2024-02-19T20:41:42.123 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-19T20:41:42.124 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566727 --verbose --owner scheduled_yuriw@teuthology --description rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} --name yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566727/orig.config.yaml 2024-02-19T20:41:42.157 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T20:41:42.232 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566727 branch: pacific-release description: rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} email: null first_in_suite: false job_id: '7566727' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: pacific-release ceph: conf: client: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mgrc: 20 debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon warn on pool no app: false paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 crush_tunables: jewel flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(MON_DOWN\) - \(OSDMAP_FLAGS\) - flag\(s\) set - \(CACHE_POOL_NO_HIT_SET\) - \(PG_ - \(OSD_ - mons down: null - reached quota - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) mon_bind_msgr2: false sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: pacific-release sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 264 sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: pacific-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 targets: smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKXeXo44orPSizxUoHEwGL0jM2Z+DSjc1UXCU9yhbhT8G1mP2VXCzJWMGf23UNEZln9z9RXM6QIFHFdgLfHOolQ= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA6jJ4zYMBuiRZrks/R4fN/SUBXMxsHBdM6sjXLOjLfzYKXFLDmg21GyuY++/Envxy0Q870W5QaMOmuxbSLPpmg= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_19:25:49 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-19T20:41:42.232 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa; will attempt to use it 2024-02-19T20:41:42.233 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks 2024-02-19T20:41:42.233 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-19T20:41:42.234 INFO:teuthology.task.internal:Checking packages... 2024-02-19T20:41:42.262 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '81bd20d634209c7cb82c18be12b4b5a05643ebf1' 2024-02-19T20:41:42.262 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T20:41:42.262 INFO:teuthology.packaging:ref: None 2024-02-19T20:41:42.262 INFO:teuthology.packaging:tag: None 2024-02-19T20:41:42.262 INFO:teuthology.packaging:branch: pacific-release 2024-02-19T20:41:42.262 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T20:41:42.262 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific-release 2024-02-19T20:41:42.417 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-548.g81bd20d6 2024-02-19T20:41:42.419 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-19T20:41:42.425 INFO:teuthology.task.internal:no buildpackages task found 2024-02-19T20:41:42.425 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-19T20:41:42.461 INFO:teuthology.task.internal:Saving configuration 2024-02-19T20:41:42.474 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-19T20:41:42.499 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T20:41:42.524 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566727', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 20:35:52.352211', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKXeXo44orPSizxUoHEwGL0jM2Z+DSjc1UXCU9yhbhT8G1mP2VXCzJWMGf23UNEZln9z9RXM6QIFHFdgLfHOolQ='} 2024-02-19T20:41:42.552 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566727', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 20:35:52.350719', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA6jJ4zYMBuiRZrks/R4fN/SUBXMxsHBdM6sjXLOjLfzYKXFLDmg21GyuY++/Envxy0Q870W5QaMOmuxbSLPpmg='} 2024-02-19T20:41:42.552 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-19T20:41:42.558 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-02-19T20:41:42.558 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-02-19T20:41:42.558 INFO:teuthology.run_tasks:Running task console_log... 2024-02-19T20:41:42.639 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd13c6730d0>, signals=[15]) 2024-02-19T20:41:42.639 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-19T20:41:42.646 INFO:teuthology.task.internal:Opening connections... 2024-02-19T20:41:42.646 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2024-02-19T20:41:42.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:41:42.727 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2024-02-19T20:41:42.729 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:41:42.805 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-19T20:41:42.812 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2024-02-19T20:41:42.835 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2024-02-19T20:41:42.835 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2024-02-19T20:41:42.901 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2024-02-19T20:41:42.901 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="8" 2024-02-19T20:41:42.901 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2024-02-19T20:41:42.902 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2024-02-19T20:41:42.902 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="8" 2024-02-19T20:41:42.902 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el8" 2024-02-19T20:41:42.902 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T20:41:42.902 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2024-02-19T20:41:42.902 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T20:41:42.903 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2024-02-19T20:41:42.903 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T20:41:42.903 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T20:41:42.903 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T20:41:42.904 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2024-02-19T20:41:42.926 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2024-02-19T20:41:42.949 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2024-02-19T20:41:42.949 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2024-02-19T20:41:43.010 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2024-02-19T20:41:43.010 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="8" 2024-02-19T20:41:43.010 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2024-02-19T20:41:43.010 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2024-02-19T20:41:43.010 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="8" 2024-02-19T20:41:43.011 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el8" 2024-02-19T20:41:43.011 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T20:41:43.011 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2024-02-19T20:41:43.011 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T20:41:43.011 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2024-02-19T20:41:43.011 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T20:41:43.011 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T20:41:43.011 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T20:41:43.011 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2024-02-19T20:41:43.035 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-19T20:41:43.048 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-19T20:41:43.077 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-19T20:41:43.078 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2024-02-19T20:41:43.080 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2024-02-19T20:41:43.104 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-19T20:41:43.122 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-19T20:41:43.122 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2024-02-19T20:41:43.147 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2024-02-19T20:41:43.186 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-19T20:41:43.243 INFO:teuthology.run_tasks:Running task kernel... 2024-02-19T20:41:43.258 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-19T20:41:43.259 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-19T20:41:43.259 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T20:41:43.259 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T20:41:43.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T20:41:43.280 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-02-19T20:41:43.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T20:41:43.281 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2024-02-19T20:41:43.340 INFO:teuthology.orchestra.run.smithi163.stdout:4.18.0-489.el8.x86_64 2024-02-19T20:41:43.341 INFO:teuthology.task.kernel:Running kernel on smithi163: 4.18.0-489.el8.x86_64 2024-02-19T20:41:43.341 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum install -y kernel 2024-02-19T20:41:43.342 INFO:teuthology.orchestra.run.smithi193.stdout:4.18.0-489.el8.x86_64 2024-02-19T20:41:43.342 INFO:teuthology.task.kernel:Running kernel on smithi193: 4.18.0-489.el8.x86_64 2024-02-19T20:41:43.342 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum install -y kernel 2024-02-19T20:41:47.070 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 8 - AppStream 22 MB/s | 27 MB 00:01 2024-02-19T20:41:47.208 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - AppStream 21 MB/s | 27 MB 00:01 2024-02-19T20:41:52.578 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 8 - BaseOS 17 MB/s | 10 MB 00:00 2024-02-19T20:41:52.641 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - BaseOS 16 MB/s | 10 MB 00:00 2024-02-19T20:41:54.462 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 8 - Extras 63 kB/s | 18 kB 00:00 2024-02-19T20:41:54.716 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - Extras 38 kB/s | 18 kB 00:00 2024-02-19T20:41:55.029 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 8 - Extras common packages 15 kB/s | 6.9 kB 00:00 2024-02-19T20:41:55.267 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 6.9 kB 00:00 2024-02-19T20:41:55.356 INFO:teuthology.orchestra.run.smithi193.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2024-02-19T20:41:55.560 INFO:teuthology.orchestra.run.smithi163.stdout:Copr repo for python3-asyncssh owned by ceph 17 kB/s | 3.5 kB 00:00 2024-02-19T20:41:57.313 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 9.7 MB/s | 16 MB 00:01 2024-02-19T20:42:01.077 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 436 kB/s | 24 kB 00:00 2024-02-19T20:42:04.074 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T20:42:04.074 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T20:42:04.191 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:42:04.192 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:42:04.192 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-02-19T20:42:04.192 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:42:04.192 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:42:04.192 INFO:teuthology.orchestra.run.smithi163.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T20:42:04.192 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-02-19T20:42:04.192 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T20:42:04.192 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T20:42:04.192 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:42:04.192 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:42:04.193 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:42:04.193 INFO:teuthology.orchestra.run.smithi163.stdout:Install 3 Packages 2024-02-19T20:42:04.193 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:42:04.193 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 90 M 2024-02-19T20:42:04.193 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 97 M 2024-02-19T20:42:04.193 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:42:04.622 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 1.8 MB/s | 16 MB 00:09 2024-02-19T20:42:04.787 INFO:teuthology.orchestra.run.smithi163.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 25 MB/s | 10 MB 00:00 2024-02-19T20:42:05.366 INFO:teuthology.orchestra.run.smithi163.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 37 MB/s | 36 MB 00:00 2024-02-19T20:42:05.854 INFO:teuthology.orchestra.run.smithi163.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 29 MB/s | 43 MB 00:01 2024-02-19T20:42:05.854 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:42:05.855 INFO:teuthology.orchestra.run.smithi163.stdout:Total 54 MB/s | 90 MB 00:01 2024-02-19T20:42:06.533 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:42:06.979 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:42:06.979 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:42:08.215 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:42:08.239 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:42:08.421 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 427 kB/s | 24 kB 00:00 2024-02-19T20:42:11.039 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:42:11.393 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T20:42:11.394 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T20:42:11.497 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:42:11.498 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:42:11.498 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-02-19T20:42:11.498 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:42:11.498 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:42:11.498 INFO:teuthology.orchestra.run.smithi193.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T20:42:11.498 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-02-19T20:42:11.498 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T20:42:11.498 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T20:42:11.498 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:42:11.499 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:42:11.499 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:42:11.499 INFO:teuthology.orchestra.run.smithi193.stdout:Install 3 Packages 2024-02-19T20:42:11.499 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:42:11.499 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 90 M 2024-02-19T20:42:11.499 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 97 M 2024-02-19T20:42:11.500 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:42:11.542 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T20:42:12.056 INFO:teuthology.orchestra.run.smithi193.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 28 MB/s | 10 MB 00:00 2024-02-19T20:42:12.565 INFO:teuthology.orchestra.run.smithi193.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 41 MB/s | 36 MB 00:00 2024-02-19T20:42:12.999 INFO:teuthology.orchestra.run.smithi193.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 33 MB/s | 43 MB 00:01 2024-02-19T20:42:12.999 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:42:12.999 INFO:teuthology.orchestra.run.smithi193.stdout:Total 60 MB/s | 90 MB 00:01 2024-02-19T20:42:13.681 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:42:14.102 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:42:14.102 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:42:15.120 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T20:42:15.338 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:42:15.355 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:42:15.507 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T20:42:18.123 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:42:18.600 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T20:42:18.933 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T20:42:19.324 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T20:42:22.132 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T20:42:22.500 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T20:42:25.926 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T20:42:26.232 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T20:42:41.327 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T20:42:41.350 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T20:42:41.698 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T20:42:41.698 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T20:42:41.698 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T20:42:42.166 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T20:42:42.166 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:42:42.166 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:42:42.166 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T20:42:42.166 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T20:42:42.166 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:42:42.166 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:42:42.338 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T20:42:42.560 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T20:42:42.560 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T20:42:42.560 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T20:42:42.560 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T20:42:42.560 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T20:42:42.561 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-19T20:42:42.561 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi163.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T20:42:42.561 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T20:42:42.771 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T20:42:42.772 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T20:42:42.772 DEBUG:teuthology.orchestra.run.smithi163:> sudo rpm -qi grub2-tools 2024-02-19T20:42:42.806 INFO:teuthology.orchestra.run.smithi163.stdout:Name : grub2-tools 2024-02-19T20:42:42.807 INFO:teuthology.orchestra.run.smithi163.stdout:Epoch : 1 2024-02-19T20:42:42.807 INFO:teuthology.orchestra.run.smithi163.stdout:Version : 2.02 2024-02-19T20:42:42.807 INFO:teuthology.orchestra.run.smithi163.stdout:Release : 129.el8 2024-02-19T20:42:42.807 INFO:teuthology.orchestra.run.smithi163.stdout:Architecture: x86_64 2024-02-19T20:42:42.807 INFO:teuthology.orchestra.run.smithi163.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T20:42:42.807 INFO:teuthology.orchestra.run.smithi163.stdout:Group : System Environment/Base 2024-02-19T20:42:42.807 INFO:teuthology.orchestra.run.smithi163.stdout:Size : 9265586 2024-02-19T20:42:42.807 INFO:teuthology.orchestra.run.smithi163.stdout:License : GPLv3+ 2024-02-19T20:42:42.807 INFO:teuthology.orchestra.run.smithi163.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T20:42:42.807 INFO:teuthology.orchestra.run.smithi163.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T20:42:42.807 INFO:teuthology.orchestra.run.smithi163.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T20:42:42.807 INFO:teuthology.orchestra.run.smithi163.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T20:42:42.808 INFO:teuthology.orchestra.run.smithi163.stdout:Relocations : (not relocatable) 2024-02-19T20:42:42.808 INFO:teuthology.orchestra.run.smithi163.stdout:Packager : CentOS BuildSystem 2024-02-19T20:42:42.808 INFO:teuthology.orchestra.run.smithi163.stdout:Vendor : CentOS 2024-02-19T20:42:42.808 INFO:teuthology.orchestra.run.smithi163.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T20:42:42.808 INFO:teuthology.orchestra.run.smithi163.stdout:Summary : Support tools for GRUB. 2024-02-19T20:42:42.808 INFO:teuthology.orchestra.run.smithi163.stdout:Description : 2024-02-19T20:42:42.808 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:42:42.808 INFO:teuthology.orchestra.run.smithi163.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T20:42:42.808 INFO:teuthology.orchestra.run.smithi163.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T20:42:42.808 INFO:teuthology.orchestra.run.smithi163.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T20:42:42.808 INFO:teuthology.orchestra.run.smithi163.stdout:hardware devices. 2024-02-19T20:42:42.809 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:42:42.809 INFO:teuthology.orchestra.run.smithi163.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T20:42:42.811 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T20:42:42.811 INFO:teuthology.task.kernel:Updating grub on smithi163 to boot 4.18.0-539.el8.x86_64 2024-02-19T20:42:42.812 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T20:42:44.248 INFO:teuthology.orchestra.run.smithi163.stderr:Generating grub configuration file ... 2024-02-19T20:42:47.067 INFO:teuthology.orchestra.run.smithi163.stderr:done 2024-02-19T20:42:47.070 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2024-02-19T20:42:47.102 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.vSLNHz1c5k 2024-02-19T20:42:47.102 DEBUG:teuthology.orchestra.run.smithi163:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vSLNHz1c5k 2024-02-19T20:42:47.175 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.vSLNHz1c5k 2024-02-19T20:42:47.323 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.vSLNHz1c5k is 6KB 2024-02-19T20:42:47.372 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.vSLNHz1c5k 2024-02-19T20:42:47.389 DEBUG:teuthology.orchestra.run.smithi163:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T20:42:47.462 INFO:teuthology.orchestra.run.smithi163.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T20:42:47.462 INFO:teuthology.orchestra.run.smithi163.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T20:42:47.462 INFO:teuthology.orchestra.run.smithi163.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T20:42:47.462 INFO:teuthology.orchestra.run.smithi163.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T20:42:47.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T20:42:47.613 DEBUG:teuthology.orchestra.run.smithi163:> sudo shutdown -r now 2024-02-19T20:42:48.051 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T20:42:48.075 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T20:42:48.425 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T20:42:48.425 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T20:42:48.425 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T20:42:48.915 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T20:42:48.915 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:42:48.916 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:42:48.916 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T20:42:48.916 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T20:42:48.916 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:42:48.916 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:42:49.065 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T20:42:49.280 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T20:42:49.280 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T20:42:49.280 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T20:42:49.280 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T20:42:49.280 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T20:42:49.280 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-19T20:42:49.280 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi193.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T20:42:49.281 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T20:42:49.491 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T20:42:49.492 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T20:42:49.492 DEBUG:teuthology.orchestra.run.smithi193:> sudo rpm -qi grub2-tools 2024-02-19T20:42:49.526 INFO:teuthology.orchestra.run.smithi193.stdout:Name : grub2-tools 2024-02-19T20:42:49.526 INFO:teuthology.orchestra.run.smithi193.stdout:Epoch : 1 2024-02-19T20:42:49.526 INFO:teuthology.orchestra.run.smithi193.stdout:Version : 2.02 2024-02-19T20:42:49.526 INFO:teuthology.orchestra.run.smithi193.stdout:Release : 129.el8 2024-02-19T20:42:49.526 INFO:teuthology.orchestra.run.smithi193.stdout:Architecture: x86_64 2024-02-19T20:42:49.526 INFO:teuthology.orchestra.run.smithi193.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T20:42:49.526 INFO:teuthology.orchestra.run.smithi193.stdout:Group : System Environment/Base 2024-02-19T20:42:49.526 INFO:teuthology.orchestra.run.smithi193.stdout:Size : 9265586 2024-02-19T20:42:49.527 INFO:teuthology.orchestra.run.smithi193.stdout:License : GPLv3+ 2024-02-19T20:42:49.527 INFO:teuthology.orchestra.run.smithi193.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T20:42:49.527 INFO:teuthology.orchestra.run.smithi193.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T20:42:49.527 INFO:teuthology.orchestra.run.smithi193.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T20:42:49.527 INFO:teuthology.orchestra.run.smithi193.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T20:42:49.527 INFO:teuthology.orchestra.run.smithi193.stdout:Relocations : (not relocatable) 2024-02-19T20:42:49.527 INFO:teuthology.orchestra.run.smithi193.stdout:Packager : CentOS BuildSystem 2024-02-19T20:42:49.527 INFO:teuthology.orchestra.run.smithi193.stdout:Vendor : CentOS 2024-02-19T20:42:49.527 INFO:teuthology.orchestra.run.smithi193.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T20:42:49.527 INFO:teuthology.orchestra.run.smithi193.stdout:Summary : Support tools for GRUB. 2024-02-19T20:42:49.527 INFO:teuthology.orchestra.run.smithi193.stdout:Description : 2024-02-19T20:42:49.527 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:42:49.527 INFO:teuthology.orchestra.run.smithi193.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T20:42:49.527 INFO:teuthology.orchestra.run.smithi193.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T20:42:49.527 INFO:teuthology.orchestra.run.smithi193.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T20:42:49.528 INFO:teuthology.orchestra.run.smithi193.stdout:hardware devices. 2024-02-19T20:42:49.528 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:42:49.528 INFO:teuthology.orchestra.run.smithi193.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T20:42:49.529 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T20:42:49.529 INFO:teuthology.task.kernel:Updating grub on smithi193 to boot 4.18.0-539.el8.x86_64 2024-02-19T20:42:49.529 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T20:42:51.168 INFO:teuthology.orchestra.run.smithi193.stderr:Generating grub configuration file ... 2024-02-19T20:42:52.957 INFO:teuthology.orchestra.run.smithi193.stderr:done 2024-02-19T20:42:52.958 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2024-02-19T20:42:52.982 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.tKuZjmz0tJ 2024-02-19T20:42:52.982 DEBUG:teuthology.orchestra.run.smithi193:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tKuZjmz0tJ 2024-02-19T20:42:53.056 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.tKuZjmz0tJ 2024-02-19T20:42:53.194 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.tKuZjmz0tJ is 6KB 2024-02-19T20:42:53.243 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.tKuZjmz0tJ 2024-02-19T20:42:53.260 DEBUG:teuthology.orchestra.run.smithi193:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T20:42:53.334 INFO:teuthology.orchestra.run.smithi193.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T20:42:53.334 INFO:teuthology.orchestra.run.smithi193.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T20:42:53.335 INFO:teuthology.orchestra.run.smithi193.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T20:42:53.335 INFO:teuthology.orchestra.run.smithi193.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T20:42:53.336 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T20:42:53.506 DEBUG:teuthology.orchestra.run.smithi193:> sudo shutdown -r now 2024-02-19T20:43:17.625 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-19T20:43:17.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-02-19T20:43:17.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:23.513 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-19T20:43:23.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-02-19T20:43:23.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:36.124 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-02-19T20:43:45.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-02-19T20:43:45.133 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:48.188 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-02-19T20:44:00.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-02-19T20:44:00.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:03.260 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-02-19T20:44:18.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-02-19T20:44:18.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:23.520 DEBUG:teuthology.orchestra.remote:timed out 2024-02-19T20:44:32.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-02-19T20:44:32.529 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:32.867 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-02-19T20:44:33.244 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-02-19T20:44:33.244 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-539.el8.x86_64"... 2024-02-19T20:44:33.244 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2024-02-19T20:44:33.261 INFO:teuthology.orchestra.run.smithi193.stdout:4.18.0-539.el8.x86_64 2024-02-19T20:44:33.261 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T20:44:33.261 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T20:44:33.261 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T20:44:34.261 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-19T20:44:34.262 DEBUG:teuthology.orchestra.run.smithi193:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T20:44:34.347 INFO:teuthology.orchestra.run.smithi193.stdout:ttyS1 2024-02-19T20:44:34.365 DEBUG:teuthology.parallel:result is None 2024-02-19T20:44:50.184 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-02-19T20:44:50.491 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-02-19T20:44:50.491 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-539.el8.x86_64"... 2024-02-19T20:44:50.492 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-02-19T20:44:50.509 INFO:teuthology.orchestra.run.smithi163.stdout:4.18.0-539.el8.x86_64 2024-02-19T20:44:50.510 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T20:44:50.510 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T20:44:50.510 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T20:44:51.512 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-19T20:44:51.512 DEBUG:teuthology.orchestra.run.smithi163:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T20:44:51.584 INFO:teuthology.orchestra.run.smithi163.stdout:ttyS1 2024-02-19T20:44:51.601 DEBUG:teuthology.parallel:result is None 2024-02-19T20:44:51.601 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-19T20:44:51.608 INFO:teuthology.task.internal:Creating test directory... 2024-02-19T20:44:51.609 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T20:44:51.611 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T20:44:51.635 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-19T20:44:51.731 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-19T20:44:51.746 INFO:teuthology.task.internal:Creating archive directory... 2024-02-19T20:44:51.747 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T20:44:51.750 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T20:44:51.790 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-19T20:44:51.800 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-19T20:44:51.800 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T20:44:51.809 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T20:44:51.848 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T20:44:51.856 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T20:44:51.858 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T20:44:51.867 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T20:44:51.868 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-19T20:44:51.875 INFO:teuthology.task.internal:Configuring sudo... 2024-02-19T20:44:51.875 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T20:44:51.902 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T20:44:51.937 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-19T20:44:51.953 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-19T20:44:51.954 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T20:44:51.969 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T20:44:51.997 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T20:44:52.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T20:44:52.119 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T20:44:52.203 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T20:44:52.243 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T20:44:52.243 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T20:44:52.311 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T20:44:52.360 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T20:44:52.419 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T20:44:52.505 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T20:44:52.551 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-02-19T20:44:52.551 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T20:44:52.620 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2024-02-19T20:44:52.622 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2024-02-19T20:44:52.700 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T20:44:52.735 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T20:44:53.066 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-19T20:44:53.075 INFO:teuthology.task.internal:Starting timer... 2024-02-19T20:44:53.075 INFO:teuthology.run_tasks:Running task pcp... 2024-02-19T20:44:53.183 INFO:teuthology.run_tasks:Running task selinux... 2024-02-19T20:44:53.307 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-19T20:44:53.307 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2024-02-19T20:44:53.388 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: 2024-02-19T20:44:53.390 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2024-02-19T20:44:53.457 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: 2024-02-19T20:44:53.459 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-19T20:44:53.459 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2024-02-19T20:44:53.486 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2024-02-19T20:44:53.486 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2024-02-19T20:44:53.532 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2024-02-19T20:44:53.532 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi163.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2024-02-19T20:44:53.532 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T20:44:53.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T20:44:53.559 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T20:44:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T20:44:53.600 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-19T20:44:53.600 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2024-02-19T20:44:53.630 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2024-02-19T20:44:53.666 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-19T20:44:53.675 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-19T20:44:53.692 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-19T20:44:53.694 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi163.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-19T20:50:36.507 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi163.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2024-02-19T20:50:36.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-02-19T20:50:36.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:50:36.585 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-02-19T20:50:36.653 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-02-19T20:50:36.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-02-19T20:50:36.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:50:36.724 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-02-19T20:50:36.797 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-02-19T20:50:36.797 INFO:teuthology.run_tasks:Running task clock... 2024-02-19T20:50:36.809 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-19T20:50:36.809 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T20:50:36.810 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T20:50:36.812 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T20:50:36.812 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T20:50:36.841 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T20:50:36.854 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T20:50:36.868 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T20:50:36.881 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T20:50:36.921 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: ntpd: command not found 2024-02-19T20:50:36.932 INFO:teuthology.orchestra.run.smithi163.stdout:506 Cannot talk to daemon 2024-02-19T20:50:36.946 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T20:50:36.959 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T20:50:36.960 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: ntpd: command not found 2024-02-19T20:50:36.972 INFO:teuthology.orchestra.run.smithi193.stdout:506 Cannot talk to daemon 2024-02-19T20:50:36.986 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T20:50:36.999 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T20:50:37.022 INFO:teuthology.orchestra.run.smithi163.stderr:bash: ntpq: command not found 2024-02-19T20:50:37.025 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T20:50:37.026 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-02-19T20:50:37.026 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T20:50:37.026 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T20:50:37.026 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T20:50:37.026 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T20:50:37.068 INFO:teuthology.orchestra.run.smithi193.stderr:bash: ntpq: command not found 2024-02-19T20:50:37.073 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T20:50:37.073 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-02-19T20:50:37.073 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T20:50:37.073 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T20:50:37.073 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T20:50:37.073 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T20:50:37.073 INFO:teuthology.run_tasks:Running task pexec... 2024-02-19T20:50:37.082 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-19T20:50:37.082 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi163.front.sepia.ceph.com 2024-02-19T20:50:37.083 DEBUG:teuthology.orchestra.run.smithi163:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T20:50:37.083 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi193.front.sepia.ceph.com 2024-02-19T20:50:37.083 DEBUG:teuthology.orchestra.run.smithi193:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T20:50:37.870 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:21 ago on Mon 19 Feb 2024 08:49:16 PM UTC. 2024-02-19T20:50:37.899 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:21 ago on Mon 19 Feb 2024 08:49:16 PM UTC. 2024-02-19T20:50:39.866 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:50:39.867 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:50:39.867 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-02-19T20:50:39.867 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:50:39.867 INFO:teuthology.orchestra.run.smithi163.stdout:Resetting modules: 2024-02-19T20:50:39.867 INFO:teuthology.orchestra.run.smithi163.stdout: container-tools 2024-02-19T20:50:39.867 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:50:39.867 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:50:39.867 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:50:39.867 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:50:39.912 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:50:39.913 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:50:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-02-19T20:50:39.913 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:50:39.913 INFO:teuthology.orchestra.run.smithi193.stdout:Resetting modules: 2024-02-19T20:50:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: container-tools 2024-02-19T20:50:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:50:39.913 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:50:39.914 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:50:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:50:40.115 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:50:40.164 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:50:40.978 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:24 ago on Mon 19 Feb 2024 08:49:16 PM UTC. 2024-02-19T20:50:41.054 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:24 ago on Mon 19 Feb 2024 08:49:16 PM UTC. 2024-02-19T20:50:43.024 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:50:43.047 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:50:43.059 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:50:43.059 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-02-19T20:50:43.059 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:50:43.059 INFO:teuthology.orchestra.run.smithi163.stdout:Installing group/module packages: 2024-02-19T20:50:43.060 INFO:teuthology.orchestra.run.smithi163.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-19T20:50:43.060 INFO:teuthology.orchestra.run.smithi163.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-19T20:50:43.060 INFO:teuthology.orchestra.run.smithi163.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-19T20:50:43.060 INFO:teuthology.orchestra.run.smithi163.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-19T20:50:43.060 INFO:teuthology.orchestra.run.smithi163.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-19T20:50:43.060 INFO:teuthology.orchestra.run.smithi163.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-19T20:50:43.060 INFO:teuthology.orchestra.run.smithi163.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-19T20:50:43.060 INFO:teuthology.orchestra.run.smithi163.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-19T20:50:43.060 INFO:teuthology.orchestra.run.smithi163.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-19T20:50:43.060 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-02-19T20:50:43.060 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-19T20:50:43.060 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-19T20:50:43.060 INFO:teuthology.orchestra.run.smithi163.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-19T20:50:43.061 INFO:teuthology.orchestra.run.smithi163.stdout:Installing weak dependencies: 2024-02-19T20:50:43.061 INFO:teuthology.orchestra.run.smithi163.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-19T20:50:43.061 INFO:teuthology.orchestra.run.smithi163.stdout:Installing module profiles: 2024-02-19T20:50:43.061 INFO:teuthology.orchestra.run.smithi163.stdout: container-tools/common 2024-02-19T20:50:43.061 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:50:43.061 INFO:teuthology.orchestra.run.smithi163.stdout:Enabling module streams: 2024-02-19T20:50:43.061 INFO:teuthology.orchestra.run.smithi163.stdout: container-tools rhel8 2024-02-19T20:50:43.061 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:50:43.061 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:50:43.061 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:50:43.061 INFO:teuthology.orchestra.run.smithi163.stdout:Install 13 Packages 2024-02-19T20:50:43.061 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:50:43.062 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 28 M 2024-02-19T20:50:43.062 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 92 M 2024-02-19T20:50:43.062 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:50:43.082 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:50:43.082 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-02-19T20:50:43.082 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:50:43.083 INFO:teuthology.orchestra.run.smithi193.stdout:Installing group/module packages: 2024-02-19T20:50:43.083 INFO:teuthology.orchestra.run.smithi193.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-19T20:50:43.083 INFO:teuthology.orchestra.run.smithi193.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-19T20:50:43.083 INFO:teuthology.orchestra.run.smithi193.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-19T20:50:43.083 INFO:teuthology.orchestra.run.smithi193.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-19T20:50:43.083 INFO:teuthology.orchestra.run.smithi193.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-19T20:50:43.083 INFO:teuthology.orchestra.run.smithi193.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-19T20:50:43.083 INFO:teuthology.orchestra.run.smithi193.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-19T20:50:43.083 INFO:teuthology.orchestra.run.smithi193.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-19T20:50:43.083 INFO:teuthology.orchestra.run.smithi193.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-19T20:50:43.083 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-02-19T20:50:43.083 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-19T20:50:43.083 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-19T20:50:43.083 INFO:teuthology.orchestra.run.smithi193.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-19T20:50:43.083 INFO:teuthology.orchestra.run.smithi193.stdout:Installing weak dependencies: 2024-02-19T20:50:43.084 INFO:teuthology.orchestra.run.smithi193.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-19T20:50:43.084 INFO:teuthology.orchestra.run.smithi193.stdout:Installing module profiles: 2024-02-19T20:50:43.084 INFO:teuthology.orchestra.run.smithi193.stdout: container-tools/common 2024-02-19T20:50:43.084 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:50:43.084 INFO:teuthology.orchestra.run.smithi193.stdout:Enabling module streams: 2024-02-19T20:50:43.084 INFO:teuthology.orchestra.run.smithi193.stdout: container-tools rhel8 2024-02-19T20:50:43.084 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:50:43.084 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:50:43.084 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:50:43.084 INFO:teuthology.orchestra.run.smithi193.stdout:Install 13 Packages 2024-02-19T20:50:43.084 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:50:43.085 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 28 M 2024-02-19T20:50:43.085 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 92 M 2024-02-19T20:50:43.085 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:50:43.390 INFO:teuthology.orchestra.run.smithi163.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 2.3 MB/s | 688 kB 00:00 2024-02-19T20:50:43.423 INFO:teuthology.orchestra.run.smithi163.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 3.1 MB/s | 1.0 MB 00:00 2024-02-19T20:50:43.449 INFO:teuthology.orchestra.run.smithi163.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 648 kB/s | 38 kB 00:00 2024-02-19T20:50:43.482 INFO:teuthology.orchestra.run.smithi163.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 4.3 MB/s | 257 kB 00:00 2024-02-19T20:50:43.551 INFO:teuthology.orchestra.run.smithi193.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 2.5 MB/s | 688 kB 00:00 2024-02-19T20:50:43.558 INFO:teuthology.orchestra.run.smithi163.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 2.3 MB/s | 180 kB 00:00 2024-02-19T20:50:43.585 INFO:teuthology.orchestra.run.smithi193.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 3.3 MB/s | 1.0 MB 00:00 2024-02-19T20:50:43.611 INFO:teuthology.orchestra.run.smithi193.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 646 kB/s | 38 kB 00:00 2024-02-19T20:50:43.625 INFO:teuthology.orchestra.run.smithi163.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 370 kB/s | 25 kB 00:00 2024-02-19T20:50:43.644 INFO:teuthology.orchestra.run.smithi193.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 4.3 MB/s | 257 kB 00:00 2024-02-19T20:50:43.683 INFO:teuthology.orchestra.run.smithi163.stdout:(7/13): netavark-1.10.2-1.module_el8+885+7da147 18 MB/s | 4.3 MB 00:00 2024-02-19T20:50:43.702 INFO:teuthology.orchestra.run.smithi193.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 3.0 MB/s | 180 kB 00:00 2024-02-19T20:50:43.709 INFO:teuthology.orchestra.run.smithi163.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.1 MB/s | 94 kB 00:00 2024-02-19T20:50:43.761 INFO:teuthology.orchestra.run.smithi193.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 425 kB/s | 25 kB 00:00 2024-02-19T20:50:43.818 INFO:teuthology.orchestra.run.smithi163.stdout:(9/13): buildah-1.33.5-1.module_el8+885+7da147f 14 MB/s | 9.7 MB 00:00 2024-02-19T20:50:43.828 INFO:teuthology.orchestra.run.smithi193.stdout:(7/13): netavark-1.10.2-1.module_el8+885+7da147 20 MB/s | 4.3 MB 00:00 2024-02-19T20:50:43.853 INFO:teuthology.orchestra.run.smithi193.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.0 MB/s | 94 kB 00:00 2024-02-19T20:50:43.876 INFO:teuthology.orchestra.run.smithi163.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 905 kB/s | 53 kB 00:00 2024-02-19T20:50:43.918 INFO:teuthology.orchestra.run.smithi163.stdout:(11/13): toolbox-0.0.99.5-1.module_el8+847+7863 12 MB/s | 2.5 MB 00:00 2024-02-19T20:50:43.943 INFO:teuthology.orchestra.run.smithi163.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 614 kB/s | 41 kB 00:00 2024-02-19T20:50:43.970 INFO:teuthology.orchestra.run.smithi193.stdout:(9/13): buildah-1.33.5-1.module_el8+885+7da147f 14 MB/s | 9.7 MB 00:00 2024-02-19T20:50:44.029 INFO:teuthology.orchestra.run.smithi193.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 903 kB/s | 53 kB 00:00 2024-02-19T20:50:44.111 INFO:teuthology.orchestra.run.smithi163.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 21 MB/s | 8.8 MB 00:00 2024-02-19T20:50:44.111 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:50:44.111 INFO:teuthology.orchestra.run.smithi163.stdout:Total 26 MB/s | 28 MB 00:01 2024-02-19T20:50:44.113 INFO:teuthology.orchestra.run.smithi193.stdout:(11/13): toolbox-0.0.99.5-1.module_el8+847+7863 9.8 MB/s | 2.5 MB 00:00 2024-02-19T20:50:44.138 INFO:teuthology.orchestra.run.smithi193.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 377 kB/s | 41 kB 00:00 2024-02-19T20:50:44.247 INFO:teuthology.orchestra.run.smithi193.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 21 MB/s | 8.8 MB 00:00 2024-02-19T20:50:44.247 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:50:44.247 INFO:teuthology.orchestra.run.smithi193.stdout:Total 24 MB/s | 28 MB 00:01 2024-02-19T20:50:44.381 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:50:44.402 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:50:44.402 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:50:44.516 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:50:44.536 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:50:44.537 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:50:44.662 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:50:44.665 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:50:44.802 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:50:44.806 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:50:45.322 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:50:45.473 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:50:46.268 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-19T20:50:46.419 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-19T20:50:46.478 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T20:50:46.640 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-19T20:50:46.640 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T20:50:46.810 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-19T20:50:46.835 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-19T20:50:46.995 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-19T20:50:47.076 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-19T20:50:47.213 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-19T20:50:47.675 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-19T20:50:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-19T20:50:48.643 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-19T20:50:48.808 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-19T20:50:48.868 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-19T20:50:49.023 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-19T20:50:49.062 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-19T20:50:49.217 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-19T20:50:49.482 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-19T20:50:49.708 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T20:50:49.712 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-19T20:50:49.922 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T20:50:49.923 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T20:50:50.114 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T20:50:50.166 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T20:50:50.368 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T20:50:50.542 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T20:50:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-19T20:50:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T20:50:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-19T20:50:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-19T20:50:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-19T20:50:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-19T20:50:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-19T20:50:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-19T20:50:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-19T20:50:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-19T20:50:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T20:50:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T20:50:50.777 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T20:50:50.777 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-19T20:50:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T20:50:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-19T20:50:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-19T20:50:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-19T20:50:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-19T20:50:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-19T20:50:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-19T20:50:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-19T20:50:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-19T20:50:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T20:50:50.778 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T20:50:51.095 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-19T20:50:51.095 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:50:51.095 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:50:51.095 INFO:teuthology.orchestra.run.smithi163.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-19T20:50:51.095 INFO:teuthology.orchestra.run.smithi163.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-19T20:50:51.095 INFO:teuthology.orchestra.run.smithi163.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-19T20:50:51.095 INFO:teuthology.orchestra.run.smithi163.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-19T20:50:51.095 INFO:teuthology.orchestra.run.smithi163.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-19T20:50:51.095 INFO:teuthology.orchestra.run.smithi163.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-19T20:50:51.095 INFO:teuthology.orchestra.run.smithi163.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-19T20:50:51.096 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-19T20:50:51.096 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-19T20:50:51.096 INFO:teuthology.orchestra.run.smithi163.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-19T20:50:51.096 INFO:teuthology.orchestra.run.smithi163.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-19T20:50:51.096 INFO:teuthology.orchestra.run.smithi163.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-19T20:50:51.096 INFO:teuthology.orchestra.run.smithi163.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-19T20:50:51.096 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:50:51.096 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:50:51.339 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-19T20:50:51.339 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:50:51.340 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:50:51.340 INFO:teuthology.orchestra.run.smithi193.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-19T20:50:51.340 INFO:teuthology.orchestra.run.smithi193.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-19T20:50:51.340 INFO:teuthology.orchestra.run.smithi193.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-19T20:50:51.340 INFO:teuthology.orchestra.run.smithi193.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-19T20:50:51.340 INFO:teuthology.orchestra.run.smithi193.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-19T20:50:51.340 INFO:teuthology.orchestra.run.smithi193.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-19T20:50:51.340 INFO:teuthology.orchestra.run.smithi193.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-19T20:50:51.340 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-19T20:50:51.340 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-19T20:50:51.340 INFO:teuthology.orchestra.run.smithi193.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-19T20:50:51.340 INFO:teuthology.orchestra.run.smithi193.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-19T20:50:51.340 INFO:teuthology.orchestra.run.smithi193.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-19T20:50:51.341 INFO:teuthology.orchestra.run.smithi193.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-19T20:50:51.341 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:50:51.341 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:50:51.386 DEBUG:teuthology.parallel:result is None 2024-02-19T20:50:51.682 DEBUG:teuthology.parallel:result is None 2024-02-19T20:50:51.683 INFO:teuthology.run_tasks:Running task install... 2024-02-19T20:50:51.693 DEBUG:teuthology.task.install:project ceph 2024-02-19T20:50:51.693 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'}} 2024-02-19T20:50:51.693 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'} 2024-02-19T20:50:51.693 INFO:teuthology.task.install:Using flavor: default 2024-02-19T20:50:51.703 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-19T20:50:51.703 INFO:teuthology.task.install:extra packages: [] 2024-02-19T20:50:51.704 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1', 'tag': None, 'wait_for_package': False} 2024-02-19T20:50:51.704 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T20:50:51.707 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1', 'tag': None, 'wait_for_package': False} 2024-02-19T20:50:51.708 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T20:50:51.858 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/ 2024-02-19T20:50:51.858 INFO:teuthology.task.install.rpm:Package version is 16.2.14-548.g81bd20d6 2024-02-19T20:50:51.873 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/ 2024-02-19T20:50:51.873 INFO:teuthology.task.install.rpm:Package version is 16.2.14-548.g81bd20d6 2024-02-19T20:50:51.979 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T20:50:51.979 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T20:50:51.980 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T20:50:52.008 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T20:50:52.008 DEBUG:teuthology.orchestra.run.smithi163:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81bd20d634209c7cb82c18be12b4b5a05643ebf1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T20:50:52.076 DEBUG:teuthology.orchestra.run.smithi163:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T20:50:52.153 DEBUG:teuthology.orchestra.run.smithi163:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T20:50:52.186 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/pacific/81bd20d634209c7cb82c18be12b4b5a05643ebf1/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T20:50:52.187 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-02-19T20:50:52.187 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T20:50:52.214 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T20:50:52.215 DEBUG:teuthology.orchestra.run.smithi193:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81bd20d634209c7cb82c18be12b4b5a05643ebf1/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T20:50:52.219 INFO:teuthology.orchestra.run.smithi163.stdout:check_obsoletes = 1 2024-02-19T20:50:52.220 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum clean all 2024-02-19T20:50:52.284 DEBUG:teuthology.orchestra.run.smithi193:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T20:50:52.360 DEBUG:teuthology.orchestra.run.smithi193:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T20:50:52.426 INFO:teuthology.orchestra.run.smithi193.stdout:check_obsoletes = 1 2024-02-19T20:50:52.427 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum clean all 2024-02-19T20:50:52.478 INFO:teuthology.orchestra.run.smithi163.stdout:76 files removed 2024-02-19T20:50:52.506 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-radosgw 2024-02-19T20:50:52.703 INFO:teuthology.orchestra.run.smithi193.stdout:76 files removed 2024-02-19T20:50:52.731 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw 2024-02-19T20:50:53.454 INFO:teuthology.orchestra.run.smithi163.stdout:ceph packages for x86_64 142 kB/s | 76 kB 00:00 2024-02-19T20:50:53.649 INFO:teuthology.orchestra.run.smithi193.stdout:ceph packages for x86_64 150 kB/s | 76 kB 00:00 2024-02-19T20:50:53.854 INFO:teuthology.orchestra.run.smithi163.stdout:ceph noarch packages 42 kB/s | 15 kB 00:00 2024-02-19T20:50:54.042 INFO:teuthology.orchestra.run.smithi193.stdout:ceph noarch packages 43 kB/s | 15 kB 00:00 2024-02-19T20:50:54.163 INFO:teuthology.orchestra.run.smithi163.stdout:ceph source packages 5.2 kB/s | 1.4 kB 00:00 2024-02-19T20:50:54.418 INFO:teuthology.orchestra.run.smithi193.stdout:ceph source packages 4.2 kB/s | 1.4 kB 00:00 2024-02-19T20:50:55.989 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - AppStream 15 MB/s | 27 MB 00:01 2024-02-19T20:50:56.269 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 8 - AppStream 15 MB/s | 27 MB 00:01 2024-02-19T20:51:01.669 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 8 - BaseOS 19 MB/s | 10 MB 00:00 2024-02-19T20:51:01.840 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - BaseOS 10 MB/s | 10 MB 00:01 2024-02-19T20:51:03.728 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 8 - Extras 38 kB/s | 18 kB 00:00 2024-02-19T20:51:03.916 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - Extras 39 kB/s | 18 kB 00:00 2024-02-19T20:51:04.154 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 6.9 kB 00:00 2024-02-19T20:51:04.341 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 6.9 kB 00:00 2024-02-19T20:51:04.976 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 8 - PowerTools 8.9 MB/s | 4.8 MB 00:00 2024-02-19T20:51:05.097 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 8 - PowerTools 5.7 MB/s | 4.8 MB 00:00 2024-02-19T20:51:06.403 INFO:teuthology.orchestra.run.smithi163.stdout:Copr repo for python3-asyncssh owned by ceph 6.8 kB/s | 3.5 kB 00:00 2024-02-19T20:51:06.423 INFO:teuthology.orchestra.run.smithi193.stdout:Copr repo for python3-asyncssh owned by ceph 8.6 kB/s | 3.5 kB 00:00 2024-02-19T20:51:08.351 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 8.9 MB/s | 16 MB 00:01 2024-02-19T20:51:12.199 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 606 kB/s | 24 kB 00:00 2024-02-19T20:51:15.772 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:51:15.773 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:51:15.773 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-02-19T20:51:15.773 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:51:15.773 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:51:15.773 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 12 M 2024-02-19T20:51:15.773 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrading: 2024-02-19T20:51:15.773 INFO:teuthology.orchestra.run.smithi193.stdout: librados2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.7 M 2024-02-19T20:51:15.773 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 4.2 M 2024-02-19T20:51:15.773 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-02-19T20:51:15.774 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 6.1 M 2024-02-19T20:51:15.774 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 24 M 2024-02-19T20:51:15.774 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 26 k 2024-02-19T20:51:15.774 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-19T20:51:15.774 INFO:teuthology.orchestra.run.smithi193.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-19T20:51:15.774 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 825 k 2024-02-19T20:51:15.774 INFO:teuthology.orchestra.run.smithi193.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-19T20:51:15.774 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-19T20:51:15.774 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 494 k 2024-02-19T20:51:15.774 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-19T20:51:15.775 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.8 M 2024-02-19T20:51:15.775 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-19T20:51:15.775 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-19T20:51:15.775 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 45 k 2024-02-19T20:51:15.775 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 99 k 2024-02-19T20:51:15.775 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 215 k 2024-02-19T20:51:15.775 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-19T20:51:15.775 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 389 k 2024-02-19T20:51:15.775 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 369 k 2024-02-19T20:51:15.775 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 114 k 2024-02-19T20:51:15.775 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:51:15.775 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:51:15.776 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:51:15.776 INFO:teuthology.orchestra.run.smithi193.stdout:Install 21 Packages 2024-02-19T20:51:15.776 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrade 2 Packages 2024-02-19T20:51:15.776 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:51:15.776 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 58 M 2024-02-19T20:51:15.776 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:51:17.191 INFO:teuthology.orchestra.run.smithi193.stdout:(1/23): ceph-base-16.2.14-548.g81bd20d6.el8.x86 6.7 MB/s | 6.1 MB 00:00 2024-02-19T20:51:17.275 INFO:teuthology.orchestra.run.smithi193.stdout:(2/23): ceph-selinux-16.2.14-548.g81bd20d6.el8. 308 kB/s | 26 kB 00:00 2024-02-19T20:51:17.551 INFO:teuthology.orchestra.run.smithi193.stdout:(3/23): libcephfs2-16.2.14-548.g81bd20d6.el8.x8 2.9 MB/s | 825 kB 00:00 2024-02-19T20:51:17.667 INFO:teuthology.orchestra.run.smithi193.stdout:(4/23): ceph-radosgw-16.2.14-548.g81bd20d6.el8. 8.4 MB/s | 12 MB 00:01 2024-02-19T20:51:17.701 INFO:teuthology.orchestra.run.smithi193.stdout:(5/23): libradosstriper1-16.2.14-548.g81bd20d6. 3.2 MB/s | 494 kB 00:00 2024-02-19T20:51:17.768 INFO:teuthology.orchestra.run.smithi193.stdout:(6/23): python3-ceph-argparse-16.2.14-548.g81bd 676 kB/s | 45 kB 00:00 2024-02-19T20:51:17.843 INFO:teuthology.orchestra.run.smithi193.stdout:(7/23): python3-ceph-common-16.2.14-548.g81bd20 1.3 MB/s | 99 kB 00:00 2024-02-19T20:51:17.944 INFO:teuthology.orchestra.run.smithi193.stdout:(8/23): python3-cephfs-16.2.14-548.g81bd20d6.el 2.1 MB/s | 215 kB 00:00 2024-02-19T20:51:18.162 INFO:teuthology.orchestra.run.smithi193.stdout:(9/23): python3-rados-16.2.14-548.g81bd20d6.el8 1.7 MB/s | 389 kB 00:00 2024-02-19T20:51:18.495 INFO:teuthology.orchestra.run.smithi193.stdout:(10/23): librgw2-16.2.14-548.g81bd20d6.el8.x86_ 4.6 MB/s | 3.8 MB 00:00 2024-02-19T20:51:18.828 INFO:teuthology.orchestra.run.smithi193.stdout:(11/23): python3-rbd-16.2.14-548.g81bd20d6.el8. 554 kB/s | 369 kB 00:00 2024-02-19T20:51:18.854 INFO:teuthology.orchestra.run.smithi193.stdout:(12/23): python3-rgw-16.2.14-548.g81bd20d6.el8. 319 kB/s | 114 kB 00:00 2024-02-19T20:51:19.187 INFO:teuthology.orchestra.run.smithi193.stdout:(13/23): ceph-common-16.2.14-548.g81bd20d6.el8. 8.3 MB/s | 24 MB 00:02 2024-02-19T20:51:19.246 INFO:teuthology.orchestra.run.smithi193.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 662 kB/s | 259 kB 00:00 2024-02-19T20:51:19.330 INFO:teuthology.orchestra.run.smithi193.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 310 kB/s | 44 kB 00:00 2024-02-19T20:51:19.355 INFO:teuthology.orchestra.run.smithi193.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 433 kB/s | 47 kB 00:00 2024-02-19T20:51:19.380 INFO:teuthology.orchestra.run.smithi193.stdout:(17/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.2 MB/s | 677 kB 00:00 2024-02-19T20:51:19.831 INFO:teuthology.orchestra.run.smithi193.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 131 kB/s | 59 kB 00:00 2024-02-19T20:51:19.906 INFO:teuthology.orchestra.run.smithi193.stdout:(19/23): leveldb-1.22-1.el8.x86_64.rpm 329 kB/s | 181 kB 00:00 2024-02-19T20:51:20.014 INFO:teuthology.orchestra.run.smithi193.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 447 kB/s | 306 kB 00:00 2024-02-19T20:51:20.065 INFO:teuthology.orchestra.run.smithi193.stdout:(21/23): libunwind-1.3.1-3.el8.x86_64.rpm 321 kB/s | 75 kB 00:00 2024-02-19T20:51:20.323 INFO:teuthology.orchestra.run.smithi193.stdout:(22/23): librados2-16.2.14-548.g81bd20d6.el8.x8 9.0 MB/s | 3.7 MB 00:00 2024-02-19T20:51:20.399 INFO:teuthology.orchestra.run.smithi193.stdout:(23/23): librbd1-16.2.14-548.g81bd20d6.el8.x86_ 11 MB/s | 4.2 MB 00:00 2024-02-19T20:51:20.401 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:51:20.401 INFO:teuthology.orchestra.run.smithi193.stdout:Total 13 MB/s | 58 MB 00:04 2024-02-19T20:51:20.443 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:51:20.490 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:51:20.491 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:51:20.833 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:51:20.836 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:51:21.836 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:51:22.028 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-19T20:51:22.246 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-19T20:51:22.633 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-19T20:51:22.838 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-19T20:51:22.949 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-19T20:51:23.194 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 4/25 2024-02-19T20:51:23.388 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-19T20:51:23.803 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-19T20:51:23.987 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 6/25 2024-02-19T20:51:24.037 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 6/25 2024-02-19T20:51:24.274 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-02-19T20:51:24.879 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-02-19T20:51:25.079 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 9/25 2024-02-19T20:51:25.152 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 9/25 2024-02-19T20:51:25.335 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-02-19T20:51:25.509 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8. 11/25 2024-02-19T20:51:25.698 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 12/25 2024-02-19T20:51:25.930 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 13/25 2024-02-19T20:51:26.137 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 14/25 2024-02-19T20:51:26.287 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 15/25 2024-02-19T20:51:26.359 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 15/25 2024-02-19T20:51:26.656 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-19T20:51:26.829 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-19T20:51:26.998 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-19T20:51:27.212 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x8 19/25 2024-02-19T20:51:29.676 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-19T20:51:30.012 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-19T20:51:30.719 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-19T20:51:31.020 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 21/25 2024-02-19T20:51:31.132 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 21/25 2024-02-19T20:51:31.303 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-19T20:51:50.372 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-19T20:51:50.373 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /sys 2024-02-19T20:51:50.373 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /proc 2024-02-19T20:51:50.373 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /dev 2024-02-19T20:51:50.373 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /run 2024-02-19T20:51:50.373 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /mnt 2024-02-19T20:51:50.373 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /var/tmp 2024-02-19T20:51:50.373 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /home 2024-02-19T20:51:50.373 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /tmp 2024-02-19T20:51:50.373 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /dev 2024-02-19T20:51:50.373 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:51:51.566 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 23/25 2024-02-19T20:51:51.640 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 23/25 2024-02-19T20:51:51.640 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-19T20:51:51.852 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-19T20:51:51.852 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-19T20:51:52.538 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-19T20:51:52.538 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 1/25 2024-02-19T20:51:52.538 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2/25 2024-02-19T20:51:52.538 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-19T20:51:52.538 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 4/25 2024-02-19T20:51:52.538 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-19T20:51:52.538 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 6/25 2024-02-19T20:51:52.538 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 7/25 2024-02-19T20:51:52.539 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8. 8/25 2024-02-19T20:51:52.539 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x8 9/25 2024-02-19T20:51:52.539 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 10/25 2024-02-19T20:51:52.539 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 11/25 2024-02-19T20:51:52.539 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 12/25 2024-02-19T20:51:52.539 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 13/25 2024-02-19T20:51:52.539 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-19T20:51:52.539 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-19T20:51:52.539 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-19T20:51:52.539 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-19T20:51:52.539 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-19T20:51:52.539 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-02-19T20:51:52.539 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-02-19T20:51:52.540 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-02-19T20:51:52.540 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-19T20:51:52.540 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-19T20:51:52.540 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 24/25 2024-02-19T20:51:53.388 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-19T20:51:53.389 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:51:53.389 INFO:teuthology.orchestra.run.smithi193.stdout:Upgraded: 2024-02-19T20:51:53.389 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:51:53.389 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:51:53.389 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:51:53.389 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:51:53.389 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:51:53.389 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:51:53.389 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:51:53.389 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-19T20:51:53.389 INFO:teuthology.orchestra.run.smithi193.stdout: leveldb-1.22-1.el8.x86_64 2024-02-19T20:51:53.389 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:51:53.390 INFO:teuthology.orchestra.run.smithi193.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-19T20:51:53.390 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-19T20:51:53.390 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:51:53.390 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-19T20:51:53.390 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:51:53.390 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-19T20:51:53.390 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-19T20:51:53.390 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:51:53.390 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:51:53.390 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:51:53.390 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-19T20:51:53.390 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:51:53.390 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:51:53.391 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:51:53.391 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:51:53.391 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:51:53.701 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-test 2024-02-19T20:51:54.467 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:42 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:51:56.019 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:51:56.020 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:51:56.020 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-02-19T20:51:56.020 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:51:56.020 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:51:56.020 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 52 M 2024-02-19T20:51:56.021 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-02-19T20:51:56.021 INFO:teuthology.orchestra.run.smithi193.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-19T20:51:56.021 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 174 k 2024-02-19T20:51:56.021 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-19T20:51:56.021 INFO:teuthology.orchestra.run.smithi193.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-19T20:51:56.021 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-19T20:51:56.021 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:51:56.021 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:51:56.021 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:51:56.021 INFO:teuthology.orchestra.run.smithi193.stdout:Install 6 Packages 2024-02-19T20:51:56.021 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:51:56.022 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 53 M 2024-02-19T20:51:56.022 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 197 M 2024-02-19T20:51:56.022 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:51:56.357 INFO:teuthology.orchestra.run.smithi193.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 865 kB/s | 203 kB 00:00 2024-02-19T20:51:56.449 INFO:teuthology.orchestra.run.smithi193.stdout:(2/6): oniguruma-6.8.2-3.el8.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-02-19T20:51:56.550 INFO:teuthology.orchestra.run.smithi193.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.2 MB/s | 323 kB 00:00 2024-02-19T20:51:56.608 INFO:teuthology.orchestra.run.smithi193.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-02-19T20:51:56.633 INFO:teuthology.orchestra.run.smithi193.stdout:(5/6): libcephsqlite-16.2.14-548.g81bd20d6.el8. 340 kB/s | 174 kB 00:00 2024-02-19T20:52:06.788 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 274 kB/s | 16 MB 01:00 2024-02-19T20:52:10.547 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 368 kB/s | 24 kB 00:00 2024-02-19T20:52:14.052 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:52:14.054 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:52:14.054 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-02-19T20:52:14.054 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:52:14.054 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:52:14.054 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 12 M 2024-02-19T20:52:14.054 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrading: 2024-02-19T20:52:14.054 INFO:teuthology.orchestra.run.smithi163.stdout: librados2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.7 M 2024-02-19T20:52:14.055 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 4.2 M 2024-02-19T20:52:14.055 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-02-19T20:52:14.055 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 6.1 M 2024-02-19T20:52:14.055 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 24 M 2024-02-19T20:52:14.055 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 26 k 2024-02-19T20:52:14.055 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-19T20:52:14.055 INFO:teuthology.orchestra.run.smithi163.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-19T20:52:14.055 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 825 k 2024-02-19T20:52:14.055 INFO:teuthology.orchestra.run.smithi163.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-19T20:52:14.055 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-19T20:52:14.055 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 494 k 2024-02-19T20:52:14.056 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-19T20:52:14.056 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2 x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.8 M 2024-02-19T20:52:14.056 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-19T20:52:14.056 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-19T20:52:14.056 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 45 k 2024-02-19T20:52:14.056 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 99 k 2024-02-19T20:52:14.056 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 215 k 2024-02-19T20:52:14.056 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-19T20:52:14.056 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 389 k 2024-02-19T20:52:14.056 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 369 k 2024-02-19T20:52:14.056 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 114 k 2024-02-19T20:52:14.056 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:52:14.056 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:52:14.056 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:52:14.057 INFO:teuthology.orchestra.run.smithi163.stdout:Install 21 Packages 2024-02-19T20:52:14.057 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrade 2 Packages 2024-02-19T20:52:14.057 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:52:14.057 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 58 M 2024-02-19T20:52:14.057 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:52:15.640 INFO:teuthology.orchestra.run.smithi163.stdout:(1/23): ceph-base-16.2.14-548.g81bd20d6.el8.x86 6.2 MB/s | 6.1 MB 00:00 2024-02-19T20:52:15.682 INFO:teuthology.orchestra.run.smithi163.stdout:(2/23): ceph-selinux-16.2.14-548.g81bd20d6.el8. 618 kB/s | 26 kB 00:00 2024-02-19T20:52:15.849 INFO:teuthology.orchestra.run.smithi163.stdout:(3/23): libcephfs2-16.2.14-548.g81bd20d6.el8.x8 4.8 MB/s | 825 kB 00:00 2024-02-19T20:52:15.974 INFO:teuthology.orchestra.run.smithi163.stdout:(4/23): libradosstriper1-16.2.14-548.g81bd20d6. 3.9 MB/s | 494 kB 00:00 2024-02-19T20:52:16.566 INFO:teuthology.orchestra.run.smithi163.stdout:(5/23): librgw2-16.2.14-548.g81bd20d6.el8.x86_6 6.4 MB/s | 3.8 MB 00:00 2024-02-19T20:52:16.816 INFO:teuthology.orchestra.run.smithi163.stdout:(6/23): ceph-common-16.2.14-548.g81bd20d6.el8.x 11 MB/s | 24 MB 00:02 2024-02-19T20:52:16.841 INFO:teuthology.orchestra.run.smithi163.stdout:(7/23): python3-ceph-argparse-16.2.14-548.g81bd 163 kB/s | 45 kB 00:00 2024-02-19T20:52:16.867 INFO:teuthology.orchestra.run.smithi163.stdout:(8/23): python3-ceph-common-16.2.14-548.g81bd20 1.9 MB/s | 99 kB 00:00 2024-02-19T20:52:16.892 INFO:teuthology.orchestra.run.smithi163.stdout:(9/23): python3-cephfs-16.2.14-548.g81bd20d6.el 4.2 MB/s | 215 kB 00:00 2024-02-19T20:52:16.917 INFO:teuthology.orchestra.run.smithi163.stdout:(10/23): python3-rados-16.2.14-548.g81bd20d6.el 7.6 MB/s | 389 kB 00:00 2024-02-19T20:52:16.959 INFO:teuthology.orchestra.run.smithi163.stdout:(11/23): python3-rbd-16.2.14-548.g81bd20d6.el8. 5.4 MB/s | 369 kB 00:00 2024-02-19T20:52:16.986 INFO:teuthology.orchestra.run.smithi163.stdout:(12/23): python3-rgw-16.2.14-548.g81bd20d6.el8. 1.7 MB/s | 114 kB 00:00 2024-02-19T20:52:17.235 INFO:teuthology.orchestra.run.smithi163.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2024-02-19T20:52:17.260 INFO:teuthology.orchestra.run.smithi163.stdout:(14/23): librdkafka-1.6.1-1.el8.x86_64.rpm 2.2 MB/s | 677 kB 00:00 2024-02-19T20:52:17.294 INFO:teuthology.orchestra.run.smithi163.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 748 kB/s | 44 kB 00:00 2024-02-19T20:52:17.378 INFO:teuthology.orchestra.run.smithi163.stdout:(16/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 402 kB/s | 47 kB 00:00 2024-02-19T20:52:18.512 INFO:teuthology.orchestra.run.smithi163.stdout:(17/23): ceph-radosgw-16.2.14-548.g81bd20d6.el8 3.0 MB/s | 12 MB 00:03 2024-02-19T20:52:18.561 INFO:teuthology.orchestra.run.smithi163.stdout:(18/23): leveldb-1.22-1.el8.x86_64.rpm 153 kB/s | 181 kB 00:01 2024-02-19T20:52:18.905 INFO:teuthology.orchestra.run.smithi193.stdout:(6/6): ceph-test-16.2.14-548.g81bd20d6.el8.x86_ 2.3 MB/s | 52 MB 00:22 2024-02-19T20:52:18.907 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:52:18.908 INFO:teuthology.orchestra.run.smithi193.stdout:Total 2.3 MB/s | 53 MB 00:22 2024-02-19T20:52:18.949 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:52:18.962 INFO:teuthology.orchestra.run.smithi163.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 187 kB/s | 75 kB 00:00 2024-02-19T20:52:18.969 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:52:18.970 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:52:19.046 INFO:teuthology.orchestra.run.smithi163.stdout:(20/23): liboath-2.6.2-3.el8.x86_64.rpm 110 kB/s | 59 kB 00:00 2024-02-19T20:52:19.258 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:52:19.261 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:52:19.304 INFO:teuthology.orchestra.run.smithi163.stdout:(21/23): librados2-16.2.14-548.g81bd20d6.el8.x8 11 MB/s | 3.7 MB 00:00 2024-02-19T20:52:19.546 INFO:teuthology.orchestra.run.smithi163.stdout:(22/23): librbd1-16.2.14-548.g81bd20d6.el8.x86_ 8.4 MB/s | 4.2 MB 00:00 2024-02-19T20:52:19.571 INFO:teuthology.orchestra.run.smithi163.stdout:(23/23): gperftools-libs-2.7-9.el8.x86_64.rpm 134 kB/s | 306 kB 00:02 2024-02-19T20:52:19.574 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:52:19.574 INFO:teuthology.orchestra.run.smithi163.stdout:Total 10 MB/s | 58 MB 00:05 2024-02-19T20:52:19.616 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:52:19.663 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:52:19.663 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:52:19.757 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:52:19.989 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-19T20:52:20.017 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:52:20.020 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:52:20.172 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-19T20:52:20.286 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T20:52:20.369 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T20:52:20.567 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-19T20:52:20.753 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 5/6 2024-02-19T20:52:21.024 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:52:21.217 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-02-19T20:52:21.390 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-19T20:52:21.781 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-02-19T20:52:21.982 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-19T20:52:22.071 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-19T20:52:22.299 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 4/25 2024-02-19T20:52:22.472 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-19T20:52:22.888 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-19T20:52:23.073 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 6/25 2024-02-19T20:52:23.136 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 6/25 2024-02-19T20:52:23.349 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-02-19T20:52:23.853 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-02-19T20:52:24.015 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 9/25 2024-02-19T20:52:24.092 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 9/25 2024-02-19T20:52:24.262 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-02-19T20:52:24.437 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8. 11/25 2024-02-19T20:52:24.618 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 12/25 2024-02-19T20:52:24.824 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 13/25 2024-02-19T20:52:25.014 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 14/25 2024-02-19T20:52:25.156 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 15/25 2024-02-19T20:52:25.223 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 15/25 2024-02-19T20:52:25.377 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 5/6 2024-02-19T20:52:25.445 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-02-19T20:52:25.631 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-02-19T20:52:25.750 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 6/6 2024-02-19T20:52:25.792 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-02-19T20:52:25.990 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x8 19/25 2024-02-19T20:52:26.048 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 6/6 2024-02-19T20:52:26.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 1/6 2024-02-19T20:52:26.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 2/6 2024-02-19T20:52:26.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-19T20:52:26.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-19T20:52:26.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-19T20:52:26.483 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-19T20:52:26.483 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:52:26.483 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:52:26.483 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:26.483 INFO:teuthology.orchestra.run.smithi193.stdout: jq-1.6-8.el8.x86_64 2024-02-19T20:52:26.484 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:26.484 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-19T20:52:26.484 INFO:teuthology.orchestra.run.smithi193.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-19T20:52:26.484 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-19T20:52:26.484 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:52:26.484 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:52:26.712 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph 2024-02-19T20:52:27.469 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:15 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:52:28.464 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-19T20:52:28.731 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-19T20:52:29.017 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:52:29.021 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:52:29.021 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-02-19T20:52:29.021 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:52:29.021 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:52:29.021 INFO:teuthology.orchestra.run.smithi193.stdout: ceph x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 6.4 k 2024-02-19T20:52:29.021 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-02-19T20:52:29.021 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 2.4 M 2024-02-19T20:52:29.021 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 1.6 M 2024-02-19T20:52:29.021 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 245 k 2024-02-19T20:52:29.021 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 5.2 M 2024-02-19T20:52:29.021 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 19 M 2024-02-19T20:52:29.021 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-19T20:52:29.022 INFO:teuthology.orchestra.run.smithi193.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-19T20:52:29.022 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-19T20:52:29.022 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-19T20:52:29.022 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-19T20:52:29.022 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-19T20:52:29.022 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-19T20:52:29.022 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-19T20:52:29.022 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-19T20:52:29.022 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-19T20:52:29.022 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-19T20:52:29.022 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-19T20:52:29.022 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-19T20:52:29.022 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-19T20:52:29.023 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-19T20:52:29.023 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-19T20:52:29.023 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-19T20:52:29.023 INFO:teuthology.orchestra.run.smithi193.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-19T20:52:29.023 INFO:teuthology.orchestra.run.smithi193.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-19T20:52:29.023 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-19T20:52:29.023 INFO:teuthology.orchestra.run.smithi193.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-19T20:52:29.023 INFO:teuthology.orchestra.run.smithi193.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-19T20:52:29.023 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-19T20:52:29.023 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-19T20:52:29.023 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-19T20:52:29.023 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-19T20:52:29.023 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:52:29.023 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:52:29.024 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:52:29.024 INFO:teuthology.orchestra.run.smithi193.stdout:Install 32 Packages 2024-02-19T20:52:29.024 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:52:29.025 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 32 M 2024-02-19T20:52:29.025 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 104 M 2024-02-19T20:52:29.025 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:52:29.430 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 20/25 2024-02-19T20:52:29.696 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 21/25 2024-02-19T20:52:29.803 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 21/25 2024-02-19T20:52:29.833 INFO:teuthology.orchestra.run.smithi193.stdout:(1/32): ceph-16.2.14-548.g81bd20d6.el8.x86_64.r 48 kB/s | 6.4 kB 00:00 2024-02-19T20:52:29.947 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-19T20:52:30.333 INFO:teuthology.orchestra.run.smithi193.stdout:(2/32): ceph-mon-16.2.14-548.g81bd20d6.el8.x86_ 10 MB/s | 5.2 MB 00:00 2024-02-19T20:52:30.684 INFO:teuthology.orchestra.run.smithi193.stdout:(3/32): ceph-mgr-16.2.14-548.g81bd20d6.el8.x86_ 1.7 MB/s | 1.6 MB 00:00 2024-02-19T20:52:30.801 INFO:teuthology.orchestra.run.smithi193.stdout:(4/32): ceph-mgr-modules-core-16.2.14-548.g81bd 2.1 MB/s | 245 kB 00:00 2024-02-19T20:52:30.901 INFO:teuthology.orchestra.run.smithi193.stdout:(5/32): ceph-mds-16.2.14-548.g81bd20d6.el8.x86_ 2.0 MB/s | 2.4 MB 00:01 2024-02-19T20:52:31.001 INFO:teuthology.orchestra.run.smithi193.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 784 kB/s | 157 kB 00:00 2024-02-19T20:52:31.035 INFO:teuthology.orchestra.run.smithi193.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 291 kB/s | 39 kB 00:00 2024-02-19T20:52:31.068 INFO:teuthology.orchestra.run.smithi193.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.5 MB/s | 103 kB 00:00 2024-02-19T20:52:31.119 INFO:teuthology.orchestra.run.smithi193.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 681 kB/s | 57 kB 00:00 2024-02-19T20:52:31.336 INFO:teuthology.orchestra.run.smithi193.stdout:(10/32): ceph-osd-16.2.14-548.g81bd20d6.el8.x86 19 MB/s | 19 MB 00:01 2024-02-19T20:52:31.361 INFO:teuthology.orchestra.run.smithi193.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.5 MB/s | 457 kB 00:00 2024-02-19T20:52:31.403 INFO:teuthology.orchestra.run.smithi193.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 875 kB/s | 249 kB 00:00 2024-02-19T20:52:31.512 INFO:teuthology.orchestra.run.smithi193.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 726 kB/s | 109 kB 00:00 2024-02-19T20:52:31.537 INFO:teuthology.orchestra.run.smithi193.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 3.1 MB/s | 641 kB 00:00 2024-02-19T20:52:31.687 INFO:teuthology.orchestra.run.smithi193.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 154 kB/s | 44 kB 00:00 2024-02-19T20:52:31.963 INFO:teuthology.orchestra.run.smithi193.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 411 kB/s | 185 kB 00:00 2024-02-19T20:52:31.988 INFO:teuthology.orchestra.run.smithi193.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 383 kB/s | 173 kB 00:00 2024-02-19T20:52:32.022 INFO:teuthology.orchestra.run.smithi193.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.1 MB/s | 384 kB 00:00 2024-02-19T20:52:32.055 INFO:teuthology.orchestra.run.smithi193.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 120 kB/s | 11 kB 00:00 2024-02-19T20:52:32.089 INFO:teuthology.orchestra.run.smithi193.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 174 kB/s | 18 kB 00:00 2024-02-19T20:52:32.116 INFO:teuthology.orchestra.run.smithi193.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 528 kB/s | 49 kB 00:00 2024-02-19T20:52:32.164 INFO:teuthology.orchestra.run.smithi193.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 539 kB/s | 59 kB 00:00 2024-02-19T20:52:32.206 INFO:teuthology.orchestra.run.smithi193.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 169 kB/s | 16 kB 00:00 2024-02-19T20:52:32.265 INFO:teuthology.orchestra.run.smithi193.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 187 kB/s | 19 kB 00:00 2024-02-19T20:52:32.290 INFO:teuthology.orchestra.run.smithi193.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2024-02-19T20:52:32.315 INFO:teuthology.orchestra.run.smithi193.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 220 kB/s | 24 kB 00:00 2024-02-19T20:52:32.366 INFO:teuthology.orchestra.run.smithi193.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2024-02-19T20:52:32.391 INFO:teuthology.orchestra.run.smithi193.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2024-02-19T20:52:32.441 INFO:teuthology.orchestra.run.smithi193.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2024-02-19T20:52:32.500 INFO:teuthology.orchestra.run.smithi193.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 788 kB/s | 85 kB 00:00 2024-02-19T20:52:32.550 INFO:teuthology.orchestra.run.smithi193.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2024-02-19T20:52:32.567 INFO:teuthology.orchestra.run.smithi193.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 183 kB/s | 23 kB 00:00 2024-02-19T20:52:32.568 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:52:32.568 INFO:teuthology.orchestra.run.smithi193.stdout:Total 9.0 MB/s | 32 MB 00:03 2024-02-19T20:52:32.649 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:52:32.687 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:52:32.687 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:52:33.000 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:52:33.004 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:52:34.118 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:52:34.341 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-19T20:52:34.534 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-19T20:52:34.824 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-19T20:52:35.057 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-19T20:52:35.288 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-19T20:52:35.484 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-19T20:52:35.669 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-19T20:52:35.847 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-19T20:52:36.080 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-19T20:52:36.284 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-19T20:52:36.514 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-19T20:52:36.745 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-19T20:52:37.104 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-19T20:52:37.377 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-19T20:52:37.580 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-19T20:52:37.850 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-19T20:52:38.075 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T20:52:38.351 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-19T20:52:38.564 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-19T20:52:38.760 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-19T20:52:38.955 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-19T20:52:39.164 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-19T20:52:39.386 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-19T20:52:39.618 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-19T20:52:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-19T20:52:40.246 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-19T20:52:40.585 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8. 27/32 2024-02-19T20:52:40.774 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 28/32 2024-02-19T20:52:42.304 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 28/32 2024-02-19T20:52:42.483 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 29/32 2024-02-19T20:52:42.926 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 29/32 2024-02-19T20:52:43.125 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 30/32 2024-02-19T20:52:43.341 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 30/32 2024-02-19T20:52:43.549 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 31/32 2024-02-19T20:52:43.566 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 31/32 2024-02-19T20:52:43.690 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 32/32 2024-02-19T20:52:44.098 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 32/32 2024-02-19T20:52:44.098 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 1/32 2024-02-19T20:52:44.098 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 2/32 2024-02-19T20:52:44.098 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 3/32 2024-02-19T20:52:44.098 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 4/32 2024-02-19T20:52:44.098 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 5/32 2024-02-19T20:52:44.098 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8. 6/32 2024-02-19T20:52:44.098 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-19T20:52:44.098 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-19T20:52:44.098 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-19T20:52:44.098 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-19T20:52:44.098 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-19T20:52:44.099 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-19T20:52:44.099 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-19T20:52:44.099 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-19T20:52:44.099 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-19T20:52:44.099 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-19T20:52:44.099 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T20:52:44.099 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-19T20:52:44.099 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-19T20:52:44.099 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-19T20:52:44.099 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-19T20:52:44.099 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-19T20:52:44.099 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-19T20:52:44.099 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-19T20:52:44.099 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-19T20:52:44.100 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-19T20:52:44.100 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-19T20:52:44.100 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-19T20:52:44.100 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-19T20:52:44.100 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-19T20:52:44.101 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-19T20:52:44.986 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-19T20:52:44.986 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:52:44.987 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:52:44.987 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:44.987 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:44.987 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:44.987 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:52:44.987 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:44.987 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:44.987 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-19T20:52:44.987 INFO:teuthology.orchestra.run.smithi193.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-19T20:52:44.987 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-19T20:52:44.987 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-19T20:52:44.987 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-19T20:52:44.987 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-19T20:52:44.987 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-19T20:52:44.988 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-19T20:52:44.988 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-19T20:52:44.988 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-19T20:52:44.988 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-19T20:52:44.988 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-19T20:52:44.988 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-19T20:52:44.988 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend-2.6-1.el8.noarch 2024-02-19T20:52:44.988 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-19T20:52:44.988 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-19T20:52:44.988 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-19T20:52:44.988 INFO:teuthology.orchestra.run.smithi193.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-19T20:52:44.988 INFO:teuthology.orchestra.run.smithi193.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-19T20:52:44.988 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-19T20:52:44.988 INFO:teuthology.orchestra.run.smithi193.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-19T20:52:44.989 INFO:teuthology.orchestra.run.smithi193.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-19T20:52:44.989 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-19T20:52:44.989 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-19T20:52:44.989 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-19T20:52:44.989 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-19T20:52:44.989 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:52:44.989 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:52:45.253 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-base 2024-02-19T20:52:46.013 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:34 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:52:47.518 INFO:teuthology.orchestra.run.smithi193.stdout:Package ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:52:47.565 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:52:47.566 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-02-19T20:52:47.566 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:52:47.678 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install cephadm 2024-02-19T20:52:48.425 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:36 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:52:49.296 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-19T20:52:49.296 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /sys 2024-02-19T20:52:49.296 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /proc 2024-02-19T20:52:49.296 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /dev 2024-02-19T20:52:49.296 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /run 2024-02-19T20:52:49.297 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /mnt 2024-02-19T20:52:49.297 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /var/tmp 2024-02-19T20:52:49.297 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /home 2024-02-19T20:52:49.297 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /tmp 2024-02-19T20:52:49.297 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /dev 2024-02-19T20:52:49.297 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:52:49.980 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:52:49.981 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:52:49.981 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-02-19T20:52:49.981 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:52:49.981 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:52:49.981 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 85 k 2024-02-19T20:52:49.981 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:52:49.981 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:52:49.981 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:52:49.981 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-02-19T20:52:49.981 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:52:49.981 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 85 k 2024-02-19T20:52:49.981 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 352 k 2024-02-19T20:52:49.982 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:52:50.159 INFO:teuthology.orchestra.run.smithi193.stdout:cephadm-16.2.14-548.g81bd20d6.el8.noarch.rpm 480 kB/s | 85 kB 00:00 2024-02-19T20:52:50.160 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:52:50.160 INFO:teuthology.orchestra.run.smithi193.stdout:Total 477 kB/s | 85 kB 00:00 2024-02-19T20:52:50.160 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:52:50.167 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:52:50.167 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:52:50.169 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:52:50.174 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:52:50.219 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:52:50.442 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 23/25 2024-02-19T20:52:50.516 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 23/25 2024-02-19T20:52:50.516 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-19T20:52:50.721 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-02-19T20:52:50.721 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-19T20:52:50.721 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-19T20:52:50.910 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-19T20:52:51.138 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-19T20:52:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-02-19T20:52:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 1/25 2024-02-19T20:52:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2/25 2024-02-19T20:52:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 3/25 2024-02-19T20:52:51.381 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 4/25 2024-02-19T20:52:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 5/25 2024-02-19T20:52:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_6 6/25 2024-02-19T20:52:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 7/25 2024-02-19T20:52:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8. 8/25 2024-02-19T20:52:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x8 9/25 2024-02-19T20:52:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 10/25 2024-02-19T20:52:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 11/25 2024-02-19T20:52:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 12/25 2024-02-19T20:52:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 13/25 2024-02-19T20:52:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-02-19T20:52:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-02-19T20:52:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-02-19T20:52:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-02-19T20:52:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-02-19T20:52:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-02-19T20:52:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-02-19T20:52:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-02-19T20:52:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 22/25 2024-02-19T20:52:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-02-19T20:52:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 24/25 2024-02-19T20:52:51.667 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-19T20:52:51.667 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:52:51.667 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:52:51.667 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:52:51.668 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:52:51.668 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:52:51.897 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T20:52:52.237 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-02-19T20:52:52.237 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:52:52.237 INFO:teuthology.orchestra.run.smithi163.stdout:Upgraded: 2024-02-19T20:52:52.237 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:52.237 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:52.237 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:52:52.237 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:52.238 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:52.238 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:52.238 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:52.238 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-19T20:52:52.238 INFO:teuthology.orchestra.run.smithi163.stdout: leveldb-1.22-1.el8.x86_64 2024-02-19T20:52:52.238 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:52.238 INFO:teuthology.orchestra.run.smithi163.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-19T20:52:52.238 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-19T20:52:52.238 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:52.238 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-19T20:52:52.238 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:52.238 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-19T20:52:52.238 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-19T20:52:52.239 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:52.239 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:52.239 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:52.239 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-19T20:52:52.239 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:52.239 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:52.239 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:52.239 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:52:52.239 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:52:52.500 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-test 2024-02-19T20:52:52.662 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:40 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:52:53.267 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:43 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:52:54.205 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:52:54.206 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:52:54.206 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-02-19T20:52:54.206 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:52:54.206 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:52:54.206 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 161 k 2024-02-19T20:52:54.206 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:52:54.206 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:52:54.206 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:52:54.206 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-02-19T20:52:54.207 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:52:54.207 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 161 k 2024-02-19T20:52:54.207 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 470 k 2024-02-19T20:52:54.207 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:52:54.690 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-immutable-object-cache-16.2.14-548.g81bd20 334 kB/s | 161 kB 00:00 2024-02-19T20:52:54.691 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:52:54.691 INFO:teuthology.orchestra.run.smithi193.stdout:Total 333 kB/s | 161 kB 00:00 2024-02-19T20:52:54.691 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:52:54.702 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:52:54.702 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:52:54.749 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:52:54.752 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:52:54.829 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:52:54.830 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:52:54.830 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-02-19T20:52:54.830 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:52:54.830 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:52:54.830 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 52 M 2024-02-19T20:52:54.830 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-02-19T20:52:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-19T20:52:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 174 k 2024-02-19T20:52:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-19T20:52:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-19T20:52:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-19T20:52:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:52:54.831 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:52:54.831 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:52:54.831 INFO:teuthology.orchestra.run.smithi163.stdout:Install 6 Packages 2024-02-19T20:52:54.831 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:52:54.832 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 53 M 2024-02-19T20:52:54.832 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 197 M 2024-02-19T20:52:54.832 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:52:54.865 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:52:55.072 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-19T20:52:55.124 INFO:teuthology.orchestra.run.smithi163.stdout:(1/6): libcephsqlite-16.2.14-548.g81bd20d6.el8. 882 kB/s | 174 kB 00:00 2024-02-19T20:52:55.158 INFO:teuthology.orchestra.run.smithi163.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 879 kB/s | 203 kB 00:00 2024-02-19T20:52:55.259 INFO:teuthology.orchestra.run.smithi163.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.2 MB/s | 323 kB 00:00 2024-02-19T20:52:55.334 INFO:teuthology.orchestra.run.smithi163.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 941 kB/s | 71 kB 00:00 2024-02-19T20:52:55.359 INFO:teuthology.orchestra.run.smithi163.stdout:(5/6): oniguruma-6.8.2-3.el8.x86_64.rpm 802 kB/s | 188 kB 00:00 2024-02-19T20:52:55.469 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-19T20:52:55.821 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-19T20:52:55.821 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:52:55.821 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:52:55.821 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:52:55.821 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:52:55.822 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:52:56.053 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr 2024-02-19T20:52:56.814 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:44 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:52:57.684 INFO:teuthology.orchestra.run.smithi163.stdout:(6/6): ceph-test-16.2.14-548.g81bd20d6.el8.x86_ 19 MB/s | 52 MB 00:02 2024-02-19T20:52:57.686 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:52:57.686 INFO:teuthology.orchestra.run.smithi163.stdout:Total 19 MB/s | 53 MB 00:02 2024-02-19T20:52:57.729 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:52:57.750 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:52:57.751 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:52:58.039 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:52:58.042 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:52:58.318 INFO:teuthology.orchestra.run.smithi193.stdout:Package ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:52:58.366 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:52:58.367 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-02-19T20:52:58.368 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:52:58.469 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T20:52:58.531 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:52:58.739 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-19T20:52:58.936 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-19T20:52:59.067 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T20:52:59.147 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T20:52:59.222 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:47 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:52:59.356 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-19T20:52:59.542 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 5/6 2024-02-19T20:53:00.759 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:53:00.760 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:00.760 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-02-19T20:53:00.760 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:00.760 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:53:00.761 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 3.1 M 2024-02-19T20:53:00.761 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-02-19T20:53:00.761 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 24 k 2024-02-19T20:53:00.761 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 15 k 2024-02-19T20:53:00.761 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-19T20:53:00.761 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-19T20:53:00.761 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-19T20:53:00.761 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:00.761 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:53:00.761 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:00.761 INFO:teuthology.orchestra.run.smithi193.stdout:Install 6 Packages 2024-02-19T20:53:00.761 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:00.762 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 3.4 M 2024-02-19T20:53:00.762 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 22 M 2024-02-19T20:53:00.762 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:53:01.461 INFO:teuthology.orchestra.run.smithi193.stdout:(1/6): ceph-grafana-dashboards-16.2.14-548.g81b 70 kB/s | 24 kB 00:00 2024-02-19T20:53:01.486 INFO:teuthology.orchestra.run.smithi193.stdout:(2/6): ceph-prometheus-alerts-16.2.14-548.g81bd 40 kB/s | 15 kB 00:00 2024-02-19T20:53:01.595 INFO:teuthology.orchestra.run.smithi193.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 319 kB/s | 43 kB 00:00 2024-02-19T20:53:01.712 INFO:teuthology.orchestra.run.smithi193.stdout:(4/6): ceph-mgr-dashboard-16.2.14-548.g81bd20d6 5.2 MB/s | 3.1 MB 00:00 2024-02-19T20:53:01.821 INFO:teuthology.orchestra.run.smithi193.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 101 kB/s | 34 kB 00:00 2024-02-19T20:53:02.137 INFO:teuthology.orchestra.run.smithi193.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 362 kB/s | 196 kB 00:00 2024-02-19T20:53:02.139 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:53:02.140 INFO:teuthology.orchestra.run.smithi193.stdout:Total 2.5 MB/s | 3.4 MB 00:01 2024-02-19T20:53:02.150 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:53:02.161 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:53:02.161 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:53:02.315 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:53:02.318 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:53:02.603 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:53:02.809 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-19T20:53:03.071 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-19T20:53:03.273 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-19T20:53:03.402 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.n 4/6 2024-02-19T20:53:04.191 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 5/6 2024-02-19T20:53:04.422 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8. 5/6 2024-02-19T20:53:04.423 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 6/6 2024-02-19T20:53:04.661 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 6/6 2024-02-19T20:53:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 6/6 2024-02-19T20:53:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 1/6 2024-02-19T20:53:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 2/6 2024-02-19T20:53:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-19T20:53:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-19T20:53:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-19T20:53:04.936 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 6/6 2024-02-19T20:53:04.936 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8. 1/6 2024-02-19T20:53:04.936 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 2/6 2024-02-19T20:53:04.936 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.n 3/6 2024-02-19T20:53:04.936 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-19T20:53:04.936 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-19T20:53:05.146 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-19T20:53:05.146 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:05.146 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:53:05.146 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:53:05.146 INFO:teuthology.orchestra.run.smithi163.stdout: jq-1.6-8.el8.x86_64 2024-02-19T20:53:05.146 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:53:05.146 INFO:teuthology.orchestra.run.smithi163.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-02-19T20:53:05.146 INFO:teuthology.orchestra.run.smithi163.stdout: socat-1.7.4.1-1.el8.x86_64 2024-02-19T20:53:05.146 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-02-19T20:53:05.146 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:05.146 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:53:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-19T20:53:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:05.326 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:53:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:53:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:53:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:53:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-19T20:53:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-19T20:53:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-19T20:53:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:05.327 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:53:05.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph 2024-02-19T20:53:05.451 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T20:53:06.121 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:56 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:53:06.207 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:54 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:53:07.678 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:53:07.681 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:53:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-02-19T20:53:07.681 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:53:07.681 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:53:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: ceph x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 6.4 k 2024-02-19T20:53:07.682 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-02-19T20:53:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 2.4 M 2024-02-19T20:53:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 1.6 M 2024-02-19T20:53:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 245 k 2024-02-19T20:53:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 5.2 M 2024-02-19T20:53:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 19 M 2024-02-19T20:53:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-19T20:53:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-19T20:53:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-19T20:53:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-19T20:53:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-19T20:53:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-19T20:53:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-19T20:53:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-19T20:53:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-19T20:53:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-19T20:53:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-19T20:53:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-19T20:53:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-19T20:53:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-19T20:53:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-19T20:53:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-19T20:53:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-19T20:53:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-19T20:53:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-19T20:53:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-19T20:53:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-19T20:53:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-19T20:53:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-19T20:53:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-19T20:53:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-19T20:53:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-19T20:53:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:07.684 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:53:07.684 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:53:07.684 INFO:teuthology.orchestra.run.smithi163.stdout:Install 32 Packages 2024-02-19T20:53:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:07.685 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 32 M 2024-02-19T20:53:07.685 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 104 M 2024-02-19T20:53:07.685 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:53:07.764 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:53:07.765 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-02-19T20:53:07.765 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-02-19T20:53:07.765 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-02-19T20:53:07.765 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:53:07.766 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 8.6 M 2024-02-19T20:53:07.766 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-02-19T20:53:07.766 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-19T20:53:07.766 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-19T20:53:07.766 INFO:teuthology.orchestra.run.smithi193.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-19T20:53:07.766 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-19T20:53:07.766 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-19T20:53:07.766 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-19T20:53:07.766 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-19T20:53:07.766 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:07.766 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:53:07.766 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-02-19T20:53:07.766 INFO:teuthology.orchestra.run.smithi193.stdout:Install 8 Packages 2024-02-19T20:53:07.766 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:07.767 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 37 M 2024-02-19T20:53:07.767 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 210 M 2024-02-19T20:53:07.767 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:53:08.468 INFO:teuthology.orchestra.run.smithi193.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 9.9 MB/s | 4.7 MB 00:00 2024-02-19T20:53:08.535 INFO:teuthology.orchestra.run.smithi193.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 8.9 MB/s | 4.9 MB 00:00 2024-02-19T20:53:08.604 INFO:teuthology.orchestra.run.smithi163.stdout:(1/32): ceph-16.2.14-548.g81bd20d6.el8.x86_64.r 17 kB/s | 6.4 kB 00:00 2024-02-19T20:53:08.644 INFO:teuthology.orchestra.run.smithi193.stdout:(3/8): ceph-mgr-diskprediction-local-16.2.14-54 13 MB/s | 8.6 MB 00:00 2024-02-19T20:53:08.669 INFO:teuthology.orchestra.run.smithi193.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.6 MB/s | 225 kB 00:00 2024-02-19T20:53:08.736 INFO:teuthology.orchestra.run.smithi193.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 14 MB/s | 3.7 MB 00:00 2024-02-19T20:53:08.887 INFO:teuthology.orchestra.run.smithi193.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 172 kB 00:00 2024-02-19T20:53:08.912 INFO:teuthology.orchestra.run.smithi193.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.6 MB/s | 645 kB 00:00 2024-02-19T20:53:09.055 INFO:teuthology.orchestra.run.smithi163.stdout:(2/32): ceph-mgr-16.2.14-548.g81bd20d6.el8.x86_ 2.0 MB/s | 1.6 MB 00:00 2024-02-19T20:53:09.155 INFO:teuthology.orchestra.run.smithi163.stdout:(3/32): ceph-mds-16.2.14-548.g81bd20d6.el8.x86_ 2.6 MB/s | 2.4 MB 00:00 2024-02-19T20:53:09.289 INFO:teuthology.orchestra.run.smithi163.stdout:(4/32): ceph-mgr-modules-core-16.2.14-548.g81bd 1.8 MB/s | 245 kB 00:00 2024-02-19T20:53:09.358 INFO:teuthology.orchestra.run.smithi193.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 20 MB/s | 14 MB 00:00 2024-02-19T20:53:09.359 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:53:09.359 INFO:teuthology.orchestra.run.smithi193.stdout:Total 23 MB/s | 37 MB 00:01 2024-02-19T20:53:09.494 INFO:teuthology.orchestra.run.smithi163.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 786 kB/s | 157 kB 00:00 2024-02-19T20:53:09.547 INFO:teuthology.orchestra.run.smithi163.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 666 kB/s | 39 kB 00:00 2024-02-19T20:53:09.583 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:53:09.602 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:53:09.602 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:53:09.616 INFO:teuthology.orchestra.run.smithi163.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.5 MB/s | 103 kB 00:00 2024-02-19T20:53:09.717 INFO:teuthology.orchestra.run.smithi163.stdout:(8/32): ceph-mon-16.2.14-548.g81bd20d6.el8.x86_ 4.7 MB/s | 5.2 MB 00:01 2024-02-19T20:53:09.751 INFO:teuthology.orchestra.run.smithi163.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 426 kB/s | 57 kB 00:00 2024-02-19T20:53:09.899 INFO:teuthology.orchestra.run.smithi163.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.6 MB/s | 249 kB 00:00 2024-02-19T20:53:09.924 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:53:09.926 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:53:09.991 INFO:teuthology.orchestra.run.smithi163.stdout:(11/32): python3-cryptography-3.2.1-7.el8.x86_6 6.9 MB/s | 641 kB 00:00 2024-02-19T20:53:10.017 INFO:teuthology.orchestra.run.smithi163.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.5 MB/s | 457 kB 00:00 2024-02-19T20:53:10.042 INFO:teuthology.orchestra.run.smithi163.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 2.1 MB/s | 109 kB 00:00 2024-02-19T20:53:10.451 INFO:teuthology.orchestra.run.smithi163.stdout:(14/32): ceph-osd-16.2.14-548.g81bd20d6.el8.x86 13 MB/s | 19 MB 00:01 2024-02-19T20:53:10.467 INFO:teuthology.orchestra.run.smithi163.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 97 kB/s | 44 kB 00:00 2024-02-19T20:53:10.500 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:53:10.576 INFO:teuthology.orchestra.run.smithi163.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 347 kB/s | 185 kB 00:00 2024-02-19T20:53:10.644 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T20:53:10.693 INFO:teuthology.orchestra.run.smithi163.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2024-02-19T20:53:10.800 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T20:53:10.810 INFO:teuthology.orchestra.run.smithi163.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 151 kB/s | 18 kB 00:00 2024-02-19T20:53:10.885 INFO:teuthology.orchestra.run.smithi163.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 921 kB/s | 384 kB 00:00 2024-02-19T20:53:10.927 INFO:teuthology.orchestra.run.smithi163.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 416 kB/s | 49 kB 00:00 2024-02-19T20:53:10.944 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T20:53:10.986 INFO:teuthology.orchestra.run.smithi163.stdout:(21/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 323 kB/s | 173 kB 00:00 2024-02-19T20:53:11.011 INFO:teuthology.orchestra.run.smithi163.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 467 kB/s | 59 kB 00:00 2024-02-19T20:53:11.112 INFO:teuthology.orchestra.run.smithi163.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 124 kB/s | 16 kB 00:00 2024-02-19T20:53:11.137 INFO:teuthology.orchestra.run.smithi163.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 149 kB/s | 19 kB 00:00 2024-02-19T20:53:11.162 INFO:teuthology.orchestra.run.smithi163.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2024-02-19T20:53:11.237 INFO:teuthology.orchestra.run.smithi163.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 190 kB/s | 24 kB 00:00 2024-02-19T20:53:11.263 INFO:teuthology.orchestra.run.smithi163.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 233 kB/s | 29 kB 00:00 2024-02-19T20:53:11.288 INFO:teuthology.orchestra.run.smithi163.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 219 kB/s | 27 kB 00:00 2024-02-19T20:53:11.397 INFO:teuthology.orchestra.run.smithi163.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.8 MB/s | 251 kB 00:00 2024-02-19T20:53:11.422 INFO:teuthology.orchestra.run.smithi163.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 638 kB/s | 85 kB 00:00 2024-02-19T20:53:11.464 INFO:teuthology.orchestra.run.smithi163.stdout:(31/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.1 MB/s | 255 kB 00:00 2024-02-19T20:53:11.522 INFO:teuthology.orchestra.run.smithi163.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 183 kB/s | 23 kB 00:00 2024-02-19T20:53:11.524 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:53:11.524 INFO:teuthology.orchestra.run.smithi163.stdout:Total 8.3 MB/s | 32 MB 00:03 2024-02-19T20:53:11.542 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T20:53:11.604 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:53:11.643 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:53:11.643 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:53:11.719 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T20:53:11.958 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:53:11.961 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:53:12.320 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T20:53:12.477 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T20:53:13.059 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:53:13.153 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T20:53:13.269 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-19T20:53:13.481 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-19T20:53:13.740 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-19T20:53:13.996 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-19T20:53:14.186 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-19T20:53:14.227 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T20:53:14.383 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-19T20:53:14.559 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-19T20:53:14.753 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-19T20:53:14.971 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-19T20:53:14.985 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T20:53:15.209 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-19T20:53:15.464 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-19T20:53:15.720 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-19T20:53:16.131 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-19T20:53:16.359 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-19T20:53:16.578 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-19T20:53:16.750 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T20:53:16.830 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-19T20:53:17.024 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T20:53:17.340 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-19T20:53:17.569 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-19T20:53:17.750 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-19T20:53:17.978 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-19T20:53:18.066 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-19T20:53:18.213 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-19T20:53:18.251 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 8/8 2024-02-19T20:53:18.434 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-19T20:53:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 8/8 2024-02-19T20:53:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 1/8 2024-02-19T20:53:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-19T20:53:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-19T20:53:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-19T20:53:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T20:53:18.565 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-19T20:53:18.566 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-19T20:53:18.657 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-19T20:53:18.970 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-19T20:53:18.973 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-19T20:53:18.973 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:18.973 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:53:18.973 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:53:18.973 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-19T20:53:18.974 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-19T20:53:18.974 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-19T20:53:18.974 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-19T20:53:18.974 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-19T20:53:18.974 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-19T20:53:18.974 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-19T20:53:18.974 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:18.974 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:53:19.101 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-rook 2024-02-19T20:53:19.258 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-19T20:53:19.606 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8. 27/32 2024-02-19T20:53:19.804 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 28/32 2024-02-19T20:53:19.855 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:07 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:53:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 28/32 2024-02-19T20:53:21.409 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:53:21.411 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-02-19T20:53:21.411 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:21.411 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:53:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 27 k 2024-02-19T20:53:21.411 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-02-19T20:53:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-19T20:53:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-19T20:53:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-19T20:53:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-19T20:53:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-19T20:53:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-19T20:53:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-19T20:53:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-19T20:53:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-19T20:53:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-19T20:53:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-19T20:53:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-19T20:53:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:21.412 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:53:21.412 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:21.413 INFO:teuthology.orchestra.run.smithi193.stdout:Install 13 Packages 2024-02-19T20:53:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:21.413 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 1.9 M 2024-02-19T20:53:21.413 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 24 M 2024-02-19T20:53:21.413 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:53:21.505 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 29/32 2024-02-19T20:53:21.949 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 29/32 2024-02-19T20:53:22.164 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 30/32 2024-02-19T20:53:22.204 INFO:teuthology.orchestra.run.smithi193.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 158 kB/s | 20 kB 00:00 2024-02-19T20:53:22.229 INFO:teuthology.orchestra.run.smithi193.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 175 kB/s | 27 kB 00:00 2024-02-19T20:53:22.330 INFO:teuthology.orchestra.run.smithi193.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2024-02-19T20:53:22.355 INFO:teuthology.orchestra.run.smithi193.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 879 kB/s | 110 kB 00:00 2024-02-19T20:53:22.380 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 30/32 2024-02-19T20:53:22.380 INFO:teuthology.orchestra.run.smithi193.stdout:(5/13): ceph-mgr-rook-16.2.14-548.g81bd20d6.el8 90 kB/s | 27 kB 00:00 2024-02-19T20:53:22.464 INFO:teuthology.orchestra.run.smithi193.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 393 kB/s | 43 kB 00:00 2024-02-19T20:53:22.526 INFO:teuthology.orchestra.run.smithi193.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 789 kB/s | 155 kB 00:00 2024-02-19T20:53:22.563 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 31/32 2024-02-19T20:53:22.581 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 31/32 2024-02-19T20:53:22.721 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 32/32 2024-02-19T20:53:22.756 INFO:teuthology.orchestra.run.smithi193.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 56 kB/s | 16 kB 00:00 2024-02-19T20:53:22.781 INFO:teuthology.orchestra.run.smithi193.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 82 kB/s | 33 kB 00:00 2024-02-19T20:53:22.915 INFO:teuthology.orchestra.run.smithi193.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 214 kB/s | 81 kB 00:00 2024-02-19T20:53:22.940 INFO:teuthology.orchestra.run.smithi193.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 405 kB/s | 64 kB 00:00 2024-02-19T20:53:23.032 INFO:teuthology.orchestra.run.smithi193.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 524 kB/s | 61 kB 00:00 2024-02-19T20:53:23.131 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 32/32 2024-02-19T20:53:23.131 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 1/32 2024-02-19T20:53:23.131 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 2/32 2024-02-19T20:53:23.131 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 3/32 2024-02-19T20:53:23.131 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 4/32 2024-02-19T20:53:23.131 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 5/32 2024-02-19T20:53:23.131 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8. 6/32 2024-02-19T20:53:23.131 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-19T20:53:23.131 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-19T20:53:23.131 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-19T20:53:23.131 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-19T20:53:23.131 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-19T20:53:23.132 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-19T20:53:23.132 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-19T20:53:23.132 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-19T20:53:23.132 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-19T20:53:23.132 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-19T20:53:23.132 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T20:53:23.132 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-19T20:53:23.132 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-19T20:53:23.132 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-19T20:53:23.132 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-19T20:53:23.132 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-19T20:53:23.133 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-19T20:53:23.133 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-19T20:53:23.133 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-19T20:53:23.133 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-19T20:53:23.134 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-19T20:53:23.134 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-19T20:53:23.134 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-19T20:53:23.134 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-19T20:53:23.134 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-19T20:53:23.249 INFO:teuthology.orchestra.run.smithi193.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.4 MB/s | 1.2 MB 00:00 2024-02-19T20:53:23.252 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:53:23.252 INFO:teuthology.orchestra.run.smithi193.stdout:Total 1.0 MB/s | 1.9 MB 00:01 2024-02-19T20:53:23.309 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:53:23.325 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:53:23.325 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:53:23.413 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:53:23.416 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:53:23.946 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-19T20:53:23.946 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:23.946 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:53:23.947 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:53:23.947 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:53:23.947 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:53:23.947 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:53:23.947 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:53:23.947 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:53:23.947 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-19T20:53:23.947 INFO:teuthology.orchestra.run.smithi163.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-19T20:53:23.947 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-19T20:53:23.947 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-19T20:53:23.947 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-19T20:53:23.947 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-19T20:53:23.947 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-19T20:53:23.947 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-19T20:53:23.947 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-19T20:53:23.948 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-19T20:53:23.948 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-19T20:53:23.948 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-19T20:53:23.948 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-19T20:53:23.948 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend-2.6-1.el8.noarch 2024-02-19T20:53:23.948 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-19T20:53:23.948 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-19T20:53:23.948 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-19T20:53:23.948 INFO:teuthology.orchestra.run.smithi163.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-19T20:53:23.948 INFO:teuthology.orchestra.run.smithi163.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-19T20:53:23.948 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-19T20:53:23.948 INFO:teuthology.orchestra.run.smithi163.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-19T20:53:23.948 INFO:teuthology.orchestra.run.smithi163.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-19T20:53:23.949 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-19T20:53:23.949 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-19T20:53:23.949 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-19T20:53:23.949 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-19T20:53:23.949 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:23.949 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:53:24.010 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:53:24.227 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-19T20:53:24.452 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-19T20:53:24.677 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-19T20:53:24.706 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-base 2024-02-19T20:53:24.890 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-19T20:53:25.131 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-19T20:53:25.380 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-19T20:53:25.469 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:15 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:53:25.655 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-19T20:53:25.896 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-19T20:53:26.934 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-19T20:53:26.945 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-base-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:53:26.991 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:53:26.992 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-02-19T20:53:26.992 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:53:27.095 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install cephadm 2024-02-19T20:53:27.181 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-19T20:53:27.406 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-19T20:53:27.582 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-19T20:53:27.733 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 13/13 2024-02-19T20:53:27.848 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:17 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:53:28.041 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 13/13 2024-02-19T20:53:28.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 1/13 2024-02-19T20:53:28.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-19T20:53:28.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-19T20:53:28.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-19T20:53:28.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-19T20:53:28.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-19T20:53:28.041 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-19T20:53:28.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-19T20:53:28.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-19T20:53:28.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-19T20:53:28.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-19T20:53:28.042 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-19T20:53:28.489 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-19T20:53:28.489 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:28.489 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:53:28.489 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:53:28.489 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-19T20:53:28.489 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-19T20:53:28.489 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-19T20:53:28.490 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-19T20:53:28.490 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-19T20:53:28.490 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-19T20:53:28.490 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-19T20:53:28.490 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-19T20:53:28.490 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-19T20:53:28.490 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-19T20:53:28.490 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-19T20:53:28.490 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-19T20:53:28.490 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:28.490 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:53:28.617 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T20:53:29.375 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:17 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:53:29.398 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:53:29.398 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:53:29.398 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-02-19T20:53:29.398 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:53:29.399 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:53:29.399 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 85 k 2024-02-19T20:53:29.399 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:29.399 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:53:29.399 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:53:29.399 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-02-19T20:53:29.399 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:29.399 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 85 k 2024-02-19T20:53:29.399 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 352 k 2024-02-19T20:53:29.399 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:53:29.803 INFO:teuthology.orchestra.run.smithi163.stdout:cephadm-16.2.14-548.g81bd20d6.el8.noarch.rpm 211 kB/s | 85 kB 00:00 2024-02-19T20:53:29.803 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:53:29.804 INFO:teuthology.orchestra.run.smithi163.stdout:Total 211 kB/s | 85 kB 00:00 2024-02-19T20:53:29.804 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:53:29.811 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:53:29.811 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:53:29.813 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:53:29.816 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:53:29.862 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:53:30.382 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-19T20:53:30.524 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-19T20:53:30.764 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-19T20:53:30.931 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:53:30.932 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:30.932 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-02-19T20:53:30.932 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:30.932 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:53:30.933 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 116 k 2024-02-19T20:53:30.933 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-02-19T20:53:30.933 INFO:teuthology.orchestra.run.smithi193.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-19T20:53:30.933 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-19T20:53:30.933 INFO:teuthology.orchestra.run.smithi193.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-19T20:53:30.933 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-19T20:53:30.933 INFO:teuthology.orchestra.run.smithi193.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-19T20:53:30.933 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:30.933 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:53:30.933 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:30.933 INFO:teuthology.orchestra.run.smithi193.stdout:Install 6 Packages 2024-02-19T20:53:30.933 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:30.934 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 5.9 M 2024-02-19T20:53:30.934 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 24 M 2024-02-19T20:53:30.934 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:53:31.116 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/1 2024-02-19T20:53:31.117 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:31.117 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:53:31.117 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:53:31.117 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:31.117 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:53:31.334 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T20:53:31.567 INFO:teuthology.orchestra.run.smithi193.stdout:(1/6): ceph-mgr-cephadm-16.2.14-548.g81bd20d6.e 626 kB/s | 116 kB 00:00 2024-02-19T20:53:31.684 INFO:teuthology.orchestra.run.smithi193.stdout:(2/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.9 MB/s | 578 kB 00:00 2024-02-19T20:53:31.826 INFO:teuthology.orchestra.run.smithi193.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 11 MB/s | 4.8 MB 00:00 2024-02-19T20:53:31.852 INFO:teuthology.orchestra.run.smithi193.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 75 kB/s | 21 kB 00:00 2024-02-19T20:53:32.095 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:22 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:53:32.185 INFO:teuthology.orchestra.run.smithi193.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 136 kB/s | 49 kB 00:00 2024-02-19T20:53:32.278 INFO:teuthology.orchestra.run.smithi193.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 600 kB/s | 355 kB 00:00 2024-02-19T20:53:32.280 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:53:32.280 INFO:teuthology.orchestra.run.smithi193.stdout:Total 4.4 MB/s | 5.9 MB 00:01 2024-02-19T20:53:32.331 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:53:32.341 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:53:32.341 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:53:32.443 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:53:32.446 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:53:32.714 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:53:32.989 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-19T20:53:33.267 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-19T20:53:33.654 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:53:33.655 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:53:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-02-19T20:53:33.656 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:53:33.656 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:53:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 161 k 2024-02-19T20:53:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:33.656 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:53:33.656 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:53:33.656 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-02-19T20:53:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:33.656 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 161 k 2024-02-19T20:53:33.656 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 470 k 2024-02-19T20:53:33.656 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:53:33.833 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-immutable-object-cache-16.2.14-548.g81bd20 912 kB/s | 161 kB 00:00 2024-02-19T20:53:33.834 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:53:33.834 INFO:teuthology.orchestra.run.smithi163.stdout:Total 905 kB/s | 161 kB 00:00 2024-02-19T20:53:33.834 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:53:33.845 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:53:33.845 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:53:33.892 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:53:33.895 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:53:34.009 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:53:34.195 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-19T20:53:34.221 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-19T20:53:34.561 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-19T20:53:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-19T20:53:34.841 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-19T20:53:35.031 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 6/6 2024-02-19T20:53:35.325 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 6/6 2024-02-19T20:53:35.325 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/6 2024-02-19T20:53:35.325 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-19T20:53:35.326 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-19T20:53:35.326 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-19T20:53:35.326 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-19T20:53:35.666 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6. 1/1 2024-02-19T20:53:35.666 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:35.666 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:53:35.666 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:53:35.667 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:35.667 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:53:35.906 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-19T20:53:35.906 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:35.907 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:53:35.907 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:53:35.907 INFO:teuthology.orchestra.run.smithi193.stdout: python3-apipkg-1.5-6.el8.noarch 2024-02-19T20:53:35.907 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-19T20:53:35.907 INFO:teuthology.orchestra.run.smithi193.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-02-19T20:53:35.907 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-02-19T20:53:35.907 INFO:teuthology.orchestra.run.smithi193.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-02-19T20:53:35.907 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:35.907 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:53:36.039 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-fuse 2024-02-19T20:53:36.795 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:24 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:53:36.840 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr 2024-02-19T20:53:37.614 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:27 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:53:38.354 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:53:38.354 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:38.354 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-02-19T20:53:38.354 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:38.354 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:53:38.355 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 911 k 2024-02-19T20:53:38.355 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-02-19T20:53:38.355 INFO:teuthology.orchestra.run.smithi193.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-19T20:53:38.355 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:38.355 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:53:38.355 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:38.355 INFO:teuthology.orchestra.run.smithi193.stdout:Install 2 Packages 2024-02-19T20:53:38.355 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:38.356 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 994 k 2024-02-19T20:53:38.356 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 2.8 M 2024-02-19T20:53:38.356 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:53:38.739 INFO:teuthology.orchestra.run.smithi193.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 598 kB/s | 83 kB 00:00 2024-02-19T20:53:39.116 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-mgr-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:53:39.163 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:53:39.164 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-02-19T20:53:39.164 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:53:39.264 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T20:53:39.539 INFO:teuthology.orchestra.run.smithi193.stdout:(2/2): ceph-fuse-16.2.14-548.g81bd20d6.el8.x86_ 970 kB/s | 911 kB 00:00 2024-02-19T20:53:39.540 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:53:39.540 INFO:teuthology.orchestra.run.smithi193.stdout:Total 839 kB/s | 994 kB 00:01 2024-02-19T20:53:39.552 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:53:39.574 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:53:39.574 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:53:39.668 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:53:39.671 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:53:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:53:40.020 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:30 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:53:40.096 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-19T20:53:40.322 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2/2 2024-02-19T20:53:40.721 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2/2 2024-02-19T20:53:40.721 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/2 2024-02-19T20:53:41.123 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-19T20:53:41.123 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:41.123 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:53:41.124 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-19T20:53:41.124 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:41.124 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:53:41.342 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librados-devel 2024-02-19T20:53:41.577 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:53:41.577 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:53:41.578 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-02-19T20:53:41.578 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:53:41.578 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:53:41.578 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 3.1 M 2024-02-19T20:53:41.578 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-02-19T20:53:41.578 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 24 k 2024-02-19T20:53:41.578 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 15 k 2024-02-19T20:53:41.578 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-19T20:53:41.578 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-19T20:53:41.578 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-19T20:53:41.578 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:41.578 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:53:41.578 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:53:41.578 INFO:teuthology.orchestra.run.smithi163.stdout:Install 6 Packages 2024-02-19T20:53:41.579 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:41.579 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 3.4 M 2024-02-19T20:53:41.579 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 22 M 2024-02-19T20:53:41.579 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:53:42.105 INFO:teuthology.orchestra.run.smithi163.stdout:(1/6): ceph-prometheus-alerts-16.2.14-548.g81bd 104 kB/s | 15 kB 00:00 2024-02-19T20:53:42.109 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:30 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:53:42.205 INFO:teuthology.orchestra.run.smithi163.stdout:(2/6): python3-jwt-1.6.1-2.el8.noarch.rpm 428 kB/s | 43 kB 00:00 2024-02-19T20:53:42.305 INFO:teuthology.orchestra.run.smithi163.stdout:(3/6): ceph-grafana-dashboards-16.2.14-548.g81b 71 kB/s | 24 kB 00:00 2024-02-19T20:53:42.506 INFO:teuthology.orchestra.run.smithi163.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 112 kB/s | 34 kB 00:00 2024-02-19T20:53:42.664 INFO:teuthology.orchestra.run.smithi163.stdout:(5/6): ceph-mgr-dashboard-16.2.14-548.g81bd20d6 4.5 MB/s | 3.1 MB 00:00 2024-02-19T20:53:42.781 INFO:teuthology.orchestra.run.smithi163.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 413 kB/s | 196 kB 00:00 2024-02-19T20:53:42.783 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:53:42.783 INFO:teuthology.orchestra.run.smithi163.stdout:Total 2.8 MB/s | 3.4 MB 00:01 2024-02-19T20:53:42.795 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:53:42.806 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:53:42.806 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:53:42.959 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:53:42.963 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:53:43.223 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:53:43.441 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-19T20:53:43.650 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:53:43.651 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:43.651 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-02-19T20:53:43.651 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:43.651 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:53:43.651 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 131 k 2024-02-19T20:53:43.651 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:43.651 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:53:43.651 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:43.651 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-02-19T20:53:43.651 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:43.651 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 131 k 2024-02-19T20:53:43.651 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 432 k 2024-02-19T20:53:43.652 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:53:43.703 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-19T20:53:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-19T20:53:44.024 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.n 4/6 2024-02-19T20:53:44.052 INFO:teuthology.orchestra.run.smithi193.stdout:librados-devel-16.2.14-548.g81bd20d6.el8.x86_64 327 kB/s | 131 kB 00:00 2024-02-19T20:53:44.053 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:53:44.053 INFO:teuthology.orchestra.run.smithi193.stdout:Total 326 kB/s | 131 kB 00:00 2024-02-19T20:53:44.053 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:53:44.058 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:53:44.059 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:53:44.108 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:53:44.111 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:53:44.219 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:53:44.429 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:53:44.748 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:53:45.051 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8. 5/6 2024-02-19T20:53:45.103 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:53:45.103 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:45.103 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:53:45.103 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:53:45.103 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:45.103 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:53:45.299 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 6/6 2024-02-19T20:53:45.321 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install libcephfs2 2024-02-19T20:53:45.584 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 6/6 2024-02-19T20:53:45.584 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8. 1/6 2024-02-19T20:53:45.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarc 2/6 2024-02-19T20:53:45.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.n 3/6 2024-02-19T20:53:45.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-19T20:53:45.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-19T20:53:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-19T20:53:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:45.962 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:53:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:53:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:53:45.962 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:53:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-19T20:53:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-19T20:53:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-19T20:53:45.963 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:45.963 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:53:46.077 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T20:53:46.083 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:34 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:53:46.829 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:36 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:53:47.574 INFO:teuthology.orchestra.run.smithi193.stdout:Package libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:53:47.620 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:53:47.621 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-02-19T20:53:47.621 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:53:47.950 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install libcephfs-devel 2024-02-19T20:53:48.388 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:53:48.389 INFO:teuthology.orchestra.run.smithi163.stdout:============================================================================================ 2024-02-19T20:53:48.389 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-02-19T20:53:48.389 INFO:teuthology.orchestra.run.smithi163.stdout:============================================================================================ 2024-02-19T20:53:48.389 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:53:48.389 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 8.6 M 2024-02-19T20:53:48.389 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-02-19T20:53:48.389 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-19T20:53:48.389 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-19T20:53:48.389 INFO:teuthology.orchestra.run.smithi163.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-19T20:53:48.390 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-19T20:53:48.390 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-19T20:53:48.390 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-19T20:53:48.390 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-19T20:53:48.390 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:48.390 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:53:48.390 INFO:teuthology.orchestra.run.smithi163.stdout:============================================================================================ 2024-02-19T20:53:48.390 INFO:teuthology.orchestra.run.smithi163.stdout:Install 8 Packages 2024-02-19T20:53:48.390 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:48.391 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 37 M 2024-02-19T20:53:48.391 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 210 M 2024-02-19T20:53:48.391 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:53:48.715 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:36 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:53:49.197 INFO:teuthology.orchestra.run.smithi163.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 10 MB/s | 4.9 MB 00:00 2024-02-19T20:53:49.289 INFO:teuthology.orchestra.run.smithi163.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 8.2 MB/s | 4.7 MB 00:00 2024-02-19T20:53:49.348 INFO:teuthology.orchestra.run.smithi163.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 3.8 MB/s | 225 kB 00:00 2024-02-19T20:53:49.415 INFO:teuthology.orchestra.run.smithi163.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 17 MB/s | 3.7 MB 00:00 2024-02-19T20:53:49.615 INFO:teuthology.orchestra.run.smithi163.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 3.2 MB/s | 645 kB 00:00 2024-02-19T20:53:49.665 INFO:teuthology.orchestra.run.smithi163.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 3.4 MB/s | 172 kB 00:00 2024-02-19T20:53:49.841 INFO:teuthology.orchestra.run.smithi163.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 29 MB/s | 14 MB 00:00 2024-02-19T20:53:49.941 INFO:teuthology.orchestra.run.smithi163.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.14-54 7.0 MB/s | 8.6 MB 00:01 2024-02-19T20:53:49.942 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:53:49.942 INFO:teuthology.orchestra.run.smithi163.stdout:Total 24 MB/s | 37 MB 00:01 2024-02-19T20:53:50.173 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:53:50.191 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:53:50.191 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:53:50.250 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:53:50.251 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:50.251 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-02-19T20:53:50.251 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:50.251 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:53:50.251 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 25 k 2024-02-19T20:53:50.252 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:50.252 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:53:50.252 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:53:50.252 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-02-19T20:53:50.252 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:50.252 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 25 k 2024-02-19T20:53:50.252 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 107 k 2024-02-19T20:53:50.252 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:53:50.386 INFO:teuthology.orchestra.run.smithi193.stdout:libcephfs-devel-16.2.14-548.g81bd20d6.el8.x86_6 191 kB/s | 25 kB 00:00 2024-02-19T20:53:50.386 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:53:50.387 INFO:teuthology.orchestra.run.smithi193.stdout:Total 189 kB/s | 25 kB 00:00 2024-02-19T20:53:50.387 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:53:50.391 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:53:50.391 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:53:50.393 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:53:50.396 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:53:50.444 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:53:50.525 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:53:50.528 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:53:50.644 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:53:50.910 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:53:51.103 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:53:51.242 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T20:53:51.264 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:53:51.264 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:51.264 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:53:51.264 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:53:51.264 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:53:51.264 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:53:51.356 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T20:53:51.384 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librados2 2024-02-19T20:53:51.491 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T20:53:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T20:53:52.135 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:40 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:53:52.241 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T20:53:52.860 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T20:53:53.616 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T20:53:53.631 INFO:teuthology.orchestra.run.smithi193.stdout:Package librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:53:53.678 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:53:53.679 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-02-19T20:53:53.679 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:53:53.793 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librbd1 2024-02-19T20:53:54.311 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T20:53:54.551 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:42 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:53:54.619 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T20:53:54.849 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T20:53:56.058 INFO:teuthology.orchestra.run.smithi193.stdout:Package librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:53:56.106 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:53:56.107 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-02-19T20:53:56.107 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:53:56.208 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rados 2024-02-19T20:53:56.618 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T20:53:56.960 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:44 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:53:57.915 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-19T20:53:58.082 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 8/8 2024-02-19T20:53:58.398 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 8/8 2024-02-19T20:53:58.399 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d 1/8 2024-02-19T20:53:58.399 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-19T20:53:58.399 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-19T20:53:58.399 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-19T20:53:58.399 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T20:53:58.399 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-19T20:53:58.399 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-19T20:53:58.463 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:53:58.509 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:53:58.510 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-02-19T20:53:58.511 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:53:58.710 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rgw 2024-02-19T20:53:58.811 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-19T20:53:58.811 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:58.812 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:53:58.812 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:53:58.812 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-19T20:53:58.812 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-19T20:53:58.812 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-19T20:53:58.812 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-19T20:53:58.812 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-19T20:53:58.812 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-19T20:53:58.812 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-19T20:53:58.812 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:53:58.812 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:53:59.004 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-rook 2024-02-19T20:53:59.459 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:47 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:53:59.760 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:49 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:54:00.967 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:54:01.013 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:54:01.014 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-02-19T20:54:01.014 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:54:01.113 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-cephfs 2024-02-19T20:54:01.326 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:54:01.328 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:01.328 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-02-19T20:54:01.328 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:01.328 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:54:01.328 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 27 k 2024-02-19T20:54:01.328 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-02-19T20:54:01.328 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-19T20:54:01.328 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-19T20:54:01.328 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-19T20:54:01.329 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-19T20:54:01.329 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-19T20:54:01.329 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-19T20:54:01.329 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-19T20:54:01.329 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-19T20:54:01.329 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-19T20:54:01.329 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-19T20:54:01.329 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-19T20:54:01.329 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-19T20:54:01.329 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:01.329 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:54:01.330 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:01.330 INFO:teuthology.orchestra.run.smithi163.stdout:Install 13 Packages 2024-02-19T20:54:01.330 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:01.330 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 1.9 M 2024-02-19T20:54:01.330 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 24 M 2024-02-19T20:54:01.330 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:54:01.609 INFO:teuthology.orchestra.run.smithi163.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 157 kB/s | 20 kB 00:00 2024-02-19T20:54:01.635 INFO:teuthology.orchestra.run.smithi163.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 174 kB/s | 27 kB 00:00 2024-02-19T20:54:01.702 INFO:teuthology.orchestra.run.smithi163.stdout:(3/13): ceph-mgr-rook-16.2.14-548.g81bd20d6.el8 124 kB/s | 27 kB 00:00 2024-02-19T20:54:01.736 INFO:teuthology.orchestra.run.smithi163.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2024-02-19T20:54:01.761 INFO:teuthology.orchestra.run.smithi163.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 874 kB/s | 110 kB 00:00 2024-02-19T20:54:01.836 INFO:teuthology.orchestra.run.smithi163.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 439 kB/s | 33 kB 00:00 2024-02-19T20:54:01.861 INFO:teuthology.orchestra.run.smithi163.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 339 kB/s | 43 kB 00:00 2024-02-19T20:54:01.880 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:49 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:54:01.887 INFO:teuthology.orchestra.run.smithi163.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 323 kB/s | 16 kB 00:00 2024-02-19T20:54:01.912 INFO:teuthology.orchestra.run.smithi163.stdout:(9/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 737 kB/s | 155 kB 00:00 2024-02-19T20:54:01.946 INFO:teuthology.orchestra.run.smithi163.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 966 kB/s | 81 kB 00:00 2024-02-19T20:54:01.979 INFO:teuthology.orchestra.run.smithi163.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.8 MB/s | 61 kB 00:00 2024-02-19T20:54:02.004 INFO:teuthology.orchestra.run.smithi163.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 697 kB/s | 64 kB 00:00 2024-02-19T20:54:02.097 INFO:teuthology.orchestra.run.smithi163.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.6 MB/s | 1.2 MB 00:00 2024-02-19T20:54:02.097 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:54:02.097 INFO:teuthology.orchestra.run.smithi163.stdout:Total 2.5 MB/s | 1.9 MB 00:00 2024-02-19T20:54:02.148 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:54:02.163 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:54:02.163 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:54:02.249 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:54:02.252 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:54:02.806 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:54:03.017 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-19T20:54:03.249 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-19T20:54:03.383 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:54:03.429 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:54:03.430 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-02-19T20:54:03.430 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:54:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-19T20:54:03.530 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rbd 2024-02-19T20:54:03.679 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-19T20:54:03.870 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-19T20:54:04.110 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-19T20:54:04.280 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:52 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:54:04.376 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-19T20:54:04.601 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-19T20:54:05.620 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-19T20:54:05.793 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:54:05.840 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:54:05.842 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-02-19T20:54:05.842 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:54:05.896 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-19T20:54:05.948 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-fuse 2024-02-19T20:54:06.129 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-19T20:54:06.304 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-19T20:54:06.456 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 13/13 2024-02-19T20:54:06.709 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:54 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:54:06.733 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 13/13 2024-02-19T20:54:06.734 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 1/13 2024-02-19T20:54:06.734 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-19T20:54:06.734 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-19T20:54:06.734 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-19T20:54:06.734 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-19T20:54:06.734 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-19T20:54:06.734 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-19T20:54:06.734 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-19T20:54:06.734 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-19T20:54:06.734 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-19T20:54:06.734 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-19T20:54:06.735 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-19T20:54:07.171 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-19T20:54:07.171 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:07.171 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:54:07.171 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:54:07.172 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-19T20:54:07.172 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-19T20:54:07.172 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-19T20:54:07.172 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-19T20:54:07.172 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-19T20:54:07.172 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-19T20:54:07.172 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-19T20:54:07.172 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-19T20:54:07.172 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-19T20:54:07.172 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-19T20:54:07.172 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-19T20:54:07.172 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-19T20:54:07.172 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:07.172 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:54:07.287 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T20:54:08.042 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:58 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:54:08.253 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:54:08.254 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:54:08.254 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-02-19T20:54:08.254 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:54:08.254 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:54:08.254 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 88 k 2024-02-19T20:54:08.254 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:54:08.254 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:54:08.254 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:54:08.254 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-02-19T20:54:08.254 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:54:08.255 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 88 k 2024-02-19T20:54:08.255 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 240 k 2024-02-19T20:54:08.255 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:54:08.542 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-fuse-16.2.14-548.g81bd20d6.el8.x86_64.rpm 308 kB/s | 88 kB 00:00 2024-02-19T20:54:08.542 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:54:08.543 INFO:teuthology.orchestra.run.smithi193.stdout:Total 307 kB/s | 88 kB 00:00 2024-02-19T20:54:08.543 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:54:08.552 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:54:08.553 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:54:08.602 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:54:08.605 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:54:08.709 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:54:08.883 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:09.162 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:09.575 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:09.575 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:54:09.575 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:54:09.575 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:54:09.575 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:54:09.575 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:54:09.710 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:54:09.711 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:09.711 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-02-19T20:54:09.711 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:09.711 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:54:09.711 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm noarch 2:16.2.14-548.g81bd20d6.el8 ceph-noarch 116 k 2024-02-19T20:54:09.711 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-02-19T20:54:09.711 INFO:teuthology.orchestra.run.smithi163.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-19T20:54:09.711 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-19T20:54:09.711 INFO:teuthology.orchestra.run.smithi163.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-19T20:54:09.711 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-19T20:54:09.712 INFO:teuthology.orchestra.run.smithi163.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-19T20:54:09.712 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:09.712 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:54:09.712 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:09.712 INFO:teuthology.orchestra.run.smithi163.stdout:Install 6 Packages 2024-02-19T20:54:09.712 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:09.712 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 5.9 M 2024-02-19T20:54:09.713 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 24 M 2024-02-19T20:54:09.713 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:54:09.856 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-mirror 2024-02-19T20:54:10.059 INFO:teuthology.orchestra.run.smithi163.stdout:(1/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 2.0 MB/s | 578 kB 00:00 2024-02-19T20:54:10.092 INFO:teuthology.orchestra.run.smithi163.stdout:(2/6): ceph-mgr-cephadm-16.2.14-548.g81bd20d6.e 373 kB/s | 116 kB 00:00 2024-02-19T20:54:10.251 INFO:teuthology.orchestra.run.smithi163.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 10 MB/s | 4.8 MB 00:00 2024-02-19T20:54:10.560 INFO:teuthology.orchestra.run.smithi163.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 42 kB/s | 21 kB 00:00 2024-02-19T20:54:10.613 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:58 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:54:10.685 INFO:teuthology.orchestra.run.smithi163.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 113 kB/s | 49 kB 00:00 2024-02-19T20:54:11.752 INFO:teuthology.orchestra.run.smithi163.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 214 kB/s | 355 kB 00:01 2024-02-19T20:54:11.752 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:54:11.752 INFO:teuthology.orchestra.run.smithi163.stdout:Total 2.9 MB/s | 5.9 MB 00:02 2024-02-19T20:54:11.806 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:54:11.817 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:54:11.817 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:54:11.917 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:54:11.921 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:54:12.177 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:54:12.178 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:54:12.178 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-02-19T20:54:12.178 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:54:12.178 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:54:12.178 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.7 M 2024-02-19T20:54:12.178 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:54:12.178 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:54:12.178 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:54:12.178 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-02-19T20:54:12.178 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:54:12.179 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 3.7 M 2024-02-19T20:54:12.179 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 15 M 2024-02-19T20:54:12.179 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:54:12.196 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:54:12.445 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-19T20:54:12.629 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-mirror-16.2.14-548.g81bd20d6.el8.x86_64.rpm 8.2 MB/s | 3.7 MB 00:00 2024-02-19T20:54:12.630 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:54:12.630 INFO:teuthology.orchestra.run.smithi193.stdout:Total 8.1 MB/s | 3.7 MB 00:00 2024-02-19T20:54:12.630 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:54:12.642 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:54:12.642 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:54:12.698 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-19T20:54:12.704 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:54:12.707 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:54:13.138 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:54:13.342 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:13.635 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-19T20:54:13.753 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:13.968 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-19T20:54:14.183 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:14.183 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:54:14.183 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:54:14.183 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:54:14.183 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:54:14.183 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:54:14.184 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-19T20:54:14.399 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-nbd 2024-02-19T20:54:14.446 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 6/6 2024-02-19T20:54:14.737 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 6/6 2024-02-19T20:54:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 1/6 2024-02-19T20:54:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-19T20:54:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-19T20:54:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-19T20:54:14.738 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-19T20:54:15.132 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-19T20:54:15.132 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:15.132 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:54:15.132 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm-2:16.2.14-548.g81bd20d6.el8.noarch 2024-02-19T20:54:15.132 INFO:teuthology.orchestra.run.smithi163.stdout: python3-apipkg-1.5-6.el8.noarch 2024-02-19T20:54:15.132 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel-2.5.1-7.el8.noarch 2024-02-19T20:54:15.132 INFO:teuthology.orchestra.run.smithi163.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-02-19T20:54:15.132 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-02-19T20:54:15.133 INFO:teuthology.orchestra.run.smithi163.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-02-19T20:54:15.133 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:15.133 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:54:15.160 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:03:03 ago on Mon 19 Feb 2024 08:51:12 PM UTC. 2024-02-19T20:54:15.270 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-fuse 2024-02-19T20:54:16.028 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:06 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:54:16.721 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-02-19T20:54:16.722 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:54:16.722 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-02-19T20:54:16.722 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:54:16.722 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-02-19T20:54:16.723 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 175 k 2024-02-19T20:54:16.723 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:54:16.723 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-02-19T20:54:16.723 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-02-19T20:54:16.723 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-02-19T20:54:16.723 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:54:16.723 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 175 k 2024-02-19T20:54:16.723 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 523 k 2024-02-19T20:54:16.723 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-02-19T20:54:16.942 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-nbd-16.2.14-548.g81bd20d6.el8.x86_64.rpm 802 kB/s | 175 kB 00:00 2024-02-19T20:54:16.942 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:54:16.942 INFO:teuthology.orchestra.run.smithi193.stdout:Total 797 kB/s | 175 kB 00:00 2024-02-19T20:54:16.943 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-02-19T20:54:16.953 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-02-19T20:54:16.953 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-02-19T20:54:17.003 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-02-19T20:54:17.006 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-02-19T20:54:17.142 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-02-19T20:54:17.373 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:17.570 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:54:17.571 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:17.571 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-02-19T20:54:17.571 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:17.571 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:54:17.571 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 911 k 2024-02-19T20:54:17.571 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-02-19T20:54:17.571 INFO:teuthology.orchestra.run.smithi163.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-19T20:54:17.571 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:17.571 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:54:17.572 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:17.572 INFO:teuthology.orchestra.run.smithi163.stdout:Install 2 Packages 2024-02-19T20:54:17.572 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:17.572 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 994 k 2024-02-19T20:54:17.572 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 2.8 M 2024-02-19T20:54:17.573 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:54:17.648 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:17.737 INFO:teuthology.orchestra.run.smithi163.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 670 kB/s | 83 kB 00:00 2024-02-19T20:54:18.034 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:18.034 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:54:18.034 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-02-19T20:54:18.035 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:54:18.035 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:54:18.035 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-02-19T20:54:18.252 DEBUG:teuthology.parallel:result is None 2024-02-19T20:54:18.646 INFO:teuthology.orchestra.run.smithi163.stdout:(2/2): ceph-fuse-16.2.14-548.g81bd20d6.el8.x86_ 883 kB/s | 911 kB 00:01 2024-02-19T20:54:18.646 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:54:18.647 INFO:teuthology.orchestra.run.smithi163.stdout:Total 925 kB/s | 994 kB 00:01 2024-02-19T20:54:18.659 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:54:18.681 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:54:18.681 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:54:18.775 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:54:18.777 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:54:18.943 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:54:19.218 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-19T20:54:19.884 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2/2 2024-02-19T20:54:20.376 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2/2 2024-02-19T20:54:20.377 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/2 2024-02-19T20:54:20.777 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-19T20:54:20.778 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:20.778 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:54:20.778 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-19T20:54:20.778 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:20.778 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:54:20.976 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados-devel 2024-02-19T20:54:21.748 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:11 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:54:23.288 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:54:23.289 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-02-19T20:54:23.289 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:23.289 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:54:23.289 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 131 k 2024-02-19T20:54:23.290 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:23.290 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:54:23.290 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:23.290 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-02-19T20:54:23.290 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:23.290 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 131 k 2024-02-19T20:54:23.290 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 432 k 2024-02-19T20:54:23.290 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:54:23.479 INFO:teuthology.orchestra.run.smithi163.stdout:librados-devel-16.2.14-548.g81bd20d6.el8.x86_64 696 kB/s | 131 kB 00:00 2024-02-19T20:54:23.479 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:54:23.479 INFO:teuthology.orchestra.run.smithi163.stdout:Total 691 kB/s | 131 kB 00:00 2024-02-19T20:54:23.480 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:54:23.485 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:54:23.485 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:54:23.534 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:54:23.537 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:54:23.660 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:54:23.851 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:24.149 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:24.550 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:24.551 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:24.551 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:54:24.551 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:54:24.551 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:24.551 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:54:24.779 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs2 2024-02-19T20:54:25.550 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:15 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:54:27.052 INFO:teuthology.orchestra.run.smithi163.stdout:Package libcephfs2-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:54:27.096 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:54:27.097 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-02-19T20:54:27.097 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:54:27.201 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs-devel 2024-02-19T20:54:27.956 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:17 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:54:29.506 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:54:29.507 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:29.507 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-02-19T20:54:29.507 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:29.507 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:54:29.507 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 25 k 2024-02-19T20:54:29.507 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:29.507 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:54:29.507 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:29.507 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-02-19T20:54:29.507 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:29.508 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 25 k 2024-02-19T20:54:29.508 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 107 k 2024-02-19T20:54:29.508 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:54:29.645 INFO:teuthology.orchestra.run.smithi163.stdout:libcephfs-devel-16.2.14-548.g81bd20d6.el8.x86_6 186 kB/s | 25 kB 00:00 2024-02-19T20:54:29.646 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:54:29.646 INFO:teuthology.orchestra.run.smithi163.stdout:Total 184 kB/s | 25 kB 00:00 2024-02-19T20:54:29.646 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:54:29.650 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:54:29.650 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:54:29.652 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:54:29.655 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:54:29.701 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:54:29.925 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:30.179 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:30.511 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:54:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:54:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:30.511 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:54:30.633 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados2 2024-02-19T20:54:31.387 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:21 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:54:32.876 INFO:teuthology.orchestra.run.smithi163.stdout:Package librados2-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:54:32.922 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:54:32.923 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-02-19T20:54:32.923 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:54:33.016 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librbd1 2024-02-19T20:54:33.772 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:23 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:54:35.276 INFO:teuthology.orchestra.run.smithi163.stdout:Package librbd1-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:54:35.323 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:54:35.324 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-02-19T20:54:35.324 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:54:35.439 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rados 2024-02-19T20:54:36.191 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:26 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:54:37.685 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rados-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:54:37.732 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:54:37.733 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-02-19T20:54:37.733 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:54:37.841 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rgw 2024-02-19T20:54:38.594 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:28 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:54:40.103 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rgw-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:54:40.151 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:54:40.151 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-02-19T20:54:40.151 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:54:40.254 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-cephfs 2024-02-19T20:54:41.028 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:31 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:54:42.559 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-cephfs-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:54:42.606 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:54:42.607 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-02-19T20:54:42.607 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:54:42.719 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rbd 2024-02-19T20:54:43.482 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:33 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:54:44.995 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rbd-2:16.2.14-548.g81bd20d6.el8.x86_64 is already installed. 2024-02-19T20:54:45.041 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:54:45.042 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-02-19T20:54:45.042 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:54:45.160 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-fuse 2024-02-19T20:54:45.917 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:35 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:54:47.453 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:54:47.454 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:47.454 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-02-19T20:54:47.455 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:47.455 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:54:47.455 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 88 k 2024-02-19T20:54:47.455 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:47.455 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:54:47.455 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:47.455 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-02-19T20:54:47.455 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:47.455 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 88 k 2024-02-19T20:54:47.455 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 240 k 2024-02-19T20:54:47.455 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:54:47.726 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-fuse-16.2.14-548.g81bd20d6.el8.x86_64.rpm 326 kB/s | 88 kB 00:00 2024-02-19T20:54:47.727 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:54:47.727 INFO:teuthology.orchestra.run.smithi163.stdout:Total 325 kB/s | 88 kB 00:00 2024-02-19T20:54:47.727 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:54:47.737 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:54:47.737 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:54:47.786 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:54:47.789 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:54:47.891 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:54:48.356 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:48.659 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:49.045 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:49.045 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:49.045 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:54:49.045 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:54:49.045 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:49.045 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:54:49.258 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-mirror 2024-02-19T20:54:50.014 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:40 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:54:51.577 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:54:51.578 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:51.578 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-02-19T20:54:51.578 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:51.579 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:54:51.579 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 3.7 M 2024-02-19T20:54:51.579 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:51.579 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:54:51.579 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:51.579 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-02-19T20:54:51.579 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:51.579 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 3.7 M 2024-02-19T20:54:51.579 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 15 M 2024-02-19T20:54:51.579 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:54:53.190 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-mirror-16.2.14-548.g81bd20d6.el8.x86_64.rpm 2.3 MB/s | 3.7 MB 00:01 2024-02-19T20:54:53.191 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:54:53.191 INFO:teuthology.orchestra.run.smithi163.stdout:Total 2.3 MB/s | 3.7 MB 00:01 2024-02-19T20:54:53.192 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:54:53.205 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:54:53.205 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:54:53.272 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:54:53.275 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:54:53.707 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:54:53.998 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:54.405 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:54.818 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:54.818 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:54.818 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:54:54.818 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:54:54.819 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:54.819 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:54:55.040 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-nbd 2024-02-19T20:54:55.810 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:45 ago on Mon 19 Feb 2024 08:52:10 PM UTC. 2024-02-19T20:54:57.356 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-02-19T20:54:57.356 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:57.357 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-02-19T20:54:57.357 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:57.357 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-02-19T20:54:57.357 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd x86_64 2:16.2.14-548.g81bd20d6.el8 ceph 175 k 2024-02-19T20:54:57.357 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:57.357 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-02-19T20:54:57.357 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-02-19T20:54:57.357 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-02-19T20:54:57.357 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:57.357 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 175 k 2024-02-19T20:54:57.357 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 523 k 2024-02-19T20:54:57.357 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-02-19T20:54:57.695 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-nbd-16.2.14-548.g81bd20d6.el8.x86_64.rpm 519 kB/s | 175 kB 00:00 2024-02-19T20:54:57.695 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-02-19T20:54:57.695 INFO:teuthology.orchestra.run.smithi163.stdout:Total 518 kB/s | 175 kB 00:00 2024-02-19T20:54:57.696 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-02-19T20:54:57.706 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-02-19T20:54:57.706 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-02-19T20:54:57.756 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-02-19T20:54:57.759 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-02-19T20:54:57.874 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-02-19T20:54:58.070 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:58.359 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:58.717 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 1/1 2024-02-19T20:54:58.717 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:58.717 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-02-19T20:54:58.717 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-2:16.2.14-548.g81bd20d6.el8.x86_64 2024-02-19T20:54:58.717 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T20:54:58.717 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-02-19T20:54:58.962 DEBUG:teuthology.parallel:result is None 2024-02-19T20:54:58.963 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T20:54:59.303 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T20:54:59.325 INFO:teuthology.orchestra.run.smithi163.stdout:16.2.14-548.g81bd20d6.el8 2024-02-19T20:54:59.326 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-548.g81bd20d6.el8 2024-02-19T20:54:59.326 INFO:teuthology.task.install:The correct ceph version 16.2.14-548.g81bd20d6 is installed. 2024-02-19T20:54:59.328 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T20:54:59.629 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T20:54:59.655 INFO:teuthology.orchestra.run.smithi193.stdout:16.2.14-548.g81bd20d6.el8 2024-02-19T20:54:59.656 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-548.g81bd20d6.el8 2024-02-19T20:54:59.656 INFO:teuthology.task.install:The correct ceph version 16.2.14-548.g81bd20d6 is installed. 2024-02-19T20:54:59.658 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-19T20:54:59.658 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T20:54:59.658 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T20:54:59.688 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-02-19T20:54:59.688 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T20:54:59.729 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-19T20:54:59.730 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T20:54:59.730 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T20:54:59.758 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T20:54:59.825 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-02-19T20:54:59.826 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T20:54:59.856 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T20:54:59.925 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-19T20:54:59.925 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T20:54:59.925 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T20:54:59.952 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T20:55:00.019 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-02-19T20:55:00.019 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T20:55:00.048 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T20:55:00.117 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-19T20:55:00.117 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T20:55:00.117 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T20:55:00.144 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T20:55:00.211 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-02-19T20:55:00.211 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T20:55:00.240 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T20:55:00.308 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-19T20:55:00.383 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', 'flag\\(s\\) set', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(PG_', '\\(OSD_', {'mons down': None}, 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'mon_bind_msgr2': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1', 'cephadm_mode': 'root'} 2024-02-19T20:55:00.383 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T20:55:00.387 INFO:tasks.cephadm:Cluster fsid is 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:55:00.387 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-19T20:55:00.387 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.163:6789]', 'mon.c': '[v1:172.21.15.163:6790]', 'mon.b': '[v1:172.21.15.193:6789]'} 2024-02-19T20:55:00.387 INFO:tasks.cephadm:First mon is mon.a on smithi163 2024-02-19T20:55:00.388 INFO:tasks.cephadm:First mgr is y 2024-02-19T20:55:00.388 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-19T20:55:00.388 DEBUG:teuthology.orchestra.run.smithi163:> sudo hostname $(hostname -s) 2024-02-19T20:55:00.415 DEBUG:teuthology.orchestra.run.smithi193:> sudo hostname $(hostname -s) 2024-02-19T20:55:00.442 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph-ci.git ref 81bd20d634209c7cb82c18be12b4b5a05643ebf1)... 2024-02-19T20:55:00.443 DEBUG:teuthology.orchestra.run.smithi163:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/81bd20d634209c7cb82c18be12b4b5a05643ebf1/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T20:55:00.832 INFO:teuthology.orchestra.run.smithi163.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Feb 19 20:55 /home/ubuntu/cephtest/cephadm 2024-02-19T20:55:00.832 DEBUG:teuthology.orchestra.run.smithi193:> curl --silent https://raw.githubusercontent.com/ceph/ceph-ci/81bd20d634209c7cb82c18be12b4b5a05643ebf1/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T20:55:01.222 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Feb 19 20:55 /home/ubuntu/cephtest/cephadm 2024-02-19T20:55:01.222 DEBUG:teuthology.orchestra.run.smithi163:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T20:55:01.244 DEBUG:teuthology.orchestra.run.smithi193:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T20:55:01.276 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph 2024-02-19T20:55:01.314 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph 2024-02-19T20:55:01.346 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 777 /etc/ceph 2024-02-19T20:55:01.383 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /etc/ceph 2024-02-19T20:55:01.416 INFO:tasks.cephadm:Writing seed config... 2024-02-19T20:55:01.417 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-19T20:55:01.417 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-19T20:55:01.417 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-19T20:55:01.417 INFO:tasks.cephadm: override: [client] debug objecter = 20 2024-02-19T20:55:01.417 INFO:tasks.cephadm: override: [client] debug rados = 20 2024-02-19T20:55:01.417 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-02-19T20:55:01.417 INFO:tasks.cephadm: override: [global] ms type = async 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [mon] debug mgrc = 20 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-02-19T20:55:01.418 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-02-19T20:55:01.419 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-02-19T20:55:01.419 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-19T20:55:01.419 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-02-19T20:55:01.419 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T20:55:01.419 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-19T20:55:01.442 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 2591809a-cf69-11ee-95bb-87774f69a715 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mgrc = 20 debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon warn on pool no app = False paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 debug objecter = 20 debug rados = 20 2024-02-19T20:55:01.442 DEBUG:teuthology.orchestra.run.smithi163:mon.a> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@mon.a.service 2024-02-19T20:55:01.485 DEBUG:teuthology.orchestra.run.smithi163:mgr.y> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@mgr.y.service 2024-02-19T20:55:01.513 INFO:journalctl@ceph.mon.a.smithi163.stdout:-- Logs begin at Mon 2024-02-19 20:44:35 UTC. -- 2024-02-19T20:55:01.514 INFO:tasks.cephadm:Bootstrapping... 2024-02-19T20:55:01.514 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 -v bootstrap --fsid 2591809a-cf69-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.163:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-19T20:55:01.541 INFO:journalctl@ceph.mgr.y.smithi163.stdout:-- Logs begin at Mon 2024-02-19 20:44:35 UTC. -- 2024-02-19T20:55:01.792 INFO:teuthology.orchestra.run.smithi163.stderr:-------------------------------------------------------------------------------- 2024-02-19T20:55:01.793 INFO:teuthology.orchestra.run.smithi163.stderr:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1', '-v', 'bootstrap', '--fsid', '2591809a-cf69-11ee-95bb-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.163:6789]', '--skip-admin-label'] 2024-02-19T20:55:02.731 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: 4.9.3-dev 2024-02-19T20:55:02.742 INFO:teuthology.orchestra.run.smithi163.stderr:Verifying podman|docker is present... 2024-02-19T20:55:02.766 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: 4.9.3-dev 2024-02-19T20:55:02.778 INFO:teuthology.orchestra.run.smithi163.stderr:Verifying lvm2 is present... 2024-02-19T20:55:02.778 INFO:teuthology.orchestra.run.smithi163.stderr:Verifying time synchronization is in place... 2024-02-19T20:55:02.785 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T20:55:02.792 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: inactive 2024-02-19T20:55:02.799 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: enabled 2024-02-19T20:55:02.805 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: active 2024-02-19T20:55:02.805 INFO:teuthology.orchestra.run.smithi163.stderr:Unit chronyd.service is enabled and running 2024-02-19T20:55:02.805 INFO:teuthology.orchestra.run.smithi163.stderr:Repeating the final host check... 2024-02-19T20:55:02.826 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: 4.9.3-dev 2024-02-19T20:55:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:podman (/bin/podman) version 4.9.3 is present 2024-02-19T20:55:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl is present 2024-02-19T20:55:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:lvcreate is present 2024-02-19T20:55:02.844 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T20:55:02.850 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: inactive 2024-02-19T20:55:02.856 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: enabled 2024-02-19T20:55:02.862 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: active 2024-02-19T20:55:02.863 INFO:teuthology.orchestra.run.smithi163.stderr:Unit chronyd.service is enabled and running 2024-02-19T20:55:02.863 INFO:teuthology.orchestra.run.smithi163.stderr:Host looks OK 2024-02-19T20:55:02.863 INFO:teuthology.orchestra.run.smithi163.stderr:Cluster fsid: 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:55:02.863 INFO:teuthology.orchestra.run.smithi163.stderr:Acquiring lock 139673889450472 on /run/cephadm/2591809a-cf69-11ee-95bb-87774f69a715.lock 2024-02-19T20:55:02.863 INFO:teuthology.orchestra.run.smithi163.stderr:Lock 139673889450472 acquired on /run/cephadm/2591809a-cf69-11ee-95bb-87774f69a715.lock 2024-02-19T20:55:02.863 INFO:teuthology.orchestra.run.smithi163.stderr:Verifying IP 172.21.15.163 port 6789 ... 2024-02-19T20:55:02.864 INFO:teuthology.orchestra.run.smithi163.stderr:Base mon IP(s) is [172.21.15.163:6789], mon addrv is [v1:172.21.15.163:6789] 2024-02-19T20:55:02.867 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.163 metric 100 2024-02-19T20:55:02.867 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.163 metric 100 2024-02-19T20:55:02.869 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-19T20:55:02.870 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-19T20:55:02.870 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1723sec hoplimit 64 pref medium 2024-02-19T20:55:02.872 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-19T20:55:02.872 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-19T20:55:02.873 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-19T20:55:02.873 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-19T20:55:02.873 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:7d5b/64 scope link 2024-02-19T20:55:02.873 INFO:teuthology.orchestra.run.smithi163.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-19T20:55:02.873 INFO:teuthology.orchestra.run.smithi163.stderr:Mon IP `172.21.15.163` is in CIDR network `172.21.0.0/20` 2024-02-19T20:55:02.873 INFO:teuthology.orchestra.run.smithi163.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-02-19T20:55:02.874 INFO:teuthology.orchestra.run.smithi163.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-19T20:55:02.875 INFO:teuthology.orchestra.run.smithi163.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1... 2024-02-19T20:55:02.975 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1... 2024-02-19T20:55:03.153 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: Getting image source signatures 2024-02-19T20:55:03.162 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: Copying blob sha256:ec93555fb25f6296fcd66f232359fb768587e082c45b15dd379853eb0f28a963 2024-02-19T20:55:03.162 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: Copying blob sha256:0f642e61a11afb42c0ccb5b05f5809250ad643ed143780d6ff49a5a4c492fc38 2024-02-19T20:55:28.927 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: Copying config sha256:823b921721f579487e02cd293080702537bcba9f4f96b9041a06d7563de513aa 2024-02-19T20:55:29.044 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: Writing manifest to image destination 2024-02-19T20:55:29.255 INFO:teuthology.orchestra.run.smithi163.stderr:/bin/podman: 823b921721f579487e02cd293080702537bcba9f4f96b9041a06d7563de513aa 2024-02-19T20:55:29.868 INFO:teuthology.orchestra.run.smithi163.stderr:ceph: ceph version 16.2.14-548-g81bd20d6 (81bd20d634209c7cb82c18be12b4b5a05643ebf1) pacific (stable) 2024-02-19T20:55:30.452 INFO:teuthology.orchestra.run.smithi163.stderr:Ceph version: ceph version 16.2.14-548-g81bd20d6 (81bd20d634209c7cb82c18be12b4b5a05643ebf1) pacific (stable) 2024-02-19T20:55:30.452 INFO:teuthology.orchestra.run.smithi163.stderr:Extracting ceph user uid/gid from container image... 2024-02-19T20:55:30.929 INFO:teuthology.orchestra.run.smithi163.stderr:stat: 167 167 2024-02-19T20:55:31.519 INFO:teuthology.orchestra.run.smithi163.stderr:Creating initial keys... 2024-02-19T20:55:32.068 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-authtool: AQBDwNNlsrkWOxAAvH4Q3yAiOnBYyQd6lsMojw== 2024-02-19T20:55:33.081 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-authtool: AQBFwNNljrNFARAAnJf4GJJdA1gZ5E3qRaH4qg== 2024-02-19T20:55:34.062 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-authtool: AQBGwNNlQ9pgABAAIBY7cmzXo1ZHYoRFk/fyow== 2024-02-19T20:55:34.630 INFO:teuthology.orchestra.run.smithi163.stderr:Creating initial monmap... 2024-02-19T20:55:35.151 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T20:55:35.152 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:55:35.152 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T20:55:35.695 INFO:teuthology.orchestra.run.smithi163.stderr:monmaptool for a [v1:172.21.15.163:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T20:55:35.695 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/monmaptool: set fsid to 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:55:35.695 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T20:55:35.695 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-02-19T20:55:35.695 INFO:teuthology.orchestra.run.smithi163.stderr:Creating mon... 2024-02-19T20:55:36.237 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.138+0000 7ff69d173880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T20:55:36.237 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.138+0000 7ff69d173880 1 imported monmap: 2024-02-19T20:55:36.237 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-19T20:55:36.237 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: fsid 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:55:36.237 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: last_changed 2024-02-19T20:55:35.071537+0000 2024-02-19T20:55:36.237 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: created 2024-02-19T20:55:35.071537+0000 2024-02-19T20:55:36.237 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-19T20:55:36.237 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-19T20:55:36.238 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 0: v1:172.21.15.163:6789/0 mon.a 2024-02-19T20:55:36.238 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.238 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.138+0000 7ff69d173880 0 /usr/bin/ceph-mon: set fsid to 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:55:36.238 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T20:55:36.238 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.238 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T20:55:36.238 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Compile date Feb 17 2024 2024-02-19T20:55:36.238 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: DB SUMMARY 2024-02-19T20:55:36.238 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.238 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-02-19T20:55:36.239 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.239 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-02-19T20:55:36.239 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.239 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.error_if_exists: 0 2024-02-19T20:55:36.239 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.create_if_missing: 1 2024-02-19T20:55:36.239 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T20:55:36.239 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.env: 0x556474628080 2024-02-19T20:55:36.239 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.fs: Posix File System 2024-02-19T20:55:36.239 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.info_log: 0x556475e6ba00 2024-02-19T20:55:36.239 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T20:55:36.239 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.statistics: (nil) 2024-02-19T20:55:36.239 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.use_fsync: 0 2024-02-19T20:55:36.239 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T20:55:36.239 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T20:55:36.240 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T20:55:36.240 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T20:55:36.240 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T20:55:36.240 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T20:55:36.240 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T20:55:36.240 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T20:55:36.240 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T20:55:36.240 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T20:55:36.240 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T20:55:36.240 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.db_log_dir: 2024-02-19T20:55:36.240 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-19T20:55:36.240 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T20:55:36.240 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T20:55:36.240 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T20:55:36.241 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T20:55:36.241 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T20:55:36.241 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T20:55:36.241 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T20:55:36.241 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T20:55:36.241 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T20:55:36.241 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T20:55:36.241 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.write_buffer_manager: 0x556475e71950 2024-02-19T20:55:36.241 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T20:55:36.241 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T20:55:36.241 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T20:55:36.241 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T20:55:36.241 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T20:55:36.241 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T20:55:36.241 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T20:55:36.242 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T20:55:36.242 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T20:55:36.242 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.unordered_write: 0 2024-02-19T20:55:36.242 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T20:55:36.242 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T20:55:36.242 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T20:55:36.242 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T20:55:36.242 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.row_cache: None 2024-02-19T20:55:36.242 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.wal_filter: None 2024-02-19T20:55:36.242 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T20:55:36.242 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T20:55:36.242 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T20:55:36.242 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.two_write_queues: 0 2024-02-19T20:55:36.242 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T20:55:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.atomic_flush: 0 2024-02-19T20:55:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T20:55:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T20:55:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T20:55:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T20:55:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T20:55:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T20:55:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T20:55:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T20:55:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T20:55:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T20:55:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T20:55:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T20:55:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T20:55:36.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T20:55:36.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T20:55:36.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_open_files: -1 2024-02-19T20:55:36.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T20:55:36.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T20:55:36.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T20:55:36.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T20:55:36.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Compression algorithms supported: 2024-02-19T20:55:36.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T20:55:36.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: kZSTD supported: 0 2024-02-19T20:55:36.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: kXpressCompression supported: 0 2024-02-19T20:55:36.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T20:55:36.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T20:55:36.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T20:55:36.244 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: kZlibCompression supported: 1 2024-02-19T20:55:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T20:55:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T20:55:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-19T20:55:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-02-19T20:55:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T20:55:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T20:55:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.merge_operator: 2024-02-19T20:55:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compaction_filter: None 2024-02-19T20:55:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T20:55:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T20:55:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T20:55:36.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556475d81d18) 2024-02-19T20:55:36.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-19T20:55:36.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T20:55:36.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T20:55:36.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-19T20:55:36.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-19T20:55:36.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-19T20:55:36.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-19T20:55:36.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-19T20:55:36.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-19T20:55:36.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-19T20:55:36.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-19T20:55:36.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_cache: 0x556475db6d10 2024-02-19T20:55:36.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-19T20:55:36.246 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-19T20:55:36.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-19T20:55:36.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-19T20:55:36.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-19T20:55:36.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-19T20:55:36.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-19T20:55:36.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-19T20:55:36.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-19T20:55:36.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-19T20:55:36.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-19T20:55:36.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-19T20:55:36.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-19T20:55:36.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-19T20:55:36.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-19T20:55:36.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T20:55:36.247 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-19T20:55:36.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-19T20:55:36.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-19T20:55:36.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-19T20:55:36.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-19T20:55:36.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-19T20:55:36.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T20:55:36.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T20:55:36.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compression: NoCompression 2024-02-19T20:55:36.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T20:55:36.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T20:55:36.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T20:55:36.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.num_levels: 7 2024-02-19T20:55:36.248 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T20:55:36.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T20:55:36.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T20:55:36.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T20:55:36.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T20:55:36.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T20:55:36.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T20:55:36.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T20:55:36.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T20:55:36.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T20:55:36.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T20:55:36.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T20:55:36.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T20:55:36.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T20:55:36.249 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T20:55:36.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T20:55:36.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T20:55:36.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T20:55:36.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T20:55:36.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T20:55:36.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T20:55:36.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T20:55:36.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T20:55:36.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T20:55:36.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T20:55:36.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T20:55:36.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T20:55:36.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T20:55:36.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T20:55:36.250 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T20:55:36.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T20:55:36.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T20:55:36.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T20:55:36.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T20:55:36.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T20:55:36.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T20:55:36.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T20:55:36.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T20:55:36.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T20:55:36.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T20:55:36.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T20:55:36.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T20:55:36.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T20:55:36.251 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T20:55:36.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T20:55:36.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T20:55:36.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T20:55:36.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.table_properties_collectors: 2024-02-19T20:55:36.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T20:55:36.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T20:55:36.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T20:55:36.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T20:55:36.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T20:55:36.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.bloom_locality: 0 2024-02-19T20:55:36.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T20:55:36.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T20:55:36.252 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T20:55:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T20:55:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T20:55:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.ttl: 2592000 2024-02-19T20:55:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.139+0000 7ff69d173880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T20:55:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.140+0000 7ff69d173880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T20:55:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.140+0000 7ff69d173880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-19T20:55:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.141+0000 7ff69d173880 4 rocksdb: DB pointer 0x556475e7f800 2024-02-19T20:55:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.141+0000 7ff685fe6700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T20:55:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.141+0000 7ff685fe6700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T20:55:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-19T20:55:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T20:55:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T20:55:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T20:55:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T20:55:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T20:55:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T20:55:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T20:55:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T20:55:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:55:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:55:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T20:55:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T20:55:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T20:55:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:55:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:55:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T20:55:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T20:55:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T20:55:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T20:55:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T20:55:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-19T20:55:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:55:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:55:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T20:55:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-19T20:55:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T20:55:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:55:36.255 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:55:36.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T20:55:36.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T20:55:36.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T20:55:36.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:55:36.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:55:36.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T20:55:36.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T20:55:36.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T20:55:36.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T20:55:36.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T20:55:36.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-19T20:55:36.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:55:36.256 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:55:36.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T20:55:36.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-19T20:55:36.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: 2024-02-19T20:55:36.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.142+0000 7ff69d173880 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-19T20:55:36.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.142+0000 7ff69d173880 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-19T20:55:36.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph-mon: debug 2024-02-19T20:55:36.142+0000 7ff69d173880 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-02-19T20:55:36.718 INFO:teuthology.orchestra.run.smithi163.stderr:create mon.a on 2024-02-19T20:55:36.731 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-19T20:55:36.823 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-19T20:55:36.916 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-2591809a-cf69-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-2591809a-cf69-11ee-95bb-87774f69a715.target. 2024-02-19T20:55:36.916 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-2591809a-cf69-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-2591809a-cf69-11ee-95bb-87774f69a715.target. 2024-02-19T20:55:37.111 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Failed to reset failed state of unit ceph-2591809a-cf69-11ee-95bb-87774f69a715@mon.a.service: Unit ceph-2591809a-cf69-11ee-95bb-87774f69a715@mon.a.service not loaded. 2024-02-19T20:55:37.118 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Created symlink /etc/systemd/system/ceph-2591809a-cf69-11ee-95bb-87774f69a715.target.wants/ceph-2591809a-cf69-11ee-95bb-87774f69a715@mon.a.service → /etc/systemd/system/ceph-2591809a-cf69-11ee-95bb-87774f69a715@.service. 2024-02-19T20:55:37.785 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: disabled 2024-02-19T20:55:37.792 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: inactive 2024-02-19T20:55:37.821 INFO:teuthology.orchestra.run.smithi163.stderr:firewalld.service is not enabled 2024-02-19T20:55:37.821 INFO:teuthology.orchestra.run.smithi163.stderr:Not possible to enable service . firewalld.service is not available 2024-02-19T20:55:37.821 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for mon to start... 2024-02-19T20:55:37.821 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for mon... 2024-02-19T20:55:38.057 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[37721]: cluster 2024-02-19T20:55:37.777020+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-19T20:55:38.308 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.326+0000 7fcb0cf1f700 1 Processor -- start 2024-02-19T20:55:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.326+0000 7fcb0cf1f700 1 -- start start 2024-02-19T20:55:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.326+0000 7fcb0cf1f700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcb081001b0 con 0x7fcb08070c20 2024-02-19T20:55:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.327+0000 7fcb0659c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fcb08070c20 0x7fcb080fea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:47250/0 (socket says 172.21.15.163:47250) 2024-02-19T20:55:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.327+0000 7fcb0659c700 1 -- 172.21.15.163:0/62632362 learned_addr learned my addr 172.21.15.163:0/62632362 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.327+0000 7fcb0559a700 1 -- 172.21.15.163:0/62632362 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022772093 0 0) 0x7fcb081001b0 con 0x7fcb08070c20 2024-02-19T20:55:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.327+0000 7fcb0559a700 1 -- 172.21.15.163:0/62632362 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf00033f0 con 0x7fcb08070c20 2024-02-19T20:55:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.327+0000 7fcb0559a700 1 -- 172.21.15.163:0/62632362 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4283436298 0 0) 0x7fcaf00033f0 con 0x7fcb08070c20 2024-02-19T20:55:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.327+0000 7fcb0559a700 1 -- 172.21.15.163:0/62632362 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb08101350 con 0x7fcb08070c20 2024-02-19T20:55:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.327+0000 7fcb0559a700 1 -- 172.21.15.163:0/62632362 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fcafc001590 con 0x7fcb08070c20 2024-02-19T20:55:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.327+0000 7fcb0559a700 1 -- 172.21.15.163:0/62632362 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcafc001880 con 0x7fcb08070c20 2024-02-19T20:55:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.327+0000 7fcb0559a700 1 -- 172.21.15.163:0/62632362 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fcafc001bd0 con 0x7fcb08070c20 2024-02-19T20:55:38.309 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.328+0000 7fcb0cf1f700 1 -- 172.21.15.163:0/62632362 >> v1:172.21.15.163:6789/0 conn(0x7fcb08070c20 legacy=0x7fcb080fea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:38.310 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.328+0000 7fcb0cf1f700 1 -- 172.21.15.163:0/62632362 shutdown_connections 2024-02-19T20:55:38.310 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.328+0000 7fcb0cf1f700 1 -- 172.21.15.163:0/62632362 >> 172.21.15.163:0/62632362 conn(0x7fcb080f64d0 msgr2=0x7fcb080f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:38.310 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.328+0000 7fcb0cf1f700 1 -- 172.21.15.163:0/62632362 shutdown_connections 2024-02-19T20:55:38.310 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.328+0000 7fcb0cf1f700 1 -- 172.21.15.163:0/62632362 wait complete. 2024-02-19T20:55:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.328+0000 7fcb0cf1f700 1 librados: starting msgr at 2024-02-19T20:55:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.328+0000 7fcb0cf1f700 1 librados: starting objecter 2024-02-19T20:55:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.329+0000 7fcb0cf1f700 1 Processor -- start 2024-02-19T20:55:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.329+0000 7fcb0cf1f700 1 -- start start 2024-02-19T20:55:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.329+0000 7fcb0cf1f700 1 librados: setting wanted keys 2024-02-19T20:55:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.329+0000 7fcb0cf1f700 1 librados: calling monclient init 2024-02-19T20:55:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.329+0000 7fcb0cf1f700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcb0810a700 con 0x7fcb08070c20 2024-02-19T20:55:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.329+0000 7fcb0659c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fcb08070c20 0x7fcb08109ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:47262/0 (socket says 172.21.15.163:47262) 2024-02-19T20:55:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.329+0000 7fcb0659c700 1 -- 172.21.15.163:0/1101214024 learned_addr learned my addr 172.21.15.163:0/1101214024 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.329+0000 7fcaf77fe700 10 client.?.objecter ms_handle_connect 0x7fcb08070c20 2024-02-19T20:55:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.329+0000 7fcaf77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:55:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.329+0000 7fcaf77fe700 1 -- 172.21.15.163:0/1101214024 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136768840 0 0) 0x7fcb0810a700 con 0x7fcb08070c20 2024-02-19T20:55:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.329+0000 7fcaf77fe700 1 -- 172.21.15.163:0/1101214024 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf0004400 con 0x7fcb08070c20 2024-02-19T20:55:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.330+0000 7fcaf77fe700 1 -- 172.21.15.163:0/1101214024 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3646790703 0 0) 0x7fcaf0004400 con 0x7fcb08070c20 2024-02-19T20:55:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.330+0000 7fcaf77fe700 1 -- 172.21.15.163:0/1101214024 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb0810a700 con 0x7fcb08070c20 2024-02-19T20:55:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.330+0000 7fcaf77fe700 1 -- 172.21.15.163:0/1101214024 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fcafc001590 con 0x7fcb08070c20 2024-02-19T20:55:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.330+0000 7fcaf77fe700 1 -- 172.21.15.163:0/1101214024 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2205066267 0 0) 0x7fcb0810a700 con 0x7fcb08070c20 2024-02-19T20:55:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.330+0000 7fcaf77fe700 1 -- 172.21.15.163:0/1101214024 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0810a890 con 0x7fcb08070c20 2024-02-19T20:55:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.330+0000 7fcb0cf1f700 1 -- 172.21.15.163:0/1101214024 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb0810ab40 con 0x7fcb08070c20 2024-02-19T20:55:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.330+0000 7fcb0cf1f700 10 client.4098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:55:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.330+0000 7fcb0cf1f700 1 -- 172.21.15.163:0/1101214024 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb0810e830 con 0x7fcb08070c20 2024-02-19T20:55:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.330+0000 7fcb0cf1f700 1 librados: init done 2024-02-19T20:55:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.330+0000 7fcaf77fe700 1 -- 172.21.15.163:0/1101214024 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcafc004a20 con 0x7fcb08070c20 2024-02-19T20:55:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.330+0000 7fcaf77fe700 1 -- 172.21.15.163:0/1101214024 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fcafc004d30 con 0x7fcb08070c20 2024-02-19T20:55:38.315 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.331+0000 7fcaf77fe700 1 -- 172.21.15.163:0/1101214024 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 1) v1 ==== 495+0+0 (unknown 2835230805 0 0) 0x7fcafc006180 con 0x7fcb08070c20 2024-02-19T20:55:38.315 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.331+0000 7fcaf77fe700 1 -- 172.21.15.163:0/1101214024 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 2758453317 0 0) 0x7fcafc0056c0 con 0x7fcb08070c20 2024-02-19T20:55:38.315 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.331+0000 7fcaf77fe700 10 client.4098.objecter ms_dispatch 0x7fcb08001550 osd_map(1..1 src has 1..1) v4 2024-02-19T20:55:38.315 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.331+0000 7fcaf77fe700 3 client.4098.objecter handle_osd_map got epochs [1,1] > 0 2024-02-19T20:55:38.315 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.331+0000 7fcaf77fe700 3 client.4098.objecter handle_osd_map decoding full epoch 1 2024-02-19T20:55:38.315 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.331+0000 7fcaf77fe700 20 client.4098.objecter dump_active .. 0 homeless 2024-02-19T20:55:38.315 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.331+0000 7fcb0cf1f700 1 -- 172.21.15.163:0/1101214024 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae8004f50 con 0x7fcb08070c20 2024-02-19T20:55:38.315 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.333+0000 7fcaf77fe700 1 -- 172.21.15.163:0/1101214024 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (unknown 1092875540 0 259193764) 0x7fcafc004e90 con 0x7fcb08070c20 2024-02-19T20:55:38.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.400+0000 7fcb0cf1f700 1 -- 172.21.15.163:0/1101214024 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fcae8005a10 con 0x7fcb08070c20 2024-02-19T20:55:38.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.400+0000 7fcaf77fe700 1 -- 172.21.15.163:0/1101214024 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (unknown 1155462804 0 680035854) 0x7fcafc0059a0 con 0x7fcb08070c20 2024-02-19T20:55:38.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: cluster: 2024-02-19T20:55:38.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: id: 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:55:38.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-19T20:55:38.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: services: 2024-02-19T20:55:38.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.607929s) 2024-02-19T20:55:38.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-19T20:55:38.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-19T20:55:38.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: data: 2024-02-19T20:55:38.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-19T20:55:38.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-19T20:55:38.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-19T20:55:38.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: pgs: 2024-02-19T20:55:38.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.400+0000 7fcb0cf1f700 10 librados: watch_flush enter 2024-02-19T20:55:38.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.401+0000 7fcb0cf1f700 10 librados: watch_flush exit 2024-02-19T20:55:38.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.401+0000 7fcb0cf1f700 20 client.4098.objecter shutdown clearing up homeless session... 2024-02-19T20:55:38.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.401+0000 7fcb0cf1f700 10 client.4098.objecter successfully canceled tick 2024-02-19T20:55:38.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.401+0000 7fcb0cf1f700 1 -- 172.21.15.163:0/1101214024 >> v1:172.21.15.163:6789/0 conn(0x7fcb08070c20 legacy=0x7fcb08109ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:38.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.401+0000 7fcb0cf1f700 1 -- 172.21.15.163:0/1101214024 shutdown_connections 2024-02-19T20:55:38.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.401+0000 7fcb0cf1f700 1 -- 172.21.15.163:0/1101214024 >> 172.21.15.163:0/1101214024 conn(0x7fcb080f64d0 msgr2=0x7fcb080fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:38.385 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.401+0000 7fcb0cf1f700 1 -- 172.21.15.163:0/1101214024 shutdown_connections 2024-02-19T20:55:38.385 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.401+0000 7fcb0cf1f700 1 -- 172.21.15.163:0/1101214024 wait complete. 2024-02-19T20:55:38.385 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:38.401+0000 7fcb0cf1f700 1 librados: shutdown 2024-02-19T20:55:38.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[37721]: cluster 2024-02-19T20:55:37.793732+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-19T20:55:38.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[37721]: cluster 2024-02-19T20:55:37.793901+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=v1:172.21.15.163:6789/0} removed_ranks: {} 2024-02-19T20:55:38.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[37721]: cluster 2024-02-19T20:55:37.797258+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2024-02-19T20:55:38.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[37721]: cluster 2024-02-19T20:55:37.799114+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-19T20:55:38.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[37721]: cluster 2024-02-19T20:55:37.799647+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-19T20:55:38.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[37721]: audit 2024-02-19T20:55:38.401622+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? v1:172.21.15.163:0/1101214024' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-19T20:55:38.979 INFO:teuthology.orchestra.run.smithi163.stderr:mon is available 2024-02-19T20:55:38.980 INFO:teuthology.orchestra.run.smithi163.stderr:Assimilating anything we can from ceph.conf... 2024-02-19T20:55:39.477 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.478+0000 7efeba9ce700 1 Processor -- start 2024-02-19T20:55:39.478 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.478+0000 7efeba9ce700 1 -- start start 2024-02-19T20:55:39.478 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.478+0000 7efeba9ce700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efeb41001f0 con 0x7efeb4070be0 2024-02-19T20:55:39.478 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.478+0000 7efeb3fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7efeb4070be0 0x7efeb40fead0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:47270/0 (socket says 172.21.15.163:47270) 2024-02-19T20:55:39.478 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.478+0000 7efeb3fff700 1 -- 172.21.15.163:0/3432992691 learned_addr learned my addr 172.21.15.163:0/3432992691 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:39.478 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.478+0000 7efeb2ffd700 1 -- 172.21.15.163:0/3432992691 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761017145 0 0) 0x7efeb41001f0 con 0x7efeb4070be0 2024-02-19T20:55:39.478 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.478+0000 7efeb2ffd700 1 -- 172.21.15.163:0/3432992691 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9c0033f0 con 0x7efeb4070be0 2024-02-19T20:55:39.478 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.479+0000 7efeb2ffd700 1 -- 172.21.15.163:0/3432992691 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1830248002 0 0) 0x7efe9c0033f0 con 0x7efeb4070be0 2024-02-19T20:55:39.478 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.479+0000 7efeb2ffd700 1 -- 172.21.15.163:0/3432992691 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb4101390 con 0x7efeb4070be0 2024-02-19T20:55:39.478 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.479+0000 7efeb2ffd700 1 -- 172.21.15.163:0/3432992691 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7efea4001590 con 0x7efeb4070be0 2024-02-19T20:55:39.478 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.479+0000 7efeb2ffd700 1 -- 172.21.15.163:0/3432992691 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efea40018f0 con 0x7efeb4070be0 2024-02-19T20:55:39.478 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.479+0000 7efeb2ffd700 1 -- 172.21.15.163:0/3432992691 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7efea4001c40 con 0x7efeb4070be0 2024-02-19T20:55:39.478 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.479+0000 7efeba9ce700 1 -- 172.21.15.163:0/3432992691 >> v1:172.21.15.163:6789/0 conn(0x7efeb4070be0 legacy=0x7efeb40fead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:39.478 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.479+0000 7efeba9ce700 1 -- 172.21.15.163:0/3432992691 shutdown_connections 2024-02-19T20:55:39.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.479+0000 7efeba9ce700 1 -- 172.21.15.163:0/3432992691 >> 172.21.15.163:0/3432992691 conn(0x7efeb40f64d0 msgr2=0x7efeb40f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:39.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.479+0000 7efeba9ce700 1 -- 172.21.15.163:0/3432992691 shutdown_connections 2024-02-19T20:55:39.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.479+0000 7efeba9ce700 1 -- 172.21.15.163:0/3432992691 wait complete. 2024-02-19T20:55:39.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.480+0000 7efeba9ce700 1 librados: starting msgr at 2024-02-19T20:55:39.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.480+0000 7efeba9ce700 1 librados: starting objecter 2024-02-19T20:55:39.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.480+0000 7efeba9ce700 1 Processor -- start 2024-02-19T20:55:39.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.480+0000 7efeba9ce700 1 -- start start 2024-02-19T20:55:39.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.480+0000 7efeba9ce700 1 librados: setting wanted keys 2024-02-19T20:55:39.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.480+0000 7efeba9ce700 1 librados: calling monclient init 2024-02-19T20:55:39.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.480+0000 7efeba9ce700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efeb410a6f0 con 0x7efeb4070be0 2024-02-19T20:55:39.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.480+0000 7efeb3fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7efeb4070be0 0x7efeb4109fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:47278/0 (socket says 172.21.15.163:47278) 2024-02-19T20:55:39.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.480+0000 7efeb3fff700 1 -- 172.21.15.163:0/615391492 learned_addr learned my addr 172.21.15.163:0/615391492 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:39.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.480+0000 7efeb17fa700 10 client.?.objecter ms_handle_connect 0x7efeb4070be0 2024-02-19T20:55:39.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.480+0000 7efeb17fa700 10 client.?.objecter resend_mon_ops 2024-02-19T20:55:39.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.480+0000 7efeb17fa700 1 -- 172.21.15.163:0/615391492 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731241728 0 0) 0x7efeb410a6f0 con 0x7efeb4070be0 2024-02-19T20:55:39.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.480+0000 7efeb17fa700 1 -- 172.21.15.163:0/615391492 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9c004400 con 0x7efeb4070be0 2024-02-19T20:55:39.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeb17fa700 1 -- 172.21.15.163:0/615391492 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 589810928 0 0) 0x7efe9c004400 con 0x7efeb4070be0 2024-02-19T20:55:39.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeb17fa700 1 -- 172.21.15.163:0/615391492 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efeb410a6f0 con 0x7efeb4070be0 2024-02-19T20:55:39.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeb17fa700 1 -- 172.21.15.163:0/615391492 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7efea4001550 con 0x7efeb4070be0 2024-02-19T20:55:39.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeb17fa700 1 -- 172.21.15.163:0/615391492 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2531225991 0 0) 0x7efeb410a6f0 con 0x7efeb4070be0 2024-02-19T20:55:39.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeb17fa700 1 -- 172.21.15.163:0/615391492 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb410a880 con 0x7efeb4070be0 2024-02-19T20:55:39.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeba9ce700 1 -- 172.21.15.163:0/615391492 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb410ab30 con 0x7efeb4070be0 2024-02-19T20:55:39.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeba9ce700 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:55:39.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeba9ce700 1 -- 172.21.15.163:0/615391492 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb410e820 con 0x7efeb4070be0 2024-02-19T20:55:39.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeba9ce700 1 librados: init done 2024-02-19T20:55:39.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeb17fa700 1 -- 172.21.15.163:0/615391492 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efea4004bf0 con 0x7efeb4070be0 2024-02-19T20:55:39.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeb17fa700 1 -- 172.21.15.163:0/615391492 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7efea4004f00 con 0x7efeb4070be0 2024-02-19T20:55:39.481 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeb17fa700 1 -- 172.21.15.163:0/615391492 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 1) v1 ==== 495+0+0 (unknown 2835230805 0 0) 0x7efea40052c0 con 0x7efeb4070be0 2024-02-19T20:55:39.481 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeb17fa700 1 -- 172.21.15.163:0/615391492 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 2758453317 0 0) 0x7efea4006250 con 0x7efeb4070be0 2024-02-19T20:55:39.481 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeb17fa700 10 client.4100.objecter ms_dispatch 0x7efeb4001550 osd_map(1..1 src has 1..1) v4 2024-02-19T20:55:39.481 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeb17fa700 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2024-02-19T20:55:39.481 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeb17fa700 3 client.4100.objecter handle_osd_map decoding full epoch 1 2024-02-19T20:55:39.481 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.481+0000 7efeb17fa700 20 client.4100.objecter dump_active .. 0 homeless 2024-02-19T20:55:39.481 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.482+0000 7efeba9ce700 1 -- 172.21.15.163:0/615391492 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea0004f50 con 0x7efeb4070be0 2024-02-19T20:55:39.481 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.484+0000 7efeb17fa700 1 -- 172.21.15.163:0/615391492 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (unknown 1092875540 0 259193764) 0x7efea4005700 con 0x7efeb4070be0 2024-02-19T20:55:39.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.549+0000 7efeba9ce700 1 -- 172.21.15.163:0/615391492 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7efea0005780 con 0x7efeb4070be0 2024-02-19T20:55:39.534 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.552+0000 7efeb17fa700 1 -- 172.21.15.163:0/615391492 <== mon.0 v1:172.21.15.163:6789/0 10 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7efea4005920 con 0x7efeb4070be0 2024-02-19T20:55:39.534 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.552+0000 7efeb17fa700 1 -- 172.21.15.163:0/615391492 <== mon.0 v1:172.21.15.163:6789/0 11 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+343 (unknown 1213389831 0 1636998692) 0x7efea4007670 con 0x7efeb4070be0 2024-02-19T20:55:39.534 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.534 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: [global] 2024-02-19T20:55:39.534 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: fsid = 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:55:39.534 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-19T20:55:39.534 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon_host = [v1:172.21.15.163:6789] 2024-02-19T20:55:39.534 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-19T20:55:39.534 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-19T20:55:39.534 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-19T20:55:39.535 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-19T20:55:39.535 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.535 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: [mgr] 2024-02-19T20:55:39.535 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-19T20:55:39.535 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.535 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: [osd] 2024-02-19T20:55:39.535 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-19T20:55:39.535 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-19T20:55:39.535 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.553+0000 7efeba9ce700 10 librados: watch_flush enter 2024-02-19T20:55:39.535 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.553+0000 7efeba9ce700 10 librados: watch_flush exit 2024-02-19T20:55:39.535 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.553+0000 7efeba9ce700 20 client.4100.objecter shutdown clearing up homeless session... 2024-02-19T20:55:39.535 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.553+0000 7efeba9ce700 10 client.4100.objecter successfully canceled tick 2024-02-19T20:55:39.535 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.553+0000 7efeba9ce700 1 -- 172.21.15.163:0/615391492 >> v1:172.21.15.163:6789/0 conn(0x7efeb4070be0 legacy=0x7efeb4109fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:39.535 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.553+0000 7efeba9ce700 1 -- 172.21.15.163:0/615391492 shutdown_connections 2024-02-19T20:55:39.536 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.553+0000 7efeba9ce700 1 -- 172.21.15.163:0/615391492 >> 172.21.15.163:0/615391492 conn(0x7efeb40f64d0 msgr2=0x7efeb40fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:39.536 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.553+0000 7efeba9ce700 1 -- 172.21.15.163:0/615391492 shutdown_connections 2024-02-19T20:55:39.536 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.553+0000 7efeba9ce700 1 -- 172.21.15.163:0/615391492 wait complete. 2024-02-19T20:55:39.536 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:39.554+0000 7efeba9ce700 1 librados: shutdown 2024-02-19T20:55:39.824 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[37721]: audit 2024-02-19T20:55:39.549576+0000 mon.a (mon.0) 8 : audit [INF] from='client.? v1:172.21.15.163:0/615391492' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-19T20:55:39.824 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[37721]: audit 2024-02-19T20:55:39.552457+0000 mon.a (mon.0) 9 : audit [INF] from='client.? v1:172.21.15.163:0/615391492' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-19T20:55:40.085 INFO:teuthology.orchestra.run.smithi163.stderr:Generating new minimal ceph.conf... 2024-02-19T20:55:40.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.596+0000 7f4047419700 1 Processor -- start 2024-02-19T20:55:40.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.596+0000 7f4047419700 1 -- start start 2024-02-19T20:55:40.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.596+0000 7f4047419700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f40401001b0 con 0x7f4040070c20 2024-02-19T20:55:40.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.597+0000 7f40451b5700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f4040070c20 0x7f40400fea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:47284/0 (socket says 172.21.15.163:47284) 2024-02-19T20:55:40.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.597+0000 7f40451b5700 1 -- 172.21.15.163:0/1206032004 learned_addr learned my addr 172.21.15.163:0/1206032004 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:40.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.597+0000 7f4037fff700 1 -- 172.21.15.163:0/1206032004 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765006043 0 0) 0x7f40401001b0 con 0x7f4040070c20 2024-02-19T20:55:40.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.597+0000 7f4037fff700 1 -- 172.21.15.163:0/1206032004 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f402c0033f0 con 0x7f4040070c20 2024-02-19T20:55:40.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.597+0000 7f4037fff700 1 -- 172.21.15.163:0/1206032004 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2408686036 0 0) 0x7f402c0033f0 con 0x7f4040070c20 2024-02-19T20:55:40.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.597+0000 7f4037fff700 1 -- 172.21.15.163:0/1206032004 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4040101350 con 0x7f4040070c20 2024-02-19T20:55:40.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.597+0000 7f4037fff700 1 -- 172.21.15.163:0/1206032004 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f403c001590 con 0x7f4040070c20 2024-02-19T20:55:40.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.597+0000 7f4037fff700 1 -- 172.21.15.163:0/1206032004 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f403c001d10 con 0x7f4040070c20 2024-02-19T20:55:40.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.597+0000 7f4037fff700 1 -- 172.21.15.163:0/1206032004 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f403c0056e0 con 0x7f4040070c20 2024-02-19T20:55:40.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.598+0000 7f4047419700 1 -- 172.21.15.163:0/1206032004 >> v1:172.21.15.163:6789/0 conn(0x7f4040070c20 legacy=0x7f40400fea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:40.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.598+0000 7f4047419700 1 -- 172.21.15.163:0/1206032004 shutdown_connections 2024-02-19T20:55:40.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.598+0000 7f4047419700 1 -- 172.21.15.163:0/1206032004 >> 172.21.15.163:0/1206032004 conn(0x7f40400f64d0 msgr2=0x7f40400f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:40.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.598+0000 7f4047419700 1 -- 172.21.15.163:0/1206032004 shutdown_connections 2024-02-19T20:55:40.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.598+0000 7f4047419700 1 -- 172.21.15.163:0/1206032004 wait complete. 2024-02-19T20:55:40.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.598+0000 7f4047419700 1 librados: starting msgr at 2024-02-19T20:55:40.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.598+0000 7f4047419700 1 librados: starting objecter 2024-02-19T20:55:40.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.599+0000 7f4047419700 1 Processor -- start 2024-02-19T20:55:40.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.599+0000 7f4047419700 1 -- start start 2024-02-19T20:55:40.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.599+0000 7f4047419700 1 librados: setting wanted keys 2024-02-19T20:55:40.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.599+0000 7f4047419700 1 librados: calling monclient init 2024-02-19T20:55:40.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.599+0000 7f4047419700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f40400fc4a0 con 0x7f4040070c20 2024-02-19T20:55:40.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.599+0000 7f40451b5700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f4040070c20 0x7f40400fa990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:47296/0 (socket says 172.21.15.163:47296) 2024-02-19T20:55:40.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.599+0000 7f40451b5700 1 -- 172.21.15.163:0/3499182339 learned_addr learned my addr 172.21.15.163:0/3499182339 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:40.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.599+0000 7f40367fc700 10 client.?.objecter ms_handle_connect 0x7f4040070c20 2024-02-19T20:55:40.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.599+0000 7f40367fc700 10 client.?.objecter resend_mon_ops 2024-02-19T20:55:40.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.599+0000 7f40367fc700 1 -- 172.21.15.163:0/3499182339 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534570119 0 0) 0x7f40400fc4a0 con 0x7f4040070c20 2024-02-19T20:55:40.582 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.599+0000 7f40367fc700 1 -- 172.21.15.163:0/3499182339 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f402c005580 con 0x7f4040070c20 2024-02-19T20:55:40.582 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.599+0000 7f40367fc700 1 -- 172.21.15.163:0/3499182339 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4120460845 0 0) 0x7f402c005580 con 0x7f4040070c20 2024-02-19T20:55:40.582 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f40367fc700 1 -- 172.21.15.163:0/3499182339 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40400fc4a0 con 0x7f4040070c20 2024-02-19T20:55:40.582 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f40367fc700 1 -- 172.21.15.163:0/3499182339 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f403c000fe0 con 0x7f4040070c20 2024-02-19T20:55:40.582 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f40367fc700 1 -- 172.21.15.163:0/3499182339 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2104988285 0 0) 0x7f40400fc4a0 con 0x7f4040070c20 2024-02-19T20:55:40.582 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f40367fc700 1 -- 172.21.15.163:0/3499182339 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40400fc630 con 0x7f4040070c20 2024-02-19T20:55:40.582 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f4047419700 1 -- 172.21.15.163:0/3499182339 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40400fc8e0 con 0x7f4040070c20 2024-02-19T20:55:40.582 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f4047419700 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:55:40.583 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f4047419700 1 -- 172.21.15.163:0/3499182339 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40400fb3a0 con 0x7f4040070c20 2024-02-19T20:55:40.583 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f4047419700 1 librados: init done 2024-02-19T20:55:40.583 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f40367fc700 1 -- 172.21.15.163:0/3499182339 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f403c0046b0 con 0x7f4040070c20 2024-02-19T20:55:40.583 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f40367fc700 1 -- 172.21.15.163:0/3499182339 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f403c006120 con 0x7f4040070c20 2024-02-19T20:55:40.583 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f40367fc700 1 -- 172.21.15.163:0/3499182339 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 1) v1 ==== 495+0+0 (unknown 2835230805 0 0) 0x7f403c006560 con 0x7f4040070c20 2024-02-19T20:55:40.583 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f40367fc700 1 -- 172.21.15.163:0/3499182339 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 2758453317 0 0) 0x7f403c007510 con 0x7f4040070c20 2024-02-19T20:55:40.583 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f40367fc700 10 client.4102.objecter ms_dispatch 0x7f4040001550 osd_map(1..1 src has 1..1) v4 2024-02-19T20:55:40.583 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f40367fc700 3 client.4102.objecter handle_osd_map got epochs [1,1] > 0 2024-02-19T20:55:40.583 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f40367fc700 3 client.4102.objecter handle_osd_map decoding full epoch 1 2024-02-19T20:55:40.583 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.600+0000 7f40367fc700 20 client.4102.objecter dump_active .. 0 homeless 2024-02-19T20:55:40.583 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.601+0000 7f4047419700 1 -- 172.21.15.163:0/3499182339 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4024004f50 con 0x7f4040070c20 2024-02-19T20:55:40.583 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.603+0000 7f40367fc700 1 -- 172.21.15.163:0/3499182339 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (unknown 1092875540 0 259193764) 0x7f40400fc8e0 con 0x7f4040070c20 2024-02-19T20:55:40.646 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.667+0000 7f4047419700 1 -- 172.21.15.163:0/3499182339 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f4024005200 con 0x7f4040070c20 2024-02-19T20:55:40.647 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.667+0000 7f40367fc700 1 -- 172.21.15.163:0/3499182339 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+148 (unknown 1452402520 0 3953335000) 0x7f403c005b10 con 0x7f4040070c20 2024-02-19T20:55:40.648 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.668+0000 7f4047419700 10 librados: watch_flush enter 2024-02-19T20:55:40.648 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.668+0000 7f4047419700 10 librados: watch_flush exit 2024-02-19T20:55:40.648 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.668+0000 7f4047419700 20 client.4102.objecter shutdown clearing up homeless session... 2024-02-19T20:55:40.648 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.668+0000 7f4047419700 10 client.4102.objecter successfully canceled tick 2024-02-19T20:55:40.649 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.668+0000 7f4047419700 1 -- 172.21.15.163:0/3499182339 >> v1:172.21.15.163:6789/0 conn(0x7f4040070c20 legacy=0x7f40400fa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:40.649 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.668+0000 7f4047419700 1 -- 172.21.15.163:0/3499182339 shutdown_connections 2024-02-19T20:55:40.649 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.668+0000 7f4047419700 1 -- 172.21.15.163:0/3499182339 >> 172.21.15.163:0/3499182339 conn(0x7f40400f64d0 msgr2=0x7f40400fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:40.649 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.668+0000 7f4047419700 1 -- 172.21.15.163:0/3499182339 shutdown_connections 2024-02-19T20:55:40.649 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.668+0000 7f4047419700 1 -- 172.21.15.163:0/3499182339 wait complete. 2024-02-19T20:55:40.649 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:40.668+0000 7f4047419700 1 librados: shutdown 2024-02-19T20:55:40.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[37721]: audit 2024-02-19T20:55:40.667778+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? v1:172.21.15.163:0/3499182339' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:55:41.183 INFO:teuthology.orchestra.run.smithi163.stderr:Restarting the monitor... 2024-02-19T20:55:41.440 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:41 smithi163 systemd[1]: Stopping Ceph mon.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T20:55:41.440 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:41 smithi163 bash[38045]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon.a" found: no such container 2024-02-19T20:55:41.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[37721]: debug 2024-02-19T20:55:41.460+0000 7f23dbd31700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T20:55:41.728 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[37721]: debug 2024-02-19T20:55:41.460+0000 7f23dbd31700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T20:55:42.202 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 bash[38083]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a 2024-02-19T20:55:42.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 systemd[1]: ceph-2591809a-cf69-11ee-95bb-87774f69a715@mon.a.service: Succeeded. 2024-02-19T20:55:42.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 systemd[1]: Stopped Ceph mon.a for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T20:55:42.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 systemd[1]: Starting Ceph mon.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T20:55:42.925 INFO:teuthology.orchestra.run.smithi163.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-19T20:55:43.124 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.876+0000 7fb61d011880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T20:55:43.124 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.876+0000 7fb61d011880 0 ceph version 16.2.14-548-g81bd20d6 (81bd20d634209c7cb82c18be12b4b5a05643ebf1) pacific (stable), process ceph-mon, pid 7 2024-02-19T20:55:43.124 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.876+0000 7fb61d011880 0 pidfile_write: ignore empty --pid-file 2024-02-19T20:55:43.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 0 load: jerasure load: lrc load: isa 2024-02-19T20:55:43.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T20:55:43.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T20:55:43.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Compile date Feb 17 2024 2024-02-19T20:55:43.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: DB SUMMARY 2024-02-19T20:55:43.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: CURRENT file: CURRENT 2024-02-19T20:55:43.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-19T20:55:43.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-19T20:55:43.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2024-02-19T20:55:43.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 91703 ; 2024-02-19T20:55:43.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.error_if_exists: 0 2024-02-19T20:55:43.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.create_if_missing: 0 2024-02-19T20:55:43.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T20:55:43.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.env: 0x558420240080 2024-02-19T20:55:43.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.fs: Posix File System 2024-02-19T20:55:43.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.info_log: 0x5584214d79c0 2024-02-19T20:55:43.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T20:55:43.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.statistics: (nil) 2024-02-19T20:55:43.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.use_fsync: 0 2024-02-19T20:55:43.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T20:55:43.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T20:55:43.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T20:55:43.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T20:55:43.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T20:55:43.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T20:55:43.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T20:55:43.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T20:55:43.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T20:55:43.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T20:55:43.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T20:55:43.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.db_log_dir: 2024-02-19T20:55:43.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-19T20:55:43.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T20:55:43.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T20:55:43.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T20:55:43.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T20:55:43.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T20:55:43.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T20:55:43.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T20:55:43.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T20:55:43.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T20:55:43.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T20:55:43.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.write_buffer_manager: 0x5584221d4360 2024-02-19T20:55:43.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T20:55:43.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T20:55:43.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T20:55:43.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T20:55:43.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T20:55:43.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T20:55:43.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T20:55:43.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T20:55:43.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T20:55:43.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.unordered_write: 0 2024-02-19T20:55:43.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T20:55:43.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T20:55:43.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T20:55:43.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T20:55:43.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.row_cache: None 2024-02-19T20:55:43.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.wal_filter: None 2024-02-19T20:55:43.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T20:55:43.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T20:55:43.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T20:55:43.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.two_write_queues: 0 2024-02-19T20:55:43.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T20:55:43.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.atomic_flush: 0 2024-02-19T20:55:43.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T20:55:43.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T20:55:43.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T20:55:43.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T20:55:43.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T20:55:43.133 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T20:55:43.133 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T20:55:43.133 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T20:55:43.133 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T20:55:43.133 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T20:55:43.133 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T20:55:43.133 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T20:55:43.133 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T20:55:43.133 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T20:55:43.133 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T20:55:43.133 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.max_open_files: -1 2024-02-19T20:55:43.133 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T20:55:43.134 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T20:55:43.134 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T20:55:43.134 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T20:55:43.134 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Compression algorithms supported: 2024-02-19T20:55:43.134 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T20:55:43.134 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: kZSTD supported: 0 2024-02-19T20:55:43.134 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: kXpressCompression supported: 0 2024-02-19T20:55:43.134 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T20:55:43.134 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T20:55:43.134 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T20:55:43.134 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: kZlibCompression supported: 1 2024-02-19T20:55:43.134 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T20:55:43.134 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.881+0000 7fb61d011880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T20:55:43.134 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2024-02-19T20:55:43.135 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.135 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T20:55:43.135 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.135 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T20:55:43.135 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.merge_operator: 2024-02-19T20:55:43.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compaction_filter: None 2024-02-19T20:55:43.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T20:55:43.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T20:55:43.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T20:55:43.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558421487d10) 2024-02-19T20:55:43.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cache_index_and_filter_blocks: 1 2024-02-19T20:55:43.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T20:55:43.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T20:55:43.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: pin_top_level_index_and_filter: 1 2024-02-19T20:55:43.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: index_type: 0 2024-02-19T20:55:43.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: data_block_index_type: 0 2024-02-19T20:55:43.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: index_shortening: 1 2024-02-19T20:55:43.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: data_block_hash_table_util_ratio: 0.750000 2024-02-19T20:55:43.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: hash_index_allow_collision: 1 2024-02-19T20:55:43.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: checksum: 1 2024-02-19T20:55:43.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: no_block_cache: 0 2024-02-19T20:55:43.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: block_cache: 0x5584214bcf10 2024-02-19T20:55:43.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: block_cache_name: BinnedLRUCache 2024-02-19T20:55:43.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: block_cache_options: 2024-02-19T20:55:43.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: capacity : 536870912 2024-02-19T20:55:43.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: num_shard_bits : 4 2024-02-19T20:55:43.137 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: strict_capacity_limit : 0 2024-02-19T20:55:43.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: high_pri_pool_ratio: 0.000 2024-02-19T20:55:43.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: block_cache_compressed: (nil) 2024-02-19T20:55:43.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: persistent_cache: (nil) 2024-02-19T20:55:43.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: block_size: 4096 2024-02-19T20:55:43.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: block_size_deviation: 10 2024-02-19T20:55:43.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: block_restart_interval: 16 2024-02-19T20:55:43.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: index_block_restart_interval: 1 2024-02-19T20:55:43.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: metadata_block_size: 4096 2024-02-19T20:55:43.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: partition_filters: 0 2024-02-19T20:55:43.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: use_delta_encoding: 1 2024-02-19T20:55:43.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T20:55:43.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: whole_key_filtering: 1 2024-02-19T20:55:43.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: verify_compression: 0 2024-02-19T20:55:43.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: read_amp_bytes_per_bit: 0 2024-02-19T20:55:43.138 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: format_version: 2 2024-02-19T20:55:43.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: enable_index_compression: 1 2024-02-19T20:55:43.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: block_align: 0 2024-02-19T20:55:43.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T20:55:43.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T20:55:43.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compression: NoCompression 2024-02-19T20:55:43.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T20:55:43.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T20:55:43.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T20:55:43.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.num_levels: 7 2024-02-19T20:55:43.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T20:55:43.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T20:55:43.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T20:55:43.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T20:55:43.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T20:55:43.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T20:55:43.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T20:55:43.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T20:55:43.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T20:55:43.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T20:55:43.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T20:55:43.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T20:55:43.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T20:55:43.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T20:55:43.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T20:55:43.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T20:55:43.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T20:55:43.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T20:55:43.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T20:55:43.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T20:55:43.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T20:55:43.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T20:55:43.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T20:55:43.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T20:55:43.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T20:55:43.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T20:55:43.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T20:55:43.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T20:55:43.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T20:55:43.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T20:55:43.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T20:55:43.142 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T20:55:43.142 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T20:55:43.142 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T20:55:43.142 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T20:55:43.142 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T20:55:43.142 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T20:55:43.142 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T20:55:43.142 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T20:55:43.142 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T20:55:43.142 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T20:55:43.142 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T20:55:43.142 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T20:55:43.142 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T20:55:43.142 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T20:55:43.142 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T20:55:43.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T20:55:43.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.table_properties_collectors: 2024-02-19T20:55:43.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T20:55:43.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T20:55:43.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T20:55:43.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T20:55:43.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T20:55:43.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.bloom_locality: 0 2024-02-19T20:55:43.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T20:55:43.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T20:55:43.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T20:55:43.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T20:55:43.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T20:55:43.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.ttl: 2592000 2024-02-19T20:55:43.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T20:55:43.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T20:55:43.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-19T20:55:43.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708376142883186, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-19T20:55:43.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.882+0000 7fb61d011880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-19T20:55:43.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.883+0000 7fb61d011880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-19T20:55:43.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.883+0000 7fb61d011880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708376142884044, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 87638, "table_properties": {"data_size": 85741, "index_size": 374, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 11481, "raw_average_key_size": 49, "raw_value_size": 79385, "raw_average_value_size": 339, "num_data_blocks": 12, "num_entries": 234, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708376142, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-19T20:55:43.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.883+0000 7fb61d011880 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-19T20:55:43.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.884+0000 7fb61d011880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708376142884444, "job": 1, "event": "recovery_finished"} 2024-02-19T20:55:43.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.884+0000 7fb61d011880 4 rocksdb: DB pointer 0x558421585800 2024-02-19T20:55:43.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.885+0000 7fb603e03700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T20:55:43.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.885+0000 7fb603e03700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T20:55:43.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: ** DB Stats ** 2024-02-19T20:55:43.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T20:55:43.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T20:55:43.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T20:55:43.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T20:55:43.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T20:55:43.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T20:55:43.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T20:55:43.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: ** Compaction Stats [default] ** 2024-02-19T20:55:43.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:55:43.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:55:43.146 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: L0 2/0 87.20 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.7 0.00 0.00 1 0.001 0 0 2024-02-19T20:55:43.146 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Sum 2/0 87.20 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.7 0.00 0.00 1 0.001 0 0 2024-02-19T20:55:43.146 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.7 0.00 0.00 1 0.001 0 0 2024-02-19T20:55:43.146 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.146 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: ** Compaction Stats [default] ** 2024-02-19T20:55:43.146 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:55:43.146 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:55:43.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.7 0.00 0.00 1 0.001 0 0 2024-02-19T20:55:43.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T20:55:43.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T20:55:43.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T20:55:43.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T20:55:43.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T20:55:43.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T20:55:43.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Cumulative compaction: 0.00 GB write, 29.91 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:55:43.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Interval compaction: 0.00 GB write, 29.91 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:55:43.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T20:55:43.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T20:55:43.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: ** Compaction Stats [default] ** 2024-02-19T20:55:43.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:55:43.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:55:43.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: L0 2/0 87.20 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.7 0.00 0.00 1 0.001 0 0 2024-02-19T20:55:43.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Sum 2/0 87.20 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.7 0.00 0.00 1 0.001 0 0 2024-02-19T20:55:43.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T20:55:43.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: ** Compaction Stats [default] ** 2024-02-19T20:55:43.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:55:43.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:55:43.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.7 0.00 0.00 1 0.001 0 0 2024-02-19T20:55:43.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T20:55:43.148 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T20:55:43.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T20:55:43.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T20:55:43.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T20:55:43.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T20:55:43.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Cumulative compaction: 0.00 GB write, 29.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:55:43.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:55:43.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T20:55:43.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T20:55:43.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.885+0000 7fb61d011880 0 starting mon.a rank 0 at public addrs v1:172.21.15.163:6789/0 at bind addrs v1:172.21.15.163:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:55:43.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.886+0000 7fb61d011880 1 mon.a@-1(???) e1 preinit fsid 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:55:43.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.886+0000 7fb61d011880 0 mon.a@-1(???).mds e1 new map 2024-02-19T20:55:43.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.886+0000 7fb61d011880 0 mon.a@-1(???).mds e1 print_map 2024-02-19T20:55:43.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: e1 2024-02-19T20:55:43.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T20:55:43.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T20:55:43.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: legacy client fscid: -1 2024-02-19T20:55:43.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: No filesystems configured 2024-02-19T20:55:43.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: 2024-02-19T20:55:43.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.886+0000 7fb61d011880 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-19T20:55:43.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.886+0000 7fb61d011880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T20:55:43.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.886+0000 7fb61d011880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T20:55:43.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.886+0000 7fb61d011880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T20:55:43.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:40.667778+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? v1:172.21.15.163:0/3499182339' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:55:43.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T20:55:42.886+0000 7fb61d011880 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-19T20:55:43.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 bash[38209]: f9ad2d9bdfb3412a825fa152a97c23cc892c73f8f9b03d34fbd42d0b9a0259a5 2024-02-19T20:55:43.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 systemd[1]: Started Ceph mon.a for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T20:55:43.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:55:42.902866+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-02-19T20:55:43.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:55:42.902924+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=v1:172.21.15.163:6789/0} removed_ranks: {} 2024-02-19T20:55:43.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:55:42.903306+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2024-02-19T20:55:43.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:55:42.903333+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-19T20:55:43.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:55:42.903706+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-19T20:55:43.429 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.447+0000 7f323a0d0700 1 Processor -- start 2024-02-19T20:55:43.429 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.447+0000 7f323a0d0700 1 -- start start 2024-02-19T20:55:43.429 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.447+0000 7f323a0d0700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f32341001e0 con 0x7f3234070c00 2024-02-19T20:55:43.429 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.448+0000 7f32337fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f3234070c00 0x7f32340feac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:47302/0 (socket says 172.21.15.163:47302) 2024-02-19T20:55:43.429 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.448+0000 7f32337fe700 1 -- 172.21.15.163:0/767346534 learned_addr learned my addr 172.21.15.163:0/767346534 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:43.429 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.448+0000 7f32327fc700 1 -- 172.21.15.163:0/767346534 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840771165 0 0) 0x7f32341001e0 con 0x7f3234070c00 2024-02-19T20:55:43.430 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.448+0000 7f32327fc700 1 -- 172.21.15.163:0/767346534 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f321c0033f0 con 0x7f3234070c00 2024-02-19T20:55:43.430 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.448+0000 7f32327fc700 1 -- 172.21.15.163:0/767346534 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2672223245 0 0) 0x7f321c0033f0 con 0x7f3234070c00 2024-02-19T20:55:43.432 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.448+0000 7f32327fc700 1 -- 172.21.15.163:0/767346534 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3234101380 con 0x7f3234070c00 2024-02-19T20:55:43.432 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.448+0000 7f32327fc700 1 -- 172.21.15.163:0/767346534 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f3228001590 con 0x7f3234070c00 2024-02-19T20:55:43.432 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.449+0000 7f32327fc700 1 -- 172.21.15.163:0/767346534 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f3228001d10 con 0x7f3234070c00 2024-02-19T20:55:43.432 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.449+0000 7f32327fc700 1 -- 172.21.15.163:0/767346534 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f32280058c0 con 0x7f3234070c00 2024-02-19T20:55:43.432 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.449+0000 7f323a0d0700 1 -- 172.21.15.163:0/767346534 >> v1:172.21.15.163:6789/0 conn(0x7f3234070c00 legacy=0x7f32340feac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:43.432 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.449+0000 7f323a0d0700 1 -- 172.21.15.163:0/767346534 shutdown_connections 2024-02-19T20:55:43.432 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.449+0000 7f323a0d0700 1 -- 172.21.15.163:0/767346534 >> 172.21.15.163:0/767346534 conn(0x7f32340f6500 msgr2=0x7f32340f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:43.432 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.449+0000 7f323a0d0700 1 -- 172.21.15.163:0/767346534 shutdown_connections 2024-02-19T20:55:43.433 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.449+0000 7f323a0d0700 1 -- 172.21.15.163:0/767346534 wait complete. 2024-02-19T20:55:43.433 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.450+0000 7f323a0d0700 1 librados: starting msgr at 2024-02-19T20:55:43.433 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.450+0000 7f323a0d0700 1 librados: starting objecter 2024-02-19T20:55:43.433 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.450+0000 7f323a0d0700 1 Processor -- start 2024-02-19T20:55:43.433 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.450+0000 7f323a0d0700 1 -- start start 2024-02-19T20:55:43.433 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.450+0000 7f323a0d0700 1 librados: setting wanted keys 2024-02-19T20:55:43.433 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.450+0000 7f323a0d0700 1 librados: calling monclient init 2024-02-19T20:55:43.433 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.450+0000 7f323a0d0700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f323410af00 con 0x7f3234070c00 2024-02-19T20:55:43.433 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f32337fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f3234070c00 0x7f323410a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:47314/0 (socket says 172.21.15.163:47314) 2024-02-19T20:55:43.433 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f32337fe700 1 -- 172.21.15.163:0/553758992 learned_addr learned my addr 172.21.15.163:0/553758992 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:43.433 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 10 client.?.objecter ms_handle_connect 0x7f3234070c00 2024-02-19T20:55:43.433 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T20:55:43.433 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 1 -- 172.21.15.163:0/553758992 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661261182 0 0) 0x7f323410af00 con 0x7f3234070c00 2024-02-19T20:55:43.433 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 1 -- 172.21.15.163:0/553758992 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f321c005550 con 0x7f3234070c00 2024-02-19T20:55:43.433 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 1 -- 172.21.15.163:0/553758992 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2212248946 0 0) 0x7f321c005550 con 0x7f3234070c00 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 1 -- 172.21.15.163:0/553758992 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f323410af00 con 0x7f3234070c00 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 1 -- 172.21.15.163:0/553758992 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f3228004e40 con 0x7f3234070c00 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 1 -- 172.21.15.163:0/553758992 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2338169325 0 0) 0x7f323410af00 con 0x7f3234070c00 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 1 -- 172.21.15.163:0/553758992 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323410b090 con 0x7f3234070c00 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f323a0d0700 1 -- 172.21.15.163:0/553758992 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323410b340 con 0x7f3234070c00 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f323a0d0700 10 client.14098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f323a0d0700 1 -- 172.21.15.163:0/553758992 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f323410f030 con 0x7f3234070c00 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f323a0d0700 1 librados: init done 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 1 -- 172.21.15.163:0/553758992 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f3228001000 con 0x7f3234070c00 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 1 -- 172.21.15.163:0/553758992 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f3228006170 con 0x7f3234070c00 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 1 -- 172.21.15.163:0/553758992 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 1) v1 ==== 495+0+0 (unknown 2835230805 0 0) 0x7f3228006630 con 0x7f3234070c00 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 1 -- 172.21.15.163:0/553758992 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 2758453317 0 0) 0x7f32280075d0 con 0x7f3234070c00 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 10 client.14098.objecter ms_dispatch 0x7f3234001550 osd_map(1..1 src has 1..1) v4 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 3 client.14098.objecter handle_osd_map got epochs [1,1] > 0 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 3 client.14098.objecter handle_osd_map decoding full epoch 1 2024-02-19T20:55:43.434 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.451+0000 7f3230ff9700 20 client.14098.objecter dump_active .. 0 homeless 2024-02-19T20:55:43.435 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.452+0000 7f323a0d0700 1 -- 172.21.15.163:0/553758992 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3220004f50 con 0x7f3234070c00 2024-02-19T20:55:43.435 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.454+0000 7f3230ff9700 1 -- 172.21.15.163:0/553758992 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (unknown 1092875540 0 259193764) 0x7f3228005ac0 con 0x7f3234070c00 2024-02-19T20:55:43.499 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.518+0000 7f323a0d0700 1 -- 172.21.15.163:0/553758992 --> v1:172.21.15.163:6789/0 -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f3220005780 con 0x7f3234070c00 2024-02-19T20:55:43.500 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.520+0000 7f3230ff9700 1 -- 172.21.15.163:0/553758992 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (unknown 534127980 0 0) 0x7f3228005570 con 0x7f3234070c00 2024-02-19T20:55:43.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.521+0000 7f323a0d0700 10 librados: watch_flush enter 2024-02-19T20:55:43.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.521+0000 7f323a0d0700 10 librados: watch_flush exit 2024-02-19T20:55:43.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.521+0000 7f323a0d0700 20 client.14098.objecter shutdown clearing up homeless session... 2024-02-19T20:55:43.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.521+0000 7f323a0d0700 10 client.14098.objecter successfully canceled tick 2024-02-19T20:55:43.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.521+0000 7f323a0d0700 1 -- 172.21.15.163:0/553758992 >> v1:172.21.15.163:6789/0 conn(0x7f3234070c00 legacy=0x7f323410a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:43.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.521+0000 7f323a0d0700 1 -- 172.21.15.163:0/553758992 shutdown_connections 2024-02-19T20:55:43.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.521+0000 7f323a0d0700 1 -- 172.21.15.163:0/553758992 >> 172.21.15.163:0/553758992 conn(0x7f32340f6500 msgr2=0x7f32340fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:43.504 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.521+0000 7f323a0d0700 1 -- 172.21.15.163:0/553758992 shutdown_connections 2024-02-19T20:55:43.504 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.521+0000 7f323a0d0700 1 -- 172.21.15.163:0/553758992 wait complete. 2024-02-19T20:55:43.504 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:43.521+0000 7f323a0d0700 1 librados: shutdown 2024-02-19T20:55:44.154 INFO:teuthology.orchestra.run.smithi163.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-19T20:55:44.157 INFO:teuthology.orchestra.run.smithi163.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-19T20:55:44.157 INFO:teuthology.orchestra.run.smithi163.stderr:Creating mgr... 2024-02-19T20:55:44.157 INFO:teuthology.orchestra.run.smithi163.stderr:Verifying port 9283 ... 2024-02-19T20:55:44.283 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Failed to reset failed state of unit ceph-2591809a-cf69-11ee-95bb-87774f69a715@mgr.y.service: Unit ceph-2591809a-cf69-11ee-95bb-87774f69a715@mgr.y.service not loaded. 2024-02-19T20:55:44.289 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: Created symlink /etc/systemd/system/ceph-2591809a-cf69-11ee-95bb-87774f69a715.target.wants/ceph-2591809a-cf69-11ee-95bb-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-2591809a-cf69-11ee-95bb-87774f69a715@.service. 2024-02-19T20:55:44.478 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:44 smithi163 systemd[1]: Starting Ceph mgr.y for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T20:55:44.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:43.521023+0000 mon.a (mon.0) 6 : audit [INF] from='client.? v1:172.21.15.163:0/553758992' entity='client.admin' 2024-02-19T20:55:46.612 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: disabled 2024-02-19T20:55:46.619 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: inactive 2024-02-19T20:55:46.620 INFO:teuthology.orchestra.run.smithi163.stderr:firewalld.service is not enabled 2024-02-19T20:55:46.620 INFO:teuthology.orchestra.run.smithi163.stderr:Not possible to enable service . firewalld.service is not available 2024-02-19T20:55:46.628 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: disabled 2024-02-19T20:55:46.636 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: inactive 2024-02-19T20:55:46.636 INFO:teuthology.orchestra.run.smithi163.stderr:firewalld.service is not enabled 2024-02-19T20:55:46.636 INFO:teuthology.orchestra.run.smithi163.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-19T20:55:46.636 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for mgr to start... 2024-02-19T20:55:46.637 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for mgr... 2024-02-19T20:55:46.853 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:46 smithi163 bash[38483]: c0b46b2f24d7d59e3fe34afc45ade3a96af275ec1f50d05348ce83b1ea71e530 2024-02-19T20:55:46.853 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:46 smithi163 systemd[1]: Started Ceph mgr.y for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T20:55:46.853 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:46.679+0000 7fa9e5231680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-19T20:55:46.853 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:46.762+0000 7fa9e5231680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-19T20:55:47.191 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.211+0000 7fb71ec87700 1 Processor -- start 2024-02-19T20:55:47.196 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.211+0000 7fb71ec87700 1 -- start start 2024-02-19T20:55:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.211+0000 7fb71ec87700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb718100170 con 0x7fb718070c60 2024-02-19T20:55:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.212+0000 7fb71ca23700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fb718070c60 0x7fb7180fea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:32866/0 (socket says 172.21.15.163:32866) 2024-02-19T20:55:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.212+0000 7fb71ca23700 1 -- 172.21.15.163:0/1403125819 learned_addr learned my addr 172.21.15.163:0/1403125819 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.212+0000 7fb7177fe700 1 -- 172.21.15.163:0/1403125819 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378194023 0 0) 0x7fb718100170 con 0x7fb718070c60 2024-02-19T20:55:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.212+0000 7fb7177fe700 1 -- 172.21.15.163:0/1403125819 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7080033f0 con 0x7fb718070c60 2024-02-19T20:55:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.212+0000 7fb7177fe700 1 -- 172.21.15.163:0/1403125819 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2425541470 0 0) 0x7fb7080033f0 con 0x7fb718070c60 2024-02-19T20:55:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.212+0000 7fb7177fe700 1 -- 172.21.15.163:0/1403125819 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb718101310 con 0x7fb718070c60 2024-02-19T20:55:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.212+0000 7fb7177fe700 1 -- 172.21.15.163:0/1403125819 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fb70c001590 con 0x7fb718070c60 2024-02-19T20:55:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.212+0000 7fb7177fe700 1 -- 172.21.15.163:0/1403125819 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7fb70c001d10 con 0x7fb718070c60 2024-02-19T20:55:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.212+0000 7fb7177fe700 1 -- 172.21.15.163:0/1403125819 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fb70c005750 con 0x7fb718070c60 2024-02-19T20:55:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.213+0000 7fb71ec87700 1 -- 172.21.15.163:0/1403125819 >> v1:172.21.15.163:6789/0 conn(0x7fb718070c60 legacy=0x7fb7180fea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.213+0000 7fb71ec87700 1 -- 172.21.15.163:0/1403125819 shutdown_connections 2024-02-19T20:55:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.213+0000 7fb71ec87700 1 -- 172.21.15.163:0/1403125819 >> 172.21.15.163:0/1403125819 conn(0x7fb7180f64d0 msgr2=0x7fb7180f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.213+0000 7fb71ec87700 1 -- 172.21.15.163:0/1403125819 shutdown_connections 2024-02-19T20:55:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.213+0000 7fb71ec87700 1 -- 172.21.15.163:0/1403125819 wait complete. 2024-02-19T20:55:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.214+0000 7fb71ec87700 1 librados: starting msgr at 2024-02-19T20:55:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.214+0000 7fb71ec87700 1 librados: starting objecter 2024-02-19T20:55:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.214+0000 7fb71ec87700 1 Processor -- start 2024-02-19T20:55:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.214+0000 7fb71ec87700 1 -- start start 2024-02-19T20:55:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.214+0000 7fb71ec87700 1 librados: setting wanted keys 2024-02-19T20:55:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.214+0000 7fb71ec87700 1 librados: calling monclient init 2024-02-19T20:55:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.214+0000 7fb71ec87700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb71810aee0 con 0x7fb718070c60 2024-02-19T20:55:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.214+0000 7fb71ca23700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fb718070c60 0x7fb71810a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:32876/0 (socket says 172.21.15.163:32876) 2024-02-19T20:55:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.214+0000 7fb71ca23700 1 -- 172.21.15.163:0/454375419 learned_addr learned my addr 172.21.15.163:0/454375419 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.214+0000 7fb715ffb700 10 client.?.objecter ms_handle_connect 0x7fb718070c60 2024-02-19T20:55:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.214+0000 7fb715ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T20:55:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.214+0000 7fb715ffb700 1 -- 172.21.15.163:0/454375419 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142755642 0 0) 0x7fb71810aee0 con 0x7fb718070c60 2024-02-19T20:55:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.214+0000 7fb715ffb700 1 -- 172.21.15.163:0/454375419 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb708005580 con 0x7fb718070c60 2024-02-19T20:55:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.215+0000 7fb715ffb700 1 -- 172.21.15.163:0/454375419 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 483286425 0 0) 0x7fb708005580 con 0x7fb718070c60 2024-02-19T20:55:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.215+0000 7fb715ffb700 1 -- 172.21.15.163:0/454375419 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb71810aee0 con 0x7fb718070c60 2024-02-19T20:55:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.215+0000 7fb715ffb700 1 -- 172.21.15.163:0/454375419 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fb70c000fe0 con 0x7fb718070c60 2024-02-19T20:55:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.215+0000 7fb715ffb700 1 -- 172.21.15.163:0/454375419 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 955058233 0 0) 0x7fb71810aee0 con 0x7fb718070c60 2024-02-19T20:55:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.215+0000 7fb715ffb700 1 -- 172.21.15.163:0/454375419 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71810b070 con 0x7fb718070c60 2024-02-19T20:55:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.215+0000 7fb71ec87700 1 -- 172.21.15.163:0/454375419 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71810b320 con 0x7fb718070c60 2024-02-19T20:55:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.215+0000 7fb71ec87700 10 client.14102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:55:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.215+0000 7fb71ec87700 1 -- 172.21.15.163:0/454375419 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71810f010 con 0x7fb718070c60 2024-02-19T20:55:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.215+0000 7fb71ec87700 1 librados: init done 2024-02-19T20:55:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.215+0000 7fb715ffb700 1 -- 172.21.15.163:0/454375419 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7fb70c005740 con 0x7fb718070c60 2024-02-19T20:55:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.215+0000 7fb715ffb700 1 -- 172.21.15.163:0/454375419 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fb70c006230 con 0x7fb718070c60 2024-02-19T20:55:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.216+0000 7fb715ffb700 1 -- 172.21.15.163:0/454375419 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 1) v1 ==== 495+0+0 (unknown 2835230805 0 0) 0x7fb70c006730 con 0x7fb718070c60 2024-02-19T20:55:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.216+0000 7fb715ffb700 1 -- 172.21.15.163:0/454375419 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 2758453317 0 0) 0x7fb70c0076f0 con 0x7fb718070c60 2024-02-19T20:55:47.200 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.216+0000 7fb715ffb700 10 client.14102.objecter ms_dispatch 0x7fb718001550 osd_map(1..1 src has 1..1) v4 2024-02-19T20:55:47.200 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.216+0000 7fb715ffb700 3 client.14102.objecter handle_osd_map got epochs [1,1] > 0 2024-02-19T20:55:47.200 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.216+0000 7fb715ffb700 3 client.14102.objecter handle_osd_map decoding full epoch 1 2024-02-19T20:55:47.200 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.216+0000 7fb715ffb700 20 client.14102.objecter dump_active .. 0 homeless 2024-02-19T20:55:47.200 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.216+0000 7fb71ec87700 1 -- 172.21.15.163:0/454375419 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6fc004f50 con 0x7fb718070c60 2024-02-19T20:55:47.200 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.219+0000 7fb715ffb700 1 -- 172.21.15.163:0/454375419 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (unknown 1092875540 0 259193764) 0x7fb70c005a30 con 0x7fb718070c60 2024-02-19T20:55:47.269 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.288+0000 7fb71ec87700 1 -- 172.21.15.163:0/454375419 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb6fc005200 con 0x7fb718070c60 2024-02-19T20:55:47.269 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.288+0000 7fb715ffb700 1 -- 172.21.15.163:0/454375419 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1241 (unknown 4201413639 0 4040938070) 0x7fb70c005c70 con 0x7fb718070c60 2024-02-19T20:55:47.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: { 2024-02-19T20:55:47.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "fsid": "2591809a-cf69-11ee-95bb-87774f69a715", 2024-02-19T20:55:47.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "health": { 2024-02-19T20:55:47.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T20:55:47.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T20:55:47.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T20:55:47.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:47.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T20:55:47.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T20:55:47.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 0 2024-02-19T20:55:47.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2024-02-19T20:55:47.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T20:55:47.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "a" 2024-02-19T20:55:47.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2024-02-19T20:55:47.271 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-02-19T20:55:47.271 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T20:55:47.271 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T20:55:47.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T20:55:47.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T20:55:47.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:47.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T20:55:47.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T20:55:47.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T20:55:47.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T20:55:47.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T20:55:47.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T20:55:47.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T20:55:47.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T20:55:47.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:47.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T20:55:47.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T20:55:47.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T20:55:47.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T20:55:47.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T20:55:47.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T20:55:47.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T20:55:47.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T20:55:47.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T20:55:47.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:47.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T20:55:47.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T20:55:47.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T20:55:47.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T20:55:47.274 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "available": false, 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "iostat", 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "nfs", 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "restful" 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "services": {} 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "modified": "2024-02-19T20:55:37.795499+0000", 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "services": {} 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:47.275 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T20:55:47.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: } 2024-02-19T20:55:47.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.289+0000 7fb71ec87700 10 librados: watch_flush enter 2024-02-19T20:55:47.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.289+0000 7fb71ec87700 10 librados: watch_flush exit 2024-02-19T20:55:47.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.289+0000 7fb71ec87700 20 client.14102.objecter shutdown clearing up homeless session... 2024-02-19T20:55:47.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.289+0000 7fb71ec87700 10 client.14102.objecter successfully canceled tick 2024-02-19T20:55:47.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.290+0000 7fb71ec87700 1 -- 172.21.15.163:0/454375419 >> v1:172.21.15.163:6789/0 conn(0x7fb718070c60 legacy=0x7fb71810a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:47.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.290+0000 7fb71ec87700 1 -- 172.21.15.163:0/454375419 shutdown_connections 2024-02-19T20:55:47.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.290+0000 7fb71ec87700 1 -- 172.21.15.163:0/454375419 >> 172.21.15.163:0/454375419 conn(0x7fb7180f64d0 msgr2=0x7fb7180fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:47.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.290+0000 7fb71ec87700 1 -- 172.21.15.163:0/454375419 shutdown_connections 2024-02-19T20:55:47.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.290+0000 7fb71ec87700 1 -- 172.21.15.163:0/454375419 wait complete. 2024-02-19T20:55:47.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:47.290+0000 7fb71ec87700 1 librados: shutdown 2024-02-19T20:55:47.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:47.288515+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? v1:172.21.15.163:0/454375419' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T20:55:47.478 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:47.143+0000 7fa9e5231680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-19T20:55:47.866 INFO:teuthology.orchestra.run.smithi163.stderr:mgr not available, waiting (1/15)... 2024-02-19T20:55:48.187 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:47.915+0000 7fa9e5231680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-19T20:55:48.187 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:47.998+0000 7fa9e5231680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-19T20:55:48.477 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:48.207+0000 7fa9e5231680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-19T20:55:49.416 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:49.145+0000 7fa9e5231680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-19T20:55:49.416 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:49.270+0000 7fa9e5231680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-19T20:55:49.416 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:49.347+0000 7fa9e5231680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-19T20:55:49.727 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:49.436+0000 7fa9e5231680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-19T20:55:49.728 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:49.631+0000 7fa9e5231680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-19T20:55:50.149 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:49.909+0000 7fa9e5231680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-19T20:55:50.149 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:50.008+0000 7fa9e5231680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-19T20:55:50.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.454+0000 7fa4ad4ff700 1 Processor -- start 2024-02-19T20:55:50.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.454+0000 7fa4ad4ff700 1 -- start start 2024-02-19T20:55:50.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.454+0000 7fa4ad4ff700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa4a80fc9a0 con 0x7fa4a80fbe70 2024-02-19T20:55:50.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.454+0000 7fa4a6ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fa4a80fbe70 0x7fa4a80fc290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:32886/0 (socket says 172.21.15.163:32886) 2024-02-19T20:55:50.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.454+0000 7fa4a6ffd700 1 -- 172.21.15.163:0/3969841280 learned_addr learned my addr 172.21.15.163:0/3969841280 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:50.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.454+0000 7fa4a5ffb700 1 -- 172.21.15.163:0/3969841280 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428026299 0 0) 0x7fa498001110 con 0x7fa4a80fbe70 2024-02-19T20:55:50.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.454+0000 7fa4a5ffb700 1 -- 172.21.15.163:0/3969841280 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4900033f0 con 0x7fa4a80fbe70 2024-02-19T20:55:50.454 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.455+0000 7fa4a5ffb700 1 -- 172.21.15.163:0/3969841280 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1983708502 0 0) 0x7fa4900033f0 con 0x7fa4a80fbe70 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.455+0000 7fa4a5ffb700 1 -- 172.21.15.163:0/3969841280 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a81012b0 con 0x7fa4a80fbe70 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.455+0000 7fa4a5ffb700 1 -- 172.21.15.163:0/3969841280 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fa498001720 con 0x7fa4a80fbe70 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.455+0000 7fa4a5ffb700 1 -- 172.21.15.163:0/3969841280 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7fa498004770 con 0x7fa4a80fbe70 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.455+0000 7fa4a5ffb700 1 -- 172.21.15.163:0/3969841280 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fa4980058f0 con 0x7fa4a80fbe70 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.455+0000 7fa4ad4ff700 1 -- 172.21.15.163:0/3969841280 >> v1:172.21.15.163:6789/0 conn(0x7fa4a80fbe70 legacy=0x7fa4a80fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.455+0000 7fa4ad4ff700 1 -- 172.21.15.163:0/3969841280 shutdown_connections 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.455+0000 7fa4ad4ff700 1 -- 172.21.15.163:0/3969841280 >> 172.21.15.163:0/3969841280 conn(0x7fa4a80f7610 msgr2=0x7fa4a80f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.455+0000 7fa4ad4ff700 1 -- 172.21.15.163:0/3969841280 shutdown_connections 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.455+0000 7fa4ad4ff700 1 -- 172.21.15.163:0/3969841280 wait complete. 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.456+0000 7fa4ad4ff700 1 librados: starting msgr at 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.456+0000 7fa4ad4ff700 1 librados: starting objecter 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.456+0000 7fa4ad4ff700 1 Processor -- start 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.456+0000 7fa4ad4ff700 1 -- start start 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.456+0000 7fa4ad4ff700 1 librados: setting wanted keys 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.456+0000 7fa4ad4ff700 1 librados: calling monclient init 2024-02-19T20:55:50.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa4ad4ff700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa4a810b040 con 0x7fa4a80fbe70 2024-02-19T20:55:50.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa4a6ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fa4a80fbe70 0x7fa4a810a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:32902/0 (socket says 172.21.15.163:32902) 2024-02-19T20:55:50.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa4a6ffd700 1 -- 172.21.15.163:0/3038950994 learned_addr learned my addr 172.21.15.163:0/3038950994 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:50.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa48ffff700 10 client.?.objecter ms_handle_connect 0x7fa4a80fbe70 2024-02-19T20:55:50.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa48ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T20:55:50.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa48ffff700 1 -- 172.21.15.163:0/3038950994 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656060528 0 0) 0x7fa4a810b040 con 0x7fa4a80fbe70 2024-02-19T20:55:50.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa48ffff700 1 -- 172.21.15.163:0/3038950994 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa490005580 con 0x7fa4a80fbe70 2024-02-19T20:55:50.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa48ffff700 1 -- 172.21.15.163:0/3038950994 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 164808302 0 0) 0x7fa490005580 con 0x7fa4a80fbe70 2024-02-19T20:55:50.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa48ffff700 1 -- 172.21.15.163:0/3038950994 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4a810b040 con 0x7fa4a80fbe70 2024-02-19T20:55:50.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa48ffff700 1 -- 172.21.15.163:0/3038950994 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fa498004d00 con 0x7fa4a80fbe70 2024-02-19T20:55:50.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa48ffff700 1 -- 172.21.15.163:0/3038950994 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2471493486 0 0) 0x7fa4a810b040 con 0x7fa4a80fbe70 2024-02-19T20:55:50.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa48ffff700 1 -- 172.21.15.163:0/3038950994 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a810b1d0 con 0x7fa4a80fbe70 2024-02-19T20:55:50.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa4ad4ff700 1 -- 172.21.15.163:0/3038950994 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a810b480 con 0x7fa4a80fbe70 2024-02-19T20:55:50.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa4ad4ff700 10 client.14104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:55:50.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa4ad4ff700 1 -- 172.21.15.163:0/3038950994 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a810f170 con 0x7fa4a80fbe70 2024-02-19T20:55:50.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa4ad4ff700 1 librados: init done 2024-02-19T20:55:50.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa48ffff700 1 -- 172.21.15.163:0/3038950994 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7fa498004a70 con 0x7fa4a80fbe70 2024-02-19T20:55:50.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa48ffff700 1 -- 172.21.15.163:0/3038950994 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fa498006240 con 0x7fa4a80fbe70 2024-02-19T20:55:50.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.457+0000 7fa48ffff700 1 -- 172.21.15.163:0/3038950994 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 1) v1 ==== 495+0+0 (unknown 2835230805 0 0) 0x7fa4980066c0 con 0x7fa4a80fbe70 2024-02-19T20:55:50.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.458+0000 7fa48ffff700 1 -- 172.21.15.163:0/3038950994 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 2758453317 0 0) 0x7fa498007630 con 0x7fa4a80fbe70 2024-02-19T20:55:50.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.458+0000 7fa48ffff700 10 client.14104.objecter ms_dispatch 0x7fa4a8001550 osd_map(1..1 src has 1..1) v4 2024-02-19T20:55:50.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.458+0000 7fa48ffff700 3 client.14104.objecter handle_osd_map got epochs [1,1] > 0 2024-02-19T20:55:50.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.458+0000 7fa48ffff700 3 client.14104.objecter handle_osd_map decoding full epoch 1 2024-02-19T20:55:50.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.458+0000 7fa48ffff700 20 client.14104.objecter dump_active .. 0 homeless 2024-02-19T20:55:50.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.458+0000 7fa4ad4ff700 1 -- 172.21.15.163:0/3038950994 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa494004f50 con 0x7fa4a80fbe70 2024-02-19T20:55:50.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.460+0000 7fa48ffff700 1 -- 172.21.15.163:0/3038950994 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (unknown 1092875540 0 259193764) 0x7fa4a80fc9a0 con 0x7fa4a80fbe70 2024-02-19T20:55:50.516 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.529+0000 7fa4ad4ff700 1 -- 172.21.15.163:0/3038950994 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa494005200 con 0x7fa4a80fbe70 2024-02-19T20:55:50.516 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.529+0000 7fa48ffff700 1 -- 172.21.15.163:0/3038950994 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1241 (unknown 4201413639 0 655552215) 0x7fa498005bd0 con 0x7fa4a80fbe70 2024-02-19T20:55:50.517 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.517 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: { 2024-02-19T20:55:50.517 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "fsid": "2591809a-cf69-11ee-95bb-87774f69a715", 2024-02-19T20:55:50.517 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "health": { 2024-02-19T20:55:50.517 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T20:55:50.517 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T20:55:50.517 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T20:55:50.517 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:50.517 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T20:55:50.517 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T20:55:50.517 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 0 2024-02-19T20:55:50.517 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2024-02-19T20:55:50.517 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T20:55:50.517 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "a" 2024-02-19T20:55:50.517 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2024-02-19T20:55:50.518 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum_age": 7, 2024-02-19T20:55:50.518 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T20:55:50.518 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T20:55:50.518 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T20:55:50.518 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T20:55:50.518 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:50.518 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T20:55:50.518 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T20:55:50.518 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T20:55:50.518 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T20:55:50.518 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T20:55:50.518 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T20:55:50.518 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T20:55:50.518 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T20:55:50.518 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:50.519 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T20:55:50.519 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T20:55:50.519 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T20:55:50.519 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T20:55:50.519 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T20:55:50.519 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T20:55:50.519 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T20:55:50.519 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T20:55:50.519 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T20:55:50.519 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:50.519 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T20:55:50.519 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T20:55:50.519 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T20:55:50.519 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T20:55:50.520 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:50.520 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T20:55:50.520 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "available": false, 2024-02-19T20:55:50.520 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T20:55:50.520 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T20:55:50.520 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "iostat", 2024-02-19T20:55:50.520 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "nfs", 2024-02-19T20:55:50.520 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "restful" 2024-02-19T20:55:50.520 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2024-02-19T20:55:50.520 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "services": {} 2024-02-19T20:55:50.520 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:50.520 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T20:55:50.520 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T20:55:50.520 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "modified": "2024-02-19T20:55:37.795499+0000", 2024-02-19T20:55:50.521 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "services": {} 2024-02-19T20:55:50.521 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:50.521 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T20:55:50.521 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: } 2024-02-19T20:55:50.521 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.531+0000 7fa4ad4ff700 10 librados: watch_flush enter 2024-02-19T20:55:50.521 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.531+0000 7fa4ad4ff700 10 librados: watch_flush exit 2024-02-19T20:55:50.521 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.531+0000 7fa4ad4ff700 20 client.14104.objecter shutdown clearing up homeless session... 2024-02-19T20:55:50.521 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.531+0000 7fa4ad4ff700 10 client.14104.objecter successfully canceled tick 2024-02-19T20:55:50.521 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.531+0000 7fa4ad4ff700 1 -- 172.21.15.163:0/3038950994 >> v1:172.21.15.163:6789/0 conn(0x7fa4a80fbe70 legacy=0x7fa4a810a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:50.521 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.531+0000 7fa4ad4ff700 1 -- 172.21.15.163:0/3038950994 shutdown_connections 2024-02-19T20:55:50.521 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.531+0000 7fa4ad4ff700 1 -- 172.21.15.163:0/3038950994 >> 172.21.15.163:0/3038950994 conn(0x7fa4a80f7610 msgr2=0x7fa4a80f8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:50.521 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.531+0000 7fa4ad4ff700 1 -- 172.21.15.163:0/3038950994 shutdown_connections 2024-02-19T20:55:50.521 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.531+0000 7fa4ad4ff700 1 -- 172.21.15.163:0/3038950994 wait complete. 2024-02-19T20:55:50.521 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:50.531+0000 7fa4ad4ff700 1 librados: shutdown 2024-02-19T20:55:50.830 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:50.530785+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? v1:172.21.15.163:0/3038950994' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T20:55:50.831 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:50.766+0000 7fa9e5231680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-19T20:55:51.096 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:50.850+0000 7fa9e5231680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-19T20:55:51.096 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:50.942+0000 7fa9e5231680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-19T20:55:51.146 INFO:teuthology.orchestra.run.smithi163.stderr:mgr not available, waiting (2/15)... 2024-02-19T20:55:51.358 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:51.116+0000 7fa9e5231680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-19T20:55:51.620 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:51.378+0000 7fa9e5231680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-19T20:55:51.620 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:51.507+0000 7fa9e5231680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-19T20:55:51.871 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:51.639+0000 7fa9e5231680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-19T20:55:51.871 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:51.809+0000 7fa9e5231680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-19T20:55:52.228 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:51.891+0000 7fa9e5231680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-19T20:55:52.228 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:55:51.905197+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2024-02-19T20:55:52.228 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:55:51.907992+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00287039s) 2024-02-19T20:55:52.228 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:51.909049+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 v1:172.21.15.163:0/1137783322' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T20:55:52.228 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:51.909257+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 v1:172.21.15.163:0/1137783322' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T20:55:52.228 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:51.909432+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 v1:172.21.15.163:0/1137783322' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T20:55:52.228 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:51.909581+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 v1:172.21.15.163:0/1137783322' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T20:55:52.228 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:51.909750+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 v1:172.21.15.163:0/1137783322' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T20:55:52.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:51.909913+0000 mon.a (mon.0) 16 : audit [DBG] from='mgr.14100 v1:172.21.15.163:0/1137783322' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T20:55:52.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:51.910082+0000 mon.a (mon.0) 17 : audit [DBG] from='mgr.14100 v1:172.21.15.163:0/1137783322' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-19T20:55:52.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:51.910266+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.14100 v1:172.21.15.163:0/1137783322' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-19T20:55:52.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:55:51.922893+0000 mon.a (mon.0) 19 : cluster [INF] Manager daemon y is now available 2024-02-19T20:55:52.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:51.932162+0000 mon.a (mon.0) 20 : audit [INF] from='mgr.14100 v1:172.21.15.163:0/1137783322' entity='mgr.y' 2024-02-19T20:55:52.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:51.936430+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 v1:172.21.15.163:0/1137783322' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-19T20:55:52.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:51.939056+0000 mon.a (mon.0) 22 : audit [INF] from='mgr.14100 v1:172.21.15.163:0/1137783322' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-19T20:55:52.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:51.941686+0000 mon.a (mon.0) 23 : audit [INF] from='mgr.14100 v1:172.21.15.163:0/1137783322' entity='mgr.y' 2024-02-19T20:55:52.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:51.943534+0000 mon.a (mon.0) 24 : audit [INF] from='mgr.14100 v1:172.21.15.163:0/1137783322' entity='mgr.y' 2024-02-19T20:55:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.676+0000 7f0e44a06700 1 Processor -- start 2024-02-19T20:55:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.676+0000 7f0e44a06700 1 -- start start 2024-02-19T20:55:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.676+0000 7f0e44a06700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0e400fc8c0 con 0x7f0e40070be0 2024-02-19T20:55:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.676+0000 7f0e3e59c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f0e40070be0 0x7f0e40100ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:32946/0 (socket says 172.21.15.163:32946) 2024-02-19T20:55:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.677+0000 7f0e3e59c700 1 -- 172.21.15.163:0/716909607 learned_addr learned my addr 172.21.15.163:0/716909607 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.677+0000 7f0e3d59a700 1 -- 172.21.15.163:0/716909607 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3182025682 0 0) 0x7f0e400fc8c0 con 0x7f0e40070be0 2024-02-19T20:55:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.677+0000 7f0e3d59a700 1 -- 172.21.15.163:0/716909607 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e280033f0 con 0x7f0e40070be0 2024-02-19T20:55:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.677+0000 7f0e3d59a700 1 -- 172.21.15.163:0/716909607 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 713741775 0 0) 0x7f0e280033f0 con 0x7f0e40070be0 2024-02-19T20:55:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.677+0000 7f0e3d59a700 1 -- 172.21.15.163:0/716909607 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e40103600 con 0x7f0e40070be0 2024-02-19T20:55:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.677+0000 7f0e3d59a700 1 -- 172.21.15.163:0/716909607 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f0e34001590 con 0x7f0e40070be0 2024-02-19T20:55:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.677+0000 7f0e3d59a700 1 -- 172.21.15.163:0/716909607 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f0e34001c40 con 0x7f0e40070be0 2024-02-19T20:55:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.677+0000 7f0e3d59a700 1 -- 172.21.15.163:0/716909607 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f0e34005610 con 0x7f0e40070be0 2024-02-19T20:55:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.678+0000 7f0e44a06700 1 -- 172.21.15.163:0/716909607 >> v1:172.21.15.163:6789/0 conn(0x7f0e40070be0 legacy=0x7f0e40100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.678+0000 7f0e44a06700 1 -- 172.21.15.163:0/716909607 shutdown_connections 2024-02-19T20:55:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.678+0000 7f0e44a06700 1 -- 172.21.15.163:0/716909607 >> 172.21.15.163:0/716909607 conn(0x7f0e400f64d0 msgr2=0x7f0e400f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.678+0000 7f0e44a06700 1 -- 172.21.15.163:0/716909607 shutdown_connections 2024-02-19T20:55:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.678+0000 7f0e44a06700 1 -- 172.21.15.163:0/716909607 wait complete. 2024-02-19T20:55:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.678+0000 7f0e44a06700 1 librados: starting msgr at 2024-02-19T20:55:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.678+0000 7f0e44a06700 1 librados: starting objecter 2024-02-19T20:55:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.679+0000 7f0e44a06700 1 Processor -- start 2024-02-19T20:55:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.679+0000 7f0e44a06700 1 -- start start 2024-02-19T20:55:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.679+0000 7f0e44a06700 1 librados: setting wanted keys 2024-02-19T20:55:53.662 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.679+0000 7f0e44a06700 1 librados: calling monclient init 2024-02-19T20:55:53.662 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.679+0000 7f0e44a06700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0e4010d130 con 0x7f0e40070be0 2024-02-19T20:55:53.662 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.679+0000 7f0e3e59c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f0e40070be0 0x7f0e4010ca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:32952/0 (socket says 172.21.15.163:32952) 2024-02-19T20:55:53.662 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.679+0000 7f0e3e59c700 1 -- 172.21.15.163:0/2251245165 learned_addr learned my addr 172.21.15.163:0/2251245165 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:53.662 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.679+0000 7f0e2f7fe700 10 client.?.objecter ms_handle_connect 0x7f0e40070be0 2024-02-19T20:55:53.662 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.679+0000 7f0e2f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:55:53.662 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.679+0000 7f0e2f7fe700 1 -- 172.21.15.163:0/2251245165 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939108982 0 0) 0x7f0e4010d130 con 0x7f0e40070be0 2024-02-19T20:55:53.662 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.679+0000 7f0e2f7fe700 1 -- 172.21.15.163:0/2251245165 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e280055d0 con 0x7f0e40070be0 2024-02-19T20:55:53.662 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.679+0000 7f0e2f7fe700 1 -- 172.21.15.163:0/2251245165 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2665809069 0 0) 0x7f0e280055d0 con 0x7f0e40070be0 2024-02-19T20:55:53.663 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.679+0000 7f0e2f7fe700 1 -- 172.21.15.163:0/2251245165 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e4010d130 con 0x7f0e40070be0 2024-02-19T20:55:53.663 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.679+0000 7f0e2f7fe700 1 -- 172.21.15.163:0/2251245165 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f0e34000ff0 con 0x7f0e40070be0 2024-02-19T20:55:53.663 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.680+0000 7f0e2f7fe700 1 -- 172.21.15.163:0/2251245165 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2215993160 0 0) 0x7f0e4010d130 con 0x7f0e40070be0 2024-02-19T20:55:53.663 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.680+0000 7f0e2f7fe700 1 -- 172.21.15.163:0/2251245165 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4010d2c0 con 0x7f0e40070be0 2024-02-19T20:55:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.680+0000 7f0e44a06700 1 -- 172.21.15.163:0/2251245165 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4010d570 con 0x7f0e40070be0 2024-02-19T20:55:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.680+0000 7f0e44a06700 10 client.14110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:55:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.680+0000 7f0e44a06700 1 -- 172.21.15.163:0/2251245165 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e40111260 con 0x7f0e40070be0 2024-02-19T20:55:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.680+0000 7f0e44a06700 1 librados: init done 2024-02-19T20:55:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.680+0000 7f0e2f7fe700 1 -- 172.21.15.163:0/2251245165 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f0e34001840 con 0x7f0e40070be0 2024-02-19T20:55:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.680+0000 7f0e2f7fe700 1 -- 172.21.15.163:0/2251245165 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f0e34006110 con 0x7f0e40070be0 2024-02-19T20:55:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.680+0000 7f0e2f7fe700 1 -- 172.21.15.163:0/2251245165 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 3) v1 ==== 38465+0+0 (unknown 3642819058 0 0) 0x7f0e3400f9a0 con 0x7f0e40070be0 2024-02-19T20:55:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.681+0000 7f0e2f7fe700 1 -- 172.21.15.163:0/2251245165 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 2758453317 0 0) 0x7f0e3403f940 con 0x7f0e40070be0 2024-02-19T20:55:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.681+0000 7f0e2f7fe700 10 client.14110.objecter ms_dispatch 0x7f0e40001550 osd_map(1..1 src has 1..1) v4 2024-02-19T20:55:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.681+0000 7f0e2f7fe700 3 client.14110.objecter handle_osd_map got epochs [1,1] > 0 2024-02-19T20:55:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.681+0000 7f0e2f7fe700 3 client.14110.objecter handle_osd_map decoding full epoch 1 2024-02-19T20:55:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.681+0000 7f0e2f7fe700 20 client.14110.objecter dump_active .. 0 homeless 2024-02-19T20:55:53.664 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.681+0000 7f0e2f7fe700 10 client.14110.objecter ms_handle_connect 0x7f0e28035070 2024-02-19T20:55:53.665 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.681+0000 7f0e44a06700 1 -- 172.21.15.163:0/2251245165 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e20004f50 con 0x7f0e40070be0 2024-02-19T20:55:53.666 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.686+0000 7f0e2f7fe700 1 -- 172.21.15.163:0/2251245165 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0e34011080 con 0x7f0e40070be0 2024-02-19T20:55:53.888 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.908+0000 7f0e44a06700 1 -- 172.21.15.163:0/2251245165 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0e20005200 con 0x7f0e40070be0 2024-02-19T20:55:53.891 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.908+0000 7f0e2f7fe700 1 -- 172.21.15.163:0/2251245165 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1241 (unknown 4201413639 0 1266553094) 0x7f0e34014850 con 0x7f0e40070be0 2024-02-19T20:55:53.891 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.891 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: { 2024-02-19T20:55:53.891 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "fsid": "2591809a-cf69-11ee-95bb-87774f69a715", 2024-02-19T20:55:53.891 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "health": { 2024-02-19T20:55:53.891 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T20:55:53.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T20:55:53.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T20:55:53.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:53.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T20:55:53.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T20:55:53.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 0 2024-02-19T20:55:53.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2024-02-19T20:55:53.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T20:55:53.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "a" 2024-02-19T20:55:53.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2024-02-19T20:55:53.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-02-19T20:55:53.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T20:55:53.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T20:55:53.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T20:55:53.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T20:55:53.893 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:53.893 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T20:55:53.893 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T20:55:53.893 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T20:55:53.897 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:53.898 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "available": true, 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "iostat", 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "nfs", 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "restful" 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ], 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "services": {} 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T20:55:53.899 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T20:55:53.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "modified": "2024-02-19T20:55:37.795499+0000", 2024-02-19T20:55:53.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "services": {} 2024-02-19T20:55:53.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: }, 2024-02-19T20:55:53.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T20:55:53.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: } 2024-02-19T20:55:53.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.910+0000 7f0e44a06700 10 librados: watch_flush enter 2024-02-19T20:55:53.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.910+0000 7f0e44a06700 10 librados: watch_flush exit 2024-02-19T20:55:53.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.910+0000 7f0e44a06700 20 client.14110.objecter shutdown clearing up homeless session... 2024-02-19T20:55:53.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.910+0000 7f0e44a06700 10 client.14110.objecter successfully canceled tick 2024-02-19T20:55:53.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.910+0000 7f0e44a06700 1 -- 172.21.15.163:0/2251245165 >> v1:172.21.15.163:6800/2248999541 conn(0x7f0e28035070 legacy=0x7f0e28037530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:53.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.910+0000 7f0e44a06700 1 -- 172.21.15.163:0/2251245165 >> v1:172.21.15.163:6789/0 conn(0x7f0e40070be0 legacy=0x7f0e4010ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:53.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.910+0000 7f0e44a06700 1 -- 172.21.15.163:0/2251245165 shutdown_connections 2024-02-19T20:55:53.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.910+0000 7f0e44a06700 1 -- 172.21.15.163:0/2251245165 >> 172.21.15.163:0/2251245165 conn(0x7f0e400f64d0 msgr2=0x7f0e400f7120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:53.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.910+0000 7f0e44a06700 1 -- 172.21.15.163:0/2251245165 shutdown_connections 2024-02-19T20:55:53.901 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.911+0000 7f0e44a06700 1 -- 172.21.15.163:0/2251245165 wait complete. 2024-02-19T20:55:53.901 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:53.911+0000 7f0e44a06700 1 librados: shutdown 2024-02-19T20:55:54.227 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:55:52.911312+0000 mon.a (mon.0) 25 : cluster [DBG] mgrmap e3: y(active, since 1.00619s) 2024-02-19T20:55:54.477 INFO:teuthology.orchestra.run.smithi163.stderr:mgr is available 2024-02-19T20:55:54.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:53.910057+0000 mon.a (mon.0) 26 : audit [DBG] from='client.? v1:172.21.15.163:0/2251245165' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T20:55:55.003 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.021+0000 7efeb8194700 1 Processor -- start 2024-02-19T20:55:55.003 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.021+0000 7efeb8194700 1 -- start start 2024-02-19T20:55:55.004 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.022+0000 7efeb8194700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efeb00fce10 con 0x7efeb00fc2e0 2024-02-19T20:55:55.004 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.022+0000 7efeb5f30700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7efeb00fc2e0 0x7efeb00fc700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:35722/0 (socket says 172.21.15.163:35722) 2024-02-19T20:55:55.004 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.022+0000 7efeb5f30700 1 -- 172.21.15.163:0/1339643584 learned_addr learned my addr 172.21.15.163:0/1339643584 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:55.004 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.022+0000 7efeb4f2e700 1 -- 172.21.15.163:0/1339643584 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216744422 0 0) 0x7efeb00fce10 con 0x7efeb00fc2e0 2024-02-19T20:55:55.004 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.022+0000 7efeb4f2e700 1 -- 172.21.15.163:0/1339643584 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9c0033f0 con 0x7efeb00fc2e0 2024-02-19T20:55:55.004 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.022+0000 7efeb4f2e700 1 -- 172.21.15.163:0/1339643584 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 977079538 0 0) 0x7efe9c0033f0 con 0x7efeb00fc2e0 2024-02-19T20:55:55.004 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.022+0000 7efeb4f2e700 1 -- 172.21.15.163:0/1339643584 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb0100320 con 0x7efeb00fc2e0 2024-02-19T20:55:55.004 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.022+0000 7efeb4f2e700 1 -- 172.21.15.163:0/1339643584 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7efeac001590 con 0x7efeb00fc2e0 2024-02-19T20:55:55.004 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.022+0000 7efeb4f2e700 1 -- 172.21.15.163:0/1339643584 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7efeac001d10 con 0x7efeb00fc2e0 2024-02-19T20:55:55.004 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.022+0000 7efeb4f2e700 1 -- 172.21.15.163:0/1339643584 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7efeac005750 con 0x7efeb00fc2e0 2024-02-19T20:55:55.004 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.023+0000 7efeb8194700 1 -- 172.21.15.163:0/1339643584 >> v1:172.21.15.163:6789/0 conn(0x7efeb00fc2e0 legacy=0x7efeb00fc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:55.004 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.023+0000 7efeb8194700 1 -- 172.21.15.163:0/1339643584 shutdown_connections 2024-02-19T20:55:55.005 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.023+0000 7efeb8194700 1 -- 172.21.15.163:0/1339643584 >> 172.21.15.163:0/1339643584 conn(0x7efeb00f5c40 msgr2=0x7efeb00f8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:55.005 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.023+0000 7efeb8194700 1 -- 172.21.15.163:0/1339643584 shutdown_connections 2024-02-19T20:55:55.005 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.023+0000 7efeb8194700 1 -- 172.21.15.163:0/1339643584 wait complete. 2024-02-19T20:55:55.008 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.024+0000 7efeb8194700 1 librados: starting msgr at 2024-02-19T20:55:55.008 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.024+0000 7efeb8194700 1 librados: starting objecter 2024-02-19T20:55:55.008 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.024+0000 7efeb8194700 1 Processor -- start 2024-02-19T20:55:55.008 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.024+0000 7efeb8194700 1 -- start start 2024-02-19T20:55:55.008 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.024+0000 7efeb8194700 1 librados: setting wanted keys 2024-02-19T20:55:55.008 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.024+0000 7efeb8194700 1 librados: calling monclient init 2024-02-19T20:55:55.008 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.024+0000 7efeb8194700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efeb0109dc0 con 0x7efeb00fc2e0 2024-02-19T20:55:55.008 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.024+0000 7efeb5f30700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7efeb00fc2e0 0x7efeb01096b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:35724/0 (socket says 172.21.15.163:35724) 2024-02-19T20:55:55.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.024+0000 7efeb5f30700 1 -- 172.21.15.163:0/818614609 learned_addr learned my addr 172.21.15.163:0/818614609 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:55.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.024+0000 7efea6ffd700 10 client.?.objecter ms_handle_connect 0x7efeb00fc2e0 2024-02-19T20:55:55.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.024+0000 7efea6ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T20:55:55.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.024+0000 7efea6ffd700 1 -- 172.21.15.163:0/818614609 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827773971 0 0) 0x7efeb0109dc0 con 0x7efeb00fc2e0 2024-02-19T20:55:55.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.024+0000 7efea6ffd700 1 -- 172.21.15.163:0/818614609 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9c005580 con 0x7efeb00fc2e0 2024-02-19T20:55:55.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.025+0000 7efea6ffd700 1 -- 172.21.15.163:0/818614609 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3678162864 0 0) 0x7efe9c005580 con 0x7efeb00fc2e0 2024-02-19T20:55:55.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.025+0000 7efea6ffd700 1 -- 172.21.15.163:0/818614609 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efeb0109dc0 con 0x7efeb00fc2e0 2024-02-19T20:55:55.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.025+0000 7efea6ffd700 1 -- 172.21.15.163:0/818614609 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7efeac000ff0 con 0x7efeb00fc2e0 2024-02-19T20:55:55.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.025+0000 7efea6ffd700 1 -- 172.21.15.163:0/818614609 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 816299730 0 0) 0x7efeb0109dc0 con 0x7efeb00fc2e0 2024-02-19T20:55:55.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.025+0000 7efea6ffd700 1 -- 172.21.15.163:0/818614609 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb0109f50 con 0x7efeb00fc2e0 2024-02-19T20:55:55.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.025+0000 7efeb8194700 1 -- 172.21.15.163:0/818614609 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb010a200 con 0x7efeb00fc2e0 2024-02-19T20:55:55.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.025+0000 7efeb8194700 10 client.14112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:55:55.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.025+0000 7efeb8194700 1 -- 172.21.15.163:0/818614609 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb010def0 con 0x7efeb00fc2e0 2024-02-19T20:55:55.009 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.025+0000 7efeb8194700 1 librados: init done 2024-02-19T20:55:55.010 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.025+0000 7efea6ffd700 1 -- 172.21.15.163:0/818614609 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7efeac0046b0 con 0x7efeb00fc2e0 2024-02-19T20:55:55.010 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.025+0000 7efea6ffd700 1 -- 172.21.15.163:0/818614609 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7efeac0061a0 con 0x7efeb00fc2e0 2024-02-19T20:55:55.010 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.026+0000 7efea6ffd700 1 -- 172.21.15.163:0/818614609 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 38571+0+0 (unknown 3892717071 0 0) 0x7efeac006320 con 0x7efeb00fc2e0 2024-02-19T20:55:55.010 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.026+0000 7efea6ffd700 1 -- 172.21.15.163:0/818614609 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 2758453317 0 0) 0x7efeac03ebf0 con 0x7efeb00fc2e0 2024-02-19T20:55:55.010 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.026+0000 7efea6ffd700 10 client.14112.objecter ms_dispatch 0x7efeb0001550 osd_map(1..1 src has 1..1) v4 2024-02-19T20:55:55.010 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.026+0000 7efea6ffd700 3 client.14112.objecter handle_osd_map got epochs [1,1] > 0 2024-02-19T20:55:55.010 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.026+0000 7efea6ffd700 3 client.14112.objecter handle_osd_map decoding full epoch 1 2024-02-19T20:55:55.010 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.026+0000 7efea6ffd700 20 client.14112.objecter dump_active .. 0 homeless 2024-02-19T20:55:55.010 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.026+0000 7efeb8194700 1 -- 172.21.15.163:0/818614609 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe94004f50 con 0x7efeb00fc2e0 2024-02-19T20:55:55.010 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.026+0000 7efea6ffd700 10 client.14112.objecter ms_handle_connect 0x7efe9c035220 2024-02-19T20:55:55.011 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.031+0000 7efea6ffd700 1 -- 172.21.15.163:0/818614609 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7efeac007a40 con 0x7efeb00fc2e0 2024-02-19T20:55:55.175 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.195+0000 7efeb8194700 1 -- 172.21.15.163:0/818614609 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7efe94005780 con 0x7efeb00fc2e0 2024-02-19T20:55:55.178 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.195+0000 7efea6ffd700 1 -- 172.21.15.163:0/818614609 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+307 (unknown 1187553405 0 1830681963) 0x7efeac0148e0 con 0x7efeb00fc2e0 2024-02-19T20:55:55.178 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.178 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: [global] 2024-02-19T20:55:55.178 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: fsid = 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:55:55.178 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: [mgr] 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: [osd] 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.197+0000 7efeb8194700 10 librados: watch_flush enter 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.197+0000 7efeb8194700 10 librados: watch_flush exit 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.197+0000 7efeb8194700 20 client.14112.objecter shutdown clearing up homeless session... 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.197+0000 7efeb8194700 10 client.14112.objecter successfully canceled tick 2024-02-19T20:55:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.197+0000 7efeb8194700 1 -- 172.21.15.163:0/818614609 >> v1:172.21.15.163:6800/2248999541 conn(0x7efe9c035220 legacy=0x7efe9c0376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:55.180 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.197+0000 7efeb8194700 1 -- 172.21.15.163:0/818614609 >> v1:172.21.15.163:6789/0 conn(0x7efeb00fc2e0 legacy=0x7efeb01096b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:55.180 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.197+0000 7efeb8194700 1 -- 172.21.15.163:0/818614609 shutdown_connections 2024-02-19T20:55:55.180 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.197+0000 7efeb8194700 1 -- 172.21.15.163:0/818614609 >> 172.21.15.163:0/818614609 conn(0x7efeb00f5c40 msgr2=0x7efeb00f6890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:55.180 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.197+0000 7efeb8194700 1 -- 172.21.15.163:0/818614609 shutdown_connections 2024-02-19T20:55:55.180 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.197+0000 7efeb8194700 1 -- 172.21.15.163:0/818614609 wait complete. 2024-02-19T20:55:55.180 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:55.197+0000 7efeb8194700 1 librados: shutdown 2024-02-19T20:55:55.724 INFO:teuthology.orchestra.run.smithi163.stderr:Enabling cephadm module... 2024-02-19T20:55:56.227 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.246+0000 7fda033d1700 1 Processor -- start 2024-02-19T20:55:56.227 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.246+0000 7fda033d1700 1 -- start start 2024-02-19T20:55:56.227 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.246+0000 7fda033d1700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd9fc0fdf20 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.227 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.246+0000 7fda0116d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fd9fc0fd3f0 0x7fd9fc0fd810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:35734/0 (socket says 172.21.15.163:35734) 2024-02-19T20:55:56.228 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.246+0000 7fda0116d700 1 -- 172.21.15.163:0/502221816 learned_addr learned my addr 172.21.15.163:0/502221816 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:56.228 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.246+0000 7fd9f3fff700 1 -- 172.21.15.163:0/502221816 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125589581 0 0) 0x7fd9fc0fdf20 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.228 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.246+0000 7fd9f3fff700 1 -- 172.21.15.163:0/502221816 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9e80033f0 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.228 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.247+0000 7fd9f3fff700 1 -- 172.21.15.163:0/502221816 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4001487651 0 0) 0x7fd9e80033f0 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.228 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.247+0000 7fd9f3fff700 1 -- 172.21.15.163:0/502221816 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9fc0fe0b0 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.228 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:55:54.912133+0000 mon.a (mon.0) 27 : cluster [DBG] mgrmap e4: y(active, since 3s) 2024-02-19T20:55:56.228 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:55.196572+0000 mon.a (mon.0) 28 : audit [INF] from='client.? v1:172.21.15.163:0/818614609' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-19T20:55:56.231 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.247+0000 7fd9f3fff700 1 -- 172.21.15.163:0/502221816 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fd9ec001590 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.231 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.247+0000 7fd9f3fff700 1 -- 172.21.15.163:0/502221816 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7fd9ec001d10 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.231 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.247+0000 7fd9f3fff700 1 -- 172.21.15.163:0/502221816 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fd9ec005750 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.232 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.247+0000 7fda033d1700 1 -- 172.21.15.163:0/502221816 >> v1:172.21.15.163:6789/0 conn(0x7fd9fc0fd3f0 legacy=0x7fd9fc0fd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:56.232 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.247+0000 7fda033d1700 1 -- 172.21.15.163:0/502221816 shutdown_connections 2024-02-19T20:55:56.232 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.247+0000 7fda033d1700 1 -- 172.21.15.163:0/502221816 >> 172.21.15.163:0/502221816 conn(0x7fd9fc0f5cd0 msgr2=0x7fd9fc0f80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:56.232 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.247+0000 7fda033d1700 1 -- 172.21.15.163:0/502221816 shutdown_connections 2024-02-19T20:55:56.232 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.247+0000 7fda033d1700 1 -- 172.21.15.163:0/502221816 wait complete. 2024-02-19T20:55:56.232 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.248+0000 7fda033d1700 1 librados: starting msgr at 2024-02-19T20:55:56.232 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.248+0000 7fda033d1700 1 librados: starting objecter 2024-02-19T20:55:56.232 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.248+0000 7fda033d1700 1 Processor -- start 2024-02-19T20:55:56.232 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.248+0000 7fda033d1700 1 -- start start 2024-02-19T20:55:56.232 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.248+0000 7fda033d1700 1 librados: setting wanted keys 2024-02-19T20:55:56.232 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.248+0000 7fda033d1700 1 librados: calling monclient init 2024-02-19T20:55:56.232 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.248+0000 7fda033d1700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd9fc10ae80 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.232 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.248+0000 7fda0116d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fd9fc0fd3f0 0x7fd9fc10a770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:35740/0 (socket says 172.21.15.163:35740) 2024-02-19T20:55:56.233 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.248+0000 7fda0116d700 1 -- 172.21.15.163:0/3346687401 learned_addr learned my addr 172.21.15.163:0/3346687401 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:55:56.233 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.248+0000 7fd9f27fc700 10 client.?.objecter ms_handle_connect 0x7fd9fc0fd3f0 2024-02-19T20:55:56.233 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.248+0000 7fd9f27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T20:55:56.233 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.249+0000 7fd9f27fc700 1 -- 172.21.15.163:0/3346687401 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475159976 0 0) 0x7fd9fc10ae80 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.233 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.249+0000 7fd9f27fc700 1 -- 172.21.15.163:0/3346687401 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9e8005580 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.233 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.249+0000 7fd9f27fc700 1 -- 172.21.15.163:0/3346687401 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4070745322 0 0) 0x7fd9e8005580 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.233 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.249+0000 7fd9f27fc700 1 -- 172.21.15.163:0/3346687401 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9fc10ae80 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.233 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.249+0000 7fd9f27fc700 1 -- 172.21.15.163:0/3346687401 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fd9ec000fe0 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.233 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.249+0000 7fd9f27fc700 1 -- 172.21.15.163:0/3346687401 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4081167728 0 0) 0x7fd9fc10ae80 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.233 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.249+0000 7fd9f27fc700 1 -- 172.21.15.163:0/3346687401 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9fc10b010 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.233 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.249+0000 7fda033d1700 1 -- 172.21.15.163:0/3346687401 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9fc10b2c0 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.233 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.249+0000 7fda033d1700 10 client.14114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:55:56.233 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.249+0000 7fda033d1700 1 -- 172.21.15.163:0/3346687401 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9fc10efb0 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.233 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.249+0000 7fda033d1700 1 librados: init done 2024-02-19T20:55:56.234 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.249+0000 7fd9f27fc700 1 -- 172.21.15.163:0/3346687401 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7fd9ec001620 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.234 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.249+0000 7fd9f27fc700 1 -- 172.21.15.163:0/3346687401 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fd9ec006270 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.234 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.250+0000 7fd9f27fc700 1 -- 172.21.15.163:0/3346687401 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 38571+0+0 (unknown 3892717071 0 0) 0x7fd9ec00fb50 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.234 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.250+0000 7fda033d1700 1 -- 172.21.15.163:0/3346687401 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e0004f50 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.234 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.250+0000 7fd9f27fc700 1 -- 172.21.15.163:0/3346687401 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (unknown 2758453317 0 0) 0x7fd9ec03ed00 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.234 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.250+0000 7fd9f27fc700 10 client.14114.objecter ms_dispatch 0x7fd9fc001550 osd_map(1..1 src has 1..1) v4 2024-02-19T20:55:56.234 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.250+0000 7fd9f27fc700 3 client.14114.objecter handle_osd_map got epochs [1,1] > 0 2024-02-19T20:55:56.234 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.250+0000 7fd9f27fc700 3 client.14114.objecter handle_osd_map decoding full epoch 1 2024-02-19T20:55:56.234 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.250+0000 7fd9f27fc700 20 client.14114.objecter dump_active .. 0 homeless 2024-02-19T20:55:56.234 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.250+0000 7fd9f27fc700 10 client.14114.objecter ms_handle_connect 0x7fd9e80395a0 2024-02-19T20:55:56.235 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.254+0000 7fd9f27fc700 1 -- 172.21.15.163:0/3346687401 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd9ec011400 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.437 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.456+0000 7fda033d1700 1 -- 172.21.15.163:0/3346687401 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fd9e0005200 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.895 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.914+0000 7fd9f27fc700 1 -- 172.21.15.163:0/3346687401 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mgrmap(e 5) v1 ==== 38582+0+0 (unknown 387226343 0 0) 0x7fd9ec03e2f0 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.896 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.916+0000 7fd9f27fc700 1 -- 172.21.15.163:0/3346687401 <== mon.0 v1:172.21.15.163:6789/0 11 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (unknown 2263024820 0 0) 0x7fd9ec0131c0 con 0x7fd9fc0fd3f0 2024-02-19T20:55:56.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.918+0000 7fda033d1700 10 librados: watch_flush enter 2024-02-19T20:55:56.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.918+0000 7fda033d1700 10 librados: watch_flush exit 2024-02-19T20:55:56.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.918+0000 7fda033d1700 20 client.14114.objecter shutdown clearing up homeless session... 2024-02-19T20:55:56.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.918+0000 7fda033d1700 10 client.14114.objecter successfully canceled tick 2024-02-19T20:55:56.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.918+0000 7fda033d1700 1 -- 172.21.15.163:0/3346687401 >> v1:172.21.15.163:6800/2248999541 conn(0x7fd9e80395a0 legacy=0x7fd9e803ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:56.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.918+0000 7fda033d1700 1 -- 172.21.15.163:0/3346687401 >> v1:172.21.15.163:6789/0 conn(0x7fd9fc0fd3f0 legacy=0x7fd9fc10a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:55:56.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.918+0000 7fda033d1700 1 -- 172.21.15.163:0/3346687401 shutdown_connections 2024-02-19T20:55:56.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.918+0000 7fda033d1700 1 -- 172.21.15.163:0/3346687401 >> 172.21.15.163:0/3346687401 conn(0x7fd9fc0f5cd0 msgr2=0x7fd9fc0f68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:55:56.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.918+0000 7fda033d1700 1 -- 172.21.15.163:0/3346687401 shutdown_connections 2024-02-19T20:55:56.900 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.918+0000 7fda033d1700 1 -- 172.21.15.163:0/3346687401 wait complete. 2024-02-19T20:55:56.901 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:55:56.919+0000 7fda033d1700 1 librados: shutdown 2024-02-19T20:55:57.227 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:56.458423+0000 mon.a (mon.0) 29 : audit [INF] from='client.? v1:172.21.15.163:0/3346687401' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-19T20:55:57.228 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ignoring --setuser ceph since I am not root 2024-02-19T20:55:57.228 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ignoring --setgroup ceph since I am not root 2024-02-19T20:55:57.228 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:57.072+0000 7f35e3bab680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-19T20:55:57.228 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:57.152+0000 7f35e3bab680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-19T20:55:57.894 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:57.520+0000 7f35e3bab680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-19T20:55:58.227 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:55:56.917118+0000 mon.a (mon.0) 30 : audit [INF] from='client.? v1:172.21.15.163:0/3346687401' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-19T20:55:58.227 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:55:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:55:56.917206+0000 mon.a (mon.0) 31 : cluster [DBG] mgrmap e5: y(active, since 5s) 2024-02-19T20:55:58.534 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:58.259+0000 7f35e3bab680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-19T20:55:58.535 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:58.344+0000 7f35e3bab680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-19T20:55:58.819 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:58.554+0000 7f35e3bab680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-19T20:55:59.727 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:59.483+0000 7f35e3bab680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-19T20:55:59.728 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:59.608+0000 7f35e3bab680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-19T20:55:59.728 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:59.686+0000 7f35e3bab680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-19T20:56:00.225 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:59.775+0000 7f35e3bab680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-19T20:56:00.225 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:55:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:55:59.971+0000 7f35e3bab680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-19T20:56:00.477 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:00.244+0000 7f35e3bab680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-19T20:56:00.477 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:00.342+0000 7f35e3bab680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-19T20:56:01.311 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:01.055+0000 7f35e3bab680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-19T20:56:01.311 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:01.134+0000 7f35e3bab680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-19T20:56:01.311 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:01.224+0000 7f35e3bab680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-19T20:56:01.641 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:01.399+0000 7f35e3bab680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-19T20:56:01.900 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:01.660+0000 7f35e3bab680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-19T20:56:01.900 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:01.789+0000 7f35e3bab680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-19T20:56:02.151 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:01.920+0000 7f35e3bab680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-19T20:56:02.151 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:02.089+0000 7f35e3bab680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-19T20:56:02.477 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:02.171+0000 7f35e3bab680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-19T20:56:02.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:02.185389+0000 mon.a (mon.0) 32 : cluster [INF] Active manager daemon y restarted 2024-02-19T20:56:02.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:02.185644+0000 mon.a (mon.0) 33 : cluster [INF] Activating manager daemon y 2024-02-19T20:56:02.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:02.190402+0000 mon.a (mon.0) 34 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-19T20:56:02.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:02.190503+0000 mon.a (mon.0) 35 : cluster [DBG] mgrmap e6: y(active, starting, since 0.00494677s) 2024-02-19T20:56:02.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:02.191550+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-19T20:56:02.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:02.192014+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-19T20:56:02.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:02.192619+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T20:56:02.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:02.192749+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T20:56:02.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:02.192871+0000 mon.a (mon.0) 40 : audit [DBG] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T20:56:02.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:02.206669+0000 mon.a (mon.0) 41 : cluster [INF] Manager daemon y is now available 2024-02-19T20:56:02.479 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:02.218726+0000 mon.a (mon.0) 42 : audit [INF] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' 2024-02-19T20:56:03.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:56:02.216112+0000 mgr.y (mgr.14116) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-02-19T20:56:03.728 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:02.304533+0000 mon.a (mon.0) 43 : audit [INF] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' 2024-02-19T20:56:03.728 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:02.305477+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:56:03.728 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:02.307206+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:56:03.728 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:02.309052+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:56:03.728 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:02.322289+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-19T20:56:03.728 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:02.326185+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-19T20:56:03.728 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:03.190890+0000 mon.a (mon.0) 49 : cluster [DBG] mgrmap e7: y(active, since 1.00532s) 2024-02-19T20:56:05.379 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.671+0000 7f1b6d0b3700 1 Processor -- start 2024-02-19T20:56:05.379 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.671+0000 7f1b6d0b3700 1 -- start start 2024-02-19T20:56:05.379 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.671+0000 7f1b6d0b3700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1b680730c0 con 0x7f1b68072ca0 2024-02-19T20:56:05.379 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.671+0000 7f1b66d9d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f1b68072ca0 0x7f1b68071240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46750/0 (socket says 172.21.15.163:46750) 2024-02-19T20:56:05.379 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.671+0000 7f1b66d9d700 1 -- 172.21.15.163:0/2032330616 learned_addr learned my addr 172.21.15.163:0/2032330616 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:05.379 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.671+0000 7f1b65d9b700 1 -- 172.21.15.163:0/2032330616 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283258759 0 0) 0x7f1b680730c0 con 0x7f1b68072ca0 2024-02-19T20:56:05.379 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.672+0000 7f1b65d9b700 1 -- 172.21.15.163:0/2032330616 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b500033f0 con 0x7f1b68072ca0 2024-02-19T20:56:05.379 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.672+0000 7f1b65d9b700 1 -- 172.21.15.163:0/2032330616 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 946225031 0 0) 0x7f1b500033f0 con 0x7f1b68072ca0 2024-02-19T20:56:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.672+0000 7f1b65d9b700 1 -- 172.21.15.163:0/2032330616 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b68071950 con 0x7f1b68072ca0 2024-02-19T20:56:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.672+0000 7f1b65d9b700 1 -- 172.21.15.163:0/2032330616 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f1b58001590 con 0x7f1b68072ca0 2024-02-19T20:56:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.672+0000 7f1b65d9b700 1 -- 172.21.15.163:0/2032330616 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f1b58001d10 con 0x7f1b68072ca0 2024-02-19T20:56:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.672+0000 7f1b65d9b700 1 -- 172.21.15.163:0/2032330616 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f1b58005750 con 0x7f1b68072ca0 2024-02-19T20:56:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.672+0000 7f1b6d0b3700 1 -- 172.21.15.163:0/2032330616 >> v1:172.21.15.163:6789/0 conn(0x7f1b68072ca0 legacy=0x7f1b68071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.672+0000 7f1b6d0b3700 1 -- 172.21.15.163:0/2032330616 shutdown_connections 2024-02-19T20:56:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.672+0000 7f1b6d0b3700 1 -- 172.21.15.163:0/2032330616 >> 172.21.15.163:0/2032330616 conn(0x7f1b680f7610 msgr2=0x7f1b680f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.672+0000 7f1b6d0b3700 1 -- 172.21.15.163:0/2032330616 shutdown_connections 2024-02-19T20:56:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.672+0000 7f1b6d0b3700 1 -- 172.21.15.163:0/2032330616 wait complete. 2024-02-19T20:56:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.673+0000 7f1b6d0b3700 1 librados: starting msgr at 2024-02-19T20:56:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.673+0000 7f1b6d0b3700 1 librados: starting objecter 2024-02-19T20:56:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.673+0000 7f1b6d0b3700 1 Processor -- start 2024-02-19T20:56:05.380 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.674+0000 7f1b6d0b3700 1 -- start start 2024-02-19T20:56:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.674+0000 7f1b6d0b3700 1 librados: setting wanted keys 2024-02-19T20:56:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.674+0000 7f1b6d0b3700 1 librados: calling monclient init 2024-02-19T20:56:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.674+0000 7f1b6d0b3700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1b6810d0a0 con 0x7f1b68072ca0 2024-02-19T20:56:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.674+0000 7f1b66d9d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f1b68072ca0 0x7f1b6810c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46760/0 (socket says 172.21.15.163:46760) 2024-02-19T20:56:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.674+0000 7f1b66d9d700 1 -- 172.21.15.163:0/3002825755 learned_addr learned my addr 172.21.15.163:0/3002825755 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.674+0000 7f1b4ffff700 10 client.?.objecter ms_handle_connect 0x7f1b68072ca0 2024-02-19T20:56:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.674+0000 7f1b4ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.674+0000 7f1b4ffff700 1 -- 172.21.15.163:0/3002825755 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522420434 0 0) 0x7f1b6810d0a0 con 0x7f1b68072ca0 2024-02-19T20:56:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.674+0000 7f1b4ffff700 1 -- 172.21.15.163:0/3002825755 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b50005580 con 0x7f1b68072ca0 2024-02-19T20:56:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.674+0000 7f1b4ffff700 1 -- 172.21.15.163:0/3002825755 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2987064575 0 0) 0x7f1b50005580 con 0x7f1b68072ca0 2024-02-19T20:56:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.674+0000 7f1b4ffff700 1 -- 172.21.15.163:0/3002825755 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b6810d0a0 con 0x7f1b68072ca0 2024-02-19T20:56:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.674+0000 7f1b4ffff700 1 -- 172.21.15.163:0/3002825755 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f1b58000fe0 con 0x7f1b68072ca0 2024-02-19T20:56:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.675+0000 7f1b4ffff700 1 -- 172.21.15.163:0/3002825755 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3552932797 0 0) 0x7f1b6810d0a0 con 0x7f1b68072ca0 2024-02-19T20:56:05.381 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.675+0000 7f1b4ffff700 1 -- 172.21.15.163:0/3002825755 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6810d230 con 0x7f1b68072ca0 2024-02-19T20:56:05.382 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.675+0000 7f1b6d0b3700 1 -- 172.21.15.163:0/3002825755 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6810d4e0 con 0x7f1b68072ca0 2024-02-19T20:56:05.382 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.675+0000 7f1b6d0b3700 10 client.14122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:05.382 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.675+0000 7f1b6d0b3700 1 -- 172.21.15.163:0/3002825755 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b681111d0 con 0x7f1b68072ca0 2024-02-19T20:56:05.382 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.675+0000 7f1b6d0b3700 1 librados: init done 2024-02-19T20:56:05.382 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.675+0000 7f1b4ffff700 1 -- 172.21.15.163:0/3002825755 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f1b58001b30 con 0x7f1b68072ca0 2024-02-19T20:56:05.382 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.675+0000 7f1b4ffff700 1 -- 172.21.15.163:0/3002825755 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f1b580061c0 con 0x7f1b68072ca0 2024-02-19T20:56:05.382 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.676+0000 7f1b4ffff700 1 -- 172.21.15.163:0/3002825755 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 532658122 0 0) 0x7f1b5800fac0 con 0x7f1b68072ca0 2024-02-19T20:56:05.382 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.676+0000 7f1b6d0b3700 1 -- 172.21.15.163:0/3002825755 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b54004f50 con 0x7f1b68072ca0 2024-02-19T20:56:05.382 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.676+0000 7f1b4ffff700 1 -- 172.21.15.163:0/3002825755 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 1409061398 0 0) 0x7f1b58004790 con 0x7f1b68072ca0 2024-02-19T20:56:05.382 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.676+0000 7f1b4ffff700 10 client.14122.objecter ms_dispatch 0x7f1b68001550 osd_map(2..2 src has 1..2) v4 2024-02-19T20:56:05.382 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.676+0000 7f1b4ffff700 3 client.14122.objecter handle_osd_map got epochs [2,2] > 0 2024-02-19T20:56:05.382 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.676+0000 7f1b4ffff700 3 client.14122.objecter handle_osd_map decoding full epoch 2 2024-02-19T20:56:05.382 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.676+0000 7f1b4ffff700 20 client.14122.objecter dump_active .. 0 homeless 2024-02-19T20:56:05.382 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.676+0000 7f1b4ffff700 10 client.14122.objecter ms_handle_connect 0x7f1b50035230 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.681+0000 7f1b4ffff700 1 -- 172.21.15.163:0/3002825755 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1b580111f0 con 0x7f1b68072ca0 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.885+0000 7f1b6d0b3700 1 -- 172.21.15.163:0/3002825755 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f1b54005a10 con 0x7f1b68072ca0 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.885+0000 7f1b4ffff700 1 -- 172.21.15.163:0/3002825755 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v8) v1 ==== 56+0+88 (unknown 3921957458 0 3691202818) 0x7f1b580149c0 con 0x7f1b68072ca0 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.887+0000 7f1b6d0b3700 10 librados: watch_flush enter 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.887+0000 7f1b6d0b3700 10 librados: watch_flush exit 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.887+0000 7f1b6d0b3700 20 client.14122.objecter shutdown clearing up homeless session... 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.887+0000 7f1b6d0b3700 10 client.14122.objecter successfully canceled tick 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.887+0000 7f1b6d0b3700 1 -- 172.21.15.163:0/3002825755 >> v1:172.21.15.163:6800/2731225015 conn(0x7f1b50035230 legacy=0x7f1b500376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.887+0000 7f1b6d0b3700 1 -- 172.21.15.163:0/3002825755 >> v1:172.21.15.163:6789/0 conn(0x7f1b68072ca0 legacy=0x7f1b6810c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.887+0000 7f1b6d0b3700 1 -- 172.21.15.163:0/3002825755 shutdown_connections 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.887+0000 7f1b6d0b3700 1 -- 172.21.15.163:0/3002825755 >> 172.21.15.163:0/3002825755 conn(0x7f1b680f7610 msgr2=0x7f1b68100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.887+0000 7f1b6d0b3700 1 -- 172.21.15.163:0/3002825755 shutdown_connections 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.887+0000 7f1b6d0b3700 1 -- 172.21.15.163:0/3002825755 wait complete. 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:04.887+0000 7f1b6d0b3700 1 librados: shutdown 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: { 2024-02-19T20:56:05.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 8, 2024-02-19T20:56:05.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "available": true, 2024-02-19T20:56:05.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-19T20:56:05.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-19T20:56:05.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: } 2024-02-19T20:56:05.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:04.307192+0000 mon.a (mon.0) 50 : cluster [DBG] mgrmap e8: y(active, since 2s) 2024-02-19T20:56:05.728 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:04.886696+0000 mon.a (mon.0) 51 : audit [DBG] from='client.? v1:172.21.15.163:0/3002825755' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T20:56:06.950 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for the mgr to restart... 2024-02-19T20:56:06.950 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for mgr epoch 8... 2024-02-19T20:56:07.495 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.515+0000 7f1b35e90700 1 Processor -- start 2024-02-19T20:56:07.499 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.515+0000 7f1b35e90700 1 -- start start 2024-02-19T20:56:07.499 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.515+0000 7f1b35e90700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1b301001b0 con 0x7f1b30070c20 2024-02-19T20:56:07.499 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.515+0000 7f1b2f7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f1b30070c20 0x7f1b300fea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46772/0 (socket says 172.21.15.163:46772) 2024-02-19T20:56:07.499 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.515+0000 7f1b2f7fe700 1 -- 172.21.15.163:0/1645562855 learned_addr learned my addr 172.21.15.163:0/1645562855 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:07.499 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.516+0000 7f1b2e7fc700 1 -- 172.21.15.163:0/1645562855 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400534332 0 0) 0x7f1b301001b0 con 0x7f1b30070c20 2024-02-19T20:56:07.499 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.516+0000 7f1b2e7fc700 1 -- 172.21.15.163:0/1645562855 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b180033f0 con 0x7f1b30070c20 2024-02-19T20:56:07.499 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.516+0000 7f1b2e7fc700 1 -- 172.21.15.163:0/1645562855 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2682514269 0 0) 0x7f1b180033f0 con 0x7f1b30070c20 2024-02-19T20:56:07.499 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.516+0000 7f1b2e7fc700 1 -- 172.21.15.163:0/1645562855 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b30101350 con 0x7f1b30070c20 2024-02-19T20:56:07.499 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.516+0000 7f1b2e7fc700 1 -- 172.21.15.163:0/1645562855 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f1b24001590 con 0x7f1b30070c20 2024-02-19T20:56:07.499 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.516+0000 7f1b2e7fc700 1 -- 172.21.15.163:0/1645562855 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f1b24001d10 con 0x7f1b30070c20 2024-02-19T20:56:07.499 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.516+0000 7f1b2e7fc700 1 -- 172.21.15.163:0/1645562855 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f1b240056e0 con 0x7f1b30070c20 2024-02-19T20:56:07.499 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.517+0000 7f1b35e90700 1 -- 172.21.15.163:0/1645562855 >> v1:172.21.15.163:6789/0 conn(0x7f1b30070c20 legacy=0x7f1b300fea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:07.499 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.517+0000 7f1b35e90700 1 -- 172.21.15.163:0/1645562855 shutdown_connections 2024-02-19T20:56:07.499 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.517+0000 7f1b35e90700 1 -- 172.21.15.163:0/1645562855 >> 172.21.15.163:0/1645562855 conn(0x7f1b300f64d0 msgr2=0x7f1b300f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:07.500 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.517+0000 7f1b35e90700 1 -- 172.21.15.163:0/1645562855 shutdown_connections 2024-02-19T20:56:07.500 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.517+0000 7f1b35e90700 1 -- 172.21.15.163:0/1645562855 wait complete. 2024-02-19T20:56:07.500 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.517+0000 7f1b35e90700 1 librados: starting msgr at 2024-02-19T20:56:07.500 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.517+0000 7f1b35e90700 1 librados: starting objecter 2024-02-19T20:56:07.500 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.517+0000 7f1b35e90700 1 Processor -- start 2024-02-19T20:56:07.500 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.517+0000 7f1b35e90700 1 -- start start 2024-02-19T20:56:07.500 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.517+0000 7f1b35e90700 1 librados: setting wanted keys 2024-02-19T20:56:07.500 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.517+0000 7f1b35e90700 1 librados: calling monclient init 2024-02-19T20:56:07.500 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.518+0000 7f1b35e90700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1b300fc4a0 con 0x7f1b30070c20 2024-02-19T20:56:07.500 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.518+0000 7f1b2f7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f1b30070c20 0x7f1b300fa990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46780/0 (socket says 172.21.15.163:46780) 2024-02-19T20:56:07.500 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.518+0000 7f1b2f7fe700 1 -- 172.21.15.163:0/2099461146 learned_addr learned my addr 172.21.15.163:0/2099461146 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:07.500 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.518+0000 7f1b2cff9700 10 client.?.objecter ms_handle_connect 0x7f1b30070c20 2024-02-19T20:56:07.500 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.518+0000 7f1b2cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:07.501 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.518+0000 7f1b2cff9700 1 -- 172.21.15.163:0/2099461146 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323062989 0 0) 0x7f1b300fc4a0 con 0x7f1b30070c20 2024-02-19T20:56:07.501 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.518+0000 7f1b2cff9700 1 -- 172.21.15.163:0/2099461146 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b18005580 con 0x7f1b30070c20 2024-02-19T20:56:07.501 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.518+0000 7f1b2cff9700 1 -- 172.21.15.163:0/2099461146 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1184674480 0 0) 0x7f1b18005580 con 0x7f1b30070c20 2024-02-19T20:56:07.502 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.518+0000 7f1b2cff9700 1 -- 172.21.15.163:0/2099461146 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b300fc4a0 con 0x7f1b30070c20 2024-02-19T20:56:07.502 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.518+0000 7f1b2cff9700 1 -- 172.21.15.163:0/2099461146 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f1b24000fe0 con 0x7f1b30070c20 2024-02-19T20:56:07.502 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.518+0000 7f1b2cff9700 1 -- 172.21.15.163:0/2099461146 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3904740892 0 0) 0x7f1b300fc4a0 con 0x7f1b30070c20 2024-02-19T20:56:07.502 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.518+0000 7f1b2cff9700 1 -- 172.21.15.163:0/2099461146 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b300fc630 con 0x7f1b30070c20 2024-02-19T20:56:07.502 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.518+0000 7f1b35e90700 1 -- 172.21.15.163:0/2099461146 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b300fc8e0 con 0x7f1b30070c20 2024-02-19T20:56:07.502 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.519+0000 7f1b35e90700 10 client.14124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:07.502 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.519+0000 7f1b35e90700 1 -- 172.21.15.163:0/2099461146 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b300fb3a0 con 0x7f1b30070c20 2024-02-19T20:56:07.502 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.519+0000 7f1b35e90700 1 librados: init done 2024-02-19T20:56:07.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.519+0000 7f1b2cff9700 1 -- 172.21.15.163:0/2099461146 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f1b240046b0 con 0x7f1b30070c20 2024-02-19T20:56:07.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.519+0000 7f1b2cff9700 1 -- 172.21.15.163:0/2099461146 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f1b24006120 con 0x7f1b30070c20 2024-02-19T20:56:07.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.519+0000 7f1b2cff9700 1 -- 172.21.15.163:0/2099461146 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 532658122 0 0) 0x7f1b2400fa20 con 0x7f1b30070c20 2024-02-19T20:56:07.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.519+0000 7f1b2cff9700 1 -- 172.21.15.163:0/2099461146 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 1409061398 0 0) 0x7f1b2403fac0 con 0x7f1b30070c20 2024-02-19T20:56:07.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.519+0000 7f1b2cff9700 10 client.14124.objecter ms_dispatch 0x7f1b30001550 osd_map(2..2 src has 1..2) v4 2024-02-19T20:56:07.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.519+0000 7f1b2cff9700 3 client.14124.objecter handle_osd_map got epochs [2,2] > 0 2024-02-19T20:56:07.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.519+0000 7f1b2cff9700 3 client.14124.objecter handle_osd_map decoding full epoch 2 2024-02-19T20:56:07.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.519+0000 7f1b35e90700 1 -- 172.21.15.163:0/2099461146 --> v1:172.21.15.163:6800/2731225015 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b1c000d40 con 0x7f1b180352c0 2024-02-19T20:56:07.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.519+0000 7f1b2cff9700 20 client.14124.objecter dump_active .. 0 homeless 2024-02-19T20:56:07.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.520+0000 7f1b2cff9700 10 client.14124.objecter ms_handle_connect 0x7f1b180352c0 2024-02-19T20:56:07.503 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.520+0000 7f1b2cff9700 1 -- 172.21.15.163:0/2099461146 <== mgr.14116 v1:172.21.15.163:6800/2731225015 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (unknown 0 0 4198994610) 0x7f1b1c000d40 con 0x7f1b180352c0 2024-02-19T20:56:07.506 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.525+0000 7f1b35e90700 1 -- 172.21.15.163:0/2099461146 --> v1:172.21.15.163:6800/2731225015 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f1b1c002610 con 0x7f1b180352c0 2024-02-19T20:56:07.506 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.526+0000 7f1b2cff9700 1 -- 172.21.15.163:0/2099461146 <== mgr.14116 v1:172.21.15.163:6800/2731225015 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (unknown 0 0 1849557100) 0x7f1b1c002610 con 0x7f1b180352c0 2024-02-19T20:56:07.506 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: { 2024-02-19T20:56:07.507 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mgrmap_epoch": 8, 2024-02-19T20:56:07.507 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "initialized": true 2024-02-19T20:56:07.507 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: } 2024-02-19T20:56:07.507 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.526+0000 7f1b35e90700 10 librados: watch_flush enter 2024-02-19T20:56:07.507 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.526+0000 7f1b35e90700 10 librados: watch_flush exit 2024-02-19T20:56:07.507 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.526+0000 7f1b35e90700 20 client.14124.objecter shutdown clearing up homeless session... 2024-02-19T20:56:07.507 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.526+0000 7f1b35e90700 10 client.14124.objecter successfully canceled tick 2024-02-19T20:56:07.508 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.526+0000 7f1b35e90700 1 -- 172.21.15.163:0/2099461146 >> v1:172.21.15.163:6800/2731225015 conn(0x7f1b180352c0 legacy=0x7f1b18037780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:07.508 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.526+0000 7f1b35e90700 1 -- 172.21.15.163:0/2099461146 >> v1:172.21.15.163:6789/0 conn(0x7f1b30070c20 legacy=0x7f1b300fa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:07.508 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.526+0000 7f1b35e90700 1 -- 172.21.15.163:0/2099461146 shutdown_connections 2024-02-19T20:56:07.508 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.526+0000 7f1b35e90700 1 -- 172.21.15.163:0/2099461146 >> 172.21.15.163:0/2099461146 conn(0x7f1b300f64d0 msgr2=0x7f1b300fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:07.508 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.526+0000 7f1b35e90700 1 -- 172.21.15.163:0/2099461146 shutdown_connections 2024-02-19T20:56:07.508 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.526+0000 7f1b35e90700 1 -- 172.21.15.163:0/2099461146 wait complete. 2024-02-19T20:56:07.508 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:07.526+0000 7f1b35e90700 1 librados: shutdown 2024-02-19T20:56:08.122 INFO:teuthology.orchestra.run.smithi163.stderr:mgr epoch 8 is available 2024-02-19T20:56:08.123 INFO:teuthology.orchestra.run.smithi163.stderr:Setting orchestrator backend to cephadm... 2024-02-19T20:56:08.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:07.521534+0000 mgr.y (mgr.14116) 2 : audit [DBG] from='client.14124 v1:172.21.15.163:0/2099461146' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T20:56:08.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:07.527201+0000 mgr.y (mgr.14116) 3 : audit [DBG] from='client.14124 v1:172.21.15.163:0/2099461146' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T20:56:08.672 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.688+0000 7f3317d81700 1 Processor -- start 2024-02-19T20:56:08.672 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.688+0000 7f3317d81700 1 -- start start 2024-02-19T20:56:08.672 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.689+0000 7f3317d81700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f33100fb8b0 con 0x7f33100fad80 2024-02-19T20:56:08.672 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.689+0000 7f3315b1d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f33100fad80 0x7f33100fb1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46794/0 (socket says 172.21.15.163:46794) 2024-02-19T20:56:08.672 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.689+0000 7f3315b1d700 1 -- 172.21.15.163:0/2278326103 learned_addr learned my addr 172.21.15.163:0/2278326103 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:08.672 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.689+0000 7f3314b1b700 1 -- 172.21.15.163:0/2278326103 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924527825 0 0) 0x7f3300001110 con 0x7f33100fad80 2024-02-19T20:56:08.672 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.689+0000 7f3314b1b700 1 -- 172.21.15.163:0/2278326103 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32fc0033f0 con 0x7f33100fad80 2024-02-19T20:56:08.672 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.689+0000 7f3314b1b700 1 -- 172.21.15.163:0/2278326103 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 773377614 0 0) 0x7f32fc0033f0 con 0x7f33100fad80 2024-02-19T20:56:08.672 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.689+0000 7f3314b1b700 1 -- 172.21.15.163:0/2278326103 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33101001c0 con 0x7f33100fad80 2024-02-19T20:56:08.672 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.689+0000 7f3314b1b700 1 -- 172.21.15.163:0/2278326103 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f3300001720 con 0x7f33100fad80 2024-02-19T20:56:08.673 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.689+0000 7f3314b1b700 1 -- 172.21.15.163:0/2278326103 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f3300004770 con 0x7f33100fad80 2024-02-19T20:56:08.673 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.689+0000 7f3314b1b700 1 -- 172.21.15.163:0/2278326103 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f33000058f0 con 0x7f33100fad80 2024-02-19T20:56:08.673 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.690+0000 7f3317d81700 1 -- 172.21.15.163:0/2278326103 >> v1:172.21.15.163:6789/0 conn(0x7f33100fad80 legacy=0x7f33100fb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:08.673 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.690+0000 7f3317d81700 1 -- 172.21.15.163:0/2278326103 shutdown_connections 2024-02-19T20:56:08.673 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.690+0000 7f3317d81700 1 -- 172.21.15.163:0/2278326103 >> 172.21.15.163:0/2278326103 conn(0x7f33100f6520 msgr2=0x7f33100f8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:08.673 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.690+0000 7f3317d81700 1 -- 172.21.15.163:0/2278326103 shutdown_connections 2024-02-19T20:56:08.673 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.690+0000 7f3317d81700 1 -- 172.21.15.163:0/2278326103 wait complete. 2024-02-19T20:56:08.673 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.690+0000 7f3317d81700 1 librados: starting msgr at 2024-02-19T20:56:08.673 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.690+0000 7f3317d81700 1 librados: starting objecter 2024-02-19T20:56:08.673 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.690+0000 7f3317d81700 1 Processor -- start 2024-02-19T20:56:08.673 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.691+0000 7f3317d81700 1 -- start start 2024-02-19T20:56:08.673 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.691+0000 7f3317d81700 1 librados: setting wanted keys 2024-02-19T20:56:08.673 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.691+0000 7f3317d81700 1 librados: calling monclient init 2024-02-19T20:56:08.673 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.691+0000 7f3317d81700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f33100f8530 con 0x7f33100fad80 2024-02-19T20:56:08.673 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.691+0000 7f3315b1d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f33100fad80 0x7f33100f6a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46804/0 (socket says 172.21.15.163:46804) 2024-02-19T20:56:08.674 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.691+0000 7f3315b1d700 1 -- 172.21.15.163:0/1618792147 learned_addr learned my addr 172.21.15.163:0/1618792147 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:08.674 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.691+0000 7f3306ffd700 10 client.?.objecter ms_handle_connect 0x7f33100fad80 2024-02-19T20:56:08.674 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.691+0000 7f3306ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:08.674 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.691+0000 7f3306ffd700 1 -- 172.21.15.163:0/1618792147 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013819955 0 0) 0x7f33100f8530 con 0x7f33100fad80 2024-02-19T20:56:08.674 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.691+0000 7f3306ffd700 1 -- 172.21.15.163:0/1618792147 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32fc005580 con 0x7f33100fad80 2024-02-19T20:56:08.675 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.691+0000 7f3306ffd700 1 -- 172.21.15.163:0/1618792147 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2970053108 0 0) 0x7f32fc005580 con 0x7f33100fad80 2024-02-19T20:56:08.675 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.691+0000 7f3306ffd700 1 -- 172.21.15.163:0/1618792147 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33100f8530 con 0x7f33100fad80 2024-02-19T20:56:08.675 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.691+0000 7f3306ffd700 1 -- 172.21.15.163:0/1618792147 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f3300004d00 con 0x7f33100fad80 2024-02-19T20:56:08.675 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.692+0000 7f3306ffd700 1 -- 172.21.15.163:0/1618792147 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 771925611 0 0) 0x7f33100f8530 con 0x7f33100fad80 2024-02-19T20:56:08.675 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.692+0000 7f3306ffd700 1 -- 172.21.15.163:0/1618792147 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33100f86c0 con 0x7f33100fad80 2024-02-19T20:56:08.676 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.692+0000 7f3317d81700 1 -- 172.21.15.163:0/1618792147 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33100f8970 con 0x7f33100fad80 2024-02-19T20:56:08.676 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.692+0000 7f3317d81700 10 client.14126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:08.676 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.692+0000 7f3317d81700 1 -- 172.21.15.163:0/1618792147 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33100f7430 con 0x7f33100fad80 2024-02-19T20:56:08.676 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.692+0000 7f3317d81700 1 librados: init done 2024-02-19T20:56:08.676 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.692+0000 7f3306ffd700 1 -- 172.21.15.163:0/1618792147 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f33000048e0 con 0x7f33100fad80 2024-02-19T20:56:08.676 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.692+0000 7f3306ffd700 1 -- 172.21.15.163:0/1618792147 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f3300006250 con 0x7f33100fad80 2024-02-19T20:56:08.676 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.693+0000 7f3306ffd700 1 -- 172.21.15.163:0/1618792147 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 532658122 0 0) 0x7f330000fa70 con 0x7f33100fad80 2024-02-19T20:56:08.676 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.693+0000 7f3317d81700 1 -- 172.21.15.163:0/1618792147 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f4004f50 con 0x7f33100fad80 2024-02-19T20:56:08.676 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.693+0000 7f3306ffd700 1 -- 172.21.15.163:0/1618792147 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 1409061398 0 0) 0x7f330003ea50 con 0x7f33100fad80 2024-02-19T20:56:08.676 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.693+0000 7f3306ffd700 10 client.14126.objecter ms_dispatch 0x7f3310001550 osd_map(2..2 src has 1..2) v4 2024-02-19T20:56:08.676 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.693+0000 7f3306ffd700 3 client.14126.objecter handle_osd_map got epochs [2,2] > 0 2024-02-19T20:56:08.676 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.693+0000 7f3306ffd700 3 client.14126.objecter handle_osd_map decoding full epoch 2 2024-02-19T20:56:08.676 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.693+0000 7f3306ffd700 20 client.14126.objecter dump_active .. 0 homeless 2024-02-19T20:56:08.676 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.693+0000 7f3306ffd700 10 client.14126.objecter ms_handle_connect 0x7f32fc0351e0 2024-02-19T20:56:08.677 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.697+0000 7f3306ffd700 1 -- 172.21.15.163:0/1618792147 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3300010ea0 con 0x7f33100fad80 2024-02-19T20:56:08.851 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.871+0000 7f3317d81700 1 -- 172.21.15.163:0/1618792147 --> v1:172.21.15.163:6800/2731225015 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f32f4000bc0 con 0x7f32fc0351e0 2024-02-19T20:56:08.910 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.876+0000 7f3306ffd700 1 -- 172.21.15.163:0/1618792147 <== mgr.14116 v1:172.21.15.163:6800/2731225015 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7f32f4000bc0 con 0x7f32fc0351e0 2024-02-19T20:56:08.910 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.878+0000 7f3317d81700 10 librados: watch_flush enter 2024-02-19T20:56:08.910 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.878+0000 7f3317d81700 10 librados: watch_flush exit 2024-02-19T20:56:08.910 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.878+0000 7f3317d81700 20 client.14126.objecter shutdown clearing up homeless session... 2024-02-19T20:56:08.910 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.878+0000 7f3317d81700 10 client.14126.objecter successfully canceled tick 2024-02-19T20:56:08.910 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.878+0000 7f3317d81700 1 -- 172.21.15.163:0/1618792147 >> v1:172.21.15.163:6800/2731225015 conn(0x7f32fc0351e0 legacy=0x7f32fc0376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:08.911 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.878+0000 7f3317d81700 1 -- 172.21.15.163:0/1618792147 >> v1:172.21.15.163:6789/0 conn(0x7f33100fad80 legacy=0x7f33100f6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:08.911 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.878+0000 7f3317d81700 1 -- 172.21.15.163:0/1618792147 shutdown_connections 2024-02-19T20:56:08.911 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.878+0000 7f3317d81700 1 -- 172.21.15.163:0/1618792147 >> 172.21.15.163:0/1618792147 conn(0x7f33100f6520 msgr2=0x7f331006fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:08.911 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.878+0000 7f3317d81700 1 -- 172.21.15.163:0/1618792147 shutdown_connections 2024-02-19T20:56:08.911 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.878+0000 7f3317d81700 1 -- 172.21.15.163:0/1618792147 wait complete. 2024-02-19T20:56:08.911 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:08.878+0000 7f3317d81700 1 librados: shutdown 2024-02-19T20:56:09.955 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.971+0000 7fc5a03e5700 1 Processor -- start 2024-02-19T20:56:09.955 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.971+0000 7fc5a03e5700 1 -- start start 2024-02-19T20:56:09.956 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.971+0000 7fc5a03e5700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc5980730e0 con 0x7fc598072cc0 2024-02-19T20:56:09.956 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.971+0000 7fc59e181700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fc598072cc0 0x7fc598071260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46810/0 (socket says 172.21.15.163:46810) 2024-02-19T20:56:09.956 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.971+0000 7fc59e181700 1 -- 172.21.15.163:0/84614686 learned_addr learned my addr 172.21.15.163:0/84614686 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:09.956 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.971+0000 7fc59d17f700 1 -- 172.21.15.163:0/84614686 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218840438 0 0) 0x7fc5980730e0 con 0x7fc598072cc0 2024-02-19T20:56:09.956 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.971+0000 7fc59d17f700 1 -- 172.21.15.163:0/84614686 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5840033f0 con 0x7fc598072cc0 2024-02-19T20:56:09.956 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.972+0000 7fc59d17f700 1 -- 172.21.15.163:0/84614686 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 855961411 0 0) 0x7fc5840033f0 con 0x7fc598072cc0 2024-02-19T20:56:09.956 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.972+0000 7fc59d17f700 1 -- 172.21.15.163:0/84614686 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc598071970 con 0x7fc598072cc0 2024-02-19T20:56:09.956 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.972+0000 7fc59d17f700 1 -- 172.21.15.163:0/84614686 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc594001590 con 0x7fc598072cc0 2024-02-19T20:56:09.956 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.972+0000 7fc59d17f700 1 -- 172.21.15.163:0/84614686 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7fc594001c40 con 0x7fc598072cc0 2024-02-19T20:56:09.956 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.972+0000 7fc59d17f700 1 -- 172.21.15.163:0/84614686 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc594005680 con 0x7fc598072cc0 2024-02-19T20:56:09.956 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.972+0000 7fc5a03e5700 1 -- 172.21.15.163:0/84614686 >> v1:172.21.15.163:6789/0 conn(0x7fc598072cc0 legacy=0x7fc598071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:09.956 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.972+0000 7fc5a03e5700 1 -- 172.21.15.163:0/84614686 shutdown_connections 2024-02-19T20:56:09.956 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.972+0000 7fc5a03e5700 1 -- 172.21.15.163:0/84614686 >> 172.21.15.163:0/84614686 conn(0x7fc5980f7610 msgr2=0x7fc5980f9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:09.956 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.972+0000 7fc5a03e5700 1 -- 172.21.15.163:0/84614686 shutdown_connections 2024-02-19T20:56:09.957 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.972+0000 7fc5a03e5700 1 -- 172.21.15.163:0/84614686 wait complete. 2024-02-19T20:56:09.957 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.973+0000 7fc5a03e5700 1 librados: starting msgr at 2024-02-19T20:56:09.957 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.973+0000 7fc5a03e5700 1 librados: starting objecter 2024-02-19T20:56:09.957 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.973+0000 7fc5a03e5700 1 Processor -- start 2024-02-19T20:56:09.957 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.973+0000 7fc5a03e5700 1 -- start start 2024-02-19T20:56:09.957 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.973+0000 7fc5a03e5700 1 librados: setting wanted keys 2024-02-19T20:56:09.957 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.973+0000 7fc5a03e5700 1 librados: calling monclient init 2024-02-19T20:56:09.957 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.973+0000 7fc5a03e5700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc59810d110 con 0x7fc598072cc0 2024-02-19T20:56:09.957 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.973+0000 7fc59e181700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fc598072cc0 0x7fc59810ca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46818/0 (socket says 172.21.15.163:46818) 2024-02-19T20:56:09.957 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.973+0000 7fc59e181700 1 -- 172.21.15.163:0/3219050198 learned_addr learned my addr 172.21.15.163:0/3219050198 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:09.957 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.973+0000 7fc58b7fe700 10 client.?.objecter ms_handle_connect 0x7fc598072cc0 2024-02-19T20:56:09.957 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.973+0000 7fc58b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:09.957 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.974+0000 7fc58b7fe700 1 -- 172.21.15.163:0/3219050198 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349408758 0 0) 0x7fc59810d110 con 0x7fc598072cc0 2024-02-19T20:56:09.957 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.974+0000 7fc58b7fe700 1 -- 172.21.15.163:0/3219050198 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5840055d0 con 0x7fc598072cc0 2024-02-19T20:56:09.957 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.974+0000 7fc58b7fe700 1 -- 172.21.15.163:0/3219050198 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3865074718 0 0) 0x7fc5840055d0 con 0x7fc598072cc0 2024-02-19T20:56:09.958 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.974+0000 7fc58b7fe700 1 -- 172.21.15.163:0/3219050198 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc59810d110 con 0x7fc598072cc0 2024-02-19T20:56:09.958 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.974+0000 7fc58b7fe700 1 -- 172.21.15.163:0/3219050198 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc594000ff0 con 0x7fc598072cc0 2024-02-19T20:56:09.958 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.974+0000 7fc58b7fe700 1 -- 172.21.15.163:0/3219050198 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1712866960 0 0) 0x7fc59810d110 con 0x7fc598072cc0 2024-02-19T20:56:09.958 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.974+0000 7fc58b7fe700 1 -- 172.21.15.163:0/3219050198 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59810d2a0 con 0x7fc598072cc0 2024-02-19T20:56:09.958 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.974+0000 7fc5a03e5700 1 -- 172.21.15.163:0/3219050198 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59810d550 con 0x7fc598072cc0 2024-02-19T20:56:09.958 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.974+0000 7fc5a03e5700 10 client.14128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:09.958 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.974+0000 7fc5a03e5700 1 -- 172.21.15.163:0/3219050198 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc598111240 con 0x7fc598072cc0 2024-02-19T20:56:09.958 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.974+0000 7fc5a03e5700 1 librados: init done 2024-02-19T20:56:09.958 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.974+0000 7fc58b7fe700 1 -- 172.21.15.163:0/3219050198 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7fc594004cf0 con 0x7fc598072cc0 2024-02-19T20:56:09.958 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.974+0000 7fc58b7fe700 1 -- 172.21.15.163:0/3219050198 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc594006130 con 0x7fc598072cc0 2024-02-19T20:56:09.958 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.975+0000 7fc58b7fe700 1 -- 172.21.15.163:0/3219050198 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 532658122 0 0) 0x7fc59400f950 con 0x7fc598072cc0 2024-02-19T20:56:09.959 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.975+0000 7fc5a03e5700 1 -- 172.21.15.163:0/3219050198 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc57c004f50 con 0x7fc598072cc0 2024-02-19T20:56:09.959 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.975+0000 7fc58b7fe700 1 -- 172.21.15.163:0/3219050198 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 1409061398 0 0) 0x7fc59403fab0 con 0x7fc598072cc0 2024-02-19T20:56:09.959 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.975+0000 7fc58b7fe700 10 client.14128.objecter ms_dispatch 0x7fc598001550 osd_map(2..2 src has 1..2) v4 2024-02-19T20:56:09.959 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.975+0000 7fc58b7fe700 3 client.14128.objecter handle_osd_map got epochs [2,2] > 0 2024-02-19T20:56:09.959 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.975+0000 7fc58b7fe700 3 client.14128.objecter handle_osd_map decoding full epoch 2 2024-02-19T20:56:09.959 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.975+0000 7fc58b7fe700 20 client.14128.objecter dump_active .. 0 homeless 2024-02-19T20:56:09.959 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.976+0000 7fc58b7fe700 10 client.14128.objecter ms_handle_connect 0x7fc5840352a0 2024-02-19T20:56:09.960 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:09.980+0000 7fc58b7fe700 1 -- 172.21.15.163:0/3219050198 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc594011200 con 0x7fc598072cc0 2024-02-19T20:56:10.123 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:10.143+0000 7fc5a03e5700 1 -- 172.21.15.163:0/3219050198 --> v1:172.21.15.163:6800/2731225015 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fc57c000bc0 con 0x7fc5840352a0 2024-02-19T20:56:10.124 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:10.143+0000 7fc58b7fe700 1 -- 172.21.15.163:0/3219050198 <== mgr.14116 v1:172.21.15.163:6800/2731225015 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (unknown 0 0 2070689548) 0x7fc57c000bc0 con 0x7fc5840352a0 2024-02-19T20:56:10.124 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: value unchanged 2024-02-19T20:56:10.126 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:10.145+0000 7fc5a03e5700 10 librados: watch_flush enter 2024-02-19T20:56:10.126 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:10.145+0000 7fc5a03e5700 10 librados: watch_flush exit 2024-02-19T20:56:10.126 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:10.145+0000 7fc5a03e5700 20 client.14128.objecter shutdown clearing up homeless session... 2024-02-19T20:56:10.126 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:10.145+0000 7fc5a03e5700 10 client.14128.objecter successfully canceled tick 2024-02-19T20:56:10.127 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:10.145+0000 7fc5a03e5700 1 -- 172.21.15.163:0/3219050198 >> v1:172.21.15.163:6800/2731225015 conn(0x7fc5840352a0 legacy=0x7fc584037760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:10.127 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:10.145+0000 7fc5a03e5700 1 -- 172.21.15.163:0/3219050198 >> v1:172.21.15.163:6789/0 conn(0x7fc598072cc0 legacy=0x7fc59810ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:10.127 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:10.145+0000 7fc5a03e5700 1 -- 172.21.15.163:0/3219050198 shutdown_connections 2024-02-19T20:56:10.127 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:10.145+0000 7fc5a03e5700 1 -- 172.21.15.163:0/3219050198 >> 172.21.15.163:0/3219050198 conn(0x7fc5980f7610 msgr2=0x7fc598100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:10.127 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:10.145+0000 7fc5a03e5700 1 -- 172.21.15.163:0/3219050198 shutdown_connections 2024-02-19T20:56:10.127 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:10.145+0000 7fc5a03e5700 1 -- 172.21.15.163:0/3219050198 wait complete. 2024-02-19T20:56:10.127 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:10.145+0000 7fc5a03e5700 1 librados: shutdown 2024-02-19T20:56:10.146 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:08.873574+0000 mgr.y (mgr.14116) 4 : audit [DBG] from='client.14126 v1:172.21.15.163:0/1618792147' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:56:10.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:08.877530+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' 2024-02-19T20:56:10.147 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:08.891014+0000 mon.a (mon.0) 53 : audit [DBG] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:56:10.767 INFO:teuthology.orchestra.run.smithi163.stderr:Generating ssh key... 2024-02-19T20:56:11.022 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:10.144837+0000 mgr.y (mgr.14116) 5 : audit [DBG] from='client.14128 v1:172.21.15.163:0/3219050198' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:56:11.286 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.300+0000 7fe6971c8700 1 Processor -- start 2024-02-19T20:56:11.286 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.300+0000 7fe6971c8700 1 -- start start 2024-02-19T20:56:11.286 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.301+0000 7fe6971c8700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe690100250 con 0x7fe690070b80 2024-02-19T20:56:11.286 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.301+0000 7fe694f64700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fe690070b80 0x7fe6900feb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46820/0 (socket says 172.21.15.163:46820) 2024-02-19T20:56:11.286 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.301+0000 7fe694f64700 1 -- 172.21.15.163:0/2894346050 learned_addr learned my addr 172.21.15.163:0/2894346050 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:11.286 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.301+0000 7fe68f7fe700 1 -- 172.21.15.163:0/2894346050 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201494775 0 0) 0x7fe690100250 con 0x7fe690070b80 2024-02-19T20:56:11.286 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.301+0000 7fe68f7fe700 1 -- 172.21.15.163:0/2894346050 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6800033f0 con 0x7fe690070b80 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.301+0000 7fe68f7fe700 1 -- 172.21.15.163:0/2894346050 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2081152981 0 0) 0x7fe6800033f0 con 0x7fe690070b80 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.301+0000 7fe68f7fe700 1 -- 172.21.15.163:0/2894346050 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6901013f0 con 0x7fe690070b80 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.301+0000 7fe68f7fe700 1 -- 172.21.15.163:0/2894346050 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fe684001590 con 0x7fe690070b80 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.301+0000 7fe68f7fe700 1 -- 172.21.15.163:0/2894346050 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7fe684001d10 con 0x7fe690070b80 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.301+0000 7fe68f7fe700 1 -- 172.21.15.163:0/2894346050 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fe684005750 con 0x7fe690070b80 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.302+0000 7fe6971c8700 1 -- 172.21.15.163:0/2894346050 >> v1:172.21.15.163:6789/0 conn(0x7fe690070b80 legacy=0x7fe6900feb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.302+0000 7fe6971c8700 1 -- 172.21.15.163:0/2894346050 shutdown_connections 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.302+0000 7fe6971c8700 1 -- 172.21.15.163:0/2894346050 >> 172.21.15.163:0/2894346050 conn(0x7fe6900f64d0 msgr2=0x7fe6900f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.302+0000 7fe6971c8700 1 -- 172.21.15.163:0/2894346050 shutdown_connections 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.302+0000 7fe6971c8700 1 -- 172.21.15.163:0/2894346050 wait complete. 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.302+0000 7fe6971c8700 1 librados: starting msgr at 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.302+0000 7fe6971c8700 1 librados: starting objecter 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.303+0000 7fe6971c8700 1 Processor -- start 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.303+0000 7fe6971c8700 1 -- start start 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.303+0000 7fe6971c8700 1 librados: setting wanted keys 2024-02-19T20:56:11.287 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.303+0000 7fe6971c8700 1 librados: calling monclient init 2024-02-19T20:56:11.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.303+0000 7fe6971c8700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6900fc540 con 0x7fe690070b80 2024-02-19T20:56:11.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.303+0000 7fe694f64700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fe690070b80 0x7fe6900faa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46828/0 (socket says 172.21.15.163:46828) 2024-02-19T20:56:11.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.303+0000 7fe694f64700 1 -- 172.21.15.163:0/2310042235 learned_addr learned my addr 172.21.15.163:0/2310042235 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:11.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.303+0000 7fe68dffb700 10 client.?.objecter ms_handle_connect 0x7fe690070b80 2024-02-19T20:56:11.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.303+0000 7fe68dffb700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:11.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.303+0000 7fe68dffb700 1 -- 172.21.15.163:0/2310042235 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297218202 0 0) 0x7fe6900fc540 con 0x7fe690070b80 2024-02-19T20:56:11.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.303+0000 7fe68dffb700 1 -- 172.21.15.163:0/2310042235 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe680005580 con 0x7fe690070b80 2024-02-19T20:56:11.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.303+0000 7fe68dffb700 1 -- 172.21.15.163:0/2310042235 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2867186345 0 0) 0x7fe680005580 con 0x7fe690070b80 2024-02-19T20:56:11.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.303+0000 7fe68dffb700 1 -- 172.21.15.163:0/2310042235 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6900fc540 con 0x7fe690070b80 2024-02-19T20:56:11.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.303+0000 7fe68dffb700 1 -- 172.21.15.163:0/2310042235 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fe684000ff0 con 0x7fe690070b80 2024-02-19T20:56:11.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.304+0000 7fe68dffb700 1 -- 172.21.15.163:0/2310042235 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2583260976 0 0) 0x7fe6900fc540 con 0x7fe690070b80 2024-02-19T20:56:11.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.304+0000 7fe68dffb700 1 -- 172.21.15.163:0/2310042235 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6900fc6d0 con 0x7fe690070b80 2024-02-19T20:56:11.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.304+0000 7fe6971c8700 1 -- 172.21.15.163:0/2310042235 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6900fc980 con 0x7fe690070b80 2024-02-19T20:56:11.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.304+0000 7fe68dffb700 1 -- 172.21.15.163:0/2310042235 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7fe6840046b0 con 0x7fe690070b80 2024-02-19T20:56:11.288 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.304+0000 7fe68dffb700 1 -- 172.21.15.163:0/2310042235 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fe684006210 con 0x7fe690070b80 2024-02-19T20:56:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.304+0000 7fe6971c8700 10 client.14130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.304+0000 7fe6971c8700 1 -- 172.21.15.163:0/2310042235 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6900fb440 con 0x7fe690070b80 2024-02-19T20:56:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.304+0000 7fe6971c8700 1 librados: init done 2024-02-19T20:56:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.304+0000 7fe68dffb700 1 -- 172.21.15.163:0/2310042235 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 532658122 0 0) 0x7fe68400faf0 con 0x7fe690070b80 2024-02-19T20:56:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.305+0000 7fe68dffb700 1 -- 172.21.15.163:0/2310042235 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 1409061398 0 0) 0x7fe68403eaf0 con 0x7fe690070b80 2024-02-19T20:56:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.305+0000 7fe68dffb700 10 client.14130.objecter ms_dispatch 0x7fe690001550 osd_map(2..2 src has 1..2) v4 2024-02-19T20:56:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.305+0000 7fe68dffb700 3 client.14130.objecter handle_osd_map got epochs [2,2] > 0 2024-02-19T20:56:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.305+0000 7fe68dffb700 3 client.14130.objecter handle_osd_map decoding full epoch 2 2024-02-19T20:56:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.305+0000 7fe68dffb700 20 client.14130.objecter dump_active .. 0 homeless 2024-02-19T20:56:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.305+0000 7fe68dffb700 10 client.14130.objecter ms_handle_connect 0x7fe680035280 2024-02-19T20:56:11.289 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.305+0000 7fe6971c8700 1 -- 172.21.15.163:0/2310042235 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe69005a310 con 0x7fe690070b80 2024-02-19T20:56:11.290 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.309+0000 7fe68dffb700 1 -- 172.21.15.163:0/2310042235 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe684014b00 con 0x7fe690070b80 2024-02-19T20:56:11.455 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.474+0000 7fe6971c8700 1 -- 172.21.15.163:0/2310042235 --> v1:172.21.15.163:6800/2731225015 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6900fb6f0 con 0x7fe680035280 2024-02-19T20:56:11.705 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.724+0000 7fe68dffb700 1 -- 172.21.15.163:0/2310042235 <== mgr.14116 v1:172.21.15.163:6800/2731225015 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7fe6900fb6f0 con 0x7fe680035280 2024-02-19T20:56:11.707 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.726+0000 7fe6971c8700 10 librados: watch_flush enter 2024-02-19T20:56:11.708 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.726+0000 7fe6971c8700 10 librados: watch_flush exit 2024-02-19T20:56:11.708 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.726+0000 7fe6971c8700 20 client.14130.objecter shutdown clearing up homeless session... 2024-02-19T20:56:11.708 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.726+0000 7fe6971c8700 10 client.14130.objecter successfully canceled tick 2024-02-19T20:56:11.708 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.726+0000 7fe6971c8700 1 -- 172.21.15.163:0/2310042235 >> v1:172.21.15.163:6800/2731225015 conn(0x7fe680035280 legacy=0x7fe680037740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:11.708 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.726+0000 7fe6971c8700 1 -- 172.21.15.163:0/2310042235 >> v1:172.21.15.163:6789/0 conn(0x7fe690070b80 legacy=0x7fe6900faa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:11.708 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.727+0000 7fe6971c8700 1 -- 172.21.15.163:0/2310042235 shutdown_connections 2024-02-19T20:56:11.708 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.727+0000 7fe6971c8700 1 -- 172.21.15.163:0/2310042235 >> 172.21.15.163:0/2310042235 conn(0x7fe6900f64d0 msgr2=0x7fe6900fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:11.708 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.727+0000 7fe6971c8700 1 -- 172.21.15.163:0/2310042235 shutdown_connections 2024-02-19T20:56:11.708 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.727+0000 7fe6971c8700 1 -- 172.21.15.163:0/2310042235 wait complete. 2024-02-19T20:56:11.708 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:11.727+0000 7fe6971c8700 1 librados: shutdown 2024-02-19T20:56:11.977 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: Generating public/private rsa key pair. 2024-02-19T20:56:11.978 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: Your identification has been saved in /tmp/tmpb3dip6i9/key. 2024-02-19T20:56:11.978 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: Your public key has been saved in /tmp/tmpb3dip6i9/key.pub. 2024-02-19T20:56:11.978 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: The key fingerprint is: 2024-02-19T20:56:11.978 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: SHA256:lOSK3MJNNrlXeRHBeri8i0INlFY9T7h0Cgr9FlGwGCQ ceph-2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:56:11.978 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: The key's randomart image is: 2024-02-19T20:56:11.978 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: +---[RSA 3072]----+ 2024-02-19T20:56:11.978 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: | Eooo==.o+o | 2024-02-19T20:56:11.978 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: | ..=*oo=.+. | 2024-02-19T20:56:11.978 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: | +*o=+oX. | 2024-02-19T20:56:11.978 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: | o *o=o.=.o | 2024-02-19T20:56:11.978 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: | = =+S. o | 2024-02-19T20:56:11.978 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: | .... o | 2024-02-19T20:56:11.978 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: | . . | 2024-02-19T20:56:11.979 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: | . .. | 2024-02-19T20:56:11.979 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: | .. .. | 2024-02-19T20:56:11.979 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: +----[SHA256]-----+ 2024-02-19T20:56:12.816 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.832+0000 7f8b3f2f8700 1 Processor -- start 2024-02-19T20:56:12.816 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.833+0000 7f8b3f2f8700 1 -- start start 2024-02-19T20:56:12.816 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.833+0000 7f8b3f2f8700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8b380fc8c0 con 0x7f8b38070be0 2024-02-19T20:56:12.816 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.833+0000 7f8b3d094700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8b38070be0 0x7f8b38100ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46836/0 (socket says 172.21.15.163:46836) 2024-02-19T20:56:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.833+0000 7f8b3d094700 1 -- 172.21.15.163:0/1855328913 learned_addr learned my addr 172.21.15.163:0/1855328913 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.833+0000 7f8b2ffff700 1 -- 172.21.15.163:0/1855328913 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196566343 0 0) 0x7f8b380fc8c0 con 0x7f8b38070be0 2024-02-19T20:56:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.833+0000 7f8b2ffff700 1 -- 172.21.15.163:0/1855328913 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b240033f0 con 0x7f8b38070be0 2024-02-19T20:56:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.833+0000 7f8b2ffff700 1 -- 172.21.15.163:0/1855328913 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1121027364 0 0) 0x7f8b240033f0 con 0x7f8b38070be0 2024-02-19T20:56:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.833+0000 7f8b2ffff700 1 -- 172.21.15.163:0/1855328913 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b38103600 con 0x7f8b38070be0 2024-02-19T20:56:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.834+0000 7f8b2ffff700 1 -- 172.21.15.163:0/1855328913 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f8b28001590 con 0x7f8b38070be0 2024-02-19T20:56:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.834+0000 7f8b2ffff700 1 -- 172.21.15.163:0/1855328913 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f8b28001c40 con 0x7f8b38070be0 2024-02-19T20:56:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.834+0000 7f8b2ffff700 1 -- 172.21.15.163:0/1855328913 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f8b28005610 con 0x7f8b38070be0 2024-02-19T20:56:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.834+0000 7f8b3f2f8700 1 -- 172.21.15.163:0/1855328913 >> v1:172.21.15.163:6789/0 conn(0x7f8b38070be0 legacy=0x7f8b38100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.834+0000 7f8b3f2f8700 1 -- 172.21.15.163:0/1855328913 shutdown_connections 2024-02-19T20:56:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.834+0000 7f8b3f2f8700 1 -- 172.21.15.163:0/1855328913 >> 172.21.15.163:0/1855328913 conn(0x7f8b380f64d0 msgr2=0x7f8b380f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.834+0000 7f8b3f2f8700 1 -- 172.21.15.163:0/1855328913 shutdown_connections 2024-02-19T20:56:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.834+0000 7f8b3f2f8700 1 -- 172.21.15.163:0/1855328913 wait complete. 2024-02-19T20:56:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.835+0000 7f8b3f2f8700 1 librados: starting msgr at 2024-02-19T20:56:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.835+0000 7f8b3f2f8700 1 librados: starting objecter 2024-02-19T20:56:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.835+0000 7f8b3f2f8700 1 Processor -- start 2024-02-19T20:56:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.835+0000 7f8b3f2f8700 1 -- start start 2024-02-19T20:56:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.835+0000 7f8b3f2f8700 1 librados: setting wanted keys 2024-02-19T20:56:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.835+0000 7f8b3f2f8700 1 librados: calling monclient init 2024-02-19T20:56:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.835+0000 7f8b3f2f8700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8b3810d130 con 0x7f8b38070be0 2024-02-19T20:56:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.835+0000 7f8b3d094700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8b38070be0 0x7f8b3810ca20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46844/0 (socket says 172.21.15.163:46844) 2024-02-19T20:56:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.835+0000 7f8b3d094700 1 -- 172.21.15.163:0/1136174655 learned_addr learned my addr 172.21.15.163:0/1136174655 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.835+0000 7f8b2e7fc700 10 client.?.objecter ms_handle_connect 0x7f8b38070be0 2024-02-19T20:56:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.835+0000 7f8b2e7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.836+0000 7f8b2e7fc700 1 -- 172.21.15.163:0/1136174655 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232646596 0 0) 0x7f8b3810d130 con 0x7f8b38070be0 2024-02-19T20:56:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.836+0000 7f8b2e7fc700 1 -- 172.21.15.163:0/1136174655 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b240055d0 con 0x7f8b38070be0 2024-02-19T20:56:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.836+0000 7f8b2e7fc700 1 -- 172.21.15.163:0/1136174655 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1269083098 0 0) 0x7f8b240055d0 con 0x7f8b38070be0 2024-02-19T20:56:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.836+0000 7f8b2e7fc700 1 -- 172.21.15.163:0/1136174655 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b3810d130 con 0x7f8b38070be0 2024-02-19T20:56:12.820 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.836+0000 7f8b2e7fc700 1 -- 172.21.15.163:0/1136174655 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f8b28000ff0 con 0x7f8b38070be0 2024-02-19T20:56:12.820 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.836+0000 7f8b2e7fc700 1 -- 172.21.15.163:0/1136174655 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 44439391 0 0) 0x7f8b3810d130 con 0x7f8b38070be0 2024-02-19T20:56:12.820 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.836+0000 7f8b2e7fc700 1 -- 172.21.15.163:0/1136174655 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3810d2c0 con 0x7f8b38070be0 2024-02-19T20:56:12.820 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.836+0000 7f8b3f2f8700 1 -- 172.21.15.163:0/1136174655 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3810d570 con 0x7f8b38070be0 2024-02-19T20:56:12.820 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.836+0000 7f8b3f2f8700 10 client.14132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:12.820 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.836+0000 7f8b3f2f8700 1 -- 172.21.15.163:0/1136174655 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b38111260 con 0x7f8b38070be0 2024-02-19T20:56:12.820 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.836+0000 7f8b3f2f8700 1 librados: init done 2024-02-19T20:56:12.820 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.836+0000 7f8b2e7fc700 1 -- 172.21.15.163:0/1136174655 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f8b28004830 con 0x7f8b38070be0 2024-02-19T20:56:12.820 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.836+0000 7f8b2e7fc700 1 -- 172.21.15.163:0/1136174655 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f8b280060f0 con 0x7f8b38070be0 2024-02-19T20:56:12.820 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.837+0000 7f8b2e7fc700 1 -- 172.21.15.163:0/1136174655 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 532658122 0 0) 0x7f8b2800f910 con 0x7f8b38070be0 2024-02-19T20:56:12.820 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.837+0000 7f8b2e7fc700 1 -- 172.21.15.163:0/1136174655 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 1409061398 0 0) 0x7f8b2803faf0 con 0x7f8b38070be0 2024-02-19T20:56:12.820 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.837+0000 7f8b2e7fc700 10 client.14132.objecter ms_dispatch 0x7f8b38001550 osd_map(2..2 src has 1..2) v4 2024-02-19T20:56:12.820 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.837+0000 7f8b2e7fc700 3 client.14132.objecter handle_osd_map got epochs [2,2] > 0 2024-02-19T20:56:12.821 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.837+0000 7f8b2e7fc700 3 client.14132.objecter handle_osd_map decoding full epoch 2 2024-02-19T20:56:12.821 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.837+0000 7f8b2e7fc700 20 client.14132.objecter dump_active .. 0 homeless 2024-02-19T20:56:12.821 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.837+0000 7f8b3f2f8700 1 -- 172.21.15.163:0/1136174655 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b1c004f50 con 0x7f8b38070be0 2024-02-19T20:56:12.821 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.838+0000 7f8b2e7fc700 10 client.14132.objecter ms_handle_connect 0x7f8b240352d0 2024-02-19T20:56:12.822 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:12.841+0000 7f8b2e7fc700 1 -- 172.21.15.163:0/1136174655 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8b28011030 con 0x7f8b38070be0 2024-02-19T20:56:12.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:11.476405+0000 mgr.y (mgr.14116) 6 : audit [DBG] from='client.14130 v1:172.21.15.163:0/2310042235' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:56:12.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:56:11.476614+0000 mgr.y (mgr.14116) 7 : cephadm [INF] Generating ssh key... 2024-02-19T20:56:12.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:11.720578+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' 2024-02-19T20:56:12.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:11.725368+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' 2024-02-19T20:56:12.987 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:13.006+0000 7f8b3f2f8700 1 -- 172.21.15.163:0/1136174655 --> v1:172.21.15.163:6800/2731225015 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b1c000bc0 con 0x7f8b240352d0 2024-02-19T20:56:12.987 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:13.006+0000 7f8b2e7fc700 1 -- 172.21.15.163:0/1136174655 <== mgr.14116 v1:172.21.15.163:6800/2731225015 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (unknown 0 0 1588896316) 0x7f8b1c000bc0 con 0x7f8b240352d0 2024-02-19T20:56:12.987 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj5Y6sK17FSD2N7lErQNP+TrvdJgPU9svRxd1S9WyX7YBkoiRT0MG1DPRsYsds1jwTe0tQXSVxTRv3X0UNqYrT3f7ByC+i3lGjUSZknKBVU9RCnWGMhMZznTYZ9U34liEBFUEqOMjM79po5xADfbw7q4vPDn3a+gatAN7IZNaWcgzYSaGFcWZ3XanS5b7IsrFl2lDipjglM7ONqoiBOllgDzsobDq93h7W3CGh2XVaHGairvU4k+h7kSVENJHA4tWJjpQ24I/oFHtct5PLzhy/1Qg3jegnoaiHqZIGjkhHAlofSSuQZ6rCbhmzKEH8xS8PalutMuyYBys117yYwg3sPEMPLZyEjItNIP2slQmA0iQzZRPN27rUyf6y7aHDU8eW25PGyMS2KnR7E28KuzY/gAIkUnrreeB4zCcDPlJCnoJYcPQ0e1Ofo49AheJC1z2tuvRP+idzLplVo+wl5V8oCc5xE4KrOJI9u58DQV65fa70j4wdT1mEptVI+AqoAlc= ceph-2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:56:12.989 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:13.008+0000 7f8b3f2f8700 10 librados: watch_flush enter 2024-02-19T20:56:12.989 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:13.008+0000 7f8b3f2f8700 10 librados: watch_flush exit 2024-02-19T20:56:12.989 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:13.008+0000 7f8b3f2f8700 20 client.14132.objecter shutdown clearing up homeless session... 2024-02-19T20:56:12.990 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:13.008+0000 7f8b3f2f8700 10 client.14132.objecter successfully canceled tick 2024-02-19T20:56:12.990 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:13.008+0000 7f8b3f2f8700 1 -- 172.21.15.163:0/1136174655 >> v1:172.21.15.163:6800/2731225015 conn(0x7f8b240352d0 legacy=0x7f8b24037790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:12.990 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:13.008+0000 7f8b3f2f8700 1 -- 172.21.15.163:0/1136174655 >> v1:172.21.15.163:6789/0 conn(0x7f8b38070be0 legacy=0x7f8b3810ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:12.990 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:13.008+0000 7f8b3f2f8700 1 -- 172.21.15.163:0/1136174655 shutdown_connections 2024-02-19T20:56:12.990 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:13.008+0000 7f8b3f2f8700 1 -- 172.21.15.163:0/1136174655 >> 172.21.15.163:0/1136174655 conn(0x7f8b380f64d0 msgr2=0x7f8b380f7120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:12.990 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:13.008+0000 7f8b3f2f8700 1 -- 172.21.15.163:0/1136174655 shutdown_connections 2024-02-19T20:56:12.990 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:13.008+0000 7f8b3f2f8700 1 -- 172.21.15.163:0/1136174655 wait complete. 2024-02-19T20:56:12.990 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:13.009+0000 7f8b3f2f8700 1 librados: shutdown 2024-02-19T20:56:13.554 INFO:teuthology.orchestra.run.smithi163.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-19T20:56:13.554 INFO:teuthology.orchestra.run.smithi163.stderr:Adding key to root@localhost authorized_keys... 2024-02-19T20:56:13.555 INFO:teuthology.orchestra.run.smithi163.stderr:Adding host smithi163... 2024-02-19T20:56:14.087 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.101+0000 7f69c976d700 1 Processor -- start 2024-02-19T20:56:14.087 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.101+0000 7f69c976d700 1 -- start start 2024-02-19T20:56:14.087 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.101+0000 7f69c976d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f69c40fb8d0 con 0x7f69c40fada0 2024-02-19T20:56:14.087 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.102+0000 7f69c2ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f69c40fada0 0x7f69c40fb1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46856/0 (socket says 172.21.15.163:46856) 2024-02-19T20:56:14.087 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.102+0000 7f69c2ffd700 1 -- 172.21.15.163:0/914553196 learned_addr learned my addr 172.21.15.163:0/914553196 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:14.087 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.102+0000 7f69c1ffb700 1 -- 172.21.15.163:0/914553196 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956950253 0 0) 0x7f69b8001110 con 0x7f69c40fada0 2024-02-19T20:56:14.087 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.102+0000 7f69c1ffb700 1 -- 172.21.15.163:0/914553196 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69ac0033f0 con 0x7f69c40fada0 2024-02-19T20:56:14.087 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.102+0000 7f69c1ffb700 1 -- 172.21.15.163:0/914553196 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 600434766 0 0) 0x7f69ac0033f0 con 0x7f69c40fada0 2024-02-19T20:56:14.087 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.102+0000 7f69c1ffb700 1 -- 172.21.15.163:0/914553196 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c41001e0 con 0x7f69c40fada0 2024-02-19T20:56:14.087 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.102+0000 7f69c1ffb700 1 -- 172.21.15.163:0/914553196 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f69b8001720 con 0x7f69c40fada0 2024-02-19T20:56:14.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.102+0000 7f69c1ffb700 1 -- 172.21.15.163:0/914553196 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f69b8004770 con 0x7f69c40fada0 2024-02-19T20:56:14.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.102+0000 7f69c1ffb700 1 -- 172.21.15.163:0/914553196 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f69b80058f0 con 0x7f69c40fada0 2024-02-19T20:56:14.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.103+0000 7f69c976d700 1 -- 172.21.15.163:0/914553196 >> v1:172.21.15.163:6789/0 conn(0x7f69c40fada0 legacy=0x7f69c40fb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:14.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.103+0000 7f69c976d700 1 -- 172.21.15.163:0/914553196 shutdown_connections 2024-02-19T20:56:14.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.103+0000 7f69c976d700 1 -- 172.21.15.163:0/914553196 >> 172.21.15.163:0/914553196 conn(0x7f69c40f6500 msgr2=0x7f69c40f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:14.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.103+0000 7f69c976d700 1 -- 172.21.15.163:0/914553196 shutdown_connections 2024-02-19T20:56:14.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.103+0000 7f69c976d700 1 -- 172.21.15.163:0/914553196 wait complete. 2024-02-19T20:56:14.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.103+0000 7f69c976d700 1 librados: starting msgr at 2024-02-19T20:56:14.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.103+0000 7f69c976d700 1 librados: starting objecter 2024-02-19T20:56:14.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.103+0000 7f69c976d700 1 Processor -- start 2024-02-19T20:56:14.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.104+0000 7f69c976d700 1 -- start start 2024-02-19T20:56:14.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.104+0000 7f69c976d700 1 librados: setting wanted keys 2024-02-19T20:56:14.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.104+0000 7f69c976d700 1 librados: calling monclient init 2024-02-19T20:56:14.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.104+0000 7f69c976d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f69c410b460 con 0x7f69c40fada0 2024-02-19T20:56:14.088 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.104+0000 7f69c2ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f69c40fada0 0x7f69c410fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46868/0 (socket says 172.21.15.163:46868) 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.104+0000 7f69c2ffd700 1 -- 172.21.15.163:0/798257816 learned_addr learned my addr 172.21.15.163:0/798257816 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.104+0000 7f69abfff700 10 client.?.objecter ms_handle_connect 0x7f69c40fada0 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.104+0000 7f69abfff700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.104+0000 7f69abfff700 1 -- 172.21.15.163:0/798257816 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947596340 0 0) 0x7f69c410b460 con 0x7f69c40fada0 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.104+0000 7f69abfff700 1 -- 172.21.15.163:0/798257816 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69ac005580 con 0x7f69c40fada0 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.104+0000 7f69abfff700 1 -- 172.21.15.163:0/798257816 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1278377833 0 0) 0x7f69ac005580 con 0x7f69c40fada0 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.104+0000 7f69abfff700 1 -- 172.21.15.163:0/798257816 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69c410b460 con 0x7f69c40fada0 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.104+0000 7f69abfff700 1 -- 172.21.15.163:0/798257816 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f69b8004d00 con 0x7f69c40fada0 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.104+0000 7f69abfff700 1 -- 172.21.15.163:0/798257816 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3102544328 0 0) 0x7f69c410b460 con 0x7f69c40fada0 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.105+0000 7f69abfff700 1 -- 172.21.15.163:0/798257816 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c4070c30 con 0x7f69c40fada0 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.105+0000 7f69c976d700 1 -- 172.21.15.163:0/798257816 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c4070ee0 con 0x7f69c40fada0 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.105+0000 7f69c976d700 10 client.14134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.105+0000 7f69c976d700 1 -- 172.21.15.163:0/798257816 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c41113d0 con 0x7f69c40fada0 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.105+0000 7f69c976d700 1 librados: init done 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.105+0000 7f69abfff700 1 -- 172.21.15.163:0/798257816 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f69b8001330 con 0x7f69c40fada0 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.105+0000 7f69abfff700 1 -- 172.21.15.163:0/798257816 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f69b8006170 con 0x7f69c40fada0 2024-02-19T20:56:14.089 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.106+0000 7f69abfff700 1 -- 172.21.15.163:0/798257816 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 532658122 0 0) 0x7f69b800fa10 con 0x7f69c40fada0 2024-02-19T20:56:14.090 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.106+0000 7f69c976d700 1 -- 172.21.15.163:0/798257816 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b0004f50 con 0x7f69c40fada0 2024-02-19T20:56:14.090 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.106+0000 7f69abfff700 1 -- 172.21.15.163:0/798257816 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 1409061398 0 0) 0x7f69b803f910 con 0x7f69c40fada0 2024-02-19T20:56:14.090 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.106+0000 7f69abfff700 10 client.14134.objecter ms_dispatch 0x7f69c4001550 osd_map(2..2 src has 1..2) v4 2024-02-19T20:56:14.090 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.106+0000 7f69abfff700 3 client.14134.objecter handle_osd_map got epochs [2,2] > 0 2024-02-19T20:56:14.090 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.106+0000 7f69abfff700 3 client.14134.objecter handle_osd_map decoding full epoch 2 2024-02-19T20:56:14.090 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.106+0000 7f69abfff700 20 client.14134.objecter dump_active .. 0 homeless 2024-02-19T20:56:14.090 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.106+0000 7f69abfff700 10 client.14134.objecter ms_handle_connect 0x7f69ac035370 2024-02-19T20:56:14.090 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.110+0000 7f69abfff700 1 -- 172.21.15.163:0/798257816 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f69b8014860 con 0x7f69c40fada0 2024-02-19T20:56:14.257 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.276+0000 7f69c976d700 1 -- 172.21.15.163:0/798257816 --> v1:172.21.15.163:6800/2731225015 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi163", "addr": "172.21.15.163", "target": ["mon-mgr", ""]}) v1 -- 0x7f69b0000bc0 con 0x7f69ac035370 2024-02-19T20:56:14.278 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:13.007939+0000 mgr.y (mgr.14116) 8 : audit [DBG] from='client.14132 v1:172.21.15.163:0/1136174655' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:56:14.727 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [80B blob data] 2024-02-19T20:56:14.877 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: Added host 'smithi163' with addr '172.21.15.163' 2024-02-19T20:56:14.877 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.895+0000 7f69abfff700 1 -- 172.21.15.163:0/798257816 <== mgr.14116 v1:172.21.15.163:6800/2731225015 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (unknown 0 0 3409930226) 0x7f69b0000bc0 con 0x7f69ac035370 2024-02-19T20:56:14.879 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.898+0000 7f69c976d700 10 librados: watch_flush enter 2024-02-19T20:56:14.879 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.898+0000 7f69c976d700 10 librados: watch_flush exit 2024-02-19T20:56:14.879 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.898+0000 7f69c976d700 20 client.14134.objecter shutdown clearing up homeless session... 2024-02-19T20:56:14.879 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.898+0000 7f69c976d700 10 client.14134.objecter successfully canceled tick 2024-02-19T20:56:14.879 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.898+0000 7f69c976d700 1 -- 172.21.15.163:0/798257816 >> v1:172.21.15.163:6800/2731225015 conn(0x7f69ac035370 legacy=0x7f69ac037810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:14.879 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.898+0000 7f69c976d700 1 -- 172.21.15.163:0/798257816 >> v1:172.21.15.163:6789/0 conn(0x7f69c40fada0 legacy=0x7f69c410fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:14.879 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.898+0000 7f69c976d700 1 -- 172.21.15.163:0/798257816 shutdown_connections 2024-02-19T20:56:14.879 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.898+0000 7f69c976d700 1 -- 172.21.15.163:0/798257816 >> 172.21.15.163:0/798257816 conn(0x7f69c40f6500 msgr2=0x7f69c40f7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:14.880 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.899+0000 7f69c976d700 1 -- 172.21.15.163:0/798257816 shutdown_connections 2024-02-19T20:56:14.880 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.899+0000 7f69c976d700 1 -- 172.21.15.163:0/798257816 wait complete. 2024-02-19T20:56:14.880 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:14.899+0000 7f69c976d700 1 librados: shutdown 2024-02-19T20:56:15.427 INFO:teuthology.orchestra.run.smithi163.stderr:Deploying unmanaged mon service... 2024-02-19T20:56:16.189 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:14.278507+0000 mgr.y (mgr.14116) 9 : audit [DBG] from='client.14134 v1:172.21.15.163:0/798257816' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi163", "addr": "172.21.15.163", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:56:16.190 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:14.896588+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' 2024-02-19T20:56:16.190 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:56:14.896866+0000 mgr.y (mgr.14116) 10 : cephadm [INF] Added host smithi163 2024-02-19T20:56:16.190 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:14.917784+0000 mon.a (mon.0) 57 : audit [DBG] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:56:16.291 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.118+0000 7f7878743700 1 Processor -- start 2024-02-19T20:56:16.291 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.118+0000 7f7878743700 1 -- start start 2024-02-19T20:56:16.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.119+0000 7f7878743700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f78701001c0 con 0x7f7870070c20 2024-02-19T20:56:16.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.119+0000 7f78764df700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f7870070c20 0x7f78700feaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:59202/0 (socket says 172.21.15.163:59202) 2024-02-19T20:56:16.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.119+0000 7f78764df700 1 -- 172.21.15.163:0/2754008020 learned_addr learned my addr 172.21.15.163:0/2754008020 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:16.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.119+0000 7f78754dd700 1 -- 172.21.15.163:0/2754008020 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912952541 0 0) 0x7f78701001c0 con 0x7f7870070c20 2024-02-19T20:56:16.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.119+0000 7f78754dd700 1 -- 172.21.15.163:0/2754008020 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f785c0033f0 con 0x7f7870070c20 2024-02-19T20:56:16.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.119+0000 7f78754dd700 1 -- 172.21.15.163:0/2754008020 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2652319342 0 0) 0x7f785c0033f0 con 0x7f7870070c20 2024-02-19T20:56:16.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.119+0000 7f78754dd700 1 -- 172.21.15.163:0/2754008020 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7870101360 con 0x7f7870070c20 2024-02-19T20:56:16.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.119+0000 7f78754dd700 1 -- 172.21.15.163:0/2754008020 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f7860001590 con 0x7f7870070c20 2024-02-19T20:56:16.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.120+0000 7f78754dd700 1 -- 172.21.15.163:0/2754008020 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f7860001d10 con 0x7f7870070c20 2024-02-19T20:56:16.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.120+0000 7f78754dd700 1 -- 172.21.15.163:0/2754008020 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f78600056e0 con 0x7f7870070c20 2024-02-19T20:56:16.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.120+0000 7f7878743700 1 -- 172.21.15.163:0/2754008020 >> v1:172.21.15.163:6789/0 conn(0x7f7870070c20 legacy=0x7f78700feaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:16.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.120+0000 7f7878743700 1 -- 172.21.15.163:0/2754008020 shutdown_connections 2024-02-19T20:56:16.292 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.120+0000 7f7878743700 1 -- 172.21.15.163:0/2754008020 >> 172.21.15.163:0/2754008020 conn(0x7f78700f6500 msgr2=0x7f78700f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:16.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.120+0000 7f7878743700 1 -- 172.21.15.163:0/2754008020 shutdown_connections 2024-02-19T20:56:16.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.120+0000 7f7878743700 1 -- 172.21.15.163:0/2754008020 wait complete. 2024-02-19T20:56:16.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f7878743700 1 librados: starting msgr at 2024-02-19T20:56:16.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f7878743700 1 librados: starting objecter 2024-02-19T20:56:16.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f7878743700 1 Processor -- start 2024-02-19T20:56:16.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f7878743700 1 -- start start 2024-02-19T20:56:16.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f7878743700 1 librados: setting wanted keys 2024-02-19T20:56:16.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f7878743700 1 librados: calling monclient init 2024-02-19T20:56:16.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f7878743700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f787010aeb0 con 0x7f7870070c20 2024-02-19T20:56:16.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f78764df700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f7870070c20 0x7f787010a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:59204/0 (socket says 172.21.15.163:59204) 2024-02-19T20:56:16.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f78764df700 1 -- 172.21.15.163:0/3701729869 learned_addr learned my addr 172.21.15.163:0/3701729869 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:16.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f78677fe700 10 client.?.objecter ms_handle_connect 0x7f7870070c20 2024-02-19T20:56:16.293 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f78677fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:16.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f78677fe700 1 -- 172.21.15.163:0/3701729869 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747796640 0 0) 0x7f787010aeb0 con 0x7f7870070c20 2024-02-19T20:56:16.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f78677fe700 1 -- 172.21.15.163:0/3701729869 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f785c005580 con 0x7f7870070c20 2024-02-19T20:56:16.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f78677fe700 1 -- 172.21.15.163:0/3701729869 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3067619027 0 0) 0x7f785c005580 con 0x7f7870070c20 2024-02-19T20:56:16.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f78677fe700 1 -- 172.21.15.163:0/3701729869 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f787010aeb0 con 0x7f7870070c20 2024-02-19T20:56:16.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.121+0000 7f78677fe700 1 -- 172.21.15.163:0/3701729869 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f7860000fe0 con 0x7f7870070c20 2024-02-19T20:56:16.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.122+0000 7f78677fe700 1 -- 172.21.15.163:0/3701729869 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 303883673 0 0) 0x7f787010aeb0 con 0x7f7870070c20 2024-02-19T20:56:16.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.122+0000 7f78677fe700 1 -- 172.21.15.163:0/3701729869 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787010b040 con 0x7f7870070c20 2024-02-19T20:56:16.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.122+0000 7f7878743700 1 -- 172.21.15.163:0/3701729869 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f787010b2f0 con 0x7f7870070c20 2024-02-19T20:56:16.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.122+0000 7f7878743700 10 client.14136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:16.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.122+0000 7f7878743700 1 -- 172.21.15.163:0/3701729869 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f787010efe0 con 0x7f7870070c20 2024-02-19T20:56:16.294 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.122+0000 7f7878743700 1 librados: init done 2024-02-19T20:56:16.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.122+0000 7f78677fe700 1 -- 172.21.15.163:0/3701729869 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f7860001b30 con 0x7f7870070c20 2024-02-19T20:56:16.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.122+0000 7f78677fe700 1 -- 172.21.15.163:0/3701729869 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f78600061c0 con 0x7f7870070c20 2024-02-19T20:56:16.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.122+0000 7f78677fe700 1 -- 172.21.15.163:0/3701729869 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 532658122 0 0) 0x7f7860007490 con 0x7f7870070c20 2024-02-19T20:56:16.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.122+0000 7f7878743700 1 -- 172.21.15.163:0/3701729869 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7854004f50 con 0x7f7870070c20 2024-02-19T20:56:16.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.123+0000 7f78677fe700 1 -- 172.21.15.163:0/3701729869 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 1409061398 0 0) 0x7f786003ede0 con 0x7f7870070c20 2024-02-19T20:56:16.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.123+0000 7f78677fe700 10 client.14136.objecter ms_dispatch 0x7f7870001550 osd_map(2..2 src has 1..2) v4 2024-02-19T20:56:16.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.123+0000 7f78677fe700 3 client.14136.objecter handle_osd_map got epochs [2,2] > 0 2024-02-19T20:56:16.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.123+0000 7f78677fe700 3 client.14136.objecter handle_osd_map decoding full epoch 2 2024-02-19T20:56:16.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.123+0000 7f78677fe700 20 client.14136.objecter dump_active .. 0 homeless 2024-02-19T20:56:16.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.123+0000 7f78677fe700 10 client.14136.objecter ms_handle_connect 0x7f785c035230 2024-02-19T20:56:16.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.127+0000 7f78677fe700 1 -- 172.21.15.163:0/3701729869 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f7860007cb0 con 0x7f7870070c20 2024-02-19T20:56:16.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.314+0000 7f7878743700 1 -- 172.21.15.163:0/3701729869 --> v1:172.21.15.163:6800/2731225015 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7854000bc0 con 0x7f785c035230 2024-02-19T20:56:16.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.319+0000 7f78677fe700 1 -- 172.21.15.163:0/3701729869 <== mgr.14116 v1:172.21.15.163:6800/2731225015 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 3265049985) 0x7f7854000bc0 con 0x7f785c035230 2024-02-19T20:56:16.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-19T20:56:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.321+0000 7f7878743700 10 librados: watch_flush enter 2024-02-19T20:56:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.321+0000 7f7878743700 10 librados: watch_flush exit 2024-02-19T20:56:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.321+0000 7f7878743700 20 client.14136.objecter shutdown clearing up homeless session... 2024-02-19T20:56:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.321+0000 7f7878743700 10 client.14136.objecter successfully canceled tick 2024-02-19T20:56:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.321+0000 7f7878743700 1 -- 172.21.15.163:0/3701729869 >> v1:172.21.15.163:6800/2731225015 conn(0x7f785c035230 legacy=0x7f785c0376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.321+0000 7f7878743700 1 -- 172.21.15.163:0/3701729869 >> v1:172.21.15.163:6789/0 conn(0x7f7870070c20 legacy=0x7f787010a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:16.303 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.321+0000 7f7878743700 1 -- 172.21.15.163:0/3701729869 shutdown_connections 2024-02-19T20:56:16.303 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.321+0000 7f7878743700 1 -- 172.21.15.163:0/3701729869 >> 172.21.15.163:0/3701729869 conn(0x7f78700f6500 msgr2=0x7f78700fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:16.303 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.321+0000 7f7878743700 1 -- 172.21.15.163:0/3701729869 shutdown_connections 2024-02-19T20:56:16.303 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.321+0000 7f7878743700 1 -- 172.21.15.163:0/3701729869 wait complete. 2024-02-19T20:56:16.303 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:16.321+0000 7f7878743700 1 librados: shutdown 2024-02-19T20:56:17.658 INFO:teuthology.orchestra.run.smithi163.stderr:Deploying unmanaged mgr service... 2024-02-19T20:56:17.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:16.316098+0000 mgr.y (mgr.14116) 11 : audit [DBG] from='client.14136 v1:172.21.15.163:0/3701729869' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:56:17.662 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:56:16.317287+0000 mgr.y (mgr.14116) 12 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-19T20:56:17.662 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:16.320237+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' 2024-02-19T20:56:18.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.224+0000 7f8eb9ba2700 1 Processor -- start 2024-02-19T20:56:18.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.224+0000 7f8eb9ba2700 1 -- start start 2024-02-19T20:56:18.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.224+0000 7f8eb9ba2700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8eb40fb8b0 con 0x7f8eb40fad80 2024-02-19T20:56:18.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.224+0000 7f8eb37fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8eb40fad80 0x7f8eb40fb1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:59212/0 (socket says 172.21.15.163:59212) 2024-02-19T20:56:18.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.224+0000 7f8eb37fe700 1 -- 172.21.15.163:0/3905247096 learned_addr learned my addr 172.21.15.163:0/3905247096 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:18.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.224+0000 7f8eb27fc700 1 -- 172.21.15.163:0/3905247096 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767649678 0 0) 0x7f8ea4001110 con 0x7f8eb40fad80 2024-02-19T20:56:18.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.224+0000 7f8eb27fc700 1 -- 172.21.15.163:0/3905247096 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e9c0033f0 con 0x7f8eb40fad80 2024-02-19T20:56:18.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.224+0000 7f8eb27fc700 1 -- 172.21.15.163:0/3905247096 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1840146802 0 0) 0x7f8e9c0033f0 con 0x7f8eb40fad80 2024-02-19T20:56:18.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.224+0000 7f8eb27fc700 1 -- 172.21.15.163:0/3905247096 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb41001c0 con 0x7f8eb40fad80 2024-02-19T20:56:18.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.224+0000 7f8eb27fc700 1 -- 172.21.15.163:0/3905247096 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f8ea4001720 con 0x7f8eb40fad80 2024-02-19T20:56:18.444 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.225+0000 7f8eb27fc700 1 -- 172.21.15.163:0/3905247096 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f8ea4004770 con 0x7f8eb40fad80 2024-02-19T20:56:18.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.225+0000 7f8eb27fc700 1 -- 172.21.15.163:0/3905247096 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f8ea40058f0 con 0x7f8eb40fad80 2024-02-19T20:56:18.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.225+0000 7f8eb9ba2700 1 -- 172.21.15.163:0/3905247096 >> v1:172.21.15.163:6789/0 conn(0x7f8eb40fad80 legacy=0x7f8eb40fb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:18.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.225+0000 7f8eb9ba2700 1 -- 172.21.15.163:0/3905247096 shutdown_connections 2024-02-19T20:56:18.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.225+0000 7f8eb9ba2700 1 -- 172.21.15.163:0/3905247096 >> 172.21.15.163:0/3905247096 conn(0x7f8eb40f6520 msgr2=0x7f8eb40f8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:18.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.225+0000 7f8eb9ba2700 1 -- 172.21.15.163:0/3905247096 shutdown_connections 2024-02-19T20:56:18.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.225+0000 7f8eb9ba2700 1 -- 172.21.15.163:0/3905247096 wait complete. 2024-02-19T20:56:18.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.225+0000 7f8eb9ba2700 1 librados: starting msgr at 2024-02-19T20:56:18.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.225+0000 7f8eb9ba2700 1 librados: starting objecter 2024-02-19T20:56:18.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.226+0000 7f8eb9ba2700 1 Processor -- start 2024-02-19T20:56:18.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.226+0000 7f8eb9ba2700 1 -- start start 2024-02-19T20:56:18.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.226+0000 7f8eb9ba2700 1 librados: setting wanted keys 2024-02-19T20:56:18.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.226+0000 7f8eb9ba2700 1 librados: calling monclient init 2024-02-19T20:56:18.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.226+0000 7f8eb9ba2700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8eb410b430 con 0x7f8eb40fad80 2024-02-19T20:56:18.445 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.226+0000 7f8eb37fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8eb40fad80 0x7f8eb410fa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:59214/0 (socket says 172.21.15.163:59214) 2024-02-19T20:56:18.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.226+0000 7f8eb37fe700 1 -- 172.21.15.163:0/3190470308 learned_addr learned my addr 172.21.15.163:0/3190470308 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:18.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.226+0000 7f8eb0ff9700 10 client.?.objecter ms_handle_connect 0x7f8eb40fad80 2024-02-19T20:56:18.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.226+0000 7f8eb0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:18.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.226+0000 7f8eb0ff9700 1 -- 172.21.15.163:0/3190470308 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494669518 0 0) 0x7f8eb410b430 con 0x7f8eb40fad80 2024-02-19T20:56:18.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.226+0000 7f8eb0ff9700 1 -- 172.21.15.163:0/3190470308 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e9c005580 con 0x7f8eb40fad80 2024-02-19T20:56:18.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.226+0000 7f8eb0ff9700 1 -- 172.21.15.163:0/3190470308 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1959976143 0 0) 0x7f8e9c005580 con 0x7f8eb40fad80 2024-02-19T20:56:18.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.226+0000 7f8eb0ff9700 1 -- 172.21.15.163:0/3190470308 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8eb410b430 con 0x7f8eb40fad80 2024-02-19T20:56:18.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.226+0000 7f8eb0ff9700 1 -- 172.21.15.163:0/3190470308 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f8ea4004d00 con 0x7f8eb40fad80 2024-02-19T20:56:18.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.227+0000 7f8eb0ff9700 1 -- 172.21.15.163:0/3190470308 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3219254717 0 0) 0x7f8eb410b430 con 0x7f8eb40fad80 2024-02-19T20:56:18.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.227+0000 7f8eb0ff9700 1 -- 172.21.15.163:0/3190470308 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb4070c30 con 0x7f8eb40fad80 2024-02-19T20:56:18.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.227+0000 7f8eb9ba2700 1 -- 172.21.15.163:0/3190470308 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb4070ee0 con 0x7f8eb40fad80 2024-02-19T20:56:18.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.227+0000 7f8eb9ba2700 10 client.14138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:18.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.227+0000 7f8eb9ba2700 1 -- 172.21.15.163:0/3190470308 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb41113a0 con 0x7f8eb40fad80 2024-02-19T20:56:18.446 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.227+0000 7f8eb9ba2700 1 librados: init done 2024-02-19T20:56:18.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.227+0000 7f8eb0ff9700 1 -- 172.21.15.163:0/3190470308 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1049+0+0 (unknown 2011830636 0 0) 0x7f8ea40048e0 con 0x7f8eb40fad80 2024-02-19T20:56:18.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.227+0000 7f8eb0ff9700 1 -- 172.21.15.163:0/3190470308 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f8ea4006290 con 0x7f8eb40fad80 2024-02-19T20:56:18.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.227+0000 7f8eb0ff9700 1 -- 172.21.15.163:0/3190470308 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 532658122 0 0) 0x7f8ea400fab0 con 0x7f8eb40fad80 2024-02-19T20:56:18.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.228+0000 7f8eb9ba2700 1 -- 172.21.15.163:0/3190470308 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea0004f50 con 0x7f8eb40fad80 2024-02-19T20:56:18.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.228+0000 7f8eb0ff9700 1 -- 172.21.15.163:0/3190470308 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 1409061398 0 0) 0x7f8ea403eb00 con 0x7f8eb40fad80 2024-02-19T20:56:18.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.228+0000 7f8eb0ff9700 10 client.14138.objecter ms_dispatch 0x7f8eb4001550 osd_map(2..2 src has 1..2) v4 2024-02-19T20:56:18.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.228+0000 7f8eb0ff9700 3 client.14138.objecter handle_osd_map got epochs [2,2] > 0 2024-02-19T20:56:18.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.228+0000 7f8eb0ff9700 3 client.14138.objecter handle_osd_map decoding full epoch 2 2024-02-19T20:56:18.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.228+0000 7f8eb0ff9700 20 client.14138.objecter dump_active .. 0 homeless 2024-02-19T20:56:18.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.228+0000 7f8eb0ff9700 10 client.14138.objecter ms_handle_connect 0x7f8e9c035230 2024-02-19T20:56:18.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.232+0000 7f8eb0ff9700 1 -- 172.21.15.163:0/3190470308 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8ea4010ff0 con 0x7f8eb40fad80 2024-02-19T20:56:18.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.381+0000 7f8eb0ff9700 1 -- 172.21.15.163:0/3190470308 <== mon.0 v1:172.21.15.163:6789/0 10 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8ea4010ff0 con 0x7f8eb40fad80 2024-02-19T20:56:18.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.411+0000 7f8eb9ba2700 1 -- 172.21.15.163:0/3190470308 --> v1:172.21.15.163:6800/2731225015 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f8ea0000bc0 con 0x7f8e9c035230 2024-02-19T20:56:18.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.416+0000 7f8eb0ff9700 1 -- 172.21.15.163:0/3190470308 <== mgr.14116 v1:172.21.15.163:6800/2731225015 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 325935098) 0x7f8ea0000bc0 con 0x7f8e9c035230 2024-02-19T20:56:18.447 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.418+0000 7f8eb9ba2700 10 librados: watch_flush enter 2024-02-19T20:56:18.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.418+0000 7f8eb9ba2700 10 librados: watch_flush exit 2024-02-19T20:56:18.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.418+0000 7f8eb9ba2700 20 client.14138.objecter shutdown clearing up homeless session... 2024-02-19T20:56:18.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.418+0000 7f8eb9ba2700 10 client.14138.objecter successfully canceled tick 2024-02-19T20:56:18.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.418+0000 7f8eb9ba2700 1 -- 172.21.15.163:0/3190470308 >> v1:172.21.15.163:6800/2731225015 conn(0x7f8e9c035230 legacy=0x7f8e9c0376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:18.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.418+0000 7f8eb9ba2700 1 -- 172.21.15.163:0/3190470308 >> v1:172.21.15.163:6789/0 conn(0x7f8eb40fad80 legacy=0x7f8eb410fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:18.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.418+0000 7f8eb9ba2700 1 -- 172.21.15.163:0/3190470308 shutdown_connections 2024-02-19T20:56:18.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.418+0000 7f8eb9ba2700 1 -- 172.21.15.163:0/3190470308 >> 172.21.15.163:0/3190470308 conn(0x7f8eb40f6520 msgr2=0x7f8eb40f7120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:18.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.418+0000 7f8eb9ba2700 1 -- 172.21.15.163:0/3190470308 shutdown_connections 2024-02-19T20:56:18.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.418+0000 7f8eb9ba2700 1 -- 172.21.15.163:0/3190470308 wait complete. 2024-02-19T20:56:18.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:18.418+0000 7f8eb9ba2700 1 librados: shutdown 2024-02-19T20:56:18.448 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-19T20:56:19.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.593+0000 7fca0d3dd700 1 Processor -- start 2024-02-19T20:56:19.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.593+0000 7fca0d3dd700 1 -- start start 2024-02-19T20:56:19.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.593+0000 7fca0d3dd700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fca080fb8c0 con 0x7fca080fad90 2024-02-19T20:56:19.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.593+0000 7fca06ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fca080fad90 0x7fca080fb1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:59224/0 (socket says 172.21.15.163:59224) 2024-02-19T20:56:19.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.593+0000 7fca06ffd700 1 -- 172.21.15.163:0/3435793319 learned_addr learned my addr 172.21.15.163:0/3435793319 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:19.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.593+0000 7fca05ffb700 1 -- 172.21.15.163:0/3435793319 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293602418 0 0) 0x7fc9f8001110 con 0x7fca080fad90 2024-02-19T20:56:19.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.594+0000 7fca05ffb700 1 -- 172.21.15.163:0/3435793319 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f00033f0 con 0x7fca080fad90 2024-02-19T20:56:19.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.594+0000 7fca05ffb700 1 -- 172.21.15.163:0/3435793319 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3708851453 0 0) 0x7fc9f00033f0 con 0x7fca080fad90 2024-02-19T20:56:19.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.594+0000 7fca05ffb700 1 -- 172.21.15.163:0/3435793319 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca081001d0 con 0x7fca080fad90 2024-02-19T20:56:19.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.594+0000 7fca05ffb700 1 -- 172.21.15.163:0/3435793319 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc9f8001720 con 0x7fca080fad90 2024-02-19T20:56:19.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.594+0000 7fca05ffb700 1 -- 172.21.15.163:0/3435793319 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fc9f8004770 con 0x7fca080fad90 2024-02-19T20:56:19.575 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.594+0000 7fca05ffb700 1 -- 172.21.15.163:0/3435793319 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc9f8005920 con 0x7fca080fad90 2024-02-19T20:56:19.576 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.594+0000 7fca0d3dd700 1 -- 172.21.15.163:0/3435793319 >> v1:172.21.15.163:6789/0 conn(0x7fca080fad90 legacy=0x7fca080fb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:19.576 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.595+0000 7fca0d3dd700 1 -- 172.21.15.163:0/3435793319 shutdown_connections 2024-02-19T20:56:19.576 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.595+0000 7fca0d3dd700 1 -- 172.21.15.163:0/3435793319 >> 172.21.15.163:0/3435793319 conn(0x7fca080f64d0 msgr2=0x7fca080f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:19.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.595+0000 7fca0d3dd700 1 -- 172.21.15.163:0/3435793319 shutdown_connections 2024-02-19T20:56:19.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.595+0000 7fca0d3dd700 1 -- 172.21.15.163:0/3435793319 wait complete. 2024-02-19T20:56:19.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.595+0000 7fca0d3dd700 1 librados: starting msgr at 2024-02-19T20:56:19.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.595+0000 7fca0d3dd700 1 librados: starting objecter 2024-02-19T20:56:19.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.595+0000 7fca0d3dd700 1 Processor -- start 2024-02-19T20:56:19.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.596+0000 7fca0d3dd700 1 -- start start 2024-02-19T20:56:19.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.596+0000 7fca0d3dd700 1 librados: setting wanted keys 2024-02-19T20:56:19.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.596+0000 7fca0d3dd700 1 librados: calling monclient init 2024-02-19T20:56:19.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.596+0000 7fca0d3dd700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fca08070c30 con 0x7fca080fad90 2024-02-19T20:56:19.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.596+0000 7fca06ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fca080fad90 0x7fca0810d680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:59236/0 (socket says 172.21.15.163:59236) 2024-02-19T20:56:19.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.596+0000 7fca06ffd700 1 -- 172.21.15.163:0/1141863473 learned_addr learned my addr 172.21.15.163:0/1141863473 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:19.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.596+0000 7fc9effff700 10 client.?.objecter ms_handle_connect 0x7fca080fad90 2024-02-19T20:56:19.579 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.596+0000 7fc9effff700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:19.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.596+0000 7fc9effff700 1 -- 172.21.15.163:0/1141863473 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994489716 0 0) 0x7fca08070c30 con 0x7fca080fad90 2024-02-19T20:56:19.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.596+0000 7fc9effff700 1 -- 172.21.15.163:0/1141863473 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f0005580 con 0x7fca080fad90 2024-02-19T20:56:19.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.596+0000 7fc9effff700 1 -- 172.21.15.163:0/1141863473 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3292180276 0 0) 0x7fc9f0005580 con 0x7fca080fad90 2024-02-19T20:56:19.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.596+0000 7fc9effff700 1 -- 172.21.15.163:0/1141863473 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca08070c30 con 0x7fca080fad90 2024-02-19T20:56:19.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.596+0000 7fc9effff700 1 -- 172.21.15.163:0/1141863473 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc9f8000fe0 con 0x7fca080fad90 2024-02-19T20:56:19.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.597+0000 7fc9effff700 1 -- 172.21.15.163:0/1141863473 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 83944239 0 0) 0x7fca08070c30 con 0x7fca080fad90 2024-02-19T20:56:19.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.597+0000 7fc9effff700 1 -- 172.21.15.163:0/1141863473 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca08070dc0 con 0x7fca080fad90 2024-02-19T20:56:19.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.597+0000 7fca0d3dd700 1 -- 172.21.15.163:0/1141863473 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0810eda0 con 0x7fca080fad90 2024-02-19T20:56:19.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.597+0000 7fca0d3dd700 10 client.14140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:19.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.597+0000 7fca0d3dd700 1 -- 172.21.15.163:0/1141863473 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0810f1d0 con 0x7fca080fad90 2024-02-19T20:56:19.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.597+0000 7fca0d3dd700 1 librados: init done 2024-02-19T20:56:19.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.597+0000 7fc9effff700 1 -- 172.21.15.163:0/1141863473 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fc9f80012d0 con 0x7fca080fad90 2024-02-19T20:56:19.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.597+0000 7fc9effff700 1 -- 172.21.15.163:0/1141863473 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc9f8006250 con 0x7fca080fad90 2024-02-19T20:56:19.580 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.598+0000 7fc9effff700 1 -- 172.21.15.163:0/1141863473 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 532658122 0 0) 0x7fc9f800fb30 con 0x7fca080fad90 2024-02-19T20:56:19.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.598+0000 7fca0d3dd700 1 -- 172.21.15.163:0/1141863473 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f4004f50 con 0x7fca080fad90 2024-02-19T20:56:19.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.598+0000 7fc9effff700 1 -- 172.21.15.163:0/1141863473 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 1409061398 0 0) 0x7fc9f803eca0 con 0x7fca080fad90 2024-02-19T20:56:19.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.598+0000 7fc9effff700 10 client.14140.objecter ms_dispatch 0x7fca08001550 osd_map(2..2 src has 1..2) v4 2024-02-19T20:56:19.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.598+0000 7fc9effff700 3 client.14140.objecter handle_osd_map got epochs [2,2] > 0 2024-02-19T20:56:19.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.598+0000 7fc9effff700 3 client.14140.objecter handle_osd_map decoding full epoch 2 2024-02-19T20:56:19.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.598+0000 7fc9effff700 20 client.14140.objecter dump_active .. 0 homeless 2024-02-19T20:56:19.581 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.598+0000 7fc9effff700 10 client.14140.objecter ms_handle_connect 0x7fc9f0035220 2024-02-19T20:56:19.587 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.604+0000 7fc9effff700 1 -- 172.21.15.163:0/1141863473 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc9f80110c0 con 0x7fca080fad90 2024-02-19T20:56:19.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:18.382405+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' 2024-02-19T20:56:19.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:18.413812+0000 mgr.y (mgr.14116) 13 : audit [DBG] from='client.14138 v1:172.21.15.163:0/3190470308' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:56:19.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:56:18.415121+0000 mgr.y (mgr.14116) 14 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-19T20:56:19.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:18.417426+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' 2024-02-19T20:56:19.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:18.704978+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' 2024-02-19T20:56:19.777 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.797+0000 7fca0d3dd700 1 -- 172.21.15.163:0/1141863473 --> v1:172.21.15.163:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fc9f4005200 con 0x7fca080fad90 2024-02-19T20:56:19.780 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.800+0000 7fc9effff700 1 -- 172.21.15.163:0/1141863473 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (unknown 2684808024 0 0) 0x7fc9f80131a0 con 0x7fca080fad90 2024-02-19T20:56:19.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.802+0000 7fca0d3dd700 10 librados: watch_flush enter 2024-02-19T20:56:19.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.802+0000 7fca0d3dd700 10 librados: watch_flush exit 2024-02-19T20:56:19.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.802+0000 7fca0d3dd700 20 client.14140.objecter shutdown clearing up homeless session... 2024-02-19T20:56:19.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.802+0000 7fca0d3dd700 10 client.14140.objecter successfully canceled tick 2024-02-19T20:56:19.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.802+0000 7fca0d3dd700 1 -- 172.21.15.163:0/1141863473 >> v1:172.21.15.163:6800/2731225015 conn(0x7fc9f0035220 legacy=0x7fc9f00376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:19.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.802+0000 7fca0d3dd700 1 -- 172.21.15.163:0/1141863473 >> v1:172.21.15.163:6789/0 conn(0x7fca080fad90 legacy=0x7fca0810d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:19.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.802+0000 7fca0d3dd700 1 -- 172.21.15.163:0/1141863473 shutdown_connections 2024-02-19T20:56:19.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.802+0000 7fca0d3dd700 1 -- 172.21.15.163:0/1141863473 >> 172.21.15.163:0/1141863473 conn(0x7fca080f64d0 msgr2=0x7fca080f7120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:19.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.802+0000 7fca0d3dd700 1 -- 172.21.15.163:0/1141863473 shutdown_connections 2024-02-19T20:56:19.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.802+0000 7fca0d3dd700 1 -- 172.21.15.163:0/1141863473 wait complete. 2024-02-19T20:56:19.783 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:19.802+0000 7fca0d3dd700 1 librados: shutdown 2024-02-19T20:56:21.024 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:19.801330+0000 mon.a (mon.0) 62 : audit [INF] from='client.? v1:172.21.15.163:0/1141863473' entity='client.admin' 2024-02-19T20:56:21.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.312+0000 7f9812109700 1 Processor -- start 2024-02-19T20:56:21.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.312+0000 7f9812109700 1 -- start start 2024-02-19T20:56:21.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.312+0000 7f9812109700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f980c100200 con 0x7f980c070be0 2024-02-19T20:56:21.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.313+0000 7f980b7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f980c070be0 0x7f980c0feae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:59238/0 (socket says 172.21.15.163:59238) 2024-02-19T20:56:21.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.313+0000 7f980b7fe700 1 -- 172.21.15.163:0/1722785661 learned_addr learned my addr 172.21.15.163:0/1722785661 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:21.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.313+0000 7f980a7fc700 1 -- 172.21.15.163:0/1722785661 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387168570 0 0) 0x7f980c100200 con 0x7f980c070be0 2024-02-19T20:56:21.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.313+0000 7f980a7fc700 1 -- 172.21.15.163:0/1722785661 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97f40033f0 con 0x7f980c070be0 2024-02-19T20:56:21.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.313+0000 7f980a7fc700 1 -- 172.21.15.163:0/1722785661 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 518111272 0 0) 0x7f97f40033f0 con 0x7f980c070be0 2024-02-19T20:56:21.295 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.313+0000 7f980a7fc700 1 -- 172.21.15.163:0/1722785661 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c1013a0 con 0x7f980c070be0 2024-02-19T20:56:21.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.313+0000 7f980a7fc700 1 -- 172.21.15.163:0/1722785661 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f9800001590 con 0x7f980c070be0 2024-02-19T20:56:21.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.313+0000 7f980a7fc700 1 -- 172.21.15.163:0/1722785661 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9800001c50 con 0x7f980c070be0 2024-02-19T20:56:21.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.314+0000 7f980a7fc700 1 -- 172.21.15.163:0/1722785661 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f98000056b0 con 0x7f980c070be0 2024-02-19T20:56:21.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.314+0000 7f9812109700 1 -- 172.21.15.163:0/1722785661 >> v1:172.21.15.163:6789/0 conn(0x7f980c070be0 legacy=0x7f980c0feae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:21.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.314+0000 7f9812109700 1 -- 172.21.15.163:0/1722785661 shutdown_connections 2024-02-19T20:56:21.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.314+0000 7f9812109700 1 -- 172.21.15.163:0/1722785661 >> 172.21.15.163:0/1722785661 conn(0x7f980c0f6500 msgr2=0x7f980c0f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:21.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.314+0000 7f9812109700 1 -- 172.21.15.163:0/1722785661 shutdown_connections 2024-02-19T20:56:21.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.314+0000 7f9812109700 1 -- 172.21.15.163:0/1722785661 wait complete. 2024-02-19T20:56:21.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.315+0000 7f9812109700 1 librados: starting msgr at 2024-02-19T20:56:21.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.315+0000 7f9812109700 1 librados: starting objecter 2024-02-19T20:56:21.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.315+0000 7f9812109700 1 Processor -- start 2024-02-19T20:56:21.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.315+0000 7f9812109700 1 -- start start 2024-02-19T20:56:21.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.315+0000 7f9812109700 1 librados: setting wanted keys 2024-02-19T20:56:21.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.315+0000 7f9812109700 1 librados: calling monclient init 2024-02-19T20:56:21.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.315+0000 7f9812109700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f980c10af50 con 0x7f980c070be0 2024-02-19T20:56:21.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.315+0000 7f980b7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f980c070be0 0x7f980c10a840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:59254/0 (socket says 172.21.15.163:59254) 2024-02-19T20:56:21.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.315+0000 7f980b7fe700 1 -- 172.21.15.163:0/1974087999 learned_addr learned my addr 172.21.15.163:0/1974087999 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:21.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.316+0000 7f9808ff9700 10 client.?.objecter ms_handle_connect 0x7f980c070be0 2024-02-19T20:56:21.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.316+0000 7f9808ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:21.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.316+0000 7f9808ff9700 1 -- 172.21.15.163:0/1974087999 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067808419 0 0) 0x7f980c10af50 con 0x7f980c070be0 2024-02-19T20:56:21.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.316+0000 7f9808ff9700 1 -- 172.21.15.163:0/1974087999 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97f40055d0 con 0x7f980c070be0 2024-02-19T20:56:21.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.316+0000 7f9808ff9700 1 -- 172.21.15.163:0/1974087999 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1745148545 0 0) 0x7f97f40055d0 con 0x7f980c070be0 2024-02-19T20:56:21.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.316+0000 7f9808ff9700 1 -- 172.21.15.163:0/1974087999 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f980c10af50 con 0x7f980c070be0 2024-02-19T20:56:21.299 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.316+0000 7f9808ff9700 1 -- 172.21.15.163:0/1974087999 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f98000046b0 con 0x7f980c070be0 2024-02-19T20:56:21.300 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.316+0000 7f9808ff9700 1 -- 172.21.15.163:0/1974087999 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2713788544 0 0) 0x7f980c10af50 con 0x7f980c070be0 2024-02-19T20:56:21.300 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.316+0000 7f9808ff9700 1 -- 172.21.15.163:0/1974087999 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c10b0e0 con 0x7f980c070be0 2024-02-19T20:56:21.300 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.316+0000 7f9812109700 1 -- 172.21.15.163:0/1974087999 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980c10b390 con 0x7f980c070be0 2024-02-19T20:56:21.300 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.316+0000 7f9812109700 10 client.14142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:21.300 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.316+0000 7f9812109700 1 -- 172.21.15.163:0/1974087999 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f980c10f080 con 0x7f980c070be0 2024-02-19T20:56:21.300 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.316+0000 7f9812109700 1 librados: init done 2024-02-19T20:56:21.300 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.316+0000 7f9808ff9700 1 -- 172.21.15.163:0/1974087999 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9800001850 con 0x7f980c070be0 2024-02-19T20:56:21.300 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.316+0000 7f9808ff9700 1 -- 172.21.15.163:0/1974087999 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f98000060c0 con 0x7f980c070be0 2024-02-19T20:56:21.300 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.317+0000 7f9808ff9700 1 -- 172.21.15.163:0/1974087999 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 532658122 0 0) 0x7f980000f900 con 0x7f980c070be0 2024-02-19T20:56:21.300 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.317+0000 7f9808ff9700 1 -- 172.21.15.163:0/1974087999 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 1409061398 0 0) 0x7f980003ebe0 con 0x7f980c070be0 2024-02-19T20:56:21.300 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.317+0000 7f9808ff9700 10 client.14142.objecter ms_dispatch 0x7f980c001550 osd_map(2..2 src has 1..2) v4 2024-02-19T20:56:21.300 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.317+0000 7f9808ff9700 3 client.14142.objecter handle_osd_map got epochs [2,2] > 0 2024-02-19T20:56:21.300 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.317+0000 7f9808ff9700 3 client.14142.objecter handle_osd_map decoding full epoch 2 2024-02-19T20:56:21.301 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.317+0000 7f9808ff9700 20 client.14142.objecter dump_active .. 0 homeless 2024-02-19T20:56:21.301 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.317+0000 7f9812109700 1 -- 172.21.15.163:0/1974087999 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f8004f50 con 0x7f980c070be0 2024-02-19T20:56:21.301 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.318+0000 7f9808ff9700 10 client.14142.objecter ms_handle_connect 0x7f97f4035340 2024-02-19T20:56:21.304 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.324+0000 7f9808ff9700 1 -- 172.21.15.163:0/1974087999 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9800011020 con 0x7f980c070be0 2024-02-19T20:56:21.475 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.494+0000 7f9812109700 1 -- 172.21.15.163:0/1974087999 --> v1:172.21.15.163:6789/0 -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f97f8005780 con 0x7f980c070be0 2024-02-19T20:56:21.477 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.497+0000 7f9808ff9700 1 -- 172.21.15.163:0/1974087999 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (unknown 1496729260 0 0) 0x7f9800013100 con 0x7f980c070be0 2024-02-19T20:56:21.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.499+0000 7f9812109700 10 librados: watch_flush enter 2024-02-19T20:56:21.479 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.499+0000 7f9812109700 10 librados: watch_flush exit 2024-02-19T20:56:21.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.499+0000 7f9812109700 20 client.14142.objecter shutdown clearing up homeless session... 2024-02-19T20:56:21.480 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.499+0000 7f9812109700 10 client.14142.objecter successfully canceled tick 2024-02-19T20:56:21.483 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.502+0000 7f9812109700 1 -- 172.21.15.163:0/1974087999 >> v1:172.21.15.163:6800/2731225015 conn(0x7f97f4035340 legacy=0x7f97f4037800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:21.483 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.502+0000 7f9812109700 1 -- 172.21.15.163:0/1974087999 >> v1:172.21.15.163:6789/0 conn(0x7f980c070be0 legacy=0x7f980c10a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:21.483 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.502+0000 7f9812109700 1 -- 172.21.15.163:0/1974087999 shutdown_connections 2024-02-19T20:56:21.483 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.502+0000 7f9812109700 1 -- 172.21.15.163:0/1974087999 >> 172.21.15.163:0/1974087999 conn(0x7f980c0f6500 msgr2=0x7f980c0fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:21.483 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.502+0000 7f9812109700 1 -- 172.21.15.163:0/1974087999 shutdown_connections 2024-02-19T20:56:21.483 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.502+0000 7f9812109700 1 -- 172.21.15.163:0/1974087999 wait complete. 2024-02-19T20:56:21.483 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:21.502+0000 7f9812109700 1 librados: shutdown 2024-02-19T20:56:22.227 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:21.075164+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.14116 v1:172.21.15.163:0/4063117323' entity='mgr.y' 2024-02-19T20:56:22.227 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:21.498771+0000 mon.a (mon.0) 64 : audit [INF] from='client.? v1:172.21.15.163:0/1974087999' entity='client.admin' 2024-02-19T20:56:22.988 INFO:teuthology.orchestra.run.smithi163.stderr:Enabling the dashboard module... 2024-02-19T20:56:23.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:22.194209+0000 mgr.y (mgr.14116) 15 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:56:23.526 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.542+0000 7f435efd9700 1 Processor -- start 2024-02-19T20:56:23.527 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.542+0000 7f435efd9700 1 -- start start 2024-02-19T20:56:23.527 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.542+0000 7f435efd9700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f43580fb8d0 con 0x7f43580fada0 2024-02-19T20:56:23.527 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.542+0000 7f435cd75700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f43580fada0 0x7f43580fb1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:59256/0 (socket says 172.21.15.163:59256) 2024-02-19T20:56:23.527 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.542+0000 7f435cd75700 1 -- 172.21.15.163:0/1237388081 learned_addr learned my addr 172.21.15.163:0/1237388081 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:23.527 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.543+0000 7f43577fe700 1 -- 172.21.15.163:0/1237388081 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963028834 0 0) 0x7f434c001110 con 0x7f43580fada0 2024-02-19T20:56:23.527 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.543+0000 7f43577fe700 1 -- 172.21.15.163:0/1237388081 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43480033f0 con 0x7f43580fada0 2024-02-19T20:56:23.527 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.543+0000 7f43577fe700 1 -- 172.21.15.163:0/1237388081 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1949274192 0 0) 0x7f43480033f0 con 0x7f43580fada0 2024-02-19T20:56:23.527 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.543+0000 7f43577fe700 1 -- 172.21.15.163:0/1237388081 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43581001e0 con 0x7f43580fada0 2024-02-19T20:56:23.527 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.543+0000 7f43577fe700 1 -- 172.21.15.163:0/1237388081 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f434c001720 con 0x7f43580fada0 2024-02-19T20:56:23.527 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.543+0000 7f43577fe700 1 -- 172.21.15.163:0/1237388081 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f434c004770 con 0x7f43580fada0 2024-02-19T20:56:23.527 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.543+0000 7f43577fe700 1 -- 172.21.15.163:0/1237388081 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f434c005920 con 0x7f43580fada0 2024-02-19T20:56:23.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.543+0000 7f435efd9700 1 -- 172.21.15.163:0/1237388081 >> v1:172.21.15.163:6789/0 conn(0x7f43580fada0 legacy=0x7f43580fb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:23.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.543+0000 7f435efd9700 1 -- 172.21.15.163:0/1237388081 shutdown_connections 2024-02-19T20:56:23.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.543+0000 7f435efd9700 1 -- 172.21.15.163:0/1237388081 >> 172.21.15.163:0/1237388081 conn(0x7f43580f6500 msgr2=0x7f43580f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:23.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.544+0000 7f435efd9700 1 -- 172.21.15.163:0/1237388081 shutdown_connections 2024-02-19T20:56:23.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.544+0000 7f435efd9700 1 -- 172.21.15.163:0/1237388081 wait complete. 2024-02-19T20:56:23.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.544+0000 7f435efd9700 1 librados: starting msgr at 2024-02-19T20:56:23.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.544+0000 7f435efd9700 1 librados: starting objecter 2024-02-19T20:56:23.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.544+0000 7f435efd9700 1 Processor -- start 2024-02-19T20:56:23.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.544+0000 7f435efd9700 1 -- start start 2024-02-19T20:56:23.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.544+0000 7f435efd9700 1 librados: setting wanted keys 2024-02-19T20:56:23.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.544+0000 7f435efd9700 1 librados: calling monclient init 2024-02-19T20:56:23.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.545+0000 7f435efd9700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f435810b090 con 0x7f43580fada0 2024-02-19T20:56:23.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.545+0000 7f435cd75700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f43580fada0 0x7f435810a980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:59260/0 (socket says 172.21.15.163:59260) 2024-02-19T20:56:23.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.545+0000 7f435cd75700 1 -- 172.21.15.163:0/3964839494 learned_addr learned my addr 172.21.15.163:0/3964839494 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:23.528 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.545+0000 7f4355ffb700 10 client.?.objecter ms_handle_connect 0x7f43580fada0 2024-02-19T20:56:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.545+0000 7f4355ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.545+0000 7f4355ffb700 1 -- 172.21.15.163:0/3964839494 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899712557 0 0) 0x7f435810b090 con 0x7f43580fada0 2024-02-19T20:56:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.545+0000 7f4355ffb700 1 -- 172.21.15.163:0/3964839494 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4348005580 con 0x7f43580fada0 2024-02-19T20:56:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.545+0000 7f4355ffb700 1 -- 172.21.15.163:0/3964839494 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1676171382 0 0) 0x7f4348005580 con 0x7f43580fada0 2024-02-19T20:56:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.545+0000 7f4355ffb700 1 -- 172.21.15.163:0/3964839494 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f435810b090 con 0x7f43580fada0 2024-02-19T20:56:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.545+0000 7f4355ffb700 1 -- 172.21.15.163:0/3964839494 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f434c000fe0 con 0x7f43580fada0 2024-02-19T20:56:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.545+0000 7f4355ffb700 1 -- 172.21.15.163:0/3964839494 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 155813393 0 0) 0x7f435810b090 con 0x7f43580fada0 2024-02-19T20:56:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.545+0000 7f4355ffb700 1 -- 172.21.15.163:0/3964839494 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435810b220 con 0x7f43580fada0 2024-02-19T20:56:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.545+0000 7f435efd9700 1 -- 172.21.15.163:0/3964839494 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435810b4d0 con 0x7f43580fada0 2024-02-19T20:56:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.546+0000 7f4355ffb700 1 -- 172.21.15.163:0/3964839494 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f434c0049e0 con 0x7f43580fada0 2024-02-19T20:56:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.546+0000 7f4355ffb700 1 -- 172.21.15.163:0/3964839494 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f434c006380 con 0x7f43580fada0 2024-02-19T20:56:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.546+0000 7f435efd9700 10 client.14144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.546+0000 7f435efd9700 1 -- 172.21.15.163:0/3964839494 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f435810f1c0 con 0x7f43580fada0 2024-02-19T20:56:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.546+0000 7f435efd9700 1 librados: init done 2024-02-19T20:56:23.529 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.546+0000 7f435efd9700 1 -- 172.21.15.163:0/3964839494 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4358046ce0 con 0x7f43580fada0 2024-02-19T20:56:23.530 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.547+0000 7f4355ffb700 1 -- 172.21.15.163:0/3964839494 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 8) v1 ==== 38582+0+0 (unknown 532658122 0 0) 0x7f434c004b50 con 0x7f43580fada0 2024-02-19T20:56:23.530 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.547+0000 7f4355ffb700 1 -- 172.21.15.163:0/3964839494 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 1409061398 0 0) 0x7f434c03ef90 con 0x7f43580fada0 2024-02-19T20:56:23.531 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.547+0000 7f4355ffb700 10 client.14144.objecter ms_dispatch 0x7f4358001550 osd_map(2..2 src has 1..2) v4 2024-02-19T20:56:23.531 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.547+0000 7f4355ffb700 3 client.14144.objecter handle_osd_map got epochs [2,2] > 0 2024-02-19T20:56:23.531 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.547+0000 7f4355ffb700 3 client.14144.objecter handle_osd_map decoding full epoch 2 2024-02-19T20:56:23.531 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.547+0000 7f4355ffb700 20 client.14144.objecter dump_active .. 0 homeless 2024-02-19T20:56:23.531 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.547+0000 7f4355ffb700 10 client.14144.objecter ms_handle_connect 0x7f4348035230 2024-02-19T20:56:23.531 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.551+0000 7f4355ffb700 1 -- 172.21.15.163:0/3964839494 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f434c014080 con 0x7f43580fada0 2024-02-19T20:56:23.737 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:23.757+0000 7f435efd9700 1 -- 172.21.15.163:0/3964839494 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f43580fb8d0 con 0x7f43580fada0 2024-02-19T20:56:24.178 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:24.197+0000 7f4355ffb700 1 -- 172.21.15.163:0/3964839494 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mgrmap(e 9) v1 ==== 38595+0+0 (unknown 3240904757 0 0) 0x7f434c03e150 con 0x7f43580fada0 2024-02-19T20:56:24.179 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:24.199+0000 7f4355ffb700 1 -- 172.21.15.163:0/3964839494 <== mon.0 v1:172.21.15.163:6789/0 11 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (unknown 1498667528 0 0) 0x7f434c0142b0 con 0x7f43580fada0 2024-02-19T20:56:24.181 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:24.200+0000 7f435efd9700 10 librados: watch_flush enter 2024-02-19T20:56:24.181 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:24.200+0000 7f435efd9700 10 librados: watch_flush exit 2024-02-19T20:56:24.181 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:24.200+0000 7f435efd9700 20 client.14144.objecter shutdown clearing up homeless session... 2024-02-19T20:56:24.182 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:24.200+0000 7f435efd9700 10 client.14144.objecter successfully canceled tick 2024-02-19T20:56:24.182 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:24.201+0000 7f435efd9700 1 -- 172.21.15.163:0/3964839494 >> v1:172.21.15.163:6800/2731225015 conn(0x7f4348035230 legacy=0x7f43480376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:24.182 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:24.201+0000 7f435efd9700 1 -- 172.21.15.163:0/3964839494 >> v1:172.21.15.163:6789/0 conn(0x7f43580fada0 legacy=0x7f435810a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:24.182 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:24.201+0000 7f435efd9700 1 -- 172.21.15.163:0/3964839494 shutdown_connections 2024-02-19T20:56:24.182 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:24.201+0000 7f435efd9700 1 -- 172.21.15.163:0/3964839494 >> 172.21.15.163:0/3964839494 conn(0x7f43580f6500 msgr2=0x7f435806fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:24.182 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:24.201+0000 7f435efd9700 1 -- 172.21.15.163:0/3964839494 shutdown_connections 2024-02-19T20:56:24.182 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:24.201+0000 7f435efd9700 1 -- 172.21.15.163:0/3964839494 wait complete. 2024-02-19T20:56:24.182 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:24.201+0000 7f435efd9700 1 librados: shutdown 2024-02-19T20:56:24.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:23.758681+0000 mon.a (mon.0) 65 : audit [INF] from='client.? v1:172.21.15.163:0/3964839494' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-19T20:56:24.477 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ignoring --setuser ceph since I am not root 2024-02-19T20:56:24.477 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ignoring --setgroup ceph since I am not root 2024-02-19T20:56:24.477 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:24.363+0000 7f2560335680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-19T20:56:24.477 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:24.445+0000 7f2560335680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-19T20:56:24.816 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:24.816+0000 7f2560335680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-19T20:56:25.423 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.441+0000 7f3c502bc700 1 Processor -- start 2024-02-19T20:56:25.423 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.442+0000 7f3c502bc700 1 -- start start 2024-02-19T20:56:25.423 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.442+0000 7f3c502bc700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3c481001b0 con 0x7f3c48070c20 2024-02-19T20:56:25.423 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.442+0000 7f3c4e058700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f3c48070c20 0x7f3c480fea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43780/0 (socket says 172.21.15.163:43780) 2024-02-19T20:56:25.423 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.442+0000 7f3c4e058700 1 -- 172.21.15.163:0/2305404604 learned_addr learned my addr 172.21.15.163:0/2305404604 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:25.424 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.442+0000 7f3c4d056700 1 -- 172.21.15.163:0/2305404604 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093699502 0 0) 0x7f3c481001b0 con 0x7f3c48070c20 2024-02-19T20:56:25.424 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.442+0000 7f3c4d056700 1 -- 172.21.15.163:0/2305404604 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c340033f0 con 0x7f3c48070c20 2024-02-19T20:56:25.424 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.443+0000 7f3c4d056700 1 -- 172.21.15.163:0/2305404604 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 384172793 0 0) 0x7f3c340033f0 con 0x7f3c48070c20 2024-02-19T20:56:25.424 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.443+0000 7f3c4d056700 1 -- 172.21.15.163:0/2305404604 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c48101350 con 0x7f3c48070c20 2024-02-19T20:56:25.424 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.443+0000 7f3c4d056700 1 -- 172.21.15.163:0/2305404604 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f3c44001590 con 0x7f3c48070c20 2024-02-19T20:56:25.424 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.443+0000 7f3c4d056700 1 -- 172.21.15.163:0/2305404604 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3c44001c50 con 0x7f3c48070c20 2024-02-19T20:56:25.424 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.443+0000 7f3c4d056700 1 -- 172.21.15.163:0/2305404604 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f3c44005640 con 0x7f3c48070c20 2024-02-19T20:56:25.426 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.443+0000 7f3c502bc700 1 -- 172.21.15.163:0/2305404604 >> v1:172.21.15.163:6789/0 conn(0x7f3c48070c20 legacy=0x7f3c480fea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:25.426 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.444+0000 7f3c502bc700 1 -- 172.21.15.163:0/2305404604 shutdown_connections 2024-02-19T20:56:25.426 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.444+0000 7f3c502bc700 1 -- 172.21.15.163:0/2305404604 >> 172.21.15.163:0/2305404604 conn(0x7f3c480f64d0 msgr2=0x7f3c480f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:25.426 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.444+0000 7f3c502bc700 1 -- 172.21.15.163:0/2305404604 shutdown_connections 2024-02-19T20:56:25.426 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.444+0000 7f3c502bc700 1 -- 172.21.15.163:0/2305404604 wait complete. 2024-02-19T20:56:25.426 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.444+0000 7f3c502bc700 1 librados: starting msgr at 2024-02-19T20:56:25.426 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.444+0000 7f3c502bc700 1 librados: starting objecter 2024-02-19T20:56:25.426 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.444+0000 7f3c502bc700 1 Processor -- start 2024-02-19T20:56:25.426 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.444+0000 7f3c502bc700 1 -- start start 2024-02-19T20:56:25.426 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.444+0000 7f3c502bc700 1 librados: setting wanted keys 2024-02-19T20:56:25.427 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.444+0000 7f3c502bc700 1 librados: calling monclient init 2024-02-19T20:56:25.427 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.444+0000 7f3c502bc700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3c480fc4a0 con 0x7f3c48070c20 2024-02-19T20:56:25.427 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c4e058700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f3c48070c20 0x7f3c480fa990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43792/0 (socket says 172.21.15.163:43792) 2024-02-19T20:56:25.427 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c4e058700 1 -- 172.21.15.163:0/132502019 learned_addr learned my addr 172.21.15.163:0/132502019 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:25.427 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c3f779700 10 client.?.objecter ms_handle_connect 0x7f3c48070c20 2024-02-19T20:56:25.427 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c3f779700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:25.427 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c3f779700 1 -- 172.21.15.163:0/132502019 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761016496 0 0) 0x7f3c480fc4a0 con 0x7f3c48070c20 2024-02-19T20:56:25.427 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c3f779700 1 -- 172.21.15.163:0/132502019 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c340055d0 con 0x7f3c48070c20 2024-02-19T20:56:25.427 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c3f779700 1 -- 172.21.15.163:0/132502019 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1434059016 0 0) 0x7f3c340055d0 con 0x7f3c48070c20 2024-02-19T20:56:25.428 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c3f779700 1 -- 172.21.15.163:0/132502019 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c480fc4a0 con 0x7f3c48070c20 2024-02-19T20:56:25.428 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c3f779700 1 -- 172.21.15.163:0/132502019 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f3c44004a70 con 0x7f3c48070c20 2024-02-19T20:56:25.428 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c3f779700 1 -- 172.21.15.163:0/132502019 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1542282629 0 0) 0x7f3c480fc4a0 con 0x7f3c48070c20 2024-02-19T20:56:25.428 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c3f779700 1 -- 172.21.15.163:0/132502019 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c480fc630 con 0x7f3c48070c20 2024-02-19T20:56:25.428 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c502bc700 1 -- 172.21.15.163:0/132502019 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c480fc8e0 con 0x7f3c48070c20 2024-02-19T20:56:25.428 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c502bc700 10 client.14148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:25.428 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c502bc700 1 -- 172.21.15.163:0/132502019 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c480fb3a0 con 0x7f3c48070c20 2024-02-19T20:56:25.428 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c502bc700 1 librados: init done 2024-02-19T20:56:25.428 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c3f779700 1 -- 172.21.15.163:0/132502019 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3c440056b0 con 0x7f3c48070c20 2024-02-19T20:56:25.428 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.445+0000 7f3c3f779700 1 -- 172.21.15.163:0/132502019 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f3c440060c0 con 0x7f3c48070c20 2024-02-19T20:56:25.428 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.446+0000 7f3c3f779700 1 -- 172.21.15.163:0/132502019 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 9) v1 ==== 38595+0+0 (unknown 3240904757 0 0) 0x7f3c4400f8f0 con 0x7f3c48070c20 2024-02-19T20:56:25.428 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.446+0000 7f3c502bc700 1 -- 172.21.15.163:0/132502019 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2c004f50 con 0x7f3c48070c20 2024-02-19T20:56:25.428 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.446+0000 7f3c4d857700 1 -- 172.21.15.163:0/132502019 >> v1:172.21.15.163:6800/2731225015 conn(0x7f3c340353c0 legacy=0x7f3c34037880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.163:6800/2731225015 2024-02-19T20:56:25.429 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.446+0000 7f3c3f779700 1 -- 172.21.15.163:0/132502019 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 1409061398 0 0) 0x7f3c4403fa60 con 0x7f3c48070c20 2024-02-19T20:56:25.429 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.446+0000 7f3c3f779700 10 client.14148.objecter ms_dispatch 0x7f3c48001550 osd_map(2..2 src has 1..2) v4 2024-02-19T20:56:25.429 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.446+0000 7f3c3f779700 3 client.14148.objecter handle_osd_map got epochs [2,2] > 0 2024-02-19T20:56:25.429 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.446+0000 7f3c3f779700 3 client.14148.objecter handle_osd_map decoding full epoch 2 2024-02-19T20:56:25.429 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.446+0000 7f3c3f779700 20 client.14148.objecter dump_active .. 0 homeless 2024-02-19T20:56:25.431 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.451+0000 7f3c3f779700 1 -- 172.21.15.163:0/132502019 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3c44010f10 con 0x7f3c48070c20 2024-02-19T20:56:25.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:24.200188+0000 mon.a (mon.0) 66 : audit [INF] from='client.? v1:172.21.15.163:0/3964839494' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-19T20:56:25.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:24.200243+0000 mon.a (mon.0) 67 : cluster [DBG] mgrmap e9: y(active, since 22s) 2024-02-19T20:56:25.627 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.647+0000 7f3c4d857700 1 -- 172.21.15.163:0/132502019 >> v1:172.21.15.163:6800/2731225015 conn(0x7f3c340353c0 legacy=0x7f3c34037880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.163:6800/2731225015 2024-02-19T20:56:25.680 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.699+0000 7f3c502bc700 1 -- 172.21.15.163:0/132502019 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3c2c005a10 con 0x7f3c48070c20 2024-02-19T20:56:25.681 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.700+0000 7f3c3f779700 1 -- 172.21.15.163:0/132502019 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (unknown 2005831594 0 1748205903) 0x7f3c44012ff0 con 0x7f3c48070c20 2024-02-19T20:56:25.681 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: { 2024-02-19T20:56:25.681 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-19T20:56:25.681 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "available": true, 2024-02-19T20:56:25.681 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-19T20:56:25.681 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-19T20:56:25.681 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: } 2024-02-19T20:56:25.683 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.701+0000 7f3c502bc700 10 librados: watch_flush enter 2024-02-19T20:56:25.683 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.701+0000 7f3c502bc700 10 librados: watch_flush exit 2024-02-19T20:56:25.683 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.702+0000 7f3c502bc700 20 client.14148.objecter shutdown clearing up homeless session... 2024-02-19T20:56:25.683 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.702+0000 7f3c502bc700 10 client.14148.objecter successfully canceled tick 2024-02-19T20:56:25.683 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.702+0000 7f3c502bc700 1 -- 172.21.15.163:0/132502019 >> v1:172.21.15.163:6800/2731225015 conn(0x7f3c340353c0 legacy=0x7f3c34037880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T20:56:25.683 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.702+0000 7f3c502bc700 1 -- 172.21.15.163:0/132502019 >> v1:172.21.15.163:6789/0 conn(0x7f3c48070c20 legacy=0x7f3c480fa990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:25.683 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.702+0000 7f3c502bc700 1 -- 172.21.15.163:0/132502019 shutdown_connections 2024-02-19T20:56:25.683 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.702+0000 7f3c502bc700 1 -- 172.21.15.163:0/132502019 >> 172.21.15.163:0/132502019 conn(0x7f3c480f64d0 msgr2=0x7f3c480fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:25.683 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.702+0000 7f3c502bc700 1 -- 172.21.15.163:0/132502019 shutdown_connections 2024-02-19T20:56:25.683 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.702+0000 7f3c502bc700 1 -- 172.21.15.163:0/132502019 wait complete. 2024-02-19T20:56:25.683 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:25.702+0000 7f3c502bc700 1 librados: shutdown 2024-02-19T20:56:25.870 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:25.587+0000 7f2560335680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-19T20:56:25.870 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:25.672+0000 7f2560335680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-19T20:56:26.177 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:25.890+0000 7f2560335680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-19T20:56:26.288 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for the mgr to restart... 2024-02-19T20:56:26.288 INFO:teuthology.orchestra.run.smithi163.stderr:Waiting for mgr epoch 9... 2024-02-19T20:56:26.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:25.701194+0000 mon.a (mon.0) 68 : audit [DBG] from='client.? v1:172.21.15.163:0/132502019' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T20:56:26.890 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.908+0000 7ff48d3be700 1 Processor -- start 2024-02-19T20:56:26.890 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.908+0000 7ff48d3be700 1 -- start start 2024-02-19T20:56:26.890 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.908+0000 7ff48d3be700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff4880fb8c0 con 0x7ff4880fad90 2024-02-19T20:56:26.890 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.908+0000 7ff486ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ff4880fad90 0x7ff4880fb1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43794/0 (socket says 172.21.15.163:43794) 2024-02-19T20:56:26.890 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.908+0000 7ff486ffd700 1 -- 172.21.15.163:0/94299710 learned_addr learned my addr 172.21.15.163:0/94299710 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:26.890 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.909+0000 7ff485ffb700 1 -- 172.21.15.163:0/94299710 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431808548 0 0) 0x7ff478001110 con 0x7ff4880fad90 2024-02-19T20:56:26.890 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.909+0000 7ff485ffb700 1 -- 172.21.15.163:0/94299710 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4700033f0 con 0x7ff4880fad90 2024-02-19T20:56:26.890 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.909+0000 7ff485ffb700 1 -- 172.21.15.163:0/94299710 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1947329597 0 0) 0x7ff4700033f0 con 0x7ff4880fad90 2024-02-19T20:56:26.890 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.909+0000 7ff485ffb700 1 -- 172.21.15.163:0/94299710 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4881001d0 con 0x7ff4880fad90 2024-02-19T20:56:26.890 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.909+0000 7ff485ffb700 1 -- 172.21.15.163:0/94299710 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7ff478001720 con 0x7ff4880fad90 2024-02-19T20:56:26.891 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.909+0000 7ff485ffb700 1 -- 172.21.15.163:0/94299710 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff478004770 con 0x7ff4880fad90 2024-02-19T20:56:26.891 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.909+0000 7ff485ffb700 1 -- 172.21.15.163:0/94299710 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7ff478005840 con 0x7ff4880fad90 2024-02-19T20:56:26.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.910+0000 7ff48d3be700 1 -- 172.21.15.163:0/94299710 >> v1:172.21.15.163:6789/0 conn(0x7ff4880fad90 legacy=0x7ff4880fb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:26.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.910+0000 7ff48d3be700 1 -- 172.21.15.163:0/94299710 shutdown_connections 2024-02-19T20:56:26.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.910+0000 7ff48d3be700 1 -- 172.21.15.163:0/94299710 >> 172.21.15.163:0/94299710 conn(0x7ff4880f64d0 msgr2=0x7ff4880f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:26.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.910+0000 7ff48d3be700 1 -- 172.21.15.163:0/94299710 shutdown_connections 2024-02-19T20:56:26.892 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.910+0000 7ff48d3be700 1 -- 172.21.15.163:0/94299710 wait complete. 2024-02-19T20:56:26.893 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.910+0000 7ff48d3be700 1 librados: starting msgr at 2024-02-19T20:56:26.893 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.911+0000 7ff48d3be700 1 librados: starting objecter 2024-02-19T20:56:26.893 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.911+0000 7ff48d3be700 1 Processor -- start 2024-02-19T20:56:26.893 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.911+0000 7ff48d3be700 1 -- start start 2024-02-19T20:56:26.893 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.911+0000 7ff48d3be700 1 librados: setting wanted keys 2024-02-19T20:56:26.893 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.911+0000 7ff48d3be700 1 librados: calling monclient init 2024-02-19T20:56:26.893 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.911+0000 7ff48d3be700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff48810b0b0 con 0x7ff4880fad90 2024-02-19T20:56:26.893 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.911+0000 7ff486ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ff4880fad90 0x7ff48810a9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43804/0 (socket says 172.21.15.163:43804) 2024-02-19T20:56:26.893 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.911+0000 7ff486ffd700 1 -- 172.21.15.163:0/2006993148 learned_addr learned my addr 172.21.15.163:0/2006993148 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:26.893 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.911+0000 7ff46ffff700 10 client.?.objecter ms_handle_connect 0x7ff4880fad90 2024-02-19T20:56:26.893 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.911+0000 7ff46ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:26.894 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.911+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473736105 0 0) 0x7ff48810b0b0 con 0x7ff4880fad90 2024-02-19T20:56:26.894 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.911+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4700055d0 con 0x7ff4880fad90 2024-02-19T20:56:26.895 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.912+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3226880589 0 0) 0x7ff4700055d0 con 0x7ff4880fad90 2024-02-19T20:56:26.895 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.912+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff48810b0b0 con 0x7ff4880fad90 2024-02-19T20:56:26.895 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.912+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7ff478004c50 con 0x7ff4880fad90 2024-02-19T20:56:26.895 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.912+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1883341247 0 0) 0x7ff48810b0b0 con 0x7ff4880fad90 2024-02-19T20:56:26.895 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.912+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48810b240 con 0x7ff4880fad90 2024-02-19T20:56:26.895 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.912+0000 7ff48d3be700 1 -- 172.21.15.163:0/2006993148 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff48810b4f0 con 0x7ff4880fad90 2024-02-19T20:56:26.895 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.912+0000 7ff48d3be700 10 client.14150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:26.895 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.912+0000 7ff48d3be700 1 -- 172.21.15.163:0/2006993148 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48810f1e0 con 0x7ff4880fad90 2024-02-19T20:56:26.895 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.912+0000 7ff48d3be700 1 librados: init done 2024-02-19T20:56:26.895 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.913+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff4780049c0 con 0x7ff4880fad90 2024-02-19T20:56:26.895 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.913+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7ff478006170 con 0x7ff4880fad90 2024-02-19T20:56:26.895 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.913+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 9) v1 ==== 38595+0+0 (unknown 3240904757 0 0) 0x7ff47800fa60 con 0x7ff4880fad90 2024-02-19T20:56:26.895 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.913+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 --> v1:172.21.15.163:6800/2731225015 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff470039110 con 0x7ff470035440 2024-02-19T20:56:26.895 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.913+0000 7ff4867fc700 1 -- 172.21.15.163:0/2006993148 >> v1:172.21.15.163:6800/2731225015 conn(0x7ff470035440 legacy=0x7ff470037900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.163:6800/2731225015 2024-02-19T20:56:26.896 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.913+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 883+0+0 (unknown 1409061398 0 0) 0x7ff47803fb70 con 0x7ff4880fad90 2024-02-19T20:56:26.896 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.913+0000 7ff46ffff700 10 client.14150.objecter ms_dispatch 0x7ff488001550 osd_map(2..2 src has 1..2) v4 2024-02-19T20:56:26.896 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.913+0000 7ff46ffff700 3 client.14150.objecter handle_osd_map got epochs [2,2] > 0 2024-02-19T20:56:26.896 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.913+0000 7ff46ffff700 3 client.14150.objecter handle_osd_map decoding full epoch 2 2024-02-19T20:56:26.896 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:26.913+0000 7ff46ffff700 20 client.14150.objecter dump_active .. 0 homeless 2024-02-19T20:56:27.094 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:27.114+0000 7ff4867fc700 1 -- 172.21.15.163:0/2006993148 >> v1:172.21.15.163:6800/2731225015 conn(0x7ff470035440 legacy=0x7ff470037900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.163:6800/2731225015 2024-02-19T20:56:27.127 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:26.847+0000 7f2560335680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-19T20:56:27.127 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:26.980+0000 7f2560335680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-19T20:56:27.127 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:27.058+0000 7f2560335680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-19T20:56:27.477 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:27.147+0000 7f2560335680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-19T20:56:27.477 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:27.342+0000 7f2560335680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-19T20:56:27.495 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:27.515+0000 7ff4867fc700 1 -- 172.21.15.163:0/2006993148 >> v1:172.21.15.163:6800/2731225015 conn(0x7ff470035440 legacy=0x7ff470037900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.163:6800/2731225015 2024-02-19T20:56:27.977 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:27.618+0000 7f2560335680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-19T20:56:27.977 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:27.716+0000 7f2560335680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-19T20:56:28.296 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:28.316+0000 7ff4867fc700 1 -- 172.21.15.163:0/2006993148 >> v1:172.21.15.163:6800/2731225015 conn(0x7ff470035440 legacy=0x7ff470037900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.163:6800/2731225015 2024-02-19T20:56:28.727 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:28.458+0000 7f2560335680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-19T20:56:28.727 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:28.541+0000 7f2560335680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-19T20:56:28.727 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:28.632+0000 7f2560335680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-19T20:56:29.047 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:28.808+0000 7f2560335680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-19T20:56:29.314 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:29.066+0000 7f2560335680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-19T20:56:29.315 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:29.198+0000 7f2560335680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-19T20:56:29.569 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:29.334+0000 7f2560335680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-19T20:56:29.569 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:29.505+0000 7f2560335680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-19T20:56:29.569 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T20:56:29.589+0000 7f2560335680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-19T20:56:29.584 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:29.604+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mgrmap(e 10) v1 ==== 38454+0+0 (unknown 2197569270 0 0) 0x7ff478005b30 con 0x7ff4880fad90 2024-02-19T20:56:29.584 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:29.604+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 >> v1:172.21.15.163:6800/2731225015 conn(0x7ff470035440 legacy=0x7ff470037900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T20:56:29.977 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [80B blob data] 2024-02-19T20:56:29.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:29.603014+0000 mon.a (mon.0) 69 : cluster [INF] Active manager daemon y restarted 2024-02-19T20:56:29.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:29.603231+0000 mon.a (mon.0) 70 : cluster [INF] Activating manager daemon y 2024-02-19T20:56:29.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:29.606184+0000 mon.a (mon.0) 71 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-19T20:56:29.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:29.606284+0000 mon.a (mon.0) 72 : cluster [DBG] mgrmap e10: y(active, starting, since 0.00314555s) 2024-02-19T20:56:29.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:29.607170+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-19T20:56:29.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:29.607836+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-19T20:56:29.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:29.608582+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T20:56:29.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:29.608742+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T20:56:29.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:29.608885+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T20:56:29.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:29.622766+0000 mon.a (mon.0) 78 : cluster [INF] Manager daemon y is now available 2024-02-19T20:56:30.586 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.606+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mgrmap(e 11) v1 ==== 38489+0+0 (unknown 1053557347 0 0) 0x7ff47803f0b0 con 0x7ff4880fad90 2024-02-19T20:56:30.586 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.606+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 --> v1:172.21.15.163:6800/2735092549 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff470039110 con 0x7ff47803f790 2024-02-19T20:56:30.586 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.607+0000 7ff46ffff700 10 client.14150.objecter ms_handle_connect 0x7ff47803f790 2024-02-19T20:56:30.586 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.607+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (unknown 0 0 4198994610) 0x7ff470039110 con 0x7ff47803f790 2024-02-19T20:56:30.595 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.613+0000 7ff48d3be700 1 -- 172.21.15.163:0/2006993148 --> v1:172.21.15.163:6800/2735092549 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff474002440 con 0x7ff47803f790 2024-02-19T20:56:30.595 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.614+0000 7ff46ffff700 1 -- 172.21.15.163:0/2006993148 <== mgr.14146 v1:172.21.15.163:6800/2735092549 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (unknown 0 0 3086460295) 0x7ff474002440 con 0x7ff47803f790 2024-02-19T20:56:30.595 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: { 2024-02-19T20:56:30.595 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-19T20:56:30.595 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: "initialized": true 2024-02-19T20:56:30.595 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: } 2024-02-19T20:56:30.595 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.614+0000 7ff48d3be700 10 librados: watch_flush enter 2024-02-19T20:56:30.595 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.614+0000 7ff48d3be700 10 librados: watch_flush exit 2024-02-19T20:56:30.595 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.614+0000 7ff48d3be700 20 client.14150.objecter shutdown clearing up homeless session... 2024-02-19T20:56:30.595 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.614+0000 7ff48d3be700 10 client.14150.objecter successfully canceled tick 2024-02-19T20:56:30.595 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.614+0000 7ff48d3be700 1 -- 172.21.15.163:0/2006993148 >> v1:172.21.15.163:6800/2735092549 conn(0x7ff47803f790 legacy=0x7ff47003c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:30.595 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.614+0000 7ff48d3be700 1 -- 172.21.15.163:0/2006993148 >> v1:172.21.15.163:6789/0 conn(0x7ff4880fad90 legacy=0x7ff48810a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:30.596 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.614+0000 7ff48d3be700 1 -- 172.21.15.163:0/2006993148 shutdown_connections 2024-02-19T20:56:30.596 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.614+0000 7ff48d3be700 1 -- 172.21.15.163:0/2006993148 >> 172.21.15.163:0/2006993148 conn(0x7ff4880f64d0 msgr2=0x7ff48806fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:30.596 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.614+0000 7ff48d3be700 1 -- 172.21.15.163:0/2006993148 shutdown_connections 2024-02-19T20:56:30.596 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.614+0000 7ff48d3be700 1 -- 172.21.15.163:0/2006993148 wait complete. 2024-02-19T20:56:30.596 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:30.614+0000 7ff48d3be700 1 librados: shutdown 2024-02-19T20:56:30.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:29.715592+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:56:30.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:29.716952+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:56:30.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:29.741598+0000 mon.a (mon.0) 81 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-19T20:56:30.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:29.745039+0000 mon.a (mon.0) 82 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-19T20:56:30.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:30.608317+0000 mon.a (mon.0) 83 : cluster [DBG] mgrmap e11: y(active, since 1.00518s) 2024-02-19T20:56:31.523 INFO:teuthology.orchestra.run.smithi163.stderr:mgr epoch 9 is available 2024-02-19T20:56:31.523 INFO:teuthology.orchestra.run.smithi163.stderr:Generating a dashboard self-signed certificate... 2024-02-19T20:56:32.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.114+0000 7fc660ef4700 1 Processor -- start 2024-02-19T20:56:32.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.114+0000 7fc660ef4700 1 -- start start 2024-02-19T20:56:32.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.114+0000 7fc660ef4700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc65c100170 con 0x7fc65c070c60 2024-02-19T20:56:32.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.114+0000 7fc65a59c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fc65c070c60 0x7fc65c0fea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43872/0 (socket says 172.21.15.163:43872) 2024-02-19T20:56:32.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.114+0000 7fc65a59c700 1 -- 172.21.15.163:0/4289417809 learned_addr learned my addr 172.21.15.163:0/4289417809 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:32.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.114+0000 7fc65959a700 1 -- 172.21.15.163:0/4289417809 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987962398 0 0) 0x7fc65c100170 con 0x7fc65c070c60 2024-02-19T20:56:32.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.115+0000 7fc65959a700 1 -- 172.21.15.163:0/4289417809 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6440033f0 con 0x7fc65c070c60 2024-02-19T20:56:32.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.115+0000 7fc65959a700 1 -- 172.21.15.163:0/4289417809 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3801066347 0 0) 0x7fc6440033f0 con 0x7fc65c070c60 2024-02-19T20:56:32.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.115+0000 7fc65959a700 1 -- 172.21.15.163:0/4289417809 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c101310 con 0x7fc65c070c60 2024-02-19T20:56:32.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.115+0000 7fc65959a700 1 -- 172.21.15.163:0/4289417809 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc64c001590 con 0x7fc65c070c60 2024-02-19T20:56:32.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.115+0000 7fc65959a700 1 -- 172.21.15.163:0/4289417809 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fc64c001d20 con 0x7fc65c070c60 2024-02-19T20:56:32.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.115+0000 7fc65959a700 1 -- 172.21.15.163:0/4289417809 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc64c005780 con 0x7fc65c070c60 2024-02-19T20:56:32.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.115+0000 7fc660ef4700 1 -- 172.21.15.163:0/4289417809 >> v1:172.21.15.163:6789/0 conn(0x7fc65c070c60 legacy=0x7fc65c0fea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:32.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.116+0000 7fc660ef4700 1 -- 172.21.15.163:0/4289417809 shutdown_connections 2024-02-19T20:56:32.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.116+0000 7fc660ef4700 1 -- 172.21.15.163:0/4289417809 >> 172.21.15.163:0/4289417809 conn(0x7fc65c0f6510 msgr2=0x7fc65c0f8930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:32.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.116+0000 7fc660ef4700 1 -- 172.21.15.163:0/4289417809 shutdown_connections 2024-02-19T20:56:32.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.116+0000 7fc660ef4700 1 -- 172.21.15.163:0/4289417809 wait complete. 2024-02-19T20:56:32.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.116+0000 7fc660ef4700 1 librados: starting msgr at 2024-02-19T20:56:32.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.116+0000 7fc660ef4700 1 librados: starting objecter 2024-02-19T20:56:32.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.116+0000 7fc660ef4700 1 Processor -- start 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.116+0000 7fc660ef4700 1 -- start start 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.116+0000 7fc660ef4700 1 librados: setting wanted keys 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.116+0000 7fc660ef4700 1 librados: calling monclient init 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc660ef4700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc65c10aed0 con 0x7fc65c070c60 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc65a59c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fc65c070c60 0x7fc65c10a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43876/0 (socket says 172.21.15.163:43876) 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc65a59c700 1 -- 172.21.15.163:0/712411672 learned_addr learned my addr 172.21.15.163:0/712411672 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc64b7fe700 10 client.?.objecter ms_handle_connect 0x7fc65c070c60 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc64b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc64b7fe700 1 -- 172.21.15.163:0/712411672 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650042626 0 0) 0x7fc65c10aed0 con 0x7fc65c070c60 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc64b7fe700 1 -- 172.21.15.163:0/712411672 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc644005580 con 0x7fc65c070c60 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc64b7fe700 1 -- 172.21.15.163:0/712411672 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2655814559 0 0) 0x7fc644005580 con 0x7fc65c070c60 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc64b7fe700 1 -- 172.21.15.163:0/712411672 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc65c10aed0 con 0x7fc65c070c60 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc64b7fe700 1 -- 172.21.15.163:0/712411672 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc64c000fe0 con 0x7fc65c070c60 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc64b7fe700 1 -- 172.21.15.163:0/712411672 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 797541114 0 0) 0x7fc65c10aed0 con 0x7fc65c070c60 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc64b7fe700 1 -- 172.21.15.163:0/712411672 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c10b060 con 0x7fc65c070c60 2024-02-19T20:56:32.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc660ef4700 1 -- 172.21.15.163:0/712411672 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65c10b310 con 0x7fc65c070c60 2024-02-19T20:56:32.104 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc660ef4700 10 client.14156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:32.104 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc660ef4700 1 -- 172.21.15.163:0/712411672 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65c10f000 con 0x7fc65c070c60 2024-02-19T20:56:32.104 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.117+0000 7fc660ef4700 1 librados: init done 2024-02-19T20:56:32.104 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.118+0000 7fc64b7fe700 1 -- 172.21.15.163:0/712411672 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fc64c004970 con 0x7fc65c070c60 2024-02-19T20:56:32.104 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.118+0000 7fc64b7fe700 1 -- 172.21.15.163:0/712411672 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc64c006100 con 0x7fc65c070c60 2024-02-19T20:56:32.104 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.118+0000 7fc660ef4700 1 -- 172.21.15.163:0/712411672 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc63c004f50 con 0x7fc65c070c60 2024-02-19T20:56:32.104 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.119+0000 7fc64b7fe700 1 -- 172.21.15.163:0/712411672 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 11) v1 ==== 38489+0+0 (unknown 1053557347 0 0) 0x7fc64c001430 con 0x7fc65c070c60 2024-02-19T20:56:32.104 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.119+0000 7fc64b7fe700 1 -- 172.21.15.163:0/712411672 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 1228221725 0 0) 0x7fc64c03ea90 con 0x7fc65c070c60 2024-02-19T20:56:32.104 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.119+0000 7fc64b7fe700 10 client.14156.objecter ms_dispatch 0x7fc65c001550 osd_map(3..3 src has 1..3) v4 2024-02-19T20:56:32.104 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.119+0000 7fc64b7fe700 3 client.14156.objecter handle_osd_map got epochs [3,3] > 0 2024-02-19T20:56:32.104 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.119+0000 7fc64b7fe700 3 client.14156.objecter handle_osd_map decoding full epoch 3 2024-02-19T20:56:32.105 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.119+0000 7fc64b7fe700 20 client.14156.objecter dump_active .. 0 homeless 2024-02-19T20:56:32.105 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.119+0000 7fc64b7fe700 10 client.14156.objecter ms_handle_connect 0x7fc6440351b0 2024-02-19T20:56:32.105 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.124+0000 7fc64b7fe700 1 -- 172.21.15.163:0/712411672 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc64c013980 con 0x7fc65c070c60 2024-02-19T20:56:32.122 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.143+0000 7fc64b7fe700 1 -- 172.21.15.163:0/712411672 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mgrmap(e 12) v1 ==== 38595+0+0 (unknown 2865551019 0 0) 0x7fc64c03e160 con 0x7fc65c070c60 2024-02-19T20:56:32.277 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.297+0000 7fc660ef4700 1 -- 172.21.15.163:0/712411672 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fc63c000bc0 con 0x7fc6440351b0 2024-02-19T20:56:32.380 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.400+0000 7fc64b7fe700 1 -- 172.21.15.163:0/712411672 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (unknown 0 0 3317752739) 0x7fc63c000bc0 con 0x7fc6440351b0 2024-02-19T20:56:32.380 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-19T20:56:32.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.402+0000 7fc660ef4700 10 librados: watch_flush enter 2024-02-19T20:56:32.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.402+0000 7fc660ef4700 10 librados: watch_flush exit 2024-02-19T20:56:32.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.402+0000 7fc660ef4700 20 client.14156.objecter shutdown clearing up homeless session... 2024-02-19T20:56:32.383 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.402+0000 7fc660ef4700 10 client.14156.objecter successfully canceled tick 2024-02-19T20:56:32.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.402+0000 7fc660ef4700 1 -- 172.21.15.163:0/712411672 >> v1:172.21.15.163:6800/2735092549 conn(0x7fc6440351b0 legacy=0x7fc644037670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:32.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.402+0000 7fc660ef4700 1 -- 172.21.15.163:0/712411672 >> v1:172.21.15.163:6789/0 conn(0x7fc65c070c60 legacy=0x7fc65c10a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:32.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.402+0000 7fc660ef4700 1 -- 172.21.15.163:0/712411672 shutdown_connections 2024-02-19T20:56:32.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.402+0000 7fc660ef4700 1 -- 172.21.15.163:0/712411672 >> 172.21.15.163:0/712411672 conn(0x7fc65c0f6510 msgr2=0x7fc65c0fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:32.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.403+0000 7fc660ef4700 1 -- 172.21.15.163:0/712411672 shutdown_connections 2024-02-19T20:56:32.384 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.403+0000 7fc660ef4700 1 -- 172.21.15.163:0/712411672 wait complete. 2024-02-19T20:56:32.385 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:32.403+0000 7fc660ef4700 1 librados: shutdown 2024-02-19T20:56:32.405 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:30.607755+0000 mgr.y (mgr.14146) 1 : audit [DBG] from='client.14150 v1:172.21.15.163:0/2006993148' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T20:56:32.405 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:30.614310+0000 mgr.y (mgr.14146) 2 : audit [DBG] from='client.14150 v1:172.21.15.163:0/2006993148' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T20:56:32.405 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:31.143492+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:56:33.378 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:32.144932+0000 mon.a (mon.0) 85 : cluster [DBG] mgrmap e12: y(active, since 2s) 2024-02-19T20:56:33.378 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:32.399137+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:56:33.378 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:32.400908+0000 mon.a (mon.0) 87 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:56:33.878 INFO:teuthology.orchestra.run.smithi163.stderr:Creating initial admin user... 2024-02-19T20:56:34.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:32.298597+0000 mgr.y (mgr.14146) 3 : audit [DBG] from='client.14156 v1:172.21.15.163:0/712411672' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:56:34.467 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.480+0000 7f2dd31ec700 1 Processor -- start 2024-02-19T20:56:34.467 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.480+0000 7f2dd31ec700 1 -- start start 2024-02-19T20:56:34.467 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.480+0000 7f2dd31ec700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2dcc100190 con 0x7f2dcc070c40 2024-02-19T20:56:34.467 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.481+0000 7f2dd0f88700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f2dcc070c40 0x7f2dcc0fea70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57794/0 (socket says 172.21.15.163:57794) 2024-02-19T20:56:34.467 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.481+0000 7f2dd0f88700 1 -- 172.21.15.163:0/1022887354 learned_addr learned my addr 172.21.15.163:0/1022887354 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:34.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.481+0000 7f2dcb7fe700 1 -- 172.21.15.163:0/1022887354 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205346556 0 0) 0x7f2dcc100190 con 0x7f2dcc070c40 2024-02-19T20:56:34.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.481+0000 7f2dcb7fe700 1 -- 172.21.15.163:0/1022887354 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dbc0033f0 con 0x7f2dcc070c40 2024-02-19T20:56:34.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.481+0000 7f2dcb7fe700 1 -- 172.21.15.163:0/1022887354 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3743035762 0 0) 0x7f2dbc0033f0 con 0x7f2dcc070c40 2024-02-19T20:56:34.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.481+0000 7f2dcb7fe700 1 -- 172.21.15.163:0/1022887354 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc101330 con 0x7f2dcc070c40 2024-02-19T20:56:34.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.481+0000 7f2dcb7fe700 1 -- 172.21.15.163:0/1022887354 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f2dc0001590 con 0x7f2dcc070c40 2024-02-19T20:56:34.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.481+0000 7f2dcb7fe700 1 -- 172.21.15.163:0/1022887354 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f2dc0001d20 con 0x7f2dcc070c40 2024-02-19T20:56:34.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.481+0000 7f2dcb7fe700 1 -- 172.21.15.163:0/1022887354 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f2dc0005780 con 0x7f2dcc070c40 2024-02-19T20:56:34.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.482+0000 7f2dd31ec700 1 -- 172.21.15.163:0/1022887354 >> v1:172.21.15.163:6789/0 conn(0x7f2dcc070c40 legacy=0x7f2dcc0fea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:34.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.482+0000 7f2dd31ec700 1 -- 172.21.15.163:0/1022887354 shutdown_connections 2024-02-19T20:56:34.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.482+0000 7f2dd31ec700 1 -- 172.21.15.163:0/1022887354 >> 172.21.15.163:0/1022887354 conn(0x7f2dcc0f64d0 msgr2=0x7f2dcc0f8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:34.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.482+0000 7f2dd31ec700 1 -- 172.21.15.163:0/1022887354 shutdown_connections 2024-02-19T20:56:34.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.482+0000 7f2dd31ec700 1 -- 172.21.15.163:0/1022887354 wait complete. 2024-02-19T20:56:34.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.482+0000 7f2dd31ec700 1 librados: starting msgr at 2024-02-19T20:56:34.468 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.482+0000 7f2dd31ec700 1 librados: starting objecter 2024-02-19T20:56:34.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.482+0000 7f2dd31ec700 1 Processor -- start 2024-02-19T20:56:34.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.482+0000 7f2dd31ec700 1 -- start start 2024-02-19T20:56:34.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.482+0000 7f2dd31ec700 1 librados: setting wanted keys 2024-02-19T20:56:34.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.482+0000 7f2dd31ec700 1 librados: calling monclient init 2024-02-19T20:56:34.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.482+0000 7f2dd31ec700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2dcc0fc480 con 0x7f2dcc070c40 2024-02-19T20:56:34.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.482+0000 7f2dd0f88700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f2dcc070c40 0x7f2dcc0fa970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57806/0 (socket says 172.21.15.163:57806) 2024-02-19T20:56:34.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.482+0000 7f2dd0f88700 1 -- 172.21.15.163:0/4006186197 learned_addr learned my addr 172.21.15.163:0/4006186197 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:34.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.483+0000 7f2dc9ffb700 10 client.?.objecter ms_handle_connect 0x7f2dcc070c40 2024-02-19T20:56:34.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.483+0000 7f2dc9ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:34.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.483+0000 7f2dc9ffb700 1 -- 172.21.15.163:0/4006186197 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177032735 0 0) 0x7f2dcc0fc480 con 0x7f2dcc070c40 2024-02-19T20:56:34.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.483+0000 7f2dc9ffb700 1 -- 172.21.15.163:0/4006186197 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dbc005580 con 0x7f2dcc070c40 2024-02-19T20:56:34.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.483+0000 7f2dc9ffb700 1 -- 172.21.15.163:0/4006186197 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3517159012 0 0) 0x7f2dbc005580 con 0x7f2dcc070c40 2024-02-19T20:56:34.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.483+0000 7f2dc9ffb700 1 -- 172.21.15.163:0/4006186197 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dcc0fc480 con 0x7f2dcc070c40 2024-02-19T20:56:34.469 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.483+0000 7f2dc9ffb700 1 -- 172.21.15.163:0/4006186197 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f2dc0000fe0 con 0x7f2dcc070c40 2024-02-19T20:56:34.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.483+0000 7f2dc9ffb700 1 -- 172.21.15.163:0/4006186197 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2505333969 0 0) 0x7f2dcc0fc480 con 0x7f2dcc070c40 2024-02-19T20:56:34.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.483+0000 7f2dc9ffb700 1 -- 172.21.15.163:0/4006186197 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc0fc610 con 0x7f2dcc070c40 2024-02-19T20:56:34.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.483+0000 7f2dd31ec700 1 -- 172.21.15.163:0/4006186197 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dcc0fc8c0 con 0x7f2dcc070c40 2024-02-19T20:56:34.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.483+0000 7f2dc9ffb700 1 -- 172.21.15.163:0/4006186197 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f2dc0004970 con 0x7f2dcc070c40 2024-02-19T20:56:34.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.483+0000 7f2dc9ffb700 1 -- 172.21.15.163:0/4006186197 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f2dc0006240 con 0x7f2dcc070c40 2024-02-19T20:56:34.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.483+0000 7f2dd31ec700 10 client.14158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:34.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.483+0000 7f2dd31ec700 1 -- 172.21.15.163:0/4006186197 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dcc0fb380 con 0x7f2dcc070c40 2024-02-19T20:56:34.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.483+0000 7f2dd31ec700 1 librados: init done 2024-02-19T20:56:34.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.484+0000 7f2dd31ec700 1 -- 172.21.15.163:0/4006186197 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dcc046ce0 con 0x7f2dcc070c40 2024-02-19T20:56:34.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.485+0000 7f2dc9ffb700 1 -- 172.21.15.163:0/4006186197 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 12) v1 ==== 38595+0+0 (unknown 2865551019 0 0) 0x7f2dc0001430 con 0x7f2dcc070c40 2024-02-19T20:56:34.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.485+0000 7f2dc9ffb700 1 -- 172.21.15.163:0/4006186197 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 1228221725 0 0) 0x7f2dc003eb60 con 0x7f2dcc070c40 2024-02-19T20:56:34.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.485+0000 7f2dc9ffb700 10 client.14158.objecter ms_dispatch 0x7f2dcc001550 osd_map(3..3 src has 1..3) v4 2024-02-19T20:56:34.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.485+0000 7f2dc9ffb700 3 client.14158.objecter handle_osd_map got epochs [3,3] > 0 2024-02-19T20:56:34.470 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.485+0000 7f2dc9ffb700 3 client.14158.objecter handle_osd_map decoding full epoch 3 2024-02-19T20:56:34.471 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.485+0000 7f2dc9ffb700 20 client.14158.objecter dump_active .. 0 homeless 2024-02-19T20:56:34.471 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.485+0000 7f2dc9ffb700 10 client.14158.objecter ms_handle_connect 0x7f2dbc0352d0 2024-02-19T20:56:34.471 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.489+0000 7f2dc9ffb700 1 -- 172.21.15.163:0/4006186197 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f2dc00138f0 con 0x7f2dcc070c40 2024-02-19T20:56:34.647 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.668+0000 7f2dd31ec700 1 -- 172.21.15.163:0/4006186197 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f2dcc100f90 con 0x7f2dbc0352d0 2024-02-19T20:56:34.873 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.894+0000 7f2dc9ffb700 1 -- 172.21.15.163:0/4006186197 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (unknown 0 0 2027259617) 0x7f2dcc100f90 con 0x7f2dbc0352d0 2024-02-19T20:56:34.874 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$MgWQM2EnummlymPDS2EWS.lQABzLC1z0vrxmcEg67gh6028Q7D0u2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708376194, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-19T20:56:34.876 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.896+0000 7f2dd31ec700 10 librados: watch_flush enter 2024-02-19T20:56:34.876 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.896+0000 7f2dd31ec700 10 librados: watch_flush exit 2024-02-19T20:56:34.876 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.896+0000 7f2dd31ec700 20 client.14158.objecter shutdown clearing up homeless session... 2024-02-19T20:56:34.876 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.896+0000 7f2dd31ec700 10 client.14158.objecter successfully canceled tick 2024-02-19T20:56:34.877 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.896+0000 7f2dd31ec700 1 -- 172.21.15.163:0/4006186197 >> v1:172.21.15.163:6800/2735092549 conn(0x7f2dbc0352d0 legacy=0x7f2dbc037790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:34.877 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.896+0000 7f2dd31ec700 1 -- 172.21.15.163:0/4006186197 >> v1:172.21.15.163:6789/0 conn(0x7f2dcc070c40 legacy=0x7f2dcc0fa970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:34.877 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.896+0000 7f2dd31ec700 1 -- 172.21.15.163:0/4006186197 shutdown_connections 2024-02-19T20:56:34.877 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.896+0000 7f2dd31ec700 1 -- 172.21.15.163:0/4006186197 >> 172.21.15.163:0/4006186197 conn(0x7f2dcc0f64d0 msgr2=0x7f2dcc0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:34.877 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.897+0000 7f2dd31ec700 1 -- 172.21.15.163:0/4006186197 shutdown_connections 2024-02-19T20:56:34.877 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.897+0000 7f2dd31ec700 1 -- 172.21.15.163:0/4006186197 wait complete. 2024-02-19T20:56:34.877 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:34.897+0000 7f2dd31ec700 1 librados: shutdown 2024-02-19T20:56:35.527 INFO:teuthology.orchestra.run.smithi163.stderr:Fetching dashboard port number... 2024-02-19T20:56:36.100 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.116+0000 7fec0a919700 1 Processor -- start 2024-02-19T20:56:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.116+0000 7fec0a919700 1 -- start start 2024-02-19T20:56:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.116+0000 7fec0a919700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fec041001c0 con 0x7fec04070c20 2024-02-19T20:56:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.116+0000 7fec03fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fec04070c20 0x7fec040feaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57814/0 (socket says 172.21.15.163:57814) 2024-02-19T20:56:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.116+0000 7fec03fff700 1 -- 172.21.15.163:0/1599970989 learned_addr learned my addr 172.21.15.163:0/1599970989 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.117+0000 7fec02ffd700 1 -- 172.21.15.163:0/1599970989 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406304770 0 0) 0x7fec041001c0 con 0x7fec04070c20 2024-02-19T20:56:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.117+0000 7fec02ffd700 1 -- 172.21.15.163:0/1599970989 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febec0033f0 con 0x7fec04070c20 2024-02-19T20:56:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.117+0000 7fec02ffd700 1 -- 172.21.15.163:0/1599970989 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2254271775 0 0) 0x7febec0033f0 con 0x7fec04070c20 2024-02-19T20:56:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.117+0000 7fec02ffd700 1 -- 172.21.15.163:0/1599970989 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec04101360 con 0x7fec04070c20 2024-02-19T20:56:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.117+0000 7fec02ffd700 1 -- 172.21.15.163:0/1599970989 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7febf4001590 con 0x7fec04070c20 2024-02-19T20:56:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.117+0000 7fec02ffd700 1 -- 172.21.15.163:0/1599970989 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7febf4001d20 con 0x7fec04070c20 2024-02-19T20:56:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.117+0000 7fec02ffd700 1 -- 172.21.15.163:0/1599970989 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7febf4005710 con 0x7fec04070c20 2024-02-19T20:56:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.118+0000 7fec0a919700 1 -- 172.21.15.163:0/1599970989 >> v1:172.21.15.163:6789/0 conn(0x7fec04070c20 legacy=0x7fec040feaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.118+0000 7fec0a919700 1 -- 172.21.15.163:0/1599970989 shutdown_connections 2024-02-19T20:56:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.118+0000 7fec0a919700 1 -- 172.21.15.163:0/1599970989 >> 172.21.15.163:0/1599970989 conn(0x7fec040f6500 msgr2=0x7fec040f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:36.101 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.118+0000 7fec0a919700 1 -- 172.21.15.163:0/1599970989 shutdown_connections 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.118+0000 7fec0a919700 1 -- 172.21.15.163:0/1599970989 wait complete. 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.118+0000 7fec0a919700 1 librados: starting msgr at 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.118+0000 7fec0a919700 1 librados: starting objecter 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.118+0000 7fec0a919700 1 Processor -- start 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.119+0000 7fec0a919700 1 -- start start 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.119+0000 7fec0a919700 1 librados: setting wanted keys 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.119+0000 7fec0a919700 1 librados: calling monclient init 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.119+0000 7fec0a919700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fec0410aed0 con 0x7fec04070c20 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.119+0000 7fec03fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fec04070c20 0x7fec0410a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57816/0 (socket says 172.21.15.163:57816) 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.119+0000 7fec03fff700 1 -- 172.21.15.163:0/2865585915 learned_addr learned my addr 172.21.15.163:0/2865585915 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.119+0000 7fec017fa700 10 client.?.objecter ms_handle_connect 0x7fec04070c20 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.119+0000 7fec017fa700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.119+0000 7fec017fa700 1 -- 172.21.15.163:0/2865585915 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947188377 0 0) 0x7fec0410aed0 con 0x7fec04070c20 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.119+0000 7fec017fa700 1 -- 172.21.15.163:0/2865585915 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febec005580 con 0x7fec04070c20 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.119+0000 7fec017fa700 1 -- 172.21.15.163:0/2865585915 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2352283728 0 0) 0x7febec005580 con 0x7fec04070c20 2024-02-19T20:56:36.102 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.120+0000 7fec017fa700 1 -- 172.21.15.163:0/2865585915 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec0410aed0 con 0x7fec04070c20 2024-02-19T20:56:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.120+0000 7fec017fa700 1 -- 172.21.15.163:0/2865585915 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7febf4004800 con 0x7fec04070c20 2024-02-19T20:56:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.120+0000 7fec017fa700 1 -- 172.21.15.163:0/2865585915 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2273129194 0 0) 0x7fec0410aed0 con 0x7fec04070c20 2024-02-19T20:56:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.120+0000 7fec017fa700 1 -- 172.21.15.163:0/2865585915 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0410b060 con 0x7fec04070c20 2024-02-19T20:56:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.120+0000 7fec0a919700 1 -- 172.21.15.163:0/2865585915 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec0410b310 con 0x7fec04070c20 2024-02-19T20:56:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.120+0000 7fec0a919700 10 client.14160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.120+0000 7fec0a919700 1 -- 172.21.15.163:0/2865585915 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec0410f000 con 0x7fec04070c20 2024-02-19T20:56:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.120+0000 7fec0a919700 1 librados: init done 2024-02-19T20:56:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.120+0000 7fec017fa700 1 -- 172.21.15.163:0/2865585915 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7febf4001100 con 0x7fec04070c20 2024-02-19T20:56:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.120+0000 7fec017fa700 1 -- 172.21.15.163:0/2865585915 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7febf4006120 con 0x7fec04070c20 2024-02-19T20:56:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.121+0000 7fec017fa700 1 -- 172.21.15.163:0/2865585915 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 4600635 0 0) 0x7febf400fa40 con 0x7fec04070c20 2024-02-19T20:56:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.121+0000 7fec017fa700 1 -- 172.21.15.163:0/2865585915 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 1228221725 0 0) 0x7febf403fd60 con 0x7fec04070c20 2024-02-19T20:56:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.121+0000 7fec017fa700 10 client.14160.objecter ms_dispatch 0x7fec04001550 osd_map(3..3 src has 1..3) v4 2024-02-19T20:56:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.121+0000 7fec017fa700 3 client.14160.objecter handle_osd_map got epochs [3,3] > 0 2024-02-19T20:56:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.121+0000 7fec017fa700 3 client.14160.objecter handle_osd_map decoding full epoch 3 2024-02-19T20:56:36.103 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.121+0000 7fec017fa700 20 client.14160.objecter dump_active .. 0 homeless 2024-02-19T20:56:36.104 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.121+0000 7fec0a919700 1 -- 172.21.15.163:0/2865585915 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf0004f50 con 0x7fec04070c20 2024-02-19T20:56:36.104 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.121+0000 7fec017fa700 10 client.14160.objecter ms_handle_connect 0x7febec035460 2024-02-19T20:56:36.104 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.125+0000 7fec017fa700 1 -- 172.21.15.163:0/2865585915 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7febf4014e00 con 0x7fec04070c20 2024-02-19T20:56:36.227 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:34.669298+0000 mgr.y (mgr.14146) 4 : audit [DBG] from='client.14158 v1:172.21.15.163:0/4006186197' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:56:36.227 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:34.894525+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:56:36.267 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.287+0000 7fec0a919700 1 -- 172.21.15.163:0/2865585915 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7febf0005780 con 0x7fec04070c20 2024-02-19T20:56:36.267 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.288+0000 7fec017fa700 1 -- 172.21.15.163:0/2865585915 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (unknown 3713421687 0 83753974) 0x7febf4013270 con 0x7fec04070c20 2024-02-19T20:56:36.267 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 8443 2024-02-19T20:56:36.269 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.289+0000 7fec0a919700 10 librados: watch_flush enter 2024-02-19T20:56:36.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.290+0000 7fec0a919700 10 librados: watch_flush exit 2024-02-19T20:56:36.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.290+0000 7fec0a919700 20 client.14160.objecter shutdown clearing up homeless session... 2024-02-19T20:56:36.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.290+0000 7fec0a919700 10 client.14160.objecter successfully canceled tick 2024-02-19T20:56:36.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.290+0000 7fec0a919700 1 -- 172.21.15.163:0/2865585915 >> v1:172.21.15.163:6800/2735092549 conn(0x7febec035460 legacy=0x7febec037900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:36.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.290+0000 7fec0a919700 1 -- 172.21.15.163:0/2865585915 >> v1:172.21.15.163:6789/0 conn(0x7fec04070c20 legacy=0x7fec0410a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:36.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.290+0000 7fec0a919700 1 -- 172.21.15.163:0/2865585915 shutdown_connections 2024-02-19T20:56:36.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.290+0000 7fec0a919700 1 -- 172.21.15.163:0/2865585915 >> 172.21.15.163:0/2865585915 conn(0x7fec040f6500 msgr2=0x7fec040fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:36.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.290+0000 7fec0a919700 1 -- 172.21.15.163:0/2865585915 shutdown_connections 2024-02-19T20:56:36.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.290+0000 7fec0a919700 1 -- 172.21.15.163:0/2865585915 wait complete. 2024-02-19T20:56:36.270 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:36.290+0000 7fec0a919700 1 librados: shutdown 2024-02-19T20:56:36.876 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: disabled 2024-02-19T20:56:36.884 INFO:teuthology.orchestra.run.smithi163.stderr:systemctl: inactive 2024-02-19T20:56:36.884 INFO:teuthology.orchestra.run.smithi163.stderr:firewalld.service is not enabled 2024-02-19T20:56:36.885 INFO:teuthology.orchestra.run.smithi163.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-19T20:56:36.887 INFO:teuthology.orchestra.run.smithi163.stderr:Ceph Dashboard is now available at: 2024-02-19T20:56:36.887 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-02-19T20:56:36.887 INFO:teuthology.orchestra.run.smithi163.stderr: URL: https://smithi163.front.sepia.ceph.com:8443/ 2024-02-19T20:56:36.887 INFO:teuthology.orchestra.run.smithi163.stderr: User: admin 2024-02-19T20:56:36.888 INFO:teuthology.orchestra.run.smithi163.stderr: Password: 6x0gfj2rss 2024-02-19T20:56:36.888 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-02-19T20:56:36.888 INFO:teuthology.orchestra.run.smithi163.stderr:Enabling autotune for osd_memory_target 2024-02-19T20:56:36.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:35.895993+0000 mon.a (mon.0) 89 : cluster [DBG] mgrmap e13: y(active, since 6s) 2024-02-19T20:56:36.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:36.288423+0000 mon.a (mon.0) 90 : audit [DBG] from='client.? v1:172.21.15.163:0/2865585915' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-19T20:56:37.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.474+0000 7fb73d756700 1 Processor -- start 2024-02-19T20:56:37.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.474+0000 7fb73d756700 1 -- start start 2024-02-19T20:56:37.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.474+0000 7fb73d756700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb7381001a0 con 0x7fb738070c20 2024-02-19T20:56:37.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.474+0000 7fb736ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fb738070c20 0x7fb7380fea80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57818/0 (socket says 172.21.15.163:57818) 2024-02-19T20:56:37.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.474+0000 7fb736ffd700 1 -- 172.21.15.163:0/2016930246 learned_addr learned my addr 172.21.15.163:0/2016930246 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:37.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.474+0000 7fb735ffb700 1 -- 172.21.15.163:0/2016930246 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383917314 0 0) 0x7fb7381001a0 con 0x7fb738070c20 2024-02-19T20:56:37.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.474+0000 7fb735ffb700 1 -- 172.21.15.163:0/2016930246 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7200033f0 con 0x7fb738070c20 2024-02-19T20:56:37.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.475+0000 7fb735ffb700 1 -- 172.21.15.163:0/2016930246 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4020037585 0 0) 0x7fb7200033f0 con 0x7fb738070c20 2024-02-19T20:56:37.456 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.475+0000 7fb735ffb700 1 -- 172.21.15.163:0/2016930246 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb738101340 con 0x7fb738070c20 2024-02-19T20:56:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.475+0000 7fb735ffb700 1 -- 172.21.15.163:0/2016930246 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fb728001590 con 0x7fb738070c20 2024-02-19T20:56:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.475+0000 7fb735ffb700 1 -- 172.21.15.163:0/2016930246 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb728001c50 con 0x7fb738070c20 2024-02-19T20:56:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.475+0000 7fb735ffb700 1 -- 172.21.15.163:0/2016930246 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fb728005640 con 0x7fb738070c20 2024-02-19T20:56:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.475+0000 7fb73d756700 1 -- 172.21.15.163:0/2016930246 >> v1:172.21.15.163:6789/0 conn(0x7fb738070c20 legacy=0x7fb7380fea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.475+0000 7fb73d756700 1 -- 172.21.15.163:0/2016930246 shutdown_connections 2024-02-19T20:56:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.475+0000 7fb73d756700 1 -- 172.21.15.163:0/2016930246 >> 172.21.15.163:0/2016930246 conn(0x7fb7380f6520 msgr2=0x7fb7380f8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.475+0000 7fb73d756700 1 -- 172.21.15.163:0/2016930246 shutdown_connections 2024-02-19T20:56:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.475+0000 7fb73d756700 1 -- 172.21.15.163:0/2016930246 wait complete. 2024-02-19T20:56:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.476+0000 7fb73d756700 1 librados: starting msgr at 2024-02-19T20:56:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.476+0000 7fb73d756700 1 librados: starting objecter 2024-02-19T20:56:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.476+0000 7fb73d756700 1 Processor -- start 2024-02-19T20:56:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.476+0000 7fb73d756700 1 -- start start 2024-02-19T20:56:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.476+0000 7fb73d756700 1 librados: setting wanted keys 2024-02-19T20:56:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.476+0000 7fb73d756700 1 librados: calling monclient init 2024-02-19T20:56:37.458 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.476+0000 7fb73d756700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb73810aec0 con 0x7fb738070c20 2024-02-19T20:56:37.458 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.476+0000 7fb736ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fb738070c20 0x7fb73810a7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57828/0 (socket says 172.21.15.163:57828) 2024-02-19T20:56:37.458 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.476+0000 7fb736ffd700 1 -- 172.21.15.163:0/2907751405 learned_addr learned my addr 172.21.15.163:0/2907751405 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:37.458 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.477+0000 7fb71ffff700 10 client.?.objecter ms_handle_connect 0x7fb738070c20 2024-02-19T20:56:37.459 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.477+0000 7fb71ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:37.459 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.477+0000 7fb71ffff700 1 -- 172.21.15.163:0/2907751405 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2266977594 0 0) 0x7fb73810aec0 con 0x7fb738070c20 2024-02-19T20:56:37.459 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.477+0000 7fb71ffff700 1 -- 172.21.15.163:0/2907751405 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7200055d0 con 0x7fb738070c20 2024-02-19T20:56:37.459 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.477+0000 7fb71ffff700 1 -- 172.21.15.163:0/2907751405 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3901106479 0 0) 0x7fb7200055d0 con 0x7fb738070c20 2024-02-19T20:56:37.459 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.477+0000 7fb71ffff700 1 -- 172.21.15.163:0/2907751405 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb73810aec0 con 0x7fb738070c20 2024-02-19T20:56:37.459 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.477+0000 7fb71ffff700 1 -- 172.21.15.163:0/2907751405 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fb728004a70 con 0x7fb738070c20 2024-02-19T20:56:37.459 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.477+0000 7fb71ffff700 1 -- 172.21.15.163:0/2907751405 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4282443353 0 0) 0x7fb73810aec0 con 0x7fb738070c20 2024-02-19T20:56:37.459 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.477+0000 7fb71ffff700 1 -- 172.21.15.163:0/2907751405 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73810b050 con 0x7fb738070c20 2024-02-19T20:56:37.459 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.477+0000 7fb73d756700 1 -- 172.21.15.163:0/2907751405 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73810b300 con 0x7fb738070c20 2024-02-19T20:56:37.459 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.477+0000 7fb73d756700 10 client.14162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.477+0000 7fb73d756700 1 -- 172.21.15.163:0/2907751405 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73810eff0 con 0x7fb738070c20 2024-02-19T20:56:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.477+0000 7fb73d756700 1 librados: init done 2024-02-19T20:56:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.477+0000 7fb71ffff700 1 -- 172.21.15.163:0/2907751405 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb7280056b0 con 0x7fb738070c20 2024-02-19T20:56:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.477+0000 7fb71ffff700 1 -- 172.21.15.163:0/2907751405 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fb728006120 con 0x7fb738070c20 2024-02-19T20:56:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.478+0000 7fb71ffff700 1 -- 172.21.15.163:0/2907751405 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 4600635 0 0) 0x7fb72800f980 con 0x7fb738070c20 2024-02-19T20:56:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.478+0000 7fb73d756700 1 -- 172.21.15.163:0/2907751405 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb724004f50 con 0x7fb738070c20 2024-02-19T20:56:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.478+0000 7fb71ffff700 1 -- 172.21.15.163:0/2907751405 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 1228221725 0 0) 0x7fb72803ee30 con 0x7fb738070c20 2024-02-19T20:56:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.478+0000 7fb71ffff700 10 client.14162.objecter ms_dispatch 0x7fb738001550 osd_map(3..3 src has 1..3) v4 2024-02-19T20:56:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.478+0000 7fb71ffff700 3 client.14162.objecter handle_osd_map got epochs [3,3] > 0 2024-02-19T20:56:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.478+0000 7fb71ffff700 3 client.14162.objecter handle_osd_map decoding full epoch 3 2024-02-19T20:56:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.478+0000 7fb71ffff700 20 client.14162.objecter dump_active .. 0 homeless 2024-02-19T20:56:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.479+0000 7fb71ffff700 10 client.14162.objecter ms_handle_connect 0x7fb720035440 2024-02-19T20:56:37.462 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.483+0000 7fb71ffff700 1 -- 172.21.15.163:0/2907751405 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb728011160 con 0x7fb738070c20 2024-02-19T20:56:37.628 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.649+0000 7fb73d756700 1 -- 172.21.15.163:0/2907751405 --> v1:172.21.15.163:6789/0 -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fb724005780 con 0x7fb738070c20 2024-02-19T20:56:37.628 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.649+0000 7fb71ffff700 1 -- 172.21.15.163:0/2907751405 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (unknown 689008156 0 0) 0x7fb728013260 con 0x7fb738070c20 2024-02-19T20:56:37.631 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.651+0000 7fb73d756700 10 librados: watch_flush enter 2024-02-19T20:56:37.631 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.651+0000 7fb73d756700 10 librados: watch_flush exit 2024-02-19T20:56:37.631 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.651+0000 7fb73d756700 20 client.14162.objecter shutdown clearing up homeless session... 2024-02-19T20:56:37.631 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.651+0000 7fb73d756700 10 client.14162.objecter successfully canceled tick 2024-02-19T20:56:37.631 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.651+0000 7fb73d756700 1 -- 172.21.15.163:0/2907751405 >> v1:172.21.15.163:6800/2735092549 conn(0x7fb720035440 legacy=0x7fb720037900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:37.631 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.651+0000 7fb73d756700 1 -- 172.21.15.163:0/2907751405 >> v1:172.21.15.163:6789/0 conn(0x7fb738070c20 legacy=0x7fb73810a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:37.631 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.651+0000 7fb73d756700 1 -- 172.21.15.163:0/2907751405 shutdown_connections 2024-02-19T20:56:37.631 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.651+0000 7fb73d756700 1 -- 172.21.15.163:0/2907751405 >> 172.21.15.163:0/2907751405 conn(0x7fb7380f6520 msgr2=0x7fb7380fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:37.631 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.651+0000 7fb73d756700 1 -- 172.21.15.163:0/2907751405 shutdown_connections 2024-02-19T20:56:37.631 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.651+0000 7fb73d756700 1 -- 172.21.15.163:0/2907751405 wait complete. 2024-02-19T20:56:37.631 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:37.651+0000 7fb73d756700 1 librados: shutdown 2024-02-19T20:56:39.011 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.027+0000 7f88bc7e0700 1 Processor -- start 2024-02-19T20:56:39.012 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.027+0000 7f88bc7e0700 1 -- start start 2024-02-19T20:56:39.012 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.028+0000 7f88bc7e0700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f88b4100180 con 0x7f88b4070c60 2024-02-19T20:56:39.012 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.028+0000 7f88ba57c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f88b4070c60 0x7f88b40fea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57842/0 (socket says 172.21.15.163:57842) 2024-02-19T20:56:39.012 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.028+0000 7f88ba57c700 1 -- 172.21.15.163:0/905142140 learned_addr learned my addr 172.21.15.163:0/905142140 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:39.012 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.028+0000 7f88b957a700 1 -- 172.21.15.163:0/905142140 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455255998 0 0) 0x7f88b4100180 con 0x7f88b4070c60 2024-02-19T20:56:39.012 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.028+0000 7f88b957a700 1 -- 172.21.15.163:0/905142140 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a00033f0 con 0x7f88b4070c60 2024-02-19T20:56:39.012 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.028+0000 7f88b957a700 1 -- 172.21.15.163:0/905142140 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 216420983 0 0) 0x7f88a00033f0 con 0x7f88b4070c60 2024-02-19T20:56:39.012 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.028+0000 7f88b957a700 1 -- 172.21.15.163:0/905142140 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b4101320 con 0x7f88b4070c60 2024-02-19T20:56:39.012 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.028+0000 7f88b957a700 1 -- 172.21.15.163:0/905142140 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f88a4001590 con 0x7f88b4070c60 2024-02-19T20:56:39.012 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.028+0000 7f88b957a700 1 -- 172.21.15.163:0/905142140 <== mon.0 v1:172.21.15.163:6789/0 4 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f88a4001d20 con 0x7f88b4070c60 2024-02-19T20:56:39.012 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.028+0000 7f88b957a700 1 -- 172.21.15.163:0/905142140 <== mon.0 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f88a4005780 con 0x7f88b4070c60 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.029+0000 7f88bc7e0700 1 -- 172.21.15.163:0/905142140 >> v1:172.21.15.163:6789/0 conn(0x7f88b4070c60 legacy=0x7f88b40fea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.029+0000 7f88bc7e0700 1 -- 172.21.15.163:0/905142140 shutdown_connections 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.029+0000 7f88bc7e0700 1 -- 172.21.15.163:0/905142140 >> 172.21.15.163:0/905142140 conn(0x7f88b40f6500 msgr2=0x7f88b40f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.029+0000 7f88bc7e0700 1 -- 172.21.15.163:0/905142140 shutdown_connections 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.029+0000 7f88bc7e0700 1 -- 172.21.15.163:0/905142140 wait complete. 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.029+0000 7f88bc7e0700 1 librados: starting msgr at 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.029+0000 7f88bc7e0700 1 librados: starting objecter 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.030+0000 7f88bc7e0700 1 Processor -- start 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.030+0000 7f88bc7e0700 1 -- start start 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.030+0000 7f88bc7e0700 1 librados: setting wanted keys 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.030+0000 7f88bc7e0700 1 librados: calling monclient init 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.030+0000 7f88bc7e0700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f88b40fc470 con 0x7f88b4070c60 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.030+0000 7f88ba57c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f88b4070c60 0x7f88b40fa960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57858/0 (socket says 172.21.15.163:57858) 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.030+0000 7f88ba57c700 1 -- 172.21.15.163:0/3803083282 learned_addr learned my addr 172.21.15.163:0/3803083282 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.030+0000 7f88ab7fe700 10 client.?.objecter ms_handle_connect 0x7f88b4070c60 2024-02-19T20:56:39.013 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.030+0000 7f88ab7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:39.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.030+0000 7f88ab7fe700 1 -- 172.21.15.163:0/3803083282 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601543723 0 0) 0x7f88b40fc470 con 0x7f88b4070c60 2024-02-19T20:56:39.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.030+0000 7f88ab7fe700 1 -- 172.21.15.163:0/3803083282 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a0005580 con 0x7f88b4070c60 2024-02-19T20:56:39.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.030+0000 7f88ab7fe700 1 -- 172.21.15.163:0/3803083282 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2357454796 0 0) 0x7f88a0005580 con 0x7f88b4070c60 2024-02-19T20:56:39.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.030+0000 7f88ab7fe700 1 -- 172.21.15.163:0/3803083282 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88b40fc470 con 0x7f88b4070c60 2024-02-19T20:56:39.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.030+0000 7f88ab7fe700 1 -- 172.21.15.163:0/3803083282 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f88a4000fe0 con 0x7f88b4070c60 2024-02-19T20:56:39.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.031+0000 7f88ab7fe700 1 -- 172.21.15.163:0/3803083282 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1793650982 0 0) 0x7f88b40fc470 con 0x7f88b4070c60 2024-02-19T20:56:39.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.031+0000 7f88ab7fe700 1 -- 172.21.15.163:0/3803083282 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b40fc600 con 0x7f88b4070c60 2024-02-19T20:56:39.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.031+0000 7f88bc7e0700 1 -- 172.21.15.163:0/3803083282 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b40fc8b0 con 0x7f88b4070c60 2024-02-19T20:56:39.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.031+0000 7f88bc7e0700 10 client.14164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:39.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.031+0000 7f88bc7e0700 1 -- 172.21.15.163:0/3803083282 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b40fb370 con 0x7f88b4070c60 2024-02-19T20:56:39.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.031+0000 7f88bc7e0700 1 librados: init done 2024-02-19T20:56:39.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.031+0000 7f88ab7fe700 1 -- 172.21.15.163:0/3803083282 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f88a4004970 con 0x7f88b4070c60 2024-02-19T20:56:39.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.031+0000 7f88ab7fe700 1 -- 172.21.15.163:0/3803083282 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f88a40060e0 con 0x7f88b4070c60 2024-02-19T20:56:39.014 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.032+0000 7f88ab7fe700 1 -- 172.21.15.163:0/3803083282 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 4600635 0 0) 0x7f88a400f960 con 0x7f88b4070c60 2024-02-19T20:56:39.015 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.032+0000 7f88bc7e0700 1 -- 172.21.15.163:0/3803083282 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8898004f50 con 0x7f88b4070c60 2024-02-19T20:56:39.015 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.032+0000 7f88ab7fe700 1 -- 172.21.15.163:0/3803083282 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 1228221725 0 0) 0x7f88a403ecf0 con 0x7f88b4070c60 2024-02-19T20:56:39.015 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.032+0000 7f88ab7fe700 10 client.14164.objecter ms_dispatch 0x7f88b4001550 osd_map(3..3 src has 1..3) v4 2024-02-19T20:56:39.015 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.032+0000 7f88ab7fe700 3 client.14164.objecter handle_osd_map got epochs [3,3] > 0 2024-02-19T20:56:39.015 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.032+0000 7f88ab7fe700 3 client.14164.objecter handle_osd_map decoding full epoch 3 2024-02-19T20:56:39.015 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.032+0000 7f88ab7fe700 20 client.14164.objecter dump_active .. 0 homeless 2024-02-19T20:56:39.015 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.032+0000 7f88ab7fe700 10 client.14164.objecter ms_handle_connect 0x7f88a0035440 2024-02-19T20:56:39.015 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.036+0000 7f88ab7fe700 1 -- 172.21.15.163:0/3803083282 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f88a4011020 con 0x7f88b4070c60 2024-02-19T20:56:39.271 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.292+0000 7f88bc7e0700 1 -- 172.21.15.163:0/3803083282 --> v1:172.21.15.163:6789/0 -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f8898005780 con 0x7f88b4070c60 2024-02-19T20:56:39.273 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.294+0000 7f88ab7fe700 1 -- 172.21.15.163:0/3803083282 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v24)=0 set mgr/dashboard/cluster/status v24) v1 ==== 153+0+0 (unknown 2554848587 0 0) 0x7f88a4013100 con 0x7f88b4070c60 2024-02-19T20:56:39.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.296+0000 7f88bc7e0700 10 librados: watch_flush enter 2024-02-19T20:56:39.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.296+0000 7f88bc7e0700 10 librados: watch_flush exit 2024-02-19T20:56:39.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.296+0000 7f88bc7e0700 20 client.14164.objecter shutdown clearing up homeless session... 2024-02-19T20:56:39.276 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.296+0000 7f88bc7e0700 10 client.14164.objecter successfully canceled tick 2024-02-19T20:56:39.277 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.296+0000 7f88bc7e0700 1 -- 172.21.15.163:0/3803083282 >> v1:172.21.15.163:6800/2735092549 conn(0x7f88a0035440 legacy=0x7f88a0037900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:39.277 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.296+0000 7f88bc7e0700 1 -- 172.21.15.163:0/3803083282 >> v1:172.21.15.163:6789/0 conn(0x7f88b4070c60 legacy=0x7f88b40fa960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:39.277 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.296+0000 7f88bc7e0700 1 -- 172.21.15.163:0/3803083282 shutdown_connections 2024-02-19T20:56:39.277 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.296+0000 7f88bc7e0700 1 -- 172.21.15.163:0/3803083282 >> 172.21.15.163:0/3803083282 conn(0x7f88b40f6500 msgr2=0x7f88b40fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:39.277 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.296+0000 7f88bc7e0700 1 -- 172.21.15.163:0/3803083282 shutdown_connections 2024-02-19T20:56:39.277 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.296+0000 7f88bc7e0700 1 -- 172.21.15.163:0/3803083282 wait complete. 2024-02-19T20:56:39.277 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: 2024-02-19T20:56:39.296+0000 7f88bc7e0700 1 librados: shutdown 2024-02-19T20:56:39.277 INFO:teuthology.orchestra.run.smithi163.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-19T20:56:39.932 INFO:teuthology.orchestra.run.smithi163.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-19T20:56:39.932 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-02-19T20:56:39.932 INFO:teuthology.orchestra.run.smithi163.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-19T20:56:39.932 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-02-19T20:56:39.932 INFO:teuthology.orchestra.run.smithi163.stderr:Or, if you are only running a single cluster on this host: 2024-02-19T20:56:39.932 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-02-19T20:56:39.932 INFO:teuthology.orchestra.run.smithi163.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-19T20:56:39.932 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-02-19T20:56:39.932 INFO:teuthology.orchestra.run.smithi163.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-19T20:56:39.932 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-02-19T20:56:39.932 INFO:teuthology.orchestra.run.smithi163.stderr: ceph telemetry on 2024-02-19T20:56:39.932 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-02-19T20:56:39.932 INFO:teuthology.orchestra.run.smithi163.stderr:For more information see: 2024-02-19T20:56:39.932 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-02-19T20:56:39.933 INFO:teuthology.orchestra.run.smithi163.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2024-02-19T20:56:39.933 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-02-19T20:56:39.933 INFO:teuthology.orchestra.run.smithi163.stderr:Bootstrap complete. 2024-02-19T20:56:39.969 INFO:tasks.cephadm:Fetching config... 2024-02-19T20:56:39.970 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T20:56:39.970 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-19T20:56:40.029 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-19T20:56:40.029 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T20:56:40.029 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-19T20:56:40.085 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-19T20:56:40.086 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T20:56:40.086 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-19T20:56:40.152 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-19T20:56:40.152 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T20:56:40.152 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-19T20:56:40.210 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-19T20:56:40.211 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj5Y6sK17FSD2N7lErQNP+TrvdJgPU9svRxd1S9WyX7YBkoiRT0MG1DPRsYsds1jwTe0tQXSVxTRv3X0UNqYrT3f7ByC+i3lGjUSZknKBVU9RCnWGMhMZznTYZ9U34liEBFUEqOMjM79po5xADfbw7q4vPDn3a+gatAN7IZNaWcgzYSaGFcWZ3XanS5b7IsrFl2lDipjglM7ONqoiBOllgDzsobDq93h7W3CGh2XVaHGairvU4k+h7kSVENJHA4tWJjpQ24I/oFHtct5PLzhy/1Qg3jegnoaiHqZIGjkhHAlofSSuQZ6rCbhmzKEH8xS8PalutMuyYBys117yYwg3sPEMPLZyEjItNIP2slQmA0iQzZRPN27rUyf6y7aHDU8eW25PGyMS2KnR7E28KuzY/gAIkUnrreeB4zCcDPlJCnoJYcPQ0e1Ofo49AheJC1z2tuvRP+idzLplVo+wl5V8oCc5xE4KrOJI9u58DQV65fa70j4wdT1mEptVI+AqoAlc= ceph-2591809a-cf69-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T20:56:40.291 INFO:teuthology.orchestra.run.smithi163.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj5Y6sK17FSD2N7lErQNP+TrvdJgPU9svRxd1S9WyX7YBkoiRT0MG1DPRsYsds1jwTe0tQXSVxTRv3X0UNqYrT3f7ByC+i3lGjUSZknKBVU9RCnWGMhMZznTYZ9U34liEBFUEqOMjM79po5xADfbw7q4vPDn3a+gatAN7IZNaWcgzYSaGFcWZ3XanS5b7IsrFl2lDipjglM7ONqoiBOllgDzsobDq93h7W3CGh2XVaHGairvU4k+h7kSVENJHA4tWJjpQ24I/oFHtct5PLzhy/1Qg3jegnoaiHqZIGjkhHAlofSSuQZ6rCbhmzKEH8xS8PalutMuyYBys117yYwg3sPEMPLZyEjItNIP2slQmA0iQzZRPN27rUyf6y7aHDU8eW25PGyMS2KnR7E28KuzY/gAIkUnrreeB4zCcDPlJCnoJYcPQ0e1Ofo49AheJC1z2tuvRP+idzLplVo+wl5V8oCc5xE4KrOJI9u58DQV65fa70j4wdT1mEptVI+AqoAlc= ceph-2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:56:40.302 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj5Y6sK17FSD2N7lErQNP+TrvdJgPU9svRxd1S9WyX7YBkoiRT0MG1DPRsYsds1jwTe0tQXSVxTRv3X0UNqYrT3f7ByC+i3lGjUSZknKBVU9RCnWGMhMZznTYZ9U34liEBFUEqOMjM79po5xADfbw7q4vPDn3a+gatAN7IZNaWcgzYSaGFcWZ3XanS5b7IsrFl2lDipjglM7ONqoiBOllgDzsobDq93h7W3CGh2XVaHGairvU4k+h7kSVENJHA4tWJjpQ24I/oFHtct5PLzhy/1Qg3jegnoaiHqZIGjkhHAlofSSuQZ6rCbhmzKEH8xS8PalutMuyYBys117yYwg3sPEMPLZyEjItNIP2slQmA0iQzZRPN27rUyf6y7aHDU8eW25PGyMS2KnR7E28KuzY/gAIkUnrreeB4zCcDPlJCnoJYcPQ0e1Ofo49AheJC1z2tuvRP+idzLplVo+wl5V8oCc5xE4KrOJI9u58DQV65fa70j4wdT1mEptVI+AqoAlc= ceph-2591809a-cf69-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T20:56:40.345 INFO:teuthology.orchestra.run.smithi193.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj5Y6sK17FSD2N7lErQNP+TrvdJgPU9svRxd1S9WyX7YBkoiRT0MG1DPRsYsds1jwTe0tQXSVxTRv3X0UNqYrT3f7ByC+i3lGjUSZknKBVU9RCnWGMhMZznTYZ9U34liEBFUEqOMjM79po5xADfbw7q4vPDn3a+gatAN7IZNaWcgzYSaGFcWZ3XanS5b7IsrFl2lDipjglM7ONqoiBOllgDzsobDq93h7W3CGh2XVaHGairvU4k+h7kSVENJHA4tWJjpQ24I/oFHtct5PLzhy/1Qg3jegnoaiHqZIGjkhHAlofSSuQZ6rCbhmzKEH8xS8PalutMuyYBys117yYwg3sPEMPLZyEjItNIP2slQmA0iQzZRPN27rUyf6y7aHDU8eW25PGyMS2KnR7E28KuzY/gAIkUnrreeB4zCcDPlJCnoJYcPQ0e1Ofo49AheJC1z2tuvRP+idzLplVo+wl5V8oCc5xE4KrOJI9u58DQV65fa70j4wdT1mEptVI+AqoAlc= ceph-2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:56:40.357 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-19T20:56:40.728 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:39.294861+0000 mon.a (mon.0) 91 : audit [INF] from='client.? v1:172.21.15.163:0/3803083282' entity='client.admin' 2024-02-19T20:56:42.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.518+0000 7fa521b60700 1 -- 172.21.15.163:0/1462506449 >> v1:172.21.15.163:6789/0 conn(0x7fa51c061190 legacy=0x7fa51c0fa0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.518+0000 7fa521b60700 1 -- 172.21.15.163:0/1462506449 shutdown_connections 2024-02-19T20:56:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.518+0000 7fa521b60700 1 -- 172.21.15.163:0/1462506449 >> 172.21.15.163:0/1462506449 conn(0x7fa51c0f1b30 msgr2=0x7fa51c0f3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.518+0000 7fa521b60700 1 -- 172.21.15.163:0/1462506449 shutdown_connections 2024-02-19T20:56:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.518+0000 7fa521b60700 1 -- 172.21.15.163:0/1462506449 wait complete. 2024-02-19T20:56:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.519+0000 7fa521b60700 1 librados: starting msgr at 2024-02-19T20:56:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.519+0000 7fa521b60700 1 librados: starting objecter 2024-02-19T20:56:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.519+0000 7fa521b60700 1 Processor -- start 2024-02-19T20:56:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.519+0000 7fa521b60700 1 -- start start 2024-02-19T20:56:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.519+0000 7fa521b60700 1 librados: setting wanted keys 2024-02-19T20:56:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.519+0000 7fa521b60700 1 librados: calling monclient init 2024-02-19T20:56:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.519+0000 7fa521b60700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa51c0f7ac0 con 0x7fa51c061190 2024-02-19T20:56:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.519+0000 7fa51b7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fa51c061190 0x7fa51c0f5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57880/0 (socket says 172.21.15.163:57880) 2024-02-19T20:56:42.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.519+0000 7fa51b7fe700 1 -- 172.21.15.163:0/161763508 learned_addr learned my addr 172.21.15.163:0/161763508 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.519+0000 7fa518ff9700 10 client.?.objecter ms_handle_connect 0x7fa51c061190 2024-02-19T20:56:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.519+0000 7fa518ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.519+0000 7fa518ff9700 1 -- 172.21.15.163:0/161763508 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976031748 0 0) 0x7fa51c0f7ac0 con 0x7fa51c061190 2024-02-19T20:56:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.519+0000 7fa518ff9700 1 -- 172.21.15.163:0/161763508 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa504005580 con 0x7fa51c061190 2024-02-19T20:56:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.520+0000 7fa518ff9700 1 -- 172.21.15.163:0/161763508 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3284177292 0 0) 0x7fa504005580 con 0x7fa51c061190 2024-02-19T20:56:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.520+0000 7fa518ff9700 1 -- 172.21.15.163:0/161763508 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa51c0f7ac0 con 0x7fa51c061190 2024-02-19T20:56:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.520+0000 7fa518ff9700 1 -- 172.21.15.163:0/161763508 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fa50c000ed0 con 0x7fa51c061190 2024-02-19T20:56:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.520+0000 7fa518ff9700 1 -- 172.21.15.163:0/161763508 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3504832449 0 0) 0x7fa51c0f7ac0 con 0x7fa51c061190 2024-02-19T20:56:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.520+0000 7fa518ff9700 1 -- 172.21.15.163:0/161763508 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c0f7c50 con 0x7fa51c061190 2024-02-19T20:56:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.520+0000 7fa521b60700 1 -- 172.21.15.163:0/161763508 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51c0f7f00 con 0x7fa51c061190 2024-02-19T20:56:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.520+0000 7fa521b60700 10 client.14166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.520+0000 7fa521b60700 1 -- 172.21.15.163:0/161763508 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51c0f69c0 con 0x7fa51c061190 2024-02-19T20:56:42.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.520+0000 7fa521b60700 1 librados: init done 2024-02-19T20:56:42.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.520+0000 7fa518ff9700 1 -- 172.21.15.163:0/161763508 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fa50c0011f0 con 0x7fa51c061190 2024-02-19T20:56:42.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.520+0000 7fa518ff9700 1 -- 172.21.15.163:0/161763508 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fa50c003860 con 0x7fa51c061190 2024-02-19T20:56:42.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.521+0000 7fa518ff9700 1 -- 172.21.15.163:0/161763508 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 4600635 0 0) 0x7fa50c003aa0 con 0x7fa51c061190 2024-02-19T20:56:42.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.521+0000 7fa518ff9700 1 -- 172.21.15.163:0/161763508 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 1228221725 0 0) 0x7fa50c03f9c0 con 0x7fa51c061190 2024-02-19T20:56:42.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.521+0000 7fa518ff9700 10 client.14166.objecter ms_dispatch 0x7fa51c001550 osd_map(3..3 src has 1..3) v4 2024-02-19T20:56:42.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.521+0000 7fa518ff9700 3 client.14166.objecter handle_osd_map got epochs [3,3] > 0 2024-02-19T20:56:42.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.521+0000 7fa518ff9700 3 client.14166.objecter handle_osd_map decoding full epoch 3 2024-02-19T20:56:42.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.521+0000 7fa518ff9700 20 client.14166.objecter dump_active .. 0 homeless 2024-02-19T20:56:42.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.521+0000 7fa521b60700 1 -- 172.21.15.163:0/161763508 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa508004f50 con 0x7fa51c061190 2024-02-19T20:56:42.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.521+0000 7fa518ff9700 10 client.14166.objecter ms_handle_connect 0x7fa504035420 2024-02-19T20:56:42.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.525+0000 7fa518ff9700 1 -- 172.21.15.163:0/161763508 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa50c014ac0 con 0x7fa51c061190 2024-02-19T20:56:42.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.693+0000 7fa521b60700 1 -- 172.21.15.163:0/161763508 --> v1:172.21.15.163:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fa508005780 con 0x7fa51c061190 2024-02-19T20:56:42.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.696+0000 7fa518ff9700 1 -- 172.21.15.163:0/161763508 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (unknown 3028693289 0 0) 0x7fa50c012f30 con 0x7fa51c061190 2024-02-19T20:56:42.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.698+0000 7fa521b60700 10 librados: watch_flush enter 2024-02-19T20:56:42.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.698+0000 7fa521b60700 10 librados: watch_flush exit 2024-02-19T20:56:42.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.698+0000 7fa521b60700 20 client.14166.objecter shutdown clearing up homeless session... 2024-02-19T20:56:42.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.698+0000 7fa521b60700 10 client.14166.objecter successfully canceled tick 2024-02-19T20:56:42.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.698+0000 7fa521b60700 1 -- 172.21.15.163:0/161763508 >> v1:172.21.15.163:6800/2735092549 conn(0x7fa504035420 legacy=0x7fa5040378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:42.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.698+0000 7fa521b60700 1 -- 172.21.15.163:0/161763508 >> v1:172.21.15.163:6789/0 conn(0x7fa51c061190 legacy=0x7fa51c0f5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:42.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.698+0000 7fa521b60700 1 -- 172.21.15.163:0/161763508 shutdown_connections 2024-02-19T20:56:42.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.698+0000 7fa521b60700 1 -- 172.21.15.163:0/161763508 >> 172.21.15.163:0/161763508 conn(0x7fa51c0f1b30 msgr2=0x7fa51c0f8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:42.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.698+0000 7fa521b60700 1 -- 172.21.15.163:0/161763508 shutdown_connections 2024-02-19T20:56:42.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.698+0000 7fa521b60700 1 -- 172.21.15.163:0/161763508 wait complete. 2024-02-19T20:56:42.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:42.698+0000 7fa521b60700 1 librados: shutdown 2024-02-19T20:56:43.337 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-19T20:56:43.337 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-19T20:56:43.859 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:42.697422+0000 mon.a (mon.0) 92 : audit [INF] from='client.? v1:172.21.15.163:0/161763508' entity='client.admin' 2024-02-19T20:56:45.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.852+0000 7f5a0b4a6700 1 -- 172.21.15.163:0/76796875 >> v1:172.21.15.163:6789/0 conn(0x7f5a040f63d0 legacy=0x7f5a040f67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:45.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.853+0000 7f5a0b4a6700 1 -- 172.21.15.163:0/76796875 shutdown_connections 2024-02-19T20:56:45.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.853+0000 7f5a0b4a6700 1 -- 172.21.15.163:0/76796875 >> 172.21.15.163:0/76796875 conn(0x7f5a040f1b30 msgr2=0x7f5a040f3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:45.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.853+0000 7f5a0b4a6700 1 -- 172.21.15.163:0/76796875 shutdown_connections 2024-02-19T20:56:45.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.853+0000 7f5a0b4a6700 1 -- 172.21.15.163:0/76796875 wait complete. 2024-02-19T20:56:45.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.853+0000 7f5a0b4a6700 1 librados: starting msgr at 2024-02-19T20:56:45.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.853+0000 7f5a0b4a6700 1 librados: starting objecter 2024-02-19T20:56:45.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.853+0000 7f5a0b4a6700 1 Processor -- start 2024-02-19T20:56:45.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.853+0000 7f5a0b4a6700 1 -- start start 2024-02-19T20:56:45.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.853+0000 7f5a0b4a6700 1 librados: setting wanted keys 2024-02-19T20:56:45.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.853+0000 7f5a0b4a6700 1 librados: calling monclient init 2024-02-19T20:56:45.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.853+0000 7f5a0b4a6700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5a040610c0 con 0x7f5a040f63d0 2024-02-19T20:56:45.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f5a09242700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f5a040f63d0 0x7f5a04109870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:42582/0 (socket says 172.21.15.163:42582) 2024-02-19T20:56:45.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f5a09242700 1 -- 172.21.15.163:0/1454917584 learned_addr learned my addr 172.21.15.163:0/1454917584 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:56:45.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f59fa7fc700 10 client.?.objecter ms_handle_connect 0x7f5a040f63d0 2024-02-19T20:56:45.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f59fa7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T20:56:45.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f59fa7fc700 1 -- 172.21.15.163:0/1454917584 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289499344 0 0) 0x7f5a040610c0 con 0x7f5a040f63d0 2024-02-19T20:56:45.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f59fa7fc700 1 -- 172.21.15.163:0/1454917584 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59f0005580 con 0x7f5a040f63d0 2024-02-19T20:56:45.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f59fa7fc700 1 -- 172.21.15.163:0/1454917584 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3020468832 0 0) 0x7f59f0005580 con 0x7f5a040f63d0 2024-02-19T20:56:45.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f59fa7fc700 1 -- 172.21.15.163:0/1454917584 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a040610c0 con 0x7f5a040f63d0 2024-02-19T20:56:45.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f59fa7fc700 1 -- 172.21.15.163:0/1454917584 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f5a000020c0 con 0x7f5a040f63d0 2024-02-19T20:56:45.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f59fa7fc700 1 -- 172.21.15.163:0/1454917584 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 545630714 0 0) 0x7f5a040610c0 con 0x7f5a040f63d0 2024-02-19T20:56:45.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f59fa7fc700 1 -- 172.21.15.163:0/1454917584 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a04061250 con 0x7f5a040f63d0 2024-02-19T20:56:45.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f5a0b4a6700 1 -- 172.21.15.163:0/1454917584 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0410af90 con 0x7f5a040f63d0 2024-02-19T20:56:45.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f5a0b4a6700 10 client.14168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:56:45.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f5a0b4a6700 1 -- 172.21.15.163:0/1454917584 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0410b3c0 con 0x7f5a040f63d0 2024-02-19T20:56:45.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f5a0b4a6700 1 librados: init done 2024-02-19T20:56:45.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f59fa7fc700 1 -- 172.21.15.163:0/1454917584 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f5a000028a0 con 0x7f5a040f63d0 2024-02-19T20:56:45.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.854+0000 7f59fa7fc700 1 -- 172.21.15.163:0/1454917584 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f5a00003870 con 0x7f5a040f63d0 2024-02-19T20:56:45.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.855+0000 7f59fa7fc700 1 -- 172.21.15.163:0/1454917584 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 4600635 0 0) 0x7f5a000039f0 con 0x7f5a040f63d0 2024-02-19T20:56:45.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.855+0000 7f5a0b4a6700 1 -- 172.21.15.163:0/1454917584 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e8004f50 con 0x7f5a040f63d0 2024-02-19T20:56:45.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.855+0000 7f59fa7fc700 1 -- 172.21.15.163:0/1454917584 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 1228221725 0 0) 0x7f5a0003eca0 con 0x7f5a040f63d0 2024-02-19T20:56:45.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.855+0000 7f59fa7fc700 10 client.14168.objecter ms_dispatch 0x7f5a04001550 osd_map(3..3 src has 1..3) v4 2024-02-19T20:56:45.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.855+0000 7f59fa7fc700 3 client.14168.objecter handle_osd_map got epochs [3,3] > 0 2024-02-19T20:56:45.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.855+0000 7f59fa7fc700 3 client.14168.objecter handle_osd_map decoding full epoch 3 2024-02-19T20:56:45.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.855+0000 7f59fa7fc700 20 client.14168.objecter dump_active .. 0 homeless 2024-02-19T20:56:45.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.856+0000 7f59fa7fc700 10 client.14168.objecter ms_handle_connect 0x7f59f0035430 2024-02-19T20:56:45.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:45.860+0000 7f59fa7fc700 1 -- 172.21.15.163:0/1454917584 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5a00003d60 con 0x7f5a040f63d0 2024-02-19T20:56:46.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:46.039+0000 7f5a0b4a6700 1 -- 172.21.15.163:0/1454917584 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f59e8000bc0 con 0x7f59f0035430 2024-02-19T20:56:46.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:46.042+0000 7f59fa7fc700 1 -- 172.21.15.163:0/1454917584 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7f59e8000bc0 con 0x7f59f0035430 2024-02-19T20:56:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:46.044+0000 7f5a0b4a6700 10 librados: watch_flush enter 2024-02-19T20:56:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:46.044+0000 7f5a0b4a6700 10 librados: watch_flush exit 2024-02-19T20:56:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:46.044+0000 7f5a0b4a6700 20 client.14168.objecter shutdown clearing up homeless session... 2024-02-19T20:56:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:46.044+0000 7f5a0b4a6700 10 client.14168.objecter successfully canceled tick 2024-02-19T20:56:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:46.044+0000 7f5a0b4a6700 1 -- 172.21.15.163:0/1454917584 >> v1:172.21.15.163:6800/2735092549 conn(0x7f59f0035430 legacy=0x7f59f00378f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:46.044+0000 7f5a0b4a6700 1 -- 172.21.15.163:0/1454917584 >> v1:172.21.15.163:6789/0 conn(0x7f5a040f63d0 legacy=0x7f5a04109870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:56:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:46.044+0000 7f5a0b4a6700 1 -- 172.21.15.163:0/1454917584 shutdown_connections 2024-02-19T20:56:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:46.044+0000 7f5a0b4a6700 1 -- 172.21.15.163:0/1454917584 >> 172.21.15.163:0/1454917584 conn(0x7f5a040f1b30 msgr2=0x7f5a040f3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:56:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:46.044+0000 7f5a0b4a6700 1 -- 172.21.15.163:0/1454917584 shutdown_connections 2024-02-19T20:56:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:46.044+0000 7f5a0b4a6700 1 -- 172.21.15.163:0/1454917584 wait complete. 2024-02-19T20:56:46.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:56:46.044+0000 7f5a0b4a6700 1 librados: shutdown 2024-02-19T20:56:46.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:45.654303+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:56:46.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:45.964262+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi163", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:56:46.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:45.968295+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:56:46.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:45.969506+0000 mon.a (mon.0) 96 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:56:46.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:45.972989+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:56:46.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:46.043305+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:56:46.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:46.087568+0000 mon.a (mon.0) 99 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:56:46.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:46.088810+0000 mon.a (mon.0) 100 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:56:46.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:46.089321+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:56:46.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:56:46.090263+0000 mgr.y (mgr.14146) 6 : cephadm [INF] Updating smithi163:/etc/ceph/ceph.conf 2024-02-19T20:56:46.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:56:46.105991+0000 mgr.y (mgr.14146) 7 : cephadm [INF] Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-02-19T20:56:46.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:46.136916+0000 mon.a (mon.0) 102 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:56:46.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:56:46.139096+0000 mon.a (mon.0) 103 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:56:46.683 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi193 2024-02-19T20:56:46.683 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-02-19T20:56:46.683 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.conf 2024-02-19T20:56:46.705 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-02-19T20:56:46.705 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T20:56:46.766 INFO:tasks.cephadm:Adding host smithi193 to orchestrator... 2024-02-19T20:56:46.766 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch host add smithi193 2024-02-19T20:56:50.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:49.610180+0000 mgr.y (mgr.14146) 8 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:56:52.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:51.610595+0000 mgr.y (mgr.14146) 9 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:56:54.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:53.610964+0000 mgr.y (mgr.14146) 10 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:56:56.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:55.611204+0000 mgr.y (mgr.14146) 11 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:56:58.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:56:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:57.611428+0000 mgr.y (mgr.14146) 12 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:00.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:56:59.611680+0000 mgr.y (mgr.14146) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:02.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:01.611916+0000 mgr.y (mgr.14146) 14 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:04.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:03.612242+0000 mgr.y (mgr.14146) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:06.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:05.612587+0000 mgr.y (mgr.14146) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:08.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:07.612940+0000 mgr.y (mgr.14146) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:10.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:09.613212+0000 mgr.y (mgr.14146) 18 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:12.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:11.613542+0000 mgr.y (mgr.14146) 19 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:14.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:13.613924+0000 mgr.y (mgr.14146) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:16.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:15.614286+0000 mgr.y (mgr.14146) 21 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:18.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.874+0000 7f5669837700 1 -- 172.21.15.193:0/1276200147 >> v1:172.21.15.163:6789/0 conn(0x7f56640f63d0 legacy=0x7f56640f67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:18.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.874+0000 7f5669837700 1 -- 172.21.15.193:0/1276200147 shutdown_connections 2024-02-19T20:57:18.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.874+0000 7f5669837700 1 -- 172.21.15.193:0/1276200147 >> 172.21.15.193:0/1276200147 conn(0x7f56640f1b30 msgr2=0x7f56640f3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:18.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.874+0000 7f5669837700 1 -- 172.21.15.193:0/1276200147 shutdown_connections 2024-02-19T20:57:18.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.874+0000 7f5669837700 1 -- 172.21.15.193:0/1276200147 wait complete. 2024-02-19T20:57:18.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.875+0000 7f5669837700 1 librados: starting msgr at 2024-02-19T20:57:18.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.875+0000 7f5669837700 1 librados: starting objecter 2024-02-19T20:57:18.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.875+0000 7f5669837700 1 Processor -- start 2024-02-19T20:57:18.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.875+0000 7f5669837700 1 -- start start 2024-02-19T20:57:18.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.875+0000 7f5669837700 1 librados: setting wanted keys 2024-02-19T20:57:18.858 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.875+0000 7f5669837700 1 librados: calling monclient init 2024-02-19T20:57:18.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.875+0000 7f5669837700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f56640610c0 con 0x7f56640f63d0 2024-02-19T20:57:18.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f5662ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f56640f63d0 0x7f56641098a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.193:33110/0 (socket says 172.21.15.193:33110) 2024-02-19T20:57:18.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f5662ffd700 1 -- 172.21.15.193:0/1558325461 learned_addr learned my addr 172.21.15.193:0/1558325461 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T20:57:18.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f564bfff700 10 client.?.objecter ms_handle_connect 0x7f56640f63d0 2024-02-19T20:57:18.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f564bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:18.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f564bfff700 1 -- 172.21.15.193:0/1558325461 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602194705 0 0) 0x7f56640610c0 con 0x7f56640f63d0 2024-02-19T20:57:18.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f564bfff700 1 -- 172.21.15.193:0/1558325461 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f564c0055d0 con 0x7f56640f63d0 2024-02-19T20:57:18.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f564bfff700 1 -- 172.21.15.193:0/1558325461 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3625138845 0 0) 0x7f564c0055d0 con 0x7f56640f63d0 2024-02-19T20:57:18.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f564bfff700 1 -- 172.21.15.193:0/1558325461 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56640610c0 con 0x7f56640f63d0 2024-02-19T20:57:18.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f564bfff700 1 -- 172.21.15.193:0/1558325461 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f5654002020 con 0x7f56640f63d0 2024-02-19T20:57:18.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f564bfff700 1 -- 172.21.15.193:0/1558325461 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2754404907 0 0) 0x7f56640610c0 con 0x7f56640f63d0 2024-02-19T20:57:18.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f564bfff700 1 -- 172.21.15.193:0/1558325461 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5664061250 con 0x7f56640f63d0 2024-02-19T20:57:18.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f5669837700 1 -- 172.21.15.193:0/1558325461 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566410afc0 con 0x7f56640f63d0 2024-02-19T20:57:18.859 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f5669837700 10 client.14170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:57:18.860 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f5669837700 1 -- 172.21.15.193:0/1558325461 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f566410b3f0 con 0x7f56640f63d0 2024-02-19T20:57:18.860 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f5669837700 1 librados: init done 2024-02-19T20:57:18.860 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f564bfff700 1 -- 172.21.15.193:0/1558325461 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f56540026e0 con 0x7f56640f63d0 2024-02-19T20:57:18.860 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.876+0000 7f564bfff700 1 -- 172.21.15.193:0/1558325461 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f5654003790 con 0x7f56640f63d0 2024-02-19T20:57:18.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.877+0000 7f564bfff700 1 -- 172.21.15.193:0/1558325461 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 4600635 0 0) 0x7f56540038f0 con 0x7f56640f63d0 2024-02-19T20:57:18.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.877+0000 7f5669837700 1 -- 172.21.15.193:0/1558325461 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5650004f50 con 0x7f56640f63d0 2024-02-19T20:57:18.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.877+0000 7f564bfff700 1 -- 172.21.15.193:0/1558325461 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 1228221725 0 0) 0x7f565403edc0 con 0x7f56640f63d0 2024-02-19T20:57:18.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.877+0000 7f564bfff700 10 client.14170.objecter ms_dispatch 0x7f5664001550 osd_map(3..3 src has 1..3) v4 2024-02-19T20:57:18.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.877+0000 7f564bfff700 3 client.14170.objecter handle_osd_map got epochs [3,3] > 0 2024-02-19T20:57:18.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.877+0000 7f564bfff700 3 client.14170.objecter handle_osd_map decoding full epoch 3 2024-02-19T20:57:18.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.878+0000 7f564bfff700 20 client.14170.objecter dump_active .. 0 homeless 2024-02-19T20:57:18.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.878+0000 7f564bfff700 10 client.14170.objecter ms_handle_connect 0x7f564c035470 2024-02-19T20:57:18.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:18.882+0000 7f564bfff700 1 -- 172.21.15.193:0/1558325461 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5654003c40 con 0x7f56640f63d0 2024-02-19T20:57:19.029 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:19.048+0000 7f5669837700 1 -- 172.21.15.193:0/1558325461 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}) v1 -- 0x7f5650000bc0 con 0x7f564c035470 2024-02-19T20:57:19.228 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:57:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [90B blob data] 2024-02-19T20:57:19.228 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:17.614567+0000 mgr.y (mgr.14146) 22 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:19.646 INFO:teuthology.orchestra.run.smithi193.stdout:Added host 'smithi193' with addr '172.21.15.193' 2024-02-19T20:57:19.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:19.664+0000 7f564bfff700 1 -- 172.21.15.193:0/1558325461 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (unknown 0 0 3783357353) 0x7f5650000bc0 con 0x7f564c035470 2024-02-19T20:57:19.647 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:19.666+0000 7f5669837700 10 librados: watch_flush enter 2024-02-19T20:57:19.648 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:19.666+0000 7f5669837700 10 librados: watch_flush exit 2024-02-19T20:57:19.648 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:19.666+0000 7f5669837700 20 client.14170.objecter shutdown clearing up homeless session... 2024-02-19T20:57:19.648 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:19.666+0000 7f5669837700 10 client.14170.objecter successfully canceled tick 2024-02-19T20:57:19.648 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:19.666+0000 7f5669837700 1 -- 172.21.15.193:0/1558325461 >> v1:172.21.15.163:6800/2735092549 conn(0x7f564c035470 legacy=0x7f564c037930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:19.648 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:19.666+0000 7f5669837700 1 -- 172.21.15.193:0/1558325461 >> v1:172.21.15.163:6789/0 conn(0x7f56640f63d0 legacy=0x7f56641098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:19.648 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:19.666+0000 7f5669837700 1 -- 172.21.15.193:0/1558325461 shutdown_connections 2024-02-19T20:57:19.648 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:19.666+0000 7f5669837700 1 -- 172.21.15.193:0/1558325461 >> 172.21.15.193:0/1558325461 conn(0x7f56640f1b30 msgr2=0x7f56640f3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:19.648 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:19.666+0000 7f5669837700 1 -- 172.21.15.193:0/1558325461 shutdown_connections 2024-02-19T20:57:19.648 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:19.666+0000 7f5669837700 1 -- 172.21.15.193:0/1558325461 wait complete. 2024-02-19T20:57:19.648 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:19.666+0000 7f5669837700 1 librados: shutdown 2024-02-19T20:57:19.978 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 20:57:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [80B blob data] 2024-02-19T20:57:19.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:19.050571+0000 mgr.y (mgr.14146) 23 : audit [DBG] from='client.14170 v1:172.21.15.193:0/1558325461' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:57:19.979 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:19.664541+0000 mon.a (mon.0) 104 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:19.979 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:19.689021+0000 mon.a (mon.0) 105 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:57:20.277 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-19T20:57:21.228 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:19.614898+0000 mgr.y (mgr.14146) 24 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:21.228 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:19.664885+0000 mgr.y (mgr.14146) 25 : cephadm [INF] Added host smithi193 2024-02-19T20:57:21.228 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:20.369837+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:22.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.197+0000 7f0531241700 1 -- 172.21.15.193:0/2089053315 >> v1:172.21.15.163:6789/0 conn(0x7f052c061130 legacy=0x7f052c0fa110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:22.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.197+0000 7f0531241700 1 -- 172.21.15.193:0/2089053315 shutdown_connections 2024-02-19T20:57:22.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.197+0000 7f0531241700 1 -- 172.21.15.193:0/2089053315 >> 172.21.15.193:0/2089053315 conn(0x7f052c0f1b30 msgr2=0x7f052c0f3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:22.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.197+0000 7f0531241700 1 -- 172.21.15.193:0/2089053315 shutdown_connections 2024-02-19T20:57:22.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.197+0000 7f0531241700 1 -- 172.21.15.193:0/2089053315 wait complete. 2024-02-19T20:57:22.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.197+0000 7f0531241700 1 librados: starting msgr at 2024-02-19T20:57:22.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.197+0000 7f0531241700 1 librados: starting objecter 2024-02-19T20:57:22.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.198+0000 7f0531241700 1 Processor -- start 2024-02-19T20:57:22.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.198+0000 7f0531241700 1 -- start start 2024-02-19T20:57:22.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.198+0000 7f0531241700 1 librados: setting wanted keys 2024-02-19T20:57:22.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.198+0000 7f0531241700 1 librados: calling monclient init 2024-02-19T20:57:22.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.198+0000 7f0531241700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f052c0f7b20 con 0x7f052c061130 2024-02-19T20:57:22.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.198+0000 7f052ad9d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f052c061130 0x7f052c0f6010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.193:33118/0 (socket says 172.21.15.193:33118) 2024-02-19T20:57:22.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.198+0000 7f052ad9d700 1 -- 172.21.15.193:0/1400087609 learned_addr learned my addr 172.21.15.193:0/1400087609 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T20:57:22.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.199+0000 7f0513fff700 10 client.?.objecter ms_handle_connect 0x7f052c061130 2024-02-19T20:57:22.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.199+0000 7f0513fff700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:22.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.199+0000 7f0513fff700 1 -- 172.21.15.193:0/1400087609 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829574792 0 0) 0x7f052c0f7b20 con 0x7f052c061130 2024-02-19T20:57:22.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.199+0000 7f0513fff700 1 -- 172.21.15.193:0/1400087609 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0514005580 con 0x7f052c061130 2024-02-19T20:57:22.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.199+0000 7f0513fff700 1 -- 172.21.15.193:0/1400087609 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1790986931 0 0) 0x7f0514005580 con 0x7f052c061130 2024-02-19T20:57:22.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.199+0000 7f0513fff700 1 -- 172.21.15.193:0/1400087609 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f052c0f7b20 con 0x7f052c061130 2024-02-19T20:57:22.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.199+0000 7f0513fff700 1 -- 172.21.15.193:0/1400087609 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f0520000ed0 con 0x7f052c061130 2024-02-19T20:57:22.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.199+0000 7f0513fff700 1 -- 172.21.15.193:0/1400087609 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 843808166 0 0) 0x7f052c0f7b20 con 0x7f052c061130 2024-02-19T20:57:22.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.199+0000 7f0513fff700 1 -- 172.21.15.193:0/1400087609 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c0f7cb0 con 0x7f052c061130 2024-02-19T20:57:22.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.199+0000 7f0531241700 1 -- 172.21.15.193:0/1400087609 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052c0f7f60 con 0x7f052c061130 2024-02-19T20:57:22.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.199+0000 7f0531241700 10 client.14172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:57:22.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.199+0000 7f0531241700 1 -- 172.21.15.193:0/1400087609 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f052c0f6a20 con 0x7f052c061130 2024-02-19T20:57:22.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.199+0000 7f0531241700 1 librados: init done 2024-02-19T20:57:22.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.199+0000 7f0513fff700 1 -- 172.21.15.193:0/1400087609 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f05200028a0 con 0x7f052c061130 2024-02-19T20:57:22.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.199+0000 7f0513fff700 1 -- 172.21.15.193:0/1400087609 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7f0520003990 con 0x7f052c061130 2024-02-19T20:57:22.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.200+0000 7f0513fff700 1 -- 172.21.15.193:0/1400087609 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 4600635 0 0) 0x7f0520003b10 con 0x7f052c061130 2024-02-19T20:57:22.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.200+0000 7f0513fff700 1 -- 172.21.15.193:0/1400087609 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 1228221725 0 0) 0x7f052003fab0 con 0x7f052c061130 2024-02-19T20:57:22.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.200+0000 7f0531241700 1 -- 172.21.15.193:0/1400087609 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0518004f50 con 0x7f052c061130 2024-02-19T20:57:22.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.200+0000 7f0513fff700 10 client.14172.objecter ms_dispatch 0x7f052c001550 osd_map(3..3 src has 1..3) v4 2024-02-19T20:57:22.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.200+0000 7f0513fff700 3 client.14172.objecter handle_osd_map got epochs [3,3] > 0 2024-02-19T20:57:22.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.200+0000 7f0513fff700 3 client.14172.objecter handle_osd_map decoding full epoch 3 2024-02-19T20:57:22.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.200+0000 7f0513fff700 20 client.14172.objecter dump_active .. 0 homeless 2024-02-19T20:57:22.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.201+0000 7f0513fff700 10 client.14172.objecter ms_handle_connect 0x7f0514035350 2024-02-19T20:57:22.186 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.205+0000 7f0513fff700 1 -- 172.21.15.193:0/1400087609 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0520014bb0 con 0x7f052c061130 2024-02-19T20:57:22.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.391+0000 7f0531241700 1 -- 172.21.15.193:0/1400087609 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0518000bc0 con 0x7f0514035350 2024-02-19T20:57:22.374 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.392+0000 7f0513fff700 1 -- 172.21.15.193:0/1400087609 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (unknown 0 0 3686260431) 0x7f0518000bc0 con 0x7f0514035350 2024-02-19T20:57:22.374 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:57:22.374 INFO:teuthology.orchestra.run.smithi193.stdout:[{"addr": "172.21.15.163", "hostname": "smithi163", "labels": [], "status": ""}, {"addr": "172.21.15.193", "hostname": "smithi193", "labels": [], "status": ""}] 2024-02-19T20:57:22.376 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.394+0000 7f0531241700 10 librados: watch_flush enter 2024-02-19T20:57:22.376 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.394+0000 7f0531241700 10 librados: watch_flush exit 2024-02-19T20:57:22.376 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.394+0000 7f0531241700 20 client.14172.objecter shutdown clearing up homeless session... 2024-02-19T20:57:22.376 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.394+0000 7f0531241700 10 client.14172.objecter successfully canceled tick 2024-02-19T20:57:22.376 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.394+0000 7f0531241700 1 -- 172.21.15.193:0/1400087609 >> v1:172.21.15.163:6800/2735092549 conn(0x7f0514035350 legacy=0x7f0514037810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:22.376 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.394+0000 7f0531241700 1 -- 172.21.15.193:0/1400087609 >> v1:172.21.15.163:6789/0 conn(0x7f052c061130 legacy=0x7f052c0f6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:22.376 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.394+0000 7f0531241700 1 -- 172.21.15.193:0/1400087609 shutdown_connections 2024-02-19T20:57:22.376 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.394+0000 7f0531241700 1 -- 172.21.15.193:0/1400087609 >> 172.21.15.193:0/1400087609 conn(0x7f052c0f1b30 msgr2=0x7f052c0f8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:22.376 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.394+0000 7f0531241700 1 -- 172.21.15.193:0/1400087609 shutdown_connections 2024-02-19T20:57:22.376 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.394+0000 7f0531241700 1 -- 172.21.15.193:0/1400087609 wait complete. 2024-02-19T20:57:22.376 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:22.395+0000 7f0531241700 1 librados: shutdown 2024-02-19T20:57:22.728 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:21.342664+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:22.728 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:21.615317+0000 mgr.y (mgr.14146) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:22.964 INFO:tasks.cephadm:Setting crush tunables to jewel 2024-02-19T20:57:22.964 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd crush tunables jewel 2024-02-19T20:57:23.685 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:22.393198+0000 mgr.y (mgr.14146) 27 : audit [DBG] from='client.14172 v1:172.21.15.193:0/1400087609' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T20:57:24.446 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:23.615562+0000 mgr.y (mgr.14146) 28 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:24.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.945+0000 7fc874f35700 1 -- 172.21.15.163:0/2796978072 >> v1:172.21.15.163:6789/0 conn(0x7fc8700fd4f0 legacy=0x7fc8700fd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:24.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.945+0000 7fc874f35700 1 -- 172.21.15.163:0/2796978072 shutdown_connections 2024-02-19T20:57:24.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.945+0000 7fc874f35700 1 -- 172.21.15.163:0/2796978072 >> 172.21.15.163:0/2796978072 conn(0x7fc8700f1770 msgr2=0x7fc8700f3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:24.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.945+0000 7fc874f35700 1 -- 172.21.15.163:0/2796978072 shutdown_connections 2024-02-19T20:57:24.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.945+0000 7fc874f35700 1 -- 172.21.15.163:0/2796978072 wait complete. 2024-02-19T20:57:24.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.946+0000 7fc874f35700 1 librados: starting msgr at 2024-02-19T20:57:24.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.946+0000 7fc874f35700 1 librados: starting objecter 2024-02-19T20:57:24.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.946+0000 7fc874f35700 1 Processor -- start 2024-02-19T20:57:24.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.946+0000 7fc874f35700 1 -- start start 2024-02-19T20:57:24.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.946+0000 7fc874f35700 1 librados: setting wanted keys 2024-02-19T20:57:24.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.946+0000 7fc874f35700 1 librados: calling monclient init 2024-02-19T20:57:24.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.946+0000 7fc874f35700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc8700f8ab0 con 0x7fc8700fd4f0 2024-02-19T20:57:24.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc86e59c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fc8700fd4f0 0x7fc8700f6ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:39714/0 (socket says 172.21.15.163:39714) 2024-02-19T20:57:24.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc86e59c700 1 -- 172.21.15.163:0/2049438788 learned_addr learned my addr 172.21.15.163:0/2049438788 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:57:24.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc85f7fe700 10 client.?.objecter ms_handle_connect 0x7fc8700fd4f0 2024-02-19T20:57:24.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc85f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:24.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc85f7fe700 1 -- 172.21.15.163:0/2049438788 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660415376 0 0) 0x7fc8700f8ab0 con 0x7fc8700fd4f0 2024-02-19T20:57:24.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc85f7fe700 1 -- 172.21.15.163:0/2049438788 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc858005580 con 0x7fc8700fd4f0 2024-02-19T20:57:24.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc85f7fe700 1 -- 172.21.15.163:0/2049438788 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1730044109 0 0) 0x7fc858005580 con 0x7fc8700fd4f0 2024-02-19T20:57:24.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc85f7fe700 1 -- 172.21.15.163:0/2049438788 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8700f8ab0 con 0x7fc8700fd4f0 2024-02-19T20:57:24.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc85f7fe700 1 -- 172.21.15.163:0/2049438788 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc864002050 con 0x7fc8700fd4f0 2024-02-19T20:57:24.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc85f7fe700 1 -- 172.21.15.163:0/2049438788 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2401159020 0 0) 0x7fc8700f8ab0 con 0x7fc8700fd4f0 2024-02-19T20:57:24.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc85f7fe700 1 -- 172.21.15.163:0/2049438788 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8700f73b0 con 0x7fc8700fd4f0 2024-02-19T20:57:24.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc874f35700 1 -- 172.21.15.163:0/2049438788 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8700f7660 con 0x7fc8700fd4f0 2024-02-19T20:57:24.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc874f35700 10 client.14174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:57:24.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc874f35700 1 -- 172.21.15.163:0/2049438788 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc870111f70 con 0x7fc8700fd4f0 2024-02-19T20:57:24.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc874f35700 1 librados: init done 2024-02-19T20:57:24.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc85f7fe700 1 -- 172.21.15.163:0/2049438788 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fc8640027c0 con 0x7fc8700fd4f0 2024-02-19T20:57:24.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.947+0000 7fc85f7fe700 1 -- 172.21.15.163:0/2049438788 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc864003760 con 0x7fc8700fd4f0 2024-02-19T20:57:24.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.948+0000 7fc85f7fe700 1 -- 172.21.15.163:0/2049438788 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 4600635 0 0) 0x7fc8640038e0 con 0x7fc8700fd4f0 2024-02-19T20:57:24.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.948+0000 7fc85f7fe700 1 -- 172.21.15.163:0/2049438788 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1012+0+0 (unknown 1228221725 0 0) 0x7fc86403ec50 con 0x7fc8700fd4f0 2024-02-19T20:57:24.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.948+0000 7fc85f7fe700 10 client.14174.objecter ms_dispatch 0x7fc870001550 osd_map(3..3 src has 1..3) v4 2024-02-19T20:57:24.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.948+0000 7fc85f7fe700 3 client.14174.objecter handle_osd_map got epochs [3,3] > 0 2024-02-19T20:57:24.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.948+0000 7fc85f7fe700 3 client.14174.objecter handle_osd_map decoding full epoch 3 2024-02-19T20:57:24.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.948+0000 7fc85f7fe700 20 client.14174.objecter dump_active .. 0 homeless 2024-02-19T20:57:24.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.948+0000 7fc874f35700 1 -- 172.21.15.163:0/2049438788 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc850004f50 con 0x7fc8700fd4f0 2024-02-19T20:57:24.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.948+0000 7fc85f7fe700 10 client.14174.objecter ms_handle_connect 0x7fc8580353d0 2024-02-19T20:57:24.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:24.953+0000 7fc85f7fe700 1 -- 172.21.15.163:0/2049438788 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc864010f60 con 0x7fc8700fd4f0 2024-02-19T20:57:25.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:25.119+0000 7fc874f35700 1 -- 172.21.15.163:0/2049438788 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "jewel"} v 0) v1 -- 0x7fc850005200 con 0x7fc8700fd4f0 2024-02-19T20:57:25.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:25.352+0000 7fc85f7fe700 1 -- 172.21.15.163:0/2049438788 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "jewel"}]=0 adjusted tunables profile to jewel v4) v1 ==== 120+0+0 (unknown 2099115640 0 0) 0x7fc864014760 con 0x7fc8700fd4f0 2024-02-19T20:57:25.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:25.354+0000 7fc874f35700 10 librados: watch_flush enter 2024-02-19T20:57:25.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:25.354+0000 7fc874f35700 10 librados: watch_flush exit 2024-02-19T20:57:25.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:25.354+0000 7fc874f35700 20 client.14174.objecter shutdown clearing up homeless session... 2024-02-19T20:57:25.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:25.354+0000 7fc874f35700 10 client.14174.objecter successfully canceled tick 2024-02-19T20:57:25.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:25.354+0000 7fc874f35700 1 -- 172.21.15.163:0/2049438788 >> v1:172.21.15.163:6800/2735092549 conn(0x7fc8580353d0 legacy=0x7fc858037890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:25.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:25.354+0000 7fc874f35700 1 -- 172.21.15.163:0/2049438788 >> v1:172.21.15.163:6789/0 conn(0x7fc8700fd4f0 legacy=0x7fc8700f6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:25.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:25.354+0000 7fc874f35700 1 -- 172.21.15.163:0/2049438788 shutdown_connections 2024-02-19T20:57:25.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:25.354+0000 7fc874f35700 1 -- 172.21.15.163:0/2049438788 >> 172.21.15.163:0/2049438788 conn(0x7fc8700f1770 msgr2=0x7fc8700ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:25.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:25.354+0000 7fc874f35700 1 -- 172.21.15.163:0/2049438788 shutdown_connections 2024-02-19T20:57:25.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:25.354+0000 7fc874f35700 1 -- 172.21.15.163:0/2049438788 wait complete. 2024-02-19T20:57:25.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:25.355+0000 7fc874f35700 1 librados: shutdown 2024-02-19T20:57:25.336 INFO:teuthology.orchestra.run.smithi163.stderr:adjusted tunables profile to jewel 2024-02-19T20:57:25.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:25.120499+0000 mon.a (mon.0) 108 : audit [INF] from='client.? v1:172.21.15.163:0/2049438788' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-02-19T20:57:25.933 INFO:tasks.cephadm:Adding mon.a on smithi163 2024-02-19T20:57:25.933 INFO:tasks.cephadm:Adding mon.c on smithi163 2024-02-19T20:57:25.933 INFO:tasks.cephadm:Adding mon.b on smithi193 2024-02-19T20:57:25.933 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch apply mon '3;smithi163:[v1:172.21.15.163:6789]=a;smithi163:[v1:172.21.15.163:6790]=c;smithi193:[v1:172.21.15.193:6789]=b' 2024-02-19T20:57:26.728 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:25.352899+0000 mon.a (mon.0) 109 : audit [INF] from='client.? v1:172.21.15.163:0/2049438788' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-02-19T20:57:26.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:25.352940+0000 mon.a (mon.0) 110 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-19T20:57:26.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:25.566260+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:26.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:25.615878+0000 mgr.y (mgr.14146) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:26.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:25.879802+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:26.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:25.880474+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:57:26.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:25.881272+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:26.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:25.881915+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:57:26.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:25.882970+0000 mgr.y (mgr.14146) 30 : cephadm [INF] Updating smithi193:/etc/ceph/ceph.conf 2024-02-19T20:57:26.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:25.913869+0000 mgr.y (mgr.14146) 31 : cephadm [INF] Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-02-19T20:57:26.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:25.944246+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:26.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:25.949955+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:27.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.648+0000 7fc83d88f700 1 -- 172.21.15.193:0/563119238 >> v1:172.21.15.163:6789/0 conn(0x7fc8380fc420 legacy=0x7fc8380fc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:27.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.648+0000 7fc83d88f700 1 -- 172.21.15.193:0/563119238 shutdown_connections 2024-02-19T20:57:27.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.648+0000 7fc83d88f700 1 -- 172.21.15.193:0/563119238 >> 172.21.15.193:0/563119238 conn(0x7fc83806c300 msgr2=0x7fc83806c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:27.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.648+0000 7fc83d88f700 1 -- 172.21.15.193:0/563119238 shutdown_connections 2024-02-19T20:57:27.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.648+0000 7fc83d88f700 1 -- 172.21.15.193:0/563119238 wait complete. 2024-02-19T20:57:27.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.648+0000 7fc83d88f700 1 librados: starting msgr at 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.648+0000 7fc83d88f700 1 librados: starting objecter 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.649+0000 7fc83d88f700 1 Processor -- start 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.649+0000 7fc83d88f700 1 -- start start 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.649+0000 7fc83d88f700 1 librados: setting wanted keys 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.649+0000 7fc83d88f700 1 librados: calling monclient init 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.649+0000 7fc83d88f700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc8381082e0 con 0x7fc8380fc420 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.649+0000 7fc836ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fc8380fc420 0x7fc838107bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.193:53280/0 (socket says 172.21.15.193:53280) 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.649+0000 7fc836ffd700 1 -- 172.21.15.193:0/2139743327 learned_addr learned my addr 172.21.15.193:0/2139743327 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.649+0000 7fc81ffff700 10 client.?.objecter ms_handle_connect 0x7fc8380fc420 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.650+0000 7fc81ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.650+0000 7fc81ffff700 1 -- 172.21.15.193:0/2139743327 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371457014 0 0) 0x7fc8381082e0 con 0x7fc8380fc420 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.650+0000 7fc81ffff700 1 -- 172.21.15.193:0/2139743327 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc820005580 con 0x7fc8380fc420 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.650+0000 7fc81ffff700 1 -- 172.21.15.193:0/2139743327 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2936336141 0 0) 0x7fc820005580 con 0x7fc8380fc420 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.650+0000 7fc81ffff700 1 -- 172.21.15.193:0/2139743327 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8381082e0 con 0x7fc8380fc420 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.650+0000 7fc81ffff700 1 -- 172.21.15.193:0/2139743327 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc828000ed0 con 0x7fc8380fc420 2024-02-19T20:57:27.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.650+0000 7fc81ffff700 1 -- 172.21.15.193:0/2139743327 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2843183002 0 0) 0x7fc8381082e0 con 0x7fc8380fc420 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.650+0000 7fc81ffff700 1 -- 172.21.15.193:0/2139743327 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc838108470 con 0x7fc8380fc420 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.650+0000 7fc83d88f700 1 -- 172.21.15.193:0/2139743327 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc838108720 con 0x7fc8380fc420 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.650+0000 7fc83d88f700 10 client.14176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.650+0000 7fc83d88f700 1 -- 172.21.15.193:0/2139743327 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc83810fc40 con 0x7fc8380fc420 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.650+0000 7fc83d88f700 1 librados: init done 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.650+0000 7fc81ffff700 1 -- 172.21.15.193:0/2139743327 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fc8280027c0 con 0x7fc8380fc420 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.650+0000 7fc81ffff700 1 -- 172.21.15.193:0/2139743327 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fc8280038a0 con 0x7fc8380fc420 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.651+0000 7fc81ffff700 1 -- 172.21.15.193:0/2139743327 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 4600635 0 0) 0x7fc828002320 con 0x7fc8380fc420 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.651+0000 7fc83d88f700 1 -- 172.21.15.193:0/2139743327 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc824004f50 con 0x7fc8380fc420 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.651+0000 7fc81ffff700 1 -- 172.21.15.193:0/2139743327 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (unknown 1462125392 0 0) 0x7fc82803efb0 con 0x7fc8380fc420 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.651+0000 7fc81ffff700 10 client.14176.objecter ms_dispatch 0x7fc838001550 osd_map(4..4 src has 1..4) v4 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.651+0000 7fc81ffff700 3 client.14176.objecter handle_osd_map got epochs [4,4] > 0 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.651+0000 7fc81ffff700 3 client.14176.objecter handle_osd_map decoding full epoch 4 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.651+0000 7fc81ffff700 20 client.14176.objecter dump_active .. 0 homeless 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.651+0000 7fc81ffff700 10 client.14176.objecter ms_handle_connect 0x7fc820035300 2024-02-19T20:57:27.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.656+0000 7fc81ffff700 1 -- 172.21.15.193:0/2139743327 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc828014140 con 0x7fc8380fc420 2024-02-19T20:57:27.809 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.827+0000 7fc83d88f700 1 -- 172.21.15.193:0/2139743327 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi163:[v1:172.21.15.163:6789]=a;smithi163:[v1:172.21.15.163:6790]=c;smithi193:[v1:172.21.15.193:6789]=b", "target": ["mon-mgr", ""]}) v1 -- 0x7fc824000ca0 con 0x7fc820035300 2024-02-19T20:57:27.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.836+0000 7fc81ffff700 1 -- 172.21.15.193:0/2139743327 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 3265049985) 0x7fc824000ca0 con 0x7fc820035300 2024-02-19T20:57:27.817 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mon update... 2024-02-19T20:57:27.819 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.837+0000 7fc83d88f700 10 librados: watch_flush enter 2024-02-19T20:57:27.819 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.838+0000 7fc83d88f700 10 librados: watch_flush exit 2024-02-19T20:57:27.819 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.838+0000 7fc83d88f700 20 client.14176.objecter shutdown clearing up homeless session... 2024-02-19T20:57:27.819 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.838+0000 7fc83d88f700 10 client.14176.objecter successfully canceled tick 2024-02-19T20:57:27.819 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.838+0000 7fc83d88f700 1 -- 172.21.15.193:0/2139743327 >> v1:172.21.15.163:6800/2735092549 conn(0x7fc820035300 legacy=0x7fc8200377c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:27.820 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.838+0000 7fc83d88f700 1 -- 172.21.15.193:0/2139743327 >> v1:172.21.15.163:6789/0 conn(0x7fc8380fc420 legacy=0x7fc838107bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:27.820 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.838+0000 7fc83d88f700 1 -- 172.21.15.193:0/2139743327 shutdown_connections 2024-02-19T20:57:27.820 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.838+0000 7fc83d88f700 1 -- 172.21.15.193:0/2139743327 >> 172.21.15.193:0/2139743327 conn(0x7fc83806c300 msgr2=0x7fc8380f5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:27.820 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.838+0000 7fc83d88f700 1 -- 172.21.15.193:0/2139743327 shutdown_connections 2024-02-19T20:57:27.820 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.838+0000 7fc83d88f700 1 -- 172.21.15.193:0/2139743327 wait complete. 2024-02-19T20:57:27.820 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:27.838+0000 7fc83d88f700 1 librados: shutdown 2024-02-19T20:57:28.452 DEBUG:teuthology.orchestra.run.smithi163:mon.c> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@mon.c.service 2024-02-19T20:57:28.455 DEBUG:teuthology.orchestra.run.smithi193:mon.b> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@mon.b.service 2024-02-19T20:57:28.457 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-19T20:57:28.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T20:57:28.485 INFO:journalctl@ceph.mon.b.smithi193.stdout:-- Logs begin at Mon 2024-02-19 20:44:17 UTC. -- 2024-02-19T20:57:28.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:-- Logs begin at Mon 2024-02-19 20:44:35 UTC. -- 2024-02-19T20:57:29.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:27.616151+0000 mgr.y (mgr.14146) 32 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:29.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:27.829653+0000 mgr.y (mgr.14146) 33 : audit [DBG] from='client.14176 v1:172.21.15.193:0/2139743327' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi163:[v1:172.21.15.163:6789]=a;smithi163:[v1:172.21.15.163:6790]=c;smithi193:[v1:172.21.15.193:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:57:29.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:27.832054+0000 mgr.y (mgr.14146) 34 : cephadm [INF] Saving service mon spec with placement smithi163:[v1:172.21.15.163:6789]=a;smithi163:[v1:172.21.15.163:6790]=c;smithi193:[v1:172.21.15.193:6789]=b;count:3 2024-02-19T20:57:29.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:27.836937+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:29.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:27.869472+0000 mon.a (mon.0) 119 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:57:29.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:27.871204+0000 mon.a (mon.0) 120 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:29.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:27.871929+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:57:29.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:27.878026+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:29.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:27.879934+0000 mon.a (mon.0) 123 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T20:57:29.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:27.880602+0000 mon.a (mon.0) 124 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:29.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:27.881210+0000 mgr.y (mgr.14146) 35 : cephadm [INF] Deploying daemon mon.b on smithi193 2024-02-19T20:57:31.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:29.616485+0000 mgr.y (mgr.14146) 36 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:31.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.526+0000 7fb490542700 1 -- 172.21.15.193:0/3301716306 >> v1:172.21.15.163:6789/0 conn(0x7fb4880fc3e0 legacy=0x7fb4880fc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:31.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.526+0000 7fb490542700 1 -- 172.21.15.193:0/3301716306 shutdown_connections 2024-02-19T20:57:31.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.526+0000 7fb490542700 1 -- 172.21.15.193:0/3301716306 >> 172.21.15.193:0/3301716306 conn(0x7fb4880f1770 msgr2=0x7fb4880f3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:31.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.526+0000 7fb490542700 1 -- 172.21.15.193:0/3301716306 shutdown_connections 2024-02-19T20:57:31.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.526+0000 7fb490542700 1 -- 172.21.15.193:0/3301716306 wait complete. 2024-02-19T20:57:31.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.527+0000 7fb490542700 1 librados: starting msgr at 2024-02-19T20:57:31.510 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.527+0000 7fb490542700 1 librados: starting objecter 2024-02-19T20:57:31.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.527+0000 7fb490542700 1 Processor -- start 2024-02-19T20:57:31.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.527+0000 7fb490542700 1 -- start start 2024-02-19T20:57:31.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.527+0000 7fb490542700 1 librados: setting wanted keys 2024-02-19T20:57:31.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.527+0000 7fb490542700 1 librados: calling monclient init 2024-02-19T20:57:31.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.527+0000 7fb490542700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb48810c190 con 0x7fb4880fc3e0 2024-02-19T20:57:31.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb48e2de700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fb4880fc3e0 0x7fb48810ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.193:53310/0 (socket says 172.21.15.193:53310) 2024-02-19T20:57:31.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb48e2de700 1 -- 172.21.15.193:0/1304791822 learned_addr learned my addr 172.21.15.193:0/1304791822 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T20:57:31.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb47f7fe700 10 client.?.objecter ms_handle_connect 0x7fb4880fc3e0 2024-02-19T20:57:31.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb47f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:31.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb47f7fe700 1 -- 172.21.15.193:0/1304791822 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334390170 0 0) 0x7fb48810c190 con 0x7fb4880fc3e0 2024-02-19T20:57:31.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb47f7fe700 1 -- 172.21.15.193:0/1304791822 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb474005580 con 0x7fb4880fc3e0 2024-02-19T20:57:31.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb47f7fe700 1 -- 172.21.15.193:0/1304791822 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3129333386 0 0) 0x7fb474005580 con 0x7fb4880fc3e0 2024-02-19T20:57:31.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb47f7fe700 1 -- 172.21.15.193:0/1304791822 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb48810c190 con 0x7fb4880fc3e0 2024-02-19T20:57:31.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb47f7fe700 1 -- 172.21.15.193:0/1304791822 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fb484000ed0 con 0x7fb4880fc3e0 2024-02-19T20:57:31.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb47f7fe700 1 -- 172.21.15.193:0/1304791822 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 412546260 0 0) 0x7fb48810c190 con 0x7fb4880fc3e0 2024-02-19T20:57:31.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb47f7fe700 1 -- 172.21.15.193:0/1304791822 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48810c320 con 0x7fb4880fc3e0 2024-02-19T20:57:31.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb490542700 1 -- 172.21.15.193:0/1304791822 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48810c5d0 con 0x7fb4880fc3e0 2024-02-19T20:57:31.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb490542700 10 client.14178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:57:31.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb490542700 1 -- 172.21.15.193:0/1304791822 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48810ead0 con 0x7fb4880fc3e0 2024-02-19T20:57:31.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb490542700 1 librados: init done 2024-02-19T20:57:31.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb47f7fe700 1 -- 172.21.15.193:0/1304791822 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb484002390 con 0x7fb4880fc3e0 2024-02-19T20:57:31.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.528+0000 7fb47f7fe700 1 -- 172.21.15.193:0/1304791822 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 3107114179 0 0) 0x7fb484003850 con 0x7fb4880fc3e0 2024-02-19T20:57:31.513 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.529+0000 7fb47f7fe700 1 -- 172.21.15.193:0/1304791822 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 4600635 0 0) 0x7fb4840039d0 con 0x7fb4880fc3e0 2024-02-19T20:57:31.513 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.529+0000 7fb490542700 1 -- 172.21.15.193:0/1304791822 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46c004f50 con 0x7fb4880fc3e0 2024-02-19T20:57:31.513 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.529+0000 7fb47f7fe700 1 -- 172.21.15.193:0/1304791822 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (unknown 1462125392 0 0) 0x7fb48403eee0 con 0x7fb4880fc3e0 2024-02-19T20:57:31.513 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.529+0000 7fb47f7fe700 10 client.14178.objecter ms_dispatch 0x7fb488001550 osd_map(4..4 src has 1..4) v4 2024-02-19T20:57:31.513 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.529+0000 7fb47f7fe700 3 client.14178.objecter handle_osd_map got epochs [4,4] > 0 2024-02-19T20:57:31.513 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.529+0000 7fb47f7fe700 3 client.14178.objecter handle_osd_map decoding full epoch 4 2024-02-19T20:57:31.514 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.529+0000 7fb47f7fe700 20 client.14178.objecter dump_active .. 0 homeless 2024-02-19T20:57:31.514 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.530+0000 7fb47f7fe700 10 client.14178.objecter ms_handle_connect 0x7fb474035380 2024-02-19T20:57:31.515 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.534+0000 7fb47f7fe700 1 -- 172.21.15.193:0/1304791822 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb484003d70 con 0x7fb4880fc3e0 2024-02-19T20:57:31.749 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.767+0000 7fb490542700 1 -- 172.21.15.193:0/1304791822 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb46c005200 con 0x7fb4880fc3e0 2024-02-19T20:57:31.749 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.768+0000 7fb47f7fe700 1 -- 172.21.15.193:0/1304791822 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+675 (unknown 2237029548 0 3253270676) 0x7fb484013080 con 0x7fb4880fc3e0 2024-02-19T20:57:31.750 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:57:31.750 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"2591809a-cf69-11ee-95bb-87774f69a715","modified":"2024-02-19T20:55:35.071537Z","created":"2024-02-19T20:55:35.071537Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T20:57:31.751 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.770+0000 7fb490542700 10 librados: watch_flush enter 2024-02-19T20:57:31.751 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.770+0000 7fb490542700 10 librados: watch_flush exit 2024-02-19T20:57:31.752 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.770+0000 7fb490542700 20 client.14178.objecter shutdown clearing up homeless session... 2024-02-19T20:57:31.752 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.770+0000 7fb490542700 10 client.14178.objecter successfully canceled tick 2024-02-19T20:57:31.752 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.770+0000 7fb490542700 1 -- 172.21.15.193:0/1304791822 >> v1:172.21.15.163:6800/2735092549 conn(0x7fb474035380 legacy=0x7fb474037840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:31.752 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.770+0000 7fb490542700 1 -- 172.21.15.193:0/1304791822 >> v1:172.21.15.163:6789/0 conn(0x7fb4880fc3e0 legacy=0x7fb48810ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:31.752 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.770+0000 7fb490542700 1 -- 172.21.15.193:0/1304791822 shutdown_connections 2024-02-19T20:57:31.752 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.770+0000 7fb490542700 1 -- 172.21.15.193:0/1304791822 >> 172.21.15.193:0/1304791822 conn(0x7fb4880f1770 msgr2=0x7fb4880fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:31.752 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.770+0000 7fb490542700 1 -- 172.21.15.193:0/1304791822 shutdown_connections 2024-02-19T20:57:31.752 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.770+0000 7fb490542700 1 -- 172.21.15.193:0/1304791822 wait complete. 2024-02-19T20:57:31.752 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:31.770+0000 7fb490542700 1 librados: shutdown 2024-02-19T20:57:31.753 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-02-19T20:57:32.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:31.769391+0000 mon.a (mon.0) 125 : audit [DBG] from='client.? v1:172.21.15.193:0/1304791822' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T20:57:32.798 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:32 smithi193 systemd[1]: Starting Ceph mon.b for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T20:57:33.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:31.616829+0000 mgr.y (mgr.14146) 37 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:33.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.401+0000 7f8b0d61d880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T20:57:33.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.401+0000 7f8b0d61d880 0 ceph version 16.2.14-548-g81bd20d6 (81bd20d634209c7cb82c18be12b4b5a05643ebf1) pacific (stable), process ceph-mon, pid 7 2024-02-19T20:57:33.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.401+0000 7f8b0d61d880 0 pidfile_write: ignore empty --pid-file 2024-02-19T20:57:33.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.405+0000 7f8b0d61d880 0 load: jerasure load: lrc load: isa 2024-02-19T20:57:33.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T20:57:33.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T20:57:33.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Compile date Feb 17 2024 2024-02-19T20:57:33.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: DB SUMMARY 2024-02-19T20:57:33.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: CURRENT file: CURRENT 2024-02-19T20:57:33.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.805 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-19T20:57:33.805 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.805 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-02-19T20:57:33.805 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.805 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-19T20:57:33.805 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.805 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000003.log size: 476 ; 2024-02-19T20:57:33.805 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.805 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.error_if_exists: 0 2024-02-19T20:57:33.805 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.create_if_missing: 0 2024-02-19T20:57:33.805 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T20:57:33.805 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.env: 0x5604a1801080 2024-02-19T20:57:33.805 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.fs: Posix File System 2024-02-19T20:57:33.805 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.info_log: 0x5604a3057980 2024-02-19T20:57:33.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T20:57:33.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.statistics: (nil) 2024-02-19T20:57:33.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.use_fsync: 0 2024-02-19T20:57:33.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T20:57:33.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T20:57:33.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T20:57:33.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T20:57:33.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T20:57:33.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T20:57:33.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T20:57:33.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T20:57:33.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T20:57:33.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T20:57:33.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T20:57:33.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.db_log_dir: 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.write_buffer_manager: 0x5604a3d543c0 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T20:57:33.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T20:57:33.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T20:57:33.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T20:57:33.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T20:57:33.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T20:57:33.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T20:57:33.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.unordered_write: 0 2024-02-19T20:57:33.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T20:57:33.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T20:57:33.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T20:57:33.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T20:57:33.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.row_cache: None 2024-02-19T20:57:33.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.wal_filter: None 2024-02-19T20:57:33.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T20:57:33.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T20:57:33.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T20:57:33.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.two_write_queues: 0 2024-02-19T20:57:33.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T20:57:33.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.atomic_flush: 0 2024-02-19T20:57:33.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T20:57:33.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T20:57:33.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T20:57:33.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T20:57:33.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T20:57:33.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T20:57:33.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T20:57:33.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T20:57:33.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T20:57:33.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T20:57:33.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T20:57:33.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T20:57:33.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T20:57:33.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T20:57:33.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T20:57:33.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_open_files: -1 2024-02-19T20:57:33.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T20:57:33.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T20:57:33.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T20:57:33.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T20:57:33.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Compression algorithms supported: 2024-02-19T20:57:33.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T20:57:33.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: kZSTD supported: 0 2024-02-19T20:57:33.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: kXpressCompression supported: 0 2024-02-19T20:57:33.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T20:57:33.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T20:57:33.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: kZlibCompression supported: 1 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.merge_operator: 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compaction_filter: None 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5604a3007d10) 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cache_index_and_filter_blocks: 1 2024-02-19T20:57:33.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T20:57:33.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T20:57:33.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: pin_top_level_index_and_filter: 1 2024-02-19T20:57:33.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: index_type: 0 2024-02-19T20:57:33.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: data_block_index_type: 0 2024-02-19T20:57:33.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: index_shortening: 1 2024-02-19T20:57:33.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: data_block_hash_table_util_ratio: 0.750000 2024-02-19T20:57:33.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: hash_index_allow_collision: 1 2024-02-19T20:57:33.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: checksum: 1 2024-02-19T20:57:33.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: no_block_cache: 0 2024-02-19T20:57:33.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: block_cache: 0x5604a303cf10 2024-02-19T20:57:33.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: block_cache_name: BinnedLRUCache 2024-02-19T20:57:33.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: block_cache_options: 2024-02-19T20:57:33.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: capacity : 536870912 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: num_shard_bits : 4 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: strict_capacity_limit : 0 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: high_pri_pool_ratio: 0.000 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: block_cache_compressed: (nil) 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: persistent_cache: (nil) 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: block_size: 4096 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: block_size_deviation: 10 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: block_restart_interval: 16 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: index_block_restart_interval: 1 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: metadata_block_size: 4096 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: partition_filters: 0 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: use_delta_encoding: 1 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: whole_key_filtering: 1 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: verify_compression: 0 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: read_amp_bytes_per_bit: 0 2024-02-19T20:57:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: format_version: 2 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: enable_index_compression: 1 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: block_align: 0 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compression: NoCompression 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.num_levels: 7 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T20:57:33.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T20:57:33.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T20:57:33.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T20:57:33.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T20:57:33.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T20:57:33.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T20:57:33.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T20:57:33.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T20:57:33.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T20:57:33.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T20:57:33.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T20:57:33.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T20:57:33.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T20:57:33.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T20:57:33.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T20:57:33.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T20:57:33.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.table_properties_collectors: 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T20:57:33.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.bloom_locality: 0 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.ttl: 2592000 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.406+0000 7f8b0d61d880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.407+0000 7f8b0d61d880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708376253408148, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.407+0000 7f8b0d61d880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.407+0000 7f8b0d61d880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-19T20:57:33.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.407+0000 7f8b0d61d880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708376253408579, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1402, "table_properties": {"data_size": 488, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708376253, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-19T20:57:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.407+0000 7f8b0d61d880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-02-19T20:57:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.407+0000 7f8b0d61d880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708376253408940, "job": 1, "event": "recovery_finished"} 2024-02-19T20:57:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.408+0000 7f8b0d61d880 4 rocksdb: DB pointer 0x5604a3105800 2024-02-19T20:57:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.408+0000 7f8af440f700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T20:57:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.408+0000 7f8af440f700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T20:57:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: ** DB Stats ** 2024-02-19T20:57:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T20:57:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T20:57:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T20:57:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T20:57:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T20:57:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T20:57:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T20:57:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: ** Compaction Stats [default] ** 2024-02-19T20:57:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:57:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:57:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: L0 1/0 1.37 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 2024-02-19T20:57:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Sum 1/0 1.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 2024-02-19T20:57:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 2024-02-19T20:57:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: ** Compaction Stats [default] ** 2024-02-19T20:57:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:57:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:57:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.4 0.00 0.00 1 0.000 0 0 2024-02-19T20:57:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T20:57:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T20:57:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T20:57:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T20:57:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T20:57:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T20:57:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:57:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:57:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T20:57:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T20:57:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: ** Compaction Stats [default] ** 2024-02-19T20:57:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:57:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:57:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: L0 1/0 1.37 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 2024-02-19T20:57:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Sum 1/0 1.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 2024-02-19T20:57:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T20:57:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: ** Compaction Stats [default] ** 2024-02-19T20:57:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:57:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:57:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.4 0.00 0.00 1 0.000 0 0 2024-02-19T20:57:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T20:57:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T20:57:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T20:57:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T20:57:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T20:57:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T20:57:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:57:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:57:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T20:57:33.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T20:57:33.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.408+0000 7f8b0d61d880 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-19T20:57:33.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.408+0000 7f8b0d61d880 0 using public_addrv v1:172.21.15.193:6789/0 2024-02-19T20:57:33.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.409+0000 7f8b0d61d880 0 starting mon.b rank -1 at public addrs v1:172.21.15.193:6789/0 at bind addrs v1:172.21.15.193:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:57:33.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.410+0000 7f8b0d61d880 1 mon.b@-1(???) e0 preinit fsid 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:57:33.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.426+0000 7f8af6c14700 0 mon.b@-1(synchronizing).mds e1 new map 2024-02-19T20:57:33.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.426+0000 7f8af6c14700 0 mon.b@-1(synchronizing).mds e1 print_map 2024-02-19T20:57:33.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: e1 2024-02-19T20:57:33.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T20:57:33.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T20:57:33.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: legacy client fscid: -1 2024-02-19T20:57:33.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: No filesystems configured 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.426+0000 7f8af6c14700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.426+0000 7f8af240b700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #9. Immutable memtables: 0. 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.426+0000 7f8afc41f700 4 rocksdb: (Original Log Time 2024/02/19-20:57:33.427885) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.426+0000 7f8afc41f700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 9 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.426+0000 7f8afc41f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708376253427935, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 928, "num_deletes": 15, "total_data_size": 3611811, "memory_usage": 3629016, "flush_reason": "Manual Compaction"} 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.426+0000 7f8afc41f700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #10: started 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.426+0000 7f8af6c14700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.427+0000 7f8af6c14700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.427+0000 7f8af6c14700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.427+0000 7f8af6c14700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.427+0000 7f8af6c14700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.427+0000 7f8af6c14700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-19T20:57:33.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.427+0000 7f8af6c14700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T20:57:33.826 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.427+0000 7f8af6c14700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T20:57:33.826 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.427+0000 7f8af6c14700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T20:57:33.826 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:31.616829+0000 mgr.y (mgr.14146) 37 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:33.826 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T20:57:33.427+0000 7f8af6c14700 1 mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-19T20:57:33.826 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 bash[39300]: 9d66d5d71d837962b42acf17599c3d3eca95cdbc1e5721c6c133e84c0b433dd3 2024-02-19T20:57:33.826 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:33 smithi193 systemd[1]: Started Ceph mon.b for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T20:57:33.847 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-19T20:57:33.848 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T20:57:38.465 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.483+0000 7f11f7cc9700 1 -- 172.21.15.193:0/2182520058 >> v1:172.21.15.163:6789/0 conn(0x7f11f00fc420 legacy=0x7f11f00fc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:38.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.483+0000 7f11f7cc9700 1 -- 172.21.15.193:0/2182520058 shutdown_connections 2024-02-19T20:57:38.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.483+0000 7f11f7cc9700 1 -- 172.21.15.193:0/2182520058 >> 172.21.15.193:0/2182520058 conn(0x7f11f006c300 msgr2=0x7f11f006c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:38.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.483+0000 7f11f7cc9700 1 -- 172.21.15.193:0/2182520058 shutdown_connections 2024-02-19T20:57:38.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.484+0000 7f11f7cc9700 1 -- 172.21.15.193:0/2182520058 wait complete. 2024-02-19T20:57:38.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.484+0000 7f11f7cc9700 1 librados: starting msgr at 2024-02-19T20:57:38.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.484+0000 7f11f7cc9700 1 librados: starting objecter 2024-02-19T20:57:38.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.484+0000 7f11f7cc9700 1 Processor -- start 2024-02-19T20:57:38.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.484+0000 7f11f7cc9700 1 -- start start 2024-02-19T20:57:38.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.484+0000 7f11f7cc9700 1 librados: setting wanted keys 2024-02-19T20:57:38.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.484+0000 7f11f7cc9700 1 librados: calling monclient init 2024-02-19T20:57:38.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.485+0000 7f11f7cc9700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f11f01130c0 con 0x7f11f010d290 2024-02-19T20:57:38.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.485+0000 7f11f7cc9700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f11f0114260 con 0x7f11f00fc420 2024-02-19T20:57:38.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.485+0000 7f11f5a65700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f11f00fc420 0x7f11f010cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:51250/0 (socket says 172.21.15.193:51250) 2024-02-19T20:57:38.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.485+0000 7f11f5a65700 1 -- 172.21.15.193:0/763208478 learned_addr learned my addr 172.21.15.193:0/763208478 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T20:57:38.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.485+0000 7f11e6ffd700 10 client.?.objecter ms_handle_connect 0x7f11f00fc420 2024-02-19T20:57:38.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.485+0000 7f11e6ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:38.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.485+0000 7f11e6ffd700 10 client.?.objecter ms_handle_connect 0x7f11f010d290 2024-02-19T20:57:38.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.485+0000 7f11e6ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:38.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.485+0000 7f11e6ffd700 1 -- 172.21.15.193:0/763208478 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934071152 0 0) 0x7f11f01130c0 con 0x7f11f010d290 2024-02-19T20:57:38.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.485+0000 7f11e6ffd700 1 -- 172.21.15.193:0/763208478 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11dc003030 con 0x7f11f010d290 2024-02-19T20:57:38.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.486+0000 7f11e6ffd700 1 -- 172.21.15.193:0/763208478 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1345250527 0 0) 0x7f11dc003030 con 0x7f11f010d290 2024-02-19T20:57:38.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.486+0000 7f11e6ffd700 1 -- 172.21.15.193:0/763208478 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11f01130c0 con 0x7f11f010d290 2024-02-19T20:57:38.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.486+0000 7f11e6ffd700 1 -- 172.21.15.193:0/763208478 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 239+0+0 (unknown 164393484 0 0) 0x7f11ec001800 con 0x7f11f010d290 2024-02-19T20:57:38.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.487+0000 7f11e6ffd700 1 -- 172.21.15.193:0/763208478 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2322462051 0 0) 0x7f11f01130c0 con 0x7f11f010d290 2024-02-19T20:57:38.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.487+0000 7f11e6ffd700 1 -- 172.21.15.193:0/763208478 >> v1:172.21.15.193:6789/0 conn(0x7f11f00fc420 legacy=0x7f11f010cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:38.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.487+0000 7f11e6ffd700 1 -- 172.21.15.193:0/763208478 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f0115400 con 0x7f11f010d290 2024-02-19T20:57:38.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.487+0000 7f11f7cc9700 1 -- 172.21.15.193:0/763208478 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f01132b0 con 0x7f11f010d290 2024-02-19T20:57:38.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.487+0000 7f11f7cc9700 10 client.14182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:57:38.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.487+0000 7f11f7cc9700 1 -- 172.21.15.193:0/763208478 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f01137a0 con 0x7f11f010d290 2024-02-19T20:57:38.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.487+0000 7f11f7cc9700 1 librados: init done 2024-02-19T20:57:38.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.487+0000 7f11e6ffd700 1 -- 172.21.15.193:0/763208478 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f11ec0023a0 con 0x7f11f010d290 2024-02-19T20:57:38.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.487+0000 7f11e6ffd700 1 -- 172.21.15.193:0/763208478 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 239+0+0 (unknown 164393484 0 0) 0x7f11ec003270 con 0x7f11f010d290 2024-02-19T20:57:38.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.488+0000 7f11e6ffd700 1 -- 172.21.15.193:0/763208478 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 4600635 0 0) 0x7f11ec00caf0 con 0x7f11f010d290 2024-02-19T20:57:38.470 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.488+0000 7f11e6ffd700 1 -- 172.21.15.193:0/763208478 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (unknown 1462125392 0 0) 0x7f11ec03cf20 con 0x7f11f010d290 2024-02-19T20:57:38.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.488+0000 7f11e6ffd700 10 client.14182.objecter ms_dispatch 0x7f11f0001550 osd_map(4..4 src has 1..4) v4 2024-02-19T20:57:38.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.488+0000 7f11e6ffd700 3 client.14182.objecter handle_osd_map got epochs [4,4] > 0 2024-02-19T20:57:38.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.488+0000 7f11e6ffd700 3 client.14182.objecter handle_osd_map decoding full epoch 4 2024-02-19T20:57:38.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.488+0000 7f11e6ffd700 20 client.14182.objecter dump_active .. 0 homeless 2024-02-19T20:57:38.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.488+0000 7f11f7cc9700 1 -- 172.21.15.193:0/763208478 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d4004f50 con 0x7f11f010d290 2024-02-19T20:57:38.471 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.488+0000 7f11e6ffd700 10 client.14182.objecter ms_handle_connect 0x7f11dc035530 2024-02-19T20:57:38.475 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.493+0000 7f11e6ffd700 1 -- 172.21.15.193:0/763208478 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f11ec011fc0 con 0x7f11f010d290 2024-02-19T20:57:38.707 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.725+0000 7f11f7cc9700 1 -- 172.21.15.193:0/763208478 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11d4005200 con 0x7f11f010d290 2024-02-19T20:57:38.707 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.725+0000 7f11e6ffd700 1 -- 172.21.15.193:0/763208478 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+893 (unknown 3557084514 0 314461009) 0x7f11ec010430 con 0x7f11f010d290 2024-02-19T20:57:38.707 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:57:38.708 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":2,"fsid":"2591809a-cf69-11ee-95bb-87774f69a715","modified":"2024-02-19T20:57:33.431931Z","created":"2024-02-19T20:55:35.071537Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-19T20:57:38.709 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.727+0000 7f11f7cc9700 10 librados: watch_flush enter 2024-02-19T20:57:38.709 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.727+0000 7f11f7cc9700 10 librados: watch_flush exit 2024-02-19T20:57:38.709 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.727+0000 7f11f7cc9700 20 client.14182.objecter shutdown clearing up homeless session... 2024-02-19T20:57:38.709 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.727+0000 7f11f7cc9700 10 client.14182.objecter successfully canceled tick 2024-02-19T20:57:38.709 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.727+0000 7f11f7cc9700 1 -- 172.21.15.193:0/763208478 >> v1:172.21.15.163:6800/2735092549 conn(0x7f11dc035530 legacy=0x7f11dc0379f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:38.709 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.727+0000 7f11f7cc9700 1 -- 172.21.15.193:0/763208478 >> v1:172.21.15.163:6789/0 conn(0x7f11f010d290 legacy=0x7f11f01119a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:38.710 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.727+0000 7f11f7cc9700 1 -- 172.21.15.193:0/763208478 shutdown_connections 2024-02-19T20:57:38.710 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.727+0000 7f11f7cc9700 1 -- 172.21.15.193:0/763208478 >> 172.21.15.193:0/763208478 conn(0x7f11f006c300 msgr2=0x7f11f0060d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:38.710 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.727+0000 7f11f7cc9700 1 -- 172.21.15.193:0/763208478 shutdown_connections 2024-02-19T20:57:38.710 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.727+0000 7f11f7cc9700 1 -- 172.21.15.193:0/763208478 wait complete. 2024-02-19T20:57:38.710 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:38.727+0000 7f11f7cc9700 1 librados: shutdown 2024-02-19T20:57:38.710 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 2 2024-02-19T20:57:38.802 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:33.434452+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-19T20:57:38.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:33.434637+0000 mon.a (mon.0) 128 : cluster [INF] mon.a calling monitor election 2024-02-19T20:57:38.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:33.450185+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:38.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:33.617132+0000 mgr.y (mgr.14146) 38 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:38.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:34.431812+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:38.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:35.431620+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:38.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:35.432737+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-02-19T20:57:38.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:35.617536+0000 mgr.y (mgr.14146) 39 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:38.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:36.432007+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:38.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:37.431902+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:38.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:37.617787+0000 mgr.y (mgr.14146) 40 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:38.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:38.431920+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:38.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:38.464715+0000 mon.a (mon.0) 135 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-19T20:57:38.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:38.469855+0000 mon.a (mon.0) 136 : cluster [DBG] monmap e2: 2 mons at {a=v1:172.21.15.163:6789/0,b=v1:172.21.15.193:6789/0} removed_ranks: {} 2024-02-19T20:57:38.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:38.481707+0000 mon.a (mon.0) 137 : cluster [DBG] fsmap 2024-02-19T20:57:38.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:38.481737+0000 mon.a (mon.0) 138 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-19T20:57:38.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:38.481946+0000 mon.a (mon.0) 139 : cluster [DBG] mgrmap e13: y(active, since 68s) 2024-02-19T20:57:38.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:38.482078+0000 mon.a (mon.0) 140 : cluster [INF] overall HEALTH_OK 2024-02-19T20:57:38.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:38.486867+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:38.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:38.488043+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T20:57:38.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:38.488776+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:38.931 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:33.434452+0000 mon.a (mon.0) 127 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-19T20:57:38.931 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:33.434637+0000 mon.a (mon.0) 128 : cluster [INF] mon.a calling monitor election 2024-02-19T20:57:38.931 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:33.450185+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:38.931 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:33.617132+0000 mgr.y (mgr.14146) 38 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:38.931 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:34.431812+0000 mon.a (mon.0) 130 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:38.931 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:35.431620+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:38.931 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:35.432737+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-02-19T20:57:38.931 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:35.617536+0000 mgr.y (mgr.14146) 39 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:38.931 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:36.432007+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:38.931 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:37.431902+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:38.932 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:37.617787+0000 mgr.y (mgr.14146) 40 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:38.932 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:38.431920+0000 mon.a (mon.0) 134 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:38.932 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:38.464715+0000 mon.a (mon.0) 135 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-19T20:57:38.932 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:38.469855+0000 mon.a (mon.0) 136 : cluster [DBG] monmap e2: 2 mons at {a=v1:172.21.15.163:6789/0,b=v1:172.21.15.193:6789/0} removed_ranks: {} 2024-02-19T20:57:38.932 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:38.481707+0000 mon.a (mon.0) 137 : cluster [DBG] fsmap 2024-02-19T20:57:38.932 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:38.481737+0000 mon.a (mon.0) 138 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-19T20:57:38.932 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:38.481946+0000 mon.a (mon.0) 139 : cluster [DBG] mgrmap e13: y(active, since 68s) 2024-02-19T20:57:38.932 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:38.482078+0000 mon.a (mon.0) 140 : cluster [INF] overall HEALTH_OK 2024-02-19T20:57:38.932 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:38.486867+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:38.932 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:38.488043+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T20:57:38.932 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:38.488776+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:39.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:57:38.489406+0000 mgr.y (mgr.14146) 41 : cephadm [INF] Deploying daemon mon.c on smithi163 2024-02-19T20:57:39.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:38.726823+0000 mon.a (mon.0) 144 : audit [DBG] from='client.? v1:172.21.15.193:0/763208478' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T20:57:39.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:39.432052+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:39.807 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:38.489406+0000 mgr.y (mgr.14146) 41 : cephadm [INF] Deploying daemon mon.c on smithi163 2024-02-19T20:57:39.807 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:38.726823+0000 mon.a (mon.0) 144 : audit [DBG] from='client.? v1:172.21.15.193:0/763208478' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T20:57:39.807 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:39.432052+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:40.666 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:39.618194+0000 mgr.y (mgr.14146) 42 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:40.802 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:39.618194+0000 mgr.y (mgr.14146) 42 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:41.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 systemd[1]: Starting Ceph mon.c for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T20:57:41.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.642+0000 7f442f3fc880 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T20:57:41.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.642+0000 7f442f3fc880 0 ceph version 16.2.14-548-g81bd20d6 (81bd20d634209c7cb82c18be12b4b5a05643ebf1) pacific (stable), process ceph-mon, pid 7 2024-02-19T20:57:41.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.643+0000 7f442f3fc880 0 pidfile_write: ignore empty --pid-file 2024-02-19T20:57:41.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.647+0000 7f442f3fc880 0 load: jerasure load: lrc load: isa 2024-02-19T20:57:41.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T20:57:41.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T20:57:41.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Compile date Feb 17 2024 2024-02-19T20:57:41.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: DB SUMMARY 2024-02-19T20:57:41.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: CURRENT file: CURRENT 2024-02-19T20:57:41.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: IDENTITY file: IDENTITY 2024-02-19T20:57:41.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: MANIFEST file: MANIFEST-000001 size: 13 Bytes 2024-02-19T20:57:41.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-19T20:57:41.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000003.log size: 566 ; 2024-02-19T20:57:41.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.error_if_exists: 0 2024-02-19T20:57:41.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.create_if_missing: 0 2024-02-19T20:57:41.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T20:57:41.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.env: 0x55df8cdb7080 2024-02-19T20:57:41.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.fs: Posix File System 2024-02-19T20:57:41.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.info_log: 0x55df8eedd9e0 2024-02-19T20:57:41.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T20:57:41.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.statistics: (nil) 2024-02-19T20:57:41.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.use_fsync: 0 2024-02-19T20:57:41.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T20:57:41.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T20:57:41.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T20:57:41.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T20:57:41.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T20:57:41.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T20:57:41.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T20:57:41.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T20:57:41.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T20:57:41.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T20:57:41.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T20:57:41.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.db_log_dir: 2024-02-19T20:57:41.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-02-19T20:57:41.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T20:57:41.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T20:57:41.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T20:57:41.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T20:57:41.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T20:57:41.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T20:57:41.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T20:57:41.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T20:57:41.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T20:57:41.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T20:57:41.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.write_buffer_manager: 0x55df8fb58420 2024-02-19T20:57:41.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T20:57:41.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T20:57:41.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T20:57:41.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T20:57:41.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T20:57:41.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T20:57:41.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T20:57:41.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.unordered_write: 0 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.row_cache: None 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.wal_filter: None 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.two_write_queues: 0 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.atomic_flush: 0 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T20:57:41.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T20:57:41.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T20:57:41.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T20:57:41.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T20:57:41.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T20:57:41.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T20:57:41.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T20:57:41.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T20:57:41.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T20:57:41.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T20:57:41.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T20:57:41.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T20:57:41.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T20:57:41.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T20:57:41.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.max_open_files: -1 2024-02-19T20:57:41.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T20:57:41.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T20:57:41.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T20:57:41.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T20:57:41.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Compression algorithms supported: 2024-02-19T20:57:41.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T20:57:41.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: kZSTD supported: 0 2024-02-19T20:57:41.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: kXpressCompression supported: 0 2024-02-19T20:57:41.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T20:57:41.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T20:57:41.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T20:57:41.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: kZlibCompression supported: 1 2024-02-19T20:57:41.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T20:57:41.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.648+0000 7f442f3fc880 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.merge_operator: 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compaction_filter: None 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55df8ee8dd10) 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cache_index_and_filter_blocks: 1 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: pin_top_level_index_and_filter: 1 2024-02-19T20:57:41.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: index_type: 0 2024-02-19T20:57:41.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: data_block_index_type: 0 2024-02-19T20:57:41.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: index_shortening: 1 2024-02-19T20:57:41.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: data_block_hash_table_util_ratio: 0.750000 2024-02-19T20:57:41.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: hash_index_allow_collision: 1 2024-02-19T20:57:41.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: checksum: 1 2024-02-19T20:57:41.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: no_block_cache: 0 2024-02-19T20:57:41.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: block_cache: 0x55df8eec2f10 2024-02-19T20:57:41.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: block_cache_name: BinnedLRUCache 2024-02-19T20:57:41.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: block_cache_options: 2024-02-19T20:57:41.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: capacity : 536870912 2024-02-19T20:57:41.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: num_shard_bits : 4 2024-02-19T20:57:41.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: strict_capacity_limit : 0 2024-02-19T20:57:41.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: high_pri_pool_ratio: 0.000 2024-02-19T20:57:41.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: block_cache_compressed: (nil) 2024-02-19T20:57:41.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: persistent_cache: (nil) 2024-02-19T20:57:41.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: block_size: 4096 2024-02-19T20:57:41.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: block_size_deviation: 10 2024-02-19T20:57:41.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: block_restart_interval: 16 2024-02-19T20:57:41.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: index_block_restart_interval: 1 2024-02-19T20:57:41.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: metadata_block_size: 4096 2024-02-19T20:57:41.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: partition_filters: 0 2024-02-19T20:57:41.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: use_delta_encoding: 1 2024-02-19T20:57:41.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T20:57:41.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: whole_key_filtering: 1 2024-02-19T20:57:41.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: verify_compression: 0 2024-02-19T20:57:41.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: read_amp_bytes_per_bit: 0 2024-02-19T20:57:41.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: format_version: 2 2024-02-19T20:57:41.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: enable_index_compression: 1 2024-02-19T20:57:41.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: block_align: 0 2024-02-19T20:57:41.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T20:57:41.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T20:57:41.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compression: NoCompression 2024-02-19T20:57:41.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T20:57:41.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T20:57:41.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T20:57:41.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.num_levels: 7 2024-02-19T20:57:41.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T20:57:41.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T20:57:41.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T20:57:41.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T20:57:41.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T20:57:41.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T20:57:41.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T20:57:41.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T20:57:41.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T20:57:41.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T20:57:41.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T20:57:41.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T20:57:41.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T20:57:41.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T20:57:41.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T20:57:41.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T20:57:41.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T20:57:41.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T20:57:41.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T20:57:41.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T20:57:41.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T20:57:41.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T20:57:41.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T20:57:41.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T20:57:41.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T20:57:41.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T20:57:41.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T20:57:41.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T20:57:41.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T20:57:41.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T20:57:41.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T20:57:41.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T20:57:41.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T20:57:41.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T20:57:41.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T20:57:41.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T20:57:41.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T20:57:41.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T20:57:41.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T20:57:41.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T20:57:41.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T20:57:41.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T20:57:41.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T20:57:41.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T20:57:41.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T20:57:41.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T20:57:41.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T20:57:41.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.table_properties_collectors: 2024-02-19T20:57:41.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T20:57:41.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T20:57:41.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T20:57:41.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T20:57:41.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T20:57:41.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.bloom_locality: 0 2024-02-19T20:57:41.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T20:57:41.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T20:57:41.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T20:57:41.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T20:57:41.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T20:57:41.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.ttl: 2592000 2024-02-19T20:57:41.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T20:57:41.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T20:57:41.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-19T20:57:41.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708376261651193, "job": 1, "event": "recovery_started", "log_files": [3]} 2024-02-19T20:57:41.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.649+0000 7f442f3fc880 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #3 mode 2 2024-02-19T20:57:41.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.650+0000 7f442f3fc880 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-19T20:57:41.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.650+0000 7f442f3fc880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708376261651867, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 4, "file_size": 1492, "table_properties": {"data_size": 578, "index_size": 40, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708376261, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-19T20:57:41.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.650+0000 7f442f3fc880 4 rocksdb: [version_set.cc:3826] Creating manifest 5 2024-02-19T20:57:41.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.651+0000 7f442f3fc880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708376261652392, "job": 1, "event": "recovery_finished"} 2024-02-19T20:57:41.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.652+0000 7f442f3fc880 4 rocksdb: DB pointer 0x55df8ef8b800 2024-02-19T20:57:41.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.652+0000 7f44161ee700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T20:57:41.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.652+0000 7f44161ee700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T20:57:41.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: ** DB Stats ** 2024-02-19T20:57:41.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T20:57:41.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T20:57:41.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T20:57:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T20:57:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T20:57:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T20:57:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T20:57:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: ** Compaction Stats [default] ** 2024-02-19T20:57:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:57:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:57:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: L0 1/0 1.46 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.3 0.00 0.00 1 0.001 0 0 2024-02-19T20:57:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Sum 1/0 1.46 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.3 0.00 0.00 1 0.001 0 0 2024-02-19T20:57:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.3 0.00 0.00 1 0.001 0 0 2024-02-19T20:57:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: ** Compaction Stats [default] ** 2024-02-19T20:57:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:57:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.3 0.00 0.00 1 0.001 0 0 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Cumulative compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Interval compaction: 0.00 GB write, 0.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: ** Compaction Stats [default] ** 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:57:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:57:42.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: L0 1/0 1.46 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.3 0.00 0.00 1 0.001 0 0 2024-02-19T20:57:42.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Sum 1/0 1.46 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.3 0.00 0.00 1 0.001 0 0 2024-02-19T20:57:42.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T20:57:42.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:42.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: ** Compaction Stats [default] ** 2024-02-19T20:57:42.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T20:57:42.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T20:57:42.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.3 0.00 0.00 1 0.001 0 0 2024-02-19T20:57:42.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T20:57:42.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T20:57:42.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T20:57:42.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T20:57:42.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T20:57:42.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T20:57:42.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Cumulative compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:57:42.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T20:57:42.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T20:57:42.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:42.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T20:57:42.002 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:42.002 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.652+0000 7f442f3fc880 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-19T20:57:42.002 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.652+0000 7f442f3fc880 0 using public_addrv v1:172.21.15.163:6790/0 2024-02-19T20:57:42.002 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.653+0000 7f442f3fc880 0 starting mon.c rank -1 at public addrs v1:172.21.15.163:6790/0 at bind addrs v1:172.21.15.163:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:57:42.002 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.654+0000 7f442f3fc880 1 mon.c@-1(???) e0 preinit fsid 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:57:42.002 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.672+0000 7f44141ea700 4 rocksdb: [db_impl/db_impl_write.cc:1668] [default] New memtable created with log file: #9. Immutable memtables: 0. 2024-02-19T20:57:42.002 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:42.002 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f441e1fe700 4 rocksdb: (Original Log Time 2024/02/19-20:57:41.674244) [db_impl/db_impl_compaction_flush.cc:2198] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-02-19T20:57:42.002 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f441e1fe700 4 rocksdb: [flush_job.cc:321] [default] [JOB 3] Flushing memtable with next log file: 9 2024-02-19T20:57:42.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:42.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f441e1fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708376261674305, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 962, "num_deletes": 12, "total_data_size": 3761059, "memory_usage": 3779064, "flush_reason": "Manual Compaction"} 2024-02-19T20:57:42.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f441e1fe700 4 rocksdb: [flush_job.cc:350] [default] [JOB 3] Level-0 flush table #10: started 2024-02-19T20:57:42.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f44189f3700 0 mon.c@-1(synchronizing).mds e1 new map 2024-02-19T20:57:42.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f44189f3700 0 mon.c@-1(synchronizing).mds e1 print_map 2024-02-19T20:57:42.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: e1 2024-02-19T20:57:42.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T20:57:42.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T20:57:42.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: legacy client fscid: -1 2024-02-19T20:57:42.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:42.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: No filesystems configured 2024-02-19T20:57:42.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: 2024-02-19T20:57:42.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f44189f3700 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-19T20:57:42.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f44189f3700 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-19T20:57:42.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f44189f3700 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-19T20:57:42.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f44189f3700 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-19T20:57:42.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f44189f3700 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-19T20:57:42.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f44189f3700 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-19T20:57:42.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f44189f3700 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-19T20:57:42.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f44189f3700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T20:57:42.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f44189f3700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T20:57:42.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.673+0000 7f44189f3700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T20:57:42.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:39.618194+0000 mgr.y (mgr.14146) 42 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:42.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T20:57:41.674+0000 7f44189f3700 1 mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-02-19T20:57:42.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 bash[45131]: 9583f754b74474b2f5020900a1a67b1189bef2bfa8a6f05166f9f44ec8c78fc6 2024-02-19T20:57:42.004 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:41 smithi163 systemd[1]: Started Ceph mon.c for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T20:57:43.849 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-19T20:57:43.849 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T20:57:46.911 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:41.618688+0000 mgr.y (mgr.14146) 43 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:46.911 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:41.682721+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-19T20:57:46.911 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:41.682915+0000 mon.a (mon.0) 148 : cluster [INF] mon.a calling monitor election 2024-02-19T20:57:46.911 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:41.690586+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-02-19T20:57:46.911 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:41.696342+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:46.912 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:41.696492+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:46.912 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:42.679358+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:46.912 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:43.619094+0000 mgr.y (mgr.14146) 44 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:46.912 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:43.679363+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:46.912 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:43.681182+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-02-19T20:57:46.912 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:44.679540+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:46.912 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:45.619519+0000 mgr.y (mgr.14146) 45 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:46.912 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:45.679703+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:46.912 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:46.679905+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:46.912 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:46.710691+0000 mon.a (mon.0) 156 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-19T20:57:46.912 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:46.720822+0000 mon.a (mon.0) 157 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.163:6789/0,b=v1:172.21.15.193:6789/0,c=v1:172.21.15.163:6790/0} removed_ranks: {} 2024-02-19T20:57:46.912 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:46.732815+0000 mon.a (mon.0) 158 : cluster [DBG] fsmap 2024-02-19T20:57:46.913 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:46.732862+0000 mon.a (mon.0) 159 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-19T20:57:46.913 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:46.733064+0000 mon.a (mon.0) 160 : cluster [DBG] mgrmap e13: y(active, since 77s) 2024-02-19T20:57:46.913 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:46.733333+0000 mon.a (mon.0) 161 : cluster [INF] overall HEALTH_OK 2024-02-19T20:57:46.913 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:46.736913+0000 mon.a (mon.0) 162 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:46.913 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:46.738297+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:57:47.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:41.618688+0000 mgr.y (mgr.14146) 43 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:47.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:41.682721+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-19T20:57:47.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:41.682915+0000 mon.a (mon.0) 148 : cluster [INF] mon.a calling monitor election 2024-02-19T20:57:47.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:41.690586+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-02-19T20:57:47.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:41.696342+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:47.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:41.696492+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:47.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:42.679358+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:47.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:43.619094+0000 mgr.y (mgr.14146) 44 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:47.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:43.679363+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:47.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:43.681182+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-02-19T20:57:47.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:44.679540+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:47.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:45.619519+0000 mgr.y (mgr.14146) 45 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:47.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:45.679703+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:47.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:46.679905+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:47.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:46.710691+0000 mon.a (mon.0) 156 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-19T20:57:47.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:46.720822+0000 mon.a (mon.0) 157 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.163:6789/0,b=v1:172.21.15.193:6789/0,c=v1:172.21.15.163:6790/0} removed_ranks: {} 2024-02-19T20:57:47.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:46.732815+0000 mon.a (mon.0) 158 : cluster [DBG] fsmap 2024-02-19T20:57:47.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:41.618688+0000 mgr.y (mgr.14146) 43 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:47.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:41.682721+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-19T20:57:47.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:41.682915+0000 mon.a (mon.0) 148 : cluster [INF] mon.a calling monitor election 2024-02-19T20:57:47.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:41.690586+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-02-19T20:57:47.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:41.696342+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T20:57:47.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:41.696492+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:47.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:42.679358+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:47.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:43.619094+0000 mgr.y (mgr.14146) 44 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:47.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:43.679363+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:47.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:43.681182+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-02-19T20:57:47.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:44.679540+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:47.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:45.619519+0000 mgr.y (mgr.14146) 45 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:47.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:45.679703+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:47.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:46.679905+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:47.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:46.710691+0000 mon.a (mon.0) 156 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-19T20:57:47.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:46.720822+0000 mon.a (mon.0) 157 : cluster [DBG] monmap e3: 3 mons at {a=v1:172.21.15.163:6789/0,b=v1:172.21.15.193:6789/0,c=v1:172.21.15.163:6790/0} removed_ranks: {} 2024-02-19T20:57:47.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:46.732815+0000 mon.a (mon.0) 158 : cluster [DBG] fsmap 2024-02-19T20:57:47.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:46.732862+0000 mon.a (mon.0) 159 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-19T20:57:47.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:46.733064+0000 mon.a (mon.0) 160 : cluster [DBG] mgrmap e13: y(active, since 77s) 2024-02-19T20:57:47.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:46.733333+0000 mon.a (mon.0) 161 : cluster [INF] overall HEALTH_OK 2024-02-19T20:57:47.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:46.736913+0000 mon.a (mon.0) 162 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:47.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:46.738297+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:57:47.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:46.732862+0000 mon.a (mon.0) 159 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-19T20:57:47.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:46.733064+0000 mon.a (mon.0) 160 : cluster [DBG] mgrmap e13: y(active, since 77s) 2024-02-19T20:57:47.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:46.733333+0000 mon.a (mon.0) 161 : cluster [INF] overall HEALTH_OK 2024-02-19T20:57:47.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:46.736913+0000 mon.a (mon.0) 162 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:47.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:46.738297+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:57:47.951 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.950+0000 7f9be0769700 1 -- 172.21.15.193:0/771088805 >> v1:172.21.15.163:6789/0 conn(0x7f9bd80fc420 legacy=0x7f9bd80fc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:47.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.950+0000 7f9be0769700 1 -- 172.21.15.193:0/771088805 shutdown_connections 2024-02-19T20:57:47.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.950+0000 7f9be0769700 1 -- 172.21.15.193:0/771088805 >> 172.21.15.193:0/771088805 conn(0x7f9bd806c300 msgr2=0x7f9bd806c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:47.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.950+0000 7f9be0769700 1 -- 172.21.15.193:0/771088805 shutdown_connections 2024-02-19T20:57:47.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.950+0000 7f9be0769700 1 -- 172.21.15.193:0/771088805 wait complete. 2024-02-19T20:57:47.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9be0769700 1 librados: starting msgr at 2024-02-19T20:57:47.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9be0769700 1 librados: starting objecter 2024-02-19T20:57:47.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9be0769700 1 Processor -- start 2024-02-19T20:57:47.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9be0769700 1 -- start start 2024-02-19T20:57:47.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9be0769700 1 librados: setting wanted keys 2024-02-19T20:57:47.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9be0769700 1 librados: calling monclient init 2024-02-19T20:57:47.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9be0769700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9bd8116fb0 con 0x7f9bd81134a0 2024-02-19T20:57:47.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9be0769700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9bd8118150 con 0x7f9bd8108410 2024-02-19T20:57:47.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9be0769700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9bd81192f0 con 0x7f9bd80fc420 2024-02-19T20:57:47.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9bddd04700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f9bd8108410 0x7f9bd8111d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:51342/0 (socket says 172.21.15.193:51342) 2024-02-19T20:57:47.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9bddd04700 1 -- 172.21.15.193:0/927278080 learned_addr learned my addr 172.21.15.193:0/927278080 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T20:57:47.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9bcf7fe700 10 client.?.objecter ms_handle_connect 0x7f9bd8108410 2024-02-19T20:57:47.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9bcf7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:47.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9bcf7fe700 10 client.?.objecter ms_handle_connect 0x7f9bd81134a0 2024-02-19T20:57:47.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9bcf7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:47.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9bcf7fe700 10 client.?.objecter ms_handle_connect 0x7f9bd80fc420 2024-02-19T20:57:47.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9bcf7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:47.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858241637 0 0) 0x7f9bd8118150 con 0x7f9bd8108410 2024-02-19T20:57:47.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bc4003ae0 con 0x7f9bd8108410 2024-02-19T20:57:47.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771665221 0 0) 0x7f9bd8116fb0 con 0x7f9bd81134a0 2024-02-19T20:57:47.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.951+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd8118150 con 0x7f9bd81134a0 2024-02-19T20:57:47.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.952+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3395894465 0 0) 0x7f9bc4003ae0 con 0x7f9bd8108410 2024-02-19T20:57:47.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.952+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bd8116fb0 con 0x7f9bd8108410 2024-02-19T20:57:47.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.952+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9bd4001840 con 0x7f9bd8108410 2024-02-19T20:57:47.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.952+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1682756900 0 0) 0x7f9bd8116fb0 con 0x7f9bd8108410 2024-02-19T20:57:47.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.952+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 >> v1:172.21.15.163:6790/0 conn(0x7f9bd80fc420 legacy=0x7f9bd8107d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:47.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.952+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 >> v1:172.21.15.163:6789/0 conn(0x7f9bd81134a0 legacy=0x7f9bd8115890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:47.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.952+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd811a4d0 con 0x7f9bd8108410 2024-02-19T20:57:47.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.952+0000 7f9be0769700 1 -- 172.21.15.193:0/927278080 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd81171a0 con 0x7f9bd8108410 2024-02-19T20:57:47.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.952+0000 7f9be0769700 10 client.24100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:57:47.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.952+0000 7f9be0769700 1 -- 172.21.15.193:0/927278080 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd81176d0 con 0x7f9bd8108410 2024-02-19T20:57:47.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.952+0000 7f9be0769700 1 librados: init done 2024-02-19T20:57:47.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.952+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9bd4004c90 con 0x7f9bd8108410 2024-02-19T20:57:47.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.952+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9bd4005ba0 con 0x7f9bd8108410 2024-02-19T20:57:47.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.952+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 4600635 0 0) 0x7f9bd400f4e0 con 0x7f9bd8108410 2024-02-19T20:57:47.955 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.953+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (unknown 1462125392 0 0) 0x7f9bd403ea10 con 0x7f9bd8108410 2024-02-19T20:57:47.955 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.953+0000 7f9bcf7fe700 10 client.24100.objecter ms_dispatch 0x7f9bd8001550 osd_map(4..4 src has 1..4) v4 2024-02-19T20:57:47.955 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.953+0000 7f9bcf7fe700 3 client.24100.objecter handle_osd_map got epochs [4,4] > 0 2024-02-19T20:57:47.955 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.953+0000 7f9bcf7fe700 3 client.24100.objecter handle_osd_map decoding full epoch 4 2024-02-19T20:57:47.955 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.953+0000 7f9bcf7fe700 20 client.24100.objecter dump_active .. 0 homeless 2024-02-19T20:57:47.955 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.953+0000 7f9be0769700 1 -- 172.21.15.193:0/927278080 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bbc004f50 con 0x7f9bd8108410 2024-02-19T20:57:47.955 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.953+0000 7f9bcf7fe700 10 client.24100.objecter ms_handle_connect 0x7f9bc40352f0 2024-02-19T20:57:47.955 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:47.959+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9bd4010d60 con 0x7f9bd8108410 2024-02-19T20:57:48.053 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:47.679896+0000 mon.a (mon.0) 164 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:48.053 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:47.692776+0000 mon.a (mon.0) 165 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:48.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:48.197+0000 7f9be0769700 1 -- 172.21.15.193:0/927278080 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9bbc005200 con 0x7f9bd8108410 2024-02-19T20:57:48.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:48.197+0000 7f9bcf7fe700 1 -- 172.21.15.193:0/927278080 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1111 (unknown 409899127 0 545308006) 0x7f9bd4012e40 con 0x7f9bd8108410 2024-02-19T20:57:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-02-19T20:57:48.180 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":3,"fsid":"2591809a-cf69-11ee-95bb-87774f69a715","modified":"2024-02-19T20:57:41.680222Z","created":"2024-02-19T20:55:35.071537Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6790","nonce":0}]},"addr":"172.21.15.163:6790/0","public_addr":"172.21.15.163:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-02-19T20:57:48.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:48.199+0000 7f9be0769700 10 librados: watch_flush enter 2024-02-19T20:57:48.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:48.199+0000 7f9be0769700 10 librados: watch_flush exit 2024-02-19T20:57:48.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:48.199+0000 7f9be0769700 20 client.24100.objecter shutdown clearing up homeless session... 2024-02-19T20:57:48.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:48.199+0000 7f9be0769700 10 client.24100.objecter successfully canceled tick 2024-02-19T20:57:48.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:48.199+0000 7f9be0769700 1 -- 172.21.15.193:0/927278080 >> v1:172.21.15.163:6800/2735092549 conn(0x7f9bc40352f0 legacy=0x7f9bc40377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:48.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:48.199+0000 7f9be0769700 1 -- 172.21.15.193:0/927278080 >> v1:172.21.15.193:6789/0 conn(0x7f9bd8108410 legacy=0x7f9bd8111d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:48.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:48.199+0000 7f9be0769700 1 -- 172.21.15.193:0/927278080 shutdown_connections 2024-02-19T20:57:48.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:48.199+0000 7f9be0769700 1 -- 172.21.15.193:0/927278080 >> 172.21.15.193:0/927278080 conn(0x7f9bd806c300 msgr2=0x7f9bd80f53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:48.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:48.199+0000 7f9be0769700 1 -- 172.21.15.193:0/927278080 shutdown_connections 2024-02-19T20:57:48.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:48.199+0000 7f9be0769700 1 -- 172.21.15.193:0/927278080 wait complete. 2024-02-19T20:57:48.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:48.199+0000 7f9be0769700 1 librados: shutdown 2024-02-19T20:57:48.182 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 3 2024-02-19T20:57:48.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:47.679896+0000 mon.a (mon.0) 164 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:48.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:47.692776+0000 mon.a (mon.0) 165 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:48.229 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:47.679896+0000 mon.a (mon.0) 164 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T20:57:48.229 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:47.692776+0000 mon.a (mon.0) 165 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:49.053 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:47.619799+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:49.053 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:48.198548+0000 mon.b (mon.1) 3 : audit [DBG] from='client.? v1:172.21.15.193:0/927278080' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T20:57:49.053 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:48.692331+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:49.053 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:48.693042+0000 mon.a (mon.0) 167 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:49.053 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:48.693831+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:57:49.053 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:48.714991+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:49.053 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:48.723215+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:49.054 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:48.729072+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:49.054 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:48.730591+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T20:57:49.054 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:48.731239+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T20:57:49.054 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:48.731848+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:49.074 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-19T20:57:49.074 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph config generate-minimal-conf 2024-02-19T20:57:49.201 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:47.619799+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:49.201 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:48.198548+0000 mon.b (mon.1) 3 : audit [DBG] from='client.? v1:172.21.15.193:0/927278080' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T20:57:49.202 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:48.692331+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:49.202 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:48.693042+0000 mon.a (mon.0) 167 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:49.202 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:48.693831+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:57:49.202 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:48.714991+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:49.202 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:48.723215+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:49.202 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:48.729072+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:49.202 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:48.730591+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T20:57:49.202 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:48.731239+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T20:57:49.202 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:48.731848+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:49.202 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:47.619799+0000 mgr.y (mgr.14146) 46 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:49.202 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:48.198548+0000 mon.b (mon.1) 3 : audit [DBG] from='client.? v1:172.21.15.193:0/927278080' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T20:57:49.202 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:48.692331+0000 mon.a (mon.0) 166 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:49.202 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:48.693042+0000 mon.a (mon.0) 167 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:49.203 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:48.693831+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:57:49.203 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:48.714991+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:49.203 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:48.723215+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:49.203 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:48.729072+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:49.203 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:48.730591+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T20:57:49.203 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:48.731239+0000 mon.a (mon.0) 173 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T20:57:49.203 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:48.731848+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:50.093 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:57:48.694948+0000 mgr.y (mgr.14146) 47 : cephadm [INF] Updating smithi163:/etc/ceph/ceph.conf 2024-02-19T20:57:50.093 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:57:48.695218+0000 mgr.y (mgr.14146) 48 : cephadm [INF] Updating smithi193:/etc/ceph/ceph.conf 2024-02-19T20:57:50.093 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:57:48.730325+0000 mgr.y (mgr.14146) 49 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-19T20:57:50.093 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:57:48.732418+0000 mgr.y (mgr.14146) 50 : cephadm [INF] Reconfiguring daemon mon.a on smithi163 2024-02-19T20:57:50.093 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:49.748528+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:50.093 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:48.694948+0000 mgr.y (mgr.14146) 47 : cephadm [INF] Updating smithi163:/etc/ceph/ceph.conf 2024-02-19T20:57:50.093 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:48.695218+0000 mgr.y (mgr.14146) 48 : cephadm [INF] Updating smithi193:/etc/ceph/ceph.conf 2024-02-19T20:57:50.094 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:48.730325+0000 mgr.y (mgr.14146) 49 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-19T20:57:50.094 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:48.732418+0000 mgr.y (mgr.14146) 50 : cephadm [INF] Reconfiguring daemon mon.a on smithi163 2024-02-19T20:57:50.094 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:49.748528+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:50.303 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:57:48.694948+0000 mgr.y (mgr.14146) 47 : cephadm [INF] Updating smithi163:/etc/ceph/ceph.conf 2024-02-19T20:57:50.303 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:57:48.695218+0000 mgr.y (mgr.14146) 48 : cephadm [INF] Updating smithi193:/etc/ceph/ceph.conf 2024-02-19T20:57:50.303 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:57:48.730325+0000 mgr.y (mgr.14146) 49 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2024-02-19T20:57:50.303 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:57:48.732418+0000 mgr.y (mgr.14146) 50 : cephadm [INF] Reconfiguring daemon mon.a on smithi163 2024-02-19T20:57:50.303 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:49.748528+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:51.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:49.620215+0000 mgr.y (mgr.14146) 51 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:51.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:50.153009+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:51.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:50.153444+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-19T20:57:51.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:50.153662+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T20:57:51.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:50.154196+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T20:57:51.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:50.154767+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:51.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:50.155246+0000 mgr.y (mgr.14146) 53 : cephadm [INF] Reconfiguring daemon mon.c on smithi163 2024-02-19T20:57:51.473 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:49.620215+0000 mgr.y (mgr.14146) 51 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:51.473 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:50.153009+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:51.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:57:50.153444+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-19T20:57:51.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:50.153662+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T20:57:51.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:50.154196+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T20:57:51.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:50.154767+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:51.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:57:50.155246+0000 mgr.y (mgr.14146) 53 : cephadm [INF] Reconfiguring daemon mon.c on smithi163 2024-02-19T20:57:51.553 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:49.620215+0000 mgr.y (mgr.14146) 51 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:51.553 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:50.153009+0000 mon.a (mon.0) 176 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:51.553 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:57:50.153444+0000 mgr.y (mgr.14146) 52 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-02-19T20:57:51.553 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:50.153662+0000 mon.a (mon.0) 177 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T20:57:51.553 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:50.154196+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T20:57:51.553 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:50.154767+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:51.553 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:57:50.155246+0000 mgr.y (mgr.14146) 53 : cephadm [INF] Reconfiguring daemon mon.c on smithi163 2024-02-19T20:57:52.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.300+0000 7f963b2ff700 1 -- 172.21.15.163:0/3791577719 >> v1:172.21.15.163:6789/0 conn(0x7f96341030c0 legacy=0x7f9634105540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:52.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.301+0000 7f963b2ff700 1 -- 172.21.15.163:0/3791577719 shutdown_connections 2024-02-19T20:57:52.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.301+0000 7f963b2ff700 1 -- 172.21.15.163:0/3791577719 >> 172.21.15.163:0/3791577719 conn(0x7f963406c760 msgr2=0x7f963406cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:52.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.301+0000 7f963b2ff700 1 -- 172.21.15.163:0/3791577719 shutdown_connections 2024-02-19T20:57:52.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.301+0000 7f963b2ff700 1 -- 172.21.15.163:0/3791577719 wait complete. 2024-02-19T20:57:52.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.301+0000 7f963b2ff700 1 librados: starting msgr at 2024-02-19T20:57:52.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.301+0000 7f963b2ff700 1 librados: starting objecter 2024-02-19T20:57:52.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.301+0000 7f963b2ff700 1 Processor -- start 2024-02-19T20:57:52.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.301+0000 7f963b2ff700 1 -- start start 2024-02-19T20:57:52.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.301+0000 7f963b2ff700 1 librados: setting wanted keys 2024-02-19T20:57:52.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.301+0000 7f963b2ff700 1 librados: calling monclient init 2024-02-19T20:57:52.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f963b2ff700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f96340fbe00 con 0x7f96341030c0 2024-02-19T20:57:52.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f963b2ff700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f96340fbfd0 con 0x7f96340ff580 2024-02-19T20:57:52.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f963b2ff700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f96340fc1c0 con 0x7f96340fc780 2024-02-19T20:57:52.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f963909b700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f96340fc780 0x7f96340fb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:36660/0 (socket says 172.21.15.163:36660) 2024-02-19T20:57:52.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f963909b700 1 -- 172.21.15.163:0/3705887176 learned_addr learned my addr 172.21.15.163:0/3705887176 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:57:52.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 10 client.?.objecter ms_handle_connect 0x7f96340fc780 2024-02-19T20:57:52.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:52.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 10 client.?.objecter ms_handle_connect 0x7f96341030c0 2024-02-19T20:57:52.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:52.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765788717 0 0) 0x7f96340fc1c0 con 0x7f96340fc780 2024-02-19T20:57:52.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9620003030 con 0x7f96340fc780 2024-02-19T20:57:52.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 10 client.?.objecter ms_handle_connect 0x7f96340ff580 2024-02-19T20:57:52.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:52.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845079260 0 0) 0x7f96340fbe00 con 0x7f96341030c0 2024-02-19T20:57:52.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96340fc1c0 con 0x7f96341030c0 2024-02-19T20:57:52.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662829040 0 0) 0x7f96340fbfd0 con 0x7f96340ff580 2024-02-19T20:57:52.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96340fbe00 con 0x7f96340ff580 2024-02-19T20:57:52.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1963927590 0 0) 0x7f9620003030 con 0x7f96340fc780 2024-02-19T20:57:52.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96340fbfd0 con 0x7f96340fc780 2024-02-19T20:57:52.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9628001940 con 0x7f96340fc780 2024-02-19T20:57:52.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3622635829 0 0) 0x7f96340fc1c0 con 0x7f96341030c0 2024-02-19T20:57:52.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9620003030 con 0x7f96341030c0 2024-02-19T20:57:52.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f962c0017b0 con 0x7f96341030c0 2024-02-19T20:57:52.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2256949805 0 0) 0x7f96340fbe00 con 0x7f96340ff580 2024-02-19T20:57:52.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96340fc1c0 con 0x7f96340ff580 2024-02-19T20:57:52.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f96300019f0 con 0x7f96340ff580 2024-02-19T20:57:52.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 279994938 0 0) 0x7f96340fbfd0 con 0x7f96340fc780 2024-02-19T20:57:52.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 >> v1:172.21.15.193:6789/0 conn(0x7f96340ff580 legacy=0x7f9634114b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:52.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 >> v1:172.21.15.163:6789/0 conn(0x7f96341030c0 legacy=0x7f9634118250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:52.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963411ca00 con 0x7f96340fc780 2024-02-19T20:57:52.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.302+0000 7f963b2ff700 1 -- 172.21.15.163:0/3705887176 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96341199d0 con 0x7f96340fc780 2024-02-19T20:57:52.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.303+0000 7f963b2ff700 10 client.34103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:57:52.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.303+0000 7f963b2ff700 1 -- 172.21.15.163:0/3705887176 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9634119ec0 con 0x7f96340fc780 2024-02-19T20:57:52.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.303+0000 7f963b2ff700 1 librados: init done 2024-02-19T20:57:52.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.303+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9628002350 con 0x7f96340fc780 2024-02-19T20:57:52.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.303+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9628005a70 con 0x7f96340fc780 2024-02-19T20:57:52.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.303+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 4600635 0 0) 0x7f962800f2d0 con 0x7f96340fc780 2024-02-19T20:57:52.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.303+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (unknown 1462125392 0 0) 0x7f962803e760 con 0x7f96340fc780 2024-02-19T20:57:52.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.303+0000 7f96267fc700 10 client.34103.objecter ms_dispatch 0x7f9634001550 osd_map(4..4 src has 1..4) v4 2024-02-19T20:57:52.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.303+0000 7f96267fc700 3 client.34103.objecter handle_osd_map got epochs [4,4] > 0 2024-02-19T20:57:52.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.303+0000 7f96267fc700 3 client.34103.objecter handle_osd_map decoding full epoch 4 2024-02-19T20:57:52.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.303+0000 7f96267fc700 20 client.34103.objecter dump_active .. 0 homeless 2024-02-19T20:57:52.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.304+0000 7f963b2ff700 1 -- 172.21.15.163:0/3705887176 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9618004f20 con 0x7f96340fc780 2024-02-19T20:57:52.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.304+0000 7f96267fc700 10 client.34103.objecter ms_handle_connect 0x7f96200353e0 2024-02-19T20:57:52.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.308+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9628010ab0 con 0x7f96340fc780 2024-02-19T20:57:52.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.473+0000 7f963b2ff700 1 -- 172.21.15.163:0/3705887176 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f96180051d0 con 0x7f96340fc780 2024-02-19T20:57:52.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.473+0000 7f96267fc700 1 -- 172.21.15.163:0/3705887176 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+193 (unknown 2189217731 0 2014289794) 0x7f9628012b90 con 0x7f96340fc780 2024-02-19T20:57:52.456 INFO:teuthology.orchestra.run.smithi163.stdout:# minimal ceph.conf for 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:57:52.456 INFO:teuthology.orchestra.run.smithi163.stdout:[global] 2024-02-19T20:57:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: fsid = 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T20:57:52.456 INFO:teuthology.orchestra.run.smithi163.stdout: mon_host = 172.21.15.163:6789/0 172.21.15.193:6789/0 v1:172.21.15.163:6790/0 2024-02-19T20:57:52.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.475+0000 7f963b2ff700 10 librados: watch_flush enter 2024-02-19T20:57:52.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.475+0000 7f963b2ff700 10 librados: watch_flush exit 2024-02-19T20:57:52.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.475+0000 7f963b2ff700 20 client.34103.objecter shutdown clearing up homeless session... 2024-02-19T20:57:52.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.475+0000 7f963b2ff700 10 client.34103.objecter successfully canceled tick 2024-02-19T20:57:52.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.475+0000 7f963b2ff700 1 -- 172.21.15.163:0/3705887176 >> v1:172.21.15.163:6800/2735092549 conn(0x7f96200353e0 legacy=0x7f96200378a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:52.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.475+0000 7f963b2ff700 1 -- 172.21.15.163:0/3705887176 >> v1:172.21.15.163:6790/0 conn(0x7f96340fc780 legacy=0x7f96340fb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:52.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.475+0000 7f963b2ff700 1 -- 172.21.15.163:0/3705887176 shutdown_connections 2024-02-19T20:57:52.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.475+0000 7f963b2ff700 1 -- 172.21.15.163:0/3705887176 >> 172.21.15.163:0/3705887176 conn(0x7f963406c760 msgr2=0x7f9634060d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:52.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.475+0000 7f963b2ff700 1 -- 172.21.15.163:0/3705887176 shutdown_connections 2024-02-19T20:57:52.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.475+0000 7f963b2ff700 1 -- 172.21.15.163:0/3705887176 wait complete. 2024-02-19T20:57:52.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:57:52.475+0000 7f963b2ff700 1 librados: shutdown 2024-02-19T20:57:53.150 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-19T20:57:53.150 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T20:57:53.150 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.conf 2024-02-19T20:57:53.181 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T20:57:53.181 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T20:57:53.249 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-02-19T20:57:53.249 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.conf 2024-02-19T20:57:53.280 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-02-19T20:57:53.280 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T20:57:53.303 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:51.620661+0000 mgr.y (mgr.14146) 54 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:53.303 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:52.474646+0000 mon.c (mon.2) 2 : audit [DBG] from='client.? v1:172.21.15.163:0/3705887176' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:53.303 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:52.636159+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:53.303 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:52.636802+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T20:57:53.303 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:52.637328+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T20:57:53.303 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:52.637799+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:53.329 INFO:tasks.cephadm:Adding mgr.y on smithi163 2024-02-19T20:57:53.330 INFO:tasks.cephadm:Adding mgr.x on smithi193 2024-02-19T20:57:53.330 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch apply mgr '2;smithi163=y;smithi193=x' 2024-02-19T20:57:53.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:51.620661+0000 mgr.y (mgr.14146) 54 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:53.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:52.474646+0000 mon.c (mon.2) 2 : audit [DBG] from='client.? v1:172.21.15.163:0/3705887176' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:53.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:52.636159+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:53.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:52.636802+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T20:57:53.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:52.637328+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T20:57:53.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:52.637799+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:53.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:51.620661+0000 mgr.y (mgr.14146) 54 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:53.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:52.474646+0000 mon.c (mon.2) 2 : audit [DBG] from='client.? v1:172.21.15.163:0/3705887176' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:53.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:52.636159+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:53.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:52.636802+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T20:57:53.481 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:52.637328+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T20:57:53.481 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:52.637799+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:54.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:52.636596+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-19T20:57:54.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:52.638212+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Reconfiguring daemon mon.b on smithi193 2024-02-19T20:57:54.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:53.954700+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:54.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:53.955816+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:57:54.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:53.957647+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:54.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:53.958338+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:57:54.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:53.964538+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:54.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:57:52.636596+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-19T20:57:54.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:57:52.638212+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Reconfiguring daemon mon.b on smithi193 2024-02-19T20:57:54.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:53.954700+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:54.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:53.955816+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:57:54.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:53.957647+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:54.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:53.958338+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:57:54.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:53.964538+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:54.553 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:57:52.636596+0000 mgr.y (mgr.14146) 55 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-02-19T20:57:54.553 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:57:52.638212+0000 mgr.y (mgr.14146) 56 : cephadm [INF] Reconfiguring daemon mon.b on smithi193 2024-02-19T20:57:54.553 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:53.954700+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:54.553 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:53.955816+0000 mon.a (mon.0) 185 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:57:54.553 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:53.957647+0000 mon.a (mon.0) 186 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:54.553 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:53.958338+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:57:54.554 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:53.964538+0000 mon.a (mon.0) 188 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:55.290 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:53.620949+0000 mgr.y (mgr.14146) 57 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:55.479 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:53.620949+0000 mgr.y (mgr.14146) 57 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:55.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:53.620949+0000 mgr.y (mgr.14146) 57 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:55.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.499+0000 7f07385a7700 1 -- 172.21.15.193:0/464121054 >> v1:172.21.15.193:6789/0 conn(0x7f07300f6720 legacy=0x7f07300f6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:55.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.499+0000 7f07385a7700 1 -- 172.21.15.193:0/464121054 shutdown_connections 2024-02-19T20:57:55.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.499+0000 7f07385a7700 1 -- 172.21.15.193:0/464121054 >> 172.21.15.193:0/464121054 conn(0x7f07300f1e60 msgr2=0x7f07300f42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:55.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.499+0000 7f07385a7700 1 -- 172.21.15.193:0/464121054 shutdown_connections 2024-02-19T20:57:55.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.499+0000 7f07385a7700 1 -- 172.21.15.193:0/464121054 wait complete. 2024-02-19T20:57:55.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.499+0000 7f07385a7700 1 librados: starting msgr at 2024-02-19T20:57:55.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.499+0000 7f07385a7700 1 librados: starting objecter 2024-02-19T20:57:55.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.499+0000 7f07385a7700 1 Processor -- start 2024-02-19T20:57:55.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.499+0000 7f07385a7700 1 -- start start 2024-02-19T20:57:55.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.499+0000 7f07385a7700 1 librados: setting wanted keys 2024-02-19T20:57:55.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.499+0000 7f07385a7700 1 librados: calling monclient init 2024-02-19T20:57:55.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07385a7700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0730112ed0 con 0x7f07300f6720 2024-02-19T20:57:55.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07385a7700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0730114070 con 0x7f07300fe820 2024-02-19T20:57:55.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07385a7700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0730115210 con 0x7f07300fabf0 2024-02-19T20:57:55.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f0736b44700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f07300fe820 0x7f07301117b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:51358/0 (socket says 172.21.15.193:51358) 2024-02-19T20:57:55.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f0736b44700 1 -- 172.21.15.193:0/453309291 learned_addr learned my addr 172.21.15.193:0/453309291 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T20:57:55.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 10 client.?.objecter ms_handle_connect 0x7f07300fe820 2024-02-19T20:57:55.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:55.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 10 client.?.objecter ms_handle_connect 0x7f07300f6720 2024-02-19T20:57:55.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:55.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 10 client.?.objecter ms_handle_connect 0x7f07300fabf0 2024-02-19T20:57:55.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:57:55.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291783781 0 0) 0x7f0730114070 con 0x7f07300fe820 2024-02-19T20:57:55.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f071c002220 con 0x7f07300fe820 2024-02-19T20:57:55.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233968790 0 0) 0x7f0730115210 con 0x7f07300fabf0 2024-02-19T20:57:55.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0730114070 con 0x7f07300fabf0 2024-02-19T20:57:55.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467433601 0 0) 0x7f0730112ed0 con 0x7f07300f6720 2024-02-19T20:57:55.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0730115210 con 0x7f07300f6720 2024-02-19T20:57:55.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3497841281 0 0) 0x7f071c002220 con 0x7f07300fe820 2024-02-19T20:57:55.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0730112ed0 con 0x7f07300fe820 2024-02-19T20:57:55.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0728001c80 con 0x7f07300fe820 2024-02-19T20:57:55.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3271632801 0 0) 0x7f0730114070 con 0x7f07300fabf0 2024-02-19T20:57:55.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f071c002220 con 0x7f07300fabf0 2024-02-19T20:57:55.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2797998505 0 0) 0x7f0730115210 con 0x7f07300f6720 2024-02-19T20:57:55.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0730114070 con 0x7f07300f6720 2024-02-19T20:57:55.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f072c0018c0 con 0x7f07300fabf0 2024-02-19T20:57:55.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1358102964 0 0) 0x7f0730112ed0 con 0x7f07300fe820 2024-02-19T20:57:55.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 >> v1:172.21.15.163:6790/0 conn(0x7f07300fabf0 legacy=0x7f073010e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:55.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 >> v1:172.21.15.163:6789/0 conn(0x7f07300f6720 legacy=0x7f073010a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:55.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07301163b0 con 0x7f07300fe820 2024-02-19T20:57:55.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.500+0000 7f07385a7700 1 -- 172.21.15.193:0/453309291 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07301130c0 con 0x7f07300fe820 2024-02-19T20:57:55.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.501+0000 7f07385a7700 10 client.24112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:57:55.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.501+0000 7f07385a7700 1 -- 172.21.15.193:0/453309291 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07301135b0 con 0x7f07300fe820 2024-02-19T20:57:55.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.501+0000 7f07385a7700 1 librados: init done 2024-02-19T20:57:55.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.501+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f0728002690 con 0x7f07300fe820 2024-02-19T20:57:55.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.501+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0728005da0 con 0x7f07300fe820 2024-02-19T20:57:55.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.501+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 4600635 0 0) 0x7f072800f600 con 0x7f07300fe820 2024-02-19T20:57:55.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.501+0000 7f07385a7700 1 -- 172.21.15.193:0/453309291 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0714004f20 con 0x7f07300fe820 2024-02-19T20:57:55.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.501+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (unknown 1462125392 0 0) 0x7f072803ec40 con 0x7f07300fe820 2024-02-19T20:57:55.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.501+0000 7f07277fe700 10 client.24112.objecter ms_dispatch 0x7f0730001550 osd_map(4..4 src has 1..4) v4 2024-02-19T20:57:55.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.501+0000 7f07277fe700 3 client.24112.objecter handle_osd_map got epochs [4,4] > 0 2024-02-19T20:57:55.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.501+0000 7f07277fe700 3 client.24112.objecter handle_osd_map decoding full epoch 4 2024-02-19T20:57:55.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.501+0000 7f07277fe700 20 client.24112.objecter dump_active .. 0 homeless 2024-02-19T20:57:55.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.502+0000 7f07277fe700 10 client.24112.objecter ms_handle_connect 0x7f071c0353e0 2024-02-19T20:57:55.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.506+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0728010e30 con 0x7f07300fe820 2024-02-19T20:57:55.659 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.676+0000 7f07385a7700 1 -- 172.21.15.193:0/453309291 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi163=y;smithi193=x", "target": ["mon-mgr", ""]}) v1 -- 0x7f0714000b90 con 0x7f071c0353e0 2024-02-19T20:57:55.665 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.683+0000 7f07277fe700 1 -- 172.21.15.193:0/453309291 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 325935098) 0x7f0714000b90 con 0x7f071c0353e0 2024-02-19T20:57:55.666 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mgr update... 2024-02-19T20:57:55.667 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.685+0000 7f07385a7700 10 librados: watch_flush enter 2024-02-19T20:57:55.667 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.685+0000 7f07385a7700 10 librados: watch_flush exit 2024-02-19T20:57:55.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.685+0000 7f07385a7700 20 client.24112.objecter shutdown clearing up homeless session... 2024-02-19T20:57:55.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.685+0000 7f07385a7700 10 client.24112.objecter successfully canceled tick 2024-02-19T20:57:55.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.685+0000 7f07385a7700 1 -- 172.21.15.193:0/453309291 >> v1:172.21.15.163:6800/2735092549 conn(0x7f071c0353e0 legacy=0x7f071c0378a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:55.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.685+0000 7f07385a7700 1 -- 172.21.15.193:0/453309291 >> v1:172.21.15.193:6789/0 conn(0x7f07300fe820 legacy=0x7f07301117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:57:55.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.685+0000 7f07385a7700 1 -- 172.21.15.193:0/453309291 shutdown_connections 2024-02-19T20:57:55.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.685+0000 7f07385a7700 1 -- 172.21.15.193:0/453309291 >> 172.21.15.193:0/453309291 conn(0x7f07300f1e60 msgr2=0x7f07300fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:57:55.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.685+0000 7f07385a7700 1 -- 172.21.15.193:0/453309291 shutdown_connections 2024-02-19T20:57:55.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.685+0000 7f07385a7700 1 -- 172.21.15.193:0/453309291 wait complete. 2024-02-19T20:57:55.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:57:55.685+0000 7f07385a7700 1 librados: shutdown 2024-02-19T20:57:56.268 DEBUG:teuthology.orchestra.run.smithi193:mgr.x> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@mgr.x.service 2024-02-19T20:57:56.271 INFO:tasks.cephadm:Deploying OSDs... 2024-02-19T20:57:56.271 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T20:57:56.271 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/scratch_devs of=/dev/stdout 2024-02-19T20:57:56.294 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-19T20:57:56.294 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_1 2024-02-19T20:57:56.302 INFO:journalctl@ceph.mgr.x.smithi193.stdout:-- Logs begin at Mon 2024-02-19 20:44:17 UTC. -- 2024-02-19T20:57:56.355 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-19T20:57:56.355 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T20:57:56.355 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 6h/6d Inode: 167527 Links: 1 2024-02-19T20:57:56.355 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T20:57:56.355 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T20:57:56.355 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-02-19 20:57:41.080662495 +0000 2024-02-19T20:57:56.355 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-02-19 20:50:05.521761528 +0000 2024-02-19T20:57:56.355 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-02-19 20:50:05.521761528 +0000 2024-02-19T20:57:56.355 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: - 2024-02-19T20:57:56.356 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-19T20:57:56.424 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-02-19T20:57:56.424 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-02-19T20:57:56.425 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000288154 s, 1.8 MB/s 2024-02-19T20:57:56.426 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-19T20:57:56.486 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_2 2024-02-19T20:57:56.545 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-19T20:57:56.546 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T20:57:56.546 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 6h/6d Inode: 163389 Links: 1 2024-02-19T20:57:56.546 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T20:57:56.546 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T20:57:56.546 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-02-19 20:57:41.081662473 +0000 2024-02-19T20:57:56.546 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-02-19 20:50:05.836754487 +0000 2024-02-19T20:57:56.546 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-02-19 20:50:05.836754487 +0000 2024-02-19T20:57:56.546 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: - 2024-02-19T20:57:56.546 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-19T20:57:56.613 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-02-19T20:57:56.613 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-02-19T20:57:56.613 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000238431 s, 2.1 MB/s 2024-02-19T20:57:56.614 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-19T20:57:56.675 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_3 2024-02-19T20:57:56.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:55.621292+0000 mgr.y (mgr.14146) 58 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:56.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:55.679120+0000 mgr.y (mgr.14146) 59 : audit [DBG] from='client.24112 v1:172.21.15.193:0/453309291' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi163=y;smithi193=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:57:56.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:55.680546+0000 mgr.y (mgr.14146) 60 : cephadm [INF] Saving service mgr spec with placement smithi163=y;smithi193=x;count:2 2024-02-19T20:57:56.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:55.684141+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:56.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:55.734546+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:57:56.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:55.736239+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:56.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:55.737026+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:57:56.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:55.741561+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:56.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:55.742939+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T20:57:56.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:55.746102+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-19T20:57:56.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:55.746910+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T20:57:56.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:55.747713+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:56.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:55.748319+0000 mgr.y (mgr.14146) 61 : cephadm [INF] Deploying daemon mgr.x on smithi193 2024-02-19T20:57:56.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:55.621292+0000 mgr.y (mgr.14146) 58 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:56.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:55.679120+0000 mgr.y (mgr.14146) 59 : audit [DBG] from='client.24112 v1:172.21.15.193:0/453309291' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi163=y;smithi193=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:57:56.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:57:55.680546+0000 mgr.y (mgr.14146) 60 : cephadm [INF] Saving service mgr spec with placement smithi163=y;smithi193=x;count:2 2024-02-19T20:57:56.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:55.684141+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:56.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:55.734546+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:57:56.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:55.736239+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:56.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:55.737026+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:57:56.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:55.741561+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:56.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:55.742939+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T20:57:56.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:55.746102+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-19T20:57:56.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:55.746910+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T20:57:56.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:55.747713+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:56.732 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:57:55.748319+0000 mgr.y (mgr.14146) 61 : cephadm [INF] Deploying daemon mgr.x on smithi193 2024-02-19T20:57:56.733 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-19T20:57:56.733 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T20:57:56.733 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 6h/6d Inode: 168235 Links: 1 2024-02-19T20:57:56.734 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T20:57:56.734 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T20:57:56.734 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-02-19 20:57:41.081662473 +0000 2024-02-19T20:57:56.734 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-02-19 20:50:06.172746976 +0000 2024-02-19T20:57:56.734 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-02-19 20:50:06.172746976 +0000 2024-02-19T20:57:56.734 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: - 2024-02-19T20:57:56.734 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-19T20:57:56.799 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-02-19T20:57:56.799 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-02-19T20:57:56.799 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000213293 s, 2.4 MB/s 2024-02-19T20:57:56.800 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-19T20:57:56.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:55.621292+0000 mgr.y (mgr.14146) 58 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:56.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:55.679120+0000 mgr.y (mgr.14146) 59 : audit [DBG] from='client.24112 v1:172.21.15.193:0/453309291' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi163=y;smithi193=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:57:56.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:57:55.680546+0000 mgr.y (mgr.14146) 60 : cephadm [INF] Saving service mgr spec with placement smithi163=y;smithi193=x;count:2 2024-02-19T20:57:56.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:55.684141+0000 mon.a (mon.0) 189 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:56.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:55.734546+0000 mon.a (mon.0) 190 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:57:56.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:55.736239+0000 mon.a (mon.0) 191 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:56.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:55.737026+0000 mon.a (mon.0) 192 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:57:56.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:55.741561+0000 mon.a (mon.0) 193 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:56.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:55.742939+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T20:57:56.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:55.746102+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-19T20:57:56.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:55.746910+0000 mon.a (mon.0) 196 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T20:57:56.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:55.747713+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:57:56.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:57:55.748319+0000 mgr.y (mgr.14146) 61 : cephadm [INF] Deploying daemon mgr.x on smithi193 2024-02-19T20:57:56.859 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_4 2024-02-19T20:57:56.917 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-19T20:57:56.917 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T20:57:56.917 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 6h/6d Inode: 163510 Links: 1 2024-02-19T20:57:56.917 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T20:57:56.918 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T20:57:56.918 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-02-19 20:57:41.081662473 +0000 2024-02-19T20:57:56.918 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-02-19 20:50:06.524739107 +0000 2024-02-19T20:57:56.918 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-02-19 20:50:06.524739107 +0000 2024-02-19T20:57:56.918 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: - 2024-02-19T20:57:56.918 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-19T20:57:56.982 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-02-19T20:57:56.982 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-02-19T20:57:56.983 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000226423 s, 2.3 MB/s 2024-02-19T20:57:56.983 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-19T20:57:57.042 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-02-19T20:57:57.042 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2024-02-19T20:57:57.062 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-19T20:57:57.062 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2024-02-19T20:57:57.120 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-19T20:57:57.120 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T20:57:57.121 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 6h/6d Inode: 166129 Links: 1 2024-02-19T20:57:57.121 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T20:57:57.121 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T20:57:57.121 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-02-19 20:57:32.504668621 +0000 2024-02-19T20:57:57.121 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-02-19 20:50:05.511181185 +0000 2024-02-19T20:57:57.121 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-02-19 20:50:05.511181185 +0000 2024-02-19T20:57:57.121 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2024-02-19T20:57:57.121 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-19T20:57:57.187 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-02-19T20:57:57.187 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-02-19T20:57:57.187 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000228663 s, 2.2 MB/s 2024-02-19T20:57:57.188 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-19T20:57:57.247 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2024-02-19T20:57:57.305 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-19T20:57:57.305 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T20:57:57.305 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 6h/6d Inode: 162589 Links: 1 2024-02-19T20:57:57.306 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T20:57:57.306 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T20:57:57.306 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-02-19 20:57:32.504668621 +0000 2024-02-19T20:57:57.306 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-02-19 20:50:05.839175706 +0000 2024-02-19T20:57:57.306 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-02-19 20:50:05.839175706 +0000 2024-02-19T20:57:57.306 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2024-02-19T20:57:57.306 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-19T20:57:57.371 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-02-19T20:57:57.371 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-02-19T20:57:57.371 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000213589 s, 2.4 MB/s 2024-02-19T20:57:57.372 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-19T20:57:57.430 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2024-02-19T20:57:57.488 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-19T20:57:57.488 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T20:57:57.488 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 6h/6d Inode: 167815 Links: 1 2024-02-19T20:57:57.488 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T20:57:57.488 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T20:57:57.488 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-02-19 20:57:32.504668621 +0000 2024-02-19T20:57:57.488 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-02-19 20:50:06.176170077 +0000 2024-02-19T20:57:57.488 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-02-19 20:50:06.176170077 +0000 2024-02-19T20:57:57.488 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2024-02-19T20:57:57.488 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-19T20:57:57.596 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-02-19T20:57:57.596 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-02-19T20:57:57.596 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000172814 s, 3.0 MB/s 2024-02-19T20:57:57.597 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-19T20:57:57.615 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2024-02-19T20:57:57.674 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-19T20:57:57.674 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T20:57:57.675 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 6h/6d Inode: 165249 Links: 1 2024-02-19T20:57:57.675 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T20:57:57.675 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T20:57:57.675 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-02-19 20:57:57.569252357 +0000 2024-02-19T20:57:57.675 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-02-19 20:50:06.493164782 +0000 2024-02-19T20:57:57.675 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-02-19 20:50:06.493164782 +0000 2024-02-19T20:57:57.675 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2024-02-19T20:57:57.675 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-19T20:57:57.740 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:57:57 smithi193 systemd[1]: Starting Ceph mgr.x for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T20:57:57.741 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-02-19T20:57:57.741 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-02-19T20:57:57.741 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.00022784 s, 2.2 MB/s 2024-02-19T20:57:57.743 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-19T20:57:57.803 INFO:tasks.cephadm:Deploying osd.0 on smithi163 with /dev/vg_nvme/lv_4... 2024-02-19T20:57:57.804 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-19T20:57:58.668 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:57:58 smithi193 bash[40771]: e051ef8dc53aac904458b09ff1a618f7a2669343b546aa167ae2c41333b80e5d 2024-02-19T20:57:58.668 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:57:58 smithi193 systemd[1]: Started Ceph mgr.x for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T20:57:58.668 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:57:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:57:58.369+0000 7fdbc2744680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-19T20:57:58.668 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:57:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:57:58.455+0000 7fdbc2744680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-19T20:57:58.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:57.621539+0000 mgr.y (mgr.14146) 62 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:58.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:58.417907+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:58.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:57:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:58.419309+0000 mon.a (mon.0) 199 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:57:58.980 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:57.621539+0000 mgr.y (mgr.14146) 62 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:58.980 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:58.417907+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:58.980 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:57:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:58.419309+0000 mon.a (mon.0) 199 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:57:59.045 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:57:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:57:58.844+0000 7fdbc2744680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-19T20:57:59.045 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:57.621539+0000 mgr.y (mgr.14146) 62 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:57:59.045 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:58.417907+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:57:59.045 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:57:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:58.419309+0000 mon.a (mon.0) 199 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:57:59.886 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:57:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:57:59.611+0000 7fdbc2744680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-19T20:57:59.886 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:57:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:57:59.694+0000 7fdbc2744680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-19T20:57:59.989 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-19T20:57:59.989 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-19T20:57:59.990 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-02-19T20:57:59.990 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-02-19T20:57:59.990 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011009 s, 952 MB/s 2024-02-19T20:57:59.990 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-02-19T20:58:00.648 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi163:vg_nvme/lv_4 2024-02-19T20:58:00.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:59.427613+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:00.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:59.428171+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:00.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:59.428757+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:00.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:59.434859+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:00.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:59.436174+0000 mgr.y (mgr.14146) 63 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-19T20:58:00.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:59.436392+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T20:58:00.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:59.436963+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T20:58:00.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:59.437391+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:00.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:57:59.437799+0000 mgr.y (mgr.14146) 64 : cephadm [INF] Reconfiguring daemon mgr.y on smithi163 2024-02-19T20:58:00.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:57:59.621787+0000 mgr.y (mgr.14146) 65 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:00.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:57:59.754909+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:00.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:59.427613+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:00.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:59.428171+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:00.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:59.428757+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:00.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:59.434859+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:00.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:57:59.436174+0000 mgr.y (mgr.14146) 63 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-19T20:58:00.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:59.436392+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T20:58:00.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:59.436963+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T20:58:00.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:59.437391+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:00.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:57:59.437799+0000 mgr.y (mgr.14146) 64 : cephadm [INF] Reconfiguring daemon mgr.y on smithi163 2024-02-19T20:58:00.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:57:59.621787+0000 mgr.y (mgr.14146) 65 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:00.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:57:59.754909+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:00.803 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:57:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:57:59.903+0000 7fdbc2744680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-19T20:58:00.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:59.427613+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:00.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:59.428171+0000 mon.a (mon.0) 201 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:00.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:59.428757+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:00.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:59.434859+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:00.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:57:59.436174+0000 mgr.y (mgr.14146) 63 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-02-19T20:58:00.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:59.436392+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T20:58:00.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:59.436963+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T20:58:00.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:59.437391+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:00.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:57:59.437799+0000 mgr.y (mgr.14146) 64 : cephadm [INF] Reconfiguring daemon mgr.y on smithi163 2024-02-19T20:58:00.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:57:59.621787+0000 mgr.y (mgr.14146) 65 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:00.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:57:59.754909+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:01.108 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:00.837+0000 7fdbc2744680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-19T20:58:01.108 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:00.961+0000 7fdbc2744680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-19T20:58:01.108 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:01.038+0000 7fdbc2744680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-19T20:58:01.553 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:01.125+0000 7fdbc2744680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-19T20:58:01.553 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:01.320+0000 7fdbc2744680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-19T20:58:02.053 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:01.595+0000 7fdbc2744680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-19T20:58:02.053 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:01.691+0000 7fdbc2744680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-19T20:58:02.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:01.392752+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:02.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:01.393661+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:58:02.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:01.394855+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:02.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:01.395348+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:02.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:01.399497+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:02.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:01.622043+0000 mgr.y (mgr.14146) 66 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:02.651 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:01.392752+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:02.651 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:01.393661+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:58:02.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:01.394855+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:02.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:01.395348+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:02.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:01.399497+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:02.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:01.622043+0000 mgr.y (mgr.14146) 66 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:02.727 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:02.405+0000 7fdbc2744680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-19T20:58:02.727 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:02.485+0000 7fdbc2744680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-19T20:58:02.727 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:02.576+0000 7fdbc2744680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-19T20:58:02.728 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:01.392752+0000 mon.a (mon.0) 208 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:02.728 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:01.393661+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:58:02.728 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:01.394855+0000 mon.a (mon.0) 210 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:02.728 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:01.395348+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:02.728 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:01.399497+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:02.728 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:01.622043+0000 mgr.y (mgr.14146) 66 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:02.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.850+0000 7fc200c93700 1 -- 172.21.15.163:0/2269005170 >> v1:172.21.15.163:6790/0 conn(0x7fc1fc06cb70 legacy=0x7fc1fc06b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:58:02.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.850+0000 7fc200c93700 1 -- 172.21.15.163:0/2269005170 shutdown_connections 2024-02-19T20:58:02.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.850+0000 7fc200c93700 1 -- 172.21.15.163:0/2269005170 >> 172.21.15.163:0/2269005170 conn(0x7fc1fc0f2fb0 msgr2=0x7fc1fc0f53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:58:02.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.850+0000 7fc200c93700 1 -- 172.21.15.163:0/2269005170 shutdown_connections 2024-02-19T20:58:02.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.850+0000 7fc200c93700 1 -- 172.21.15.163:0/2269005170 wait complete. 2024-02-19T20:58:02.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.850+0000 7fc200c93700 1 librados: starting msgr at 2024-02-19T20:58:02.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.850+0000 7fc200c93700 1 librados: starting objecter 2024-02-19T20:58:02.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc200c93700 1 Processor -- start 2024-02-19T20:58:02.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc200c93700 1 -- start start 2024-02-19T20:58:02.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc200c93700 1 librados: setting wanted keys 2024-02-19T20:58:02.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc200c93700 1 librados: calling monclient init 2024-02-19T20:58:02.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc200c93700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc1fc10a570 con 0x7fc1fc06b780 2024-02-19T20:58:02.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc200c93700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc1fc115dd0 con 0x7fc1fc06cb70 2024-02-19T20:58:02.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc200c93700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc1fc116fb0 con 0x7fc1fc101ef0 2024-02-19T20:58:02.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc1fa59c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fc1fc06b780 0x7fc1fc1099a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:49032/0 (socket says 172.21.15.163:49032) 2024-02-19T20:58:02.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc1fa59c700 1 -- 172.21.15.163:0/1063441556 learned_addr learned my addr 172.21.15.163:0/1063441556 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:58:02.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc1eb7fe700 10 client.?.objecter ms_handle_connect 0x7fc1fc101ef0 2024-02-19T20:58:02.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc1eb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:58:02.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc1eb7fe700 10 client.?.objecter ms_handle_connect 0x7fc1fc06b780 2024-02-19T20:58:02.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc1eb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:58:02.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc1eb7fe700 10 client.?.objecter ms_handle_connect 0x7fc1fc06cb70 2024-02-19T20:58:02.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc1eb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:58:02.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999579786 0 0) 0x7fc1fc10a570 con 0x7fc1fc06b780 2024-02-19T20:58:02.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e4003800 con 0x7fc1fc06b780 2024-02-19T20:58:02.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119259378 0 0) 0x7fc1fc116fb0 con 0x7fc1fc101ef0 2024-02-19T20:58:02.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1fc10a570 con 0x7fc1fc101ef0 2024-02-19T20:58:02.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632954251 0 0) 0x7fc1fc115dd0 con 0x7fc1fc06cb70 2024-02-19T20:58:02.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.851+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1fc116fb0 con 0x7fc1fc06cb70 2024-02-19T20:58:02.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3059211626 0 0) 0x7fc1e4003800 con 0x7fc1fc06b780 2024-02-19T20:58:02.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1fc115dd0 con 0x7fc1fc06b780 2024-02-19T20:58:02.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc1f0002ea0 con 0x7fc1fc06b780 2024-02-19T20:58:02.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1354374582 0 0) 0x7fc1fc116fb0 con 0x7fc1fc06cb70 2024-02-19T20:58:02.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1e4003800 con 0x7fc1fc06cb70 2024-02-19T20:58:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 429181663 0 0) 0x7fc1fc10a570 con 0x7fc1fc101ef0 2024-02-19T20:58:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1fc116fb0 con 0x7fc1fc101ef0 2024-02-19T20:58:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc1ec0019f0 con 0x7fc1fc06cb70 2024-02-19T20:58:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc1f4001b80 con 0x7fc1fc101ef0 2024-02-19T20:58:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1536479654 0 0) 0x7fc1fc115dd0 con 0x7fc1fc06b780 2024-02-19T20:58:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 >> v1:172.21.15.163:6790/0 conn(0x7fc1fc101ef0 legacy=0x7fc1fc1136a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:58:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 >> v1:172.21.15.193:6789/0 conn(0x7fc1fc06cb70 legacy=0x7fc1fc10ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:58:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc118190 con 0x7fc1fc06b780 2024-02-19T20:58:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc200c93700 1 -- 172.21.15.163:0/1063441556 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1fc114e20 con 0x7fc1fc06b780 2024-02-19T20:58:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc200c93700 10 client.14211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:58:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc200c93700 1 -- 172.21.15.163:0/1063441556 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1fc115310 con 0x7fc1fc06b780 2024-02-19T20:58:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc200c93700 1 librados: init done 2024-02-19T20:58:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fc1f0002070 con 0x7fc1fc06b780 2024-02-19T20:58:02.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.852+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc1f0003b10 con 0x7fc1fc06b780 2024-02-19T20:58:02.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.853+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 13) v1 ==== 38639+0+0 (unknown 4600635 0 0) 0x7fc1f0003d10 con 0x7fc1fc06b780 2024-02-19T20:58:02.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.853+0000 7fc200c93700 1 -- 172.21.15.163:0/1063441556 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1dc004f20 con 0x7fc1fc06b780 2024-02-19T20:58:02.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.853+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1012+0+0 (unknown 1462125392 0 0) 0x7fc1f003ede0 con 0x7fc1fc06b780 2024-02-19T20:58:02.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.853+0000 7fc1eb7fe700 10 client.14211.objecter ms_dispatch 0x7fc1fc001550 osd_map(4..4 src has 1..4) v4 2024-02-19T20:58:02.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.853+0000 7fc1eb7fe700 3 client.14211.objecter handle_osd_map got epochs [4,4] > 0 2024-02-19T20:58:02.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.853+0000 7fc1eb7fe700 3 client.14211.objecter handle_osd_map decoding full epoch 4 2024-02-19T20:58:02.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.853+0000 7fc1eb7fe700 20 client.14211.objecter dump_active .. 0 homeless 2024-02-19T20:58:02.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.853+0000 7fc1eb7fe700 10 client.14211.objecter ms_handle_connect 0x7fc1e4035410 2024-02-19T20:58:02.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:02.857+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc1f0011250 con 0x7fc1fc06b780 2024-02-19T20:58:03.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:03.027+0000 7fc200c93700 1 -- 172.21.15.163:0/1063441556 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1dc000b90 con 0x7fc1e4035410 2024-02-19T20:58:03.244 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:02.744+0000 7fdbc2744680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-19T20:58:03.244 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:03.003+0000 7fdbc2744680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-19T20:58:03.244 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:03.131+0000 7fdbc2744680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-19T20:58:03.495 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:03.261+0000 7fdbc2744680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-19T20:58:03.495 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:03.430+0000 7fdbc2744680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-19T20:58:03.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:03.028626+0000 mgr.y (mgr.14146) 67 : audit [DBG] from='client.14211 v1:172.21.15.163:0/1063441556' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:58:03.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:03.030893+0000 mon.a (mon.0) 213 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T20:58:03.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:03.032757+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T20:58:03.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:03.033198+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:03.721 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:03.028626+0000 mgr.y (mgr.14146) 67 : audit [DBG] from='client.14211 v1:172.21.15.163:0/1063441556' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:58:03.721 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:03.030893+0000 mon.a (mon.0) 213 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T20:58:03.721 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:03.032757+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T20:58:03.721 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:03.033198+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:03.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:03.028626+0000 mgr.y (mgr.14146) 67 : audit [DBG] from='client.14211 v1:172.21.15.163:0/1063441556' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:58:03.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:03.030893+0000 mon.a (mon.0) 213 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T20:58:03.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:03.032757+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T20:58:03.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:03.033198+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:03.803 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 20:58:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T20:58:03.512+0000 7fdbc2744680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-19T20:58:04.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:04.407+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 7720783 0 0) 0x7fc1f00117d0 con 0x7fc1fc06b780 2024-02-19T20:58:04.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:03.526616+0000 mon.a (mon.0) 216 : cluster [DBG] Standby manager daemon x started 2024-02-19T20:58:04.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:03.527626+0000 mon.c (mon.2) 3 : audit [DBG] from='mgr.? v1:172.21.15.193:0/1434739173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-19T20:58:04.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:03.527919+0000 mon.c (mon.2) 4 : audit [DBG] from='mgr.? v1:172.21.15.193:0/1434739173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T20:58:04.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:03.528804+0000 mon.c (mon.2) 5 : audit [DBG] from='mgr.? v1:172.21.15.193:0/1434739173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-19T20:58:04.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:03.529082+0000 mon.c (mon.2) 6 : audit [DBG] from='mgr.? v1:172.21.15.193:0/1434739173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T20:58:04.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:03.622258+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:04.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:03.526616+0000 mon.a (mon.0) 216 : cluster [DBG] Standby manager daemon x started 2024-02-19T20:58:04.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:03.527626+0000 mon.c (mon.2) 3 : audit [DBG] from='mgr.? v1:172.21.15.193:0/1434739173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-19T20:58:04.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:03.527919+0000 mon.c (mon.2) 4 : audit [DBG] from='mgr.? v1:172.21.15.193:0/1434739173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T20:58:04.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:03.528804+0000 mon.c (mon.2) 5 : audit [DBG] from='mgr.? v1:172.21.15.193:0/1434739173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-19T20:58:04.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:03.529082+0000 mon.c (mon.2) 6 : audit [DBG] from='mgr.? v1:172.21.15.193:0/1434739173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T20:58:04.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:03.622258+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:04.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:03.526616+0000 mon.a (mon.0) 216 : cluster [DBG] Standby manager daemon x started 2024-02-19T20:58:04.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:03.527626+0000 mon.c (mon.2) 3 : audit [DBG] from='mgr.? v1:172.21.15.193:0/1434739173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-19T20:58:04.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:03.527919+0000 mon.c (mon.2) 4 : audit [DBG] from='mgr.? v1:172.21.15.193:0/1434739173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T20:58:04.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:03.528804+0000 mon.c (mon.2) 5 : audit [DBG] from='mgr.? v1:172.21.15.193:0/1434739173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-19T20:58:04.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:03.529082+0000 mon.c (mon.2) 6 : audit [DBG] from='mgr.? v1:172.21.15.193:0/1434739173' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T20:58:04.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:03.622258+0000 mgr.y (mgr.14146) 68 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:05.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:04.408918+0000 mon.a (mon.0) 217 : cluster [DBG] mgrmap e14: y(active, since 94s), standbys: x 2024-02-19T20:58:05.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:04.409035+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-19T20:58:05.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:05.372719+0000 mon.c (mon.2) 7 : audit [INF] from='client.? v1:172.21.15.163:0/1657910190' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088"}]: dispatch 2024-02-19T20:58:05.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:05.373028+0000 mon.a (mon.0) 219 : audit [INF] from='client.34127 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088"}]: dispatch 2024-02-19T20:58:05.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:05.376769+0000 mon.a (mon.0) 220 : audit [INF] from='client.34127 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088"}]': finished 2024-02-19T20:58:05.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:05.376818+0000 mon.a (mon.0) 221 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-19T20:58:05.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:05.376937+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:05.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:04.408918+0000 mon.a (mon.0) 217 : cluster [DBG] mgrmap e14: y(active, since 94s), standbys: x 2024-02-19T20:58:05.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:04.409035+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-19T20:58:05.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:05.372719+0000 mon.c (mon.2) 7 : audit [INF] from='client.? v1:172.21.15.163:0/1657910190' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088"}]: dispatch 2024-02-19T20:58:05.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:05.373028+0000 mon.a (mon.0) 219 : audit [INF] from='client.34127 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088"}]: dispatch 2024-02-19T20:58:05.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:05.376769+0000 mon.a (mon.0) 220 : audit [INF] from='client.34127 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088"}]': finished 2024-02-19T20:58:05.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:05.376818+0000 mon.a (mon.0) 221 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-19T20:58:05.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:05.376937+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:05.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:04.408918+0000 mon.a (mon.0) 217 : cluster [DBG] mgrmap e14: y(active, since 94s), standbys: x 2024-02-19T20:58:05.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:04.409035+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-19T20:58:05.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:05.372719+0000 mon.c (mon.2) 7 : audit [INF] from='client.? v1:172.21.15.163:0/1657910190' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088"}]: dispatch 2024-02-19T20:58:05.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:05.373028+0000 mon.a (mon.0) 219 : audit [INF] from='client.34127 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088"}]: dispatch 2024-02-19T20:58:05.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:05.376769+0000 mon.a (mon.0) 220 : audit [INF] from='client.34127 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088"}]': finished 2024-02-19T20:58:05.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:05.376818+0000 mon.a (mon.0) 221 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-02-19T20:58:05.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:05.376937+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:06.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:05.622450+0000 mgr.y (mgr.14146) 69 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:06.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:05.941417+0000 mon.c (mon.2) 8 : audit [DBG] from='client.? v1:172.21.15.163:0/2509232294' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T20:58:06.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:05.622450+0000 mgr.y (mgr.14146) 69 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:06.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:05.941417+0000 mon.c (mon.2) 8 : audit [DBG] from='client.? v1:172.21.15.163:0/2509232294' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T20:58:06.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:05.622450+0000 mgr.y (mgr.14146) 69 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:06.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:05.941417+0000 mon.c (mon.2) 8 : audit [DBG] from='client.? v1:172.21.15.163:0/2509232294' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T20:58:07.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:07.853+0000 7fc1fbfff700 10 client.14211.objecter tick 2024-02-19T20:58:09.221 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:07.622755+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:09.222 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:07.622755+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:09.303 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:07.622755+0000 mgr.y (mgr.14146) 70 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:11.099 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:09.623008+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:11.099 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:09.623008+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:11.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:09.623008+0000 mgr.y (mgr.14146) 71 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:12.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:11.423376+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T20:58:12.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:11.423932+0000 mon.a (mon.0) 224 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:12.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:11.423376+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T20:58:12.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:11.423932+0000 mon.a (mon.0) 224 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:12.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:11.423376+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T20:58:12.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:11.423932+0000 mon.a (mon.0) 224 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:12.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:12.853+0000 7fc1fbfff700 10 client.14211.objecter tick 2024-02-19T20:58:13.048 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:58:11.424397+0000 mgr.y (mgr.14146) 72 : cephadm [INF] Deploying daemon osd.0 on smithi163 2024-02-19T20:58:13.048 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:11.623321+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:13.049 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:58:11.424397+0000 mgr.y (mgr.14146) 72 : cephadm [INF] Deploying daemon osd.0 on smithi163 2024-02-19T20:58:13.049 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:11.623321+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:13.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:58:11.424397+0000 mgr.y (mgr.14146) 72 : cephadm [INF] Deploying daemon osd.0 on smithi163 2024-02-19T20:58:13.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:11.623321+0000 mgr.y (mgr.14146) 73 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:15.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:13.623554+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:15.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:13.623554+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:15.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:13.623554+0000 mgr.y (mgr.14146) 74 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:17.124 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:15.623836+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:17.124 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:16.406070+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:17.124 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:16.445546+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:58:17.124 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:15.623836+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:17.124 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:16.406070+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:17.124 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:16.445546+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:58:17.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:15.623836+0000 mgr.y (mgr.14146) 75 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:17.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:16.406070+0000 mon.a (mon.0) 225 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:17.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:16.445546+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:58:17.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:17.854+0000 7fc1fbfff700 10 client.14211.objecter tick 2024-02-19T20:58:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:17.624061+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:18.863386+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:18.864076+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:18.864696+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:18.868757+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:19.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:17.624061+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:19.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:18.863386+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:19.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:18.864076+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:19.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:18.864696+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:19.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:18.868757+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:19.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:17.624061+0000 mgr.y (mgr.14146) 76 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:19.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:18.863386+0000 mon.a (mon.0) 227 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:19.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:18.864076+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:19.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:18.864696+0000 mon.a (mon.0) 229 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:19.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:18.868757+0000 mon.a (mon.0) 230 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:19.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:19.553+0000 7fc1eb7fe700 1 -- 172.21.15.163:0/1063441556 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 183196380) 0x7fc1dc000b90 con 0x7fc1e4035410 2024-02-19T20:58:19.537 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 0 on host 'smithi163' 2024-02-19T20:58:19.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:19.555+0000 7fc200c93700 10 librados: watch_flush enter 2024-02-19T20:58:19.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:19.555+0000 7fc200c93700 10 librados: watch_flush exit 2024-02-19T20:58:19.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:19.555+0000 7fc200c93700 20 client.14211.objecter shutdown clearing up homeless session... 2024-02-19T20:58:19.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:19.555+0000 7fc200c93700 10 client.14211.objecter successfully canceled tick 2024-02-19T20:58:19.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:19.555+0000 7fc200c93700 1 -- 172.21.15.163:0/1063441556 >> v1:172.21.15.163:6800/2735092549 conn(0x7fc1e4035410 legacy=0x7fc1e40378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:58:19.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:19.555+0000 7fc200c93700 1 -- 172.21.15.163:0/1063441556 >> v1:172.21.15.163:6789/0 conn(0x7fc1fc06b780 legacy=0x7fc1fc1099a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:58:19.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:19.555+0000 7fc200c93700 1 -- 172.21.15.163:0/1063441556 shutdown_connections 2024-02-19T20:58:19.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:19.555+0000 7fc200c93700 1 -- 172.21.15.163:0/1063441556 >> 172.21.15.163:0/1063441556 conn(0x7fc1fc0f2fb0 msgr2=0x7fc1fc0f4c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:58:19.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:19.555+0000 7fc200c93700 1 -- 172.21.15.163:0/1063441556 shutdown_connections 2024-02-19T20:58:19.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:19.555+0000 7fc200c93700 1 -- 172.21.15.163:0/1063441556 wait complete. 2024-02-19T20:58:19.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:19.555+0000 7fc200c93700 1 librados: shutdown 2024-02-19T20:58:20.148 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:19.524487+0000 mon.a (mon.0) 231 : audit [INF] from='osd.0 v1:172.21.15.163:6801/1559418115' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-19T20:58:20.148 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:19.553931+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:20.148 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:19.585815+0000 mon.a (mon.0) 233 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:58:20.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:19.524487+0000 mon.a (mon.0) 231 : audit [INF] from='osd.0 v1:172.21.15.163:6801/1559418115' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-19T20:58:20.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:19.553931+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:20.149 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:19.585815+0000 mon.a (mon.0) 233 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:58:20.177 DEBUG:teuthology.orchestra.run.smithi163:osd.0> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.0.service 2024-02-19T20:58:20.220 INFO:tasks.cephadm:Deploying osd.1 on smithi163 with /dev/vg_nvme/lv_3... 2024-02-19T20:58:20.220 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-19T20:58:20.250 INFO:journalctl@ceph.osd.0.smithi163.stdout:-- Logs begin at Mon 2024-02-19 20:44:35 UTC. -- 2024-02-19T20:58:20.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:19.524487+0000 mon.a (mon.0) 231 : audit [INF] from='osd.0 v1:172.21.15.163:6801/1559418115' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-19T20:58:20.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:19.553931+0000 mon.a (mon.0) 232 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:20.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:19.585815+0000 mon.a (mon.0) 233 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:58:21.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:19.624282+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:21.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:19.944643+0000 mon.a (mon.0) 234 : audit [INF] from='osd.0 v1:172.21.15.163:6801/1559418115' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-19T20:58:21.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:19.944684+0000 mon.a (mon.0) 235 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-19T20:58:21.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:19.944798+0000 mon.a (mon.0) 236 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:21.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:19.945110+0000 mon.a (mon.0) 237 : audit [INF] from='osd.0 v1:172.21.15.163:6801/1559418115' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-02-19T20:58:21.231 INFO:journalctl@ceph.osd.0.smithi163.stdout:Feb 19 20:58:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-0[48619]: debug 2024-02-19T20:58:20.948+0000 7f63410bf700 -1 osd.0 0 waiting for initial osdmap 2024-02-19T20:58:21.232 INFO:journalctl@ceph.osd.0.smithi163.stdout:Feb 19 20:58:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-0[48619]: debug 2024-02-19T20:58:20.951+0000 7f633c038700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T20:58:21.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:19.624282+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:21.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:19.944643+0000 mon.a (mon.0) 234 : audit [INF] from='osd.0 v1:172.21.15.163:6801/1559418115' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-19T20:58:21.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:19.944684+0000 mon.a (mon.0) 235 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-19T20:58:21.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:19.944798+0000 mon.a (mon.0) 236 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:21.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:19.945110+0000 mon.a (mon.0) 237 : audit [INF] from='osd.0 v1:172.21.15.163:6801/1559418115' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-02-19T20:58:21.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:19.624282+0000 mgr.y (mgr.14146) 77 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:21.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:19.944643+0000 mon.a (mon.0) 234 : audit [INF] from='osd.0 v1:172.21.15.163:6801/1559418115' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-19T20:58:21.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:19.944684+0000 mon.a (mon.0) 235 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-02-19T20:58:21.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:19.944798+0000 mon.a (mon.0) 236 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:21.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:19.945110+0000 mon.a (mon.0) 237 : audit [INF] from='osd.0 v1:172.21.15.163:6801/1559418115' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-02-19T20:58:22.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:20.946733+0000 mon.a (mon.0) 238 : audit [INF] from='osd.0 v1:172.21.15.163:6801/1559418115' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-02-19T20:58:22.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:20.946790+0000 mon.a (mon.0) 239 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-19T20:58:22.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:20.947132+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:22.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:20.950043+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:22.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:21.949729+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:22.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:20.946733+0000 mon.a (mon.0) 238 : audit [INF] from='osd.0 v1:172.21.15.163:6801/1559418115' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-02-19T20:58:22.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:20.946790+0000 mon.a (mon.0) 239 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-19T20:58:22.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:20.947132+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:22.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:20.950043+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:22.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:21.949729+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:22.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:20.946733+0000 mon.a (mon.0) 238 : audit [INF] from='osd.0 v1:172.21.15.163:6801/1559418115' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-02-19T20:58:22.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:20.946790+0000 mon.a (mon.0) 239 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-02-19T20:58:22.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:20.947132+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:22.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:20.950043+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:22.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:21.949729+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:23.226 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:20.504743+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:58:23.226 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:20.504820+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:58:23.226 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:21.624516+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:23.226 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:21.959043+0000 mon.a (mon.0) 243 : cluster [INF] osd.0 v1:172.21.15.163:6801/1559418115 boot 2024-02-19T20:58:23.226 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:21.959068+0000 mon.a (mon.0) 244 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-19T20:58:23.227 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:21.959585+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:23.227 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:20.504743+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:58:23.227 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:20.504820+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:58:23.227 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:21.624516+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:23.227 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:21.959043+0000 mon.a (mon.0) 243 : cluster [INF] osd.0 v1:172.21.15.163:6801/1559418115 boot 2024-02-19T20:58:23.227 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:21.959068+0000 mon.a (mon.0) 244 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-19T20:58:23.227 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:21.959585+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:23.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:20.504743+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:58:23.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:20.504820+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:58:23.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:21.624516+0000 mgr.y (mgr.14146) 78 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T20:58:23.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:21.959043+0000 mon.a (mon.0) 243 : cluster [INF] osd.0 v1:172.21.15.163:6801/1559418115 boot 2024-02-19T20:58:23.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:21.959068+0000 mon.a (mon.0) 244 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-02-19T20:58:23.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:21.959585+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T20:58:23.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:22.960838+0000 mon.a (mon.0) 246 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-19T20:58:23.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:23.763326+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:23.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:23.764149+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:58:23.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:23.768122+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:23.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:23.768912+0000 mon.a (mon.0) 250 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:23.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:23.769503+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:23.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:23.773442+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-19T20:58:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-19T20:58:24.044 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-02-19T20:58:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-02-19T20:58:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0116319 s, 901 MB/s 2024-02-19T20:58:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-02-19T20:58:24.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:22.960838+0000 mon.a (mon.0) 246 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-19T20:58:24.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:23.763326+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:24.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:23.764149+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:58:24.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:23.768122+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:24.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:23.768912+0000 mon.a (mon.0) 250 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:24.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:23.769503+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:24.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:23.773442+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:24.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:22.960838+0000 mon.a (mon.0) 246 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-02-19T20:58:24.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:23.763326+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:24.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:23.764149+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:58:24.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:23.768122+0000 mon.a (mon.0) 249 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:24.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:23.768912+0000 mon.a (mon.0) 250 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:24.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:23.769503+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:24.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:23.773442+0000 mon.a (mon.0) 252 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:24.741 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi163:vg_nvme/lv_3 2024-02-19T20:58:24.981 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:23.624812+0000 mgr.y (mgr.14146) 79 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:24.981 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:58:23.759151+0000 mgr.y (mgr.14146) 80 : cephadm [INF] Detected new or changed devices on smithi163 2024-02-19T20:58:24.981 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:58:23.764636+0000 mgr.y (mgr.14146) 81 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 16098M 2024-02-19T20:58:24.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:23.624812+0000 mgr.y (mgr.14146) 79 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:24.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:58:23.759151+0000 mgr.y (mgr.14146) 80 : cephadm [INF] Detected new or changed devices on smithi163 2024-02-19T20:58:24.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:58:23.764636+0000 mgr.y (mgr.14146) 81 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 16098M 2024-02-19T20:58:25.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:23.624812+0000 mgr.y (mgr.14146) 79 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:25.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:58:23.759151+0000 mgr.y (mgr.14146) 80 : cephadm [INF] Detected new or changed devices on smithi163 2024-02-19T20:58:25.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:58:23.764636+0000 mgr.y (mgr.14146) 81 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 16098M 2024-02-19T20:58:26.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.578+0000 7fc322925700 1 -- 172.21.15.163:0/2060148152 >> v1:172.21.15.163:6789/0 conn(0x7fc31c0fbd00 legacy=0x7fc31c0fe0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:58:26.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.578+0000 7fc322925700 1 -- 172.21.15.163:0/2060148152 shutdown_connections 2024-02-19T20:58:26.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.578+0000 7fc322925700 1 -- 172.21.15.163:0/2060148152 >> 172.21.15.163:0/2060148152 conn(0x7fc31c0f2fb0 msgr2=0x7fc31c0f53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:58:26.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.578+0000 7fc322925700 1 -- 172.21.15.163:0/2060148152 shutdown_connections 2024-02-19T20:58:26.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.578+0000 7fc322925700 1 -- 172.21.15.163:0/2060148152 wait complete. 2024-02-19T20:58:26.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.578+0000 7fc322925700 1 librados: starting msgr at 2024-02-19T20:58:26.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.578+0000 7fc322925700 1 librados: starting objecter 2024-02-19T20:58:26.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.578+0000 7fc322925700 1 Processor -- start 2024-02-19T20:58:26.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.578+0000 7fc322925700 1 -- start start 2024-02-19T20:58:26.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.578+0000 7fc322925700 1 librados: setting wanted keys 2024-02-19T20:58:26.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.578+0000 7fc322925700 1 librados: calling monclient init 2024-02-19T20:58:26.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc322925700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc31c114af0 con 0x7fc31c0fbd00 2024-02-19T20:58:26.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc322925700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc31c115cd0 con 0x7fc31c0f7830 2024-02-19T20:58:26.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc322925700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc31c116ed0 con 0x7fc31c0ff930 2024-02-19T20:58:26.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc320ec2700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fc31c0ff930 0x7fc31c1131b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:50794/0 (socket says 172.21.15.163:50794) 2024-02-19T20:58:26.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc31b7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fc31c0fbd00 0x7fc31c10f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44244/0 (socket says 172.21.15.163:44244) 2024-02-19T20:58:26.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc320ec2700 1 -- 172.21.15.163:0/2295144951 learned_addr learned my addr 172.21.15.163:0/2295144951 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:58:26.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 10 client.?.objecter ms_handle_connect 0x7fc31c0ff930 2024-02-19T20:58:26.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 10 client.?.objecter resend_mon_ops 2024-02-19T20:58:26.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 10 client.?.objecter ms_handle_connect 0x7fc31c0fbd00 2024-02-19T20:58:26.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 10 client.?.objecter resend_mon_ops 2024-02-19T20:58:26.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340954437 0 0) 0x7fc31c116ed0 con 0x7fc31c0ff930 2024-02-19T20:58:26.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc308003110 con 0x7fc31c0ff930 2024-02-19T20:58:26.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320331588 0 0) 0x7fc31c114af0 con 0x7fc31c0fbd00 2024-02-19T20:58:26.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc31c116ed0 con 0x7fc31c0fbd00 2024-02-19T20:58:26.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 10 client.?.objecter ms_handle_connect 0x7fc31c0f7830 2024-02-19T20:58:26.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 10 client.?.objecter resend_mon_ops 2024-02-19T20:58:26.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235144880 0 0) 0x7fc31c115cd0 con 0x7fc31c0f7830 2024-02-19T20:58:26.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc31c114af0 con 0x7fc31c0f7830 2024-02-19T20:58:26.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1012114549 0 0) 0x7fc31c116ed0 con 0x7fc31c0fbd00 2024-02-19T20:58:26.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc31c115cd0 con 0x7fc31c0fbd00 2024-02-19T20:58:26.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 154609233 0 0) 0x7fc308003110 con 0x7fc31c0ff930 2024-02-19T20:58:26.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc31c116ed0 con 0x7fc31c0ff930 2024-02-19T20:58:26.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc30c002bb0 con 0x7fc31c0fbd00 2024-02-19T20:58:26.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.579+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc314002c10 con 0x7fc31c0ff930 2024-02-19T20:58:26.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.580+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3685102290 0 0) 0x7fc31c116ed0 con 0x7fc31c0ff930 2024-02-19T20:58:26.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.580+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 >> v1:172.21.15.193:6789/0 conn(0x7fc31c0f7830 legacy=0x7fc31c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:58:26.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.580+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 >> v1:172.21.15.163:6789/0 conn(0x7fc31c0fbd00 legacy=0x7fc31c10f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:58:26.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.580+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31c1180d0 con 0x7fc31c0ff930 2024-02-19T20:58:26.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.580+0000 7fc322925700 1 -- 172.21.15.163:0/2295144951 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc31c114ce0 con 0x7fc31c0ff930 2024-02-19T20:58:26.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.580+0000 7fc322925700 10 client.34148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:58:26.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.580+0000 7fc322925700 1 -- 172.21.15.163:0/2295144951 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc31c115150 con 0x7fc31c0ff930 2024-02-19T20:58:26.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.580+0000 7fc322925700 1 librados: init done 2024-02-19T20:58:26.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.580+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fc314002db0 con 0x7fc31c0ff930 2024-02-19T20:58:26.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.580+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc314005cf0 con 0x7fc31c0ff930 2024-02-19T20:58:26.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.581+0000 7fc322925700 1 -- 172.21.15.163:0/2295144951 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2fc004f20 con 0x7fc31c0ff930 2024-02-19T20:58:26.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.581+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 7720783 0 0) 0x7fc314018c00 con 0x7fc31c0ff930 2024-02-19T20:58:26.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.581+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1484+0+0 (unknown 2307110634 0 0) 0x7fc314077530 con 0x7fc31c0ff930 2024-02-19T20:58:26.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.581+0000 7fc3197fa700 10 client.34148.objecter ms_dispatch 0x7fc31c001550 osd_map(9..9 src has 1..9) v4 2024-02-19T20:58:26.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.581+0000 7fc3197fa700 3 client.34148.objecter handle_osd_map got epochs [9,9] > 0 2024-02-19T20:58:26.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.581+0000 7fc3197fa700 3 client.34148.objecter handle_osd_map decoding full epoch 9 2024-02-19T20:58:26.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.581+0000 7fc3197fa700 20 client.34148.objecter dump_active .. 0 homeless 2024-02-19T20:58:26.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.582+0000 7fc3197fa700 10 client.34148.objecter ms_handle_connect 0x7fc308063b10 2024-02-19T20:58:26.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.586+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc314048960 con 0x7fc31c0ff930 2024-02-19T20:58:26.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:26.757+0000 7fc322925700 1 -- 172.21.15.163:0/2295144951 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2fc000b90 con 0x7fc308063b10 2024-02-19T20:58:27.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:25.625195+0000 mgr.y (mgr.14146) 82 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:27.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:26.761657+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T20:58:27.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:26.763443+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T20:58:27.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:26.763889+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:27.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:25.625195+0000 mgr.y (mgr.14146) 82 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:27.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:26.761657+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T20:58:27.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:26.763443+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T20:58:27.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:26.763889+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:27.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:25.625195+0000 mgr.y (mgr.14146) 82 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:27.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:26.761657+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T20:58:27.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:26.763443+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T20:58:27.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:26.763889+0000 mon.a (mon.0) 255 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:28.111 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:26.759695+0000 mgr.y (mgr.14146) 83 : audit [DBG] from='client.34148 v1:172.21.15.163:0/2295144951' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:58:28.111 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:26.759695+0000 mgr.y (mgr.14146) 83 : audit [DBG] from='client.34148 v1:172.21.15.163:0/2295144951' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:58:28.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:26.759695+0000 mgr.y (mgr.14146) 83 : audit [DBG] from='client.34148 v1:172.21.15.163:0/2295144951' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:58:29.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:27.625535+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:29.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:27.625535+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:29.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:27.625535+0000 mgr.y (mgr.14146) 84 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:30.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:29.106789+0000 mon.a (mon.0) 256 : audit [INF] from='client.? v1:172.21.15.163:0/3138349908' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e74a65fa-cb9c-42c3-9293-602cf7e9dc67"}]: dispatch 2024-02-19T20:58:30.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:29.113145+0000 mon.a (mon.0) 257 : audit [INF] from='client.? v1:172.21.15.163:0/3138349908' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e74a65fa-cb9c-42c3-9293-602cf7e9dc67"}]': finished 2024-02-19T20:58:30.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:29.113184+0000 mon.a (mon.0) 258 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-19T20:58:30.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:29.113314+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:30.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:29.691241+0000 mon.a (mon.0) 260 : audit [DBG] from='client.? v1:172.21.15.163:0/1505695416' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T20:58:30.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:29.106789+0000 mon.a (mon.0) 256 : audit [INF] from='client.? v1:172.21.15.163:0/3138349908' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e74a65fa-cb9c-42c3-9293-602cf7e9dc67"}]: dispatch 2024-02-19T20:58:30.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:29.113145+0000 mon.a (mon.0) 257 : audit [INF] from='client.? v1:172.21.15.163:0/3138349908' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e74a65fa-cb9c-42c3-9293-602cf7e9dc67"}]': finished 2024-02-19T20:58:30.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:29.113184+0000 mon.a (mon.0) 258 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-19T20:58:30.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:29.113314+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:30.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:29.691241+0000 mon.a (mon.0) 260 : audit [DBG] from='client.? v1:172.21.15.163:0/1505695416' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T20:58:30.304 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:29.106789+0000 mon.a (mon.0) 256 : audit [INF] from='client.? v1:172.21.15.163:0/3138349908' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e74a65fa-cb9c-42c3-9293-602cf7e9dc67"}]: dispatch 2024-02-19T20:58:30.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:29.113145+0000 mon.a (mon.0) 257 : audit [INF] from='client.? v1:172.21.15.163:0/3138349908' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e74a65fa-cb9c-42c3-9293-602cf7e9dc67"}]': finished 2024-02-19T20:58:30.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:29.113184+0000 mon.a (mon.0) 258 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-02-19T20:58:30.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:29.113314+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:30.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:29.691241+0000 mon.a (mon.0) 260 : audit [DBG] from='client.? v1:172.21.15.163:0/1505695416' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T20:58:31.481 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:29.625769+0000 mgr.y (mgr.14146) 85 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:31.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:29.625769+0000 mgr.y (mgr.14146) 85 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:31.554 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:29.625769+0000 mgr.y (mgr.14146) 85 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:31.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:31.581+0000 7fc322124700 10 client.34148.objecter tick 2024-02-19T20:58:33.481 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:31.626142+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:33.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:31.626142+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:33.554 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:31.626142+0000 mgr.y (mgr.14146) 86 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:35.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:33.626470+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:35.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:33.626470+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:35.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:33.626470+0000 mgr.y (mgr.14146) 87 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:36.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:35.242514+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T20:58:36.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:35.243084+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:36.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:58:35.243539+0000 mgr.y (mgr.14146) 88 : cephadm [INF] Deploying daemon osd.1 on smithi163 2024-02-19T20:58:36.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:35.242514+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T20:58:36.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:35.243084+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:36.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:58:35.243539+0000 mgr.y (mgr.14146) 88 : cephadm [INF] Deploying daemon osd.1 on smithi163 2024-02-19T20:58:36.554 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:35.242514+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T20:58:36.554 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:35.243084+0000 mon.a (mon.0) 262 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:36.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:58:35.243539+0000 mgr.y (mgr.14146) 88 : cephadm [INF] Deploying daemon osd.1 on smithi163 2024-02-19T20:58:36.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:36.582+0000 7fc322124700 10 client.34148.objecter tick 2024-02-19T20:58:37.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:35.626671+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:37.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:35.626671+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:37.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:35.626671+0000 mgr.y (mgr.14146) 89 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:39.353 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:37.626978+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:39.353 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:37.626978+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:39.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:37.626978+0000 mgr.y (mgr.14146) 90 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:41.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:39.627411+0000 mgr.y (mgr.14146) 91 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:41.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:40.234885+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:41.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:40.275001+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:58:41.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:39.627411+0000 mgr.y (mgr.14146) 91 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:41.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:40.234885+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:41.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:40.275001+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:58:41.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:39.627411+0000 mgr.y (mgr.14146) 91 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:41.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:40.234885+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:41.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:40.275001+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:58:41.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:41.583+0000 7fc322124700 10 client.34148.objecter tick 2024-02-19T20:58:43.402 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:41.627695+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:43.403 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:42.776050+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:43.403 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:42.776918+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:43.403 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:42.777605+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:43.403 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:42.782095+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:43.403 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:41.627695+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:43.403 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:42.776050+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:43.403 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:42.776918+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:43.403 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:42.777605+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:43.403 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:42.782095+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:43.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:43.441+0000 7fc3197fa700 1 -- 172.21.15.163:0/2295144951 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 1459532904) 0x7fc2fc000b90 con 0x7fc308063b10 2024-02-19T20:58:43.425 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 1 on host 'smithi163' 2024-02-19T20:58:43.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:43.442+0000 7fc322925700 10 librados: watch_flush enter 2024-02-19T20:58:43.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:43.442+0000 7fc322925700 10 librados: watch_flush exit 2024-02-19T20:58:43.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:43.442+0000 7fc322925700 20 client.34148.objecter shutdown clearing up homeless session... 2024-02-19T20:58:43.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:43.442+0000 7fc322925700 10 client.34148.objecter successfully canceled tick 2024-02-19T20:58:43.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:43.443+0000 7fc322925700 1 -- 172.21.15.163:0/2295144951 >> v1:172.21.15.163:6800/2735092549 conn(0x7fc308063b10 legacy=0x7fc308065fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:58:43.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:43.443+0000 7fc322925700 1 -- 172.21.15.163:0/2295144951 >> v1:172.21.15.163:6790/0 conn(0x7fc31c0ff930 legacy=0x7fc31c1131b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:58:43.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:43.443+0000 7fc322925700 1 -- 172.21.15.163:0/2295144951 shutdown_connections 2024-02-19T20:58:43.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:43.443+0000 7fc322925700 1 -- 172.21.15.163:0/2295144951 >> 172.21.15.163:0/2295144951 conn(0x7fc31c0f2fb0 msgr2=0x7fc31c0ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:58:43.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:43.443+0000 7fc322925700 1 -- 172.21.15.163:0/2295144951 shutdown_connections 2024-02-19T20:58:43.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:43.443+0000 7fc322925700 1 -- 172.21.15.163:0/2295144951 wait complete. 2024-02-19T20:58:43.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:43.443+0000 7fc322925700 1 librados: shutdown 2024-02-19T20:58:43.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:41.627695+0000 mgr.y (mgr.14146) 92 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:43.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:42.776050+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:43.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:42.776918+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:43.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:42.777605+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:43.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:42.782095+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:44.050 DEBUG:teuthology.orchestra.run.smithi163:osd.1> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.1.service 2024-02-19T20:58:44.052 INFO:tasks.cephadm:Deploying osd.2 on smithi163 with /dev/vg_nvme/lv_2... 2024-02-19T20:58:44.052 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-19T20:58:44.081 INFO:journalctl@ceph.osd.1.smithi163.stdout:-- Logs begin at Mon 2024-02-19 20:44:35 UTC. -- 2024-02-19T20:58:44.311 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:43.322833+0000 mon.a (mon.0) 269 : audit [INF] from='osd.1 v1:172.21.15.163:6805/3125047432' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-19T20:58:44.311 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:43.441648+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:44.311 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:43.448830+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:58:44.312 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:43.322833+0000 mon.a (mon.0) 269 : audit [INF] from='osd.1 v1:172.21.15.163:6805/3125047432' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-19T20:58:44.312 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:43.441648+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:44.312 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:43.448830+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:58:44.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:43.322833+0000 mon.a (mon.0) 269 : audit [INF] from='osd.1 v1:172.21.15.163:6805/3125047432' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-19T20:58:44.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:43.441648+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:44.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:43.448830+0000 mon.a (mon.0) 271 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:58:45.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:43.628024+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:45.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:44.245340+0000 mon.a (mon.0) 272 : audit [INF] from='osd.1 v1:172.21.15.163:6805/3125047432' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-19T20:58:45.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:44.245403+0000 mon.a (mon.0) 273 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-19T20:58:45.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:44.245800+0000 mon.a (mon.0) 274 : audit [INF] from='osd.1 v1:172.21.15.163:6805/3125047432' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-02-19T20:58:45.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:44.256681+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:45.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:43.628024+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:45.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:44.245340+0000 mon.a (mon.0) 272 : audit [INF] from='osd.1 v1:172.21.15.163:6805/3125047432' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-19T20:58:45.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:44.245403+0000 mon.a (mon.0) 273 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-19T20:58:45.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:44.245800+0000 mon.a (mon.0) 274 : audit [INF] from='osd.1 v1:172.21.15.163:6805/3125047432' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-02-19T20:58:45.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:44.256681+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:45.483 INFO:journalctl@ceph.osd.1.smithi163.stdout:Feb 19 20:58:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-1[53263]: debug 2024-02-19T20:58:45.250+0000 7f78fa8ef700 -1 osd.1 0 waiting for initial osdmap 2024-02-19T20:58:45.483 INFO:journalctl@ceph.osd.1.smithi163.stdout:Feb 19 20:58:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-1[53263]: debug 2024-02-19T20:58:45.253+0000 7f78f4866700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T20:58:45.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:43.628024+0000 mgr.y (mgr.14146) 93 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:45.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:44.245340+0000 mon.a (mon.0) 272 : audit [INF] from='osd.1 v1:172.21.15.163:6805/3125047432' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-19T20:58:45.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:44.245403+0000 mon.a (mon.0) 273 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-02-19T20:58:45.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:44.245800+0000 mon.a (mon.0) 274 : audit [INF] from='osd.1 v1:172.21.15.163:6805/3125047432' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-02-19T20:58:45.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:44.256681+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:46.372 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-19T20:58:46.372 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-19T20:58:46.372 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-02-19T20:58:46.372 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-02-19T20:58:46.372 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013087 s, 801 MB/s 2024-02-19T20:58:46.373 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-02-19T20:58:46.409 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:45.246647+0000 mon.a (mon.0) 276 : audit [INF] from='osd.1 v1:172.21.15.163:6805/3125047432' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-02-19T20:58:46.409 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:45.246709+0000 mon.a (mon.0) 277 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-19T20:58:46.409 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:45.247039+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:46.409 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:45.250929+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:46.409 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:45.246647+0000 mon.a (mon.0) 276 : audit [INF] from='osd.1 v1:172.21.15.163:6805/3125047432' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-02-19T20:58:46.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:45.246709+0000 mon.a (mon.0) 277 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-19T20:58:46.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:45.247039+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:46.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:45.250929+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:46.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:45.246647+0000 mon.a (mon.0) 276 : audit [INF] from='osd.1 v1:172.21.15.163:6805/3125047432' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-02-19T20:58:46.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:45.246709+0000 mon.a (mon.0) 277 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-02-19T20:58:46.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:45.247039+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:46.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:45.250929+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:47.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:44.362074+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:58:47.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:44.362139+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:58:47.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:45.628450+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:47.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:46.252079+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:47.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:46.256560+0000 mon.a (mon.0) 281 : cluster [INF] osd.1 v1:172.21.15.163:6805/3125047432 boot 2024-02-19T20:58:47.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:46.256588+0000 mon.a (mon.0) 282 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-19T20:58:47.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:46.256868+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:47.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:44.362074+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:58:47.606 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:44.362139+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:58:47.606 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:45.628450+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:47.606 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:46.252079+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:47.606 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:46.256560+0000 mon.a (mon.0) 281 : cluster [INF] osd.1 v1:172.21.15.163:6805/3125047432 boot 2024-02-19T20:58:47.606 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:46.256588+0000 mon.a (mon.0) 282 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-19T20:58:47.606 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:46.256868+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:47.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:44.362074+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:58:47.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:44.362139+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:58:47.606 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:45.628450+0000 mgr.y (mgr.14146) 94 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T20:58:47.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:46.252079+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:47.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:46.256560+0000 mon.a (mon.0) 281 : cluster [INF] osd.1 v1:172.21.15.163:6805/3125047432 boot 2024-02-19T20:58:47.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:46.256588+0000 mon.a (mon.0) 282 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-02-19T20:58:47.607 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:46.256868+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T20:58:47.649 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi163:vg_nvme/lv_2 2024-02-19T20:58:48.471 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:47.261332+0000 mon.a (mon.0) 284 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-19T20:58:48.471 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:47.261332+0000 mon.a (mon.0) 284 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-19T20:58:48.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:47.261332+0000 mon.a (mon.0) 284 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-02-19T20:58:49.314 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:47.628790+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:49.314 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:48.509818+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:49.314 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:47.628790+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:49.314 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:48.509818+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:49.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:47.628790+0000 mgr.y (mgr.14146) 95 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:49.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:48.509818+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:49.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.796+0000 7f409c7c2700 1 -- 172.21.15.163:0/1557519363 >> v1:172.21.15.163:6789/0 conn(0x7f40940fac00 legacy=0x7f40940fcff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:58:49.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.796+0000 7f409c7c2700 1 -- 172.21.15.163:0/1557519363 shutdown_connections 2024-02-19T20:58:49.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.796+0000 7f409c7c2700 1 -- 172.21.15.163:0/1557519363 >> 172.21.15.163:0/1557519363 conn(0x7f40940f1eb0 msgr2=0x7f40940f42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:58:49.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.797+0000 7f409c7c2700 1 -- 172.21.15.163:0/1557519363 shutdown_connections 2024-02-19T20:58:49.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.797+0000 7f409c7c2700 1 -- 172.21.15.163:0/1557519363 wait complete. 2024-02-19T20:58:49.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.797+0000 7f409c7c2700 1 librados: starting msgr at 2024-02-19T20:58:49.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.797+0000 7f409c7c2700 1 librados: starting objecter 2024-02-19T20:58:49.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.797+0000 7f409c7c2700 1 Processor -- start 2024-02-19T20:58:49.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.797+0000 7f409c7c2700 1 -- start start 2024-02-19T20:58:49.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.797+0000 7f409c7c2700 1 librados: setting wanted keys 2024-02-19T20:58:49.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.797+0000 7f409c7c2700 1 librados: calling monclient init 2024-02-19T20:58:49.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.797+0000 7f409c7c2700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4094112ac0 con 0x7f40940fe830 2024-02-19T20:58:49.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.797+0000 7f409c7c2700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4094113c60 con 0x7f40940fac00 2024-02-19T20:58:49.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.797+0000 7f409c7c2700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4094114e00 con 0x7f40940f6730 2024-02-19T20:58:49.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.797+0000 7f409a55e700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f40940f6730 0x7f409410a540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:42694/0 (socket says 172.21.15.163:42694) 2024-02-19T20:58:49.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.797+0000 7f409a55e700 1 -- 172.21.15.163:0/3277368004 learned_addr learned my addr 172.21.15.163:0/3277368004 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:58:49.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 10 client.?.objecter ms_handle_connect 0x7f40940f6730 2024-02-19T20:58:49.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:58:49.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 10 client.?.objecter ms_handle_connect 0x7f40940fe830 2024-02-19T20:58:49.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:58:49.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464565728 0 0) 0x7f4094114e00 con 0x7f40940f6730 2024-02-19T20:58:49.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4080003030 con 0x7f40940f6730 2024-02-19T20:58:49.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 10 client.?.objecter ms_handle_connect 0x7f40940fac00 2024-02-19T20:58:49.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T20:58:49.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294080151 0 0) 0x7f4094112ac0 con 0x7f40940fe830 2024-02-19T20:58:49.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4094114e00 con 0x7f40940fe830 2024-02-19T20:58:49.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627803354 0 0) 0x7f4094113c60 con 0x7f40940fac00 2024-02-19T20:58:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4094112ac0 con 0x7f40940fac00 2024-02-19T20:58:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1585260394 0 0) 0x7f4094114e00 con 0x7f40940fe830 2024-02-19T20:58:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4094113c60 con 0x7f40940fe830 2024-02-19T20:58:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f408c001c40 con 0x7f40940fe830 2024-02-19T20:58:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2998033694 0 0) 0x7f4080003030 con 0x7f40940f6730 2024-02-19T20:58:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4094114e00 con 0x7f40940f6730 2024-02-19T20:58:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f40840018e0 con 0x7f40940f6730 2024-02-19T20:58:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2033687493 0 0) 0x7f4094113c60 con 0x7f40940fe830 2024-02-19T20:58:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 >> v1:172.21.15.163:6790/0 conn(0x7f40940f6730 legacy=0x7f409410a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:58:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 >> v1:172.21.15.193:6789/0 conn(0x7f40940fac00 legacy=0x7f409410dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:58:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4094115fa0 con 0x7f40940fe830 2024-02-19T20:58:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f409c7c2700 1 -- 172.21.15.163:0/3277368004 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4094112cb0 con 0x7f40940fe830 2024-02-19T20:58:49.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f409c7c2700 10 client.14265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:58:49.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f409c7c2700 1 -- 172.21.15.163:0/3277368004 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40941131a0 con 0x7f40940fe830 2024-02-19T20:58:49.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f409c7c2700 1 librados: init done 2024-02-19T20:58:49.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f408c002410 con 0x7f40940fe830 2024-02-19T20:58:49.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.798+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f408c003720 con 0x7f40940fe830 2024-02-19T20:58:49.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.799+0000 7f409c7c2700 1 -- 172.21.15.163:0/3277368004 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4078004f20 con 0x7f40940fe830 2024-02-19T20:58:49.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.800+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 7720783 0 0) 0x7f408c001f70 con 0x7f40940fe830 2024-02-19T20:58:49.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.800+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(14..14 src has 1..14) v4 ==== 1776+0+0 (unknown 2428299200 0 0) 0x7f408c074f40 con 0x7f40940fe830 2024-02-19T20:58:49.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.800+0000 7f408b7fe700 10 client.14265.objecter ms_dispatch 0x7f4094001550 osd_map(14..14 src has 1..14) v4 2024-02-19T20:58:49.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.800+0000 7f408b7fe700 3 client.14265.objecter handle_osd_map got epochs [14,14] > 0 2024-02-19T20:58:49.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.800+0000 7f408b7fe700 3 client.14265.objecter handle_osd_map decoding full epoch 14 2024-02-19T20:58:49.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.800+0000 7f408b7fe700 20 client.14265.objecter dump_active .. 0 homeless 2024-02-19T20:58:49.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.800+0000 7f408b7fe700 10 client.14265.objecter ms_handle_connect 0x7f4080063b60 2024-02-19T20:58:49.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.804+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f408c0460a0 con 0x7f40940fe830 2024-02-19T20:58:49.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:49.992+0000 7f409c7c2700 1 -- 172.21.15.163:0/3277368004 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f4078000b90 con 0x7f4080063b60 2024-02-19T20:58:50.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:58:48.504785+0000 mgr.y (mgr.14146) 96 : cephadm [INF] Detected new or changed devices on smithi163 2024-02-19T20:58:50.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:49.629070+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:50.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:49.923450+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:58:50.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:49.924032+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:58:50.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:58:49.924418+0000 mgr.y (mgr.14146) 98 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 8049M 2024-02-19T20:58:50.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:49.928470+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:50.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:49.929434+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:50.515 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:49.930020+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:50.516 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:49.934254+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:50.516 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:49.993669+0000 mgr.y (mgr.14146) 99 : audit [DBG] from='client.14265 v1:172.21.15.163:0/3277368004' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:58:50.516 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:49.995573+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T20:58:50.516 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:49.997411+0000 mon.a (mon.0) 293 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T20:58:50.516 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:49.997880+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:50.516 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:58:48.504785+0000 mgr.y (mgr.14146) 96 : cephadm [INF] Detected new or changed devices on smithi163 2024-02-19T20:58:50.516 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:49.629070+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:50.517 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:49.923450+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:58:50.517 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:49.924032+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:58:50.517 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:58:49.924418+0000 mgr.y (mgr.14146) 98 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 8049M 2024-02-19T20:58:50.517 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:49.928470+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:50.517 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:49.929434+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:50.517 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:49.930020+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:50.517 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:49.934254+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:50.517 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:49.993669+0000 mgr.y (mgr.14146) 99 : audit [DBG] from='client.14265 v1:172.21.15.163:0/3277368004' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:58:50.517 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:49.995573+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T20:58:50.517 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:49.997411+0000 mon.a (mon.0) 293 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T20:58:50.517 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:49.997880+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:50.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:58:48.504785+0000 mgr.y (mgr.14146) 96 : cephadm [INF] Detected new or changed devices on smithi163 2024-02-19T20:58:50.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:49.629070+0000 mgr.y (mgr.14146) 97 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:50.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:49.923450+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:58:50.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:49.924032+0000 mon.a (mon.0) 287 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:58:50.555 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:58:49.924418+0000 mgr.y (mgr.14146) 98 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 8049M 2024-02-19T20:58:50.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:49.928470+0000 mon.a (mon.0) 288 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:50.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:49.929434+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:50.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:49.930020+0000 mon.a (mon.0) 290 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:58:50.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:49.934254+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:58:50.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:49.993669+0000 mgr.y (mgr.14146) 99 : audit [DBG] from='client.14265 v1:172.21.15.163:0/3277368004' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:58:50.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:49.995573+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T20:58:50.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:49.997411+0000 mon.a (mon.0) 293 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T20:58:50.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:49.997880+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:52.982 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:51.629427+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:52.982 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:52.477665+0000 mon.a (mon.0) 295 : audit [INF] from='client.? v1:172.21.15.163:0/1666021388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6980a625-cdc9-487e-8f8c-7997ba716a97"}]: dispatch 2024-02-19T20:58:52.982 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:52.482882+0000 mon.a (mon.0) 296 : audit [INF] from='client.? v1:172.21.15.163:0/1666021388' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6980a625-cdc9-487e-8f8c-7997ba716a97"}]': finished 2024-02-19T20:58:52.982 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:52.482940+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-02-19T20:58:52.982 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:52.483060+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T20:58:52.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:51.629427+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:52.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:52.477665+0000 mon.a (mon.0) 295 : audit [INF] from='client.? v1:172.21.15.163:0/1666021388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6980a625-cdc9-487e-8f8c-7997ba716a97"}]: dispatch 2024-02-19T20:58:52.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:52.482882+0000 mon.a (mon.0) 296 : audit [INF] from='client.? v1:172.21.15.163:0/1666021388' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6980a625-cdc9-487e-8f8c-7997ba716a97"}]': finished 2024-02-19T20:58:52.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:52.482940+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-02-19T20:58:52.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:52.483060+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T20:58:53.055 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:51.629427+0000 mgr.y (mgr.14146) 100 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:53.055 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:52.477665+0000 mon.a (mon.0) 295 : audit [INF] from='client.? v1:172.21.15.163:0/1666021388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6980a625-cdc9-487e-8f8c-7997ba716a97"}]: dispatch 2024-02-19T20:58:53.055 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:52.482882+0000 mon.a (mon.0) 296 : audit [INF] from='client.? v1:172.21.15.163:0/1666021388' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6980a625-cdc9-487e-8f8c-7997ba716a97"}]': finished 2024-02-19T20:58:53.055 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:52.482940+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-02-19T20:58:53.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:52.483060+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T20:58:54.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:53.075028+0000 mon.a (mon.0) 299 : audit [DBG] from='client.? v1:172.21.15.163:0/2399932734' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T20:58:54.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:53.075028+0000 mon.a (mon.0) 299 : audit [DBG] from='client.? v1:172.21.15.163:0/2399932734' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T20:58:54.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:53.075028+0000 mon.a (mon.0) 299 : audit [DBG] from='client.? v1:172.21.15.163:0/2399932734' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T20:58:54.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:54.799+0000 7f409bfc1700 10 client.14265.objecter tick 2024-02-19T20:58:55.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:53.629797+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:55.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:53.629797+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:55.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:53.629797+0000 mgr.y (mgr.14146) 101 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:57.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:55.630230+0000 mgr.y (mgr.14146) 102 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:57.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:55.630230+0000 mgr.y (mgr.14146) 102 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:57.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:55.630230+0000 mgr.y (mgr.14146) 102 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 580 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:59.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:57.630642+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:59.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:58.615158+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T20:58:59.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:58:58.615722+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:59.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:57.630642+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:59.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:58.615158+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T20:58:59.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:58:58.615722+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:59.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:57.630642+0000 mgr.y (mgr.14146) 103 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:58:59.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:58.615158+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T20:58:59.306 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:58:58.615722+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:58:59.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:58:59.799+0000 7f409bfc1700 10 client.14265.objecter tick 2024-02-19T20:58:59.982 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:58:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:58:58.616187+0000 mgr.y (mgr.14146) 104 : cephadm [INF] Deploying daemon osd.2 on smithi163 2024-02-19T20:58:59.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:58:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:58:58.616187+0000 mgr.y (mgr.14146) 104 : cephadm [INF] Deploying daemon osd.2 on smithi163 2024-02-19T20:59:00.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:58:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:58:58.616187+0000 mgr.y (mgr.14146) 104 : cephadm [INF] Deploying daemon osd.2 on smithi163 2024-02-19T20:59:01.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:58:59.630971+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:59:01.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:58:59.630971+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:59:01.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:58:59.630971+0000 mgr.y (mgr.14146) 105 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:59:02.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:01.631238+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:59:02.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:01.631238+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:59:03.305 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:01.631238+0000 mgr.y (mgr.14146) 106 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:59:04.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:03.588808+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:04.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:03.627560+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:04.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:03.631642+0000 mgr.y (mgr.14146) 107 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:59:04.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:03.588808+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:04.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:03.627560+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:04.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:03.631642+0000 mgr.y (mgr.14146) 107 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:59:04.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:04.800+0000 7f409bfc1700 10 client.14265.objecter tick 2024-02-19T20:59:05.055 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:03.588808+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:05.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:03.627560+0000 mon.a (mon.0) 303 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:05.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:03.631642+0000 mgr.y (mgr.14146) 107 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:59:06.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:06.989+0000 7f408b7fe700 1 -- 172.21.15.163:0/3277368004 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 2998962100) 0x7f4078000b90 con 0x7f4080063b60 2024-02-19T20:59:06.974 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 2 on host 'smithi163' 2024-02-19T20:59:06.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:06.990+0000 7f409c7c2700 10 librados: watch_flush enter 2024-02-19T20:59:06.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:06.990+0000 7f409c7c2700 10 librados: watch_flush exit 2024-02-19T20:59:06.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:06.990+0000 7f409c7c2700 20 client.14265.objecter shutdown clearing up homeless session... 2024-02-19T20:59:06.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:06.990+0000 7f409c7c2700 10 client.14265.objecter successfully canceled tick 2024-02-19T20:59:06.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:06.991+0000 7f409c7c2700 1 -- 172.21.15.163:0/3277368004 >> v1:172.21.15.163:6800/2735092549 conn(0x7f4080063b60 legacy=0x7f4080066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:59:06.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:06.991+0000 7f409c7c2700 1 -- 172.21.15.163:0/3277368004 >> v1:172.21.15.163:6789/0 conn(0x7f40940fe830 legacy=0x7f40941113a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:59:06.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:06.991+0000 7f409c7c2700 1 -- 172.21.15.163:0/3277368004 shutdown_connections 2024-02-19T20:59:06.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:06.991+0000 7f409c7c2700 1 -- 172.21.15.163:0/3277368004 >> 172.21.15.163:0/3277368004 conn(0x7f40940f1eb0 msgr2=0x7f4094100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:59:06.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:06.991+0000 7f409c7c2700 1 -- 172.21.15.163:0/3277368004 shutdown_connections 2024-02-19T20:59:06.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:06.991+0000 7f409c7c2700 1 -- 172.21.15.163:0/3277368004 wait complete. 2024-02-19T20:59:06.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:06.991+0000 7f409c7c2700 1 librados: shutdown 2024-02-19T20:59:07.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:05.631883+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:59:07.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:06.161108+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:07.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:06.161836+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:07.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:06.162483+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:07.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:06.168855+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:07.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:06.696814+0000 mon.a (mon.0) 308 : audit [INF] from='osd.2 v1:172.21.15.163:6809/1563158487' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-19T20:59:07.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:06.985541+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:07.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:06.989050+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:07.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:05.631883+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:59:07.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:06.161108+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:07.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:06.161836+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:07.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:06.162483+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:07.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:06.168855+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:07.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:06.696814+0000 mon.a (mon.0) 308 : audit [INF] from='osd.2 v1:172.21.15.163:6809/1563158487' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-19T20:59:07.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:06.985541+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:07.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:06.989050+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:07.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:05.631883+0000 mgr.y (mgr.14146) 108 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:59:07.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:06.161108+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:07.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:06.161836+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:07.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:06.162483+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:07.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:06.168855+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:07.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:06.696814+0000 mon.a (mon.0) 308 : audit [INF] from='osd.2 v1:172.21.15.163:6809/1563158487' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-19T20:59:07.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:06.985541+0000 mon.a (mon.0) 309 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:07.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:06.989050+0000 mon.a (mon.0) 310 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:07.737 DEBUG:teuthology.orchestra.run.smithi163:osd.2> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.2.service 2024-02-19T20:59:07.739 INFO:tasks.cephadm:Deploying osd.3 on smithi163 with /dev/vg_nvme/lv_1... 2024-02-19T20:59:07.739 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-19T20:59:07.770 INFO:journalctl@ceph.osd.2.smithi163.stdout:-- Logs begin at Mon 2024-02-19 20:44:35 UTC. -- 2024-02-19T20:59:08.482 INFO:journalctl@ceph.osd.2.smithi163.stdout:Feb 19 20:59:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-2[57708]: debug 2024-02-19T20:59:08.177+0000 7f5b4f694700 -1 osd.2 0 waiting for initial osdmap 2024-02-19T20:59:08.483 INFO:journalctl@ceph.osd.2.smithi163.stdout:Feb 19 20:59:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-2[57708]: debug 2024-02-19T20:59:08.180+0000 7f5b4ae0e700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T20:59:08.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:07.173152+0000 mon.a (mon.0) 311 : audit [INF] from='osd.2 v1:172.21.15.163:6809/1563158487' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-19T20:59:08.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:07.173192+0000 mon.a (mon.0) 312 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-02-19T20:59:08.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:07.173331+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T20:59:08.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:07.173750+0000 mon.a (mon.0) 314 : audit [INF] from='osd.2 v1:172.21.15.163:6809/1563158487' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-02-19T20:59:08.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:07.173152+0000 mon.a (mon.0) 311 : audit [INF] from='osd.2 v1:172.21.15.163:6809/1563158487' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-19T20:59:08.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:07.173192+0000 mon.a (mon.0) 312 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-02-19T20:59:08.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:07.173331+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T20:59:08.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:07.173750+0000 mon.a (mon.0) 314 : audit [INF] from='osd.2 v1:172.21.15.163:6809/1563158487' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-02-19T20:59:08.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:07.173152+0000 mon.a (mon.0) 311 : audit [INF] from='osd.2 v1:172.21.15.163:6809/1563158487' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-19T20:59:08.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:07.173192+0000 mon.a (mon.0) 312 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-02-19T20:59:08.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:07.173331+0000 mon.a (mon.0) 313 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T20:59:08.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:07.173750+0000 mon.a (mon.0) 314 : audit [INF] from='osd.2 v1:172.21.15.163:6809/1563158487' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-02-19T20:59:09.700 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:07.632231+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:59:09.701 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:08.175804+0000 mon.a (mon.0) 315 : audit [INF] from='osd.2 v1:172.21.15.163:6809/1563158487' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-02-19T20:59:09.701 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:08.175852+0000 mon.a (mon.0) 316 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-19T20:59:09.701 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:08.176104+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T20:59:09.701 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:08.179169+0000 mon.a (mon.0) 318 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T20:59:09.701 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:07.632231+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:59:09.701 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:08.175804+0000 mon.a (mon.0) 315 : audit [INF] from='osd.2 v1:172.21.15.163:6809/1563158487' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-02-19T20:59:09.701 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:08.175852+0000 mon.a (mon.0) 316 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-19T20:59:09.701 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:08.176104+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T20:59:09.701 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:08.179169+0000 mon.a (mon.0) 318 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T20:59:09.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:07.632231+0000 mgr.y (mgr.14146) 109 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T20:59:09.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:08.175804+0000 mon.a (mon.0) 315 : audit [INF] from='osd.2 v1:172.21.15.163:6809/1563158487' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-02-19T20:59:09.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:08.175852+0000 mon.a (mon.0) 316 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-02-19T20:59:09.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:08.176104+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T20:59:09.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:08.179169+0000 mon.a (mon.0) 318 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T20:59:10.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:07.723568+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:59:10.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:07.723628+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:59:10.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:09.176662+0000 mon.a (mon.0) 319 : cluster [INF] osd.2 v1:172.21.15.163:6809/1563158487 boot 2024-02-19T20:59:10.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:09.176691+0000 mon.a (mon.0) 320 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-02-19T20:59:10.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:09.177108+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T20:59:10.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:09.177409+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-19T20:59:10.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:09.632511+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:10.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:10.179568+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-19T20:59:10.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:10.179637+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-02-19T20:59:10.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:10.180721+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-19T20:59:10.734 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:07.723568+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:59:10.734 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:07.723628+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:59:10.734 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:09.176662+0000 mon.a (mon.0) 319 : cluster [INF] osd.2 v1:172.21.15.163:6809/1563158487 boot 2024-02-19T20:59:10.734 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:09.176691+0000 mon.a (mon.0) 320 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-02-19T20:59:10.734 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:09.177108+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T20:59:10.734 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:09.177409+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-19T20:59:10.734 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:09.632511+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:10.734 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:10.179568+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-19T20:59:10.734 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:10.179637+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-02-19T20:59:10.734 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:10.180721+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-19T20:59:10.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:07.723568+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:59:10.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:07.723628+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:59:10.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:09.176662+0000 mon.a (mon.0) 319 : cluster [INF] osd.2 v1:172.21.15.163:6809/1563158487 boot 2024-02-19T20:59:10.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:09.176691+0000 mon.a (mon.0) 320 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-02-19T20:59:10.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:09.177108+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T20:59:10.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:09.177409+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-19T20:59:10.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:09.632511+0000 mgr.y (mgr.14146) 110 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:10.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:10.179568+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-19T20:59:10.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:10.179637+0000 mon.a (mon.0) 324 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-02-19T20:59:10.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:10.180721+0000 mon.a (mon.0) 325 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-19T20:59:10.906 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-19T20:59:10.906 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-19T20:59:10.906 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-02-19T20:59:10.906 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-02-19T20:59:10.906 INFO:teuthology.orchestra.run.smithi163.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113796 s, 921 MB/s 2024-02-19T20:59:10.907 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-02-19T20:59:11.992 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi163:vg_nvme/lv_1 2024-02-19T20:59:12.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:11.185135+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-19T20:59:12.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:11.185221+0000 mon.a (mon.0) 327 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-02-19T20:59:12.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:12.118612+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:12.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:12.119418+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:12.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:12.119936+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:12.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:12.120411+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:12.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:12.125799+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:12.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:12.126722+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:12.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:12.127265+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:12.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:12.131390+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:12.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:11.185135+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-19T20:59:12.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:11.185221+0000 mon.a (mon.0) 327 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-02-19T20:59:12.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:12.118612+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:12.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:12.119418+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:12.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:12.119936+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:12.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:12.120411+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:12.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:12.125799+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:12.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:12.126722+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:12.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:12.127265+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:12.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:12.131390+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:12.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:11.185135+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-19T20:59:12.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:11.185221+0000 mon.a (mon.0) 327 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-02-19T20:59:12.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:12.118612+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:12.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:12.119418+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:12.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:12.119936+0000 mon.a (mon.0) 330 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:12.557 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:12.120411+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:12.557 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:12.125799+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:12.557 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:12.126722+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:12.557 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:12.127265+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:12.557 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:12.131390+0000 mon.a (mon.0) 335 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:13.411 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:11.632815+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v91: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:13.412 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:59:12.113426+0000 mgr.y (mgr.14146) 112 : cephadm [INF] Detected new or changed devices on smithi163 2024-02-19T20:59:13.412 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:59:12.120784+0000 mgr.y (mgr.14146) 113 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 5366M 2024-02-19T20:59:13.412 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:12.185705+0000 mon.a (mon.0) 336 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-19T20:59:13.412 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:11.632815+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v91: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:13.412 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:59:12.113426+0000 mgr.y (mgr.14146) 112 : cephadm [INF] Detected new or changed devices on smithi163 2024-02-19T20:59:13.412 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:59:12.120784+0000 mgr.y (mgr.14146) 113 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 5366M 2024-02-19T20:59:13.412 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:12.185705+0000 mon.a (mon.0) 336 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-19T20:59:13.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:11.632815+0000 mgr.y (mgr.14146) 111 : cluster [DBG] pgmap v91: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:13.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:59:12.113426+0000 mgr.y (mgr.14146) 112 : cephadm [INF] Detected new or changed devices on smithi163 2024-02-19T20:59:13.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:59:12.120784+0000 mgr.y (mgr.14146) 113 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 5366M 2024-02-19T20:59:13.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:12.185705+0000 mon.a (mon.0) 336 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-02-19T20:59:13.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.910+0000 7f7c3978d700 1 -- 172.21.15.163:0/1673752484 >> v1:172.21.15.163:6790/0 conn(0x7f7c340ff920 legacy=0x7f7c34101de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:59:13.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.910+0000 7f7c3978d700 1 -- 172.21.15.163:0/1673752484 shutdown_connections 2024-02-19T20:59:13.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.910+0000 7f7c3978d700 1 -- 172.21.15.163:0/1673752484 >> 172.21.15.163:0/1673752484 conn(0x7f7c340f1e60 msgr2=0x7f7c340f42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:59:13.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.910+0000 7f7c3978d700 1 -- 172.21.15.163:0/1673752484 shutdown_connections 2024-02-19T20:59:13.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.910+0000 7f7c3978d700 1 -- 172.21.15.163:0/1673752484 wait complete. 2024-02-19T20:59:13.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.910+0000 7f7c3978d700 1 librados: starting msgr at 2024-02-19T20:59:13.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.910+0000 7f7c3978d700 1 librados: starting objecter 2024-02-19T20:59:13.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.910+0000 7f7c3978d700 1 Processor -- start 2024-02-19T20:59:13.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c3978d700 1 -- start start 2024-02-19T20:59:13.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c3978d700 1 librados: setting wanted keys 2024-02-19T20:59:13.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c3978d700 1 librados: calling monclient init 2024-02-19T20:59:13.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c3978d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7c34108320 con 0x7f7c340ff920 2024-02-19T20:59:13.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c3978d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7c34113b80 con 0x7f7c340611a0 2024-02-19T20:59:13.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c3978d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7c34114d60 con 0x7f7c340fbbc0 2024-02-19T20:59:13.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c327fc700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f7c340fbbc0 0x7f7c3410dd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:48612/0 (socket says 172.21.15.163:48612) 2024-02-19T20:59:13.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c327fc700 1 -- 172.21.15.163:0/792599038 learned_addr learned my addr 172.21.15.163:0/792599038 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T20:59:13.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c1bfff700 10 client.?.objecter ms_handle_connect 0x7f7c340fbbc0 2024-02-19T20:59:13.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c1bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T20:59:13.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c1bfff700 10 client.?.objecter ms_handle_connect 0x7f7c340ff920 2024-02-19T20:59:13.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c1bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T20:59:13.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c1bfff700 10 client.?.objecter ms_handle_connect 0x7f7c340611a0 2024-02-19T20:59:13.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c1bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T20:59:13.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749750506 0 0) 0x7f7c34114d60 con 0x7f7c340fbbc0 2024-02-19T20:59:13.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c1c003800 con 0x7f7c340fbbc0 2024-02-19T20:59:13.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094969933 0 0) 0x7f7c34108320 con 0x7f7c340ff920 2024-02-19T20:59:13.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c34114d60 con 0x7f7c340ff920 2024-02-19T20:59:13.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362191394 0 0) 0x7f7c34113b80 con 0x7f7c340611a0 2024-02-19T20:59:13.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c34108320 con 0x7f7c340611a0 2024-02-19T20:59:13.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4081175559 0 0) 0x7f7c34114d60 con 0x7f7c340ff920 2024-02-19T20:59:13.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c34113b80 con 0x7f7c340ff920 2024-02-19T20:59:13.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.911+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7c2c0032e0 con 0x7f7c340ff920 2024-02-19T20:59:13.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 374393116 0 0) 0x7f7c34108320 con 0x7f7c340611a0 2024-02-19T20:59:13.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c34114d60 con 0x7f7c340611a0 2024-02-19T20:59:13.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 453502831 0 0) 0x7f7c1c003800 con 0x7f7c340fbbc0 2024-02-19T20:59:13.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c34108320 con 0x7f7c340fbbc0 2024-02-19T20:59:13.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7c28001060 con 0x7f7c340611a0 2024-02-19T20:59:13.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7c24001900 con 0x7f7c340fbbc0 2024-02-19T20:59:13.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2623395061 0 0) 0x7f7c34113b80 con 0x7f7c340ff920 2024-02-19T20:59:13.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 >> v1:172.21.15.163:6790/0 conn(0x7f7c340fbbc0 legacy=0x7f7c3410dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:59:13.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 >> v1:172.21.15.193:6789/0 conn(0x7f7c340611a0 legacy=0x7f7c34107750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:59:13.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c34115f40 con 0x7f7c340ff920 2024-02-19T20:59:13.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c3978d700 1 -- 172.21.15.163:0/792599038 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c34112bd0 con 0x7f7c340ff920 2024-02-19T20:59:13.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c3978d700 10 client.14292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:59:13.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c3978d700 1 -- 172.21.15.163:0/792599038 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c341130c0 con 0x7f7c340ff920 2024-02-19T20:59:13.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c3978d700 1 librados: init done 2024-02-19T20:59:13.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f7c2c001ed0 con 0x7f7c340ff920 2024-02-19T20:59:13.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.912+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7c2c003bd0 con 0x7f7c340ff920 2024-02-19T20:59:13.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.913+0000 7f7c3978d700 1 -- 172.21.15.163:0/792599038 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c20004f20 con 0x7f7c340ff920 2024-02-19T20:59:13.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.913+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 7720783 0 0) 0x7f7c2c0023d0 con 0x7f7c340ff920 2024-02-19T20:59:13.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.914+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(21..21 src has 1..21) v4 ==== 2484+0+0 (unknown 1246178309 0 0) 0x7f7c2c075670 con 0x7f7c340ff920 2024-02-19T20:59:13.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.914+0000 7f7c1bfff700 10 client.14292.objecter ms_dispatch 0x7f7c34001550 osd_map(21..21 src has 1..21) v4 2024-02-19T20:59:13.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.914+0000 7f7c1bfff700 3 client.14292.objecter handle_osd_map got epochs [21,21] > 0 2024-02-19T20:59:13.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.914+0000 7f7c1bfff700 3 client.14292.objecter handle_osd_map decoding full epoch 21 2024-02-19T20:59:13.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.914+0000 7f7c1bfff700 20 client.14292.objecter dump_active .. 0 homeless 2024-02-19T20:59:13.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.914+0000 7f7c1bfff700 10 client.14292.objecter ms_handle_connect 0x7f7c1c063c60 2024-02-19T20:59:13.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:13.918+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f7c2c046630 con 0x7f7c340ff920 2024-02-19T20:59:14.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:14.089+0000 7f7c3978d700 1 -- 172.21.15.163:0/792599038 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c20000b90 con 0x7f7c1c063c60 2024-02-19T20:59:14.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:14.093460+0000 mon.a (mon.0) 337 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T20:59:14.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:14.095219+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T20:59:14.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:14.095659+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:14.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:14.093460+0000 mon.a (mon.0) 337 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T20:59:14.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:14.095219+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T20:59:14.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:14.095659+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:14.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:14.093460+0000 mon.a (mon.0) 337 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T20:59:14.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:14.095219+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T20:59:14.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:14.095659+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:15.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:13.633174+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:15.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:14.091545+0000 mgr.y (mgr.14146) 115 : audit [DBG] from='client.14292 v1:172.21.15.163:0/792599038' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:59:15.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:13.633174+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:15.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:14.091545+0000 mgr.y (mgr.14146) 115 : audit [DBG] from='client.14292 v1:172.21.15.163:0/792599038' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:59:15.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:13.633174+0000 mgr.y (mgr.14146) 114 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:15.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:14.091545+0000 mgr.y (mgr.14146) 115 : audit [DBG] from='client.14292 v1:172.21.15.163:0/792599038' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:59:17.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:15.633460+0000 mgr.y (mgr.14146) 116 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:17.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:16.593154+0000 mon.c (mon.2) 9 : audit [INF] from='client.? v1:172.21.15.163:0/2211692565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5"}]: dispatch 2024-02-19T20:59:17.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:16.593458+0000 mon.a (mon.0) 340 : audit [INF] from='client.34208 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5"}]: dispatch 2024-02-19T20:59:17.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:16.597417+0000 mon.a (mon.0) 341 : audit [INF] from='client.34208 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5"}]': finished 2024-02-19T20:59:17.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:16.597476+0000 mon.a (mon.0) 342 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-02-19T20:59:17.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:16.597580+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T20:59:17.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:17.189673+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? v1:172.21.15.163:0/836645855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T20:59:17.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:15.633460+0000 mgr.y (mgr.14146) 116 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:17.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:16.593154+0000 mon.c (mon.2) 9 : audit [INF] from='client.? v1:172.21.15.163:0/2211692565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5"}]: dispatch 2024-02-19T20:59:17.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:16.593458+0000 mon.a (mon.0) 340 : audit [INF] from='client.34208 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5"}]: dispatch 2024-02-19T20:59:17.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:16.597417+0000 mon.a (mon.0) 341 : audit [INF] from='client.34208 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5"}]': finished 2024-02-19T20:59:17.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:16.597476+0000 mon.a (mon.0) 342 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-02-19T20:59:17.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:16.597580+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T20:59:17.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:17.189673+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? v1:172.21.15.163:0/836645855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T20:59:17.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:15.633460+0000 mgr.y (mgr.14146) 116 : cluster [DBG] pgmap v94: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:17.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:16.593154+0000 mon.c (mon.2) 9 : audit [INF] from='client.? v1:172.21.15.163:0/2211692565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5"}]: dispatch 2024-02-19T20:59:17.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:16.593458+0000 mon.a (mon.0) 340 : audit [INF] from='client.34208 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5"}]: dispatch 2024-02-19T20:59:17.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:16.597417+0000 mon.a (mon.0) 341 : audit [INF] from='client.34208 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5"}]': finished 2024-02-19T20:59:17.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:16.597476+0000 mon.a (mon.0) 342 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2024-02-19T20:59:17.557 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:16.597580+0000 mon.a (mon.0) 343 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T20:59:17.557 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:17.189673+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? v1:172.21.15.163:0/836645855' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T20:59:18.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:18.913+0000 7f7c38f8c700 10 client.14292.objecter tick 2024-02-19T20:59:19.450 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:17.633844+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:19.450 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:17.633844+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:19.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:17.633844+0000 mgr.y (mgr.14146) 117 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:21.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:19.634127+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:21.451 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:19.634127+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:21.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:19.634127+0000 mgr.y (mgr.14146) 118 : cluster [DBG] pgmap v97: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:23.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:21.634394+0000 mgr.y (mgr.14146) 119 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:23.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:22.825211+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T20:59:23.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:22.825750+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:23.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:21.634394+0000 mgr.y (mgr.14146) 119 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:23.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:22.825211+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T20:59:23.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:22.825750+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:23.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:21.634394+0000 mgr.y (mgr.14146) 119 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:23.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:22.825211+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T20:59:23.557 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:22.825750+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:23.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:23.914+0000 7f7c38f8c700 10 client.14292.objecter tick 2024-02-19T20:59:24.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:59:22.826189+0000 mgr.y (mgr.14146) 120 : cephadm [INF] Deploying daemon osd.3 on smithi163 2024-02-19T20:59:24.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:59:22.826189+0000 mgr.y (mgr.14146) 120 : cephadm [INF] Deploying daemon osd.3 on smithi163 2024-02-19T20:59:24.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:59:22.826189+0000 mgr.y (mgr.14146) 120 : cephadm [INF] Deploying daemon osd.3 on smithi163 2024-02-19T20:59:25.402 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:23.634856+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:25.402 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:23.634856+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:25.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:23.634856+0000 mgr.y (mgr.14146) 121 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:27.471 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:25.635191+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:27.471 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:25.635191+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:27.556 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:25.635191+0000 mgr.y (mgr.14146) 122 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:28.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:28.915+0000 7f7c38f8c700 10 client.14292.objecter tick 2024-02-19T20:59:29.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:27.635584+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:29.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:27.792111+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:29.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:27.828846+0000 mon.a (mon.0) 347 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:29.169 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:27.635584+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:29.169 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:27.792111+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:29.169 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:27.828846+0000 mon.a (mon.0) 347 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:29.169 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:27.635584+0000 mgr.y (mgr.14146) 123 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:29.169 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:27.792111+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:29.169 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:27.828846+0000 mon.a (mon.0) 347 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:30.830 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:29.635970+0000 mgr.y (mgr.14146) 124 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:30.830 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:30.505781+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:30.830 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:30.506666+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:30.830 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:30.507304+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:30.830 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:30.511516+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:30.830 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:29.635970+0000 mgr.y (mgr.14146) 124 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:30.830 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:30.505781+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:30.830 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:30.506666+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:30.831 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:30.507304+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:30.831 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:30.511516+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:31.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:29.635970+0000 mgr.y (mgr.14146) 124 : cluster [DBG] pgmap v102: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:31.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:30.505781+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:31.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:30.506666+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:31.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:30.507304+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:31.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:30.511516+0000 mon.a (mon.0) 351 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:31.125 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 3 on host 'smithi163' 2024-02-19T20:59:31.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:31.139+0000 7f7c1bfff700 1 -- 172.21.15.163:0/792599038 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 4006963456) 0x7f7c20000b90 con 0x7f7c1c063c60 2024-02-19T20:59:31.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:31.141+0000 7f7c3978d700 10 librados: watch_flush enter 2024-02-19T20:59:31.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:31.142+0000 7f7c3978d700 10 librados: watch_flush exit 2024-02-19T20:59:31.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:31.142+0000 7f7c3978d700 20 client.14292.objecter shutdown clearing up homeless session... 2024-02-19T20:59:31.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:31.142+0000 7f7c3978d700 10 client.14292.objecter successfully canceled tick 2024-02-19T20:59:31.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:31.142+0000 7f7c3978d700 1 -- 172.21.15.163:0/792599038 >> v1:172.21.15.163:6800/2735092549 conn(0x7f7c1c063c60 legacy=0x7f7c1c066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:59:31.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:31.142+0000 7f7c3978d700 1 -- 172.21.15.163:0/792599038 >> v1:172.21.15.163:6789/0 conn(0x7f7c340ff920 legacy=0x7f7c34111450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:59:31.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:31.142+0000 7f7c3978d700 1 -- 172.21.15.163:0/792599038 shutdown_connections 2024-02-19T20:59:31.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:31.142+0000 7f7c3978d700 1 -- 172.21.15.163:0/792599038 >> 172.21.15.163:0/792599038 conn(0x7f7c340f1e60 msgr2=0x7f7c340f4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:59:31.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:31.142+0000 7f7c3978d700 1 -- 172.21.15.163:0/792599038 shutdown_connections 2024-02-19T20:59:31.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:31.142+0000 7f7c3978d700 1 -- 172.21.15.163:0/792599038 wait complete. 2024-02-19T20:59:31.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T20:59:31.142+0000 7f7c3978d700 1 librados: shutdown 2024-02-19T20:59:31.759 DEBUG:teuthology.orchestra.run.smithi163:osd.3> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.3.service 2024-02-19T20:59:31.761 INFO:tasks.cephadm:Deploying osd.4 on smithi193 with /dev/vg_nvme/lv_4... 2024-02-19T20:59:31.761 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-19T20:59:31.795 INFO:journalctl@ceph.osd.3.smithi163.stdout:-- Logs begin at Mon 2024-02-19 20:44:35 UTC. -- 2024-02-19T20:59:32.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:30.903102+0000 mon.c (mon.2) 11 : audit [INF] from='osd.3 v1:172.21.15.163:6813/2112279769' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T20:59:32.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:30.903379+0000 mon.a (mon.0) 352 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T20:59:32.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:31.139397+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:32.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:31.178597+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:32.070 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:30.903102+0000 mon.c (mon.2) 11 : audit [INF] from='osd.3 v1:172.21.15.163:6813/2112279769' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T20:59:32.070 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:30.903379+0000 mon.a (mon.0) 352 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T20:59:32.070 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:31.139397+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:32.070 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:31.178597+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:32.071 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:30.903102+0000 mon.c (mon.2) 11 : audit [INF] from='osd.3 v1:172.21.15.163:6813/2112279769' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T20:59:32.071 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:30.903379+0000 mon.a (mon.0) 352 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T20:59:32.071 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:31.139397+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:32.071 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:31.178597+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:32.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:31.636243+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:32.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:31.801460+0000 mon.a (mon.0) 355 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-19T20:59:32.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:31.801530+0000 mon.a (mon.0) 356 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-19T20:59:32.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:31.801685+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T20:59:32.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:31.802343+0000 mon.c (mon.2) 12 : audit [INF] from='osd.3 v1:172.21.15.163:6813/2112279769' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-02-19T20:59:32.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:31.802631+0000 mon.a (mon.0) 358 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-02-19T20:59:32.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:31.636243+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:32.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:31.801460+0000 mon.a (mon.0) 355 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-19T20:59:32.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:31.801530+0000 mon.a (mon.0) 356 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-19T20:59:32.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:31.801685+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T20:59:32.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:31.802343+0000 mon.c (mon.2) 12 : audit [INF] from='osd.3 v1:172.21.15.163:6813/2112279769' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-02-19T20:59:32.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:31.802631+0000 mon.a (mon.0) 358 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-02-19T20:59:32.981 INFO:journalctl@ceph.osd.3.smithi163.stdout:Feb 19 20:59:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-3[62371]: debug 2024-02-19T20:59:32.808+0000 7f396f23c700 -1 osd.3 0 waiting for initial osdmap 2024-02-19T20:59:32.981 INFO:journalctl@ceph.osd.3.smithi163.stdout:Feb 19 20:59:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-3[62371]: debug 2024-02-19T20:59:32.812+0000 7f396b9b8700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T20:59:33.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:31.636243+0000 mgr.y (mgr.14146) 125 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:33.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:31.801460+0000 mon.a (mon.0) 355 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-19T20:59:33.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:31.801530+0000 mon.a (mon.0) 356 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2024-02-19T20:59:33.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:31.801685+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T20:59:33.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:31.802343+0000 mon.c (mon.2) 12 : audit [INF] from='osd.3 v1:172.21.15.163:6813/2112279769' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-02-19T20:59:33.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:31.802631+0000 mon.a (mon.0) 358 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-02-19T20:59:33.948 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-19T20:59:33.948 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-19T20:59:33.948 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-02-19T20:59:33.948 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-02-19T20:59:33.948 INFO:teuthology.orchestra.run.smithi193.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110176 s, 952 MB/s 2024-02-19T20:59:33.949 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-02-19T20:59:34.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:32.802191+0000 mon.a (mon.0) 359 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-02-19T20:59:34.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:32.802263+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-19T20:59:34.056 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:32.802583+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T20:59:34.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:32.806526+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T20:59:34.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:32.802191+0000 mon.a (mon.0) 359 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-02-19T20:59:34.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:32.802263+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-19T20:59:34.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:32.802583+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T20:59:34.234 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:32.806526+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T20:59:34.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:32.802191+0000 mon.a (mon.0) 359 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-02-19T20:59:34.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:32.802263+0000 mon.a (mon.0) 360 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-02-19T20:59:34.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:32.802583+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T20:59:34.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:32.806526+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T20:59:34.631 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_4 2024-02-19T20:59:35.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:31.893916+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:59:35.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:31.893995+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:59:35.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:33.636556+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:35.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:33.807694+0000 mon.a (mon.0) 363 : cluster [INF] osd.3 v1:172.21.15.163:6813/2112279769 boot 2024-02-19T20:59:35.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:33.807737+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-02-19T20:59:35.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:33.808400+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T20:59:35.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:34.756099+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:35.057 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:34.756940+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:35.058 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:34.757462+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:35.058 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:34.757918+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:35.058 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:34.758614+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:35.058 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:34.762152+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:35.058 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:34.763616+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:35.058 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:34.764126+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:35.058 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:34.770272+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:35.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:31.893916+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:59:35.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:31.893995+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:59:35.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:33.636556+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:35.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:33.807694+0000 mon.a (mon.0) 363 : cluster [INF] osd.3 v1:172.21.15.163:6813/2112279769 boot 2024-02-19T20:59:35.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:33.807737+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-02-19T20:59:35.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:33.808400+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T20:59:35.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:34.756099+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:35.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:34.756940+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:35.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:34.757462+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:35.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:34.757918+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:35.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:34.758614+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:35.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:34.762152+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:35.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:34.763616+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:35.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:34.764126+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:35.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:34.770272+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:35.234 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:31.893916+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:59:35.234 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:31.893995+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:59:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:33.636556+0000 mgr.y (mgr.14146) 126 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T20:59:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:33.807694+0000 mon.a (mon.0) 363 : cluster [INF] osd.3 v1:172.21.15.163:6813/2112279769 boot 2024-02-19T20:59:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:33.807737+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2024-02-19T20:59:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:33.808400+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T20:59:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:34.756099+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:34.756940+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:34.757462+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:34.757918+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:34.758614+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:34.762152+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:34.763616+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:34.764126+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:34.770272+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:35.950 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:59:34.751775+0000 mgr.y (mgr.14146) 127 : cephadm [INF] Detected new or changed devices on smithi163 2024-02-19T20:59:35.950 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:59:34.758977+0000 mgr.y (mgr.14146) 128 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 4024M 2024-02-19T20:59:35.950 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:34.824094+0000 mon.a (mon.0) 375 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-19T20:59:36.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:59:34.751775+0000 mgr.y (mgr.14146) 127 : cephadm [INF] Detected new or changed devices on smithi163 2024-02-19T20:59:36.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:59:34.758977+0000 mgr.y (mgr.14146) 128 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 4024M 2024-02-19T20:59:36.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:34.824094+0000 mon.a (mon.0) 375 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-19T20:59:36.234 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:59:34.751775+0000 mgr.y (mgr.14146) 127 : cephadm [INF] Detected new or changed devices on smithi163 2024-02-19T20:59:36.234 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:59:34.758977+0000 mgr.y (mgr.14146) 128 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 4024M 2024-02-19T20:59:36.234 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:34.824094+0000 mon.a (mon.0) 375 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2024-02-19T20:59:36.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.488+0000 7f72c37a4700 1 -- 172.21.15.193:0/3774970875 >> v1:172.21.15.193:6789/0 conn(0x7f72bc0fe820 legacy=0x7f72bc100ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:59:36.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.489+0000 7f72c37a4700 1 -- 172.21.15.193:0/3774970875 shutdown_connections 2024-02-19T20:59:36.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.489+0000 7f72c37a4700 1 -- 172.21.15.193:0/3774970875 >> 172.21.15.193:0/3774970875 conn(0x7f72bc0f1e60 msgr2=0x7f72bc0f42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:59:36.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.489+0000 7f72c37a4700 1 -- 172.21.15.193:0/3774970875 shutdown_connections 2024-02-19T20:59:36.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.489+0000 7f72c37a4700 1 -- 172.21.15.193:0/3774970875 wait complete. 2024-02-19T20:59:36.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.489+0000 7f72c37a4700 1 librados: starting msgr at 2024-02-19T20:59:36.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.489+0000 7f72c37a4700 1 librados: starting objecter 2024-02-19T20:59:36.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.489+0000 7f72c37a4700 1 Processor -- start 2024-02-19T20:59:36.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.489+0000 7f72c37a4700 1 -- start start 2024-02-19T20:59:36.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.489+0000 7f72c37a4700 1 librados: setting wanted keys 2024-02-19T20:59:36.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.489+0000 7f72c37a4700 1 librados: calling monclient init 2024-02-19T20:59:36.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.489+0000 7f72c37a4700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f72bc1075c0 con 0x7f72bc0f6720 2024-02-19T20:59:36.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.489+0000 7f72c37a4700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f72bc112e20 con 0x7f72bc0fabf0 2024-02-19T20:59:36.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.489+0000 7f72c37a4700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f72bc114000 con 0x7f72bc0fe820 2024-02-19T20:59:36.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.489+0000 7f72c0d3f700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f72bc0fabf0 0x7f72bc10cfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:40030/0 (socket says 172.21.15.193:40030) 2024-02-19T20:59:36.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.489+0000 7f72c0d3f700 1 -- 172.21.15.193:0/1515963620 learned_addr learned my addr 172.21.15.193:0/1515963620 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T20:59:36.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 10 client.?.objecter ms_handle_connect 0x7f72bc0fabf0 2024-02-19T20:59:36.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T20:59:36.491 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 10 client.?.objecter ms_handle_connect 0x7f72bc0f6720 2024-02-19T20:59:36.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T20:59:36.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 10 client.?.objecter ms_handle_connect 0x7f72bc0fe820 2024-02-19T20:59:36.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T20:59:36.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379943655 0 0) 0x7f72bc112e20 con 0x7f72bc0fabf0 2024-02-19T20:59:36.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a8002220 con 0x7f72bc0fabf0 2024-02-19T20:59:36.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242589250 0 0) 0x7f72bc1075c0 con 0x7f72bc0f6720 2024-02-19T20:59:36.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72bc112e20 con 0x7f72bc0f6720 2024-02-19T20:59:36.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396109935 0 0) 0x7f72bc114000 con 0x7f72bc0fe820 2024-02-19T20:59:36.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72bc1075c0 con 0x7f72bc0fe820 2024-02-19T20:59:36.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2394172517 0 0) 0x7f72a8002220 con 0x7f72bc0fabf0 2024-02-19T20:59:36.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72bc114000 con 0x7f72bc0fabf0 2024-02-19T20:59:36.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f72ac001980 con 0x7f72bc0fabf0 2024-02-19T20:59:36.492 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 5185676 0 0) 0x7f72bc114000 con 0x7f72bc0fabf0 2024-02-19T20:59:36.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 >> v1:172.21.15.163:6790/0 conn(0x7f72bc0fe820 legacy=0x7f72bc1106f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:59:36.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 >> v1:172.21.15.163:6789/0 conn(0x7f72bc0f6720 legacy=0x7f72bc1069f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:59:36.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72bc1151e0 con 0x7f72bc0fabf0 2024-02-19T20:59:36.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72c37a4700 1 -- 172.21.15.193:0/1515963620 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72bc111e70 con 0x7f72bc0fabf0 2024-02-19T20:59:36.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72c37a4700 10 client.24232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T20:59:36.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72c37a4700 1 -- 172.21.15.193:0/1515963620 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72bc1122e0 con 0x7f72bc0fabf0 2024-02-19T20:59:36.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72c37a4700 1 librados: init done 2024-02-19T20:59:36.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f72ac001d10 con 0x7f72bc0fabf0 2024-02-19T20:59:36.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.490+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f72ac005bc0 con 0x7f72bc0fabf0 2024-02-19T20:59:36.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.491+0000 7f72c37a4700 1 -- 172.21.15.193:0/1515963620 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a0004f50 con 0x7f72bc0fabf0 2024-02-19T20:59:36.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.491+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 7720783 0 0) 0x7f72ac002950 con 0x7f72bc0fabf0 2024-02-19T20:59:36.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.492+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 2776+0+0 (unknown 2782070150 0 0) 0x7f72ac0777f0 con 0x7f72bc0fabf0 2024-02-19T20:59:36.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.492+0000 7f72b27fc700 10 client.24232.objecter ms_dispatch 0x7f72bc001550 osd_map(26..26 src has 1..26) v4 2024-02-19T20:59:36.493 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.492+0000 7f72b27fc700 3 client.24232.objecter handle_osd_map got epochs [26,26] > 0 2024-02-19T20:59:36.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.492+0000 7f72b27fc700 3 client.24232.objecter handle_osd_map decoding full epoch 26 2024-02-19T20:59:36.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.492+0000 7f72b27fc700 20 client.24232.objecter dump_active .. 0 homeless 2024-02-19T20:59:36.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.492+0000 7f72b27fc700 10 client.24232.objecter ms_handle_connect 0x7f72a80637e0 2024-02-19T20:59:36.494 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.496+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f72ac0485d0 con 0x7f72bc0fabf0 2024-02-19T20:59:36.652 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:36.666+0000 7f72c37a4700 1 -- 172.21.15.193:0/1515963620 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f72a0000bc0 con 0x7f72a80637e0 2024-02-19T20:59:37.141 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:35.636970+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:37.141 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:36.670986+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T20:59:37.141 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:36.673468+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T20:59:37.141 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:36.674102+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:37.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:35.636970+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:37.234 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:36.670986+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T20:59:37.234 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:36.673468+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T20:59:37.234 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:36.674102+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:37.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:35.636970+0000 mgr.y (mgr.14146) 129 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:37.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:36.670986+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T20:59:37.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:36.673468+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T20:59:37.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:36.674102+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:38.001 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:36.668415+0000 mgr.y (mgr.14146) 130 : audit [DBG] from='client.24232 v1:172.21.15.193:0/1515963620' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:59:38.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:36.668415+0000 mgr.y (mgr.14146) 130 : audit [DBG] from='client.24232 v1:172.21.15.193:0/1515963620' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:59:38.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:36.668415+0000 mgr.y (mgr.14146) 130 : audit [DBG] from='client.24232 v1:172.21.15.193:0/1515963620' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T20:59:39.233 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:37.637423+0000 mgr.y (mgr.14146) 131 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:39.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:37.637423+0000 mgr.y (mgr.14146) 131 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:39.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:37.637423+0000 mgr.y (mgr.14146) 131 : cluster [DBG] pgmap v110: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:40.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:39.043347+0000 mon.b (mon.1) 4 : audit [INF] from='client.? v1:172.21.15.193:0/2600563164' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0"}]: dispatch 2024-02-19T20:59:40.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:39.043905+0000 mon.a (mon.0) 379 : audit [INF] from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0"}]: dispatch 2024-02-19T20:59:40.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:39.048809+0000 mon.a (mon.0) 380 : audit [INF] from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0"}]': finished 2024-02-19T20:59:40.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:39.048876+0000 mon.a (mon.0) 381 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2024-02-19T20:59:40.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:39.049065+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:40.234 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:39.639947+0000 mon.b (mon.1) 5 : audit [DBG] from='client.? v1:172.21.15.193:0/3217654022' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T20:59:40.234 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:39.043347+0000 mon.b (mon.1) 4 : audit [INF] from='client.? v1:172.21.15.193:0/2600563164' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0"}]: dispatch 2024-02-19T20:59:40.234 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:39.043905+0000 mon.a (mon.0) 379 : audit [INF] from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0"}]: dispatch 2024-02-19T20:59:40.234 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:39.048809+0000 mon.a (mon.0) 380 : audit [INF] from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0"}]': finished 2024-02-19T20:59:40.234 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:39.048876+0000 mon.a (mon.0) 381 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2024-02-19T20:59:40.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:39.049065+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:40.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:39.639947+0000 mon.b (mon.1) 5 : audit [DBG] from='client.? v1:172.21.15.193:0/3217654022' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T20:59:40.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:39.043347+0000 mon.b (mon.1) 4 : audit [INF] from='client.? v1:172.21.15.193:0/2600563164' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0"}]: dispatch 2024-02-19T20:59:40.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:39.043905+0000 mon.a (mon.0) 379 : audit [INF] from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0"}]: dispatch 2024-02-19T20:59:40.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:39.048809+0000 mon.a (mon.0) 380 : audit [INF] from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0"}]': finished 2024-02-19T20:59:40.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:39.048876+0000 mon.a (mon.0) 381 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2024-02-19T20:59:40.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:39.049065+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:40.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:39.639947+0000 mon.b (mon.1) 5 : audit [DBG] from='client.? v1:172.21.15.193:0/3217654022' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T20:59:41.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:39.637746+0000 mgr.y (mgr.14146) 132 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:41.477 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:41.491+0000 7f72c2fa3700 10 client.24232.objecter tick 2024-02-19T20:59:41.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:39.637746+0000 mgr.y (mgr.14146) 132 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:41.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:39.637746+0000 mgr.y (mgr.14146) 132 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:43.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:41.638157+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:43.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:41.638157+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:43.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:41.638157+0000 mgr.y (mgr.14146) 133 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:45.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:43.638607+0000 mgr.y (mgr.14146) 134 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:45.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:43.638607+0000 mgr.y (mgr.14146) 134 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:45.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:43.638607+0000 mgr.y (mgr.14146) 134 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:46.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:45.075268+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T20:59:46.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:45.075994+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:46.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:59:45.076649+0000 mgr.y (mgr.14146) 135 : cephadm [INF] Deploying daemon osd.4 on smithi193 2024-02-19T20:59:46.478 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:46.492+0000 7f72c2fa3700 10 client.24232.objecter tick 2024-02-19T20:59:46.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:45.075268+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T20:59:46.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:45.075994+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:46.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:59:45.076649+0000 mgr.y (mgr.14146) 135 : cephadm [INF] Deploying daemon osd.4 on smithi193 2024-02-19T20:59:46.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:45.075268+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T20:59:46.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:45.075994+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:46.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:59:45.076649+0000 mgr.y (mgr.14146) 135 : cephadm [INF] Deploying daemon osd.4 on smithi193 2024-02-19T20:59:47.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:45.573938+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:59:47.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:45.574015+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:59:47.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:45.638962+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:47.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:46.063081+0000 mon.a (mon.0) 385 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-19T20:59:47.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:46.063364+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:47.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:45.573938+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:59:47.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:45.574015+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:59:47.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:45.638962+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:47.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:46.063081+0000 mon.a (mon.0) 385 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-19T20:59:47.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:46.063364+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:47.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:45.573938+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:59:47.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:45.574015+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:59:47.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:45.638962+0000 mgr.y (mgr.14146) 136 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:47.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:46.063081+0000 mon.a (mon.0) 385 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2024-02-19T20:59:47.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:46.063364+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:49.308 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:47.639335+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:49.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:47.639335+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:49.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:47.639335+0000 mgr.y (mgr.14146) 137 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:51.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:49.639738+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:51.308 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:50.081675+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:51.308 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:50.123856+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:51.479 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:51.493+0000 7f72c2fa3700 10 client.24232.objecter tick 2024-02-19T20:59:51.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:49.639738+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:51.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:50.081675+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:51.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:50.123856+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:51.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:49.639738+0000 mgr.y (mgr.14146) 138 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:51.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:50.081675+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:51.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:50.123856+0000 mon.a (mon.0) 388 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:53.425 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:51.640265+0000 mgr.y (mgr.14146) 139 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:53.425 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:52.403925+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:53.425 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:52.707913+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:53.425 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:52.708832+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:53.426 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:52.715517+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:53.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:53.461+0000 7f72b27fc700 1 -- 172.21.15.193:0/1515963620 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 2417249745) 0x7f72a0000bc0 con 0x7f72a80637e0 2024-02-19T20:59:53.448 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 4 on host 'smithi193' 2024-02-19T20:59:53.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:53.463+0000 7f72c37a4700 10 librados: watch_flush enter 2024-02-19T20:59:53.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:53.463+0000 7f72c37a4700 10 librados: watch_flush exit 2024-02-19T20:59:53.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:53.463+0000 7f72c37a4700 20 client.24232.objecter shutdown clearing up homeless session... 2024-02-19T20:59:53.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:53.463+0000 7f72c37a4700 10 client.24232.objecter successfully canceled tick 2024-02-19T20:59:53.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:53.463+0000 7f72c37a4700 1 -- 172.21.15.193:0/1515963620 >> v1:172.21.15.163:6800/2735092549 conn(0x7f72a80637e0 legacy=0x7f72a8065ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:59:53.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:53.463+0000 7f72c37a4700 1 -- 172.21.15.193:0/1515963620 >> v1:172.21.15.193:6789/0 conn(0x7f72bc0fabf0 legacy=0x7f72bc10cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T20:59:53.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:53.463+0000 7f72c37a4700 1 -- 172.21.15.193:0/1515963620 shutdown_connections 2024-02-19T20:59:53.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:53.463+0000 7f72c37a4700 1 -- 172.21.15.193:0/1515963620 >> 172.21.15.193:0/1515963620 conn(0x7f72bc0f1e60 msgr2=0x7f72bc0fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T20:59:53.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:53.464+0000 7f72c37a4700 1 -- 172.21.15.193:0/1515963620 shutdown_connections 2024-02-19T20:59:53.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:53.464+0000 7f72c37a4700 1 -- 172.21.15.193:0/1515963620 wait complete. 2024-02-19T20:59:53.451 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T20:59:53.464+0000 7f72c37a4700 1 librados: shutdown 2024-02-19T20:59:53.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:51.640265+0000 mgr.y (mgr.14146) 139 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:53.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:52.403925+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:53.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:52.707913+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:53.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:52.708832+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:53.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:52.715517+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:53.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:51.640265+0000 mgr.y (mgr.14146) 139 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:53.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:52.403925+0000 mon.a (mon.0) 389 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:53.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:52.707913+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:53.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:52.708832+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:53.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:52.715517+0000 mon.a (mon.0) 392 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:54.116 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:53.194097+0000 mon.b (mon.1) 6 : audit [INF] from='osd.4 v1:172.21.15.193:6800/3129716965' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T20:59:54.117 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:53.194493+0000 mon.a (mon.0) 393 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T20:59:54.117 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:53.462047+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:54.117 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:53.483055+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:54.144 DEBUG:teuthology.orchestra.run.smithi193:osd.4> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.4.service 2024-02-19T20:59:54.146 INFO:tasks.cephadm:Deploying osd.5 on smithi193 with /dev/vg_nvme/lv_3... 2024-02-19T20:59:54.146 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-19T20:59:54.177 INFO:journalctl@ceph.osd.4.smithi193.stdout:-- Logs begin at Mon 2024-02-19 20:44:17 UTC. -- 2024-02-19T20:59:54.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:53.194097+0000 mon.b (mon.1) 6 : audit [INF] from='osd.4 v1:172.21.15.193:6800/3129716965' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T20:59:54.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:53.194493+0000 mon.a (mon.0) 393 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T20:59:54.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:53.462047+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:54.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:53.483055+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:54.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:53.194097+0000 mon.b (mon.1) 6 : audit [INF] from='osd.4 v1:172.21.15.193:6800/3129716965' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T20:59:54.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:53.194493+0000 mon.a (mon.0) 393 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T20:59:54.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:53.462047+0000 mon.a (mon.0) 394 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:54.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:53.483055+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T20:59:55.395 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 20:59:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-4[43484]: debug 2024-02-19T20:59:55.097+0000 7f9151e45700 -1 osd.4 0 waiting for initial osdmap 2024-02-19T20:59:55.395 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 20:59:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-4[43484]: debug 2024-02-19T20:59:55.101+0000 7f914e5c1700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T20:59:55.395 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:53.640749+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:55.395 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:54.091312+0000 mon.a (mon.0) 396 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-19T20:59:55.395 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:54.091425+0000 mon.a (mon.0) 397 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-19T20:59:55.395 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:54.091573+0000 mon.a (mon.0) 398 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:55.395 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:54.091871+0000 mon.b (mon.1) 7 : audit [INF] from='osd.4 v1:172.21.15.193:6800/3129716965' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T20:59:55.395 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:54.092229+0000 mon.a (mon.0) 399 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T20:59:55.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:53.640749+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:55.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:54.091312+0000 mon.a (mon.0) 396 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-19T20:59:55.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:54.091425+0000 mon.a (mon.0) 397 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-19T20:59:55.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:54.091573+0000 mon.a (mon.0) 398 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:55.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:54.091871+0000 mon.b (mon.1) 7 : audit [INF] from='osd.4 v1:172.21.15.193:6800/3129716965' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T20:59:55.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:54.092229+0000 mon.a (mon.0) 399 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T20:59:55.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:53.640749+0000 mgr.y (mgr.14146) 140 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:55.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:54.091312+0000 mon.a (mon.0) 396 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-19T20:59:55.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:54.091425+0000 mon.a (mon.0) 397 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2024-02-19T20:59:55.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:54.091573+0000 mon.a (mon.0) 398 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:55.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:54.091871+0000 mon.b (mon.1) 7 : audit [INF] from='osd.4 v1:172.21.15.193:6800/3129716965' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T20:59:55.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:54.092229+0000 mon.a (mon.0) 399 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T20:59:56.280 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:54.696303+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:59:56.280 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:54.696435+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:59:56.280 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:55.093536+0000 mon.a (mon.0) 400 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-02-19T20:59:56.280 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:55.093686+0000 mon.a (mon.0) 401 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-19T20:59:56.280 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:55.094125+0000 mon.a (mon.0) 402 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:56.280 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:55.099088+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:56.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:54.696303+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:59:56.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:54.696435+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:59:56.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:55.093536+0000 mon.a (mon.0) 400 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-02-19T20:59:56.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:55.093686+0000 mon.a (mon.0) 401 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-19T20:59:56.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:55.094125+0000 mon.a (mon.0) 402 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:56.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:55.099088+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:56.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:54.696303+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:59:56.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:54.696435+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:59:56.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:55.093536+0000 mon.a (mon.0) 400 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-02-19T20:59:56.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:55.093686+0000 mon.a (mon.0) 401 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2024-02-19T20:59:56.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:55.094125+0000 mon.a (mon.0) 402 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:56.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:55.099088+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:57.247 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:54.171098+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:59:57.247 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:54.171177+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:59:57.247 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:55.641115+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:57.247 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:56.096414+0000 mon.a (mon.0) 404 : cluster [INF] osd.4 v1:172.21.15.193:6800/3129716965 boot 2024-02-19T20:59:57.248 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:56.096472+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-02-19T20:59:57.248 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:56.096643+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:57.248 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:57.098423+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-19T20:59:57.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:54.171098+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:59:57.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:54.171177+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:59:57.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:55.641115+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:57.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:56.096414+0000 mon.a (mon.0) 404 : cluster [INF] osd.4 v1:172.21.15.193:6800/3129716965 boot 2024-02-19T20:59:57.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:56.096472+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-02-19T20:59:57.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:56.096643+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:57.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:57.098423+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-19T20:59:57.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:54.171098+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T20:59:57.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:54.171177+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T20:59:57.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:55.641115+0000 mgr.y (mgr.14146) 141 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T20:59:57.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:56.096414+0000 mon.a (mon.0) 404 : cluster [INF] osd.4 v1:172.21.15.193:6800/3129716965 boot 2024-02-19T20:59:57.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:56.096472+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2024-02-19T20:59:57.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:56.096643+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T20:59:57.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:57.098423+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2024-02-19T20:59:58.024 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-19T20:59:58.025 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-19T20:59:58.025 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-02-19T20:59:58.025 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-02-19T20:59:58.025 INFO:teuthology.orchestra.run.smithi193.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110142 s, 952 MB/s 2024-02-19T20:59:58.025 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-02-19T20:59:58.357 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:59:57.168877+0000 mgr.y (mgr.14146) 142 : cephadm [INF] Detected new or changed devices on smithi193 2024-02-19T20:59:58.357 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:57.173598+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:58.357 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:57.174489+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:58.357 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T20:59:57.175026+0000 mgr.y (mgr.14146) 143 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 17122M 2024-02-19T20:59:58.357 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:57.179418+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:58.357 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:57.180280+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:58.357 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:57.180858+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:58.357 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T20:59:57.187467+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:58.358 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:58.100801+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-19T20:59:58.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:59:57.168877+0000 mgr.y (mgr.14146) 142 : cephadm [INF] Detected new or changed devices on smithi193 2024-02-19T20:59:58.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:57.173598+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:58.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:57.174489+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:58.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T20:59:57.175026+0000 mgr.y (mgr.14146) 143 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 17122M 2024-02-19T20:59:58.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:57.179418+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:58.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:57.180280+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:58.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:57.180858+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:58.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T20:59:57.187467+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:58.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:58.100801+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-19T20:59:58.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:59:57.168877+0000 mgr.y (mgr.14146) 142 : cephadm [INF] Detected new or changed devices on smithi193 2024-02-19T20:59:58.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:57.173598+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:58.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:57.174489+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T20:59:58.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T20:59:57.175026+0000 mgr.y (mgr.14146) 143 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 17122M 2024-02-19T20:59:58.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:57.179418+0000 mon.a (mon.0) 410 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:58.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:57.180280+0000 mon.a (mon.0) 411 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T20:59:58.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:57.180858+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T20:59:58.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T20:59:57.187467+0000 mon.a (mon.0) 413 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T20:59:58.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:58.100801+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2024-02-19T20:59:58.673 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_3 2024-02-19T20:59:59.343 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 20:59:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:57.641425+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v126: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T20:59:59.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 20:59:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:57.641425+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v126: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T20:59:59.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 20:59:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:57.641425+0000 mgr.y (mgr.14146) 144 : cluster [DBG] pgmap v126: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:00.358 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:00.000153+0000 mon.a (mon.0) 415 : cluster [INF] overall HEALTH_OK 2024-02-19T21:00:00.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:00.000153+0000 mon.a (mon.0) 415 : cluster [INF] overall HEALTH_OK 2024-02-19T21:00:00.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:00.000153+0000 mon.a (mon.0) 415 : cluster [INF] overall HEALTH_OK 2024-02-19T21:00:00.641 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.641+0000 7fefa2d35700 1 -- 172.21.15.193:0/3557388905 >> v1:172.21.15.193:6789/0 conn(0x7fef9c0fbd00 legacy=0x7fef9c0fe0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:00:00.641 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.641+0000 7fefa2d35700 1 -- 172.21.15.193:0/3557388905 shutdown_connections 2024-02-19T21:00:00.641 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.641+0000 7fefa2d35700 1 -- 172.21.15.193:0/3557388905 >> 172.21.15.193:0/3557388905 conn(0x7fef9c0f2fb0 msgr2=0x7fef9c0f53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:00:00.641 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.641+0000 7fefa2d35700 1 -- 172.21.15.193:0/3557388905 shutdown_connections 2024-02-19T21:00:00.641 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.641+0000 7fefa2d35700 1 -- 172.21.15.193:0/3557388905 wait complete. 2024-02-19T21:00:00.641 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fefa2d35700 1 librados: starting msgr at 2024-02-19T21:00:00.641 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fefa2d35700 1 librados: starting objecter 2024-02-19T21:00:00.641 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fefa2d35700 1 Processor -- start 2024-02-19T21:00:00.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fefa2d35700 1 -- start start 2024-02-19T21:00:00.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fefa2d35700 1 librados: setting wanted keys 2024-02-19T21:00:00.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fefa2d35700 1 librados: calling monclient init 2024-02-19T21:00:00.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fefa2d35700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fef9c1084c0 con 0x7fef9c0fbd00 2024-02-19T21:00:00.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fefa2d35700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fef9c113d20 con 0x7fef9c0ff930 2024-02-19T21:00:00.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fefa2d35700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fef9c114f00 con 0x7fef9c0f7830 2024-02-19T21:00:00.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fefa12d2700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fef9c0ff930 0x7fef9c1115f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:50174/0 (socket says 172.21.15.193:50174) 2024-02-19T21:00:00.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fefa12d2700 1 -- 172.21.15.193:0/981640739 learned_addr learned my addr 172.21.15.193:0/981640739 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T21:00:00.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fef99ffb700 10 client.?.objecter ms_handle_connect 0x7fef9c0ff930 2024-02-19T21:00:00.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fef99ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:00:00.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fef99ffb700 10 client.?.objecter ms_handle_connect 0x7fef9c0f7830 2024-02-19T21:00:00.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fef99ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:00:00.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fef99ffb700 10 client.?.objecter ms_handle_connect 0x7fef9c0fbd00 2024-02-19T21:00:00.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fef99ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:00:00.642 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337960978 0 0) 0x7fef9c113d20 con 0x7fef9c0ff930 2024-02-19T21:00:00.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef8c0033f0 con 0x7fef9c0ff930 2024-02-19T21:00:00.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.642+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318670801 0 0) 0x7fef9c114f00 con 0x7fef9c0f7830 2024-02-19T21:00:00.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef9c113d20 con 0x7fef9c0f7830 2024-02-19T21:00:00.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795696548 0 0) 0x7fef9c1084c0 con 0x7fef9c0fbd00 2024-02-19T21:00:00.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef9c114f00 con 0x7fef9c0fbd00 2024-02-19T21:00:00.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2648551571 0 0) 0x7fef8c0033f0 con 0x7fef9c0ff930 2024-02-19T21:00:00.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef9c1084c0 con 0x7fef9c0ff930 2024-02-19T21:00:00.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fef94002cb0 con 0x7fef9c0ff930 2024-02-19T21:00:00.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2430635156 0 0) 0x7fef9c113d20 con 0x7fef9c0f7830 2024-02-19T21:00:00.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef8c0033f0 con 0x7fef9c0f7830 2024-02-19T21:00:00.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fef900018c0 con 0x7fef9c0f7830 2024-02-19T21:00:00.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1435730826 0 0) 0x7fef9c114f00 con 0x7fef9c0fbd00 2024-02-19T21:00:00.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef9c113d20 con 0x7fef9c0fbd00 2024-02-19T21:00:00.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fef88002d60 con 0x7fef9c0fbd00 2024-02-19T21:00:00.643 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1563892844 0 0) 0x7fef9c1084c0 con 0x7fef9c0ff930 2024-02-19T21:00:00.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 >> v1:172.21.15.163:6790/0 conn(0x7fef9c0f7830 legacy=0x7fef9c1078f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:00:00.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 >> v1:172.21.15.163:6789/0 conn(0x7fef9c0fbd00 legacy=0x7fef9c10dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:00:00.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c1160e0 con 0x7fef9c0ff930 2024-02-19T21:00:00.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fefa2d35700 1 -- 172.21.15.193:0/981640739 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c1150f0 con 0x7fef9c0ff930 2024-02-19T21:00:00.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fef94001b90 con 0x7fef9c0ff930 2024-02-19T21:00:00.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fef94005e40 con 0x7fef9c0ff930 2024-02-19T21:00:00.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fefa2d35700 10 client.24259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:00:00.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fefa2d35700 1 -- 172.21.15.193:0/981640739 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c1155e0 con 0x7fef9c0ff930 2024-02-19T21:00:00.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.643+0000 7fefa2d35700 1 librados: init done 2024-02-19T21:00:00.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.644+0000 7fefa2d35700 1 -- 172.21.15.193:0/981640739 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c047540 con 0x7fef9c0ff930 2024-02-19T21:00:00.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.644+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 7720783 0 0) 0x7fef94018d70 con 0x7fef9c0ff930 2024-02-19T21:00:00.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.645+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(33..33 src has 1..33) v4 ==== 3178+0+0 (unknown 3521433550 0 0) 0x7fef94077df0 con 0x7fef9c0ff930 2024-02-19T21:00:00.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.645+0000 7fef99ffb700 10 client.24259.objecter ms_dispatch 0x7fef9c001550 osd_map(33..33 src has 1..33) v4 2024-02-19T21:00:00.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.645+0000 7fef99ffb700 3 client.24259.objecter handle_osd_map got epochs [33,33] > 0 2024-02-19T21:00:00.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.645+0000 7fef99ffb700 3 client.24259.objecter handle_osd_map decoding full epoch 33 2024-02-19T21:00:00.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.645+0000 7fef99ffb700 20 client.24259.objecter dump_active .. 0 homeless 2024-02-19T21:00:00.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.645+0000 7fef99ffb700 10 client.24259.objecter ms_handle_connect 0x7fef8c063be0 2024-02-19T21:00:00.645 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.649+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fef94048a10 con 0x7fef9c0ff930 2024-02-19T21:00:00.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:00.817+0000 7fefa2d35700 1 -- 172.21.15.193:0/981640739 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fef9c1159e0 con 0x7fef8c063be0 2024-02-19T21:00:01.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T20:59:59.641802+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v128: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:01.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:00.822414+0000 mon.a (mon.0) 416 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T21:00:01.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:00.824682+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T21:00:01.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:00.825346+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:01.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T20:59:59.641802+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v128: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:01.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:00.822414+0000 mon.a (mon.0) 416 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T21:00:01.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:00.824682+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T21:00:01.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:00.825346+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:01.486 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T20:59:59.641802+0000 mgr.y (mgr.14146) 145 : cluster [DBG] pgmap v128: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:01.486 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:00.822414+0000 mon.a (mon.0) 416 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T21:00:01.486 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:00.824682+0000 mon.a (mon.0) 417 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T21:00:01.486 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:00.825346+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:02.320 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:00.819961+0000 mgr.y (mgr.14146) 146 : audit [DBG] from='client.24259 v1:172.21.15.193:0/981640739' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:00:02.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:00.819961+0000 mgr.y (mgr.14146) 146 : audit [DBG] from='client.24259 v1:172.21.15.193:0/981640739' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:00:02.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:00.819961+0000 mgr.y (mgr.14146) 146 : audit [DBG] from='client.24259 v1:172.21.15.193:0/981640739' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:00:03.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:01.642297+0000 mgr.y (mgr.14146) 147 : cluster [DBG] pgmap v129: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:03.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:03.067646+0000 mon.b (mon.1) 8 : audit [INF] from='client.? v1:172.21.15.193:0/1310975581' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f"}]: dispatch 2024-02-19T21:00:03.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:03.068185+0000 mon.a (mon.0) 419 : audit [INF] from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f"}]: dispatch 2024-02-19T21:00:03.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:03.072993+0000 mon.a (mon.0) 420 : audit [INF] from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f"}]': finished 2024-02-19T21:00:03.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:03.073052+0000 mon.a (mon.0) 421 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-02-19T21:00:03.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:03.073202+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:00:03.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:01.642297+0000 mgr.y (mgr.14146) 147 : cluster [DBG] pgmap v129: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:03.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:03.067646+0000 mon.b (mon.1) 8 : audit [INF] from='client.? v1:172.21.15.193:0/1310975581' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f"}]: dispatch 2024-02-19T21:00:03.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:03.068185+0000 mon.a (mon.0) 419 : audit [INF] from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f"}]: dispatch 2024-02-19T21:00:03.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:03.072993+0000 mon.a (mon.0) 420 : audit [INF] from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f"}]': finished 2024-02-19T21:00:03.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:03.073052+0000 mon.a (mon.0) 421 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-02-19T21:00:03.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:03.073202+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:00:03.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:01.642297+0000 mgr.y (mgr.14146) 147 : cluster [DBG] pgmap v129: 1 pgs: 1 peering; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:03.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:03.067646+0000 mon.b (mon.1) 8 : audit [INF] from='client.? v1:172.21.15.193:0/1310975581' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f"}]: dispatch 2024-02-19T21:00:03.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:03.068185+0000 mon.a (mon.0) 419 : audit [INF] from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f"}]: dispatch 2024-02-19T21:00:03.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:03.072993+0000 mon.a (mon.0) 420 : audit [INF] from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f"}]': finished 2024-02-19T21:00:03.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:03.073052+0000 mon.a (mon.0) 421 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2024-02-19T21:00:03.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:03.073202+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:00:04.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:03.676227+0000 mon.b (mon.1) 9 : audit [DBG] from='client.? v1:172.21.15.193:0/1482101528' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T21:00:04.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:03.676227+0000 mon.b (mon.1) 9 : audit [DBG] from='client.? v1:172.21.15.193:0/1482101528' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T21:00:04.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:03.676227+0000 mon.b (mon.1) 9 : audit [DBG] from='client.? v1:172.21.15.193:0/1482101528' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T21:00:05.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:03.642699+0000 mgr.y (mgr.14146) 148 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:05.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:03.642699+0000 mgr.y (mgr.14146) 148 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:05.557 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:03.642699+0000 mgr.y (mgr.14146) 148 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:05.631 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:05.645+0000 7fefa2534700 10 client.24259.objecter tick 2024-02-19T21:00:07.408 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:05.643235+0000 mgr.y (mgr.14146) 149 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:07.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:05.643235+0000 mgr.y (mgr.14146) 149 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:07.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:05.643235+0000 mgr.y (mgr.14146) 149 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:09.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:07.643622+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:09.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:07.643622+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:09.557 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:07.643622+0000 mgr.y (mgr.14146) 150 : cluster [DBG] pgmap v133: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:10.466 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:09.383172+0000 mon.a (mon.0) 423 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T21:00:10.466 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:09.383915+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:10.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:09.383172+0000 mon.a (mon.0) 423 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T21:00:10.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:09.383915+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:10.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:09.383172+0000 mon.a (mon.0) 423 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T21:00:10.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:09.383915+0000 mon.a (mon.0) 424 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:10.632 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:10.646+0000 7fefa2534700 10 client.24259.objecter tick 2024-02-19T21:00:11.307 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:00:09.384535+0000 mgr.y (mgr.14146) 151 : cephadm [INF] Deploying daemon osd.5 on smithi193 2024-02-19T21:00:11.308 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:09.644118+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:11.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:00:09.384535+0000 mgr.y (mgr.14146) 151 : cephadm [INF] Deploying daemon osd.5 on smithi193 2024-02-19T21:00:11.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:09.644118+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:11.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:00:09.384535+0000 mgr.y (mgr.14146) 151 : cephadm [INF] Deploying daemon osd.5 on smithi193 2024-02-19T21:00:11.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:09.644118+0000 mgr.y (mgr.14146) 152 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:13.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:11.644534+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:13.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:11.644534+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:13.521 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:11.644534+0000 mgr.y (mgr.14146) 153 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:15.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:13.644865+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:15.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:14.396493+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:00:15.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:14.401751+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:15.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:13.644865+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:15.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:14.396493+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:00:15.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:14.401751+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:15.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:13.644865+0000 mgr.y (mgr.14146) 154 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:15.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:14.396493+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:00:15.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:14.401751+0000 mon.a (mon.0) 426 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:15.633 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:15.646+0000 7fefa2534700 10 client.24259.objecter tick 2024-02-19T21:00:17.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:15.645308+0000 mgr.y (mgr.14146) 155 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:17.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:17.038328+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:17.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:17.039114+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:17.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:17.039937+0000 mon.a (mon.0) 429 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:00:17.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:17.046834+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:17.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:15.645308+0000 mgr.y (mgr.14146) 155 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:17.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:17.038328+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:17.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:17.039114+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:17.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:17.039937+0000 mon.a (mon.0) 429 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:00:17.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:17.046834+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:17.519 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:15.645308+0000 mgr.y (mgr.14146) 155 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:17.519 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:17.038328+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:17.519 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:17.039114+0000 mon.a (mon.0) 428 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:17.519 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:17.039937+0000 mon.a (mon.0) 429 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:00:17.520 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:17.046834+0000 mon.a (mon.0) 430 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:17.800 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 5 on host 'smithi193' 2024-02-19T21:00:17.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:17.813+0000 7fef99ffb700 1 -- 172.21.15.193:0/981640739 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 3422659429) 0x7fef9c1159e0 con 0x7fef8c063be0 2024-02-19T21:00:17.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:17.815+0000 7fefa2d35700 10 librados: watch_flush enter 2024-02-19T21:00:17.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:17.815+0000 7fefa2d35700 10 librados: watch_flush exit 2024-02-19T21:00:17.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:17.815+0000 7fefa2d35700 20 client.24259.objecter shutdown clearing up homeless session... 2024-02-19T21:00:17.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:17.815+0000 7fefa2d35700 10 client.24259.objecter successfully canceled tick 2024-02-19T21:00:17.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:17.815+0000 7fefa2d35700 1 -- 172.21.15.193:0/981640739 >> v1:172.21.15.163:6800/2735092549 conn(0x7fef8c063be0 legacy=0x7fef8c0660a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:00:17.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:17.815+0000 7fefa2d35700 1 -- 172.21.15.193:0/981640739 >> v1:172.21.15.193:6789/0 conn(0x7fef9c0ff930 legacy=0x7fef9c1115f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:00:17.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:17.815+0000 7fefa2d35700 1 -- 172.21.15.193:0/981640739 shutdown_connections 2024-02-19T21:00:17.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:17.815+0000 7fefa2d35700 1 -- 172.21.15.193:0/981640739 >> 172.21.15.193:0/981640739 conn(0x7fef9c0f2fb0 msgr2=0x7fef9c0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:00:17.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:17.815+0000 7fefa2d35700 1 -- 172.21.15.193:0/981640739 shutdown_connections 2024-02-19T21:00:17.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:17.815+0000 7fefa2d35700 1 -- 172.21.15.193:0/981640739 wait complete. 2024-02-19T21:00:17.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:17.815+0000 7fefa2d35700 1 librados: shutdown 2024-02-19T21:00:18.436 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:17.536061+0000 mon.b (mon.1) 10 : audit [INF] from='osd.5 v1:172.21.15.193:6804/805161991' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T21:00:18.436 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:17.536494+0000 mon.a (mon.0) 431 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T21:00:18.437 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:17.813519+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:18.437 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:17.814603+0000 mon.a (mon.0) 433 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:00:18.458 DEBUG:teuthology.orchestra.run.smithi193:osd.5> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.5.service 2024-02-19T21:00:18.460 INFO:tasks.cephadm:Deploying osd.6 on smithi193 with /dev/vg_nvme/lv_2... 2024-02-19T21:00:18.461 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-19T21:00:18.491 INFO:journalctl@ceph.osd.5.smithi193.stdout:-- Logs begin at Mon 2024-02-19 20:44:17 UTC. -- 2024-02-19T21:00:18.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:17.536061+0000 mon.b (mon.1) 10 : audit [INF] from='osd.5 v1:172.21.15.193:6804/805161991' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T21:00:18.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:17.536494+0000 mon.a (mon.0) 431 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T21:00:18.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:17.813519+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:18.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:17.814603+0000 mon.a (mon.0) 433 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:00:18.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:17.536061+0000 mon.b (mon.1) 10 : audit [INF] from='osd.5 v1:172.21.15.193:6804/805161991' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T21:00:18.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:17.536494+0000 mon.a (mon.0) 431 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T21:00:18.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:17.813519+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:18.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:17.814603+0000 mon.a (mon.0) 433 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:00:19.558 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:00:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:00:19.260+0000 7faad7630700 -1 osd.5 0 waiting for initial osdmap 2024-02-19T21:00:19.559 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:00:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:00:19.264+0000 7faacfda4700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T21:00:19.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:17.645741+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:19.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:18.252091+0000 mon.a (mon.0) 434 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-19T21:00:19.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:18.252153+0000 mon.a (mon.0) 435 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-19T21:00:19.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:18.252295+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:00:19.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:18.254321+0000 mon.b (mon.1) 11 : audit [INF] from='osd.5 v1:172.21.15.193:6804/805161991' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:00:19.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:18.254822+0000 mon.a (mon.0) 437 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:00:19.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:17.645741+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:19.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:18.252091+0000 mon.a (mon.0) 434 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-19T21:00:19.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:18.252153+0000 mon.a (mon.0) 435 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-19T21:00:19.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:18.252295+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:00:19.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:18.254321+0000 mon.b (mon.1) 11 : audit [INF] from='osd.5 v1:172.21.15.193:6804/805161991' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:00:19.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:18.254822+0000 mon.a (mon.0) 437 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:00:19.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:17.645741+0000 mgr.y (mgr.14146) 156 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:19.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:18.252091+0000 mon.a (mon.0) 434 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-19T21:00:19.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:18.252153+0000 mon.a (mon.0) 435 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2024-02-19T21:00:19.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:18.252295+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:00:19.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:18.254321+0000 mon.b (mon.1) 11 : audit [INF] from='osd.5 v1:172.21.15.193:6804/805161991' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:00:19.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:18.254822+0000 mon.a (mon.0) 437 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:00:20.308 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:19.253364+0000 mon.a (mon.0) 438 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-02-19T21:00:20.308 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:19.253465+0000 mon.a (mon.0) 439 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-19T21:00:20.309 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:19.254027+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:00:20.309 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:19.261232+0000 mon.a (mon.0) 441 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:00:20.309 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:20.255082+0000 mon.a (mon.0) 442 : cluster [INF] osd.5 v1:172.21.15.193:6804/805161991 boot 2024-02-19T21:00:20.309 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:20.255138+0000 mon.a (mon.0) 443 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-02-19T21:00:20.309 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:20.255334+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:00:20.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:19.253364+0000 mon.a (mon.0) 438 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-02-19T21:00:20.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:19.253465+0000 mon.a (mon.0) 439 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-19T21:00:20.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:19.254027+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:00:20.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:19.261232+0000 mon.a (mon.0) 441 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:00:20.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:20.255082+0000 mon.a (mon.0) 442 : cluster [INF] osd.5 v1:172.21.15.193:6804/805161991 boot 2024-02-19T21:00:20.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:20.255138+0000 mon.a (mon.0) 443 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-02-19T21:00:20.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:20.255334+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:00:20.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:19.253364+0000 mon.a (mon.0) 438 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-02-19T21:00:20.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:19.253465+0000 mon.a (mon.0) 439 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2024-02-19T21:00:20.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:19.254027+0000 mon.a (mon.0) 440 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:00:20.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:19.261232+0000 mon.a (mon.0) 441 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:00:20.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:20.255082+0000 mon.a (mon.0) 442 : cluster [INF] osd.5 v1:172.21.15.193:6804/805161991 boot 2024-02-19T21:00:20.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:20.255138+0000 mon.a (mon.0) 443 : cluster [DBG] osdmap e37: 6 total, 6 up, 6 in 2024-02-19T21:00:20.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:20.255334+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:00:21.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:18.529568+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:21.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:18.529620+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:21.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:19.646114+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:21.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:21.256334+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-02-19T21:00:21.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:18.529568+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:21.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:18.529620+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:21.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:19.646114+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:21.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:21.256334+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-02-19T21:00:21.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:18.529568+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:21.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:18.529620+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:21.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:19.646114+0000 mgr.y (mgr.14146) 157 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 0 B data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T21:00:21.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:21.256334+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2024-02-19T21:00:22.028 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-19T21:00:22.028 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-19T21:00:22.028 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-02-19T21:00:22.028 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-02-19T21:00:22.028 INFO:teuthology.orchestra.run.smithi193.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111227 s, 943 MB/s 2024-02-19T21:00:22.028 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-02-19T21:00:22.701 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_2 2024-02-19T21:00:23.382 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:21.646480+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:23.382 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:22.257594+0000 mon.a (mon.0) 446 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-19T21:00:23.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:21.646480+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:23.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:22.257594+0000 mon.a (mon.0) 446 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-19T21:00:23.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:21.646480+0000 mgr.y (mgr.14146) 158 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:23.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:22.257594+0000 mon.a (mon.0) 446 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2024-02-19T21:00:24.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:22.538681+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:00:24.558 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:22.542047+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:00:24.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:00:23.423776+0000 mgr.y (mgr.14146) 159 : cephadm [INF] Detected new or changed devices on smithi193 2024-02-19T21:00:24.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:23.428686+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:24.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:23.429677+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:00:24.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:23.430439+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:00:24.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:00:23.430935+0000 mgr.y (mgr.14146) 160 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 8561M 2024-02-19T21:00:24.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:23.435305+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:24.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:23.436306+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:24.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:23.436996+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:00:24.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:23.441677+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:24.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:23.646789+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:24.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:22.538681+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:00:24.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:22.542047+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:00:24.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:00:23.423776+0000 mgr.y (mgr.14146) 159 : cephadm [INF] Detected new or changed devices on smithi193 2024-02-19T21:00:24.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:23.428686+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:24.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:23.429677+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:00:24.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:23.430439+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:00:24.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:00:23.430935+0000 mgr.y (mgr.14146) 160 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 8561M 2024-02-19T21:00:24.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:23.435305+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:24.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:23.436306+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:24.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:23.436996+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:00:24.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:23.441677+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:24.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:23.646789+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:24.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:22.538681+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:00:24.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:22.542047+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:00:24.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:00:23.423776+0000 mgr.y (mgr.14146) 159 : cephadm [INF] Detected new or changed devices on smithi193 2024-02-19T21:00:24.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:23.428686+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:24.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:23.429677+0000 mon.a (mon.0) 448 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:00:24.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:23.430439+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:00:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:00:23.430935+0000 mgr.y (mgr.14146) 160 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 8561M 2024-02-19T21:00:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:23.435305+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:23.436306+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:23.436996+0000 mon.a (mon.0) 452 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:00:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:23.441677+0000 mon.a (mon.0) 453 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:23.646789+0000 mgr.y (mgr.14146) 161 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:24.758 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.767+0000 7fcde898c700 1 -- 172.21.15.193:0/2190966494 >> v1:172.21.15.193:6789/0 conn(0x7fcde40f6720 legacy=0x7fcde40f6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:00:24.759 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.767+0000 7fcde898c700 1 -- 172.21.15.193:0/2190966494 shutdown_connections 2024-02-19T21:00:24.759 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.767+0000 7fcde898c700 1 -- 172.21.15.193:0/2190966494 >> 172.21.15.193:0/2190966494 conn(0x7fcde40f1e60 msgr2=0x7fcde40f42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:00:24.759 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.768+0000 7fcde898c700 1 -- 172.21.15.193:0/2190966494 shutdown_connections 2024-02-19T21:00:24.759 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.768+0000 7fcde898c700 1 -- 172.21.15.193:0/2190966494 wait complete. 2024-02-19T21:00:24.759 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.768+0000 7fcde898c700 1 librados: starting msgr at 2024-02-19T21:00:24.759 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.768+0000 7fcde898c700 1 librados: starting objecter 2024-02-19T21:00:24.759 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.768+0000 7fcde898c700 1 Processor -- start 2024-02-19T21:00:24.759 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.768+0000 7fcde898c700 1 -- start start 2024-02-19T21:00:24.759 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.768+0000 7fcde898c700 1 librados: setting wanted keys 2024-02-19T21:00:24.759 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.768+0000 7fcde898c700 1 librados: calling monclient init 2024-02-19T21:00:24.759 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.768+0000 7fcde898c700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcde4112ed0 con 0x7fcde40fabf0 2024-02-19T21:00:24.759 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.768+0000 7fcde898c700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcde4114070 con 0x7fcde40fe820 2024-02-19T21:00:24.760 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.768+0000 7fcde898c700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcde4115210 con 0x7fcde40f6720 2024-02-19T21:00:24.760 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcde2d9d700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fcde40fe820 0x7fcde41117b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:46012/0 (socket says 172.21.15.193:46012) 2024-02-19T21:00:24.760 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcde2d9d700 1 -- 172.21.15.193:0/656558327 learned_addr learned my addr 172.21.15.193:0/656558327 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T21:00:24.760 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 10 client.?.objecter ms_handle_connect 0x7fcde40fe820 2024-02-19T21:00:24.760 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:00:24.760 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 10 client.?.objecter ms_handle_connect 0x7fcde40fabf0 2024-02-19T21:00:24.760 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:00:24.760 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 10 client.?.objecter ms_handle_connect 0x7fcde40f6720 2024-02-19T21:00:24.760 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:00:24.760 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009459618 0 0) 0x7fcde4114070 con 0x7fcde40fe820 2024-02-19T21:00:24.761 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdcc002220 con 0x7fcde40fe820 2024-02-19T21:00:24.761 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798214363 0 0) 0x7fcde4112ed0 con 0x7fcde40fabf0 2024-02-19T21:00:24.761 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde4114070 con 0x7fcde40fabf0 2024-02-19T21:00:24.761 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943434349 0 0) 0x7fcde4115210 con 0x7fcde40f6720 2024-02-19T21:00:24.761 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde4112ed0 con 0x7fcde40f6720 2024-02-19T21:00:24.761 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 443373909 0 0) 0x7fcdcc002220 con 0x7fcde40fe820 2024-02-19T21:00:24.761 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcde4115210 con 0x7fcde40fe820 2024-02-19T21:00:24.761 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcddc001c80 con 0x7fcde40fe820 2024-02-19T21:00:24.761 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3748007409 0 0) 0x7fcde4114070 con 0x7fcde40fabf0 2024-02-19T21:00:24.761 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdcc002220 con 0x7fcde40fabf0 2024-02-19T21:00:24.761 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcdd8001880 con 0x7fcde40fabf0 2024-02-19T21:00:24.761 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2666038427 0 0) 0x7fcde4112ed0 con 0x7fcde40f6720 2024-02-19T21:00:24.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcde4114070 con 0x7fcde40f6720 2024-02-19T21:00:24.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcdd40018c0 con 0x7fcde40f6720 2024-02-19T21:00:24.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2909004283 0 0) 0x7fcde4115210 con 0x7fcde40fe820 2024-02-19T21:00:24.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 >> v1:172.21.15.163:6790/0 conn(0x7fcde40f6720 legacy=0x7fcde410a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:00:24.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 >> v1:172.21.15.163:6789/0 conn(0x7fcde40fabf0 legacy=0x7fcde410e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:00:24.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde41163b0 con 0x7fcde40fe820 2024-02-19T21:00:24.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcde898c700 1 -- 172.21.15.193:0/656558327 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde41130c0 con 0x7fcde40fe820 2024-02-19T21:00:24.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcde898c700 10 client.24286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:00:24.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcde898c700 1 -- 172.21.15.193:0/656558327 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde41135b0 con 0x7fcde40fe820 2024-02-19T21:00:24.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.769+0000 7fcde898c700 1 librados: init done 2024-02-19T21:00:24.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.770+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fcddc002010 con 0x7fcde40fe820 2024-02-19T21:00:24.762 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.770+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcddc005f40 con 0x7fcde40fe820 2024-02-19T21:00:24.763 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.770+0000 7fcde898c700 1 -- 172.21.15.193:0/656558327 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc4004f20 con 0x7fcde40fe820 2024-02-19T21:00:24.763 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.771+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 7720783 0 0) 0x7fcddc018d90 con 0x7fcde40fe820 2024-02-19T21:00:24.763 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.771+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(39..39 src has 1..39) v4 ==== 3470+0+0 (unknown 560659782 0 0) 0x7fcddc077d20 con 0x7fcde40fe820 2024-02-19T21:00:24.763 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.771+0000 7fcdd37fe700 10 client.24286.objecter ms_dispatch 0x7fcde4001550 osd_map(39..39 src has 1..39) v4 2024-02-19T21:00:24.763 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.771+0000 7fcdd37fe700 3 client.24286.objecter handle_osd_map got epochs [39,39] > 0 2024-02-19T21:00:24.763 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.771+0000 7fcdd37fe700 3 client.24286.objecter handle_osd_map decoding full epoch 39 2024-02-19T21:00:24.763 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.771+0000 7fcdd37fe700 20 client.24286.objecter dump_active .. 0 homeless 2024-02-19T21:00:24.764 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.772+0000 7fcdd37fe700 10 client.24286.objecter ms_handle_connect 0x7fcdcc063b40 2024-02-19T21:00:24.764 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.775+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fcddc0a2f60 con 0x7fcde40fe820 2024-02-19T21:00:24.930 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:24.943+0000 7fcde898c700 1 -- 172.21.15.193:0/656558327 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdc4000b90 con 0x7fcdcc063b40 2024-02-19T21:00:25.644 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:24.945317+0000 mgr.y (mgr.14146) 162 : audit [DBG] from='client.24286 v1:172.21.15.193:0/656558327' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:00:25.644 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:24.947529+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T21:00:25.644 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:24.949866+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T21:00:25.644 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:24.950535+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:25.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:24.945317+0000 mgr.y (mgr.14146) 162 : audit [DBG] from='client.24286 v1:172.21.15.193:0/656558327' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:00:25.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:24.947529+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T21:00:25.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:24.949866+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T21:00:25.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:24.950535+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:25.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:24.945317+0000 mgr.y (mgr.14146) 162 : audit [DBG] from='client.24286 v1:172.21.15.193:0/656558327' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:00:25.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:24.947529+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T21:00:25.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:24.949866+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T21:00:25.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:24.950535+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:26.627 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:25.647130+0000 mgr.y (mgr.14146) 163 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:26.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:25.647130+0000 mgr.y (mgr.14146) 163 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:26.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:25.647130+0000 mgr.y (mgr.14146) 163 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:27.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:27.411970+0000 mon.b (mon.1) 12 : audit [INF] from='client.? v1:172.21.15.193:0/1125595271' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "302c95df-3842-4818-8afd-d79d8184684f"}]: dispatch 2024-02-19T21:00:27.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:27.412689+0000 mon.a (mon.0) 457 : audit [INF] from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "302c95df-3842-4818-8afd-d79d8184684f"}]: dispatch 2024-02-19T21:00:27.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:27.417353+0000 mon.a (mon.0) 458 : audit [INF] from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "302c95df-3842-4818-8afd-d79d8184684f"}]': finished 2024-02-19T21:00:27.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:27.417495+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2024-02-19T21:00:27.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:27.417627+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:27.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:27.411970+0000 mon.b (mon.1) 12 : audit [INF] from='client.? v1:172.21.15.193:0/1125595271' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "302c95df-3842-4818-8afd-d79d8184684f"}]: dispatch 2024-02-19T21:00:27.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:27.412689+0000 mon.a (mon.0) 457 : audit [INF] from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "302c95df-3842-4818-8afd-d79d8184684f"}]: dispatch 2024-02-19T21:00:27.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:27.417353+0000 mon.a (mon.0) 458 : audit [INF] from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "302c95df-3842-4818-8afd-d79d8184684f"}]': finished 2024-02-19T21:00:27.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:27.417495+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2024-02-19T21:00:27.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:27.417627+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:27.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:27.411970+0000 mon.b (mon.1) 12 : audit [INF] from='client.? v1:172.21.15.193:0/1125595271' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "302c95df-3842-4818-8afd-d79d8184684f"}]: dispatch 2024-02-19T21:00:27.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:27.412689+0000 mon.a (mon.0) 457 : audit [INF] from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "302c95df-3842-4818-8afd-d79d8184684f"}]: dispatch 2024-02-19T21:00:27.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:27.417353+0000 mon.a (mon.0) 458 : audit [INF] from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "302c95df-3842-4818-8afd-d79d8184684f"}]': finished 2024-02-19T21:00:27.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:27.417495+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e40: 7 total, 6 up, 7 in 2024-02-19T21:00:27.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:27.417627+0000 mon.a (mon.0) 460 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:28.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:27.590384+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:28.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:27.590474+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:28.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:27.647481+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:28.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:28.041641+0000 mon.b (mon.1) 13 : audit [DBG] from='client.? v1:172.21.15.193:0/242630422' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T21:00:28.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:27.590384+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:28.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:27.590474+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:28.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:27.647481+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:28.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:28.041641+0000 mon.b (mon.1) 13 : audit [DBG] from='client.? v1:172.21.15.193:0/242630422' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T21:00:28.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:27.590384+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:28.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:27.590474+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:28.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:27.647481+0000 mgr.y (mgr.14146) 164 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:28.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:28.041641+0000 mon.b (mon.1) 13 : audit [DBG] from='client.? v1:172.21.15.193:0/242630422' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T21:00:29.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:28.461750+0000 mon.a (mon.0) 461 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-02-19T21:00:29.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:28.461908+0000 mon.a (mon.0) 462 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:29.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:28.461750+0000 mon.a (mon.0) 461 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-02-19T21:00:29.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:28.461908+0000 mon.a (mon.0) 462 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:29.757 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:29.770+0000 7fcde3fff700 10 client.24286.objecter tick 2024-02-19T21:00:29.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:28.461750+0000 mon.a (mon.0) 461 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2024-02-19T21:00:29.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:28.461908+0000 mon.a (mon.0) 462 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:30.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:29.647976+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:30.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:29.647976+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:30.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:29.647976+0000 mgr.y (mgr.14146) 165 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:32.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:31.648416+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:32.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:31.648416+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:32.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:31.648416+0000 mgr.y (mgr.14146) 166 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:34.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:33.584877+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-19T21:00:34.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:33.585616+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:34.235 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:33.584877+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-19T21:00:34.235 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:33.585616+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:34.294 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:33.584877+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-19T21:00:34.294 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:33.585616+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:34.758 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:34.771+0000 7fcde3fff700 10 client.24286.objecter tick 2024-02-19T21:00:35.206 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:00:33.586271+0000 mgr.y (mgr.14146) 167 : cephadm [INF] Deploying daemon osd.6 on smithi193 2024-02-19T21:00:35.206 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:33.648715+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:35.207 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:34.805203+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:35.207 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:34.805314+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:00:33.586271+0000 mgr.y (mgr.14146) 167 : cephadm [INF] Deploying daemon osd.6 on smithi193 2024-02-19T21:00:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:33.648715+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:34.805203+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:35.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:34.805314+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:35.235 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:00:33.586271+0000 mgr.y (mgr.14146) 167 : cephadm [INF] Deploying daemon osd.6 on smithi193 2024-02-19T21:00:35.235 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:33.648715+0000 mgr.y (mgr.14146) 168 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:35.235 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:34.805203+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:35.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:34.805314+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:36.213 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:34.946122+0000 mon.a (mon.0) 465 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2024-02-19T21:00:36.213 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:34.946523+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:36.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:34.946122+0000 mon.a (mon.0) 465 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2024-02-19T21:00:36.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:34.946523+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:36.235 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:34.946122+0000 mon.a (mon.0) 465 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2024-02-19T21:00:36.235 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:34.946523+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:37.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:35.649334+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:37.235 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:35.649334+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:37.309 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:35.649334+0000 mgr.y (mgr.14146) 169 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:39.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:37.649667+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:39.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:38.759593+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:39.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:38.795549+0000 mon.a (mon.0) 468 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:00:39.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:37.649667+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:39.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:38.759593+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:39.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:38.795549+0000 mon.a (mon.0) 468 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:00:39.254 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:37.649667+0000 mgr.y (mgr.14146) 170 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:39.255 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:38.759593+0000 mon.a (mon.0) 467 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:39.255 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:38.795549+0000 mon.a (mon.0) 468 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:00:39.760 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:39.772+0000 7fcde3fff700 10 client.24286.objecter tick 2024-02-19T21:00:41.040 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:39.650132+0000 mgr.y (mgr.14146) 171 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:41.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:39.650132+0000 mgr.y (mgr.14146) 171 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:41.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:39.650132+0000 mgr.y (mgr.14146) 171 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:42.224 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:42.235+0000 7fcdd37fe700 1 -- 172.21.15.193:0/656558327 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 675254457) 0x7fcdc4000b90 con 0x7fcdcc063b40 2024-02-19T21:00:42.224 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 6 on host 'smithi193' 2024-02-19T21:00:42.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:42.237+0000 7fcde898c700 10 librados: watch_flush enter 2024-02-19T21:00:42.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:42.237+0000 7fcde898c700 10 librados: watch_flush exit 2024-02-19T21:00:42.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:42.238+0000 7fcde898c700 20 client.24286.objecter shutdown clearing up homeless session... 2024-02-19T21:00:42.225 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:42.238+0000 7fcde898c700 10 client.24286.objecter successfully canceled tick 2024-02-19T21:00:42.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:42.238+0000 7fcde898c700 1 -- 172.21.15.193:0/656558327 >> v1:172.21.15.163:6800/2735092549 conn(0x7fcdcc063b40 legacy=0x7fcdcc066000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:00:42.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:42.238+0000 7fcde898c700 1 -- 172.21.15.193:0/656558327 >> v1:172.21.15.193:6789/0 conn(0x7fcde40fe820 legacy=0x7fcde41117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:00:42.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:42.238+0000 7fcde898c700 1 -- 172.21.15.193:0/656558327 shutdown_connections 2024-02-19T21:00:42.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:42.238+0000 7fcde898c700 1 -- 172.21.15.193:0/656558327 >> 172.21.15.193:0/656558327 conn(0x7fcde40f1e60 msgr2=0x7fcde40fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:00:42.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:42.238+0000 7fcde898c700 1 -- 172.21.15.193:0/656558327 shutdown_connections 2024-02-19T21:00:42.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:42.238+0000 7fcde898c700 1 -- 172.21.15.193:0/656558327 wait complete. 2024-02-19T21:00:42.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:42.238+0000 7fcde898c700 1 librados: shutdown 2024-02-19T21:00:42.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:41.335570+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:42.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:41.336343+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:42.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:41.337095+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:00:42.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:41.341681+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:42.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:41.620593+0000 mon.b (mon.1) 14 : audit [INF] from='osd.6 v1:172.21.15.193:6808/787604615' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-19T21:00:42.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:41.621003+0000 mon.a (mon.0) 473 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-19T21:00:42.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:41.650579+0000 mgr.y (mgr.14146) 172 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:42.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:42.236187+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:42.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:42.259735+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:00:42.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:41.335570+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:42.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:41.336343+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:42.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:41.337095+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:00:42.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:41.341681+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:42.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:41.620593+0000 mon.b (mon.1) 14 : audit [INF] from='osd.6 v1:172.21.15.193:6808/787604615' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-19T21:00:42.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:41.621003+0000 mon.a (mon.0) 473 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-19T21:00:42.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:41.650579+0000 mgr.y (mgr.14146) 172 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:42.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:42.236187+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:42.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:42.259735+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:00:42.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:41.335570+0000 mon.a (mon.0) 469 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:42.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:41.336343+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:42.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:41.337095+0000 mon.a (mon.0) 471 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:00:42.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:41.341681+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:42.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:41.620593+0000 mon.b (mon.1) 14 : audit [INF] from='osd.6 v1:172.21.15.193:6808/787604615' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-19T21:00:42.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:41.621003+0000 mon.a (mon.0) 473 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-19T21:00:42.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:41.650579+0000 mgr.y (mgr.14146) 172 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:42.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:42.236187+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:42.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:42.259735+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:00:42.941 DEBUG:teuthology.orchestra.run.smithi193:osd.6> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.6.service 2024-02-19T21:00:42.943 INFO:tasks.cephadm:Deploying osd.7 on smithi193 with /dev/vg_nvme/lv_1... 2024-02-19T21:00:42.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-19T21:00:42.973 INFO:journalctl@ceph.osd.6.smithi193.stdout:-- Logs begin at Mon 2024-02-19 20:44:17 UTC. -- 2024-02-19T21:00:43.461 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:42.346375+0000 mon.a (mon.0) 476 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-19T21:00:43.462 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:42.346463+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-19T21:00:43.462 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:42.346606+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:43.462 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:42.351296+0000 mon.b (mon.1) 15 : audit [INF] from='osd.6 v1:172.21.15.193:6808/787604615' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:00:43.462 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:42.351699+0000 mon.a (mon.0) 479 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:00:43.462 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:00:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:00:43.353+0000 7f824bb86700 -1 osd.6 0 waiting for initial osdmap 2024-02-19T21:00:43.462 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:00:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:00:43.360+0000 7f8247300700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T21:00:43.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:42.346375+0000 mon.a (mon.0) 476 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-19T21:00:43.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:42.346463+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-19T21:00:43.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:42.346606+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:43.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:42.351296+0000 mon.b (mon.1) 15 : audit [INF] from='osd.6 v1:172.21.15.193:6808/787604615' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:00:43.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:42.351699+0000 mon.a (mon.0) 479 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:00:43.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:42.346375+0000 mon.a (mon.0) 476 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-19T21:00:43.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:42.346463+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2024-02-19T21:00:43.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:42.346606+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:43.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:42.351296+0000 mon.b (mon.1) 15 : audit [INF] from='osd.6 v1:172.21.15.193:6808/787604615' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:00:43.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:42.351699+0000 mon.a (mon.0) 479 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:00:44.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:43.348849+0000 mon.a (mon.0) 480 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-02-19T21:00:44.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:43.348946+0000 mon.a (mon.0) 481 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2024-02-19T21:00:44.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:43.349399+0000 mon.a (mon.0) 482 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:44.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:43.354630+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:44.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:43.650986+0000 mgr.y (mgr.14146) 173 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:44.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:43.348849+0000 mon.a (mon.0) 480 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-02-19T21:00:44.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:43.348946+0000 mon.a (mon.0) 481 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2024-02-19T21:00:44.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:43.349399+0000 mon.a (mon.0) 482 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:44.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:43.354630+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:44.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:43.650986+0000 mgr.y (mgr.14146) 173 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:44.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:43.348849+0000 mon.a (mon.0) 480 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-02-19T21:00:44.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:43.348946+0000 mon.a (mon.0) 481 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2024-02-19T21:00:44.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:43.349399+0000 mon.a (mon.0) 482 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:44.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:43.354630+0000 mon.a (mon.0) 483 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:44.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:43.650986+0000 mgr.y (mgr.14146) 173 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 0 B data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:00:45.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:42.575237+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:45.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:42.575290+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:45.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:44.352048+0000 mon.a (mon.0) 484 : cluster [INF] osd.6 v1:172.21.15.193:6808/787604615 boot 2024-02-19T21:00:45.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:44.352100+0000 mon.a (mon.0) 485 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2024-02-19T21:00:45.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:44.352238+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:45.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:45.351554+0000 mon.a (mon.0) 487 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2024-02-19T21:00:45.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:42.575237+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:45.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:42.575290+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:45.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:44.352048+0000 mon.a (mon.0) 484 : cluster [INF] osd.6 v1:172.21.15.193:6808/787604615 boot 2024-02-19T21:00:45.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:44.352100+0000 mon.a (mon.0) 485 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2024-02-19T21:00:45.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:44.352238+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:45.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:45.351554+0000 mon.a (mon.0) 487 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2024-02-19T21:00:45.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:42.575237+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:45.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:42.575290+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:45.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:44.352048+0000 mon.a (mon.0) 484 : cluster [INF] osd.6 v1:172.21.15.193:6808/787604615 boot 2024-02-19T21:00:45.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:44.352100+0000 mon.a (mon.0) 485 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2024-02-19T21:00:45.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:44.352238+0000 mon.a (mon.0) 486 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:00:45.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:45.351554+0000 mon.a (mon.0) 487 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2024-02-19T21:00:46.188 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-19T21:00:46.188 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-19T21:00:46.188 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-02-19T21:00:46.188 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-02-19T21:00:46.188 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144893 s, 724 MB/s 2024-02-19T21:00:46.188 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-02-19T21:00:46.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:45.651522+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:46.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:46.353618+0000 mon.a (mon.0) 488 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-19T21:00:46.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:45.651522+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:46.735 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:46.353618+0000 mon.a (mon.0) 488 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-19T21:00:46.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:45.651522+0000 mgr.y (mgr.14146) 174 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:46.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:46.353618+0000 mon.a (mon.0) 488 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2024-02-19T21:00:47.038 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_1 2024-02-19T21:00:48.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:47.651841+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:48.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:00:47.675786+0000 mgr.y (mgr.14146) 176 : cephadm [INF] Detected new or changed devices on smithi193 2024-02-19T21:00:48.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:47.680663+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:48.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:47.681652+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:00:48.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:47.682418+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:00:48.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:47.683138+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:00:48.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:00:47.683684+0000 mgr.y (mgr.14146) 177 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 5707M 2024-02-19T21:00:48.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:47.689991+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:48.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:47.690848+0000 mon.a (mon.0) 494 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:48.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:47.691535+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:00:48.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:47.698339+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:48.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:47.651841+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:48.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:00:47.675786+0000 mgr.y (mgr.14146) 176 : cephadm [INF] Detected new or changed devices on smithi193 2024-02-19T21:00:48.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:47.680663+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:48.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:47.681652+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:00:48.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:47.682418+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:00:48.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:47.683138+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:00:48.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:00:47.683684+0000 mgr.y (mgr.14146) 177 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 5707M 2024-02-19T21:00:48.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:47.689991+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:48.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:47.690848+0000 mon.a (mon.0) 494 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:48.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:47.691535+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:00:48.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:47.698339+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:48.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:47.651841+0000 mgr.y (mgr.14146) 175 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:48.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:00:47.675786+0000 mgr.y (mgr.14146) 176 : cephadm [INF] Detected new or changed devices on smithi193 2024-02-19T21:00:48.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:47.680663+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:48.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:47.681652+0000 mon.a (mon.0) 490 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:00:48.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:47.682418+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:00:48.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:47.683138+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:00:48.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:00:47.683684+0000 mgr.y (mgr.14146) 177 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 5707M 2024-02-19T21:00:48.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:47.689991+0000 mon.a (mon.0) 493 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:48.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:47.690848+0000 mon.a (mon.0) 494 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:48.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:47.691535+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:00:48.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:47.698339+0000 mon.a (mon.0) 496 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:00:49.044 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.056+0000 7ff8c1d9b700 1 -- 172.21.15.193:0/1067706525 >> v1:172.21.15.193:6789/0 conn(0x7ff8bc0ff930 legacy=0x7ff8bc101df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:00:49.044 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.056+0000 7ff8c1d9b700 1 -- 172.21.15.193:0/1067706525 shutdown_connections 2024-02-19T21:00:49.044 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.056+0000 7ff8c1d9b700 1 -- 172.21.15.193:0/1067706525 >> 172.21.15.193:0/1067706525 conn(0x7ff8bc0f2fb0 msgr2=0x7ff8bc0f53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:00:49.044 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.056+0000 7ff8c1d9b700 1 -- 172.21.15.193:0/1067706525 shutdown_connections 2024-02-19T21:00:49.045 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.056+0000 7ff8c1d9b700 1 -- 172.21.15.193:0/1067706525 wait complete. 2024-02-19T21:00:49.047 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8c1d9b700 1 librados: starting msgr at 2024-02-19T21:00:49.047 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8c1d9b700 1 librados: starting objecter 2024-02-19T21:00:49.047 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8c1d9b700 1 Processor -- start 2024-02-19T21:00:49.047 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8c1d9b700 1 -- start start 2024-02-19T21:00:49.047 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8c1d9b700 1 librados: setting wanted keys 2024-02-19T21:00:49.047 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8c1d9b700 1 librados: calling monclient init 2024-02-19T21:00:49.047 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8c1d9b700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff8bc114d60 con 0x7ff8bc0f7830 2024-02-19T21:00:49.047 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8c1d9b700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff8bc115f40 con 0x7ff8bc0ff930 2024-02-19T21:00:49.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8c1d9b700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff8bc117140 con 0x7ff8bc0fbd00 2024-02-19T21:00:49.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8bbfff700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7ff8bc0ff930 0x7ff8bc113420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:48110/0 (socket says 172.21.15.193:48110) 2024-02-19T21:00:49.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8bbfff700 1 -- 172.21.15.193:0/2065506749 learned_addr learned my addr 172.21.15.193:0/2065506749 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T21:00:49.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8b8ff9700 10 client.?.objecter ms_handle_connect 0x7ff8bc0ff930 2024-02-19T21:00:49.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8b8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:00:49.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8b8ff9700 10 client.?.objecter ms_handle_connect 0x7ff8bc0f7830 2024-02-19T21:00:49.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8b8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:00:49.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888677309 0 0) 0x7ff8bc115f40 con 0x7ff8bc0ff930 2024-02-19T21:00:49.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a4002220 con 0x7ff8bc0ff930 2024-02-19T21:00:49.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8b8ff9700 10 client.?.objecter ms_handle_connect 0x7ff8bc0fbd00 2024-02-19T21:00:49.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8b8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:00:49.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.057+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268464631 0 0) 0x7ff8bc114d60 con 0x7ff8bc0f7830 2024-02-19T21:00:49.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8bc115f40 con 0x7ff8bc0f7830 2024-02-19T21:00:49.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402602135 0 0) 0x7ff8bc117140 con 0x7ff8bc0fbd00 2024-02-19T21:00:49.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8bc114d60 con 0x7ff8bc0fbd00 2024-02-19T21:00:49.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 135557376 0 0) 0x7ff8a4002220 con 0x7ff8bc0ff930 2024-02-19T21:00:49.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8bc117140 con 0x7ff8bc0ff930 2024-02-19T21:00:49.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff8b4001e20 con 0x7ff8bc0ff930 2024-02-19T21:00:49.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3581218120 0 0) 0x7ff8bc117140 con 0x7ff8bc0ff930 2024-02-19T21:00:49.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 >> v1:172.21.15.163:6790/0 conn(0x7ff8bc0fbd00 legacy=0x7ff8bc10fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:00:49.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 >> v1:172.21.15.163:6789/0 conn(0x7ff8bc0f7830 legacy=0x7ff8bc06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:00:49.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8bc118340 con 0x7ff8bc0ff930 2024-02-19T21:00:49.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8c1d9b700 1 -- 172.21.15.193:0/2065506749 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8bc114f50 con 0x7ff8bc0ff930 2024-02-19T21:00:49.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8c1d9b700 10 client.24313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:00:49.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8c1d9b700 1 -- 172.21.15.193:0/2065506749 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8bc115440 con 0x7ff8bc0ff930 2024-02-19T21:00:49.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8c1d9b700 1 librados: init done 2024-02-19T21:00:49.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff8b4002da0 con 0x7ff8bc0ff930 2024-02-19T21:00:49.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.058+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff8b4003cd0 con 0x7ff8bc0ff930 2024-02-19T21:00:49.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.059+0000 7ff8c1d9b700 1 -- 172.21.15.193:0/2065506749 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a8004f50 con 0x7ff8bc0ff930 2024-02-19T21:00:49.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.059+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 7720783 0 0) 0x7ff8b4003310 con 0x7ff8bc0ff930 2024-02-19T21:00:49.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.060+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(47..47 src has 1..47) v4 ==== 3762+0+0 (unknown 3190511858 0 0) 0x7ff8b40782c0 con 0x7ff8bc0ff930 2024-02-19T21:00:49.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.060+0000 7ff8b8ff9700 10 client.24313.objecter ms_dispatch 0x7ff8bc001550 osd_map(47..47 src has 1..47) v4 2024-02-19T21:00:49.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.060+0000 7ff8b8ff9700 3 client.24313.objecter handle_osd_map got epochs [47,47] > 0 2024-02-19T21:00:49.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.060+0000 7ff8b8ff9700 3 client.24313.objecter handle_osd_map decoding full epoch 47 2024-02-19T21:00:49.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.060+0000 7ff8b8ff9700 20 client.24313.objecter dump_active .. 0 homeless 2024-02-19T21:00:49.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.060+0000 7ff8b8ff9700 10 client.24313.objecter ms_handle_connect 0x7ff8a4063a50 2024-02-19T21:00:49.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.064+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff8b4048de0 con 0x7ff8bc0ff930 2024-02-19T21:00:49.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:49.234+0000 7ff8c1d9b700 1 -- 172.21.15.193:0/2065506749 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8a8000bc0 con 0x7ff8a4063a50 2024-02-19T21:00:49.928 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:49.236325+0000 mgr.y (mgr.14146) 178 : audit [DBG] from='client.24313 v1:172.21.15.193:0/2065506749' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:00:49.928 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:49.238845+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T21:00:49.928 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:49.241093+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T21:00:49.928 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:49.241748+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:49.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:49.236325+0000 mgr.y (mgr.14146) 178 : audit [DBG] from='client.24313 v1:172.21.15.193:0/2065506749' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:00:49.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:49.238845+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T21:00:49.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:49.241093+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T21:00:49.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:49.241748+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:49.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:49.236325+0000 mgr.y (mgr.14146) 178 : audit [DBG] from='client.24313 v1:172.21.15.193:0/2065506749' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:00:49.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:49.238845+0000 mon.a (mon.0) 497 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T21:00:49.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:49.241093+0000 mon.a (mon.0) 498 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T21:00:49.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:49.241748+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:50.941 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:49.652193+0000 mgr.y (mgr.14146) 179 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:50.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:49.652193+0000 mgr.y (mgr.14146) 179 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:50.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:49.652193+0000 mgr.y (mgr.14146) 179 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:51.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:50.702131+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:51.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:50.702187+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:51.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:50.702131+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:51.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:50.702187+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:52.059 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:50.702131+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:52.059 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:50.702187+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:52.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:51.652627+0000 mgr.y (mgr.14146) 180 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:52.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:51.692011+0000 mon.b (mon.1) 16 : audit [INF] from='client.? v1:172.21.15.193:0/359473975' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f"}]: dispatch 2024-02-19T21:00:52.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:51.692528+0000 mon.a (mon.0) 500 : audit [INF] from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f"}]: dispatch 2024-02-19T21:00:52.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:51.699356+0000 mon.a (mon.0) 501 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2024-02-19T21:00:52.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:51.702541+0000 mon.a (mon.0) 502 : audit [INF] from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f"}]': finished 2024-02-19T21:00:52.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:51.702604+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e49: 8 total, 7 up, 8 in 2024-02-19T21:00:52.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:51.702727+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:00:52.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:52.263521+0000 mon.b (mon.1) 17 : audit [DBG] from='client.? v1:172.21.15.193:0/2554479598' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T21:00:52.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:51.652627+0000 mgr.y (mgr.14146) 180 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:52.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:51.692011+0000 mon.b (mon.1) 16 : audit [INF] from='client.? v1:172.21.15.193:0/359473975' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f"}]: dispatch 2024-02-19T21:00:52.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:51.692528+0000 mon.a (mon.0) 500 : audit [INF] from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f"}]: dispatch 2024-02-19T21:00:52.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:51.699356+0000 mon.a (mon.0) 501 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2024-02-19T21:00:52.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:51.702541+0000 mon.a (mon.0) 502 : audit [INF] from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f"}]': finished 2024-02-19T21:00:52.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:51.702604+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e49: 8 total, 7 up, 8 in 2024-02-19T21:00:52.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:51.702727+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:00:52.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:52.263521+0000 mon.b (mon.1) 17 : audit [DBG] from='client.? v1:172.21.15.193:0/2554479598' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T21:00:53.059 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:51.652627+0000 mgr.y (mgr.14146) 180 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:53.059 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:51.692011+0000 mon.b (mon.1) 16 : audit [INF] from='client.? v1:172.21.15.193:0/359473975' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f"}]: dispatch 2024-02-19T21:00:53.059 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:51.692528+0000 mon.a (mon.0) 500 : audit [INF] from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f"}]: dispatch 2024-02-19T21:00:53.059 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:51.699356+0000 mon.a (mon.0) 501 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2024-02-19T21:00:53.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:51.702541+0000 mon.a (mon.0) 502 : audit [INF] from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f"}]': finished 2024-02-19T21:00:53.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:51.702604+0000 mon.a (mon.0) 503 : cluster [DBG] osdmap e49: 8 total, 7 up, 8 in 2024-02-19T21:00:53.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:51.702727+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:00:53.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:52.263521+0000 mon.b (mon.1) 17 : audit [DBG] from='client.? v1:172.21.15.193:0/2554479598' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T21:00:54.047 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:54.059+0000 7ff8c159a700 10 client.24313.objecter tick 2024-02-19T21:00:55.144 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:53.653048+0000 mgr.y (mgr.14146) 181 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:55.235 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:53.653048+0000 mgr.y (mgr.14146) 181 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:55.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:53.653048+0000 mgr.y (mgr.14146) 181 : cluster [DBG] pgmap v171: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:57.059 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:55.653425+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:57.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:55.653425+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:57.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:55.653425+0000 mgr.y (mgr.14146) 182 : cluster [DBG] pgmap v172: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:59.048 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:00:59.060+0000 7ff8c159a700 10 client.24313.objecter tick 2024-02-19T21:00:59.059 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:57.342637+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:59.059 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:57.342697+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:59.059 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:57.653753+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:59.059 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:57.956638+0000 mon.a (mon.0) 505 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2024-02-19T21:00:59.059 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:57.956786+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:00:59.059 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:57.983448+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-19T21:00:59.059 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:00:57.984126+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:59.059 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:00:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:00:57.984716+0000 mgr.y (mgr.14146) 184 : cephadm [INF] Deploying daemon osd.7 on smithi193 2024-02-19T21:00:59.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:57.342637+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:59.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:57.342697+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:59.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:57.653753+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:59.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:57.956638+0000 mon.a (mon.0) 505 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2024-02-19T21:00:59.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:57.956786+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:00:59.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:57.983448+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-19T21:00:59.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:00:57.984126+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:59.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:00:57.984716+0000 mgr.y (mgr.14146) 184 : cephadm [INF] Deploying daemon osd.7 on smithi193 2024-02-19T21:00:59.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:57.342637+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:00:59.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:57.342697+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:00:59.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:57.653753+0000 mgr.y (mgr.14146) 183 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:00:59.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:57.956638+0000 mon.a (mon.0) 505 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2024-02-19T21:00:59.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:57.956786+0000 mon.a (mon.0) 506 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:00:59.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:57.983448+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-19T21:00:59.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:00:57.984126+0000 mon.a (mon.0) 508 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:00:59.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:00:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:00:57.984716+0000 mgr.y (mgr.14146) 184 : cephadm [INF] Deploying daemon osd.7 on smithi193 2024-02-19T21:01:01.053 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:00:59.654215+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:01:01.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:00:59.654215+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:01:01.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:00:59.654215+0000 mgr.y (mgr.14146) 185 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:01:03.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:01.654659+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:01:03.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:01.654659+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:01:03.309 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:01.654659+0000 mgr.y (mgr.14146) 186 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:01:04.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:04.061+0000 7ff8c159a700 10 client.24313.objecter tick 2024-02-19T21:01:04.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:03.200608+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:04.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:03.203079+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:04.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:03.200608+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:04.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:03.203079+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:04.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:03.200608+0000 mon.a (mon.0) 509 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:04.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:03.203079+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:05.211 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:03.654989+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:01:05.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:03.654989+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:01:05.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:03.654989+0000 mgr.y (mgr.14146) 187 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:01:06.686 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:06.698+0000 7ff8b8ff9700 1 -- 172.21.15.193:0/2065506749 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 1948941837) 0x7ff8a8000bc0 con 0x7ff8a4063a50 2024-02-19T21:01:06.686 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 7 on host 'smithi193' 2024-02-19T21:01:06.688 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:06.700+0000 7ff8c1d9b700 10 librados: watch_flush enter 2024-02-19T21:01:06.688 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:06.700+0000 7ff8c1d9b700 10 librados: watch_flush exit 2024-02-19T21:01:06.688 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:06.700+0000 7ff8c1d9b700 20 client.24313.objecter shutdown clearing up homeless session... 2024-02-19T21:01:06.688 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:06.700+0000 7ff8c1d9b700 10 client.24313.objecter successfully canceled tick 2024-02-19T21:01:06.689 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:06.701+0000 7ff8c1d9b700 1 -- 172.21.15.193:0/2065506749 >> v1:172.21.15.163:6800/2735092549 conn(0x7ff8a4063a50 legacy=0x7ff8a4065f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:06.689 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:06.701+0000 7ff8c1d9b700 1 -- 172.21.15.193:0/2065506749 >> v1:172.21.15.193:6789/0 conn(0x7ff8bc0ff930 legacy=0x7ff8bc113420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:06.689 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:06.701+0000 7ff8c1d9b700 1 -- 172.21.15.193:0/2065506749 shutdown_connections 2024-02-19T21:01:06.689 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:06.701+0000 7ff8c1d9b700 1 -- 172.21.15.193:0/2065506749 >> 172.21.15.193:0/2065506749 conn(0x7ff8bc0f2fb0 msgr2=0x7ff8bc0f3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:06.689 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:06.701+0000 7ff8c1d9b700 1 -- 172.21.15.193:0/2065506749 shutdown_connections 2024-02-19T21:01:06.689 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:06.701+0000 7ff8c1d9b700 1 -- 172.21.15.193:0/2065506749 wait complete. 2024-02-19T21:01:06.689 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:06.701+0000 7ff8c1d9b700 1 librados: shutdown 2024-02-19T21:01:07.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:05.655517+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:01:07.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:05.959358+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:07.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:06.304737+0000 mon.b (mon.1) 18 : audit [INF] from='osd.7 v1:172.21.15.193:6812/157927432' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-19T21:01:07.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:06.305227+0000 mon.a (mon.0) 512 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-19T21:01:07.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:06.365661+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:07.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:06.366462+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:07.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:06.372762+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:07.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:06.698054+0000 mon.a (mon.0) 516 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:07.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:06.739727+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:07.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:05.655517+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:01:07.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:05.959358+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:07.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:06.304737+0000 mon.b (mon.1) 18 : audit [INF] from='osd.7 v1:172.21.15.193:6812/157927432' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-19T21:01:07.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:06.305227+0000 mon.a (mon.0) 512 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-19T21:01:07.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:06.365661+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:07.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:06.366462+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:07.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:06.372762+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:07.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:06.698054+0000 mon.a (mon.0) 516 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:07.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:06.739727+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:07.287 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:05.655517+0000 mgr.y (mgr.14146) 188 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:01:07.288 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:05.959358+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:07.288 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:06.304737+0000 mon.b (mon.1) 18 : audit [INF] from='osd.7 v1:172.21.15.193:6812/157927432' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-19T21:01:07.288 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:06.305227+0000 mon.a (mon.0) 512 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-19T21:01:07.288 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:06.365661+0000 mon.a (mon.0) 513 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:07.288 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:06.366462+0000 mon.a (mon.0) 514 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:07.288 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:06.372762+0000 mon.a (mon.0) 515 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:07.288 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:06.698054+0000 mon.a (mon.0) 516 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:07.288 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:06.739727+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:07.307 DEBUG:teuthology.orchestra.run.smithi193:osd.7> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.7.service 2024-02-19T21:01:07.310 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-19T21:01:07.310 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd stat -f json 2024-02-19T21:01:07.338 INFO:journalctl@ceph.osd.7.smithi193.stdout:-- Logs begin at Mon 2024-02-19 20:44:17 UTC. -- 2024-02-19T21:01:08.210 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:07.215283+0000 mon.a (mon.0) 518 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-19T21:01:08.210 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:07.215370+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2024-02-19T21:01:08.210 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:07.215509+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:01:08.210 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:07.215784+0000 mon.b (mon.1) 19 : audit [INF] from='osd.7 v1:172.21.15.193:6812/157927432' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:01:08.210 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:07.216192+0000 mon.a (mon.0) 521 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:01:08.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:07.215283+0000 mon.a (mon.0) 518 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-19T21:01:08.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:07.215370+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2024-02-19T21:01:08.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:07.215509+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:01:08.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:07.215784+0000 mon.b (mon.1) 19 : audit [INF] from='osd.7 v1:172.21.15.193:6812/157927432' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:01:08.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:07.216192+0000 mon.a (mon.0) 521 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:01:08.559 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:01:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:01:08.224+0000 7fe7ca2de700 -1 osd.7 0 waiting for initial osdmap 2024-02-19T21:01:08.559 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:01:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:01:08.230+0000 7fe7c3253700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T21:01:08.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:07.215283+0000 mon.a (mon.0) 518 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-19T21:01:08.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:07.215370+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2024-02-19T21:01:08.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:07.215509+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:01:08.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:07.215784+0000 mon.b (mon.1) 19 : audit [INF] from='osd.7 v1:172.21.15.193:6812/157927432' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:01:08.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:07.216192+0000 mon.a (mon.0) 521 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-02-19T21:01:09.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.164+0000 7fb6ad186700 1 -- 172.21.15.163:0/1278444697 >> v1:172.21.15.163:6790/0 conn(0x7fb6a80fbcf0 legacy=0x7fb6a80fe0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:09.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.164+0000 7fb6ad186700 1 -- 172.21.15.163:0/1278444697 shutdown_connections 2024-02-19T21:01:09.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.164+0000 7fb6ad186700 1 -- 172.21.15.163:0/1278444697 >> 172.21.15.163:0/1278444697 conn(0x7fb6a80f2f80 msgr2=0x7fb6a80f53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:09.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.164+0000 7fb6ad186700 1 -- 172.21.15.163:0/1278444697 shutdown_connections 2024-02-19T21:01:09.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.164+0000 7fb6ad186700 1 -- 172.21.15.163:0/1278444697 wait complete. 2024-02-19T21:01:09.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.165+0000 7fb6ad186700 1 librados: starting msgr at 2024-02-19T21:01:09.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.165+0000 7fb6ad186700 1 librados: starting objecter 2024-02-19T21:01:09.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.165+0000 7fb6ad186700 1 Processor -- start 2024-02-19T21:01:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.165+0000 7fb6ad186700 1 -- start start 2024-02-19T21:01:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.165+0000 7fb6ad186700 1 librados: setting wanted keys 2024-02-19T21:01:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.165+0000 7fb6ad186700 1 librados: calling monclient init 2024-02-19T21:01:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb6ad186700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6a8114b60 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb6ad186700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6a8115d40 con 0x7fb6a80ff920 2024-02-19T21:01:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb6ad186700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6a8116f40 con 0x7fb6a80f7820 2024-02-19T21:01:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb6a659c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fb6a80fbcf0 0x7fb6a810f8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:49388/0 (socket says 172.21.15.163:49388) 2024-02-19T21:01:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb6a659c700 1 -- 172.21.15.163:0/3594900485 learned_addr learned my addr 172.21.15.163:0/3594900485 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:01:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb6a6d9d700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fb6a80f7820 0x7fb6a806c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:37456/0 (socket says 172.21.15.163:37456) 2024-02-19T21:01:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 10 client.?.objecter ms_handle_connect 0x7fb6a80fbcf0 2024-02-19T21:01:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 10 client.?.objecter ms_handle_connect 0x7fb6a80f7820 2024-02-19T21:01:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:09.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 10 client.?.objecter ms_handle_connect 0x7fb6a80ff920 2024-02-19T21:01:09.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:09.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074548396 0 0) 0x7fb6a8114b60 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6900033f0 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119399820 0 0) 0x7fb6a8116f40 con 0x7fb6a80f7820 2024-02-19T21:01:09.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a8114b60 con 0x7fb6a80f7820 2024-02-19T21:01:09.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720225443 0 0) 0x7fb6a8115d40 con 0x7fb6a80ff920 2024-02-19T21:01:09.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a8116f40 con 0x7fb6a80ff920 2024-02-19T21:01:09.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 334202861 0 0) 0x7fb6a8114b60 con 0x7fb6a80f7820 2024-02-19T21:01:09.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6a8115d40 con 0x7fb6a80f7820 2024-02-19T21:01:09.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb6980018c0 con 0x7fb6a80f7820 2024-02-19T21:01:09.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3196222477 0 0) 0x7fb6900033f0 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6a8114b60 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb69c0021f0 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 54696229 0 0) 0x7fb6a8116f40 con 0x7fb6a80ff920 2024-02-19T21:01:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6900033f0 con 0x7fb6a80ff920 2024-02-19T21:01:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb6a0001bb0 con 0x7fb6a80ff920 2024-02-19T21:01:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2032215184 0 0) 0x7fb6a8114b60 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 >> v1:172.21.15.163:6790/0 conn(0x7fb6a80f7820 legacy=0x7fb6a806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 >> v1:172.21.15.193:6789/0 conn(0x7fb6a80ff920 legacy=0x7fb6a8113220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a8118140 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb6ad186700 1 -- 172.21.15.163:0/3594900485 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a8117130 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb6ad186700 10 client.14424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:01:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.166+0000 7fb6ad186700 1 -- 172.21.15.163:0/3594900485 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a8117620 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.167+0000 7fb6ad186700 1 librados: init done 2024-02-19T21:01:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.167+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb69c001180 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.167+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb69c0038f0 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.167+0000 7fb6ad186700 1 -- 172.21.15.163:0/3594900485 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb694004f20 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.168+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 7720783 0 0) 0x7fb69c003af0 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.168+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(52..52 src has 1..52) v4 ==== 3914+0+0 (unknown 2633792893 0 0) 0x7fb69c077f20 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.168+0000 7fb68ffff700 10 client.14424.objecter ms_dispatch 0x7fb6a8001550 osd_map(52..52 src has 1..52) v4 2024-02-19T21:01:09.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.168+0000 7fb68ffff700 3 client.14424.objecter handle_osd_map got epochs [52,52] > 0 2024-02-19T21:01:09.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.168+0000 7fb68ffff700 3 client.14424.objecter handle_osd_map decoding full epoch 52 2024-02-19T21:01:09.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.168+0000 7fb68ffff700 20 client.14424.objecter dump_active .. 0 homeless 2024-02-19T21:01:09.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.169+0000 7fb68ffff700 10 client.14424.objecter ms_handle_connect 0x7fb690063b40 2024-02-19T21:01:09.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.172+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb69c0489a0 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.209 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:07.655908+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:01:09.209 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:08.217384+0000 mon.a (mon.0) 522 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-02-19T21:01:09.209 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:08.217466+0000 mon.a (mon.0) 523 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2024-02-19T21:01:09.209 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:08.217789+0000 mon.a (mon.0) 524 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:01:09.209 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:08.222834+0000 mon.a (mon.0) 525 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:01:09.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.340+0000 7fb6ad186700 1 -- 172.21.15.163:0/3594900485 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fb6940051d0 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.341+0000 7fb68ffff700 1 -- 172.21.15.163:0/3594900485 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v53) v1 ==== 74+0+130 (unknown 4062121142 0 2997127448) 0x7fb69c04a8d0 con 0x7fb6a80fbcf0 2024-02-19T21:01:09.329 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:01:09.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.342+0000 7fb6ad186700 10 librados: watch_flush enter 2024-02-19T21:01:09.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.342+0000 7fb6ad186700 10 librados: watch_flush exit 2024-02-19T21:01:09.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.342+0000 7fb6ad186700 20 client.14424.objecter shutdown clearing up homeless session... 2024-02-19T21:01:09.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.342+0000 7fb6ad186700 10 client.14424.objecter successfully canceled tick 2024-02-19T21:01:09.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.342+0000 7fb6ad186700 1 -- 172.21.15.163:0/3594900485 >> v1:172.21.15.163:6800/2735092549 conn(0x7fb690063b40 legacy=0x7fb690066000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:09.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.342+0000 7fb6ad186700 1 -- 172.21.15.163:0/3594900485 >> v1:172.21.15.163:6789/0 conn(0x7fb6a80fbcf0 legacy=0x7fb6a810f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:09.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.342+0000 7fb6ad186700 1 -- 172.21.15.163:0/3594900485 shutdown_connections 2024-02-19T21:01:09.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.342+0000 7fb6ad186700 1 -- 172.21.15.163:0/3594900485 >> 172.21.15.163:0/3594900485 conn(0x7fb6a80f2f80 msgr2=0x7fb6a80f5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:09.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.343+0000 7fb6ad186700 1 -- 172.21.15.163:0/3594900485 shutdown_connections 2024-02-19T21:01:09.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.343+0000 7fb6ad186700 1 -- 172.21.15.163:0/3594900485 wait complete. 2024-02-19T21:01:09.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:09.343+0000 7fb6ad186700 1 librados: shutdown 2024-02-19T21:01:09.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:07.655908+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:01:09.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:08.217384+0000 mon.a (mon.0) 522 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-02-19T21:01:09.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:08.217466+0000 mon.a (mon.0) 523 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2024-02-19T21:01:09.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:08.217789+0000 mon.a (mon.0) 524 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:01:09.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:08.222834+0000 mon.a (mon.0) 525 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:01:09.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:07.655908+0000 mgr.y (mgr.14146) 189 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:01:09.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:08.217384+0000 mon.a (mon.0) 522 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-02-19T21:01:09.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:08.217466+0000 mon.a (mon.0) 523 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2024-02-19T21:01:09.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:08.217789+0000 mon.a (mon.0) 524 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:01:09.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:08.222834+0000 mon.a (mon.0) 525 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:01:09.945 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":53,"num_osds":8,"num_up_osds":8,"osd_up_since":1708376469,"num_in_osds":8,"osd_in_since":1708376451,"num_remapped_pgs":1} 2024-02-19T21:01:09.946 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi163 2024-02-19T21:01:09.946 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi163=foo.a' 2024-02-19T21:01:10.435 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:07.350176+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:01:10.435 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:07.350221+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:01:10.435 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:09.219229+0000 mon.a (mon.0) 526 : cluster [INF] osd.7 v1:172.21.15.193:6812/157927432 boot 2024-02-19T21:01:10.435 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:09.219281+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-02-19T21:01:10.435 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:09.219434+0000 mon.a (mon.0) 528 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:01:10.435 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:09.341285+0000 mon.a (mon.0) 529 : audit [DBG] from='client.? v1:172.21.15.163:0/3594900485' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-19T21:01:10.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:07.350176+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:01:10.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:07.350221+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:01:10.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:09.219229+0000 mon.a (mon.0) 526 : cluster [INF] osd.7 v1:172.21.15.193:6812/157927432 boot 2024-02-19T21:01:10.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:09.219281+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-02-19T21:01:10.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:09.219434+0000 mon.a (mon.0) 528 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:01:10.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:09.341285+0000 mon.a (mon.0) 529 : audit [DBG] from='client.? v1:172.21.15.163:0/3594900485' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-19T21:01:10.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:07.350176+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:01:10.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:07.350221+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:01:10.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:09.219229+0000 mon.a (mon.0) 526 : cluster [INF] osd.7 v1:172.21.15.193:6812/157927432 boot 2024-02-19T21:01:10.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:09.219281+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2024-02-19T21:01:10.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:09.219434+0000 mon.a (mon.0) 528 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:01:10.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:09.341285+0000 mon.a (mon.0) 529 : audit [DBG] from='client.? v1:172.21.15.163:0/3594900485' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-19T21:01:11.385 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:09.656333+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:11.385 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:10.220453+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-19T21:01:11.385 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:10.483577+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:11.385 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:10.484501+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:11.385 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:10.485157+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:11.385 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:10.485751+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:11.385 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:10.486329+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:11.385 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:10.490491+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:11.385 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:10.491271+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:11.386 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:10.491899+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:11.386 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:10.495958+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:11.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:09.656333+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:11.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:10.220453+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-19T21:01:11.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:10.483577+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:11.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:10.484501+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:11.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:10.485157+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:11.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:10.485751+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:11.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:10.486329+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:11.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:10.490491+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:11.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:10.491271+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:11.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:10.491899+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:11.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:10.495958+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:11.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:09.656333+0000 mgr.y (mgr.14146) 190 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:11.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:10.220453+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2024-02-19T21:01:11.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:10.483577+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:11.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:10.484501+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:11.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:10.485157+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:11.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:10.485751+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:11.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:10.486329+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:11.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:10.490491+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:11.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:10.491271+0000 mon.a (mon.0) 537 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:11.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:10.491899+0000 mon.a (mon.0) 538 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:11.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:10.495958+0000 mon.a (mon.0) 539 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:11.846 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.855+0000 7fc5dac99700 1 -- 172.21.15.193:0/1640210880 >> v1:172.21.15.193:6789/0 conn(0x7fc5d40fbd00 legacy=0x7fc5d40fe0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:11.846 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.855+0000 7fc5dac99700 1 -- 172.21.15.193:0/1640210880 shutdown_connections 2024-02-19T21:01:11.846 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.855+0000 7fc5dac99700 1 -- 172.21.15.193:0/1640210880 >> 172.21.15.193:0/1640210880 conn(0x7fc5d40f2fb0 msgr2=0x7fc5d40f53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:11.846 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.855+0000 7fc5dac99700 1 -- 172.21.15.193:0/1640210880 shutdown_connections 2024-02-19T21:01:11.846 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.855+0000 7fc5dac99700 1 -- 172.21.15.193:0/1640210880 wait complete. 2024-02-19T21:01:11.846 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.855+0000 7fc5dac99700 1 librados: starting msgr at 2024-02-19T21:01:11.846 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.855+0000 7fc5dac99700 1 librados: starting objecter 2024-02-19T21:01:11.846 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.855+0000 7fc5dac99700 1 Processor -- start 2024-02-19T21:01:11.846 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.855+0000 7fc5dac99700 1 -- start start 2024-02-19T21:01:11.846 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.855+0000 7fc5dac99700 1 librados: setting wanted keys 2024-02-19T21:01:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.855+0000 7fc5dac99700 1 librados: calling monclient init 2024-02-19T21:01:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.855+0000 7fc5dac99700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc5d4108540 con 0x7fc5d40ff930 2024-02-19T21:01:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.855+0000 7fc5dac99700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc5d4113da0 con 0x7fc5d40f7830 2024-02-19T21:01:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.855+0000 7fc5dac99700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc5d4114f80 con 0x7fc5d40fbd00 2024-02-19T21:01:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d8a35700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fc5d40f7830 0x7fc5d4107970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:53170/0 (socket says 172.21.15.193:53170) 2024-02-19T21:01:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d8a35700 1 -- 172.21.15.193:0/1395329836 learned_addr learned my addr 172.21.15.193:0/1395329836 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T21:01:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 10 client.?.objecter ms_handle_connect 0x7fc5d40f7830 2024-02-19T21:01:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 10 client.?.objecter ms_handle_connect 0x7fc5d40fbd00 2024-02-19T21:01:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 10 client.?.objecter ms_handle_connect 0x7fc5d40ff930 2024-02-19T21:01:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233910189 0 0) 0x7fc5d4113da0 con 0x7fc5d40f7830 2024-02-19T21:01:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c40033f0 con 0x7fc5d40f7830 2024-02-19T21:01:11.847 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632757715 0 0) 0x7fc5d4114f80 con 0x7fc5d40fbd00 2024-02-19T21:01:11.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d4113da0 con 0x7fc5d40fbd00 2024-02-19T21:01:11.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763975810 0 0) 0x7fc5d4108540 con 0x7fc5d40ff930 2024-02-19T21:01:11.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d4114f80 con 0x7fc5d40ff930 2024-02-19T21:01:11.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 593452768 0 0) 0x7fc5c40033f0 con 0x7fc5d40f7830 2024-02-19T21:01:11.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5d4108540 con 0x7fc5d40f7830 2024-02-19T21:01:11.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc5c8002950 con 0x7fc5d40f7830 2024-02-19T21:01:11.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1858734584 0 0) 0x7fc5d4113da0 con 0x7fc5d40fbd00 2024-02-19T21:01:11.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5c40033f0 con 0x7fc5d40fbd00 2024-02-19T21:01:11.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc5c0001ec0 con 0x7fc5d40fbd00 2024-02-19T21:01:11.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2931564723 0 0) 0x7fc5d4114f80 con 0x7fc5d40ff930 2024-02-19T21:01:11.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5d4113da0 con 0x7fc5d40ff930 2024-02-19T21:01:11.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc5cc001bc0 con 0x7fc5d40ff930 2024-02-19T21:01:11.848 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 71855392 0 0) 0x7fc5d4108540 con 0x7fc5d40f7830 2024-02-19T21:01:11.849 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 >> v1:172.21.15.163:6790/0 conn(0x7fc5d40fbd00 legacy=0x7fc5d410df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:11.849 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 >> v1:172.21.15.163:6789/0 conn(0x7fc5d40ff930 legacy=0x7fc5d4111670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:11.849 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d4116160 con 0x7fc5d40f7830 2024-02-19T21:01:11.849 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5dac99700 1 -- 172.21.15.193:0/1395329836 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d4115170 con 0x7fc5d40f7830 2024-02-19T21:01:11.849 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fc5c8001810 con 0x7fc5d40f7830 2024-02-19T21:01:11.849 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.856+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc5c8005b10 con 0x7fc5d40f7830 2024-02-19T21:01:11.849 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.857+0000 7fc5dac99700 10 client.24346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:01:11.849 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.857+0000 7fc5dac99700 1 -- 172.21.15.193:0/1395329836 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d4115660 con 0x7fc5d40f7830 2024-02-19T21:01:11.849 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.857+0000 7fc5dac99700 1 librados: init done 2024-02-19T21:01:11.850 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.858+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 7720783 0 0) 0x7fc5c8001370 con 0x7fc5d40f7830 2024-02-19T21:01:11.850 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.858+0000 7fc5dac99700 1 -- 172.21.15.193:0/1395329836 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d4047540 con 0x7fc5d40f7830 2024-02-19T21:01:11.850 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.858+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(55..55 src has 1..55) v4 ==== 4054+0+0 (unknown 1811980291 0 0) 0x7fc5c8077cc0 con 0x7fc5d40f7830 2024-02-19T21:01:11.850 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.858+0000 7fc5d1ffb700 10 client.24346.objecter ms_dispatch 0x7fc5d4001550 osd_map(55..55 src has 1..55) v4 2024-02-19T21:01:11.850 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.858+0000 7fc5d1ffb700 3 client.24346.objecter handle_osd_map got epochs [55,55] > 0 2024-02-19T21:01:11.850 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.858+0000 7fc5d1ffb700 3 client.24346.objecter handle_osd_map decoding full epoch 55 2024-02-19T21:01:11.850 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.858+0000 7fc5d1ffb700 20 client.24346.objecter dump_active .. 0 homeless 2024-02-19T21:01:11.850 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.859+0000 7fc5d1ffb700 10 client.24346.objecter ms_handle_connect 0x7fc5c4063b00 2024-02-19T21:01:11.851 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:11.862+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc5c80485a0 con 0x7fc5d40f7830 2024-02-19T21:01:12.023 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:12.035+0000 7fc5dac99700 1 -- 172.21.15.193:0/1395329836 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi163=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5d4115a00 con 0x7fc5c4063b00 2024-02-19T21:01:12.030 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:12.041+0000 7fc5d1ffb700 1 -- 172.21.15.193:0/1395329836 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (unknown 0 0 1123153589) 0x7fc5d4115a00 con 0x7fc5c4063b00 2024-02-19T21:01:12.030 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled rgw.foo.a update... 2024-02-19T21:01:12.032 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:12.043+0000 7fc5dac99700 10 librados: watch_flush enter 2024-02-19T21:01:12.032 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:12.043+0000 7fc5dac99700 10 librados: watch_flush exit 2024-02-19T21:01:12.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:12.043+0000 7fc5dac99700 20 client.24346.objecter shutdown clearing up homeless session... 2024-02-19T21:01:12.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:12.043+0000 7fc5dac99700 10 client.24346.objecter successfully canceled tick 2024-02-19T21:01:12.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:12.044+0000 7fc5dac99700 1 -- 172.21.15.193:0/1395329836 >> v1:172.21.15.163:6800/2735092549 conn(0x7fc5c4063b00 legacy=0x7fc5c4065fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:12.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:12.044+0000 7fc5dac99700 1 -- 172.21.15.193:0/1395329836 >> v1:172.21.15.193:6789/0 conn(0x7fc5d40f7830 legacy=0x7fc5d4107970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:12.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:12.044+0000 7fc5dac99700 1 -- 172.21.15.193:0/1395329836 shutdown_connections 2024-02-19T21:01:12.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:12.044+0000 7fc5dac99700 1 -- 172.21.15.193:0/1395329836 >> 172.21.15.193:0/1395329836 conn(0x7fc5d40f2fb0 msgr2=0x7fc5d40ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:12.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:12.044+0000 7fc5dac99700 1 -- 172.21.15.193:0/1395329836 shutdown_connections 2024-02-19T21:01:12.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:12.044+0000 7fc5dac99700 1 -- 172.21.15.193:0/1395329836 wait complete. 2024-02-19T21:01:12.033 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:12.044+0000 7fc5dac99700 1 librados: shutdown 2024-02-19T21:01:12.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:10.478937+0000 mgr.y (mgr.14146) 191 : cephadm [INF] Detected new or changed devices on smithi193 2024-02-19T21:01:12.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:10.486775+0000 mgr.y (mgr.14146) 192 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 4280M 2024-02-19T21:01:12.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:11.228414+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-19T21:01:12.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:12.042630+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:12.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:12.066008+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:12.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:12.067626+0000 mon.a (mon.0) 543 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:12.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:12.068379+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:12.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:12.075434+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:12.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:12.084118+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:12.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:12.084697+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-19T21:01:12.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:12.090214+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-19T21:01:12.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:12.096550+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:12.311 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:12.097366+0000 mon.a (mon.0) 550 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:12.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:10.478937+0000 mgr.y (mgr.14146) 191 : cephadm [INF] Detected new or changed devices on smithi193 2024-02-19T21:01:12.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:10.486775+0000 mgr.y (mgr.14146) 192 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 4280M 2024-02-19T21:01:12.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:11.228414+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-19T21:01:12.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:12.042630+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:12.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:12.066008+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:12.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:12.067626+0000 mon.a (mon.0) 543 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:12.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:12.068379+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:12.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:12.075434+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:12.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:12.084118+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:12.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:12.084697+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-19T21:01:12.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:12.090214+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-19T21:01:12.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:12.096550+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:12.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:12.097366+0000 mon.a (mon.0) 550 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:12.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:10.478937+0000 mgr.y (mgr.14146) 191 : cephadm [INF] Detected new or changed devices on smithi193 2024-02-19T21:01:12.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:10.486775+0000 mgr.y (mgr.14146) 192 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 4280M 2024-02-19T21:01:12.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:11.228414+0000 mon.a (mon.0) 540 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2024-02-19T21:01:12.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:12.042630+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:12.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:12.066008+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:12.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:12.067626+0000 mon.a (mon.0) 543 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:12.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:12.068379+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:12.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:12.075434+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:12.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:12.084118+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:12.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:12.084697+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-19T21:01:12.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:12.090214+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-19T21:01:12.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:12.096550+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:12.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:12.097366+0000 mon.a (mon.0) 550 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:12.727 DEBUG:teuthology.orchestra.run.smithi163:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@rgw.foo.a.service 2024-02-19T21:01:12.730 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi193 2024-02-19T21:01:12.730 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool create iscsi 3 3 replicated 2024-02-19T21:01:12.758 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:-- Logs begin at Mon 2024-02-19 20:44:35 UTC. -- 2024-02-19T21:01:13.451 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:11.656734+0000 mgr.y (mgr.14146) 193 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:13.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:11.764788+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:01:13.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:11.764862+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:01:13.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:12.037134+0000 mgr.y (mgr.14146) 194 : audit [DBG] from='client.24346 v1:172.21.15.193:0/1395329836' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi163=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:01:13.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:12.038722+0000 mgr.y (mgr.14146) 195 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi163=foo.a;count:1 2024-02-19T21:01:13.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:12.078214+0000 mgr.y (mgr.14146) 196 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi163=foo.a;count:1 2024-02-19T21:01:13.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:12.098039+0000 mgr.y (mgr.14146) 197 : cephadm [INF] Deploying daemon rgw.foo.a on smithi163 2024-02-19T21:01:13.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:12.241085+0000 mon.a (mon.0) 551 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-19T21:01:13.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:11.656734+0000 mgr.y (mgr.14146) 193 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:13.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:11.764788+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:01:13.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:11.764862+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:01:13.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:12.037134+0000 mgr.y (mgr.14146) 194 : audit [DBG] from='client.24346 v1:172.21.15.193:0/1395329836' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi163=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:01:13.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:12.038722+0000 mgr.y (mgr.14146) 195 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi163=foo.a;count:1 2024-02-19T21:01:13.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:12.078214+0000 mgr.y (mgr.14146) 196 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi163=foo.a;count:1 2024-02-19T21:01:13.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:12.098039+0000 mgr.y (mgr.14146) 197 : cephadm [INF] Deploying daemon rgw.foo.a on smithi163 2024-02-19T21:01:13.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:12.241085+0000 mon.a (mon.0) 551 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-19T21:01:13.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:11.656734+0000 mgr.y (mgr.14146) 193 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:13.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:11.764788+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:01:13.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:11.764862+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:01:13.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:12.037134+0000 mgr.y (mgr.14146) 194 : audit [DBG] from='client.24346 v1:172.21.15.193:0/1395329836' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi163=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:01:13.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:12.038722+0000 mgr.y (mgr.14146) 195 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi163=foo.a;count:1 2024-02-19T21:01:13.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:12.078214+0000 mgr.y (mgr.14146) 196 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi163=foo.a;count:1 2024-02-19T21:01:13.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:12.098039+0000 mgr.y (mgr.14146) 197 : cephadm [INF] Deploying daemon rgw.foo.a on smithi163 2024-02-19T21:01:13.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:12.241085+0000 mon.a (mon.0) 551 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2024-02-19T21:01:13.855 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Feb 19 21:01:13 smithi163 systemd[1]: Starting Ceph rgw.foo.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:01:14.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.539+0000 7f06867c0700 1 -- 172.21.15.193:0/3950374869 >> v1:172.21.15.193:6789/0 conn(0x7f0680061180 legacy=0x7f06800fa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:14.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.539+0000 7f06867c0700 1 -- 172.21.15.193:0/3950374869 shutdown_connections 2024-02-19T21:01:14.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.539+0000 7f06867c0700 1 -- 172.21.15.193:0/3950374869 >> 172.21.15.193:0/3950374869 conn(0x7f06800f1e60 msgr2=0x7f06800f42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:14.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.540+0000 7f06867c0700 1 -- 172.21.15.193:0/3950374869 shutdown_connections 2024-02-19T21:01:14.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.540+0000 7f06867c0700 1 -- 172.21.15.193:0/3950374869 wait complete. 2024-02-19T21:01:14.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.540+0000 7f06867c0700 1 librados: starting msgr at 2024-02-19T21:01:14.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.540+0000 7f06867c0700 1 librados: starting objecter 2024-02-19T21:01:14.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.540+0000 7f06867c0700 1 Processor -- start 2024-02-19T21:01:14.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.540+0000 7f06867c0700 1 -- start start 2024-02-19T21:01:14.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.540+0000 7f06867c0700 1 librados: setting wanted keys 2024-02-19T21:01:14.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.540+0000 7f06867c0700 1 librados: calling monclient init 2024-02-19T21:01:14.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.540+0000 7f06867c0700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0680114cd0 con 0x7f0680061180 2024-02-19T21:01:14.531 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.540+0000 7f06867c0700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0680115eb0 con 0x7f06800ff910 2024-02-19T21:01:14.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.540+0000 7f06867c0700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f06801170b0 con 0x7f06800fbbe0 2024-02-19T21:01:14.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.540+0000 7f0684d5d700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f06800ff910 0x7f0680113390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:53196/0 (socket says 172.21.15.193:53196) 2024-02-19T21:01:14.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.540+0000 7f0684d5d700 1 -- 172.21.15.193:0/1319936461 learned_addr learned my addr 172.21.15.193:0/1319936461 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T21:01:14.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.540+0000 7f067dffb700 10 client.?.objecter ms_handle_connect 0x7f06800ff910 2024-02-19T21:01:14.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.540+0000 7f067dffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:14.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 10 client.?.objecter ms_handle_connect 0x7f06800fbbe0 2024-02-19T21:01:14.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:14.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 10 client.?.objecter ms_handle_connect 0x7f0680061180 2024-02-19T21:01:14.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:14.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535092348 0 0) 0x7f0680115eb0 con 0x7f06800ff910 2024-02-19T21:01:14.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f066c003630 con 0x7f06800ff910 2024-02-19T21:01:14.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065136369 0 0) 0x7f06801170b0 con 0x7f06800fbbe0 2024-02-19T21:01:14.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0680115eb0 con 0x7f06800fbbe0 2024-02-19T21:01:14.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1606664538 0 0) 0x7f066c003630 con 0x7f06800ff910 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06801170b0 con 0x7f06800ff910 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0678002c90 con 0x7f06800ff910 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2562167615 0 0) 0x7f0680115eb0 con 0x7f06800fbbe0 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f066c003630 con 0x7f06800fbbe0 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0668001800 con 0x7f06800fbbe0 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 20126983 0 0) 0x7f06801170b0 con 0x7f06800ff910 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 >> v1:172.21.15.163:6790/0 conn(0x7f06800fbbe0 legacy=0x7f068010fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 >> v1:172.21.15.163:6789/0 conn(0x7f0680061180 legacy=0x7f06800f7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06801182b0 con 0x7f06800ff910 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f06867c0700 1 -- 172.21.15.193:0/1319936461 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06801172a0 con 0x7f06800ff910 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f06867c0700 10 client.24361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f06867c0700 1 -- 172.21.15.193:0/1319936461 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0680117790 con 0x7f06800ff910 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f06867c0700 1 librados: init done 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f0678001b50 con 0x7f06800ff910 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.541+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0678005e10 con 0x7f06800ff910 2024-02-19T21:01:14.533 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.542+0000 7f06867c0700 1 -- 172.21.15.193:0/1319936461 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0660004f20 con 0x7f06800ff910 2024-02-19T21:01:14.534 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.542+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 7720783 0 0) 0x7f06780016b0 con 0x7f06800ff910 2024-02-19T21:01:14.534 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.543+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(56..56 src has 1..56) v4 ==== 4054+0+0 (unknown 3211146260 0 0) 0x7f0678077fd0 con 0x7f06800ff910 2024-02-19T21:01:14.534 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.543+0000 7f067dffb700 10 client.24361.objecter ms_dispatch 0x7f0680001550 osd_map(56..56 src has 1..56) v4 2024-02-19T21:01:14.534 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.543+0000 7f067dffb700 3 client.24361.objecter handle_osd_map got epochs [56,56] > 0 2024-02-19T21:01:14.534 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.543+0000 7f067dffb700 3 client.24361.objecter handle_osd_map decoding full epoch 56 2024-02-19T21:01:14.535 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.543+0000 7f067dffb700 20 client.24361.objecter dump_active .. 0 homeless 2024-02-19T21:01:14.535 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.543+0000 7f067dffb700 10 client.24361.objecter ms_handle_connect 0x7f066c063bf0 2024-02-19T21:01:14.536 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.547+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0678048890 con 0x7f06800ff910 2024-02-19T21:01:14.698 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:14.709+0000 7f06867c0700 1 -- 172.21.15.193:0/1319936461 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7f0660005750 con 0x7f06800ff910 2024-02-19T21:01:14.736 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Feb 19 21:01:14 smithi163 bash[65194]: d6cb2c9348ecc871942f7389e0d250dd48de374db749e9d8576f560bd404ed7f 2024-02-19T21:01:14.736 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Feb 19 21:01:14 smithi163 systemd[1]: Started Ceph rgw.foo.a for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T21:01:15.244 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:15.255+0000 7f067dffb700 1 -- 172.21.15.193:0/1319936461 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'iscsi' created v57) v1 ==== 154+0+0 (unknown 3423797607 0 0) 0x7f067804a7c0 con 0x7f06800ff910 2024-02-19T21:01:15.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:15.257+0000 7f06867c0700 10 librados: watch_flush enter 2024-02-19T21:01:15.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:15.257+0000 7f06867c0700 10 librados: watch_flush exit 2024-02-19T21:01:15.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:15.257+0000 7f06867c0700 20 client.24361.objecter shutdown clearing up homeless session... 2024-02-19T21:01:15.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:15.257+0000 7f06867c0700 10 client.24361.objecter successfully canceled tick 2024-02-19T21:01:15.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:15.257+0000 7f06867c0700 1 -- 172.21.15.193:0/1319936461 >> v1:172.21.15.163:6800/2735092549 conn(0x7f066c063bf0 legacy=0x7f066c0660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:15.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:15.257+0000 7f06867c0700 1 -- 172.21.15.193:0/1319936461 >> v1:172.21.15.193:6789/0 conn(0x7f06800ff910 legacy=0x7f0680113390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:15.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:15.257+0000 7f06867c0700 1 -- 172.21.15.193:0/1319936461 shutdown_connections 2024-02-19T21:01:15.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:15.257+0000 7f06867c0700 1 -- 172.21.15.193:0/1319936461 >> 172.21.15.193:0/1319936461 conn(0x7f06800f1e60 msgr2=0x7f0680101dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:15.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:15.257+0000 7f06867c0700 1 -- 172.21.15.193:0/1319936461 shutdown_connections 2024-02-19T21:01:15.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:15.257+0000 7f06867c0700 1 -- 172.21.15.193:0/1319936461 wait complete. 2024-02-19T21:01:15.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:15.257+0000 7f06867c0700 1 librados: shutdown 2024-02-19T21:01:15.247 INFO:teuthology.orchestra.run.smithi193.stderr:pool 'iscsi' created 2024-02-19T21:01:15.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:13.657051+0000 mgr.y (mgr.14146) 198 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:15.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:14.665743+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:15.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:14.666790+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:15.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:14.710589+0000 mon.b (mon.1) 20 : audit [INF] from='client.? v1:172.21.15.193:0/1319936461' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-19T21:01:15.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:14.711092+0000 mon.a (mon.0) 554 : audit [INF] from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-19T21:01:15.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:14.965937+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:15.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:13.657051+0000 mgr.y (mgr.14146) 198 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:15.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:14.665743+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:15.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:14.666790+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:15.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:14.710589+0000 mon.b (mon.1) 20 : audit [INF] from='client.? v1:172.21.15.193:0/1319936461' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-19T21:01:15.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:14.711092+0000 mon.a (mon.0) 554 : audit [INF] from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-19T21:01:15.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:14.965937+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:15.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:13.657051+0000 mgr.y (mgr.14146) 198 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:15.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:14.665743+0000 mon.a (mon.0) 552 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:15.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:14.666790+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:15.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:14.710589+0000 mon.b (mon.1) 20 : audit [INF] from='client.? v1:172.21.15.193:0/1319936461' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-19T21:01:15.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:14.711092+0000 mon.a (mon.0) 554 : audit [INF] from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-19T21:01:15.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:14.965937+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:15.875 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool application enable iscsi rbd 2024-02-19T21:01:16.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:15.254808+0000 mon.a (mon.0) 556 : audit [INF] from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-19T21:01:16.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:15.254897+0000 mon.a (mon.0) 557 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-19T21:01:16.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:15.256149+0000 mon.c (mon.2) 13 : audit [INF] from='client.? v1:172.21.15.163:0/2750222700' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-19T21:01:16.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:15.256562+0000 mon.a (mon.0) 558 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-19T21:01:16.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:15.965366+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:16.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:15.966199+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:16.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:15.966781+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:16.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:15.254808+0000 mon.a (mon.0) 556 : audit [INF] from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-19T21:01:16.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:15.254897+0000 mon.a (mon.0) 557 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-19T21:01:16.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:15.256149+0000 mon.c (mon.2) 13 : audit [INF] from='client.? v1:172.21.15.163:0/2750222700' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-19T21:01:16.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:15.256562+0000 mon.a (mon.0) 558 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-19T21:01:16.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:15.965366+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:16.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:15.966199+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:16.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:15.966781+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:16.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:15.254808+0000 mon.a (mon.0) 556 : audit [INF] from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-19T21:01:16.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:15.254897+0000 mon.a (mon.0) 557 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2024-02-19T21:01:16.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:15.256149+0000 mon.c (mon.2) 13 : audit [INF] from='client.? v1:172.21.15.163:0/2750222700' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-19T21:01:16.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:15.256562+0000 mon.a (mon.0) 558 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-19T21:01:16.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:15.965366+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:16.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:15.966199+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:16.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:15.966781+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:17.466 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:15.657410+0000 mgr.y (mgr.14146) 199 : cluster [DBG] pgmap v190: 36 pgs: 5 creating+peering, 1 active+clean, 30 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:17.466 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:15.970709+0000 mgr.y (mgr.14146) 200 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-19T21:01:17.466 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:16.263540+0000 mon.a (mon.0) 562 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-19T21:01:17.466 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:16.263708+0000 mon.a (mon.0) 563 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-19T21:01:17.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.690+0000 7fca87942700 1 -- 172.21.15.193:0/1601967082 >> v1:172.21.15.193:6789/0 conn(0x7fca800fe820 legacy=0x7fca80100ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:17.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.690+0000 7fca87942700 1 -- 172.21.15.193:0/1601967082 shutdown_connections 2024-02-19T21:01:17.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.690+0000 7fca87942700 1 -- 172.21.15.193:0/1601967082 >> 172.21.15.193:0/1601967082 conn(0x7fca800f1e80 msgr2=0x7fca800f42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:17.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.691+0000 7fca87942700 1 -- 172.21.15.193:0/1601967082 shutdown_connections 2024-02-19T21:01:17.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.691+0000 7fca87942700 1 -- 172.21.15.193:0/1601967082 wait complete. 2024-02-19T21:01:17.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.691+0000 7fca87942700 1 librados: starting msgr at 2024-02-19T21:01:17.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.691+0000 7fca87942700 1 librados: starting objecter 2024-02-19T21:01:17.683 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.691+0000 7fca87942700 1 Processor -- start 2024-02-19T21:01:17.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.691+0000 7fca87942700 1 -- start start 2024-02-19T21:01:17.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.691+0000 7fca87942700 1 librados: setting wanted keys 2024-02-19T21:01:17.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.691+0000 7fca87942700 1 librados: calling monclient init 2024-02-19T21:01:17.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.691+0000 7fca87942700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fca80108430 con 0x7fca800f6720 2024-02-19T21:01:17.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.691+0000 7fca87942700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fca80113c90 con 0x7fca800fe820 2024-02-19T21:01:17.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.691+0000 7fca87942700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fca80114e70 con 0x7fca800fabf0 2024-02-19T21:01:17.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.691+0000 7fca85edf700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fca800fe820 0x7fca80111560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:35920/0 (socket says 172.21.15.193:35920) 2024-02-19T21:01:17.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.691+0000 7fca85edf700 1 -- 172.21.15.193:0/1206827204 learned_addr learned my addr 172.21.15.193:0/1206827204 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T21:01:17.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.691+0000 7fca767fc700 10 client.?.objecter ms_handle_connect 0x7fca800fe820 2024-02-19T21:01:17.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.691+0000 7fca767fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:17.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 10 client.?.objecter ms_handle_connect 0x7fca800fabf0 2024-02-19T21:01:17.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:17.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 10 client.?.objecter ms_handle_connect 0x7fca800f6720 2024-02-19T21:01:17.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:17.684 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225796279 0 0) 0x7fca80113c90 con 0x7fca800fe820 2024-02-19T21:01:17.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca6c003630 con 0x7fca800fe820 2024-02-19T21:01:17.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714755194 0 0) 0x7fca80114e70 con 0x7fca800fabf0 2024-02-19T21:01:17.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca80113c90 con 0x7fca800fabf0 2024-02-19T21:01:17.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409111846 0 0) 0x7fca80108430 con 0x7fca800f6720 2024-02-19T21:01:17.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca80114e70 con 0x7fca800f6720 2024-02-19T21:01:17.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3043197565 0 0) 0x7fca6c003630 con 0x7fca800fe820 2024-02-19T21:01:17.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca80108430 con 0x7fca800fe820 2024-02-19T21:01:17.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fca780023e0 con 0x7fca800fe820 2024-02-19T21:01:17.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3083975044 0 0) 0x7fca80113c90 con 0x7fca800fabf0 2024-02-19T21:01:17.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca6c003630 con 0x7fca800fabf0 2024-02-19T21:01:17.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fca7c0018c0 con 0x7fca800fabf0 2024-02-19T21:01:17.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 175706610 0 0) 0x7fca80114e70 con 0x7fca800f6720 2024-02-19T21:01:17.685 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca80113c90 con 0x7fca800f6720 2024-02-19T21:01:17.686 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 490425856 0 0) 0x7fca80108430 con 0x7fca800fe820 2024-02-19T21:01:17.686 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 >> v1:172.21.15.163:6790/0 conn(0x7fca800fabf0 legacy=0x7fca8010de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:17.686 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 >> v1:172.21.15.163:6789/0 conn(0x7fca800f6720 legacy=0x7fca80107860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:17.686 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca80116050 con 0x7fca800fe820 2024-02-19T21:01:17.686 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca87942700 1 -- 172.21.15.193:0/1206827204 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca80115060 con 0x7fca800fe820 2024-02-19T21:01:17.686 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca87942700 10 client.24391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:01:17.686 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca87942700 1 -- 172.21.15.193:0/1206827204 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca801154d0 con 0x7fca800fe820 2024-02-19T21:01:17.686 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca87942700 1 librados: init done 2024-02-19T21:01:17.686 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fca78002680 con 0x7fca800fe820 2024-02-19T21:01:17.686 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.692+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fca78003c40 con 0x7fca800fe820 2024-02-19T21:01:17.687 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.693+0000 7fca87942700 1 -- 172.21.15.193:0/1206827204 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca64004f20 con 0x7fca800fe820 2024-02-19T21:01:17.687 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.693+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 7720783 0 0) 0x7fca78003500 con 0x7fca800fe820 2024-02-19T21:01:17.687 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.694+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(59..59 src has 1..59) v4 ==== 5111+0+0 (unknown 1929349024 0 0) 0x7fca78078680 con 0x7fca800fe820 2024-02-19T21:01:17.687 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.694+0000 7fca767fc700 10 client.24391.objecter ms_dispatch 0x7fca80001550 osd_map(59..59 src has 1..59) v4 2024-02-19T21:01:17.687 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.694+0000 7fca767fc700 3 client.24391.objecter handle_osd_map got epochs [59,59] > 0 2024-02-19T21:01:17.687 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.694+0000 7fca767fc700 3 client.24391.objecter handle_osd_map decoding full epoch 59 2024-02-19T21:01:17.687 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.694+0000 7fca767fc700 20 client.24391.objecter dump_active .. 0 homeless 2024-02-19T21:01:17.687 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.694+0000 7fca767fc700 10 client.24391.objecter ms_handle_connect 0x7fca6c063cb0 2024-02-19T21:01:17.687 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.698+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fca78048b80 con 0x7fca800fe820 2024-02-19T21:01:17.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:15.657410+0000 mgr.y (mgr.14146) 199 : cluster [DBG] pgmap v190: 36 pgs: 5 creating+peering, 1 active+clean, 30 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:17.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:15.970709+0000 mgr.y (mgr.14146) 200 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-19T21:01:17.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:16.263540+0000 mon.a (mon.0) 562 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-19T21:01:17.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:16.263708+0000 mon.a (mon.0) 563 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-19T21:01:17.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:15.657410+0000 mgr.y (mgr.14146) 199 : cluster [DBG] pgmap v190: 36 pgs: 5 creating+peering, 1 active+clean, 30 unknown; 0 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:17.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:15.970709+0000 mgr.y (mgr.14146) 200 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-02-19T21:01:17.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:16.263540+0000 mon.a (mon.0) 562 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-19T21:01:17.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:16.263708+0000 mon.a (mon.0) 563 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2024-02-19T21:01:17.850 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:17.861+0000 7fca87942700 1 -- 172.21.15.193:0/1206827204 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"} v 0) v1 -- 0x7fca64005750 con 0x7fca800fe820 2024-02-19T21:01:18.265 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:18.276+0000 7fca767fc700 1 -- 172.21.15.193:0/1206827204 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]=0 enabled application 'rbd' on pool 'iscsi' v60) v1 ==== 147+0+0 (unknown 2906445475 0 0) 0x7fca7804aab0 con 0x7fca800fe820 2024-02-19T21:01:18.268 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:18.279+0000 7fca87942700 10 librados: watch_flush enter 2024-02-19T21:01:18.270 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:18.279+0000 7fca87942700 10 librados: watch_flush exit 2024-02-19T21:01:18.270 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:18.279+0000 7fca87942700 20 client.24391.objecter shutdown clearing up homeless session... 2024-02-19T21:01:18.270 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:18.279+0000 7fca87942700 10 client.24391.objecter successfully canceled tick 2024-02-19T21:01:18.270 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:18.279+0000 7fca87942700 1 -- 172.21.15.193:0/1206827204 >> v1:172.21.15.163:6800/2735092549 conn(0x7fca6c063cb0 legacy=0x7fca6c066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:18.270 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:18.279+0000 7fca87942700 1 -- 172.21.15.193:0/1206827204 >> v1:172.21.15.193:6789/0 conn(0x7fca800fe820 legacy=0x7fca80111560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:18.270 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:18.280+0000 7fca87942700 1 -- 172.21.15.193:0/1206827204 shutdown_connections 2024-02-19T21:01:18.270 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:18.280+0000 7fca87942700 1 -- 172.21.15.193:0/1206827204 >> 172.21.15.193:0/1206827204 conn(0x7fca800f1e80 msgr2=0x7fca80102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:18.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:18.282+0000 7fca87942700 1 -- 172.21.15.193:0/1206827204 shutdown_connections 2024-02-19T21:01:18.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:18.282+0000 7fca87942700 1 -- 172.21.15.193:0/1206827204 wait complete. 2024-02-19T21:01:18.271 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:18.282+0000 7fca87942700 1 librados: shutdown 2024-02-19T21:01:18.271 INFO:teuthology.orchestra.run.smithi193.stderr:enabled application 'rbd' on pool 'iscsi' 2024-02-19T21:01:18.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:17.266280+0000 mon.a (mon.0) 564 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-19T21:01:18.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:17.269087+0000 mon.c (mon.2) 14 : audit [INF] from='client.? v1:172.21.15.163:0/2202902387' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-19T21:01:18.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:17.269571+0000 mon.a (mon.0) 565 : audit [INF] from='client.34382 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-19T21:01:18.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:17.271946+0000 mon.c (mon.2) 15 : audit [INF] from='client.? v1:172.21.15.163:0/2750222700' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-19T21:01:18.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:17.272515+0000 mon.a (mon.0) 566 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-19T21:01:18.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:17.863114+0000 mon.b (mon.1) 21 : audit [INF] from='client.? v1:172.21.15.193:0/1206827204' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-19T21:01:18.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:17.863743+0000 mon.a (mon.0) 567 : audit [INF] from='client.24391 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-19T21:01:18.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:17.266280+0000 mon.a (mon.0) 564 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-19T21:01:18.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:17.269087+0000 mon.c (mon.2) 14 : audit [INF] from='client.? v1:172.21.15.163:0/2202902387' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-19T21:01:18.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:17.269571+0000 mon.a (mon.0) 565 : audit [INF] from='client.34382 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-19T21:01:18.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:17.271946+0000 mon.c (mon.2) 15 : audit [INF] from='client.? v1:172.21.15.163:0/2750222700' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-19T21:01:18.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:17.272515+0000 mon.a (mon.0) 566 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-19T21:01:18.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:17.863114+0000 mon.b (mon.1) 21 : audit [INF] from='client.? v1:172.21.15.193:0/1206827204' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-19T21:01:18.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:17.863743+0000 mon.a (mon.0) 567 : audit [INF] from='client.24391 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-19T21:01:18.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:17.266280+0000 mon.a (mon.0) 564 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2024-02-19T21:01:18.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:17.269087+0000 mon.c (mon.2) 14 : audit [INF] from='client.? v1:172.21.15.163:0/2202902387' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-19T21:01:18.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:17.269571+0000 mon.a (mon.0) 565 : audit [INF] from='client.34382 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-19T21:01:18.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:17.271946+0000 mon.c (mon.2) 15 : audit [INF] from='client.? v1:172.21.15.163:0/2750222700' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-19T21:01:18.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:17.272515+0000 mon.a (mon.0) 566 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-19T21:01:18.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:17.863114+0000 mon.b (mon.1) 21 : audit [INF] from='client.? v1:172.21.15.193:0/1206827204' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-19T21:01:18.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:17.863743+0000 mon.a (mon.0) 567 : audit [INF] from='client.24391 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2024-02-19T21:01:18.979 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch apply iscsi iscsi user password --placement '1;smithi193=iscsi.a' 2024-02-19T21:01:19.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:17.657859+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v193: 68 pgs: 5 creating+peering, 9 active+clean, 54 unknown; 92 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:19.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:18.276828+0000 mon.a (mon.0) 568 : audit [INF] from='client.34382 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-19T21:01:19.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:18.277025+0000 mon.a (mon.0) 569 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-19T21:01:19.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:18.277103+0000 mon.a (mon.0) 570 : audit [INF] from='client.24391 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2024-02-19T21:01:19.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:18.277171+0000 mon.a (mon.0) 571 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-19T21:01:19.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:17.657859+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v193: 68 pgs: 5 creating+peering, 9 active+clean, 54 unknown; 92 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:19.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:18.276828+0000 mon.a (mon.0) 568 : audit [INF] from='client.34382 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-19T21:01:19.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:18.277025+0000 mon.a (mon.0) 569 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-19T21:01:19.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:18.277103+0000 mon.a (mon.0) 570 : audit [INF] from='client.24391 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2024-02-19T21:01:19.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:18.277171+0000 mon.a (mon.0) 571 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-19T21:01:19.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:17.657859+0000 mgr.y (mgr.14146) 201 : cluster [DBG] pgmap v193: 68 pgs: 5 creating+peering, 9 active+clean, 54 unknown; 92 B data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:19.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:18.276828+0000 mon.a (mon.0) 568 : audit [INF] from='client.34382 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-19T21:01:19.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:18.277025+0000 mon.a (mon.0) 569 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-19T21:01:19.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:18.277103+0000 mon.a (mon.0) 570 : audit [INF] from='client.24391 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2024-02-19T21:01:19.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:18.277171+0000 mon.a (mon.0) 571 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2024-02-19T21:01:20.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:19.276919+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-19T21:01:20.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:19.280923+0000 mon.c (mon.2) 16 : audit [INF] from='client.? v1:172.21.15.163:0/2202902387' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-19T21:01:20.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:19.281116+0000 mon.c (mon.2) 17 : audit [INF] from='client.? v1:172.21.15.163:0/2750222700' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-19T21:01:20.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:19.285111+0000 mon.a (mon.0) 573 : audit [INF] from='client.34382 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-19T21:01:20.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:19.285538+0000 mon.a (mon.0) 574 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-19T21:01:20.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:20.277511+0000 mon.a (mon.0) 575 : audit [INF] from='client.34382 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-19T21:01:20.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:20.277610+0000 mon.a (mon.0) 576 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-19T21:01:20.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:20.277682+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-19T21:01:20.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:19.276919+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-19T21:01:20.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:19.280923+0000 mon.c (mon.2) 16 : audit [INF] from='client.? v1:172.21.15.163:0/2202902387' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-19T21:01:20.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:19.281116+0000 mon.c (mon.2) 17 : audit [INF] from='client.? v1:172.21.15.163:0/2750222700' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-19T21:01:20.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:19.285111+0000 mon.a (mon.0) 573 : audit [INF] from='client.34382 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-19T21:01:20.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:19.285538+0000 mon.a (mon.0) 574 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-19T21:01:20.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:20.277511+0000 mon.a (mon.0) 575 : audit [INF] from='client.34382 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-19T21:01:20.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:20.277610+0000 mon.a (mon.0) 576 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-19T21:01:20.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:20.277682+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-19T21:01:20.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:19.276919+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2024-02-19T21:01:20.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:19.280923+0000 mon.c (mon.2) 16 : audit [INF] from='client.? v1:172.21.15.163:0/2202902387' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-19T21:01:20.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:19.281116+0000 mon.c (mon.2) 17 : audit [INF] from='client.? v1:172.21.15.163:0/2750222700' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-19T21:01:20.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:19.285111+0000 mon.a (mon.0) 573 : audit [INF] from='client.34382 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-19T21:01:20.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:19.285538+0000 mon.a (mon.0) 574 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-19T21:01:20.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:20.277511+0000 mon.a (mon.0) 575 : audit [INF] from='client.34382 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-19T21:01:20.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:20.277610+0000 mon.a (mon.0) 576 : audit [INF] from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-19T21:01:20.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:20.277682+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2024-02-19T21:01:20.862 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.842+0000 7f9f11e0d700 1 -- 172.21.15.193:0/1473560464 >> v1:172.21.15.193:6789/0 conn(0x7f9f0c0fbc10 legacy=0x7f9f0c0fe060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:20.862 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.842+0000 7f9f11e0d700 1 -- 172.21.15.193:0/1473560464 shutdown_connections 2024-02-19T21:01:20.862 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.842+0000 7f9f11e0d700 1 -- 172.21.15.193:0/1473560464 >> 172.21.15.193:0/1473560464 conn(0x7f9f0c0f1e60 msgr2=0x7f9f0c0f42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:20.862 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.842+0000 7f9f11e0d700 1 -- 172.21.15.193:0/1473560464 shutdown_connections 2024-02-19T21:01:20.862 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.842+0000 7f9f11e0d700 1 -- 172.21.15.193:0/1473560464 wait complete. 2024-02-19T21:01:20.862 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.842+0000 7f9f11e0d700 1 librados: starting msgr at 2024-02-19T21:01:20.862 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.842+0000 7f9f11e0d700 1 librados: starting objecter 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.842+0000 7f9f11e0d700 1 Processor -- start 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.842+0000 7f9f11e0d700 1 -- start start 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.842+0000 7f9f11e0d700 1 librados: setting wanted keys 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.842+0000 7f9f11e0d700 1 librados: calling monclient init 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f11e0d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9f0c1084f0 con 0x7f9f0c0ff920 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f11e0d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9f0c113d50 con 0x7f9f0c0fbc10 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f11e0d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9f0c114f30 con 0x7f9f0c0611e0 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f0affd700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f9f0c0fbc10 0x7f9f0c10def0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:35940/0 (socket says 172.21.15.193:35940) 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f0b7fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f9f0c0611e0 0x7f9f0c107920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.193:54658/0 (socket says 172.21.15.193:54658) 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f0affd700 1 -- 172.21.15.193:0/1280132506 learned_addr learned my addr 172.21.15.193:0/1280132506 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 10 client.?.objecter ms_handle_connect 0x7f9f0c0ff920 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 10 client.?.objecter ms_handle_connect 0x7f9f0c0fbc10 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 10 client.?.objecter ms_handle_connect 0x7f9f0c0611e0 2024-02-19T21:01:20.863 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:20.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274790929 0 0) 0x7f9f0c114f30 con 0x7f9f0c0611e0 2024-02-19T21:01:20.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef40033f0 con 0x7f9f0c0611e0 2024-02-19T21:01:20.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025489227 0 0) 0x7f9f0c1084f0 con 0x7f9f0c0ff920 2024-02-19T21:01:20.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f0c114f30 con 0x7f9f0c0ff920 2024-02-19T21:01:20.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672722304 0 0) 0x7f9f0c113d50 con 0x7f9f0c0fbc10 2024-02-19T21:01:20.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f0c1084f0 con 0x7f9f0c0fbc10 2024-02-19T21:01:20.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4018618760 0 0) 0x7f9ef40033f0 con 0x7f9f0c0611e0 2024-02-19T21:01:20.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f0c113d50 con 0x7f9f0c0611e0 2024-02-19T21:01:20.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9f000019d0 con 0x7f9f0c0611e0 2024-02-19T21:01:20.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2797153048 0 0) 0x7f9f0c1084f0 con 0x7f9f0c0fbc10 2024-02-19T21:01:20.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ef40033f0 con 0x7f9f0c0fbc10 2024-02-19T21:01:20.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9efc002d60 con 0x7f9f0c0fbc10 2024-02-19T21:01:20.864 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 388775291 0 0) 0x7f9f0c113d50 con 0x7f9f0c0611e0 2024-02-19T21:01:20.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 >> v1:172.21.15.193:6789/0 conn(0x7f9f0c0fbc10 legacy=0x7f9f0c10def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:20.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 >> v1:172.21.15.163:6789/0 conn(0x7f9f0c0ff920 legacy=0x7f9f0c111620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:20.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c116110 con 0x7f9f0c0611e0 2024-02-19T21:01:20.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f11e0d700 1 -- 172.21.15.193:0/1280132506 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0c113f40 con 0x7f9f0c0611e0 2024-02-19T21:01:20.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f11e0d700 10 client.34406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:01:20.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f11e0d700 1 -- 172.21.15.193:0/1280132506 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f0c114430 con 0x7f9f0c0611e0 2024-02-19T21:01:20.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f11e0d700 1 librados: init done 2024-02-19T21:01:20.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9f000023c0 con 0x7f9f0c0611e0 2024-02-19T21:01:20.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.843+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9f00005a00 con 0x7f9f0c0611e0 2024-02-19T21:01:20.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.844+0000 7f9f11e0d700 1 -- 172.21.15.193:0/1280132506 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef8004f20 con 0x7f9f0c0611e0 2024-02-19T21:01:20.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.845+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 7720783 0 0) 0x7f9f00018850 con 0x7f9f0c0611e0 2024-02-19T21:01:20.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.845+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(62..62 src has 1..62) v4 ==== 5502+0+0 (unknown 719645118 0 0) 0x7f9f00078130 con 0x7f9f0c0611e0 2024-02-19T21:01:20.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.845+0000 7f9f08ff9700 10 client.34406.objecter ms_dispatch 0x7f9f0c001550 osd_map(62..62 src has 1..62) v4 2024-02-19T21:01:20.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.845+0000 7f9f08ff9700 3 client.34406.objecter handle_osd_map got epochs [62,62] > 0 2024-02-19T21:01:20.865 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.845+0000 7f9f08ff9700 3 client.34406.objecter handle_osd_map decoding full epoch 62 2024-02-19T21:01:20.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.845+0000 7f9f08ff9700 20 client.34406.objecter dump_active .. 0 homeless 2024-02-19T21:01:20.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.845+0000 7f9f08ff9700 10 client.34406.objecter ms_handle_connect 0x7f9ef40639b0 2024-02-19T21:01:20.866 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:20.849+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9f00048400 con 0x7f9f0c0611e0 2024-02-19T21:01:21.009 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:21.020+0000 7f9f11e0d700 1 -- 172.21.15.193:0/1280132506 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi193=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ef8000c40 con 0x7f9ef40639b0 2024-02-19T21:01:21.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:21.027+0000 7f9f08ff9700 1 -- 172.21.15.193:0/1280132506 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (unknown 0 0 3282307126) 0x7f9ef8000c40 con 0x7f9ef40639b0 2024-02-19T21:01:21.016 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled iscsi.iscsi update... 2024-02-19T21:01:21.019 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:21.028+0000 7f9f11e0d700 10 librados: watch_flush enter 2024-02-19T21:01:21.019 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:21.028+0000 7f9f11e0d700 10 librados: watch_flush exit 2024-02-19T21:01:21.019 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:21.028+0000 7f9f11e0d700 20 client.34406.objecter shutdown clearing up homeless session... 2024-02-19T21:01:21.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:21.028+0000 7f9f11e0d700 10 client.34406.objecter successfully canceled tick 2024-02-19T21:01:21.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:21.028+0000 7f9f11e0d700 1 -- 172.21.15.193:0/1280132506 >> v1:172.21.15.163:6800/2735092549 conn(0x7f9ef40639b0 legacy=0x7f9ef4065e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:21.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:21.028+0000 7f9f11e0d700 1 -- 172.21.15.193:0/1280132506 >> v1:172.21.15.163:6790/0 conn(0x7f9f0c0611e0 legacy=0x7f9f0c107920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:21.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:21.029+0000 7f9f11e0d700 1 -- 172.21.15.193:0/1280132506 shutdown_connections 2024-02-19T21:01:21.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:21.029+0000 7f9f11e0d700 1 -- 172.21.15.193:0/1280132506 >> 172.21.15.193:0/1280132506 conn(0x7f9f0c0f1e60 msgr2=0x7f9f0c101de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:21.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:21.029+0000 7f9f11e0d700 1 -- 172.21.15.193:0/1280132506 shutdown_connections 2024-02-19T21:01:21.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:21.029+0000 7f9f11e0d700 1 -- 172.21.15.193:0/1280132506 wait complete. 2024-02-19T21:01:21.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:21.029+0000 7f9f11e0d700 1 librados: shutdown 2024-02-19T21:01:21.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:19.658538+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v196: 100 pgs: 13 creating+peering, 31 active+clean, 56 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 855 B/s wr, 3 op/s 2024-02-19T21:01:21.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:21.027777+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:21.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:21.279982+0000 mon.a (mon.0) 579 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-19T21:01:21.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:21.284213+0000 mon.b (mon.1) 22 : audit [INF] from='client.? v1:172.21.15.163:0/1737694266' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-19T21:01:21.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:21.287470+0000 mon.a (mon.0) 580 : audit [INF] from='client.24397 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-19T21:01:21.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:21.287781+0000 mon.a (mon.0) 581 : audit [INF] from='client.24400 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-19T21:01:21.708 DEBUG:teuthology.orchestra.run.smithi193:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@iscsi.iscsi.a.service 2024-02-19T21:01:21.711 INFO:tasks.cephadm:Adding prometheus.a on smithi193 2024-02-19T21:01:21.711 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch apply prometheus '1;smithi193=a' 2024-02-19T21:01:21.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:19.658538+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v196: 100 pgs: 13 creating+peering, 31 active+clean, 56 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 855 B/s wr, 3 op/s 2024-02-19T21:01:21.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:21.027777+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:21.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:21.279982+0000 mon.a (mon.0) 579 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-19T21:01:21.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:21.284213+0000 mon.b (mon.1) 22 : audit [INF] from='client.? v1:172.21.15.163:0/1737694266' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-19T21:01:21.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:21.287470+0000 mon.a (mon.0) 580 : audit [INF] from='client.24397 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-19T21:01:21.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:21.287781+0000 mon.a (mon.0) 581 : audit [INF] from='client.24400 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-19T21:01:21.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:19.658538+0000 mgr.y (mgr.14146) 202 : cluster [DBG] pgmap v196: 100 pgs: 13 creating+peering, 31 active+clean, 56 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 855 B/s wr, 3 op/s 2024-02-19T21:01:21.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:21.027777+0000 mon.a (mon.0) 578 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:21.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:21.279982+0000 mon.a (mon.0) 579 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2024-02-19T21:01:21.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:21.284213+0000 mon.b (mon.1) 22 : audit [INF] from='client.? v1:172.21.15.163:0/1737694266' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-19T21:01:21.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:21.287470+0000 mon.a (mon.0) 580 : audit [INF] from='client.24397 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-19T21:01:21.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:21.287781+0000 mon.a (mon.0) 581 : audit [INF] from='client.24400 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-19T21:01:21.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:-- Logs begin at Mon 2024-02-19 20:44:17 UTC. -- 2024-02-19T21:01:22.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:21.022230+0000 mgr.y (mgr.14146) 203 : audit [DBG] from='client.34406 v1:172.21.15.193:0/1280132506' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi193=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:01:22.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:21.024100+0000 mgr.y (mgr.14146) 204 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi193=iscsi.a;count:1 2024-02-19T21:01:22.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:21.287070+0000 mon.b (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.163:0/3997262145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-19T21:01:22.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:21.658999+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v199: 132 pgs: 17 creating+peering, 49 active+clean, 66 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-19T21:01:22.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:22.282338+0000 mon.a (mon.0) 582 : audit [INF] from='client.24397 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-19T21:01:22.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:22.282456+0000 mon.a (mon.0) 583 : audit [INF] from='client.24400 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-19T21:01:22.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:22.282529+0000 mon.a (mon.0) 584 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-19T21:01:22.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:22.287457+0000 mon.b (mon.1) 24 : audit [INF] from='client.? v1:172.21.15.163:0/1737694266' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-19T21:01:22.452 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:22.288142+0000 mon.a (mon.0) 585 : audit [INF] from='client.24397 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-19T21:01:22.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:21.022230+0000 mgr.y (mgr.14146) 203 : audit [DBG] from='client.34406 v1:172.21.15.193:0/1280132506' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi193=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:01:22.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:21.024100+0000 mgr.y (mgr.14146) 204 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi193=iscsi.a;count:1 2024-02-19T21:01:22.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:21.287070+0000 mon.b (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.163:0/3997262145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-19T21:01:22.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:21.658999+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v199: 132 pgs: 17 creating+peering, 49 active+clean, 66 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-19T21:01:22.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:22.282338+0000 mon.a (mon.0) 582 : audit [INF] from='client.24397 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-19T21:01:22.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:22.282456+0000 mon.a (mon.0) 583 : audit [INF] from='client.24400 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-19T21:01:22.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:22.282529+0000 mon.a (mon.0) 584 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-19T21:01:22.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:22.287457+0000 mon.b (mon.1) 24 : audit [INF] from='client.? v1:172.21.15.163:0/1737694266' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-19T21:01:22.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:22.288142+0000 mon.a (mon.0) 585 : audit [INF] from='client.24397 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-19T21:01:22.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:21.022230+0000 mgr.y (mgr.14146) 203 : audit [DBG] from='client.34406 v1:172.21.15.193:0/1280132506' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi193=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:01:22.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:21.024100+0000 mgr.y (mgr.14146) 204 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi193=iscsi.a;count:1 2024-02-19T21:01:22.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:21.287070+0000 mon.b (mon.1) 23 : audit [INF] from='client.? v1:172.21.15.163:0/3997262145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-19T21:01:22.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:21.658999+0000 mgr.y (mgr.14146) 205 : cluster [DBG] pgmap v199: 132 pgs: 17 creating+peering, 49 active+clean, 66 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-19T21:01:22.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:22.282338+0000 mon.a (mon.0) 582 : audit [INF] from='client.24397 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-19T21:01:22.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:22.282456+0000 mon.a (mon.0) 583 : audit [INF] from='client.24400 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-19T21:01:22.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:22.282529+0000 mon.a (mon.0) 584 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2024-02-19T21:01:22.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:22.287457+0000 mon.b (mon.1) 24 : audit [INF] from='client.? v1:172.21.15.163:0/1737694266' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-19T21:01:22.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:22.288142+0000 mon.a (mon.0) 585 : audit [INF] from='client.24397 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-19T21:01:23.295 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:22.290210+0000 mon.b (mon.1) 25 : audit [INF] from='client.? v1:172.21.15.163:0/3997262145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-19T21:01:23.295 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:22.290851+0000 mon.a (mon.0) 586 : audit [INF] from='client.24400 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-19T21:01:23.295 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:23.284449+0000 mon.a (mon.0) 587 : audit [INF] from='client.24397 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-19T21:01:23.295 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:23.284551+0000 mon.a (mon.0) 588 : audit [INF] from='client.24400 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-19T21:01:23.295 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:23.284616+0000 mon.a (mon.0) 589 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-19T21:01:23.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:22.290210+0000 mon.b (mon.1) 25 : audit [INF] from='client.? v1:172.21.15.163:0/3997262145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-19T21:01:23.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:22.290851+0000 mon.a (mon.0) 586 : audit [INF] from='client.24400 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-19T21:01:23.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:23.284449+0000 mon.a (mon.0) 587 : audit [INF] from='client.24397 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-19T21:01:23.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:23.284551+0000 mon.a (mon.0) 588 : audit [INF] from='client.24400 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-19T21:01:23.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:23.284616+0000 mon.a (mon.0) 589 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-19T21:01:23.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:22.290210+0000 mon.b (mon.1) 25 : audit [INF] from='client.? v1:172.21.15.163:0/3997262145' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-19T21:01:23.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:22.290851+0000 mon.a (mon.0) 586 : audit [INF] from='client.24400 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-19T21:01:23.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:23.284449+0000 mon.a (mon.0) 587 : audit [INF] from='client.24397 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-19T21:01:23.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:23.284551+0000 mon.a (mon.0) 588 : audit [INF] from='client.24400 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-19T21:01:23.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:23.284616+0000 mon.a (mon.0) 589 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2024-02-19T21:01:23.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.789+0000 7fe92f26e700 1 -- 172.21.15.193:0/435156193 >> v1:172.21.15.193:6789/0 conn(0x7fe9280fabf0 legacy=0x7fe9280fcfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:23.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.789+0000 7fe92f26e700 1 -- 172.21.15.193:0/435156193 shutdown_connections 2024-02-19T21:01:23.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.789+0000 7fe92f26e700 1 -- 172.21.15.193:0/435156193 >> 172.21.15.193:0/435156193 conn(0x7fe9280f1e60 msgr2=0x7fe9280f42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:23.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.789+0000 7fe92f26e700 1 -- 172.21.15.193:0/435156193 shutdown_connections 2024-02-19T21:01:23.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.789+0000 7fe92f26e700 1 -- 172.21.15.193:0/435156193 wait complete. 2024-02-19T21:01:23.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe92f26e700 1 librados: starting msgr at 2024-02-19T21:01:23.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe92f26e700 1 librados: starting objecter 2024-02-19T21:01:23.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe92f26e700 1 Processor -- start 2024-02-19T21:01:23.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe92f26e700 1 -- start start 2024-02-19T21:01:23.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe92f26e700 1 librados: setting wanted keys 2024-02-19T21:01:23.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe92f26e700 1 librados: calling monclient init 2024-02-19T21:01:23.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe92f26e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe9280614c0 con 0x7fe9280fabf0 2024-02-19T21:01:23.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe92f26e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe928113af0 con 0x7fe9280fe820 2024-02-19T21:01:23.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe92f26e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe928114cf0 con 0x7fe9280f6720 2024-02-19T21:01:23.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe92d80b700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fe9280fe820 0x7fe928111360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:35954/0 (socket says 172.21.15.193:35954) 2024-02-19T21:01:23.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe92d80b700 1 -- 172.21.15.193:0/1896866144 learned_addr learned my addr 172.21.15.193:0/1896866144 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T21:01:23.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe91e777700 10 client.?.objecter ms_handle_connect 0x7fe9280fe820 2024-02-19T21:01:23.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe91e777700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:23.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453668016 0 0) 0x7fe928113af0 con 0x7fe9280fe820 2024-02-19T21:01:23.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9140038e0 con 0x7fe9280fe820 2024-02-19T21:01:23.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe91e777700 10 client.?.objecter ms_handle_connect 0x7fe9280f6720 2024-02-19T21:01:23.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe91e777700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:23.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe91e777700 10 client.?.objecter ms_handle_connect 0x7fe9280fabf0 2024-02-19T21:01:23.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe91e777700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:23.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982102953 0 0) 0x7fe928114cf0 con 0x7fe9280f6720 2024-02-19T21:01:23.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe928113af0 con 0x7fe9280f6720 2024-02-19T21:01:23.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553522211 0 0) 0x7fe928105c00 con 0x7fe9280fabf0 2024-02-19T21:01:23.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe928114cf0 con 0x7fe9280fabf0 2024-02-19T21:01:23.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.790+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1955465397 0 0) 0x7fe9140038e0 con 0x7fe9280fe820 2024-02-19T21:01:23.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe928105c00 con 0x7fe9280fe820 2024-02-19T21:01:23.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe920001c40 con 0x7fe9280fe820 2024-02-19T21:01:23.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3612375350 0 0) 0x7fe928113af0 con 0x7fe9280f6720 2024-02-19T21:01:23.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9140038e0 con 0x7fe9280f6720 2024-02-19T21:01:23.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe9240019f0 con 0x7fe9280f6720 2024-02-19T21:01:23.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2977721285 0 0) 0x7fe928114cf0 con 0x7fe9280fabf0 2024-02-19T21:01:23.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe928113af0 con 0x7fe9280fabf0 2024-02-19T21:01:23.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe918002b80 con 0x7fe9280fabf0 2024-02-19T21:01:23.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2467399243 0 0) 0x7fe928105c00 con 0x7fe9280fe820 2024-02-19T21:01:23.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 >> v1:172.21.15.163:6790/0 conn(0x7fe9280f6720 legacy=0x7fe92810a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:23.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 >> v1:172.21.15.163:6789/0 conn(0x7fe9280fabf0 legacy=0x7fe92810dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:23.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe928115ef0 con 0x7fe9280fe820 2024-02-19T21:01:23.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe92f26e700 1 -- 172.21.15.193:0/1896866144 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe928114ee0 con 0x7fe9280fe820 2024-02-19T21:01:23.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe92f26e700 10 client.24433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:01:23.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe92f26e700 1 -- 172.21.15.193:0/1896866144 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe928115350 con 0x7fe9280fe820 2024-02-19T21:01:23.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe92f26e700 1 librados: init done 2024-02-19T21:01:23.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fe920002630 con 0x7fe9280fe820 2024-02-19T21:01:23.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.791+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe920005d00 con 0x7fe9280fe820 2024-02-19T21:01:23.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.792+0000 7fe92f26e700 1 -- 172.21.15.193:0/1896866144 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90c004f20 con 0x7fe9280fe820 2024-02-19T21:01:23.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.792+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 14) v1 ==== 77030+0+0 (unknown 7720783 0 0) 0x7fe920018c30 con 0x7fe9280fe820 2024-02-19T21:01:23.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.793+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(65..65 src has 1..65) v4 ==== 5884+0+0 (unknown 3811228291 0 0) 0x7fe920078580 con 0x7fe9280fe820 2024-02-19T21:01:23.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.793+0000 7fe91e777700 10 client.24433.objecter ms_dispatch 0x7fe928001550 osd_map(65..65 src has 1..65) v4 2024-02-19T21:01:23.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.793+0000 7fe91e777700 3 client.24433.objecter handle_osd_map got epochs [65,65] > 0 2024-02-19T21:01:23.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.793+0000 7fe91e777700 3 client.24433.objecter handle_osd_map decoding full epoch 65 2024-02-19T21:01:23.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.793+0000 7fe91e777700 20 client.24433.objecter dump_active .. 0 homeless 2024-02-19T21:01:23.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.793+0000 7fe91e777700 10 client.24433.objecter ms_handle_connect 0x7fe914063c10 2024-02-19T21:01:23.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.797+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe920048730 con 0x7fe9280fe820 2024-02-19T21:01:23.978 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.989+0000 7fe92f26e700 1 -- 172.21.15.193:0/1896866144 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi193=a", "target": ["mon-mgr", ""]}) v1 -- 0x7fe90c000b90 con 0x7fe914063c10 2024-02-19T21:01:23.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.995+0000 7fe91e777700 1 -- 172.21.15.193:0/1896866144 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (unknown 0 0 1342662408) 0x7fe90c000b90 con 0x7fe914063c10 2024-02-19T21:01:23.984 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled prometheus update... 2024-02-19T21:01:23.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.997+0000 7fe92f26e700 10 librados: watch_flush enter 2024-02-19T21:01:23.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.997+0000 7fe92f26e700 10 librados: watch_flush exit 2024-02-19T21:01:23.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.997+0000 7fe92f26e700 20 client.24433.objecter shutdown clearing up homeless session... 2024-02-19T21:01:23.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.997+0000 7fe92f26e700 10 client.24433.objecter successfully canceled tick 2024-02-19T21:01:23.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.997+0000 7fe92f26e700 1 -- 172.21.15.193:0/1896866144 >> v1:172.21.15.163:6800/2735092549 conn(0x7fe914063c10 legacy=0x7fe9140660d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:23.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.997+0000 7fe92f26e700 1 -- 172.21.15.193:0/1896866144 >> v1:172.21.15.193:6789/0 conn(0x7fe9280fe820 legacy=0x7fe928111360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:23.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.997+0000 7fe92f26e700 1 -- 172.21.15.193:0/1896866144 shutdown_connections 2024-02-19T21:01:23.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.997+0000 7fe92f26e700 1 -- 172.21.15.193:0/1896866144 >> 172.21.15.193:0/1896866144 conn(0x7fe9280f1e60 msgr2=0x7fe928100ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:23.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.997+0000 7fe92f26e700 1 -- 172.21.15.193:0/1896866144 shutdown_connections 2024-02-19T21:01:23.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.997+0000 7fe92f26e700 1 -- 172.21.15.193:0/1896866144 wait complete. 2024-02-19T21:01:23.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:23.997+0000 7fe92f26e700 1 librados: shutdown 2024-02-19T21:01:24.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:23.163697+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:01:24.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:23.163789+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:01:24.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:23.441298+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:24.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:23.448828+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:24.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:23.455748+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:24.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:23.459751+0000 mgr.y (mgr.14146) 206 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2024-02-19T21:01:24.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:23.466585+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-19T21:01:24.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:23.469762+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-19T21:01:24.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:23.477583+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:24.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:23.478357+0000 mgr.y (mgr.14146) 207 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi193 2024-02-19T21:01:24.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:23.659454+0000 mgr.y (mgr.14146) 208 : cluster [DBG] pgmap v202: 132 pgs: 20 creating+peering, 84 active+clean, 28 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:01:24.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:23.991164+0000 mgr.y (mgr.14146) 209 : audit [DBG] from='client.24433 v1:172.21.15.193:0/1896866144' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi193=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:01:24.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:23.992641+0000 mgr.y (mgr.14146) 210 : cephadm [INF] Saving service prometheus spec with placement smithi193=a;count:1 2024-02-19T21:01:24.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:23.995882+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:24.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:23.163697+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:01:24.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:23.163789+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:01:24.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:23.441298+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:24.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:23.448828+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:23.455748+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:23.459751+0000 mgr.y (mgr.14146) 206 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2024-02-19T21:01:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:23.466585+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-19T21:01:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:23.469762+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-19T21:01:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:23.477583+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:23.478357+0000 mgr.y (mgr.14146) 207 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi193 2024-02-19T21:01:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:23.659454+0000 mgr.y (mgr.14146) 208 : cluster [DBG] pgmap v202: 132 pgs: 20 creating+peering, 84 active+clean, 28 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:01:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:23.991164+0000 mgr.y (mgr.14146) 209 : audit [DBG] from='client.24433 v1:172.21.15.193:0/1896866144' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi193=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:01:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:23.992641+0000 mgr.y (mgr.14146) 210 : cephadm [INF] Saving service prometheus spec with placement smithi193=a;count:1 2024-02-19T21:01:24.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:23.995882+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:24.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:23.163697+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:01:24.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:23.163789+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:01:24.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:23.441298+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:24.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:23.448828+0000 mon.a (mon.0) 591 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:24.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:23.455748+0000 mon.a (mon.0) 592 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:24.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:23.459751+0000 mgr.y (mgr.14146) 206 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2024-02-19T21:01:24.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:23.466585+0000 mon.a (mon.0) 593 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-19T21:01:24.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:23.469762+0000 mon.a (mon.0) 594 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-19T21:01:24.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:23.477583+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:24.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:23.478357+0000 mgr.y (mgr.14146) 207 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi193 2024-02-19T21:01:24.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:23.659454+0000 mgr.y (mgr.14146) 208 : cluster [DBG] pgmap v202: 132 pgs: 20 creating+peering, 84 active+clean, 28 unknown; 1.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:01:24.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:23.991164+0000 mgr.y (mgr.14146) 209 : audit [DBG] from='client.24433 v1:172.21.15.193:0/1896866144' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi193=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:01:24.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:23.992641+0000 mgr.y (mgr.14146) 210 : cephadm [INF] Saving service prometheus spec with placement smithi193=a;count:1 2024-02-19T21:01:24.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:23.995882+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:25.296 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:01:25 smithi193 systemd[1]: Starting Ceph iscsi.iscsi.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:01:25.634 DEBUG:teuthology.orchestra.run.smithi193:prometheus.a> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@prometheus.a.service 2024-02-19T21:01:25.636 INFO:tasks.cephadm:Adding node-exporter.a on smithi163 2024-02-19T21:01:25.636 INFO:tasks.cephadm:Adding node-exporter.b on smithi193 2024-02-19T21:01:25.637 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch apply node-exporter '2;smithi163=a;smithi193=b' 2024-02-19T21:01:25.667 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:-- Logs begin at Mon 2024-02-19 20:44:17 UTC. -- 2024-02-19T21:01:26.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:25.477804+0000 mon.a (mon.0) 597 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-02-19T21:01:26.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:25.660085+0000 mgr.y (mgr.14146) 211 : cluster [DBG] pgmap v203: 132 pgs: 8 creating+peering, 124 active+clean; 3.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 2.6 KiB/s wr, 85 op/s 2024-02-19T21:01:26.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:25.477804+0000 mon.a (mon.0) 597 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-02-19T21:01:26.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:25.660085+0000 mgr.y (mgr.14146) 211 : cluster [DBG] pgmap v203: 132 pgs: 8 creating+peering, 124 active+clean; 3.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 2.6 KiB/s wr, 85 op/s 2024-02-19T21:01:26.810 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:01:26 smithi193 bash[60231]: log file path now is '/var/log/tcmu-runner.log' 2024-02-19T21:01:26.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:25.477804+0000 mon.a (mon.0) 597 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-02-19T21:01:26.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:25.660085+0000 mgr.y (mgr.14146) 211 : cluster [DBG] pgmap v203: 132 pgs: 8 creating+peering, 124 active+clean; 3.0 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 2.6 KiB/s wr, 85 op/s 2024-02-19T21:01:27.700 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.709+0000 7fc161531700 1 -- 172.21.15.193:0/570839328 >> v1:172.21.15.193:6789/0 conn(0x7fc15c0f5af0 legacy=0x7fc15c0f5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:27.700 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.709+0000 7fc161531700 1 -- 172.21.15.193:0/570839328 shutdown_connections 2024-02-19T21:01:27.700 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.709+0000 7fc161531700 1 -- 172.21.15.193:0/570839328 >> 172.21.15.193:0/570839328 conn(0x7fc15c0f1610 msgr2=0x7fc15c0f3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:27.700 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.710+0000 7fc161531700 1 -- 172.21.15.193:0/570839328 shutdown_connections 2024-02-19T21:01:27.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.710+0000 7fc161531700 1 -- 172.21.15.193:0/570839328 wait complete. 2024-02-19T21:01:27.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.710+0000 7fc161531700 1 librados: starting msgr at 2024-02-19T21:01:27.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.710+0000 7fc161531700 1 librados: starting objecter 2024-02-19T21:01:27.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.710+0000 7fc161531700 1 Processor -- start 2024-02-19T21:01:27.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.710+0000 7fc161531700 1 -- start start 2024-02-19T21:01:27.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.710+0000 7fc161531700 1 librados: setting wanted keys 2024-02-19T21:01:27.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.710+0000 7fc161531700 1 librados: calling monclient init 2024-02-19T21:01:27.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.710+0000 7fc161531700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc15c108650 con 0x7fc15c0f5af0 2024-02-19T21:01:27.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.710+0000 7fc161531700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc15c113eb0 con 0x7fc15c0fbe40 2024-02-19T21:01:27.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.711+0000 7fc161531700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc15c115090 con 0x7fc15c0ff950 2024-02-19T21:01:27.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.711+0000 7fc15a7fc700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fc15c0fbe40 0x7fc15c10e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:59358/0 (socket says 172.21.15.193:59358) 2024-02-19T21:01:27.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.711+0000 7fc15a7fc700 1 -- 172.21.15.193:0/3911435992 learned_addr learned my addr 172.21.15.193:0/3911435992 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T21:01:27.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.711+0000 7fc143fff700 10 client.?.objecter ms_handle_connect 0x7fc15c0fbe40 2024-02-19T21:01:27.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.711+0000 7fc143fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:27.701 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.711+0000 7fc143fff700 10 client.?.objecter ms_handle_connect 0x7fc15c0f5af0 2024-02-19T21:01:27.702 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.711+0000 7fc143fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:27.702 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.711+0000 7fc143fff700 10 client.?.objecter ms_handle_connect 0x7fc15c0ff950 2024-02-19T21:01:27.702 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.711+0000 7fc143fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:27.702 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.711+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911841470 0 0) 0x7fc15c108650 con 0x7fc15c0f5af0 2024-02-19T21:01:27.702 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.711+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc144002220 con 0x7fc15c0f5af0 2024-02-19T21:01:27.703 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.713+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200032138 0 0) 0x7fc15c113eb0 con 0x7fc15c0fbe40 2024-02-19T21:01:27.703 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc15c108650 con 0x7fc15c0fbe40 2024-02-19T21:01:27.703 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652544775 0 0) 0x7fc15c115090 con 0x7fc15c0ff950 2024-02-19T21:01:27.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc15c113eb0 con 0x7fc15c0ff950 2024-02-19T21:01:27.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3454782883 0 0) 0x7fc144002220 con 0x7fc15c0f5af0 2024-02-19T21:01:27.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc15c115090 con 0x7fc15c0f5af0 2024-02-19T21:01:27.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc14c0010c0 con 0x7fc15c0f5af0 2024-02-19T21:01:27.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1595316204 0 0) 0x7fc15c108650 con 0x7fc15c0fbe40 2024-02-19T21:01:27.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc144002220 con 0x7fc15c0fbe40 2024-02-19T21:01:27.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc150001900 con 0x7fc15c0fbe40 2024-02-19T21:01:27.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3306811883 0 0) 0x7fc15c115090 con 0x7fc15c0f5af0 2024-02-19T21:01:27.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 >> v1:172.21.15.163:6790/0 conn(0x7fc15c0ff950 legacy=0x7fc15c111780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:27.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 >> v1:172.21.15.193:6789/0 conn(0x7fc15c0fbe40 legacy=0x7fc15c10e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:27.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15c116270 con 0x7fc15c0f5af0 2024-02-19T21:01:27.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc161531700 1 -- 172.21.15.193:0/3911435992 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc15c112f00 con 0x7fc15c0f5af0 2024-02-19T21:01:27.704 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc161531700 10 client.14508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:01:27.705 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc161531700 1 -- 172.21.15.193:0/3911435992 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15c113370 con 0x7fc15c0f5af0 2024-02-19T21:01:27.705 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc161531700 1 librados: init done 2024-02-19T21:01:27.705 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fc14c001cb0 con 0x7fc15c0f5af0 2024-02-19T21:01:27.705 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.714+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc14c0035e0 con 0x7fc15c0f5af0 2024-02-19T21:01:27.705 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.715+0000 7fc161531700 1 -- 172.21.15.193:0/3911435992 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc148004f20 con 0x7fc15c0f5af0 2024-02-19T21:01:27.706 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.715+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 15) v1 ==== 77081+0+0 (unknown 2383824409 0 0) 0x7fc14c0020f0 con 0x7fc15c0f5af0 2024-02-19T21:01:27.706 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.716+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(65..65 src has 1..65) v4 ==== 5884+0+0 (unknown 3811228291 0 0) 0x7fc14c075ef0 con 0x7fc15c0f5af0 2024-02-19T21:01:27.706 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.716+0000 7fc143fff700 10 client.14508.objecter ms_dispatch 0x7fc15c001550 osd_map(65..65 src has 1..65) v4 2024-02-19T21:01:27.706 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.716+0000 7fc143fff700 3 client.14508.objecter handle_osd_map got epochs [65,65] > 0 2024-02-19T21:01:27.706 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.716+0000 7fc143fff700 3 client.14508.objecter handle_osd_map decoding full epoch 65 2024-02-19T21:01:27.706 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.716+0000 7fc143fff700 20 client.14508.objecter dump_active .. 0 homeless 2024-02-19T21:01:27.706 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.716+0000 7fc143fff700 10 client.14508.objecter ms_handle_connect 0x7fc144063a10 2024-02-19T21:01:27.709 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.720+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc14c045f40 con 0x7fc15c0f5af0 2024-02-19T21:01:27.886 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.897+0000 7fc161531700 1 -- 172.21.15.193:0/3911435992 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi163=a;smithi193=b", "target": ["mon-mgr", ""]}) v1 -- 0x7fc148000b90 con 0x7fc144063a10 2024-02-19T21:01:27.893 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.904+0000 7fc143fff700 1 -- 172.21.15.193:0/3911435992 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (unknown 0 0 240551134) 0x7fc148000b90 con 0x7fc144063a10 2024-02-19T21:01:27.893 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled node-exporter update... 2024-02-19T21:01:27.895 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.906+0000 7fc161531700 10 librados: watch_flush enter 2024-02-19T21:01:27.895 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.906+0000 7fc161531700 10 librados: watch_flush exit 2024-02-19T21:01:27.895 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.906+0000 7fc161531700 20 client.14508.objecter shutdown clearing up homeless session... 2024-02-19T21:01:27.895 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.906+0000 7fc161531700 10 client.14508.objecter successfully canceled tick 2024-02-19T21:01:27.895 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.906+0000 7fc161531700 1 -- 172.21.15.193:0/3911435992 >> v1:172.21.15.163:6800/2735092549 conn(0x7fc144063a10 legacy=0x7fc144065ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:27.896 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.906+0000 7fc161531700 1 -- 172.21.15.193:0/3911435992 >> v1:172.21.15.163:6789/0 conn(0x7fc15c0f5af0 legacy=0x7fc15c107a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:27.896 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.906+0000 7fc161531700 1 -- 172.21.15.193:0/3911435992 shutdown_connections 2024-02-19T21:01:27.896 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.906+0000 7fc161531700 1 -- 172.21.15.193:0/3911435992 >> 172.21.15.193:0/3911435992 conn(0x7fc15c0f1610 msgr2=0x7fc15c0ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:27.896 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.906+0000 7fc161531700 1 -- 172.21.15.193:0/3911435992 shutdown_connections 2024-02-19T21:01:27.896 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.906+0000 7fc161531700 1 -- 172.21.15.193:0/3911435992 wait complete. 2024-02-19T21:01:27.896 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:27.906+0000 7fc161531700 1 librados: shutdown 2024-02-19T21:01:28.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:01:28 smithi193 bash[60395]: b12659232c2d345b227ac31a24fce5846726c2dc8e940d216c5ace588c43cd68 2024-02-19T21:01:28.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:01:28 smithi193 systemd[1]: Started Ceph iscsi.iscsi.a for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T21:01:28.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:01:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug Started the configuration object watcher 2024-02-19T21:01:28.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:01:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug Checking for config object changes every 1s 2024-02-19T21:01:28.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:01:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug Processing osd blocklist entries for this node 2024-02-19T21:01:28.801 DEBUG:teuthology.orchestra.run.smithi163:node-exporter.a> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@node-exporter.a.service 2024-02-19T21:01:28.805 DEBUG:teuthology.orchestra.run.smithi193:node-exporter.b> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@node-exporter.b.service 2024-02-19T21:01:28.807 INFO:tasks.cephadm:Adding alertmanager.a on smithi163 2024-02-19T21:01:28.807 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch apply alertmanager '1;smithi163=a' 2024-02-19T21:01:28.841 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:-- Logs begin at Mon 2024-02-19 20:44:17 UTC. -- 2024-02-19T21:01:28.843 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:-- Logs begin at Mon 2024-02-19 20:44:35 UTC. -- 2024-02-19T21:01:29.060 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:01:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug Reading the configuration object to update local LIO configuration 2024-02-19T21:01:29.060 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:01:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug Configuration does not have an entry for this host(smithi193.front.sepia.ceph.com) - nothing to define to LIO 2024-02-19T21:01:29.060 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:01:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-02-19T21:01:29.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:27.660814+0000 mgr.y (mgr.14146) 212 : cluster [DBG] pgmap v204: 132 pgs: 132 active+clean; 4.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 4.7 KiB/s wr, 149 op/s 2024-02-19T21:01:29.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:27.899122+0000 mgr.y (mgr.14146) 213 : audit [DBG] from='client.14508 v1:172.21.15.193:0/3911435992' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi163=a;smithi193=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:01:29.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:27.900622+0000 mgr.y (mgr.14146) 214 : cephadm [INF] Saving service node-exporter spec with placement smithi163=a;smithi193=b;count:2 2024-02-19T21:01:29.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:27.905123+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:29.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:28.479165+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:29.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:28.480574+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:29.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:28.866304+0000 mon.b (mon.1) 26 : audit [DBG] from='client.? v1:172.21.15.193:0/347386030' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-19T21:01:29.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:27.660814+0000 mgr.y (mgr.14146) 212 : cluster [DBG] pgmap v204: 132 pgs: 132 active+clean; 4.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 4.7 KiB/s wr, 149 op/s 2024-02-19T21:01:29.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:27.899122+0000 mgr.y (mgr.14146) 213 : audit [DBG] from='client.14508 v1:172.21.15.193:0/3911435992' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi163=a;smithi193=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:01:29.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:27.900622+0000 mgr.y (mgr.14146) 214 : cephadm [INF] Saving service node-exporter spec with placement smithi163=a;smithi193=b;count:2 2024-02-19T21:01:29.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:27.905123+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:29.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:28.479165+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:29.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:28.480574+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:29.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:28.866304+0000 mon.b (mon.1) 26 : audit [DBG] from='client.? v1:172.21.15.193:0/347386030' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-19T21:01:29.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:27.660814+0000 mgr.y (mgr.14146) 212 : cluster [DBG] pgmap v204: 132 pgs: 132 active+clean; 4.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 4.7 KiB/s wr, 149 op/s 2024-02-19T21:01:29.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:27.899122+0000 mgr.y (mgr.14146) 213 : audit [DBG] from='client.14508 v1:172.21.15.193:0/3911435992' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi163=a;smithi193=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:01:29.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:27.900622+0000 mgr.y (mgr.14146) 214 : cephadm [INF] Saving service node-exporter spec with placement smithi163=a;smithi193=b;count:2 2024-02-19T21:01:29.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:27.905123+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:29.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:28.479165+0000 mon.a (mon.0) 599 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:29.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:28.480574+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:29.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:28.866304+0000 mon.b (mon.1) 26 : audit [DBG] from='client.? v1:172.21.15.193:0/347386030' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-19T21:01:30.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:29.632096+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-02-19T21:01:30.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:29.632446+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-19T21:01:30.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:29.632640+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-19T21:01:30.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:29.632825+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-19T21:01:30.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:29.632096+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-02-19T21:01:30.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:29.632446+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-19T21:01:30.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:29.632640+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-19T21:01:30.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:29.632825+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-19T21:01:30.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:29.632096+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-02-19T21:01:30.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:29.632446+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-19T21:01:30.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:29.632640+0000 mon.a (mon.0) 603 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-19T21:01:30.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:29.632825+0000 mon.a (mon.0) 604 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-19T21:01:30.954 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:29.661563+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v205: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-02-19T21:01:30.954 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:29.919487+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-02-19T21:01:30.954 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:29.919584+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-19T21:01:30.954 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:29.919647+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-02-19T21:01:30.955 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:29.919724+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-02-19T21:01:30.955 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:29.919787+0000 mon.a (mon.0) 609 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-19T21:01:30.955 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:29.976655+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:31.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.208+0000 7fa6f9b42700 1 -- 172.21.15.193:0/2352371729 >> v1:172.21.15.193:6789/0 conn(0x7fa6f40ff920 legacy=0x7fa6f4101de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:31.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.208+0000 7fa6f9b42700 1 -- 172.21.15.193:0/2352371729 shutdown_connections 2024-02-19T21:01:31.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.208+0000 7fa6f9b42700 1 -- 172.21.15.193:0/2352371729 >> 172.21.15.193:0/2352371729 conn(0x7fa6f40f2f80 msgr2=0x7fa6f40f53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:31.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.208+0000 7fa6f9b42700 1 -- 172.21.15.193:0/2352371729 shutdown_connections 2024-02-19T21:01:31.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.208+0000 7fa6f9b42700 1 -- 172.21.15.193:0/2352371729 wait complete. 2024-02-19T21:01:31.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.208+0000 7fa6f9b42700 1 librados: starting msgr at 2024-02-19T21:01:31.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.208+0000 7fa6f9b42700 1 librados: starting objecter 2024-02-19T21:01:31.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.208+0000 7fa6f9b42700 1 Processor -- start 2024-02-19T21:01:31.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f9b42700 1 -- start start 2024-02-19T21:01:31.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f9b42700 1 librados: setting wanted keys 2024-02-19T21:01:31.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f9b42700 1 librados: calling monclient init 2024-02-19T21:01:31.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f9b42700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa6f4114ea0 con 0x7fa6f40fbcf0 2024-02-19T21:01:31.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f9b42700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa6f4116080 con 0x7fa6f40ff920 2024-02-19T21:01:31.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f9b42700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa6f4117280 con 0x7fa6f40f7820 2024-02-19T21:01:31.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f3fff700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fa6f40ff920 0x7fa6f4113560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:59452/0 (socket says 172.21.15.193:59452) 2024-02-19T21:01:31.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f3fff700 1 -- 172.21.15.193:0/4132392332 learned_addr learned my addr 172.21.15.193:0/4132392332 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T21:01:31.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 10 client.?.objecter ms_handle_connect 0x7fa6f40ff920 2024-02-19T21:01:31.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:31.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 10 client.?.objecter ms_handle_connect 0x7fa6f40fbcf0 2024-02-19T21:01:31.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:31.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 10 client.?.objecter ms_handle_connect 0x7fa6f40f7820 2024-02-19T21:01:31.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:31.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 626663493 0 0) 0x7fa6f4116080 con 0x7fa6f40ff920 2024-02-19T21:01:31.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6dc002220 con 0x7fa6f40ff920 2024-02-19T21:01:31.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997707484 0 0) 0x7fa6f4114ea0 con 0x7fa6f40fbcf0 2024-02-19T21:01:31.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f4116080 con 0x7fa6f40fbcf0 2024-02-19T21:01:31.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646059743 0 0) 0x7fa6f4117280 con 0x7fa6f40f7820 2024-02-19T21:01:31.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f4114ea0 con 0x7fa6f40f7820 2024-02-19T21:01:31.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1734321552 0 0) 0x7fa6dc002220 con 0x7fa6f40ff920 2024-02-19T21:01:31.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6f4117280 con 0x7fa6f40ff920 2024-02-19T21:01:31.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa6ec001de0 con 0x7fa6f40ff920 2024-02-19T21:01:31.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4184612078 0 0) 0x7fa6f4116080 con 0x7fa6f40fbcf0 2024-02-19T21:01:31.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6dc002220 con 0x7fa6f40fbcf0 2024-02-19T21:01:31.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa6e80017b0 con 0x7fa6f40fbcf0 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1909992521 0 0) 0x7fa6f4114ea0 con 0x7fa6f40f7820 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6f4116080 con 0x7fa6f40f7820 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.209+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4084812808 0 0) 0x7fa6f4117280 con 0x7fa6f40ff920 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.210+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 >> v1:172.21.15.163:6790/0 conn(0x7fa6f40f7820 legacy=0x7fa6f406c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.210+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 >> v1:172.21.15.163:6789/0 conn(0x7fa6f40fbcf0 legacy=0x7fa6f410fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.210+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f4118480 con 0x7fa6f40ff920 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.210+0000 7fa6f9b42700 1 -- 172.21.15.193:0/4132392332 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f4115090 con 0x7fa6f40ff920 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.210+0000 7fa6f9b42700 10 client.24469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.210+0000 7fa6f9b42700 1 -- 172.21.15.193:0/4132392332 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f4115500 con 0x7fa6f40ff920 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.210+0000 7fa6f9b42700 1 librados: init done 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.210+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fa6ec0030f0 con 0x7fa6f40ff920 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.210+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa6ec003c60 con 0x7fa6f40ff920 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.211+0000 7fa6f9b42700 1 -- 172.21.15.193:0/4132392332 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f4060650 con 0x7fa6f40ff920 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.211+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 15) v1 ==== 77081+0+0 (unknown 2383824409 0 0) 0x7fa6ec003ea0 con 0x7fa6f40ff920 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.212+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(67..67 src has 1..67) v4 ==== 6000+0+0 (unknown 1224682740 0 0) 0x7fa6ec078d60 con 0x7fa6f40ff920 2024-02-19T21:01:31.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.212+0000 7fa6f0ff9700 10 client.24469.objecter ms_dispatch 0x7fa6f4001550 osd_map(67..67 src has 1..67) v4 2024-02-19T21:01:31.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.212+0000 7fa6f0ff9700 3 client.24469.objecter handle_osd_map got epochs [67,67] > 0 2024-02-19T21:01:31.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.212+0000 7fa6f0ff9700 3 client.24469.objecter handle_osd_map decoding full epoch 67 2024-02-19T21:01:31.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.212+0000 7fa6f0ff9700 20 client.24469.objecter dump_active .. 0 homeless 2024-02-19T21:01:31.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.212+0000 7fa6f0ff9700 10 client.24469.objecter ms_handle_connect 0x7fa6dc063ce0 2024-02-19T21:01:31.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.216+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa6f4060650 con 0x7fa6f40ff920 2024-02-19T21:01:31.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:29.661563+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v205: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-02-19T21:01:31.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:29.919487+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-02-19T21:01:31.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:29.919584+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-19T21:01:31.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:29.919647+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-02-19T21:01:31.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:29.919724+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-02-19T21:01:31.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:29.919787+0000 mon.a (mon.0) 609 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-19T21:01:31.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:29.976655+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:31.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:29.661563+0000 mgr.y (mgr.14146) 215 : cluster [DBG] pgmap v205: 132 pgs: 132 active+clean; 5.3 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-02-19T21:01:31.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:29.919487+0000 mon.a (mon.0) 605 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-02-19T21:01:31.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:29.919584+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-19T21:01:31.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:29.919647+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-02-19T21:01:31.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:29.919724+0000 mon.a (mon.0) 608 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-02-19T21:01:31.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:29.919787+0000 mon.a (mon.0) 609 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2024-02-19T21:01:31.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:29.976655+0000 mon.a (mon.0) 610 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:31.398 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.409+0000 7fa6f9b42700 1 -- 172.21.15.193:0/4132392332 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi163=a", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6f4060650 con 0x7fa6dc063ce0 2024-02-19T21:01:31.404 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.415+0000 7fa6f0ff9700 1 -- 172.21.15.193:0/4132392332 <== mgr.14146 v1:172.21.15.163:6800/2735092549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (unknown 0 0 1850065467) 0x7fa6f4060650 con 0x7fa6dc063ce0 2024-02-19T21:01:31.405 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled alertmanager update... 2024-02-19T21:01:31.407 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.417+0000 7fa6f9b42700 10 librados: watch_flush enter 2024-02-19T21:01:31.407 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.417+0000 7fa6f9b42700 10 librados: watch_flush exit 2024-02-19T21:01:31.407 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.417+0000 7fa6f9b42700 20 client.24469.objecter shutdown clearing up homeless session... 2024-02-19T21:01:31.407 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.417+0000 7fa6f9b42700 10 client.24469.objecter successfully canceled tick 2024-02-19T21:01:31.407 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.417+0000 7fa6f9b42700 1 -- 172.21.15.193:0/4132392332 >> v1:172.21.15.163:6800/2735092549 conn(0x7fa6dc063ce0 legacy=0x7fa6dc0661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:31.407 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.417+0000 7fa6f9b42700 1 -- 172.21.15.193:0/4132392332 >> v1:172.21.15.193:6789/0 conn(0x7fa6f40ff920 legacy=0x7fa6f4113560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:31.407 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.417+0000 7fa6f9b42700 1 -- 172.21.15.193:0/4132392332 shutdown_connections 2024-02-19T21:01:31.407 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.417+0000 7fa6f9b42700 1 -- 172.21.15.193:0/4132392332 >> 172.21.15.193:0/4132392332 conn(0x7fa6f40f2f80 msgr2=0x7fa6f40f3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:31.407 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.418+0000 7fa6f9b42700 1 -- 172.21.15.193:0/4132392332 shutdown_connections 2024-02-19T21:01:31.408 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.418+0000 7fa6f9b42700 1 -- 172.21.15.193:0/4132392332 wait complete. 2024-02-19T21:01:31.408 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:31.418+0000 7fa6f9b42700 1 librados: shutdown 2024-02-19T21:01:32.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:30.919295+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-19T21:01:32.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:31.416430+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:32.236 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:30.919295+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-19T21:01:32.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:31.416430+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:32.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:30.919295+0000 mon.a (mon.0) 611 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2024-02-19T21:01:32.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:31.416430+0000 mon.a (mon.0) 612 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:32.246 DEBUG:teuthology.orchestra.run.smithi163:alertmanager.a> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@alertmanager.a.service 2024-02-19T21:01:32.249 INFO:tasks.cephadm:Adding grafana.a on smithi193 2024-02-19T21:01:32.249 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph orch apply grafana '1;smithi193=a' 2024-02-19T21:01:32.282 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:-- Logs begin at Mon 2024-02-19 20:44:35 UTC. -- 2024-02-19T21:01:33.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:31.411166+0000 mgr.y (mgr.14146) 216 : audit [DBG] from='client.24469 v1:172.21.15.193:0/4132392332' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi163=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:01:33.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:31.412604+0000 mgr.y (mgr.14146) 217 : cephadm [INF] Saving service alertmanager spec with placement smithi163=a;count:1 2024-02-19T21:01:33.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:31.662024+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v208: 132 pgs: 1 peering, 131 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 56 KiB/s rd, 4.2 KiB/s wr, 136 op/s 2024-02-19T21:01:33.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:32.571954+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:33.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:32.572868+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:33.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:32.573688+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:33.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:32.580859+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:33.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:32.586046+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-19T21:01:33.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:31.411166+0000 mgr.y (mgr.14146) 216 : audit [DBG] from='client.24469 v1:172.21.15.193:0/4132392332' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi163=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:01:33.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:31.412604+0000 mgr.y (mgr.14146) 217 : cephadm [INF] Saving service alertmanager spec with placement smithi163=a;count:1 2024-02-19T21:01:33.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:31.662024+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v208: 132 pgs: 1 peering, 131 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 56 KiB/s rd, 4.2 KiB/s wr, 136 op/s 2024-02-19T21:01:33.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:32.571954+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:33.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:32.572868+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:33.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:32.573688+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:33.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:32.580859+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:33.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:32.586046+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-19T21:01:33.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:31.411166+0000 mgr.y (mgr.14146) 216 : audit [DBG] from='client.24469 v1:172.21.15.193:0/4132392332' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi163=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T21:01:33.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:31.412604+0000 mgr.y (mgr.14146) 217 : cephadm [INF] Saving service alertmanager spec with placement smithi163=a;count:1 2024-02-19T21:01:33.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:31.662024+0000 mgr.y (mgr.14146) 218 : cluster [DBG] pgmap v208: 132 pgs: 1 peering, 131 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 56 KiB/s rd, 4.2 KiB/s wr, 136 op/s 2024-02-19T21:01:33.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:32.571954+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:33.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:32.572868+0000 mon.a (mon.0) 614 : audit [DBG] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:33.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:32.573688+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:33.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:32.580859+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' 2024-02-19T21:01:33.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:32.586046+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-19T21:01:33.810 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ignoring --setuser ceph since I am not root 2024-02-19T21:01:33.811 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ignoring --setgroup ceph since I am not root 2024-02-19T21:01:33.811 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:33.739+0000 7fb72212e680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-19T21:01:33.939 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ignoring --setuser ceph since I am not root 2024-02-19T21:01:33.939 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ignoring --setgroup ceph since I am not root 2024-02-19T21:01:33.939 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:33.747+0000 7f8d7c347680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-19T21:01:33.939 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:33.828+0000 7f8d7c347680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-19T21:01:34.111 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:33.821+0000 7fb72212e680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-19T21:01:34.111 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:32.952470+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-19T21:01:34.111 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:33.588075+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-19T21:01:34.111 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:33.588126+0000 mon.a (mon.0) 620 : cluster [DBG] mgrmap e16: y(active, since 5m), standbys: x 2024-02-19T21:01:34.236 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:34.188+0000 7f8d7c347680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-19T21:01:34.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:32.952470+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-19T21:01:34.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:33.588075+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-19T21:01:34.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:33.588126+0000 mon.a (mon.0) 620 : cluster [DBG] mgrmap e16: y(active, since 5m), standbys: x 2024-02-19T21:01:34.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:32.952470+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2024-02-19T21:01:34.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:33.588075+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.14146 v1:172.21.15.163:0/2611007343' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-19T21:01:34.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:33.588126+0000 mon.a (mon.0) 620 : cluster [DBG] mgrmap e16: y(active, since 5m), standbys: x 2024-02-19T21:01:34.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.390+0000 7f5882010700 1 -- 172.21.15.193:0/2638397261 >> v1:172.21.15.193:6789/0 conn(0x7f587c0ff930 legacy=0x7f587c101dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:34.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.390+0000 7f5882010700 1 -- 172.21.15.193:0/2638397261 shutdown_connections 2024-02-19T21:01:34.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.390+0000 7f5882010700 1 -- 172.21.15.193:0/2638397261 >> 172.21.15.193:0/2638397261 conn(0x7f587c0f1610 msgr2=0x7f587c0f3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:34.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.390+0000 7f5882010700 1 -- 172.21.15.193:0/2638397261 shutdown_connections 2024-02-19T21:01:34.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.390+0000 7f5882010700 1 -- 172.21.15.193:0/2638397261 wait complete. 2024-02-19T21:01:34.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.391+0000 7f5882010700 1 librados: starting msgr at 2024-02-19T21:01:34.381 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.391+0000 7f5882010700 1 librados: starting objecter 2024-02-19T21:01:34.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.391+0000 7f5882010700 1 Processor -- start 2024-02-19T21:01:34.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.391+0000 7f5882010700 1 -- start start 2024-02-19T21:01:34.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.391+0000 7f5882010700 1 librados: setting wanted keys 2024-02-19T21:01:34.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.391+0000 7f5882010700 1 librados: calling monclient init 2024-02-19T21:01:34.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.391+0000 7f5882010700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f587c1084c0 con 0x7f587c0f5ad0 2024-02-19T21:01:34.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.391+0000 7f5882010700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f587c113d20 con 0x7f587c0ff930 2024-02-19T21:01:34.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.391+0000 7f5882010700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f587c114f00 con 0x7f587c0fbe20 2024-02-19T21:01:34.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.391+0000 7f587bfff700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f587c0ff930 0x7f587c1115f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:59492/0 (socket says 172.21.15.193:59492) 2024-02-19T21:01:34.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f587bfff700 1 -- 172.21.15.193:0/1976425738 learned_addr learned my addr 172.21.15.193:0/1976425738 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T21:01:34.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 10 client.?.objecter ms_handle_connect 0x7f587c0ff930 2024-02-19T21:01:34.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:34.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 10 client.?.objecter ms_handle_connect 0x7f587c0f5ad0 2024-02-19T21:01:34.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:34.382 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 10 client.?.objecter ms_handle_connect 0x7f587c0fbe20 2024-02-19T21:01:34.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:34.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123015272 0 0) 0x7f587c113d20 con 0x7f587c0ff930 2024-02-19T21:01:34.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58640033f0 con 0x7f587c0ff930 2024-02-19T21:01:34.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198394190 0 0) 0x7f587c1084c0 con 0x7f587c0f5ad0 2024-02-19T21:01:34.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f587c113d20 con 0x7f587c0f5ad0 2024-02-19T21:01:34.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56502958 0 0) 0x7f587c114f00 con 0x7f587c0fbe20 2024-02-19T21:01:34.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f587c1084c0 con 0x7f587c0fbe20 2024-02-19T21:01:34.385 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3892697024 0 0) 0x7f587c113d20 con 0x7f587c0f5ad0 2024-02-19T21:01:34.385 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f587c114f00 con 0x7f587c0f5ad0 2024-02-19T21:01:34.385 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3724493851 0 0) 0x7f58640033f0 con 0x7f587c0ff930 2024-02-19T21:01:34.385 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f587c113d20 con 0x7f587c0ff930 2024-02-19T21:01:34.385 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f586c001840 con 0x7f587c0f5ad0 2024-02-19T21:01:34.385 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f58740030e0 con 0x7f587c0ff930 2024-02-19T21:01:34.385 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3233024383 0 0) 0x7f587c1084c0 con 0x7f587c0fbe20 2024-02-19T21:01:34.385 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58640033f0 con 0x7f587c0fbe20 2024-02-19T21:01:34.385 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f58700018c0 con 0x7f587c0fbe20 2024-02-19T21:01:34.385 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1958018175 0 0) 0x7f587c114f00 con 0x7f587c0f5ad0 2024-02-19T21:01:34.385 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 >> v1:172.21.15.163:6790/0 conn(0x7f587c0fbe20 legacy=0x7f587c10dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:34.385 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 >> v1:172.21.15.193:6789/0 conn(0x7f587c0ff930 legacy=0x7f587c1115f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:34.386 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587c1160e0 con 0x7f587c0f5ad0 2024-02-19T21:01:34.386 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5882010700 1 -- 172.21.15.193:0/1976425738 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f587c113f10 con 0x7f587c0f5ad0 2024-02-19T21:01:34.386 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5882010700 10 client.14556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:01:34.386 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5882010700 1 -- 172.21.15.193:0/1976425738 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f587c114400 con 0x7f587c0f5ad0 2024-02-19T21:01:34.386 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5882010700 1 librados: init done 2024-02-19T21:01:34.386 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f586c001ec0 con 0x7f587c0f5ad0 2024-02-19T21:01:34.386 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.392+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f586c003160 con 0x7f587c0f5ad0 2024-02-19T21:01:34.386 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.393+0000 7f5882010700 1 -- 172.21.15.193:0/1976425738 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5868004f20 con 0x7f587c0f5ad0 2024-02-19T21:01:34.386 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.394+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 16) v1 ==== 77095+0+0 (unknown 3106140564 0 0) 0x7f586c001a20 con 0x7f587c0f5ad0 2024-02-19T21:01:34.386 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.394+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(68..68 src has 1..68) v4 ==== 6000+0+0 (unknown 2452188071 0 0) 0x7f586c075cf0 con 0x7f587c0f5ad0 2024-02-19T21:01:34.386 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.394+0000 7f5878ff9700 10 client.14556.objecter ms_dispatch 0x7f587c001550 osd_map(68..68 src has 1..68) v4 2024-02-19T21:01:34.386 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.394+0000 7f5878ff9700 3 client.14556.objecter handle_osd_map got epochs [68,68] > 0 2024-02-19T21:01:34.386 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.394+0000 7f5878ff9700 3 client.14556.objecter handle_osd_map decoding full epoch 68 2024-02-19T21:01:34.386 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.394+0000 7f587affd700 1 -- 172.21.15.193:0/1976425738 >> v1:172.21.15.163:6800/2735092549 conn(0x7f5864063d20 legacy=0x7f58640661e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.163:6800/2735092549 2024-02-19T21:01:34.387 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.394+0000 7f5878ff9700 20 client.14556.objecter dump_active .. 0 homeless 2024-02-19T21:01:34.387 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.398+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f586c045c30 con 0x7f587c0f5ad0 2024-02-19T21:01:34.560 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:34.197+0000 7fb72212e680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-19T21:01:34.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.578+0000 7f5882010700 1 -- 172.21.15.193:0/1976425738 --> v1:172.21.15.163:6800/2735092549 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi193=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f5868000b90 con 0x7f5864063d20 2024-02-19T21:01:34.584 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.595+0000 7f587affd700 1 -- 172.21.15.193:0/1976425738 >> v1:172.21.15.163:6800/2735092549 conn(0x7f5864063d20 legacy=0x7f58640661e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.163:6800/2735092549 2024-02-19T21:01:34.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:34.996+0000 7f587affd700 1 -- 172.21.15.193:0/1976425738 >> v1:172.21.15.163:6800/2735092549 conn(0x7f5864063d20 legacy=0x7f58640661e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.163:6800/2735092549 2024-02-19T21:01:35.219 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:34.933+0000 7f8d7c347680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-19T21:01:35.219 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:35.017+0000 7f8d7c347680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-19T21:01:35.219 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:35.228+0000 7f8d7c347680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-19T21:01:35.237 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:34.954+0000 7fb72212e680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-19T21:01:35.237 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:35.038+0000 7fb72212e680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-19T21:01:35.560 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:35.248+0000 7fb72212e680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-19T21:01:35.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:35.797+0000 7f587affd700 1 -- 172.21.15.193:0/1976425738 >> v1:172.21.15.163:6800/2735092549 conn(0x7f5864063d20 legacy=0x7f58640661e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.163:6800/2735092549 2024-02-19T21:01:36.451 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:36.162+0000 7f8d7c347680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-19T21:01:36.451 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:36.291+0000 7f8d7c347680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-19T21:01:36.451 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:36.372+0000 7f8d7c347680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-19T21:01:36.466 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:36.181+0000 7fb72212e680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-19T21:01:36.466 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:36.309+0000 7fb72212e680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-19T21:01:36.466 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:36.387+0000 7fb72212e680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-19T21:01:36.736 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:36.461+0000 7f8d7c347680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-19T21:01:36.737 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:36.657+0000 7f8d7c347680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-19T21:01:36.783 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:36.476+0000 7fb72212e680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-19T21:01:36.783 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:36.674+0000 7fb72212e680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-19T21:01:37.040 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:36.953+0000 7fb72212e680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-19T21:01:37.040 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:35.739416+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:01:37.040 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:35.739484+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:01:37.041 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:35.796925+0000 mon.a (mon.0) 621 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-19T21:01:37.041 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:36.293205+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:01:37.041 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:36.293263+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:01:37.236 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:36.939+0000 7f8d7c347680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-19T21:01:37.237 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:37.037+0000 7f8d7c347680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-19T21:01:37.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:35.739416+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:01:37.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:35.739484+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:01:37.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:35.796925+0000 mon.a (mon.0) 621 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-19T21:01:37.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:36.293205+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:01:37.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:36.293263+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:01:37.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:35.739416+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:01:37.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:35.739484+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:01:37.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:35.796925+0000 mon.a (mon.0) 621 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-02-19T21:01:37.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:36.293205+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:01:37.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:36.293263+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:01:37.310 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:37.051+0000 7fb72212e680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-19T21:01:37.387 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:37.398+0000 7f587affd700 1 -- 172.21.15.193:0/1976425738 >> v1:172.21.15.163:6800/2735092549 conn(0x7f5864063d20 legacy=0x7f58640661e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.163:6800/2735092549 2024-02-19T21:01:38.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:36.804142+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-19T21:01:38.060 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:37.769+0000 7fb72212e680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-19T21:01:38.060 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:37.851+0000 7fb72212e680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-19T21:01:38.060 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:37.944+0000 7fb72212e680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-19T21:01:38.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:36.804142+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-19T21:01:38.091 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:36.804142+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-02-19T21:01:38.091 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:37.754+0000 7f8d7c347680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-19T21:01:38.091 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:37.837+0000 7f8d7c347680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-19T21:01:38.091 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:37.927+0000 7f8d7c347680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-19T21:01:38.091 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:38.101+0000 7f8d7c347680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-19T21:01:38.369 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:38.117+0000 7fb72212e680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-19T21:01:38.616 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:38.363+0000 7f8d7c347680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-19T21:01:38.616 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:38.493+0000 7f8d7c347680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-19T21:01:38.632 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:38.379+0000 7fb72212e680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-19T21:01:38.632 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:38.509+0000 7fb72212e680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-19T21:01:38.870 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:38.626+0000 7f8d7c347680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-19T21:01:38.870 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:38.797+0000 7f8d7c347680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-19T21:01:38.870 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:01:38.880+0000 7f8d7c347680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-19T21:01:38.887 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:38.642+0000 7fb72212e680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-19T21:01:38.887 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:38.814+0000 7fb72212e680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-19T21:01:38.887 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: debug 2024-02-19T21:01:38.897+0000 7fb72212e680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-19T21:01:38.896 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:38.906+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mgrmap(e 17) v1 ==== 76859+0+0 (unknown 4215209820 0 0) 0x7f586c073940 con 0x7f587c0f5ad0 2024-02-19T21:01:38.896 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:38.906+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 >> v1:172.21.15.163:6800/2735092549 conn(0x7f5864063d20 legacy=0x7f58640661e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T21:01:39.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:37.951645+0000 mon.a (mon.0) 623 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-19T21:01:39.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:38.896212+0000 mon.a (mon.0) 624 : cluster [INF] Active manager daemon y restarted 2024-02-19T21:01:39.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:38.896631+0000 mon.a (mon.0) 625 : cluster [INF] Activating manager daemon y 2024-02-19T21:01:39.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:38.906413+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-19T21:01:39.126 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:38.906926+0000 mon.a (mon.0) 627 : cluster [DBG] mgrmap e17: y(active, starting, since 0.0103887s), standbys: x 2024-02-19T21:01:39.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.907792+0000 mon.b (mon.1) 27 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-19T21:01:39.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.907948+0000 mon.b (mon.1) 28 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T21:01:39.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.908110+0000 mon.b (mon.1) 29 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T21:01:39.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.908966+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-19T21:01:39.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.909132+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-19T21:01:39.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.909321+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T21:01:39.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.909526+0000 mon.b (mon.1) 33 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T21:01:39.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.909713+0000 mon.b (mon.1) 34 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T21:01:39.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.909905+0000 mon.b (mon.1) 35 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T21:01:39.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.910094+0000 mon.b (mon.1) 36 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T21:01:39.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.910287+0000 mon.b (mon.1) 37 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:01:39.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.910471+0000 mon.b (mon.1) 38 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:01:39.127 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.910661+0000 mon.b (mon.1) 39 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:01:39.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.910854+0000 mon.b (mon.1) 40 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T21:01:39.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.910985+0000 mon.b (mon.1) 41 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T21:01:39.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.911424+0000 mon.b (mon.1) 42 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T21:01:39.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:38.911868+0000 mon.a (mon.0) 628 : cluster [DBG] Standby manager daemon x restarted 2024-02-19T21:01:39.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:38.911947+0000 mon.a (mon.0) 629 : cluster [DBG] Standby manager daemon x started 2024-02-19T21:01:39.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.914047+0000 mon.a (mon.0) 630 : audit [DBG] from='mgr.? v1:172.21.15.193:0/3524108028' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-19T21:01:39.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.914509+0000 mon.a (mon.0) 631 : audit [DBG] from='mgr.? v1:172.21.15.193:0/3524108028' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T21:01:39.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.916406+0000 mon.a (mon.0) 632 : audit [DBG] from='mgr.? v1:172.21.15.193:0/3524108028' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-19T21:01:39.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:38.917360+0000 mon.a (mon.0) 633 : audit [DBG] from='mgr.? v1:172.21.15.193:0/3524108028' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T21:01:39.128 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:38.926887+0000 mon.a (mon.0) 634 : cluster [INF] Manager daemon y is now available 2024-02-19T21:01:39.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:37.951645+0000 mon.a (mon.0) 623 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-19T21:01:39.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:38.896212+0000 mon.a (mon.0) 624 : cluster [INF] Active manager daemon y restarted 2024-02-19T21:01:39.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:38.896631+0000 mon.a (mon.0) 625 : cluster [INF] Activating manager daemon y 2024-02-19T21:01:39.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:38.906413+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-19T21:01:39.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:38.906926+0000 mon.a (mon.0) 627 : cluster [DBG] mgrmap e17: y(active, starting, since 0.0103887s), standbys: x 2024-02-19T21:01:39.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.907792+0000 mon.b (mon.1) 27 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-19T21:01:39.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.907948+0000 mon.b (mon.1) 28 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T21:01:39.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.908110+0000 mon.b (mon.1) 29 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T21:01:39.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.908966+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-19T21:01:39.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.909132+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-19T21:01:39.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.909321+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T21:01:39.129 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.909526+0000 mon.b (mon.1) 33 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T21:01:39.130 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.909713+0000 mon.b (mon.1) 34 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T21:01:39.130 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.909905+0000 mon.b (mon.1) 35 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T21:01:39.130 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.910094+0000 mon.b (mon.1) 36 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T21:01:39.130 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.910287+0000 mon.b (mon.1) 37 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:01:39.130 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.910471+0000 mon.b (mon.1) 38 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:01:39.130 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.910661+0000 mon.b (mon.1) 39 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:01:39.130 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.910854+0000 mon.b (mon.1) 40 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T21:01:39.130 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.910985+0000 mon.b (mon.1) 41 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T21:01:39.130 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.911424+0000 mon.b (mon.1) 42 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T21:01:39.130 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:38.911868+0000 mon.a (mon.0) 628 : cluster [DBG] Standby manager daemon x restarted 2024-02-19T21:01:39.130 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:38.911947+0000 mon.a (mon.0) 629 : cluster [DBG] Standby manager daemon x started 2024-02-19T21:01:39.130 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.914047+0000 mon.a (mon.0) 630 : audit [DBG] from='mgr.? v1:172.21.15.193:0/3524108028' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-19T21:01:39.130 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.914509+0000 mon.a (mon.0) 631 : audit [DBG] from='mgr.? v1:172.21.15.193:0/3524108028' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T21:01:39.130 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.916406+0000 mon.a (mon.0) 632 : audit [DBG] from='mgr.? v1:172.21.15.193:0/3524108028' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-19T21:01:39.131 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:38.917360+0000 mon.a (mon.0) 633 : audit [DBG] from='mgr.? v1:172.21.15.193:0/3524108028' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T21:01:39.131 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:38.926887+0000 mon.a (mon.0) 634 : cluster [INF] Manager daemon y is now available 2024-02-19T21:01:39.131 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:01:39] ENGINE Bus STARTING 2024-02-19T21:01:39.131 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [80B blob data] 2024-02-19T21:01:39.131 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [80B blob data] 2024-02-19T21:01:39.137 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: [19/Feb/2024:21:01:38] ENGINE Bus STARTING 2024-02-19T21:01:39.137 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: CherryPy Checker: 2024-02-19T21:01:39.137 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: The Application mounted at '' has an empty config. 2024-02-19T21:01:39.138 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: 2024-02-19T21:01:39.138 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: [19/Feb/2024:21:01:39] ENGINE Serving on http://:::9283 2024-02-19T21:01:39.138 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:01:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: [19/Feb/2024:21:01:39] ENGINE Bus STARTED 2024-02-19T21:01:39.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:37.951645+0000 mon.a (mon.0) 623 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-02-19T21:01:39.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:38.896212+0000 mon.a (mon.0) 624 : cluster [INF] Active manager daemon y restarted 2024-02-19T21:01:39.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:38.896631+0000 mon.a (mon.0) 625 : cluster [INF] Activating manager daemon y 2024-02-19T21:01:39.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:38.906413+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-02-19T21:01:39.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:38.906926+0000 mon.a (mon.0) 627 : cluster [DBG] mgrmap e17: y(active, starting, since 0.0103887s), standbys: x 2024-02-19T21:01:39.138 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.907792+0000 mon.b (mon.1) 27 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-19T21:01:39.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.907948+0000 mon.b (mon.1) 28 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-19T21:01:39.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.908110+0000 mon.b (mon.1) 29 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-19T21:01:39.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.908966+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-19T21:01:39.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.909132+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-19T21:01:39.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.909321+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T21:01:39.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.909526+0000 mon.b (mon.1) 33 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T21:01:39.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.909713+0000 mon.b (mon.1) 34 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T21:01:39.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.909905+0000 mon.b (mon.1) 35 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T21:01:39.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.910094+0000 mon.b (mon.1) 36 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T21:01:39.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.910287+0000 mon.b (mon.1) 37 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T21:01:39.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.910471+0000 mon.b (mon.1) 38 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-19T21:01:39.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.910661+0000 mon.b (mon.1) 39 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-19T21:01:39.139 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.910854+0000 mon.b (mon.1) 40 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T21:01:39.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.910985+0000 mon.b (mon.1) 41 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T21:01:39.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.911424+0000 mon.b (mon.1) 42 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T21:01:39.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:38.911868+0000 mon.a (mon.0) 628 : cluster [DBG] Standby manager daemon x restarted 2024-02-19T21:01:39.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:38.911947+0000 mon.a (mon.0) 629 : cluster [DBG] Standby manager daemon x started 2024-02-19T21:01:39.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.914047+0000 mon.a (mon.0) 630 : audit [DBG] from='mgr.? v1:172.21.15.193:0/3524108028' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-19T21:01:39.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.914509+0000 mon.a (mon.0) 631 : audit [DBG] from='mgr.? v1:172.21.15.193:0/3524108028' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T21:01:39.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.916406+0000 mon.a (mon.0) 632 : audit [DBG] from='mgr.? v1:172.21.15.193:0/3524108028' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-19T21:01:39.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:38.917360+0000 mon.a (mon.0) 633 : audit [DBG] from='mgr.? v1:172.21.15.193:0/3524108028' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T21:01:39.140 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:38.926887+0000 mon.a (mon.0) 634 : cluster [INF] Manager daemon y is now available 2024-02-19T21:01:39.383 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.393+0000 7f588180f700 10 client.14556.objecter tick 2024-02-19T21:01:39.487 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: CherryPy Checker: 2024-02-19T21:01:39.487 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: The Application mounted at '' has an empty config. 2024-02-19T21:01:39.487 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: 2024-02-19T21:01:39.487 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:01:39] ENGINE Serving on http://:::9283 2024-02-19T21:01:39.487 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:01:39] ENGINE Bus STARTED 2024-02-19T21:01:39.895 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.905+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mon.0 v1:172.21.15.163:6789/0 11 ==== mgrmap(e 18) v1 ==== 76894+0+0 (unknown 1679436957 0 0) 0x7f586c073cc0 con 0x7f587c0f5ad0 2024-02-19T21:01:39.895 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.906+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi193=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f5868000b90 con 0x7f586406e2f0 2024-02-19T21:01:39.897 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.907+0000 7f5878ff9700 10 client.14556.objecter ms_handle_connect 0x7f586406e2f0 2024-02-19T21:01:39.906 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.916+0000 7f5878ff9700 1 -- 172.21.15.193:0/1976425738 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (unknown 0 0 664801700) 0x7f5868000b90 con 0x7f586406e2f0 2024-02-19T21:01:39.906 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled grafana update... 2024-02-19T21:01:39.910 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.919+0000 7f5882010700 10 librados: watch_flush enter 2024-02-19T21:01:39.910 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.919+0000 7f5882010700 10 librados: watch_flush exit 2024-02-19T21:01:39.910 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.919+0000 7f5882010700 20 client.14556.objecter shutdown clearing up homeless session... 2024-02-19T21:01:39.910 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.919+0000 7f5882010700 10 client.14556.objecter successfully canceled tick 2024-02-19T21:01:39.910 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.920+0000 7f5882010700 1 -- 172.21.15.193:0/1976425738 >> v1:172.21.15.163:6800/1552514681 conn(0x7f586406e2f0 legacy=0x7f58640706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:39.910 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.920+0000 7f5882010700 1 -- 172.21.15.193:0/1976425738 >> v1:172.21.15.163:6789/0 conn(0x7f587c0f5ad0 legacy=0x7f587c1078f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:39.910 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.920+0000 7f587bfff700 1 -- 172.21.15.193:0/1976425738 reap_dead start 2024-02-19T21:01:39.910 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.920+0000 7f5882010700 1 -- 172.21.15.193:0/1976425738 shutdown_connections 2024-02-19T21:01:39.910 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.920+0000 7f5882010700 1 -- 172.21.15.193:0/1976425738 >> 172.21.15.193:0/1976425738 conn(0x7f587c0f1610 msgr2=0x7f587c0f19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:39.910 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.920+0000 7f5882010700 1 -- 172.21.15.193:0/1976425738 shutdown_connections 2024-02-19T21:01:39.910 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.920+0000 7f5882010700 1 -- 172.21.15.193:0/1976425738 wait complete. 2024-02-19T21:01:39.911 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:39.920+0000 7f5882010700 1 librados: shutdown 2024-02-19T21:01:40.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:39.025830+0000 mon.b (mon.1) 43 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:40.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:39.027195+0000 mon.b (mon.1) 44 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:40.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:39.054752+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:40.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:39.071754+0000 mon.b (mon.1) 45 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:01:40.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:39.072316+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:01:40.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:39.090162+0000 mon.b (mon.1) 46 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-19T21:01:40.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:39.090605+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-19T21:01:40.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:39.907707+0000 mon.a (mon.0) 638 : cluster [DBG] mgrmap e18: y(active, since 1.01116s), standbys: x 2024-02-19T21:01:40.237 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:39.916500+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:40.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:39.025830+0000 mon.b (mon.1) 43 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:40.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:39.027195+0000 mon.b (mon.1) 44 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:40.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:39.054752+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:40.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:39.071754+0000 mon.b (mon.1) 45 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:01:40.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:39.072316+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:01:40.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:39.090162+0000 mon.b (mon.1) 46 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-19T21:01:40.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:39.090605+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-19T21:01:40.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:39.907707+0000 mon.a (mon.0) 638 : cluster [DBG] mgrmap e18: y(active, since 1.01116s), standbys: x 2024-02-19T21:01:40.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:39.916500+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:40.310 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:01:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:01:40.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:39.025830+0000 mon.b (mon.1) 43 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:40.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:39.027195+0000 mon.b (mon.1) 44 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:01:40.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:39.054752+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:40.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:39.071754+0000 mon.b (mon.1) 45 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:01:40.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:39.072316+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-19T21:01:40.311 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:39.090162+0000 mon.b (mon.1) 46 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-19T21:01:40.311 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:39.090605+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-19T21:01:40.311 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:39.907707+0000 mon.a (mon.0) 638 : cluster [DBG] mgrmap e18: y(active, since 1.01116s), standbys: x 2024-02-19T21:01:40.311 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:39.916500+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:40.965 DEBUG:teuthology.orchestra.run.smithi193:grafana.a> sudo journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@grafana.a.service 2024-02-19T21:01:40.967 INFO:tasks.cephadm:Setting up client nodes... 2024-02-19T21:01:40.967 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-19T21:01:40.998 INFO:journalctl@ceph.grafana.a.smithi193.stdout:-- Logs begin at Mon 2024-02-19 20:44:17 UTC. -- 2024-02-19T21:01:41.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:39.912856+0000 mgr.y (mgr.14550) 4 : cephadm [INF] Saving service grafana spec with placement smithi193=a;count:1 2024-02-19T21:01:41.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:40.518463+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:41.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:40.800004+0000 mon.b (mon.1) 47 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:40.800472+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:40.801206+0000 mon.b (mon.1) 48 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:40.801482+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:40.802021+0000 mon.b (mon.1) 49 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:40.802290+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:40.802768+0000 mon.b (mon.1) 50 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:40.803026+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.738 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:40.807928+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:41.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:39.912856+0000 mgr.y (mgr.14550) 4 : cephadm [INF] Saving service grafana spec with placement smithi193=a;count:1 2024-02-19T21:01:41.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:40.518463+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:41.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:40.800004+0000 mon.b (mon.1) 47 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:40.800472+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:40.801206+0000 mon.b (mon.1) 48 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:40.801482+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:40.802021+0000 mon.b (mon.1) 49 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:40.802290+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:40.802768+0000 mon.b (mon.1) 50 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:40.803026+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:40.807928+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:41.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:39.912856+0000 mgr.y (mgr.14550) 4 : cephadm [INF] Saving service grafana spec with placement smithi193=a;count:1 2024-02-19T21:01:41.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:40.518463+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:41.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:40.800004+0000 mon.b (mon.1) 47 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:40.800472+0000 mon.a (mon.0) 641 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:40.801206+0000 mon.b (mon.1) 48 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:40.801482+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:40.802021+0000 mon.b (mon.1) 49 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:40.802290+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:40.802768+0000 mon.b (mon.1) 50 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:40.803026+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:41.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:40.807928+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:40.803528+0000 mgr.y (mgr.14550) 5 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 3768M 2024-02-19T21:01:42.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:40.910435+0000 mgr.y (mgr.14550) 6 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:42.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:41.743960+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:41.912197+0000 mon.a (mon.0) 647 : cluster [DBG] mgrmap e19: y(active, since 3s), standbys: x 2024-02-19T21:01:42.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:42.031702+0000 mon.b (mon.1) 51 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:42.032182+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:42.033019+0000 mon.b (mon.1) 52 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:42.033387+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:42.034309+0000 mon.b (mon.1) 53 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:42.034670+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:42.035394+0000 mon.b (mon.1) 54 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:42.035727+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:42.040450+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.608 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:42.041460+0000 mon.b (mon.1) 55 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:42.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:42.042097+0000 mon.b (mon.1) 56 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:42.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:42.104295+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:42.126078+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:42.130967+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:40.803528+0000 mgr.y (mgr.14550) 5 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 3768M 2024-02-19T21:01:42.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:40.910435+0000 mgr.y (mgr.14550) 6 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:42.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:41.743960+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:41.912197+0000 mon.a (mon.0) 647 : cluster [DBG] mgrmap e19: y(active, since 3s), standbys: x 2024-02-19T21:01:42.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:42.031702+0000 mon.b (mon.1) 51 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:42.032182+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:42.033019+0000 mon.b (mon.1) 52 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.610 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:42.033387+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.610 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:42.034309+0000 mon.b (mon.1) 53 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.610 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:42.034670+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.610 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:42.035394+0000 mon.b (mon.1) 54 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.610 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:42.035727+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.610 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:42.040450+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.610 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:42.041460+0000 mon.b (mon.1) 55 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:42.610 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:42.042097+0000 mon.b (mon.1) 56 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:42.610 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:42.104295+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.610 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:42.126078+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.610 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:42.130967+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:40.803528+0000 mgr.y (mgr.14550) 5 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 3768M 2024-02-19T21:01:42.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:40.910435+0000 mgr.y (mgr.14550) 6 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:42.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:41.743960+0000 mon.a (mon.0) 646 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:41.912197+0000 mon.a (mon.0) 647 : cluster [DBG] mgrmap e19: y(active, since 3s), standbys: x 2024-02-19T21:01:42.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:42.031702+0000 mon.b (mon.1) 51 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:42.032182+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:42.033019+0000 mon.b (mon.1) 52 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:42.033387+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:42.034309+0000 mon.b (mon.1) 53 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:42.034670+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:42.035394+0000 mon.b (mon.1) 54 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:42.035727+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:01:42.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:42.040450+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:42.041460+0000 mon.b (mon.1) 55 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:01:42.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:42.042097+0000 mon.b (mon.1) 56 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:01:42.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:42.104295+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:42.126078+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:42.130967+0000 mon.a (mon.0) 655 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:42.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.889+0000 7f393d2dc700 1 -- 172.21.15.163:0/3661743188 >> v1:172.21.15.163:6790/0 conn(0x7f39381041b0 legacy=0x7f3938106630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:42.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.889+0000 7f393d2dc700 1 -- 172.21.15.163:0/3661743188 shutdown_connections 2024-02-19T21:01:42.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.889+0000 7f393d2dc700 1 -- 172.21.15.163:0/3661743188 >> 172.21.15.163:0/3661743188 conn(0x7f39380f3450 msgr2=0x7f39380f5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:42.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.889+0000 7f393d2dc700 1 -- 172.21.15.163:0/3661743188 shutdown_connections 2024-02-19T21:01:42.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.889+0000 7f393d2dc700 1 -- 172.21.15.163:0/3661743188 wait complete. 2024-02-19T21:01:42.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.889+0000 7f393d2dc700 1 librados: starting msgr at 2024-02-19T21:01:42.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.889+0000 7f393d2dc700 1 librados: starting objecter 2024-02-19T21:01:42.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f393d2dc700 1 Processor -- start 2024-02-19T21:01:42.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f393d2dc700 1 -- start start 2024-02-19T21:01:42.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f393d2dc700 1 librados: setting wanted keys 2024-02-19T21:01:42.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f393d2dc700 1 librados: calling monclient init 2024-02-19T21:01:42.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f393d2dc700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3938117270 con 0x7f39381041b0 2024-02-19T21:01:42.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f393d2dc700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3938118430 con 0x7f3938100670 2024-02-19T21:01:42.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f393d2dc700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3938119630 con 0x7f39380fd870 2024-02-19T21:01:42.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f39377fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f39381041b0 0x7f3938115930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:45842/0 (socket says 172.21.15.163:45842) 2024-02-19T21:01:42.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f3936ffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f39380fd870 0x7f39380f9850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38910/0 (socket says 172.21.15.163:38910) 2024-02-19T21:01:42.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f39377fe700 1 -- 172.21.15.163:0/2308482374 learned_addr learned my addr 172.21.15.163:0/2308482374 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:01:42.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f391ffff700 10 client.?.objecter ms_handle_connect 0x7f39380fd870 2024-02-19T21:01:42.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f391ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:42.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f391ffff700 10 client.?.objecter ms_handle_connect 0x7f39381041b0 2024-02-19T21:01:42.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f391ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:42.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f391ffff700 10 client.?.objecter ms_handle_connect 0x7f3938100670 2024-02-19T21:01:42.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f391ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:42.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158875199 0 0) 0x7f3938119630 con 0x7f39380fd870 2024-02-19T21:01:42.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39200033f0 con 0x7f39380fd870 2024-02-19T21:01:42.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720959900 0 0) 0x7f3938117270 con 0x7f39381041b0 2024-02-19T21:01:42.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3938119630 con 0x7f39381041b0 2024-02-19T21:01:42.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232628870 0 0) 0x7f3938118430 con 0x7f3938100670 2024-02-19T21:01:42.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.890+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3938117270 con 0x7f3938100670 2024-02-19T21:01:42.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1480298067 0 0) 0x7f39200033f0 con 0x7f39380fd870 2024-02-19T21:01:42.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3938118430 con 0x7f39380fd870 2024-02-19T21:01:42.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f392c001940 con 0x7f39380fd870 2024-02-19T21:01:42.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1305012361 0 0) 0x7f3938119630 con 0x7f39381041b0 2024-02-19T21:01:42.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39200033f0 con 0x7f39381041b0 2024-02-19T21:01:42.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3930002310 con 0x7f39381041b0 2024-02-19T21:01:42.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4235488673 0 0) 0x7f3938117270 con 0x7f3938100670 2024-02-19T21:01:42.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3938119630 con 0x7f3938100670 2024-02-19T21:01:42.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f39280019f0 con 0x7f3938100670 2024-02-19T21:01:42.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3087985198 0 0) 0x7f3938118430 con 0x7f39380fd870 2024-02-19T21:01:42.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 >> v1:172.21.15.193:6789/0 conn(0x7f3938100670 legacy=0x7f39380f9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:42.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 >> v1:172.21.15.163:6789/0 conn(0x7f39381041b0 legacy=0x7f3938115930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:42.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393811a830 con 0x7f39380fd870 2024-02-19T21:01:42.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f393d2dc700 1 -- 172.21.15.163:0/2308482374 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3938117460 con 0x7f39380fd870 2024-02-19T21:01:42.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f393d2dc700 10 client.34499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:01:42.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f393d2dc700 1 -- 172.21.15.163:0/2308482374 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39381178d0 con 0x7f39380fd870 2024-02-19T21:01:42.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f393d2dc700 1 librados: init done 2024-02-19T21:01:42.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f392c002cc0 con 0x7f39380fd870 2024-02-19T21:01:42.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.891+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f392c005b50 con 0x7f39380fd870 2024-02-19T21:01:42.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.892+0000 7f393d2dc700 1 -- 172.21.15.163:0/2308482374 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3924004f20 con 0x7f39380fd870 2024-02-19T21:01:42.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.892+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f392c0189e0 con 0x7f39380fd870 2024-02-19T21:01:42.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.893+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6172+0+0 (unknown 109389188 0 0) 0x7f392c0787b0 con 0x7f39380fd870 2024-02-19T21:01:42.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.893+0000 7f391ffff700 10 client.34499.objecter ms_dispatch 0x7f3938001550 osd_map(72..72 src has 1..72) v4 2024-02-19T21:01:42.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.893+0000 7f391ffff700 3 client.34499.objecter handle_osd_map got epochs [72,72] > 0 2024-02-19T21:01:42.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.893+0000 7f391ffff700 3 client.34499.objecter handle_osd_map decoding full epoch 72 2024-02-19T21:01:42.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.893+0000 7f391ffff700 20 client.34499.objecter dump_active .. 0 homeless 2024-02-19T21:01:42.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.893+0000 7f391ffff700 10 client.34499.objecter ms_handle_connect 0x7f3920063d00 2024-02-19T21:01:42.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:42.897+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f392c001f20 con 0x7f39380fd870 2024-02-19T21:01:43.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:43.127+0000 7f393d2dc700 1 -- 172.21.15.163:0/2308482374 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f3924005750 con 0x7f39380fd870 2024-02-19T21:01:43.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:43.134+0000 7f391ffff700 1 -- 172.21.15.163:0/2308482374 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (unknown 57899324 0 2729476309) 0x7f392c051e80 con 0x7f39380fd870 2024-02-19T21:01:43.123 INFO:teuthology.orchestra.run.smithi163.stdout:[client.0] 2024-02-19T21:01:43.123 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQC3wdNlRGq0BxAASoeAxWBRWGBejWKAtrUPng== 2024-02-19T21:01:43.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:43.136+0000 7f393d2dc700 10 librados: watch_flush enter 2024-02-19T21:01:43.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:43.136+0000 7f393d2dc700 10 librados: watch_flush exit 2024-02-19T21:01:43.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:43.136+0000 7f393d2dc700 20 client.34499.objecter shutdown clearing up homeless session... 2024-02-19T21:01:43.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:43.136+0000 7f393d2dc700 10 client.34499.objecter successfully canceled tick 2024-02-19T21:01:43.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:43.136+0000 7f393d2dc700 1 -- 172.21.15.163:0/2308482374 >> v1:172.21.15.163:6800/1552514681 conn(0x7f3920063d00 legacy=0x7f39200661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:43.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:43.136+0000 7f393d2dc700 1 -- 172.21.15.163:0/2308482374 >> v1:172.21.15.163:6790/0 conn(0x7f39380fd870 legacy=0x7f39380f9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:43.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:43.136+0000 7f393d2dc700 1 -- 172.21.15.163:0/2308482374 shutdown_connections 2024-02-19T21:01:43.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:43.136+0000 7f393d2dc700 1 -- 172.21.15.163:0/2308482374 >> 172.21.15.163:0/2308482374 conn(0x7f39380f3450 msgr2=0x7f3938107620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:43.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:43.136+0000 7f393d2dc700 1 -- 172.21.15.163:0/2308482374 shutdown_connections 2024-02-19T21:01:43.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:43.136+0000 7f393d2dc700 1 -- 172.21.15.163:0/2308482374 wait complete. 2024-02-19T21:01:43.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:43.136+0000 7f393d2dc700 1 librados: shutdown 2024-02-19T21:01:43.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:42.036381+0000 mgr.y (mgr.14550) 7 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 4024M 2024-02-19T21:01:43.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:42.043206+0000 mgr.y (mgr.14550) 8 : cephadm [INF] Updating smithi163:/etc/ceph/ceph.conf 2024-02-19T21:01:43.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:42.043428+0000 mgr.y (mgr.14550) 9 : cephadm [INF] Updating smithi193:/etc/ceph/ceph.conf 2024-02-19T21:01:43.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:42.069872+0000 mgr.y (mgr.14550) 10 : cephadm [INF] Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-02-19T21:01:43.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:42.073848+0000 mgr.y (mgr.14550) 11 : cephadm [INF] Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-02-19T21:01:43.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:42.142682+0000 mgr.y (mgr.14550) 12 : cephadm [INF] Deploying daemon alertmanager.a on smithi163 2024-02-19T21:01:43.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:43.128748+0000 mon.c (mon.2) 18 : audit [INF] from='client.? v1:172.21.15.163:0/2308482374' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T21:01:43.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:43.129108+0000 mon.a (mon.0) 656 : audit [INF] from='client.34499 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T21:01:43.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:43.132461+0000 mon.a (mon.0) 657 : audit [INF] from='client.34499 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T21:01:43.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:42.036381+0000 mgr.y (mgr.14550) 7 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 4024M 2024-02-19T21:01:43.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:42.043206+0000 mgr.y (mgr.14550) 8 : cephadm [INF] Updating smithi163:/etc/ceph/ceph.conf 2024-02-19T21:01:43.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:42.043428+0000 mgr.y (mgr.14550) 9 : cephadm [INF] Updating smithi193:/etc/ceph/ceph.conf 2024-02-19T21:01:43.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:42.069872+0000 mgr.y (mgr.14550) 10 : cephadm [INF] Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-02-19T21:01:43.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:42.073848+0000 mgr.y (mgr.14550) 11 : cephadm [INF] Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-02-19T21:01:43.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:42.142682+0000 mgr.y (mgr.14550) 12 : cephadm [INF] Deploying daemon alertmanager.a on smithi163 2024-02-19T21:01:43.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:43.128748+0000 mon.c (mon.2) 18 : audit [INF] from='client.? v1:172.21.15.163:0/2308482374' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T21:01:43.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:43.129108+0000 mon.a (mon.0) 656 : audit [INF] from='client.34499 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T21:01:43.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:43.132461+0000 mon.a (mon.0) 657 : audit [INF] from='client.34499 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T21:01:43.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:42.036381+0000 mgr.y (mgr.14550) 7 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 4024M 2024-02-19T21:01:43.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:42.043206+0000 mgr.y (mgr.14550) 8 : cephadm [INF] Updating smithi163:/etc/ceph/ceph.conf 2024-02-19T21:01:43.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:42.043428+0000 mgr.y (mgr.14550) 9 : cephadm [INF] Updating smithi193:/etc/ceph/ceph.conf 2024-02-19T21:01:43.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:42.069872+0000 mgr.y (mgr.14550) 10 : cephadm [INF] Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-02-19T21:01:43.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:42.073848+0000 mgr.y (mgr.14550) 11 : cephadm [INF] Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-02-19T21:01:43.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:42.142682+0000 mgr.y (mgr.14550) 12 : cephadm [INF] Deploying daemon alertmanager.a on smithi163 2024-02-19T21:01:43.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:43.128748+0000 mon.c (mon.2) 18 : audit [INF] from='client.? v1:172.21.15.163:0/2308482374' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T21:01:43.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:43.129108+0000 mon.a (mon.0) 656 : audit [INF] from='client.34499 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T21:01:43.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:43.132461+0000 mon.a (mon.0) 657 : audit [INF] from='client.34499 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T21:01:44.098 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T21:01:44.098 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-19T21:01:44.098 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-19T21:01:44.140 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-19T21:01:44.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:42.910715+0000 mgr.y (mgr.14550) 13 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:44.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:42.910715+0000 mgr.y (mgr.14550) 13 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:44.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:42.910715+0000 mgr.y (mgr.14550) 13 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:01:46.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.131+0000 7f01ad5e2700 1 -- 172.21.15.193:0/1036164508 >> v1:172.21.15.193:6789/0 conn(0x7f01a80fd870 legacy=0x7f01a80fdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:46.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.131+0000 7f01ad5e2700 1 -- 172.21.15.193:0/1036164508 shutdown_connections 2024-02-19T21:01:46.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.131+0000 7f01ad5e2700 1 -- 172.21.15.193:0/1036164508 >> 172.21.15.193:0/1036164508 conn(0x7f01a80f3490 msgr2=0x7f01a80f58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:46.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.131+0000 7f01ad5e2700 1 -- 172.21.15.193:0/1036164508 shutdown_connections 2024-02-19T21:01:46.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.131+0000 7f01ad5e2700 1 -- 172.21.15.193:0/1036164508 wait complete. 2024-02-19T21:01:46.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.132+0000 7f01ad5e2700 1 librados: starting msgr at 2024-02-19T21:01:46.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.132+0000 7f01ad5e2700 1 librados: starting objecter 2024-02-19T21:01:46.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.132+0000 7f01ad5e2700 1 Processor -- start 2024-02-19T21:01:46.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.132+0000 7f01ad5e2700 1 -- start start 2024-02-19T21:01:46.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.132+0000 7f01ad5e2700 1 librados: setting wanted keys 2024-02-19T21:01:46.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.132+0000 7f01ad5e2700 1 librados: calling monclient init 2024-02-19T21:01:46.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.132+0000 7f01ad5e2700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f01a8117330 con 0x7f01a80fd870 2024-02-19T21:01:46.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.132+0000 7f01ad5e2700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f01a8118510 con 0x7f01a8100670 2024-02-19T21:01:46.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.132+0000 7f01ad5e2700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f01a8119710 con 0x7f01a81041b0 2024-02-19T21:01:46.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f01a67fc700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f01a8100670 0x7f01a80fa2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:54408/0 (socket says 172.21.15.193:54408) 2024-02-19T21:01:46.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f01a67fc700 1 -- 172.21.15.193:0/611215779 learned_addr learned my addr 172.21.15.193:0/611215779 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-02-19T21:01:46.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 10 client.?.objecter ms_handle_connect 0x7f01a8100670 2024-02-19T21:01:46.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:46.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 10 client.?.objecter ms_handle_connect 0x7f01a80fd870 2024-02-19T21:01:46.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:46.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 10 client.?.objecter ms_handle_connect 0x7f01a81041b0 2024-02-19T21:01:46.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:46.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29115937 0 0) 0x7f01a8118510 con 0x7f01a8100670 2024-02-19T21:01:46.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01900033f0 con 0x7f01a8100670 2024-02-19T21:01:46.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463108998 0 0) 0x7f01a8117330 con 0x7f01a80fd870 2024-02-19T21:01:46.126 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a8118510 con 0x7f01a80fd870 2024-02-19T21:01:46.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028620195 0 0) 0x7f01a8119710 con 0x7f01a81041b0 2024-02-19T21:01:46.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a8117330 con 0x7f01a81041b0 2024-02-19T21:01:46.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3999407914 0 0) 0x7f01900033f0 con 0x7f01a8100670 2024-02-19T21:01:46.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01a8119710 con 0x7f01a8100670 2024-02-19T21:01:46.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f019c001980 con 0x7f01a8100670 2024-02-19T21:01:46.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4152622481 0 0) 0x7f01a8117330 con 0x7f01a81041b0 2024-02-19T21:01:46.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01900033f0 con 0x7f01a81041b0 2024-02-19T21:01:46.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3422046589 0 0) 0x7f01a8118510 con 0x7f01a80fd870 2024-02-19T21:01:46.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01a8117330 con 0x7f01a80fd870 2024-02-19T21:01:46.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f01a0001bc0 con 0x7f01a81041b0 2024-02-19T21:01:46.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0198001ec0 con 0x7f01a80fd870 2024-02-19T21:01:46.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2565389001 0 0) 0x7f01a8119710 con 0x7f01a8100670 2024-02-19T21:01:46.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 >> v1:172.21.15.163:6790/0 conn(0x7f01a81041b0 legacy=0x7f01a81159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:46.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 >> v1:172.21.15.163:6789/0 conn(0x7f01a80fd870 legacy=0x7f01a80f9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:46.127 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a811a910 con 0x7f01a8100670 2024-02-19T21:01:46.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f01ad5e2700 1 -- 172.21.15.193:0/611215779 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a8117520 con 0x7f01a8100670 2024-02-19T21:01:46.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f01ad5e2700 10 client.24511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:01:46.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f01ad5e2700 1 -- 172.21.15.193:0/611215779 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a8117a10 con 0x7f01a8100670 2024-02-19T21:01:46.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f01ad5e2700 1 librados: init done 2024-02-19T21:01:46.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f019c002100 con 0x7f01a8100670 2024-02-19T21:01:46.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.133+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f019c005ae0 con 0x7f01a8100670 2024-02-19T21:01:46.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.134+0000 7f01ad5e2700 1 -- 172.21.15.193:0/611215779 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0194004f20 con 0x7f01a8100670 2024-02-19T21:01:46.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.134+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f019c018970 con 0x7f01a8100670 2024-02-19T21:01:46.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.135+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6172+0+0 (unknown 109389188 0 0) 0x7f019c078530 con 0x7f01a8100670 2024-02-19T21:01:46.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.135+0000 7f018ffff700 10 client.24511.objecter ms_dispatch 0x7f01a8001550 osd_map(72..72 src has 1..72) v4 2024-02-19T21:01:46.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.135+0000 7f018ffff700 3 client.24511.objecter handle_osd_map got epochs [72,72] > 0 2024-02-19T21:01:46.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.135+0000 7f018ffff700 3 client.24511.objecter handle_osd_map decoding full epoch 72 2024-02-19T21:01:46.129 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.135+0000 7f018ffff700 20 client.24511.objecter dump_active .. 0 homeless 2024-02-19T21:01:46.129 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.135+0000 7f018ffff700 10 client.24511.objecter ms_handle_connect 0x7f0190063dc0 2024-02-19T21:01:46.129 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.139+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f019c048410 con 0x7f01a8100670 2024-02-19T21:01:46.357 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.368+0000 7f01ad5e2700 1 -- 172.21.15.193:0/611215779 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f0194005750 con 0x7f01a8100670 2024-02-19T21:01:46.362 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.372+0000 7f018ffff700 1 -- 172.21.15.193:0/611215779 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) v1 ==== 170+0+59 (unknown 3508017507 0 4017691069) 0x7f019c04bc10 con 0x7f01a8100670 2024-02-19T21:01:46.362 INFO:teuthology.orchestra.run.smithi193.stdout:[client.1] 2024-02-19T21:01:46.362 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQC6wdNlRJkTFhAAf10Vv8jTzJuysuMagW/I5w== 2024-02-19T21:01:46.364 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.374+0000 7f01ad5e2700 10 librados: watch_flush enter 2024-02-19T21:01:46.364 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.374+0000 7f01ad5e2700 10 librados: watch_flush exit 2024-02-19T21:01:46.364 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.374+0000 7f01ad5e2700 20 client.24511.objecter shutdown clearing up homeless session... 2024-02-19T21:01:46.364 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.374+0000 7f01ad5e2700 10 client.24511.objecter successfully canceled tick 2024-02-19T21:01:46.364 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.374+0000 7f01ad5e2700 1 -- 172.21.15.193:0/611215779 >> v1:172.21.15.163:6800/1552514681 conn(0x7f0190063dc0 legacy=0x7f0190066280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:46.364 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.374+0000 7f01ad5e2700 1 -- 172.21.15.193:0/611215779 >> v1:172.21.15.193:6789/0 conn(0x7f01a8100670 legacy=0x7f01a80fa2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:46.364 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.374+0000 7f01ad5e2700 1 -- 172.21.15.193:0/611215779 shutdown_connections 2024-02-19T21:01:46.364 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.374+0000 7f01ad5e2700 1 -- 172.21.15.193:0/611215779 >> 172.21.15.193:0/611215779 conn(0x7f01a80f3490 msgr2=0x7f01a8107600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:46.364 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.374+0000 7f01ad5e2700 1 -- 172.21.15.193:0/611215779 shutdown_connections 2024-02-19T21:01:46.365 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.374+0000 7f01ad5e2700 1 -- 172.21.15.193:0/611215779 wait complete. 2024-02-19T21:01:46.365 INFO:teuthology.orchestra.run.smithi193.stderr:2024-02-19T21:01:46.374+0000 7f01ad5e2700 1 librados: shutdown 2024-02-19T21:01:46.740 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:44.911393+0000 mgr.y (mgr.14550) 14 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-02-19T21:01:46.741 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:46.369673+0000 mon.b (mon.1) 57 : audit [INF] from='client.? v1:172.21.15.193:0/611215779' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T21:01:46.741 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:46.370227+0000 mon.a (mon.0) 658 : audit [INF] from='client.24511 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T21:01:46.741 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:46.373411+0000 mon.a (mon.0) 659 : audit [INF] from='client.24511 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T21:01:46.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:44.911393+0000 mgr.y (mgr.14550) 14 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-02-19T21:01:46.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:46.369673+0000 mon.b (mon.1) 57 : audit [INF] from='client.? v1:172.21.15.193:0/611215779' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T21:01:46.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:46.370227+0000 mon.a (mon.0) 658 : audit [INF] from='client.24511 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T21:01:46.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:46.373411+0000 mon.a (mon.0) 659 : audit [INF] from='client.24511 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T21:01:46.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:44.911393+0000 mgr.y (mgr.14550) 14 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-02-19T21:01:46.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:46.369673+0000 mon.b (mon.1) 57 : audit [INF] from='client.? v1:172.21.15.193:0/611215779' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T21:01:46.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:46.370227+0000 mon.a (mon.0) 658 : audit [INF] from='client.24511 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T21:01:46.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:46.373411+0000 mon.a (mon.0) 659 : audit [INF] from='client.24511 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T21:01:47.025 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-02-19T21:01:47.025 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-19T21:01:47.025 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-19T21:01:47.067 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-19T21:01:48.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:46.911906+0000 mgr.y (mgr.14550) 15 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-02-19T21:01:48.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:48.256657+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:01:48.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:48.259808+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:01:48.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:46.911906+0000 mgr.y (mgr.14550) 15 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-02-19T21:01:48.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:48.256657+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:01:48.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:48.259808+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:01:48.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:46.911906+0000 mgr.y (mgr.14550) 15 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-02-19T21:01:48.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:48.256657+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:01:48.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:48.259808+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:01:49.068 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:01:50.176 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:01:50 smithi163 systemd[1]: Starting Ceph alertmanager.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:01:50.310 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:01:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:01:50.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:48.912365+0000 mgr.y (mgr.14550) 16 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 568 B/s rd, 0 op/s 2024-02-19T21:01:50.902 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:48.912365+0000 mgr.y (mgr.14550) 16 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 568 B/s rd, 0 op/s 2024-02-19T21:01:50.903 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:48.912365+0000 mgr.y (mgr.14550) 16 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 568 B/s rd, 0 op/s 2024-02-19T21:01:51.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.411+0000 7f176af5d700 1 -- 172.21.15.163:0/2572957966 >> v1:172.21.15.163:6790/0 conn(0x7f17640fc870 legacy=0x7f17640fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:51.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.411+0000 7f176af5d700 1 -- 172.21.15.163:0/2572957966 shutdown_connections 2024-02-19T21:01:51.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.411+0000 7f176af5d700 1 -- 172.21.15.163:0/2572957966 >> 172.21.15.163:0/2572957966 conn(0x7f17640f2470 msgr2=0x7f17640f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:51.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.411+0000 7f176af5d700 1 -- 172.21.15.163:0/2572957966 shutdown_connections 2024-02-19T21:01:51.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.411+0000 7f176af5d700 1 -- 172.21.15.163:0/2572957966 wait complete. 2024-02-19T21:01:51.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f176af5d700 1 librados: starting msgr at 2024-02-19T21:01:51.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f176af5d700 1 librados: starting objecter 2024-02-19T21:01:51.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f176af5d700 1 Processor -- start 2024-02-19T21:01:51.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f176af5d700 1 -- start start 2024-02-19T21:01:51.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f176af5d700 1 librados: setting wanted keys 2024-02-19T21:01:51.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f176af5d700 1 librados: calling monclient init 2024-02-19T21:01:51.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f176af5d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1764116240 con 0x7f17640fc870 2024-02-19T21:01:51.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f176af5d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1764117420 con 0x7f17641031b0 2024-02-19T21:01:51.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f176af5d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1764118620 con 0x7f17640ff670 2024-02-19T21:01:51.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f1768cf9700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f17640fc870 0x7f17640f8850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57920/0 (socket says 172.21.15.163:57920) 2024-02-19T21:01:51.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f1768cf9700 1 -- 172.21.15.163:0/2968889068 learned_addr learned my addr 172.21.15.163:0/2968889068 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:01:51.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f1761ffb700 10 client.?.objecter ms_handle_connect 0x7f17640ff670 2024-02-19T21:01:51.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f1761ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:51.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f1761ffb700 10 client.?.objecter ms_handle_connect 0x7f17640fc870 2024-02-19T21:01:51.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f1761ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:51.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f1761ffb700 10 client.?.objecter ms_handle_connect 0x7f17641031b0 2024-02-19T21:01:51.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f1761ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:51.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971865098 0 0) 0x7f1764118620 con 0x7f17640ff670 2024-02-19T21:01:51.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.412+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17540038e0 con 0x7f17640ff670 2024-02-19T21:01:51.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668222288 0 0) 0x7f1764116240 con 0x7f17640fc870 2024-02-19T21:01:51.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1764118620 con 0x7f17640fc870 2024-02-19T21:01:51.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474213161 0 0) 0x7f1764117420 con 0x7f17641031b0 2024-02-19T21:01:51.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1764116240 con 0x7f17641031b0 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2745366029 0 0) 0x7f17540038e0 con 0x7f17640ff670 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1764117420 con 0x7f17640ff670 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1750001620 con 0x7f17640ff670 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 86127026 0 0) 0x7f1764118620 con 0x7f17640fc870 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17540038e0 con 0x7f17640fc870 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1758002e60 con 0x7f17640fc870 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 44506562 0 0) 0x7f1764116240 con 0x7f17641031b0 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1764118620 con 0x7f17641031b0 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f175c001b80 con 0x7f17641031b0 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 485944845 0 0) 0x7f1764117420 con 0x7f17640ff670 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 >> v1:172.21.15.193:6789/0 conn(0x7f17641031b0 legacy=0x7f1764114900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 >> v1:172.21.15.163:6789/0 conn(0x7f17640fc870 legacy=0x7f17640f8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1764119820 con 0x7f17640ff670 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f176af5d700 1 -- 172.21.15.163:0/2968889068 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1764116430 con 0x7f17640ff670 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f1750002200 con 0x7f17640ff670 2024-02-19T21:01:51.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f17500031c0 con 0x7f17640ff670 2024-02-19T21:01:51.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f176af5d700 10 client.34511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:01:51.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f176af5d700 1 -- 172.21.15.163:0/2968889068 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17641168a0 con 0x7f17640ff670 2024-02-19T21:01:51.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.413+0000 7f176af5d700 1 librados: init done 2024-02-19T21:01:51.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.414+0000 7f176af5d700 1 -- 172.21.15.163:0/2968889068 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176405f700 con 0x7f17640ff670 2024-02-19T21:01:51.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.414+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f1750001d60 con 0x7f17640ff670 2024-02-19T21:01:51.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.415+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6172+0+0 (unknown 109389188 0 0) 0x7f1750075c60 con 0x7f17640ff670 2024-02-19T21:01:51.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.415+0000 7f1761ffb700 10 client.34511.objecter ms_dispatch 0x7f1764001550 osd_map(72..72 src has 1..72) v4 2024-02-19T21:01:51.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.415+0000 7f1761ffb700 3 client.34511.objecter handle_osd_map got epochs [72,72] > 0 2024-02-19T21:01:51.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.415+0000 7f1761ffb700 3 client.34511.objecter handle_osd_map decoding full epoch 72 2024-02-19T21:01:51.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.415+0000 7f1761ffb700 20 client.34511.objecter dump_active .. 0 homeless 2024-02-19T21:01:51.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.415+0000 7f1761ffb700 10 client.34511.objecter ms_handle_connect 0x7f1754063cf0 2024-02-19T21:01:51.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.419+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1750049810 con 0x7f17640ff670 2024-02-19T21:01:51.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.602+0000 7f176af5d700 1 -- 172.21.15.163:0/2968889068 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1764047540 con 0x7f17640ff670 2024-02-19T21:01:51.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.602+0000 7f1761ffb700 1 -- 172.21.15.163:0/2968889068 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+19240 (unknown 1686360691 0 3632484784) 0x7f1750047c20 con 0x7f17640ff670 2024-02-19T21:01:51.592 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:01:51.592 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":72,"fsid":"2591809a-cf69-11ee-95bb-87774f69a715","created":"2024-02-19T20:55:37.794885+0000","modified":"2024-02-19T21:01:38.896376+0000","last_up_change":"2024-02-19T21:01:09.214134+0000","last_in_change":"2024-02-19T21:00:51.699017+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-19T20:59:09.425169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-19T21:01:14.658945+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-19T21:01:14.958800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-19T21:01:16.543816+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-19T21:01:18.521482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-19T21:01:20.586564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"4a14b406-7c21-40f3-8b12-8cf9df811088","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6801","nonce":1559418115}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6802","nonce":1559418115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6804","nonce":1559418115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6803","nonce":1559418115}]},"public_addr":"172.21.15.163:6801/1559418115","cluster_addr":"172.21.15.163:6802/1559418115","heartbeat_back_addr":"172.21.15.163:6804/1559418115","heartbeat_front_addr":"172.21.15.163:6803/1559418115","state":["exists","up"]},{"osd":1,"uuid":"e74a65fa-cb9c-42c3-9293-602cf7e9dc67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6805","nonce":3125047432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6806","nonce":3125047432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6808","nonce":3125047432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6807","nonce":3125047432}]},"public_addr":"172.21.15.163:6805/3125047432","cluster_addr":"172.21.15.163:6806/3125047432","heartbeat_back_addr":"172.21.15.163:6808/3125047432","heartbeat_front_addr":"172.21.15.163:6807/3125047432","state":["exists","up"]},{"osd":2,"uuid":"6980a625-cdc9-487e-8f8c-7997ba716a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6809","nonce":1563158487}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6810","nonce":1563158487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6812","nonce":1563158487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6811","nonce":1563158487}]},"public_addr":"172.21.15.163:6809/1563158487","cluster_addr":"172.21.15.163:6810/1563158487","heartbeat_back_addr":"172.21.15.163:6812/1563158487","heartbeat_front_addr":"172.21.15.163:6811/1563158487","state":["exists","up"]},{"osd":3,"uuid":"359e2ee0-9d1d-4074-8119-2d69bc2ca3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6813","nonce":2112279769}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6814","nonce":2112279769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6816","nonce":2112279769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6815","nonce":2112279769}]},"public_addr":"172.21.15.163:6813/2112279769","cluster_addr":"172.21.15.163:6814/2112279769","heartbeat_back_addr":"172.21.15.163:6816/2112279769","heartbeat_front_addr":"172.21.15.163:6815/2112279769","state":["exists","up"]},{"osd":4,"uuid":"2bf5ef05-292b-4693-85a9-97991f2091e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6800","nonce":3129716965}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6801","nonce":3129716965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6803","nonce":3129716965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6802","nonce":3129716965}]},"public_addr":"172.21.15.193:6800/3129716965","cluster_addr":"172.21.15.193:6801/3129716965","heartbeat_back_addr":"172.21.15.193:6803/3129716965","heartbeat_front_addr":"172.21.15.193:6802/3129716965","state":["exists","up"]},{"osd":5,"uuid":"59065f9b-9240-414b-9c91-e91cda4aed6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6804","nonce":805161991}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6805","nonce":805161991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6807","nonce":805161991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6806","nonce":805161991}]},"public_addr":"172.21.15.193:6804/805161991","cluster_addr":"172.21.15.193:6805/805161991","heartbeat_back_addr":"172.21.15.193:6807/805161991","heartbeat_front_addr":"172.21.15.193:6806/805161991","state":["exists","up"]},{"osd":6,"uuid":"302c95df-3842-4818-8afd-d79d8184684f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6808","nonce":787604615}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6809","nonce":787604615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6811","nonce":787604615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6810","nonce":787604615}]},"public_addr":"172.21.15.193:6808/787604615","cluster_addr":"172.21.15.193:6809/787604615","heartbeat_back_addr":"172.21.15.193:6811/787604615","heartbeat_front_addr":"172.21.15.193:6810/787604615","state":["exists","up"]},{"osd":7,"uuid":"9ebeff27-c2f8-4a66-923f-a0d7573ddd3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6812","nonce":157927432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6813","nonce":157927432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6815","nonce":157927432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6814","nonce":157927432}]},"public_addr":"172.21.15.193:6812/157927432","cluster_addr":"172.21.15.193:6813/157927432","heartbeat_back_addr":"172.21.15.193:6815/157927432","heartbeat_front_addr":"172.21.15.193:6814/157927432","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:00:50.702191+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:11.764868+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:36.293275+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:35.739499+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:00:57.342698+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:23.163793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:00:42.575293+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:07.350223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":7,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:0/1427774891":"2024-02-20T21:01:38.896227+0000","172.21.15.163:0/3282102947":"2024-02-20T21:01:38.896227+0000","172.21.15.163:6800/2735092549":"2024-02-20T21:01:38.896227+0000","172.21.15.163:0/1235588846":"2024-02-20T20:56:29.603030+0000","172.21.15.163:6800/2731225015":"2024-02-20T20:56:29.603030+0000","172.21.15.163:0/506491651":"2024-02-20T20:56:02.185409+0000","172.21.15.163:0/168117826":"2024-02-20T21:01:38.896227+0000","172.21.15.163:0/1458009465":"2024-02-20T20:56:29.603030+0000","172.21.15.163:0/541631472":"2024-02-20T20:56:02.185409+0000","172.21.15.163:6800/2248999541":"2024-02-20T20:56:02.185409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T21:01:51.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.604+0000 7f176af5d700 10 librados: watch_flush enter 2024-02-19T21:01:51.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.604+0000 7f176af5d700 10 librados: watch_flush exit 2024-02-19T21:01:51.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.604+0000 7f176af5d700 20 client.34511.objecter shutdown clearing up homeless session... 2024-02-19T21:01:51.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.604+0000 7f176af5d700 10 client.34511.objecter successfully canceled tick 2024-02-19T21:01:51.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.604+0000 7f176af5d700 1 -- 172.21.15.163:0/2968889068 >> v1:172.21.15.163:6800/1552514681 conn(0x7f1754063cf0 legacy=0x7f17540661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:51.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.604+0000 7f176af5d700 1 -- 172.21.15.163:0/2968889068 >> v1:172.21.15.163:6790/0 conn(0x7f17640ff670 legacy=0x7f17640f8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:51.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.604+0000 7f176af5d700 1 -- 172.21.15.163:0/2968889068 shutdown_connections 2024-02-19T21:01:51.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.604+0000 7f176af5d700 1 -- 172.21.15.163:0/2968889068 >> 172.21.15.163:0/2968889068 conn(0x7f17640f2470 msgr2=0x7f1764106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:51.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.605+0000 7f176af5d700 1 -- 172.21.15.163:0/2968889068 shutdown_connections 2024-02-19T21:01:51.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.605+0000 7f176af5d700 1 -- 172.21.15.163:0/2968889068 wait complete. 2024-02-19T21:01:51.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:51.605+0000 7f176af5d700 1 librados: shutdown 2024-02-19T21:01:51.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:49.919391+0000 mgr.y (mgr.14550) 17 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:01:51.961 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:49.919391+0000 mgr.y (mgr.14550) 17 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:01:51.961 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:49.919391+0000 mgr.y (mgr.14550) 17 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:01:52.533 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-19T20:59:09.425169+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-19T21:01:14.658945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-19T21:01:14.958800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-19T21:01:16.543816+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-19T21:01:18.521482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-19T21:01:20.586564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}] 2024-02-19T21:01:52.534 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool get device_health_metrics pg_num 2024-02-19T21:01:52.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:50.912943+0000 mgr.y (mgr.14550) 18 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-02-19T21:01:52.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:51.602507+0000 mon.c (mon.2) 19 : audit [DBG] from='client.? v1:172.21.15.163:0/2968889068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T21:01:52.817 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:01:52.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:50.912943+0000 mgr.y (mgr.14550) 18 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-02-19T21:01:52.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:51.602507+0000 mon.c (mon.2) 19 : audit [DBG] from='client.? v1:172.21.15.163:0/2968889068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T21:01:52.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:50.912943+0000 mgr.y (mgr.14550) 18 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-02-19T21:01:52.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:51.602507+0000 mon.c (mon.2) 19 : audit [DBG] from='client.? v1:172.21.15.163:0/2968889068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T21:01:52.987 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:01:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[67708]: level=info ts=2024-02-19T21:01:52.669Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-19T21:01:52.987 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:01:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[67708]: level=info ts=2024-02-19T21:01:52.669Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-19T21:01:52.988 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:01:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[67708]: level=info ts=2024-02-19T21:01:52.671Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.163 port=9094 2024-02-19T21:01:52.988 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:01:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[67708]: level=info ts=2024-02-19T21:01:52.672Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-19T21:01:52.988 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:01:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[67708]: level=info ts=2024-02-19T21:01:52.697Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-19T21:01:52.988 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:01:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[67708]: level=info ts=2024-02-19T21:01:52.697Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-19T21:01:52.988 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:01:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[67708]: level=info ts=2024-02-19T21:01:52.700Z caller=main.go:518 msg=Listening address=:9093 2024-02-19T21:01:52.988 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:01:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[67708]: level=info ts=2024-02-19T21:01:52.700Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-19T21:01:52.988 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:01:52 smithi163 bash[67681]: 4ded753f36990dcf22bcb37d085b0761108ffddd88649d061d914bdb3198a1de 2024-02-19T21:01:52.988 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:01:52 smithi163 systemd[1]: Started Ceph alertmanager.a for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T21:01:53.715 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:53 smithi163 systemd[1]: Starting Ceph node-exporter.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:01:53.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:52.829134+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:53.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:52.833258+0000 mgr.y (mgr.14550) 19 : cephadm [INF] Deploying daemon node-exporter.a on smithi163 2024-02-19T21:01:53.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:52.913327+0000 mgr.y (mgr.14550) 20 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:01:53.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:52.829134+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:53.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:52.833258+0000 mgr.y (mgr.14550) 19 : cephadm [INF] Deploying daemon node-exporter.a on smithi163 2024-02-19T21:01:53.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:52.913327+0000 mgr.y (mgr.14550) 20 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:01:54.311 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:52.829134+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:54.311 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:52.833258+0000 mgr.y (mgr.14550) 19 : cephadm [INF] Deploying daemon node-exporter.a on smithi163 2024-02-19T21:01:54.311 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:52.913327+0000 mgr.y (mgr.14550) 20 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:01:54.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.550+0000 7f627fc5c700 1 -- 172.21.15.163:0/2730494144 >> v1:172.21.15.163:6789/0 conn(0x7f62781042b0 legacy=0x7f6278106730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:54.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.550+0000 7f627fc5c700 1 -- 172.21.15.163:0/2730494144 shutdown_connections 2024-02-19T21:01:54.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.550+0000 7f627fc5c700 1 -- 172.21.15.163:0/2730494144 >> 172.21.15.163:0/2730494144 conn(0x7f62780f3590 msgr2=0x7f62780f59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:54.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.550+0000 7f627fc5c700 1 -- 172.21.15.163:0/2730494144 shutdown_connections 2024-02-19T21:01:54.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.550+0000 7f627fc5c700 1 -- 172.21.15.163:0/2730494144 wait complete. 2024-02-19T21:01:54.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.550+0000 7f627fc5c700 1 librados: starting msgr at 2024-02-19T21:01:54.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.550+0000 7f627fc5c700 1 librados: starting objecter 2024-02-19T21:01:54.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f627fc5c700 1 Processor -- start 2024-02-19T21:01:54.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f627fc5c700 1 -- start start 2024-02-19T21:01:54.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f627fc5c700 1 librados: setting wanted keys 2024-02-19T21:01:54.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f627fc5c700 1 librados: calling monclient init 2024-02-19T21:01:54.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f627fc5c700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f62781173a0 con 0x7f6278100770 2024-02-19T21:01:54.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f627fc5c700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6278118540 con 0x7f62781042b0 2024-02-19T21:01:54.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f627fc5c700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6278119740 con 0x7f62780fd970 2024-02-19T21:01:54.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f627d9f8700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f62780fd970 0x7f62780f9af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45126/0 (socket says 172.21.15.163:45126) 2024-02-19T21:01:54.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f627d9f8700 1 -- 172.21.15.163:0/1162461679 learned_addr learned my addr 172.21.15.163:0/1162461679 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:01:54.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 10 client.?.objecter ms_handle_connect 0x7f6278100770 2024-02-19T21:01:54.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:54.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 10 client.?.objecter ms_handle_connect 0x7f62780fd970 2024-02-19T21:01:54.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:54.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 10 client.?.objecter ms_handle_connect 0x7f62781042b0 2024-02-19T21:01:54.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:54.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430425438 0 0) 0x7f6278119740 con 0x7f62780fd970 2024-02-19T21:01:54.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6264003030 con 0x7f62780fd970 2024-02-19T21:01:54.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983505952 0 0) 0x7f62781173a0 con 0x7f6278100770 2024-02-19T21:01:54.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6278119740 con 0x7f6278100770 2024-02-19T21:01:54.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338816270 0 0) 0x7f6278118540 con 0x7f62781042b0 2024-02-19T21:01:54.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62781173a0 con 0x7f62781042b0 2024-02-19T21:01:54.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2294303648 0 0) 0x7f6264003030 con 0x7f62780fd970 2024-02-19T21:01:54.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6278118540 con 0x7f62780fd970 2024-02-19T21:01:54.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6274001a70 con 0x7f62780fd970 2024-02-19T21:01:54.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4053670829 0 0) 0x7f6278119740 con 0x7f6278100770 2024-02-19T21:01:54.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6264003030 con 0x7f6278100770 2024-02-19T21:01:54.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.551+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6268001840 con 0x7f6278100770 2024-02-19T21:01:54.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.552+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3780790221 0 0) 0x7f62781173a0 con 0x7f62781042b0 2024-02-19T21:01:54.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.552+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6278119740 con 0x7f62781042b0 2024-02-19T21:01:54.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.552+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6270002250 con 0x7f62781042b0 2024-02-19T21:01:54.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.552+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2684598456 0 0) 0x7f6278118540 con 0x7f62780fd970 2024-02-19T21:01:54.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.552+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 >> v1:172.21.15.193:6789/0 conn(0x7f62781042b0 legacy=0x7f6278115a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:54.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.552+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 >> v1:172.21.15.163:6789/0 conn(0x7f6278100770 legacy=0x7f62780fa200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:54.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.552+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627811a940 con 0x7f62780fd970 2024-02-19T21:01:54.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.552+0000 7f627fc5c700 1 -- 172.21.15.163:0/1162461679 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6278119930 con 0x7f62780fd970 2024-02-19T21:01:54.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.552+0000 7f627fc5c700 10 client.34517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:01:54.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.552+0000 7f627fc5c700 1 -- 172.21.15.163:0/1162461679 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6278119e20 con 0x7f62780fd970 2024-02-19T21:01:54.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.552+0000 7f627fc5c700 1 librados: init done 2024-02-19T21:01:54.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.552+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f62740024b0 con 0x7f62780fd970 2024-02-19T21:01:54.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.552+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6274005ab0 con 0x7f62780fd970 2024-02-19T21:01:54.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.553+0000 7f627fc5c700 1 -- 172.21.15.163:0/1162461679 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f625c004f20 con 0x7f62780fd970 2024-02-19T21:01:54.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.553+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6274002010 con 0x7f62780fd970 2024-02-19T21:01:54.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.553+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6172+0+0 (unknown 109389188 0 0) 0x7f6274078540 con 0x7f62780fd970 2024-02-19T21:01:54.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.553+0000 7f626effd700 10 client.34517.objecter ms_dispatch 0x7f6278001550 osd_map(72..72 src has 1..72) v4 2024-02-19T21:01:54.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.553+0000 7f626effd700 3 client.34517.objecter handle_osd_map got epochs [72,72] > 0 2024-02-19T21:01:54.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.553+0000 7f626effd700 3 client.34517.objecter handle_osd_map decoding full epoch 72 2024-02-19T21:01:54.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.554+0000 7f626effd700 20 client.34517.objecter dump_active .. 0 homeless 2024-02-19T21:01:54.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.554+0000 7f626effd700 10 client.34517.objecter ms_handle_connect 0x7f6264063c50 2024-02-19T21:01:54.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.558+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f627404c0f0 con 0x7f62780fd970 2024-02-19T21:01:54.662 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:54 smithi163 bash[68067]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-19T21:01:54.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.724+0000 7f627fc5c700 1 -- 172.21.15.163:0/1162461679 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f625c005750 con 0x7f62780fd970 2024-02-19T21:01:54.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.725+0000 7f626effd700 1 -- 172.21.15.163:0/1162461679 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v72) v1 ==== 110+0+10 (unknown 2135966190 0 2170607528) 0x7f627404a500 con 0x7f62780fd970 2024-02-19T21:01:54.714 INFO:teuthology.orchestra.run.smithi163.stdout:pg_num: 1 2024-02-19T21:01:54.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.726+0000 7f627fc5c700 10 librados: watch_flush enter 2024-02-19T21:01:54.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.726+0000 7f627fc5c700 10 librados: watch_flush exit 2024-02-19T21:01:54.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.726+0000 7f627fc5c700 20 client.34517.objecter shutdown clearing up homeless session... 2024-02-19T21:01:54.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.726+0000 7f627fc5c700 10 client.34517.objecter successfully canceled tick 2024-02-19T21:01:54.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.726+0000 7f627fc5c700 1 -- 172.21.15.163:0/1162461679 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6264063c50 legacy=0x7f6264066110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:54.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.726+0000 7f627fc5c700 1 -- 172.21.15.163:0/1162461679 >> v1:172.21.15.163:6790/0 conn(0x7f62780fd970 legacy=0x7f62780f9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:54.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.726+0000 7f627fc5c700 1 -- 172.21.15.163:0/1162461679 shutdown_connections 2024-02-19T21:01:54.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.726+0000 7f627fc5c700 1 -- 172.21.15.163:0/1162461679 >> 172.21.15.163:0/1162461679 conn(0x7f62780f3590 msgr2=0x7f62780f66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:54.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.727+0000 7f627fc5c700 1 -- 172.21.15.163:0/1162461679 shutdown_connections 2024-02-19T21:01:54.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.727+0000 7f627fc5c700 1 -- 172.21.15.163:0/1162461679 wait complete. 2024-02-19T21:01:54.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:54.727+0000 7f627fc5c700 1 librados: shutdown 2024-02-19T21:01:54.987 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:54 smithi163 bash[68067]: Getting image source signatures 2024-02-19T21:01:54.987 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:54 smithi163 bash[68067]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-19T21:01:54.987 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:54 smithi163 bash[68067]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-19T21:01:54.987 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:54 smithi163 bash[68067]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-19T21:01:54.987 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:01:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[67708]: level=info ts=2024-02-19T21:01:54.672Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000030474s 2024-02-19T21:01:55.437 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:54.071601+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:55.437 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:54.725197+0000 mon.c (mon.2) 20 : audit [DBG] from='client.? v1:172.21.15.163:0/1162461679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-19T21:01:55.437 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:54.071601+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:55.437 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:54.725197+0000 mon.c (mon.2) 20 : audit [DBG] from='client.? v1:172.21.15.163:0/1162461679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-19T21:01:55.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool get .rgw.root pg_num 2024-02-19T21:01:55.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:54.071601+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:55.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:54.725197+0000 mon.c (mon.2) 20 : audit [DBG] from='client.? v1:172.21.15.163:0/1162461679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2024-02-19T21:01:55.805 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:01:56.128 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:54.913848+0000 mgr.y (mgr.14550) 21 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:01:56.128 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:55 smithi163 bash[68067]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-19T21:01:56.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:54.913848+0000 mgr.y (mgr.14550) 21 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:01:56.487 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 bash[68067]: Writing manifest to image destination 2024-02-19T21:01:56.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:54.913848+0000 mgr.y (mgr.14550) 21 : cluster [DBG] pgmap v11: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:01:57.139 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.859Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-19T21:01:57.139 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.859Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-19T21:01:57.139 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-19T21:01:57.139 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-19T21:01:57.139 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-19T21:01:57.139 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=arp 2024-02-19T21:01:57.139 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-19T21:01:57.139 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-19T21:01:57.139 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-19T21:01:57.139 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-19T21:01:57.140 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-19T21:01:57.140 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-19T21:01:57.140 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-19T21:01:57.140 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-19T21:01:57.140 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=edac 2024-02-19T21:01:57.140 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-19T21:01:57.140 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-19T21:01:57.140 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-19T21:01:57.140 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-19T21:01:57.140 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-19T21:01:57.140 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-19T21:01:57.140 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-19T21:01:57.140 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-19T21:01:57.141 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-19T21:01:57.141 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-19T21:01:57.141 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-19T21:01:57.141 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-19T21:01:57.141 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-19T21:01:57.141 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-19T21:01:57.141 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-19T21:01:57.141 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-19T21:01:57.141 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=os 2024-02-19T21:01:57.141 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-19T21:01:57.141 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-19T21:01:57.141 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-19T21:01:57.141 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-19T21:01:57.141 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-19T21:01:57.141 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-19T21:01:57.142 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=stat 2024-02-19T21:01:57.142 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-19T21:01:57.142 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-19T21:01:57.142 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-19T21:01:57.142 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=time 2024-02-19T21:01:57.142 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-19T21:01:57.142 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=uname 2024-02-19T21:01:57.142 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-19T21:01:57.142 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-19T21:01:57.142 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-19T21:01:57.142 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.860Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-19T21:01:57.142 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a[68282]: ts=2024-02-19T21:01:56.861Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-19T21:01:57.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.949+0000 7faccc40f700 1 -- 172.21.15.163:0/2768165745 >> v1:172.21.15.163:6789/0 conn(0x7facc41031b0 legacy=0x7facc4105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:57.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.949+0000 7faccc40f700 1 -- 172.21.15.163:0/2768165745 shutdown_connections 2024-02-19T21:01:57.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.949+0000 7faccc40f700 1 -- 172.21.15.163:0/2768165745 >> 172.21.15.163:0/2768165745 conn(0x7facc40f2470 msgr2=0x7facc40f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:57.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.949+0000 7faccc40f700 1 -- 172.21.15.163:0/2768165745 shutdown_connections 2024-02-19T21:01:57.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.949+0000 7faccc40f700 1 -- 172.21.15.163:0/2768165745 wait complete. 2024-02-19T21:01:57.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.950+0000 7faccc40f700 1 librados: starting msgr at 2024-02-19T21:01:57.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.950+0000 7faccc40f700 1 librados: starting objecter 2024-02-19T21:01:57.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.950+0000 7faccc40f700 1 Processor -- start 2024-02-19T21:01:57.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.950+0000 7faccc40f700 1 -- start start 2024-02-19T21:01:57.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.950+0000 7faccc40f700 1 librados: setting wanted keys 2024-02-19T21:01:57.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.950+0000 7faccc40f700 1 librados: calling monclient init 2024-02-19T21:01:57.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.950+0000 7faccc40f700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7facc41171e0 con 0x7facc40fc870 2024-02-19T21:01:57.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.950+0000 7faccc40f700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7facc41183c0 con 0x7facc41031b0 2024-02-19T21:01:57.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.950+0000 7faccc40f700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7facc41195c0 con 0x7facc40ff670 2024-02-19T21:01:57.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.950+0000 7facc99aa700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7facc40ff670 0x7facc40f9ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45148/0 (socket says 172.21.15.163:45148) 2024-02-19T21:01:57.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.950+0000 7facc99aa700 1 -- 172.21.15.163:0/2900696221 learned_addr learned my addr 172.21.15.163:0/2900696221 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:01:57.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.950+0000 7facbb7fe700 10 client.?.objecter ms_handle_connect 0x7facc40ff670 2024-02-19T21:01:57.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.950+0000 7facbb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:57.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.950+0000 7facbb7fe700 10 client.?.objecter ms_handle_connect 0x7facc41031b0 2024-02-19T21:01:57.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.950+0000 7facbb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:57.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.951+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437436263 0 0) 0x7facc41195c0 con 0x7facc40ff670 2024-02-19T21:01:57.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.951+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb0003030 con 0x7facc40ff670 2024-02-19T21:01:57.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.951+0000 7facbb7fe700 10 client.?.objecter ms_handle_connect 0x7facc40fc870 2024-02-19T21:01:57.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.951+0000 7facbb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:01:57.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.951+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1572152513 0 0) 0x7facb0003030 con 0x7facc40ff670 2024-02-19T21:01:57.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.951+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facc41195c0 con 0x7facc40ff670 2024-02-19T21:01:57.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.951+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7facc0001950 con 0x7facc40ff670 2024-02-19T21:01:57.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.951+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744369849 0 0) 0x7facc41171e0 con 0x7facc40fc870 2024-02-19T21:01:57.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.951+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb0003030 con 0x7facc40fc870 2024-02-19T21:01:57.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.951+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3289548637 0 0) 0x7facb0003030 con 0x7facc40fc870 2024-02-19T21:01:57.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.951+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facc41171e0 con 0x7facc40fc870 2024-02-19T21:01:57.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.951+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7facb4001840 con 0x7facc40fc870 2024-02-19T21:01:57.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.953+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 490873185 0 0) 0x7facc41171e0 con 0x7facc40fc870 2024-02-19T21:01:57.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.954+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 >> v1:172.21.15.163:6790/0 conn(0x7facc40ff670 legacy=0x7facc40f9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:57.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.954+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 >> v1:172.21.15.193:6789/0 conn(0x7facc41031b0 legacy=0x7facc41158a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:57.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.954+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc411a7c0 con 0x7facc40fc870 2024-02-19T21:01:57.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.954+0000 7faccc40f700 1 -- 172.21.15.163:0/2900696221 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc41185b0 con 0x7facc40fc870 2024-02-19T21:01:57.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.954+0000 7faccc40f700 10 client.14598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:01:57.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.954+0000 7faccc40f700 1 -- 172.21.15.163:0/2900696221 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7facc4118a20 con 0x7facc40fc870 2024-02-19T21:01:57.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.954+0000 7faccc40f700 1 librados: init done 2024-02-19T21:01:57.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.955+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7facb4001f00 con 0x7facc40fc870 2024-02-19T21:01:57.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.955+0000 7faccc40f700 1 -- 172.21.15.163:0/2900696221 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc405f700 con 0x7facc40fc870 2024-02-19T21:01:57.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.955+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7facb40032a0 con 0x7facc40fc870 2024-02-19T21:01:57.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.955+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7facb40161f0 con 0x7facc40fc870 2024-02-19T21:01:57.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.956+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(72..72 src has 1..72) v4 ==== 6172+0+0 (unknown 109389188 0 0) 0x7facb4076c90 con 0x7facc40fc870 2024-02-19T21:01:57.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.956+0000 7facbb7fe700 10 client.14598.objecter ms_dispatch 0x7facc4001550 osd_map(72..72 src has 1..72) v4 2024-02-19T21:01:57.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.956+0000 7facbb7fe700 3 client.14598.objecter handle_osd_map got epochs [72,72] > 0 2024-02-19T21:01:57.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.956+0000 7facbb7fe700 3 client.14598.objecter handle_osd_map decoding full epoch 72 2024-02-19T21:01:57.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.956+0000 7facbb7fe700 20 client.14598.objecter dump_active .. 0 homeless 2024-02-19T21:01:57.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.956+0000 7facbb7fe700 10 client.14598.objecter ms_handle_connect 0x7facb0063c80 2024-02-19T21:01:57.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:57.960+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7facb404a7e0 con 0x7facc40fc870 2024-02-19T21:01:58.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:58.130+0000 7faccc40f700 1 -- 172.21.15.163:0/2900696221 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"} v 0) v1 -- 0x7facc40f80d0 con 0x7facc40fc870 2024-02-19T21:01:58.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:58.131+0000 7facbb7fe700 1 -- 172.21.15.163:0/2900696221 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]=0 v72) v1 ==== 98+0+11 (unknown 1234939541 0 3519293104) 0x7facb4048bf0 con 0x7facc40fc870 2024-02-19T21:01:58.121 INFO:teuthology.orchestra.run.smithi163.stdout:pg_num: 32 2024-02-19T21:01:58.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:58.132+0000 7faccc40f700 10 librados: watch_flush enter 2024-02-19T21:01:58.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:58.133+0000 7faccc40f700 10 librados: watch_flush exit 2024-02-19T21:01:58.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:58.133+0000 7faccc40f700 20 client.14598.objecter shutdown clearing up homeless session... 2024-02-19T21:01:58.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:58.133+0000 7faccc40f700 10 client.14598.objecter successfully canceled tick 2024-02-19T21:01:58.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:58.133+0000 7faccc40f700 1 -- 172.21.15.163:0/2900696221 >> v1:172.21.15.163:6800/1552514681 conn(0x7facb0063c80 legacy=0x7facb0066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:58.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:58.133+0000 7faccc40f700 1 -- 172.21.15.163:0/2900696221 >> v1:172.21.15.163:6789/0 conn(0x7facc40fc870 legacy=0x7facc40f97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:01:58.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:58.133+0000 7faccc40f700 1 -- 172.21.15.163:0/2900696221 shutdown_connections 2024-02-19T21:01:58.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:58.133+0000 7faccc40f700 1 -- 172.21.15.163:0/2900696221 >> 172.21.15.163:0/2900696221 conn(0x7facc40f2470 msgr2=0x7facc4106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:01:58.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:58.133+0000 7faccc40f700 1 -- 172.21.15.163:0/2900696221 shutdown_connections 2024-02-19T21:01:58.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:58.133+0000 7faccc40f700 1 -- 172.21.15.163:0/2900696221 wait complete. 2024-02-19T21:01:58.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:01:58.133+0000 7faccc40f700 1 librados: shutdown 2024-02-19T21:01:58.396 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:56.914227+0000 mgr.y (mgr.14550) 22 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:01:58.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:56.914227+0000 mgr.y (mgr.14550) 22 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:01:58.445 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:58 smithi163 bash[68067]: d814a34de719db68e69f61fb4d4501d72b28ea6bf9651e28bd95eda61cfa0f65 2024-02-19T21:01:58.445 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:01:58 smithi163 systemd[1]: Started Ceph node-exporter.a for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T21:01:58.445 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:56.914227+0000 mgr.y (mgr.14550) 22 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:01:58.737 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool get iscsi pg_num 2024-02-19T21:01:59.006 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:01:59.061 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:01:58 smithi193 systemd[1]: Starting Ceph node-exporter.b for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:01:59.061 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:01:58 smithi193 bash[62121]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-19T21:01:59.433 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:58.132244+0000 mon.a (mon.0) 662 : audit [DBG] from='client.? v1:172.21.15.163:0/2900696221' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2024-02-19T21:01:59.433 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:58.140460+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:59.433 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:01:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:01:58.141893+0000 mgr.y (mgr.14550) 23 : cephadm [INF] Deploying daemon node-exporter.b on smithi193 2024-02-19T21:01:59.433 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:58.132244+0000 mon.a (mon.0) 662 : audit [DBG] from='client.? v1:172.21.15.163:0/2900696221' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2024-02-19T21:01:59.433 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:58.140460+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:59.433 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:01:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:01:58.141893+0000 mgr.y (mgr.14550) 23 : cephadm [INF] Deploying daemon node-exporter.b on smithi193 2024-02-19T21:01:59.561 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:01:59 smithi193 bash[62121]: Getting image source signatures 2024-02-19T21:01:59.561 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:01:59 smithi193 bash[62121]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-19T21:01:59.561 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:01:59 smithi193 bash[62121]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-19T21:01:59.561 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:01:59 smithi193 bash[62121]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-19T21:01:59.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:58.132244+0000 mon.a (mon.0) 662 : audit [DBG] from='client.? v1:172.21.15.163:0/2900696221' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2024-02-19T21:01:59.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:58.140460+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:01:59.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:01:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:01:58.141893+0000 mgr.y (mgr.14550) 23 : cephadm [INF] Deploying daemon node-exporter.b on smithi193 2024-02-19T21:02:00.134 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:58.914672+0000 mgr.y (mgr.14550) 24 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:02:00.310 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:58.914672+0000 mgr.y (mgr.14550) 24 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:02:00.311 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:01:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:02:00.392 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:58.914672+0000 mgr.y (mgr.14550) 24 : cluster [DBG] pgmap v13: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:02:00.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.618+0000 7f90a929d700 1 -- 172.21.15.163:0/2097782158 >> v1:172.21.15.163:6789/0 conn(0x7f90a40f8920 legacy=0x7f90a40f8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:00.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.618+0000 7f90a929d700 1 -- 172.21.15.163:0/2097782158 shutdown_connections 2024-02-19T21:02:00.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.618+0000 7f90a929d700 1 -- 172.21.15.163:0/2097782158 >> 172.21.15.163:0/2097782158 conn(0x7f90a40f2450 msgr2=0x7f90a40f4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:00.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.618+0000 7f90a929d700 1 -- 172.21.15.163:0/2097782158 shutdown_connections 2024-02-19T21:02:00.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.618+0000 7f90a929d700 1 -- 172.21.15.163:0/2097782158 wait complete. 2024-02-19T21:02:00.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.619+0000 7f90a929d700 1 librados: starting msgr at 2024-02-19T21:02:00.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.619+0000 7f90a929d700 1 librados: starting objecter 2024-02-19T21:02:00.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.619+0000 7f90a929d700 1 Processor -- start 2024-02-19T21:02:00.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.619+0000 7f90a929d700 1 -- start start 2024-02-19T21:02:00.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.619+0000 7f90a929d700 1 librados: setting wanted keys 2024-02-19T21:02:00.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.619+0000 7f90a929d700 1 librados: calling monclient init 2024-02-19T21:02:00.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f90a929d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f90a4117310 con 0x7f90a40f8920 2024-02-19T21:02:00.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f90a929d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f90a41184f0 con 0x7f90a41042b0 2024-02-19T21:02:00.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f90a929d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f90a41196f0 con 0x7f90a40f94d0 2024-02-19T21:02:00.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f90a2ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f90a40f8920 0x7f90a40ff450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:34250/0 (socket says 172.21.15.163:34250) 2024-02-19T21:02:00.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f90a2ffd700 1 -- 172.21.15.163:0/1743987778 learned_addr learned my addr 172.21.15.163:0/1743987778 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:00.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 10 client.?.objecter ms_handle_connect 0x7f90a40f94d0 2024-02-19T21:02:00.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:00.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 10 client.?.objecter ms_handle_connect 0x7f90a40f8920 2024-02-19T21:02:00.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:00.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899305956 0 0) 0x7f90a41196f0 con 0x7f90a40f94d0 2024-02-19T21:02:00.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f908c0033f0 con 0x7f90a40f94d0 2024-02-19T21:02:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 10 client.?.objecter ms_handle_connect 0x7f90a41042b0 2024-02-19T21:02:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039161691 0 0) 0x7f90a4117310 con 0x7f90a40f8920 2024-02-19T21:02:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a41196f0 con 0x7f90a40f8920 2024-02-19T21:02:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4270726715 0 0) 0x7f908c0033f0 con 0x7f90a40f94d0 2024-02-19T21:02:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90a4117310 con 0x7f90a40f94d0 2024-02-19T21:02:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2096645344 0 0) 0x7f90a41196f0 con 0x7f90a40f8920 2024-02-19T21:02:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f908c0033f0 con 0x7f90a40f8920 2024-02-19T21:02:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f90980018c0 con 0x7f90a40f94d0 2024-02-19T21:02:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581967456 0 0) 0x7f90a41184f0 con 0x7f90a41042b0 2024-02-19T21:02:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a41196f0 con 0x7f90a41042b0 2024-02-19T21:02:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f90940020b0 con 0x7f90a40f8920 2024-02-19T21:02:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4102714139 0 0) 0x7f90a4117310 con 0x7f90a40f94d0 2024-02-19T21:02:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 >> v1:172.21.15.193:6789/0 conn(0x7f90a41042b0 legacy=0x7f90a41159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 >> v1:172.21.15.163:6789/0 conn(0x7f90a40f8920 legacy=0x7f90a40ff450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:00.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a411a8f0 con 0x7f90a40f94d0 2024-02-19T21:02:00.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.620+0000 7f90a929d700 1 -- 172.21.15.163:0/1743987778 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a4117500 con 0x7f90a40f94d0 2024-02-19T21:02:00.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.621+0000 7f90a929d700 10 client.34529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:00.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.621+0000 7f90a929d700 1 -- 172.21.15.163:0/1743987778 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a4117970 con 0x7f90a40f94d0 2024-02-19T21:02:00.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.621+0000 7f90a929d700 1 librados: init done 2024-02-19T21:02:00.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.621+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9098001f80 con 0x7f90a40f94d0 2024-02-19T21:02:00.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.621+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9098003270 con 0x7f90a40f94d0 2024-02-19T21:02:00.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.621+0000 7f90a929d700 1 -- 172.21.15.163:0/1743987778 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9090004f20 con 0x7f90a40f94d0 2024-02-19T21:02:00.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.622+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9098001ae0 con 0x7f90a40f94d0 2024-02-19T21:02:00.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.622+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 6172+0+0 (unknown 296875052 0 0) 0x7f9098075d40 con 0x7f90a40f94d0 2024-02-19T21:02:00.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.622+0000 7f908bfff700 10 client.34529.objecter ms_dispatch 0x7f90a4001550 osd_map(73..73 src has 1..73) v4 2024-02-19T21:02:00.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.622+0000 7f908bfff700 3 client.34529.objecter handle_osd_map got epochs [73,73] > 0 2024-02-19T21:02:00.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.622+0000 7f908bfff700 3 client.34529.objecter handle_osd_map decoding full epoch 73 2024-02-19T21:02:00.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.622+0000 7f908bfff700 20 client.34529.objecter dump_active .. 0 homeless 2024-02-19T21:02:00.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.623+0000 7f908bfff700 10 client.34529.objecter ms_handle_connect 0x7f908c063cb0 2024-02-19T21:02:00.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.627+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f90980a0f60 con 0x7f90a40f94d0 2024-02-19T21:02:00.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.790+0000 7f90a929d700 1 -- 172.21.15.163:0/1743987778 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"} v 0) v1 -- 0x7f9090005750 con 0x7f90a40f94d0 2024-02-19T21:02:00.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.791+0000 7f908bfff700 1 -- 172.21.15.163:0/1743987778 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]=0 v73) v1 ==== 94+0+10 (unknown 468571128 0 2787501894) 0x7f90980498d0 con 0x7f90a40f94d0 2024-02-19T21:02:00.781 INFO:teuthology.orchestra.run.smithi163.stdout:pg_num: 3 2024-02-19T21:02:00.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.792+0000 7f90a929d700 10 librados: watch_flush enter 2024-02-19T21:02:00.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.792+0000 7f90a929d700 10 librados: watch_flush exit 2024-02-19T21:02:00.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.792+0000 7f90a929d700 20 client.34529.objecter shutdown clearing up homeless session... 2024-02-19T21:02:00.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.792+0000 7f90a929d700 10 client.34529.objecter successfully canceled tick 2024-02-19T21:02:00.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.793+0000 7f90a929d700 1 -- 172.21.15.163:0/1743987778 >> v1:172.21.15.163:6800/1552514681 conn(0x7f908c063cb0 legacy=0x7f908c066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:00.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.793+0000 7f90a929d700 1 -- 172.21.15.163:0/1743987778 >> v1:172.21.15.163:6790/0 conn(0x7f90a40f94d0 legacy=0x7f90a40ffb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:00.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.793+0000 7f90a929d700 1 -- 172.21.15.163:0/1743987778 shutdown_connections 2024-02-19T21:02:00.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.793+0000 7f90a929d700 1 -- 172.21.15.163:0/1743987778 >> 172.21.15.163:0/1743987778 conn(0x7f90a40f2450 msgr2=0x7f90a4107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:00.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.793+0000 7f90a929d700 1 -- 172.21.15.163:0/1743987778 shutdown_connections 2024-02-19T21:02:00.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.793+0000 7f90a929d700 1 -- 172.21.15.163:0/1743987778 wait complete. 2024-02-19T21:02:00.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:00.793+0000 7f90a929d700 1 librados: shutdown 2024-02-19T21:02:01.311 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:00 smithi193 bash[62121]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-19T21:02:01.311 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 bash[62121]: Writing manifest to image destination 2024-02-19T21:02:01.311 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:59.638588+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:01.311 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:01:59.638640+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:01.311 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:01:59.930185+0000 mgr.y (mgr.14550) 25 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:01.311 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:00.142545+0000 mon.b (mon.1) 58 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-19T21:02:01.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:00.142737+0000 mon.a (mon.0) 664 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-19T21:02:01.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:00.143579+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-19T21:02:01.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:00.146627+0000 mon.b (mon.1) 59 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-19T21:02:01.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:00.146995+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-19T21:02:01.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:00.792395+0000 mon.c (mon.2) 21 : audit [DBG] from='client.? v1:172.21.15.163:0/1743987778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2024-02-19T21:02:01.409 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:59.638588+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:01.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:01:59.638640+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:01.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:01:59.930185+0000 mgr.y (mgr.14550) 25 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:01.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:00.142545+0000 mon.b (mon.1) 58 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-19T21:02:01.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:00.142737+0000 mon.a (mon.0) 664 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-19T21:02:01.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:00.143579+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-19T21:02:01.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:00.146627+0000 mon.b (mon.1) 59 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-19T21:02:01.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:00.146995+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-19T21:02:01.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:00.792395+0000 mon.c (mon.2) 21 : audit [DBG] from='client.? v1:172.21.15.163:0/1743987778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2024-02-19T21:02:01.410 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:59.638588+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:01.410 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:01:59.638640+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:01.410 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:01:59.930185+0000 mgr.y (mgr.14550) 25 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:01.411 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:00.142545+0000 mon.b (mon.1) 58 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-19T21:02:01.411 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:00.142737+0000 mon.a (mon.0) 664 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-02-19T21:02:01.411 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:00.143579+0000 mon.a (mon.0) 665 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-19T21:02:01.411 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:00.146627+0000 mon.b (mon.1) 59 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-19T21:02:01.411 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:00.146995+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2024-02-19T21:02:01.411 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:00.792395+0000 mon.c (mon.2) 21 : audit [DBG] from='client.? v1:172.21.15.163:0/1743987778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2024-02-19T21:02:01.429 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool get default.rgw.log pg_num 2024-02-19T21:02:01.696 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:02.061 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.808Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-19T21:02:02.062 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.808Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-19T21:02:02.062 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.809Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-19T21:02:02.062 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.809Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-19T21:02:02.062 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-19T21:02:02.062 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=arp 2024-02-19T21:02:02.062 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-19T21:02:02.062 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-19T21:02:02.062 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-19T21:02:02.062 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-19T21:02:02.062 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-19T21:02:02.062 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-19T21:02:02.062 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-19T21:02:02.062 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-19T21:02:02.062 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=edac 2024-02-19T21:02:02.062 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-19T21:02:02.063 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-19T21:02:02.063 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-19T21:02:02.063 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-19T21:02:02.063 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-19T21:02:02.063 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-19T21:02:02.063 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-19T21:02:02.063 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-19T21:02:02.063 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-19T21:02:02.063 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-19T21:02:02.063 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-19T21:02:02.063 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-19T21:02:02.063 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-19T21:02:02.063 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-19T21:02:02.063 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-19T21:02:02.063 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-19T21:02:02.064 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=os 2024-02-19T21:02:02.064 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-19T21:02:02.064 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-19T21:02:02.064 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-19T21:02:02.064 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-19T21:02:02.064 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-19T21:02:02.064 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-19T21:02:02.064 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=stat 2024-02-19T21:02:02.064 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-19T21:02:02.064 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-19T21:02:02.064 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-19T21:02:02.064 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=time 2024-02-19T21:02:02.064 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-19T21:02:02.064 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=uname 2024-02-19T21:02:02.064 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-19T21:02:02.065 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-19T21:02:02.065 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-19T21:02:02.065 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-19T21:02:02.065 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b[62172]: ts=2024-02-19T21:02:01.810Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-19T21:02:02.065 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 bash[62121]: cfe9103fb9e890483599a51c00416baa8526b4ab5f3945dcd45419da0a590133 2024-02-19T21:02:02.065 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:02:01 smithi193 systemd[1]: Started Ceph node-exporter.b for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T21:02:02.156 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:00.915198+0000 mgr.y (mgr.14550) 26 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:02.156 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:01.151863+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-19T21:02:02.156 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:01.151931+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-02-19T21:02:02.156 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:01.972452+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:02.156 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:00.915198+0000 mgr.y (mgr.14550) 26 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:02.156 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:01.151863+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-19T21:02:02.156 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:01.151931+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-02-19T21:02:02.157 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:01.972452+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:02.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:00.915198+0000 mgr.y (mgr.14550) 26 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:02.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:01.151863+0000 mon.a (mon.0) 667 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2024-02-19T21:02:02.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:01.151931+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-02-19T21:02:02.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:01.972452+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:02.851 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[67708]: level=info ts=2024-02-19T21:02:02.674Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.001874971s 2024-02-19T21:02:03.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.372+0000 7f1fb601f700 1 -- 172.21.15.163:0/1246754350 >> v1:172.21.15.163:6789/0 conn(0x7f1fb00ff670 legacy=0x7f1fb0101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:03.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.372+0000 7f1fb601f700 1 -- 172.21.15.163:0/1246754350 shutdown_connections 2024-02-19T21:02:03.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.372+0000 7f1fb601f700 1 -- 172.21.15.163:0/1246754350 >> 172.21.15.163:0/1246754350 conn(0x7f1fb00f2470 msgr2=0x7f1fb00f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:03.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.372+0000 7f1fb601f700 1 -- 172.21.15.163:0/1246754350 shutdown_connections 2024-02-19T21:02:03.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.372+0000 7f1fb601f700 1 -- 172.21.15.163:0/1246754350 wait complete. 2024-02-19T21:02:03.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.372+0000 7f1fb601f700 1 librados: starting msgr at 2024-02-19T21:02:03.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.372+0000 7f1fb601f700 1 librados: starting objecter 2024-02-19T21:02:03.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1fb601f700 1 Processor -- start 2024-02-19T21:02:03.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1fb601f700 1 -- start start 2024-02-19T21:02:03.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1fb601f700 1 librados: setting wanted keys 2024-02-19T21:02:03.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1fb601f700 1 librados: calling monclient init 2024-02-19T21:02:03.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1fb601f700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1fb0116170 con 0x7f1fb00ff670 2024-02-19T21:02:03.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1fb601f700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1fb0117350 con 0x7f1fb00fc870 2024-02-19T21:02:03.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1fb601f700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1fb0118550 con 0x7f1fb01031b0 2024-02-19T21:02:03.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1faffff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f1fb01031b0 0x7f1fb0114830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45174/0 (socket says 172.21.15.163:45174) 2024-02-19T21:02:03.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1faffff700 1 -- 172.21.15.163:0/1233093495 learned_addr learned my addr 172.21.15.163:0/1233093495 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:03.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1facff9700 10 client.?.objecter ms_handle_connect 0x7f1fb01031b0 2024-02-19T21:02:03.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1facff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:03.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1facff9700 10 client.?.objecter ms_handle_connect 0x7f1fb00ff670 2024-02-19T21:02:03.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1facff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:03.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1facff9700 10 client.?.objecter ms_handle_connect 0x7f1fb00fc870 2024-02-19T21:02:03.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1facff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:03.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588447714 0 0) 0x7f1fb0118550 con 0x7f1fb01031b0 2024-02-19T21:02:03.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f980033f0 con 0x7f1fb01031b0 2024-02-19T21:02:03.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.373+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330204396 0 0) 0x7f1fb0116170 con 0x7f1fb00ff670 2024-02-19T21:02:03.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb0118550 con 0x7f1fb00ff670 2024-02-19T21:02:03.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036280789 0 0) 0x7f1fb0117350 con 0x7f1fb00fc870 2024-02-19T21:02:03.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fb0116170 con 0x7f1fb00fc870 2024-02-19T21:02:03.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3139644352 0 0) 0x7f1f980033f0 con 0x7f1fb01031b0 2024-02-19T21:02:03.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fb0117350 con 0x7f1fb01031b0 2024-02-19T21:02:03.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1fa8002cd0 con 0x7f1fb01031b0 2024-02-19T21:02:03.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1337917244 0 0) 0x7f1fb0118550 con 0x7f1fb00ff670 2024-02-19T21:02:03.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f980033f0 con 0x7f1fb00ff670 2024-02-19T21:02:03.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1fa0002130 con 0x7f1fb00ff670 2024-02-19T21:02:03.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3813400578 0 0) 0x7f1fb0116170 con 0x7f1fb00fc870 2024-02-19T21:02:03.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fb0118550 con 0x7f1fb00fc870 2024-02-19T21:02:03.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1fa40019d0 con 0x7f1fb00fc870 2024-02-19T21:02:03.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1714678990 0 0) 0x7f1fb0117350 con 0x7f1fb01031b0 2024-02-19T21:02:03.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 >> v1:172.21.15.193:6789/0 conn(0x7f1fb00fc870 legacy=0x7f1fb00f8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:03.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 >> v1:172.21.15.163:6789/0 conn(0x7f1fb00ff670 legacy=0x7f1fb00f8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:03.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb0119750 con 0x7f1fb01031b0 2024-02-19T21:02:03.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1fb601f700 1 -- 172.21.15.163:0/1233093495 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb0116360 con 0x7f1fb01031b0 2024-02-19T21:02:03.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1fb601f700 10 client.34535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:03.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1fb601f700 1 -- 172.21.15.163:0/1233093495 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb0116850 con 0x7f1fb01031b0 2024-02-19T21:02:03.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1fb601f700 1 librados: init done 2024-02-19T21:02:03.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f1fa80030a0 con 0x7f1fb01031b0 2024-02-19T21:02:03.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.374+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1fa8005fb0 con 0x7f1fb01031b0 2024-02-19T21:02:03.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.375+0000 7f1fb601f700 1 -- 172.21.15.163:0/1233093495 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f9c004f20 con 0x7f1fb01031b0 2024-02-19T21:02:03.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.375+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f1fa80019e0 con 0x7f1fb01031b0 2024-02-19T21:02:03.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.376+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 6172+0+0 (unknown 686589652 0 0) 0x7f1fa8078990 con 0x7f1fb01031b0 2024-02-19T21:02:03.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.376+0000 7f1facff9700 10 client.34535.objecter ms_dispatch 0x7f1fb0001550 osd_map(74..74 src has 1..74) v4 2024-02-19T21:02:03.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.376+0000 7f1facff9700 3 client.34535.objecter handle_osd_map got epochs [74,74] > 0 2024-02-19T21:02:03.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.376+0000 7f1facff9700 3 client.34535.objecter handle_osd_map decoding full epoch 74 2024-02-19T21:02:03.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.376+0000 7f1facff9700 20 client.34535.objecter dump_active .. 0 homeless 2024-02-19T21:02:03.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.376+0000 7f1facff9700 10 client.34535.objecter ms_handle_connect 0x7f1f98063c10 2024-02-19T21:02:03.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.380+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1fa804c540 con 0x7f1fb01031b0 2024-02-19T21:02:03.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:02:01.982594+0000 mgr.y (mgr.14550) 27 : cephadm [INF] Deploying daemon prometheus.a on smithi193 2024-02-19T21:02:03.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:02:01.982594+0000 mgr.y (mgr.14550) 27 : cephadm [INF] Deploying daemon prometheus.a on smithi193 2024-02-19T21:02:03.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.542+0000 7f1fb601f700 1 -- 172.21.15.163:0/1233093495 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"} v 0) v1 -- 0x7f1f9c005750 con 0x7f1fb01031b0 2024-02-19T21:02:03.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.542+0000 7f1facff9700 1 -- 172.21.15.163:0/1233093495 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]=0 v74) v1 ==== 104+0+11 (unknown 808570673 0 3519293104) 0x7f1fa804a950 con 0x7f1fb01031b0 2024-02-19T21:02:03.532 INFO:teuthology.orchestra.run.smithi163.stdout:pg_num: 32 2024-02-19T21:02:03.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.544+0000 7f1fb601f700 10 librados: watch_flush enter 2024-02-19T21:02:03.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.544+0000 7f1fb601f700 10 librados: watch_flush exit 2024-02-19T21:02:03.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.544+0000 7f1fb601f700 20 client.34535.objecter shutdown clearing up homeless session... 2024-02-19T21:02:03.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.544+0000 7f1fb601f700 10 client.34535.objecter successfully canceled tick 2024-02-19T21:02:03.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.544+0000 7f1fb601f700 1 -- 172.21.15.163:0/1233093495 >> v1:172.21.15.163:6800/1552514681 conn(0x7f1f98063c10 legacy=0x7f1f980660d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:03.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.544+0000 7f1fb601f700 1 -- 172.21.15.163:0/1233093495 >> v1:172.21.15.163:6790/0 conn(0x7f1fb01031b0 legacy=0x7f1fb0114830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:03.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.544+0000 7f1fb601f700 1 -- 172.21.15.163:0/1233093495 shutdown_connections 2024-02-19T21:02:03.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.544+0000 7f1fb601f700 1 -- 172.21.15.163:0/1233093495 >> 172.21.15.163:0/1233093495 conn(0x7f1fb00f2470 msgr2=0x7f1fb0106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:03.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.544+0000 7f1fb601f700 1 -- 172.21.15.163:0/1233093495 shutdown_connections 2024-02-19T21:02:03.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.544+0000 7f1fb601f700 1 -- 172.21.15.163:0/1233093495 wait complete. 2024-02-19T21:02:03.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:03.544+0000 7f1fb601f700 1 librados: shutdown 2024-02-19T21:02:03.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:02:01.982594+0000 mgr.y (mgr.14550) 27 : cephadm [INF] Deploying daemon prometheus.a on smithi193 2024-02-19T21:02:04.113 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool get default.rgw.control pg_num 2024-02-19T21:02:04.388 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:04.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:02.915542+0000 mgr.y (mgr.14550) 28 : cluster [DBG] pgmap v17: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:02:04.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:03.243768+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:04.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:03.243860+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:04.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:03.543780+0000 mon.c (mon.2) 22 : audit [DBG] from='client.? v1:172.21.15.163:0/1233093495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2024-02-19T21:02:04.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:04.085034+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:04.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:02.915542+0000 mgr.y (mgr.14550) 28 : cluster [DBG] pgmap v17: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:02:04.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:03.243768+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:04.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:03.243860+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:04.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:03.543780+0000 mon.c (mon.2) 22 : audit [DBG] from='client.? v1:172.21.15.163:0/1233093495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2024-02-19T21:02:04.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:04.085034+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:04.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:02.915542+0000 mgr.y (mgr.14550) 28 : cluster [DBG] pgmap v17: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:02:04.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:03.243768+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:04.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:03.243860+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:04.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:03.543780+0000 mon.c (mon.2) 22 : audit [DBG] from='client.? v1:172.21.15.163:0/1233093495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2024-02-19T21:02:04.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:04.085034+0000 mon.a (mon.0) 670 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:05.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:04.160867+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-02-19T21:02:05.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:04.160867+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-02-19T21:02:05.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:04.160867+0000 mon.a (mon.0) 671 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-02-19T21:02:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.077+0000 7fcac9c4b700 1 -- 172.21.15.163:0/4236259145 >> v1:172.21.15.163:6789/0 conn(0x7fcac41042b0 legacy=0x7fcac4106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.077+0000 7fcac9c4b700 1 -- 172.21.15.163:0/4236259145 shutdown_connections 2024-02-19T21:02:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.077+0000 7fcac9c4b700 1 -- 172.21.15.163:0/4236259145 >> 172.21.15.163:0/4236259145 conn(0x7fcac40f2470 msgr2=0x7fcac40f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.077+0000 7fcac9c4b700 1 -- 172.21.15.163:0/4236259145 shutdown_connections 2024-02-19T21:02:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.077+0000 7fcac9c4b700 1 -- 172.21.15.163:0/4236259145 wait complete. 2024-02-19T21:02:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.078+0000 7fcac9c4b700 1 librados: starting msgr at 2024-02-19T21:02:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.078+0000 7fcac9c4b700 1 librados: starting objecter 2024-02-19T21:02:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.078+0000 7fcac9c4b700 1 Processor -- start 2024-02-19T21:02:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.078+0000 7fcac9c4b700 1 -- start start 2024-02-19T21:02:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.078+0000 7fcac9c4b700 1 librados: setting wanted keys 2024-02-19T21:02:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.078+0000 7fcac9c4b700 1 librados: calling monclient init 2024-02-19T21:02:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.078+0000 7fcac9c4b700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcac41195e0 con 0x7fcac40f8980 2024-02-19T21:02:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.078+0000 7fcac9c4b700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcac411a7c0 con 0x7fcac41042b0 2024-02-19T21:02:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.078+0000 7fcac9c4b700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcac411b9c0 con 0x7fcac40f9530 2024-02-19T21:02:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac37fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fcac40f8980 0x7fcac40fd300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:45358/0 (socket says 172.21.15.163:45358) 2024-02-19T21:02:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac37fe700 1 -- 172.21.15.163:0/2297154307 learned_addr learned my addr 172.21.15.163:0/2297154307 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 10 client.?.objecter ms_handle_connect 0x7fcac40f8980 2024-02-19T21:02:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 10 client.?.objecter ms_handle_connect 0x7fcac41042b0 2024-02-19T21:02:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 10 client.?.objecter ms_handle_connect 0x7fcac40f9530 2024-02-19T21:02:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112332223 0 0) 0x7fcac41195e0 con 0x7fcac40f8980 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcab00033f0 con 0x7fcac40f8980 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341997913 0 0) 0x7fcac411a7c0 con 0x7fcac41042b0 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac41195e0 con 0x7fcac41042b0 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676661970 0 0) 0x7fcac411b9c0 con 0x7fcac40f9530 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcac411a7c0 con 0x7fcac40f9530 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4055795473 0 0) 0x7fcab00033f0 con 0x7fcac40f8980 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcac411b9c0 con 0x7fcac40f8980 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcaac0019d0 con 0x7fcac40f8980 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4197190079 0 0) 0x7fcac41195e0 con 0x7fcac41042b0 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcab00033f0 con 0x7fcac41042b0 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3677223780 0 0) 0x7fcac411a7c0 con 0x7fcac40f9530 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcac41195e0 con 0x7fcac40f9530 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcabc002180 con 0x7fcac41042b0 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcab4001800 con 0x7fcac40f9530 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2425009135 0 0) 0x7fcac411b9c0 con 0x7fcac40f8980 2024-02-19T21:02:06.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 >> v1:172.21.15.163:6790/0 conn(0x7fcac40f9530 legacy=0x7fcac40fda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 >> v1:172.21.15.193:6789/0 conn(0x7fcac41042b0 legacy=0x7fcac4117d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac411cbc0 con 0x7fcac40f8980 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac9c4b700 1 -- 172.21.15.163:0/2297154307 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac411a9b0 con 0x7fcac40f8980 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac9c4b700 10 client.14616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac9c4b700 1 -- 172.21.15.163:0/2297154307 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac411ae20 con 0x7fcac40f8980 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac9c4b700 1 librados: init done 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fcaac002510 con 0x7fcac40f8980 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.079+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcaac005bb0 con 0x7fcac40f8980 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.080+0000 7fcac9c4b700 1 -- 172.21.15.163:0/2297154307 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa4004f20 con 0x7fcac40f8980 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.081+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fcaac018a40 con 0x7fcac40f8980 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.081+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(76..76 src has 1..76) v4 ==== 6172+0+0 (unknown 1515610916 0 0) 0x7fcaac078600 con 0x7fcac40f8980 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.081+0000 7fcac0ff9700 10 client.14616.objecter ms_dispatch 0x7fcac4001550 osd_map(76..76 src has 1..76) v4 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.081+0000 7fcac0ff9700 3 client.14616.objecter handle_osd_map got epochs [76,76] > 0 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.081+0000 7fcac0ff9700 3 client.14616.objecter handle_osd_map decoding full epoch 76 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.081+0000 7fcac0ff9700 20 client.14616.objecter dump_active .. 0 homeless 2024-02-19T21:02:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.081+0000 7fcac0ff9700 10 client.14616.objecter ms_handle_connect 0x7fcab0063d70 2024-02-19T21:02:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.085+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fcaac04c2a0 con 0x7fcac40f8980 2024-02-19T21:02:06.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.248+0000 7fcac9c4b700 1 -- 172.21.15.163:0/2297154307 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"} v 0) v1 -- 0x7fcaa4005750 con 0x7fcac40f8980 2024-02-19T21:02:06.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.249+0000 7fcac0ff9700 1 -- 172.21.15.163:0/2297154307 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]=0 v76) v1 ==== 108+0+11 (unknown 1856407266 0 3519293104) 0x7fcaac04a6b0 con 0x7fcac40f8980 2024-02-19T21:02:06.239 INFO:teuthology.orchestra.run.smithi163.stdout:pg_num: 32 2024-02-19T21:02:06.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.250+0000 7fcac9c4b700 10 librados: watch_flush enter 2024-02-19T21:02:06.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.250+0000 7fcac9c4b700 10 librados: watch_flush exit 2024-02-19T21:02:06.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.251+0000 7fcac9c4b700 20 client.14616.objecter shutdown clearing up homeless session... 2024-02-19T21:02:06.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.251+0000 7fcac9c4b700 10 client.14616.objecter successfully canceled tick 2024-02-19T21:02:06.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.251+0000 7fcac9c4b700 1 -- 172.21.15.163:0/2297154307 >> v1:172.21.15.163:6800/1552514681 conn(0x7fcab0063d70 legacy=0x7fcab0066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:06.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.251+0000 7fcac9c4b700 1 -- 172.21.15.163:0/2297154307 >> v1:172.21.15.163:6789/0 conn(0x7fcac40f8980 legacy=0x7fcac40fd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:06.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.251+0000 7fcac9c4b700 1 -- 172.21.15.163:0/2297154307 shutdown_connections 2024-02-19T21:02:06.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.251+0000 7fcac9c4b700 1 -- 172.21.15.163:0/2297154307 >> 172.21.15.163:0/2297154307 conn(0x7fcac40f2470 msgr2=0x7fcac41073c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:06.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.251+0000 7fcac9c4b700 1 -- 172.21.15.163:0/2297154307 shutdown_connections 2024-02-19T21:02:06.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.251+0000 7fcac9c4b700 1 -- 172.21.15.163:0/2297154307 wait complete. 2024-02-19T21:02:06.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:06.251+0000 7fcac9c4b700 1 librados: shutdown 2024-02-19T21:02:06.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:04.916230+0000 mgr.y (mgr.14550) 29 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:02:06.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:05.118963+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:06.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:05.119014+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:06.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:05.177270+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-02-19T21:02:06.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:04.916230+0000 mgr.y (mgr.14550) 29 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:02:06.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:05.118963+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:06.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:05.119014+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:06.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:05.177270+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-02-19T21:02:06.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:04.916230+0000 mgr.y (mgr.14550) 29 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:02:06.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:05.118963+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:06.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:05.119014+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:06.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:05.177270+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-02-19T21:02:06.928 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool get default.rgw.meta pg_num 2024-02-19T21:02:07.168 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:06.250422+0000 mon.a (mon.0) 673 : audit [DBG] from='client.? v1:172.21.15.163:0/2297154307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2024-02-19T21:02:07.199 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:07.440 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:06.250422+0000 mon.a (mon.0) 673 : audit [DBG] from='client.? v1:172.21.15.163:0/2297154307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2024-02-19T21:02:07.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:06.250422+0000 mon.a (mon.0) 673 : audit [DBG] from='client.? v1:172.21.15.163:0/2297154307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2024-02-19T21:02:08.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:06.916572+0000 mgr.y (mgr.14550) 30 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:02:08.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:06.916572+0000 mgr.y (mgr.14550) 30 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:02:08.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:06.916572+0000 mgr.y (mgr.14550) 30 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:02:08.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.785+0000 7fd69a730700 1 -- 172.21.15.163:0/1653556260 >> v1:172.21.15.163:6790/0 conn(0x7fd6941031b0 legacy=0x7fd694105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:08.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.785+0000 7fd69a730700 1 -- 172.21.15.163:0/1653556260 shutdown_connections 2024-02-19T21:02:08.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.785+0000 7fd69a730700 1 -- 172.21.15.163:0/1653556260 >> 172.21.15.163:0/1653556260 conn(0x7fd6940f2470 msgr2=0x7fd6940f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:08.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.785+0000 7fd69a730700 1 -- 172.21.15.163:0/1653556260 shutdown_connections 2024-02-19T21:02:08.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.785+0000 7fd69a730700 1 -- 172.21.15.163:0/1653556260 wait complete. 2024-02-19T21:02:08.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.786+0000 7fd69a730700 1 librados: starting msgr at 2024-02-19T21:02:08.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.786+0000 7fd69a730700 1 librados: starting objecter 2024-02-19T21:02:08.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.786+0000 7fd69a730700 1 Processor -- start 2024-02-19T21:02:08.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.786+0000 7fd69a730700 1 -- start start 2024-02-19T21:02:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.786+0000 7fd69a730700 1 librados: setting wanted keys 2024-02-19T21:02:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.786+0000 7fd69a730700 1 librados: calling monclient init 2024-02-19T21:02:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.786+0000 7fd69a730700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd694119520 con 0x7fd6940fc870 2024-02-19T21:02:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.786+0000 7fd69a730700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd69411a700 con 0x7fd6940ff670 2024-02-19T21:02:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.786+0000 7fd69a730700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd69411b900 con 0x7fd6941031b0 2024-02-19T21:02:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.786+0000 7fd693fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fd6940fc870 0x7fd6940609e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:45376/0 (socket says 172.21.15.163:45376) 2024-02-19T21:02:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.786+0000 7fd693fff700 1 -- 172.21.15.163:0/1775727436 learned_addr learned my addr 172.21.15.163:0/1775727436 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 10 client.?.objecter ms_handle_connect 0x7fd6940fc870 2024-02-19T21:02:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 10 client.?.objecter ms_handle_connect 0x7fd6941031b0 2024-02-19T21:02:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 10 client.?.objecter ms_handle_connect 0x7fd6940ff670 2024-02-19T21:02:08.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572703829 0 0) 0x7fd694119520 con 0x7fd6940fc870 2024-02-19T21:02:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd67c003030 con 0x7fd6940fc870 2024-02-19T21:02:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366136709 0 0) 0x7fd69411b900 con 0x7fd6941031b0 2024-02-19T21:02:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd694119520 con 0x7fd6941031b0 2024-02-19T21:02:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967223148 0 0) 0x7fd69411a700 con 0x7fd6940ff670 2024-02-19T21:02:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd69411b900 con 0x7fd6940ff670 2024-02-19T21:02:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2599695056 0 0) 0x7fd67c003030 con 0x7fd6940fc870 2024-02-19T21:02:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd69411a700 con 0x7fd6940fc870 2024-02-19T21:02:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd6840019c0 con 0x7fd6940fc870 2024-02-19T21:02:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 110506392 0 0) 0x7fd694119520 con 0x7fd6941031b0 2024-02-19T21:02:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd67c003030 con 0x7fd6941031b0 2024-02-19T21:02:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd68c001360 con 0x7fd6941031b0 2024-02-19T21:02:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2663458819 0 0) 0x7fd69411b900 con 0x7fd6940ff670 2024-02-19T21:02:08.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd694119520 con 0x7fd6940ff670 2024-02-19T21:02:08.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd688001840 con 0x7fd6940ff670 2024-02-19T21:02:08.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4244629251 0 0) 0x7fd69411a700 con 0x7fd6940fc870 2024-02-19T21:02:08.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 >> v1:172.21.15.163:6790/0 conn(0x7fd6941031b0 legacy=0x7fd694117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:08.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 >> v1:172.21.15.193:6789/0 conn(0x7fd6940ff670 legacy=0x7fd6940610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:08.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69411cb00 con 0x7fd6940fc870 2024-02-19T21:02:08.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd69a730700 1 -- 172.21.15.163:0/1775727436 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69411baf0 con 0x7fd6940fc870 2024-02-19T21:02:08.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd69a730700 10 client.14622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:08.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd69a730700 1 -- 172.21.15.163:0/1775727436 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69411bfe0 con 0x7fd6940fc870 2024-02-19T21:02:08.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd69a730700 1 librados: init done 2024-02-19T21:02:08.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fd6840023b0 con 0x7fd6940fc870 2024-02-19T21:02:08.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.787+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd684005ae0 con 0x7fd6940fc870 2024-02-19T21:02:08.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.788+0000 7fd69a730700 1 -- 172.21.15.163:0/1775727436 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd680004f20 con 0x7fd6940fc870 2024-02-19T21:02:08.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.789+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fd684001f10 con 0x7fd6940fc870 2024-02-19T21:02:08.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.789+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(76..76 src has 1..76) v4 ==== 6172+0+0 (unknown 1515610916 0 0) 0x7fd684078430 con 0x7fd6940fc870 2024-02-19T21:02:08.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.789+0000 7fd6917fa700 10 client.14622.objecter ms_dispatch 0x7fd694001550 osd_map(76..76 src has 1..76) v4 2024-02-19T21:02:08.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.789+0000 7fd6917fa700 3 client.14622.objecter handle_osd_map got epochs [76,76] > 0 2024-02-19T21:02:08.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.789+0000 7fd6917fa700 3 client.14622.objecter handle_osd_map decoding full epoch 76 2024-02-19T21:02:08.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.789+0000 7fd6917fa700 20 client.14622.objecter dump_active .. 0 homeless 2024-02-19T21:02:08.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.789+0000 7fd6917fa700 10 client.14622.objecter ms_handle_connect 0x7fd67c063d10 2024-02-19T21:02:08.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.793+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd68404be10 con 0x7fd6940fc870 2024-02-19T21:02:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.955+0000 7fd69a730700 1 -- 172.21.15.163:0/1775727436 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"} v 0) v1 -- 0x7fd680005750 con 0x7fd6940fc870 2024-02-19T21:02:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.956+0000 7fd6917fa700 1 -- 172.21.15.163:0/1775727436 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]=0 v76) v1 ==== 105+0+11 (unknown 1735117824 0 3519293104) 0x7fd68404a4b0 con 0x7fd6940fc870 2024-02-19T21:02:08.946 INFO:teuthology.orchestra.run.smithi163.stdout:pg_num: 32 2024-02-19T21:02:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.957+0000 7fd69a730700 10 librados: watch_flush enter 2024-02-19T21:02:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.957+0000 7fd69a730700 10 librados: watch_flush exit 2024-02-19T21:02:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.957+0000 7fd69a730700 20 client.14622.objecter shutdown clearing up homeless session... 2024-02-19T21:02:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.957+0000 7fd69a730700 10 client.14622.objecter successfully canceled tick 2024-02-19T21:02:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.958+0000 7fd69a730700 1 -- 172.21.15.163:0/1775727436 >> v1:172.21.15.163:6800/1552514681 conn(0x7fd67c063d10 legacy=0x7fd67c0661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.958+0000 7fd69a730700 1 -- 172.21.15.163:0/1775727436 >> v1:172.21.15.163:6789/0 conn(0x7fd6940fc870 legacy=0x7fd6940609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:08.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.958+0000 7fd69a730700 1 -- 172.21.15.163:0/1775727436 shutdown_connections 2024-02-19T21:02:08.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.958+0000 7fd69a730700 1 -- 172.21.15.163:0/1775727436 >> 172.21.15.163:0/1775727436 conn(0x7fd6940f2470 msgr2=0x7fd694106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:08.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.958+0000 7fd69a730700 1 -- 172.21.15.163:0/1775727436 shutdown_connections 2024-02-19T21:02:08.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.958+0000 7fd69a730700 1 -- 172.21.15.163:0/1775727436 wait complete. 2024-02-19T21:02:08.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:08.958+0000 7fd69a730700 1 librados: shutdown 2024-02-19T21:02:09.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:08.957421+0000 mon.a (mon.0) 674 : audit [DBG] from='client.? v1:172.21.15.163:0/1775727436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2024-02-19T21:02:09.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:08.957421+0000 mon.a (mon.0) 674 : audit [DBG] from='client.? v1:172.21.15.163:0/1775727436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2024-02-19T21:02:09.541 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:08.957421+0000 mon.a (mon.0) 674 : audit [DBG] from='client.? v1:172.21.15.163:0/1775727436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2024-02-19T21:02:09.619 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-19T21:02:09.619 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-19T21:02:09.619 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph mgr dump --format=json 2024-02-19T21:02:09.887 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:10.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:02:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:02:10.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:08.916943+0000 mgr.y (mgr.14550) 31 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 659 B/s rd, 0 op/s 2024-02-19T21:02:10.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:08.916943+0000 mgr.y (mgr.14550) 31 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 659 B/s rd, 0 op/s 2024-02-19T21:02:10.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:08.916943+0000 mgr.y (mgr.14550) 31 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 659 B/s rd, 0 op/s 2024-02-19T21:02:11.397 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:09.940011+0000 mgr.y (mgr.14550) 32 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:11.397 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:09.940011+0000 mgr.y (mgr.14550) 32 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:11.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:09.940011+0000 mgr.y (mgr.14550) 32 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:11.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.701+0000 7fd5f95aa700 1 -- 172.21.15.163:0/3597719283 >> v1:172.21.15.163:6790/0 conn(0x7fd5f40fd970 legacy=0x7fd5f40fdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:11.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.701+0000 7fd5f95aa700 1 -- 172.21.15.163:0/3597719283 shutdown_connections 2024-02-19T21:02:11.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.701+0000 7fd5f95aa700 1 -- 172.21.15.163:0/3597719283 >> 172.21.15.163:0/3597719283 conn(0x7fd5f40f3590 msgr2=0x7fd5f40f59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:11.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.701+0000 7fd5f95aa700 1 -- 172.21.15.163:0/3597719283 shutdown_connections 2024-02-19T21:02:11.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.701+0000 7fd5f95aa700 1 -- 172.21.15.163:0/3597719283 wait complete. 2024-02-19T21:02:11.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.701+0000 7fd5f95aa700 1 librados: starting msgr at 2024-02-19T21:02:11.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.701+0000 7fd5f95aa700 1 librados: starting objecter 2024-02-19T21:02:11.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.701+0000 7fd5f95aa700 1 Processor -- start 2024-02-19T21:02:11.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.701+0000 7fd5f95aa700 1 -- start start 2024-02-19T21:02:11.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.701+0000 7fd5f95aa700 1 librados: setting wanted keys 2024-02-19T21:02:11.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.701+0000 7fd5f95aa700 1 librados: calling monclient init 2024-02-19T21:02:11.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5f95aa700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd5f4119440 con 0x7fd5f4100770 2024-02-19T21:02:11.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5f95aa700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd5f411a5e0 con 0x7fd5f41042b0 2024-02-19T21:02:11.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5f95aa700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd5f411b780 con 0x7fd5f40fd970 2024-02-19T21:02:11.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5f2ffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fd5f40fd970 0x7fd5f40fc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:46924/0 (socket says 172.21.15.163:46924) 2024-02-19T21:02:11.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5f2ffd700 1 -- 172.21.15.163:0/3622462143 learned_addr learned my addr 172.21.15.163:0/3622462143 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:11.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 10 client.?.objecter ms_handle_connect 0x7fd5f40fd970 2024-02-19T21:02:11.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:11.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 10 client.?.objecter ms_handle_connect 0x7fd5f4100770 2024-02-19T21:02:11.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 10 client.?.objecter ms_handle_connect 0x7fd5f41042b0 2024-02-19T21:02:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017625039 0 0) 0x7fd5f4119440 con 0x7fd5f4100770 2024-02-19T21:02:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5dc003630 con 0x7fd5f4100770 2024-02-19T21:02:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025581039 0 0) 0x7fd5f411b780 con 0x7fd5f40fd970 2024-02-19T21:02:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5f4119440 con 0x7fd5f40fd970 2024-02-19T21:02:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479396416 0 0) 0x7fd5f411a5e0 con 0x7fd5f41042b0 2024-02-19T21:02:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5f411b780 con 0x7fd5f41042b0 2024-02-19T21:02:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2709078862 0 0) 0x7fd5dc003630 con 0x7fd5f4100770 2024-02-19T21:02:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5f411a5e0 con 0x7fd5f4100770 2024-02-19T21:02:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd5e40019f0 con 0x7fd5f4100770 2024-02-19T21:02:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 144008012 0 0) 0x7fd5f4119440 con 0x7fd5f40fd970 2024-02-19T21:02:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5dc003630 con 0x7fd5f40fd970 2024-02-19T21:02:11.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd5e8001f50 con 0x7fd5f40fd970 2024-02-19T21:02:11.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1565790353 0 0) 0x7fd5f411b780 con 0x7fd5f41042b0 2024-02-19T21:02:11.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5f4119440 con 0x7fd5f41042b0 2024-02-19T21:02:11.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.702+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd5ec001bc0 con 0x7fd5f41042b0 2024-02-19T21:02:11.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.703+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2393609877 0 0) 0x7fd5f411a5e0 con 0x7fd5f4100770 2024-02-19T21:02:11.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.703+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 >> v1:172.21.15.163:6790/0 conn(0x7fd5f40fd970 legacy=0x7fd5f40fc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:11.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.703+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 >> v1:172.21.15.193:6789/0 conn(0x7fd5f41042b0 legacy=0x7fd5f4117d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:11.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.703+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f411c920 con 0x7fd5f4100770 2024-02-19T21:02:11.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.703+0000 7fd5f95aa700 1 -- 172.21.15.163:0/3622462143 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f4119630 con 0x7fd5f4100770 2024-02-19T21:02:11.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.703+0000 7fd5f95aa700 10 client.14628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:11.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.703+0000 7fd5f95aa700 1 -- 172.21.15.163:0/3622462143 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f4119b20 con 0x7fd5f4100770 2024-02-19T21:02:11.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.703+0000 7fd5f95aa700 1 librados: init done 2024-02-19T21:02:11.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.703+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fd5e40022a0 con 0x7fd5f4100770 2024-02-19T21:02:11.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.703+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd5e4003290 con 0x7fd5f4100770 2024-02-19T21:02:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.704+0000 7fd5f95aa700 1 -- 172.21.15.163:0/3622462143 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e0004f20 con 0x7fd5f4100770 2024-02-19T21:02:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.704+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fd5e40033f0 con 0x7fd5f4100770 2024-02-19T21:02:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.704+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(76..76 src has 1..76) v4 ==== 6172+0+0 (unknown 1515610916 0 0) 0x7fd5e4075ba0 con 0x7fd5f4100770 2024-02-19T21:02:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.704+0000 7fd5dbfff700 10 client.14628.objecter ms_dispatch 0x7fd5f4001550 osd_map(76..76 src has 1..76) v4 2024-02-19T21:02:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.704+0000 7fd5dbfff700 3 client.14628.objecter handle_osd_map got epochs [76,76] > 0 2024-02-19T21:02:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.704+0000 7fd5dbfff700 3 client.14628.objecter handle_osd_map decoding full epoch 76 2024-02-19T21:02:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.705+0000 7fd5dbfff700 20 client.14628.objecter dump_active .. 0 homeless 2024-02-19T21:02:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.705+0000 7fd5dbfff700 10 client.14628.objecter ms_handle_connect 0x7fd5dc063dd0 2024-02-19T21:02:11.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.709+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd5e4049730 con 0x7fd5f4100770 2024-02-19T21:02:11.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.916+0000 7fd5f95aa700 1 -- 172.21.15.163:0/3622462143 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fd5e00051d0 con 0x7fd5f4100770 2024-02-19T21:02:11.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.919+0000 7fd5dbfff700 1 -- 172.21.15.163:0/3622462143 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+151943 (unknown 3383617513 0 2318807745) 0x7fd5e4047b40 con 0x7fd5f4100770 2024-02-19T21:02:11.912 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:02:11.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.921+0000 7fd5f95aa700 10 librados: watch_flush enter 2024-02-19T21:02:11.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.921+0000 7fd5f95aa700 10 librados: watch_flush exit 2024-02-19T21:02:11.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.921+0000 7fd5f95aa700 20 client.14628.objecter shutdown clearing up homeless session... 2024-02-19T21:02:11.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.921+0000 7fd5f95aa700 10 client.14628.objecter successfully canceled tick 2024-02-19T21:02:11.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.921+0000 7fd5f95aa700 1 -- 172.21.15.163:0/3622462143 >> v1:172.21.15.163:6800/1552514681 conn(0x7fd5dc063dd0 legacy=0x7fd5dc066290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:11.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.921+0000 7fd5f95aa700 1 -- 172.21.15.163:0/3622462143 >> v1:172.21.15.163:6789/0 conn(0x7fd5f4100770 legacy=0x7fd5f40fcca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:11.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.922+0000 7fd5f95aa700 1 -- 172.21.15.163:0/3622462143 shutdown_connections 2024-02-19T21:02:11.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.922+0000 7fd5f95aa700 1 -- 172.21.15.163:0/3622462143 >> 172.21.15.163:0/3622462143 conn(0x7fd5f40f3590 msgr2=0x7fd5f4107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:11.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.922+0000 7fd5f95aa700 1 -- 172.21.15.163:0/3622462143 shutdown_connections 2024-02-19T21:02:11.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.922+0000 7fd5f95aa700 1 -- 172.21.15.163:0/3622462143 wait complete. 2024-02-19T21:02:11.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:11.922+0000 7fd5f95aa700 1 librados: shutdown 2024-02-19T21:02:12.311 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 systemd[1]: Starting Ceph prometheus.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:02:12.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:10.917558+0000 mgr.y (mgr.14550) 33 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:12.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:11.918347+0000 mon.a (mon.0) 675 : audit [DBG] from='client.? v1:172.21.15.163:0/3622462143' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-19T21:02:12.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:10.917558+0000 mgr.y (mgr.14550) 33 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:12.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:11.918347+0000 mon.a (mon.0) 675 : audit [DBG] from='client.? v1:172.21.15.163:0/3622462143' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-19T21:02:12.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:10.917558+0000 mgr.y (mgr.14550) 33 : cluster [DBG] pgmap v23: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:12.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:11.918347+0000 mon.a (mon.0) 675 : audit [DBG] from='client.? v1:172.21.15.163:0/3622462143' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-19T21:02:12.552 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":19,"active_gid":14550,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6800","nonce":1552514681}]},"active_addr":"172.21.15.163:6800/1552514681","active_change":"2024-02-19T21:01:38.896527+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[{"gid":14544,"name":"x","mgr_features":4540138314316775423,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.163:8443/","prometheus":"http://172.21.15.163:9283/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":72,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.163:0","nonce":2351957517}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.163:0","nonce":2366913468}]}]} 2024-02-19T21:02:12.555 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-19T21:02:12.555 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-19T21:02:12.556 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-19T21:02:12.811 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.554Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-19T21:02:12.812 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.554Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-19T21:02:12.812 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.554Z caller=main.go:518 level=info host_details="(Linux 4.18.0-539.el8.x86_64 #1 SMP Mon Feb 5 18:34:56 UTC 2024 x86_64 smithi193 (none))" 2024-02-19T21:02:12.812 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.554Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-19T21:02:12.812 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.554Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-19T21:02:12.812 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.555Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-19T21:02:12.812 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.556Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-19T21:02:12.812 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.557Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-19T21:02:12.812 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.558Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-19T21:02:12.812 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.558Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.979µs 2024-02-19T21:02:12.812 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.558Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-19T21:02:12.812 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.559Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-02-19T21:02:12.812 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.559Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.601µs wal_replay_duration=307.083µs total_replay_duration=359.651µs 2024-02-19T21:02:12.813 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.559Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-19T21:02:12.813 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.559Z caller=main.go:947 level=info msg="TSDB started" 2024-02-19T21:02:12.813 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.559Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-19T21:02:12.813 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.575Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.82365ms db_storage=585ns remote_storage=1.202µs web_handler=343ns query_engine=885ns scrape=429.562µs scrape_sd=21.579µs notify=37.594µs notify_sd=10.131µs rules=15.104086ms 2024-02-19T21:02:12.813 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:02:12.575Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-19T21:02:12.813 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 bash[62633]: 728b12e28f64b1d242f77e87053e4f3b9b86a3038be0dd50fda6d0624e1aaf77 2024-02-19T21:02:12.813 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:02:12 smithi193 systemd[1]: Started Ceph prometheus.a for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T21:02:12.833 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:12.836 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:12] ENGINE Bus STOPPING 2024-02-19T21:02:12.837 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-19T21:02:12.837 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:12] ENGINE Bus STOPPED 2024-02-19T21:02:12.837 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:12] ENGINE Bus STARTING 2024-02-19T21:02:12.837 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:12] ENGINE Serving on http://:::9283 2024-02-19T21:02:13.094 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:12] ENGINE Bus STARTED 2024-02-19T21:02:13.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:12.656714+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:13.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:12.713516+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:13.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:12.719228+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:13.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:12.723253+0000 mon.b (mon.1) 60 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T21:02:13.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:12.723686+0000 mgr.y (mgr.14550) 34 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T21:02:13.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:12.728267+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:13.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:02:12.739296+0000 mgr.y (mgr.14550) 35 : cephadm [INF] Deploying daemon grafana.a on smithi193 2024-02-19T21:02:13.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:12.918123+0000 mgr.y (mgr.14550) 36 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:02:13.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:12.656714+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:13.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:12.713516+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:13.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:12.719228+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:13.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:12.723253+0000 mon.b (mon.1) 60 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T21:02:13.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:12.723686+0000 mgr.y (mgr.14550) 34 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T21:02:13.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:12.728267+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:13.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:02:12.739296+0000 mgr.y (mgr.14550) 35 : cephadm [INF] Deploying daemon grafana.a on smithi193 2024-02-19T21:02:13.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:12.918123+0000 mgr.y (mgr.14550) 36 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:02:14.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:12.656714+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:14.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:12.713516+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:14.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:12.719228+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:14.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:12.723253+0000 mon.b (mon.1) 60 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T21:02:14.062 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:12.723686+0000 mgr.y (mgr.14550) 34 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T21:02:14.062 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:12.728267+0000 mon.a (mon.0) 679 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:14.062 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:02:12.739296+0000 mgr.y (mgr.14550) 35 : cephadm [INF] Deploying daemon grafana.a on smithi193 2024-02-19T21:02:14.062 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:12.918123+0000 mgr.y (mgr.14550) 36 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:02:14.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.511+0000 7f6d93e58700 1 -- 172.21.15.163:0/1790680681 >> v1:172.21.15.163:6789/0 conn(0x7f6d8c0fc870 legacy=0x7f6d8c0fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:14.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.512+0000 7f6d93e58700 1 -- 172.21.15.163:0/1790680681 shutdown_connections 2024-02-19T21:02:14.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.512+0000 7f6d93e58700 1 -- 172.21.15.163:0/1790680681 >> 172.21.15.163:0/1790680681 conn(0x7f6d8c0f2470 msgr2=0x7f6d8c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:14.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.512+0000 7f6d93e58700 1 -- 172.21.15.163:0/1790680681 shutdown_connections 2024-02-19T21:02:14.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.512+0000 7f6d93e58700 1 -- 172.21.15.163:0/1790680681 wait complete. 2024-02-19T21:02:14.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.512+0000 7f6d93e58700 1 librados: starting msgr at 2024-02-19T21:02:14.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.512+0000 7f6d93e58700 1 librados: starting objecter 2024-02-19T21:02:14.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.512+0000 7f6d93e58700 1 Processor -- start 2024-02-19T21:02:14.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.512+0000 7f6d93e58700 1 -- start start 2024-02-19T21:02:14.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.512+0000 7f6d93e58700 1 librados: setting wanted keys 2024-02-19T21:02:14.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.512+0000 7f6d93e58700 1 librados: calling monclient init 2024-02-19T21:02:14.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.512+0000 7f6d93e58700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6d8c1173b0 con 0x7f6d8c0fc870 2024-02-19T21:02:14.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.512+0000 7f6d93e58700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6d8c118590 con 0x7f6d8c1031b0 2024-02-19T21:02:14.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.512+0000 7f6d93e58700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6d8c119790 con 0x7f6d8c0ff670 2024-02-19T21:02:14.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.512+0000 7f6d91bf4700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f6d8c0fc870 0x7f6d8c0f9b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:41050/0 (socket says 172.21.15.163:41050) 2024-02-19T21:02:14.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d91bf4700 1 -- 172.21.15.163:0/1392880547 learned_addr learned my addr 172.21.15.163:0/1392880547 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:14.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 10 client.?.objecter ms_handle_connect 0x7f6d8c0fc870 2024-02-19T21:02:14.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:14.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 10 client.?.objecter ms_handle_connect 0x7f6d8c0ff670 2024-02-19T21:02:14.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:14.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 10 client.?.objecter ms_handle_connect 0x7f6d8c1031b0 2024-02-19T21:02:14.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:14.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543827045 0 0) 0x7f6d8c119790 con 0x7f6d8c0ff670 2024-02-19T21:02:14.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d74003800 con 0x7f6d8c0ff670 2024-02-19T21:02:14.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907241304 0 0) 0x7f6d8c1173b0 con 0x7f6d8c0fc870 2024-02-19T21:02:14.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d8c119790 con 0x7f6d8c0fc870 2024-02-19T21:02:14.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265869022 0 0) 0x7f6d8c118590 con 0x7f6d8c1031b0 2024-02-19T21:02:14.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d8c1173b0 con 0x7f6d8c1031b0 2024-02-19T21:02:14.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3655396404 0 0) 0x7f6d74003800 con 0x7f6d8c0ff670 2024-02-19T21:02:14.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d8c118590 con 0x7f6d8c0ff670 2024-02-19T21:02:14.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6d800018c0 con 0x7f6d8c0ff670 2024-02-19T21:02:14.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3664930441 0 0) 0x7f6d8c119790 con 0x7f6d8c0fc870 2024-02-19T21:02:14.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d74003800 con 0x7f6d8c0fc870 2024-02-19T21:02:14.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6d84002260 con 0x7f6d8c0fc870 2024-02-19T21:02:14.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1215885023 0 0) 0x7f6d8c1173b0 con 0x7f6d8c1031b0 2024-02-19T21:02:14.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d8c119790 con 0x7f6d8c1031b0 2024-02-19T21:02:14.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6d7c001620 con 0x7f6d8c1031b0 2024-02-19T21:02:14.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3977636051 0 0) 0x7f6d8c118590 con 0x7f6d8c0ff670 2024-02-19T21:02:14.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 >> v1:172.21.15.193:6789/0 conn(0x7f6d8c1031b0 legacy=0x7f6d8c115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:14.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 >> v1:172.21.15.163:6789/0 conn(0x7f6d8c0fc870 legacy=0x7f6d8c0f9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:14.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c11a990 con 0x7f6d8c0ff670 2024-02-19T21:02:14.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d93e58700 1 -- 172.21.15.163:0/1392880547 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8c1175a0 con 0x7f6d8c0ff670 2024-02-19T21:02:14.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d93e58700 10 client.34559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:14.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d93e58700 1 -- 172.21.15.163:0/1392880547 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8c117a10 con 0x7f6d8c0ff670 2024-02-19T21:02:14.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d93e58700 1 librados: init done 2024-02-19T21:02:14.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6d80001f80 con 0x7f6d8c0ff670 2024-02-19T21:02:14.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.513+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6d800032a0 con 0x7f6d8c0ff670 2024-02-19T21:02:14.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.514+0000 7f6d93e58700 1 -- 172.21.15.163:0/1392880547 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d78004f20 con 0x7f6d8c0ff670 2024-02-19T21:02:14.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.515+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6d80001ae0 con 0x7f6d8c0ff670 2024-02-19T21:02:14.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.515+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(76..76 src has 1..76) v4 ==== 6172+0+0 (unknown 1515610916 0 0) 0x7f6d80075d40 con 0x7f6d8c0ff670 2024-02-19T21:02:14.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.515+0000 7f6d8affd700 10 client.34559.objecter ms_dispatch 0x7f6d8c001550 osd_map(76..76 src has 1..76) v4 2024-02-19T21:02:14.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.515+0000 7f6d8affd700 3 client.34559.objecter handle_osd_map got epochs [76,76] > 0 2024-02-19T21:02:14.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.515+0000 7f6d8affd700 3 client.34559.objecter handle_osd_map decoding full epoch 76 2024-02-19T21:02:14.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.515+0000 7f6d8affd700 20 client.34559.objecter dump_active .. 0 homeless 2024-02-19T21:02:14.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.515+0000 7f6d8affd700 10 client.34559.objecter ms_handle_connect 0x7f6d74063d90 2024-02-19T21:02:14.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.519+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6d800498f0 con 0x7f6d8c0ff670 2024-02-19T21:02:14.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.683+0000 7f6d93e58700 1 -- 172.21.15.163:0/1392880547 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d780051d0 con 0x7f6d8c0ff670 2024-02-19T21:02:14.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.684+0000 7f6d8affd700 1 -- 172.21.15.163:0/1392880547 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+19240 (unknown 2435502076 0 639989584) 0x7f6d80047d00 con 0x7f6d8c0ff670 2024-02-19T21:02:14.674 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:02:14.675 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":76,"fsid":"2591809a-cf69-11ee-95bb-87774f69a715","created":"2024-02-19T20:55:37.794885+0000","modified":"2024-02-19T21:02:05.168060+0000","last_up_change":"2024-02-19T21:01:09.214134+0000","last_in_change":"2024-02-19T21:00:51.699017+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-19T20:59:09.425169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-19T21:01:14.658945+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-19T21:01:14.958800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-19T21:01:16.543816+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-19T21:01:18.521482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-19T21:01:20.586564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"4a14b406-7c21-40f3-8b12-8cf9df811088","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6801","nonce":1559418115}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6802","nonce":1559418115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6804","nonce":1559418115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6803","nonce":1559418115}]},"public_addr":"172.21.15.163:6801/1559418115","cluster_addr":"172.21.15.163:6802/1559418115","heartbeat_back_addr":"172.21.15.163:6804/1559418115","heartbeat_front_addr":"172.21.15.163:6803/1559418115","state":["exists","up"]},{"osd":1,"uuid":"e74a65fa-cb9c-42c3-9293-602cf7e9dc67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6805","nonce":3125047432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6806","nonce":3125047432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6808","nonce":3125047432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6807","nonce":3125047432}]},"public_addr":"172.21.15.163:6805/3125047432","cluster_addr":"172.21.15.163:6806/3125047432","heartbeat_back_addr":"172.21.15.163:6808/3125047432","heartbeat_front_addr":"172.21.15.163:6807/3125047432","state":["exists","up"]},{"osd":2,"uuid":"6980a625-cdc9-487e-8f8c-7997ba716a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6809","nonce":1563158487}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6810","nonce":1563158487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6812","nonce":1563158487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6811","nonce":1563158487}]},"public_addr":"172.21.15.163:6809/1563158487","cluster_addr":"172.21.15.163:6810/1563158487","heartbeat_back_addr":"172.21.15.163:6812/1563158487","heartbeat_front_addr":"172.21.15.163:6811/1563158487","state":["exists","up"]},{"osd":3,"uuid":"359e2ee0-9d1d-4074-8119-2d69bc2ca3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6813","nonce":2112279769}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6814","nonce":2112279769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6816","nonce":2112279769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6815","nonce":2112279769}]},"public_addr":"172.21.15.163:6813/2112279769","cluster_addr":"172.21.15.163:6814/2112279769","heartbeat_back_addr":"172.21.15.163:6816/2112279769","heartbeat_front_addr":"172.21.15.163:6815/2112279769","state":["exists","up"]},{"osd":4,"uuid":"2bf5ef05-292b-4693-85a9-97991f2091e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6800","nonce":3129716965}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6801","nonce":3129716965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6803","nonce":3129716965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6802","nonce":3129716965}]},"public_addr":"172.21.15.193:6800/3129716965","cluster_addr":"172.21.15.193:6801/3129716965","heartbeat_back_addr":"172.21.15.193:6803/3129716965","heartbeat_front_addr":"172.21.15.193:6802/3129716965","state":["exists","up"]},{"osd":5,"uuid":"59065f9b-9240-414b-9c91-e91cda4aed6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6804","nonce":805161991}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6805","nonce":805161991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6807","nonce":805161991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6806","nonce":805161991}]},"public_addr":"172.21.15.193:6804/805161991","cluster_addr":"172.21.15.193:6805/805161991","heartbeat_back_addr":"172.21.15.193:6807/805161991","heartbeat_front_addr":"172.21.15.193:6806/805161991","state":["exists","up"]},{"osd":6,"uuid":"302c95df-3842-4818-8afd-d79d8184684f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6808","nonce":787604615}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6809","nonce":787604615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6811","nonce":787604615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6810","nonce":787604615}]},"public_addr":"172.21.15.193:6808/787604615","cluster_addr":"172.21.15.193:6809/787604615","heartbeat_back_addr":"172.21.15.193:6811/787604615","heartbeat_front_addr":"172.21.15.193:6810/787604615","state":["exists","up"]},{"osd":7,"uuid":"9ebeff27-c2f8-4a66-923f-a0d7573ddd3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6812","nonce":157927432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6813","nonce":157927432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6815","nonce":157927432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6814","nonce":157927432}]},"public_addr":"172.21.15.193:6812/157927432","cluster_addr":"172.21.15.193:6813/157927432","heartbeat_back_addr":"172.21.15.193:6815/157927432","heartbeat_front_addr":"172.21.15.193:6814/157927432","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:02:03.243866+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:11.764868+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:36.293275+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:35.739499+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:59.638643+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:23.163793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:02:05.119016+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:07.350223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":7,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:0/1427774891":"2024-02-20T21:01:38.896227+0000","172.21.15.163:0/3282102947":"2024-02-20T21:01:38.896227+0000","172.21.15.163:6800/2735092549":"2024-02-20T21:01:38.896227+0000","172.21.15.163:0/1235588846":"2024-02-20T20:56:29.603030+0000","172.21.15.163:6800/2731225015":"2024-02-20T20:56:29.603030+0000","172.21.15.163:0/506491651":"2024-02-20T20:56:02.185409+0000","172.21.15.163:0/168117826":"2024-02-20T21:01:38.896227+0000","172.21.15.163:0/1458009465":"2024-02-20T20:56:29.603030+0000","172.21.15.163:0/541631472":"2024-02-20T20:56:02.185409+0000","172.21.15.163:6800/2248999541":"2024-02-20T20:56:02.185409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T21:02:14.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.685+0000 7f6d93e58700 10 librados: watch_flush enter 2024-02-19T21:02:14.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.685+0000 7f6d93e58700 10 librados: watch_flush exit 2024-02-19T21:02:14.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.685+0000 7f6d93e58700 20 client.34559.objecter shutdown clearing up homeless session... 2024-02-19T21:02:14.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.685+0000 7f6d93e58700 10 client.34559.objecter successfully canceled tick 2024-02-19T21:02:14.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.686+0000 7f6d93e58700 1 -- 172.21.15.163:0/1392880547 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6d74063d90 legacy=0x7f6d74066250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:14.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.686+0000 7f6d93e58700 1 -- 172.21.15.163:0/1392880547 >> v1:172.21.15.163:6790/0 conn(0x7f6d8c0ff670 legacy=0x7f6d8c0fa230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:14.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.686+0000 7f6d93e58700 1 -- 172.21.15.163:0/1392880547 shutdown_connections 2024-02-19T21:02:14.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.686+0000 7f6d93e58700 1 -- 172.21.15.163:0/1392880547 >> 172.21.15.163:0/1392880547 conn(0x7f6d8c0f2470 msgr2=0x7f6d8c061200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:14.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.686+0000 7f6d93e58700 1 -- 172.21.15.163:0/1392880547 shutdown_connections 2024-02-19T21:02:14.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.686+0000 7f6d93e58700 1 -- 172.21.15.163:0/1392880547 wait complete. 2024-02-19T21:02:14.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:14.686+0000 7f6d93e58700 1 librados: shutdown 2024-02-19T21:02:15.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:14.095812+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:15.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:14.684718+0000 mon.c (mon.2) 23 : audit [DBG] from='client.? v1:172.21.15.163:0/1392880547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T21:02:15.334 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:14.095812+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:15.334 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:14.684718+0000 mon.c (mon.2) 23 : audit [DBG] from='client.? v1:172.21.15.163:0/1392880547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T21:02:15.357 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-19T21:02:15.357 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-19T21:02:15.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:14.095812+0000 mon.a (mon.0) 680 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:15.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:14.684718+0000 mon.c (mon.2) 23 : audit [DBG] from='client.? v1:172.21.15.163:0/1392880547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T21:02:15.630 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:16.110 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:14.918771+0000 mgr.y (mgr.14550) 37 : cluster [DBG] pgmap v25: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:16.110 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:14.918771+0000 mgr.y (mgr.14550) 37 : cluster [DBG] pgmap v25: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:16.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:14.918771+0000 mgr.y (mgr.14550) 37 : cluster [DBG] pgmap v25: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:17.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.756+0000 7f842be38700 1 -- 172.21.15.163:0/3908472341 >> v1:172.21.15.163:6790/0 conn(0x7f84240ff670 legacy=0x7f8424101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:17.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.756+0000 7f842be38700 1 -- 172.21.15.163:0/3908472341 shutdown_connections 2024-02-19T21:02:17.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.756+0000 7f842be38700 1 -- 172.21.15.163:0/3908472341 >> 172.21.15.163:0/3908472341 conn(0x7f84240f2470 msgr2=0x7f84240f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:17.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.756+0000 7f842be38700 1 -- 172.21.15.163:0/3908472341 shutdown_connections 2024-02-19T21:02:17.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.756+0000 7f842be38700 1 -- 172.21.15.163:0/3908472341 wait complete. 2024-02-19T21:02:17.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.756+0000 7f842be38700 1 librados: starting msgr at 2024-02-19T21:02:17.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.756+0000 7f842be38700 1 librados: starting objecter 2024-02-19T21:02:17.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.756+0000 7f842be38700 1 Processor -- start 2024-02-19T21:02:17.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.756+0000 7f842be38700 1 -- start start 2024-02-19T21:02:17.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.756+0000 7f842be38700 1 librados: setting wanted keys 2024-02-19T21:02:17.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.756+0000 7f842be38700 1 librados: calling monclient init 2024-02-19T21:02:17.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f842be38700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8424117310 con 0x7f84240fc870 2024-02-19T21:02:17.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f842be38700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f84241184f0 con 0x7f84241031b0 2024-02-19T21:02:17.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f842be38700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f84241196f0 con 0x7f84240ff670 2024-02-19T21:02:17.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f8429bd4700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f84240fc870 0x7f84240f9a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:41060/0 (socket says 172.21.15.163:41060) 2024-02-19T21:02:17.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f8429bd4700 1 -- 172.21.15.163:0/906228705 learned_addr learned my addr 172.21.15.163:0/906228705 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:17.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f84293d3700 1 --1- 172.21.15.163:0/906228705 >> v1:172.21.15.163:6790/0 conn(0x7f84240ff670 0x7f84240fa190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:39082/0 (socket says 172.21.15.163:39082) 2024-02-19T21:02:17.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 10 client.?.objecter ms_handle_connect 0x7f84240fc870 2024-02-19T21:02:17.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 10 client.?.objecter ms_handle_connect 0x7f84240ff670 2024-02-19T21:02:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 10 client.?.objecter ms_handle_connect 0x7f84241031b0 2024-02-19T21:02:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009292669 0 0) 0x7f8424117310 con 0x7f84240fc870 2024-02-19T21:02:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8410003030 con 0x7f84240fc870 2024-02-19T21:02:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870998421 0 0) 0x7f84241196f0 con 0x7f84240ff670 2024-02-19T21:02:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8424117310 con 0x7f84240ff670 2024-02-19T21:02:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173449388 0 0) 0x7f84241184f0 con 0x7f84241031b0 2024-02-19T21:02:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84241196f0 con 0x7f84241031b0 2024-02-19T21:02:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 577786043 0 0) 0x7f8410003030 con 0x7f84240fc870 2024-02-19T21:02:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84241184f0 con 0x7f84240fc870 2024-02-19T21:02:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8414001940 con 0x7f84240fc870 2024-02-19T21:02:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3290727634 0 0) 0x7f8424117310 con 0x7f84240ff670 2024-02-19T21:02:17.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8410003030 con 0x7f84240ff670 2024-02-19T21:02:17.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3017480264 0 0) 0x7f84241196f0 con 0x7f84241031b0 2024-02-19T21:02:17.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8424117310 con 0x7f84241031b0 2024-02-19T21:02:17.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8420000fe0 con 0x7f84240ff670 2024-02-19T21:02:17.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.757+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f841c0019a0 con 0x7f84241031b0 2024-02-19T21:02:17.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.758+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1669861081 0 0) 0x7f84241184f0 con 0x7f84240fc870 2024-02-19T21:02:17.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.758+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 >> v1:172.21.15.163:6790/0 conn(0x7f84240ff670 legacy=0x7f84240fa190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:17.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.758+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 >> v1:172.21.15.193:6789/0 conn(0x7f84241031b0 legacy=0x7f84241159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:17.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.758+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842411a8f0 con 0x7f84240fc870 2024-02-19T21:02:17.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.758+0000 7f842be38700 1 -- 172.21.15.163:0/906228705 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84241198e0 con 0x7f84240fc870 2024-02-19T21:02:17.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.758+0000 7f842be38700 10 client.14640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:17.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.758+0000 7f842be38700 1 -- 172.21.15.163:0/906228705 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8424119dd0 con 0x7f84240fc870 2024-02-19T21:02:17.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.758+0000 7f842be38700 1 librados: init done 2024-02-19T21:02:17.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.758+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8414001cd0 con 0x7f84240fc870 2024-02-19T21:02:17.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.758+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8414005ca0 con 0x7f84240fc870 2024-02-19T21:02:17.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.759+0000 7f842be38700 1 -- 172.21.15.163:0/906228705 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8408004f20 con 0x7f84240fc870 2024-02-19T21:02:17.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.759+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f8414002c60 con 0x7f84240fc870 2024-02-19T21:02:17.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.759+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(76..76 src has 1..76) v4 ==== 6172+0+0 (unknown 1515610916 0 0) 0x7f8414078650 con 0x7f84240fc870 2024-02-19T21:02:17.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.759+0000 7f841affd700 10 client.14640.objecter ms_dispatch 0x7f8424001550 osd_map(76..76 src has 1..76) v4 2024-02-19T21:02:17.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.759+0000 7f841affd700 3 client.14640.objecter handle_osd_map got epochs [76,76] > 0 2024-02-19T21:02:17.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.759+0000 7f841affd700 3 client.14640.objecter handle_osd_map decoding full epoch 76 2024-02-19T21:02:17.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.759+0000 7f841affd700 20 client.14640.objecter dump_active .. 0 homeless 2024-02-19T21:02:17.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.760+0000 7f841affd700 10 client.14640.objecter ms_handle_connect 0x7f8410068190 2024-02-19T21:02:17.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.764+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f841404c200 con 0x7f84240fc870 2024-02-19T21:02:17.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.928+0000 7f842be38700 1 -- 172.21.15.163:0/906228705 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84080051d0 con 0x7f84240fc870 2024-02-19T21:02:17.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.928+0000 7f841affd700 1 -- 172.21.15.163:0/906228705 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+19240 (unknown 2435502076 0 639989584) 0x7f841404a610 con 0x7f84240fc870 2024-02-19T21:02:17.919 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:02:17.919 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":76,"fsid":"2591809a-cf69-11ee-95bb-87774f69a715","created":"2024-02-19T20:55:37.794885+0000","modified":"2024-02-19T21:02:05.168060+0000","last_up_change":"2024-02-19T21:01:09.214134+0000","last_in_change":"2024-02-19T21:00:51.699017+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-19T20:59:09.425169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-19T21:01:14.658945+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-19T21:01:14.958800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-19T21:01:16.543816+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-19T21:01:18.521482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-19T21:01:20.586564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"4a14b406-7c21-40f3-8b12-8cf9df811088","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6801","nonce":1559418115}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6802","nonce":1559418115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6804","nonce":1559418115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6803","nonce":1559418115}]},"public_addr":"172.21.15.163:6801/1559418115","cluster_addr":"172.21.15.163:6802/1559418115","heartbeat_back_addr":"172.21.15.163:6804/1559418115","heartbeat_front_addr":"172.21.15.163:6803/1559418115","state":["exists","up"]},{"osd":1,"uuid":"e74a65fa-cb9c-42c3-9293-602cf7e9dc67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6805","nonce":3125047432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6806","nonce":3125047432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6808","nonce":3125047432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6807","nonce":3125047432}]},"public_addr":"172.21.15.163:6805/3125047432","cluster_addr":"172.21.15.163:6806/3125047432","heartbeat_back_addr":"172.21.15.163:6808/3125047432","heartbeat_front_addr":"172.21.15.163:6807/3125047432","state":["exists","up"]},{"osd":2,"uuid":"6980a625-cdc9-487e-8f8c-7997ba716a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6809","nonce":1563158487}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6810","nonce":1563158487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6812","nonce":1563158487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6811","nonce":1563158487}]},"public_addr":"172.21.15.163:6809/1563158487","cluster_addr":"172.21.15.163:6810/1563158487","heartbeat_back_addr":"172.21.15.163:6812/1563158487","heartbeat_front_addr":"172.21.15.163:6811/1563158487","state":["exists","up"]},{"osd":3,"uuid":"359e2ee0-9d1d-4074-8119-2d69bc2ca3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6813","nonce":2112279769}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6814","nonce":2112279769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6816","nonce":2112279769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6815","nonce":2112279769}]},"public_addr":"172.21.15.163:6813/2112279769","cluster_addr":"172.21.15.163:6814/2112279769","heartbeat_back_addr":"172.21.15.163:6816/2112279769","heartbeat_front_addr":"172.21.15.163:6815/2112279769","state":["exists","up"]},{"osd":4,"uuid":"2bf5ef05-292b-4693-85a9-97991f2091e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6800","nonce":3129716965}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6801","nonce":3129716965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6803","nonce":3129716965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6802","nonce":3129716965}]},"public_addr":"172.21.15.193:6800/3129716965","cluster_addr":"172.21.15.193:6801/3129716965","heartbeat_back_addr":"172.21.15.193:6803/3129716965","heartbeat_front_addr":"172.21.15.193:6802/3129716965","state":["exists","up"]},{"osd":5,"uuid":"59065f9b-9240-414b-9c91-e91cda4aed6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6804","nonce":805161991}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6805","nonce":805161991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6807","nonce":805161991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6806","nonce":805161991}]},"public_addr":"172.21.15.193:6804/805161991","cluster_addr":"172.21.15.193:6805/805161991","heartbeat_back_addr":"172.21.15.193:6807/805161991","heartbeat_front_addr":"172.21.15.193:6806/805161991","state":["exists","up"]},{"osd":6,"uuid":"302c95df-3842-4818-8afd-d79d8184684f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6808","nonce":787604615}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6809","nonce":787604615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6811","nonce":787604615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6810","nonce":787604615}]},"public_addr":"172.21.15.193:6808/787604615","cluster_addr":"172.21.15.193:6809/787604615","heartbeat_back_addr":"172.21.15.193:6811/787604615","heartbeat_front_addr":"172.21.15.193:6810/787604615","state":["exists","up"]},{"osd":7,"uuid":"9ebeff27-c2f8-4a66-923f-a0d7573ddd3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6812","nonce":157927432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6813","nonce":157927432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6815","nonce":157927432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6814","nonce":157927432}]},"public_addr":"172.21.15.193:6812/157927432","cluster_addr":"172.21.15.193:6813/157927432","heartbeat_back_addr":"172.21.15.193:6815/157927432","heartbeat_front_addr":"172.21.15.193:6814/157927432","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:02:03.243866+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:11.764868+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:36.293275+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:35.739499+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:59.638643+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:23.163793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:02:05.119016+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:01:07.350223+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":7,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:0/1427774891":"2024-02-20T21:01:38.896227+0000","172.21.15.163:0/3282102947":"2024-02-20T21:01:38.896227+0000","172.21.15.163:6800/2735092549":"2024-02-20T21:01:38.896227+0000","172.21.15.163:0/1235588846":"2024-02-20T20:56:29.603030+0000","172.21.15.163:6800/2731225015":"2024-02-20T20:56:29.603030+0000","172.21.15.163:0/506491651":"2024-02-20T20:56:02.185409+0000","172.21.15.163:0/168117826":"2024-02-20T21:01:38.896227+0000","172.21.15.163:0/1458009465":"2024-02-20T20:56:29.603030+0000","172.21.15.163:0/541631472":"2024-02-20T20:56:02.185409+0000","172.21.15.163:6800/2248999541":"2024-02-20T20:56:02.185409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T21:02:17.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.930+0000 7f842be38700 10 librados: watch_flush enter 2024-02-19T21:02:17.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.930+0000 7f842be38700 10 librados: watch_flush exit 2024-02-19T21:02:17.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.930+0000 7f842be38700 20 client.14640.objecter shutdown clearing up homeless session... 2024-02-19T21:02:17.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.930+0000 7f842be38700 10 client.14640.objecter successfully canceled tick 2024-02-19T21:02:17.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.931+0000 7f842be38700 1 -- 172.21.15.163:0/906228705 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8410068190 legacy=0x7f841006a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:17.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.931+0000 7f842be38700 1 -- 172.21.15.163:0/906228705 >> v1:172.21.15.163:6789/0 conn(0x7f84240fc870 legacy=0x7f84240f9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:17.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.931+0000 7f842be38700 1 -- 172.21.15.163:0/906228705 shutdown_connections 2024-02-19T21:02:17.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.931+0000 7f842be38700 1 -- 172.21.15.163:0/906228705 >> 172.21.15.163:0/906228705 conn(0x7f84240f2470 msgr2=0x7f8424061200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:17.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.931+0000 7f842be38700 1 -- 172.21.15.163:0/906228705 shutdown_connections 2024-02-19T21:02:17.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.931+0000 7f842be38700 1 -- 172.21.15.163:0/906228705 wait complete. 2024-02-19T21:02:17.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:17.931+0000 7f842be38700 1 librados: shutdown 2024-02-19T21:02:18.061 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:02:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:02:18.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:16.919320+0000 mgr.y (mgr.14550) 38 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-02-19T21:02:18.237 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:17.028433+0000 osd.4 (osd.4) 7 : cluster [DBG] 2.f scrub starts 2024-02-19T21:02:18.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:17.029607+0000 osd.4 (osd.4) 8 : cluster [DBG] 2.f scrub ok 2024-02-19T21:02:18.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:17.929512+0000 mon.a (mon.0) 681 : audit [DBG] from='client.? v1:172.21.15.163:0/906228705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T21:02:18.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:16.919320+0000 mgr.y (mgr.14550) 38 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-02-19T21:02:18.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:17.028433+0000 osd.4 (osd.4) 7 : cluster [DBG] 2.f scrub starts 2024-02-19T21:02:18.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:17.029607+0000 osd.4 (osd.4) 8 : cluster [DBG] 2.f scrub ok 2024-02-19T21:02:18.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:17.929512+0000 mon.a (mon.0) 681 : audit [DBG] from='client.? v1:172.21.15.163:0/906228705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T21:02:18.555 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-19T21:02:18.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:16.919320+0000 mgr.y (mgr.14550) 38 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-02-19T21:02:18.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:17.028433+0000 osd.4 (osd.4) 7 : cluster [DBG] 2.f scrub starts 2024-02-19T21:02:18.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:17.029607+0000 osd.4 (osd.4) 8 : cluster [DBG] 2.f scrub ok 2024-02-19T21:02:18.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:17.929512+0000 mon.a (mon.0) 681 : audit [DBG] from='client.? v1:172.21.15.163:0/906228705' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T21:02:18.825 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:20.311 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:02:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:02:20.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:18.593183+0000 osd.1 (osd.1) 7 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:02:20.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:18.594040+0000 osd.1 (osd.1) 8 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:02:20.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:18.706450+0000 osd.0 (osd.0) 9 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:02:20.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:18.707915+0000 osd.0 (osd.0) 10 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:02:20.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:18.919645+0000 mgr.y (mgr.14550) 39 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:02:20.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:18.967637+0000 osd.1 (osd.1) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:20.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:18.967697+0000 osd.1 (osd.1) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:20.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:19.041440+0000 osd.4 (osd.4) 9 : cluster [DBG] 4.e scrub starts 2024-02-19T21:02:20.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:19.042645+0000 osd.4 (osd.4) 10 : cluster [DBG] 4.e scrub ok 2024-02-19T21:02:20.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:19.111443+0000 mon.a (mon.0) 682 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-19T21:02:20.350 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:18.593183+0000 osd.1 (osd.1) 7 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:02:20.350 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:18.594040+0000 osd.1 (osd.1) 8 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:02:20.350 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:18.706450+0000 osd.0 (osd.0) 9 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:02:20.351 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:18.707915+0000 osd.0 (osd.0) 10 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:02:20.351 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:18.919645+0000 mgr.y (mgr.14550) 39 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:02:20.351 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:18.967637+0000 osd.1 (osd.1) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:20.351 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:18.967697+0000 osd.1 (osd.1) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:20.351 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:19.041440+0000 osd.4 (osd.4) 9 : cluster [DBG] 4.e scrub starts 2024-02-19T21:02:20.351 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:19.042645+0000 osd.4 (osd.4) 10 : cluster [DBG] 4.e scrub ok 2024-02-19T21:02:20.351 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:19.111443+0000 mon.a (mon.0) 682 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-19T21:02:20.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:18.593183+0000 osd.1 (osd.1) 7 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:02:20.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:18.594040+0000 osd.1 (osd.1) 8 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:02:20.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:18.706450+0000 osd.0 (osd.0) 9 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:02:20.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:18.707915+0000 osd.0 (osd.0) 10 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:02:20.351 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:18.919645+0000 mgr.y (mgr.14550) 39 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:02:20.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:18.967637+0000 osd.1 (osd.1) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:20.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:18.967697+0000 osd.1 (osd.1) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:20.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:19.041440+0000 osd.4 (osd.4) 9 : cluster [DBG] 4.e scrub starts 2024-02-19T21:02:20.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:19.042645+0000 osd.4 (osd.4) 10 : cluster [DBG] 4.e scrub ok 2024-02-19T21:02:20.352 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:19.111443+0000 mon.a (mon.0) 682 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-02-19T21:02:20.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.587+0000 7f2c278a0700 1 -- 172.21.15.163:0/3787837415 >> v1:172.21.15.163:6790/0 conn(0x7f2c201042c0 legacy=0x7f2c20106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:20.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.587+0000 7f2c278a0700 1 -- 172.21.15.163:0/3787837415 shutdown_connections 2024-02-19T21:02:20.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.587+0000 7f2c278a0700 1 -- 172.21.15.163:0/3787837415 >> 172.21.15.163:0/3787837415 conn(0x7f2c200f35a0 msgr2=0x7f2c200f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:20.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.588+0000 7f2c278a0700 1 -- 172.21.15.163:0/3787837415 shutdown_connections 2024-02-19T21:02:20.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.588+0000 7f2c278a0700 1 -- 172.21.15.163:0/3787837415 wait complete. 2024-02-19T21:02:20.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.588+0000 7f2c278a0700 1 librados: starting msgr at 2024-02-19T21:02:20.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.588+0000 7f2c278a0700 1 librados: starting objecter 2024-02-19T21:02:20.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.588+0000 7f2c278a0700 1 Processor -- start 2024-02-19T21:02:20.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.588+0000 7f2c278a0700 1 -- start start 2024-02-19T21:02:20.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.588+0000 7f2c278a0700 1 librados: setting wanted keys 2024-02-19T21:02:20.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.588+0000 7f2c278a0700 1 librados: calling monclient init 2024-02-19T21:02:20.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.588+0000 7f2c278a0700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2c200fcf30 con 0x7f2c200fd980 2024-02-19T21:02:20.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.588+0000 7f2c278a0700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2c200fd100 con 0x7f2c20100780 2024-02-19T21:02:20.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.588+0000 7f2c278a0700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2c200fd2f0 con 0x7f2c201042c0 2024-02-19T21:02:20.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.588+0000 7f2c25e3d700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f2c201042c0 0x7f2c20118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:39098/0 (socket says 172.21.15.163:39098) 2024-02-19T21:02:20.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.588+0000 7f2c25e3d700 1 -- 172.21.15.163:0/2922844427 learned_addr learned my addr 172.21.15.163:0/2922844427 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:20.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 10 client.?.objecter ms_handle_connect 0x7f2c201042c0 2024-02-19T21:02:20.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:20.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 10 client.?.objecter ms_handle_connect 0x7f2c20100780 2024-02-19T21:02:20.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:20.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 10 client.?.objecter ms_handle_connect 0x7f2c200fd980 2024-02-19T21:02:20.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:20.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608561260 0 0) 0x7f2c200fd2f0 con 0x7f2c201042c0 2024-02-19T21:02:20.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c0c0033f0 con 0x7f2c201042c0 2024-02-19T21:02:20.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197770835 0 0) 0x7f2c200fd100 con 0x7f2c20100780 2024-02-19T21:02:20.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c200fd2f0 con 0x7f2c20100780 2024-02-19T21:02:20.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402951029 0 0) 0x7f2c200fcf30 con 0x7f2c200fd980 2024-02-19T21:02:20.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c200fd100 con 0x7f2c200fd980 2024-02-19T21:02:20.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3359042264 0 0) 0x7f2c0c0033f0 con 0x7f2c201042c0 2024-02-19T21:02:20.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c200fcf30 con 0x7f2c201042c0 2024-02-19T21:02:20.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2c18001360 con 0x7f2c201042c0 2024-02-19T21:02:20.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4101535777 0 0) 0x7f2c200fd100 con 0x7f2c200fd980 2024-02-19T21:02:20.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c0c0033f0 con 0x7f2c200fd980 2024-02-19T21:02:20.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4279577852 0 0) 0x7f2c200fd2f0 con 0x7f2c20100780 2024-02-19T21:02:20.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c200fd100 con 0x7f2c20100780 2024-02-19T21:02:20.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2c1c001830 con 0x7f2c200fd980 2024-02-19T21:02:20.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2c10001840 con 0x7f2c20100780 2024-02-19T21:02:20.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2940767697 0 0) 0x7f2c200fcf30 con 0x7f2c201042c0 2024-02-19T21:02:20.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 >> v1:172.21.15.193:6789/0 conn(0x7f2c20100780 legacy=0x7f2c20114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:20.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 >> v1:172.21.15.163:6789/0 conn(0x7f2c200fd980 legacy=0x7f2c200fc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:20.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2011cb90 con 0x7f2c201042c0 2024-02-19T21:02:20.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c278a0700 1 -- 172.21.15.163:0/2922844427 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c20119bc0 con 0x7f2c201042c0 2024-02-19T21:02:20.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c278a0700 10 client.34571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c278a0700 1 -- 172.21.15.163:0/2922844427 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2011a030 con 0x7f2c201042c0 2024-02-19T21:02:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c278a0700 1 librados: init done 2024-02-19T21:02:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f2c180026a0 con 0x7f2c201042c0 2024-02-19T21:02:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.589+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2c18003aa0 con 0x7f2c201042c0 2024-02-19T21:02:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.590+0000 7f2c278a0700 10 client.34571.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:02:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.590+0000 7f2c278a0700 15 client.34571.objecter _session_command_op_assign -1 1 2024-02-19T21:02:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.590+0000 7f2c278a0700 20 client.34571.objecter _get_session osd=-1 returning homeless 2024-02-19T21:02:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.590+0000 7f2c278a0700 10 client.34571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.590+0000 7f2c278a0700 1 -- 172.21.15.163:0/2922844427 --> v1:172.21.15.163:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2c040028b0 con 0x7f2c201042c0 2024-02-19T21:02:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.590+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f2c18003ce0 con 0x7f2c201042c0 2024-02-19T21:02:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(77..77 src has 1..77) v4 ==== 6172+0+0 (unknown 1298313868 0 0) 0x7f2c18078cc0 con 0x7f2c201042c0 2024-02-19T21:02:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 10 client.34571.objecter ms_dispatch 0x7f2c20001550 osd_map(77..77 src has 1..77) v4 2024-02-19T21:02:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 3 client.34571.objecter handle_osd_map got epochs [77,77] > 0 2024-02-19T21:02:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 3 client.34571.objecter handle_osd_map decoding full epoch 77 2024-02-19T21:02:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 10 client.34571.objecter checking command 1 2024-02-19T21:02:20.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 20 client.34571.objecter _get_session s=0x7f2c18001360 osd=0 3 2024-02-19T21:02:20.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 20 client.34571.objecter put_session s=0x7f2c18001360 osd=0 3 2024-02-19T21:02:20.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 15 client.34571.objecter _session_command_op_remove -1 1 2024-02-19T21:02:20.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 20 client.34571.objecter _get_session s=0x7f2c18001360 osd=0 3 2024-02-19T21:02:20.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 20 client.34571.objecter get_session s=0x7f2c18001360 osd=0 3 2024-02-19T21:02:20.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 15 client.34571.objecter _session_command_op_assign 0 1 2024-02-19T21:02:20.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 20 client.34571.objecter put_session s=0x7f2c18001360 osd=0 4 2024-02-19T21:02:20.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 10 client.34571.objecter _send_command 1 2024-02-19T21:02:20.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c0c072170 con 0x7f2c0c06e460 2024-02-19T21:02:20.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 20 client.34571.objecter dump_active .. 0 homeless 2024-02-19T21:02:20.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_get_version_reply(handle=1 version=77) v2 ==== 24+0+0 (unknown 369529703 0 0) 0x7f2c180790d0 con 0x7f2c201042c0 2024-02-19T21:02:20.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 10 client.34571.objecter ms_handle_connect 0x7f2c0c063c80 2024-02-19T21:02:20.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.591+0000 7f2c167fc700 10 client.34571.objecter ms_handle_connect 0x7f2c0c06e460 2024-02-19T21:02:20.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.592+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2c0c072170 con 0x7f2c0c06e460 2024-02-19T21:02:20.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.592+0000 7f2c167fc700 10 client.34571.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:02:20.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.592+0000 7f2c167fc700 10 client.34571.objecter _finish_command 1 = system:0 2024-02-19T21:02:20.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.592+0000 7f2c167fc700 20 client.34571.objecter put_session s=0x7f2c18001360 osd=0 4 2024-02-19T21:02:20.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.592+0000 7f2c167fc700 15 client.34571.objecter _session_command_op_remove 0 1 2024-02-19T21:02:20.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 10 client.34571.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:02:20.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 15 client.34571.objecter _session_command_op_assign -1 2 2024-02-19T21:02:20.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 20 client.34571.objecter _get_session s=0x7f2c18001360 osd=0 3 2024-02-19T21:02:20.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 20 client.34571.objecter put_session s=0x7f2c18001360 osd=0 3 2024-02-19T21:02:20.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 20 client.34571.objecter _get_session s=0x7f2c18001360 osd=0 3 2024-02-19T21:02:20.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 15 client.34571.objecter _session_command_op_remove -1 2 2024-02-19T21:02:20.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 20 client.34571.objecter get_session s=0x7f2c18001360 osd=0 3 2024-02-19T21:02:20.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 15 client.34571.objecter _session_command_op_assign 0 2 2024-02-19T21:02:20.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 20 client.34571.objecter put_session s=0x7f2c18001360 osd=0 4 2024-02-19T21:02:20.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 10 client.34571.objecter _send_command 2 2024-02-19T21:02:20.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 1 -- 172.21.15.163:0/2922844427 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2c04001140 con 0x7f2c0c06e460 2024-02-19T21:02:20.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c167fc700 1 -- 172.21.15.163:0/2922844427 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1002017476) 0x7f2c04001140 con 0x7f2c0c06e460 2024-02-19T21:02:20.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c167fc700 10 client.34571.objecter ms_dispatch 0x7f2c20001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:02:20.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c167fc700 10 client.34571.objecter _finish_command 2 = system:0 2024-02-19T21:02:20.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c167fc700 20 client.34571.objecter put_session s=0x7f2c18001360 osd=0 4 2024-02-19T21:02:20.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c167fc700 15 client.34571.objecter _session_command_op_remove 0 2 2024-02-19T21:02:20.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 10 librados: watch_flush enter 2024-02-19T21:02:20.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 10 librados: watch_flush exit 2024-02-19T21:02:20.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 10 client.34571.objecter close_session for osd.0 2024-02-19T21:02:20.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 1 -- 172.21.15.163:0/2922844427 >> v1:172.21.15.163:6801/1559418115 conn(0x7f2c0c06e460 legacy=0x7f2c0c070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:20.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 20 client.34571.objecter put_session s=0x7f2c18001360 osd=0 1 2024-02-19T21:02:20.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 20 client.34571.objecter shutdown clearing up homeless session... 2024-02-19T21:02:20.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 10 client.34571.objecter successfully canceled tick 2024-02-19T21:02:20.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 1 -- 172.21.15.163:0/2922844427 >> v1:172.21.15.163:6800/1552514681 conn(0x7f2c0c063c80 legacy=0x7f2c0c066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:20.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 1 -- 172.21.15.163:0/2922844427 >> v1:172.21.15.163:6790/0 conn(0x7f2c201042c0 legacy=0x7f2c20118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:20.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c25e3d700 1 -- 172.21.15.163:0/2922844427 reap_dead start 2024-02-19T21:02:20.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 1 -- 172.21.15.163:0/2922844427 shutdown_connections 2024-02-19T21:02:20.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 1 -- 172.21.15.163:0/2922844427 >> 172.21.15.163:0/2922844427 conn(0x7f2c200f35a0 msgr2=0x7f2c20107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:20.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 1 -- 172.21.15.163:0/2922844427 shutdown_connections 2024-02-19T21:02:20.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.606+0000 7f2c278a0700 1 -- 172.21.15.163:0/2922844427 wait complete. 2024-02-19T21:02:20.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:20.607+0000 7f2c278a0700 1 librados: shutdown 2024-02-19T21:02:21.203 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:19.950489+0000 mgr.y (mgr.14550) 40 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:21.203 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:19.950489+0000 mgr.y (mgr.14550) 40 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:21.222 INFO:teuthology.orchestra.run.smithi163.stdout:34359738418 2024-02-19T21:02:21.222 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-19T21:02:21.491 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:21.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:19.950489+0000 mgr.y (mgr.14550) 40 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:22.112 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:20.543621+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:22.112 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:20.543687+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:22.112 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:20.572231+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:02:22.112 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:20.575128+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:02:22.112 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:20.920523+0000 mgr.y (mgr.14550) 41 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:22.112 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:20.969346+0000 osd.3 (osd.3) 7 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:02:22.113 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:20.970211+0000 osd.3 (osd.3) 8 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:02:22.113 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:21.122399+0000 mon.a (mon.0) 683 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-19T21:02:22.113 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:21.211685+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.c scrub starts 2024-02-19T21:02:22.113 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:21.212800+0000 osd.2 (osd.2) 8 : cluster [DBG] 2.c scrub ok 2024-02-19T21:02:22.113 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:21.921452+0000 osd.3 (osd.3) 9 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:02:22.113 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:21.922966+0000 osd.3 (osd.3) 10 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:02:22.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:20.543621+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:22.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:20.543687+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:22.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:20.572231+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:02:22.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:20.575128+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:02:22.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:20.920523+0000 mgr.y (mgr.14550) 41 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:22.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:20.969346+0000 osd.3 (osd.3) 7 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:02:22.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:20.970211+0000 osd.3 (osd.3) 8 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:02:22.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:21.122399+0000 mon.a (mon.0) 683 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-19T21:02:22.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:21.211685+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.c scrub starts 2024-02-19T21:02:22.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:21.212800+0000 osd.2 (osd.2) 8 : cluster [DBG] 2.c scrub ok 2024-02-19T21:02:22.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:21.921452+0000 osd.3 (osd.3) 9 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:02:22.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:21.922966+0000 osd.3 (osd.3) 10 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:02:22.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:20.543621+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:22.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:20.543687+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:22.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:20.572231+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:02:22.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:20.575128+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:02:22.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:20.920523+0000 mgr.y (mgr.14550) 41 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:22.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:20.969346+0000 osd.3 (osd.3) 7 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:02:22.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:20.970211+0000 osd.3 (osd.3) 8 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:02:22.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:21.122399+0000 mon.a (mon.0) 683 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-02-19T21:02:22.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:21.211685+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.c scrub starts 2024-02-19T21:02:22.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:21.212800+0000 osd.2 (osd.2) 8 : cluster [DBG] 2.c scrub ok 2024-02-19T21:02:22.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:21.921452+0000 osd.3 (osd.3) 9 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:02:22.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:21.922966+0000 osd.3 (osd.3) 10 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:02:23.188 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:21.496063+0000 osd.1 (osd.1) 11 : cluster [DBG] 5.c scrub starts 2024-02-19T21:02:23.188 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:21.497824+0000 osd.1 (osd.1) 12 : cluster [DBG] 5.c scrub ok 2024-02-19T21:02:23.188 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:21.656148+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:02:23.188 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:21.657329+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:02:23.188 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:21.692904+0000 osd.0 (osd.0) 11 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:02:23.188 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:21.693672+0000 osd.0 (osd.0) 12 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:02:23.188 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:22.215356+0000 osd.2 (osd.2) 9 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:02:23.189 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:22.217216+0000 osd.2 (osd.2) 10 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:02:23.189 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:21.496063+0000 osd.1 (osd.1) 11 : cluster [DBG] 5.c scrub starts 2024-02-19T21:02:23.189 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:21.497824+0000 osd.1 (osd.1) 12 : cluster [DBG] 5.c scrub ok 2024-02-19T21:02:23.189 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:21.656148+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:02:23.189 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:21.657329+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:02:23.189 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:21.692904+0000 osd.0 (osd.0) 11 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:02:23.189 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:21.693672+0000 osd.0 (osd.0) 12 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:02:23.189 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:22.215356+0000 osd.2 (osd.2) 9 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:02:23.189 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:22.217216+0000 osd.2 (osd.2) 10 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:02:23.225 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:21.496063+0000 osd.1 (osd.1) 11 : cluster [DBG] 5.c scrub starts 2024-02-19T21:02:23.225 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:21.497824+0000 osd.1 (osd.1) 12 : cluster [DBG] 5.c scrub ok 2024-02-19T21:02:23.225 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:21.656148+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:02:23.225 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:21.657329+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:02:23.225 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:21.692904+0000 osd.0 (osd.0) 11 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:02:23.226 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:21.693672+0000 osd.0 (osd.0) 12 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:02:23.226 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:22.215356+0000 osd.2 (osd.2) 9 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:02:23.226 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:22.217216+0000 osd.2 (osd.2) 10 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:02:23.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.399+0000 7f8491f2f700 1 -- 172.21.15.163:0/1395446470 >> v1:172.21.15.163:6789/0 conn(0x7f848c1042c0 legacy=0x7f848c106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:23.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.399+0000 7f8491f2f700 1 -- 172.21.15.163:0/1395446470 shutdown_connections 2024-02-19T21:02:23.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.399+0000 7f8491f2f700 1 -- 172.21.15.163:0/1395446470 >> 172.21.15.163:0/1395446470 conn(0x7f848c0f35a0 msgr2=0x7f848c0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:23.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.399+0000 7f8491f2f700 1 -- 172.21.15.163:0/1395446470 shutdown_connections 2024-02-19T21:02:23.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.399+0000 7f8491f2f700 1 -- 172.21.15.163:0/1395446470 wait complete. 2024-02-19T21:02:23.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.400+0000 7f8491f2f700 1 librados: starting msgr at 2024-02-19T21:02:23.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.400+0000 7f8491f2f700 1 librados: starting objecter 2024-02-19T21:02:23.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.400+0000 7f8491f2f700 1 Processor -- start 2024-02-19T21:02:23.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8491f2f700 1 -- start start 2024-02-19T21:02:23.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8491f2f700 1 librados: setting wanted keys 2024-02-19T21:02:23.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8491f2f700 1 librados: calling monclient init 2024-02-19T21:02:23.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8491f2f700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f848c0fcf30 con 0x7f848c0fd980 2024-02-19T21:02:23.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8491f2f700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f848c0fd0c0 con 0x7f848c1042c0 2024-02-19T21:02:23.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8491f2f700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f848c0fd2b0 con 0x7f848c100780 2024-02-19T21:02:23.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f848affd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f848c100780 0x7f848c114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:39116/0 (socket says 172.21.15.163:39116) 2024-02-19T21:02:23.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f848b7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f848c0fd980 0x7f848c0fc790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:41080/0 (socket says 172.21.15.163:41080) 2024-02-19T21:02:23.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f848affd700 1 -- 172.21.15.163:0/2702851333 learned_addr learned my addr 172.21.15.163:0/2702851333 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:23.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 10 client.?.objecter ms_handle_connect 0x7f848c100780 2024-02-19T21:02:23.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:23.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 10 client.?.objecter ms_handle_connect 0x7f848c0fd980 2024-02-19T21:02:23.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:23.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 10 client.?.objecter ms_handle_connect 0x7f848c1042c0 2024-02-19T21:02:23.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:23.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842756150 0 0) 0x7f848c0fcf30 con 0x7f848c0fd980 2024-02-19T21:02:23.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8474003800 con 0x7f848c0fd980 2024-02-19T21:02:23.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009224212 0 0) 0x7f848c0fd2b0 con 0x7f848c100780 2024-02-19T21:02:23.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f848c0fcf30 con 0x7f848c100780 2024-02-19T21:02:23.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084708626 0 0) 0x7f848c0fd0c0 con 0x7f848c1042c0 2024-02-19T21:02:23.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f848c0fd2b0 con 0x7f848c1042c0 2024-02-19T21:02:23.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 271337347 0 0) 0x7f8474003800 con 0x7f848c0fd980 2024-02-19T21:02:23.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f848c0fd0c0 con 0x7f848c0fd980 2024-02-19T21:02:23.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f847c0019f0 con 0x7f848c0fd980 2024-02-19T21:02:23.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 658497426 0 0) 0x7f848c0fcf30 con 0x7f848c100780 2024-02-19T21:02:23.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8474003800 con 0x7f848c100780 2024-02-19T21:02:23.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2116907667 0 0) 0x7f848c0fd2b0 con 0x7f848c1042c0 2024-02-19T21:02:23.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f848c0fcf30 con 0x7f848c1042c0 2024-02-19T21:02:23.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8480001840 con 0x7f848c100780 2024-02-19T21:02:23.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.401+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8484002fe0 con 0x7f848c1042c0 2024-02-19T21:02:23.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.402+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3254269238 0 0) 0x7f848c0fd0c0 con 0x7f848c0fd980 2024-02-19T21:02:23.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.402+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 >> v1:172.21.15.163:6790/0 conn(0x7f848c100780 legacy=0x7f848c114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:23.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.402+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 >> v1:172.21.15.193:6789/0 conn(0x7f848c1042c0 legacy=0x7f848c118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:23.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.402+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848c11cb90 con 0x7f848c0fd980 2024-02-19T21:02:23.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.402+0000 7f8491f2f700 1 -- 172.21.15.163:0/2702851333 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f848c119bc0 con 0x7f848c0fd980 2024-02-19T21:02:23.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.402+0000 7f8491f2f700 10 client.14652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:23.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.402+0000 7f8491f2f700 1 -- 172.21.15.163:0/2702851333 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f848c11a0b0 con 0x7f848c0fd980 2024-02-19T21:02:23.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.402+0000 7f8491f2f700 1 librados: init done 2024-02-19T21:02:23.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.402+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f847c002360 con 0x7f848c0fd980 2024-02-19T21:02:23.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.402+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f847c0059e0 con 0x7f848c0fd980 2024-02-19T21:02:23.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.402+0000 7f8491f2f700 10 client.14652.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:02:23.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.402+0000 7f8491f2f700 15 client.14652.objecter _session_command_op_assign -1 1 2024-02-19T21:02:23.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.402+0000 7f8491f2f700 20 client.14652.objecter _get_session osd=-1 returning homeless 2024-02-19T21:02:23.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.402+0000 7f8491f2f700 10 client.14652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:23.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.402+0000 7f8491f2f700 1 -- 172.21.15.163:0/2702851333 --> v1:172.21.15.163:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f84780028b0 con 0x7f848c0fd980 2024-02-19T21:02:23.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.403+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f847c0024d0 con 0x7f848c0fd980 2024-02-19T21:02:23.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.403+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(78..78 src has 1..78) v4 ==== 6172+0+0 (unknown 1950335092 0 0) 0x7f847c078430 con 0x7f848c0fd980 2024-02-19T21:02:23.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.403+0000 7f8488ff9700 10 client.14652.objecter ms_dispatch 0x7f848c001550 osd_map(78..78 src has 1..78) v4 2024-02-19T21:02:23.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.403+0000 7f8488ff9700 3 client.14652.objecter handle_osd_map got epochs [78,78] > 0 2024-02-19T21:02:23.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.403+0000 7f8488ff9700 3 client.14652.objecter handle_osd_map decoding full epoch 78 2024-02-19T21:02:23.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.404+0000 7f8488ff9700 10 client.14652.objecter checking command 1 2024-02-19T21:02:23.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.404+0000 7f8488ff9700 20 client.14652.objecter _get_session s=0x7f847406e0e0 osd=1 3 2024-02-19T21:02:23.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.404+0000 7f8488ff9700 20 client.14652.objecter put_session s=0x7f847406e0e0 osd=1 3 2024-02-19T21:02:23.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.404+0000 7f8488ff9700 15 client.14652.objecter _session_command_op_remove -1 1 2024-02-19T21:02:23.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.404+0000 7f8488ff9700 20 client.14652.objecter _get_session s=0x7f847406e0e0 osd=1 3 2024-02-19T21:02:23.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.404+0000 7f8488ff9700 20 client.14652.objecter get_session s=0x7f847406e0e0 osd=1 3 2024-02-19T21:02:23.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.404+0000 7f8488ff9700 15 client.14652.objecter _session_command_op_assign 1 1 2024-02-19T21:02:23.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.404+0000 7f8488ff9700 20 client.14652.objecter put_session s=0x7f847406e0e0 osd=1 4 2024-02-19T21:02:23.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.404+0000 7f8488ff9700 10 client.14652.objecter _send_command 1 2024-02-19T21:02:23.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.404+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 --> v1:172.21.15.163:6805/3125047432 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f84740724a0 con 0x7f847406e790 2024-02-19T21:02:23.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.404+0000 7f8488ff9700 20 client.14652.objecter dump_active .. 0 homeless 2024-02-19T21:02:23.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.404+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (unknown 100594580 0 0) 0x7f847c078810 con 0x7f848c0fd980 2024-02-19T21:02:23.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.404+0000 7f8488ff9700 10 client.14652.objecter ms_handle_connect 0x7f8474063dc0 2024-02-19T21:02:23.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.404+0000 7f8488ff9700 10 client.14652.objecter ms_handle_connect 0x7f847406e790 2024-02-19T21:02:23.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.405+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f84740724a0 con 0x7f847406e790 2024-02-19T21:02:23.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.405+0000 7f8488ff9700 10 client.14652.objecter ms_dispatch 0x7f848c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:02:23.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.405+0000 7f8488ff9700 10 client.14652.objecter _finish_command 1 = system:0 2024-02-19T21:02:23.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.405+0000 7f8488ff9700 20 client.14652.objecter put_session s=0x7f847406e0e0 osd=1 4 2024-02-19T21:02:23.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.405+0000 7f8488ff9700 15 client.14652.objecter _session_command_op_remove 1 1 2024-02-19T21:02:23.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.418+0000 7f8491f2f700 10 client.14652.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:02:23.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.418+0000 7f8491f2f700 15 client.14652.objecter _session_command_op_assign -1 2 2024-02-19T21:02:23.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.418+0000 7f8491f2f700 20 client.14652.objecter _get_session s=0x7f847406e0e0 osd=1 3 2024-02-19T21:02:23.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.418+0000 7f8491f2f700 20 client.14652.objecter put_session s=0x7f847406e0e0 osd=1 3 2024-02-19T21:02:23.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.418+0000 7f8491f2f700 20 client.14652.objecter _get_session s=0x7f847406e0e0 osd=1 3 2024-02-19T21:02:23.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.418+0000 7f8491f2f700 15 client.14652.objecter _session_command_op_remove -1 2 2024-02-19T21:02:23.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.418+0000 7f8491f2f700 20 client.14652.objecter get_session s=0x7f847406e0e0 osd=1 3 2024-02-19T21:02:23.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.418+0000 7f8491f2f700 15 client.14652.objecter _session_command_op_assign 1 2 2024-02-19T21:02:23.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.418+0000 7f8491f2f700 20 client.14652.objecter put_session s=0x7f847406e0e0 osd=1 4 2024-02-19T21:02:23.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.418+0000 7f8491f2f700 10 client.14652.objecter _send_command 2 2024-02-19T21:02:23.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.418+0000 7f8491f2f700 1 -- 172.21.15.163:0/2702851333 --> v1:172.21.15.163:6805/3125047432 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8478003be0 con 0x7f847406e790 2024-02-19T21:02:23.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8488ff9700 1 -- 172.21.15.163:0/2702851333 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 4200525082) 0x7f8478003be0 con 0x7f847406e790 2024-02-19T21:02:23.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8488ff9700 10 client.14652.objecter ms_dispatch 0x7f848c001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:02:23.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8488ff9700 10 client.14652.objecter _finish_command 2 = system:0 2024-02-19T21:02:23.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8488ff9700 20 client.14652.objecter put_session s=0x7f847406e0e0 osd=1 4 2024-02-19T21:02:23.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8488ff9700 15 client.14652.objecter _session_command_op_remove 1 2 2024-02-19T21:02:23.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8491f2f700 10 librados: watch_flush enter 2024-02-19T21:02:23.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8491f2f700 10 librados: watch_flush exit 2024-02-19T21:02:23.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8491f2f700 10 client.14652.objecter close_session for osd.1 2024-02-19T21:02:23.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8491f2f700 1 -- 172.21.15.163:0/2702851333 >> v1:172.21.15.163:6805/3125047432 conn(0x7f847406e790 legacy=0x7f8474070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:23.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8491f2f700 20 client.14652.objecter put_session s=0x7f847406e0e0 osd=1 1 2024-02-19T21:02:23.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8491f2f700 20 client.14652.objecter shutdown clearing up homeless session... 2024-02-19T21:02:23.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8491f2f700 10 client.14652.objecter successfully canceled tick 2024-02-19T21:02:23.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8491f2f700 1 -- 172.21.15.163:0/2702851333 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8474063dc0 legacy=0x7f8474066280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:23.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8491f2f700 1 -- 172.21.15.163:0/2702851333 >> v1:172.21.15.163:6789/0 conn(0x7f848c0fd980 legacy=0x7f848c0fc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:23.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f848bfff700 1 -- 172.21.15.163:0/2702851333 reap_dead start 2024-02-19T21:02:23.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8491f2f700 1 -- 172.21.15.163:0/2702851333 shutdown_connections 2024-02-19T21:02:23.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8491f2f700 1 -- 172.21.15.163:0/2702851333 >> 172.21.15.163:0/2702851333 conn(0x7f848c0f35a0 msgr2=0x7f848c107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:23.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8491f2f700 1 -- 172.21.15.163:0/2702851333 shutdown_connections 2024-02-19T21:02:23.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8491f2f700 1 -- 172.21.15.163:0/2702851333 wait complete. 2024-02-19T21:02:23.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:23.419+0000 7f8491f2f700 1 librados: shutdown 2024-02-19T21:02:23.738 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:02:23] "GET /metrics HTTP/1.1" 200 172145 "" "Prometheus/2.33.4" 2024-02-19T21:02:24.188 INFO:teuthology.orchestra.run.smithi163.stdout:55834574892 2024-02-19T21:02:24.188 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-19T21:02:24.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:22.515323+0000 osd.1 (osd.1) 13 : cluster [DBG] 2.d scrub starts 2024-02-19T21:02:24.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:22.516129+0000 osd.1 (osd.1) 14 : cluster [DBG] 2.d scrub ok 2024-02-19T21:02:24.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:22.920932+0000 mgr.y (mgr.14550) 42 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:02:24.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:22.937354+0000 osd.3 (osd.3) 11 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:02:24.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:22.939186+0000 osd.3 (osd.3) 12 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:02:24.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:23.099049+0000 osd.4 (osd.4) 11 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:02:24.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:23.099928+0000 osd.4 (osd.4) 12 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:02:24.457 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:24.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:22.515323+0000 osd.1 (osd.1) 13 : cluster [DBG] 2.d scrub starts 2024-02-19T21:02:24.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:22.516129+0000 osd.1 (osd.1) 14 : cluster [DBG] 2.d scrub ok 2024-02-19T21:02:24.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:22.920932+0000 mgr.y (mgr.14550) 42 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:02:24.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:22.937354+0000 osd.3 (osd.3) 11 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:02:24.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:22.939186+0000 osd.3 (osd.3) 12 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:02:24.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:23.099049+0000 osd.4 (osd.4) 11 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:02:24.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:23.099928+0000 osd.4 (osd.4) 12 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:02:24.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:22.515323+0000 osd.1 (osd.1) 13 : cluster [DBG] 2.d scrub starts 2024-02-19T21:02:24.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:22.516129+0000 osd.1 (osd.1) 14 : cluster [DBG] 2.d scrub ok 2024-02-19T21:02:24.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:22.920932+0000 mgr.y (mgr.14550) 42 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:02:24.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:22.937354+0000 osd.3 (osd.3) 11 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:02:24.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:22.939186+0000 osd.3 (osd.3) 12 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:02:24.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:23.099049+0000 osd.4 (osd.4) 11 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:02:24.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:23.099928+0000 osd.4 (osd.4) 12 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:02:26.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.216+0000 7f8714238700 1 -- 172.21.15.163:0/3551097421 >> v1:172.21.15.163:6790/0 conn(0x7f870c1042c0 legacy=0x7f870c106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:26.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.216+0000 7f8714238700 1 -- 172.21.15.163:0/3551097421 shutdown_connections 2024-02-19T21:02:26.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.216+0000 7f8714238700 1 -- 172.21.15.163:0/3551097421 >> 172.21.15.163:0/3551097421 conn(0x7f870c0f35a0 msgr2=0x7f870c0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:26.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.216+0000 7f8714238700 1 -- 172.21.15.163:0/3551097421 shutdown_connections 2024-02-19T21:02:26.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.216+0000 7f8714238700 1 -- 172.21.15.163:0/3551097421 wait complete. 2024-02-19T21:02:26.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8714238700 1 librados: starting msgr at 2024-02-19T21:02:26.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8714238700 1 librados: starting objecter 2024-02-19T21:02:26.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8714238700 1 Processor -- start 2024-02-19T21:02:26.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8714238700 1 -- start start 2024-02-19T21:02:26.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8714238700 1 librados: setting wanted keys 2024-02-19T21:02:26.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8714238700 1 librados: calling monclient init 2024-02-19T21:02:26.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8714238700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f870c0fcee0 con 0x7f870c0fd980 2024-02-19T21:02:26.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8714238700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f870c0fd0b0 con 0x7f870c1042c0 2024-02-19T21:02:26.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8714238700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f870c0fd280 con 0x7f870c100780 2024-02-19T21:02:26.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8711fd4700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f870c0fd980 0x7f870c0fc7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:52602/0 (socket says 172.21.15.163:52602) 2024-02-19T21:02:26.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8711fd4700 1 -- 172.21.15.163:0/1226310690 learned_addr learned my addr 172.21.15.163:0/1226310690 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:26.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f87117d3700 1 --1- 172.21.15.163:0/1226310690 >> v1:172.21.15.163:6790/0 conn(0x7f870c100780 0x7f870c114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:52120/0 (socket says 172.21.15.163:52120) 2024-02-19T21:02:26.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8702ffd700 10 client.?.objecter ms_handle_connect 0x7f870c0fd980 2024-02-19T21:02:26.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8702ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:26.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8702ffd700 10 client.?.objecter ms_handle_connect 0x7f870c100780 2024-02-19T21:02:26.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8702ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:26.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8702ffd700 10 client.?.objecter ms_handle_connect 0x7f870c1042c0 2024-02-19T21:02:26.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8702ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:26.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24592112 0 0) 0x7f870c0fcee0 con 0x7f870c0fd980 2024-02-19T21:02:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f8003030 con 0x7f870c0fd980 2024-02-19T21:02:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311960512 0 0) 0x7f870c0fd280 con 0x7f870c100780 2024-02-19T21:02:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f870c0fcee0 con 0x7f870c100780 2024-02-19T21:02:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3423674473 0 0) 0x7f870c0fd0b0 con 0x7f870c1042c0 2024-02-19T21:02:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.217+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f870c0fd280 con 0x7f870c1042c0 2024-02-19T21:02:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1255575421 0 0) 0x7f86f8003030 con 0x7f870c0fd980 2024-02-19T21:02:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f870c0fd0b0 con 0x7f870c0fd980 2024-02-19T21:02:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f86fc002900 con 0x7f870c0fd980 2024-02-19T21:02:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2282196887 0 0) 0x7f870c0fcee0 con 0x7f870c100780 2024-02-19T21:02:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86f8003030 con 0x7f870c100780 2024-02-19T21:02:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f87080018a0 con 0x7f870c100780 2024-02-19T21:02:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2781843083 0 0) 0x7f870c0fd280 con 0x7f870c1042c0 2024-02-19T21:02:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f870c0fcee0 con 0x7f870c1042c0 2024-02-19T21:02:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8704001360 con 0x7f870c1042c0 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1538062731 0 0) 0x7f870c0fd0b0 con 0x7f870c0fd980 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 >> v1:172.21.15.163:6790/0 conn(0x7f870c100780 legacy=0x7f870c114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 >> v1:172.21.15.193:6789/0 conn(0x7f870c1042c0 legacy=0x7f870c118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870c11cb90 con 0x7f870c0fd980 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8714238700 1 -- 172.21.15.163:0/1226310690 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f870c119bc0 con 0x7f870c0fd980 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8714238700 10 client.14658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8714238700 1 -- 172.21.15.163:0/1226310690 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f870c11a030 con 0x7f870c0fd980 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8714238700 1 librados: init done 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f86fc002c90 con 0x7f870c0fd980 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.218+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f86fc005bc0 con 0x7f870c0fd980 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.219+0000 7f8714238700 10 client.14658.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.219+0000 7f8714238700 15 client.14658.objecter _session_command_op_assign -1 1 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.219+0000 7f8714238700 20 client.14658.objecter _get_session osd=-1 returning homeless 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.219+0000 7f8714238700 10 client.14658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.219+0000 7f8714238700 1 -- 172.21.15.163:0/1226310690 --> v1:172.21.15.163:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f86f00028b0 con 0x7f870c0fd980 2024-02-19T21:02:26.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.219+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f86fc001940 con 0x7f870c0fd980 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(78..78 src has 1..78) v4 ==== 6172+0+0 (unknown 1950335092 0 0) 0x7f86fc0784c0 con 0x7f870c0fd980 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 10 client.14658.objecter ms_dispatch 0x7f870c001550 osd_map(78..78 src has 1..78) v4 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 3 client.14658.objecter handle_osd_map got epochs [78,78] > 0 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 3 client.14658.objecter handle_osd_map decoding full epoch 78 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 10 client.14658.objecter checking command 1 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 20 client.14658.objecter _get_session s=0x7f8704001360 osd=2 3 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 20 client.14658.objecter put_session s=0x7f8704001360 osd=2 3 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 15 client.14658.objecter _session_command_op_remove -1 1 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 20 client.14658.objecter _get_session s=0x7f8704001360 osd=2 3 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 20 client.14658.objecter get_session s=0x7f8704001360 osd=2 3 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 15 client.14658.objecter _session_command_op_assign 2 1 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 20 client.14658.objecter put_session s=0x7f8704001360 osd=2 4 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 10 client.14658.objecter _send_command 1 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 --> v1:172.21.15.163:6809/1563158487 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86f8072110 con 0x7f86f806e400 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 20 client.14658.objecter dump_active .. 0 homeless 2024-02-19T21:02:26.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (unknown 100594580 0 0) 0x7f86fc0788a0 con 0x7f870c0fd980 2024-02-19T21:02:26.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 10 client.14658.objecter ms_handle_connect 0x7f86f8063db0 2024-02-19T21:02:26.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.220+0000 7f8702ffd700 10 client.14658.objecter ms_handle_connect 0x7f86f806e400 2024-02-19T21:02:26.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.221+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f86f8072110 con 0x7f86f806e400 2024-02-19T21:02:26.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.221+0000 7f8702ffd700 10 client.14658.objecter ms_dispatch 0x7f870c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:02:26.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.221+0000 7f8702ffd700 10 client.14658.objecter _finish_command 1 = system:0 2024-02-19T21:02:26.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.221+0000 7f8702ffd700 20 client.14658.objecter put_session s=0x7f8704001360 osd=2 4 2024-02-19T21:02:26.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.221+0000 7f8702ffd700 15 client.14658.objecter _session_command_op_remove 2 1 2024-02-19T21:02:26.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 10 client.14658.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:02:26.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 15 client.14658.objecter _session_command_op_assign -1 2 2024-02-19T21:02:26.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 20 client.14658.objecter _get_session s=0x7f8704001360 osd=2 3 2024-02-19T21:02:26.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 20 client.14658.objecter put_session s=0x7f8704001360 osd=2 3 2024-02-19T21:02:26.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 20 client.14658.objecter _get_session s=0x7f8704001360 osd=2 3 2024-02-19T21:02:26.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 15 client.14658.objecter _session_command_op_remove -1 2 2024-02-19T21:02:26.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 20 client.14658.objecter get_session s=0x7f8704001360 osd=2 3 2024-02-19T21:02:26.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 15 client.14658.objecter _session_command_op_assign 2 2 2024-02-19T21:02:26.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 20 client.14658.objecter put_session s=0x7f8704001360 osd=2 4 2024-02-19T21:02:26.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 10 client.14658.objecter _send_command 2 2024-02-19T21:02:26.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 1 -- 172.21.15.163:0/1226310690 --> v1:172.21.15.163:6809/1563158487 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f86f0003be0 con 0x7f86f806e400 2024-02-19T21:02:26.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8702ffd700 1 -- 172.21.15.163:0/1226310690 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1016321414) 0x7f86f0003be0 con 0x7f86f806e400 2024-02-19T21:02:26.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8702ffd700 10 client.14658.objecter ms_dispatch 0x7f870c001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:02:26.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8702ffd700 10 client.14658.objecter _finish_command 2 = system:0 2024-02-19T21:02:26.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8702ffd700 20 client.14658.objecter put_session s=0x7f8704001360 osd=2 4 2024-02-19T21:02:26.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8702ffd700 15 client.14658.objecter _session_command_op_remove 2 2 2024-02-19T21:02:26.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 10 librados: watch_flush enter 2024-02-19T21:02:26.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 10 librados: watch_flush exit 2024-02-19T21:02:26.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 10 client.14658.objecter close_session for osd.2 2024-02-19T21:02:26.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 1 -- 172.21.15.163:0/1226310690 >> v1:172.21.15.163:6809/1563158487 conn(0x7f86f806e400 legacy=0x7f86f80708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:26.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 20 client.14658.objecter put_session s=0x7f8704001360 osd=2 1 2024-02-19T21:02:26.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 20 client.14658.objecter shutdown clearing up homeless session... 2024-02-19T21:02:26.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 10 client.14658.objecter successfully canceled tick 2024-02-19T21:02:26.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 1 -- 172.21.15.163:0/1226310690 >> v1:172.21.15.163:6800/1552514681 conn(0x7f86f8063db0 legacy=0x7f86f8066270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:26.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 1 -- 172.21.15.163:0/1226310690 >> v1:172.21.15.163:6789/0 conn(0x7f870c0fd980 legacy=0x7f870c0fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:26.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f87127d5700 1 -- 172.21.15.163:0/1226310690 reap_dead start 2024-02-19T21:02:26.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 1 -- 172.21.15.163:0/1226310690 shutdown_connections 2024-02-19T21:02:26.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 1 -- 172.21.15.163:0/1226310690 >> 172.21.15.163:0/1226310690 conn(0x7f870c0f35a0 msgr2=0x7f870c107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:26.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 1 -- 172.21.15.163:0/1226310690 shutdown_connections 2024-02-19T21:02:26.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 1 -- 172.21.15.163:0/1226310690 wait complete. 2024-02-19T21:02:26.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:26.235+0000 7f8714238700 1 librados: shutdown 2024-02-19T21:02:26.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:24.921598+0000 mgr.y (mgr.14550) 43 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:26.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:24.948019+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:02:26.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:24.948827+0000 osd.3 (osd.3) 14 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:02:26.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:25.253584+0000 osd.2 (osd.2) 11 : cluster [DBG] 5.d scrub starts 2024-02-19T21:02:26.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:25.255639+0000 osd.2 (osd.2) 12 : cluster [DBG] 5.d scrub ok 2024-02-19T21:02:26.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:25.323332+0000 osd.5 (osd.5) 7 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:02:26.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:25.326437+0000 osd.5 (osd.5) 8 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:02:26.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:24.921598+0000 mgr.y (mgr.14550) 43 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:26.239 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:24.948019+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:02:26.239 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:24.948827+0000 osd.3 (osd.3) 14 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:02:26.239 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:25.253584+0000 osd.2 (osd.2) 11 : cluster [DBG] 5.d scrub starts 2024-02-19T21:02:26.239 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:25.255639+0000 osd.2 (osd.2) 12 : cluster [DBG] 5.d scrub ok 2024-02-19T21:02:26.239 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:25.323332+0000 osd.5 (osd.5) 7 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:02:26.239 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:25.326437+0000 osd.5 (osd.5) 8 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:02:26.539 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:24.921598+0000 mgr.y (mgr.14550) 43 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:26.539 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:24.948019+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:02:26.539 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:24.948827+0000 osd.3 (osd.3) 14 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:02:26.539 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:25.253584+0000 osd.2 (osd.2) 11 : cluster [DBG] 5.d scrub starts 2024-02-19T21:02:26.539 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:25.255639+0000 osd.2 (osd.2) 12 : cluster [DBG] 5.d scrub ok 2024-02-19T21:02:26.539 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:25.323332+0000 osd.5 (osd.5) 7 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:02:26.539 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:25.326437+0000 osd.5 (osd.5) 8 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:02:26.941 INFO:teuthology.orchestra.run.smithi163.stdout:77309411368 2024-02-19T21:02:26.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-19T21:02:27.215 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:27.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:26.071777+0000 osd.4 (osd.4) 13 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:02:27.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:26.073618+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:02:27.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:26.071777+0000 osd.4 (osd.4) 13 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:02:27.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:26.073618+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:02:27.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:26.071777+0000 osd.4 (osd.4) 13 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:02:27.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:26.073618+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:02:27.846 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:02:27.848 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 systemd[1]: Started Ceph grafana.a for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T21:02:27.848 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-02-19T21:02:27.848 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-02-19T21:02:27.848 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-02-19T21:02:27.848 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-02-19T21:02:27.848 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-02-19T21:02:27.848 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-02-19T21:02:27.848 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-19T21:02:27.848 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-02-19T21:02:27.848 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-02-19T21:02:27.848 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-02-19T21:02:27.848 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-02-19T21:02:27.848 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-02-19T21:02:27.849 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-02-19T21:02:27.850 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-02-19T21:02:27.851 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-02-19T21:02:27.851 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-02-19T21:02:27.851 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-02-19T21:02:27.851 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-02-19T21:02:27.851 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-02-19T21:02:27.851 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-02-19T21:02:27.851 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-02-19T21:02:27.851 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-02-19T21:02:27.851 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-02-19T21:02:27.851 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-02-19T21:02:27.851 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-02-19T21:02:27.851 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-02-19T21:02:27.851 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-19T21:02:27.851 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-02-19T21:02:27.851 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-19T21:02:27.852 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-02-19T21:02:27.852 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-19T21:02:27.852 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-19T21:02:27.852 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-02-19T21:02:27.852 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-02-19T21:02:27.852 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-02-19T21:02:27.852 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-02-19T21:02:27.852 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-02-19T21:02:27.852 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-02-19T21:02:27.852 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-02-19T21:02:27.852 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-02-19T21:02:27.852 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-02-19T21:02:27.852 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-02-19T21:02:27.852 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-02-19T21:02:27.853 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-02-19T21:02:27.853 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-02-19T21:02:27.853 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-02-19T21:02:27.853 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-02-19T21:02:27.853 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-02-19T21:02:27.853 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-19T21:02:27.853 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-02-19T21:02:27.853 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-02-19T21:02:27.853 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-02-19T21:02:27.853 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-02-19T21:02:27.853 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-02-19T21:02:27.853 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-02-19T21:02:27.853 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-02-19T21:02:27.853 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-02-19T21:02:27.854 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-02-19T21:02:27.854 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-02-19T21:02:27.854 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-02-19T21:02:27.854 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-02-19T21:02:27.854 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-02-19T21:02:27.854 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-02-19T21:02:27.854 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-02-19T21:02:27.854 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-02-19T21:02:27.854 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-02-19T21:02:27.854 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-02-19T21:02:27.854 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-02-19T21:02:27.854 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-02-19T21:02:27.854 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-02-19T21:02:27.854 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-02-19T21:02:27.854 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-02-19T21:02:27.855 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-02-19T21:02:27.856 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-19T21:02:27.856 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-02-19T21:02:27.856 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-19T21:02:27.856 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-02-19T21:02:27.856 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-02-19T21:02:27.856 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-19T21:02:27.856 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-02-19T21:02:27.856 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-02-19T21:02:27.856 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-02-19T21:02:27.856 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-02-19T21:02:27.856 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-02-19T21:02:27.856 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-02-19T21:02:27.857 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-19T21:02:27.858 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-19T21:02:28.100 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-19T21:02:28.100 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-02-19T21:02:28.100 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-19T21:02:28.100 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-02-19T21:02:28.100 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-02-19T21:02:28.100 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-02-19T21:02:28.100 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-02-19T21:02:28.101 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-02-19T21:02:28.101 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-02-19T21:02:28.101 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-02-19T21:02:28.101 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-02-19T21:02:28.101 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-02-19T21:02:28.101 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-02-19T21:02:28.101 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-02-19T21:02:28.101 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-19T21:02:28.101 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-02-19T21:02:28.101 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-02-19T21:02:28.101 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-02-19T21:02:28.102 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-02-19T21:02:28.103 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-02-19T21:02:28.103 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-02-19T21:02:28.103 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-02-19T21:02:28.103 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-02-19T21:02:28.103 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-02-19T21:02:28.103 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-19T21:02:28.103 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-19T21:02:28.103 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-19T21:02:28.103 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-02-19T21:02:28.103 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-19T21:02:28.103 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-02-19T21:02:28.103 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-02-19T21:02:28.103 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-02-19T21:02:28.103 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-02-19T21:02:28.103 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-02-19T21:02:28.104 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-02-19T21:02:28.104 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-02-19T21:02:28.104 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-02-19T21:02:28.104 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-02-19T21:02:28.104 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-02-19T21:02:28.104 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-02-19T21:02:28.104 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-02-19T21:02:28.104 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-02-19T21:02:28.104 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-19T21:02:28.104 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-19T21:02:28.104 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-19T21:02:28.104 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-19T21:02:28.104 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-02-19T21:02:28.104 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-02-19T21:02:28.105 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-02-19T21:02:28.105 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-02-19T21:02:28.105 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-19T21:02:28.105 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-02-19T21:02:28.105 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-02-19T21:02:28.105 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-02-19T21:02:28.105 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-02-19T21:02:28.105 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-02-19T21:02:28.105 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-02-19T21:02:28.105 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-02-19T21:02:28.105 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-02-19T21:02:28.105 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-02-19T21:02:28.105 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-02-19T21:02:28.105 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-02-19T21:02:28.105 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-02-19T21:02:28.106 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-02-19T21:02:28.106 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-02-19T21:02:28.106 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-02-19T21:02:28.106 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-19T21:02:28.106 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-19T21:02:28.106 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-02-19T21:02:28.106 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-02-19T21:02:28.106 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-02-19T21:02:28.106 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-02-19T21:02:28.106 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-02-19T21:02:28.106 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-02-19T21:02:28.106 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-02-19T21:02:28.106 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-02-19T21:02:28.106 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-02-19T21:02:28.107 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-02-19T21:02:28.107 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-02-19T21:02:28.107 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-19T21:02:28.107 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-02-19T21:02:28.107 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-02-19T21:02:28.107 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-02-19T21:02:28.107 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-02-19T21:02:28.107 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-02-19T21:02:28.107 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-19T21:02:28.107 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-02-19T21:02:28.107 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-02-19T21:02:28.107 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-02-19T21:02:28.107 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-02-19T21:02:28.107 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-02-19T21:02:28.107 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-02-19T21:02:28.108 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-02-19T21:02:28.108 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-02-19T21:02:28.108 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-02-19T21:02:28.108 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-02-19T21:02:28.108 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-02-19T21:02:28.108 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-02-19T21:02:28.108 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-02-19T21:02:28.108 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-02-19T21:02:28.108 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-02-19T21:02:28.108 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-02-19T21:02:28.108 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-02-19T21:02:28.108 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:27+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-02-19T21:02:28.108 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-02-19T21:02:28.108 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-02-19T21:02:28.108 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-02-19T21:02:28.109 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-02-19T21:02:28.109 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-02-19T21:02:28.109 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-02-19T21:02:28.109 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-02-19T21:02:28.109 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-02-19T21:02:28.109 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-02-19T21:02:28.109 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-02-19T21:02:28.109 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-02-19T21:02:28.109 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-02-19T21:02:28.109 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-19T21:02:28.109 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-19T21:02:28.109 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-19T21:02:28.109 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-02-19T21:02:28.109 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-02-19T21:02:28.110 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-19T21:02:28.110 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-19T21:02:28.110 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-02-19T21:02:28.110 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-19T21:02:28.110 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-02-19T21:02:28.110 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-02-19T21:02:28.110 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-02-19T21:02:28.110 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-19T21:02:28.110 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-02-19T21:02:28.110 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-02-19T21:02:28.110 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-02-19T21:02:28.110 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-02-19T21:02:28.110 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-19T21:02:28.110 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-02-19T21:02:28.111 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-02-19T21:02:28.111 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-02-19T21:02:28.111 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-02-19T21:02:28.111 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-02-19T21:02:28.111 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-19T21:02:28.111 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-02-19T21:02:28.111 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-02-19T21:02:28.111 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-19T21:02:28.111 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-02-19T21:02:28.111 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-19T21:02:28.111 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-19T21:02:28.111 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-19T21:02:28.111 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-02-19T21:02:28.111 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-02-19T21:02:28.112 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-02-19T21:02:28.112 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-02-19T21:02:28.112 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-02-19T21:02:28.112 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-19T21:02:28.112 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-02-19T21:02:28.112 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-02-19T21:02:28.112 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-02-19T21:02:28.112 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-02-19T21:02:28.112 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-02-19T21:02:28.112 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-02-19T21:02:28.112 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-02-19T21:02:28.112 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-02-19T21:02:28.113 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-02-19T21:02:28.113 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-02-19T21:02:28.113 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-02-19T21:02:28.113 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-02-19T21:02:28.113 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-02-19T21:02:28.113 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-02-19T21:02:28.113 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-02-19T21:02:28.113 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-02-19T21:02:28.113 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-02-19T21:02:28.113 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-02-19T21:02:28.113 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-02-19T21:02:28.113 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-02-19T21:02:28.113 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-02-19T21:02:28.113 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-02-19T21:02:28.114 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-02-19T21:02:28.114 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-02-19T21:02:28.114 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-02-19T21:02:28.114 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-02-19T21:02:28.114 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-02-19T21:02:28.114 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-02-19T21:02:28.114 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-02-19T21:02:28.114 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-02-19T21:02:28.114 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-02-19T21:02:28.114 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-02-19T21:02:28.114 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-02-19T21:02:28.114 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-02-19T21:02:28.114 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-02-19T21:02:28.114 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-02-19T21:02:28.115 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-02-19T21:02:28.115 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-02-19T21:02:28.115 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-02-19T21:02:28.115 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-02-19T21:02:28.115 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-02-19T21:02:28.115 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-02-19T21:02:28.115 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-02-19T21:02:28.115 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=501.791571ms 2024-02-19T21:02:28.115 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-19T21:02:28.115 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-19T21:02:28.397 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:26.921980+0000 mgr.y (mgr.14550) 44 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:28.397 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:27.097047+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:02:28.397 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:27.098839+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:02:28.397 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:27.723955+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:28.397 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:27.727633+0000 mon.b (mon.1) 61 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:02:28.397 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:26.921980+0000 mgr.y (mgr.14550) 44 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:28.398 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:27.097047+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:02:28.398 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:27.098839+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:02:28.398 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:27.723955+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:28.398 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:27.727633+0000 mon.b (mon.1) 61 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:02:28.498 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:26.921980+0000 mgr.y (mgr.14550) 44 : cluster [DBG] pgmap v33: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:28.498 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:27.097047+0000 osd.4 (osd.4) 15 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:02:28.498 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:27.098839+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:02:28.498 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:27.723955+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:28.498 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:27.727633+0000 mon.b (mon.1) 61 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:02:28.498 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-19T21:02:28.498 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-19T21:02:28.498 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-19T21:02:28.498 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-19T21:02:28.498 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-02-19T21:02:28.499 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-19T21:02:28.499 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-19T21:02:28.499 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-19T21:02:28.499 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:02:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:02:28+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-19T21:02:29.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.193+0000 7f155c69b700 1 -- 172.21.15.163:0/3025825588 >> v1:172.21.15.163:6790/0 conn(0x7f15541031b0 legacy=0x7f1554105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:29.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.193+0000 7f155c69b700 1 -- 172.21.15.163:0/3025825588 shutdown_connections 2024-02-19T21:02:29.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.193+0000 7f155c69b700 1 -- 172.21.15.163:0/3025825588 >> 172.21.15.163:0/3025825588 conn(0x7f15540f2470 msgr2=0x7f15540f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:29.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.193+0000 7f155c69b700 1 -- 172.21.15.163:0/3025825588 shutdown_connections 2024-02-19T21:02:29.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.193+0000 7f155c69b700 1 -- 172.21.15.163:0/3025825588 wait complete. 2024-02-19T21:02:29.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.193+0000 7f155c69b700 1 librados: starting msgr at 2024-02-19T21:02:29.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.193+0000 7f155c69b700 1 librados: starting objecter 2024-02-19T21:02:29.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.193+0000 7f155c69b700 1 Processor -- start 2024-02-19T21:02:29.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.194+0000 7f155c69b700 1 -- start start 2024-02-19T21:02:29.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.194+0000 7f155c69b700 1 librados: setting wanted keys 2024-02-19T21:02:29.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.194+0000 7f155c69b700 1 librados: calling monclient init 2024-02-19T21:02:29.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.194+0000 7f155c69b700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15541163a0 con 0x7f15540fc870 2024-02-19T21:02:29.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.194+0000 7f155c69b700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1554117580 con 0x7f15541031b0 2024-02-19T21:02:29.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.194+0000 7f155c69b700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1554118780 con 0x7f15540ff670 2024-02-19T21:02:29.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.194+0000 7f155a437700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f15540fc870 0x7f15540f8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:52618/0 (socket says 172.21.15.163:52618) 2024-02-19T21:02:29.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.194+0000 7f155a437700 1 -- 172.21.15.163:0/1703134790 learned_addr learned my addr 172.21.15.163:0/1703134790 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:29.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.194+0000 7f154b7fe700 10 client.?.objecter ms_handle_connect 0x7f15540fc870 2024-02-19T21:02:29.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.194+0000 7f154b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:29.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.194+0000 7f154b7fe700 10 client.?.objecter ms_handle_connect 0x7f15540ff670 2024-02-19T21:02:29.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.194+0000 7f154b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:29.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.194+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538195384 0 0) 0x7f15541163a0 con 0x7f15540fc870 2024-02-19T21:02:29.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1540003030 con 0x7f15540fc870 2024-02-19T21:02:29.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97845973 0 0) 0x7f1554118780 con 0x7f15540ff670 2024-02-19T21:02:29.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15541163a0 con 0x7f15540ff670 2024-02-19T21:02:29.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 10 client.?.objecter ms_handle_connect 0x7f15541031b0 2024-02-19T21:02:29.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:29.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3202707061 0 0) 0x7f15541163a0 con 0x7f15540ff670 2024-02-19T21:02:29.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1554118780 con 0x7f15540ff670 2024-02-19T21:02:29.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1544001780 con 0x7f15540ff670 2024-02-19T21:02:29.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3981308206 0 0) 0x7f1540003030 con 0x7f15540fc870 2024-02-19T21:02:29.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15541163a0 con 0x7f15540fc870 2024-02-19T21:02:29.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1550002990 con 0x7f15540fc870 2024-02-19T21:02:29.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654084215 0 0) 0x7f1554117580 con 0x7f15541031b0 2024-02-19T21:02:29.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1540003030 con 0x7f15541031b0 2024-02-19T21:02:29.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1084864682 0 0) 0x7f1554118780 con 0x7f15540ff670 2024-02-19T21:02:29.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 >> v1:172.21.15.193:6789/0 conn(0x7f15541031b0 legacy=0x7f1554114a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:29.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 >> v1:172.21.15.163:6789/0 conn(0x7f15540fc870 legacy=0x7f15540f8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:29.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1554119980 con 0x7f15540ff670 2024-02-19T21:02:29.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f155c69b700 1 -- 172.21.15.163:0/1703134790 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1554118970 con 0x7f15540ff670 2024-02-19T21:02:29.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f155c69b700 10 client.34589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:29.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f155c69b700 1 -- 172.21.15.163:0/1703134790 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1554118e60 con 0x7f15540ff670 2024-02-19T21:02:29.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f155c69b700 1 librados: init done 2024-02-19T21:02:29.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f1544001e00 con 0x7f15540ff670 2024-02-19T21:02:29.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.195+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1544003270 con 0x7f15540ff670 2024-02-19T21:02:29.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.196+0000 7f155c69b700 10 client.34589.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:02:29.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.196+0000 7f155c69b700 15 client.34589.objecter _session_command_op_assign -1 1 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.196+0000 7f155c69b700 20 client.34589.objecter _get_session osd=-1 returning homeless 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.196+0000 7f155c69b700 10 client.34589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.196+0000 7f155c69b700 1 -- 172.21.15.163:0/1703134790 --> v1:172.21.15.163:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f15380028b0 con 0x7f15540ff670 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.196+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f1544004500 con 0x7f15540ff670 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(78..78 src has 1..78) v4 ==== 6172+0+0 (unknown 1950335092 0 0) 0x7f1544075ca0 con 0x7f15540ff670 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 10 client.34589.objecter ms_dispatch 0x7f1554001550 osd_map(78..78 src has 1..78) v4 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 3 client.34589.objecter handle_osd_map got epochs [78,78] > 0 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 3 client.34589.objecter handle_osd_map decoding full epoch 78 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 10 client.34589.objecter checking command 1 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 20 client.34589.objecter _get_session s=0x7f1540070170 osd=3 3 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 20 client.34589.objecter put_session s=0x7f1540070170 osd=3 3 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 15 client.34589.objecter _session_command_op_remove -1 1 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 20 client.34589.objecter _get_session s=0x7f1540070170 osd=3 3 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 20 client.34589.objecter get_session s=0x7f1540070170 osd=3 3 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 15 client.34589.objecter _session_command_op_assign 3 1 2024-02-19T21:02:29.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 20 client.34589.objecter put_session s=0x7f1540070170 osd=3 4 2024-02-19T21:02:29.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 10 client.34589.objecter _send_command 1 2024-02-19T21:02:29.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 --> v1:172.21.15.163:6813/2112279769 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1540074490 con 0x7f1540070820 2024-02-19T21:02:29.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 20 client.34589.objecter dump_active .. 0 homeless 2024-02-19T21:02:29.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (unknown 100594580 0 0) 0x7f15440760d0 con 0x7f15540ff670 2024-02-19T21:02:29.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 10 client.34589.objecter ms_handle_connect 0x7f1540065ee0 2024-02-19T21:02:29.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.197+0000 7f154b7fe700 10 client.34589.objecter ms_handle_connect 0x7f1540070820 2024-02-19T21:02:29.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.198+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1540074490 con 0x7f1540070820 2024-02-19T21:02:29.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.198+0000 7f154b7fe700 10 client.34589.objecter ms_dispatch 0x7f1554001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:02:29.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.198+0000 7f154b7fe700 10 client.34589.objecter _finish_command 1 = system:0 2024-02-19T21:02:29.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.198+0000 7f154b7fe700 20 client.34589.objecter put_session s=0x7f1540070170 osd=3 4 2024-02-19T21:02:29.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.198+0000 7f154b7fe700 15 client.34589.objecter _session_command_op_remove 3 1 2024-02-19T21:02:29.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 10 client.34589.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:02:29.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 15 client.34589.objecter _session_command_op_assign -1 2 2024-02-19T21:02:29.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 20 client.34589.objecter _get_session s=0x7f1540070170 osd=3 3 2024-02-19T21:02:29.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 20 client.34589.objecter put_session s=0x7f1540070170 osd=3 3 2024-02-19T21:02:29.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 20 client.34589.objecter _get_session s=0x7f1540070170 osd=3 3 2024-02-19T21:02:29.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 15 client.34589.objecter _session_command_op_remove -1 2 2024-02-19T21:02:29.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 20 client.34589.objecter get_session s=0x7f1540070170 osd=3 3 2024-02-19T21:02:29.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 15 client.34589.objecter _session_command_op_assign 3 2 2024-02-19T21:02:29.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 20 client.34589.objecter put_session s=0x7f1540070170 osd=3 4 2024-02-19T21:02:29.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 10 client.34589.objecter _send_command 2 2024-02-19T21:02:29.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 1 -- 172.21.15.163:0/1703134790 --> v1:172.21.15.163:6813/2112279769 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1538001140 con 0x7f1540070820 2024-02-19T21:02:29.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f154b7fe700 1 -- 172.21.15.163:0/1703134790 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 4066359062) 0x7f1538001140 con 0x7f1540070820 2024-02-19T21:02:29.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f154b7fe700 10 client.34589.objecter ms_dispatch 0x7f1554001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:02:29.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f154b7fe700 10 client.34589.objecter _finish_command 2 = system:0 2024-02-19T21:02:29.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f154b7fe700 20 client.34589.objecter put_session s=0x7f1540070170 osd=3 4 2024-02-19T21:02:29.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f154b7fe700 15 client.34589.objecter _session_command_op_remove 3 2 2024-02-19T21:02:29.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 10 librados: watch_flush enter 2024-02-19T21:02:29.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 10 librados: watch_flush exit 2024-02-19T21:02:29.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 10 client.34589.objecter close_session for osd.3 2024-02-19T21:02:29.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 1 -- 172.21.15.163:0/1703134790 >> v1:172.21.15.163:6813/2112279769 conn(0x7f1540070820 legacy=0x7f1540072cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:29.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 20 client.34589.objecter put_session s=0x7f1540070170 osd=3 1 2024-02-19T21:02:29.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 20 client.34589.objecter shutdown clearing up homeless session... 2024-02-19T21:02:29.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 10 client.34589.objecter successfully canceled tick 2024-02-19T21:02:29.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 1 -- 172.21.15.163:0/1703134790 >> v1:172.21.15.163:6800/1552514681 conn(0x7f1540065ee0 legacy=0x7f15400683a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:29.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 1 -- 172.21.15.163:0/1703134790 >> v1:172.21.15.163:6790/0 conn(0x7f15540ff670 legacy=0x7f15540f92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:29.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155ac38700 1 -- 172.21.15.163:0/1703134790 reap_dead start 2024-02-19T21:02:29.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 1 -- 172.21.15.163:0/1703134790 shutdown_connections 2024-02-19T21:02:29.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 1 -- 172.21.15.163:0/1703134790 >> 172.21.15.163:0/1703134790 conn(0x7f15540f2470 msgr2=0x7f1554106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:29.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 1 -- 172.21.15.163:0/1703134790 shutdown_connections 2024-02-19T21:02:29.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 1 -- 172.21.15.163:0/1703134790 wait complete. 2024-02-19T21:02:29.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:29.213+0000 7f155c69b700 1 librados: shutdown 2024-02-19T21:02:29.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:27.565002+0000 osd.7 (osd.7) 9 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:02:29.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:27.567899+0000 osd.7 (osd.7) 10 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:02:29.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:27.760805+0000 osd.6 (osd.6) 7 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:02:29.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:27.761677+0000 osd.6 (osd.6) 8 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:02:29.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:29.104292+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:29.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:27.565002+0000 osd.7 (osd.7) 9 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:02:29.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:27.567899+0000 osd.7 (osd.7) 10 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:02:29.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:27.760805+0000 osd.6 (osd.6) 7 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:02:29.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:27.761677+0000 osd.6 (osd.6) 8 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:02:29.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:29.104292+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:29.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:27.565002+0000 osd.7 (osd.7) 9 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:02:29.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:27.567899+0000 osd.7 (osd.7) 10 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:02:29.489 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:27.760805+0000 osd.6 (osd.6) 7 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:02:29.489 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:27.761677+0000 osd.6 (osd.6) 8 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:02:29.489 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:29.104292+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:30.311 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:02:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:02:30.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:28.468236+0000 osd.1 (osd.1) 15 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:02:30.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:28.471127+0000 osd.1 (osd.1) 16 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:02:30.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:28.922340+0000 mgr.y (mgr.14550) 45 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:02:30.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:28.468236+0000 osd.1 (osd.1) 15 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:02:30.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:28.471127+0000 osd.1 (osd.1) 16 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:02:30.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:28.922340+0000 mgr.y (mgr.14550) 45 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:02:30.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:28.468236+0000 osd.1 (osd.1) 15 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:02:30.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:28.471127+0000 osd.1 (osd.1) 16 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:02:30.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:28.922340+0000 mgr.y (mgr.14550) 45 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:02:30.585 INFO:teuthology.orchestra.run.smithi163.stdout:107374182436 2024-02-19T21:02:30.586 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-19T21:02:31.224 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:31.457 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:29.477324+0000 osd.1 (osd.1) 17 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:02:31.457 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:29.480272+0000 osd.1 (osd.1) 18 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:02:31.457 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:29.683001+0000 osd.6 (osd.6) 9 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:02:31.457 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:29.684694+0000 osd.6 (osd.6) 10 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:02:31.458 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:29.954489+0000 mgr.y (mgr.14550) 46 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:31.458 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:30.068416+0000 osd.4 (osd.4) 17 : cluster [DBG] 4.d scrub starts 2024-02-19T21:02:31.458 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:30.070227+0000 osd.4 (osd.4) 18 : cluster [DBG] 4.d scrub ok 2024-02-19T21:02:31.458 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:29.477324+0000 osd.1 (osd.1) 17 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:02:31.458 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:29.480272+0000 osd.1 (osd.1) 18 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:02:31.458 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:29.683001+0000 osd.6 (osd.6) 9 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:02:31.458 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:29.684694+0000 osd.6 (osd.6) 10 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:02:31.458 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:29.954489+0000 mgr.y (mgr.14550) 46 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:31.458 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:30.068416+0000 osd.4 (osd.4) 17 : cluster [DBG] 4.d scrub starts 2024-02-19T21:02:31.458 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:30.070227+0000 osd.4 (osd.4) 18 : cluster [DBG] 4.d scrub ok 2024-02-19T21:02:31.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:29.477324+0000 osd.1 (osd.1) 17 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:02:31.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:29.480272+0000 osd.1 (osd.1) 18 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:02:31.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:29.683001+0000 osd.6 (osd.6) 9 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:02:31.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:29.684694+0000 osd.6 (osd.6) 10 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:02:31.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:29.954489+0000 mgr.y (mgr.14550) 46 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:31.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:30.068416+0000 osd.4 (osd.4) 17 : cluster [DBG] 4.d scrub starts 2024-02-19T21:02:31.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:30.070227+0000 osd.4 (osd.4) 18 : cluster [DBG] 4.d scrub ok 2024-02-19T21:02:32.447 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:30.722005+0000 osd.6 (osd.6) 11 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:02:32.447 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:30.724904+0000 osd.6 (osd.6) 12 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:02:32.447 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:30.922945+0000 mgr.y (mgr.14550) 47 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:32.447 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:31.354164+0000 osd.2 (osd.2) 13 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:02:32.447 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:31.357036+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:02:32.447 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:32.059091+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:32.447 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:30.722005+0000 osd.6 (osd.6) 11 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:02:32.447 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:30.724904+0000 osd.6 (osd.6) 12 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:02:32.447 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:30.922945+0000 mgr.y (mgr.14550) 47 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:32.447 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:31.354164+0000 osd.2 (osd.2) 13 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:02:32.448 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:31.357036+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:02:32.448 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:32.059091+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:32.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:30.722005+0000 osd.6 (osd.6) 11 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:02:32.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:30.724904+0000 osd.6 (osd.6) 12 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:02:32.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:30.922945+0000 mgr.y (mgr.14550) 47 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:32.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:31.354164+0000 osd.2 (osd.2) 13 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:02:32.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:31.357036+0000 osd.2 (osd.2) 14 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:02:32.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:32.059091+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:33.205 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:32.038980+0000 osd.4 (osd.4) 19 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:02:33.206 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:32.040725+0000 osd.4 (osd.4) 20 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:02:33.206 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:32.548348+0000 osd.0 (osd.0) 13 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:02:33.206 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:32.550186+0000 osd.0 (osd.0) 14 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:02:33.206 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:32.038980+0000 osd.4 (osd.4) 19 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:02:33.206 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:32.040725+0000 osd.4 (osd.4) 20 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:02:33.206 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:32.548348+0000 osd.0 (osd.0) 13 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:02:33.206 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:32.550186+0000 osd.0 (osd.0) 14 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:02:33.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:32.038980+0000 osd.4 (osd.4) 19 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:02:33.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:32.040725+0000 osd.4 (osd.4) 20 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:02:33.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:32.548348+0000 osd.0 (osd.0) 13 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:02:33.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:32.550186+0000 osd.0 (osd.0) 14 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:02:33.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.744+0000 7fbd4acb4700 1 -- 172.21.15.163:0/1567763663 >> v1:172.21.15.163:6789/0 conn(0x7fbd44103190 legacy=0x7fbd44105650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:33.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.744+0000 7fbd4acb4700 1 -- 172.21.15.163:0/1567763663 shutdown_connections 2024-02-19T21:02:33.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.744+0000 7fbd4acb4700 1 -- 172.21.15.163:0/1567763663 >> 172.21.15.163:0/1567763663 conn(0x7fbd4406fdc0 msgr2=0x7fbd440f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:33.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.744+0000 7fbd4acb4700 1 -- 172.21.15.163:0/1567763663 shutdown_connections 2024-02-19T21:02:33.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.744+0000 7fbd4acb4700 1 -- 172.21.15.163:0/1567763663 wait complete. 2024-02-19T21:02:33.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd4acb4700 1 librados: starting msgr at 2024-02-19T21:02:33.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd4acb4700 1 librados: starting objecter 2024-02-19T21:02:33.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd4acb4700 1 Processor -- start 2024-02-19T21:02:33.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd4acb4700 1 -- start start 2024-02-19T21:02:33.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd4acb4700 1 librados: setting wanted keys 2024-02-19T21:02:33.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd4acb4700 1 librados: calling monclient init 2024-02-19T21:02:33.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd4acb4700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbd44117450 con 0x7fbd440fc8b0 2024-02-19T21:02:33.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd4acb4700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbd44118630 con 0x7fbd440ff650 2024-02-19T21:02:33.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd4acb4700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbd44119830 con 0x7fbd44103190 2024-02-19T21:02:33.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd48a50700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fbd440fc8b0 0x7fbd440f9be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:52638/0 (socket says 172.21.15.163:52638) 2024-02-19T21:02:33.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd48a50700 1 -- 172.21.15.163:0/4128155926 learned_addr learned my addr 172.21.15.163:0/4128155926 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:33.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd41ffb700 10 client.?.objecter ms_handle_connect 0x7fbd440fc8b0 2024-02-19T21:02:33.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd41ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:33.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd41ffb700 10 client.?.objecter ms_handle_connect 0x7fbd44103190 2024-02-19T21:02:33.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd41ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:33.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd41ffb700 10 client.?.objecter ms_handle_connect 0x7fbd440ff650 2024-02-19T21:02:33.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd41ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:33.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840805752 0 0) 0x7fbd44117450 con 0x7fbd440fc8b0 2024-02-19T21:02:33.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd34003030 con 0x7fbd440fc8b0 2024-02-19T21:02:33.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459960812 0 0) 0x7fbd44119830 con 0x7fbd44103190 2024-02-19T21:02:33.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd44117450 con 0x7fbd44103190 2024-02-19T21:02:33.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927165880 0 0) 0x7fbd44118630 con 0x7fbd440ff650 2024-02-19T21:02:33.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.745+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd44119830 con 0x7fbd440ff650 2024-02-19T21:02:33.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 331038171 0 0) 0x7fbd34003030 con 0x7fbd440fc8b0 2024-02-19T21:02:33.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd44118630 con 0x7fbd440fc8b0 2024-02-19T21:02:33.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbd38001a70 con 0x7fbd440fc8b0 2024-02-19T21:02:33.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2879122988 0 0) 0x7fbd44117450 con 0x7fbd44103190 2024-02-19T21:02:33.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd34003030 con 0x7fbd44103190 2024-02-19T21:02:33.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbd3c0017b0 con 0x7fbd44103190 2024-02-19T21:02:33.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1351714913 0 0) 0x7fbd44119830 con 0x7fbd440ff650 2024-02-19T21:02:33.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd44117450 con 0x7fbd440ff650 2024-02-19T21:02:33.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbd30001840 con 0x7fbd440ff650 2024-02-19T21:02:33.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2851259001 0 0) 0x7fbd44118630 con 0x7fbd440fc8b0 2024-02-19T21:02:33.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 >> v1:172.21.15.163:6790/0 conn(0x7fbd44103190 legacy=0x7fbd44115b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:33.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 >> v1:172.21.15.193:6789/0 conn(0x7fbd440ff650 legacy=0x7fbd440fa310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:33.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4411aa30 con 0x7fbd440fc8b0 2024-02-19T21:02:33.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd4acb4700 1 -- 172.21.15.163:0/4128155926 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd44119a20 con 0x7fbd440fc8b0 2024-02-19T21:02:33.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fbd380024d0 con 0x7fbd440fc8b0 2024-02-19T21:02:33.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbd38005c00 con 0x7fbd440fc8b0 2024-02-19T21:02:33.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd4acb4700 10 client.14670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:33.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd4acb4700 1 -- 172.21.15.163:0/4128155926 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd44119e90 con 0x7fbd440fc8b0 2024-02-19T21:02:33.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.746+0000 7fbd4acb4700 1 librados: init done 2024-02-19T21:02:33.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd4acb4700 10 client.14670.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:02:33.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd4acb4700 15 client.14670.objecter _session_command_op_assign -1 1 2024-02-19T21:02:33.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd4acb4700 20 client.14670.objecter _get_session osd=-1 returning homeless 2024-02-19T21:02:33.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd4acb4700 10 client.14670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:33.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd4acb4700 1 -- 172.21.15.163:0/4128155926 --> v1:172.21.15.163:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbd4405f700 con 0x7fbd440fc8b0 2024-02-19T21:02:33.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fbd38018b50 con 0x7fbd440fc8b0 2024-02-19T21:02:33.741 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:02:33] "GET /metrics HTTP/1.1" 200 172148 "" "Prometheus/2.33.4" 2024-02-19T21:02:33.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(78..78 src has 1..78) v4 ==== 6172+0+0 (unknown 1950335092 0 0) 0x7fbd38078700 con 0x7fbd440fc8b0 2024-02-19T21:02:33.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 10 client.14670.objecter ms_dispatch 0x7fbd44001550 osd_map(78..78 src has 1..78) v4 2024-02-19T21:02:33.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 3 client.14670.objecter handle_osd_map got epochs [78,78] > 0 2024-02-19T21:02:33.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 3 client.14670.objecter handle_osd_map decoding full epoch 78 2024-02-19T21:02:33.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 10 client.14670.objecter checking command 1 2024-02-19T21:02:33.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 20 client.14670.objecter _get_session s=0x7fbd34072200 osd=4 3 2024-02-19T21:02:33.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 20 client.14670.objecter put_session s=0x7fbd34072200 osd=4 3 2024-02-19T21:02:33.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 15 client.14670.objecter _session_command_op_remove -1 1 2024-02-19T21:02:33.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 20 client.14670.objecter _get_session s=0x7fbd34072200 osd=4 3 2024-02-19T21:02:33.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 20 client.14670.objecter get_session s=0x7fbd34072200 osd=4 3 2024-02-19T21:02:33.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 15 client.14670.objecter _session_command_op_assign 4 1 2024-02-19T21:02:33.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 20 client.14670.objecter put_session s=0x7fbd34072200 osd=4 4 2024-02-19T21:02:33.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 10 client.14670.objecter _send_command 1 2024-02-19T21:02:33.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 --> v1:172.21.15.193:6800/3129716965 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd340765c0 con 0x7fbd340728b0 2024-02-19T21:02:33.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 20 client.14670.objecter dump_active .. 0 homeless 2024-02-19T21:02:33.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.748+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (unknown 100594580 0 0) 0x7fbd38078b30 con 0x7fbd440fc8b0 2024-02-19T21:02:33.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.749+0000 7fbd41ffb700 10 client.14670.objecter ms_handle_connect 0x7fbd340680f0 2024-02-19T21:02:33.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.749+0000 7fbd41ffb700 10 client.14670.objecter ms_handle_connect 0x7fbd340728b0 2024-02-19T21:02:33.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.750+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbd340765c0 con 0x7fbd340728b0 2024-02-19T21:02:33.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.750+0000 7fbd41ffb700 10 client.14670.objecter ms_dispatch 0x7fbd44001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:02:33.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.750+0000 7fbd41ffb700 10 client.14670.objecter _finish_command 1 = system:0 2024-02-19T21:02:33.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.750+0000 7fbd41ffb700 20 client.14670.objecter put_session s=0x7fbd34072200 osd=4 4 2024-02-19T21:02:33.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.750+0000 7fbd41ffb700 15 client.14670.objecter _session_command_op_remove 4 1 2024-02-19T21:02:33.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd4acb4700 10 client.14670.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:02:33.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd4acb4700 15 client.14670.objecter _session_command_op_assign -1 2 2024-02-19T21:02:33.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd4acb4700 20 client.14670.objecter _get_session s=0x7fbd34072200 osd=4 3 2024-02-19T21:02:33.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd4acb4700 20 client.14670.objecter put_session s=0x7fbd34072200 osd=4 3 2024-02-19T21:02:33.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd4acb4700 20 client.14670.objecter _get_session s=0x7fbd34072200 osd=4 3 2024-02-19T21:02:33.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd4acb4700 15 client.14670.objecter _session_command_op_remove -1 2 2024-02-19T21:02:33.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd4acb4700 20 client.14670.objecter get_session s=0x7fbd34072200 osd=4 3 2024-02-19T21:02:33.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd4acb4700 15 client.14670.objecter _session_command_op_assign 4 2 2024-02-19T21:02:33.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd4acb4700 20 client.14670.objecter put_session s=0x7fbd34072200 osd=4 4 2024-02-19T21:02:33.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd4acb4700 10 client.14670.objecter _send_command 2 2024-02-19T21:02:33.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd4acb4700 1 -- 172.21.15.163:0/4128155926 --> v1:172.21.15.193:6800/3129716965 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbd44047540 con 0x7fbd340728b0 2024-02-19T21:02:33.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd41ffb700 1 -- 172.21.15.163:0/4128155926 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 3860663012) 0x7fbd44047540 con 0x7fbd340728b0 2024-02-19T21:02:33.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd41ffb700 10 client.14670.objecter ms_dispatch 0x7fbd44001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:02:33.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd41ffb700 10 client.14670.objecter _finish_command 2 = system:0 2024-02-19T21:02:33.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd41ffb700 20 client.14670.objecter put_session s=0x7fbd34072200 osd=4 4 2024-02-19T21:02:33.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.766+0000 7fbd41ffb700 15 client.14670.objecter _session_command_op_remove 4 2 2024-02-19T21:02:33.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.767+0000 7fbd4acb4700 10 librados: watch_flush enter 2024-02-19T21:02:33.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.767+0000 7fbd4acb4700 10 librados: watch_flush exit 2024-02-19T21:02:33.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.767+0000 7fbd4acb4700 10 client.14670.objecter close_session for osd.4 2024-02-19T21:02:33.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.767+0000 7fbd4acb4700 1 -- 172.21.15.163:0/4128155926 >> v1:172.21.15.193:6800/3129716965 conn(0x7fbd340728b0 legacy=0x7fbd34074d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:33.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.767+0000 7fbd4acb4700 20 client.14670.objecter put_session s=0x7fbd34072200 osd=4 1 2024-02-19T21:02:33.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.767+0000 7fbd4acb4700 20 client.14670.objecter shutdown clearing up homeless session... 2024-02-19T21:02:33.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.767+0000 7fbd4acb4700 10 client.14670.objecter successfully canceled tick 2024-02-19T21:02:33.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.767+0000 7fbd4acb4700 1 -- 172.21.15.163:0/4128155926 >> v1:172.21.15.163:6800/1552514681 conn(0x7fbd340680f0 legacy=0x7fbd3406a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:33.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.767+0000 7fbd4acb4700 1 -- 172.21.15.163:0/4128155926 >> v1:172.21.15.163:6789/0 conn(0x7fbd440fc8b0 legacy=0x7fbd440f9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:33.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.767+0000 7fbd49251700 1 -- 172.21.15.163:0/4128155926 reap_dead start 2024-02-19T21:02:33.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.767+0000 7fbd4acb4700 1 -- 172.21.15.163:0/4128155926 shutdown_connections 2024-02-19T21:02:33.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.767+0000 7fbd4acb4700 1 -- 172.21.15.163:0/4128155926 >> 172.21.15.163:0/4128155926 conn(0x7fbd4406fdc0 msgr2=0x7fbd440f4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:33.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.767+0000 7fbd4acb4700 1 -- 172.21.15.163:0/4128155926 shutdown_connections 2024-02-19T21:02:33.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.767+0000 7fbd4acb4700 1 -- 172.21.15.163:0/4128155926 wait complete. 2024-02-19T21:02:33.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:33.767+0000 7fbd4acb4700 1 librados: shutdown 2024-02-19T21:02:34.416 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:32.485553+0000 osd.7 (osd.7) 11 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:02:34.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:32.488580+0000 osd.7 (osd.7) 12 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:02:34.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:32.775831+0000 osd.6 (osd.6) 13 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:02:34.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:32.777822+0000 osd.6 (osd.6) 14 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:02:34.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:32.923446+0000 mgr.y (mgr.14550) 48 : cluster [DBG] pgmap v36: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 867 B/s rd, 0 op/s 2024-02-19T21:02:34.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:33.556235+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:34.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:33.558264+0000 mon.b (mon.1) 62 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:02:34.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:33.559015+0000 mon.b (mon.1) 63 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:02:34.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:33.563364+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:34.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:02:33.566702+0000 mgr.y (mgr.14550) 49 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-19T21:02:34.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:02:33.569439+0000 mgr.y (mgr.14550) 50 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi163 2024-02-19T21:02:34.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:32.485553+0000 osd.7 (osd.7) 11 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:02:34.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:32.488580+0000 osd.7 (osd.7) 12 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:02:34.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:32.775831+0000 osd.6 (osd.6) 13 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:02:34.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:32.777822+0000 osd.6 (osd.6) 14 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:02:34.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:32.923446+0000 mgr.y (mgr.14550) 48 : cluster [DBG] pgmap v36: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 867 B/s rd, 0 op/s 2024-02-19T21:02:34.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:33.556235+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:34.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:33.558264+0000 mon.b (mon.1) 62 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:02:34.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:33.559015+0000 mon.b (mon.1) 63 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:02:34.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:33.563364+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:34.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:02:33.566702+0000 mgr.y (mgr.14550) 49 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-19T21:02:34.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:02:33.569439+0000 mgr.y (mgr.14550) 50 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi163 2024-02-19T21:02:34.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:32.485553+0000 osd.7 (osd.7) 11 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:02:34.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:32.488580+0000 osd.7 (osd.7) 12 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:02:34.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:32.775831+0000 osd.6 (osd.6) 13 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:02:34.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:32.777822+0000 osd.6 (osd.6) 14 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:02:34.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:32.923446+0000 mgr.y (mgr.14550) 48 : cluster [DBG] pgmap v36: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 867 B/s rd, 0 op/s 2024-02-19T21:02:34.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:33.556235+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:34.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:33.558264+0000 mon.b (mon.1) 62 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:02:34.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:33.559015+0000 mon.b (mon.1) 63 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:02:34.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:33.563364+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:34.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:02:33.566702+0000 mgr.y (mgr.14550) 49 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-02-19T21:02:34.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:02:33.569439+0000 mgr.y (mgr.14550) 50 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi163 2024-02-19T21:02:34.932 INFO:teuthology.orchestra.run.smithi163.stdout:133143986209 2024-02-19T21:02:34.933 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-19T21:02:35.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:33.965292+0000 osd.3 (osd.3) 15 : cluster [DBG] 4.17 deep-scrub starts 2024-02-19T21:02:35.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:33.968176+0000 osd.3 (osd.3) 16 : cluster [DBG] 4.17 deep-scrub ok 2024-02-19T21:02:35.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:34.273485+0000 osd.2 (osd.2) 15 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:02:35.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:34.274647+0000 osd.2 (osd.2) 16 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:02:35.238 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:33.965292+0000 osd.3 (osd.3) 15 : cluster [DBG] 4.17 deep-scrub starts 2024-02-19T21:02:35.239 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:33.968176+0000 osd.3 (osd.3) 16 : cluster [DBG] 4.17 deep-scrub ok 2024-02-19T21:02:35.239 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:34.273485+0000 osd.2 (osd.2) 15 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:02:35.239 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:34.274647+0000 osd.2 (osd.2) 16 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:02:35.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:33.965292+0000 osd.3 (osd.3) 15 : cluster [DBG] 4.17 deep-scrub starts 2024-02-19T21:02:35.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:33.968176+0000 osd.3 (osd.3) 16 : cluster [DBG] 4.17 deep-scrub ok 2024-02-19T21:02:35.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:34.273485+0000 osd.2 (osd.2) 15 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:02:35.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:34.274647+0000 osd.2 (osd.2) 16 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:02:36.155 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:34.924122+0000 mgr.y (mgr.14550) 51 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:36.156 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:35.249644+0000 osd.2 (osd.2) 17 : cluster [DBG] 5.8 deep-scrub starts 2024-02-19T21:02:36.156 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:35.252564+0000 osd.2 (osd.2) 18 : cluster [DBG] 5.8 deep-scrub ok 2024-02-19T21:02:36.156 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:35.500774+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:02:36.156 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:35.502902+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:02:36.156 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:34.924122+0000 mgr.y (mgr.14550) 51 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:36.156 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:35.249644+0000 osd.2 (osd.2) 17 : cluster [DBG] 5.8 deep-scrub starts 2024-02-19T21:02:36.156 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:35.252564+0000 osd.2 (osd.2) 18 : cluster [DBG] 5.8 deep-scrub ok 2024-02-19T21:02:36.156 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:35.500774+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:02:36.156 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:35.502902+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:02:36.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:34.924122+0000 mgr.y (mgr.14550) 51 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:36.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:35.249644+0000 osd.2 (osd.2) 17 : cluster [DBG] 5.8 deep-scrub starts 2024-02-19T21:02:36.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:35.252564+0000 osd.2 (osd.2) 18 : cluster [DBG] 5.8 deep-scrub ok 2024-02-19T21:02:36.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:35.500774+0000 osd.0 (osd.0) 15 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:02:36.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:35.502902+0000 osd.0 (osd.0) 16 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:02:37.303 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:37.488 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:37 smithi163 systemd[1]: Stopping Ceph alertmanager.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:02:37.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:35.543599+0000 osd.7 (osd.7) 13 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:02:37.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:35.546522+0000 osd.7 (osd.7) 14 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:02:37.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:36.147369+0000 osd.5 (osd.5) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:37.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:36.147446+0000 osd.5 (osd.5) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:37.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:36.161975+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-02-19T21:02:37.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:35.543599+0000 osd.7 (osd.7) 13 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:02:37.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:35.546522+0000 osd.7 (osd.7) 14 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:02:37.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:36.147369+0000 osd.5 (osd.5) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:37.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:36.147446+0000 osd.5 (osd.5) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:37.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:36.161975+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-02-19T21:02:37.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:35.543599+0000 osd.7 (osd.7) 13 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:02:37.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:35.546522+0000 osd.7 (osd.7) 14 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:02:37.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:36.147369+0000 osd.5 (osd.5) 9 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:37.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:36.147446+0000 osd.5 (osd.5) 10 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:37.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:36.161975+0000 mon.a (mon.0) 689 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-02-19T21:02:37.988 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:37 smithi163 bash[71724]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager.a" found: no such container 2024-02-19T21:02:38.061 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:02:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:02:38.488 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[67708]: level=info ts=2024-02-19T21:02:38.254Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-19T21:02:38.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:36.581274+0000 osd.7 (osd.7) 15 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:02:38.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:36.584996+0000 osd.7 (osd.7) 16 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:02:38.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:36.924551+0000 mgr.y (mgr.14550) 52 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:38.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:37.257653+0000 osd.2 (osd.2) 19 : cluster [DBG] 2.e deep-scrub starts 2024-02-19T21:02:38.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:37.258577+0000 osd.2 (osd.2) 20 : cluster [DBG] 2.e deep-scrub ok 2024-02-19T21:02:38.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:37.964082+0000 mon.a (mon.0) 690 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-02-19T21:02:38.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:36.581274+0000 osd.7 (osd.7) 15 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:02:38.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:36.584996+0000 osd.7 (osd.7) 16 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:02:38.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:36.924551+0000 mgr.y (mgr.14550) 52 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:38.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:37.257653+0000 osd.2 (osd.2) 19 : cluster [DBG] 2.e deep-scrub starts 2024-02-19T21:02:38.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:37.258577+0000 osd.2 (osd.2) 20 : cluster [DBG] 2.e deep-scrub ok 2024-02-19T21:02:38.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:37.964082+0000 mon.a (mon.0) 690 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-02-19T21:02:38.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:36.581274+0000 osd.7 (osd.7) 15 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:02:38.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:36.584996+0000 osd.7 (osd.7) 16 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:02:38.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:36.924551+0000 mgr.y (mgr.14550) 52 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:38.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:37.257653+0000 osd.2 (osd.2) 19 : cluster [DBG] 2.e deep-scrub starts 2024-02-19T21:02:38.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:37.258577+0000 osd.2 (osd.2) 20 : cluster [DBG] 2.e deep-scrub ok 2024-02-19T21:02:38.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:37.964082+0000 mon.a (mon.0) 690 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-02-19T21:02:39.488 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:39 smithi163 bash[71802]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a 2024-02-19T21:02:39.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:37.841285+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:02:39.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:37.842537+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:02:39.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:37.841285+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:02:39.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:37.842537+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:02:39.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:37.841285+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:02:39.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:37.842537+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:02:39.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.917+0000 7fd6efae3700 1 -- 172.21.15.163:0/3245240054 >> v1:172.21.15.163:6790/0 conn(0x7fd6e80fc880 legacy=0x7fd6e80fcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:39.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.917+0000 7fd6efae3700 1 -- 172.21.15.163:0/3245240054 shutdown_connections 2024-02-19T21:02:39.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.917+0000 7fd6efae3700 1 -- 172.21.15.163:0/3245240054 >> 172.21.15.163:0/3245240054 conn(0x7fd6e80f24a0 msgr2=0x7fd6e80f48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:39.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.917+0000 7fd6efae3700 1 -- 172.21.15.163:0/3245240054 shutdown_connections 2024-02-19T21:02:39.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.917+0000 7fd6efae3700 1 -- 172.21.15.163:0/3245240054 wait complete. 2024-02-19T21:02:39.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.917+0000 7fd6efae3700 1 librados: starting msgr at 2024-02-19T21:02:39.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.917+0000 7fd6efae3700 1 librados: starting objecter 2024-02-19T21:02:39.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.917+0000 7fd6efae3700 1 Processor -- start 2024-02-19T21:02:39.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6efae3700 1 -- start start 2024-02-19T21:02:39.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6efae3700 1 librados: setting wanted keys 2024-02-19T21:02:39.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6efae3700 1 librados: calling monclient init 2024-02-19T21:02:39.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6efae3700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd6e81172a0 con 0x7fd6e81031c0 2024-02-19T21:02:39.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6efae3700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd6e8118480 con 0x7fd6e80fc880 2024-02-19T21:02:39.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6efae3700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd6e8119680 con 0x7fd6e80ff680 2024-02-19T21:02:39.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6ed07e700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fd6e80ff680 0x7fd6e80fa140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:34506/0 (socket says 172.21.15.163:34506) 2024-02-19T21:02:39.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6ed07e700 1 -- 172.21.15.163:0/3110542565 learned_addr learned my addr 172.21.15.163:0/3110542565 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:39.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 10 client.?.objecter ms_handle_connect 0x7fd6e80ff680 2024-02-19T21:02:39.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:39.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 10 client.?.objecter ms_handle_connect 0x7fd6e81031c0 2024-02-19T21:02:39.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:39.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 10 client.?.objecter ms_handle_connect 0x7fd6e80fc880 2024-02-19T21:02:39.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:39.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818195445 0 0) 0x7fd6e8119680 con 0x7fd6e80ff680 2024-02-19T21:02:39.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6d40038e0 con 0x7fd6e80ff680 2024-02-19T21:02:39.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438924891 0 0) 0x7fd6e81172a0 con 0x7fd6e81031c0 2024-02-19T21:02:39.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6e8119680 con 0x7fd6e81031c0 2024-02-19T21:02:39.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064105123 0 0) 0x7fd6e8118480 con 0x7fd6e80fc880 2024-02-19T21:02:39.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6e81172a0 con 0x7fd6e80fc880 2024-02-19T21:02:39.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 470511444 0 0) 0x7fd6d40038e0 con 0x7fd6e80ff680 2024-02-19T21:02:39.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6e8118480 con 0x7fd6e80ff680 2024-02-19T21:02:39.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2836176000 0 0) 0x7fd6e8119680 con 0x7fd6e81031c0 2024-02-19T21:02:39.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6d40038e0 con 0x7fd6e81031c0 2024-02-19T21:02:39.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd6d80019f0 con 0x7fd6e80ff680 2024-02-19T21:02:39.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd6e0001b00 con 0x7fd6e81031c0 2024-02-19T21:02:39.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 589118447 0 0) 0x7fd6e81172a0 con 0x7fd6e80fc880 2024-02-19T21:02:39.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6e8119680 con 0x7fd6e80fc880 2024-02-19T21:02:39.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.918+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd6e4002c90 con 0x7fd6e80fc880 2024-02-19T21:02:39.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.919+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4122523739 0 0) 0x7fd6e8118480 con 0x7fd6e80ff680 2024-02-19T21:02:39.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.919+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 >> v1:172.21.15.193:6789/0 conn(0x7fd6e80fc880 legacy=0x7fd6e80f9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:39.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.919+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 >> v1:172.21.15.163:6789/0 conn(0x7fd6e81031c0 legacy=0x7fd6e8115960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:39.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.919+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e811a880 con 0x7fd6e80ff680 2024-02-19T21:02:39.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.919+0000 7fd6efae3700 1 -- 172.21.15.163:0/3110542565 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e8117490 con 0x7fd6e80ff680 2024-02-19T21:02:39.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.919+0000 7fd6efae3700 10 client.34601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:39.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.919+0000 7fd6efae3700 1 -- 172.21.15.163:0/3110542565 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e8117980 con 0x7fd6e80ff680 2024-02-19T21:02:39.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.919+0000 7fd6efae3700 1 librados: init done 2024-02-19T21:02:39.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.919+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fd6d8002480 con 0x7fd6e80ff680 2024-02-19T21:02:39.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.919+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd6d8005ac0 con 0x7fd6e80ff680 2024-02-19T21:02:39.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.919+0000 7fd6efae3700 10 client.34601.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:02:39.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.919+0000 7fd6efae3700 15 client.34601.objecter _session_command_op_assign -1 1 2024-02-19T21:02:39.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.919+0000 7fd6efae3700 20 client.34601.objecter _get_session osd=-1 returning homeless 2024-02-19T21:02:39.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.919+0000 7fd6efae3700 10 client.34601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:39.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.919+0000 7fd6efae3700 1 -- 172.21.15.163:0/3110542565 --> v1:172.21.15.163:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd6cc0028b0 con 0x7fd6e80ff680 2024-02-19T21:02:39.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.920+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fd6d8001fe0 con 0x7fd6e80ff680 2024-02-19T21:02:39.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.920+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(80..80 src has 1..80) v4 ==== 6172+0+0 (unknown 3380790453 0 0) 0x7fd6d8078550 con 0x7fd6e80ff680 2024-02-19T21:02:39.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.920+0000 7fd6def78700 10 client.34601.objecter ms_dispatch 0x7fd6e8001550 osd_map(80..80 src has 1..80) v4 2024-02-19T21:02:39.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.920+0000 7fd6def78700 3 client.34601.objecter handle_osd_map got epochs [80,80] > 0 2024-02-19T21:02:39.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.920+0000 7fd6def78700 3 client.34601.objecter handle_osd_map decoding full epoch 80 2024-02-19T21:02:39.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.920+0000 7fd6def78700 10 client.34601.objecter checking command 1 2024-02-19T21:02:39.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.921+0000 7fd6def78700 20 client.34601.objecter _get_session s=0x7fd6d406dff0 osd=5 3 2024-02-19T21:02:39.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.921+0000 7fd6def78700 20 client.34601.objecter put_session s=0x7fd6d406dff0 osd=5 3 2024-02-19T21:02:39.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.921+0000 7fd6def78700 15 client.34601.objecter _session_command_op_remove -1 1 2024-02-19T21:02:39.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.921+0000 7fd6def78700 20 client.34601.objecter _get_session s=0x7fd6d406dff0 osd=5 3 2024-02-19T21:02:39.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.921+0000 7fd6def78700 20 client.34601.objecter get_session s=0x7fd6d406dff0 osd=5 3 2024-02-19T21:02:39.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.921+0000 7fd6def78700 15 client.34601.objecter _session_command_op_assign 5 1 2024-02-19T21:02:39.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.921+0000 7fd6def78700 20 client.34601.objecter put_session s=0x7fd6d406dff0 osd=5 4 2024-02-19T21:02:39.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.921+0000 7fd6def78700 10 client.34601.objecter _send_command 1 2024-02-19T21:02:39.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.921+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 --> v1:172.21.15.193:6804/805161991 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6d40723b0 con 0x7fd6d406e6a0 2024-02-19T21:02:39.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.921+0000 7fd6def78700 20 client.34601.objecter dump_active .. 0 homeless 2024-02-19T21:02:39.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.921+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (unknown 3184342506 0 0) 0x7fd6d8078980 con 0x7fd6e80ff680 2024-02-19T21:02:39.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.921+0000 7fd6def78700 10 client.34601.objecter ms_handle_connect 0x7fd6d4063d50 2024-02-19T21:02:39.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.921+0000 7fd6def78700 10 client.34601.objecter ms_handle_connect 0x7fd6d406e6a0 2024-02-19T21:02:39.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.922+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fd6d40723b0 con 0x7fd6d406e6a0 2024-02-19T21:02:39.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.922+0000 7fd6def78700 10 client.34601.objecter ms_dispatch 0x7fd6e8001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:02:39.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.922+0000 7fd6def78700 10 client.34601.objecter _finish_command 1 = system:0 2024-02-19T21:02:39.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.922+0000 7fd6def78700 20 client.34601.objecter put_session s=0x7fd6d406dff0 osd=5 4 2024-02-19T21:02:39.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.922+0000 7fd6def78700 15 client.34601.objecter _session_command_op_remove 5 1 2024-02-19T21:02:39.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.935+0000 7fd6efae3700 10 client.34601.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:02:39.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.935+0000 7fd6efae3700 15 client.34601.objecter _session_command_op_assign -1 2 2024-02-19T21:02:39.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.935+0000 7fd6efae3700 20 client.34601.objecter _get_session s=0x7fd6d406dff0 osd=5 3 2024-02-19T21:02:39.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.935+0000 7fd6efae3700 20 client.34601.objecter put_session s=0x7fd6d406dff0 osd=5 3 2024-02-19T21:02:39.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.935+0000 7fd6efae3700 20 client.34601.objecter _get_session s=0x7fd6d406dff0 osd=5 3 2024-02-19T21:02:39.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.935+0000 7fd6efae3700 15 client.34601.objecter _session_command_op_remove -1 2 2024-02-19T21:02:39.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.935+0000 7fd6efae3700 20 client.34601.objecter get_session s=0x7fd6d406dff0 osd=5 3 2024-02-19T21:02:39.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.935+0000 7fd6efae3700 15 client.34601.objecter _session_command_op_assign 5 2 2024-02-19T21:02:39.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.935+0000 7fd6efae3700 20 client.34601.objecter put_session s=0x7fd6d406dff0 osd=5 4 2024-02-19T21:02:39.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.935+0000 7fd6efae3700 10 client.34601.objecter _send_command 2 2024-02-19T21:02:39.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.935+0000 7fd6efae3700 1 -- 172.21.15.163:0/3110542565 --> v1:172.21.15.193:6804/805161991 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd6cc001140 con 0x7fd6d406e6a0 2024-02-19T21:02:39.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.936+0000 7fd6def78700 1 -- 172.21.15.163:0/3110542565 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 1699381006) 0x7fd6cc001140 con 0x7fd6d406e6a0 2024-02-19T21:02:39.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.936+0000 7fd6def78700 10 client.34601.objecter ms_dispatch 0x7fd6e8001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:02:39.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.936+0000 7fd6def78700 10 client.34601.objecter _finish_command 2 = system:0 2024-02-19T21:02:39.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.936+0000 7fd6def78700 20 client.34601.objecter put_session s=0x7fd6d406dff0 osd=5 4 2024-02-19T21:02:39.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.936+0000 7fd6def78700 15 client.34601.objecter _session_command_op_remove 5 2 2024-02-19T21:02:39.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.936+0000 7fd6efae3700 10 librados: watch_flush enter 2024-02-19T21:02:39.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.936+0000 7fd6efae3700 10 librados: watch_flush exit 2024-02-19T21:02:39.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.936+0000 7fd6efae3700 10 client.34601.objecter close_session for osd.5 2024-02-19T21:02:39.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.936+0000 7fd6efae3700 1 -- 172.21.15.163:0/3110542565 >> v1:172.21.15.193:6804/805161991 conn(0x7fd6d406e6a0 legacy=0x7fd6d4070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:39.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.936+0000 7fd6efae3700 20 client.34601.objecter put_session s=0x7fd6d406dff0 osd=5 1 2024-02-19T21:02:39.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.936+0000 7fd6efae3700 20 client.34601.objecter shutdown clearing up homeless session... 2024-02-19T21:02:39.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.936+0000 7fd6efae3700 10 client.34601.objecter successfully canceled tick 2024-02-19T21:02:39.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.936+0000 7fd6efae3700 1 -- 172.21.15.163:0/3110542565 >> v1:172.21.15.163:6800/1552514681 conn(0x7fd6d4063d50 legacy=0x7fd6d4066210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:39.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.936+0000 7fd6efae3700 1 -- 172.21.15.163:0/3110542565 >> v1:172.21.15.163:6790/0 conn(0x7fd6e80ff680 legacy=0x7fd6e80fa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:39.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.936+0000 7fd6ee080700 1 -- 172.21.15.163:0/3110542565 reap_dead start 2024-02-19T21:02:39.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.937+0000 7fd6efae3700 1 -- 172.21.15.163:0/3110542565 shutdown_connections 2024-02-19T21:02:39.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.937+0000 7fd6efae3700 1 -- 172.21.15.163:0/3110542565 >> 172.21.15.163:0/3110542565 conn(0x7fd6e80f24a0 msgr2=0x7fd6e8061340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:39.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.937+0000 7fd6efae3700 1 -- 172.21.15.163:0/3110542565 shutdown_connections 2024-02-19T21:02:39.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.937+0000 7fd6efae3700 1 -- 172.21.15.163:0/3110542565 wait complete. 2024-02-19T21:02:39.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:39.937+0000 7fd6efae3700 1 librados: shutdown 2024-02-19T21:02:40.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:38.629984+0000 osd.7 (osd.7) 17 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:02:40.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:38.632946+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:02:40.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:38.925014+0000 mgr.y (mgr.14550) 53 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:02:40.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:39.091162+0000 osd.4 (osd.4) 21 : cluster [DBG] 6.17 deep-scrub starts 2024-02-19T21:02:40.238 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:39.092149+0000 osd.4 (osd.4) 22 : cluster [DBG] 6.17 deep-scrub ok 2024-02-19T21:02:40.239 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:39.440928+0000 osd.0 (osd.0) 17 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:02:40.239 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:39.442527+0000 osd.0 (osd.0) 18 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:02:40.239 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:38.629984+0000 osd.7 (osd.7) 17 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:02:40.239 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:38.632946+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:02:40.239 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:38.925014+0000 mgr.y (mgr.14550) 53 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:02:40.239 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:39.091162+0000 osd.4 (osd.4) 21 : cluster [DBG] 6.17 deep-scrub starts 2024-02-19T21:02:40.239 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:39.092149+0000 osd.4 (osd.4) 22 : cluster [DBG] 6.17 deep-scrub ok 2024-02-19T21:02:40.239 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:39.440928+0000 osd.0 (osd.0) 17 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:02:40.239 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:39.442527+0000 osd.0 (osd.0) 18 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:02:40.312 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:02:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:02:40.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:38.629984+0000 osd.7 (osd.7) 17 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:02:40.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:38.632946+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:02:40.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:38.925014+0000 mgr.y (mgr.14550) 53 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:02:40.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:39.091162+0000 osd.4 (osd.4) 21 : cluster [DBG] 6.17 deep-scrub starts 2024-02-19T21:02:40.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:39.092149+0000 osd.4 (osd.4) 22 : cluster [DBG] 6.17 deep-scrub ok 2024-02-19T21:02:40.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:39.440928+0000 osd.0 (osd.0) 17 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:02:40.312 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:39.442527+0000 osd.0 (osd.0) 18 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:02:40.713 INFO:teuthology.orchestra.run.smithi163.stdout:158913789982 2024-02-19T21:02:40.714 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-19T21:02:40.988 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:40 smithi163 systemd[1]: ceph-2591809a-cf69-11ee-95bb-87774f69a715@alertmanager.a.service: Succeeded. 2024-02-19T21:02:40.988 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:40 smithi163 systemd[1]: Stopped Ceph alertmanager.a for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T21:02:40.988 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:40 smithi163 systemd[1]: Starting Ceph alertmanager.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:02:41.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:39.777780+0000 osd.6 (osd.6) 17 : cluster [DBG] 4.a scrub starts 2024-02-19T21:02:41.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:39.779885+0000 osd.6 (osd.6) 18 : cluster [DBG] 4.a scrub ok 2024-02-19T21:02:41.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:39.950304+0000 osd.3 (osd.3) 17 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:02:41.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:39.952620+0000 osd.3 (osd.3) 18 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:02:41.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:39.964568+0000 mgr.y (mgr.14550) 54 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:41.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:40.103080+0000 osd.4 (osd.4) 23 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:02:41.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:40.105137+0000 osd.4 (osd.4) 24 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:02:41.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:39.777780+0000 osd.6 (osd.6) 17 : cluster [DBG] 4.a scrub starts 2024-02-19T21:02:41.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:39.779885+0000 osd.6 (osd.6) 18 : cluster [DBG] 4.a scrub ok 2024-02-19T21:02:41.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:39.950304+0000 osd.3 (osd.3) 17 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:02:41.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:39.952620+0000 osd.3 (osd.3) 18 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:02:41.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:39.964568+0000 mgr.y (mgr.14550) 54 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:41.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:40.103080+0000 osd.4 (osd.4) 23 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:02:41.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:40.105137+0000 osd.4 (osd.4) 24 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:02:41.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:39.777780+0000 osd.6 (osd.6) 17 : cluster [DBG] 4.a scrub starts 2024-02-19T21:02:41.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:39.779885+0000 osd.6 (osd.6) 18 : cluster [DBG] 4.a scrub ok 2024-02-19T21:02:41.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:39.950304+0000 osd.3 (osd.3) 17 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:02:41.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:39.952620+0000 osd.3 (osd.3) 18 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:02:41.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:39.964568+0000 mgr.y (mgr.14550) 54 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:41.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:40.103080+0000 osd.4 (osd.4) 23 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:02:41.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:40.105137+0000 osd.4 (osd.4) 24 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:02:41.636 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:41.768 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE Bus STOPPING 2024-02-19T21:02:41.769 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-19T21:02:41.769 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE Bus STOPPED 2024-02-19T21:02:41.769 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE Bus STARTING 2024-02-19T21:02:41.769 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE Serving on http://:::9283 2024-02-19T21:02:41.769 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE Bus STARTED 2024-02-19T21:02:41.769 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=info ts=2024-02-19T21:02:41.504Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-19T21:02:41.769 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=info ts=2024-02-19T21:02:41.505Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-19T21:02:41.769 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=info ts=2024-02-19T21:02:41.505Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.163 port=9094 2024-02-19T21:02:41.769 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=info ts=2024-02-19T21:02:41.507Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-19T21:02:41.769 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=info ts=2024-02-19T21:02:41.532Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-19T21:02:41.770 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=info ts=2024-02-19T21:02:41.532Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-19T21:02:41.770 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=info ts=2024-02-19T21:02:41.534Z caller=main.go:518 msg=Listening address=:9093 2024-02-19T21:02:41.770 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=info ts=2024-02-19T21:02:41.534Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-19T21:02:41.770 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:41 smithi163 bash[72045]: f071782bcdbb014f3cf8817d7860f8974d389ac7bffc4ceb8f62889e53148744 2024-02-19T21:02:41.770 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:41 smithi163 systemd[1]: Started Ceph alertmanager.a for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T21:02:42.140 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE Bus STOPPING 2024-02-19T21:02:42.140 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-19T21:02:42.140 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE Bus STOPPED 2024-02-19T21:02:42.140 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE Bus STARTING 2024-02-19T21:02:42.141 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE Serving on http://:::9283 2024-02-19T21:02:42.141 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE Bus STARTED 2024-02-19T21:02:42.141 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE Bus STOPPING 2024-02-19T21:02:42.141 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-19T21:02:42.141 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE Bus STOPPED 2024-02-19T21:02:42.141 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:41] ENGINE Bus STARTING 2024-02-19T21:02:42.141 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:42] ENGINE Serving on http://:::9283 2024-02-19T21:02:42.141 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:02:42] ENGINE Bus STARTED 2024-02-19T21:02:42.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:40.597390+0000 osd.1 (osd.1) 19 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:02:42.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:40.600405+0000 osd.1 (osd.1) 20 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:02:42.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:40.625164+0000 osd.7 (osd.7) 19 : cluster [DBG] 2.b scrub starts 2024-02-19T21:02:42.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:40.628627+0000 osd.7 (osd.7) 20 : cluster [DBG] 2.b scrub ok 2024-02-19T21:02:42.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:40.925697+0000 mgr.y (mgr.14550) 55 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:42.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:41.192040+0000 osd.5 (osd.5) 11 : cluster [DBG] 6.d scrub starts 2024-02-19T21:02:42.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:41.194987+0000 osd.5 (osd.5) 12 : cluster [DBG] 6.d scrub ok 2024-02-19T21:02:42.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.639619+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:42.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.644464+0000 mon.b (mon.1) 64 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T21:02:42.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.646683+0000 mon.b (mon.1) 65 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi163.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T21:02:42.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.652318+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:42.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.658439+0000 mon.b (mon.1) 66 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T21:02:42.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.661857+0000 mon.b (mon.1) 67 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi193.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T21:02:42.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.667002+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:42.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.671504+0000 mon.b (mon.1) 68 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T21:02:42.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.674824+0000 mon.b (mon.1) 69 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi193.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T21:02:42.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.680245+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:42.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.686972+0000 mon.b (mon.1) 70 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:02:42.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:40.597390+0000 osd.1 (osd.1) 19 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:02:42.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:40.600405+0000 osd.1 (osd.1) 20 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:02:42.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:40.625164+0000 osd.7 (osd.7) 19 : cluster [DBG] 2.b scrub starts 2024-02-19T21:02:42.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:40.628627+0000 osd.7 (osd.7) 20 : cluster [DBG] 2.b scrub ok 2024-02-19T21:02:42.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:40.925697+0000 mgr.y (mgr.14550) 55 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:42.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:41.192040+0000 osd.5 (osd.5) 11 : cluster [DBG] 6.d scrub starts 2024-02-19T21:02:42.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:41.194987+0000 osd.5 (osd.5) 12 : cluster [DBG] 6.d scrub ok 2024-02-19T21:02:42.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.639619+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:42.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.644464+0000 mon.b (mon.1) 64 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T21:02:42.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.646683+0000 mon.b (mon.1) 65 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi163.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T21:02:42.477 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.652318+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:42.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.658439+0000 mon.b (mon.1) 66 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T21:02:42.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.661857+0000 mon.b (mon.1) 67 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi193.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T21:02:42.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.667002+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:42.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.671504+0000 mon.b (mon.1) 68 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T21:02:42.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.674824+0000 mon.b (mon.1) 69 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi193.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T21:02:42.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.680245+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:42.478 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.686972+0000 mon.b (mon.1) 70 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:02:42.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:40.597390+0000 osd.1 (osd.1) 19 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:02:42.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:40.600405+0000 osd.1 (osd.1) 20 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:02:42.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:40.625164+0000 osd.7 (osd.7) 19 : cluster [DBG] 2.b scrub starts 2024-02-19T21:02:42.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:40.628627+0000 osd.7 (osd.7) 20 : cluster [DBG] 2.b scrub ok 2024-02-19T21:02:42.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:40.925697+0000 mgr.y (mgr.14550) 55 : cluster [DBG] pgmap v42: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:42.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:41.192040+0000 osd.5 (osd.5) 11 : cluster [DBG] 6.d scrub starts 2024-02-19T21:02:42.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:41.194987+0000 osd.5 (osd.5) 12 : cluster [DBG] 6.d scrub ok 2024-02-19T21:02:42.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.639619+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:42.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.644464+0000 mon.b (mon.1) 64 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T21:02:42.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.646683+0000 mon.b (mon.1) 65 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi163.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T21:02:42.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.652318+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:42.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.658439+0000 mon.b (mon.1) 66 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T21:02:42.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.661857+0000 mon.b (mon.1) 67 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi193.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T21:02:42.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.667002+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:42.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.671504+0000 mon.b (mon.1) 68 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T21:02:42.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.674824+0000 mon.b (mon.1) 69 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi193.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T21:02:42.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.680245+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:42.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.686972+0000 mon.b (mon.1) 70 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:02:43.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.234+0000 7f6f26693700 1 -- 172.21.15.163:0/1952109800 >> v1:172.21.15.163:6790/0 conn(0x7f6f201042c0 legacy=0x7f6f20106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:43.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.234+0000 7f6f26693700 1 -- 172.21.15.163:0/1952109800 shutdown_connections 2024-02-19T21:02:43.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.234+0000 7f6f26693700 1 -- 172.21.15.163:0/1952109800 >> 172.21.15.163:0/1952109800 conn(0x7f6f200f35a0 msgr2=0x7f6f200f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:43.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.234+0000 7f6f26693700 1 -- 172.21.15.163:0/1952109800 shutdown_connections 2024-02-19T21:02:43.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.234+0000 7f6f26693700 1 -- 172.21.15.163:0/1952109800 wait complete. 2024-02-19T21:02:43.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.235+0000 7f6f26693700 1 librados: starting msgr at 2024-02-19T21:02:43.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.235+0000 7f6f26693700 1 librados: starting objecter 2024-02-19T21:02:43.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.235+0000 7f6f26693700 1 Processor -- start 2024-02-19T21:02:43.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.235+0000 7f6f26693700 1 -- start start 2024-02-19T21:02:43.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.235+0000 7f6f26693700 1 librados: setting wanted keys 2024-02-19T21:02:43.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.235+0000 7f6f26693700 1 librados: calling monclient init 2024-02-19T21:02:43.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.235+0000 7f6f26693700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6f201171e0 con 0x7f6f200fd980 2024-02-19T21:02:43.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.235+0000 7f6f26693700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6f201183c0 con 0x7f6f201042c0 2024-02-19T21:02:43.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.235+0000 7f6f26693700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6f201195c0 con 0x7f6f20100780 2024-02-19T21:02:43.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.235+0000 7f6f1f7fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f6f20100780 0x7f6f200f9eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:34524/0 (socket says 172.21.15.163:34524) 2024-02-19T21:02:43.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.235+0000 7f6f1ffff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f6f200fd980 0x7f6f200f97a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:39692/0 (socket says 172.21.15.163:39692) 2024-02-19T21:02:43.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.235+0000 7f6f1f7fe700 1 -- 172.21.15.163:0/2361898336 learned_addr learned my addr 172.21.15.163:0/2361898336 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:43.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 10 client.?.objecter ms_handle_connect 0x7f6f20100780 2024-02-19T21:02:43.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:43.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 10 client.?.objecter ms_handle_connect 0x7f6f200fd980 2024-02-19T21:02:43.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:43.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 10 client.?.objecter ms_handle_connect 0x7f6f201042c0 2024-02-19T21:02:43.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:43.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288150292 0 0) 0x7f6f201195c0 con 0x7f6f20100780 2024-02-19T21:02:43.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f0c003030 con 0x7f6f20100780 2024-02-19T21:02:43.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440543927 0 0) 0x7f6f201171e0 con 0x7f6f200fd980 2024-02-19T21:02:43.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f201195c0 con 0x7f6f200fd980 2024-02-19T21:02:43.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512371811 0 0) 0x7f6f201183c0 con 0x7f6f201042c0 2024-02-19T21:02:43.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f201171e0 con 0x7f6f201042c0 2024-02-19T21:02:43.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4077563393 0 0) 0x7f6f201195c0 con 0x7f6f200fd980 2024-02-19T21:02:43.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f201183c0 con 0x7f6f200fd980 2024-02-19T21:02:43.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6f10001970 con 0x7f6f200fd980 2024-02-19T21:02:43.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 263964509 0 0) 0x7f6f0c003030 con 0x7f6f20100780 2024-02-19T21:02:43.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f201195c0 con 0x7f6f20100780 2024-02-19T21:02:43.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 18344385 0 0) 0x7f6f201171e0 con 0x7f6f201042c0 2024-02-19T21:02:43.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f0c003030 con 0x7f6f201042c0 2024-02-19T21:02:43.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6f080018c0 con 0x7f6f20100780 2024-02-19T21:02:43.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6f18003190 con 0x7f6f201042c0 2024-02-19T21:02:43.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1566919150 0 0) 0x7f6f201183c0 con 0x7f6f200fd980 2024-02-19T21:02:43.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 >> v1:172.21.15.163:6790/0 conn(0x7f6f20100780 legacy=0x7f6f200f9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:43.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 >> v1:172.21.15.193:6789/0 conn(0x7f6f201042c0 legacy=0x7f6f20115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:43.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2011a7c0 con 0x7f6f200fd980 2024-02-19T21:02:43.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f26693700 1 -- 172.21.15.163:0/2361898336 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f201173d0 con 0x7f6f200fd980 2024-02-19T21:02:43.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f26693700 10 client.14682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:43.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f26693700 1 -- 172.21.15.163:0/2361898336 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f201178c0 con 0x7f6f200fd980 2024-02-19T21:02:43.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f26693700 1 librados: init done 2024-02-19T21:02:43.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6f100022e0 con 0x7f6f200fd980 2024-02-19T21:02:43.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.236+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6f100059e0 con 0x7f6f200fd980 2024-02-19T21:02:43.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.237+0000 7f6f26693700 10 client.14682.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:02:43.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.237+0000 7f6f26693700 15 client.14682.objecter _session_command_op_assign -1 1 2024-02-19T21:02:43.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.237+0000 7f6f26693700 20 client.14682.objecter _get_session osd=-1 returning homeless 2024-02-19T21:02:43.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.237+0000 7f6f26693700 10 client.14682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:43.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.237+0000 7f6f26693700 1 -- 172.21.15.163:0/2361898336 --> v1:172.21.15.163:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6f000028b0 con 0x7f6f200fd980 2024-02-19T21:02:43.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6f10002450 con 0x7f6f200fd980 2024-02-19T21:02:43.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(80..80 src has 1..80) v4 ==== 6172+0+0 (unknown 3380790453 0 0) 0x7f6f10078410 con 0x7f6f200fd980 2024-02-19T21:02:43.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 10 client.14682.objecter ms_dispatch 0x7f6f20001550 osd_map(80..80 src has 1..80) v4 2024-02-19T21:02:43.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 3 client.14682.objecter handle_osd_map got epochs [80,80] > 0 2024-02-19T21:02:43.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 3 client.14682.objecter handle_osd_map decoding full epoch 80 2024-02-19T21:02:43.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 10 client.14682.objecter checking command 1 2024-02-19T21:02:43.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 20 client.14682.objecter _get_session s=0x7f6f0c06df80 osd=6 3 2024-02-19T21:02:43.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 20 client.14682.objecter put_session s=0x7f6f0c06df80 osd=6 3 2024-02-19T21:02:43.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 15 client.14682.objecter _session_command_op_remove -1 1 2024-02-19T21:02:43.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 20 client.14682.objecter _get_session s=0x7f6f0c06df80 osd=6 3 2024-02-19T21:02:43.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 20 client.14682.objecter get_session s=0x7f6f0c06df80 osd=6 3 2024-02-19T21:02:43.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 15 client.14682.objecter _session_command_op_assign 6 1 2024-02-19T21:02:43.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 20 client.14682.objecter put_session s=0x7f6f0c06df80 osd=6 4 2024-02-19T21:02:43.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 10 client.14682.objecter _send_command 1 2024-02-19T21:02:43.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 --> v1:172.21.15.193:6808/787604615 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f0c072340 con 0x7f6f0c06e630 2024-02-19T21:02:43.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 20 client.14682.objecter dump_active .. 0 homeless 2024-02-19T21:02:43.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (unknown 3184342506 0 0) 0x7f6f100787c0 con 0x7f6f200fd980 2024-02-19T21:02:43.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.238+0000 7f6f1d7fa700 10 client.14682.objecter ms_handle_connect 0x7f6f0c063ce0 2024-02-19T21:02:43.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.239+0000 7f6f1d7fa700 10 client.14682.objecter ms_handle_connect 0x7f6f0c06e630 2024-02-19T21:02:43.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.239+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6f0c072340 con 0x7f6f0c06e630 2024-02-19T21:02:43.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.239+0000 7f6f1d7fa700 10 client.14682.objecter ms_dispatch 0x7f6f20001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:02:43.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.239+0000 7f6f1d7fa700 10 client.14682.objecter _finish_command 1 = system:0 2024-02-19T21:02:43.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.239+0000 7f6f1d7fa700 20 client.14682.objecter put_session s=0x7f6f0c06df80 osd=6 4 2024-02-19T21:02:43.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.239+0000 7f6f1d7fa700 15 client.14682.objecter _session_command_op_remove 6 1 2024-02-19T21:02:43.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.253+0000 7f6f26693700 10 client.14682.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:02:43.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.253+0000 7f6f26693700 15 client.14682.objecter _session_command_op_assign -1 2 2024-02-19T21:02:43.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.253+0000 7f6f26693700 20 client.14682.objecter _get_session s=0x7f6f0c06df80 osd=6 3 2024-02-19T21:02:43.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.253+0000 7f6f26693700 20 client.14682.objecter put_session s=0x7f6f0c06df80 osd=6 3 2024-02-19T21:02:43.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.253+0000 7f6f26693700 20 client.14682.objecter _get_session s=0x7f6f0c06df80 osd=6 3 2024-02-19T21:02:43.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.253+0000 7f6f26693700 15 client.14682.objecter _session_command_op_remove -1 2 2024-02-19T21:02:43.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.253+0000 7f6f26693700 20 client.14682.objecter get_session s=0x7f6f0c06df80 osd=6 3 2024-02-19T21:02:43.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.253+0000 7f6f26693700 15 client.14682.objecter _session_command_op_assign 6 2 2024-02-19T21:02:43.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.253+0000 7f6f26693700 20 client.14682.objecter put_session s=0x7f6f0c06df80 osd=6 4 2024-02-19T21:02:43.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.253+0000 7f6f26693700 10 client.14682.objecter _send_command 2 2024-02-19T21:02:43.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.253+0000 7f6f26693700 1 -- 172.21.15.163:0/2361898336 --> v1:172.21.15.193:6808/787604615 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6f00003be0 con 0x7f6f0c06e630 2024-02-19T21:02:43.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f1d7fa700 1 -- 172.21.15.163:0/2361898336 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 1358482268) 0x7f6f00003be0 con 0x7f6f0c06e630 2024-02-19T21:02:43.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f1d7fa700 10 client.14682.objecter ms_dispatch 0x7f6f20001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:02:43.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f1d7fa700 10 client.14682.objecter _finish_command 2 = system:0 2024-02-19T21:02:43.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f1d7fa700 20 client.14682.objecter put_session s=0x7f6f0c06df80 osd=6 4 2024-02-19T21:02:43.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f1d7fa700 15 client.14682.objecter _session_command_op_remove 6 2 2024-02-19T21:02:43.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f26693700 10 librados: watch_flush enter 2024-02-19T21:02:43.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f26693700 10 librados: watch_flush exit 2024-02-19T21:02:43.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f26693700 10 client.14682.objecter close_session for osd.6 2024-02-19T21:02:43.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f26693700 1 -- 172.21.15.163:0/2361898336 >> v1:172.21.15.193:6808/787604615 conn(0x7f6f0c06e630 legacy=0x7f6f0c070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:43.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f26693700 20 client.14682.objecter put_session s=0x7f6f0c06df80 osd=6 1 2024-02-19T21:02:43.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f26693700 20 client.14682.objecter shutdown clearing up homeless session... 2024-02-19T21:02:43.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f26693700 10 client.14682.objecter successfully canceled tick 2024-02-19T21:02:43.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f26693700 1 -- 172.21.15.163:0/2361898336 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6f0c063ce0 legacy=0x7f6f0c0661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:43.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f26693700 1 -- 172.21.15.163:0/2361898336 >> v1:172.21.15.163:6789/0 conn(0x7f6f200fd980 legacy=0x7f6f200f97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:43.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f24c30700 1 -- 172.21.15.163:0/2361898336 reap_dead start 2024-02-19T21:02:43.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f26693700 1 -- 172.21.15.163:0/2361898336 shutdown_connections 2024-02-19T21:02:43.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f26693700 1 -- 172.21.15.163:0/2361898336 >> 172.21.15.163:0/2361898336 conn(0x7f6f200f35a0 msgr2=0x7f6f200f64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:43.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f26693700 1 -- 172.21.15.163:0/2361898336 shutdown_connections 2024-02-19T21:02:43.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f26693700 1 -- 172.21.15.163:0/2361898336 wait complete. 2024-02-19T21:02:43.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:43.254+0000 7f6f26693700 1 librados: shutdown 2024-02-19T21:02:43.266 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:41.613121+0000 osd.7 (osd.7) 21 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:02:43.266 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:41.616725+0000 osd.7 (osd.7) 22 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:02:43.266 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.645103+0000 mgr.y (mgr.14550) 56 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T21:02:43.266 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.647090+0000 mgr.y (mgr.14550) 57 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi163.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T21:02:43.267 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.658967+0000 mgr.y (mgr.14550) 58 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T21:02:43.267 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.662302+0000 mgr.y (mgr.14550) 59 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi193.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T21:02:43.267 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.672033+0000 mgr.y (mgr.14550) 60 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T21:02:43.267 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:41.675290+0000 mgr.y (mgr.14550) 61 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi193.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T21:02:43.267 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:41.964011+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:02:43.267 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:41.964804+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:02:43.267 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:42.122308+0000 osd.4 (osd.4) 25 : cluster [DBG] 6.e scrub starts 2024-02-19T21:02:43.267 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:42.123294+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.e scrub ok 2024-02-19T21:02:43.267 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:41.613121+0000 osd.7 (osd.7) 21 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:02:43.267 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:41.616725+0000 osd.7 (osd.7) 22 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:02:43.267 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.645103+0000 mgr.y (mgr.14550) 56 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T21:02:43.267 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.647090+0000 mgr.y (mgr.14550) 57 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi163.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T21:02:43.267 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.658967+0000 mgr.y (mgr.14550) 58 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T21:02:43.268 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.662302+0000 mgr.y (mgr.14550) 59 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi193.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T21:02:43.268 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.672033+0000 mgr.y (mgr.14550) 60 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T21:02:43.268 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:41.675290+0000 mgr.y (mgr.14550) 61 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi193.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T21:02:43.268 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:41.964011+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:02:43.268 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:41.964804+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:02:43.268 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:42.122308+0000 osd.4 (osd.4) 25 : cluster [DBG] 6.e scrub starts 2024-02-19T21:02:43.268 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:42.123294+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.e scrub ok 2024-02-19T21:02:43.467 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:41.613121+0000 osd.7 (osd.7) 21 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:02:43.467 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:41.616725+0000 osd.7 (osd.7) 22 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:02:43.468 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.645103+0000 mgr.y (mgr.14550) 56 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T21:02:43.468 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.647090+0000 mgr.y (mgr.14550) 57 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi163.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T21:02:43.468 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.658967+0000 mgr.y (mgr.14550) 58 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T21:02:43.468 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.662302+0000 mgr.y (mgr.14550) 59 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi193.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T21:02:43.468 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.672033+0000 mgr.y (mgr.14550) 60 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T21:02:43.468 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:41.675290+0000 mgr.y (mgr.14550) 61 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi193.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T21:02:43.468 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:41.964011+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:02:43.468 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:41.964804+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:02:43.468 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:42.122308+0000 osd.4 (osd.4) 25 : cluster [DBG] 6.e scrub starts 2024-02-19T21:02:43.468 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:42.123294+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.e scrub ok 2024-02-19T21:02:43.539 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:02:43] "GET /metrics HTTP/1.1" 200 172139 "" "Prometheus/2.33.4" 2024-02-19T21:02:43.540 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=info ts=2024-02-19T21:02:43.507Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000375615s 2024-02-19T21:02:44.112 INFO:teuthology.orchestra.run.smithi163.stdout:193273528345 2024-02-19T21:02:44.112 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-19T21:02:44.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:42.522089+0000 osd.1 (osd.1) 21 : cluster [DBG] 2.a scrub starts 2024-02-19T21:02:44.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:42.525337+0000 osd.1 (osd.1) 22 : cluster [DBG] 2.a scrub ok 2024-02-19T21:02:44.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:42.926027+0000 mgr.y (mgr.14550) 62 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:02:44.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:42.522089+0000 osd.1 (osd.1) 21 : cluster [DBG] 2.a scrub starts 2024-02-19T21:02:44.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:42.525337+0000 osd.1 (osd.1) 22 : cluster [DBG] 2.a scrub ok 2024-02-19T21:02:44.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:42.926027+0000 mgr.y (mgr.14550) 62 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:02:44.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:42.522089+0000 osd.1 (osd.1) 21 : cluster [DBG] 2.a scrub starts 2024-02-19T21:02:44.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:42.525337+0000 osd.1 (osd.1) 22 : cluster [DBG] 2.a scrub ok 2024-02-19T21:02:44.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:42.926027+0000 mgr.y (mgr.14550) 62 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:02:44.627 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:45.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:43.789326+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:02:45.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:43.790202+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:02:45.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:43.789326+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:02:45.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:43.790202+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:02:45.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:43.789326+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:02:45.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:43.790202+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:02:46.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.284+0000 7f10fa215700 1 -- 172.21.15.163:0/154340254 >> v1:172.21.15.163:6790/0 conn(0x7f10f4100780 legacy=0x7f10f4102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:46.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.285+0000 7f10fa215700 1 -- 172.21.15.163:0/154340254 shutdown_connections 2024-02-19T21:02:46.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.285+0000 7f10fa215700 1 -- 172.21.15.163:0/154340254 >> 172.21.15.163:0/154340254 conn(0x7f10f40f35a0 msgr2=0x7f10f40f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:46.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.285+0000 7f10fa215700 1 -- 172.21.15.163:0/154340254 shutdown_connections 2024-02-19T21:02:46.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.285+0000 7f10fa215700 1 -- 172.21.15.163:0/154340254 wait complete. 2024-02-19T21:02:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.285+0000 7f10fa215700 1 librados: starting msgr at 2024-02-19T21:02:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.285+0000 7f10fa215700 1 librados: starting objecter 2024-02-19T21:02:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.285+0000 7f10fa215700 1 Processor -- start 2024-02-19T21:02:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.285+0000 7f10fa215700 1 -- start start 2024-02-19T21:02:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.285+0000 7f10fa215700 1 librados: setting wanted keys 2024-02-19T21:02:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.285+0000 7f10fa215700 1 librados: calling monclient init 2024-02-19T21:02:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10fa215700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10f406cce0 con 0x7f10f41042c0 2024-02-19T21:02:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10fa215700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10f406ce70 con 0x7f10f40fd980 2024-02-19T21:02:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10fa215700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10f411bb80 con 0x7f10f4100780 2024-02-19T21:02:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f3fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f10f41042c0 0x7f10f4118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57942/0 (socket says 172.21.15.163:57942) 2024-02-19T21:02:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f3fff700 1 -- 172.21.15.163:0/1575412500 learned_addr learned my addr 172.21.15.163:0/1575412500 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 10 client.?.objecter ms_handle_connect 0x7f10f41042c0 2024-02-19T21:02:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 10 client.?.objecter ms_handle_connect 0x7f10f40fd980 2024-02-19T21:02:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920658 0 0) 0x7f10f406cce0 con 0x7f10f41042c0 2024-02-19T21:02:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e00033f0 con 0x7f10f41042c0 2024-02-19T21:02:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844076732 0 0) 0x7f10f406ce70 con 0x7f10f40fd980 2024-02-19T21:02:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f406cce0 con 0x7f10f40fd980 2024-02-19T21:02:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 10 client.?.objecter ms_handle_connect 0x7f10f4100780 2024-02-19T21:02:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944435789 0 0) 0x7f10f411bb80 con 0x7f10f4100780 2024-02-19T21:02:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f406ce70 con 0x7f10f4100780 2024-02-19T21:02:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3845725763 0 0) 0x7f10e00033f0 con 0x7f10f41042c0 2024-02-19T21:02:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10f411bb80 con 0x7f10f41042c0 2024-02-19T21:02:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f10ec001c00 con 0x7f10f41042c0 2024-02-19T21:02:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2143472385 0 0) 0x7f10f406cce0 con 0x7f10f40fd980 2024-02-19T21:02:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10e00033f0 con 0x7f10f40fd980 2024-02-19T21:02:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.286+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f10dc0018c0 con 0x7f10f40fd980 2024-02-19T21:02:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.287+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4068788303 0 0) 0x7f10f406ce70 con 0x7f10f4100780 2024-02-19T21:02:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.287+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10f406cce0 con 0x7f10f4100780 2024-02-19T21:02:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.287+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f10e4002d60 con 0x7f10f4100780 2024-02-19T21:02:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.287+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 128744479 0 0) 0x7f10f411bb80 con 0x7f10f41042c0 2024-02-19T21:02:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.287+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 >> v1:172.21.15.163:6790/0 conn(0x7f10f4100780 legacy=0x7f10f4114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.287+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 >> v1:172.21.15.193:6789/0 conn(0x7f10f40fd980 legacy=0x7f10f40fc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.287+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f411cd20 con 0x7f10f41042c0 2024-02-19T21:02:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.287+0000 7f10fa215700 1 -- 172.21.15.163:0/1575412500 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f411bd70 con 0x7f10f41042c0 2024-02-19T21:02:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.287+0000 7f10fa215700 10 client.14688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.287+0000 7f10fa215700 1 -- 172.21.15.163:0/1575412500 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f411c1e0 con 0x7f10f41042c0 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.287+0000 7f10fa215700 1 librados: init done 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.288+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f10ec0022c0 con 0x7f10f41042c0 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.288+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f10ec005d10 con 0x7f10f41042c0 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.288+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f10ec018c60 con 0x7f10f41042c0 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.288+0000 7f10fa215700 10 client.14688.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.288+0000 7f10fa215700 15 client.14688.objecter _session_command_op_assign -1 1 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.288+0000 7f10fa215700 20 client.14688.objecter _get_session osd=-1 returning homeless 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.288+0000 7f10fa215700 10 client.14688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.288+0000 7f10fa215700 1 -- 172.21.15.163:0/1575412500 --> v1:172.21.15.163:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f10d40028b0 con 0x7f10f41042c0 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(80..80 src has 1..80) v4 ==== 6172+0+0 (unknown 3380790453 0 0) 0x7f10ec078880 con 0x7f10f41042c0 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 10 client.14688.objecter ms_dispatch 0x7f10f4001550 osd_map(80..80 src has 1..80) v4 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 3 client.14688.objecter handle_osd_map got epochs [80,80] > 0 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 3 client.14688.objecter handle_osd_map decoding full epoch 80 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 10 client.14688.objecter checking command 1 2024-02-19T21:02:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 20 client.14688.objecter _get_session s=0x7f10e006df80 osd=7 3 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 20 client.14688.objecter put_session s=0x7f10e006df80 osd=7 3 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 15 client.14688.objecter _session_command_op_remove -1 1 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 20 client.14688.objecter _get_session s=0x7f10e006df80 osd=7 3 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 20 client.14688.objecter get_session s=0x7f10e006df80 osd=7 3 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 15 client.14688.objecter _session_command_op_assign 7 1 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 20 client.14688.objecter put_session s=0x7f10e006df80 osd=7 4 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 10 client.14688.objecter _send_command 1 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 --> v1:172.21.15.193:6812/157927432 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10e0072340 con 0x7f10e006e630 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 20 client.14688.objecter dump_active .. 0 homeless 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_get_version_reply(handle=1 version=80) v2 ==== 24+0+0 (unknown 3184342506 0 0) 0x7f10ec078c70 con 0x7f10f41042c0 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 10 client.14688.objecter ms_handle_connect 0x7f10e0063ce0 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.289+0000 7f10f0ff9700 10 client.14688.objecter ms_handle_connect 0x7f10e006e630 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.290+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f10e0072340 con 0x7f10e006e630 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.290+0000 7f10f0ff9700 10 client.14688.objecter ms_dispatch 0x7f10f4001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.290+0000 7f10f0ff9700 10 client.14688.objecter _finish_command 1 = system:0 2024-02-19T21:02:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.290+0000 7f10f0ff9700 20 client.14688.objecter put_session s=0x7f10e006df80 osd=7 4 2024-02-19T21:02:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.290+0000 7f10f0ff9700 15 client.14688.objecter _session_command_op_remove 7 1 2024-02-19T21:02:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.304+0000 7f10fa215700 10 client.14688.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:02:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.304+0000 7f10fa215700 15 client.14688.objecter _session_command_op_assign -1 2 2024-02-19T21:02:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.304+0000 7f10fa215700 20 client.14688.objecter _get_session s=0x7f10e006df80 osd=7 3 2024-02-19T21:02:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.304+0000 7f10fa215700 20 client.14688.objecter put_session s=0x7f10e006df80 osd=7 3 2024-02-19T21:02:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.304+0000 7f10fa215700 20 client.14688.objecter _get_session s=0x7f10e006df80 osd=7 3 2024-02-19T21:02:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.304+0000 7f10fa215700 15 client.14688.objecter _session_command_op_remove -1 2 2024-02-19T21:02:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.304+0000 7f10fa215700 20 client.14688.objecter get_session s=0x7f10e006df80 osd=7 3 2024-02-19T21:02:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.304+0000 7f10fa215700 15 client.14688.objecter _session_command_op_assign 7 2 2024-02-19T21:02:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.304+0000 7f10fa215700 20 client.14688.objecter put_session s=0x7f10e006df80 osd=7 4 2024-02-19T21:02:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.304+0000 7f10fa215700 10 client.14688.objecter _send_command 2 2024-02-19T21:02:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.304+0000 7f10fa215700 1 -- 172.21.15.163:0/1575412500 --> v1:172.21.15.193:6812/157927432 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f10d4001140 con 0x7f10e006e630 2024-02-19T21:02:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10f0ff9700 1 -- 172.21.15.163:0/1575412500 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 4108596023) 0x7f10d4001140 con 0x7f10e006e630 2024-02-19T21:02:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10f0ff9700 10 client.14688.objecter ms_dispatch 0x7f10f4001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:02:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10f0ff9700 10 client.14688.objecter _finish_command 2 = system:0 2024-02-19T21:02:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10f0ff9700 20 client.14688.objecter put_session s=0x7f10e006df80 osd=7 4 2024-02-19T21:02:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10f0ff9700 15 client.14688.objecter _session_command_op_remove 7 2 2024-02-19T21:02:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10fa215700 10 librados: watch_flush enter 2024-02-19T21:02:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10fa215700 10 librados: watch_flush exit 2024-02-19T21:02:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10fa215700 10 client.14688.objecter close_session for osd.7 2024-02-19T21:02:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10fa215700 1 -- 172.21.15.163:0/1575412500 >> v1:172.21.15.193:6812/157927432 conn(0x7f10e006e630 legacy=0x7f10e0070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10fa215700 20 client.14688.objecter put_session s=0x7f10e006df80 osd=7 1 2024-02-19T21:02:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10fa215700 20 client.14688.objecter shutdown clearing up homeless session... 2024-02-19T21:02:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10fa215700 10 client.14688.objecter successfully canceled tick 2024-02-19T21:02:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10fa215700 1 -- 172.21.15.163:0/1575412500 >> v1:172.21.15.163:6800/1552514681 conn(0x7f10e0063ce0 legacy=0x7f10e00661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10fa215700 1 -- 172.21.15.163:0/1575412500 >> v1:172.21.15.163:6789/0 conn(0x7f10f41042c0 legacy=0x7f10f4118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10f3fff700 1 -- 172.21.15.163:0/1575412500 reap_dead start 2024-02-19T21:02:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10fa215700 1 -- 172.21.15.163:0/1575412500 shutdown_connections 2024-02-19T21:02:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10fa215700 1 -- 172.21.15.163:0/1575412500 >> 172.21.15.163:0/1575412500 conn(0x7f10f40f35a0 msgr2=0x7f10f4107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10fa215700 1 -- 172.21.15.163:0/1575412500 shutdown_connections 2024-02-19T21:02:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10fa215700 1 -- 172.21.15.163:0/1575412500 wait complete. 2024-02-19T21:02:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:46.305+0000 7f10fa215700 1 librados: shutdown 2024-02-19T21:02:46.319 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:44.559734+0000 osd.7 (osd.7) 23 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:02:46.319 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:44.563248+0000 osd.7 (osd.7) 24 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:02:46.319 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:44.926713+0000 mgr.y (mgr.14550) 63 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:02:46.320 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:45.221067+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:02:46.320 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:45.221960+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:02:46.320 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:46.061820+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:46.320 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:44.559734+0000 osd.7 (osd.7) 23 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:02:46.320 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:44.563248+0000 osd.7 (osd.7) 24 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:02:46.320 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:44.926713+0000 mgr.y (mgr.14550) 63 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:02:46.320 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:45.221067+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:02:46.320 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:45.221960+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:02:46.320 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:46.061820+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:46.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:44.559734+0000 osd.7 (osd.7) 23 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:02:46.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:44.563248+0000 osd.7 (osd.7) 24 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:02:46.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:44.926713+0000 mgr.y (mgr.14550) 63 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:02:46.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:45.221067+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:02:46.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:45.221960+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:02:46.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:46.061820+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:46.889 INFO:teuthology.orchestra.run.smithi163.stdout:227633266708 2024-02-19T21:02:46.889 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-19T21:02:47.198 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:45.575888+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:02:47.198 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:45.578971+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:02:47.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:45.575888+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:02:47.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:45.578971+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:02:47.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:45.575888+0000 osd.7 (osd.7) 25 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:02:47.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:45.578971+0000 osd.7 (osd.7) 26 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:02:47.634 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:48.062 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:02:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:02:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:02:48.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:46.557125+0000 osd.1 (osd.1) 23 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:02:48.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:46.558734+0000 osd.1 (osd.1) 24 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:02:48.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:46.562292+0000 osd.7 (osd.7) 27 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:02:48.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:46.565376+0000 osd.7 (osd.7) 28 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:02:48.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:46.839711+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:02:48.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:46.842727+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:02:48.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:46.927073+0000 mgr.y (mgr.14550) 64 : cluster [DBG] pgmap v45: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:48.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:47.237004+0000 osd.2 (osd.2) 23 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:02:48.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:47.238466+0000 osd.2 (osd.2) 24 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:02:48.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:46.557125+0000 osd.1 (osd.1) 23 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:02:48.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:46.558734+0000 osd.1 (osd.1) 24 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:02:48.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:46.562292+0000 osd.7 (osd.7) 27 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:02:48.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:46.565376+0000 osd.7 (osd.7) 28 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:02:48.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:46.839711+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:02:48.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:46.842727+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:02:48.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:46.927073+0000 mgr.y (mgr.14550) 64 : cluster [DBG] pgmap v45: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:48.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:47.237004+0000 osd.2 (osd.2) 23 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:02:48.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:47.238466+0000 osd.2 (osd.2) 24 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:02:48.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:46.557125+0000 osd.1 (osd.1) 23 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:02:48.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:46.558734+0000 osd.1 (osd.1) 24 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:02:48.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:46.562292+0000 osd.7 (osd.7) 27 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:02:48.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:46.565376+0000 osd.7 (osd.7) 28 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:02:48.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:46.839711+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:02:48.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:46.842727+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:02:48.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:46.927073+0000 mgr.y (mgr.14550) 64 : cluster [DBG] pgmap v45: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:48.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:47.237004+0000 osd.2 (osd.2) 23 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:02:48.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:47.238466+0000 osd.2 (osd.2) 24 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:02:49.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:47.590264+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:02:49.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:47.593128+0000 osd.0 (osd.0) 20 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:02:49.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:47.599811+0000 osd.1 (osd.1) 25 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:02:49.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:47.601121+0000 osd.1 (osd.1) 26 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:02:49.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:47.804181+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:02:49.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:47.805167+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:02:49.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:48.025425+0000 osd.4 (osd.4) 27 : cluster [DBG] 5.e scrub starts 2024-02-19T21:02:49.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:48.027064+0000 osd.4 (osd.4) 28 : cluster [DBG] 5.e scrub ok 2024-02-19T21:02:49.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:48.861925+0000 osd.3 (osd.3) 21 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:02:49.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:48.863848+0000 osd.3 (osd.3) 22 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:02:49.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:47.590264+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:02:49.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:47.593128+0000 osd.0 (osd.0) 20 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:02:49.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:47.599811+0000 osd.1 (osd.1) 25 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:02:49.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:47.601121+0000 osd.1 (osd.1) 26 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:02:49.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:47.804181+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:02:49.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:47.805167+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:02:49.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:48.025425+0000 osd.4 (osd.4) 27 : cluster [DBG] 5.e scrub starts 2024-02-19T21:02:49.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:48.027064+0000 osd.4 (osd.4) 28 : cluster [DBG] 5.e scrub ok 2024-02-19T21:02:49.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:48.861925+0000 osd.3 (osd.3) 21 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:02:49.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:48.863848+0000 osd.3 (osd.3) 22 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:02:49.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:47.590264+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:02:49.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:47.593128+0000 osd.0 (osd.0) 20 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:02:49.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:47.599811+0000 osd.1 (osd.1) 25 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:02:49.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:47.601121+0000 osd.1 (osd.1) 26 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:02:49.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:47.804181+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:02:49.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:47.805167+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:02:49.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:48.025425+0000 osd.4 (osd.4) 27 : cluster [DBG] 5.e scrub starts 2024-02-19T21:02:49.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:48.027064+0000 osd.4 (osd.4) 28 : cluster [DBG] 5.e scrub ok 2024-02-19T21:02:49.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:48.861925+0000 osd.3 (osd.3) 21 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:02:49.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:48.863848+0000 osd.3 (osd.3) 22 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:02:49.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.863+0000 7f63d1823700 1 -- 172.21.15.163:0/1145864543 >> v1:172.21.15.163:6789/0 conn(0x7f63cc103150 legacy=0x7f63cc1055d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:49.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.863+0000 7f63d1823700 1 -- 172.21.15.163:0/1145864543 shutdown_connections 2024-02-19T21:02:49.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.863+0000 7f63d1823700 1 -- 172.21.15.163:0/1145864543 >> 172.21.15.163:0/1145864543 conn(0x7f63cc0f23d0 msgr2=0x7f63cc0f47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:49.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.863+0000 7f63d1823700 1 -- 172.21.15.163:0/1145864543 shutdown_connections 2024-02-19T21:02:49.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.863+0000 7f63d1823700 1 -- 172.21.15.163:0/1145864543 wait complete. 2024-02-19T21:02:49.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.864+0000 7f63d1823700 1 librados: starting msgr at 2024-02-19T21:02:49.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.864+0000 7f63d1823700 1 librados: starting objecter 2024-02-19T21:02:49.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.864+0000 7f63d1823700 1 Processor -- start 2024-02-19T21:02:49.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.864+0000 7f63d1823700 1 -- start start 2024-02-19T21:02:49.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.864+0000 7f63d1823700 1 librados: setting wanted keys 2024-02-19T21:02:49.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.864+0000 7f63d1823700 1 librados: calling monclient init 2024-02-19T21:02:49.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.864+0000 7f63d1823700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f63cc117280 con 0x7f63cc0ff610 2024-02-19T21:02:49.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.864+0000 7f63d1823700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f63cc118460 con 0x7f63cc103150 2024-02-19T21:02:49.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.864+0000 7f63d1823700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f63cc119660 con 0x7f63cc0fc810 2024-02-19T21:02:49.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63caffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f63cc0fc810 0x7f63cc060ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:60342/0 (socket says 172.21.15.163:60342) 2024-02-19T21:02:49.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63caffd700 1 -- 172.21.15.163:0/869584007 learned_addr learned my addr 172.21.15.163:0/869584007 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:49.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63ca7fc700 1 --1- 172.21.15.163:0/869584007 >> v1:172.21.15.163:6789/0 conn(0x7f63cc0ff610 0x7f63cc0612b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57968/0 (socket says 172.21.15.163:57968) 2024-02-19T21:02:49.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 10 client.?.objecter ms_handle_connect 0x7f63cc0fc810 2024-02-19T21:02:49.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:49.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 10 client.?.objecter ms_handle_connect 0x7f63cc103150 2024-02-19T21:02:49.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:49.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387802960 0 0) 0x7f63cc119660 con 0x7f63cc0fc810 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b40038e0 con 0x7f63cc0fc810 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 10 client.?.objecter ms_handle_connect 0x7f63cc0ff610 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434904727 0 0) 0x7f63cc118460 con 0x7f63cc103150 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63cc119660 con 0x7f63cc103150 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3688477667 0 0) 0x7f63cc119660 con 0x7f63cc103150 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63cc118460 con 0x7f63cc103150 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f63c40030e0 con 0x7f63cc103150 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830056380 0 0) 0x7f63cc117280 con 0x7f63cc0ff610 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63cc119660 con 0x7f63cc0ff610 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3033278298 0 0) 0x7f63b40038e0 con 0x7f63cc0fc810 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63cc117280 con 0x7f63cc0fc810 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f63c00019f0 con 0x7f63cc0fc810 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1262771588 0 0) 0x7f63cc118460 con 0x7f63cc103150 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 >> v1:172.21.15.163:6790/0 conn(0x7f63cc0fc810 legacy=0x7f63cc060ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:49.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 >> v1:172.21.15.163:6789/0 conn(0x7f63cc0ff610 legacy=0x7f63cc0612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:49.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc11a860 con 0x7f63cc103150 2024-02-19T21:02:49.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63d1823700 1 -- 172.21.15.163:0/869584007 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc118650 con 0x7f63cc103150 2024-02-19T21:02:49.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63d1823700 10 client.24619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:49.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63d1823700 1 -- 172.21.15.163:0/869584007 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc118ac0 con 0x7f63cc103150 2024-02-19T21:02:49.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.865+0000 7f63d1823700 1 librados: init done 2024-02-19T21:02:49.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.866+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f63c40026f0 con 0x7f63cc103150 2024-02-19T21:02:49.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.866+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f63c4003b30 con 0x7f63cc103150 2024-02-19T21:02:49.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.866+0000 7f63d1823700 1 -- 172.21.15.163:0/869584007 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b8004f20 con 0x7f63cc103150 2024-02-19T21:02:49.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.867+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f63c40021f0 con 0x7f63cc103150 2024-02-19T21:02:49.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.867+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(80..80 src has 1..80) v4 ==== 6172+0+0 (unknown 3380790453 0 0) 0x7f63c40764d0 con 0x7f63cc103150 2024-02-19T21:02:49.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.867+0000 7f63b3fff700 10 client.24619.objecter ms_dispatch 0x7f63cc001550 osd_map(80..80 src has 1..80) v4 2024-02-19T21:02:49.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.867+0000 7f63b3fff700 3 client.24619.objecter handle_osd_map got epochs [80,80] > 0 2024-02-19T21:02:49.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.867+0000 7f63b3fff700 3 client.24619.objecter handle_osd_map decoding full epoch 80 2024-02-19T21:02:49.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.867+0000 7f63b3fff700 20 client.24619.objecter dump_active .. 0 homeless 2024-02-19T21:02:49.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.867+0000 7f63b3fff700 10 client.24619.objecter ms_handle_connect 0x7f63b4063b80 2024-02-19T21:02:49.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:49.876+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f63c404a080 con 0x7f63cc103150 2024-02-19T21:02:50.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:50.061+0000 7f63d1823700 1 -- 172.21.15.163:0/869584007 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f63b80051d0 con 0x7f63cc103150 2024-02-19T21:02:50.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:50.062+0000 7f63b3fff700 1 -- 172.21.15.163:0/869584007 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 739478307) 0x7f63c40484f0 con 0x7f63cc103150 2024-02-19T21:02:50.053 INFO:teuthology.orchestra.run.smithi163.stdout:34359738423 2024-02-19T21:02:50.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:50.064+0000 7f63d1823700 10 librados: watch_flush enter 2024-02-19T21:02:50.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:50.064+0000 7f63d1823700 10 librados: watch_flush exit 2024-02-19T21:02:50.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:50.064+0000 7f63d1823700 20 client.24619.objecter shutdown clearing up homeless session... 2024-02-19T21:02:50.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:50.064+0000 7f63d1823700 10 client.24619.objecter successfully canceled tick 2024-02-19T21:02:50.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:50.064+0000 7f63d1823700 1 -- 172.21.15.163:0/869584007 >> v1:172.21.15.163:6800/1552514681 conn(0x7f63b4063b80 legacy=0x7f63b4066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:50.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:50.064+0000 7f63d1823700 1 -- 172.21.15.163:0/869584007 >> v1:172.21.15.193:6789/0 conn(0x7f63cc103150 legacy=0x7f63cc115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:50.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:50.064+0000 7f63d1823700 1 -- 172.21.15.163:0/869584007 shutdown_connections 2024-02-19T21:02:50.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:50.064+0000 7f63d1823700 1 -- 172.21.15.163:0/869584007 >> 172.21.15.163:0/869584007 conn(0x7f63cc0f23d0 msgr2=0x7f63cc1065c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:50.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:50.064+0000 7f63d1823700 1 -- 172.21.15.163:0/869584007 shutdown_connections 2024-02-19T21:02:50.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:50.064+0000 7f63d1823700 1 -- 172.21.15.163:0/869584007 wait complete. 2024-02-19T21:02:50.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:50.064+0000 7f63d1823700 1 librados: shutdown 2024-02-19T21:02:50.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:02:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:02:50.369 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:48.569514+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:02:50.369 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:48.572856+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:02:50.369 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:48.927408+0000 mgr.y (mgr.14550) 65 : cluster [DBG] pgmap v46: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-02-19T21:02:50.369 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:49.053388+0000 osd.5 (osd.5) 13 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:02:50.369 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:49.056403+0000 osd.5 (osd.5) 14 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:02:50.369 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:49.174086+0000 osd.2 (osd.2) 25 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:02:50.369 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:49.176961+0000 osd.2 (osd.2) 26 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:02:50.369 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:49.314499+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:50.369 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:49.711565+0000 mon.b (mon.1) 71 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:02:50.369 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:49.712647+0000 mon.b (mon.1) 72 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:02:50.370 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:49.717247+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:50.370 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:49.721892+0000 mon.b (mon.1) 73 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:02:50.370 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:49.862045+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.17 deep-scrub starts 2024-02-19T21:02:50.370 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:49.863028+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.17 deep-scrub ok 2024-02-19T21:02:50.370 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:50.063336+0000 mon.b (mon.1) 74 : audit [DBG] from='client.? v1:172.21.15.163:0/869584007' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-19T21:02:50.370 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:48.569514+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:02:50.370 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:48.572856+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:02:50.370 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:48.927408+0000 mgr.y (mgr.14550) 65 : cluster [DBG] pgmap v46: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-02-19T21:02:50.370 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:49.053388+0000 osd.5 (osd.5) 13 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:02:50.370 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:49.056403+0000 osd.5 (osd.5) 14 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:02:50.370 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:49.174086+0000 osd.2 (osd.2) 25 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:02:50.370 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:49.176961+0000 osd.2 (osd.2) 26 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:02:50.370 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:49.314499+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:50.371 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:49.711565+0000 mon.b (mon.1) 71 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:02:50.371 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:49.712647+0000 mon.b (mon.1) 72 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:02:50.371 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:49.717247+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:50.371 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:49.721892+0000 mon.b (mon.1) 73 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:02:50.371 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:49.862045+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.17 deep-scrub starts 2024-02-19T21:02:50.371 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:49.863028+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.17 deep-scrub ok 2024-02-19T21:02:50.371 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:50.063336+0000 mon.b (mon.1) 74 : audit [DBG] from='client.? v1:172.21.15.163:0/869584007' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-19T21:02:50.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:48.569514+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:02:50.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:48.572856+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:02:50.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:48.927408+0000 mgr.y (mgr.14550) 65 : cluster [DBG] pgmap v46: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 933 B/s rd, 0 op/s 2024-02-19T21:02:50.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:49.053388+0000 osd.5 (osd.5) 13 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:02:50.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:49.056403+0000 osd.5 (osd.5) 14 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:02:50.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:49.174086+0000 osd.2 (osd.2) 25 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:02:50.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:49.176961+0000 osd.2 (osd.2) 26 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:02:50.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:49.314499+0000 mon.a (mon.0) 696 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:50.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:49.711565+0000 mon.b (mon.1) 71 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:02:50.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:49.712647+0000 mon.b (mon.1) 72 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:02:50.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:49.717247+0000 mon.a (mon.0) 697 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:50.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:49.721892+0000 mon.b (mon.1) 73 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:02:50.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:49.862045+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.17 deep-scrub starts 2024-02-19T21:02:50.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:49.863028+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.17 deep-scrub ok 2024-02-19T21:02:50.563 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:50.063336+0000 mon.b (mon.1) 74 : audit [DBG] from='client.? v1:172.21.15.163:0/869584007' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-19T21:02:51.002 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738423 for osd.0 2024-02-19T21:02:51.002 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-19T21:02:51.515 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:51.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:49.974459+0000 mgr.y (mgr.14550) 66 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:51.738 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:49.974459+0000 mgr.y (mgr.14550) 66 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:51.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:49.974459+0000 mgr.y (mgr.14550) 66 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:02:51.739 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:02:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=info ts=2024-02-19T21:02:51.508Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.001263018s 2024-02-19T21:02:52.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:50.919999+0000 osd.6 (osd.6) 25 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:02:52.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:50.920944+0000 osd.6 (osd.6) 26 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:02:52.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:50.928186+0000 mgr.y (mgr.14550) 67 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:52.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:51.535878+0000 osd.3 (osd.3) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:52.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:51.535931+0000 osd.3 (osd.3) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:52.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:51.609543+0000 osd.2 (osd.2) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:52.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:51.609607+0000 osd.2 (osd.2) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:52.738 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:50.919999+0000 osd.6 (osd.6) 25 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:02:52.738 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:50.920944+0000 osd.6 (osd.6) 26 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:02:52.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:50.928186+0000 mgr.y (mgr.14550) 67 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:52.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:51.535878+0000 osd.3 (osd.3) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:52.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:51.535931+0000 osd.3 (osd.3) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:52.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:51.609543+0000 osd.2 (osd.2) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:52.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:51.609607+0000 osd.2 (osd.2) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:52.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:50.919999+0000 osd.6 (osd.6) 25 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:02:52.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:50.920944+0000 osd.6 (osd.6) 26 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:02:52.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:50.928186+0000 mgr.y (mgr.14550) 67 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:52.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:51.535878+0000 osd.3 (osd.3) 25 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:52.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:51.535931+0000 osd.3 (osd.3) 26 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:52.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:51.609543+0000 osd.2 (osd.2) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:02:52.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:51.609607+0000 osd.2 (osd.2) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:02:53.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.313+0000 7f808f00e700 1 -- 172.21.15.163:0/82439255 >> v1:172.21.15.163:6790/0 conn(0x7f80881031b0 legacy=0x7f8088105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:53.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.313+0000 7f808f00e700 1 -- 172.21.15.163:0/82439255 shutdown_connections 2024-02-19T21:02:53.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.313+0000 7f808f00e700 1 -- 172.21.15.163:0/82439255 >> 172.21.15.163:0/82439255 conn(0x7f80880f2470 msgr2=0x7f80880f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:53.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.314+0000 7f808f00e700 1 -- 172.21.15.163:0/82439255 shutdown_connections 2024-02-19T21:02:53.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.314+0000 7f808f00e700 1 -- 172.21.15.163:0/82439255 wait complete. 2024-02-19T21:02:53.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.314+0000 7f808f00e700 1 librados: starting msgr at 2024-02-19T21:02:53.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.314+0000 7f808f00e700 1 librados: starting objecter 2024-02-19T21:02:53.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.314+0000 7f808f00e700 1 Processor -- start 2024-02-19T21:02:53.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.314+0000 7f808f00e700 1 -- start start 2024-02-19T21:02:53.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.314+0000 7f808f00e700 1 librados: setting wanted keys 2024-02-19T21:02:53.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.314+0000 7f808f00e700 1 librados: calling monclient init 2024-02-19T21:02:53.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.314+0000 7f808f00e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f80880fbe70 con 0x7f80880fc870 2024-02-19T21:02:53.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.314+0000 7f808f00e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f80880fc040 con 0x7f80881031b0 2024-02-19T21:02:53.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.314+0000 7f808f00e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f80880fc230 con 0x7f80880ff670 2024-02-19T21:02:53.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f808cdaa700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f80880fc870 0x7f80880fb6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57980/0 (socket says 172.21.15.163:57980) 2024-02-19T21:02:53.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f808cdaa700 1 -- 172.21.15.163:0/75904120 learned_addr learned my addr 172.21.15.163:0/75904120 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:53.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8087fff700 1 --1- 172.21.15.163:0/75904120 >> v1:172.21.15.163:6790/0 conn(0x7f80880ff670 0x7f8088114c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:60356/0 (socket says 172.21.15.163:60356) 2024-02-19T21:02:53.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 10 client.?.objecter ms_handle_connect 0x7f80880ff670 2024-02-19T21:02:53.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:53.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 10 client.?.objecter ms_handle_connect 0x7f80880fc870 2024-02-19T21:02:53.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:53.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 10 client.?.objecter ms_handle_connect 0x7f80881031b0 2024-02-19T21:02:53.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:53.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278205824 0 0) 0x7f80880fbe70 con 0x7f80880fc870 2024-02-19T21:02:53.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8078003800 con 0x7f80880fc870 2024-02-19T21:02:53.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605602557 0 0) 0x7f80880fc040 con 0x7f80881031b0 2024-02-19T21:02:53.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80880fbe70 con 0x7f80881031b0 2024-02-19T21:02:53.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4224546294 0 0) 0x7f8078003800 con 0x7f80880fc870 2024-02-19T21:02:53.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80880fc040 con 0x7f80880fc870 2024-02-19T21:02:53.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f807c0019f0 con 0x7f80880fc870 2024-02-19T21:02:53.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1886872687 0 0) 0x7f80880fbe70 con 0x7f80881031b0 2024-02-19T21:02:53.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8078003800 con 0x7f80881031b0 2024-02-19T21:02:53.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f80800031a0 con 0x7f80881031b0 2024-02-19T21:02:53.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 541489723 0 0) 0x7f80880fc040 con 0x7f80880fc870 2024-02-19T21:02:53.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 >> v1:172.21.15.163:6790/0 conn(0x7f80880ff670 legacy=0x7f8088114c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:53.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 >> v1:172.21.15.193:6789/0 conn(0x7f80881031b0 legacy=0x7f8088118340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:53.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808811ca90 con 0x7f80880fc870 2024-02-19T21:02:53.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f808f00e700 1 -- 172.21.15.163:0/75904120 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8088119ac0 con 0x7f80880fc870 2024-02-19T21:02:53.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f807c002450 con 0x7f80880fc870 2024-02-19T21:02:53.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.315+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f807c005b40 con 0x7f80880fc870 2024-02-19T21:02:53.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.316+0000 7f808f00e700 10 client.14700.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:53.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.316+0000 7f808f00e700 1 -- 172.21.15.163:0/75904120 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8088119fb0 con 0x7f80880fc870 2024-02-19T21:02:53.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.316+0000 7f808f00e700 1 librados: init done 2024-02-19T21:02:53.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.317+0000 7f808f00e700 1 -- 172.21.15.163:0/75904120 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f808805f700 con 0x7f80880fc870 2024-02-19T21:02:53.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.317+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f807c005cc0 con 0x7f80880fc870 2024-02-19T21:02:53.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.318+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(82..82 src has 1..82) v4 ==== 6172+0+0 (unknown 3890964453 0 0) 0x7f807c0785e0 con 0x7f80880fc870 2024-02-19T21:02:53.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.318+0000 7f8085ffb700 10 client.14700.objecter ms_dispatch 0x7f8088001550 osd_map(82..82 src has 1..82) v4 2024-02-19T21:02:53.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.318+0000 7f8085ffb700 3 client.14700.objecter handle_osd_map got epochs [82,82] > 0 2024-02-19T21:02:53.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.318+0000 7f8085ffb700 3 client.14700.objecter handle_osd_map decoding full epoch 82 2024-02-19T21:02:53.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.318+0000 7f8085ffb700 20 client.14700.objecter dump_active .. 0 homeless 2024-02-19T21:02:53.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.318+0000 7f8085ffb700 10 client.14700.objecter ms_handle_connect 0x7f8078063b40 2024-02-19T21:02:53.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.322+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f807c04c170 con 0x7f80880fc870 2024-02-19T21:02:53.322 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:51.628600+0000 osd.0 (osd.0) 23 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:02:53.322 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:51.631654+0000 osd.0 (osd.0) 24 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:02:53.322 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:52.018615+0000 osd.5 (osd.5) 15 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:02:53.323 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:52.021597+0000 osd.5 (osd.5) 16 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:02:53.323 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:52.326051+0000 mon.a (mon.0) 698 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-02-19T21:02:53.323 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:52.857578+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:02:53.323 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:52.859448+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:02:53.323 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:52.967090+0000 mon.a (mon.0) 699 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-19T21:02:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.494+0000 7f808f00e700 1 -- 172.21.15.163:0/75904120 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8088108de0 con 0x7f80880fc870 2024-02-19T21:02:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.494+0000 7f8085ffb700 1 -- 172.21.15.163:0/75904120 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 2868210388) 0x7f807c04a580 con 0x7f80880fc870 2024-02-19T21:02:53.485 INFO:teuthology.orchestra.run.smithi163.stdout:55834574898 2024-02-19T21:02:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.495+0000 7f808f00e700 10 librados: watch_flush enter 2024-02-19T21:02:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.496+0000 7f808f00e700 10 librados: watch_flush exit 2024-02-19T21:02:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.496+0000 7f808f00e700 20 client.14700.objecter shutdown clearing up homeless session... 2024-02-19T21:02:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.496+0000 7f808f00e700 10 client.14700.objecter successfully canceled tick 2024-02-19T21:02:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.496+0000 7f808f00e700 1 -- 172.21.15.163:0/75904120 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8078063b40 legacy=0x7f8078066000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.496+0000 7f808f00e700 1 -- 172.21.15.163:0/75904120 >> v1:172.21.15.163:6789/0 conn(0x7f80880fc870 legacy=0x7f80880fb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.496+0000 7f808f00e700 1 -- 172.21.15.163:0/75904120 shutdown_connections 2024-02-19T21:02:53.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.496+0000 7f808f00e700 1 -- 172.21.15.163:0/75904120 >> 172.21.15.163:0/75904120 conn(0x7f80880f2470 msgr2=0x7f8088106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:53.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.496+0000 7f808f00e700 1 -- 172.21.15.163:0/75904120 shutdown_connections 2024-02-19T21:02:53.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.496+0000 7f808f00e700 1 -- 172.21.15.163:0/75904120 wait complete. 2024-02-19T21:02:53.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:53.496+0000 7f808f00e700 1 librados: shutdown 2024-02-19T21:02:53.611 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:02:53] "GET /metrics HTTP/1.1" 200 172139 "" "Prometheus/2.33.4" 2024-02-19T21:02:53.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:51.628600+0000 osd.0 (osd.0) 23 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:02:53.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:51.631654+0000 osd.0 (osd.0) 24 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:02:53.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:52.018615+0000 osd.5 (osd.5) 15 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:02:53.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:52.021597+0000 osd.5 (osd.5) 16 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:02:53.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:52.326051+0000 mon.a (mon.0) 698 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-02-19T21:02:53.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:52.857578+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:02:53.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:52.859448+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:02:53.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:52.967090+0000 mon.a (mon.0) 699 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-19T21:02:53.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:51.628600+0000 osd.0 (osd.0) 23 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:02:53.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:51.631654+0000 osd.0 (osd.0) 24 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:02:53.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:52.018615+0000 osd.5 (osd.5) 15 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:02:53.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:52.021597+0000 osd.5 (osd.5) 16 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:02:53.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:52.326051+0000 mon.a (mon.0) 698 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-02-19T21:02:53.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:52.857578+0000 osd.3 (osd.3) 27 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:02:53.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:52.859448+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:02:53.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:52.967090+0000 mon.a (mon.0) 699 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-02-19T21:02:54.478 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:52.555447+0000 osd.1 (osd.1) 27 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:02:54.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:52.556337+0000 osd.1 (osd.1) 28 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:02:54.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:52.642349+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.b scrub starts 2024-02-19T21:02:54.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:52.643829+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.b scrub ok 2024-02-19T21:02:54.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:52.928536+0000 mgr.y (mgr.14550) 68 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:54.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:52.984021+0000 osd.5 (osd.5) 17 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:02:54.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:52.986909+0000 osd.5 (osd.5) 18 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:02:54.479 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:53.495499+0000 mon.a (mon.0) 700 : audit [DBG] from='client.? v1:172.21.15.163:0/75904120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-19T21:02:54.479 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:52.555447+0000 osd.1 (osd.1) 27 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:02:54.479 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:52.556337+0000 osd.1 (osd.1) 28 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:02:54.479 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:52.642349+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.b scrub starts 2024-02-19T21:02:54.479 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:52.643829+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.b scrub ok 2024-02-19T21:02:54.479 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:52.928536+0000 mgr.y (mgr.14550) 68 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:54.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:52.984021+0000 osd.5 (osd.5) 17 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:02:54.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:52.986909+0000 osd.5 (osd.5) 18 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:02:54.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:53.495499+0000 mon.a (mon.0) 700 : audit [DBG] from='client.? v1:172.21.15.163:0/75904120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-19T21:02:54.481 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574898 for osd.1 2024-02-19T21:02:54.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-19T21:02:54.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:52.555447+0000 osd.1 (osd.1) 27 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:02:54.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:52.556337+0000 osd.1 (osd.1) 28 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:02:54.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:52.642349+0000 osd.0 (osd.0) 25 : cluster [DBG] 4.b scrub starts 2024-02-19T21:02:54.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:52.643829+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.b scrub ok 2024-02-19T21:02:54.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:52.928536+0000 mgr.y (mgr.14550) 68 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:02:54.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:52.984021+0000 osd.5 (osd.5) 17 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:02:54.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:52.986909+0000 osd.5 (osd.5) 18 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:02:54.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:53.495499+0000 mon.a (mon.0) 700 : audit [DBG] from='client.? v1:172.21.15.163:0/75904120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-19T21:02:54.864 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:55.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:53.522666+0000 osd.1 (osd.1) 29 : cluster [DBG] 4.f scrub starts 2024-02-19T21:02:55.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:53.526136+0000 osd.1 (osd.1) 30 : cluster [DBG] 4.f scrub ok 2024-02-19T21:02:55.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:54.196872+0000 osd.2 (osd.2) 29 : cluster [DBG] 6.f scrub starts 2024-02-19T21:02:55.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:54.197709+0000 osd.2 (osd.2) 30 : cluster [DBG] 6.f scrub ok 2024-02-19T21:02:55.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:53.522666+0000 osd.1 (osd.1) 29 : cluster [DBG] 4.f scrub starts 2024-02-19T21:02:55.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:53.526136+0000 osd.1 (osd.1) 30 : cluster [DBG] 4.f scrub ok 2024-02-19T21:02:55.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:54.196872+0000 osd.2 (osd.2) 29 : cluster [DBG] 6.f scrub starts 2024-02-19T21:02:55.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:54.197709+0000 osd.2 (osd.2) 30 : cluster [DBG] 6.f scrub ok 2024-02-19T21:02:55.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:53.522666+0000 osd.1 (osd.1) 29 : cluster [DBG] 4.f scrub starts 2024-02-19T21:02:55.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:53.526136+0000 osd.1 (osd.1) 30 : cluster [DBG] 4.f scrub ok 2024-02-19T21:02:55.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:54.196872+0000 osd.2 (osd.2) 29 : cluster [DBG] 6.f scrub starts 2024-02-19T21:02:55.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:54.197709+0000 osd.2 (osd.2) 30 : cluster [DBG] 6.f scrub ok 2024-02-19T21:02:56.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:54.543630+0000 osd.1 (osd.1) 31 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:02:56.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:54.544826+0000 osd.1 (osd.1) 32 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:02:56.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:54.929047+0000 mgr.y (mgr.14550) 69 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:56.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:55.212882+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.b scrub starts 2024-02-19T21:02:56.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:55.213686+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.b scrub ok 2024-02-19T21:02:56.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:55.583976+0000 osd.7 (osd.7) 29 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:02:56.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:55.586851+0000 osd.7 (osd.7) 30 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:02:56.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:55.772892+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:56.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:54.543630+0000 osd.1 (osd.1) 31 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:02:56.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:54.544826+0000 osd.1 (osd.1) 32 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:02:56.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:54.929047+0000 mgr.y (mgr.14550) 69 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:56.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:55.212882+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.b scrub starts 2024-02-19T21:02:56.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:55.213686+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.b scrub ok 2024-02-19T21:02:56.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:55.583976+0000 osd.7 (osd.7) 29 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:02:56.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:55.586851+0000 osd.7 (osd.7) 30 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:02:56.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:55.772892+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:56.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:54.543630+0000 osd.1 (osd.1) 31 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:02:56.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:54.544826+0000 osd.1 (osd.1) 32 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:02:56.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:54.929047+0000 mgr.y (mgr.14550) 69 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:56.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:55.212882+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.b scrub starts 2024-02-19T21:02:56.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:55.213686+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.b scrub ok 2024-02-19T21:02:56.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:55.583976+0000 osd.7 (osd.7) 29 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:02:56.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:55.586851+0000 osd.7 (osd.7) 30 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:02:56.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:55.772892+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:57.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.014+0000 7f7ebb018700 1 -- 172.21.15.163:0/1270074485 >> v1:172.21.15.163:6790/0 conn(0x7f7eb40ff670 legacy=0x7f7eb4101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:57.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.014+0000 7f7ebb018700 1 -- 172.21.15.163:0/1270074485 shutdown_connections 2024-02-19T21:02:57.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.014+0000 7f7ebb018700 1 -- 172.21.15.163:0/1270074485 >> 172.21.15.163:0/1270074485 conn(0x7f7eb40f2470 msgr2=0x7f7eb40f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:57.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.014+0000 7f7ebb018700 1 -- 172.21.15.163:0/1270074485 shutdown_connections 2024-02-19T21:02:57.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.014+0000 7f7ebb018700 1 -- 172.21.15.163:0/1270074485 wait complete. 2024-02-19T21:02:57.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7ebb018700 1 librados: starting msgr at 2024-02-19T21:02:57.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7ebb018700 1 librados: starting objecter 2024-02-19T21:02:57.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7ebb018700 1 Processor -- start 2024-02-19T21:02:57.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7ebb018700 1 -- start start 2024-02-19T21:02:57.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7ebb018700 1 librados: setting wanted keys 2024-02-19T21:02:57.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7ebb018700 1 librados: calling monclient init 2024-02-19T21:02:57.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7ebb018700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7eb4119570 con 0x7f7eb41031b0 2024-02-19T21:02:57.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7ebb018700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7eb411a770 con 0x7f7eb40ff670 2024-02-19T21:02:57.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7ebb018700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7eb411b970 con 0x7f7eb40fc870 2024-02-19T21:02:57.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7eb95b5700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f7eb41031b0 0x7f7eb4117c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33826/0 (socket says 172.21.15.163:33826) 2024-02-19T21:02:57.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7eb8db4700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f7eb40fc870 0x7f7eb40609e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:57946/0 (socket says 172.21.15.163:57946) 2024-02-19T21:02:57.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7eb95b5700 1 -- 172.21.15.163:0/297334899 learned_addr learned my addr 172.21.15.163:0/297334899 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:57.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7eb1ffb700 10 client.?.objecter ms_handle_connect 0x7f7eb41031b0 2024-02-19T21:02:57.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7eb1ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:57.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7eb1ffb700 10 client.?.objecter ms_handle_connect 0x7f7eb40fc870 2024-02-19T21:02:57.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7eb1ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:57.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7eb1ffb700 10 client.?.objecter ms_handle_connect 0x7f7eb40ff670 2024-02-19T21:02:57.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.015+0000 7f7eb1ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:57.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073822835 0 0) 0x7f7eb411b970 con 0x7f7eb40fc870 2024-02-19T21:02:57.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e9c003030 con 0x7f7eb40fc870 2024-02-19T21:02:57.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883093956 0 0) 0x7f7eb4119570 con 0x7f7eb41031b0 2024-02-19T21:02:57.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb411b970 con 0x7f7eb41031b0 2024-02-19T21:02:57.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782029260 0 0) 0x7f7eb411a770 con 0x7f7eb40ff670 2024-02-19T21:02:57.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb4119570 con 0x7f7eb40ff670 2024-02-19T21:02:57.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 857741364 0 0) 0x7f7e9c003030 con 0x7f7eb40fc870 2024-02-19T21:02:57.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7eb411a770 con 0x7f7eb40fc870 2024-02-19T21:02:57.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7ea4001840 con 0x7f7eb40fc870 2024-02-19T21:02:57.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3925593189 0 0) 0x7f7eb411b970 con 0x7f7eb41031b0 2024-02-19T21:02:57.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e9c003030 con 0x7f7eb41031b0 2024-02-19T21:02:57.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7eac001d70 con 0x7f7eb41031b0 2024-02-19T21:02:57.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1756612627 0 0) 0x7f7eb4119570 con 0x7f7eb40ff670 2024-02-19T21:02:57.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7eb411b970 con 0x7f7eb40ff670 2024-02-19T21:02:57.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7ea80010b0 con 0x7f7eb40ff670 2024-02-19T21:02:57.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1076802323 0 0) 0x7f7eb411a770 con 0x7f7eb40fc870 2024-02-19T21:02:57.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 >> v1:172.21.15.193:6789/0 conn(0x7f7eb40ff670 legacy=0x7f7eb40610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:57.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 >> v1:172.21.15.163:6789/0 conn(0x7f7eb41031b0 legacy=0x7f7eb4117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:57.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb411cb70 con 0x7f7eb40fc870 2024-02-19T21:02:57.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7ebb018700 1 -- 172.21.15.163:0/297334899 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb4119760 con 0x7f7eb40fc870 2024-02-19T21:02:57.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f7ea4002210 con 0x7f7eb40fc870 2024-02-19T21:02:57.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7ea4005920 con 0x7f7eb40fc870 2024-02-19T21:02:57.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7ebb018700 10 client.34628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:57.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7ebb018700 1 -- 172.21.15.163:0/297334899 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb4119bd0 con 0x7f7eb40fc870 2024-02-19T21:02:57.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.016+0000 7f7ebb018700 1 librados: init done 2024-02-19T21:02:57.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.017+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f7ea4001d70 con 0x7f7eb40fc870 2024-02-19T21:02:57.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.017+0000 7f7ebb018700 1 -- 172.21.15.163:0/297334899 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb405f700 con 0x7f7eb40fc870 2024-02-19T21:02:57.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.018+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(82..82 src has 1..82) v4 ==== 6172+0+0 (unknown 3890964453 0 0) 0x7f7ea40783c0 con 0x7f7eb40fc870 2024-02-19T21:02:57.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.018+0000 7f7eb1ffb700 10 client.34628.objecter ms_dispatch 0x7f7eb4001550 osd_map(82..82 src has 1..82) v4 2024-02-19T21:02:57.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.018+0000 7f7eb1ffb700 3 client.34628.objecter handle_osd_map got epochs [82,82] > 0 2024-02-19T21:02:57.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.018+0000 7f7eb1ffb700 3 client.34628.objecter handle_osd_map decoding full epoch 82 2024-02-19T21:02:57.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.018+0000 7f7eb1ffb700 20 client.34628.objecter dump_active .. 0 homeless 2024-02-19T21:02:57.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.018+0000 7f7eb1ffb700 10 client.34628.objecter ms_handle_connect 0x7f7e9c063cd0 2024-02-19T21:02:57.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.022+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f7ea404bf10 con 0x7f7eb40fc870 2024-02-19T21:02:57.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.185+0000 7f7ebb018700 1 -- 172.21.15.163:0/297334899 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f7eb4047540 con 0x7f7eb40fc870 2024-02-19T21:02:57.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.185+0000 7f7eb1ffb700 1 -- 172.21.15.163:0/297334899 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 3109443431) 0x7f7ea404a320 con 0x7f7eb40fc870 2024-02-19T21:02:57.177 INFO:teuthology.orchestra.run.smithi163.stdout:77309411374 2024-02-19T21:02:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.187+0000 7f7ebb018700 10 librados: watch_flush enter 2024-02-19T21:02:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.187+0000 7f7ebb018700 10 librados: watch_flush exit 2024-02-19T21:02:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.187+0000 7f7ebb018700 20 client.34628.objecter shutdown clearing up homeless session... 2024-02-19T21:02:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.187+0000 7f7ebb018700 10 client.34628.objecter successfully canceled tick 2024-02-19T21:02:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.187+0000 7f7ebb018700 1 -- 172.21.15.163:0/297334899 >> v1:172.21.15.163:6800/1552514681 conn(0x7f7e9c063cd0 legacy=0x7f7e9c066190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.187+0000 7f7ebb018700 1 -- 172.21.15.163:0/297334899 >> v1:172.21.15.163:6790/0 conn(0x7f7eb40fc870 legacy=0x7f7eb40609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.187+0000 7f7ebb018700 1 -- 172.21.15.163:0/297334899 shutdown_connections 2024-02-19T21:02:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.187+0000 7f7ebb018700 1 -- 172.21.15.163:0/297334899 >> 172.21.15.163:0/297334899 conn(0x7f7eb40f2470 msgr2=0x7f7eb4106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.187+0000 7f7ebb018700 1 -- 172.21.15.163:0/297334899 shutdown_connections 2024-02-19T21:02:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.187+0000 7f7ebb018700 1 -- 172.21.15.163:0/297334899 wait complete. 2024-02-19T21:02:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:57.187+0000 7f7ebb018700 1 librados: shutdown 2024-02-19T21:02:57.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:55.564948+0000 osd.1 (osd.1) 33 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:02:57.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:55.568589+0000 osd.1 (osd.1) 34 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:02:57.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:56.463476+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:57.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:56.465075+0000 mon.b (mon.1) 75 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:02:57.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:56.465887+0000 mon.b (mon.1) 76 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:02:57.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:56.470473+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:57.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:56.509628+0000 osd.0 (osd.0) 27 : cluster [DBG] 5.5 deep-scrub starts 2024-02-19T21:02:57.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:56.510470+0000 osd.0 (osd.0) 28 : cluster [DBG] 5.5 deep-scrub ok 2024-02-19T21:02:57.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:57.186640+0000 mon.c (mon.2) 24 : audit [DBG] from='client.? v1:172.21.15.163:0/297334899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-19T21:02:57.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:55.564948+0000 osd.1 (osd.1) 33 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:02:57.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:55.568589+0000 osd.1 (osd.1) 34 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:02:57.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:56.463476+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:57.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:56.465075+0000 mon.b (mon.1) 75 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:02:57.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:56.465887+0000 mon.b (mon.1) 76 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:02:57.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:56.470473+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:57.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:56.509628+0000 osd.0 (osd.0) 27 : cluster [DBG] 5.5 deep-scrub starts 2024-02-19T21:02:57.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:56.510470+0000 osd.0 (osd.0) 28 : cluster [DBG] 5.5 deep-scrub ok 2024-02-19T21:02:57.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:57.186640+0000 mon.c (mon.2) 24 : audit [DBG] from='client.? v1:172.21.15.163:0/297334899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-19T21:02:57.764 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:55.564948+0000 osd.1 (osd.1) 33 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:02:57.764 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:55.568589+0000 osd.1 (osd.1) 34 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:02:57.764 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:56.463476+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:57.764 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:56.465075+0000 mon.b (mon.1) 75 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:02:57.764 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:56.465887+0000 mon.b (mon.1) 76 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:02:57.765 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:56.470473+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:02:57.765 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:56.509628+0000 osd.0 (osd.0) 27 : cluster [DBG] 5.5 deep-scrub starts 2024-02-19T21:02:57.765 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:56.510470+0000 osd.0 (osd.0) 28 : cluster [DBG] 5.5 deep-scrub ok 2024-02-19T21:02:57.765 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:57.186640+0000 mon.c (mon.2) 24 : audit [DBG] from='client.? v1:172.21.15.163:0/297334899' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-19T21:02:57.811 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411374 for osd.2 2024-02-19T21:02:57.811 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-19T21:02:58.062 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:02:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:02:58.096 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:02:58.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:56.535455+0000 osd.1 (osd.1) 35 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:02:58.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:56.539294+0000 osd.1 (osd.1) 36 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:02:58.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:56.929385+0000 mgr.y (mgr.14550) 70 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:58.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:57.017927+0000 osd.5 (osd.5) 19 : cluster [DBG] 4.15 scrub starts 2024-02-19T21:02:58.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:57.021399+0000 osd.5 (osd.5) 20 : cluster [DBG] 4.15 scrub ok 2024-02-19T21:02:58.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:57.502082+0000 osd.0 (osd.0) 29 : cluster [DBG] 3.1 deep-scrub starts 2024-02-19T21:02:58.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:57.505025+0000 osd.0 (osd.0) 30 : cluster [DBG] 3.1 deep-scrub ok 2024-02-19T21:02:58.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:57.549970+0000 osd.7 (osd.7) 31 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:02:58.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:57.552878+0000 osd.7 (osd.7) 32 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:02:58.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:57.817502+0000 osd.3 (osd.3) 29 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:02:58.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:57.819118+0000 osd.3 (osd.3) 30 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:02:58.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:57.996508+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.1 deep-scrub starts 2024-02-19T21:02:58.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:57.997417+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.1 deep-scrub ok 2024-02-19T21:02:58.622 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:56.535455+0000 osd.1 (osd.1) 35 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:02:58.622 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:56.539294+0000 osd.1 (osd.1) 36 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:02:58.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:56.929385+0000 mgr.y (mgr.14550) 70 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:58.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:57.017927+0000 osd.5 (osd.5) 19 : cluster [DBG] 4.15 scrub starts 2024-02-19T21:02:58.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:57.021399+0000 osd.5 (osd.5) 20 : cluster [DBG] 4.15 scrub ok 2024-02-19T21:02:58.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:57.502082+0000 osd.0 (osd.0) 29 : cluster [DBG] 3.1 deep-scrub starts 2024-02-19T21:02:58.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:57.505025+0000 osd.0 (osd.0) 30 : cluster [DBG] 3.1 deep-scrub ok 2024-02-19T21:02:58.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:57.549970+0000 osd.7 (osd.7) 31 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:02:58.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:57.552878+0000 osd.7 (osd.7) 32 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:02:58.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:57.817502+0000 osd.3 (osd.3) 29 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:02:58.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:57.819118+0000 osd.3 (osd.3) 30 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:02:58.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:57.996508+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.1 deep-scrub starts 2024-02-19T21:02:58.623 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:02:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:57.997417+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.1 deep-scrub ok 2024-02-19T21:02:58.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:56.535455+0000 osd.1 (osd.1) 35 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:02:58.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:56.539294+0000 osd.1 (osd.1) 36 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:02:58.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:56.929385+0000 mgr.y (mgr.14550) 70 : cluster [DBG] pgmap v52: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:02:58.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:57.017927+0000 osd.5 (osd.5) 19 : cluster [DBG] 4.15 scrub starts 2024-02-19T21:02:58.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:57.021399+0000 osd.5 (osd.5) 20 : cluster [DBG] 4.15 scrub ok 2024-02-19T21:02:58.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:57.502082+0000 osd.0 (osd.0) 29 : cluster [DBG] 3.1 deep-scrub starts 2024-02-19T21:02:58.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:57.505025+0000 osd.0 (osd.0) 30 : cluster [DBG] 3.1 deep-scrub ok 2024-02-19T21:02:58.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:57.549970+0000 osd.7 (osd.7) 31 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:02:58.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:57.552878+0000 osd.7 (osd.7) 32 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:02:58.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:57.817502+0000 osd.3 (osd.3) 29 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:02:58.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:57.819118+0000 osd.3 (osd.3) 30 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:02:58.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:57.996508+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.1 deep-scrub starts 2024-02-19T21:02:58.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:02:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:57.997417+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.1 deep-scrub ok 2024-02-19T21:02:59.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.882+0000 7ffa706e4700 1 -- 172.21.15.163:0/732501988 >> v1:172.21.15.163:6789/0 conn(0x7ffa680f94e0 legacy=0x7ffa681029b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:59.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.882+0000 7ffa706e4700 1 -- 172.21.15.163:0/732501988 shutdown_connections 2024-02-19T21:02:59.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.882+0000 7ffa706e4700 1 -- 172.21.15.163:0/732501988 >> 172.21.15.163:0/732501988 conn(0x7ffa680f24a0 msgr2=0x7ffa680f48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:02:59.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.882+0000 7ffa706e4700 1 -- 172.21.15.163:0/732501988 shutdown_connections 2024-02-19T21:02:59.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.882+0000 7ffa706e4700 1 -- 172.21.15.163:0/732501988 wait complete. 2024-02-19T21:02:59.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.882+0000 7ffa706e4700 1 librados: starting msgr at 2024-02-19T21:02:59.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.882+0000 7ffa706e4700 1 librados: starting objecter 2024-02-19T21:02:59.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.882+0000 7ffa706e4700 1 Processor -- start 2024-02-19T21:02:59.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.883+0000 7ffa706e4700 1 -- start start 2024-02-19T21:02:59.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.883+0000 7ffa706e4700 1 librados: setting wanted keys 2024-02-19T21:02:59.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.883+0000 7ffa706e4700 1 librados: calling monclient init 2024-02-19T21:02:59.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.883+0000 7ffa706e4700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa68119450 con 0x7ffa681042c0 2024-02-19T21:02:59.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.883+0000 7ffa706e4700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa6811a5f0 con 0x7ffa680f8930 2024-02-19T21:02:59.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.883+0000 7ffa706e4700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa6811b790 con 0x7ffa680f94e0 2024-02-19T21:02:59.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.883+0000 7ffa6dc7f700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7ffa680f94e0 0x7ffa68102870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:57974/0 (socket says 172.21.15.163:57974) 2024-02-19T21:02:59.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.883+0000 7ffa6dc7f700 1 -- 172.21.15.163:0/2303751425 learned_addr learned my addr 172.21.15.163:0/2303751425 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:02:59.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.883+0000 7ffa6ec81700 1 --1- 172.21.15.163:0/2303751425 >> v1:172.21.15.163:6789/0 conn(0x7ffa681042c0 0x7ffa68117d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33838/0 (socket says 172.21.15.163:33838) 2024-02-19T21:02:59.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.883+0000 7ffa5f7fe700 10 client.?.objecter ms_handle_connect 0x7ffa680f94e0 2024-02-19T21:02:59.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.883+0000 7ffa5f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:59.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.883+0000 7ffa5f7fe700 10 client.?.objecter ms_handle_connect 0x7ffa681042c0 2024-02-19T21:02:59.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.883+0000 7ffa5f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:59.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.883+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454824144 0 0) 0x7ffa6811b790 con 0x7ffa680f94e0 2024-02-19T21:02:59.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa54003030 con 0x7ffa680f94e0 2024-02-19T21:02:59.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 10 client.?.objecter ms_handle_connect 0x7ffa680f8930 2024-02-19T21:02:59.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:02:59.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429728478 0 0) 0x7ffa68119450 con 0x7ffa681042c0 2024-02-19T21:02:59.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa6811b790 con 0x7ffa681042c0 2024-02-19T21:02:59.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419896707 0 0) 0x7ffa6811a5f0 con 0x7ffa680f8930 2024-02-19T21:02:59.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa68119450 con 0x7ffa680f8930 2024-02-19T21:02:59.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2467673477 0 0) 0x7ffa6811b790 con 0x7ffa681042c0 2024-02-19T21:02:59.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa6811a5f0 con 0x7ffa681042c0 2024-02-19T21:02:59.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa60001c80 con 0x7ffa681042c0 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1179266081 0 0) 0x7ffa54003030 con 0x7ffa680f94e0 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa6811b790 con 0x7ffa680f94e0 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa58002050 con 0x7ffa680f94e0 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2946420302 0 0) 0x7ffa68119450 con 0x7ffa680f8930 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa54003030 con 0x7ffa680f8930 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa64001880 con 0x7ffa680f8930 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2434243199 0 0) 0x7ffa6811a5f0 con 0x7ffa681042c0 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 >> v1:172.21.15.163:6790/0 conn(0x7ffa680f94e0 legacy=0x7ffa68102870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 >> v1:172.21.15.193:6789/0 conn(0x7ffa680f8930 legacy=0x7ffa68102160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6811c990 con 0x7ffa681042c0 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa706e4700 1 -- 172.21.15.163:0/2303751425 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa68119640 con 0x7ffa681042c0 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa706e4700 10 client.14712.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa706e4700 1 -- 172.21.15.163:0/2303751425 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa68119ab0 con 0x7ffa681042c0 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa706e4700 1 librados: init done 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ffa600026c0 con 0x7ffa681042c0 2024-02-19T21:02:59.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.884+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa60005db0 con 0x7ffa681042c0 2024-02-19T21:02:59.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.885+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ffa60007060 con 0x7ffa681042c0 2024-02-19T21:02:59.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.886+0000 7ffa706e4700 1 -- 172.21.15.163:0/2303751425 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c004f20 con 0x7ffa681042c0 2024-02-19T21:02:59.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.886+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(82..82 src has 32..82) v4 ==== 6172+0+0 (unknown 3838378995 0 0) 0x7ffa60079720 con 0x7ffa681042c0 2024-02-19T21:02:59.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.886+0000 7ffa5f7fe700 10 client.14712.objecter ms_dispatch 0x7ffa68001550 osd_map(82..82 src has 32..82) v4 2024-02-19T21:02:59.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.886+0000 7ffa5f7fe700 3 client.14712.objecter handle_osd_map got epochs [82,82] > 0 2024-02-19T21:02:59.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.886+0000 7ffa5f7fe700 3 client.14712.objecter handle_osd_map decoding full epoch 82 2024-02-19T21:02:59.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.886+0000 7ffa5f7fe700 20 client.14712.objecter dump_active .. 0 homeless 2024-02-19T21:02:59.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.886+0000 7ffa5f7fe700 10 client.14712.objecter ms_handle_connect 0x7ffa54063d30 2024-02-19T21:02:59.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:02:59.890+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ffa6004d2d0 con 0x7ffa681042c0 2024-02-19T21:03:00.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:00.056+0000 7ffa706e4700 1 -- 172.21.15.163:0/2303751425 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ffa4c0051d0 con 0x7ffa681042c0 2024-02-19T21:03:00.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:00.057+0000 7ffa5f7fe700 1 -- 172.21.15.163:0/2303751425 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (unknown 383520633 0 1737839293) 0x7ffa6004b6e0 con 0x7ffa681042c0 2024-02-19T21:03:00.049 INFO:teuthology.orchestra.run.smithi163.stdout:107374182442 2024-02-19T21:03:00.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:00.059+0000 7ffa706e4700 10 librados: watch_flush enter 2024-02-19T21:03:00.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:00.059+0000 7ffa706e4700 10 librados: watch_flush exit 2024-02-19T21:03:00.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:00.059+0000 7ffa706e4700 20 client.14712.objecter shutdown clearing up homeless session... 2024-02-19T21:03:00.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:00.059+0000 7ffa706e4700 10 client.14712.objecter successfully canceled tick 2024-02-19T21:03:00.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:00.059+0000 7ffa706e4700 1 -- 172.21.15.163:0/2303751425 >> v1:172.21.15.163:6800/1552514681 conn(0x7ffa54063d30 legacy=0x7ffa540661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:00.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:00.059+0000 7ffa706e4700 1 -- 172.21.15.163:0/2303751425 >> v1:172.21.15.163:6789/0 conn(0x7ffa681042c0 legacy=0x7ffa68117d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:00.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:00.059+0000 7ffa706e4700 1 -- 172.21.15.163:0/2303751425 shutdown_connections 2024-02-19T21:03:00.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:00.059+0000 7ffa706e4700 1 -- 172.21.15.163:0/2303751425 >> 172.21.15.163:0/2303751425 conn(0x7ffa680f24a0 msgr2=0x7ffa680f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:00.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:00.059+0000 7ffa706e4700 1 -- 172.21.15.163:0/2303751425 shutdown_connections 2024-02-19T21:03:00.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:00.059+0000 7ffa706e4700 1 -- 172.21.15.163:0/2303751425 wait complete. 2024-02-19T21:03:00.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:00.059+0000 7ffa706e4700 1 librados: shutdown 2024-02-19T21:03:00.312 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:02:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:03:00.333 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:58.929684+0000 mgr.y (mgr.14550) 71 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:03:00.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:58.983565+0000 osd.5 (osd.5) 21 : cluster [DBG] 2.2 scrub starts 2024-02-19T21:03:00.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:02:58.986454+0000 osd.5 (osd.5) 22 : cluster [DBG] 2.2 scrub ok 2024-02-19T21:03:00.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:00.058427+0000 mon.a (mon.0) 704 : audit [DBG] from='client.? v1:172.21.15.163:0/2303751425' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-19T21:03:00.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:58.929684+0000 mgr.y (mgr.14550) 71 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:03:00.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:58.983565+0000 osd.5 (osd.5) 21 : cluster [DBG] 2.2 scrub starts 2024-02-19T21:03:00.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:02:58.986454+0000 osd.5 (osd.5) 22 : cluster [DBG] 2.2 scrub ok 2024-02-19T21:03:00.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:00.058427+0000 mon.a (mon.0) 704 : audit [DBG] from='client.? v1:172.21.15.163:0/2303751425' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-19T21:03:00.807 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182436 got 107374182442 for osd.3 2024-02-19T21:03:00.807 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-19T21:03:00.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:58.929684+0000 mgr.y (mgr.14550) 71 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:03:00.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:58.983565+0000 osd.5 (osd.5) 21 : cluster [DBG] 2.2 scrub starts 2024-02-19T21:03:00.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:02:58.986454+0000 osd.5 (osd.5) 22 : cluster [DBG] 2.2 scrub ok 2024-02-19T21:03:00.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:00.058427+0000 mon.a (mon.0) 704 : audit [DBG] from='client.? v1:172.21.15.163:0/2303751425' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-19T21:03:01.080 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:01.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:02:59.985156+0000 mgr.y (mgr.14550) 72 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:01.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:02:59.985156+0000 mgr.y (mgr.14550) 72 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:01.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:02:59.985156+0000 mgr.y (mgr.14550) 72 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:02.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:00.567723+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:03:02.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:00.570674+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:03:02.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:00.930315+0000 mgr.y (mgr.14550) 73 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:03:02.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:01.001683+0000 osd.5 (osd.5) 23 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:03:02.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:01.005439+0000 osd.5 (osd.5) 24 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:03:02.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:01.168384+0000 osd.2 (osd.2) 33 : cluster [DBG] 5.a scrub starts 2024-02-19T21:03:02.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:01.169272+0000 osd.2 (osd.2) 34 : cluster [DBG] 5.a scrub ok 2024-02-19T21:03:02.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:01.493765+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:03:02.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:01.496829+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:03:02.559 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:00.567723+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:03:02.560 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:00.570674+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:03:02.560 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:00.930315+0000 mgr.y (mgr.14550) 73 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:03:02.560 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:01.001683+0000 osd.5 (osd.5) 23 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:03:02.560 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:01.005439+0000 osd.5 (osd.5) 24 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:03:02.560 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:01.168384+0000 osd.2 (osd.2) 33 : cluster [DBG] 5.a scrub starts 2024-02-19T21:03:02.560 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:01.169272+0000 osd.2 (osd.2) 34 : cluster [DBG] 5.a scrub ok 2024-02-19T21:03:02.560 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:01.493765+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:03:02.560 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:01.496829+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:03:02.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.818+0000 7f4b9857e700 1 -- 172.21.15.163:0/58261190 >> v1:172.21.15.163:6790/0 conn(0x7f4b90109000 legacy=0x7f4b9010b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:02.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.818+0000 7f4b9857e700 1 -- 172.21.15.163:0/58261190 shutdown_connections 2024-02-19T21:03:02.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.818+0000 7f4b9857e700 1 -- 172.21.15.163:0/58261190 >> 172.21.15.163:0/58261190 conn(0x7f4b90068e80 msgr2=0x7f4b90066170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:02.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.819+0000 7f4b9857e700 1 -- 172.21.15.163:0/58261190 shutdown_connections 2024-02-19T21:03:02.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.819+0000 7f4b9857e700 1 -- 172.21.15.163:0/58261190 wait complete. 2024-02-19T21:03:02.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.819+0000 7f4b9857e700 1 librados: starting msgr at 2024-02-19T21:03:02.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.819+0000 7f4b9857e700 1 librados: starting objecter 2024-02-19T21:03:02.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.819+0000 7f4b9857e700 1 Processor -- start 2024-02-19T21:03:02.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.819+0000 7f4b9857e700 1 -- start start 2024-02-19T21:03:02.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.819+0000 7f4b9857e700 1 librados: setting wanted keys 2024-02-19T21:03:02.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.819+0000 7f4b9857e700 1 librados: calling monclient init 2024-02-19T21:03:02.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:00.567723+0000 osd.0 (osd.0) 31 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:03:02.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:00.570674+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:03:02.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:00.930315+0000 mgr.y (mgr.14550) 73 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:03:02.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:01.001683+0000 osd.5 (osd.5) 23 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:03:02.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:01.005439+0000 osd.5 (osd.5) 24 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:03:02.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:01.168384+0000 osd.2 (osd.2) 33 : cluster [DBG] 5.a scrub starts 2024-02-19T21:03:02.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:01.169272+0000 osd.2 (osd.2) 34 : cluster [DBG] 5.a scrub ok 2024-02-19T21:03:02.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:01.493765+0000 osd.7 (osd.7) 33 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:03:02.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:01.496829+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:03:02.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b9857e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b9010bf20 con 0x7f4b90109000 2024-02-19T21:03:02.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b9857e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b90123030 con 0x7f4b9010cb10 2024-02-19T21:03:02.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b9857e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b901241d0 con 0x7f4b90060d40 2024-02-19T21:03:02.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b95b19700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f4b90109000 0x7f4b9011d1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33852/0 (socket says 172.21.15.163:33852) 2024-02-19T21:03:02.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b9631a700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f4b90060d40 0x7f4b9010b330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:57994/0 (socket says 172.21.15.163:57994) 2024-02-19T21:03:02.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b95b19700 1 -- 172.21.15.163:0/3667438462 learned_addr learned my addr 172.21.15.163:0/3667438462 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:02.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 10 client.?.objecter ms_handle_connect 0x7f4b90109000 2024-02-19T21:03:02.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:02.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 10 client.?.objecter ms_handle_connect 0x7f4b90060d40 2024-02-19T21:03:02.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:02.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990873684 0 0) 0x7f4b9010bf20 con 0x7f4b90109000 2024-02-19T21:03:02.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b7c002220 con 0x7f4b90109000 2024-02-19T21:03:02.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 10 client.?.objecter ms_handle_connect 0x7f4b9010cb10 2024-02-19T21:03:02.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:02.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112965320 0 0) 0x7f4b901241d0 con 0x7f4b90060d40 2024-02-19T21:03:02.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b9010bf20 con 0x7f4b90060d40 2024-02-19T21:03:02.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289423277 0 0) 0x7f4b90123030 con 0x7f4b9010cb10 2024-02-19T21:03:02.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b901241d0 con 0x7f4b9010cb10 2024-02-19T21:03:02.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3697754030 0 0) 0x7f4b7c002220 con 0x7f4b90109000 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b90123030 con 0x7f4b90109000 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4b80002ea0 con 0x7f4b90109000 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4258094550 0 0) 0x7f4b9010bf20 con 0x7f4b90060d40 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b7c002220 con 0x7f4b90060d40 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4b8c0019f0 con 0x7f4b90060d40 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1545028599 0 0) 0x7f4b901241d0 con 0x7f4b9010cb10 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b9010bf20 con 0x7f4b9010cb10 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4b88001c40 con 0x7f4b9010cb10 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.820+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3998442400 0 0) 0x7f4b90123030 con 0x7f4b90109000 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.821+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 >> v1:172.21.15.163:6790/0 conn(0x7f4b90060d40 legacy=0x7f4b9010b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.821+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 >> v1:172.21.15.193:6789/0 conn(0x7f4b9010cb10 legacy=0x7f4b90120900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.821+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b901253d0 con 0x7f4b90109000 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.821+0000 7f4b9857e700 1 -- 172.21.15.163:0/3667438462 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b901243c0 con 0x7f4b90109000 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.821+0000 7f4b9857e700 10 client.14718.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.821+0000 7f4b9857e700 1 -- 172.21.15.163:0/3667438462 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b90124830 con 0x7f4b90109000 2024-02-19T21:03:02.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.821+0000 7f4b9857e700 1 librados: init done 2024-02-19T21:03:02.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.821+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f4b800026f0 con 0x7f4b90109000 2024-02-19T21:03:02.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.821+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4b800038c0 con 0x7f4b90109000 2024-02-19T21:03:02.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.822+0000 7f4b9857e700 1 -- 172.21.15.163:0/3667438462 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b74004f20 con 0x7f4b90109000 2024-02-19T21:03:02.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.822+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f4b80003140 con 0x7f4b90109000 2024-02-19T21:03:02.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.822+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(82..82 src has 32..82) v4 ==== 6172+0+0 (unknown 3838378995 0 0) 0x7f4b80078800 con 0x7f4b90109000 2024-02-19T21:03:02.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.822+0000 7f4b877fe700 10 client.14718.objecter ms_dispatch 0x7f4b90001550 osd_map(82..82 src has 32..82) v4 2024-02-19T21:03:02.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.822+0000 7f4b877fe700 3 client.14718.objecter handle_osd_map got epochs [82,82] > 0 2024-02-19T21:03:02.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.822+0000 7f4b877fe700 3 client.14718.objecter handle_osd_map decoding full epoch 82 2024-02-19T21:03:02.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.823+0000 7f4b877fe700 20 client.14718.objecter dump_active .. 0 homeless 2024-02-19T21:03:02.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.823+0000 7f4b877fe700 10 client.14718.objecter ms_handle_connect 0x7f4b7c063d80 2024-02-19T21:03:02.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.827+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f4b8004c470 con 0x7f4b90109000 2024-02-19T21:03:02.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.990+0000 7f4b9857e700 1 -- 172.21.15.163:0/3667438462 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4b740051d0 con 0x7f4b90109000 2024-02-19T21:03:02.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.991+0000 7f4b877fe700 1 -- 172.21.15.163:0/3667438462 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (unknown 1823589948 0 963721773) 0x7f4b8004a8e0 con 0x7f4b90109000 2024-02-19T21:03:02.983 INFO:teuthology.orchestra.run.smithi163.stdout:133143986215 2024-02-19T21:03:02.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.992+0000 7f4b9857e700 10 librados: watch_flush enter 2024-02-19T21:03:02.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.992+0000 7f4b9857e700 10 librados: watch_flush exit 2024-02-19T21:03:02.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.992+0000 7f4b9857e700 20 client.14718.objecter shutdown clearing up homeless session... 2024-02-19T21:03:02.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.992+0000 7f4b9857e700 10 client.14718.objecter successfully canceled tick 2024-02-19T21:03:02.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.992+0000 7f4b9857e700 1 -- 172.21.15.163:0/3667438462 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4b7c063d80 legacy=0x7f4b7c066240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:02.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.993+0000 7f4b9857e700 1 -- 172.21.15.163:0/3667438462 >> v1:172.21.15.163:6789/0 conn(0x7f4b90109000 legacy=0x7f4b9011d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:02.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.993+0000 7f4b9857e700 1 -- 172.21.15.163:0/3667438462 shutdown_connections 2024-02-19T21:03:02.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.993+0000 7f4b9857e700 1 -- 172.21.15.163:0/3667438462 >> 172.21.15.163:0/3667438462 conn(0x7f4b90068e80 msgr2=0x7f4b90106fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:02.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.993+0000 7f4b9857e700 1 -- 172.21.15.163:0/3667438462 shutdown_connections 2024-02-19T21:03:02.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.993+0000 7f4b9857e700 1 -- 172.21.15.163:0/3667438462 wait complete. 2024-02-19T21:03:02.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:02.993+0000 7f4b9857e700 1 librados: shutdown 2024-02-19T21:03:03.701 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:01.529189+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.4 deep-scrub starts 2024-02-19T21:03:03.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:01.530099+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.4 deep-scrub ok 2024-02-19T21:03:03.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:02.992466+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? v1:172.21.15.163:0/3667438462' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-19T21:03:03.702 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:03:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:03:03] "GET /metrics HTTP/1.1" 200 172155 "" "Prometheus/2.33.4" 2024-02-19T21:03:03.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:01.529189+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.4 deep-scrub starts 2024-02-19T21:03:03.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:01.530099+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.4 deep-scrub ok 2024-02-19T21:03:03.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:02.992466+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? v1:172.21.15.163:0/3667438462' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-19T21:03:03.721 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986209 got 133143986215 for osd.4 2024-02-19T21:03:03.722 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-19T21:03:03.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:01.529189+0000 osd.1 (osd.1) 37 : cluster [DBG] 2.4 deep-scrub starts 2024-02-19T21:03:03.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:01.530099+0000 osd.1 (osd.1) 38 : cluster [DBG] 2.4 deep-scrub ok 2024-02-19T21:03:03.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:02.992466+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? v1:172.21.15.163:0/3667438462' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-19T21:03:03.992 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:04.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:02.930657+0000 mgr.y (mgr.14550) 74 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:04.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:03.497042+0000 osd.7 (osd.7) 35 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:03:04.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:03.499954+0000 osd.7 (osd.7) 36 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:03:04.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:03.775544+0000 osd.3 (osd.3) 31 : cluster [DBG] 5.13 deep-scrub starts 2024-02-19T21:03:04.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:03.776389+0000 osd.3 (osd.3) 32 : cluster [DBG] 5.13 deep-scrub ok 2024-02-19T21:03:04.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:04.082406+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.c deep-scrub starts 2024-02-19T21:03:04.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:04.086277+0000 osd.4 (osd.4) 32 : cluster [DBG] 4.c deep-scrub ok 2024-02-19T21:03:04.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:02.930657+0000 mgr.y (mgr.14550) 74 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:04.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:03.497042+0000 osd.7 (osd.7) 35 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:03:04.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:03.499954+0000 osd.7 (osd.7) 36 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:03:04.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:03.775544+0000 osd.3 (osd.3) 31 : cluster [DBG] 5.13 deep-scrub starts 2024-02-19T21:03:04.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:03.776389+0000 osd.3 (osd.3) 32 : cluster [DBG] 5.13 deep-scrub ok 2024-02-19T21:03:04.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:04.082406+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.c deep-scrub starts 2024-02-19T21:03:04.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:04.086277+0000 osd.4 (osd.4) 32 : cluster [DBG] 4.c deep-scrub ok 2024-02-19T21:03:04.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:02.930657+0000 mgr.y (mgr.14550) 74 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:04.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:03.497042+0000 osd.7 (osd.7) 35 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:03:04.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:03.499954+0000 osd.7 (osd.7) 36 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:03:04.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:03.775544+0000 osd.3 (osd.3) 31 : cluster [DBG] 5.13 deep-scrub starts 2024-02-19T21:03:04.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:03.776389+0000 osd.3 (osd.3) 32 : cluster [DBG] 5.13 deep-scrub ok 2024-02-19T21:03:04.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:04.082406+0000 osd.4 (osd.4) 31 : cluster [DBG] 4.c deep-scrub starts 2024-02-19T21:03:04.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:04.086277+0000 osd.4 (osd.4) 32 : cluster [DBG] 4.c deep-scrub ok 2024-02-19T21:03:05.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.745+0000 7f347f72a700 1 -- 172.21.15.163:0/3317865997 >> v1:172.21.15.163:6790/0 conn(0x7f34780f88c0 legacy=0x7f34780f8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:05.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.745+0000 7f347f72a700 1 -- 172.21.15.163:0/3317865997 shutdown_connections 2024-02-19T21:03:05.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.745+0000 7f347f72a700 1 -- 172.21.15.163:0/3317865997 >> 172.21.15.163:0/3317865997 conn(0x7f34780f23d0 msgr2=0x7f34780f47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:05.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.745+0000 7f347f72a700 1 -- 172.21.15.163:0/3317865997 shutdown_connections 2024-02-19T21:03:05.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.745+0000 7f347f72a700 1 -- 172.21.15.163:0/3317865997 wait complete. 2024-02-19T21:03:05.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.745+0000 7f347f72a700 1 librados: starting msgr at 2024-02-19T21:03:05.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.745+0000 7f347f72a700 1 librados: starting objecter 2024-02-19T21:03:05.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.745+0000 7f347f72a700 1 Processor -- start 2024-02-19T21:03:05.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.745+0000 7f347f72a700 1 -- start start 2024-02-19T21:03:05.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.745+0000 7f347f72a700 1 librados: setting wanted keys 2024-02-19T21:03:05.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.745+0000 7f347f72a700 1 librados: calling monclient init 2024-02-19T21:03:05.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f347f72a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3478102a20 con 0x7f34780f88c0 2024-02-19T21:03:05.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f347f72a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3478102bf0 con 0x7f34780f9470 2024-02-19T21:03:05.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f347f72a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3478102de0 con 0x7f3478104250 2024-02-19T21:03:05.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f347d4c6700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f34780f88c0 0x7f3478102280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33568/0 (socket says 172.21.15.163:33568) 2024-02-19T21:03:05.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f347d4c6700 1 -- 172.21.15.163:0/3709049629 learned_addr learned my addr 172.21.15.163:0/3709049629 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 10 client.?.objecter ms_handle_connect 0x7f3478104250 2024-02-19T21:03:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 10 client.?.objecter ms_handle_connect 0x7f34780f88c0 2024-02-19T21:03:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 10 client.?.objecter ms_handle_connect 0x7f34780f9470 2024-02-19T21:03:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562422960 0 0) 0x7f3478102de0 con 0x7f3478104250 2024-02-19T21:03:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3464003030 con 0x7f3478104250 2024-02-19T21:03:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232756142 0 0) 0x7f3478102a20 con 0x7f34780f88c0 2024-02-19T21:03:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3478102de0 con 0x7f34780f88c0 2024-02-19T21:03:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439220730 0 0) 0x7f3478102bf0 con 0x7f34780f9470 2024-02-19T21:03:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3478102a20 con 0x7f34780f9470 2024-02-19T21:03:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 111367326 0 0) 0x7f3464003030 con 0x7f3478104250 2024-02-19T21:03:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3478102bf0 con 0x7f3478104250 2024-02-19T21:03:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3470001d70 con 0x7f3478104250 2024-02-19T21:03:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1450275986 0 0) 0x7f3478102de0 con 0x7f34780f88c0 2024-02-19T21:03:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3464003030 con 0x7f34780f88c0 2024-02-19T21:03:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f34680020f0 con 0x7f34780f88c0 2024-02-19T21:03:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1866153029 0 0) 0x7f3478102a20 con 0x7f34780f9470 2024-02-19T21:03:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3478102de0 con 0x7f34780f9470 2024-02-19T21:03:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.746+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f34740018c0 con 0x7f34780f9470 2024-02-19T21:03:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.747+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3885287185 0 0) 0x7f3478102bf0 con 0x7f3478104250 2024-02-19T21:03:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.747+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 >> v1:172.21.15.193:6789/0 conn(0x7f34780f9470 legacy=0x7f3478114ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.747+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 >> v1:172.21.15.163:6789/0 conn(0x7f34780f88c0 legacy=0x7f3478102280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.747+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347811cb20 con 0x7f3478104250 2024-02-19T21:03:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.747+0000 7f347f72a700 1 -- 172.21.15.163:0/3709049629 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3478119b50 con 0x7f3478104250 2024-02-19T21:03:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.747+0000 7f347f72a700 10 client.34646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.747+0000 7f347f72a700 1 -- 172.21.15.163:0/3709049629 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3478119fc0 con 0x7f3478104250 2024-02-19T21:03:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.747+0000 7f347f72a700 1 librados: init done 2024-02-19T21:03:05.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:04.483112+0000 osd.7 (osd.7) 37 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:03:05.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:04.486062+0000 osd.7 (osd.7) 38 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:03:05.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:05.072779+0000 osd.4 (osd.4) 33 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:03:05.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:05.073658+0000 osd.4 (osd.4) 34 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:03:05.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:04.483112+0000 osd.7 (osd.7) 37 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:03:05.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:04.486062+0000 osd.7 (osd.7) 38 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:03:05.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:05.072779+0000 osd.4 (osd.4) 33 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:03:05.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:05.073658+0000 osd.4 (osd.4) 34 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:03:05.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.747+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3470002620 con 0x7f3478104250 2024-02-19T21:03:05.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.747+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f34700036b0 con 0x7f3478104250 2024-02-19T21:03:05.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.748+0000 7f347f72a700 1 -- 172.21.15.163:0/3709049629 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f345c004f20 con 0x7f3478104250 2024-02-19T21:03:05.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.748+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f3470003810 con 0x7f3478104250 2024-02-19T21:03:05.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.749+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(82..82 src has 32..82) v4 ==== 6172+0+0 (unknown 3838378995 0 0) 0x7f3470075f50 con 0x7f3478104250 2024-02-19T21:03:05.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.749+0000 7f346e7fc700 10 client.34646.objecter ms_dispatch 0x7f3478001550 osd_map(82..82 src has 32..82) v4 2024-02-19T21:03:05.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.749+0000 7f346e7fc700 3 client.34646.objecter handle_osd_map got epochs [82,82] > 0 2024-02-19T21:03:05.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.749+0000 7f346e7fc700 3 client.34646.objecter handle_osd_map decoding full epoch 82 2024-02-19T21:03:05.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.749+0000 7f346e7fc700 20 client.34646.objecter dump_active .. 0 homeless 2024-02-19T21:03:05.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.749+0000 7f346e7fc700 10 client.34646.objecter ms_handle_connect 0x7f3464063ce0 2024-02-19T21:03:05.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.753+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3470049c50 con 0x7f3478104250 2024-02-19T21:03:05.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:04.483112+0000 osd.7 (osd.7) 37 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:03:05.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:04.486062+0000 osd.7 (osd.7) 38 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:03:05.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:05.072779+0000 osd.4 (osd.4) 33 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:03:05.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:05.073658+0000 osd.4 (osd.4) 34 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:03:05.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.916+0000 7f347f72a700 1 -- 172.21.15.163:0/3709049629 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f345c0051d0 con 0x7f3478104250 2024-02-19T21:03:05.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.916+0000 7f346e7fc700 1 -- 172.21.15.163:0/3709049629 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (unknown 2131975755 0 2597790923) 0x7f3470048060 con 0x7f3478104250 2024-02-19T21:03:05.908 INFO:teuthology.orchestra.run.smithi163.stdout:158913789987 2024-02-19T21:03:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.918+0000 7f347f72a700 10 librados: watch_flush enter 2024-02-19T21:03:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.918+0000 7f347f72a700 10 librados: watch_flush exit 2024-02-19T21:03:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.918+0000 7f347f72a700 20 client.34646.objecter shutdown clearing up homeless session... 2024-02-19T21:03:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.918+0000 7f347f72a700 10 client.34646.objecter successfully canceled tick 2024-02-19T21:03:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.918+0000 7f347f72a700 1 -- 172.21.15.163:0/3709049629 >> v1:172.21.15.163:6800/1552514681 conn(0x7f3464063ce0 legacy=0x7f34640661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.918+0000 7f347f72a700 1 -- 172.21.15.163:0/3709049629 >> v1:172.21.15.163:6790/0 conn(0x7f3478104250 legacy=0x7f34781183d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.918+0000 7f347f72a700 1 -- 172.21.15.163:0/3709049629 shutdown_connections 2024-02-19T21:03:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.918+0000 7f347f72a700 1 -- 172.21.15.163:0/3709049629 >> 172.21.15.163:0/3709049629 conn(0x7f34780f23d0 msgr2=0x7f34781076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:05.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.918+0000 7f347f72a700 1 -- 172.21.15.163:0/3709049629 shutdown_connections 2024-02-19T21:03:05.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.918+0000 7f347f72a700 1 -- 172.21.15.163:0/3709049629 wait complete. 2024-02-19T21:03:05.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:05.918+0000 7f347f72a700 1 librados: shutdown 2024-02-19T21:03:06.501 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:04.643941+0000 osd.0 (osd.0) 33 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:03:06.502 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:04.644684+0000 osd.0 (osd.0) 34 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:03:06.502 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:04.931377+0000 mgr.y (mgr.14550) 75 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:03:06.502 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:05.002186+0000 osd.5 (osd.5) 25 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:03:06.502 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:05.005284+0000 osd.5 (osd.5) 26 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:03:06.502 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:05.733019+0000 osd.3 (osd.3) 33 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:03:06.502 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:05.733973+0000 osd.3 (osd.3) 34 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:03:06.502 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:05.917824+0000 mon.c (mon.2) 25 : audit [DBG] from='client.? v1:172.21.15.163:0/3709049629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-19T21:03:06.502 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:04.643941+0000 osd.0 (osd.0) 33 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:03:06.502 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:04.644684+0000 osd.0 (osd.0) 34 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:03:06.502 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:04.931377+0000 mgr.y (mgr.14550) 75 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:03:06.502 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:05.002186+0000 osd.5 (osd.5) 25 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:03:06.503 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:05.005284+0000 osd.5 (osd.5) 26 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:03:06.503 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:05.733019+0000 osd.3 (osd.3) 33 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:03:06.503 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:05.733973+0000 osd.3 (osd.3) 34 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:03:06.503 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:05.917824+0000 mon.c (mon.2) 25 : audit [DBG] from='client.? v1:172.21.15.163:0/3709049629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-19T21:03:06.522 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789982 got 158913789987 for osd.5 2024-02-19T21:03:06.522 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-19T21:03:06.792 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:06.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:04.643941+0000 osd.0 (osd.0) 33 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:03:06.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:04.644684+0000 osd.0 (osd.0) 34 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:03:06.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:04.931377+0000 mgr.y (mgr.14550) 75 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:03:06.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:05.002186+0000 osd.5 (osd.5) 25 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:03:06.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:05.005284+0000 osd.5 (osd.5) 26 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:03:06.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:05.733019+0000 osd.3 (osd.3) 33 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:03:06.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:05.733973+0000 osd.3 (osd.3) 34 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:03:06.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:05.917824+0000 mon.c (mon.2) 25 : audit [DBG] from='client.? v1:172.21.15.163:0/3709049629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-19T21:03:07.365 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:05.997752+0000 osd.5 (osd.5) 27 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:03:07.365 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:06.000727+0000 osd.5 (osd.5) 28 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:03:07.365 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:06.455180+0000 osd.1 (osd.1) 39 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:03:07.365 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:06.456205+0000 osd.1 (osd.1) 40 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:03:07.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:05.997752+0000 osd.5 (osd.5) 27 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:03:07.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:06.000727+0000 osd.5 (osd.5) 28 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:03:07.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:06.455180+0000 osd.1 (osd.1) 39 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:03:07.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:06.456205+0000 osd.1 (osd.1) 40 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:03:07.764 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:05.997752+0000 osd.5 (osd.5) 27 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:03:07.764 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:06.000727+0000 osd.5 (osd.5) 28 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:03:07.764 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:06.455180+0000 osd.1 (osd.1) 39 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:03:07.764 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:06.456205+0000 osd.1 (osd.1) 40 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:03:08.062 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:03:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:03:08.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.442+0000 7fd2f2ae7700 1 -- 172.21.15.163:0/1523837033 >> v1:172.21.15.163:6789/0 conn(0x7fd2ec0f94b0 legacy=0x7fd2ec102a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:08.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.442+0000 7fd2f2ae7700 1 -- 172.21.15.163:0/1523837033 shutdown_connections 2024-02-19T21:03:08.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.442+0000 7fd2f2ae7700 1 -- 172.21.15.163:0/1523837033 >> 172.21.15.163:0/1523837033 conn(0x7fd2ec0f2470 msgr2=0x7fd2ec0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:08.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.442+0000 7fd2f2ae7700 1 -- 172.21.15.163:0/1523837033 shutdown_connections 2024-02-19T21:03:08.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.442+0000 7fd2f2ae7700 1 -- 172.21.15.163:0/1523837033 wait complete. 2024-02-19T21:03:08.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.442+0000 7fd2f2ae7700 1 librados: starting msgr at 2024-02-19T21:03:08.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.442+0000 7fd2f2ae7700 1 librados: starting objecter 2024-02-19T21:03:08.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2f2ae7700 1 Processor -- start 2024-02-19T21:03:08.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2f2ae7700 1 -- start start 2024-02-19T21:03:08.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2f2ae7700 1 librados: setting wanted keys 2024-02-19T21:03:08.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2f2ae7700 1 librados: calling monclient init 2024-02-19T21:03:08.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2f2ae7700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd2ec102cb0 con 0x7fd2ec104290 2024-02-19T21:03:08.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2f2ae7700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd2ec102e80 con 0x7fd2ec0f94b0 2024-02-19T21:03:08.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2f2ae7700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd2ec11bb50 con 0x7fd2ec0f89c0 2024-02-19T21:03:08.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2f0883700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fd2ec0f89c0 0x7fd2ec102510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38622/0 (socket says 172.21.15.163:38622) 2024-02-19T21:03:08.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2f0883700 1 -- 172.21.15.163:0/55560802 learned_addr learned my addr 172.21.15.163:0/55560802 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:08.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 10 client.?.objecter ms_handle_connect 0x7fd2ec0f89c0 2024-02-19T21:03:08.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:08.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 10 client.?.objecter ms_handle_connect 0x7fd2ec104290 2024-02-19T21:03:08.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:08.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 10 client.?.objecter ms_handle_connect 0x7fd2ec0f94b0 2024-02-19T21:03:08.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:08.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594379929 0 0) 0x7fd2ec11bb50 con 0x7fd2ec0f89c0 2024-02-19T21:03:08.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2dc003030 con 0x7fd2ec0f89c0 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518305942 0 0) 0x7fd2ec102cb0 con 0x7fd2ec104290 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ec11bb50 con 0x7fd2ec104290 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353827959 0 0) 0x7fd2ec102e80 con 0x7fd2ec0f94b0 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ec102cb0 con 0x7fd2ec0f94b0 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 959089630 0 0) 0x7fd2ec11bb50 con 0x7fd2ec104290 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2ec102e80 con 0x7fd2ec104290 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3323573420 0 0) 0x7fd2dc003030 con 0x7fd2ec0f89c0 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2ec11bb50 con 0x7fd2ec0f89c0 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.443+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd2e4001d30 con 0x7fd2ec104290 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.444+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd2e0001980 con 0x7fd2ec0f89c0 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.444+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1083349921 0 0) 0x7fd2ec102cb0 con 0x7fd2ec0f94b0 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.444+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2dc003030 con 0x7fd2ec0f94b0 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.444+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd2d8001f90 con 0x7fd2ec0f94b0 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.444+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 261388002 0 0) 0x7fd2ec102e80 con 0x7fd2ec104290 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.444+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 >> v1:172.21.15.163:6790/0 conn(0x7fd2ec0f89c0 legacy=0x7fd2ec102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:08.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.444+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 >> v1:172.21.15.193:6789/0 conn(0x7fd2ec0f94b0 legacy=0x7fd2ec114ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:08.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.444+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec11ccf0 con 0x7fd2ec104290 2024-02-19T21:03:08.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.444+0000 7fd2f2ae7700 1 -- 172.21.15.163:0/55560802 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ec11bd40 con 0x7fd2ec104290 2024-02-19T21:03:08.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.444+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fd2e40026b0 con 0x7fd2ec104290 2024-02-19T21:03:08.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.444+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd2e4003670 con 0x7fd2ec104290 2024-02-19T21:03:08.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.444+0000 7fd2f2ae7700 10 client.14730.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:08.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.444+0000 7fd2f2ae7700 1 -- 172.21.15.163:0/55560802 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ec11c1b0 con 0x7fd2ec104290 2024-02-19T21:03:08.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.444+0000 7fd2f2ae7700 1 librados: init done 2024-02-19T21:03:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.445+0000 7fd2f2ae7700 1 -- 172.21.15.163:0/55560802 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ec05f700 con 0x7fd2ec104290 2024-02-19T21:03:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.445+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fd2e4002210 con 0x7fd2ec104290 2024-02-19T21:03:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.445+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(82..82 src has 32..82) v4 ==== 6172+0+0 (unknown 3838378995 0 0) 0x7fd2e4076030 con 0x7fd2ec104290 2024-02-19T21:03:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.445+0000 7fd2e9f76700 10 client.14730.objecter ms_dispatch 0x7fd2ec001550 osd_map(82..82 src has 32..82) v4 2024-02-19T21:03:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.445+0000 7fd2e9f76700 3 client.14730.objecter handle_osd_map got epochs [82,82] > 0 2024-02-19T21:03:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.445+0000 7fd2e9f76700 3 client.14730.objecter handle_osd_map decoding full epoch 82 2024-02-19T21:03:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.446+0000 7fd2e9f76700 20 client.14730.objecter dump_active .. 0 homeless 2024-02-19T21:03:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.446+0000 7fd2e9f76700 10 client.14730.objecter ms_handle_connect 0x7fd2dc063cb0 2024-02-19T21:03:08.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.450+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd2e4049bc0 con 0x7fd2ec104290 2024-02-19T21:03:08.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:06.931715+0000 mgr.y (mgr.14550) 76 : cluster [DBG] pgmap v57: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:03:08.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:06.988283+0000 osd.5 (osd.5) 29 : cluster [DBG] 5.f scrub starts 2024-02-19T21:03:08.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:06.991195+0000 osd.5 (osd.5) 30 : cluster [DBG] 5.f scrub ok 2024-02-19T21:03:08.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:07.470088+0000 osd.1 (osd.1) 41 : cluster [DBG] 6.1 deep-scrub starts 2024-02-19T21:03:08.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:07.473065+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.1 deep-scrub ok 2024-02-19T21:03:08.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:08.044630+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:03:08.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:08.046636+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:03:08.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:06.931715+0000 mgr.y (mgr.14550) 76 : cluster [DBG] pgmap v57: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:03:08.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:06.988283+0000 osd.5 (osd.5) 29 : cluster [DBG] 5.f scrub starts 2024-02-19T21:03:08.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:06.991195+0000 osd.5 (osd.5) 30 : cluster [DBG] 5.f scrub ok 2024-02-19T21:03:08.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:07.470088+0000 osd.1 (osd.1) 41 : cluster [DBG] 6.1 deep-scrub starts 2024-02-19T21:03:08.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:07.473065+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.1 deep-scrub ok 2024-02-19T21:03:08.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:08.044630+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:03:08.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:08.046636+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:03:08.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.613+0000 7fd2f2ae7700 1 -- 172.21.15.163:0/55560802 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd2ec109fb0 con 0x7fd2ec104290 2024-02-19T21:03:08.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.613+0000 7fd2e9f76700 1 -- 172.21.15.163:0/55560802 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (unknown 1274345170 0 2411572773) 0x7fd2e4047fd0 con 0x7fd2ec104290 2024-02-19T21:03:08.605 INFO:teuthology.orchestra.run.smithi163.stdout:193273528350 2024-02-19T21:03:08.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.615+0000 7fd2f2ae7700 10 librados: watch_flush enter 2024-02-19T21:03:08.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.615+0000 7fd2f2ae7700 10 librados: watch_flush exit 2024-02-19T21:03:08.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.615+0000 7fd2f2ae7700 20 client.14730.objecter shutdown clearing up homeless session... 2024-02-19T21:03:08.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.615+0000 7fd2f2ae7700 10 client.14730.objecter successfully canceled tick 2024-02-19T21:03:08.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.615+0000 7fd2f2ae7700 1 -- 172.21.15.163:0/55560802 >> v1:172.21.15.163:6800/1552514681 conn(0x7fd2dc063cb0 legacy=0x7fd2dc066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:08.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.615+0000 7fd2f2ae7700 1 -- 172.21.15.163:0/55560802 >> v1:172.21.15.163:6789/0 conn(0x7fd2ec104290 legacy=0x7fd2ec118410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:08.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.615+0000 7fd2f2ae7700 1 -- 172.21.15.163:0/55560802 shutdown_connections 2024-02-19T21:03:08.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.615+0000 7fd2f2ae7700 1 -- 172.21.15.163:0/55560802 >> 172.21.15.163:0/55560802 conn(0x7fd2ec0f2470 msgr2=0x7fd2ec103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:08.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.615+0000 7fd2f2ae7700 1 -- 172.21.15.163:0/55560802 shutdown_connections 2024-02-19T21:03:08.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.615+0000 7fd2f2ae7700 1 -- 172.21.15.163:0/55560802 wait complete. 2024-02-19T21:03:08.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:08.615+0000 7fd2f2ae7700 1 librados: shutdown 2024-02-19T21:03:08.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:06.931715+0000 mgr.y (mgr.14550) 76 : cluster [DBG] pgmap v57: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:03:08.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:06.988283+0000 osd.5 (osd.5) 29 : cluster [DBG] 5.f scrub starts 2024-02-19T21:03:08.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:06.991195+0000 osd.5 (osd.5) 30 : cluster [DBG] 5.f scrub ok 2024-02-19T21:03:08.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:07.470088+0000 osd.1 (osd.1) 41 : cluster [DBG] 6.1 deep-scrub starts 2024-02-19T21:03:08.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:07.473065+0000 osd.1 (osd.1) 42 : cluster [DBG] 6.1 deep-scrub ok 2024-02-19T21:03:08.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:08.044630+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:03:08.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:08.046636+0000 osd.4 (osd.4) 36 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:03:09.355 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528345 got 193273528350 for osd.6 2024-02-19T21:03:09.355 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-19T21:03:09.625 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:09.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:07.667118+0000 osd.0 (osd.0) 35 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:03:09.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:07.668938+0000 osd.0 (osd.0) 36 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:03:09.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:08.615112+0000 mon.a (mon.0) 706 : audit [DBG] from='client.? v1:172.21.15.163:0/55560802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-19T21:03:09.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:08.741083+0000 osd.3 (osd.3) 35 : cluster [DBG] 4.14 scrub starts 2024-02-19T21:03:09.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:08.742862+0000 osd.3 (osd.3) 36 : cluster [DBG] 4.14 scrub ok 2024-02-19T21:03:09.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:09.009422+0000 osd.4 (osd.4) 37 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:03:09.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:09.010338+0000 osd.4 (osd.4) 38 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:03:09.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:07.667118+0000 osd.0 (osd.0) 35 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:03:09.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:07.668938+0000 osd.0 (osd.0) 36 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:03:09.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:08.615112+0000 mon.a (mon.0) 706 : audit [DBG] from='client.? v1:172.21.15.163:0/55560802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-19T21:03:09.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:08.741083+0000 osd.3 (osd.3) 35 : cluster [DBG] 4.14 scrub starts 2024-02-19T21:03:09.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:08.742862+0000 osd.3 (osd.3) 36 : cluster [DBG] 4.14 scrub ok 2024-02-19T21:03:09.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:09.009422+0000 osd.4 (osd.4) 37 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:03:09.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:09.010338+0000 osd.4 (osd.4) 38 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:03:09.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:07.667118+0000 osd.0 (osd.0) 35 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:03:09.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:07.668938+0000 osd.0 (osd.0) 36 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:03:09.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:08.615112+0000 mon.a (mon.0) 706 : audit [DBG] from='client.? v1:172.21.15.163:0/55560802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-19T21:03:09.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:08.741083+0000 osd.3 (osd.3) 35 : cluster [DBG] 4.14 scrub starts 2024-02-19T21:03:09.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:08.742862+0000 osd.3 (osd.3) 36 : cluster [DBG] 4.14 scrub ok 2024-02-19T21:03:09.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:09.009422+0000 osd.4 (osd.4) 37 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:03:09.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:09.010338+0000 osd.4 (osd.4) 38 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:03:10.312 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:03:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:03:10.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:08.635022+0000 osd.0 (osd.0) 37 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:03:10.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:08.635936+0000 osd.0 (osd.0) 38 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:03:10.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:08.932090+0000 mgr.y (mgr.14550) 77 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:03:10.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:08.981799+0000 osd.5 (osd.5) 31 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:03:10.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:08.984714+0000 osd.5 (osd.5) 32 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:03:10.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:08.635022+0000 osd.0 (osd.0) 37 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:03:10.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:08.635936+0000 osd.0 (osd.0) 38 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:03:10.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:08.932090+0000 mgr.y (mgr.14550) 77 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:03:10.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:08.981799+0000 osd.5 (osd.5) 31 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:03:10.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:08.984714+0000 osd.5 (osd.5) 32 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:03:10.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:08.635022+0000 osd.0 (osd.0) 37 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:03:10.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:08.635936+0000 osd.0 (osd.0) 38 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:03:10.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:08.932090+0000 mgr.y (mgr.14550) 77 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:03:10.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:08.981799+0000 osd.5 (osd.5) 31 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:03:10.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:08.984714+0000 osd.5 (osd.5) 32 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:03:11.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.468+0000 7f161a730700 1 -- 172.21.15.163:0/2999188671 >> v1:172.21.15.163:6790/0 conn(0x7f16140fd980 legacy=0x7f16140fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:11.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.468+0000 7f161a730700 1 -- 172.21.15.163:0/2999188671 shutdown_connections 2024-02-19T21:03:11.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.468+0000 7f161a730700 1 -- 172.21.15.163:0/2999188671 >> 172.21.15.163:0/2999188671 conn(0x7f16140f35a0 msgr2=0x7f16140f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:11.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.468+0000 7f161a730700 1 -- 172.21.15.163:0/2999188671 shutdown_connections 2024-02-19T21:03:11.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.468+0000 7f161a730700 1 -- 172.21.15.163:0/2999188671 wait complete. 2024-02-19T21:03:11.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f161a730700 1 librados: starting msgr at 2024-02-19T21:03:11.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f161a730700 1 librados: starting objecter 2024-02-19T21:03:11.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f161a730700 1 Processor -- start 2024-02-19T21:03:11.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f161a730700 1 -- start start 2024-02-19T21:03:11.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f161a730700 1 librados: setting wanted keys 2024-02-19T21:03:11.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f161a730700 1 librados: calling monclient init 2024-02-19T21:03:11.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f161a730700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f16140fcee0 con 0x7f16140fd980 2024-02-19T21:03:11.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f161a730700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f16140fd070 con 0x7f16141042c0 2024-02-19T21:03:11.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f161a730700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f16140fd260 con 0x7f1614100780 2024-02-19T21:03:11.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f1613fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f16140fd980 0x7f16140fc7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33614/0 (socket says 172.21.15.163:33614) 2024-02-19T21:03:11.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f1613fff700 1 -- 172.21.15.163:0/1590359120 learned_addr learned my addr 172.21.15.163:0/1590359120 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:11.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f16117fa700 10 client.?.objecter ms_handle_connect 0x7f16140fd980 2024-02-19T21:03:11.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f16117fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:11.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f16117fa700 10 client.?.objecter ms_handle_connect 0x7f1614100780 2024-02-19T21:03:11.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f16117fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:11.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f16117fa700 10 client.?.objecter ms_handle_connect 0x7f16141042c0 2024-02-19T21:03:11.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f16117fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:11.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.469+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178317555 0 0) 0x7f16140fcee0 con 0x7f16140fd980 2024-02-19T21:03:11.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15fc0038e0 con 0x7f16140fd980 2024-02-19T21:03:11.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938878953 0 0) 0x7f16140fd260 con 0x7f1614100780 2024-02-19T21:03:11.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16140fcee0 con 0x7f1614100780 2024-02-19T21:03:11.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751504399 0 0) 0x7f16140fd070 con 0x7f16141042c0 2024-02-19T21:03:11.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16140fd260 con 0x7f16141042c0 2024-02-19T21:03:11.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3172619539 0 0) 0x7f15fc0038e0 con 0x7f16140fd980 2024-02-19T21:03:11.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16140fd070 con 0x7f16140fd980 2024-02-19T21:03:11.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1604002e60 con 0x7f16140fd980 2024-02-19T21:03:11.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3836201597 0 0) 0x7f16140fcee0 con 0x7f1614100780 2024-02-19T21:03:11.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15fc0038e0 con 0x7f1614100780 2024-02-19T21:03:11.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1608001620 con 0x7f1614100780 2024-02-19T21:03:11.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2904564494 0 0) 0x7f16140fd260 con 0x7f16141042c0 2024-02-19T21:03:11.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16140fcee0 con 0x7f16141042c0 2024-02-19T21:03:11.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f160c001b00 con 0x7f16141042c0 2024-02-19T21:03:11.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1521913052 0 0) 0x7f16140fd070 con 0x7f16140fd980 2024-02-19T21:03:11.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 >> v1:172.21.15.163:6790/0 conn(0x7f1614100780 legacy=0x7f1614114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:11.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 >> v1:172.21.15.193:6789/0 conn(0x7f16141042c0 legacy=0x7f1614118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:11.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161411cb90 con 0x7f16140fd980 2024-02-19T21:03:11.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f161a730700 1 -- 172.21.15.163:0/1590359120 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1614119bc0 con 0x7f16140fd980 2024-02-19T21:03:11.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f161a730700 10 client.14736.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:11.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f161a730700 1 -- 172.21.15.163:0/1590359120 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f161411a0b0 con 0x7f16140fd980 2024-02-19T21:03:11.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f161a730700 1 librados: init done 2024-02-19T21:03:11.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f16040024f0 con 0x7f16140fd980 2024-02-19T21:03:11.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.470+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f16040038c0 con 0x7f16140fd980 2024-02-19T21:03:11.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.471+0000 7f161a730700 1 -- 172.21.15.163:0/1590359120 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1600004f20 con 0x7f16140fd980 2024-02-19T21:03:11.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.471+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f1604003a20 con 0x7f16140fd980 2024-02-19T21:03:11.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.472+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(83..83 src has 32..83) v4 ==== 6172+0+0 (unknown 4093497947 0 0) 0x7f1604078a00 con 0x7f16140fd980 2024-02-19T21:03:11.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.472+0000 7f16117fa700 10 client.14736.objecter ms_dispatch 0x7f1614001550 osd_map(83..83 src has 32..83) v4 2024-02-19T21:03:11.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.472+0000 7f16117fa700 3 client.14736.objecter handle_osd_map got epochs [83,83] > 0 2024-02-19T21:03:11.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.472+0000 7f16117fa700 3 client.14736.objecter handle_osd_map decoding full epoch 83 2024-02-19T21:03:11.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.472+0000 7f16117fa700 20 client.14736.objecter dump_active .. 0 homeless 2024-02-19T21:03:11.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.472+0000 7f16117fa700 10 client.14736.objecter ms_handle_connect 0x7f15fc063cc0 2024-02-19T21:03:11.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.476+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f160404c5b0 con 0x7f16140fd980 2024-02-19T21:03:11.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:09.996087+0000 mgr.y (mgr.14550) 78 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:11.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:10.014853+0000 osd.5 (osd.5) 33 : cluster [DBG] 6.a scrub starts 2024-02-19T21:03:11.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:10.017759+0000 osd.5 (osd.5) 34 : cluster [DBG] 6.a scrub ok 2024-02-19T21:03:11.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:09.996087+0000 mgr.y (mgr.14550) 78 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:11.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:10.014853+0000 osd.5 (osd.5) 33 : cluster [DBG] 6.a scrub starts 2024-02-19T21:03:11.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:10.017759+0000 osd.5 (osd.5) 34 : cluster [DBG] 6.a scrub ok 2024-02-19T21:03:11.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.641+0000 7f161a730700 1 -- 172.21.15.163:0/1590359120 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f16000051d0 con 0x7f16140fd980 2024-02-19T21:03:11.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.641+0000 7f16117fa700 1 -- 172.21.15.163:0/1590359120 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (unknown 1482059429 0 2207062030) 0x7f160404aa20 con 0x7f16140fd980 2024-02-19T21:03:11.633 INFO:teuthology.orchestra.run.smithi163.stdout:227633266713 2024-02-19T21:03:11.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.643+0000 7f161a730700 10 librados: watch_flush enter 2024-02-19T21:03:11.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.643+0000 7f161a730700 10 librados: watch_flush exit 2024-02-19T21:03:11.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.643+0000 7f161a730700 20 client.14736.objecter shutdown clearing up homeless session... 2024-02-19T21:03:11.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.643+0000 7f161a730700 10 client.14736.objecter successfully canceled tick 2024-02-19T21:03:11.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.643+0000 7f161a730700 1 -- 172.21.15.163:0/1590359120 >> v1:172.21.15.163:6800/1552514681 conn(0x7f15fc063cc0 legacy=0x7f15fc066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:11.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.643+0000 7f161a730700 1 -- 172.21.15.163:0/1590359120 >> v1:172.21.15.163:6789/0 conn(0x7f16140fd980 legacy=0x7f16140fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:11.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.643+0000 7f161a730700 1 -- 172.21.15.163:0/1590359120 shutdown_connections 2024-02-19T21:03:11.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.643+0000 7f161a730700 1 -- 172.21.15.163:0/1590359120 >> 172.21.15.163:0/1590359120 conn(0x7f16140f35a0 msgr2=0x7f1614107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:11.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.643+0000 7f161a730700 1 -- 172.21.15.163:0/1590359120 shutdown_connections 2024-02-19T21:03:11.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.643+0000 7f161a730700 1 -- 172.21.15.163:0/1590359120 wait complete. 2024-02-19T21:03:11.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:11.643+0000 7f161a730700 1 librados: shutdown 2024-02-19T21:03:11.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:09.996087+0000 mgr.y (mgr.14550) 78 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:11.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:10.014853+0000 osd.5 (osd.5) 33 : cluster [DBG] 6.a scrub starts 2024-02-19T21:03:11.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:10.017759+0000 osd.5 (osd.5) 34 : cluster [DBG] 6.a scrub ok 2024-02-19T21:03:12.322 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266708 got 227633266713 for osd.7 2024-02-19T21:03:12.323 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-19T21:03:12.323 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:03:12.598 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:12.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:10.625301+0000 osd.0 (osd.0) 39 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:03:12.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:10.626861+0000 osd.0 (osd.0) 40 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:03:12.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:10.929013+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:03:12.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:10.929142+0000 osd.6 (osd.6) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:03:12.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:10.932680+0000 mgr.y (mgr.14550) 79 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:03:12.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:11.059760+0000 osd.5 (osd.5) 35 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:03:12.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:11.063218+0000 osd.5 (osd.5) 36 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:03:12.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:11.389022+0000 mon.a (mon.0) 707 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-19T21:03:12.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:11.595041+0000 osd.7 (osd.7) 39 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:03:12.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:11.598594+0000 osd.7 (osd.7) 40 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:03:12.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:11.642906+0000 mon.a (mon.0) 708 : audit [DBG] from='client.? v1:172.21.15.163:0/1590359120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-19T21:03:12.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:12.080261+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.1e deep-scrub starts 2024-02-19T21:03:12.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:12.081221+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.1e deep-scrub ok 2024-02-19T21:03:12.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:10.625301+0000 osd.0 (osd.0) 39 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:03:12.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:10.626861+0000 osd.0 (osd.0) 40 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:03:12.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:10.929013+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:03:12.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:10.929142+0000 osd.6 (osd.6) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:03:12.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:10.932680+0000 mgr.y (mgr.14550) 79 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:03:12.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:11.059760+0000 osd.5 (osd.5) 35 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:03:12.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:11.063218+0000 osd.5 (osd.5) 36 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:03:12.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:11.389022+0000 mon.a (mon.0) 707 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-19T21:03:12.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:11.595041+0000 osd.7 (osd.7) 39 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:03:12.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:11.598594+0000 osd.7 (osd.7) 40 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:03:12.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:11.642906+0000 mon.a (mon.0) 708 : audit [DBG] from='client.? v1:172.21.15.163:0/1590359120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-19T21:03:12.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:12.080261+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.1e deep-scrub starts 2024-02-19T21:03:12.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:12.081221+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.1e deep-scrub ok 2024-02-19T21:03:12.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:10.625301+0000 osd.0 (osd.0) 39 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:03:12.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:10.626861+0000 osd.0 (osd.0) 40 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:03:12.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:10.929013+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:03:12.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:10.929142+0000 osd.6 (osd.6) 28 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:03:12.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:10.932680+0000 mgr.y (mgr.14550) 79 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:03:12.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:11.059760+0000 osd.5 (osd.5) 35 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:03:12.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:11.063218+0000 osd.5 (osd.5) 36 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:03:12.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:11.389022+0000 mon.a (mon.0) 707 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-02-19T21:03:12.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:11.595041+0000 osd.7 (osd.7) 39 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:03:12.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:11.598594+0000 osd.7 (osd.7) 40 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:03:12.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:11.642906+0000 mon.a (mon.0) 708 : audit [DBG] from='client.? v1:172.21.15.163:0/1590359120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-19T21:03:12.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:12.080261+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.1e deep-scrub starts 2024-02-19T21:03:12.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:12.081221+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.1e deep-scrub ok 2024-02-19T21:03:13.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:12.733319+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:03:13.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:12.734214+0000 osd.3 (osd.3) 38 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:03:13.739 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:03:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:03:13] "GET /metrics HTTP/1.1" 200 172167 "" "Prometheus/2.33.4" 2024-02-19T21:03:13.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:12.733319+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:03:13.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:12.734214+0000 osd.3 (osd.3) 38 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:03:13.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:12.733319+0000 osd.3 (osd.3) 37 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:03:13.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:12.734214+0000 osd.3 (osd.3) 38 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:03:14.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.304+0000 7f946772e700 1 -- 172.21.15.163:0/300482671 >> v1:172.21.15.163:6789/0 conn(0x7f94600fd910 legacy=0x7f94600fdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:14.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.304+0000 7f946772e700 1 -- 172.21.15.163:0/300482671 shutdown_connections 2024-02-19T21:03:14.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.304+0000 7f946772e700 1 -- 172.21.15.163:0/300482671 >> 172.21.15.163:0/300482671 conn(0x7f94600f3530 msgr2=0x7f94600f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:14.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.304+0000 7f946772e700 1 -- 172.21.15.163:0/300482671 shutdown_connections 2024-02-19T21:03:14.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.304+0000 7f946772e700 1 -- 172.21.15.163:0/300482671 wait complete. 2024-02-19T21:03:14.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.304+0000 7f946772e700 1 librados: starting msgr at 2024-02-19T21:03:14.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.304+0000 7f946772e700 1 librados: starting objecter 2024-02-19T21:03:14.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.304+0000 7f946772e700 1 Processor -- start 2024-02-19T21:03:14.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.304+0000 7f946772e700 1 -- start start 2024-02-19T21:03:14.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.304+0000 7f946772e700 1 librados: setting wanted keys 2024-02-19T21:03:14.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.304+0000 7f946772e700 1 librados: calling monclient init 2024-02-19T21:03:14.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f946772e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f94601172c0 con 0x7f94600fd910 2024-02-19T21:03:14.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f946772e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f94601184a0 con 0x7f9460100710 2024-02-19T21:03:14.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f946772e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f94601196a0 con 0x7f9460104250 2024-02-19T21:03:14.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94654ca700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f94600fd910 0x7f94600f98f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:34506/0 (socket says 172.21.15.163:34506) 2024-02-19T21:03:14.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f9465ccb700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f9460104250 0x7f9460115980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45196/0 (socket says 172.21.15.163:45196) 2024-02-19T21:03:14.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94654ca700 1 -- 172.21.15.163:0/1839329750 learned_addr learned my addr 172.21.15.163:0/1839329750 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:14.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 10 client.?.objecter ms_handle_connect 0x7f9460104250 2024-02-19T21:03:14.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:14.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 10 client.?.objecter ms_handle_connect 0x7f94600fd910 2024-02-19T21:03:14.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:14.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 10 client.?.objecter ms_handle_connect 0x7f9460100710 2024-02-19T21:03:14.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:14.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143236871 0 0) 0x7f94601196a0 con 0x7f9460104250 2024-02-19T21:03:14.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f944c003800 con 0x7f9460104250 2024-02-19T21:03:14.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484385012 0 0) 0x7f94601172c0 con 0x7f94600fd910 2024-02-19T21:03:14.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94601196a0 con 0x7f94600fd910 2024-02-19T21:03:14.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055688283 0 0) 0x7f94601184a0 con 0x7f9460100710 2024-02-19T21:03:14.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94601172c0 con 0x7f9460100710 2024-02-19T21:03:14.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2610969169 0 0) 0x7f944c003800 con 0x7f9460104250 2024-02-19T21:03:14.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94601184a0 con 0x7f9460104250 2024-02-19T21:03:14.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9458001d70 con 0x7f9460104250 2024-02-19T21:03:14.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2642589082 0 0) 0x7f94601196a0 con 0x7f94600fd910 2024-02-19T21:03:14.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f944c003800 con 0x7f94600fd910 2024-02-19T21:03:14.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f945c0015b0 con 0x7f94600fd910 2024-02-19T21:03:14.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1586009260 0 0) 0x7f94601172c0 con 0x7f9460100710 2024-02-19T21:03:14.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94601196a0 con 0x7f9460100710 2024-02-19T21:03:14.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.305+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f94500018c0 con 0x7f9460100710 2024-02-19T21:03:14.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.306+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2032258251 0 0) 0x7f94601184a0 con 0x7f9460104250 2024-02-19T21:03:14.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.306+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 >> v1:172.21.15.193:6789/0 conn(0x7f9460100710 legacy=0x7f94600fa020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:14.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.306+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 >> v1:172.21.15.163:6789/0 conn(0x7f94600fd910 legacy=0x7f94600f98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:14.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.306+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946011a8a0 con 0x7f9460104250 2024-02-19T21:03:14.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.306+0000 7f946772e700 1 -- 172.21.15.163:0/1839329750 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94601174b0 con 0x7f9460104250 2024-02-19T21:03:14.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.306+0000 7f946772e700 10 client.34664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:14.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.306+0000 7f946772e700 1 -- 172.21.15.163:0/1839329750 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9460117920 con 0x7f9460104250 2024-02-19T21:03:14.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.306+0000 7f946772e700 1 librados: init done 2024-02-19T21:03:14.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.306+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9458002730 con 0x7f9460104250 2024-02-19T21:03:14.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.306+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9458005d50 con 0x7f9460104250 2024-02-19T21:03:14.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.307+0000 7f946772e700 1 -- 172.21.15.163:0/1839329750 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9444004f20 con 0x7f9460104250 2024-02-19T21:03:14.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.307+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9458002290 con 0x7f9460104250 2024-02-19T21:03:14.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.307+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(83..83 src has 32..83) v4 ==== 6172+0+0 (unknown 4093497947 0 0) 0x7f9458078760 con 0x7f9460104250 2024-02-19T21:03:14.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.307+0000 7f94567fc700 10 client.34664.objecter ms_dispatch 0x7f9460001550 osd_map(83..83 src has 32..83) v4 2024-02-19T21:03:14.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.307+0000 7f94567fc700 3 client.34664.objecter handle_osd_map got epochs [83,83] > 0 2024-02-19T21:03:14.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.307+0000 7f94567fc700 3 client.34664.objecter handle_osd_map decoding full epoch 83 2024-02-19T21:03:14.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.308+0000 7f94567fc700 20 client.34664.objecter dump_active .. 0 homeless 2024-02-19T21:03:14.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.308+0000 7f94567fc700 10 client.34664.objecter ms_handle_connect 0x7f944c063cb0 2024-02-19T21:03:14.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.312+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f945804c310 con 0x7f9460104250 2024-02-19T21:03:14.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.474+0000 7f946772e700 1 -- 172.21.15.163:0/1839329750 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9444000b90 con 0x7f944c063cb0 2024-02-19T21:03:14.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.484+0000 7f94567fc700 1 -- 172.21.15.163:0/1839329750 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357340 (unknown 2965378022 0 2219292828) 0x7f9444000b90 con 0x7f944c063cb0 2024-02-19T21:03:14.480 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:03:14.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.487+0000 7f946772e700 10 librados: watch_flush enter 2024-02-19T21:03:14.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.487+0000 7f946772e700 10 librados: watch_flush exit 2024-02-19T21:03:14.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.487+0000 7f946772e700 20 client.34664.objecter shutdown clearing up homeless session... 2024-02-19T21:03:14.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.487+0000 7f946772e700 10 client.34664.objecter successfully canceled tick 2024-02-19T21:03:14.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.487+0000 7f946772e700 1 -- 172.21.15.163:0/1839329750 >> v1:172.21.15.163:6800/1552514681 conn(0x7f944c063cb0 legacy=0x7f944c066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:14.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.488+0000 7f946772e700 1 -- 172.21.15.163:0/1839329750 >> v1:172.21.15.163:6790/0 conn(0x7f9460104250 legacy=0x7f9460115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:14.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.488+0000 7f946772e700 1 -- 172.21.15.163:0/1839329750 shutdown_connections 2024-02-19T21:03:14.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.488+0000 7f946772e700 1 -- 172.21.15.163:0/1839329750 >> 172.21.15.163:0/1839329750 conn(0x7f94600f3530 msgr2=0x7f94601076c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:14.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.488+0000 7f946772e700 1 -- 172.21.15.163:0/1839329750 shutdown_connections 2024-02-19T21:03:14.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.488+0000 7f946772e700 1 -- 172.21.15.163:0/1839329750 wait complete. 2024-02-19T21:03:14.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:14.488+0000 7f946772e700 1 librados: shutdown 2024-02-19T21:03:14.484 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:03:14.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:12.933008+0000 mgr.y (mgr.14550) 80 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:14.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:13.093766+0000 osd.5 (osd.5) 37 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:03:14.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:13.096663+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:03:14.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:12.933008+0000 mgr.y (mgr.14550) 80 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:14.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:13.093766+0000 osd.5 (osd.5) 37 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:03:14.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:13.096663+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:03:14.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:12.933008+0000 mgr.y (mgr.14550) 80 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:14.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:13.093766+0000 osd.5 (osd.5) 37 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:03:14.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:13.096663+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:03:15.273 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-02-19T21:03:12.932853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":826,"num_read_kb":578,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":474,"ondisk_log_size":474,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2383380,"kb_used_data":6612,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747577836,"statfs":{"total":767960285184,"available":765519704064,"internally_reserved":0,"allocated":6770688,"data_stored":2789053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002359"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.327168+0000","last_change":"2024-02-19T21:01:22.609338+0000","last_active":"2024-02-19T21:02:52.327168+0000","last_peered":"2024-02-19T21:02:52.327168+0000","last_clean":"2024-02-19T21:02:52.327168+0000","last_became_active":"2024-02-19T21:01:22.609059+0000","last_became_peered":"2024-02-19T21:01:22.609059+0000","last_unstale":"2024-02-19T21:02:52.327168+0000","last_undegraded":"2024-02-19T21:02:52.327168+0000","last_fullsized":"2024-02-19T21:02:52.327168+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:01:21.273400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:05.073746+0000","last_change":"2024-02-19T21:03:05.073746+0000","last_active":"2024-02-19T21:03:05.073746+0000","last_peered":"2024-02-19T21:03:05.073746+0000","last_clean":"2024-02-19T21:03:05.073746+0000","last_became_active":"2024-02-19T21:01:20.289208+0000","last_became_peered":"2024-02-19T21:01:20.289208+0000","last_unstale":"2024-02-19T21:03:05.073746+0000","last_undegraded":"2024-02-19T21:03:05.073746+0000","last_fullsized":"2024-02-19T21:03:05.073746+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.073664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.073664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:04.644754+0000","last_change":"2024-02-19T21:03:04.644754+0000","last_active":"2024-02-19T21:03:04.644754+0000","last_peered":"2024-02-19T21:03:04.644754+0000","last_clean":"2024-02-19T21:03:04.644754+0000","last_became_active":"2024-02-19T21:01:16.278753+0000","last_became_peered":"2024-02-19T21:01:16.278753+0000","last_unstale":"2024-02-19T21:03:04.644754+0000","last_undegraded":"2024-02-19T21:03:04.644754+0000","last_fullsized":"2024-02-19T21:03:04.644754+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.644689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.644689+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":61,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.327106+0000","last_change":"2024-02-19T21:02:48.863931+0000","last_active":"2024-02-19T21:02:52.327106+0000","last_peered":"2024-02-19T21:02:52.327106+0000","last_clean":"2024-02-19T21:02:52.327106+0000","last_became_active":"2024-02-19T21:01:18.283222+0000","last_became_peered":"2024-02-19T21:01:18.283222+0000","last_unstale":"2024-02-19T21:02:52.327106+0000","last_undegraded":"2024-02-19T21:02:52.327106+0000","last_fullsized":"2024-02-19T21:02:52.327106+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:48.863852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.863852+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.325850+0000","last_change":"2024-02-19T21:02:23.100006+0000","last_active":"2024-02-19T21:02:52.325850+0000","last_peered":"2024-02-19T21:02:52.325850+0000","last_clean":"2024-02-19T21:02:52.325850+0000","last_became_active":"2024-02-19T21:01:22.293432+0000","last_became_peered":"2024-02-19T21:01:22.293432+0000","last_unstale":"2024-02-19T21:02:52.325850+0000","last_undegraded":"2024-02-19T21:02:52.325850+0000","last_fullsized":"2024-02-19T21:02:52.325850+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:23.099932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:23.099932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.456017+0000","last_change":"2024-02-19T21:02:29.480341+0000","last_active":"2024-02-19T21:02:52.456017+0000","last_peered":"2024-02-19T21:02:52.456017+0000","last_clean":"2024-02-19T21:02:52.456017+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:02:52.456017+0000","last_undegraded":"2024-02-19T21:02:52.456017+0000","last_fullsized":"2024-02-19T21:02:52.456017+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:29.480275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:29.480275+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326446+0000","last_change":"2024-02-19T21:02:20.970293+0000","last_active":"2024-02-19T21:02:52.326446+0000","last_peered":"2024-02-19T21:02:52.326446+0000","last_clean":"2024-02-19T21:02:52.326446+0000","last_became_active":"2024-02-19T21:01:16.272132+0000","last_became_peered":"2024-02-19T21:01:16.272132+0000","last_unstale":"2024-02-19T21:02:52.326446+0000","last_undegraded":"2024-02-19T21:02:52.326446+0000","last_fullsized":"2024-02-19T21:02:52.326446+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:20.970216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:20.970216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":47,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.325802+0000","last_change":"2024-02-19T21:01:18.287914+0000","last_active":"2024-02-19T21:02:52.325802+0000","last_peered":"2024-02-19T21:02:52.325802+0000","last_clean":"2024-02-19T21:02:52.325802+0000","last_became_active":"2024-02-19T21:01:18.287645+0000","last_became_peered":"2024-02-19T21:01:18.287645+0000","last_unstale":"2024-02-19T21:02:52.325802+0000","last_undegraded":"2024-02-19T21:02:52.325802+0000","last_fullsized":"2024-02-19T21:02:52.325802+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:01:17.258309+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455286+0000","last_change":"2024-02-19T21:01:22.293588+0000","last_active":"2024-02-19T21:02:52.455286+0000","last_peered":"2024-02-19T21:02:52.455286+0000","last_clean":"2024-02-19T21:02:52.455286+0000","last_became_active":"2024-02-19T21:01:22.293282+0000","last_became_peered":"2024-02-19T21:01:22.293282+0000","last_unstale":"2024-02-19T21:02:52.455286+0000","last_undegraded":"2024-02-19T21:02:52.455286+0000","last_fullsized":"2024-02-19T21:02:52.455286+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:01:21.273400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.456173+0000","last_change":"2024-02-19T21:02:35.546613+0000","last_active":"2024-02-19T21:02:52.456173+0000","last_peered":"2024-02-19T21:02:52.456173+0000","last_clean":"2024-02-19T21:02:52.456173+0000","last_became_active":"2024-02-19T21:01:20.287940+0000","last_became_peered":"2024-02-19T21:01:20.287940+0000","last_unstale":"2024-02-19T21:02:52.456173+0000","last_undegraded":"2024-02-19T21:02:52.456173+0000","last_fullsized":"2024-02-19T21:02:52.456173+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:35.546526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:35.546526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.475727+0000","last_change":"2024-02-19T21:02:20.575203+0000","last_active":"2024-02-19T21:02:52.475727+0000","last_peered":"2024-02-19T21:02:52.475727+0000","last_clean":"2024-02-19T21:02:52.475727+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:02:52.475727+0000","last_undegraded":"2024-02-19T21:02:52.475727+0000","last_fullsized":"2024-02-19T21:02:52.475727+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:20.575133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:20.575133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":46,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326150+0000","last_change":"2024-02-19T21:02:27.098930+0000","last_active":"2024-02-19T21:02:52.326150+0000","last_peered":"2024-02-19T21:02:52.326150+0000","last_clean":"2024-02-19T21:02:52.326150+0000","last_became_active":"2024-02-19T21:01:18.285735+0000","last_became_peered":"2024-02-19T21:01:18.285735+0000","last_unstale":"2024-02-19T21:02:52.326150+0000","last_undegraded":"2024-02-19T21:02:52.326150+0000","last_fullsized":"2024-02-19T21:02:52.326150+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:02:27.098844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:27.098844+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:00.570766+0000","last_change":"2024-02-19T21:03:00.570766+0000","last_active":"2024-02-19T21:03:00.570766+0000","last_peered":"2024-02-19T21:03:00.570766+0000","last_clean":"2024-02-19T21:03:00.570766+0000","last_became_active":"2024-02-19T21:01:22.292714+0000","last_became_peered":"2024-02-19T21:01:22.292714+0000","last_unstale":"2024-02-19T21:03:00.570766+0000","last_undegraded":"2024-02-19T21:03:00.570766+0000","last_fullsized":"2024-02-19T21:03:00.570766+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:00.570677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:00.570677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455754+0000","last_change":"2024-02-19T21:01:20.293436+0000","last_active":"2024-02-19T21:02:52.455754+0000","last_peered":"2024-02-19T21:02:52.455754+0000","last_clean":"2024-02-19T21:02:52.455754+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:02:52.455754+0000","last_undegraded":"2024-02-19T21:02:52.455754+0000","last_fullsized":"2024-02-19T21:02:52.455754+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:01:19.270373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":37,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:57.552979+0000","last_change":"2024-02-19T21:02:57.552979+0000","last_active":"2024-02-19T21:02:57.552979+0000","last_peered":"2024-02-19T21:02:57.552979+0000","last_clean":"2024-02-19T21:02:57.552979+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:02:57.552979+0000","last_undegraded":"2024-02-19T21:02:57.552979+0000","last_fullsized":"2024-02-19T21:02:57.552979+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.552886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.552886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":52,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.325895+0000","last_change":"2024-02-19T21:02:26.073713+0000","last_active":"2024-02-19T21:02:52.325895+0000","last_peered":"2024-02-19T21:02:52.325895+0000","last_clean":"2024-02-19T21:02:52.325895+0000","last_became_active":"2024-02-19T21:01:18.285269+0000","last_became_peered":"2024-02-19T21:01:18.285269+0000","last_unstale":"2024-02-19T21:02:52.325895+0000","last_undegraded":"2024-02-19T21:02:52.325895+0000","last_fullsized":"2024-02-19T21:02:52.325895+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:26.073625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:26.073625+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.325880+0000","last_change":"2024-02-19T21:02:24.948910+0000","last_active":"2024-02-19T21:02:52.325880+0000","last_peered":"2024-02-19T21:02:52.325880+0000","last_clean":"2024-02-19T21:02:52.325880+0000","last_became_active":"2024-02-19T21:01:22.607099+0000","last_became_peered":"2024-02-19T21:01:22.607099+0000","last_unstale":"2024-02-19T21:02:52.325880+0000","last_undegraded":"2024-02-19T21:02:52.325880+0000","last_fullsized":"2024-02-19T21:02:52.325880+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:24.948832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:24.948832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:09.010420+0000","last_change":"2024-02-19T21:03:09.010420+0000","last_active":"2024-02-19T21:03:09.010420+0000","last_peered":"2024-02-19T21:03:09.010420+0000","last_clean":"2024-02-19T21:03:09.010420+0000","last_became_active":"2024-02-19T21:01:20.291672+0000","last_became_peered":"2024-02-19T21:01:20.291672+0000","last_unstale":"2024-02-19T21:03:09.010420+0000","last_undegraded":"2024-02-19T21:03:09.010420+0000","last_fullsized":"2024-02-19T21:03:09.010420+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:09.010341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:09.010341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.327278+0000","last_change":"2024-02-19T21:01:16.275072+0000","last_active":"2024-02-19T21:02:52.327278+0000","last_peered":"2024-02-19T21:02:52.327278+0000","last_clean":"2024-02-19T21:02:52.327278+0000","last_became_active":"2024-02-19T21:01:16.274811+0000","last_became_peered":"2024-02-19T21:01:16.274811+0000","last_unstale":"2024-02-19T21:02:52.327278+0000","last_undegraded":"2024-02-19T21:02:52.327278+0000","last_fullsized":"2024-02-19T21:02:52.327278+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:01:15.241725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":59,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:57.819231+0000","last_change":"2024-02-19T21:02:57.819231+0000","last_active":"2024-02-19T21:02:57.819231+0000","last_peered":"2024-02-19T21:02:57.819231+0000","last_clean":"2024-02-19T21:02:57.819231+0000","last_became_active":"2024-02-19T21:01:18.285926+0000","last_became_peered":"2024-02-19T21:01:18.285926+0000","last_unstale":"2024-02-19T21:02:57.819231+0000","last_undegraded":"2024-02-19T21:02:57.819231+0000","last_fullsized":"2024-02-19T21:02:57.819231+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:02:57.819125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.819125+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326558+0000","last_change":"2024-02-19T21:01:22.607724+0000","last_active":"2024-02-19T21:02:52.326558+0000","last_peered":"2024-02-19T21:02:52.326558+0000","last_clean":"2024-02-19T21:02:52.326558+0000","last_became_active":"2024-02-19T21:01:22.607435+0000","last_became_peered":"2024-02-19T21:01:22.607435+0000","last_unstale":"2024-02-19T21:02:52.326558+0000","last_undegraded":"2024-02-19T21:02:52.326558+0000","last_fullsized":"2024-02-19T21:02:52.326558+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:01:21.273400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:54.544941+0000","last_change":"2024-02-19T21:02:54.544941+0000","last_active":"2024-02-19T21:02:54.544941+0000","last_peered":"2024-02-19T21:02:54.544941+0000","last_clean":"2024-02-19T21:02:54.544941+0000","last_became_active":"2024-02-19T21:01:20.290705+0000","last_became_peered":"2024-02-19T21:01:20.290705+0000","last_unstale":"2024-02-19T21:02:54.544941+0000","last_undegraded":"2024-02-19T21:02:54.544941+0000","last_fullsized":"2024-02-19T21:02:54.544941+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:54.544834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:54.544834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.456236+0000","last_change":"2024-02-19T21:02:47.805245+0000","last_active":"2024-02-19T21:02:52.456236+0000","last_peered":"2024-02-19T21:02:52.456236+0000","last_clean":"2024-02-19T21:02:52.456236+0000","last_became_active":"2024-02-19T21:01:16.274524+0000","last_became_peered":"2024-02-19T21:01:16.274524+0000","last_unstale":"2024-02-19T21:02:52.456236+0000","last_undegraded":"2024-02-19T21:02:52.456236+0000","last_fullsized":"2024-02-19T21:02:52.456236+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.805171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.805171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":61,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455369+0000","last_change":"2024-02-19T21:02:47.238547+0000","last_active":"2024-02-19T21:02:52.455369+0000","last_peered":"2024-02-19T21:02:52.455369+0000","last_clean":"2024-02-19T21:02:52.455369+0000","last_became_active":"2024-02-19T21:01:18.289712+0000","last_became_peered":"2024-02-19T21:01:18.289712+0000","last_unstale":"2024-02-19T21:02:52.455369+0000","last_undegraded":"2024-02-19T21:02:52.455369+0000","last_fullsized":"2024-02-19T21:02:52.455369+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:47.238470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.238470+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:06.456310+0000","last_change":"2024-02-19T21:03:06.456310+0000","last_active":"2024-02-19T21:03:06.456310+0000","last_peered":"2024-02-19T21:03:06.456310+0000","last_clean":"2024-02-19T21:03:06.456310+0000","last_became_active":"2024-02-19T21:01:22.293230+0000","last_became_peered":"2024-02-19T21:01:22.293230+0000","last_unstale":"2024-02-19T21:03:06.456310+0000","last_undegraded":"2024-02-19T21:03:06.456310+0000","last_fullsized":"2024-02-19T21:03:06.456310+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.456212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.456212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:08.636037+0000","last_change":"2024-02-19T21:03:08.636037+0000","last_active":"2024-02-19T21:03:08.636037+0000","last_peered":"2024-02-19T21:03:08.636037+0000","last_clean":"2024-02-19T21:03:08.636037+0000","last_became_active":"2024-02-19T21:01:20.288016+0000","last_became_peered":"2024-02-19T21:01:20.288016+0000","last_unstale":"2024-02-19T21:03:08.636037+0000","last_undegraded":"2024-02-19T21:03:08.636037+0000","last_fullsized":"2024-02-19T21:03:08.636037+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:08.635943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.635943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326948+0000","last_change":"2024-02-19T21:01:16.278289+0000","last_active":"2024-02-19T21:02:52.326948+0000","last_peered":"2024-02-19T21:02:52.326948+0000","last_clean":"2024-02-19T21:02:52.326948+0000","last_became_active":"2024-02-19T21:01:16.277826+0000","last_became_peered":"2024-02-19T21:01:16.277826+0000","last_unstale":"2024-02-19T21:02:52.326948+0000","last_undegraded":"2024-02-19T21:02:52.326948+0000","last_fullsized":"2024-02-19T21:02:52.326948+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:01:15.241725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.456313+0000","last_change":"2024-02-19T21:02:29.684765+0000","last_active":"2024-02-19T21:02:52.456313+0000","last_peered":"2024-02-19T21:02:52.456313+0000","last_clean":"2024-02-19T21:02:52.456313+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:02:52.456313+0000","last_undegraded":"2024-02-19T21:02:52.456313+0000","last_fullsized":"2024-02-19T21:02:52.456313+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:29.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:29.684698+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.475780+0000","last_change":"2024-02-19T21:02:36.585127+0000","last_active":"2024-02-19T21:02:52.475780+0000","last_peered":"2024-02-19T21:02:52.475780+0000","last_clean":"2024-02-19T21:02:52.475780+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:02:52.475780+0000","last_undegraded":"2024-02-19T21:02:52.475780+0000","last_fullsized":"2024-02-19T21:02:52.475780+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:02:36.585007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:36.585007+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":68,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:04.441943+0000","last_change":"2024-02-19T21:02:37.842634+0000","last_active":"2024-02-19T21:03:04.441943+0000","last_peered":"2024-02-19T21:03:04.441943+0000","last_clean":"2024-02-19T21:03:04.441943+0000","last_became_active":"2024-02-19T21:01:20.290567+0000","last_became_peered":"2024-02-19T21:01:20.290567+0000","last_unstale":"2024-02-19T21:03:04.441943+0000","last_undegraded":"2024-02-19T21:03:04.441943+0000","last_fullsized":"2024-02-19T21:03:04.441943+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:37.842545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:37.842545+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:06.000805+0000","last_change":"2024-02-19T21:03:06.000805+0000","last_active":"2024-02-19T21:03:06.000805+0000","last_peered":"2024-02-19T21:03:06.000805+0000","last_clean":"2024-02-19T21:03:06.000805+0000","last_became_active":"2024-02-19T21:01:16.272284+0000","last_became_peered":"2024-02-19T21:01:16.272284+0000","last_unstale":"2024-02-19T21:03:06.000805+0000","last_undegraded":"2024-02-19T21:03:06.000805+0000","last_fullsized":"2024-02-19T21:03:06.000805+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.000732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.000732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":51,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326554+0000","last_change":"2024-02-19T21:02:32.550264+0000","last_active":"2024-02-19T21:02:52.326554+0000","last_peered":"2024-02-19T21:02:52.326554+0000","last_clean":"2024-02-19T21:02:52.326554+0000","last_became_active":"2024-02-19T21:01:18.293865+0000","last_became_peered":"2024-02-19T21:01:18.293865+0000","last_unstale":"2024-02-19T21:02:52.326554+0000","last_undegraded":"2024-02-19T21:02:52.326554+0000","last_fullsized":"2024-02-19T21:02:52.326554+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:02:32.550190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.550190+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.327395+0000","last_change":"2024-02-19T21:02:41.964875+0000","last_active":"2024-02-19T21:02:52.327395+0000","last_peered":"2024-02-19T21:02:52.327395+0000","last_clean":"2024-02-19T21:02:52.327395+0000","last_became_active":"2024-02-19T21:01:22.608997+0000","last_became_peered":"2024-02-19T21:01:22.608997+0000","last_unstale":"2024-02-19T21:02:52.327395+0000","last_undegraded":"2024-02-19T21:02:52.327395+0000","last_fullsized":"2024-02-19T21:02:52.327395+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:41.964809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.964809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:04.486168+0000","last_change":"2024-02-19T21:03:04.486168+0000","last_active":"2024-02-19T21:03:04.486168+0000","last_peered":"2024-02-19T21:03:04.486168+0000","last_clean":"2024-02-19T21:03:04.486168+0000","last_became_active":"2024-02-19T21:01:20.287961+0000","last_became_peered":"2024-02-19T21:01:20.287961+0000","last_unstale":"2024-02-19T21:03:04.486168+0000","last_undegraded":"2024-02-19T21:03:04.486168+0000","last_fullsized":"2024-02-19T21:03:04.486168+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.486067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.486067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.456777+0000","last_change":"2024-02-19T21:02:43.790280+0000","last_active":"2024-02-19T21:02:52.456777+0000","last_peered":"2024-02-19T21:02:52.456777+0000","last_clean":"2024-02-19T21:02:52.456777+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:02:52.456777+0000","last_undegraded":"2024-02-19T21:02:52.456777+0000","last_fullsized":"2024-02-19T21:02:52.456777+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:43.790206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:43.790206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.859565+0000","last_change":"2024-02-19T21:02:52.859531+0000","last_active":"2024-02-19T21:02:52.859565+0000","last_peered":"2024-02-19T21:02:52.859565+0000","last_clean":"2024-02-19T21:02:52.859565+0000","last_became_active":"2024-02-19T21:01:18.293793+0000","last_became_peered":"2024-02-19T21:01:18.293793+0000","last_unstale":"2024-02-19T21:02:52.859565+0000","last_undegraded":"2024-02-19T21:02:52.859565+0000","last_fullsized":"2024-02-19T21:02:52.859565+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:52.859453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.859453+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.456003+0000","last_change":"2024-02-19T21:02:45.579070+0000","last_active":"2024-02-19T21:02:52.456003+0000","last_peered":"2024-02-19T21:02:52.456003+0000","last_clean":"2024-02-19T21:02:52.456003+0000","last_became_active":"2024-02-19T21:01:22.297693+0000","last_became_peered":"2024-02-19T21:01:22.297693+0000","last_unstale":"2024-02-19T21:02:52.456003+0000","last_undegraded":"2024-02-19T21:02:52.456003+0000","last_fullsized":"2024-02-19T21:02:52.456003+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:45.578978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:45.578978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455568+0000","last_change":"2024-02-19T21:02:30.724973+0000","last_active":"2024-02-19T21:02:52.455568+0000","last_peered":"2024-02-19T21:02:52.455568+0000","last_clean":"2024-02-19T21:02:52.455568+0000","last_became_active":"2024-02-19T21:01:20.290962+0000","last_became_peered":"2024-02-19T21:01:20.290962+0000","last_unstale":"2024-02-19T21:02:52.455568+0000","last_undegraded":"2024-02-19T21:02:52.455568+0000","last_fullsized":"2024-02-19T21:02:52.455568+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:30.724908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:30.724908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.476173+0000","last_change":"2024-02-19T21:02:52.021686+0000","last_active":"2024-02-19T21:02:52.476173+0000","last_peered":"2024-02-19T21:02:52.476173+0000","last_clean":"2024-02-19T21:02:52.476173+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:02:52.476173+0000","last_undegraded":"2024-02-19T21:02:52.476173+0000","last_fullsized":"2024-02-19T21:02:52.476173+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.021603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.021603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.325708+0000","last_change":"2024-02-19T21:02:22.939275+0000","last_active":"2024-02-19T21:02:52.325708+0000","last_peered":"2024-02-19T21:02:52.325708+0000","last_clean":"2024-02-19T21:02:52.325708+0000","last_became_active":"2024-02-19T21:01:18.287874+0000","last_became_peered":"2024-02-19T21:01:18.287874+0000","last_unstale":"2024-02-19T21:02:52.325708+0000","last_undegraded":"2024-02-19T21:02:52.325708+0000","last_fullsized":"2024-02-19T21:02:52.325708+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:02:22.939192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:22.939192+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:05.734069+0000","last_change":"2024-02-19T21:03:05.734069+0000","last_active":"2024-02-19T21:03:05.734069+0000","last_peered":"2024-02-19T21:03:05.734069+0000","last_clean":"2024-02-19T21:03:05.734069+0000","last_became_active":"2024-02-19T21:01:22.292704+0000","last_became_peered":"2024-02-19T21:01:22.292704+0000","last_unstale":"2024-02-19T21:03:05.734069+0000","last_undegraded":"2024-02-19T21:03:05.734069+0000","last_fullsized":"2024-02-19T21:03:05.734069+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.733980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.733980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455794+0000","last_change":"2024-02-19T21:02:40.600482+0000","last_active":"2024-02-19T21:02:52.455794+0000","last_peered":"2024-02-19T21:02:52.455794+0000","last_clean":"2024-02-19T21:02:52.455794+0000","last_became_active":"2024-02-19T21:01:20.289020+0000","last_became_peered":"2024-02-19T21:01:20.289020+0000","last_unstale":"2024-02-19T21:02:52.455794+0000","last_undegraded":"2024-02-19T21:02:52.455794+0000","last_fullsized":"2024-02-19T21:02:52.455794+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:40.600410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.600410+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455941+0000","last_change":"2024-02-19T21:02:18.594112+0000","last_active":"2024-02-19T21:02:52.455941+0000","last_peered":"2024-02-19T21:02:52.455941+0000","last_clean":"2024-02-19T21:02:52.455941+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:02:52.455941+0000","last_undegraded":"2024-02-19T21:02:52.455941+0000","last_fullsized":"2024-02-19T21:02:52.455941+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:18.594043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:18.594043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326035+0000","last_change":"2024-02-19T21:02:32.040809+0000","last_active":"2024-02-19T21:02:52.326035+0000","last_peered":"2024-02-19T21:02:52.326035+0000","last_clean":"2024-02-19T21:02:52.326035+0000","last_became_active":"2024-02-19T21:01:18.286128+0000","last_became_peered":"2024-02-19T21:01:18.286128+0000","last_unstale":"2024-02-19T21:02:52.326035+0000","last_undegraded":"2024-02-19T21:02:52.326035+0000","last_fullsized":"2024-02-19T21:02:52.326035+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:32.040731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.040731+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.325198+0000","last_change":"2024-02-19T21:02:51.631722+0000","last_active":"2024-02-19T21:02:52.325198+0000","last_peered":"2024-02-19T21:02:52.325198+0000","last_clean":"2024-02-19T21:02:52.325198+0000","last_became_active":"2024-02-19T21:01:22.293093+0000","last_became_peered":"2024-02-19T21:01:22.293093+0000","last_unstale":"2024-02-19T21:02:52.325198+0000","last_undegraded":"2024-02-19T21:02:52.325198+0000","last_fullsized":"2024-02-19T21:02:52.325198+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:51.631657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:51.631657+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:03.776497+0000","last_change":"2024-02-19T21:03:03.776497+0000","last_active":"2024-02-19T21:03:03.776497+0000","last_peered":"2024-02-19T21:03:03.776497+0000","last_clean":"2024-02-19T21:03:03.776497+0000","last_became_active":"2024-02-19T21:01:20.289691+0000","last_became_peered":"2024-02-19T21:01:20.289691+0000","last_unstale":"2024-02-19T21:03:03.776497+0000","last_undegraded":"2024-02-19T21:03:03.776497+0000","last_fullsized":"2024-02-19T21:03:03.776497+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_clean_scrub_stamp":"2024-02-19T21:03:03.776397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.456161+0000","last_change":"2024-02-19T21:02:21.657425+0000","last_active":"2024-02-19T21:02:52.456161+0000","last_peered":"2024-02-19T21:02:52.456161+0000","last_clean":"2024-02-19T21:02:52.456161+0000","last_became_active":"2024-02-19T21:01:16.271316+0000","last_became_peered":"2024-02-19T21:01:16.271316+0000","last_unstale":"2024-02-19T21:02:52.456161+0000","last_undegraded":"2024-02-19T21:02:52.456161+0000","last_fullsized":"2024-02-19T21:02:52.456161+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.657334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.657334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455272+0000","last_change":"2024-02-19T21:01:31.611926+0000","last_active":"2024-02-19T21:02:52.455272+0000","last_peered":"2024-02-19T21:02:52.455272+0000","last_clean":"2024-02-19T21:02:52.455272+0000","last_became_active":"2024-02-19T21:01:30.922577+0000","last_became_peered":"2024-02-19T21:01:30.922577+0000","last_unstale":"2024-02-19T21:02:52.455272+0000","last_undegraded":"2024-02-19T21:02:52.455272+0000","last_fullsized":"2024-02-19T21:02:52.455272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:01:17.258309+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326668+0000","last_change":"2024-02-19T21:02:39.092233+0000","last_active":"2024-02-19T21:02:52.326668+0000","last_peered":"2024-02-19T21:02:52.326668+0000","last_clean":"2024-02-19T21:02:52.326668+0000","last_became_active":"2024-02-19T21:01:22.299969+0000","last_became_peered":"2024-02-19T21:01:22.299969+0000","last_unstale":"2024-02-19T21:02:52.326668+0000","last_undegraded":"2024-02-19T21:02:52.326668+0000","last_fullsized":"2024-02-19T21:02:52.326668+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.092154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":60,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:04.441764+0000","last_change":"2024-02-19T21:01:20.284282+0000","last_active":"2024-02-19T21:03:04.441764+0000","last_peered":"2024-02-19T21:03:04.441764+0000","last_clean":"2024-02-19T21:03:04.441764+0000","last_became_active":"2024-02-19T21:01:20.283943+0000","last_became_peered":"2024-02-19T21:01:20.283943+0000","last_unstale":"2024-02-19T21:03:04.441764+0000","last_undegraded":"2024-02-19T21:03:04.441764+0000","last_fullsized":"2024-02-19T21:03:04.441764+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:01:19.270373+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.325772+0000","last_change":"2024-02-19T21:02:21.693740+0000","last_active":"2024-02-19T21:02:52.325772+0000","last_peered":"2024-02-19T21:02:52.325772+0000","last_clean":"2024-02-19T21:02:52.325772+0000","last_became_active":"2024-02-19T21:01:16.273181+0000","last_became_peered":"2024-02-19T21:01:16.273181+0000","last_unstale":"2024-02-19T21:02:52.325772+0000","last_undegraded":"2024-02-19T21:02:52.325772+0000","last_fullsized":"2024-02-19T21:02:52.325772+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.693675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.693675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":52,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:57.021496+0000","last_change":"2024-02-19T21:02:57.021496+0000","last_active":"2024-02-19T21:02:57.021496+0000","last_peered":"2024-02-19T21:02:57.021496+0000","last_clean":"2024-02-19T21:02:57.021496+0000","last_became_active":"2024-02-19T21:01:18.293786+0000","last_became_peered":"2024-02-19T21:01:18.293786+0000","last_unstale":"2024-02-19T21:02:57.021496+0000","last_undegraded":"2024-02-19T21:02:57.021496+0000","last_fullsized":"2024-02-19T21:02:57.021496+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:57.021403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.021403+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326222+0000","last_change":"2024-02-19T21:02:39.442600+0000","last_active":"2024-02-19T21:02:52.326222+0000","last_peered":"2024-02-19T21:02:52.326222+0000","last_clean":"2024-02-19T21:02:52.326222+0000","last_became_active":"2024-02-19T21:01:22.297188+0000","last_became_peered":"2024-02-19T21:01:22.297188+0000","last_unstale":"2024-02-19T21:02:52.326222+0000","last_undegraded":"2024-02-19T21:02:52.326222+0000","last_fullsized":"2024-02-19T21:02:52.326222+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:02:39.442533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.442533+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":60,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:04.441944+0000","last_change":"2024-02-19T21:01:20.289075+0000","last_active":"2024-02-19T21:03:04.441944+0000","last_peered":"2024-02-19T21:03:04.441944+0000","last_clean":"2024-02-19T21:03:04.441944+0000","last_became_active":"2024-02-19T21:01:20.288799+0000","last_became_peered":"2024-02-19T21:01:20.288799+0000","last_unstale":"2024-02-19T21:03:04.441944+0000","last_undegraded":"2024-02-19T21:03:04.441944+0000","last_fullsized":"2024-02-19T21:03:04.441944+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:01:19.270373+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455507+0000","last_change":"2024-02-19T21:02:49.056495+0000","last_active":"2024-02-19T21:02:52.455507+0000","last_peered":"2024-02-19T21:02:52.455507+0000","last_clean":"2024-02-19T21:02:52.455507+0000","last_became_active":"2024-02-19T21:01:16.271608+0000","last_became_peered":"2024-02-19T21:01:16.271608+0000","last_unstale":"2024-02-19T21:02:52.455507+0000","last_undegraded":"2024-02-19T21:02:52.455507+0000","last_fullsized":"2024-02-19T21:02:52.455507+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.056412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.056412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":51,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:08.742946+0000","last_change":"2024-02-19T21:03:08.742946+0000","last_active":"2024-02-19T21:03:08.742946+0000","last_peered":"2024-02-19T21:03:08.742946+0000","last_clean":"2024-02-19T21:03:08.742946+0000","last_became_active":"2024-02-19T21:01:18.294369+0000","last_became_peered":"2024-02-19T21:01:18.294369+0000","last_unstale":"2024-02-19T21:03:08.742946+0000","last_undegraded":"2024-02-19T21:03:08.742946+0000","last_fullsized":"2024-02-19T21:03:08.742946+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:03:08.742876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.742876+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.456397+0000","last_change":"2024-02-19T21:02:38.633030+0000","last_active":"2024-02-19T21:02:52.456397+0000","last_peered":"2024-02-19T21:02:52.456397+0000","last_clean":"2024-02-19T21:02:52.456397+0000","last_became_active":"2024-02-19T21:01:22.608285+0000","last_became_peered":"2024-02-19T21:01:22.608285+0000","last_unstale":"2024-02-19T21:02:52.456397+0000","last_undegraded":"2024-02-19T21:02:52.456397+0000","last_fullsized":"2024-02-19T21:02:52.456397+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:38.632950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:38.632950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:08.984824+0000","last_change":"2024-02-19T21:03:08.984824+0000","last_active":"2024-02-19T21:03:08.984824+0000","last_peered":"2024-02-19T21:03:08.984824+0000","last_clean":"2024-02-19T21:03:08.984824+0000","last_became_active":"2024-02-19T21:01:20.288589+0000","last_became_peered":"2024-02-19T21:01:20.288589+0000","last_unstale":"2024-02-19T21:03:08.984824+0000","last_undegraded":"2024-02-19T21:03:08.984824+0000","last_fullsized":"2024-02-19T21:03:08.984824+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:08.984720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.984720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455647+0000","last_change":"2024-02-19T21:02:46.842812+0000","last_active":"2024-02-19T21:02:52.455647+0000","last_peered":"2024-02-19T21:02:52.455647+0000","last_clean":"2024-02-19T21:02:52.455647+0000","last_became_active":"2024-02-19T21:01:16.271210+0000","last_became_peered":"2024-02-19T21:01:16.271210+0000","last_unstale":"2024-02-19T21:02:52.455647+0000","last_undegraded":"2024-02-19T21:02:52.455647+0000","last_fullsized":"2024-02-19T21:02:52.455647+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:46.842732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.842732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":48,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.325798+0000","last_change":"2024-02-19T21:02:33.968248+0000","last_active":"2024-02-19T21:02:52.325798+0000","last_peered":"2024-02-19T21:02:52.325798+0000","last_clean":"2024-02-19T21:02:52.325798+0000","last_became_active":"2024-02-19T21:01:18.293913+0000","last_became_peered":"2024-02-19T21:01:18.293913+0000","last_unstale":"2024-02-19T21:02:52.325798+0000","last_undegraded":"2024-02-19T21:02:52.325798+0000","last_fullsized":"2024-02-19T21:02:52.325798+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:02:33.968182+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455180+0000","last_change":"2024-02-19T21:02:34.274719+0000","last_active":"2024-02-19T21:02:52.455180+0000","last_peered":"2024-02-19T21:02:52.455180+0000","last_clean":"2024-02-19T21:02:52.455180+0000","last_became_active":"2024-02-19T21:01:22.297086+0000","last_became_peered":"2024-02-19T21:01:22.297086+0000","last_unstale":"2024-02-19T21:02:52.455180+0000","last_undegraded":"2024-02-19T21:02:52.455180+0000","last_fullsized":"2024-02-19T21:02:52.455180+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:34.274650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:34.274650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.325956+0000","last_change":"2024-02-19T21:02:49.863142+0000","last_active":"2024-02-19T21:02:52.325956+0000","last_peered":"2024-02-19T21:02:52.325956+0000","last_clean":"2024-02-19T21:02:52.325956+0000","last_became_active":"2024-02-19T21:01:20.285298+0000","last_became_peered":"2024-02-19T21:01:20.285298+0000","last_unstale":"2024-02-19T21:02:52.325956+0000","last_undegraded":"2024-02-19T21:02:52.325956+0000","last_fullsized":"2024-02-19T21:02:52.325956+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.863037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.454878+0000","last_change":"2024-02-19T21:02:49.177055+0000","last_active":"2024-02-19T21:02:52.454878+0000","last_peered":"2024-02-19T21:02:52.454878+0000","last_clean":"2024-02-19T21:02:52.454878+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:02:52.454878+0000","last_undegraded":"2024-02-19T21:02:52.454878+0000","last_fullsized":"2024-02-19T21:02:52.454878+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.176969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.176969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":52,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:07.669010+0000","last_change":"2024-02-19T21:03:07.669010+0000","last_active":"2024-02-19T21:03:07.669010+0000","last_peered":"2024-02-19T21:03:07.669010+0000","last_clean":"2024-02-19T21:03:07.669010+0000","last_became_active":"2024-02-19T21:01:18.293773+0000","last_became_peered":"2024-02-19T21:01:18.293773+0000","last_unstale":"2024-02-19T21:03:07.669010+0000","last_undegraded":"2024-02-19T21:03:07.669010+0000","last_fullsized":"2024-02-19T21:03:07.669010+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:07.668943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:07.668943+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.327356+0000","last_change":"2024-02-19T21:01:22.296193+0000","last_active":"2024-02-19T21:02:52.327356+0000","last_peered":"2024-02-19T21:02:52.327356+0000","last_clean":"2024-02-19T21:02:52.327356+0000","last_became_active":"2024-02-19T21:01:22.295912+0000","last_became_peered":"2024-02-19T21:01:22.295912+0000","last_unstale":"2024-02-19T21:02:52.327356+0000","last_undegraded":"2024-02-19T21:02:52.327356+0000","last_fullsized":"2024-02-19T21:02:52.327356+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:01:21.273400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"58'2","reported_seq":51,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326586+0000","last_change":"2024-02-19T21:02:17.029682+0000","last_active":"2024-02-19T21:02:52.326586+0000","last_peered":"2024-02-19T21:02:52.326586+0000","last_clean":"2024-02-19T21:02:52.326586+0000","last_became_active":"2024-02-19T21:01:16.273545+0000","last_became_peered":"2024-02-19T21:01:16.273545+0000","last_unstale":"2024-02-19T21:02:52.326586+0000","last_undegraded":"2024-02-19T21:02:52.326586+0000","last_fullsized":"2024-02-19T21:02:52.326586+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-02-19T21:02:17.029611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:17.029611+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":59,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:08.046722+0000","last_change":"2024-02-19T21:03:08.046722+0000","last_active":"2024-02-19T21:03:08.046722+0000","last_peered":"2024-02-19T21:03:08.046722+0000","last_clean":"2024-02-19T21:03:08.046722+0000","last_became_active":"2024-02-19T21:01:18.285672+0000","last_became_peered":"2024-02-19T21:01:18.285672+0000","last_unstale":"2024-02-19T21:03:08.046722+0000","last_undegraded":"2024-02-19T21:03:08.046722+0000","last_fullsized":"2024-02-19T21:03:08.046722+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:03:08.046641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.046641+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455040+0000","last_change":"2024-02-19T21:02:35.252642+0000","last_active":"2024-02-19T21:02:52.455040+0000","last_peered":"2024-02-19T21:02:52.455040+0000","last_clean":"2024-02-19T21:02:52.455040+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:02:52.455040+0000","last_undegraded":"2024-02-19T21:02:52.455040+0000","last_fullsized":"2024-02-19T21:02:52.455040+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:35.252569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:35.252569+0000","last_clean_scrub_stamp":"2024-02-19T21:02:35.252569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.325200+0000","last_change":"2024-02-19T21:02:47.593210+0000","last_active":"2024-02-19T21:02:52.325200+0000","last_peered":"2024-02-19T21:02:52.325200+0000","last_clean":"2024-02-19T21:02:52.325200+0000","last_became_active":"2024-02-19T21:01:22.292133+0000","last_became_peered":"2024-02-19T21:01:22.292133+0000","last_unstale":"2024-02-19T21:02:52.325200+0000","last_undegraded":"2024-02-19T21:02:52.325200+0000","last_fullsized":"2024-02-19T21:02:52.325200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.593132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.593132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:01.530211+0000","last_change":"2024-02-19T21:03:01.530211+0000","last_active":"2024-02-19T21:03:01.530211+0000","last_peered":"2024-02-19T21:03:01.530211+0000","last_clean":"2024-02-19T21:03:01.530211+0000","last_became_active":"2024-02-19T21:01:30.926676+0000","last_became_peered":"2024-02-19T21:01:30.926676+0000","last_unstale":"2024-02-19T21:03:01.530211+0000","last_undegraded":"2024-02-19T21:03:01.530211+0000","last_fullsized":"2024-02-19T21:03:01.530211+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.530108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":51,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:56.539408+0000","last_change":"2024-02-19T21:02:56.539408+0000","last_active":"2024-02-19T21:02:56.539408+0000","last_peered":"2024-02-19T21:02:56.539408+0000","last_clean":"2024-02-19T21:02:56.539408+0000","last_became_active":"2024-02-19T21:01:18.285891+0000","last_became_peered":"2024-02-19T21:01:18.285891+0000","last_unstale":"2024-02-19T21:02:56.539408+0000","last_undegraded":"2024-02-19T21:02:56.539408+0000","last_fullsized":"2024-02-19T21:02:56.539408+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:02:56.539303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.539303+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":63,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:04.441736+0000","last_change":"2024-02-19T21:01:20.289497+0000","last_active":"2024-02-19T21:03:04.441736+0000","last_peered":"2024-02-19T21:03:04.441736+0000","last_clean":"2024-02-19T21:03:04.441736+0000","last_became_active":"2024-02-19T21:01:20.288918+0000","last_became_peered":"2024-02-19T21:01:20.288918+0000","last_unstale":"2024-02-19T21:03:04.441736+0000","last_undegraded":"2024-02-19T21:03:04.441736+0000","last_fullsized":"2024-02-19T21:03:04.441736+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:01:19.270373+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326507+0000","last_change":"2024-02-19T21:01:22.291717+0000","last_active":"2024-02-19T21:02:52.326507+0000","last_peered":"2024-02-19T21:02:52.326507+0000","last_clean":"2024-02-19T21:02:52.326507+0000","last_became_active":"2024-02-19T21:01:22.291396+0000","last_became_peered":"2024-02-19T21:01:22.291396+0000","last_unstale":"2024-02-19T21:02:52.326507+0000","last_undegraded":"2024-02-19T21:02:52.326507+0000","last_fullsized":"2024-02-19T21:02:52.326507+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:01:21.273400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:58.986556+0000","last_change":"2024-02-19T21:02:58.986556+0000","last_active":"2024-02-19T21:02:58.986556+0000","last_peered":"2024-02-19T21:02:58.986556+0000","last_clean":"2024-02-19T21:02:58.986556+0000","last_became_active":"2024-02-19T21:01:16.270159+0000","last_became_peered":"2024-02-19T21:01:16.270159+0000","last_unstale":"2024-02-19T21:02:58.986556+0000","last_undegraded":"2024-02-19T21:02:58.986556+0000","last_fullsized":"2024-02-19T21:02:58.986556+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:58.986458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:58.986458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":107,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:04.441759+0000","last_change":"2024-02-19T21:02:46.558811+0000","last_active":"2024-02-19T21:03:04.441759+0000","last_peered":"2024-02-19T21:03:04.441759+0000","last_clean":"2024-02-19T21:03:04.441759+0000","last_became_active":"2024-02-19T21:01:18.289076+0000","last_became_peered":"2024-02-19T21:01:18.289076+0000","last_unstale":"2024-02-19T21:03:04.441759+0000","last_undegraded":"2024-02-19T21:03:04.441759+0000","last_fullsized":"2024-02-19T21:03:04.441759+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-19T21:02:46.558739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.558739+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:56.510539+0000","last_change":"2024-02-19T21:02:56.510539+0000","last_active":"2024-02-19T21:02:56.510539+0000","last_peered":"2024-02-19T21:02:56.510539+0000","last_clean":"2024-02-19T21:02:56.510539+0000","last_became_active":"2024-02-19T21:01:20.290758+0000","last_became_peered":"2024-02-19T21:01:20.290758+0000","last_unstale":"2024-02-19T21:02:56.510539+0000","last_undegraded":"2024-02-19T21:02:56.510539+0000","last_fullsized":"2024-02-19T21:02:56.510539+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.510474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.475914+0000","last_change":"2024-02-19T21:02:32.488666+0000","last_active":"2024-02-19T21:02:52.475914+0000","last_peered":"2024-02-19T21:02:52.475914+0000","last_clean":"2024-02-19T21:02:52.475914+0000","last_became_active":"2024-02-19T21:01:22.607883+0000","last_became_peered":"2024-02-19T21:01:22.607883+0000","last_unstale":"2024-02-19T21:02:52.475914+0000","last_undegraded":"2024-02-19T21:02:52.475914+0000","last_fullsized":"2024-02-19T21:02:52.475914+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:32.488584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.488584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":148,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:08.723389+0000","last_change":"2024-02-19T21:02:55.568674+0000","last_active":"2024-02-19T21:03:08.723389+0000","last_peered":"2024-02-19T21:03:08.723389+0000","last_clean":"2024-02-19T21:03:08.723389+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:03:08.723389+0000","last_undegraded":"2024-02-19T21:03:08.723389+0000","last_fullsized":"2024-02-19T21:03:08.723389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:02:55.568593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.568593+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":107,"num_read_kb":102,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.454841+0000","last_change":"2024-02-19T21:02:31.357109+0000","last_active":"2024-02-19T21:02:52.454841+0000","last_peered":"2024-02-19T21:02:52.454841+0000","last_clean":"2024-02-19T21:02:52.454841+0000","last_became_active":"2024-02-19T21:01:16.276032+0000","last_became_peered":"2024-02-19T21:01:16.276032+0000","last_unstale":"2024-02-19T21:02:52.454841+0000","last_undegraded":"2024-02-19T21:02:52.454841+0000","last_fullsized":"2024-02-19T21:02:52.454841+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:31.357041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:31.357041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455008+0000","last_change":"2024-02-19T21:02:22.217326+0000","last_active":"2024-02-19T21:02:52.455008+0000","last_peered":"2024-02-19T21:02:52.455008+0000","last_clean":"2024-02-19T21:02:52.455008+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:02:52.455008+0000","last_undegraded":"2024-02-19T21:02:52.455008+0000","last_fullsized":"2024-02-19T21:02:52.455008+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-02-19T21:02:22.217222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:22.217222+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455403+0000","last_change":"2024-02-19T21:02:45.222033+0000","last_active":"2024-02-19T21:02:52.455403+0000","last_peered":"2024-02-19T21:02:52.455403+0000","last_clean":"2024-02-19T21:02:52.455403+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:02:52.455403+0000","last_undegraded":"2024-02-19T21:02:52.455403+0000","last_fullsized":"2024-02-19T21:02:52.455403+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:45.221965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:45.221965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.476000+0000","last_change":"2024-02-19T21:02:25.326504+0000","last_active":"2024-02-19T21:02:52.476000+0000","last_peered":"2024-02-19T21:02:52.476000+0000","last_clean":"2024-02-19T21:02:52.476000+0000","last_became_active":"2024-02-19T21:01:22.292299+0000","last_became_peered":"2024-02-19T21:01:22.292299+0000","last_unstale":"2024-02-19T21:02:52.476000+0000","last_undegraded":"2024-02-19T21:02:52.476000+0000","last_fullsized":"2024-02-19T21:02:52.476000+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:25.326441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:25.326441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326626+0000","last_change":"2024-02-19T21:01:16.274575+0000","last_active":"2024-02-19T21:02:52.326626+0000","last_peered":"2024-02-19T21:02:52.326626+0000","last_clean":"2024-02-19T21:02:52.326626+0000","last_became_active":"2024-02-19T21:01:16.274290+0000","last_became_peered":"2024-02-19T21:01:16.274290+0000","last_unstale":"2024-02-19T21:02:52.326626+0000","last_undegraded":"2024-02-19T21:02:52.326626+0000","last_fullsized":"2024-02-19T21:02:52.326626+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:01:15.241725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.987017+0000","last_change":"2024-02-19T21:02:52.986989+0000","last_active":"2024-02-19T21:02:52.987017+0000","last_peered":"2024-02-19T21:02:52.987017+0000","last_clean":"2024-02-19T21:02:52.987017+0000","last_became_active":"2024-02-19T21:01:16.276807+0000","last_became_peered":"2024-02-19T21:01:16.276807+0000","last_unstale":"2024-02-19T21:02:52.987017+0000","last_undegraded":"2024-02-19T21:02:52.987017+0000","last_fullsized":"2024-02-19T21:02:52.987017+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.986913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.986913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455253+0000","last_change":"2024-02-19T21:01:48.259896+0000","last_active":"2024-02-19T21:02:52.455253+0000","last_peered":"2024-02-19T21:02:52.455253+0000","last_clean":"2024-02-19T21:02:52.455253+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:02:52.455253+0000","last_undegraded":"2024-02-19T21:02:52.455253+0000","last_fullsized":"2024-02-19T21:02:52.455253+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:48.259812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:01:48.259812+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":58,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455963+0000","last_change":"2024-02-19T21:02:32.777906+0000","last_active":"2024-02-19T21:02:52.455963+0000","last_peered":"2024-02-19T21:02:52.455963+0000","last_clean":"2024-02-19T21:02:52.455963+0000","last_became_active":"2024-02-19T21:01:18.293784+0000","last_became_peered":"2024-02-19T21:01:18.293784+0000","last_unstale":"2024-02-19T21:02:52.455963+0000","last_undegraded":"2024-02-19T21:02:52.455963+0000","last_fullsized":"2024-02-19T21:02:52.455963+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:02:32.777827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.777827+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:55.586944+0000","last_change":"2024-02-19T21:02:55.586944+0000","last_active":"2024-02-19T21:02:55.586944+0000","last_peered":"2024-02-19T21:02:55.586944+0000","last_clean":"2024-02-19T21:02:55.586944+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:02:55.586944+0000","last_undegraded":"2024-02-19T21:02:55.586944+0000","last_fullsized":"2024-02-19T21:02:55.586944+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:55.586854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.586854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.556474+0000","last_change":"2024-02-19T21:02:52.556438+0000","last_active":"2024-02-19T21:02:52.556474+0000","last_peered":"2024-02-19T21:02:52.556474+0000","last_clean":"2024-02-19T21:02:52.556474+0000","last_became_active":"2024-02-19T21:01:22.298477+0000","last_became_peered":"2024-02-19T21:01:22.298477+0000","last_unstale":"2024-02-19T21:02:52.556474+0000","last_undegraded":"2024-02-19T21:02:52.556474+0000","last_fullsized":"2024-02-19T21:02:52.556474+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.556344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.556344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:57.505093+0000","last_change":"2024-02-19T21:02:57.505093+0000","last_active":"2024-02-19T21:02:57.505093+0000","last_peered":"2024-02-19T21:02:57.505093+0000","last_clean":"2024-02-19T21:02:57.505093+0000","last_became_active":"2024-02-19T21:01:16.274398+0000","last_became_peered":"2024-02-19T21:01:16.274398+0000","last_unstale":"2024-02-19T21:02:57.505093+0000","last_undegraded":"2024-02-19T21:02:57.505093+0000","last_fullsized":"2024-02-19T21:02:57.505093+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.505029+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":44,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455210+0000","last_change":"2024-02-19T21:02:41.616831+0000","last_active":"2024-02-19T21:02:52.455210+0000","last_peered":"2024-02-19T21:02:52.455210+0000","last_clean":"2024-02-19T21:02:52.455210+0000","last_became_active":"2024-02-19T21:01:16.275904+0000","last_became_peered":"2024-02-19T21:01:16.275904+0000","last_unstale":"2024-02-19T21:02:52.455210+0000","last_undegraded":"2024-02-19T21:02:52.455210+0000","last_fullsized":"2024-02-19T21:02:52.455210+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:02:41.616729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.616729+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":54,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326299+0000","last_change":"2024-02-19T21:02:18.707988+0000","last_active":"2024-02-19T21:02:52.326299+0000","last_peered":"2024-02-19T21:02:52.326299+0000","last_clean":"2024-02-19T21:02:52.326299+0000","last_became_active":"2024-02-19T21:01:18.287139+0000","last_became_peered":"2024-02-19T21:01:18.287139+0000","last_unstale":"2024-02-19T21:02:52.326299+0000","last_undegraded":"2024-02-19T21:02:52.326299+0000","last_fullsized":"2024-02-19T21:02:52.326299+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:02:18.707921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:18.707921+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:05.005360+0000","last_change":"2024-02-19T21:03:05.005360+0000","last_active":"2024-02-19T21:03:05.005360+0000","last_peered":"2024-02-19T21:03:05.005360+0000","last_clean":"2024-02-19T21:03:05.005360+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:03:05.005360+0000","last_undegraded":"2024-02-19T21:03:05.005360+0000","last_fullsized":"2024-02-19T21:03:05.005360+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.005288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.005288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:07.473156+0000","last_change":"2024-02-19T21:03:07.473156+0000","last_active":"2024-02-19T21:03:07.473156+0000","last_peered":"2024-02-19T21:03:07.473156+0000","last_clean":"2024-02-19T21:03:07.473156+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:03:07.473156+0000","last_undegraded":"2024-02-19T21:03:07.473156+0000","last_fullsized":"2024-02-19T21:03:07.473156+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:03:07.473070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.475963+0000","last_change":"2024-02-19T21:02:46.565447+0000","last_active":"2024-02-19T21:02:52.475963+0000","last_peered":"2024-02-19T21:02:52.475963+0000","last_clean":"2024-02-19T21:02:52.475963+0000","last_became_active":"2024-02-19T21:01:16.274215+0000","last_became_peered":"2024-02-19T21:01:16.274215+0000","last_unstale":"2024-02-19T21:02:52.475963+0000","last_undegraded":"2024-02-19T21:02:52.475963+0000","last_fullsized":"2024-02-19T21:02:52.475963+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:46.565380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.565380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":72,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.325696+0000","last_change":"2024-02-19T21:02:35.502975+0000","last_active":"2024-02-19T21:02:52.325696+0000","last_peered":"2024-02-19T21:02:52.325696+0000","last_clean":"2024-02-19T21:02:52.325696+0000","last_became_active":"2024-02-19T21:01:18.293791+0000","last_became_peered":"2024-02-19T21:01:18.293791+0000","last_unstale":"2024-02-19T21:02:52.325696+0000","last_undegraded":"2024-02-19T21:02:52.325696+0000","last_fullsized":"2024-02-19T21:02:52.325696+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:02:35.502906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:35.502906+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.456657+0000","last_change":"2024-02-19T21:02:27.761781+0000","last_active":"2024-02-19T21:02:52.456657+0000","last_peered":"2024-02-19T21:02:52.456657+0000","last_clean":"2024-02-19T21:02:52.456657+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:02:52.456657+0000","last_undegraded":"2024-02-19T21:02:52.456657+0000","last_fullsized":"2024-02-19T21:02:52.456657+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:27.761685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:27.761685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326307+0000","last_change":"2024-02-19T21:01:22.298108+0000","last_active":"2024-02-19T21:02:52.326307+0000","last_peered":"2024-02-19T21:02:52.326307+0000","last_clean":"2024-02-19T21:02:52.326307+0000","last_became_active":"2024-02-19T21:01:22.297707+0000","last_became_peered":"2024-02-19T21:01:22.297707+0000","last_unstale":"2024-02-19T21:02:52.326307+0000","last_undegraded":"2024-02-19T21:02:52.326307+0000","last_fullsized":"2024-02-19T21:02:52.326307+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:01:21.273400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.456120+0000","last_change":"2024-02-19T21:02:47.601190+0000","last_active":"2024-02-19T21:02:52.456120+0000","last_peered":"2024-02-19T21:02:52.456120+0000","last_clean":"2024-02-19T21:02:52.456120+0000","last_became_active":"2024-02-19T21:01:16.271340+0000","last_became_peered":"2024-02-19T21:01:16.271340+0000","last_unstale":"2024-02-19T21:02:52.456120+0000","last_undegraded":"2024-02-19T21:02:52.456120+0000","last_fullsized":"2024-02-19T21:02:52.456120+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.601127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.601127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":68,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.325626+0000","last_change":"2024-02-19T21:02:39.952691+0000","last_active":"2024-02-19T21:02:52.325626+0000","last_peered":"2024-02-19T21:02:52.325626+0000","last_clean":"2024-02-19T21:02:52.325626+0000","last_became_active":"2024-02-19T21:01:18.293812+0000","last_became_peered":"2024-02-19T21:01:18.293812+0000","last_unstale":"2024-02-19T21:02:52.325626+0000","last_undegraded":"2024-02-19T21:02:52.325626+0000","last_fullsized":"2024-02-19T21:02:52.325626+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:02:39.952624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.952624+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:57.997535+0000","last_change":"2024-02-19T21:02:57.997535+0000","last_active":"2024-02-19T21:02:57.997535+0000","last_peered":"2024-02-19T21:02:57.997535+0000","last_clean":"2024-02-19T21:02:57.997535+0000","last_became_active":"2024-02-19T21:01:20.291079+0000","last_became_peered":"2024-02-19T21:01:20.291079+0000","last_unstale":"2024-02-19T21:02:57.997535+0000","last_undegraded":"2024-02-19T21:02:57.997535+0000","last_fullsized":"2024-02-19T21:02:57.997535+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.997427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:01.496926+0000","last_change":"2024-02-19T21:03:01.496926+0000","last_active":"2024-02-19T21:03:01.496926+0000","last_peered":"2024-02-19T21:03:01.496926+0000","last_clean":"2024-02-19T21:03:01.496926+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:03:01.496926+0000","last_undegraded":"2024-02-19T21:03:01.496926+0000","last_fullsized":"2024-02-19T21:03:01.496926+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.496833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.496833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.456628+0000","last_change":"2024-02-19T21:02:50.921031+0000","last_active":"2024-02-19T21:02:52.456628+0000","last_peered":"2024-02-19T21:02:52.456628+0000","last_clean":"2024-02-19T21:02:52.456628+0000","last_became_active":"2024-02-19T21:01:16.274462+0000","last_became_peered":"2024-02-19T21:01:16.274462+0000","last_unstale":"2024-02-19T21:02:52.456628+0000","last_undegraded":"2024-02-19T21:02:52.456628+0000","last_fullsized":"2024-02-19T21:02:52.456628+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:50.920948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:50.920948+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":57,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.325842+0000","last_change":"2024-02-19T21:02:40.105227+0000","last_active":"2024-02-19T21:02:52.325842+0000","last_peered":"2024-02-19T21:02:52.325842+0000","last_clean":"2024-02-19T21:02:52.325842+0000","last_became_active":"2024-02-19T21:01:18.282702+0000","last_became_peered":"2024-02-19T21:01:18.282702+0000","last_unstale":"2024-02-19T21:02:52.325842+0000","last_undegraded":"2024-02-19T21:02:52.325842+0000","last_fullsized":"2024-02-19T21:02:52.325842+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:02:40.105141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.105141+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":65,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:04.441784+0000","last_change":"2024-02-19T21:02:21.923048+0000","last_active":"2024-02-19T21:03:04.441784+0000","last_peered":"2024-02-19T21:03:04.441784+0000","last_clean":"2024-02-19T21:03:04.441784+0000","last_became_active":"2024-02-19T21:01:20.286505+0000","last_became_peered":"2024-02-19T21:01:20.286505+0000","last_unstale":"2024-02-19T21:03:04.441784+0000","last_undegraded":"2024-02-19T21:03:04.441784+0000","last_fullsized":"2024-02-19T21:03:04.441784+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:21.922971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.922971+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.325793+0000","last_change":"2024-02-19T21:01:22.607484+0000","last_active":"2024-02-19T21:02:52.325793+0000","last_peered":"2024-02-19T21:02:52.325793+0000","last_clean":"2024-02-19T21:02:52.325793+0000","last_became_active":"2024-02-19T21:01:22.607007+0000","last_became_peered":"2024-02-19T21:01:22.607007+0000","last_unstale":"2024-02-19T21:02:52.325793+0000","last_undegraded":"2024-02-19T21:02:52.325793+0000","last_fullsized":"2024-02-19T21:02:52.325793+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:01:21.273400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:06.991276+0000","last_change":"2024-02-19T21:03:06.991276+0000","last_active":"2024-02-19T21:03:06.991276+0000","last_peered":"2024-02-19T21:03:06.991276+0000","last_clean":"2024-02-19T21:03:06.991276+0000","last_became_active":"2024-02-19T21:01:20.289375+0000","last_became_peered":"2024-02-19T21:01:20.289375+0000","last_unstale":"2024-02-19T21:03:06.991276+0000","last_undegraded":"2024-02-19T21:03:06.991276+0000","last_fullsized":"2024-02-19T21:03:06.991276+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.991199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.991199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455298+0000","last_change":"2024-02-19T21:02:27.567995+0000","last_active":"2024-02-19T21:02:52.455298+0000","last_peered":"2024-02-19T21:02:52.455298+0000","last_clean":"2024-02-19T21:02:52.455298+0000","last_became_active":"2024-02-19T21:01:16.275520+0000","last_became_peered":"2024-02-19T21:01:16.275520+0000","last_unstale":"2024-02-19T21:02:52.455298+0000","last_undegraded":"2024-02-19T21:02:52.455298+0000","last_fullsized":"2024-02-19T21:02:52.455298+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:27.567903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:27.567903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326600+0000","last_change":"2024-02-19T21:02:19.042725+0000","last_active":"2024-02-19T21:02:52.326600+0000","last_peered":"2024-02-19T21:02:52.326600+0000","last_clean":"2024-02-19T21:02:52.326600+0000","last_became_active":"2024-02-19T21:01:18.285670+0000","last_became_peered":"2024-02-19T21:01:18.285670+0000","last_unstale":"2024-02-19T21:02:52.326600+0000","last_undegraded":"2024-02-19T21:02:52.326600+0000","last_fullsized":"2024-02-19T21:02:52.326600+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:19.042650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:19.042650+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.475913+0000","last_change":"2024-02-19T21:02:41.195082+0000","last_active":"2024-02-19T21:02:52.475913+0000","last_peered":"2024-02-19T21:02:52.475913+0000","last_clean":"2024-02-19T21:02:52.475913+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:02:52.475913+0000","last_undegraded":"2024-02-19T21:02:52.475913+0000","last_fullsized":"2024-02-19T21:02:52.475913+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:41.194993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.194993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":65,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:04.441946+0000","last_change":"2024-02-19T21:02:48.027160+0000","last_active":"2024-02-19T21:03:04.441946+0000","last_peered":"2024-02-19T21:03:04.441946+0000","last_clean":"2024-02-19T21:03:04.441946+0000","last_became_active":"2024-02-19T21:01:20.293341+0000","last_became_peered":"2024-02-19T21:01:20.293341+0000","last_unstale":"2024-02-19T21:03:04.441946+0000","last_undegraded":"2024-02-19T21:03:04.441946+0000","last_fullsized":"2024-02-19T21:03:04.441946+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:48.027070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.027070+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.456101+0000","last_change":"2024-02-19T21:02:28.471214+0000","last_active":"2024-02-19T21:02:52.456101+0000","last_peered":"2024-02-19T21:02:52.456101+0000","last_clean":"2024-02-19T21:02:52.456101+0000","last_became_active":"2024-02-19T21:01:16.274939+0000","last_became_peered":"2024-02-19T21:01:16.274939+0000","last_unstale":"2024-02-19T21:02:52.456101+0000","last_undegraded":"2024-02-19T21:02:52.456101+0000","last_fullsized":"2024-02-19T21:02:52.456101+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:28.471131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:28.471131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":61,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.526211+0000","last_change":"2024-02-19T21:02:53.526211+0000","last_active":"2024-02-19T21:02:53.526211+0000","last_peered":"2024-02-19T21:02:53.526211+0000","last_clean":"2024-02-19T21:02:53.526211+0000","last_became_active":"2024-02-19T21:01:18.285534+0000","last_became_peered":"2024-02-19T21:01:18.285534+0000","last_unstale":"2024-02-19T21:02:53.526211+0000","last_undegraded":"2024-02-19T21:02:53.526211+0000","last_fullsized":"2024-02-19T21:02:53.526211+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:53.526141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:53.526141+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326396+0000","last_change":"2024-02-19T21:02:42.123367+0000","last_active":"2024-02-19T21:02:52.326396+0000","last_peered":"2024-02-19T21:02:52.326396+0000","last_clean":"2024-02-19T21:02:52.326396+0000","last_became_active":"2024-02-19T21:01:22.299993+0000","last_became_peered":"2024-02-19T21:01:22.299993+0000","last_unstale":"2024-02-19T21:02:52.326396+0000","last_undegraded":"2024-02-19T21:02:52.326396+0000","last_fullsized":"2024-02-19T21:02:52.326396+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:42.123299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:42.123299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":65,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:04.441742+0000","last_change":"2024-02-19T21:02:25.255767+0000","last_active":"2024-02-19T21:03:04.441742+0000","last_peered":"2024-02-19T21:03:04.441742+0000","last_clean":"2024-02-19T21:03:04.441742+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:03:04.441742+0000","last_undegraded":"2024-02-19T21:03:04.441742+0000","last_fullsized":"2024-02-19T21:03:04.441742+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:25.255648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:25.255648+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455642+0000","last_change":"2024-02-19T21:02:42.525447+0000","last_active":"2024-02-19T21:02:52.455642+0000","last_peered":"2024-02-19T21:02:52.455642+0000","last_clean":"2024-02-19T21:02:52.455642+0000","last_became_active":"2024-02-19T21:01:16.274086+0000","last_became_peered":"2024-02-19T21:01:16.274086+0000","last_unstale":"2024-02-19T21:02:52.455642+0000","last_undegraded":"2024-02-19T21:02:52.455642+0000","last_fullsized":"2024-02-19T21:02:52.455642+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:42.525345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:42.525345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"82'15","reported_seq":56,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:04.086368+0000","last_change":"2024-02-19T21:03:04.086368+0000","last_active":"2024-02-19T21:03:04.086368+0000","last_peered":"2024-02-19T21:03:04.086368+0000","last_clean":"2024-02-19T21:03:04.086368+0000","last_became_active":"2024-02-19T21:01:18.282805+0000","last_became_peered":"2024-02-19T21:01:18.282805+0000","last_unstale":"2024-02-19T21:03:04.086368+0000","last_undegraded":"2024-02-19T21:03:04.086368+0000","last_fullsized":"2024-02-19T21:03:04.086368+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.086282+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:54.197783+0000","last_change":"2024-02-19T21:02:54.197783+0000","last_active":"2024-02-19T21:02:54.197783+0000","last_peered":"2024-02-19T21:02:54.197783+0000","last_clean":"2024-02-19T21:02:54.197783+0000","last_became_active":"2024-02-19T21:01:22.298941+0000","last_became_peered":"2024-02-19T21:01:22.298941+0000","last_unstale":"2024-02-19T21:02:54.197783+0000","last_undegraded":"2024-02-19T21:02:54.197783+0000","last_fullsized":"2024-02-19T21:02:54.197783+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:54.197714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:54.197714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455474+0000","last_change":"2024-02-19T21:02:21.497900+0000","last_active":"2024-02-19T21:02:52.455474+0000","last_peered":"2024-02-19T21:02:52.455474+0000","last_clean":"2024-02-19T21:02:52.455474+0000","last_became_active":"2024-02-19T21:01:20.290687+0000","last_became_peered":"2024-02-19T21:01:20.290687+0000","last_unstale":"2024-02-19T21:02:52.455474+0000","last_undegraded":"2024-02-19T21:02:52.455474+0000","last_fullsized":"2024-02-19T21:02:52.455474+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.497831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.497831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":43,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455623+0000","last_change":"2024-02-19T21:02:40.628725+0000","last_active":"2024-02-19T21:02:52.455623+0000","last_peered":"2024-02-19T21:02:52.455623+0000","last_clean":"2024-02-19T21:02:52.455623+0000","last_became_active":"2024-02-19T21:01:16.274937+0000","last_became_peered":"2024-02-19T21:01:16.274937+0000","last_unstale":"2024-02-19T21:02:52.455623+0000","last_undegraded":"2024-02-19T21:02:52.455623+0000","last_fullsized":"2024-02-19T21:02:52.455623+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:02:40.628632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.628632+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":91,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.326446+0000","last_change":"2024-02-19T21:02:30.070342+0000","last_active":"2024-02-19T21:02:52.326446+0000","last_peered":"2024-02-19T21:02:52.326446+0000","last_clean":"2024-02-19T21:02:52.326446+0000","last_became_active":"2024-02-19T21:01:30.923214+0000","last_became_peered":"2024-02-19T21:01:30.923214+0000","last_unstale":"2024-02-19T21:02:52.326446+0000","last_undegraded":"2024-02-19T21:02:52.326446+0000","last_fullsized":"2024-02-19T21:02:52.326446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:02:30.070236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:30.070236+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:03.500049+0000","last_change":"2024-02-19T21:03:03.500049+0000","last_active":"2024-02-19T21:03:03.500049+0000","last_peered":"2024-02-19T21:03:03.500049+0000","last_clean":"2024-02-19T21:03:03.500049+0000","last_became_active":"2024-02-19T21:01:22.291759+0000","last_became_peered":"2024-02-19T21:01:22.291759+0000","last_unstale":"2024-02-19T21:03:03.500049+0000","last_undegraded":"2024-02-19T21:03:03.500049+0000","last_fullsized":"2024-02-19T21:03:03.500049+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:03.499960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:03.499960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:55.213761+0000","last_change":"2024-02-19T21:02:55.213761+0000","last_active":"2024-02-19T21:02:55.213761+0000","last_peered":"2024-02-19T21:02:55.213761+0000","last_clean":"2024-02-19T21:02:55.213761+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:02:55.213761+0000","last_undegraded":"2024-02-19T21:02:55.213761+0000","last_fullsized":"2024-02-19T21:02:55.213761+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:55.213692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.213692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455323+0000","last_change":"2024-02-19T21:02:21.212907+0000","last_active":"2024-02-19T21:02:52.455323+0000","last_peered":"2024-02-19T21:02:52.455323+0000","last_clean":"2024-02-19T21:02:52.455323+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:02:52.455323+0000","last_undegraded":"2024-02-19T21:02:52.455323+0000","last_fullsized":"2024-02-19T21:02:52.455323+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.212808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.212808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":67,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.456156+0000","last_change":"2024-02-19T21:02:39.779974+0000","last_active":"2024-02-19T21:02:52.456156+0000","last_peered":"2024-02-19T21:02:52.456156+0000","last_clean":"2024-02-19T21:02:52.456156+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:02:52.456156+0000","last_undegraded":"2024-02-19T21:02:52.456156+0000","last_fullsized":"2024-02-19T21:02:52.456156+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:02:39.779890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.779890+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.325256+0000","last_change":"2024-02-19T21:02:48.572976+0000","last_active":"2024-02-19T21:02:52.325256+0000","last_peered":"2024-02-19T21:02:52.325256+0000","last_clean":"2024-02-19T21:02:52.325256+0000","last_became_active":"2024-02-19T21:01:22.297998+0000","last_became_peered":"2024-02-19T21:01:22.297998+0000","last_unstale":"2024-02-19T21:02:52.325256+0000","last_undegraded":"2024-02-19T21:02:52.325256+0000","last_fullsized":"2024-02-19T21:02:52.325256+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:48.572877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.572877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:01.169350+0000","last_change":"2024-02-19T21:03:01.169350+0000","last_active":"2024-02-19T21:03:01.169350+0000","last_peered":"2024-02-19T21:03:01.169350+0000","last_clean":"2024-02-19T21:03:01.169350+0000","last_became_active":"2024-02-19T21:01:20.290566+0000","last_became_peered":"2024-02-19T21:01:20.290566+0000","last_unstale":"2024-02-19T21:03:01.169350+0000","last_undegraded":"2024-02-19T21:03:01.169350+0000","last_fullsized":"2024-02-19T21:03:01.169350+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.169279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.169279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455969+0000","last_change":"2024-02-19T21:02:22.516204+0000","last_active":"2024-02-19T21:02:52.455969+0000","last_peered":"2024-02-19T21:02:52.455969+0000","last_clean":"2024-02-19T21:02:52.455969+0000","last_became_active":"2024-02-19T21:01:16.270931+0000","last_became_peered":"2024-02-19T21:01:16.270931+0000","last_unstale":"2024-02-19T21:02:52.455969+0000","last_undegraded":"2024-02-19T21:02:52.455969+0000","last_fullsized":"2024-02-19T21:02:52.455969+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:22.516135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:22.516135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":52,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.643923+0000","last_change":"2024-02-19T21:02:52.643899+0000","last_active":"2024-02-19T21:02:52.643923+0000","last_peered":"2024-02-19T21:02:52.643923+0000","last_clean":"2024-02-19T21:02:52.643923+0000","last_became_active":"2024-02-19T21:01:18.288322+0000","last_became_peered":"2024-02-19T21:01:18.288322+0000","last_unstale":"2024-02-19T21:02:52.643923+0000","last_undegraded":"2024-02-19T21:02:52.643923+0000","last_fullsized":"2024-02-19T21:02:52.643923+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:52.643835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.643835+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455451+0000","last_change":"2024-02-19T21:01:22.607463+0000","last_active":"2024-02-19T21:02:52.455451+0000","last_peered":"2024-02-19T21:02:52.455451+0000","last_clean":"2024-02-19T21:02:52.455451+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:02:52.455451+0000","last_undegraded":"2024-02-19T21:02:52.455451+0000","last_fullsized":"2024-02-19T21:02:52.455451+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:01:21.273400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.455538+0000","last_change":"2024-02-19T21:02:37.258648+0000","last_active":"2024-02-19T21:02:52.455538+0000","last_peered":"2024-02-19T21:02:52.455538+0000","last_clean":"2024-02-19T21:02:52.455538+0000","last_became_active":"2024-02-19T21:01:16.271911+0000","last_became_peered":"2024-02-19T21:01:16.271911+0000","last_unstale":"2024-02-19T21:02:52.455538+0000","last_undegraded":"2024-02-19T21:02:52.455538+0000","last_fullsized":"2024-02-19T21:02:52.455538+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:37.258580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:37.258580+0000","last_clean_scrub_stamp":"2024-02-19T21:02:37.258580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":61,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:01.005530+0000","last_change":"2024-02-19T21:03:01.005530+0000","last_active":"2024-02-19T21:03:01.005530+0000","last_peered":"2024-02-19T21:03:01.005530+0000","last_clean":"2024-02-19T21:03:01.005530+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:03:01.005530+0000","last_undegraded":"2024-02-19T21:03:01.005530+0000","last_fullsized":"2024-02-19T21:03:01.005530+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:03:01.005445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.005445+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":64,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:04.441972+0000","last_change":"2024-02-19T21:02:44.563371+0000","last_active":"2024-02-19T21:03:04.441972+0000","last_peered":"2024-02-19T21:03:04.441972+0000","last_clean":"2024-02-19T21:03:04.441972+0000","last_became_active":"2024-02-19T21:01:20.288720+0000","last_became_peered":"2024-02-19T21:01:20.288720+0000","last_unstale":"2024-02-19T21:03:04.441972+0000","last_undegraded":"2024-02-19T21:03:04.441972+0000","last_fullsized":"2024-02-19T21:03:04.441972+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:44.563256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:44.563256+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":399,"ondisk_log_size":399,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":107,"num_read_kb":102,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266713,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":851968,"data_stored":349415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.53500000000000003}]},{"osd":1,"last update":"Mon Feb 19 21:02:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.5}]},{"osd":2,"last update":"Mon Feb 19 21:02:09 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.58899999999999997}]},{"osd":3,"last update":"Mon Feb 19 21:02:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.54800000000000004}]},{"osd":4,"last update":"Mon Feb 19 21:02:09 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.433},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.61299999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:02:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.626}]},{"osd":6,"last update":"Mon Feb 19 21:02:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.65700000000000003}]}]},{"osd":6,"up_from":45,"seq":193273528351,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297900,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447252,"statfs":{"total":95995035648,"available":95689986048,"internally_reserved":0,"allocated":823296,"data_stored":347879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.48799999999999999,"15min":0.49199999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.40100000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.49399999999999999,"15min":0.498},"min":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.57299999999999995}]},{"osd":2,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51300000000000001,"15min":0.51700000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.49299999999999999,"15min":0.49099999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.45800000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.503,"15min":0.497},"min":{"1min":0.38900000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.498}]},{"osd":4,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.39300000000000002,"15min":0.375},"min":{"1min":0.29299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.41299999999999998,"15min":0.40300000000000002},"min":{"1min":0.28899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.48099999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.42999999999999999,"15min":0.41199999999999998},"min":{"1min":0.34000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.46300000000000002,"15min":0.44700000000000001},"min":{"1min":0.39900000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.59599999999999997}]},{"osd":7,"last update":"Mon Feb 19 21:02:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.443},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.52600000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297936,"kb_used_data":840,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447216,"statfs":{"total":95995035648,"available":95689949184,"internally_reserved":0,"allocated":860160,"data_stored":349506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35899999999999999,"15min":0.34799999999999998},"min":{"1min":0.22500000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.38,"15min":0.35999999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:02:12 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.56299999999999994,"5min":0.62,"15min":0.62},"last":0.314},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.41399999999999998,"15min":0.41799999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.27300000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:02:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.41399999999999998,"15min":0.42399999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.51800000000000002,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.45500000000000002,"15min":0.46300000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58799999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.40200000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68300000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71699999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.73699999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57799999999999996,"15min":0.58399999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72799999999999998,"5min":0.872,"15min":0.872},"last":0.68700000000000006}]},{"osd":6,"last update":"Mon Feb 19 21:02:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.57699999999999996,"15min":0.58599999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71299999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63,"15min":0.64300000000000002},"min":{"1min":0.44600000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.68100000000000005,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Feb 19 21:02:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.58499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":851968,"data_stored":349069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.372,"15min":0.315},"min":{"1min":0.21299999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.46899999999999997,"5min":4.2830000000000004,"15min":4.2830000000000004},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.33300000000000002,"15min":0.308},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.48299999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.28000000000000003}]},{"osd":2,"last update":"Mon Feb 19 21:02:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.40600000000000003,"15min":0.38100000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":3.121,"5min":3.121,"15min":3.121},"last":0.222},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.42799999999999999,"15min":0.39000000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":4.2569999999999997,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.33800000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.47099999999999997,"15min":0.45100000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.503,"5min":4.4470000000000001,"15min":4.4470000000000001},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.48399999999999999,"15min":0.46100000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.504,"5min":4.3070000000000004,"15min":4.3070000000000004},"last":0.308}]},{"osd":4,"last update":"Mon Feb 19 21:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52600000000000002,"15min":0.51800000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55400000000000005,"15min":0.54000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73999999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.54500000000000004}]},{"osd":5,"last update":"Mon Feb 19 21:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.628,"15min":0.64100000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88300000000000001,"5min":1.581,"15min":1.581},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.78800000000000003,"15min":0.82799999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76900000000000002,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.1499999999999999,"5min":1.381,"15min":1.381},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.79600000000000004,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.58799999999999997}]},{"osd":7,"last update":"Mon Feb 19 21:02:10 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":4.226,"5min":4.226,"15min":4.226},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.57199999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":820,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":839680,"data_stored":348657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.36299999999999999,"15min":0.35399999999999998},"min":{"1min":0.245,"5min":0.222,"15min":0.222},"max":{"1min":0.47399999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.35699999999999998,"15min":0.34699999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.59399999999999997,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.40300000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:02:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.36199999999999999,"15min":0.35199999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.54800000000000004,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.40200000000000002,"15min":0.39500000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56100000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.46300000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:02:40 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.48499999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.46200000000000002,"15min":0.47399999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.56699999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.51600000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.58099999999999996,"15min":0.59999999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71899999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.60899999999999999,"15min":0.64300000000000002},"min":{"1min":0.38400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.74199999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.76500000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:02:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.63500000000000001,"15min":0.65500000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70899999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80300000000000005,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.69599999999999995}]},{"osd":6,"last update":"Mon Feb 19 21:02:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.62,"15min":0.629},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77800000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.622,"15min":0.63200000000000001},"min":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.76300000000000001,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.79100000000000004}]},{"osd":7,"last update":"Mon Feb 19 21:02:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.77500000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182445,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297940,"kb_used_data":844,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447212,"statfs":{"total":95995035648,"available":95689945088,"internally_reserved":0,"allocated":864256,"data_stored":348682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.35199999999999998,"15min":0.35399999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.39300000000000002,"15min":0.40300000000000002},"min":{"1min":0.27500000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.51600000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.39300000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.38600000000000001,"15min":0.39600000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.39500000000000002,"15min":0.39900000000000002},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59199999999999997,"5min":1.054,"15min":1.054},"last":0.34000000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41499999999999998,"15min":0.42199999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56399999999999995,"5min":0.625,"15min":0.625},"last":0.375},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.40999999999999998,"15min":0.41899999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.52900000000000003,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.29599999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.41199999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82799999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55700000000000005,"15min":0.56599999999999995},"min":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.83399999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.53100000000000003}]},{"osd":5,"last update":"Mon Feb 19 21:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79900000000000004,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55800000000000005,"15min":0.56200000000000006},"min":{"1min":0.38200000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72399999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:02:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56699999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85199999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.55700000000000005}]},{"osd":7,"last update":"Mon Feb 19 21:02:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.50900000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986217,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":832,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":851968,"data_stored":348695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.52500000000000002,"15min":0.54800000000000004},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.67800000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.50700000000000001,"15min":0.52300000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.59899999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.56299999999999994}]},{"osd":1,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.754,"5min":2.149,"15min":2.149},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.436,"5min":0.502,"15min":0.51900000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57799999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.55200000000000005}]},{"osd":2,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.53300000000000003,"15min":0.55700000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66400000000000003,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.52000000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.51700000000000002,"15min":0.52500000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.56499999999999995,"15min":0.56999999999999995},"min":{"1min":0.33400000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64200000000000002,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":0.57199999999999995}]},{"osd":5,"last update":"Mon Feb 19 21:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.41799999999999998,"15min":0.42799999999999999},"min":{"1min":0.26800000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.501,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.29899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.53600000000000003,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.40300000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.377,"15min":0.36899999999999999},"min":{"1min":0.28699999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.5},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.42899999999999999,"15min":0.42199999999999999},"min":{"1min":0.36199999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.45400000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:02:11 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.58499999999999996}]}]},{"osd":5,"up_from":37,"seq":158913789988,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297904,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447248,"statfs":{"total":95995035648,"available":95689981952,"internally_reserved":0,"allocated":827392,"data_stored":347150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.66500000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.53400000000000003,"15min":0.54300000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.70199999999999996,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.501}]},{"osd":1,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52000000000000002,"15min":0.52500000000000002},"min":{"1min":0.38,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.52400000000000002,"15min":0.53200000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.58799999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.45900000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.53200000000000003,"15min":0.54000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.63200000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.53000000000000003,"15min":0.53500000000000003},"min":{"1min":0.40799999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.65400000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.50800000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52700000000000002,"15min":0.53500000000000003},"min":{"1min":0.36099999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.57899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.40600000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71199999999999997,"5min":0.747,"15min":0.747},"last":0.39900000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.39200000000000002,"15min":0.38600000000000001},"min":{"1min":0.30099999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.56299999999999994,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.39400000000000002,"15min":0.38400000000000001},"min":{"1min":0.30099999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.434}]},{"osd":6,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42499999999999999,"15min":0.41699999999999998},"min":{"1min":0.311,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.441,"15min":0.432},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.624,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.52400000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:02:14 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.56699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:03:15.282 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:03:15.555 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:15.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:14.475169+0000 mgr.y (mgr.14550) 81 : audit [DBG] from='client.34664 v1:172.21.15.163:0/1839329750' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:03:15.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:14.475169+0000 mgr.y (mgr.14550) 81 : audit [DBG] from='client.34664 v1:172.21.15.163:0/1839329750' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:03:15.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:14.475169+0000 mgr.y (mgr.14550) 81 : audit [DBG] from='client.34664 v1:172.21.15.163:0/1839329750' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:03:16.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:14.933501+0000 mgr.y (mgr.14550) 82 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:16.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:16.045326+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:03:16.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:16.047252+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:03:16.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:14.933501+0000 mgr.y (mgr.14550) 82 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:16.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:16.045326+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:03:16.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:16.047252+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:03:16.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:14.933501+0000 mgr.y (mgr.14550) 82 : cluster [DBG] pgmap v62: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:16.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:16.045326+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:03:16.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:16.047252+0000 osd.4 (osd.4) 42 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:03:17.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.182+0000 7f9001755700 1 -- 172.21.15.163:0/3104995639 >> v1:172.21.15.163:6790/0 conn(0x7f8ffc0fd960 legacy=0x7f8ffc0fdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:17.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.182+0000 7f9001755700 1 -- 172.21.15.163:0/3104995639 shutdown_connections 2024-02-19T21:03:17.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.182+0000 7f9001755700 1 -- 172.21.15.163:0/3104995639 >> 172.21.15.163:0/3104995639 conn(0x7f8ffc06cc00 msgr2=0x7f8ffc0f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:17.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.182+0000 7f9001755700 1 -- 172.21.15.163:0/3104995639 shutdown_connections 2024-02-19T21:03:17.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.182+0000 7f9001755700 1 -- 172.21.15.163:0/3104995639 wait complete. 2024-02-19T21:03:17.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.182+0000 7f9001755700 1 librados: starting msgr at 2024-02-19T21:03:17.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.182+0000 7f9001755700 1 librados: starting objecter 2024-02-19T21:03:17.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.182+0000 7f9001755700 1 Processor -- start 2024-02-19T21:03:17.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f9001755700 1 -- start start 2024-02-19T21:03:17.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f9001755700 1 librados: setting wanted keys 2024-02-19T21:03:17.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f9001755700 1 librados: calling monclient init 2024-02-19T21:03:17.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f9001755700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8ffc0fcfc0 con 0x7f8ffc1007b0 2024-02-19T21:03:17.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f9001755700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8ffc0fd150 con 0x7f8ffc0fd960 2024-02-19T21:03:17.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f9001755700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8ffc0fd340 con 0x7f8ffc1042c0 2024-02-19T21:03:17.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8ffa7fc700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8ffc1007b0 0x7f8ffc114d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:34532/0 (socket says 172.21.15.163:34532) 2024-02-19T21:03:17.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8ffa7fc700 1 -- 172.21.15.163:0/1379314935 learned_addr learned my addr 172.21.15.163:0/1379314935 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:17.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 10 client.?.objecter ms_handle_connect 0x7f8ffc1007b0 2024-02-19T21:03:17.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:17.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 10 client.?.objecter ms_handle_connect 0x7f8ffc1042c0 2024-02-19T21:03:17.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:17.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 10 client.?.objecter ms_handle_connect 0x7f8ffc0fd960 2024-02-19T21:03:17.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:17.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312879434 0 0) 0x7f8ffc0fcfc0 con 0x7f8ffc1007b0 2024-02-19T21:03:17.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe4003800 con 0x7f8ffc1007b0 2024-02-19T21:03:17.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114938938 0 0) 0x7f8ffc0fd340 con 0x7f8ffc1042c0 2024-02-19T21:03:17.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ffc0fcfc0 con 0x7f8ffc1042c0 2024-02-19T21:03:17.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790772406 0 0) 0x7f8ffc0fd150 con 0x7f8ffc0fd960 2024-02-19T21:03:17.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ffc0fd340 con 0x7f8ffc0fd960 2024-02-19T21:03:17.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1361540360 0 0) 0x7f8fe4003800 con 0x7f8ffc1007b0 2024-02-19T21:03:17.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ffc0fd150 con 0x7f8ffc1007b0 2024-02-19T21:03:17.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 101575362 0 0) 0x7f8ffc0fcfc0 con 0x7f8ffc1042c0 2024-02-19T21:03:17.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fe4003800 con 0x7f8ffc1042c0 2024-02-19T21:03:17.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8fec001a70 con 0x7f8ffc1007b0 2024-02-19T21:03:17.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.183+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8ff4001bc0 con 0x7f8ffc1042c0 2024-02-19T21:03:17.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.184+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2819857245 0 0) 0x7f8fe4003800 con 0x7f8ffc1042c0 2024-02-19T21:03:17.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.184+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 >> v1:172.21.15.193:6789/0 conn(0x7f8ffc0fd960 legacy=0x7f8ffc0fc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:17.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.184+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 >> v1:172.21.15.163:6789/0 conn(0x7f8ffc1007b0 legacy=0x7f8ffc114d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:17.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.184+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ffc0fd530 con 0x7f8ffc1042c0 2024-02-19T21:03:17.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.184+0000 7f9001755700 1 -- 172.21.15.163:0/1379314935 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ffc119bd0 con 0x7f8ffc1042c0 2024-02-19T21:03:17.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.184+0000 7f9001755700 10 client.34670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:17.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.184+0000 7f9001755700 1 -- 172.21.15.163:0/1379314935 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ffc11a0c0 con 0x7f8ffc1042c0 2024-02-19T21:03:17.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.184+0000 7f9001755700 1 librados: init done 2024-02-19T21:03:17.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.184+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8ff4002200 con 0x7f8ffc1042c0 2024-02-19T21:03:17.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.184+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8ff4003460 con 0x7f8ffc1042c0 2024-02-19T21:03:17.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.185+0000 7f9001755700 1 -- 172.21.15.163:0/1379314935 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe8004f20 con 0x7f8ffc1042c0 2024-02-19T21:03:17.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.185+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f8ff4001d20 con 0x7f8ffc1042c0 2024-02-19T21:03:17.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.186+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(83..83 src has 32..83) v4 ==== 6172+0+0 (unknown 4093497947 0 0) 0x7f8ff4075ee0 con 0x7f8ffc1042c0 2024-02-19T21:03:17.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.186+0000 7f8fe3fff700 10 client.34670.objecter ms_dispatch 0x7f8ffc001550 osd_map(83..83 src has 32..83) v4 2024-02-19T21:03:17.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.186+0000 7f8fe3fff700 3 client.34670.objecter handle_osd_map got epochs [83,83] > 0 2024-02-19T21:03:17.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.186+0000 7f8fe3fff700 3 client.34670.objecter handle_osd_map decoding full epoch 83 2024-02-19T21:03:17.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.186+0000 7f8fe3fff700 20 client.34670.objecter dump_active .. 0 homeless 2024-02-19T21:03:17.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.186+0000 7f8fe3fff700 10 client.34670.objecter ms_handle_connect 0x7f8fe4063c70 2024-02-19T21:03:17.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.190+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8ff4049a90 con 0x7f8ffc1042c0 2024-02-19T21:03:17.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.350+0000 7f9001755700 1 -- 172.21.15.163:0/1379314935 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8fe8000b90 con 0x7f8fe4063c70 2024-02-19T21:03:17.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.360+0000 7f8fe3fff700 1 -- 172.21.15.163:0/1379314935 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357393 (unknown 2965378022 0 2307020898) 0x7f8fe8000b90 con 0x7f8fe4063c70 2024-02-19T21:03:17.355 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:03:17.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.363+0000 7f9001755700 10 librados: watch_flush enter 2024-02-19T21:03:17.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.363+0000 7f9001755700 10 librados: watch_flush exit 2024-02-19T21:03:17.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.363+0000 7f9001755700 20 client.34670.objecter shutdown clearing up homeless session... 2024-02-19T21:03:17.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.363+0000 7f9001755700 10 client.34670.objecter successfully canceled tick 2024-02-19T21:03:17.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.363+0000 7f9001755700 1 -- 172.21.15.163:0/1379314935 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8fe4063c70 legacy=0x7f8fe4066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:17.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.363+0000 7f9001755700 1 -- 172.21.15.163:0/1379314935 >> v1:172.21.15.163:6790/0 conn(0x7f8ffc1042c0 legacy=0x7f8ffc118450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:17.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.363+0000 7f9001755700 1 -- 172.21.15.163:0/1379314935 shutdown_connections 2024-02-19T21:03:17.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.363+0000 7f9001755700 1 -- 172.21.15.163:0/1379314935 >> 172.21.15.163:0/1379314935 conn(0x7f8ffc06cc00 msgr2=0x7f8ffc107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:17.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.363+0000 7f9001755700 1 -- 172.21.15.163:0/1379314935 shutdown_connections 2024-02-19T21:03:17.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.363+0000 7f9001755700 1 -- 172.21.15.163:0/1379314935 wait complete. 2024-02-19T21:03:17.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:17.363+0000 7f9001755700 1 librados: shutdown 2024-02-19T21:03:17.359 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:03:17.711 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:17.087739+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.2 scrub starts 2024-02-19T21:03:17.712 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:17.088691+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.2 scrub ok 2024-02-19T21:03:17.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:17.087739+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.2 scrub starts 2024-02-19T21:03:17.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:17.088691+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.2 scrub ok 2024-02-19T21:03:17.763 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:17.087739+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.2 scrub starts 2024-02-19T21:03:17.763 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:17.088691+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.2 scrub ok 2024-02-19T21:03:18.062 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:03:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:03:18.131 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-02-19T21:03:16.933647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":831,"num_read_kb":583,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":474,"ondisk_log_size":474,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2383236,"kb_used_data":4788,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747577980,"statfs":{"total":767960285184,"available":765519851520,"internally_reserved":0,"allocated":4902912,"data_stored":2239373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002119"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.966454+0000","last_change":"2024-02-19T21:01:22.609338+0000","last_active":"2024-02-19T21:02:52.966454+0000","last_peered":"2024-02-19T21:02:52.966454+0000","last_clean":"2024-02-19T21:02:52.966454+0000","last_became_active":"2024-02-19T21:01:22.609059+0000","last_became_peered":"2024-02-19T21:01:22.609059+0000","last_unstale":"2024-02-19T21:02:52.966454+0000","last_undegraded":"2024-02-19T21:02:52.966454+0000","last_fullsized":"2024-02-19T21:02:52.966454+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:01:21.273400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:05.073773+0000","last_change":"2024-02-19T21:03:05.073746+0000","last_active":"2024-02-19T21:03:05.073773+0000","last_peered":"2024-02-19T21:03:05.073773+0000","last_clean":"2024-02-19T21:03:05.073773+0000","last_became_active":"2024-02-19T21:01:20.289208+0000","last_became_peered":"2024-02-19T21:01:20.289208+0000","last_unstale":"2024-02-19T21:03:05.073773+0000","last_undegraded":"2024-02-19T21:03:05.073773+0000","last_fullsized":"2024-02-19T21:03:05.073773+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.073664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.073664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:04.644779+0000","last_change":"2024-02-19T21:03:04.644754+0000","last_active":"2024-02-19T21:03:04.644779+0000","last_peered":"2024-02-19T21:03:04.644779+0000","last_clean":"2024-02-19T21:03:04.644779+0000","last_became_active":"2024-02-19T21:01:16.278753+0000","last_became_peered":"2024-02-19T21:01:16.278753+0000","last_unstale":"2024-02-19T21:03:04.644779+0000","last_undegraded":"2024-02-19T21:03:04.644779+0000","last_fullsized":"2024-02-19T21:03:04.644779+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.644689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.644689+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":62,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.967476+0000","last_change":"2024-02-19T21:02:48.863931+0000","last_active":"2024-02-19T21:02:52.967476+0000","last_peered":"2024-02-19T21:02:52.967476+0000","last_clean":"2024-02-19T21:02:52.967476+0000","last_became_active":"2024-02-19T21:01:18.283222+0000","last_became_peered":"2024-02-19T21:01:18.283222+0000","last_unstale":"2024-02-19T21:02:52.967476+0000","last_undegraded":"2024-02-19T21:02:52.967476+0000","last_fullsized":"2024-02-19T21:02:52.967476+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:48.863852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.863852+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965502+0000","last_change":"2024-02-19T21:02:23.100006+0000","last_active":"2024-02-19T21:02:52.965502+0000","last_peered":"2024-02-19T21:02:52.965502+0000","last_clean":"2024-02-19T21:02:52.965502+0000","last_became_active":"2024-02-19T21:01:22.293432+0000","last_became_peered":"2024-02-19T21:01:22.293432+0000","last_unstale":"2024-02-19T21:02:52.965502+0000","last_undegraded":"2024-02-19T21:02:52.965502+0000","last_fullsized":"2024-02-19T21:02:52.965502+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:23.099932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:23.099932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056115+0000","last_change":"2024-02-19T21:02:29.480341+0000","last_active":"2024-02-19T21:02:53.056115+0000","last_peered":"2024-02-19T21:02:53.056115+0000","last_clean":"2024-02-19T21:02:53.056115+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:02:53.056115+0000","last_undegraded":"2024-02-19T21:02:53.056115+0000","last_fullsized":"2024-02-19T21:02:53.056115+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:29.480275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:29.480275+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.968564+0000","last_change":"2024-02-19T21:02:20.970293+0000","last_active":"2024-02-19T21:02:52.968564+0000","last_peered":"2024-02-19T21:02:52.968564+0000","last_clean":"2024-02-19T21:02:52.968564+0000","last_became_active":"2024-02-19T21:01:16.272132+0000","last_became_peered":"2024-02-19T21:01:16.272132+0000","last_unstale":"2024-02-19T21:02:52.968564+0000","last_undegraded":"2024-02-19T21:02:52.968564+0000","last_fullsized":"2024-02-19T21:02:52.968564+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:20.970216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:20.970216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":48,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965496+0000","last_change":"2024-02-19T21:01:18.287914+0000","last_active":"2024-02-19T21:02:52.965496+0000","last_peered":"2024-02-19T21:02:52.965496+0000","last_clean":"2024-02-19T21:02:52.965496+0000","last_became_active":"2024-02-19T21:01:18.287645+0000","last_became_peered":"2024-02-19T21:01:18.287645+0000","last_unstale":"2024-02-19T21:02:52.965496+0000","last_undegraded":"2024-02-19T21:02:52.965496+0000","last_fullsized":"2024-02-19T21:02:52.965496+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:01:17.258309+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:13.096737+0000","last_change":"2024-02-19T21:03:13.096737+0000","last_active":"2024-02-19T21:03:13.096737+0000","last_peered":"2024-02-19T21:03:13.096737+0000","last_clean":"2024-02-19T21:03:13.096737+0000","last_became_active":"2024-02-19T21:01:22.293282+0000","last_became_peered":"2024-02-19T21:01:22.293282+0000","last_unstale":"2024-02-19T21:03:13.096737+0000","last_undegraded":"2024-02-19T21:03:13.096737+0000","last_fullsized":"2024-02-19T21:03:13.096737+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:13.096667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:13.096667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056288+0000","last_change":"2024-02-19T21:02:35.546613+0000","last_active":"2024-02-19T21:02:53.056288+0000","last_peered":"2024-02-19T21:02:53.056288+0000","last_clean":"2024-02-19T21:02:53.056288+0000","last_became_active":"2024-02-19T21:01:20.287940+0000","last_became_peered":"2024-02-19T21:01:20.287940+0000","last_unstale":"2024-02-19T21:02:53.056288+0000","last_undegraded":"2024-02-19T21:02:53.056288+0000","last_fullsized":"2024-02-19T21:02:53.056288+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:35.546526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:35.546526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.055844+0000","last_change":"2024-02-19T21:02:20.575203+0000","last_active":"2024-02-19T21:02:53.055844+0000","last_peered":"2024-02-19T21:02:53.055844+0000","last_clean":"2024-02-19T21:02:53.055844+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:02:53.055844+0000","last_undegraded":"2024-02-19T21:02:53.055844+0000","last_fullsized":"2024-02-19T21:02:53.055844+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:20.575133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:20.575133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":47,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965792+0000","last_change":"2024-02-19T21:02:27.098930+0000","last_active":"2024-02-19T21:02:52.965792+0000","last_peered":"2024-02-19T21:02:52.965792+0000","last_clean":"2024-02-19T21:02:52.965792+0000","last_became_active":"2024-02-19T21:01:18.285735+0000","last_became_peered":"2024-02-19T21:01:18.285735+0000","last_unstale":"2024-02-19T21:02:52.965792+0000","last_undegraded":"2024-02-19T21:02:52.965792+0000","last_fullsized":"2024-02-19T21:02:52.965792+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:02:27.098844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:27.098844+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:00.570790+0000","last_change":"2024-02-19T21:03:00.570766+0000","last_active":"2024-02-19T21:03:00.570790+0000","last_peered":"2024-02-19T21:03:00.570790+0000","last_clean":"2024-02-19T21:03:00.570790+0000","last_became_active":"2024-02-19T21:01:22.292714+0000","last_became_peered":"2024-02-19T21:01:22.292714+0000","last_unstale":"2024-02-19T21:03:00.570790+0000","last_undegraded":"2024-02-19T21:03:00.570790+0000","last_fullsized":"2024-02-19T21:03:00.570790+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:00.570677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:00.570677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":30,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.995973+0000","last_change":"2024-02-19T21:01:20.293436+0000","last_active":"2024-02-19T21:02:52.995973+0000","last_peered":"2024-02-19T21:02:52.995973+0000","last_clean":"2024-02-19T21:02:52.995973+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:02:52.995973+0000","last_undegraded":"2024-02-19T21:02:52.995973+0000","last_fullsized":"2024-02-19T21:02:52.995973+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:01:19.270373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:57.553024+0000","last_change":"2024-02-19T21:02:57.552979+0000","last_active":"2024-02-19T21:02:57.553024+0000","last_peered":"2024-02-19T21:02:57.553024+0000","last_clean":"2024-02-19T21:02:57.553024+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:02:57.553024+0000","last_undegraded":"2024-02-19T21:02:57.553024+0000","last_fullsized":"2024-02-19T21:02:57.553024+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.552886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.552886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":53,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965612+0000","last_change":"2024-02-19T21:02:26.073713+0000","last_active":"2024-02-19T21:02:52.965612+0000","last_peered":"2024-02-19T21:02:52.965612+0000","last_clean":"2024-02-19T21:02:52.965612+0000","last_became_active":"2024-02-19T21:01:18.285269+0000","last_became_peered":"2024-02-19T21:01:18.285269+0000","last_unstale":"2024-02-19T21:02:52.965612+0000","last_undegraded":"2024-02-19T21:02:52.965612+0000","last_fullsized":"2024-02-19T21:02:52.965612+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:26.073625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:26.073625+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.968229+0000","last_change":"2024-02-19T21:02:24.948910+0000","last_active":"2024-02-19T21:02:52.968229+0000","last_peered":"2024-02-19T21:02:52.968229+0000","last_clean":"2024-02-19T21:02:52.968229+0000","last_became_active":"2024-02-19T21:01:22.607099+0000","last_became_peered":"2024-02-19T21:01:22.607099+0000","last_unstale":"2024-02-19T21:02:52.968229+0000","last_undegraded":"2024-02-19T21:02:52.968229+0000","last_fullsized":"2024-02-19T21:02:52.968229+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:24.948832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:24.948832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:09.010450+0000","last_change":"2024-02-19T21:03:09.010420+0000","last_active":"2024-02-19T21:03:09.010450+0000","last_peered":"2024-02-19T21:03:09.010450+0000","last_clean":"2024-02-19T21:03:09.010450+0000","last_became_active":"2024-02-19T21:01:20.291672+0000","last_became_peered":"2024-02-19T21:01:20.291672+0000","last_unstale":"2024-02-19T21:03:09.010450+0000","last_undegraded":"2024-02-19T21:03:09.010450+0000","last_fullsized":"2024-02-19T21:03:09.010450+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:09.010341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:09.010341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.967598+0000","last_change":"2024-02-19T21:01:16.275072+0000","last_active":"2024-02-19T21:02:52.967598+0000","last_peered":"2024-02-19T21:02:52.967598+0000","last_clean":"2024-02-19T21:02:52.967598+0000","last_became_active":"2024-02-19T21:01:16.274811+0000","last_became_peered":"2024-02-19T21:01:16.274811+0000","last_unstale":"2024-02-19T21:02:52.967598+0000","last_undegraded":"2024-02-19T21:02:52.967598+0000","last_fullsized":"2024-02-19T21:02:52.967598+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:01:15.241725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":60,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:57.819268+0000","last_change":"2024-02-19T21:02:57.819231+0000","last_active":"2024-02-19T21:02:57.819268+0000","last_peered":"2024-02-19T21:02:57.819268+0000","last_clean":"2024-02-19T21:02:57.819268+0000","last_became_active":"2024-02-19T21:01:18.285926+0000","last_became_peered":"2024-02-19T21:01:18.285926+0000","last_unstale":"2024-02-19T21:02:57.819268+0000","last_undegraded":"2024-02-19T21:02:57.819268+0000","last_fullsized":"2024-02-19T21:02:57.819268+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:02:57.819125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.819125+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.081314+0000","last_change":"2024-02-19T21:03:12.081314+0000","last_active":"2024-02-19T21:03:12.081314+0000","last_peered":"2024-02-19T21:03:12.081314+0000","last_clean":"2024-02-19T21:03:12.081314+0000","last_became_active":"2024-02-19T21:01:22.607435+0000","last_became_peered":"2024-02-19T21:01:22.607435+0000","last_unstale":"2024-02-19T21:03:12.081314+0000","last_undegraded":"2024-02-19T21:03:12.081314+0000","last_fullsized":"2024-02-19T21:03:12.081314+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:03:12.081227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:54.544978+0000","last_change":"2024-02-19T21:02:54.544941+0000","last_active":"2024-02-19T21:02:54.544978+0000","last_peered":"2024-02-19T21:02:54.544978+0000","last_clean":"2024-02-19T21:02:54.544978+0000","last_became_active":"2024-02-19T21:01:20.290705+0000","last_became_peered":"2024-02-19T21:01:20.290705+0000","last_unstale":"2024-02-19T21:02:54.544978+0000","last_undegraded":"2024-02-19T21:02:54.544978+0000","last_fullsized":"2024-02-19T21:02:54.544978+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:54.544834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:54.544834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.055841+0000","last_change":"2024-02-19T21:02:47.805245+0000","last_active":"2024-02-19T21:02:53.055841+0000","last_peered":"2024-02-19T21:02:53.055841+0000","last_clean":"2024-02-19T21:02:53.055841+0000","last_became_active":"2024-02-19T21:01:16.274524+0000","last_became_peered":"2024-02-19T21:01:16.274524+0000","last_unstale":"2024-02-19T21:02:53.055841+0000","last_undegraded":"2024-02-19T21:02:53.055841+0000","last_fullsized":"2024-02-19T21:02:53.055841+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.805171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.805171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":62,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965843+0000","last_change":"2024-02-19T21:02:47.238547+0000","last_active":"2024-02-19T21:02:52.965843+0000","last_peered":"2024-02-19T21:02:52.965843+0000","last_clean":"2024-02-19T21:02:52.965843+0000","last_became_active":"2024-02-19T21:01:18.289712+0000","last_became_peered":"2024-02-19T21:01:18.289712+0000","last_unstale":"2024-02-19T21:02:52.965843+0000","last_undegraded":"2024-02-19T21:02:52.965843+0000","last_fullsized":"2024-02-19T21:02:52.965843+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:47.238470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.238470+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:06.456338+0000","last_change":"2024-02-19T21:03:06.456310+0000","last_active":"2024-02-19T21:03:06.456338+0000","last_peered":"2024-02-19T21:03:06.456338+0000","last_clean":"2024-02-19T21:03:06.456338+0000","last_became_active":"2024-02-19T21:01:22.293230+0000","last_became_peered":"2024-02-19T21:01:22.293230+0000","last_unstale":"2024-02-19T21:03:06.456338+0000","last_undegraded":"2024-02-19T21:03:06.456338+0000","last_fullsized":"2024-02-19T21:03:06.456338+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.456212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.456212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:08.636073+0000","last_change":"2024-02-19T21:03:08.636037+0000","last_active":"2024-02-19T21:03:08.636073+0000","last_peered":"2024-02-19T21:03:08.636073+0000","last_clean":"2024-02-19T21:03:08.636073+0000","last_became_active":"2024-02-19T21:01:20.288016+0000","last_became_peered":"2024-02-19T21:01:20.288016+0000","last_unstale":"2024-02-19T21:03:08.636073+0000","last_undegraded":"2024-02-19T21:03:08.636073+0000","last_fullsized":"2024-02-19T21:03:08.636073+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:08.635943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.635943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.967126+0000","last_change":"2024-02-19T21:01:16.278289+0000","last_active":"2024-02-19T21:02:52.967126+0000","last_peered":"2024-02-19T21:02:52.967126+0000","last_clean":"2024-02-19T21:02:52.967126+0000","last_became_active":"2024-02-19T21:01:16.277826+0000","last_became_peered":"2024-02-19T21:01:16.277826+0000","last_unstale":"2024-02-19T21:02:52.967126+0000","last_undegraded":"2024-02-19T21:02:52.967126+0000","last_fullsized":"2024-02-19T21:02:52.967126+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:01:15.241725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":56,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056872+0000","last_change":"2024-02-19T21:02:29.684765+0000","last_active":"2024-02-19T21:02:53.056872+0000","last_peered":"2024-02-19T21:02:53.056872+0000","last_clean":"2024-02-19T21:02:53.056872+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:02:53.056872+0000","last_undegraded":"2024-02-19T21:02:53.056872+0000","last_fullsized":"2024-02-19T21:02:53.056872+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:29.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:29.684698+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056059+0000","last_change":"2024-02-19T21:02:36.585127+0000","last_active":"2024-02-19T21:02:53.056059+0000","last_peered":"2024-02-19T21:02:53.056059+0000","last_clean":"2024-02-19T21:02:53.056059+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:02:53.056059+0000","last_undegraded":"2024-02-19T21:02:53.056059+0000","last_fullsized":"2024-02-19T21:02:53.056059+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:02:36.585007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:36.585007+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":70,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.599379+0000","last_change":"2024-02-19T21:02:37.842634+0000","last_active":"2024-02-19T21:03:11.599379+0000","last_peered":"2024-02-19T21:03:11.599379+0000","last_clean":"2024-02-19T21:03:11.599379+0000","last_became_active":"2024-02-19T21:01:20.290567+0000","last_became_peered":"2024-02-19T21:01:20.290567+0000","last_unstale":"2024-02-19T21:03:11.599379+0000","last_undegraded":"2024-02-19T21:03:11.599379+0000","last_fullsized":"2024-02-19T21:03:11.599379+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:37.842545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:37.842545+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:06.000850+0000","last_change":"2024-02-19T21:03:06.000805+0000","last_active":"2024-02-19T21:03:06.000850+0000","last_peered":"2024-02-19T21:03:06.000850+0000","last_clean":"2024-02-19T21:03:06.000850+0000","last_became_active":"2024-02-19T21:01:16.272284+0000","last_became_peered":"2024-02-19T21:01:16.272284+0000","last_unstale":"2024-02-19T21:03:06.000850+0000","last_undegraded":"2024-02-19T21:03:06.000850+0000","last_fullsized":"2024-02-19T21:03:06.000850+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.000732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.000732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":52,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056870+0000","last_change":"2024-02-19T21:02:32.550264+0000","last_active":"2024-02-19T21:02:53.056870+0000","last_peered":"2024-02-19T21:02:53.056870+0000","last_clean":"2024-02-19T21:02:53.056870+0000","last_became_active":"2024-02-19T21:01:18.293865+0000","last_became_peered":"2024-02-19T21:01:18.293865+0000","last_unstale":"2024-02-19T21:02:53.056870+0000","last_undegraded":"2024-02-19T21:02:53.056870+0000","last_fullsized":"2024-02-19T21:02:53.056870+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:02:32.550190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.550190+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.967731+0000","last_change":"2024-02-19T21:02:41.964875+0000","last_active":"2024-02-19T21:02:52.967731+0000","last_peered":"2024-02-19T21:02:52.967731+0000","last_clean":"2024-02-19T21:02:52.967731+0000","last_became_active":"2024-02-19T21:01:22.608997+0000","last_became_peered":"2024-02-19T21:01:22.608997+0000","last_unstale":"2024-02-19T21:02:52.967731+0000","last_undegraded":"2024-02-19T21:02:52.967731+0000","last_fullsized":"2024-02-19T21:02:52.967731+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:41.964809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.964809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:04.486197+0000","last_change":"2024-02-19T21:03:04.486168+0000","last_active":"2024-02-19T21:03:04.486197+0000","last_peered":"2024-02-19T21:03:04.486197+0000","last_clean":"2024-02-19T21:03:04.486197+0000","last_became_active":"2024-02-19T21:01:20.287961+0000","last_became_peered":"2024-02-19T21:01:20.287961+0000","last_unstale":"2024-02-19T21:03:04.486197+0000","last_undegraded":"2024-02-19T21:03:04.486197+0000","last_fullsized":"2024-02-19T21:03:04.486197+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.486067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.486067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056969+0000","last_change":"2024-02-19T21:02:43.790280+0000","last_active":"2024-02-19T21:02:53.056969+0000","last_peered":"2024-02-19T21:02:53.056969+0000","last_clean":"2024-02-19T21:02:53.056969+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:02:53.056969+0000","last_undegraded":"2024-02-19T21:02:53.056969+0000","last_fullsized":"2024-02-19T21:02:53.056969+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:43.790206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:43.790206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":56,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.967391+0000","last_change":"2024-02-19T21:02:52.859531+0000","last_active":"2024-02-19T21:02:52.967391+0000","last_peered":"2024-02-19T21:02:52.967391+0000","last_clean":"2024-02-19T21:02:52.967391+0000","last_became_active":"2024-02-19T21:01:18.293793+0000","last_became_peered":"2024-02-19T21:01:18.293793+0000","last_unstale":"2024-02-19T21:02:52.967391+0000","last_undegraded":"2024-02-19T21:02:52.967391+0000","last_fullsized":"2024-02-19T21:02:52.967391+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:52.859453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.859453+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056143+0000","last_change":"2024-02-19T21:02:45.579070+0000","last_active":"2024-02-19T21:02:53.056143+0000","last_peered":"2024-02-19T21:02:53.056143+0000","last_clean":"2024-02-19T21:02:53.056143+0000","last_became_active":"2024-02-19T21:01:22.297693+0000","last_became_peered":"2024-02-19T21:01:22.297693+0000","last_unstale":"2024-02-19T21:02:53.056143+0000","last_undegraded":"2024-02-19T21:02:53.056143+0000","last_fullsized":"2024-02-19T21:02:53.056143+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:45.578978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:45.578978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.055643+0000","last_change":"2024-02-19T21:02:30.724973+0000","last_active":"2024-02-19T21:02:53.055643+0000","last_peered":"2024-02-19T21:02:53.055643+0000","last_clean":"2024-02-19T21:02:53.055643+0000","last_became_active":"2024-02-19T21:01:20.290962+0000","last_became_peered":"2024-02-19T21:01:20.290962+0000","last_unstale":"2024-02-19T21:02:53.055643+0000","last_undegraded":"2024-02-19T21:02:53.055643+0000","last_fullsized":"2024-02-19T21:02:53.055643+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:30.724908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:30.724908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.996267+0000","last_change":"2024-02-19T21:02:52.021686+0000","last_active":"2024-02-19T21:02:52.996267+0000","last_peered":"2024-02-19T21:02:52.996267+0000","last_clean":"2024-02-19T21:02:52.996267+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:02:52.996267+0000","last_undegraded":"2024-02-19T21:02:52.996267+0000","last_fullsized":"2024-02-19T21:02:52.996267+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.021603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.021603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":43,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.966504+0000","last_change":"2024-02-19T21:02:22.939275+0000","last_active":"2024-02-19T21:02:52.966504+0000","last_peered":"2024-02-19T21:02:52.966504+0000","last_clean":"2024-02-19T21:02:52.966504+0000","last_became_active":"2024-02-19T21:01:18.287874+0000","last_became_peered":"2024-02-19T21:01:18.287874+0000","last_unstale":"2024-02-19T21:02:52.966504+0000","last_undegraded":"2024-02-19T21:02:52.966504+0000","last_fullsized":"2024-02-19T21:02:52.966504+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:02:22.939192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:22.939192+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:05.734107+0000","last_change":"2024-02-19T21:03:05.734069+0000","last_active":"2024-02-19T21:03:05.734107+0000","last_peered":"2024-02-19T21:03:05.734107+0000","last_clean":"2024-02-19T21:03:05.734107+0000","last_became_active":"2024-02-19T21:01:22.292704+0000","last_became_peered":"2024-02-19T21:01:22.292704+0000","last_unstale":"2024-02-19T21:03:05.734107+0000","last_undegraded":"2024-02-19T21:03:05.734107+0000","last_fullsized":"2024-02-19T21:03:05.734107+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.733980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.733980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056890+0000","last_change":"2024-02-19T21:02:40.600482+0000","last_active":"2024-02-19T21:02:53.056890+0000","last_peered":"2024-02-19T21:02:53.056890+0000","last_clean":"2024-02-19T21:02:53.056890+0000","last_became_active":"2024-02-19T21:01:20.289020+0000","last_became_peered":"2024-02-19T21:01:20.289020+0000","last_unstale":"2024-02-19T21:02:53.056890+0000","last_undegraded":"2024-02-19T21:02:53.056890+0000","last_fullsized":"2024-02-19T21:02:53.056890+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:40.600410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.600410+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056363+0000","last_change":"2024-02-19T21:02:18.594112+0000","last_active":"2024-02-19T21:02:53.056363+0000","last_peered":"2024-02-19T21:02:53.056363+0000","last_clean":"2024-02-19T21:02:53.056363+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:02:53.056363+0000","last_undegraded":"2024-02-19T21:02:53.056363+0000","last_fullsized":"2024-02-19T21:02:53.056363+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:18.594043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:18.594043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":56,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965697+0000","last_change":"2024-02-19T21:02:32.040809+0000","last_active":"2024-02-19T21:02:52.965697+0000","last_peered":"2024-02-19T21:02:52.965697+0000","last_clean":"2024-02-19T21:02:52.965697+0000","last_became_active":"2024-02-19T21:01:18.286128+0000","last_became_peered":"2024-02-19T21:01:18.286128+0000","last_unstale":"2024-02-19T21:02:52.965697+0000","last_undegraded":"2024-02-19T21:02:52.965697+0000","last_fullsized":"2024-02-19T21:02:52.965697+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:32.040731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.040731+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.055771+0000","last_change":"2024-02-19T21:02:51.631722+0000","last_active":"2024-02-19T21:02:53.055771+0000","last_peered":"2024-02-19T21:02:53.055771+0000","last_clean":"2024-02-19T21:02:53.055771+0000","last_became_active":"2024-02-19T21:01:22.293093+0000","last_became_peered":"2024-02-19T21:01:22.293093+0000","last_unstale":"2024-02-19T21:02:53.055771+0000","last_undegraded":"2024-02-19T21:02:53.055771+0000","last_fullsized":"2024-02-19T21:02:53.055771+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:51.631657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:51.631657+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:03.776534+0000","last_change":"2024-02-19T21:03:03.776497+0000","last_active":"2024-02-19T21:03:03.776534+0000","last_peered":"2024-02-19T21:03:03.776534+0000","last_clean":"2024-02-19T21:03:03.776534+0000","last_became_active":"2024-02-19T21:01:20.289691+0000","last_became_peered":"2024-02-19T21:01:20.289691+0000","last_unstale":"2024-02-19T21:03:03.776534+0000","last_undegraded":"2024-02-19T21:03:03.776534+0000","last_fullsized":"2024-02-19T21:03:03.776534+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_clean_scrub_stamp":"2024-02-19T21:03:03.776397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056630+0000","last_change":"2024-02-19T21:02:21.657425+0000","last_active":"2024-02-19T21:02:53.056630+0000","last_peered":"2024-02-19T21:02:53.056630+0000","last_clean":"2024-02-19T21:02:53.056630+0000","last_became_active":"2024-02-19T21:01:16.271316+0000","last_became_peered":"2024-02-19T21:01:16.271316+0000","last_unstale":"2024-02-19T21:02:53.056630+0000","last_undegraded":"2024-02-19T21:02:53.056630+0000","last_fullsized":"2024-02-19T21:02:53.056630+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.657334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.657334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.995759+0000","last_change":"2024-02-19T21:01:31.611926+0000","last_active":"2024-02-19T21:02:52.995759+0000","last_peered":"2024-02-19T21:02:52.995759+0000","last_clean":"2024-02-19T21:02:52.995759+0000","last_became_active":"2024-02-19T21:01:30.922577+0000","last_became_peered":"2024-02-19T21:01:30.922577+0000","last_unstale":"2024-02-19T21:02:52.995759+0000","last_undegraded":"2024-02-19T21:02:52.995759+0000","last_fullsized":"2024-02-19T21:02:52.995759+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:01:17.258309+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.966256+0000","last_change":"2024-02-19T21:02:39.092233+0000","last_active":"2024-02-19T21:02:52.966256+0000","last_peered":"2024-02-19T21:02:52.966256+0000","last_clean":"2024-02-19T21:02:52.966256+0000","last_became_active":"2024-02-19T21:01:22.299969+0000","last_became_peered":"2024-02-19T21:01:22.299969+0000","last_unstale":"2024-02-19T21:02:52.966256+0000","last_undegraded":"2024-02-19T21:02:52.966256+0000","last_fullsized":"2024-02-19T21:02:52.966256+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.092154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":62,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.396290+0000","last_change":"2024-02-19T21:01:20.284282+0000","last_active":"2024-02-19T21:03:11.396290+0000","last_peered":"2024-02-19T21:03:11.396290+0000","last_clean":"2024-02-19T21:03:11.396290+0000","last_became_active":"2024-02-19T21:01:20.283943+0000","last_became_peered":"2024-02-19T21:01:20.283943+0000","last_unstale":"2024-02-19T21:03:11.396290+0000","last_undegraded":"2024-02-19T21:03:11.396290+0000","last_fullsized":"2024-02-19T21:03:11.396290+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:01:19.270373+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056192+0000","last_change":"2024-02-19T21:02:21.693740+0000","last_active":"2024-02-19T21:02:53.056192+0000","last_peered":"2024-02-19T21:02:53.056192+0000","last_clean":"2024-02-19T21:02:53.056192+0000","last_became_active":"2024-02-19T21:01:16.273181+0000","last_became_peered":"2024-02-19T21:01:16.273181+0000","last_unstale":"2024-02-19T21:02:53.056192+0000","last_undegraded":"2024-02-19T21:02:53.056192+0000","last_fullsized":"2024-02-19T21:02:53.056192+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.693675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.693675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":53,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:57.021524+0000","last_change":"2024-02-19T21:02:57.021496+0000","last_active":"2024-02-19T21:02:57.021524+0000","last_peered":"2024-02-19T21:02:57.021524+0000","last_clean":"2024-02-19T21:02:57.021524+0000","last_became_active":"2024-02-19T21:01:18.293786+0000","last_became_peered":"2024-02-19T21:01:18.293786+0000","last_unstale":"2024-02-19T21:02:57.021524+0000","last_undegraded":"2024-02-19T21:02:57.021524+0000","last_fullsized":"2024-02-19T21:02:57.021524+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:57.021403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.021403+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056298+0000","last_change":"2024-02-19T21:02:39.442600+0000","last_active":"2024-02-19T21:02:53.056298+0000","last_peered":"2024-02-19T21:02:53.056298+0000","last_clean":"2024-02-19T21:02:53.056298+0000","last_became_active":"2024-02-19T21:01:22.297188+0000","last_became_peered":"2024-02-19T21:01:22.297188+0000","last_unstale":"2024-02-19T21:02:53.056298+0000","last_undegraded":"2024-02-19T21:02:53.056298+0000","last_fullsized":"2024-02-19T21:02:53.056298+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:02:39.442533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.442533+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":67,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079933+0000","last_change":"2024-02-19T21:03:11.063302+0000","last_active":"2024-02-19T21:03:12.079933+0000","last_peered":"2024-02-19T21:03:12.079933+0000","last_clean":"2024-02-19T21:03:12.079933+0000","last_became_active":"2024-02-19T21:01:20.288799+0000","last_became_peered":"2024-02-19T21:01:20.288799+0000","last_unstale":"2024-02-19T21:03:12.079933+0000","last_undegraded":"2024-02-19T21:03:12.079933+0000","last_fullsized":"2024-02-19T21:03:12.079933+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:11.063222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:11.063222+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.995841+0000","last_change":"2024-02-19T21:02:49.056495+0000","last_active":"2024-02-19T21:02:52.995841+0000","last_peered":"2024-02-19T21:02:52.995841+0000","last_clean":"2024-02-19T21:02:52.995841+0000","last_became_active":"2024-02-19T21:01:16.271608+0000","last_became_peered":"2024-02-19T21:01:16.271608+0000","last_unstale":"2024-02-19T21:02:52.995841+0000","last_undegraded":"2024-02-19T21:02:52.995841+0000","last_fullsized":"2024-02-19T21:02:52.995841+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.056412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.056412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":52,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:08.742977+0000","last_change":"2024-02-19T21:03:08.742946+0000","last_active":"2024-02-19T21:03:08.742977+0000","last_peered":"2024-02-19T21:03:08.742977+0000","last_clean":"2024-02-19T21:03:08.742977+0000","last_became_active":"2024-02-19T21:01:18.294369+0000","last_became_peered":"2024-02-19T21:01:18.294369+0000","last_unstale":"2024-02-19T21:03:08.742977+0000","last_undegraded":"2024-02-19T21:03:08.742977+0000","last_fullsized":"2024-02-19T21:03:08.742977+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:03:08.742876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.742876+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.055759+0000","last_change":"2024-02-19T21:02:38.633030+0000","last_active":"2024-02-19T21:02:53.055759+0000","last_peered":"2024-02-19T21:02:53.055759+0000","last_clean":"2024-02-19T21:02:53.055759+0000","last_became_active":"2024-02-19T21:01:22.608285+0000","last_became_peered":"2024-02-19T21:01:22.608285+0000","last_unstale":"2024-02-19T21:02:53.055759+0000","last_undegraded":"2024-02-19T21:02:53.055759+0000","last_fullsized":"2024-02-19T21:02:53.055759+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:38.632950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:38.632950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:08.984853+0000","last_change":"2024-02-19T21:03:08.984824+0000","last_active":"2024-02-19T21:03:08.984853+0000","last_peered":"2024-02-19T21:03:08.984853+0000","last_clean":"2024-02-19T21:03:08.984853+0000","last_became_active":"2024-02-19T21:01:20.288589+0000","last_became_peered":"2024-02-19T21:01:20.288589+0000","last_unstale":"2024-02-19T21:03:08.984853+0000","last_undegraded":"2024-02-19T21:03:08.984853+0000","last_fullsized":"2024-02-19T21:03:08.984853+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:08.984720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.984720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.055720+0000","last_change":"2024-02-19T21:02:46.842812+0000","last_active":"2024-02-19T21:02:53.055720+0000","last_peered":"2024-02-19T21:02:53.055720+0000","last_clean":"2024-02-19T21:02:53.055720+0000","last_became_active":"2024-02-19T21:01:16.271210+0000","last_became_peered":"2024-02-19T21:01:16.271210+0000","last_unstale":"2024-02-19T21:02:53.055720+0000","last_undegraded":"2024-02-19T21:02:53.055720+0000","last_fullsized":"2024-02-19T21:02:53.055720+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:46.842732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.842732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":49,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.968152+0000","last_change":"2024-02-19T21:02:33.968248+0000","last_active":"2024-02-19T21:02:52.968152+0000","last_peered":"2024-02-19T21:02:52.968152+0000","last_clean":"2024-02-19T21:02:52.968152+0000","last_became_active":"2024-02-19T21:01:18.293913+0000","last_became_peered":"2024-02-19T21:01:18.293913+0000","last_unstale":"2024-02-19T21:02:52.968152+0000","last_undegraded":"2024-02-19T21:02:52.968152+0000","last_fullsized":"2024-02-19T21:02:52.968152+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:02:33.968182+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965541+0000","last_change":"2024-02-19T21:02:34.274719+0000","last_active":"2024-02-19T21:02:52.965541+0000","last_peered":"2024-02-19T21:02:52.965541+0000","last_clean":"2024-02-19T21:02:52.965541+0000","last_became_active":"2024-02-19T21:01:22.297086+0000","last_became_peered":"2024-02-19T21:01:22.297086+0000","last_unstale":"2024-02-19T21:02:52.965541+0000","last_undegraded":"2024-02-19T21:02:52.965541+0000","last_fullsized":"2024-02-19T21:02:52.965541+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:34.274650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:34.274650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.968313+0000","last_change":"2024-02-19T21:02:49.863142+0000","last_active":"2024-02-19T21:02:52.968313+0000","last_peered":"2024-02-19T21:02:52.968313+0000","last_clean":"2024-02-19T21:02:52.968313+0000","last_became_active":"2024-02-19T21:01:20.285298+0000","last_became_peered":"2024-02-19T21:01:20.285298+0000","last_unstale":"2024-02-19T21:02:52.968313+0000","last_undegraded":"2024-02-19T21:02:52.968313+0000","last_fullsized":"2024-02-19T21:02:52.968313+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.863037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965355+0000","last_change":"2024-02-19T21:02:49.177055+0000","last_active":"2024-02-19T21:02:52.965355+0000","last_peered":"2024-02-19T21:02:52.965355+0000","last_clean":"2024-02-19T21:02:52.965355+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:02:52.965355+0000","last_undegraded":"2024-02-19T21:02:52.965355+0000","last_fullsized":"2024-02-19T21:02:52.965355+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.176969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.176969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":53,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:07.669036+0000","last_change":"2024-02-19T21:03:07.669010+0000","last_active":"2024-02-19T21:03:07.669036+0000","last_peered":"2024-02-19T21:03:07.669036+0000","last_clean":"2024-02-19T21:03:07.669036+0000","last_became_active":"2024-02-19T21:01:18.293773+0000","last_became_peered":"2024-02-19T21:01:18.293773+0000","last_unstale":"2024-02-19T21:03:07.669036+0000","last_undegraded":"2024-02-19T21:03:07.669036+0000","last_fullsized":"2024-02-19T21:03:07.669036+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:07.668943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:07.668943+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.967690+0000","last_change":"2024-02-19T21:01:22.296193+0000","last_active":"2024-02-19T21:02:52.967690+0000","last_peered":"2024-02-19T21:02:52.967690+0000","last_clean":"2024-02-19T21:02:52.967690+0000","last_became_active":"2024-02-19T21:01:22.295912+0000","last_became_peered":"2024-02-19T21:01:22.295912+0000","last_unstale":"2024-02-19T21:02:52.967690+0000","last_undegraded":"2024-02-19T21:02:52.967690+0000","last_fullsized":"2024-02-19T21:02:52.967690+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:01:21.273400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"58'2","reported_seq":52,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.966141+0000","last_change":"2024-02-19T21:02:17.029682+0000","last_active":"2024-02-19T21:02:52.966141+0000","last_peered":"2024-02-19T21:02:52.966141+0000","last_clean":"2024-02-19T21:02:52.966141+0000","last_became_active":"2024-02-19T21:01:16.273545+0000","last_became_peered":"2024-02-19T21:01:16.273545+0000","last_unstale":"2024-02-19T21:02:52.966141+0000","last_undegraded":"2024-02-19T21:02:52.966141+0000","last_fullsized":"2024-02-19T21:02:52.966141+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'2","last_scrub_stamp":"2024-02-19T21:02:17.029611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:17.029611+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":60,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:08.046760+0000","last_change":"2024-02-19T21:03:08.046722+0000","last_active":"2024-02-19T21:03:08.046760+0000","last_peered":"2024-02-19T21:03:08.046760+0000","last_clean":"2024-02-19T21:03:08.046760+0000","last_became_active":"2024-02-19T21:01:18.285672+0000","last_became_peered":"2024-02-19T21:01:18.285672+0000","last_unstale":"2024-02-19T21:03:08.046760+0000","last_undegraded":"2024-02-19T21:03:08.046760+0000","last_fullsized":"2024-02-19T21:03:08.046760+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:03:08.046641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.046641+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965686+0000","last_change":"2024-02-19T21:02:35.252642+0000","last_active":"2024-02-19T21:02:52.965686+0000","last_peered":"2024-02-19T21:02:52.965686+0000","last_clean":"2024-02-19T21:02:52.965686+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:02:52.965686+0000","last_undegraded":"2024-02-19T21:02:52.965686+0000","last_fullsized":"2024-02-19T21:02:52.965686+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:35.252569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:35.252569+0000","last_clean_scrub_stamp":"2024-02-19T21:02:35.252569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.055766+0000","last_change":"2024-02-19T21:02:47.593210+0000","last_active":"2024-02-19T21:02:53.055766+0000","last_peered":"2024-02-19T21:02:53.055766+0000","last_clean":"2024-02-19T21:02:53.055766+0000","last_became_active":"2024-02-19T21:01:22.292133+0000","last_became_peered":"2024-02-19T21:01:22.292133+0000","last_unstale":"2024-02-19T21:02:53.055766+0000","last_undegraded":"2024-02-19T21:02:53.055766+0000","last_fullsized":"2024-02-19T21:02:53.055766+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.593132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.593132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":48,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:01.530244+0000","last_change":"2024-02-19T21:03:01.530211+0000","last_active":"2024-02-19T21:03:01.530244+0000","last_peered":"2024-02-19T21:03:01.530244+0000","last_clean":"2024-02-19T21:03:01.530244+0000","last_became_active":"2024-02-19T21:01:30.926676+0000","last_became_peered":"2024-02-19T21:01:30.926676+0000","last_unstale":"2024-02-19T21:03:01.530244+0000","last_undegraded":"2024-02-19T21:03:01.530244+0000","last_fullsized":"2024-02-19T21:03:01.530244+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.530108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":52,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:56.539448+0000","last_change":"2024-02-19T21:02:56.539408+0000","last_active":"2024-02-19T21:02:56.539448+0000","last_peered":"2024-02-19T21:02:56.539448+0000","last_clean":"2024-02-19T21:02:56.539448+0000","last_became_active":"2024-02-19T21:01:18.285891+0000","last_became_peered":"2024-02-19T21:01:18.285891+0000","last_unstale":"2024-02-19T21:02:56.539448+0000","last_undegraded":"2024-02-19T21:02:56.539448+0000","last_fullsized":"2024-02-19T21:02:56.539448+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:02:56.539303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.539303+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":70,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393509+0000","last_change":"2024-02-19T21:03:10.626933+0000","last_active":"2024-02-19T21:03:11.393509+0000","last_peered":"2024-02-19T21:03:11.393509+0000","last_clean":"2024-02-19T21:03:11.393509+0000","last_became_active":"2024-02-19T21:01:20.288918+0000","last_became_peered":"2024-02-19T21:01:20.288918+0000","last_unstale":"2024-02-19T21:03:11.393509+0000","last_undegraded":"2024-02-19T21:03:11.393509+0000","last_fullsized":"2024-02-19T21:03:11.393509+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:10.626866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:10.626866+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.734312+0000","last_change":"2024-02-19T21:03:12.734312+0000","last_active":"2024-02-19T21:03:12.734312+0000","last_peered":"2024-02-19T21:03:12.734312+0000","last_clean":"2024-02-19T21:03:12.734312+0000","last_became_active":"2024-02-19T21:01:22.291396+0000","last_became_peered":"2024-02-19T21:01:22.291396+0000","last_unstale":"2024-02-19T21:03:12.734312+0000","last_undegraded":"2024-02-19T21:03:12.734312+0000","last_fullsized":"2024-02-19T21:03:12.734312+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:12.734219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:12.734219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:58.986604+0000","last_change":"2024-02-19T21:02:58.986556+0000","last_active":"2024-02-19T21:02:58.986604+0000","last_peered":"2024-02-19T21:02:58.986604+0000","last_clean":"2024-02-19T21:02:58.986604+0000","last_became_active":"2024-02-19T21:01:16.270159+0000","last_became_peered":"2024-02-19T21:01:16.270159+0000","last_unstale":"2024-02-19T21:02:58.986604+0000","last_undegraded":"2024-02-19T21:02:58.986604+0000","last_fullsized":"2024-02-19T21:02:58.986604+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:58.986458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:58.986458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":109,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445891+0000","last_change":"2024-02-19T21:02:46.558811+0000","last_active":"2024-02-19T21:03:12.445891+0000","last_peered":"2024-02-19T21:03:12.445891+0000","last_clean":"2024-02-19T21:03:12.445891+0000","last_became_active":"2024-02-19T21:01:18.289076+0000","last_became_peered":"2024-02-19T21:01:18.289076+0000","last_unstale":"2024-02-19T21:03:12.445891+0000","last_undegraded":"2024-02-19T21:03:12.445891+0000","last_fullsized":"2024-02-19T21:03:12.445891+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-19T21:02:46.558739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.558739+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:56.510563+0000","last_change":"2024-02-19T21:02:56.510539+0000","last_active":"2024-02-19T21:02:56.510563+0000","last_peered":"2024-02-19T21:02:56.510563+0000","last_clean":"2024-02-19T21:02:56.510563+0000","last_became_active":"2024-02-19T21:01:20.290758+0000","last_became_peered":"2024-02-19T21:01:20.290758+0000","last_unstale":"2024-02-19T21:02:56.510563+0000","last_undegraded":"2024-02-19T21:02:56.510563+0000","last_fullsized":"2024-02-19T21:02:56.510563+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.510474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056363+0000","last_change":"2024-02-19T21:02:32.488666+0000","last_active":"2024-02-19T21:02:53.056363+0000","last_peered":"2024-02-19T21:02:53.056363+0000","last_clean":"2024-02-19T21:02:53.056363+0000","last_became_active":"2024-02-19T21:01:22.607883+0000","last_became_peered":"2024-02-19T21:01:22.607883+0000","last_unstale":"2024-02-19T21:02:53.056363+0000","last_undegraded":"2024-02-19T21:02:53.056363+0000","last_fullsized":"2024-02-19T21:02:53.056363+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:32.488584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.488584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":154,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:13.730994+0000","last_change":"2024-02-19T21:02:55.568674+0000","last_active":"2024-02-19T21:03:13.730994+0000","last_peered":"2024-02-19T21:03:13.730994+0000","last_clean":"2024-02-19T21:03:13.730994+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:03:13.730994+0000","last_undegraded":"2024-02-19T21:03:13.730994+0000","last_fullsized":"2024-02-19T21:03:13.730994+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:02:55.568593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.568593+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.966350+0000","last_change":"2024-02-19T21:02:31.357109+0000","last_active":"2024-02-19T21:02:52.966350+0000","last_peered":"2024-02-19T21:02:52.966350+0000","last_clean":"2024-02-19T21:02:52.966350+0000","last_became_active":"2024-02-19T21:01:16.276032+0000","last_became_peered":"2024-02-19T21:01:16.276032+0000","last_unstale":"2024-02-19T21:02:52.966350+0000","last_undegraded":"2024-02-19T21:02:52.966350+0000","last_fullsized":"2024-02-19T21:02:52.966350+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:31.357041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:31.357041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":43,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965503+0000","last_change":"2024-02-19T21:02:22.217326+0000","last_active":"2024-02-19T21:02:52.965503+0000","last_peered":"2024-02-19T21:02:52.965503+0000","last_clean":"2024-02-19T21:02:52.965503+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:02:52.965503+0000","last_undegraded":"2024-02-19T21:02:52.965503+0000","last_fullsized":"2024-02-19T21:02:52.965503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-02-19T21:02:22.217222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:22.217222+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965782+0000","last_change":"2024-02-19T21:02:45.222033+0000","last_active":"2024-02-19T21:02:52.965782+0000","last_peered":"2024-02-19T21:02:52.965782+0000","last_clean":"2024-02-19T21:02:52.965782+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:02:52.965782+0000","last_undegraded":"2024-02-19T21:02:52.965782+0000","last_fullsized":"2024-02-19T21:02:52.965782+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:45.221965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:45.221965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.996331+0000","last_change":"2024-02-19T21:02:25.326504+0000","last_active":"2024-02-19T21:02:52.996331+0000","last_peered":"2024-02-19T21:02:52.996331+0000","last_clean":"2024-02-19T21:02:52.996331+0000","last_became_active":"2024-02-19T21:01:22.292299+0000","last_became_peered":"2024-02-19T21:01:22.292299+0000","last_unstale":"2024-02-19T21:02:52.996331+0000","last_undegraded":"2024-02-19T21:02:52.996331+0000","last_fullsized":"2024-02-19T21:02:52.996331+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:25.326441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:25.326441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.966502+0000","last_change":"2024-02-19T21:01:16.274575+0000","last_active":"2024-02-19T21:02:52.966502+0000","last_peered":"2024-02-19T21:02:52.966502+0000","last_clean":"2024-02-19T21:02:52.966502+0000","last_became_active":"2024-02-19T21:01:16.274290+0000","last_became_peered":"2024-02-19T21:01:16.274290+0000","last_unstale":"2024-02-19T21:02:52.966502+0000","last_undegraded":"2024-02-19T21:02:52.966502+0000","last_fullsized":"2024-02-19T21:02:52.966502+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:01:15.241725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.995920+0000","last_change":"2024-02-19T21:02:52.986989+0000","last_active":"2024-02-19T21:02:52.995920+0000","last_peered":"2024-02-19T21:02:52.995920+0000","last_clean":"2024-02-19T21:02:52.995920+0000","last_became_active":"2024-02-19T21:01:16.276807+0000","last_became_peered":"2024-02-19T21:01:16.276807+0000","last_unstale":"2024-02-19T21:02:52.995920+0000","last_undegraded":"2024-02-19T21:02:52.995920+0000","last_fullsized":"2024-02-19T21:02:52.995920+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.986913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.986913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":47,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.598713+0000","last_change":"2024-02-19T21:03:11.598713+0000","last_active":"2024-02-19T21:03:11.598713+0000","last_peered":"2024-02-19T21:03:11.598713+0000","last_clean":"2024-02-19T21:03:11.598713+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:03:11.598713+0000","last_undegraded":"2024-02-19T21:03:11.598713+0000","last_fullsized":"2024-02-19T21:03:11.598713+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:11.598603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:03:11.598603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":59,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056566+0000","last_change":"2024-02-19T21:02:32.777906+0000","last_active":"2024-02-19T21:02:53.056566+0000","last_peered":"2024-02-19T21:02:53.056566+0000","last_clean":"2024-02-19T21:02:53.056566+0000","last_became_active":"2024-02-19T21:01:18.293784+0000","last_became_peered":"2024-02-19T21:01:18.293784+0000","last_unstale":"2024-02-19T21:02:53.056566+0000","last_undegraded":"2024-02-19T21:02:53.056566+0000","last_fullsized":"2024-02-19T21:02:53.056566+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:02:32.777827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.777827+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:55.586966+0000","last_change":"2024-02-19T21:02:55.586944+0000","last_active":"2024-02-19T21:02:55.586966+0000","last_peered":"2024-02-19T21:02:55.586966+0000","last_clean":"2024-02-19T21:02:55.586966+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:02:55.586966+0000","last_undegraded":"2024-02-19T21:02:55.586966+0000","last_fullsized":"2024-02-19T21:02:55.586966+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:55.586854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.586854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056707+0000","last_change":"2024-02-19T21:02:52.556438+0000","last_active":"2024-02-19T21:02:53.056707+0000","last_peered":"2024-02-19T21:02:53.056707+0000","last_clean":"2024-02-19T21:02:53.056707+0000","last_became_active":"2024-02-19T21:01:22.298477+0000","last_became_peered":"2024-02-19T21:01:22.298477+0000","last_unstale":"2024-02-19T21:02:53.056707+0000","last_undegraded":"2024-02-19T21:02:53.056707+0000","last_fullsized":"2024-02-19T21:02:53.056707+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.556344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.556344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:57.505121+0000","last_change":"2024-02-19T21:02:57.505093+0000","last_active":"2024-02-19T21:02:57.505121+0000","last_peered":"2024-02-19T21:02:57.505121+0000","last_clean":"2024-02-19T21:02:57.505121+0000","last_became_active":"2024-02-19T21:01:16.274398+0000","last_became_peered":"2024-02-19T21:01:16.274398+0000","last_unstale":"2024-02-19T21:02:57.505121+0000","last_undegraded":"2024-02-19T21:02:57.505121+0000","last_fullsized":"2024-02-19T21:02:57.505121+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.505029+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":45,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056052+0000","last_change":"2024-02-19T21:02:41.616831+0000","last_active":"2024-02-19T21:02:53.056052+0000","last_peered":"2024-02-19T21:02:53.056052+0000","last_clean":"2024-02-19T21:02:53.056052+0000","last_became_active":"2024-02-19T21:01:16.275904+0000","last_became_peered":"2024-02-19T21:01:16.275904+0000","last_unstale":"2024-02-19T21:02:53.056052+0000","last_undegraded":"2024-02-19T21:02:53.056052+0000","last_fullsized":"2024-02-19T21:02:53.056052+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:02:41.616729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.616729+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":55,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056381+0000","last_change":"2024-02-19T21:02:18.707988+0000","last_active":"2024-02-19T21:02:53.056381+0000","last_peered":"2024-02-19T21:02:53.056381+0000","last_clean":"2024-02-19T21:02:53.056381+0000","last_became_active":"2024-02-19T21:01:18.287139+0000","last_became_peered":"2024-02-19T21:01:18.287139+0000","last_unstale":"2024-02-19T21:02:53.056381+0000","last_undegraded":"2024-02-19T21:02:53.056381+0000","last_fullsized":"2024-02-19T21:02:53.056381+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:02:18.707921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:18.707921+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:05.005388+0000","last_change":"2024-02-19T21:03:05.005360+0000","last_active":"2024-02-19T21:03:05.005388+0000","last_peered":"2024-02-19T21:03:05.005388+0000","last_clean":"2024-02-19T21:03:05.005388+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:03:05.005388+0000","last_undegraded":"2024-02-19T21:03:05.005388+0000","last_fullsized":"2024-02-19T21:03:05.005388+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.005288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.005288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:07.473181+0000","last_change":"2024-02-19T21:03:07.473156+0000","last_active":"2024-02-19T21:03:07.473181+0000","last_peered":"2024-02-19T21:03:07.473181+0000","last_clean":"2024-02-19T21:03:07.473181+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:03:07.473181+0000","last_undegraded":"2024-02-19T21:03:07.473181+0000","last_fullsized":"2024-02-19T21:03:07.473181+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:03:07.473070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056397+0000","last_change":"2024-02-19T21:02:46.565447+0000","last_active":"2024-02-19T21:02:53.056397+0000","last_peered":"2024-02-19T21:02:53.056397+0000","last_clean":"2024-02-19T21:02:53.056397+0000","last_became_active":"2024-02-19T21:01:16.274215+0000","last_became_peered":"2024-02-19T21:01:16.274215+0000","last_unstale":"2024-02-19T21:02:53.056397+0000","last_undegraded":"2024-02-19T21:02:53.056397+0000","last_fullsized":"2024-02-19T21:02:53.056397+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:46.565380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.565380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":73,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056018+0000","last_change":"2024-02-19T21:02:35.502975+0000","last_active":"2024-02-19T21:02:53.056018+0000","last_peered":"2024-02-19T21:02:53.056018+0000","last_clean":"2024-02-19T21:02:53.056018+0000","last_became_active":"2024-02-19T21:01:18.293791+0000","last_became_peered":"2024-02-19T21:01:18.293791+0000","last_unstale":"2024-02-19T21:02:53.056018+0000","last_undegraded":"2024-02-19T21:02:53.056018+0000","last_fullsized":"2024-02-19T21:02:53.056018+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:02:35.502906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:35.502906+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.055898+0000","last_change":"2024-02-19T21:02:27.761781+0000","last_active":"2024-02-19T21:02:53.055898+0000","last_peered":"2024-02-19T21:02:53.055898+0000","last_clean":"2024-02-19T21:02:53.055898+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:02:53.055898+0000","last_undegraded":"2024-02-19T21:02:53.055898+0000","last_fullsized":"2024-02-19T21:02:53.055898+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:27.761685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:27.761685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965766+0000","last_change":"2024-02-19T21:01:22.298108+0000","last_active":"2024-02-19T21:02:52.965766+0000","last_peered":"2024-02-19T21:02:52.965766+0000","last_clean":"2024-02-19T21:02:52.965766+0000","last_became_active":"2024-02-19T21:01:22.297707+0000","last_became_peered":"2024-02-19T21:01:22.297707+0000","last_unstale":"2024-02-19T21:02:52.965766+0000","last_undegraded":"2024-02-19T21:02:52.965766+0000","last_fullsized":"2024-02-19T21:02:52.965766+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:01:21.273400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056707+0000","last_change":"2024-02-19T21:02:47.601190+0000","last_active":"2024-02-19T21:02:53.056707+0000","last_peered":"2024-02-19T21:02:53.056707+0000","last_clean":"2024-02-19T21:02:53.056707+0000","last_became_active":"2024-02-19T21:01:16.271340+0000","last_became_peered":"2024-02-19T21:01:16.271340+0000","last_unstale":"2024-02-19T21:02:53.056707+0000","last_undegraded":"2024-02-19T21:02:53.056707+0000","last_fullsized":"2024-02-19T21:02:53.056707+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.601127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.601127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":69,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.966383+0000","last_change":"2024-02-19T21:02:39.952691+0000","last_active":"2024-02-19T21:02:52.966383+0000","last_peered":"2024-02-19T21:02:52.966383+0000","last_clean":"2024-02-19T21:02:52.966383+0000","last_became_active":"2024-02-19T21:01:18.293812+0000","last_became_peered":"2024-02-19T21:01:18.293812+0000","last_unstale":"2024-02-19T21:02:52.966383+0000","last_undegraded":"2024-02-19T21:02:52.966383+0000","last_fullsized":"2024-02-19T21:02:52.966383+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:02:39.952624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.952624+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:57.997574+0000","last_change":"2024-02-19T21:02:57.997535+0000","last_active":"2024-02-19T21:02:57.997574+0000","last_peered":"2024-02-19T21:02:57.997574+0000","last_clean":"2024-02-19T21:02:57.997574+0000","last_became_active":"2024-02-19T21:01:20.291079+0000","last_became_peered":"2024-02-19T21:01:20.291079+0000","last_unstale":"2024-02-19T21:02:57.997574+0000","last_undegraded":"2024-02-19T21:02:57.997574+0000","last_fullsized":"2024-02-19T21:02:57.997574+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.997427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:01.496957+0000","last_change":"2024-02-19T21:03:01.496926+0000","last_active":"2024-02-19T21:03:01.496957+0000","last_peered":"2024-02-19T21:03:01.496957+0000","last_clean":"2024-02-19T21:03:01.496957+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:03:01.496957+0000","last_undegraded":"2024-02-19T21:03:01.496957+0000","last_fullsized":"2024-02-19T21:03:01.496957+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.496833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.496833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056951+0000","last_change":"2024-02-19T21:02:50.921031+0000","last_active":"2024-02-19T21:02:53.056951+0000","last_peered":"2024-02-19T21:02:53.056951+0000","last_clean":"2024-02-19T21:02:53.056951+0000","last_became_active":"2024-02-19T21:01:16.274462+0000","last_became_peered":"2024-02-19T21:01:16.274462+0000","last_unstale":"2024-02-19T21:02:53.056951+0000","last_undegraded":"2024-02-19T21:02:53.056951+0000","last_fullsized":"2024-02-19T21:02:53.056951+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:50.920948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:50.920948+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":58,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965538+0000","last_change":"2024-02-19T21:02:40.105227+0000","last_active":"2024-02-19T21:02:52.965538+0000","last_peered":"2024-02-19T21:02:52.965538+0000","last_clean":"2024-02-19T21:02:52.965538+0000","last_became_active":"2024-02-19T21:01:18.282702+0000","last_became_peered":"2024-02-19T21:01:18.282702+0000","last_unstale":"2024-02-19T21:02:52.965538+0000","last_undegraded":"2024-02-19T21:02:52.965538+0000","last_fullsized":"2024-02-19T21:02:52.965538+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:02:40.105141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.105141+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":67,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.395460+0000","last_change":"2024-02-19T21:02:21.923048+0000","last_active":"2024-02-19T21:03:11.395460+0000","last_peered":"2024-02-19T21:03:11.395460+0000","last_clean":"2024-02-19T21:03:11.395460+0000","last_became_active":"2024-02-19T21:01:20.286505+0000","last_became_peered":"2024-02-19T21:01:20.286505+0000","last_unstale":"2024-02-19T21:03:11.395460+0000","last_undegraded":"2024-02-19T21:03:11.395460+0000","last_fullsized":"2024-02-19T21:03:11.395460+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:21.922971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.922971+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.967843+0000","last_change":"2024-02-19T21:01:22.607484+0000","last_active":"2024-02-19T21:02:52.967843+0000","last_peered":"2024-02-19T21:02:52.967843+0000","last_clean":"2024-02-19T21:02:52.967843+0000","last_became_active":"2024-02-19T21:01:22.607007+0000","last_became_peered":"2024-02-19T21:01:22.607007+0000","last_unstale":"2024-02-19T21:02:52.967843+0000","last_undegraded":"2024-02-19T21:02:52.967843+0000","last_fullsized":"2024-02-19T21:02:52.967843+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:01:21.273400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:06.991306+0000","last_change":"2024-02-19T21:03:06.991276+0000","last_active":"2024-02-19T21:03:06.991306+0000","last_peered":"2024-02-19T21:03:06.991306+0000","last_clean":"2024-02-19T21:03:06.991306+0000","last_became_active":"2024-02-19T21:01:20.289375+0000","last_became_peered":"2024-02-19T21:01:20.289375+0000","last_unstale":"2024-02-19T21:03:06.991306+0000","last_undegraded":"2024-02-19T21:03:06.991306+0000","last_fullsized":"2024-02-19T21:03:06.991306+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.991199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.991199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.055896+0000","last_change":"2024-02-19T21:02:27.567995+0000","last_active":"2024-02-19T21:02:53.055896+0000","last_peered":"2024-02-19T21:02:53.055896+0000","last_clean":"2024-02-19T21:02:53.055896+0000","last_became_active":"2024-02-19T21:01:16.275520+0000","last_became_peered":"2024-02-19T21:01:16.275520+0000","last_unstale":"2024-02-19T21:02:53.055896+0000","last_undegraded":"2024-02-19T21:02:53.055896+0000","last_fullsized":"2024-02-19T21:02:53.055896+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:27.567903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:27.567903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":56,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.966160+0000","last_change":"2024-02-19T21:02:19.042725+0000","last_active":"2024-02-19T21:02:52.966160+0000","last_peered":"2024-02-19T21:02:52.966160+0000","last_clean":"2024-02-19T21:02:52.966160+0000","last_became_active":"2024-02-19T21:01:18.285670+0000","last_became_peered":"2024-02-19T21:01:18.285670+0000","last_unstale":"2024-02-19T21:02:52.966160+0000","last_undegraded":"2024-02-19T21:02:52.966160+0000","last_fullsized":"2024-02-19T21:02:52.966160+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:19.042650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:19.042650+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.996259+0000","last_change":"2024-02-19T21:02:41.195082+0000","last_active":"2024-02-19T21:02:52.996259+0000","last_peered":"2024-02-19T21:02:52.996259+0000","last_clean":"2024-02-19T21:02:52.996259+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:02:52.996259+0000","last_undegraded":"2024-02-19T21:02:52.996259+0000","last_fullsized":"2024-02-19T21:02:52.996259+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:41.194993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.194993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":67,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390823+0000","last_change":"2024-02-19T21:02:48.027160+0000","last_active":"2024-02-19T21:03:11.390823+0000","last_peered":"2024-02-19T21:03:11.390823+0000","last_clean":"2024-02-19T21:03:11.390823+0000","last_became_active":"2024-02-19T21:01:20.293341+0000","last_became_peered":"2024-02-19T21:01:20.293341+0000","last_unstale":"2024-02-19T21:03:11.390823+0000","last_undegraded":"2024-02-19T21:03:11.390823+0000","last_fullsized":"2024-02-19T21:03:11.390823+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:48.027070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.027070+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056266+0000","last_change":"2024-02-19T21:02:28.471214+0000","last_active":"2024-02-19T21:02:53.056266+0000","last_peered":"2024-02-19T21:02:53.056266+0000","last_clean":"2024-02-19T21:02:53.056266+0000","last_became_active":"2024-02-19T21:01:16.274939+0000","last_became_peered":"2024-02-19T21:01:16.274939+0000","last_unstale":"2024-02-19T21:02:53.056266+0000","last_undegraded":"2024-02-19T21:02:53.056266+0000","last_fullsized":"2024-02-19T21:02:53.056266+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:28.471131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:28.471131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":62,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.526237+0000","last_change":"2024-02-19T21:02:53.526211+0000","last_active":"2024-02-19T21:02:53.526237+0000","last_peered":"2024-02-19T21:02:53.526237+0000","last_clean":"2024-02-19T21:02:53.526237+0000","last_became_active":"2024-02-19T21:01:18.285534+0000","last_became_peered":"2024-02-19T21:01:18.285534+0000","last_unstale":"2024-02-19T21:02:53.526237+0000","last_undegraded":"2024-02-19T21:02:53.526237+0000","last_fullsized":"2024-02-19T21:02:53.526237+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:53.526141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:53.526141+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965997+0000","last_change":"2024-02-19T21:02:42.123367+0000","last_active":"2024-02-19T21:02:52.965997+0000","last_peered":"2024-02-19T21:02:52.965997+0000","last_clean":"2024-02-19T21:02:52.965997+0000","last_became_active":"2024-02-19T21:01:22.299993+0000","last_became_peered":"2024-02-19T21:01:22.299993+0000","last_unstale":"2024-02-19T21:02:52.965997+0000","last_undegraded":"2024-02-19T21:02:52.965997+0000","last_fullsized":"2024-02-19T21:02:52.965997+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:42.123299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:42.123299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":67,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.432367+0000","last_change":"2024-02-19T21:02:25.255767+0000","last_active":"2024-02-19T21:03:12.432367+0000","last_peered":"2024-02-19T21:03:12.432367+0000","last_clean":"2024-02-19T21:03:12.432367+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:03:12.432367+0000","last_undegraded":"2024-02-19T21:03:12.432367+0000","last_fullsized":"2024-02-19T21:03:12.432367+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:25.255648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:25.255648+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056726+0000","last_change":"2024-02-19T21:02:42.525447+0000","last_active":"2024-02-19T21:02:53.056726+0000","last_peered":"2024-02-19T21:02:53.056726+0000","last_clean":"2024-02-19T21:02:53.056726+0000","last_became_active":"2024-02-19T21:01:16.274086+0000","last_became_peered":"2024-02-19T21:01:16.274086+0000","last_unstale":"2024-02-19T21:02:53.056726+0000","last_undegraded":"2024-02-19T21:02:53.056726+0000","last_fullsized":"2024-02-19T21:02:53.056726+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:42.525345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:42.525345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"82'15","reported_seq":57,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:04.086425+0000","last_change":"2024-02-19T21:03:04.086368+0000","last_active":"2024-02-19T21:03:04.086425+0000","last_peered":"2024-02-19T21:03:04.086425+0000","last_clean":"2024-02-19T21:03:04.086425+0000","last_became_active":"2024-02-19T21:01:18.282805+0000","last_became_peered":"2024-02-19T21:01:18.282805+0000","last_unstale":"2024-02-19T21:03:04.086425+0000","last_undegraded":"2024-02-19T21:03:04.086425+0000","last_fullsized":"2024-02-19T21:03:04.086425+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.086282+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:54.197807+0000","last_change":"2024-02-19T21:02:54.197783+0000","last_active":"2024-02-19T21:02:54.197807+0000","last_peered":"2024-02-19T21:02:54.197807+0000","last_clean":"2024-02-19T21:02:54.197807+0000","last_became_active":"2024-02-19T21:01:22.298941+0000","last_became_peered":"2024-02-19T21:01:22.298941+0000","last_unstale":"2024-02-19T21:02:54.197807+0000","last_undegraded":"2024-02-19T21:02:54.197807+0000","last_fullsized":"2024-02-19T21:02:54.197807+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:54.197714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:54.197714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056794+0000","last_change":"2024-02-19T21:02:21.497900+0000","last_active":"2024-02-19T21:02:53.056794+0000","last_peered":"2024-02-19T21:02:53.056794+0000","last_clean":"2024-02-19T21:02:53.056794+0000","last_became_active":"2024-02-19T21:01:20.290687+0000","last_became_peered":"2024-02-19T21:01:20.290687+0000","last_unstale":"2024-02-19T21:02:53.056794+0000","last_undegraded":"2024-02-19T21:02:53.056794+0000","last_fullsized":"2024-02-19T21:02:53.056794+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.497831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.497831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":44,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056239+0000","last_change":"2024-02-19T21:02:40.628725+0000","last_active":"2024-02-19T21:02:53.056239+0000","last_peered":"2024-02-19T21:02:53.056239+0000","last_clean":"2024-02-19T21:02:53.056239+0000","last_became_active":"2024-02-19T21:01:16.274937+0000","last_became_peered":"2024-02-19T21:01:16.274937+0000","last_unstale":"2024-02-19T21:02:53.056239+0000","last_undegraded":"2024-02-19T21:02:53.056239+0000","last_fullsized":"2024-02-19T21:02:53.056239+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:02:40.628632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.628632+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":92,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.966136+0000","last_change":"2024-02-19T21:02:30.070342+0000","last_active":"2024-02-19T21:02:52.966136+0000","last_peered":"2024-02-19T21:02:52.966136+0000","last_clean":"2024-02-19T21:02:52.966136+0000","last_became_active":"2024-02-19T21:01:30.923214+0000","last_became_peered":"2024-02-19T21:01:30.923214+0000","last_unstale":"2024-02-19T21:02:52.966136+0000","last_undegraded":"2024-02-19T21:02:52.966136+0000","last_fullsized":"2024-02-19T21:02:52.966136+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:02:30.070236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:30.070236+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":32,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:03.500096+0000","last_change":"2024-02-19T21:03:03.500049+0000","last_active":"2024-02-19T21:03:03.500096+0000","last_peered":"2024-02-19T21:03:03.500096+0000","last_clean":"2024-02-19T21:03:03.500096+0000","last_became_active":"2024-02-19T21:01:22.291759+0000","last_became_peered":"2024-02-19T21:01:22.291759+0000","last_unstale":"2024-02-19T21:03:03.500096+0000","last_undegraded":"2024-02-19T21:03:03.500096+0000","last_fullsized":"2024-02-19T21:03:03.500096+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:03.499960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:03.499960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:55.213784+0000","last_change":"2024-02-19T21:02:55.213761+0000","last_active":"2024-02-19T21:02:55.213784+0000","last_peered":"2024-02-19T21:02:55.213784+0000","last_clean":"2024-02-19T21:02:55.213784+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:02:55.213784+0000","last_undegraded":"2024-02-19T21:02:55.213784+0000","last_fullsized":"2024-02-19T21:02:55.213784+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:55.213692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.213692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965762+0000","last_change":"2024-02-19T21:02:21.212907+0000","last_active":"2024-02-19T21:02:52.965762+0000","last_peered":"2024-02-19T21:02:52.965762+0000","last_clean":"2024-02-19T21:02:52.965762+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:02:52.965762+0000","last_undegraded":"2024-02-19T21:02:52.965762+0000","last_fullsized":"2024-02-19T21:02:52.965762+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.212808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.212808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":68,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.055761+0000","last_change":"2024-02-19T21:02:39.779974+0000","last_active":"2024-02-19T21:02:53.055761+0000","last_peered":"2024-02-19T21:02:53.055761+0000","last_clean":"2024-02-19T21:02:53.055761+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:02:53.055761+0000","last_undegraded":"2024-02-19T21:02:53.055761+0000","last_fullsized":"2024-02-19T21:02:53.055761+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:02:39.779890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.779890+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.055835+0000","last_change":"2024-02-19T21:02:48.572976+0000","last_active":"2024-02-19T21:02:53.055835+0000","last_peered":"2024-02-19T21:02:53.055835+0000","last_clean":"2024-02-19T21:02:53.055835+0000","last_became_active":"2024-02-19T21:01:22.297998+0000","last_became_peered":"2024-02-19T21:01:22.297998+0000","last_unstale":"2024-02-19T21:02:53.055835+0000","last_undegraded":"2024-02-19T21:02:53.055835+0000","last_fullsized":"2024-02-19T21:02:53.055835+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:48.572877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.572877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":35,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:01.169378+0000","last_change":"2024-02-19T21:03:01.169350+0000","last_active":"2024-02-19T21:03:01.169378+0000","last_peered":"2024-02-19T21:03:01.169378+0000","last_clean":"2024-02-19T21:03:01.169378+0000","last_became_active":"2024-02-19T21:01:20.290566+0000","last_became_peered":"2024-02-19T21:01:20.290566+0000","last_unstale":"2024-02-19T21:03:01.169378+0000","last_undegraded":"2024-02-19T21:03:01.169378+0000","last_fullsized":"2024-02-19T21:03:01.169378+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.169279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.169279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056445+0000","last_change":"2024-02-19T21:02:22.516204+0000","last_active":"2024-02-19T21:02:53.056445+0000","last_peered":"2024-02-19T21:02:53.056445+0000","last_clean":"2024-02-19T21:02:53.056445+0000","last_became_active":"2024-02-19T21:01:16.270931+0000","last_became_peered":"2024-02-19T21:01:16.270931+0000","last_unstale":"2024-02-19T21:02:53.056445+0000","last_undegraded":"2024-02-19T21:02:53.056445+0000","last_fullsized":"2024-02-19T21:02:53.056445+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:22.516135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:22.516135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":53,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:53.056288+0000","last_change":"2024-02-19T21:02:52.643899+0000","last_active":"2024-02-19T21:02:53.056288+0000","last_peered":"2024-02-19T21:02:53.056288+0000","last_clean":"2024-02-19T21:02:53.056288+0000","last_became_active":"2024-02-19T21:01:18.288322+0000","last_became_peered":"2024-02-19T21:01:18.288322+0000","last_unstale":"2024-02-19T21:02:53.056288+0000","last_undegraded":"2024-02-19T21:02:53.056288+0000","last_fullsized":"2024-02-19T21:02:53.056288+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:52.643835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.643835+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:10.017872+0000","last_change":"2024-02-19T21:03:10.017843+0000","last_active":"2024-02-19T21:03:10.017872+0000","last_peered":"2024-02-19T21:03:10.017872+0000","last_clean":"2024-02-19T21:03:10.017872+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:03:10.017872+0000","last_undegraded":"2024-02-19T21:03:10.017872+0000","last_fullsized":"2024-02-19T21:03:10.017872+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:10.017764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:10.017764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":39,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:02:52.965863+0000","last_change":"2024-02-19T21:02:37.258648+0000","last_active":"2024-02-19T21:02:52.965863+0000","last_peered":"2024-02-19T21:02:52.965863+0000","last_clean":"2024-02-19T21:02:52.965863+0000","last_became_active":"2024-02-19T21:01:16.271911+0000","last_became_peered":"2024-02-19T21:01:16.271911+0000","last_unstale":"2024-02-19T21:02:52.965863+0000","last_undegraded":"2024-02-19T21:02:52.965863+0000","last_fullsized":"2024-02-19T21:02:52.965863+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:37.258580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:37.258580+0000","last_clean_scrub_stamp":"2024-02-19T21:02:37.258580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":62,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-02-19T21:03:01.005562+0000","last_change":"2024-02-19T21:03:01.005530+0000","last_active":"2024-02-19T21:03:01.005562+0000","last_peered":"2024-02-19T21:03:01.005562+0000","last_clean":"2024-02-19T21:03:01.005562+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:03:01.005562+0000","last_undegraded":"2024-02-19T21:03:01.005562+0000","last_fullsized":"2024-02-19T21:03:01.005562+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:03:01.005445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.005445+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":66,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596438+0000","last_change":"2024-02-19T21:02:44.563371+0000","last_active":"2024-02-19T21:03:11.596438+0000","last_peered":"2024-02-19T21:03:11.596438+0000","last_clean":"2024-02-19T21:03:11.596438+0000","last_became_active":"2024-02-19T21:01:20.288720+0000","last_became_peered":"2024-02-19T21:01:20.288720+0000","last_unstale":"2024-02-19T21:03:11.596438+0000","last_undegraded":"2024-02-19T21:03:11.596438+0000","last_fullsized":"2024-02-19T21:03:11.596438+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:44.563256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:44.563256+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":399,"ondisk_log_size":399,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266714,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297940,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447212,"statfs":{"total":95995035648,"available":95689945088,"internally_reserved":0,"allocated":618496,"data_stored":280705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.502,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.57499999999999996,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63500000000000001,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.504}]},{"osd":1,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.51300000000000001,"15min":0.52000000000000002},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78200000000000003,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51300000000000001,"15min":0.51900000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.61499999999999999,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48699999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.499,"15min":0.5},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.433,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70599999999999996,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.61099999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.39600000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.67800000000000005,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.505,"5min":0.496,"15min":0.49399999999999999},"min":{"1min":0.38700000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.59999999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.48599999999999999,"15min":0.498},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58199999999999996,"5min":1.373,"15min":1.373},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.629}]},{"osd":5,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.46000000000000002,"15min":0.46100000000000002},"min":{"1min":0.31900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66800000000000004,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.63900000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.34699999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.64700000000000002,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.56699999999999995,"15min":0.57599999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76000000000000001,"5min":1.262,"15min":1.262},"last":0.67600000000000005}]}]},{"osd":6,"up_from":45,"seq":193273528352,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297912,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447240,"statfs":{"total":95995035648,"available":95689973760,"internally_reserved":0,"allocated":589824,"data_stored":279169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.48799999999999999,"15min":0.49199999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.47699999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.49399999999999999,"15min":0.498},"min":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.58099999999999996}]},{"osd":2,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51300000000000001,"15min":0.51700000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.49299999999999999,"15min":0.49099999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.54400000000000004}]},{"osd":3,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.503,"15min":0.497},"min":{"1min":0.38900000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.503}]},{"osd":4,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.39300000000000002,"15min":0.375},"min":{"1min":0.29299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.41299999999999998,"15min":0.40300000000000002},"min":{"1min":0.28899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.34399999999999997}]},{"osd":5,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.42999999999999999,"15min":0.41199999999999998},"min":{"1min":0.34000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.46300000000000002,"15min":0.44700000000000001},"min":{"1min":0.39900000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.41199999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:02:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.437},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.46200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297948,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447204,"statfs":{"total":95995035648,"available":95689936896,"internally_reserved":0,"allocated":626688,"data_stored":280796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35899999999999999,"15min":0.34799999999999998},"min":{"1min":0.22500000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.38,"15min":0.35999999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:02:12 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.56299999999999994,"5min":0.62,"15min":0.62},"last":0.314},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.41399999999999998,"15min":0.41799999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.27300000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:02:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.41399999999999998,"15min":0.42399999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.51800000000000002,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.45500000000000002,"15min":0.46300000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58799999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.40200000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68300000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71699999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.73699999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57799999999999996,"15min":0.58399999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72799999999999998,"5min":0.872,"15min":0.872},"last":0.68700000000000006}]},{"osd":6,"last update":"Mon Feb 19 21:02:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.57699999999999996,"15min":0.58599999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71299999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63,"15min":0.64300000000000002},"min":{"1min":0.44600000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.68100000000000005,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Feb 19 21:02:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.58499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738429,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297940,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447212,"statfs":{"total":95995035648,"available":95689945088,"internally_reserved":0,"allocated":618496,"data_stored":280359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.372,"15min":0.315},"min":{"1min":0.21299999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.46899999999999997,"5min":4.2830000000000004,"15min":4.2830000000000004},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.33300000000000002,"15min":0.308},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.48299999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.32600000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:02:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.40600000000000003,"15min":0.38100000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":3.121,"5min":3.121,"15min":3.121},"last":0.315},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.42799999999999999,"15min":0.39000000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":4.2569999999999997,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.38}]},{"osd":3,"last update":"Mon Feb 19 21:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.47099999999999997,"15min":0.45100000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.503,"5min":4.4470000000000001,"15min":4.4470000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.373,"5min":0.48399999999999999,"15min":0.46100000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.504,"5min":4.3070000000000004,"15min":4.3070000000000004},"last":0.35799999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52600000000000002,"15min":0.51800000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55400000000000005,"15min":0.54000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73999999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.48199999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.628,"15min":0.64100000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88300000000000001,"5min":1.581,"15min":1.581},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.78800000000000003,"15min":0.82799999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76900000000000002,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.48799999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.1499999999999999,"5min":1.381,"15min":1.381},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.79600000000000004,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.51800000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.76900000000000002,"15min":0.79900000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2070000000000001,"5min":4.226,"15min":4.226},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.625,"15min":0.63100000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.218,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.55000000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":606208,"data_stored":279947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.36299999999999999,"15min":0.35399999999999998},"min":{"1min":0.245,"5min":0.222,"15min":0.222},"max":{"1min":0.47399999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.35699999999999998,"15min":0.34699999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.59399999999999997,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.311}]},{"osd":1,"last update":"Mon Feb 19 21:02:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.36199999999999999,"15min":0.35199999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.54800000000000004,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.377,"5min":0.40200000000000002,"15min":0.39500000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56100000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.95099999999999996}]},{"osd":3,"last update":"Mon Feb 19 21:02:40 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.48499999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.46200000000000002,"15min":0.47399999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.56699999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":1.0309999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.58099999999999996,"15min":0.59999999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71899999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":1.47},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.60899999999999999,"15min":0.64300000000000002},"min":{"1min":0.38400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.74199999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.79700000000000004}]},{"osd":5,"last update":"Mon Feb 19 21:02:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.63500000000000001,"15min":0.65500000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70899999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80300000000000005,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.97599999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:02:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.62,"15min":0.629},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77800000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":1.5069999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.622,"15min":0.63200000000000001},"min":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.76300000000000001,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.90000000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:02:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":1.4410000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182446,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297952,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447200,"statfs":{"total":95995035648,"available":95689932800,"internally_reserved":0,"allocated":630784,"data_stored":279972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.35199999999999998,"15min":0.35399999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.39300000000000002,"15min":0.40300000000000002},"min":{"1min":0.27500000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.51600000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.313}]},{"osd":1,"last update":"Mon Feb 19 21:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.38600000000000001,"15min":0.39600000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.39500000000000002,"15min":0.39900000000000002},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59199999999999997,"5min":1.054,"15min":1.054},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41499999999999998,"15min":0.42199999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56399999999999995,"5min":0.625,"15min":0.625},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.40999999999999998,"15min":0.41899999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.52900000000000003,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.32400000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.41199999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82799999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55700000000000005,"15min":0.56599999999999995},"min":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.83399999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.46100000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.57199999999999995,"15min":0.57899999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79900000000000004,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55800000000000005,"15min":0.56200000000000006},"min":{"1min":0.38200000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72399999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.54600000000000004}]},{"osd":6,"last update":"Mon Feb 19 21:02:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56699999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85199999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.59899999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.39900000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56799999999999995,"15min":0.57099999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.57999999999999996}]}]},{"osd":4,"up_from":31,"seq":133143986218,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297700,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447452,"statfs":{"total":95995035648,"available":95690190848,"internally_reserved":0,"allocated":618496,"data_stored":279985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.52500000000000002,"15min":0.54800000000000004},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.67800000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.435,"5min":0.50700000000000001,"15min":0.52300000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.59899999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.67800000000000005}]},{"osd":1,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.754,"5min":2.149,"15min":2.149},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.502,"15min":0.51900000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57799999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.623}]},{"osd":2,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.53300000000000003,"15min":0.55700000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66400000000000003,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.70699999999999996}]},{"osd":3,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.51700000000000002,"15min":0.52500000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.56499999999999995,"15min":0.56999999999999995},"min":{"1min":0.33400000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64200000000000002,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":0.64800000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.41799999999999998,"15min":0.42799999999999999},"min":{"1min":0.26800000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.501,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.29899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.53600000000000003,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.36099999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.377,"15min":0.36899999999999999},"min":{"1min":0.28699999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.42899999999999999,"15min":0.42199999999999999},"min":{"1min":0.36199999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.52400000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.442,"15min":0.432},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.46800000000000003,"15min":0.45900000000000002},"min":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.46899999999999997}]}]},{"osd":5,"up_from":37,"seq":158913789989,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":593920,"data_stored":278440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.66500000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.53400000000000003,"15min":0.54300000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.70199999999999996,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.66900000000000004}]},{"osd":1,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52000000000000002,"15min":0.52500000000000002},"min":{"1min":0.38,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.52400000000000002,"15min":0.53200000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.58799999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.68899999999999995}]},{"osd":2,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.53200000000000003,"15min":0.54000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.63200000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.53000000000000003,"15min":0.53500000000000003},"min":{"1min":0.40799999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.65400000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.53000000000000003}]},{"osd":3,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52700000000000002,"15min":0.53500000000000003},"min":{"1min":0.36099999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.57899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.40600000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71199999999999997,"5min":0.747,"15min":0.747},"last":0.61799999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.39200000000000002,"15min":0.38600000000000001},"min":{"1min":0.30099999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.56299999999999994,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.39400000000000002,"15min":0.38400000000000001},"min":{"1min":0.30099999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.63}]},{"osd":6,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42499999999999999,"15min":0.41699999999999998},"min":{"1min":0.311,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.441,"15min":0.432},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.624,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.57899999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:02:14 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.60499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:03:18.137 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-19T21:03:18.137 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-19T21:03:18.137 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-19T21:03:18.137 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph health --format=json 2024-02-19T21:03:18.406 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:16.933856+0000 mgr.y (mgr.14550) 83 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:18.411 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:18.665 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:16.933856+0000 mgr.y (mgr.14550) 83 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:18.665 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:17.351350+0000 mgr.y (mgr.14550) 84 : audit [DBG] from='client.34670 v1:172.21.15.163:0/1379314935' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:03:18.665 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:17.624500+0000 osd.3 (osd.3) 39 : cluster [DBG] 6.b deep-scrub starts 2024-02-19T21:03:18.666 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:17.625705+0000 osd.3 (osd.3) 40 : cluster [DBG] 6.b deep-scrub ok 2024-02-19T21:03:18.666 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:17.351350+0000 mgr.y (mgr.14550) 84 : audit [DBG] from='client.34670 v1:172.21.15.163:0/1379314935' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:03:18.666 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:17.624500+0000 osd.3 (osd.3) 39 : cluster [DBG] 6.b deep-scrub starts 2024-02-19T21:03:18.666 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:17.625705+0000 osd.3 (osd.3) 40 : cluster [DBG] 6.b deep-scrub ok 2024-02-19T21:03:18.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:16.933856+0000 mgr.y (mgr.14550) 83 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:18.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:17.351350+0000 mgr.y (mgr.14550) 84 : audit [DBG] from='client.34670 v1:172.21.15.163:0/1379314935' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:03:18.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:17.624500+0000 osd.3 (osd.3) 39 : cluster [DBG] 6.b deep-scrub starts 2024-02-19T21:03:18.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:17.625705+0000 osd.3 (osd.3) 40 : cluster [DBG] 6.b deep-scrub ok 2024-02-19T21:03:20.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.102+0000 7fcd35ca1700 1 -- 172.21.15.163:0/3084969488 >> v1:172.21.15.163:6790/0 conn(0x7fcd300ff670 legacy=0x7fcd30101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:20.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.102+0000 7fcd35ca1700 1 -- 172.21.15.163:0/3084969488 shutdown_connections 2024-02-19T21:03:20.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.102+0000 7fcd35ca1700 1 -- 172.21.15.163:0/3084969488 >> 172.21.15.163:0/3084969488 conn(0x7fcd300f2470 msgr2=0x7fcd300f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:20.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.102+0000 7fcd35ca1700 1 -- 172.21.15.163:0/3084969488 shutdown_connections 2024-02-19T21:03:20.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.102+0000 7fcd35ca1700 1 -- 172.21.15.163:0/3084969488 wait complete. 2024-02-19T21:03:20.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.103+0000 7fcd35ca1700 1 librados: starting msgr at 2024-02-19T21:03:20.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.103+0000 7fcd35ca1700 1 librados: starting objecter 2024-02-19T21:03:20.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.103+0000 7fcd35ca1700 1 Processor -- start 2024-02-19T21:03:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.103+0000 7fcd35ca1700 1 -- start start 2024-02-19T21:03:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.103+0000 7fcd35ca1700 1 librados: setting wanted keys 2024-02-19T21:03:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.103+0000 7fcd35ca1700 1 librados: calling monclient init 2024-02-19T21:03:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd35ca1700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcd301172c0 con 0x7fcd300ff670 2024-02-19T21:03:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd35ca1700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcd301184a0 con 0x7fcd300fc870 2024-02-19T21:03:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd35ca1700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcd301196a0 con 0x7fcd301031b0 2024-02-19T21:03:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2effd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fcd300ff670 0x7fcd300613a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:34544/0 (socket says 172.21.15.163:34544) 2024-02-19T21:03:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2effd700 1 -- 172.21.15.163:0/3453168434 learned_addr learned my addr 172.21.15.163:0/3453168434 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 10 client.?.objecter ms_handle_connect 0x7fcd300ff670 2024-02-19T21:03:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 10 client.?.objecter ms_handle_connect 0x7fcd301031b0 2024-02-19T21:03:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 10 client.?.objecter ms_handle_connect 0x7fcd300fc870 2024-02-19T21:03:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:20.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430062528 0 0) 0x7fcd301172c0 con 0x7fcd300ff670 2024-02-19T21:03:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd18003030 con 0x7fcd300ff670 2024-02-19T21:03:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37815580 0 0) 0x7fcd301196a0 con 0x7fcd301031b0 2024-02-19T21:03:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd301172c0 con 0x7fcd301031b0 2024-02-19T21:03:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391347061 0 0) 0x7fcd301184a0 con 0x7fcd300fc870 2024-02-19T21:03:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd301196a0 con 0x7fcd300fc870 2024-02-19T21:03:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1378765359 0 0) 0x7fcd18003030 con 0x7fcd300ff670 2024-02-19T21:03:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd301184a0 con 0x7fcd300ff670 2024-02-19T21:03:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcd240015d0 con 0x7fcd300ff670 2024-02-19T21:03:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1612219236 0 0) 0x7fcd301172c0 con 0x7fcd301031b0 2024-02-19T21:03:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd18003030 con 0x7fcd301031b0 2024-02-19T21:03:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcd28001b00 con 0x7fcd301031b0 2024-02-19T21:03:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1003794459 0 0) 0x7fcd301196a0 con 0x7fcd300fc870 2024-02-19T21:03:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd301172c0 con 0x7fcd300fc870 2024-02-19T21:03:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcd200018a0 con 0x7fcd300fc870 2024-02-19T21:03:20.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 75944653 0 0) 0x7fcd301184a0 con 0x7fcd300ff670 2024-02-19T21:03:20.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 >> v1:172.21.15.163:6790/0 conn(0x7fcd301031b0 legacy=0x7fcd30115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:20.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 >> v1:172.21.15.193:6789/0 conn(0x7fcd300fc870 legacy=0x7fcd30060c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:20.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.104+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3011a8a0 con 0x7fcd300ff670 2024-02-19T21:03:20.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.105+0000 7fcd35ca1700 1 -- 172.21.15.163:0/3453168434 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd30119890 con 0x7fcd300ff670 2024-02-19T21:03:20.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.105+0000 7fcd35ca1700 10 client.14754.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:20.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.105+0000 7fcd35ca1700 1 -- 172.21.15.163:0/3453168434 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd30119d80 con 0x7fcd300ff670 2024-02-19T21:03:20.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.105+0000 7fcd35ca1700 1 librados: init done 2024-02-19T21:03:20.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.105+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fcd24002400 con 0x7fcd300ff670 2024-02-19T21:03:20.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.105+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcd24003810 con 0x7fcd300ff670 2024-02-19T21:03:20.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.106+0000 7fcd35ca1700 1 -- 172.21.15.163:0/3453168434 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1c004f20 con 0x7fcd300ff670 2024-02-19T21:03:20.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.106+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fcd24001840 con 0x7fcd300ff670 2024-02-19T21:03:20.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.106+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(83..83 src has 32..83) v4 ==== 6172+0+0 (unknown 4093497947 0 0) 0x7fcd24078860 con 0x7fcd300ff670 2024-02-19T21:03:20.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.106+0000 7fcd2cff9700 10 client.14754.objecter ms_dispatch 0x7fcd30001550 osd_map(83..83 src has 32..83) v4 2024-02-19T21:03:20.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.106+0000 7fcd2cff9700 3 client.14754.objecter handle_osd_map got epochs [83,83] > 0 2024-02-19T21:03:20.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.106+0000 7fcd2cff9700 3 client.14754.objecter handle_osd_map decoding full epoch 83 2024-02-19T21:03:20.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.106+0000 7fcd2cff9700 20 client.14754.objecter dump_active .. 0 homeless 2024-02-19T21:03:20.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.106+0000 7fcd2cff9700 10 client.14754.objecter ms_handle_connect 0x7fcd18063c50 2024-02-19T21:03:20.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.110+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fcd2404c470 con 0x7fcd300ff670 2024-02-19T21:03:20.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.313+0000 7fcd35ca1700 1 -- 172.21.15.163:0/3453168434 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcd1c005750 con 0x7fcd300ff670 2024-02-19T21:03:20.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.313+0000 7fcd2cff9700 1 -- 172.21.15.163:0/3453168434 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (unknown 3730786198 0 3397129104) 0x7fcd2404a8e0 con 0x7fcd300ff670 2024-02-19T21:03:20.304 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:03:20.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.315+0000 7fcd35ca1700 10 librados: watch_flush enter 2024-02-19T21:03:20.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.315+0000 7fcd35ca1700 10 librados: watch_flush exit 2024-02-19T21:03:20.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.315+0000 7fcd35ca1700 20 client.14754.objecter shutdown clearing up homeless session... 2024-02-19T21:03:20.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.315+0000 7fcd35ca1700 10 client.14754.objecter successfully canceled tick 2024-02-19T21:03:20.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.315+0000 7fcd35ca1700 1 -- 172.21.15.163:0/3453168434 >> v1:172.21.15.163:6800/1552514681 conn(0x7fcd18063c50 legacy=0x7fcd18066110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:20.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.315+0000 7fcd35ca1700 1 -- 172.21.15.163:0/3453168434 >> v1:172.21.15.163:6789/0 conn(0x7fcd300ff670 legacy=0x7fcd300613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:20.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.315+0000 7fcd35ca1700 1 -- 172.21.15.163:0/3453168434 shutdown_connections 2024-02-19T21:03:20.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.315+0000 7fcd35ca1700 1 -- 172.21.15.163:0/3453168434 >> 172.21.15.163:0/3453168434 conn(0x7fcd300f2470 msgr2=0x7fcd30106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:20.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.315+0000 7fcd35ca1700 1 -- 172.21.15.163:0/3453168434 shutdown_connections 2024-02-19T21:03:20.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.315+0000 7fcd35ca1700 1 -- 172.21.15.163:0/3453168434 wait complete. 2024-02-19T21:03:20.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:20.315+0000 7fcd35ca1700 1 librados: shutdown 2024-02-19T21:03:20.312 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:03:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:03:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:18.934323+0000 mgr.y (mgr.14550) 85 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:19.664721+0000 osd.3 (osd.3) 41 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:03:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:19.665642+0000 osd.3 (osd.3) 42 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:03:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:20.091870+0000 osd.1 (osd.1) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:03:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:20.091923+0000 osd.1 (osd.1) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:03:20.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:20.313794+0000 mon.a (mon.0) 709 : audit [DBG] from='client.? v1:172.21.15.163:0/3453168434' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-19T21:03:20.654 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:18.934323+0000 mgr.y (mgr.14550) 85 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:20.655 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:19.664721+0000 osd.3 (osd.3) 41 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:03:20.655 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:19.665642+0000 osd.3 (osd.3) 42 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:03:20.655 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:20.091870+0000 osd.1 (osd.1) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:03:20.655 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:20.091923+0000 osd.1 (osd.1) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:03:20.655 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:20.313794+0000 mon.a (mon.0) 709 : audit [DBG] from='client.? v1:172.21.15.163:0/3453168434' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-19T21:03:20.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:18.934323+0000 mgr.y (mgr.14550) 85 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:20.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:19.664721+0000 osd.3 (osd.3) 41 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:03:20.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:19.665642+0000 osd.3 (osd.3) 42 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:03:20.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:20.091870+0000 osd.1 (osd.1) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:03:20.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:20.091923+0000 osd.1 (osd.1) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:03:20.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:20.313794+0000 mon.a (mon.0) 709 : audit [DBG] from='client.? v1:172.21.15.163:0/3453168434' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-19T21:03:20.972 INFO:teuthology.orchestra.run.smithi163.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-19T21:03:20.972 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-19T21:03:20.973 INFO:tasks.cephadm:Setup complete, yielding 2024-02-19T21:03:20.973 INFO:teuthology.run_tasks:Running task thrashosds... 2024-02-19T21:03:20.984 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-19T21:03:20.984 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'pacific-release'}, 'ceph': {'conf': {'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(MON_DOWN\\)', '\\(OSDMAP_FLAGS\\)', 'flag\\(s\\) set', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(PG_', '\\(OSD_', {'mons down': None}, 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'mon_bind_msgr2': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'pacific-release', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'}} 2024-02-19T21:03:20.984 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-19T21:03:20.985 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-02-19T21:03:21.252 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:21.698 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:20.007105+0000 mgr.y (mgr.14550) 86 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:21.699 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:20.424549+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-02-19T21:03:21.699 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:20.444534+0000 osd.1 (osd.1) 45 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:03:21.699 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:20.445472+0000 osd.1 (osd.1) 46 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:03:21.699 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:20.636523+0000 osd.3 (osd.3) 43 : cluster [DBG] 2.1b scrub starts 2024-02-19T21:03:21.699 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:20.637460+0000 osd.3 (osd.3) 44 : cluster [DBG] 2.1b scrub ok 2024-02-19T21:03:21.699 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:21.161813+0000 osd.4 (osd.4) 45 : cluster [DBG] 2.f deep-scrub starts 2024-02-19T21:03:21.699 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:21.164279+0000 osd.4 (osd.4) 46 : cluster [DBG] 2.f deep-scrub ok 2024-02-19T21:03:21.699 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:20.007105+0000 mgr.y (mgr.14550) 86 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:21.699 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:20.424549+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-02-19T21:03:21.699 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:20.444534+0000 osd.1 (osd.1) 45 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:03:21.699 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:20.445472+0000 osd.1 (osd.1) 46 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:03:21.699 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:20.636523+0000 osd.3 (osd.3) 43 : cluster [DBG] 2.1b scrub starts 2024-02-19T21:03:21.699 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:20.637460+0000 osd.3 (osd.3) 44 : cluster [DBG] 2.1b scrub ok 2024-02-19T21:03:21.700 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:21.161813+0000 osd.4 (osd.4) 45 : cluster [DBG] 2.f deep-scrub starts 2024-02-19T21:03:21.700 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:21.164279+0000 osd.4 (osd.4) 46 : cluster [DBG] 2.f deep-scrub ok 2024-02-19T21:03:21.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:20.007105+0000 mgr.y (mgr.14550) 86 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:21.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:20.424549+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-02-19T21:03:21.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:20.444534+0000 osd.1 (osd.1) 45 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:03:21.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:20.445472+0000 osd.1 (osd.1) 46 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:03:21.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:20.636523+0000 osd.3 (osd.3) 43 : cluster [DBG] 2.1b scrub starts 2024-02-19T21:03:21.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:20.637460+0000 osd.3 (osd.3) 44 : cluster [DBG] 2.1b scrub ok 2024-02-19T21:03:21.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:21.161813+0000 osd.4 (osd.4) 45 : cluster [DBG] 2.f deep-scrub starts 2024-02-19T21:03:21.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:21.164279+0000 osd.4 (osd.4) 46 : cluster [DBG] 2.f deep-scrub ok 2024-02-19T21:03:22.685 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:20.934726+0000 mgr.y (mgr.14550) 87 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:03:22.685 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:21.618415+0000 osd.3 (osd.3) 45 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:03:22.685 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:21.619384+0000 osd.3 (osd.3) 46 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:03:22.685 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:20.934726+0000 mgr.y (mgr.14550) 87 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:03:22.685 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:21.618415+0000 osd.3 (osd.3) 45 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:03:22.686 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:21.619384+0000 osd.3 (osd.3) 46 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:03:22.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:20.934726+0000 mgr.y (mgr.14550) 87 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:03:22.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:21.618415+0000 osd.3 (osd.3) 45 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:03:22.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:21.619384+0000 osd.3 (osd.3) 46 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:03:22.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.954+0000 7f975d757700 1 -- 172.21.15.163:0/1279690454 >> v1:172.21.15.163:6789/0 conn(0x7f97581007c0 legacy=0x7f9758102bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:22.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.954+0000 7f975d757700 1 -- 172.21.15.163:0/1279690454 shutdown_connections 2024-02-19T21:03:22.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.954+0000 7f975d757700 1 -- 172.21.15.163:0/1279690454 >> 172.21.15.163:0/1279690454 conn(0x7f975806cc00 msgr2=0x7f97580f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:22.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.954+0000 7f975d757700 1 -- 172.21.15.163:0/1279690454 shutdown_connections 2024-02-19T21:03:22.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.954+0000 7f975d757700 1 -- 172.21.15.163:0/1279690454 wait complete. 2024-02-19T21:03:22.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.954+0000 7f975d757700 1 librados: starting msgr at 2024-02-19T21:03:22.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.954+0000 7f975d757700 1 librados: starting objecter 2024-02-19T21:03:22.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.954+0000 7f975d757700 1 Processor -- start 2024-02-19T21:03:22.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f975d757700 1 -- start start 2024-02-19T21:03:22.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f975d757700 1 librados: setting wanted keys 2024-02-19T21:03:22.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f975d757700 1 librados: calling monclient init 2024-02-19T21:03:22.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f975d757700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f97580fd040 con 0x7f9758104330 2024-02-19T21:03:22.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f975d757700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f97580fd210 con 0x7f97581007c0 2024-02-19T21:03:22.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f975d757700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f97580fd3e0 con 0x7f97580fd9d0 2024-02-19T21:03:22.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f9756ffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f97580fd9d0 0x7f97580fc8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45234/0 (socket says 172.21.15.163:45234) 2024-02-19T21:03:22.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f9756ffd700 1 -- 172.21.15.163:0/2108204378 learned_addr learned my addr 172.21.15.163:0/2108204378 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:22.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 10 client.?.objecter ms_handle_connect 0x7f97580fd9d0 2024-02-19T21:03:22.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:22.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 10 client.?.objecter ms_handle_connect 0x7f9758104330 2024-02-19T21:03:22.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:22.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 10 client.?.objecter ms_handle_connect 0x7f97581007c0 2024-02-19T21:03:22.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:22.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321926379 0 0) 0x7f97580fd3e0 con 0x7f97580fd9d0 2024-02-19T21:03:22.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97400038e0 con 0x7f97580fd9d0 2024-02-19T21:03:22.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355561232 0 0) 0x7f97580fd040 con 0x7f9758104330 2024-02-19T21:03:22.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97580fd3e0 con 0x7f9758104330 2024-02-19T21:03:22.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969207064 0 0) 0x7f97580fd210 con 0x7f97581007c0 2024-02-19T21:03:22.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97580fd040 con 0x7f97581007c0 2024-02-19T21:03:22.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1657267019 0 0) 0x7f97580fd3e0 con 0x7f9758104330 2024-02-19T21:03:22.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97580fd210 con 0x7f9758104330 2024-02-19T21:03:22.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9750001d70 con 0x7f9758104330 2024-02-19T21:03:22.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2905405708 0 0) 0x7f97400038e0 con 0x7f97580fd9d0 2024-02-19T21:03:22.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97580fd3e0 con 0x7f97580fd9d0 2024-02-19T21:03:22.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f974c0018c0 con 0x7f97580fd9d0 2024-02-19T21:03:22.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1547972659 0 0) 0x7f97580fd040 con 0x7f97581007c0 2024-02-19T21:03:22.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97400038e0 con 0x7f97581007c0 2024-02-19T21:03:22.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.955+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9748002020 con 0x7f97581007c0 2024-02-19T21:03:22.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.956+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2712616984 0 0) 0x7f97580fd210 con 0x7f9758104330 2024-02-19T21:03:22.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.956+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 >> v1:172.21.15.163:6790/0 conn(0x7f97580fd9d0 legacy=0x7f97580fc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:22.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.956+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 >> v1:172.21.15.193:6789/0 conn(0x7f97581007c0 legacy=0x7f9758114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:22.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.956+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975811cc20 con 0x7f9758104330 2024-02-19T21:03:22.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.956+0000 7f975d757700 1 -- 172.21.15.163:0/2108204378 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9758119c50 con 0x7f9758104330 2024-02-19T21:03:22.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.956+0000 7f975d757700 10 client.14760.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:22.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.956+0000 7f975d757700 1 -- 172.21.15.163:0/2108204378 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f975811a0c0 con 0x7f9758104330 2024-02-19T21:03:22.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.956+0000 7f975d757700 1 librados: init done 2024-02-19T21:03:22.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.956+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9750002620 con 0x7f9758104330 2024-02-19T21:03:22.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.956+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f97500035d0 con 0x7f9758104330 2024-02-19T21:03:22.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.957+0000 7f975d757700 1 -- 172.21.15.163:0/2108204378 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9744004f20 con 0x7f9758104330 2024-02-19T21:03:22.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.957+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9750003730 con 0x7f9758104330 2024-02-19T21:03:22.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.958+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(84..84 src has 32..84) v4 ==== 6172+0+0 (unknown 2524177923 0 0) 0x7f9750075f20 con 0x7f9758104330 2024-02-19T21:03:22.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.958+0000 7f973ffff700 10 client.14760.objecter ms_dispatch 0x7f9758001550 osd_map(84..84 src has 32..84) v4 2024-02-19T21:03:22.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.958+0000 7f973ffff700 3 client.14760.objecter handle_osd_map got epochs [84,84] > 0 2024-02-19T21:03:22.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.958+0000 7f973ffff700 3 client.14760.objecter handle_osd_map decoding full epoch 84 2024-02-19T21:03:22.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.958+0000 7f973ffff700 20 client.14760.objecter dump_active .. 0 homeless 2024-02-19T21:03:22.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.958+0000 7f973ffff700 10 client.14760.objecter ms_handle_connect 0x7f9740063c20 2024-02-19T21:03:22.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:22.962+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f975004aa40 con 0x7f9758104330 2024-02-19T21:03:23.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:23.132+0000 7f975d757700 1 -- 172.21.15.163:0/2108204378 --> v1:172.21.15.163:6789/0 -- mon_command([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}] v 0) v1 -- 0x7f9744005750 con 0x7f9758104330 2024-02-19T21:03:23.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:23.136+0000 7f973ffff700 1 -- 172.21.15.163:0/2108204378 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}]=0 v27)=0 v27) v1 ==== 136+0+0 (unknown 398629962 0 0) 0x7f9750048e50 con 0x7f9758104330 2024-02-19T21:03:23.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:23.138+0000 7f975d757700 10 librados: watch_flush enter 2024-02-19T21:03:23.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:23.139+0000 7f975d757700 10 librados: watch_flush exit 2024-02-19T21:03:23.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:23.139+0000 7f975d757700 20 client.14760.objecter shutdown clearing up homeless session... 2024-02-19T21:03:23.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:23.139+0000 7f975d757700 10 client.14760.objecter successfully canceled tick 2024-02-19T21:03:23.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:23.139+0000 7f975d757700 1 -- 172.21.15.163:0/2108204378 >> v1:172.21.15.163:6800/1552514681 conn(0x7f9740063c20 legacy=0x7f97400660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:23.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:23.139+0000 7f975d757700 1 -- 172.21.15.163:0/2108204378 >> v1:172.21.15.163:6789/0 conn(0x7f9758104330 legacy=0x7f97581184d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:23.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:23.139+0000 7f975d757700 1 -- 172.21.15.163:0/2108204378 shutdown_connections 2024-02-19T21:03:23.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:23.139+0000 7f975d757700 1 -- 172.21.15.163:0/2108204378 >> 172.21.15.163:0/2108204378 conn(0x7f975806cc00 msgr2=0x7f97581077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:23.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:23.139+0000 7f975d757700 1 -- 172.21.15.163:0/2108204378 shutdown_connections 2024-02-19T21:03:23.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:23.139+0000 7f975d757700 1 -- 172.21.15.163:0/2108204378 wait complete. 2024-02-19T21:03:23.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:23.139+0000 7f975d757700 1 librados: shutdown 2024-02-19T21:03:23.427 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:03:23] ENGINE Bus STOPPING 2024-02-19T21:03:23.427 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:03:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-19T21:03:23.427 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:03:23] ENGINE Bus STOPPED 2024-02-19T21:03:23.427 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:03:23] ENGINE Bus STARTING 2024-02-19T21:03:23.427 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:03:23] ENGINE Serving on http://:::9283 2024-02-19T21:03:23.427 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:03:23] ENGINE Bus STARTED 2024-02-19T21:03:23.693 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:03:23] "GET /metrics HTTP/1.1" 200 172167 "" "Prometheus/2.33.4" 2024-02-19T21:03:23.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:22.494412+0000 osd.0 (osd.0) 41 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:03:23.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:22.495749+0000 osd.0 (osd.0) 42 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:03:23.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:23.136864+0000 mon.a (mon.0) 711 : audit [INF] from='client.? v1:172.21.15.163:0/2108204378' entity='client.admin' 2024-02-19T21:03:23.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:23.190392+0000 mon.b (mon.1) 77 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:03:23.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:23.192367+0000 mon.b (mon.1) 78 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:03:23.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:23.193307+0000 mon.b (mon.1) 79 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:03:23.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:23.201223+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:03:23.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:22.494412+0000 osd.0 (osd.0) 41 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:03:23.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:22.495749+0000 osd.0 (osd.0) 42 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:03:23.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:23.136864+0000 mon.a (mon.0) 711 : audit [INF] from='client.? v1:172.21.15.163:0/2108204378' entity='client.admin' 2024-02-19T21:03:23.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:23.190392+0000 mon.b (mon.1) 77 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:03:23.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:23.192367+0000 mon.b (mon.1) 78 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:03:23.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:23.193307+0000 mon.b (mon.1) 79 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:03:23.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:23.201223+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:03:23.713 INFO:tasks.thrashosds:Beginning thrashosds... 2024-02-19T21:03:23.713 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-19T21:03:23.713 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:03:23.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:22.494412+0000 osd.0 (osd.0) 41 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:03:23.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:22.495749+0000 osd.0 (osd.0) 42 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:03:23.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:23.136864+0000 mon.a (mon.0) 711 : audit [INF] from='client.? v1:172.21.15.163:0/2108204378' entity='client.admin' 2024-02-19T21:03:23.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:23.190392+0000 mon.b (mon.1) 77 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:03:23.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:23.192367+0000 mon.b (mon.1) 78 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:03:23.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:23.193307+0000 mon.b (mon.1) 79 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:03:23.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:23.201223+0000 mon.a (mon.0) 712 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:03:23.982 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:24.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:22.935040+0000 mgr.y (mgr.14550) 88 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:24.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:22.981082+0000 osd.5 (osd.5) 39 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:03:24.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:22.984541+0000 osd.5 (osd.5) 40 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:03:24.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:23.655680+0000 osd.3 (osd.3) 47 : cluster [DBG] 6.c scrub starts 2024-02-19T21:03:24.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:23.656679+0000 osd.3 (osd.3) 48 : cluster [DBG] 6.c scrub ok 2024-02-19T21:03:24.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:22.935040+0000 mgr.y (mgr.14550) 88 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:24.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:22.981082+0000 osd.5 (osd.5) 39 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:03:24.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:22.984541+0000 osd.5 (osd.5) 40 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:03:24.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:23.655680+0000 osd.3 (osd.3) 47 : cluster [DBG] 6.c scrub starts 2024-02-19T21:03:24.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:23.656679+0000 osd.3 (osd.3) 48 : cluster [DBG] 6.c scrub ok 2024-02-19T21:03:24.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:22.935040+0000 mgr.y (mgr.14550) 88 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:24.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:22.981082+0000 osd.5 (osd.5) 39 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:03:24.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:22.984541+0000 osd.5 (osd.5) 40 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:03:24.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:23.655680+0000 osd.3 (osd.3) 47 : cluster [DBG] 6.c scrub starts 2024-02-19T21:03:24.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:23.656679+0000 osd.3 (osd.3) 48 : cluster [DBG] 6.c scrub ok 2024-02-19T21:03:25.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.900+0000 7fcca91d8700 1 -- 172.21.15.163:0/1493712152 >> v1:172.21.15.163:6789/0 conn(0x7fcca4100780 legacy=0x7fcca4102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:25.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.900+0000 7fcca91d8700 1 -- 172.21.15.163:0/1493712152 shutdown_connections 2024-02-19T21:03:25.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.900+0000 7fcca91d8700 1 -- 172.21.15.163:0/1493712152 >> 172.21.15.163:0/1493712152 conn(0x7fcca40f35a0 msgr2=0x7fcca40f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:25.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.900+0000 7fcca91d8700 1 -- 172.21.15.163:0/1493712152 shutdown_connections 2024-02-19T21:03:25.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.900+0000 7fcca91d8700 1 -- 172.21.15.163:0/1493712152 wait complete. 2024-02-19T21:03:25.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.900+0000 7fcca91d8700 1 librados: starting msgr at 2024-02-19T21:03:25.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.900+0000 7fcca91d8700 1 librados: starting objecter 2024-02-19T21:03:25.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.900+0000 7fcca91d8700 1 Processor -- start 2024-02-19T21:03:25.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcca91d8700 1 -- start start 2024-02-19T21:03:25.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcca91d8700 1 librados: setting wanted keys 2024-02-19T21:03:25.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcca91d8700 1 librados: calling monclient init 2024-02-19T21:03:25.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcca91d8700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcca4117130 con 0x7fcca41042c0 2024-02-19T21:03:25.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcca91d8700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcca4118310 con 0x7fcca4100780 2024-02-19T21:03:25.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcca91d8700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcca4119510 con 0x7fcca40fd980 2024-02-19T21:03:25.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcca2d9d700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fcca40fd980 0x7fcca40f96f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:33720/0 (socket says 172.21.15.163:33720) 2024-02-19T21:03:25.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcca359e700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fcca41042c0 0x7fcca4115940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:58514/0 (socket says 172.21.15.163:58514) 2024-02-19T21:03:25.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcca2d9d700 1 -- 172.21.15.163:0/692656312 learned_addr learned my addr 172.21.15.163:0/692656312 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:25.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcc8bfff700 10 client.?.objecter ms_handle_connect 0x7fcca40fd980 2024-02-19T21:03:25.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcc8bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:25.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcc8bfff700 10 client.?.objecter ms_handle_connect 0x7fcca41042c0 2024-02-19T21:03:25.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcc8bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:25.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcc8bfff700 10 client.?.objecter ms_handle_connect 0x7fcca4100780 2024-02-19T21:03:25.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcc8bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:25.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783514768 0 0) 0x7fcca4119510 con 0x7fcca40fd980 2024-02-19T21:03:25.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc8c003030 con 0x7fcca40fd980 2024-02-19T21:03:25.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346613871 0 0) 0x7fcca4117130 con 0x7fcca41042c0 2024-02-19T21:03:25.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca4119510 con 0x7fcca41042c0 2024-02-19T21:03:25.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403611781 0 0) 0x7fcca4118310 con 0x7fcca4100780 2024-02-19T21:03:25.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca4117130 con 0x7fcca4100780 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.901+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2691921559 0 0) 0x7fcca4119510 con 0x7fcca41042c0 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcca4118310 con 0x7fcca41042c0 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2742521252 0 0) 0x7fcc8c003030 con 0x7fcca40fd980 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcca4119510 con 0x7fcca40fd980 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcc9c001b80 con 0x7fcca41042c0 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcc980019f0 con 0x7fcca40fd980 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3842704409 0 0) 0x7fcca4117130 con 0x7fcca4100780 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc8c003030 con 0x7fcca4100780 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcc94002020 con 0x7fcca4100780 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2315332816 0 0) 0x7fcca4118310 con 0x7fcca41042c0 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 >> v1:172.21.15.163:6790/0 conn(0x7fcca40fd980 legacy=0x7fcca40f96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 >> v1:172.21.15.193:6789/0 conn(0x7fcca4100780 legacy=0x7fcca40f9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca411a710 con 0x7fcca41042c0 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcca91d8700 1 -- 172.21.15.163:0/692656312 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca4117320 con 0x7fcca41042c0 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcca91d8700 10 client.14766.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:25.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcca91d8700 1 -- 172.21.15.163:0/692656312 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca4117810 con 0x7fcca41042c0 2024-02-19T21:03:25.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcca91d8700 1 librados: init done 2024-02-19T21:03:25.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fcc9c002700 con 0x7fcca41042c0 2024-02-19T21:03:25.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.902+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcc9c005d70 con 0x7fcca41042c0 2024-02-19T21:03:25.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.903+0000 7fcca91d8700 1 -- 172.21.15.163:0/692656312 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc90004f20 con 0x7fcca41042c0 2024-02-19T21:03:25.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.903+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fcc9c018ce0 con 0x7fcca41042c0 2024-02-19T21:03:25.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.903+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(84..84 src has 32..84) v4 ==== 6172+0+0 (unknown 2524177923 0 0) 0x7fcc9c0788e0 con 0x7fcca41042c0 2024-02-19T21:03:25.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.903+0000 7fcc8bfff700 10 client.14766.objecter ms_dispatch 0x7fcca4001550 osd_map(84..84 src has 32..84) v4 2024-02-19T21:03:25.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.903+0000 7fcc8bfff700 3 client.14766.objecter handle_osd_map got epochs [84,84] > 0 2024-02-19T21:03:25.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.903+0000 7fcc8bfff700 3 client.14766.objecter handle_osd_map decoding full epoch 84 2024-02-19T21:03:25.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.904+0000 7fcc8bfff700 20 client.14766.objecter dump_active .. 0 homeless 2024-02-19T21:03:25.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.904+0000 7fcc8bfff700 10 client.14766.objecter ms_handle_connect 0x7fcc8c063cb0 2024-02-19T21:03:25.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:25.908+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fcc9c04c490 con 0x7fcca41042c0 2024-02-19T21:03:26.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:26.069+0000 7fcca91d8700 1 -- 172.21.15.163:0/692656312 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcc90000b90 con 0x7fcc8c063cb0 2024-02-19T21:03:26.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:26.078+0000 7fcc8bfff700 1 -- 172.21.15.163:0/692656312 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357394 (unknown 2965378022 0 3759528334) 0x7fcc90000b90 con 0x7fcc8c063cb0 2024-02-19T21:03:26.075 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:03:26.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:26.081+0000 7fcca91d8700 10 librados: watch_flush enter 2024-02-19T21:03:26.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:26.081+0000 7fcca91d8700 10 librados: watch_flush exit 2024-02-19T21:03:26.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:26.081+0000 7fcca91d8700 20 client.14766.objecter shutdown clearing up homeless session... 2024-02-19T21:03:26.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:26.081+0000 7fcca91d8700 10 client.14766.objecter successfully canceled tick 2024-02-19T21:03:26.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:26.082+0000 7fcca91d8700 1 -- 172.21.15.163:0/692656312 >> v1:172.21.15.163:6800/1552514681 conn(0x7fcc8c063cb0 legacy=0x7fcc8c066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:26.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:26.082+0000 7fcca91d8700 1 -- 172.21.15.163:0/692656312 >> v1:172.21.15.163:6789/0 conn(0x7fcca41042c0 legacy=0x7fcca4115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:26.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:26.082+0000 7fcca91d8700 1 -- 172.21.15.163:0/692656312 shutdown_connections 2024-02-19T21:03:26.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:26.082+0000 7fcca91d8700 1 -- 172.21.15.163:0/692656312 >> 172.21.15.163:0/692656312 conn(0x7fcca40f35a0 msgr2=0x7fcca4107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:26.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:26.082+0000 7fcca91d8700 1 -- 172.21.15.163:0/692656312 shutdown_connections 2024-02-19T21:03:26.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:26.082+0000 7fcca91d8700 1 -- 172.21.15.163:0/692656312 wait complete. 2024-02-19T21:03:26.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:26.082+0000 7fcca91d8700 1 librados: shutdown 2024-02-19T21:03:26.079 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:03:26.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:24.935816+0000 mgr.y (mgr.14550) 89 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:26.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:25.518459+0000 osd.7 (osd.7) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:03:26.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:25.518527+0000 osd.7 (osd.7) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:03:26.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:24.935816+0000 mgr.y (mgr.14550) 89 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:26.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:25.518459+0000 osd.7 (osd.7) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:03:26.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:25.518527+0000 osd.7 (osd.7) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:03:26.799 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-02-19T21:03:24.935228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":841,"num_read_kb":593,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":476,"ondisk_log_size":476,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381612,"kb_used_data":4820,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579604,"statfs":{"total":767960285184,"available":765521514496,"internally_reserved":0,"allocated":4935680,"data_stored":2261261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002081"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.395740+0000","last_change":"2024-02-19T21:01:22.609338+0000","last_active":"2024-02-19T21:03:11.395740+0000","last_peered":"2024-02-19T21:03:11.395740+0000","last_clean":"2024-02-19T21:03:11.395740+0000","last_became_active":"2024-02-19T21:01:22.609059+0000","last_became_peered":"2024-02-19T21:01:22.609059+0000","last_unstale":"2024-02-19T21:03:11.395740+0000","last_undegraded":"2024-02-19T21:03:11.395740+0000","last_fullsized":"2024-02-19T21:03:11.395740+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:01:21.273400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390200+0000","last_change":"2024-02-19T21:03:05.073746+0000","last_active":"2024-02-19T21:03:11.390200+0000","last_peered":"2024-02-19T21:03:11.390200+0000","last_clean":"2024-02-19T21:03:11.390200+0000","last_became_active":"2024-02-19T21:01:20.289208+0000","last_became_peered":"2024-02-19T21:01:20.289208+0000","last_unstale":"2024-02-19T21:03:11.390200+0000","last_undegraded":"2024-02-19T21:03:11.390200+0000","last_fullsized":"2024-02-19T21:03:11.390200+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.073664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.073664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393353+0000","last_change":"2024-02-19T21:03:04.644754+0000","last_active":"2024-02-19T21:03:11.393353+0000","last_peered":"2024-02-19T21:03:11.393353+0000","last_clean":"2024-02-19T21:03:11.393353+0000","last_became_active":"2024-02-19T21:01:16.278753+0000","last_became_peered":"2024-02-19T21:01:16.278753+0000","last_unstale":"2024-02-19T21:03:11.393353+0000","last_undegraded":"2024-02-19T21:03:11.393353+0000","last_fullsized":"2024-02-19T21:03:11.393353+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.644689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.644689+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":63,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.395829+0000","last_change":"2024-02-19T21:02:48.863931+0000","last_active":"2024-02-19T21:03:11.395829+0000","last_peered":"2024-02-19T21:03:11.395829+0000","last_clean":"2024-02-19T21:03:11.395829+0000","last_became_active":"2024-02-19T21:01:18.283222+0000","last_became_peered":"2024-02-19T21:01:18.283222+0000","last_unstale":"2024-02-19T21:03:11.395829+0000","last_undegraded":"2024-02-19T21:03:11.395829+0000","last_fullsized":"2024-02-19T21:03:11.395829+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:48.863852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.863852+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390362+0000","last_change":"2024-02-19T21:02:23.100006+0000","last_active":"2024-02-19T21:03:11.390362+0000","last_peered":"2024-02-19T21:03:11.390362+0000","last_clean":"2024-02-19T21:03:11.390362+0000","last_became_active":"2024-02-19T21:01:22.293432+0000","last_became_peered":"2024-02-19T21:01:22.293432+0000","last_unstale":"2024-02-19T21:03:11.390362+0000","last_undegraded":"2024-02-19T21:03:11.390362+0000","last_fullsized":"2024-02-19T21:03:11.390362+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:23.099932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:23.099932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445014+0000","last_change":"2024-02-19T21:02:29.480341+0000","last_active":"2024-02-19T21:03:12.445014+0000","last_peered":"2024-02-19T21:03:12.445014+0000","last_clean":"2024-02-19T21:03:12.445014+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:03:12.445014+0000","last_undegraded":"2024-02-19T21:03:12.445014+0000","last_fullsized":"2024-02-19T21:03:12.445014+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:29.480275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:29.480275+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.396465+0000","last_change":"2024-02-19T21:02:20.970293+0000","last_active":"2024-02-19T21:03:11.396465+0000","last_peered":"2024-02-19T21:03:11.396465+0000","last_clean":"2024-02-19T21:03:11.396465+0000","last_became_active":"2024-02-19T21:01:16.272132+0000","last_became_peered":"2024-02-19T21:01:16.272132+0000","last_unstale":"2024-02-19T21:03:11.396465+0000","last_undegraded":"2024-02-19T21:03:11.396465+0000","last_fullsized":"2024-02-19T21:03:11.396465+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:20.970216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:20.970216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:16.047374+0000","last_change":"2024-02-19T21:03:16.047345+0000","last_active":"2024-02-19T21:03:16.047374+0000","last_peered":"2024-02-19T21:03:16.047374+0000","last_clean":"2024-02-19T21:03:16.047374+0000","last_became_active":"2024-02-19T21:01:18.287645+0000","last_became_peered":"2024-02-19T21:01:18.287645+0000","last_unstale":"2024-02-19T21:03:16.047374+0000","last_undegraded":"2024-02-19T21:03:16.047374+0000","last_fullsized":"2024-02-19T21:03:16.047374+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:16.047258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:16.047258+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:13.096766+0000","last_change":"2024-02-19T21:03:13.096737+0000","last_active":"2024-02-19T21:03:13.096766+0000","last_peered":"2024-02-19T21:03:13.096766+0000","last_clean":"2024-02-19T21:03:13.096766+0000","last_became_active":"2024-02-19T21:01:22.293282+0000","last_became_peered":"2024-02-19T21:01:22.293282+0000","last_unstale":"2024-02-19T21:03:13.096766+0000","last_undegraded":"2024-02-19T21:03:13.096766+0000","last_fullsized":"2024-02-19T21:03:13.096766+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:13.096667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:13.096667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596578+0000","last_change":"2024-02-19T21:02:35.546613+0000","last_active":"2024-02-19T21:03:11.596578+0000","last_peered":"2024-02-19T21:03:11.596578+0000","last_clean":"2024-02-19T21:03:11.596578+0000","last_became_active":"2024-02-19T21:01:20.287940+0000","last_became_peered":"2024-02-19T21:01:20.287940+0000","last_unstale":"2024-02-19T21:03:11.596578+0000","last_undegraded":"2024-02-19T21:03:11.596578+0000","last_fullsized":"2024-02-19T21:03:11.596578+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:35.546526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:35.546526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596749+0000","last_change":"2024-02-19T21:02:20.575203+0000","last_active":"2024-02-19T21:03:11.596749+0000","last_peered":"2024-02-19T21:03:11.596749+0000","last_clean":"2024-02-19T21:03:11.596749+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:03:11.596749+0000","last_undegraded":"2024-02-19T21:03:11.596749+0000","last_fullsized":"2024-02-19T21:03:11.596749+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:20.575133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:20.575133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390720+0000","last_change":"2024-02-19T21:02:27.098930+0000","last_active":"2024-02-19T21:03:11.390720+0000","last_peered":"2024-02-19T21:03:11.390720+0000","last_clean":"2024-02-19T21:03:11.390720+0000","last_became_active":"2024-02-19T21:01:18.285735+0000","last_became_peered":"2024-02-19T21:01:18.285735+0000","last_unstale":"2024-02-19T21:03:11.390720+0000","last_undegraded":"2024-02-19T21:03:11.390720+0000","last_fullsized":"2024-02-19T21:03:11.390720+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:02:27.098844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:27.098844+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393117+0000","last_change":"2024-02-19T21:03:00.570766+0000","last_active":"2024-02-19T21:03:11.393117+0000","last_peered":"2024-02-19T21:03:11.393117+0000","last_clean":"2024-02-19T21:03:11.393117+0000","last_became_active":"2024-02-19T21:01:22.292714+0000","last_became_peered":"2024-02-19T21:01:22.292714+0000","last_unstale":"2024-02-19T21:03:11.393117+0000","last_undegraded":"2024-02-19T21:03:11.393117+0000","last_fullsized":"2024-02-19T21:03:11.393117+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:00.570677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:00.570677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079804+0000","last_change":"2024-02-19T21:01:20.293436+0000","last_active":"2024-02-19T21:03:12.079804+0000","last_peered":"2024-02-19T21:03:12.079804+0000","last_clean":"2024-02-19T21:03:12.079804+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:03:12.079804+0000","last_undegraded":"2024-02-19T21:03:12.079804+0000","last_fullsized":"2024-02-19T21:03:12.079804+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:01:19.270373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596676+0000","last_change":"2024-02-19T21:02:57.552979+0000","last_active":"2024-02-19T21:03:11.596676+0000","last_peered":"2024-02-19T21:03:11.596676+0000","last_clean":"2024-02-19T21:03:11.596676+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:03:11.596676+0000","last_undegraded":"2024-02-19T21:03:11.596676+0000","last_fullsized":"2024-02-19T21:03:11.596676+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.552886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.552886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390433+0000","last_change":"2024-02-19T21:02:26.073713+0000","last_active":"2024-02-19T21:03:11.390433+0000","last_peered":"2024-02-19T21:03:11.390433+0000","last_clean":"2024-02-19T21:03:11.390433+0000","last_became_active":"2024-02-19T21:01:18.285269+0000","last_became_peered":"2024-02-19T21:01:18.285269+0000","last_unstale":"2024-02-19T21:03:11.390433+0000","last_undegraded":"2024-02-19T21:03:11.390433+0000","last_fullsized":"2024-02-19T21:03:11.390433+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:26.073625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:26.073625+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.396393+0000","last_change":"2024-02-19T21:02:24.948910+0000","last_active":"2024-02-19T21:03:11.396393+0000","last_peered":"2024-02-19T21:03:11.396393+0000","last_clean":"2024-02-19T21:03:11.396393+0000","last_became_active":"2024-02-19T21:01:22.607099+0000","last_became_peered":"2024-02-19T21:01:22.607099+0000","last_unstale":"2024-02-19T21:03:11.396393+0000","last_undegraded":"2024-02-19T21:03:11.396393+0000","last_fullsized":"2024-02-19T21:03:11.396393+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:24.948832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:24.948832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390940+0000","last_change":"2024-02-19T21:03:09.010420+0000","last_active":"2024-02-19T21:03:11.390940+0000","last_peered":"2024-02-19T21:03:11.390940+0000","last_clean":"2024-02-19T21:03:11.390940+0000","last_became_active":"2024-02-19T21:01:20.291672+0000","last_became_peered":"2024-02-19T21:01:20.291672+0000","last_unstale":"2024-02-19T21:03:11.390940+0000","last_undegraded":"2024-02-19T21:03:11.390940+0000","last_fullsized":"2024-02-19T21:03:11.390940+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:09.010341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:09.010341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.637528+0000","last_change":"2024-02-19T21:03:20.637528+0000","last_active":"2024-02-19T21:03:20.637528+0000","last_peered":"2024-02-19T21:03:20.637528+0000","last_clean":"2024-02-19T21:03:20.637528+0000","last_became_active":"2024-02-19T21:01:16.274811+0000","last_became_peered":"2024-02-19T21:01:16.274811+0000","last_unstale":"2024-02-19T21:03:20.637528+0000","last_undegraded":"2024-02-19T21:03:20.637528+0000","last_fullsized":"2024-02-19T21:03:20.637528+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:20.637465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:20.637465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":61,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.396228+0000","last_change":"2024-02-19T21:02:57.819231+0000","last_active":"2024-02-19T21:03:11.396228+0000","last_peered":"2024-02-19T21:03:11.396228+0000","last_clean":"2024-02-19T21:03:11.396228+0000","last_became_active":"2024-02-19T21:01:18.285926+0000","last_became_peered":"2024-02-19T21:01:18.285926+0000","last_unstale":"2024-02-19T21:03:11.396228+0000","last_undegraded":"2024-02-19T21:03:11.396228+0000","last_fullsized":"2024-02-19T21:03:11.396228+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:02:57.819125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.819125+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.081355+0000","last_change":"2024-02-19T21:03:12.081314+0000","last_active":"2024-02-19T21:03:12.081355+0000","last_peered":"2024-02-19T21:03:12.081355+0000","last_clean":"2024-02-19T21:03:12.081355+0000","last_became_active":"2024-02-19T21:01:22.607435+0000","last_became_peered":"2024-02-19T21:01:22.607435+0000","last_unstale":"2024-02-19T21:03:12.081355+0000","last_undegraded":"2024-02-19T21:03:12.081355+0000","last_fullsized":"2024-02-19T21:03:12.081355+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:03:12.081227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445115+0000","last_change":"2024-02-19T21:02:54.544941+0000","last_active":"2024-02-19T21:03:12.445115+0000","last_peered":"2024-02-19T21:03:12.445115+0000","last_clean":"2024-02-19T21:03:12.445115+0000","last_became_active":"2024-02-19T21:01:20.290705+0000","last_became_peered":"2024-02-19T21:01:20.290705+0000","last_unstale":"2024-02-19T21:03:12.445115+0000","last_undegraded":"2024-02-19T21:03:12.445115+0000","last_fullsized":"2024-02-19T21:03:12.445115+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:54.544834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:54.544834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.598772+0000","last_change":"2024-02-19T21:02:47.805245+0000","last_active":"2024-02-19T21:03:11.598772+0000","last_peered":"2024-02-19T21:03:11.598772+0000","last_clean":"2024-02-19T21:03:11.598772+0000","last_became_active":"2024-02-19T21:01:16.274524+0000","last_became_peered":"2024-02-19T21:01:16.274524+0000","last_unstale":"2024-02-19T21:03:11.598772+0000","last_undegraded":"2024-02-19T21:03:11.598772+0000","last_fullsized":"2024-02-19T21:03:11.598772+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.805171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.805171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":63,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.433438+0000","last_change":"2024-02-19T21:02:47.238547+0000","last_active":"2024-02-19T21:03:12.433438+0000","last_peered":"2024-02-19T21:03:12.433438+0000","last_clean":"2024-02-19T21:03:12.433438+0000","last_became_active":"2024-02-19T21:01:18.289712+0000","last_became_peered":"2024-02-19T21:01:18.289712+0000","last_unstale":"2024-02-19T21:03:12.433438+0000","last_undegraded":"2024-02-19T21:03:12.433438+0000","last_fullsized":"2024-02-19T21:03:12.433438+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:47.238470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.238470+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.446016+0000","last_change":"2024-02-19T21:03:06.456310+0000","last_active":"2024-02-19T21:03:12.446016+0000","last_peered":"2024-02-19T21:03:12.446016+0000","last_clean":"2024-02-19T21:03:12.446016+0000","last_became_active":"2024-02-19T21:01:22.293230+0000","last_became_peered":"2024-02-19T21:01:22.293230+0000","last_unstale":"2024-02-19T21:03:12.446016+0000","last_undegraded":"2024-02-19T21:03:12.446016+0000","last_fullsized":"2024-02-19T21:03:12.446016+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.456212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.456212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393656+0000","last_change":"2024-02-19T21:03:08.636037+0000","last_active":"2024-02-19T21:03:11.393656+0000","last_peered":"2024-02-19T21:03:11.393656+0000","last_clean":"2024-02-19T21:03:11.393656+0000","last_became_active":"2024-02-19T21:01:20.288016+0000","last_became_peered":"2024-02-19T21:01:20.288016+0000","last_unstale":"2024-02-19T21:03:11.393656+0000","last_undegraded":"2024-02-19T21:03:11.393656+0000","last_fullsized":"2024-02-19T21:03:11.393656+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:08.635943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.635943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:21.619457+0000","last_change":"2024-02-19T21:03:21.619457+0000","last_active":"2024-02-19T21:03:21.619457+0000","last_peered":"2024-02-19T21:03:21.619457+0000","last_clean":"2024-02-19T21:03:21.619457+0000","last_became_active":"2024-02-19T21:01:16.277826+0000","last_became_peered":"2024-02-19T21:01:16.277826+0000","last_unstale":"2024-02-19T21:03:21.619457+0000","last_undegraded":"2024-02-19T21:03:21.619457+0000","last_fullsized":"2024-02-19T21:03:21.619457+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:21.619389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:21.619389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":57,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.599346+0000","last_change":"2024-02-19T21:02:29.684765+0000","last_active":"2024-02-19T21:03:11.599346+0000","last_peered":"2024-02-19T21:03:11.599346+0000","last_clean":"2024-02-19T21:03:11.599346+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:03:11.599346+0000","last_undegraded":"2024-02-19T21:03:11.599346+0000","last_fullsized":"2024-02-19T21:03:11.599346+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:29.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:29.684698+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.597047+0000","last_change":"2024-02-19T21:02:36.585127+0000","last_active":"2024-02-19T21:03:11.597047+0000","last_peered":"2024-02-19T21:03:11.597047+0000","last_clean":"2024-02-19T21:03:11.597047+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:03:11.597047+0000","last_undegraded":"2024-02-19T21:03:11.597047+0000","last_fullsized":"2024-02-19T21:03:11.597047+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:02:36.585007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:36.585007+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":73,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.425426+0000","last_change":"2024-02-19T21:02:37.842634+0000","last_active":"2024-02-19T21:03:20.425426+0000","last_peered":"2024-02-19T21:03:20.425426+0000","last_clean":"2024-02-19T21:03:20.425426+0000","last_became_active":"2024-02-19T21:01:20.290567+0000","last_became_peered":"2024-02-19T21:01:20.290567+0000","last_unstale":"2024-02-19T21:03:20.425426+0000","last_undegraded":"2024-02-19T21:03:20.425426+0000","last_fullsized":"2024-02-19T21:03:20.425426+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:37.842545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:37.842545+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079313+0000","last_change":"2024-02-19T21:03:06.000805+0000","last_active":"2024-02-19T21:03:12.079313+0000","last_peered":"2024-02-19T21:03:12.079313+0000","last_clean":"2024-02-19T21:03:12.079313+0000","last_became_active":"2024-02-19T21:01:16.272284+0000","last_became_peered":"2024-02-19T21:01:16.272284+0000","last_unstale":"2024-02-19T21:03:12.079313+0000","last_undegraded":"2024-02-19T21:03:12.079313+0000","last_fullsized":"2024-02-19T21:03:12.079313+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.000732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.000732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":53,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393617+0000","last_change":"2024-02-19T21:02:32.550264+0000","last_active":"2024-02-19T21:03:11.393617+0000","last_peered":"2024-02-19T21:03:11.393617+0000","last_clean":"2024-02-19T21:03:11.393617+0000","last_became_active":"2024-02-19T21:01:18.293865+0000","last_became_peered":"2024-02-19T21:01:18.293865+0000","last_unstale":"2024-02-19T21:03:11.393617+0000","last_undegraded":"2024-02-19T21:03:11.393617+0000","last_fullsized":"2024-02-19T21:03:11.393617+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:02:32.550190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.550190+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.396000+0000","last_change":"2024-02-19T21:02:41.964875+0000","last_active":"2024-02-19T21:03:11.396000+0000","last_peered":"2024-02-19T21:03:11.396000+0000","last_clean":"2024-02-19T21:03:11.396000+0000","last_became_active":"2024-02-19T21:01:22.608997+0000","last_became_peered":"2024-02-19T21:01:22.608997+0000","last_unstale":"2024-02-19T21:03:11.396000+0000","last_undegraded":"2024-02-19T21:03:11.396000+0000","last_fullsized":"2024-02-19T21:03:11.396000+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:41.964809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.964809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596434+0000","last_change":"2024-02-19T21:03:04.486168+0000","last_active":"2024-02-19T21:03:11.596434+0000","last_peered":"2024-02-19T21:03:11.596434+0000","last_clean":"2024-02-19T21:03:11.596434+0000","last_became_active":"2024-02-19T21:01:20.287961+0000","last_became_peered":"2024-02-19T21:01:20.287961+0000","last_unstale":"2024-02-19T21:03:11.596434+0000","last_undegraded":"2024-02-19T21:03:11.596434+0000","last_fullsized":"2024-02-19T21:03:11.596434+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.486067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.486067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.599433+0000","last_change":"2024-02-19T21:02:43.790280+0000","last_active":"2024-02-19T21:03:11.599433+0000","last_peered":"2024-02-19T21:03:11.599433+0000","last_clean":"2024-02-19T21:03:11.599433+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:03:11.599433+0000","last_undegraded":"2024-02-19T21:03:11.599433+0000","last_fullsized":"2024-02-19T21:03:11.599433+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:43.790206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:43.790206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":57,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.395796+0000","last_change":"2024-02-19T21:02:52.859531+0000","last_active":"2024-02-19T21:03:11.395796+0000","last_peered":"2024-02-19T21:03:11.395796+0000","last_clean":"2024-02-19T21:03:11.395796+0000","last_became_active":"2024-02-19T21:01:18.293793+0000","last_became_peered":"2024-02-19T21:01:18.293793+0000","last_unstale":"2024-02-19T21:03:11.395796+0000","last_undegraded":"2024-02-19T21:03:11.395796+0000","last_fullsized":"2024-02-19T21:03:11.395796+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:52.859453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.859453+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596374+0000","last_change":"2024-02-19T21:02:45.579070+0000","last_active":"2024-02-19T21:03:11.596374+0000","last_peered":"2024-02-19T21:03:11.596374+0000","last_clean":"2024-02-19T21:03:11.596374+0000","last_became_active":"2024-02-19T21:01:22.297693+0000","last_became_peered":"2024-02-19T21:01:22.297693+0000","last_unstale":"2024-02-19T21:03:11.596374+0000","last_undegraded":"2024-02-19T21:03:11.596374+0000","last_fullsized":"2024-02-19T21:03:11.596374+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:45.578978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:45.578978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.598557+0000","last_change":"2024-02-19T21:02:30.724973+0000","last_active":"2024-02-19T21:03:11.598557+0000","last_peered":"2024-02-19T21:03:11.598557+0000","last_clean":"2024-02-19T21:03:11.598557+0000","last_became_active":"2024-02-19T21:01:20.290962+0000","last_became_peered":"2024-02-19T21:01:20.290962+0000","last_unstale":"2024-02-19T21:03:11.598557+0000","last_undegraded":"2024-02-19T21:03:11.598557+0000","last_fullsized":"2024-02-19T21:03:11.598557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:30.724908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:30.724908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.080134+0000","last_change":"2024-02-19T21:02:52.021686+0000","last_active":"2024-02-19T21:03:12.080134+0000","last_peered":"2024-02-19T21:03:12.080134+0000","last_clean":"2024-02-19T21:03:12.080134+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:03:12.080134+0000","last_undegraded":"2024-02-19T21:03:12.080134+0000","last_fullsized":"2024-02-19T21:03:12.080134+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.021603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.021603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.395598+0000","last_change":"2024-02-19T21:02:22.939275+0000","last_active":"2024-02-19T21:03:11.395598+0000","last_peered":"2024-02-19T21:03:11.395598+0000","last_clean":"2024-02-19T21:03:11.395598+0000","last_became_active":"2024-02-19T21:01:18.287874+0000","last_became_peered":"2024-02-19T21:01:18.287874+0000","last_unstale":"2024-02-19T21:03:11.395598+0000","last_undegraded":"2024-02-19T21:03:11.395598+0000","last_fullsized":"2024-02-19T21:03:11.395598+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:02:22.939192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:22.939192+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.395541+0000","last_change":"2024-02-19T21:03:05.734069+0000","last_active":"2024-02-19T21:03:11.395541+0000","last_peered":"2024-02-19T21:03:11.395541+0000","last_clean":"2024-02-19T21:03:11.395541+0000","last_became_active":"2024-02-19T21:01:22.292704+0000","last_became_peered":"2024-02-19T21:01:22.292704+0000","last_unstale":"2024-02-19T21:03:11.395541+0000","last_undegraded":"2024-02-19T21:03:11.395541+0000","last_fullsized":"2024-02-19T21:03:11.395541+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.733980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.733980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445018+0000","last_change":"2024-02-19T21:02:40.600482+0000","last_active":"2024-02-19T21:03:12.445018+0000","last_peered":"2024-02-19T21:03:12.445018+0000","last_clean":"2024-02-19T21:03:12.445018+0000","last_became_active":"2024-02-19T21:01:20.289020+0000","last_became_peered":"2024-02-19T21:01:20.289020+0000","last_unstale":"2024-02-19T21:03:12.445018+0000","last_undegraded":"2024-02-19T21:03:12.445018+0000","last_fullsized":"2024-02-19T21:03:12.445018+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:40.600410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.600410+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.445580+0000","last_change":"2024-02-19T21:03:20.445543+0000","last_active":"2024-02-19T21:03:20.445580+0000","last_peered":"2024-02-19T21:03:20.445580+0000","last_clean":"2024-02-19T21:03:20.445580+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:03:20.445580+0000","last_undegraded":"2024-02-19T21:03:20.445580+0000","last_fullsized":"2024-02-19T21:03:20.445580+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:20.445477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:20.445477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":57,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390557+0000","last_change":"2024-02-19T21:02:32.040809+0000","last_active":"2024-02-19T21:03:11.390557+0000","last_peered":"2024-02-19T21:03:11.390557+0000","last_clean":"2024-02-19T21:03:11.390557+0000","last_became_active":"2024-02-19T21:01:18.286128+0000","last_became_peered":"2024-02-19T21:01:18.286128+0000","last_unstale":"2024-02-19T21:03:11.390557+0000","last_undegraded":"2024-02-19T21:03:11.390557+0000","last_fullsized":"2024-02-19T21:03:11.390557+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:32.040731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.040731+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.392789+0000","last_change":"2024-02-19T21:02:51.631722+0000","last_active":"2024-02-19T21:03:11.392789+0000","last_peered":"2024-02-19T21:03:11.392789+0000","last_clean":"2024-02-19T21:03:11.392789+0000","last_became_active":"2024-02-19T21:01:22.293093+0000","last_became_peered":"2024-02-19T21:01:22.293093+0000","last_unstale":"2024-02-19T21:03:11.392789+0000","last_undegraded":"2024-02-19T21:03:11.392789+0000","last_fullsized":"2024-02-19T21:03:11.392789+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:51.631657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:51.631657+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.395921+0000","last_change":"2024-02-19T21:03:03.776497+0000","last_active":"2024-02-19T21:03:11.395921+0000","last_peered":"2024-02-19T21:03:11.395921+0000","last_clean":"2024-02-19T21:03:11.395921+0000","last_became_active":"2024-02-19T21:01:20.289691+0000","last_became_peered":"2024-02-19T21:01:20.289691+0000","last_unstale":"2024-02-19T21:03:11.395921+0000","last_undegraded":"2024-02-19T21:03:11.395921+0000","last_fullsized":"2024-02-19T21:03:11.395921+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_clean_scrub_stamp":"2024-02-19T21:03:03.776397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.599152+0000","last_change":"2024-02-19T21:02:21.657425+0000","last_active":"2024-02-19T21:03:11.599152+0000","last_peered":"2024-02-19T21:03:11.599152+0000","last_clean":"2024-02-19T21:03:11.599152+0000","last_became_active":"2024-02-19T21:01:16.271316+0000","last_became_peered":"2024-02-19T21:01:16.271316+0000","last_unstale":"2024-02-19T21:03:11.599152+0000","last_undegraded":"2024-02-19T21:03:11.599152+0000","last_fullsized":"2024-02-19T21:03:11.599152+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.657334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.657334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:22.984626+0000","last_change":"2024-02-19T21:03:22.984626+0000","last_active":"2024-02-19T21:03:22.984626+0000","last_peered":"2024-02-19T21:03:22.984626+0000","last_clean":"2024-02-19T21:03:22.984626+0000","last_became_active":"2024-02-19T21:01:30.922577+0000","last_became_peered":"2024-02-19T21:01:30.922577+0000","last_unstale":"2024-02-19T21:03:22.984626+0000","last_undegraded":"2024-02-19T21:03:22.984626+0000","last_fullsized":"2024-02-19T21:03:22.984626+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:22.984547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:22.984547+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.391137+0000","last_change":"2024-02-19T21:02:39.092233+0000","last_active":"2024-02-19T21:03:11.391137+0000","last_peered":"2024-02-19T21:03:11.391137+0000","last_clean":"2024-02-19T21:03:11.391137+0000","last_became_active":"2024-02-19T21:01:22.299969+0000","last_became_peered":"2024-02-19T21:01:22.299969+0000","last_unstale":"2024-02-19T21:03:11.391137+0000","last_undegraded":"2024-02-19T21:03:11.391137+0000","last_fullsized":"2024-02-19T21:03:11.391137+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.092154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":65,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.426058+0000","last_change":"2024-02-19T21:01:20.284282+0000","last_active":"2024-02-19T21:03:20.426058+0000","last_peered":"2024-02-19T21:03:20.426058+0000","last_clean":"2024-02-19T21:03:20.426058+0000","last_became_active":"2024-02-19T21:01:20.283943+0000","last_became_peered":"2024-02-19T21:01:20.283943+0000","last_unstale":"2024-02-19T21:03:20.426058+0000","last_undegraded":"2024-02-19T21:03:20.426058+0000","last_fullsized":"2024-02-19T21:03:20.426058+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:01:19.270373+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393335+0000","last_change":"2024-02-19T21:02:21.693740+0000","last_active":"2024-02-19T21:03:11.393335+0000","last_peered":"2024-02-19T21:03:11.393335+0000","last_clean":"2024-02-19T21:03:11.393335+0000","last_became_active":"2024-02-19T21:01:16.273181+0000","last_became_peered":"2024-02-19T21:01:16.273181+0000","last_unstale":"2024-02-19T21:03:11.393335+0000","last_undegraded":"2024-02-19T21:03:11.393335+0000","last_fullsized":"2024-02-19T21:03:11.393335+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.693675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.693675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079828+0000","last_change":"2024-02-19T21:02:57.021496+0000","last_active":"2024-02-19T21:03:12.079828+0000","last_peered":"2024-02-19T21:03:12.079828+0000","last_clean":"2024-02-19T21:03:12.079828+0000","last_became_active":"2024-02-19T21:01:18.293786+0000","last_became_peered":"2024-02-19T21:01:18.293786+0000","last_unstale":"2024-02-19T21:03:12.079828+0000","last_undegraded":"2024-02-19T21:03:12.079828+0000","last_fullsized":"2024-02-19T21:03:12.079828+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:57.021403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.021403+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.392931+0000","last_change":"2024-02-19T21:02:39.442600+0000","last_active":"2024-02-19T21:03:11.392931+0000","last_peered":"2024-02-19T21:03:11.392931+0000","last_clean":"2024-02-19T21:03:11.392931+0000","last_became_active":"2024-02-19T21:01:22.297188+0000","last_became_peered":"2024-02-19T21:01:22.297188+0000","last_unstale":"2024-02-19T21:03:11.392931+0000","last_undegraded":"2024-02-19T21:03:11.392931+0000","last_fullsized":"2024-02-19T21:03:11.392931+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:02:39.442533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.442533+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.541980+0000","last_change":"2024-02-19T21:03:11.063302+0000","last_active":"2024-02-19T21:03:20.541980+0000","last_peered":"2024-02-19T21:03:20.541980+0000","last_clean":"2024-02-19T21:03:20.541980+0000","last_became_active":"2024-02-19T21:01:20.288799+0000","last_became_peered":"2024-02-19T21:01:20.288799+0000","last_unstale":"2024-02-19T21:03:20.541980+0000","last_undegraded":"2024-02-19T21:03:20.541980+0000","last_fullsized":"2024-02-19T21:03:20.541980+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:11.063222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:11.063222+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079545+0000","last_change":"2024-02-19T21:02:49.056495+0000","last_active":"2024-02-19T21:03:12.079545+0000","last_peered":"2024-02-19T21:03:12.079545+0000","last_clean":"2024-02-19T21:03:12.079545+0000","last_became_active":"2024-02-19T21:01:16.271608+0000","last_became_peered":"2024-02-19T21:01:16.271608+0000","last_unstale":"2024-02-19T21:03:12.079545+0000","last_undegraded":"2024-02-19T21:03:12.079545+0000","last_fullsized":"2024-02-19T21:03:12.079545+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.056412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.056412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":53,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.396124+0000","last_change":"2024-02-19T21:03:08.742946+0000","last_active":"2024-02-19T21:03:11.396124+0000","last_peered":"2024-02-19T21:03:11.396124+0000","last_clean":"2024-02-19T21:03:11.396124+0000","last_became_active":"2024-02-19T21:01:18.294369+0000","last_became_peered":"2024-02-19T21:01:18.294369+0000","last_unstale":"2024-02-19T21:03:11.396124+0000","last_undegraded":"2024-02-19T21:03:11.396124+0000","last_fullsized":"2024-02-19T21:03:11.396124+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:03:08.742876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.742876+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596968+0000","last_change":"2024-02-19T21:02:38.633030+0000","last_active":"2024-02-19T21:03:11.596968+0000","last_peered":"2024-02-19T21:03:11.596968+0000","last_clean":"2024-02-19T21:03:11.596968+0000","last_became_active":"2024-02-19T21:01:22.608285+0000","last_became_peered":"2024-02-19T21:01:22.608285+0000","last_unstale":"2024-02-19T21:03:11.596968+0000","last_undegraded":"2024-02-19T21:03:11.596968+0000","last_fullsized":"2024-02-19T21:03:11.596968+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:38.632950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:38.632950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.080120+0000","last_change":"2024-02-19T21:03:08.984824+0000","last_active":"2024-02-19T21:03:12.080120+0000","last_peered":"2024-02-19T21:03:12.080120+0000","last_clean":"2024-02-19T21:03:12.080120+0000","last_became_active":"2024-02-19T21:01:20.288589+0000","last_became_peered":"2024-02-19T21:01:20.288589+0000","last_unstale":"2024-02-19T21:03:12.080120+0000","last_undegraded":"2024-02-19T21:03:12.080120+0000","last_fullsized":"2024-02-19T21:03:12.080120+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:08.984720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.984720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.598593+0000","last_change":"2024-02-19T21:02:46.842812+0000","last_active":"2024-02-19T21:03:11.598593+0000","last_peered":"2024-02-19T21:03:11.598593+0000","last_clean":"2024-02-19T21:03:11.598593+0000","last_became_active":"2024-02-19T21:01:16.271210+0000","last_became_peered":"2024-02-19T21:01:16.271210+0000","last_unstale":"2024-02-19T21:03:11.598593+0000","last_undegraded":"2024-02-19T21:03:11.598593+0000","last_fullsized":"2024-02-19T21:03:11.598593+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:46.842732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.842732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.396349+0000","last_change":"2024-02-19T21:02:33.968248+0000","last_active":"2024-02-19T21:03:11.396349+0000","last_peered":"2024-02-19T21:03:11.396349+0000","last_clean":"2024-02-19T21:03:11.396349+0000","last_became_active":"2024-02-19T21:01:18.293913+0000","last_became_peered":"2024-02-19T21:01:18.293913+0000","last_unstale":"2024-02-19T21:03:11.396349+0000","last_undegraded":"2024-02-19T21:03:11.396349+0000","last_fullsized":"2024-02-19T21:03:11.396349+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:02:33.968182+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.433027+0000","last_change":"2024-02-19T21:02:34.274719+0000","last_active":"2024-02-19T21:03:12.433027+0000","last_peered":"2024-02-19T21:03:12.433027+0000","last_clean":"2024-02-19T21:03:12.433027+0000","last_became_active":"2024-02-19T21:01:22.297086+0000","last_became_peered":"2024-02-19T21:01:22.297086+0000","last_unstale":"2024-02-19T21:03:12.433027+0000","last_undegraded":"2024-02-19T21:03:12.433027+0000","last_fullsized":"2024-02-19T21:03:12.433027+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:34.274650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:34.274650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.396431+0000","last_change":"2024-02-19T21:02:49.863142+0000","last_active":"2024-02-19T21:03:11.396431+0000","last_peered":"2024-02-19T21:03:11.396431+0000","last_clean":"2024-02-19T21:03:11.396431+0000","last_became_active":"2024-02-19T21:01:20.285298+0000","last_became_peered":"2024-02-19T21:01:20.285298+0000","last_unstale":"2024-02-19T21:03:11.396431+0000","last_undegraded":"2024-02-19T21:03:11.396431+0000","last_fullsized":"2024-02-19T21:03:11.396431+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.863037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.432866+0000","last_change":"2024-02-19T21:02:49.177055+0000","last_active":"2024-02-19T21:03:12.432866+0000","last_peered":"2024-02-19T21:03:12.432866+0000","last_clean":"2024-02-19T21:03:12.432866+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:03:12.432866+0000","last_undegraded":"2024-02-19T21:03:12.432866+0000","last_fullsized":"2024-02-19T21:03:12.432866+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.176969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.176969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393503+0000","last_change":"2024-02-19T21:03:07.669010+0000","last_active":"2024-02-19T21:03:11.393503+0000","last_peered":"2024-02-19T21:03:11.393503+0000","last_clean":"2024-02-19T21:03:11.393503+0000","last_became_active":"2024-02-19T21:01:18.293773+0000","last_became_peered":"2024-02-19T21:01:18.293773+0000","last_unstale":"2024-02-19T21:03:11.393503+0000","last_undegraded":"2024-02-19T21:03:11.393503+0000","last_fullsized":"2024-02-19T21:03:11.393503+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:07.668943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:07.668943+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:17.625801+0000","last_change":"2024-02-19T21:03:17.625775+0000","last_active":"2024-02-19T21:03:17.625801+0000","last_peered":"2024-02-19T21:03:17.625801+0000","last_clean":"2024-02-19T21:03:17.625801+0000","last_became_active":"2024-02-19T21:01:22.295912+0000","last_became_peered":"2024-02-19T21:01:22.295912+0000","last_unstale":"2024-02-19T21:03:17.625801+0000","last_undegraded":"2024-02-19T21:03:17.625801+0000","last_fullsized":"2024-02-19T21:03:17.625801+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:03:17.625711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":60,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:21.164366+0000","last_change":"2024-02-19T21:03:21.164366+0000","last_active":"2024-02-19T21:03:21.164366+0000","last_peered":"2024-02-19T21:03:21.164366+0000","last_clean":"2024-02-19T21:03:21.164366+0000","last_became_active":"2024-02-19T21:01:16.273545+0000","last_became_peered":"2024-02-19T21:01:16.273545+0000","last_unstale":"2024-02-19T21:03:21.164366+0000","last_undegraded":"2024-02-19T21:03:21.164366+0000","last_fullsized":"2024-02-19T21:03:21.164366+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:03:21.164286+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":61,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390469+0000","last_change":"2024-02-19T21:03:08.046722+0000","last_active":"2024-02-19T21:03:11.390469+0000","last_peered":"2024-02-19T21:03:11.390469+0000","last_clean":"2024-02-19T21:03:11.390469+0000","last_became_active":"2024-02-19T21:01:18.285672+0000","last_became_peered":"2024-02-19T21:01:18.285672+0000","last_unstale":"2024-02-19T21:03:11.390469+0000","last_undegraded":"2024-02-19T21:03:11.390469+0000","last_fullsized":"2024-02-19T21:03:11.390469+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:03:08.046641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.046641+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.432296+0000","last_change":"2024-02-19T21:02:35.252642+0000","last_active":"2024-02-19T21:03:12.432296+0000","last_peered":"2024-02-19T21:03:12.432296+0000","last_clean":"2024-02-19T21:03:12.432296+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:03:12.432296+0000","last_undegraded":"2024-02-19T21:03:12.432296+0000","last_fullsized":"2024-02-19T21:03:12.432296+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:35.252569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:35.252569+0000","last_clean_scrub_stamp":"2024-02-19T21:02:35.252569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.392809+0000","last_change":"2024-02-19T21:02:47.593210+0000","last_active":"2024-02-19T21:03:11.392809+0000","last_peered":"2024-02-19T21:03:11.392809+0000","last_clean":"2024-02-19T21:03:11.392809+0000","last_became_active":"2024-02-19T21:01:22.292133+0000","last_became_peered":"2024-02-19T21:01:22.292133+0000","last_unstale":"2024-02-19T21:03:11.392809+0000","last_undegraded":"2024-02-19T21:03:11.392809+0000","last_fullsized":"2024-02-19T21:03:11.392809+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.593132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.593132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445932+0000","last_change":"2024-02-19T21:03:01.530211+0000","last_active":"2024-02-19T21:03:12.445932+0000","last_peered":"2024-02-19T21:03:12.445932+0000","last_clean":"2024-02-19T21:03:12.445932+0000","last_became_active":"2024-02-19T21:01:30.926676+0000","last_became_peered":"2024-02-19T21:01:30.926676+0000","last_unstale":"2024-02-19T21:03:12.445932+0000","last_undegraded":"2024-02-19T21:03:12.445932+0000","last_fullsized":"2024-02-19T21:03:12.445932+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.530108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":53,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445772+0000","last_change":"2024-02-19T21:02:56.539408+0000","last_active":"2024-02-19T21:03:12.445772+0000","last_peered":"2024-02-19T21:03:12.445772+0000","last_clean":"2024-02-19T21:03:12.445772+0000","last_became_active":"2024-02-19T21:01:18.285891+0000","last_became_peered":"2024-02-19T21:01:18.285891+0000","last_unstale":"2024-02-19T21:03:12.445772+0000","last_undegraded":"2024-02-19T21:03:12.445772+0000","last_fullsized":"2024-02-19T21:03:12.445772+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:02:56.539303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.539303+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":73,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.541280+0000","last_change":"2024-02-19T21:03:10.626933+0000","last_active":"2024-02-19T21:03:20.541280+0000","last_peered":"2024-02-19T21:03:20.541280+0000","last_clean":"2024-02-19T21:03:20.541280+0000","last_became_active":"2024-02-19T21:01:20.288918+0000","last_became_peered":"2024-02-19T21:01:20.288918+0000","last_unstale":"2024-02-19T21:03:20.541280+0000","last_undegraded":"2024-02-19T21:03:20.541280+0000","last_fullsized":"2024-02-19T21:03:20.541280+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:10.626866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:10.626866+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.734351+0000","last_change":"2024-02-19T21:03:12.734312+0000","last_active":"2024-02-19T21:03:12.734351+0000","last_peered":"2024-02-19T21:03:12.734351+0000","last_clean":"2024-02-19T21:03:12.734351+0000","last_became_active":"2024-02-19T21:01:22.291396+0000","last_became_peered":"2024-02-19T21:01:22.291396+0000","last_unstale":"2024-02-19T21:03:12.734351+0000","last_undegraded":"2024-02-19T21:03:12.734351+0000","last_fullsized":"2024-02-19T21:03:12.734351+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:12.734219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:12.734219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079597+0000","last_change":"2024-02-19T21:02:58.986556+0000","last_active":"2024-02-19T21:03:12.079597+0000","last_peered":"2024-02-19T21:03:12.079597+0000","last_clean":"2024-02-19T21:03:12.079597+0000","last_became_active":"2024-02-19T21:01:16.270159+0000","last_became_peered":"2024-02-19T21:01:16.270159+0000","last_unstale":"2024-02-19T21:03:12.079597+0000","last_undegraded":"2024-02-19T21:03:12.079597+0000","last_fullsized":"2024-02-19T21:03:12.079597+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:58.986458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:58.986458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":112,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.541578+0000","last_change":"2024-02-19T21:02:46.558811+0000","last_active":"2024-02-19T21:03:20.541578+0000","last_peered":"2024-02-19T21:03:20.541578+0000","last_clean":"2024-02-19T21:03:20.541578+0000","last_became_active":"2024-02-19T21:01:18.289076+0000","last_became_peered":"2024-02-19T21:01:18.289076+0000","last_unstale":"2024-02-19T21:03:20.541578+0000","last_undegraded":"2024-02-19T21:03:20.541578+0000","last_fullsized":"2024-02-19T21:03:20.541578+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-19T21:02:46.558739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.558739+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393332+0000","last_change":"2024-02-19T21:02:56.510539+0000","last_active":"2024-02-19T21:03:11.393332+0000","last_peered":"2024-02-19T21:03:11.393332+0000","last_clean":"2024-02-19T21:03:11.393332+0000","last_became_active":"2024-02-19T21:01:20.290758+0000","last_became_peered":"2024-02-19T21:01:20.290758+0000","last_unstale":"2024-02-19T21:03:11.393332+0000","last_undegraded":"2024-02-19T21:03:11.393332+0000","last_fullsized":"2024-02-19T21:03:11.393332+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.510474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.597024+0000","last_change":"2024-02-19T21:02:32.488666+0000","last_active":"2024-02-19T21:03:11.597024+0000","last_peered":"2024-02-19T21:03:11.597024+0000","last_clean":"2024-02-19T21:03:11.597024+0000","last_became_active":"2024-02-19T21:01:22.607883+0000","last_became_peered":"2024-02-19T21:01:22.607883+0000","last_unstale":"2024-02-19T21:03:11.597024+0000","last_undegraded":"2024-02-19T21:03:11.597024+0000","last_fullsized":"2024-02-19T21:03:11.597024+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:32.488584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.488584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":165,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:23.747720+0000","last_change":"2024-02-19T21:02:55.568674+0000","last_active":"2024-02-19T21:03:23.747720+0000","last_peered":"2024-02-19T21:03:23.747720+0000","last_clean":"2024-02-19T21:03:23.747720+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:03:23.747720+0000","last_undegraded":"2024-02-19T21:03:23.747720+0000","last_fullsized":"2024-02-19T21:03:23.747720+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:02:55.568593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.568593+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":122,"num_read_kb":117,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.433155+0000","last_change":"2024-02-19T21:02:31.357109+0000","last_active":"2024-02-19T21:03:12.433155+0000","last_peered":"2024-02-19T21:03:12.433155+0000","last_clean":"2024-02-19T21:03:12.433155+0000","last_became_active":"2024-02-19T21:01:16.276032+0000","last_became_peered":"2024-02-19T21:01:16.276032+0000","last_unstale":"2024-02-19T21:03:12.433155+0000","last_undegraded":"2024-02-19T21:03:12.433155+0000","last_fullsized":"2024-02-19T21:03:12.433155+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:31.357041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:31.357041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.433447+0000","last_change":"2024-02-19T21:02:22.217326+0000","last_active":"2024-02-19T21:03:12.433447+0000","last_peered":"2024-02-19T21:03:12.433447+0000","last_clean":"2024-02-19T21:03:12.433447+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:03:12.433447+0000","last_undegraded":"2024-02-19T21:03:12.433447+0000","last_fullsized":"2024-02-19T21:03:12.433447+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-02-19T21:02:22.217222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:22.217222+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.432565+0000","last_change":"2024-02-19T21:02:45.222033+0000","last_active":"2024-02-19T21:03:12.432565+0000","last_peered":"2024-02-19T21:03:12.432565+0000","last_clean":"2024-02-19T21:03:12.432565+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:03:12.432565+0000","last_undegraded":"2024-02-19T21:03:12.432565+0000","last_fullsized":"2024-02-19T21:03:12.432565+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:45.221965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:45.221965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.080285+0000","last_change":"2024-02-19T21:02:25.326504+0000","last_active":"2024-02-19T21:03:12.080285+0000","last_peered":"2024-02-19T21:03:12.080285+0000","last_clean":"2024-02-19T21:03:12.080285+0000","last_became_active":"2024-02-19T21:01:22.292299+0000","last_became_peered":"2024-02-19T21:01:22.292299+0000","last_unstale":"2024-02-19T21:03:12.080285+0000","last_undegraded":"2024-02-19T21:03:12.080285+0000","last_fullsized":"2024-02-19T21:03:12.080285+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:25.326441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:25.326441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:19.665735+0000","last_change":"2024-02-19T21:03:19.665710+0000","last_active":"2024-02-19T21:03:19.665735+0000","last_peered":"2024-02-19T21:03:19.665735+0000","last_clean":"2024-02-19T21:03:19.665735+0000","last_became_active":"2024-02-19T21:01:16.274290+0000","last_became_peered":"2024-02-19T21:01:16.274290+0000","last_unstale":"2024-02-19T21:03:19.665735+0000","last_undegraded":"2024-02-19T21:03:19.665735+0000","last_fullsized":"2024-02-19T21:03:19.665735+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:19.665646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:19.665646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079670+0000","last_change":"2024-02-19T21:02:52.986989+0000","last_active":"2024-02-19T21:03:12.079670+0000","last_peered":"2024-02-19T21:03:12.079670+0000","last_clean":"2024-02-19T21:03:12.079670+0000","last_became_active":"2024-02-19T21:01:16.276807+0000","last_became_peered":"2024-02-19T21:01:16.276807+0000","last_unstale":"2024-02-19T21:03:12.079670+0000","last_undegraded":"2024-02-19T21:03:12.079670+0000","last_fullsized":"2024-02-19T21:03:12.079670+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.986913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.986913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.598756+0000","last_change":"2024-02-19T21:03:11.598713+0000","last_active":"2024-02-19T21:03:11.598756+0000","last_peered":"2024-02-19T21:03:11.598756+0000","last_clean":"2024-02-19T21:03:11.598756+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:03:11.598756+0000","last_undegraded":"2024-02-19T21:03:11.598756+0000","last_fullsized":"2024-02-19T21:03:11.598756+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:11.598603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:03:11.598603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":60,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.599400+0000","last_change":"2024-02-19T21:02:32.777906+0000","last_active":"2024-02-19T21:03:11.599400+0000","last_peered":"2024-02-19T21:03:11.599400+0000","last_clean":"2024-02-19T21:03:11.599400+0000","last_became_active":"2024-02-19T21:01:18.293784+0000","last_became_peered":"2024-02-19T21:01:18.293784+0000","last_unstale":"2024-02-19T21:03:11.599400+0000","last_undegraded":"2024-02-19T21:03:11.599400+0000","last_fullsized":"2024-02-19T21:03:11.599400+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:02:32.777827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.777827+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596822+0000","last_change":"2024-02-19T21:02:55.586944+0000","last_active":"2024-02-19T21:03:11.596822+0000","last_peered":"2024-02-19T21:03:11.596822+0000","last_clean":"2024-02-19T21:03:11.596822+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:03:11.596822+0000","last_undegraded":"2024-02-19T21:03:11.596822+0000","last_fullsized":"2024-02-19T21:03:11.596822+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:55.586854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.586854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445711+0000","last_change":"2024-02-19T21:02:52.556438+0000","last_active":"2024-02-19T21:03:12.445711+0000","last_peered":"2024-02-19T21:03:12.445711+0000","last_clean":"2024-02-19T21:03:12.445711+0000","last_became_active":"2024-02-19T21:01:22.298477+0000","last_became_peered":"2024-02-19T21:01:22.298477+0000","last_unstale":"2024-02-19T21:03:12.445711+0000","last_undegraded":"2024-02-19T21:03:12.445711+0000","last_fullsized":"2024-02-19T21:03:12.445711+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.556344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.556344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393070+0000","last_change":"2024-02-19T21:02:57.505093+0000","last_active":"2024-02-19T21:03:11.393070+0000","last_peered":"2024-02-19T21:03:11.393070+0000","last_clean":"2024-02-19T21:03:11.393070+0000","last_became_active":"2024-02-19T21:01:16.274398+0000","last_became_peered":"2024-02-19T21:01:16.274398+0000","last_unstale":"2024-02-19T21:03:11.393070+0000","last_undegraded":"2024-02-19T21:03:11.393070+0000","last_fullsized":"2024-02-19T21:03:11.393070+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.505029+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":46,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596209+0000","last_change":"2024-02-19T21:02:41.616831+0000","last_active":"2024-02-19T21:03:11.596209+0000","last_peered":"2024-02-19T21:03:11.596209+0000","last_clean":"2024-02-19T21:03:11.596209+0000","last_became_active":"2024-02-19T21:01:16.275904+0000","last_became_peered":"2024-02-19T21:01:16.275904+0000","last_unstale":"2024-02-19T21:03:11.596209+0000","last_undegraded":"2024-02-19T21:03:11.596209+0000","last_fullsized":"2024-02-19T21:03:11.596209+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:02:41.616729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.616729+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":61,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:22.495828+0000","last_change":"2024-02-19T21:03:22.495828+0000","last_active":"2024-02-19T21:03:22.495828+0000","last_peered":"2024-02-19T21:03:22.495828+0000","last_clean":"2024-02-19T21:03:22.495828+0000","last_became_active":"2024-02-19T21:01:18.287139+0000","last_became_peered":"2024-02-19T21:01:18.287139+0000","last_unstale":"2024-02-19T21:03:22.495828+0000","last_undegraded":"2024-02-19T21:03:22.495828+0000","last_fullsized":"2024-02-19T21:03:22.495828+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:03:22.495754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:22.495754+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.080207+0000","last_change":"2024-02-19T21:03:05.005360+0000","last_active":"2024-02-19T21:03:12.080207+0000","last_peered":"2024-02-19T21:03:12.080207+0000","last_clean":"2024-02-19T21:03:12.080207+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:03:12.080207+0000","last_undegraded":"2024-02-19T21:03:12.080207+0000","last_fullsized":"2024-02-19T21:03:12.080207+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.005288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.005288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445481+0000","last_change":"2024-02-19T21:03:07.473156+0000","last_active":"2024-02-19T21:03:12.445481+0000","last_peered":"2024-02-19T21:03:12.445481+0000","last_clean":"2024-02-19T21:03:12.445481+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:03:12.445481+0000","last_undegraded":"2024-02-19T21:03:12.445481+0000","last_fullsized":"2024-02-19T21:03:12.445481+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:03:07.473070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.597080+0000","last_change":"2024-02-19T21:02:46.565447+0000","last_active":"2024-02-19T21:03:11.597080+0000","last_peered":"2024-02-19T21:03:11.597080+0000","last_clean":"2024-02-19T21:03:11.597080+0000","last_became_active":"2024-02-19T21:01:16.274215+0000","last_became_peered":"2024-02-19T21:01:16.274215+0000","last_unstale":"2024-02-19T21:03:11.597080+0000","last_undegraded":"2024-02-19T21:03:11.597080+0000","last_fullsized":"2024-02-19T21:03:11.597080+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:46.565380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.565380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":74,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393257+0000","last_change":"2024-02-19T21:02:35.502975+0000","last_active":"2024-02-19T21:03:11.393257+0000","last_peered":"2024-02-19T21:03:11.393257+0000","last_clean":"2024-02-19T21:03:11.393257+0000","last_became_active":"2024-02-19T21:01:18.293791+0000","last_became_peered":"2024-02-19T21:01:18.293791+0000","last_unstale":"2024-02-19T21:03:11.393257+0000","last_undegraded":"2024-02-19T21:03:11.393257+0000","last_fullsized":"2024-02-19T21:03:11.393257+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:02:35.502906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:35.502906+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.598846+0000","last_change":"2024-02-19T21:02:27.761781+0000","last_active":"2024-02-19T21:03:11.598846+0000","last_peered":"2024-02-19T21:03:11.598846+0000","last_clean":"2024-02-19T21:03:11.598846+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:03:11.598846+0000","last_undegraded":"2024-02-19T21:03:11.598846+0000","last_fullsized":"2024-02-19T21:03:11.598846+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:27.761685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:27.761685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:17.088819+0000","last_change":"2024-02-19T21:03:17.088777+0000","last_active":"2024-02-19T21:03:17.088819+0000","last_peered":"2024-02-19T21:03:17.088819+0000","last_clean":"2024-02-19T21:03:17.088819+0000","last_became_active":"2024-02-19T21:01:22.297707+0000","last_became_peered":"2024-02-19T21:01:22.297707+0000","last_unstale":"2024-02-19T21:03:17.088819+0000","last_undegraded":"2024-02-19T21:03:17.088819+0000","last_fullsized":"2024-02-19T21:03:17.088819+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:17.088695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:17.088695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.446215+0000","last_change":"2024-02-19T21:02:47.601190+0000","last_active":"2024-02-19T21:03:12.446215+0000","last_peered":"2024-02-19T21:03:12.446215+0000","last_clean":"2024-02-19T21:03:12.446215+0000","last_became_active":"2024-02-19T21:01:16.271340+0000","last_became_peered":"2024-02-19T21:01:16.271340+0000","last_unstale":"2024-02-19T21:03:12.446215+0000","last_undegraded":"2024-02-19T21:03:12.446215+0000","last_fullsized":"2024-02-19T21:03:12.446215+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.601127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.601127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":70,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.395532+0000","last_change":"2024-02-19T21:02:39.952691+0000","last_active":"2024-02-19T21:03:11.395532+0000","last_peered":"2024-02-19T21:03:11.395532+0000","last_clean":"2024-02-19T21:03:11.395532+0000","last_became_active":"2024-02-19T21:01:18.293812+0000","last_became_peered":"2024-02-19T21:01:18.293812+0000","last_unstale":"2024-02-19T21:03:11.395532+0000","last_undegraded":"2024-02-19T21:03:11.395532+0000","last_fullsized":"2024-02-19T21:03:11.395532+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:02:39.952624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.952624+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390581+0000","last_change":"2024-02-19T21:02:57.997535+0000","last_active":"2024-02-19T21:03:11.390581+0000","last_peered":"2024-02-19T21:03:11.390581+0000","last_clean":"2024-02-19T21:03:11.390581+0000","last_became_active":"2024-02-19T21:01:20.291079+0000","last_became_peered":"2024-02-19T21:01:20.291079+0000","last_unstale":"2024-02-19T21:03:11.390581+0000","last_undegraded":"2024-02-19T21:03:11.390581+0000","last_fullsized":"2024-02-19T21:03:11.390581+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.997427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596883+0000","last_change":"2024-02-19T21:03:01.496926+0000","last_active":"2024-02-19T21:03:11.596883+0000","last_peered":"2024-02-19T21:03:11.596883+0000","last_clean":"2024-02-19T21:03:11.596883+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:03:11.596883+0000","last_undegraded":"2024-02-19T21:03:11.596883+0000","last_fullsized":"2024-02-19T21:03:11.596883+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.496833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.496833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.599402+0000","last_change":"2024-02-19T21:02:50.921031+0000","last_active":"2024-02-19T21:03:11.599402+0000","last_peered":"2024-02-19T21:03:11.599402+0000","last_clean":"2024-02-19T21:03:11.599402+0000","last_became_active":"2024-02-19T21:01:16.274462+0000","last_became_peered":"2024-02-19T21:01:16.274462+0000","last_unstale":"2024-02-19T21:03:11.599402+0000","last_undegraded":"2024-02-19T21:03:11.599402+0000","last_fullsized":"2024-02-19T21:03:11.599402+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:50.920948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:50.920948+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":59,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390461+0000","last_change":"2024-02-19T21:02:40.105227+0000","last_active":"2024-02-19T21:03:11.390461+0000","last_peered":"2024-02-19T21:03:11.390461+0000","last_clean":"2024-02-19T21:03:11.390461+0000","last_became_active":"2024-02-19T21:01:18.282702+0000","last_became_peered":"2024-02-19T21:01:18.282702+0000","last_unstale":"2024-02-19T21:03:11.390461+0000","last_undegraded":"2024-02-19T21:03:11.390461+0000","last_fullsized":"2024-02-19T21:03:11.390461+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:02:40.105141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.105141+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.425397+0000","last_change":"2024-02-19T21:02:21.923048+0000","last_active":"2024-02-19T21:03:20.425397+0000","last_peered":"2024-02-19T21:03:20.425397+0000","last_clean":"2024-02-19T21:03:20.425397+0000","last_became_active":"2024-02-19T21:01:20.286505+0000","last_became_peered":"2024-02-19T21:01:20.286505+0000","last_unstale":"2024-02-19T21:03:20.425397+0000","last_undegraded":"2024-02-19T21:03:20.425397+0000","last_fullsized":"2024-02-19T21:03:20.425397+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:21.922971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.922971+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:23.656785+0000","last_change":"2024-02-19T21:03:23.656785+0000","last_active":"2024-02-19T21:03:23.656785+0000","last_peered":"2024-02-19T21:03:23.656785+0000","last_clean":"2024-02-19T21:03:23.656785+0000","last_became_active":"2024-02-19T21:01:22.607007+0000","last_became_peered":"2024-02-19T21:01:22.607007+0000","last_unstale":"2024-02-19T21:03:23.656785+0000","last_undegraded":"2024-02-19T21:03:23.656785+0000","last_fullsized":"2024-02-19T21:03:23.656785+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:23.656688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:23.656688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.080129+0000","last_change":"2024-02-19T21:03:06.991276+0000","last_active":"2024-02-19T21:03:12.080129+0000","last_peered":"2024-02-19T21:03:12.080129+0000","last_clean":"2024-02-19T21:03:12.080129+0000","last_became_active":"2024-02-19T21:01:20.289375+0000","last_became_peered":"2024-02-19T21:01:20.289375+0000","last_unstale":"2024-02-19T21:03:12.080129+0000","last_undegraded":"2024-02-19T21:03:12.080129+0000","last_fullsized":"2024-02-19T21:03:12.080129+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.991199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.991199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596317+0000","last_change":"2024-02-19T21:02:27.567995+0000","last_active":"2024-02-19T21:03:11.596317+0000","last_peered":"2024-02-19T21:03:11.596317+0000","last_clean":"2024-02-19T21:03:11.596317+0000","last_became_active":"2024-02-19T21:01:16.275520+0000","last_became_peered":"2024-02-19T21:01:16.275520+0000","last_unstale":"2024-02-19T21:03:11.596317+0000","last_undegraded":"2024-02-19T21:03:11.596317+0000","last_fullsized":"2024-02-19T21:03:11.596317+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:27.567903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:27.567903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":57,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390969+0000","last_change":"2024-02-19T21:02:19.042725+0000","last_active":"2024-02-19T21:03:11.390969+0000","last_peered":"2024-02-19T21:03:11.390969+0000","last_clean":"2024-02-19T21:03:11.390969+0000","last_became_active":"2024-02-19T21:01:18.285670+0000","last_became_peered":"2024-02-19T21:01:18.285670+0000","last_unstale":"2024-02-19T21:03:11.390969+0000","last_undegraded":"2024-02-19T21:03:11.390969+0000","last_fullsized":"2024-02-19T21:03:11.390969+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:19.042650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:19.042650+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.080111+0000","last_change":"2024-02-19T21:02:41.195082+0000","last_active":"2024-02-19T21:03:12.080111+0000","last_peered":"2024-02-19T21:03:12.080111+0000","last_clean":"2024-02-19T21:03:12.080111+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:03:12.080111+0000","last_undegraded":"2024-02-19T21:03:12.080111+0000","last_fullsized":"2024-02-19T21:03:12.080111+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:41.194993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.194993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.543028+0000","last_change":"2024-02-19T21:02:48.027160+0000","last_active":"2024-02-19T21:03:20.543028+0000","last_peered":"2024-02-19T21:03:20.543028+0000","last_clean":"2024-02-19T21:03:20.543028+0000","last_became_active":"2024-02-19T21:01:20.293341+0000","last_became_peered":"2024-02-19T21:01:20.293341+0000","last_unstale":"2024-02-19T21:03:20.543028+0000","last_undegraded":"2024-02-19T21:03:20.543028+0000","last_fullsized":"2024-02-19T21:03:20.543028+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:48.027070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.027070+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445665+0000","last_change":"2024-02-19T21:02:28.471214+0000","last_active":"2024-02-19T21:03:12.445665+0000","last_peered":"2024-02-19T21:03:12.445665+0000","last_clean":"2024-02-19T21:03:12.445665+0000","last_became_active":"2024-02-19T21:01:16.274939+0000","last_became_peered":"2024-02-19T21:01:16.274939+0000","last_unstale":"2024-02-19T21:03:12.445665+0000","last_undegraded":"2024-02-19T21:03:12.445665+0000","last_fullsized":"2024-02-19T21:03:12.445665+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:28.471131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:28.471131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":63,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.446231+0000","last_change":"2024-02-19T21:02:53.526211+0000","last_active":"2024-02-19T21:03:12.446231+0000","last_peered":"2024-02-19T21:03:12.446231+0000","last_clean":"2024-02-19T21:03:12.446231+0000","last_became_active":"2024-02-19T21:01:18.285534+0000","last_became_peered":"2024-02-19T21:01:18.285534+0000","last_unstale":"2024-02-19T21:03:12.446231+0000","last_undegraded":"2024-02-19T21:03:12.446231+0000","last_fullsized":"2024-02-19T21:03:12.446231+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:53.526141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:53.526141+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390780+0000","last_change":"2024-02-19T21:02:42.123367+0000","last_active":"2024-02-19T21:03:11.390780+0000","last_peered":"2024-02-19T21:03:11.390780+0000","last_clean":"2024-02-19T21:03:11.390780+0000","last_became_active":"2024-02-19T21:01:22.299993+0000","last_became_peered":"2024-02-19T21:01:22.299993+0000","last_unstale":"2024-02-19T21:03:11.390780+0000","last_undegraded":"2024-02-19T21:03:11.390780+0000","last_fullsized":"2024-02-19T21:03:11.390780+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:42.123299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:42.123299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.423430+0000","last_change":"2024-02-19T21:02:25.255767+0000","last_active":"2024-02-19T21:03:20.423430+0000","last_peered":"2024-02-19T21:03:20.423430+0000","last_clean":"2024-02-19T21:03:20.423430+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:03:20.423430+0000","last_undegraded":"2024-02-19T21:03:20.423430+0000","last_fullsized":"2024-02-19T21:03:20.423430+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:25.255648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:25.255648+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445534+0000","last_change":"2024-02-19T21:02:42.525447+0000","last_active":"2024-02-19T21:03:12.445534+0000","last_peered":"2024-02-19T21:03:12.445534+0000","last_clean":"2024-02-19T21:03:12.445534+0000","last_became_active":"2024-02-19T21:01:16.274086+0000","last_became_peered":"2024-02-19T21:01:16.274086+0000","last_unstale":"2024-02-19T21:03:12.445534+0000","last_undegraded":"2024-02-19T21:03:12.445534+0000","last_fullsized":"2024-02-19T21:03:12.445534+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:42.525345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:42.525345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"82'15","reported_seq":58,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390563+0000","last_change":"2024-02-19T21:03:04.086368+0000","last_active":"2024-02-19T21:03:11.390563+0000","last_peered":"2024-02-19T21:03:11.390563+0000","last_clean":"2024-02-19T21:03:11.390563+0000","last_became_active":"2024-02-19T21:01:18.282805+0000","last_became_peered":"2024-02-19T21:01:18.282805+0000","last_unstale":"2024-02-19T21:03:11.390563+0000","last_undegraded":"2024-02-19T21:03:11.390563+0000","last_fullsized":"2024-02-19T21:03:11.390563+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.086282+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.433544+0000","last_change":"2024-02-19T21:02:54.197783+0000","last_active":"2024-02-19T21:03:12.433544+0000","last_peered":"2024-02-19T21:03:12.433544+0000","last_clean":"2024-02-19T21:03:12.433544+0000","last_became_active":"2024-02-19T21:01:22.298941+0000","last_became_peered":"2024-02-19T21:01:22.298941+0000","last_unstale":"2024-02-19T21:03:12.433544+0000","last_undegraded":"2024-02-19T21:03:12.433544+0000","last_fullsized":"2024-02-19T21:03:12.433544+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:54.197714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:54.197714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445046+0000","last_change":"2024-02-19T21:02:21.497900+0000","last_active":"2024-02-19T21:03:12.445046+0000","last_peered":"2024-02-19T21:03:12.445046+0000","last_clean":"2024-02-19T21:03:12.445046+0000","last_became_active":"2024-02-19T21:01:20.290687+0000","last_became_peered":"2024-02-19T21:01:20.290687+0000","last_unstale":"2024-02-19T21:03:12.445046+0000","last_undegraded":"2024-02-19T21:03:12.445046+0000","last_fullsized":"2024-02-19T21:03:12.445046+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.497831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.497831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596492+0000","last_change":"2024-02-19T21:02:40.628725+0000","last_active":"2024-02-19T21:03:11.596492+0000","last_peered":"2024-02-19T21:03:11.596492+0000","last_clean":"2024-02-19T21:03:11.596492+0000","last_became_active":"2024-02-19T21:01:16.274937+0000","last_became_peered":"2024-02-19T21:01:16.274937+0000","last_unstale":"2024-02-19T21:03:11.596492+0000","last_undegraded":"2024-02-19T21:03:11.596492+0000","last_fullsized":"2024-02-19T21:03:11.596492+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:02:40.628632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.628632+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":93,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.391135+0000","last_change":"2024-02-19T21:02:30.070342+0000","last_active":"2024-02-19T21:03:11.391135+0000","last_peered":"2024-02-19T21:03:11.391135+0000","last_clean":"2024-02-19T21:03:11.391135+0000","last_became_active":"2024-02-19T21:01:30.923214+0000","last_became_peered":"2024-02-19T21:01:30.923214+0000","last_unstale":"2024-02-19T21:03:11.391135+0000","last_undegraded":"2024-02-19T21:03:11.391135+0000","last_fullsized":"2024-02-19T21:03:11.391135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:02:30.070236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:30.070236+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596138+0000","last_change":"2024-02-19T21:03:03.500049+0000","last_active":"2024-02-19T21:03:11.596138+0000","last_peered":"2024-02-19T21:03:11.596138+0000","last_clean":"2024-02-19T21:03:11.596138+0000","last_became_active":"2024-02-19T21:01:22.291759+0000","last_became_peered":"2024-02-19T21:01:22.291759+0000","last_unstale":"2024-02-19T21:03:11.596138+0000","last_undegraded":"2024-02-19T21:03:11.596138+0000","last_fullsized":"2024-02-19T21:03:11.596138+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:03.499960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:03.499960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.432286+0000","last_change":"2024-02-19T21:02:55.213761+0000","last_active":"2024-02-19T21:03:12.432286+0000","last_peered":"2024-02-19T21:03:12.432286+0000","last_clean":"2024-02-19T21:03:12.432286+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:03:12.432286+0000","last_undegraded":"2024-02-19T21:03:12.432286+0000","last_fullsized":"2024-02-19T21:03:12.432286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:55.213692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.213692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.433353+0000","last_change":"2024-02-19T21:02:21.212907+0000","last_active":"2024-02-19T21:03:12.433353+0000","last_peered":"2024-02-19T21:03:12.433353+0000","last_clean":"2024-02-19T21:03:12.433353+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:03:12.433353+0000","last_undegraded":"2024-02-19T21:03:12.433353+0000","last_fullsized":"2024-02-19T21:03:12.433353+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.212808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.212808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":69,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.598706+0000","last_change":"2024-02-19T21:02:39.779974+0000","last_active":"2024-02-19T21:03:11.598706+0000","last_peered":"2024-02-19T21:03:11.598706+0000","last_clean":"2024-02-19T21:03:11.598706+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:03:11.598706+0000","last_undegraded":"2024-02-19T21:03:11.598706+0000","last_fullsized":"2024-02-19T21:03:11.598706+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:02:39.779890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.779890+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393062+0000","last_change":"2024-02-19T21:02:48.572976+0000","last_active":"2024-02-19T21:03:11.393062+0000","last_peered":"2024-02-19T21:03:11.393062+0000","last_clean":"2024-02-19T21:03:11.393062+0000","last_became_active":"2024-02-19T21:01:22.297998+0000","last_became_peered":"2024-02-19T21:01:22.297998+0000","last_unstale":"2024-02-19T21:03:11.393062+0000","last_undegraded":"2024-02-19T21:03:11.393062+0000","last_fullsized":"2024-02-19T21:03:11.393062+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:48.572877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.572877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.432286+0000","last_change":"2024-02-19T21:03:01.169350+0000","last_active":"2024-02-19T21:03:12.432286+0000","last_peered":"2024-02-19T21:03:12.432286+0000","last_clean":"2024-02-19T21:03:12.432286+0000","last_became_active":"2024-02-19T21:01:20.290566+0000","last_became_peered":"2024-02-19T21:01:20.290566+0000","last_unstale":"2024-02-19T21:03:12.432286+0000","last_undegraded":"2024-02-19T21:03:12.432286+0000","last_fullsized":"2024-02-19T21:03:12.432286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.169279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.169279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.446066+0000","last_change":"2024-02-19T21:02:22.516204+0000","last_active":"2024-02-19T21:03:12.446066+0000","last_peered":"2024-02-19T21:03:12.446066+0000","last_clean":"2024-02-19T21:03:12.446066+0000","last_became_active":"2024-02-19T21:01:16.270931+0000","last_became_peered":"2024-02-19T21:01:16.270931+0000","last_unstale":"2024-02-19T21:03:12.446066+0000","last_undegraded":"2024-02-19T21:03:12.446066+0000","last_fullsized":"2024-02-19T21:03:12.446066+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:22.516135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:22.516135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393417+0000","last_change":"2024-02-19T21:02:52.643899+0000","last_active":"2024-02-19T21:03:11.393417+0000","last_peered":"2024-02-19T21:03:11.393417+0000","last_clean":"2024-02-19T21:03:11.393417+0000","last_became_active":"2024-02-19T21:01:18.288322+0000","last_became_peered":"2024-02-19T21:01:18.288322+0000","last_unstale":"2024-02-19T21:03:11.393417+0000","last_undegraded":"2024-02-19T21:03:11.393417+0000","last_fullsized":"2024-02-19T21:03:11.393417+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:52.643835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.643835+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079517+0000","last_change":"2024-02-19T21:03:10.017843+0000","last_active":"2024-02-19T21:03:12.079517+0000","last_peered":"2024-02-19T21:03:12.079517+0000","last_clean":"2024-02-19T21:03:12.079517+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:03:12.079517+0000","last_undegraded":"2024-02-19T21:03:12.079517+0000","last_fullsized":"2024-02-19T21:03:12.079517+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:10.017764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:10.017764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.435524+0000","last_change":"2024-02-19T21:02:37.258648+0000","last_active":"2024-02-19T21:03:12.435524+0000","last_peered":"2024-02-19T21:03:12.435524+0000","last_clean":"2024-02-19T21:03:12.435524+0000","last_became_active":"2024-02-19T21:01:16.271911+0000","last_became_peered":"2024-02-19T21:01:16.271911+0000","last_unstale":"2024-02-19T21:03:12.435524+0000","last_undegraded":"2024-02-19T21:03:12.435524+0000","last_fullsized":"2024-02-19T21:03:12.435524+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:37.258580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:37.258580+0000","last_clean_scrub_stamp":"2024-02-19T21:02:37.258580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":63,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079403+0000","last_change":"2024-02-19T21:03:01.005530+0000","last_active":"2024-02-19T21:03:12.079403+0000","last_peered":"2024-02-19T21:03:12.079403+0000","last_clean":"2024-02-19T21:03:12.079403+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:03:12.079403+0000","last_undegraded":"2024-02-19T21:03:12.079403+0000","last_fullsized":"2024-02-19T21:03:12.079403+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:03:01.005445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.005445+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":69,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.541255+0000","last_change":"2024-02-19T21:02:44.563371+0000","last_active":"2024-02-19T21:03:20.541255+0000","last_peered":"2024-02-19T21:03:20.541255+0000","last_clean":"2024-02-19T21:03:20.541255+0000","last_became_active":"2024-02-19T21:01:20.288720+0000","last_became_peered":"2024-02-19T21:01:20.288720+0000","last_unstale":"2024-02-19T21:03:20.541255+0000","last_undegraded":"2024-02-19T21:03:20.541255+0000","last_fullsized":"2024-02-19T21:03:20.541255+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:44.563256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:44.563256+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":399,"ondisk_log_size":399,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":122,"num_read_kb":117,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266716,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297712,"kb_used_data":608,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":622592,"data_stored":283441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.502,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.57499999999999996,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63500000000000001,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.28499999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.51300000000000001,"15min":0.52000000000000002},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78200000000000003,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51300000000000001,"15min":0.51900000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.61499999999999999,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.35999999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.499,"15min":0.5},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.313},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.433,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70599999999999996,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.39700000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.39600000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.67800000000000005,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.496,"15min":0.49399999999999999},"min":{"1min":0.38700000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.39000000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.48599999999999999,"15min":0.498},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58199999999999996,"5min":1.373,"15min":1.373},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.55600000000000005}]},{"osd":5,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.46000000000000002,"15min":0.46100000000000002},"min":{"1min":0.31900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66800000000000004,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.47699999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.34699999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.64700000000000002,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.56699999999999995,"15min":0.57599999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76000000000000001,"5min":1.262,"15min":1.262},"last":0.498}]}]},{"osd":6,"up_from":45,"seq":193273528354,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297676,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447476,"statfs":{"total":95995035648,"available":95690215424,"internally_reserved":0,"allocated":593920,"data_stored":281905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.48799999999999999,"15min":0.49199999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.58799999999999997}]},{"osd":1,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.49399999999999999,"15min":0.498},"min":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.754}]},{"osd":2,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51300000000000001,"15min":0.51700000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.49299999999999999,"15min":0.49099999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.624}]},{"osd":3,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.503,"15min":0.497},"min":{"1min":0.38900000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.65800000000000003}]},{"osd":4,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.39300000000000002,"15min":0.375},"min":{"1min":0.29299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.41299999999999998,"15min":0.40300000000000002},"min":{"1min":0.28899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.51100000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.42999999999999999,"15min":0.41199999999999998},"min":{"1min":0.34000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.46300000000000002,"15min":0.44700000000000001},"min":{"1min":0.39900000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.56399999999999995}]},{"osd":7,"last update":"Mon Feb 19 21:03:16 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.38600000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67400000000000004,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65300000000000002,"5min":1.534,"15min":1.534},"last":0.57599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":630784,"data_stored":283532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35899999999999999,"15min":0.34799999999999998},"min":{"1min":0.22500000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.38,"15min":0.35999999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:03:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.40300000000000002,"15min":0.40600000000000003},"min":{"1min":0.27500000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.499,"5min":0.62,"15min":0.62},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.40300000000000002,"15min":0.41499999999999998},"min":{"1min":0.26900000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.52800000000000002,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.313}]},{"osd":3,"last update":"Mon Feb 19 21:02:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.41399999999999998,"15min":0.42399999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.51800000000000002,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.45500000000000002,"15min":0.46300000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58799999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.34699999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68300000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71699999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.33100000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57799999999999996,"15min":0.58399999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72799999999999998,"5min":0.872,"15min":0.872},"last":1.052}]},{"osd":6,"last update":"Mon Feb 19 21:02:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.57699999999999996,"15min":0.58599999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71299999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63,"15min":0.64300000000000002},"min":{"1min":0.44600000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.68100000000000005,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.41299999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:03:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.45700000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.47699999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.747,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.441}]}]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297712,"kb_used_data":608,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":622592,"data_stored":283095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.372,"15min":0.315},"min":{"1min":0.21299999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.46899999999999997,"5min":4.2830000000000004,"15min":4.2830000000000004},"last":0.318},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.33300000000000002,"15min":0.308},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.48299999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.22700000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.39600000000000002,"15min":0.378},"min":{"1min":0.222,"5min":0.219,"15min":0.219},"max":{"1min":0.45300000000000001,"5min":3.121,"15min":3.121},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.42499999999999999,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.497,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.25700000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.47099999999999997,"15min":0.45100000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.503,"5min":4.4470000000000001,"15min":4.4470000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.48399999999999999,"15min":0.46100000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.504,"5min":4.3070000000000004,"15min":4.3070000000000004},"last":0.28199999999999997}]},{"osd":4,"last update":"Mon Feb 19 21:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52600000000000002,"15min":0.51800000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55400000000000005,"15min":0.54000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73999999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.55000000000000004}]},{"osd":5,"last update":"Mon Feb 19 21:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.628,"15min":0.64100000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88300000000000001,"5min":1.581,"15min":1.581},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.78800000000000003,"15min":0.82799999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76900000000000002,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.59399999999999997}]},{"osd":6,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.1499999999999999,"5min":1.381,"15min":1.381},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.79600000000000004,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.50900000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.76900000000000002,"15min":0.79900000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2070000000000001,"5min":4.226,"15min":4.226},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.625,"15min":0.63100000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.218,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.56299999999999994}]}]},{"osd":2,"up_from":18,"seq":77309411380,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297692,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447460,"statfs":{"total":95995035648,"available":95690199040,"internally_reserved":0,"allocated":610304,"data_stored":282683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36199999999999999,"15min":0.35399999999999998},"min":{"1min":0.23000000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.55500000000000005,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.373},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.35399999999999998,"15min":0.34599999999999997},"min":{"1min":0.252,"5min":0.214,"15min":0.214},"max":{"1min":0.40300000000000002,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.41799999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.35299999999999998},"min":{"1min":0.23300000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.432},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40500000000000003,"15min":0.39600000000000002},"min":{"1min":0.28799999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.47199999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:02:40 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.48499999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.46200000000000002,"15min":0.47399999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.56699999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.442}]},{"osd":4,"last update":"Mon Feb 19 21:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.58099999999999996,"15min":0.59999999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71899999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.60899999999999999,"15min":0.64300000000000002},"min":{"1min":0.38400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.74199999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.48699999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:02:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.63500000000000001,"15min":0.65500000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70899999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80300000000000005,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.53000000000000003}]},{"osd":6,"last update":"Mon Feb 19 21:02:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.62,"15min":0.629},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77800000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.624},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.622,"15min":0.63200000000000001},"min":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.76300000000000001,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.57999999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.58899999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182448,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297716,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":634880,"data_stored":282708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.35199999999999998,"15min":0.35399999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.39300000000000002,"15min":0.40300000000000002},"min":{"1min":0.27500000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.51600000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.317}]},{"osd":1,"last update":"Mon Feb 19 21:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.38600000000000001,"15min":0.39600000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.39500000000000002,"15min":0.39900000000000002},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59199999999999997,"5min":1.054,"15min":1.054},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41499999999999998,"15min":0.42199999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56399999999999995,"5min":0.625,"15min":0.625},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.40999999999999998,"15min":0.41899999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.52900000000000003,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.37}]},{"osd":4,"last update":"Mon Feb 19 21:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.41199999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82799999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55700000000000005,"15min":0.56599999999999995},"min":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.83399999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.48299999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55100000000000005,"15min":0.56000000000000005},"min":{"1min":0.38500000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.501}]},{"osd":6,"last update":"Mon Feb 19 21:02:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56699999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85199999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.56799999999999995}]},{"osd":7,"last update":"Mon Feb 19 21:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.39900000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56799999999999995,"15min":0.57099999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.58499999999999996}]}]},{"osd":4,"up_from":31,"seq":133143986220,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297704,"kb_used_data":608,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447448,"statfs":{"total":95995035648,"available":95690186752,"internally_reserved":0,"allocated":622592,"data_stored":282721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.52500000000000002,"15min":0.54800000000000004},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.67800000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.50700000000000001,"15min":0.52300000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.59899999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.439}]},{"osd":1,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.754,"5min":2.149,"15min":2.149},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.502,"15min":0.51900000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57799999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.48599999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.53300000000000003,"15min":0.55700000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66400000000000003,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.50900000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.51700000000000002,"15min":0.52500000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.502,"5min":0.56499999999999995,"15min":0.56999999999999995},"min":{"1min":0.33400000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64200000000000002,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":0.61899999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.41799999999999998,"15min":0.42799999999999999},"min":{"1min":0.26800000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.501,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.29899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.53600000000000003,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.66100000000000003}]},{"osd":6,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.377,"15min":0.36899999999999999},"min":{"1min":0.28699999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.42899999999999999,"15min":0.42199999999999999},"min":{"1min":0.36199999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.63500000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.442,"15min":0.432},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.46800000000000003,"15min":0.45900000000000002},"min":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.64500000000000002}]}]},{"osd":5,"up_from":37,"seq":158913789991,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297680,"kb_used_data":584,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447472,"statfs":{"total":95995035648,"available":95690211328,"internally_reserved":0,"allocated":598016,"data_stored":281176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.66500000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.53400000000000003,"15min":0.54300000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.70199999999999996,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.44800000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52000000000000002,"15min":0.52500000000000002},"min":{"1min":0.38,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.52400000000000002,"15min":0.53200000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.58799999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.53200000000000003,"15min":0.54000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.63200000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.53000000000000003,"15min":0.53500000000000003},"min":{"1min":0.40799999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.65400000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.54800000000000004}]},{"osd":3,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52700000000000002,"15min":0.53500000000000003},"min":{"1min":0.36099999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.57899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.40600000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71199999999999997,"5min":0.747,"15min":0.747},"last":0.39800000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.39200000000000002,"15min":0.38600000000000001},"min":{"1min":0.30099999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.56299999999999994,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.39400000000000002,"15min":0.38400000000000001},"min":{"1min":0.30099999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.433}]},{"osd":6,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42499999999999999,"15min":0.41699999999999998},"min":{"1min":0.311,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.441,"15min":0.432},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.624,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.501}]},{"osd":7,"last update":"Mon Feb 19 21:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.503,"15min":0.502},"min":{"1min":0.35699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.63100000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.50800000000000001,"15min":0.505},"min":{"1min":0.36599999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.66000000000000003,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.56399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:03:26.805 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:03:26.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:24.935816+0000 mgr.y (mgr.14550) 89 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:26.812 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:25.518459+0000 osd.7 (osd.7) 41 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:03:26.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:25.518527+0000 osd.7 (osd.7) 42 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:03:27.075 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:27.571 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:26.070616+0000 mgr.y (mgr.14550) 90 : audit [DBG] from='client.14766 v1:172.21.15.163:0/692656312' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:03:27.571 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:26.447230+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-02-19T21:03:27.571 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:26.668334+0000 osd.3 (osd.3) 49 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:03:27.571 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:26.669790+0000 osd.3 (osd.3) 50 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:03:27.571 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:26.790380+0000 osd.7 (osd.7) 43 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:03:27.571 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:26.793259+0000 osd.7 (osd.7) 44 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:03:27.572 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:26.070616+0000 mgr.y (mgr.14550) 90 : audit [DBG] from='client.14766 v1:172.21.15.163:0/692656312' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:03:27.572 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:26.447230+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-02-19T21:03:27.572 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:26.668334+0000 osd.3 (osd.3) 49 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:03:27.572 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:26.669790+0000 osd.3 (osd.3) 50 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:03:27.572 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:26.790380+0000 osd.7 (osd.7) 43 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:03:27.572 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:26.793259+0000 osd.7 (osd.7) 44 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:03:27.765 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:26.070616+0000 mgr.y (mgr.14550) 90 : audit [DBG] from='client.14766 v1:172.21.15.163:0/692656312' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:03:27.765 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:26.447230+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-02-19T21:03:27.765 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:26.668334+0000 osd.3 (osd.3) 49 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:03:27.765 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:26.669790+0000 osd.3 (osd.3) 50 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:03:27.765 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:26.790380+0000 osd.7 (osd.7) 43 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:03:27.765 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:26.793259+0000 osd.7 (osd.7) 44 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:03:28.063 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:03:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:03:28.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:26.919639+0000 osd.5 (osd.5) 41 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:03:28.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:26.922567+0000 osd.5 (osd.5) 42 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:03:28.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:26.936175+0000 mgr.y (mgr.14550) 91 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:03:28.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:27.283977+0000 osd.4 (osd.4) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:03:28.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:27.284023+0000 osd.4 (osd.4) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:03:28.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:27.452260+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-02-19T21:03:28.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:27.632539+0000 osd.3 (osd.3) 51 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:03:28.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:27.633488+0000 osd.3 (osd.3) 52 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:03:28.643 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:26.919639+0000 osd.5 (osd.5) 41 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:03:28.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:26.922567+0000 osd.5 (osd.5) 42 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:03:28.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:26.936175+0000 mgr.y (mgr.14550) 91 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:03:28.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:27.283977+0000 osd.4 (osd.4) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:03:28.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:27.284023+0000 osd.4 (osd.4) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:03:28.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:27.452260+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-02-19T21:03:28.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:27.632539+0000 osd.3 (osd.3) 51 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:03:28.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:27.633488+0000 osd.3 (osd.3) 52 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:03:28.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:26.919639+0000 osd.5 (osd.5) 41 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:03:28.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:26.922567+0000 osd.5 (osd.5) 42 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:03:28.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:26.936175+0000 mgr.y (mgr.14550) 91 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:03:28.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:27.283977+0000 osd.4 (osd.4) 47 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:03:28.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:27.284023+0000 osd.4 (osd.4) 48 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:03:28.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:27.452260+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-02-19T21:03:28.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:27.632539+0000 osd.3 (osd.3) 51 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:03:28.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:27.633488+0000 osd.3 (osd.3) 52 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:03:28.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.856+0000 7f321cd9e700 1 -- 172.21.15.163:0/3519020921 >> v1:172.21.15.163:6790/0 conn(0x7f3218103020 legacy=0x7f32181054a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:28.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.856+0000 7f321cd9e700 1 -- 172.21.15.163:0/3519020921 shutdown_connections 2024-02-19T21:03:28.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.856+0000 7f321cd9e700 1 -- 172.21.15.163:0/3519020921 >> 172.21.15.163:0/3519020921 conn(0x7f32180f22c0 msgr2=0x7f32180f46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:28.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.856+0000 7f321cd9e700 1 -- 172.21.15.163:0/3519020921 shutdown_connections 2024-02-19T21:03:28.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.856+0000 7f321cd9e700 1 -- 172.21.15.163:0/3519020921 wait complete. 2024-02-19T21:03:28.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.856+0000 7f321cd9e700 1 librados: starting msgr at 2024-02-19T21:03:28.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.856+0000 7f321cd9e700 1 librados: starting objecter 2024-02-19T21:03:28.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.857+0000 7f321cd9e700 1 Processor -- start 2024-02-19T21:03:28.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.857+0000 7f321cd9e700 1 -- start start 2024-02-19T21:03:28.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.857+0000 7f321cd9e700 1 librados: setting wanted keys 2024-02-19T21:03:28.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.857+0000 7f321cd9e700 1 librados: calling monclient init 2024-02-19T21:03:28.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.857+0000 7f321cd9e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3218116110 con 0x7f3218103020 2024-02-19T21:03:28.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.857+0000 7f321cd9e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f32181172b0 con 0x7f32180fc6e0 2024-02-19T21:03:28.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.857+0000 7f321cd9e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f32181184b0 con 0x7f32180ff4e0 2024-02-19T21:03:28.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.857+0000 7f3216ffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f32180ff4e0 0x7f32180f8d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:33742/0 (socket says 172.21.15.163:33742) 2024-02-19T21:03:28.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.857+0000 7f3216ffd700 1 -- 172.21.15.163:0/2467727181 learned_addr learned my addr 172.21.15.163:0/2467727181 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:28.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.857+0000 7f3214ff9700 10 client.?.objecter ms_handle_connect 0x7f32180ff4e0 2024-02-19T21:03:28.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.857+0000 7f3214ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:28.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.857+0000 7f3214ff9700 10 client.?.objecter ms_handle_connect 0x7f3218103020 2024-02-19T21:03:28.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.857+0000 7f3214ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:28.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.857+0000 7f3214ff9700 10 client.?.objecter ms_handle_connect 0x7f32180fc6e0 2024-02-19T21:03:28.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.857+0000 7f3214ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:28.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542957271 0 0) 0x7f3218116110 con 0x7f3218103020 2024-02-19T21:03:28.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3200003030 con 0x7f3218103020 2024-02-19T21:03:28.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296261251 0 0) 0x7f32181184b0 con 0x7f32180ff4e0 2024-02-19T21:03:28.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3218116110 con 0x7f32180ff4e0 2024-02-19T21:03:28.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308360996 0 0) 0x7f32181172b0 con 0x7f32180fc6e0 2024-02-19T21:03:28.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32181184b0 con 0x7f32180fc6e0 2024-02-19T21:03:28.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1191531158 0 0) 0x7f3218116110 con 0x7f32180ff4e0 2024-02-19T21:03:28.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32181172b0 con 0x7f32180ff4e0 2024-02-19T21:03:28.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3882888211 0 0) 0x7f3200003030 con 0x7f3218103020 2024-02-19T21:03:28.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3218116110 con 0x7f3218103020 2024-02-19T21:03:28.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 192180319 0 0) 0x7f32181184b0 con 0x7f32180fc6e0 2024-02-19T21:03:28.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3200003030 con 0x7f32180fc6e0 2024-02-19T21:03:28.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f32080019f0 con 0x7f32180ff4e0 2024-02-19T21:03:28.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3210001ce0 con 0x7f3218103020 2024-02-19T21:03:28.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f320c0017c0 con 0x7f32180fc6e0 2024-02-19T21:03:28.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1169347997 0 0) 0x7f32181172b0 con 0x7f32180ff4e0 2024-02-19T21:03:28.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 >> v1:172.21.15.193:6789/0 conn(0x7f32180fc6e0 legacy=0x7f32180f8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:28.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 >> v1:172.21.15.163:6789/0 conn(0x7f3218103020 legacy=0x7f32181147a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:28.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32181196b0 con 0x7f32180ff4e0 2024-02-19T21:03:28.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f321cd9e700 1 -- 172.21.15.163:0/2467727181 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3218116300 con 0x7f32180ff4e0 2024-02-19T21:03:28.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f321cd9e700 10 client.34694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:28.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f321cd9e700 1 -- 172.21.15.163:0/2467727181 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32181167f0 con 0x7f32180ff4e0 2024-02-19T21:03:28.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f321cd9e700 1 librados: init done 2024-02-19T21:03:28.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3208002460 con 0x7f32180ff4e0 2024-02-19T21:03:28.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.858+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3208005ae0 con 0x7f32180ff4e0 2024-02-19T21:03:28.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.859+0000 7f321cd9e700 1 -- 172.21.15.163:0/2467727181 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3204004f20 con 0x7f32180ff4e0 2024-02-19T21:03:28.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.860+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f3208005c40 con 0x7f32180ff4e0 2024-02-19T21:03:28.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.860+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(86..86 src has 32..86) v4 ==== 6172+0+0 (unknown 3088729427 0 0) 0x7f32080784a0 con 0x7f32180ff4e0 2024-02-19T21:03:28.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.860+0000 7f3214ff9700 10 client.34694.objecter ms_dispatch 0x7f3218001550 osd_map(86..86 src has 32..86) v4 2024-02-19T21:03:28.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.860+0000 7f3214ff9700 3 client.34694.objecter handle_osd_map got epochs [86,86] > 0 2024-02-19T21:03:28.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.860+0000 7f3214ff9700 3 client.34694.objecter handle_osd_map decoding full epoch 86 2024-02-19T21:03:28.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.860+0000 7f3214ff9700 20 client.34694.objecter dump_active .. 0 homeless 2024-02-19T21:03:28.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.860+0000 7f3214ff9700 10 client.34694.objecter ms_handle_connect 0x7f3200065f90 2024-02-19T21:03:28.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:28.864+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f320804c050 con 0x7f32180ff4e0 2024-02-19T21:03:29.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:29.027+0000 7f321cd9e700 1 -- 172.21.15.163:0/2467727181 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3204000b90 con 0x7f3200065f90 2024-02-19T21:03:29.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:29.036+0000 7f3214ff9700 1 -- 172.21.15.163:0/2467727181 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357391 (unknown 2965378022 0 1221581896) 0x7f3204000b90 con 0x7f3200065f90 2024-02-19T21:03:29.033 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:03:29.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:29.039+0000 7f321cd9e700 10 librados: watch_flush enter 2024-02-19T21:03:29.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:29.039+0000 7f321cd9e700 10 librados: watch_flush exit 2024-02-19T21:03:29.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:29.039+0000 7f321cd9e700 20 client.34694.objecter shutdown clearing up homeless session... 2024-02-19T21:03:29.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:29.039+0000 7f321cd9e700 10 client.34694.objecter successfully canceled tick 2024-02-19T21:03:29.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:29.039+0000 7f321cd9e700 1 -- 172.21.15.163:0/2467727181 >> v1:172.21.15.163:6800/1552514681 conn(0x7f3200065f90 legacy=0x7f3200068450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:29.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:29.039+0000 7f321cd9e700 1 -- 172.21.15.163:0/2467727181 >> v1:172.21.15.163:6790/0 conn(0x7f32180ff4e0 legacy=0x7f32180f8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:29.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:29.039+0000 7f321cd9e700 1 -- 172.21.15.163:0/2467727181 shutdown_connections 2024-02-19T21:03:29.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:29.039+0000 7f321cd9e700 1 -- 172.21.15.163:0/2467727181 >> 172.21.15.163:0/2467727181 conn(0x7f32180f22c0 msgr2=0x7f3218106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:29.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:29.039+0000 7f321cd9e700 1 -- 172.21.15.163:0/2467727181 shutdown_connections 2024-02-19T21:03:29.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:29.039+0000 7f321cd9e700 1 -- 172.21.15.163:0/2467727181 wait complete. 2024-02-19T21:03:29.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:29.039+0000 7f321cd9e700 1 librados: shutdown 2024-02-19T21:03:29.037 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:03:29.701 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:27.959624+0000 osd.2 (osd.2) 35 : cluster [DBG] 4.7 deep-scrub starts 2024-02-19T21:03:29.702 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:27.966518+0000 osd.2 (osd.2) 36 : cluster [DBG] 4.7 deep-scrub ok 2024-02-19T21:03:29.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:27.959624+0000 osd.2 (osd.2) 35 : cluster [DBG] 4.7 deep-scrub starts 2024-02-19T21:03:29.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:27.966518+0000 osd.2 (osd.2) 36 : cluster [DBG] 4.7 deep-scrub ok 2024-02-19T21:03:29.721 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-02-19T21:03:28.936351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":841,"num_read_kb":593,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":476,"ondisk_log_size":476,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2381612,"kb_used_data":4820,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579604,"statfs":{"total":767960285184,"available":765521514496,"internally_reserved":0,"allocated":4935680,"data_stored":2261261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001848"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.395740+0000","last_change":"2024-02-19T21:01:22.609338+0000","last_active":"2024-02-19T21:03:11.395740+0000","last_peered":"2024-02-19T21:03:11.395740+0000","last_clean":"2024-02-19T21:03:11.395740+0000","last_became_active":"2024-02-19T21:01:22.609059+0000","last_became_peered":"2024-02-19T21:01:22.609059+0000","last_unstale":"2024-02-19T21:03:11.395740+0000","last_undegraded":"2024-02-19T21:03:11.395740+0000","last_fullsized":"2024-02-19T21:03:11.395740+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:01:21.273400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390200+0000","last_change":"2024-02-19T21:03:05.073746+0000","last_active":"2024-02-19T21:03:11.390200+0000","last_peered":"2024-02-19T21:03:11.390200+0000","last_clean":"2024-02-19T21:03:11.390200+0000","last_became_active":"2024-02-19T21:01:20.289208+0000","last_became_peered":"2024-02-19T21:01:20.289208+0000","last_unstale":"2024-02-19T21:03:11.390200+0000","last_undegraded":"2024-02-19T21:03:11.390200+0000","last_fullsized":"2024-02-19T21:03:11.390200+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.073664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.073664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393353+0000","last_change":"2024-02-19T21:03:04.644754+0000","last_active":"2024-02-19T21:03:11.393353+0000","last_peered":"2024-02-19T21:03:11.393353+0000","last_clean":"2024-02-19T21:03:11.393353+0000","last_became_active":"2024-02-19T21:01:16.278753+0000","last_became_peered":"2024-02-19T21:01:16.278753+0000","last_unstale":"2024-02-19T21:03:11.393353+0000","last_undegraded":"2024-02-19T21:03:11.393353+0000","last_fullsized":"2024-02-19T21:03:11.393353+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.644689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.644689+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":63,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.395829+0000","last_change":"2024-02-19T21:02:48.863931+0000","last_active":"2024-02-19T21:03:11.395829+0000","last_peered":"2024-02-19T21:03:11.395829+0000","last_clean":"2024-02-19T21:03:11.395829+0000","last_became_active":"2024-02-19T21:01:18.283222+0000","last_became_peered":"2024-02-19T21:01:18.283222+0000","last_unstale":"2024-02-19T21:03:11.395829+0000","last_undegraded":"2024-02-19T21:03:11.395829+0000","last_fullsized":"2024-02-19T21:03:11.395829+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:48.863852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.863852+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390362+0000","last_change":"2024-02-19T21:02:23.100006+0000","last_active":"2024-02-19T21:03:11.390362+0000","last_peered":"2024-02-19T21:03:11.390362+0000","last_clean":"2024-02-19T21:03:11.390362+0000","last_became_active":"2024-02-19T21:01:22.293432+0000","last_became_peered":"2024-02-19T21:01:22.293432+0000","last_unstale":"2024-02-19T21:03:11.390362+0000","last_undegraded":"2024-02-19T21:03:11.390362+0000","last_fullsized":"2024-02-19T21:03:11.390362+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:23.099932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:23.099932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445014+0000","last_change":"2024-02-19T21:02:29.480341+0000","last_active":"2024-02-19T21:03:12.445014+0000","last_peered":"2024-02-19T21:03:12.445014+0000","last_clean":"2024-02-19T21:03:12.445014+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:03:12.445014+0000","last_undegraded":"2024-02-19T21:03:12.445014+0000","last_fullsized":"2024-02-19T21:03:12.445014+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:29.480275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:29.480275+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.396465+0000","last_change":"2024-02-19T21:02:20.970293+0000","last_active":"2024-02-19T21:03:11.396465+0000","last_peered":"2024-02-19T21:03:11.396465+0000","last_clean":"2024-02-19T21:03:11.396465+0000","last_became_active":"2024-02-19T21:01:16.272132+0000","last_became_peered":"2024-02-19T21:01:16.272132+0000","last_unstale":"2024-02-19T21:03:11.396465+0000","last_undegraded":"2024-02-19T21:03:11.396465+0000","last_fullsized":"2024-02-19T21:03:11.396465+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:20.970216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:20.970216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:16.047374+0000","last_change":"2024-02-19T21:03:16.047345+0000","last_active":"2024-02-19T21:03:16.047374+0000","last_peered":"2024-02-19T21:03:16.047374+0000","last_clean":"2024-02-19T21:03:16.047374+0000","last_became_active":"2024-02-19T21:01:18.287645+0000","last_became_peered":"2024-02-19T21:01:18.287645+0000","last_unstale":"2024-02-19T21:03:16.047374+0000","last_undegraded":"2024-02-19T21:03:16.047374+0000","last_fullsized":"2024-02-19T21:03:16.047374+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:16.047258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:16.047258+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:13.096766+0000","last_change":"2024-02-19T21:03:13.096737+0000","last_active":"2024-02-19T21:03:13.096766+0000","last_peered":"2024-02-19T21:03:13.096766+0000","last_clean":"2024-02-19T21:03:13.096766+0000","last_became_active":"2024-02-19T21:01:22.293282+0000","last_became_peered":"2024-02-19T21:01:22.293282+0000","last_unstale":"2024-02-19T21:03:13.096766+0000","last_undegraded":"2024-02-19T21:03:13.096766+0000","last_fullsized":"2024-02-19T21:03:13.096766+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:13.096667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:13.096667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596578+0000","last_change":"2024-02-19T21:02:35.546613+0000","last_active":"2024-02-19T21:03:11.596578+0000","last_peered":"2024-02-19T21:03:11.596578+0000","last_clean":"2024-02-19T21:03:11.596578+0000","last_became_active":"2024-02-19T21:01:20.287940+0000","last_became_peered":"2024-02-19T21:01:20.287940+0000","last_unstale":"2024-02-19T21:03:11.596578+0000","last_undegraded":"2024-02-19T21:03:11.596578+0000","last_fullsized":"2024-02-19T21:03:11.596578+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:35.546526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:35.546526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596749+0000","last_change":"2024-02-19T21:02:20.575203+0000","last_active":"2024-02-19T21:03:11.596749+0000","last_peered":"2024-02-19T21:03:11.596749+0000","last_clean":"2024-02-19T21:03:11.596749+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:03:11.596749+0000","last_undegraded":"2024-02-19T21:03:11.596749+0000","last_fullsized":"2024-02-19T21:03:11.596749+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:20.575133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:20.575133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390720+0000","last_change":"2024-02-19T21:02:27.098930+0000","last_active":"2024-02-19T21:03:11.390720+0000","last_peered":"2024-02-19T21:03:11.390720+0000","last_clean":"2024-02-19T21:03:11.390720+0000","last_became_active":"2024-02-19T21:01:18.285735+0000","last_became_peered":"2024-02-19T21:01:18.285735+0000","last_unstale":"2024-02-19T21:03:11.390720+0000","last_undegraded":"2024-02-19T21:03:11.390720+0000","last_fullsized":"2024-02-19T21:03:11.390720+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:02:27.098844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:27.098844+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393117+0000","last_change":"2024-02-19T21:03:00.570766+0000","last_active":"2024-02-19T21:03:11.393117+0000","last_peered":"2024-02-19T21:03:11.393117+0000","last_clean":"2024-02-19T21:03:11.393117+0000","last_became_active":"2024-02-19T21:01:22.292714+0000","last_became_peered":"2024-02-19T21:01:22.292714+0000","last_unstale":"2024-02-19T21:03:11.393117+0000","last_undegraded":"2024-02-19T21:03:11.393117+0000","last_fullsized":"2024-02-19T21:03:11.393117+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:00.570677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:00.570677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079804+0000","last_change":"2024-02-19T21:01:20.293436+0000","last_active":"2024-02-19T21:03:12.079804+0000","last_peered":"2024-02-19T21:03:12.079804+0000","last_clean":"2024-02-19T21:03:12.079804+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:03:12.079804+0000","last_undegraded":"2024-02-19T21:03:12.079804+0000","last_fullsized":"2024-02-19T21:03:12.079804+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:01:19.270373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596676+0000","last_change":"2024-02-19T21:02:57.552979+0000","last_active":"2024-02-19T21:03:11.596676+0000","last_peered":"2024-02-19T21:03:11.596676+0000","last_clean":"2024-02-19T21:03:11.596676+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:03:11.596676+0000","last_undegraded":"2024-02-19T21:03:11.596676+0000","last_fullsized":"2024-02-19T21:03:11.596676+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.552886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.552886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390433+0000","last_change":"2024-02-19T21:02:26.073713+0000","last_active":"2024-02-19T21:03:11.390433+0000","last_peered":"2024-02-19T21:03:11.390433+0000","last_clean":"2024-02-19T21:03:11.390433+0000","last_became_active":"2024-02-19T21:01:18.285269+0000","last_became_peered":"2024-02-19T21:01:18.285269+0000","last_unstale":"2024-02-19T21:03:11.390433+0000","last_undegraded":"2024-02-19T21:03:11.390433+0000","last_fullsized":"2024-02-19T21:03:11.390433+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:26.073625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:26.073625+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.396393+0000","last_change":"2024-02-19T21:02:24.948910+0000","last_active":"2024-02-19T21:03:11.396393+0000","last_peered":"2024-02-19T21:03:11.396393+0000","last_clean":"2024-02-19T21:03:11.396393+0000","last_became_active":"2024-02-19T21:01:22.607099+0000","last_became_peered":"2024-02-19T21:01:22.607099+0000","last_unstale":"2024-02-19T21:03:11.396393+0000","last_undegraded":"2024-02-19T21:03:11.396393+0000","last_fullsized":"2024-02-19T21:03:11.396393+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:24.948832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:24.948832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390940+0000","last_change":"2024-02-19T21:03:09.010420+0000","last_active":"2024-02-19T21:03:11.390940+0000","last_peered":"2024-02-19T21:03:11.390940+0000","last_clean":"2024-02-19T21:03:11.390940+0000","last_became_active":"2024-02-19T21:01:20.291672+0000","last_became_peered":"2024-02-19T21:01:20.291672+0000","last_unstale":"2024-02-19T21:03:11.390940+0000","last_undegraded":"2024-02-19T21:03:11.390940+0000","last_fullsized":"2024-02-19T21:03:11.390940+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:09.010341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:09.010341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.637528+0000","last_change":"2024-02-19T21:03:20.637528+0000","last_active":"2024-02-19T21:03:20.637528+0000","last_peered":"2024-02-19T21:03:20.637528+0000","last_clean":"2024-02-19T21:03:20.637528+0000","last_became_active":"2024-02-19T21:01:16.274811+0000","last_became_peered":"2024-02-19T21:01:16.274811+0000","last_unstale":"2024-02-19T21:03:20.637528+0000","last_undegraded":"2024-02-19T21:03:20.637528+0000","last_fullsized":"2024-02-19T21:03:20.637528+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:20.637465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:20.637465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":61,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.396228+0000","last_change":"2024-02-19T21:02:57.819231+0000","last_active":"2024-02-19T21:03:11.396228+0000","last_peered":"2024-02-19T21:03:11.396228+0000","last_clean":"2024-02-19T21:03:11.396228+0000","last_became_active":"2024-02-19T21:01:18.285926+0000","last_became_peered":"2024-02-19T21:01:18.285926+0000","last_unstale":"2024-02-19T21:03:11.396228+0000","last_undegraded":"2024-02-19T21:03:11.396228+0000","last_fullsized":"2024-02-19T21:03:11.396228+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:02:57.819125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.819125+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.081355+0000","last_change":"2024-02-19T21:03:12.081314+0000","last_active":"2024-02-19T21:03:12.081355+0000","last_peered":"2024-02-19T21:03:12.081355+0000","last_clean":"2024-02-19T21:03:12.081355+0000","last_became_active":"2024-02-19T21:01:22.607435+0000","last_became_peered":"2024-02-19T21:01:22.607435+0000","last_unstale":"2024-02-19T21:03:12.081355+0000","last_undegraded":"2024-02-19T21:03:12.081355+0000","last_fullsized":"2024-02-19T21:03:12.081355+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:03:12.081227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445115+0000","last_change":"2024-02-19T21:02:54.544941+0000","last_active":"2024-02-19T21:03:12.445115+0000","last_peered":"2024-02-19T21:03:12.445115+0000","last_clean":"2024-02-19T21:03:12.445115+0000","last_became_active":"2024-02-19T21:01:20.290705+0000","last_became_peered":"2024-02-19T21:01:20.290705+0000","last_unstale":"2024-02-19T21:03:12.445115+0000","last_undegraded":"2024-02-19T21:03:12.445115+0000","last_fullsized":"2024-02-19T21:03:12.445115+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:54.544834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:54.544834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.598772+0000","last_change":"2024-02-19T21:02:47.805245+0000","last_active":"2024-02-19T21:03:11.598772+0000","last_peered":"2024-02-19T21:03:11.598772+0000","last_clean":"2024-02-19T21:03:11.598772+0000","last_became_active":"2024-02-19T21:01:16.274524+0000","last_became_peered":"2024-02-19T21:01:16.274524+0000","last_unstale":"2024-02-19T21:03:11.598772+0000","last_undegraded":"2024-02-19T21:03:11.598772+0000","last_fullsized":"2024-02-19T21:03:11.598772+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.805171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.805171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":63,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.433438+0000","last_change":"2024-02-19T21:02:47.238547+0000","last_active":"2024-02-19T21:03:12.433438+0000","last_peered":"2024-02-19T21:03:12.433438+0000","last_clean":"2024-02-19T21:03:12.433438+0000","last_became_active":"2024-02-19T21:01:18.289712+0000","last_became_peered":"2024-02-19T21:01:18.289712+0000","last_unstale":"2024-02-19T21:03:12.433438+0000","last_undegraded":"2024-02-19T21:03:12.433438+0000","last_fullsized":"2024-02-19T21:03:12.433438+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:47.238470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.238470+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.446016+0000","last_change":"2024-02-19T21:03:06.456310+0000","last_active":"2024-02-19T21:03:12.446016+0000","last_peered":"2024-02-19T21:03:12.446016+0000","last_clean":"2024-02-19T21:03:12.446016+0000","last_became_active":"2024-02-19T21:01:22.293230+0000","last_became_peered":"2024-02-19T21:01:22.293230+0000","last_unstale":"2024-02-19T21:03:12.446016+0000","last_undegraded":"2024-02-19T21:03:12.446016+0000","last_fullsized":"2024-02-19T21:03:12.446016+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.456212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.456212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393656+0000","last_change":"2024-02-19T21:03:08.636037+0000","last_active":"2024-02-19T21:03:11.393656+0000","last_peered":"2024-02-19T21:03:11.393656+0000","last_clean":"2024-02-19T21:03:11.393656+0000","last_became_active":"2024-02-19T21:01:20.288016+0000","last_became_peered":"2024-02-19T21:01:20.288016+0000","last_unstale":"2024-02-19T21:03:11.393656+0000","last_undegraded":"2024-02-19T21:03:11.393656+0000","last_fullsized":"2024-02-19T21:03:11.393656+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:08.635943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.635943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":40,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:21.619457+0000","last_change":"2024-02-19T21:03:21.619457+0000","last_active":"2024-02-19T21:03:21.619457+0000","last_peered":"2024-02-19T21:03:21.619457+0000","last_clean":"2024-02-19T21:03:21.619457+0000","last_became_active":"2024-02-19T21:01:16.277826+0000","last_became_peered":"2024-02-19T21:01:16.277826+0000","last_unstale":"2024-02-19T21:03:21.619457+0000","last_undegraded":"2024-02-19T21:03:21.619457+0000","last_fullsized":"2024-02-19T21:03:21.619457+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:21.619389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:21.619389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":57,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.599346+0000","last_change":"2024-02-19T21:02:29.684765+0000","last_active":"2024-02-19T21:03:11.599346+0000","last_peered":"2024-02-19T21:03:11.599346+0000","last_clean":"2024-02-19T21:03:11.599346+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:03:11.599346+0000","last_undegraded":"2024-02-19T21:03:11.599346+0000","last_fullsized":"2024-02-19T21:03:11.599346+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:29.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:29.684698+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.597047+0000","last_change":"2024-02-19T21:02:36.585127+0000","last_active":"2024-02-19T21:03:11.597047+0000","last_peered":"2024-02-19T21:03:11.597047+0000","last_clean":"2024-02-19T21:03:11.597047+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:03:11.597047+0000","last_undegraded":"2024-02-19T21:03:11.597047+0000","last_fullsized":"2024-02-19T21:03:11.597047+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:02:36.585007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:36.585007+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":73,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.425426+0000","last_change":"2024-02-19T21:02:37.842634+0000","last_active":"2024-02-19T21:03:20.425426+0000","last_peered":"2024-02-19T21:03:20.425426+0000","last_clean":"2024-02-19T21:03:20.425426+0000","last_became_active":"2024-02-19T21:01:20.290567+0000","last_became_peered":"2024-02-19T21:01:20.290567+0000","last_unstale":"2024-02-19T21:03:20.425426+0000","last_undegraded":"2024-02-19T21:03:20.425426+0000","last_fullsized":"2024-02-19T21:03:20.425426+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:37.842545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:37.842545+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079313+0000","last_change":"2024-02-19T21:03:06.000805+0000","last_active":"2024-02-19T21:03:12.079313+0000","last_peered":"2024-02-19T21:03:12.079313+0000","last_clean":"2024-02-19T21:03:12.079313+0000","last_became_active":"2024-02-19T21:01:16.272284+0000","last_became_peered":"2024-02-19T21:01:16.272284+0000","last_unstale":"2024-02-19T21:03:12.079313+0000","last_undegraded":"2024-02-19T21:03:12.079313+0000","last_fullsized":"2024-02-19T21:03:12.079313+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.000732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.000732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":53,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393617+0000","last_change":"2024-02-19T21:02:32.550264+0000","last_active":"2024-02-19T21:03:11.393617+0000","last_peered":"2024-02-19T21:03:11.393617+0000","last_clean":"2024-02-19T21:03:11.393617+0000","last_became_active":"2024-02-19T21:01:18.293865+0000","last_became_peered":"2024-02-19T21:01:18.293865+0000","last_unstale":"2024-02-19T21:03:11.393617+0000","last_undegraded":"2024-02-19T21:03:11.393617+0000","last_fullsized":"2024-02-19T21:03:11.393617+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:02:32.550190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.550190+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.396000+0000","last_change":"2024-02-19T21:02:41.964875+0000","last_active":"2024-02-19T21:03:11.396000+0000","last_peered":"2024-02-19T21:03:11.396000+0000","last_clean":"2024-02-19T21:03:11.396000+0000","last_became_active":"2024-02-19T21:01:22.608997+0000","last_became_peered":"2024-02-19T21:01:22.608997+0000","last_unstale":"2024-02-19T21:03:11.396000+0000","last_undegraded":"2024-02-19T21:03:11.396000+0000","last_fullsized":"2024-02-19T21:03:11.396000+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:41.964809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.964809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596434+0000","last_change":"2024-02-19T21:03:04.486168+0000","last_active":"2024-02-19T21:03:11.596434+0000","last_peered":"2024-02-19T21:03:11.596434+0000","last_clean":"2024-02-19T21:03:11.596434+0000","last_became_active":"2024-02-19T21:01:20.287961+0000","last_became_peered":"2024-02-19T21:01:20.287961+0000","last_unstale":"2024-02-19T21:03:11.596434+0000","last_undegraded":"2024-02-19T21:03:11.596434+0000","last_fullsized":"2024-02-19T21:03:11.596434+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.486067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.486067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.599433+0000","last_change":"2024-02-19T21:02:43.790280+0000","last_active":"2024-02-19T21:03:11.599433+0000","last_peered":"2024-02-19T21:03:11.599433+0000","last_clean":"2024-02-19T21:03:11.599433+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:03:11.599433+0000","last_undegraded":"2024-02-19T21:03:11.599433+0000","last_fullsized":"2024-02-19T21:03:11.599433+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:43.790206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:43.790206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":57,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.395796+0000","last_change":"2024-02-19T21:02:52.859531+0000","last_active":"2024-02-19T21:03:11.395796+0000","last_peered":"2024-02-19T21:03:11.395796+0000","last_clean":"2024-02-19T21:03:11.395796+0000","last_became_active":"2024-02-19T21:01:18.293793+0000","last_became_peered":"2024-02-19T21:01:18.293793+0000","last_unstale":"2024-02-19T21:03:11.395796+0000","last_undegraded":"2024-02-19T21:03:11.395796+0000","last_fullsized":"2024-02-19T21:03:11.395796+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:52.859453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.859453+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596374+0000","last_change":"2024-02-19T21:02:45.579070+0000","last_active":"2024-02-19T21:03:11.596374+0000","last_peered":"2024-02-19T21:03:11.596374+0000","last_clean":"2024-02-19T21:03:11.596374+0000","last_became_active":"2024-02-19T21:01:22.297693+0000","last_became_peered":"2024-02-19T21:01:22.297693+0000","last_unstale":"2024-02-19T21:03:11.596374+0000","last_undegraded":"2024-02-19T21:03:11.596374+0000","last_fullsized":"2024-02-19T21:03:11.596374+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:45.578978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:45.578978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.598557+0000","last_change":"2024-02-19T21:02:30.724973+0000","last_active":"2024-02-19T21:03:11.598557+0000","last_peered":"2024-02-19T21:03:11.598557+0000","last_clean":"2024-02-19T21:03:11.598557+0000","last_became_active":"2024-02-19T21:01:20.290962+0000","last_became_peered":"2024-02-19T21:01:20.290962+0000","last_unstale":"2024-02-19T21:03:11.598557+0000","last_undegraded":"2024-02-19T21:03:11.598557+0000","last_fullsized":"2024-02-19T21:03:11.598557+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:30.724908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:30.724908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.080134+0000","last_change":"2024-02-19T21:02:52.021686+0000","last_active":"2024-02-19T21:03:12.080134+0000","last_peered":"2024-02-19T21:03:12.080134+0000","last_clean":"2024-02-19T21:03:12.080134+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:03:12.080134+0000","last_undegraded":"2024-02-19T21:03:12.080134+0000","last_fullsized":"2024-02-19T21:03:12.080134+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.021603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.021603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.395598+0000","last_change":"2024-02-19T21:02:22.939275+0000","last_active":"2024-02-19T21:03:11.395598+0000","last_peered":"2024-02-19T21:03:11.395598+0000","last_clean":"2024-02-19T21:03:11.395598+0000","last_became_active":"2024-02-19T21:01:18.287874+0000","last_became_peered":"2024-02-19T21:01:18.287874+0000","last_unstale":"2024-02-19T21:03:11.395598+0000","last_undegraded":"2024-02-19T21:03:11.395598+0000","last_fullsized":"2024-02-19T21:03:11.395598+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:02:22.939192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:22.939192+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.395541+0000","last_change":"2024-02-19T21:03:05.734069+0000","last_active":"2024-02-19T21:03:11.395541+0000","last_peered":"2024-02-19T21:03:11.395541+0000","last_clean":"2024-02-19T21:03:11.395541+0000","last_became_active":"2024-02-19T21:01:22.292704+0000","last_became_peered":"2024-02-19T21:01:22.292704+0000","last_unstale":"2024-02-19T21:03:11.395541+0000","last_undegraded":"2024-02-19T21:03:11.395541+0000","last_fullsized":"2024-02-19T21:03:11.395541+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.733980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.733980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445018+0000","last_change":"2024-02-19T21:02:40.600482+0000","last_active":"2024-02-19T21:03:12.445018+0000","last_peered":"2024-02-19T21:03:12.445018+0000","last_clean":"2024-02-19T21:03:12.445018+0000","last_became_active":"2024-02-19T21:01:20.289020+0000","last_became_peered":"2024-02-19T21:01:20.289020+0000","last_unstale":"2024-02-19T21:03:12.445018+0000","last_undegraded":"2024-02-19T21:03:12.445018+0000","last_fullsized":"2024-02-19T21:03:12.445018+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:40.600410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.600410+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.445580+0000","last_change":"2024-02-19T21:03:20.445543+0000","last_active":"2024-02-19T21:03:20.445580+0000","last_peered":"2024-02-19T21:03:20.445580+0000","last_clean":"2024-02-19T21:03:20.445580+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:03:20.445580+0000","last_undegraded":"2024-02-19T21:03:20.445580+0000","last_fullsized":"2024-02-19T21:03:20.445580+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:20.445477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:20.445477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":57,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390557+0000","last_change":"2024-02-19T21:02:32.040809+0000","last_active":"2024-02-19T21:03:11.390557+0000","last_peered":"2024-02-19T21:03:11.390557+0000","last_clean":"2024-02-19T21:03:11.390557+0000","last_became_active":"2024-02-19T21:01:18.286128+0000","last_became_peered":"2024-02-19T21:01:18.286128+0000","last_unstale":"2024-02-19T21:03:11.390557+0000","last_undegraded":"2024-02-19T21:03:11.390557+0000","last_fullsized":"2024-02-19T21:03:11.390557+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:32.040731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.040731+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.392789+0000","last_change":"2024-02-19T21:02:51.631722+0000","last_active":"2024-02-19T21:03:11.392789+0000","last_peered":"2024-02-19T21:03:11.392789+0000","last_clean":"2024-02-19T21:03:11.392789+0000","last_became_active":"2024-02-19T21:01:22.293093+0000","last_became_peered":"2024-02-19T21:01:22.293093+0000","last_unstale":"2024-02-19T21:03:11.392789+0000","last_undegraded":"2024-02-19T21:03:11.392789+0000","last_fullsized":"2024-02-19T21:03:11.392789+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:51.631657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:51.631657+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.395921+0000","last_change":"2024-02-19T21:03:03.776497+0000","last_active":"2024-02-19T21:03:11.395921+0000","last_peered":"2024-02-19T21:03:11.395921+0000","last_clean":"2024-02-19T21:03:11.395921+0000","last_became_active":"2024-02-19T21:01:20.289691+0000","last_became_peered":"2024-02-19T21:01:20.289691+0000","last_unstale":"2024-02-19T21:03:11.395921+0000","last_undegraded":"2024-02-19T21:03:11.395921+0000","last_fullsized":"2024-02-19T21:03:11.395921+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_clean_scrub_stamp":"2024-02-19T21:03:03.776397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.599152+0000","last_change":"2024-02-19T21:02:21.657425+0000","last_active":"2024-02-19T21:03:11.599152+0000","last_peered":"2024-02-19T21:03:11.599152+0000","last_clean":"2024-02-19T21:03:11.599152+0000","last_became_active":"2024-02-19T21:01:16.271316+0000","last_became_peered":"2024-02-19T21:01:16.271316+0000","last_unstale":"2024-02-19T21:03:11.599152+0000","last_undegraded":"2024-02-19T21:03:11.599152+0000","last_fullsized":"2024-02-19T21:03:11.599152+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.657334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.657334+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":42,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:22.984626+0000","last_change":"2024-02-19T21:03:22.984626+0000","last_active":"2024-02-19T21:03:22.984626+0000","last_peered":"2024-02-19T21:03:22.984626+0000","last_clean":"2024-02-19T21:03:22.984626+0000","last_became_active":"2024-02-19T21:01:30.922577+0000","last_became_peered":"2024-02-19T21:01:30.922577+0000","last_unstale":"2024-02-19T21:03:22.984626+0000","last_undegraded":"2024-02-19T21:03:22.984626+0000","last_fullsized":"2024-02-19T21:03:22.984626+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:22.984547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:22.984547+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.391137+0000","last_change":"2024-02-19T21:02:39.092233+0000","last_active":"2024-02-19T21:03:11.391137+0000","last_peered":"2024-02-19T21:03:11.391137+0000","last_clean":"2024-02-19T21:03:11.391137+0000","last_became_active":"2024-02-19T21:01:22.299969+0000","last_became_peered":"2024-02-19T21:01:22.299969+0000","last_unstale":"2024-02-19T21:03:11.391137+0000","last_undegraded":"2024-02-19T21:03:11.391137+0000","last_fullsized":"2024-02-19T21:03:11.391137+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.092154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":65,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.426058+0000","last_change":"2024-02-19T21:01:20.284282+0000","last_active":"2024-02-19T21:03:20.426058+0000","last_peered":"2024-02-19T21:03:20.426058+0000","last_clean":"2024-02-19T21:03:20.426058+0000","last_became_active":"2024-02-19T21:01:20.283943+0000","last_became_peered":"2024-02-19T21:01:20.283943+0000","last_unstale":"2024-02-19T21:03:20.426058+0000","last_undegraded":"2024-02-19T21:03:20.426058+0000","last_fullsized":"2024-02-19T21:03:20.426058+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:01:19.270373+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393335+0000","last_change":"2024-02-19T21:02:21.693740+0000","last_active":"2024-02-19T21:03:11.393335+0000","last_peered":"2024-02-19T21:03:11.393335+0000","last_clean":"2024-02-19T21:03:11.393335+0000","last_became_active":"2024-02-19T21:01:16.273181+0000","last_became_peered":"2024-02-19T21:01:16.273181+0000","last_unstale":"2024-02-19T21:03:11.393335+0000","last_undegraded":"2024-02-19T21:03:11.393335+0000","last_fullsized":"2024-02-19T21:03:11.393335+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.693675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.693675+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079828+0000","last_change":"2024-02-19T21:02:57.021496+0000","last_active":"2024-02-19T21:03:12.079828+0000","last_peered":"2024-02-19T21:03:12.079828+0000","last_clean":"2024-02-19T21:03:12.079828+0000","last_became_active":"2024-02-19T21:01:18.293786+0000","last_became_peered":"2024-02-19T21:01:18.293786+0000","last_unstale":"2024-02-19T21:03:12.079828+0000","last_undegraded":"2024-02-19T21:03:12.079828+0000","last_fullsized":"2024-02-19T21:03:12.079828+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:57.021403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.021403+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.392931+0000","last_change":"2024-02-19T21:02:39.442600+0000","last_active":"2024-02-19T21:03:11.392931+0000","last_peered":"2024-02-19T21:03:11.392931+0000","last_clean":"2024-02-19T21:03:11.392931+0000","last_became_active":"2024-02-19T21:01:22.297188+0000","last_became_peered":"2024-02-19T21:01:22.297188+0000","last_unstale":"2024-02-19T21:03:11.392931+0000","last_undegraded":"2024-02-19T21:03:11.392931+0000","last_fullsized":"2024-02-19T21:03:11.392931+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:02:39.442533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.442533+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.541980+0000","last_change":"2024-02-19T21:03:11.063302+0000","last_active":"2024-02-19T21:03:20.541980+0000","last_peered":"2024-02-19T21:03:20.541980+0000","last_clean":"2024-02-19T21:03:20.541980+0000","last_became_active":"2024-02-19T21:01:20.288799+0000","last_became_peered":"2024-02-19T21:01:20.288799+0000","last_unstale":"2024-02-19T21:03:20.541980+0000","last_undegraded":"2024-02-19T21:03:20.541980+0000","last_fullsized":"2024-02-19T21:03:20.541980+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:11.063222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:11.063222+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079545+0000","last_change":"2024-02-19T21:02:49.056495+0000","last_active":"2024-02-19T21:03:12.079545+0000","last_peered":"2024-02-19T21:03:12.079545+0000","last_clean":"2024-02-19T21:03:12.079545+0000","last_became_active":"2024-02-19T21:01:16.271608+0000","last_became_peered":"2024-02-19T21:01:16.271608+0000","last_unstale":"2024-02-19T21:03:12.079545+0000","last_undegraded":"2024-02-19T21:03:12.079545+0000","last_fullsized":"2024-02-19T21:03:12.079545+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.056412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.056412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":53,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.396124+0000","last_change":"2024-02-19T21:03:08.742946+0000","last_active":"2024-02-19T21:03:11.396124+0000","last_peered":"2024-02-19T21:03:11.396124+0000","last_clean":"2024-02-19T21:03:11.396124+0000","last_became_active":"2024-02-19T21:01:18.294369+0000","last_became_peered":"2024-02-19T21:01:18.294369+0000","last_unstale":"2024-02-19T21:03:11.396124+0000","last_undegraded":"2024-02-19T21:03:11.396124+0000","last_fullsized":"2024-02-19T21:03:11.396124+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:03:08.742876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.742876+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596968+0000","last_change":"2024-02-19T21:02:38.633030+0000","last_active":"2024-02-19T21:03:11.596968+0000","last_peered":"2024-02-19T21:03:11.596968+0000","last_clean":"2024-02-19T21:03:11.596968+0000","last_became_active":"2024-02-19T21:01:22.608285+0000","last_became_peered":"2024-02-19T21:01:22.608285+0000","last_unstale":"2024-02-19T21:03:11.596968+0000","last_undegraded":"2024-02-19T21:03:11.596968+0000","last_fullsized":"2024-02-19T21:03:11.596968+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:38.632950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:38.632950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.080120+0000","last_change":"2024-02-19T21:03:08.984824+0000","last_active":"2024-02-19T21:03:12.080120+0000","last_peered":"2024-02-19T21:03:12.080120+0000","last_clean":"2024-02-19T21:03:12.080120+0000","last_became_active":"2024-02-19T21:01:20.288589+0000","last_became_peered":"2024-02-19T21:01:20.288589+0000","last_unstale":"2024-02-19T21:03:12.080120+0000","last_undegraded":"2024-02-19T21:03:12.080120+0000","last_fullsized":"2024-02-19T21:03:12.080120+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:08.984720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.984720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.598593+0000","last_change":"2024-02-19T21:02:46.842812+0000","last_active":"2024-02-19T21:03:11.598593+0000","last_peered":"2024-02-19T21:03:11.598593+0000","last_clean":"2024-02-19T21:03:11.598593+0000","last_became_active":"2024-02-19T21:01:16.271210+0000","last_became_peered":"2024-02-19T21:01:16.271210+0000","last_unstale":"2024-02-19T21:03:11.598593+0000","last_undegraded":"2024-02-19T21:03:11.598593+0000","last_fullsized":"2024-02-19T21:03:11.598593+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:46.842732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.842732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.396349+0000","last_change":"2024-02-19T21:02:33.968248+0000","last_active":"2024-02-19T21:03:11.396349+0000","last_peered":"2024-02-19T21:03:11.396349+0000","last_clean":"2024-02-19T21:03:11.396349+0000","last_became_active":"2024-02-19T21:01:18.293913+0000","last_became_peered":"2024-02-19T21:01:18.293913+0000","last_unstale":"2024-02-19T21:03:11.396349+0000","last_undegraded":"2024-02-19T21:03:11.396349+0000","last_fullsized":"2024-02-19T21:03:11.396349+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:02:33.968182+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.433027+0000","last_change":"2024-02-19T21:02:34.274719+0000","last_active":"2024-02-19T21:03:12.433027+0000","last_peered":"2024-02-19T21:03:12.433027+0000","last_clean":"2024-02-19T21:03:12.433027+0000","last_became_active":"2024-02-19T21:01:22.297086+0000","last_became_peered":"2024-02-19T21:01:22.297086+0000","last_unstale":"2024-02-19T21:03:12.433027+0000","last_undegraded":"2024-02-19T21:03:12.433027+0000","last_fullsized":"2024-02-19T21:03:12.433027+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:34.274650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:34.274650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.396431+0000","last_change":"2024-02-19T21:02:49.863142+0000","last_active":"2024-02-19T21:03:11.396431+0000","last_peered":"2024-02-19T21:03:11.396431+0000","last_clean":"2024-02-19T21:03:11.396431+0000","last_became_active":"2024-02-19T21:01:20.285298+0000","last_became_peered":"2024-02-19T21:01:20.285298+0000","last_unstale":"2024-02-19T21:03:11.396431+0000","last_undegraded":"2024-02-19T21:03:11.396431+0000","last_fullsized":"2024-02-19T21:03:11.396431+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.863037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.432866+0000","last_change":"2024-02-19T21:02:49.177055+0000","last_active":"2024-02-19T21:03:12.432866+0000","last_peered":"2024-02-19T21:03:12.432866+0000","last_clean":"2024-02-19T21:03:12.432866+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:03:12.432866+0000","last_undegraded":"2024-02-19T21:03:12.432866+0000","last_fullsized":"2024-02-19T21:03:12.432866+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.176969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.176969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393503+0000","last_change":"2024-02-19T21:03:07.669010+0000","last_active":"2024-02-19T21:03:11.393503+0000","last_peered":"2024-02-19T21:03:11.393503+0000","last_clean":"2024-02-19T21:03:11.393503+0000","last_became_active":"2024-02-19T21:01:18.293773+0000","last_became_peered":"2024-02-19T21:01:18.293773+0000","last_unstale":"2024-02-19T21:03:11.393503+0000","last_undegraded":"2024-02-19T21:03:11.393503+0000","last_fullsized":"2024-02-19T21:03:11.393503+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:07.668943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:07.668943+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:17.625801+0000","last_change":"2024-02-19T21:03:17.625775+0000","last_active":"2024-02-19T21:03:17.625801+0000","last_peered":"2024-02-19T21:03:17.625801+0000","last_clean":"2024-02-19T21:03:17.625801+0000","last_became_active":"2024-02-19T21:01:22.295912+0000","last_became_peered":"2024-02-19T21:01:22.295912+0000","last_unstale":"2024-02-19T21:03:17.625801+0000","last_undegraded":"2024-02-19T21:03:17.625801+0000","last_fullsized":"2024-02-19T21:03:17.625801+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:03:17.625711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":60,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:21.164366+0000","last_change":"2024-02-19T21:03:21.164366+0000","last_active":"2024-02-19T21:03:21.164366+0000","last_peered":"2024-02-19T21:03:21.164366+0000","last_clean":"2024-02-19T21:03:21.164366+0000","last_became_active":"2024-02-19T21:01:16.273545+0000","last_became_peered":"2024-02-19T21:01:16.273545+0000","last_unstale":"2024-02-19T21:03:21.164366+0000","last_undegraded":"2024-02-19T21:03:21.164366+0000","last_fullsized":"2024-02-19T21:03:21.164366+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:03:21.164286+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":61,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390469+0000","last_change":"2024-02-19T21:03:08.046722+0000","last_active":"2024-02-19T21:03:11.390469+0000","last_peered":"2024-02-19T21:03:11.390469+0000","last_clean":"2024-02-19T21:03:11.390469+0000","last_became_active":"2024-02-19T21:01:18.285672+0000","last_became_peered":"2024-02-19T21:01:18.285672+0000","last_unstale":"2024-02-19T21:03:11.390469+0000","last_undegraded":"2024-02-19T21:03:11.390469+0000","last_fullsized":"2024-02-19T21:03:11.390469+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:03:08.046641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.046641+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.432296+0000","last_change":"2024-02-19T21:02:35.252642+0000","last_active":"2024-02-19T21:03:12.432296+0000","last_peered":"2024-02-19T21:03:12.432296+0000","last_clean":"2024-02-19T21:03:12.432296+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:03:12.432296+0000","last_undegraded":"2024-02-19T21:03:12.432296+0000","last_fullsized":"2024-02-19T21:03:12.432296+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:35.252569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:35.252569+0000","last_clean_scrub_stamp":"2024-02-19T21:02:35.252569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.392809+0000","last_change":"2024-02-19T21:02:47.593210+0000","last_active":"2024-02-19T21:03:11.392809+0000","last_peered":"2024-02-19T21:03:11.392809+0000","last_clean":"2024-02-19T21:03:11.392809+0000","last_became_active":"2024-02-19T21:01:22.292133+0000","last_became_peered":"2024-02-19T21:01:22.292133+0000","last_unstale":"2024-02-19T21:03:11.392809+0000","last_undegraded":"2024-02-19T21:03:11.392809+0000","last_fullsized":"2024-02-19T21:03:11.392809+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.593132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.593132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445932+0000","last_change":"2024-02-19T21:03:01.530211+0000","last_active":"2024-02-19T21:03:12.445932+0000","last_peered":"2024-02-19T21:03:12.445932+0000","last_clean":"2024-02-19T21:03:12.445932+0000","last_became_active":"2024-02-19T21:01:30.926676+0000","last_became_peered":"2024-02-19T21:01:30.926676+0000","last_unstale":"2024-02-19T21:03:12.445932+0000","last_undegraded":"2024-02-19T21:03:12.445932+0000","last_fullsized":"2024-02-19T21:03:12.445932+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.530108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":53,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445772+0000","last_change":"2024-02-19T21:02:56.539408+0000","last_active":"2024-02-19T21:03:12.445772+0000","last_peered":"2024-02-19T21:03:12.445772+0000","last_clean":"2024-02-19T21:03:12.445772+0000","last_became_active":"2024-02-19T21:01:18.285891+0000","last_became_peered":"2024-02-19T21:01:18.285891+0000","last_unstale":"2024-02-19T21:03:12.445772+0000","last_undegraded":"2024-02-19T21:03:12.445772+0000","last_fullsized":"2024-02-19T21:03:12.445772+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:02:56.539303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.539303+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":73,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.541280+0000","last_change":"2024-02-19T21:03:10.626933+0000","last_active":"2024-02-19T21:03:20.541280+0000","last_peered":"2024-02-19T21:03:20.541280+0000","last_clean":"2024-02-19T21:03:20.541280+0000","last_became_active":"2024-02-19T21:01:20.288918+0000","last_became_peered":"2024-02-19T21:01:20.288918+0000","last_unstale":"2024-02-19T21:03:20.541280+0000","last_undegraded":"2024-02-19T21:03:20.541280+0000","last_fullsized":"2024-02-19T21:03:20.541280+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:10.626866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:10.626866+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.734351+0000","last_change":"2024-02-19T21:03:12.734312+0000","last_active":"2024-02-19T21:03:12.734351+0000","last_peered":"2024-02-19T21:03:12.734351+0000","last_clean":"2024-02-19T21:03:12.734351+0000","last_became_active":"2024-02-19T21:01:22.291396+0000","last_became_peered":"2024-02-19T21:01:22.291396+0000","last_unstale":"2024-02-19T21:03:12.734351+0000","last_undegraded":"2024-02-19T21:03:12.734351+0000","last_fullsized":"2024-02-19T21:03:12.734351+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:12.734219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:12.734219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079597+0000","last_change":"2024-02-19T21:02:58.986556+0000","last_active":"2024-02-19T21:03:12.079597+0000","last_peered":"2024-02-19T21:03:12.079597+0000","last_clean":"2024-02-19T21:03:12.079597+0000","last_became_active":"2024-02-19T21:01:16.270159+0000","last_became_peered":"2024-02-19T21:01:16.270159+0000","last_unstale":"2024-02-19T21:03:12.079597+0000","last_undegraded":"2024-02-19T21:03:12.079597+0000","last_fullsized":"2024-02-19T21:03:12.079597+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:58.986458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:58.986458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":112,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.541578+0000","last_change":"2024-02-19T21:02:46.558811+0000","last_active":"2024-02-19T21:03:20.541578+0000","last_peered":"2024-02-19T21:03:20.541578+0000","last_clean":"2024-02-19T21:03:20.541578+0000","last_became_active":"2024-02-19T21:01:18.289076+0000","last_became_peered":"2024-02-19T21:01:18.289076+0000","last_unstale":"2024-02-19T21:03:20.541578+0000","last_undegraded":"2024-02-19T21:03:20.541578+0000","last_fullsized":"2024-02-19T21:03:20.541578+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-19T21:02:46.558739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.558739+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393332+0000","last_change":"2024-02-19T21:02:56.510539+0000","last_active":"2024-02-19T21:03:11.393332+0000","last_peered":"2024-02-19T21:03:11.393332+0000","last_clean":"2024-02-19T21:03:11.393332+0000","last_became_active":"2024-02-19T21:01:20.290758+0000","last_became_peered":"2024-02-19T21:01:20.290758+0000","last_unstale":"2024-02-19T21:03:11.393332+0000","last_undegraded":"2024-02-19T21:03:11.393332+0000","last_fullsized":"2024-02-19T21:03:11.393332+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.510474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.597024+0000","last_change":"2024-02-19T21:02:32.488666+0000","last_active":"2024-02-19T21:03:11.597024+0000","last_peered":"2024-02-19T21:03:11.597024+0000","last_clean":"2024-02-19T21:03:11.597024+0000","last_became_active":"2024-02-19T21:01:22.607883+0000","last_became_peered":"2024-02-19T21:01:22.607883+0000","last_unstale":"2024-02-19T21:03:11.597024+0000","last_undegraded":"2024-02-19T21:03:11.597024+0000","last_fullsized":"2024-02-19T21:03:11.597024+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:32.488584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.488584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":165,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:23.747720+0000","last_change":"2024-02-19T21:02:55.568674+0000","last_active":"2024-02-19T21:03:23.747720+0000","last_peered":"2024-02-19T21:03:23.747720+0000","last_clean":"2024-02-19T21:03:23.747720+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:03:23.747720+0000","last_undegraded":"2024-02-19T21:03:23.747720+0000","last_fullsized":"2024-02-19T21:03:23.747720+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:02:55.568593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.568593+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":122,"num_read_kb":117,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.433155+0000","last_change":"2024-02-19T21:02:31.357109+0000","last_active":"2024-02-19T21:03:12.433155+0000","last_peered":"2024-02-19T21:03:12.433155+0000","last_clean":"2024-02-19T21:03:12.433155+0000","last_became_active":"2024-02-19T21:01:16.276032+0000","last_became_peered":"2024-02-19T21:01:16.276032+0000","last_unstale":"2024-02-19T21:03:12.433155+0000","last_undegraded":"2024-02-19T21:03:12.433155+0000","last_fullsized":"2024-02-19T21:03:12.433155+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:31.357041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:31.357041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'13","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.433447+0000","last_change":"2024-02-19T21:02:22.217326+0000","last_active":"2024-02-19T21:03:12.433447+0000","last_peered":"2024-02-19T21:03:12.433447+0000","last_clean":"2024-02-19T21:03:12.433447+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:03:12.433447+0000","last_undegraded":"2024-02-19T21:03:12.433447+0000","last_fullsized":"2024-02-19T21:03:12.433447+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2024-02-19T21:02:22.217222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:22.217222+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.432565+0000","last_change":"2024-02-19T21:02:45.222033+0000","last_active":"2024-02-19T21:03:12.432565+0000","last_peered":"2024-02-19T21:03:12.432565+0000","last_clean":"2024-02-19T21:03:12.432565+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:03:12.432565+0000","last_undegraded":"2024-02-19T21:03:12.432565+0000","last_fullsized":"2024-02-19T21:03:12.432565+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:45.221965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:45.221965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.080285+0000","last_change":"2024-02-19T21:02:25.326504+0000","last_active":"2024-02-19T21:03:12.080285+0000","last_peered":"2024-02-19T21:03:12.080285+0000","last_clean":"2024-02-19T21:03:12.080285+0000","last_became_active":"2024-02-19T21:01:22.292299+0000","last_became_peered":"2024-02-19T21:01:22.292299+0000","last_unstale":"2024-02-19T21:03:12.080285+0000","last_undegraded":"2024-02-19T21:03:12.080285+0000","last_fullsized":"2024-02-19T21:03:12.080285+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:25.326441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:25.326441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:19.665735+0000","last_change":"2024-02-19T21:03:19.665710+0000","last_active":"2024-02-19T21:03:19.665735+0000","last_peered":"2024-02-19T21:03:19.665735+0000","last_clean":"2024-02-19T21:03:19.665735+0000","last_became_active":"2024-02-19T21:01:16.274290+0000","last_became_peered":"2024-02-19T21:01:16.274290+0000","last_unstale":"2024-02-19T21:03:19.665735+0000","last_undegraded":"2024-02-19T21:03:19.665735+0000","last_fullsized":"2024-02-19T21:03:19.665735+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:19.665646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:19.665646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079670+0000","last_change":"2024-02-19T21:02:52.986989+0000","last_active":"2024-02-19T21:03:12.079670+0000","last_peered":"2024-02-19T21:03:12.079670+0000","last_clean":"2024-02-19T21:03:12.079670+0000","last_became_active":"2024-02-19T21:01:16.276807+0000","last_became_peered":"2024-02-19T21:01:16.276807+0000","last_unstale":"2024-02-19T21:03:12.079670+0000","last_undegraded":"2024-02-19T21:03:12.079670+0000","last_fullsized":"2024-02-19T21:03:12.079670+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.986913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.986913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.598756+0000","last_change":"2024-02-19T21:03:11.598713+0000","last_active":"2024-02-19T21:03:11.598756+0000","last_peered":"2024-02-19T21:03:11.598756+0000","last_clean":"2024-02-19T21:03:11.598756+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:03:11.598756+0000","last_undegraded":"2024-02-19T21:03:11.598756+0000","last_fullsized":"2024-02-19T21:03:11.598756+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:11.598603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:03:11.598603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":60,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.599400+0000","last_change":"2024-02-19T21:02:32.777906+0000","last_active":"2024-02-19T21:03:11.599400+0000","last_peered":"2024-02-19T21:03:11.599400+0000","last_clean":"2024-02-19T21:03:11.599400+0000","last_became_active":"2024-02-19T21:01:18.293784+0000","last_became_peered":"2024-02-19T21:01:18.293784+0000","last_unstale":"2024-02-19T21:03:11.599400+0000","last_undegraded":"2024-02-19T21:03:11.599400+0000","last_fullsized":"2024-02-19T21:03:11.599400+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:02:32.777827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.777827+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596822+0000","last_change":"2024-02-19T21:02:55.586944+0000","last_active":"2024-02-19T21:03:11.596822+0000","last_peered":"2024-02-19T21:03:11.596822+0000","last_clean":"2024-02-19T21:03:11.596822+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:03:11.596822+0000","last_undegraded":"2024-02-19T21:03:11.596822+0000","last_fullsized":"2024-02-19T21:03:11.596822+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:55.586854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.586854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445711+0000","last_change":"2024-02-19T21:02:52.556438+0000","last_active":"2024-02-19T21:03:12.445711+0000","last_peered":"2024-02-19T21:03:12.445711+0000","last_clean":"2024-02-19T21:03:12.445711+0000","last_became_active":"2024-02-19T21:01:22.298477+0000","last_became_peered":"2024-02-19T21:01:22.298477+0000","last_unstale":"2024-02-19T21:03:12.445711+0000","last_undegraded":"2024-02-19T21:03:12.445711+0000","last_fullsized":"2024-02-19T21:03:12.445711+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.556344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.556344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393070+0000","last_change":"2024-02-19T21:02:57.505093+0000","last_active":"2024-02-19T21:03:11.393070+0000","last_peered":"2024-02-19T21:03:11.393070+0000","last_clean":"2024-02-19T21:03:11.393070+0000","last_became_active":"2024-02-19T21:01:16.274398+0000","last_became_peered":"2024-02-19T21:01:16.274398+0000","last_unstale":"2024-02-19T21:03:11.393070+0000","last_undegraded":"2024-02-19T21:03:11.393070+0000","last_fullsized":"2024-02-19T21:03:11.393070+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.505029+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":46,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596209+0000","last_change":"2024-02-19T21:02:41.616831+0000","last_active":"2024-02-19T21:03:11.596209+0000","last_peered":"2024-02-19T21:03:11.596209+0000","last_clean":"2024-02-19T21:03:11.596209+0000","last_became_active":"2024-02-19T21:01:16.275904+0000","last_became_peered":"2024-02-19T21:01:16.275904+0000","last_unstale":"2024-02-19T21:03:11.596209+0000","last_undegraded":"2024-02-19T21:03:11.596209+0000","last_fullsized":"2024-02-19T21:03:11.596209+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:02:41.616729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.616729+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":61,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:22.495828+0000","last_change":"2024-02-19T21:03:22.495828+0000","last_active":"2024-02-19T21:03:22.495828+0000","last_peered":"2024-02-19T21:03:22.495828+0000","last_clean":"2024-02-19T21:03:22.495828+0000","last_became_active":"2024-02-19T21:01:18.287139+0000","last_became_peered":"2024-02-19T21:01:18.287139+0000","last_unstale":"2024-02-19T21:03:22.495828+0000","last_undegraded":"2024-02-19T21:03:22.495828+0000","last_fullsized":"2024-02-19T21:03:22.495828+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:03:22.495754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:22.495754+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.080207+0000","last_change":"2024-02-19T21:03:05.005360+0000","last_active":"2024-02-19T21:03:12.080207+0000","last_peered":"2024-02-19T21:03:12.080207+0000","last_clean":"2024-02-19T21:03:12.080207+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:03:12.080207+0000","last_undegraded":"2024-02-19T21:03:12.080207+0000","last_fullsized":"2024-02-19T21:03:12.080207+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.005288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.005288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445481+0000","last_change":"2024-02-19T21:03:07.473156+0000","last_active":"2024-02-19T21:03:12.445481+0000","last_peered":"2024-02-19T21:03:12.445481+0000","last_clean":"2024-02-19T21:03:12.445481+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:03:12.445481+0000","last_undegraded":"2024-02-19T21:03:12.445481+0000","last_fullsized":"2024-02-19T21:03:12.445481+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:03:07.473070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.597080+0000","last_change":"2024-02-19T21:02:46.565447+0000","last_active":"2024-02-19T21:03:11.597080+0000","last_peered":"2024-02-19T21:03:11.597080+0000","last_clean":"2024-02-19T21:03:11.597080+0000","last_became_active":"2024-02-19T21:01:16.274215+0000","last_became_peered":"2024-02-19T21:01:16.274215+0000","last_unstale":"2024-02-19T21:03:11.597080+0000","last_undegraded":"2024-02-19T21:03:11.597080+0000","last_fullsized":"2024-02-19T21:03:11.597080+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:46.565380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.565380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":74,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393257+0000","last_change":"2024-02-19T21:02:35.502975+0000","last_active":"2024-02-19T21:03:11.393257+0000","last_peered":"2024-02-19T21:03:11.393257+0000","last_clean":"2024-02-19T21:03:11.393257+0000","last_became_active":"2024-02-19T21:01:18.293791+0000","last_became_peered":"2024-02-19T21:01:18.293791+0000","last_unstale":"2024-02-19T21:03:11.393257+0000","last_undegraded":"2024-02-19T21:03:11.393257+0000","last_fullsized":"2024-02-19T21:03:11.393257+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:02:35.502906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:35.502906+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.598846+0000","last_change":"2024-02-19T21:02:27.761781+0000","last_active":"2024-02-19T21:03:11.598846+0000","last_peered":"2024-02-19T21:03:11.598846+0000","last_clean":"2024-02-19T21:03:11.598846+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:03:11.598846+0000","last_undegraded":"2024-02-19T21:03:11.598846+0000","last_fullsized":"2024-02-19T21:03:11.598846+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:27.761685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:27.761685+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:17.088819+0000","last_change":"2024-02-19T21:03:17.088777+0000","last_active":"2024-02-19T21:03:17.088819+0000","last_peered":"2024-02-19T21:03:17.088819+0000","last_clean":"2024-02-19T21:03:17.088819+0000","last_became_active":"2024-02-19T21:01:22.297707+0000","last_became_peered":"2024-02-19T21:01:22.297707+0000","last_unstale":"2024-02-19T21:03:17.088819+0000","last_undegraded":"2024-02-19T21:03:17.088819+0000","last_fullsized":"2024-02-19T21:03:17.088819+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:17.088695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:17.088695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.446215+0000","last_change":"2024-02-19T21:02:47.601190+0000","last_active":"2024-02-19T21:03:12.446215+0000","last_peered":"2024-02-19T21:03:12.446215+0000","last_clean":"2024-02-19T21:03:12.446215+0000","last_became_active":"2024-02-19T21:01:16.271340+0000","last_became_peered":"2024-02-19T21:01:16.271340+0000","last_unstale":"2024-02-19T21:03:12.446215+0000","last_undegraded":"2024-02-19T21:03:12.446215+0000","last_fullsized":"2024-02-19T21:03:12.446215+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.601127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.601127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":70,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.395532+0000","last_change":"2024-02-19T21:02:39.952691+0000","last_active":"2024-02-19T21:03:11.395532+0000","last_peered":"2024-02-19T21:03:11.395532+0000","last_clean":"2024-02-19T21:03:11.395532+0000","last_became_active":"2024-02-19T21:01:18.293812+0000","last_became_peered":"2024-02-19T21:01:18.293812+0000","last_unstale":"2024-02-19T21:03:11.395532+0000","last_undegraded":"2024-02-19T21:03:11.395532+0000","last_fullsized":"2024-02-19T21:03:11.395532+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:02:39.952624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.952624+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390581+0000","last_change":"2024-02-19T21:02:57.997535+0000","last_active":"2024-02-19T21:03:11.390581+0000","last_peered":"2024-02-19T21:03:11.390581+0000","last_clean":"2024-02-19T21:03:11.390581+0000","last_became_active":"2024-02-19T21:01:20.291079+0000","last_became_peered":"2024-02-19T21:01:20.291079+0000","last_unstale":"2024-02-19T21:03:11.390581+0000","last_undegraded":"2024-02-19T21:03:11.390581+0000","last_fullsized":"2024-02-19T21:03:11.390581+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.997427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596883+0000","last_change":"2024-02-19T21:03:01.496926+0000","last_active":"2024-02-19T21:03:11.596883+0000","last_peered":"2024-02-19T21:03:11.596883+0000","last_clean":"2024-02-19T21:03:11.596883+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:03:11.596883+0000","last_undegraded":"2024-02-19T21:03:11.596883+0000","last_fullsized":"2024-02-19T21:03:11.596883+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.496833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.496833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.599402+0000","last_change":"2024-02-19T21:02:50.921031+0000","last_active":"2024-02-19T21:03:11.599402+0000","last_peered":"2024-02-19T21:03:11.599402+0000","last_clean":"2024-02-19T21:03:11.599402+0000","last_became_active":"2024-02-19T21:01:16.274462+0000","last_became_peered":"2024-02-19T21:01:16.274462+0000","last_unstale":"2024-02-19T21:03:11.599402+0000","last_undegraded":"2024-02-19T21:03:11.599402+0000","last_fullsized":"2024-02-19T21:03:11.599402+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:50.920948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:50.920948+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":59,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390461+0000","last_change":"2024-02-19T21:02:40.105227+0000","last_active":"2024-02-19T21:03:11.390461+0000","last_peered":"2024-02-19T21:03:11.390461+0000","last_clean":"2024-02-19T21:03:11.390461+0000","last_became_active":"2024-02-19T21:01:18.282702+0000","last_became_peered":"2024-02-19T21:01:18.282702+0000","last_unstale":"2024-02-19T21:03:11.390461+0000","last_undegraded":"2024-02-19T21:03:11.390461+0000","last_fullsized":"2024-02-19T21:03:11.390461+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:02:40.105141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.105141+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.425397+0000","last_change":"2024-02-19T21:02:21.923048+0000","last_active":"2024-02-19T21:03:20.425397+0000","last_peered":"2024-02-19T21:03:20.425397+0000","last_clean":"2024-02-19T21:03:20.425397+0000","last_became_active":"2024-02-19T21:01:20.286505+0000","last_became_peered":"2024-02-19T21:01:20.286505+0000","last_unstale":"2024-02-19T21:03:20.425397+0000","last_undegraded":"2024-02-19T21:03:20.425397+0000","last_fullsized":"2024-02-19T21:03:20.425397+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:21.922971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.922971+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:23.656785+0000","last_change":"2024-02-19T21:03:23.656785+0000","last_active":"2024-02-19T21:03:23.656785+0000","last_peered":"2024-02-19T21:03:23.656785+0000","last_clean":"2024-02-19T21:03:23.656785+0000","last_became_active":"2024-02-19T21:01:22.607007+0000","last_became_peered":"2024-02-19T21:01:22.607007+0000","last_unstale":"2024-02-19T21:03:23.656785+0000","last_undegraded":"2024-02-19T21:03:23.656785+0000","last_fullsized":"2024-02-19T21:03:23.656785+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:23.656688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:23.656688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.080129+0000","last_change":"2024-02-19T21:03:06.991276+0000","last_active":"2024-02-19T21:03:12.080129+0000","last_peered":"2024-02-19T21:03:12.080129+0000","last_clean":"2024-02-19T21:03:12.080129+0000","last_became_active":"2024-02-19T21:01:20.289375+0000","last_became_peered":"2024-02-19T21:01:20.289375+0000","last_unstale":"2024-02-19T21:03:12.080129+0000","last_undegraded":"2024-02-19T21:03:12.080129+0000","last_fullsized":"2024-02-19T21:03:12.080129+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.991199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.991199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596317+0000","last_change":"2024-02-19T21:02:27.567995+0000","last_active":"2024-02-19T21:03:11.596317+0000","last_peered":"2024-02-19T21:03:11.596317+0000","last_clean":"2024-02-19T21:03:11.596317+0000","last_became_active":"2024-02-19T21:01:16.275520+0000","last_became_peered":"2024-02-19T21:01:16.275520+0000","last_unstale":"2024-02-19T21:03:11.596317+0000","last_undegraded":"2024-02-19T21:03:11.596317+0000","last_fullsized":"2024-02-19T21:03:11.596317+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:27.567903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:27.567903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":57,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390969+0000","last_change":"2024-02-19T21:02:19.042725+0000","last_active":"2024-02-19T21:03:11.390969+0000","last_peered":"2024-02-19T21:03:11.390969+0000","last_clean":"2024-02-19T21:03:11.390969+0000","last_became_active":"2024-02-19T21:01:18.285670+0000","last_became_peered":"2024-02-19T21:01:18.285670+0000","last_unstale":"2024-02-19T21:03:11.390969+0000","last_undegraded":"2024-02-19T21:03:11.390969+0000","last_fullsized":"2024-02-19T21:03:11.390969+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:19.042650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:19.042650+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.080111+0000","last_change":"2024-02-19T21:02:41.195082+0000","last_active":"2024-02-19T21:03:12.080111+0000","last_peered":"2024-02-19T21:03:12.080111+0000","last_clean":"2024-02-19T21:03:12.080111+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:03:12.080111+0000","last_undegraded":"2024-02-19T21:03:12.080111+0000","last_fullsized":"2024-02-19T21:03:12.080111+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:41.194993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.194993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.543028+0000","last_change":"2024-02-19T21:02:48.027160+0000","last_active":"2024-02-19T21:03:20.543028+0000","last_peered":"2024-02-19T21:03:20.543028+0000","last_clean":"2024-02-19T21:03:20.543028+0000","last_became_active":"2024-02-19T21:01:20.293341+0000","last_became_peered":"2024-02-19T21:01:20.293341+0000","last_unstale":"2024-02-19T21:03:20.543028+0000","last_undegraded":"2024-02-19T21:03:20.543028+0000","last_fullsized":"2024-02-19T21:03:20.543028+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:48.027070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.027070+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445665+0000","last_change":"2024-02-19T21:02:28.471214+0000","last_active":"2024-02-19T21:03:12.445665+0000","last_peered":"2024-02-19T21:03:12.445665+0000","last_clean":"2024-02-19T21:03:12.445665+0000","last_became_active":"2024-02-19T21:01:16.274939+0000","last_became_peered":"2024-02-19T21:01:16.274939+0000","last_unstale":"2024-02-19T21:03:12.445665+0000","last_undegraded":"2024-02-19T21:03:12.445665+0000","last_fullsized":"2024-02-19T21:03:12.445665+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:28.471131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:28.471131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":63,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.446231+0000","last_change":"2024-02-19T21:02:53.526211+0000","last_active":"2024-02-19T21:03:12.446231+0000","last_peered":"2024-02-19T21:03:12.446231+0000","last_clean":"2024-02-19T21:03:12.446231+0000","last_became_active":"2024-02-19T21:01:18.285534+0000","last_became_peered":"2024-02-19T21:01:18.285534+0000","last_unstale":"2024-02-19T21:03:12.446231+0000","last_undegraded":"2024-02-19T21:03:12.446231+0000","last_fullsized":"2024-02-19T21:03:12.446231+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:53.526141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:53.526141+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390780+0000","last_change":"2024-02-19T21:02:42.123367+0000","last_active":"2024-02-19T21:03:11.390780+0000","last_peered":"2024-02-19T21:03:11.390780+0000","last_clean":"2024-02-19T21:03:11.390780+0000","last_became_active":"2024-02-19T21:01:22.299993+0000","last_became_peered":"2024-02-19T21:01:22.299993+0000","last_unstale":"2024-02-19T21:03:11.390780+0000","last_undegraded":"2024-02-19T21:03:11.390780+0000","last_fullsized":"2024-02-19T21:03:11.390780+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:42.123299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:42.123299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":70,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.423430+0000","last_change":"2024-02-19T21:02:25.255767+0000","last_active":"2024-02-19T21:03:20.423430+0000","last_peered":"2024-02-19T21:03:20.423430+0000","last_clean":"2024-02-19T21:03:20.423430+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:03:20.423430+0000","last_undegraded":"2024-02-19T21:03:20.423430+0000","last_fullsized":"2024-02-19T21:03:20.423430+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:25.255648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:25.255648+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445534+0000","last_change":"2024-02-19T21:02:42.525447+0000","last_active":"2024-02-19T21:03:12.445534+0000","last_peered":"2024-02-19T21:03:12.445534+0000","last_clean":"2024-02-19T21:03:12.445534+0000","last_became_active":"2024-02-19T21:01:16.274086+0000","last_became_peered":"2024-02-19T21:01:16.274086+0000","last_unstale":"2024-02-19T21:03:12.445534+0000","last_undegraded":"2024-02-19T21:03:12.445534+0000","last_fullsized":"2024-02-19T21:03:12.445534+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:42.525345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:42.525345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"82'15","reported_seq":58,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.390563+0000","last_change":"2024-02-19T21:03:04.086368+0000","last_active":"2024-02-19T21:03:11.390563+0000","last_peered":"2024-02-19T21:03:11.390563+0000","last_clean":"2024-02-19T21:03:11.390563+0000","last_became_active":"2024-02-19T21:01:18.282805+0000","last_became_peered":"2024-02-19T21:01:18.282805+0000","last_unstale":"2024-02-19T21:03:11.390563+0000","last_undegraded":"2024-02-19T21:03:11.390563+0000","last_fullsized":"2024-02-19T21:03:11.390563+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.086282+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.433544+0000","last_change":"2024-02-19T21:02:54.197783+0000","last_active":"2024-02-19T21:03:12.433544+0000","last_peered":"2024-02-19T21:03:12.433544+0000","last_clean":"2024-02-19T21:03:12.433544+0000","last_became_active":"2024-02-19T21:01:22.298941+0000","last_became_peered":"2024-02-19T21:01:22.298941+0000","last_unstale":"2024-02-19T21:03:12.433544+0000","last_undegraded":"2024-02-19T21:03:12.433544+0000","last_fullsized":"2024-02-19T21:03:12.433544+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:54.197714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:54.197714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.445046+0000","last_change":"2024-02-19T21:02:21.497900+0000","last_active":"2024-02-19T21:03:12.445046+0000","last_peered":"2024-02-19T21:03:12.445046+0000","last_clean":"2024-02-19T21:03:12.445046+0000","last_became_active":"2024-02-19T21:01:20.290687+0000","last_became_peered":"2024-02-19T21:01:20.290687+0000","last_unstale":"2024-02-19T21:03:12.445046+0000","last_undegraded":"2024-02-19T21:03:12.445046+0000","last_fullsized":"2024-02-19T21:03:12.445046+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.497831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.497831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596492+0000","last_change":"2024-02-19T21:02:40.628725+0000","last_active":"2024-02-19T21:03:11.596492+0000","last_peered":"2024-02-19T21:03:11.596492+0000","last_clean":"2024-02-19T21:03:11.596492+0000","last_became_active":"2024-02-19T21:01:16.274937+0000","last_became_peered":"2024-02-19T21:01:16.274937+0000","last_unstale":"2024-02-19T21:03:11.596492+0000","last_undegraded":"2024-02-19T21:03:11.596492+0000","last_fullsized":"2024-02-19T21:03:11.596492+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:02:40.628632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.628632+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":93,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.391135+0000","last_change":"2024-02-19T21:02:30.070342+0000","last_active":"2024-02-19T21:03:11.391135+0000","last_peered":"2024-02-19T21:03:11.391135+0000","last_clean":"2024-02-19T21:03:11.391135+0000","last_became_active":"2024-02-19T21:01:30.923214+0000","last_became_peered":"2024-02-19T21:01:30.923214+0000","last_unstale":"2024-02-19T21:03:11.391135+0000","last_undegraded":"2024-02-19T21:03:11.391135+0000","last_fullsized":"2024-02-19T21:03:11.391135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:02:30.070236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:30.070236+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.596138+0000","last_change":"2024-02-19T21:03:03.500049+0000","last_active":"2024-02-19T21:03:11.596138+0000","last_peered":"2024-02-19T21:03:11.596138+0000","last_clean":"2024-02-19T21:03:11.596138+0000","last_became_active":"2024-02-19T21:01:22.291759+0000","last_became_peered":"2024-02-19T21:01:22.291759+0000","last_unstale":"2024-02-19T21:03:11.596138+0000","last_undegraded":"2024-02-19T21:03:11.596138+0000","last_fullsized":"2024-02-19T21:03:11.596138+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:03.499960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:03.499960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.432286+0000","last_change":"2024-02-19T21:02:55.213761+0000","last_active":"2024-02-19T21:03:12.432286+0000","last_peered":"2024-02-19T21:03:12.432286+0000","last_clean":"2024-02-19T21:03:12.432286+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:03:12.432286+0000","last_undegraded":"2024-02-19T21:03:12.432286+0000","last_fullsized":"2024-02-19T21:03:12.432286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:55.213692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.213692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.433353+0000","last_change":"2024-02-19T21:02:21.212907+0000","last_active":"2024-02-19T21:03:12.433353+0000","last_peered":"2024-02-19T21:03:12.433353+0000","last_clean":"2024-02-19T21:03:12.433353+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:03:12.433353+0000","last_undegraded":"2024-02-19T21:03:12.433353+0000","last_fullsized":"2024-02-19T21:03:12.433353+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.212808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.212808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":69,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.598706+0000","last_change":"2024-02-19T21:02:39.779974+0000","last_active":"2024-02-19T21:03:11.598706+0000","last_peered":"2024-02-19T21:03:11.598706+0000","last_clean":"2024-02-19T21:03:11.598706+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:03:11.598706+0000","last_undegraded":"2024-02-19T21:03:11.598706+0000","last_fullsized":"2024-02-19T21:03:11.598706+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:02:39.779890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.779890+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393062+0000","last_change":"2024-02-19T21:02:48.572976+0000","last_active":"2024-02-19T21:03:11.393062+0000","last_peered":"2024-02-19T21:03:11.393062+0000","last_clean":"2024-02-19T21:03:11.393062+0000","last_became_active":"2024-02-19T21:01:22.297998+0000","last_became_peered":"2024-02-19T21:01:22.297998+0000","last_unstale":"2024-02-19T21:03:11.393062+0000","last_undegraded":"2024-02-19T21:03:11.393062+0000","last_fullsized":"2024-02-19T21:03:11.393062+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:48.572877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.572877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.432286+0000","last_change":"2024-02-19T21:03:01.169350+0000","last_active":"2024-02-19T21:03:12.432286+0000","last_peered":"2024-02-19T21:03:12.432286+0000","last_clean":"2024-02-19T21:03:12.432286+0000","last_became_active":"2024-02-19T21:01:20.290566+0000","last_became_peered":"2024-02-19T21:01:20.290566+0000","last_unstale":"2024-02-19T21:03:12.432286+0000","last_undegraded":"2024-02-19T21:03:12.432286+0000","last_fullsized":"2024-02-19T21:03:12.432286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.169279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.169279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.446066+0000","last_change":"2024-02-19T21:02:22.516204+0000","last_active":"2024-02-19T21:03:12.446066+0000","last_peered":"2024-02-19T21:03:12.446066+0000","last_clean":"2024-02-19T21:03:12.446066+0000","last_became_active":"2024-02-19T21:01:16.270931+0000","last_became_peered":"2024-02-19T21:01:16.270931+0000","last_unstale":"2024-02-19T21:03:12.446066+0000","last_undegraded":"2024-02-19T21:03:12.446066+0000","last_fullsized":"2024-02-19T21:03:12.446066+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:22.516135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:22.516135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:11.393417+0000","last_change":"2024-02-19T21:02:52.643899+0000","last_active":"2024-02-19T21:03:11.393417+0000","last_peered":"2024-02-19T21:03:11.393417+0000","last_clean":"2024-02-19T21:03:11.393417+0000","last_became_active":"2024-02-19T21:01:18.288322+0000","last_became_peered":"2024-02-19T21:01:18.288322+0000","last_unstale":"2024-02-19T21:03:11.393417+0000","last_undegraded":"2024-02-19T21:03:11.393417+0000","last_fullsized":"2024-02-19T21:03:11.393417+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:52.643835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.643835+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079517+0000","last_change":"2024-02-19T21:03:10.017843+0000","last_active":"2024-02-19T21:03:12.079517+0000","last_peered":"2024-02-19T21:03:12.079517+0000","last_clean":"2024-02-19T21:03:12.079517+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:03:12.079517+0000","last_undegraded":"2024-02-19T21:03:12.079517+0000","last_fullsized":"2024-02-19T21:03:12.079517+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:10.017764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:10.017764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.435524+0000","last_change":"2024-02-19T21:02:37.258648+0000","last_active":"2024-02-19T21:03:12.435524+0000","last_peered":"2024-02-19T21:03:12.435524+0000","last_clean":"2024-02-19T21:03:12.435524+0000","last_became_active":"2024-02-19T21:01:16.271911+0000","last_became_peered":"2024-02-19T21:01:16.271911+0000","last_unstale":"2024-02-19T21:03:12.435524+0000","last_undegraded":"2024-02-19T21:03:12.435524+0000","last_fullsized":"2024-02-19T21:03:12.435524+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:37.258580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:37.258580+0000","last_clean_scrub_stamp":"2024-02-19T21:02:37.258580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":63,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-02-19T21:03:12.079403+0000","last_change":"2024-02-19T21:03:01.005530+0000","last_active":"2024-02-19T21:03:12.079403+0000","last_peered":"2024-02-19T21:03:12.079403+0000","last_clean":"2024-02-19T21:03:12.079403+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:03:12.079403+0000","last_undegraded":"2024-02-19T21:03:12.079403+0000","last_fullsized":"2024-02-19T21:03:12.079403+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:03:01.005445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.005445+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":69,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-02-19T21:03:20.541255+0000","last_change":"2024-02-19T21:02:44.563371+0000","last_active":"2024-02-19T21:03:20.541255+0000","last_peered":"2024-02-19T21:03:20.541255+0000","last_clean":"2024-02-19T21:03:20.541255+0000","last_became_active":"2024-02-19T21:01:20.288720+0000","last_became_peered":"2024-02-19T21:01:20.288720+0000","last_unstale":"2024-02-19T21:03:20.541255+0000","last_undegraded":"2024-02-19T21:03:20.541255+0000","last_fullsized":"2024-02-19T21:03:20.541255+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:44.563256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:44.563256+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":399,"ondisk_log_size":399,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":122,"num_read_kb":117,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266716,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297712,"kb_used_data":608,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":622592,"data_stored":283441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.502,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.57499999999999996,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63500000000000001,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.28499999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.51300000000000001,"15min":0.52000000000000002},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78200000000000003,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51300000000000001,"15min":0.51900000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.61499999999999999,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.35999999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.499,"15min":0.5},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.313},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.433,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70599999999999996,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.39700000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.39600000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.67800000000000005,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.496,"15min":0.49399999999999999},"min":{"1min":0.38700000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.39000000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.48599999999999999,"15min":0.498},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58199999999999996,"5min":1.373,"15min":1.373},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.55600000000000005}]},{"osd":5,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.46000000000000002,"15min":0.46100000000000002},"min":{"1min":0.31900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66800000000000004,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.47699999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.34699999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.64700000000000002,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.56699999999999995,"15min":0.57599999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76000000000000001,"5min":1.262,"15min":1.262},"last":0.498}]}]},{"osd":6,"up_from":45,"seq":193273528354,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297676,"kb_used_data":580,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447476,"statfs":{"total":95995035648,"available":95690215424,"internally_reserved":0,"allocated":593920,"data_stored":281905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.48799999999999999,"15min":0.49199999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.58799999999999997}]},{"osd":1,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.49399999999999999,"15min":0.498},"min":{"1min":0.34499999999999997,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.754}]},{"osd":2,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51300000000000001,"15min":0.51700000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.49299999999999999,"15min":0.49099999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.624}]},{"osd":3,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.503,"15min":0.497},"min":{"1min":0.38900000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.65800000000000003}]},{"osd":4,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.39300000000000002,"15min":0.375},"min":{"1min":0.29299999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.363,"5min":1.363,"15min":1.363},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.41299999999999998,"15min":0.40300000000000002},"min":{"1min":0.28899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.51100000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.42999999999999999,"15min":0.41199999999999998},"min":{"1min":0.34000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.46300000000000002,"15min":0.44700000000000001},"min":{"1min":0.39900000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.56399999999999995}]},{"osd":7,"last update":"Mon Feb 19 21:03:16 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.38600000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67400000000000004,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65300000000000002,"5min":1.534,"15min":1.534},"last":0.57599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":616,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":630784,"data_stored":283532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35899999999999999,"15min":0.34799999999999998},"min":{"1min":0.22500000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.38,"15min":0.35999999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:03:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.40300000000000002,"15min":0.40600000000000003},"min":{"1min":0.27500000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.499,"5min":0.62,"15min":0.62},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.40300000000000002,"15min":0.41499999999999998},"min":{"1min":0.26900000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.52800000000000002,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.313}]},{"osd":3,"last update":"Mon Feb 19 21:02:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.41399999999999998,"15min":0.42399999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.51800000000000002,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.45500000000000002,"15min":0.46300000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58799999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.34699999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68300000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71699999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.33100000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57799999999999996,"15min":0.58399999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72799999999999998,"5min":0.872,"15min":0.872},"last":1.052}]},{"osd":6,"last update":"Mon Feb 19 21:02:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.57699999999999996,"15min":0.58599999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71299999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.63,"15min":0.64300000000000002},"min":{"1min":0.44600000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.68100000000000005,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.41299999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:03:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.45700000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.47699999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.747,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.441}]}]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297712,"kb_used_data":608,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":622592,"data_stored":283095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.372,"15min":0.315},"min":{"1min":0.21299999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.46899999999999997,"5min":4.2830000000000004,"15min":4.2830000000000004},"last":0.318},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.33300000000000002,"15min":0.308},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.48299999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.22700000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.39600000000000002,"15min":0.378},"min":{"1min":0.222,"5min":0.219,"15min":0.219},"max":{"1min":0.45300000000000001,"5min":3.121,"15min":3.121},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.42499999999999999,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.497,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.25700000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.47099999999999997,"15min":0.45100000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.503,"5min":4.4470000000000001,"15min":4.4470000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.48399999999999999,"15min":0.46100000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.504,"5min":4.3070000000000004,"15min":4.3070000000000004},"last":0.28199999999999997}]},{"osd":4,"last update":"Mon Feb 19 21:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52600000000000002,"15min":0.51800000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55400000000000005,"15min":0.54000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73999999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.55000000000000004}]},{"osd":5,"last update":"Mon Feb 19 21:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.628,"15min":0.64100000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88300000000000001,"5min":1.581,"15min":1.581},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.78800000000000003,"15min":0.82799999999999996},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76900000000000002,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.59399999999999997}]},{"osd":6,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.1499999999999999,"5min":1.381,"15min":1.381},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.64200000000000002,"15min":0.65600000000000003},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.79600000000000004,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.50900000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.76900000000000002,"15min":0.79900000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2070000000000001,"5min":4.226,"15min":4.226},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.625,"15min":0.63100000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.218,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.56299999999999994}]}]},{"osd":2,"up_from":18,"seq":77309411380,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297692,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447460,"statfs":{"total":95995035648,"available":95690199040,"internally_reserved":0,"allocated":610304,"data_stored":282683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36199999999999999,"15min":0.35399999999999998},"min":{"1min":0.23000000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.55500000000000005,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.373},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.35399999999999998,"15min":0.34599999999999997},"min":{"1min":0.252,"5min":0.214,"15min":0.214},"max":{"1min":0.40300000000000002,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.41799999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.35299999999999998},"min":{"1min":0.23300000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.432},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40500000000000003,"15min":0.39600000000000002},"min":{"1min":0.28799999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.47199999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:02:40 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.48499999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.46200000000000002,"15min":0.47399999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.56699999999999995,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.442}]},{"osd":4,"last update":"Mon Feb 19 21:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.58099999999999996,"15min":0.59999999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71899999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.60899999999999999,"15min":0.64300000000000002},"min":{"1min":0.38400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.74199999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.48699999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:02:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.63500000000000001,"15min":0.65500000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70899999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.63700000000000001,"15min":0.65200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80300000000000005,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.53000000000000003}]},{"osd":6,"last update":"Mon Feb 19 21:02:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.62,"15min":0.629},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77800000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.624},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.622,"15min":0.63200000000000001},"min":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.76300000000000001,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.57999999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.58899999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182448,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297716,"kb_used_data":620,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":634880,"data_stored":282708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.35199999999999998,"15min":0.35399999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.39300000000000002,"15min":0.40300000000000002},"min":{"1min":0.27500000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.51600000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.317}]},{"osd":1,"last update":"Mon Feb 19 21:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.38600000000000001,"15min":0.39600000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.39500000000000002,"15min":0.39900000000000002},"min":{"1min":0.26500000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59199999999999997,"5min":1.054,"15min":1.054},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:02:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41499999999999998,"15min":0.42199999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.56399999999999995,"5min":0.625,"15min":0.625},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.40999999999999998,"15min":0.41899999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.52900000000000003,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.37}]},{"osd":4,"last update":"Mon Feb 19 21:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.41199999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82799999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55700000000000005,"15min":0.56599999999999995},"min":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.83399999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.48299999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55100000000000005,"15min":0.56000000000000005},"min":{"1min":0.38500000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.501}]},{"osd":6,"last update":"Mon Feb 19 21:02:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56699999999999995,"15min":0.56999999999999995},"min":{"1min":0.39000000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85199999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.56799999999999995}]},{"osd":7,"last update":"Mon Feb 19 21:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.39900000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56799999999999995,"15min":0.57099999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.58499999999999996}]}]},{"osd":4,"up_from":31,"seq":133143986220,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297704,"kb_used_data":608,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447448,"statfs":{"total":95995035648,"available":95690186752,"internally_reserved":0,"allocated":622592,"data_stored":282721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.52500000000000002,"15min":0.54800000000000004},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.67800000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.50700000000000001,"15min":0.52300000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.59899999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.439}]},{"osd":1,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.754,"5min":2.149,"15min":2.149},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.502,"15min":0.51900000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57799999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.48599999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.53300000000000003,"15min":0.55700000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66400000000000003,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.50900000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.51700000000000002,"15min":0.52500000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.502,"5min":0.56499999999999995,"15min":0.56999999999999995},"min":{"1min":0.33400000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64200000000000002,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":0.61899999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:02:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.41799999999999998,"15min":0.42799999999999999},"min":{"1min":0.26800000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.501,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.29899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.53600000000000003,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.66100000000000003}]},{"osd":6,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.377,"15min":0.36899999999999999},"min":{"1min":0.28699999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.42899999999999999,"15min":0.42199999999999999},"min":{"1min":0.36199999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.63500000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.442,"15min":0.432},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.46800000000000003,"15min":0.45900000000000002},"min":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.64500000000000002}]}]},{"osd":5,"up_from":37,"seq":158913789991,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297680,"kb_used_data":584,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447472,"statfs":{"total":95995035648,"available":95690211328,"internally_reserved":0,"allocated":598016,"data_stored":281176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51200000000000001,"15min":0.51600000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.66500000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.53400000000000003,"15min":0.54300000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.70199999999999996,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.44800000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52000000000000002,"15min":0.52500000000000002},"min":{"1min":0.38,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.52400000000000002,"15min":0.53200000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.58799999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.53200000000000003,"15min":0.54000000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.63200000000000001,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.53000000000000003,"15min":0.53500000000000003},"min":{"1min":0.40799999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.65400000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.54800000000000004}]},{"osd":3,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52700000000000002,"15min":0.53500000000000003},"min":{"1min":0.36099999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.57899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54900000000000004,"15min":0.55300000000000005},"min":{"1min":0.40600000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71199999999999997,"5min":0.747,"15min":0.747},"last":0.39800000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:02:22 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.39200000000000002,"15min":0.38600000000000001},"min":{"1min":0.30099999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.56299999999999994,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.39400000000000002,"15min":0.38400000000000001},"min":{"1min":0.30099999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.433}]},{"osd":6,"last update":"Mon Feb 19 21:02:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.42499999999999999,"15min":0.41699999999999998},"min":{"1min":0.311,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.441,"15min":0.432},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.624,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.501}]},{"osd":7,"last update":"Mon Feb 19 21:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.503,"15min":0.502},"min":{"1min":0.35699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.63100000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.50800000000000001,"15min":0.505},"min":{"1min":0.36599999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.66000000000000003,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.56399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:03:29.727 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-19T21:03:29.727 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd dump 2024-02-19T21:03:29.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:27.959624+0000 osd.2 (osd.2) 35 : cluster [DBG] 4.7 deep-scrub starts 2024-02-19T21:03:29.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:27.966518+0000 osd.2 (osd.2) 36 : cluster [DBG] 4.7 deep-scrub ok 2024-02-19T21:03:29.995 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:30.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:03:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:03:30.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:28.936528+0000 mgr.y (mgr.14550) 92 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:03:30.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:29.028699+0000 mgr.y (mgr.14550) 93 : audit [DBG] from='client.34694 v1:172.21.15.163:0/2467727181' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:03:30.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:28.936528+0000 mgr.y (mgr.14550) 92 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:03:30.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:29.028699+0000 mgr.y (mgr.14550) 93 : audit [DBG] from='client.34694 v1:172.21.15.163:0/2467727181' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:03:30.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:28.936528+0000 mgr.y (mgr.14550) 92 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:03:30.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:29.028699+0000 mgr.y (mgr.14550) 93 : audit [DBG] from='client.34694 v1:172.21.15.163:0/2467727181' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:03:31.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.573+0000 7fa0ebee8700 1 -- 172.21.15.163:0/3248186443 >> v1:172.21.15.163:6790/0 conn(0x7fa0e40fc870 legacy=0x7fa0e40fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:31.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.573+0000 7fa0ebee8700 1 -- 172.21.15.163:0/3248186443 shutdown_connections 2024-02-19T21:03:31.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.573+0000 7fa0ebee8700 1 -- 172.21.15.163:0/3248186443 >> 172.21.15.163:0/3248186443 conn(0x7fa0e40f2470 msgr2=0x7fa0e40f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:31.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.574+0000 7fa0ebee8700 1 -- 172.21.15.163:0/3248186443 shutdown_connections 2024-02-19T21:03:31.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.574+0000 7fa0ebee8700 1 -- 172.21.15.163:0/3248186443 wait complete. 2024-02-19T21:03:31.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.574+0000 7fa0ebee8700 1 librados: starting msgr at 2024-02-19T21:03:31.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.574+0000 7fa0ebee8700 1 librados: starting objecter 2024-02-19T21:03:31.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.574+0000 7fa0ebee8700 1 Processor -- start 2024-02-19T21:03:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.574+0000 7fa0ebee8700 1 -- start start 2024-02-19T21:03:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.574+0000 7fa0ebee8700 1 librados: setting wanted keys 2024-02-19T21:03:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.574+0000 7fa0ebee8700 1 librados: calling monclient init 2024-02-19T21:03:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.574+0000 7fa0ebee8700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa0e41160d0 con 0x7fa0e40ff670 2024-02-19T21:03:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.574+0000 7fa0ebee8700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa0e41172d0 con 0x7fa0e41031b0 2024-02-19T21:03:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.574+0000 7fa0ebee8700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa0e41184d0 con 0x7fa0e40fc870 2024-02-19T21:03:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.574+0000 7fa0e9c84700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fa0e40fc870 0x7fa0e40f8850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:33752/0 (socket says 172.21.15.163:33752) 2024-02-19T21:03:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.574+0000 7fa0e9c84700 1 -- 172.21.15.163:0/811811409 learned_addr learned my addr 172.21.15.163:0/811811409 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.574+0000 7fa0e9483700 1 --1- 172.21.15.163:0/811811409 >> v1:172.21.15.163:6789/0 conn(0x7fa0e40ff670 0x7fa0e40f8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:58548/0 (socket says 172.21.15.163:58548) 2024-02-19T21:03:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 10 client.?.objecter ms_handle_connect 0x7fa0e40fc870 2024-02-19T21:03:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 10 client.?.objecter ms_handle_connect 0x7fa0e40ff670 2024-02-19T21:03:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:31.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 10 client.?.objecter ms_handle_connect 0x7fa0e41031b0 2024-02-19T21:03:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966810532 0 0) 0x7fa0e41184d0 con 0x7fa0e40fc870 2024-02-19T21:03:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d00038e0 con 0x7fa0e40fc870 2024-02-19T21:03:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401500691 0 0) 0x7fa0e41160d0 con 0x7fa0e40ff670 2024-02-19T21:03:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e41184d0 con 0x7fa0e40ff670 2024-02-19T21:03:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301286287 0 0) 0x7fa0e41172d0 con 0x7fa0e41031b0 2024-02-19T21:03:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e41160d0 con 0x7fa0e41031b0 2024-02-19T21:03:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1790290539 0 0) 0x7fa0d00038e0 con 0x7fa0e40fc870 2024-02-19T21:03:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0e41172d0 con 0x7fa0e40fc870 2024-02-19T21:03:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa0e0002e10 con 0x7fa0e40fc870 2024-02-19T21:03:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3795741303 0 0) 0x7fa0e41160d0 con 0x7fa0e41031b0 2024-02-19T21:03:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0d00038e0 con 0x7fa0e41031b0 2024-02-19T21:03:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa0dc001d70 con 0x7fa0e41031b0 2024-02-19T21:03:31.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1653012954 0 0) 0x7fa0e41184d0 con 0x7fa0e40ff670 2024-02-19T21:03:31.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0e41160d0 con 0x7fa0e40ff670 2024-02-19T21:03:31.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa0d4001840 con 0x7fa0e40ff670 2024-02-19T21:03:31.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2269321234 0 0) 0x7fa0e41172d0 con 0x7fa0e40fc870 2024-02-19T21:03:31.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 >> v1:172.21.15.193:6789/0 conn(0x7fa0e41031b0 legacy=0x7fa0e4114830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:31.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 >> v1:172.21.15.163:6789/0 conn(0x7fa0e40ff670 legacy=0x7fa0e40f8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:31.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e41196d0 con 0x7fa0e40fc870 2024-02-19T21:03:31.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0ebee8700 1 -- 172.21.15.163:0/811811409 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e41162c0 con 0x7fa0e40fc870 2024-02-19T21:03:31.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0ebee8700 10 client.34700.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:31.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0ebee8700 1 -- 172.21.15.163:0/811811409 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e4116730 con 0x7fa0e40fc870 2024-02-19T21:03:31.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.575+0000 7fa0ebee8700 1 librados: init done 2024-02-19T21:03:31.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.576+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fa0e00023a0 con 0x7fa0e40fc870 2024-02-19T21:03:31.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.576+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa0e00038f0 con 0x7fa0e40fc870 2024-02-19T21:03:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.576+0000 7fa0ebee8700 1 -- 172.21.15.163:0/811811409 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c8004f20 con 0x7fa0e40fc870 2024-02-19T21:03:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.576+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fa0e0003a50 con 0x7fa0e40fc870 2024-02-19T21:03:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.577+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(87..87 src has 32..87) v4 ==== 6172+0+0 (unknown 2939065595 0 0) 0x7fa0e00788f0 con 0x7fa0e40fc870 2024-02-19T21:03:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.577+0000 7fa0daffd700 10 client.34700.objecter ms_dispatch 0x7fa0e4001550 osd_map(87..87 src has 32..87) v4 2024-02-19T21:03:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.577+0000 7fa0daffd700 3 client.34700.objecter handle_osd_map got epochs [87,87] > 0 2024-02-19T21:03:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.577+0000 7fa0daffd700 3 client.34700.objecter handle_osd_map decoding full epoch 87 2024-02-19T21:03:31.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.577+0000 7fa0daffd700 20 client.34700.objecter dump_active .. 0 homeless 2024-02-19T21:03:31.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.577+0000 7fa0daffd700 10 client.34700.objecter ms_handle_connect 0x7fa0d0063cd0 2024-02-19T21:03:31.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.581+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa0e004c4a0 con 0x7fa0e40fc870 2024-02-19T21:03:31.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.744+0000 7fa0ebee8700 1 -- 172.21.15.163:0/811811409 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fa0c80059e0 con 0x7fa0e40fc870 2024-02-19T21:03:31.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.744+0000 7fa0daffd700 1 -- 172.21.15.163:0/811811409 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) v1 ==== 56+0+4029 (unknown 1124525651 0 958204748) 0x7fa0e004a910 con 0x7fa0e40fc870 2024-02-19T21:03:31.737 INFO:teuthology.orchestra.run.smithi163.stdout:epoch 87 2024-02-19T21:03:31.737 INFO:teuthology.orchestra.run.smithi163.stdout:fsid 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T21:03:31.737 INFO:teuthology.orchestra.run.smithi163.stdout:created 2024-02-19T20:55:37.794885+0000 2024-02-19T21:03:31.737 INFO:teuthology.orchestra.run.smithi163.stdout:modified 2024-02-19T21:03:31.459714+0000 2024-02-19T21:03:31.737 INFO:teuthology.orchestra.run.smithi163.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-02-19T21:03:31.737 INFO:teuthology.orchestra.run.smithi163.stdout:crush_version 18 2024-02-19T21:03:31.737 INFO:teuthology.orchestra.run.smithi163.stdout:full_ratio 0.95 2024-02-19T21:03:31.737 INFO:teuthology.orchestra.run.smithi163.stdout:backfillfull_ratio 0.9 2024-02-19T21:03:31.737 INFO:teuthology.orchestra.run.smithi163.stdout:nearfull_ratio 0.85 2024-02-19T21:03:31.737 INFO:teuthology.orchestra.run.smithi163.stdout:require_min_compat_client luminous 2024-02-19T21:03:31.737 INFO:teuthology.orchestra.run.smithi163.stdout:min_compat_client luminous 2024-02-19T21:03:31.737 INFO:teuthology.orchestra.run.smithi163.stdout:require_osd_release pacific 2024-02-19T21:03:31.737 INFO:teuthology.orchestra.run.smithi163.stdout:stretch_mode_enabled false 2024-02-19T21:03:31.738 INFO:teuthology.orchestra.run.smithi163.stdout:pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 74 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth 2024-02-19T21:03:31.738 INFO:teuthology.orchestra.run.smithi163.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 59 flags hashpspool stripe_width 0 application rgw 2024-02-19T21:03:31.738 INFO:teuthology.orchestra.run.smithi163.stdout:pool 3 'iscsi' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 60 flags hashpspool stripe_width 0 application rbd 2024-02-19T21:03:31.738 INFO:teuthology.orchestra.run.smithi163.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 61 flags hashpspool stripe_width 0 application rgw 2024-02-19T21:03:31.738 INFO:teuthology.orchestra.run.smithi163.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 63 flags hashpspool stripe_width 0 application rgw 2024-02-19T21:03:31.738 INFO:teuthology.orchestra.run.smithi163.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 65 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw 2024-02-19T21:03:31.738 INFO:teuthology.orchestra.run.smithi163.stdout:max_osd 8 2024-02-19T21:03:31.738 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0 up in weight 1 up_from 8 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.163:6801/1559418115 v1:172.21.15.163:6802/1559418115 exists,up 4a14b406-7c21-40f3-8b12-8cf9df811088 2024-02-19T21:03:31.738 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1 up in weight 1 up_from 13 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.163:6805/3125047432 v1:172.21.15.163:6806/3125047432 exists,up e74a65fa-cb9c-42c3-9293-602cf7e9dc67 2024-02-19T21:03:31.738 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2 up in weight 1 up_from 18 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.163:6809/1563158487 v1:172.21.15.163:6810/1563158487 exists,up 6980a625-cdc9-487e-8f8c-7997ba716a97 2024-02-19T21:03:31.738 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3 up in weight 1 up_from 25 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.163:6813/2112279769 v1:172.21.15.163:6814/2112279769 exists,up 359e2ee0-9d1d-4074-8119-2d69bc2ca3a5 2024-02-19T21:03:31.738 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4 up in weight 1 up_from 31 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6800/3129716965 v1:172.21.15.193:6801/3129716965 exists,up 2bf5ef05-292b-4693-85a9-97991f2091e0 2024-02-19T21:03:31.738 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5 up in weight 1 up_from 37 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6804/805161991 v1:172.21.15.193:6805/805161991 exists,up 59065f9b-9240-414b-9c91-e91cda4aed6f 2024-02-19T21:03:31.739 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6 up in weight 1 up_from 45 up_thru 61 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6808/787604615 v1:172.21.15.193:6809/787604615 exists,up 302c95df-3842-4818-8afd-d79d8184684f 2024-02-19T21:03:31.739 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7 up in weight 1 up_from 53 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6812/157927432 v1:172.21.15.193:6813/157927432 exists,up 9ebeff27-c2f8-4a66-923f-a0d7573ddd3f 2024-02-19T21:03:31.739 INFO:teuthology.orchestra.run.smithi163.stdout:pg_upmap_items 2.4 [7,2] 2024-02-19T21:03:31.739 INFO:teuthology.orchestra.run.smithi163.stdout:pg_upmap_items 4.7 [1,2] 2024-02-19T21:03:31.739 INFO:teuthology.orchestra.run.smithi163.stdout:pg_upmap_items 4.d [1,5] 2024-02-19T21:03:31.739 INFO:teuthology.orchestra.run.smithi163.stdout:pg_upmap_items 4.12 [1,5] 2024-02-19T21:03:31.739 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/1427774891 expires 2024-02-20T21:01:38.896227+0000 2024-02-19T21:03:31.739 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/3282102947 expires 2024-02-20T21:01:38.896227+0000 2024-02-19T21:03:31.739 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:6800/2735092549 expires 2024-02-20T21:01:38.896227+0000 2024-02-19T21:03:31.739 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/1235588846 expires 2024-02-20T20:56:29.603030+0000 2024-02-19T21:03:31.739 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:6800/2731225015 expires 2024-02-20T20:56:29.603030+0000 2024-02-19T21:03:31.739 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/506491651 expires 2024-02-20T20:56:02.185409+0000 2024-02-19T21:03:31.739 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/168117826 expires 2024-02-20T21:01:38.896227+0000 2024-02-19T21:03:31.739 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/1458009465 expires 2024-02-20T20:56:29.603030+0000 2024-02-19T21:03:31.739 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:0/541631472 expires 2024-02-20T20:56:02.185409+0000 2024-02-19T21:03:31.740 INFO:teuthology.orchestra.run.smithi163.stdout:blocklist 172.21.15.163:6800/2248999541 expires 2024-02-20T20:56:02.185409+0000 2024-02-19T21:03:31.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.746+0000 7fa0ebee8700 10 librados: watch_flush enter 2024-02-19T21:03:31.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.746+0000 7fa0ebee8700 10 librados: watch_flush exit 2024-02-19T21:03:31.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.746+0000 7fa0ebee8700 20 client.34700.objecter shutdown clearing up homeless session... 2024-02-19T21:03:31.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.746+0000 7fa0ebee8700 10 client.34700.objecter successfully canceled tick 2024-02-19T21:03:31.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.746+0000 7fa0ebee8700 1 -- 172.21.15.163:0/811811409 >> v1:172.21.15.163:6800/1552514681 conn(0x7fa0d0063cd0 legacy=0x7fa0d0066190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:31.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.746+0000 7fa0ebee8700 1 -- 172.21.15.163:0/811811409 >> v1:172.21.15.163:6790/0 conn(0x7fa0e40fc870 legacy=0x7fa0e40f8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:31.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.747+0000 7fa0ebee8700 1 -- 172.21.15.163:0/811811409 shutdown_connections 2024-02-19T21:03:31.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.747+0000 7fa0ebee8700 1 -- 172.21.15.163:0/811811409 >> 172.21.15.163:0/811811409 conn(0x7fa0e40f2470 msgr2=0x7fa0e4106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:31.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.747+0000 7fa0ebee8700 1 -- 172.21.15.163:0/811811409 shutdown_connections 2024-02-19T21:03:31.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.747+0000 7fa0ebee8700 1 -- 172.21.15.163:0/811811409 wait complete. 2024-02-19T21:03:31.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:31.747+0000 7fa0ebee8700 1 librados: shutdown 2024-02-19T21:03:31.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:30.018008+0000 mgr.y (mgr.14550) 94 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:31.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:30.462057+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.d scrub starts 2024-02-19T21:03:31.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:30.462864+0000 osd.1 (osd.1) 48 : cluster [DBG] 2.d scrub ok 2024-02-19T21:03:31.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:30.018008+0000 mgr.y (mgr.14550) 94 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:31.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:30.462057+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.d scrub starts 2024-02-19T21:03:31.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:30.462864+0000 osd.1 (osd.1) 48 : cluster [DBG] 2.d scrub ok 2024-02-19T21:03:31.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:30.018008+0000 mgr.y (mgr.14550) 94 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:31.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:30.462057+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.d scrub starts 2024-02-19T21:03:31.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:30.462864+0000 osd.1 (osd.1) 48 : cluster [DBG] 2.d scrub ok 2024-02-19T21:03:32.411 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.163:6801/1559418115 v1:172.21.15.163:6802/1559418115 exists,up 4a14b406-7c21-40f3-8b12-8cf9df811088', 'osd.1 up in weight 1 up_from 13 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.163:6805/3125047432 v1:172.21.15.163:6806/3125047432 exists,up e74a65fa-cb9c-42c3-9293-602cf7e9dc67', 'osd.2 up in weight 1 up_from 18 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.163:6809/1563158487 v1:172.21.15.163:6810/1563158487 exists,up 6980a625-cdc9-487e-8f8c-7997ba716a97', 'osd.3 up in weight 1 up_from 25 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.163:6813/2112279769 v1:172.21.15.163:6814/2112279769 exists,up 359e2ee0-9d1d-4074-8119-2d69bc2ca3a5', 'osd.4 up in weight 1 up_from 31 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6800/3129716965 v1:172.21.15.193:6801/3129716965 exists,up 2bf5ef05-292b-4693-85a9-97991f2091e0', 'osd.5 up in weight 1 up_from 37 up_thru 66 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6804/805161991 v1:172.21.15.193:6805/805161991 exists,up 59065f9b-9240-414b-9c91-e91cda4aed6f', 'osd.6 up in weight 1 up_from 45 up_thru 61 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6808/787604615 v1:172.21.15.193:6809/787604615 exists,up 302c95df-3842-4818-8afd-d79d8184684f', 'osd.7 up in weight 1 up_from 53 up_thru 63 down_at 0 last_clean_interval [0,0) v1:172.21.15.193:6812/157927432 v1:172.21.15.193:6813/157927432 exists,up 9ebeff27-c2f8-4a66-923f-a0d7573ddd3f'] 2024-02-19T21:03:32.412 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon mon.a config show 2024-02-19T21:03:32.687 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:32.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:30.937087+0000 mgr.y (mgr.14550) 95 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:03:32.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:31.058855+0000 osd.0 (osd.0) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:03:32.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:31.058934+0000 osd.0 (osd.0) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:03:32.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:31.466073+0000 mon.a (mon.0) 715 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-19T21:03:32.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:31.745813+0000 mon.c (mon.2) 26 : audit [DBG] from='client.? v1:172.21.15.163:0/811811409' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-19T21:03:32.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:30.937087+0000 mgr.y (mgr.14550) 95 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:03:32.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:31.058855+0000 osd.0 (osd.0) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:03:32.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:31.058934+0000 osd.0 (osd.0) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:03:32.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:31.466073+0000 mon.a (mon.0) 715 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-19T21:03:32.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:31.745813+0000 mon.c (mon.2) 26 : audit [DBG] from='client.? v1:172.21.15.163:0/811811409' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-19T21:03:32.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:30.937087+0000 mgr.y (mgr.14550) 95 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:03:32.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:31.058855+0000 osd.0 (osd.0) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:03:32.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:31.058934+0000 osd.0 (osd.0) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:03:32.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:31.466073+0000 mon.a (mon.0) 715 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-02-19T21:03:32.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:31.745813+0000 mon.c (mon.2) 26 : audit [DBG] from='client.? v1:172.21.15.163:0/811811409' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-19T21:03:33.739 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:03:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:03:33] "GET /metrics HTTP/1.1" 200 172164 "" "Prometheus/2.33.4" 2024-02-19T21:03:33.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:32.720017+0000 osd.3 (osd.3) 53 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:03:33.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:32.721869+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:03:33.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:32.720017+0000 osd.3 (osd.3) 53 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:03:33.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:32.721869+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:03:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:32.720017+0000 osd.3 (osd.3) 53 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:03:33.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:32.721869+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:03:34.518 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:03:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "name": "mon.a", 2024-02-19T21:03:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster": "ceph", 2024-02-19T21:03:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-02-19T21:03:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "admin_socket_mode": "", 2024-02-19T21:03:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-02-19T21:03:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_client_required": "cephx, none", 2024-02-19T21:03:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_cluster_required": "cephx", 2024-02-19T21:03:34.518 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_debug": "true", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_service_required": "cephx", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_service_ticket_ttl": "240.000000", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "auth_supported": "", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_aio": "true", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_aio_max_queue_depth": "1024", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_aio_poll_ms": "250", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_aio_reap_max": "16", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_async_discard": "false", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_block_size": "4096", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_debug_aio": "false", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_debug_inflight_ios": "false", 2024-02-19T21:03:34.519 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_enable_discard": "false", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_flock_retry": "3", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_flock_retry_interval": "0.100000", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_inject_crash": "0", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_inject_crash_flush_delay": "2", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_ioring": "false", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_ioring_hipri": "false", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_ioring_sqthread_poll": "false", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bdev_type": "", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_alloc_size": "1048576", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_allocator": "hybrid", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_buffered_io": "true", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_check_for_zeros": "false", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_check_volume_selector_often": "false", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-02-19T21:03:34.520 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_compact_log_sync": "false", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_log_compact_min_size": "16777216", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_log_replay_check_allocations": "true", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_max_log_runway": "4194304", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_max_prefetch": "1048576", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_min_flush_size": "524288", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_min_log_runway": "1048576", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_replay_recovery": "false", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_shared_alloc_size": "65536", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_sync_write": "false", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-02-19T21:03:34.521 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_allocator": "hybrid", 2024-02-19T21:03:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-02-19T21:03:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-02-19T21:03:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-02-19T21:03:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-02-19T21:03:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-02-19T21:03:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-02-19T21:03:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_blobid_prealloc": "10240", 2024-02-19T21:03:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_create": "true", 2024-02-19T21:03:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_db_create": "false", 2024-02-19T21:03:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_db_path": "", 2024-02-19T21:03:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_db_size": "0", 2024-02-19T21:03:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_path": "", 2024-02-19T21:03:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_preallocate_file": "false", 2024-02-19T21:03:34.522 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_size": "107374182400", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_wal_create": "false", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_wal_path": "", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_block_wal_size": "100663296", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_bluefs": "true", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_bluefs_env_mirror": "false", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_autotune": "true", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_size": "0", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-02-19T21:03:34.523 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-02-19T21:03:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_type": "2q", 2024-02-19T21:03:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_clone_cow": "true", 2024-02-19T21:03:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_algorithm": "snappy", 2024-02-19T21:03:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_max_blob_size": "0", 2024-02-19T21:03:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-02-19T21:03:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-02-19T21:03:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_min_blob_size": "0", 2024-02-19T21:03:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-02-19T21:03:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_min_blob_size_ssd": "8192", 2024-02-19T21:03:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_mode": "none", 2024-02-19T21:03:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-02-19T21:03:34.524 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_csum_type": "crc32c", 2024-02-19T21:03:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_enforce_settings": "default", 2024-02-19T21:03:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_freelist": "false", 2024-02-19T21:03:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_fsck_abort": "false", 2024-02-19T21:03:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_inject_bug21040": "false", 2024-02-19T21:03:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-02-19T21:03:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_inject_read_err": "false", 2024-02-19T21:03:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_legacy_omap": "false", 2024-02-19T21:03:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-02-19T21:03:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-02-19T21:03:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-02-19T21:03:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-02-19T21:03:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_prefill": "0.000000", 2024-02-19T21:03:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-02-19T21:03:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-02-19T21:03:34.525 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_small_allocations": "0", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_default_buffered_read": "true", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_default_buffered_write": "false", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_deferred_batch_ops": "0", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-02-19T21:03:34.526 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_on_mkfs": "true", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_on_mount": "false", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_on_umount": "false", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_ignore_data_csum": "false", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-02-19T21:03:34.527 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_kvbackend": "rocksdb", 2024-02-19T21:03:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-02-19T21:03:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-02-19T21:03:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_log_op_age": "5.000000", 2024-02-19T21:03:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_max_alloc_size": "0", 2024-02-19T21:03:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_max_blob_size": "0", 2024-02-19T21:03:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-02-19T21:03:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-02-19T21:03:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_max_defer_interval": "3.000000", 2024-02-19T21:03:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_max_deferred_txc": "32", 2024-02-19T21:03:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_min_alloc_size": "0", 2024-02-19T21:03:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-02-19T21:03:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-02-19T21:03:34.528 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_nid_prealloc": "1024", 2024-02-19T21:03:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_prefer_deferred_size": "0", 2024-02-19T21:03:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-02-19T21:03:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-02-19T21:03:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_retry_disk_reads": "3", 2024-02-19T21:03:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_rocksdb_cf": "true", 2024-02-19T21:03:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-02-19T21:03:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-02-19T21:03:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_rocksdb_options_annex": "", 2024-02-19T21:03:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_spdk_coremask": "0x1", 2024-02-19T21:03:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_spdk_io_sleep": "5", 2024-02-19T21:03:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_spdk_max_io_completion": "0", 2024-02-19T21:03:34.529 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_spdk_mem": "512", 2024-02-19T21:03:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_sync_submit_transaction": "false", 2024-02-19T21:03:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_throttle_bytes": "67108864", 2024-02-19T21:03:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_throttle_cost_per_io": "0", 2024-02-19T21:03:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-02-19T21:03:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-02-19T21:03:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-02-19T21:03:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-02-19T21:03:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_tracing": "false", 2024-02-19T21:03:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-02-19T21:03:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_volume_selection_reserved": "0", 2024-02-19T21:03:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-02-19T21:03:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-02-19T21:03:34.530 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-02-19T21:03:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-02-19T21:03:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-02-19T21:03:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-02-19T21:03:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-02-19T21:03:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_action_update_interval": "2", 2024-02-19T21:03:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-02-19T21:03:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-02-19T21:03:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-02-19T21:03:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-02-19T21:03:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_mount_timeout": "10", 2024-02-19T21:03:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-02-19T21:03:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-02-19T21:03:34.531 INFO:teuthology.orchestra.run.smithi163.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-02-19T21:03:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-02-19T21:03:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-02-19T21:03:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-02-19T21:03:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "cephx_cluster_require_signatures": "false", 2024-02-19T21:03:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "cephx_cluster_require_version": "2", 2024-02-19T21:03:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "cephx_require_signatures": "false", 2024-02-19T21:03:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "cephx_require_version": "2", 2024-02-19T21:03:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "cephx_service_require_signatures": "false", 2024-02-19T21:03:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "cephx_service_require_version": "2", 2024-02-19T21:03:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "cephx_sign_messages": "true", 2024-02-19T21:03:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "chdir": "", 2024-02-19T21:03:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "client_acl_type": "", 2024-02-19T21:03:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "client_asio_thread_count": "2", 2024-02-19T21:03:34.532 INFO:teuthology.orchestra.run.smithi163.stdout: "client_cache_mid": "0.750000", 2024-02-19T21:03:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "client_cache_size": "16384", 2024-02-19T21:03:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "client_caps_release_delay": "5", 2024-02-19T21:03:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "client_check_pool_perm": "true", 2024-02-19T21:03:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "client_collect_and_send_global_metrics": "false", 2024-02-19T21:03:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "client_debug_force_sync_read": "false", 2024-02-19T21:03:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "client_debug_getattr_caps": "false", 2024-02-19T21:03:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "client_debug_inject_tick_delay": "0", 2024-02-19T21:03:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-02-19T21:03:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "client_die_on_failed_remount": "false", 2024-02-19T21:03:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "client_dirsize_rbytes": "true", 2024-02-19T21:03:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "client_force_lazyio": "false", 2024-02-19T21:03:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "client_fs": "", 2024-02-19T21:03:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "client_inject_fixed_oldest_tid": "false", 2024-02-19T21:03:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "client_inject_release_failure": "false", 2024-02-19T21:03:34.533 INFO:teuthology.orchestra.run.smithi163.stdout: "client_max_inline_size": "4096", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_max_retries_on_remount_failure": "5", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_mds_namespace": "", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_metadata": "", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_mount_gid": "-1", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_mount_timeout": "300", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_mount_uid": "-1", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_mountpoint": "/", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_notify_timeout": "10", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_oc": "true", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_oc_max_dirty": "104857600", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_oc_max_dirty_age": "5.000000", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_oc_max_objects": "1000", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_oc_size": "209715200", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_oc_target_dirty": "8388608", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_permissions": "true", 2024-02-19T21:03:34.534 INFO:teuthology.orchestra.run.smithi163.stdout: "client_quota": "true", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "client_quota_df": "true", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "client_readahead_max_bytes": "0", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "client_readahead_max_periods": "4", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "client_readahead_min": "131072", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "client_reconnect_stale": "false", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "client_shutdown_timeout": "30", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "client_snapdir": ".snap", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "client_tick_interval": "1", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "client_trace": "", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "client_try_dentry_invalidate": "false", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "client_use_faked_inos": "false", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "client_use_random_mds": "false", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "clog_to_graylog": "false", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "clog_to_graylog_port": "12201", 2024-02-19T21:03:34.535 INFO:teuthology.orchestra.run.smithi163.stdout: "clog_to_monitors": "default=true", 2024-02-19T21:03:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "clog_to_syslog": "false", 2024-02-19T21:03:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-02-19T21:03:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "clog_to_syslog_level": "info", 2024-02-19T21:03:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "-", 2024-02-19T21:03:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_network": "", 2024-02-19T21:03:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_network_interface": "", 2024-02-19T21:03:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "compressor_zlib_isal": "false", 2024-02-19T21:03:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "compressor_zlib_level": "5", 2024-02-19T21:03:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "compressor_zlib_winsize": "-15", 2024-02-19T21:03:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "compressor_zstd_level": "1", 2024-02-19T21:03:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7fa249cc0191869e2bcc743303b0da933d04d152b2397edb19ec992e48553ab9", 2024-02-19T21:03:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-02-19T21:03:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "crimson_osd_obc_lru_size": "10", 2024-02-19T21:03:34.536 INFO:teuthology.orchestra.run.smithi163.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-02-19T21:03:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_location": "", 2024-02-19T21:03:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_location_hook": "", 2024-02-19T21:03:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_location_hook_timeout": "10", 2024-02-19T21:03:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "daemonize": "false", 2024-02-19T21:03:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_allow_any_pool_priority": "false", 2024-02-19T21:03:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_asok": "1/5", 2024-02-19T21:03:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_asok_assert_abort": "false", 2024-02-19T21:03:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_asserts_on_shutdown": "true", 2024-02-19T21:03:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_auth": "1/5", 2024-02-19T21:03:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_bdev": "1/3", 2024-02-19T21:03:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_bluefs": "1/5", 2024-02-19T21:03:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_bluestore": "1/5", 2024-02-19T21:03:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_buffer": "0/1", 2024-02-19T21:03:34.537 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_cephfs_mirror": "0/5", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_cephsqlite": "0/5", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_civetweb": "1/10", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_client": "0/5", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_compressor": "1/5", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_context": "0/1", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_crush": "1/1", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_crypto": "1/5", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_deliberately_leak_memory": "false", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_disable_randomized_ping": "false", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_dpdk": "1/5", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_eventtrace": "1/5", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_filer": "0/1", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_filestore": "1/3", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_finisher": "1/1", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_fuse": "1/5", 2024-02-19T21:03:34.538 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_heartbeat_testing_span": "0", 2024-02-19T21:03:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_heartbeatmap": "1/5", 2024-02-19T21:03:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_immutable_obj_cache": "0/5", 2024-02-19T21:03:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_javaclient": "1/5", 2024-02-19T21:03:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_journal": "1/3", 2024-02-19T21:03:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_journaler": "0/5", 2024-02-19T21:03:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_kstore": "1/5", 2024-02-19T21:03:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_leveldb": "4/5", 2024-02-19T21:03:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_lockdep": "0/1", 2024-02-19T21:03:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mds": "1/5", 2024-02-19T21:03:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mds_balancer": "1/5", 2024-02-19T21:03:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mds_locker": "1/5", 2024-02-19T21:03:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mds_log": "1/5", 2024-02-19T21:03:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mds_log_expire": "1/5", 2024-02-19T21:03:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mds_migrator": "1/5", 2024-02-19T21:03:34.539 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_memdb": "4/5", 2024-02-19T21:03:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mgr": "2/5", 2024-02-19T21:03:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mgrc": "20/20", 2024-02-19T21:03:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_mon": "20/20", 2024-02-19T21:03:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_monc": "0/10", 2024-02-19T21:03:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_ms": "1/1", 2024-02-19T21:03:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_none": "0/5", 2024-02-19T21:03:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_objclass": "0/5", 2024-02-19T21:03:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_objectcacher": "0/5", 2024-02-19T21:03:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_objecter": "0/1", 2024-02-19T21:03:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_optracker": "0/5", 2024-02-19T21:03:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_osd": "1/5", 2024-02-19T21:03:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_paxos": "20/20", 2024-02-19T21:03:34.540 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_perfcounter": "1/5", 2024-02-19T21:03:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_prioritycache": "1/5", 2024-02-19T21:03:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rados": "0/5", 2024-02-19T21:03:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rbd": "0/5", 2024-02-19T21:03:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rbd_mirror": "0/5", 2024-02-19T21:03:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rbd_pwl": "0/5", 2024-02-19T21:03:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rbd_replay": "0/5", 2024-02-19T21:03:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_refs": "0/0", 2024-02-19T21:03:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_reserver": "1/1", 2024-02-19T21:03:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rgw": "1/5", 2024-02-19T21:03:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rgw_sync": "1/5", 2024-02-19T21:03:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_rocksdb": "4/5", 2024-02-19T21:03:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_striper": "0/1", 2024-02-19T21:03:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_test": "0/5", 2024-02-19T21:03:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_throttle": "1/1", 2024-02-19T21:03:34.541 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_timer": "0/1", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "debug_tp": "0/5", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "device_failure_prediction_mode": "none", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "err_to_graylog": "false", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "err_to_stderr": "true", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "err_to_syslog": "false", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "event_tracing": "false", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "fake_statfs_for_testing": "0", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "fatal_signal_handlers": "true", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "filer_max_purge_ops": "10", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "filer_max_truncate_ops": "128", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_apply_finisher_threads": "1", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_blackhole": "false", 2024-02-19T21:03:34.542 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_btrfs_clone_range": "true", 2024-02-19T21:03:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_btrfs_snap": "true", 2024-02-19T21:03:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_caller_concurrency": "10", 2024-02-19T21:03:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_collect_device_partition_information": "true", 2024-02-19T21:03:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_commit_timeout": "600.000000", 2024-02-19T21:03:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_debug_inject_read_err": "false", 2024-02-19T21:03:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_debug_omap_check": "false", 2024-02-19T21:03:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_debug_random_read_err": "0.000000", 2024-02-19T21:03:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_debug_verify_split": "false", 2024-02-19T21:03:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_dump_file": "", 2024-02-19T21:03:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-02-19T21:03:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-02-19T21:03:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_fadvise": "true", 2024-02-19T21:03:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_fail_eio": "true", 2024-02-19T21:03:34.543 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_fd_cache_shards": "16", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_fd_cache_size": "128", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_fiemap": "false", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_fiemap_threshold": "4096", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_index_retry_probability": "0.000000", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_inject_stall": "0", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_journal_parallel": "false", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_journal_trailing": "false", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_journal_writeahead": "false", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_kill_at": "0", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattr_size": "0", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-02-19T21:03:34.544 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattrs": "0", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattrs_other": "2", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_sync_interval": "5.000000", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_xattr_value_size": "0", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_merge_threshold": "-10", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_min_sync_interval": "0.010000", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_odsync_write": "false", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_omap_backend": "rocksdb", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_omap_backend_path": "", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_omap_header_cache_size": "1024", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_ondisk_finisher_threads": "1", 2024-02-19T21:03:34.545 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-02-19T21:03:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_op_thread_timeout": "60", 2024-02-19T21:03:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_op_threads": "2", 2024-02-19T21:03:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_punch_hole": "false", 2024-02-19T21:03:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-02-19T21:03:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-02-19T21:03:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-02-19T21:03:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-02-19T21:03:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-02-19T21:03:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_max_bytes": "104857600", 2024-02-19T21:03:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-02-19T21:03:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-02-19T21:03:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-02-19T21:03:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_queue_max_ops": "50", 2024-02-19T21:03:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-02-19T21:03:34.546 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_seek_data_hole": "false", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_sloppy_crc": "false", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_splice": "false", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_split_multiple": "2", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_split_rand_factor": "20", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_update_to": "1000", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_enable": "true", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-02-19T21:03:34.547 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-02-19T21:03:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-02-19T21:03:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-02-19T21:03:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-02-19T21:03:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_xfs_extsize": "false", 2024-02-19T21:03:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "filestore_zfs_snap": "false", 2024-02-19T21:03:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "fio_dir": "/tmp/fio", 2024-02-19T21:03:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "fsid": "2591809a-cf69-11ee-95bb-87774f69a715", 2024-02-19T21:03:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_allow_other": "true", 2024-02-19T21:03:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_atomic_o_trunc": "true", 2024-02-19T21:03:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_big_writes": "true", 2024-02-19T21:03:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_debug": "false", 2024-02-19T21:03:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_default_permissions": "false", 2024-02-19T21:03:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_disable_pagecache": "false", 2024-02-19T21:03:34.548 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_max_write": "0", 2024-02-19T21:03:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_multithreaded": "true", 2024-02-19T21:03:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_require_active_mds": "true", 2024-02-19T21:03:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_set_user_groups": "true", 2024-02-19T21:03:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_splice_move": "true", 2024-02-19T21:03:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_splice_read": "true", 2024-02-19T21:03:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_splice_write": "true", 2024-02-19T21:03:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_syncfs_on_mksnap": "true", 2024-02-19T21:03:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "fuse_use_invalidate_cb": "true", 2024-02-19T21:03:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-02-19T21:03:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "gss_target_name": "ceph", 2024-02-19T21:03:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_file": "", 2024-02-19T21:03:34.549 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_inject_failure": "0", 2024-02-19T21:03:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_interval": "5", 2024-02-19T21:03:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "host": "", 2024-02-19T21:03:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-02-19T21:03:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-02-19T21:03:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_max_size": "1073741824", 2024-02-19T21:03:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-02-19T21:03:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-02-19T21:03:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-02-19T21:03:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-02-19T21:03:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-02-19T21:03:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-02-19T21:03:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-02-19T21:03:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-02-19T21:03:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-02-19T21:03:34.550 INFO:teuthology.orchestra.run.smithi163.stdout: "immutable_object_cache_watermark": "0.900000", 2024-02-19T21:03:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "inject_early_sigterm": "false", 2024-02-19T21:03:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_aio": "true", 2024-02-19T21:03:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_align_min_size": "65536", 2024-02-19T21:03:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_block_align": "true", 2024-02-19T21:03:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_block_size": "4096", 2024-02-19T21:03:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_dio": "true", 2024-02-19T21:03:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_discard": "false", 2024-02-19T21:03:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_force_aio": "false", 2024-02-19T21:03:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_ignore_corruption": "false", 2024-02-19T21:03:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_max_write_bytes": "10485760", 2024-02-19T21:03:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_max_write_entries": "100", 2024-02-19T21:03:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_replay_from": "0", 2024-02-19T21:03:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_throttle_high_multiple": "0.000000", 2024-02-19T21:03:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-02-19T21:03:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-02-19T21:03:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_throttle_max_multiple": "0.000000", 2024-02-19T21:03:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_write_header_frequency": "0", 2024-02-19T21:03:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "journal_zero_on_create": "false", 2024-02-19T21:03:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "journaler_prefetch_periods": "10", 2024-02-19T21:03:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "journaler_prezero_periods": "5", 2024-02-19T21:03:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "journaler_write_head_interval": "15", 2024-02-19T21:03:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "key": "", 2024-02-19T21:03:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "keyfile": "", 2024-02-19T21:03:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-02-19T21:03:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_backend": "rocksdb", 2024-02-19T21:03:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_default_stripe_size": "65536", 2024-02-19T21:03:34.552 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_fsck_on_mount": "false", 2024-02-19T21:03:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_fsck_on_mount_deep": "true", 2024-02-19T21:03:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_max_bytes": "67108864", 2024-02-19T21:03:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_max_ops": "512", 2024-02-19T21:03:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_nid_prealloc": "1024", 2024-02-19T21:03:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_onode_map_size": "1024", 2024-02-19T21:03:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-02-19T21:03:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_sync_submit_transaction": "false", 2024-02-19T21:03:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "kstore_sync_transaction": "false", 2024-02-19T21:03:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "leveldb_block_size": "65536", 2024-02-19T21:03:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "leveldb_bloom_size": "0", 2024-02-19T21:03:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "leveldb_cache_size": "536870912", 2024-02-19T21:03:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "leveldb_compact_on_mount": "false", 2024-02-19T21:03:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "leveldb_compression": "false", 2024-02-19T21:03:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "leveldb_log": "", 2024-02-19T21:03:34.553 INFO:teuthology.orchestra.run.smithi163.stdout: "leveldb_log_to_ceph_log": "true", 2024-02-19T21:03:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "leveldb_max_open_files": "0", 2024-02-19T21:03:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "leveldb_paranoid": "false", 2024-02-19T21:03:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "leveldb_write_buffer_size": "33554432", 2024-02-19T21:03:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "librados_thread_count": "2", 2024-02-19T21:03:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "lockdep": "false", 2024-02-19T21:03:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "lockdep_force_backtrace": "false", 2024-02-19T21:03:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "log_coarse_timestamps": "true", 2024-02-19T21:03:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-02-19T21:03:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "log_flush_on_exit": "false", 2024-02-19T21:03:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "log_graylog_host": "127.0.0.1", 2024-02-19T21:03:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "log_graylog_port": "12201", 2024-02-19T21:03:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "log_max_new": "1000", 2024-02-19T21:03:34.554 INFO:teuthology.orchestra.run.smithi163.stdout: "log_max_recent": "10000", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "log_stderr_prefix": "debug ", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "log_stop_at_utilization": "0.970000", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_file": "true", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_graylog": "false", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_stderr": "false", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "log_to_syslog": "false", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "max_rotating_auth_attempts": "10", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_action_on_write_error": "1", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_alternate_name_max": "8192", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_asio_thread_count": "2", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_export_pin": "true", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_fragment_dirs": "true", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_fragment_interval": "5", 2024-02-19T21:03:34.555 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_fragment_size_max": "100000", 2024-02-19T21:03:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_idle_threshold": "0.000000", 2024-02-19T21:03:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_interval": "10", 2024-02-19T21:03:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_max": "-1", 2024-02-19T21:03:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_max_until": "-1", 2024-02-19T21:03:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_merge_size": "50", 2024-02-19T21:03:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_midchunk": "0.300000", 2024-02-19T21:03:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_min_rebalance": "0.100000", 2024-02-19T21:03:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_min_start": "0.200000", 2024-02-19T21:03:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_minchunk": "0.001000", 2024-02-19T21:03:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_mode": "0", 2024-02-19T21:03:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_need_max": "1.200000", 2024-02-19T21:03:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_need_min": "0.800000", 2024-02-19T21:03:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-02-19T21:03:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_sample_interval": "3.000000", 2024-02-19T21:03:34.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_split_bits": "3", 2024-02-19T21:03:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_split_rd": "25000.000000", 2024-02-19T21:03:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_split_size": "10000", 2024-02-19T21:03:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_split_wr": "10000.000000", 2024-02-19T21:03:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_target_decay": "10.000000", 2024-02-19T21:03:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-02-19T21:03:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_beacon_grace": "15.000000", 2024-02-19T21:03:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_beacon_interval": "4.000000", 2024-02-19T21:03:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_beacon_mon_down_grace": "60", 2024-02-19T21:03:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache_memory_limit": "4294967296", 2024-02-19T21:03:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache_mid": "0.700000", 2024-02-19T21:03:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache_release_free_interval": "10", 2024-02-19T21:03:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache_reservation": "0.050000", 2024-02-19T21:03:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-02-19T21:03:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache_trim_interval": "1", 2024-02-19T21:03:34.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache_trim_threshold": "262144", 2024-02-19T21:03:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-02-19T21:03:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-02-19T21:03:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_client_delegate_inos_pct": "50", 2024-02-19T21:03:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_client_prealloc_inos": "1000", 2024-02-19T21:03:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-02-19T21:03:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_connect_bootstrapping": "false", 2024-02-19T21:03:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_damage_table_max_entries": "10000", 2024-02-19T21:03:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-02-19T21:03:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_debug_auth_pins": "false", 2024-02-19T21:03:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_debug_frag": "false", 2024-02-19T21:03:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_debug_scatterstat": "false", 2024-02-19T21:03:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_debug_subtrees": "false", 2024-02-19T21:03:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_decay_halflife": "5.000000", 2024-02-19T21:03:34.558 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_default_dir_hash": "2", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_defer_session_stale": "true", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_deny_all_reconnect": "false", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dir_keys_per_op": "16384", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dir_max_commit_size": "10", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dir_max_entries": "0", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dirstat_min_interval": "1.000000", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dump_cache_after_rejoin": "false", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dump_cache_on_map": "false", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dump_cache_threshold_file": "0", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_early_reply": "true", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_enable_op_tracker": "true", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_enforce_unique_name": "true", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_export_ephemeral_distributed": "true", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-02-19T21:03:34.559 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_export_ephemeral_random": "true", 2024-02-19T21:03:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-02-19T21:03:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_forward_all_requests_to_auth": "false", 2024-02-19T21:03:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-02-19T21:03:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-02-19T21:03:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-02-19T21:03:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_health_cache_threshold": "1.500000", 2024-02-19T21:03:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_health_summarize_threshold": "10", 2024-02-19T21:03:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_heartbeat_grace": "15.000000", 2024-02-19T21:03:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_heartbeat_reset_grace": "1000", 2024-02-19T21:03:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_inject_health_dummy": "false", 2024-02-19T21:03:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-02-19T21:03:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_inject_migrator_session_race": "false", 2024-02-19T21:03:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-02-19T21:03:34.560 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-02-19T21:03:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-02-19T21:03:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_join_fs": "", 2024-02-19T21:03:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_journal_format": "1", 2024-02-19T21:03:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_create_at": "0", 2024-02-19T21:03:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_export_at": "0", 2024-02-19T21:03:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_import_at": "0", 2024-02-19T21:03:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_journal_at": "0", 2024-02-19T21:03:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_journal_expire_at": "0", 2024-02-19T21:03:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_journal_replay_at": "0", 2024-02-19T21:03:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_link_at": "0", 2024-02-19T21:03:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_mdstable_at": "0", 2024-02-19T21:03:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_openc_at": "0", 2024-02-19T21:03:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_rename_at": "0", 2024-02-19T21:03:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-02-19T21:03:34.561 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_events_per_segment": "1024", 2024-02-19T21:03:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_max_events": "-1", 2024-02-19T21:03:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_max_segments": "128", 2024-02-19T21:03:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_pause": "false", 2024-02-19T21:03:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_segment_size": "0", 2024-02-19T21:03:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_skip_corrupt_events": "false", 2024-02-19T21:03:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log_warn_factor": "2.000000", 2024-02-19T21:03:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_caps_per_client": "1048576", 2024-02-19T21:03:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_completed_flushes": "100000", 2024-02-19T21:03:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_completed_requests": "100000", 2024-02-19T21:03:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_export_size": "20971520", 2024-02-19T21:03:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_file_recover": "32", 2024-02-19T21:03:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_purge_files": "64", 2024-02-19T21:03:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_purge_ops": "8192", 2024-02-19T21:03:34.562 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-02-19T21:03:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-02-19T21:03:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_snaps_per_dir": "100", 2024-02-19T21:03:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_max_xattr_pairs_size": "65536", 2024-02-19T21:03:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_metrics_update_interval": "2", 2024-02-19T21:03:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_min_caps_per_client": "100", 2024-02-19T21:03:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_min_caps_working_set": "10000", 2024-02-19T21:03:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-02-19T21:03:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_numa_node": "-1", 2024-02-19T21:03:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-02-19T21:03:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_op_complaint_time": "30.000000", 2024-02-19T21:03:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_op_history_duration": "600", 2024-02-19T21:03:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_op_history_size": "20", 2024-02-19T21:03:34.563 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_op_log_threshold": "5", 2024-02-19T21:03:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_ping_grace": "15", 2024-02-19T21:03:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_ping_interval": "5", 2024-02-19T21:03:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-02-19T21:03:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-02-19T21:03:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_recall_max_caps": "30000", 2024-02-19T21:03:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-02-19T21:03:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_recall_max_decay_threshold": "131072", 2024-02-19T21:03:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-02-19T21:03:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_recall_warning_threshold": "262144", 2024-02-19T21:03:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_reconnect_timeout": "45.000000", 2024-02-19T21:03:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_replay_interval": "1.000000", 2024-02-19T21:03:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-02-19T21:03:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-02-19T21:03:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_root_ino_gid": "0", 2024-02-19T21:03:34.564 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_root_ino_uid": "0", 2024-02-19T21:03:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-02-19T21:03:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_blocklist_on_evict": "true", 2024-02-19T21:03:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_blocklist_on_timeout": "true", 2024-02-19T21:03:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-02-19T21:03:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-02-19T21:03:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-02-19T21:03:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-02-19T21:03:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-02-19T21:03:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_session_metadata_threshold": "16777216", 2024-02-19T21:03:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-02-19T21:03:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_shutdown_check": "0", 2024-02-19T21:03:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_skip_ino": "0", 2024-02-19T21:03:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_sleep_rank_change": "0.000000", 2024-02-19T21:03:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_snap_max_uid": "4294967294", 2024-02-19T21:03:34.565 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_snap_min_uid": "0", 2024-02-19T21:03:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_snap_rstat": "false", 2024-02-19T21:03:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_standby_replay_damaged": "false", 2024-02-19T21:03:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_task_status_update_interval": "2.000000", 2024-02-19T21:03:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_thrash_exports": "0", 2024-02-19T21:03:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_thrash_fragments": "0", 2024-02-19T21:03:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_tick_interval": "5.000000", 2024-02-19T21:03:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_valgrind_exit": "false", 2024-02-19T21:03:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_verify_backtrace": "1", 2024-02-19T21:03:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_verify_scatter": "false", 2024-02-19T21:03:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_wipe_ino_prealloc": "false", 2024-02-19T21:03:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_wipe_sessions": "false", 2024-02-19T21:03:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "mempool_debug": "false", 2024-02-19T21:03:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "memstore_debug_omit_block_device_write": "false", 2024-02-19T21:03:34.566 INFO:teuthology.orchestra.run.smithi163.stdout: "memstore_device_bytes": "1073741824", 2024-02-19T21:03:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "memstore_page_set": "false", 2024-02-19T21:03:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "memstore_page_size": "65536", 2024-02-19T21:03:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_client_bytes": "134217728", 2024-02-19T21:03:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_client_messages": "512", 2024-02-19T21:03:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-02-19T21:03:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_connect_retry_interval": "1.000000", 2024-02-19T21:03:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-02-19T21:03:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-02-19T21:03:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_disabled_modules": "", 2024-02-19T21:03:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-02-19T21:03:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_max_pg_num_change": "128", 2024-02-19T21:03:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_mds_bytes": "134217728", 2024-02-19T21:03:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_mds_messages": "128", 2024-02-19T21:03:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-02-19T21:03:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_mon_bytes": "134217728", 2024-02-19T21:03:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_mon_messages": "128", 2024-02-19T21:03:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_osd_bytes": "536870912", 2024-02-19T21:03:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_osd_messages": "8192", 2024-02-19T21:03:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_service_beacon_grace": "60.000000", 2024-02-19T21:03:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_standby_modules": "true", 2024-02-19T21:03:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_stats_period": "5", 2024-02-19T21:03:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_stats_threshold": "5", 2024-02-19T21:03:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_tick_period": "2", 2024-02-19T21:03:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-02-19T21:03:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_accept_timeout_factor": "2.000000", 2024-02-19T21:03:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_allow_pool_delete": "true", 2024-02-19T21:03:34.568 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_allow_pool_size_one": "false", 2024-02-19T21:03:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_auth_validate_all_caps": "true", 2024-02-19T21:03:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-02-19T21:03:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-02-19T21:03:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_bytes": "104857600", 2024-02-19T21:03:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_directed_command_retry": "2", 2024-02-19T21:03:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_hunt_interval": "3.000000", 2024-02-19T21:03:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-02-19T21:03:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-02-19T21:03:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-02-19T21:03:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_hunt_parallel": "3", 2024-02-19T21:03:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_log_interval": "1.000000", 2024-02-19T21:03:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-02-19T21:03:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_ping_interval": "10.000000", 2024-02-19T21:03:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_client_ping_timeout": "30.000000", 2024-02-19T21:03:34.569 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_clock_drift_allowed": "1.000000", 2024-02-19T21:03:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-02-19T21:03:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-02-19T21:03:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_file_level": "debug", 2024-02-19T21:03:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_file": "true", 2024-02-19T21:03:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_graylog": "false", 2024-02-19T21:03:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-02-19T21:03:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-02-19T21:03:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_stderr": "true", 2024-02-19T21:03:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-02-19T21:03:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-02-19T21:03:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-02-19T21:03:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_compact_on_bootstrap": "false", 2024-02-19T21:03:34.570 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_compact_on_start": "false", 2024-02-19T21:03:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_compact_on_trim": "true", 2024-02-19T21:03:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_con_tracker_persist_interval": "10", 2024-02-19T21:03:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_con_tracker_score_halflife": "43200", 2024-02-19T21:03:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_config_key_max_entry_size": "65536", 2024-02-19T21:03:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_cpu_threads": "4", 2024-02-19T21:03:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_crush_min_required_version": "hammer", 2024-02-19T21:03:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_daemon_bytes": "419430400", 2024-02-19T21:03:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-02-19T21:03:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_data_avail_crit": "5", 2024-02-19T21:03:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_data_avail_warn": "5", 2024-02-19T21:03:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_data_size_warn": "16106127360", 2024-02-19T21:03:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_block_osdmap_trim": "false", 2024-02-19T21:03:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-02-19T21:03:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_dump_json": "false", 2024-02-19T21:03:34.571 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-02-19T21:03:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_dump_transactions": "false", 2024-02-19T21:03:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_extra_checks": "false", 2024-02-19T21:03:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-02-19T21:03:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-02-19T21:03:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_no_require_octopus": "false", 2024-02-19T21:03:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_no_require_pacific": "false", 2024-02-19T21:03:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-02-19T21:03:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_delta_reset_interval": "10.000000", 2024-02-19T21:03:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_dns_srv_name": "ceph-mon", 2024-02-19T21:03:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_down_mkfs_grace": "60", 2024-02-19T21:03:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_election_default_strategy": "1", 2024-02-19T21:03:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_election_timeout": "5.000000", 2024-02-19T21:03:34.572 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_elector_ping_divisor": "2", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_elector_ping_timeout": "2.000000", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_enable_op_tracker": "true", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_fake_pool_delete": "false", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_force_quorum_join": "false", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_globalid_prealloc": "10000", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_health_detail_to_clog": "true", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_health_log_update_period": "5", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_health_max_detail": "50", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_health_to_clog": "true", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_health_to_clog_interval": "600", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_host": "172.21.15.163:6789/0", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_host_override": "", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_initial_members": "", 2024-02-19T21:03:34.573 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_keyvaluedb": "rocksdb", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_lease": "5.000000", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_log_max_summary": "50", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_log_entries_per_event": "4096", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_log_epochs": "500", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_mdsmap_epochs": "500", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_mgrmap_epochs": "500", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_osd": "10000", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_pg_per_osd": "10000", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_pool_pg_num": "65536", 2024-02-19T21:03:34.574 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-02-19T21:03:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-02-19T21:03:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mds_force_trim_to": "0", 2024-02-19T21:03:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mds_skip_sanity": "false", 2024-02-19T21:03:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_memory_autotune": "true", 2024-02-19T21:03:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_memory_target": "2147483648", 2024-02-19T21:03:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mgr_beacon_grace": "30", 2024-02-19T21:03:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-02-19T21:03:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mgr_digest_period": "5", 2024-02-19T21:03:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mgr_inactive_grace": "60", 2024-02-19T21:03:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mgr_mkfs_grace": "240", 2024-02-19T21:03:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-02-19T21:03:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_min_osdmap_epochs": "50", 2024-02-19T21:03:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_op_complaint_time": "30", 2024-02-19T21:03:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_op_history_duration": "600", 2024-02-19T21:03:34.575 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_op_history_size": "20", 2024-02-19T21:03:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_op_history_slow_op_size": "20", 2024-02-19T21:03:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_op_history_slow_op_threshold": "10", 2024-02-19T21:03:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_op_log_threshold": "5", 2024-02-19T21:03:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-02-19T21:03:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-02-19T21:03:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-02-19T21:03:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_auto_mark_in": "false", 2024-02-19T21:03:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_auto_mark_new_in": "true", 2024-02-19T21:03:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-02-19T21:03:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-02-19T21:03:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_cache_size": "500", 2024-02-19T21:03:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_cache_size_min": "134217728", 2024-02-19T21:03:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_crush_smoke_test": "true", 2024-02-19T21:03:34.576 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_destroyed_out_interval": "600", 2024-02-19T21:03:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_down_out_interval": "600", 2024-02-19T21:03:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-02-19T21:03:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-02-19T21:03:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_force_trim_to": "0", 2024-02-19T21:03:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_full_ratio": "0.950000", 2024-02-19T21:03:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-02-19T21:03:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_laggy_halflife": "3600", 2024-02-19T21:03:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_laggy_max_interval": "300", 2024-02-19T21:03:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_laggy_weight": "0.300000", 2024-02-19T21:03:34.577 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-02-19T21:03:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_max_creating_pgs": "1024", 2024-02-19T21:03:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_max_initial_pgs": "1024", 2024-02-19T21:03:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_min_down_reporters": "2", 2024-02-19T21:03:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-02-19T21:03:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-02-19T21:03:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-02-19T21:03:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_prime_pg_temp": "true", 2024-02-19T21:03:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-02-19T21:03:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-02-19T21:03:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_report_timeout": "900", 2024-02-19T21:03:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-02-19T21:03:34.578 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_warn_num_repaired": "10", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osd_warn_op_age": "32.000000", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osdmap_full_prune_interval": "2", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osdmap_full_prune_min": "15", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pg_stuck_threshold": "60", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pg_warn_min_objects": "10000", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pg_warn_min_per_osd": "0", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pool_quota_crit_threshold": "0", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_pool_quota_warn_threshold": "0", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_probe_timeout": "2.000000", 2024-02-19T21:03:34.579 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_reweight_max_change": "0.050000", 2024-02-19T21:03:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_reweight_max_osds": "4", 2024-02-19T21:03:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-02-19T21:03:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-02-19T21:03:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-02-19T21:03:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-02-19T21:03:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-02-19T21:03:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_scrub_interval": "86400", 2024-02-19T21:03:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_scrub_max_keys": "100", 2024-02-19T21:03:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_scrub_timeout": "300", 2024-02-19T21:03:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_session_timeout": "300", 2024-02-19T21:03:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_smart_report_timeout": "5", 2024-02-19T21:03:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_stat_smooth_intervals": "6", 2024-02-19T21:03:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-02-19T21:03:34.580 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_stretch_pool_min_size": "2", 2024-02-19T21:03:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_stretch_pool_size": "4", 2024-02-19T21:03:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-02-19T21:03:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_subscribe_interval": "86400.000000", 2024-02-19T21:03:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_sync_debug": "false", 2024-02-19T21:03:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_sync_max_payload_keys": "2000", 2024-02-19T21:03:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_sync_max_payload_size": "1048576", 2024-02-19T21:03:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_sync_provider_kill_at": "0", 2024-02-19T21:03:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_sync_requester_kill_at": "0", 2024-02-19T21:03:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_sync_timeout": "60.000000", 2024-02-19T21:03:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_target_pg_per_osd": "100", 2024-02-19T21:03:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_tick_interval": "5", 2024-02-19T21:03:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_timecheck_interval": "300.000000", 2024-02-19T21:03:34.581 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-02-19T21:03:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_older_version_delay": "604800", 2024-02-19T21:03:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-02-19T21:03:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-02-19T21:03:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-02-19T21:03:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-02-19T21:03:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-02-19T21:03:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-02-19T21:03:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_misplaced": "false", 2024-02-19T21:03:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-02-19T21:03:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_older_version": "true", 2024-02-19T21:03:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-02-19T21:03:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_pool_no_app": "false", 2024-02-19T21:03:34.582 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-02-19T21:03:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-02-19T21:03:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-02-19T21:03:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-02-19T21:03:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_on_too_few_osds": "false", 2024-02-19T21:03:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-02-19T21:03:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-02-19T21:03:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "monmap": "", 2024-02-19T21:03:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_op_threads": "3", 2024-02-19T21:03:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_buffer_size": "131072", 2024-02-19T21:03:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_cm": "false", 2024-02-19T21:03:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_device_name": "", 2024-02-19T21:03:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_dscp": "96", 2024-02-19T21:03:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-02-19T21:03:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_gid_idx": "0", 2024-02-19T21:03:34.583 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_local_gid": "", 2024-02-19T21:03:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_polling_us": "1000", 2024-02-19T21:03:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_port_num": "1", 2024-02-19T21:03:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-02-19T21:03:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-02-19T21:03:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_roce_ver": "1", 2024-02-19T21:03:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_send_buffers": "1024", 2024-02-19T21:03:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_sl": "3", 2024-02-19T21:03:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_support_srq": "true", 2024-02-19T21:03:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_rdma_type": "ib", 2024-02-19T21:03:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_async_reap_threshold": "5", 2024-02-19T21:03:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_before_connect": "false", 2024-02-19T21:03:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_ipv4": "true", 2024-02-19T21:03:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_ipv6": "false", 2024-02-19T21:03:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_msgr1": "true", 2024-02-19T21:03:34.584 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_msgr2": "false", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_port_max": "7300", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_port_min": "6800", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_prefer_ipv4": "false", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_retry_count": "3", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_bind_retry_delay": "5", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_blackhole_client": "false", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_blackhole_mds": "false", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_blackhole_mgr": "false", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_blackhole_mon": "false", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_blackhole_osd": "false", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_client_mode": "crc secure", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_cluster_mode": "crc secure", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_cluster_type": "", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_connection_idle_timeout": "900", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_connection_ready_timeout": "10", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_crc_data": "true", 2024-02-19T21:03:34.585 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_crc_header": "true", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_die_on_bad_msg": "false", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_die_on_bug": "true", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_die_on_old_message": "true", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_die_on_skipped_message": "false", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_die_on_unhandled_msg": "false", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_coremask": "0xF", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_hugepages": "", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_hw_flow_control": "true", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_lro": "true", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_memory_channel": "4", 2024-02-19T21:03:34.586 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_pmd": "", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_port_id": "0", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dump_corrupt_message_level": "1", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_dump_on_send": "false", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_initial_backoff": "0.200000", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_inject_delay_max": "1.000000", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_inject_delay_probability": "0.000000", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_inject_delay_type": "", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_inject_internal_delays": "0.000000", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_inject_socket_failures": "0", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_learn_addr_from_peer": "true", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_max_accept_failures": "4", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_max_backoff": "15.000000", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_mon_client_mode": "secure crc", 2024-02-19T21:03:34.587 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_mon_cluster_mode": "secure crc", 2024-02-19T21:03:34.588 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_mon_service_mode": "secure crc", 2024-02-19T21:03:34.588 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-02-19T21:03:34.588 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_pq_min_cost": "65536", 2024-02-19T21:03:34.588 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_public_type": "", 2024-02-19T21:03:34.588 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_service_mode": "crc secure", 2024-02-19T21:03:34.588 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_tcp_listen_backlog": "512", 2024-02-19T21:03:34.588 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_tcp_nodelay": "true", 2024-02-19T21:03:34.588 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-02-19T21:03:34.588 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_tcp_rcvbuf": "0", 2024-02-19T21:03:34.588 INFO:teuthology.orchestra.run.smithi163.stdout: "ms_type": "async+posix", 2024-02-19T21:03:34.588 INFO:teuthology.orchestra.run.smithi163.stdout: "no_config_file": "false", 2024-02-19T21:03:34.588 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_completion_locks_per_session": "32", 2024-02-19T21:03:34.588 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_debug_inject_relock_delay": "false", 2024-02-19T21:03:34.588 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_inflight_op_bytes": "104857600", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_inflight_ops": "1024", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_inject_no_watch_ping": "false", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_tick_interval": "5.000000", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter_timeout": "10.000000", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "objectstore_blackhole": "false", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "openssl_engine_opts": "", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_agent_delay_time": "5.000000", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_agent_hist_halflife": "1000", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_agent_max_low_ops": "2", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_agent_max_ops": "4", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_agent_quantize_effort": "0.100000", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_agent_slop": "0.020000", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-02-19T21:03:34.589 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_allow_recovery_below_min_size": "true", 2024-02-19T21:03:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_asio_thread_count": "2", 2024-02-19T21:03:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_async_recovery_min_cost": "100", 2024-02-19T21:03:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_auto_mark_unfound_lost": "false", 2024-02-19T21:03:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_backfill_retry_interval": "30.000000", 2024-02-19T21:03:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_backfill_scan_max": "512", 2024-02-19T21:03:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_backfill_scan_min": "64", 2024-02-19T21:03:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_backoff_on_degraded": "false", 2024-02-19T21:03:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_backoff_on_peering": "false", 2024-02-19T21:03:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_backoff_on_unfound": "true", 2024-02-19T21:03:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_beacon_report_interval": "300", 2024-02-19T21:03:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_bench_duration": "30", 2024-02-19T21:03:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-02-19T21:03:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_bench_max_block_size": "67108864", 2024-02-19T21:03:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_bench_small_size_max_iops": "100", 2024-02-19T21:03:34.590 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_blkin_trace_all": "false", 2024-02-19T21:03:34.591 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-02-19T21:03:34.591 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-02-19T21:03:34.591 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_check_for_log_corruption": "false", 2024-02-19T21:03:34.591 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-02-19T21:03:34.591 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-19T21:03:34.591 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-02-19T21:03:34.591 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-19T21:03:34.591 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_class_update_on_start": "true", 2024-02-19T21:03:34.591 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_client_message_cap": "256", 2024-02-19T21:03:34.591 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_client_message_size_cap": "524288000", 2024-02-19T21:03:34.591 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_client_op_priority": "63", 2024-02-19T21:03:34.591 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_client_watch_timeout": "30", 2024-02-19T21:03:34.591 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_command_max_records": "256", 2024-02-19T21:03:34.591 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_command_thread_suicide_timeout": "900", 2024-02-19T21:03:34.592 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_command_thread_timeout": "600", 2024-02-19T21:03:34.592 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_compact_on_start": "false", 2024-02-19T21:03:34.592 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-02-19T21:03:34.592 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_crush_chooseleaf_type": "1", 2024-02-19T21:03:34.592 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_crush_initial_weight": "-1.000000", 2024-02-19T21:03:34.592 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_crush_update_on_start": "true", 2024-02-19T21:03:34.592 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_crush_update_weight_set": "true", 2024-02-19T21:03:34.592 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-02-19T21:03:34.592 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-02-19T21:03:34.592 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-02-19T21:03:34.592 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_drop_ping_duration": "0", 2024-02-19T21:03:34.592 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-02-19T21:03:34.592 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_feed_pullee": "-1", 2024-02-19T21:03:34.592 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-02-19T21:03:34.593 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-02-19T21:03:34.593 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-02-19T21:03:34.593 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_misdirected_ops": "false", 2024-02-19T21:03:34.593 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_no_acting_change": "false", 2024-02-19T21:03:34.593 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_no_purge_strays": "false", 2024-02-19T21:03:34.593 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_op_order": "false", 2024-02-19T21:03:34.593 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_pg_log_writeout": "false", 2024-02-19T21:03:34.593 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_pretend_recovery_active": "false", 2024-02-19T21:03:34.593 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-02-19T21:03:34.593 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-02-19T21:03:34.593 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_shutdown": "false", 2024-02-19T21:03:34.593 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-02-19T21:03:34.593 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-02-19T21:03:34.593 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_verify_cached_snaps": "false", 2024-02-19T21:03:34.593 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_verify_missing_on_start": "false", 2024-02-19T21:03:34.594 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_verify_snaps": "false", 2024-02-19T21:03:34.594 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-02-19T21:03:34.594 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-02-19T21:03:34.594 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_deep_scrub_keys": "1024", 2024-02-19T21:03:34.594 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-02-19T21:03:34.594 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-02-19T21:03:34.594 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-02-19T21:03:34.594 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_deep_scrub_stride": "524288", 2024-02-19T21:03:34.594 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-02-19T21:03:34.594 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_default_data_pool_replay_window": "45", 2024-02-19T21:03:34.594 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_default_notify_timeout": "30", 2024-02-19T21:03:34.594 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_delete_sleep": "0.000000", 2024-02-19T21:03:34.594 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-02-19T21:03:34.594 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-02-19T21:03:34.594 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-02-19T21:03:34.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_discard_disconnected_ops": "true", 2024-02-19T21:03:34.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_enable_op_tracker": "true", 2024-02-19T21:03:34.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_erasure_code_plugins": "jerasure lrc isa", 2024-02-19T21:03:34.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-02-19T21:03:34.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-02-19T21:03:34.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_fast_info": "true", 2024-02-19T21:03:34.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_fast_shutdown": "true", 2024-02-19T21:03:34.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-02-19T21:03:34.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-02-19T21:03:34.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-02-19T21:03:34.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-02-19T21:03:34.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_function_tracing": "false", 2024-02-19T21:03:34.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_heartbeat_grace": "20", 2024-02-19T21:03:34.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_heartbeat_interval": "6", 2024-02-19T21:03:34.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-02-19T21:03:34.596 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_heartbeat_min_peers": "10", 2024-02-19T21:03:34.596 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_heartbeat_min_size": "2000", 2024-02-19T21:03:34.596 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_heartbeat_stale": "600", 2024-02-19T21:03:34.596 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-02-19T21:03:34.596 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_hit_set_max_size": "100000", 2024-02-19T21:03:34.596 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_hit_set_min_size": "1000", 2024-02-19T21:03:34.596 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-02-19T21:03:34.596 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-02-19T21:03:34.596 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-02-19T21:03:34.596 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-02-19T21:03:34.596 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-02-19T21:03:34.596 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_journal_flush_on_shutdown": "true", 2024-02-19T21:03:34.596 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_journal_size": "5120", 2024-02-19T21:03:34.596 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_kill_backfill_at": "0", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_loop_before_reset_tphandle": "64", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_map_cache_size": "50", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_map_dedup": "true", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_map_message_max": "40", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_map_message_max_bytes": "10485760", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_map_share_max_epochs": "40", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_attr_name_len": "100", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_attr_size": "0", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_backfills": "1", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_markdown_count": "5", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_markdown_period": "600", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_object_name_len": "2048", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_object_namespace_len": "256", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_object_size": "134217728", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_omap_entries_per_request": "1024", 2024-02-19T21:03:34.597 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_pg_blocked_by": "16", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_pg_log_entries": "10000", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_pgls": "1024", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_push_cost": "8388608", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_push_objects": "10", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_scrubs": "1", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_trimming_pgs": "2", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_write_op_reply_len": "64", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_max_write_size": "90", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_cost_per_byte_usec": "0.000000", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_cost_per_byte_usec_hdd": "5.200000", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_cost_per_byte_usec_ssd": "0.011000", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_cost_per_io_usec": "0.000000", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_cost_per_io_usec_hdd": "25000.000000", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_cost_per_io_usec_ssd": "50.000000", 2024-02-19T21:03:34.598 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-02-19T21:03:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-02-19T21:03:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-02-19T21:03:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_profile": "high_client_ops", 2024-02-19T21:03:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-02-19T21:03:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_background_best_effort_lim": "999999", 2024-02-19T21:03:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_background_best_effort_res": "1", 2024-02-19T21:03:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-02-19T21:03:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_background_recovery_lim": "999999", 2024-02-19T21:03:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_background_recovery_res": "1", 2024-02-19T21:03:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-02-19T21:03:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_client_lim": "999999", 2024-02-19T21:03:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_client_res": "1", 2024-02-19T21:03:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-02-19T21:03:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mclock_skip_benchmark": "false", 2024-02-19T21:03:34.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_memory_base": "805306368", 2024-02-19T21:03:34.600 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_memory_cache_min": "134217728", 2024-02-19T21:03:34.600 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-02-19T21:03:34.600 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-02-19T21:03:34.600 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_memory_target": "4294967296", 2024-02-19T21:03:34.600 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_memory_target_autotune": "false", 2024-02-19T21:03:34.600 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-02-19T21:03:34.600 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_min_pg_log_entries": "250", 2024-02-19T21:03:34.600 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_min_recovery_priority": "0", 2024-02-19T21:03:34.600 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mon_heartbeat_interval": "30", 2024-02-19T21:03:34.600 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-02-19T21:03:34.600 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mon_report_interval": "5", 2024-02-19T21:03:34.600 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mon_report_max_in_flight": "2", 2024-02-19T21:03:34.601 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-02-19T21:03:34.601 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_num_cache_shards": "32", 2024-02-19T21:03:34.601 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_num_op_tracker_shard": "32", 2024-02-19T21:03:34.601 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_numa_auto_affinity": "true", 2024-02-19T21:03:34.601 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_numa_node": "-1", 2024-02-19T21:03:34.601 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_numa_prefer_iface": "true", 2024-02-19T21:03:34.601 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-02-19T21:03:34.601 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_objecter_finishers": "1", 2024-02-19T21:03:34.601 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_objectstore": "bluestore", 2024-02-19T21:03:34.601 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_objectstore_fuse": "false", 2024-02-19T21:03:34.601 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_objectstore_tracing": "false", 2024-02-19T21:03:34.601 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_complaint_time": "30.000000", 2024-02-19T21:03:34.601 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_history_duration": "600", 2024-02-19T21:03:34.601 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_history_size": "20", 2024-02-19T21:03:34.601 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_history_slow_op_size": "20", 2024-02-19T21:03:34.602 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-02-19T21:03:34.602 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_log_threshold": "5", 2024-02-19T21:03:34.602 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_num_shards": "0", 2024-02-19T21:03:34.602 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_num_shards_hdd": "5", 2024-02-19T21:03:34.602 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_num_shards_ssd": "8", 2024-02-19T21:03:34.602 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_num_threads_per_shard": "0", 2024-02-19T21:03:34.602 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-02-19T21:03:34.602 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-02-19T21:03:34.602 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-02-19T21:03:34.602 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_pq_min_cost": "65536", 2024-02-19T21:03:34.602 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_queue": "wpq", 2024-02-19T21:03:34.602 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_queue_cut_off": "high", 2024-02-19T21:03:34.602 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_thread_suicide_timeout": "150", 2024-02-19T21:03:34.602 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_op_thread_timeout": "15", 2024-02-19T21:03:34.603 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_open_classes_on_start": "true", 2024-02-19T21:03:34.603 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_os_flags": "0", 2024-02-19T21:03:34.603 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_peering_op_priority": "255", 2024-02-19T21:03:34.603 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_delete_cost": "1048576", 2024-02-19T21:03:34.603 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_delete_priority": "5", 2024-02-19T21:03:34.603 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-02-19T21:03:34.603 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-02-19T21:03:34.603 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_log_dups_tracked": "3000", 2024-02-19T21:03:34.603 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_log_trim_max": "10000", 2024-02-19T21:03:34.603 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_log_trim_min": "100", 2024-02-19T21:03:34.603 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-02-19T21:03:34.603 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_object_context_cache_count": "64", 2024-02-19T21:03:34.604 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pg_stat_report_interval_max": "500", 2024-02-19T21:03:34.604 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-02-19T21:03:34.604 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-02-19T21:03:34.604 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-02-19T21:03:34.604 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-02-19T21:03:34.604 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-02-19T21:03:34.604 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-02-19T21:03:34.604 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_crush_rule": "-1", 2024-02-19T21:03:34.604 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_ec_fast_read": "false", 2024-02-19T21:03:34.604 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd", 2024-02-19T21:03:34.604 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_flag_bulk": "false", 2024-02-19T21:03:34.604 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-02-19T21:03:34.604 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_flag_nodelete": "false", 2024-02-19T21:03:34.605 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-02-19T21:03:34.605 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-02-19T21:03:34.605 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_flags": "0", 2024-02-19T21:03:34.605 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-02-19T21:03:34.605 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_min_size": "0", 2024-02-19T21:03:34.605 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-02-19T21:03:34.605 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_pg_num": "32", 2024-02-19T21:03:34.605 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_pgp_num": "0", 2024-02-19T21:03:34.605 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-02-19T21:03:34.605 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_size": "3", 2024-02-19T21:03:34.605 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_default_type": "replicated", 2024-02-19T21:03:34.605 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-02-19T21:03:34.605 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pool_use_gmt_hitset": "true", 2024-02-19T21:03:34.605 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_push_per_object_cost": "1000", 2024-02-19T21:03:34.606 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_read_ec_check_for_errors": "false", 2024-02-19T21:03:34.606 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recover_clone_overlap": "true", 2024-02-19T21:03:34.606 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recover_clone_overlap_limit": "10", 2024-02-19T21:03:34.606 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_cost": "20971520", 2024-02-19T21:03:34.606 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_delay_start": "0.000000", 2024-02-19T21:03:34.606 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_max_active": "0", 2024-02-19T21:03:34.606 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_max_active_hdd": "3", 2024-02-19T21:03:34.606 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_max_active_ssd": "10", 2024-02-19T21:03:34.606 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_max_chunk": "8388608", 2024-02-19T21:03:34.606 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-02-19T21:03:34.606 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_max_single_start": "1", 2024-02-19T21:03:34.606 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_op_priority": "3", 2024-02-19T21:03:34.606 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_op_warn_multiple": "16", 2024-02-19T21:03:34.606 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_priority": "5", 2024-02-19T21:03:34.606 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_retry_interval": "30.000000", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_sleep": "0.000000", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_repair_during_recovery": "false", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_requested_scrub_priority": "120", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_rollback_to_cluster_snap": "", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_auto_repair": "false", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_begin_hour": "0", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_begin_week_day": "0", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_chunk_max": "25", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_chunk_min": "5", 2024-02-19T21:03:34.607 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_cost": "52428800", 2024-02-19T21:03:34.608 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_during_recovery": "false", 2024-02-19T21:03:34.608 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_end_hour": "0", 2024-02-19T21:03:34.608 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_end_week_day": "0", 2024-02-19T21:03:34.608 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-02-19T21:03:34.608 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-02-19T21:03:34.608 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_invalid_stats": "true", 2024-02-19T21:03:34.608 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_load_threshold": "0.500000", 2024-02-19T21:03:34.608 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_max_interval": "604800.000000", 2024-02-19T21:03:34.608 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_max_preemptions": "5", 2024-02-19T21:03:34.608 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_min_interval": "86400.000000", 2024-02-19T21:03:34.608 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_priority": "5", 2024-02-19T21:03:34.608 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_scrub_sleep": "0.000000", 2024-02-19T21:03:34.608 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_shutdown_pgref_assert": "false", 2024-02-19T21:03:34.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_skip_data_digest": "false", 2024-02-19T21:03:34.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_smart_report_timeout": "5", 2024-02-19T21:03:34.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_snap_trim_cost": "1048576", 2024-02-19T21:03:34.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_snap_trim_priority": "5", 2024-02-19T21:03:34.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_snap_trim_sleep": "0.000000", 2024-02-19T21:03:34.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-02-19T21:03:34.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-02-19T21:03:34.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-02-19T21:03:34.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-02-19T21:03:34.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_target_transaction_size": "30", 2024-02-19T21:03:34.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-02-19T21:03:34.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-02-19T21:03:34.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-02-19T21:03:34.610 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-02-19T21:03:34.610 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-02-19T21:03:34.610 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-02-19T21:03:34.610 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-02-19T21:03:34.610 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_default_cache_mode": "writeback", 2024-02-19T21:03:34.610 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-02-19T21:03:34.610 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-02-19T21:03:34.610 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_tracing": "false", 2024-02-19T21:03:34.610 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_use_stale_snap": "false", 2024-02-19T21:03:34.610 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-02-19T21:03:34.610 INFO:teuthology.orchestra.run.smithi163.stdout: "osdc_blkin_trace_all": "false", 2024-02-19T21:03:34.610 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_kill_at": "0", 2024-02-19T21:03:34.610 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_max_join_drift": "10", 2024-02-19T21:03:34.610 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_min": "500", 2024-02-19T21:03:34.610 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_min_wait": "0.050000", 2024-02-19T21:03:34.611 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_propose_interval": "1.000000", 2024-02-19T21:03:34.611 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_service_trim_max": "500", 2024-02-19T21:03:34.611 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_service_trim_max_multiplier": "20", 2024-02-19T21:03:34.611 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_service_trim_min": "10", 2024-02-19T21:03:34.611 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_stash_full_interval": "25", 2024-02-19T21:03:34.611 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_trim_max": "500", 2024-02-19T21:03:34.611 INFO:teuthology.orchestra.run.smithi163.stdout: "paxos_trim_min": "250", 2024-02-19T21:03:34.611 INFO:teuthology.orchestra.run.smithi163.stdout: "perf": "true", 2024-02-19T21:03:34.611 INFO:teuthology.orchestra.run.smithi163.stdout: "pid_file": "", 2024-02-19T21:03:34.611 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-02-19T21:03:34.611 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-02-19T21:03:34.611 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "-", 2024-02-19T21:03:34.611 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrv": "", 2024-02-19T21:03:34.611 INFO:teuthology.orchestra.run.smithi163.stdout: "public_bind_addr": "-", 2024-02-19T21:03:34.612 INFO:teuthology.orchestra.run.smithi163.stdout: "public_network": "", 2024-02-19T21:03:34.612 INFO:teuthology.orchestra.run.smithi163.stdout: "public_network_interface": "", 2024-02-19T21:03:34.612 INFO:teuthology.orchestra.run.smithi163.stdout: "qat_compressor_enabled": "false", 2024-02-19T21:03:34.612 INFO:teuthology.orchestra.run.smithi163.stdout: "rados_mon_op_timeout": "0", 2024-02-19T21:03:34.612 INFO:teuthology.orchestra.run.smithi163.stdout: "rados_osd_op_timeout": "0", 2024-02-19T21:03:34.612 INFO:teuthology.orchestra.run.smithi163.stdout: "rados_tracing": "false", 2024-02-19T21:03:34.612 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_atime_update_interval": "60", 2024-02-19T21:03:34.612 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-02-19T21:03:34.612 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_balance_parent_reads": "false", 2024-02-19T21:03:34.612 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_balance_snap_reads": "false", 2024-02-19T21:03:34.613 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_blkin_trace_all": "false", 2024-02-19T21:03:34.613 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_blocklist_expire_seconds": "0", 2024-02-19T21:03:34.613 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_blocklist_on_break_lock": "true", 2024-02-19T21:03:34.613 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache": "true", 2024-02-19T21:03:34.613 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_block_writes_upfront": "false", 2024-02-19T21:03:34.613 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_max_dirty": "25165824", 2024-02-19T21:03:34.613 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-02-19T21:03:34.613 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_max_dirty_object": "0", 2024-02-19T21:03:34.613 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_policy": "writearound", 2024-02-19T21:03:34.613 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_size": "33554432", 2024-02-19T21:03:34.613 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_target_dirty": "16777216", 2024-02-19T21:03:34.613 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-02-19T21:03:34.613 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_clone_copy_on_read": "false", 2024-02-19T21:03:34.614 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_compression_hint": "none", 2024-02-19T21:03:34.614 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_concurrent_management_ops": "10", 2024-02-19T21:03:34.614 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-02-19T21:03:34.614 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_clone_format": "auto", 2024-02-19T21:03:34.614 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_data_pool": "", 2024-02-19T21:03:34.614 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_features": "61", 2024-02-19T21:03:34.614 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_format": "2", 2024-02-19T21:03:34.614 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_map_options": "", 2024-02-19T21:03:34.614 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_order": "22", 2024-02-19T21:03:34.614 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_pool": "rbd", 2024-02-19T21:03:34.614 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-02-19T21:03:34.614 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_stripe_count": "0", 2024-02-19T21:03:34.614 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_default_stripe_unit": "0", 2024-02-19T21:03:34.614 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_disable_zero_copy_writes": "true", 2024-02-19T21:03:34.614 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_discard_granularity_bytes": "65536", 2024-02-19T21:03:34.615 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-02-19T21:03:34.615 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_enable_alloc_hint": "true", 2024-02-19T21:03:34.615 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-02-19T21:03:34.615 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_io_scheduler": "simple", 2024-02-19T21:03:34.615 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-02-19T21:03:34.615 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_commit_age": "5.000000", 2024-02-19T21:03:34.615 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-02-19T21:03:34.615 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-02-19T21:03:34.615 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-02-19T21:03:34.615 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-02-19T21:03:34.615 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_object_flush_interval": "0", 2024-02-19T21:03:34.615 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-02-19T21:03:34.615 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-02-19T21:03:34.615 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_order": "24", 2024-02-19T21:03:34.616 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_pool": "", 2024-02-19T21:03:34.616 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_journal_splay_width": "4", 2024-02-19T21:03:34.616 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_localize_parent_reads": "false", 2024-02-19T21:03:34.616 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_localize_snap_reads": "false", 2024-02-19T21:03:34.616 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-02-19T21:03:34.616 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-02-19T21:03:34.616 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-02-19T21:03:34.616 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-02-19T21:03:34.616 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-02-19T21:03:34.616 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-02-19T21:03:34.616 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_image_policy_type": "simple", 2024-02-19T21:03:34.616 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-02-19T21:03:34.616 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-02-19T21:03:34.616 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-02-19T21:03:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-02-19T21:03:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-02-19T21:03:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-02-19T21:03:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-02-19T21:03:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_memory_autotune": "true", 2024-02-19T21:03:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_memory_base": "805306368", 2024-02-19T21:03:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-02-19T21:03:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-02-19T21:03:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-02-19T21:03:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-02-19T21:03:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_memory_target": "4294967296", 2024-02-19T21:03:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-02-19T21:03:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-02-19T21:03:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-02-19T21:03:34.617 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirroring_delete_delay": "0", 2024-02-19T21:03:34.618 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-02-19T21:03:34.618 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirroring_replay_delay": "0", 2024-02-19T21:03:34.618 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-02-19T21:03:34.618 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-02-19T21:03:34.618 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_move_to_trash_on_remove": "false", 2024-02-19T21:03:34.618 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-02-19T21:03:34.618 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_mtime_update_interval": "60", 2024-02-19T21:03:34.618 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_non_blocking_aio": "true", 2024-02-19T21:03:34.618 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_op_thread_timeout": "60", 2024-02-19T21:03:34.618 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_op_threads": "1", 2024-02-19T21:03:34.618 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_parent_cache_enabled": "false", 2024-02-19T21:03:34.618 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_persistent_cache_mode": "disabled", 2024-02-19T21:03:34.618 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_persistent_cache_path": "/tmp", 2024-02-19T21:03:34.618 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_persistent_cache_size": "1073741824", 2024-02-19T21:03:34.619 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_plugins": "", 2024-02-19T21:03:34.619 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_bps_burst": "0", 2024-02-19T21:03:34.619 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-02-19T21:03:34.619 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_bps_limit": "0", 2024-02-19T21:03:34.619 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_iops_burst": "0", 2024-02-19T21:03:34.619 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-02-19T21:03:34.619 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_iops_limit": "0", 2024-02-19T21:03:34.619 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_read_bps_burst": "0", 2024-02-19T21:03:34.619 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-02-19T21:03:34.619 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_read_bps_limit": "0", 2024-02-19T21:03:34.619 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_read_iops_burst": "0", 2024-02-19T21:03:34.619 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-02-19T21:03:34.619 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_read_iops_limit": "0", 2024-02-19T21:03:34.620 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_schedule_tick_min": "50", 2024-02-19T21:03:34.620 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_write_bps_burst": "0", 2024-02-19T21:03:34.620 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-02-19T21:03:34.620 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_write_bps_limit": "0", 2024-02-19T21:03:34.620 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_write_iops_burst": "0", 2024-02-19T21:03:34.620 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-02-19T21:03:34.620 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_qos_write_iops_limit": "0", 2024-02-19T21:03:34.620 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_quiesce_notification_attempts": "10", 2024-02-19T21:03:34.620 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_read_from_replica_policy": "default", 2024-02-19T21:03:34.620 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-02-19T21:03:34.620 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_readahead_max_bytes": "524288", 2024-02-19T21:03:34.620 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_readahead_trigger_requests": "10", 2024-02-19T21:03:34.621 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_request_timed_out_seconds": "30", 2024-02-19T21:03:34.621 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_skip_partial_discard": "true", 2024-02-19T21:03:34.621 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-02-19T21:03:34.621 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_tracing": "false", 2024-02-19T21:03:34.621 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_validate_names": "true", 2024-02-19T21:03:34.621 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd_validate_pool": "true", 2024-02-19T21:03:34.621 INFO:teuthology.orchestra.run.smithi163.stdout: "restapi_base_url": "", 2024-02-19T21:03:34.621 INFO:teuthology.orchestra.run.smithi163.stdout: "restapi_log_level": "", 2024-02-19T21:03:34.621 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_acl_grants_max_num": "100", 2024-02-19T21:03:34.621 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_admin_entry": "admin", 2024-02-19T21:03:34.621 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_barbican_url": "", 2024-02-19T21:03:34.621 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-02-19T21:03:34.621 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-02-19T21:03:34.621 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_bucket_index_max_aio": "128", 2024-02-19T21:03:34.622 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-02-19T21:03:34.622 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_bucket_quota_ttl": "600", 2024-02-19T21:03:34.622 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_cache_enabled": "true", 2024-02-19T21:03:34.622 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_cache_expiry_interval": "900", 2024-02-19T21:03:34.622 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_cache_lru_size": "10000", 2024-02-19T21:03:34.622 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_content_length_compat": "false", 2024-02-19T21:03:34.622 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_copy_obj_progress": "true", 2024-02-19T21:03:34.622 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-02-19T21:03:34.622 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_cors_rules_max_num": "100", 2024-02-19T21:03:34.622 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_cross_domain_policy": "", 2024-02-19T21:03:34.622 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_default_encryption_key": "", 2024-02-19T21:03:34.622 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_addr": "", 2024-02-19T21:03:34.623 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_ca_path": "", 2024-02-19T21:03:34.623 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_client_cert": "", 2024-02-19T21:03:34.623 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_client_key": "", 2024-02-19T21:03:34.623 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-02-19T21:03:34.623 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_password": "", 2024-02-19T21:03:34.623 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-02-19T21:03:34.623 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_kmip_username": "", 2024-02-19T21:03:34.623 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_require_ssl": "true", 2024-02-19T21:03:34.623 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-02-19T21:03:34.623 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-02-19T21:03:34.623 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_suppress_logs": "true", 2024-02-19T21:03:34.623 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_addr": "", 2024-02-19T21:03:34.623 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_auth": "token", 2024-02-19T21:03:34.623 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_namespace": "", 2024-02-19T21:03:34.623 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_prefix": "", 2024-02-19T21:03:34.624 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-02-19T21:03:34.624 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-02-19T21:03:34.624 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-02-19T21:03:34.624 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-02-19T21:03:34.624 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_token_file": "", 2024-02-19T21:03:34.624 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-02-19T21:03:34.624 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_curl_buffersize": "524288", 2024-02-19T21:03:34.624 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_curl_low_speed_limit": "1024", 2024-02-19T21:03:34.624 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_curl_low_speed_time": "300", 2024-02-19T21:03:34.624 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-02-19T21:03:34.624 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-02-19T21:03:34.624 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_data_log_changes_size": "1000", 2024-02-19T21:03:34.624 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_data_log_num_shards": "128", 2024-02-19T21:03:34.625 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-02-19T21:03:34.625 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_data_log_window": "30", 2024-02-19T21:03:34.625 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_data_notify_interval_msec": "200", 2024-02-19T21:03:34.625 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-02-19T21:03:34.625 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-02-19T21:03:34.625 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_default_data_log_backing": "fifo", 2024-02-19T21:03:34.625 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-02-19T21:03:34.625 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_default_region_info_oid": "default.region", 2024-02-19T21:03:34.625 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-02-19T21:03:34.625 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-02-19T21:03:34.625 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_defer_to_bucket_acls": "", 2024-02-19T21:03:34.625 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-02-19T21:03:34.625 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-02-19T21:03:34.625 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-02-19T21:03:34.626 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-02-19T21:03:34.626 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-02-19T21:03:34.626 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-02-19T21:03:34.626 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-02-19T21:03:34.626 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-02-19T21:03:34.626 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_data_res": "500.000000", 2024-02-19T21:03:34.626 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-02-19T21:03:34.626 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-02-19T21:03:34.626 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-02-19T21:03:34.626 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-02-19T21:03:34.626 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dns_name": "", 2024-02-19T21:03:34.626 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dns_s3website_name": "", 2024-02-19T21:03:34.626 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_dynamic_resharding": "true", 2024-02-19T21:03:34.626 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-02-19T21:03:34.627 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enable_gc_threads": "true", 2024-02-19T21:03:34.627 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enable_lc_threads": "true", 2024-02-19T21:03:34.627 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enable_ops_log": "false", 2024-02-19T21:03:34.627 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enable_quota_threads": "true", 2024-02-19T21:03:34.627 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enable_static_website": "false", 2024-02-19T21:03:34.627 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enable_usage_log": "false", 2024-02-19T21:03:34.627 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_enforce_swift_acls": "true", 2024-02-19T21:03:34.627 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_exit_timeout_secs": "120", 2024-02-19T21:03:34.627 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_expose_bucket": "false", 2024-02-19T21:03:34.627 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_extended_http_attrs": "", 2024-02-19T21:03:34.627 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_fcgi_socket_backlog": "1024", 2024-02-19T21:03:34.627 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-02-19T21:03:34.627 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_frontends": "beast port=7480", 2024-02-19T21:03:34.627 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_max_concurrent_io": "10", 2024-02-19T21:03:34.628 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_max_deferred": "50", 2024-02-19T21:03:34.628 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-02-19T21:03:34.628 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_max_objs": "32", 2024-02-19T21:03:34.628 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_max_queue_size": "134213632", 2024-02-19T21:03:34.628 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_max_trim_chunk": "16", 2024-02-19T21:03:34.628 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_obj_min_wait": "7200", 2024-02-19T21:03:34.628 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_processor_max_time": "3600", 2024-02-19T21:03:34.628 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_gc_processor_period": "3600", 2024-02-19T21:03:34.628 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-02-19T21:03:34.628 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_get_obj_window_size": "16777216", 2024-02-19T21:03:34.628 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_healthcheck_disabling_path": "", 2024-02-19T21:03:34.628 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_host": "", 2024-02-19T21:03:34.628 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ignore_get_invalid_range": "false", 2024-02-19T21:03:34.629 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_init_timeout": "300", 2024-02-19T21:03:34.629 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-02-19T21:03:34.629 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-02-19T21:03:34.629 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-02-19T21:03:34.629 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_domain": "", 2024-02-19T21:03:34.629 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_password": "", 2024-02-19T21:03:34.629 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_password_path": "", 2024-02-19T21:03:34.629 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_project": "", 2024-02-19T21:03:34.629 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_tenant": "", 2024-02-19T21:03:34.629 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_token": "", 2024-02-19T21:03:34.629 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_token_path": "", 2024-02-19T21:03:34.629 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_admin_user": "", 2024-02-19T21:03:34.629 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_api_version": "2", 2024-02-19T21:03:34.630 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_barbican_domain": "", 2024-02-19T21:03:34.630 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_barbican_password": "", 2024-02-19T21:03:34.630 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_barbican_project": "", 2024-02-19T21:03:34.630 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_barbican_tenant": "", 2024-02-19T21:03:34.630 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_barbican_user": "", 2024-02-19T21:03:34.630 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_implicit_tenants": "false", 2024-02-19T21:03:34.630 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_token_cache_size": "10000", 2024-02-19T21:03:34.630 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_url": "", 2024-02-19T21:03:34.630 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_keystone_verify_ssl": "true", 2024-02-19T21:03:34.630 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lc_debug_interval": "-1", 2024-02-19T21:03:34.631 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lc_lock_max_time": "90", 2024-02-19T21:03:34.631 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lc_max_objs": "32", 2024-02-19T21:03:34.631 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lc_max_rules": "1000", 2024-02-19T21:03:34.631 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lc_max_worker": "3", 2024-02-19T21:03:34.631 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lc_max_wp_worker": "3", 2024-02-19T21:03:34.631 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lc_thread_delay": "0", 2024-02-19T21:03:34.631 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-02-19T21:03:34.631 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ldap_dnattr": "uid", 2024-02-19T21:03:34.631 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-02-19T21:03:34.631 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ldap_searchfilter": "", 2024-02-19T21:03:34.631 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-02-19T21:03:34.631 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ldap_uri": "ldaps://", 2024-02-19T21:03:34.631 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-02-19T21:03:34.631 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-02-19T21:03:34.632 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-02-19T21:03:34.632 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_log_http_headers": "", 2024-02-19T21:03:34.632 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_log_nonexistent_bucket": "false", 2024-02-19T21:03:34.632 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-02-19T21:03:34.632 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_log_object_name_utc": "false", 2024-02-19T21:03:34.632 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-02-19T21:03:34.632 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_attr_name_len": "0", 2024-02-19T21:03:34.632 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_attr_size": "0", 2024-02-19T21:03:34.632 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_attrs_num_in_req": "0", 2024-02-19T21:03:34.632 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_chunk_size": "4194304", 2024-02-19T21:03:34.632 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_concurrent_requests": "1024", 2024-02-19T21:03:34.632 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_dynamic_shards": "1999", 2024-02-19T21:03:34.632 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_listing_results": "1000", 2024-02-19T21:03:34.633 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_notify_retries": "10", 2024-02-19T21:03:34.633 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_objs_per_shard": "100000", 2024-02-19T21:03:34.633 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_put_param_size": "1048576", 2024-02-19T21:03:34.633 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_put_size": "5368709120", 2024-02-19T21:03:34.633 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_max_slo_entries": "1000", 2024-02-19T21:03:34.633 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_md_log_max_shards": "64", 2024-02-19T21:03:34.633 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_md_notify_interval_msec": "200", 2024-02-19T21:03:34.633 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-02-19T21:03:34.633 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_mp_lock_max_time": "600", 2024-02-19T21:03:34.633 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-02-19T21:03:34.633 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_multipart_min_part_size": "5242880", 2024-02-19T21:03:34.633 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-02-19T21:03:34.633 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-02-19T21:03:34.634 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_fhcache_size": "2017", 2024-02-19T21:03:34.634 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-02-19T21:03:34.634 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_lru_lanes": "5", 2024-02-19T21:03:34.634 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_max_gc": "300", 2024-02-19T21:03:34.634 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-02-19T21:03:34.634 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_run_gc_threads": "false", 2024-02-19T21:03:34.634 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_run_lc_threads": "false", 2024-02-19T21:03:34.634 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_run_quota_threads": "false", 2024-02-19T21:03:34.634 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_run_sync_thread": "false", 2024-02-19T21:03:34.634 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-02-19T21:03:34.634 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-02-19T21:03:34.634 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_num_async_rados_threads": "32", 2024-02-19T21:03:34.634 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_num_control_oids": "8", 2024-02-19T21:03:34.634 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_num_rados_handles": "1", 2024-02-19T21:03:34.634 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_numa_node": "-1", 2024-02-19T21:03:34.635 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_obj_stripe_size": "4194304", 2024-02-19T21:03:34.635 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-02-19T21:03:34.635 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_objexp_chunk_size": "100", 2024-02-19T21:03:34.635 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_objexp_gc_interval": "600", 2024-02-19T21:03:34.635 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_objexp_hints_num_shards": "127", 2024-02-19T21:03:34.635 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-02-19T21:03:34.635 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-02-19T21:03:34.635 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_op_thread_timeout": "600", 2024-02-19T21:03:34.635 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_op_tracing": "false", 2024-02-19T21:03:34.635 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_opa_token": "", 2024-02-19T21:03:34.635 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_opa_url": "", 2024-02-19T21:03:34.635 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_opa_verify_ssl": "true", 2024-02-19T21:03:34.635 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-02-19T21:03:34.635 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ops_log_file_path": "", 2024-02-19T21:03:34.635 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ops_log_rados": "true", 2024-02-19T21:03:34.636 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_ops_log_socket_path": "", 2024-02-19T21:03:34.636 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-02-19T21:03:34.636 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-02-19T21:03:34.636 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_period_push_interval": "2.000000", 2024-02-19T21:03:34.636 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_period_push_interval_max": "30.000000", 2024-02-19T21:03:34.636 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_period_root_pool": ".rgw.root", 2024-02-19T21:03:34.636 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_port": "", 2024-02-19T21:03:34.636 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_print_continue": "true", 2024-02-19T21:03:34.636 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_print_prohibited_content_length": "false", 2024-02-19T21:03:34.636 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-02-19T21:03:34.636 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-02-19T21:03:34.636 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-02-19T21:03:34.637 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-02-19T21:03:34.637 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_rados_tracing": "false", 2024-02-19T21:03:34.637 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_realm": "", 2024-02-19T21:03:34.637 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-02-19T21:03:34.637 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_region": "", 2024-02-19T21:03:34.637 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_region_root_pool": ".rgw.root", 2024-02-19T21:03:34.637 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_relaxed_region_enforcement": "false", 2024-02-19T21:03:34.637 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-02-19T21:03:34.637 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-02-19T21:03:34.637 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_request_uri": "", 2024-02-19T21:03:34.637 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_reshard_batch_size": "64", 2024-02-19T21:03:34.637 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-02-19T21:03:34.637 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_reshard_max_aio": "128", 2024-02-19T21:03:34.637 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_reshard_num_logs": "16", 2024-02-19T21:03:34.637 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_reshard_thread_interval": "600", 2024-02-19T21:03:34.638 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_resolve_cname": "false", 2024-02-19T21:03:34.638 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_rest_getusage_op_compat": "false", 2024-02-19T21:03:34.638 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_run_sync_thread": "true", 2024-02-19T21:03:34.638 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-02-19T21:03:34.638 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_s3_auth_use_keystone": "false", 2024-02-19T21:03:34.638 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_s3_auth_use_ldap": "false", 2024-02-19T21:03:34.638 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_s3_auth_use_rados": "true", 2024-02-19T21:03:34.638 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_s3_auth_use_sts": "false", 2024-02-19T21:03:34.638 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_s3_success_create_obj_status": "0", 2024-02-19T21:03:34.638 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-02-19T21:03:34.638 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_scheduler_type": "throttler", 2024-02-19T21:03:34.638 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_script_uri": "", 2024-02-19T21:03:34.638 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_service_provider_name": "", 2024-02-19T21:03:34.638 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-02-19T21:03:34.638 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_socket_path": "", 2024-02-19T21:03:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sts_client_id": "", 2024-02-19T21:03:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sts_client_secret": "", 2024-02-19T21:03:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sts_entry": "sts", 2024-02-19T21:03:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sts_key": "sts", 2024-02-19T21:03:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sts_max_session_duration": "43200", 2024-02-19T21:03:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sts_min_session_duration": "900", 2024-02-19T21:03:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sts_token_introspection_url": "", 2024-02-19T21:03:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_account_in_url": "false", 2024-02-19T21:03:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_auth_entry": "auth", 2024-02-19T21:03:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_auth_url": "", 2024-02-19T21:03:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_custom_header": "", 2024-02-19T21:03:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_enforce_content_length": "false", 2024-02-19T21:03:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_need_stats": "true", 2024-02-19T21:03:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_tenant_name": "", 2024-02-19T21:03:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_token_expiration": "86400", 2024-02-19T21:03:34.640 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_url": "", 2024-02-19T21:03:34.640 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_url_prefix": "swift", 2024-02-19T21:03:34.640 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_swift_versioning_enabled": "false", 2024-02-19T21:03:34.640 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-02-19T21:03:34.640 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_lease_period": "120", 2024-02-19T21:03:34.640 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-02-19T21:03:34.640 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_log_trim_interval": "1200", 2024-02-19T21:03:34.640 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-02-19T21:03:34.640 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-02-19T21:03:34.640 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-02-19T21:03:34.640 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_obj_etag_verify": "false", 2024-02-19T21:03:34.640 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_trace_history_size": "4096", 2024-02-19T21:03:34.640 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-02-19T21:03:34.640 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-02-19T21:03:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_thread_pool_size": "512", 2024-02-19T21:03:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_torrent_comment": "", 2024-02-19T21:03:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_torrent_createby": "", 2024-02-19T21:03:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_torrent_encoding": "", 2024-02-19T21:03:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_torrent_flag": "false", 2024-02-19T21:03:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_torrent_origin": "", 2024-02-19T21:03:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_torrent_sha_unit": "524288", 2024-02-19T21:03:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_torrent_tracker": "", 2024-02-19T21:03:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_trust_forwarded_https": "false", 2024-02-19T21:03:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-02-19T21:03:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_usage_log_tick_interval": "30", 2024-02-19T21:03:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_usage_max_shards": "32", 2024-02-19T21:03:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_usage_max_user_shards": "1", 2024-02-19T21:03:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_use_opa_authz": "false", 2024-02-19T21:03:34.641 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-02-19T21:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_default_quota_max_size": "-1", 2024-02-19T21:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_max_buckets": "1000", 2024-02-19T21:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-02-19T21:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-02-19T21:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_quota_sync_interval": "86400", 2024-02-19T21:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-02-19T21:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_user_unique_email": "true", 2024-02-19T21:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_verify_ssl": "true", 2024-02-19T21:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_website_routing_rules_max_num": "50", 2024-02-19T21:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_zone": "", 2024-02-19T21:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-02-19T21:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_zonegroup": "", 2024-02-19T21:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-02-19T21:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_block_size": "4096", 2024-02-19T21:03:34.642 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-02-19T21:03:34.643 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-02-19T21:03:34.643 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-02-19T21:03:34.643 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-02-19T21:03:34.643 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cache_shard_bits": "4", 2024-02-19T21:03:34.643 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cache_size": "536870912", 2024-02-19T21:03:34.643 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cache_type": "binned_lru", 2024-02-19T21:03:34.643 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cf_compact_on_deletion": "false", 2024-02-19T21:03:34.643 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-02-19T21:03:34.643 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-02-19T21:03:34.643 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_collect_compaction_stats": "false", 2024-02-19T21:03:34.643 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_collect_extended_stats": "false", 2024-02-19T21:03:34.643 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_collect_memory_stats": "false", 2024-02-19T21:03:34.643 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-02-19T21:03:34.643 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_index_type": "binary_search", 2024-02-19T21:03:34.644 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_log_to_ceph_log": "true", 2024-02-19T21:03:34.644 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_metadata_block_size": "4096", 2024-02-19T21:03:34.644 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_partition_filters": "false", 2024-02-19T21:03:34.644 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_perf": "false", 2024-02-19T21:03:34.644 INFO:teuthology.orchestra.run.smithi163.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-02-19T21:03:34.644 INFO:teuthology.orchestra.run.smithi163.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-02-19T21:03:34.644 INFO:teuthology.orchestra.run.smithi163.stdout: "rotating_keys_renewal_timeout": "10", 2024-02-19T21:03:34.644 INFO:teuthology.orchestra.run.smithi163.stdout: "run_dir": "/var/run/ceph", 2024-02-19T21:03:34.644 INFO:teuthology.orchestra.run.smithi163.stdout: "setgroup": "ceph", 2024-02-19T21:03:34.644 INFO:teuthology.orchestra.run.smithi163.stdout: "setuser": "ceph", 2024-02-19T21:03:34.644 INFO:teuthology.orchestra.run.smithi163.stdout: "setuser_match_path": "", 2024-02-19T21:03:34.644 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_misplaced_ratio": "0.050000", 2024-02-19T21:03:34.645 INFO:teuthology.orchestra.run.smithi163.stdout: "thp": "false", 2024-02-19T21:03:34.645 INFO:teuthology.orchestra.run.smithi163.stdout: "threadpool_default_timeout": "60", 2024-02-19T21:03:34.645 INFO:teuthology.orchestra.run.smithi163.stdout: "threadpool_empty_queue_max_wait": "2", 2024-02-19T21:03:34.645 INFO:teuthology.orchestra.run.smithi163.stdout: "throttler_perf_counter": "true" 2024-02-19T21:03:34.645 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:03:34.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:32.937505+0000 mgr.y (mgr.14550) 96 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-02-19T21:03:34.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:33.048043+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:03:34.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:33.048921+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:03:34.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:33.734997+0000 osd.3 (osd.3) 55 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:03:34.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:33.736361+0000 osd.3 (osd.3) 56 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:03:34.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:32.937505+0000 mgr.y (mgr.14550) 96 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-02-19T21:03:34.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:33.048043+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:03:34.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:33.048921+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:03:34.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:33.734997+0000 osd.3 (osd.3) 55 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:03:34.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:33.736361+0000 osd.3 (osd.3) 56 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:03:34.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:32.937505+0000 mgr.y (mgr.14550) 96 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-02-19T21:03:34.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:33.048043+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:03:34.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:33.048921+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:03:34.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:33.734997+0000 osd.3 (osd.3) 55 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:03:34.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:33.736361+0000 osd.3 (osd.3) 56 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:03:35.272 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-02-19T21:03:35.543 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:35.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:35.085102+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.e scrub starts 2024-02-19T21:03:35.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:35.086574+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.e scrub ok 2024-02-19T21:03:35.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:35.085102+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.e scrub starts 2024-02-19T21:03:35.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:35.086574+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.e scrub ok 2024-02-19T21:03:35.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:35.085102+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.e scrub starts 2024-02-19T21:03:35.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:35.086574+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.e scrub ok 2024-02-19T21:03:36.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:34.938148+0000 mgr.y (mgr.14550) 97 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:03:36.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:34.938148+0000 mgr.y (mgr.14550) 97 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:03:36.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:34.938148+0000 mgr.y (mgr.14550) 97 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:03:37.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.270+0000 7fa400176700 1 -- 172.21.15.163:0/4172637981 >> v1:172.21.15.163:6790/0 conn(0x7fa3f81042c0 legacy=0x7fa3f8106760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:37.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.270+0000 7fa400176700 1 -- 172.21.15.163:0/4172637981 shutdown_connections 2024-02-19T21:03:37.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.270+0000 7fa400176700 1 -- 172.21.15.163:0/4172637981 >> 172.21.15.163:0/4172637981 conn(0x7fa3f806fdc0 msgr2=0x7fa3f80f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:37.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.270+0000 7fa400176700 1 -- 172.21.15.163:0/4172637981 shutdown_connections 2024-02-19T21:03:37.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.270+0000 7fa400176700 1 -- 172.21.15.163:0/4172637981 wait complete. 2024-02-19T21:03:37.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.271+0000 7fa400176700 1 librados: starting msgr at 2024-02-19T21:03:37.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.271+0000 7fa400176700 1 librados: starting objecter 2024-02-19T21:03:37.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.271+0000 7fa400176700 1 Processor -- start 2024-02-19T21:03:37.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.271+0000 7fa400176700 1 -- start start 2024-02-19T21:03:37.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.271+0000 7fa400176700 1 librados: setting wanted keys 2024-02-19T21:03:37.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.271+0000 7fa400176700 1 librados: calling monclient init 2024-02-19T21:03:37.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.271+0000 7fa400176700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa3f80fd000 con 0x7fa3f81042c0 2024-02-19T21:03:37.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.271+0000 7fa400176700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa3f80fd1d0 con 0x7fa3f80fd9a0 2024-02-19T21:03:37.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.271+0000 7fa400176700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa3f80fd3c0 con 0x7fa3f8100780 2024-02-19T21:03:37.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.271+0000 7fa3fd711700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fa3f8100780 0x7fa3f8114d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:48990/0 (socket says 172.21.15.163:48990) 2024-02-19T21:03:37.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.271+0000 7fa3fd711700 1 -- 172.21.15.163:0/4229413356 learned_addr learned my addr 172.21.15.163:0/4229413356 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:37.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 10 client.?.objecter ms_handle_connect 0x7fa3f8100780 2024-02-19T21:03:37.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:37.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 10 client.?.objecter ms_handle_connect 0x7fa3f81042c0 2024-02-19T21:03:37.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:37.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 10 client.?.objecter ms_handle_connect 0x7fa3f80fd9a0 2024-02-19T21:03:37.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:37.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052776907 0 0) 0x7fa3f80fd3c0 con 0x7fa3f8100780 2024-02-19T21:03:37.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3e4003800 con 0x7fa3f8100780 2024-02-19T21:03:37.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262681730 0 0) 0x7fa3f80fd000 con 0x7fa3f81042c0 2024-02-19T21:03:37.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f80fd3c0 con 0x7fa3f81042c0 2024-02-19T21:03:37.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761446399 0 0) 0x7fa3f80fd1d0 con 0x7fa3f80fd9a0 2024-02-19T21:03:37.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f80fd000 con 0x7fa3f80fd9a0 2024-02-19T21:03:37.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3579024921 0 0) 0x7fa3f80fd3c0 con 0x7fa3f81042c0 2024-02-19T21:03:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3f80fd1d0 con 0x7fa3f81042c0 2024-02-19T21:03:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3f0003190 con 0x7fa3f81042c0 2024-02-19T21:03:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3248946081 0 0) 0x7fa3f80fd000 con 0x7fa3f80fd9a0 2024-02-19T21:03:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3f80fd3c0 con 0x7fa3f80fd9a0 2024-02-19T21:03:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1230840863 0 0) 0x7fa3e4003800 con 0x7fa3f8100780 2024-02-19T21:03:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3f80fd000 con 0x7fa3f8100780 2024-02-19T21:03:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3e8001840 con 0x7fa3f80fd9a0 2024-02-19T21:03:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3f40019f0 con 0x7fa3f8100780 2024-02-19T21:03:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 835195708 0 0) 0x7fa3f80fd1d0 con 0x7fa3f81042c0 2024-02-19T21:03:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.163:6790/0 conn(0x7fa3f8100780 legacy=0x7fa3f8114d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.193:6789/0 conn(0x7fa3f80fd9a0 legacy=0x7fa3f80fc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f811cbb0 con 0x7fa3f81042c0 2024-02-19T21:03:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa400176700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f8119be0 con 0x7fa3f81042c0 2024-02-19T21:03:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa400176700 10 client.14784.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa400176700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f811a050 con 0x7fa3f81042c0 2024-02-19T21:03:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa400176700 1 librados: init done 2024-02-19T21:03:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fa3f00014c0 con 0x7fa3f81042c0 2024-02-19T21:03:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.272+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3f0003960 con 0x7fa3f81042c0 2024-02-19T21:03:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.273+0000 7fa400176700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3dc004f40 con 0x7fa3f81042c0 2024-02-19T21:03:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.273+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fa3f0002650 con 0x7fa3f81042c0 2024-02-19T21:03:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.274+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(87..87 src has 32..87) v4 ==== 6172+0+0 (unknown 2939065595 0 0) 0x7fa3f0076350 con 0x7fa3f81042c0 2024-02-19T21:03:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.274+0000 7fa3eeffd700 10 client.14784.objecter ms_dispatch 0x7fa3f8001550 osd_map(87..87 src has 32..87) v4 2024-02-19T21:03:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.274+0000 7fa3eeffd700 3 client.14784.objecter handle_osd_map got epochs [87,87] > 0 2024-02-19T21:03:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.274+0000 7fa3eeffd700 3 client.14784.objecter handle_osd_map decoding full epoch 87 2024-02-19T21:03:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.274+0000 7fa3eeffd700 20 client.14784.objecter dump_active .. 0 homeless 2024-02-19T21:03:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.274+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1111 (unknown 409899127 0 545308006) 0x7fa3f0078740 con 0x7fa3f81042c0 2024-02-19T21:03:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.274+0000 7fa3eeffd700 10 client.14784.objecter ms_handle_connect 0x7fa3e4063ce0 2024-02-19T21:03:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.274+0000 7fa400176700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fa3dc004d40 con 0x7fa3dc002730 2024-02-19T21:03:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.274+0000 7fa3eeffd700 10 client.14784.objecter ms_handle_connect 0x7fa3dc002730 2024-02-19T21:03:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.274+0000 7fa3eeffd700 10 client.14784.objecter resend_mon_ops 2024-02-19T21:03:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.275+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518250794 0 0) 0x7fa3dc004d40 con 0x7fa3dc002730 2024-02-19T21:03:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.275+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f80fd1d0 con 0x7fa3dc002730 2024-02-19T21:03:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.275+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3573539224 0 0) 0x7fa3f80fd1d0 con 0x7fa3dc002730 2024-02-19T21:03:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.275+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3dc004d40 con 0x7fa3dc002730 2024-02-19T21:03:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.275+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3f4001fe0 con 0x7fa3dc002730 2024-02-19T21:03:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.275+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 59574362 0 0) 0x7fa3dc004d40 con 0x7fa3dc002730 2024-02-19T21:03:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.275+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6789/0 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3dc00a390 con 0x7fa3dc002730 2024-02-19T21:03:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.275+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+8167 (unknown 0 0 1565185916) 0x7fa3f4002300 con 0x7fa3dc002730 2024-02-19T21:03:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.275+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.163:6789/0 conn(0x7fa3dc002730 legacy=0x7fa3dc0054b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.275+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.163:6789/0 conn(0x7fa3dc002730 legacy=0x7fa3dc0054b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-19T21:03:37.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.282+0000 7fa400176700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fa3dc00bcd0 con 0x7fa3dc00a770 2024-02-19T21:03:37.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.282+0000 7fa3eeffd700 10 client.14784.objecter ms_handle_connect 0x7fa3dc00a770 2024-02-19T21:03:37.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.282+0000 7fa3eeffd700 10 client.14784.objecter resend_mon_ops 2024-02-19T21:03:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.282+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025230869 0 0) 0x7fa3dc00bcd0 con 0x7fa3dc00a770 2024-02-19T21:03:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.282+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3dc004d40 con 0x7fa3dc00a770 2024-02-19T21:03:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.282+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1179584063 0 0) 0x7fa3dc004d40 con 0x7fa3dc00a770 2024-02-19T21:03:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.282+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3dc00bcd0 con 0x7fa3dc00a770 2024-02-19T21:03:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.282+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3f4005160 con 0x7fa3dc00a770 2024-02-19T21:03:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.282+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3136526525 0 0) 0x7fa3dc00bcd0 con 0x7fa3dc00a770 2024-02-19T21:03:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.282+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6789/0 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7fa3dc00bf40 con 0x7fa3dc00a770 2024-02-19T21:03:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.283+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.0 v1:172.21.15.163:6789/0 5 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '0' (not observed, change may require restart) ) v1 ==== 83+0+3 (unknown 1093878403 0 2423995997) 0x7fa3f4005440 con 0x7fa3dc00a770 2024-02-19T21:03:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.283+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.163:6789/0 conn(0x7fa3dc00a770 legacy=0x7fa3dc008be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.283+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.163:6789/0 conn(0x7fa3dc00a770 legacy=0x7fa3dc008be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-19T21:03:37.276 INFO:teuthology.orchestra.run.smithi163.stdout:mon.a: {} 2024-02-19T21:03:37.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.284+0000 7fa400176700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fa3dc00a070 con 0x7fa3dc009b80 2024-02-19T21:03:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.284+0000 7fa3eeffd700 10 client.14784.objecter ms_handle_connect 0x7fa3dc009b80 2024-02-19T21:03:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.284+0000 7fa3eeffd700 10 client.14784.objecter resend_mon_ops 2024-02-19T21:03:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.284+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669465701 0 0) 0x7fa3dc00a070 con 0x7fa3dc009b80 2024-02-19T21:03:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.284+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3dc00bcd0 con 0x7fa3dc009b80 2024-02-19T21:03:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.284+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4133259237 0 0) 0x7fa3dc00bcd0 con 0x7fa3dc009b80 2024-02-19T21:03:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.284+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3dc00a070 con 0x7fa3dc009b80 2024-02-19T21:03:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.284+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3f4005940 con 0x7fa3dc009b80 2024-02-19T21:03:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.285+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 969575450 0 0) 0x7fa3dc00a070 con 0x7fa3dc009b80 2024-02-19T21:03:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.285+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.193:6789/0 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3dc0128a0 con 0x7fa3dc009b80 2024-02-19T21:03:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.285+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.1 v1:172.21.15.193:6789/0 5 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+8167 (unknown 0 0 1565185916) 0x7fa3f4005be0 con 0x7fa3dc009b80 2024-02-19T21:03:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.285+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.193:6789/0 conn(0x7fa3dc009b80 legacy=0x7fa3dc001d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.285+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.193:6789/0 conn(0x7fa3dc009b80 legacy=0x7fa3dc001d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-19T21:03:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.285+0000 7fa3fe713700 1 -- 172.21.15.163:0/4229413356 reap_dead start 2024-02-19T21:03:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.291+0000 7fa400176700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fa3dc014070 con 0x7fa3dc012bc0 2024-02-19T21:03:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.292+0000 7fa3eeffd700 10 client.14784.objecter ms_handle_connect 0x7fa3dc012bc0 2024-02-19T21:03:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.292+0000 7fa3eeffd700 10 client.14784.objecter resend_mon_ops 2024-02-19T21:03:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.292+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250890835 0 0) 0x7fa3dc014070 con 0x7fa3dc012bc0 2024-02-19T21:03:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.292+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3dc00a070 con 0x7fa3dc012bc0 2024-02-19T21:03:37.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.292+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3416761066 0 0) 0x7fa3dc00a070 con 0x7fa3dc012bc0 2024-02-19T21:03:37.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.292+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3dc014070 con 0x7fa3dc012bc0 2024-02-19T21:03:37.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.292+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3f4006270 con 0x7fa3dc012bc0 2024-02-19T21:03:37.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.292+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2614391982 0 0) 0x7fa3dc014070 con 0x7fa3dc012bc0 2024-02-19T21:03:37.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.292+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.193:6789/0 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7fa3dc0142a0 con 0x7fa3dc012bc0 2024-02-19T21:03:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.294+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.1 v1:172.21.15.193:6789/0 5 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '0' (not observed, change may require restart) ) v1 ==== 83+0+3 (unknown 1093878403 0 2423995997) 0x7fa3f4006520 con 0x7fa3dc012bc0 2024-02-19T21:03:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.294+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.193:6789/0 conn(0x7fa3dc012bc0 legacy=0x7fa3dc011140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.294+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.193:6789/0 conn(0x7fa3dc012bc0 legacy=0x7fa3dc011140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-19T21:03:37.286 INFO:teuthology.orchestra.run.smithi163.stdout:mon.b: {} 2024-02-19T21:03:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.294+0000 7fa400176700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fa3dc012600 con 0x7fa3dc011a00 2024-02-19T21:03:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.294+0000 7fa3eeffd700 10 client.14784.objecter ms_handle_connect 0x7fa3dc011a00 2024-02-19T21:03:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.294+0000 7fa3eeffd700 10 client.14784.objecter resend_mon_ops 2024-02-19T21:03:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.294+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044989129 0 0) 0x7fa3dc012600 con 0x7fa3dc011a00 2024-02-19T21:03:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.294+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3dc014070 con 0x7fa3dc011a00 2024-02-19T21:03:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.294+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3526183257 0 0) 0x7fa3dc014070 con 0x7fa3dc011a00 2024-02-19T21:03:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.294+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3dc012600 con 0x7fa3dc011a00 2024-02-19T21:03:37.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.294+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3f4006a20 con 0x7fa3dc011a00 2024-02-19T21:03:37.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.295+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4107352091 0 0) 0x7fa3dc012600 con 0x7fa3dc011a00 2024-02-19T21:03:37.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.295+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6790/0 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa3dc01ac00 con 0x7fa3dc011a00 2024-02-19T21:03:37.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.295+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.2 v1:172.21.15.163:6790/0 5 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+8167 (unknown 0 0 1565185916) 0x7fa3f4006a40 con 0x7fa3dc011a00 2024-02-19T21:03:37.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.295+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.163:6790/0 conn(0x7fa3dc011a00 legacy=0x7fa3dc011e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:37.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.295+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.163:6790/0 conn(0x7fa3dc011a00 legacy=0x7fa3dc011e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-19T21:03:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.301+0000 7fa400176700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fa3dc01b190 con 0x7fa3dc01ad70 2024-02-19T21:03:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.301+0000 7fa3eeffd700 10 client.14784.objecter ms_handle_connect 0x7fa3dc01ad70 2024-02-19T21:03:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.301+0000 7fa3eeffd700 10 client.14784.objecter resend_mon_ops 2024-02-19T21:03:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.301+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475163855 0 0) 0x7fa3dc01b190 con 0x7fa3dc01ad70 2024-02-19T21:03:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.301+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3dc014070 con 0x7fa3dc01ad70 2024-02-19T21:03:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.302+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1035035026 0 0) 0x7fa3dc014070 con 0x7fa3dc01ad70 2024-02-19T21:03:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.302+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3dc01b190 con 0x7fa3dc01ad70 2024-02-19T21:03:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.302+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3f4006f40 con 0x7fa3dc01ad70 2024-02-19T21:03:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.302+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1509832919 0 0) 0x7fa3dc01b190 con 0x7fa3dc01ad70 2024-02-19T21:03:37.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.302+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 --> v1:172.21.15.163:6790/0 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) v1 -- 0x7fa3dc01c410 con 0x7fa3dc01ad70 2024-02-19T21:03:37.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.303+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 <== mon.2 v1:172.21.15.163:6790/0 5 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '0' (not observed, change may require restart) ) v1 ==== 83+0+3 (unknown 1093878403 0 2423995997) 0x7fa3f4007180 con 0x7fa3dc01ad70 2024-02-19T21:03:37.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.303+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.163:6790/0 conn(0x7fa3dc01ad70 legacy=0x7fa3dc0194a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:37.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.303+0000 7fa3eeffd700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.163:6790/0 conn(0x7fa3dc01ad70 legacy=0x7fa3dc0194a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-19T21:03:37.296 INFO:teuthology.orchestra.run.smithi163.stdout:mon.c: {} 2024-02-19T21:03:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.303+0000 7fa400176700 10 librados: watch_flush enter 2024-02-19T21:03:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.303+0000 7fa400176700 10 librados: watch_flush exit 2024-02-19T21:03:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.303+0000 7fa400176700 20 client.14784.objecter shutdown clearing up homeless session... 2024-02-19T21:03:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.303+0000 7fa400176700 10 client.14784.objecter successfully canceled tick 2024-02-19T21:03:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.303+0000 7fa400176700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.163:6800/1552514681 conn(0x7fa3e4063ce0 legacy=0x7fa3e40661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.303+0000 7fa400176700 1 -- 172.21.15.163:0/4229413356 >> v1:172.21.15.163:6789/0 conn(0x7fa3f81042c0 legacy=0x7fa3f8118460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.303+0000 7fa3fe713700 1 -- 172.21.15.163:0/4229413356 reap_dead start 2024-02-19T21:03:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.303+0000 7fa400176700 1 -- 172.21.15.163:0/4229413356 shutdown_connections 2024-02-19T21:03:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.303+0000 7fa400176700 1 -- 172.21.15.163:0/4229413356 >> 172.21.15.163:0/4229413356 conn(0x7fa3f806fdc0 msgr2=0x7fa3f80f4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.304+0000 7fa400176700 1 -- 172.21.15.163:0/4229413356 shutdown_connections 2024-02-19T21:03:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.304+0000 7fa400176700 1 -- 172.21.15.163:0/4229413356 wait complete. 2024-02-19T21:03:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:37.304+0000 7fa400176700 1 librados: shutdown 2024-02-19T21:03:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-19T21:03:37.297 INFO:teuthology.orchestra.run.smithi163.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-19T21:03:37.298 INFO:teuthology.orchestra.run.smithi163.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2024-02-19T21:03:37.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:35.996652+0000 osd.2 (osd.2) 37 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:03:37.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:35.999571+0000 osd.2 (osd.2) 38 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:03:37.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:36.003614+0000 osd.6 (osd.6) 31 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:03:37.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:36.004575+0000 osd.6 (osd.6) 32 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:03:37.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:37.275336+0000 mon.a (mon.0) 716 : audit [DBG] from='client.? v1:172.21.15.163:0/4229413356' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:03:37.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:35.996652+0000 osd.2 (osd.2) 37 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:03:37.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:35.999571+0000 osd.2 (osd.2) 38 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:03:37.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:36.003614+0000 osd.6 (osd.6) 31 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:03:37.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:36.004575+0000 osd.6 (osd.6) 32 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:03:37.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:37.275336+0000 mon.a (mon.0) 716 : audit [DBG] from='client.? v1:172.21.15.163:0/4229413356' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:03:37.764 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:35.996652+0000 osd.2 (osd.2) 37 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:03:37.765 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:35.999571+0000 osd.2 (osd.2) 38 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:03:37.765 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:36.003614+0000 osd.6 (osd.6) 31 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:03:37.765 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:36.004575+0000 osd.6 (osd.6) 32 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:03:37.765 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:37.275336+0000 mon.a (mon.0) 716 : audit [DBG] from='client.? v1:172.21.15.163:0/4229413356' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:03:38.051 INFO:teuthology.run_tasks:Running task workunit... 2024-02-19T21:03:38.066 INFO:tasks.workunit:Pulling workunits from ref 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:03:38.066 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-19T21:03:38.066 DEBUG:teuthology.orchestra.run.smithi163:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-19T21:03:38.067 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-02-19T21:03:38.067 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:03:38.067 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:03:38.067 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-02-19T21:03:38.067 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-19T21:03:38.067 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-19T21:03:38.067 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-02-19T21:03:38.068 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-02-19T21:03:38.068 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:03:38.068 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-02-19T21:03:38.068 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:03:38.069 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-02-19T21:03:38.069 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-19T21:03:38.069 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-02-19T21:03:38.069 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:03:38.071 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:03:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:03:38.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:03:38.092 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-02-19T21:03:38.093 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-02-19T21:03:38.107 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:38.129 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:03:38.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:38.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:03:38.150 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-02-19T21:03:38.150 DEBUG:teuthology.orchestra.run.smithi163:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-02-19T21:03:38.209 INFO:tasks.workunit:timeout=3h 2024-02-19T21:03:38.209 INFO:tasks.workunit:cleanup=True 2024-02-19T21:03:38.209 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:03:38.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:03:38.263 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:38.267 INFO:tasks.workunit.client.0.smithi163.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-19T21:03:38.284 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:03:38.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:38.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:03:38.374 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:38.377 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:38.388 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:03:38.422 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:38.671 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:36.938550+0000 mgr.y (mgr.14550) 98 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:03:38.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:36.938550+0000 mgr.y (mgr.14550) 98 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:03:38.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:36.938550+0000 mgr.y (mgr.14550) 98 : cluster [DBG] pgmap v77: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:03:38.976 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:03:38.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:38.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:03:39.083 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:03:39.115 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:39.136 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:03:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:39.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:03:39.240 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:03:39.269 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:40.118 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:03:40.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:40.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:03:40.222 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:03:40.255 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:40.278 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:03:40.280 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:03:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:03:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:40.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:03:40.382 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:03:40.412 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:40.561 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:03:40.563 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:40.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:03:40.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:38.938965+0000 mgr.y (mgr.14550) 99 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:40.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:39.466153+0000 osd.0 (osd.0) 45 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:03:40.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:39.467054+0000 osd.0 (osd.0) 46 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:03:40.600 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:40.664 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:03:40.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:38.938965+0000 mgr.y (mgr.14550) 99 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:40.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:39.466153+0000 osd.0 (osd.0) 45 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:03:40.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:39.467054+0000 osd.0 (osd.0) 46 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:03:40.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:38.938965+0000 mgr.y (mgr.14550) 99 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:40.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:39.466153+0000 osd.0 (osd.0) 45 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:03:40.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:39.467054+0000 osd.0 (osd.0) 46 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:03:40.691 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:40.839 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:03:40.842 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:40.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:03:40.943 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:03:40.975 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:40.997 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:03:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:41.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:03:41.101 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:03:41.129 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:41.289 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:03:41.289 INFO:teuthology.orchestra.run.smithi163.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-19T21:03:41.289 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:03:41.377 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:03:41.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:41.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:03:41.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.480+0000 7fd2d84d8700 1 -- 172.21.15.163:0/3763215904 >> v1:172.21.15.163:6789/0 conn(0x7fd2d01042c0 legacy=0x7fd2d0106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:41.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.480+0000 7fd2d84d8700 1 -- 172.21.15.163:0/3763215904 shutdown_connections 2024-02-19T21:03:41.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.480+0000 7fd2d84d8700 1 -- 172.21.15.163:0/3763215904 >> 172.21.15.163:0/3763215904 conn(0x7fd2d00f35a0 msgr2=0x7fd2d00f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:41.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.480+0000 7fd2d84d8700 1 -- 172.21.15.163:0/3763215904 shutdown_connections 2024-02-19T21:03:41.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.480+0000 7fd2d84d8700 1 -- 172.21.15.163:0/3763215904 wait complete. 2024-02-19T21:03:41.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.481+0000 7fd2d84d8700 1 librados: starting msgr at 2024-02-19T21:03:41.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.481+0000 7fd2d84d8700 1 librados: starting objecter 2024-02-19T21:03:41.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.481+0000 7fd2d84d8700 1 Processor -- start 2024-02-19T21:03:41.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.481+0000 7fd2d84d8700 1 -- start start 2024-02-19T21:03:41.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.481+0000 7fd2d84d8700 1 librados: setting wanted keys 2024-02-19T21:03:41.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.481+0000 7fd2d84d8700 1 librados: calling monclient init 2024-02-19T21:03:41.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.481+0000 7fd2d84d8700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd2d01194e0 con 0x7fd2d01042c0 2024-02-19T21:03:41.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.481+0000 7fd2d84d8700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd2d011a6c0 con 0x7fd2d0100780 2024-02-19T21:03:41.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.481+0000 7fd2d84d8700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd2d011b8c0 con 0x7fd2d00fd980 2024-02-19T21:03:41.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.481+0000 7fd2d6274700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fd2d00fd980 0x7fd2d0102420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:49026/0 (socket says 172.21.15.163:49026) 2024-02-19T21:03:41.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.481+0000 7fd2d6274700 1 -- 172.21.15.163:0/996921367 learned_addr learned my addr 172.21.15.163:0/996921367 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:41.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.481+0000 7fd2c77fe700 10 client.?.objecter ms_handle_connect 0x7fd2d00fd980 2024-02-19T21:03:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.481+0000 7fd2c77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.481+0000 7fd2c77fe700 10 client.?.objecter ms_handle_connect 0x7fd2d0100780 2024-02-19T21:03:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.481+0000 7fd2c77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230789603 0 0) 0x7fd2d011b8c0 con 0x7fd2d00fd980 2024-02-19T21:03:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2bc0033f0 con 0x7fd2d00fd980 2024-02-19T21:03:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326631204 0 0) 0x7fd2d011a6c0 con 0x7fd2d0100780 2024-02-19T21:03:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d011b8c0 con 0x7fd2d0100780 2024-02-19T21:03:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 10 client.?.objecter ms_handle_connect 0x7fd2d01042c0 2024-02-19T21:03:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267705243 0 0) 0x7fd2d01194e0 con 0x7fd2d01042c0 2024-02-19T21:03:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d011a6c0 con 0x7fd2d01042c0 2024-02-19T21:03:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1292859762 0 0) 0x7fd2bc0033f0 con 0x7fd2d00fd980 2024-02-19T21:03:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2d01194e0 con 0x7fd2d00fd980 2024-02-19T21:03:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd2cc001980 con 0x7fd2d00fd980 2024-02-19T21:03:41.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1326840102 0 0) 0x7fd2d011b8c0 con 0x7fd2d0100780 2024-02-19T21:03:41.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2bc0033f0 con 0x7fd2d0100780 2024-02-19T21:03:41.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd2c00019d0 con 0x7fd2d0100780 2024-02-19T21:03:41.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 438189290 0 0) 0x7fd2d011a6c0 con 0x7fd2d01042c0 2024-02-19T21:03:41.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2d011b8c0 con 0x7fd2d01042c0 2024-02-19T21:03:41.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd2c8002370 con 0x7fd2d01042c0 2024-02-19T21:03:41.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4004917840 0 0) 0x7fd2d01194e0 con 0x7fd2d00fd980 2024-02-19T21:03:41.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 >> v1:172.21.15.193:6789/0 conn(0x7fd2d0100780 legacy=0x7fd2d0102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:41.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 >> v1:172.21.15.163:6789/0 conn(0x7fd2d01042c0 legacy=0x7fd2d0117d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:41.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d011cac0 con 0x7fd2d00fd980 2024-02-19T21:03:41.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2d84d8700 1 -- 172.21.15.163:0/996921367 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d011a8b0 con 0x7fd2d00fd980 2024-02-19T21:03:41.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2d84d8700 10 client.34718.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:41.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2d84d8700 1 -- 172.21.15.163:0/996921367 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d011ad20 con 0x7fd2d00fd980 2024-02-19T21:03:41.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2d84d8700 1 librados: init done 2024-02-19T21:03:41.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fd2cc002c70 con 0x7fd2d00fd980 2024-02-19T21:03:41.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.482+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd2cc005ab0 con 0x7fd2d00fd980 2024-02-19T21:03:41.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.483+0000 7fd2d84d8700 1 -- 172.21.15.163:0/996921367 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b4004f20 con 0x7fd2d00fd980 2024-02-19T21:03:41.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.483+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fd2cc001e50 con 0x7fd2d00fd980 2024-02-19T21:03:41.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.484+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(87..87 src has 32..87) v4 ==== 6172+0+0 (unknown 2939065595 0 0) 0x7fd2cc0784b0 con 0x7fd2d00fd980 2024-02-19T21:03:41.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.484+0000 7fd2c77fe700 10 client.34718.objecter ms_dispatch 0x7fd2d0001550 osd_map(87..87 src has 32..87) v4 2024-02-19T21:03:41.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.484+0000 7fd2c77fe700 3 client.34718.objecter handle_osd_map got epochs [87,87] > 0 2024-02-19T21:03:41.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.484+0000 7fd2c77fe700 3 client.34718.objecter handle_osd_map decoding full epoch 87 2024-02-19T21:03:41.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.484+0000 7fd2c77fe700 20 client.34718.objecter dump_active .. 0 homeless 2024-02-19T21:03:41.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.484+0000 7fd2c77fe700 10 client.34718.objecter ms_handle_connect 0x7fd2bc063df0 2024-02-19T21:03:41.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.488+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd2cc04c040 con 0x7fd2d00fd980 2024-02-19T21:03:41.482 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:03:41.510 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:41.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:41.669+0000 7fd2d84d8700 1 -- 172.21.15.163:0/996921367 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fd2b4005750 con 0x7fd2d00fd980 2024-02-19T21:03:41.708 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:03:41.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:40.029088+0000 mgr.y (mgr.14550) 100 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:41.710 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:40.029088+0000 mgr.y (mgr.14550) 100 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:41.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:41.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:03:41.717 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:41.812 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:03:41.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:40.029088+0000 mgr.y (mgr.14550) 100 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:41.844 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:41.864 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:03:41.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:41.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:03:41.968 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:03:41.996 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:42.193 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:03:42.196 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:42.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:03:42.297 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:03:42.329 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:42.350 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:03:42.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:42.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:03:42.454 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:03:42.484 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:42.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:42.513+0000 7fd2c77fe700 1 -- 172.21.15.163:0/996921367 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v88) v1 ==== 87+0+0 (unknown 578100353 0 0) 0x7fd2cc04a450 con 0x7fd2d00fd980 2024-02-19T21:03:42.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:42.516+0000 7fd2d84d8700 10 librados: watch_flush enter 2024-02-19T21:03:42.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:42.516+0000 7fd2d84d8700 10 librados: watch_flush exit 2024-02-19T21:03:42.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:42.516+0000 7fd2d84d8700 20 client.34718.objecter shutdown clearing up homeless session... 2024-02-19T21:03:42.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:42.516+0000 7fd2d84d8700 10 client.34718.objecter successfully canceled tick 2024-02-19T21:03:42.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:42.517+0000 7fd2d84d8700 1 -- 172.21.15.163:0/996921367 >> v1:172.21.15.163:6800/1552514681 conn(0x7fd2bc063df0 legacy=0x7fd2bc0662b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:42.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:42.517+0000 7fd2d84d8700 1 -- 172.21.15.163:0/996921367 >> v1:172.21.15.163:6790/0 conn(0x7fd2d00fd980 legacy=0x7fd2d0102420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:42.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:42.522+0000 7fd2d84d8700 1 -- 172.21.15.163:0/996921367 shutdown_connections 2024-02-19T21:03:42.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:42.522+0000 7fd2d84d8700 1 -- 172.21.15.163:0/996921367 >> 172.21.15.163:0/996921367 conn(0x7fd2d00f35a0 msgr2=0x7fd2d0107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:42.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:42.524+0000 7fd2d84d8700 1 -- 172.21.15.163:0/996921367 shutdown_connections 2024-02-19T21:03:42.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:42.525+0000 7fd2d84d8700 1 -- 172.21.15.163:0/996921367 wait complete. 2024-02-19T21:03:42.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:42.525+0000 7fd2d84d8700 1 librados: shutdown 2024-02-19T21:03:42.521 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:03:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:42.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:03:42.525 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:03:42.625 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:03:42.655 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:40.939681+0000 mgr.y (mgr.14550) 101 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:42.656 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:41.671097+0000 mon.c (mon.2) 27 : audit [INF] from='client.? v1:172.21.15.163:0/996921367' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:03:42.656 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:41.671363+0000 mon.a (mon.0) 717 : audit [INF] from='client.34718 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:03:42.658 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:42.680 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:03:42.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:42.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:03:42.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:40.939681+0000 mgr.y (mgr.14550) 101 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:42.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:41.671097+0000 mon.c (mon.2) 27 : audit [INF] from='client.? v1:172.21.15.163:0/996921367' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:03:42.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:41.671363+0000 mon.a (mon.0) 717 : audit [INF] from='client.34718 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:03:42.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:40.939681+0000 mgr.y (mgr.14550) 101 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:42.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:41.671097+0000 mon.c (mon.2) 27 : audit [INF] from='client.? v1:172.21.15.163:0/996921367' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:03:42.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:41.671363+0000 mon.a (mon.0) 717 : audit [INF] from='client.34718 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:03:42.783 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:03:42.810 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:43.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:42.499906+0000 mon.a (mon.0) 718 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:03:43.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:42.513700+0000 mon.a (mon.0) 719 : audit [INF] from='client.34718 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:03:43.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:42.513754+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-19T21:03:43.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:42.499906+0000 mon.a (mon.0) 718 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:03:43.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:42.513700+0000 mon.a (mon.0) 719 : audit [INF] from='client.34718 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:03:43.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:42.513754+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-19T21:03:43.740 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:03:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:03:43] "GET /metrics HTTP/1.1" 200 172169 "" "Prometheus/2.33.4" 2024-02-19T21:03:43.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:42.499906+0000 mon.a (mon.0) 718 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:03:43.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:42.513700+0000 mon.a (mon.0) 719 : audit [INF] from='client.34718 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:03:43.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:42.513754+0000 mon.a (mon.0) 720 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-02-19T21:03:44.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:42.940061+0000 mgr.y (mgr.14550) 102 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:44.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:42.940061+0000 mgr.y (mgr.14550) 102 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:44.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:42.940061+0000 mgr.y (mgr.14550) 102 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:45.560 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:03:45.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:45.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:03:45.668 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:03:45.698 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:45.796 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:03:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:45.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:03:45.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.901+0000 7fba80abc700 1 -- 172.21.15.163:0/1635681317 >> v1:172.21.15.163:6789/0 conn(0x7fba7c0f94d0 legacy=0x7fba7c1029a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:45.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.901+0000 7fba80abc700 1 -- 172.21.15.163:0/1635681317 shutdown_connections 2024-02-19T21:03:45.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.901+0000 7fba80abc700 1 -- 172.21.15.163:0/1635681317 >> 172.21.15.163:0/1635681317 conn(0x7fba7c0f2470 msgr2=0x7fba7c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:45.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.901+0000 7fba80abc700 1 -- 172.21.15.163:0/1635681317 shutdown_connections 2024-02-19T21:03:45.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.901+0000 7fba80abc700 1 -- 172.21.15.163:0/1635681317 wait complete. 2024-02-19T21:03:45.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.902+0000 7fba80abc700 1 librados: starting msgr at 2024-02-19T21:03:45.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.902+0000 7fba80abc700 1 librados: starting objecter 2024-02-19T21:03:45.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.902+0000 7fba80abc700 1 Processor -- start 2024-02-19T21:03:45.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.902+0000 7fba80abc700 1 -- start start 2024-02-19T21:03:45.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.902+0000 7fba80abc700 1 librados: setting wanted keys 2024-02-19T21:03:45.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.902+0000 7fba80abc700 1 librados: calling monclient init 2024-02-19T21:03:45.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.902+0000 7fba80abc700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba7c117400 con 0x7fba7c1042b0 2024-02-19T21:03:45.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.902+0000 7fba80abc700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba7c1185e0 con 0x7fba7c0f94d0 2024-02-19T21:03:45.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.902+0000 7fba80abc700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba7c1197e0 con 0x7fba7c0f8920 2024-02-19T21:03:45.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.902+0000 7fba7a59c700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fba7c0f8920 0x7fba7c0ff6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:52142/0 (socket says 172.21.15.163:52142) 2024-02-19T21:03:45.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.902+0000 7fba7a59c700 1 -- 172.21.15.163:0/2590083148 learned_addr learned my addr 172.21.15.163:0/2590083148 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:45.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 10 client.?.objecter ms_handle_connect 0x7fba7c0f8920 2024-02-19T21:03:45.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:45.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 10 client.?.objecter ms_handle_connect 0x7fba7c1042b0 2024-02-19T21:03:45.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:45.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 10 client.?.objecter ms_handle_connect 0x7fba7c0f94d0 2024-02-19T21:03:45.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:45.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957946054 0 0) 0x7fba7c1197e0 con 0x7fba7c0f8920 2024-02-19T21:03:45.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba640038e0 con 0x7fba7c0f8920 2024-02-19T21:03:45.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836652788 0 0) 0x7fba7c117400 con 0x7fba7c1042b0 2024-02-19T21:03:45.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba7c1197e0 con 0x7fba7c1042b0 2024-02-19T21:03:45.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967237319 0 0) 0x7fba7c1185e0 con 0x7fba7c0f94d0 2024-02-19T21:03:45.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba7c117400 con 0x7fba7c0f94d0 2024-02-19T21:03:45.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1861349615 0 0) 0x7fba640038e0 con 0x7fba7c0f8920 2024-02-19T21:03:45.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba7c1185e0 con 0x7fba7c0f8920 2024-02-19T21:03:45.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fba6c0019f0 con 0x7fba7c0f8920 2024-02-19T21:03:45.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 605184819 0 0) 0x7fba7c1197e0 con 0x7fba7c1042b0 2024-02-19T21:03:45.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba640038e0 con 0x7fba7c1042b0 2024-02-19T21:03:45.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3034427997 0 0) 0x7fba7c117400 con 0x7fba7c0f94d0 2024-02-19T21:03:45.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba7c1197e0 con 0x7fba7c0f94d0 2024-02-19T21:03:45.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fba74001b00 con 0x7fba7c1042b0 2024-02-19T21:03:45.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fba70001f50 con 0x7fba7c0f94d0 2024-02-19T21:03:45.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4091551188 0 0) 0x7fba7c1185e0 con 0x7fba7c0f8920 2024-02-19T21:03:45.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 >> v1:172.21.15.193:6789/0 conn(0x7fba7c0f94d0 legacy=0x7fba7c0ffdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:45.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 >> v1:172.21.15.163:6789/0 conn(0x7fba7c1042b0 legacy=0x7fba7c115ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:45.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7c11a9e0 con 0x7fba7c0f8920 2024-02-19T21:03:45.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7c1199d0 con 0x7fba7c0f8920 2024-02-19T21:03:45.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba80abc700 10 client.34724.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:45.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7c119e40 con 0x7fba7c0f8920 2024-02-19T21:03:45.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba80abc700 1 librados: init done 2024-02-19T21:03:45.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fba6c002360 con 0x7fba7c0f8920 2024-02-19T21:03:45.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.903+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fba6c005a30 con 0x7fba7c0f8920 2024-02-19T21:03:45.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.904+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fba5c004f00 con 0x7fba7c0f8920 2024-02-19T21:03:45.900 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:03:45.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.905+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fba6c0024d0 con 0x7fba7c0f8920 2024-02-19T21:03:45.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.905+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(88..88 src has 32..88) v4 ==== 6172+0+0 (unknown 1929784803 0 0) 0x7fba6c0784a0 con 0x7fba7c0f8920 2024-02-19T21:03:45.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.905+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 osd_map(88..88 src has 32..88) v4 2024-02-19T21:03:45.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.905+0000 7fba6b7fe700 3 client.34724.objecter handle_osd_map got epochs [88,88] > 0 2024-02-19T21:03:45.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.905+0000 7fba6b7fe700 3 client.34724.objecter handle_osd_map decoding full epoch 88 2024-02-19T21:03:45.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.906+0000 7fba6b7fe700 20 client.34724.objecter dump_active .. 0 homeless 2024-02-19T21:03:45.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.906+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v88) v1 ==== 54+0+16 (unknown 822001828 0 2420000192) 0x7fba6c07a890 con 0x7fba7c0f8920 2024-02-19T21:03:45.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.906+0000 7fba6b7fe700 10 client.34724.objecter ms_handle_connect 0x7fba64063dd0 2024-02-19T21:03:45.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.906+0000 7fba80abc700 10 client.34724.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:45.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.906+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 1 2024-02-19T21:03:45.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.906+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0051a0 osd=0 3 2024-02-19T21:03:45.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.906+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0051a0 osd=0 3 2024-02-19T21:03:45.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.906+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0051a0 osd=0 3 2024-02-19T21:03:45.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.906+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 1 2024-02-19T21:03:45.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.906+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c0051a0 osd=0 3 2024-02-19T21:03:45.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.906+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 0 1 2024-02-19T21:03:45.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.906+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0051a0 osd=0 4 2024-02-19T21:03:45.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.906+0000 7fba80abc700 10 client.34724.objecter _send_command 1 2024-02-19T21:03:45.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.906+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba5c004970 con 0x7fba5c004530 2024-02-19T21:03:45.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.906+0000 7fba6b7fe700 10 client.34724.objecter ms_handle_connect 0x7fba5c004530 2024-02-19T21:03:45.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.907+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba5c004970 con 0x7fba5c004530 2024-02-19T21:03:45.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.907+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:03:45.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.907+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 1 = system:0 2024-02-19T21:03:45.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.907+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c0051a0 osd=0 4 2024-02-19T21:03:45.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.907+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 0 1 2024-02-19T21:03:45.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.922+0000 7fba80abc700 10 client.34724.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:03:45.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.922+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 2 2024-02-19T21:03:45.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.922+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0051a0 osd=0 3 2024-02-19T21:03:45.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.922+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0051a0 osd=0 3 2024-02-19T21:03:45.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.922+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0051a0 osd=0 3 2024-02-19T21:03:45.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.922+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 2 2024-02-19T21:03:45.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.922+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c0051a0 osd=0 3 2024-02-19T21:03:45.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.922+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 0 2 2024-02-19T21:03:45.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.922+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0051a0 osd=0 4 2024-02-19T21:03:45.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.922+0000 7fba80abc700 10 client.34724.objecter _send_command 2 2024-02-19T21:03:45.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.922+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba5c004d50 con 0x7fba5c004530 2024-02-19T21:03:45.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.924+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba5c004d50 con 0x7fba5c004530 2024-02-19T21:03:45.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.924+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:03:45.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.924+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:03:45.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.924+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c0051a0 osd=0 4 2024-02-19T21:03:45.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.924+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 0 2 2024-02-19T21:03:45.917 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:03:45.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.924+0000 7fba80abc700 10 client.34724.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:45.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.924+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 3 2024-02-19T21:03:45.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.925+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c004260 osd=1 3 2024-02-19T21:03:45.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.925+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c004260 osd=1 3 2024-02-19T21:03:45.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.925+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c004260 osd=1 3 2024-02-19T21:03:45.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.925+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 3 2024-02-19T21:03:45.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.925+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c004260 osd=1 3 2024-02-19T21:03:45.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.925+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 1 3 2024-02-19T21:03:45.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.925+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c004260 osd=1 4 2024-02-19T21:03:45.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.925+0000 7fba80abc700 10 client.34724.objecter _send_command 3 2024-02-19T21:03:45.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.925+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba5c00bef0 con 0x7fba5c002090 2024-02-19T21:03:45.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.925+0000 7fba6b7fe700 10 client.34724.objecter ms_handle_connect 0x7fba5c002090 2024-02-19T21:03:45.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.926+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba5c00bef0 con 0x7fba5c002090 2024-02-19T21:03:45.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.926+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:03:45.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.926+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 3 = system:0 2024-02-19T21:03:45.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.926+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c004260 osd=1 4 2024-02-19T21:03:45.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.926+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 1 3 2024-02-19T21:03:45.931 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:45.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.943+0000 7fba80abc700 10 client.34724.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:03:45.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.943+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 4 2024-02-19T21:03:45.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.943+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c004260 osd=1 3 2024-02-19T21:03:45.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.943+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c004260 osd=1 3 2024-02-19T21:03:45.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.943+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c004260 osd=1 3 2024-02-19T21:03:45.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.943+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 4 2024-02-19T21:03:45.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.943+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c004260 osd=1 3 2024-02-19T21:03:45.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.943+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 1 4 2024-02-19T21:03:45.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.943+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c004260 osd=1 4 2024-02-19T21:03:45.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.943+0000 7fba80abc700 10 client.34724.objecter _send_command 4 2024-02-19T21:03:45.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.943+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba5c0016b0 con 0x7fba5c002090 2024-02-19T21:03:45.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.944+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba5c0016b0 con 0x7fba5c002090 2024-02-19T21:03:45.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.944+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:03:45.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.944+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:03:45.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.944+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c004260 osd=1 4 2024-02-19T21:03:45.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.944+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 1 4 2024-02-19T21:03:45.937 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:03:45.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.945+0000 7fba80abc700 10 client.34724.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:45.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.945+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 5 2024-02-19T21:03:45.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.945+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0035f0 osd=2 3 2024-02-19T21:03:45.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.945+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0035f0 osd=2 3 2024-02-19T21:03:45.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.945+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0035f0 osd=2 3 2024-02-19T21:03:45.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.945+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 5 2024-02-19T21:03:45.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.945+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c0035f0 osd=2 3 2024-02-19T21:03:45.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.945+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 2 5 2024-02-19T21:03:45.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.945+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0035f0 osd=2 4 2024-02-19T21:03:45.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.945+0000 7fba80abc700 10 client.34724.objecter _send_command 5 2024-02-19T21:03:45.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.945+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba5c00dd60 con 0x7fba5c00d1e0 2024-02-19T21:03:45.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.945+0000 7fba6b7fe700 10 client.34724.objecter ms_handle_connect 0x7fba5c00d1e0 2024-02-19T21:03:45.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.946+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba5c00dd60 con 0x7fba5c00d1e0 2024-02-19T21:03:45.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.946+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:03:45.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.946+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 5 = system:0 2024-02-19T21:03:45.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.946+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c0035f0 osd=2 4 2024-02-19T21:03:45.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.946+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 2 5 2024-02-19T21:03:45.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.962+0000 7fba80abc700 10 client.34724.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:03:45.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.962+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 6 2024-02-19T21:03:45.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.962+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0035f0 osd=2 3 2024-02-19T21:03:45.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.962+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0035f0 osd=2 3 2024-02-19T21:03:45.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.962+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0035f0 osd=2 3 2024-02-19T21:03:45.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.962+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 6 2024-02-19T21:03:45.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.962+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c0035f0 osd=2 3 2024-02-19T21:03:45.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.962+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 2 6 2024-02-19T21:03:45.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.962+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0035f0 osd=2 4 2024-02-19T21:03:45.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.962+0000 7fba80abc700 10 client.34724.objecter _send_command 6 2024-02-19T21:03:45.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.962+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba5c00e6a0 con 0x7fba5c00d1e0 2024-02-19T21:03:45.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.964+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba5c00e6a0 con 0x7fba5c00d1e0 2024-02-19T21:03:45.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.964+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:03:45.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.964+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:03:45.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.964+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c0035f0 osd=2 4 2024-02-19T21:03:45.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.964+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 2 6 2024-02-19T21:03:45.957 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:03:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.965+0000 7fba80abc700 10 client.34724.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.965+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 7 2024-02-19T21:03:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.965+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0033d0 osd=3 3 2024-02-19T21:03:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.965+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0033d0 osd=3 3 2024-02-19T21:03:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.965+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0033d0 osd=3 3 2024-02-19T21:03:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.965+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 7 2024-02-19T21:03:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.965+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c0033d0 osd=3 3 2024-02-19T21:03:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.965+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 3 7 2024-02-19T21:03:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.965+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0033d0 osd=3 4 2024-02-19T21:03:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.965+0000 7fba80abc700 10 client.34724.objecter _send_command 7 2024-02-19T21:03:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.965+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba5c0159e0 con 0x7fba5c011dc0 2024-02-19T21:03:45.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.965+0000 7fba6b7fe700 10 client.34724.objecter ms_handle_connect 0x7fba5c011dc0 2024-02-19T21:03:45.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.966+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba5c0159e0 con 0x7fba5c011dc0 2024-02-19T21:03:45.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.966+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:03:45.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.966+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 7 = system:0 2024-02-19T21:03:45.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.966+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c0033d0 osd=3 4 2024-02-19T21:03:45.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.966+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 3 7 2024-02-19T21:03:45.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.980+0000 7fba80abc700 10 client.34724.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:03:45.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.980+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 8 2024-02-19T21:03:45.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.980+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0033d0 osd=3 3 2024-02-19T21:03:45.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.980+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0033d0 osd=3 3 2024-02-19T21:03:45.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.980+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0033d0 osd=3 3 2024-02-19T21:03:45.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.980+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 8 2024-02-19T21:03:45.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.980+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c0033d0 osd=3 3 2024-02-19T21:03:45.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.980+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 3 8 2024-02-19T21:03:45.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.980+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0033d0 osd=3 4 2024-02-19T21:03:45.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.980+0000 7fba80abc700 10 client.34724.objecter _send_command 8 2024-02-19T21:03:45.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.980+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba5c00e170 con 0x7fba5c011dc0 2024-02-19T21:03:45.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba5c00e170 con 0x7fba5c011dc0 2024-02-19T21:03:45.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:03:45.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:03:45.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c0033d0 osd=3 4 2024-02-19T21:03:45.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 3 8 2024-02-19T21:03:45.974 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:03:45.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba80abc700 10 client.34724.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:45.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 9 2024-02-19T21:03:45.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c00df30 osd=4 3 2024-02-19T21:03:45.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c00df30 osd=4 3 2024-02-19T21:03:45.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c00df30 osd=4 3 2024-02-19T21:03:45.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 9 2024-02-19T21:03:45.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c00df30 osd=4 3 2024-02-19T21:03:45.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 4 9 2024-02-19T21:03:45.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c00df30 osd=4 4 2024-02-19T21:03:45.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba80abc700 10 client.34724.objecter _send_command 9 2024-02-19T21:03:45.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba5c00e110 con 0x7fba5c00cd40 2024-02-19T21:03:45.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.982+0000 7fba6b7fe700 10 client.34724.objecter ms_handle_connect 0x7fba5c00cd40 2024-02-19T21:03:45.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.983+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba5c00e110 con 0x7fba5c00cd40 2024-02-19T21:03:45.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.983+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:03:45.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.983+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 9 = system:0 2024-02-19T21:03:45.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.983+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c00df30 osd=4 4 2024-02-19T21:03:45.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.983+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 4 9 2024-02-19T21:03:45.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.999+0000 7fba80abc700 10 client.34724.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:03:45.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.999+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 10 2024-02-19T21:03:45.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.999+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c00df30 osd=4 3 2024-02-19T21:03:45.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.999+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c00df30 osd=4 3 2024-02-19T21:03:45.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.999+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c00df30 osd=4 3 2024-02-19T21:03:45.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.999+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 10 2024-02-19T21:03:45.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.999+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c00df30 osd=4 3 2024-02-19T21:03:45.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.999+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 4 10 2024-02-19T21:03:45.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.999+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c00df30 osd=4 4 2024-02-19T21:03:45.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.999+0000 7fba80abc700 10 client.34724.objecter _send_command 10 2024-02-19T21:03:45.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:45.999+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba5c00e110 con 0x7fba5c00cd40 2024-02-19T21:03:45.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba5c00e110 con 0x7fba5c00cd40 2024-02-19T21:03:45.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:03:45.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:03:45.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c00df30 osd=4 4 2024-02-19T21:03:45.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 4 10 2024-02-19T21:03:45.993 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:03:45.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba80abc700 10 client.34724.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:45.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 11 2024-02-19T21:03:45.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c00e110 osd=5 3 2024-02-19T21:03:45.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c00e110 osd=5 3 2024-02-19T21:03:45.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c00e110 osd=5 3 2024-02-19T21:03:45.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 11 2024-02-19T21:03:45.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c00e110 osd=5 3 2024-02-19T21:03:45.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 5 11 2024-02-19T21:03:45.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c00e110 osd=5 4 2024-02-19T21:03:45.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba80abc700 10 client.34724.objecter _send_command 11 2024-02-19T21:03:45.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba5c017f60 con 0x7fba5c017970 2024-02-19T21:03:45.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.001+0000 7fba6b7fe700 10 client.34724.objecter ms_handle_connect 0x7fba5c017970 2024-02-19T21:03:45.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.002+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba5c017f60 con 0x7fba5c017970 2024-02-19T21:03:45.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.002+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:03:45.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.002+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 11 = system:0 2024-02-19T21:03:45.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.002+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c00e110 osd=5 4 2024-02-19T21:03:45.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.002+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 5 11 2024-02-19T21:03:46.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.017+0000 7fba80abc700 10 client.34724.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:03:46.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.017+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 12 2024-02-19T21:03:46.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.017+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c00e110 osd=5 3 2024-02-19T21:03:46.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.017+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c00e110 osd=5 3 2024-02-19T21:03:46.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.017+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c00e110 osd=5 3 2024-02-19T21:03:46.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.017+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 12 2024-02-19T21:03:46.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.017+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c00e110 osd=5 3 2024-02-19T21:03:46.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.017+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 5 12 2024-02-19T21:03:46.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.017+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c00e110 osd=5 4 2024-02-19T21:03:46.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.017+0000 7fba80abc700 10 client.34724.objecter _send_command 12 2024-02-19T21:03:46.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.017+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba5c0181a0 con 0x7fba5c017970 2024-02-19T21:03:46.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba5c0181a0 con 0x7fba5c017970 2024-02-19T21:03:46.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:03:46.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:03:46.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c00e110 osd=5 4 2024-02-19T21:03:46.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 5 12 2024-02-19T21:03:46.011 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:03:46.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba80abc700 10 client.34724.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:46.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 13 2024-02-19T21:03:46.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0181a0 osd=6 3 2024-02-19T21:03:46.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0181a0 osd=6 3 2024-02-19T21:03:46.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0181a0 osd=6 3 2024-02-19T21:03:46.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 13 2024-02-19T21:03:46.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c0181a0 osd=6 3 2024-02-19T21:03:46.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 6 13 2024-02-19T21:03:46.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0181a0 osd=6 4 2024-02-19T21:03:46.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba80abc700 10 client.34724.objecter _send_command 13 2024-02-19T21:03:46.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.019+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba5c022b40 con 0x7fba5c01efc0 2024-02-19T21:03:46.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.020+0000 7fba6b7fe700 10 client.34724.objecter ms_handle_connect 0x7fba5c01efc0 2024-02-19T21:03:46.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.020+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba5c022b40 con 0x7fba5c01efc0 2024-02-19T21:03:46.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.020+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:03:46.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.020+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 13 = system:0 2024-02-19T21:03:46.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.020+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c0181a0 osd=6 4 2024-02-19T21:03:46.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.020+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 6 13 2024-02-19T21:03:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.035+0000 7fba80abc700 10 client.34724.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:03:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.035+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 14 2024-02-19T21:03:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.035+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0181a0 osd=6 3 2024-02-19T21:03:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.035+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0181a0 osd=6 3 2024-02-19T21:03:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.035+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c0181a0 osd=6 3 2024-02-19T21:03:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.035+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 14 2024-02-19T21:03:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.035+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c0181a0 osd=6 3 2024-02-19T21:03:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.035+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 6 14 2024-02-19T21:03:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.035+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0181a0 osd=6 4 2024-02-19T21:03:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.035+0000 7fba80abc700 10 client.34724.objecter _send_command 14 2024-02-19T21:03:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.035+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba5c022b40 con 0x7fba5c01efc0 2024-02-19T21:03:46.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba5c022b40 con 0x7fba5c01efc0 2024-02-19T21:03:46.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:03:46.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:03:46.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c0181a0 osd=6 4 2024-02-19T21:03:46.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 6 14 2024-02-19T21:03:46.029 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:03:46.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba80abc700 10 client.34724.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:46.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 15 2024-02-19T21:03:46.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c022b40 osd=7 3 2024-02-19T21:03:46.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c022b40 osd=7 3 2024-02-19T21:03:46.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c022b40 osd=7 3 2024-02-19T21:03:46.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 15 2024-02-19T21:03:46.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c022b40 osd=7 3 2024-02-19T21:03:46.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 7 15 2024-02-19T21:03:46.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c022b40 osd=7 4 2024-02-19T21:03:46.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba80abc700 10 client.34724.objecter _send_command 15 2024-02-19T21:03:46.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba5c023d90 con 0x7fba5c024840 2024-02-19T21:03:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.037+0000 7fba6b7fe700 10 client.34724.objecter ms_handle_connect 0x7fba5c024840 2024-02-19T21:03:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.038+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba5c023d90 con 0x7fba5c024840 2024-02-19T21:03:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.038+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:03:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.038+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 15 = system:0 2024-02-19T21:03:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.038+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c022b40 osd=7 4 2024-02-19T21:03:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.038+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 7 15 2024-02-19T21:03:46.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.053+0000 7fba80abc700 10 client.34724.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:03:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.053+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign -1 16 2024-02-19T21:03:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.053+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c022b40 osd=7 3 2024-02-19T21:03:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.053+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c022b40 osd=7 3 2024-02-19T21:03:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.053+0000 7fba80abc700 20 client.34724.objecter _get_session s=0x7fba5c022b40 osd=7 3 2024-02-19T21:03:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.053+0000 7fba80abc700 15 client.34724.objecter _session_command_op_remove -1 16 2024-02-19T21:03:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.053+0000 7fba80abc700 20 client.34724.objecter get_session s=0x7fba5c022b40 osd=7 3 2024-02-19T21:03:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.053+0000 7fba80abc700 15 client.34724.objecter _session_command_op_assign 7 16 2024-02-19T21:03:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.053+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c022b40 osd=7 4 2024-02-19T21:03:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.053+0000 7fba80abc700 10 client.34724.objecter _send_command 16 2024-02-19T21:03:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.053+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba5c023d90 con 0x7fba5c024840 2024-02-19T21:03:46.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.055+0000 7fba6b7fe700 1 -- 172.21.15.163:0/2590083148 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba5c023d90 con 0x7fba5c024840 2024-02-19T21:03:46.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.055+0000 7fba6b7fe700 10 client.34724.objecter ms_dispatch 0x7fba7c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:03:46.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.055+0000 7fba6b7fe700 10 client.34724.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:03:46.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.055+0000 7fba6b7fe700 20 client.34724.objecter put_session s=0x7fba5c022b40 osd=7 4 2024-02-19T21:03:46.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.055+0000 7fba6b7fe700 15 client.34724.objecter _session_command_op_remove 7 16 2024-02-19T21:03:46.048 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:03:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.055+0000 7fba80abc700 10 librados: watch_flush enter 2024-02-19T21:03:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.055+0000 7fba80abc700 10 librados: watch_flush exit 2024-02-19T21:03:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.055+0000 7fba80abc700 10 client.34724.objecter close_session for osd.0 2024-02-19T21:03:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.055+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 >> v1:172.21.15.163:6801/1559418115 conn(0x7fba5c004530 legacy=0x7fba5c002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.055+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0051a0 osd=0 1 2024-02-19T21:03:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 10 client.34724.objecter close_session for osd.1 2024-02-19T21:03:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 >> v1:172.21.15.163:6805/3125047432 conn(0x7fba5c002090 legacy=0x7fba5c00a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c004260 osd=1 1 2024-02-19T21:03:46.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 10 client.34724.objecter close_session for osd.2 2024-02-19T21:03:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 >> v1:172.21.15.163:6809/1563158487 conn(0x7fba5c00d1e0 legacy=0x7fba5c00d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0035f0 osd=2 1 2024-02-19T21:03:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 10 client.34724.objecter close_session for osd.3 2024-02-19T21:03:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 >> v1:172.21.15.163:6813/2112279769 conn(0x7fba5c011dc0 legacy=0x7fba5c0141b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0033d0 osd=3 1 2024-02-19T21:03:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 10 client.34724.objecter close_session for osd.4 2024-02-19T21:03:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 >> v1:172.21.15.193:6800/3129716965 conn(0x7fba5c00cd40 legacy=0x7fba5c015b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c00df30 osd=4 1 2024-02-19T21:03:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 10 client.34724.objecter close_session for osd.5 2024-02-19T21:03:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 >> v1:172.21.15.193:6804/805161991 conn(0x7fba5c017970 legacy=0x7fba5c01d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c00e110 osd=5 1 2024-02-19T21:03:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 10 client.34724.objecter close_session for osd.6 2024-02-19T21:03:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 >> v1:172.21.15.193:6808/787604615 conn(0x7fba5c01efc0 legacy=0x7fba5c0213b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c0181a0 osd=6 1 2024-02-19T21:03:46.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 10 client.34724.objecter close_session for osd.7 2024-02-19T21:03:46.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba7ad9d700 1 -- 172.21.15.163:0/2590083148 reap_dead start 2024-02-19T21:03:46.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 >> v1:172.21.15.193:6812/157927432 conn(0x7fba5c024840 legacy=0x7fba5c024ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:46.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba7ad9d700 1 -- 172.21.15.163:0/2590083148 reap_dead start 2024-02-19T21:03:46.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 20 client.34724.objecter put_session s=0x7fba5c022b40 osd=7 1 2024-02-19T21:03:46.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 20 client.34724.objecter shutdown clearing up homeless session... 2024-02-19T21:03:46.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 10 client.34724.objecter successfully canceled tick 2024-02-19T21:03:46.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba7ad9d700 1 -- 172.21.15.163:0/2590083148 reap_dead start 2024-02-19T21:03:46.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba7ad9d700 1 -- 172.21.15.163:0/2590083148 reap_dead start 2024-02-19T21:03:46.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 >> v1:172.21.15.163:6800/1552514681 conn(0x7fba64063dd0 legacy=0x7fba64066290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:46.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 >> v1:172.21.15.163:6790/0 conn(0x7fba7c0f8920 legacy=0x7fba7c0ff6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:46.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 shutdown_connections 2024-02-19T21:03:46.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 >> 172.21.15.163:0/2590083148 conn(0x7fba7c0f2470 msgr2=0x7fba7c107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:46.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 shutdown_connections 2024-02-19T21:03:46.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 1 -- 172.21.15.163:0/2590083148 wait complete. 2024-02-19T21:03:46.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:46.056+0000 7fba80abc700 1 librados: shutdown 2024-02-19T21:03:46.052 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:03:46.052 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:03:46.052 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:03:46.052 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:03:46.052 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:03:46.052 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:03:46.052 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:03:46.052 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:03:46.053 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:03:46.060 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:46.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:03:46.161 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:03:46.191 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:46.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:44.940633+0000 mgr.y (mgr.14550) 103 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:46.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:45.906631+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? v1:172.21.15.163:0/2590083148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:03:46.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:44.940633+0000 mgr.y (mgr.14550) 103 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:46.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:45.906631+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? v1:172.21.15.163:0/2590083148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:03:46.784 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:03:46.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:44.940633+0000 mgr.y (mgr.14550) 103 : cluster [DBG] pgmap v82: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:46.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:45.906631+0000 mon.c (mon.2) 28 : audit [DBG] from='client.? v1:172.21.15.163:0/2590083148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:03:47.182 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:03:47.182 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:03:47.192 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:03:47.192 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:03:47.193 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-19T21:03:47.193 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-19T21:03:47.237 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:47.445 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:03:47.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:47.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:03:47.549 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:03:47.582 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:47.603 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:03:47.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:47.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:03:47.707 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:03:47.737 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:47.757 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:03:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:47.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:03:47.861 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:03:47.890 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:48.063 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:03:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:03:48.274 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-19T21:03:48.441 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:03:48.443 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:48.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:03:48.523 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:46.940984+0000 mgr.y (mgr.14550) 104 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:48.545 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:03:48.572 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:48.767 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:48.770 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:03:48.773 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:46.940984+0000 mgr.y (mgr.14550) 104 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:48.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:48.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:03:48.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:46.940984+0000 mgr.y (mgr.14550) 104 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:48.875 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:03:48.893 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:03:48.902 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:49.031 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:03:49.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:49.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:03:49.136 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:03:49.168 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:49.189 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:03:49.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:49.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:03:49.293 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:03:49.319 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:49.668 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:03:49.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:49.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:03:49.771 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:03:49.799 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:49.820 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:03:49.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:49.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:03:49.933 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:03:49.959 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:50.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:03:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:03:50.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:48.941362+0000 mgr.y (mgr.14550) 105 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:50.922 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:48.941362+0000 mgr.y (mgr.14550) 105 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:50.922 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:48.941362+0000 mgr.y (mgr.14550) 105 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:50.938 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:03:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:50.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:03:51.042 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:03:51.073 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:51.094 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:03:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:51.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:03:51.198 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:03:51.227 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:51.247 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:03:51.250 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:51.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:03:51.351 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:03:51.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.171+0000 7fba323a1700 1 -- 172.21.15.163:0/652799489 >> v1:172.21.15.163:6790/0 conn(0x7fba2c0f9340 legacy=0x7fba2c102810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:51.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.171+0000 7fba323a1700 1 -- 172.21.15.163:0/652799489 shutdown_connections 2024-02-19T21:03:51.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.171+0000 7fba323a1700 1 -- 172.21.15.163:0/652799489 >> 172.21.15.163:0/652799489 conn(0x7fba2c0f22c0 msgr2=0x7fba2c0f46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:51.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.171+0000 7fba323a1700 1 -- 172.21.15.163:0/652799489 shutdown_connections 2024-02-19T21:03:51.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.171+0000 7fba323a1700 1 -- 172.21.15.163:0/652799489 wait complete. 2024-02-19T21:03:51.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba323a1700 1 librados: starting msgr at 2024-02-19T21:03:51.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba323a1700 1 librados: starting objecter 2024-02-19T21:03:51.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba323a1700 1 Processor -- start 2024-02-19T21:03:51.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba323a1700 1 -- start start 2024-02-19T21:03:51.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba323a1700 1 librados: setting wanted keys 2024-02-19T21:03:51.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba323a1700 1 librados: calling monclient init 2024-02-19T21:03:51.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba323a1700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba2c102b90 con 0x7fba2c0f9340 2024-02-19T21:03:51.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba323a1700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba2c102d60 con 0x7fba2c0f8790 2024-02-19T21:03:51.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba323a1700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba2c11ba20 con 0x7fba2c104120 2024-02-19T21:03:51.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba30b9e700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fba2c0f9340 0x7fba2c114b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44522/0 (socket says 172.21.15.163:44522) 2024-02-19T21:03:51.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba31ba0700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fba2c104120 0x7fba2c1182a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:52174/0 (socket says 172.21.15.163:52174) 2024-02-19T21:03:51.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba30b9e700 1 -- 172.21.15.163:0/3657996602 learned_addr learned my addr 172.21.15.163:0/3657996602 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:51.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba227fc700 10 client.?.objecter ms_handle_connect 0x7fba2c0f9340 2024-02-19T21:03:51.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba227fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:51.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba227fc700 10 client.?.objecter ms_handle_connect 0x7fba2c104120 2024-02-19T21:03:51.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba227fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:51.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba227fc700 10 client.?.objecter ms_handle_connect 0x7fba2c0f8790 2024-02-19T21:03:51.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba227fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:51.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.172+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965472728 0 0) 0x7fba2c102b90 con 0x7fba2c0f9340 2024-02-19T21:03:51.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba18003800 con 0x7fba2c0f9340 2024-02-19T21:03:51.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 840704866 0 0) 0x7fba2c11ba20 con 0x7fba2c104120 2024-02-19T21:03:51.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba2c102b90 con 0x7fba2c104120 2024-02-19T21:03:51.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116750224 0 0) 0x7fba2c102d60 con 0x7fba2c0f8790 2024-02-19T21:03:51.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba2c11ba20 con 0x7fba2c0f8790 2024-02-19T21:03:51.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2744202028 0 0) 0x7fba18003800 con 0x7fba2c0f9340 2024-02-19T21:03:51.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba2c102d60 con 0x7fba2c0f9340 2024-02-19T21:03:51.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fba1c001d80 con 0x7fba2c0f9340 2024-02-19T21:03:51.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3906691814 0 0) 0x7fba2c102b90 con 0x7fba2c104120 2024-02-19T21:03:51.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba18003800 con 0x7fba2c104120 2024-02-19T21:03:51.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 180855791 0 0) 0x7fba2c11ba20 con 0x7fba2c0f8790 2024-02-19T21:03:51.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba2c102b90 con 0x7fba2c0f8790 2024-02-19T21:03:51.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fba24001e00 con 0x7fba2c104120 2024-02-19T21:03:51.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fba28001800 con 0x7fba2c0f8790 2024-02-19T21:03:51.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3342667157 0 0) 0x7fba2c102d60 con 0x7fba2c0f9340 2024-02-19T21:03:51.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 >> v1:172.21.15.163:6790/0 conn(0x7fba2c104120 legacy=0x7fba2c1182a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:51.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 >> v1:172.21.15.193:6789/0 conn(0x7fba2c0f8790 legacy=0x7fba2c1021e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:51.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c11cc40 con 0x7fba2c0f9340 2024-02-19T21:03:51.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba323a1700 1 -- 172.21.15.163:0/3657996602 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2c11bc10 con 0x7fba2c0f9340 2024-02-19T21:03:51.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba323a1700 10 client.14808.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:51.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba323a1700 1 -- 172.21.15.163:0/3657996602 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2c11c080 con 0x7fba2c0f9340 2024-02-19T21:03:51.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba323a1700 1 librados: init done 2024-02-19T21:03:51.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fba1c001a10 con 0x7fba2c0f9340 2024-02-19T21:03:51.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.173+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fba1c003860 con 0x7fba2c0f9340 2024-02-19T21:03:51.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.174+0000 7fba323a1700 1 -- 172.21.15.163:0/3657996602 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba10004f20 con 0x7fba2c0f9340 2024-02-19T21:03:51.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.174+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fba1c003000 con 0x7fba2c0f9340 2024-02-19T21:03:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:03:51.382 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:03:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: "epoch": 88, 2024-02-19T21:03:51.382 INFO:teuthology.orchestra.run.smithi163.stdout: "fsid": "2591809a-cf69-11ee-95bb-87774f69a715", 2024-02-19T21:03:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "created": "2024-02-19T20:55:37.794885+0000", 2024-02-19T21:03:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "modified": "2024-02-19T21:03:42.499726+0000", 2024-02-19T21:03:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "last_up_change": "2024-02-19T21:01:09.214134+0000", 2024-02-19T21:03:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "last_in_change": "2024-02-19T21:00:51.699017+0000", 2024-02-19T21:03:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-19T21:03:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_num": 5801984, 2024-02-19T21:03:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_set": [ 2024-02-19T21:03:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "noscrub", 2024-02-19T21:03:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "pglog_hardlimit", 2024-02-19T21:03:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "purged_snapdirs", 2024-02-19T21:03:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_deletes", 2024-02-19T21:03:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "sortbitwise" 2024-02-19T21:03:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:03:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_version": 18, 2024-02-19T21:03:51.383 INFO:teuthology.orchestra.run.smithi163.stdout: "full_ratio": 0.94999998807907104, 2024-02-19T21:03:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-19T21:03:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-19T21:03:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_snapshot": "", 2024-02-19T21:03:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_max": 6, 2024-02-19T21:03:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "max_osd": 8, 2024-02-19T21:03:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "require_min_compat_client": "luminous", 2024-02-19T21:03:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "min_compat_client": "luminous", 2024-02-19T21:03:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "require_osd_release": "pacific", 2024-02-19T21:03:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "pools": [ 2024-02-19T21:03:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 1, 2024-02-19T21:03:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "device_health_metrics", 2024-02-19T21:03:51.384 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T20:59:09.425169+0000", 2024-02-19T21:03:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:03:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:03:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:03:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:03:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:03:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:03:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:03:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:03:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:03:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:03:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:03:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:03:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 1, 2024-02-19T21:03:51.385 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 1, 2024-02-19T21:03:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 1, 2024-02-19T21:03:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 1, 2024-02-19T21:03:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 1, 2024-02-19T21:03:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:03:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:03:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:03:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:03:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:03:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:03:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:03:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "74", 2024-02-19T21:03:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:03:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:03:51.386 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:03:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:03:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:03:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:03:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:03:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:03:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:03:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:03:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:03:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:03:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:03:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:03:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:03:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:03:51.387 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:03:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:03:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:03:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:03:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:03:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:03:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:03:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:03:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:03:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:03:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.388 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:03:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:03:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:03:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:03:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:03:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:03:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:03:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:03:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:03:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:03:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:03:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:03:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_max": 32, 2024-02-19T21:03:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_min": 1 2024-02-19T21:03:51.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_devicehealth": {} 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 2, 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".rgw.root", 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:14.658945+0000", 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:03:51.390 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:03:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:03:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:03:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:03:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:03:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:03:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:03:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:03:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:03:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:03:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:03:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:03:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:03:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:03:51.391 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:03:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:03:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:03:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "59", 2024-02-19T21:03:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:03:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:03:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:03:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:03:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:03:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:03:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:03:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:03:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:03:51.392 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:03:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:03:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:03:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:03:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:03:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:03:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:03:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:03:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:03:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:03:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:03:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:03:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:03:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:03:51.393 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:03:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:03:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:03:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:03:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:03:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:03:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:03:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:03:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:03:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:03:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:03:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:03:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:03:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:03:51.394 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:03:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:03:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:03:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 3, 2024-02-19T21:03:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "iscsi", 2024-02-19T21:03:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:14.958800+0000", 2024-02-19T21:03:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:03:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:03:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:03:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:03:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:03:51.395 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:03:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:03:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:03:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:03:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:03:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:03:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:03:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 3, 2024-02-19T21:03:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 3, 2024-02-19T21:03:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 3, 2024-02-19T21:03:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 3, 2024-02-19T21:03:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 3, 2024-02-19T21:03:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:03:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:03:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:03:51.396 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "60", 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:03:51.397 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:03:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:03:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:03:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:03:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:03:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:03:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:03:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:03:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:03:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:03:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:03:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:03:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:03:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:03:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:03:51.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:03:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:03:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:03:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:03:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:03:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:03:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:03:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:03:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:03:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:03:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:03:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:03:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:03:51.399 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd": {} 2024-02-19T21:03:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 4, 2024-02-19T21:03:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.log", 2024-02-19T21:03:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:16.543816+0000", 2024-02-19T21:03:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:03:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:03:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:03:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:03:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:03:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:03:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:03:51.400 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:03:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:03:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:03:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:03:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:03:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:03:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:03:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:03:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:03:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:03:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:03:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:03:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:03:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:03:51.401 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:03:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:03:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:03:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "61", 2024-02-19T21:03:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:03:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:03:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:03:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:03:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:03:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:03:51.402 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:03:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:03:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:03:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:03:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:03:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:03:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:03:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:03:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:03:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:03:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:03:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:03:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:03:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:03:51.403 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:03:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:03:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:03:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:03:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:03:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:03:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:03:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:03:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:03:51.404 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:03:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:03:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:03:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:03:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:03:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:03:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:03:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:03:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:03:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:03:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:03:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.405 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 5, 2024-02-19T21:03:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.control", 2024-02-19T21:03:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:18.521482+0000", 2024-02-19T21:03:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:03:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:03:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:03:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:03:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:03:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:03:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:03:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:03:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:03:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:03:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:03:51.406 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:03:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:03:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:03:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:03:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:03:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:03:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:03:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:03:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:03:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:03:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:03:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:03:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:03:51.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "63", 2024-02-19T21:03:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:03:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:03:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:03:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:03:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:03:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:03:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:03:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:03:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:03:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:03:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:03:51.408 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:03:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:03:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:03:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:03:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:03:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:03:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:03:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:03:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:03:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:03:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:03:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:03:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:03:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:03:51.409 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:03:51.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.410 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:03:51.410 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:03:51.410 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:03:51.410 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:03:51.410 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:03:51.410 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:03:51.410 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:03:51.410 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:03:51.410 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:03:51.410 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:03:51.410 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:03:51.410 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:03:51.410 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:03:51.411 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:03:51.411 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.411 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 6, 2024-02-19T21:03:51.411 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.meta", 2024-02-19T21:03:51.411 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:20.586564+0000", 2024-02-19T21:03:51.411 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:03:51.411 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:03:51.411 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:03:51.411 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:03:51.411 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:03:51.411 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:03:51.412 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:03:51.412 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:03:51.412 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:03:51.412 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:03:51.412 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:03:51.412 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:03:51.412 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:03:51.412 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:03:51.412 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:03:51.412 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:03:51.412 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:03:51.412 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:03:51.412 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:03:51.412 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:03:51.412 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:03:51.413 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:03:51.413 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:03:51.413 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:03:51.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.413 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "65", 2024-02-19T21:03:51.413 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:03:51.413 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:03:51.413 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:03:51.413 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:03:51.413 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:03:51.413 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:03:51.414 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:03:51.414 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:03:51.414 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:03:51.414 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:03:51.414 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:03:51.414 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:03:51.414 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:03:51.414 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:03:51.414 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:03:51.414 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:03:51.414 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:03:51.414 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:03:51.414 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:03:51.414 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:03:51.414 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:03:51.415 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:03:51.415 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:03:51.415 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:03:51.415 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:03:51.415 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:03:51.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.415 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:03:51.415 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:03:51.415 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:03:51.415 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:03:51.415 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:03:51.415 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:03:51.415 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:03:51.415 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:03:51.416 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:03:51.416 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:03:51.416 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:03:51.416 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:03:51.416 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_bias": 4 2024-02-19T21:03:51.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.175+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(88..88 src has 32..88) v4 ==== 6172+0+0 (unknown 1929784803 0 0) 0x7fba1c0788d0 con 0x7fba2c0f9340 2024-02-19T21:03:51.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.175+0000 7fba227fc700 10 client.14808.objecter ms_dispatch 0x7fba2c001550 osd_map(88..88 src has 32..88) v4 2024-02-19T21:03:51.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.175+0000 7fba227fc700 3 client.14808.objecter handle_osd_map got epochs [88,88] > 0 2024-02-19T21:03:51.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.175+0000 7fba227fc700 3 client.14808.objecter handle_osd_map decoding full epoch 88 2024-02-19T21:03:51.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.175+0000 7fba227fc700 20 client.14808.objecter dump_active .. 0 homeless 2024-02-19T21:03:51.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.175+0000 7fba227fc700 10 client.14808.objecter ms_handle_connect 0x7fba18063e10 2024-02-19T21:03:51.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.179+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fba1c04c540 con 0x7fba2c0f9340 2024-02-19T21:03:51.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.364+0000 7fba323a1700 1 -- 172.21.15.163:0/3657996602 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fba100051d0 con 0x7fba2c0f9340 2024-02-19T21:03:51.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.365+0000 7fba227fc700 1 -- 172.21.15.163:0/3657996602 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v88) v1 ==== 81+0+35969 (unknown 1608717924 0 4231827910) 0x7fba1c0039c0 con 0x7fba2c0f9340 2024-02-19T21:03:51.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.367+0000 7fba323a1700 10 librados: watch_flush enter 2024-02-19T21:03:51.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.367+0000 7fba323a1700 10 librados: watch_flush exit 2024-02-19T21:03:51.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.367+0000 7fba323a1700 20 client.14808.objecter shutdown clearing up homeless session... 2024-02-19T21:03:51.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.367+0000 7fba323a1700 10 client.14808.objecter successfully canceled tick 2024-02-19T21:03:51.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.417 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:03:51.418 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:03:51.418 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.418 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.418 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:03:51.418 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-02-19T21:03:51.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.418 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-02-19T21:03:51.418 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088", 2024-02-19T21:03:51.418 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:03:51.418 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:03:51.418 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:03:51.418 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:03:51.418 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:03:51.418 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:03:51.419 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 8, 2024-02-19T21:03:51.419 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 63, 2024-02-19T21:03:51.419 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:03:51.419 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:03:51.419 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:03:51.419 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.419 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.419 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6801", 2024-02-19T21:03:51.419 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:03:51.419 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.419 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.419 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:03:51.420 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.420 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.420 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6802", 2024-02-19T21:03:51.420 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:03:51.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.420 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.420 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:03:51.420 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.420 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.420 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6804", 2024-02-19T21:03:51.421 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:03:51.421 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.421 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.421 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:03:51.421 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.421 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.421 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6803", 2024-02-19T21:03:51.421 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:03:51.421 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.421 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.421 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6801/1559418115", 2024-02-19T21:03:51.422 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6802/1559418115", 2024-02-19T21:03:51.422 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6804/1559418115", 2024-02-19T21:03:51.422 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6803/1559418115", 2024-02-19T21:03:51.422 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:03:51.422 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:03:51.422 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:03:51.422 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.422 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-02-19T21:03:51.422 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "e74a65fa-cb9c-42c3-9293-602cf7e9dc67", 2024-02-19T21:03:51.422 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:03:51.422 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:03:51.423 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:03:51.423 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:03:51.423 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:03:51.423 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:03:51.423 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 13, 2024-02-19T21:03:51.423 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 66, 2024-02-19T21:03:51.423 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:03:51.423 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:03:51.423 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:03:51.423 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.423 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.423 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6805", 2024-02-19T21:03:51.423 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:03:51.423 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.424 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.424 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:03:51.424 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.424 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.424 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6806", 2024-02-19T21:03:51.424 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:03:51.424 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.424 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.424 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:03:51.425 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.425 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.425 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6808", 2024-02-19T21:03:51.425 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:03:51.425 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.425 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.425 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:03:51.425 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.425 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.425 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6807", 2024-02-19T21:03:51.425 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:03:51.426 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.426 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.426 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6805/3125047432", 2024-02-19T21:03:51.426 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6806/3125047432", 2024-02-19T21:03:51.426 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6808/3125047432", 2024-02-19T21:03:51.426 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6807/3125047432", 2024-02-19T21:03:51.426 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:03:51.426 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:03:51.426 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:03:51.426 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.426 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-02-19T21:03:51.427 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "6980a625-cdc9-487e-8f8c-7997ba716a97", 2024-02-19T21:03:51.427 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:03:51.427 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:03:51.427 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:03:51.427 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:03:51.427 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:03:51.427 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:03:51.427 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 18, 2024-02-19T21:03:51.427 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 66, 2024-02-19T21:03:51.427 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:03:51.427 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:03:51.427 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:03:51.427 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.428 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.428 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6809", 2024-02-19T21:03:51.428 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:03:51.428 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.428 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.428 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:03:51.428 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.428 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.428 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6810", 2024-02-19T21:03:51.428 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:03:51.428 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.429 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.429 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:03:51.429 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.429 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.429 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6812", 2024-02-19T21:03:51.429 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:03:51.429 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.429 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.429 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:03:51.429 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.430 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.430 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6811", 2024-02-19T21:03:51.430 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:03:51.430 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.430 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.430 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6809/1563158487", 2024-02-19T21:03:51.430 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6810/1563158487", 2024-02-19T21:03:51.430 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6812/1563158487", 2024-02-19T21:03:51.430 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6811/1563158487", 2024-02-19T21:03:51.430 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:03:51.430 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:03:51.430 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:03:51.430 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5", 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 25, 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 63, 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:03:51.431 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.432 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.432 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6813", 2024-02-19T21:03:51.432 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:03:51.432 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.432 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.432 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:03:51.432 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.432 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.432 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6814", 2024-02-19T21:03:51.432 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:03:51.432 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.433 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.433 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:03:51.433 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.433 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.433 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6816", 2024-02-19T21:03:51.433 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:03:51.433 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.433 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.433 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:03:51.433 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6815", 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6813/2112279769", 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6814/2112279769", 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6816/2112279769", 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6815/2112279769", 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.435 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-02-19T21:03:51.435 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0", 2024-02-19T21:03:51.435 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:03:51.435 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:03:51.435 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:03:51.435 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:03:51.435 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:03:51.435 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:03:51.435 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 31, 2024-02-19T21:03:51.435 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 66, 2024-02-19T21:03:51.435 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:03:51.435 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:03:51.435 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:03:51.435 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.436 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.436 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6800", 2024-02-19T21:03:51.436 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:03:51.436 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.436 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.436 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:03:51.436 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.436 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.436 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6801", 2024-02-19T21:03:51.436 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:03:51.436 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.436 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.437 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:03:51.437 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.437 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.437 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6803", 2024-02-19T21:03:51.437 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:03:51.437 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.437 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.437 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:03:51.437 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.437 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.438 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6802", 2024-02-19T21:03:51.438 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:03:51.438 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.438 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.438 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6800/3129716965", 2024-02-19T21:03:51.438 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6801/3129716965", 2024-02-19T21:03:51.438 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6803/3129716965", 2024-02-19T21:03:51.438 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6802/3129716965", 2024-02-19T21:03:51.438 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:03:51.438 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:03:51.438 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:03:51.438 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.439 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-02-19T21:03:51.439 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f", 2024-02-19T21:03:51.439 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:03:51.439 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:03:51.439 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:03:51.439 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:03:51.439 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:03:51.439 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:03:51.439 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 37, 2024-02-19T21:03:51.439 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 66, 2024-02-19T21:03:51.439 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:03:51.439 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:03:51.439 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:03:51.439 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.440 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.440 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6804", 2024-02-19T21:03:51.440 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:03:51.440 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.440 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.440 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:03:51.440 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.440 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.440 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6805", 2024-02-19T21:03:51.440 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:03:51.440 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.440 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.441 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:03:51.441 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.441 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.441 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6807", 2024-02-19T21:03:51.441 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:03:51.441 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.441 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.441 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:03:51.441 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.441 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.441 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6806", 2024-02-19T21:03:51.442 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:03:51.442 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.442 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.442 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6804/805161991", 2024-02-19T21:03:51.442 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6805/805161991", 2024-02-19T21:03:51.442 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6807/805161991", 2024-02-19T21:03:51.442 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6806/805161991", 2024-02-19T21:03:51.442 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:03:51.442 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:03:51.442 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:03:51.442 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.442 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "302c95df-3842-4818-8afd-d79d8184684f", 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 45, 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 61, 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.443 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6808", 2024-02-19T21:03:51.444 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:03:51.444 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.444 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.444 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:03:51.444 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.444 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.444 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6809", 2024-02-19T21:03:51.444 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:03:51.444 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.444 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.445 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:03:51.445 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.445 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.445 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6811", 2024-02-19T21:03:51.445 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:03:51.445 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.445 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.445 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:03:51.445 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.445 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.445 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6810", 2024-02-19T21:03:51.446 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:03:51.446 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.446 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.446 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6808/787604615", 2024-02-19T21:03:51.446 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6809/787604615", 2024-02-19T21:03:51.446 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6811/787604615", 2024-02-19T21:03:51.446 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6810/787604615", 2024-02-19T21:03:51.446 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:03:51.446 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:03:51.446 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:03:51.446 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f", 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 53, 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 63, 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.447 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.448 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6812", 2024-02-19T21:03:51.448 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:03:51.448 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.448 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.448 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:03:51.448 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.448 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.448 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6813", 2024-02-19T21:03:51.448 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:03:51.448 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.449 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.449 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:03:51.449 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.449 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.449 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6815", 2024-02-19T21:03:51.449 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:03:51.449 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.449 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.449 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:03:51.449 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:03:51.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.450 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:03:51.450 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6814", 2024-02-19T21:03:51.450 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:03:51.450 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.450 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.450 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6812/157927432", 2024-02-19T21:03:51.450 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6813/157927432", 2024-02-19T21:03:51.450 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6815/157927432", 2024-02-19T21:03:51.450 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6814/157927432", 2024-02-19T21:03:51.450 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:03:51.450 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:03:51.450 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:03:51.450 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.451 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.451 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:03:51.451 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_xinfo": [ 2024-02-19T21:03:51.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.451 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-02-19T21:03:51.451 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:03:51.451 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:03:51.451 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:03:51.451 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:03:51.451 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:03:51.451 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:03:31.058939+0000", 2024-02-19T21:03:51.451 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:03:51.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.452 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-02-19T21:03:51.452 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:03:51.452 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:03:51.452 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:03:51.452 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:03:51.452 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:03:51.452 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:03:20.091925+0000", 2024-02-19T21:03:51.452 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:03:51.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.452 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-02-19T21:03:51.452 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:03:51.452 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:03:51.452 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:03:51.452 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:02:51.609609+0000", 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:02:51.535933+0000", 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.453 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-02-19T21:03:51.454 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:03:51.454 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:03:51.454 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:03:51.454 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:03:51.454 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:03:51.454 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:03:27.284025+0000", 2024-02-19T21:03:51.454 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:03:51.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.368+0000 7fba323a1700 1 -- 172.21.15.163:0/3657996602 >> v1:172.21.15.163:6800/1552514681 conn(0x7fba18063e10 legacy=0x7fba180662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:51.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.368+0000 7fba323a1700 1 -- 172.21.15.163:0/3657996602 >> v1:172.21.15.163:6789/0 conn(0x7fba2c0f9340 legacy=0x7fba2c114b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:51.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.368+0000 7fba323a1700 1 -- 172.21.15.163:0/3657996602 shutdown_connections 2024-02-19T21:03:51.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.368+0000 7fba323a1700 1 -- 172.21.15.163:0/3657996602 >> 172.21.15.163:0/3657996602 conn(0x7fba2c0f22c0 msgr2=0x7fba2c1037e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:51.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.368+0000 7fba323a1700 1 -- 172.21.15.163:0/3657996602 shutdown_connections 2024-02-19T21:03:51.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.368+0000 7fba323a1700 1 -- 172.21.15.163:0/3657996602 wait complete. 2024-02-19T21:03:51.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.368+0000 7fba323a1700 1 librados: shutdown 2024-02-19T21:03:51.455 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-02-19T21:03:51.455 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:03:51.455 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:03:51.455 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:03:51.455 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:03:51.455 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:03:51.456 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:02:36.147449+0000", 2024-02-19T21:03:51.456 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:03:51.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.456 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-02-19T21:03:51.456 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:03:51.456 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:03:51.456 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:03:51.456 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:03:51.456 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:03:51.456 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:03:10.929144+0000", 2024-02-19T21:03:51.456 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:03:51.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.457 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-02-19T21:03:51.457 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:03:51.457 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:03:51.457 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:03:51.457 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:03:51.457 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:03:51.457 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:03:25.518529+0000", 2024-02-19T21:03:51.457 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:03:51.457 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.457 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:03:51.457 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap": [], 2024-02-19T21:03:51.457 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_items": [ 2024-02-19T21:03:51.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.457 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "2.4", 2024-02-19T21:03:51.458 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:03:51.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.458 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 7, 2024-02-19T21:03:51.458 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-02-19T21:03:51.458 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.458 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.458 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.7", 2024-02-19T21:03:51.458 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:03:51.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.458 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:03:51.458 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-02-19T21:03:51.458 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.458 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.d", 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 5 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.12", 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:03:51.459 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 5 2024-02-19T21:03:51.460 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.460 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:03:51.460 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.460 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:03:51.460 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_temp": [], 2024-02-19T21:03:51.460 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_temp": [], 2024-02-19T21:03:51.460 INFO:teuthology.orchestra.run.smithi163.stdout: "blocklist": { 2024-02-19T21:03:51.460 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1427774891": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:03:51.460 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3282102947": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:03:51.460 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2735092549": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:03:51.460 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1235588846": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:03:51.460 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2731225015": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:03:51.460 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/506491651": "2024-02-20T20:56:02.185409+0000", 2024-02-19T21:03:51.460 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/168117826": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:03:51.461 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1458009465": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:03:51.461 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/541631472": "2024-02-20T20:56:02.185409+0000", 2024-02-19T21:03:51.461 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2248999541": "2024-02-20T20:56:02.185409+0000" 2024-02-19T21:03:51.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.461 INFO:teuthology.orchestra.run.smithi163.stdout: "range_blocklist": {}, 2024-02-19T21:03:51.461 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profiles": { 2024-02-19T21:03:51.461 INFO:teuthology.orchestra.run.smithi163.stdout: "default": { 2024-02-19T21:03:51.461 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-02-19T21:03:51.461 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-02-19T21:03:51.461 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-02-19T21:03:51.461 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-02-19T21:03:51.461 INFO:teuthology.orchestra.run.smithi163.stdout: "ruleset-failure-domain": "osd", 2024-02-19T21:03:51.461 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van" 2024-02-19T21:03:51.461 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:03:51.462 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps_queue": [], 2024-02-19T21:03:51.462 INFO:teuthology.orchestra.run.smithi163.stdout: "new_removed_snaps": [], 2024-02-19T21:03:51.462 INFO:teuthology.orchestra.run.smithi163.stdout: "new_purged_snaps": [], 2024-02-19T21:03:51.462 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_node_flags": {}, 2024-02-19T21:03:51.462 INFO:teuthology.orchestra.run.smithi163.stdout: "device_class_flags": {}, 2024-02-19T21:03:51.462 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode": { 2024-02-19T21:03:51.462 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_enabled": false, 2024-02-19T21:03:51.462 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_bucket_count": 0, 2024-02-19T21:03:51.462 INFO:teuthology.orchestra.run.smithi163.stdout: "degraded_stretch_mode": 0, 2024-02-19T21:03:51.462 INFO:teuthology.orchestra.run.smithi163.stdout: "recovering_stretch_mode": 0, 2024-02-19T21:03:51.462 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_bucket": 0 2024-02-19T21:03:51.462 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:03:51.462 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:03:51.462 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:51.533 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:50.040273+0000 mgr.y (mgr.14550) 106 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:51.533 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:51.365628+0000 mon.a (mon.0) 721 : audit [DBG] from='client.? v1:172.21.15.163:0/3657996602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:03:51.533 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:50.040273+0000 mgr.y (mgr.14550) 106 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:51.533 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:51.365628+0000 mon.a (mon.0) 721 : audit [DBG] from='client.? v1:172.21.15.163:0/3657996602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:03:51.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.584+0000 7f258d431700 1 -- 172.21.15.163:0/1727935871 >> v1:172.21.15.163:6789/0 conn(0x7f25880f94d0 legacy=0x7f25881029a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:51.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.584+0000 7f258d431700 1 -- 172.21.15.163:0/1727935871 shutdown_connections 2024-02-19T21:03:51.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.584+0000 7f258d431700 1 -- 172.21.15.163:0/1727935871 >> 172.21.15.163:0/1727935871 conn(0x7f25880f2470 msgr2=0x7f25880f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:51.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.584+0000 7f258d431700 1 -- 172.21.15.163:0/1727935871 shutdown_connections 2024-02-19T21:03:51.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.584+0000 7f258d431700 1 -- 172.21.15.163:0/1727935871 wait complete. 2024-02-19T21:03:51.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.584+0000 7f258d431700 1 librados: starting msgr at 2024-02-19T21:03:51.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.584+0000 7f258d431700 1 librados: starting objecter 2024-02-19T21:03:51.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f258d431700 1 Processor -- start 2024-02-19T21:03:51.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f258d431700 1 -- start start 2024-02-19T21:03:51.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f258d431700 1 librados: setting wanted keys 2024-02-19T21:03:51.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f258d431700 1 librados: calling monclient init 2024-02-19T21:03:51.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f258d431700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2588102a90 con 0x7f25880f8920 2024-02-19T21:03:51.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f258d431700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2588102c60 con 0x7f25881042b0 2024-02-19T21:03:51.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f258d431700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2588102e50 con 0x7f25880f94d0 2024-02-19T21:03:51.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f2586ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f25880f8920 0x7f25881022f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44536/0 (socket says 172.21.15.163:44536) 2024-02-19T21:03:51.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f2586ffd700 1 -- 172.21.15.163:0/1197481544 learned_addr learned my addr 172.21.15.163:0/1197481544 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:51.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f256ffff700 10 client.?.objecter ms_handle_connect 0x7f25880f8920 2024-02-19T21:03:51.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f256ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:51.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f256ffff700 10 client.?.objecter ms_handle_connect 0x7f25880f94d0 2024-02-19T21:03:51.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f256ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:51.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889892631 0 0) 0x7f2588102a90 con 0x7f25880f8920 2024-02-19T21:03:51.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2570003030 con 0x7f25880f8920 2024-02-19T21:03:51.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f256ffff700 10 client.?.objecter ms_handle_connect 0x7f25881042b0 2024-02-19T21:03:51.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f256ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:51.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594666155 0 0) 0x7f2588102e50 con 0x7f25880f94d0 2024-02-19T21:03:51.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.585+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2588102a90 con 0x7f25880f94d0 2024-02-19T21:03:51.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572879046 0 0) 0x7f2588102c60 con 0x7f25881042b0 2024-02-19T21:03:51.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2588102e50 con 0x7f25881042b0 2024-02-19T21:03:51.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3074408763 0 0) 0x7f2570003030 con 0x7f25880f8920 2024-02-19T21:03:51.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2588102c60 con 0x7f25880f8920 2024-02-19T21:03:51.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1873441126 0 0) 0x7f2588102a90 con 0x7f25880f94d0 2024-02-19T21:03:51.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2570003030 con 0x7f25880f94d0 2024-02-19T21:03:51.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f25780018e0 con 0x7f25880f8920 2024-02-19T21:03:51.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f257c001ec0 con 0x7f25880f94d0 2024-02-19T21:03:51.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1952076918 0 0) 0x7f2588102e50 con 0x7f25881042b0 2024-02-19T21:03:51.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2588102a90 con 0x7f25881042b0 2024-02-19T21:03:51.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2580001c00 con 0x7f25881042b0 2024-02-19T21:03:51.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1827499499 0 0) 0x7f2588102c60 con 0x7f25880f8920 2024-02-19T21:03:51.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 >> v1:172.21.15.163:6790/0 conn(0x7f25880f94d0 legacy=0x7f2588114d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:51.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 >> v1:172.21.15.193:6789/0 conn(0x7f25881042b0 legacy=0x7f2588118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:51.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258811cb80 con 0x7f25880f8920 2024-02-19T21:03:51.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f258d431700 1 -- 172.21.15.163:0/1197481544 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2588119bb0 con 0x7f25880f8920 2024-02-19T21:03:51.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f258d431700 10 client.14814.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:51.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f258d431700 1 -- 172.21.15.163:0/1197481544 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f258811a020 con 0x7f25880f8920 2024-02-19T21:03:51.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f258d431700 1 librados: init done 2024-02-19T21:03:51.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f2578001f70 con 0x7f25880f8920 2024-02-19T21:03:51.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.586+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2578005a20 con 0x7f25880f8920 2024-02-19T21:03:51.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.587+0000 7f258d431700 1 -- 172.21.15.163:0/1197481544 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2574004f20 con 0x7f25880f8920 2024-02-19T21:03:51.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.588+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f2578001ad0 con 0x7f25880f8920 2024-02-19T21:03:51.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.588+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(88..88 src has 32..88) v4 ==== 6172+0+0 (unknown 1929784803 0 0) 0x7f2578078550 con 0x7f25880f8920 2024-02-19T21:03:51.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.588+0000 7f256ffff700 10 client.14814.objecter ms_dispatch 0x7f2588001550 osd_map(88..88 src has 32..88) v4 2024-02-19T21:03:51.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.588+0000 7f256ffff700 3 client.14814.objecter handle_osd_map got epochs [88,88] > 0 2024-02-19T21:03:51.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.588+0000 7f256ffff700 3 client.14814.objecter handle_osd_map decoding full epoch 88 2024-02-19T21:03:51.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.588+0000 7f256ffff700 20 client.14814.objecter dump_active .. 0 homeless 2024-02-19T21:03:51.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.589+0000 7f256ffff700 10 client.14814.objecter ms_handle_connect 0x7f2570063e00 2024-02-19T21:03:51.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.593+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f257804c100 con 0x7f25880f8920 2024-02-19T21:03:51.587 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:03:51.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:51.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:03:51.691 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:03:51.719 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:51.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:51.818+0000 7f258d431700 1 -- 172.21.15.163:0/1197481544 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f25740051d0 con 0x7f25880f8920 2024-02-19T21:03:51.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:50.040273+0000 mgr.y (mgr.14550) 106 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:03:51.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:51.365628+0000 mon.a (mon.0) 721 : audit [DBG] from='client.? v1:172.21.15.163:0/3657996602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:03:52.309 INFO:tasks.thrashosds.thrasher:j is {'epoch': 88, 'fsid': '2591809a-cf69-11ee-95bb-87774f69a715', 'created': '2024-02-19T20:55:37.794885+0000', 'modified': '2024-02-19T21:03:42.499726+0000', 'last_up_change': '2024-02-19T21:01:09.214134+0000', 'last_in_change': '2024-02-19T21:00:51.699017+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 6, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-19T20:59:09.425169+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-19T21:01:14.658945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-19T21:01:14.958800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-19T21:01:16.543816+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-19T21:01:18.521482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-19T21:01:20.586564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}], 'osds': [{'osd': 0, 'uuid': '4a14b406-7c21-40f3-8b12-8cf9df811088', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 63, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6801', 'nonce': 1559418115}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6802', 'nonce': 1559418115}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6804', 'nonce': 1559418115}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6803', 'nonce': 1559418115}]}, 'public_addr': '172.21.15.163:6801/1559418115', 'cluster_addr': '172.21.15.163:6802/1559418115', 'heartbeat_back_addr': '172.21.15.163:6804/1559418115', 'heartbeat_front_addr': '172.21.15.163:6803/1559418115', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e74a65fa-cb9c-42c3-9293-602cf7e9dc67', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 66, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6805', 'nonce': 3125047432}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6806', 'nonce': 3125047432}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6808', 'nonce': 3125047432}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6807', 'nonce': 3125047432}]}, 'public_addr': '172.21.15.163:6805/3125047432', 'cluster_addr': '172.21.15.163:6806/3125047432', 'heartbeat_back_addr': '172.21.15.163:6808/3125047432', 'heartbeat_front_addr': '172.21.15.163:6807/3125047432', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6980a625-cdc9-487e-8f8c-7997ba716a97', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 66, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6809', 'nonce': 1563158487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6810', 'nonce': 1563158487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6812', 'nonce': 1563158487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6811', 'nonce': 1563158487}]}, 'public_addr': '172.21.15.163:6809/1563158487', 'cluster_addr': '172.21.15.163:6810/1563158487', 'heartbeat_back_addr': '172.21.15.163:6812/1563158487', 'heartbeat_front_addr': '172.21.15.163:6811/1563158487', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '359e2ee0-9d1d-4074-8119-2d69bc2ca3a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 63, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6813', 'nonce': 2112279769}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6814', 'nonce': 2112279769}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6816', 'nonce': 2112279769}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6815', 'nonce': 2112279769}]}, 'public_addr': '172.21.15.163:6813/2112279769', 'cluster_addr': '172.21.15.163:6814/2112279769', 'heartbeat_back_addr': '172.21.15.163:6816/2112279769', 'heartbeat_front_addr': '172.21.15.163:6815/2112279769', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2bf5ef05-292b-4693-85a9-97991f2091e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 66, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6800', 'nonce': 3129716965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6801', 'nonce': 3129716965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6803', 'nonce': 3129716965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6802', 'nonce': 3129716965}]}, 'public_addr': '172.21.15.193:6800/3129716965', 'cluster_addr': '172.21.15.193:6801/3129716965', 'heartbeat_back_addr': '172.21.15.193:6803/3129716965', 'heartbeat_front_addr': '172.21.15.193:6802/3129716965', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '59065f9b-9240-414b-9c91-e91cda4aed6f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 66, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6804', 'nonce': 805161991}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6805', 'nonce': 805161991}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6807', 'nonce': 805161991}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6806', 'nonce': 805161991}]}, 'public_addr': '172.21.15.193:6804/805161991', 'cluster_addr': '172.21.15.193:6805/805161991', 'heartbeat_back_addr': '172.21.15.193:6807/805161991', 'heartbeat_front_addr': '172.21.15.193:6806/805161991', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '302c95df-3842-4818-8afd-d79d8184684f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 61, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6808', 'nonce': 787604615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6809', 'nonce': 787604615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6811', 'nonce': 787604615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6810', 'nonce': 787604615}]}, 'public_addr': '172.21.15.193:6808/787604615', 'cluster_addr': '172.21.15.193:6809/787604615', 'heartbeat_back_addr': '172.21.15.193:6811/787604615', 'heartbeat_front_addr': '172.21.15.193:6810/787604615', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '9ebeff27-c2f8-4a66-923f-a0d7573ddd3f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 63, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6812', 'nonce': 157927432}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6813', 'nonce': 157927432}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6815', 'nonce': 157927432}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6814', 'nonce': 157927432}]}, 'public_addr': '172.21.15.193:6812/157927432', 'cluster_addr': '172.21.15.193:6813/157927432', 'heartbeat_back_addr': '172.21.15.193:6815/157927432', 'heartbeat_front_addr': '172.21.15.193:6814/157927432', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:03:31.058939+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:03:20.091925+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:02:51.609609+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:02:51.535933+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:03:27.284025+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:02:36.147449+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:03:10.929144+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:03:25.518529+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.4', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.163:0/1427774891': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/3282102947': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:6800/2735092549': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/1235588846': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:6800/2731225015': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:0/506491651': '2024-02-20T20:56:02.185409+0000', '172.21.15.163:0/168117826': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/1458009465': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:0/541631472': '2024-02-20T20:56:02.185409+0000', '172.21.15.163:6800/2248999541': '2024-02-20T20:56:02.185409+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-19T21:03:52.310 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:03:52.358 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:52.359 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:52.374 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:03:52.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:52.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:03:52.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:03:52.509 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:52.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:52.552+0000 7f256ffff700 1 -- 172.21.15.163:0/1197481544 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v89) v1 ==== 97+0+0 (unknown 2770807327 0 0) 0x7f257804a510 con 0x7f25880f8920 2024-02-19T21:03:52.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:52.554+0000 7f258d431700 10 librados: watch_flush enter 2024-02-19T21:03:52.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:52.555+0000 7f258d431700 10 librados: watch_flush exit 2024-02-19T21:03:52.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:52.555+0000 7f258d431700 20 client.14814.objecter shutdown clearing up homeless session... 2024-02-19T21:03:52.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:52.555+0000 7f258d431700 10 client.14814.objecter successfully canceled tick 2024-02-19T21:03:52.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:52.555+0000 7f258d431700 1 -- 172.21.15.163:0/1197481544 >> v1:172.21.15.163:6800/1552514681 conn(0x7f2570063e00 legacy=0x7f25700662c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:52.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:52.555+0000 7f258d431700 1 -- 172.21.15.163:0/1197481544 >> v1:172.21.15.163:6789/0 conn(0x7f25880f8920 legacy=0x7f25881022f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:52.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:52.556+0000 7f258d431700 1 -- 172.21.15.163:0/1197481544 shutdown_connections 2024-02-19T21:03:52.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:52.556+0000 7f258d431700 1 -- 172.21.15.163:0/1197481544 >> 172.21.15.163:0/1197481544 conn(0x7f25880f2470 msgr2=0x7f2588107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:52.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:52.556+0000 7f258d431700 1 -- 172.21.15.163:0/1197481544 shutdown_connections 2024-02-19T21:03:52.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:52.556+0000 7f258d431700 1 -- 172.21.15.163:0/1197481544 wait complete. 2024-02-19T21:03:52.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:52.556+0000 7f258d431700 1 librados: shutdown 2024-02-19T21:03:52.549 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:03:52.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:50.942124+0000 mgr.y (mgr.14550) 107 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:52.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:51.819834+0000 mon.a (mon.0) 722 : audit [INF] from='client.? v1:172.21.15.163:0/1197481544' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:03:52.577 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:50.942124+0000 mgr.y (mgr.14550) 107 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:52.577 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:51.819834+0000 mon.a (mon.0) 722 : audit [INF] from='client.? v1:172.21.15.163:0/1197481544' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:03:52.615 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:03:52.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:52.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:03:52.720 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:03:52.752 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:52.772 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:03:52.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:52.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:03:52.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:50.942124+0000 mgr.y (mgr.14550) 107 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:52.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:51.819834+0000 mon.a (mon.0) 722 : audit [INF] from='client.? v1:172.21.15.163:0/1197481544' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:03:52.879 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:03:52.910 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:52.931 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:03:52.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:52.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:03:53.035 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:03:53.064 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:53.714 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:03:53.715 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:03:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:03:53] "GET /metrics HTTP/1.1" 200 172169 "" "Prometheus/2.33.4" 2024-02-19T21:03:53.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:52.538198+0000 mon.a (mon.0) 723 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:03:53.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:52.553349+0000 mon.a (mon.0) 724 : audit [INF] from='client.? v1:172.21.15.163:0/1197481544' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:03:53.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:52.553409+0000 mon.a (mon.0) 725 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-02-19T21:03:53.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:52.538198+0000 mon.a (mon.0) 723 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:03:53.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:52.553349+0000 mon.a (mon.0) 724 : audit [INF] from='client.? v1:172.21.15.163:0/1197481544' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:03:53.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:52.553409+0000 mon.a (mon.0) 725 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-02-19T21:03:53.717 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:53.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:03:53.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:52.538198+0000 mon.a (mon.0) 723 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:03:53.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:52.553349+0000 mon.a (mon.0) 724 : audit [INF] from='client.? v1:172.21.15.163:0/1197481544' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:03:53.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:52.553409+0000 mon.a (mon.0) 725 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-02-19T21:03:53.818 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:03:53.850 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:53.871 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:03:53.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:53.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:03:53.975 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:03:54.004 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:54.024 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:03:54.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:54.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:03:54.128 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:03:54.156 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:54.707 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:03:54.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:54.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:03:54.811 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:03:54.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:52.942445+0000 mgr.y (mgr.14550) 108 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:54.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:54.072473+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:03:54.838 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:52.942445+0000 mgr.y (mgr.14550) 108 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:54.838 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:54.072473+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:03:54.838 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:52.942445+0000 mgr.y (mgr.14550) 108 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:54.839 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:54.072473+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:03:54.841 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:55.041 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:03:55.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:55.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:03:55.145 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:03:55.179 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:55.257 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:03:55.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:55.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:03:55.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.279+0000 7f581247d700 1 -- 172.21.15.163:0/3948573574 >> v1:172.21.15.163:6790/0 conn(0x7f580c100780 legacy=0x7f580c102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:55.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.279+0000 7f581247d700 1 -- 172.21.15.163:0/3948573574 shutdown_connections 2024-02-19T21:03:55.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.279+0000 7f581247d700 1 -- 172.21.15.163:0/3948573574 >> 172.21.15.163:0/3948573574 conn(0x7f580c0f35a0 msgr2=0x7f580c0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:55.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.279+0000 7f581247d700 1 -- 172.21.15.163:0/3948573574 shutdown_connections 2024-02-19T21:03:55.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.279+0000 7f581247d700 1 -- 172.21.15.163:0/3948573574 wait complete. 2024-02-19T21:03:55.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.279+0000 7f581247d700 1 librados: starting msgr at 2024-02-19T21:03:55.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.279+0000 7f581247d700 1 librados: starting objecter 2024-02-19T21:03:55.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.280+0000 7f581247d700 1 Processor -- start 2024-02-19T21:03:55.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.280+0000 7f581247d700 1 -- start start 2024-02-19T21:03:55.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.280+0000 7f581247d700 1 librados: setting wanted keys 2024-02-19T21:03:55.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.280+0000 7f581247d700 1 librados: calling monclient init 2024-02-19T21:03:55.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.280+0000 7f581247d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f580c115210 con 0x7f580c100780 2024-02-19T21:03:55.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.280+0000 7f581247d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f580c1163f0 con 0x7f580c1042c0 2024-02-19T21:03:55.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.280+0000 7f581247d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f580c1175f0 con 0x7f580c0fd980 2024-02-19T21:03:55.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.280+0000 7f580bfff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f580c0fd980 0x7f580c102580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:58764/0 (socket says 172.21.15.163:58764) 2024-02-19T21:03:55.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.280+0000 7f580bfff700 1 -- 172.21.15.163:0/604313359 learned_addr learned my addr 172.21.15.163:0/604313359 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:55.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.280+0000 7f58097fa700 10 client.?.objecter ms_handle_connect 0x7f580c0fd980 2024-02-19T21:03:55.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.280+0000 7f58097fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:55.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.280+0000 7f58097fa700 10 client.?.objecter ms_handle_connect 0x7f580c100780 2024-02-19T21:03:55.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.280+0000 7f58097fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:55.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308862966 0 0) 0x7f580c1175f0 con 0x7f580c0fd980 2024-02-19T21:03:55.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57f40033f0 con 0x7f580c0fd980 2024-02-19T21:03:55.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 10 client.?.objecter ms_handle_connect 0x7f580c1042c0 2024-02-19T21:03:55.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:55.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886745322 0 0) 0x7f580c115210 con 0x7f580c100780 2024-02-19T21:03:55.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f580c1175f0 con 0x7f580c100780 2024-02-19T21:03:55.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248315201 0 0) 0x7f580c1163f0 con 0x7f580c1042c0 2024-02-19T21:03:55.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f580c115210 con 0x7f580c1042c0 2024-02-19T21:03:55.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1803965720 0 0) 0x7f57f40033f0 con 0x7f580c0fd980 2024-02-19T21:03:55.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f580c1163f0 con 0x7f580c0fd980 2024-02-19T21:03:55.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f57fc0029d0 con 0x7f580c0fd980 2024-02-19T21:03:55.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 217200216 0 0) 0x7f580c1175f0 con 0x7f580c100780 2024-02-19T21:03:55.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57f40033f0 con 0x7f580c100780 2024-02-19T21:03:55.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5800002d60 con 0x7f580c100780 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2398896598 0 0) 0x7f580c115210 con 0x7f580c1042c0 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f580c1175f0 con 0x7f580c1042c0 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2119995867 0 0) 0x7f580c1163f0 con 0x7f580c0fd980 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 >> v1:172.21.15.193:6789/0 conn(0x7f580c1042c0 legacy=0x7f580c113920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 >> v1:172.21.15.163:6789/0 conn(0x7f580c100780 legacy=0x7f580c102c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c1187f0 con 0x7f580c0fd980 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580c115400 con 0x7f580c0fd980 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f581247d700 10 client.34742.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f580c1158f0 con 0x7f580c0fd980 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f581247d700 1 librados: init done 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.281+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f57fc002da0 con 0x7f580c0fd980 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.282+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f57fc005c30 con 0x7f580c0fd980 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.282+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f57f8004f00 con 0x7f580c0fd980 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.283+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f57fc018b40 con 0x7f580c0fd980 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.283+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(89..89 src has 32..89) v4 ==== 6172+0+0 (unknown 1681062987 0 0) 0x7f57fc0786a0 con 0x7f580c0fd980 2024-02-19T21:03:55.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.283+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 osd_map(89..89 src has 32..89) v4 2024-02-19T21:03:55.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.283+0000 7f58097fa700 3 client.34742.objecter handle_osd_map got epochs [89,89] > 0 2024-02-19T21:03:55.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.283+0000 7f58097fa700 3 client.34742.objecter handle_osd_map decoding full epoch 89 2024-02-19T21:03:55.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.283+0000 7f58097fa700 20 client.34742.objecter dump_active .. 0 homeless 2024-02-19T21:03:55.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.283+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v89) v1 ==== 54+0+16 (unknown 974353933 0 2420000192) 0x7f57fc07aaf0 con 0x7f580c0fd980 2024-02-19T21:03:55.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.284+0000 7f58097fa700 10 client.34742.objecter ms_handle_connect 0x7f57f4068060 2024-02-19T21:03:55.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.284+0000 7f581247d700 10 client.34742.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:55.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.284+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 1 2024-02-19T21:03:55.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.284+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f80051a0 osd=0 3 2024-02-19T21:03:55.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.284+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f80051a0 osd=0 3 2024-02-19T21:03:55.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.284+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f80051a0 osd=0 3 2024-02-19T21:03:55.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.284+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 1 2024-02-19T21:03:55.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.284+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f80051a0 osd=0 3 2024-02-19T21:03:55.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.284+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 0 1 2024-02-19T21:03:55.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.284+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f80051a0 osd=0 4 2024-02-19T21:03:55.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.284+0000 7f581247d700 10 client.34742.objecter _send_command 1 2024-02-19T21:03:55.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.284+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57f8004970 con 0x7f57f8004530 2024-02-19T21:03:55.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.284+0000 7f58097fa700 10 client.34742.objecter ms_handle_connect 0x7f57f8004530 2024-02-19T21:03:55.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.285+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f57f8004970 con 0x7f57f8004530 2024-02-19T21:03:55.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.285+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:03:55.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.285+0000 7f58097fa700 10 client.34742.objecter _finish_command 1 = system:0 2024-02-19T21:03:55.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.285+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f80051a0 osd=0 4 2024-02-19T21:03:55.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.285+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 0 1 2024-02-19T21:03:55.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.300+0000 7f581247d700 10 client.34742.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:03:55.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.300+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 2 2024-02-19T21:03:55.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.300+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f80051a0 osd=0 3 2024-02-19T21:03:55.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.300+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f80051a0 osd=0 3 2024-02-19T21:03:55.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.300+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f80051a0 osd=0 3 2024-02-19T21:03:55.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.300+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 2 2024-02-19T21:03:55.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.300+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f80051a0 osd=0 3 2024-02-19T21:03:55.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.300+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 0 2 2024-02-19T21:03:55.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.300+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f80051a0 osd=0 4 2024-02-19T21:03:55.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.300+0000 7f581247d700 10 client.34742.objecter _send_command 2 2024-02-19T21:03:55.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.300+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f57f8004d50 con 0x7f57f8004530 2024-02-19T21:03:55.294 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:03:55.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.301+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f57f8004d50 con 0x7f57f8004530 2024-02-19T21:03:55.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.301+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:03:55.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.301+0000 7f58097fa700 10 client.34742.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.301+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f80051a0 osd=0 4 2024-02-19T21:03:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.301+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 0 2 2024-02-19T21:03:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.301+0000 7f581247d700 10 client.34742.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.301+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 3 2024-02-19T21:03:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.302+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f8004260 osd=1 3 2024-02-19T21:03:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.302+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f8004260 osd=1 3 2024-02-19T21:03:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.302+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f8004260 osd=1 3 2024-02-19T21:03:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.302+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 3 2024-02-19T21:03:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.302+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f8004260 osd=1 3 2024-02-19T21:03:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.302+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 1 3 2024-02-19T21:03:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.302+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f8004260 osd=1 4 2024-02-19T21:03:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.302+0000 7f581247d700 10 client.34742.objecter _send_command 3 2024-02-19T21:03:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.302+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57f800bef0 con 0x7f57f8002090 2024-02-19T21:03:55.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.302+0000 7f58097fa700 10 client.34742.objecter ms_handle_connect 0x7f57f8002090 2024-02-19T21:03:55.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.303+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f57f800bef0 con 0x7f57f8002090 2024-02-19T21:03:55.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.303+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:03:55.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.303+0000 7f58097fa700 10 client.34742.objecter _finish_command 3 = system:0 2024-02-19T21:03:55.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.303+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f8004260 osd=1 4 2024-02-19T21:03:55.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.303+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 1 3 2024-02-19T21:03:55.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.324+0000 7f581247d700 10 client.34742.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:03:55.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.324+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 4 2024-02-19T21:03:55.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.324+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f8004260 osd=1 3 2024-02-19T21:03:55.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.324+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f8004260 osd=1 3 2024-02-19T21:03:55.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.324+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f8004260 osd=1 3 2024-02-19T21:03:55.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.324+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 4 2024-02-19T21:03:55.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.324+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f8004260 osd=1 3 2024-02-19T21:03:55.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.324+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 1 4 2024-02-19T21:03:55.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.324+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f8004260 osd=1 4 2024-02-19T21:03:55.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.324+0000 7f581247d700 10 client.34742.objecter _send_command 4 2024-02-19T21:03:55.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.324+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f57f80016b0 con 0x7f57f8002090 2024-02-19T21:03:55.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f57f80016b0 con 0x7f57f8002090 2024-02-19T21:03:55.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:03:55.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f58097fa700 10 client.34742.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f8004260 osd=1 4 2024-02-19T21:03:55.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 1 4 2024-02-19T21:03:55.319 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:03:55.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f581247d700 10 client.34742.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:55.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 5 2024-02-19T21:03:55.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f80035f0 osd=2 3 2024-02-19T21:03:55.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f80035f0 osd=2 3 2024-02-19T21:03:55.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f80035f0 osd=2 3 2024-02-19T21:03:55.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 5 2024-02-19T21:03:55.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f80035f0 osd=2 3 2024-02-19T21:03:55.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 2 5 2024-02-19T21:03:55.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f80035f0 osd=2 4 2024-02-19T21:03:55.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f581247d700 10 client.34742.objecter _send_command 5 2024-02-19T21:03:55.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.326+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57f800ddd0 con 0x7f57f800d210 2024-02-19T21:03:55.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.327+0000 7f58097fa700 10 client.34742.objecter ms_handle_connect 0x7f57f800d210 2024-02-19T21:03:55.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.327+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f57f800ddd0 con 0x7f57f800d210 2024-02-19T21:03:55.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.327+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:03:55.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.327+0000 7f58097fa700 10 client.34742.objecter _finish_command 5 = system:0 2024-02-19T21:03:55.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.327+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f80035f0 osd=2 4 2024-02-19T21:03:55.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.327+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 2 5 2024-02-19T21:03:55.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.355+0000 7f581247d700 10 client.34742.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:03:55.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.355+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 6 2024-02-19T21:03:55.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.355+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f80035f0 osd=2 3 2024-02-19T21:03:55.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.355+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f80035f0 osd=2 3 2024-02-19T21:03:55.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.355+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f80035f0 osd=2 3 2024-02-19T21:03:55.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.355+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 6 2024-02-19T21:03:55.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.355+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f80035f0 osd=2 3 2024-02-19T21:03:55.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.355+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 2 6 2024-02-19T21:03:55.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.355+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f80035f0 osd=2 4 2024-02-19T21:03:55.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.355+0000 7f581247d700 10 client.34742.objecter _send_command 6 2024-02-19T21:03:55.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.355+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f57f8003120 con 0x7f57f800d210 2024-02-19T21:03:55.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f57f8003120 con 0x7f57f800d210 2024-02-19T21:03:55.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:03:55.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f58097fa700 10 client.34742.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f80035f0 osd=2 4 2024-02-19T21:03:55.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 2 6 2024-02-19T21:03:55.350 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:03:55.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f581247d700 10 client.34742.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:55.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 7 2024-02-19T21:03:55.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f80118b0 osd=3 3 2024-02-19T21:03:55.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f80118b0 osd=3 3 2024-02-19T21:03:55.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f80118b0 osd=3 3 2024-02-19T21:03:55.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 7 2024-02-19T21:03:55.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f80118b0 osd=3 3 2024-02-19T21:03:55.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 3 7 2024-02-19T21:03:55.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f80118b0 osd=3 4 2024-02-19T21:03:55.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f581247d700 10 client.34742.objecter _send_command 7 2024-02-19T21:03:55.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.357+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57f8015ae0 con 0x7f57f8011f30 2024-02-19T21:03:55.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.358+0000 7f58097fa700 10 client.34742.objecter ms_handle_connect 0x7f57f8011f30 2024-02-19T21:03:55.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.359+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f57f8015ae0 con 0x7f57f8011f30 2024-02-19T21:03:55.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.359+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:03:55.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.359+0000 7f58097fa700 10 client.34742.objecter _finish_command 7 = system:0 2024-02-19T21:03:55.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.359+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f80118b0 osd=3 4 2024-02-19T21:03:55.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.359+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 3 7 2024-02-19T21:03:55.361 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:03:55.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.384+0000 7f581247d700 10 client.34742.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:03:55.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.384+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 8 2024-02-19T21:03:55.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.384+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f80118b0 osd=3 3 2024-02-19T21:03:55.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.384+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f80118b0 osd=3 3 2024-02-19T21:03:55.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.384+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f80118b0 osd=3 3 2024-02-19T21:03:55.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.384+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 8 2024-02-19T21:03:55.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.384+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f80118b0 osd=3 3 2024-02-19T21:03:55.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.384+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 3 8 2024-02-19T21:03:55.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.384+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f80118b0 osd=3 4 2024-02-19T21:03:55.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.384+0000 7f581247d700 10 client.34742.objecter _send_command 8 2024-02-19T21:03:55.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.384+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f57f800e1b0 con 0x7f57f8011f30 2024-02-19T21:03:55.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f57f800e1b0 con 0x7f57f8011f30 2024-02-19T21:03:55.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:03:55.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f58097fa700 10 client.34742.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f80118b0 osd=3 4 2024-02-19T21:03:55.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 3 8 2024-02-19T21:03:55.378 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:03:55.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f581247d700 10 client.34742.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:55.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 9 2024-02-19T21:03:55.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f800df70 osd=4 3 2024-02-19T21:03:55.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f800df70 osd=4 3 2024-02-19T21:03:55.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f800df70 osd=4 3 2024-02-19T21:03:55.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 9 2024-02-19T21:03:55.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f800df70 osd=4 3 2024-02-19T21:03:55.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 4 9 2024-02-19T21:03:55.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f800df70 osd=4 4 2024-02-19T21:03:55.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f581247d700 10 client.34742.objecter _send_command 9 2024-02-19T21:03:55.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.385+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57f800e120 con 0x7f57f800cd70 2024-02-19T21:03:55.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.386+0000 7f58097fa700 10 client.34742.objecter ms_handle_connect 0x7f57f800cd70 2024-02-19T21:03:55.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.387+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f57f800e120 con 0x7f57f800cd70 2024-02-19T21:03:55.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.387+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:03:55.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.387+0000 7f58097fa700 10 client.34742.objecter _finish_command 9 = system:0 2024-02-19T21:03:55.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.387+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f800df70 osd=4 4 2024-02-19T21:03:55.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.387+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 4 9 2024-02-19T21:03:55.393 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:55.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.415+0000 7f581247d700 10 client.34742.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:03:55.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.415+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 10 2024-02-19T21:03:55.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.415+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f800df70 osd=4 3 2024-02-19T21:03:55.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.415+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f800df70 osd=4 3 2024-02-19T21:03:55.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.415+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f800df70 osd=4 3 2024-02-19T21:03:55.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.415+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 10 2024-02-19T21:03:55.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.415+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f800df70 osd=4 3 2024-02-19T21:03:55.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.415+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 4 10 2024-02-19T21:03:55.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.415+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f800df70 osd=4 4 2024-02-19T21:03:55.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.415+0000 7f581247d700 10 client.34742.objecter _send_command 10 2024-02-19T21:03:55.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.415+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f57f800e120 con 0x7f57f800cd70 2024-02-19T21:03:55.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f57f800e120 con 0x7f57f800cd70 2024-02-19T21:03:55.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:03:55.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f58097fa700 10 client.34742.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f800df70 osd=4 4 2024-02-19T21:03:55.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 4 10 2024-02-19T21:03:55.409 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:03:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f581247d700 10 client.34742.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 11 2024-02-19T21:03:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f800e120 osd=5 3 2024-02-19T21:03:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f800e120 osd=5 3 2024-02-19T21:03:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f800e120 osd=5 3 2024-02-19T21:03:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 11 2024-02-19T21:03:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f800e120 osd=5 3 2024-02-19T21:03:55.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 5 11 2024-02-19T21:03:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f800e120 osd=5 4 2024-02-19T21:03:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f581247d700 10 client.34742.objecter _send_command 11 2024-02-19T21:03:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57f80180a0 con 0x7f57f8017a70 2024-02-19T21:03:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.417+0000 7f58097fa700 10 client.34742.objecter ms_handle_connect 0x7f57f8017a70 2024-02-19T21:03:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.418+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f57f80180a0 con 0x7f57f8017a70 2024-02-19T21:03:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.418+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:03:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.418+0000 7f58097fa700 10 client.34742.objecter _finish_command 11 = system:0 2024-02-19T21:03:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.418+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f800e120 osd=5 4 2024-02-19T21:03:55.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.418+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 5 11 2024-02-19T21:03:55.415 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:03:55.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:55.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:03:55.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.446+0000 7f581247d700 10 client.34742.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:03:55.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.446+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 12 2024-02-19T21:03:55.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.446+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f800e120 osd=5 3 2024-02-19T21:03:55.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.446+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f800e120 osd=5 3 2024-02-19T21:03:55.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.446+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f800e120 osd=5 3 2024-02-19T21:03:55.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.446+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 12 2024-02-19T21:03:55.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.446+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f800e120 osd=5 3 2024-02-19T21:03:55.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.446+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 5 12 2024-02-19T21:03:55.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.446+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f800e120 osd=5 4 2024-02-19T21:03:55.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.446+0000 7f581247d700 10 client.34742.objecter _send_command 12 2024-02-19T21:03:55.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.446+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f57f8018270 con 0x7f57f8017a70 2024-02-19T21:03:55.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f57f8018270 con 0x7f57f8017a70 2024-02-19T21:03:55.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:03:55.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f58097fa700 10 client.34742.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f800e120 osd=5 4 2024-02-19T21:03:55.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 5 12 2024-02-19T21:03:55.441 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:03:55.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f581247d700 10 client.34742.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:55.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 13 2024-02-19T21:03:55.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f8018060 osd=6 3 2024-02-19T21:03:55.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f8018060 osd=6 3 2024-02-19T21:03:55.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f8018060 osd=6 3 2024-02-19T21:03:55.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 13 2024-02-19T21:03:55.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f8018060 osd=6 3 2024-02-19T21:03:55.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 6 13 2024-02-19T21:03:55.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f8018060 osd=6 4 2024-02-19T21:03:55.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f581247d700 10 client.34742.objecter _send_command 13 2024-02-19T21:03:55.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.448+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57f8022cb0 con 0x7f57f801f0e0 2024-02-19T21:03:55.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.449+0000 7f58097fa700 10 client.34742.objecter ms_handle_connect 0x7f57f801f0e0 2024-02-19T21:03:55.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.449+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f57f8022cb0 con 0x7f57f801f0e0 2024-02-19T21:03:55.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.449+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:03:55.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.449+0000 7f58097fa700 10 client.34742.objecter _finish_command 13 = system:0 2024-02-19T21:03:55.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.449+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f8018060 osd=6 4 2024-02-19T21:03:55.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.449+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 6 13 2024-02-19T21:03:55.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.478+0000 7f581247d700 10 client.34742.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:03:55.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.478+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 14 2024-02-19T21:03:55.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.478+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f8018060 osd=6 3 2024-02-19T21:03:55.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.478+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f8018060 osd=6 3 2024-02-19T21:03:55.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.478+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f8018060 osd=6 3 2024-02-19T21:03:55.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.478+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 14 2024-02-19T21:03:55.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.478+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f8018060 osd=6 3 2024-02-19T21:03:55.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.478+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 6 14 2024-02-19T21:03:55.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.478+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f8018060 osd=6 4 2024-02-19T21:03:55.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.478+0000 7f581247d700 10 client.34742.objecter _send_command 14 2024-02-19T21:03:55.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.478+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f57f8022cb0 con 0x7f57f801f0e0 2024-02-19T21:03:55.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.479+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f57f8022cb0 con 0x7f57f801f0e0 2024-02-19T21:03:55.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.479+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:03:55.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.479+0000 7f58097fa700 10 client.34742.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.479+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f8018060 osd=6 4 2024-02-19T21:03:55.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.479+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 6 14 2024-02-19T21:03:55.473 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:03:55.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.480+0000 7f581247d700 10 client.34742.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:03:55.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.480+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 15 2024-02-19T21:03:55.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.480+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f8022cb0 osd=7 3 2024-02-19T21:03:55.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.480+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f8022cb0 osd=7 3 2024-02-19T21:03:55.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.480+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f8022cb0 osd=7 3 2024-02-19T21:03:55.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.480+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 15 2024-02-19T21:03:55.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.480+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f8022cb0 osd=7 3 2024-02-19T21:03:55.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.480+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 7 15 2024-02-19T21:03:55.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.480+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f8022cb0 osd=7 4 2024-02-19T21:03:55.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.480+0000 7f581247d700 10 client.34742.objecter _send_command 15 2024-02-19T21:03:55.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.480+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57f8023ea0 con 0x7f57f8024970 2024-02-19T21:03:55.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.480+0000 7f58097fa700 10 client.34742.objecter ms_handle_connect 0x7f57f8024970 2024-02-19T21:03:55.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.481+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f57f8023ea0 con 0x7f57f8024970 2024-02-19T21:03:55.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.481+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:03:55.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.481+0000 7f58097fa700 10 client.34742.objecter _finish_command 15 = system:0 2024-02-19T21:03:55.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.481+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f8022cb0 osd=7 4 2024-02-19T21:03:55.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.481+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 7 15 2024-02-19T21:03:55.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.509+0000 7f581247d700 10 client.34742.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:03:55.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.509+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign -1 16 2024-02-19T21:03:55.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.509+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f8022cb0 osd=7 3 2024-02-19T21:03:55.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.509+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f8022cb0 osd=7 3 2024-02-19T21:03:55.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.509+0000 7f581247d700 20 client.34742.objecter _get_session s=0x7f57f8022cb0 osd=7 3 2024-02-19T21:03:55.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.509+0000 7f581247d700 15 client.34742.objecter _session_command_op_remove -1 16 2024-02-19T21:03:55.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.509+0000 7f581247d700 20 client.34742.objecter get_session s=0x7f57f8022cb0 osd=7 3 2024-02-19T21:03:55.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.509+0000 7f581247d700 15 client.34742.objecter _session_command_op_assign 7 16 2024-02-19T21:03:55.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.509+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f8022cb0 osd=7 4 2024-02-19T21:03:55.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.509+0000 7f581247d700 10 client.34742.objecter _send_command 16 2024-02-19T21:03:55.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.509+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f57f8023e60 con 0x7f57f8024970 2024-02-19T21:03:55.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.511+0000 7f58097fa700 1 -- 172.21.15.163:0/604313359 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f57f8023e60 con 0x7f57f8024970 2024-02-19T21:03:55.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.511+0000 7f58097fa700 10 client.34742.objecter ms_dispatch 0x7f580c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:03:55.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.511+0000 7f58097fa700 10 client.34742.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.511+0000 7f58097fa700 20 client.34742.objecter put_session s=0x7f57f8022cb0 osd=7 4 2024-02-19T21:03:55.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.511+0000 7f58097fa700 15 client.34742.objecter _session_command_op_remove 7 16 2024-02-19T21:03:55.504 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:03:55.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.511+0000 7f581247d700 10 librados: watch_flush enter 2024-02-19T21:03:55.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.511+0000 7f581247d700 10 librados: watch_flush exit 2024-02-19T21:03:55.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.511+0000 7f581247d700 10 client.34742.objecter close_session for osd.0 2024-02-19T21:03:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.511+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 >> v1:172.21.15.163:6801/1559418115 conn(0x7f57f8004530 legacy=0x7f57f8002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.511+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f80051a0 osd=0 1 2024-02-19T21:03:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.511+0000 7f581247d700 10 client.34742.objecter close_session for osd.1 2024-02-19T21:03:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.511+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 >> v1:172.21.15.163:6805/3125047432 conn(0x7f57f8002090 legacy=0x7f57f800a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f8004260 osd=1 1 2024-02-19T21:03:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 10 client.34742.objecter close_session for osd.2 2024-02-19T21:03:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 >> v1:172.21.15.163:6809/1563158487 conn(0x7f57f800d210 legacy=0x7f57f800d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f80035f0 osd=2 1 2024-02-19T21:03:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 10 client.34742.objecter close_session for osd.3 2024-02-19T21:03:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 >> v1:172.21.15.163:6813/2112279769 conn(0x7f57f8011f30 legacy=0x7f57f8014320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f80118b0 osd=3 1 2024-02-19T21:03:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 10 client.34742.objecter close_session for osd.4 2024-02-19T21:03:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 >> v1:172.21.15.193:6800/3129716965 conn(0x7f57f800cd70 legacy=0x7f57f8015c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:55.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f800df70 osd=4 1 2024-02-19T21:03:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 10 client.34742.objecter close_session for osd.5 2024-02-19T21:03:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 >> v1:172.21.15.193:6804/805161991 conn(0x7f57f8017a70 legacy=0x7f57f801d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f800e120 osd=5 1 2024-02-19T21:03:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 10 client.34742.objecter close_session for osd.6 2024-02-19T21:03:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 >> v1:172.21.15.193:6808/787604615 conn(0x7f57f801f0e0 legacy=0x7f57f80214d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f8018060 osd=6 1 2024-02-19T21:03:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 10 client.34742.objecter close_session for osd.7 2024-02-19T21:03:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 >> v1:172.21.15.193:6812/157927432 conn(0x7f57f8024970 legacy=0x7f57f8024df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f5810a1a700 1 -- 172.21.15.163:0/604313359 reap_dead start 2024-02-19T21:03:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 20 client.34742.objecter put_session s=0x7f57f8022cb0 osd=7 1 2024-02-19T21:03:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 20 client.34742.objecter shutdown clearing up homeless session... 2024-02-19T21:03:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 10 client.34742.objecter successfully canceled tick 2024-02-19T21:03:55.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f5810a1a700 1 -- 172.21.15.163:0/604313359 reap_dead start 2024-02-19T21:03:55.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f5810a1a700 1 -- 172.21.15.163:0/604313359 reap_dead start 2024-02-19T21:03:55.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f5810a1a700 1 -- 172.21.15.163:0/604313359 reap_dead start 2024-02-19T21:03:55.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f5810a1a700 1 -- 172.21.15.163:0/604313359 reap_dead start 2024-02-19T21:03:55.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 >> v1:172.21.15.163:6800/1552514681 conn(0x7f57f4068060 legacy=0x7f57f406a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:55.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.512+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 >> v1:172.21.15.163:6790/0 conn(0x7f580c0fd980 legacy=0x7f580c102580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:55.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.513+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 shutdown_connections 2024-02-19T21:03:55.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.513+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 >> 172.21.15.163:0/604313359 conn(0x7f580c0f35a0 msgr2=0x7f580c107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:55.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.513+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 shutdown_connections 2024-02-19T21:03:55.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.513+0000 7f581247d700 1 -- 172.21.15.163:0/604313359 wait complete. 2024-02-19T21:03:55.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:55.513+0000 7f581247d700 1 librados: shutdown 2024-02-19T21:03:55.509 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.509 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.509 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.509 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.509 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.509 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.509 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.509 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:03:55.518 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:03:55.558 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:55.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:55.284633+0000 mon.c (mon.2) 29 : audit [DBG] from='client.? v1:172.21.15.163:0/604313359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:03:55.819 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:03:55.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:55.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:55.284633+0000 mon.c (mon.2) 29 : audit [DBG] from='client.? v1:172.21.15.163:0/604313359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:03:55.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:03:55.827 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:55.284633+0000 mon.c (mon.2) 29 : audit [DBG] from='client.? v1:172.21.15.163:0/604313359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:03:55.928 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:03:55.961 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:56.244 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:03:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:56.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:03:56.322 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:03:56.352 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:03:56.384 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:56.406 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:03:56.408 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:56.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:03:56.509 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:03:56.539 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:56.560 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:03:56.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:56.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:03:56.610 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:56.663 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:03:56.696 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:56.714 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:03:56.714 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:03:56.714 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-02-19T21:03:56.714 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-02-19T21:03:56.714 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:03:56.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:54.943012+0000 mgr.y (mgr.14550) 109 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:56.990 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:54.943012+0000 mgr.y (mgr.14550) 109 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:56.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:54.943012+0000 mgr.y (mgr.14550) 109 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:57.550 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:03:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:57.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:03:57.659 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:03:57.691 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:57.712 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:03:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:57.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:03:57.816 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:03:57.852 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:58.063 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:03:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:03:58.121 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-19T21:03:58.396 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:03:58.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:03:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:56.943370+0000 mgr.y (mgr.14550) 110 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:58.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:03:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:56.943370+0000 mgr.y (mgr.14550) 110 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:58.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:03:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:56.943370+0000 mgr.y (mgr.14550) 110 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:03:59.331 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:03:59.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:59.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:03:59.390 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:03:59.437 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:03:59.469 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:59.491 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:03:59.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:59.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:03:59.595 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:03:59.625 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:59.645 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:03:59.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:03:59.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:03:59.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.697+0000 7f8f9c1bc700 1 -- 172.21.15.163:0/1355567393 >> v1:172.21.15.163:6789/0 conn(0x7f8f940f8980 legacy=0x7f8f940f8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:59.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.698+0000 7f8f9c1bc700 1 -- 172.21.15.163:0/1355567393 shutdown_connections 2024-02-19T21:03:59.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.698+0000 7f8f9c1bc700 1 -- 172.21.15.163:0/1355567393 >> 172.21.15.163:0/1355567393 conn(0x7f8f940f2450 msgr2=0x7f8f940f4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:59.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.698+0000 7f8f9c1bc700 1 -- 172.21.15.163:0/1355567393 shutdown_connections 2024-02-19T21:03:59.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.698+0000 7f8f9c1bc700 1 -- 172.21.15.163:0/1355567393 wait complete. 2024-02-19T21:03:59.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.698+0000 7f8f9c1bc700 1 librados: starting msgr at 2024-02-19T21:03:59.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.698+0000 7f8f9c1bc700 1 librados: starting objecter 2024-02-19T21:03:59.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.698+0000 7f8f9c1bc700 1 Processor -- start 2024-02-19T21:03:59.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.698+0000 7f8f9c1bc700 1 -- start start 2024-02-19T21:03:59.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.698+0000 7f8f9c1bc700 1 librados: setting wanted keys 2024-02-19T21:03:59.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.698+0000 7f8f9c1bc700 1 librados: calling monclient init 2024-02-19T21:03:59.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f9c1bc700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8f941173a0 con 0x7f8f94104280 2024-02-19T21:03:59.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f9c1bc700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8f94118580 con 0x7f8f940f9470 2024-02-19T21:03:59.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f9c1bc700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8f94119780 con 0x7f8f940f8980 2024-02-19T21:03:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f9a759700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8f94104280 0x7f8f94115a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57738/0 (socket says 172.21.15.163:57738) 2024-02-19T21:03:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f9a759700 1 -- 172.21.15.163:0/1026562181 learned_addr learned my addr 172.21.15.163:0/1026562181 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:03:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f8affd700 10 client.?.objecter ms_handle_connect 0x7f8f940f9470 2024-02-19T21:03:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f8affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f8affd700 10 client.?.objecter ms_handle_connect 0x7f8f94104280 2024-02-19T21:03:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f8affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f8affd700 10 client.?.objecter ms_handle_connect 0x7f8f940f8980 2024-02-19T21:03:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f8affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:03:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596045989 0 0) 0x7f8f94118580 con 0x7f8f940f9470 2024-02-19T21:03:59.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f80003030 con 0x7f8f940f9470 2024-02-19T21:03:59.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970739996 0 0) 0x7f8f941173a0 con 0x7f8f94104280 2024-02-19T21:03:59.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f94118580 con 0x7f8f94104280 2024-02-19T21:03:59.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429403651 0 0) 0x7f8f94119780 con 0x7f8f940f8980 2024-02-19T21:03:59.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.699+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f941173a0 con 0x7f8f940f8980 2024-02-19T21:03:59.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2251012489 0 0) 0x7f8f80003030 con 0x7f8f940f9470 2024-02-19T21:03:59.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f94119780 con 0x7f8f940f9470 2024-02-19T21:03:59.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 968807786 0 0) 0x7f8f941173a0 con 0x7f8f940f8980 2024-02-19T21:03:59.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f80003030 con 0x7f8f940f8980 2024-02-19T21:03:59.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8f840028f0 con 0x7f8f940f9470 2024-02-19T21:03:59.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8f90001990 con 0x7f8f940f8980 2024-02-19T21:03:59.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1345665065 0 0) 0x7f8f94118580 con 0x7f8f94104280 2024-02-19T21:03:59.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f941173a0 con 0x7f8f94104280 2024-02-19T21:03:59.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8f8c001be0 con 0x7f8f94104280 2024-02-19T21:03:59.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4032836507 0 0) 0x7f8f94119780 con 0x7f8f940f9470 2024-02-19T21:03:59.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 >> v1:172.21.15.163:6790/0 conn(0x7f8f940f8980 legacy=0x7f8f940ff730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:59.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 >> v1:172.21.15.163:6789/0 conn(0x7f8f94104280 legacy=0x7f8f94115a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:59.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9411a980 con 0x7f8f940f9470 2024-02-19T21:03:59.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f9c1bc700 1 -- 172.21.15.163:0/1026562181 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f94117590 con 0x7f8f940f9470 2024-02-19T21:03:59.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f9c1bc700 10 client.24748.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:03:59.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f9c1bc700 1 -- 172.21.15.163:0/1026562181 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f94117a00 con 0x7f8f940f9470 2024-02-19T21:03:59.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f9c1bc700 1 librados: init done 2024-02-19T21:03:59.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8f84002a50 con 0x7f8f940f9470 2024-02-19T21:03:59.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.700+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8f840032b0 con 0x7f8f940f9470 2024-02-19T21:03:59.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.701+0000 7f8f9c1bc700 1 -- 172.21.15.163:0/1026562181 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f78004f20 con 0x7f8f940f9470 2024-02-19T21:03:59.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.701+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f8f84001330 con 0x7f8f940f9470 2024-02-19T21:03:59.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.702+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(89..89 src has 32..89) v4 ==== 6172+0+0 (unknown 1681062987 0 0) 0x7f8f84075ce0 con 0x7f8f940f9470 2024-02-19T21:03:59.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.702+0000 7f8f8affd700 10 client.24748.objecter ms_dispatch 0x7f8f94001550 osd_map(89..89 src has 32..89) v4 2024-02-19T21:03:59.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.702+0000 7f8f8affd700 3 client.24748.objecter handle_osd_map got epochs [89,89] > 0 2024-02-19T21:03:59.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.702+0000 7f8f8affd700 3 client.24748.objecter handle_osd_map decoding full epoch 89 2024-02-19T21:03:59.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.702+0000 7f8f8affd700 20 client.24748.objecter dump_active .. 0 homeless 2024-02-19T21:03:59.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.702+0000 7f8f8affd700 10 client.24748.objecter ms_handle_connect 0x7f8f80068250 2024-02-19T21:03:59.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.706+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8f84049890 con 0x7f8f940f9470 2024-02-19T21:03:59.749 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:03:59.786 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:03:59.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.957+0000 7f8f9c1bc700 1 -- 172.21.15.163:0/1026562181 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8f78000b90 con 0x7f8f80068250 2024-02-19T21:03:59.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.976+0000 7f8f8affd700 1 -- 172.21.15.163:0/1026562181 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357181 (unknown 2965378022 0 1756019927) 0x7f8f78000b90 con 0x7f8f80068250 2024-02-19T21:03:59.972 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:03:59.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.980+0000 7f8f9c1bc700 10 librados: watch_flush enter 2024-02-19T21:03:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.980+0000 7f8f9c1bc700 10 librados: watch_flush exit 2024-02-19T21:03:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.980+0000 7f8f9c1bc700 20 client.24748.objecter shutdown clearing up homeless session... 2024-02-19T21:03:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.980+0000 7f8f9c1bc700 10 client.24748.objecter successfully canceled tick 2024-02-19T21:03:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.980+0000 7f8f9c1bc700 1 -- 172.21.15.163:0/1026562181 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8f80068250 legacy=0x7f8f8006a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.980+0000 7f8f9c1bc700 1 -- 172.21.15.163:0/1026562181 >> v1:172.21.15.193:6789/0 conn(0x7f8f940f9470 legacy=0x7f8f940ffe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:03:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.980+0000 7f8f9c1bc700 1 -- 172.21.15.163:0/1026562181 shutdown_connections 2024-02-19T21:03:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.980+0000 7f8f9c1bc700 1 -- 172.21.15.163:0/1026562181 >> 172.21.15.163:0/1026562181 conn(0x7f8f940f2450 msgr2=0x7f8f941076f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:03:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.980+0000 7f8f9c1bc700 1 -- 172.21.15.163:0/1026562181 shutdown_connections 2024-02-19T21:03:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.980+0000 7f8f9c1bc700 1 -- 172.21.15.163:0/1026562181 wait complete. 2024-02-19T21:03:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:03:59.980+0000 7f8f9c1bc700 1 librados: shutdown 2024-02-19T21:03:59.976 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:04:00.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:04:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:04:00.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:03:58.943761+0000 mgr.y (mgr.14550) 111 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:04:00.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:03:58.943761+0000 mgr.y (mgr.14550) 111 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:04:00.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:03:58.943761+0000 mgr.y (mgr.14550) 111 : cluster [DBG] pgmap v90: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:04:00.821 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:01.343 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:01.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:01.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:01.452 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:01.484 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:01.506 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:01.509 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:01.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:01.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:03:59.957559+0000 mgr.y (mgr.14550) 112 : audit [DBG] from='client.24748 v1:172.21.15.163:0/1026562181' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:04:01.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:00.046168+0000 mgr.y (mgr.14550) 113 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:01.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:03:59.957559+0000 mgr.y (mgr.14550) 112 : audit [DBG] from='client.24748 v1:172.21.15.163:0/1026562181' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:04:01.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:00.046168+0000 mgr.y (mgr.14550) 113 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:01.610 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:01.647 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:01.799 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:01.809 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:01.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:01.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:03:59.957559+0000 mgr.y (mgr.14550) 112 : audit [DBG] from='client.24748 v1:172.21.15.163:0/1026562181' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:04:01.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:00.046168+0000 mgr.y (mgr.14550) 113 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:01.911 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:01.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.734+0000 7fde24669700 1 -- 172.21.15.163:0/4562967 >> v1:172.21.15.193:6789/0 conn(0x7fde1c0faff0 legacy=0x7fde1c10cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:01.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.735+0000 7fde24669700 1 -- 172.21.15.163:0/4562967 shutdown_connections 2024-02-19T21:04:01.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.735+0000 7fde24669700 1 -- 172.21.15.163:0/4562967 >> 172.21.15.163:0/4562967 conn(0x7fde1c06fdc0 msgr2=0x7fde1c0f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:01.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.735+0000 7fde24669700 1 -- 172.21.15.163:0/4562967 shutdown_connections 2024-02-19T21:04:01.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.735+0000 7fde24669700 1 -- 172.21.15.163:0/4562967 wait complete. 2024-02-19T21:04:01.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.735+0000 7fde24669700 1 librados: starting msgr at 2024-02-19T21:04:01.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.735+0000 7fde24669700 1 librados: starting objecter 2024-02-19T21:04:01.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde24669700 1 Processor -- start 2024-02-19T21:04:01.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde24669700 1 -- start start 2024-02-19T21:04:01.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde24669700 1 librados: setting wanted keys 2024-02-19T21:04:01.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde24669700 1 librados: calling monclient init 2024-02-19T21:04:01.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde24669700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fde1c11d9d0 con 0x7fde1c0fc780 2024-02-19T21:04:01.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde24669700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fde1c11ebb0 con 0x7fde1c0fa430 2024-02-19T21:04:01.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde24669700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fde1c11fdb0 con 0x7fde1c0faff0 2024-02-19T21:04:01.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde21c04700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fde1c0faff0 0x7fde1c10a1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:58794/0 (socket says 172.21.15.163:58794) 2024-02-19T21:04:01.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde21c04700 1 -- 172.21.15.163:0/799236926 learned_addr learned my addr 172.21.15.163:0/799236926 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:04:01.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde137fe700 10 client.?.objecter ms_handle_connect 0x7fde1c0faff0 2024-02-19T21:04:01.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde137fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:01.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde137fe700 10 client.?.objecter ms_handle_connect 0x7fde1c0fc780 2024-02-19T21:04:01.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde137fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:01.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde137fe700 10 client.?.objecter ms_handle_connect 0x7fde1c0fa430 2024-02-19T21:04:01.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde137fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:01.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.736+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839701649 0 0) 0x7fde1c11d9d0 con 0x7fde1c0fc780 2024-02-19T21:04:01.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde080033f0 con 0x7fde1c0fc780 2024-02-19T21:04:01.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533370111 0 0) 0x7fde1c11ebb0 con 0x7fde1c0fa430 2024-02-19T21:04:01.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde1c11d9d0 con 0x7fde1c0fa430 2024-02-19T21:04:01.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885659553 0 0) 0x7fde1c11fdb0 con 0x7fde1c0faff0 2024-02-19T21:04:01.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde1c11ebb0 con 0x7fde1c0faff0 2024-02-19T21:04:01.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 252824952 0 0) 0x7fde080033f0 con 0x7fde1c0fc780 2024-02-19T21:04:01.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde1c11fdb0 con 0x7fde1c0fc780 2024-02-19T21:04:01.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 521560711 0 0) 0x7fde1c11d9d0 con 0x7fde1c0fa430 2024-02-19T21:04:01.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde080033f0 con 0x7fde1c0fa430 2024-02-19T21:04:01.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fde14002240 con 0x7fde1c0fc780 2024-02-19T21:04:01.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fde18001840 con 0x7fde1c0fa430 2024-02-19T21:04:01.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3042629790 0 0) 0x7fde1c11ebb0 con 0x7fde1c0faff0 2024-02-19T21:04:01.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde1c11d9d0 con 0x7fde1c0faff0 2024-02-19T21:04:01.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fde0c0029d0 con 0x7fde1c0faff0 2024-02-19T21:04:01.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1235569047 0 0) 0x7fde1c11fdb0 con 0x7fde1c0fc780 2024-02-19T21:04:01.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 >> v1:172.21.15.163:6790/0 conn(0x7fde1c0faff0 legacy=0x7fde1c10a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:01.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 >> v1:172.21.15.193:6789/0 conn(0x7fde1c0fa430 legacy=0x7fde1c109ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:01.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c120fb0 con 0x7fde1c0fc780 2024-02-19T21:04:01.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde24669700 1 -- 172.21.15.163:0/799236926 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1c11dbc0 con 0x7fde1c0fc780 2024-02-19T21:04:01.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde24669700 10 client.14832.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:01.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde24669700 1 -- 172.21.15.163:0/799236926 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1c11e0b0 con 0x7fde1c0fc780 2024-02-19T21:04:01.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.737+0000 7fde24669700 1 librados: init done 2024-02-19T21:04:01.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.739+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fde140015b0 con 0x7fde1c0fc780 2024-02-19T21:04:01.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.739+0000 7fde24669700 1 -- 172.21.15.163:0/799236926 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1c05f700 con 0x7fde1c0fc780 2024-02-19T21:04:01.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.739+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fde140039d0 con 0x7fde1c0fc780 2024-02-19T21:04:01.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.741+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fde14001c00 con 0x7fde1c0fc780 2024-02-19T21:04:01.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.742+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(89..89 src has 32..89) v4 ==== 6172+0+0 (unknown 1681062987 0 0) 0x7fde14076280 con 0x7fde1c0fc780 2024-02-19T21:04:01.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.742+0000 7fde137fe700 10 client.14832.objecter ms_dispatch 0x7fde1c001550 osd_map(89..89 src has 32..89) v4 2024-02-19T21:04:01.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.742+0000 7fde137fe700 3 client.14832.objecter handle_osd_map got epochs [89,89] > 0 2024-02-19T21:04:01.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.742+0000 7fde137fe700 3 client.14832.objecter handle_osd_map decoding full epoch 89 2024-02-19T21:04:01.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.742+0000 7fde137fe700 20 client.14832.objecter dump_active .. 0 homeless 2024-02-19T21:04:01.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.742+0000 7fde137fe700 10 client.14832.objecter ms_handle_connect 0x7fde08063c70 2024-02-19T21:04:01.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:01.748+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fde14049ef0 con 0x7fde1c0fc780 2024-02-19T21:04:01.943 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:01.964 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:01.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:02.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:02.059+0000 7fde24669700 1 -- 172.21.15.163:0/799236926 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fde1c10f2f0 con 0x7fde1c0fc780 2024-02-19T21:04:02.067 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:02.097 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:02.116 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:02.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:02.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:02.221 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:02.255 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:02.328 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:02.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:02.432 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:04:02.463 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:02.483 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:04:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:02.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:04:02.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:02.580+0000 7fde137fe700 1 -- 172.21.15.163:0/799236926 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v90) v1 ==== 91+0+0 (unknown 1353147662 0 0) 0x7fde14048360 con 0x7fde1c0fc780 2024-02-19T21:04:02.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:02.583+0000 7fde24669700 10 librados: watch_flush enter 2024-02-19T21:04:02.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:02.583+0000 7fde24669700 10 librados: watch_flush exit 2024-02-19T21:04:02.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:02.583+0000 7fde24669700 20 client.14832.objecter shutdown clearing up homeless session... 2024-02-19T21:04:02.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:02.583+0000 7fde24669700 10 client.14832.objecter successfully canceled tick 2024-02-19T21:04:02.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:02.583+0000 7fde24669700 1 -- 172.21.15.163:0/799236926 >> v1:172.21.15.163:6800/1552514681 conn(0x7fde08063c70 legacy=0x7fde08066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:02.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:02.583+0000 7fde24669700 1 -- 172.21.15.163:0/799236926 >> v1:172.21.15.163:6789/0 conn(0x7fde1c0fc780 legacy=0x7fde1c11c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:02.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:02.585+0000 7fde24669700 1 -- 172.21.15.163:0/799236926 shutdown_connections 2024-02-19T21:04:02.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:02.585+0000 7fde24669700 1 -- 172.21.15.163:0/799236926 >> 172.21.15.163:0/799236926 conn(0x7fde1c06fdc0 msgr2=0x7fde1c10dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:02.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:02.585+0000 7fde24669700 1 -- 172.21.15.163:0/799236926 shutdown_connections 2024-02-19T21:04:02.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:02.585+0000 7fde24669700 1 -- 172.21.15.163:0/799236926 wait complete. 2024-02-19T21:04:02.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:02.585+0000 7fde24669700 1 librados: shutdown 2024-02-19T21:04:02.580 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:04:02.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:00.944484+0000 mgr.y (mgr.14550) 114 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:04:02.582 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:02.059687+0000 mon.a (mon.0) 727 : audit [INF] from='client.? v1:172.21.15.163:0/799236926' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:04:02.582 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:00.944484+0000 mgr.y (mgr.14550) 114 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:04:02.582 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:02.059687+0000 mon.a (mon.0) 727 : audit [INF] from='client.? v1:172.21.15.163:0/799236926' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:04:02.586 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:02.616 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:02.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:00.944484+0000 mgr.y (mgr.14550) 114 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:04:02.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:02.059687+0000 mon.a (mon.0) 727 : audit [INF] from='client.? v1:172.21.15.163:0/799236926' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:04:02.874 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:02.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:02.981 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:04:03.010 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:03.068 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:04:03.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:03.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:04:03.173 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:03.205 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:03.226 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:03.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:03.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:03.330 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:03.359 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:03.378 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:03.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:03.482 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:03.510 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:03.529 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:03.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:03.593 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:04:03] "GET /metrics HTTP/1.1" 200 172242 "" "Prometheus/2.33.4" 2024-02-19T21:04:03.594 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:02.575553+0000 mon.a (mon.0) 728 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:04:03.594 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:02.579681+0000 mon.a (mon.0) 729 : audit [INF] from='client.? v1:172.21.15.163:0/799236926' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:04:03.594 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:02.579739+0000 mon.a (mon.0) 730 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-02-19T21:04:03.594 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:02.780387+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:04:03.594 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:02.781550+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:04:03.594 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:03.077306+0000 osd.5 (osd.5) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:03.594 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:03.077384+0000 osd.5 (osd.5) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:03.594 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:03.150723+0000 osd.4 (osd.4) 51 : cluster [DBG] 4.d scrub starts 2024-02-19T21:04:03.594 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:03.152811+0000 osd.4 (osd.4) 52 : cluster [DBG] 4.d scrub ok 2024-02-19T21:04:03.633 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:03.659 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:03.678 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:03.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:03.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:03.740 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-02-19T21:03:58.943533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":871,"num_read_kb":623,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":481,"ondisk_log_size":481,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2382056,"kb_used_data":5288,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579160,"statfs":{"total":767960285184,"available":765521059840,"internally_reserved":0,"allocated":5414912,"data_stored":2510657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002399"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.514655+0000","last_change":"2024-02-19T21:03:27.633572+0000","last_active":"2024-02-19T21:03:42.514655+0000","last_peered":"2024-02-19T21:03:42.514655+0000","last_clean":"2024-02-19T21:03:42.514655+0000","last_became_active":"2024-02-19T21:01:22.609059+0000","last_became_peered":"2024-02-19T21:01:22.609059+0000","last_unstale":"2024-02-19T21:03:42.514655+0000","last_undegraded":"2024-02-19T21:03:42.514655+0000","last_fullsized":"2024-02-19T21:03:42.514655+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:27.633492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:27.633492+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.507997+0000","last_change":"2024-02-19T21:03:05.073746+0000","last_active":"2024-02-19T21:03:42.507997+0000","last_peered":"2024-02-19T21:03:42.507997+0000","last_clean":"2024-02-19T21:03:42.507997+0000","last_became_active":"2024-02-19T21:01:20.289208+0000","last_became_peered":"2024-02-19T21:01:20.289208+0000","last_unstale":"2024-02-19T21:03:42.507997+0000","last_undegraded":"2024-02-19T21:03:42.507997+0000","last_fullsized":"2024-02-19T21:03:42.507997+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.073664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.073664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.528790+0000","last_change":"2024-02-19T21:03:04.644754+0000","last_active":"2024-02-19T21:03:42.528790+0000","last_peered":"2024-02-19T21:03:42.528790+0000","last_clean":"2024-02-19T21:03:42.528790+0000","last_became_active":"2024-02-19T21:01:16.278753+0000","last_became_peered":"2024-02-19T21:01:16.278753+0000","last_unstale":"2024-02-19T21:03:42.528790+0000","last_undegraded":"2024-02-19T21:03:42.528790+0000","last_fullsized":"2024-02-19T21:03:42.528790+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.644689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.644689+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.513284+0000","last_change":"2024-02-19T21:02:48.863931+0000","last_active":"2024-02-19T21:03:42.513284+0000","last_peered":"2024-02-19T21:03:42.513284+0000","last_clean":"2024-02-19T21:03:42.513284+0000","last_became_active":"2024-02-19T21:01:18.283222+0000","last_became_peered":"2024-02-19T21:01:18.283222+0000","last_unstale":"2024-02-19T21:03:42.513284+0000","last_undegraded":"2024-02-19T21:03:42.513284+0000","last_fullsized":"2024-02-19T21:03:42.513284+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:48.863852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.863852+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508222+0000","last_change":"2024-02-19T21:02:23.100006+0000","last_active":"2024-02-19T21:03:42.508222+0000","last_peered":"2024-02-19T21:03:42.508222+0000","last_clean":"2024-02-19T21:03:42.508222+0000","last_became_active":"2024-02-19T21:01:22.293432+0000","last_became_peered":"2024-02-19T21:01:22.293432+0000","last_unstale":"2024-02-19T21:03:42.508222+0000","last_undegraded":"2024-02-19T21:03:42.508222+0000","last_fullsized":"2024-02-19T21:03:42.508222+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:23.099932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:23.099932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.512690+0000","last_change":"2024-02-19T21:02:29.480341+0000","last_active":"2024-02-19T21:03:42.512690+0000","last_peered":"2024-02-19T21:03:42.512690+0000","last_clean":"2024-02-19T21:03:42.512690+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:03:42.512690+0000","last_undegraded":"2024-02-19T21:03:42.512690+0000","last_fullsized":"2024-02-19T21:03:42.512690+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:29.480275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:29.480275+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.517010+0000","last_change":"2024-02-19T21:02:20.970293+0000","last_active":"2024-02-19T21:03:42.517010+0000","last_peered":"2024-02-19T21:03:42.517010+0000","last_clean":"2024-02-19T21:03:42.517010+0000","last_became_active":"2024-02-19T21:01:16.272132+0000","last_became_peered":"2024-02-19T21:01:16.272132+0000","last_unstale":"2024-02-19T21:03:42.517010+0000","last_undegraded":"2024-02-19T21:03:42.517010+0000","last_fullsized":"2024-02-19T21:03:42.517010+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:20.970216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:20.970216+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508222+0000","last_change":"2024-02-19T21:03:16.047345+0000","last_active":"2024-02-19T21:03:42.508222+0000","last_peered":"2024-02-19T21:03:42.508222+0000","last_clean":"2024-02-19T21:03:42.508222+0000","last_became_active":"2024-02-19T21:01:18.287645+0000","last_became_peered":"2024-02-19T21:01:18.287645+0000","last_unstale":"2024-02-19T21:03:42.508222+0000","last_undegraded":"2024-02-19T21:03:42.508222+0000","last_fullsized":"2024-02-19T21:03:42.508222+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:16.047258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:16.047258+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523389+0000","last_change":"2024-02-19T21:03:13.096737+0000","last_active":"2024-02-19T21:03:42.523389+0000","last_peered":"2024-02-19T21:03:42.523389+0000","last_clean":"2024-02-19T21:03:42.523389+0000","last_became_active":"2024-02-19T21:01:22.293282+0000","last_became_peered":"2024-02-19T21:01:22.293282+0000","last_unstale":"2024-02-19T21:03:42.523389+0000","last_undegraded":"2024-02-19T21:03:42.523389+0000","last_fullsized":"2024-02-19T21:03:42.523389+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:13.096667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:13.096667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523927+0000","last_change":"2024-02-19T21:02:35.546613+0000","last_active":"2024-02-19T21:03:42.523927+0000","last_peered":"2024-02-19T21:03:42.523927+0000","last_clean":"2024-02-19T21:03:42.523927+0000","last_became_active":"2024-02-19T21:01:20.287940+0000","last_became_peered":"2024-02-19T21:01:20.287940+0000","last_unstale":"2024-02-19T21:03:42.523927+0000","last_undegraded":"2024-02-19T21:03:42.523927+0000","last_fullsized":"2024-02-19T21:03:42.523927+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:35.546526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:35.546526+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524193+0000","last_change":"2024-02-19T21:03:26.793343+0000","last_active":"2024-02-19T21:03:42.524193+0000","last_peered":"2024-02-19T21:03:42.524193+0000","last_clean":"2024-02-19T21:03:42.524193+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:03:42.524193+0000","last_undegraded":"2024-02-19T21:03:42.524193+0000","last_fullsized":"2024-02-19T21:03:42.524193+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:26.793263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:26.793263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508421+0000","last_change":"2024-02-19T21:02:27.098930+0000","last_active":"2024-02-19T21:03:42.508421+0000","last_peered":"2024-02-19T21:03:42.508421+0000","last_clean":"2024-02-19T21:03:42.508421+0000","last_became_active":"2024-02-19T21:01:18.285735+0000","last_became_peered":"2024-02-19T21:01:18.285735+0000","last_unstale":"2024-02-19T21:03:42.508421+0000","last_undegraded":"2024-02-19T21:03:42.508421+0000","last_fullsized":"2024-02-19T21:03:42.508421+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:02:27.098844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:27.098844+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.527574+0000","last_change":"2024-02-19T21:03:00.570766+0000","last_active":"2024-02-19T21:03:42.527574+0000","last_peered":"2024-02-19T21:03:42.527574+0000","last_clean":"2024-02-19T21:03:42.527574+0000","last_became_active":"2024-02-19T21:01:22.292714+0000","last_became_peered":"2024-02-19T21:01:22.292714+0000","last_unstale":"2024-02-19T21:03:42.527574+0000","last_undegraded":"2024-02-19T21:03:42.527574+0000","last_fullsized":"2024-02-19T21:03:42.527574+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:00.570677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:00.570677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524478+0000","last_change":"2024-02-19T21:03:26.922649+0000","last_active":"2024-02-19T21:03:42.524478+0000","last_peered":"2024-02-19T21:03:42.524478+0000","last_clean":"2024-02-19T21:03:42.524478+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:03:42.524478+0000","last_undegraded":"2024-02-19T21:03:42.524478+0000","last_fullsized":"2024-02-19T21:03:42.524478+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:26.922572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:26.922572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524402+0000","last_change":"2024-02-19T21:02:57.552979+0000","last_active":"2024-02-19T21:03:42.524402+0000","last_peered":"2024-02-19T21:03:42.524402+0000","last_clean":"2024-02-19T21:03:42.524402+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:03:42.524402+0000","last_undegraded":"2024-02-19T21:03:42.524402+0000","last_fullsized":"2024-02-19T21:03:42.524402+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.552886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.552886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508325+0000","last_change":"2024-02-19T21:02:26.073713+0000","last_active":"2024-02-19T21:03:42.508325+0000","last_peered":"2024-02-19T21:03:42.508325+0000","last_clean":"2024-02-19T21:03:42.508325+0000","last_became_active":"2024-02-19T21:01:18.285269+0000","last_became_peered":"2024-02-19T21:01:18.285269+0000","last_unstale":"2024-02-19T21:03:42.508325+0000","last_undegraded":"2024-02-19T21:03:42.508325+0000","last_fullsized":"2024-02-19T21:03:42.508325+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:26.073625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:26.073625+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.516372+0000","last_change":"2024-02-19T21:02:24.948910+0000","last_active":"2024-02-19T21:03:42.516372+0000","last_peered":"2024-02-19T21:03:42.516372+0000","last_clean":"2024-02-19T21:03:42.516372+0000","last_became_active":"2024-02-19T21:01:22.607099+0000","last_became_peered":"2024-02-19T21:01:22.607099+0000","last_unstale":"2024-02-19T21:03:42.516372+0000","last_undegraded":"2024-02-19T21:03:42.516372+0000","last_fullsized":"2024-02-19T21:03:42.516372+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:24.948832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:24.948832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508556+0000","last_change":"2024-02-19T21:03:09.010420+0000","last_active":"2024-02-19T21:03:42.508556+0000","last_peered":"2024-02-19T21:03:42.508556+0000","last_clean":"2024-02-19T21:03:42.508556+0000","last_became_active":"2024-02-19T21:01:20.291672+0000","last_became_peered":"2024-02-19T21:01:20.291672+0000","last_unstale":"2024-02-19T21:03:42.508556+0000","last_undegraded":"2024-02-19T21:03:42.508556+0000","last_fullsized":"2024-02-19T21:03:42.508556+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:09.010341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:09.010341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.515067+0000","last_change":"2024-02-19T21:03:20.637528+0000","last_active":"2024-02-19T21:03:42.515067+0000","last_peered":"2024-02-19T21:03:42.515067+0000","last_clean":"2024-02-19T21:03:42.515067+0000","last_became_active":"2024-02-19T21:01:16.274811+0000","last_became_peered":"2024-02-19T21:01:16.274811+0000","last_unstale":"2024-02-19T21:03:42.515067+0000","last_undegraded":"2024-02-19T21:03:42.515067+0000","last_fullsized":"2024-02-19T21:03:42.515067+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:20.637465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:20.637465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.516124+0000","last_change":"2024-02-19T21:02:57.819231+0000","last_active":"2024-02-19T21:03:42.516124+0000","last_peered":"2024-02-19T21:03:42.516124+0000","last_clean":"2024-02-19T21:03:42.516124+0000","last_became_active":"2024-02-19T21:01:18.285926+0000","last_became_peered":"2024-02-19T21:01:18.285926+0000","last_unstale":"2024-02-19T21:03:42.516124+0000","last_undegraded":"2024-02-19T21:03:42.516124+0000","last_fullsized":"2024-02-19T21:03:42.516124+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:02:57.819125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.819125+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508909+0000","last_change":"2024-02-19T21:03:12.081314+0000","last_active":"2024-02-19T21:03:42.508909+0000","last_peered":"2024-02-19T21:03:42.508909+0000","last_clean":"2024-02-19T21:03:42.508909+0000","last_became_active":"2024-02-19T21:01:22.607435+0000","last_became_peered":"2024-02-19T21:01:22.607435+0000","last_unstale":"2024-02-19T21:03:42.508909+0000","last_undegraded":"2024-02-19T21:03:42.508909+0000","last_fullsized":"2024-02-19T21:03:42.508909+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:03:12.081227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.513684+0000","last_change":"2024-02-19T21:02:54.544941+0000","last_active":"2024-02-19T21:03:42.513684+0000","last_peered":"2024-02-19T21:03:42.513684+0000","last_clean":"2024-02-19T21:03:42.513684+0000","last_became_active":"2024-02-19T21:01:20.290705+0000","last_became_peered":"2024-02-19T21:01:20.290705+0000","last_unstale":"2024-02-19T21:03:42.513684+0000","last_undegraded":"2024-02-19T21:03:42.513684+0000","last_fullsized":"2024-02-19T21:03:42.513684+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:54.544834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:54.544834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.525684+0000","last_change":"2024-02-19T21:02:47.805245+0000","last_active":"2024-02-19T21:03:42.525684+0000","last_peered":"2024-02-19T21:03:42.525684+0000","last_clean":"2024-02-19T21:03:42.525684+0000","last_became_active":"2024-02-19T21:01:16.274524+0000","last_became_peered":"2024-02-19T21:01:16.274524+0000","last_unstale":"2024-02-19T21:03:42.525684+0000","last_undegraded":"2024-02-19T21:03:42.525684+0000","last_fullsized":"2024-02-19T21:03:42.525684+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.805171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.805171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524222+0000","last_change":"2024-02-19T21:02:47.238547+0000","last_active":"2024-02-19T21:03:42.524222+0000","last_peered":"2024-02-19T21:03:42.524222+0000","last_clean":"2024-02-19T21:03:42.524222+0000","last_became_active":"2024-02-19T21:01:18.289712+0000","last_became_peered":"2024-02-19T21:01:18.289712+0000","last_unstale":"2024-02-19T21:03:42.524222+0000","last_undegraded":"2024-02-19T21:03:42.524222+0000","last_fullsized":"2024-02-19T21:03:42.524222+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:47.238470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.238470+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.513294+0000","last_change":"2024-02-19T21:03:06.456310+0000","last_active":"2024-02-19T21:03:42.513294+0000","last_peered":"2024-02-19T21:03:42.513294+0000","last_clean":"2024-02-19T21:03:42.513294+0000","last_became_active":"2024-02-19T21:01:22.293230+0000","last_became_peered":"2024-02-19T21:01:22.293230+0000","last_unstale":"2024-02-19T21:03:42.513294+0000","last_undegraded":"2024-02-19T21:03:42.513294+0000","last_fullsized":"2024-02-19T21:03:42.513294+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.456212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.456212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.528763+0000","last_change":"2024-02-19T21:03:08.636037+0000","last_active":"2024-02-19T21:03:42.528763+0000","last_peered":"2024-02-19T21:03:42.528763+0000","last_clean":"2024-02-19T21:03:42.528763+0000","last_became_active":"2024-02-19T21:01:20.288016+0000","last_became_peered":"2024-02-19T21:01:20.288016+0000","last_unstale":"2024-02-19T21:03:42.528763+0000","last_undegraded":"2024-02-19T21:03:42.528763+0000","last_fullsized":"2024-02-19T21:03:42.528763+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:08.635943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.635943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.512960+0000","last_change":"2024-02-19T21:03:21.619457+0000","last_active":"2024-02-19T21:03:42.512960+0000","last_peered":"2024-02-19T21:03:42.512960+0000","last_clean":"2024-02-19T21:03:42.512960+0000","last_became_active":"2024-02-19T21:01:16.277826+0000","last_became_peered":"2024-02-19T21:01:16.277826+0000","last_unstale":"2024-02-19T21:03:42.512960+0000","last_undegraded":"2024-02-19T21:03:42.512960+0000","last_fullsized":"2024-02-19T21:03:42.512960+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:21.619389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:21.619389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":62,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.525181+0000","last_change":"2024-02-19T21:02:29.684765+0000","last_active":"2024-02-19T21:03:42.525181+0000","last_peered":"2024-02-19T21:03:42.525181+0000","last_clean":"2024-02-19T21:03:42.525181+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:03:42.525181+0000","last_undegraded":"2024-02-19T21:03:42.525181+0000","last_fullsized":"2024-02-19T21:03:42.525181+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:29.684698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:29.684698+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"65'1","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524727+0000","last_change":"2024-02-19T21:02:36.585127+0000","last_active":"2024-02-19T21:03:42.524727+0000","last_peered":"2024-02-19T21:03:42.524727+0000","last_clean":"2024-02-19T21:03:42.524727+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:03:42.524727+0000","last_undegraded":"2024-02-19T21:03:42.524727+0000","last_fullsized":"2024-02-19T21:03:42.524727+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:02:36.585007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:36.585007+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":84,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-19T21:03:54.453206+0000","last_change":"2024-02-19T21:02:37.842634+0000","last_active":"2024-02-19T21:03:54.453206+0000","last_peered":"2024-02-19T21:03:54.453206+0000","last_clean":"2024-02-19T21:03:54.453206+0000","last_became_active":"2024-02-19T21:01:20.290567+0000","last_became_peered":"2024-02-19T21:01:20.290567+0000","last_unstale":"2024-02-19T21:03:54.453206+0000","last_undegraded":"2024-02-19T21:03:54.453206+0000","last_fullsized":"2024-02-19T21:03:54.453206+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:37.842545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:37.842545+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523450+0000","last_change":"2024-02-19T21:03:06.000805+0000","last_active":"2024-02-19T21:03:42.523450+0000","last_peered":"2024-02-19T21:03:42.523450+0000","last_clean":"2024-02-19T21:03:42.523450+0000","last_became_active":"2024-02-19T21:01:16.272284+0000","last_became_peered":"2024-02-19T21:01:16.272284+0000","last_unstale":"2024-02-19T21:03:42.523450+0000","last_undegraded":"2024-02-19T21:03:42.523450+0000","last_fullsized":"2024-02-19T21:03:42.523450+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.000732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.000732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.528694+0000","last_change":"2024-02-19T21:02:32.550264+0000","last_active":"2024-02-19T21:03:42.528694+0000","last_peered":"2024-02-19T21:03:42.528694+0000","last_clean":"2024-02-19T21:03:42.528694+0000","last_became_active":"2024-02-19T21:01:18.293865+0000","last_became_peered":"2024-02-19T21:01:18.293865+0000","last_unstale":"2024-02-19T21:03:42.528694+0000","last_undegraded":"2024-02-19T21:03:42.528694+0000","last_fullsized":"2024-02-19T21:03:42.528694+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:02:32.550190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.550190+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.515321+0000","last_change":"2024-02-19T21:02:41.964875+0000","last_active":"2024-02-19T21:03:42.515321+0000","last_peered":"2024-02-19T21:03:42.515321+0000","last_clean":"2024-02-19T21:03:42.515321+0000","last_became_active":"2024-02-19T21:01:22.608997+0000","last_became_peered":"2024-02-19T21:01:22.608997+0000","last_unstale":"2024-02-19T21:03:42.515321+0000","last_undegraded":"2024-02-19T21:03:42.515321+0000","last_fullsized":"2024-02-19T21:03:42.515321+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:41.964809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.964809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524881+0000","last_change":"2024-02-19T21:03:04.486168+0000","last_active":"2024-02-19T21:03:42.524881+0000","last_peered":"2024-02-19T21:03:42.524881+0000","last_clean":"2024-02-19T21:03:42.524881+0000","last_became_active":"2024-02-19T21:01:20.287961+0000","last_became_peered":"2024-02-19T21:01:20.287961+0000","last_unstale":"2024-02-19T21:03:42.524881+0000","last_undegraded":"2024-02-19T21:03:42.524881+0000","last_fullsized":"2024-02-19T21:03:42.524881+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.486067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.486067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.525400+0000","last_change":"2024-02-19T21:02:43.790280+0000","last_active":"2024-02-19T21:03:42.525400+0000","last_peered":"2024-02-19T21:03:42.525400+0000","last_clean":"2024-02-19T21:03:42.525400+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:03:42.525400+0000","last_undegraded":"2024-02-19T21:03:42.525400+0000","last_fullsized":"2024-02-19T21:03:42.525400+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:43.790206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:43.790206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":62,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.513185+0000","last_change":"2024-02-19T21:02:52.859531+0000","last_active":"2024-02-19T21:03:42.513185+0000","last_peered":"2024-02-19T21:03:42.513185+0000","last_clean":"2024-02-19T21:03:42.513185+0000","last_became_active":"2024-02-19T21:01:18.293793+0000","last_became_peered":"2024-02-19T21:01:18.293793+0000","last_unstale":"2024-02-19T21:03:42.513185+0000","last_undegraded":"2024-02-19T21:03:42.513185+0000","last_fullsized":"2024-02-19T21:03:42.513185+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:52.859453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.859453+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523658+0000","last_change":"2024-02-19T21:02:45.579070+0000","last_active":"2024-02-19T21:03:42.523658+0000","last_peered":"2024-02-19T21:03:42.523658+0000","last_clean":"2024-02-19T21:03:42.523658+0000","last_became_active":"2024-02-19T21:01:22.297693+0000","last_became_peered":"2024-02-19T21:01:22.297693+0000","last_unstale":"2024-02-19T21:03:42.523658+0000","last_undegraded":"2024-02-19T21:03:42.523658+0000","last_fullsized":"2024-02-19T21:03:42.523658+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:45.578978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:45.578978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523776+0000","last_change":"2024-02-19T21:02:30.724973+0000","last_active":"2024-02-19T21:03:42.523776+0000","last_peered":"2024-02-19T21:03:42.523776+0000","last_clean":"2024-02-19T21:03:42.523776+0000","last_became_active":"2024-02-19T21:01:20.290962+0000","last_became_peered":"2024-02-19T21:01:20.290962+0000","last_unstale":"2024-02-19T21:03:42.523776+0000","last_undegraded":"2024-02-19T21:03:42.523776+0000","last_fullsized":"2024-02-19T21:03:42.523776+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:30.724908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:30.724908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523808+0000","last_change":"2024-02-19T21:02:52.021686+0000","last_active":"2024-02-19T21:03:42.523808+0000","last_peered":"2024-02-19T21:03:42.523808+0000","last_clean":"2024-02-19T21:03:42.523808+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:03:42.523808+0000","last_undegraded":"2024-02-19T21:03:42.523808+0000","last_fullsized":"2024-02-19T21:03:42.523808+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.021603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.021603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.513660+0000","last_change":"2024-02-19T21:03:32.722001+0000","last_active":"2024-02-19T21:03:42.513660+0000","last_peered":"2024-02-19T21:03:42.513660+0000","last_clean":"2024-02-19T21:03:42.513660+0000","last_became_active":"2024-02-19T21:01:18.287874+0000","last_became_peered":"2024-02-19T21:01:18.287874+0000","last_unstale":"2024-02-19T21:03:42.513660+0000","last_undegraded":"2024-02-19T21:03:42.513660+0000","last_fullsized":"2024-02-19T21:03:42.513660+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:03:32.721877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:32.721877+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.512772+0000","last_change":"2024-02-19T21:03:05.734069+0000","last_active":"2024-02-19T21:03:42.512772+0000","last_peered":"2024-02-19T21:03:42.512772+0000","last_clean":"2024-02-19T21:03:42.512772+0000","last_became_active":"2024-02-19T21:01:22.292704+0000","last_became_peered":"2024-02-19T21:01:22.292704+0000","last_unstale":"2024-02-19T21:03:42.512772+0000","last_undegraded":"2024-02-19T21:03:42.512772+0000","last_fullsized":"2024-02-19T21:03:42.512772+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.733980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.733980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.513215+0000","last_change":"2024-02-19T21:02:40.600482+0000","last_active":"2024-02-19T21:03:42.513215+0000","last_peered":"2024-02-19T21:03:42.513215+0000","last_clean":"2024-02-19T21:03:42.513215+0000","last_became_active":"2024-02-19T21:01:20.289020+0000","last_became_peered":"2024-02-19T21:01:20.289020+0000","last_unstale":"2024-02-19T21:03:42.513215+0000","last_undegraded":"2024-02-19T21:03:42.513215+0000","last_fullsized":"2024-02-19T21:03:42.513215+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:40.600410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.600410+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.513476+0000","last_change":"2024-02-19T21:03:20.445543+0000","last_active":"2024-02-19T21:03:42.513476+0000","last_peered":"2024-02-19T21:03:42.513476+0000","last_clean":"2024-02-19T21:03:42.513476+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:03:42.513476+0000","last_undegraded":"2024-02-19T21:03:42.513476+0000","last_fullsized":"2024-02-19T21:03:42.513476+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:20.445477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:20.445477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":62,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508275+0000","last_change":"2024-02-19T21:02:32.040809+0000","last_active":"2024-02-19T21:03:42.508275+0000","last_peered":"2024-02-19T21:03:42.508275+0000","last_clean":"2024-02-19T21:03:42.508275+0000","last_became_active":"2024-02-19T21:01:18.286128+0000","last_became_peered":"2024-02-19T21:01:18.286128+0000","last_unstale":"2024-02-19T21:03:42.508275+0000","last_undegraded":"2024-02-19T21:03:42.508275+0000","last_fullsized":"2024-02-19T21:03:42.508275+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:32.040731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.040731+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.526838+0000","last_change":"2024-02-19T21:02:51.631722+0000","last_active":"2024-02-19T21:03:42.526838+0000","last_peered":"2024-02-19T21:03:42.526838+0000","last_clean":"2024-02-19T21:03:42.526838+0000","last_became_active":"2024-02-19T21:01:22.293093+0000","last_became_peered":"2024-02-19T21:01:22.293093+0000","last_unstale":"2024-02-19T21:03:42.526838+0000","last_undegraded":"2024-02-19T21:03:42.526838+0000","last_fullsized":"2024-02-19T21:03:42.526838+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:51.631657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:51.631657+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.515143+0000","last_change":"2024-02-19T21:03:03.776497+0000","last_active":"2024-02-19T21:03:42.515143+0000","last_peered":"2024-02-19T21:03:42.515143+0000","last_clean":"2024-02-19T21:03:42.515143+0000","last_became_active":"2024-02-19T21:01:20.289691+0000","last_became_peered":"2024-02-19T21:01:20.289691+0000","last_unstale":"2024-02-19T21:03:42.515143+0000","last_undegraded":"2024-02-19T21:03:42.515143+0000","last_fullsized":"2024-02-19T21:03:42.515143+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_clean_scrub_stamp":"2024-02-19T21:03:03.776397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524545+0000","last_change":"2024-02-19T21:03:33.049004+0000","last_active":"2024-02-19T21:03:42.524545+0000","last_peered":"2024-02-19T21:03:42.524545+0000","last_clean":"2024-02-19T21:03:42.524545+0000","last_became_active":"2024-02-19T21:01:16.271316+0000","last_became_peered":"2024-02-19T21:01:16.271316+0000","last_unstale":"2024-02-19T21:03:42.524545+0000","last_undegraded":"2024-02-19T21:03:42.524545+0000","last_fullsized":"2024-02-19T21:03:42.524545+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:33.048928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:33.048928+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523574+0000","last_change":"2024-02-19T21:03:22.984626+0000","last_active":"2024-02-19T21:03:42.523574+0000","last_peered":"2024-02-19T21:03:42.523574+0000","last_clean":"2024-02-19T21:03:42.523574+0000","last_became_active":"2024-02-19T21:01:30.922577+0000","last_became_peered":"2024-02-19T21:01:30.922577+0000","last_unstale":"2024-02-19T21:03:42.523574+0000","last_undegraded":"2024-02-19T21:03:42.523574+0000","last_fullsized":"2024-02-19T21:03:42.523574+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:22.984547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:22.984547+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508988+0000","last_change":"2024-02-19T21:02:39.092233+0000","last_active":"2024-02-19T21:03:42.508988+0000","last_peered":"2024-02-19T21:03:42.508988+0000","last_clean":"2024-02-19T21:03:42.508988+0000","last_became_active":"2024-02-19T21:01:22.299969+0000","last_became_peered":"2024-02-19T21:01:22.299969+0000","last_unstale":"2024-02-19T21:03:42.508988+0000","last_undegraded":"2024-02-19T21:03:42.508988+0000","last_fullsized":"2024-02-19T21:03:42.508988+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.092154+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-19T21:03:52.557345+0000","last_change":"2024-02-19T21:03:26.669863+0000","last_active":"2024-02-19T21:03:52.557345+0000","last_peered":"2024-02-19T21:03:52.557345+0000","last_clean":"2024-02-19T21:03:52.557345+0000","last_became_active":"2024-02-19T21:01:20.283943+0000","last_became_peered":"2024-02-19T21:01:20.283943+0000","last_unstale":"2024-02-19T21:03:52.557345+0000","last_undegraded":"2024-02-19T21:03:52.557345+0000","last_fullsized":"2024-02-19T21:03:52.557345+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:26.669794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:26.669794+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.528087+0000","last_change":"2024-02-19T21:03:39.467124+0000","last_active":"2024-02-19T21:03:42.528087+0000","last_peered":"2024-02-19T21:03:42.528087+0000","last_clean":"2024-02-19T21:03:42.528087+0000","last_became_active":"2024-02-19T21:01:16.273181+0000","last_became_peered":"2024-02-19T21:01:16.273181+0000","last_unstale":"2024-02-19T21:03:42.528087+0000","last_undegraded":"2024-02-19T21:03:42.528087+0000","last_fullsized":"2024-02-19T21:03:42.528087+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:39.467059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:39.467059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.525206+0000","last_change":"2024-02-19T21:02:57.021496+0000","last_active":"2024-02-19T21:03:42.525206+0000","last_peered":"2024-02-19T21:03:42.525206+0000","last_clean":"2024-02-19T21:03:42.525206+0000","last_became_active":"2024-02-19T21:01:18.293786+0000","last_became_peered":"2024-02-19T21:01:18.293786+0000","last_unstale":"2024-02-19T21:03:42.525206+0000","last_undegraded":"2024-02-19T21:03:42.525206+0000","last_fullsized":"2024-02-19T21:03:42.525206+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:57.021403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.021403+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.527541+0000","last_change":"2024-02-19T21:02:39.442600+0000","last_active":"2024-02-19T21:03:42.527541+0000","last_peered":"2024-02-19T21:03:42.527541+0000","last_clean":"2024-02-19T21:03:42.527541+0000","last_became_active":"2024-02-19T21:01:22.297188+0000","last_became_peered":"2024-02-19T21:01:22.297188+0000","last_unstale":"2024-02-19T21:03:42.527541+0000","last_undegraded":"2024-02-19T21:03:42.527541+0000","last_fullsized":"2024-02-19T21:03:42.527541+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:02:39.442533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.442533+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-19T21:03:52.553504+0000","last_change":"2024-02-19T21:03:11.063302+0000","last_active":"2024-02-19T21:03:52.553504+0000","last_peered":"2024-02-19T21:03:52.553504+0000","last_clean":"2024-02-19T21:03:52.553504+0000","last_became_active":"2024-02-19T21:01:20.288799+0000","last_became_peered":"2024-02-19T21:01:20.288799+0000","last_unstale":"2024-02-19T21:03:52.553504+0000","last_undegraded":"2024-02-19T21:03:52.553504+0000","last_fullsized":"2024-02-19T21:03:52.553504+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:11.063222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:11.063222+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523810+0000","last_change":"2024-02-19T21:02:49.056495+0000","last_active":"2024-02-19T21:03:42.523810+0000","last_peered":"2024-02-19T21:03:42.523810+0000","last_clean":"2024-02-19T21:03:42.523810+0000","last_became_active":"2024-02-19T21:01:16.271608+0000","last_became_peered":"2024-02-19T21:01:16.271608+0000","last_unstale":"2024-02-19T21:03:42.523810+0000","last_undegraded":"2024-02-19T21:03:42.523810+0000","last_fullsized":"2024-02-19T21:03:42.523810+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.056412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.056412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.515182+0000","last_change":"2024-02-19T21:03:08.742946+0000","last_active":"2024-02-19T21:03:42.515182+0000","last_peered":"2024-02-19T21:03:42.515182+0000","last_clean":"2024-02-19T21:03:42.515182+0000","last_became_active":"2024-02-19T21:01:18.294369+0000","last_became_peered":"2024-02-19T21:01:18.294369+0000","last_unstale":"2024-02-19T21:03:42.515182+0000","last_undegraded":"2024-02-19T21:03:42.515182+0000","last_fullsized":"2024-02-19T21:03:42.515182+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:03:08.742876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.742876+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524107+0000","last_change":"2024-02-19T21:02:38.633030+0000","last_active":"2024-02-19T21:03:42.524107+0000","last_peered":"2024-02-19T21:03:42.524107+0000","last_clean":"2024-02-19T21:03:42.524107+0000","last_became_active":"2024-02-19T21:01:22.608285+0000","last_became_peered":"2024-02-19T21:01:22.608285+0000","last_unstale":"2024-02-19T21:03:42.524107+0000","last_undegraded":"2024-02-19T21:03:42.524107+0000","last_fullsized":"2024-02-19T21:03:42.524107+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:38.632950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:38.632950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523731+0000","last_change":"2024-02-19T21:03:08.984824+0000","last_active":"2024-02-19T21:03:42.523731+0000","last_peered":"2024-02-19T21:03:42.523731+0000","last_clean":"2024-02-19T21:03:42.523731+0000","last_became_active":"2024-02-19T21:01:20.288589+0000","last_became_peered":"2024-02-19T21:01:20.288589+0000","last_unstale":"2024-02-19T21:03:42.523731+0000","last_undegraded":"2024-02-19T21:03:42.523731+0000","last_fullsized":"2024-02-19T21:03:42.523731+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:08.984720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.984720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523854+0000","last_change":"2024-02-19T21:02:46.842812+0000","last_active":"2024-02-19T21:03:42.523854+0000","last_peered":"2024-02-19T21:03:42.523854+0000","last_clean":"2024-02-19T21:03:42.523854+0000","last_became_active":"2024-02-19T21:01:16.271210+0000","last_became_peered":"2024-02-19T21:01:16.271210+0000","last_unstale":"2024-02-19T21:03:42.523854+0000","last_undegraded":"2024-02-19T21:03:42.523854+0000","last_fullsized":"2024-02-19T21:03:42.523854+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:46.842732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.842732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.516291+0000","last_change":"2024-02-19T21:02:33.968248+0000","last_active":"2024-02-19T21:03:42.516291+0000","last_peered":"2024-02-19T21:03:42.516291+0000","last_clean":"2024-02-19T21:03:42.516291+0000","last_became_active":"2024-02-19T21:01:18.293913+0000","last_became_peered":"2024-02-19T21:01:18.293913+0000","last_unstale":"2024-02-19T21:03:42.516291+0000","last_undegraded":"2024-02-19T21:03:42.516291+0000","last_fullsized":"2024-02-19T21:03:42.516291+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:02:33.968182+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524064+0000","last_change":"2024-02-19T21:02:34.274719+0000","last_active":"2024-02-19T21:03:42.524064+0000","last_peered":"2024-02-19T21:03:42.524064+0000","last_clean":"2024-02-19T21:03:42.524064+0000","last_became_active":"2024-02-19T21:01:22.297086+0000","last_became_peered":"2024-02-19T21:01:22.297086+0000","last_unstale":"2024-02-19T21:03:42.524064+0000","last_undegraded":"2024-02-19T21:03:42.524064+0000","last_fullsized":"2024-02-19T21:03:42.524064+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:34.274650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:34.274650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.516449+0000","last_change":"2024-02-19T21:02:49.863142+0000","last_active":"2024-02-19T21:03:42.516449+0000","last_peered":"2024-02-19T21:03:42.516449+0000","last_clean":"2024-02-19T21:03:42.516449+0000","last_became_active":"2024-02-19T21:01:20.285298+0000","last_became_peered":"2024-02-19T21:01:20.285298+0000","last_unstale":"2024-02-19T21:03:42.516449+0000","last_undegraded":"2024-02-19T21:03:42.516449+0000","last_fullsized":"2024-02-19T21:03:42.516449+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.863037+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523497+0000","last_change":"2024-02-19T21:02:49.177055+0000","last_active":"2024-02-19T21:03:42.523497+0000","last_peered":"2024-02-19T21:03:42.523497+0000","last_clean":"2024-02-19T21:03:42.523497+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:03:42.523497+0000","last_undegraded":"2024-02-19T21:03:42.523497+0000","last_fullsized":"2024-02-19T21:03:42.523497+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.176969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.176969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.528436+0000","last_change":"2024-02-19T21:03:07.669010+0000","last_active":"2024-02-19T21:03:42.528436+0000","last_peered":"2024-02-19T21:03:42.528436+0000","last_clean":"2024-02-19T21:03:42.528436+0000","last_became_active":"2024-02-19T21:01:18.293773+0000","last_became_peered":"2024-02-19T21:01:18.293773+0000","last_unstale":"2024-02-19T21:03:42.528436+0000","last_undegraded":"2024-02-19T21:03:42.528436+0000","last_fullsized":"2024-02-19T21:03:42.528436+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:07.668943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:07.668943+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.515234+0000","last_change":"2024-02-19T21:03:17.625775+0000","last_active":"2024-02-19T21:03:42.515234+0000","last_peered":"2024-02-19T21:03:42.515234+0000","last_clean":"2024-02-19T21:03:42.515234+0000","last_became_active":"2024-02-19T21:01:22.295912+0000","last_became_peered":"2024-02-19T21:01:22.295912+0000","last_unstale":"2024-02-19T21:03:42.515234+0000","last_undegraded":"2024-02-19T21:03:42.515234+0000","last_fullsized":"2024-02-19T21:03:42.515234+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:03:17.625711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508842+0000","last_change":"2024-02-19T21:03:21.164366+0000","last_active":"2024-02-19T21:03:42.508842+0000","last_peered":"2024-02-19T21:03:42.508842+0000","last_clean":"2024-02-19T21:03:42.508842+0000","last_became_active":"2024-02-19T21:01:16.273545+0000","last_became_peered":"2024-02-19T21:01:16.273545+0000","last_unstale":"2024-02-19T21:03:42.508842+0000","last_undegraded":"2024-02-19T21:03:42.508842+0000","last_fullsized":"2024-02-19T21:03:42.508842+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:03:21.164286+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508303+0000","last_change":"2024-02-19T21:03:08.046722+0000","last_active":"2024-02-19T21:03:42.508303+0000","last_peered":"2024-02-19T21:03:42.508303+0000","last_clean":"2024-02-19T21:03:42.508303+0000","last_became_active":"2024-02-19T21:01:18.285672+0000","last_became_peered":"2024-02-19T21:01:18.285672+0000","last_unstale":"2024-02-19T21:03:42.508303+0000","last_undegraded":"2024-02-19T21:03:42.508303+0000","last_fullsized":"2024-02-19T21:03:42.508303+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:03:08.046641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.046641+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523414+0000","last_change":"2024-02-19T21:02:35.252642+0000","last_active":"2024-02-19T21:03:42.523414+0000","last_peered":"2024-02-19T21:03:42.523414+0000","last_clean":"2024-02-19T21:03:42.523414+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:03:42.523414+0000","last_undegraded":"2024-02-19T21:03:42.523414+0000","last_fullsized":"2024-02-19T21:03:42.523414+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:35.252569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:35.252569+0000","last_clean_scrub_stamp":"2024-02-19T21:02:35.252569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.526904+0000","last_change":"2024-02-19T21:02:47.593210+0000","last_active":"2024-02-19T21:03:42.526904+0000","last_peered":"2024-02-19T21:03:42.526904+0000","last_clean":"2024-02-19T21:03:42.526904+0000","last_became_active":"2024-02-19T21:01:22.292133+0000","last_became_peered":"2024-02-19T21:01:22.292133+0000","last_unstale":"2024-02-19T21:03:42.526904+0000","last_undegraded":"2024-02-19T21:03:42.526904+0000","last_fullsized":"2024-02-19T21:03:42.526904+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.593132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.593132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.515582+0000","last_change":"2024-02-19T21:03:01.530211+0000","last_active":"2024-02-19T21:03:42.515582+0000","last_peered":"2024-02-19T21:03:42.515582+0000","last_clean":"2024-02-19T21:03:42.515582+0000","last_became_active":"2024-02-19T21:01:30.926676+0000","last_became_peered":"2024-02-19T21:01:30.926676+0000","last_unstale":"2024-02-19T21:03:42.515582+0000","last_undegraded":"2024-02-19T21:03:42.515582+0000","last_fullsized":"2024-02-19T21:03:42.515582+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.530108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.513136+0000","last_change":"2024-02-19T21:02:56.539408+0000","last_active":"2024-02-19T21:03:42.513136+0000","last_peered":"2024-02-19T21:03:42.513136+0000","last_clean":"2024-02-19T21:03:42.513136+0000","last_became_active":"2024-02-19T21:01:18.285891+0000","last_became_peered":"2024-02-19T21:01:18.285891+0000","last_unstale":"2024-02-19T21:03:42.513136+0000","last_undegraded":"2024-02-19T21:03:42.513136+0000","last_fullsized":"2024-02-19T21:03:42.513136+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:02:56.539303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.539303+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":84,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-19T21:03:52.552730+0000","last_change":"2024-02-19T21:03:10.626933+0000","last_active":"2024-02-19T21:03:52.552730+0000","last_peered":"2024-02-19T21:03:52.552730+0000","last_clean":"2024-02-19T21:03:52.552730+0000","last_became_active":"2024-02-19T21:01:20.288918+0000","last_became_peered":"2024-02-19T21:01:20.288918+0000","last_unstale":"2024-02-19T21:03:52.552730+0000","last_undegraded":"2024-02-19T21:03:52.552730+0000","last_fullsized":"2024-02-19T21:03:52.552730+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:10.626866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:10.626866+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.517098+0000","last_change":"2024-02-19T21:03:12.734312+0000","last_active":"2024-02-19T21:03:42.517098+0000","last_peered":"2024-02-19T21:03:42.517098+0000","last_clean":"2024-02-19T21:03:42.517098+0000","last_became_active":"2024-02-19T21:01:22.291396+0000","last_became_peered":"2024-02-19T21:01:22.291396+0000","last_unstale":"2024-02-19T21:03:42.517098+0000","last_undegraded":"2024-02-19T21:03:42.517098+0000","last_fullsized":"2024-02-19T21:03:42.517098+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:12.734219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:12.734219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523887+0000","last_change":"2024-02-19T21:02:58.986556+0000","last_active":"2024-02-19T21:03:42.523887+0000","last_peered":"2024-02-19T21:03:42.523887+0000","last_clean":"2024-02-19T21:03:42.523887+0000","last_became_active":"2024-02-19T21:01:16.270159+0000","last_became_peered":"2024-02-19T21:01:16.270159+0000","last_unstale":"2024-02-19T21:03:42.523887+0000","last_undegraded":"2024-02-19T21:03:42.523887+0000","last_fullsized":"2024-02-19T21:03:42.523887+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:58.986458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:58.986458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":123,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-19T21:03:54.453186+0000","last_change":"2024-02-19T21:02:46.558811+0000","last_active":"2024-02-19T21:03:54.453186+0000","last_peered":"2024-02-19T21:03:54.453186+0000","last_clean":"2024-02-19T21:03:54.453186+0000","last_became_active":"2024-02-19T21:01:18.289076+0000","last_became_peered":"2024-02-19T21:01:18.289076+0000","last_unstale":"2024-02-19T21:03:54.453186+0000","last_undegraded":"2024-02-19T21:03:54.453186+0000","last_fullsized":"2024-02-19T21:03:54.453186+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-19T21:02:46.558739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.558739+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.527828+0000","last_change":"2024-02-19T21:02:56.510539+0000","last_active":"2024-02-19T21:03:42.527828+0000","last_peered":"2024-02-19T21:03:42.527828+0000","last_clean":"2024-02-19T21:03:42.527828+0000","last_became_active":"2024-02-19T21:01:20.290758+0000","last_became_peered":"2024-02-19T21:01:20.290758+0000","last_unstale":"2024-02-19T21:03:42.527828+0000","last_undegraded":"2024-02-19T21:03:42.527828+0000","last_fullsized":"2024-02-19T21:03:42.527828+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.510474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524328+0000","last_change":"2024-02-19T21:02:32.488666+0000","last_active":"2024-02-19T21:03:42.524328+0000","last_peered":"2024-02-19T21:03:42.524328+0000","last_clean":"2024-02-19T21:03:42.524328+0000","last_became_active":"2024-02-19T21:01:22.607883+0000","last_became_peered":"2024-02-19T21:01:22.607883+0000","last_unstale":"2024-02-19T21:03:42.524328+0000","last_undegraded":"2024-02-19T21:03:42.524328+0000","last_fullsized":"2024-02-19T21:03:42.524328+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:32.488584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.488584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":200,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-19T21:03:54.452632+0000","last_change":"2024-02-19T21:02:55.568674+0000","last_active":"2024-02-19T21:03:54.452632+0000","last_peered":"2024-02-19T21:03:54.452632+0000","last_clean":"2024-02-19T21:03:54.452632+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:03:54.452632+0000","last_undegraded":"2024-02-19T21:03:54.452632+0000","last_fullsized":"2024-02-19T21:03:54.452632+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:02:55.568593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.568593+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":152,"num_read_kb":147,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524363+0000","last_change":"2024-02-19T21:03:35.999651+0000","last_active":"2024-02-19T21:03:42.524363+0000","last_peered":"2024-02-19T21:03:42.524363+0000","last_clean":"2024-02-19T21:03:42.524363+0000","last_became_active":"2024-02-19T21:01:16.276032+0000","last_became_peered":"2024-02-19T21:01:16.276032+0000","last_unstale":"2024-02-19T21:03:42.524363+0000","last_undegraded":"2024-02-19T21:03:42.524363+0000","last_fullsized":"2024-02-19T21:03:42.524363+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:35.999575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:35.999575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523582+0000","last_change":"2024-02-19T21:03:27.966621+0000","last_active":"2024-02-19T21:03:42.523582+0000","last_peered":"2024-02-19T21:03:42.523582+0000","last_clean":"2024-02-19T21:03:42.523582+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:03:42.523582+0000","last_undegraded":"2024-02-19T21:03:42.523582+0000","last_fullsized":"2024-02-19T21:03:42.523582+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:03:27.966528+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523793+0000","last_change":"2024-02-19T21:02:45.222033+0000","last_active":"2024-02-19T21:03:42.523793+0000","last_peered":"2024-02-19T21:03:42.523793+0000","last_clean":"2024-02-19T21:03:42.523793+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:03:42.523793+0000","last_undegraded":"2024-02-19T21:03:42.523793+0000","last_fullsized":"2024-02-19T21:03:42.523793+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:45.221965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:45.221965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.525528+0000","last_change":"2024-02-19T21:02:25.326504+0000","last_active":"2024-02-19T21:03:42.525528+0000","last_peered":"2024-02-19T21:03:42.525528+0000","last_clean":"2024-02-19T21:03:42.525528+0000","last_became_active":"2024-02-19T21:01:22.292299+0000","last_became_peered":"2024-02-19T21:01:22.292299+0000","last_unstale":"2024-02-19T21:03:42.525528+0000","last_undegraded":"2024-02-19T21:03:42.525528+0000","last_fullsized":"2024-02-19T21:03:42.525528+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:25.326441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:25.326441+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.512755+0000","last_change":"2024-02-19T21:03:19.665710+0000","last_active":"2024-02-19T21:03:42.512755+0000","last_peered":"2024-02-19T21:03:42.512755+0000","last_clean":"2024-02-19T21:03:42.512755+0000","last_became_active":"2024-02-19T21:01:16.274290+0000","last_became_peered":"2024-02-19T21:01:16.274290+0000","last_unstale":"2024-02-19T21:03:42.512755+0000","last_undegraded":"2024-02-19T21:03:42.512755+0000","last_fullsized":"2024-02-19T21:03:42.512755+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:19.665646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:19.665646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524393+0000","last_change":"2024-02-19T21:02:52.986989+0000","last_active":"2024-02-19T21:03:42.524393+0000","last_peered":"2024-02-19T21:03:42.524393+0000","last_clean":"2024-02-19T21:03:42.524393+0000","last_became_active":"2024-02-19T21:01:16.276807+0000","last_became_peered":"2024-02-19T21:01:16.276807+0000","last_unstale":"2024-02-19T21:03:42.524393+0000","last_undegraded":"2024-02-19T21:03:42.524393+0000","last_fullsized":"2024-02-19T21:03:42.524393+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.986913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.986913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524722+0000","last_change":"2024-02-19T21:03:11.598713+0000","last_active":"2024-02-19T21:03:42.524722+0000","last_peered":"2024-02-19T21:03:42.524722+0000","last_clean":"2024-02-19T21:03:42.524722+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:03:42.524722+0000","last_undegraded":"2024-02-19T21:03:42.524722+0000","last_fullsized":"2024-02-19T21:03:42.524722+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:11.598603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:03:11.598603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":65,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524859+0000","last_change":"2024-02-19T21:02:32.777906+0000","last_active":"2024-02-19T21:03:42.524859+0000","last_peered":"2024-02-19T21:03:42.524859+0000","last_clean":"2024-02-19T21:03:42.524859+0000","last_became_active":"2024-02-19T21:01:18.293784+0000","last_became_peered":"2024-02-19T21:01:18.293784+0000","last_unstale":"2024-02-19T21:03:42.524859+0000","last_undegraded":"2024-02-19T21:03:42.524859+0000","last_fullsized":"2024-02-19T21:03:42.524859+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:02:32.777827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:32.777827+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524548+0000","last_change":"2024-02-19T21:02:55.586944+0000","last_active":"2024-02-19T21:03:42.524548+0000","last_peered":"2024-02-19T21:03:42.524548+0000","last_clean":"2024-02-19T21:03:42.524548+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:03:42.524548+0000","last_undegraded":"2024-02-19T21:03:42.524548+0000","last_fullsized":"2024-02-19T21:03:42.524548+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:55.586854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.586854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.515159+0000","last_change":"2024-02-19T21:02:52.556438+0000","last_active":"2024-02-19T21:03:42.515159+0000","last_peered":"2024-02-19T21:03:42.515159+0000","last_clean":"2024-02-19T21:03:42.515159+0000","last_became_active":"2024-02-19T21:01:22.298477+0000","last_became_peered":"2024-02-19T21:01:22.298477+0000","last_unstale":"2024-02-19T21:03:42.515159+0000","last_undegraded":"2024-02-19T21:03:42.515159+0000","last_fullsized":"2024-02-19T21:03:42.515159+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.556344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.556344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.527342+0000","last_change":"2024-02-19T21:02:57.505093+0000","last_active":"2024-02-19T21:03:42.527342+0000","last_peered":"2024-02-19T21:03:42.527342+0000","last_clean":"2024-02-19T21:03:42.527342+0000","last_became_active":"2024-02-19T21:01:16.274398+0000","last_became_peered":"2024-02-19T21:01:16.274398+0000","last_unstale":"2024-02-19T21:03:42.527342+0000","last_undegraded":"2024-02-19T21:03:42.527342+0000","last_fullsized":"2024-02-19T21:03:42.527342+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.505029+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524635+0000","last_change":"2024-02-19T21:02:41.616831+0000","last_active":"2024-02-19T21:03:42.524635+0000","last_peered":"2024-02-19T21:03:42.524635+0000","last_clean":"2024-02-19T21:03:42.524635+0000","last_became_active":"2024-02-19T21:01:16.275904+0000","last_became_peered":"2024-02-19T21:01:16.275904+0000","last_unstale":"2024-02-19T21:03:42.524635+0000","last_undegraded":"2024-02-19T21:03:42.524635+0000","last_fullsized":"2024-02-19T21:03:42.524635+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:02:41.616729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.616729+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.527622+0000","last_change":"2024-02-19T21:03:22.495828+0000","last_active":"2024-02-19T21:03:42.527622+0000","last_peered":"2024-02-19T21:03:42.527622+0000","last_clean":"2024-02-19T21:03:42.527622+0000","last_became_active":"2024-02-19T21:01:18.287139+0000","last_became_peered":"2024-02-19T21:01:18.287139+0000","last_unstale":"2024-02-19T21:03:42.527622+0000","last_undegraded":"2024-02-19T21:03:42.527622+0000","last_fullsized":"2024-02-19T21:03:42.527622+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:03:22.495754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:22.495754+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.525450+0000","last_change":"2024-02-19T21:03:05.005360+0000","last_active":"2024-02-19T21:03:42.525450+0000","last_peered":"2024-02-19T21:03:42.525450+0000","last_clean":"2024-02-19T21:03:42.525450+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:03:42.525450+0000","last_undegraded":"2024-02-19T21:03:42.525450+0000","last_fullsized":"2024-02-19T21:03:42.525450+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.005288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.005288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.512612+0000","last_change":"2024-02-19T21:03:07.473156+0000","last_active":"2024-02-19T21:03:42.512612+0000","last_peered":"2024-02-19T21:03:42.512612+0000","last_clean":"2024-02-19T21:03:42.512612+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:03:42.512612+0000","last_undegraded":"2024-02-19T21:03:42.512612+0000","last_fullsized":"2024-02-19T21:03:42.512612+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:03:07.473070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524404+0000","last_change":"2024-02-19T21:02:46.565447+0000","last_active":"2024-02-19T21:03:42.524404+0000","last_peered":"2024-02-19T21:03:42.524404+0000","last_clean":"2024-02-19T21:03:42.524404+0000","last_became_active":"2024-02-19T21:01:16.274215+0000","last_became_peered":"2024-02-19T21:01:16.274215+0000","last_unstale":"2024-02-19T21:03:42.524404+0000","last_undegraded":"2024-02-19T21:03:42.524404+0000","last_fullsized":"2024-02-19T21:03:42.524404+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:46.565380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.565380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":79,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.528010+0000","last_change":"2024-02-19T21:02:35.502975+0000","last_active":"2024-02-19T21:03:42.528010+0000","last_peered":"2024-02-19T21:03:42.528010+0000","last_clean":"2024-02-19T21:03:42.528010+0000","last_became_active":"2024-02-19T21:01:18.293791+0000","last_became_peered":"2024-02-19T21:01:18.293791+0000","last_unstale":"2024-02-19T21:03:42.528010+0000","last_undegraded":"2024-02-19T21:03:42.528010+0000","last_fullsized":"2024-02-19T21:03:42.528010+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:02:35.502906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:35.502906+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.525725+0000","last_change":"2024-02-19T21:03:36.004677+0000","last_active":"2024-02-19T21:03:42.525725+0000","last_peered":"2024-02-19T21:03:42.525725+0000","last_clean":"2024-02-19T21:03:42.525725+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:03:42.525725+0000","last_undegraded":"2024-02-19T21:03:42.525725+0000","last_fullsized":"2024-02-19T21:03:42.525725+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:36.004580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:36.004580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508581+0000","last_change":"2024-02-19T21:03:17.088777+0000","last_active":"2024-02-19T21:03:42.508581+0000","last_peered":"2024-02-19T21:03:42.508581+0000","last_clean":"2024-02-19T21:03:42.508581+0000","last_became_active":"2024-02-19T21:01:22.297707+0000","last_became_peered":"2024-02-19T21:01:22.297707+0000","last_unstale":"2024-02-19T21:03:42.508581+0000","last_undegraded":"2024-02-19T21:03:42.508581+0000","last_fullsized":"2024-02-19T21:03:42.508581+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:17.088695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:17.088695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.513985+0000","last_change":"2024-02-19T21:02:47.601190+0000","last_active":"2024-02-19T21:03:42.513985+0000","last_peered":"2024-02-19T21:03:42.513985+0000","last_clean":"2024-02-19T21:03:42.513985+0000","last_became_active":"2024-02-19T21:01:16.271340+0000","last_became_peered":"2024-02-19T21:01:16.271340+0000","last_unstale":"2024-02-19T21:03:42.513985+0000","last_undegraded":"2024-02-19T21:03:42.513985+0000","last_fullsized":"2024-02-19T21:03:42.513985+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.601127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.601127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":75,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.513522+0000","last_change":"2024-02-19T21:02:39.952691+0000","last_active":"2024-02-19T21:03:42.513522+0000","last_peered":"2024-02-19T21:03:42.513522+0000","last_clean":"2024-02-19T21:03:42.513522+0000","last_became_active":"2024-02-19T21:01:18.293812+0000","last_became_peered":"2024-02-19T21:01:18.293812+0000","last_unstale":"2024-02-19T21:03:42.513522+0000","last_undegraded":"2024-02-19T21:03:42.513522+0000","last_fullsized":"2024-02-19T21:03:42.513522+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:02:39.952624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.952624+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508332+0000","last_change":"2024-02-19T21:02:57.997535+0000","last_active":"2024-02-19T21:03:42.508332+0000","last_peered":"2024-02-19T21:03:42.508332+0000","last_clean":"2024-02-19T21:03:42.508332+0000","last_became_active":"2024-02-19T21:01:20.291079+0000","last_became_peered":"2024-02-19T21:01:20.291079+0000","last_unstale":"2024-02-19T21:03:42.508332+0000","last_undegraded":"2024-02-19T21:03:42.508332+0000","last_fullsized":"2024-02-19T21:03:42.508332+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.997427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524117+0000","last_change":"2024-02-19T21:03:01.496926+0000","last_active":"2024-02-19T21:03:42.524117+0000","last_peered":"2024-02-19T21:03:42.524117+0000","last_clean":"2024-02-19T21:03:42.524117+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:03:42.524117+0000","last_undegraded":"2024-02-19T21:03:42.524117+0000","last_fullsized":"2024-02-19T21:03:42.524117+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.496833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.496833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.525322+0000","last_change":"2024-02-19T21:02:50.921031+0000","last_active":"2024-02-19T21:03:42.525322+0000","last_peered":"2024-02-19T21:03:42.525322+0000","last_clean":"2024-02-19T21:03:42.525322+0000","last_became_active":"2024-02-19T21:01:16.274462+0000","last_became_peered":"2024-02-19T21:01:16.274462+0000","last_unstale":"2024-02-19T21:03:42.525322+0000","last_undegraded":"2024-02-19T21:03:42.525322+0000","last_fullsized":"2024-02-19T21:03:42.525322+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:50.920948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:50.920948+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":64,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508210+0000","last_change":"2024-02-19T21:02:40.105227+0000","last_active":"2024-02-19T21:03:42.508210+0000","last_peered":"2024-02-19T21:03:42.508210+0000","last_clean":"2024-02-19T21:03:42.508210+0000","last_became_active":"2024-02-19T21:01:18.282702+0000","last_became_peered":"2024-02-19T21:01:18.282702+0000","last_unstale":"2024-02-19T21:03:42.508210+0000","last_undegraded":"2024-02-19T21:03:42.508210+0000","last_fullsized":"2024-02-19T21:03:42.508210+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:02:40.105141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.105141+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":86,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-19T21:03:52.555653+0000","last_change":"2024-02-19T21:03:33.736454+0000","last_active":"2024-02-19T21:03:52.555653+0000","last_peered":"2024-02-19T21:03:52.555653+0000","last_clean":"2024-02-19T21:03:52.555653+0000","last_became_active":"2024-02-19T21:01:20.286505+0000","last_became_peered":"2024-02-19T21:01:20.286505+0000","last_unstale":"2024-02-19T21:03:52.555653+0000","last_undegraded":"2024-02-19T21:03:52.555653+0000","last_fullsized":"2024-02-19T21:03:52.555653+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:33.736366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:33.736366+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.515093+0000","last_change":"2024-02-19T21:03:23.656785+0000","last_active":"2024-02-19T21:03:42.515093+0000","last_peered":"2024-02-19T21:03:42.515093+0000","last_clean":"2024-02-19T21:03:42.515093+0000","last_became_active":"2024-02-19T21:01:22.607007+0000","last_became_peered":"2024-02-19T21:01:22.607007+0000","last_unstale":"2024-02-19T21:03:42.515093+0000","last_undegraded":"2024-02-19T21:03:42.515093+0000","last_fullsized":"2024-02-19T21:03:42.515093+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:23.656688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:23.656688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.525602+0000","last_change":"2024-02-19T21:03:06.991276+0000","last_active":"2024-02-19T21:03:42.525602+0000","last_peered":"2024-02-19T21:03:42.525602+0000","last_clean":"2024-02-19T21:03:42.525602+0000","last_became_active":"2024-02-19T21:01:20.289375+0000","last_became_peered":"2024-02-19T21:01:20.289375+0000","last_unstale":"2024-02-19T21:03:42.525602+0000","last_undegraded":"2024-02-19T21:03:42.525602+0000","last_fullsized":"2024-02-19T21:03:42.525602+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.991199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.991199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524780+0000","last_change":"2024-02-19T21:02:27.567995+0000","last_active":"2024-02-19T21:03:42.524780+0000","last_peered":"2024-02-19T21:03:42.524780+0000","last_clean":"2024-02-19T21:03:42.524780+0000","last_became_active":"2024-02-19T21:01:16.275520+0000","last_became_peered":"2024-02-19T21:01:16.275520+0000","last_unstale":"2024-02-19T21:03:42.524780+0000","last_undegraded":"2024-02-19T21:03:42.524780+0000","last_fullsized":"2024-02-19T21:03:42.524780+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:27.567903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:27.567903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":67,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.509004+0000","last_change":"2024-02-19T21:03:35.086667+0000","last_active":"2024-02-19T21:03:42.509004+0000","last_peered":"2024-02-19T21:03:42.509004+0000","last_clean":"2024-02-19T21:03:42.509004+0000","last_became_active":"2024-02-19T21:01:18.285670+0000","last_became_peered":"2024-02-19T21:01:18.285670+0000","last_unstale":"2024-02-19T21:03:42.509004+0000","last_undegraded":"2024-02-19T21:03:42.509004+0000","last_fullsized":"2024-02-19T21:03:42.509004+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:03:35.086579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:35.086579+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.525370+0000","last_change":"2024-02-19T21:02:41.195082+0000","last_active":"2024-02-19T21:03:42.525370+0000","last_peered":"2024-02-19T21:03:42.525370+0000","last_clean":"2024-02-19T21:03:42.525370+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:03:42.525370+0000","last_undegraded":"2024-02-19T21:03:42.525370+0000","last_fullsized":"2024-02-19T21:03:42.525370+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:41.194993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.194993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-19T21:03:54.454740+0000","last_change":"2024-02-19T21:02:48.027160+0000","last_active":"2024-02-19T21:03:54.454740+0000","last_peered":"2024-02-19T21:03:54.454740+0000","last_clean":"2024-02-19T21:03:54.454740+0000","last_became_active":"2024-02-19T21:01:20.293341+0000","last_became_peered":"2024-02-19T21:01:20.293341+0000","last_unstale":"2024-02-19T21:03:54.454740+0000","last_undegraded":"2024-02-19T21:03:54.454740+0000","last_fullsized":"2024-02-19T21:03:54.454740+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:48.027070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.027070+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.512810+0000","last_change":"2024-02-19T21:02:28.471214+0000","last_active":"2024-02-19T21:03:42.512810+0000","last_peered":"2024-02-19T21:03:42.512810+0000","last_clean":"2024-02-19T21:03:42.512810+0000","last_became_active":"2024-02-19T21:01:16.274939+0000","last_became_peered":"2024-02-19T21:01:16.274939+0000","last_unstale":"2024-02-19T21:03:42.512810+0000","last_undegraded":"2024-02-19T21:03:42.512810+0000","last_fullsized":"2024-02-19T21:03:42.512810+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:28.471131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:28.471131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.514152+0000","last_change":"2024-02-19T21:02:53.526211+0000","last_active":"2024-02-19T21:03:42.514152+0000","last_peered":"2024-02-19T21:03:42.514152+0000","last_clean":"2024-02-19T21:03:42.514152+0000","last_became_active":"2024-02-19T21:01:18.285534+0000","last_became_peered":"2024-02-19T21:01:18.285534+0000","last_unstale":"2024-02-19T21:03:42.514152+0000","last_undegraded":"2024-02-19T21:03:42.514152+0000","last_fullsized":"2024-02-19T21:03:42.514152+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:53.526141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:53.526141+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508712+0000","last_change":"2024-02-19T21:02:42.123367+0000","last_active":"2024-02-19T21:03:42.508712+0000","last_peered":"2024-02-19T21:03:42.508712+0000","last_clean":"2024-02-19T21:03:42.508712+0000","last_became_active":"2024-02-19T21:01:22.299993+0000","last_became_peered":"2024-02-19T21:01:22.299993+0000","last_unstale":"2024-02-19T21:03:42.508712+0000","last_undegraded":"2024-02-19T21:03:42.508712+0000","last_fullsized":"2024-02-19T21:03:42.508712+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:42.123299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:42.123299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-19T21:03:54.453807+0000","last_change":"2024-02-19T21:02:25.255767+0000","last_active":"2024-02-19T21:03:54.453807+0000","last_peered":"2024-02-19T21:03:54.453807+0000","last_clean":"2024-02-19T21:03:54.453807+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:03:54.453807+0000","last_undegraded":"2024-02-19T21:03:54.453807+0000","last_fullsized":"2024-02-19T21:03:54.453807+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:25.255648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:25.255648+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.513051+0000","last_change":"2024-02-19T21:02:42.525447+0000","last_active":"2024-02-19T21:03:42.513051+0000","last_peered":"2024-02-19T21:03:42.513051+0000","last_clean":"2024-02-19T21:03:42.513051+0000","last_became_active":"2024-02-19T21:01:16.274086+0000","last_became_peered":"2024-02-19T21:01:16.274086+0000","last_unstale":"2024-02-19T21:03:42.513051+0000","last_undegraded":"2024-02-19T21:03:42.513051+0000","last_fullsized":"2024-02-19T21:03:42.513051+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:42.525345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:42.525345+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"82'15","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508431+0000","last_change":"2024-02-19T21:03:04.086368+0000","last_active":"2024-02-19T21:03:42.508431+0000","last_peered":"2024-02-19T21:03:42.508431+0000","last_clean":"2024-02-19T21:03:42.508431+0000","last_became_active":"2024-02-19T21:01:18.282805+0000","last_became_peered":"2024-02-19T21:01:18.282805+0000","last_unstale":"2024-02-19T21:03:42.508431+0000","last_undegraded":"2024-02-19T21:03:42.508431+0000","last_fullsized":"2024-02-19T21:03:42.508431+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.086282+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523665+0000","last_change":"2024-02-19T21:02:54.197783+0000","last_active":"2024-02-19T21:03:42.523665+0000","last_peered":"2024-02-19T21:03:42.523665+0000","last_clean":"2024-02-19T21:03:42.523665+0000","last_became_active":"2024-02-19T21:01:22.298941+0000","last_became_peered":"2024-02-19T21:01:22.298941+0000","last_unstale":"2024-02-19T21:03:42.523665+0000","last_undegraded":"2024-02-19T21:03:42.523665+0000","last_fullsized":"2024-02-19T21:03:42.523665+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:54.197714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:54.197714+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.515261+0000","last_change":"2024-02-19T21:02:21.497900+0000","last_active":"2024-02-19T21:03:42.515261+0000","last_peered":"2024-02-19T21:03:42.515261+0000","last_clean":"2024-02-19T21:03:42.515261+0000","last_became_active":"2024-02-19T21:01:20.290687+0000","last_became_peered":"2024-02-19T21:01:20.290687+0000","last_unstale":"2024-02-19T21:03:42.515261+0000","last_undegraded":"2024-02-19T21:03:42.515261+0000","last_fullsized":"2024-02-19T21:03:42.515261+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.497831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.497831+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523720+0000","last_change":"2024-02-19T21:02:40.628725+0000","last_active":"2024-02-19T21:03:42.523720+0000","last_peered":"2024-02-19T21:03:42.523720+0000","last_clean":"2024-02-19T21:03:42.523720+0000","last_became_active":"2024-02-19T21:01:16.274937+0000","last_became_peered":"2024-02-19T21:01:16.274937+0000","last_unstale":"2024-02-19T21:03:42.523720+0000","last_undegraded":"2024-02-19T21:03:42.523720+0000","last_fullsized":"2024-02-19T21:03:42.523720+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:02:40.628632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.628632+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":98,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.508862+0000","last_change":"2024-02-19T21:02:30.070342+0000","last_active":"2024-02-19T21:03:42.508862+0000","last_peered":"2024-02-19T21:03:42.508862+0000","last_clean":"2024-02-19T21:03:42.508862+0000","last_became_active":"2024-02-19T21:01:30.923214+0000","last_became_peered":"2024-02-19T21:01:30.923214+0000","last_unstale":"2024-02-19T21:03:42.508862+0000","last_undegraded":"2024-02-19T21:03:42.508862+0000","last_fullsized":"2024-02-19T21:03:42.508862+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:02:30.070236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:30.070236+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524582+0000","last_change":"2024-02-19T21:03:03.500049+0000","last_active":"2024-02-19T21:03:42.524582+0000","last_peered":"2024-02-19T21:03:42.524582+0000","last_clean":"2024-02-19T21:03:42.524582+0000","last_became_active":"2024-02-19T21:01:22.291759+0000","last_became_peered":"2024-02-19T21:01:22.291759+0000","last_unstale":"2024-02-19T21:03:42.524582+0000","last_undegraded":"2024-02-19T21:03:42.524582+0000","last_fullsized":"2024-02-19T21:03:42.524582+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:03.499960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:03.499960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523614+0000","last_change":"2024-02-19T21:02:55.213761+0000","last_active":"2024-02-19T21:03:42.523614+0000","last_peered":"2024-02-19T21:03:42.523614+0000","last_clean":"2024-02-19T21:03:42.523614+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:03:42.523614+0000","last_undegraded":"2024-02-19T21:03:42.523614+0000","last_fullsized":"2024-02-19T21:03:42.523614+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:55.213692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.213692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.524180+0000","last_change":"2024-02-19T21:02:21.212907+0000","last_active":"2024-02-19T21:03:42.524180+0000","last_peered":"2024-02-19T21:03:42.524180+0000","last_clean":"2024-02-19T21:03:42.524180+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:03:42.524180+0000","last_undegraded":"2024-02-19T21:03:42.524180+0000","last_fullsized":"2024-02-19T21:03:42.524180+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:21.212808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:21.212808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.525634+0000","last_change":"2024-02-19T21:02:39.779974+0000","last_active":"2024-02-19T21:03:42.525634+0000","last_peered":"2024-02-19T21:03:42.525634+0000","last_clean":"2024-02-19T21:03:42.525634+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:03:42.525634+0000","last_undegraded":"2024-02-19T21:03:42.525634+0000","last_fullsized":"2024-02-19T21:03:42.525634+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:02:39.779890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.779890+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.527101+0000","last_change":"2024-02-19T21:02:48.572976+0000","last_active":"2024-02-19T21:03:42.527101+0000","last_peered":"2024-02-19T21:03:42.527101+0000","last_clean":"2024-02-19T21:03:42.527101+0000","last_became_active":"2024-02-19T21:01:22.297998+0000","last_became_peered":"2024-02-19T21:01:22.297998+0000","last_unstale":"2024-02-19T21:03:42.527101+0000","last_undegraded":"2024-02-19T21:03:42.527101+0000","last_fullsized":"2024-02-19T21:03:42.527101+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:48.572877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.572877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523455+0000","last_change":"2024-02-19T21:03:01.169350+0000","last_active":"2024-02-19T21:03:42.523455+0000","last_peered":"2024-02-19T21:03:42.523455+0000","last_clean":"2024-02-19T21:03:42.523455+0000","last_became_active":"2024-02-19T21:01:20.290566+0000","last_became_peered":"2024-02-19T21:01:20.290566+0000","last_unstale":"2024-02-19T21:03:42.523455+0000","last_undegraded":"2024-02-19T21:03:42.523455+0000","last_fullsized":"2024-02-19T21:03:42.523455+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.169279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.169279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.513553+0000","last_change":"2024-02-19T21:03:30.462938+0000","last_active":"2024-02-19T21:03:42.513553+0000","last_peered":"2024-02-19T21:03:42.513553+0000","last_clean":"2024-02-19T21:03:42.513553+0000","last_became_active":"2024-02-19T21:01:16.270931+0000","last_became_peered":"2024-02-19T21:01:16.270931+0000","last_unstale":"2024-02-19T21:03:42.513553+0000","last_undegraded":"2024-02-19T21:03:42.513553+0000","last_fullsized":"2024-02-19T21:03:42.513553+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:30.462868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:30.462868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":59,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.528160+0000","last_change":"2024-02-19T21:02:52.643899+0000","last_active":"2024-02-19T21:03:42.528160+0000","last_peered":"2024-02-19T21:03:42.528160+0000","last_clean":"2024-02-19T21:03:42.528160+0000","last_became_active":"2024-02-19T21:01:18.288322+0000","last_became_peered":"2024-02-19T21:01:18.288322+0000","last_unstale":"2024-02-19T21:03:42.528160+0000","last_undegraded":"2024-02-19T21:03:42.528160+0000","last_fullsized":"2024-02-19T21:03:42.528160+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:52.643835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.643835+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523728+0000","last_change":"2024-02-19T21:03:10.017843+0000","last_active":"2024-02-19T21:03:42.523728+0000","last_peered":"2024-02-19T21:03:42.523728+0000","last_clean":"2024-02-19T21:03:42.523728+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:03:42.523728+0000","last_undegraded":"2024-02-19T21:03:42.523728+0000","last_fullsized":"2024-02-19T21:03:42.523728+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:10.017764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:10.017764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523864+0000","last_change":"2024-02-19T21:02:37.258648+0000","last_active":"2024-02-19T21:03:42.523864+0000","last_peered":"2024-02-19T21:03:42.523864+0000","last_clean":"2024-02-19T21:03:42.523864+0000","last_became_active":"2024-02-19T21:01:16.271911+0000","last_became_peered":"2024-02-19T21:01:16.271911+0000","last_unstale":"2024-02-19T21:03:42.523864+0000","last_undegraded":"2024-02-19T21:03:42.523864+0000","last_fullsized":"2024-02-19T21:03:42.523864+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:37.258580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:37.258580+0000","last_clean_scrub_stamp":"2024-02-19T21:02:37.258580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-19T21:03:42.523349+0000","last_change":"2024-02-19T21:03:01.005530+0000","last_active":"2024-02-19T21:03:42.523349+0000","last_peered":"2024-02-19T21:03:42.523349+0000","last_clean":"2024-02-19T21:03:42.523349+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:03:42.523349+0000","last_undegraded":"2024-02-19T21:03:42.523349+0000","last_fullsized":"2024-02-19T21:03:42.523349+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:03:01.005445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.005445+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-02-19T21:03:54.454064+0000","last_change":"2024-02-19T21:02:44.563371+0000","last_active":"2024-02-19T21:03:54.454064+0000","last_peered":"2024-02-19T21:03:54.454064+0000","last_clean":"2024-02-19T21:03:54.454064+0000","last_became_active":"2024-02-19T21:01:20.288720+0000","last_became_peered":"2024-02-19T21:01:20.288720+0000","last_unstale":"2024-02-19T21:03:54.454064+0000","last_undegraded":"2024-02-19T21:03:54.454064+0000","last_fullsized":"2024-02-19T21:03:54.454064+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:44.563256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:44.563256+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":152,"num_read_kb":147,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266722,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297764,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447388,"statfs":{"total":95995035648,"available":95690125312,"internally_reserved":0,"allocated":684032,"data_stored":315411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.502,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.57499999999999996,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.432},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.32400000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63500000000000001,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.51300000000000001,"15min":0.52000000000000002},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78200000000000003,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.51300000000000001,"15min":0.51900000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.61499999999999999,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.36499999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.499,"15min":0.5},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.433,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70599999999999996,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.46300000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.39600000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.67800000000000005,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.505,"5min":0.496,"15min":0.49399999999999999},"min":{"1min":0.38700000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.47799999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.48599999999999999,"15min":0.498},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58199999999999996,"5min":1.373,"15min":1.373},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.748,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.46000000000000002,"15min":0.46100000000000002},"min":{"1min":0.31900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66800000000000004,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.56799999999999995}]},{"osd":6,"last update":"Mon Feb 19 21:03:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.34699999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.64700000000000002,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.56699999999999995,"15min":0.57599999999999996},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.76000000000000001,"5min":1.262,"15min":1.262},"last":0.62}]}]},{"osd":6,"up_from":45,"seq":193273528360,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297736,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447416,"statfs":{"total":95995035648,"available":95690153984,"internally_reserved":0,"allocated":655360,"data_stored":313875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.495,"15min":0.49399999999999999},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69399999999999995,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.38100000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.37,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.67500000000000004,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.505},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.501,"15min":0.5},"min":{"1min":0.34100000000000003,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.754,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.44700000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51200000000000001,"15min":0.51700000000000002},"min":{"1min":0.35299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.49299999999999999,"15min":0.49099999999999999},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.624,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.34699999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.41299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73899999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51400000000000001,"15min":0.5},"min":{"1min":0.41299999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.95299999999999996}]},{"osd":4,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.40999999999999998,"15min":0.38100000000000001},"min":{"1min":0.33900000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.628,"5min":1.363,"15min":1.363},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.42399999999999999,"15min":0.40600000000000003},"min":{"1min":0.311,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66400000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.42399999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.45000000000000001,"15min":0.41799999999999998},"min":{"1min":0.29799999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68000000000000005,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.47599999999999998,"15min":0.45200000000000001},"min":{"1min":0.33600000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69399999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.57699999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:03:16 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.38600000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67400000000000004,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65300000000000002,"5min":1.534,"15min":1.534},"last":0.45600000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297772,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447380,"statfs":{"total":95995035648,"available":95690117120,"internally_reserved":0,"allocated":692224,"data_stored":315502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35899999999999999,"15min":0.34799999999999998},"min":{"1min":0.22500000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":1.403},{"interface":"front","average":{"1min":0.373,"5min":0.38,"15min":0.35999999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.64400000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:03:14 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.40300000000000002,"15min":0.40600000000000003},"min":{"1min":0.27500000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.499,"5min":0.62,"15min":0.62},"last":1.3520000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.40300000000000002,"15min":0.41499999999999998},"min":{"1min":0.26900000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.52800000000000002,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.58299999999999996}]},{"osd":3,"last update":"Mon Feb 19 21:03:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40799999999999997,"15min":0.42199999999999999},"min":{"1min":0.26400000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.438,"15min":0.45700000000000002},"min":{"1min":0.29099999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.51500000000000001,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":1.4810000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55300000000000005,"15min":0.55700000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.68300000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71699999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":1.381}]},{"osd":5,"last update":"Mon Feb 19 21:03:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.58299999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71699999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.42599999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.56299999999999994}]},{"osd":6,"last update":"Mon Feb 19 21:03:49 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.64000000000000001},"min":{"1min":0.41299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.878,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":1.429}]},{"osd":7,"last update":"Mon Feb 19 21:03:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58199999999999996,"15min":0.58499999999999996},"min":{"1min":0.45700000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.47699999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.747,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":1.4570000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297764,"kb_used_data":668,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447388,"statfs":{"total":95995035648,"available":95690125312,"internally_reserved":0,"allocated":684032,"data_stored":315065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.372,"15min":0.315},"min":{"1min":0.21299999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.46899999999999997,"5min":4.2830000000000004,"15min":4.2830000000000004},"last":0.317},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.33300000000000002,"15min":0.308},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.48299999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.25600000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.39600000000000002,"15min":0.378},"min":{"1min":0.222,"5min":0.219,"15min":0.219},"max":{"1min":0.45300000000000001,"5min":3.121,"15min":3.121},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.42499999999999999,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.497,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.375}]},{"osd":3,"last update":"Mon Feb 19 21:03:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.46000000000000002,"15min":0.44700000000000001},"min":{"1min":0.26700000000000002,"5min":0.249,"15min":0.249},"max":{"1min":1.028,"5min":4.4470000000000001,"15min":4.4470000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.47499999999999998,"15min":0.45800000000000002},"min":{"1min":0.28199999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.95299999999999996,"5min":4.3070000000000004,"15min":4.3070000000000004},"last":0.41699999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:02:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52600000000000002,"15min":0.51800000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55400000000000005,"15min":0.54000000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73999999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.502}]},{"osd":5,"last update":"Mon Feb 19 21:03:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.621,"15min":0.63900000000000001},"min":{"1min":0.41799999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.79500000000000004,"5min":1.581,"15min":1.581},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.75,"15min":0.81599999999999995},"min":{"1min":0.435,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.2290000000000001,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.40100000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:03:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.36699999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72799999999999998,"5min":1.381,"15min":1.381},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85399999999999998,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.53000000000000003}]},{"osd":7,"last update":"Mon Feb 19 21:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.76900000000000002,"15min":0.79900000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.2070000000000001,"5min":4.226,"15min":4.226},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.625,"15min":0.63100000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.218,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.47399999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297752,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":671744,"data_stored":314653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.36199999999999999,"15min":0.35399999999999998},"min":{"1min":0.23000000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.55500000000000005,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.35399999999999998,"15min":0.34599999999999997},"min":{"1min":0.252,"5min":0.214,"15min":0.214},"max":{"1min":0.40300000000000002,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.48399999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.35299999999999998},"min":{"1min":0.23300000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.442},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40500000000000003,"15min":0.39600000000000002},"min":{"1min":0.28799999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.45500000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:03:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.437,"15min":0.45300000000000001},"min":{"1min":0.28599999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.45000000000000001,"15min":0.46999999999999997},"min":{"1min":0.28499999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.27600000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.58099999999999996,"15min":0.59999999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71899999999999997,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.60899999999999999,"15min":0.64300000000000002},"min":{"1min":0.38400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.74199999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.58999999999999997}]},{"osd":5,"last update":"Mon Feb 19 21:03:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.61199999999999999,"15min":0.64700000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85199999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.625,"15min":0.64800000000000002},"min":{"1min":0.45600000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:02:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.62,"15min":0.629},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77800000000000002,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.622,"15min":0.63200000000000001},"min":{"1min":0.41899999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.76300000000000001,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.60499999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.58999999999999997,"15min":0.58799999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.60999999999999999,"15min":0.60799999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.51100000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297776,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447376,"statfs":{"total":95995035648,"available":95690113024,"internally_reserved":0,"allocated":696320,"data_stored":314678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:41 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.34799999999999998,"15min":0.35199999999999998},"min":{"1min":0.218,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.47299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.377,"15min":0.39800000000000002},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.50700000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.36899999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:03:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.374,"15min":0.39200000000000002},"min":{"1min":0.23300000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.47399999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.38100000000000001,"15min":0.39500000000000002},"min":{"1min":0.25700000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.45700000000000002,"5min":1.054,"15min":1.054},"last":0.252}]},{"osd":2,"last update":"Mon Feb 19 21:03:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40400000000000003,"15min":0.41799999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.48399999999999999,"5min":0.625,"15min":0.625},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.39400000000000002,"15min":0.41399999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.50700000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.433}]},{"osd":4,"last update":"Mon Feb 19 21:02:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.41199999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82799999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55700000000000005,"15min":0.56599999999999995},"min":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.83399999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.64600000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55100000000000005,"15min":0.56000000000000005},"min":{"1min":0.38500000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.68100000000000005}]},{"osd":6,"last update":"Mon Feb 19 21:03:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55700000000000005,"15min":0.56699999999999995},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.69999999999999996,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.624},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58399999999999996,"15min":0.59199999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.65700000000000003}]},{"osd":7,"last update":"Mon Feb 19 21:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56200000000000006,"15min":0.56100000000000005},"min":{"1min":0.39900000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56799999999999995,"15min":0.57099999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.72899999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986226,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297752,"kb_used_data":656,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447400,"statfs":{"total":95995035648,"available":95690137600,"internally_reserved":0,"allocated":671744,"data_stored":308327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.52500000000000002,"15min":0.54800000000000004},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.67800000000000005,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.50700000000000001,"15min":0.52300000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.59899999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.36699999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.754,"5min":2.149,"15min":2.149},"last":0.441},{"interface":"front","average":{"1min":0.436,"5min":0.502,"15min":0.51900000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.57799999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.64200000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.53300000000000003,"15min":0.55700000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.66400000000000003,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.42299999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.51700000000000002,"15min":0.52500000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.56499999999999995,"15min":0.56999999999999995},"min":{"1min":0.33400000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64200000000000002,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":0.501}]},{"osd":5,"last update":"Mon Feb 19 21:03:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.42399999999999999,"15min":0.42999999999999999},"min":{"1min":0.30099999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.68100000000000005,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.438,"15min":0.434},"min":{"1min":0.35299999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66100000000000003,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.39100000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:03:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.41399999999999998,"15min":0.38200000000000001},"min":{"1min":0.38100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.433},"min":{"1min":0.45400000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.54500000000000004}]},{"osd":7,"last update":"Mon Feb 19 21:03:12 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.442,"15min":0.432},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.46800000000000003,"15min":0.45900000000000002},"min":{"1min":0.40600000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.56699999999999995}]}]},{"osd":5,"up_from":37,"seq":158913789997,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297740,"kb_used_data":644,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447412,"statfs":{"total":95995035648,"available":95690149888,"internally_reserved":0,"allocated":659456,"data_stored":313146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.495,"15min":0.51000000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.65200000000000002,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.51500000000000001,"15min":0.53700000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.66900000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.505,"15min":0.52000000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.57899999999999996,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51500000000000001,"15min":0.52900000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68899999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.48899999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.51700000000000002,"15min":0.53500000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60299999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51900000000000002,"15min":0.53100000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69099999999999995,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.503}]},{"osd":3,"last update":"Mon Feb 19 21:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51800000000000002,"15min":0.53200000000000003},"min":{"1min":0.37,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.63500000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66100000000000003,"5min":0.747,"15min":0.747},"last":0.54500000000000004}]},{"osd":4,"last update":"Mon Feb 19 21:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40300000000000002,"15min":0.39000000000000001},"min":{"1min":0.28000000000000003,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.58999999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.41299999999999998,"15min":0.39000000000000001},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.59599999999999997}]},{"osd":6,"last update":"Mon Feb 19 21:03:47 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.442,"15min":0.42299999999999999},"min":{"1min":0.33500000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.443},{"interface":"front","average":{"1min":0.502,"5min":0.45600000000000002,"15min":0.437},"min":{"1min":0.33400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.623,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.626}]},{"osd":7,"last update":"Mon Feb 19 21:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.503,"15min":0.502},"min":{"1min":0.35699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.63100000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.50800000000000001,"15min":0.505},"min":{"1min":0.36599999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.66000000000000003,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.55600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:04:03.746 INFO:tasks.thrashosds.thrasher:Setting 2.d to [1, 4, 2] 2024-02-19T21:04:03.746 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.d', '1', '4', '2'] 2024-02-19T21:04:03.746 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pg-upmap 2.d 1 4 2 2024-02-19T21:04:03.782 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:04:03.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:02.575553+0000 mon.a (mon.0) 728 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:04:03.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:02.579681+0000 mon.a (mon.0) 729 : audit [INF] from='client.? v1:172.21.15.163:0/799236926' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:04:03.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:02.579739+0000 mon.a (mon.0) 730 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-02-19T21:04:03.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:02.780387+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:04:03.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:02.781550+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:04:03.807 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:03.077306+0000 osd.5 (osd.5) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:03.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:03.077384+0000 osd.5 (osd.5) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:03.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:03.150723+0000 osd.4 (osd.4) 51 : cluster [DBG] 4.d scrub starts 2024-02-19T21:04:03.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:03.152811+0000 osd.4 (osd.4) 52 : cluster [DBG] 4.d scrub ok 2024-02-19T21:04:03.809 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:03.827 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:04:03.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:03.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:04:03.864 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:02.575553+0000 mon.a (mon.0) 728 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:04:03.864 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:02.579681+0000 mon.a (mon.0) 729 : audit [INF] from='client.? v1:172.21.15.163:0/799236926' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:04:03.864 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:02.579739+0000 mon.a (mon.0) 730 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-02-19T21:04:03.864 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:02.780387+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:04:03.865 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:02.781550+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:04:03.865 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:03.077306+0000 osd.5 (osd.5) 43 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:03.865 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:03.077384+0000 osd.5 (osd.5) 44 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:03.865 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:03.150723+0000 osd.4 (osd.4) 51 : cluster [DBG] 4.d scrub starts 2024-02-19T21:04:03.865 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:03.152811+0000 osd.4 (osd.4) 52 : cluster [DBG] 4.d scrub ok 2024-02-19T21:04:03.931 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:04:03.958 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:03.976 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:04:03.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:03.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:04:04.081 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:04:04.118 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:04.226 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:04:04.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:04.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:04:04.330 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:04:04.357 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:04.376 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:04:04.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:04.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:04:04.479 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:04.516 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:04.547 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:04.563 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:04.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:04.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:04.611 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:04:04.611 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-02-19T21:04:04.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-02-19T21:04:04.611 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-02-19T21:04:04.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.611 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.34382.0:57 5.15 e62/61 5:a93a5511:::notify.2:head v 62'1, mlcod=0'0)", 2024-02-19T21:04:04.611 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:20.297991+0000", 2024-02-19T21:04:04.611 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 164.31424121500001, 2024-02-19T21:04:04.611 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002747598, 2024-02-19T21:04:04.611 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.611 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.612 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.612 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.297991+0000", 2024-02-19T21:04:04.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.612 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.297991+0000", 2024-02-19T21:04:04.612 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:04:04.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.612 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.297989+0000", 2024-02-19T21:04:04.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3156000000000001e-05 2024-02-19T21:04:04.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.298002+0000", 2024-02-19T21:04:04.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7439e-05 2024-02-19T21:04:04.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.613 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.613 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.298019+0000", 2024-02-19T21:04:04.613 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4839999999999993e-06 2024-02-19T21:04:04.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.614 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.614 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.298029+0000", 2024-02-19T21:04:04.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4378e-05 2024-02-19T21:04:04.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.614 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.614 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.298083+0000", 2024-02-19T21:04:04.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3189000000000001e-05 2024-02-19T21:04:04.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.614 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.614 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.298106+0000", 2024-02-19T21:04:04.614 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0025863330000000001 2024-02-19T21:04:04.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.615 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.300693+0000", 2024-02-19T21:04:04.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5989000000000003e-05 2024-02-19T21:04:04.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.615 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.615 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.300738+0000", 2024-02-19T21:04:04.615 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002747598 2024-02-19T21:04:04.615 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.615 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.615 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.615 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.34382.0:64 5.3 5:c609908c:::notify.5:head [watch watch cookie 94067146299952] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-19T21:04:04.616 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:20.305155+0000", 2024-02-19T21:04:04.616 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 164.307076857, 2024-02-19T21:04:04.616 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001374412, 2024-02-19T21:04:04.616 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.616 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.616 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:04.616 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.34382", 2024-02-19T21:04:04.616 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2202902387", 2024-02-19T21:04:04.616 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 64 2024-02-19T21:04:04.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.616 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.616 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.616 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305155+0000", 2024-02-19T21:04:04.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305155+0000", 2024-02-19T21:04:04.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:04:04.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.617 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305153+0000", 2024-02-19T21:04:04.617 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0520000000000004e-06 2024-02-19T21:04:04.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.617 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.618 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305160+0000", 2024-02-19T21:04:04.618 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1044e-05 2024-02-19T21:04:04.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.618 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.618 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305171+0000", 2024-02-19T21:04:04.618 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4100000000000002e-06 2024-02-19T21:04:04.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.618 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.618 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305179+0000", 2024-02-19T21:04:04.618 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0627000000000001e-05 2024-02-19T21:04:04.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.619 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.619 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305239+0000", 2024-02-19T21:04:04.619 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9283000000000006e-05 2024-02-19T21:04:04.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.619 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-02-19T21:04:04.619 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305329+0000", 2024-02-19T21:04:04.619 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018149999999999999 2024-02-19T21:04:04.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.619 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.619 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305510+0000", 2024-02-19T21:04:04.619 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6387e-05 2024-02-19T21:04:04.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305577+0000", 2024-02-19T21:04:04.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016600499999999999 2024-02-19T21:04:04.620 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.620 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-02-19T21:04:04.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305743+0000", 2024-02-19T21:04:04.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00050379199999999998 2024-02-19T21:04:04.620 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.620 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.620 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:04.620 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.306246+0000", 2024-02-19T21:04:04.620 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000119929 2024-02-19T21:04:04.620 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.306366+0000", 2024-02-19T21:04:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4347e-05 2024-02-19T21:04:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.306421+0000", 2024-02-19T21:04:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8727999999999998e-05 2024-02-19T21:04:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.306469+0000", 2024-02-19T21:04:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0242999999999999e-05 2024-02-19T21:04:04.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.306530+0000", 2024-02-19T21:04:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001374412 2024-02-19T21:04:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.34352.0:102 5.3 5:c609908c:::notify.5:head [watch watch cookie 94681540392576] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-19T21:04:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:20.305645+0000", 2024-02-19T21:04:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 164.30658737100001, 2024-02-19T21:04:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00188138, 2024-02-19T21:04:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.622 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.34352", 2024-02-19T21:04:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:04:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 102 2024-02-19T21:04:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305645+0000", 2024-02-19T21:04:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.623 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305645+0000", 2024-02-19T21:04:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305643+0000", 2024-02-19T21:04:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0360000000000003e-06 2024-02-19T21:04:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305649+0000", 2024-02-19T21:04:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9330000000000007e-06 2024-02-19T21:04:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.624 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305658+0000", 2024-02-19T21:04:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0699999999999997e-06 2024-02-19T21:04:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305663+0000", 2024-02-19T21:04:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000312221 2024-02-19T21:04:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.305975+0000", 2024-02-19T21:04:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5775e-05 2024-02-19T21:04:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-02-19T21:04:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.306041+0000", 2024-02-19T21:04:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00053208400000000005 2024-02-19T21:04:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.306573+0000", 2024-02-19T21:04:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9234000000000007e-05 2024-02-19T21:04:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.306663+0000", 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012673 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.306789+0000", 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030942399999999998 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.307099+0000", 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000271456 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.627 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.307370+0000", 2024-02-19T21:04:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1264000000000002e-05 2024-02-19T21:04:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.307421+0000", 2024-02-19T21:04:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9607999999999997e-05 2024-02-19T21:04:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.307471+0000", 2024-02-19T21:04:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5101e-05 2024-02-19T21:04:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.307526+0000", 2024-02-19T21:04:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00188138 2024-02-19T21:04:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:4 4.b 4:d242335b:gc::gc.2:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.291285+0000", 2024-02-19T21:04:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 161.32094758599999, 2024-02-19T21:04:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001331273, 2024-02-19T21:04:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.629 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 4 2024-02-19T21:04:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.291285+0000", 2024-02-19T21:04:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.630 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.291285+0000", 2024-02-19T21:04:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:04:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.291281+0000", 2024-02-19T21:04:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1036000000000001e-05 2024-02-19T21:04:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.291293+0000", 2024-02-19T21:04:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3210999999999999e-05 2024-02-19T21:04:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.631 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.291306+0000", 2024-02-19T21:04:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3469999999999999e-06 2024-02-19T21:04:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.291314+0000", 2024-02-19T21:04:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9110000000000002e-05 2024-02-19T21:04:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.291363+0000", 2024-02-19T21:04:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010644199999999999 2024-02-19T21:04:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.632 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.291470+0000", 2024-02-19T21:04:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00026153899999999998 2024-02-19T21:04:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,4", 2024-02-19T21:04:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.291731+0000", 2024-02-19T21:04:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00053336200000000001 2024-02-19T21:04:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.292265+0000", 2024-02-19T21:04:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019106599999999999 2024-02-19T21:04:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.633 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.292456+0000", 2024-02-19T21:04:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0385999999999999e-05 2024-02-19T21:04:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.292536+0000", 2024-02-19T21:04:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9410000000000003e-05 2024-02-19T21:04:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.292585+0000", 2024-02-19T21:04:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0392e-05 2024-02-19T21:04:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.292616+0000", 2024-02-19T21:04:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001331273 2024-02-19T21:04:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.24400.0:57 4.1a e65/59 4:5909e671:gc::gc.27:head v 65'3, mlcod=65'1)", 2024-02-19T21:04:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.343272+0000", 2024-02-19T21:04:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 161.26896045300001, 2024-02-19T21:04:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020378150000000001, 2024-02-19T21:04:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.635 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.343272+0000", 2024-02-19T21:04:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.343272+0000", 2024-02-19T21:04:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:04:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.343270+0000", 2024-02-19T21:04:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0943e-05 2024-02-19T21:04:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.343281+0000", 2024-02-19T21:04:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4807000000000001e-05 2024-02-19T21:04:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.343296+0000", 2024-02-19T21:04:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2280000000000001e-06 2024-02-19T21:04:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.343303+0000", 2024-02-19T21:04:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7355000000000002e-05 2024-02-19T21:04:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.343360+0000", 2024-02-19T21:04:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9936999999999999e-05 2024-02-19T21:04:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.343380+0000", 2024-02-19T21:04:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018982109999999999 2024-02-19T21:04:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.345278+0000", 2024-02-19T21:04:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1127000000000002e-05 2024-02-19T21:04:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.345309+0000", 2024-02-19T21:04:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020378150000000001 2024-02-19T21:04:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.639 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:75 4.3 4:c3a682d4:reshard::reshard.0000000004:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.351097+0000", 2024-02-19T21:04:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 161.26113504700001, 2024-02-19T21:04:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013642789999999999, 2024-02-19T21:04:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 75 2024-02-19T21:04:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.640 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.351097+0000", 2024-02-19T21:04:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.351097+0000", 2024-02-19T21:04:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:04:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.351095+0000", 2024-02-19T21:04:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0459999999999994e-06 2024-02-19T21:04:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.351104+0000", 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3331e-05 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.351117+0000", 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8120000000000002e-06 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.351124+0000", 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9283000000000002e-05 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.351174+0000", 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012208799999999999 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.351296+0000", 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000175506 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,7", 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.351471+0000", 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00084158400000000004 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.352313+0000", 2024-02-19T21:04:04.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7861999999999999e-05 2024-02-19T21:04:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.352331+0000", 2024-02-19T21:04:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4345000000000003e-05 2024-02-19T21:04:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.352375+0000", 2024-02-19T21:04:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9128999999999997e-05 2024-02-19T21:04:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.352424+0000", 2024-02-19T21:04:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7243000000000002e-05 2024-02-19T21:04:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.352461+0000", 2024-02-19T21:04:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013642789999999999 2024-02-19T21:04:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:84 4.6 4:66924383:reshard::reshard.0000000005:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.354639+0000", 2024-02-19T21:04:04.645 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 161.25759314699999, 2024-02-19T21:04:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0017789150000000001, 2024-02-19T21:04:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 84 2024-02-19T21:04:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354639+0000", 2024-02-19T21:04:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354639+0000", 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354637+0000", 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0080000000000008e-06 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354646+0000", 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3969e-05 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354660+0000", 2024-02-19T21:04:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7580000000000003e-06 2024-02-19T21:04:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354667+0000", 2024-02-19T21:04:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1893e-05 2024-02-19T21:04:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354709+0000", 2024-02-19T21:04:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010564399999999999 2024-02-19T21:04:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354815+0000", 2024-02-19T21:04:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019095900000000001 2024-02-19T21:04:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,2", 2024-02-19T21:04:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.355005+0000", 2024-02-19T21:04:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00098326300000000006 2024-02-19T21:04:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.355989+0000", 2024-02-19T21:04:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3359999999999997e-06 2024-02-19T21:04:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.355996+0000", 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00033548300000000003 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.356332+0000", 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6428999999999999e-05 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.356378+0000", 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9907000000000003e-05 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.356418+0000", 2024-02-19T21:04:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0017789150000000001 2024-02-19T21:04:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:88 4.6 4:66924383:reshard::reshard.0000000005:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.357192+0000", 2024-02-19T21:04:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 161.25503993699999, 2024-02-19T21:04:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013833560000000001, 2024-02-19T21:04:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.651 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 88 2024-02-19T21:04:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357192+0000", 2024-02-19T21:04:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357192+0000", 2024-02-19T21:04:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357191+0000", 2024-02-19T21:04:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2819999999999999e-06 2024-02-19T21:04:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357198+0000", 2024-02-19T21:04:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.481e-06 2024-02-19T21:04:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357206+0000", 2024-02-19T21:04:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1690000000000002e-06 2024-02-19T21:04:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357210+0000", 2024-02-19T21:04:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2955999999999998e-05 2024-02-19T21:04:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357273+0000", 2024-02-19T21:04:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1359999999999994e-05 2024-02-19T21:04:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357344+0000", 2024-02-19T21:04:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017449499999999999 2024-02-19T21:04:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,2", 2024-02-19T21:04:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357519+0000", 2024-02-19T21:04:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00063798999999999995 2024-02-19T21:04:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.358157+0000", 2024-02-19T21:04:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000164277 2024-02-19T21:04:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.358321+0000", 2024-02-19T21:04:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000181187 2024-02-19T21:04:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.358502+0000", 2024-02-19T21:04:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3575000000000003e-05 2024-02-19T21:04:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.358536+0000", 2024-02-19T21:04:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9926999999999999e-05 2024-02-19T21:04:04.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.358576+0000", 2024-02-19T21:04:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013833560000000001 2024-02-19T21:04:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:99 4.16 4:6a988a9b:reshard::reshard.0000000007:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.361793+0000", 2024-02-19T21:04:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 161.250438694, 2024-02-19T21:04:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024377980000000001, 2024-02-19T21:04:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.657 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 99 2024-02-19T21:04:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361793+0000", 2024-02-19T21:04:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361793+0000", 2024-02-19T21:04:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:04:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361792+0000", 2024-02-19T21:04:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0849999999999997e-06 2024-02-19T21:04:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361800+0000", 2024-02-19T21:04:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.252e-05 2024-02-19T21:04:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.659 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361812+0000", 2024-02-19T21:04:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7989999999999996e-06 2024-02-19T21:04:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361819+0000", 2024-02-19T21:04:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2968e-05 2024-02-19T21:04:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361852+0000", 2024-02-19T21:04:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1804000000000004e-05 2024-02-19T21:04:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361944+0000", 2024-02-19T21:04:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015612199999999999 2024-02-19T21:04:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-02-19T21:04:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.362100+0000", 2024-02-19T21:04:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00066182100000000005 2024-02-19T21:04:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.362762+0000", 2024-02-19T21:04:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020385100000000001 2024-02-19T21:04:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.362966+0000", 2024-02-19T21:04:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012093799999999999 2024-02-19T21:04:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.364175+0000", 2024-02-19T21:04:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3618000000000002e-05 2024-02-19T21:04:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.364209+0000", 2024-02-19T21:04:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2694e-05 2024-02-19T21:04:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.364231+0000", 2024-02-19T21:04:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024377980000000001 2024-02-19T21:04:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:105 4.16 4:6a988a9b:reshard::reshard.0000000007:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.364932+0000", 2024-02-19T21:04:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 161.24730012000001, 2024-02-19T21:04:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015032789999999999, 2024-02-19T21:04:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.663 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 105 2024-02-19T21:04:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.364932+0000", 2024-02-19T21:04:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.364932+0000", 2024-02-19T21:04:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.364931+0000", 2024-02-19T21:04:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1430000000000003e-06 2024-02-19T21:04:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.364937+0000", 2024-02-19T21:04:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0513000000000001e-05 2024-02-19T21:04:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.364947+0000", 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3864e-05 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.364961+0000", 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8472e-05 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.365000+0000", 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0494000000000006e-05 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.365070+0000", 2024-02-19T21:04:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019557400000000001 2024-02-19T21:04:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-02-19T21:04:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.365266+0000", 2024-02-19T21:04:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00046623200000000002 2024-02-19T21:04:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.365732+0000", 2024-02-19T21:04:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2222999999999998e-05 2024-02-19T21:04:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.667 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.365784+0000", 2024-02-19T21:04:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00058620500000000002 2024-02-19T21:04:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.366370+0000", 2024-02-19T21:04:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2839000000000001e-05 2024-02-19T21:04:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.366413+0000", 2024-02-19T21:04:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2152000000000001e-05 2024-02-19T21:04:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.366435+0000", 2024-02-19T21:04:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015032789999999999 2024-02-19T21:04:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:145 4.b 4:d242335b:gc::gc.2:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.379482+0000", 2024-02-19T21:04:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 161.232750127, 2024-02-19T21:04:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001871827, 2024-02-19T21:04:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.669 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 145 2024-02-19T21:04:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.379482+0000", 2024-02-19T21:04:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.379482+0000", 2024-02-19T21:04:04.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:04:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.379481+0000", 2024-02-19T21:04:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1741e-05 2024-02-19T21:04:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.379493+0000", 2024-02-19T21:04:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.678e-06 2024-02-19T21:04:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.379500+0000", 2024-02-19T21:04:04.671 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8449999999999992e-06 2024-02-19T21:04:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.379509+0000", 2024-02-19T21:04:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1999999999999999e-05 2024-02-19T21:04:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.379541+0000", 2024-02-19T21:04:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3707999999999994e-05 2024-02-19T21:04:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.379605+0000", 2024-02-19T21:04:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000172039 2024-02-19T21:04:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,4", 2024-02-19T21:04:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.379777+0000", 2024-02-19T21:04:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00031396400000000002 2024-02-19T21:04:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.380091+0000", 2024-02-19T21:04:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00037085599999999997 2024-02-19T21:04:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.380462+0000", 2024-02-19T21:04:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00083914099999999995 2024-02-19T21:04:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.381301+0000", 2024-02-19T21:04:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1306999999999997e-05 2024-02-19T21:04:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.381332+0000", 2024-02-19T21:04:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1489e-05 2024-02-19T21:04:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.381354+0000", 2024-02-19T21:04:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001871827 2024-02-19T21:04:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:238 4.b 4:d201668e:::obj_delete_at_hint.0000000023:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.417449+0000", 2024-02-19T21:04:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 161.19478316799999, 2024-02-19T21:04:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001369959, 2024-02-19T21:04:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.675 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 238 2024-02-19T21:04:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.417449+0000", 2024-02-19T21:04:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.417449+0000", 2024-02-19T21:04:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:04:04.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.417447+0000", 2024-02-19T21:04:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4109999999999999e-06 2024-02-19T21:04:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.417455+0000", 2024-02-19T21:04:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7909999999999999e-06 2024-02-19T21:04:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.417464+0000", 2024-02-19T21:04:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0200000000000002e-06 2024-02-19T21:04:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.417469+0000", 2024-02-19T21:04:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000107477 2024-02-19T21:04:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.417577+0000", 2024-02-19T21:04:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000117039 2024-02-19T21:04:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.417694+0000", 2024-02-19T21:04:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000279086 2024-02-19T21:04:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,4", 2024-02-19T21:04:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.417973+0000", 2024-02-19T21:04:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00028998799999999998 2024-02-19T21:04:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.418263+0000", 2024-02-19T21:04:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00038420099999999998 2024-02-19T21:04:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.418647+0000", 2024-02-19T21:04:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8982999999999998e-05 2024-02-19T21:04:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.418736+0000", 2024-02-19T21:04:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6159999999999999e-05 2024-02-19T21:04:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.418782+0000", 2024-02-19T21:04:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6488e-05 2024-02-19T21:04:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.418819+0000", 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001369959 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:279 4.3 4:c623b173:gc::gc.16:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.434757+0000", 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 161.17747537700001, 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021730149999999999, 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:04.681 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 279 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.434757+0000", 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.434757+0000", 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.434755+0000", 2024-02-19T21:04:04.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4700000000000001e-06 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.434761+0000", 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7059999999999993e-06 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.434770+0000", 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1962e-05 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.434782+0000", 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1117e-05 2024-02-19T21:04:04.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.434813+0000", 2024-02-19T21:04:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1461999999999999e-05 2024-02-19T21:04:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.434904+0000", 2024-02-19T21:04:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016347399999999999 2024-02-19T21:04:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,7", 2024-02-19T21:04:04.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.435068+0000", 2024-02-19T21:04:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025502899999999999 2024-02-19T21:04:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.435323+0000", 2024-02-19T21:04:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00027088399999999999 2024-02-19T21:04:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.435594+0000", 2024-02-19T21:04:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012876980000000001 2024-02-19T21:04:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.436881+0000", 2024-02-19T21:04:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7188000000000001e-05 2024-02-19T21:04:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.436908+0000", 2024-02-19T21:04:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1274e-05 2024-02-19T21:04:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.436930+0000", 2024-02-19T21:04:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021730149999999999 2024-02-19T21:04:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.686 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:288 4.16 4:6849fd06:gc::gc.17:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.440521+0000", 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 161.171710953, 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001621471, 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 288 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.440521+0000", 2024-02-19T21:04:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.440521+0000", 2024-02-19T21:04:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:04:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.440520+0000", 2024-02-19T21:04:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5520000000000004e-06 2024-02-19T21:04:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.440526+0000", 2024-02-19T21:04:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9870000000000002e-06 2024-02-19T21:04:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.440533+0000", 2024-02-19T21:04:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.794e-06 2024-02-19T21:04:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.440536+0000", 2024-02-19T21:04:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1010000000000003e-05 2024-02-19T21:04:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.440567+0000", 2024-02-19T21:04:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6656000000000006e-05 2024-02-19T21:04:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.440644+0000", 2024-02-19T21:04:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017984300000000001 2024-02-19T21:04:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,7", 2024-02-19T21:04:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.440824+0000", 2024-02-19T21:04:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00067994599999999996 2024-02-19T21:04:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.441504+0000", 2024-02-19T21:04:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00021554000000000001 2024-02-19T21:04:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.441719+0000", 2024-02-19T21:04:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00034970900000000002 2024-02-19T21:04:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.442069+0000", 2024-02-19T21:04:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1171000000000003e-05 2024-02-19T21:04:04.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.442110+0000", 2024-02-19T21:04:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2301000000000001e-05 2024-02-19T21:04:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.442143+0000", 2024-02-19T21:04:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001621471 2024-02-19T21:04:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.24400.0:294 4.1a e65/59 4:5ae4284b:::obj_delete_at_hint.0000000032:head v 65'6, mlcod=65'4)", 2024-02-19T21:04:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.445736+0000", 2024-02-19T21:04:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 161.166495989, 2024-02-19T21:04:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032415450000000002, 2024-02-19T21:04:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.445736+0000", 2024-02-19T21:04:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.445736+0000", 2024-02-19T21:04:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.445735+0000", 2024-02-19T21:04:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0334e-05 2024-02-19T21:04:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.445745+0000", 2024-02-19T21:04:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6455000000000001e-05 2024-02-19T21:04:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.445761+0000", 2024-02-19T21:04:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4750000000000001e-06 2024-02-19T21:04:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.445767+0000", 2024-02-19T21:04:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7163000000000002e-05 2024-02-19T21:04:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.445814+0000", 2024-02-19T21:04:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9539e-05 2024-02-19T21:04:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.445833+0000", 2024-02-19T21:04:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031103989999999998 2024-02-19T21:04:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.448944+0000", 2024-02-19T21:04:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3797000000000002e-05 2024-02-19T21:04:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.448978+0000", 2024-02-19T21:04:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032415450000000002 2024-02-19T21:04:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.696 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.24400.0:306 4.7 e65/59 4:e622f76a:gc::gc.20:head v 65'10, mlcod=65'8)", 2024-02-19T21:04:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.454013+0000", 2024-02-19T21:04:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 161.158219444, 2024-02-19T21:04:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002101608, 2024-02-19T21:04:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454013+0000", 2024-02-19T21:04:04.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454013+0000", 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454011+0000", 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0278000000000001e-05 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454021+0000", 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4664000000000001e-05 2024-02-19T21:04:04.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454036+0000", 2024-02-19T21:04:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0160000000000003e-06 2024-02-19T21:04:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454043+0000", 2024-02-19T21:04:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6957999999999999e-05 2024-02-19T21:04:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454100+0000", 2024-02-19T21:04:04.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5361000000000001e-05 2024-02-19T21:04:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454115+0000", 2024-02-19T21:04:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019646109999999998 2024-02-19T21:04:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.456080+0000", 2024-02-19T21:04:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4518999999999999e-05 2024-02-19T21:04:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.456114+0000", 2024-02-19T21:04:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002101608 2024-02-19T21:04:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24493.0:2 3.1 3:90ab8ff9:::rbd_task:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected e72)", 2024-02-19T21:04:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:39.077446+0000", 2024-02-19T21:04:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 145.534786056, 2024-02-19T21:04:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001699742, 2024-02-19T21:04:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:04:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:04.701 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24493", 2024-02-19T21:04:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2351957517", 2024-02-19T21:04:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:04:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.077446+0000", 2024-02-19T21:04:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.077446+0000", 2024-02-19T21:04:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:04:04.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.077444+0000", 2024-02-19T21:04:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6389999999999992e-06 2024-02-19T21:04:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.077451+0000", 2024-02-19T21:04:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6719999999999993e-06 2024-02-19T21:04:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.077460+0000", 2024-02-19T21:04:04.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8130000000000001e-06 2024-02-19T21:04:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.077466+0000", 2024-02-19T21:04:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7800999999999999e-05 2024-02-19T21:04:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.077514+0000", 2024-02-19T21:04:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6689999999999997e-06 2024-02-19T21:04:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:04:04.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.077518+0000", 2024-02-19T21:04:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015491820000000001 2024-02-19T21:04:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.079067+0000", 2024-02-19T21:04:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8435000000000004e-05 2024-02-19T21:04:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.079146+0000", 2024-02-19T21:04:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001699742 2024-02-19T21:04:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.705 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.706 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.34352.0:501 5.3 5:c609908c:::notify.5:head [watch ping cookie 94681540392576] snapc 0=[] ondisk+write+known_if_redirected e77)", 2024-02-19T21:04:04.706 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:02:19.433530+0000", 2024-02-19T21:04:04.706 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 105.17870245500001, 2024-02-19T21:04:04.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002245567, 2024-02-19T21:04:04.706 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.706 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:04:04.706 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:04.706 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.34352", 2024-02-19T21:04:04.706 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:04:04.706 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 501 2024-02-19T21:04:04.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.706 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:02:19.433530+0000", 2024-02-19T21:04:04.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:02:19.433530+0000", 2024-02-19T21:04:04.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999924 2024-02-19T21:04:04.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:02:19.433522+0000", 2024-02-19T21:04:04.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5729999999999999e-05 2024-02-19T21:04:04.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:02:19.433548+0000", 2024-02-19T21:04:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3291999999999997e-05 2024-02-19T21:04:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:02:19.433601+0000", 2024-02-19T21:04:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3473000000000003e-05 2024-02-19T21:04:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:02:19.433634+0000", 2024-02-19T21:04:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6590999999999998e-05 2024-02-19T21:04:04.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:02:19.433721+0000", 2024-02-19T21:04:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0560000000000005e-06 2024-02-19T21:04:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:04:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:02:19.433729+0000", 2024-02-19T21:04:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019375099999999999 2024-02-19T21:04:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:02:19.435666+0000", 2024-02-19T21:04:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1607999999999999e-05 2024-02-19T21:04:04.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:02:19.435708+0000", 2024-02-19T21:04:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7139999999999998e-05 2024-02-19T21:04:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:02:19.435775+0000", 2024-02-19T21:04:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002245567 2024-02-19T21:04:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.710 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDScrubReserve(2.1d REQUEST e85)", 2024-02-19T21:04:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:03:26.788521+0000", 2024-02-19T21:04:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.823711517, 2024-02-19T21:04:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001599184, 2024-02-19T21:04:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:04:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:03:26.788521+0000", 2024-02-19T21:04:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:03:26.788521+0000", 2024-02-19T21:04:04.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:04:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:03:26.788520+0000", 2024-02-19T21:04:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5990000000000004e-06 2024-02-19T21:04:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:03:26.788525+0000", 2024-02-19T21:04:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6962e-05 2024-02-19T21:04:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:03:26.788552+0000", 2024-02-19T21:04:04.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0199e-05 2024-02-19T21:04:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:03:26.788562+0000", 2024-02-19T21:04:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1536000000000001e-05 2024-02-19T21:04:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:03:26.788633+0000", 2024-02-19T21:04:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0709999999999995e-06 2024-02-19T21:04:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:04:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:03:26.788641+0000", 2024-02-19T21:04:04.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001432914 2024-02-19T21:04:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:03:26.790074+0000", 2024-02-19T21:04:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6705e-05 2024-02-19T21:04:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:03:26.790091+0000", 2024-02-19T21:04:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8802999999999998e-05 2024-02-19T21:04:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:03:26.790120+0000", 2024-02-19T21:04:04.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001599184 2024-02-19T21:04:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDScrubReserve(2.c REQUEST e91)", 2024-02-19T21:04:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:03.761098+0000", 2024-02-19T21:04:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 0.85113441499999998, 2024-02-19T21:04:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003990141, 2024-02-19T21:04:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:04:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:04.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:03.761098+0000", 2024-02-19T21:04:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:03.761098+0000", 2024-02-19T21:04:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:04:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:03.761097+0000", 2024-02-19T21:04:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.614e-06 2024-02-19T21:04:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:03.761101+0000", 2024-02-19T21:04:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7700999999999999e-05 2024-02-19T21:04:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:03.761119+0000", 2024-02-19T21:04:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7690000000000008e-06 2024-02-19T21:04:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:03.761128+0000", 2024-02-19T21:04:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000577555 2024-02-19T21:04:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:03.761706+0000", 2024-02-19T21:04:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.046e-06 2024-02-19T21:04:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:04:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:03.761713+0000", 2024-02-19T21:04:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033308650000000001 2024-02-19T21:04:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:03.765044+0000", 2024-02-19T21:04:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3908e-05 2024-02-19T21:04:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:03.765058+0000", 2024-02-19T21:04:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0086e-05 2024-02-19T21:04:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:03.765088+0000", 2024-02-19T21:04:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003990141 2024-02-19T21:04:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:04.719 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:04.719 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:04:04.720 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:04.746 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:04.764 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:04.767 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:04.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:04.828 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:02.944857+0000 mgr.y (mgr.14550) 115 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:04:04.828 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:03.601704+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-02-19T21:04:04.829 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:03.805843+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:04:04.829 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:03.806895+0000 osd.3 (osd.3) 60 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:04:04.829 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:04.542528+0000 osd.0 (osd.0) 47 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:04:04.829 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:04.545123+0000 osd.0 (osd.0) 48 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:04:04.829 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:02.944857+0000 mgr.y (mgr.14550) 115 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:04:04.829 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:03.601704+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-02-19T21:04:04.829 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:03.805843+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:04:04.829 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:03.806895+0000 osd.3 (osd.3) 60 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:04:04.829 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:04.542528+0000 osd.0 (osd.0) 47 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:04:04.829 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:04.545123+0000 osd.0 (osd.0) 48 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:04:04.868 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:04.892 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:02.944857+0000 mgr.y (mgr.14550) 115 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:04:04.892 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:03.601704+0000 mon.a (mon.0) 731 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-02-19T21:04:04.892 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:03.805843+0000 osd.3 (osd.3) 59 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:04:04.892 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:03.806895+0000 osd.3 (osd.3) 60 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:04:04.892 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:04.542528+0000 osd.0 (osd.0) 47 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:04:04.892 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:04.545123+0000 osd.0 (osd.0) 48 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:04:04.894 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:04.913 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:04.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:04.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:05.016 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:05.054 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:05.457 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:05.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:03.766105+0000 osd.2 (osd.2) 39 : cluster [DBG] 2.c scrub starts 2024-02-19T21:04:05.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:03.767706+0000 osd.2 (osd.2) 40 : cluster [DBG] 2.c scrub ok 2024-02-19T21:04:05.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:04.385866+0000 osd.1 (osd.1) 49 : cluster [DBG] 5.c scrub starts 2024-02-19T21:04:05.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:04.386729+0000 osd.1 (osd.1) 50 : cluster [DBG] 5.c scrub ok 2024-02-19T21:04:05.915 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:05.133292+0000 osd.5 (osd.5) 45 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:04:05.915 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:05.136183+0000 osd.5 (osd.5) 46 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:04:05.915 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:05.556641+0000 osd.0 (osd.0) 49 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:04:05.915 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:05.558757+0000 osd.0 (osd.0) 50 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:04:05.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:03.766105+0000 osd.2 (osd.2) 39 : cluster [DBG] 2.c scrub starts 2024-02-19T21:04:05.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:03.767706+0000 osd.2 (osd.2) 40 : cluster [DBG] 2.c scrub ok 2024-02-19T21:04:05.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:04.385866+0000 osd.1 (osd.1) 49 : cluster [DBG] 5.c scrub starts 2024-02-19T21:04:05.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:04.386729+0000 osd.1 (osd.1) 50 : cluster [DBG] 5.c scrub ok 2024-02-19T21:04:05.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:05.133292+0000 osd.5 (osd.5) 45 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:04:05.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:05.136183+0000 osd.5 (osd.5) 46 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:04:05.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:05.556641+0000 osd.0 (osd.0) 49 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:04:05.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:05.558757+0000 osd.0 (osd.0) 50 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:04:05.916 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:05.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:05.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:06.021 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:06.051 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:03.766105+0000 osd.2 (osd.2) 39 : cluster [DBG] 2.c scrub starts 2024-02-19T21:04:06.051 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:03.767706+0000 osd.2 (osd.2) 40 : cluster [DBG] 2.c scrub ok 2024-02-19T21:04:06.051 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:04.385866+0000 osd.1 (osd.1) 49 : cluster [DBG] 5.c scrub starts 2024-02-19T21:04:06.051 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:04.386729+0000 osd.1 (osd.1) 50 : cluster [DBG] 5.c scrub ok 2024-02-19T21:04:06.051 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:05.133292+0000 osd.5 (osd.5) 45 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:04:06.051 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:05.136183+0000 osd.5 (osd.5) 46 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:04:06.051 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:05.556641+0000 osd.0 (osd.0) 49 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:04:06.051 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:05.558757+0000 osd.0 (osd.0) 50 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:04:06.053 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:06.074 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:06.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:06.178 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:04:06.209 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:06.230 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:04:06.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:06.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:04:06.238 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:04:06.334 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:06.376 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:06.589 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-19T21:04:06.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:04.810357+0000 osd.6 (osd.6) 33 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:04:06.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:04.812139+0000 osd.6 (osd.6) 34 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:04:06.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:04.945590+0000 mgr.y (mgr.14550) 116 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:04:06.990 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:04.810357+0000 osd.6 (osd.6) 33 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:04:06.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:04.812139+0000 osd.6 (osd.6) 34 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:04:06.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:04.945590+0000 mgr.y (mgr.14550) 116 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:04:07.063 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:04.810357+0000 osd.6 (osd.6) 33 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:04:07.063 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:04.812139+0000 osd.6 (osd.6) 34 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:04:07.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:04.945590+0000 mgr.y (mgr.14550) 116 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:04:07.513 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:07.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:07.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:07.617 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:07.649 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:07.670 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:07.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:07.774 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:07.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:05.786190+0000 osd.6 (osd.6) 35 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:04:07.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:05.787410+0000 osd.6 (osd.6) 36 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:04:07.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:06.380123+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:04:07.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:06.383552+0000 osd.1 (osd.1) 52 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:04:07.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:06.595997+0000 osd.0 (osd.0) 51 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:04:07.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:06.599431+0000 osd.0 (osd.0) 52 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:04:07.814 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:06.804884+0000 osd.3 (osd.3) 61 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:04:07.814 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:06.807182+0000 osd.3 (osd.3) 62 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:04:07.814 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:06.877981+0000 osd.7 (osd.7) 45 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:04:07.814 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:06.880895+0000 osd.7 (osd.7) 46 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:04:07.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:05.786190+0000 osd.6 (osd.6) 35 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:04:07.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:05.787410+0000 osd.6 (osd.6) 36 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:04:07.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:06.380123+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:04:07.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:06.383552+0000 osd.1 (osd.1) 52 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:04:07.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:06.595997+0000 osd.0 (osd.0) 51 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:04:07.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:06.599431+0000 osd.0 (osd.0) 52 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:04:07.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:06.804884+0000 osd.3 (osd.3) 61 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:04:07.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:06.807182+0000 osd.3 (osd.3) 62 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:04:07.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:06.877981+0000 osd.7 (osd.7) 45 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:04:07.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:06.880895+0000 osd.7 (osd.7) 46 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:04:07.816 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:08.064 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:04:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:04:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:04:08.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:05.786190+0000 osd.6 (osd.6) 35 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:04:08.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:05.787410+0000 osd.6 (osd.6) 36 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:04:08.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:06.380123+0000 osd.1 (osd.1) 51 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:04:08.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:06.383552+0000 osd.1 (osd.1) 52 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:04:08.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:06.595997+0000 osd.0 (osd.0) 51 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:04:08.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:06.599431+0000 osd.0 (osd.0) 52 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:04:08.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:06.804884+0000 osd.3 (osd.3) 61 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:04:08.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:06.807182+0000 osd.3 (osd.3) 62 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:04:08.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:06.877981+0000 osd.7 (osd.7) 45 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:04:08.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:06.880895+0000 osd.7 (osd.7) 46 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:04:08.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:06.945971+0000 mgr.y (mgr.14550) 117 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:04:08.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:07.587497+0000 osd.0 (osd.0) 53 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:04:08.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:07.592200+0000 osd.0 (osd.0) 54 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:04:08.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:08.176875+0000 osd.4 (osd.4) 53 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:04:08.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:08.177332+0000 mon.a (mon.0) 732 : audit [DBG] from='client.? v1:172.21.15.163:0/2259049776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:04:08.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:08.178219+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:04:08.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:08.180583+0000 osd.5 (osd.5) 47 : cluster [DBG] 6.d scrub starts 2024-02-19T21:04:08.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:08.185712+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.d scrub ok 2024-02-19T21:04:08.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:06.945971+0000 mgr.y (mgr.14550) 117 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:04:08.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:07.587497+0000 osd.0 (osd.0) 53 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:04:08.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:07.592200+0000 osd.0 (osd.0) 54 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:04:08.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:08.176875+0000 osd.4 (osd.4) 53 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:04:08.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:08.177332+0000 mon.a (mon.0) 732 : audit [DBG] from='client.? v1:172.21.15.163:0/2259049776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:04:08.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:08.178219+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:04:08.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:08.180583+0000 osd.5 (osd.5) 47 : cluster [DBG] 6.d scrub starts 2024-02-19T21:04:08.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:08.185712+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.d scrub ok 2024-02-19T21:04:08.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.173+0000 7f0078118700 1 -- 172.21.15.163:0/3947065506 >> v1:172.21.15.193:6789/0 conn(0x7f00700fc870 legacy=0x7f00700fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:08.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.173+0000 7f0078118700 1 -- 172.21.15.163:0/3947065506 shutdown_connections 2024-02-19T21:04:08.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.173+0000 7f0078118700 1 -- 172.21.15.163:0/3947065506 >> 172.21.15.163:0/3947065506 conn(0x7f00700f2470 msgr2=0x7f00700f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:08.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.173+0000 7f0078118700 1 -- 172.21.15.163:0/3947065506 shutdown_connections 2024-02-19T21:04:08.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.173+0000 7f0078118700 1 -- 172.21.15.163:0/3947065506 wait complete. 2024-02-19T21:04:08.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.173+0000 7f0078118700 1 librados: starting msgr at 2024-02-19T21:04:08.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.173+0000 7f0078118700 1 librados: starting objecter 2024-02-19T21:04:08.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.174+0000 7f0078118700 1 Processor -- start 2024-02-19T21:04:08.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.174+0000 7f0078118700 1 -- start start 2024-02-19T21:04:08.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.174+0000 7f0078118700 1 librados: setting wanted keys 2024-02-19T21:04:08.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.174+0000 7f0078118700 1 librados: calling monclient init 2024-02-19T21:04:08.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.174+0000 7f0078118700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0070117310 con 0x7f00701031b0 2024-02-19T21:04:08.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.174+0000 7f0078118700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f00701184f0 con 0x7f00700ff670 2024-02-19T21:04:08.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.174+0000 7f0078118700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f00701196f0 con 0x7f00700fc870 2024-02-19T21:04:08.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.174+0000 7f00766b5700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f00701031b0 0x7f00701159d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:42878/0 (socket says 172.21.15.163:42878) 2024-02-19T21:04:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.174+0000 7f00766b5700 1 -- 172.21.15.163:0/2259049776 learned_addr learned my addr 172.21.15.163:0/2259049776 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:04:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.174+0000 7f0066ffd700 10 client.?.objecter ms_handle_connect 0x7f00701031b0 2024-02-19T21:04:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.174+0000 7f0066ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.174+0000 7f0066ffd700 10 client.?.objecter ms_handle_connect 0x7f00700fc870 2024-02-19T21:04:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.174+0000 7f0066ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282850232 0 0) 0x7f0070117310 con 0x7f00701031b0 2024-02-19T21:04:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f005c003110 con 0x7f00701031b0 2024-02-19T21:04:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 10 client.?.objecter ms_handle_connect 0x7f00700ff670 2024-02-19T21:04:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820979928 0 0) 0x7f00701196f0 con 0x7f00700fc870 2024-02-19T21:04:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0070117310 con 0x7f00700fc870 2024-02-19T21:04:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116943794 0 0) 0x7f00701184f0 con 0x7f00700ff670 2024-02-19T21:04:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00701196f0 con 0x7f00700ff670 2024-02-19T21:04:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2027634506 0 0) 0x7f005c003110 con 0x7f00701031b0 2024-02-19T21:04:08.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00701184f0 con 0x7f00701031b0 2024-02-19T21:04:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0068001be0 con 0x7f00701031b0 2024-02-19T21:04:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2205247981 0 0) 0x7f00701196f0 con 0x7f00700ff670 2024-02-19T21:04:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f005c003110 con 0x7f00700ff670 2024-02-19T21:04:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0060001880 con 0x7f00700ff670 2024-02-19T21:04:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1905751624 0 0) 0x7f00701184f0 con 0x7f00701031b0 2024-02-19T21:04:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 >> v1:172.21.15.163:6790/0 conn(0x7f00700fc870 legacy=0x7f0070060b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 >> v1:172.21.15.193:6789/0 conn(0x7f00700ff670 legacy=0x7f00700612e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007011a8f0 con 0x7f00701031b0 2024-02-19T21:04:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0070119880 con 0x7f00701031b0 2024-02-19T21:04:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0078118700 10 client.14838.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0070119cf0 con 0x7f00701031b0 2024-02-19T21:04:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.175+0000 7f0078118700 1 librados: init done 2024-02-19T21:04:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.176+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f007005f700 con 0x7f00701031b0 2024-02-19T21:04:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.176+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f00680021e0 con 0x7f00701031b0 2024-02-19T21:04:08.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.176+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0068005cf0 con 0x7f00701031b0 2024-02-19T21:04:08.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.178+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f0068001d40 con 0x7f00701031b0 2024-02-19T21:04:08.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(91..91 src has 32..91) v4 ==== 6172+0+0 (unknown 1247502107 0 0) 0x7f0068078820 con 0x7f00701031b0 2024-02-19T21:04:08.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 osd_map(91..91 src has 32..91) v4 2024-02-19T21:04:08.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0066ffd700 3 client.14838.objecter handle_osd_map got epochs [91,91] > 0 2024-02-19T21:04:08.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0066ffd700 3 client.14838.objecter handle_osd_map decoding full epoch 91 2024-02-19T21:04:08.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0066ffd700 20 client.14838.objecter dump_active .. 0 homeless 2024-02-19T21:04:08.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v91) v1 ==== 54+0+16 (unknown 801643359 0 2420000192) 0x7f0068078be0 con 0x7f00701031b0 2024-02-19T21:04:08.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0078118700 10 client.14838.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:08.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 1 2024-02-19T21:04:08.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070047540 osd=0 3 2024-02-19T21:04:08.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070047540 osd=0 3 2024-02-19T21:04:08.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070047540 osd=0 3 2024-02-19T21:04:08.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 1 2024-02-19T21:04:08.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f0070047540 osd=0 3 2024-02-19T21:04:08.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 0 1 2024-02-19T21:04:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070047540 osd=0 4 2024-02-19T21:04:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0078118700 10 client.14838.objecter _send_command 1 2024-02-19T21:04:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.179+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00700fa640 con 0x7f0070118680 2024-02-19T21:04:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.180+0000 7f0066ffd700 10 client.14838.objecter ms_handle_connect 0x7f0070118680 2024-02-19T21:04:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.180+0000 7f0066ffd700 10 client.14838.objecter ms_handle_connect 0x7f005c063dc0 2024-02-19T21:04:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.181+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f00700fa640 con 0x7f0070118680 2024-02-19T21:04:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.181+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:04:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.181+0000 7f0066ffd700 10 client.14838.objecter _finish_command 1 = system:0 2024-02-19T21:04:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.181+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f0070047540 osd=0 4 2024-02-19T21:04:08.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.181+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 0 1 2024-02-19T21:04:08.949 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:04:08.949 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:04:08.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.209+0000 7f0078118700 10 client.14838.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:08.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.209+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 2 2024-02-19T21:04:08.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.209+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070047540 osd=0 3 2024-02-19T21:04:08.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.209+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070047540 osd=0 3 2024-02-19T21:04:08.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.209+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070047540 osd=0 3 2024-02-19T21:04:08.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.209+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 2 2024-02-19T21:04:08.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.209+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f0070047540 osd=0 3 2024-02-19T21:04:08.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.209+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 0 2 2024-02-19T21:04:08.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.209+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070047540 osd=0 4 2024-02-19T21:04:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.209+0000 7f0078118700 10 client.14838.objecter _send_command 2 2024-02-19T21:04:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.209+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f00700f80a0 con 0x7f0070118680 2024-02-19T21:04:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f00700f80a0 con 0x7f0070118680 2024-02-19T21:04:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0066ffd700 10 client.14838.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f0070047540 osd=0 4 2024-02-19T21:04:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 0 2 2024-02-19T21:04:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0078118700 10 client.14838.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 3 2024-02-19T21:04:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070107cb0 osd=1 3 2024-02-19T21:04:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070107cb0 osd=1 3 2024-02-19T21:04:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070107cb0 osd=1 3 2024-02-19T21:04:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 3 2024-02-19T21:04:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f0070107cb0 osd=1 3 2024-02-19T21:04:08.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 1 3 2024-02-19T21:04:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070107cb0 osd=1 4 2024-02-19T21:04:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0078118700 10 client.14838.objecter _send_command 3 2024-02-19T21:04:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.211+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00700283d0 con 0x7f00700f2d60 2024-02-19T21:04:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.212+0000 7f0066ffd700 10 client.14838.objecter ms_handle_connect 0x7f00700f2d60 2024-02-19T21:04:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.212+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f00700283d0 con 0x7f00700f2d60 2024-02-19T21:04:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.212+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:04:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.212+0000 7f0066ffd700 10 client.14838.objecter _finish_command 3 = system:0 2024-02-19T21:04:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.212+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f0070107cb0 osd=1 4 2024-02-19T21:04:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.212+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 1 3 2024-02-19T21:04:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.240+0000 7f0078118700 10 client.14838.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.240+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 4 2024-02-19T21:04:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.240+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070107cb0 osd=1 3 2024-02-19T21:04:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.240+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070107cb0 osd=1 3 2024-02-19T21:04:08.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.240+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070107cb0 osd=1 3 2024-02-19T21:04:08.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.240+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 4 2024-02-19T21:04:08.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.240+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f0070107cb0 osd=1 3 2024-02-19T21:04:08.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.240+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 1 4 2024-02-19T21:04:08.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.240+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070107cb0 osd=1 4 2024-02-19T21:04:08.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.240+0000 7f0078118700 10 client.14838.objecter _send_command 4 2024-02-19T21:04:08.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.240+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f00700287b0 con 0x7f00700f2d60 2024-02-19T21:04:08.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f00700287b0 con 0x7f00700f2d60 2024-02-19T21:04:08.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:08.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0066ffd700 10 client.14838.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:08.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f0070107cb0 osd=1 4 2024-02-19T21:04:08.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 1 4 2024-02-19T21:04:08.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0078118700 10 client.14838.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:08.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 5 2024-02-19T21:04:08.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007011a220 osd=2 3 2024-02-19T21:04:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007011a220 osd=2 3 2024-02-19T21:04:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007011a220 osd=2 3 2024-02-19T21:04:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 5 2024-02-19T21:04:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f007011a220 osd=2 3 2024-02-19T21:04:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 2 5 2024-02-19T21:04:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007011a220 osd=2 4 2024-02-19T21:04:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0078118700 10 client.14838.objecter _send_command 5 2024-02-19T21:04:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f007011a390 con 0x7f0070059e00 2024-02-19T21:04:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.243+0000 7f0066ffd700 10 client.14838.objecter ms_handle_connect 0x7f0070059e00 2024-02-19T21:04:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.244+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f007011a390 con 0x7f0070059e00 2024-02-19T21:04:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.244+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:04:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.244+0000 7f0066ffd700 10 client.14838.objecter _finish_command 5 = system:0 2024-02-19T21:04:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.244+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f007011a220 osd=2 4 2024-02-19T21:04:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.244+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 2 5 2024-02-19T21:04:08.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.261+0000 7f0078118700 10 client.14838.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:08.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.261+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 6 2024-02-19T21:04:08.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.261+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007011a220 osd=2 3 2024-02-19T21:04:08.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.261+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007011a220 osd=2 3 2024-02-19T21:04:08.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.261+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007011a220 osd=2 3 2024-02-19T21:04:08.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.261+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 6 2024-02-19T21:04:08.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.261+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f007011a220 osd=2 3 2024-02-19T21:04:08.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.261+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 2 6 2024-02-19T21:04:08.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.261+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007011a220 osd=2 4 2024-02-19T21:04:08.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.261+0000 7f0078118700 10 client.14838.objecter _send_command 6 2024-02-19T21:04:08.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.261+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0070125810 con 0x7f0070059e00 2024-02-19T21:04:08.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0070125810 con 0x7f0070059e00 2024-02-19T21:04:08.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:08.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0066ffd700 10 client.14838.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:08.955 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:04:08.955 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:04:08.955 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:04:08.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f007011a220 osd=2 4 2024-02-19T21:04:08.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 2 6 2024-02-19T21:04:08.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0078118700 10 client.14838.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:08.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 7 2024-02-19T21:04:08.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070059360 osd=3 3 2024-02-19T21:04:08.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070059360 osd=3 3 2024-02-19T21:04:08.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070059360 osd=3 3 2024-02-19T21:04:08.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 7 2024-02-19T21:04:08.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f0070059360 osd=3 3 2024-02-19T21:04:08.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 3 7 2024-02-19T21:04:08.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070059360 osd=3 4 2024-02-19T21:04:08.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0078118700 10 client.14838.objecter _send_command 7 2024-02-19T21:04:08.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0070121c40 con 0x7f00700594d0 2024-02-19T21:04:08.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.263+0000 7f0066ffd700 10 client.14838.objecter ms_handle_connect 0x7f00700594d0 2024-02-19T21:04:08.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.264+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0070121c40 con 0x7f00700594d0 2024-02-19T21:04:08.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.264+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:04:08.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.264+0000 7f0066ffd700 10 client.14838.objecter _finish_command 7 = system:0 2024-02-19T21:04:08.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.264+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f0070059360 osd=3 4 2024-02-19T21:04:08.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.264+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 3 7 2024-02-19T21:04:08.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.278+0000 7f0078118700 10 client.14838.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.278+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 8 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.278+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070059360 osd=3 3 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.278+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070059360 osd=3 3 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.278+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070059360 osd=3 3 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.278+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 8 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.278+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f0070059360 osd=3 3 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.278+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 3 8 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.278+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070059360 osd=3 4 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.278+0000 7f0078118700 10 client.14838.objecter _send_command 8 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.278+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f007012a290 con 0x7f00700594d0 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f007012a290 con 0x7f00700594d0 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0066ffd700 10 client.14838.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f0070059360 osd=3 4 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 3 8 2024-02-19T21:04:08.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0078118700 10 client.14838.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:08.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 9 2024-02-19T21:04:08.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007012a290 osd=4 3 2024-02-19T21:04:08.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007012a290 osd=4 3 2024-02-19T21:04:08.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007012a290 osd=4 3 2024-02-19T21:04:08.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 9 2024-02-19T21:04:08.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f007012a290 osd=4 3 2024-02-19T21:04:08.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 4 9 2024-02-19T21:04:08.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007012a290 osd=4 4 2024-02-19T21:04:08.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0078118700 10 client.14838.objecter _send_command 9 2024-02-19T21:04:08.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.280+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f007012c5a0 con 0x7f007012a910 2024-02-19T21:04:08.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.281+0000 7f0066ffd700 10 client.14838.objecter ms_handle_connect 0x7f007012a910 2024-02-19T21:04:08.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.282+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f007012c5a0 con 0x7f007012a910 2024-02-19T21:04:08.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.282+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:04:08.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.282+0000 7f0066ffd700 10 client.14838.objecter _finish_command 9 = system:0 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.282+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f007012a290 osd=4 4 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.282+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 4 9 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.302+0000 7f0078118700 10 client.14838.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.302+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 10 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.302+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007012a290 osd=4 3 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.302+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007012a290 osd=4 3 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.302+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007012a290 osd=4 3 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.302+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 10 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.302+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f007012a290 osd=4 3 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.302+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 4 10 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.302+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007012a290 osd=4 4 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.302+0000 7f0078118700 10 client.14838.objecter _send_command 10 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.302+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f007012c5a0 con 0x7f007012a910 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f007012c5a0 con 0x7f007012a910 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:08.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0066ffd700 10 client.14838.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:08.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f007012a290 osd=4 4 2024-02-19T21:04:08.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 4 10 2024-02-19T21:04:08.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0078118700 10 client.14838.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:08.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 11 2024-02-19T21:04:08.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007012c5a0 osd=5 3 2024-02-19T21:04:08.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007012c5a0 osd=5 3 2024-02-19T21:04:08.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007012c5a0 osd=5 3 2024-02-19T21:04:08.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 11 2024-02-19T21:04:08.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f007012c5a0 osd=5 3 2024-02-19T21:04:08.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 5 11 2024-02-19T21:04:08.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007012c5a0 osd=5 4 2024-02-19T21:04:08.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0078118700 10 client.14838.objecter _send_command 11 2024-02-19T21:04:08.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.304+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f007012c9a0 con 0x7f007012f0d0 2024-02-19T21:04:08.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.305+0000 7f0066ffd700 10 client.14838.objecter ms_handle_connect 0x7f007012f0d0 2024-02-19T21:04:08.961 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:04:08.961 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:04:08.961 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:04:08.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.306+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f007012c9a0 con 0x7f007012f0d0 2024-02-19T21:04:08.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.306+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:04:08.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.306+0000 7f0066ffd700 10 client.14838.objecter _finish_command 11 = system:0 2024-02-19T21:04:08.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.306+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f007012c5a0 osd=5 4 2024-02-19T21:04:08.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.306+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 5 11 2024-02-19T21:04:08.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.330+0000 7f0078118700 10 client.14838.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:08.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.330+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 12 2024-02-19T21:04:08.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.330+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007012c5a0 osd=5 3 2024-02-19T21:04:08.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.330+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007012c5a0 osd=5 3 2024-02-19T21:04:08.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.330+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007012c5a0 osd=5 3 2024-02-19T21:04:08.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.330+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 12 2024-02-19T21:04:08.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.330+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f007012c5a0 osd=5 3 2024-02-19T21:04:08.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.330+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 5 12 2024-02-19T21:04:08.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.330+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007012c5a0 osd=5 4 2024-02-19T21:04:08.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.330+0000 7f0078118700 10 client.14838.objecter _send_command 12 2024-02-19T21:04:08.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.330+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f007012c9a0 con 0x7f007012f0d0 2024-02-19T21:04:08.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f007012c9a0 con 0x7f007012f0d0 2024-02-19T21:04:08.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0066ffd700 10 client.14838.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f007012c5a0 osd=5 4 2024-02-19T21:04:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 5 12 2024-02-19T21:04:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0078118700 10 client.14838.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 13 2024-02-19T21:04:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007012c9a0 osd=6 3 2024-02-19T21:04:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007012c9a0 osd=6 3 2024-02-19T21:04:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007012c9a0 osd=6 3 2024-02-19T21:04:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 13 2024-02-19T21:04:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f007012c9a0 osd=6 3 2024-02-19T21:04:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 6 13 2024-02-19T21:04:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007012c9a0 osd=6 4 2024-02-19T21:04:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0078118700 10 client.14838.objecter _send_command 13 2024-02-19T21:04:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.333+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0070133cc0 con 0x7f0070125ef0 2024-02-19T21:04:08.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.334+0000 7f0066ffd700 10 client.14838.objecter ms_handle_connect 0x7f0070125ef0 2024-02-19T21:04:08.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.334+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0070133cc0 con 0x7f0070125ef0 2024-02-19T21:04:08.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.334+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:04:08.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.334+0000 7f0066ffd700 10 client.14838.objecter _finish_command 13 = system:0 2024-02-19T21:04:08.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.334+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f007012c9a0 osd=6 4 2024-02-19T21:04:08.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.334+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 6 13 2024-02-19T21:04:08.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.361+0000 7f0078118700 10 client.14838.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:08.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.361+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 14 2024-02-19T21:04:08.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.361+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007012c9a0 osd=6 3 2024-02-19T21:04:08.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.361+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007012c9a0 osd=6 3 2024-02-19T21:04:08.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.361+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f007012c9a0 osd=6 3 2024-02-19T21:04:08.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.361+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 14 2024-02-19T21:04:08.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.361+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f007012c9a0 osd=6 3 2024-02-19T21:04:08.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.361+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 6 14 2024-02-19T21:04:08.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.361+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007012c9a0 osd=6 4 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.361+0000 7f0078118700 10 client.14838.objecter _send_command 14 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.361+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0070133cc0 con 0x7f0070125ef0 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0070133cc0 con 0x7f0070125ef0 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0066ffd700 10 client.14838.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f007012c9a0 osd=6 4 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 6 14 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0078118700 10 client.14838.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 15 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070133cc0 osd=7 3 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070133cc0 osd=7 3 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070133cc0 osd=7 3 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 15 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f0070133cc0 osd=7 3 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 7 15 2024-02-19T21:04:08.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070133cc0 osd=7 4 2024-02-19T21:04:08.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0078118700 10 client.14838.objecter _send_command 15 2024-02-19T21:04:08.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f007013b3f0 con 0x7f00701378e0 2024-02-19T21:04:08.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.365+0000 7f0066ffd700 10 client.14838.objecter ms_handle_connect 0x7f00701378e0 2024-02-19T21:04:08.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.366+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f007013b3f0 con 0x7f00701378e0 2024-02-19T21:04:08.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.366+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:04:08.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.366+0000 7f0066ffd700 10 client.14838.objecter _finish_command 15 = system:0 2024-02-19T21:04:08.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.366+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f0070133cc0 osd=7 4 2024-02-19T21:04:08.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.366+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 7 15 2024-02-19T21:04:08.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.381+0000 7f0078118700 10 client.14838.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:08.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.381+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign -1 16 2024-02-19T21:04:08.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.381+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070133cc0 osd=7 3 2024-02-19T21:04:08.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.381+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070133cc0 osd=7 3 2024-02-19T21:04:08.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.381+0000 7f0078118700 20 client.14838.objecter _get_session s=0x7f0070133cc0 osd=7 3 2024-02-19T21:04:08.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.381+0000 7f0078118700 15 client.14838.objecter _session_command_op_remove -1 16 2024-02-19T21:04:08.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.381+0000 7f0078118700 20 client.14838.objecter get_session s=0x7f0070133cc0 osd=7 3 2024-02-19T21:04:08.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.381+0000 7f0078118700 15 client.14838.objecter _session_command_op_assign 7 16 2024-02-19T21:04:08.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.381+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070133cc0 osd=7 4 2024-02-19T21:04:08.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.381+0000 7f0078118700 10 client.14838.objecter _send_command 16 2024-02-19T21:04:08.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.381+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f007013b3f0 con 0x7f00701378e0 2024-02-19T21:04:08.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0066ffd700 1 -- 172.21.15.163:0/2259049776 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f007013b3f0 con 0x7f00701378e0 2024-02-19T21:04:08.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0066ffd700 10 client.14838.objecter ms_dispatch 0x7f0070001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:08.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0066ffd700 10 client.14838.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:08.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0066ffd700 20 client.14838.objecter put_session s=0x7f0070133cc0 osd=7 4 2024-02-19T21:04:08.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0066ffd700 15 client.14838.objecter _session_command_op_remove 7 16 2024-02-19T21:04:08.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 10 librados: watch_flush enter 2024-02-19T21:04:08.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 10 librados: watch_flush exit 2024-02-19T21:04:08.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 10 client.14838.objecter close_session for osd.0 2024-02-19T21:04:08.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 >> v1:172.21.15.163:6801/1559418115 conn(0x7f0070118680 legacy=0x7f0070118a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:08.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070047540 osd=0 1 2024-02-19T21:04:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 10 client.14838.objecter close_session for osd.1 2024-02-19T21:04:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 >> v1:172.21.15.163:6805/3125047432 conn(0x7f00700f2d60 legacy=0x7f00700f31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070107cb0 osd=1 1 2024-02-19T21:04:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 10 client.14838.objecter close_session for osd.2 2024-02-19T21:04:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 >> v1:172.21.15.163:6809/1563158487 conn(0x7f0070059e00 legacy=0x7f00701240f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007011a220 osd=2 1 2024-02-19T21:04:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 10 client.14838.objecter close_session for osd.3 2024-02-19T21:04:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 >> v1:172.21.15.163:6813/2112279769 conn(0x7f00700594d0 legacy=0x7f0070128b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070059360 osd=3 1 2024-02-19T21:04:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 10 client.14838.objecter close_session for osd.4 2024-02-19T21:04:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 >> v1:172.21.15.193:6800/3129716965 conn(0x7f007012a910 legacy=0x7f007012ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f00766b5700 1 -- 172.21.15.163:0/2259049776 reap_dead start 2024-02-19T21:04:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007012a290 osd=4 1 2024-02-19T21:04:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 10 client.14838.objecter close_session for osd.5 2024-02-19T21:04:08.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 >> v1:172.21.15.193:6804/805161991 conn(0x7f007012f0d0 legacy=0x7f00701314c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:08.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f00766b5700 1 -- 172.21.15.163:0/2259049776 reap_dead start 2024-02-19T21:04:08.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f00766b5700 1 -- 172.21.15.163:0/2259049776 reap_dead start 2024-02-19T21:04:08.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007012c5a0 osd=5 1 2024-02-19T21:04:08.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 10 client.14838.objecter close_session for osd.6 2024-02-19T21:04:08.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 >> v1:172.21.15.193:6808/787604615 conn(0x7f0070125ef0 legacy=0x7f0070126370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:08.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f007012c9a0 osd=6 1 2024-02-19T21:04:08.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 10 client.14838.objecter close_session for osd.7 2024-02-19T21:04:08.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 >> v1:172.21.15.193:6812/157927432 conn(0x7f00701378e0 legacy=0x7f0070139cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:08.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 20 client.14838.objecter put_session s=0x7f0070133cc0 osd=7 1 2024-02-19T21:04:08.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 20 client.14838.objecter shutdown clearing up homeless session... 2024-02-19T21:04:08.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.383+0000 7f0078118700 10 client.14838.objecter successfully canceled tick 2024-02-19T21:04:08.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.384+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 >> v1:172.21.15.163:6800/1552514681 conn(0x7f005c063dc0 legacy=0x7f005c066280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:08.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.384+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 >> v1:172.21.15.163:6789/0 conn(0x7f00701031b0 legacy=0x7f00701159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:08.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.384+0000 7f00766b5700 1 -- 172.21.15.163:0/2259049776 reap_dead start 2024-02-19T21:04:08.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.384+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 shutdown_connections 2024-02-19T21:04:08.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.384+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 >> 172.21.15.163:0/2259049776 conn(0x7f00700f2470 msgr2=0x7f0070103620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:08.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.385+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 shutdown_connections 2024-02-19T21:04:08.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.385+0000 7f0078118700 1 -- 172.21.15.163:0/2259049776 wait complete. 2024-02-19T21:04:08.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:08.385+0000 7f0078118700 1 librados: shutdown 2024-02-19T21:04:08.970 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:04:08.970 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:04:08.970 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:04:08.970 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:04:08.970 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:04:08.970 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:04:08.970 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:04:08.970 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:04:09.063 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:06.945971+0000 mgr.y (mgr.14550) 117 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:04:09.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:07.587497+0000 osd.0 (osd.0) 53 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:04:09.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:07.592200+0000 osd.0 (osd.0) 54 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:04:09.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:08.176875+0000 osd.4 (osd.4) 53 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:04:09.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:08.177332+0000 mon.a (mon.0) 732 : audit [DBG] from='client.? v1:172.21.15.163:0/2259049776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:04:09.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:08.178219+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:04:09.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:08.180583+0000 osd.5 (osd.5) 47 : cluster [DBG] 6.d scrub starts 2024-02-19T21:04:09.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:08.185712+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.d scrub ok 2024-02-19T21:04:09.638 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:09.676 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:09.683 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:09.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:09.784 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:09.826 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:09.830 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:09.927 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:07.805811+0000 osd.2 (osd.2) 41 : cluster [DBG] 5.d scrub starts 2024-02-19T21:04:09.927 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:07.808936+0000 osd.2 (osd.2) 42 : cluster [DBG] 5.d scrub ok 2024-02-19T21:04:09.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:08.613279+0000 osd.0 (osd.0) 55 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:04:09.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:08.616388+0000 osd.0 (osd.0) 56 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:04:09.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:08.835147+0000 osd.3 (osd.3) 63 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:04:09.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:08.837644+0000 osd.3 (osd.3) 64 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:04:09.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:08.855155+0000 osd.7 (osd.7) 47 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:04:09.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:08.857969+0000 osd.7 (osd.7) 48 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:04:09.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:09.137437+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:04:09.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:09.139339+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:04:09.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:09.177230+0000 osd.5 (osd.5) 49 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:04:09.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:09.180155+0000 osd.5 (osd.5) 50 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:04:09.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:09.568584+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:04:09.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:09.570964+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:04:09.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:07.805811+0000 osd.2 (osd.2) 41 : cluster [DBG] 5.d scrub starts 2024-02-19T21:04:09.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:07.808936+0000 osd.2 (osd.2) 42 : cluster [DBG] 5.d scrub ok 2024-02-19T21:04:09.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:08.613279+0000 osd.0 (osd.0) 55 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:04:09.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:08.616388+0000 osd.0 (osd.0) 56 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:04:09.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:08.835147+0000 osd.3 (osd.3) 63 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:04:09.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:08.837644+0000 osd.3 (osd.3) 64 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:04:09.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:08.855155+0000 osd.7 (osd.7) 47 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:04:09.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:08.857969+0000 osd.7 (osd.7) 48 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:04:09.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:09.137437+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:04:09.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:09.139339+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:04:09.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:09.177230+0000 osd.5 (osd.5) 49 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:04:09.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:09.180155+0000 osd.5 (osd.5) 50 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:04:09.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:09.568584+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:04:09.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:09.570964+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:04:09.931 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:09.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:09.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:10.038 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:10.041 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:07.805811+0000 osd.2 (osd.2) 41 : cluster [DBG] 5.d scrub starts 2024-02-19T21:04:10.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:07.808936+0000 osd.2 (osd.2) 42 : cluster [DBG] 5.d scrub ok 2024-02-19T21:04:10.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:08.613279+0000 osd.0 (osd.0) 55 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:04:10.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:08.616388+0000 osd.0 (osd.0) 56 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:04:10.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:08.835147+0000 osd.3 (osd.3) 63 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:04:10.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:08.837644+0000 osd.3 (osd.3) 64 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:04:10.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:08.855155+0000 osd.7 (osd.7) 47 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:04:10.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:08.857969+0000 osd.7 (osd.7) 48 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:04:10.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:09.137437+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:04:10.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:09.139339+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:04:10.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:09.177230+0000 osd.5 (osd.5) 49 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:04:10.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:09.180155+0000 osd.5 (osd.5) 50 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:04:10.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:09.568584+0000 osd.0 (osd.0) 57 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:04:10.042 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:09.570964+0000 osd.0 (osd.0) 58 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:04:10.079 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:10.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:04:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:04:10.936 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:10.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:10.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:08.754191+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:04:10.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:08.757093+0000 osd.6 (osd.6) 38 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:04:10.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:08.829868+0000 osd.2 (osd.2) 43 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:04:10.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:08.830987+0000 osd.2 (osd.2) 44 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:04:10.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:08.946359+0000 mgr.y (mgr.14550) 118 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:04:10.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:10.945 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:08.754191+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:04:10.945 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:08.757093+0000 osd.6 (osd.6) 38 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:04:10.945 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:08.829868+0000 osd.2 (osd.2) 43 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:04:10.945 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:08.830987+0000 osd.2 (osd.2) 44 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:04:10.946 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:08.946359+0000 mgr.y (mgr.14550) 118 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:04:10.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.966+0000 7fadeed31700 1 -- 172.21.15.163:0/3617497169 >> v1:172.21.15.193:6789/0 conn(0x7fade81031c0 legacy=0x7fade8105640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:10.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.967+0000 7fadeed31700 1 -- 172.21.15.163:0/3617497169 shutdown_connections 2024-02-19T21:04:10.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.967+0000 7fadeed31700 1 -- 172.21.15.163:0/3617497169 >> 172.21.15.163:0/3617497169 conn(0x7fade80f24a0 msgr2=0x7fade80f48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:10.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.967+0000 7fadeed31700 1 -- 172.21.15.163:0/3617497169 shutdown_connections 2024-02-19T21:04:10.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.967+0000 7fadeed31700 1 -- 172.21.15.163:0/3617497169 wait complete. 2024-02-19T21:04:10.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.967+0000 7fadeed31700 1 librados: starting msgr at 2024-02-19T21:04:10.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.967+0000 7fadeed31700 1 librados: starting objecter 2024-02-19T21:04:10.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.967+0000 7fadeed31700 1 Processor -- start 2024-02-19T21:04:10.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.967+0000 7fadeed31700 1 -- start start 2024-02-19T21:04:10.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.967+0000 7fadeed31700 1 librados: setting wanted keys 2024-02-19T21:04:10.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.967+0000 7fadeed31700 1 librados: calling monclient init 2024-02-19T21:04:10.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.968+0000 7fadeed31700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fade81152b0 con 0x7fade80ff680 2024-02-19T21:04:10.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.968+0000 7fadeed31700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fade8116490 con 0x7fade80fc880 2024-02-19T21:04:10.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.968+0000 7fadeed31700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fade8117690 con 0x7fade81031c0 2024-02-19T21:04:10.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.968+0000 7fadecacd700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fade80fc880 0x7fade80f85d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:39726/0 (socket says 172.21.15.163:39726) 2024-02-19T21:04:10.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.968+0000 7fade7fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fade80ff680 0x7fade80f8ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:42904/0 (socket says 172.21.15.163:42904) 2024-02-19T21:04:10.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.968+0000 7fadecacd700 1 -- 172.21.15.163:0/2607116942 learned_addr learned my addr 172.21.15.163:0/2607116942 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:04:10.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.968+0000 7faded2ce700 1 --1- 172.21.15.163:0/2607116942 >> v1:172.21.15.163:6790/0 conn(0x7fade81031c0 0x7fade8113970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:41374/0 (socket says 172.21.15.163:41374) 2024-02-19T21:04:10.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.968+0000 7fade5ffb700 10 client.?.objecter ms_handle_connect 0x7fade80fc880 2024-02-19T21:04:10.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.968+0000 7fade5ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.968+0000 7fade5ffb700 10 client.?.objecter ms_handle_connect 0x7fade81031c0 2024-02-19T21:04:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.968+0000 7fade5ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.968+0000 7fade5ffb700 10 client.?.objecter ms_handle_connect 0x7fade80ff680 2024-02-19T21:04:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.968+0000 7fade5ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.968+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983213304 0 0) 0x7fade8116490 con 0x7fade80fc880 2024-02-19T21:04:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.968+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd80033f0 con 0x7fade80fc880 2024-02-19T21:04:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 724796525 0 0) 0x7fadd80033f0 con 0x7fade80fc880 2024-02-19T21:04:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fade8116490 con 0x7fade80fc880 2024-02-19T21:04:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287603883 0 0) 0x7fade81152b0 con 0x7fade80ff680 2024-02-19T21:04:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd80033f0 con 0x7fade80ff680 2024-02-19T21:04:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103662532 0 0) 0x7fade8117690 con 0x7fade81031c0 2024-02-19T21:04:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade81152b0 con 0x7fade81031c0 2024-02-19T21:04:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7faddc001980 con 0x7fade80fc880 2024-02-19T21:04:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3030441407 0 0) 0x7fade81152b0 con 0x7fade81031c0 2024-02-19T21:04:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fade8117690 con 0x7fade81031c0 2024-02-19T21:04:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fade0003110 con 0x7fade81031c0 2024-02-19T21:04:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3820411933 0 0) 0x7fadd80033f0 con 0x7fade80ff680 2024-02-19T21:04:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fade81152b0 con 0x7fade80ff680 2024-02-19T21:04:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fadd40017c0 con 0x7fade80ff680 2024-02-19T21:04:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1171053441 0 0) 0x7fade8116490 con 0x7fade80fc880 2024-02-19T21:04:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 >> v1:172.21.15.163:6790/0 conn(0x7fade81031c0 legacy=0x7fade8113970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 >> v1:172.21.15.163:6789/0 conn(0x7fade80ff680 legacy=0x7fade80f8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade8118890 con 0x7fade80fc880 2024-02-19T21:04:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.969+0000 7fadeed31700 1 -- 172.21.15.163:0/2607116942 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade8117880 con 0x7fade80fc880 2024-02-19T21:04:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.970+0000 7fadeed31700 10 client.24766.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.970+0000 7fadeed31700 1 -- 172.21.15.163:0/2607116942 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fade8117cf0 con 0x7fade80fc880 2024-02-19T21:04:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.970+0000 7fadeed31700 1 librados: init done 2024-02-19T21:04:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.970+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7faddc001f80 con 0x7fade80fc880 2024-02-19T21:04:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.970+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7faddc005a50 con 0x7fade80fc880 2024-02-19T21:04:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.971+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7faddc001ae0 con 0x7fade80fc880 2024-02-19T21:04:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.971+0000 7fadeed31700 1 -- 172.21.15.163:0/2607116942 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadcc004f20 con 0x7fade80fc880 2024-02-19T21:04:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.971+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(91..91 src has 32..91) v4 ==== 6172+0+0 (unknown 1247502107 0 0) 0x7faddc0784d0 con 0x7fade80fc880 2024-02-19T21:04:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.971+0000 7fade5ffb700 10 client.24766.objecter ms_dispatch 0x7fade8001550 osd_map(91..91 src has 32..91) v4 2024-02-19T21:04:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.971+0000 7fade5ffb700 3 client.24766.objecter handle_osd_map got epochs [91,91] > 0 2024-02-19T21:04:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.971+0000 7fade5ffb700 3 client.24766.objecter handle_osd_map decoding full epoch 91 2024-02-19T21:04:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.972+0000 7fade5ffb700 20 client.24766.objecter dump_active .. 0 homeless 2024-02-19T21:04:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.972+0000 7fade5ffb700 10 client.24766.objecter ms_handle_connect 0x7fadd8068090 2024-02-19T21:04:10.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:10.976+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7faddc04c020 con 0x7fade80fc880 2024-02-19T21:04:11.046 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:11.063 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:08.754191+0000 osd.6 (osd.6) 37 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:04:11.063 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:08.757093+0000 osd.6 (osd.6) 38 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:04:11.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:08.829868+0000 osd.2 (osd.2) 43 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:04:11.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:08.830987+0000 osd.2 (osd.2) 44 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:04:11.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:08.946359+0000 mgr.y (mgr.14550) 118 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:04:11.079 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:11.100 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:11.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:11.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:11.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:11.209+0000 7fadeed31700 1 -- 172.21.15.163:0/2607116942 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd pg-upmap", "pgid": "2.d", "id": [1, 4, 2]} v 0) v1 -- 0x7fadcc005750 con 0x7fade80fc880 2024-02-19T21:04:11.203 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:11.234 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:11.363 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:11.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:11.467 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:11.502 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:11.575 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:11.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:11.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:11.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:11.651+0000 7fade5ffb700 1 -- 172.21.15.163:0/2607116942 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "2.d", "id": [1, 4, 2]}]=0 set 2.d pg_upmap mapping to [1,4,2] v92) v1 ==== 127+0+0 (unknown 1025431468 0 0) 0x7faddc04a430 con 0x7fade80fc880 2024-02-19T21:04:11.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:11.654+0000 7fadeed31700 10 librados: watch_flush enter 2024-02-19T21:04:11.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:11.657+0000 7fadeed31700 10 librados: watch_flush exit 2024-02-19T21:04:11.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:11.657+0000 7fadeed31700 20 client.24766.objecter shutdown clearing up homeless session... 2024-02-19T21:04:11.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:11.657+0000 7fadeed31700 10 client.24766.objecter successfully canceled tick 2024-02-19T21:04:11.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:11.657+0000 7fadeed31700 1 -- 172.21.15.163:0/2607116942 >> v1:172.21.15.163:6800/1552514681 conn(0x7fadd8068090 legacy=0x7fadd806a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:11.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:11.657+0000 7fadeed31700 1 -- 172.21.15.163:0/2607116942 >> v1:172.21.15.193:6789/0 conn(0x7fade80fc880 legacy=0x7fade80f85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:11.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:11.658+0000 7fadeed31700 1 -- 172.21.15.163:0/2607116942 shutdown_connections 2024-02-19T21:04:11.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:11.658+0000 7fadeed31700 1 -- 172.21.15.163:0/2607116942 >> 172.21.15.163:0/2607116942 conn(0x7fade80f24a0 msgr2=0x7fade8060b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:11.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:11.658+0000 7fadeed31700 1 -- 172.21.15.163:0/2607116942 shutdown_connections 2024-02-19T21:04:11.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:11.658+0000 7fadeed31700 1 -- 172.21.15.163:0/2607116942 wait complete. 2024-02-19T21:04:11.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:11.658+0000 7fadeed31700 1 librados: shutdown 2024-02-19T21:04:11.653 INFO:teuthology.orchestra.run.smithi163.stderr:set 2.d pg_upmap mapping to [1,4,2] 2024-02-19T21:04:11.670 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:04:11.680 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:11.712 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:11.733 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:11.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:11.775 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:10.049658+0000 mgr.y (mgr.14550) 119 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:11.775 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:10.861784+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:04:11.775 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:10.862919+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:04:11.775 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:10.888206+0000 osd.7 (osd.7) 49 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:04:11.775 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:10.891108+0000 osd.7 (osd.7) 50 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:04:11.775 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:11.163292+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:04:11.775 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:11.165192+0000 osd.4 (osd.4) 58 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:04:11.775 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:11.211491+0000 mon.b (mon.1) 80 : audit [INF] from='client.? v1:172.21.15.163:0/2607116942' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.d", "id": [1, 4, 2]}]: dispatch 2024-02-19T21:04:11.775 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:11.211974+0000 mon.a (mon.0) 733 : audit [INF] from='client.24766 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.d", "id": [1, 4, 2]}]: dispatch 2024-02-19T21:04:11.775 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:11.300567+0000 osd.1 (osd.1) 53 : cluster [DBG] 2.a scrub starts 2024-02-19T21:04:11.775 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:11.303563+0000 osd.1 (osd.1) 54 : cluster [DBG] 2.a scrub ok 2024-02-19T21:04:11.776 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:10.049658+0000 mgr.y (mgr.14550) 119 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:11.776 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:10.861784+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:04:11.776 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:10.862919+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:04:11.776 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:10.888206+0000 osd.7 (osd.7) 49 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:04:11.776 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:10.891108+0000 osd.7 (osd.7) 50 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:04:11.776 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:11.163292+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:04:11.776 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:11.165192+0000 osd.4 (osd.4) 58 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:04:11.776 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:11.211491+0000 mon.b (mon.1) 80 : audit [INF] from='client.? v1:172.21.15.163:0/2607116942' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.d", "id": [1, 4, 2]}]: dispatch 2024-02-19T21:04:11.776 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:11.211974+0000 mon.a (mon.0) 733 : audit [INF] from='client.24766 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.d", "id": [1, 4, 2]}]: dispatch 2024-02-19T21:04:11.776 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:11.300567+0000 osd.1 (osd.1) 53 : cluster [DBG] 2.a scrub starts 2024-02-19T21:04:11.776 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:11.303563+0000 osd.1 (osd.1) 54 : cluster [DBG] 2.a scrub ok 2024-02-19T21:04:11.837 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:04:11.866 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:11.886 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:04:11.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:11.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:04:11.990 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:04:12.016 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:10.049658+0000 mgr.y (mgr.14550) 119 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:12.016 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:10.861784+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:04:12.016 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:10.862919+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:04:12.016 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:10.888206+0000 osd.7 (osd.7) 49 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:04:12.016 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:10.891108+0000 osd.7 (osd.7) 50 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:04:12.016 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:11.163292+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:04:12.016 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:11.165192+0000 osd.4 (osd.4) 58 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:04:12.016 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:11.211491+0000 mon.b (mon.1) 80 : audit [INF] from='client.? v1:172.21.15.163:0/2607116942' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.d", "id": [1, 4, 2]}]: dispatch 2024-02-19T21:04:12.016 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:11.211974+0000 mon.a (mon.0) 733 : audit [INF] from='client.24766 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.d", "id": [1, 4, 2]}]: dispatch 2024-02-19T21:04:12.016 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:11.300567+0000 osd.1 (osd.1) 53 : cluster [DBG] 2.a scrub starts 2024-02-19T21:04:12.016 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:11.303563+0000 osd.1 (osd.1) 54 : cluster [DBG] 2.a scrub ok 2024-02-19T21:04:12.018 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:12.036 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:04:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:12.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:04:12.140 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:04:12.168 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:12.187 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:04:12.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:12.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:04:12.290 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:12.317 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:12.336 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:12.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:12.439 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:12.465 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:12.484 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:12.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:12.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:12.587 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:12.622 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:12.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:10.946968+0000 mgr.y (mgr.14550) 120 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:04:12.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:11.651712+0000 mon.a (mon.0) 734 : audit [INF] from='client.24766 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.d", "id": [1, 4, 2]}]': finished 2024-02-19T21:04:12.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:11.651775+0000 mon.a (mon.0) 735 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-19T21:04:12.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:11.840092+0000 osd.3 (osd.3) 67 : cluster [DBG] 5.17 scrub starts 2024-02-19T21:04:12.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:11.841177+0000 osd.3 (osd.3) 68 : cluster [DBG] 5.17 scrub ok 2024-02-19T21:04:12.910 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:11.897706+0000 osd.7 (osd.7) 51 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:04:12.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:11.900618+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:04:12.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:12.132049+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:04:12.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:12.133624+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:04:12.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:12.302959+0000 osd.1 (osd.1) 55 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:04:12.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:12.305880+0000 osd.1 (osd.1) 56 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:04:12.911 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:10.946968+0000 mgr.y (mgr.14550) 120 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:04:12.911 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:11.651712+0000 mon.a (mon.0) 734 : audit [INF] from='client.24766 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.d", "id": [1, 4, 2]}]': finished 2024-02-19T21:04:12.911 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:11.651775+0000 mon.a (mon.0) 735 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-19T21:04:12.911 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:11.840092+0000 osd.3 (osd.3) 67 : cluster [DBG] 5.17 scrub starts 2024-02-19T21:04:12.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:11.841177+0000 osd.3 (osd.3) 68 : cluster [DBG] 5.17 scrub ok 2024-02-19T21:04:12.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:11.897706+0000 osd.7 (osd.7) 51 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:04:12.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:11.900618+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:04:12.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:12.132049+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:04:12.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:12.133624+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:04:12.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:12.302959+0000 osd.1 (osd.1) 55 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:04:12.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:12.305880+0000 osd.1 (osd.1) 56 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:04:13.063 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:10.946968+0000 mgr.y (mgr.14550) 120 : cluster [DBG] pgmap v98: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:04:13.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:11.651712+0000 mon.a (mon.0) 734 : audit [INF] from='client.24766 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.d", "id": [1, 4, 2]}]': finished 2024-02-19T21:04:13.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:11.651775+0000 mon.a (mon.0) 735 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-02-19T21:04:13.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:11.840092+0000 osd.3 (osd.3) 67 : cluster [DBG] 5.17 scrub starts 2024-02-19T21:04:13.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:11.841177+0000 osd.3 (osd.3) 68 : cluster [DBG] 5.17 scrub ok 2024-02-19T21:04:13.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:11.897706+0000 osd.7 (osd.7) 51 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:04:13.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:11.900618+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:04:13.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:12.132049+0000 osd.4 (osd.4) 59 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:04:13.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:12.133624+0000 osd.4 (osd.4) 60 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:04:13.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:12.302959+0000 osd.1 (osd.1) 55 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:04:13.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:12.305880+0000 osd.1 (osd.1) 56 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:04:13.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.214+0000 7f9254a2e700 1 -- 172.21.15.163:0/2669234242 >> v1:172.21.15.163:6790/0 conn(0x7f92500ff670 legacy=0x7f9250101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:13.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.214+0000 7f9254a2e700 1 -- 172.21.15.163:0/2669234242 shutdown_connections 2024-02-19T21:04:13.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.214+0000 7f9254a2e700 1 -- 172.21.15.163:0/2669234242 >> 172.21.15.163:0/2669234242 conn(0x7f92500f2470 msgr2=0x7f92500f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:13.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.214+0000 7f9254a2e700 1 -- 172.21.15.163:0/2669234242 shutdown_connections 2024-02-19T21:04:13.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.214+0000 7f9254a2e700 1 -- 172.21.15.163:0/2669234242 wait complete. 2024-02-19T21:04:13.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.215+0000 7f9254a2e700 1 librados: starting msgr at 2024-02-19T21:04:13.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.215+0000 7f9254a2e700 1 librados: starting objecter 2024-02-19T21:04:13.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.215+0000 7f9254a2e700 1 Processor -- start 2024-02-19T21:04:13.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.215+0000 7f9254a2e700 1 -- start start 2024-02-19T21:04:13.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.215+0000 7f9254a2e700 1 librados: setting wanted keys 2024-02-19T21:04:13.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.215+0000 7f9254a2e700 1 librados: calling monclient init 2024-02-19T21:04:13.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.215+0000 7f9254a2e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f925006a150 con 0x7f92500fc870 2024-02-19T21:04:13.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.215+0000 7f9254a2e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f925006a320 con 0x7f92501031b0 2024-02-19T21:04:13.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.215+0000 7f9254a2e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f925006a510 con 0x7f92500ff670 2024-02-19T21:04:13.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f924dd9b700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f92500ff670 0x7f92500f8b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:41386/0 (socket says 172.21.15.163:41386) 2024-02-19T21:04:13.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f924dd9b700 1 -- 172.21.15.163:0/2415483144 learned_addr learned my addr 172.21.15.163:0/2415483144 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:04:13.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 10 client.?.objecter ms_handle_connect 0x7f92500ff670 2024-02-19T21:04:13.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:13.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409820829 0 0) 0x7f925006a510 con 0x7f92500ff670 2024-02-19T21:04:13.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9238003030 con 0x7f92500ff670 2024-02-19T21:04:13.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 10 client.?.objecter ms_handle_connect 0x7f92501031b0 2024-02-19T21:04:13.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:13.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 10 client.?.objecter ms_handle_connect 0x7f92500fc870 2024-02-19T21:04:13.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:13.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2451138783 0 0) 0x7f9238003030 con 0x7f92500ff670 2024-02-19T21:04:13.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f925006a510 con 0x7f92500ff670 2024-02-19T21:04:13.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9244002e00 con 0x7f92500ff670 2024-02-19T21:04:13.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997948973 0 0) 0x7f925006a320 con 0x7f92501031b0 2024-02-19T21:04:13.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9238003030 con 0x7f92501031b0 2024-02-19T21:04:13.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736990778 0 0) 0x7f925006a150 con 0x7f92500fc870 2024-02-19T21:04:13.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f925006a320 con 0x7f92500fc870 2024-02-19T21:04:13.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1145464483 0 0) 0x7f925006a510 con 0x7f92500ff670 2024-02-19T21:04:13.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 >> v1:172.21.15.193:6789/0 conn(0x7f92501031b0 legacy=0x7f9250069820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:13.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 >> v1:172.21.15.163:6789/0 conn(0x7f92500fc870 legacy=0x7f92500f8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:13.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.216+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925006a700 con 0x7f92500ff670 2024-02-19T21:04:13.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.217+0000 7f9254a2e700 1 -- 172.21.15.163:0/2415483144 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925006a9b0 con 0x7f92500ff670 2024-02-19T21:04:13.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.217+0000 7f9254a2e700 10 client.34772.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:13.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.217+0000 7f9254a2e700 1 -- 172.21.15.163:0/2415483144 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9250127000 con 0x7f92500ff670 2024-02-19T21:04:13.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.217+0000 7f9254a2e700 1 librados: init done 2024-02-19T21:04:13.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.218+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f92440024f0 con 0x7f92500ff670 2024-02-19T21:04:13.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.218+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9244003920 con 0x7f92500ff670 2024-02-19T21:04:13.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.219+0000 7f9254a2e700 1 -- 172.21.15.163:0/2415483144 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925005f700 con 0x7f92500ff670 2024-02-19T21:04:13.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.220+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9244002050 con 0x7f92500ff670 2024-02-19T21:04:13.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.220+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(93..93 src has 43..93) v4 ==== 6205+0+0 (unknown 2034849678 0 0) 0x7f9244078a80 con 0x7f92500ff670 2024-02-19T21:04:13.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.220+0000 7f923f7fe700 10 client.34772.objecter ms_dispatch 0x7f9250001550 osd_map(93..93 src has 43..93) v4 2024-02-19T21:04:13.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.220+0000 7f923f7fe700 3 client.34772.objecter handle_osd_map got epochs [93,93] > 0 2024-02-19T21:04:13.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.220+0000 7f923f7fe700 3 client.34772.objecter handle_osd_map decoding full epoch 93 2024-02-19T21:04:13.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.220+0000 7f923f7fe700 20 client.34772.objecter dump_active .. 0 homeless 2024-02-19T21:04:13.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.220+0000 7f923f7fe700 10 client.34772.objecter ms_handle_connect 0x7f9238063bc0 2024-02-19T21:04:13.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.228+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f924404c670 con 0x7f92500ff670 2024-02-19T21:04:13.479 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:13.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:13.533 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:04:13.533 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:04:13.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.554+0000 7f9254a2e700 1 -- 172.21.15.163:0/2415483144 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f92500f80a0 con 0x7f92500ff670 2024-02-19T21:04:13.557 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:04:13] "GET /metrics HTTP/1.1" 200 172251 "" "Prometheus/2.33.4" 2024-02-19T21:04:13.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:04:13.617 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:13.638 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:04:13.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:13.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:04:13.742 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:13.773 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:13.855 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:11.751633+0000 osd.6 (osd.6) 39 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:04:13.855 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:11.754520+0000 osd.6 (osd.6) 40 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:04:13.855 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:11.887614+0000 osd.2 (osd.2) 45 : cluster [DBG] 2.e scrub starts 2024-02-19T21:04:13.855 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:11.889771+0000 osd.2 (osd.2) 46 : cluster [DBG] 2.e scrub ok 2024-02-19T21:04:13.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:12.665857+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-19T21:04:13.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:12.947379+0000 mgr.y (mgr.14550) 121 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:04:13.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:13.202716+0000 osd.5 (osd.5) 51 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:04:13.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:13.205592+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:04:13.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:13.334129+0000 osd.1 (osd.1) 57 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:04:13.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:13.337491+0000 osd.1 (osd.1) 58 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:04:13.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:13.555911+0000 mon.c (mon.2) 30 : audit [INF] from='client.? v1:172.21.15.163:0/2415483144' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:13.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:13.556257+0000 mon.a (mon.0) 737 : audit [INF] from='client.34772 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:13.856 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:11.751633+0000 osd.6 (osd.6) 39 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:04:13.856 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:11.754520+0000 osd.6 (osd.6) 40 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:04:13.856 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:11.887614+0000 osd.2 (osd.2) 45 : cluster [DBG] 2.e scrub starts 2024-02-19T21:04:13.856 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:11.889771+0000 osd.2 (osd.2) 46 : cluster [DBG] 2.e scrub ok 2024-02-19T21:04:13.856 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:12.665857+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-19T21:04:13.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:12.947379+0000 mgr.y (mgr.14550) 121 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:04:13.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:13.202716+0000 osd.5 (osd.5) 51 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:04:13.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:13.205592+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:04:13.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:13.334129+0000 osd.1 (osd.1) 57 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:04:13.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:13.337491+0000 osd.1 (osd.1) 58 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:04:13.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:13.555911+0000 mon.c (mon.2) 30 : audit [INF] from='client.? v1:172.21.15.163:0/2415483144' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:13.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:13.556257+0000 mon.a (mon.0) 737 : audit [INF] from='client.34772 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:13.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.984+0000 7f923f7fe700 1 -- 172.21.15.163:0/2415483144 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v94) v1 ==== 101+0+0 (unknown 2024750469 0 0) 0x7f924404aae0 con 0x7f92500ff670 2024-02-19T21:04:13.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.986+0000 7f9254a2e700 10 librados: watch_flush enter 2024-02-19T21:04:13.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.986+0000 7f9254a2e700 10 librados: watch_flush exit 2024-02-19T21:04:13.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.986+0000 7f9254a2e700 20 client.34772.objecter shutdown clearing up homeless session... 2024-02-19T21:04:13.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.986+0000 7f9254a2e700 10 client.34772.objecter successfully canceled tick 2024-02-19T21:04:13.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.986+0000 7f9254a2e700 1 -- 172.21.15.163:0/2415483144 >> v1:172.21.15.163:6800/1552514681 conn(0x7f9238063bc0 legacy=0x7f9238066080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:13.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.986+0000 7f9254a2e700 1 -- 172.21.15.163:0/2415483144 >> v1:172.21.15.163:6790/0 conn(0x7f92500ff670 legacy=0x7f92500f8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:13.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.987+0000 7f9254a2e700 1 -- 172.21.15.163:0/2415483144 shutdown_connections 2024-02-19T21:04:13.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.987+0000 7f9254a2e700 1 -- 172.21.15.163:0/2415483144 >> 172.21.15.163:0/2415483144 conn(0x7f92500f2470 msgr2=0x7f92500601c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:13.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.987+0000 7f9254a2e700 1 -- 172.21.15.163:0/2415483144 shutdown_connections 2024-02-19T21:04:13.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.987+0000 7f9254a2e700 1 -- 172.21.15.163:0/2415483144 wait complete. 2024-02-19T21:04:13.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:13.987+0000 7f9254a2e700 1 librados: shutdown 2024-02-19T21:04:13.981 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:04:14.063 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:11.751633+0000 osd.6 (osd.6) 39 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:04:14.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:11.754520+0000 osd.6 (osd.6) 40 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:04:14.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:11.887614+0000 osd.2 (osd.2) 45 : cluster [DBG] 2.e scrub starts 2024-02-19T21:04:14.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:11.889771+0000 osd.2 (osd.2) 46 : cluster [DBG] 2.e scrub ok 2024-02-19T21:04:14.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:12.665857+0000 mon.a (mon.0) 736 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-02-19T21:04:14.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:12.947379+0000 mgr.y (mgr.14550) 121 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:04:14.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:13.202716+0000 osd.5 (osd.5) 51 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:04:14.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:13.205592+0000 osd.5 (osd.5) 52 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:04:14.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:13.334129+0000 osd.1 (osd.1) 57 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:04:14.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:13.337491+0000 osd.1 (osd.1) 58 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:04:14.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:13.555911+0000 mon.c (mon.2) 30 : audit [INF] from='client.? v1:172.21.15.163:0/2415483144' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:14.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:13.556257+0000 mon.a (mon.0) 737 : audit [INF] from='client.34772 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:14.422 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:14.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:14.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:14.532 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:14.561 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:14.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:13.815653+0000 osd.6 (osd.6) 41 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:04:14.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:13.816541+0000 osd.6 (osd.6) 42 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:04:14.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:13.938228+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:04:14.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:13.941128+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:04:14.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:13.980200+0000 mon.a (mon.0) 738 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:04:14.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:13.980234+0000 mon.a (mon.0) 739 : cluster [INF] Cluster is now healthy 2024-02-19T21:04:14.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:13.984435+0000 mon.a (mon.0) 740 : audit [INF] from='client.34772 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:04:14.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:13.984501+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-02-19T21:04:14.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:14.212120+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:04:14.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:14.215011+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:04:14.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:13.815653+0000 osd.6 (osd.6) 41 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:04:14.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:13.816541+0000 osd.6 (osd.6) 42 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:04:14.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:13.938228+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:04:14.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:13.941128+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:04:14.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:13.980200+0000 mon.a (mon.0) 738 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:04:14.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:13.980234+0000 mon.a (mon.0) 739 : cluster [INF] Cluster is now healthy 2024-02-19T21:04:14.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:13.984435+0000 mon.a (mon.0) 740 : audit [INF] from='client.34772 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:04:14.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:13.984501+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-02-19T21:04:14.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:14.212120+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:04:14.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:14.215011+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:04:14.917 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:14.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:14.952 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:15.024 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:04:15.055 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:13.815653+0000 osd.6 (osd.6) 41 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:04:15.055 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:13.816541+0000 osd.6 (osd.6) 42 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:04:15.055 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:13.938228+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:04:15.055 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:13.941128+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:04:15.055 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:13.980200+0000 mon.a (mon.0) 738 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:04:15.055 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:13.980234+0000 mon.a (mon.0) 739 : cluster [INF] Cluster is now healthy 2024-02-19T21:04:15.055 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:13.984435+0000 mon.a (mon.0) 740 : audit [INF] from='client.34772 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:04:15.055 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:13.984501+0000 mon.a (mon.0) 741 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-02-19T21:04:15.055 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:14.212120+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:04:15.055 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:14.215011+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:04:15.057 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:15.078 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:04:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:15.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:04:15.182 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:15.211 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:15.230 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:15.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:15.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:15.318 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-19T21:04:15.334 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:15.393 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:15.449 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:15.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:15.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:15.556 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:15.585 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:15.605 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:15.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:15.708 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:04:15.736 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:15.756 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:04:15.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:15.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:04:15.860 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:15.887 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:14.823290+0000 osd.3 (osd.3) 69 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:04:15.887 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:14.826265+0000 osd.3 (osd.3) 70 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:04:15.887 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:14.948151+0000 mgr.y (mgr.14550) 122 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:04:15.887 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:15.097992+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.17 scrub starts 2024-02-19T21:04:15.887 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:15.099201+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.17 scrub ok 2024-02-19T21:04:15.887 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:15.184022+0000 osd.5 (osd.5) 55 : cluster [DBG] 4.15 scrub starts 2024-02-19T21:04:15.887 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:15.187577+0000 osd.5 (osd.5) 56 : cluster [DBG] 4.15 scrub ok 2024-02-19T21:04:15.887 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:14.823290+0000 osd.3 (osd.3) 69 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:04:15.887 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:14.826265+0000 osd.3 (osd.3) 70 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:04:15.887 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:14.948151+0000 mgr.y (mgr.14550) 122 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:04:15.888 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:15.097992+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.17 scrub starts 2024-02-19T21:04:15.888 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:15.099201+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.17 scrub ok 2024-02-19T21:04:15.888 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:15.184022+0000 osd.5 (osd.5) 55 : cluster [DBG] 4.15 scrub starts 2024-02-19T21:04:15.888 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:15.187577+0000 osd.5 (osd.5) 56 : cluster [DBG] 4.15 scrub ok 2024-02-19T21:04:15.890 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:16.063 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:14.823290+0000 osd.3 (osd.3) 69 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:04:16.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:14.826265+0000 osd.3 (osd.3) 70 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:04:16.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:14.948151+0000 mgr.y (mgr.14550) 122 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:04:16.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:15.097992+0000 osd.4 (osd.4) 61 : cluster [DBG] 6.17 scrub starts 2024-02-19T21:04:16.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:15.099201+0000 osd.4 (osd.4) 62 : cluster [DBG] 6.17 scrub ok 2024-02-19T21:04:16.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:15.184022+0000 osd.5 (osd.5) 55 : cluster [DBG] 4.15 scrub starts 2024-02-19T21:04:16.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:15.187577+0000 osd.5 (osd.5) 56 : cluster [DBG] 4.15 scrub ok 2024-02-19T21:04:16.107 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:16.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:16.058209+0000 osd.4 (osd.4) 63 : cluster [DBG] 5.e scrub starts 2024-02-19T21:04:16.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:16.059643+0000 osd.4 (osd.4) 64 : cluster [DBG] 5.e scrub ok 2024-02-19T21:04:16.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:16.214847+0000 osd.5 (osd.5) 57 : cluster [DBG] 5.f scrub starts 2024-02-19T21:04:16.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:16.217739+0000 osd.5 (osd.5) 58 : cluster [DBG] 5.f scrub ok 2024-02-19T21:04:16.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:16.273518+0000 osd.1 (osd.1) 59 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:04:16.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:16.276511+0000 osd.1 (osd.1) 60 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:04:16.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:16.058209+0000 osd.4 (osd.4) 63 : cluster [DBG] 5.e scrub starts 2024-02-19T21:04:16.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:16.059643+0000 osd.4 (osd.4) 64 : cluster [DBG] 5.e scrub ok 2024-02-19T21:04:16.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:16.214847+0000 osd.5 (osd.5) 57 : cluster [DBG] 5.f scrub starts 2024-02-19T21:04:16.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:16.217739+0000 osd.5 (osd.5) 58 : cluster [DBG] 5.f scrub ok 2024-02-19T21:04:16.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:16.273518+0000 osd.1 (osd.1) 59 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:04:16.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:16.276511+0000 osd.1 (osd.1) 60 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:04:16.748 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:16.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:16.801 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:04:16.855 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:16.897 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:16.971 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:16.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:16.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:16.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.990+0000 7f86cea14700 1 -- 172.21.15.163:0/1249651218 >> v1:172.21.15.193:6789/0 conn(0x7f86c81031b0 legacy=0x7f86c8105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:16.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.990+0000 7f86cea14700 1 -- 172.21.15.163:0/1249651218 shutdown_connections 2024-02-19T21:04:16.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.990+0000 7f86cea14700 1 -- 172.21.15.163:0/1249651218 >> 172.21.15.163:0/1249651218 conn(0x7f86c80f2470 msgr2=0x7f86c80f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:16.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.990+0000 7f86cea14700 1 -- 172.21.15.163:0/1249651218 shutdown_connections 2024-02-19T21:04:16.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.990+0000 7f86cea14700 1 -- 172.21.15.163:0/1249651218 wait complete. 2024-02-19T21:04:16.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.991+0000 7f86cea14700 1 librados: starting msgr at 2024-02-19T21:04:16.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.991+0000 7f86cea14700 1 librados: starting objecter 2024-02-19T21:04:16.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.991+0000 7f86cea14700 1 Processor -- start 2024-02-19T21:04:16.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.991+0000 7f86cea14700 1 -- start start 2024-02-19T21:04:16.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.991+0000 7f86cea14700 1 librados: setting wanted keys 2024-02-19T21:04:16.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.991+0000 7f86cea14700 1 librados: calling monclient init 2024-02-19T21:04:16.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.991+0000 7f86cea14700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86c8110890 con 0x7f86c80fc870 2024-02-19T21:04:16.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.991+0000 7f86cea14700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86c8110a20 con 0x7f86c80ff670 2024-02-19T21:04:16.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.991+0000 7f86cea14700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86c8110bf0 con 0x7f86c81031b0 2024-02-19T21:04:16.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c77fe700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f86c80ff670 0x7f86c80f8b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:57558/0 (socket says 172.21.15.163:57558) 2024-02-19T21:04:16.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86ccfb1700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f86c81031b0 0x7f86c811c610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:47822/0 (socket says 172.21.15.163:47822) 2024-02-19T21:04:16.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c77fe700 1 -- 172.21.15.163:0/1352552202 learned_addr learned my addr 172.21.15.163:0/1352552202 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:04:16.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c57fa700 10 client.?.objecter ms_handle_connect 0x7f86c80ff670 2024-02-19T21:04:16.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c57fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:16.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c57fa700 10 client.?.objecter ms_handle_connect 0x7f86c80fc870 2024-02-19T21:04:16.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c57fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:16.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c57fa700 10 client.?.objecter ms_handle_connect 0x7f86c81031b0 2024-02-19T21:04:16.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c57fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:16.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372531050 0 0) 0x7f86c8110a20 con 0x7f86c80ff670 2024-02-19T21:04:16.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b0003800 con 0x7f86c80ff670 2024-02-19T21:04:16.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1533261830 0 0) 0x7f86b0003800 con 0x7f86c80ff670 2024-02-19T21:04:16.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86c8110a20 con 0x7f86c80ff670 2024-02-19T21:04:16.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989733352 0 0) 0x7f86c8110bf0 con 0x7f86c81031b0 2024-02-19T21:04:16.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b0003800 con 0x7f86c81031b0 2024-02-19T21:04:16.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f86bc001940 con 0x7f86c80ff670 2024-02-19T21:04:16.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056548061 0 0) 0x7f86c8110890 con 0x7f86c80fc870 2024-02-19T21:04:16.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.992+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c8110bf0 con 0x7f86c80fc870 2024-02-19T21:04:16.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.993+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1739802065 0 0) 0x7f86c8110a20 con 0x7f86c80ff670 2024-02-19T21:04:16.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.993+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 >> v1:172.21.15.163:6790/0 conn(0x7f86c81031b0 legacy=0x7f86c811c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:16.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.993+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 >> v1:172.21.15.163:6789/0 conn(0x7f86c80fc870 legacy=0x7f86c80f8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:16.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.993+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c8120da0 con 0x7f86c80ff670 2024-02-19T21:04:16.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.993+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c811dd90 con 0x7f86c80ff670 2024-02-19T21:04:16.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.993+0000 7f86cea14700 10 client.24778.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:16.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.993+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c811e280 con 0x7f86c80ff670 2024-02-19T21:04:16.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.993+0000 7f86cea14700 1 librados: init done 2024-02-19T21:04:16.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.995+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f86bc002470 con 0x7f86c80ff670 2024-02-19T21:04:16.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.995+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f86bc005bc0 con 0x7f86c80ff670 2024-02-19T21:04:16.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.995+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f86bc006e50 con 0x7f86c80ff670 2024-02-19T21:04:16.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.995+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(94..94 src has 43..94) v4 ==== 6205+0+0 (unknown 3021565180 0 0) 0x7f86bc0795a0 con 0x7f86c80ff670 2024-02-19T21:04:16.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.995+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 osd_map(94..94 src has 43..94) v4 2024-02-19T21:04:16.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.995+0000 7f86c57fa700 3 client.24778.objecter handle_osd_map got epochs [94,94] > 0 2024-02-19T21:04:16.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.995+0000 7f86c57fa700 3 client.24778.objecter handle_osd_map decoding full epoch 94 2024-02-19T21:04:16.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.995+0000 7f86c57fa700 20 client.24778.objecter dump_active .. 0 homeless 2024-02-19T21:04:16.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.996+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f86c805f700 con 0x7f86c80ff670 2024-02-19T21:04:16.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.996+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v94) v1 ==== 54+0+16 (unknown 244545362 0 2420000192) 0x7f86bc04d150 con 0x7f86c80ff670 2024-02-19T21:04:16.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.996+0000 7f86c57fa700 10 client.24778.objecter ms_handle_connect 0x7f86b0063ef0 2024-02-19T21:04:16.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.996+0000 7f86cea14700 10 client.24778.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:16.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.996+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 1 2024-02-19T21:04:16.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.996+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80f80a0 osd=0 3 2024-02-19T21:04:16.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.996+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80f80a0 osd=0 3 2024-02-19T21:04:16.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.996+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80f80a0 osd=0 3 2024-02-19T21:04:16.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.996+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 1 2024-02-19T21:04:16.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.996+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c80f80a0 osd=0 3 2024-02-19T21:04:16.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.996+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 0 1 2024-02-19T21:04:16.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.996+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80f80a0 osd=0 4 2024-02-19T21:04:16.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.996+0000 7f86cea14700 10 client.24778.objecter _send_command 1 2024-02-19T21:04:16.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.996+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86c8108e10 con 0x7f86c80598f0 2024-02-19T21:04:16.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.997+0000 7f86c57fa700 10 client.24778.objecter ms_handle_connect 0x7f86c80598f0 2024-02-19T21:04:16.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.997+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f86c8108e10 con 0x7f86c80598f0 2024-02-19T21:04:16.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.997+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:04:16.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.997+0000 7f86c57fa700 10 client.24778.objecter _finish_command 1 = system:0 2024-02-19T21:04:16.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.997+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c80f80a0 osd=0 4 2024-02-19T21:04:16.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:16.997+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 0 1 2024-02-19T21:04:17.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.024+0000 7f86cea14700 10 client.24778.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:17.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.024+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 2 2024-02-19T21:04:17.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.024+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80f80a0 osd=0 3 2024-02-19T21:04:17.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.024+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80f80a0 osd=0 3 2024-02-19T21:04:17.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.024+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80f80a0 osd=0 3 2024-02-19T21:04:17.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.024+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 2 2024-02-19T21:04:17.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.024+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c80f80a0 osd=0 3 2024-02-19T21:04:17.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.024+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 0 2 2024-02-19T21:04:17.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.024+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80f80a0 osd=0 4 2024-02-19T21:04:17.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.024+0000 7f86cea14700 10 client.24778.objecter _send_command 2 2024-02-19T21:04:17.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.024+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f86c8028610 con 0x7f86c80598f0 2024-02-19T21:04:17.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f86c8028610 con 0x7f86c80598f0 2024-02-19T21:04:17.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:17.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86c57fa700 10 client.24778.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c80f80a0 osd=0 4 2024-02-19T21:04:17.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 0 2 2024-02-19T21:04:17.020 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:04:17.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86cea14700 10 client.24778.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:17.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 3 2024-02-19T21:04:17.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80287e0 osd=1 3 2024-02-19T21:04:17.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80287e0 osd=1 3 2024-02-19T21:04:17.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80287e0 osd=1 3 2024-02-19T21:04:17.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 3 2024-02-19T21:04:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c80287e0 osd=1 3 2024-02-19T21:04:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 1 3 2024-02-19T21:04:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80287e0 osd=1 4 2024-02-19T21:04:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86cea14700 10 client.24778.objecter _send_command 3 2024-02-19T21:04:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.026+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86c8059f10 con 0x7f86c8121440 2024-02-19T21:04:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.027+0000 7f86c57fa700 10 client.24778.objecter ms_handle_connect 0x7f86c8121440 2024-02-19T21:04:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.028+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f86c8059f10 con 0x7f86c8121440 2024-02-19T21:04:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.028+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:04:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.028+0000 7f86c57fa700 10 client.24778.objecter _finish_command 3 = system:0 2024-02-19T21:04:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.028+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c80287e0 osd=1 4 2024-02-19T21:04:17.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.028+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 1 3 2024-02-19T21:04:17.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.054+0000 7f86cea14700 10 client.24778.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:17.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.054+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 4 2024-02-19T21:04:17.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.054+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80287e0 osd=1 3 2024-02-19T21:04:17.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.054+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80287e0 osd=1 3 2024-02-19T21:04:17.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.054+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80287e0 osd=1 3 2024-02-19T21:04:17.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.054+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 4 2024-02-19T21:04:17.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.054+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c80287e0 osd=1 3 2024-02-19T21:04:17.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.054+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 1 4 2024-02-19T21:04:17.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.054+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80287e0 osd=1 4 2024-02-19T21:04:17.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.054+0000 7f86cea14700 10 client.24778.objecter _send_command 4 2024-02-19T21:04:17.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.054+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f86c805a0b0 con 0x7f86c8121440 2024-02-19T21:04:17.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f86c805a0b0 con 0x7f86c8121440 2024-02-19T21:04:17.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:17.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86c57fa700 10 client.24778.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c80287e0 osd=1 4 2024-02-19T21:04:17.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 1 4 2024-02-19T21:04:17.049 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:04:17.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86cea14700 10 client.24778.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:17.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 5 2024-02-19T21:04:17.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80595a0 osd=2 3 2024-02-19T21:04:17.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80595a0 osd=2 3 2024-02-19T21:04:17.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80595a0 osd=2 3 2024-02-19T21:04:17.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 5 2024-02-19T21:04:17.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c80595a0 osd=2 3 2024-02-19T21:04:17.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 2 5 2024-02-19T21:04:17.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80595a0 osd=2 4 2024-02-19T21:04:17.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86cea14700 10 client.24778.objecter _send_command 5 2024-02-19T21:04:17.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86c8059710 con 0x7f86c80f4540 2024-02-19T21:04:17.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.056+0000 7f86c57fa700 10 client.24778.objecter ms_handle_connect 0x7f86c80f4540 2024-02-19T21:04:17.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.057+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f86c8059710 con 0x7f86c80f4540 2024-02-19T21:04:17.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.057+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:04:17.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.057+0000 7f86c57fa700 10 client.24778.objecter _finish_command 5 = system:0 2024-02-19T21:04:17.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.057+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c80595a0 osd=2 4 2024-02-19T21:04:17.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.057+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 2 5 2024-02-19T21:04:17.063 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:16.058209+0000 osd.4 (osd.4) 63 : cluster [DBG] 5.e scrub starts 2024-02-19T21:04:17.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:16.059643+0000 osd.4 (osd.4) 64 : cluster [DBG] 5.e scrub ok 2024-02-19T21:04:17.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:16.214847+0000 osd.5 (osd.5) 57 : cluster [DBG] 5.f scrub starts 2024-02-19T21:04:17.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:16.217739+0000 osd.5 (osd.5) 58 : cluster [DBG] 5.f scrub ok 2024-02-19T21:04:17.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:16.273518+0000 osd.1 (osd.1) 59 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:04:17.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:16.276511+0000 osd.1 (osd.1) 60 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:04:17.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.083+0000 7f86cea14700 10 client.24778.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:17.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.083+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 6 2024-02-19T21:04:17.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.083+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80595a0 osd=2 3 2024-02-19T21:04:17.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.083+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80595a0 osd=2 3 2024-02-19T21:04:17.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.083+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80595a0 osd=2 3 2024-02-19T21:04:17.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.083+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 6 2024-02-19T21:04:17.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.083+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c80595a0 osd=2 3 2024-02-19T21:04:17.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.083+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 2 6 2024-02-19T21:04:17.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.083+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80595a0 osd=2 4 2024-02-19T21:04:17.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.083+0000 7f86cea14700 10 client.24778.objecter _send_command 6 2024-02-19T21:04:17.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.083+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f86c80f3aa0 con 0x7f86c80f4540 2024-02-19T21:04:17.078 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:04:17.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f86c80f3aa0 con 0x7f86c80f4540 2024-02-19T21:04:17.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:17.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86c57fa700 10 client.24778.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c80595a0 osd=2 4 2024-02-19T21:04:17.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 2 6 2024-02-19T21:04:17.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86cea14700 10 client.24778.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:17.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 7 2024-02-19T21:04:17.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80f3eb0 osd=3 3 2024-02-19T21:04:17.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80f3eb0 osd=3 3 2024-02-19T21:04:17.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80f3eb0 osd=3 3 2024-02-19T21:04:17.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 7 2024-02-19T21:04:17.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c80f3eb0 osd=3 3 2024-02-19T21:04:17.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 3 7 2024-02-19T21:04:17.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80f3eb0 osd=3 4 2024-02-19T21:04:17.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86cea14700 10 client.24778.objecter _send_command 7 2024-02-19T21:04:17.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.085+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86c812e0b0 con 0x7f86c8125c30 2024-02-19T21:04:17.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.086+0000 7f86c57fa700 10 client.24778.objecter ms_handle_connect 0x7f86c8125c30 2024-02-19T21:04:17.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.087+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f86c812e0b0 con 0x7f86c8125c30 2024-02-19T21:04:17.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.087+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:04:17.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.087+0000 7f86c57fa700 10 client.24778.objecter _finish_command 7 = system:0 2024-02-19T21:04:17.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.087+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c80f3eb0 osd=3 4 2024-02-19T21:04:17.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.087+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 3 7 2024-02-19T21:04:17.081 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:17.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.107+0000 7f86cea14700 10 client.24778.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:17.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.107+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 8 2024-02-19T21:04:17.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.107+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80f3eb0 osd=3 3 2024-02-19T21:04:17.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.107+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80f3eb0 osd=3 3 2024-02-19T21:04:17.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.107+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c80f3eb0 osd=3 3 2024-02-19T21:04:17.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.107+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 8 2024-02-19T21:04:17.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.107+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c80f3eb0 osd=3 3 2024-02-19T21:04:17.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.107+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 3 8 2024-02-19T21:04:17.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.107+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80f3eb0 osd=3 4 2024-02-19T21:04:17.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.107+0000 7f86cea14700 10 client.24778.objecter _send_command 8 2024-02-19T21:04:17.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.107+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f86c8125190 con 0x7f86c8125c30 2024-02-19T21:04:17.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.109+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f86c8125190 con 0x7f86c8125c30 2024-02-19T21:04:17.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.109+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:17.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.109+0000 7f86c57fa700 10 client.24778.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.109+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c80f3eb0 osd=3 4 2024-02-19T21:04:17.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.109+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 3 8 2024-02-19T21:04:17.103 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:04:17.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.110+0000 7f86cea14700 10 client.24778.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:17.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.110+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 9 2024-02-19T21:04:17.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.110+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c8125190 osd=4 3 2024-02-19T21:04:17.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.110+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c8125190 osd=4 3 2024-02-19T21:04:17.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.110+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c8125190 osd=4 3 2024-02-19T21:04:17.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.110+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 9 2024-02-19T21:04:17.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.110+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c8125190 osd=4 3 2024-02-19T21:04:17.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.110+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 4 9 2024-02-19T21:04:17.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.110+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c8125190 osd=4 4 2024-02-19T21:04:17.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.110+0000 7f86cea14700 10 client.24778.objecter _send_command 9 2024-02-19T21:04:17.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.110+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86c8125480 con 0x7f86c812a4e0 2024-02-19T21:04:17.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.110+0000 7f86c57fa700 10 client.24778.objecter ms_handle_connect 0x7f86c812a4e0 2024-02-19T21:04:17.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.111+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f86c8125480 con 0x7f86c812a4e0 2024-02-19T21:04:17.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.111+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:04:17.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.111+0000 7f86c57fa700 10 client.24778.objecter _finish_command 9 = system:0 2024-02-19T21:04:17.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.111+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c8125190 osd=4 4 2024-02-19T21:04:17.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.111+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 4 9 2024-02-19T21:04:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:17.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.139+0000 7f86cea14700 10 client.24778.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:17.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.139+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 10 2024-02-19T21:04:17.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.139+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c8125190 osd=4 3 2024-02-19T21:04:17.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.139+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c8125190 osd=4 3 2024-02-19T21:04:17.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.139+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c8125190 osd=4 3 2024-02-19T21:04:17.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.139+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 10 2024-02-19T21:04:17.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.139+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c8125190 osd=4 3 2024-02-19T21:04:17.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.139+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 4 10 2024-02-19T21:04:17.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.139+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c8125190 osd=4 4 2024-02-19T21:04:17.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.139+0000 7f86cea14700 10 client.24778.objecter _send_command 10 2024-02-19T21:04:17.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.139+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f86c8125480 con 0x7f86c812a4e0 2024-02-19T21:04:17.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.140+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f86c8125480 con 0x7f86c812a4e0 2024-02-19T21:04:17.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.140+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:17.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.140+0000 7f86c57fa700 10 client.24778.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.140+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c8125190 osd=4 4 2024-02-19T21:04:17.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.140+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 4 10 2024-02-19T21:04:17.134 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:04:17.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.141+0000 7f86cea14700 10 client.24778.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:17.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.141+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 11 2024-02-19T21:04:17.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.141+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c8125480 osd=5 3 2024-02-19T21:04:17.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.141+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c8125480 osd=5 3 2024-02-19T21:04:17.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.141+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c8125480 osd=5 3 2024-02-19T21:04:17.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.141+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 11 2024-02-19T21:04:17.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.141+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c8125480 osd=5 3 2024-02-19T21:04:17.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.141+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 5 11 2024-02-19T21:04:17.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.141+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c8125480 osd=5 4 2024-02-19T21:04:17.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.141+0000 7f86cea14700 10 client.24778.objecter _send_command 11 2024-02-19T21:04:17.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.141+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86c8130560 con 0x7f86c812ffc0 2024-02-19T21:04:17.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.141+0000 7f86c57fa700 10 client.24778.objecter ms_handle_connect 0x7f86c812ffc0 2024-02-19T21:04:17.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.142+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f86c8130560 con 0x7f86c812ffc0 2024-02-19T21:04:17.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.142+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:04:17.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.142+0000 7f86c57fa700 10 client.24778.objecter _finish_command 11 = system:0 2024-02-19T21:04:17.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.142+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c8125480 osd=5 4 2024-02-19T21:04:17.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.142+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 5 11 2024-02-19T21:04:17.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.159+0000 7f86cea14700 10 client.24778.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:17.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.159+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 12 2024-02-19T21:04:17.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.159+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c8125480 osd=5 3 2024-02-19T21:04:17.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.159+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c8125480 osd=5 3 2024-02-19T21:04:17.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.159+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c8125480 osd=5 3 2024-02-19T21:04:17.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.159+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 12 2024-02-19T21:04:17.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.159+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c8125480 osd=5 3 2024-02-19T21:04:17.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.159+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 5 12 2024-02-19T21:04:17.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.159+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c8125480 osd=5 4 2024-02-19T21:04:17.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.159+0000 7f86cea14700 10 client.24778.objecter _send_command 12 2024-02-19T21:04:17.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.159+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f86c8130560 con 0x7f86c812ffc0 2024-02-19T21:04:17.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f86c8130560 con 0x7f86c812ffc0 2024-02-19T21:04:17.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:17.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86c57fa700 10 client.24778.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c8125480 osd=5 4 2024-02-19T21:04:17.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 5 12 2024-02-19T21:04:17.154 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:04:17.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86cea14700 10 client.24778.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:17.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 13 2024-02-19T21:04:17.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c8130560 osd=6 3 2024-02-19T21:04:17.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c8130560 osd=6 3 2024-02-19T21:04:17.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c8130560 osd=6 3 2024-02-19T21:04:17.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 13 2024-02-19T21:04:17.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c8130560 osd=6 3 2024-02-19T21:04:17.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 6 13 2024-02-19T21:04:17.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c8130560 osd=6 4 2024-02-19T21:04:17.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86cea14700 10 client.24778.objecter _send_command 13 2024-02-19T21:04:17.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.161+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86c813b180 con 0x7f86c8137630 2024-02-19T21:04:17.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.162+0000 7f86c57fa700 10 client.24778.objecter ms_handle_connect 0x7f86c8137630 2024-02-19T21:04:17.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.162+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f86c813b180 con 0x7f86c8137630 2024-02-19T21:04:17.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.162+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:04:17.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.162+0000 7f86c57fa700 10 client.24778.objecter _finish_command 13 = system:0 2024-02-19T21:04:17.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.162+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c8130560 osd=6 4 2024-02-19T21:04:17.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.162+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 6 13 2024-02-19T21:04:17.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.177+0000 7f86cea14700 10 client.24778.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:17.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.177+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 14 2024-02-19T21:04:17.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.177+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c8130560 osd=6 3 2024-02-19T21:04:17.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.177+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c8130560 osd=6 3 2024-02-19T21:04:17.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.177+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c8130560 osd=6 3 2024-02-19T21:04:17.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.177+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 14 2024-02-19T21:04:17.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.177+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c8130560 osd=6 3 2024-02-19T21:04:17.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.177+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 6 14 2024-02-19T21:04:17.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.177+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c8130560 osd=6 4 2024-02-19T21:04:17.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.177+0000 7f86cea14700 10 client.24778.objecter _send_command 14 2024-02-19T21:04:17.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.177+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f86c813c930 con 0x7f86c8137630 2024-02-19T21:04:17.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f86c813c930 con 0x7f86c8137630 2024-02-19T21:04:17.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:17.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86c57fa700 10 client.24778.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c8130560 osd=6 4 2024-02-19T21:04:17.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 6 14 2024-02-19T21:04:17.172 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:04:17.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86cea14700 10 client.24778.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:17.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 15 2024-02-19T21:04:17.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c813c930 osd=7 3 2024-02-19T21:04:17.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c813c930 osd=7 3 2024-02-19T21:04:17.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c813c930 osd=7 3 2024-02-19T21:04:17.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 15 2024-02-19T21:04:17.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c813c930 osd=7 3 2024-02-19T21:04:17.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 7 15 2024-02-19T21:04:17.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c813c930 osd=7 4 2024-02-19T21:04:17.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86cea14700 10 client.24778.objecter _send_command 15 2024-02-19T21:04:17.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.179+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f86c813d440 con 0x7f86c813cfb0 2024-02-19T21:04:17.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.180+0000 7f86c57fa700 10 client.24778.objecter ms_handle_connect 0x7f86c813cfb0 2024-02-19T21:04:17.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.180+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f86c813d440 con 0x7f86c813cfb0 2024-02-19T21:04:17.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.180+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:04:17.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.180+0000 7f86c57fa700 10 client.24778.objecter _finish_command 15 = system:0 2024-02-19T21:04:17.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.180+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c813c930 osd=7 4 2024-02-19T21:04:17.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.180+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 7 15 2024-02-19T21:04:17.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.194+0000 7f86cea14700 10 client.24778.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:17.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.194+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign -1 16 2024-02-19T21:04:17.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.194+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c813c930 osd=7 3 2024-02-19T21:04:17.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.194+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c813c930 osd=7 3 2024-02-19T21:04:17.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.194+0000 7f86cea14700 20 client.24778.objecter _get_session s=0x7f86c813c930 osd=7 3 2024-02-19T21:04:17.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.194+0000 7f86cea14700 15 client.24778.objecter _session_command_op_remove -1 16 2024-02-19T21:04:17.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.194+0000 7f86cea14700 20 client.24778.objecter get_session s=0x7f86c813c930 osd=7 3 2024-02-19T21:04:17.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.194+0000 7f86cea14700 15 client.24778.objecter _session_command_op_assign 7 16 2024-02-19T21:04:17.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.194+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c813c930 osd=7 4 2024-02-19T21:04:17.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.195+0000 7f86cea14700 10 client.24778.objecter _send_command 16 2024-02-19T21:04:17.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.195+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f86c813d440 con 0x7f86c813cfb0 2024-02-19T21:04:17.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.196+0000 7f86c57fa700 1 -- 172.21.15.163:0/1352552202 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f86c813d440 con 0x7f86c813cfb0 2024-02-19T21:04:17.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.196+0000 7f86c57fa700 10 client.24778.objecter ms_dispatch 0x7f86c8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:17.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.196+0000 7f86c57fa700 10 client.24778.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.196+0000 7f86c57fa700 20 client.24778.objecter put_session s=0x7f86c813c930 osd=7 4 2024-02-19T21:04:17.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.196+0000 7f86c57fa700 15 client.24778.objecter _session_command_op_remove 7 16 2024-02-19T21:04:17.190 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:04:17.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 10 librados: watch_flush enter 2024-02-19T21:04:17.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 10 librados: watch_flush exit 2024-02-19T21:04:17.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 10 client.24778.objecter close_session for osd.0 2024-02-19T21:04:17.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 >> v1:172.21.15.163:6801/1559418115 conn(0x7f86c80598f0 legacy=0x7f86c8120540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:17.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80f80a0 osd=0 1 2024-02-19T21:04:17.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 10 client.24778.objecter close_session for osd.1 2024-02-19T21:04:17.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 >> v1:172.21.15.163:6805/3125047432 conn(0x7f86c8121440 legacy=0x7f86c8123830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:17.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80287e0 osd=1 1 2024-02-19T21:04:17.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 10 client.24778.objecter close_session for osd.2 2024-02-19T21:04:17.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 >> v1:172.21.15.163:6809/1563158487 conn(0x7f86c80f4540 legacy=0x7f86c81280e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:17.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80595a0 osd=2 1 2024-02-19T21:04:17.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 10 client.24778.objecter close_session for osd.3 2024-02-19T21:04:17.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 >> v1:172.21.15.163:6813/2112279769 conn(0x7f86c8125c30 legacy=0x7f86c812c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c80f3eb0 osd=3 1 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 10 client.24778.objecter close_session for osd.4 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 >> v1:172.21.15.193:6800/3129716965 conn(0x7f86c812a4e0 legacy=0x7f86c812f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c8125190 osd=4 1 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 10 client.24778.objecter close_session for osd.5 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 >> v1:172.21.15.193:6804/805161991 conn(0x7f86c812ffc0 legacy=0x7f86c8135a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86ccfb1700 1 -- 172.21.15.163:0/1352552202 reap_dead start 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c8125480 osd=5 1 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 10 client.24778.objecter close_session for osd.6 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 >> v1:172.21.15.193:6808/787604615 conn(0x7f86c8137630 legacy=0x7f86c8139a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86ccfb1700 1 -- 172.21.15.163:0/1352552202 reap_dead start 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c8130560 osd=6 1 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 10 client.24778.objecter close_session for osd.7 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 >> v1:172.21.15.193:6812/157927432 conn(0x7f86c813cfb0 legacy=0x7f86c813b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 20 client.24778.objecter put_session s=0x7f86c813c930 osd=7 1 2024-02-19T21:04:17.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 20 client.24778.objecter shutdown clearing up homeless session... 2024-02-19T21:04:17.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 10 client.24778.objecter successfully canceled tick 2024-02-19T21:04:17.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86ccfb1700 1 -- 172.21.15.163:0/1352552202 reap_dead start 2024-02-19T21:04:17.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86ccfb1700 1 -- 172.21.15.163:0/1352552202 reap_dead start 2024-02-19T21:04:17.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 >> v1:172.21.15.163:6800/1552514681 conn(0x7f86b0063ef0 legacy=0x7f86b00663b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:17.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 >> v1:172.21.15.193:6789/0 conn(0x7f86c80ff670 legacy=0x7f86c80f8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:17.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 shutdown_connections 2024-02-19T21:04:17.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 >> 172.21.15.163:0/1352552202 conn(0x7f86c80f2470 msgr2=0x7f86c80601c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:17.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 shutdown_connections 2024-02-19T21:04:17.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.197+0000 7f86cea14700 1 -- 172.21.15.163:0/1352552202 wait complete. 2024-02-19T21:04:17.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:17.198+0000 7f86cea14700 1 librados: shutdown 2024-02-19T21:04:17.194 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.194 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.195 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.195 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.195 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.195 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.195 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.195 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:04:17.220 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:17.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:17.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:17.324 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:17.356 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:17.377 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:17.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:17.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:17.509 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:17.531 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:17.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:17.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:17.635 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:17.663 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:17.817 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:17.820 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:15.867669+0000 osd.2 (osd.2) 47 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:04:17.820 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:15.868719+0000 osd.2 (osd.2) 48 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:04:17.820 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:16.951561+0000 mgr.y (mgr.14550) 123 : cluster [DBG] pgmap v104: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-19T21:04:17.820 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:16.997549+0000 mon.b (mon.1) 81 : audit [DBG] from='client.? v1:172.21.15.163:0/1352552202' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:04:17.820 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:15.867669+0000 osd.2 (osd.2) 47 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:04:17.820 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:15.868719+0000 osd.2 (osd.2) 48 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:04:17.820 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:16.951561+0000 mgr.y (mgr.14550) 123 : cluster [DBG] pgmap v104: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-19T21:04:17.821 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:16.997549+0000 mon.b (mon.1) 81 : audit [DBG] from='client.? v1:172.21.15.163:0/1352552202' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:04:17.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:17.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:17.922 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:17.951 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:04:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:04:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:04:17.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:15.867669+0000 osd.2 (osd.2) 47 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:04:17.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:15.868719+0000 osd.2 (osd.2) 48 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:04:17.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:16.951561+0000 mgr.y (mgr.14550) 123 : cluster [DBG] pgmap v104: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-19T21:04:17.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:16.997549+0000 mon.b (mon.1) 81 : audit [DBG] from='client.? v1:172.21.15.163:0/1352552202' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:04:17.954 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:17.976 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:17.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:18.080 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:18.112 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:18.259 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:18.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:18.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:18.365 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:18.392 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:18.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:16.863152+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 deep-scrub starts 2024-02-19T21:04:18.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:16.866109+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.8 deep-scrub ok 2024-02-19T21:04:18.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:17.539913+0000 osd.2 (osd.2) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:18.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:17.540006+0000 osd.2 (osd.2) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:18.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:17.692857+0000 mon.a (mon.0) 742 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-02-19T21:04:18.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:18.241700+0000 osd.1 (osd.1) 61 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:04:18.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:18.242551+0000 osd.1 (osd.1) 62 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:04:18.841 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:18.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:18.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:18.950 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:18.980 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:16.863152+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 deep-scrub starts 2024-02-19T21:04:18.980 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:16.866109+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.8 deep-scrub ok 2024-02-19T21:04:18.980 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:17.539913+0000 osd.2 (osd.2) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:18.980 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:17.540006+0000 osd.2 (osd.2) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:18.980 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:17.692857+0000 mon.a (mon.0) 742 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-02-19T21:04:18.980 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:18.241700+0000 osd.1 (osd.1) 61 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:04:18.980 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:18.242551+0000 osd.1 (osd.1) 62 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:04:18.982 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:18.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:16.863152+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 deep-scrub starts 2024-02-19T21:04:18.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:16.866109+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.8 deep-scrub ok 2024-02-19T21:04:18.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:17.539913+0000 osd.2 (osd.2) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:18.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:17.540006+0000 osd.2 (osd.2) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:18.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:17.692857+0000 mon.a (mon.0) 742 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-02-19T21:04:18.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:18.241700+0000 osd.1 (osd.1) 61 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:04:18.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:18.242551+0000 osd.1 (osd.1) 62 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:04:19.005 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:19.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:19.108 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:04:19.139 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:19.179 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:04:19.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:19.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:04:19.284 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:19.285 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:19.315 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:04:19.315 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:04:19.315 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-02-19T21:04:19.315 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-02-19T21:04:19.315 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:04:19.323 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:19.662 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:19.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:19.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:19.767 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:04:19.799 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:19.820 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:04:19.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:19.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:04:19.924 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:19.948 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:04:19.948 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:04:19.948 INFO:tasks.thrashosds.thrasher:fixing pg num pool iscsi 2024-02-19T21:04:19.948 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:04:19.951 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:17.829844+0000 osd.2 (osd.2) 53 : cluster [DBG] 5.a scrub starts 2024-02-19T21:04:19.951 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:17.831059+0000 osd.2 (osd.2) 54 : cluster [DBG] 5.a scrub ok 2024-02-19T21:04:19.951 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:18.873275+0000 osd.7 (osd.7) 55 : cluster [DBG] 6.1c deep-scrub starts 2024-02-19T21:04:19.951 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:18.877380+0000 osd.7 (osd.7) 56 : cluster [DBG] 6.1c deep-scrub ok 2024-02-19T21:04:19.951 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:18.951944+0000 mgr.y (mgr.14550) 124 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 813 B/s rd, 0 op/s 2024-02-19T21:04:19.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:19.301333+0000 osd.5 (osd.5) 59 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:04:19.952 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:19.314234+0000 osd.5 (osd.5) 60 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:04:19.953 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:19.972 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:19.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:19.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:19.979 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:17.829844+0000 osd.2 (osd.2) 53 : cluster [DBG] 5.a scrub starts 2024-02-19T21:04:19.979 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:17.831059+0000 osd.2 (osd.2) 54 : cluster [DBG] 5.a scrub ok 2024-02-19T21:04:19.979 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:18.873275+0000 osd.7 (osd.7) 55 : cluster [DBG] 6.1c deep-scrub starts 2024-02-19T21:04:19.979 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:18.877380+0000 osd.7 (osd.7) 56 : cluster [DBG] 6.1c deep-scrub ok 2024-02-19T21:04:19.979 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:18.951944+0000 mgr.y (mgr.14550) 124 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 813 B/s rd, 0 op/s 2024-02-19T21:04:19.979 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:19.301333+0000 osd.5 (osd.5) 59 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:04:19.980 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:19.314234+0000 osd.5 (osd.5) 60 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:04:19.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:17.829844+0000 osd.2 (osd.2) 53 : cluster [DBG] 5.a scrub starts 2024-02-19T21:04:19.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:17.831059+0000 osd.2 (osd.2) 54 : cluster [DBG] 5.a scrub ok 2024-02-19T21:04:19.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:18.873275+0000 osd.7 (osd.7) 55 : cluster [DBG] 6.1c deep-scrub starts 2024-02-19T21:04:19.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:18.877380+0000 osd.7 (osd.7) 56 : cluster [DBG] 6.1c deep-scrub ok 2024-02-19T21:04:19.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:18.951944+0000 mgr.y (mgr.14550) 124 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 813 B/s rd, 0 op/s 2024-02-19T21:04:19.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:19.301333+0000 osd.5 (osd.5) 59 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:04:19.980 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:19.314234+0000 osd.5 (osd.5) 60 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:04:20.076 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:20.111 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:20.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:04:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:04:20.366 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-19T21:04:20.375 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:20.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:20.478 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:20.515 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:20.545 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:20.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:20.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:20.650 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:20.682 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:20.693 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:19.405908+0000 osd.3 (osd.3) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:20.693 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:19.405973+0000 osd.3 (osd.3) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:20.693 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:19.702732+0000 mon.a (mon.0) 743 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-02-19T21:04:20.693 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:19.855419+0000 osd.7 (osd.7) 57 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:04:20.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:19.858680+0000 osd.7 (osd.7) 58 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:04:20.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:19.883795+0000 osd.6 (osd.6) 43 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:04:20.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:19.885792+0000 osd.6 (osd.6) 44 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:04:20.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:20.053578+0000 mgr.y (mgr.14550) 125 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:20.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:20.237693+0000 osd.1 (osd.1) 63 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:04:20.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:20.241317+0000 osd.1 (osd.1) 64 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:04:20.704 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:20.706 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:20.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:20.807 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:04:20.838 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:20.858 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:04:20.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:20.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:04:20.864 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:04:20.951 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:20.962 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:04:20.989 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:19.405908+0000 osd.3 (osd.3) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:20.989 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:19.405973+0000 osd.3 (osd.3) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:20.989 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:19.702732+0000 mon.a (mon.0) 743 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-02-19T21:04:20.989 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:19.855419+0000 osd.7 (osd.7) 57 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:04:20.989 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:19.858680+0000 osd.7 (osd.7) 58 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:04:20.989 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:19.883795+0000 osd.6 (osd.6) 43 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:04:20.989 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:19.885792+0000 osd.6 (osd.6) 44 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:04:20.989 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:20.053578+0000 mgr.y (mgr.14550) 125 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:20.989 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:20.237693+0000 osd.1 (osd.1) 63 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:04:20.989 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:20.241317+0000 osd.1 (osd.1) 64 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:04:20.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:19.405908+0000 osd.3 (osd.3) 71 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:20.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:19.405973+0000 osd.3 (osd.3) 72 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:20.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:19.702732+0000 mon.a (mon.0) 743 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-02-19T21:04:20.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:19.855419+0000 osd.7 (osd.7) 57 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:04:20.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:19.858680+0000 osd.7 (osd.7) 58 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:04:20.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:19.883795+0000 osd.6 (osd.6) 43 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:04:20.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:19.885792+0000 osd.6 (osd.6) 44 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:04:20.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:20.053578+0000 mgr.y (mgr.14550) 125 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:20.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:20.237693+0000 osd.1 (osd.1) 63 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:04:20.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:20.241317+0000 osd.1 (osd.1) 64 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:04:20.991 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:21.010 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:04:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:21.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:04:21.114 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:21.154 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:21.746 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:21.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:19.844208+0000 osd.2 (osd.2) 55 : cluster [DBG] 6.f deep-scrub starts 2024-02-19T21:04:21.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:19.845589+0000 osd.2 (osd.2) 56 : cluster [DBG] 6.f deep-scrub ok 2024-02-19T21:04:21.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:20.834491+0000 osd.6 (osd.6) 45 : cluster [DBG] 4.a scrub starts 2024-02-19T21:04:21.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:20.836604+0000 osd.6 (osd.6) 46 : cluster [DBG] 4.a scrub ok 2024-02-19T21:04:21.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:20.952719+0000 mgr.y (mgr.14550) 126 : cluster [DBG] pgmap v108: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-19T21:04:21.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:21.030569+0000 osd.4 (osd.4) 65 : cluster [DBG] 6.e deep-scrub starts 2024-02-19T21:04:21.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:21.031572+0000 osd.4 (osd.4) 66 : cluster [DBG] 6.e deep-scrub ok 2024-02-19T21:04:21.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:21.204713+0000 osd.1 (osd.1) 65 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:04:21.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:21.205818+0000 osd.1 (osd.1) 66 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:04:21.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:19.844208+0000 osd.2 (osd.2) 55 : cluster [DBG] 6.f deep-scrub starts 2024-02-19T21:04:21.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:19.845589+0000 osd.2 (osd.2) 56 : cluster [DBG] 6.f deep-scrub ok 2024-02-19T21:04:21.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:20.834491+0000 osd.6 (osd.6) 45 : cluster [DBG] 4.a scrub starts 2024-02-19T21:04:21.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:20.836604+0000 osd.6 (osd.6) 46 : cluster [DBG] 4.a scrub ok 2024-02-19T21:04:21.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:20.952719+0000 mgr.y (mgr.14550) 126 : cluster [DBG] pgmap v108: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-19T21:04:21.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:21.030569+0000 osd.4 (osd.4) 65 : cluster [DBG] 6.e deep-scrub starts 2024-02-19T21:04:21.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:21.031572+0000 osd.4 (osd.4) 66 : cluster [DBG] 6.e deep-scrub ok 2024-02-19T21:04:21.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:21.204713+0000 osd.1 (osd.1) 65 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:04:21.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:21.205818+0000 osd.1 (osd.1) 66 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:04:22.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:19.844208+0000 osd.2 (osd.2) 55 : cluster [DBG] 6.f deep-scrub starts 2024-02-19T21:04:22.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:19.845589+0000 osd.2 (osd.2) 56 : cluster [DBG] 6.f deep-scrub ok 2024-02-19T21:04:22.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:20.834491+0000 osd.6 (osd.6) 45 : cluster [DBG] 4.a scrub starts 2024-02-19T21:04:22.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:20.836604+0000 osd.6 (osd.6) 46 : cluster [DBG] 4.a scrub ok 2024-02-19T21:04:22.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:20.952719+0000 mgr.y (mgr.14550) 126 : cluster [DBG] pgmap v108: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-19T21:04:22.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:21.030569+0000 osd.4 (osd.4) 65 : cluster [DBG] 6.e deep-scrub starts 2024-02-19T21:04:22.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:21.031572+0000 osd.4 (osd.4) 66 : cluster [DBG] 6.e deep-scrub ok 2024-02-19T21:04:22.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:21.204713+0000 osd.1 (osd.1) 65 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:04:22.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:21.205818+0000 osd.1 (osd.1) 66 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:04:22.611 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:22.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:22.722 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:22.755 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:22.776 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:22.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:22.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:22.879 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:04:22.909 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:22.929 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:04:22.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:20.872752+0000 osd.2 (osd.2) 57 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:04:22.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:20.874345+0000 osd.2 (osd.2) 58 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:04:22.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:21.656489+0000 osd.0 (osd.0) 59 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:04:22.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:21.659612+0000 osd.0 (osd.0) 60 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:04:22.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:21.827872+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:04:22.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:21.828859+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:04:22.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:21.949118+0000 osd.7 (osd.7) 59 : cluster [DBG] 2.b scrub starts 2024-02-19T21:04:22.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:21.952578+0000 osd.7 (osd.7) 60 : cluster [DBG] 2.b scrub ok 2024-02-19T21:04:22.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:22.167223+0000 osd.1 (osd.1) 67 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:04:22.934 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:22.168588+0000 osd.1 (osd.1) 68 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:04:22.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:22.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:04:22.934 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:20.872752+0000 osd.2 (osd.2) 57 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:04:22.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:20.874345+0000 osd.2 (osd.2) 58 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:04:22.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:21.656489+0000 osd.0 (osd.0) 59 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:04:22.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:21.659612+0000 osd.0 (osd.0) 60 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:04:22.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:21.827872+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:04:22.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:21.828859+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:04:22.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:21.949118+0000 osd.7 (osd.7) 59 : cluster [DBG] 2.b scrub starts 2024-02-19T21:04:22.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:21.952578+0000 osd.7 (osd.7) 60 : cluster [DBG] 2.b scrub ok 2024-02-19T21:04:22.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:22.167223+0000 osd.1 (osd.1) 67 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:04:22.935 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:22.168588+0000 osd.1 (osd.1) 68 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:04:23.035 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:23.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:20.872752+0000 osd.2 (osd.2) 57 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:04:23.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:20.874345+0000 osd.2 (osd.2) 58 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:04:23.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:21.656489+0000 osd.0 (osd.0) 59 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:04:23.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:21.659612+0000 osd.0 (osd.0) 60 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:04:23.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:21.827872+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:04:23.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:21.828859+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:04:23.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:21.949118+0000 osd.7 (osd.7) 59 : cluster [DBG] 2.b scrub starts 2024-02-19T21:04:23.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:21.952578+0000 osd.7 (osd.7) 60 : cluster [DBG] 2.b scrub ok 2024-02-19T21:04:23.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:22.167223+0000 osd.1 (osd.1) 67 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:04:23.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:22.168588+0000 osd.1 (osd.1) 68 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:04:23.070 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:23.555 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:04:23] "GET /metrics HTTP/1.1" 200 172251 "" "Prometheus/2.33.4" 2024-02-19T21:04:23.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:22.761963+0000 osd.3 (osd.3) 73 : cluster [DBG] 4.14 scrub starts 2024-02-19T21:04:23.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:22.763922+0000 osd.3 (osd.3) 74 : cluster [DBG] 4.14 scrub ok 2024-02-19T21:04:23.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:22.799687+0000 osd.6 (osd.6) 49 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:04:23.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:22.800572+0000 osd.6 (osd.6) 50 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:04:23.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:22.953100+0000 mgr.y (mgr.14550) 127 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:04:23.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:23.206764+0000 mon.b (mon.1) 82 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:04:23.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:23.551269+0000 mon.b (mon.1) 83 : audit [INF] from='client.? v1:172.21.15.163:0/287987821' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:04:23.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:23.551802+0000 mon.a (mon.0) 744 : audit [INF] from='client.24781 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:04:23.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:23.625935+0000 mon.b (mon.1) 84 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:04:23.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:23.626696+0000 mon.b (mon.1) 85 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:04:23.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:23.632576+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:04:23.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:22.761963+0000 osd.3 (osd.3) 73 : cluster [DBG] 4.14 scrub starts 2024-02-19T21:04:23.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:22.763922+0000 osd.3 (osd.3) 74 : cluster [DBG] 4.14 scrub ok 2024-02-19T21:04:23.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:22.799687+0000 osd.6 (osd.6) 49 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:04:23.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:22.800572+0000 osd.6 (osd.6) 50 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:04:23.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:22.953100+0000 mgr.y (mgr.14550) 127 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:04:23.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:23.206764+0000 mon.b (mon.1) 82 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:04:23.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:23.551269+0000 mon.b (mon.1) 83 : audit [INF] from='client.? v1:172.21.15.163:0/287987821' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:04:23.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:23.551802+0000 mon.a (mon.0) 744 : audit [INF] from='client.24781 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:04:23.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:23.625935+0000 mon.b (mon.1) 84 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:04:23.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:23.626696+0000 mon.b (mon.1) 85 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:04:23.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:23.632576+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:04:24.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:22.761963+0000 osd.3 (osd.3) 73 : cluster [DBG] 4.14 scrub starts 2024-02-19T21:04:24.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:22.763922+0000 osd.3 (osd.3) 74 : cluster [DBG] 4.14 scrub ok 2024-02-19T21:04:24.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:22.799687+0000 osd.6 (osd.6) 49 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:04:24.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:22.800572+0000 osd.6 (osd.6) 50 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:04:24.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:22.953100+0000 mgr.y (mgr.14550) 127 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:04:24.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:23.206764+0000 mon.b (mon.1) 82 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:04:24.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:23.551269+0000 mon.b (mon.1) 83 : audit [INF] from='client.? v1:172.21.15.163:0/287987821' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:04:24.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:23.551802+0000 mon.a (mon.0) 744 : audit [INF] from='client.24781 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:04:24.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:23.625935+0000 mon.b (mon.1) 84 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:04:24.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:23.626696+0000 mon.b (mon.1) 85 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:04:24.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:23.632576+0000 mon.a (mon.0) 745 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:04:24.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.120+0000 7fd9ef815700 1 -- 172.21.15.163:0/3466490544 >> v1:172.21.15.163:6789/0 conn(0x7fd9e80ff670 legacy=0x7fd9e8101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:24.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.120+0000 7fd9ef815700 1 -- 172.21.15.163:0/3466490544 shutdown_connections 2024-02-19T21:04:24.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.120+0000 7fd9ef815700 1 -- 172.21.15.163:0/3466490544 >> 172.21.15.163:0/3466490544 conn(0x7fd9e80f2470 msgr2=0x7fd9e80f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:24.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.120+0000 7fd9ef815700 1 -- 172.21.15.163:0/3466490544 shutdown_connections 2024-02-19T21:04:24.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.120+0000 7fd9ef815700 1 -- 172.21.15.163:0/3466490544 wait complete. 2024-02-19T21:04:24.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.121+0000 7fd9ef815700 1 librados: starting msgr at 2024-02-19T21:04:24.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.121+0000 7fd9ef815700 1 librados: starting objecter 2024-02-19T21:04:24.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.121+0000 7fd9ef815700 1 Processor -- start 2024-02-19T21:04:24.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.121+0000 7fd9ef815700 1 -- start start 2024-02-19T21:04:24.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.121+0000 7fd9ef815700 1 librados: setting wanted keys 2024-02-19T21:04:24.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.121+0000 7fd9ef815700 1 librados: calling monclient init 2024-02-19T21:04:24.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.121+0000 7fd9ef815700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd9e811db30 con 0x7fd9e80fc870 2024-02-19T21:04:24.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.121+0000 7fd9ef815700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd9e811ecd0 con 0x7fd9e80ff670 2024-02-19T21:04:24.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.121+0000 7fd9ef815700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd9e811fe70 con 0x7fd9e81031b0 2024-02-19T21:04:24.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.121+0000 7fd9eddb2700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fd9e81031b0 0x7fd9e811c410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:47844/0 (socket says 172.21.15.163:47844) 2024-02-19T21:04:24.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.121+0000 7fd9eddb2700 1 -- 172.21.15.163:0/816365348 learned_addr learned my addr 172.21.15.163:0/816365348 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:04:24.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 10 client.?.objecter ms_handle_connect 0x7fd9e81031b0 2024-02-19T21:04:24.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:24.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 10 client.?.objecter ms_handle_connect 0x7fd9e80fc870 2024-02-19T21:04:24.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:24.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 10 client.?.objecter ms_handle_connect 0x7fd9e80ff670 2024-02-19T21:04:24.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:24.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126420311 0 0) 0x7fd9e811fe70 con 0x7fd9e81031b0 2024-02-19T21:04:24.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d4003630 con 0x7fd9e81031b0 2024-02-19T21:04:24.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857243747 0 0) 0x7fd9e811db30 con 0x7fd9e80fc870 2024-02-19T21:04:24.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9e811fe70 con 0x7fd9e80fc870 2024-02-19T21:04:24.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877511286 0 0) 0x7fd9e811ecd0 con 0x7fd9e80ff670 2024-02-19T21:04:24.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9e811db30 con 0x7fd9e80ff670 2024-02-19T21:04:24.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2399397463 0 0) 0x7fd9e811db30 con 0x7fd9e80ff670 2024-02-19T21:04:24.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9e811ecd0 con 0x7fd9e80ff670 2024-02-19T21:04:24.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3713398100 0 0) 0x7fd9d4003630 con 0x7fd9e81031b0 2024-02-19T21:04:24.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9e811db30 con 0x7fd9e81031b0 2024-02-19T21:04:24.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd9d8002e30 con 0x7fd9e80ff670 2024-02-19T21:04:24.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.122+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd9e0001cb0 con 0x7fd9e81031b0 2024-02-19T21:04:24.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.123+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1877202556 0 0) 0x7fd9e811fe70 con 0x7fd9e80fc870 2024-02-19T21:04:24.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.123+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9d4003630 con 0x7fd9e80fc870 2024-02-19T21:04:24.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.123+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd9e4001630 con 0x7fd9e80fc870 2024-02-19T21:04:24.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.123+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3653223726 0 0) 0x7fd9e811db30 con 0x7fd9e81031b0 2024-02-19T21:04:24.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.123+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 >> v1:172.21.15.193:6789/0 conn(0x7fd9e80ff670 legacy=0x7fd9e80f8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:24.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.123+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 >> v1:172.21.15.163:6789/0 conn(0x7fd9e80fc870 legacy=0x7fd9e80f8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:24.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.123+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e8121010 con 0x7fd9e81031b0 2024-02-19T21:04:24.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.123+0000 7fd9ef815700 1 -- 172.21.15.163:0/816365348 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e811dd20 con 0x7fd9e81031b0 2024-02-19T21:04:24.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.123+0000 7fd9ef815700 10 client.34784.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:24.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.123+0000 7fd9ef815700 1 -- 172.21.15.163:0/816365348 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e811e190 con 0x7fd9e81031b0 2024-02-19T21:04:24.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.123+0000 7fd9ef815700 1 librados: init done 2024-02-19T21:04:24.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.123+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fd9e00023f0 con 0x7fd9e81031b0 2024-02-19T21:04:24.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.123+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd9e0005d90 con 0x7fd9e81031b0 2024-02-19T21:04:24.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.124+0000 7fd9ef815700 1 -- 172.21.15.163:0/816365348 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9cc004f20 con 0x7fd9e81031b0 2024-02-19T21:04:24.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.125+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fd9e0001f50 con 0x7fd9e81031b0 2024-02-19T21:04:24.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.126+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(97..97 src has 43..97) v4 ==== 6205+0+0 (unknown 2951543353 0 0) 0x7fd9e0078890 con 0x7fd9e81031b0 2024-02-19T21:04:24.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.126+0000 7fd9de7fc700 10 client.34784.objecter ms_dispatch 0x7fd9e8001550 osd_map(97..97 src has 43..97) v4 2024-02-19T21:04:24.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.126+0000 7fd9de7fc700 3 client.34784.objecter handle_osd_map got epochs [97,97] > 0 2024-02-19T21:04:24.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.126+0000 7fd9de7fc700 3 client.34784.objecter handle_osd_map decoding full epoch 97 2024-02-19T21:04:24.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.126+0000 7fd9de7fc700 20 client.34784.objecter dump_active .. 0 homeless 2024-02-19T21:04:24.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.129+0000 7fd9de7fc700 10 client.34784.objecter ms_handle_connect 0x7fd9d4063e40 2024-02-19T21:04:24.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.134+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd9e004c3c0 con 0x7fd9e81031b0 2024-02-19T21:04:24.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.302+0000 7f42e580e700 1 -- 172.21.15.163:0/3467858140 >> v1:172.21.15.163:6789/0 conn(0x7f42e00634c0 legacy=0x7f42e010d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:24.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.302+0000 7f42e580e700 1 -- 172.21.15.163:0/3467858140 shutdown_connections 2024-02-19T21:04:24.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.302+0000 7f42e580e700 1 -- 172.21.15.163:0/3467858140 >> 172.21.15.163:0/3467858140 conn(0x7f42e006fdc0 msgr2=0x7f42e00f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:24.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.308+0000 7f42e580e700 1 -- 172.21.15.163:0/3467858140 shutdown_connections 2024-02-19T21:04:24.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.310+0000 7f42e580e700 1 -- 172.21.15.163:0/3467858140 wait complete. 2024-02-19T21:04:24.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.310+0000 7f42e580e700 1 librados: starting msgr at 2024-02-19T21:04:24.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.310+0000 7f42e580e700 1 librados: starting objecter 2024-02-19T21:04:24.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.310+0000 7f42e580e700 1 Processor -- start 2024-02-19T21:04:24.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.311+0000 7f42e580e700 1 -- start start 2024-02-19T21:04:24.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.311+0000 7f42e580e700 1 librados: setting wanted keys 2024-02-19T21:04:24.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.311+0000 7f42e580e700 1 librados: calling monclient init 2024-02-19T21:04:24.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.311+0000 7f42e580e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f42e0109720 con 0x7f42e006a860 2024-02-19T21:04:24.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.311+0000 7f42e580e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f42e010a920 con 0x7f42e00ff140 2024-02-19T21:04:24.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.311+0000 7f42e580e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f42e010bb20 con 0x7f42e00634c0 2024-02-19T21:04:24.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.311+0000 7f42df7fe700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f42e00ff140 0x7f42e0107de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:57566/0 (socket says 172.21.15.163:57566) 2024-02-19T21:04:24.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.311+0000 7f42df7fe700 1 -- 172.21.15.163:0/287987821 learned_addr learned my addr 172.21.15.163:0/287987821 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:04:24.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.311+0000 7f42c7fff700 10 client.?.objecter ms_handle_connect 0x7f42e00ff140 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.311+0000 7f42c7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.312+0000 7f42c7fff700 1 -- 172.21.15.163:0/287987821 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911494982 0 0) 0x7f42e010a920 con 0x7f42e00ff140 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.312+0000 7f42c7fff700 1 -- 172.21.15.163:0/287987821 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42c8002220 con 0x7f42e00ff140 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.312+0000 7f42c7fff700 1 -- 172.21.15.163:0/287987821 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4043058604 0 0) 0x7f42c8002220 con 0x7f42e00ff140 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.312+0000 7f42c7fff700 1 -- 172.21.15.163:0/287987821 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42e010a920 con 0x7f42e00ff140 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.312+0000 7f42c7fff700 1 -- 172.21.15.163:0/287987821 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f42d4002130 con 0x7f42e00ff140 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.312+0000 7f42c7fff700 1 -- 172.21.15.163:0/287987821 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 146142103 0 0) 0x7f42e010a920 con 0x7f42e00ff140 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.312+0000 7f42c7fff700 1 -- 172.21.15.163:0/287987821 >> v1:172.21.15.163:6790/0 conn(0x7f42e00634c0 legacy=0x7f42e00fe7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.312+0000 7f42c7fff700 1 -- 172.21.15.163:0/287987821 >> v1:172.21.15.163:6789/0 conn(0x7f42e006a860 legacy=0x7f42e0104610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.312+0000 7f42c7fff700 1 -- 172.21.15.163:0/287987821 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e010cd20 con 0x7f42e00ff140 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.312+0000 7f42e580e700 1 -- 172.21.15.163:0/287987821 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e010bd10 con 0x7f42e00ff140 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.312+0000 7f42e580e700 10 client.24781.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.312+0000 7f42e580e700 1 -- 172.21.15.163:0/287987821 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e010c180 con 0x7f42e00ff140 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.312+0000 7f42e580e700 1 librados: init done 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.312+0000 7f42c7fff700 1 -- 172.21.15.163:0/287987821 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f42d40028d0 con 0x7f42e00ff140 2024-02-19T21:04:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.312+0000 7f42c7fff700 1 -- 172.21.15.163:0/287987821 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f42d40039c0 con 0x7f42e00ff140 2024-02-19T21:04:24.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.314+0000 7f42e580e700 1 -- 172.21.15.163:0/287987821 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42cc004f50 con 0x7f42e00ff140 2024-02-19T21:04:24.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.314+0000 7f42c7fff700 1 -- 172.21.15.163:0/287987821 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f42d4003c50 con 0x7f42e00ff140 2024-02-19T21:04:24.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.314+0000 7f42c7fff700 1 -- 172.21.15.163:0/287987821 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(96..96 src has 43..96) v4 ==== 6205+0+0 (unknown 394739384 0 0) 0x7f42d40789b0 con 0x7f42e00ff140 2024-02-19T21:04:24.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.314+0000 7f42c7fff700 10 client.24781.objecter ms_dispatch 0x7f42e0001550 osd_map(96..96 src has 43..96) v4 2024-02-19T21:04:24.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.314+0000 7f42c7fff700 3 client.24781.objecter handle_osd_map got epochs [96,96] > 0 2024-02-19T21:04:24.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.314+0000 7f42c7fff700 3 client.24781.objecter handle_osd_map decoding full epoch 96 2024-02-19T21:04:24.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.314+0000 7f42c7fff700 20 client.24781.objecter dump_active .. 0 homeless 2024-02-19T21:04:24.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.315+0000 7f42c7fff700 10 client.24781.objecter ms_handle_connect 0x7f42c8063a70 2024-02-19T21:04:24.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.322+0000 7f42c7fff700 1 -- 172.21.15.163:0/287987821 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f42d404c5a0 con 0x7f42e00ff140 2024-02-19T21:04:24.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.549+0000 7f42e580e700 1 -- 172.21.15.163:0/287987821 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f42cc005780 con 0x7f42e00ff140 2024-02-19T21:04:24.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.724+0000 7f42c7fff700 1 -- 172.21.15.163:0/287987821 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v97) v1 ==== 87+0+0 (unknown 72982671 0 0) 0x7f42d404aa10 con 0x7f42e00ff140 2024-02-19T21:04:24.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.728+0000 7f42e580e700 10 librados: watch_flush enter 2024-02-19T21:04:24.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.728+0000 7f42e580e700 10 librados: watch_flush exit 2024-02-19T21:04:24.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.728+0000 7f42e580e700 20 client.24781.objecter shutdown clearing up homeless session... 2024-02-19T21:04:24.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.728+0000 7f42e580e700 10 client.24781.objecter successfully canceled tick 2024-02-19T21:04:24.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.728+0000 7f42e580e700 1 -- 172.21.15.163:0/287987821 >> v1:172.21.15.163:6800/1552514681 conn(0x7f42c8063a70 legacy=0x7f42c8065f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:24.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.728+0000 7f42e580e700 1 -- 172.21.15.163:0/287987821 >> v1:172.21.15.193:6789/0 conn(0x7f42e00ff140 legacy=0x7f42e0107de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:24.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.728+0000 7f42e580e700 1 -- 172.21.15.163:0/287987821 shutdown_connections 2024-02-19T21:04:24.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.728+0000 7f42e580e700 1 -- 172.21.15.163:0/287987821 >> 172.21.15.163:0/287987821 conn(0x7f42e006fdc0 msgr2=0x7f42e006d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:24.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.728+0000 7f42e580e700 1 -- 172.21.15.163:0/287987821 shutdown_connections 2024-02-19T21:04:24.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.728+0000 7f42e580e700 1 -- 172.21.15.163:0/287987821 wait complete. 2024-02-19T21:04:24.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:23.728+0000 7f42e580e700 1 librados: shutdown 2024-02-19T21:04:24.195 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:04:24.379 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:24.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.408+0000 7fd9ef815700 1 -- 172.21.15.163:0/816365348 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9cc000b90 con 0x7fd9d4063e40 2024-02-19T21:04:24.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.422+0000 7fd9de7fc700 1 -- 172.21.15.163:0/816365348 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+357119 (unknown 2965378022 0 3526310952) 0x7fd9cc000b90 con 0x7fd9d4063e40 2024-02-19T21:04:24.420 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:04:24.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.426+0000 7fd9ef815700 10 librados: watch_flush enter 2024-02-19T21:04:24.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.426+0000 7fd9ef815700 10 librados: watch_flush exit 2024-02-19T21:04:24.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.426+0000 7fd9ef815700 20 client.34784.objecter shutdown clearing up homeless session... 2024-02-19T21:04:24.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.426+0000 7fd9ef815700 10 client.34784.objecter successfully canceled tick 2024-02-19T21:04:24.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.426+0000 7fd9ef815700 1 -- 172.21.15.163:0/816365348 >> v1:172.21.15.163:6800/1552514681 conn(0x7fd9d4063e40 legacy=0x7fd9d4066300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:24.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.426+0000 7fd9ef815700 1 -- 172.21.15.163:0/816365348 >> v1:172.21.15.163:6790/0 conn(0x7fd9e81031b0 legacy=0x7fd9e811c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:24.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.426+0000 7fd9ef815700 1 -- 172.21.15.163:0/816365348 shutdown_connections 2024-02-19T21:04:24.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.426+0000 7fd9ef815700 1 -- 172.21.15.163:0/816365348 >> 172.21.15.163:0/816365348 conn(0x7fd9e80f2470 msgr2=0x7fd9e80601c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:24.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.426+0000 7fd9ef815700 1 -- 172.21.15.163:0/816365348 shutdown_connections 2024-02-19T21:04:24.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.426+0000 7fd9ef815700 1 -- 172.21.15.163:0/816365348 wait complete. 2024-02-19T21:04:24.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:24.426+0000 7fd9ef815700 1 librados: shutdown 2024-02-19T21:04:24.424 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:04:24.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:23.708613+0000 mon.a (mon.0) 746 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:04:24.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:23.725572+0000 mon.a (mon.0) 747 : audit [INF] from='client.24781 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:04:24.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:23.725656+0000 mon.a (mon.0) 748 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-19T21:04:24.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:24.041819+0000 osd.7 (osd.7) 61 : cluster [DBG] 5.9 deep-scrub starts 2024-02-19T21:04:24.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:24.046030+0000 osd.7 (osd.7) 62 : cluster [DBG] 5.9 deep-scrub ok 2024-02-19T21:04:24.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:24.217632+0000 osd.5 (osd.5) 61 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:04:24.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:24.409562+0000 mgr.y (mgr.14550) 128 : audit [DBG] from='client.34784 v1:172.21.15.163:0/816365348' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:04:24.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:23.708613+0000 mon.a (mon.0) 746 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:04:24.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:23.725572+0000 mon.a (mon.0) 747 : audit [INF] from='client.24781 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:04:24.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:23.725656+0000 mon.a (mon.0) 748 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-19T21:04:24.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:24.041819+0000 osd.7 (osd.7) 61 : cluster [DBG] 5.9 deep-scrub starts 2024-02-19T21:04:24.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:24.046030+0000 osd.7 (osd.7) 62 : cluster [DBG] 5.9 deep-scrub ok 2024-02-19T21:04:24.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:24.217632+0000 osd.5 (osd.5) 61 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:04:24.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:24.409562+0000 mgr.y (mgr.14550) 128 : audit [DBG] from='client.34784 v1:172.21.15.163:0/816365348' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:04:25.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:23.708613+0000 mon.a (mon.0) 746 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:04:25.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:23.725572+0000 mon.a (mon.0) 747 : audit [INF] from='client.24781 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:04:25.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:23.725656+0000 mon.a (mon.0) 748 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-02-19T21:04:25.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:24.041819+0000 osd.7 (osd.7) 61 : cluster [DBG] 5.9 deep-scrub starts 2024-02-19T21:04:25.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:24.046030+0000 osd.7 (osd.7) 62 : cluster [DBG] 5.9 deep-scrub ok 2024-02-19T21:04:25.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:24.217632+0000 osd.5 (osd.5) 61 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:04:25.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:24.409562+0000 mgr.y (mgr.14550) 128 : audit [DBG] from='client.34784 v1:172.21.15.163:0/816365348' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:04:25.721 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:24.953854+0000 mgr.y (mgr.14550) 129 : cluster [DBG] pgmap v111: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-19T21:04:25.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:24.953854+0000 mgr.y (mgr.14550) 129 : cluster [DBG] pgmap v111: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-19T21:04:26.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:24.953854+0000 mgr.y (mgr.14550) 129 : cluster [DBG] pgmap v111: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-19T21:04:26.915 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:04:28.064 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:04:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:04:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:04:28.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:26.954245+0000 mgr.y (mgr.14550) 130 : cluster [DBG] pgmap v112: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:04:28.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:26.954245+0000 mgr.y (mgr.14550) 130 : cluster [DBG] pgmap v112: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:04:28.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:26.954245+0000 mgr.y (mgr.14550) 130 : cluster [DBG] pgmap v112: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:04:28.953 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:28.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:29.058 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:04:29.086 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:29.410 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-02-19T21:04:23.720879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5796,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":896,"num_read_kb":648,"num_write":428,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":482,"ondisk_log_size":482,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2382060,"kb_used_data":5204,"kb_used_omap":0,"kb_used_meta":2376704,"kb_avail":747579156,"statfs":{"total":767960285184,"available":765521055744,"internally_reserved":0,"allocated":5328896,"data_stored":2502545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2433744896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.769548"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.695778+0000","last_change":"2024-02-19T21:03:27.633572+0000","last_active":"2024-02-19T21:04:17.695778+0000","last_peered":"2024-02-19T21:04:17.695778+0000","last_clean":"2024-02-19T21:04:17.695778+0000","last_became_active":"2024-02-19T21:01:22.609059+0000","last_became_peered":"2024-02-19T21:01:22.609059+0000","last_unstale":"2024-02-19T21:04:17.695778+0000","last_undegraded":"2024-02-19T21:04:17.695778+0000","last_fullsized":"2024-02-19T21:04:17.695778+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:27.633492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:27.633492+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.692796+0000","last_change":"2024-02-19T21:03:05.073746+0000","last_active":"2024-02-19T21:04:17.692796+0000","last_peered":"2024-02-19T21:04:17.692796+0000","last_clean":"2024-02-19T21:04:17.692796+0000","last_became_active":"2024-02-19T21:01:20.289208+0000","last_became_peered":"2024-02-19T21:01:20.289208+0000","last_unstale":"2024-02-19T21:04:17.692796+0000","last_undegraded":"2024-02-19T21:04:17.692796+0000","last_fullsized":"2024-02-19T21:04:17.692796+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.073664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.073664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:13.983654+0000","last_change":"2024-02-19T21:03:04.644754+0000","last_active":"2024-02-19T21:04:13.983654+0000","last_peered":"2024-02-19T21:04:13.983654+0000","last_clean":"2024-02-19T21:04:13.983654+0000","last_became_active":"2024-02-19T21:01:16.278753+0000","last_became_peered":"2024-02-19T21:01:16.278753+0000","last_unstale":"2024-02-19T21:04:13.983654+0000","last_undegraded":"2024-02-19T21:04:13.983654+0000","last_fullsized":"2024-02-19T21:04:13.983654+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.644689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.644689+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":80,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.695987+0000","last_change":"2024-02-19T21:04:14.826385+0000","last_active":"2024-02-19T21:04:17.695987+0000","last_peered":"2024-02-19T21:04:17.695987+0000","last_clean":"2024-02-19T21:04:17.695987+0000","last_became_active":"2024-02-19T21:01:18.283222+0000","last_became_peered":"2024-02-19T21:01:18.283222+0000","last_unstale":"2024-02-19T21:04:17.695987+0000","last_undegraded":"2024-02-19T21:04:17.695987+0000","last_fullsized":"2024-02-19T21:04:17.695987+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:04:14.826273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:14.826273+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":51,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693016+0000","last_change":"2024-02-19T21:04:08.178302+0000","last_active":"2024-02-19T21:04:17.693016+0000","last_peered":"2024-02-19T21:04:17.693016+0000","last_clean":"2024-02-19T21:04:17.693016+0000","last_became_active":"2024-02-19T21:01:22.293432+0000","last_became_peered":"2024-02-19T21:01:22.293432+0000","last_unstale":"2024-02-19T21:04:17.693016+0000","last_undegraded":"2024-02-19T21:04:17.693016+0000","last_fullsized":"2024-02-19T21:04:17.693016+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.178225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.178225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:16.276652+0000","last_change":"2024-02-19T21:04:16.276617+0000","last_active":"2024-02-19T21:04:16.276652+0000","last_peered":"2024-02-19T21:04:16.276652+0000","last_clean":"2024-02-19T21:04:16.276652+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:04:16.276652+0000","last_undegraded":"2024-02-19T21:04:16.276652+0000","last_fullsized":"2024-02-19T21:04:16.276652+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:16.276518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.276518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.696288+0000","last_change":"2024-02-19T21:04:02.781654+0000","last_active":"2024-02-19T21:04:17.696288+0000","last_peered":"2024-02-19T21:04:17.696288+0000","last_clean":"2024-02-19T21:04:17.696288+0000","last_became_active":"2024-02-19T21:01:16.272132+0000","last_became_peered":"2024-02-19T21:01:16.272132+0000","last_unstale":"2024-02-19T21:04:17.696288+0000","last_undegraded":"2024-02-19T21:04:17.696288+0000","last_fullsized":"2024-02-19T21:04:17.696288+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:02.781556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:02.781556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":66,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.692949+0000","last_change":"2024-02-19T21:03:16.047345+0000","last_active":"2024-02-19T21:04:17.692949+0000","last_peered":"2024-02-19T21:04:17.692949+0000","last_clean":"2024-02-19T21:04:17.692949+0000","last_became_active":"2024-02-19T21:01:18.287645+0000","last_became_peered":"2024-02-19T21:01:18.287645+0000","last_unstale":"2024-02-19T21:04:17.692949+0000","last_undegraded":"2024-02-19T21:04:17.692949+0000","last_fullsized":"2024-02-19T21:04:17.692949+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:16.047258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:16.047258+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":51,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:19.314417+0000","last_change":"2024-02-19T21:04:19.314335+0000","last_active":"2024-02-19T21:04:19.314417+0000","last_peered":"2024-02-19T21:04:19.314417+0000","last_clean":"2024-02-19T21:04:19.314417+0000","last_became_active":"2024-02-19T21:01:22.293282+0000","last_became_peered":"2024-02-19T21:01:22.293282+0000","last_unstale":"2024-02-19T21:04:19.314417+0000","last_undegraded":"2024-02-19T21:04:19.314417+0000","last_fullsized":"2024-02-19T21:04:19.314417+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:19.314241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.314241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743508+0000","last_change":"2024-02-19T21:04:10.891192+0000","last_active":"2024-02-19T21:04:17.743508+0000","last_peered":"2024-02-19T21:04:17.743508+0000","last_clean":"2024-02-19T21:04:17.743508+0000","last_became_active":"2024-02-19T21:01:20.287940+0000","last_became_peered":"2024-02-19T21:01:20.287940+0000","last_unstale":"2024-02-19T21:04:17.743508+0000","last_undegraded":"2024-02-19T21:04:17.743508+0000","last_fullsized":"2024-02-19T21:04:17.743508+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:10.891113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:10.891113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":56,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743739+0000","last_change":"2024-02-19T21:03:26.793343+0000","last_active":"2024-02-19T21:04:17.743739+0000","last_peered":"2024-02-19T21:04:17.743739+0000","last_clean":"2024-02-19T21:04:17.743739+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:04:17.743739+0000","last_undegraded":"2024-02-19T21:04:17.743739+0000","last_fullsized":"2024-02-19T21:04:17.743739+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:26.793263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:26.793263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693317+0000","last_change":"2024-02-19T21:04:12.133700+0000","last_active":"2024-02-19T21:04:17.693317+0000","last_peered":"2024-02-19T21:04:17.693317+0000","last_clean":"2024-02-19T21:04:17.693317+0000","last_became_active":"2024-02-19T21:01:18.285735+0000","last_became_peered":"2024-02-19T21:01:18.285735+0000","last_unstale":"2024-02-19T21:04:17.693317+0000","last_undegraded":"2024-02-19T21:04:17.693317+0000","last_fullsized":"2024-02-19T21:04:17.693317+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:04:12.133628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:12.133628+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:13.983455+0000","last_change":"2024-02-19T21:03:00.570766+0000","last_active":"2024-02-19T21:04:13.983455+0000","last_peered":"2024-02-19T21:04:13.983455+0000","last_clean":"2024-02-19T21:04:13.983455+0000","last_became_active":"2024-02-19T21:01:22.292714+0000","last_became_peered":"2024-02-19T21:01:22.292714+0000","last_unstale":"2024-02-19T21:04:13.983455+0000","last_undegraded":"2024-02-19T21:04:13.983455+0000","last_fullsized":"2024-02-19T21:04:13.983455+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:00.570677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:00.570677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.742839+0000","last_change":"2024-02-19T21:03:26.922649+0000","last_active":"2024-02-19T21:04:17.742839+0000","last_peered":"2024-02-19T21:04:17.742839+0000","last_clean":"2024-02-19T21:04:17.742839+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:04:17.742839+0000","last_undegraded":"2024-02-19T21:04:17.742839+0000","last_fullsized":"2024-02-19T21:04:17.742839+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:26.922572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:26.922572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":51,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743040+0000","last_change":"2024-02-19T21:02:57.552979+0000","last_active":"2024-02-19T21:04:17.743040+0000","last_peered":"2024-02-19T21:04:17.743040+0000","last_clean":"2024-02-19T21:04:17.743040+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:04:17.743040+0000","last_undegraded":"2024-02-19T21:04:17.743040+0000","last_fullsized":"2024-02-19T21:04:17.743040+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.552886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.552886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":71,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693171+0000","last_change":"2024-02-19T21:04:11.165275+0000","last_active":"2024-02-19T21:04:17.693171+0000","last_peered":"2024-02-19T21:04:17.693171+0000","last_clean":"2024-02-19T21:04:17.693171+0000","last_became_active":"2024-02-19T21:01:18.285269+0000","last_became_peered":"2024-02-19T21:01:18.285269+0000","last_unstale":"2024-02-19T21:04:17.693171+0000","last_undegraded":"2024-02-19T21:04:17.693171+0000","last_fullsized":"2024-02-19T21:04:17.693171+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:11.165197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.165197+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":51,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.696878+0000","last_change":"2024-02-19T21:04:03.806998+0000","last_active":"2024-02-19T21:04:17.696878+0000","last_peered":"2024-02-19T21:04:17.696878+0000","last_clean":"2024-02-19T21:04:17.696878+0000","last_became_active":"2024-02-19T21:01:22.607099+0000","last_became_peered":"2024-02-19T21:01:22.607099+0000","last_unstale":"2024-02-19T21:04:17.696878+0000","last_undegraded":"2024-02-19T21:04:17.696878+0000","last_fullsized":"2024-02-19T21:04:17.696878+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:03.806903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:03.806903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693386+0000","last_change":"2024-02-19T21:03:09.010420+0000","last_active":"2024-02-19T21:04:17.693386+0000","last_peered":"2024-02-19T21:04:17.693386+0000","last_clean":"2024-02-19T21:04:17.693386+0000","last_became_active":"2024-02-19T21:01:20.291672+0000","last_became_peered":"2024-02-19T21:01:20.291672+0000","last_unstale":"2024-02-19T21:04:17.693386+0000","last_undegraded":"2024-02-19T21:04:17.693386+0000","last_fullsized":"2024-02-19T21:04:17.693386+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:09.010341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:09.010341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.696023+0000","last_change":"2024-02-19T21:03:20.637528+0000","last_active":"2024-02-19T21:04:17.696023+0000","last_peered":"2024-02-19T21:04:17.696023+0000","last_clean":"2024-02-19T21:04:17.696023+0000","last_became_active":"2024-02-19T21:01:16.274811+0000","last_became_peered":"2024-02-19T21:01:16.274811+0000","last_unstale":"2024-02-19T21:04:17.696023+0000","last_undegraded":"2024-02-19T21:04:17.696023+0000","last_fullsized":"2024-02-19T21:04:17.696023+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:20.637465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:20.637465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.696325+0000","last_change":"2024-02-19T21:02:57.819231+0000","last_active":"2024-02-19T21:04:17.696325+0000","last_peered":"2024-02-19T21:04:17.696325+0000","last_clean":"2024-02-19T21:04:17.696325+0000","last_became_active":"2024-02-19T21:01:18.285926+0000","last_became_peered":"2024-02-19T21:01:18.285926+0000","last_unstale":"2024-02-19T21:04:17.696325+0000","last_undegraded":"2024-02-19T21:04:17.696325+0000","last_fullsized":"2024-02-19T21:04:17.696325+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:02:57.819125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.819125+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693768+0000","last_change":"2024-02-19T21:03:12.081314+0000","last_active":"2024-02-19T21:04:17.693768+0000","last_peered":"2024-02-19T21:04:17.693768+0000","last_clean":"2024-02-19T21:04:17.693768+0000","last_became_active":"2024-02-19T21:01:22.607435+0000","last_became_peered":"2024-02-19T21:01:22.607435+0000","last_unstale":"2024-02-19T21:04:17.693768+0000","last_undegraded":"2024-02-19T21:04:17.693768+0000","last_fullsized":"2024-02-19T21:04:17.693768+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:03:12.081227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:14.313733+0000","last_change":"2024-02-19T21:02:54.544941+0000","last_active":"2024-02-19T21:04:14.313733+0000","last_peered":"2024-02-19T21:04:14.313733+0000","last_clean":"2024-02-19T21:04:14.313733+0000","last_became_active":"2024-02-19T21:01:20.290705+0000","last_became_peered":"2024-02-19T21:01:20.290705+0000","last_unstale":"2024-02-19T21:04:14.313733+0000","last_undegraded":"2024-02-19T21:04:14.313733+0000","last_fullsized":"2024-02-19T21:04:14.313733+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:54.544834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:54.544834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.744263+0000","last_change":"2024-02-19T21:02:47.805245+0000","last_active":"2024-02-19T21:04:17.744263+0000","last_peered":"2024-02-19T21:04:17.744263+0000","last_clean":"2024-02-19T21:04:17.744263+0000","last_became_active":"2024-02-19T21:01:16.274524+0000","last_became_peered":"2024-02-19T21:01:16.274524+0000","last_unstale":"2024-02-19T21:04:17.744263+0000","last_undegraded":"2024-02-19T21:04:17.744263+0000","last_fullsized":"2024-02-19T21:04:17.744263+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.805171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.805171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":75,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.694714+0000","last_change":"2024-02-19T21:02:47.238547+0000","last_active":"2024-02-19T21:04:17.694714+0000","last_peered":"2024-02-19T21:04:17.694714+0000","last_clean":"2024-02-19T21:04:17.694714+0000","last_became_active":"2024-02-19T21:01:18.289712+0000","last_became_peered":"2024-02-19T21:01:18.289712+0000","last_unstale":"2024-02-19T21:04:17.694714+0000","last_undegraded":"2024-02-19T21:04:17.694714+0000","last_fullsized":"2024-02-19T21:04:17.694714+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:47.238470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.238470+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:14.313407+0000","last_change":"2024-02-19T21:03:06.456310+0000","last_active":"2024-02-19T21:04:14.313407+0000","last_peered":"2024-02-19T21:04:14.313407+0000","last_clean":"2024-02-19T21:04:14.313407+0000","last_became_active":"2024-02-19T21:01:22.293230+0000","last_became_peered":"2024-02-19T21:01:22.293230+0000","last_unstale":"2024-02-19T21:04:14.313407+0000","last_undegraded":"2024-02-19T21:04:14.313407+0000","last_fullsized":"2024-02-19T21:04:14.313407+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.456212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.456212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:13.984718+0000","last_change":"2024-02-19T21:03:08.636037+0000","last_active":"2024-02-19T21:04:13.984718+0000","last_peered":"2024-02-19T21:04:13.984718+0000","last_clean":"2024-02-19T21:04:13.984718+0000","last_became_active":"2024-02-19T21:01:20.288016+0000","last_became_peered":"2024-02-19T21:01:20.288016+0000","last_unstale":"2024-02-19T21:04:13.984718+0000","last_undegraded":"2024-02-19T21:04:13.984718+0000","last_fullsized":"2024-02-19T21:04:13.984718+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:08.635943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.635943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.695666+0000","last_change":"2024-02-19T21:03:21.619457+0000","last_active":"2024-02-19T21:04:17.695666+0000","last_peered":"2024-02-19T21:04:17.695666+0000","last_clean":"2024-02-19T21:04:17.695666+0000","last_became_active":"2024-02-19T21:01:16.277826+0000","last_became_peered":"2024-02-19T21:01:16.277826+0000","last_unstale":"2024-02-19T21:04:17.695666+0000","last_undegraded":"2024-02-19T21:04:17.695666+0000","last_fullsized":"2024-02-19T21:04:17.695666+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:21.619389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:21.619389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":74,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.744617+0000","last_change":"2024-02-19T21:04:04.812216+0000","last_active":"2024-02-19T21:04:17.744617+0000","last_peered":"2024-02-19T21:04:17.744617+0000","last_clean":"2024-02-19T21:04:17.744617+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:04:17.744617+0000","last_undegraded":"2024-02-19T21:04:17.744617+0000","last_fullsized":"2024-02-19T21:04:17.744617+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:04:04.812144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:04.812144+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:18.877486+0000","last_change":"2024-02-19T21:04:18.877455+0000","last_active":"2024-02-19T21:04:18.877486+0000","last_peered":"2024-02-19T21:04:18.877486+0000","last_clean":"2024-02-19T21:04:18.877486+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:04:18.877486+0000","last_undegraded":"2024-02-19T21:04:18.877486+0000","last_fullsized":"2024-02-19T21:04:18.877486+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:04:18.877385+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":101,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:19.455286+0000","last_change":"2024-02-19T21:04:05.787496+0000","last_active":"2024-02-19T21:04:19.455286+0000","last_peered":"2024-02-19T21:04:19.455286+0000","last_clean":"2024-02-19T21:04:19.455286+0000","last_became_active":"2024-02-19T21:01:20.290567+0000","last_became_peered":"2024-02-19T21:01:20.290567+0000","last_unstale":"2024-02-19T21:04:19.455286+0000","last_undegraded":"2024-02-19T21:04:19.455286+0000","last_fullsized":"2024-02-19T21:04:19.455286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:05.787415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:05.787415+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.742240+0000","last_change":"2024-02-19T21:03:06.000805+0000","last_active":"2024-02-19T21:04:17.742240+0000","last_peered":"2024-02-19T21:04:17.742240+0000","last_clean":"2024-02-19T21:04:17.742240+0000","last_became_active":"2024-02-19T21:01:16.272284+0000","last_became_peered":"2024-02-19T21:01:16.272284+0000","last_unstale":"2024-02-19T21:04:17.742240+0000","last_undegraded":"2024-02-19T21:04:17.742240+0000","last_fullsized":"2024-02-19T21:04:17.742240+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.000732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.000732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":69,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:13.984642+0000","last_change":"2024-02-19T21:04:04.545245+0000","last_active":"2024-02-19T21:04:13.984642+0000","last_peered":"2024-02-19T21:04:13.984642+0000","last_clean":"2024-02-19T21:04:13.984642+0000","last_became_active":"2024-02-19T21:01:18.293865+0000","last_became_peered":"2024-02-19T21:01:18.293865+0000","last_unstale":"2024-02-19T21:04:13.984642+0000","last_undegraded":"2024-02-19T21:04:13.984642+0000","last_fullsized":"2024-02-19T21:04:13.984642+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:04:04.545131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:04.545131+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":51,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.696168+0000","last_change":"2024-02-19T21:04:10.863028+0000","last_active":"2024-02-19T21:04:17.696168+0000","last_peered":"2024-02-19T21:04:17.696168+0000","last_clean":"2024-02-19T21:04:17.696168+0000","last_became_active":"2024-02-19T21:01:22.608997+0000","last_became_peered":"2024-02-19T21:01:22.608997+0000","last_unstale":"2024-02-19T21:04:17.696168+0000","last_undegraded":"2024-02-19T21:04:17.696168+0000","last_fullsized":"2024-02-19T21:04:17.696168+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:10.862926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:10.862926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.744052+0000","last_change":"2024-02-19T21:03:04.486168+0000","last_active":"2024-02-19T21:04:17.744052+0000","last_peered":"2024-02-19T21:04:17.744052+0000","last_clean":"2024-02-19T21:04:17.744052+0000","last_became_active":"2024-02-19T21:01:20.287961+0000","last_became_peered":"2024-02-19T21:01:20.287961+0000","last_unstale":"2024-02-19T21:04:17.744052+0000","last_undegraded":"2024-02-19T21:04:17.744052+0000","last_fullsized":"2024-02-19T21:04:17.744052+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.486067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.486067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.744751+0000","last_change":"2024-02-19T21:04:13.816617+0000","last_active":"2024-02-19T21:04:17.744751+0000","last_peered":"2024-02-19T21:04:17.744751+0000","last_clean":"2024-02-19T21:04:17.744751+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:04:17.744751+0000","last_undegraded":"2024-02-19T21:04:17.744751+0000","last_fullsized":"2024-02-19T21:04:17.744751+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.816545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.816545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":69,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.695895+0000","last_change":"2024-02-19T21:02:52.859531+0000","last_active":"2024-02-19T21:04:17.695895+0000","last_peered":"2024-02-19T21:04:17.695895+0000","last_clean":"2024-02-19T21:04:17.695895+0000","last_became_active":"2024-02-19T21:01:18.293793+0000","last_became_peered":"2024-02-19T21:01:18.293793+0000","last_unstale":"2024-02-19T21:04:17.695895+0000","last_undegraded":"2024-02-19T21:04:17.695895+0000","last_fullsized":"2024-02-19T21:04:17.695895+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:52.859453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.859453+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":50,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743042+0000","last_change":"2024-02-19T21:04:13.941207+0000","last_active":"2024-02-19T21:04:17.743042+0000","last_peered":"2024-02-19T21:04:17.743042+0000","last_clean":"2024-02-19T21:04:17.743042+0000","last_became_active":"2024-02-19T21:01:22.297693+0000","last_became_peered":"2024-02-19T21:01:22.297693+0000","last_unstale":"2024-02-19T21:04:17.743042+0000","last_undegraded":"2024-02-19T21:04:17.743042+0000","last_fullsized":"2024-02-19T21:04:17.743042+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.941132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.941132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.742516+0000","last_change":"2024-02-19T21:04:11.754594+0000","last_active":"2024-02-19T21:04:17.742516+0000","last_peered":"2024-02-19T21:04:17.742516+0000","last_clean":"2024-02-19T21:04:17.742516+0000","last_became_active":"2024-02-19T21:01:20.290962+0000","last_became_peered":"2024-02-19T21:01:20.290962+0000","last_unstale":"2024-02-19T21:04:17.742516+0000","last_undegraded":"2024-02-19T21:04:17.742516+0000","last_fullsized":"2024-02-19T21:04:17.742516+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.754525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.754525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.745296+0000","last_change":"2024-02-19T21:04:13.205667+0000","last_active":"2024-02-19T21:04:17.745296+0000","last_peered":"2024-02-19T21:04:17.745296+0000","last_clean":"2024-02-19T21:04:17.745296+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:04:17.745296+0000","last_undegraded":"2024-02-19T21:04:17.745296+0000","last_fullsized":"2024-02-19T21:04:17.745296+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.205596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.205596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":61,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.695637+0000","last_change":"2024-02-19T21:03:32.722001+0000","last_active":"2024-02-19T21:04:17.695637+0000","last_peered":"2024-02-19T21:04:17.695637+0000","last_clean":"2024-02-19T21:04:17.695637+0000","last_became_active":"2024-02-19T21:01:18.287874+0000","last_became_peered":"2024-02-19T21:01:18.287874+0000","last_unstale":"2024-02-19T21:04:17.695637+0000","last_undegraded":"2024-02-19T21:04:17.695637+0000","last_fullsized":"2024-02-19T21:04:17.695637+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:03:32.721877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:32.721877+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.695520+0000","last_change":"2024-02-19T21:03:05.734069+0000","last_active":"2024-02-19T21:04:17.695520+0000","last_peered":"2024-02-19T21:04:17.695520+0000","last_clean":"2024-02-19T21:04:17.695520+0000","last_became_active":"2024-02-19T21:01:22.292704+0000","last_became_peered":"2024-02-19T21:01:22.292704+0000","last_unstale":"2024-02-19T21:04:17.695520+0000","last_undegraded":"2024-02-19T21:04:17.695520+0000","last_fullsized":"2024-02-19T21:04:17.695520+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.733980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.733980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:14.311203+0000","last_change":"2024-02-19T21:04:13.337608+0000","last_active":"2024-02-19T21:04:14.311203+0000","last_peered":"2024-02-19T21:04:14.311203+0000","last_clean":"2024-02-19T21:04:14.311203+0000","last_became_active":"2024-02-19T21:01:20.289020+0000","last_became_peered":"2024-02-19T21:01:20.289020+0000","last_unstale":"2024-02-19T21:04:14.311203+0000","last_undegraded":"2024-02-19T21:04:14.311203+0000","last_fullsized":"2024-02-19T21:04:14.311203+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.337498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.337498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:14.313581+0000","last_change":"2024-02-19T21:03:20.445543+0000","last_active":"2024-02-19T21:04:14.313581+0000","last_peered":"2024-02-19T21:04:14.313581+0000","last_clean":"2024-02-19T21:04:14.313581+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:04:14.313581+0000","last_undegraded":"2024-02-19T21:04:14.313581+0000","last_fullsized":"2024-02-19T21:04:14.313581+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:20.445477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:20.445477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":74,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693162+0000","last_change":"2024-02-19T21:04:09.139464+0000","last_active":"2024-02-19T21:04:17.693162+0000","last_peered":"2024-02-19T21:04:17.693162+0000","last_clean":"2024-02-19T21:04:17.693162+0000","last_became_active":"2024-02-19T21:01:18.286128+0000","last_became_peered":"2024-02-19T21:01:18.286128+0000","last_unstale":"2024-02-19T21:04:17.693162+0000","last_undegraded":"2024-02-19T21:04:17.693162+0000","last_fullsized":"2024-02-19T21:04:17.693162+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:04:09.139344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:09.139344+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:13.982992+0000","last_change":"2024-02-19T21:04:08.616495+0000","last_active":"2024-02-19T21:04:13.982992+0000","last_peered":"2024-02-19T21:04:13.982992+0000","last_clean":"2024-02-19T21:04:13.982992+0000","last_became_active":"2024-02-19T21:01:22.293093+0000","last_became_peered":"2024-02-19T21:01:22.293093+0000","last_unstale":"2024-02-19T21:04:13.982992+0000","last_undegraded":"2024-02-19T21:04:13.982992+0000","last_fullsized":"2024-02-19T21:04:13.982992+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.616396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.616396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.696086+0000","last_change":"2024-02-19T21:03:03.776497+0000","last_active":"2024-02-19T21:04:17.696086+0000","last_peered":"2024-02-19T21:04:17.696086+0000","last_clean":"2024-02-19T21:04:17.696086+0000","last_became_active":"2024-02-19T21:01:20.289691+0000","last_became_peered":"2024-02-19T21:01:20.289691+0000","last_unstale":"2024-02-19T21:04:17.696086+0000","last_undegraded":"2024-02-19T21:04:17.696086+0000","last_fullsized":"2024-02-19T21:04:17.696086+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_clean_scrub_stamp":"2024-02-19T21:03:03.776397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743505+0000","last_change":"2024-02-19T21:03:33.049004+0000","last_active":"2024-02-19T21:04:17.743505+0000","last_peered":"2024-02-19T21:04:17.743505+0000","last_clean":"2024-02-19T21:04:17.743505+0000","last_became_active":"2024-02-19T21:01:16.271316+0000","last_became_peered":"2024-02-19T21:01:16.271316+0000","last_unstale":"2024-02-19T21:04:17.743505+0000","last_undegraded":"2024-02-19T21:04:17.743505+0000","last_fullsized":"2024-02-19T21:04:17.743505+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:33.048928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:33.048928+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":54,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.742332+0000","last_change":"2024-02-19T21:03:22.984626+0000","last_active":"2024-02-19T21:04:17.742332+0000","last_peered":"2024-02-19T21:04:17.742332+0000","last_clean":"2024-02-19T21:04:17.742332+0000","last_became_active":"2024-02-19T21:01:30.922577+0000","last_became_peered":"2024-02-19T21:01:30.922577+0000","last_unstale":"2024-02-19T21:04:17.742332+0000","last_undegraded":"2024-02-19T21:04:17.742332+0000","last_fullsized":"2024-02-19T21:04:17.742332+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:22.984547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:22.984547+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":51,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693818+0000","last_change":"2024-02-19T21:04:15.099340+0000","last_active":"2024-02-19T21:04:17.693818+0000","last_peered":"2024-02-19T21:04:17.693818+0000","last_clean":"2024-02-19T21:04:17.693818+0000","last_became_active":"2024-02-19T21:01:22.299969+0000","last_became_peered":"2024-02-19T21:01:22.299969+0000","last_unstale":"2024-02-19T21:04:17.693818+0000","last_undegraded":"2024-02-19T21:04:17.693818+0000","last_fullsized":"2024-02-19T21:04:17.693818+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:15.099208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_clean_scrub_stamp":"2024-02-19T21:04:15.099208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":93,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:19.455066+0000","last_change":"2024-02-19T21:03:26.669863+0000","last_active":"2024-02-19T21:04:19.455066+0000","last_peered":"2024-02-19T21:04:19.455066+0000","last_clean":"2024-02-19T21:04:19.455066+0000","last_became_active":"2024-02-19T21:01:20.283943+0000","last_became_peered":"2024-02-19T21:01:20.283943+0000","last_unstale":"2024-02-19T21:04:19.455066+0000","last_undegraded":"2024-02-19T21:04:19.455066+0000","last_fullsized":"2024-02-19T21:04:19.455066+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:26.669794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:26.669794+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:13.984187+0000","last_change":"2024-02-19T21:03:39.467124+0000","last_active":"2024-02-19T21:04:13.984187+0000","last_peered":"2024-02-19T21:04:13.984187+0000","last_clean":"2024-02-19T21:04:13.984187+0000","last_became_active":"2024-02-19T21:01:16.273181+0000","last_became_peered":"2024-02-19T21:01:16.273181+0000","last_unstale":"2024-02-19T21:04:13.984187+0000","last_undegraded":"2024-02-19T21:04:13.984187+0000","last_fullsized":"2024-02-19T21:04:13.984187+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:39.467059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:39.467059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":71,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.742957+0000","last_change":"2024-02-19T21:04:15.187687+0000","last_active":"2024-02-19T21:04:17.742957+0000","last_peered":"2024-02-19T21:04:17.742957+0000","last_clean":"2024-02-19T21:04:17.742957+0000","last_became_active":"2024-02-19T21:01:18.293786+0000","last_became_peered":"2024-02-19T21:01:18.293786+0000","last_unstale":"2024-02-19T21:04:17.742957+0000","last_undegraded":"2024-02-19T21:04:17.742957+0000","last_fullsized":"2024-02-19T21:04:17.742957+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:15.187587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:15.187587+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:13.984279+0000","last_change":"2024-02-19T21:04:05.558879+0000","last_active":"2024-02-19T21:04:13.984279+0000","last_peered":"2024-02-19T21:04:13.984279+0000","last_clean":"2024-02-19T21:04:13.984279+0000","last_became_active":"2024-02-19T21:01:22.297188+0000","last_became_peered":"2024-02-19T21:01:22.297188+0000","last_unstale":"2024-02-19T21:04:13.984279+0000","last_undegraded":"2024-02-19T21:04:13.984279+0000","last_fullsized":"2024-02-19T21:04:13.984279+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:04:05.558765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:05.558765+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":93,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:19.455416+0000","last_change":"2024-02-19T21:03:11.063302+0000","last_active":"2024-02-19T21:04:19.455416+0000","last_peered":"2024-02-19T21:04:19.455416+0000","last_clean":"2024-02-19T21:04:19.455416+0000","last_became_active":"2024-02-19T21:01:20.288799+0000","last_became_peered":"2024-02-19T21:01:20.288799+0000","last_unstale":"2024-02-19T21:04:19.455416+0000","last_undegraded":"2024-02-19T21:04:19.455416+0000","last_fullsized":"2024-02-19T21:04:19.455416+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:11.063222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:11.063222+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.742440+0000","last_change":"2024-02-19T21:04:09.180232+0000","last_active":"2024-02-19T21:04:17.742440+0000","last_peered":"2024-02-19T21:04:17.742440+0000","last_clean":"2024-02-19T21:04:17.742440+0000","last_became_active":"2024-02-19T21:01:16.271608+0000","last_became_peered":"2024-02-19T21:01:16.271608+0000","last_unstale":"2024-02-19T21:04:17.742440+0000","last_undegraded":"2024-02-19T21:04:17.742440+0000","last_fullsized":"2024-02-19T21:04:17.742440+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:09.180159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:09.180159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.696292+0000","last_change":"2024-02-19T21:03:08.742946+0000","last_active":"2024-02-19T21:04:17.696292+0000","last_peered":"2024-02-19T21:04:17.696292+0000","last_clean":"2024-02-19T21:04:17.696292+0000","last_became_active":"2024-02-19T21:01:18.294369+0000","last_became_peered":"2024-02-19T21:01:18.294369+0000","last_unstale":"2024-02-19T21:04:17.696292+0000","last_undegraded":"2024-02-19T21:04:17.696292+0000","last_fullsized":"2024-02-19T21:04:17.696292+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:03:08.742876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.742876+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":50,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743651+0000","last_change":"2024-02-19T21:04:06.880972+0000","last_active":"2024-02-19T21:04:17.743651+0000","last_peered":"2024-02-19T21:04:17.743651+0000","last_clean":"2024-02-19T21:04:17.743651+0000","last_became_active":"2024-02-19T21:01:22.608285+0000","last_became_peered":"2024-02-19T21:01:22.608285+0000","last_unstale":"2024-02-19T21:04:17.743651+0000","last_undegraded":"2024-02-19T21:04:17.743651+0000","last_fullsized":"2024-02-19T21:04:17.743651+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:06.880899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.880899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.745192+0000","last_change":"2024-02-19T21:03:08.984824+0000","last_active":"2024-02-19T21:04:17.745192+0000","last_peered":"2024-02-19T21:04:17.745192+0000","last_clean":"2024-02-19T21:04:17.745192+0000","last_became_active":"2024-02-19T21:01:20.288589+0000","last_became_peered":"2024-02-19T21:01:20.288589+0000","last_unstale":"2024-02-19T21:04:17.745192+0000","last_undegraded":"2024-02-19T21:04:17.745192+0000","last_fullsized":"2024-02-19T21:04:17.745192+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:08.984720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.984720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.742721+0000","last_change":"2024-02-19T21:04:08.757186+0000","last_active":"2024-02-19T21:04:17.742721+0000","last_peered":"2024-02-19T21:04:17.742721+0000","last_clean":"2024-02-19T21:04:17.742721+0000","last_became_active":"2024-02-19T21:01:16.271210+0000","last_became_peered":"2024-02-19T21:01:16.271210+0000","last_unstale":"2024-02-19T21:04:17.742721+0000","last_undegraded":"2024-02-19T21:04:17.742721+0000","last_fullsized":"2024-02-19T21:04:17.742721+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.757098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.757098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":67,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.696836+0000","last_change":"2024-02-19T21:04:06.807310+0000","last_active":"2024-02-19T21:04:17.696836+0000","last_peered":"2024-02-19T21:04:17.696836+0000","last_clean":"2024-02-19T21:04:17.696836+0000","last_became_active":"2024-02-19T21:01:18.293913+0000","last_became_peered":"2024-02-19T21:01:18.293913+0000","last_unstale":"2024-02-19T21:04:17.696836+0000","last_undegraded":"2024-02-19T21:04:17.696836+0000","last_fullsized":"2024-02-19T21:04:17.696836+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:04:06.807190+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.807190+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":51,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.694421+0000","last_change":"2024-02-19T21:04:08.831108+0000","last_active":"2024-02-19T21:04:17.694421+0000","last_peered":"2024-02-19T21:04:17.694421+0000","last_clean":"2024-02-19T21:04:17.694421+0000","last_became_active":"2024-02-19T21:01:22.297086+0000","last_became_peered":"2024-02-19T21:01:22.297086+0000","last_unstale":"2024-02-19T21:04:17.694421+0000","last_undegraded":"2024-02-19T21:04:17.694421+0000","last_fullsized":"2024-02-19T21:04:17.694421+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.830994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.830994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.696913+0000","last_change":"2024-02-19T21:04:11.841348+0000","last_active":"2024-02-19T21:04:17.696913+0000","last_peered":"2024-02-19T21:04:17.696913+0000","last_clean":"2024-02-19T21:04:17.696913+0000","last_became_active":"2024-02-19T21:01:20.285298+0000","last_became_peered":"2024-02-19T21:01:20.285298+0000","last_unstale":"2024-02-19T21:04:17.696913+0000","last_undegraded":"2024-02-19T21:04:17.696913+0000","last_fullsized":"2024-02-19T21:04:17.696913+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.841184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.841184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.694773+0000","last_change":"2024-02-19T21:02:49.177055+0000","last_active":"2024-02-19T21:04:17.694773+0000","last_peered":"2024-02-19T21:04:17.694773+0000","last_clean":"2024-02-19T21:04:17.694773+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:04:17.694773+0000","last_undegraded":"2024-02-19T21:04:17.694773+0000","last_fullsized":"2024-02-19T21:04:17.694773+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.176969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.176969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":70,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:13.984451+0000","last_change":"2024-02-19T21:04:09.571081+0000","last_active":"2024-02-19T21:04:13.984451+0000","last_peered":"2024-02-19T21:04:13.984451+0000","last_clean":"2024-02-19T21:04:13.984451+0000","last_became_active":"2024-02-19T21:01:18.293773+0000","last_became_peered":"2024-02-19T21:01:18.293773+0000","last_unstale":"2024-02-19T21:04:13.984451+0000","last_undegraded":"2024-02-19T21:04:13.984451+0000","last_fullsized":"2024-02-19T21:04:13.984451+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:09.570972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:09.570972+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.696114+0000","last_change":"2024-02-19T21:03:17.625775+0000","last_active":"2024-02-19T21:04:17.696114+0000","last_peered":"2024-02-19T21:04:17.696114+0000","last_clean":"2024-02-19T21:04:17.696114+0000","last_became_active":"2024-02-19T21:01:22.295912+0000","last_became_peered":"2024-02-19T21:01:22.295912+0000","last_unstale":"2024-02-19T21:04:17.696114+0000","last_undegraded":"2024-02-19T21:04:17.696114+0000","last_fullsized":"2024-02-19T21:04:17.696114+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:03:17.625711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":72,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693690+0000","last_change":"2024-02-19T21:03:21.164366+0000","last_active":"2024-02-19T21:04:17.693690+0000","last_peered":"2024-02-19T21:04:17.693690+0000","last_clean":"2024-02-19T21:04:17.693690+0000","last_became_active":"2024-02-19T21:01:16.273545+0000","last_became_peered":"2024-02-19T21:01:16.273545+0000","last_unstale":"2024-02-19T21:04:17.693690+0000","last_undegraded":"2024-02-19T21:04:17.693690+0000","last_fullsized":"2024-02-19T21:04:17.693690+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:03:21.164286+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":73,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693070+0000","last_change":"2024-02-19T21:03:08.046722+0000","last_active":"2024-02-19T21:04:17.693070+0000","last_peered":"2024-02-19T21:04:17.693070+0000","last_clean":"2024-02-19T21:04:17.693070+0000","last_became_active":"2024-02-19T21:01:18.285672+0000","last_became_peered":"2024-02-19T21:01:18.285672+0000","last_unstale":"2024-02-19T21:04:17.693070+0000","last_undegraded":"2024-02-19T21:04:17.693070+0000","last_fullsized":"2024-02-19T21:04:17.693070+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:03:08.046641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.046641+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.694691+0000","last_change":"2024-02-19T21:04:16.866220+0000","last_active":"2024-02-19T21:04:17.694691+0000","last_peered":"2024-02-19T21:04:17.694691+0000","last_clean":"2024-02-19T21:04:17.694691+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:04:17.694691+0000","last_undegraded":"2024-02-19T21:04:17.694691+0000","last_fullsized":"2024-02-19T21:04:17.694691+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.866116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":50,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:13.983080+0000","last_change":"2024-02-19T21:04:06.599565+0000","last_active":"2024-02-19T21:04:13.983080+0000","last_peered":"2024-02-19T21:04:13.983080+0000","last_clean":"2024-02-19T21:04:13.983080+0000","last_became_active":"2024-02-19T21:01:22.292133+0000","last_became_peered":"2024-02-19T21:01:22.292133+0000","last_unstale":"2024-02-19T21:04:13.983080+0000","last_undegraded":"2024-02-19T21:04:13.983080+0000","last_fullsized":"2024-02-19T21:04:13.983080+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:06.599439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.599439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":65,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:18.242620+0000","last_change":"2024-02-19T21:04:18.242620+0000","last_active":"2024-02-19T21:04:18.242620+0000","last_peered":"2024-02-19T21:04:18.242620+0000","last_clean":"2024-02-19T21:04:18.242620+0000","last_became_active":"2024-02-19T21:01:30.926676+0000","last_became_peered":"2024-02-19T21:01:30.926676+0000","last_unstale":"2024-02-19T21:04:18.242620+0000","last_undegraded":"2024-02-19T21:04:18.242620+0000","last_fullsized":"2024-02-19T21:04:18.242620+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:18.242556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:04:18.242556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":64,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:14.311128+0000","last_change":"2024-02-19T21:02:56.539408+0000","last_active":"2024-02-19T21:04:14.311128+0000","last_peered":"2024-02-19T21:04:14.311128+0000","last_clean":"2024-02-19T21:04:14.311128+0000","last_became_active":"2024-02-19T21:01:18.285891+0000","last_became_peered":"2024-02-19T21:01:18.285891+0000","last_unstale":"2024-02-19T21:04:14.311128+0000","last_undegraded":"2024-02-19T21:04:14.311128+0000","last_fullsized":"2024-02-19T21:04:14.311128+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:02:56.539303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.539303+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":95,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.742927+0000","last_change":"2024-02-19T21:03:10.626933+0000","last_active":"2024-02-19T21:04:17.742927+0000","last_peered":"2024-02-19T21:04:17.742927+0000","last_clean":"2024-02-19T21:04:17.742927+0000","last_became_active":"2024-02-19T21:01:20.288918+0000","last_became_peered":"2024-02-19T21:01:20.288918+0000","last_unstale":"2024-02-19T21:04:17.742927+0000","last_undegraded":"2024-02-19T21:04:17.742927+0000","last_fullsized":"2024-02-19T21:04:17.742927+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:10.626866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:10.626866+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.696746+0000","last_change":"2024-02-19T21:03:12.734312+0000","last_active":"2024-02-19T21:04:17.696746+0000","last_peered":"2024-02-19T21:04:17.696746+0000","last_clean":"2024-02-19T21:04:17.696746+0000","last_became_active":"2024-02-19T21:01:22.291396+0000","last_became_peered":"2024-02-19T21:01:22.291396+0000","last_unstale":"2024-02-19T21:04:17.696746+0000","last_undegraded":"2024-02-19T21:04:17.696746+0000","last_fullsized":"2024-02-19T21:04:17.696746+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:12.734219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:12.734219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.742537+0000","last_change":"2024-02-19T21:02:58.986556+0000","last_active":"2024-02-19T21:04:17.742537+0000","last_peered":"2024-02-19T21:04:17.742537+0000","last_clean":"2024-02-19T21:04:17.742537+0000","last_became_active":"2024-02-19T21:01:16.270159+0000","last_became_peered":"2024-02-19T21:01:16.270159+0000","last_unstale":"2024-02-19T21:04:17.742537+0000","last_undegraded":"2024-02-19T21:04:17.742537+0000","last_fullsized":"2024-02-19T21:04:17.742537+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:58.986458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:58.986458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":139,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.744194+0000","last_change":"2024-02-19T21:04:06.383697+0000","last_active":"2024-02-19T21:04:17.744194+0000","last_peered":"2024-02-19T21:04:17.744194+0000","last_clean":"2024-02-19T21:04:17.744194+0000","last_became_active":"2024-02-19T21:01:18.289076+0000","last_became_peered":"2024-02-19T21:01:18.289076+0000","last_unstale":"2024-02-19T21:04:17.744194+0000","last_undegraded":"2024-02-19T21:04:17.744194+0000","last_fullsized":"2024-02-19T21:04:17.744194+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-19T21:04:06.383559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.383559+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:13.983436+0000","last_change":"2024-02-19T21:02:56.510539+0000","last_active":"2024-02-19T21:04:13.983436+0000","last_peered":"2024-02-19T21:04:13.983436+0000","last_clean":"2024-02-19T21:04:13.983436+0000","last_became_active":"2024-02-19T21:01:20.290758+0000","last_became_peered":"2024-02-19T21:01:20.290758+0000","last_unstale":"2024-02-19T21:04:13.983436+0000","last_undegraded":"2024-02-19T21:04:13.983436+0000","last_fullsized":"2024-02-19T21:04:13.983436+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.510474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":50,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.744156+0000","last_change":"2024-02-19T21:04:11.900738+0000","last_active":"2024-02-19T21:04:17.744156+0000","last_peered":"2024-02-19T21:04:17.744156+0000","last_clean":"2024-02-19T21:04:17.744156+0000","last_became_active":"2024-02-19T21:01:22.607883+0000","last_became_peered":"2024-02-19T21:01:22.607883+0000","last_unstale":"2024-02-19T21:04:17.744156+0000","last_undegraded":"2024-02-19T21:04:17.744156+0000","last_fullsized":"2024-02-19T21:04:17.744156+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.900625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.900625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":231,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:18.843555+0000","last_change":"2024-02-19T21:02:55.568674+0000","last_active":"2024-02-19T21:04:18.843555+0000","last_peered":"2024-02-19T21:04:18.843555+0000","last_clean":"2024-02-19T21:04:18.843555+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:04:18.843555+0000","last_undegraded":"2024-02-19T21:04:18.843555+0000","last_fullsized":"2024-02-19T21:04:18.843555+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:02:55.568593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.568593+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":177,"num_read_kb":172,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.694735+0000","last_change":"2024-02-19T21:03:35.999651+0000","last_active":"2024-02-19T21:04:17.694735+0000","last_peered":"2024-02-19T21:04:17.694735+0000","last_clean":"2024-02-19T21:04:17.694735+0000","last_became_active":"2024-02-19T21:01:16.276032+0000","last_became_peered":"2024-02-19T21:01:16.276032+0000","last_unstale":"2024-02-19T21:04:17.694735+0000","last_undegraded":"2024-02-19T21:04:17.694735+0000","last_fullsized":"2024-02-19T21:04:17.694735+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:35.999575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:35.999575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":66,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.694444+0000","last_change":"2024-02-19T21:03:27.966621+0000","last_active":"2024-02-19T21:04:17.694444+0000","last_peered":"2024-02-19T21:04:17.694444+0000","last_clean":"2024-02-19T21:04:17.694444+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:04:17.694444+0000","last_undegraded":"2024-02-19T21:04:17.694444+0000","last_fullsized":"2024-02-19T21:04:17.694444+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:03:27.966528+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.694623+0000","last_change":"2024-02-19T21:04:15.868824+0000","last_active":"2024-02-19T21:04:17.694623+0000","last_peered":"2024-02-19T21:04:17.694623+0000","last_clean":"2024-02-19T21:04:17.694623+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:04:17.694623+0000","last_undegraded":"2024-02-19T21:04:17.694623+0000","last_fullsized":"2024-02-19T21:04:17.694623+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:15.868726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:15.868726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":51,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743997+0000","last_change":"2024-02-19T21:04:05.136254+0000","last_active":"2024-02-19T21:04:17.743997+0000","last_peered":"2024-02-19T21:04:17.743997+0000","last_clean":"2024-02-19T21:04:17.743997+0000","last_became_active":"2024-02-19T21:01:22.292299+0000","last_became_peered":"2024-02-19T21:01:22.292299+0000","last_unstale":"2024-02-19T21:04:17.743997+0000","last_undegraded":"2024-02-19T21:04:17.743997+0000","last_fullsized":"2024-02-19T21:04:17.743997+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:05.136186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:05.136186+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.696057+0000","last_change":"2024-02-19T21:03:19.665710+0000","last_active":"2024-02-19T21:04:17.696057+0000","last_peered":"2024-02-19T21:04:17.696057+0000","last_clean":"2024-02-19T21:04:17.696057+0000","last_became_active":"2024-02-19T21:01:16.274290+0000","last_became_peered":"2024-02-19T21:01:16.274290+0000","last_unstale":"2024-02-19T21:04:17.696057+0000","last_undegraded":"2024-02-19T21:04:17.696057+0000","last_fullsized":"2024-02-19T21:04:17.696057+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:19.665646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:19.665646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.742813+0000","last_change":"2024-02-19T21:04:14.215108+0000","last_active":"2024-02-19T21:04:17.742813+0000","last_peered":"2024-02-19T21:04:17.742813+0000","last_clean":"2024-02-19T21:04:17.742813+0000","last_became_active":"2024-02-19T21:01:16.276807+0000","last_became_peered":"2024-02-19T21:01:16.276807+0000","last_unstale":"2024-02-19T21:04:17.742813+0000","last_undegraded":"2024-02-19T21:04:17.742813+0000","last_fullsized":"2024-02-19T21:04:17.742813+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:14.215016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:14.215016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743910+0000","last_change":"2024-02-19T21:03:11.598713+0000","last_active":"2024-02-19T21:04:17.743910+0000","last_peered":"2024-02-19T21:04:17.743910+0000","last_clean":"2024-02-19T21:04:17.743910+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:04:17.743910+0000","last_undegraded":"2024-02-19T21:04:17.743910+0000","last_fullsized":"2024-02-19T21:04:17.743910+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:11.598603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:03:11.598603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-02-19T21:04:19.885862+0000","last_change":"2024-02-19T21:04:19.885862+0000","last_active":"2024-02-19T21:04:19.885862+0000","last_peered":"2024-02-19T21:04:19.885862+0000","last_clean":"2024-02-19T21:04:19.885862+0000","last_became_active":"2024-02-19T21:01:18.293784+0000","last_became_peered":"2024-02-19T21:01:18.293784+0000","last_unstale":"2024-02-19T21:04:19.885862+0000","last_undegraded":"2024-02-19T21:04:19.885862+0000","last_fullsized":"2024-02-19T21:04:19.885862+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:04:19.885797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.885797+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":52,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-02-19T21:04:19.858793+0000","last_change":"2024-02-19T21:04:19.858793+0000","last_active":"2024-02-19T21:04:19.858793+0000","last_peered":"2024-02-19T21:04:19.858793+0000","last_clean":"2024-02-19T21:04:19.858793+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:04:19.858793+0000","last_undegraded":"2024-02-19T21:04:19.858793+0000","last_fullsized":"2024-02-19T21:04:19.858793+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:19.858688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.858688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:14.311845+0000","last_change":"2024-02-19T21:02:52.556438+0000","last_active":"2024-02-19T21:04:14.311845+0000","last_peered":"2024-02-19T21:04:14.311845+0000","last_clean":"2024-02-19T21:04:14.311845+0000","last_became_active":"2024-02-19T21:01:22.298477+0000","last_became_peered":"2024-02-19T21:01:22.298477+0000","last_unstale":"2024-02-19T21:04:14.311845+0000","last_undegraded":"2024-02-19T21:04:14.311845+0000","last_fullsized":"2024-02-19T21:04:14.311845+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:52.556344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.556344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:13.983988+0000","last_change":"2024-02-19T21:02:57.505093+0000","last_active":"2024-02-19T21:04:13.983988+0000","last_peered":"2024-02-19T21:04:13.983988+0000","last_clean":"2024-02-19T21:04:13.983988+0000","last_became_active":"2024-02-19T21:01:16.274398+0000","last_became_peered":"2024-02-19T21:01:16.274398+0000","last_unstale":"2024-02-19T21:04:13.983988+0000","last_undegraded":"2024-02-19T21:04:13.983988+0000","last_fullsized":"2024-02-19T21:04:13.983988+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.505029+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":58,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743893+0000","last_change":"2024-02-19T21:02:41.616831+0000","last_active":"2024-02-19T21:04:17.743893+0000","last_peered":"2024-02-19T21:04:17.743893+0000","last_clean":"2024-02-19T21:04:17.743893+0000","last_became_active":"2024-02-19T21:01:16.275904+0000","last_became_peered":"2024-02-19T21:01:16.275904+0000","last_unstale":"2024-02-19T21:04:17.743893+0000","last_undegraded":"2024-02-19T21:04:17.743893+0000","last_fullsized":"2024-02-19T21:04:17.743893+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:02:41.616729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.616729+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:13.984368+0000","last_change":"2024-02-19T21:03:22.495828+0000","last_active":"2024-02-19T21:04:13.984368+0000","last_peered":"2024-02-19T21:04:13.984368+0000","last_clean":"2024-02-19T21:04:13.984368+0000","last_became_active":"2024-02-19T21:01:18.287139+0000","last_became_peered":"2024-02-19T21:01:18.287139+0000","last_unstale":"2024-02-19T21:04:13.984368+0000","last_undegraded":"2024-02-19T21:04:13.984368+0000","last_fullsized":"2024-02-19T21:04:13.984368+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:03:22.495754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:22.495754+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743902+0000","last_change":"2024-02-19T21:03:05.005360+0000","last_active":"2024-02-19T21:04:17.743902+0000","last_peered":"2024-02-19T21:04:17.743902+0000","last_clean":"2024-02-19T21:04:17.743902+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:04:17.743902+0000","last_undegraded":"2024-02-19T21:04:17.743902+0000","last_fullsized":"2024-02-19T21:04:17.743902+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.005288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.005288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:14.312225+0000","last_change":"2024-02-19T21:03:07.473156+0000","last_active":"2024-02-19T21:04:14.312225+0000","last_peered":"2024-02-19T21:04:14.312225+0000","last_clean":"2024-02-19T21:04:14.312225+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:04:14.312225+0000","last_undegraded":"2024-02-19T21:04:14.312225+0000","last_fullsized":"2024-02-19T21:04:14.312225+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:03:07.473070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":51,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.744239+0000","last_change":"2024-02-19T21:02:46.565447+0000","last_active":"2024-02-19T21:04:17.744239+0000","last_peered":"2024-02-19T21:04:17.744239+0000","last_clean":"2024-02-19T21:04:17.744239+0000","last_became_active":"2024-02-19T21:01:16.274215+0000","last_became_peered":"2024-02-19T21:01:16.274215+0000","last_unstale":"2024-02-19T21:04:17.744239+0000","last_undegraded":"2024-02-19T21:04:17.744239+0000","last_fullsized":"2024-02-19T21:04:17.744239+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:46.565380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.565380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":90,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:13.984108+0000","last_change":"2024-02-19T21:04:07.592349+0000","last_active":"2024-02-19T21:04:13.984108+0000","last_peered":"2024-02-19T21:04:13.984108+0000","last_clean":"2024-02-19T21:04:13.984108+0000","last_became_active":"2024-02-19T21:01:18.293791+0000","last_became_peered":"2024-02-19T21:01:18.293791+0000","last_unstale":"2024-02-19T21:04:13.984108+0000","last_undegraded":"2024-02-19T21:04:13.984108+0000","last_fullsized":"2024-02-19T21:04:13.984108+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:04:07.592209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:07.592209+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.744304+0000","last_change":"2024-02-19T21:03:36.004677+0000","last_active":"2024-02-19T21:04:17.744304+0000","last_peered":"2024-02-19T21:04:17.744304+0000","last_clean":"2024-02-19T21:04:17.744304+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:04:17.744304+0000","last_undegraded":"2024-02-19T21:04:17.744304+0000","last_fullsized":"2024-02-19T21:04:17.744304+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:36.004580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:36.004580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693495+0000","last_change":"2024-02-19T21:03:17.088777+0000","last_active":"2024-02-19T21:04:17.693495+0000","last_peered":"2024-02-19T21:04:17.693495+0000","last_clean":"2024-02-19T21:04:17.693495+0000","last_became_active":"2024-02-19T21:01:22.297707+0000","last_became_peered":"2024-02-19T21:01:22.297707+0000","last_unstale":"2024-02-19T21:04:17.693495+0000","last_undegraded":"2024-02-19T21:04:17.693495+0000","last_fullsized":"2024-02-19T21:04:17.693495+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:17.088695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:17.088695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":51,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:14.311080+0000","last_change":"2024-02-19T21:02:47.601190+0000","last_active":"2024-02-19T21:04:14.311080+0000","last_peered":"2024-02-19T21:04:14.311080+0000","last_clean":"2024-02-19T21:04:14.311080+0000","last_became_active":"2024-02-19T21:01:16.271340+0000","last_became_peered":"2024-02-19T21:01:16.271340+0000","last_unstale":"2024-02-19T21:04:14.311080+0000","last_undegraded":"2024-02-19T21:04:14.311080+0000","last_fullsized":"2024-02-19T21:04:14.311080+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:47.601127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:47.601127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":87,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.695542+0000","last_change":"2024-02-19T21:04:08.837761+0000","last_active":"2024-02-19T21:04:17.695542+0000","last_peered":"2024-02-19T21:04:17.695542+0000","last_clean":"2024-02-19T21:04:17.695542+0000","last_became_active":"2024-02-19T21:01:18.293812+0000","last_became_peered":"2024-02-19T21:01:18.293812+0000","last_unstale":"2024-02-19T21:04:17.695542+0000","last_undegraded":"2024-02-19T21:04:17.695542+0000","last_fullsized":"2024-02-19T21:04:17.695542+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:04:08.837652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.837652+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693136+0000","last_change":"2024-02-19T21:02:57.997535+0000","last_active":"2024-02-19T21:04:17.693136+0000","last_peered":"2024-02-19T21:04:17.693136+0000","last_clean":"2024-02-19T21:04:17.693136+0000","last_became_active":"2024-02-19T21:01:20.291079+0000","last_became_peered":"2024-02-19T21:01:20.291079+0000","last_unstale":"2024-02-19T21:04:17.693136+0000","last_undegraded":"2024-02-19T21:04:17.693136+0000","last_fullsized":"2024-02-19T21:04:17.693136+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.997427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743596+0000","last_change":"2024-02-19T21:03:01.496926+0000","last_active":"2024-02-19T21:04:17.743596+0000","last_peered":"2024-02-19T21:04:17.743596+0000","last_clean":"2024-02-19T21:04:17.743596+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:04:17.743596+0000","last_undegraded":"2024-02-19T21:04:17.743596+0000","last_fullsized":"2024-02-19T21:04:17.743596+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.496833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.496833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.744709+0000","last_change":"2024-02-19T21:02:50.921031+0000","last_active":"2024-02-19T21:04:17.744709+0000","last_peered":"2024-02-19T21:04:17.744709+0000","last_clean":"2024-02-19T21:04:17.744709+0000","last_became_active":"2024-02-19T21:01:16.274462+0000","last_became_peered":"2024-02-19T21:01:16.274462+0000","last_unstale":"2024-02-19T21:04:17.744709+0000","last_undegraded":"2024-02-19T21:04:17.744709+0000","last_fullsized":"2024-02-19T21:04:17.744709+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:50.920948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:50.920948+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":71,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693015+0000","last_change":"2024-02-19T21:02:40.105227+0000","last_active":"2024-02-19T21:04:17.693015+0000","last_peered":"2024-02-19T21:04:17.693015+0000","last_clean":"2024-02-19T21:04:17.693015+0000","last_became_active":"2024-02-19T21:01:18.282702+0000","last_became_peered":"2024-02-19T21:01:18.282702+0000","last_unstale":"2024-02-19T21:04:17.693015+0000","last_undegraded":"2024-02-19T21:04:17.693015+0000","last_fullsized":"2024-02-19T21:04:17.693015+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:02:40.105141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.105141+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":98,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:19.454901+0000","last_change":"2024-02-19T21:03:33.736454+0000","last_active":"2024-02-19T21:04:19.454901+0000","last_peered":"2024-02-19T21:04:19.454901+0000","last_clean":"2024-02-19T21:04:19.454901+0000","last_became_active":"2024-02-19T21:01:20.286505+0000","last_became_peered":"2024-02-19T21:01:20.286505+0000","last_unstale":"2024-02-19T21:04:19.454901+0000","last_undegraded":"2024-02-19T21:04:19.454901+0000","last_fullsized":"2024-02-19T21:04:19.454901+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:33.736366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:33.736366+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.696205+0000","last_change":"2024-02-19T21:03:23.656785+0000","last_active":"2024-02-19T21:04:17.696205+0000","last_peered":"2024-02-19T21:04:17.696205+0000","last_clean":"2024-02-19T21:04:17.696205+0000","last_became_active":"2024-02-19T21:01:22.607007+0000","last_became_peered":"2024-02-19T21:01:22.607007+0000","last_unstale":"2024-02-19T21:04:17.696205+0000","last_undegraded":"2024-02-19T21:04:17.696205+0000","last_fullsized":"2024-02-19T21:04:17.696205+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:23.656688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:23.656688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743764+0000","last_change":"2024-02-19T21:04:16.217810+0000","last_active":"2024-02-19T21:04:17.743764+0000","last_peered":"2024-02-19T21:04:17.743764+0000","last_clean":"2024-02-19T21:04:17.743764+0000","last_became_active":"2024-02-19T21:01:20.289375+0000","last_became_peered":"2024-02-19T21:01:20.289375+0000","last_unstale":"2024-02-19T21:04:17.743764+0000","last_undegraded":"2024-02-19T21:04:17.743764+0000","last_fullsized":"2024-02-19T21:04:17.743764+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:16.217744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.217744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":56,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743978+0000","last_change":"2024-02-19T21:04:08.858046+0000","last_active":"2024-02-19T21:04:17.743978+0000","last_peered":"2024-02-19T21:04:17.743978+0000","last_clean":"2024-02-19T21:04:17.743978+0000","last_became_active":"2024-02-19T21:01:16.275520+0000","last_became_peered":"2024-02-19T21:01:16.275520+0000","last_unstale":"2024-02-19T21:04:17.743978+0000","last_undegraded":"2024-02-19T21:04:17.743978+0000","last_fullsized":"2024-02-19T21:04:17.743978+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.857973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.857973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":74,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693783+0000","last_change":"2024-02-19T21:03:35.086667+0000","last_active":"2024-02-19T21:04:17.693783+0000","last_peered":"2024-02-19T21:04:17.693783+0000","last_clean":"2024-02-19T21:04:17.693783+0000","last_became_active":"2024-02-19T21:01:18.285670+0000","last_became_peered":"2024-02-19T21:01:18.285670+0000","last_unstale":"2024-02-19T21:04:17.693783+0000","last_undegraded":"2024-02-19T21:04:17.693783+0000","last_fullsized":"2024-02-19T21:04:17.693783+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:03:35.086579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:35.086579+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":51,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743384+0000","last_change":"2024-02-19T21:04:08.185818+0000","last_active":"2024-02-19T21:04:17.743384+0000","last_peered":"2024-02-19T21:04:17.743384+0000","last_clean":"2024-02-19T21:04:17.743384+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:04:17.743384+0000","last_undegraded":"2024-02-19T21:04:17.743384+0000","last_fullsized":"2024-02-19T21:04:17.743384+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.185717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.185717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":98,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:19.455297+0000","last_change":"2024-02-19T21:04:16.059715+0000","last_active":"2024-02-19T21:04:19.455297+0000","last_peered":"2024-02-19T21:04:19.455297+0000","last_clean":"2024-02-19T21:04:19.455297+0000","last_became_active":"2024-02-19T21:01:20.293341+0000","last_became_peered":"2024-02-19T21:01:20.293341+0000","last_unstale":"2024-02-19T21:04:19.455297+0000","last_undegraded":"2024-02-19T21:04:19.455297+0000","last_fullsized":"2024-02-19T21:04:19.455297+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:16.059648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.059648+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:14.312430+0000","last_change":"2024-02-19T21:04:12.305976+0000","last_active":"2024-02-19T21:04:14.312430+0000","last_peered":"2024-02-19T21:04:14.312430+0000","last_clean":"2024-02-19T21:04:14.312430+0000","last_became_active":"2024-02-19T21:01:16.274939+0000","last_became_peered":"2024-02-19T21:01:16.274939+0000","last_unstale":"2024-02-19T21:04:14.312430+0000","last_undegraded":"2024-02-19T21:04:14.312430+0000","last_fullsized":"2024-02-19T21:04:14.312430+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:12.305886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:12.305886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":74,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:14.311173+0000","last_change":"2024-02-19T21:02:53.526211+0000","last_active":"2024-02-19T21:04:14.311173+0000","last_peered":"2024-02-19T21:04:14.311173+0000","last_clean":"2024-02-19T21:04:14.311173+0000","last_became_active":"2024-02-19T21:01:18.285534+0000","last_became_peered":"2024-02-19T21:01:18.285534+0000","last_unstale":"2024-02-19T21:04:14.311173+0000","last_undegraded":"2024-02-19T21:04:14.311173+0000","last_fullsized":"2024-02-19T21:04:14.311173+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:53.526141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:53.526141+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693576+0000","last_change":"2024-02-19T21:02:42.123367+0000","last_active":"2024-02-19T21:04:17.693576+0000","last_peered":"2024-02-19T21:04:17.693576+0000","last_clean":"2024-02-19T21:04:17.693576+0000","last_became_active":"2024-02-19T21:01:22.299993+0000","last_became_peered":"2024-02-19T21:01:22.299993+0000","last_unstale":"2024-02-19T21:04:17.693576+0000","last_undegraded":"2024-02-19T21:04:17.693576+0000","last_fullsized":"2024-02-19T21:04:17.693576+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:42.123299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:42.123299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":98,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:19.455508+0000","last_change":"2024-02-19T21:04:07.809065+0000","last_active":"2024-02-19T21:04:19.455508+0000","last_peered":"2024-02-19T21:04:19.455508+0000","last_clean":"2024-02-19T21:04:19.455508+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:04:19.455508+0000","last_undegraded":"2024-02-19T21:04:19.455508+0000","last_fullsized":"2024-02-19T21:04:19.455508+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:07.808944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:07.808944+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":56,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:14.311039+0000","last_change":"2024-02-19T21:04:11.303670+0000","last_active":"2024-02-19T21:04:14.311039+0000","last_peered":"2024-02-19T21:04:14.311039+0000","last_clean":"2024-02-19T21:04:14.311039+0000","last_became_active":"2024-02-19T21:01:16.274086+0000","last_became_peered":"2024-02-19T21:01:16.274086+0000","last_unstale":"2024-02-19T21:04:14.311039+0000","last_undegraded":"2024-02-19T21:04:14.311039+0000","last_fullsized":"2024-02-19T21:04:14.311039+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.303572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.303572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"82'15","reported_seq":70,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693288+0000","last_change":"2024-02-19T21:03:04.086368+0000","last_active":"2024-02-19T21:04:17.693288+0000","last_peered":"2024-02-19T21:04:17.693288+0000","last_clean":"2024-02-19T21:04:17.693288+0000","last_became_active":"2024-02-19T21:01:18.282805+0000","last_became_peered":"2024-02-19T21:01:18.282805+0000","last_unstale":"2024-02-19T21:04:17.693288+0000","last_undegraded":"2024-02-19T21:04:17.693288+0000","last_fullsized":"2024-02-19T21:04:17.693288+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.086282+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":51,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-02-19T21:04:19.845711+0000","last_change":"2024-02-19T21:04:19.845711+0000","last_active":"2024-02-19T21:04:19.845711+0000","last_peered":"2024-02-19T21:04:19.845711+0000","last_clean":"2024-02-19T21:04:19.845711+0000","last_became_active":"2024-02-19T21:01:22.298941+0000","last_became_peered":"2024-02-19T21:01:22.298941+0000","last_unstale":"2024-02-19T21:04:19.845711+0000","last_undegraded":"2024-02-19T21:04:19.845711+0000","last_fullsized":"2024-02-19T21:04:19.845711+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.845609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:14.311918+0000","last_change":"2024-02-19T21:04:04.386811+0000","last_active":"2024-02-19T21:04:14.311918+0000","last_peered":"2024-02-19T21:04:14.311918+0000","last_clean":"2024-02-19T21:04:14.311918+0000","last_became_active":"2024-02-19T21:01:20.290687+0000","last_became_peered":"2024-02-19T21:01:20.290687+0000","last_unstale":"2024-02-19T21:04:14.311918+0000","last_undegraded":"2024-02-19T21:04:14.311918+0000","last_fullsized":"2024-02-19T21:04:14.311918+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:04.386735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:04.386735+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743329+0000","last_change":"2024-02-19T21:02:40.628725+0000","last_active":"2024-02-19T21:04:17.743329+0000","last_peered":"2024-02-19T21:04:17.743329+0000","last_clean":"2024-02-19T21:04:17.743329+0000","last_became_active":"2024-02-19T21:01:16.274937+0000","last_became_peered":"2024-02-19T21:01:16.274937+0000","last_unstale":"2024-02-19T21:04:17.743329+0000","last_undegraded":"2024-02-19T21:04:17.743329+0000","last_fullsized":"2024-02-19T21:04:17.743329+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:02:40.628632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.628632+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":110,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.693850+0000","last_change":"2024-02-19T21:04:03.152903+0000","last_active":"2024-02-19T21:04:17.693850+0000","last_peered":"2024-02-19T21:04:17.693850+0000","last_clean":"2024-02-19T21:04:17.693850+0000","last_became_active":"2024-02-19T21:01:30.923214+0000","last_became_peered":"2024-02-19T21:01:30.923214+0000","last_unstale":"2024-02-19T21:04:17.693850+0000","last_undegraded":"2024-02-19T21:04:17.693850+0000","last_fullsized":"2024-02-19T21:04:17.693850+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:04:03.152817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:03.152817+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":45,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.743763+0000","last_change":"2024-02-19T21:03:03.500049+0000","last_active":"2024-02-19T21:04:17.743763+0000","last_peered":"2024-02-19T21:04:17.743763+0000","last_clean":"2024-02-19T21:04:17.743763+0000","last_became_active":"2024-02-19T21:01:22.291759+0000","last_became_peered":"2024-02-19T21:01:22.291759+0000","last_unstale":"2024-02-19T21:04:17.743763+0000","last_undegraded":"2024-02-19T21:04:17.743763+0000","last_fullsized":"2024-02-19T21:04:17.743763+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:03.499960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:03.499960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":48,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.694106+0000","last_change":"2024-02-19T21:02:55.213761+0000","last_active":"2024-02-19T21:04:17.694106+0000","last_peered":"2024-02-19T21:04:17.694106+0000","last_clean":"2024-02-19T21:04:17.694106+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:04:17.694106+0000","last_undegraded":"2024-02-19T21:04:17.694106+0000","last_fullsized":"2024-02-19T21:04:17.694106+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:55.213692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.213692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.694623+0000","last_change":"2024-02-19T21:04:03.767826+0000","last_active":"2024-02-19T21:04:17.694623+0000","last_peered":"2024-02-19T21:04:17.694623+0000","last_clean":"2024-02-19T21:04:17.694623+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:04:17.694623+0000","last_undegraded":"2024-02-19T21:04:17.694623+0000","last_fullsized":"2024-02-19T21:04:17.694623+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:03.767713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:03.767713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":81,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.744216+0000","last_change":"2024-02-19T21:02:39.779974+0000","last_active":"2024-02-19T21:04:17.744216+0000","last_peered":"2024-02-19T21:04:17.744216+0000","last_clean":"2024-02-19T21:04:17.744216+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:04:17.744216+0000","last_undegraded":"2024-02-19T21:04:17.744216+0000","last_fullsized":"2024-02-19T21:04:17.744216+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:02:39.779890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:39.779890+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:13.983728+0000","last_change":"2024-02-19T21:02:48.572976+0000","last_active":"2024-02-19T21:04:13.983728+0000","last_peered":"2024-02-19T21:04:13.983728+0000","last_clean":"2024-02-19T21:04:13.983728+0000","last_became_active":"2024-02-19T21:01:22.297998+0000","last_became_peered":"2024-02-19T21:01:22.297998+0000","last_unstale":"2024-02-19T21:04:13.983728+0000","last_undegraded":"2024-02-19T21:04:13.983728+0000","last_fullsized":"2024-02-19T21:04:13.983728+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:48.572877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:02:48.572877+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":53,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.831206+0000","last_change":"2024-02-19T21:04:17.831164+0000","last_active":"2024-02-19T21:04:17.831206+0000","last_peered":"2024-02-19T21:04:17.831206+0000","last_clean":"2024-02-19T21:04:17.831206+0000","last_became_active":"2024-02-19T21:01:20.290566+0000","last_became_peered":"2024-02-19T21:01:20.290566+0000","last_unstale":"2024-02-19T21:04:17.831206+0000","last_undegraded":"2024-02-19T21:04:17.831206+0000","last_fullsized":"2024-02-19T21:04:17.831206+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:17.831065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:17.831065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:14.313650+0000","last_change":"2024-02-19T21:04:12.927016+0000","last_active":"2024-02-19T21:04:14.313650+0000","last_peered":"2024-02-19T21:04:14.313650+0000","last_clean":"2024-02-19T21:04:14.313650+0000","last_became_active":"2024-02-19T21:04:12.926645+0000","last_became_peered":"2024-02-19T21:04:12.926645+0000","last_unstale":"2024-02-19T21:04:14.313650+0000","last_undegraded":"2024-02-19T21:04:14.313650+0000","last_fullsized":"2024-02-19T21:04:14.313650+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:30.462868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:30.462868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-02-19T21:04:13.984270+0000","last_change":"2024-02-19T21:02:52.643899+0000","last_active":"2024-02-19T21:04:13.984270+0000","last_peered":"2024-02-19T21:04:13.984270+0000","last_clean":"2024-02-19T21:04:13.984270+0000","last_became_active":"2024-02-19T21:01:18.288322+0000","last_became_peered":"2024-02-19T21:01:18.288322+0000","last_unstale":"2024-02-19T21:04:13.984270+0000","last_undegraded":"2024-02-19T21:04:13.984270+0000","last_fullsized":"2024-02-19T21:04:13.984270+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:52.643835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.643835+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":46,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.742443+0000","last_change":"2024-02-19T21:03:10.017843+0000","last_active":"2024-02-19T21:04:17.742443+0000","last_peered":"2024-02-19T21:04:17.742443+0000","last_clean":"2024-02-19T21:04:17.742443+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:04:17.742443+0000","last_undegraded":"2024-02-19T21:04:17.742443+0000","last_fullsized":"2024-02-19T21:04:17.742443+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:10.017764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:10.017764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":57,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.694793+0000","last_change":"2024-02-19T21:04:11.889869+0000","last_active":"2024-02-19T21:04:17.694793+0000","last_peered":"2024-02-19T21:04:17.694793+0000","last_clean":"2024-02-19T21:04:17.694793+0000","last_became_active":"2024-02-19T21:01:16.271911+0000","last_became_peered":"2024-02-19T21:01:16.271911+0000","last_unstale":"2024-02-19T21:04:17.694793+0000","last_undegraded":"2024-02-19T21:04:17.694793+0000","last_fullsized":"2024-02-19T21:04:17.694793+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.889778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:37.258580+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.889778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":75,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:17.742315+0000","last_change":"2024-02-19T21:03:01.005530+0000","last_active":"2024-02-19T21:04:17.742315+0000","last_peered":"2024-02-19T21:04:17.742315+0000","last_clean":"2024-02-19T21:04:17.742315+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:04:17.742315+0000","last_undegraded":"2024-02-19T21:04:17.742315+0000","last_fullsized":"2024-02-19T21:04:17.742315+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:03:01.005445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.005445+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"65'8","reported_seq":92,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-02-19T21:04:19.455284+0000","last_change":"2024-02-19T21:02:44.563371+0000","last_active":"2024-02-19T21:04:19.455284+0000","last_peered":"2024-02-19T21:04:19.455284+0000","last_clean":"2024-02-19T21:04:19.455284+0000","last_became_active":"2024-02-19T21:01:20.288720+0000","last_became_peered":"2024-02-19T21:01:20.288720+0000","last_unstale":"2024-02-19T21:04:19.455284+0000","last_undegraded":"2024-02-19T21:04:19.455284+0000","last_fullsized":"2024-02-19T21:04:19.455284+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:02:44.563256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:44.563256+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":177,"num_read_kb":172,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266727,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297748,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447404,"statfs":{"total":95995035648,"available":95690141696,"internally_reserved":0,"allocated":667648,"data_stored":311911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.49299999999999999,"15min":0.504},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.77400000000000002,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.78200000000000003,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.45000000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50900000000000001,"15min":0.51900000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.67400000000000004,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":3.5190000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51200000000000001,"15min":0.51800000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.40500000000000003}]},{"osd":2,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.49399999999999999,"15min":0.498},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53000000000000003,"15min":0.52400000000000002},"min":{"1min":0.39700000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73999999999999999,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.59399999999999997}]},{"osd":3,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.71199999999999997,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":3.4510000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.504,"15min":0.497},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.55800000000000005}]},{"osd":4,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.33900000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73799999999999999,"5min":1.373,"15min":1.373},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.496,"15min":0.48999999999999999},"min":{"1min":0.32000000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.81100000000000005,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.60799999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.47299999999999998,"15min":0.46500000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64900000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52400000000000002,"15min":0.51600000000000001},"min":{"1min":0.38800000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80800000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.66900000000000004}]},{"osd":6,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.54000000000000004,"15min":0.54100000000000004},"min":{"1min":0.34899999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82299999999999995,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.57899999999999996,"15min":0.57999999999999996},"min":{"1min":0.44500000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83799999999999997,"5min":1.262,"15min":1.262},"last":0.68999999999999995}]}]},{"osd":6,"up_from":45,"seq":193273528365,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297720,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":638976,"data_stored":310375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.495,"15min":0.49399999999999999},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69399999999999995,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.29999999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.37,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.67500000000000004,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.501,"15min":0.5},"min":{"1min":0.34100000000000003,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.754,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.60199999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51200000000000001,"15min":0.51700000000000002},"min":{"1min":0.35299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.49299999999999999,"15min":0.49099999999999999},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.624,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.51900000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.41299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73899999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51400000000000001,"15min":0.5},"min":{"1min":0.41299999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.40400000000000003}]},{"osd":4,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.40999999999999998,"15min":0.38100000000000001},"min":{"1min":0.33900000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.628,"5min":1.363,"15min":1.363},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.42399999999999999,"15min":0.40600000000000003},"min":{"1min":0.311,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66400000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.32100000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.45000000000000001,"15min":0.41799999999999998},"min":{"1min":0.29799999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68000000000000005,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.372},{"interface":"front","average":{"1min":0.505,"5min":0.47599999999999998,"15min":0.45200000000000001},"min":{"1min":0.33600000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69399999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.35999999999999999}]},{"osd":7,"last update":"Mon Feb 19 21:03:16 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.38600000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67400000000000004,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65300000000000002,"5min":1.534,"15min":1.534},"last":0.39600000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297760,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447392,"statfs":{"total":95995035648,"available":95690129408,"internally_reserved":0,"allocated":679936,"data_stored":312317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.36699999999999999,"15min":0.35099999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38200000000000001,"15min":0.36099999999999999},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":1.2529999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:04:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41499999999999998,"15min":0.40999999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29199999999999998,"5min":0.255,"15min":0.255},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":1.0049999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:03:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40799999999999997,"15min":0.42199999999999999},"min":{"1min":0.26400000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.438,"15min":0.45700000000000002},"min":{"1min":0.29099999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.51500000000000001,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.93600000000000005}]},{"osd":4,"last update":"Mon Feb 19 21:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.56599999999999995,"15min":0.56100000000000005},"min":{"1min":0.42599999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57599999999999996,"15min":0.56599999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.90900000000000003}]},{"osd":5,"last update":"Mon Feb 19 21:03:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.58299999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71699999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.42599999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.80100000000000005}]},{"osd":6,"last update":"Mon Feb 19 21:03:49 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.64000000000000001},"min":{"1min":0.41299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.878,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.879}]},{"osd":7,"last update":"Mon Feb 19 21:04:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.60299999999999998,"15min":0.59199999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":1.05},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.61799999999999999,"15min":0.60699999999999998},"min":{"1min":0.441,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.95799999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297748,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447404,"statfs":{"total":95995035648,"available":95690141696,"internally_reserved":0,"allocated":667648,"data_stored":311565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39200000000000002,"15min":0.32200000000000001},"min":{"1min":0.24299999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.90100000000000002,"5min":4.2830000000000004,"15min":4.2830000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.34699999999999998,"15min":0.313},"min":{"1min":0.22700000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:03:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.39600000000000002,"15min":0.378},"min":{"1min":0.222,"5min":0.219,"15min":0.219},"max":{"1min":0.45300000000000001,"5min":3.121,"15min":3.121},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.42499999999999999,"15min":0.38900000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.497,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.439}]},{"osd":3,"last update":"Mon Feb 19 21:03:41 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.46000000000000002,"15min":0.44700000000000001},"min":{"1min":0.26700000000000002,"5min":0.249,"15min":0.249},"max":{"1min":1.028,"5min":4.4470000000000001,"15min":4.4470000000000001},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.47499999999999998,"15min":0.45800000000000002},"min":{"1min":0.28199999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.95299999999999996,"5min":4.3070000000000004,"15min":4.3070000000000004},"last":0.41299999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53400000000000003,"15min":0.52100000000000002},"min":{"1min":0.35399999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.83899999999999997,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55600000000000005,"15min":0.54100000000000004},"min":{"1min":0.40600000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82399999999999995,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.80000000000000004}]},{"osd":5,"last update":"Mon Feb 19 21:03:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.621,"15min":0.63900000000000001},"min":{"1min":0.41799999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.79500000000000004,"5min":1.581,"15min":1.581},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.75,"15min":0.81599999999999995},"min":{"1min":0.435,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.2290000000000001,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.64400000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:03:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.36699999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72799999999999998,"5min":1.381,"15min":1.381},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85399999999999998,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.82699999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:04:16 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.73199999999999998,"15min":0.78600000000000003},"min":{"1min":0.51100000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.875,"5min":4.226,"15min":4.226},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.45300000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.82499999999999996,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.64900000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297748,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447404,"statfs":{"total":95995035648,"available":95690141696,"internally_reserved":0,"allocated":667648,"data_stored":317600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.41399999999999998,"15min":0.371},"min":{"1min":0.23000000000000001,"5min":0.222,"15min":0.222},"max":{"1min":4.6150000000000002,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":4.6150000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.42599999999999999,"15min":0.37},"min":{"1min":0.254,"5min":0.214,"15min":0.214},"max":{"1min":5.3609999999999998,"5min":5.3609999999999998,"15min":5.3609999999999998},"last":1.048}]},{"osd":1,"last update":"Mon Feb 19 21:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.42699999999999999,"15min":0.373},"min":{"1min":0.26500000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.6989999999999998,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":4.6989999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.46100000000000002,"15min":0.41499999999999998},"min":{"1min":0.29899999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.782,"5min":4.782,"15min":4.782},"last":4.782}]},{"osd":3,"last update":"Mon Feb 19 21:03:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.437,"15min":0.45300000000000001},"min":{"1min":0.28599999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.45000000000000001,"15min":0.46999999999999997},"min":{"1min":0.28499999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":4.8399999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.628,"15min":0.61599999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.2880000000000003,"5min":5.2880000000000003,"15min":5.2880000000000003},"last":0.752},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59799999999999998,"15min":0.64000000000000001},"min":{"1min":0.373,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79700000000000004,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.86199999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:03:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.61199999999999999,"15min":0.64700000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85199999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.625,"15min":0.64800000000000002},"min":{"1min":0.45600000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":4.7320000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.623,"15min":0.63},"min":{"1min":0.40000000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61699999999999999,"15min":0.63100000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90000000000000002,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.81299999999999994}]},{"osd":7,"last update":"Mon Feb 19 21:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.65000000000000002,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.758,"5min":4.758,"15min":4.758},"last":4.758},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.67200000000000004,"15min":0.628},"min":{"1min":0.51100000000000001,"5min":0.379,"15min":0.379},"max":{"1min":5.2430000000000003,"5min":5.2430000000000003,"15min":5.2430000000000003},"last":0.72799999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297772,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447380,"statfs":{"total":95995035648,"available":95690117120,"internally_reserved":0,"allocated":692224,"data_stored":317625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:41 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.34799999999999998,"15min":0.35199999999999998},"min":{"1min":0.218,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.47299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.377,"15min":0.39800000000000002},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.50700000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.39300000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:03:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.374,"15min":0.39200000000000002},"min":{"1min":0.23300000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.47399999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.38100000000000001,"15min":0.39500000000000002},"min":{"1min":0.25700000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.45700000000000002,"5min":1.054,"15min":1.054},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Feb 19 21:03:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40400000000000003,"15min":0.41799999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.48399999999999999,"5min":0.625,"15min":0.625},"last":0.375},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.39400000000000002,"15min":0.41399999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.50700000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.439}]},{"osd":4,"last update":"Mon Feb 19 21:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84299999999999997,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.503},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55300000000000005,"15min":0.56499999999999995},"min":{"1min":0.41999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82499999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.51800000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:03:24 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.55100000000000005,"15min":0.56000000000000005},"min":{"1min":0.38500000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Feb 19 21:03:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55700000000000005,"15min":0.56699999999999995},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.69999999999999996,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58399999999999996,"15min":0.59199999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.57899999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56899999999999995,"15min":0.56299999999999994},"min":{"1min":0.43099999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57999999999999996,"15min":0.57499999999999996},"min":{"1min":0.43099999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.55800000000000005}]}]},{"osd":4,"up_from":31,"seq":133143986231,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297836,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447316,"statfs":{"total":95995035648,"available":95690051584,"internally_reserved":0,"allocated":667648,"data_stored":311191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:04 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51300000000000001,"15min":0.54400000000000004},"min":{"1min":0.33700000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.505,"15min":0.52200000000000002},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.55300000000000005}]},{"osd":1,"last update":"Mon Feb 19 21:04:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.40999999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.78900000000000003,"5min":2.149,"15min":2.149},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.499,"15min":0.51800000000000002},"min":{"1min":0.35899999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Mon Feb 19 21:04:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52800000000000002},"min":{"1min":0.38600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.55400000000000005},"min":{"1min":0.36599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70699999999999996,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.51600000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:04:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.50900000000000001,"15min":0.52300000000000002},"min":{"1min":0.33800000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63700000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56499999999999995,"15min":0.56999999999999995},"min":{"1min":0.40600000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68300000000000005,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":0.53200000000000003}]},{"osd":5,"last update":"Mon Feb 19 21:03:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.42399999999999999,"15min":0.42999999999999999},"min":{"1min":0.30099999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.68100000000000005,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":0.44},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.438,"15min":0.434},"min":{"1min":0.35299999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.66100000000000003,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.48799999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:03:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.41399999999999998,"15min":0.38200000000000001},"min":{"1min":0.38100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.433},"min":{"1min":0.45400000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59699999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.46800000000000003,"15min":0.44},"min":{"1min":0.38600000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.49299999999999999,"15min":0.46700000000000003},"min":{"1min":0.41399999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.622}]}]},{"osd":5,"up_from":37,"seq":158913790002,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297728,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":93447424,"statfs":{"total":95995035648,"available":95690162176,"internally_reserved":0,"allocated":647168,"data_stored":309961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.495,"15min":0.51000000000000001},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.65200000000000002,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.438},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.51500000000000001,"15min":0.53700000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.66900000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.46899999999999997}]},{"osd":1,"last update":"Mon Feb 19 21:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.505,"15min":0.52000000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.57899999999999996,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.51500000000000001,"15min":0.52900000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.68899999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":2.9430000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.51700000000000002,"15min":0.53500000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.60299999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51900000000000002,"15min":0.53100000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69099999999999995,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.50700000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51800000000000002,"15min":0.53200000000000003},"min":{"1min":0.37,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.63500000000000001,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.5},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66100000000000003,"5min":0.747,"15min":0.747},"last":0.92800000000000005}]},{"osd":4,"last update":"Mon Feb 19 21:03:25 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40300000000000002,"15min":0.39000000000000001},"min":{"1min":0.28000000000000003,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.58999999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.41299999999999998,"15min":0.39000000000000001},"min":{"1min":0.34100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.38800000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:03:47 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.442,"15min":0.42299999999999999},"min":{"1min":0.33500000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.45600000000000002,"15min":0.437},"min":{"1min":0.33400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.623,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.42999999999999999}]},{"osd":7,"last update":"Mon Feb 19 21:03:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.503,"15min":0.502},"min":{"1min":0.35699999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.63100000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.50800000000000001,"15min":0.505},"min":{"1min":0.36599999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.66000000000000003,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.45500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:04:29.416 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool set iscsi pgp_num 3 2024-02-19T21:04:29.498 INFO:tasks.workunit.client.0.smithi163.stderr:Updating files: 96% (11646/12088) Updating files: 97% (11726/12088) Updating files: 98% (11847/12088) Updating files: 99% (11968/12088) Updating files: 100% (12088/12088) Updating files: 100% (12088/12088), done. 2024-02-19T21:04:30.135 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:04:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:30.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:04:30.239 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:30.240 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:28.954668+0000 mgr.y (mgr.14550) 131 : cluster [DBG] pgmap v113: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:04:30.240 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:28.954668+0000 mgr.y (mgr.14550) 131 : cluster [DBG] pgmap v113: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:04:30.270 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:28.954668+0000 mgr.y (mgr.14550) 131 : cluster [DBG] pgmap v113: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:04:30.270 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:04:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:04:30.273 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:30.296 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:30.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:30.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:30.400 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:30.430 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:30.447 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:04:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-02-19T21:04:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-02-19T21:04:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-02-19T21:04:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.34352.0:88 4.2 4:4347d321:::bucket.sync-source-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected e60)", 2024-02-19T21:04:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:18.282685+0000", 2024-02-19T21:04:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 192.17068355699999, 2024-02-19T21:04:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059293260000000004, 2024-02-19T21:04:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:04:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.34352", 2024-02-19T21:04:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:04:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 88 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:18.282685+0000", 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:18.282685+0000", 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:18.282682+0000", 2024-02-19T21:04:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1722000000000001e-05 2024-02-19T21:04:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:18.282694+0000", 2024-02-19T21:04:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5998000000000001e-05 2024-02-19T21:04:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:18.282720+0000", 2024-02-19T21:04:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9259999999999995e-06 2024-02-19T21:04:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:18.282730+0000", 2024-02-19T21:04:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0058088209999999996 2024-02-19T21:04:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:18.288539+0000", 2024-02-19T21:04:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5637999999999996e-05 2024-02-19T21:04:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:18.288614+0000", 2024-02-19T21:04:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059293260000000004 2024-02-19T21:04:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.34382.0:57 5.15 e62/61 5:a93a5511:::notify.2:head v 62'1, mlcod=0'0)", 2024-02-19T21:04:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:20.298016+0000", 2024-02-19T21:04:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 190.15535243400001, 2024-02-19T21:04:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027503549999999999, 2024-02-19T21:04:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.298016+0000", 2024-02-19T21:04:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.298016+0000", 2024-02-19T21:04:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.298015+0000", 2024-02-19T21:04:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0511e-05 2024-02-19T21:04:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.298025+0000", 2024-02-19T21:04:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5102999999999999e-05 2024-02-19T21:04:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.298040+0000", 2024-02-19T21:04:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1280000000000008e-06 2024-02-19T21:04:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.298048+0000", 2024-02-19T21:04:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0217999999999999e-05 2024-02-19T21:04:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.298099+0000", 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6841e-05 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.298115+0000", 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002610205 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.300726+0000", 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1010000000000002e-05 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:20.300767+0000", 2024-02-19T21:04:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027503549999999999 2024-02-19T21:04:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:16 4.4 4:227f63b5:gc::gc.14:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.309367+0000", 2024-02-19T21:04:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 187.14400181400001, 2024-02-19T21:04:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0022811699999999999, 2024-02-19T21:04:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 16 2024-02-19T21:04:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.309367+0000", 2024-02-19T21:04:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.309367+0000", 2024-02-19T21:04:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999962 2024-02-19T21:04:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.309363+0000", 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4864e-05 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.309378+0000", 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1733999999999999e-05 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.309400+0000", 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6299999999999993e-06 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.309409+0000", 2024-02-19T21:04:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7166000000000002e-05 2024-02-19T21:04:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.309457+0000", 2024-02-19T21:04:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000109162 2024-02-19T21:04:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.309566+0000", 2024-02-19T21:04:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00021239300000000001 2024-02-19T21:04:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,3", 2024-02-19T21:04:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.309778+0000", 2024-02-19T21:04:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00045009599999999998 2024-02-19T21:04:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.310228+0000", 2024-02-19T21:04:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00027521400000000003 2024-02-19T21:04:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.310504+0000", 2024-02-19T21:04:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010746690000000001 2024-02-19T21:04:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.311578+0000", 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8980000000000003e-05 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.311617+0000", 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0858000000000002e-05 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.311648+0000", 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0022811699999999999 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:45 4.f 4:f1983ba0:::obj_delete_at_hint.0000000001:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.337478+0000", 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 187.115890216, 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019032070000000001, 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 45 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.337478+0000", 2024-02-19T21:04:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.337478+0000", 2024-02-19T21:04:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:04:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.337476+0000", 2024-02-19T21:04:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0298e-05 2024-02-19T21:04:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.337486+0000", 2024-02-19T21:04:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3118999999999999e-05 2024-02-19T21:04:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.337499+0000", 2024-02-19T21:04:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7011999999999999e-05 2024-02-19T21:04:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.337516+0000", 2024-02-19T21:04:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4668e-05 2024-02-19T21:04:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.337561+0000", 2024-02-19T21:04:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000144253 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.337705+0000", 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000223799 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,4", 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.337929+0000", 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061855299999999998 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.338548+0000", 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00022030899999999999 2024-02-19T21:04:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.338768+0000", 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00050178500000000003 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.339270+0000", 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5923e-05 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.339326+0000", 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6125999999999998e-05 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.339382+0000", 2024-02-19T21:04:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019032070000000001 2024-02-19T21:04:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:78 4.4 4:25ca39bb:::obj_delete_at_hint.0000000005:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.352769+0000", 2024-02-19T21:04:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 187.10059981800001, 2024-02-19T21:04:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033028380000000002, 2024-02-19T21:04:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 78 2024-02-19T21:04:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.352769+0000", 2024-02-19T21:04:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.352769+0000", 2024-02-19T21:04:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.352768+0000", 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8580000000000002e-06 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.352774+0000", 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5259999999999996e-06 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.352782+0000", 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7950000000000008e-06 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.352790+0000", 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8624999999999997e-05 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.352828+0000", 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011442699999999999 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.352943+0000", 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000178264 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,3", 2024-02-19T21:04:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.353121+0000", 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00075157399999999997 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.353873+0000", 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00031202399999999999 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354185+0000", 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001795657 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.355980+0000", 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8334000000000002e-05 2024-02-19T21:04:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.356029+0000", 2024-02-19T21:04:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3068000000000002e-05 2024-02-19T21:04:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.356072+0000", 2024-02-19T21:04:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033028380000000002 2024-02-19T21:04:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:85 4.7 4:e47c9322:gc::gc.29:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.354765+0000", 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 187.098603693, 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001723451, 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 85 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354765+0000", 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354765+0000", 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354763+0000", 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6429999999999998e-06 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354770+0000", 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3395e-05 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354784+0000", 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5100000000000004e-06 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354790+0000", 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9739e-05 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354830+0000", 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9339999999999997e-05 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.354899+0000", 2024-02-19T21:04:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000165802 2024-02-19T21:04:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,5", 2024-02-19T21:04:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.355065+0000", 2024-02-19T21:04:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00035444399999999998 2024-02-19T21:04:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.355419+0000", 2024-02-19T21:04:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00081656399999999998 2024-02-19T21:04:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.356236+0000", 2024-02-19T21:04:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015295799999999999 2024-02-19T21:04:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.356389+0000", 2024-02-19T21:04:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2506999999999997e-05 2024-02-19T21:04:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.356441+0000", 2024-02-19T21:04:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7055000000000003e-05 2024-02-19T21:04:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.356488+0000", 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001723451 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:89 4.4 4:25ca39bb:::obj_delete_at_hint.0000000005:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.357279+0000", 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 187.096089353, 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052540659999999999, 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 89 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357279+0000", 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357279+0000", 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357278+0000", 2024-02-19T21:04:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0260000000000007e-06 2024-02-19T21:04:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357286+0000", 2024-02-19T21:04:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2959000000000001e-05 2024-02-19T21:04:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357299+0000", 2024-02-19T21:04:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7799999999999997e-06 2024-02-19T21:04:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357305+0000", 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011208 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357417+0000", 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1769999999999998e-05 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357498+0000", 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000189991 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,3", 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357688+0000", 2024-02-19T21:04:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039970700000000001 2024-02-19T21:04:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.358088+0000", 2024-02-19T21:04:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00065727299999999999 2024-02-19T21:04:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.358745+0000", 2024-02-19T21:04:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0037274840000000001 2024-02-19T21:04:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.362473+0000", 2024-02-19T21:04:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4486000000000002e-05 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.362507+0000", 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6041e-05 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.362533+0000", 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052540659999999999 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:98 4.f 4:f24cb4a8:gc::gc.31:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.361641+0000", 2024-02-19T21:04:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 187.09172741500001, 2024-02-19T21:04:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002807351, 2024-02-19T21:04:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 98 2024-02-19T21:04:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361641+0000", 2024-02-19T21:04:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361641+0000", 2024-02-19T21:04:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:04:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361639+0000", 2024-02-19T21:04:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2598e-05 2024-02-19T21:04:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361651+0000", 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6228000000000001e-05 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361667+0000", 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7540000000000003e-06 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361673+0000", 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4055999999999999e-05 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361727+0000", 2024-02-19T21:04:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8653000000000006e-05 2024-02-19T21:04:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.361826+0000", 2024-02-19T21:04:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019010300000000001 2024-02-19T21:04:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,4", 2024-02-19T21:04:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.362016+0000", 2024-02-19T21:04:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00065497000000000005 2024-02-19T21:04:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.362671+0000", 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016720120000000001 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.364343+0000", 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5018999999999998e-05 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.364378+0000", 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5701000000000002e-05 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.364424+0000", 2024-02-19T21:04:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4890000000000001e-05 2024-02-19T21:04:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.364449+0000", 2024-02-19T21:04:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002807351 2024-02-19T21:04:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:108 4.f 4:f24cb4a8:gc::gc.31:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.366430+0000", 2024-02-19T21:04:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 187.086938506, 2024-02-19T21:04:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001620612, 2024-02-19T21:04:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 108 2024-02-19T21:04:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.366430+0000", 2024-02-19T21:04:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.366430+0000", 2024-02-19T21:04:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.366429+0000", 2024-02-19T21:04:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3369999999999996e-06 2024-02-19T21:04:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.366435+0000", 2024-02-19T21:04:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1646e-05 2024-02-19T21:04:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.366447+0000", 2024-02-19T21:04:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8720000000000001e-06 2024-02-19T21:04:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.366452+0000", 2024-02-19T21:04:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8484999999999999e-05 2024-02-19T21:04:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.366480+0000", 2024-02-19T21:04:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6982000000000003e-05 2024-02-19T21:04:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.366547+0000", 2024-02-19T21:04:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014014299999999999 2024-02-19T21:04:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,4", 2024-02-19T21:04:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.366687+0000", 2024-02-19T21:04:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00064621899999999996 2024-02-19T21:04:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.367333+0000", 2024-02-19T21:04:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4667999999999999e-05 2024-02-19T21:04:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.367348+0000", 2024-02-19T21:04:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00063102999999999998 2024-02-19T21:04:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.367979+0000", 2024-02-19T21:04:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9844e-05 2024-02-19T21:04:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.368019+0000", 2024-02-19T21:04:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1891999999999999e-05 2024-02-19T21:04:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.368051+0000", 2024-02-19T21:04:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001620612 2024-02-19T21:04:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:148 4.4 4:2166d8e1:gc::gc.3:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.381512+0000", 2024-02-19T21:04:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 187.07185653499999, 2024-02-19T21:04:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001706384, 2024-02-19T21:04:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 148 2024-02-19T21:04:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.381512+0000", 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.381512+0000", 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.381511+0000", 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0590000000000003e-06 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.381519+0000", 2024-02-19T21:04:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3089e-05 2024-02-19T21:04:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.381532+0000", 2024-02-19T21:04:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2830000000000004e-06 2024-02-19T21:04:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.381539+0000", 2024-02-19T21:04:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9718000000000001e-05 2024-02-19T21:04:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.381589+0000", 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0224999999999998e-05 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.381669+0000", 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016736100000000001 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,3", 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.381836+0000", 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00075323399999999998 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.382589+0000", 2024-02-19T21:04:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3968999999999999e-05 2024-02-19T21:04:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.382633+0000", 2024-02-19T21:04:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00050735600000000004 2024-02-19T21:04:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.383141+0000", 2024-02-19T21:04:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6992e-05 2024-02-19T21:04:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.383188+0000", 2024-02-19T21:04:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0710000000000002e-05 2024-02-19T21:04:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.383219+0000", 2024-02-19T21:04:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001706384 2024-02-19T21:04:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14487.0:68 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch watch cookie 94661314107488] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.404401+0000", 2024-02-19T21:04:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 187.04896790999999, 2024-02-19T21:04:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015372949999999999, 2024-02-19T21:04:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14487", 2024-02-19T21:04:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/177516907", 2024-02-19T21:04:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 68 2024-02-19T21:04:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.404401+0000", 2024-02-19T21:04:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.404401+0000", 2024-02-19T21:04:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.404399+0000", 2024-02-19T21:04:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7469999999999996e-06 2024-02-19T21:04:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.404404+0000", 2024-02-19T21:04:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7480000000000005e-06 2024-02-19T21:04:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.404414+0000", 2024-02-19T21:04:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1340000000000001e-06 2024-02-19T21:04:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.404420+0000", 2024-02-19T21:04:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1838999999999997e-05 2024-02-19T21:04:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.404462+0000", 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8289999999999998e-05 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.404540+0000", 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000166498 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,3", 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.404707+0000", 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00031816 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.405025+0000", 2024-02-19T21:04:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029175300000000001 2024-02-19T21:04:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.405317+0000", 2024-02-19T21:04:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00055833 2024-02-19T21:04:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.405875+0000", 2024-02-19T21:04:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9036000000000001e-05 2024-02-19T21:04:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.405904+0000", 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.417e-05 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.405938+0000", 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015372949999999999 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.24400.0:283 4.1f e65/59 4:fd8d39b6:::obj_delete_at_hint.0000000030:head v 65'5, mlcod=65'3)", 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.438342+0000", 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 187.01502704500001, 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027857810000000002, 2024-02-19T21:04:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.438342+0000", 2024-02-19T21:04:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.438342+0000", 2024-02-19T21:04:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.438340+0000", 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7710000000000001e-06 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.438347+0000", 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0080000000000008e-06 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.438356+0000", 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7270000000000004e-06 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.438362+0000", 2024-02-19T21:04:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9784999999999999e-05 2024-02-19T21:04:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.438412+0000", 2024-02-19T21:04:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8583000000000001e-05 2024-02-19T21:04:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.438430+0000", 2024-02-19T21:04:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026527040000000001 2024-02-19T21:04:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.441083+0000", 2024-02-19T21:04:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4468999999999997e-05 2024-02-19T21:04:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.441127+0000", 2024-02-19T21:04:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0027857810000000002 2024-02-19T21:04:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.24400.0:290 4.d e65/59 4:b019ea83:gc::gc.18:head v 65'4, mlcod=65'2)", 2024-02-19T21:04:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.442541+0000", 2024-02-19T21:04:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 187.010827839, 2024-02-19T21:04:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019967269999999998, 2024-02-19T21:04:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.442541+0000", 2024-02-19T21:04:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.442541+0000", 2024-02-19T21:04:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:04:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.442540+0000", 2024-02-19T21:04:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8199999999999999e-05 2024-02-19T21:04:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.442558+0000", 2024-02-19T21:04:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6139999999999998e-06 2024-02-19T21:04:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.442567+0000", 2024-02-19T21:04:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.942e-06 2024-02-19T21:04:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.442573+0000", 2024-02-19T21:04:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0516999999999998e-05 2024-02-19T21:04:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.442624+0000", 2024-02-19T21:04:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0338000000000001e-05 2024-02-19T21:04:30.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.442644+0000", 2024-02-19T21:04:30.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001868792 2024-02-19T21:04:30.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.444513+0000", 2024-02-19T21:04:30.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4485e-05 2024-02-19T21:04:30.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.444538+0000", 2024-02-19T21:04:30.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019967269999999998 2024-02-19T21:04:30.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.516 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.516 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.24400.0:291 4.18 e65/59 4:1aba6385:::obj_delete_at_hint.0000000031:head v 65'9, mlcod=65'7)", 2024-02-19T21:04:30.516 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.443296+0000", 2024-02-19T21:04:30.516 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 187.01007297300001, 2024-02-19T21:04:30.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001786263, 2024-02-19T21:04:30.516 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.516 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.516 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.443296+0000", 2024-02-19T21:04:30.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.443296+0000", 2024-02-19T21:04:30.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:04:30.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.443295+0000", 2024-02-19T21:04:30.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0400000000000004e-06 2024-02-19T21:04:30.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.443302+0000", 2024-02-19T21:04:30.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6039999999999999e-06 2024-02-19T21:04:30.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.443311+0000", 2024-02-19T21:04:30.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0860000000000008e-06 2024-02-19T21:04:30.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.443320+0000", 2024-02-19T21:04:30.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6038999999999998e-05 2024-02-19T21:04:30.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.443356+0000", 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7173e-05 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.443373+0000", 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001677011 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.445050+0000", 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1486999999999999e-05 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.445082+0000", 2024-02-19T21:04:30.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001786263 2024-02-19T21:04:30.520 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.520 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.520 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.520 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:298 4.4 4:24c8e1ae:gc::gc.19:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:30.520 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.448050+0000", 2024-02-19T21:04:30.520 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 187.00531842300001, 2024-02-19T21:04:30.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002538766, 2024-02-19T21:04:30.520 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.520 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.520 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:30.521 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:30.521 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:30.521 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 298 2024-02-19T21:04:30.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.521 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.448050+0000", 2024-02-19T21:04:30.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.448050+0000", 2024-02-19T21:04:30.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:04:30.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.448048+0000", 2024-02-19T21:04:30.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5589999999999994e-06 2024-02-19T21:04:30.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.448057+0000", 2024-02-19T21:04:30.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0528e-05 2024-02-19T21:04:30.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.448068+0000", 2024-02-19T21:04:30.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9429999999999999e-06 2024-02-19T21:04:30.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.448073+0000", 2024-02-19T21:04:30.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9691000000000003e-05 2024-02-19T21:04:30.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.448113+0000", 2024-02-19T21:04:30.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6507000000000005e-05 2024-02-19T21:04:30.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.448189+0000", 2024-02-19T21:04:30.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000156625 2024-02-19T21:04:30.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,3", 2024-02-19T21:04:30.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.448346+0000", 2024-02-19T21:04:30.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00089276100000000005 2024-02-19T21:04:30.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.449239+0000", 2024-02-19T21:04:30.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00045478700000000001 2024-02-19T21:04:30.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.449693+0000", 2024-02-19T21:04:30.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00080419400000000005 2024-02-19T21:04:30.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:30.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.450498+0000", 2024-02-19T21:04:30.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.041e-05 2024-02-19T21:04:30.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.450548+0000", 2024-02-19T21:04:30.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1069999999999998e-05 2024-02-19T21:04:30.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.450589+0000", 2024-02-19T21:04:30.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002538766 2024-02-19T21:04:30.526 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.526 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.526 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.526 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.24400.0:301 4.d e65/59 4:b72b1478:::obj_delete_at_hint.0000000033:head v 65'6, mlcod=65'4)", 2024-02-19T21:04:30.526 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.451625+0000", 2024-02-19T21:04:30.526 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 187.001743662, 2024-02-19T21:04:30.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001705315, 2024-02-19T21:04:30.526 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.526 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.527 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.451625+0000", 2024-02-19T21:04:30.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.451625+0000", 2024-02-19T21:04:30.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:30.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.451624+0000", 2024-02-19T21:04:30.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2040000000000002e-06 2024-02-19T21:04:30.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.451630+0000", 2024-02-19T21:04:30.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7190000000000004e-06 2024-02-19T21:04:30.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.451637+0000", 2024-02-19T21:04:30.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9829999999999998e-06 2024-02-19T21:04:30.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.451641+0000", 2024-02-19T21:04:30.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2639999999999997e-05 2024-02-19T21:04:30.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.451693+0000", 2024-02-19T21:04:30.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9496000000000001e-05 2024-02-19T21:04:30.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.451713+0000", 2024-02-19T21:04:30.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015883920000000001 2024-02-19T21:04:30.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.453301+0000", 2024-02-19T21:04:30.530 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9074e-05 2024-02-19T21:04:30.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.530 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.530 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.453330+0000", 2024-02-19T21:04:30.530 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001705315 2024-02-19T21:04:30.530 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.530 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.530 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.530 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24400.0:306 4.7 4:e622f76a:gc::gc.20:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:04:30.530 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.453684+0000", 2024-02-19T21:04:30.530 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 186.999684231, 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002637805, 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24400", 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 306 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.453684+0000", 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.531 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.453684+0000", 2024-02-19T21:04:30.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:04:30.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.453683+0000", 2024-02-19T21:04:30.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.669e-06 2024-02-19T21:04:30.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.453689+0000", 2024-02-19T21:04:30.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3610000000000007e-06 2024-02-19T21:04:30.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.453699+0000", 2024-02-19T21:04:30.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8049999999999997e-06 2024-02-19T21:04:30.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.453704+0000", 2024-02-19T21:04:30.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5447999999999997e-05 2024-02-19T21:04:30.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.453739+0000", 2024-02-19T21:04:30.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4993e-05 2024-02-19T21:04:30.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.453794+0000", 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014785899999999999 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,5", 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.453942+0000", 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00061753199999999996 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454559+0000", 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015151699999999999 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.456075+0000", 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015080599999999999 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.456225+0000", 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5402000000000003e-05 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.456281+0000", 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1449000000000002e-05 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.536 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.456322+0000", 2024-02-19T21:04:30.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002637805 2024-02-19T21:04:30.536 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.536 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.536 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.536 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.24400.0:307 4.d e65/59 4:b72b1478:::obj_delete_at_hint.0000000033:head v 65'7, mlcod=65'5)", 2024-02-19T21:04:30.536 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.454278+0000", 2024-02-19T21:04:30.536 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 186.999090298, 2024-02-19T21:04:30.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019566039999999998, 2024-02-19T21:04:30.536 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.536 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:30.536 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454278+0000", 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454278+0000", 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454277+0000", 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9969999999999993e-06 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454287+0000", 2024-02-19T21:04:30.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1231e-05 2024-02-19T21:04:30.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.538 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454298+0000", 2024-02-19T21:04:30.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1009999999999997e-06 2024-02-19T21:04:30.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.538 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454304+0000", 2024-02-19T21:04:30.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4209000000000002e-05 2024-02-19T21:04:30.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454359+0000", 2024-02-19T21:04:30.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8578999999999999e-05 2024-02-19T21:04:30.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.454377+0000", 2024-02-19T21:04:30.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001834086 2024-02-19T21:04:30.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:30.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.456211+0000", 2024-02-19T21:04:30.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3798000000000002e-05 2024-02-19T21:04:30.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.540 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.456235+0000", 2024-02-19T21:04:30.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019566039999999998 2024-02-19T21:04:30.540 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.540 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.540 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.540 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24493.0:3 3.0 3:377edd40:::rbd_namespace:head [call rbd.namespace_list in=17b] snapc 0=[] ondisk+read+known_if_redirected e72)", 2024-02-19T21:04:30.540 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:39.085713+0000", 2024-02-19T21:04:30.540 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 171.36765528500001, 2024-02-19T21:04:30.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014564490000000001, 2024-02-19T21:04:30.541 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.541 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:04:30.541 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:30.541 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24493", 2024-02-19T21:04:30.541 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2351957517", 2024-02-19T21:04:30.541 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3 2024-02-19T21:04:30.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.541 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.541 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.541 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.085713+0000", 2024-02-19T21:04:30.541 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.541 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.085713+0000", 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.085711+0000", 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2059999999999996e-06 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.085720+0000", 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8210000000000008e-06 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.085729+0000", 2024-02-19T21:04:30.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2129999999999996e-06 2024-02-19T21:04:30.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.085735+0000", 2024-02-19T21:04:30.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1165999999999999e-05 2024-02-19T21:04:30.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.085776+0000", 2024-02-19T21:04:30.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7639999999999996e-06 2024-02-19T21:04:30.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:04:30.544 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.085781+0000", 2024-02-19T21:04:30.544 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012911039999999999 2024-02-19T21:04:30.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.544 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.544 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.087072+0000", 2024-02-19T21:04:30.544 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7757000000000006e-05 2024-02-19T21:04:30.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.544 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.544 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:39.087170+0000", 2024-02-19T21:04:30.544 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014564490000000001 2024-02-19T21:04:30.544 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.544 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.545 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.545 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.545 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDScrubReserve(6.19 REQUEST e95)", 2024-02-19T21:04:30.545 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:19.296499+0000", 2024-02-19T21:04:30.545 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 11.156869422, 2024-02-19T21:04:30.545 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001449704, 2024-02-19T21:04:30.545 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:30.545 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:04:30.545 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:30.545 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.545 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:19.296499+0000", 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:19.296499+0000", 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999962 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:19.296495+0000", 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4194e-05 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:19.296509+0000", 2024-02-19T21:04:30.546 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0460000000000001e-05 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:19.296530+0000", 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.111e-05 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:19.296541+0000", 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013476619999999999 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:19.297889+0000", 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1297999999999998e-05 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.548 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:30.548 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:19.297910+0000", 2024-02-19T21:04:30.548 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8973000000000001e-05 2024-02-19T21:04:30.548 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:30.548 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:30.548 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:30.548 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:19.297949+0000", 2024-02-19T21:04:30.548 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001449704 2024-02-19T21:04:30.548 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.548 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.548 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.548 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:30.548 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:30.548 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:04:30.549 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:30.550 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:30.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:30.550 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:30.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:30.552 INFO:tasks.workunit.client.0.smithi163.stderr:Note: switching to '81bd20d634209c7cb82c18be12b4b5a05643ebf1'. 2024-02-19T21:04:30.552 INFO:tasks.workunit.client.0.smithi163.stderr: 2024-02-19T21:04:30.552 INFO:tasks.workunit.client.0.smithi163.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-19T21:04:30.552 INFO:tasks.workunit.client.0.smithi163.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-19T21:04:30.552 INFO:tasks.workunit.client.0.smithi163.stderr:state without impacting any branches by switching back to a branch. 2024-02-19T21:04:30.552 INFO:tasks.workunit.client.0.smithi163.stderr: 2024-02-19T21:04:30.552 INFO:tasks.workunit.client.0.smithi163.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-19T21:04:30.552 INFO:tasks.workunit.client.0.smithi163.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-19T21:04:30.552 INFO:tasks.workunit.client.0.smithi163.stderr: 2024-02-19T21:04:30.552 INFO:tasks.workunit.client.0.smithi163.stderr: git switch -c 2024-02-19T21:04:30.552 INFO:tasks.workunit.client.0.smithi163.stderr: 2024-02-19T21:04:30.553 INFO:tasks.workunit.client.0.smithi163.stderr:Or undo this operation with: 2024-02-19T21:04:30.553 INFO:tasks.workunit.client.0.smithi163.stderr: 2024-02-19T21:04:30.553 INFO:tasks.workunit.client.0.smithi163.stderr: git switch - 2024-02-19T21:04:30.553 INFO:tasks.workunit.client.0.smithi163.stderr: 2024-02-19T21:04:30.553 INFO:tasks.workunit.client.0.smithi163.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-19T21:04:30.553 INFO:tasks.workunit.client.0.smithi163.stderr: 2024-02-19T21:04:30.553 INFO:tasks.workunit.client.0.smithi163.stderr:HEAD is now at 81bd20d6342 Merge pull request #55523 from ljflores/wip-tracker-64343-pacific 2024-02-19T21:04:30.559 DEBUG:teuthology.orchestra.run.smithi163:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-19T21:04:30.633 INFO:tasks.workunit.client.0.smithi163.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-19T21:04:30.635 INFO:tasks.workunit.client.0.smithi163.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-19T21:04:30.635 INFO:tasks.workunit.client.0.smithi163.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-19T21:04:30.651 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:30.680 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:30.700 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:30.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:30.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:30.803 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:04:30.832 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:30.851 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:04:30.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:30.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:04:30.955 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:30.984 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:31.003 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:31.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:31.108 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:31.135 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:31.154 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:31.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:31.172 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:30.054728+0000 mgr.y (mgr.14550) 132 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:31.172 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:30.054728+0000 mgr.y (mgr.14550) 132 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:31.257 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:31.284 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:31.314 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:30.054728+0000 mgr.y (mgr.14550) 132 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:31.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-19T21:04:32.241 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:30.955322+0000 mgr.y (mgr.14550) 133 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:04:32.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:30.955322+0000 mgr.y (mgr.14550) 133 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:04:32.250 INFO:tasks.workunit.client.0.smithi163.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-19T21:04:32.314 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:30.955322+0000 mgr.y (mgr.14550) 133 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:04:32.334 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:32.336 INFO:tasks.workunit.client.0.smithi163.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-19T21:04:32.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:32.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:32.387 INFO:tasks.workunit.client.0.smithi163.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-19T21:04:32.388 INFO:tasks.workunit.client.0.smithi163.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-19T21:04:32.389 INFO:tasks.workunit.client.0.smithi163.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-19T21:04:32.426 INFO:tasks.workunit.client.0.smithi163.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-19T21:04:32.430 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-02-19T21:04:32.430 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-19T21:04:32.438 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:32.448 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-02-19T21:04:32.449 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-02-19T21:04:32.449 DEBUG:teuthology.orchestra.run.smithi163:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=81bd20d634209c7cb82c18be12b4b5a05643ebf1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-02-19T21:04:32.470 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:32.491 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:32.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:32.536 INFO:tasks.workunit.client.0.smithi163.stderr:+ parallel=1 2024-02-19T21:04:32.537 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' '' = --serial ']' 2024-02-19T21:04:32.537 INFO:tasks.workunit.client.0.smithi163.stderr:+ color= 2024-02-19T21:04:32.537 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' -t 1 ']' 2024-02-19T21:04:32.537 INFO:tasks.workunit.client.0.smithi163.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-02-19T21:04:32.537 INFO:tasks.workunit.client.0.smithi163.stderr:+ declare -A pids 2024-02-19T21:04:32.537 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.537 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.537 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_aio 2024-02-19T21:04:32.537 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_aio' 2024-02-19T21:04:32.537 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_aio 2024-02-19T21:04:32.537 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.538 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_aio 2024-02-19T21:04:32.538 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83840 2024-02-19T21:04:32.538 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_aio on pid 83840' 2024-02-19T21:04:32.538 INFO:tasks.workunit.client.0.smithi163.stdout:test api_aio on pid 83840 2024-02-19T21:04:32.540 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83840 2024-02-19T21:04:32.540 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.540 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.540 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-02-19T21:04:32.540 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_aio_pp 2024-02-19T21:04:32.540 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_aio_pp' 2024-02-19T21:04:32.540 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_aio_pp 2024-02-19T21:04:32.540 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.540 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_aio 2024-02-19T21:04:32.540 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_aio.log 2024-02-19T21:04:32.540 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_aio: /' 2024-02-19T21:04:32.540 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_aio_pp 2024-02-19T21:04:32.540 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83848 2024-02-19T21:04:32.541 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_aio_pp on pid 83848' 2024-02-19T21:04:32.541 INFO:tasks.workunit.client.0.smithi163.stdout:test api_aio_pp on pid 83848 2024-02-19T21:04:32.542 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83848 2024-02-19T21:04:32.542 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.542 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.542 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-02-19T21:04:32.542 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_io 2024-02-19T21:04:32.542 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_io' 2024-02-19T21:04:32.542 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_io 2024-02-19T21:04:32.542 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.542 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_aio_pp 2024-02-19T21:04:32.542 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_aio_pp: /' 2024-02-19T21:04:32.542 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-02-19T21:04:32.543 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_io 2024-02-19T21:04:32.543 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83859 2024-02-19T21:04:32.543 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_io on pid 83859' 2024-02-19T21:04:32.543 INFO:tasks.workunit.client.0.smithi163.stdout:test api_io on pid 83859 2024-02-19T21:04:32.545 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83859 2024-02-19T21:04:32.545 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.545 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.545 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-02-19T21:04:32.545 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_io_pp 2024-02-19T21:04:32.545 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_io_pp' 2024-02-19T21:04:32.545 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_io_pp 2024-02-19T21:04:32.546 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.546 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_io 2024-02-19T21:04:32.546 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_io.log 2024-02-19T21:04:32.546 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_io: /' 2024-02-19T21:04:32.546 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_io_pp 2024-02-19T21:04:32.546 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83867 2024-02-19T21:04:32.546 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_io_pp on pid 83867' 2024-02-19T21:04:32.546 INFO:tasks.workunit.client.0.smithi163.stdout:test api_io_pp on pid 83867 2024-02-19T21:04:32.547 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83867 2024-02-19T21:04:32.547 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.547 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.548 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-02-19T21:04:32.548 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_asio 2024-02-19T21:04:32.548 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_asio' 2024-02-19T21:04:32.548 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_asio 2024-02-19T21:04:32.548 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.548 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_io_pp 2024-02-19T21:04:32.548 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-02-19T21:04:32.548 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_io_pp: /' 2024-02-19T21:04:32.549 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_asio 2024-02-19T21:04:32.549 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83875 2024-02-19T21:04:32.549 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_asio on pid 83875' 2024-02-19T21:04:32.549 INFO:tasks.workunit.client.0.smithi163.stdout:test api_asio on pid 83875 2024-02-19T21:04:32.550 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83875 2024-02-19T21:04:32.550 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.551 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.551 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-02-19T21:04:32.551 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_list 2024-02-19T21:04:32.551 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_list' 2024-02-19T21:04:32.551 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_list 2024-02-19T21:04:32.551 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.551 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_asio 2024-02-19T21:04:32.551 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_asio.log 2024-02-19T21:04:32.551 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_asio: /' 2024-02-19T21:04:32.551 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_list 2024-02-19T21:04:32.551 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83883 2024-02-19T21:04:32.552 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_list on pid 83883' 2024-02-19T21:04:32.552 INFO:tasks.workunit.client.0.smithi163.stdout:test api_list on pid 83883 2024-02-19T21:04:32.553 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83883 2024-02-19T21:04:32.553 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.553 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-02-19T21:04:32.553 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.554 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_lock 2024-02-19T21:04:32.554 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_lock' 2024-02-19T21:04:32.554 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_lock 2024-02-19T21:04:32.554 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.554 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_list 2024-02-19T21:04:32.554 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_list.log 2024-02-19T21:04:32.554 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_list: /' 2024-02-19T21:04:32.554 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_lock 2024-02-19T21:04:32.554 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83891 2024-02-19T21:04:32.554 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_lock on pid 83891' 2024-02-19T21:04:32.554 INFO:tasks.workunit.client.0.smithi163.stdout:test api_lock on pid 83891 2024-02-19T21:04:32.556 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83891 2024-02-19T21:04:32.556 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.556 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-02-19T21:04:32.556 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.556 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_lock_pp 2024-02-19T21:04:32.556 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_lock_pp' 2024-02-19T21:04:32.556 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_lock_pp 2024-02-19T21:04:32.556 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.556 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_lock 2024-02-19T21:04:32.556 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_lock.log 2024-02-19T21:04:32.556 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_lock: /' 2024-02-19T21:04:32.556 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_lock_pp 2024-02-19T21:04:32.556 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83899 2024-02-19T21:04:32.557 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_lock_pp on pid 83899' 2024-02-19T21:04:32.557 INFO:tasks.workunit.client.0.smithi163.stdout:test api_lock_pp on pid 83899 2024-02-19T21:04:32.558 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83899 2024-02-19T21:04:32.558 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.558 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-02-19T21:04:32.558 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.558 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_misc 2024-02-19T21:04:32.558 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_misc' 2024-02-19T21:04:32.558 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_misc 2024-02-19T21:04:32.558 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.558 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_lock_pp 2024-02-19T21:04:32.558 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-02-19T21:04:32.558 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_lock_pp: /' 2024-02-19T21:04:32.559 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_misc 2024-02-19T21:04:32.559 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83907 2024-02-19T21:04:32.559 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_misc on pid 83907' 2024-02-19T21:04:32.559 INFO:tasks.workunit.client.0.smithi163.stdout:test api_misc on pid 83907 2024-02-19T21:04:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83907 2024-02-19T21:04:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-02-19T21:04:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_misc_pp 2024-02-19T21:04:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_misc_pp' 2024-02-19T21:04:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_misc_pp 2024-02-19T21:04:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_misc 2024-02-19T21:04:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_misc.log 2024-02-19T21:04:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_misc: /' 2024-02-19T21:04:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_misc_pp 2024-02-19T21:04:32.561 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83915 2024-02-19T21:04:32.562 INFO:tasks.workunit.client.0.smithi163.stdout:test api_misc_pp on pid 83915 2024-02-19T21:04:32.563 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_misc_pp on pid 83915' 2024-02-19T21:04:32.563 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83915 2024-02-19T21:04:32.563 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.563 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.563 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-02-19T21:04:32.563 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_tier_pp 2024-02-19T21:04:32.563 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_tier_pp' 2024-02-19T21:04:32.563 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_tier_pp 2024-02-19T21:04:32.564 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.564 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_misc_pp 2024-02-19T21:04:32.564 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-02-19T21:04:32.564 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_misc_pp: /' 2024-02-19T21:04:32.564 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_tier_pp 2024-02-19T21:04:32.564 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83923 2024-02-19T21:04:32.564 INFO:tasks.workunit.client.0.smithi163.stdout:test api_tier_pp on pid 83923 2024-02-19T21:04:32.566 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_tier_pp on pid 83923' 2024-02-19T21:04:32.566 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83923 2024-02-19T21:04:32.566 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.566 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.566 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-02-19T21:04:32.566 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_pool 2024-02-19T21:04:32.567 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_pool' 2024-02-19T21:04:32.567 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_pool 2024-02-19T21:04:32.567 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.567 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_tier_pp 2024-02-19T21:04:32.567 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-02-19T21:04:32.567 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_tier_pp: /' 2024-02-19T21:04:32.567 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_pool 2024-02-19T21:04:32.567 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83931 2024-02-19T21:04:32.567 INFO:tasks.workunit.client.0.smithi163.stdout:test api_pool on pid 83931 2024-02-19T21:04:32.569 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_pool on pid 83931' 2024-02-19T21:04:32.569 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83931 2024-02-19T21:04:32.569 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.569 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.569 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-02-19T21:04:32.570 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_snapshots 2024-02-19T21:04:32.570 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_snapshots' 2024-02-19T21:04:32.570 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_snapshots 2024-02-19T21:04:32.570 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.570 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_pool 2024-02-19T21:04:32.570 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_pool.log 2024-02-19T21:04:32.570 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_pool: /' 2024-02-19T21:04:32.570 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_snapshots 2024-02-19T21:04:32.570 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83939 2024-02-19T21:04:32.570 INFO:tasks.workunit.client.0.smithi163.stdout:test api_snapshots on pid 83939 2024-02-19T21:04:32.571 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_snapshots on pid 83939' 2024-02-19T21:04:32.572 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83939 2024-02-19T21:04:32.572 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.572 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-02-19T21:04:32.572 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.572 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_snapshots_pp 2024-02-19T21:04:32.572 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_snapshots_pp' 2024-02-19T21:04:32.572 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_snapshots_pp 2024-02-19T21:04:32.572 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.572 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_snapshots 2024-02-19T21:04:32.572 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-02-19T21:04:32.572 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_snapshots: /' 2024-02-19T21:04:32.572 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_snapshots_pp 2024-02-19T21:04:32.573 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83947 2024-02-19T21:04:32.573 INFO:tasks.workunit.client.0.smithi163.stdout:test api_snapshots_pp on pid 83947 2024-02-19T21:04:32.574 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_snapshots_pp on pid 83947' 2024-02-19T21:04:32.574 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83947 2024-02-19T21:04:32.574 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.574 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-02-19T21:04:32.574 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.574 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_stat 2024-02-19T21:04:32.575 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_stat' 2024-02-19T21:04:32.575 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_stat 2024-02-19T21:04:32.575 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.575 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_snapshots_pp 2024-02-19T21:04:32.575 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-02-19T21:04:32.575 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-02-19T21:04:32.575 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_stat 2024-02-19T21:04:32.575 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83955 2024-02-19T21:04:32.575 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_stat on pid 83955' 2024-02-19T21:04:32.575 INFO:tasks.workunit.client.0.smithi163.stdout:test api_stat on pid 83955 2024-02-19T21:04:32.577 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83955 2024-02-19T21:04:32.577 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.577 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.577 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-02-19T21:04:32.577 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_stat_pp 2024-02-19T21:04:32.577 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_stat_pp' 2024-02-19T21:04:32.577 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_stat_pp 2024-02-19T21:04:32.577 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.577 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_stat 2024-02-19T21:04:32.577 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_stat.log 2024-02-19T21:04:32.577 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_stat: /' 2024-02-19T21:04:32.577 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_stat_pp 2024-02-19T21:04:32.578 INFO:tasks.workunit.client.0.smithi163.stdout:test api_stat_pp on pid 83963 2024-02-19T21:04:32.578 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83963 2024-02-19T21:04:32.579 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_stat_pp on pid 83963' 2024-02-19T21:04:32.579 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83963 2024-02-19T21:04:32.579 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.579 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.579 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-02-19T21:04:32.579 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_watch_notify 2024-02-19T21:04:32.579 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_watch_notify' 2024-02-19T21:04:32.579 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_watch_notify 2024-02-19T21:04:32.579 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.579 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_stat_pp 2024-02-19T21:04:32.579 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-02-19T21:04:32.579 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_stat_pp: /' 2024-02-19T21:04:32.579 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_watch_notify 2024-02-19T21:04:32.580 INFO:tasks.workunit.client.0.smithi163.stdout:test api_watch_notify on pid 83971 2024-02-19T21:04:32.581 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83971 2024-02-19T21:04:32.581 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_watch_notify on pid 83971' 2024-02-19T21:04:32.581 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83971 2024-02-19T21:04:32.581 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.581 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.581 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-02-19T21:04:32.581 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_watch_notify_pp 2024-02-19T21:04:32.581 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_watch_notify_pp' 2024-02-19T21:04:32.581 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_watch_notify_pp 2024-02-19T21:04:32.581 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.581 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_watch_notify 2024-02-19T21:04:32.581 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-02-19T21:04:32.581 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_watch_notify: /' 2024-02-19T21:04:32.582 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_watch_notify_pp 2024-02-19T21:04:32.582 INFO:tasks.workunit.client.0.smithi163.stdout:test api_watch_notify_pp on pid 83979 2024-02-19T21:04:32.583 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83979 2024-02-19T21:04:32.583 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_watch_notify_pp on pid 83979' 2024-02-19T21:04:32.583 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83979 2024-02-19T21:04:32.583 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.583 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.583 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-02-19T21:04:32.584 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_cmd 2024-02-19T21:04:32.584 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_cmd' 2024-02-19T21:04:32.584 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_cmd 2024-02-19T21:04:32.584 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.584 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_watch_notify_pp 2024-02-19T21:04:32.585 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-02-19T21:04:32.585 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-02-19T21:04:32.585 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_cmd 2024-02-19T21:04:32.585 INFO:tasks.workunit.client.0.smithi163.stdout:test api_cmd on pid 83987 2024-02-19T21:04:32.587 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83987 2024-02-19T21:04:32.587 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_cmd on pid 83987' 2024-02-19T21:04:32.587 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83987 2024-02-19T21:04:32.588 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.588 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.588 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-02-19T21:04:32.588 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_cmd_pp 2024-02-19T21:04:32.588 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_cmd_pp' 2024-02-19T21:04:32.588 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_cmd_pp 2024-02-19T21:04:32.588 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.589 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_cmd 2024-02-19T21:04:32.589 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_cmd.log 2024-02-19T21:04:32.589 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_cmd: /' 2024-02-19T21:04:32.589 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_cmd_pp 2024-02-19T21:04:32.589 INFO:tasks.workunit.client.0.smithi163.stdout:test api_cmd_pp on pid 83995 2024-02-19T21:04:32.590 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83995 2024-02-19T21:04:32.590 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_cmd_pp on pid 83995' 2024-02-19T21:04:32.591 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=83995 2024-02-19T21:04:32.591 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.591 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.591 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-02-19T21:04:32.591 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_service 2024-02-19T21:04:32.591 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_service' 2024-02-19T21:04:32.591 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_service 2024-02-19T21:04:32.591 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.591 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_cmd_pp 2024-02-19T21:04:32.591 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-02-19T21:04:32.591 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-02-19T21:04:32.591 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_service 2024-02-19T21:04:32.591 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=84003 2024-02-19T21:04:32.591 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_service on pid 84003' 2024-02-19T21:04:32.592 INFO:tasks.workunit.client.0.smithi163.stdout:test api_service on pid 84003 2024-02-19T21:04:32.593 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=84003 2024-02-19T21:04:32.593 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.593 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.593 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-02-19T21:04:32.593 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_service_pp 2024-02-19T21:04:32.594 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_service_pp' 2024-02-19T21:04:32.594 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_service_pp 2024-02-19T21:04:32.594 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.594 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_service 2024-02-19T21:04:32.594 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_service.log 2024-02-19T21:04:32.594 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_service: /' 2024-02-19T21:04:32.594 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_service_pp 2024-02-19T21:04:32.594 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=84011 2024-02-19T21:04:32.594 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_service_pp on pid 84011' 2024-02-19T21:04:32.594 INFO:tasks.workunit.client.0.smithi163.stdout:test api_service_pp on pid 84011 2024-02-19T21:04:32.596 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=84011 2024-02-19T21:04:32.596 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.596 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.596 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-02-19T21:04:32.596 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_c_write_operations 2024-02-19T21:04:32.596 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_c_write_operations' 2024-02-19T21:04:32.596 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_c_write_operations 2024-02-19T21:04:32.596 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.596 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_service_pp 2024-02-19T21:04:32.596 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-02-19T21:04:32.597 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_service_pp: /' 2024-02-19T21:04:32.597 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_c_write_operations 2024-02-19T21:04:32.597 INFO:tasks.workunit.client.0.smithi163.stdout:test api_c_write_operations on pid 84019 2024-02-19T21:04:32.598 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:32.598 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=84019 2024-02-19T21:04:32.598 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_c_write_operations on pid 84019' 2024-02-19T21:04:32.598 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=84019 2024-02-19T21:04:32.598 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-02-19T21:04:32.599 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.599 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.599 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s api_c_read_operations 2024-02-19T21:04:32.599 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' api_c_read_operations' 2024-02-19T21:04:32.599 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo api_c_read_operations 2024-02-19T21:04:32.599 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.599 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_c_write_operations 2024-02-19T21:04:32.599 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-02-19T21:04:32.599 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-02-19T21:04:32.599 INFO:tasks.workunit.client.0.smithi163.stdout:test api_c_read_operations on pid 84027 2024-02-19T21:04:32.601 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=api_c_read_operations 2024-02-19T21:04:32.601 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=84027 2024-02-19T21:04:32.601 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test api_c_read_operations on pid 84027' 2024-02-19T21:04:32.601 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=84027 2024-02-19T21:04:32.601 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-02-19T21:04:32.601 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.601 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.601 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s list_parallel 2024-02-19T21:04:32.601 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' list_parallel' 2024-02-19T21:04:32.601 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo list_parallel 2024-02-19T21:04:32.601 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.601 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_api_c_read_operations 2024-02-19T21:04:32.601 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-02-19T21:04:32.601 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-02-19T21:04:32.602 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=list_parallel 2024-02-19T21:04:32.602 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=84035 2024-02-19T21:04:32.602 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test list_parallel on pid 84035' 2024-02-19T21:04:32.602 INFO:tasks.workunit.client.0.smithi163.stdout:test list_parallel on pid 84035 2024-02-19T21:04:32.603 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=84035 2024-02-19T21:04:32.603 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.603 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.603 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-02-19T21:04:32.603 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s open_pools_parallel 2024-02-19T21:04:32.603 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' open_pools_parallel' 2024-02-19T21:04:32.604 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo open_pools_parallel 2024-02-19T21:04:32.604 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.604 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_list_parallel 2024-02-19T21:04:32.604 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_list_parallel.log 2024-02-19T21:04:32.604 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ list_parallel: /' 2024-02-19T21:04:32.604 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=open_pools_parallel 2024-02-19T21:04:32.604 INFO:tasks.workunit.client.0.smithi163.stdout:test open_pools_parallel on pid 84043 2024-02-19T21:04:32.605 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=84043 2024-02-19T21:04:32.605 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test open_pools_parallel on pid 84043' 2024-02-19T21:04:32.606 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=84043 2024-02-19T21:04:32.606 INFO:tasks.workunit.client.0.smithi163.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-02-19T21:04:32.606 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.606 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-02-19T21:04:32.606 INFO:tasks.workunit.client.0.smithi163.stderr:++ printf %25s delete_pools_parallel 2024-02-19T21:04:32.606 INFO:tasks.workunit.client.0.smithi163.stderr:+ r=' delete_pools_parallel' 2024-02-19T21:04:32.606 INFO:tasks.workunit.client.0.smithi163.stderr:++ echo delete_pools_parallel 2024-02-19T21:04:32.606 INFO:tasks.workunit.client.0.smithi163.stderr:++ awk '{print $1}' 2024-02-19T21:04:32.606 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_open_pools_parallel 2024-02-19T21:04:32.606 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-02-19T21:04:32.606 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-02-19T21:04:32.606 INFO:tasks.workunit.client.0.smithi163.stderr:+ ff=delete_pools_parallel 2024-02-19T21:04:32.607 INFO:tasks.workunit.client.0.smithi163.stdout:test delete_pools_parallel on pid 84051 2024-02-19T21:04:32.607 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=84051 2024-02-19T21:04:32.607 INFO:tasks.workunit.client.0.smithi163.stderr:+ echo 'test delete_pools_parallel on pid 84051' 2024-02-19T21:04:32.607 INFO:tasks.workunit.client.0.smithi163.stderr:+ pids[$f]=84051 2024-02-19T21:04:32.607 INFO:tasks.workunit.client.0.smithi163.stderr:+ ret=0 2024-02-19T21:04:32.608 INFO:tasks.workunit.client.0.smithi163.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-02-19T21:04:32.608 INFO:tasks.workunit.client.0.smithi163.stderr:+ '[' 1 -eq 1 ']' 2024-02-19T21:04:32.608 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:04:32.608 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83939 2024-02-19T21:04:32.608 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83939 2024-02-19T21:04:32.608 INFO:tasks.workunit.client.0.smithi163.stderr:+ ceph_test_rados_delete_pools_parallel 2024-02-19T21:04:32.608 INFO:tasks.workunit.client.0.smithi163.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-02-19T21:04:32.608 INFO:tasks.workunit.client.0.smithi163.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-02-19T21:04:32.631 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:33.740 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:04:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:04:33] "GET /metrics HTTP/1.1" 200 172246 "" "Prometheus/2.33.4" 2024-02-19T21:04:33.995 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:33.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:34.099 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:04:34.130 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:34.151 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:04:34.154 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:34.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:04:34.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.104+0000 7f1a96130700 1 -- 172.21.15.163:0/4070238237 >> v1:172.21.15.163:6790/0 conn(0x7f1a900fd970 legacy=0x7f1a900fdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:34.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.104+0000 7f1a96130700 1 -- 172.21.15.163:0/4070238237 shutdown_connections 2024-02-19T21:04:34.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.104+0000 7f1a96130700 1 -- 172.21.15.163:0/4070238237 >> 172.21.15.163:0/4070238237 conn(0x7f1a900f3590 msgr2=0x7f1a900f59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:34.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.104+0000 7f1a96130700 1 -- 172.21.15.163:0/4070238237 shutdown_connections 2024-02-19T21:04:34.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.104+0000 7f1a96130700 1 -- 172.21.15.163:0/4070238237 wait complete. 2024-02-19T21:04:34.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.104+0000 7f1a96130700 1 librados: starting msgr at 2024-02-19T21:04:34.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.104+0000 7f1a96130700 1 librados: starting objecter 2024-02-19T21:04:34.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.104+0000 7f1a96130700 1 Processor -- start 2024-02-19T21:04:34.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a96130700 1 -- start start 2024-02-19T21:04:34.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a96130700 1 librados: setting wanted keys 2024-02-19T21:04:34.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a96130700 1 librados: calling monclient init 2024-02-19T21:04:34.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a96130700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1a9006ccf0 con 0x7f1a901042b0 2024-02-19T21:04:34.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a96130700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1a9006ce80 con 0x7f1a900fd970 2024-02-19T21:04:34.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a96130700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1a9011bb70 con 0x7f1a90100770 2024-02-19T21:04:34.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a8effd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f1a90100770 0x7f1a90114d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:53354/0 (socket says 172.21.15.163:53354) 2024-02-19T21:04:34.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a8effd700 1 -- 172.21.15.163:0/2458544903 learned_addr learned my addr 172.21.15.163:0/2458544903 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:04:34.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a8cff9700 10 client.?.objecter ms_handle_connect 0x7f1a901042b0 2024-02-19T21:04:34.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a8cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:34.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a8cff9700 10 client.?.objecter ms_handle_connect 0x7f1a90100770 2024-02-19T21:04:34.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a8cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:34.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164968191 0 0) 0x7f1a9011bb70 con 0x7f1a90100770 2024-02-19T21:04:34.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a78003030 con 0x7f1a90100770 2024-02-19T21:04:34.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a8cff9700 10 client.?.objecter ms_handle_connect 0x7f1a900fd970 2024-02-19T21:04:34.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a8cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:34.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37212509 0 0) 0x7f1a9006ce80 con 0x7f1a900fd970 2024-02-19T21:04:34.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a9011bb70 con 0x7f1a900fd970 2024-02-19T21:04:34.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3460425658 0 0) 0x7f1a78003030 con 0x7f1a90100770 2024-02-19T21:04:34.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a9006ce80 con 0x7f1a90100770 2024-02-19T21:04:34.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.105+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1a840018c0 con 0x7f1a90100770 2024-02-19T21:04:34.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.106+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 604968554 0 0) 0x7f1a9006ce80 con 0x7f1a90100770 2024-02-19T21:04:34.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.106+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 >> v1:172.21.15.193:6789/0 conn(0x7f1a900fd970 legacy=0x7f1a900fc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:34.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.106+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 >> v1:172.21.15.163:6789/0 conn(0x7f1a901042b0 legacy=0x7f1a90118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:34.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.106+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9011cd10 con 0x7f1a90100770 2024-02-19T21:04:34.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.106+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a9011bd60 con 0x7f1a90100770 2024-02-19T21:04:34.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.106+0000 7f1a96130700 10 client.34940.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:34.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.106+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a9011c210 con 0x7f1a90100770 2024-02-19T21:04:34.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.106+0000 7f1a96130700 1 librados: init done 2024-02-19T21:04:34.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.106+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f1a84001f80 con 0x7f1a90100770 2024-02-19T21:04:34.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.106+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1a84005a00 con 0x7f1a90100770 2024-02-19T21:04:34.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.106+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1a7c004f00 con 0x7f1a90100770 2024-02-19T21:04:34.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.107+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f1a84001ae0 con 0x7f1a90100770 2024-02-19T21:04:34.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.107+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(98..98 src has 43..98) v4 ==== 6943+0+0 (unknown 1756307953 0 0) 0x7f1a84078850 con 0x7f1a90100770 2024-02-19T21:04:34.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.107+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 osd_map(98..98 src has 43..98) v4 2024-02-19T21:04:34.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.107+0000 7f1a8cff9700 3 client.34940.objecter handle_osd_map got epochs [98,98] > 0 2024-02-19T21:04:34.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.107+0000 7f1a8cff9700 3 client.34940.objecter handle_osd_map decoding full epoch 98 2024-02-19T21:04:34.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.108+0000 7f1a8cff9700 20 client.34940.objecter dump_active .. 0 homeless 2024-02-19T21:04:34.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.108+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v98) v1 ==== 54+0+16 (unknown 2227819007 0 2420000192) 0x7f1a84078c10 con 0x7f1a90100770 2024-02-19T21:04:34.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.108+0000 7f1a8cff9700 10 client.34940.objecter ms_handle_connect 0x7f1a78063810 2024-02-19T21:04:34.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.108+0000 7f1a96130700 10 client.34940.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:34.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.108+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 1 2024-02-19T21:04:34.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.108+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c0051a0 osd=0 3 2024-02-19T21:04:34.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.108+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c0051a0 osd=0 3 2024-02-19T21:04:34.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.108+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c0051a0 osd=0 3 2024-02-19T21:04:34.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.108+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 1 2024-02-19T21:04:34.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.108+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c0051a0 osd=0 3 2024-02-19T21:04:34.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.108+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 0 1 2024-02-19T21:04:34.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.108+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c0051a0 osd=0 4 2024-02-19T21:04:34.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.108+0000 7f1a96130700 10 client.34940.objecter _send_command 1 2024-02-19T21:04:34.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.108+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a7c004950 con 0x7f1a7c004530 2024-02-19T21:04:34.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.108+0000 7f1a8cff9700 10 client.34940.objecter ms_handle_connect 0x7f1a7c004530 2024-02-19T21:04:34.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.109+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1a7c004950 con 0x7f1a7c004530 2024-02-19T21:04:34.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.109+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:04:34.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.109+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 1 = system:0 2024-02-19T21:04:34.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.109+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c0051a0 osd=0 4 2024-02-19T21:04:34.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.109+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 0 1 2024-02-19T21:04:34.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.124+0000 7f1a96130700 10 client.34940.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:34.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.124+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 2 2024-02-19T21:04:34.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.124+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c0051a0 osd=0 3 2024-02-19T21:04:34.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.124+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c0051a0 osd=0 3 2024-02-19T21:04:34.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.124+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c0051a0 osd=0 3 2024-02-19T21:04:34.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.124+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 2 2024-02-19T21:04:34.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.124+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c0051a0 osd=0 3 2024-02-19T21:04:34.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.125+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 0 2 2024-02-19T21:04:34.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.125+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c0051a0 osd=0 4 2024-02-19T21:04:34.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.125+0000 7f1a96130700 10 client.34940.objecter _send_command 2 2024-02-19T21:04:34.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.125+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1a7c004d30 con 0x7f1a7c004530 2024-02-19T21:04:34.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1a7c004d30 con 0x7f1a7c004530 2024-02-19T21:04:34.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:34.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c0051a0 osd=0 4 2024-02-19T21:04:34.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 0 2 2024-02-19T21:04:34.219 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:04:34.219 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:04:34.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a96130700 10 client.34940.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:34.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 3 2024-02-19T21:04:34.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c004230 osd=1 3 2024-02-19T21:04:34.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c004230 osd=1 3 2024-02-19T21:04:34.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c004230 osd=1 3 2024-02-19T21:04:34.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 3 2024-02-19T21:04:34.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c004230 osd=1 3 2024-02-19T21:04:34.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 1 3 2024-02-19T21:04:34.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c004230 osd=1 4 2024-02-19T21:04:34.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a96130700 10 client.34940.objecter _send_command 3 2024-02-19T21:04:34.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.126+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a7c00bf30 con 0x7f1a7c003b00 2024-02-19T21:04:34.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.127+0000 7f1a8cff9700 10 client.34940.objecter ms_handle_connect 0x7f1a7c003b00 2024-02-19T21:04:34.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.127+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1a7c00bf30 con 0x7f1a7c003b00 2024-02-19T21:04:34.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.127+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.127+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 3 = system:0 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.127+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c004230 osd=1 4 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.127+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 1 3 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.144+0000 7f1a96130700 10 client.34940.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.144+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 4 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.144+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c004230 osd=1 3 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.144+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c004230 osd=1 3 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.144+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c004230 osd=1 3 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.144+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 4 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.144+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c004230 osd=1 3 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.144+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 1 4 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.144+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c004230 osd=1 4 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.144+0000 7f1a96130700 10 client.34940.objecter _send_command 4 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.144+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1a7c00d220 con 0x7f1a7c003b00 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.145+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1a7c00d220 con 0x7f1a7c003b00 2024-02-19T21:04:34.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.145+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:34.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.145+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.145+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c004230 osd=1 4 2024-02-19T21:04:34.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.145+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 1 4 2024-02-19T21:04:34.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a96130700 10 client.34940.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:34.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 5 2024-02-19T21:04:34.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c003360 osd=2 3 2024-02-19T21:04:34.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c003360 osd=2 3 2024-02-19T21:04:34.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c003360 osd=2 3 2024-02-19T21:04:34.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 5 2024-02-19T21:04:34.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c003360 osd=2 3 2024-02-19T21:04:34.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 2 5 2024-02-19T21:04:34.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c003360 osd=2 4 2024-02-19T21:04:34.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a96130700 10 client.34940.objecter _send_command 5 2024-02-19T21:04:34.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a7c010d20 con 0x7f1a7c002070 2024-02-19T21:04:34.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a8cff9700 10 client.34940.objecter ms_handle_connect 0x7f1a7c002070 2024-02-19T21:04:34.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1a7c010d20 con 0x7f1a7c002070 2024-02-19T21:04:34.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:04:34.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 5 = system:0 2024-02-19T21:04:34.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c003360 osd=2 4 2024-02-19T21:04:34.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.146+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 2 5 2024-02-19T21:04:34.223 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:04:34.223 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:04:34.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.161+0000 7f1a96130700 10 client.34940.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:34.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.161+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 6 2024-02-19T21:04:34.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.161+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c003360 osd=2 3 2024-02-19T21:04:34.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.161+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c003360 osd=2 3 2024-02-19T21:04:34.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.161+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c003360 osd=2 3 2024-02-19T21:04:34.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.161+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 6 2024-02-19T21:04:34.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.161+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c003360 osd=2 3 2024-02-19T21:04:34.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.161+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 2 6 2024-02-19T21:04:34.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.161+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c003360 osd=2 4 2024-02-19T21:04:34.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.161+0000 7f1a96130700 10 client.34940.objecter _send_command 6 2024-02-19T21:04:34.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.161+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1a7c001690 con 0x7f1a7c002070 2024-02-19T21:04:34.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1a7c001690 con 0x7f1a7c002070 2024-02-19T21:04:34.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:34.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c003360 osd=2 4 2024-02-19T21:04:34.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 2 6 2024-02-19T21:04:34.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a96130700 10 client.34940.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:34.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 7 2024-02-19T21:04:34.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c00c870 osd=3 3 2024-02-19T21:04:34.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c00c870 osd=3 3 2024-02-19T21:04:34.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c00c870 osd=3 3 2024-02-19T21:04:34.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 7 2024-02-19T21:04:34.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c00c870 osd=3 3 2024-02-19T21:04:34.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 3 7 2024-02-19T21:04:34.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c00c870 osd=3 4 2024-02-19T21:04:34.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a96130700 10 client.34940.objecter _send_command 7 2024-02-19T21:04:34.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a7c0119f0 con 0x7f1a7c010ec0 2024-02-19T21:04:34.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.163+0000 7f1a8cff9700 10 client.34940.objecter ms_handle_connect 0x7f1a7c010ec0 2024-02-19T21:04:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.164+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1a7c0119f0 con 0x7f1a7c010ec0 2024-02-19T21:04:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.164+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:04:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.164+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 7 = system:0 2024-02-19T21:04:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.164+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c00c870 osd=3 4 2024-02-19T21:04:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.164+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 3 7 2024-02-19T21:04:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.180+0000 7f1a96130700 10 client.34940.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.180+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 8 2024-02-19T21:04:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.180+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c00c870 osd=3 3 2024-02-19T21:04:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.180+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c00c870 osd=3 3 2024-02-19T21:04:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.180+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c00c870 osd=3 3 2024-02-19T21:04:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.180+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 8 2024-02-19T21:04:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.180+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c00c870 osd=3 3 2024-02-19T21:04:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.180+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 3 8 2024-02-19T21:04:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.180+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c00c870 osd=3 4 2024-02-19T21:04:34.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.180+0000 7f1a96130700 10 client.34940.objecter _send_command 8 2024-02-19T21:04:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.180+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1a7c00c2e0 con 0x7f1a7c010ec0 2024-02-19T21:04:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1a7c00c2e0 con 0x7f1a7c010ec0 2024-02-19T21:04:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c00c870 osd=3 4 2024-02-19T21:04:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 3 8 2024-02-19T21:04:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a96130700 10 client.34940.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 9 2024-02-19T21:04:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c00c0a0 osd=4 3 2024-02-19T21:04:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c00c0a0 osd=4 3 2024-02-19T21:04:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c00c0a0 osd=4 3 2024-02-19T21:04:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 9 2024-02-19T21:04:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c00c0a0 osd=4 3 2024-02-19T21:04:34.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 4 9 2024-02-19T21:04:34.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c00c0a0 osd=4 4 2024-02-19T21:04:34.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a96130700 10 client.34940.objecter _send_command 9 2024-02-19T21:04:34.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.181+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a7c00c210 con 0x7f1a7c016bf0 2024-02-19T21:04:34.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.182+0000 7f1a8cff9700 10 client.34940.objecter ms_handle_connect 0x7f1a7c016bf0 2024-02-19T21:04:34.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.183+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1a7c00c210 con 0x7f1a7c016bf0 2024-02-19T21:04:34.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.183+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:04:34.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.183+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 9 = system:0 2024-02-19T21:04:34.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.183+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c00c0a0 osd=4 4 2024-02-19T21:04:34.229 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:04:34.229 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:04:34.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.183+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 4 9 2024-02-19T21:04:34.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.197+0000 7f1a96130700 10 client.34940.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:34.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.197+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 10 2024-02-19T21:04:34.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.197+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c00c0a0 osd=4 3 2024-02-19T21:04:34.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.197+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c00c0a0 osd=4 3 2024-02-19T21:04:34.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.197+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c00c0a0 osd=4 3 2024-02-19T21:04:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.197+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 10 2024-02-19T21:04:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.197+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c00c0a0 osd=4 3 2024-02-19T21:04:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.197+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 4 10 2024-02-19T21:04:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.197+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c00c0a0 osd=4 4 2024-02-19T21:04:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.197+0000 7f1a96130700 10 client.34940.objecter _send_command 10 2024-02-19T21:04:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.197+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1a7c00c210 con 0x7f1a7c016bf0 2024-02-19T21:04:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1a7c00c210 con 0x7f1a7c016bf0 2024-02-19T21:04:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c00c0a0 osd=4 4 2024-02-19T21:04:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 4 10 2024-02-19T21:04:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a96130700 10 client.34940.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 11 2024-02-19T21:04:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c00c210 osd=5 3 2024-02-19T21:04:34.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c00c210 osd=5 3 2024-02-19T21:04:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c00c210 osd=5 3 2024-02-19T21:04:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 11 2024-02-19T21:04:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c00c210 osd=5 3 2024-02-19T21:04:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 5 11 2024-02-19T21:04:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c00c210 osd=5 4 2024-02-19T21:04:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a96130700 10 client.34940.objecter _send_command 11 2024-02-19T21:04:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.199+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a7c01e810 con 0x7f1a7c01ac10 2024-02-19T21:04:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.200+0000 7f1a8cff9700 10 client.34940.objecter ms_handle_connect 0x7f1a7c01ac10 2024-02-19T21:04:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.201+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1a7c01e810 con 0x7f1a7c01ac10 2024-02-19T21:04:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.201+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:04:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.201+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 11 = system:0 2024-02-19T21:04:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.201+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c00c210 osd=5 4 2024-02-19T21:04:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.201+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 5 11 2024-02-19T21:04:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.215+0000 7f1a96130700 10 client.34940.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:34.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.215+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 12 2024-02-19T21:04:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.215+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c00c210 osd=5 3 2024-02-19T21:04:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.215+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c00c210 osd=5 3 2024-02-19T21:04:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.215+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c00c210 osd=5 3 2024-02-19T21:04:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.215+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 12 2024-02-19T21:04:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.215+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c00c210 osd=5 3 2024-02-19T21:04:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.215+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 5 12 2024-02-19T21:04:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.215+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c00c210 osd=5 4 2024-02-19T21:04:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.215+0000 7f1a96130700 10 client.34940.objecter _send_command 12 2024-02-19T21:04:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.215+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1a7c01e810 con 0x7f1a7c01ac10 2024-02-19T21:04:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1a7c01e810 con 0x7f1a7c01ac10 2024-02-19T21:04:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c00c210 osd=5 4 2024-02-19T21:04:34.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 5 12 2024-02-19T21:04:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a96130700 10 client.34940.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 13 2024-02-19T21:04:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c01e810 osd=6 3 2024-02-19T21:04:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c01e810 osd=6 3 2024-02-19T21:04:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c01e810 osd=6 3 2024-02-19T21:04:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 13 2024-02-19T21:04:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c01e810 osd=6 3 2024-02-19T21:04:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 6 13 2024-02-19T21:04:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c01e810 osd=6 4 2024-02-19T21:04:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a96130700 10 client.34940.objecter _send_command 13 2024-02-19T21:04:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.217+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a7c01fa60 con 0x7f1a7c020510 2024-02-19T21:04:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.218+0000 7f1a8cff9700 10 client.34940.objecter ms_handle_connect 0x7f1a7c020510 2024-02-19T21:04:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.218+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1a7c01fa60 con 0x7f1a7c020510 2024-02-19T21:04:34.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.218+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.218+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 13 = system:0 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.218+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c01e810 osd=6 4 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.218+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 6 13 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.233+0000 7f1a96130700 10 client.34940.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.233+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 14 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.233+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c01e810 osd=6 3 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.233+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c01e810 osd=6 3 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.233+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c01e810 osd=6 3 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.233+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 14 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.233+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c01e810 osd=6 3 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.233+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 6 14 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.233+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c01e810 osd=6 4 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.233+0000 7f1a96130700 10 client.34940.objecter _send_command 14 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.233+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1a7c01fa60 con 0x7f1a7c020510 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1a7c01fa60 con 0x7f1a7c020510 2024-02-19T21:04:34.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:34.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c01e810 osd=6 4 2024-02-19T21:04:34.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 6 14 2024-02-19T21:04:34.235 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:04:34.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a96130700 10 client.34940.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:34.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 15 2024-02-19T21:04:34.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c01fa60 osd=7 3 2024-02-19T21:04:34.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c01fa60 osd=7 3 2024-02-19T21:04:34.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c01fa60 osd=7 3 2024-02-19T21:04:34.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 15 2024-02-19T21:04:34.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c01fa60 osd=7 3 2024-02-19T21:04:34.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 7 15 2024-02-19T21:04:34.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c01fa60 osd=7 4 2024-02-19T21:04:34.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a96130700 10 client.34940.objecter _send_command 15 2024-02-19T21:04:34.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a7c0271e0 con 0x7f1a7c0236d0 2024-02-19T21:04:34.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.235+0000 7f1a8cff9700 10 client.34940.objecter ms_handle_connect 0x7f1a7c0236d0 2024-02-19T21:04:34.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.236+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1a7c0271e0 con 0x7f1a7c0236d0 2024-02-19T21:04:34.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.236+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:04:34.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.236+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 15 = system:0 2024-02-19T21:04:34.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.236+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c01fa60 osd=7 4 2024-02-19T21:04:34.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.236+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 7 15 2024-02-19T21:04:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.251+0000 7f1a96130700 10 client.34940.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.251+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign -1 16 2024-02-19T21:04:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.251+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c01fa60 osd=7 3 2024-02-19T21:04:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.251+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c01fa60 osd=7 3 2024-02-19T21:04:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.251+0000 7f1a96130700 20 client.34940.objecter _get_session s=0x7f1a7c01fa60 osd=7 3 2024-02-19T21:04:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.251+0000 7f1a96130700 15 client.34940.objecter _session_command_op_remove -1 16 2024-02-19T21:04:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.251+0000 7f1a96130700 20 client.34940.objecter get_session s=0x7f1a7c01fa60 osd=7 3 2024-02-19T21:04:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.251+0000 7f1a96130700 15 client.34940.objecter _session_command_op_assign 7 16 2024-02-19T21:04:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.251+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c01fa60 osd=7 4 2024-02-19T21:04:34.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.251+0000 7f1a96130700 10 client.34940.objecter _send_command 16 2024-02-19T21:04:34.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.251+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1a7c0271e0 con 0x7f1a7c0236d0 2024-02-19T21:04:34.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a8cff9700 1 -- 172.21.15.163:0/2458544903 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1a7c0271e0 con 0x7f1a7c0236d0 2024-02-19T21:04:34.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a8cff9700 10 client.34940.objecter ms_dispatch 0x7f1a90001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:34.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a8cff9700 10 client.34940.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a8cff9700 20 client.34940.objecter put_session s=0x7f1a7c01fa60 osd=7 4 2024-02-19T21:04:34.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a8cff9700 15 client.34940.objecter _session_command_op_remove 7 16 2024-02-19T21:04:34.247 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:04:34.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 10 librados: watch_flush enter 2024-02-19T21:04:34.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 10 librados: watch_flush exit 2024-02-19T21:04:34.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 10 client.34940.objecter close_session for osd.0 2024-02-19T21:04:34.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 >> v1:172.21.15.163:6801/1559418115 conn(0x7f1a7c004530 legacy=0x7f1a7c002710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:34.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c0051a0 osd=0 1 2024-02-19T21:04:34.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 10 client.34940.objecter close_session for osd.1 2024-02-19T21:04:34.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 >> v1:172.21.15.163:6805/3125047432 conn(0x7f1a7c003b00 legacy=0x7f1a7c00a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:34.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c004230 osd=1 1 2024-02-19T21:04:34.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 10 client.34940.objecter close_session for osd.2 2024-02-19T21:04:34.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 >> v1:172.21.15.163:6809/1563158487 conn(0x7f1a7c002070 legacy=0x7f1a7c00f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:34.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c003360 osd=2 1 2024-02-19T21:04:34.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 10 client.34940.objecter close_session for osd.3 2024-02-19T21:04:34.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 >> v1:172.21.15.163:6813/2112279769 conn(0x7f1a7c010ec0 legacy=0x7f1a7c0112e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:34.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a8ffff700 1 -- 172.21.15.163:0/2458544903 reap_dead start 2024-02-19T21:04:34.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c00c870 osd=3 1 2024-02-19T21:04:34.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 10 client.34940.objecter close_session for osd.4 2024-02-19T21:04:34.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 >> v1:172.21.15.193:6800/3129716965 conn(0x7f1a7c016bf0 legacy=0x7f1a7c018fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:34.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a8ffff700 1 -- 172.21.15.163:0/2458544903 reap_dead start 2024-02-19T21:04:34.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c00c0a0 osd=4 1 2024-02-19T21:04:34.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 10 client.34940.objecter close_session for osd.5 2024-02-19T21:04:34.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 >> v1:172.21.15.193:6804/805161991 conn(0x7f1a7c01ac10 legacy=0x7f1a7c01d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:34.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c00c210 osd=5 1 2024-02-19T21:04:34.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 10 client.34940.objecter close_session for osd.6 2024-02-19T21:04:34.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 >> v1:172.21.15.193:6808/787604615 conn(0x7f1a7c020510 legacy=0x7f1a7c020970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:34.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c01e810 osd=6 1 2024-02-19T21:04:34.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 10 client.34940.objecter close_session for osd.7 2024-02-19T21:04:34.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 >> v1:172.21.15.193:6812/157927432 conn(0x7f1a7c0236d0 legacy=0x7f1a7c025ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:34.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 20 client.34940.objecter put_session s=0x7f1a7c01fa60 osd=7 1 2024-02-19T21:04:34.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 20 client.34940.objecter shutdown clearing up homeless session... 2024-02-19T21:04:34.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 10 client.34940.objecter successfully canceled tick 2024-02-19T21:04:34.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 >> v1:172.21.15.163:6800/1552514681 conn(0x7f1a78063810 legacy=0x7f1a78065cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:34.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 >> v1:172.21.15.163:6790/0 conn(0x7f1a90100770 legacy=0x7f1a90114d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:34.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a8ffff700 1 -- 172.21.15.163:0/2458544903 reap_dead start 2024-02-19T21:04:34.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 shutdown_connections 2024-02-19T21:04:34.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 >> 172.21.15.163:0/2458544903 conn(0x7f1a900f3590 msgr2=0x7f1a90107700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:34.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 shutdown_connections 2024-02-19T21:04:34.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 1 -- 172.21.15.163:0/2458544903 wait complete. 2024-02-19T21:04:34.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:34.253+0000 7f1a96130700 1 librados: shutdown 2024-02-19T21:04:34.251 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.251 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.251 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.251 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.251 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.251 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.251 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.251 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:04:34.256 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:34.284 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:35.162 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:35.735 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:35.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:35.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:35.840 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:35.873 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:35.894 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:35.896 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:35.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:35.998 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:36.027 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:36.477 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:36.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:36.582 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.437+0000 7f9e4b5df700 1 -- 172.21.15.163:0/1201913876 >> v1:172.21.15.163:6790/0 conn(0x7f9e440fc880 legacy=0x7f9e440fcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.437+0000 7f9e4b5df700 1 -- 172.21.15.163:0/1201913876 shutdown_connections 2024-02-19T21:04:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.437+0000 7f9e4b5df700 1 -- 172.21.15.163:0/1201913876 >> 172.21.15.163:0/1201913876 conn(0x7f9e440f24a0 msgr2=0x7f9e440f48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.438+0000 7f9e4b5df700 1 -- 172.21.15.163:0/1201913876 shutdown_connections 2024-02-19T21:04:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.438+0000 7f9e4b5df700 1 -- 172.21.15.163:0/1201913876 wait complete. 2024-02-19T21:04:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.438+0000 7f9e4b5df700 1 librados: starting msgr at 2024-02-19T21:04:36.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.438+0000 7f9e4b5df700 1 librados: starting objecter 2024-02-19T21:04:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.438+0000 7f9e4b5df700 1 Processor -- start 2024-02-19T21:04:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.438+0000 7f9e4b5df700 1 -- start start 2024-02-19T21:04:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.438+0000 7f9e4b5df700 1 librados: setting wanted keys 2024-02-19T21:04:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.438+0000 7f9e4b5df700 1 librados: calling monclient init 2024-02-19T21:04:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e4b5df700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9e44061a00 con 0x7f9e440fc880 2024-02-19T21:04:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e4b5df700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9e44118690 con 0x7f9e440ff680 2024-02-19T21:04:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e4b5df700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9e44119830 con 0x7f9e441031c0 2024-02-19T21:04:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e4937b700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f9e440fc880 0x7f9e44060e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:52458/0 (socket says 172.21.15.163:52458) 2024-02-19T21:04:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e4937b700 1 -- 172.21.15.163:0/459567243 learned_addr learned my addr 172.21.15.163:0/459567243 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:04:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 10 client.?.objecter ms_handle_connect 0x7f9e440fc880 2024-02-19T21:04:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 10 client.?.objecter ms_handle_connect 0x7f9e441031c0 2024-02-19T21:04:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:36.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 10 client.?.objecter ms_handle_connect 0x7f9e440ff680 2024-02-19T21:04:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285135174 0 0) 0x7f9e44119830 con 0x7f9e441031c0 2024-02-19T21:04:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e30003630 con 0x7f9e441031c0 2024-02-19T21:04:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433804538 0 0) 0x7f9e44118690 con 0x7f9e440ff680 2024-02-19T21:04:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e44119830 con 0x7f9e440ff680 2024-02-19T21:04:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3302807312 0 0) 0x7f9e30003630 con 0x7f9e441031c0 2024-02-19T21:04:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e44118690 con 0x7f9e441031c0 2024-02-19T21:04:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9e3c001c90 con 0x7f9e441031c0 2024-02-19T21:04:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1782287966 0 0) 0x7f9e44119830 con 0x7f9e440ff680 2024-02-19T21:04:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e30003630 con 0x7f9e440ff680 2024-02-19T21:04:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9e340017f0 con 0x7f9e440ff680 2024-02-19T21:04:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4102839052 0 0) 0x7f9e44118690 con 0x7f9e441031c0 2024-02-19T21:04:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 >> v1:172.21.15.193:6789/0 conn(0x7f9e440ff680 legacy=0x7f9e44112830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:36.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 >> v1:172.21.15.163:6789/0 conn(0x7f9e440fc880 legacy=0x7f9e44060e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4411aa30 con 0x7f9e441031c0 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.439+0000 7f9e4b5df700 1 -- 172.21.15.163:0/459567243 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e44119a20 con 0x7f9e441031c0 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.440+0000 7f9e4b5df700 10 client.34958.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.440+0000 7f9e4b5df700 1 -- 172.21.15.163:0/459567243 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e44119e90 con 0x7f9e441031c0 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.440+0000 7f9e4b5df700 1 librados: init done 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.440+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9e3c0013c0 con 0x7f9e441031c0 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.440+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9e3c003540 con 0x7f9e441031c0 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.440+0000 7f9e4b5df700 1 -- 172.21.15.163:0/459567243 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e28004f20 con 0x7f9e441031c0 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.441+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9e3c0029c0 con 0x7f9e441031c0 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.441+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(98..98 src has 43..98) v4 ==== 6943+0+0 (unknown 1756307953 0 0) 0x7f9e3c076360 con 0x7f9e441031c0 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.441+0000 7f9e3a7fc700 10 client.34958.objecter ms_dispatch 0x7f9e44001550 osd_map(98..98 src has 43..98) v4 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.441+0000 7f9e3a7fc700 3 client.34958.objecter handle_osd_map got epochs [98,98] > 0 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.441+0000 7f9e3a7fc700 3 client.34958.objecter handle_osd_map decoding full epoch 98 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.441+0000 7f9e3a7fc700 20 client.34958.objecter dump_active .. 0 homeless 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.441+0000 7f9e3a7fc700 10 client.34958.objecter ms_handle_connect 0x7f9e30063c50 2024-02-19T21:04:36.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.446+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9e3c049bb0 con 0x7f9e441031c0 2024-02-19T21:04:36.612 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:36.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:36.627+0000 7f9e4b5df700 1 -- 172.21.15.163:0/459567243 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"} v 0) v1 -- 0x7f9e28005750 con 0x7f9e441031c0 2024-02-19T21:04:37.063 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:37.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:37.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:37.167 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:04:37.199 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:37.221 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:04:37.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:37.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:04:37.325 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:37.353 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:37.406 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:37.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:37.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:37.512 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:04:37.519 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:37.550 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:37.571 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:37.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:37.674 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:04:37.704 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:37.724 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:04:37.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:37.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:04:37.814 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:04:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:04:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:04:37.827 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:04:37.855 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:37.875 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:04:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:37.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:04:37.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.641+0000 7f2bac9a1700 1 -- 172.21.15.163:0/2400575721 >> v1:172.21.15.163:6790/0 conn(0x7f2ba8103280 legacy=0x7f2ba8105740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:37.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.641+0000 7f2bac9a1700 1 -- 172.21.15.163:0/2400575721 shutdown_connections 2024-02-19T21:04:37.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.641+0000 7f2bac9a1700 1 -- 172.21.15.163:0/2400575721 >> 172.21.15.163:0/2400575721 conn(0x7f2ba806bff0 msgr2=0x7f2ba806c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:37.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.641+0000 7f2bac9a1700 1 -- 172.21.15.163:0/2400575721 shutdown_connections 2024-02-19T21:04:37.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.641+0000 7f2bac9a1700 1 -- 172.21.15.163:0/2400575721 wait complete. 2024-02-19T21:04:37.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.641+0000 7f2bac9a1700 1 librados: starting msgr at 2024-02-19T21:04:37.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.641+0000 7f2bac9a1700 1 librados: starting objecter 2024-02-19T21:04:37.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2bac9a1700 1 Processor -- start 2024-02-19T21:04:37.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2bac9a1700 1 -- start start 2024-02-19T21:04:37.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2bac9a1700 1 librados: setting wanted keys 2024-02-19T21:04:37.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2bac9a1700 1 librados: calling monclient init 2024-02-19T21:04:37.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2bac9a1700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ba8119720 con 0x7f2ba80fc8c0 2024-02-19T21:04:37.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2bac9a1700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ba811a920 con 0x7f2ba8103280 2024-02-19T21:04:37.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2bac9a1700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ba811bb20 con 0x7f2ba80ff680 2024-02-19T21:04:37.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2ba5d9b700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f2ba80ff680 0x7f2ba810afe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:40486/0 (socket says 172.21.15.163:40486) 2024-02-19T21:04:37.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2ba5d9b700 1 -- 172.21.15.163:0/1210592304 learned_addr learned my addr 172.21.15.163:0/1210592304 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:04:37.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2b977fe700 10 client.?.objecter ms_handle_connect 0x7f2ba80ff680 2024-02-19T21:04:37.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2b977fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:37.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238623539 0 0) 0x7f2ba811bb20 con 0x7f2ba80ff680 2024-02-19T21:04:37.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b90003630 con 0x7f2ba80ff680 2024-02-19T21:04:37.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2b977fe700 10 client.?.objecter ms_handle_connect 0x7f2ba8103280 2024-02-19T21:04:37.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2b977fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:37.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2b977fe700 10 client.?.objecter ms_handle_connect 0x7f2ba80fc8c0 2024-02-19T21:04:37.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2b977fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:37.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247713580 0 0) 0x7f2ba811a920 con 0x7f2ba8103280 2024-02-19T21:04:37.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba811bb20 con 0x7f2ba8103280 2024-02-19T21:04:37.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.642+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4010904954 0 0) 0x7f2b90003630 con 0x7f2ba80ff680 2024-02-19T21:04:37.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.643+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ba811a920 con 0x7f2ba80ff680 2024-02-19T21:04:37.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.643+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2b9c0018c0 con 0x7f2ba80ff680 2024-02-19T21:04:37.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.643+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3073274312 0 0) 0x7f2ba811bb20 con 0x7f2ba8103280 2024-02-19T21:04:37.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.643+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b90003630 con 0x7f2ba8103280 2024-02-19T21:04:37.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.643+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2ba0002240 con 0x7f2ba8103280 2024-02-19T21:04:37.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.643+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2822321248 0 0) 0x7f2ba811a920 con 0x7f2ba80ff680 2024-02-19T21:04:37.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.643+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 >> v1:172.21.15.193:6789/0 conn(0x7f2ba8103280 legacy=0x7f2ba8117f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:37.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.643+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 >> v1:172.21.15.163:6789/0 conn(0x7f2ba80fc8c0 legacy=0x7f2ba810a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:37.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.643+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba811cd20 con 0x7f2ba80ff680 2024-02-19T21:04:37.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.643+0000 7f2bac9a1700 1 -- 172.21.15.163:0/1210592304 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba811ab10 con 0x7f2ba80ff680 2024-02-19T21:04:37.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.643+0000 7f2bac9a1700 10 client.34964.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:37.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.643+0000 7f2bac9a1700 1 -- 172.21.15.163:0/1210592304 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba811af80 con 0x7f2ba80ff680 2024-02-19T21:04:37.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.643+0000 7f2bac9a1700 1 librados: init done 2024-02-19T21:04:37.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.643+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f2b9c001c70 con 0x7f2ba80ff680 2024-02-19T21:04:37.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.643+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2b9c005b70 con 0x7f2ba80ff680 2024-02-19T21:04:37.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.644+0000 7f2bac9a1700 1 -- 172.21.15.163:0/1210592304 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b88004f20 con 0x7f2ba80ff680 2024-02-19T21:04:37.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.644+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f2b9c018a20 con 0x7f2ba80ff680 2024-02-19T21:04:37.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.645+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(98..98 src has 43..98) v4 ==== 6943+0+0 (unknown 1756307953 0 0) 0x7f2b9c078890 con 0x7f2ba80ff680 2024-02-19T21:04:37.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.645+0000 7f2b977fe700 10 client.34964.objecter ms_dispatch 0x7f2ba8001550 osd_map(98..98 src has 43..98) v4 2024-02-19T21:04:37.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.645+0000 7f2b977fe700 3 client.34964.objecter handle_osd_map got epochs [98,98] > 0 2024-02-19T21:04:37.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.645+0000 7f2b977fe700 3 client.34964.objecter handle_osd_map decoding full epoch 98 2024-02-19T21:04:37.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.645+0000 7f2b977fe700 20 client.34964.objecter dump_active .. 0 homeless 2024-02-19T21:04:37.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.645+0000 7f2b977fe700 10 client.34964.objecter ms_handle_connect 0x7f2b90063b70 2024-02-19T21:04:37.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.649+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f2b9c04c140 con 0x7f2ba80ff680 2024-02-19T21:04:37.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:37.831+0000 7f2bac9a1700 1 -- 172.21.15.163:0/1210592304 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f2b880051d0 con 0x7f2ba80ff680 2024-02-19T21:04:37.978 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:38.013 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:38.225 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:38.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:38.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:38.329 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:04:38.356 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:38.375 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:04:38.377 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:38.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:04:38.458 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:38.478 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:38.497 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:04:38.497 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:04:38.506 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:38.536 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:38.539 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:38.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:38.641 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:38.669 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:38.722 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:38.725 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:38.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:38.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:32.920488+0000 mon.b (mon.1) 86 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:38.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:32.950122+0000 mon.b (mon.1) 87 : audit [DBG] from='client.? v1:172.21.15.163:0/1298291889' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-02-19T21:04:38.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:33.187457+0000 mon.a (mon.0) 749 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:38.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:32.920488+0000 mon.b (mon.1) 86 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:38.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:32.950122+0000 mon.b (mon.1) 87 : audit [DBG] from='client.? v1:172.21.15.163:0/1298291889' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-02-19T21:04:38.755 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:33.187457+0000 mon.a (mon.0) 749 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:38.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:32.920488+0000 mon.b (mon.1) 86 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:38.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:32.950122+0000 mon.b (mon.1) 87 : audit [DBG] from='client.? v1:172.21.15.163:0/1298291889' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-02-19T21:04:38.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:33.187457+0000 mon.a (mon.0) 749 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:38.827 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:38.854 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:39.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.533+0000 7f9e3a7fc700 1 -- 172.21.15.163:0/459567243 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]=0 set pool 3 pgp_num to 3 v100) v1 ==== 130+0+0 (unknown 1106698541 0 0) 0x7f9e3c047fc0 con 0x7f9e441031c0 2024-02-19T21:04:39.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.534+0000 7f2b977fe700 1 -- 172.21.15.163:0/1210592304 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v100) v1 ==== 97+0+0 (unknown 4135192107 0 0) 0x7f2b9c04a550 con 0x7f2ba80ff680 2024-02-19T21:04:39.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.538+0000 7f2bac9a1700 10 librados: watch_flush enter 2024-02-19T21:04:39.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.538+0000 7f2bac9a1700 10 librados: watch_flush exit 2024-02-19T21:04:39.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.538+0000 7f2bac9a1700 20 client.34964.objecter shutdown clearing up homeless session... 2024-02-19T21:04:39.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.538+0000 7f2bac9a1700 10 client.34964.objecter successfully canceled tick 2024-02-19T21:04:39.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.539+0000 7f9e4b5df700 10 librados: watch_flush enter 2024-02-19T21:04:39.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.539+0000 7f2bac9a1700 1 -- 172.21.15.163:0/1210592304 >> v1:172.21.15.163:6800/1552514681 conn(0x7f2b90063b70 legacy=0x7f2b90066030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:39.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.540+0000 7f2bac9a1700 1 -- 172.21.15.163:0/1210592304 >> v1:172.21.15.163:6790/0 conn(0x7f2ba80ff680 legacy=0x7f2ba810afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:39.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.540+0000 7f2bac9a1700 1 -- 172.21.15.163:0/1210592304 shutdown_connections 2024-02-19T21:04:39.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.540+0000 7f2bac9a1700 1 -- 172.21.15.163:0/1210592304 >> 172.21.15.163:0/1210592304 conn(0x7f2ba806bff0 msgr2=0x7f2ba8105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:39.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.541+0000 7f2bac9a1700 1 -- 172.21.15.163:0/1210592304 shutdown_connections 2024-02-19T21:04:39.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.541+0000 7f2bac9a1700 1 -- 172.21.15.163:0/1210592304 wait complete. 2024-02-19T21:04:39.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.541+0000 7f2bac9a1700 1 librados: shutdown 2024-02-19T21:04:39.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.545+0000 7f9e4b5df700 10 librados: watch_flush exit 2024-02-19T21:04:39.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.545+0000 7f9e4b5df700 20 client.34958.objecter shutdown clearing up homeless session... 2024-02-19T21:04:39.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.545+0000 7f9e4b5df700 10 client.34958.objecter successfully canceled tick 2024-02-19T21:04:39.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.546+0000 7f9e4b5df700 1 -- 172.21.15.163:0/459567243 >> v1:172.21.15.163:6800/1552514681 conn(0x7f9e30063c50 legacy=0x7f9e30066110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:39.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.546+0000 7f9e4b5df700 1 -- 172.21.15.163:0/459567243 >> v1:172.21.15.163:6790/0 conn(0x7f9e441031c0 legacy=0x7f9e44115f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:39.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.549+0000 7f9e4b5df700 1 -- 172.21.15.163:0/459567243 shutdown_connections 2024-02-19T21:04:39.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.549+0000 7f9e4b5df700 1 -- 172.21.15.163:0/459567243 >> 172.21.15.163:0/459567243 conn(0x7f9e440f24a0 msgr2=0x7f9e44106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:39.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.559+0000 7f9e4b5df700 1 -- 172.21.15.163:0/459567243 shutdown_connections 2024-02-19T21:04:39.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.559+0000 7f9e4b5df700 1 -- 172.21.15.163:0/459567243 wait complete. 2024-02-19T21:04:39.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:39.560+0000 7f9e4b5df700 1 librados: shutdown 2024-02-19T21:04:39.563 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:04:39.569 INFO:teuthology.orchestra.run.smithi163.stderr:set pool 3 pgp_num to 3 2024-02-19T21:04:39.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:32.963390+0000 mgr.y (mgr.14550) 134 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-02-19T21:04:39.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:33.188433+0000 mon.b (mon.1) 88 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:39.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:33.428572+0000 mon.a (mon.0) 750 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:39.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:33.439371+0000 mon.c (mon.2) 31 : audit [INF] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi163-83888-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:34.108291+0000 mon.c (mon.2) 32 : audit [DBG] from='client.? v1:172.21.15.163:0/2458544903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:04:39.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:34.964043+0000 mgr.y (mgr.14550) 135 : cluster [DBG] pgmap v117: 196 pgs: 42 creating+peering, 132 active+clean, 22 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-02-19T21:04:39.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:35.069544+0000 osd.6 (osd.6) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:35.069600+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:36.494971+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:36.495034+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:36.629976+0000 mon.c (mon.2) 33 : audit [INF] from='client.? v1:172.21.15.163:0/459567243' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-19T21:04:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:36.964437+0000 mgr.y (mgr.14550) 136 : cluster [DBG] pgmap v118: 196 pgs: 42 creating+peering, 132 active+clean, 22 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-02-19T21:04:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:37.833296+0000 mon.c (mon.2) 34 : audit [INF] from='client.? v1:172.21.15.163:0/1210592304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.208935+0000 mon.b (mon.1) 89 : audit [DBG] from='client.? v1:172.21.15.163:0/1153118870' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-19T21:04:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:38.494160+0000 mon.a (mon.0) 751 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-19T21:04:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.508610+0000 mon.a (mon.0) 752 : audit [INF] from='client.? v1:172.21.15.163:0/1288380247' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi163-83960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.508926+0000 mon.a (mon.0) 753 : audit [INF] from='client.34832 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi163-83888-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.509555+0000 mon.a (mon.0) 754 : audit [INF] from='client.34958 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-19T21:04:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.518903+0000 mon.b (mon.1) 90 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.521109+0000 mon.b (mon.1) 91 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.521341+0000 mon.b (mon.1) 92 : audit [INF] from='client.? v1:172.21.15.163:0/3705993251' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi163-83976-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.521508+0000 mon.b (mon.1) 93 : audit [INF] from='client.? v1:172.21.15.163:0/1366918899' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi163-83904-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.521711+0000 mon.b (mon.1) 94 : audit [INF] from='client.? v1:172.21.15.163:0/3187889786' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi163-83952-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.521930+0000 mon.b (mon.1) 95 : audit [INF] from='client.? v1:172.21.15.163:0/3083004620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi163-83845-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.522082+0000 mon.b (mon.1) 96 : audit [INF] from='client.? v1:172.21.15.163:0/1259005871' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi163-83928-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.522158+0000 mon.c (mon.2) 35 : audit [INF] from='client.? v1:172.21.15.163:0/1026582766' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.522249+0000 mon.b (mon.1) 97 : audit [INF] from='client.? v1:172.21.15.163:0/3832265225' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.522427+0000 mon.c (mon.2) 36 : audit [INF] from='client.? v1:172.21.15.163:0/2649387006' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.522576+0000 mon.c (mon.2) 37 : audit [INF] from='client.? v1:172.21.15.163:0/425122647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84000-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.522734+0000 mon.c (mon.2) 38 : audit [INF] from='client.? v1:172.21.15.163:0/725453317' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi163-83872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.523167+0000 mon.b (mon.1) 98 : audit [INF] from='client.? v1:172.21.15.163:0/1025237974' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi163-83896-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.523403+0000 mon.b (mon.1) 99 : audit [INF] from='client.? v1:172.21.15.163:0/3327823022' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi163-83968-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.524190+0000 mon.c (mon.2) 39 : audit [INF] from='client.? v1:172.21.15.163:0/134596152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi163-84032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.526048+0000 mon.b (mon.1) 100 : audit [INF] from='client.? v1:172.21.15.163:0/3326199758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi163-83944-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.526272+0000 mon.b (mon.1) 101 : audit [INF] from='client.? v1:172.21.15.163:0/667290477' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi163-83854-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.526577+0000 mon.c (mon.2) 40 : audit [INF] from='client.? v1:172.21.15.163:0/191688483' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi163-83864-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:38.768371+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-02-19T21:04:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.770329+0000 mon.a (mon.0) 756 : audit [INF] from='client.34964 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.770640+0000 mon.a (mon.0) 757 : audit [INF] from='client.34868 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.771831+0000 mon.a (mon.0) 758 : audit [INF] from='client.34931 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.772168+0000 mon.a (mon.0) 759 : audit [INF] from='client.34934 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84000-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.772616+0000 mon.a (mon.0) 760 : audit [INF] from='client.34889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi163-83872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.773124+0000 mon.a (mon.0) 761 : audit [INF] from='client.34916 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi163-84032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.773516+0000 mon.a (mon.0) 762 : audit [INF] from='client.34874 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi163-83864-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.775258+0000 mon.a (mon.0) 763 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.775605+0000 mon.a (mon.0) 764 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.775836+0000 mon.a (mon.0) 765 : audit [INF] from='client.24922 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi163-83976-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.776057+0000 mon.a (mon.0) 766 : audit [INF] from='client.24838 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi163-83904-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.776224+0000 mon.a (mon.0) 767 : audit [INF] from='client.24865 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi163-83952-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.776444+0000 mon.a (mon.0) 768 : audit [INF] from='client.24913 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi163-83845-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.776690+0000 mon.a (mon.0) 769 : audit [INF] from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi163-83928-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:32.963390+0000 mgr.y (mgr.14550) 134 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-02-19T21:04:39.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:33.188433+0000 mon.b (mon.1) 88 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:39.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:33.428572+0000 mon.a (mon.0) 750 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:39.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:33.439371+0000 mon.c (mon.2) 31 : audit [INF] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi163-83888-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:34.108291+0000 mon.c (mon.2) 32 : audit [DBG] from='client.? v1:172.21.15.163:0/2458544903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:04:39.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:34.964043+0000 mgr.y (mgr.14550) 135 : cluster [DBG] pgmap v117: 196 pgs: 42 creating+peering, 132 active+clean, 22 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-02-19T21:04:39.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:35.069544+0000 osd.6 (osd.6) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:39.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:35.069600+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:39.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:36.494971+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:36.495034+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:36.629976+0000 mon.c (mon.2) 33 : audit [INF] from='client.? v1:172.21.15.163:0/459567243' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-19T21:04:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:36.964437+0000 mgr.y (mgr.14550) 136 : cluster [DBG] pgmap v118: 196 pgs: 42 creating+peering, 132 active+clean, 22 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-02-19T21:04:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:37.833296+0000 mon.c (mon.2) 34 : audit [INF] from='client.? v1:172.21.15.163:0/1210592304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.208935+0000 mon.b (mon.1) 89 : audit [DBG] from='client.? v1:172.21.15.163:0/1153118870' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-19T21:04:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:38.494160+0000 mon.a (mon.0) 751 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-19T21:04:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.508610+0000 mon.a (mon.0) 752 : audit [INF] from='client.? v1:172.21.15.163:0/1288380247' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi163-83960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.508926+0000 mon.a (mon.0) 753 : audit [INF] from='client.34832 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi163-83888-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.509555+0000 mon.a (mon.0) 754 : audit [INF] from='client.34958 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-19T21:04:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.518903+0000 mon.b (mon.1) 90 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.521109+0000 mon.b (mon.1) 91 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.521341+0000 mon.b (mon.1) 92 : audit [INF] from='client.? v1:172.21.15.163:0/3705993251' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi163-83976-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.521508+0000 mon.b (mon.1) 93 : audit [INF] from='client.? v1:172.21.15.163:0/1366918899' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi163-83904-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.521711+0000 mon.b (mon.1) 94 : audit [INF] from='client.? v1:172.21.15.163:0/3187889786' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi163-83952-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.521930+0000 mon.b (mon.1) 95 : audit [INF] from='client.? v1:172.21.15.163:0/3083004620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi163-83845-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.522082+0000 mon.b (mon.1) 96 : audit [INF] from='client.? v1:172.21.15.163:0/1259005871' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi163-83928-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.522158+0000 mon.c (mon.2) 35 : audit [INF] from='client.? v1:172.21.15.163:0/1026582766' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.522249+0000 mon.b (mon.1) 97 : audit [INF] from='client.? v1:172.21.15.163:0/3832265225' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.776873+0000 mon.a (mon.0) 770 : audit [INF] from='client.24808 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.777054+0000 mon.a (mon.0) 771 : audit [INF] from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi163-83896-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.777318+0000 mon.a (mon.0) 772 : audit [INF] from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi163-83968-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.777564+0000 mon.a (mon.0) 773 : audit [INF] from='client.24895 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi163-83944-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:38.777753+0000 mon.a (mon.0) 774 : audit [INF] from='client.24898 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi163-83854-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.522427+0000 mon.c (mon.2) 36 : audit [INF] from='client.? v1:172.21.15.163:0/2649387006' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.522576+0000 mon.c (mon.2) 37 : audit [INF] from='client.? v1:172.21.15.163:0/425122647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84000-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.522734+0000 mon.c (mon.2) 38 : audit [INF] from='client.? v1:172.21.15.163:0/725453317' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi163-83872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.523167+0000 mon.b (mon.1) 98 : audit [INF] from='client.? v1:172.21.15.163:0/1025237974' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi163-83896-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.523403+0000 mon.b (mon.1) 99 : audit [INF] from='client.? v1:172.21.15.163:0/3327823022' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi163-83968-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.524190+0000 mon.c (mon.2) 39 : audit [INF] from='client.? v1:172.21.15.163:0/134596152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi163-84032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.526048+0000 mon.b (mon.1) 100 : audit [INF] from='client.? v1:172.21.15.163:0/3326199758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi163-83944-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.526272+0000 mon.b (mon.1) 101 : audit [INF] from='client.? v1:172.21.15.163:0/667290477' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi163-83854-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.526577+0000 mon.c (mon.2) 40 : audit [INF] from='client.? v1:172.21.15.163:0/191688483' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi163-83864-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:38.768371+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-02-19T21:04:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.770329+0000 mon.a (mon.0) 756 : audit [INF] from='client.34964 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.770640+0000 mon.a (mon.0) 757 : audit [INF] from='client.34868 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.771831+0000 mon.a (mon.0) 758 : audit [INF] from='client.34931 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.772168+0000 mon.a (mon.0) 759 : audit [INF] from='client.34934 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84000-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.772616+0000 mon.a (mon.0) 760 : audit [INF] from='client.34889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi163-83872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.773124+0000 mon.a (mon.0) 761 : audit [INF] from='client.34916 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi163-84032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.773516+0000 mon.a (mon.0) 762 : audit [INF] from='client.34874 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi163-83864-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.775258+0000 mon.a (mon.0) 763 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.775605+0000 mon.a (mon.0) 764 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.775836+0000 mon.a (mon.0) 765 : audit [INF] from='client.24922 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi163-83976-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.776057+0000 mon.a (mon.0) 766 : audit [INF] from='client.24838 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi163-83904-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.776224+0000 mon.a (mon.0) 767 : audit [INF] from='client.24865 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi163-83952-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.776444+0000 mon.a (mon.0) 768 : audit [INF] from='client.24913 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi163-83845-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.776690+0000 mon.a (mon.0) 769 : audit [INF] from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi163-83928-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.776873+0000 mon.a (mon.0) 770 : audit [INF] from='client.24808 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.777054+0000 mon.a (mon.0) 771 : audit [INF] from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi163-83896-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.777318+0000 mon.a (mon.0) 772 : audit [INF] from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi163-83968-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.777564+0000 mon.a (mon.0) 773 : audit [INF] from='client.24895 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi163-83944-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:38.777753+0000 mon.a (mon.0) 774 : audit [INF] from='client.24898 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi163-83854-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:32.963390+0000 mgr.y (mgr.14550) 134 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-02-19T21:04:39.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:33.188433+0000 mon.b (mon.1) 88 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:39.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:33.428572+0000 mon.a (mon.0) 750 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:33.439371+0000 mon.c (mon.2) 31 : audit [INF] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi163-83888-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:34.108291+0000 mon.c (mon.2) 32 : audit [DBG] from='client.? v1:172.21.15.163:0/2458544903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:04:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:34.964043+0000 mgr.y (mgr.14550) 135 : cluster [DBG] pgmap v117: 196 pgs: 42 creating+peering, 132 active+clean, 22 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-02-19T21:04:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:35.069544+0000 osd.6 (osd.6) 51 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:35.069600+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:36.494971+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:36.495034+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:36.629976+0000 mon.c (mon.2) 33 : audit [INF] from='client.? v1:172.21.15.163:0/459567243' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-19T21:04:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:36.964437+0000 mgr.y (mgr.14550) 136 : cluster [DBG] pgmap v118: 196 pgs: 42 creating+peering, 132 active+clean, 22 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-02-19T21:04:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:37.833296+0000 mon.c (mon.2) 34 : audit [INF] from='client.? v1:172.21.15.163:0/1210592304' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.208935+0000 mon.b (mon.1) 89 : audit [DBG] from='client.? v1:172.21.15.163:0/1153118870' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-19T21:04:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:38.494160+0000 mon.a (mon.0) 751 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-02-19T21:04:39.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.508610+0000 mon.a (mon.0) 752 : audit [INF] from='client.? v1:172.21.15.163:0/1288380247' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi163-83960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.508926+0000 mon.a (mon.0) 753 : audit [INF] from='client.34832 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi163-83888-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.509555+0000 mon.a (mon.0) 754 : audit [INF] from='client.34958 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]: dispatch 2024-02-19T21:04:39.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.518903+0000 mon.b (mon.1) 90 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:39.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.521109+0000 mon.b (mon.1) 91 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.521341+0000 mon.b (mon.1) 92 : audit [INF] from='client.? v1:172.21.15.163:0/3705993251' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi163-83976-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.521508+0000 mon.b (mon.1) 93 : audit [INF] from='client.? v1:172.21.15.163:0/1366918899' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi163-83904-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.521711+0000 mon.b (mon.1) 94 : audit [INF] from='client.? v1:172.21.15.163:0/3187889786' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi163-83952-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.521930+0000 mon.b (mon.1) 95 : audit [INF] from='client.? v1:172.21.15.163:0/3083004620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi163-83845-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.522082+0000 mon.b (mon.1) 96 : audit [INF] from='client.? v1:172.21.15.163:0/1259005871' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi163-83928-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.522158+0000 mon.c (mon.2) 35 : audit [INF] from='client.? v1:172.21.15.163:0/1026582766' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.522249+0000 mon.b (mon.1) 97 : audit [INF] from='client.? v1:172.21.15.163:0/3832265225' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.522427+0000 mon.c (mon.2) 36 : audit [INF] from='client.? v1:172.21.15.163:0/2649387006' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.522576+0000 mon.c (mon.2) 37 : audit [INF] from='client.? v1:172.21.15.163:0/425122647' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84000-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.522734+0000 mon.c (mon.2) 38 : audit [INF] from='client.? v1:172.21.15.163:0/725453317' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi163-83872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.523167+0000 mon.b (mon.1) 98 : audit [INF] from='client.? v1:172.21.15.163:0/1025237974' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi163-83896-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.523403+0000 mon.b (mon.1) 99 : audit [INF] from='client.? v1:172.21.15.163:0/3327823022' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi163-83968-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.524190+0000 mon.c (mon.2) 39 : audit [INF] from='client.? v1:172.21.15.163:0/134596152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi163-84032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.526048+0000 mon.b (mon.1) 100 : audit [INF] from='client.? v1:172.21.15.163:0/3326199758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi163-83944-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.526272+0000 mon.b (mon.1) 101 : audit [INF] from='client.? v1:172.21.15.163:0/667290477' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi163-83854-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.526577+0000 mon.c (mon.2) 40 : audit [INF] from='client.? v1:172.21.15.163:0/191688483' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi163-83864-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:38.768371+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-02-19T21:04:39.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.770329+0000 mon.a (mon.0) 756 : audit [INF] from='client.34964 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:39.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.770640+0000 mon.a (mon.0) 757 : audit [INF] from='client.34868 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.771831+0000 mon.a (mon.0) 758 : audit [INF] from='client.34931 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.772168+0000 mon.a (mon.0) 759 : audit [INF] from='client.34934 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84000-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.772616+0000 mon.a (mon.0) 760 : audit [INF] from='client.34889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi163-83872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.773124+0000 mon.a (mon.0) 761 : audit [INF] from='client.34916 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi163-84032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.773516+0000 mon.a (mon.0) 762 : audit [INF] from='client.34874 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi163-83864-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.775258+0000 mon.a (mon.0) 763 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.775605+0000 mon.a (mon.0) 764 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.775836+0000 mon.a (mon.0) 765 : audit [INF] from='client.24922 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi163-83976-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.776057+0000 mon.a (mon.0) 766 : audit [INF] from='client.24838 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi163-83904-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.776224+0000 mon.a (mon.0) 767 : audit [INF] from='client.24865 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi163-83952-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.776444+0000 mon.a (mon.0) 768 : audit [INF] from='client.24913 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi163-83845-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.776690+0000 mon.a (mon.0) 769 : audit [INF] from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi163-83928-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.776873+0000 mon.a (mon.0) 770 : audit [INF] from='client.24808 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.777054+0000 mon.a (mon.0) 771 : audit [INF] from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi163-83896-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.777318+0000 mon.a (mon.0) 772 : audit [INF] from='client.24889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi163-83968-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.777564+0000 mon.a (mon.0) 773 : audit [INF] from='client.24895 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi163-83944-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:38.777753+0000 mon.a (mon.0) 774 : audit [INF] from='client.24898 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi163-83854-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:39.855 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-02-19T21:04:39.855 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [----------] Global test environment set-up. 2024-02-19T21:04:39.855 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [----------] 12 tests from AsioRados 2024-02-19T21:04:39.855 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-02-19T21:04:39.855 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (2 ms) 2024-02-19T21:04:39.855 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-02-19T21:04:39.855 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-02-19T21:04:39.855 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-02-19T21:04:39.855 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (2 ms) 2024-02-19T21:04:39.855 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-02-19T21:04:39.856 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (6 ms) 2024-02-19T21:04:39.856 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-02-19T21:04:39.856 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (3 ms) 2024-02-19T21:04:39.856 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-02-19T21:04:39.856 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (3 ms) 2024-02-19T21:04:39.856 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-02-19T21:04:39.856 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-02-19T21:04:39.856 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-02-19T21:04:39.856 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (0 ms) 2024-02-19T21:04:39.856 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-02-19T21:04:39.856 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-02-19T21:04:39.856 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-02-19T21:04:39.856 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (2 ms) 2024-02-19T21:04:39.856 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-02-19T21:04:39.856 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (5 ms) 2024-02-19T21:04:39.857 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-02-19T21:04:39.857 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (6 ms) 2024-02-19T21:04:39.857 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [----------] 12 tests from AsioRados (32 ms total) 2024-02-19T21:04:39.857 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: 2024-02-19T21:04:39.857 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [----------] Global test environment tear-down 2024-02-19T21:04:39.857 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (6941 ms total) 2024-02-19T21:04:39.857 INFO:tasks.workunit.client.0.smithi163.stdout: api_asio: [ PASSED ] 12 tests. 2024-02-19T21:04:40.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-02-19T21:04:40.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [----------] Global test environment set-up. 2024-02-19T21:04:40.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [----------] 7 tests from LibRadosList 2024-02-19T21:04:40.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-02-19T21:04:40.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ OK ] LibRadosList.ListObjects (345 ms) 2024-02-19T21:04:40.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-02-19T21:04:40.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (40 ms) 2024-02-19T21:04:40.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-02-19T21:04:40.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: myset foo1,foo2,foo3 2024-02-19T21:04:40.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo1 2024-02-19T21:04:40.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo2 2024-02-19T21:04:40.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo3 2024-02-19T21:04:40.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: myset foo1,foo4,foo5 2024-02-19T21:04:40.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo4 2024-02-19T21:04:40.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo5 2024-02-19T21:04:40.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo1 2024-02-19T21:04:40.197 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: myset foo6,foo7 2024-02-19T21:04:40.197 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo7 2024-02-19T21:04:40.197 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo6 2024-02-19T21:04:40.197 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-02-19T21:04:40.197 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: ns1:foo4 2024-02-19T21:04:40.197 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: ns1:foo5 2024-02-19T21:04:40.197 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: ns2:foo7 2024-02-19T21:04:40.197 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: ns2:foo6 2024-02-19T21:04:40.197 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: ns1:foo1 2024-02-19T21:04:40.197 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: :foo1 2024-02-19T21:04:40.197 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: :foo2 2024-02-19T21:04:40.197 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: :foo3 2024-02-19T21:04:40.197 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (98 ms) 2024-02-19T21:04:40.197 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-02-19T21:04:40.197 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 1 0 2024-02-19T21:04:40.198 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 10 0 2024-02-19T21:04:40.198 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 13 0 2024-02-19T21:04:40.198 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 7 0 2024-02-19T21:04:40.198 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 14 0 2024-02-19T21:04:40.198 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 0 0 2024-02-19T21:04:40.198 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 15 0 2024-02-19T21:04:40.198 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 11 0 2024-02-19T21:04:40.198 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 5 0 2024-02-19T21:04:40.198 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 8 0 2024-02-19T21:04:40.198 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 6 0 2024-02-19T21:04:40.198 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 3 0 2024-02-19T21:04:40.198 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 4 0 2024-02-19T21:04:40.198 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 12 0 2024-02-19T21:04:40.198 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 9 0 2024-02-19T21:04:40.198 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 2 0 2024-02-19T21:04:40.199 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-02-19T21:04:40.199 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (80 ms) 2024-02-19T21:04:40.199 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-02-19T21:04:40.199 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: x cursor=MIN 2024-02-19T21:04:40.199 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=1 cursor=8:02547ec2:::1:head 2024-02-19T21:04:40.199 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=10 cursor=8:52ea6a34:::10:head 2024-02-19T21:04:40.199 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=13 cursor=8:566253c9:::13:head 2024-02-19T21:04:40.199 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=7 cursor=8:5c6b0b28:::7:head 2024-02-19T21:04:40.199 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=14 cursor=8:62a1935d:::14:head 2024-02-19T21:04:40.199 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=0 cursor=8:6cac518f:::0:head 2024-02-19T21:04:40.199 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=15 cursor=8:863748b0:::15:head 2024-02-19T21:04:40.199 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=11 cursor=8:89d3ae78:::11:head 2024-02-19T21:04:40.199 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=5 cursor=8:b29083e3:::5:head 2024-02-19T21:04:40.199 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=8 cursor=8:bd63b0f1:::8:head 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=6 cursor=8:c4fdafeb:::6:head 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=3 cursor=8:cfc208b3:::3:head 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=4 cursor=8:d83876eb:::4:head 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=12 cursor=8:de5d7c5f:::12:head 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=9 cursor=8:e960b815:::9:head 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > oid=2 cursor=8:f905c69b:::2:head 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: FIRST> seek to MIN oid=1 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=1 cursor=8:02547ec2:::1:head 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=10 cursor=8:52ea6a34:::10:head 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=13 cursor=8:566253c9:::13:head 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:566253c9:::13:head 2024-02-19T21:04:40.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-02-19T21:04:40.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=7 cursor=8:5c6b0b28:::7:head 2024-02-19T21:04:40.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-02-19T21:04:40.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-02-19T21:04:40.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=14 cursor=8:62a1935d:::14:head 2024-02-19T21:04:40.275 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-02-19T21:04:40.276 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-02-19T21:04:40.276 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=0 cursor=8:6cac518f:::0:head 2024-02-19T21:04:40.276 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-02-19T21:04:40.276 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-02-19T21:04:40.276 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=15 cursor=8:863748b0:::15:head 2024-02-19T21:04:40.276 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:863748b0:::15:head 2024-02-19T21:04:40.276 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-02-19T21:04:40.276 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=11 cursor=8:89d3ae78:::11:head 2024-02-19T21:04:40.276 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-02-19T21:04:40.276 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-02-19T21:04:40.276 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=5 cursor=8:b29083e3:::5:head 2024-02-19T21:04:40.276 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-02-19T21:04:40.277 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-02-19T21:04:40.277 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=8 cursor=8:bd63b0f1:::8:head 2024-02-19T21:04:40.277 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-02-19T21:04:40.277 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-02-19T21:04:40.277 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=6 cursor=8:c4fdafeb:::6:head 2024-02-19T21:04:40.277 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-02-19T21:04:40.277 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-02-19T21:04:40.277 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=3 cursor=8:cfc208b3:::3:head 2024-02-19T21:04:40.277 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-02-19T21:04:40.277 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-02-19T21:04:40.277 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=4 cursor=8:d83876eb:::4:head 2024-02-19T21:04:40.277 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-02-19T21:04:40.277 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-02-19T21:04:40.277 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=12 cursor=8:de5d7c5f:::12:head 2024-02-19T21:04:40.277 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-02-19T21:04:40.278 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-02-19T21:04:40.278 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=9 cursor=8:e960b815:::9:head 2024-02-19T21:04:40.278 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:e960b815:::9:head 2024-02-19T21:04:40.278 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-02-19T21:04:40.278 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : oid=2 cursor=8:f905c69b:::2:head 2024-02-19T21:04:40.278 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-02-19T21:04:40.278 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-02-19T21:04:40.278 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-02-19T21:04:40.278 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : cursor()=8:02547ec2:::1:head expected=8:02547ec2:::1:head 2024-02-19T21:04:40.278 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-02-19T21:04:40.278 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=1 expected=1 2024-02-19T21:04:40.278 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:566253c9:::13:head 2024-02-19T21:04:40.278 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : cursor()=8:566253c9:::13:head expected=8:566253c9:::13:head 2024-02-19T21:04:40.278 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=13 expected=13 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : cursor()=8:62a1935d:::14:head expected=8:62a1935d:::14:head 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=14 expected=14 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : cursor()=8:5c6b0b28:::7:head expected=8:5c6b0b28:::7:head 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=7 expected=7 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : cursor()=8:f905c69b:::2:head expected=8:f905c69b:::2:head 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=2 expected=2 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:e960b815:::9:head 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : cursor()=8:e960b815:::9:head expected=8:e960b815:::9:head 2024-02-19T21:04:40.279 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-02-19T21:04:40.280 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=9 expected=9 2024-02-19T21:04:40.280 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-02-19T21:04:40.280 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : cursor()=8:de5d7c5f:::12:head expected=8:de5d7c5f:::12:head 2024-02-19T21:04:40.280 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-02-19T21:04:40.280 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=12 expected=12 2024-02-19T21:04:40.280 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-02-19T21:04:40.280 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : cursor()=8:d83876eb:::4:head expected=8:d83876eb:::4:head 2024-02-19T21:04:40.280 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-02-19T21:04:40.280 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=4 expected=4 2024-02-19T21:04:40.280 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-02-19T21:04:40.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:04:40.405 INFO:tasks.workunit.client.0.smithi163.stdout: api list_parallel: process_1_[84090]: starting. 2024-02-19T21:04:40.405 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_1_[84090]: creating pool ceph_test_rados_list_parallel.smithi163-84040 2024-02-19T21:04:40.405 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_1_[84090]: created object 0... 2024-02-19T21:04:40.405 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_1_[84090]: created object 25... 2024-02-19T21:04:40.405 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_1_[84090]: created object 49... 2024-02-19T21:04:40.405 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_1_[84090]: finishing. 2024-02-19T21:04:40.405 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_1_[84090]: shutting down. 2024-02-19T21:04:40.405 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_2_[84092]: starting. 2024-02-19T21:04:40.405 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_2_[84092]: listing objects. 2024-02-19T21:04:40.405 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_2_[84092]: listed object 0... 2024-02-19T21:04:40.406 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_2_[84092]: listed object 25... 2024-02-19T21:04:40.406 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_2_[84092]: saw 50 objects 2024-02-19T21:04:40.406 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_2_[84092]: shutting down. 2024-02-19T21:04:40.406 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.406 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_3_[85862]: starting. 2024-02-19T21:04:40.406 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_3_[85862]: creating pool ceph_test_rados_list_parallel.smithi163-84040 2024-02-19T21:04:40.406 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_3_[85862]: created object 0... 2024-02-19T21:04:40.406 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_3_[85862]: created object 25... 2024-02-19T21:04:40.406 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_3_[85862]: created object 49... 2024-02-19T21:04:40.406 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_3_[85862]: finishing. 2024-02-19T21:04:40.406 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_3_[85862]: shutting down. 2024-02-19T21:04:40.406 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.407 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_4_[85863]: starting. 2024-02-19T21:04:40.407 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_4_[85863]: listing objects. 2024-02-19T21:04:40.407 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_4_[85863]: listed object 0... 2024-02-19T21:04:40.407 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_4_[85863]: listed object 25... 2024-02-19T21:04:40.407 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_4_[85863]: saw 43 objects 2024-02-19T21:04:40.407 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_4_[85863]: shutting down. 2024-02-19T21:04:40.407 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.408 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_5_[85864]: starting. 2024-02-19T21:04:40.408 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_5_[85864]: removed 25 objects... 2024-02-19T21:04:40.408 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_5_[85864]: removed half of the objects 2024-02-19T21:04:40.408 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_5_[85864]: removed 50 objects... 2024-02-19T21:04:40.408 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_5_[85864]: removed 50 objects 2024-02-19T21:04:40.408 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_5_[85864]: shutting down. 2024-02-19T21:04:40.408 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.408 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.408 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_6_[85931]: starting. 2024-02-19T21:04:40.408 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_6_[85931]: creating pool ceph_test_rados_list_parallel.smithi163-84040 2024-02-19T21:04:40.408 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_6_[85931]: created object 0... 2024-02-19T21:04:40.408 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_6_[85931]: created object 25... 2024-02-19T21:04:40.408 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_6_[85931]: created object 49... 2024-02-19T21:04:40.409 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_6_[85931]: finishing. 2024-02-19T21:04:40.409 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_6_[85931]: shutting down. 2024-02-19T21:04:40.409 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.409 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.409 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_7_[85932]: starting. 2024-02-19T21:04:40.409 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_7_[85932]: listing objects. 2024-02-19T21:04:40.409 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_7_[85932]: listed object 0... 2024-02-19T21:04:40.409 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_7_[85932]: listed object 25... 2024-02-19T21:04:40.409 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_7_[85932]: listed object 50... 2024-02-19T21:04:40.409 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_7_[85932]: saw 55 objects 2024-02-19T21:04:40.409 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_7_[85932]: shutting down. 2024-02-19T21:04:40.409 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.409 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.410 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_8_[85933]: starting. 2024-02-19T21:04:40.410 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_8_[85933]: added 25 objects... 2024-02-19T21:04:40.410 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_8_[85933]: added half of the objects 2024-02-19T21:04:40.410 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_8_[85933]: added 50 objects... 2024-02-19T21:04:40.410 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_8_[85933]: added 50 objects 2024-02-19T21:04:40.410 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_8_[85933]: shutting down. 2024-02-19T21:04:40.411 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.411 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:38.965284+0000 mgr.y (mgr.14550) 137 : cluster [DBG] pgmap v120: 868 pgs: 42 creating+peering, 132 active+clean, 694 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:04:40.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:39.512538+0000 mon.a (mon.0) 775 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:04:40.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.526017+0000 mon.a (mon.0) 776 : audit [INF] from='client.? v1:172.21.15.163:0/1288380247' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi163-83960-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.526121+0000 mon.a (mon.0) 777 : audit [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi163-83888-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.526254+0000 mon.a (mon.0) 778 : audit [INF] from='client.34958 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2024-02-19T21:04:40.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.526331+0000 mon.a (mon.0) 779 : audit [INF] from='client.34964 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:04:40.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.526399+0000 mon.a (mon.0) 780 : audit [INF] from='client.34868 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.526466+0000 mon.a (mon.0) 781 : audit [INF] from='client.34931 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83992-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.526527+0000 mon.a (mon.0) 782 : audit [INF] from='client.34934 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84000-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.526613+0000 mon.a (mon.0) 783 : audit [INF] from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi163-83872-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.526677+0000 mon.a (mon.0) 784 : audit [INF] from='client.34916 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi163-84032-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.526741+0000 mon.a (mon.0) 785 : audit [INF] from='client.34874 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi163-83864-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.526825+0000 mon.a (mon.0) 786 : audit [INF] from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.526899+0000 mon.a (mon.0) 787 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.526960+0000 mon.a (mon.0) 788 : audit [INF] from='client.24922 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi163-83976-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.527022+0000 mon.a (mon.0) 789 : audit [INF] from='client.24838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi163-83904-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.527093+0000 mon.a (mon.0) 790 : audit [INF] from='client.24865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi163-83952-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.527156+0000 mon.a (mon.0) 791 : audit [INF] from='client.24913 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi163-83845-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.527218+0000 mon.a (mon.0) 792 : audit [INF] from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi163-83928-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.527294+0000 mon.a (mon.0) 793 : audit [INF] from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.527360+0000 mon.a (mon.0) 794 : audit [INF] from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi163-83896-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.527441+0000 mon.a (mon.0) 795 : audit [INF] from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi163-83968-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.527532+0000 mon.a (mon.0) 796 : audit [INF] from='client.24895 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi163-83944-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.527597+0000 mon.a (mon.0) 797 : audit [INF] from='client.24898 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi163-83854-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:39.527717+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-02-19T21:04:40.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.536419+0000 mon.b (mon.1) 102 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi163-83984-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:40.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:39.546206+0000 mon.a (mon.0) 799 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi163-83984-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:40.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:40.229770+0000 mon.c (mon.2) 41 : audit [DBG] from='client.? v1:172.21.15.163:0/800634670' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-19T21:04:40.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:40.274311+0000 osd.1 (osd.1) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:40.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:40.274352+0000 osd.1 (osd.1) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:40.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:40.379369+0000 mon.c (mon.2) 42 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:38.965284+0000 mgr.y (mgr.14550) 137 : cluster [DBG] pgmap v120: 868 pgs: 42 creating+peering, 132 active+clean, 694 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:04:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:39.512538+0000 mon.a (mon.0) 775 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:04:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.526017+0000 mon.a (mon.0) 776 : audit [INF] from='client.? v1:172.21.15.163:0/1288380247' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi163-83960-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.526121+0000 mon.a (mon.0) 777 : audit [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi163-83888-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.526254+0000 mon.a (mon.0) 778 : audit [INF] from='client.34958 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2024-02-19T21:04:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.526331+0000 mon.a (mon.0) 779 : audit [INF] from='client.34964 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:04:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.526399+0000 mon.a (mon.0) 780 : audit [INF] from='client.34868 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.526466+0000 mon.a (mon.0) 781 : audit [INF] from='client.34931 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83992-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.526527+0000 mon.a (mon.0) 782 : audit [INF] from='client.34934 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84000-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.526613+0000 mon.a (mon.0) 783 : audit [INF] from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi163-83872-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.526677+0000 mon.a (mon.0) 784 : audit [INF] from='client.34916 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi163-84032-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.526741+0000 mon.a (mon.0) 785 : audit [INF] from='client.34874 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi163-83864-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.526825+0000 mon.a (mon.0) 786 : audit [INF] from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.526899+0000 mon.a (mon.0) 787 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.526960+0000 mon.a (mon.0) 788 : audit [INF] from='client.24922 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi163-83976-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.527022+0000 mon.a (mon.0) 789 : audit [INF] from='client.24838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi163-83904-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.527093+0000 mon.a (mon.0) 790 : audit [INF] from='client.24865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi163-83952-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.527156+0000 mon.a (mon.0) 791 : audit [INF] from='client.24913 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi163-83845-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.527218+0000 mon.a (mon.0) 792 : audit [INF] from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi163-83928-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.527294+0000 mon.a (mon.0) 793 : audit [INF] from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.527360+0000 mon.a (mon.0) 794 : audit [INF] from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi163-83896-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.527441+0000 mon.a (mon.0) 795 : audit [INF] from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi163-83968-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.527532+0000 mon.a (mon.0) 796 : audit [INF] from='client.24895 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi163-83944-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.527597+0000 mon.a (mon.0) 797 : audit [INF] from='client.24898 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi163-83854-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:39.527717+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-02-19T21:04:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.536419+0000 mon.b (mon.1) 102 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi163-83984-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:39.546206+0000 mon.a (mon.0) 799 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi163-83984-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:40.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:40.229770+0000 mon.c (mon.2) 41 : audit [DBG] from='client.? v1:172.21.15.163:0/800634670' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-19T21:04:40.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:40.274311+0000 osd.1 (osd.1) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:40.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:40.274352+0000 osd.1 (osd.1) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:40.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:40.379369+0000 mon.c (mon.2) 42 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:40.751 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.752 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_9_[86010]: starting. 2024-02-19T21:04:40.752 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_9_[86010]: creating pool ceph_test_rados_list_parallel.smithi163-84040 2024-02-19T21:04:40.752 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_9_[86010]: created object 0... 2024-02-19T21:04:40.752 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_9_[86010]: created object 25... 2024-02-19T21:04:40.752 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_9_[86010]: created object 49... 2024-02-19T21:04:40.752 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_9_[86010]: finishing. 2024-02-19T21:04:40.752 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_9_[86010]: shutting down. 2024-02-19T21:04:40.752 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.752 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.752 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.752 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_10_[86011]: starting. 2024-02-19T21:04:40.752 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_10_[86011]: listing objects. 2024-02-19T21:04:40.753 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_10_[86011]: listed object 0... 2024-02-19T21:04:40.753 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_10_[86011]: listed object 25... 2024-02-19T21:04:40.753 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_10_[86011]: listed object 50... 2024-02-19T21:04:40.753 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_10_[86011]: listed object 75... 2024-02-19T21:04:40.753 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_10_[86011]: saw 97 objects 2024-02-19T21:04:40.753 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_10_[86011]: shutting down. 2024-02-19T21:04:40.753 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.753 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.753 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.753 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_13_[86014]: starting. 2024-02-19T21:04:40.753 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_13_[86014]: removed 25 objects... 2024-02-19T21:04:40.753 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_13_[86014]: removed half of the objects 2024-02-19T21:04:40.753 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_13_[86014]: removed 50 objects... 2024-02-19T21:04:40.754 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_13_[86014]: removed 50 objects 2024-02-19T21:04:40.754 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_13_[86014]: shutting down. 2024-02-19T21:04:40.754 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.754 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.754 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.754 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_11_[86012]: starting. 2024-02-19T21:04:40.754 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_11_[86012]: added 25 objects... 2024-02-19T21:04:40.754 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_11_[86012]: added half of the objects 2024-02-19T21:04:40.754 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_11_[86012]: added 50 objects... 2024-02-19T21:04:40.754 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_11_[86012]: added 50 objects 2024-02-19T21:04:40.754 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_11_[86012]: shutting down. 2024-02-19T21:04:40.754 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.754 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.754 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.754 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_12_[86013]: starting. 2024-02-19T21:04:40.755 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_12_[86013]: added 25 objects... 2024-02-19T21:04:40.755 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_12_[86013]: added half of the objects 2024-02-19T21:04:40.755 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_12_[86013]: added 50 objects... 2024-02-19T21:04:40.755 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_12_[86013]: added 50 objects 2024-02-19T21:04:40.755 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_12_[86013]: shutting down. 2024-02-19T21:04:40.755 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.755 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.755 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.755 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.755 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_14_[86091]: starting. 2024-02-19T21:04:40.755 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_14_[86091]: creating pool ceph_test_rados_list_parallel.smithi163-84040 2024-02-19T21:04:40.755 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_14_[86091]: created object 0... 2024-02-19T21:04:40.755 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_14_[86091]: created object 25... 2024-02-19T21:04:40.755 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_14_[86091]: created object 49... 2024-02-19T21:04:40.755 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_14_[86091]: finishing. 2024-02-19T21:04:40.756 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_14_[86091]: shutting down. 2024-02-19T21:04:40.756 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.756 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.756 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.756 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:40.756 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_15_[86092]: starting. 2024-02-19T21:04:40.756 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_15_[86092]: listing objects. 2024-02-19T21:04:40.756 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_15_[86092]: listed object 0... 2024-02-19T21:04:40.807 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_15_[86092]: listed object 2 api_cmd_pp: Running main() from gmock_main.cc 2024-02-19T21:04:40.808 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-02-19T21:04:40.808 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-02-19T21:04:40.808 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-02-19T21:04:40.808 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-02-19T21:04:40.808 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (143 ms) 2024-02-19T21:04:40.808 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-02-19T21:04:40.808 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (23 ms) 2024-02-19T21:04:40.808 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-02-19T21:04:40.808 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (7810 ms) 2024-02-19T21:04:40.808 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (7976 ms total) 2024-02-19T21:04:40.808 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd_pp: 2024-02-19T21:04:40.809 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-02-19T21:04:40.809 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (7976 ms total) 2024-02-19T21:04:40.809 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-02-19T21:04:40.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:38.965284+0000 mgr.y (mgr.14550) 137 : cluster [DBG] pgmap v120: 868 pgs: 42 creating+peering, 132 active+clean, 694 unknown; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:04:40.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:39.512538+0000 mon.a (mon.0) 775 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:04:40.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.526017+0000 mon.a (mon.0) 776 : audit [INF] from='client.? v1:172.21.15.163:0/1288380247' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi163-83960-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.526121+0000 mon.a (mon.0) 777 : audit [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi163-83888-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.526254+0000 mon.a (mon.0) 778 : audit [INF] from='client.34958 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2024-02-19T21:04:40.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.526331+0000 mon.a (mon.0) 779 : audit [INF] from='client.34964 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:04:40.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.526399+0000 mon.a (mon.0) 780 : audit [INF] from='client.34868 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.526466+0000 mon.a (mon.0) 781 : audit [INF] from='client.34931 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83992-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.526527+0000 mon.a (mon.0) 782 : audit [INF] from='client.34934 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84000-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.526613+0000 mon.a (mon.0) 783 : audit [INF] from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi163-83872-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.526677+0000 mon.a (mon.0) 784 : audit [INF] from='client.34916 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi163-84032-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.526741+0000 mon.a (mon.0) 785 : audit [INF] from='client.34874 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi163-83864-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.526825+0000 mon.a (mon.0) 786 : audit [INF] from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:40.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.526899+0000 mon.a (mon.0) 787 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.526960+0000 mon.a (mon.0) 788 : audit [INF] from='client.24922 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi163-83976-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.527022+0000 mon.a (mon.0) 789 : audit [INF] from='client.24838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi163-83904-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.527093+0000 mon.a (mon.0) 790 : audit [INF] from='client.24865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi163-83952-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.527156+0000 mon.a (mon.0) 791 : audit [INF] from='client.24913 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi163-83845-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.527218+0000 mon.a (mon.0) 792 : audit [INF] from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi163-83928-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.527294+0000 mon.a (mon.0) 793 : audit [INF] from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.527360+0000 mon.a (mon.0) 794 : audit [INF] from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi163-83896-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.527441+0000 mon.a (mon.0) 795 : audit [INF] from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi163-83968-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.527532+0000 mon.a (mon.0) 796 : audit [INF] from='client.24895 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi163-83944-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.527597+0000 mon.a (mon.0) 797 : audit [INF] from='client.24898 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi163-83854-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:39.527717+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-02-19T21:04:40.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.536419+0000 mon.b (mon.1) 102 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi163-83984-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:40.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:39.546206+0000 mon.a (mon.0) 799 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi163-83984-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:40.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:40.229770+0000 mon.c (mon.2) 41 : audit [DBG] from='client.? v1:172.21.15.163:0/800634670' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-02-19T21:04:40.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:40.274311+0000 osd.1 (osd.1) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:40.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:40.274352+0000 osd.1 (osd.1) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:40.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:40.379369+0000 mon.c (mon.2) 42 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:40.827 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-02-19T21:04:40.827 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [==========] Running 16 tests from 1 test suite. 2024-02-19T21:04:40.828 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-02-19T21:04:40.828 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest 2024-02-19T21:04:40.828 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-02-19T21:04:40.828 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-02-19T21:04:40.828 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-02-19T21:04:40.828 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (180 ms) 2024-02-19T21:04:40.828 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-02-19T21:04:40.828 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (110 ms) 2024-02-19T21:04:40.828 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-02-19T21:04:40.828 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (35 ms) 2024-02-19T21:04:40.828 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-02-19T21:04:40.828 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (26 ms) 2024-02-19T21:04:40.828 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (21 ms) 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (12 ms) 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (7 ms) 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (9 ms) 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (4 ms) 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (5 ms) 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (5 ms) 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (13 ms) 2024-02-19T21:04:40.829 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-02-19T21:04:40.830 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (13 ms) 2024-02-19T21:04:40.830 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-02-19T21:04:40.830 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (10 ms) 2024-02-19T21:04:40.830 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-02-19T21:04:40.830 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (5 ms) 2024-02-19T21:04:40.830 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest (455 ms total) 2024-02-19T21:04:40.830 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: 2024-02-19T21:04:40.830 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-02-19T21:04:40.830 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [==========] 16 tests from 1 test suite ran. (7986 ms total) 2024-02-19T21:04:40.830 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_read_operations: [ PASSED ] 16 tests. 2024-02-19T21:04:40.882 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: Running main() from gmock_main.cc 2024-02-19T21:04:40.882 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-02-19T21:04:40.882 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: [----------] Global test environment set-up. 2024-02-19T21:04:40.882 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-02-19T21:04:40.882 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-02-19T21:04:40.882 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (121 ms) 2024-02-19T21:04:40.883 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-02-19T21:04:40.883 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (31 ms) 2024-02-19T21:04:40.883 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-02-19T21:04:40.883 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (7850 ms) 2024-02-19T21:04:40.883 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-02-19T21:04:40.883 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.512538+0000 mon.a [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:04:40.883 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.526017+0000 mon.a [INF] from='client.? v1:172.21.15.163:0/1288380247' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi163-83960-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.883 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.526121+0000 mon.a [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi163-83888-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.883 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.526254+0000 mon.a [INF] from='client.34958 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "iscsi", "var": "pgp_num", "val": "3"}]': finished 2024-02-19T21:04:40.883 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.526331+0000 mon.a [INF] from='client.34964 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:04:40.883 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.526399+0000 mon.a [INF] from='client.34868 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.883 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.526466+0000 mon.a [INF] from='client.34931 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83992-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.883 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.526527+0000 mon.a [INF] from='client.34934 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84000-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.883 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.526613+0000 mon.a [INF] from='client.34889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi163-83872-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.883 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.526677+0000 mon.a [INF] from='client.34916 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi163-84032-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.884 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.526741+0000 mon.a [INF] from='client.34874 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi163-83864-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:40.884 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.526825+0000 mon.a [INF] from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:40.884 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.526899+0000 mon.a [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:41.162 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-19T21:04:41.613 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.526960+0000 mon.a [INF] from='client.24922 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi163-83976-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:41.614 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.527022+0000 mon.a [INF] from='client.24838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi163-83904-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:41.614 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.527093+0000 mon.a [INF] from='client.24865 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi163-83952-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:41.614 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.527156+0000 mon.a [INF] from='client.24913 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi163-83845-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:41.614 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.527218+0000 mon.a [INF] from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi163-83928-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:41.614 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.527294+0000 mon.a [INF] from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:41.614 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.527360+0000 mon.a [INF] from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi163-83896-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:41.614 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.527441+0000 mon.a [INF] from='client.24889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi163-83968-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:41.614 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.527532+0000 mon.a [INF] from='client.24895 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi163-83944-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:41.614 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.527597+0000 mon.a [INF] from='client.24898 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi163-83854-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:41.614 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.536419+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi163-83984-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:41.615 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:39.546206+0000 mon.a [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi163-83984-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:41.615 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:40.802755+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:41.615 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:40.805322+0000 mon.a [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:41.615 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:40.857214+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:41.621 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:41.629 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:41.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:41.632 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 20245... 2024-02-19T21:04:41.633 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_15_[86092]: listed object 50... 2024-02-19T21:04:41.633 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_15_[86092]: listed object 75... 2024-02-19T21:04:41.633 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_15_[86092]: listed object 100... 2024-02-19T21:04:41.633 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_15_[86092]: listed object 125... 2024-02-19T21:04:41.633 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_15_[86092]: saw 150 objects 2024-02-19T21:04:41.633 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_15_[86092]: shutting down. 2024-02-19T21:04:41.633 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:41.633 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:41.633 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:41.633 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:41.633 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_16_[86093]: starting. 2024-02-19T21:04:41.633 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_16_[86093]: added 25 objects... 2024-02-19T21:04:41.633 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_16_[86093]: added half of the objects 2024-02-19T21:04:41.633 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_16_[86093]: added 50 objects... 2024-02-19T21:04:41.634 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_16_[86093]: added 50 objects 2024-02-19T21:04:41.634 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: process_16_[86093]: shutting down. 2024-02-19T21:04:41.634 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:41.634 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:41.634 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:41.634 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:41.634 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******************************* 2024-02-19T21:04:41.634 INFO:tasks.workunit.client.0.smithi163.stdout: list_parallel: ******* SUCCESS ********** 2024-02-19T21:04:41.731 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:41.757 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:41.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:40.065419+0000 mgr.y (mgr.14550) 138 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:41.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:40.801086+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-02-19T21:04:41.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:40.802755+0000 mon.b (mon.1) 103 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:41.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:40.805322+0000 mon.a (mon.0) 801 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:41.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:40.857214+0000 mon.b (mon.1) 104 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:41.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:40.857684+0000 mon.b (mon.1) 105 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:41.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:40.863718+0000 mon.b (mon.1) 106 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:41.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:40.872645+0000 mon.b (mon.1) 107 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:41.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:40.887858+0000 mon.c (mon.2) 43 : audit [INF] from='client.? v1:172.21.15.163:0/1426051668' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-19T21:04:41.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:40.891156+0000 client.admin (client.35186) 0 : cluster [INF] onexx 2024-02-19T21:04:41.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:41.365305+0000 mon.a (mon.0) 802 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-19T21:04:41.883 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:41.884 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:40.065419+0000 mgr.y (mgr.14550) 138 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:41.884 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:40.801086+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-02-19T21:04:41.884 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:40.802755+0000 mon.b (mon.1) 103 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:41.884 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:40.805322+0000 mon.a (mon.0) 801 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:41.884 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:40.857214+0000 mon.b (mon.1) 104 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:41.884 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:40.857684+0000 mon.b (mon.1) 105 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:41.884 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:40.863718+0000 mon.b (mon.1) 106 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:41.884 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:40.872645+0000 mon.b (mon.1) 107 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:41.884 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:40.887858+0000 mon.c (mon.2) 43 : audit [INF] from='client.? v1:172.21.15.163:0/1426051668' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-19T21:04:41.884 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:40.891156+0000 client.admin (client.35186) 0 : cluster [INF] onexx 2024-02-19T21:04:41.884 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:41.365305+0000 mon.a (mon.0) 802 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-19T21:04:41.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:41.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:41.991 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:42.023 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:42.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:40.065419+0000 mgr.y (mgr.14550) 138 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:42.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:40.801086+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-02-19T21:04:42.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:40.802755+0000 mon.b (mon.1) 103 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:42.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:40.805322+0000 mon.a (mon.0) 801 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:42.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:40.857214+0000 mon.b (mon.1) 104 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:42.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:40.857684+0000 mon.b (mon.1) 105 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:42.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:40.863718+0000 mon.b (mon.1) 106 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:42.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:40.872645+0000 mon.b (mon.1) 107 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:42.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:40.887858+0000 mon.c (mon.2) 43 : audit [INF] from='client.? v1:172.21.15.163:0/1426051668' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-19T21:04:42.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:40.891156+0000 client.admin (client.35186) 0 : cluster [INF] onexx 2024-02-19T21:04:42.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:41.365305+0000 mon.a (mon.0) 802 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-19T21:04:42.137 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:42.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.151+0000 7fdf0deed700 1 -- 172.21.15.163:0/4035286857 >> v1:172.21.15.163:6790/0 conn(0x7fdf081042c0 legacy=0x7fdf08106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:42.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.152+0000 7fdf0deed700 1 -- 172.21.15.163:0/4035286857 shutdown_connections 2024-02-19T21:04:42.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.152+0000 7fdf0deed700 1 -- 172.21.15.163:0/4035286857 >> 172.21.15.163:0/4035286857 conn(0x7fdf080f35a0 msgr2=0x7fdf080f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:42.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.152+0000 7fdf0deed700 1 -- 172.21.15.163:0/4035286857 shutdown_connections 2024-02-19T21:04:42.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.152+0000 7fdf0deed700 1 -- 172.21.15.163:0/4035286857 wait complete. 2024-02-19T21:04:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.152+0000 7fdf0deed700 1 librados: starting msgr at 2024-02-19T21:04:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.152+0000 7fdf0deed700 1 librados: starting objecter 2024-02-19T21:04:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.152+0000 7fdf0deed700 1 Processor -- start 2024-02-19T21:04:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.153+0000 7fdf0deed700 1 -- start start 2024-02-19T21:04:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.153+0000 7fdf0deed700 1 librados: setting wanted keys 2024-02-19T21:04:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.153+0000 7fdf0deed700 1 librados: calling monclient init 2024-02-19T21:04:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.153+0000 7fdf0deed700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf0806cc10 con 0x7fdf08100780 2024-02-19T21:04:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.153+0000 7fdf0deed700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf0806cde0 con 0x7fdf081042c0 2024-02-19T21:04:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.153+0000 7fdf0deed700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf0811bb80 con 0x7fdf080fd980 2024-02-19T21:04:42.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.153+0000 7fdf077fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fdf080fd980 0x7fdf080fc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:41442/0 (socket says 172.21.15.163:41442) 2024-02-19T21:04:42.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.153+0000 7fdf077fe700 1 -- 172.21.15.163:0/856391014 learned_addr learned my addr 172.21.15.163:0/856391014 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:04:42.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.153+0000 7fdf04ff9700 10 client.?.objecter ms_handle_connect 0x7fdf080fd980 2024-02-19T21:04:42.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.153+0000 7fdf04ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:42.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.153+0000 7fdf04ff9700 10 client.?.objecter ms_handle_connect 0x7fdf08100780 2024-02-19T21:04:42.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.153+0000 7fdf04ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:42.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.153+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311263337 0 0) 0x7fdf0811bb80 con 0x7fdf080fd980 2024-02-19T21:04:42.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.153+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef0002220 con 0x7fdf080fd980 2024-02-19T21:04:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.154+0000 7fdf04ff9700 10 client.?.objecter ms_handle_connect 0x7fdf081042c0 2024-02-19T21:04:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.154+0000 7fdf04ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.154+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4126467276 0 0) 0x7fdef0002220 con 0x7fdf080fd980 2024-02-19T21:04:42.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.154+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf0811bb80 con 0x7fdf080fd980 2024-02-19T21:04:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.154+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdef8001800 con 0x7fdf080fd980 2024-02-19T21:04:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.154+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3080292860 0 0) 0x7fdf0811bb80 con 0x7fdf080fd980 2024-02-19T21:04:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.154+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 >> v1:172.21.15.193:6789/0 conn(0x7fdf081042c0 legacy=0x7fdf08118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.154+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 >> v1:172.21.15.163:6789/0 conn(0x7fdf08100780 legacy=0x7fdf08114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.154+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0811cd20 con 0x7fdf080fd980 2024-02-19T21:04:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.154+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0811bd70 con 0x7fdf080fd980 2024-02-19T21:04:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.154+0000 7fdf0deed700 10 client.35198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.154+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0811c220 con 0x7fdf080fd980 2024-02-19T21:04:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.154+0000 7fdf0deed700 1 librados: init done 2024-02-19T21:04:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.154+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fdef8004c70 con 0x7fdf080fd980 2024-02-19T21:04:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.154+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdef8005b70 con 0x7fdf080fd980 2024-02-19T21:04:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.155+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fdef4004f30 con 0x7fdf080fd980 2024-02-19T21:04:42.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.156+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fdef80047d0 con 0x7fdf080fd980 2024-02-19T21:04:42.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.156+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(102..102 src has 43..102) v4 ==== 12839+0+0 (unknown 4253754181 0 0) 0x7fdef807a080 con 0x7fdf080fd980 2024-02-19T21:04:42.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.156+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 osd_map(102..102 src has 43..102) v4 2024-02-19T21:04:42.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.156+0000 7fdf04ff9700 3 client.35198.objecter handle_osd_map got epochs [102,102] > 0 2024-02-19T21:04:42.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.156+0000 7fdf04ff9700 3 client.35198.objecter handle_osd_map decoding full epoch 102 2024-02-19T21:04:42.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.156+0000 7fdf04ff9700 20 client.35198.objecter dump_active .. 0 homeless 2024-02-19T21:04:42.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.156+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v102) v1 ==== 54+0+16 (unknown 2944338779 0 2420000192) 0x7fdef807c490 con 0x7fdf080fd980 2024-02-19T21:04:42.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.157+0000 7fdf04ff9700 10 client.35198.objecter ms_handle_connect 0x7fdef0063a00 2024-02-19T21:04:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.159+0000 7fdf0deed700 10 client.35198.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.159+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 1 2024-02-19T21:04:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.159+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef40051d0 osd=0 3 2024-02-19T21:04:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.159+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef40051d0 osd=0 3 2024-02-19T21:04:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.159+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef40051d0 osd=0 3 2024-02-19T21:04:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.159+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 1 2024-02-19T21:04:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.159+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef40051d0 osd=0 3 2024-02-19T21:04:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.159+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 0 1 2024-02-19T21:04:42.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.159+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef40051d0 osd=0 4 2024-02-19T21:04:42.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.159+0000 7fdf0deed700 10 client.35198.objecter _send_command 1 2024-02-19T21:04:42.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.159+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdef4004980 con 0x7fdef4004560 2024-02-19T21:04:42.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.159+0000 7fdf04ff9700 10 client.35198.objecter ms_handle_connect 0x7fdef4004560 2024-02-19T21:04:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:42.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:42.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.160+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdef4004980 con 0x7fdef4004560 2024-02-19T21:04:42.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.160+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:04:42.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.160+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 1 = system:0 2024-02-19T21:04:42.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.160+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef40051d0 osd=0 4 2024-02-19T21:04:42.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.160+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 0 1 2024-02-19T21:04:42.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.178+0000 7fdf0deed700 10 client.35198.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:42.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.178+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 2 2024-02-19T21:04:42.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.178+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef40051d0 osd=0 3 2024-02-19T21:04:42.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.178+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef40051d0 osd=0 3 2024-02-19T21:04:42.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.178+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef40051d0 osd=0 3 2024-02-19T21:04:42.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.178+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 2 2024-02-19T21:04:42.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.178+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef40051d0 osd=0 3 2024-02-19T21:04:42.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.178+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 0 2 2024-02-19T21:04:42.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.178+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef40051d0 osd=0 4 2024-02-19T21:04:42.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.178+0000 7fdf0deed700 10 client.35198.objecter _send_command 2 2024-02-19T21:04:42.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.178+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdef4004d60 con 0x7fdef4004560 2024-02-19T21:04:42.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.179+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdef4004d60 con 0x7fdef4004560 2024-02-19T21:04:42.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.179+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:42.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.179+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.179+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef40051d0 osd=0 4 2024-02-19T21:04:42.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.179+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 0 2 2024-02-19T21:04:42.172 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:04:42.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.180+0000 7fdf0deed700 10 client.35198.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:42.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.180+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 3 2024-02-19T21:04:42.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.180+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4004010 osd=1 3 2024-02-19T21:04:42.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.180+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4004010 osd=1 3 2024-02-19T21:04:42.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.180+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4004010 osd=1 3 2024-02-19T21:04:42.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.180+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 3 2024-02-19T21:04:42.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.180+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef4004010 osd=1 3 2024-02-19T21:04:42.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.180+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 1 3 2024-02-19T21:04:42.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.180+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4004010 osd=1 4 2024-02-19T21:04:42.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.180+0000 7fdf0deed700 10 client.35198.objecter _send_command 3 2024-02-19T21:04:42.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.180+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdef400c030 con 0x7fdef4004180 2024-02-19T21:04:42.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.180+0000 7fdf04ff9700 10 client.35198.objecter ms_handle_connect 0x7fdef4004180 2024-02-19T21:04:42.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.181+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdef400c030 con 0x7fdef4004180 2024-02-19T21:04:42.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.181+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:04:42.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.181+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 3 = system:0 2024-02-19T21:04:42.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.181+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef4004010 osd=1 4 2024-02-19T21:04:42.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.181+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 1 3 2024-02-19T21:04:42.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.196+0000 7fdf0deed700 10 client.35198.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.196+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 4 2024-02-19T21:04:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.196+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4004010 osd=1 3 2024-02-19T21:04:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.196+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4004010 osd=1 3 2024-02-19T21:04:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.196+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4004010 osd=1 3 2024-02-19T21:04:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.196+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 4 2024-02-19T21:04:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.196+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef4004010 osd=1 3 2024-02-19T21:04:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.196+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 1 4 2024-02-19T21:04:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.196+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4004010 osd=1 4 2024-02-19T21:04:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.196+0000 7fdf0deed700 10 client.35198.objecter _send_command 4 2024-02-19T21:04:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.196+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdef4001880 con 0x7fdef4004180 2024-02-19T21:04:42.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdef4001880 con 0x7fdef4004180 2024-02-19T21:04:42.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:42.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef4004010 osd=1 4 2024-02-19T21:04:42.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 1 4 2024-02-19T21:04:42.191 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:04:42.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf0deed700 10 client.35198.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:42.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 5 2024-02-19T21:04:42.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4003660 osd=2 3 2024-02-19T21:04:42.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4003660 osd=2 3 2024-02-19T21:04:42.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4003660 osd=2 3 2024-02-19T21:04:42.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 5 2024-02-19T21:04:42.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef4003660 osd=2 3 2024-02-19T21:04:42.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 2 5 2024-02-19T21:04:42.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4003660 osd=2 4 2024-02-19T21:04:42.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf0deed700 10 client.35198.objecter _send_command 5 2024-02-19T21:04:42.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdef4003ea0 con 0x7fdef400d320 2024-02-19T21:04:42.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.198+0000 7fdf04ff9700 10 client.35198.objecter ms_handle_connect 0x7fdef400d320 2024-02-19T21:04:42.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.199+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdef4003ea0 con 0x7fdef400d320 2024-02-19T21:04:42.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.199+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:04:42.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.199+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 5 = system:0 2024-02-19T21:04:42.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.199+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef4003660 osd=2 4 2024-02-19T21:04:42.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.199+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 2 5 2024-02-19T21:04:42.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.213+0000 7fdf0deed700 10 client.35198.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:42.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.213+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 6 2024-02-19T21:04:42.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.213+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4003660 osd=2 3 2024-02-19T21:04:42.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.213+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4003660 osd=2 3 2024-02-19T21:04:42.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.213+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4003660 osd=2 3 2024-02-19T21:04:42.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.213+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 6 2024-02-19T21:04:42.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.213+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef4003660 osd=2 3 2024-02-19T21:04:42.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.213+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 2 6 2024-02-19T21:04:42.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.213+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4003660 osd=2 4 2024-02-19T21:04:42.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.213+0000 7fdf0deed700 10 client.35198.objecter _send_command 6 2024-02-19T21:04:42.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.213+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdef400e780 con 0x7fdef400d320 2024-02-19T21:04:42.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdef400e780 con 0x7fdef400d320 2024-02-19T21:04:42.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:42.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef4003660 osd=2 4 2024-02-19T21:04:42.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 2 6 2024-02-19T21:04:42.208 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:04:42.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf0deed700 10 client.35198.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:42.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 7 2024-02-19T21:04:42.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4003440 osd=3 3 2024-02-19T21:04:42.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4003440 osd=3 3 2024-02-19T21:04:42.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4003440 osd=3 3 2024-02-19T21:04:42.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 7 2024-02-19T21:04:42.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef4003440 osd=3 3 2024-02-19T21:04:42.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 3 7 2024-02-19T21:04:42.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4003440 osd=3 4 2024-02-19T21:04:42.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf0deed700 10 client.35198.objecter _send_command 7 2024-02-19T21:04:42.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.215+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdef4015b30 con 0x7fdef4011eb0 2024-02-19T21:04:42.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.216+0000 7fdf04ff9700 10 client.35198.objecter ms_handle_connect 0x7fdef4011eb0 2024-02-19T21:04:42.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.217+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdef4015b30 con 0x7fdef4011eb0 2024-02-19T21:04:42.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.217+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:04:42.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.217+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 7 = system:0 2024-02-19T21:04:42.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.217+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef4003440 osd=3 4 2024-02-19T21:04:42.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.217+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 3 7 2024-02-19T21:04:42.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.232+0000 7fdf0deed700 10 client.35198.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:42.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.232+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 8 2024-02-19T21:04:42.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.232+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4003440 osd=3 3 2024-02-19T21:04:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.232+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4003440 osd=3 3 2024-02-19T21:04:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.232+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4003440 osd=3 3 2024-02-19T21:04:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.232+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 8 2024-02-19T21:04:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.232+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef4003440 osd=3 3 2024-02-19T21:04:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.232+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 3 8 2024-02-19T21:04:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.232+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4003440 osd=3 4 2024-02-19T21:04:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.232+0000 7fdf0deed700 10 client.35198.objecter _send_command 8 2024-02-19T21:04:42.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.232+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdef400e1f0 con 0x7fdef4011eb0 2024-02-19T21:04:42.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdef400e1f0 con 0x7fdef4011eb0 2024-02-19T21:04:42.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:42.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef4003440 osd=3 4 2024-02-19T21:04:42.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 3 8 2024-02-19T21:04:42.227 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:04:42.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf0deed700 10 client.35198.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:42.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 9 2024-02-19T21:04:42.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef400dfb0 osd=4 3 2024-02-19T21:04:42.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef400dfb0 osd=4 3 2024-02-19T21:04:42.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef400dfb0 osd=4 3 2024-02-19T21:04:42.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 9 2024-02-19T21:04:42.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef400dfb0 osd=4 3 2024-02-19T21:04:42.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 4 9 2024-02-19T21:04:42.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef400dfb0 osd=4 4 2024-02-19T21:04:42.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf0deed700 10 client.35198.objecter _send_command 9 2024-02-19T21:04:42.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.234+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdef400e150 con 0x7fdef400ce80 2024-02-19T21:04:42.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.235+0000 7fdf04ff9700 10 client.35198.objecter ms_handle_connect 0x7fdef400ce80 2024-02-19T21:04:42.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.236+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdef400e150 con 0x7fdef400ce80 2024-02-19T21:04:42.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.236+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:04:42.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.236+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 9 = system:0 2024-02-19T21:04:42.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.236+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef400dfb0 osd=4 4 2024-02-19T21:04:42.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.236+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 4 9 2024-02-19T21:04:42.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.257+0000 7fdf0deed700 10 client.35198.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:42.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.257+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 10 2024-02-19T21:04:42.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.257+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef400dfb0 osd=4 3 2024-02-19T21:04:42.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.257+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef400dfb0 osd=4 3 2024-02-19T21:04:42.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.257+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef400dfb0 osd=4 3 2024-02-19T21:04:42.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.257+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 10 2024-02-19T21:04:42.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.257+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef400dfb0 osd=4 3 2024-02-19T21:04:42.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.257+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 4 10 2024-02-19T21:04:42.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.257+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef400dfb0 osd=4 4 2024-02-19T21:04:42.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.257+0000 7fdf0deed700 10 client.35198.objecter _send_command 10 2024-02-19T21:04:42.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.257+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdef400e150 con 0x7fdef400ce80 2024-02-19T21:04:42.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdef400e150 con 0x7fdef400ce80 2024-02-19T21:04:42.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:42.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef400dfb0 osd=4 4 2024-02-19T21:04:42.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 4 10 2024-02-19T21:04:42.251 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:04:42.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf0deed700 10 client.35198.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:42.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 11 2024-02-19T21:04:42.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef400e150 osd=5 3 2024-02-19T21:04:42.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef400e150 osd=5 3 2024-02-19T21:04:42.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef400e150 osd=5 3 2024-02-19T21:04:42.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 11 2024-02-19T21:04:42.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef400e150 osd=5 3 2024-02-19T21:04:42.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 5 11 2024-02-19T21:04:42.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef400e150 osd=5 4 2024-02-19T21:04:42.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf0deed700 10 client.35198.objecter _send_command 11 2024-02-19T21:04:42.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdef4018090 con 0x7fdef4017ac0 2024-02-19T21:04:42.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.259+0000 7fdf04ff9700 10 client.35198.objecter ms_handle_connect 0x7fdef4017ac0 2024-02-19T21:04:42.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.260+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdef4018090 con 0x7fdef4017ac0 2024-02-19T21:04:42.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.260+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:04:42.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.260+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 11 = system:0 2024-02-19T21:04:42.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.260+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef400e150 osd=5 4 2024-02-19T21:04:42.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.260+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 5 11 2024-02-19T21:04:42.260 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:42.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.283+0000 7fdf0deed700 10 client.35198.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:42.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.283+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 12 2024-02-19T21:04:42.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.283+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef400e150 osd=5 3 2024-02-19T21:04:42.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.283+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef400e150 osd=5 3 2024-02-19T21:04:42.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.283+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef400e150 osd=5 3 2024-02-19T21:04:42.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.283+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 12 2024-02-19T21:04:42.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.283+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef400e150 osd=5 3 2024-02-19T21:04:42.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.283+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 5 12 2024-02-19T21:04:42.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.283+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef400e150 osd=5 4 2024-02-19T21:04:42.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.283+0000 7fdf0deed700 10 client.35198.objecter _send_command 12 2024-02-19T21:04:42.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.283+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdef4018090 con 0x7fdef4017ac0 2024-02-19T21:04:42.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.284+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdef4018090 con 0x7fdef4017ac0 2024-02-19T21:04:42.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.284+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:42.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.284+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.284+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef400e150 osd=5 4 2024-02-19T21:04:42.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.284+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 5 12 2024-02-19T21:04:42.277 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:04:42.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.285+0000 7fdf0deed700 10 client.35198.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:42.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.285+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 13 2024-02-19T21:04:42.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.285+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4018090 osd=6 3 2024-02-19T21:04:42.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.285+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4018090 osd=6 3 2024-02-19T21:04:42.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.285+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4018090 osd=6 3 2024-02-19T21:04:42.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.285+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 13 2024-02-19T21:04:42.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.285+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef4018090 osd=6 3 2024-02-19T21:04:42.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.285+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 6 13 2024-02-19T21:04:42.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.285+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4018090 osd=6 4 2024-02-19T21:04:42.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.285+0000 7fdf0deed700 10 client.35198.objecter _send_command 13 2024-02-19T21:04:42.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.285+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdef4022d00 con 0x7fdef401f130 2024-02-19T21:04:42.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.285+0000 7fdf04ff9700 10 client.35198.objecter ms_handle_connect 0x7fdef401f130 2024-02-19T21:04:42.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.286+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdef4022d00 con 0x7fdef401f130 2024-02-19T21:04:42.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.286+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:04:42.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.286+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 13 = system:0 2024-02-19T21:04:42.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.286+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef4018090 osd=6 4 2024-02-19T21:04:42.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.286+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 6 13 2024-02-19T21:04:42.287 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:42.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.306+0000 7fdf0deed700 10 client.35198.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:42.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.306+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 14 2024-02-19T21:04:42.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.306+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4018090 osd=6 3 2024-02-19T21:04:42.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.306+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4018090 osd=6 3 2024-02-19T21:04:42.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.306+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4018090 osd=6 3 2024-02-19T21:04:42.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.306+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 14 2024-02-19T21:04:42.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.306+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef4018090 osd=6 3 2024-02-19T21:04:42.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.306+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 6 14 2024-02-19T21:04:42.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.306+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4018090 osd=6 4 2024-02-19T21:04:42.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.307+0000 7fdf0deed700 10 client.35198.objecter _send_command 14 2024-02-19T21:04:42.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.307+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdef4022d00 con 0x7fdef401f130 2024-02-19T21:04:42.304 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:04:42.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.308+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdef4022d00 con 0x7fdef401f130 2024-02-19T21:04:42.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.308+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:42.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.308+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.308+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef4018090 osd=6 4 2024-02-19T21:04:42.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.308+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 6 14 2024-02-19T21:04:42.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.312+0000 7fdf0deed700 10 client.35198.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:42.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.312+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 15 2024-02-19T21:04:42.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.312+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4022d00 osd=7 3 2024-02-19T21:04:42.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.312+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4022d00 osd=7 3 2024-02-19T21:04:42.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.312+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4022d00 osd=7 3 2024-02-19T21:04:42.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.312+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 15 2024-02-19T21:04:42.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.312+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef4022d00 osd=7 3 2024-02-19T21:04:42.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.312+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 7 15 2024-02-19T21:04:42.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.312+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4022d00 osd=7 4 2024-02-19T21:04:42.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.312+0000 7fdf0deed700 10 client.35198.objecter _send_command 15 2024-02-19T21:04:42.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.312+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdef4023f20 con 0x7fdef40249c0 2024-02-19T21:04:42.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.312+0000 7fdf04ff9700 10 client.35198.objecter ms_handle_connect 0x7fdef40249c0 2024-02-19T21:04:42.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.313+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdef4023f20 con 0x7fdef40249c0 2024-02-19T21:04:42.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.313+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:04:42.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.313+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 15 = system:0 2024-02-19T21:04:42.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.313+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef4022d00 osd=7 4 2024-02-19T21:04:42.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.313+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 7 15 2024-02-19T21:04:42.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.329+0000 7fdf0deed700 10 client.35198.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:04:42.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.329+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign -1 16 2024-02-19T21:04:42.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.329+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4022d00 osd=7 3 2024-02-19T21:04:42.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.329+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4022d00 osd=7 3 2024-02-19T21:04:42.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.329+0000 7fdf0deed700 20 client.35198.objecter _get_session s=0x7fdef4022d00 osd=7 3 2024-02-19T21:04:42.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.329+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_remove -1 16 2024-02-19T21:04:42.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.329+0000 7fdf0deed700 20 client.35198.objecter get_session s=0x7fdef4022d00 osd=7 3 2024-02-19T21:04:42.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.329+0000 7fdf0deed700 15 client.35198.objecter _session_command_op_assign 7 16 2024-02-19T21:04:42.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.329+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4022d00 osd=7 4 2024-02-19T21:04:42.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.329+0000 7fdf0deed700 10 client.35198.objecter _send_command 16 2024-02-19T21:04:42.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.329+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fdef4023f20 con 0x7fdef40249c0 2024-02-19T21:04:42.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf04ff9700 1 -- 172.21.15.163:0/856391014 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fdef4023f20 con 0x7fdef40249c0 2024-02-19T21:04:42.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf04ff9700 10 client.35198.objecter ms_dispatch 0x7fdf08001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:04:42.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf04ff9700 10 client.35198.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf04ff9700 20 client.35198.objecter put_session s=0x7fdef4022d00 osd=7 4 2024-02-19T21:04:42.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf04ff9700 15 client.35198.objecter _session_command_op_remove 7 16 2024-02-19T21:04:42.324 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:04:42.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 10 librados: watch_flush enter 2024-02-19T21:04:42.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 10 librados: watch_flush exit 2024-02-19T21:04:42.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 10 client.35198.objecter close_session for osd.0 2024-02-19T21:04:42.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 >> v1:172.21.15.163:6801/1559418115 conn(0x7fdef4004560 legacy=0x7fdef4002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:42.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef40051d0 osd=0 1 2024-02-19T21:04:42.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 10 client.35198.objecter close_session for osd.1 2024-02-19T21:04:42.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 >> v1:172.21.15.163:6805/3125047432 conn(0x7fdef4004180 legacy=0x7fdef400a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:42.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4004010 osd=1 1 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 10 client.35198.objecter close_session for osd.2 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 >> v1:172.21.15.163:6809/1563158487 conn(0x7fdef400d320 legacy=0x7fdef400d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4003660 osd=2 1 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 10 client.35198.objecter close_session for osd.3 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 >> v1:172.21.15.163:6813/2112279769 conn(0x7fdef4011eb0 legacy=0x7fdef40142a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4003440 osd=3 1 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 10 client.35198.objecter close_session for osd.4 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 >> v1:172.21.15.193:6800/3129716965 conn(0x7fdef400ce80 legacy=0x7fdef4015cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef400dfb0 osd=4 1 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 10 client.35198.objecter close_session for osd.5 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf07fff700 1 -- 172.21.15.163:0/856391014 reap_dead start 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 >> v1:172.21.15.193:6804/805161991 conn(0x7fdef4017ac0 legacy=0x7fdef401d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf07fff700 1 -- 172.21.15.163:0/856391014 reap_dead start 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf07fff700 1 -- 172.21.15.163:0/856391014 reap_dead start 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef400e150 osd=5 1 2024-02-19T21:04:42.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 10 client.35198.objecter close_session for osd.6 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 >> v1:172.21.15.193:6808/787604615 conn(0x7fdef401f130 legacy=0x7fdef4021520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4018090 osd=6 1 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 10 client.35198.objecter close_session for osd.7 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 >> v1:172.21.15.193:6812/157927432 conn(0x7fdef40249c0 legacy=0x7fdef4024e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 20 client.35198.objecter put_session s=0x7fdef4022d00 osd=7 1 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 20 client.35198.objecter shutdown clearing up homeless session... 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 10 client.35198.objecter successfully canceled tick 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 >> v1:172.21.15.163:6800/1552514681 conn(0x7fdef0063a00 legacy=0x7fdef0065ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 >> v1:172.21.15.163:6790/0 conn(0x7fdf080fd980 legacy=0x7fdf080fc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 shutdown_connections 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 >> 172.21.15.163:0/856391014 conn(0x7fdf080f35a0 msgr2=0x7fdf08107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.331+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 shutdown_connections 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.332+0000 7fdf0deed700 1 -- 172.21.15.163:0/856391014 wait complete. 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:42.332+0000 7fdf0deed700 1 librados: shutdown 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.327 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.328 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.328 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.328 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.328 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.328 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.328 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:04:42.674 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:41.038837+0000 osd.0 (osd.0) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:42.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:41.038875+0000 osd.0 (osd.0) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:42.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:41.380623+0000 mon.c (mon.2) 44 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:42.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:41.618508+0000 mon.a (mon.0) 803 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:42.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:41.622698+0000 mon.a (mon.0) 804 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-02-19T21:04:42.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:41.623303+0000 mon.c (mon.2) 45 : audit [INF] from='client.? v1:172.21.15.163:0/1426051668' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-19T21:04:42.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:41.623439+0000 client.admin (client.35186) 0 : cluster [INF] twoxx 2024-02-19T21:04:42.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:41.623772+0000 mon.a (mon.0) 805 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:42.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:41.623945+0000 mon.a (mon.0) 806 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-19T21:04:42.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:41.624119+0000 mon.a (mon.0) 807 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:42.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:41.628218+0000 mon.a (mon.0) 808 : audit [INF] from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi163-83984-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]': finished 2024-02-19T21:04:42.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:41.628425+0000 mon.a (mon.0) 809 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:42.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:41.636238+0000 mon.b (mon.1) 108 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:42.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:41.642589+0000 mon.b (mon.1) 109 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:42.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:41.650413+0000 mon.b (mon.1) 110 : audit [INF] from='client.? v1:172.21.15.163:0/2094902030' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi163-83845-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:42.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:41.038837+0000 osd.0 (osd.0) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:42.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:41.038875+0000 osd.0 (osd.0) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:42.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:41.380623+0000 mon.c (mon.2) 44 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:42.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:41.618508+0000 mon.a (mon.0) 803 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:42.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:41.622698+0000 mon.a (mon.0) 804 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-02-19T21:04:42.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:41.623303+0000 mon.c (mon.2) 45 : audit [INF] from='client.? v1:172.21.15.163:0/1426051668' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-19T21:04:42.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:41.623439+0000 client.admin (client.35186) 0 : cluster [INF] twoxx 2024-02-19T21:04:42.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:41.623772+0000 mon.a (mon.0) 805 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:42.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:41.623945+0000 mon.a (mon.0) 806 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-19T21:04:42.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:41.624119+0000 mon.a (mon.0) 807 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:42.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:41.628218+0000 mon.a (mon.0) 808 : audit [INF] from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi163-83984-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]': finished 2024-02-19T21:04:42.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:41.628425+0000 mon.a (mon.0) 809 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:42.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:41.636238+0000 mon.b (mon.1) 108 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:42.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:41.642589+0000 mon.b (mon.1) 109 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:42.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:41.650684+0000 mon.b (mon.1) 111 : audit [INF] from='client.? v1:172.21.15.163:0/3338391635' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:42.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:41.898037+0000 mon.a (mon.0) 810 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-19T21:04:42.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:41.898913+0000 mon.b (mon.1) 112 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:42.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.156648+0000 mon.c (mon.2) 46 : audit [DBG] from='client.? v1:172.21.15.163:0/856391014' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:04:42.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.317867+0000 mon.b (mon.1) 113 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.0"}]: dispatch 2024-02-19T21:04:42.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.318619+0000 mon.b (mon.1) 114 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.1"}]: dispatch 2024-02-19T21:04:42.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.319131+0000 mon.b (mon.1) 115 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.2"}]: dispatch 2024-02-19T21:04:42.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.319516+0000 mon.b (mon.1) 116 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.3"}]: dispatch 2024-02-19T21:04:42.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.319902+0000 mon.b (mon.1) 117 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.4"}]: dispatch 2024-02-19T21:04:42.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.320292+0000 mon.b (mon.1) 118 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.5"}]: dispatch 2024-02-19T21:04:42.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.320673+0000 mon.b (mon.1) 119 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.6"}]: dispatch 2024-02-19T21:04:42.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.321046+0000 mon.b (mon.1) 120 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.7"}]: dispatch 2024-02-19T21:04:42.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.321441+0000 mon.b (mon.1) 121 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.8"}]: dispatch 2024-02-19T21:04:42.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.321836+0000 mon.b (mon.1) 122 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.9"}]: dispatch 2024-02-19T21:04:42.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:41.650413+0000 mon.b (mon.1) 110 : audit [INF] from='client.? v1:172.21.15.163:0/2094902030' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi163-83845-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:42.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:41.650684+0000 mon.b (mon.1) 111 : audit [INF] from='client.? v1:172.21.15.163:0/3338391635' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:42.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:41.898037+0000 mon.a (mon.0) 810 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-19T21:04:42.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:41.898913+0000 mon.b (mon.1) 112 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:42.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.156648+0000 mon.c (mon.2) 46 : audit [DBG] from='client.? v1:172.21.15.163:0/856391014' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:04:42.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.317867+0000 mon.b (mon.1) 113 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.0"}]: dispatch 2024-02-19T21:04:42.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.318619+0000 mon.b (mon.1) 114 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.1"}]: dispatch 2024-02-19T21:04:42.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.319131+0000 mon.b (mon.1) 115 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.2"}]: dispatch 2024-02-19T21:04:42.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.319516+0000 mon.b (mon.1) 116 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.3"}]: dispatch 2024-02-19T21:04:42.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.319902+0000 mon.b (mon.1) 117 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.4"}]: dispatch 2024-02-19T21:04:42.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.320292+0000 mon.b (mon.1) 118 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.5"}]: dispatch 2024-02-19T21:04:42.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.320673+0000 mon.b (mon.1) 119 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.6"}]: dispatch 2024-02-19T21:04:42.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.321046+0000 mon.b (mon.1) 120 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.7"}]: dispatch 2024-02-19T21:04:42.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.321441+0000 mon.b (mon.1) 121 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.8"}]: dispatch 2024-02-19T21:04:42.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.321836+0000 mon.b (mon.1) 122 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.9"}]: dispatch 2024-02-19T21:04:42.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.382049+0000 mon.c (mon.2) 47 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:42.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.427888+0000 mon.a (mon.0) 811 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:42.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.430384+0000 mon.b (mon.1) 123 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:42.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.430561+0000 mon.a (mon.0) 812 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:42.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.431016+0000 mon.a (mon.0) 813 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:42.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.431420+0000 mon.a (mon.0) 814 : audit [INF] from='client.25213 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi163-83845-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:42.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.382049+0000 mon.c (mon.2) 47 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:42.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.427888+0000 mon.a (mon.0) 811 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:42.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.430384+0000 mon.b (mon.1) 123 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:42.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.430561+0000 mon.a (mon.0) 812 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:42.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.431016+0000 mon.a (mon.0) 813 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:42.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.431420+0000 mon.a (mon.0) 814 : audit [INF] from='client.25213 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi163-83845-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:41.038837+0000 osd.0 (osd.0) 61 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:43.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:41.038875+0000 osd.0 (osd.0) 62 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:43.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:41.380623+0000 mon.c (mon.2) 44 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:43.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:41.618508+0000 mon.a (mon.0) 803 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:43.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:41.622698+0000 mon.a (mon.0) 804 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-02-19T21:04:43.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:41.623303+0000 mon.c (mon.2) 45 : audit [INF] from='client.? v1:172.21.15.163:0/1426051668' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-19T21:04:43.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:41.623439+0000 client.admin (client.35186) 0 : cluster [INF] twoxx 2024-02-19T21:04:43.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:41.623772+0000 mon.a (mon.0) 805 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:43.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:41.623945+0000 mon.a (mon.0) 806 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-02-19T21:04:43.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:41.624119+0000 mon.a (mon.0) 807 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:43.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:41.628218+0000 mon.a (mon.0) 808 : audit [INF] from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi163-83984-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]': finished 2024-02-19T21:04:43.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:41.628425+0000 mon.a (mon.0) 809 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:43.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:41.636238+0000 mon.b (mon.1) 108 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:43.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:41.642589+0000 mon.b (mon.1) 109 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:43.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:41.650413+0000 mon.b (mon.1) 110 : audit [INF] from='client.? v1:172.21.15.163:0/2094902030' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi163-83845-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:41.650684+0000 mon.b (mon.1) 111 : audit [INF] from='client.? v1:172.21.15.163:0/3338391635' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:41.898037+0000 mon.a (mon.0) 810 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-02-19T21:04:43.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:41.898913+0000 mon.b (mon.1) 112 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:43.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.156648+0000 mon.c (mon.2) 46 : audit [DBG] from='client.? v1:172.21.15.163:0/856391014' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:04:43.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.317867+0000 mon.b (mon.1) 113 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.0"}]: dispatch 2024-02-19T21:04:43.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.318619+0000 mon.b (mon.1) 114 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.1"}]: dispatch 2024-02-19T21:04:43.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.319131+0000 mon.b (mon.1) 115 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.2"}]: dispatch 2024-02-19T21:04:43.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.319516+0000 mon.b (mon.1) 116 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.3"}]: dispatch 2024-02-19T21:04:43.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.319902+0000 mon.b (mon.1) 117 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.4"}]: dispatch 2024-02-19T21:04:43.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.320292+0000 mon.b (mon.1) 118 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.5"}]: dispatch 2024-02-19T21:04:43.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.320673+0000 mon.b (mon.1) 119 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.6"}]: dispatch 2024-02-19T21:04:43.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.321046+0000 mon.b (mon.1) 120 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.7"}]: dispatch 2024-02-19T21:04:43.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.321441+0000 mon.b (mon.1) 121 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.8"}]: dispatch 2024-02-19T21:04:43.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.321836+0000 mon.b (mon.1) 122 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.9"}]: dispatch 2024-02-19T21:04:43.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.382049+0000 mon.c (mon.2) 47 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:43.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.427888+0000 mon.a (mon.0) 811 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:43.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.430384+0000 mon.b (mon.1) 123 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:43.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.430561+0000 mon.a (mon.0) 812 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:43.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.431016+0000 mon.a (mon.0) 813 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:43.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.431420+0000 mon.a (mon.0) 814 : audit [INF] from='client.25213 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi163-83845-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.443 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:43.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:43.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:43.547 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:43.575 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:43.653 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:04:43.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:40.965740+0000 mgr.y (mgr.14550) 139 : cluster [DBG] pgmap v123: 548 pgs: 32 creating+peering, 452 active+clean, 64 unknown; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 8.5 KiB/s wr, 14 op/s 2024-02-19T21:04:43.674 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:04:43] "GET /metrics HTTP/1.1" 200 229240 "" "Prometheus/2.33.4" 2024-02-19T21:04:43.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:40.965740+0000 mgr.y (mgr.14550) 139 : cluster [DBG] pgmap v123: 548 pgs: 32 creating+peering, 452 active+clean, 64 unknown; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 8.5 KiB/s wr, 14 op/s 2024-02-19T21:04:43.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.318136+0000 mgr.y (mgr.14550) 140 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.0"}]: dispatch 2024-02-19T21:04:43.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.318814+0000 mgr.y (mgr.14550) 141 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.1"}]: dispatch 2024-02-19T21:04:43.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.319269+0000 mgr.y (mgr.14550) 142 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.2"}]: dispatch 2024-02-19T21:04:43.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.319651+0000 mgr.y (mgr.14550) 143 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.3"}]: dispatch 2024-02-19T21:04:43.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.320038+0000 mgr.y (mgr.14550) 144 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.4"}]: dispatch 2024-02-19T21:04:43.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.320426+0000 mgr.y (mgr.14550) 145 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.5"}]: dispatch 2024-02-19T21:04:43.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.320803+0000 mgr.y (mgr.14550) 146 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.6"}]: dispatch 2024-02-19T21:04:43.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.321177+0000 mgr.y (mgr.14550) 147 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.7"}]: dispatch 2024-02-19T21:04:43.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.321579+0000 mgr.y (mgr.14550) 148 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.8"}]: dispatch 2024-02-19T21:04:43.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.321973+0000 mgr.y (mgr.14550) 149 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.9"}]: dispatch 2024-02-19T21:04:43.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.431979+0000 mon.a (mon.0) 815 : audit [INF] from='client.25219 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.433146+0000 mon.b (mon.1) 124 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.318136+0000 mgr.y (mgr.14550) 140 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.0"}]: dispatch 2024-02-19T21:04:43.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.318814+0000 mgr.y (mgr.14550) 141 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.1"}]: dispatch 2024-02-19T21:04:43.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.319269+0000 mgr.y (mgr.14550) 142 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.2"}]: dispatch 2024-02-19T21:04:43.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.319651+0000 mgr.y (mgr.14550) 143 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.3"}]: dispatch 2024-02-19T21:04:43.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.320038+0000 mgr.y (mgr.14550) 144 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.4"}]: dispatch 2024-02-19T21:04:43.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.320426+0000 mgr.y (mgr.14550) 145 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.5"}]: dispatch 2024-02-19T21:04:43.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.320803+0000 mgr.y (mgr.14550) 146 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.6"}]: dispatch 2024-02-19T21:04:43.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.321177+0000 mgr.y (mgr.14550) 147 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.7"}]: dispatch 2024-02-19T21:04:43.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.321579+0000 mgr.y (mgr.14550) 148 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.8"}]: dispatch 2024-02-19T21:04:43.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.321973+0000 mgr.y (mgr.14550) 149 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.9"}]: dispatch 2024-02-19T21:04:43.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.431979+0000 mon.a (mon.0) 815 : audit [INF] from='client.25219 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.433146+0000 mon.b (mon.1) 124 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.433496+0000 mon.b (mon.1) 125 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi163-83960-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:42.532653+0000 osd.0 (osd.0) 63 : cluster [DBG] 28.2 deep-scrub starts 2024-02-19T21:04:43.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.433496+0000 mon.b (mon.1) 125 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi163-83960-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:42.532653+0000 osd.0 (osd.0) 63 : cluster [DBG] 28.2 deep-scrub starts 2024-02-19T21:04:43.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:42.535592+0000 osd.0 (osd.0) 64 : cluster [DBG] 28.2 deep-scrub ok 2024-02-19T21:04:43.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:42.678106+0000 mon.a (mon.0) 816 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:04:43.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.685062+0000 mon.a (mon.0) 817 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-19T21:04:43.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.690680+0000 mon.a (mon.0) 818 : audit [INF] from='client.25213 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi163-83845-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:43.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.690785+0000 mon.a (mon.0) 819 : audit [INF] from='client.25219 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:43.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:42.691900+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-19T21:04:43.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.694993+0000 mon.a (mon.0) 821 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:43.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.695143+0000 mon.b (mon.1) 126 : audit [INF] from='client.? v1:172.21.15.163:0/4074496902' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.695420+0000 mon.b (mon.1) 127 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.695467+0000 mon.a (mon.0) 822 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:43.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.695672+0000 mon.a (mon.0) 823 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.695968+0000 mon.a (mon.0) 824 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi163-83960-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:42.535592+0000 osd.0 (osd.0) 64 : cluster [DBG] 28.2 deep-scrub ok 2024-02-19T21:04:43.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:42.678106+0000 mon.a (mon.0) 816 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:04:43.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.685062+0000 mon.a (mon.0) 817 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-19T21:04:43.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.690680+0000 mon.a (mon.0) 818 : audit [INF] from='client.25213 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi163-83845-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:43.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.690785+0000 mon.a (mon.0) 819 : audit [INF] from='client.25219 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:43.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:42.691900+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-19T21:04:43.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.694993+0000 mon.a (mon.0) 821 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:43.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.695143+0000 mon.b (mon.1) 126 : audit [INF] from='client.? v1:172.21.15.163:0/4074496902' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.695420+0000 mon.b (mon.1) 127 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.695467+0000 mon.a (mon.0) 822 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:43.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.695672+0000 mon.a (mon.0) 823 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.695968+0000 mon.a (mon.0) 824 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi163-83960-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.696891+0000 mon.a (mon.0) 825 : audit [INF] from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.696891+0000 mon.a (mon.0) 825 : audit [INF] from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.697162+0000 mon.a (mon.0) 826 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.703867+0000 mon.b (mon.1) 128 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.704186+0000 mon.b (mon.1) 129 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi163-83864-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.704363+0000 mon.b (mon.1) 130 : audit [INF] from='client.? v1:172.21.15.163:0/3188847468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:42.913900+0000 osd.1 (osd.1) 71 : cluster [DBG] 28.1 deep-scrub starts 2024-02-19T21:04:43.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:42.918295+0000 osd.1 (osd.1) 72 : cluster [DBG] 28.1 deep-scrub ok 2024-02-19T21:04:43.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.964567+0000 mon.a (mon.0) 827 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.965067+0000 mon.a (mon.0) 828 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi163-83864-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:42.965397+0000 mon.a (mon.0) 829 : audit [INF] from='client.25261 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:42.966400+0000 mgr.y (mgr.14550) 150 : cluster [DBG] pgmap v126: 748 pgs: 32 creating+peering, 420 active+clean, 296 unknown; 6.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5.5 KiB/s wr, 13 op/s 2024-02-19T21:04:43.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:43.317966+0000 osd.7 (osd.7) 63 : cluster [DBG] 28.5 deep-scrub starts 2024-02-19T21:04:43.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:43.320862+0000 osd.7 (osd.7) 64 : cluster [DBG] 28.5 deep-scrub ok 2024-02-19T21:04:43.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.383225+0000 mon.c (mon.2) 48 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:43.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:43.643379+0000 osd.6 (osd.6) 53 : cluster [DBG] 28.0 deep-scrub starts 2024-02-19T21:04:43.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:43.646538+0000 osd.6 (osd.6) 54 : cluster [DBG] 28.0 deep-scrub ok 2024-02-19T21:04:43.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.697162+0000 mon.a (mon.0) 826 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.703867+0000 mon.b (mon.1) 128 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.704186+0000 mon.b (mon.1) 129 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi163-83864-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.704363+0000 mon.b (mon.1) 130 : audit [INF] from='client.? v1:172.21.15.163:0/3188847468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:42.913900+0000 osd.1 (osd.1) 71 : cluster [DBG] 28.1 deep-scrub starts 2024-02-19T21:04:43.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:42.918295+0000 osd.1 (osd.1) 72 : cluster [DBG] 28.1 deep-scrub ok 2024-02-19T21:04:43.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.964567+0000 mon.a (mon.0) 827 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.965067+0000 mon.a (mon.0) 828 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi163-83864-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:43.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:42.965397+0000 mon.a (mon.0) 829 : audit [INF] from='client.25261 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:43.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:42.966400+0000 mgr.y (mgr.14550) 150 : cluster [DBG] pgmap v126: 748 pgs: 32 creating+peering, 420 active+clean, 296 unknown; 6.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5.5 KiB/s wr, 13 op/s 2024-02-19T21:04:43.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:43.317966+0000 osd.7 (osd.7) 63 : cluster [DBG] 28.5 deep-scrub starts 2024-02-19T21:04:43.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:43.320862+0000 osd.7 (osd.7) 64 : cluster [DBG] 28.5 deep-scrub ok 2024-02-19T21:04:43.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.383225+0000 mon.c (mon.2) 48 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:43.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:43.643379+0000 osd.6 (osd.6) 53 : cluster [DBG] 28.0 deep-scrub starts 2024-02-19T21:04:43.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:43.646538+0000 osd.6 (osd.6) 54 : cluster [DBG] 28.0 deep-scrub ok 2024-02-19T21:04:44.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:40.965740+0000 mgr.y (mgr.14550) 139 : cluster [DBG] pgmap v123: 548 pgs: 32 creating+peering, 452 active+clean, 64 unknown; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 8.5 KiB/s wr, 14 op/s 2024-02-19T21:04:44.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.318136+0000 mgr.y (mgr.14550) 140 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.0"}]: dispatch 2024-02-19T21:04:44.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.318814+0000 mgr.y (mgr.14550) 141 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.1"}]: dispatch 2024-02-19T21:04:44.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.319269+0000 mgr.y (mgr.14550) 142 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.2"}]: dispatch 2024-02-19T21:04:44.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.319651+0000 mgr.y (mgr.14550) 143 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.3"}]: dispatch 2024-02-19T21:04:44.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.320038+0000 mgr.y (mgr.14550) 144 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.4"}]: dispatch 2024-02-19T21:04:44.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.320426+0000 mgr.y (mgr.14550) 145 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.5"}]: dispatch 2024-02-19T21:04:44.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.320803+0000 mgr.y (mgr.14550) 146 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.6"}]: dispatch 2024-02-19T21:04:44.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.321177+0000 mgr.y (mgr.14550) 147 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.7"}]: dispatch 2024-02-19T21:04:44.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.321579+0000 mgr.y (mgr.14550) 148 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.8"}]: dispatch 2024-02-19T21:04:44.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.321973+0000 mgr.y (mgr.14550) 149 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.9"}]: dispatch 2024-02-19T21:04:44.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.431979+0000 mon.a (mon.0) 815 : audit [INF] from='client.25219 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:44.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.433146+0000 mon.b (mon.1) 124 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:44.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.433496+0000 mon.b (mon.1) 125 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi163-83960-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:44.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:42.532653+0000 osd.0 (osd.0) 63 : cluster [DBG] 28.2 deep-scrub starts 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:42.535592+0000 osd.0 (osd.0) 64 : cluster [DBG] 28.2 deep-scrub ok 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:42.678106+0000 mon.a (mon.0) 816 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.685062+0000 mon.a (mon.0) 817 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.690680+0000 mon.a (mon.0) 818 : audit [INF] from='client.25213 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi163-83845-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.690785+0000 mon.a (mon.0) 819 : audit [INF] from='client.25219 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:42.691900+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.694993+0000 mon.a (mon.0) 821 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.695143+0000 mon.b (mon.1) 126 : audit [INF] from='client.? v1:172.21.15.163:0/4074496902' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.695420+0000 mon.b (mon.1) 127 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.695467+0000 mon.a (mon.0) 822 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.695672+0000 mon.a (mon.0) 823 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.695968+0000 mon.a (mon.0) 824 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi163-83960-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.696891+0000 mon.a (mon.0) 825 : audit [INF] from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.697162+0000 mon.a (mon.0) 826 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.703867+0000 mon.b (mon.1) 128 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:44.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.704186+0000 mon.b (mon.1) 129 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi163-83864-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:44.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.704363+0000 mon.b (mon.1) 130 : audit [INF] from='client.? v1:172.21.15.163:0/3188847468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:44.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:42.913900+0000 osd.1 (osd.1) 71 : cluster [DBG] 28.1 deep-scrub starts 2024-02-19T21:04:44.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:42.918295+0000 osd.1 (osd.1) 72 : cluster [DBG] 28.1 deep-scrub ok 2024-02-19T21:04:44.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.964567+0000 mon.a (mon.0) 827 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:44.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.965067+0000 mon.a (mon.0) 828 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi163-83864-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:44.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:42.965397+0000 mon.a (mon.0) 829 : audit [INF] from='client.25261 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:44.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:42.966400+0000 mgr.y (mgr.14550) 150 : cluster [DBG] pgmap v126: 748 pgs: 32 creating+peering, 420 active+clean, 296 unknown; 6.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 5.5 KiB/s wr, 13 op/s 2024-02-19T21:04:44.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:43.317966+0000 osd.7 (osd.7) 63 : cluster [DBG] 28.5 deep-scrub starts 2024-02-19T21:04:44.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:43.320862+0000 osd.7 (osd.7) 64 : cluster [DBG] 28.5 deep-scrub ok 2024-02-19T21:04:44.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.383225+0000 mon.c (mon.2) 48 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:44.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:43.643379+0000 osd.6 (osd.6) 53 : cluster [DBG] 28.0 deep-scrub starts 2024-02-19T21:04:44.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:43.646538+0000 osd.6 (osd.6) 54 : cluster [DBG] 28.0 deep-scrub ok 2024-02-19T21:04:44.127 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:44.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:44.231 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:44.262 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:44.337 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:44.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:44.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:44.365 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:04:44.365 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:04:44.365 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-02-19T21:04:44.365 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-02-19T21:04:44.365 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:04:44.371 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:44.441 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:44.469 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:44.488 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:44.491 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:44.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:44.592 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:44.618 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:44.637 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:44.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:44.678 INFO:tasks.workunit.client.0.smithi163.stdout:-02-19T21:04:40.857684+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:44.678 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:40.863718+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:44.678 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:40.872645+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:44.678 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:40.887858+0000 mon.c [INF] from='client.? v1:172.21.15.163:0/1426051668' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-19T21:04:44.678 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:40.891156+0000 client.admin [INF] onexx 2024-02-19T21:04:44.678 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:41.365305+0000 mon.a [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-02-19T21:04:44.678 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.431979+0000 mon.a [INF] from='client.25219 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:44.678 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.433146+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:44.678 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.433496+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi163-83960-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:44.678 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.678106+0000 mon.a [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:04:44.678 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.685062+0000 mon.a [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-02-19T21:04:44.678 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.690680+0000 mon.a [INF] from='client.25213 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi163-83845-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:44.678 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.690785+0000 mon.a [INF] from='client.25219 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:44.679 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.694993+0000 mon.a [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:44.679 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.695143+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/4074496902' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:44.679 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.695420+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:44.679 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.695467+0000 mon.a [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:44.740 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:44.766 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:44.785 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:44.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:44.889 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:44.918 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:45.041 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:45.044 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:45.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:45.145 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:45.172 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:45.190 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:45.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:45.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:45.294 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:04:45.320 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:45.338 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:04:45.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:45.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:04:45.442 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:04:45.459 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:04:45.459 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:04:45.459 INFO:tasks.thrashosds.thrasher:fixing pg num pool device_health_metrics 2024-02-19T21:04:45.459 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:04:45.468 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:45.487 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:04:45.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:45.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:04:45.590 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:04:45.616 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:45.661 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.695672+0000 mon.a [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "o open_pools_parallel: process_1_[84094]: starting. 2024-02-19T21:04:45.661 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_1_[84094]: creating pool ceph_test_rados_open_pools_parallel.smithi163-84048 2024-02-19T21:04:45.661 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_1_[84094]: created object 0... 2024-02-19T21:04:45.661 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_1_[84094]: created object 25... 2024-02-19T21:04:45.661 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_1_[84094]: created object 49... 2024-02-19T21:04:45.661 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_1_[84094]: finishing. 2024-02-19T21:04:45.661 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_1_[84094]: shutting down. 2024-02-19T21:04:45.662 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_2_[84095]: starting. 2024-02-19T21:04:45.662 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_2_[84095]: rados_pool_create. 2024-02-19T21:04:45.662 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_2_[84095]: rados_ioctx_create. 2024-02-19T21:04:45.662 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_2_[84095]: shutting down. 2024-02-19T21:04:45.662 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: ******************************* 2024-02-19T21:04:45.662 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_3_[86324]: starting. 2024-02-19T21:04:45.662 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_3_[86324]: creating pool ceph_test_rados_open_pools_parallel.smithi163-84048 2024-02-19T21:04:45.662 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_3_[86324]: created object 0... 2024-02-19T21:04:45.662 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_3_[86324]: created object 25... 2024-02-19T21:04:45.662 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_3_[86324]: created object 49... 2024-02-19T21:04:45.662 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_3_[86324]: finishing. 2024-02-19T21:04:45.662 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_3_[86324]: shutting down. 2024-02-19T21:04:45.662 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: ******************************* 2024-02-19T21:04:45.663 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_4_[86326]: starting. 2024-02-19T21:04:45.663 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_4_[86326]: rados_pool_create. 2024-02-19T21:04:45.663 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_4_[86326]: rados_ioctx_create. 2024-02-19T21:04:45.663 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: process_4_[86326]: shutting down. 2024-02-19T21:04:45.663 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: ******************************* 2024-02-19T21:04:45.663 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: ******************************* 2024-02-19T21:04:45.663 INFO:tasks.workunit.client.0.smithi163.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-02-19T21:04:45.663 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_1_[84091]: starting. 2024-02-19T21:04:45.663 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_1_[84091]: creating pool ceph_test_rados_delete_pools_parallel.smithi163-84052 2024-02-19T21:04:45.663 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_1_[84091]: created object 0... 2024-02-19T21:04:45.663 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_1_[84091]: created object 25... 2024-02-19T21:04:45.663 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_1_[84091]: created object 49... 2024-02-19T21:04:45.663 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_1_[84091]: finishing. 2024-02-19T21:04:45.663 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_1_[84091]: shutting down. 2024-02-19T21:04:45.664 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_2_[84093]: starting. 2024-02-19T21:04:45.664 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_2_[84093]: deleting pool ceph_test_rados_delete_pools_parallel.smithi163-84052 2024-02-19T21:04:45.664 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_2_[84093]: shutting down. 2024-02-19T21:04:45.664 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: ******************************* 2024-02-19T21:04:45.664 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_3_[86153]: starting. 2024-02-19T21:04:45.664 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_3_[86153]: creating pool ceph_test_rados_delete_pools_parallel.smithi163-84052 2024-02-19T21:04:45.664 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_3_[86153]: created object 0... 2024-02-19T21:04:45.664 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_3_[86153]: created object 25... 2024-02-19T21:04:45.664 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_3_[86153]: created object 49... 2024-02-19T21:04:45.664 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_3_[86153]: finishing. 2024-02-19T21:04:45.664 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_3_[86153]: shutting down. 2024-02-19T21:04:45.664 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: ******************************* 2024-02-19T21:04:45.664 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_5_[86155]: starting. 2024-02-19T21:04:45.664 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_5_[86155]: listing objects. 2024-02-19T21:04:45.665 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_5_[86155]: listed object 0... 2024-02-19T21:04:45.665 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_5_[86155]: listed object 25... 2024-02-19T21:04:45.665 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_5_[86155]: saw 50 objects 2024-02-19T21:04:45.665 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_5_[86155]: shutting down. 2024-02-19T21:04:45.665 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: ******************************* 2024-02-19T21:04:45.665 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_4_[86154]: starting. 2024-02-19T21:04:45.665 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_4_[86154]: deleting pool ceph_test_rados_delete_pools_parallel.smithi163-84052 2024-02-19T21:04:45.665 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: process_4_[86154]: shutting down. 2024-02-19T21:04:45.665 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: ******************************* 2024-02-19T21:04:45.665 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: ******************************* 2024-02-19T21:04:45.665 INFO:tasks.workunit.client.0.smithi163.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-02-19T21:04:45.689 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:04:45.693 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:45.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:04:45.705 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-19T21:04:45.746 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:45.795 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:04:45.823 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:45.847 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:04:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:45.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:04:45.919 INFO:tasks.workunit.client.0.smithi163.stdout:sd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:45.920 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.695968+0000 mon.a [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi163-83960-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:45.920 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.696891+0000 mon.a [INF] from='client.25240 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:45.920 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.697162+0000 mon.a [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:45.920 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.703867+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:45.921 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.704186+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi163-83864-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:45.921 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.704363+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/3188847468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:45.921 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.964567+0000 mon.a [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:45.921 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.965067+0000 mon.a [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi163-83864-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:45.921 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:42.965397+0000 mon.a [INF] from='client.25261 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:45.921 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.697582+0000 mon.a [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:45.922 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.697657+0000 mon.a [INF] from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi163-83960-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:45.923 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.697726+0000 mon.a [INF] from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:45.923 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.697785+0000 mon.a [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:45.923 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.697960+0000 mon.a [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:45.923 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.698049+0000 mon.a [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi163-83864-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:45.923 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.698106+0000 mon.a [INF] from='client.25261 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:45.924 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.700313+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi163-83960-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:45.924 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.701421+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi163-83872-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:45.925 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.701656+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi163-83968-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:45.925 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.702993+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/3338391635' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:45.925 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.705609+0000 mon.a [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi163-83960-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:45.926 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.706183+0000 mon.a [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi163-83872-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:45.926 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.706548+0000 mon.a [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi163-83968-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:45.927 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.710204+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi163-83864-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:45.927 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.710627+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:45.928 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.712951+0000 mon.a [INF] from='client.25219 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:45.929 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:04:45.956 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:45.984 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:46.002 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:46.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:46.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:46.106 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:46.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.697582+0000 mon.a (mon.0) 830 : audit [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.129 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.697657+0000 mon.a (mon.0) 831 : audit [INF] from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi163-83960-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.130 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.697726+0000 mon.a (mon.0) 832 : audit [INF] from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:46.130 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.697785+0000 mon.a (mon.0) 833 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:46.130 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.697960+0000 mon.a (mon.0) 834 : audit [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.130 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.698049+0000 mon.a (mon.0) 835 : audit [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi163-83864-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.130 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.698106+0000 mon.a (mon.0) 836 : audit [INF] from='client.25261 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:46.130 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:43.698198+0000 mon.a (mon.0) 837 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-02-19T21:04:46.130 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.700313+0000 mon.b (mon.1) 131 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi163-83960-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:46.130 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.701421+0000 mon.b (mon.1) 132 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi163-83872-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:46.130 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.701656+0000 mon.b (mon.1) 133 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi163-83968-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:46.131 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.697582+0000 mon.a (mon.0) 830 : audit [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.131 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.697657+0000 mon.a (mon.0) 831 : audit [INF] from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi163-83960-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.131 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.697726+0000 mon.a (mon.0) 832 : audit [INF] from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:46.131 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.697785+0000 mon.a (mon.0) 833 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:46.131 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.697960+0000 mon.a (mon.0) 834 : audit [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.131 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.698049+0000 mon.a (mon.0) 835 : audit [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi163-83864-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.131 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.698106+0000 mon.a (mon.0) 836 : audit [INF] from='client.25261 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:46.131 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:43.698198+0000 mon.a (mon.0) 837 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-02-19T21:04:46.131 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.700313+0000 mon.b (mon.1) 131 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi163-83960-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:46.131 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.701421+0000 mon.b (mon.1) 132 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi163-83872-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:46.131 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.701656+0000 mon.b (mon.1) 133 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi163-83968-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:46.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.702993+0000 mon.b (mon.1) 134 : audit [INF] from='client.? v1:172.21.15.163:0/3338391635' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:46.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.705609+0000 mon.a (mon.0) 838 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi163-83960-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:46.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.706183+0000 mon.a (mon.0) 839 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi163-83872-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:46.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.706548+0000 mon.a (mon.0) 840 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi163-83968-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:46.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.710204+0000 mon.b (mon.1) 135 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi163-83864-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:46.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.710627+0000 mon.b (mon.1) 136 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:46.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.712951+0000 mon.a (mon.0) 841 : audit [INF] from='client.25219 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:46.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.713356+0000 mon.a (mon.0) 842 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi163-83864-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:46.132 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.714056+0000 mon.a (mon.0) 843 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:46.133 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.716509+0000 mon.b (mon.1) 137 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:46.133 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.718697+0000 mon.a (mon.0) 844 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:46.133 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.702993+0000 mon.b (mon.1) 134 : audit [INF] from='client.? v1:172.21.15.163:0/3338391635' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:46.133 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.705609+0000 mon.a (mon.0) 838 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi163-83960-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:46.133 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.706183+0000 mon.a (mon.0) 839 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi163-83872-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:46.133 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.706548+0000 mon.a (mon.0) 840 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi163-83968-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:46.133 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.710204+0000 mon.b (mon.1) 135 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi163-83864-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:46.133 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.710627+0000 mon.b (mon.1) 136 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:46.133 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.712951+0000 mon.a (mon.0) 841 : audit [INF] from='client.25219 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:46.134 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.713356+0000 mon.a (mon.0) 842 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi163-83864-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:46.134 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.714056+0000 mon.a (mon.0) 843 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:46.134 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.716509+0000 mon.b (mon.1) 137 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:46.134 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.718697+0000 mon.a (mon.0) 844 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:46.134 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.775017+0000 mon.b (mon.1) 138 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.134 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.779470+0000 mon.a (mon.0) 845 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.134 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.782513+0000 mon.b (mon.1) 139 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.134 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.784786+0000 mon.a (mon.0) 846 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:46.134 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.787118+0000 mon.a (mon.0) 847 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.134 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.788219+0000 mon.b (mon.1) 140 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi163-83896-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:46.134 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.789505+0000 mon.a (mon.0) 848 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:46.134 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.793212+0000 mon.a (mon.0) 849 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:46.134 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:43.795621+0000 mon.a (mon.0) 850 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi163-83896-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:46.134 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:43.864427+0000 osd.7 (osd.7) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:46.135 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:43.864480+0000 osd.7 (osd.7) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:46.135 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:43.923400+0000 osd.1 (osd.1) 73 : cluster [DBG] 28.7 deep-scrub starts 2024-02-19T21:04:46.135 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:43.961309+0000 osd.1 (osd.1) 74 : cluster [DBG] 28.7 deep-scrub ok 2024-02-19T21:04:46.135 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:44.384128+0000 mon.c (mon.2) 49 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:46.135 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:44.685828+0000 mon.c (mon.2) 50 : audit [INF] from='client.? v1:172.21.15.163:0/1426051668' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-19T21:04:46.135 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:44.685936+0000 client.admin (client.35186) 0 : cluster [INF] threexx 2024-02-19T21:04:46.135 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:44.686094+0000 mon.a (mon.0) 851 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-19T21:04:46.135 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.775017+0000 mon.b (mon.1) 138 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.135 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.779470+0000 mon.a (mon.0) 845 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.782513+0000 mon.b (mon.1) 139 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.784786+0000 mon.a (mon.0) 846 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:46.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.787118+0000 mon.a (mon.0) 847 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.788219+0000 mon.b (mon.1) 140 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi163-83896-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:46.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.789505+0000 mon.a (mon.0) 848 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:46.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.793212+0000 mon.a (mon.0) 849 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:46.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:43.795621+0000 mon.a (mon.0) 850 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi163-83896-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:46.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:43.864427+0000 osd.7 (osd.7) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:46.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:43.864480+0000 osd.7 (osd.7) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:46.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:43.923400+0000 osd.1 (osd.1) 73 : cluster [DBG] 28.7 deep-scrub starts 2024-02-19T21:04:46.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:43.961309+0000 osd.1 (osd.1) 74 : cluster [DBG] 28.7 deep-scrub ok 2024-02-19T21:04:46.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:44.384128+0000 mon.c (mon.2) 49 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:46.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:44.685828+0000 mon.c (mon.2) 50 : audit [INF] from='client.? v1:172.21.15.163:0/1426051668' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-19T21:04:46.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:44.685936+0000 client.admin (client.35186) 0 : cluster [INF] threexx 2024-02-19T21:04:46.136 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:44.686094+0000 mon.a (mon.0) 851 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-19T21:04:46.137 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:46.159 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:46.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:46.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:46.263 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:46.291 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:46.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.697582+0000 mon.a (mon.0) 830 : audit [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.697657+0000 mon.a (mon.0) 831 : audit [INF] from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi163-83960-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.697726+0000 mon.a (mon.0) 832 : audit [INF] from='client.25240 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:46.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.697785+0000 mon.a (mon.0) 833 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:46.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.697960+0000 mon.a (mon.0) 834 : audit [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.698049+0000 mon.a (mon.0) 835 : audit [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi163-83864-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.698106+0000 mon.a (mon.0) 836 : audit [INF] from='client.25261 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:46.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:43.698198+0000 mon.a (mon.0) 837 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-02-19T21:04:46.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.700313+0000 mon.b (mon.1) 131 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi163-83960-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:46.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.701421+0000 mon.b (mon.1) 132 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi163-83872-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:46.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.701656+0000 mon.b (mon.1) 133 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi163-83968-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:46.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.702993+0000 mon.b (mon.1) 134 : audit [INF] from='client.? v1:172.21.15.163:0/3338391635' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:46.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.705609+0000 mon.a (mon.0) 838 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi163-83960-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:46.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.706183+0000 mon.a (mon.0) 839 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi163-83872-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:46.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.706548+0000 mon.a (mon.0) 840 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi163-83968-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:46.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.710204+0000 mon.b (mon.1) 135 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi163-83864-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:46.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.710627+0000 mon.b (mon.1) 136 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:46.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.712951+0000 mon.a (mon.0) 841 : audit [INF] from='client.25219 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:46.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.713356+0000 mon.a (mon.0) 842 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi163-83864-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:46.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.714056+0000 mon.a (mon.0) 843 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:46.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.716509+0000 mon.b (mon.1) 137 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:46.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.718697+0000 mon.a (mon.0) 844 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:46.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.775017+0000 mon.b (mon.1) 138 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.779470+0000 mon.a (mon.0) 845 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.782513+0000 mon.b (mon.1) 139 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.784786+0000 mon.a (mon.0) 846 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:46.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.787118+0000 mon.a (mon.0) 847 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.788219+0000 mon.b (mon.1) 140 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi163-83896-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:46.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.789505+0000 mon.a (mon.0) 848 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:46.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.793212+0000 mon.a (mon.0) 849 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:46.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:43.795621+0000 mon.a (mon.0) 850 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi163-83896-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:46.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:43.864427+0000 osd.7 (osd.7) 65 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:04:46.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:43.864480+0000 osd.7 (osd.7) 66 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:04:46.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:43.923400+0000 osd.1 (osd.1) 73 : cluster [DBG] 28.7 deep-scrub starts 2024-02-19T21:04:46.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:43.961309+0000 osd.1 (osd.1) 74 : cluster [DBG] 28.7 deep-scrub ok 2024-02-19T21:04:46.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:44.384128+0000 mon.c (mon.2) 49 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:46.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:44.685828+0000 mon.c (mon.2) 50 : audit [INF] from='client.? v1:172.21.15.163:0/1426051668' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-19T21:04:46.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:44.685936+0000 client.admin (client.35186) 0 : cluster [INF] threexx 2024-02-19T21:04:46.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:44.686094+0000 mon.a (mon.0) 851 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-19T21:04:46.389 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:46.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:46.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:46.412 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:46.494 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:46.526 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:46.547 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:46.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:46.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:46.651 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:04:46.680 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:46.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.957+0000 7f31fc83e700 1 -- 172.21.15.163:0/369538574 >> v1:172.21.15.163:6790/0 conn(0x7f31f8104820 legacy=0x7f31f8104c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:46.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.958+0000 7f31fc83e700 1 -- 172.21.15.163:0/369538574 shutdown_connections 2024-02-19T21:04:46.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.958+0000 7f31fc83e700 1 -- 172.21.15.163:0/369538574 >> 172.21.15.163:0/369538574 conn(0x7f31f8063b20 msgr2=0x7f31f8067230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:46.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.960+0000 7f31fc83e700 1 -- 172.21.15.163:0/369538574 shutdown_connections 2024-02-19T21:04:46.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.961+0000 7f31fc83e700 1 -- 172.21.15.163:0/369538574 wait complete. 2024-02-19T21:04:46.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.961+0000 7f31fc83e700 1 librados: starting msgr at 2024-02-19T21:04:46.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.961+0000 7f31fc83e700 1 librados: starting objecter 2024-02-19T21:04:46.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.962+0000 7f31fc83e700 1 Processor -- start 2024-02-19T21:04:46.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.962+0000 7f31fc83e700 1 -- start start 2024-02-19T21:04:46.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.962+0000 7f31fc83e700 1 librados: setting wanted keys 2024-02-19T21:04:46.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.962+0000 7f31fc83e700 1 librados: calling monclient init 2024-02-19T21:04:46.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.962+0000 7f31fc83e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f31f81261a0 con 0x7f31f81075c0 2024-02-19T21:04:46.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.962+0000 7f31fc83e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f31f8127340 con 0x7f31f8104820 2024-02-19T21:04:46.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.962+0000 7f31fc83e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f31f81284e0 con 0x7f31f810b130 2024-02-19T21:04:46.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.964+0000 7f31f6d9d700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f31f810b130 0x7f31f8124a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:54166/0 (socket says 172.21.15.163:54166) 2024-02-19T21:04:46.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.964+0000 7f31f6d9d700 1 -- 172.21.15.163:0/1009994265 learned_addr learned my addr 172.21.15.163:0/1009994265 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:04:46.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.964+0000 7f31e77fe700 10 client.?.objecter ms_handle_connect 0x7f31f810b130 2024-02-19T21:04:46.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.964+0000 7f31e77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:46.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.965+0000 7f31e77fe700 10 client.?.objecter ms_handle_connect 0x7f31f8104820 2024-02-19T21:04:46.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.965+0000 7f31e77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:46.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.965+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184654884 0 0) 0x7f31f8127340 con 0x7f31f8104820 2024-02-19T21:04:46.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.965+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31e0003110 con 0x7f31f8104820 2024-02-19T21:04:46.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125647520 0 0) 0x7f31f81284e0 con 0x7f31f810b130 2024-02-19T21:04:46.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31f8127340 con 0x7f31f810b130 2024-02-19T21:04:46.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31e77fe700 10 client.?.objecter ms_handle_connect 0x7f31f81075c0 2024-02-19T21:04:46.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31e77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:46.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 512192018 0 0) 0x7f31f8127340 con 0x7f31f810b130 2024-02-19T21:04:46.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31f81284e0 con 0x7f31f810b130 2024-02-19T21:04:46.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f31f0001b00 con 0x7f31f810b130 2024-02-19T21:04:46.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 604684047 0 0) 0x7f31f81284e0 con 0x7f31f810b130 2024-02-19T21:04:46.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 >> v1:172.21.15.193:6789/0 conn(0x7f31f8104820 legacy=0x7f31f8103b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:46.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 >> v1:172.21.15.163:6789/0 conn(0x7f31f81075c0 legacy=0x7f31f8121350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:46.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f81296c0 con 0x7f31f810b130 2024-02-19T21:04:46.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31fc83e700 1 -- 172.21.15.163:0/1009994265 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f8127530 con 0x7f31f810b130 2024-02-19T21:04:46.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31fc83e700 10 client.35231.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:46.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31fc83e700 1 -- 172.21.15.163:0/1009994265 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f81279a0 con 0x7f31f810b130 2024-02-19T21:04:46.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31fc83e700 1 librados: init done 2024-02-19T21:04:46.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f31f0002180 con 0x7f31f810b130 2024-02-19T21:04:46.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.966+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f31f0005e10 con 0x7f31f810b130 2024-02-19T21:04:46.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.968+0000 7f31fc83e700 1 -- 172.21.15.163:0/1009994265 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d8004f50 con 0x7f31f810b130 2024-02-19T21:04:46.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.968+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f31f0018cc0 con 0x7f31f810b130 2024-02-19T21:04:46.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.969+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(107..107 src has 43..107) v4 ==== 16167+0+0 (unknown 4024708234 0 0) 0x7f31f007af40 con 0x7f31f810b130 2024-02-19T21:04:46.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.969+0000 7f31e77fe700 10 client.35231.objecter ms_dispatch 0x7f31f8001550 osd_map(107..107 src has 43..107) v4 2024-02-19T21:04:46.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.969+0000 7f31e77fe700 3 client.35231.objecter handle_osd_map got epochs [107,107] > 0 2024-02-19T21:04:46.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.969+0000 7f31e77fe700 3 client.35231.objecter handle_osd_map decoding full epoch 107 2024-02-19T21:04:46.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.969+0000 7f31e77fe700 20 client.35231.objecter dump_active .. 0 homeless 2024-02-19T21:04:46.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.976+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f31f004c4e0 con 0x7f31f810b130 2024-02-19T21:04:46.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:46.988+0000 7f31e77fe700 10 client.35231.objecter ms_handle_connect 0x7f31e0063b30 2024-02-19T21:04:46.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:44.677003+0000 osd.6 (osd.6) 55 : cluster [DBG] 28.4 deep-scrub starts 2024-02-19T21:04:46.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:44.678214+0000 osd.6 (osd.6) 56 : cluster [DBG] 28.4 deep-scrub ok 2024-02-19T21:04:46.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:44.967751+0000 mgr.y (mgr.14550) 151 : cluster [DBG] pgmap v129: 484 pgs: 17 creating+peering, 15 creating+activating, 452 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 527 op/s 2024-02-19T21:04:46.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:45.197016+0000 osd.5 (osd.5) 62 : cluster [DBG] 28.3 deep-scrub starts 2024-02-19T21:04:46.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:45.203061+0000 osd.5 (osd.5) 63 : cluster [DBG] 28.3 deep-scrub ok 2024-02-19T21:04:46.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:45.352882+0000 osd.7 (osd.7) 67 : cluster [DBG] 28.6 deep-scrub starts 2024-02-19T21:04:46.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:45.355817+0000 osd.7 (osd.7) 68 : cluster [DBG] 28.6 deep-scrub ok 2024-02-19T21:04:46.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.384953+0000 mon.c (mon.2) 51 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:46.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.424493+0000 mon.a (mon.0) 852 : audit [INF] from='client.25219 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:46.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.425560+0000 mon.b (mon.1) 141 : audit [INF] from='client.? v1:172.21.15.163:0/3338391635' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi163-83854-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-19T21:04:46.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.662201+0000 mon.a (mon.0) 853 : audit [INF] from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]': finished 2024-02-19T21:04:46.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.662373+0000 mon.a (mon.0) 854 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]': finished 2024-02-19T21:04:46.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.662536+0000 mon.a (mon.0) 855 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.662609+0000 mon.a (mon.0) 856 : audit [INF] from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi163-83896-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:45.662770+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-02-19T21:04:46.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.663140+0000 mon.b (mon.1) 142 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:46.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.663316+0000 mon.b (mon.1) 143 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-4", "overlaypool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:46.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.663696+0000 mon.b (mon.1) 144 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi163-83896-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.922118+0000 mon.a (mon.0) 858 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi163-83904-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:46.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.928837+0000 mon.a (mon.0) 859 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-02-19T21:04:46.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.929440+0000 mon.c (mon.2) 52 : audit [INF] from='client.? v1:172.21.15.163:0/1426051668' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-19T21:04:46.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:45.929578+0000 client.admin (client.35186) 0 : cluster [INF] fourxx 2024-02-19T21:04:46.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.935930+0000 mon.a (mon.0) 860 : audit [INF] from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi163-83960-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi163-83960-6"}]': finished 2024-02-19T21:04:46.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.936022+0000 mon.a (mon.0) 861 : audit [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi163-83872-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi163-83872-22"}]': finished 2024-02-19T21:04:46.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.936121+0000 mon.a (mon.0) 862 : audit [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi163-83968-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi163-83968-7"}]': finished 2024-02-19T21:04:46.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.936196+0000 mon.a (mon.0) 863 : audit [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi163-83864-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi163-83864-15"}]': finished 2024-02-19T21:04:46.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:45.936656+0000 mon.a (mon.0) 864 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-02-19T21:04:46.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.939347+0000 mon.a (mon.0) 865 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-19T21:04:46.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.941538+0000 mon.a (mon.0) 866 : audit [INF] from='client.25219 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi163-83854-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-19T21:04:46.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.942166+0000 mon.a (mon.0) 867 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:46.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.942452+0000 mon.a (mon.0) 868 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-4", "overlaypool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:46.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.942673+0000 mon.a (mon.0) 869 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi163-83896-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:45.952730+0000 mon.c (mon.2) 53 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:46.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:46.209417+0000 mon.a (mon.0) 870 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:46.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:46.386106+0000 mon.c (mon.2) 54 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:46.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:44.677003+0000 osd.6 (osd.6) 55 : cluster [DBG] 28.4 deep-scrub starts 2024-02-19T21:04:46.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:44.678214+0000 osd.6 (osd.6) 56 : cluster [DBG] 28.4 deep-scrub ok 2024-02-19T21:04:46.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:44.967751+0000 mgr.y (mgr.14550) 151 : cluster [DBG] pgmap v129: 484 pgs: 17 creating+peering, 15 creating+activating, 452 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 527 op/s 2024-02-19T21:04:46.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:45.197016+0000 osd.5 (osd.5) 62 : cluster [DBG] 28.3 deep-scrub starts 2024-02-19T21:04:46.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:45.203061+0000 osd.5 (osd.5) 63 : cluster [DBG] 28.3 deep-scrub ok 2024-02-19T21:04:46.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:45.352882+0000 osd.7 (osd.7) 67 : cluster [DBG] 28.6 deep-scrub starts 2024-02-19T21:04:46.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:45.355817+0000 osd.7 (osd.7) 68 : cluster [DBG] 28.6 deep-scrub ok 2024-02-19T21:04:46.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.384953+0000 mon.c (mon.2) 51 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:46.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.424493+0000 mon.a (mon.0) 852 : audit [INF] from='client.25219 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:46.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.425560+0000 mon.b (mon.1) 141 : audit [INF] from='client.? v1:172.21.15.163:0/3338391635' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi163-83854-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-19T21:04:46.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.662201+0000 mon.a (mon.0) 853 : audit [INF] from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]': finished 2024-02-19T21:04:46.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.662373+0000 mon.a (mon.0) 854 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]': finished 2024-02-19T21:04:46.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.662536+0000 mon.a (mon.0) 855 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.662609+0000 mon.a (mon.0) 856 : audit [INF] from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi163-83896-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:46.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:45.662770+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-02-19T21:04:46.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.663140+0000 mon.b (mon.1) 142 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:46.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.663316+0000 mon.b (mon.1) 143 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-4", "overlaypool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:46.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.663696+0000 mon.b (mon.1) 144 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi163-83896-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.922118+0000 mon.a (mon.0) 858 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi163-83904-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:46.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.928837+0000 mon.a (mon.0) 859 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-02-19T21:04:46.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.929440+0000 mon.c (mon.2) 52 : audit [INF] from='client.? v1:172.21.15.163:0/1426051668' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-19T21:04:46.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:45.929578+0000 client.admin (client.35186) 0 : cluster [INF] fourxx 2024-02-19T21:04:46.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.935930+0000 mon.a (mon.0) 860 : audit [INF] from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi163-83960-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi163-83960-6"}]': finished 2024-02-19T21:04:46.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.936022+0000 mon.a (mon.0) 861 : audit [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi163-83872-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi163-83872-22"}]': finished 2024-02-19T21:04:46.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.936121+0000 mon.a (mon.0) 862 : audit [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi163-83968-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi163-83968-7"}]': finished 2024-02-19T21:04:46.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.936196+0000 mon.a (mon.0) 863 : audit [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi163-83864-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi163-83864-15"}]': finished 2024-02-19T21:04:46.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:45.936656+0000 mon.a (mon.0) 864 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-02-19T21:04:46.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.939347+0000 mon.a (mon.0) 865 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-19T21:04:46.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.941538+0000 mon.a (mon.0) 866 : audit [INF] from='client.25219 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi163-83854-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-19T21:04:46.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.942166+0000 mon.a (mon.0) 867 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:46.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.942452+0000 mon.a (mon.0) 868 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-4", "overlaypool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:46.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.942673+0000 mon.a (mon.0) 869 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi163-83896-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:46.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:45.952730+0000 mon.c (mon.2) 53 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:46.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:46.209417+0000 mon.a (mon.0) 870 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:46.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:46.386106+0000 mon.c (mon.2) 54 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:47.026 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.713356+0000 mon.a [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi163-83864-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_cod api_io_pp: Running main() from gmock_main.cc 2024-02-19T21:04:47.026 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-02-19T21:04:47.026 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [----------] Global test environment set-up. 2024-02-19T21:04:47.026 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-02-19T21:04:47.026 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: seed 83872 2024-02-19T21:04:47.027 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-02-19T21:04:47.027 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-02-19T21:04:47.027 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-02-19T21:04:47.027 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (270 ms) 2024-02-19T21:04:47.027 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-02-19T21:04:47.027 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (36 ms) 2024-02-19T21:04:47.027 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-02-19T21:04:47.027 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (4 ms) 2024-02-19T21:04:47.027 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-02-19T21:04:47.027 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (8 ms) 2024-02-19T21:04:47.027 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-02-19T21:04:47.027 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (6 ms) 2024-02-19T21:04:47.027 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-02-19T21:04:47.027 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (4 ms) 2024-02-19T21:04:47.027 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-02-19T21:04:47.028 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (5 ms) 2024-02-19T21:04:47.028 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-02-19T21:04:47.028 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (10 ms) 2024-02-19T21:04:47.028 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-02-19T21:04:47.028 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (12 ms) 2024-02-19T21:04:47.028 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-02-19T21:04:47.028 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (4 ms) 2024-02-19T21:04:47.028 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-02-19T21:04:47.028 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (28 ms) 2024-02-19T21:04:47.028 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-02-19T21:04:47.028 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (11 ms) 2024-02-19T21:04:47.028 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-02-19T21:04:47.028 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (7 ms) 2024-02-19T21:04:47.028 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (4 ms) 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (12 ms) 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (5 ms) 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (6 ms) 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (3 ms) 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (4 ms) 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (439 ms total) 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-02-19T21:04:47.029 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1054 ms) 2024-02-19T21:04:47.030 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-02-19T21:04:47.030 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (16 ms) 2024-02-19T21:04:47.030 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-02-19T21:04:47.030 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (5 ms) 2024-02-19T21:04:47.030 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-02-19T21:04:47.030 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (4 ms) 2024-02-19T21:04:47.030 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-02-19T21:04:47.030 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (5 ms) 2024-02-19T21:04:47.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:47.188+0000 7f31fc83e700 1 -- 172.21.15.163:0/1009994265 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f31d8005200 con 0x7f31f810b130 2024-02-19T21:04:47.278 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:04:47.279 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:47.282 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:47.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:04:47.314 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:44.677003+0000 osd.6 (osd.6) 55 : cluster [DBG] 28.4 deep-scrub starts 2024-02-19T21:04:47.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:44.678214+0000 osd.6 (osd.6) 56 : cluster [DBG] 28.4 deep-scrub ok 2024-02-19T21:04:47.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:44.967751+0000 mgr.y (mgr.14550) 151 : cluster [DBG] pgmap v129: 484 pgs: 17 creating+peering, 15 creating+activating, 452 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 527 op/s 2024-02-19T21:04:47.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:45.197016+0000 osd.5 (osd.5) 62 : cluster [DBG] 28.3 deep-scrub starts 2024-02-19T21:04:47.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:45.203061+0000 osd.5 (osd.5) 63 : cluster [DBG] 28.3 deep-scrub ok 2024-02-19T21:04:47.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:45.352882+0000 osd.7 (osd.7) 67 : cluster [DBG] 28.6 deep-scrub starts 2024-02-19T21:04:47.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:45.355817+0000 osd.7 (osd.7) 68 : cluster [DBG] 28.6 deep-scrub ok 2024-02-19T21:04:47.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.384953+0000 mon.c (mon.2) 51 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:47.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.424493+0000 mon.a (mon.0) 852 : audit [INF] from='client.25219 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi163-83854-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:47.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.425560+0000 mon.b (mon.1) 141 : audit [INF] from='client.? v1:172.21.15.163:0/3338391635' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi163-83854-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-19T21:04:47.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.662201+0000 mon.a (mon.0) 853 : audit [INF] from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]': finished 2024-02-19T21:04:47.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.662373+0000 mon.a (mon.0) 854 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]': finished 2024-02-19T21:04:47.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.662536+0000 mon.a (mon.0) 855 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:47.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.662609+0000 mon.a (mon.0) 856 : audit [INF] from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi163-83896-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:47.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:45.662770+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-02-19T21:04:47.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.663140+0000 mon.b (mon.1) 142 : audit [INF] from='client.? v1:172.21.15.163:0/1278889621' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:47.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.663316+0000 mon.b (mon.1) 143 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-4", "overlaypool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:47.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.663696+0000 mon.b (mon.1) 144 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi163-83896-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:47.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.922118+0000 mon.a (mon.0) 858 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi163-83904-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:47.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.928837+0000 mon.a (mon.0) 859 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-02-19T21:04:47.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.929440+0000 mon.c (mon.2) 52 : audit [INF] from='client.? v1:172.21.15.163:0/1426051668' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-19T21:04:47.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:45.929578+0000 client.admin (client.35186) 0 : cluster [INF] fourxx 2024-02-19T21:04:47.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.935930+0000 mon.a (mon.0) 860 : audit [INF] from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi163-83960-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi163-83960-6"}]': finished 2024-02-19T21:04:47.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.936022+0000 mon.a (mon.0) 861 : audit [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi163-83872-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi163-83872-22"}]': finished 2024-02-19T21:04:47.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.936121+0000 mon.a (mon.0) 862 : audit [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi163-83968-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi163-83968-7"}]': finished 2024-02-19T21:04:47.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.936196+0000 mon.a (mon.0) 863 : audit [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi163-83864-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi163-83864-15"}]': finished 2024-02-19T21:04:47.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:45.936656+0000 mon.a (mon.0) 864 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-02-19T21:04:47.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.939347+0000 mon.a (mon.0) 865 : audit [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-02-19T21:04:47.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.941538+0000 mon.a (mon.0) 866 : audit [INF] from='client.25219 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi163-83854-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-02-19T21:04:47.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.942166+0000 mon.a (mon.0) 867 : audit [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:47.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.942452+0000 mon.a (mon.0) 868 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-4", "overlaypool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:47.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.942673+0000 mon.a (mon.0) 869 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi163-83896-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:47.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:45.952730+0000 mon.c (mon.2) 53 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:47.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:46.209417+0000 mon.a (mon.0) 870 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:47.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:46.386106+0000 mon.c (mon.2) 54 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:47.383 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:04:47.410 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:47.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:47.954+0000 7f31e77fe700 1 -- 172.21.15.163:0/1009994265 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v108) v1 ==== 91+0+0 (unknown 2530268910 0 0) 0x7f31f004a8f0 con 0x7f31f810b130 2024-02-19T21:04:47.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:47.957+0000 7f31fc83e700 10 librados: watch_flush enter 2024-02-19T21:04:47.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:47.962+0000 7f31fc83e700 10 librados: watch_flush exit 2024-02-19T21:04:47.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:47.962+0000 7f31fc83e700 20 client.35231.objecter shutdown clearing up homeless session... 2024-02-19T21:04:47.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:47.962+0000 7f31fc83e700 10 client.35231.objecter successfully canceled tick 2024-02-19T21:04:47.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:47.963+0000 7f31fc83e700 1 -- 172.21.15.163:0/1009994265 >> v1:172.21.15.163:6800/1552514681 conn(0x7f31e0063b30 legacy=0x7f31e0065ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:47.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:47.963+0000 7f31fc83e700 1 -- 172.21.15.163:0/1009994265 >> v1:172.21.15.163:6790/0 conn(0x7f31f810b130 legacy=0x7f31f8124a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:47.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:47.966+0000 7f31fc83e700 1 -- 172.21.15.163:0/1009994265 shutdown_connections 2024-02-19T21:04:47.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:47.966+0000 7f31fc83e700 1 -- 172.21.15.163:0/1009994265 >> 172.21.15.163:0/1009994265 conn(0x7f31f8063b20 msgr2=0x7f31f8067230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:47.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:47.969+0000 7f31fc83e700 1 -- 172.21.15.163:0/1009994265 shutdown_connections 2024-02-19T21:04:47.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:47.970+0000 7f31fc83e700 1 -- 172.21.15.163:0/1009994265 wait complete. 2024-02-19T21:04:47.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:47.974+0000 7f31fc83e700 1 librados: shutdown 2024-02-19T21:04:47.974 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:04:48.064 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:04:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:04:48.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:46.932369+0000 mon.a (mon.0) 871 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-02-19T21:04:48.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:46.968373+0000 mgr.y (mgr.14550) 152 : cluster [DBG] pgmap v132: 644 pgs: 160 unknown, 17 creating+peering, 15 creating+activating, 452 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 534 op/s 2024-02-19T21:04:48.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.188820+0000 mon.c (mon.2) 55 : audit [INF] from='client.? v1:172.21.15.163:0/1009994265' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:04:48.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.204656+0000 mon.a (mon.0) 872 : audit [INF] from='client.25219 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi163-83854-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-19T21:04:48.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.204789+0000 mon.a (mon.0) 873 : audit [INF] from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi163-83984-1"}]': finished 2024-02-19T21:04:48.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.204857+0000 mon.a (mon.0) 874 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-4", "overlaypool": "test-rados-api-smithi163-83928-4-cache"}]': finished 2024-02-19T21:04:48.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.205367+0000 mon.b (mon.1) 145 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-4-cache", "mode": "writeback"}]: dispatch 2024-02-19T21:04:48.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.386921+0000 mon.c (mon.2) 56 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:48.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.447204+0000 mon.a (mon.0) 875 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:48.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:47.447411+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-19T21:04:48.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.448241+0000 mon.b (mon.1) 146 : audit [INF] from='client.? v1:172.21.15.163:0/826688492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.448429+0000 mon.b (mon.1) 147 : audit [INF] from='client.? v1:172.21.15.163:0/4284884290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi163-83845-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.450217+0000 mon.a (mon.0) 877 : audit [INF] from='client.? v1:172.21.15.163:0/3920297063' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.454079+0000 mon.a (mon.0) 878 : audit [INF] from='client.35231 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:04:48.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.462347+0000 mon.c (mon.2) 57 : audit [DBG] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-02-19T21:04:48.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.462881+0000 mon.c (mon.2) 58 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.698930+0000 mon.a (mon.0) 879 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.700586+0000 mon.a (mon.0) 880 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-4-cache", "mode": "writeback"}]: dispatch 2024-02-19T21:04:48.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.700795+0000 mon.a (mon.0) 881 : audit [INF] from='client.25321 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.700959+0000 mon.a (mon.0) 882 : audit [INF] from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi163-83845-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.243 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:46.932369+0000 mon.a (mon.0) 871 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-02-19T21:04:48.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:46.968373+0000 mgr.y (mgr.14550) 152 : cluster [DBG] pgmap v132: 644 pgs: 160 unknown, 17 creating+peering, 15 creating+activating, 452 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 534 op/s 2024-02-19T21:04:48.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.188820+0000 mon.c (mon.2) 55 : audit [INF] from='client.? v1:172.21.15.163:0/1009994265' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:04:48.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.204656+0000 mon.a (mon.0) 872 : audit [INF] from='client.25219 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi163-83854-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-19T21:04:48.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.204789+0000 mon.a (mon.0) 873 : audit [INF] from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi163-83984-1"}]': finished 2024-02-19T21:04:48.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.204857+0000 mon.a (mon.0) 874 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-4", "overlaypool": "test-rados-api-smithi163-83928-4-cache"}]': finished 2024-02-19T21:04:48.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.205367+0000 mon.b (mon.1) 145 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-4-cache", "mode": "writeback"}]: dispatch 2024-02-19T21:04:48.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.386921+0000 mon.c (mon.2) 56 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:48.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.447204+0000 mon.a (mon.0) 875 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:48.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:47.447411+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-19T21:04:48.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.448241+0000 mon.b (mon.1) 146 : audit [INF] from='client.? v1:172.21.15.163:0/826688492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.245 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.448429+0000 mon.b (mon.1) 147 : audit [INF] from='client.? v1:172.21.15.163:0/4284884290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi163-83845-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.245 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.450217+0000 mon.a (mon.0) 877 : audit [INF] from='client.? v1:172.21.15.163:0/3920297063' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:46.932369+0000 mon.a (mon.0) 871 : audit [INF] from='client.35186 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-02-19T21:04:48.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:46.968373+0000 mgr.y (mgr.14550) 152 : cluster [DBG] pgmap v132: 644 pgs: 160 unknown, 17 creating+peering, 15 creating+activating, 452 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 534 op/s 2024-02-19T21:04:48.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.188820+0000 mon.c (mon.2) 55 : audit [INF] from='client.? v1:172.21.15.163:0/1009994265' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:04:48.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.204656+0000 mon.a (mon.0) 872 : audit [INF] from='client.25219 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi163-83854-3", "field": "max_bytes", "val": "4096"}]': finished 2024-02-19T21:04:48.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.204789+0000 mon.a (mon.0) 873 : audit [INF] from='client.24835 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi163-83984-1"}]': finished 2024-02-19T21:04:48.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.204857+0000 mon.a (mon.0) 874 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-4", "overlaypool": "test-rados-api-smithi163-83928-4-cache"}]': finished 2024-02-19T21:04:48.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.205367+0000 mon.b (mon.1) 145 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-4-cache", "mode": "writeback"}]: dispatch 2024-02-19T21:04:48.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.386921+0000 mon.c (mon.2) 56 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:48.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.447204+0000 mon.a (mon.0) 875 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:48.246 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:47.447411+0000 mon.a (mon.0) 876 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-02-19T21:04:48.246 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.448241+0000 mon.b (mon.1) 146 : audit [INF] from='client.? v1:172.21.15.163:0/826688492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.246 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.448429+0000 mon.b (mon.1) 147 : audit [INF] from='client.? v1:172.21.15.163:0/4284884290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi163-83845-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.246 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.450217+0000 mon.a (mon.0) 877 : audit [INF] from='client.? v1:172.21.15.163:0/3920297063' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.454079+0000 mon.a (mon.0) 878 : audit [INF] from='client.35231 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:04:48.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.462347+0000 mon.c (mon.2) 57 : audit [DBG] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-02-19T21:04:48.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.462881+0000 mon.c (mon.2) 58 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.698930+0000 mon.a (mon.0) 879 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.700586+0000 mon.a (mon.0) 880 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-4-cache", "mode": "writeback"}]: dispatch 2024-02-19T21:04:48.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.700795+0000 mon.a (mon.0) 881 : audit [INF] from='client.25321 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.700959+0000 mon.a (mon.0) 882 : audit [INF] from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi163-83845-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.454079+0000 mon.a (mon.0) 878 : audit [INF] from='client.35231 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:04:48.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.462347+0000 mon.c (mon.2) 57 : audit [DBG] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-02-19T21:04:48.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.462881+0000 mon.c (mon.2) 58 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.698930+0000 mon.a (mon.0) 879 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.700586+0000 mon.a (mon.0) 880 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-4-cache", "mode": "writeback"}]: dispatch 2024-02-19T21:04:48.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.700795+0000 mon.a (mon.0) 881 : audit [INF] from='client.25321 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.700959+0000 mon.a (mon.0) 882 : audit [INF] from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi163-83845-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:48.562 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:04:48.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:48.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:04:48.669 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:48.696 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:48.928 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN e_profile":"testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:48.928 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.714056+0000 mon.a [INF] from='client.24835 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi163-83984-1"}]: dispatch 2024-02-19T21:04:48.928 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.716509+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:48.928 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.718697+0000 mon.a [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:48.928 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.775017+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:48.928 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.779470+0000 mon.a [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:48.929 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.782513+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:48.929 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.784786+0000 mon.a [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:48.929 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.787118+0000 mon.a [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:48.929 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.788219+0000 mon.b [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi163-83896-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:48.929 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.789505+0000 mon.a [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:48.929 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.793212+0000 mon.a [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:48.929 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:43.795621+0000 mon.a [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi163-83896-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:48.929 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:44.685828+0000 mon.c [INF] from='client.? v1:172.21.15.163:0/1426051668' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-19T21:04:48.929 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:44.685936+0000 client.admin [INF] threexx 2024-02-19T21:04:48.929 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: got: 2024-02-19T21:04:44.686094+0000 mon.a [INF] from='client.35186 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-02-19T21:04:48.929 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (8085 ms) 2024-02-19T21:04:48.929 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (16087 ms total) 2024-02-19T21:04:48.930 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: 2024-02-19T21:04:48.930 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: [----------] Global test environment tear-down 2024-02-19T21:04:48.930 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (16087 ms total) 2024-02-19T21:04:48.930 INFO:tasks.workunit.client.0.smithi163.stdout: api_cmd: [ PASSED ] 4 tests. 2024-02-19T21:04:49.582 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: Running main() from gmock_main.cc 2024-02-19T21:04:49.582 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [==========] Running 23 tests from 2 test suites. 2024-02-19T21:04:49.582 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [----------] Global test environment set-up. 2024-02-19T21:04:49.582 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [----------] 13 tests from LibRadosIo 2024-02-19T21:04:49.583 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-02-19T21:04:49.583 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (277 ms) 2024-02-19T21:04:49.583 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-02-19T21:04:49.583 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-02-19T21:04:49.583 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-02-19T21:04:49.583 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: no timeout :/ 2024-02-19T21:04:49.583 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: no timeout :/ 2024-02-19T21:04:49.583 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: no timeout :/ 2024-02-19T21:04:49.583 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: no timeout :/ 2024-02-19T21:04:49.583 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: no timeout :/ 2024-02-19T21:04:49.583 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (32 ms) 2024-02-19T21:04:49.583 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-02-19T21:04:49.583 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (14 ms) 2024-02-19T21:04:49.584 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-02-19T21:04:49.584 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIo.Checksum (9 ms) 2024-02-19T21:04:49.584 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-02-19T21:04:49.584 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (4 ms) 2024-02-19T21:04:49.584 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-02-19T21:04:49.584 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (3 ms) 2024-02-19T21:04:49.584 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-02-19T21:04:49.584 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (5 ms) 2024-02-19T21:04:49.584 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-02-19T21:04:49.584 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIo.TruncTest (5 ms) 2024-02-19T21:04:49.584 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-02-19T21:04:49.584 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (4 ms) 2024-02-19T21:04:49.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-02-19T21:04:49.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (3 ms) 2024-02-19T21:04:49.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-02-19T21:04:49.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIo.RmXattr (32 ms) 2024-02-19T21:04:49.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-02-19T21:04:49.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIo.XattrIter (8 ms) 2024-02-19T21:04:49.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [----------] 13 tests from LibRadosIo (396 ms total) 2024-02-19T21:04:49.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: 2024-02-19T21:04:49.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-02-19T21:04:49.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-02-19T21:04:49.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1053 ms) 2024-02-19T21:04:49.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-02-19T21:04:49.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (9 ms) 2024-02-19T21:04:49.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-02-19T21:04:49.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (5 ms) 2024-02-19T21:04:49.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-02-19T21:04:49.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (4 ms) 2024-02-19T21:04:49.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-02-19T21:04:49.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (8 ms) 2024-02-19T21:04:49.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-02-19T21:04:49.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (4 ms) 2024-02-19T21:04:49.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-02-19T21:04:49.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (4 ms) 2024-02-19T21:04:49.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-02-19T21:04:49.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3 ms) 2024-02-19T21:04:49.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-02-19T21:04:49.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (10 ms) 2024-02-19T21:04:49.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-02-19T21:04:49.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2024-02-19T21:04:49.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1104 ms total) 2024-02-19T21:04:49.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: 2024-02-19T21:04:49.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [----------] Global test environment tear-down 2024-02-19T21:04:49.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [==========] 23 tests from 2 test suites ran. (16742 ms total) 2024-02-19T21:04:49.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_io: [ PASSED ] 23 tests. 2024-02-19T21:04:49.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-02-19T21:04:49.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-02-19T21:04:49.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-02-19T21:04:49.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-02-19T21:04:49.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: seed 83968 2024-02-19T21:04:49.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-02-19T21:04:49.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (290 ms) 2024-02-19T21:04:49.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-02-19T21:04:49.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (13 ms) 2024-02-19T21:04:49.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-02-19T21:04:49.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-02-19T21:04:49.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-02-19T21:04:49.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (3 ms) 2024-02-19T21:04:49.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-02-19T21:04:49.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (23 ms) 2024-02-19T21:04:49.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (330 ms total) 2024-02-19T21:04:49.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: 2024-02-19T21:04:49.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-02-19T21:04:49.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-02-19T21:04:49.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1051 ms) 2024-02-19T21:04:49.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-02-19T21:04:49.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-02-19T21:04:49.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-02-19T21:04:49.589 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (7 ms) 2024-02-19T21:04:49.589 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-02-19T21:04:49.589 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (7 ms) 2024-02-19T21:04:49.589 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1065 ms total) 2024-02-19T21:04:49.589 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: 2024-02-19T21:04:49.589 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-02-19T21:04:49.589 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (16741 ms total) 2024-02-19T21:04:49.589 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-02-19T21:04:49.589 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: Running main() from gmock_main.cc 2024-02-19T21:04:49.590 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [==========] Running 8 tests from 2 test suites. 2024-02-19T21:04:49.590 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [----------] Global test environment set-up. 2024-02-19T21:04:49.590 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [----------] 4 tests from LibRadosStat 2024-02-19T21:04:49.590 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-02-19T21:04:49.590 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ OK ] LibRadosStat.Stat (276 ms) 2024-02-19T21:04:49.590 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-02-19T21:04:49.590 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ OK ] LibRadosStat.StatNS (28 ms) 2024-02-19T21:04:49.590 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-02-19T21:04:49.590 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (24 ms) 2024-02-19T21:04:49.590 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-02-19T21:04:49.590 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (4 ms) 2024-02-19T21:04:49.590 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [----------] 4 tests from LibRadosStat (332 ms total) 2024-02-19T21:04:49.590 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: 2024-02-19T21:04:49.590 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-02-19T21:04:49.590 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-02-19T21:04:49.591 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1037 ms) 2024-02-19T21:04:49.591 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-02-19T21:04:49.591 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (22 ms) 2024-02-19T21:04:49.591 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-02-19T21:04:49.591 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1 ms) 2024-02-19T21:04:49.591 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-02-19T21:04:49.591 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (4 ms) 2024-02-19T21:04:49.591 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1064 ms total) 2024-02-19T21:04:49.591 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: 2024-02-19T21:04:49.591 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [----------] Global test environment tear-down 2024-02-19T21:04:49.591 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [==========] 8 tests from 2 test suites ran. (16732 ms total) 2024-02-19T21:04:49.591 INFO:tasks.workunit.client.0.smithi163.stdout: api_stat: [ PASSED ] 8 tests. 2024-02-19T21:04:49.591 INFO:tasks.workunit.client.0.smithi163.stdout: ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-02-19T21:04:49.591 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5 ms) 2024-02-19T21:04:49.591 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-02-19T21:04:49.592 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (11 ms) 2024-02-19T21:04:49.592 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-02-19T21:04:49.592 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (4 ms) 2024-02-19T21:04:49.592 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-02-19T21:04:49.592 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (5 ms) 2024-02-19T21:04:49.592 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-02-19T21:04:49.592 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (5 ms) 2024-02-19T21:04:49.592 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-02-19T21:04:49.592 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2024-02-19T21:04:49.592 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-02-19T21:04:49.592 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-02-19T21:04:49.592 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-02-19T21:04:49.592 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (8 ms) 2024-02-19T21:04:49.593 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-02-19T21:04:49.593 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (5 ms) 2024-02-19T21:04:49.593 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-02-19T21:04:49.593 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (8 ms) 2024-02-19T21:04:49.593 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-02-19T21:04:49.593 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (5 ms) 2024-02-19T21:04:49.593 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-02-19T21:04:49.593 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (6 ms) 2024-02-19T21:04:49.593 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1152 ms total) 2024-02-19T21:04:49.593 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: 2024-02-19T21:04:49.593 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [----------] Global test environment tear-down 2024-02-19T21:04:49.593 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (16764 ms total) 2024-02-19T21:04:49.593 INFO:tasks.workunit.client.0.smithi163.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-02-19T21:04:49.646 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:49.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:47.936924+0000 mon.a (mon.0) 883 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:04:49.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:47.936945+0000 mon.a (mon.0) 884 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:04:49.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.948005+0000 mon.a (mon.0) 885 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi163-83904-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi163-83904-10"}]': finished 2024-02-19T21:04:49.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.948096+0000 mon.a (mon.0) 886 : audit [INF] from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi163-83896-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi163-83896-10"}]': finished 2024-02-19T21:04:49.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.948258+0000 mon.a (mon.0) 887 : audit [INF] from='client.? v1:172.21.15.163:0/3920297063' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.948356+0000 mon.a (mon.0) 888 : audit [INF] from='client.35231 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:04:49.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.948483+0000 mon.a (mon.0) 889 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.948651+0000 mon.a (mon.0) 890 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-4-cache", "mode": "writeback"}]': finished 2024-02-19T21:04:49.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.948740+0000 mon.a (mon.0) 891 : audit [INF] from='client.25321 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.948808+0000 mon.a (mon.0) 892 : audit [INF] from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi163-83845-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:47.948885+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-19T21:04:49.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.950158+0000 mon.c (mon.2) 59 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app2"}]: dispatch 2024-02-19T21:04:49.654 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:47.936924+0000 mon.a (mon.0) 883 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:04:49.654 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:47.936945+0000 mon.a (mon.0) 884 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:04:49.654 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.948005+0000 mon.a (mon.0) 885 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi163-83904-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi163-83904-10"}]': finished 2024-02-19T21:04:49.654 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.948096+0000 mon.a (mon.0) 886 : audit [INF] from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi163-83896-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi163-83896-10"}]': finished 2024-02-19T21:04:49.654 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.948258+0000 mon.a (mon.0) 887 : audit [INF] from='client.? v1:172.21.15.163:0/3920297063' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.654 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.948356+0000 mon.a (mon.0) 888 : audit [INF] from='client.35231 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:04:49.654 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.948483+0000 mon.a (mon.0) 889 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.654 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.948651+0000 mon.a (mon.0) 890 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-4-cache", "mode": "writeback"}]': finished 2024-02-19T21:04:49.654 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.948740+0000 mon.a (mon.0) 891 : audit [INF] from='client.25321 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.654 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.948808+0000 mon.a (mon.0) 892 : audit [INF] from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi163-83845-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.654 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:47.948885+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-19T21:04:49.654 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.950158+0000 mon.c (mon.2) 59 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app2"}]: dispatch 2024-02-19T21:04:49.655 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.951057+0000 mon.c (mon.2) 60 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:49.655 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.951691+0000 mon.a (mon.0) 894 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:49.655 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.955394+0000 mon.b (mon.1) 148 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:49.655 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.956511+0000 mon.a (mon.0) 895 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:49.655 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.957233+0000 mon.b (mon.1) 149 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:49.655 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.958181+0000 mon.b (mon.1) 150 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:49.655 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.961291+0000 mon.a (mon.0) 896 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:49.655 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.961550+0000 mon.a (mon.0) 897 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:49.655 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.969441+0000 mon.b (mon.1) 151 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:49.656 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.970502+0000 mon.c (mon.2) 61 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:49.656 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.970670+0000 mon.a (mon.0) 898 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:49.656 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:47.974872+0000 mon.a (mon.0) 899 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:49.656 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.025237+0000 mon.b (mon.1) 152 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-4"}]: dispatch 2024-02-19T21:04:49.656 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.951057+0000 mon.c (mon.2) 60 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:49.656 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.951691+0000 mon.a (mon.0) 894 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:49.656 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.955394+0000 mon.b (mon.1) 148 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:49.656 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.956511+0000 mon.a (mon.0) 895 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:49.656 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.957233+0000 mon.b (mon.1) 149 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:49.657 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.958181+0000 mon.b (mon.1) 150 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:49.657 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.961291+0000 mon.a (mon.0) 896 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:49.657 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.961550+0000 mon.a (mon.0) 897 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:49.657 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.969441+0000 mon.b (mon.1) 151 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:49.657 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.970502+0000 mon.c (mon.2) 61 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:49.657 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.970670+0000 mon.a (mon.0) 898 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:49.657 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:47.974872+0000 mon.a (mon.0) 899 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:49.657 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.025237+0000 mon.b (mon.1) 152 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-4"}]: dispatch 2024-02-19T21:04:49.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.025550+0000 mon.a (mon.0) 900 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-4"}]: dispatch 2024-02-19T21:04:49.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:48.294580+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:04:49.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:48.298481+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:04:49.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.387678+0000 mon.c (mon.2) 62 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:49.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:48.503212+0000 mon.a (mon.0) 901 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:04:49.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.509923+0000 mon.a (mon.0) 902 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.509998+0000 mon.a (mon.0) 903 : audit [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]': finished 2024-02-19T21:04:49.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.510051+0000 mon.a (mon.0) 904 : audit [INF] from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]': finished 2024-02-19T21:04:49.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.510086+0000 mon.a (mon.0) 905 : audit [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]': finished 2024-02-19T21:04:49.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.510187+0000 mon.a (mon.0) 906 : audit [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]': finished 2024-02-19T21:04:49.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.510277+0000 mon.a (mon.0) 907 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.510374+0000 mon.a (mon.0) 908 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-4"}]': finished 2024-02-19T21:04:49.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:48.510403+0000 mon.a (mon.0) 909 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-02-19T21:04:49.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.518786+0000 mon.b (mon.1) 153 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:49.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.025550+0000 mon.a (mon.0) 900 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-4"}]: dispatch 2024-02-19T21:04:49.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:48.294580+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:04:49.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:48.298481+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:04:49.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.387678+0000 mon.c (mon.2) 62 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:49.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:48.503212+0000 mon.a (mon.0) 901 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:04:49.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.509923+0000 mon.a (mon.0) 902 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.509998+0000 mon.a (mon.0) 903 : audit [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]': finished 2024-02-19T21:04:49.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.510051+0000 mon.a (mon.0) 904 : audit [INF] from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]': finished 2024-02-19T21:04:49.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.510086+0000 mon.a (mon.0) 905 : audit [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]': finished 2024-02-19T21:04:49.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.510187+0000 mon.a (mon.0) 906 : audit [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]': finished 2024-02-19T21:04:49.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.510277+0000 mon.a (mon.0) 907 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.510374+0000 mon.a (mon.0) 908 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-4"}]': finished 2024-02-19T21:04:49.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:48.510403+0000 mon.a (mon.0) 909 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-02-19T21:04:49.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.518786+0000 mon.b (mon.1) 153 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:49.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.519153+0000 mon.b (mon.1) 154 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:49.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.519179+0000 mon.c (mon.2) 63 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-02-19T21:04:49.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.520982+0000 mon.a (mon.0) 910 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:49.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.521131+0000 mon.c (mon.2) 64 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-19T21:04:49.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.521424+0000 mon.a (mon.0) 911 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:49.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.521778+0000 mon.a (mon.0) 912 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-19T21:04:49.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.523803+0000 mon.b (mon.1) 155 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:49.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.523970+0000 mon.b (mon.1) 156 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:49.661 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.524195+0000 mon.b (mon.1) 157 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:49.661 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.534933+0000 mon.a (mon.0) 913 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:49.661 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.535223+0000 mon.a (mon.0) 914 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:49.661 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.535413+0000 mon.a (mon.0) 915 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:49.661 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.592798+0000 mon.c (mon.2) 65 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pg_num": 4}]: dispatch 2024-02-19T21:04:49.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.519153+0000 mon.b (mon.1) 154 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:49.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.519179+0000 mon.c (mon.2) 63 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-02-19T21:04:49.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.520982+0000 mon.a (mon.0) 910 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:49.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.521131+0000 mon.c (mon.2) 64 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-19T21:04:49.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.521424+0000 mon.a (mon.0) 911 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:49.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.521778+0000 mon.a (mon.0) 912 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-19T21:04:49.662 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.523803+0000 mon.b (mon.1) 155 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:49.662 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.523970+0000 mon.b (mon.1) 156 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:49.662 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.524195+0000 mon.b (mon.1) 157 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:49.662 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.534933+0000 mon.a (mon.0) 913 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:49.662 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.535223+0000 mon.a (mon.0) 914 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:49.662 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.535413+0000 mon.a (mon.0) 915 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:49.662 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.592798+0000 mon.c (mon.2) 65 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pg_num": 4}]: dispatch 2024-02-19T21:04:49.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:49.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:49.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.675+0000 7fdc88090700 1 -- 172.21.15.163:0/2341691725 >> v1:172.21.15.163:6789/0 conn(0x7fdc801042c0 legacy=0x7fdc80106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:49.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.675+0000 7fdc88090700 1 -- 172.21.15.163:0/2341691725 shutdown_connections 2024-02-19T21:04:49.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.675+0000 7fdc88090700 1 -- 172.21.15.163:0/2341691725 >> 172.21.15.163:0/2341691725 conn(0x7fdc800f35a0 msgr2=0x7fdc800f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:49.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.675+0000 7fdc88090700 1 -- 172.21.15.163:0/2341691725 shutdown_connections 2024-02-19T21:04:49.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.675+0000 7fdc88090700 1 -- 172.21.15.163:0/2341691725 wait complete. 2024-02-19T21:04:49.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.675+0000 7fdc88090700 1 librados: starting msgr at 2024-02-19T21:04:49.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.675+0000 7fdc88090700 1 librados: starting objecter 2024-02-19T21:04:49.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.675+0000 7fdc88090700 1 Processor -- start 2024-02-19T21:04:49.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.675+0000 7fdc88090700 1 -- start start 2024-02-19T21:04:49.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.675+0000 7fdc88090700 1 librados: setting wanted keys 2024-02-19T21:04:49.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.675+0000 7fdc88090700 1 librados: calling monclient init 2024-02-19T21:04:49.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.676+0000 7fdc88090700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdc8006cb20 con 0x7fdc801042c0 2024-02-19T21:04:49.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.676+0000 7fdc88090700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdc8006ccb0 con 0x7fdc800fd980 2024-02-19T21:04:49.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.676+0000 7fdc88090700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdc8006cea0 con 0x7fdc80100780 2024-02-19T21:04:49.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.676+0000 7fdc8562b700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fdc80100780 0x7fdc80114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:54270/0 (socket says 172.21.15.163:54270) 2024-02-19T21:04:49.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.676+0000 7fdc8562b700 1 -- 172.21.15.163:0/1871563584 learned_addr learned my addr 172.21.15.163:0/1871563584 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:04:49.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.676+0000 7fdc8662d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fdc801042c0 0x7fdc80118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:42300/0 (socket says 172.21.15.163:42300) 2024-02-19T21:04:49.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 10 client.?.objecter ms_handle_connect 0x7fdc80100780 2024-02-19T21:04:49.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:49.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 10 client.?.objecter ms_handle_connect 0x7fdc801042c0 2024-02-19T21:04:49.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:49.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 10 client.?.objecter ms_handle_connect 0x7fdc800fd980 2024-02-19T21:04:49.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:49.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239919544 0 0) 0x7fdc8006ccb0 con 0x7fdc800fd980 2024-02-19T21:04:49.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc6c003030 con 0x7fdc800fd980 2024-02-19T21:04:49.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849726987 0 0) 0x7fdc8006cea0 con 0x7fdc80100780 2024-02-19T21:04:49.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc8006ccb0 con 0x7fdc80100780 2024-02-19T21:04:49.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927855187 0 0) 0x7fdc8006cb20 con 0x7fdc801042c0 2024-02-19T21:04:49.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc8006cea0 con 0x7fdc801042c0 2024-02-19T21:04:49.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3448782174 0 0) 0x7fdc6c003030 con 0x7fdc800fd980 2024-02-19T21:04:49.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc8006cb20 con 0x7fdc800fd980 2024-02-19T21:04:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdc7c0018c0 con 0x7fdc800fd980 2024-02-19T21:04:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3764498683 0 0) 0x7fdc8006cea0 con 0x7fdc801042c0 2024-02-19T21:04:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc6c003030 con 0x7fdc801042c0 2024-02-19T21:04:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1985453922 0 0) 0x7fdc8006ccb0 con 0x7fdc80100780 2024-02-19T21:04:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc8006cea0 con 0x7fdc80100780 2024-02-19T21:04:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdc780031a0 con 0x7fdc801042c0 2024-02-19T21:04:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdc70001860 con 0x7fdc80100780 2024-02-19T21:04:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2978992643 0 0) 0x7fdc8006cb20 con 0x7fdc800fd980 2024-02-19T21:04:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 >> v1:172.21.15.163:6790/0 conn(0x7fdc80100780 legacy=0x7fdc80114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 >> v1:172.21.15.163:6789/0 conn(0x7fdc801042c0 legacy=0x7fdc80118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8011cb90 con 0x7fdc800fd980 2024-02-19T21:04:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc88090700 1 -- 172.21.15.163:0/1871563584 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc80119bc0 con 0x7fdc800fd980 2024-02-19T21:04:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc88090700 10 client.25357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:49.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc88090700 1 -- 172.21.15.163:0/1871563584 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8011a030 con 0x7fdc800fd980 2024-02-19T21:04:49.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc88090700 1 librados: init done 2024-02-19T21:04:49.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fdc7c001f80 con 0x7fdc800fd980 2024-02-19T21:04:49.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.677+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdc7c0032a0 con 0x7fdc800fd980 2024-02-19T21:04:49.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.678+0000 7fdc88090700 1 -- 172.21.15.163:0/1871563584 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc64004f20 con 0x7fdc800fd980 2024-02-19T21:04:49.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.679+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fdc7c001ae0 con 0x7fdc800fd980 2024-02-19T21:04:49.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.679+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(110..110 src has 43..110) v4 ==== 14614+0+0 (unknown 4174331974 0 0) 0x7fdc7c077e10 con 0x7fdc800fd980 2024-02-19T21:04:49.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.679+0000 7fdc76ffd700 10 client.25357.objecter ms_dispatch 0x7fdc80001550 osd_map(110..110 src has 43..110) v4 2024-02-19T21:04:49.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.679+0000 7fdc76ffd700 3 client.25357.objecter handle_osd_map got epochs [110,110] > 0 2024-02-19T21:04:49.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.679+0000 7fdc76ffd700 3 client.25357.objecter handle_osd_map decoding full epoch 110 2024-02-19T21:04:49.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.680+0000 7fdc76ffd700 20 client.25357.objecter dump_active .. 0 homeless 2024-02-19T21:04:49.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.680+0000 7fdc76ffd700 10 client.25357.objecter ms_handle_connect 0x7fdc6c06c950 2024-02-19T21:04:49.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.684+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fdc7c049870 con 0x7fdc800fd980 2024-02-19T21:04:49.764 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:04:49.791 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:47.936924+0000 mon.a (mon.0) 883 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:04:49.791 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:47.936945+0000 mon.a (mon.0) 884 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:04:49.791 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.948005+0000 mon.a (mon.0) 885 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi163-83904-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi163-83904-10"}]': finished 2024-02-19T21:04:49.791 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.948096+0000 mon.a (mon.0) 886 : audit [INF] from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi163-83896-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi163-83896-10"}]': finished 2024-02-19T21:04:49.791 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.948258+0000 mon.a (mon.0) 887 : audit [INF] from='client.? v1:172.21.15.163:0/3920297063' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.791 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.948356+0000 mon.a (mon.0) 888 : audit [INF] from='client.35231 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:04:49.791 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.948483+0000 mon.a (mon.0) 889 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.791 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.948651+0000 mon.a (mon.0) 890 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-4-cache", "mode": "writeback"}]': finished 2024-02-19T21:04:49.791 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.948740+0000 mon.a (mon.0) 891 : audit [INF] from='client.25321 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.791 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.948808+0000 mon.a (mon.0) 892 : audit [INF] from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi163-83845-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.791 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:47.948885+0000 mon.a (mon.0) 893 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-02-19T21:04:49.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.950158+0000 mon.c (mon.2) 59 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app2"}]: dispatch 2024-02-19T21:04:49.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.951057+0000 mon.c (mon.2) 60 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:49.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.951691+0000 mon.a (mon.0) 894 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:49.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.955394+0000 mon.b (mon.1) 148 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:49.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.956511+0000 mon.a (mon.0) 895 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:49.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.957233+0000 mon.b (mon.1) 149 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:49.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.958181+0000 mon.b (mon.1) 150 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:49.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.961291+0000 mon.a (mon.0) 896 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:49.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.961550+0000 mon.a (mon.0) 897 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:49.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.969441+0000 mon.b (mon.1) 151 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:49.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.970502+0000 mon.c (mon.2) 61 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:49.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.970670+0000 mon.a (mon.0) 898 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:49.792 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:47.974872+0000 mon.a (mon.0) 899 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:49.793 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.025237+0000 mon.b (mon.1) 152 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-4"}]: dispatch 2024-02-19T21:04:49.793 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.025550+0000 mon.a (mon.0) 900 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-4"}]: dispatch 2024-02-19T21:04:49.793 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:48.294580+0000 osd.7 (osd.7) 69 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:04:49.793 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:48.298481+0000 osd.7 (osd.7) 70 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:04:49.793 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.387678+0000 mon.c (mon.2) 62 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:49.793 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:48.503212+0000 mon.a (mon.0) 901 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:04:49.793 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.509923+0000 mon.a (mon.0) 902 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi163-83912-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.793 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.509998+0000 mon.a (mon.0) 903 : audit [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi163-83864-15"}]': finished 2024-02-19T21:04:49.793 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.510051+0000 mon.a (mon.0) 904 : audit [INF] from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi163-83960-6"}]': finished 2024-02-19T21:04:49.793 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.510086+0000 mon.a (mon.0) 905 : audit [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi163-83968-7"}]': finished 2024-02-19T21:04:49.793 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.510187+0000 mon.a (mon.0) 906 : audit [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi163-83872-22"}]': finished 2024-02-19T21:04:49.793 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.510277+0000 mon.a (mon.0) 907 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:49.793 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.510374+0000 mon.a (mon.0) 908 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-4"}]': finished 2024-02-19T21:04:49.793 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:48.510403+0000 mon.a (mon.0) 909 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-02-19T21:04:49.794 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.518786+0000 mon.b (mon.1) 153 : audit [INF] from='client.? v1:172.21.15.163:0/3624732563' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:49.794 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.519153+0000 mon.b (mon.1) 154 : audit [INF] from='client.? v1:172.21.15.163:0/3498182538' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:49.794 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.519179+0000 mon.c (mon.2) 63 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-02-19T21:04:49.794 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.520982+0000 mon.a (mon.0) 910 : audit [INF] from='client.25105 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]: dispatch 2024-02-19T21:04:49.794 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.521131+0000 mon.c (mon.2) 64 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-19T21:04:49.794 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.521424+0000 mon.a (mon.0) 911 : audit [INF] from='client.25258 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]: dispatch 2024-02-19T21:04:49.794 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.521778+0000 mon.a (mon.0) 912 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-19T21:04:49.794 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.523803+0000 mon.b (mon.1) 155 : audit [INF] from='client.? v1:172.21.15.163:0/3720420201' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:49.794 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.523970+0000 mon.b (mon.1) 156 : audit [INF] from='client.? v1:172.21.15.163:0/2108766762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:49.794 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.524195+0000 mon.b (mon.1) 157 : audit [INF] from='client.? v1:172.21.15.163:0/516814289' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:49.794 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.534933+0000 mon.a (mon.0) 913 : audit [INF] from='client.25243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi163-83968-7"}]: dispatch 2024-02-19T21:04:49.794 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.535223+0000 mon.a (mon.0) 914 : audit [INF] from='client.25267 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]: dispatch 2024-02-19T21:04:49.794 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.535413+0000 mon.a (mon.0) 915 : audit [INF] from='client.25228 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi163-83960-6"}]: dispatch 2024-02-19T21:04:49.794 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.592798+0000 mon.c (mon.2) 65 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pg_num": 4}]: dispatch 2024-02-19T21:04:49.795 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:49.811 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:04:49.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:49.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:04:49.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.872+0000 7fdc88090700 1 -- 172.21.15.163:0/1871563584 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdc64000b90 con 0x7fdc6c06c950 2024-02-19T21:04:49.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.915+0000 7fdc76ffd700 1 -- 172.21.15.163:0/1871563584 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+1574223 (unknown 2965378022 0 3966981971) 0x7fdc64000b90 con 0x7fdc6c06c950 2024-02-19T21:04:49.917 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:49.929 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:04:49.960 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:49.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.925+0000 7fdc88090700 10 librados: watch_flush enter 2024-02-19T21:04:49.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.925+0000 7fdc88090700 10 librados: watch_flush exit 2024-02-19T21:04:49.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.925+0000 7fdc88090700 20 client.25357.objecter shutdown clearing up homeless session... 2024-02-19T21:04:49.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.925+0000 7fdc88090700 10 client.25357.objecter successfully canceled tick 2024-02-19T21:04:49.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.925+0000 7fdc88090700 1 -- 172.21.15.163:0/1871563584 >> v1:172.21.15.163:6800/1552514681 conn(0x7fdc6c06c950 legacy=0x7fdc6c06ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:49.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.925+0000 7fdc88090700 1 -- 172.21.15.163:0/1871563584 >> v1:172.21.15.193:6789/0 conn(0x7fdc800fd980 legacy=0x7fdc800fc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:49.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.925+0000 7fdc88090700 1 -- 172.21.15.163:0/1871563584 shutdown_connections 2024-02-19T21:04:49.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.926+0000 7fdc88090700 1 -- 172.21.15.163:0/1871563584 >> 172.21.15.163:0/1871563584 conn(0x7fdc800f35a0 msgr2=0x7fdc80107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:49.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.926+0000 7fdc88090700 1 -- 172.21.15.163:0/1871563584 shutdown_connections 2024-02-19T21:04:49.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.926+0000 7fdc88090700 1 -- 172.21.15.163:0/1871563584 wait complete. 2024-02-19T21:04:49.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:49.926+0000 7fdc88090700 1 librados: shutdown 2024-02-19T21:04:49.961 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:04:49.971 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:49.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:49.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:50.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:50.105 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:50.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:04:50.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:48.826312+0000 mon.a (mon.0) 916 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pg_num": 4}]: dispatch 2024-02-19T21:04:50.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:48.968870+0000 mgr.y (mgr.14550) 153 : cluster [DBG] pgmap v135: 564 pgs: 80 unknown, 17 creating+peering, 15 creating+activating, 452 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:04:50.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:49.003357+0000 osd.3 (osd.3) 75 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:04:50.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:49.004143+0000 osd.3 (osd.3) 76 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:04:50.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:49.284263+0000 osd.4 (osd.4) 69 : cluster [DBG] 4.c scrub starts 2024-02-19T21:04:50.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:49.285592+0000 osd.4 (osd.4) 70 : cluster [DBG] 4.c scrub ok 2024-02-19T21:04:50.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:49.379653+0000 osd.0 (osd.0) 65 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:04:50.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:49.382708+0000 osd.0 (osd.0) 66 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:04:50.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.388845+0000 mon.c (mon.2) 66 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:50.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:49.549678+0000 mon.a (mon.0) 917 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:04:50.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.555345+0000 mon.a (mon.0) 918 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]': finished 2024-02-19T21:04:50.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.555426+0000 mon.a (mon.0) 919 : audit [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]': finished 2024-02-19T21:04:50.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.555482+0000 mon.a (mon.0) 920 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-19T21:04:50.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.555543+0000 mon.a (mon.0) 921 : audit [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi163-83968-7"}]': finished 2024-02-19T21:04:50.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.555573+0000 mon.a (mon.0) 922 : audit [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]': finished 2024-02-19T21:04:50.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.555601+0000 mon.a (mon.0) 923 : audit [INF] from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi163-83960-6"}]': finished 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:48.826312+0000 mon.a (mon.0) 916 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pg_num": 4}]: dispatch 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:48.968870+0000 mgr.y (mgr.14550) 153 : cluster [DBG] pgmap v135: 564 pgs: 80 unknown, 17 creating+peering, 15 creating+activating, 452 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:49.003357+0000 osd.3 (osd.3) 75 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:49.004143+0000 osd.3 (osd.3) 76 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:49.284263+0000 osd.4 (osd.4) 69 : cluster [DBG] 4.c scrub starts 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:49.285592+0000 osd.4 (osd.4) 70 : cluster [DBG] 4.c scrub ok 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:49.379653+0000 osd.0 (osd.0) 65 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:49.382708+0000 osd.0 (osd.0) 66 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.388845+0000 mon.c (mon.2) 66 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:49.549678+0000 mon.a (mon.0) 917 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.555345+0000 mon.a (mon.0) 918 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]': finished 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.555426+0000 mon.a (mon.0) 919 : audit [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]': finished 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.555482+0000 mon.a (mon.0) 920 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.555543+0000 mon.a (mon.0) 921 : audit [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi163-83968-7"}]': finished 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.555573+0000 mon.a (mon.0) 922 : audit [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]': finished 2024-02-19T21:04:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.555601+0000 mon.a (mon.0) 923 : audit [INF] from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi163-83960-6"}]': finished 2024-02-19T21:04:50.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.555664+0000 mon.a (mon.0) 924 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pg_num": 4}]': finished 2024-02-19T21:04:50.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:49.555742+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-02-19T21:04:50.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.558191+0000 mon.a (mon.0) 926 : audit [INF] from='client.? v1:172.21.15.163:0/2459019407' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi163-83845-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:50.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.560784+0000 mon.c (mon.2) 67 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:04:50.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.561097+0000 mon.a (mon.0) 927 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:04:50.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.565813+0000 mon.c (mon.2) 68 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-19T21:04:50.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.571733+0000 mon.a (mon.0) 928 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-19T21:04:50.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.581114+0000 mon.b (mon.1) 158 : audit [INF] from='client.? v1:172.21.15.163:0/3896506064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:50.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.582006+0000 mon.a (mon.0) 929 : audit [INF] from='client.25348 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:50.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.583318+0000 mon.b (mon.1) 159 : audit [INF] from='client.? v1:172.21.15.163:0/581384461' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:50.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.585648+0000 mon.a (mon.0) 930 : audit [INF] from='client.25351 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:50.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:49.647762+0000 osd.6 (osd.6) 57 : cluster [DBG] 28.8 deep-scrub starts 2024-02-19T21:04:50.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:49.651567+0000 osd.6 (osd.6) 58 : cluster [DBG] 28.8 deep-scrub ok 2024-02-19T21:04:50.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:49.850289+0000 osd.1 (osd.1) 75 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:04:50.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:49.851386+0000 osd.1 (osd.1) 76 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:04:50.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:50.389588+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:50.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.555664+0000 mon.a (mon.0) 924 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pg_num": 4}]': finished 2024-02-19T21:04:50.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:49.555742+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-02-19T21:04:50.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.558191+0000 mon.a (mon.0) 926 : audit [INF] from='client.? v1:172.21.15.163:0/2459019407' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi163-83845-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:50.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.560784+0000 mon.c (mon.2) 67 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:04:50.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.561097+0000 mon.a (mon.0) 927 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:04:50.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.565813+0000 mon.c (mon.2) 68 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-19T21:04:50.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.571733+0000 mon.a (mon.0) 928 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-19T21:04:50.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.581114+0000 mon.b (mon.1) 158 : audit [INF] from='client.? v1:172.21.15.163:0/3896506064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:50.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.582006+0000 mon.a (mon.0) 929 : audit [INF] from='client.25348 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:50.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.583318+0000 mon.b (mon.1) 159 : audit [INF] from='client.? v1:172.21.15.163:0/581384461' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:50.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.585648+0000 mon.a (mon.0) 930 : audit [INF] from='client.25351 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:50.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:49.647762+0000 osd.6 (osd.6) 57 : cluster [DBG] 28.8 deep-scrub starts 2024-02-19T21:04:50.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:49.651567+0000 osd.6 (osd.6) 58 : cluster [DBG] 28.8 deep-scrub ok 2024-02-19T21:04:50.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:49.850289+0000 osd.1 (osd.1) 75 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:04:50.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:49.851386+0000 osd.1 (osd.1) 76 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:04:50.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:50.389588+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:50.758 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:50.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:50.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:50.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:48.826312+0000 mon.a (mon.0) 916 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pg_num": 4}]: dispatch 2024-02-19T21:04:50.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:48.968870+0000 mgr.y (mgr.14550) 153 : cluster [DBG] pgmap v135: 564 pgs: 80 unknown, 17 creating+peering, 15 creating+activating, 452 active+clean; 144 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:04:50.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:49.003357+0000 osd.3 (osd.3) 75 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:04:50.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:49.004143+0000 osd.3 (osd.3) 76 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:04:50.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:49.284263+0000 osd.4 (osd.4) 69 : cluster [DBG] 4.c scrub starts 2024-02-19T21:04:50.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:49.285592+0000 osd.4 (osd.4) 70 : cluster [DBG] 4.c scrub ok 2024-02-19T21:04:50.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:49.379653+0000 osd.0 (osd.0) 65 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:04:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:49.382708+0000 osd.0 (osd.0) 66 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:04:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.388845+0000 mon.c (mon.2) 66 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:49.549678+0000 mon.a (mon.0) 917 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:04:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.555345+0000 mon.a (mon.0) 918 : audit [INF] from='client.25105 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-4", "tierpool": "test-rados-api-smithi163-83928-4-cache"}]': finished 2024-02-19T21:04:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.555426+0000 mon.a (mon.0) 919 : audit [INF] from='client.25258 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi163-83864-15"}]': finished 2024-02-19T21:04:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.555482+0000 mon.a (mon.0) 920 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-19T21:04:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.555543+0000 mon.a (mon.0) 921 : audit [INF] from='client.25243 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi163-83968-7"}]': finished 2024-02-19T21:04:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.555573+0000 mon.a (mon.0) 922 : audit [INF] from='client.25267 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi163-83872-22"}]': finished 2024-02-19T21:04:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.555601+0000 mon.a (mon.0) 923 : audit [INF] from='client.25228 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi163-83960-6"}]': finished 2024-02-19T21:04:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.555664+0000 mon.a (mon.0) 924 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pg_num": 4}]': finished 2024-02-19T21:04:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:49.555742+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-02-19T21:04:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.558191+0000 mon.a (mon.0) 926 : audit [INF] from='client.? v1:172.21.15.163:0/2459019407' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi163-83845-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.560784+0000 mon.c (mon.2) 67 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:04:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.561097+0000 mon.a (mon.0) 927 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:04:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.565813+0000 mon.c (mon.2) 68 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-19T21:04:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.571733+0000 mon.a (mon.0) 928 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-19T21:04:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.581114+0000 mon.b (mon.1) 158 : audit [INF] from='client.? v1:172.21.15.163:0/3896506064' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.582006+0000 mon.a (mon.0) 929 : audit [INF] from='client.25348 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.583318+0000 mon.b (mon.1) 159 : audit [INF] from='client.? v1:172.21.15.163:0/581384461' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.585648+0000 mon.a (mon.0) 930 : audit [INF] from='client.25351 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:49.647762+0000 osd.6 (osd.6) 57 : cluster [DBG] 28.8 deep-scrub starts 2024-02-19T21:04:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:49.651567+0000 osd.6 (osd.6) 58 : cluster [DBG] 28.8 deep-scrub ok 2024-02-19T21:04:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:49.850289+0000 osd.1 (osd.1) 75 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:04:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:49.851386+0000 osd.1 (osd.1) 76 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:04:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:50.389588+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:50.864 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:50.891 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:51.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:04:51.439 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:51.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:51.543 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:04:51.610 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:51.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:49.817035+0000 osd.2 (osd.2) 59 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:04:51.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:49.819918+0000 osd.2 (osd.2) 60 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:04:51.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:49.872484+0000 mgr.y (mgr.14550) 154 : audit [DBG] from='client.25357 v1:172.21.15.163:0/1871563584' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:04:51.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:50.026038+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:04:51.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:50.027684+0000 osd.3 (osd.3) 78 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:04:51.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:50.071328+0000 mgr.y (mgr.14550) 155 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:51.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:50.264243+0000 osd.4 (osd.4) 71 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:04:51.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:50.266143+0000 osd.4 (osd.4) 72 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:04:51.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:50.352137+0000 osd.0 (osd.0) 67 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:04:51.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:50.353860+0000 osd.0 (osd.0) 68 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:04:51.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:50.557210+0000 mon.a (mon.0) 931 : audit [INF] from='client.? v1:172.21.15.163:0/2459019407' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi163-83845-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:51.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:50.557319+0000 mon.a (mon.0) 932 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:04:51.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:50.557405+0000 mon.a (mon.0) 933 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-19T21:04:51.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:50.557558+0000 mon.a (mon.0) 934 : audit [INF] from='client.25348 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:51.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:50.557635+0000 mon.a (mon.0) 935 : audit [INF] from='client.25351 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:51.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:50.557709+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-02-19T21:04:51.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:50.559231+0000 mon.c (mon.2) 70 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1"}]: dispatch 2024-02-19T21:04:51.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:50.560261+0000 mon.a (mon.0) 937 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1"}]: dispatch 2024-02-19T21:04:51.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:50.565165+0000 mon.c (mon.2) 71 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]: dispatch 2024-02-19T21:04:51.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:50.565854+0000 mon.a (mon.0) 938 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]: dispatch 2024-02-19T21:04:51.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:51.390515+0000 mon.c (mon.2) 72 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:51.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:49.817035+0000 osd.2 (osd.2) 59 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:04:51.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:49.819918+0000 osd.2 (osd.2) 60 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:04:51.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:49.872484+0000 mgr.y (mgr.14550) 154 : audit [DBG] from='client.25357 v1:172.21.15.163:0/1871563584' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:04:51.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:50.026038+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:04:51.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:50.027684+0000 osd.3 (osd.3) 78 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:04:51.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:50.071328+0000 mgr.y (mgr.14550) 155 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:50.264243+0000 osd.4 (osd.4) 71 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:04:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:50.266143+0000 osd.4 (osd.4) 72 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:04:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:50.352137+0000 osd.0 (osd.0) 67 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:04:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:50.353860+0000 osd.0 (osd.0) 68 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:04:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:50.557210+0000 mon.a (mon.0) 931 : audit [INF] from='client.? v1:172.21.15.163:0/2459019407' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi163-83845-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:50.557319+0000 mon.a (mon.0) 932 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:04:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:50.557405+0000 mon.a (mon.0) 933 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-19T21:04:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:50.557558+0000 mon.a (mon.0) 934 : audit [INF] from='client.25348 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:50.557635+0000 mon.a (mon.0) 935 : audit [INF] from='client.25351 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:50.557709+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-02-19T21:04:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:50.559231+0000 mon.c (mon.2) 70 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1"}]: dispatch 2024-02-19T21:04:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:50.560261+0000 mon.a (mon.0) 937 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1"}]: dispatch 2024-02-19T21:04:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:50.565165+0000 mon.c (mon.2) 71 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]: dispatch 2024-02-19T21:04:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:50.565854+0000 mon.a (mon.0) 938 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]: dispatch 2024-02-19T21:04:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:51.390515+0000 mon.c (mon.2) 72 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:51.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:49.817035+0000 osd.2 (osd.2) 59 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:04:51.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:49.819918+0000 osd.2 (osd.2) 60 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:04:51.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:49.872484+0000 mgr.y (mgr.14550) 154 : audit [DBG] from='client.25357 v1:172.21.15.163:0/1871563584' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:04:51.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:50.026038+0000 osd.3 (osd.3) 77 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:04:51.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:50.027684+0000 osd.3 (osd.3) 78 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:04:51.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:50.071328+0000 mgr.y (mgr.14550) 155 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:04:51.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:50.264243+0000 osd.4 (osd.4) 71 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:04:51.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:50.266143+0000 osd.4 (osd.4) 72 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:04:51.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:50.352137+0000 osd.0 (osd.0) 67 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:04:51.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:50.353860+0000 osd.0 (osd.0) 68 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:04:51.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:50.557210+0000 mon.a (mon.0) 931 : audit [INF] from='client.? v1:172.21.15.163:0/2459019407' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi163-83845-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:51.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:50.557319+0000 mon.a (mon.0) 932 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:04:51.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:50.557405+0000 mon.a (mon.0) 933 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-19T21:04:51.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:50.557558+0000 mon.a (mon.0) 934 : audit [INF] from='client.25348 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:51.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:50.557635+0000 mon.a (mon.0) 935 : audit [INF] from='client.25351 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:51.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:50.557709+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-02-19T21:04:51.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:50.559231+0000 mon.c (mon.2) 70 : audit [INF] from='client.? v1:172.21.15.163:0/2088464822' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1"}]: dispatch 2024-02-19T21:04:51.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:50.560261+0000 mon.a (mon.0) 937 : audit [INF] from='client.35207 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1"}]: dispatch 2024-02-19T21:04:51.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:50.565165+0000 mon.c (mon.2) 71 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]: dispatch 2024-02-19T21:04:51.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:50.565854+0000 mon.a (mon.0) 938 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]: dispatch 2024-02-19T21:04:51.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:51.390515+0000 mon.c (mon.2) 72 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:51.980 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-02-19T21:04:49.553584+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":151008107,"num_objects":249,"num_object_clones":0,"num_object_copies":747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":249,"num_whiteouts":0,"num_read":1941,"num_read_kb":49840,"num_write":1594,"num_write_kb":153096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1681,"ondisk_log_size":1681,"up":1452,"acting":1452,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1912,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2873796,"kb_used_data":451304,"kb_used_omap":0,"kb_used_meta":2421760,"kb_avail":747087420,"statfs":{"total":767960285184,"available":765017518080,"internally_reserved":0,"allocated":462135296,"data_stored":456686336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2479882240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.620004"},"pg_stats":[{"pgid":"56.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.16","version":"0'0","reported_seq":3,"reported_epoch":105,"state":"creating+activating","last_fresh":"2024-02-19T21:04:44.699446+0000","last_change":"2024-02-19T21:04:44.699446+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:44.699446+0000","last_undegraded":"2024-02-19T21:04:44.699446+0000","last_fullsized":"2024-02-19T21:04:44.699446+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.17","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.807653+0000","last_change":"2024-02-19T21:04:43.807366+0000","last_active":"2024-02-19T21:04:43.807653+0000","last_peered":"2024-02-19T21:04:43.807653+0000","last_clean":"2024-02-19T21:04:43.807653+0000","last_became_active":"2024-02-19T21:04:43.806817+0000","last_became_peered":"2024-02-19T21:04:43.806817+0000","last_unstale":"2024-02-19T21:04:43.807653+0000","last_undegraded":"2024-02-19T21:04:43.807653+0000","last_fullsized":"2024-02-19T21:04:43.807653+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.719649+0000","last_change":"2024-02-19T21:04:42.757960+0000","last_active":"2024-02-19T21:04:43.719649+0000","last_peered":"2024-02-19T21:04:43.719649+0000","last_clean":"2024-02-19T21:04:43.719649+0000","last_became_active":"2024-02-19T21:04:42.757431+0000","last_became_peered":"2024-02-19T21:04:42.757431+0000","last_unstale":"2024-02-19T21:04:43.719649+0000","last_undegraded":"2024-02-19T21:04:43.719649+0000","last_fullsized":"2024-02-19T21:04:43.719649+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.782931+0000","last_change":"2024-02-19T21:04:41.703839+0000","last_active":"2024-02-19T21:04:43.782931+0000","last_peered":"2024-02-19T21:04:43.782931+0000","last_clean":"2024-02-19T21:04:43.782931+0000","last_became_active":"2024-02-19T21:04:41.703455+0000","last_became_peered":"2024-02-19T21:04:41.703455+0000","last_unstale":"2024-02-19T21:04:43.782931+0000","last_undegraded":"2024-02-19T21:04:43.782931+0000","last_fullsized":"2024-02-19T21:04:43.782931+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.17","version":"0'0","reported_seq":3,"reported_epoch":105,"state":"creating+activating","last_fresh":"2024-02-19T21:04:44.699321+0000","last_change":"2024-02-19T21:04:44.699321+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:44.699321+0000","last_undegraded":"2024-02-19T21:04:44.699321+0000","last_fullsized":"2024-02-19T21:04:44.699321+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":["4","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.16","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.817950+0000","last_change":"2024-02-19T21:04:43.811248+0000","last_active":"2024-02-19T21:04:43.817950+0000","last_peered":"2024-02-19T21:04:43.817950+0000","last_clean":"2024-02-19T21:04:43.817950+0000","last_became_active":"2024-02-19T21:04:43.811084+0000","last_became_peered":"2024-02-19T21:04:43.811084+0000","last_unstale":"2024-02-19T21:04:43.817950+0000","last_undegraded":"2024-02-19T21:04:43.817950+0000","last_fullsized":"2024-02-19T21:04:43.817950+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.763213+0000","last_change":"2024-02-19T21:04:42.743671+0000","last_active":"2024-02-19T21:04:43.763213+0000","last_peered":"2024-02-19T21:04:43.763213+0000","last_clean":"2024-02-19T21:04:43.763213+0000","last_became_active":"2024-02-19T21:04:42.743072+0000","last_became_peered":"2024-02-19T21:04:42.743072+0000","last_unstale":"2024-02-19T21:04:43.763213+0000","last_undegraded":"2024-02-19T21:04:43.763213+0000","last_fullsized":"2024-02-19T21:04:43.763213+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.735021+0000","last_change":"2024-02-19T21:04:41.697047+0000","last_active":"2024-02-19T21:04:42.735021+0000","last_peered":"2024-02-19T21:04:42.735021+0000","last_clean":"2024-02-19T21:04:42.735021+0000","last_became_active":"2024-02-19T21:04:41.696606+0000","last_became_peered":"2024-02-19T21:04:41.696606+0000","last_unstale":"2024-02-19T21:04:42.735021+0000","last_undegraded":"2024-02-19T21:04:42.735021+0000","last_fullsized":"2024-02-19T21:04:42.735021+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"56.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.709120+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.15","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.797542+0000","last_change":"2024-02-19T21:04:43.794162+0000","last_active":"2024-02-19T21:04:43.797542+0000","last_peered":"2024-02-19T21:04:43.797542+0000","last_clean":"2024-02-19T21:04:43.797542+0000","last_became_active":"2024-02-19T21:04:43.765864+0000","last_became_peered":"2024-02-19T21:04:43.765864+0000","last_unstale":"2024-02-19T21:04:43.797542+0000","last_undegraded":"2024-02-19T21:04:43.797542+0000","last_fullsized":"2024-02-19T21:04:43.797542+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.787460+0000","last_change":"2024-02-19T21:04:42.739337+0000","last_active":"2024-02-19T21:04:43.787460+0000","last_peered":"2024-02-19T21:04:43.787460+0000","last_clean":"2024-02-19T21:04:43.787460+0000","last_became_active":"2024-02-19T21:04:42.738646+0000","last_became_peered":"2024-02-19T21:04:42.738646+0000","last_unstale":"2024-02-19T21:04:43.787460+0000","last_undegraded":"2024-02-19T21:04:43.787460+0000","last_fullsized":"2024-02-19T21:04:43.787460+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.693611+0000","last_change":"2024-02-19T21:04:41.703245+0000","last_active":"2024-02-19T21:04:42.693611+0000","last_peered":"2024-02-19T21:04:42.693611+0000","last_clean":"2024-02-19T21:04:42.693611+0000","last_became_active":"2024-02-19T21:04:41.702973+0000","last_became_peered":"2024-02-19T21:04:41.702973+0000","last_unstale":"2024-02-19T21:04:42.693611+0000","last_undegraded":"2024-02-19T21:04:42.693611+0000","last_fullsized":"2024-02-19T21:04:42.693611+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.15","version":"0'0","reported_seq":2,"reported_epoch":105,"state":"creating+activating","last_fresh":"2024-02-19T21:04:44.700345+0000","last_change":"2024-02-19T21:04:44.700514+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:44.700345+0000","last_undegraded":"2024-02-19T21:04:44.700345+0000","last_fullsized":"2024-02-19T21:04:44.700345+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":["0","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.14","version":"0'0","reported_seq":9,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.775816+0000","last_change":"2024-02-19T21:04:43.775816+0000","last_active":"2024-02-19T21:04:43.775816+0000","last_peered":"2024-02-19T21:04:43.775816+0000","last_clean":"2024-02-19T21:04:43.775816+0000","last_became_active":"2024-02-19T21:04:43.775443+0000","last_became_peered":"2024-02-19T21:04:43.775443+0000","last_unstale":"2024-02-19T21:04:43.775816+0000","last_undegraded":"2024-02-19T21:04:43.775816+0000","last_fullsized":"2024-02-19T21:04:43.775816+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.743674+0000","last_change":"2024-02-19T21:04:42.743179+0000","last_active":"2024-02-19T21:04:42.743674+0000","last_peered":"2024-02-19T21:04:42.743674+0000","last_clean":"2024-02-19T21:04:42.743674+0000","last_became_active":"2024-02-19T21:04:42.742869+0000","last_became_peered":"2024-02-19T21:04:42.742869+0000","last_unstale":"2024-02-19T21:04:42.743674+0000","last_undegraded":"2024-02-19T21:04:42.743674+0000","last_fullsized":"2024-02-19T21:04:42.743674+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.691820+0000","last_change":"2024-02-19T21:04:41.705402+0000","last_active":"2024-02-19T21:04:42.691820+0000","last_peered":"2024-02-19T21:04:42.691820+0000","last_clean":"2024-02-19T21:04:42.691820+0000","last_became_active":"2024-02-19T21:04:41.705104+0000","last_became_peered":"2024-02-19T21:04:41.705104+0000","last_unstale":"2024-02-19T21:04:42.691820+0000","last_undegraded":"2024-02-19T21:04:42.691820+0000","last_fullsized":"2024-02-19T21:04:42.691820+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.714190+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.13","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.792299+0000","last_change":"2024-02-19T21:04:43.790457+0000","last_active":"2024-02-19T21:04:43.792299+0000","last_peered":"2024-02-19T21:04:43.792299+0000","last_clean":"2024-02-19T21:04:43.792299+0000","last_became_active":"2024-02-19T21:04:43.789452+0000","last_became_peered":"2024-02-19T21:04:43.789452+0000","last_unstale":"2024-02-19T21:04:43.792299+0000","last_undegraded":"2024-02-19T21:04:43.792299+0000","last_fullsized":"2024-02-19T21:04:43.792299+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.745732+0000","last_change":"2024-02-19T21:04:42.744931+0000","last_active":"2024-02-19T21:04:42.745732+0000","last_peered":"2024-02-19T21:04:42.745732+0000","last_clean":"2024-02-19T21:04:42.745732+0000","last_became_active":"2024-02-19T21:04:42.744443+0000","last_became_peered":"2024-02-19T21:04:42.744443+0000","last_unstale":"2024-02-19T21:04:42.745732+0000","last_undegraded":"2024-02-19T21:04:42.745732+0000","last_fullsized":"2024-02-19T21:04:42.745732+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.700352+0000","last_change":"2024-02-19T21:04:41.711688+0000","last_active":"2024-02-19T21:04:42.700352+0000","last_peered":"2024-02-19T21:04:42.700352+0000","last_clean":"2024-02-19T21:04:42.700352+0000","last_became_active":"2024-02-19T21:04:41.711348+0000","last_became_peered":"2024-02-19T21:04:41.711348+0000","last_unstale":"2024-02-19T21:04:42.700352+0000","last_undegraded":"2024-02-19T21:04:42.700352+0000","last_fullsized":"2024-02-19T21:04:42.700352+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"56.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.711632+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.12","version":"0'0","reported_seq":9,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.756255+0000","last_change":"2024-02-19T21:04:43.756255+0000","last_active":"2024-02-19T21:04:43.756255+0000","last_peered":"2024-02-19T21:04:43.756255+0000","last_clean":"2024-02-19T21:04:43.756255+0000","last_became_active":"2024-02-19T21:04:43.755603+0000","last_became_peered":"2024-02-19T21:04:43.755603+0000","last_unstale":"2024-02-19T21:04:43.756255+0000","last_undegraded":"2024-02-19T21:04:43.756255+0000","last_fullsized":"2024-02-19T21:04:43.756255+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.727178+0000","last_change":"2024-02-19T21:04:42.725174+0000","last_active":"2024-02-19T21:04:42.727178+0000","last_peered":"2024-02-19T21:04:42.727178+0000","last_clean":"2024-02-19T21:04:42.727178+0000","last_became_active":"2024-02-19T21:04:42.724881+0000","last_became_peered":"2024-02-19T21:04:42.724881+0000","last_unstale":"2024-02-19T21:04:42.727178+0000","last_undegraded":"2024-02-19T21:04:42.727178+0000","last_fullsized":"2024-02-19T21:04:42.727178+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.741640+0000","last_change":"2024-02-19T21:04:41.704232+0000","last_active":"2024-02-19T21:04:43.741640+0000","last_peered":"2024-02-19T21:04:43.741640+0000","last_clean":"2024-02-19T21:04:43.741640+0000","last_became_active":"2024-02-19T21:04:41.704047+0000","last_became_peered":"2024-02-19T21:04:41.704047+0000","last_unstale":"2024-02-19T21:04:43.741640+0000","last_undegraded":"2024-02-19T21:04:43.741640+0000","last_fullsized":"2024-02-19T21:04:43.741640+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.724695+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.11","version":"0'0","reported_seq":9,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.743738+0000","last_change":"2024-02-19T21:04:43.743738+0000","last_active":"2024-02-19T21:04:43.743738+0000","last_peered":"2024-02-19T21:04:43.743738+0000","last_clean":"2024-02-19T21:04:43.743738+0000","last_became_active":"2024-02-19T21:04:43.743476+0000","last_became_peered":"2024-02-19T21:04:43.743476+0000","last_unstale":"2024-02-19T21:04:43.743738+0000","last_undegraded":"2024-02-19T21:04:43.743738+0000","last_fullsized":"2024-02-19T21:04:43.743738+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.727882+0000","last_change":"2024-02-19T21:04:42.723207+0000","last_active":"2024-02-19T21:04:42.727882+0000","last_peered":"2024-02-19T21:04:42.727882+0000","last_clean":"2024-02-19T21:04:42.727882+0000","last_became_active":"2024-02-19T21:04:42.722745+0000","last_became_peered":"2024-02-19T21:04:42.722745+0000","last_unstale":"2024-02-19T21:04:42.727882+0000","last_undegraded":"2024-02-19T21:04:42.727882+0000","last_fullsized":"2024-02-19T21:04:42.727882+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.698339+0000","last_change":"2024-02-19T21:04:41.689219+0000","last_active":"2024-02-19T21:04:42.698339+0000","last_peered":"2024-02-19T21:04:42.698339+0000","last_clean":"2024-02-19T21:04:42.698339+0000","last_became_active":"2024-02-19T21:04:41.688933+0000","last_became_peered":"2024-02-19T21:04:41.688933+0000","last_unstale":"2024-02-19T21:04:42.698339+0000","last_undegraded":"2024-02-19T21:04:42.698339+0000","last_fullsized":"2024-02-19T21:04:42.698339+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.711358+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.10","version":"0'0","reported_seq":9,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.775360+0000","last_change":"2024-02-19T21:04:43.775360+0000","last_active":"2024-02-19T21:04:43.775360+0000","last_peered":"2024-02-19T21:04:43.775360+0000","last_clean":"2024-02-19T21:04:43.775360+0000","last_became_active":"2024-02-19T21:04:43.769028+0000","last_became_peered":"2024-02-19T21:04:43.769028+0000","last_unstale":"2024-02-19T21:04:43.775360+0000","last_undegraded":"2024-02-19T21:04:43.775360+0000","last_fullsized":"2024-02-19T21:04:43.775360+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.745587+0000","last_change":"2024-02-19T21:04:42.745123+0000","last_active":"2024-02-19T21:04:42.745587+0000","last_peered":"2024-02-19T21:04:42.745587+0000","last_clean":"2024-02-19T21:04:42.745587+0000","last_became_active":"2024-02-19T21:04:42.744820+0000","last_became_peered":"2024-02-19T21:04:42.744820+0000","last_unstale":"2024-02-19T21:04:42.745587+0000","last_undegraded":"2024-02-19T21:04:42.745587+0000","last_fullsized":"2024-02-19T21:04:42.745587+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.700172+0000","last_change":"2024-02-19T21:04:41.694610+0000","last_active":"2024-02-19T21:04:42.700172+0000","last_peered":"2024-02-19T21:04:42.700172+0000","last_clean":"2024-02-19T21:04:42.700172+0000","last_became_active":"2024-02-19T21:04:41.694265+0000","last_became_peered":"2024-02-19T21:04:41.694265+0000","last_unstale":"2024-02-19T21:04:42.700172+0000","last_undegraded":"2024-02-19T21:04:42.700172+0000","last_fullsized":"2024-02-19T21:04:42.700172+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"56.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"54.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"51.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"50.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.705284+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.1f","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.823450+0000","last_change":"2024-02-19T21:04:43.817575+0000","last_active":"2024-02-19T21:04:43.823450+0000","last_peered":"2024-02-19T21:04:43.823450+0000","last_clean":"2024-02-19T21:04:43.823450+0000","last_became_active":"2024-02-19T21:04:43.817283+0000","last_became_peered":"2024-02-19T21:04:43.817283+0000","last_unstale":"2024-02-19T21:04:43.823450+0000","last_undegraded":"2024-02-19T21:04:43.823450+0000","last_fullsized":"2024-02-19T21:04:43.823450+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.742466+0000","last_change":"2024-02-19T21:04:42.742007+0000","last_active":"2024-02-19T21:04:42.742466+0000","last_peered":"2024-02-19T21:04:42.742466+0000","last_clean":"2024-02-19T21:04:42.742466+0000","last_became_active":"2024-02-19T21:04:42.741686+0000","last_became_peered":"2024-02-19T21:04:42.741686+0000","last_unstale":"2024-02-19T21:04:42.742466+0000","last_undegraded":"2024-02-19T21:04:42.742466+0000","last_fullsized":"2024-02-19T21:04:42.742466+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.785268+0000","last_change":"2024-02-19T21:04:41.713793+0000","last_active":"2024-02-19T21:04:43.785268+0000","last_peered":"2024-02-19T21:04:43.785268+0000","last_clean":"2024-02-19T21:04:43.785268+0000","last_became_active":"2024-02-19T21:04:41.713157+0000","last_became_peered":"2024-02-19T21:04:41.713157+0000","last_unstale":"2024-02-19T21:04:43.785268+0000","last_undegraded":"2024-02-19T21:04:43.785268+0000","last_fullsized":"2024-02-19T21:04:43.785268+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"54.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"51.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"50.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.717551+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.1e","version":"0'0","reported_seq":9,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.774699+0000","last_change":"2024-02-19T21:04:43.774699+0000","last_active":"2024-02-19T21:04:43.774699+0000","last_peered":"2024-02-19T21:04:43.774699+0000","last_clean":"2024-02-19T21:04:43.774699+0000","last_became_active":"2024-02-19T21:04:43.774399+0000","last_became_peered":"2024-02-19T21:04:43.774399+0000","last_unstale":"2024-02-19T21:04:43.774699+0000","last_undegraded":"2024-02-19T21:04:43.774699+0000","last_fullsized":"2024-02-19T21:04:43.774699+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.767958+0000","last_change":"2024-02-19T21:04:43.759901+0000","last_active":"2024-02-19T21:04:43.767958+0000","last_peered":"2024-02-19T21:04:43.767958+0000","last_clean":"2024-02-19T21:04:43.767958+0000","last_became_active":"2024-02-19T21:04:43.759289+0000","last_became_peered":"2024-02-19T21:04:43.759289+0000","last_unstale":"2024-02-19T21:04:43.767958+0000","last_undegraded":"2024-02-19T21:04:43.767958+0000","last_fullsized":"2024-02-19T21:04:43.767958+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.695596+0000","last_change":"2024-02-19T21:04:41.705982+0000","last_active":"2024-02-19T21:04:42.695596+0000","last_peered":"2024-02-19T21:04:42.695596+0000","last_clean":"2024-02-19T21:04:42.695596+0000","last_became_active":"2024-02-19T21:04:41.705692+0000","last_became_peered":"2024-02-19T21:04:41.705692+0000","last_unstale":"2024-02-19T21:04:42.695596+0000","last_undegraded":"2024-02-19T21:04:42.695596+0000","last_fullsized":"2024-02-19T21:04:42.695596+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"54.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"51.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"50.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.1c","version":"0'0","reported_seq":2,"reported_epoch":105,"state":"creating+activating","last_fresh":"2024-02-19T21:04:44.698875+0000","last_change":"2024-02-19T21:04:44.699162+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:44.698875+0000","last_undegraded":"2024-02-19T21:04:44.698875+0000","last_fullsized":"2024-02-19T21:04:44.698875+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":["4","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1d","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.771775+0000","last_change":"2024-02-19T21:04:43.767396+0000","last_active":"2024-02-19T21:04:43.771775+0000","last_peered":"2024-02-19T21:04:43.771775+0000","last_clean":"2024-02-19T21:04:43.771775+0000","last_became_active":"2024-02-19T21:04:43.767112+0000","last_became_peered":"2024-02-19T21:04:43.767112+0000","last_unstale":"2024-02-19T21:04:43.771775+0000","last_undegraded":"2024-02-19T21:04:43.771775+0000","last_fullsized":"2024-02-19T21:04:43.771775+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.726959+0000","last_change":"2024-02-19T21:04:42.722262+0000","last_active":"2024-02-19T21:04:42.726959+0000","last_peered":"2024-02-19T21:04:42.726959+0000","last_clean":"2024-02-19T21:04:42.726959+0000","last_became_active":"2024-02-19T21:04:42.721790+0000","last_became_peered":"2024-02-19T21:04:42.721790+0000","last_unstale":"2024-02-19T21:04:42.726959+0000","last_undegraded":"2024-02-19T21:04:42.726959+0000","last_fullsized":"2024-02-19T21:04:42.726959+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.693658+0000","last_change":"2024-02-19T21:04:41.695115+0000","last_active":"2024-02-19T21:04:42.693658+0000","last_peered":"2024-02-19T21:04:42.693658+0000","last_clean":"2024-02-19T21:04:42.693658+0000","last_became_active":"2024-02-19T21:04:41.694815+0000","last_became_peered":"2024-02-19T21:04:41.694815+0000","last_unstale":"2024-02-19T21:04:42.693658+0000","last_undegraded":"2024-02-19T21:04:42.693658+0000","last_fullsized":"2024-02-19T21:04:42.693658+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"54.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"51.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"50.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.710282+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.1c","version":"0'0","reported_seq":9,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.777932+0000","last_change":"2024-02-19T21:04:43.777932+0000","last_active":"2024-02-19T21:04:43.777932+0000","last_peered":"2024-02-19T21:04:43.777932+0000","last_clean":"2024-02-19T21:04:43.777932+0000","last_became_active":"2024-02-19T21:04:43.777189+0000","last_became_peered":"2024-02-19T21:04:43.777189+0000","last_unstale":"2024-02-19T21:04:43.777932+0000","last_undegraded":"2024-02-19T21:04:43.777932+0000","last_fullsized":"2024-02-19T21:04:43.777932+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.774528+0000","last_change":"2024-02-19T21:04:43.768740+0000","last_active":"2024-02-19T21:04:43.774528+0000","last_peered":"2024-02-19T21:04:43.774528+0000","last_clean":"2024-02-19T21:04:43.774528+0000","last_became_active":"2024-02-19T21:04:43.766339+0000","last_became_peered":"2024-02-19T21:04:43.766339+0000","last_unstale":"2024-02-19T21:04:43.774528+0000","last_undegraded":"2024-02-19T21:04:43.774528+0000","last_fullsized":"2024-02-19T21:04:43.774528+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.719812+0000","last_change":"2024-02-19T21:04:41.710440+0000","last_active":"2024-02-19T21:04:43.719812+0000","last_peered":"2024-02-19T21:04:43.719812+0000","last_clean":"2024-02-19T21:04:43.719812+0000","last_became_active":"2024-02-19T21:04:41.709507+0000","last_became_peered":"2024-02-19T21:04:41.709507+0000","last_unstale":"2024-02-19T21:04:43.719812+0000","last_undegraded":"2024-02-19T21:04:43.719812+0000","last_fullsized":"2024-02-19T21:04:43.719812+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"51.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"50.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.1a","version":"0'0","reported_seq":2,"reported_epoch":105,"state":"creating+activating","last_fresh":"2024-02-19T21:04:44.698772+0000","last_change":"2024-02-19T21:04:44.698940+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:44.698772+0000","last_undegraded":"2024-02-19T21:04:44.698772+0000","last_fullsized":"2024-02-19T21:04:44.698772+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1b","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.768649+0000","last_change":"2024-02-19T21:04:43.762980+0000","last_active":"2024-02-19T21:04:43.768649+0000","last_peered":"2024-02-19T21:04:43.768649+0000","last_clean":"2024-02-19T21:04:43.768649+0000","last_became_active":"2024-02-19T21:04:43.762392+0000","last_became_peered":"2024-02-19T21:04:43.762392+0000","last_unstale":"2024-02-19T21:04:43.768649+0000","last_undegraded":"2024-02-19T21:04:43.768649+0000","last_fullsized":"2024-02-19T21:04:43.768649+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.732212+0000","last_change":"2024-02-19T21:04:42.723102+0000","last_active":"2024-02-19T21:04:42.732212+0000","last_peered":"2024-02-19T21:04:42.732212+0000","last_clean":"2024-02-19T21:04:42.732212+0000","last_became_active":"2024-02-19T21:04:42.722678+0000","last_became_peered":"2024-02-19T21:04:42.722678+0000","last_unstale":"2024-02-19T21:04:42.732212+0000","last_undegraded":"2024-02-19T21:04:42.732212+0000","last_fullsized":"2024-02-19T21:04:42.732212+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.703139+0000","last_change":"2024-02-19T21:04:41.711642+0000","last_active":"2024-02-19T21:04:43.703139+0000","last_peered":"2024-02-19T21:04:43.703139+0000","last_clean":"2024-02-19T21:04:43.703139+0000","last_became_active":"2024-02-19T21:04:41.710717+0000","last_became_peered":"2024-02-19T21:04:41.710717+0000","last_unstale":"2024-02-19T21:04:43.703139+0000","last_undegraded":"2024-02-19T21:04:43.703139+0000","last_fullsized":"2024-02-19T21:04:43.703139+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"51.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"50.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.1b","version":"0'0","reported_seq":3,"reported_epoch":105,"state":"creating+activating","last_fresh":"2024-02-19T21:04:44.700357+0000","last_change":"2024-02-19T21:04:44.700357+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:44.700357+0000","last_undegraded":"2024-02-19T21:04:44.700357+0000","last_fullsized":"2024-02-19T21:04:44.700357+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":["0","1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.1a","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.810873+0000","last_change":"2024-02-19T21:04:43.806265+0000","last_active":"2024-02-19T21:04:43.810873+0000","last_peered":"2024-02-19T21:04:43.810873+0000","last_clean":"2024-02-19T21:04:43.810873+0000","last_became_active":"2024-02-19T21:04:43.805330+0000","last_became_peered":"2024-02-19T21:04:43.805330+0000","last_unstale":"2024-02-19T21:04:43.810873+0000","last_undegraded":"2024-02-19T21:04:43.810873+0000","last_fullsized":"2024-02-19T21:04:43.810873+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.728303+0000","last_change":"2024-02-19T21:04:42.727946+0000","last_active":"2024-02-19T21:04:42.728303+0000","last_peered":"2024-02-19T21:04:42.728303+0000","last_clean":"2024-02-19T21:04:42.728303+0000","last_became_active":"2024-02-19T21:04:42.727725+0000","last_became_peered":"2024-02-19T21:04:42.727725+0000","last_unstale":"2024-02-19T21:04:42.728303+0000","last_undegraded":"2024-02-19T21:04:42.728303+0000","last_fullsized":"2024-02-19T21:04:42.728303+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.754539+0000","last_change":"2024-02-19T21:04:41.710160+0000","last_active":"2024-02-19T21:04:43.754539+0000","last_peered":"2024-02-19T21:04:43.754539+0000","last_clean":"2024-02-19T21:04:43.754539+0000","last_became_active":"2024-02-19T21:04:41.707805+0000","last_became_peered":"2024-02-19T21:04:41.707805+0000","last_unstale":"2024-02-19T21:04:43.754539+0000","last_undegraded":"2024-02-19T21:04:43.754539+0000","last_fullsized":"2024-02-19T21:04:43.754539+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"51.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"50.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.715741+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.19","version":"0'0","reported_seq":9,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.782668+0000","last_change":"2024-02-19T21:04:43.782668+0000","last_active":"2024-02-19T21:04:43.782668+0000","last_peered":"2024-02-19T21:04:43.782668+0000","last_clean":"2024-02-19T21:04:43.782668+0000","last_became_active":"2024-02-19T21:04:43.782287+0000","last_became_peered":"2024-02-19T21:04:43.782287+0000","last_unstale":"2024-02-19T21:04:43.782668+0000","last_undegraded":"2024-02-19T21:04:43.782668+0000","last_fullsized":"2024-02-19T21:04:43.782668+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.741054+0000","last_change":"2024-02-19T21:04:42.740812+0000","last_active":"2024-02-19T21:04:42.741054+0000","last_peered":"2024-02-19T21:04:42.741054+0000","last_clean":"2024-02-19T21:04:42.741054+0000","last_became_active":"2024-02-19T21:04:42.740516+0000","last_became_peered":"2024-02-19T21:04:42.740516+0000","last_unstale":"2024-02-19T21:04:42.741054+0000","last_undegraded":"2024-02-19T21:04:42.741054+0000","last_fullsized":"2024-02-19T21:04:42.741054+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.696758+0000","last_change":"2024-02-19T21:04:41.689549+0000","last_active":"2024-02-19T21:04:42.696758+0000","last_peered":"2024-02-19T21:04:42.696758+0000","last_clean":"2024-02-19T21:04:42.696758+0000","last_became_active":"2024-02-19T21:04:41.689021+0000","last_became_peered":"2024-02-19T21:04:41.689021+0000","last_unstale":"2024-02-19T21:04:42.696758+0000","last_undegraded":"2024-02-19T21:04:42.696758+0000","last_fullsized":"2024-02-19T21:04:42.696758+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"51.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"50.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.19","version":"0'0","reported_seq":3,"reported_epoch":105,"state":"creating+activating","last_fresh":"2024-02-19T21:04:44.707862+0000","last_change":"2024-02-19T21:04:44.707862+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:44.707862+0000","last_undegraded":"2024-02-19T21:04:44.707862+0000","last_fullsized":"2024-02-19T21:04:44.707862+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":["0","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.18","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.815510+0000","last_change":"2024-02-19T21:04:43.809873+0000","last_active":"2024-02-19T21:04:43.815510+0000","last_peered":"2024-02-19T21:04:43.815510+0000","last_clean":"2024-02-19T21:04:43.815510+0000","last_became_active":"2024-02-19T21:04:43.809514+0000","last_became_peered":"2024-02-19T21:04:43.809514+0000","last_unstale":"2024-02-19T21:04:43.815510+0000","last_undegraded":"2024-02-19T21:04:43.815510+0000","last_fullsized":"2024-02-19T21:04:43.815510+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.750553+0000","last_change":"2024-02-19T21:04:42.747748+0000","last_active":"2024-02-19T21:04:42.750553+0000","last_peered":"2024-02-19T21:04:42.750553+0000","last_clean":"2024-02-19T21:04:42.750553+0000","last_became_active":"2024-02-19T21:04:42.747433+0000","last_became_peered":"2024-02-19T21:04:42.747433+0000","last_unstale":"2024-02-19T21:04:42.750553+0000","last_undegraded":"2024-02-19T21:04:42.750553+0000","last_fullsized":"2024-02-19T21:04:42.750553+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.692193+0000","last_change":"2024-02-19T21:04:41.731194+0000","last_active":"2024-02-19T21:04:42.692193+0000","last_peered":"2024-02-19T21:04:42.692193+0000","last_clean":"2024-02-19T21:04:42.692193+0000","last_became_active":"2024-02-19T21:04:41.730872+0000","last_became_peered":"2024-02-19T21:04:41.730872+0000","last_unstale":"2024-02-19T21:04:42.692193+0000","last_undegraded":"2024-02-19T21:04:42.692193+0000","last_fullsized":"2024-02-19T21:04:42.692193+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.6","version":"0'0","reported_seq":2,"reported_epoch":105,"state":"creating+activating","last_fresh":"2024-02-19T21:04:44.698963+0000","last_change":"2024-02-19T21:04:44.699125+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:44.698963+0000","last_undegraded":"2024-02-19T21:04:44.698963+0000","last_fullsized":"2024-02-19T21:04:44.698963+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":["4","2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.7","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.820015+0000","last_change":"2024-02-19T21:04:43.813144+0000","last_active":"2024-02-19T21:04:43.820015+0000","last_peered":"2024-02-19T21:04:43.820015+0000","last_clean":"2024-02-19T21:04:43.820015+0000","last_became_active":"2024-02-19T21:04:43.812473+0000","last_became_peered":"2024-02-19T21:04:43.812473+0000","last_unstale":"2024-02-19T21:04:43.820015+0000","last_undegraded":"2024-02-19T21:04:43.820015+0000","last_fullsized":"2024-02-19T21:04:43.820015+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.736401+0000","last_change":"2024-02-19T21:04:42.733191+0000","last_active":"2024-02-19T21:04:42.736401+0000","last_peered":"2024-02-19T21:04:42.736401+0000","last_clean":"2024-02-19T21:04:42.736401+0000","last_became_active":"2024-02-19T21:04:42.732868+0000","last_became_peered":"2024-02-19T21:04:42.732868+0000","last_unstale":"2024-02-19T21:04:42.736401+0000","last_undegraded":"2024-02-19T21:04:42.736401+0000","last_fullsized":"2024-02-19T21:04:42.736401+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.724545+0000","last_change":"2024-02-19T21:04:41.706804+0000","last_active":"2024-02-19T21:04:43.724545+0000","last_peered":"2024-02-19T21:04:43.724545+0000","last_clean":"2024-02-19T21:04:43.724545+0000","last_became_active":"2024-02-19T21:04:41.705621+0000","last_became_peered":"2024-02-19T21:04:41.705621+0000","last_unstale":"2024-02-19T21:04:43.724545+0000","last_undegraded":"2024-02-19T21:04:43.724545+0000","last_fullsized":"2024-02-19T21:04:43.724545+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.706799+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.6","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.779655+0000","last_change":"2024-02-19T21:04:43.779219+0000","last_active":"2024-02-19T21:04:43.779655+0000","last_peered":"2024-02-19T21:04:43.779655+0000","last_clean":"2024-02-19T21:04:43.779655+0000","last_became_active":"2024-02-19T21:04:43.778903+0000","last_became_peered":"2024-02-19T21:04:43.778903+0000","last_unstale":"2024-02-19T21:04:43.779655+0000","last_undegraded":"2024-02-19T21:04:43.779655+0000","last_fullsized":"2024-02-19T21:04:43.779655+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.763268+0000","last_change":"2024-02-19T21:04:42.743519+0000","last_active":"2024-02-19T21:04:43.763268+0000","last_peered":"2024-02-19T21:04:43.763268+0000","last_clean":"2024-02-19T21:04:43.763268+0000","last_became_active":"2024-02-19T21:04:42.742687+0000","last_became_peered":"2024-02-19T21:04:42.742687+0000","last_unstale":"2024-02-19T21:04:43.763268+0000","last_undegraded":"2024-02-19T21:04:43.763268+0000","last_fullsized":"2024-02-19T21:04:43.763268+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.695672+0000","last_change":"2024-02-19T21:04:41.731189+0000","last_active":"2024-02-19T21:04:42.695672+0000","last_peered":"2024-02-19T21:04:42.695672+0000","last_clean":"2024-02-19T21:04:42.695672+0000","last_became_active":"2024-02-19T21:04:41.730913+0000","last_became_peered":"2024-02-19T21:04:41.730913+0000","last_unstale":"2024-02-19T21:04:42.695672+0000","last_undegraded":"2024-02-19T21:04:42.695672+0000","last_fullsized":"2024-02-19T21:04:42.695672+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"56.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.705015+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.5","version":"0'0","reported_seq":9,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.774663+0000","last_change":"2024-02-19T21:04:43.774663+0000","last_active":"2024-02-19T21:04:43.774663+0000","last_peered":"2024-02-19T21:04:43.774663+0000","last_clean":"2024-02-19T21:04:43.774663+0000","last_became_active":"2024-02-19T21:04:43.774198+0000","last_became_peered":"2024-02-19T21:04:43.774198+0000","last_unstale":"2024-02-19T21:04:43.774663+0000","last_undegraded":"2024-02-19T21:04:43.774663+0000","last_fullsized":"2024-02-19T21:04:43.774663+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.759141+0000","last_change":"2024-02-19T21:04:42.750549+0000","last_active":"2024-02-19T21:04:42.759141+0000","last_peered":"2024-02-19T21:04:42.759141+0000","last_clean":"2024-02-19T21:04:42.759141+0000","last_became_active":"2024-02-19T21:04:42.750260+0000","last_became_peered":"2024-02-19T21:04:42.750260+0000","last_unstale":"2024-02-19T21:04:42.759141+0000","last_undegraded":"2024-02-19T21:04:42.759141+0000","last_fullsized":"2024-02-19T21:04:42.759141+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.718244+0000","last_change":"2024-02-19T21:04:41.732268+0000","last_active":"2024-02-19T21:04:42.718244+0000","last_peered":"2024-02-19T21:04:42.718244+0000","last_clean":"2024-02-19T21:04:42.718244+0000","last_became_active":"2024-02-19T21:04:41.731950+0000","last_became_peered":"2024-02-19T21:04:41.731950+0000","last_unstale":"2024-02-19T21:04:42.718244+0000","last_undegraded":"2024-02-19T21:04:42.718244+0000","last_fullsized":"2024-02-19T21:04:42.718244+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.707643+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.4","version":"0'0","reported_seq":9,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.775655+0000","last_change":"2024-02-19T21:04:43.775655+0000","last_active":"2024-02-19T21:04:43.775655+0000","last_peered":"2024-02-19T21:04:43.775655+0000","last_clean":"2024-02-19T21:04:43.775655+0000","last_became_active":"2024-02-19T21:04:43.775040+0000","last_became_peered":"2024-02-19T21:04:43.775040+0000","last_unstale":"2024-02-19T21:04:43.775655+0000","last_undegraded":"2024-02-19T21:04:43.775655+0000","last_fullsized":"2024-02-19T21:04:43.775655+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.782806+0000","last_change":"2024-02-19T21:04:42.738272+0000","last_active":"2024-02-19T21:04:43.782806+0000","last_peered":"2024-02-19T21:04:43.782806+0000","last_clean":"2024-02-19T21:04:43.782806+0000","last_became_active":"2024-02-19T21:04:42.737993+0000","last_became_peered":"2024-02-19T21:04:42.737993+0000","last_unstale":"2024-02-19T21:04:43.782806+0000","last_undegraded":"2024-02-19T21:04:43.782806+0000","last_fullsized":"2024-02-19T21:04:43.782806+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.726177+0000","last_change":"2024-02-19T21:04:41.705548+0000","last_active":"2024-02-19T21:04:42.726177+0000","last_peered":"2024-02-19T21:04:42.726177+0000","last_clean":"2024-02-19T21:04:42.726177+0000","last_became_active":"2024-02-19T21:04:41.705213+0000","last_became_peered":"2024-02-19T21:04:41.705213+0000","last_unstale":"2024-02-19T21:04:42.726177+0000","last_undegraded":"2024-02-19T21:04:42.726177+0000","last_fullsized":"2024-02-19T21:04:42.726177+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"56.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.709681+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"40.3","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.816359+0000","last_change":"2024-02-19T21:04:43.810488+0000","last_active":"2024-02-19T21:04:43.816359+0000","last_peered":"2024-02-19T21:04:43.816359+0000","last_clean":"2024-02-19T21:04:43.816359+0000","last_became_active":"2024-02-19T21:04:43.806612+0000","last_became_peered":"2024-02-19T21:04:43.806612+0000","last_unstale":"2024-02-19T21:04:43.816359+0000","last_undegraded":"2024-02-19T21:04:43.816359+0000","last_fullsized":"2024-02-19T21:04:43.816359+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.736898+0000","last_change":"2024-02-19T21:04:42.736310+0000","last_active":"2024-02-19T21:04:42.736898+0000","last_peered":"2024-02-19T21:04:42.736898+0000","last_clean":"2024-02-19T21:04:42.736898+0000","last_became_active":"2024-02-19T21:04:42.735695+0000","last_became_peered":"2024-02-19T21:04:42.735695+0000","last_unstale":"2024-02-19T21:04:42.736898+0000","last_undegraded":"2024-02-19T21:04:42.736898+0000","last_fullsized":"2024-02-19T21:04:42.736898+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.701461+0000","last_change":"2024-02-19T21:04:41.702103+0000","last_active":"2024-02-19T21:04:42.701461+0000","last_peered":"2024-02-19T21:04:42.701461+0000","last_clean":"2024-02-19T21:04:42.701461+0000","last_became_active":"2024-02-19T21:04:41.701803+0000","last_became_peered":"2024-02-19T21:04:41.701803+0000","last_unstale":"2024-02-19T21:04:42.701461+0000","last_undegraded":"2024-02-19T21:04:42.701461+0000","last_fullsized":"2024-02-19T21:04:42.701461+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.3","version":"0'0","reported_seq":3,"reported_epoch":105,"state":"creating+activating","last_fresh":"2024-02-19T21:04:44.704492+0000","last_change":"2024-02-19T21:04:44.704492+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:44.704492+0000","last_undegraded":"2024-02-19T21:04:44.704492+0000","last_fullsized":"2024-02-19T21:04:44.704492+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":["3","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.2","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.821430+0000","last_change":"2024-02-19T21:04:43.817244+0000","last_active":"2024-02-19T21:04:43.821430+0000","last_peered":"2024-02-19T21:04:43.821430+0000","last_clean":"2024-02-19T21:04:43.821430+0000","last_became_active":"2024-02-19T21:04:43.813523+0000","last_became_peered":"2024-02-19T21:04:43.813523+0000","last_unstale":"2024-02-19T21:04:43.821430+0000","last_undegraded":"2024-02-19T21:04:43.821430+0000","last_fullsized":"2024-02-19T21:04:43.821430+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.762247+0000","last_change":"2024-02-19T21:04:42.740863+0000","last_active":"2024-02-19T21:04:43.762247+0000","last_peered":"2024-02-19T21:04:43.762247+0000","last_clean":"2024-02-19T21:04:43.762247+0000","last_became_active":"2024-02-19T21:04:42.740203+0000","last_became_peered":"2024-02-19T21:04:42.740203+0000","last_unstale":"2024-02-19T21:04:43.762247+0000","last_undegraded":"2024-02-19T21:04:43.762247+0000","last_fullsized":"2024-02-19T21:04:43.762247+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.702770+0000","last_change":"2024-02-19T21:04:41.709335+0000","last_active":"2024-02-19T21:04:43.702770+0000","last_peered":"2024-02-19T21:04:43.702770+0000","last_clean":"2024-02-19T21:04:43.702770+0000","last_became_active":"2024-02-19T21:04:41.707392+0000","last_became_peered":"2024-02-19T21:04:41.707392+0000","last_unstale":"2024-02-19T21:04:43.702770+0000","last_undegraded":"2024-02-19T21:04:43.702770+0000","last_fullsized":"2024-02-19T21:04:43.702770+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.0","version":"0'0","reported_seq":2,"reported_epoch":105,"state":"creating+activating","last_fresh":"2024-02-19T21:04:44.698710+0000","last_change":"2024-02-19T21:04:44.698914+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:44.698710+0000","last_undegraded":"2024-02-19T21:04:44.698710+0000","last_fullsized":"2024-02-19T21:04:44.698710+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":["4","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1","version":"0'0","reported_seq":9,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.763336+0000","last_change":"2024-02-19T21:04:43.763336+0000","last_active":"2024-02-19T21:04:43.763336+0000","last_peered":"2024-02-19T21:04:43.763336+0000","last_clean":"2024-02-19T21:04:43.763336+0000","last_became_active":"2024-02-19T21:04:43.763004+0000","last_became_peered":"2024-02-19T21:04:43.763004+0000","last_unstale":"2024-02-19T21:04:43.763336+0000","last_undegraded":"2024-02-19T21:04:43.763336+0000","last_fullsized":"2024-02-19T21:04:43.763336+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.750984+0000","last_change":"2024-02-19T21:04:42.749675+0000","last_active":"2024-02-19T21:04:42.750984+0000","last_peered":"2024-02-19T21:04:42.750984+0000","last_clean":"2024-02-19T21:04:42.750984+0000","last_became_active":"2024-02-19T21:04:42.749377+0000","last_became_peered":"2024-02-19T21:04:42.749377+0000","last_unstale":"2024-02-19T21:04:42.750984+0000","last_undegraded":"2024-02-19T21:04:42.750984+0000","last_fullsized":"2024-02-19T21:04:42.750984+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.696835+0000","last_change":"2024-02-19T21:04:41.732238+0000","last_active":"2024-02-19T21:04:42.696835+0000","last_peered":"2024-02-19T21:04:42.696835+0000","last_clean":"2024-02-19T21:04:42.696835+0000","last_became_active":"2024-02-19T21:04:41.731945+0000","last_became_peered":"2024-02-19T21:04:41.731945+0000","last_unstale":"2024-02-19T21:04:42.696835+0000","last_undegraded":"2024-02-19T21:04:42.696835+0000","last_fullsized":"2024-02-19T21:04:42.696835+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"56.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.1","version":"0'0","reported_seq":2,"reported_epoch":105,"state":"creating+activating","last_fresh":"2024-02-19T21:04:44.701375+0000","last_change":"2024-02-19T21:04:44.701559+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:44.701375+0000","last_undegraded":"2024-02-19T21:04:44.701375+0000","last_fullsized":"2024-02-19T21:04:44.701375+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":["3","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.0","version":"0'0","reported_seq":9,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.754164+0000","last_change":"2024-02-19T21:04:43.754164+0000","last_active":"2024-02-19T21:04:43.754164+0000","last_peered":"2024-02-19T21:04:43.754164+0000","last_clean":"2024-02-19T21:04:43.754164+0000","last_became_active":"2024-02-19T21:04:43.753646+0000","last_became_peered":"2024-02-19T21:04:43.753646+0000","last_unstale":"2024-02-19T21:04:43.754164+0000","last_undegraded":"2024-02-19T21:04:43.754164+0000","last_fullsized":"2024-02-19T21:04:43.754164+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.741327+0000","last_change":"2024-02-19T21:04:42.740165+0000","last_active":"2024-02-19T21:04:42.741327+0000","last_peered":"2024-02-19T21:04:42.741327+0000","last_clean":"2024-02-19T21:04:42.741327+0000","last_became_active":"2024-02-19T21:04:42.739817+0000","last_became_peered":"2024-02-19T21:04:42.739817+0000","last_unstale":"2024-02-19T21:04:42.741327+0000","last_undegraded":"2024-02-19T21:04:42.741327+0000","last_fullsized":"2024-02-19T21:04:42.741327+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.702838+0000","last_change":"2024-02-19T21:04:41.709787+0000","last_active":"2024-02-19T21:04:43.702838+0000","last_peered":"2024-02-19T21:04:43.702838+0000","last_clean":"2024-02-19T21:04:43.702838+0000","last_became_active":"2024-02-19T21:04:41.709477+0000","last_became_peered":"2024-02-19T21:04:41.709477+0000","last_unstale":"2024-02-19T21:04:43.702838+0000","last_undegraded":"2024-02-19T21:04:43.702838+0000","last_fullsized":"2024-02-19T21:04:43.702838+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"56.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.709694+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.f","version":"0'0","reported_seq":9,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.799311+0000","last_change":"2024-02-19T21:04:43.799311+0000","last_active":"2024-02-19T21:04:43.799311+0000","last_peered":"2024-02-19T21:04:43.799311+0000","last_clean":"2024-02-19T21:04:43.799311+0000","last_became_active":"2024-02-19T21:04:43.798909+0000","last_became_peered":"2024-02-19T21:04:43.798909+0000","last_unstale":"2024-02-19T21:04:43.799311+0000","last_undegraded":"2024-02-19T21:04:43.799311+0000","last_fullsized":"2024-02-19T21:04:43.799311+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.735575+0000","last_change":"2024-02-19T21:04:42.734269+0000","last_active":"2024-02-19T21:04:42.735575+0000","last_peered":"2024-02-19T21:04:42.735575+0000","last_clean":"2024-02-19T21:04:42.735575+0000","last_became_active":"2024-02-19T21:04:42.733974+0000","last_became_peered":"2024-02-19T21:04:42.733974+0000","last_unstale":"2024-02-19T21:04:42.735575+0000","last_undegraded":"2024-02-19T21:04:42.735575+0000","last_fullsized":"2024-02-19T21:04:42.735575+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.739154+0000","last_change":"2024-02-19T21:04:41.700191+0000","last_active":"2024-02-19T21:04:42.739154+0000","last_peered":"2024-02-19T21:04:42.739154+0000","last_clean":"2024-02-19T21:04:42.739154+0000","last_became_active":"2024-02-19T21:04:41.699762+0000","last_became_peered":"2024-02-19T21:04:41.699762+0000","last_unstale":"2024-02-19T21:04:42.739154+0000","last_undegraded":"2024-02-19T21:04:42.739154+0000","last_fullsized":"2024-02-19T21:04:42.739154+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"56.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.710914+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.e","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.806138+0000","last_change":"2024-02-19T21:04:43.805734+0000","last_active":"2024-02-19T21:04:43.806138+0000","last_peered":"2024-02-19T21:04:43.806138+0000","last_clean":"2024-02-19T21:04:43.806138+0000","last_became_active":"2024-02-19T21:04:43.805440+0000","last_became_peered":"2024-02-19T21:04:43.805440+0000","last_unstale":"2024-02-19T21:04:43.806138+0000","last_undegraded":"2024-02-19T21:04:43.806138+0000","last_fullsized":"2024-02-19T21:04:43.806138+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.763348+0000","last_change":"2024-02-19T21:04:42.743358+0000","last_active":"2024-02-19T21:04:43.763348+0000","last_peered":"2024-02-19T21:04:43.763348+0000","last_clean":"2024-02-19T21:04:43.763348+0000","last_became_active":"2024-02-19T21:04:42.742292+0000","last_became_peered":"2024-02-19T21:04:42.742292+0000","last_unstale":"2024-02-19T21:04:43.763348+0000","last_undegraded":"2024-02-19T21:04:43.763348+0000","last_fullsized":"2024-02-19T21:04:43.763348+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.697729+0000","last_change":"2024-02-19T21:04:41.701470+0000","last_active":"2024-02-19T21:04:42.697729+0000","last_peered":"2024-02-19T21:04:42.697729+0000","last_clean":"2024-02-19T21:04:42.697729+0000","last_became_active":"2024-02-19T21:04:41.701142+0000","last_became_peered":"2024-02-19T21:04:41.701142+0000","last_unstale":"2024-02-19T21:04:42.697729+0000","last_undegraded":"2024-02-19T21:04:42.697729+0000","last_fullsized":"2024-02-19T21:04:42.697729+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"56.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.717078+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.d","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.783747+0000","last_change":"2024-02-19T21:04:43.766717+0000","last_active":"2024-02-19T21:04:43.783747+0000","last_peered":"2024-02-19T21:04:43.783747+0000","last_clean":"2024-02-19T21:04:43.783747+0000","last_became_active":"2024-02-19T21:04:43.765743+0000","last_became_peered":"2024-02-19T21:04:43.765743+0000","last_unstale":"2024-02-19T21:04:43.783747+0000","last_undegraded":"2024-02-19T21:04:43.783747+0000","last_fullsized":"2024-02-19T21:04:43.783747+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.743483+0000","last_change":"2024-02-19T21:04:42.743055+0000","last_active":"2024-02-19T21:04:42.743483+0000","last_peered":"2024-02-19T21:04:42.743483+0000","last_clean":"2024-02-19T21:04:42.743483+0000","last_became_active":"2024-02-19T21:04:42.742816+0000","last_became_peered":"2024-02-19T21:04:42.742816+0000","last_unstale":"2024-02-19T21:04:42.743483+0000","last_undegraded":"2024-02-19T21:04:42.743483+0000","last_fullsized":"2024-02-19T21:04:42.743483+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.701733+0000","last_change":"2024-02-19T21:04:41.691068+0000","last_active":"2024-02-19T21:04:42.701733+0000","last_peered":"2024-02-19T21:04:42.701733+0000","last_clean":"2024-02-19T21:04:42.701733+0000","last_became_active":"2024-02-19T21:04:41.690796+0000","last_became_peered":"2024-02-19T21:04:41.690796+0000","last_unstale":"2024-02-19T21:04:42.701733+0000","last_undegraded":"2024-02-19T21:04:42.701733+0000","last_fullsized":"2024-02-19T21:04:42.701733+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"56.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.d","version":"0'0","reported_seq":2,"reported_epoch":105,"state":"creating+activating","last_fresh":"2024-02-19T21:04:44.700693+0000","last_change":"2024-02-19T21:04:44.700846+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:44.700693+0000","last_undegraded":"2024-02-19T21:04:44.700693+0000","last_fullsized":"2024-02-19T21:04:44.700693+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.c","version":"0'0","reported_seq":9,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.780765+0000","last_change":"2024-02-19T21:04:43.780765+0000","last_active":"2024-02-19T21:04:43.780765+0000","last_peered":"2024-02-19T21:04:43.780765+0000","last_clean":"2024-02-19T21:04:43.780765+0000","last_became_active":"2024-02-19T21:04:43.780142+0000","last_became_peered":"2024-02-19T21:04:43.780142+0000","last_unstale":"2024-02-19T21:04:43.780765+0000","last_undegraded":"2024-02-19T21:04:43.780765+0000","last_fullsized":"2024-02-19T21:04:43.780765+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.750563+0000","last_change":"2024-02-19T21:04:42.750352+0000","last_active":"2024-02-19T21:04:42.750563+0000","last_peered":"2024-02-19T21:04:42.750563+0000","last_clean":"2024-02-19T21:04:42.750563+0000","last_became_active":"2024-02-19T21:04:42.750004+0000","last_became_peered":"2024-02-19T21:04:42.750004+0000","last_unstale":"2024-02-19T21:04:42.750563+0000","last_undegraded":"2024-02-19T21:04:42.750563+0000","last_fullsized":"2024-02-19T21:04:42.750563+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.739988+0000","last_change":"2024-02-19T21:04:41.710633+0000","last_active":"2024-02-19T21:04:43.739988+0000","last_peered":"2024-02-19T21:04:43.739988+0000","last_clean":"2024-02-19T21:04:43.739988+0000","last_became_active":"2024-02-19T21:04:41.709922+0000","last_became_peered":"2024-02-19T21:04:41.709922+0000","last_unstale":"2024-02-19T21:04:43.739988+0000","last_undegraded":"2024-02-19T21:04:43.739988+0000","last_fullsized":"2024-02-19T21:04:43.739988+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"56.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.a","version":"0'0","reported_seq":3,"reported_epoch":105,"state":"creating+activating","last_fresh":"2024-02-19T21:04:44.698900+0000","last_change":"2024-02-19T21:04:44.698900+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:44.698900+0000","last_undegraded":"2024-02-19T21:04:44.698900+0000","last_fullsized":"2024-02-19T21:04:44.698900+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":["4","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.b","version":"0'0","reported_seq":9,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.774826+0000","last_change":"2024-02-19T21:04:43.774826+0000","last_active":"2024-02-19T21:04:43.774826+0000","last_peered":"2024-02-19T21:04:43.774826+0000","last_clean":"2024-02-19T21:04:43.774826+0000","last_became_active":"2024-02-19T21:04:43.774561+0000","last_became_peered":"2024-02-19T21:04:43.774561+0000","last_unstale":"2024-02-19T21:04:43.774826+0000","last_undegraded":"2024-02-19T21:04:43.774826+0000","last_fullsized":"2024-02-19T21:04:43.774826+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.741466+0000","last_change":"2024-02-19T21:04:42.740324+0000","last_active":"2024-02-19T21:04:42.741466+0000","last_peered":"2024-02-19T21:04:42.741466+0000","last_clean":"2024-02-19T21:04:42.741466+0000","last_became_active":"2024-02-19T21:04:42.740036+0000","last_became_peered":"2024-02-19T21:04:42.740036+0000","last_unstale":"2024-02-19T21:04:42.741466+0000","last_undegraded":"2024-02-19T21:04:42.741466+0000","last_fullsized":"2024-02-19T21:04:42.741466+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.757082+0000","last_change":"2024-02-19T21:04:41.711047+0000","last_active":"2024-02-19T21:04:43.757082+0000","last_peered":"2024-02-19T21:04:43.757082+0000","last_clean":"2024-02-19T21:04:43.757082+0000","last_became_active":"2024-02-19T21:04:41.710772+0000","last_became_peered":"2024-02-19T21:04:41.710772+0000","last_unstale":"2024-02-19T21:04:43.757082+0000","last_undegraded":"2024-02-19T21:04:43.757082+0000","last_fullsized":"2024-02-19T21:04:43.757082+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.b","version":"0'0","reported_seq":2,"reported_epoch":105,"state":"creating+activating","last_fresh":"2024-02-19T21:04:44.712497+0000","last_change":"2024-02-19T21:04:44.712664+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:44.712497+0000","last_undegraded":"2024-02-19T21:04:44.712497+0000","last_fullsized":"2024-02-19T21:04:44.712497+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.a","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.821413+0000","last_change":"2024-02-19T21:04:43.817730+0000","last_active":"2024-02-19T21:04:43.821413+0000","last_peered":"2024-02-19T21:04:43.821413+0000","last_clean":"2024-02-19T21:04:43.821413+0000","last_became_active":"2024-02-19T21:04:43.816636+0000","last_became_peered":"2024-02-19T21:04:43.816636+0000","last_unstale":"2024-02-19T21:04:43.821413+0000","last_undegraded":"2024-02-19T21:04:43.821413+0000","last_fullsized":"2024-02-19T21:04:43.821413+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.771691+0000","last_change":"2024-02-19T21:04:43.767857+0000","last_active":"2024-02-19T21:04:43.771691+0000","last_peered":"2024-02-19T21:04:43.771691+0000","last_clean":"2024-02-19T21:04:43.771691+0000","last_became_active":"2024-02-19T21:04:43.766045+0000","last_became_peered":"2024-02-19T21:04:43.766045+0000","last_unstale":"2024-02-19T21:04:43.771691+0000","last_undegraded":"2024-02-19T21:04:43.771691+0000","last_fullsized":"2024-02-19T21:04:43.771691+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.754623+0000","last_change":"2024-02-19T21:04:41.704802+0000","last_active":"2024-02-19T21:04:43.754623+0000","last_peered":"2024-02-19T21:04:43.754623+0000","last_clean":"2024-02-19T21:04:43.754623+0000","last_became_active":"2024-02-19T21:04:41.704528+0000","last_became_peered":"2024-02-19T21:04:41.704528+0000","last_unstale":"2024-02-19T21:04:43.754623+0000","last_undegraded":"2024-02-19T21:04:43.754623+0000","last_fullsized":"2024-02-19T21:04:43.754623+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"56.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:04:43.687697+0000","last_change":"2024-02-19T21:04:43.709312+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:43.687697+0000","last_undegraded":"2024-02-19T21:04:43.687697+0000","last_fullsized":"2024-02-19T21:04:43.687697+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.9","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.771710+0000","last_change":"2024-02-19T21:04:43.768246+0000","last_active":"2024-02-19T21:04:43.771710+0000","last_peered":"2024-02-19T21:04:43.771710+0000","last_clean":"2024-02-19T21:04:43.771710+0000","last_became_active":"2024-02-19T21:04:43.767415+0000","last_became_peered":"2024-02-19T21:04:43.767415+0000","last_unstale":"2024-02-19T21:04:43.771710+0000","last_undegraded":"2024-02-19T21:04:43.771710+0000","last_fullsized":"2024-02-19T21:04:43.771710+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.746229+0000","last_change":"2024-02-19T21:04:42.743865+0000","last_active":"2024-02-19T21:04:42.746229+0000","last_peered":"2024-02-19T21:04:42.746229+0000","last_clean":"2024-02-19T21:04:42.746229+0000","last_became_active":"2024-02-19T21:04:42.743560+0000","last_became_peered":"2024-02-19T21:04:42.743560+0000","last_unstale":"2024-02-19T21:04:42.746229+0000","last_undegraded":"2024-02-19T21:04:42.746229+0000","last_fullsized":"2024-02-19T21:04:42.746229+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.723111+0000","last_change":"2024-02-19T21:04:41.684101+0000","last_active":"2024-02-19T21:04:42.723111+0000","last_peered":"2024-02-19T21:04:42.723111+0000","last_clean":"2024-02-19T21:04:42.723111+0000","last_became_active":"2024-02-19T21:04:41.683673+0000","last_became_peered":"2024-02-19T21:04:41.683673+0000","last_unstale":"2024-02-19T21:04:42.723111+0000","last_undegraded":"2024-02-19T21:04:42.723111+0000","last_fullsized":"2024-02-19T21:04:42.723111+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"56.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"55.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"53.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:49.548372+0000","last_change":"2024-02-19T21:04:49.548372+0000","last_active":"2024-02-19T21:04:49.548372+0000","last_peered":"2024-02-19T21:04:49.548372+0000","last_clean":"2024-02-19T21:04:49.548372+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:49.548372+0000","last_undegraded":"2024-02-19T21:04:49.548372+0000","last_fullsized":"2024-02-19T21:04:49.548372+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"52.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:47.936107+0000","last_change":"2024-02-19T21:04:47.936107+0000","last_active":"2024-02-19T21:04:47.936107+0000","last_peered":"2024-02-19T21:04:47.936107+0000","last_clean":"2024-02-19T21:04:47.936107+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:47.936107+0000","last_undegraded":"2024-02-19T21:04:47.936107+0000","last_fullsized":"2024-02-19T21:04:47.936107+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"46.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:04:45.920915+0000","last_change":"2024-02-19T21:04:45.920915+0000","last_active":"2024-02-19T21:04:45.920915+0000","last_peered":"2024-02-19T21:04:45.920915+0000","last_clean":"2024-02-19T21:04:45.920915+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:45.920915+0000","last_undegraded":"2024-02-19T21:04:45.920915+0000","last_fullsized":"2024-02-19T21:04:45.920915+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"41.9","version":"0'0","reported_seq":2,"reported_epoch":105,"state":"creating+activating","last_fresh":"2024-02-19T21:04:44.707729+0000","last_change":"2024-02-19T21:04:44.707870+0000","last_active":"2024-02-19T21:04:43.687697+0000","last_peered":"2024-02-19T21:04:43.687697+0000","last_clean":"2024-02-19T21:04:43.687697+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:04:44.707729+0000","last_undegraded":"2024-02-19T21:04:44.707729+0000","last_fullsized":"2024-02-19T21:04:44.707729+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.687697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.687697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":["0","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.8","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.800973+0000","last_change":"2024-02-19T21:04:43.799675+0000","last_active":"2024-02-19T21:04:43.800973+0000","last_peered":"2024-02-19T21:04:43.800973+0000","last_clean":"2024-02-19T21:04:43.800973+0000","last_became_active":"2024-02-19T21:04:43.799408+0000","last_became_peered":"2024-02-19T21:04:43.799408+0000","last_unstale":"2024-02-19T21:04:43.800973+0000","last_undegraded":"2024-02-19T21:04:43.800973+0000","last_fullsized":"2024-02-19T21:04:43.800973+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.678142+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.678142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":10,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.750906+0000","last_change":"2024-02-19T21:04:42.750499+0000","last_active":"2024-02-19T21:04:42.750906+0000","last_peered":"2024-02-19T21:04:42.750906+0000","last_clean":"2024-02-19T21:04:42.750906+0000","last_became_active":"2024-02-19T21:04:42.748567+0000","last_became_peered":"2024-02-19T21:04:42.748567+0000","last_unstale":"2024-02-19T21:04:42.750906+0000","last_undegraded":"2024-02-19T21:04:42.750906+0000","last_fullsized":"2024-02-19T21:04:42.750906+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:41.616549+0000","last_clean_scrub_stamp":"2024-02-19T21:04:41.616549+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.746939+0000","last_change":"2024-02-19T21:04:41.709136+0000","last_active":"2024-02-19T21:04:43.746939+0000","last_peered":"2024-02-19T21:04:43.746939+0000","last_clean":"2024-02-19T21:04:43.746939+0000","last_became_active":"2024-02-19T21:04:41.708832+0000","last_became_peered":"2024-02-19T21:04:41.708832+0000","last_unstale":"2024-02-19T21:04:43.746939+0000","last_undegraded":"2024-02-19T21:04:43.746939+0000","last_fullsized":"2024-02-19T21:04:43.746939+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:40.521697+0000","last_clean_scrub_stamp":"2024-02-19T21:04:40.521697+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.10","version":"100'1","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.704236+0000","last_change":"2024-02-19T21:04:39.739732+0000","last_active":"2024-02-19T21:04:43.704236+0000","last_peered":"2024-02-19T21:04:43.704236+0000","last_clean":"2024-02-19T21:04:43.704236+0000","last_became_active":"2024-02-19T21:04:39.739003+0000","last_became_peered":"2024-02-19T21:04:39.739003+0000","last_unstale":"2024-02-19T21:04:43.704236+0000","last_undegraded":"2024-02-19T21:04:43.704236+0000","last_fullsized":"2024-02-19T21:04:43.704236+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.717053+0000","last_change":"2024-02-19T21:04:39.744209+0000","last_active":"2024-02-19T21:04:43.717053+0000","last_peered":"2024-02-19T21:04:43.717053+0000","last_clean":"2024-02-19T21:04:43.717053+0000","last_became_active":"2024-02-19T21:04:39.742909+0000","last_became_peered":"2024-02-19T21:04:39.742909+0000","last_unstale":"2024-02-19T21:04:43.717053+0000","last_undegraded":"2024-02-19T21:04:43.717053+0000","last_fullsized":"2024-02-19T21:04:43.717053+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.728977+0000","last_change":"2024-02-19T21:04:44.729142+0000","last_active":"2024-02-19T21:04:44.728977+0000","last_peered":"2024-02-19T21:04:44.728977+0000","last_clean":"2024-02-19T21:04:44.728977+0000","last_became_active":"2024-02-19T21:04:39.782022+0000","last_became_peered":"2024-02-19T21:04:39.782022+0000","last_unstale":"2024-02-19T21:04:44.728977+0000","last_undegraded":"2024-02-19T21:04:44.728977+0000","last_fullsized":"2024-02-19T21:04:44.728977+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.7","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.710930+0000","last_change":"2024-02-19T21:04:41.670180+0000","last_active":"2024-02-19T21:04:42.710930+0000","last_peered":"2024-02-19T21:04:42.710930+0000","last_clean":"2024-02-19T21:04:42.710930+0000","last_became_active":"2024-02-19T21:04:39.655026+0000","last_became_peered":"2024-02-19T21:04:39.655026+0000","last_unstale":"2024-02-19T21:04:42.710930+0000","last_undegraded":"2024-02-19T21:04:42.710930+0000","last_fullsized":"2024-02-19T21:04:42.710930+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.699390+0000","last_change":"2024-02-19T21:04:39.640053+0000","last_active":"2024-02-19T21:04:42.699390+0000","last_peered":"2024-02-19T21:04:42.699390+0000","last_clean":"2024-02-19T21:04:42.699390+0000","last_became_active":"2024-02-19T21:04:39.639772+0000","last_became_peered":"2024-02-19T21:04:39.639772+0000","last_unstale":"2024-02-19T21:04:42.699390+0000","last_undegraded":"2024-02-19T21:04:42.699390+0000","last_fullsized":"2024-02-19T21:04:42.699390+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"97'9","reported_seq":111,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.726194+0000","last_change":"2024-02-19T21:04:24.046111+0000","last_active":"2024-02-19T21:04:43.726194+0000","last_peered":"2024-02-19T21:04:43.726194+0000","last_clean":"2024-02-19T21:04:43.726194+0000","last_became_active":"2024-02-19T21:01:20.288720+0000","last_became_peered":"2024-02-19T21:01:20.288720+0000","last_unstale":"2024-02-19T21:04:43.726194+0000","last_undegraded":"2024-02-19T21:04:43.726194+0000","last_fullsized":"2024-02-19T21:04:43.726194+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:04:24.046034+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":86,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.705743+0000","last_change":"2024-02-19T21:04:24.218527+0000","last_active":"2024-02-19T21:04:42.705743+0000","last_peered":"2024-02-19T21:04:42.705743+0000","last_clean":"2024-02-19T21:04:42.705743+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:04:42.705743+0000","last_undegraded":"2024-02-19T21:04:42.705743+0000","last_fullsized":"2024-02-19T21:04:42.705743+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:03:01.005445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.005445+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":65,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.735743+0000","last_change":"2024-02-19T21:04:11.889869+0000","last_active":"2024-02-19T21:04:42.735743+0000","last_peered":"2024-02-19T21:04:42.735743+0000","last_clean":"2024-02-19T21:04:42.735743+0000","last_became_active":"2024-02-19T21:01:16.271911+0000","last_became_peered":"2024-02-19T21:01:16.271911+0000","last_unstale":"2024-02-19T21:04:42.735743+0000","last_undegraded":"2024-02-19T21:04:42.735743+0000","last_fullsized":"2024-02-19T21:04:42.735743+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.889778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:37.258580+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.889778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":54,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.704604+0000","last_change":"2024-02-19T21:03:10.017843+0000","last_active":"2024-02-19T21:04:42.704604+0000","last_peered":"2024-02-19T21:04:42.704604+0000","last_clean":"2024-02-19T21:04:42.704604+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:04:42.704604+0000","last_undegraded":"2024-02-19T21:04:42.704604+0000","last_fullsized":"2024-02-19T21:04:42.704604+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:10.017764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:10.017764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.727520+0000","last_change":"2024-02-19T21:04:39.703479+0000","last_active":"2024-02-19T21:04:42.727520+0000","last_peered":"2024-02-19T21:04:42.727520+0000","last_clean":"2024-02-19T21:04:42.727520+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:04:42.727520+0000","last_undegraded":"2024-02-19T21:04:42.727520+0000","last_fullsized":"2024-02-19T21:04:42.727520+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.691833+0000","last_change":"2024-02-19T21:04:39.776349+0000","last_active":"2024-02-19T21:04:42.691833+0000","last_peered":"2024-02-19T21:04:42.691833+0000","last_clean":"2024-02-19T21:04:42.691833+0000","last_became_active":"2024-02-19T21:04:39.775082+0000","last_became_peered":"2024-02-19T21:04:39.775082+0000","last_unstale":"2024-02-19T21:04:42.691833+0000","last_undegraded":"2024-02-19T21:04:42.691833+0000","last_fullsized":"2024-02-19T21:04:42.691833+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.13","version":"102'3","reported_seq":19,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.743814+0000","last_change":"2024-02-19T21:04:39.785733+0000","last_active":"2024-02-19T21:04:42.743814+0000","last_peered":"2024-02-19T21:04:42.743814+0000","last_clean":"2024-02-19T21:04:42.743814+0000","last_became_active":"2024-02-19T21:04:39.784009+0000","last_became_peered":"2024-02-19T21:04:39.784009+0000","last_unstale":"2024-02-19T21:04:42.743814+0000","last_undegraded":"2024-02-19T21:04:42.743814+0000","last_fullsized":"2024-02-19T21:04:42.743814+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165843,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"101'4","reported_seq":18,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.764416+0000","last_change":"2024-02-19T21:04:39.747515+0000","last_active":"2024-02-19T21:04:43.764416+0000","last_peered":"2024-02-19T21:04:43.764416+0000","last_clean":"2024-02-19T21:04:43.764416+0000","last_became_active":"2024-02-19T21:04:39.746314+0000","last_became_peered":"2024-02-19T21:04:39.746314+0000","last_unstale":"2024-02-19T21:04:43.764416+0000","last_undegraded":"2024-02-19T21:04:43.764416+0000","last_fullsized":"2024-02-19T21:04:43.764416+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.697499+0000","last_change":"2024-02-19T21:04:41.675583+0000","last_active":"2024-02-19T21:04:42.697499+0000","last_peered":"2024-02-19T21:04:42.697499+0000","last_clean":"2024-02-19T21:04:42.697499+0000","last_became_active":"2024-02-19T21:04:39.752451+0000","last_became_peered":"2024-02-19T21:04:39.752451+0000","last_unstale":"2024-02-19T21:04:42.697499+0000","last_undegraded":"2024-02-19T21:04:42.697499+0000","last_fullsized":"2024-02-19T21:04:42.697499+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.4","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719021+0000","last_change":"2024-02-19T21:04:41.651323+0000","last_active":"2024-02-19T21:04:42.719021+0000","last_peered":"2024-02-19T21:04:42.719021+0000","last_clean":"2024-02-19T21:04:42.719021+0000","last_became_active":"2024-02-19T21:04:39.688570+0000","last_became_peered":"2024-02-19T21:04:39.688570+0000","last_unstale":"2024-02-19T21:04:42.719021+0000","last_undegraded":"2024-02-19T21:04:42.719021+0000","last_fullsized":"2024-02-19T21:04:42.719021+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.7","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.698169+0000","last_change":"2024-02-19T21:04:39.772957+0000","last_active":"2024-02-19T21:04:42.698169+0000","last_peered":"2024-02-19T21:04:42.698169+0000","last_clean":"2024-02-19T21:04:42.698169+0000","last_became_active":"2024-02-19T21:04:39.771391+0000","last_became_peered":"2024-02-19T21:04:39.771391+0000","last_unstale":"2024-02-19T21:04:42.698169+0000","last_undegraded":"2024-02-19T21:04:42.698169+0000","last_fullsized":"2024-02-19T21:04:42.698169+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":75,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.769824+0000","last_change":"2024-02-19T21:02:52.643899+0000","last_active":"2024-02-19T21:04:43.769824+0000","last_peered":"2024-02-19T21:04:43.769824+0000","last_clean":"2024-02-19T21:04:43.769824+0000","last_became_active":"2024-02-19T21:01:18.288322+0000","last_became_peered":"2024-02-19T21:01:18.288322+0000","last_unstale":"2024-02-19T21:04:43.769824+0000","last_undegraded":"2024-02-19T21:04:43.769824+0000","last_fullsized":"2024-02-19T21:04:43.769824+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:02:52.643835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.643835+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.695532+0000","last_change":"2024-02-19T21:04:12.927016+0000","last_active":"2024-02-19T21:04:42.695532+0000","last_peered":"2024-02-19T21:04:42.695532+0000","last_clean":"2024-02-19T21:04:42.695532+0000","last_became_active":"2024-02-19T21:04:12.926645+0000","last_became_peered":"2024-02-19T21:04:12.926645+0000","last_unstale":"2024-02-19T21:04:42.695532+0000","last_undegraded":"2024-02-19T21:04:42.695532+0000","last_fullsized":"2024-02-19T21:04:42.695532+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:30.462868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:30.462868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":61,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.725016+0000","last_change":"2024-02-19T21:04:17.831164+0000","last_active":"2024-02-19T21:04:42.725016+0000","last_peered":"2024-02-19T21:04:42.725016+0000","last_clean":"2024-02-19T21:04:42.725016+0000","last_became_active":"2024-02-19T21:01:20.290566+0000","last_became_peered":"2024-02-19T21:01:20.290566+0000","last_unstale":"2024-02-19T21:04:42.725016+0000","last_undegraded":"2024-02-19T21:04:42.725016+0000","last_fullsized":"2024-02-19T21:04:42.725016+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:17.831065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:17.831065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":60,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.765511+0000","last_change":"2024-02-19T21:04:21.659690+0000","last_active":"2024-02-19T21:04:43.765511+0000","last_peered":"2024-02-19T21:04:43.765511+0000","last_clean":"2024-02-19T21:04:43.765511+0000","last_became_active":"2024-02-19T21:01:22.297998+0000","last_became_peered":"2024-02-19T21:01:22.297998+0000","last_unstale":"2024-02-19T21:04:43.765511+0000","last_undegraded":"2024-02-19T21:04:43.765511+0000","last_fullsized":"2024-02-19T21:04:43.765511+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:21.659616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.659616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.710818+0000","last_change":"2024-02-19T21:04:39.765776+0000","last_active":"2024-02-19T21:04:42.710818+0000","last_peered":"2024-02-19T21:04:42.710818+0000","last_clean":"2024-02-19T21:04:42.710818+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:04:42.710818+0000","last_undegraded":"2024-02-19T21:04:42.710818+0000","last_fullsized":"2024-02-19T21:04:42.710818+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.695156+0000","last_change":"2024-02-19T21:04:39.635405+0000","last_active":"2024-02-19T21:04:42.695156+0000","last_peered":"2024-02-19T21:04:42.695156+0000","last_clean":"2024-02-19T21:04:42.695156+0000","last_became_active":"2024-02-19T21:04:39.635124+0000","last_became_peered":"2024-02-19T21:04:39.635124+0000","last_unstale":"2024-02-19T21:04:42.695156+0000","last_undegraded":"2024-02-19T21:04:42.695156+0000","last_fullsized":"2024-02-19T21:04:42.695156+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.12","version":"101'2","reported_seq":18,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.757293+0000","last_change":"2024-02-19T21:04:39.785340+0000","last_active":"2024-02-19T21:04:43.757293+0000","last_peered":"2024-02-19T21:04:43.757293+0000","last_clean":"2024-02-19T21:04:43.757293+0000","last_became_active":"2024-02-19T21:04:39.784930+0000","last_became_peered":"2024-02-19T21:04:39.784930+0000","last_unstale":"2024-02-19T21:04:43.757293+0000","last_undegraded":"2024-02-19T21:04:43.757293+0000","last_fullsized":"2024-02-19T21:04:43.757293+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.718774+0000","last_change":"2024-02-19T21:04:39.691020+0000","last_active":"2024-02-19T21:04:42.718774+0000","last_peered":"2024-02-19T21:04:42.718774+0000","last_clean":"2024-02-19T21:04:42.718774+0000","last_became_active":"2024-02-19T21:04:39.687895+0000","last_became_peered":"2024-02-19T21:04:39.687895+0000","last_unstale":"2024-02-19T21:04:42.718774+0000","last_undegraded":"2024-02-19T21:04:42.718774+0000","last_fullsized":"2024-02-19T21:04:42.718774+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.726455+0000","last_change":"2024-02-19T21:04:44.733089+0000","last_active":"2024-02-19T21:04:44.726455+0000","last_peered":"2024-02-19T21:04:44.726455+0000","last_clean":"2024-02-19T21:04:44.726455+0000","last_became_active":"2024-02-19T21:04:39.720682+0000","last_became_peered":"2024-02-19T21:04:39.720682+0000","last_unstale":"2024-02-19T21:04:44.726455+0000","last_undegraded":"2024-02-19T21:04:44.726455+0000","last_fullsized":"2024-02-19T21:04:44.726455+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.5","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.720328+0000","last_change":"2024-02-19T21:04:41.649076+0000","last_active":"2024-02-19T21:04:42.720328+0000","last_peered":"2024-02-19T21:04:42.720328+0000","last_clean":"2024-02-19T21:04:42.720328+0000","last_became_active":"2024-02-19T21:04:39.692019+0000","last_became_peered":"2024-02-19T21:04:39.692019+0000","last_unstale":"2024-02-19T21:04:42.720328+0000","last_undegraded":"2024-02-19T21:04:42.720328+0000","last_fullsized":"2024-02-19T21:04:42.720328+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.6","version":"100'9","reported_seq":24,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.788350+0000","last_change":"2024-02-19T21:04:39.731654+0000","last_active":"2024-02-19T21:04:43.788350+0000","last_peered":"2024-02-19T21:04:43.788350+0000","last_clean":"2024-02-19T21:04:43.788350+0000","last_became_active":"2024-02-19T21:04:39.731439+0000","last_became_peered":"2024-02-19T21:04:39.731439+0000","last_unstale":"2024-02-19T21:04:43.788350+0000","last_undegraded":"2024-02-19T21:04:43.788350+0000","last_fullsized":"2024-02-19T21:04:43.788350+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":94,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719301+0000","last_change":"2024-02-19T21:04:20.836693+0000","last_active":"2024-02-19T21:04:42.719301+0000","last_peered":"2024-02-19T21:04:42.719301+0000","last_clean":"2024-02-19T21:04:42.719301+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:04:42.719301+0000","last_undegraded":"2024-02-19T21:04:42.719301+0000","last_fullsized":"2024-02-19T21:04:42.719301+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:04:20.836609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:20.836609+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":65,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.727080+0000","last_change":"2024-02-19T21:04:03.767826+0000","last_active":"2024-02-19T21:04:42.727080+0000","last_peered":"2024-02-19T21:04:42.727080+0000","last_clean":"2024-02-19T21:04:42.727080+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:04:42.727080+0000","last_undegraded":"2024-02-19T21:04:42.727080+0000","last_fullsized":"2024-02-19T21:04:42.727080+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:03.767713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:03.767713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":56,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.743402+0000","last_change":"2024-02-19T21:02:55.213761+0000","last_active":"2024-02-19T21:04:42.743402+0000","last_peered":"2024-02-19T21:04:42.743402+0000","last_clean":"2024-02-19T21:04:42.743402+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:04:42.743402+0000","last_undegraded":"2024-02-19T21:04:42.743402+0000","last_fullsized":"2024-02-19T21:04:42.743402+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:55.213692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:55.213692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":53,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.697519+0000","last_change":"2024-02-19T21:03:03.500049+0000","last_active":"2024-02-19T21:04:42.697519+0000","last_peered":"2024-02-19T21:04:42.697519+0000","last_clean":"2024-02-19T21:04:42.697519+0000","last_became_active":"2024-02-19T21:01:22.291759+0000","last_became_peered":"2024-02-19T21:01:22.291759+0000","last_unstale":"2024-02-19T21:04:42.697519+0000","last_undegraded":"2024-02-19T21:04:42.697519+0000","last_fullsized":"2024-02-19T21:04:42.697519+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:03.499960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:03.499960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.718610+0000","last_change":"2024-02-19T21:04:39.620073+0000","last_active":"2024-02-19T21:04:42.718610+0000","last_peered":"2024-02-19T21:04:42.718610+0000","last_clean":"2024-02-19T21:04:42.718610+0000","last_became_active":"2024-02-19T21:04:39.618614+0000","last_became_peered":"2024-02-19T21:04:39.618614+0000","last_unstale":"2024-02-19T21:04:42.718610+0000","last_undegraded":"2024-02-19T21:04:42.718610+0000","last_fullsized":"2024-02-19T21:04:42.718610+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.732822+0000","last_change":"2024-02-19T21:04:39.735486+0000","last_active":"2024-02-19T21:04:42.732822+0000","last_peered":"2024-02-19T21:04:42.732822+0000","last_clean":"2024-02-19T21:04:42.732822+0000","last_became_active":"2024-02-19T21:04:39.734973+0000","last_became_peered":"2024-02-19T21:04:39.734973+0000","last_unstale":"2024-02-19T21:04:42.732822+0000","last_undegraded":"2024-02-19T21:04:42.732822+0000","last_fullsized":"2024-02-19T21:04:42.732822+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.695838+0000","last_change":"2024-02-19T21:04:39.690722+0000","last_active":"2024-02-19T21:04:42.695838+0000","last_peered":"2024-02-19T21:04:42.695838+0000","last_clean":"2024-02-19T21:04:42.695838+0000","last_became_active":"2024-02-19T21:04:39.689794+0000","last_became_peered":"2024-02-19T21:04:39.689794+0000","last_unstale":"2024-02-19T21:04:42.695838+0000","last_undegraded":"2024-02-19T21:04:42.695838+0000","last_fullsized":"2024-02-19T21:04:42.695838+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.712138+0000","last_change":"2024-02-19T21:04:39.764245+0000","last_active":"2024-02-19T21:04:43.712138+0000","last_peered":"2024-02-19T21:04:43.712138+0000","last_clean":"2024-02-19T21:04:43.712138+0000","last_became_active":"2024-02-19T21:04:39.763684+0000","last_became_peered":"2024-02-19T21:04:39.763684+0000","last_unstale":"2024-02-19T21:04:43.712138+0000","last_undegraded":"2024-02-19T21:04:43.712138+0000","last_fullsized":"2024-02-19T21:04:43.712138+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.705839+0000","last_change":"2024-02-19T21:04:41.658357+0000","last_active":"2024-02-19T21:04:42.705839+0000","last_peered":"2024-02-19T21:04:42.705839+0000","last_clean":"2024-02-19T21:04:42.705839+0000","last_became_active":"2024-02-19T21:04:39.772112+0000","last_became_peered":"2024-02-19T21:04:39.772112+0000","last_unstale":"2024-02-19T21:04:42.705839+0000","last_undegraded":"2024-02-19T21:04:42.705839+0000","last_fullsized":"2024-02-19T21:04:42.705839+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.2","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.725418+0000","last_change":"2024-02-19T21:04:41.635769+0000","last_active":"2024-02-19T21:04:42.725418+0000","last_peered":"2024-02-19T21:04:42.725418+0000","last_clean":"2024-02-19T21:04:42.725418+0000","last_became_active":"2024-02-19T21:04:39.793603+0000","last_became_peered":"2024-02-19T21:04:39.793603+0000","last_unstale":"2024-02-19T21:04:42.725418+0000","last_undegraded":"2024-02-19T21:04:42.725418+0000","last_fullsized":"2024-02-19T21:04:42.725418+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"4.d","version":"65'17","reported_seq":119,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.720678+0000","last_change":"2024-02-19T21:04:03.152903+0000","last_active":"2024-02-19T21:04:43.720678+0000","last_peered":"2024-02-19T21:04:43.720678+0000","last_clean":"2024-02-19T21:04:43.720678+0000","last_became_active":"2024-02-19T21:01:30.923214+0000","last_became_peered":"2024-02-19T21:01:30.923214+0000","last_unstale":"2024-02-19T21:04:43.720678+0000","last_undegraded":"2024-02-19T21:04:43.720678+0000","last_fullsized":"2024-02-19T21:04:43.720678+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:04:03.152817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:03.152817+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":70,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.712590+0000","last_change":"2024-02-19T21:04:21.952701+0000","last_active":"2024-02-19T21:04:42.712590+0000","last_peered":"2024-02-19T21:04:42.712590+0000","last_clean":"2024-02-19T21:04:42.712590+0000","last_became_active":"2024-02-19T21:01:16.274937+0000","last_became_peered":"2024-02-19T21:01:16.274937+0000","last_unstale":"2024-02-19T21:04:42.712590+0000","last_undegraded":"2024-02-19T21:04:42.712590+0000","last_fullsized":"2024-02-19T21:04:42.712590+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:04:21.952586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.952586+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":61,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.693682+0000","last_change":"2024-02-19T21:04:04.386811+0000","last_active":"2024-02-19T21:04:42.693682+0000","last_peered":"2024-02-19T21:04:42.693682+0000","last_clean":"2024-02-19T21:04:42.693682+0000","last_became_active":"2024-02-19T21:01:20.290687+0000","last_became_peered":"2024-02-19T21:01:20.290687+0000","last_unstale":"2024-02-19T21:04:42.693682+0000","last_undegraded":"2024-02-19T21:04:42.693682+0000","last_fullsized":"2024-02-19T21:04:42.693682+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:04.386735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:04.386735+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":59,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.740192+0000","last_change":"2024-02-19T21:04:19.845711+0000","last_active":"2024-02-19T21:04:42.740192+0000","last_peered":"2024-02-19T21:04:42.740192+0000","last_clean":"2024-02-19T21:04:42.740192+0000","last_became_active":"2024-02-19T21:01:22.298941+0000","last_became_peered":"2024-02-19T21:01:22.298941+0000","last_unstale":"2024-02-19T21:04:42.740192+0000","last_undegraded":"2024-02-19T21:04:42.740192+0000","last_fullsized":"2024-02-19T21:04:42.740192+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.845609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"100'5","reported_seq":19,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.691129+0000","last_change":"2024-02-19T21:04:39.781921+0000","last_active":"2024-02-19T21:04:42.691129+0000","last_peered":"2024-02-19T21:04:42.691129+0000","last_clean":"2024-02-19T21:04:42.691129+0000","last_became_active":"2024-02-19T21:04:39.778460+0000","last_became_peered":"2024-02-19T21:04:39.778460+0000","last_unstale":"2024-02-19T21:04:42.691129+0000","last_undegraded":"2024-02-19T21:04:42.691129+0000","last_fullsized":"2024-02-19T21:04:42.691129+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.702354+0000","last_change":"2024-02-19T21:04:39.699479+0000","last_active":"2024-02-19T21:04:43.702354+0000","last_peered":"2024-02-19T21:04:43.702354+0000","last_clean":"2024-02-19T21:04:43.702354+0000","last_became_active":"2024-02-19T21:04:39.698684+0000","last_became_peered":"2024-02-19T21:04:39.698684+0000","last_unstale":"2024-02-19T21:04:43.702354+0000","last_undegraded":"2024-02-19T21:04:43.702354+0000","last_fullsized":"2024-02-19T21:04:43.702354+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.794986+0000","last_change":"2024-02-19T21:04:39.800621+0000","last_active":"2024-02-19T21:04:43.794986+0000","last_peered":"2024-02-19T21:04:43.794986+0000","last_clean":"2024-02-19T21:04:43.794986+0000","last_became_active":"2024-02-19T21:04:39.799889+0000","last_became_peered":"2024-02-19T21:04:39.799889+0000","last_unstale":"2024-02-19T21:04:43.794986+0000","last_undegraded":"2024-02-19T21:04:43.794986+0000","last_fullsized":"2024-02-19T21:04:43.794986+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.14","version":"100'6","reported_seq":21,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.759244+0000","last_change":"2024-02-19T21:04:39.757025+0000","last_active":"2024-02-19T21:04:43.759244+0000","last_peered":"2024-02-19T21:04:43.759244+0000","last_clean":"2024-02-19T21:04:43.759244+0000","last_became_active":"2024-02-19T21:04:39.755747+0000","last_became_peered":"2024-02-19T21:04:39.755747+0000","last_unstale":"2024-02-19T21:04:43.759244+0000","last_undegraded":"2024-02-19T21:04:43.759244+0000","last_fullsized":"2024-02-19T21:04:43.759244+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1024,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.712443+0000","last_change":"2024-02-19T21:04:39.763797+0000","last_active":"2024-02-19T21:04:43.712443+0000","last_peered":"2024-02-19T21:04:43.712443+0000","last_clean":"2024-02-19T21:04:43.712443+0000","last_became_active":"2024-02-19T21:04:39.763493+0000","last_became_peered":"2024-02-19T21:04:39.763493+0000","last_unstale":"2024-02-19T21:04:43.712443+0000","last_undegraded":"2024-02-19T21:04:43.712443+0000","last_fullsized":"2024-02-19T21:04:43.712443+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.716049+0000","last_change":"2024-02-19T21:04:44.716401+0000","last_active":"2024-02-19T21:04:44.716049+0000","last_peered":"2024-02-19T21:04:44.716049+0000","last_clean":"2024-02-19T21:04:44.716049+0000","last_became_active":"2024-02-19T21:04:39.749954+0000","last_became_peered":"2024-02-19T21:04:39.749954+0000","last_unstale":"2024-02-19T21:04:44.716049+0000","last_undegraded":"2024-02-19T21:04:44.716049+0000","last_fullsized":"2024-02-19T21:04:44.716049+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.3","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.714817+0000","last_change":"2024-02-19T21:04:41.691809+0000","last_active":"2024-02-19T21:04:42.714817+0000","last_peered":"2024-02-19T21:04:42.714817+0000","last_clean":"2024-02-19T21:04:42.714817+0000","last_became_active":"2024-02-19T21:04:39.656255+0000","last_became_peered":"2024-02-19T21:04:39.656255+0000","last_unstale":"2024-02-19T21:04:42.714817+0000","last_undegraded":"2024-02-19T21:04:42.714817+0000","last_fullsized":"2024-02-19T21:04:42.714817+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"4.c","version":"82'15","reported_seq":79,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.718668+0000","last_change":"2024-02-19T21:03:04.086368+0000","last_active":"2024-02-19T21:04:43.718668+0000","last_peered":"2024-02-19T21:04:43.718668+0000","last_clean":"2024-02-19T21:04:43.718668+0000","last_became_active":"2024-02-19T21:01:18.282805+0000","last_became_peered":"2024-02-19T21:01:18.282805+0000","last_unstale":"2024-02-19T21:04:43.718668+0000","last_undegraded":"2024-02-19T21:04:43.718668+0000","last_fullsized":"2024-02-19T21:04:43.718668+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.086282+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":65,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.692465+0000","last_change":"2024-02-19T21:04:11.303670+0000","last_active":"2024-02-19T21:04:42.692465+0000","last_peered":"2024-02-19T21:04:42.692465+0000","last_clean":"2024-02-19T21:04:42.692465+0000","last_became_active":"2024-02-19T21:01:16.274086+0000","last_became_peered":"2024-02-19T21:01:16.274086+0000","last_unstale":"2024-02-19T21:04:42.692465+0000","last_undegraded":"2024-02-19T21:04:42.692465+0000","last_fullsized":"2024-02-19T21:04:42.692465+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.303572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.303572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":111,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.704685+0000","last_change":"2024-02-19T21:04:07.809065+0000","last_active":"2024-02-19T21:04:43.704685+0000","last_peered":"2024-02-19T21:04:43.704685+0000","last_clean":"2024-02-19T21:04:43.704685+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:04:43.704685+0000","last_undegraded":"2024-02-19T21:04:43.704685+0000","last_fullsized":"2024-02-19T21:04:43.704685+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:07.808944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:07.808944+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":60,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.715215+0000","last_change":"2024-02-19T21:04:21.031646+0000","last_active":"2024-02-19T21:04:43.715215+0000","last_peered":"2024-02-19T21:04:43.715215+0000","last_clean":"2024-02-19T21:04:43.715215+0000","last_became_active":"2024-02-19T21:01:22.299993+0000","last_became_peered":"2024-02-19T21:01:22.299993+0000","last_unstale":"2024-02-19T21:04:43.715215+0000","last_undegraded":"2024-02-19T21:04:43.715215+0000","last_fullsized":"2024-02-19T21:04:43.715215+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.031577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"100'5","reported_seq":20,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.749334+0000","last_change":"2024-02-19T21:04:39.746708+0000","last_active":"2024-02-19T21:04:43.749334+0000","last_peered":"2024-02-19T21:04:43.749334+0000","last_clean":"2024-02-19T21:04:43.749334+0000","last_became_active":"2024-02-19T21:04:39.742728+0000","last_became_peered":"2024-02-19T21:04:39.742728+0000","last_unstale":"2024-02-19T21:04:43.749334+0000","last_undegraded":"2024-02-19T21:04:43.749334+0000","last_fullsized":"2024-02-19T21:04:43.749334+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.794040+0000","last_change":"2024-02-19T21:04:39.766043+0000","last_active":"2024-02-19T21:04:43.794040+0000","last_peered":"2024-02-19T21:04:43.794040+0000","last_clean":"2024-02-19T21:04:43.794040+0000","last_became_active":"2024-02-19T21:04:39.765782+0000","last_became_peered":"2024-02-19T21:04:39.765782+0000","last_unstale":"2024-02-19T21:04:43.794040+0000","last_undegraded":"2024-02-19T21:04:43.794040+0000","last_fullsized":"2024-02-19T21:04:43.794040+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.724630+0000","last_change":"2024-02-19T21:04:39.764255+0000","last_active":"2024-02-19T21:04:43.724630+0000","last_peered":"2024-02-19T21:04:43.724630+0000","last_clean":"2024-02-19T21:04:43.724630+0000","last_became_active":"2024-02-19T21:04:39.763962+0000","last_became_peered":"2024-02-19T21:04:39.763962+0000","last_unstale":"2024-02-19T21:04:43.724630+0000","last_undegraded":"2024-02-19T21:04:43.724630+0000","last_fullsized":"2024-02-19T21:04:43.724630+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.17","version":"102'7","reported_seq":28,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.734126+0000","last_change":"2024-02-19T21:04:39.659021+0000","last_active":"2024-02-19T21:04:42.734126+0000","last_peered":"2024-02-19T21:04:42.734126+0000","last_clean":"2024-02-19T21:04:42.734126+0000","last_became_active":"2024-02-19T21:04:39.658450+0000","last_became_peered":"2024-02-19T21:04:39.658450+0000","last_unstale":"2024-02-19T21:04:42.734126+0000","last_undegraded":"2024-02-19T21:04:42.734126+0000","last_fullsized":"2024-02-19T21:04:42.734126+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165900,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.11","version":"100'1","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.741369+0000","last_change":"2024-02-19T21:04:39.674675+0000","last_active":"2024-02-19T21:04:43.741369+0000","last_peered":"2024-02-19T21:04:43.741369+0000","last_clean":"2024-02-19T21:04:43.741369+0000","last_became_active":"2024-02-19T21:04:39.674446+0000","last_became_peered":"2024-02-19T21:04:39.674446+0000","last_unstale":"2024-02-19T21:04:43.741369+0000","last_undegraded":"2024-02-19T21:04:43.741369+0000","last_fullsized":"2024-02-19T21:04:43.741369+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.692010+0000","last_change":"2024-02-19T21:04:41.662506+0000","last_active":"2024-02-19T21:04:42.692010+0000","last_peered":"2024-02-19T21:04:42.692010+0000","last_clean":"2024-02-19T21:04:42.692010+0000","last_became_active":"2024-02-19T21:04:39.682417+0000","last_became_peered":"2024-02-19T21:04:39.682417+0000","last_unstale":"2024-02-19T21:04:42.692010+0000","last_undegraded":"2024-02-19T21:04:42.692010+0000","last_fullsized":"2024-02-19T21:04:42.692010+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.0","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.694097+0000","last_change":"2024-02-19T21:04:41.647957+0000","last_active":"2024-02-19T21:04:42.694097+0000","last_peered":"2024-02-19T21:04:42.694097+0000","last_clean":"2024-02-19T21:04:42.694097+0000","last_became_active":"2024-02-19T21:04:39.746860+0000","last_became_peered":"2024-02-19T21:04:39.746860+0000","last_unstale":"2024-02-19T21:04:42.694097+0000","last_undegraded":"2024-02-19T21:04:42.694097+0000","last_fullsized":"2024-02-19T21:04:42.694097+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.3","version":"100'5","reported_seq":19,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.691749+0000","last_change":"2024-02-19T21:04:39.676464+0000","last_active":"2024-02-19T21:04:42.691749+0000","last_peered":"2024-02-19T21:04:42.691749+0000","last_clean":"2024-02-19T21:04:42.691749+0000","last_became_active":"2024-02-19T21:04:39.675580+0000","last_became_peered":"2024-02-19T21:04:39.675580+0000","last_unstale":"2024-02-19T21:04:42.691749+0000","last_undegraded":"2024-02-19T21:04:42.691749+0000","last_fullsized":"2024-02-19T21:04:42.691749+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":83,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.698152+0000","last_change":"2024-02-19T21:02:53.526211+0000","last_active":"2024-02-19T21:04:42.698152+0000","last_peered":"2024-02-19T21:04:42.698152+0000","last_clean":"2024-02-19T21:04:42.698152+0000","last_became_active":"2024-02-19T21:01:18.285534+0000","last_became_peered":"2024-02-19T21:01:18.285534+0000","last_unstale":"2024-02-19T21:04:42.698152+0000","last_undegraded":"2024-02-19T21:04:42.698152+0000","last_fullsized":"2024-02-19T21:04:42.698152+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:53.526141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:53.526141+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":65,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.691439+0000","last_change":"2024-02-19T21:04:12.305976+0000","last_active":"2024-02-19T21:04:42.691439+0000","last_peered":"2024-02-19T21:04:42.691439+0000","last_clean":"2024-02-19T21:04:42.691439+0000","last_became_active":"2024-02-19T21:01:16.274939+0000","last_became_peered":"2024-02-19T21:01:16.274939+0000","last_unstale":"2024-02-19T21:04:42.691439+0000","last_undegraded":"2024-02-19T21:04:42.691439+0000","last_fullsized":"2024-02-19T21:04:42.691439+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:12.305886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:12.305886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":112,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.459138+0000","last_change":"2024-02-19T21:04:16.059715+0000","last_active":"2024-02-19T21:04:44.459138+0000","last_peered":"2024-02-19T21:04:44.459138+0000","last_clean":"2024-02-19T21:04:44.459138+0000","last_became_active":"2024-02-19T21:01:20.293341+0000","last_became_peered":"2024-02-19T21:01:20.293341+0000","last_unstale":"2024-02-19T21:04:44.459138+0000","last_undegraded":"2024-02-19T21:04:44.459138+0000","last_fullsized":"2024-02-19T21:04:44.459138+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:16.059648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.059648+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":59,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.710377+0000","last_change":"2024-02-19T21:04:08.185818+0000","last_active":"2024-02-19T21:04:42.710377+0000","last_peered":"2024-02-19T21:04:42.710377+0000","last_clean":"2024-02-19T21:04:42.710377+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:04:42.710377+0000","last_undegraded":"2024-02-19T21:04:42.710377+0000","last_fullsized":"2024-02-19T21:04:42.710377+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.185717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.185717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.704068+0000","last_change":"2024-02-19T21:04:39.657326+0000","last_active":"2024-02-19T21:04:42.704068+0000","last_peered":"2024-02-19T21:04:42.704068+0000","last_clean":"2024-02-19T21:04:42.704068+0000","last_became_active":"2024-02-19T21:04:39.657052+0000","last_became_peered":"2024-02-19T21:04:39.657052+0000","last_unstale":"2024-02-19T21:04:42.704068+0000","last_undegraded":"2024-02-19T21:04:42.704068+0000","last_fullsized":"2024-02-19T21:04:42.704068+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.727686+0000","last_change":"2024-02-19T21:04:39.674008+0000","last_active":"2024-02-19T21:04:42.727686+0000","last_peered":"2024-02-19T21:04:42.727686+0000","last_clean":"2024-02-19T21:04:42.727686+0000","last_became_active":"2024-02-19T21:04:39.672497+0000","last_became_peered":"2024-02-19T21:04:39.672497+0000","last_unstale":"2024-02-19T21:04:42.727686+0000","last_undegraded":"2024-02-19T21:04:42.727686+0000","last_fullsized":"2024-02-19T21:04:42.727686+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.788986+0000","last_change":"2024-02-19T21:04:39.800157+0000","last_active":"2024-02-19T21:04:43.788986+0000","last_peered":"2024-02-19T21:04:43.788986+0000","last_clean":"2024-02-19T21:04:43.788986+0000","last_became_active":"2024-02-19T21:04:39.798967+0000","last_became_peered":"2024-02-19T21:04:39.798967+0000","last_unstale":"2024-02-19T21:04:43.788986+0000","last_undegraded":"2024-02-19T21:04:43.788986+0000","last_fullsized":"2024-02-19T21:04:43.788986+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.720310+0000","last_change":"2024-02-19T21:04:39.781493+0000","last_active":"2024-02-19T21:04:42.720310+0000","last_peered":"2024-02-19T21:04:42.720310+0000","last_clean":"2024-02-19T21:04:42.720310+0000","last_became_active":"2024-02-19T21:04:39.780767+0000","last_became_peered":"2024-02-19T21:04:39.780767+0000","last_unstale":"2024-02-19T21:04:42.720310+0000","last_undegraded":"2024-02-19T21:04:42.720310+0000","last_fullsized":"2024-02-19T21:04:42.720310+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.713312+0000","last_change":"2024-02-19T21:04:44.714728+0000","last_active":"2024-02-19T21:04:44.713312+0000","last_peered":"2024-02-19T21:04:44.713312+0000","last_clean":"2024-02-19T21:04:44.713312+0000","last_became_active":"2024-02-19T21:04:39.741733+0000","last_became_peered":"2024-02-19T21:04:39.741733+0000","last_unstale":"2024-02-19T21:04:44.713312+0000","last_undegraded":"2024-02-19T21:04:44.713312+0000","last_fullsized":"2024-02-19T21:04:44.713312+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.1","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.705102+0000","last_change":"2024-02-19T21:04:41.655311+0000","last_active":"2024-02-19T21:04:42.705102+0000","last_peered":"2024-02-19T21:04:42.705102+0000","last_clean":"2024-02-19T21:04:42.705102+0000","last_became_active":"2024-02-19T21:04:39.666823+0000","last_became_peered":"2024-02-19T21:04:39.666823+0000","last_unstale":"2024-02-19T21:04:42.705102+0000","last_undegraded":"2024-02-19T21:04:42.705102+0000","last_fullsized":"2024-02-19T21:04:42.705102+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.2","version":"0'0","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.712654+0000","last_change":"2024-02-19T21:04:39.662708+0000","last_active":"2024-02-19T21:04:43.712654+0000","last_peered":"2024-02-19T21:04:43.712654+0000","last_clean":"2024-02-19T21:04:43.712654+0000","last_became_active":"2024-02-19T21:04:39.662412+0000","last_became_peered":"2024-02-19T21:04:39.662412+0000","last_unstale":"2024-02-19T21:04:43.712654+0000","last_undegraded":"2024-02-19T21:04:43.712654+0000","last_fullsized":"2024-02-19T21:04:43.712654+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":83,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.714491+0000","last_change":"2024-02-19T21:03:35.086667+0000","last_active":"2024-02-19T21:04:43.714491+0000","last_peered":"2024-02-19T21:04:43.714491+0000","last_clean":"2024-02-19T21:04:43.714491+0000","last_became_active":"2024-02-19T21:01:18.285670+0000","last_became_peered":"2024-02-19T21:01:18.285670+0000","last_unstale":"2024-02-19T21:04:43.714491+0000","last_undegraded":"2024-02-19T21:04:43.714491+0000","last_fullsized":"2024-02-19T21:04:43.714491+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:03:35.086579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:35.086579+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":64,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.698100+0000","last_change":"2024-02-19T21:04:08.858046+0000","last_active":"2024-02-19T21:04:42.698100+0000","last_peered":"2024-02-19T21:04:42.698100+0000","last_clean":"2024-02-19T21:04:42.698100+0000","last_became_active":"2024-02-19T21:01:16.275520+0000","last_became_peered":"2024-02-19T21:01:16.275520+0000","last_unstale":"2024-02-19T21:04:42.698100+0000","last_undegraded":"2024-02-19T21:04:42.698100+0000","last_fullsized":"2024-02-19T21:04:42.698100+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.857973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.857973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":61,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.711400+0000","last_change":"2024-02-19T21:04:16.217810+0000","last_active":"2024-02-19T21:04:42.711400+0000","last_peered":"2024-02-19T21:04:42.711400+0000","last_clean":"2024-02-19T21:04:42.711400+0000","last_became_active":"2024-02-19T21:01:20.289375+0000","last_became_peered":"2024-02-19T21:01:20.289375+0000","last_unstale":"2024-02-19T21:04:42.711400+0000","last_undegraded":"2024-02-19T21:04:42.711400+0000","last_fullsized":"2024-02-19T21:04:42.711400+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:16.217744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.217744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.753632+0000","last_change":"2024-02-19T21:03:23.656785+0000","last_active":"2024-02-19T21:04:43.753632+0000","last_peered":"2024-02-19T21:04:43.753632+0000","last_clean":"2024-02-19T21:04:43.753632+0000","last_became_active":"2024-02-19T21:01:22.607007+0000","last_became_peered":"2024-02-19T21:01:22.607007+0000","last_unstale":"2024-02-19T21:04:43.753632+0000","last_undegraded":"2024-02-19T21:04:43.753632+0000","last_fullsized":"2024-02-19T21:04:43.753632+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:23.656688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:23.656688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.742826+0000","last_change":"2024-02-19T21:04:39.744505+0000","last_active":"2024-02-19T21:04:43.742826+0000","last_peered":"2024-02-19T21:04:43.742826+0000","last_clean":"2024-02-19T21:04:43.742826+0000","last_became_active":"2024-02-19T21:04:39.744231+0000","last_became_peered":"2024-02-19T21:04:39.744231+0000","last_unstale":"2024-02-19T21:04:43.742826+0000","last_undegraded":"2024-02-19T21:04:43.742826+0000","last_fullsized":"2024-02-19T21:04:43.742826+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.710453+0000","last_change":"2024-02-19T21:04:39.750580+0000","last_active":"2024-02-19T21:04:42.710453+0000","last_peered":"2024-02-19T21:04:42.710453+0000","last_clean":"2024-02-19T21:04:42.710453+0000","last_became_active":"2024-02-19T21:04:39.750293+0000","last_became_peered":"2024-02-19T21:04:39.750293+0000","last_unstale":"2024-02-19T21:04:42.710453+0000","last_undegraded":"2024-02-19T21:04:42.710453+0000","last_fullsized":"2024-02-19T21:04:42.710453+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.700685+0000","last_change":"2024-02-19T21:04:39.640944+0000","last_active":"2024-02-19T21:04:42.700685+0000","last_peered":"2024-02-19T21:04:42.700685+0000","last_clean":"2024-02-19T21:04:42.700685+0000","last_became_active":"2024-02-19T21:04:39.640667+0000","last_became_peered":"2024-02-19T21:04:39.640667+0000","last_unstale":"2024-02-19T21:04:42.700685+0000","last_undegraded":"2024-02-19T21:04:42.700685+0000","last_fullsized":"2024-02-19T21:04:42.700685+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.720598+0000","last_change":"2024-02-19T21:04:39.681382+0000","last_active":"2024-02-19T21:04:42.720598+0000","last_peered":"2024-02-19T21:04:42.720598+0000","last_clean":"2024-02-19T21:04:42.720598+0000","last_became_active":"2024-02-19T21:04:39.680986+0000","last_became_peered":"2024-02-19T21:04:39.680986+0000","last_unstale":"2024-02-19T21:04:42.720598+0000","last_undegraded":"2024-02-19T21:04:42.720598+0000","last_fullsized":"2024-02-19T21:04:42.720598+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.709043+0000","last_change":"2024-02-19T21:04:44.709932+0000","last_active":"2024-02-19T21:04:44.709043+0000","last_peered":"2024-02-19T21:04:44.709043+0000","last_clean":"2024-02-19T21:04:44.709043+0000","last_became_active":"2024-02-19T21:04:39.763428+0000","last_became_peered":"2024-02-19T21:04:39.763428+0000","last_unstale":"2024-02-19T21:04:44.709043+0000","last_undegraded":"2024-02-19T21:04:44.709043+0000","last_fullsized":"2024-02-19T21:04:44.709043+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.e","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.711276+0000","last_change":"2024-02-19T21:04:44.713056+0000","last_active":"2024-02-19T21:04:44.711276+0000","last_peered":"2024-02-19T21:04:44.711276+0000","last_clean":"2024-02-19T21:04:44.711276+0000","last_became_active":"2024-02-19T21:04:39.640368+0000","last_became_peered":"2024-02-19T21:04:39.640368+0000","last_unstale":"2024-02-19T21:04:44.711276+0000","last_undegraded":"2024-02-19T21:04:44.711276+0000","last_fullsized":"2024-02-19T21:04:44.711276+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"8.d","version":"100'5","reported_seq":19,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.709446+0000","last_change":"2024-02-19T21:04:39.631816+0000","last_active":"2024-02-19T21:04:42.709446+0000","last_peered":"2024-02-19T21:04:42.709446+0000","last_clean":"2024-02-19T21:04:42.709446+0000","last_became_active":"2024-02-19T21:04:39.631449+0000","last_became_peered":"2024-02-19T21:04:39.631449+0000","last_unstale":"2024-02-19T21:04:42.709446+0000","last_undegraded":"2024-02-19T21:04:42.709446+0000","last_fullsized":"2024-02-19T21:04:42.709446+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":112,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.459117+0000","last_change":"2024-02-19T21:03:33.736454+0000","last_active":"2024-02-19T21:04:44.459117+0000","last_peered":"2024-02-19T21:04:44.459117+0000","last_clean":"2024-02-19T21:04:44.459117+0000","last_became_active":"2024-02-19T21:01:20.286505+0000","last_became_peered":"2024-02-19T21:01:20.286505+0000","last_unstale":"2024-02-19T21:04:44.459117+0000","last_undegraded":"2024-02-19T21:04:44.459117+0000","last_fullsized":"2024-02-19T21:04:44.459117+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:33.736366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:33.736366+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":80,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.735564+0000","last_change":"2024-02-19T21:02:40.105227+0000","last_active":"2024-02-19T21:04:43.735564+0000","last_peered":"2024-02-19T21:04:43.735564+0000","last_clean":"2024-02-19T21:04:43.735564+0000","last_became_active":"2024-02-19T21:01:18.282702+0000","last_became_peered":"2024-02-19T21:01:18.282702+0000","last_unstale":"2024-02-19T21:04:43.735564+0000","last_undegraded":"2024-02-19T21:04:43.735564+0000","last_fullsized":"2024-02-19T21:04:43.735564+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:02:40.105141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:40.105141+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":65,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.720559+0000","last_change":"2024-02-19T21:04:22.800646+0000","last_active":"2024-02-19T21:04:42.720559+0000","last_peered":"2024-02-19T21:04:42.720559+0000","last_clean":"2024-02-19T21:04:42.720559+0000","last_became_active":"2024-02-19T21:01:16.274462+0000","last_became_peered":"2024-02-19T21:01:16.274462+0000","last_unstale":"2024-02-19T21:04:42.720559+0000","last_undegraded":"2024-02-19T21:04:42.720559+0000","last_fullsized":"2024-02-19T21:04:42.720559+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:22.800578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:22.800578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":53,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.716257+0000","last_change":"2024-02-19T21:03:01.496926+0000","last_active":"2024-02-19T21:04:42.716257+0000","last_peered":"2024-02-19T21:04:42.716257+0000","last_clean":"2024-02-19T21:04:42.716257+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:04:42.716257+0000","last_undegraded":"2024-02-19T21:04:42.716257+0000","last_fullsized":"2024-02-19T21:04:42.716257+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.496833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.496833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.794023+0000","last_change":"2024-02-19T21:04:39.659690+0000","last_active":"2024-02-19T21:04:43.794023+0000","last_peered":"2024-02-19T21:04:43.794023+0000","last_clean":"2024-02-19T21:04:43.794023+0000","last_became_active":"2024-02-19T21:04:39.658536+0000","last_became_peered":"2024-02-19T21:04:39.658536+0000","last_unstale":"2024-02-19T21:04:43.794023+0000","last_undegraded":"2024-02-19T21:04:43.794023+0000","last_fullsized":"2024-02-19T21:04:43.794023+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.746230+0000","last_change":"2024-02-19T21:04:39.753154+0000","last_active":"2024-02-19T21:04:43.746230+0000","last_peered":"2024-02-19T21:04:43.746230+0000","last_clean":"2024-02-19T21:04:43.746230+0000","last_became_active":"2024-02-19T21:04:39.745540+0000","last_became_peered":"2024-02-19T21:04:39.745540+0000","last_unstale":"2024-02-19T21:04:43.746230+0000","last_undegraded":"2024-02-19T21:04:43.746230+0000","last_fullsized":"2024-02-19T21:04:43.746230+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.18","version":"102'4","reported_seq":22,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.750246+0000","last_change":"2024-02-19T21:04:39.748257+0000","last_active":"2024-02-19T21:04:43.750246+0000","last_peered":"2024-02-19T21:04:43.750246+0000","last_clean":"2024-02-19T21:04:43.750246+0000","last_became_active":"2024-02-19T21:04:39.747589+0000","last_became_peered":"2024-02-19T21:04:39.747589+0000","last_unstale":"2024-02-19T21:04:43.750246+0000","last_undegraded":"2024-02-19T21:04:43.750246+0000","last_fullsized":"2024-02-19T21:04:43.750246+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50331664,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":4,"num_write_kb":51902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.710478+0000","last_change":"2024-02-19T21:04:39.647683+0000","last_active":"2024-02-19T21:04:42.710478+0000","last_peered":"2024-02-19T21:04:42.710478+0000","last_clean":"2024-02-19T21:04:42.710478+0000","last_became_active":"2024-02-19T21:04:39.646950+0000","last_became_peered":"2024-02-19T21:04:39.646950+0000","last_unstale":"2024-02-19T21:04:42.710478+0000","last_undegraded":"2024-02-19T21:04:42.710478+0000","last_fullsized":"2024-02-19T21:04:42.710478+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"102'1","reported_seq":18,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.709295+0000","last_change":"2024-02-19T21:04:41.660081+0000","last_active":"2024-02-19T21:04:42.709295+0000","last_peered":"2024-02-19T21:04:42.709295+0000","last_clean":"2024-02-19T21:04:42.709295+0000","last_became_active":"2024-02-19T21:04:39.656342+0000","last_became_peered":"2024-02-19T21:04:39.656342+0000","last_unstale":"2024-02-19T21:04:42.709295+0000","last_undegraded":"2024-02-19T21:04:42.709295+0000","last_fullsized":"2024-02-19T21:04:42.709295+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.f","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.733270+0000","last_change":"2024-02-19T21:04:41.651673+0000","last_active":"2024-02-19T21:04:42.733270+0000","last_peered":"2024-02-19T21:04:42.733270+0000","last_clean":"2024-02-19T21:04:42.733270+0000","last_became_active":"2024-02-19T21:04:39.657430+0000","last_became_peered":"2024-02-19T21:04:39.657430+0000","last_unstale":"2024-02-19T21:04:42.733270+0000","last_undegraded":"2024-02-19T21:04:42.733270+0000","last_fullsized":"2024-02-19T21:04:42.733270+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.c","version":"0'0","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.769278+0000","last_change":"2024-02-19T21:04:39.769987+0000","last_active":"2024-02-19T21:04:43.769278+0000","last_peered":"2024-02-19T21:04:43.769278+0000","last_clean":"2024-02-19T21:04:43.769278+0000","last_became_active":"2024-02-19T21:04:39.767371+0000","last_became_peered":"2024-02-19T21:04:39.767371+0000","last_unstale":"2024-02-19T21:04:43.769278+0000","last_undegraded":"2024-02-19T21:04:43.769278+0000","last_fullsized":"2024-02-19T21:04:43.769278+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":57,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.735730+0000","last_change":"2024-02-19T21:02:57.997535+0000","last_active":"2024-02-19T21:04:43.735730+0000","last_peered":"2024-02-19T21:04:43.735730+0000","last_clean":"2024-02-19T21:04:43.735730+0000","last_became_active":"2024-02-19T21:01:20.291079+0000","last_became_peered":"2024-02-19T21:01:20.291079+0000","last_unstale":"2024-02-19T21:04:43.735730+0000","last_undegraded":"2024-02-19T21:04:43.735730+0000","last_fullsized":"2024-02-19T21:04:43.735730+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.997427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":96,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.753853+0000","last_change":"2024-02-19T21:04:08.837761+0000","last_active":"2024-02-19T21:04:43.753853+0000","last_peered":"2024-02-19T21:04:43.753853+0000","last_clean":"2024-02-19T21:04:43.753853+0000","last_became_active":"2024-02-19T21:01:18.293812+0000","last_became_peered":"2024-02-19T21:01:18.293812+0000","last_unstale":"2024-02-19T21:04:43.753853+0000","last_undegraded":"2024-02-19T21:04:43.753853+0000","last_fullsized":"2024-02-19T21:04:43.753853+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:04:08.837652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.837652+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":65,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.696321+0000","last_change":"2024-02-19T21:04:21.205943+0000","last_active":"2024-02-19T21:04:42.696321+0000","last_peered":"2024-02-19T21:04:42.696321+0000","last_clean":"2024-02-19T21:04:42.696321+0000","last_became_active":"2024-02-19T21:01:16.271340+0000","last_became_peered":"2024-02-19T21:01:16.271340+0000","last_unstale":"2024-02-19T21:04:42.696321+0000","last_undegraded":"2024-02-19T21:04:42.696321+0000","last_fullsized":"2024-02-19T21:04:42.696321+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:21.205829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.205829+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.714104+0000","last_change":"2024-02-19T21:03:17.088777+0000","last_active":"2024-02-19T21:04:43.714104+0000","last_peered":"2024-02-19T21:04:43.714104+0000","last_clean":"2024-02-19T21:04:43.714104+0000","last_became_active":"2024-02-19T21:01:22.297707+0000","last_became_peered":"2024-02-19T21:01:22.297707+0000","last_unstale":"2024-02-19T21:04:43.714104+0000","last_undegraded":"2024-02-19T21:04:43.714104+0000","last_fullsized":"2024-02-19T21:04:43.714104+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:17.088695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:17.088695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.769349+0000","last_change":"2024-02-19T21:04:39.770138+0000","last_active":"2024-02-19T21:04:43.769349+0000","last_peered":"2024-02-19T21:04:43.769349+0000","last_clean":"2024-02-19T21:04:43.769349+0000","last_became_active":"2024-02-19T21:04:39.767723+0000","last_became_peered":"2024-02-19T21:04:39.767723+0000","last_unstale":"2024-02-19T21:04:43.769349+0000","last_undegraded":"2024-02-19T21:04:43.769349+0000","last_fullsized":"2024-02-19T21:04:43.769349+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.768599+0000","last_change":"2024-02-19T21:04:39.781680+0000","last_active":"2024-02-19T21:04:43.768599+0000","last_peered":"2024-02-19T21:04:43.768599+0000","last_clean":"2024-02-19T21:04:43.768599+0000","last_became_active":"2024-02-19T21:04:39.779955+0000","last_became_peered":"2024-02-19T21:04:39.779955+0000","last_unstale":"2024-02-19T21:04:43.768599+0000","last_undegraded":"2024-02-19T21:04:43.768599+0000","last_fullsized":"2024-02-19T21:04:43.768599+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"100'1","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.743333+0000","last_change":"2024-02-19T21:04:39.786049+0000","last_active":"2024-02-19T21:04:42.743333+0000","last_peered":"2024-02-19T21:04:42.743333+0000","last_clean":"2024-02-19T21:04:42.743333+0000","last_became_active":"2024-02-19T21:04:39.784742+0000","last_became_peered":"2024-02-19T21:04:39.784742+0000","last_unstale":"2024-02-19T21:04:42.743333+0000","last_undegraded":"2024-02-19T21:04:42.743333+0000","last_fullsized":"2024-02-19T21:04:42.743333+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.738677+0000","last_change":"2024-02-19T21:04:39.790336+0000","last_active":"2024-02-19T21:04:42.738677+0000","last_peered":"2024-02-19T21:04:42.738677+0000","last_clean":"2024-02-19T21:04:42.738677+0000","last_became_active":"2024-02-19T21:04:39.789356+0000","last_became_peered":"2024-02-19T21:04:39.789356+0000","last_unstale":"2024-02-19T21:04:42.738677+0000","last_undegraded":"2024-02-19T21:04:42.738677+0000","last_fullsized":"2024-02-19T21:04:42.738677+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.723152+0000","last_change":"2024-02-19T21:04:44.723328+0000","last_active":"2024-02-19T21:04:44.723152+0000","last_peered":"2024-02-19T21:04:44.723152+0000","last_clean":"2024-02-19T21:04:44.723152+0000","last_became_active":"2024-02-19T21:04:39.747808+0000","last_became_peered":"2024-02-19T21:04:39.747808+0000","last_unstale":"2024-02-19T21:04:44.723152+0000","last_undegraded":"2024-02-19T21:04:44.723152+0000","last_fullsized":"2024-02-19T21:04:44.723152+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.c","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.694046+0000","last_change":"2024-02-19T21:04:41.675024+0000","last_active":"2024-02-19T21:04:42.694046+0000","last_peered":"2024-02-19T21:04:42.694046+0000","last_clean":"2024-02-19T21:04:42.694046+0000","last_became_active":"2024-02-19T21:04:39.799979+0000","last_became_peered":"2024-02-19T21:04:39.799979+0000","last_unstale":"2024-02-19T21:04:42.694046+0000","last_undegraded":"2024-02-19T21:04:42.694046+0000","last_fullsized":"2024-02-19T21:04:42.694046+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.f","version":"0'0","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.787045+0000","last_change":"2024-02-19T21:04:39.722429+0000","last_active":"2024-02-19T21:04:43.787045+0000","last_peered":"2024-02-19T21:04:43.787045+0000","last_clean":"2024-02-19T21:04:43.787045+0000","last_became_active":"2024-02-19T21:04:39.711377+0000","last_became_peered":"2024-02-19T21:04:39.711377+0000","last_unstale":"2024-02-19T21:04:43.787045+0000","last_undegraded":"2024-02-19T21:04:43.787045+0000","last_fullsized":"2024-02-19T21:04:43.787045+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":61,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719461+0000","last_change":"2024-02-19T21:03:36.004677+0000","last_active":"2024-02-19T21:04:42.719461+0000","last_peered":"2024-02-19T21:04:42.719461+0000","last_clean":"2024-02-19T21:04:42.719461+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:04:42.719461+0000","last_undegraded":"2024-02-19T21:04:42.719461+0000","last_fullsized":"2024-02-19T21:04:42.719461+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:36.004580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:36.004580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":100,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.769686+0000","last_change":"2024-02-19T21:04:07.592349+0000","last_active":"2024-02-19T21:04:43.769686+0000","last_peered":"2024-02-19T21:04:43.769686+0000","last_clean":"2024-02-19T21:04:43.769686+0000","last_became_active":"2024-02-19T21:01:18.293791+0000","last_became_peered":"2024-02-19T21:01:18.293791+0000","last_unstale":"2024-02-19T21:04:43.769686+0000","last_undegraded":"2024-02-19T21:04:43.769686+0000","last_fullsized":"2024-02-19T21:04:43.769686+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:04:07.592209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:07.592209+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":59,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.720836+0000","last_change":"2024-02-19T21:02:46.565447+0000","last_active":"2024-02-19T21:04:42.720836+0000","last_peered":"2024-02-19T21:04:42.720836+0000","last_clean":"2024-02-19T21:04:42.720836+0000","last_became_active":"2024-02-19T21:01:16.274215+0000","last_became_peered":"2024-02-19T21:01:16.274215+0000","last_unstale":"2024-02-19T21:04:42.720836+0000","last_undegraded":"2024-02-19T21:04:42.720836+0000","last_fullsized":"2024-02-19T21:04:42.720836+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:46.565380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:46.565380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":54,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.691573+0000","last_change":"2024-02-19T21:03:07.473156+0000","last_active":"2024-02-19T21:04:42.691573+0000","last_peered":"2024-02-19T21:04:42.691573+0000","last_clean":"2024-02-19T21:04:42.691573+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:04:42.691573+0000","last_undegraded":"2024-02-19T21:04:42.691573+0000","last_fullsized":"2024-02-19T21:04:42.691573+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:03:07.473070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.740739+0000","last_change":"2024-02-19T21:04:39.719113+0000","last_active":"2024-02-19T21:04:42.740739+0000","last_peered":"2024-02-19T21:04:42.740739+0000","last_clean":"2024-02-19T21:04:42.740739+0000","last_became_active":"2024-02-19T21:04:39.718926+0000","last_became_peered":"2024-02-19T21:04:39.718926+0000","last_unstale":"2024-02-19T21:04:42.740739+0000","last_undegraded":"2024-02-19T21:04:42.740739+0000","last_fullsized":"2024-02-19T21:04:42.740739+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.695028+0000","last_change":"2024-02-19T21:04:39.783494+0000","last_active":"2024-02-19T21:04:42.695028+0000","last_peered":"2024-02-19T21:04:42.695028+0000","last_clean":"2024-02-19T21:04:42.695028+0000","last_became_active":"2024-02-19T21:04:39.777458+0000","last_became_peered":"2024-02-19T21:04:39.777458+0000","last_unstale":"2024-02-19T21:04:42.695028+0000","last_undegraded":"2024-02-19T21:04:42.695028+0000","last_fullsized":"2024-02-19T21:04:42.695028+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1e","version":"100'1","reported_seq":16,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.717156+0000","last_change":"2024-02-19T21:04:39.765598+0000","last_active":"2024-02-19T21:04:43.717156+0000","last_peered":"2024-02-19T21:04:43.717156+0000","last_clean":"2024-02-19T21:04:43.717156+0000","last_became_active":"2024-02-19T21:04:39.765165+0000","last_became_peered":"2024-02-19T21:04:39.765165+0000","last_unstale":"2024-02-19T21:04:43.717156+0000","last_undegraded":"2024-02-19T21:04:43.717156+0000","last_fullsized":"2024-02-19T21:04:43.717156+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.704398+0000","last_change":"2024-02-19T21:04:39.739577+0000","last_active":"2024-02-19T21:04:43.704398+0000","last_peered":"2024-02-19T21:04:43.704398+0000","last_clean":"2024-02-19T21:04:43.704398+0000","last_became_active":"2024-02-19T21:04:39.738827+0000","last_became_peered":"2024-02-19T21:04:39.738827+0000","last_unstale":"2024-02-19T21:04:43.704398+0000","last_undegraded":"2024-02-19T21:04:43.704398+0000","last_fullsized":"2024-02-19T21:04:43.704398+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.733342+0000","last_change":"2024-02-19T21:04:44.735019+0000","last_active":"2024-02-19T21:04:44.733342+0000","last_peered":"2024-02-19T21:04:44.733342+0000","last_clean":"2024-02-19T21:04:44.733342+0000","last_became_active":"2024-02-19T21:04:39.739319+0000","last_became_peered":"2024-02-19T21:04:39.739319+0000","last_unstale":"2024-02-19T21:04:44.733342+0000","last_undegraded":"2024-02-19T21:04:44.733342+0000","last_fullsized":"2024-02-19T21:04:44.733342+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.9","version":"102'1","reported_seq":23,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.719348+0000","last_change":"2024-02-19T21:04:44.722894+0000","last_active":"2024-02-19T21:04:44.719348+0000","last_peered":"2024-02-19T21:04:44.719348+0000","last_clean":"2024-02-19T21:04:44.719348+0000","last_became_active":"2024-02-19T21:04:39.671354+0000","last_became_peered":"2024-02-19T21:04:39.671354+0000","last_unstale":"2024-02-19T21:04:44.719348+0000","last_undegraded":"2024-02-19T21:04:44.719348+0000","last_fullsized":"2024-02-19T21:04:44.719348+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"8.a","version":"100'10","reported_seq":24,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.691891+0000","last_change":"2024-02-19T21:04:39.794115+0000","last_active":"2024-02-19T21:04:42.691891+0000","last_peered":"2024-02-19T21:04:42.691891+0000","last_clean":"2024-02-19T21:04:42.691891+0000","last_became_active":"2024-02-19T21:04:39.792916+0000","last_became_peered":"2024-02-19T21:04:39.792916+0000","last_unstale":"2024-02-19T21:04:42.691891+0000","last_undegraded":"2024-02-19T21:04:42.691891+0000","last_fullsized":"2024-02-19T21:04:42.691891+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":56,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.711980+0000","last_change":"2024-02-19T21:03:05.005360+0000","last_active":"2024-02-19T21:04:42.711980+0000","last_peered":"2024-02-19T21:04:42.711980+0000","last_clean":"2024-02-19T21:04:42.711980+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:04:42.711980+0000","last_undegraded":"2024-02-19T21:04:42.711980+0000","last_fullsized":"2024-02-19T21:04:42.711980+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.005288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.005288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":82,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.788514+0000","last_change":"2024-02-19T21:03:22.495828+0000","last_active":"2024-02-19T21:04:43.788514+0000","last_peered":"2024-02-19T21:04:43.788514+0000","last_clean":"2024-02-19T21:04:43.788514+0000","last_became_active":"2024-02-19T21:01:18.287139+0000","last_became_peered":"2024-02-19T21:01:18.287139+0000","last_unstale":"2024-02-19T21:04:43.788514+0000","last_undegraded":"2024-02-19T21:04:43.788514+0000","last_fullsized":"2024-02-19T21:04:43.788514+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:03:22.495754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:22.495754+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":66,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.697605+0000","last_change":"2024-02-19T21:02:41.616831+0000","last_active":"2024-02-19T21:04:42.697605+0000","last_peered":"2024-02-19T21:04:42.697605+0000","last_clean":"2024-02-19T21:04:42.697605+0000","last_became_active":"2024-02-19T21:01:16.275904+0000","last_became_peered":"2024-02-19T21:01:16.275904+0000","last_unstale":"2024-02-19T21:04:42.697605+0000","last_undegraded":"2024-02-19T21:04:42.697605+0000","last_fullsized":"2024-02-19T21:04:42.697605+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:02:41.616729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:41.616729+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":61,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.763063+0000","last_change":"2024-02-19T21:02:57.505093+0000","last_active":"2024-02-19T21:04:43.763063+0000","last_peered":"2024-02-19T21:04:43.763063+0000","last_clean":"2024-02-19T21:04:43.763063+0000","last_became_active":"2024-02-19T21:01:16.274398+0000","last_became_peered":"2024-02-19T21:01:16.274398+0000","last_unstale":"2024-02-19T21:04:43.763063+0000","last_undegraded":"2024-02-19T21:04:43.763063+0000","last_fullsized":"2024-02-19T21:04:43.763063+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.505029+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":59,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.693767+0000","last_change":"2024-02-19T21:04:22.168700+0000","last_active":"2024-02-19T21:04:42.693767+0000","last_peered":"2024-02-19T21:04:42.693767+0000","last_clean":"2024-02-19T21:04:42.693767+0000","last_became_active":"2024-02-19T21:01:22.298477+0000","last_became_peered":"2024-02-19T21:01:22.298477+0000","last_unstale":"2024-02-19T21:04:42.693767+0000","last_undegraded":"2024-02-19T21:04:42.693767+0000","last_fullsized":"2024-02-19T21:04:42.693767+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:22.168596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:22.168596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.714097+0000","last_change":"2024-02-19T21:04:39.665830+0000","last_active":"2024-02-19T21:04:42.714097+0000","last_peered":"2024-02-19T21:04:42.714097+0000","last_clean":"2024-02-19T21:04:42.714097+0000","last_became_active":"2024-02-19T21:04:39.662125+0000","last_became_peered":"2024-02-19T21:04:39.662125+0000","last_unstale":"2024-02-19T21:04:42.714097+0000","last_undegraded":"2024-02-19T21:04:42.714097+0000","last_fullsized":"2024-02-19T21:04:42.714097+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.725332+0000","last_change":"2024-02-19T21:04:39.678391+0000","last_active":"2024-02-19T21:04:43.725332+0000","last_peered":"2024-02-19T21:04:43.725332+0000","last_clean":"2024-02-19T21:04:43.725332+0000","last_became_active":"2024-02-19T21:04:39.678031+0000","last_became_peered":"2024-02-19T21:04:39.678031+0000","last_unstale":"2024-02-19T21:04:43.725332+0000","last_undegraded":"2024-02-19T21:04:43.725332+0000","last_fullsized":"2024-02-19T21:04:43.725332+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.710658+0000","last_change":"2024-02-19T21:04:39.656608+0000","last_active":"2024-02-19T21:04:42.710658+0000","last_peered":"2024-02-19T21:04:42.710658+0000","last_clean":"2024-02-19T21:04:42.710658+0000","last_became_active":"2024-02-19T21:04:39.655971+0000","last_became_peered":"2024-02-19T21:04:39.655971+0000","last_unstale":"2024-02-19T21:04:42.710658+0000","last_undegraded":"2024-02-19T21:04:42.710658+0000","last_fullsized":"2024-02-19T21:04:42.710658+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1b","version":"102'3","reported_seq":18,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.714737+0000","last_change":"2024-02-19T21:04:39.674256+0000","last_active":"2024-02-19T21:04:42.714737+0000","last_peered":"2024-02-19T21:04:42.714737+0000","last_clean":"2024-02-19T21:04:42.714737+0000","last_became_active":"2024-02-19T21:04:39.662877+0000","last_became_peered":"2024-02-19T21:04:39.662877+0000","last_unstale":"2024-02-19T21:04:42.714737+0000","last_undegraded":"2024-02-19T21:04:42.714737+0000","last_fullsized":"2024-02-19T21:04:42.714737+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.704109+0000","last_change":"2024-02-19T21:04:44.708504+0000","last_active":"2024-02-19T21:04:44.704109+0000","last_peered":"2024-02-19T21:04:44.704109+0000","last_clean":"2024-02-19T21:04:44.704109+0000","last_became_active":"2024-02-19T21:04:39.726696+0000","last_became_peered":"2024-02-19T21:04:39.726696+0000","last_unstale":"2024-02-19T21:04:44.704109+0000","last_undegraded":"2024-02-19T21:04:44.704109+0000","last_fullsized":"2024-02-19T21:04:44.704109+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.a","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.722147+0000","last_change":"2024-02-19T21:04:44.722735+0000","last_active":"2024-02-19T21:04:44.722147+0000","last_peered":"2024-02-19T21:04:44.722147+0000","last_clean":"2024-02-19T21:04:44.722147+0000","last_became_active":"2024-02-19T21:04:39.741461+0000","last_became_peered":"2024-02-19T21:04:39.741461+0000","last_unstale":"2024-02-19T21:04:44.722147+0000","last_undegraded":"2024-02-19T21:04:44.722147+0000","last_fullsized":"2024-02-19T21:04:44.722147+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.706013+0000","last_change":"2024-02-19T21:04:39.633236+0000","last_active":"2024-02-19T21:04:42.706013+0000","last_peered":"2024-02-19T21:04:42.706013+0000","last_clean":"2024-02-19T21:04:42.706013+0000","last_became_active":"2024-02-19T21:04:39.632936+0000","last_became_peered":"2024-02-19T21:04:39.632936+0000","last_unstale":"2024-02-19T21:04:42.706013+0000","last_undegraded":"2024-02-19T21:04:42.706013+0000","last_fullsized":"2024-02-19T21:04:42.706013+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":60,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719378+0000","last_change":"2024-02-19T21:04:19.858793+0000","last_active":"2024-02-19T21:04:42.719378+0000","last_peered":"2024-02-19T21:04:42.719378+0000","last_clean":"2024-02-19T21:04:42.719378+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:04:42.719378+0000","last_undegraded":"2024-02-19T21:04:42.719378+0000","last_fullsized":"2024-02-19T21:04:42.719378+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:19.858688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.858688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":85,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.718728+0000","last_change":"2024-02-19T21:04:19.885862+0000","last_active":"2024-02-19T21:04:42.718728+0000","last_peered":"2024-02-19T21:04:42.718728+0000","last_clean":"2024-02-19T21:04:42.718728+0000","last_became_active":"2024-02-19T21:01:18.293784+0000","last_became_peered":"2024-02-19T21:01:18.293784+0000","last_unstale":"2024-02-19T21:04:42.718728+0000","last_undegraded":"2024-02-19T21:04:42.718728+0000","last_fullsized":"2024-02-19T21:04:42.718728+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:04:19.885797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.885797+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":68,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.697955+0000","last_change":"2024-02-19T21:03:11.598713+0000","last_active":"2024-02-19T21:04:42.697955+0000","last_peered":"2024-02-19T21:04:42.697955+0000","last_clean":"2024-02-19T21:04:42.697955+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:04:42.697955+0000","last_undegraded":"2024-02-19T21:04:42.697955+0000","last_fullsized":"2024-02-19T21:04:42.697955+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:11.598603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:03:11.598603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":65,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.703776+0000","last_change":"2024-02-19T21:04:14.215108+0000","last_active":"2024-02-19T21:04:42.703776+0000","last_peered":"2024-02-19T21:04:42.703776+0000","last_clean":"2024-02-19T21:04:42.703776+0000","last_became_active":"2024-02-19T21:01:16.276807+0000","last_became_peered":"2024-02-19T21:01:16.276807+0000","last_unstale":"2024-02-19T21:04:42.703776+0000","last_undegraded":"2024-02-19T21:04:42.703776+0000","last_fullsized":"2024-02-19T21:04:42.703776+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:14.215016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:14.215016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":61,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.746950+0000","last_change":"2024-02-19T21:03:19.665710+0000","last_active":"2024-02-19T21:04:43.746950+0000","last_peered":"2024-02-19T21:04:43.746950+0000","last_clean":"2024-02-19T21:04:43.746950+0000","last_became_active":"2024-02-19T21:01:16.274290+0000","last_became_peered":"2024-02-19T21:01:16.274290+0000","last_unstale":"2024-02-19T21:04:43.746950+0000","last_undegraded":"2024-02-19T21:04:43.746950+0000","last_fullsized":"2024-02-19T21:04:43.746950+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:19.665646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:19.665646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":59,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.712064+0000","last_change":"2024-02-19T21:04:05.136254+0000","last_active":"2024-02-19T21:04:42.712064+0000","last_peered":"2024-02-19T21:04:42.712064+0000","last_clean":"2024-02-19T21:04:42.712064+0000","last_became_active":"2024-02-19T21:01:22.292299+0000","last_became_peered":"2024-02-19T21:01:22.292299+0000","last_unstale":"2024-02-19T21:04:42.712064+0000","last_undegraded":"2024-02-19T21:04:42.712064+0000","last_fullsized":"2024-02-19T21:04:42.712064+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:05.136186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:05.136186+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.700493+0000","last_change":"2024-02-19T21:04:39.667403+0000","last_active":"2024-02-19T21:04:42.700493+0000","last_peered":"2024-02-19T21:04:42.700493+0000","last_clean":"2024-02-19T21:04:42.700493+0000","last_became_active":"2024-02-19T21:04:39.666739+0000","last_became_peered":"2024-02-19T21:04:39.666739+0000","last_unstale":"2024-02-19T21:04:42.700493+0000","last_undegraded":"2024-02-19T21:04:42.700493+0000","last_fullsized":"2024-02-19T21:04:42.700493+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.784969+0000","last_change":"2024-02-19T21:04:39.767845+0000","last_active":"2024-02-19T21:04:43.784969+0000","last_peered":"2024-02-19T21:04:43.784969+0000","last_clean":"2024-02-19T21:04:43.784969+0000","last_became_active":"2024-02-19T21:04:39.767076+0000","last_became_peered":"2024-02-19T21:04:39.767076+0000","last_unstale":"2024-02-19T21:04:43.784969+0000","last_undegraded":"2024-02-19T21:04:43.784969+0000","last_fullsized":"2024-02-19T21:04:43.784969+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1f","version":"100'6","reported_seq":20,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.790671+0000","last_change":"2024-02-19T21:04:39.743491+0000","last_active":"2024-02-19T21:04:43.790671+0000","last_peered":"2024-02-19T21:04:43.790671+0000","last_clean":"2024-02-19T21:04:43.790671+0000","last_became_active":"2024-02-19T21:04:39.742131+0000","last_became_peered":"2024-02-19T21:04:39.742131+0000","last_unstale":"2024-02-19T21:04:43.790671+0000","last_undegraded":"2024-02-19T21:04:43.790671+0000","last_fullsized":"2024-02-19T21:04:43.790671+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.691700+0000","last_change":"2024-02-19T21:04:39.781741+0000","last_active":"2024-02-19T21:04:42.691700+0000","last_peered":"2024-02-19T21:04:42.691700+0000","last_clean":"2024-02-19T21:04:42.691700+0000","last_became_active":"2024-02-19T21:04:39.778201+0000","last_became_peered":"2024-02-19T21:04:39.778201+0000","last_unstale":"2024-02-19T21:04:42.691700+0000","last_undegraded":"2024-02-19T21:04:42.691700+0000","last_fullsized":"2024-02-19T21:04:42.691700+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.720504+0000","last_change":"2024-02-19T21:04:44.720677+0000","last_active":"2024-02-19T21:04:44.720504+0000","last_peered":"2024-02-19T21:04:44.720504+0000","last_clean":"2024-02-19T21:04:44.720504+0000","last_became_active":"2024-02-19T21:04:39.767812+0000","last_became_peered":"2024-02-19T21:04:39.767812+0000","last_unstale":"2024-02-19T21:04:44.720504+0000","last_undegraded":"2024-02-19T21:04:44.720504+0000","last_fullsized":"2024-02-19T21:04:44.720504+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.8","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.723395+0000","last_change":"2024-02-19T21:04:44.726371+0000","last_active":"2024-02-19T21:04:44.723395+0000","last_peered":"2024-02-19T21:04:44.723395+0000","last_clean":"2024-02-19T21:04:44.723395+0000","last_became_active":"2024-02-19T21:04:39.743067+0000","last_became_peered":"2024-02-19T21:04:39.743067+0000","last_unstale":"2024-02-19T21:04:44.723395+0000","last_undegraded":"2024-02-19T21:04:44.723395+0000","last_fullsized":"2024-02-19T21:04:44.723395+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"8.b","version":"100'2","reported_seq":17,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.730542+0000","last_change":"2024-02-19T21:04:39.723860+0000","last_active":"2024-02-19T21:04:43.730542+0000","last_peered":"2024-02-19T21:04:43.730542+0000","last_clean":"2024-02-19T21:04:43.730542+0000","last_became_active":"2024-02-19T21:04:39.715390+0000","last_became_peered":"2024-02-19T21:04:39.715390+0000","last_unstale":"2024-02-19T21:04:43.730542+0000","last_undegraded":"2024-02-19T21:04:43.730542+0000","last_fullsized":"2024-02-19T21:04:43.730542+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":61,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.736518+0000","last_change":"2024-02-19T21:04:15.868824+0000","last_active":"2024-02-19T21:04:42.736518+0000","last_peered":"2024-02-19T21:04:42.736518+0000","last_clean":"2024-02-19T21:04:42.736518+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:04:42.736518+0000","last_undegraded":"2024-02-19T21:04:42.736518+0000","last_fullsized":"2024-02-19T21:04:42.736518+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:15.868726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:15.868726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.740359+0000","last_change":"2024-02-19T21:03:27.966621+0000","last_active":"2024-02-19T21:04:42.740359+0000","last_peered":"2024-02-19T21:04:42.740359+0000","last_clean":"2024-02-19T21:04:42.740359+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:04:42.740359+0000","last_undegraded":"2024-02-19T21:04:42.740359+0000","last_fullsized":"2024-02-19T21:04:42.740359+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:03:27.966528+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":65,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.727959+0000","last_change":"2024-02-19T21:03:35.999651+0000","last_active":"2024-02-19T21:04:42.727959+0000","last_peered":"2024-02-19T21:04:42.727959+0000","last_clean":"2024-02-19T21:04:42.727959+0000","last_became_active":"2024-02-19T21:01:16.276032+0000","last_became_peered":"2024-02-19T21:01:16.276032+0000","last_unstale":"2024-02-19T21:04:42.727959+0000","last_undegraded":"2024-02-19T21:04:42.727959+0000","last_fullsized":"2024-02-19T21:04:42.727959+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:35.999575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:35.999575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":270,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.905580+0000","last_change":"2024-02-19T21:04:20.241440+0000","last_active":"2024-02-19T21:04:43.905580+0000","last_peered":"2024-02-19T21:04:43.905580+0000","last_clean":"2024-02-19T21:04:43.905580+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:04:43.905580+0000","last_undegraded":"2024-02-19T21:04:43.905580+0000","last_fullsized":"2024-02-19T21:04:43.905580+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:04:20.241327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:20.241327+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":202,"num_read_kb":197,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":58,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.720714+0000","last_change":"2024-02-19T21:04:11.900738+0000","last_active":"2024-02-19T21:04:42.720714+0000","last_peered":"2024-02-19T21:04:42.720714+0000","last_clean":"2024-02-19T21:04:42.720714+0000","last_became_active":"2024-02-19T21:01:22.607883+0000","last_became_peered":"2024-02-19T21:01:22.607883+0000","last_unstale":"2024-02-19T21:04:42.720714+0000","last_undegraded":"2024-02-19T21:04:42.720714+0000","last_fullsized":"2024-02-19T21:04:42.720714+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.900625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.900625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.696129+0000","last_change":"2024-02-19T21:04:39.692210+0000","last_active":"2024-02-19T21:04:42.696129+0000","last_peered":"2024-02-19T21:04:42.696129+0000","last_clean":"2024-02-19T21:04:42.696129+0000","last_became_active":"2024-02-19T21:04:39.691434+0000","last_became_peered":"2024-02-19T21:04:39.691434+0000","last_unstale":"2024-02-19T21:04:42.696129+0000","last_undegraded":"2024-02-19T21:04:42.696129+0000","last_fullsized":"2024-02-19T21:04:42.696129+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.759486+0000","last_change":"2024-02-19T21:04:39.765986+0000","last_active":"2024-02-19T21:04:43.759486+0000","last_peered":"2024-02-19T21:04:43.759486+0000","last_clean":"2024-02-19T21:04:43.759486+0000","last_became_active":"2024-02-19T21:04:39.765719+0000","last_became_peered":"2024-02-19T21:04:39.765719+0000","last_unstale":"2024-02-19T21:04:43.759486+0000","last_undegraded":"2024-02-19T21:04:43.759486+0000","last_fullsized":"2024-02-19T21:04:43.759486+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.715278+0000","last_change":"2024-02-19T21:04:39.701428+0000","last_active":"2024-02-19T21:04:43.715278+0000","last_peered":"2024-02-19T21:04:43.715278+0000","last_clean":"2024-02-19T21:04:43.715278+0000","last_became_active":"2024-02-19T21:04:39.698760+0000","last_became_peered":"2024-02-19T21:04:39.698760+0000","last_unstale":"2024-02-19T21:04:43.715278+0000","last_undegraded":"2024-02-19T21:04:43.715278+0000","last_fullsized":"2024-02-19T21:04:43.715278+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.692221+0000","last_change":"2024-02-19T21:04:39.761020+0000","last_active":"2024-02-19T21:04:42.692221+0000","last_peered":"2024-02-19T21:04:42.692221+0000","last_clean":"2024-02-19T21:04:42.692221+0000","last_became_active":"2024-02-19T21:04:39.760735+0000","last_became_peered":"2024-02-19T21:04:39.760735+0000","last_unstale":"2024-02-19T21:04:42.692221+0000","last_undegraded":"2024-02-19T21:04:42.692221+0000","last_fullsized":"2024-02-19T21:04:42.692221+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.709985+0000","last_change":"2024-02-19T21:04:44.713017+0000","last_active":"2024-02-19T21:04:44.709985+0000","last_peered":"2024-02-19T21:04:44.709985+0000","last_clean":"2024-02-19T21:04:44.709985+0000","last_became_active":"2024-02-19T21:04:39.698371+0000","last_became_peered":"2024-02-19T21:04:39.698371+0000","last_unstale":"2024-02-19T21:04:44.709985+0000","last_undegraded":"2024-02-19T21:04:44.709985+0000","last_fullsized":"2024-02-19T21:04:44.709985+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.b","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.733402+0000","last_change":"2024-02-19T21:04:44.735020+0000","last_active":"2024-02-19T21:04:44.733402+0000","last_peered":"2024-02-19T21:04:44.733402+0000","last_clean":"2024-02-19T21:04:44.733402+0000","last_became_active":"2024-02-19T21:04:39.719715+0000","last_became_peered":"2024-02-19T21:04:39.719715+0000","last_unstale":"2024-02-19T21:04:44.733402+0000","last_undegraded":"2024-02-19T21:04:44.733402+0000","last_fullsized":"2024-02-19T21:04:44.733402+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"8.8","version":"100'2","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.708582+0000","last_change":"2024-02-19T21:04:39.773441+0000","last_active":"2024-02-19T21:04:42.708582+0000","last_peered":"2024-02-19T21:04:42.708582+0000","last_clean":"2024-02-19T21:04:42.708582+0000","last_became_active":"2024-02-19T21:04:39.771376+0000","last_became_peered":"2024-02-19T21:04:39.771376+0000","last_unstale":"2024-02-19T21:04:42.708582+0000","last_undegraded":"2024-02-19T21:04:42.708582+0000","last_fullsized":"2024-02-19T21:04:42.708582+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":57,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.788580+0000","last_change":"2024-02-19T21:02:56.510539+0000","last_active":"2024-02-19T21:04:43.788580+0000","last_peered":"2024-02-19T21:04:43.788580+0000","last_clean":"2024-02-19T21:04:43.788580+0000","last_became_active":"2024-02-19T21:01:20.290758+0000","last_became_peered":"2024-02-19T21:01:20.290758+0000","last_unstale":"2024-02-19T21:04:43.788580+0000","last_undegraded":"2024-02-19T21:04:43.788580+0000","last_fullsized":"2024-02-19T21:04:43.788580+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.510474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":153,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.711829+0000","last_change":"2024-02-19T21:04:06.383697+0000","last_active":"2024-02-19T21:04:43.711829+0000","last_peered":"2024-02-19T21:04:43.711829+0000","last_clean":"2024-02-19T21:04:43.711829+0000","last_became_active":"2024-02-19T21:01:18.289076+0000","last_became_peered":"2024-02-19T21:01:18.289076+0000","last_unstale":"2024-02-19T21:04:43.711829+0000","last_undegraded":"2024-02-19T21:04:43.711829+0000","last_fullsized":"2024-02-19T21:04:43.711829+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-19T21:04:06.383559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.383559+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":60,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.704887+0000","last_change":"2024-02-19T21:02:58.986556+0000","last_active":"2024-02-19T21:04:42.704887+0000","last_peered":"2024-02-19T21:04:42.704887+0000","last_clean":"2024-02-19T21:04:42.704887+0000","last_became_active":"2024-02-19T21:01:16.270159+0000","last_became_peered":"2024-02-19T21:01:16.270159+0000","last_unstale":"2024-02-19T21:04:42.704887+0000","last_undegraded":"2024-02-19T21:04:42.704887+0000","last_fullsized":"2024-02-19T21:04:42.704887+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:58.986458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:58.986458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.784708+0000","last_change":"2024-02-19T21:03:12.734312+0000","last_active":"2024-02-19T21:04:43.784708+0000","last_peered":"2024-02-19T21:04:43.784708+0000","last_clean":"2024-02-19T21:04:43.784708+0000","last_became_active":"2024-02-19T21:01:22.291396+0000","last_became_peered":"2024-02-19T21:01:22.291396+0000","last_unstale":"2024-02-19T21:04:43.784708+0000","last_undegraded":"2024-02-19T21:04:43.784708+0000","last_fullsized":"2024-02-19T21:04:43.784708+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:12.734219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:12.734219+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.720227+0000","last_change":"2024-02-19T21:04:39.721663+0000","last_active":"2024-02-19T21:04:42.720227+0000","last_peered":"2024-02-19T21:04:42.720227+0000","last_clean":"2024-02-19T21:04:42.720227+0000","last_became_active":"2024-02-19T21:04:39.721416+0000","last_became_peered":"2024-02-19T21:04:39.721416+0000","last_unstale":"2024-02-19T21:04:42.720227+0000","last_undegraded":"2024-02-19T21:04:42.720227+0000","last_fullsized":"2024-02-19T21:04:42.720227+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.786636+0000","last_change":"2024-02-19T21:04:39.759375+0000","last_active":"2024-02-19T21:04:43.786636+0000","last_peered":"2024-02-19T21:04:43.786636+0000","last_clean":"2024-02-19T21:04:43.786636+0000","last_became_active":"2024-02-19T21:04:39.758965+0000","last_became_peered":"2024-02-19T21:04:39.758965+0000","last_unstale":"2024-02-19T21:04:43.786636+0000","last_undegraded":"2024-02-19T21:04:43.786636+0000","last_fullsized":"2024-02-19T21:04:43.786636+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719769+0000","last_change":"2024-02-19T21:04:39.678110+0000","last_active":"2024-02-19T21:04:42.719769+0000","last_peered":"2024-02-19T21:04:42.719769+0000","last_clean":"2024-02-19T21:04:42.719769+0000","last_became_active":"2024-02-19T21:04:39.674697+0000","last_became_peered":"2024-02-19T21:04:39.674697+0000","last_unstale":"2024-02-19T21:04:42.719769+0000","last_undegraded":"2024-02-19T21:04:42.719769+0000","last_fullsized":"2024-02-19T21:04:42.719769+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719104+0000","last_change":"2024-02-19T21:04:39.740952+0000","last_active":"2024-02-19T21:04:42.719104+0000","last_peered":"2024-02-19T21:04:42.719104+0000","last_clean":"2024-02-19T21:04:42.719104+0000","last_became_active":"2024-02-19T21:04:39.740650+0000","last_became_peered":"2024-02-19T21:04:39.740650+0000","last_unstale":"2024-02-19T21:04:42.719104+0000","last_undegraded":"2024-02-19T21:04:42.719104+0000","last_fullsized":"2024-02-19T21:04:42.719104+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.729696+0000","last_change":"2024-02-19T21:04:44.730473+0000","last_active":"2024-02-19T21:04:44.729696+0000","last_peered":"2024-02-19T21:04:44.729696+0000","last_clean":"2024-02-19T21:04:44.729696+0000","last_became_active":"2024-02-19T21:04:39.797560+0000","last_became_peered":"2024-02-19T21:04:39.797560+0000","last_unstale":"2024-02-19T21:04:44.729696+0000","last_undegraded":"2024-02-19T21:04:44.729696+0000","last_fullsized":"2024-02-19T21:04:44.729696+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.d","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.735330+0000","last_change":"2024-02-19T21:04:41.655177+0000","last_active":"2024-02-19T21:04:42.735330+0000","last_peered":"2024-02-19T21:04:42.735330+0000","last_clean":"2024-02-19T21:04:42.735330+0000","last_became_active":"2024-02-19T21:04:39.656424+0000","last_became_peered":"2024-02-19T21:04:39.656424+0000","last_unstale":"2024-02-19T21:04:42.735330+0000","last_undegraded":"2024-02-19T21:04:42.735330+0000","last_fullsized":"2024-02-19T21:04:42.735330+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.e","version":"0'0","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.788268+0000","last_change":"2024-02-19T21:04:39.674303+0000","last_active":"2024-02-19T21:04:43.788268+0000","last_peered":"2024-02-19T21:04:43.788268+0000","last_clean":"2024-02-19T21:04:43.788268+0000","last_became_active":"2024-02-19T21:04:39.674039+0000","last_became_peered":"2024-02-19T21:04:39.674039+0000","last_unstale":"2024-02-19T21:04:43.788268+0000","last_undegraded":"2024-02-19T21:04:43.788268+0000","last_fullsized":"2024-02-19T21:04:43.788268+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":110,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.458997+0000","last_change":"2024-02-19T21:03:10.626933+0000","last_active":"2024-02-19T21:04:44.458997+0000","last_peered":"2024-02-19T21:04:44.458997+0000","last_clean":"2024-02-19T21:04:44.458997+0000","last_became_active":"2024-02-19T21:01:20.288918+0000","last_became_peered":"2024-02-19T21:01:20.288918+0000","last_unstale":"2024-02-19T21:04:44.458997+0000","last_undegraded":"2024-02-19T21:04:44.458997+0000","last_fullsized":"2024-02-19T21:04:44.458997+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:10.626866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:10.626866+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":73,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.693133+0000","last_change":"2024-02-19T21:02:56.539408+0000","last_active":"2024-02-19T21:04:42.693133+0000","last_peered":"2024-02-19T21:04:42.693133+0000","last_clean":"2024-02-19T21:04:42.693133+0000","last_became_active":"2024-02-19T21:01:18.285891+0000","last_became_peered":"2024-02-19T21:01:18.285891+0000","last_unstale":"2024-02-19T21:04:42.693133+0000","last_undegraded":"2024-02-19T21:04:42.693133+0000","last_fullsized":"2024-02-19T21:04:42.693133+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:02:56.539303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.539303+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":74,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.693360+0000","last_change":"2024-02-19T21:04:18.242620+0000","last_active":"2024-02-19T21:04:42.693360+0000","last_peered":"2024-02-19T21:04:42.693360+0000","last_clean":"2024-02-19T21:04:42.693360+0000","last_became_active":"2024-02-19T21:01:30.926676+0000","last_became_peered":"2024-02-19T21:01:30.926676+0000","last_unstale":"2024-02-19T21:04:42.693360+0000","last_undegraded":"2024-02-19T21:04:42.693360+0000","last_fullsized":"2024-02-19T21:04:42.693360+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:18.242556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:04:18.242556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":60,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.785521+0000","last_change":"2024-02-19T21:04:06.599565+0000","last_active":"2024-02-19T21:04:43.785521+0000","last_peered":"2024-02-19T21:04:43.785521+0000","last_clean":"2024-02-19T21:04:43.785521+0000","last_became_active":"2024-02-19T21:01:22.292133+0000","last_became_peered":"2024-02-19T21:01:22.292133+0000","last_unstale":"2024-02-19T21:04:43.785521+0000","last_undegraded":"2024-02-19T21:04:43.785521+0000","last_fullsized":"2024-02-19T21:04:43.785521+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:06.599439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.599439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.782905+0000","last_change":"2024-02-19T21:04:39.767350+0000","last_active":"2024-02-19T21:04:43.782905+0000","last_peered":"2024-02-19T21:04:43.782905+0000","last_clean":"2024-02-19T21:04:43.782905+0000","last_became_active":"2024-02-19T21:04:39.766156+0000","last_became_peered":"2024-02-19T21:04:39.766156+0000","last_unstale":"2024-02-19T21:04:43.782905+0000","last_undegraded":"2024-02-19T21:04:43.782905+0000","last_fullsized":"2024-02-19T21:04:43.782905+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.701257+0000","last_change":"2024-02-19T21:04:39.667534+0000","last_active":"2024-02-19T21:04:42.701257+0000","last_peered":"2024-02-19T21:04:42.701257+0000","last_clean":"2024-02-19T21:04:42.701257+0000","last_became_active":"2024-02-19T21:04:39.667060+0000","last_became_peered":"2024-02-19T21:04:39.667060+0000","last_unstale":"2024-02-19T21:04:42.701257+0000","last_undegraded":"2024-02-19T21:04:42.701257+0000","last_fullsized":"2024-02-19T21:04:42.701257+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.11","version":"100'1","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.691337+0000","last_change":"2024-02-19T21:04:39.781019+0000","last_active":"2024-02-19T21:04:42.691337+0000","last_peered":"2024-02-19T21:04:42.691337+0000","last_clean":"2024-02-19T21:04:42.691337+0000","last_became_active":"2024-02-19T21:04:39.776326+0000","last_became_peered":"2024-02-19T21:04:39.776326+0000","last_unstale":"2024-02-19T21:04:42.691337+0000","last_undegraded":"2024-02-19T21:04:42.691337+0000","last_fullsized":"2024-02-19T21:04:42.691337+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.698247+0000","last_change":"2024-02-19T21:04:39.642574+0000","last_active":"2024-02-19T21:04:42.698247+0000","last_peered":"2024-02-19T21:04:42.698247+0000","last_clean":"2024-02-19T21:04:42.698247+0000","last_became_active":"2024-02-19T21:04:39.642279+0000","last_became_peered":"2024-02-19T21:04:39.642279+0000","last_unstale":"2024-02-19T21:04:42.698247+0000","last_undegraded":"2024-02-19T21:04:42.698247+0000","last_fullsized":"2024-02-19T21:04:42.698247+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.709751+0000","last_change":"2024-02-19T21:04:44.710054+0000","last_active":"2024-02-19T21:04:44.709751+0000","last_peered":"2024-02-19T21:04:44.709751+0000","last_clean":"2024-02-19T21:04:44.709751+0000","last_became_active":"2024-02-19T21:04:39.699285+0000","last_became_peered":"2024-02-19T21:04:39.699285+0000","last_unstale":"2024-02-19T21:04:44.709751+0000","last_undegraded":"2024-02-19T21:04:44.709751+0000","last_fullsized":"2024-02-19T21:04:44.709751+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.6","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.729047+0000","last_change":"2024-02-19T21:04:44.729274+0000","last_active":"2024-02-19T21:04:44.729047+0000","last_peered":"2024-02-19T21:04:44.729047+0000","last_clean":"2024-02-19T21:04:44.729047+0000","last_became_active":"2024-02-19T21:04:39.799527+0000","last_became_peered":"2024-02-19T21:04:39.799527+0000","last_unstale":"2024-02-19T21:04:44.729047+0000","last_undegraded":"2024-02-19T21:04:44.729047+0000","last_fullsized":"2024-02-19T21:04:44.729047+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719073+0000","last_change":"2024-02-19T21:04:39.656217+0000","last_active":"2024-02-19T21:04:42.719073+0000","last_peered":"2024-02-19T21:04:42.719073+0000","last_clean":"2024-02-19T21:04:42.719073+0000","last_became_active":"2024-02-19T21:04:39.654865+0000","last_became_peered":"2024-02-19T21:04:39.654865+0000","last_unstale":"2024-02-19T21:04:42.719073+0000","last_undegraded":"2024-02-19T21:04:42.719073+0000","last_fullsized":"2024-02-19T21:04:42.719073+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":61,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719993+0000","last_change":"2024-02-19T21:04:16.866220+0000","last_active":"2024-02-19T21:04:42.719993+0000","last_peered":"2024-02-19T21:04:42.719993+0000","last_clean":"2024-02-19T21:04:42.719993+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:04:42.719993+0000","last_undegraded":"2024-02-19T21:04:42.719993+0000","last_fullsized":"2024-02-19T21:04:42.719993+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.866116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":82,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.735651+0000","last_change":"2024-02-19T21:03:08.046722+0000","last_active":"2024-02-19T21:04:43.735651+0000","last_peered":"2024-02-19T21:04:43.735651+0000","last_clean":"2024-02-19T21:04:43.735651+0000","last_became_active":"2024-02-19T21:01:18.285672+0000","last_became_peered":"2024-02-19T21:01:18.285672+0000","last_unstale":"2024-02-19T21:04:43.735651+0000","last_undegraded":"2024-02-19T21:04:43.735651+0000","last_fullsized":"2024-02-19T21:04:43.735651+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:03:08.046641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.046641+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":81,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.724951+0000","last_change":"2024-02-19T21:03:21.164366+0000","last_active":"2024-02-19T21:04:43.724951+0000","last_peered":"2024-02-19T21:04:43.724951+0000","last_clean":"2024-02-19T21:04:43.724951+0000","last_became_active":"2024-02-19T21:01:16.273545+0000","last_became_peered":"2024-02-19T21:01:16.273545+0000","last_unstale":"2024-02-19T21:04:43.724951+0000","last_undegraded":"2024-02-19T21:04:43.724951+0000","last_fullsized":"2024-02-19T21:04:43.724951+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:03:21.164286+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.731584+0000","last_change":"2024-02-19T21:03:17.625775+0000","last_active":"2024-02-19T21:04:43.731584+0000","last_peered":"2024-02-19T21:04:43.731584+0000","last_clean":"2024-02-19T21:04:43.731584+0000","last_became_active":"2024-02-19T21:01:22.295912+0000","last_became_peered":"2024-02-19T21:01:22.295912+0000","last_unstale":"2024-02-19T21:04:43.731584+0000","last_undegraded":"2024-02-19T21:04:43.731584+0000","last_fullsized":"2024-02-19T21:04:43.731584+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:03:17.625711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.738751+0000","last_change":"2024-02-19T21:04:39.696886+0000","last_active":"2024-02-19T21:04:42.738751+0000","last_peered":"2024-02-19T21:04:42.738751+0000","last_clean":"2024-02-19T21:04:42.738751+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:04:42.738751+0000","last_undegraded":"2024-02-19T21:04:42.738751+0000","last_fullsized":"2024-02-19T21:04:42.738751+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.758041+0000","last_change":"2024-02-19T21:04:39.769820+0000","last_active":"2024-02-19T21:04:43.758041+0000","last_peered":"2024-02-19T21:04:43.758041+0000","last_clean":"2024-02-19T21:04:43.758041+0000","last_became_active":"2024-02-19T21:04:39.767467+0000","last_became_peered":"2024-02-19T21:04:39.767467+0000","last_unstale":"2024-02-19T21:04:43.758041+0000","last_undegraded":"2024-02-19T21:04:43.758041+0000","last_fullsized":"2024-02-19T21:04:43.758041+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.788740+0000","last_change":"2024-02-19T21:04:39.741185+0000","last_active":"2024-02-19T21:04:43.788740+0000","last_peered":"2024-02-19T21:04:43.788740+0000","last_clean":"2024-02-19T21:04:43.788740+0000","last_became_active":"2024-02-19T21:04:39.740396+0000","last_became_peered":"2024-02-19T21:04:39.740396+0000","last_unstale":"2024-02-19T21:04:43.788740+0000","last_undegraded":"2024-02-19T21:04:43.788740+0000","last_fullsized":"2024-02-19T21:04:43.788740+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.693625+0000","last_change":"2024-02-19T21:04:39.757297+0000","last_active":"2024-02-19T21:04:42.693625+0000","last_peered":"2024-02-19T21:04:42.693625+0000","last_clean":"2024-02-19T21:04:42.693625+0000","last_became_active":"2024-02-19T21:04:39.747086+0000","last_became_peered":"2024-02-19T21:04:39.747086+0000","last_unstale":"2024-02-19T21:04:42.693625+0000","last_undegraded":"2024-02-19T21:04:42.693625+0000","last_fullsized":"2024-02-19T21:04:42.693625+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.714929+0000","last_change":"2024-02-19T21:04:44.715037+0000","last_active":"2024-02-19T21:04:44.714929+0000","last_peered":"2024-02-19T21:04:44.714929+0000","last_clean":"2024-02-19T21:04:44.714929+0000","last_became_active":"2024-02-19T21:04:39.742123+0000","last_became_peered":"2024-02-19T21:04:39.742123+0000","last_unstale":"2024-02-19T21:04:44.714929+0000","last_undegraded":"2024-02-19T21:04:44.714929+0000","last_fullsized":"2024-02-19T21:04:44.714929+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.19","version":"100'1","reported_seq":23,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.715995+0000","last_change":"2024-02-19T21:04:44.716237+0000","last_active":"2024-02-19T21:04:44.715995+0000","last_peered":"2024-02-19T21:04:44.715995+0000","last_clean":"2024-02-19T21:04:44.715995+0000","last_became_active":"2024-02-19T21:04:39.796004+0000","last_became_peered":"2024-02-19T21:04:39.796004+0000","last_unstale":"2024-02-19T21:04:44.715995+0000","last_undegraded":"2024-02-19T21:04:44.715995+0000","last_fullsized":"2024-02-19T21:04:44.715995+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"8.1a","version":"100'5","reported_seq":20,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.757101+0000","last_change":"2024-02-19T21:04:39.736543+0000","last_active":"2024-02-19T21:04:43.757101+0000","last_peered":"2024-02-19T21:04:43.757101+0000","last_clean":"2024-02-19T21:04:43.757101+0000","last_became_active":"2024-02-19T21:04:39.731233+0000","last_became_peered":"2024-02-19T21:04:39.731233+0000","last_unstale":"2024-02-19T21:04:43.757101+0000","last_undegraded":"2024-02-19T21:04:43.757101+0000","last_fullsized":"2024-02-19T21:04:43.757101+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":80,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.788669+0000","last_change":"2024-02-19T21:04:09.571081+0000","last_active":"2024-02-19T21:04:43.788669+0000","last_peered":"2024-02-19T21:04:43.788669+0000","last_clean":"2024-02-19T21:04:43.788669+0000","last_became_active":"2024-02-19T21:01:18.293773+0000","last_became_peered":"2024-02-19T21:01:18.293773+0000","last_unstale":"2024-02-19T21:04:43.788669+0000","last_undegraded":"2024-02-19T21:04:43.788669+0000","last_fullsized":"2024-02-19T21:04:43.788669+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:09.570972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:09.570972+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":60,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719918+0000","last_change":"2024-02-19T21:02:49.177055+0000","last_active":"2024-02-19T21:04:42.719918+0000","last_peered":"2024-02-19T21:04:42.719918+0000","last_clean":"2024-02-19T21:04:42.719918+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:04:42.719918+0000","last_undegraded":"2024-02-19T21:04:42.719918+0000","last_fullsized":"2024-02-19T21:04:42.719918+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:49.176969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:49.176969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":62,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.788295+0000","last_change":"2024-02-19T21:04:11.841348+0000","last_active":"2024-02-19T21:04:43.788295+0000","last_peered":"2024-02-19T21:04:43.788295+0000","last_clean":"2024-02-19T21:04:43.788295+0000","last_became_active":"2024-02-19T21:01:20.285298+0000","last_became_peered":"2024-02-19T21:01:20.285298+0000","last_unstale":"2024-02-19T21:04:43.788295+0000","last_undegraded":"2024-02-19T21:04:43.788295+0000","last_fullsized":"2024-02-19T21:04:43.788295+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.841184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.841184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":59,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.726828+0000","last_change":"2024-02-19T21:04:08.831108+0000","last_active":"2024-02-19T21:04:42.726828+0000","last_peered":"2024-02-19T21:04:42.726828+0000","last_clean":"2024-02-19T21:04:42.726828+0000","last_became_active":"2024-02-19T21:01:22.297086+0000","last_became_peered":"2024-02-19T21:01:22.297086+0000","last_unstale":"2024-02-19T21:04:42.726828+0000","last_undegraded":"2024-02-19T21:04:42.726828+0000","last_fullsized":"2024-02-19T21:04:42.726828+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.830994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.830994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.712580+0000","last_change":"2024-02-19T21:04:39.639826+0000","last_active":"2024-02-19T21:04:43.712580+0000","last_peered":"2024-02-19T21:04:43.712580+0000","last_clean":"2024-02-19T21:04:43.712580+0000","last_became_active":"2024-02-19T21:04:39.639518+0000","last_became_peered":"2024-02-19T21:04:39.639518+0000","last_unstale":"2024-02-19T21:04:43.712580+0000","last_undegraded":"2024-02-19T21:04:43.712580+0000","last_fullsized":"2024-02-19T21:04:43.712580+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.764727+0000","last_change":"2024-02-19T21:04:39.726109+0000","last_active":"2024-02-19T21:04:43.764727+0000","last_peered":"2024-02-19T21:04:43.764727+0000","last_clean":"2024-02-19T21:04:43.764727+0000","last_became_active":"2024-02-19T21:04:39.725440+0000","last_became_peered":"2024-02-19T21:04:39.725440+0000","last_unstale":"2024-02-19T21:04:43.764727+0000","last_undegraded":"2024-02-19T21:04:43.764727+0000","last_fullsized":"2024-02-19T21:04:43.764727+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.720381+0000","last_change":"2024-02-19T21:04:39.679134+0000","last_active":"2024-02-19T21:04:42.720381+0000","last_peered":"2024-02-19T21:04:42.720381+0000","last_clean":"2024-02-19T21:04:42.720381+0000","last_became_active":"2024-02-19T21:04:39.678877+0000","last_became_peered":"2024-02-19T21:04:39.678877+0000","last_unstale":"2024-02-19T21:04:42.720381+0000","last_undegraded":"2024-02-19T21:04:42.720381+0000","last_fullsized":"2024-02-19T21:04:42.720381+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.770970+0000","last_change":"2024-02-19T21:04:39.737854+0000","last_active":"2024-02-19T21:04:43.770970+0000","last_peered":"2024-02-19T21:04:43.770970+0000","last_clean":"2024-02-19T21:04:43.770970+0000","last_became_active":"2024-02-19T21:04:39.737693+0000","last_became_peered":"2024-02-19T21:04:39.737693+0000","last_unstale":"2024-02-19T21:04:43.770970+0000","last_undegraded":"2024-02-19T21:04:43.770970+0000","last_fullsized":"2024-02-19T21:04:43.770970+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.716992+0000","last_change":"2024-02-19T21:04:44.722521+0000","last_active":"2024-02-19T21:04:44.716992+0000","last_peered":"2024-02-19T21:04:44.716992+0000","last_clean":"2024-02-19T21:04:44.716992+0000","last_became_active":"2024-02-19T21:04:39.678427+0000","last_became_peered":"2024-02-19T21:04:39.678427+0000","last_unstale":"2024-02-19T21:04:44.716992+0000","last_undegraded":"2024-02-19T21:04:44.716992+0000","last_fullsized":"2024-02-19T21:04:44.716992+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.18","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719701+0000","last_change":"2024-02-19T21:04:41.670877+0000","last_active":"2024-02-19T21:04:42.719701+0000","last_peered":"2024-02-19T21:04:42.719701+0000","last_clean":"2024-02-19T21:04:42.719701+0000","last_became_active":"2024-02-19T21:04:39.760845+0000","last_became_peered":"2024-02-19T21:04:39.760845+0000","last_unstale":"2024-02-19T21:04:42.719701+0000","last_undegraded":"2024-02-19T21:04:42.719701+0000","last_fullsized":"2024-02-19T21:04:42.719701+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.1b","version":"100'10","reported_seq":25,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.713897+0000","last_change":"2024-02-19T21:04:39.642209+0000","last_active":"2024-02-19T21:04:43.713897+0000","last_peered":"2024-02-19T21:04:43.713897+0000","last_clean":"2024-02-19T21:04:43.713897+0000","last_became_active":"2024-02-19T21:04:39.638760+0000","last_became_peered":"2024-02-19T21:04:39.638760+0000","last_unstale":"2024-02-19T21:04:43.713897+0000","last_undegraded":"2024-02-19T21:04:43.713897+0000","last_fullsized":"2024-02-19T21:04:43.713897+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":76,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.788225+0000","last_change":"2024-02-19T21:04:06.807310+0000","last_active":"2024-02-19T21:04:43.788225+0000","last_peered":"2024-02-19T21:04:43.788225+0000","last_clean":"2024-02-19T21:04:43.788225+0000","last_became_active":"2024-02-19T21:01:18.293913+0000","last_became_peered":"2024-02-19T21:01:18.293913+0000","last_unstale":"2024-02-19T21:04:43.788225+0000","last_undegraded":"2024-02-19T21:04:43.788225+0000","last_fullsized":"2024-02-19T21:04:43.788225+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:04:06.807190+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.807190+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":65,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.705365+0000","last_change":"2024-02-19T21:04:08.757186+0000","last_active":"2024-02-19T21:04:42.705365+0000","last_peered":"2024-02-19T21:04:42.705365+0000","last_clean":"2024-02-19T21:04:42.705365+0000","last_became_active":"2024-02-19T21:01:16.271210+0000","last_became_peered":"2024-02-19T21:01:16.271210+0000","last_unstale":"2024-02-19T21:04:42.705365+0000","last_undegraded":"2024-02-19T21:04:42.705365+0000","last_fullsized":"2024-02-19T21:04:42.705365+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.757098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.757098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":56,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.709952+0000","last_change":"2024-02-19T21:03:08.984824+0000","last_active":"2024-02-19T21:04:42.709952+0000","last_peered":"2024-02-19T21:04:42.709952+0000","last_clean":"2024-02-19T21:04:42.709952+0000","last_became_active":"2024-02-19T21:01:20.288589+0000","last_became_peered":"2024-02-19T21:01:20.288589+0000","last_unstale":"2024-02-19T21:04:42.709952+0000","last_undegraded":"2024-02-19T21:04:42.709952+0000","last_fullsized":"2024-02-19T21:04:42.709952+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:08.984720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.984720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":58,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.720318+0000","last_change":"2024-02-19T21:04:06.880972+0000","last_active":"2024-02-19T21:04:42.720318+0000","last_peered":"2024-02-19T21:04:42.720318+0000","last_clean":"2024-02-19T21:04:42.720318+0000","last_became_active":"2024-02-19T21:01:22.608285+0000","last_became_peered":"2024-02-19T21:01:22.608285+0000","last_unstale":"2024-02-19T21:04:42.720318+0000","last_undegraded":"2024-02-19T21:04:42.720318+0000","last_fullsized":"2024-02-19T21:04:42.720318+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:06.880899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.880899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.704151+0000","last_change":"2024-02-19T21:04:39.641073+0000","last_active":"2024-02-19T21:04:42.704151+0000","last_peered":"2024-02-19T21:04:42.704151+0000","last_clean":"2024-02-19T21:04:42.704151+0000","last_became_active":"2024-02-19T21:04:39.639775+0000","last_became_peered":"2024-02-19T21:04:39.639775+0000","last_unstale":"2024-02-19T21:04:42.704151+0000","last_undegraded":"2024-02-19T21:04:42.704151+0000","last_fullsized":"2024-02-19T21:04:42.704151+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.727606+0000","last_change":"2024-02-19T21:04:39.723657+0000","last_active":"2024-02-19T21:04:42.727606+0000","last_peered":"2024-02-19T21:04:42.727606+0000","last_clean":"2024-02-19T21:04:42.727606+0000","last_became_active":"2024-02-19T21:04:39.723372+0000","last_became_peered":"2024-02-19T21:04:39.723372+0000","last_unstale":"2024-02-19T21:04:42.727606+0000","last_undegraded":"2024-02-19T21:04:42.727606+0000","last_fullsized":"2024-02-19T21:04:42.727606+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"100'1","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.759008+0000","last_change":"2024-02-19T21:04:39.753081+0000","last_active":"2024-02-19T21:04:43.759008+0000","last_peered":"2024-02-19T21:04:43.759008+0000","last_clean":"2024-02-19T21:04:43.759008+0000","last_became_active":"2024-02-19T21:04:39.752718+0000","last_became_peered":"2024-02-19T21:04:39.752718+0000","last_unstale":"2024-02-19T21:04:43.759008+0000","last_undegraded":"2024-02-19T21:04:43.759008+0000","last_fullsized":"2024-02-19T21:04:43.759008+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.755912+0000","last_change":"2024-02-19T21:04:39.683920+0000","last_active":"2024-02-19T21:04:43.755912+0000","last_peered":"2024-02-19T21:04:43.755912+0000","last_clean":"2024-02-19T21:04:43.755912+0000","last_became_active":"2024-02-19T21:04:39.677947+0000","last_became_peered":"2024-02-19T21:04:39.677947+0000","last_unstale":"2024-02-19T21:04:43.755912+0000","last_undegraded":"2024-02-19T21:04:43.755912+0000","last_fullsized":"2024-02-19T21:04:43.755912+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.732921+0000","last_change":"2024-02-19T21:04:41.656233+0000","last_active":"2024-02-19T21:04:42.732921+0000","last_peered":"2024-02-19T21:04:42.732921+0000","last_clean":"2024-02-19T21:04:42.732921+0000","last_became_active":"2024-02-19T21:04:39.735226+0000","last_became_peered":"2024-02-19T21:04:39.735226+0000","last_unstale":"2024-02-19T21:04:42.732921+0000","last_undegraded":"2024-02-19T21:04:42.732921+0000","last_fullsized":"2024-02-19T21:04:42.732921+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.1b","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.714174+0000","last_change":"2024-02-19T21:04:44.714887+0000","last_active":"2024-02-19T21:04:44.714174+0000","last_peered":"2024-02-19T21:04:44.714174+0000","last_clean":"2024-02-19T21:04:44.714174+0000","last_became_active":"2024-02-19T21:04:39.639545+0000","last_became_peered":"2024-02-19T21:04:39.639545+0000","last_unstale":"2024-02-19T21:04:44.714174+0000","last_undegraded":"2024-02-19T21:04:44.714174+0000","last_fullsized":"2024-02-19T21:04:44.714174+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"8.18","version":"100'2","reported_seq":17,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.704320+0000","last_change":"2024-02-19T21:04:39.635873+0000","last_active":"2024-02-19T21:04:43.704320+0000","last_peered":"2024-02-19T21:04:43.704320+0000","last_clean":"2024-02-19T21:04:43.704320+0000","last_became_active":"2024-02-19T21:04:39.631391+0000","last_became_peered":"2024-02-19T21:04:39.631391+0000","last_unstale":"2024-02-19T21:04:43.704320+0000","last_undegraded":"2024-02-19T21:04:43.704320+0000","last_fullsized":"2024-02-19T21:04:43.704320+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":79,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.758853+0000","last_change":"2024-02-19T21:04:22.763995+0000","last_active":"2024-02-19T21:04:43.758853+0000","last_peered":"2024-02-19T21:04:43.758853+0000","last_clean":"2024-02-19T21:04:43.758853+0000","last_became_active":"2024-02-19T21:01:18.294369+0000","last_became_peered":"2024-02-19T21:01:18.294369+0000","last_unstale":"2024-02-19T21:04:43.758853+0000","last_undegraded":"2024-02-19T21:04:43.758853+0000","last_fullsized":"2024-02-19T21:04:43.758853+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:04:22.763926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:22.763926+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":65,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.704730+0000","last_change":"2024-02-19T21:04:09.180232+0000","last_active":"2024-02-19T21:04:42.704730+0000","last_peered":"2024-02-19T21:04:42.704730+0000","last_clean":"2024-02-19T21:04:42.704730+0000","last_became_active":"2024-02-19T21:01:16.271608+0000","last_became_peered":"2024-02-19T21:01:16.271608+0000","last_unstale":"2024-02-19T21:04:42.704730+0000","last_undegraded":"2024-02-19T21:04:42.704730+0000","last_fullsized":"2024-02-19T21:04:42.704730+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:09.180159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:09.180159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":106,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.745585+0000","last_change":"2024-02-19T21:03:11.063302+0000","last_active":"2024-02-19T21:04:43.745585+0000","last_peered":"2024-02-19T21:04:43.745585+0000","last_clean":"2024-02-19T21:04:43.745585+0000","last_became_active":"2024-02-19T21:01:20.288799+0000","last_became_peered":"2024-02-19T21:01:20.288799+0000","last_unstale":"2024-02-19T21:04:43.745585+0000","last_undegraded":"2024-02-19T21:04:43.745585+0000","last_fullsized":"2024-02-19T21:04:43.745585+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:11.063222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:11.063222+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":61,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.788434+0000","last_change":"2024-02-19T21:04:05.558879+0000","last_active":"2024-02-19T21:04:43.788434+0000","last_peered":"2024-02-19T21:04:43.788434+0000","last_clean":"2024-02-19T21:04:43.788434+0000","last_became_active":"2024-02-19T21:01:22.297188+0000","last_became_peered":"2024-02-19T21:01:22.297188+0000","last_unstale":"2024-02-19T21:04:43.788434+0000","last_undegraded":"2024-02-19T21:04:43.788434+0000","last_fullsized":"2024-02-19T21:04:43.788434+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:04:05.558765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:05.558765+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.793901+0000","last_change":"2024-02-19T21:04:39.688410+0000","last_active":"2024-02-19T21:04:43.793901+0000","last_peered":"2024-02-19T21:04:43.793901+0000","last_clean":"2024-02-19T21:04:43.793901+0000","last_became_active":"2024-02-19T21:04:39.687668+0000","last_became_peered":"2024-02-19T21:04:39.687668+0000","last_unstale":"2024-02-19T21:04:43.793901+0000","last_undegraded":"2024-02-19T21:04:43.793901+0000","last_fullsized":"2024-02-19T21:04:43.793901+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.791017+0000","last_change":"2024-02-19T21:04:39.754053+0000","last_active":"2024-02-19T21:04:43.791017+0000","last_peered":"2024-02-19T21:04:43.791017+0000","last_clean":"2024-02-19T21:04:43.791017+0000","last_became_active":"2024-02-19T21:04:39.753243+0000","last_became_peered":"2024-02-19T21:04:39.753243+0000","last_unstale":"2024-02-19T21:04:43.791017+0000","last_undegraded":"2024-02-19T21:04:43.791017+0000","last_fullsized":"2024-02-19T21:04:43.791017+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.720636+0000","last_change":"2024-02-19T21:04:39.749884+0000","last_active":"2024-02-19T21:04:42.720636+0000","last_peered":"2024-02-19T21:04:42.720636+0000","last_clean":"2024-02-19T21:04:42.720636+0000","last_became_active":"2024-02-19T21:04:39.748211+0000","last_became_peered":"2024-02-19T21:04:39.748211+0000","last_unstale":"2024-02-19T21:04:42.720636+0000","last_undegraded":"2024-02-19T21:04:42.720636+0000","last_fullsized":"2024-02-19T21:04:42.720636+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.b","version":"102'4","reported_seq":18,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.731277+0000","last_change":"2024-02-19T21:04:39.755140+0000","last_active":"2024-02-19T21:04:43.731277+0000","last_peered":"2024-02-19T21:04:43.731277+0000","last_clean":"2024-02-19T21:04:43.731277+0000","last_became_active":"2024-02-19T21:04:39.754518+0000","last_became_peered":"2024-02-19T21:04:39.754518+0000","last_unstale":"2024-02-19T21:04:43.731277+0000","last_undegraded":"2024-02-19T21:04:43.731277+0000","last_fullsized":"2024-02-19T21:04:43.731277+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.707283+0000","last_change":"2024-02-19T21:04:41.645424+0000","last_active":"2024-02-19T21:04:42.707283+0000","last_peered":"2024-02-19T21:04:42.707283+0000","last_clean":"2024-02-19T21:04:42.707283+0000","last_became_active":"2024-02-19T21:04:39.662426+0000","last_became_peered":"2024-02-19T21:04:39.662426+0000","last_unstale":"2024-02-19T21:04:42.707283+0000","last_undegraded":"2024-02-19T21:04:42.707283+0000","last_fullsized":"2024-02-19T21:04:42.707283+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.1a","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.704807+0000","last_change":"2024-02-19T21:04:41.670384+0000","last_active":"2024-02-19T21:04:42.704807+0000","last_peered":"2024-02-19T21:04:42.704807+0000","last_clean":"2024-02-19T21:04:42.704807+0000","last_became_active":"2024-02-19T21:04:39.750848+0000","last_became_peered":"2024-02-19T21:04:39.750848+0000","last_unstale":"2024-02-19T21:04:42.704807+0000","last_undegraded":"2024-02-19T21:04:42.704807+0000","last_fullsized":"2024-02-19T21:04:42.704807+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.700906+0000","last_change":"2024-02-19T21:04:39.667189+0000","last_active":"2024-02-19T21:04:42.700906+0000","last_peered":"2024-02-19T21:04:42.700906+0000","last_clean":"2024-02-19T21:04:42.700906+0000","last_became_active":"2024-02-19T21:04:39.666360+0000","last_became_peered":"2024-02-19T21:04:39.666360+0000","last_unstale":"2024-02-19T21:04:42.700906+0000","last_undegraded":"2024-02-19T21:04:42.700906+0000","last_fullsized":"2024-02-19T21:04:42.700906+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":79,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.709661+0000","last_change":"2024-02-19T21:04:15.187687+0000","last_active":"2024-02-19T21:04:42.709661+0000","last_peered":"2024-02-19T21:04:42.709661+0000","last_clean":"2024-02-19T21:04:42.709661+0000","last_became_active":"2024-02-19T21:01:18.293786+0000","last_became_peered":"2024-02-19T21:01:18.293786+0000","last_unstale":"2024-02-19T21:04:42.709661+0000","last_undegraded":"2024-02-19T21:04:42.709661+0000","last_fullsized":"2024-02-19T21:04:42.709661+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:15.187587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:15.187587+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":66,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.769755+0000","last_change":"2024-02-19T21:03:39.467124+0000","last_active":"2024-02-19T21:04:43.769755+0000","last_peered":"2024-02-19T21:04:43.769755+0000","last_clean":"2024-02-19T21:04:43.769755+0000","last_became_active":"2024-02-19T21:01:16.273181+0000","last_became_peered":"2024-02-19T21:01:16.273181+0000","last_unstale":"2024-02-19T21:04:43.769755+0000","last_undegraded":"2024-02-19T21:04:43.769755+0000","last_fullsized":"2024-02-19T21:04:43.769755+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:39.467059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:39.467059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":107,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.459049+0000","last_change":"2024-02-19T21:03:26.669863+0000","last_active":"2024-02-19T21:04:44.459049+0000","last_peered":"2024-02-19T21:04:44.459049+0000","last_clean":"2024-02-19T21:04:44.459049+0000","last_became_active":"2024-02-19T21:01:20.283943+0000","last_became_peered":"2024-02-19T21:01:20.283943+0000","last_unstale":"2024-02-19T21:04:44.459049+0000","last_undegraded":"2024-02-19T21:04:44.459049+0000","last_fullsized":"2024-02-19T21:04:44.459049+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:26.669794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:26.669794+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":60,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.726037+0000","last_change":"2024-02-19T21:04:15.099340+0000","last_active":"2024-02-19T21:04:43.726037+0000","last_peered":"2024-02-19T21:04:43.726037+0000","last_clean":"2024-02-19T21:04:43.726037+0000","last_became_active":"2024-02-19T21:01:22.299969+0000","last_became_peered":"2024-02-19T21:01:22.299969+0000","last_unstale":"2024-02-19T21:04:43.726037+0000","last_undegraded":"2024-02-19T21:04:43.726037+0000","last_fullsized":"2024-02-19T21:04:43.726037+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:15.099208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_clean_scrub_stamp":"2024-02-19T21:04:15.099208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":26,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.723555+0000","last_change":"2024-02-19T21:04:39.767396+0000","last_active":"2024-02-19T21:04:42.723555+0000","last_peered":"2024-02-19T21:04:42.723555+0000","last_clean":"2024-02-19T21:04:42.723555+0000","last_became_active":"2024-02-19T21:04:39.760167+0000","last_became_peered":"2024-02-19T21:04:39.760167+0000","last_unstale":"2024-02-19T21:04:42.723555+0000","last_undegraded":"2024-02-19T21:04:42.723555+0000","last_fullsized":"2024-02-19T21:04:42.723555+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.784897+0000","last_change":"2024-02-19T21:04:39.767683+0000","last_active":"2024-02-19T21:04:43.784897+0000","last_peered":"2024-02-19T21:04:43.784897+0000","last_clean":"2024-02-19T21:04:43.784897+0000","last_became_active":"2024-02-19T21:04:39.766594+0000","last_became_peered":"2024-02-19T21:04:39.766594+0000","last_unstale":"2024-02-19T21:04:43.784897+0000","last_undegraded":"2024-02-19T21:04:43.784897+0000","last_fullsized":"2024-02-19T21:04:43.784897+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"102'5","reported_seq":22,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.704326+0000","last_change":"2024-02-19T21:04:39.751144+0000","last_active":"2024-02-19T21:04:42.704326+0000","last_peered":"2024-02-19T21:04:42.704326+0000","last_clean":"2024-02-19T21:04:42.704326+0000","last_became_active":"2024-02-19T21:04:39.750165+0000","last_became_peered":"2024-02-19T21:04:39.750165+0000","last_unstale":"2024-02-19T21:04:42.704326+0000","last_undegraded":"2024-02-19T21:04:42.704326+0000","last_fullsized":"2024-02-19T21:04:42.704326+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165843,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":2,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.c","version":"102'2","reported_seq":18,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.996349+0000","last_change":"2024-02-19T21:04:39.734685+0000","last_active":"2024-02-19T21:04:42.996349+0000","last_peered":"2024-02-19T21:04:42.996349+0000","last_clean":"2024-02-19T21:04:42.996349+0000","last_became_active":"2024-02-19T21:04:39.734349+0000","last_became_peered":"2024-02-19T21:04:39.734349+0000","last_unstale":"2024-02-19T21:04:42.996349+0000","last_undegraded":"2024-02-19T21:04:42.996349+0000","last_fullsized":"2024-02-19T21:04:42.996349+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.712470+0000","last_change":"2024-02-19T21:04:41.670549+0000","last_active":"2024-02-19T21:04:42.712470+0000","last_peered":"2024-02-19T21:04:42.712470+0000","last_clean":"2024-02-19T21:04:42.712470+0000","last_became_active":"2024-02-19T21:04:39.684894+0000","last_became_peered":"2024-02-19T21:04:39.684894+0000","last_unstale":"2024-02-19T21:04:42.712470+0000","last_undegraded":"2024-02-19T21:04:42.712470+0000","last_fullsized":"2024-02-19T21:04:42.712470+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.1d","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719885+0000","last_change":"2024-02-19T21:04:41.649348+0000","last_active":"2024-02-19T21:04:42.719885+0000","last_peered":"2024-02-19T21:04:42.719885+0000","last_clean":"2024-02-19T21:04:42.719885+0000","last_became_active":"2024-02-19T21:04:39.739538+0000","last_became_peered":"2024-02-19T21:04:39.739538+0000","last_unstale":"2024-02-19T21:04:42.719885+0000","last_undegraded":"2024-02-19T21:04:42.719885+0000","last_fullsized":"2024-02-19T21:04:42.719885+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.1e","version":"0'0","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.783471+0000","last_change":"2024-02-19T21:04:39.704782+0000","last_active":"2024-02-19T21:04:43.783471+0000","last_peered":"2024-02-19T21:04:43.783471+0000","last_clean":"2024-02-19T21:04:43.783471+0000","last_became_active":"2024-02-19T21:04:39.704532+0000","last_became_peered":"2024-02-19T21:04:39.704532+0000","last_unstale":"2024-02-19T21:04:43.783471+0000","last_undegraded":"2024-02-19T21:04:43.783471+0000","last_fullsized":"2024-02-19T21:04:43.783471+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":62,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.703364+0000","last_change":"2024-02-19T21:03:22.984626+0000","last_active":"2024-02-19T21:04:42.703364+0000","last_peered":"2024-02-19T21:04:42.703364+0000","last_clean":"2024-02-19T21:04:42.703364+0000","last_became_active":"2024-02-19T21:01:30.922577+0000","last_became_peered":"2024-02-19T21:01:30.922577+0000","last_unstale":"2024-02-19T21:04:42.703364+0000","last_undegraded":"2024-02-19T21:04:42.703364+0000","last_fullsized":"2024-02-19T21:04:42.703364+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:22.984547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:22.984547+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":65,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.716056+0000","last_change":"2024-02-19T21:03:33.049004+0000","last_active":"2024-02-19T21:04:42.716056+0000","last_peered":"2024-02-19T21:04:42.716056+0000","last_clean":"2024-02-19T21:04:42.716056+0000","last_became_active":"2024-02-19T21:01:16.271316+0000","last_became_peered":"2024-02-19T21:01:16.271316+0000","last_unstale":"2024-02-19T21:04:42.716056+0000","last_undegraded":"2024-02-19T21:04:42.716056+0000","last_fullsized":"2024-02-19T21:04:42.716056+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:33.048928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:33.048928+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":57,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.731508+0000","last_change":"2024-02-19T21:03:03.776497+0000","last_active":"2024-02-19T21:04:43.731508+0000","last_peered":"2024-02-19T21:04:43.731508+0000","last_clean":"2024-02-19T21:04:43.731508+0000","last_became_active":"2024-02-19T21:01:20.289691+0000","last_became_peered":"2024-02-19T21:01:20.289691+0000","last_unstale":"2024-02-19T21:04:43.731508+0000","last_undegraded":"2024-02-19T21:04:43.731508+0000","last_fullsized":"2024-02-19T21:04:43.731508+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_clean_scrub_stamp":"2024-02-19T21:03:03.776397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":60,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.785443+0000","last_change":"2024-02-19T21:04:08.616495+0000","last_active":"2024-02-19T21:04:43.785443+0000","last_peered":"2024-02-19T21:04:43.785443+0000","last_clean":"2024-02-19T21:04:43.785443+0000","last_became_active":"2024-02-19T21:01:22.293093+0000","last_became_peered":"2024-02-19T21:01:22.293093+0000","last_unstale":"2024-02-19T21:04:43.785443+0000","last_undegraded":"2024-02-19T21:04:43.785443+0000","last_fullsized":"2024-02-19T21:04:43.785443+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.616396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.616396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.716348+0000","last_change":"2024-02-19T21:04:39.674336+0000","last_active":"2024-02-19T21:04:42.716348+0000","last_peered":"2024-02-19T21:04:42.716348+0000","last_clean":"2024-02-19T21:04:42.716348+0000","last_became_active":"2024-02-19T21:04:39.672228+0000","last_became_peered":"2024-02-19T21:04:39.672228+0000","last_unstale":"2024-02-19T21:04:42.716348+0000","last_undegraded":"2024-02-19T21:04:42.716348+0000","last_fullsized":"2024-02-19T21:04:42.716348+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.701184+0000","last_change":"2024-02-19T21:04:39.637113+0000","last_active":"2024-02-19T21:04:42.701184+0000","last_peered":"2024-02-19T21:04:42.701184+0000","last_clean":"2024-02-19T21:04:42.701184+0000","last_became_active":"2024-02-19T21:04:39.632631+0000","last_became_peered":"2024-02-19T21:04:39.632631+0000","last_unstale":"2024-02-19T21:04:42.701184+0000","last_undegraded":"2024-02-19T21:04:42.701184+0000","last_fullsized":"2024-02-19T21:04:42.701184+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.b","version":"102'3","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.714949+0000","last_change":"2024-02-19T21:04:39.657768+0000","last_active":"2024-02-19T21:04:42.714949+0000","last_peered":"2024-02-19T21:04:42.714949+0000","last_clean":"2024-02-19T21:04:42.714949+0000","last_became_active":"2024-02-19T21:04:39.655157+0000","last_became_peered":"2024-02-19T21:04:39.655157+0000","last_unstale":"2024-02-19T21:04:42.714949+0000","last_undegraded":"2024-02-19T21:04:42.714949+0000","last_fullsized":"2024-02-19T21:04:42.714949+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2807,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.764495+0000","last_change":"2024-02-19T21:04:39.747689+0000","last_active":"2024-02-19T21:04:43.764495+0000","last_peered":"2024-02-19T21:04:43.764495+0000","last_clean":"2024-02-19T21:04:43.764495+0000","last_became_active":"2024-02-19T21:04:39.747466+0000","last_became_peered":"2024-02-19T21:04:39.747466+0000","last_unstale":"2024-02-19T21:04:43.764495+0000","last_undegraded":"2024-02-19T21:04:43.764495+0000","last_fullsized":"2024-02-19T21:04:43.764495+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.729505+0000","last_change":"2024-02-19T21:04:44.729647+0000","last_active":"2024-02-19T21:04:44.729505+0000","last_peered":"2024-02-19T21:04:44.729505+0000","last_clean":"2024-02-19T21:04:44.729505+0000","last_became_active":"2024-02-19T21:04:39.793619+0000","last_became_peered":"2024-02-19T21:04:39.793619+0000","last_unstale":"2024-02-19T21:04:44.729505+0000","last_undegraded":"2024-02-19T21:04:44.729505+0000","last_fullsized":"2024-02-19T21:04:44.729505+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.1c","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.714148+0000","last_change":"2024-02-19T21:04:41.662352+0000","last_active":"2024-02-19T21:04:42.714148+0000","last_peered":"2024-02-19T21:04:42.714148+0000","last_clean":"2024-02-19T21:04:42.714148+0000","last_became_active":"2024-02-19T21:04:39.673838+0000","last_became_peered":"2024-02-19T21:04:39.673838+0000","last_unstale":"2024-02-19T21:04:42.714148+0000","last_undegraded":"2024-02-19T21:04:42.714148+0000","last_fullsized":"2024-02-19T21:04:42.714148+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.1f","version":"100'5","reported_seq":20,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.789372+0000","last_change":"2024-02-19T21:04:39.743813+0000","last_active":"2024-02-19T21:04:43.789372+0000","last_peered":"2024-02-19T21:04:43.789372+0000","last_clean":"2024-02-19T21:04:43.789372+0000","last_became_active":"2024-02-19T21:04:39.742840+0000","last_became_peered":"2024-02-19T21:04:39.742840+0000","last_unstale":"2024-02-19T21:04:43.789372+0000","last_undegraded":"2024-02-19T21:04:43.789372+0000","last_fullsized":"2024-02-19T21:04:43.789372+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":83,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.713667+0000","last_change":"2024-02-19T21:04:09.139464+0000","last_active":"2024-02-19T21:04:43.713667+0000","last_peered":"2024-02-19T21:04:43.713667+0000","last_clean":"2024-02-19T21:04:43.713667+0000","last_became_active":"2024-02-19T21:01:18.286128+0000","last_became_peered":"2024-02-19T21:01:18.286128+0000","last_unstale":"2024-02-19T21:04:43.713667+0000","last_undegraded":"2024-02-19T21:04:43.713667+0000","last_fullsized":"2024-02-19T21:04:43.713667+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:04:09.139344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:09.139344+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":65,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.695445+0000","last_change":"2024-02-19T21:03:20.445543+0000","last_active":"2024-02-19T21:04:42.695445+0000","last_peered":"2024-02-19T21:04:42.695445+0000","last_clean":"2024-02-19T21:04:42.695445+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:04:42.695445+0000","last_undegraded":"2024-02-19T21:04:42.695445+0000","last_fullsized":"2024-02-19T21:04:42.695445+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:20.445477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:20.445477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":61,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.693213+0000","last_change":"2024-02-19T21:04:13.337608+0000","last_active":"2024-02-19T21:04:42.693213+0000","last_peered":"2024-02-19T21:04:42.693213+0000","last_clean":"2024-02-19T21:04:42.693213+0000","last_became_active":"2024-02-19T21:01:20.289020+0000","last_became_peered":"2024-02-19T21:01:20.289020+0000","last_unstale":"2024-02-19T21:04:42.693213+0000","last_undegraded":"2024-02-19T21:04:42.693213+0000","last_fullsized":"2024-02-19T21:04:42.693213+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.337498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.337498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.737281+0000","last_change":"2024-02-19T21:03:05.734069+0000","last_active":"2024-02-19T21:04:43.737281+0000","last_peered":"2024-02-19T21:04:43.737281+0000","last_clean":"2024-02-19T21:04:43.737281+0000","last_became_active":"2024-02-19T21:01:22.292704+0000","last_became_peered":"2024-02-19T21:01:22.292704+0000","last_unstale":"2024-02-19T21:04:43.737281+0000","last_undegraded":"2024-02-19T21:04:43.737281+0000","last_fullsized":"2024-02-19T21:04:43.737281+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.733980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.733980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.786708+0000","last_change":"2024-02-19T21:04:39.672911+0000","last_active":"2024-02-19T21:04:43.786708+0000","last_peered":"2024-02-19T21:04:43.786708+0000","last_clean":"2024-02-19T21:04:43.786708+0000","last_became_active":"2024-02-19T21:04:39.672672+0000","last_became_peered":"2024-02-19T21:04:39.672672+0000","last_unstale":"2024-02-19T21:04:43.786708+0000","last_undegraded":"2024-02-19T21:04:43.786708+0000","last_fullsized":"2024-02-19T21:04:43.786708+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.796374+0000","last_change":"2024-02-19T21:04:39.731136+0000","last_active":"2024-02-19T21:04:43.796374+0000","last_peered":"2024-02-19T21:04:43.796374+0000","last_clean":"2024-02-19T21:04:43.796374+0000","last_became_active":"2024-02-19T21:04:39.730536+0000","last_became_peered":"2024-02-19T21:04:39.730536+0000","last_unstale":"2024-02-19T21:04:43.796374+0000","last_undegraded":"2024-02-19T21:04:43.796374+0000","last_fullsized":"2024-02-19T21:04:43.796374+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.8","version":"100'2","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.707833+0000","last_change":"2024-02-19T21:04:39.726991+0000","last_active":"2024-02-19T21:04:42.707833+0000","last_peered":"2024-02-19T21:04:42.707833+0000","last_clean":"2024-02-19T21:04:42.707833+0000","last_became_active":"2024-02-19T21:04:39.725742+0000","last_became_peered":"2024-02-19T21:04:39.725742+0000","last_unstale":"2024-02-19T21:04:42.707833+0000","last_undegraded":"2024-02-19T21:04:42.707833+0000","last_fullsized":"2024-02-19T21:04:42.707833+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.710116+0000","last_change":"2024-02-19T21:04:39.757170+0000","last_active":"2024-02-19T21:04:42.710116+0000","last_peered":"2024-02-19T21:04:42.710116+0000","last_clean":"2024-02-19T21:04:42.710116+0000","last_became_active":"2024-02-19T21:04:39.753948+0000","last_became_peered":"2024-02-19T21:04:39.753948+0000","last_unstale":"2024-02-19T21:04:42.710116+0000","last_undegraded":"2024-02-19T21:04:42.710116+0000","last_fullsized":"2024-02-19T21:04:42.710116+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.718933+0000","last_change":"2024-02-19T21:04:41.683747+0000","last_active":"2024-02-19T21:04:42.718933+0000","last_peered":"2024-02-19T21:04:42.718933+0000","last_clean":"2024-02-19T21:04:42.718933+0000","last_became_active":"2024-02-19T21:04:39.653176+0000","last_became_peered":"2024-02-19T21:04:39.653176+0000","last_unstale":"2024-02-19T21:04:42.718933+0000","last_undegraded":"2024-02-19T21:04:42.718933+0000","last_fullsized":"2024-02-19T21:04:42.718933+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.1f","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.740529+0000","last_change":"2024-02-19T21:04:41.671819+0000","last_active":"2024-02-19T21:04:42.740529+0000","last_peered":"2024-02-19T21:04:42.740529+0000","last_clean":"2024-02-19T21:04:42.740529+0000","last_became_active":"2024-02-19T21:04:39.788338+0000","last_became_peered":"2024-02-19T21:04:39.788338+0000","last_unstale":"2024-02-19T21:04:42.740529+0000","last_undegraded":"2024-02-19T21:04:42.740529+0000","last_fullsized":"2024-02-19T21:04:42.740529+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.1c","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.699473+0000","last_change":"2024-02-19T21:04:39.639950+0000","last_active":"2024-02-19T21:04:42.699473+0000","last_peered":"2024-02-19T21:04:42.699473+0000","last_clean":"2024-02-19T21:04:42.699473+0000","last_became_active":"2024-02-19T21:04:39.639552+0000","last_became_peered":"2024-02-19T21:04:39.639552+0000","last_unstale":"2024-02-19T21:04:42.699473+0000","last_undegraded":"2024-02-19T21:04:42.699473+0000","last_fullsized":"2024-02-19T21:04:42.699473+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":70,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.749482+0000","last_change":"2024-02-19T21:03:32.722001+0000","last_active":"2024-02-19T21:04:43.749482+0000","last_peered":"2024-02-19T21:04:43.749482+0000","last_clean":"2024-02-19T21:04:43.749482+0000","last_became_active":"2024-02-19T21:01:18.287874+0000","last_became_peered":"2024-02-19T21:01:18.287874+0000","last_unstale":"2024-02-19T21:04:43.749482+0000","last_undegraded":"2024-02-19T21:04:43.749482+0000","last_fullsized":"2024-02-19T21:04:43.749482+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:03:32.721877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:32.721877+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":65,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.710042+0000","last_change":"2024-02-19T21:04:13.205667+0000","last_active":"2024-02-19T21:04:42.710042+0000","last_peered":"2024-02-19T21:04:42.710042+0000","last_clean":"2024-02-19T21:04:42.710042+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:04:42.710042+0000","last_undegraded":"2024-02-19T21:04:42.710042+0000","last_fullsized":"2024-02-19T21:04:42.710042+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.205596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.205596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":61,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.705287+0000","last_change":"2024-02-19T21:04:11.754594+0000","last_active":"2024-02-19T21:04:42.705287+0000","last_peered":"2024-02-19T21:04:42.705287+0000","last_clean":"2024-02-19T21:04:42.705287+0000","last_became_active":"2024-02-19T21:01:20.290962+0000","last_became_peered":"2024-02-19T21:01:20.290962+0000","last_unstale":"2024-02-19T21:04:42.705287+0000","last_undegraded":"2024-02-19T21:04:42.705287+0000","last_fullsized":"2024-02-19T21:04:42.705287+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.754525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.754525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":58,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.705647+0000","last_change":"2024-02-19T21:04:13.941207+0000","last_active":"2024-02-19T21:04:42.705647+0000","last_peered":"2024-02-19T21:04:42.705647+0000","last_clean":"2024-02-19T21:04:42.705647+0000","last_became_active":"2024-02-19T21:01:22.297693+0000","last_became_peered":"2024-02-19T21:01:22.297693+0000","last_unstale":"2024-02-19T21:04:42.705647+0000","last_undegraded":"2024-02-19T21:04:42.705647+0000","last_fullsized":"2024-02-19T21:04:42.705647+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.941132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.941132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.752316+0000","last_change":"2024-02-19T21:04:39.717968+0000","last_active":"2024-02-19T21:04:43.752316+0000","last_peered":"2024-02-19T21:04:43.752316+0000","last_clean":"2024-02-19T21:04:43.752316+0000","last_became_active":"2024-02-19T21:04:39.716606+0000","last_became_peered":"2024-02-19T21:04:39.716606+0000","last_unstale":"2024-02-19T21:04:43.752316+0000","last_undegraded":"2024-02-19T21:04:43.752316+0000","last_fullsized":"2024-02-19T21:04:43.752316+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.713994+0000","last_change":"2024-02-19T21:04:39.742917+0000","last_active":"2024-02-19T21:04:43.713994+0000","last_peered":"2024-02-19T21:04:43.713994+0000","last_clean":"2024-02-19T21:04:43.713994+0000","last_became_active":"2024-02-19T21:04:39.742440+0000","last_became_peered":"2024-02-19T21:04:39.742440+0000","last_unstale":"2024-02-19T21:04:43.713994+0000","last_undegraded":"2024-02-19T21:04:43.713994+0000","last_fullsized":"2024-02-19T21:04:43.713994+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.701333+0000","last_change":"2024-02-19T21:04:39.658952+0000","last_active":"2024-02-19T21:04:42.701333+0000","last_peered":"2024-02-19T21:04:42.701333+0000","last_clean":"2024-02-19T21:04:42.701333+0000","last_became_active":"2024-02-19T21:04:39.658671+0000","last_became_peered":"2024-02-19T21:04:39.658671+0000","last_unstale":"2024-02-19T21:04:42.701333+0000","last_undegraded":"2024-02-19T21:04:42.701333+0000","last_fullsized":"2024-02-19T21:04:42.701333+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.698269+0000","last_change":"2024-02-19T21:04:39.776864+0000","last_active":"2024-02-19T21:04:42.698269+0000","last_peered":"2024-02-19T21:04:42.698269+0000","last_clean":"2024-02-19T21:04:42.698269+0000","last_became_active":"2024-02-19T21:04:39.775205+0000","last_became_peered":"2024-02-19T21:04:39.775205+0000","last_unstale":"2024-02-19T21:04:42.698269+0000","last_undegraded":"2024-02-19T21:04:42.698269+0000","last_fullsized":"2024-02-19T21:04:42.698269+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.699777+0000","last_change":"2024-02-19T21:04:41.665203+0000","last_active":"2024-02-19T21:04:42.699777+0000","last_peered":"2024-02-19T21:04:42.699777+0000","last_clean":"2024-02-19T21:04:42.699777+0000","last_became_active":"2024-02-19T21:04:39.751921+0000","last_became_peered":"2024-02-19T21:04:39.751921+0000","last_unstale":"2024-02-19T21:04:42.699777+0000","last_undegraded":"2024-02-19T21:04:42.699777+0000","last_fullsized":"2024-02-19T21:04:42.699777+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.1e","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.718740+0000","last_change":"2024-02-19T21:04:41.648527+0000","last_active":"2024-02-19T21:04:42.718740+0000","last_peered":"2024-02-19T21:04:42.718740+0000","last_clean":"2024-02-19T21:04:42.718740+0000","last_became_active":"2024-02-19T21:04:39.664499+0000","last_became_peered":"2024-02-19T21:04:39.664499+0000","last_unstale":"2024-02-19T21:04:42.718740+0000","last_undegraded":"2024-02-19T21:04:42.718740+0000","last_fullsized":"2024-02-19T21:04:42.718740+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.1d","version":"100'7","reported_seq":22,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.784924+0000","last_change":"2024-02-19T21:04:39.782279+0000","last_active":"2024-02-19T21:04:43.784924+0000","last_peered":"2024-02-19T21:04:43.784924+0000","last_clean":"2024-02-19T21:04:43.784924+0000","last_became_active":"2024-02-19T21:04:39.781203+0000","last_became_peered":"2024-02-19T21:04:39.781203+0000","last_unstale":"2024-02-19T21:04:43.784924+0000","last_undegraded":"2024-02-19T21:04:43.784924+0000","last_fullsized":"2024-02-19T21:04:43.784924+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":78,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.741820+0000","last_change":"2024-02-19T21:02:52.859531+0000","last_active":"2024-02-19T21:04:43.741820+0000","last_peered":"2024-02-19T21:04:43.741820+0000","last_clean":"2024-02-19T21:04:43.741820+0000","last_became_active":"2024-02-19T21:01:18.293793+0000","last_became_peered":"2024-02-19T21:01:18.293793+0000","last_unstale":"2024-02-19T21:04:43.741820+0000","last_undegraded":"2024-02-19T21:04:43.741820+0000","last_fullsized":"2024-02-19T21:04:43.741820+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:02:52.859453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:52.859453+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":65,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.720520+0000","last_change":"2024-02-19T21:04:13.816617+0000","last_active":"2024-02-19T21:04:42.720520+0000","last_peered":"2024-02-19T21:04:42.720520+0000","last_clean":"2024-02-19T21:04:42.720520+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:04:42.720520+0000","last_undegraded":"2024-02-19T21:04:42.720520+0000","last_fullsized":"2024-02-19T21:04:42.720520+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.816545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.816545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":55,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.698284+0000","last_change":"2024-02-19T21:03:04.486168+0000","last_active":"2024-02-19T21:04:42.698284+0000","last_peered":"2024-02-19T21:04:42.698284+0000","last_clean":"2024-02-19T21:04:42.698284+0000","last_became_active":"2024-02-19T21:01:20.287961+0000","last_became_peered":"2024-02-19T21:01:20.287961+0000","last_unstale":"2024-02-19T21:04:42.698284+0000","last_undegraded":"2024-02-19T21:04:42.698284+0000","last_fullsized":"2024-02-19T21:04:42.698284+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.486067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.486067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":60,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.732237+0000","last_change":"2024-02-19T21:04:10.863028+0000","last_active":"2024-02-19T21:04:43.732237+0000","last_peered":"2024-02-19T21:04:43.732237+0000","last_clean":"2024-02-19T21:04:43.732237+0000","last_became_active":"2024-02-19T21:01:22.608997+0000","last_became_peered":"2024-02-19T21:01:22.608997+0000","last_unstale":"2024-02-19T21:04:43.732237+0000","last_undegraded":"2024-02-19T21:04:43.732237+0000","last_fullsized":"2024-02-19T21:04:43.732237+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:10.862926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:10.862926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.785000+0000","last_change":"2024-02-19T21:04:39.686091+0000","last_active":"2024-02-19T21:04:43.785000+0000","last_peered":"2024-02-19T21:04:43.785000+0000","last_clean":"2024-02-19T21:04:43.785000+0000","last_became_active":"2024-02-19T21:04:39.685620+0000","last_became_peered":"2024-02-19T21:04:39.685620+0000","last_unstale":"2024-02-19T21:04:43.785000+0000","last_undegraded":"2024-02-19T21:04:43.785000+0000","last_fullsized":"2024-02-19T21:04:43.785000+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2","version":"100'5","reported_seq":22,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.763281+0000","last_change":"2024-02-19T21:04:39.748398+0000","last_active":"2024-02-19T21:04:43.763281+0000","last_peered":"2024-02-19T21:04:43.763281+0000","last_clean":"2024-02-19T21:04:43.763281+0000","last_became_active":"2024-02-19T21:04:39.747725+0000","last_became_peered":"2024-02-19T21:04:39.747725+0000","last_unstale":"2024-02-19T21:04:43.763281+0000","last_undegraded":"2024-02-19T21:04:43.763281+0000","last_fullsized":"2024-02-19T21:04:43.763281+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719348+0000","last_change":"2024-02-19T21:04:39.755843+0000","last_active":"2024-02-19T21:04:42.719348+0000","last_peered":"2024-02-19T21:04:42.719348+0000","last_clean":"2024-02-19T21:04:42.719348+0000","last_became_active":"2024-02-19T21:04:39.755561+0000","last_became_peered":"2024-02-19T21:04:39.755561+0000","last_unstale":"2024-02-19T21:04:42.719348+0000","last_undegraded":"2024-02-19T21:04:42.719348+0000","last_fullsized":"2024-02-19T21:04:42.719348+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.707123+0000","last_change":"2024-02-19T21:04:39.634256+0000","last_active":"2024-02-19T21:04:42.707123+0000","last_peered":"2024-02-19T21:04:42.707123+0000","last_clean":"2024-02-19T21:04:42.707123+0000","last_became_active":"2024-02-19T21:04:39.633131+0000","last_became_peered":"2024-02-19T21:04:39.633131+0000","last_unstale":"2024-02-19T21:04:42.707123+0000","last_undegraded":"2024-02-19T21:04:42.707123+0000","last_fullsized":"2024-02-19T21:04:42.707123+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.712411+0000","last_change":"2024-02-19T21:04:41.647452+0000","last_active":"2024-02-19T21:04:42.712411+0000","last_peered":"2024-02-19T21:04:42.712411+0000","last_clean":"2024-02-19T21:04:42.712411+0000","last_became_active":"2024-02-19T21:04:39.654263+0000","last_became_peered":"2024-02-19T21:04:39.654263+0000","last_unstale":"2024-02-19T21:04:42.712411+0000","last_undegraded":"2024-02-19T21:04:42.712411+0000","last_fullsized":"2024-02-19T21:04:42.712411+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.11","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.723464+0000","last_change":"2024-02-19T21:04:41.633110+0000","last_active":"2024-02-19T21:04:42.723464+0000","last_peered":"2024-02-19T21:04:42.723464+0000","last_clean":"2024-02-19T21:04:42.723464+0000","last_became_active":"2024-02-19T21:04:39.786576+0000","last_became_peered":"2024-02-19T21:04:39.786576+0000","last_unstale":"2024-02-19T21:04:42.723464+0000","last_undegraded":"2024-02-19T21:04:42.723464+0000","last_fullsized":"2024-02-19T21:04:42.723464+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.714186+0000","last_change":"2024-02-19T21:04:39.665998+0000","last_active":"2024-02-19T21:04:42.714186+0000","last_peered":"2024-02-19T21:04:42.714186+0000","last_clean":"2024-02-19T21:04:42.714186+0000","last_became_active":"2024-02-19T21:04:39.662519+0000","last_became_peered":"2024-02-19T21:04:39.662519+0000","last_unstale":"2024-02-19T21:04:42.714186+0000","last_undegraded":"2024-02-19T21:04:42.714186+0000","last_fullsized":"2024-02-19T21:04:42.714186+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":79,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.788915+0000","last_change":"2024-02-19T21:04:04.545245+0000","last_active":"2024-02-19T21:04:43.788915+0000","last_peered":"2024-02-19T21:04:43.788915+0000","last_clean":"2024-02-19T21:04:43.788915+0000","last_became_active":"2024-02-19T21:01:18.293865+0000","last_became_peered":"2024-02-19T21:01:18.293865+0000","last_unstale":"2024-02-19T21:04:43.788915+0000","last_undegraded":"2024-02-19T21:04:43.788915+0000","last_fullsized":"2024-02-19T21:04:43.788915+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:04:04.545131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:04.545131+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":60,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.706048+0000","last_change":"2024-02-19T21:03:06.000805+0000","last_active":"2024-02-19T21:04:42.706048+0000","last_peered":"2024-02-19T21:04:42.706048+0000","last_clean":"2024-02-19T21:04:42.706048+0000","last_became_active":"2024-02-19T21:01:16.272284+0000","last_became_peered":"2024-02-19T21:01:16.272284+0000","last_unstale":"2024-02-19T21:04:42.706048+0000","last_undegraded":"2024-02-19T21:04:42.706048+0000","last_fullsized":"2024-02-19T21:04:42.706048+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.000732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.000732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":114,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.737885+0000","last_change":"2024-02-19T21:04:05.787496+0000","last_active":"2024-02-19T21:04:43.737885+0000","last_peered":"2024-02-19T21:04:43.737885+0000","last_clean":"2024-02-19T21:04:43.737885+0000","last_became_active":"2024-02-19T21:01:20.290567+0000","last_became_peered":"2024-02-19T21:01:20.290567+0000","last_unstale":"2024-02-19T21:04:43.737885+0000","last_undegraded":"2024-02-19T21:04:43.737885+0000","last_fullsized":"2024-02-19T21:04:43.737885+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:05.787415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:05.787415+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":60,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719585+0000","last_change":"2024-02-19T21:04:18.877455+0000","last_active":"2024-02-19T21:04:42.719585+0000","last_peered":"2024-02-19T21:04:42.719585+0000","last_clean":"2024-02-19T21:04:42.719585+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:04:42.719585+0000","last_undegraded":"2024-02-19T21:04:42.719585+0000","last_fullsized":"2024-02-19T21:04:42.719585+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:04:18.877385+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.727586+0000","last_change":"2024-02-19T21:04:39.700717+0000","last_active":"2024-02-19T21:04:42.727586+0000","last_peered":"2024-02-19T21:04:42.727586+0000","last_clean":"2024-02-19T21:04:42.727586+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:04:42.727586+0000","last_undegraded":"2024-02-19T21:04:42.727586+0000","last_fullsized":"2024-02-19T21:04:42.727586+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.784844+0000","last_change":"2024-02-19T21:04:39.782441+0000","last_active":"2024-02-19T21:04:43.784844+0000","last_peered":"2024-02-19T21:04:43.784844+0000","last_clean":"2024-02-19T21:04:43.784844+0000","last_became_active":"2024-02-19T21:04:39.781604+0000","last_became_peered":"2024-02-19T21:04:39.781604+0000","last_unstale":"2024-02-19T21:04:43.784844+0000","last_undegraded":"2024-02-19T21:04:43.784844+0000","last_fullsized":"2024-02-19T21:04:43.784844+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.7","version":"102'5","reported_seq":23,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.961401+0000","last_change":"2024-02-19T21:04:43.961401+0000","last_active":"2024-02-19T21:04:43.961401+0000","last_peered":"2024-02-19T21:04:43.961401+0000","last_clean":"2024-02-19T21:04:43.961401+0000","last_became_active":"2024-02-19T21:04:39.668876+0000","last_became_peered":"2024-02-19T21:04:39.668876+0000","last_unstale":"2024-02-19T21:04:43.961401+0000","last_undegraded":"2024-02-19T21:04:43.961401+0000","last_fullsized":"2024-02-19T21:04:43.961401+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'5","last_scrub_stamp":"2024-02-19T21:04:43.961315+0000","last_deep_scrub":"102'5","last_deep_scrub_stamp":"2024-02-19T21:04:43.961315+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.961315+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165832,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":27326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.735810+0000","last_change":"2024-02-19T21:04:39.645178+0000","last_active":"2024-02-19T21:04:43.735810+0000","last_peered":"2024-02-19T21:04:43.735810+0000","last_clean":"2024-02-19T21:04:43.735810+0000","last_became_active":"2024-02-19T21:04:39.644627+0000","last_became_peered":"2024-02-19T21:04:39.644627+0000","last_unstale":"2024-02-19T21:04:43.735810+0000","last_undegraded":"2024-02-19T21:04:43.735810+0000","last_fullsized":"2024-02-19T21:04:43.735810+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.706187+0000","last_change":"2024-02-19T21:04:41.640539+0000","last_active":"2024-02-19T21:04:42.706187+0000","last_peered":"2024-02-19T21:04:42.706187+0000","last_clean":"2024-02-19T21:04:42.706187+0000","last_became_active":"2024-02-19T21:04:39.751146+0000","last_became_peered":"2024-02-19T21:04:39.751146+0000","last_unstale":"2024-02-19T21:04:42.706187+0000","last_undegraded":"2024-02-19T21:04:42.706187+0000","last_fullsized":"2024-02-19T21:04:42.706187+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.10","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.704158+0000","last_change":"2024-02-19T21:04:44.708655+0000","last_active":"2024-02-19T21:04:44.704158+0000","last_peered":"2024-02-19T21:04:44.704158+0000","last_clean":"2024-02-19T21:04:44.704158+0000","last_became_active":"2024-02-19T21:04:39.686927+0000","last_became_peered":"2024-02-19T21:04:39.686927+0000","last_unstale":"2024-02-19T21:04:44.704158+0000","last_undegraded":"2024-02-19T21:04:44.704158+0000","last_fullsized":"2024-02-19T21:04:44.704158+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"8.13","version":"100'9","reported_seq":24,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.730450+0000","last_change":"2024-02-19T21:04:39.724745+0000","last_active":"2024-02-19T21:04:43.730450+0000","last_peered":"2024-02-19T21:04:43.730450+0000","last_clean":"2024-02-19T21:04:43.730450+0000","last_became_active":"2024-02-19T21:04:39.715739+0000","last_became_peered":"2024-02-19T21:04:39.715739+0000","last_unstale":"2024-02-19T21:04:43.730450+0000","last_undegraded":"2024-02-19T21:04:43.730450+0000","last_fullsized":"2024-02-19T21:04:43.730450+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":82,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.720754+0000","last_change":"2024-02-19T21:04:04.812216+0000","last_active":"2024-02-19T21:04:42.720754+0000","last_peered":"2024-02-19T21:04:42.720754+0000","last_clean":"2024-02-19T21:04:42.720754+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:04:42.720754+0000","last_undegraded":"2024-02-19T21:04:42.720754+0000","last_fullsized":"2024-02-19T21:04:42.720754+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:04:04.812144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:04.812144+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":61,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.741460+0000","last_change":"2024-02-19T21:03:21.619457+0000","last_active":"2024-02-19T21:04:43.741460+0000","last_peered":"2024-02-19T21:04:43.741460+0000","last_clean":"2024-02-19T21:04:43.741460+0000","last_became_active":"2024-02-19T21:01:16.277826+0000","last_became_peered":"2024-02-19T21:01:16.277826+0000","last_unstale":"2024-02-19T21:04:43.741460+0000","last_undegraded":"2024-02-19T21:04:43.741460+0000","last_fullsized":"2024-02-19T21:04:43.741460+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:21.619389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:21.619389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":57,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.789054+0000","last_change":"2024-02-19T21:03:08.636037+0000","last_active":"2024-02-19T21:04:43.789054+0000","last_peered":"2024-02-19T21:04:43.789054+0000","last_clean":"2024-02-19T21:04:43.789054+0000","last_became_active":"2024-02-19T21:01:20.288016+0000","last_became_peered":"2024-02-19T21:01:20.288016+0000","last_unstale":"2024-02-19T21:04:43.789054+0000","last_undegraded":"2024-02-19T21:04:43.789054+0000","last_fullsized":"2024-02-19T21:04:43.789054+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:08.635943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.635943+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":54,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.695302+0000","last_change":"2024-02-19T21:03:06.456310+0000","last_active":"2024-02-19T21:04:42.695302+0000","last_peered":"2024-02-19T21:04:42.695302+0000","last_clean":"2024-02-19T21:04:42.695302+0000","last_became_active":"2024-02-19T21:01:22.293230+0000","last_became_peered":"2024-02-19T21:01:22.293230+0000","last_unstale":"2024-02-19T21:04:42.695302+0000","last_undegraded":"2024-02-19T21:04:42.695302+0000","last_fullsized":"2024-02-19T21:04:42.695302+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.456212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.456212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.692365+0000","last_change":"2024-02-19T21:04:39.801737+0000","last_active":"2024-02-19T21:04:42.692365+0000","last_peered":"2024-02-19T21:04:42.692365+0000","last_clean":"2024-02-19T21:04:42.692365+0000","last_became_active":"2024-02-19T21:04:39.797018+0000","last_became_peered":"2024-02-19T21:04:39.797018+0000","last_unstale":"2024-02-19T21:04:42.692365+0000","last_undegraded":"2024-02-19T21:04:42.692365+0000","last_fullsized":"2024-02-19T21:04:42.692365+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.693469+0000","last_change":"2024-02-19T21:04:39.683987+0000","last_active":"2024-02-19T21:04:42.693469+0000","last_peered":"2024-02-19T21:04:42.693469+0000","last_clean":"2024-02-19T21:04:42.693469+0000","last_became_active":"2024-02-19T21:04:39.683667+0000","last_became_peered":"2024-02-19T21:04:39.683667+0000","last_unstale":"2024-02-19T21:04:42.693469+0000","last_undegraded":"2024-02-19T21:04:42.693469+0000","last_fullsized":"2024-02-19T21:04:42.693469+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":18,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.678319+0000","last_change":"2024-02-19T21:04:44.678319+0000","last_active":"2024-02-19T21:04:44.678319+0000","last_peered":"2024-02-19T21:04:44.678319+0000","last_clean":"2024-02-19T21:04:44.678319+0000","last_became_active":"2024-02-19T21:04:39.676208+0000","last_became_peered":"2024-02-19T21:04:39.676208+0000","last_unstale":"2024-02-19T21:04:44.678319+0000","last_undegraded":"2024-02-19T21:04:44.678319+0000","last_fullsized":"2024-02-19T21:04:44.678319+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:44.678223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:44.678223+0000","last_clean_scrub_stamp":"2024-02-19T21:04:44.678223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"102'5","reported_seq":21,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.693686+0000","last_change":"2024-02-19T21:04:39.765520+0000","last_active":"2024-02-19T21:04:42.693686+0000","last_peered":"2024-02-19T21:04:42.693686+0000","last_clean":"2024-02-19T21:04:42.693686+0000","last_became_active":"2024-02-19T21:04:39.761639+0000","last_became_peered":"2024-02-19T21:04:39.761639+0000","last_unstale":"2024-02-19T21:04:42.693686+0000","last_undegraded":"2024-02-19T21:04:42.693686+0000","last_fullsized":"2024-02-19T21:04:42.693686+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.728036+0000","last_change":"2024-02-19T21:04:41.637693+0000","last_active":"2024-02-19T21:04:42.728036+0000","last_peered":"2024-02-19T21:04:42.728036+0000","last_clean":"2024-02-19T21:04:42.728036+0000","last_became_active":"2024-02-19T21:04:39.761141+0000","last_became_peered":"2024-02-19T21:04:39.761141+0000","last_unstale":"2024-02-19T21:04:42.728036+0000","last_undegraded":"2024-02-19T21:04:42.728036+0000","last_fullsized":"2024-02-19T21:04:42.728036+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.13","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.703701+0000","last_change":"2024-02-19T21:04:41.664574+0000","last_active":"2024-02-19T21:04:42.703701+0000","last_peered":"2024-02-19T21:04:42.703701+0000","last_clean":"2024-02-19T21:04:42.703701+0000","last_became_active":"2024-02-19T21:04:39.663948+0000","last_became_peered":"2024-02-19T21:04:39.663948+0000","last_unstale":"2024-02-19T21:04:42.703701+0000","last_undegraded":"2024-02-19T21:04:42.703701+0000","last_fullsized":"2024-02-19T21:04:42.703701+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.10","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.697312+0000","last_change":"2024-02-19T21:04:39.613875+0000","last_active":"2024-02-19T21:04:42.697312+0000","last_peered":"2024-02-19T21:04:42.697312+0000","last_clean":"2024-02-19T21:04:42.697312+0000","last_became_active":"2024-02-19T21:04:39.613576+0000","last_became_peered":"2024-02-19T21:04:39.613576+0000","last_unstale":"2024-02-19T21:04:42.697312+0000","last_undegraded":"2024-02-19T21:04:42.697312+0000","last_fullsized":"2024-02-19T21:04:42.697312+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":88,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.727212+0000","last_change":"2024-02-19T21:04:20.874434+0000","last_active":"2024-02-19T21:04:42.727212+0000","last_peered":"2024-02-19T21:04:42.727212+0000","last_clean":"2024-02-19T21:04:42.727212+0000","last_became_active":"2024-02-19T21:01:18.289712+0000","last_became_peered":"2024-02-19T21:01:18.289712+0000","last_unstale":"2024-02-19T21:04:42.727212+0000","last_undegraded":"2024-02-19T21:04:42.727212+0000","last_fullsized":"2024-02-19T21:04:42.727212+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:04:20.874350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:20.874350+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":65,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719389+0000","last_change":"2024-02-19T21:04:21.828958+0000","last_active":"2024-02-19T21:04:42.719389+0000","last_peered":"2024-02-19T21:04:42.719389+0000","last_clean":"2024-02-19T21:04:42.719389+0000","last_became_active":"2024-02-19T21:01:16.274524+0000","last_became_peered":"2024-02-19T21:01:16.274524+0000","last_unstale":"2024-02-19T21:04:42.719389+0000","last_undegraded":"2024-02-19T21:04:42.719389+0000","last_fullsized":"2024-02-19T21:04:42.719389+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:21.828876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.828876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":56,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.695759+0000","last_change":"2024-02-19T21:02:54.544941+0000","last_active":"2024-02-19T21:04:42.695759+0000","last_peered":"2024-02-19T21:04:42.695759+0000","last_clean":"2024-02-19T21:04:42.695759+0000","last_became_active":"2024-02-19T21:01:20.290705+0000","last_became_peered":"2024-02-19T21:01:20.290705+0000","last_unstale":"2024-02-19T21:04:42.695759+0000","last_undegraded":"2024-02-19T21:04:42.695759+0000","last_fullsized":"2024-02-19T21:04:42.695759+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:54.544834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:02:54.544834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.714582+0000","last_change":"2024-02-19T21:03:12.081314+0000","last_active":"2024-02-19T21:04:43.714582+0000","last_peered":"2024-02-19T21:04:43.714582+0000","last_clean":"2024-02-19T21:04:43.714582+0000","last_became_active":"2024-02-19T21:01:22.607435+0000","last_became_peered":"2024-02-19T21:01:22.607435+0000","last_unstale":"2024-02-19T21:04:43.714582+0000","last_undegraded":"2024-02-19T21:04:43.714582+0000","last_fullsized":"2024-02-19T21:04:43.714582+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:03:12.081227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.793971+0000","last_change":"2024-02-19T21:04:39.688234+0000","last_active":"2024-02-19T21:04:43.793971+0000","last_peered":"2024-02-19T21:04:43.793971+0000","last_clean":"2024-02-19T21:04:43.793971+0000","last_became_active":"2024-02-19T21:04:39.687307+0000","last_became_peered":"2024-02-19T21:04:39.687307+0000","last_unstale":"2024-02-19T21:04:43.793971+0000","last_undegraded":"2024-02-19T21:04:43.793971+0000","last_fullsized":"2024-02-19T21:04:43.793971+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.786974+0000","last_change":"2024-02-19T21:04:39.752416+0000","last_active":"2024-02-19T21:04:43.786974+0000","last_peered":"2024-02-19T21:04:43.786974+0000","last_clean":"2024-02-19T21:04:43.786974+0000","last_became_active":"2024-02-19T21:04:39.752128+0000","last_became_peered":"2024-02-19T21:04:39.752128+0000","last_unstale":"2024-02-19T21:04:43.786974+0000","last_undegraded":"2024-02-19T21:04:43.786974+0000","last_fullsized":"2024-02-19T21:04:43.786974+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":18,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.321025+0000","last_change":"2024-02-19T21:04:43.320971+0000","last_active":"2024-02-19T21:04:43.321025+0000","last_peered":"2024-02-19T21:04:43.321025+0000","last_clean":"2024-02-19T21:04:43.321025+0000","last_became_active":"2024-02-19T21:04:39.690799+0000","last_became_peered":"2024-02-19T21:04:39.690799+0000","last_unstale":"2024-02-19T21:04:43.321025+0000","last_undegraded":"2024-02-19T21:04:43.321025+0000","last_fullsized":"2024-02-19T21:04:43.321025+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.320871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.320871+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.320871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.712877+0000","last_change":"2024-02-19T21:04:39.745803+0000","last_active":"2024-02-19T21:04:42.712877+0000","last_peered":"2024-02-19T21:04:42.712877+0000","last_clean":"2024-02-19T21:04:42.712877+0000","last_became_active":"2024-02-19T21:04:39.745448+0000","last_became_peered":"2024-02-19T21:04:39.745448+0000","last_unstale":"2024-02-19T21:04:42.712877+0000","last_undegraded":"2024-02-19T21:04:42.712877+0000","last_fullsized":"2024-02-19T21:04:42.712877+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.707203+0000","last_change":"2024-02-19T21:04:41.645251+0000","last_active":"2024-02-19T21:04:42.707203+0000","last_peered":"2024-02-19T21:04:42.707203+0000","last_clean":"2024-02-19T21:04:42.707203+0000","last_became_active":"2024-02-19T21:04:39.642891+0000","last_became_peered":"2024-02-19T21:04:39.642891+0000","last_unstale":"2024-02-19T21:04:42.707203+0000","last_undegraded":"2024-02-19T21:04:42.707203+0000","last_fullsized":"2024-02-19T21:04:42.707203+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.12","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.706045+0000","last_change":"2024-02-19T21:04:41.640394+0000","last_active":"2024-02-19T21:04:42.706045+0000","last_peered":"2024-02-19T21:04:42.706045+0000","last_clean":"2024-02-19T21:04:42.706045+0000","last_became_active":"2024-02-19T21:04:39.804223+0000","last_became_peered":"2024-02-19T21:04:39.804223+0000","last_unstale":"2024-02-19T21:04:42.706045+0000","last_undegraded":"2024-02-19T21:04:42.706045+0000","last_fullsized":"2024-02-19T21:04:42.706045+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.11","version":"100'7","reported_seq":21,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.701042+0000","last_change":"2024-02-19T21:04:39.612171+0000","last_active":"2024-02-19T21:04:42.701042+0000","last_peered":"2024-02-19T21:04:42.701042+0000","last_clean":"2024-02-19T21:04:42.701042+0000","last_became_active":"2024-02-19T21:04:39.610484+0000","last_became_peered":"2024-02-19T21:04:39.610484+0000","last_unstale":"2024-02-19T21:04:42.701042+0000","last_undegraded":"2024-02-19T21:04:42.701042+0000","last_fullsized":"2024-02-19T21:04:42.701042+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":82,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.764579+0000","last_change":"2024-02-19T21:02:57.819231+0000","last_active":"2024-02-19T21:04:43.764579+0000","last_peered":"2024-02-19T21:04:43.764579+0000","last_clean":"2024-02-19T21:04:43.764579+0000","last_became_active":"2024-02-19T21:01:18.285926+0000","last_became_peered":"2024-02-19T21:01:18.285926+0000","last_unstale":"2024-02-19T21:04:43.764579+0000","last_undegraded":"2024-02-19T21:04:43.764579+0000","last_fullsized":"2024-02-19T21:04:43.764579+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:02:57.819125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.819125+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":61,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.731434+0000","last_change":"2024-02-19T21:03:20.637528+0000","last_active":"2024-02-19T21:04:43.731434+0000","last_peered":"2024-02-19T21:04:43.731434+0000","last_clean":"2024-02-19T21:04:43.731434+0000","last_became_active":"2024-02-19T21:01:16.274811+0000","last_became_peered":"2024-02-19T21:01:16.274811+0000","last_unstale":"2024-02-19T21:04:43.731434+0000","last_undegraded":"2024-02-19T21:04:43.731434+0000","last_fullsized":"2024-02-19T21:04:43.731434+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:20.637465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:20.637465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":57,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.718758+0000","last_change":"2024-02-19T21:03:09.010420+0000","last_active":"2024-02-19T21:04:43.718758+0000","last_peered":"2024-02-19T21:04:43.718758+0000","last_clean":"2024-02-19T21:04:43.718758+0000","last_became_active":"2024-02-19T21:01:20.291672+0000","last_became_peered":"2024-02-19T21:01:20.291672+0000","last_unstale":"2024-02-19T21:04:43.718758+0000","last_undegraded":"2024-02-19T21:04:43.718758+0000","last_fullsized":"2024-02-19T21:04:43.718758+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:09.010341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:09.010341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":60,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.788164+0000","last_change":"2024-02-19T21:04:03.806998+0000","last_active":"2024-02-19T21:04:43.788164+0000","last_peered":"2024-02-19T21:04:43.788164+0000","last_clean":"2024-02-19T21:04:43.788164+0000","last_became_active":"2024-02-19T21:01:22.607099+0000","last_became_peered":"2024-02-19T21:01:22.607099+0000","last_unstale":"2024-02-19T21:04:43.788164+0000","last_undegraded":"2024-02-19T21:04:43.788164+0000","last_fullsized":"2024-02-19T21:04:43.788164+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:03.806903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:03.806903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.691698+0000","last_change":"2024-02-19T21:04:39.781223+0000","last_active":"2024-02-19T21:04:42.691698+0000","last_peered":"2024-02-19T21:04:42.691698+0000","last_clean":"2024-02-19T21:04:42.691698+0000","last_became_active":"2024-02-19T21:04:39.777442+0000","last_became_peered":"2024-02-19T21:04:39.777442+0000","last_unstale":"2024-02-19T21:04:42.691698+0000","last_undegraded":"2024-02-19T21:04:42.691698+0000","last_fullsized":"2024-02-19T21:04:42.691698+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.697309+0000","last_change":"2024-02-19T21:04:39.787206+0000","last_active":"2024-02-19T21:04:42.697309+0000","last_peered":"2024-02-19T21:04:42.697309+0000","last_clean":"2024-02-19T21:04:42.697309+0000","last_became_active":"2024-02-19T21:04:39.786881+0000","last_became_peered":"2024-02-19T21:04:39.786881+0000","last_unstale":"2024-02-19T21:04:42.697309+0000","last_undegraded":"2024-02-19T21:04:42.697309+0000","last_fullsized":"2024-02-19T21:04:42.697309+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":19,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.756846+0000","last_change":"2024-02-19T21:04:42.535665+0000","last_active":"2024-02-19T21:04:43.756846+0000","last_peered":"2024-02-19T21:04:43.756846+0000","last_clean":"2024-02-19T21:04:43.756846+0000","last_became_active":"2024-02-19T21:04:39.716211+0000","last_became_peered":"2024-02-19T21:04:39.716211+0000","last_unstale":"2024-02-19T21:04:43.756846+0000","last_undegraded":"2024-02-19T21:04:43.756846+0000","last_fullsized":"2024-02-19T21:04:43.756846+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.535597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.535597+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.535597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.727005+0000","last_change":"2024-02-19T21:04:39.787649+0000","last_active":"2024-02-19T21:04:42.727005+0000","last_peered":"2024-02-19T21:04:42.727005+0000","last_clean":"2024-02-19T21:04:42.727005+0000","last_became_active":"2024-02-19T21:04:39.787503+0000","last_became_peered":"2024-02-19T21:04:39.787503+0000","last_unstale":"2024-02-19T21:04:42.727005+0000","last_undegraded":"2024-02-19T21:04:42.727005+0000","last_fullsized":"2024-02-19T21:04:42.727005+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.729558+0000","last_change":"2024-02-19T21:04:44.730454+0000","last_active":"2024-02-19T21:04:44.729558+0000","last_peered":"2024-02-19T21:04:44.729558+0000","last_clean":"2024-02-19T21:04:44.729558+0000","last_became_active":"2024-02-19T21:04:39.742491+0000","last_became_peered":"2024-02-19T21:04:39.742491+0000","last_unstale":"2024-02-19T21:04:44.729558+0000","last_undegraded":"2024-02-19T21:04:44.729558+0000","last_fullsized":"2024-02-19T21:04:44.729558+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.15","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.738464+0000","last_change":"2024-02-19T21:04:41.655857+0000","last_active":"2024-02-19T21:04:42.738464+0000","last_peered":"2024-02-19T21:04:42.738464+0000","last_clean":"2024-02-19T21:04:42.738464+0000","last_became_active":"2024-02-19T21:04:39.696487+0000","last_became_peered":"2024-02-19T21:04:39.696487+0000","last_unstale":"2024-02-19T21:04:42.738464+0000","last_undegraded":"2024-02-19T21:04:42.738464+0000","last_fullsized":"2024-02-19T21:04:42.738464+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.16","version":"100'5","reported_seq":20,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.783929+0000","last_change":"2024-02-19T21:04:39.742946+0000","last_active":"2024-02-19T21:04:43.783929+0000","last_peered":"2024-02-19T21:04:43.783929+0000","last_clean":"2024-02-19T21:04:43.783929+0000","last_became_active":"2024-02-19T21:04:39.741473+0000","last_became_peered":"2024-02-19T21:04:39.741473+0000","last_unstale":"2024-02-19T21:04:43.783929+0000","last_undegraded":"2024-02-19T21:04:43.783929+0000","last_fullsized":"2024-02-19T21:04:43.783929+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":80,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.713826+0000","last_change":"2024-02-19T21:04:11.165275+0000","last_active":"2024-02-19T21:04:43.713826+0000","last_peered":"2024-02-19T21:04:43.713826+0000","last_clean":"2024-02-19T21:04:43.713826+0000","last_became_active":"2024-02-19T21:01:18.285269+0000","last_became_peered":"2024-02-19T21:01:18.285269+0000","last_unstale":"2024-02-19T21:04:43.713826+0000","last_undegraded":"2024-02-19T21:04:43.713826+0000","last_fullsized":"2024-02-19T21:04:43.713826+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:11.165197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.165197+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":59,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719180+0000","last_change":"2024-02-19T21:02:57.552979+0000","last_active":"2024-02-19T21:04:42.719180+0000","last_peered":"2024-02-19T21:04:42.719180+0000","last_clean":"2024-02-19T21:04:42.719180+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:04:42.719180+0000","last_undegraded":"2024-02-19T21:04:42.719180+0000","last_fullsized":"2024-02-19T21:04:42.719180+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.552886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.552886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":56,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.703844+0000","last_change":"2024-02-19T21:03:26.922649+0000","last_active":"2024-02-19T21:04:42.703844+0000","last_peered":"2024-02-19T21:04:42.703844+0000","last_clean":"2024-02-19T21:04:42.703844+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:04:42.703844+0000","last_undegraded":"2024-02-19T21:04:42.703844+0000","last_fullsized":"2024-02-19T21:04:42.703844+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:26.922572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:26.922572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.793040+0000","last_change":"2024-02-19T21:03:00.570766+0000","last_active":"2024-02-19T21:04:43.793040+0000","last_peered":"2024-02-19T21:04:43.793040+0000","last_clean":"2024-02-19T21:04:43.793040+0000","last_became_active":"2024-02-19T21:01:22.292714+0000","last_became_peered":"2024-02-19T21:01:22.292714+0000","last_unstale":"2024-02-19T21:04:43.793040+0000","last_undegraded":"2024-02-19T21:04:43.793040+0000","last_fullsized":"2024-02-19T21:04:43.793040+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:00.570677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:00.570677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.736992+0000","last_change":"2024-02-19T21:04:39.737500+0000","last_active":"2024-02-19T21:04:42.736992+0000","last_peered":"2024-02-19T21:04:42.736992+0000","last_clean":"2024-02-19T21:04:42.736992+0000","last_became_active":"2024-02-19T21:04:39.736990+0000","last_became_peered":"2024-02-19T21:04:39.736990+0000","last_unstale":"2024-02-19T21:04:42.736992+0000","last_undegraded":"2024-02-19T21:04:42.736992+0000","last_fullsized":"2024-02-19T21:04:42.736992+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.728238+0000","last_change":"2024-02-19T21:04:39.767719+0000","last_active":"2024-02-19T21:04:42.728238+0000","last_peered":"2024-02-19T21:04:42.728238+0000","last_clean":"2024-02-19T21:04:42.728238+0000","last_became_active":"2024-02-19T21:04:39.767010+0000","last_became_peered":"2024-02-19T21:04:39.767010+0000","last_unstale":"2024-02-19T21:04:42.728238+0000","last_undegraded":"2024-02-19T21:04:42.728238+0000","last_fullsized":"2024-02-19T21:04:42.728238+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3","version":"101'1003","reported_seq":2017,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.703445+0000","last_change":"2024-02-19T21:04:39.668010+0000","last_active":"2024-02-19T21:04:42.703445+0000","last_peered":"2024-02-19T21:04:42.703445+0000","last_clean":"2024-02-19T21:04:42.703445+0000","last_became_active":"2024-02-19T21:04:39.663589+0000","last_became_peered":"2024-02-19T21:04:39.663589+0000","last_unstale":"2024-02-19T21:04:42.703445+0000","last_undegraded":"2024-02-19T21:04:42.703445+0000","last_fullsized":"2024-02-19T21:04:42.703445+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1003,"ondisk_log_size":1003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1001,"num_read_kb":1,"num_write":1002,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"102'6","reported_seq":23,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.695188+0000","last_change":"2024-02-19T21:04:39.776183+0000","last_active":"2024-02-19T21:04:42.695188+0000","last_peered":"2024-02-19T21:04:42.695188+0000","last_clean":"2024-02-19T21:04:42.695188+0000","last_became_active":"2024-02-19T21:04:39.774257+0000","last_became_peered":"2024-02-19T21:04:39.774257+0000","last_unstale":"2024-02-19T21:04:42.695188+0000","last_undegraded":"2024-02-19T21:04:42.695188+0000","last_fullsized":"2024-02-19T21:04:42.695188+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"100'1","reported_seq":18,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.736835+0000","last_change":"2024-02-19T21:04:41.670631+0000","last_active":"2024-02-19T21:04:42.736835+0000","last_peered":"2024-02-19T21:04:42.736835+0000","last_clean":"2024-02-19T21:04:42.736835+0000","last_became_active":"2024-02-19T21:04:39.703845+0000","last_became_peered":"2024-02-19T21:04:39.703845+0000","last_unstale":"2024-02-19T21:04:42.736835+0000","last_undegraded":"2024-02-19T21:04:42.736835+0000","last_fullsized":"2024-02-19T21:04:42.736835+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"11.14","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.716319+0000","last_change":"2024-02-19T21:04:41.662948+0000","last_active":"2024-02-19T21:04:42.716319+0000","last_peered":"2024-02-19T21:04:42.716319+0000","last_clean":"2024-02-19T21:04:42.716319+0000","last_became_active":"2024-02-19T21:04:39.642390+0000","last_became_peered":"2024-02-19T21:04:39.642390+0000","last_unstale":"2024-02-19T21:04:42.716319+0000","last_undegraded":"2024-02-19T21:04:42.716319+0000","last_fullsized":"2024-02-19T21:04:42.716319+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.17","version":"100'5","reported_seq":19,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.733409+0000","last_change":"2024-02-19T21:04:39.746785+0000","last_active":"2024-02-19T21:04:42.733409+0000","last_peered":"2024-02-19T21:04:42.733409+0000","last_clean":"2024-02-19T21:04:42.733409+0000","last_became_active":"2024-02-19T21:04:39.746470+0000","last_became_peered":"2024-02-19T21:04:39.746470+0000","last_unstale":"2024-02-19T21:04:42.733409+0000","last_undegraded":"2024-02-19T21:04:42.733409+0000","last_fullsized":"2024-02-19T21:04:42.733409+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":74,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.707468+0000","last_change":"2024-02-19T21:04:12.133700+0000","last_active":"2024-02-19T21:04:43.707468+0000","last_peered":"2024-02-19T21:04:43.707468+0000","last_clean":"2024-02-19T21:04:43.707468+0000","last_became_active":"2024-02-19T21:01:18.285735+0000","last_became_peered":"2024-02-19T21:01:18.285735+0000","last_unstale":"2024-02-19T21:04:43.707468+0000","last_undegraded":"2024-02-19T21:04:43.707468+0000","last_fullsized":"2024-02-19T21:04:43.707468+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:04:12.133628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:12.133628+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":64,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.720494+0000","last_change":"2024-02-19T21:03:26.793343+0000","last_active":"2024-02-19T21:04:42.720494+0000","last_peered":"2024-02-19T21:04:42.720494+0000","last_clean":"2024-02-19T21:04:42.720494+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:04:42.720494+0000","last_undegraded":"2024-02-19T21:04:42.720494+0000","last_fullsized":"2024-02-19T21:04:42.720494+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:26.793263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:26.793263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":60,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.705863+0000","last_change":"2024-02-19T21:04:10.891192+0000","last_active":"2024-02-19T21:04:42.705863+0000","last_peered":"2024-02-19T21:04:42.705863+0000","last_clean":"2024-02-19T21:04:42.705863+0000","last_became_active":"2024-02-19T21:01:20.287940+0000","last_became_peered":"2024-02-19T21:01:20.287940+0000","last_unstale":"2024-02-19T21:04:42.705863+0000","last_undegraded":"2024-02-19T21:04:42.705863+0000","last_fullsized":"2024-02-19T21:04:42.705863+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:10.891113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:10.891113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":59,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.700813+0000","last_change":"2024-02-19T21:04:19.314335+0000","last_active":"2024-02-19T21:04:42.700813+0000","last_peered":"2024-02-19T21:04:42.700813+0000","last_clean":"2024-02-19T21:04:42.700813+0000","last_became_active":"2024-02-19T21:01:22.293282+0000","last_became_peered":"2024-02-19T21:01:22.293282+0000","last_unstale":"2024-02-19T21:04:42.700813+0000","last_undegraded":"2024-02-19T21:04:42.700813+0000","last_fullsized":"2024-02-19T21:04:42.700813+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:19.314241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.314241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.790887+0000","last_change":"2024-02-19T21:04:39.714784+0000","last_active":"2024-02-19T21:04:43.790887+0000","last_peered":"2024-02-19T21:04:43.790887+0000","last_clean":"2024-02-19T21:04:43.790887+0000","last_became_active":"2024-02-19T21:04:39.714495+0000","last_became_peered":"2024-02-19T21:04:39.714495+0000","last_unstale":"2024-02-19T21:04:43.790887+0000","last_undegraded":"2024-02-19T21:04:43.790887+0000","last_fullsized":"2024-02-19T21:04:43.790887+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.698478+0000","last_change":"2024-02-19T21:04:39.688286+0000","last_active":"2024-02-19T21:04:42.698478+0000","last_peered":"2024-02-19T21:04:42.698478+0000","last_clean":"2024-02-19T21:04:42.698478+0000","last_became_active":"2024-02-19T21:04:39.687980+0000","last_became_peered":"2024-02-19T21:04:39.687980+0000","last_unstale":"2024-02-19T21:04:42.698478+0000","last_undegraded":"2024-02-19T21:04:42.698478+0000","last_fullsized":"2024-02-19T21:04:42.698478+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":18,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.646674+0000","last_change":"2024-02-19T21:04:43.646643+0000","last_active":"2024-02-19T21:04:43.646674+0000","last_peered":"2024-02-19T21:04:43.646674+0000","last_clean":"2024-02-19T21:04:43.646674+0000","last_became_active":"2024-02-19T21:04:39.726486+0000","last_became_peered":"2024-02-19T21:04:39.726486+0000","last_unstale":"2024-02-19T21:04:43.646674+0000","last_undegraded":"2024-02-19T21:04:43.646674+0000","last_fullsized":"2024-02-19T21:04:43.646674+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.646543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.646543+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.646543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.719583+0000","last_change":"2024-02-19T21:04:39.688585+0000","last_active":"2024-02-19T21:04:42.719583+0000","last_peered":"2024-02-19T21:04:42.719583+0000","last_clean":"2024-02-19T21:04:42.719583+0000","last_became_active":"2024-02-19T21:04:39.688317+0000","last_became_peered":"2024-02-19T21:04:39.688317+0000","last_unstale":"2024-02-19T21:04:42.719583+0000","last_undegraded":"2024-02-19T21:04:42.719583+0000","last_fullsized":"2024-02-19T21:04:42.719583+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.733054+0000","last_change":"2024-02-19T21:04:44.733367+0000","last_active":"2024-02-19T21:04:44.733054+0000","last_peered":"2024-02-19T21:04:44.733054+0000","last_clean":"2024-02-19T21:04:44.733054+0000","last_became_active":"2024-02-19T21:04:39.745348+0000","last_became_peered":"2024-02-19T21:04:39.745348+0000","last_unstale":"2024-02-19T21:04:44.733054+0000","last_undegraded":"2024-02-19T21:04:44.733054+0000","last_fullsized":"2024-02-19T21:04:44.733054+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.17","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.733187+0000","last_change":"2024-02-19T21:04:41.651427+0000","last_active":"2024-02-19T21:04:42.733187+0000","last_peered":"2024-02-19T21:04:42.733187+0000","last_clean":"2024-02-19T21:04:42.733187+0000","last_became_active":"2024-02-19T21:04:39.691263+0000","last_became_peered":"2024-02-19T21:04:39.691263+0000","last_unstale":"2024-02-19T21:04:42.733187+0000","last_undegraded":"2024-02-19T21:04:42.733187+0000","last_fullsized":"2024-02-19T21:04:42.733187+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.14","version":"100'2","reported_seq":17,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.718958+0000","last_change":"2024-02-19T21:04:39.628834+0000","last_active":"2024-02-19T21:04:43.718958+0000","last_peered":"2024-02-19T21:04:43.718958+0000","last_clean":"2024-02-19T21:04:43.718958+0000","last_became_active":"2024-02-19T21:04:39.628525+0000","last_became_peered":"2024-02-19T21:04:39.628525+0000","last_unstale":"2024-02-19T21:04:43.718958+0000","last_undegraded":"2024-02-19T21:04:43.718958+0000","last_fullsized":"2024-02-19T21:04:43.718958+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":75,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.703833+0000","last_change":"2024-02-19T21:03:16.047345+0000","last_active":"2024-02-19T21:04:43.703833+0000","last_peered":"2024-02-19T21:04:43.703833+0000","last_clean":"2024-02-19T21:04:43.703833+0000","last_became_active":"2024-02-19T21:01:18.287645+0000","last_became_peered":"2024-02-19T21:01:18.287645+0000","last_unstale":"2024-02-19T21:04:43.703833+0000","last_undegraded":"2024-02-19T21:04:43.703833+0000","last_fullsized":"2024-02-19T21:04:43.703833+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:16.047258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:16.047258+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":66,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.784635+0000","last_change":"2024-02-19T21:04:02.781654+0000","last_active":"2024-02-19T21:04:43.784635+0000","last_peered":"2024-02-19T21:04:43.784635+0000","last_clean":"2024-02-19T21:04:43.784635+0000","last_became_active":"2024-02-19T21:01:16.272132+0000","last_became_peered":"2024-02-19T21:01:16.272132+0000","last_unstale":"2024-02-19T21:04:43.784635+0000","last_undegraded":"2024-02-19T21:04:43.784635+0000","last_fullsized":"2024-02-19T21:04:43.784635+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:02.781556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:02.781556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":61,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.691519+0000","last_change":"2024-02-19T21:04:16.276617+0000","last_active":"2024-02-19T21:04:42.691519+0000","last_peered":"2024-02-19T21:04:42.691519+0000","last_clean":"2024-02-19T21:04:42.691519+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:04:42.691519+0000","last_undegraded":"2024-02-19T21:04:42.691519+0000","last_fullsized":"2024-02-19T21:04:42.691519+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:16.276518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.276518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":60,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.713742+0000","last_change":"2024-02-19T21:04:08.178302+0000","last_active":"2024-02-19T21:04:43.713742+0000","last_peered":"2024-02-19T21:04:43.713742+0000","last_clean":"2024-02-19T21:04:43.713742+0000","last_became_active":"2024-02-19T21:01:22.293432+0000","last_became_peered":"2024-02-19T21:01:22.293432+0000","last_unstale":"2024-02-19T21:04:43.713742+0000","last_undegraded":"2024-02-19T21:04:43.713742+0000","last_fullsized":"2024-02-19T21:04:43.713742+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.178225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.178225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.699668+0000","last_change":"2024-02-19T21:04:39.662781+0000","last_active":"2024-02-19T21:04:42.699668+0000","last_peered":"2024-02-19T21:04:42.699668+0000","last_clean":"2024-02-19T21:04:42.699668+0000","last_became_active":"2024-02-19T21:04:39.661810+0000","last_became_peered":"2024-02-19T21:04:39.661810+0000","last_unstale":"2024-02-19T21:04:42.699668+0000","last_undegraded":"2024-02-19T21:04:42.699668+0000","last_fullsized":"2024-02-19T21:04:42.699668+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.769199+0000","last_change":"2024-02-19T21:04:39.741851+0000","last_active":"2024-02-19T21:04:43.769199+0000","last_peered":"2024-02-19T21:04:43.769199+0000","last_clean":"2024-02-19T21:04:43.769199+0000","last_became_active":"2024-02-19T21:04:39.738368+0000","last_became_peered":"2024-02-19T21:04:39.738368+0000","last_unstale":"2024-02-19T21:04:43.769199+0000","last_undegraded":"2024-02-19T21:04:43.769199+0000","last_fullsized":"2024-02-19T21:04:43.769199+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"103'3","reported_seq":24,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.918406+0000","last_change":"2024-02-19T21:04:42.918366+0000","last_active":"2024-02-19T21:04:42.918406+0000","last_peered":"2024-02-19T21:04:42.918406+0000","last_clean":"2024-02-19T21:04:42.918406+0000","last_became_active":"2024-02-19T21:04:39.683344+0000","last_became_peered":"2024-02-19T21:04:39.683344+0000","last_unstale":"2024-02-19T21:04:42.918406+0000","last_undegraded":"2024-02-19T21:04:42.918406+0000","last_fullsized":"2024-02-19T21:04:42.918406+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'3","last_scrub_stamp":"2024-02-19T21:04:42.918299+0000","last_deep_scrub":"103'3","last_deep_scrub_stamp":"2024-02-19T21:04:42.918299+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.918299+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.7","version":"102'3","reported_seq":18,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.739823+0000","last_change":"2024-02-19T21:04:39.640748+0000","last_active":"2024-02-19T21:04:42.739823+0000","last_peered":"2024-02-19T21:04:42.739823+0000","last_clean":"2024-02-19T21:04:42.739823+0000","last_became_active":"2024-02-19T21:04:39.640492+0000","last_became_peered":"2024-02-19T21:04:39.640492+0000","last_unstale":"2024-02-19T21:04:42.739823+0000","last_undegraded":"2024-02-19T21:04:42.739823+0000","last_fullsized":"2024-02-19T21:04:42.739823+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":22,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-02-19T21:04:44.723191+0000","last_change":"2024-02-19T21:04:44.731993+0000","last_active":"2024-02-19T21:04:44.723191+0000","last_peered":"2024-02-19T21:04:44.723191+0000","last_clean":"2024-02-19T21:04:44.723191+0000","last_became_active":"2024-02-19T21:04:39.769796+0000","last_became_peered":"2024-02-19T21:04:39.769796+0000","last_unstale":"2024-02-19T21:04:44.723191+0000","last_undegraded":"2024-02-19T21:04:44.723191+0000","last_fullsized":"2024-02-19T21:04:44.723191+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"1"},{"start":"3","length":"1"}]},{"pgid":"11.16","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.709828+0000","last_change":"2024-02-19T21:04:41.688640+0000","last_active":"2024-02-19T21:04:42.709828+0000","last_peered":"2024-02-19T21:04:42.709828+0000","last_clean":"2024-02-19T21:04:42.709828+0000","last_became_active":"2024-02-19T21:04:39.753440+0000","last_became_peered":"2024-02-19T21:04:39.753440+0000","last_unstale":"2024-02-19T21:04:42.709828+0000","last_undegraded":"2024-02-19T21:04:42.709828+0000","last_fullsized":"2024-02-19T21:04:42.709828+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"1"}]},{"pgid":"8.15","version":"0'0","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.720168+0000","last_change":"2024-02-19T21:04:39.691058+0000","last_active":"2024-02-19T21:04:42.720168+0000","last_peered":"2024-02-19T21:04:42.720168+0000","last_clean":"2024-02-19T21:04:42.720168+0000","last_became_active":"2024-02-19T21:04:39.687677+0000","last_became_peered":"2024-02-19T21:04:39.687677+0000","last_unstale":"2024-02-19T21:04:42.720168+0000","last_undegraded":"2024-02-19T21:04:42.720168+0000","last_fullsized":"2024-02-19T21:04:42.720168+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":89,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.741908+0000","last_change":"2024-02-19T21:04:14.826385+0000","last_active":"2024-02-19T21:04:43.741908+0000","last_peered":"2024-02-19T21:04:43.741908+0000","last_clean":"2024-02-19T21:04:43.741908+0000","last_became_active":"2024-02-19T21:01:18.283222+0000","last_became_peered":"2024-02-19T21:01:18.283222+0000","last_unstale":"2024-02-19T21:04:43.741908+0000","last_undegraded":"2024-02-19T21:04:43.741908+0000","last_fullsized":"2024-02-19T21:04:43.741908+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:04:14.826273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:14.826273+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":61,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.789950+0000","last_change":"2024-02-19T21:03:04.644754+0000","last_active":"2024-02-19T21:04:43.789950+0000","last_peered":"2024-02-19T21:04:43.789950+0000","last_clean":"2024-02-19T21:04:43.789950+0000","last_became_active":"2024-02-19T21:01:16.278753+0000","last_became_peered":"2024-02-19T21:01:16.278753+0000","last_unstale":"2024-02-19T21:04:43.789950+0000","last_undegraded":"2024-02-19T21:04:43.789950+0000","last_fullsized":"2024-02-19T21:04:43.789950+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.644689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.644689+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":57,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.704122+0000","last_change":"2024-02-19T21:03:05.073746+0000","last_active":"2024-02-19T21:04:43.704122+0000","last_peered":"2024-02-19T21:04:43.704122+0000","last_clean":"2024-02-19T21:04:43.704122+0000","last_became_active":"2024-02-19T21:01:20.289208+0000","last_became_peered":"2024-02-19T21:01:20.289208+0000","last_unstale":"2024-02-19T21:04:43.704122+0000","last_undegraded":"2024-02-19T21:04:43.704122+0000","last_fullsized":"2024-02-19T21:04:43.704122+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.073664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.073664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":55,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-02-19T21:04:43.730627+0000","last_change":"2024-02-19T21:03:27.633572+0000","last_active":"2024-02-19T21:04:43.730627+0000","last_peered":"2024-02-19T21:04:43.730627+0000","last_clean":"2024-02-19T21:04:43.730627+0000","last_became_active":"2024-02-19T21:01:22.609059+0000","last_became_peered":"2024-02-19T21:01:22.609059+0000","last_unstale":"2024-02-19T21:04:43.730627+0000","last_undegraded":"2024-02-19T21:04:43.730627+0000","last_fullsized":"2024-02-19T21:04:43.730627+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:27.633492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:27.633492+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.718592+0000","last_change":"2024-02-19T21:04:39.726477+0000","last_active":"2024-02-19T21:04:42.718592+0000","last_peered":"2024-02-19T21:04:42.718592+0000","last_clean":"2024-02-19T21:04:42.718592+0000","last_became_active":"2024-02-19T21:04:39.726195+0000","last_became_peered":"2024-02-19T21:04:39.726195+0000","last_unstale":"2024-02-19T21:04:42.718592+0000","last_undegraded":"2024-02-19T21:04:42.718592+0000","last_fullsized":"2024-02-19T21:04:42.718592+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-02-19T21:04:42.704523+0000","last_change":"2024-02-19T21:04:39.751308+0000","last_active":"2024-02-19T21:04:42.704523+0000","last_peered":"2024-02-19T21:04:42.704523+0000","last_clean":"2024-02-19T21:04:42.704523+0000","last_became_active":"2024-02-19T21:04:39.750531+0000","last_became_peered":"2024-02-19T21:04:39.750531+0000","last_unstale":"2024-02-19T21:04:42.704523+0000","last_undegraded":"2024-02-19T21:04:42.704523+0000","last_fullsized":"2024-02-19T21:04:42.704523+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":56,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":55,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":54,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":52,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":51,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":50,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":46,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":35,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":150998978,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":1011,"num_read_kb":49161,"num_write":1043,"num_write_kb":152978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":453267456,"data_stored":452996934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1054,"ondisk_log_size":1054,"up":96,"acting":96,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":768,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":96,"acting":96,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":41,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":96,"acting":96,"num_store_stats":8},{"poolid":40,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":202,"num_read_kb":197,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":53,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266732,"num_pgs":234,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":355516,"kb_used_data":50228,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93389636,"statfs":{"total":95995035648,"available":95630987264,"internally_reserved":0,"allocated":51433472,"data_stored":50794917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.49299999999999999,"15min":0.504},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.77400000000000002,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.78200000000000003,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50900000000000001,"15min":0.51900000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.67400000000000004,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51200000000000001,"15min":0.51800000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.46899999999999997}]},{"osd":2,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.49399999999999999,"15min":0.498},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53000000000000003,"15min":0.52400000000000002},"min":{"1min":0.39700000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73999999999999999,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.48299999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.71199999999999997,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.504,"15min":0.497},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.315}]},{"osd":4,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.33900000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73799999999999999,"5min":1.373,"15min":1.373},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.496,"15min":0.48999999999999999},"min":{"1min":0.32000000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.81100000000000005,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.34399999999999997}]},{"osd":5,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.47299999999999998,"15min":0.46500000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64900000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52400000000000002,"15min":0.51600000000000001},"min":{"1min":0.38800000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80800000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.40699999999999997}]},{"osd":6,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.54000000000000004,"15min":0.54100000000000004},"min":{"1min":0.34899999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82299999999999995,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.57899999999999996,"15min":0.57999999999999996},"min":{"1min":0.44500000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83799999999999997,"5min":1.262,"15min":1.262},"last":0.51300000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528370,"num_pgs":236,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":376244,"kb_used_data":74888,"kb_used_omap":0,"kb_used_meta":301184,"kb_avail":93368908,"statfs":{"total":95995035648,"available":95609761792,"internally_reserved":0,"allocated":76685312,"data_stored":75962273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308412416},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.495,"15min":0.49399999999999999},"min":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69399999999999995,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.52300000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.37,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.67500000000000004,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.501,"15min":0.5},"min":{"1min":0.34100000000000003,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.754,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.46300000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51200000000000001,"15min":0.51700000000000002},"min":{"1min":0.35299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72699999999999998,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.49299999999999999,"15min":0.49099999999999999},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.624,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.49399999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.41299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73899999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.51400000000000001,"15min":0.5},"min":{"1min":0.41299999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.502}]},{"osd":4,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.40999999999999998,"15min":0.38100000000000001},"min":{"1min":0.33900000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.628,"5min":1.363,"15min":1.363},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.42399999999999999,"15min":0.40600000000000003},"min":{"1min":0.311,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66400000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.36099999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:03:51 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.45000000000000001,"15min":0.41799999999999998},"min":{"1min":0.29799999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68000000000000005,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.375},{"interface":"front","average":{"1min":0.505,"5min":0.47599999999999998,"15min":0.45200000000000001},"min":{"1min":0.33600000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69399999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.48399999999999999}]},{"osd":7,"last update":"Mon Feb 19 21:04:19 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68899999999999995,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52600000000000002,"15min":0.53300000000000003},"min":{"1min":0.39600000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65900000000000003,"5min":1.534,"15min":1.534},"last":0.47199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":149,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":351548,"kb_used_data":50244,"kb_used_omap":0,"kb_used_meta":301184,"kb_avail":93393604,"statfs":{"total":95995035648,"available":95635050496,"internally_reserved":0,"allocated":51449856,"data_stored":50792771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308412416},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.36699999999999999,"15min":0.35099999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38200000000000001,"15min":0.36099999999999999},"min":{"1min":0.22800000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.316}]},{"osd":2,"last update":"Mon Feb 19 21:04:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41499999999999998,"15min":0.40999999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29199999999999998,"5min":0.255,"15min":0.255},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.34799999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:03:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.40799999999999997,"15min":0.42199999999999999},"min":{"1min":0.26400000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.438,"15min":0.45700000000000002},"min":{"1min":0.29099999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.51500000000000001,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.33600000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.56599999999999995,"15min":0.56100000000000005},"min":{"1min":0.42599999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57599999999999996,"15min":0.56599999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:04:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.47299999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60399999999999998,"15min":0.61199999999999999},"min":{"1min":0.438,"5min":0.377,"15min":0.377},"max":{"1min":0.93999999999999995,"5min":1.052,"15min":1.052},"last":0.58899999999999997}]},{"osd":6,"last update":"Mon Feb 19 21:03:49 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.58399999999999996,"15min":0.58799999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.64000000000000001},"min":{"1min":0.41299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.878,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.67000000000000004}]},{"osd":7,"last update":"Mon Feb 19 21:04:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.60299999999999998,"15min":0.59199999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.61799999999999999,"15min":0.60699999999999998},"min":{"1min":0.441,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.69399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":255,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":355652,"kb_used_data":50264,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93389500,"statfs":{"total":95995035648,"available":95630848000,"internally_reserved":0,"allocated":51470336,"data_stored":50795597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.39200000000000002,"15min":0.32200000000000001},"min":{"1min":0.24299999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.90100000000000002,"5min":4.2830000000000004,"15min":4.2830000000000004},"last":0.223},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.34699999999999998,"15min":0.313},"min":{"1min":0.22700000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.31900000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:04:20 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40999999999999998,"15min":0.38200000000000001},"min":{"1min":0.26900000000000002,"5min":0.219,"15min":0.219},"max":{"1min":1.079,"5min":3.121,"15min":3.121},"last":0.245},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.44500000000000001,"15min":0.39500000000000002},"min":{"1min":0.25700000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.3720000000000001,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.253}]},{"osd":3,"last update":"Mon Feb 19 21:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.47399999999999998,"15min":0.45200000000000001},"min":{"1min":0.28100000000000003,"5min":0.249,"15min":0.249},"max":{"1min":1.238,"5min":4.4470000000000001,"15min":4.4470000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.48499999999999999,"15min":0.46100000000000002},"min":{"1min":0.315,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.97099999999999997,"5min":4.3070000000000004,"15min":4.3070000000000004},"last":0.378}]},{"osd":4,"last update":"Mon Feb 19 21:03:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53400000000000003,"15min":0.52100000000000002},"min":{"1min":0.35399999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.83899999999999997,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55600000000000005,"15min":0.54100000000000004},"min":{"1min":0.40600000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82399999999999995,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.29199999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:04:30 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60899999999999999,"15min":0.63500000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95599999999999996,"5min":1.581,"15min":1.581},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.70899999999999996,"15min":0.80200000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88900000000000001,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":0.30399999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:03:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.36699999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72799999999999998,"5min":1.381,"15min":1.381},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.625,"15min":0.65100000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85399999999999998,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.33400000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:04:16 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.73199999999999998,"15min":0.78600000000000003},"min":{"1min":0.51100000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.875,"5min":4.226,"15min":4.226},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.45300000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.82499999999999996,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.41499999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411396,"num_pgs":240,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":327012,"kb_used_data":25680,"kb_used_omap":0,"kb_used_meta":301184,"kb_avail":93418140,"statfs":{"total":95995035648,"available":95660175360,"internally_reserved":0,"allocated":26296320,"data_stored":25628239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308412416},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.41399999999999998,"15min":0.371},"min":{"1min":0.23000000000000001,"5min":0.222,"15min":0.222},"max":{"1min":4.6150000000000002,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.42599999999999999,"15min":0.37},"min":{"1min":0.254,"5min":0.214,"15min":0.214},"max":{"1min":5.3609999999999998,"5min":5.3609999999999998,"15min":5.3609999999999998},"last":0.77000000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.42699999999999999,"15min":0.373},"min":{"1min":0.26500000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.6989999999999998,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.441},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.46100000000000002,"15min":0.41499999999999998},"min":{"1min":0.29899999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.782,"5min":4.782,"15min":4.782},"last":0.56399999999999995}]},{"osd":3,"last update":"Mon Feb 19 21:04:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.441,"15min":0.45400000000000001},"min":{"1min":0.29599999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.78900000000000003,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.49099999999999999,"15min":0.48299999999999998},"min":{"1min":0.27600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.8399999999999999,"5min":4.8399999999999999,"15min":4.8399999999999999},"last":0.505}]},{"osd":4,"last update":"Mon Feb 19 21:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.628,"15min":0.61599999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.2880000000000003,"5min":5.2880000000000003,"15min":5.2880000000000003},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59799999999999998,"15min":0.64000000000000001},"min":{"1min":0.373,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79700000000000004,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.64100000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:04:28 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60799999999999998,"15min":0.64600000000000002},"min":{"1min":0.45000000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83999999999999997,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.67900000000000005,"15min":0.66600000000000004},"min":{"1min":0.48299999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":4.7320000000000002,"5min":4.7320000000000002,"15min":4.7320000000000002},"last":0.625}]},{"osd":6,"last update":"Mon Feb 19 21:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.623,"15min":0.63},"min":{"1min":0.40000000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61699999999999999,"15min":0.63100000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90000000000000002,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.58099999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.65000000000000002,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.758,"5min":4.758,"15min":4.758},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.67200000000000004,"15min":0.628},"min":{"1min":0.51100000000000001,"5min":0.379,"15min":0.379},"max":{"1min":5.2430000000000003,"5min":5.2430000000000003,"15min":5.2430000000000003},"last":0.65800000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182464,"num_pgs":360,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":400712,"kb_used_data":99448,"kb_used_omap":0,"kb_used_meta":301184,"kb_avail":93344440,"statfs":{"total":95995035648,"available":95584706560,"internally_reserved":0,"allocated":101834752,"data_stored":101124970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308412416},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:03:41 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.34799999999999998,"15min":0.35199999999999998},"min":{"1min":0.218,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.47299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":77.662000000000006},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.377,"15min":0.39800000000000002},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.50700000000000001,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":17.456}]},{"osd":1,"last update":"Mon Feb 19 21:03:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.374,"15min":0.39200000000000002},"min":{"1min":0.23300000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.47399999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":17.327999999999999},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.38100000000000001,"15min":0.39500000000000002},"min":{"1min":0.25700000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.45700000000000002,"5min":1.054,"15min":1.054},"last":1.3280000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:03:41 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40400000000000003,"15min":0.41799999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.48399999999999999,"5min":0.625,"15min":0.625},"last":56.890999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.39400000000000002,"15min":0.41399999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.50700000000000001,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":44.600999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84299999999999997,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":1.3109999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55300000000000005,"15min":0.56499999999999995},"min":{"1min":0.41999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82499999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":26.393999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:04:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56999999999999995,"15min":0.57799999999999996},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":17.401},{"interface":"front","average":{"1min":0.623,"5min":0.56299999999999994,"15min":0.56399999999999995},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":1.343}]},{"osd":6,"last update":"Mon Feb 19 21:03:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55700000000000005,"15min":0.56699999999999995},"min":{"1min":0.41799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.69999999999999996,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":17.367000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58399999999999996,"15min":0.59199999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.35599999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56899999999999995,"15min":0.56299999999999994},"min":{"1min":0.43099999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":17.428999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57999999999999996,"15min":0.57499999999999996},"min":{"1min":0.43099999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":44.378}]}]},{"osd":4,"up_from":31,"seq":133143986236,"num_pgs":198,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302308,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":301184,"kb_avail":93442844,"statfs":{"total":95995035648,"available":95685472256,"internally_reserved":0,"allocated":1142784,"data_stored":463272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308412416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:04 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51300000000000001,"15min":0.54400000000000004},"min":{"1min":0.33700000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.505,"15min":0.52200000000000002},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.39200000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:04:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.40999999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.78900000000000003,"5min":2.149,"15min":2.149},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.499,"15min":0.51800000000000002},"min":{"1min":0.35899999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.54300000000000004}]},{"osd":2,"last update":"Mon Feb 19 21:04:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52800000000000002},"min":{"1min":0.38600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.55400000000000005},"min":{"1min":0.36599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70699999999999996,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":1.3700000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:04:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.50900000000000001,"15min":0.52300000000000002},"min":{"1min":0.33800000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63700000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":1.994},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56499999999999995,"15min":0.56999999999999995},"min":{"1min":0.40600000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68300000000000005,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":0.29599999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:04:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.43099999999999999,"15min":0.432},"min":{"1min":0.33900000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.75700000000000001,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.45600000000000002,"15min":0.44},"min":{"1min":0.378,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.82699999999999996,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.26600000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:03:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.41399999999999998,"15min":0.38200000000000001},"min":{"1min":0.38100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.46300000000000002,"15min":0.433},"min":{"1min":0.45400000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.30599999999999999}]},{"osd":7,"last update":"Mon Feb 19 21:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.46800000000000003,"15min":0.44},"min":{"1min":0.38600000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.49299999999999999,"15min":0.46700000000000003},"min":{"1min":0.41399999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.30099999999999999}]}]},{"osd":5,"up_from":37,"seq":158913790007,"num_pgs":240,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":404804,"kb_used_data":99436,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93340348,"statfs":{"total":95995035648,"available":95580516352,"internally_reserved":0,"allocated":101822464,"data_stored":101124297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.47899999999999998,"15min":0.505},"min":{"1min":0.23899999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.66600000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.437},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.52600000000000002,"15min":0.54000000000000004},"min":{"1min":0.27300000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":0.35599999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.49399999999999999,"15min":0.51600000000000001},"min":{"1min":0.308,"5min":0.248,"15min":0.248},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.53300000000000003,"15min":0.53500000000000003},"min":{"1min":0.31900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":2.9430000000000001,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":0.41099999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.50700000000000001,"15min":0.53100000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.505,"5min":0.51200000000000001,"15min":0.52900000000000003},"min":{"1min":0.34499999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71399999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.38600000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.36399999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.53500000000000003,"15min":0.54800000000000004},"min":{"1min":0.39200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.47699999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.41799999999999998,"15min":0.39500000000000002},"min":{"1min":0.28000000000000003,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.39800000000000002},"min":{"1min":0.35999999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.495}]},{"osd":6,"last update":"Mon Feb 19 21:03:47 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.442,"15min":0.42299999999999999},"min":{"1min":0.33500000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.45600000000000002,"15min":0.437},"min":{"1min":0.33400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.623,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.46100000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:04:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.50700000000000001,"15min":0.503},"min":{"1min":0.36799999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.51900000000000002,"15min":0.50900000000000001},"min":{"1min":0.38600000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50372608,"data_stored":50334515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50360320,"data_stored":50331754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25186304,"data_stored":25166874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":100712448,"data_stored":100664403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":100700160,"data_stored":100663429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":75554816,"data_stored":75500361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":50368512,"data_stored":50334560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:04:52.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:52.008 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:04:52.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:04:52.087 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:04:52.087 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-02-19T21:04:52.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-02-19T21:04:52.088 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-02-19T21:04:52.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.088 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.24400.0:89 4.4 e65/59 4:25ca39bb:::obj_delete_at_hint.0000000005:head v 65'10, mlcod=65'8)", 2024-02-19T21:04:52.088 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:01:23.357978+0000", 2024-02-19T21:04:52.088 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 208.46739923199999, 2024-02-19T21:04:52.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0043556580000000001, 2024-02-19T21:04:52.088 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.088 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:52.088 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.088 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.088 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.088 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357978+0000", 2024-02-19T21:04:52.088 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357978+0000", 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357977+0000", 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3670000000000002e-06 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357983+0000", 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0269999999999995e-06 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357991+0000", 2024-02-19T21:04:52.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.634e-06 2024-02-19T21:04:52.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.357996+0000", 2024-02-19T21:04:52.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5027000000000001e-05 2024-02-19T21:04:52.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.358041+0000", 2024-02-19T21:04:52.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5488000000000001e-05 2024-02-19T21:04:52.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.358056+0000", 2024-02-19T21:04:52.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0042191870000000001 2024-02-19T21:04:52.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:52.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.362276+0000", 2024-02-19T21:04:52.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7707999999999997e-05 2024-02-19T21:04:52.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:01:23.362333+0000", 2024-02-19T21:04:52.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0043556580000000001 2024-02-19T21:04:52.092 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.092 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.092 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.092 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDScrubReserve(6.0 REQUEST e91)", 2024-02-19T21:04:52.092 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:06.590908+0000", 2024-02-19T21:04:52.092 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 45.234468731, 2024-02-19T21:04:52.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004846281, 2024-02-19T21:04:52.092 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.092 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:04:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:06.590908+0000", 2024-02-19T21:04:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:06.590908+0000", 2024-02-19T21:04:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999962 2024-02-19T21:04:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:06.590904+0000", 2024-02-19T21:04:52.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3129000000000001e-05 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:06.590917+0000", 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8680000000000001e-05 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:06.590936+0000", 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6016e-05 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:06.590962+0000", 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047290980000000002 2024-02-19T21:04:52.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:06.595691+0000", 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1963e-05 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:06.595713+0000", 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1347999999999998e-05 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:06.595754+0000", 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004846281 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.095 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.096 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.096 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.096 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.25213.0:2 34.6 e103/102 34:602f83fe:::foo:head v 103'1, mlcod=0'0)", 2024-02-19T21:04:52.096 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:42.752171+0000", 2024-02-19T21:04:52.096 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 9.0732058710000008, 2024-02-19T21:04:52.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045573469999999998, 2024-02-19T21:04:52.096 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.096 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:52.096 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.096 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.096 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.096 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.752171+0000", 2024-02-19T21:04:52.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.752171+0000", 2024-02-19T21:04:52.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:04:52.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.752170+0000", 2024-02-19T21:04:52.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.742e-06 2024-02-19T21:04:52.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.752177+0000", 2024-02-19T21:04:52.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2539999999999998e-06 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.752183+0000", 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7440000000000001e-06 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.752190+0000", 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112834 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.752303+0000", 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7543000000000001e-05 2024-02-19T21:04:52.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.752321+0000", 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004373816 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.756694+0000", 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3976999999999997e-05 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.756728+0000", 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045573469999999998 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.099 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.100 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.100 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.100 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.35189.0:5 36.12 36:4a6d8757:::3.obj:head [write 0~115] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-19T21:04:52.100 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:43.796288+0000", 2024-02-19T21:04:52.100 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.0290884160000004, 2024-02-19T21:04:52.100 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023332470000000001, 2024-02-19T21:04:52.100 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.100 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:52.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:52.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.35189", 2024-02-19T21:04:52.100 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3441459623", 2024-02-19T21:04:52.100 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5 2024-02-19T21:04:52.100 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.100 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796288+0000", 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796288+0000", 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796286+0000", 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9190000000000005e-06 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796295+0000", 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0984e-05 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796306+0000", 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9120000000000001e-06 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796313+0000", 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2296999999999999e-05 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.102 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796345+0000", 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9303999999999998e-05 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796424+0000", 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1744999999999997e-05 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,6", 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796516+0000", 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00058284700000000005 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.797099+0000", 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000270282 2024-02-19T21:04:52.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:52.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.797369+0000", 2024-02-19T21:04:52.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022161777000000001 2024-02-19T21:04:52.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:52.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.819531+0000", 2024-02-19T21:04:52.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.621e-05 2024-02-19T21:04:52.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:52.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.819577+0000", 2024-02-19T21:04:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3671000000000003e-05 2024-02-19T21:04:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.819621+0000", 2024-02-19T21:04:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023332470000000001 2024-02-19T21:04:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.35189.0:5 36.12 e104/102 ondisk, result = 0)", 2024-02-19T21:04:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:43.814409+0000", 2024-02-19T21:04:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.0109674789999996, 2024-02-19T21:04:52.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052169069999999998, 2024-02-19T21:04:52.106 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.106 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:04:52.106 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.106 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.814409+0000", 2024-02-19T21:04:52.106 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.106 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.814409+0000", 2024-02-19T21:04:52.106 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:52.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.814408+0000", 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0460000000000006e-06 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.814416+0000", 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6629999999999999e-06 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.814424+0000", 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4790000000000001e-06 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.814431+0000", 2024-02-19T21:04:52.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050842049999999996 2024-02-19T21:04:52.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.108 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.108 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.108 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.819515+0000", 2024-02-19T21:04:52.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4449999999999996e-06 2024-02-19T21:04:52.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.108 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.108 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.108 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.819524+0000", 2024-02-19T21:04:52.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000102687 2024-02-19T21:04:52.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.108 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.108 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.109 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.819626+0000", 2024-02-19T21:04:52.109 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0052169069999999998 2024-02-19T21:04:52.109 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.109 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.109 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.109 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.109 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.109 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(42.5s1 107/106 ECSubWrite(tid=1161, reqid=client.25228.0:1, at_version=107'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-19T21:04:52.109 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:46.975696+0000", 2024-02-19T21:04:52.109 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.8496803870000003, 2024-02-19T21:04:52.109 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0071007759999999996, 2024-02-19T21:04:52.109 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.109 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:04:52.109 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.109 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.975696+0000", 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.975696+0000", 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.975696+0000", 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3676000000000002e-05 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.975720+0000", 2024-02-19T21:04:52.111 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0539999999999997e-06 2024-02-19T21:04:52.111 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.111 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.111 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.111 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.975727+0000", 2024-02-19T21:04:52.111 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3839999999999999e-06 2024-02-19T21:04:52.111 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.111 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.111 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.111 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.975734+0000", 2024-02-19T21:04:52.111 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019610069999999999 2024-02-19T21:04:52.111 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.111 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.112 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.112 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.977695+0000", 2024-02-19T21:04:52.112 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4430000000000003e-06 2024-02-19T21:04:52.112 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.112 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.112 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:04:52.112 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.977704+0000", 2024-02-19T21:04:52.112 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050767909999999998 2024-02-19T21:04:52.112 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.112 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.112 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:04:52.112 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.982780+0000", 2024-02-19T21:04:52.112 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6790999999999999e-05 2024-02-19T21:04:52.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.113 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.113 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.113 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.982797+0000", 2024-02-19T21:04:52.113 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0071007759999999996 2024-02-19T21:04:52.113 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.113 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.113 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.113 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.113 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25228.0:13 42.2s0 42:4f5df15e:test-rados-api-smithi163-83960-10::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-19T21:04:52.113 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:47.009271+0000", 2024-02-19T21:04:52.113 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.816106306, 2024-02-19T21:04:52.114 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0022695760000000001, 2024-02-19T21:04:52.114 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.114 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:52.114 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:52.114 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25228", 2024-02-19T21:04:52.114 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/516814289", 2024-02-19T21:04:52.114 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 13 2024-02-19T21:04:52.114 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.114 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.114 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.115 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.115 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.009271+0000", 2024-02-19T21:04:52.115 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.115 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.115 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.115 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.009271+0000", 2024-02-19T21:04:52.115 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:04:52.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.115 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.115 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.115 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.009269+0000", 2024-02-19T21:04:52.115 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8399999999999995e-06 2024-02-19T21:04:52.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.115 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.116 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.116 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.009276+0000", 2024-02-19T21:04:52.116 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3859999999999999e-05 2024-02-19T21:04:52.116 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.116 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.116 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.116 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.009290+0000", 2024-02-19T21:04:52.116 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8040000000000004e-06 2024-02-19T21:04:52.116 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.116 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.116 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.116 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.009297+0000", 2024-02-19T21:04:52.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5513000000000002e-05 2024-02-19T21:04:52.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.117 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.117 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.009353+0000", 2024-02-19T21:04:52.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5414000000000007e-05 2024-02-19T21:04:52.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.117 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.117 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.009428+0000", 2024-02-19T21:04:52.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00025075199999999997 2024-02-19T21:04:52.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:04:52.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.009679+0000", 2024-02-19T21:04:52.118 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010510669999999999 2024-02-19T21:04:52.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.118 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:04:52.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.010730+0000", 2024-02-19T21:04:52.118 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00075270600000000001 2024-02-19T21:04:52.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.118 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:52.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.011482+0000", 2024-02-19T21:04:52.118 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7645000000000001e-05 2024-02-19T21:04:52.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.011540+0000", 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0022695760000000001 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25267.0:32 43.7s0 43:e38eb4e4:test-rados-api-smithi163-83872-31::foo:head [read 0~16384] snapc 0=[] ondisk+read+known_if_redirected e107)", 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:47.057029+0000", 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.7683480320000005, 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0022911559999999999, 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:04:52.119 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:52.120 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25267", 2024-02-19T21:04:52.120 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2108766762", 2024-02-19T21:04:52.120 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 32 2024-02-19T21:04:52.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.120 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.120 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.120 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.120 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.057029+0000", 2024-02-19T21:04:52.120 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.120 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.120 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.123 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.057029+0000", 2024-02-19T21:04:52.123 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:04:52.123 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.123 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.123 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.124 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.057028+0000", 2024-02-19T21:04:52.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1840000000000001e-06 2024-02-19T21:04:52.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.124 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.124 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.124 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.057032+0000", 2024-02-19T21:04:52.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3229999999999998e-06 2024-02-19T21:04:52.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.124 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.124 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.124 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.057040+0000", 2024-02-19T21:04:52.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9899999999999997e-06 2024-02-19T21:04:52.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.057045+0000", 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4602000000000003e-05 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.057110+0000", 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0427999999999997e-05 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.057180+0000", 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021395920000000001 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.059320+0000", 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0022911559999999999 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25267.0:61 43.2s0 43:4b7572a5:test-rados-api-smithi163-83872-39::foo:head [cmpext 0~4,writefull 0~4] snapc 0=[] ondisk+write+known_if_redirected e107)", 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:47.097906+0000", 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.7274710390000001, 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0038113259999999999, 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:52.126 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25267", 2024-02-19T21:04:52.127 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2108766762", 2024-02-19T21:04:52.127 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 61 2024-02-19T21:04:52.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.127 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.127 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.127 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.127 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.097906+0000", 2024-02-19T21:04:52.127 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.127 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.127 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.127 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.097906+0000", 2024-02-19T21:04:52.127 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:52.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.128 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.128 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.128 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.097905+0000", 2024-02-19T21:04:52.128 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2700000000000004e-06 2024-02-19T21:04:52.128 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.128 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.128 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.128 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.097910+0000", 2024-02-19T21:04:52.128 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2710000000000003e-06 2024-02-19T21:04:52.128 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.128 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.128 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.128 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.097917+0000", 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2229999999999998e-06 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.097922+0000", 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4975000000000003e-05 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.097957+0000", 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1472999999999999e-05 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.098019+0000", 2024-02-19T21:04:52.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0036983200000000002 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.101717+0000", 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0038113259999999999 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24865.0:14 11.18 11:1d8f7fe2:test-rados-api-smithi163-83952-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:47.966832+0000", 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.8585444459999998, 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040096484000000002, 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:52.130 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24865", 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3187889786", 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 14 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.966832+0000", 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.966832+0000", 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.966830+0000", 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7490000000000004e-06 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.966839+0000", 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0332e-05 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.966849+0000", 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1670000000000002e-06 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.966857+0000", 2024-02-19T21:04:52.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0093742600000000006 2024-02-19T21:04:52.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.976231+0000", 2024-02-19T21:04:52.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013006500000000001 2024-02-19T21:04:52.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.976361+0000", 2024-02-19T21:04:52.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000153556 2024-02-19T21:04:52.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,6", 2024-02-19T21:04:52.134 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.976514+0000", 2024-02-19T21:04:52.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00053995200000000001 2024-02-19T21:04:52.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.134 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:52.134 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.977054+0000", 2024-02-19T21:04:52.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085838589999999992 2024-02-19T21:04:52.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.134 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:52.134 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.985638+0000", 2024-02-19T21:04:52.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.021197450999999999 2024-02-19T21:04:52.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.006836+0000", 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0127000000000003e-05 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.006886+0000", 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3053999999999998e-05 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.006929+0000", 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040096484000000002 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.135 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.136 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.136 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.136 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.0.0:234 25.1f e108/99 25:f80a0358:test-rados-api-smithi163-83944-4::foo:5 v 108'6, mlcod=107'3)", 2024-02-19T21:04:52.136 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:48.002564+0000", 2024-02-19T21:04:52.136 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.8228129200000001, 2024-02-19T21:04:52.136 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078267389999999992, 2024-02-19T21:04:52.136 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.136 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:52.136 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.136 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.136 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.136 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.002564+0000", 2024-02-19T21:04:52.136 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.136 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.002564+0000", 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.002563+0000", 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2843999999999999e-05 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.002576+0000", 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7490000000000001e-06 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.138 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.002583+0000", 2024-02-19T21:04:52.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.17e-06 2024-02-19T21:04:52.138 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.138 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.002590+0000", 2024-02-19T21:04:52.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001072284 2024-02-19T21:04:52.138 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.138 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.003662+0000", 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1444e-05 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.003684+0000", 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0066861209999999997 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.010370+0000", 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0769000000000001e-05 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.010391+0000", 2024-02-19T21:04:52.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078267389999999992 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.4.0:261 40.6 e108/103 40:602f83fe:::foo:head v 108'1, mlcod=0'0)", 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:48.004914+0000", 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.8204624410000001, 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054180260000000003, 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.004914+0000", 2024-02-19T21:04:52.140 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.141 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.141 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.141 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.141 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.004914+0000", 2024-02-19T21:04:52.141 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:04:52.141 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.141 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.141 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.141 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.004914+0000", 2024-02-19T21:04:52.141 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5709999999999997e-06 2024-02-19T21:04:52.141 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.141 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.141 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.141 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.004919+0000", 2024-02-19T21:04:52.141 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5010000000000004e-06 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.004925+0000", 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.758e-06 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.004930+0000", 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012828010000000001 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.006212+0000", 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7334e-05 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.142 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.143 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.006230+0000", 2024-02-19T21:04:52.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0040766589999999998 2024-02-19T21:04:52.143 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.143 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.143 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:52.143 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.010306+0000", 2024-02-19T21:04:52.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6061999999999999e-05 2024-02-19T21:04:52.143 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.143 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.143 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.143 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.010332+0000", 2024-02-19T21:04:52.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054180260000000003 2024-02-19T21:04:52.144 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.144 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.144 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.144 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.144 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.144 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.25105.0:3 40.6 e108/103 40:602f83fe:::foo:head v 108'2, mlcod=0'0)", 2024-02-19T21:04:52.144 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:48.006594+0000", 2024-02-19T21:04:52.144 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.8187831829999999, 2024-02-19T21:04:52.144 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003763845, 2024-02-19T21:04:52.144 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.144 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:52.144 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.144 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.144 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.145 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.006594+0000", 2024-02-19T21:04:52.145 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.145 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.145 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.145 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.145 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.006594+0000", 2024-02-19T21:04:52.145 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:04:52.145 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.145 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.145 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.145 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.006593+0000", 2024-02-19T21:04:52.145 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5679999999999999e-05 2024-02-19T21:04:52.146 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.146 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.146 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.146 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.006609+0000", 2024-02-19T21:04:52.146 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2280000000000003e-06 2024-02-19T21:04:52.146 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.146 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.146 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.146 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.006615+0000", 2024-02-19T21:04:52.146 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7110000000000003e-06 2024-02-19T21:04:52.146 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.146 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.146 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.146 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.006620+0000", 2024-02-19T21:04:52.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000568349 2024-02-19T21:04:52.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.147 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.147 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.147 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.007188+0000", 2024-02-19T21:04:52.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8467000000000001e-05 2024-02-19T21:04:52.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.147 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.147 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.147 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.007207+0000", 2024-02-19T21:04:52.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031291729999999999 2024-02-19T21:04:52.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.147 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.147 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:52.148 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.010336+0000", 2024-02-19T21:04:52.148 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.175e-05 2024-02-19T21:04:52.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.148 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.148 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.148 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.010358+0000", 2024-02-19T21:04:52.148 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003763845 2024-02-19T21:04:52.148 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.148 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.148 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.148 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.148 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.4.31:3 32.6 e108/101 32:602f83fe:::foo:head v 108'1, mlcod=0'0)", 2024-02-19T21:04:52.148 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:48.013221+0000", 2024-02-19T21:04:52.149 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.812155991, 2024-02-19T21:04:52.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0043013260000000003, 2024-02-19T21:04:52.149 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.149 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:52.149 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.149 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.149 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.013221+0000", 2024-02-19T21:04:52.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.149 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.149 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.013221+0000", 2024-02-19T21:04:52.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:04:52.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.150 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.150 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.013220+0000", 2024-02-19T21:04:52.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3834e-05 2024-02-19T21:04:52.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.150 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.150 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.013234+0000", 2024-02-19T21:04:52.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3900000000000004e-06 2024-02-19T21:04:52.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.013241+0000", 2024-02-19T21:04:52.151 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2369999999999997e-06 2024-02-19T21:04:52.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.013248+0000", 2024-02-19T21:04:52.151 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017947800000000001 2024-02-19T21:04:52.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.013427+0000", 2024-02-19T21:04:52.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4666e-05 2024-02-19T21:04:52.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.013442+0000", 2024-02-19T21:04:52.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0040470130000000003 2024-02-19T21:04:52.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:52.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.017489+0000", 2024-02-19T21:04:52.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3529999999999999e-05 2024-02-19T21:04:52.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.017522+0000", 2024-02-19T21:04:52.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0043013260000000003 2024-02-19T21:04:52.153 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.153 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.153 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.153 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(50.0s1 109/108 ECSubWrite(tid=1206, reqid=client.14940.0:1, at_version=109'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-19T21:04:52.153 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:48.596468+0000", 2024-02-19T21:04:52.153 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.2289091220000001, 2024-02-19T21:04:52.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.006883335, 2024-02-19T21:04:52.153 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.153 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:04:52.154 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.596468+0000", 2024-02-19T21:04:52.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.596468+0000", 2024-02-19T21:04:52.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:04:52.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.596467+0000", 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3238000000000003e-05 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.596501+0000", 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7759999999999992e-06 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.596510+0000", 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.25e-06 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.596518+0000", 2024-02-19T21:04:52.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00077985200000000004 2024-02-19T21:04:52.156 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.156 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.156 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.156 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.597297+0000", 2024-02-19T21:04:52.156 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0900000000000001e-05 2024-02-19T21:04:52.156 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.156 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.156 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:04:52.156 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.597308+0000", 2024-02-19T21:04:52.156 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060169710000000003 2024-02-19T21:04:52.156 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.156 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.156 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:04:52.156 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.603325+0000", 2024-02-19T21:04:52.156 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5812999999999998e-05 2024-02-19T21:04:52.157 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.157 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.157 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.157 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.603351+0000", 2024-02-19T21:04:52.157 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.006883335 2024-02-19T21:04:52.157 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.157 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.157 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.157 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.157 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.157 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(51.0s2 109/108 ECSubWrite(tid=1207, reqid=client.25294.0:3, at_version=109'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-19T21:04:52.157 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:48.596600+0000", 2024-02-19T21:04:52.157 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.2287763749999998, 2024-02-19T21:04:52.157 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0067591600000000002, 2024-02-19T21:04:52.157 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.158 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:04:52.158 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.158 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.158 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.158 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.596600+0000", 2024-02-19T21:04:52.158 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.158 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.158 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.158 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.158 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.596600+0000", 2024-02-19T21:04:52.158 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:04:52.158 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.158 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.158 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.158 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.596600+0000", 2024-02-19T21:04:52.159 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3665999999999999e-05 2024-02-19T21:04:52.159 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.159 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.159 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.159 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.596614+0000", 2024-02-19T21:04:52.159 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8669999999999996e-06 2024-02-19T21:04:52.159 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.159 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.159 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.159 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.596620+0000", 2024-02-19T21:04:52.159 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9309999999999996e-06 2024-02-19T21:04:52.159 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.159 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.159 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.159 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.596626+0000", 2024-02-19T21:04:52.160 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00095589899999999996 2024-02-19T21:04:52.160 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.160 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.160 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.160 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.597582+0000", 2024-02-19T21:04:52.160 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2729999999999997e-06 2024-02-19T21:04:52.160 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.160 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.160 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:04:52.160 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.597590+0000", 2024-02-19T21:04:52.160 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057651739999999996 2024-02-19T21:04:52.160 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.160 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.160 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:04:52.160 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.603355+0000", 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9389999999999996e-06 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.603360+0000", 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0067591600000000002 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25351.0:2 56.15 56:ad551702:::test:head [create,write 0~4,writefull 0~2] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:50.578524+0000", 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.2468528679999999, 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040752544000000002, 2024-02-19T21:04:52.161 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25351", 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/581384461", 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.578524+0000", 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.578524+0000", 2024-02-19T21:04:52.162 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:04:52.163 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.163 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.163 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.163 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.578522+0000", 2024-02-19T21:04:52.163 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1309999999999995e-06 2024-02-19T21:04:52.163 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.163 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.163 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.163 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.578531+0000", 2024-02-19T21:04:52.163 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0318e-05 2024-02-19T21:04:52.163 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.163 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.163 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.163 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.578541+0000", 2024-02-19T21:04:52.164 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1940000000000001e-06 2024-02-19T21:04:52.164 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.164 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.164 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.164 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.578548+0000", 2024-02-19T21:04:52.164 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001524381 2024-02-19T21:04:52.164 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.164 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.164 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.164 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.580073+0000", 2024-02-19T21:04:52.164 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6190000000000002e-06 2024-02-19T21:04:52.164 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.164 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.165 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:04:52.165 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.580080+0000", 2024-02-19T21:04:52.165 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020204755000000001 2024-02-19T21:04:52.165 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.165 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.165 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.165 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.600285+0000", 2024-02-19T21:04:52.165 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110561 2024-02-19T21:04:52.165 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.165 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.165 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.165 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.600396+0000", 2024-02-19T21:04:52.165 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000158414 2024-02-19T21:04:52.166 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.166 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.166 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-02-19T21:04:52.166 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.600554+0000", 2024-02-19T21:04:52.166 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0043549599999999997 2024-02-19T21:04:52.166 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.166 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.166 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:52.166 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.604909+0000", 2024-02-19T21:04:52.166 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014248733 2024-02-19T21:04:52.166 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.166 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.166 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:52.166 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.619158+0000", 2024-02-19T21:04:52.166 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7781999999999999e-05 2024-02-19T21:04:52.167 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.167 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.167 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:52.167 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.619186+0000", 2024-02-19T21:04:52.167 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7846999999999998e-05 2024-02-19T21:04:52.167 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.167 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.167 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:52.167 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.619234+0000", 2024-02-19T21:04:52.167 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3025999999999998e-05 2024-02-19T21:04:52.167 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.167 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.167 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.167 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.619277+0000", 2024-02-19T21:04:52.167 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040752544000000002 2024-02-19T21:04:52.168 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.168 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.168 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.168 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.168 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.168 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.25351.0:2 56.15 e111/110 ondisk, result = 0)", 2024-02-19T21:04:52.168 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:50.610545+0000", 2024-02-19T21:04:52.168 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.2148322629999999, 2024-02-19T21:04:52.168 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.008615724, 2024-02-19T21:04:52.168 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.168 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:04:52.168 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.168 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.169 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.169 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.610545+0000", 2024-02-19T21:04:52.169 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.169 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.169 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.169 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.169 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.610545+0000", 2024-02-19T21:04:52.169 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:04:52.169 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.169 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.169 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.169 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.610544+0000", 2024-02-19T21:04:52.169 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6090000000000001e-06 2024-02-19T21:04:52.169 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.169 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.610548+0000", 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1970000000000004e-06 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.610553+0000", 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7710000000000001e-06 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.610560+0000", 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085829930000000006 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.170 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.171 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.619143+0000", 2024-02-19T21:04:52.171 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8229999999999995e-06 2024-02-19T21:04:52.171 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.171 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.171 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.171 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.619151+0000", 2024-02-19T21:04:52.171 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8199999999999992e-06 2024-02-19T21:04:52.171 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.171 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.171 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.171 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.619160+0000", 2024-02-19T21:04:52.171 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.008615724 2024-02-19T21:04:52.171 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.172 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.172 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.172 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.172 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.172 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(client.25351.0:2 56.15 e111/110 ondisk, result = 0)", 2024-02-19T21:04:52.172 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:50.614387+0000", 2024-02-19T21:04:52.172 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.2109900790000001, 2024-02-19T21:04:52.172 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0048953759999999999, 2024-02-19T21:04:52.172 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.172 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:04:52.172 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.172 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.172 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.172 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.614387+0000", 2024-02-19T21:04:52.173 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.173 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.173 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.173 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.173 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.614387+0000", 2024-02-19T21:04:52.173 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:04:52.173 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.173 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.173 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.173 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.614386+0000", 2024-02-19T21:04:52.173 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8700000000000002e-06 2024-02-19T21:04:52.173 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.173 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.173 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.174 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.614390+0000", 2024-02-19T21:04:52.174 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6230000000000002e-06 2024-02-19T21:04:52.174 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.174 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.174 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.174 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.614396+0000", 2024-02-19T21:04:52.174 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5611999999999999e-05 2024-02-19T21:04:52.174 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.174 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.174 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.174 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.614411+0000", 2024-02-19T21:04:52.174 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047650499999999998 2024-02-19T21:04:52.174 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.174 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.175 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.175 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.619176+0000", 2024-02-19T21:04:52.175 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8189999999999999e-06 2024-02-19T21:04:52.175 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.175 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.175 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.175 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.619181+0000", 2024-02-19T21:04:52.175 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000100926 2024-02-19T21:04:52.175 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.175 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.175 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.175 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.619282+0000", 2024-02-19T21:04:52.175 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0048953759999999999 2024-02-19T21:04:52.175 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.175 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.176 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.176 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.176 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.176 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25351.0:4 56.15 56:ad551702:::test:head [writefull 0~4 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-19T21:04:52.176 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:50.623662+0000", 2024-02-19T21:04:52.176 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.2017152659999999, 2024-02-19T21:04:52.176 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013390582, 2024-02-19T21:04:52.176 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:04:52.176 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25351", 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/581384461", 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 4 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.623662+0000", 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.623662+0000", 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:04:52.177 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.623660+0000", 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8069999999999996e-06 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.623666+0000", 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032233899999999999 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.623988+0000", 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1519999999999992e-06 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.178 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.623997+0000", 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0017157310000000001 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.625712+0000", 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6019000000000005e-05 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.625788+0000", 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012908099999999999 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.179 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-02-19T21:04:52.180 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.625917+0000", 2024-02-19T21:04:52.180 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0013898300000000001 2024-02-19T21:04:52.180 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.180 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.180 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:52.180 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.627307+0000", 2024-02-19T21:04:52.180 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054649970000000001 2024-02-19T21:04:52.180 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.180 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.180 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:04:52.180 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.632772+0000", 2024-02-19T21:04:52.180 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0042053749999999999 2024-02-19T21:04:52.180 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.180 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.180 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.636978+0000", 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8717000000000002e-05 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.637016+0000", 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5842e-05 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.637052+0000", 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013390582 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.181 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:04:52.182 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:04:52.182 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:04:52.182 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:04:52.233 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:52.283 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:04:52.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:52.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:04:52.388 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:04:52.422 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:52.675 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:04:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:52.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:04:52.708 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-19T21:04:52.780 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:04:52.806 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:52.824 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:04:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:52.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:04:52.928 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:52.980 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:52.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:50.969467+0000 mgr.y (mgr.14550) 156 : cluster [DBG] pgmap v138: 600 pgs: 100 creating+peering, 1 active+clean+snaptrim, 499 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 5.5 KiB/s wr, 10 op/s 2024-02-19T21:04:52.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:51.013496+0000 osd.3 (osd.3) 79 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:04:52.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:51.014241+0000 osd.3 (osd.3) 80 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:04:52.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:51.564731+0000 mon.a (mon.0) 939 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1"}]': finished 2024-02-19T21:04:52.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:51.564805+0000 mon.a (mon.0) 940 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]': finished 2024-02-19T21:04:52.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:51.564977+0000 mon.a (mon.0) 941 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-19T21:04:52.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:51.569732+0000 mon.a (mon.0) 942 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:52.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:51.569764+0000 mon.c (mon.2) 73 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "mode": "writeback"}]: dispatch 2024-02-19T21:04:52.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:51.570106+0000 mon.a (mon.0) 943 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "mode": "writeback"}]: dispatch 2024-02-19T21:04:52.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:51.575825+0000 mon.c (mon.2) 74 : audit [INF] from='client.? v1:172.21.15.163:0/1504385913' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi163-83944-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:52.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:51.576753+0000 mon.a (mon.0) 944 : audit [INF] from='client.35258 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi163-83944-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:52.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:51.594210+0000 mon.b (mon.1) 160 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:52.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:51.597690+0000 mon.a (mon.0) 945 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:52.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:51.689019+0000 mon.c (mon.2) 75 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:52.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.227319+0000 mon.a (mon.0) 946 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:52.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.247103+0000 mon.c (mon.2) 76 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:52.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.396825+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:52.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.531087+0000 mon.a (mon.0) 947 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:52.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.531893+0000 mon.c (mon.2) 78 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:52.982 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.533453+0000 mon.a (mon.0) 948 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:52.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:50.969467+0000 mgr.y (mgr.14550) 156 : cluster [DBG] pgmap v138: 600 pgs: 100 creating+peering, 1 active+clean+snaptrim, 499 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 5.5 KiB/s wr, 10 op/s 2024-02-19T21:04:52.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:51.013496+0000 osd.3 (osd.3) 79 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:04:52.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:51.014241+0000 osd.3 (osd.3) 80 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:04:52.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:51.564731+0000 mon.a (mon.0) 939 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1"}]': finished 2024-02-19T21:04:52.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:51.564805+0000 mon.a (mon.0) 940 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]': finished 2024-02-19T21:04:52.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:51.564977+0000 mon.a (mon.0) 941 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-19T21:04:52.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:51.569732+0000 mon.a (mon.0) 942 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:52.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:51.569764+0000 mon.c (mon.2) 73 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "mode": "writeback"}]: dispatch 2024-02-19T21:04:52.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:51.570106+0000 mon.a (mon.0) 943 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "mode": "writeback"}]: dispatch 2024-02-19T21:04:52.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:51.575825+0000 mon.c (mon.2) 74 : audit [INF] from='client.? v1:172.21.15.163:0/1504385913' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi163-83944-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:52.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:51.576753+0000 mon.a (mon.0) 944 : audit [INF] from='client.35258 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi163-83944-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:52.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:51.594210+0000 mon.b (mon.1) 160 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:52.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:51.597690+0000 mon.a (mon.0) 945 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:52.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:51.689019+0000 mon.c (mon.2) 75 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:52.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.227319+0000 mon.a (mon.0) 946 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:52.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.247103+0000 mon.c (mon.2) 76 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:52.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.396825+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:52.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.531087+0000 mon.a (mon.0) 947 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:52.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.531893+0000 mon.c (mon.2) 78 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:52.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.533453+0000 mon.a (mon.0) 948 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:53.024 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:53.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:53.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:50.969467+0000 mgr.y (mgr.14550) 156 : cluster [DBG] pgmap v138: 600 pgs: 100 creating+peering, 1 active+clean+snaptrim, 499 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 5.5 KiB/s wr, 10 op/s 2024-02-19T21:04:53.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:51.013496+0000 osd.3 (osd.3) 79 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:04:53.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:51.014241+0000 osd.3 (osd.3) 80 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:04:53.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:51.564731+0000 mon.a (mon.0) 939 : audit [INF] from='client.35207 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi163-83912-1","app":"app1","key":"key1"}]': finished 2024-02-19T21:04:53.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:51.564805+0000 mon.a (mon.0) 940 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]': finished 2024-02-19T21:04:53.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:51.564977+0000 mon.a (mon.0) 941 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-02-19T21:04:53.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:51.569732+0000 mon.a (mon.0) 942 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:53.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:51.569764+0000 mon.c (mon.2) 73 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "mode": "writeback"}]: dispatch 2024-02-19T21:04:53.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:51.570106+0000 mon.a (mon.0) 943 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "mode": "writeback"}]: dispatch 2024-02-19T21:04:53.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:51.575825+0000 mon.c (mon.2) 74 : audit [INF] from='client.? v1:172.21.15.163:0/1504385913' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi163-83944-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:53.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:51.576753+0000 mon.a (mon.0) 944 : audit [INF] from='client.35258 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi163-83944-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:53.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:51.594210+0000 mon.b (mon.1) 160 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:53.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:51.597690+0000 mon.a (mon.0) 945 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:53.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:51.689019+0000 mon.c (mon.2) 75 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:53.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.227319+0000 mon.a (mon.0) 946 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:53.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.247103+0000 mon.c (mon.2) 76 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:53.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.396825+0000 mon.c (mon.2) 77 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:53.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.531087+0000 mon.a (mon.0) 947 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:53.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.531893+0000 mon.c (mon.2) 78 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:53.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.533453+0000 mon.a (mon.0) 948 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:04:53.128 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:53.155 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:53.174 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:53.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:53.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:53.278 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:53.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.341+0000 7f36ed62d700 1 -- 172.21.15.163:0/1788508146 >> v1:172.21.15.193:6789/0 conn(0x7f36e80faff0 legacy=0x7f36e810cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:53.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.341+0000 7f36ed62d700 1 -- 172.21.15.163:0/1788508146 shutdown_connections 2024-02-19T21:04:53.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.341+0000 7f36ed62d700 1 -- 172.21.15.163:0/1788508146 >> 172.21.15.163:0/1788508146 conn(0x7f36e806fdc0 msgr2=0x7f36e80f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:53.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.341+0000 7f36ed62d700 1 -- 172.21.15.163:0/1788508146 shutdown_connections 2024-02-19T21:04:53.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.341+0000 7f36ed62d700 1 -- 172.21.15.163:0/1788508146 wait complete. 2024-02-19T21:04:53.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.342+0000 7f36ed62d700 1 librados: starting msgr at 2024-02-19T21:04:53.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.342+0000 7f36ed62d700 1 librados: starting objecter 2024-02-19T21:04:53.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.342+0000 7f36ed62d700 1 Processor -- start 2024-02-19T21:04:53.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.342+0000 7f36ed62d700 1 -- start start 2024-02-19T21:04:53.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.342+0000 7f36ed62d700 1 librados: setting wanted keys 2024-02-19T21:04:53.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.342+0000 7f36ed62d700 1 librados: calling monclient init 2024-02-19T21:04:53.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.342+0000 7f36ed62d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f36e8106370 con 0x7f36e80fa430 2024-02-19T21:04:53.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.342+0000 7f36ed62d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f36e8106500 con 0x7f36e80fc780 2024-02-19T21:04:53.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.342+0000 7f36ed62d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f36e81066d0 con 0x7f36e80faff0 2024-02-19T21:04:53.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.342+0000 7f36e67fc700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f36e80faff0 0x7f36e810b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:56972/0 (socket says 172.21.15.163:56972) 2024-02-19T21:04:53.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36e67fc700 1 -- 172.21.15.163:0/3521777757 learned_addr learned my addr 172.21.15.163:0/3521777757 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:04:53.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 10 client.?.objecter ms_handle_connect 0x7f36e80faff0 2024-02-19T21:04:53.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:53.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 10 client.?.objecter ms_handle_connect 0x7f36e80fc780 2024-02-19T21:04:53.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:53.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405280606 0 0) 0x7f36dc0010b0 con 0x7f36e80faff0 2024-02-19T21:04:53.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d0002220 con 0x7f36e80faff0 2024-02-19T21:04:53.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107214388 0 0) 0x7f36e8106500 con 0x7f36e80fc780 2024-02-19T21:04:53.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36dc0010b0 con 0x7f36e80fc780 2024-02-19T21:04:53.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2797644754 0 0) 0x7f36dc0010b0 con 0x7f36e80fc780 2024-02-19T21:04:53.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36e8106500 con 0x7f36e80fc780 2024-02-19T21:04:53.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f36d80017f0 con 0x7f36e80fc780 2024-02-19T21:04:53.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 995375291 0 0) 0x7f36d0002220 con 0x7f36e80faff0 2024-02-19T21:04:53.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36dc0010b0 con 0x7f36e80faff0 2024-02-19T21:04:53.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f36dc001770 con 0x7f36e80faff0 2024-02-19T21:04:53.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3791599243 0 0) 0x7f36dc0010b0 con 0x7f36e80faff0 2024-02-19T21:04:53.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 >> v1:172.21.15.193:6789/0 conn(0x7f36e80fc780 legacy=0x7f36e8124eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:53.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 >> v1:172.21.15.163:6789/0 conn(0x7f36e80fa430 legacy=0x7f36e8105b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:53.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e8129660 con 0x7f36e80faff0 2024-02-19T21:04:53.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e8126630 con 0x7f36e80faff0 2024-02-19T21:04:53.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36ed62d700 10 client.35933.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:53.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e8126aa0 con 0x7f36e80faff0 2024-02-19T21:04:53.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.343+0000 7f36ed62d700 1 librados: init done 2024-02-19T21:04:53.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.344+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f36dc001ca0 con 0x7f36e80faff0 2024-02-19T21:04:53.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.344+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f36e81026c0 con 0x7f36e80faff0 2024-02-19T21:04:53.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.344+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f36dc005a40 con 0x7f36e80faff0 2024-02-19T21:04:53.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.346+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f36dc0047d0 con 0x7f36e80faff0 2024-02-19T21:04:53.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.346+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(113..113 src has 43..113) v4 ==== 12006+0+0 (unknown 2011647627 0 0) 0x7f36dc079af0 con 0x7f36e80faff0 2024-02-19T21:04:53.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.346+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 osd_map(113..113 src has 43..113) v4 2024-02-19T21:04:53.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.346+0000 7f36cffff700 3 client.35933.objecter handle_osd_map got epochs [113,113] > 0 2024-02-19T21:04:53.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.346+0000 7f36cffff700 3 client.35933.objecter handle_osd_map decoding full epoch 113 2024-02-19T21:04:53.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.346+0000 7f36cffff700 20 client.35933.objecter dump_active .. 0 homeless 2024-02-19T21:04:53.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.346+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v113) v1 ==== 54+0+16 (unknown 891576468 0 2420000192) 0x7f36dc079e60 con 0x7f36e80faff0 2024-02-19T21:04:53.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.347+0000 7f36cffff700 10 client.35933.objecter ms_handle_connect 0x7f36d0063c10 2024-02-19T21:04:53.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.347+0000 7f36ed62d700 10 client.35933.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:53.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.347+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 1 2024-02-19T21:04:53.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.347+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e805f700 osd=0 3 2024-02-19T21:04:53.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.347+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e805f700 osd=0 3 2024-02-19T21:04:53.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.347+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e805f700 osd=0 3 2024-02-19T21:04:53.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.347+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 1 2024-02-19T21:04:53.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.347+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e805f700 osd=0 3 2024-02-19T21:04:53.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.347+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 0 1 2024-02-19T21:04:53.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.347+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e805f700 osd=0 4 2024-02-19T21:04:53.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.347+0000 7f36ed62d700 10 client.35933.objecter _send_command 1 2024-02-19T21:04:53.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.347+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36e80f7040 con 0x7f36e80598f0 2024-02-19T21:04:53.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.347+0000 7f36cffff700 10 client.35933.objecter ms_handle_connect 0x7f36e80598f0 2024-02-19T21:04:53.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.348+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f36e80f7040 con 0x7f36e80598f0 2024-02-19T21:04:53.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.348+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:04:53.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.348+0000 7f36cffff700 10 client.35933.objecter _finish_command 1 = system:0 2024-02-19T21:04:53.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.348+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e805f700 osd=0 4 2024-02-19T21:04:53.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.348+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 0 1 2024-02-19T21:04:53.376 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:53.380 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:04:53.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.382+0000 7f36ed62d700 10 client.35933.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:53.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.382+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 2 2024-02-19T21:04:53.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.382+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e805f700 osd=0 3 2024-02-19T21:04:53.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.382+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e805f700 osd=0 3 2024-02-19T21:04:53.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.382+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e805f700 osd=0 3 2024-02-19T21:04:53.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.382+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 2 2024-02-19T21:04:53.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.382+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e805f700 osd=0 3 2024-02-19T21:04:53.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.382+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 0 2 2024-02-19T21:04:53.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.382+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e805f700 osd=0 4 2024-02-19T21:04:53.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.382+0000 7f36ed62d700 10 client.35933.objecter _send_command 2 2024-02-19T21:04:53.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.382+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f36e8047540 con 0x7f36e80598f0 2024-02-19T21:04:53.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.385+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f36e8047540 con 0x7f36e80598f0 2024-02-19T21:04:53.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.385+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:53.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.385+0000 7f36cffff700 10 client.35933.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.385+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e805f700 osd=0 4 2024-02-19T21:04:53.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.385+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 0 2 2024-02-19T21:04:53.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.386+0000 7f36ed62d700 10 client.35933.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:53.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.386+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 3 2024-02-19T21:04:53.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.386+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e810f3f0 osd=1 3 2024-02-19T21:04:53.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.386+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e810f3f0 osd=1 3 2024-02-19T21:04:53.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.386+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e810f3f0 osd=1 3 2024-02-19T21:04:53.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.386+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 3 2024-02-19T21:04:53.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.386+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e810f3f0 osd=1 3 2024-02-19T21:04:53.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.386+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 1 3 2024-02-19T21:04:53.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.386+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e810f3f0 osd=1 4 2024-02-19T21:04:53.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.386+0000 7f36ed62d700 10 client.35933.objecter _send_command 3 2024-02-19T21:04:53.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.386+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36e812d3c0 con 0x7f36e81298b0 2024-02-19T21:04:53.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.386+0000 7f36cffff700 10 client.35933.objecter ms_handle_connect 0x7f36e81298b0 2024-02-19T21:04:53.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.399+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f36e812d3c0 con 0x7f36e81298b0 2024-02-19T21:04:53.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.399+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:04:53.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.399+0000 7f36cffff700 10 client.35933.objecter _finish_command 3 = system:0 2024-02-19T21:04:53.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.399+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e810f3f0 osd=1 4 2024-02-19T21:04:53.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.399+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 1 3 2024-02-19T21:04:53.407 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:53.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:53.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.440+0000 7f36ed62d700 10 client.35933.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:53.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.441+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 4 2024-02-19T21:04:53.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.441+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e810f3f0 osd=1 3 2024-02-19T21:04:53.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.441+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e810f3f0 osd=1 3 2024-02-19T21:04:53.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.441+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e810f3f0 osd=1 3 2024-02-19T21:04:53.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.441+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 4 2024-02-19T21:04:53.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.441+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e810f3f0 osd=1 3 2024-02-19T21:04:53.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.441+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 1 4 2024-02-19T21:04:53.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.441+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e810f3f0 osd=1 4 2024-02-19T21:04:53.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.441+0000 7f36ed62d700 10 client.35933.objecter _send_command 4 2024-02-19T21:04:53.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.441+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f36e8059360 con 0x7f36e81298b0 2024-02-19T21:04:53.435 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:53.445 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:04:53.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.446+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f36e8059360 con 0x7f36e81298b0 2024-02-19T21:04:53.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.446+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:53.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.446+0000 7f36cffff700 10 client.35933.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.446+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e810f3f0 osd=1 4 2024-02-19T21:04:53.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.446+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 1 4 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.446+0000 7f36ed62d700 10 client.35933.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.446+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 5 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.446+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e80f7bd0 osd=2 3 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.446+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e80f7bd0 osd=2 3 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.446+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e80f7bd0 osd=2 3 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.446+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 5 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.446+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e80f7bd0 osd=2 3 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.446+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 2 5 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.446+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e80f7bd0 osd=2 4 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.446+0000 7f36ed62d700 10 client.35933.objecter _send_command 5 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.447+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36e80f8250 con 0x7f36e812e6b0 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.447+0000 7f36cffff700 10 client.35933.objecter ms_handle_connect 0x7f36e812e6b0 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.448+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f36e80f8250 con 0x7f36e812e6b0 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.448+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.448+0000 7f36cffff700 10 client.35933.objecter _finish_command 5 = system:0 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.448+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e80f7bd0 osd=2 4 2024-02-19T21:04:53.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.448+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 2 5 2024-02-19T21:04:53.450 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:53.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.464+0000 7f36ed62d700 10 client.35933.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:53.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.464+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 6 2024-02-19T21:04:53.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.464+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e80f7bd0 osd=2 3 2024-02-19T21:04:53.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.464+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e80f7bd0 osd=2 3 2024-02-19T21:04:53.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.464+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e80f7bd0 osd=2 3 2024-02-19T21:04:53.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.464+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 6 2024-02-19T21:04:53.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.464+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e80f7bd0 osd=2 3 2024-02-19T21:04:53.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.464+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 2 6 2024-02-19T21:04:53.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.464+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e80f7bd0 osd=2 4 2024-02-19T21:04:53.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.464+0000 7f36ed62d700 10 client.35933.objecter _send_command 6 2024-02-19T21:04:53.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.464+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f36e80f83f0 con 0x7f36e812e6b0 2024-02-19T21:04:53.471 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:04:53.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.467+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f36e80f83f0 con 0x7f36e812e6b0 2024-02-19T21:04:53.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.467+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:53.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.467+0000 7f36cffff700 10 client.35933.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.467+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e80f7bd0 osd=2 4 2024-02-19T21:04:53.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.467+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 2 6 2024-02-19T21:04:53.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.467+0000 7f36ed62d700 10 client.35933.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:53.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.467+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 7 2024-02-19T21:04:53.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.468+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e80f7880 osd=3 3 2024-02-19T21:04:53.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.468+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e80f7880 osd=3 3 2024-02-19T21:04:53.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.468+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e80f7880 osd=3 3 2024-02-19T21:04:53.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.468+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 7 2024-02-19T21:04:53.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.468+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e80f7880 osd=3 3 2024-02-19T21:04:53.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.468+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 3 7 2024-02-19T21:04:53.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.468+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e80f7880 osd=3 4 2024-02-19T21:04:53.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.468+0000 7f36ed62d700 10 client.35933.objecter _send_command 7 2024-02-19T21:04:53.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.468+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36e80f79f0 con 0x7f36e812e210 2024-02-19T21:04:53.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.468+0000 7f36cffff700 10 client.35933.objecter ms_handle_connect 0x7f36e812e210 2024-02-19T21:04:53.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.470+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f36e80f79f0 con 0x7f36e812e210 2024-02-19T21:04:53.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.470+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:04:53.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.470+0000 7f36cffff700 10 client.35933.objecter _finish_command 7 = system:0 2024-02-19T21:04:53.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.470+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e80f7880 osd=3 4 2024-02-19T21:04:53.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.470+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 3 7 2024-02-19T21:04:53.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.497+0000 7f36ed62d700 10 client.35933.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:53.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.497+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 8 2024-02-19T21:04:53.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.497+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e80f7880 osd=3 3 2024-02-19T21:04:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.497+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e80f7880 osd=3 3 2024-02-19T21:04:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.497+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e80f7880 osd=3 3 2024-02-19T21:04:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.497+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 8 2024-02-19T21:04:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.497+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e80f7880 osd=3 3 2024-02-19T21:04:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.497+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 3 8 2024-02-19T21:04:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.497+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e80f7880 osd=3 4 2024-02-19T21:04:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.497+0000 7f36ed62d700 10 client.35933.objecter _send_command 8 2024-02-19T21:04:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.497+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f36e812d770 con 0x7f36e812e210 2024-02-19T21:04:53.508 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:04:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f36e812d770 con 0x7f36e812e210 2024-02-19T21:04:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36cffff700 10 client.35933.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e80f7880 osd=3 4 2024-02-19T21:04:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 3 8 2024-02-19T21:04:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36ed62d700 10 client.35933.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 9 2024-02-19T21:04:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e812d530 osd=4 3 2024-02-19T21:04:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e812d530 osd=4 3 2024-02-19T21:04:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e812d530 osd=4 3 2024-02-19T21:04:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 9 2024-02-19T21:04:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e812d530 osd=4 3 2024-02-19T21:04:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 4 9 2024-02-19T21:04:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e812d530 osd=4 4 2024-02-19T21:04:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36ed62d700 10 client.35933.objecter _send_command 9 2024-02-19T21:04:53.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.515+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36e812d6a0 con 0x7f36e8132ea0 2024-02-19T21:04:53.514 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:53.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.524+0000 7f36cffff700 10 client.35933.objecter ms_handle_connect 0x7f36e8132ea0 2024-02-19T21:04:53.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.524+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f36e812d6a0 con 0x7f36e8132ea0 2024-02-19T21:04:53.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.524+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:04:53.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.524+0000 7f36cffff700 10 client.35933.objecter _finish_command 9 = system:0 2024-02-19T21:04:53.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.524+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e812d530 osd=4 4 2024-02-19T21:04:53.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.524+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 4 9 2024-02-19T21:04:53.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.546+0000 7f36ed62d700 10 client.35933.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:53.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.546+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 10 2024-02-19T21:04:53.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.546+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e812d530 osd=4 3 2024-02-19T21:04:53.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.546+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e812d530 osd=4 3 2024-02-19T21:04:53.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.546+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e812d530 osd=4 3 2024-02-19T21:04:53.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.546+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 10 2024-02-19T21:04:53.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.546+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e812d530 osd=4 3 2024-02-19T21:04:53.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.546+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 4 10 2024-02-19T21:04:53.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.546+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e812d530 osd=4 4 2024-02-19T21:04:53.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.546+0000 7f36ed62d700 10 client.35933.objecter _send_command 10 2024-02-19T21:04:53.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.546+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f36e812d6a0 con 0x7f36e8132ea0 2024-02-19T21:04:53.544 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:04:53.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.547+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f36e812d6a0 con 0x7f36e8132ea0 2024-02-19T21:04:53.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.547+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:53.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.547+0000 7f36cffff700 10 client.35933.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.547+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e812d530 osd=4 4 2024-02-19T21:04:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.547+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 4 10 2024-02-19T21:04:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.549+0000 7f36ed62d700 10 client.35933.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.549+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 11 2024-02-19T21:04:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.549+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e812d6a0 osd=5 3 2024-02-19T21:04:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.549+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e812d6a0 osd=5 3 2024-02-19T21:04:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.549+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e812d6a0 osd=5 3 2024-02-19T21:04:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.549+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 11 2024-02-19T21:04:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.549+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e812d6a0 osd=5 3 2024-02-19T21:04:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.549+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 5 11 2024-02-19T21:04:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.549+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e812d6a0 osd=5 4 2024-02-19T21:04:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.549+0000 7f36ed62d700 10 client.35933.objecter _send_command 11 2024-02-19T21:04:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.549+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36e813d560 con 0x7f36e813b870 2024-02-19T21:04:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.550+0000 7f36cffff700 10 client.35933.objecter ms_handle_connect 0x7f36e813b870 2024-02-19T21:04:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.551+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f36e813d560 con 0x7f36e813b870 2024-02-19T21:04:53.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.551+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:04:53.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.551+0000 7f36cffff700 10 client.35933.objecter _finish_command 11 = system:0 2024-02-19T21:04:53.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.551+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e812d6a0 osd=5 4 2024-02-19T21:04:53.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.551+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 5 11 2024-02-19T21:04:53.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.577+0000 7f36ed62d700 10 client.35933.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:53.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.577+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 12 2024-02-19T21:04:53.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.577+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e812d6a0 osd=5 3 2024-02-19T21:04:53.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.577+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e812d6a0 osd=5 3 2024-02-19T21:04:53.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.577+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e812d6a0 osd=5 3 2024-02-19T21:04:53.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.577+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 12 2024-02-19T21:04:53.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.577+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e812d6a0 osd=5 3 2024-02-19T21:04:53.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.577+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 5 12 2024-02-19T21:04:53.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.577+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e812d6a0 osd=5 4 2024-02-19T21:04:53.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.577+0000 7f36ed62d700 10 client.35933.objecter _send_command 12 2024-02-19T21:04:53.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.577+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f36e813d560 con 0x7f36e813b870 2024-02-19T21:04:53.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.587+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f36e813d560 con 0x7f36e813b870 2024-02-19T21:04:53.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.587+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:53.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.587+0000 7f36cffff700 10 client.35933.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.587+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e812d6a0 osd=5 4 2024-02-19T21:04:53.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.587+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 5 12 2024-02-19T21:04:53.583 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:04:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.589+0000 7f36ed62d700 10 client.35933.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.589+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 13 2024-02-19T21:04:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.589+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e813d560 osd=6 3 2024-02-19T21:04:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.589+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e813d560 osd=6 3 2024-02-19T21:04:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.589+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e813d560 osd=6 3 2024-02-19T21:04:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.589+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 13 2024-02-19T21:04:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.589+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e813d560 osd=6 3 2024-02-19T21:04:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.589+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 6 13 2024-02-19T21:04:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.589+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e813d560 osd=6 4 2024-02-19T21:04:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.589+0000 7f36ed62d700 10 client.35933.objecter _send_command 13 2024-02-19T21:04:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.589+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36e8143b00 con 0x7f36e813fff0 2024-02-19T21:04:53.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.590+0000 7f36cffff700 10 client.35933.objecter ms_handle_connect 0x7f36e813fff0 2024-02-19T21:04:53.585 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:53.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: Running main() from gmock_main.cc 2024-02-19T21:04:53.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-02-19T21:04:53.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [----------] Global test environment set-up. 2024-02-19T21:04:53.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-02-19T21:04:53.585 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (268 ms) 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLock.LockShared (12 ms) 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1044 ms) 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1037 ms) 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (17 ms) 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLock.Unlock (14 ms) 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (10 ms) 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (15 ms) 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [----------] 8 tests from LibRadosLock (2417 ms total) 2024-02-19T21:04:53.586 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (635 ms) 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (13 ms) 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1005 ms) 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1026 ms) 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (8 ms) 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (4 ms) 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (5 ms) 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-02-19T21:04:53.587 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (3 ms) 2024-02-19T21:04:53.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (2699 ms total) 2024-02-19T21:04:53.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: 2024-02-19T21:04:53.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [----------] Global test environment tear-down 2024-02-19T21:04:53.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (20763 ms total) 2024-02-19T21:04:53.588 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock: [ PASSED ] 16 tests. 2024-02-19T21:04:53.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.591+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f36e8143b00 con 0x7f36e813fff0 2024-02-19T21:04:53.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.591+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:04:53.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.591+0000 7f36cffff700 10 client.35933.objecter _finish_command 13 = system:0 2024-02-19T21:04:53.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.591+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e813d560 osd=6 4 2024-02-19T21:04:53.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.591+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 6 13 2024-02-19T21:04:53.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.621+0000 7f36ed62d700 10 client.35933.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:53.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.621+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 14 2024-02-19T21:04:53.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.621+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e813d560 osd=6 3 2024-02-19T21:04:53.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.621+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e813d560 osd=6 3 2024-02-19T21:04:53.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.621+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e813d560 osd=6 3 2024-02-19T21:04:53.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.621+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 14 2024-02-19T21:04:53.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.621+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e813d560 osd=6 3 2024-02-19T21:04:53.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.621+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 6 14 2024-02-19T21:04:53.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.621+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e813d560 osd=6 4 2024-02-19T21:04:53.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.621+0000 7f36ed62d700 10 client.35933.objecter _send_command 14 2024-02-19T21:04:53.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.621+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f36e8143b00 con 0x7f36e813fff0 2024-02-19T21:04:53.620 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:04:53.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.626+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f36e8143b00 con 0x7f36e813fff0 2024-02-19T21:04:53.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.626+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:53.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.626+0000 7f36cffff700 10 client.35933.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.626+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e813d560 osd=6 4 2024-02-19T21:04:53.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.626+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 6 14 2024-02-19T21:04:53.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.626+0000 7f36ed62d700 10 client.35933.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:04:53.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.627+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 15 2024-02-19T21:04:53.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.627+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e8143b00 osd=7 3 2024-02-19T21:04:53.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.627+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e8143b00 osd=7 3 2024-02-19T21:04:53.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.627+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e8143b00 osd=7 3 2024-02-19T21:04:53.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.627+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 15 2024-02-19T21:04:53.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.627+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e8143b00 osd=7 3 2024-02-19T21:04:53.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.627+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 7 15 2024-02-19T21:04:53.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.627+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e8143b00 osd=7 4 2024-02-19T21:04:53.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.627+0000 7f36ed62d700 10 client.35933.objecter _send_command 15 2024-02-19T21:04:53.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.627+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36e8145e10 con 0x7f36e8144180 2024-02-19T21:04:53.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.629+0000 7f36cffff700 10 client.35933.objecter ms_handle_connect 0x7f36e8144180 2024-02-19T21:04:53.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.629+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f36e8145e10 con 0x7f36e8144180 2024-02-19T21:04:53.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.629+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:04:53.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.629+0000 7f36cffff700 10 client.35933.objecter _finish_command 15 = system:0 2024-02-19T21:04:53.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.629+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e8143b00 osd=7 4 2024-02-19T21:04:53.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.629+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 7 15 2024-02-19T21:04:53.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.658+0000 7f36ed62d700 10 client.35933.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:04:53.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.658+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign -1 16 2024-02-19T21:04:53.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.658+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e8143b00 osd=7 3 2024-02-19T21:04:53.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.658+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e8143b00 osd=7 3 2024-02-19T21:04:53.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.658+0000 7f36ed62d700 20 client.35933.objecter _get_session s=0x7f36e8143b00 osd=7 3 2024-02-19T21:04:53.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.658+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_remove -1 16 2024-02-19T21:04:53.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.658+0000 7f36ed62d700 20 client.35933.objecter get_session s=0x7f36e8143b00 osd=7 3 2024-02-19T21:04:53.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.658+0000 7f36ed62d700 15 client.35933.objecter _session_command_op_assign 7 16 2024-02-19T21:04:53.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.658+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e8143b00 osd=7 4 2024-02-19T21:04:53.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.658+0000 7f36ed62d700 10 client.35933.objecter _send_command 16 2024-02-19T21:04:53.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.658+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f36e8145e10 con 0x7f36e8144180 2024-02-19T21:04:53.657 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:04:53.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.662+0000 7f36cffff700 1 -- 172.21.15.163:0/3521777757 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f36e8145e10 con 0x7f36e8144180 2024-02-19T21:04:53.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.662+0000 7f36cffff700 10 client.35933.objecter ms_dispatch 0x7f36e8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:04:53.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.662+0000 7f36cffff700 10 client.35933.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.662+0000 7f36cffff700 20 client.35933.objecter put_session s=0x7f36e8143b00 osd=7 4 2024-02-19T21:04:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.662+0000 7f36cffff700 15 client.35933.objecter _session_command_op_remove 7 16 2024-02-19T21:04:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 10 librados: watch_flush enter 2024-02-19T21:04:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 10 librados: watch_flush exit 2024-02-19T21:04:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 10 client.35933.objecter close_session for osd.0 2024-02-19T21:04:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 >> v1:172.21.15.163:6801/1559418115 conn(0x7f36e80598f0 legacy=0x7f36e8128da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e805f700 osd=0 1 2024-02-19T21:04:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 10 client.35933.objecter close_session for osd.1 2024-02-19T21:04:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 >> v1:172.21.15.163:6805/3125047432 conn(0x7f36e81298b0 legacy=0x7f36e812bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e810f3f0 osd=1 1 2024-02-19T21:04:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 10 client.35933.objecter close_session for osd.2 2024-02-19T21:04:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 >> v1:172.21.15.163:6809/1563158487 conn(0x7f36e812e6b0 legacy=0x7f36e8130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e80f7bd0 osd=2 1 2024-02-19T21:04:53.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 10 client.35933.objecter close_session for osd.3 2024-02-19T21:04:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 >> v1:172.21.15.163:6813/2112279769 conn(0x7f36e812e210 legacy=0x7f36e8135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e80f7880 osd=3 1 2024-02-19T21:04:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 10 client.35933.objecter close_session for osd.4 2024-02-19T21:04:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 >> v1:172.21.15.193:6800/3129716965 conn(0x7f36e8132ea0 legacy=0x7f36e8139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e812d530 osd=4 1 2024-02-19T21:04:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 10 client.35933.objecter close_session for osd.5 2024-02-19T21:04:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 >> v1:172.21.15.193:6804/805161991 conn(0x7f36e813b870 legacy=0x7f36e813bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e812d6a0 osd=5 1 2024-02-19T21:04:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 10 client.35933.objecter close_session for osd.6 2024-02-19T21:04:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 >> v1:172.21.15.193:6808/787604615 conn(0x7f36e813fff0 legacy=0x7f36e81423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e813d560 osd=6 1 2024-02-19T21:04:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 10 client.35933.objecter close_session for osd.7 2024-02-19T21:04:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 >> v1:172.21.15.193:6812/157927432 conn(0x7f36e8144180 legacy=0x7f36e8144600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 20 client.35933.objecter put_session s=0x7f36e8143b00 osd=7 1 2024-02-19T21:04:53.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 20 client.35933.objecter shutdown clearing up homeless session... 2024-02-19T21:04:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.663+0000 7f36ed62d700 10 client.35933.objecter successfully canceled tick 2024-02-19T21:04:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.664+0000 7f36e77fe700 1 -- 172.21.15.163:0/3521777757 reap_dead start 2024-02-19T21:04:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.664+0000 7f36e77fe700 1 -- 172.21.15.163:0/3521777757 reap_dead start 2024-02-19T21:04:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.664+0000 7f36e77fe700 1 -- 172.21.15.163:0/3521777757 reap_dead start 2024-02-19T21:04:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.664+0000 7f36e77fe700 1 -- 172.21.15.163:0/3521777757 reap_dead start 2024-02-19T21:04:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.664+0000 7f36e77fe700 1 -- 172.21.15.163:0/3521777757 reap_dead start 2024-02-19T21:04:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.664+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 >> v1:172.21.15.163:6800/1552514681 conn(0x7f36d0063c10 legacy=0x7f36d00660d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:53.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.664+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 >> v1:172.21.15.163:6790/0 conn(0x7f36e80faff0 legacy=0x7f36e810b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.664+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 shutdown_connections 2024-02-19T21:04:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.664+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 >> 172.21.15.163:0/3521777757 conn(0x7f36e806fdc0 msgr2=0x7f36e810d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.664+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 shutdown_connections 2024-02-19T21:04:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.664+0000 7f36ed62d700 1 -- 172.21.15.163:0/3521777757 wait complete. 2024-02-19T21:04:53.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:53.665+0000 7f36ed62d700 1 librados: shutdown 2024-02-19T21:04:53.666 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.666 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.666 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.666 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.666 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.666 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.666 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.666 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:04:53.692 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:04:53] "GET /metrics HTTP/1.1" 200 229240 "" "Prometheus/2.33.4" 2024-02-19T21:04:53.966 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-02-19T21:04:53.967 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-02-19T21:04:53.967 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-02-19T21:04:53.967 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-02-19T21:04:53.967 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: seed 83904 2024-02-19T21:04:53.967 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-02-19T21:04:53.967 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (149 ms) 2024-02-19T21:04:53.967 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-02-19T21:04:53.967 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (126 ms) 2024-02-19T21:04:53.967 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-02-19T21:04:53.967 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1023 ms) 2024-02-19T21:04:53.967 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-02-19T21:04:53.967 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1050 ms) 2024-02-19T21:04:53.967 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-02-19T21:04:53.967 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (4 ms) 2024-02-19T21:04:53.967 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-02-19T21:04:53.968 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (24 ms) 2024-02-19T21:04:53.968 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-02-19T21:04:53.968 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (11 ms) 2024-02-19T21:04:53.968 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-02-19T21:04:53.968 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-02-19T21:04:53.968 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2390 ms total) 2024-02-19T21:04:53.968 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: 2024-02-19T21:04:53.968 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-02-19T21:04:53.968 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-02-19T21:04:53.968 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (644 ms) 2024-02-19T21:04:53.968 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-02-19T21:04:53.968 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (5 ms) 2024-02-19T21:04:53.968 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-02-19T21:04:53.969 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1005 ms) 2024-02-19T21:04:53.969 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-02-19T21:04:53.969 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1020 ms) 2024-02-19T21:04:53.969 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-02-19T21:04:53.969 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (6 ms) 2024-02-19T21:04:53.969 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-02-19T21:04:53.969 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (6 ms) 2024-02-19T21:04:53.969 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-02-19T21:04:53.969 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (6 ms) 2024-02-19T21:04:53.969 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-02-19T21:04:53.969 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2024-02-19T21:04:53.969 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (2695 ms total) 2024-02-19T21:04:53.969 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: 2024-02-19T21:04:53.969 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-02-19T21:04:53.969 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (21135 ms total) 2024-02-19T21:04:53.970 INFO:tasks.workunit.client.0.smithi163.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-02-19T21:04:54.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:51.819464+0000 osd.2 (osd.2) 61 : cluster [DBG] 5.b scrub starts 2024-02-19T21:04:54.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:51.822841+0000 osd.2 (osd.2) 62 : cluster [DBG] 5.b scrub ok 2024-02-19T21:04:54.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:52.024447+0000 osd.3 (osd.3) 81 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:04:54.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:52.026994+0000 osd.3 (osd.3) 82 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:04:54.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:52.562358+0000 mon.a (mon.0) 949 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:04:54.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.573528+0000 mon.a (mon.0) 950 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10"}]': finished 2024-02-19T21:04:54.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.573606+0000 mon.a (mon.0) 951 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "mode": "writeback"}]': finished 2024-02-19T21:04:54.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.573733+0000 mon.a (mon.0) 952 : audit [INF] from='client.35258 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi163-83944-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:54.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.573888+0000 mon.a (mon.0) 953 : audit [INF] from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]': finished 2024-02-19T21:04:54.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.574102+0000 mon.a (mon.0) 954 : audit [INF] from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:54.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:52.574171+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-19T21:04:54.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.588717+0000 mon.b (mon.1) 161 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:54.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.592252+0000 mon.b (mon.1) 162 : audit [INF] from='client.? v1:172.21.15.163:0/1555920557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.592397+0000 mon.b (mon.1) 163 : audit [INF] from='client.? v1:172.21.15.163:0/3689611843' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi163-83845-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.597531+0000 mon.c (mon.2) 79 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi163-83976-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:54.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.598954+0000 mon.a (mon.0) 956 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:54.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.620272+0000 mon.a (mon.0) 957 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:54.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.620739+0000 mon.a (mon.0) 958 : audit [INF] from='client.25381 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.620860+0000 mon.a (mon.0) 959 : audit [INF] from='client.25369 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi163-83845-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.635189+0000 mon.c (mon.2) 80 : audit [INF] from='client.? v1:172.21.15.163:0/856035894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.636068+0000 mon.a (mon.0) 960 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi163-83976-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:54.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:52.636641+0000 mon.a (mon.0) 961 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:53.346713+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? v1:172.21.15.163:0/3521777757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:04:54.246 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:53.397835+0000 mon.c (mon.2) 82 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:54.246 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:53.511521+0000 mon.a (mon.0) 962 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:04:54.246 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:53.511873+0000 mon.a (mon.0) 963 : cluster [WRN] pool 'PoolQuotaPP_smithi163-83854-3' is full (reached quota's max_bytes: 4 KiB) 2024-02-19T21:04:54.246 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:53.513996+0000 mon.a (mon.0) 964 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-19T21:04:54.246 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:53.558549+0000 mon.a (mon.0) 965 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi163-83904-10"}]': finished 2024-02-19T21:04:54.246 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:53.558706+0000 mon.a (mon.0) 966 : audit [INF] from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]': finished 2024-02-19T21:04:54.246 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:53.558819+0000 mon.a (mon.0) 967 : audit [INF] from='client.25381 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:54.246 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:53.558901+0000 mon.a (mon.0) 968 : audit [INF] from='client.25369 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi163-83845-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:54.246 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:53.953431+0000 mon.a (mon.0) 969 : audit [INF] from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:54.246 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:53.953598+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-19T21:04:54.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:51.819464+0000 osd.2 (osd.2) 61 : cluster [DBG] 5.b scrub starts 2024-02-19T21:04:54.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:51.822841+0000 osd.2 (osd.2) 62 : cluster [DBG] 5.b scrub ok 2024-02-19T21:04:54.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:52.024447+0000 osd.3 (osd.3) 81 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:52.026994+0000 osd.3 (osd.3) 82 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:52.562358+0000 mon.a (mon.0) 949 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.573528+0000 mon.a (mon.0) 950 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10"}]': finished 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.573606+0000 mon.a (mon.0) 951 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "mode": "writeback"}]': finished 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.573733+0000 mon.a (mon.0) 952 : audit [INF] from='client.35258 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi163-83944-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.573888+0000 mon.a (mon.0) 953 : audit [INF] from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]': finished 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.574102+0000 mon.a (mon.0) 954 : audit [INF] from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:52.574171+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.588717+0000 mon.b (mon.1) 161 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.592252+0000 mon.b (mon.1) 162 : audit [INF] from='client.? v1:172.21.15.163:0/1555920557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.592397+0000 mon.b (mon.1) 163 : audit [INF] from='client.? v1:172.21.15.163:0/3689611843' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi163-83845-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.597531+0000 mon.c (mon.2) 79 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi163-83976-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.598954+0000 mon.a (mon.0) 956 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.620272+0000 mon.a (mon.0) 957 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.620739+0000 mon.a (mon.0) 958 : audit [INF] from='client.25381 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.620860+0000 mon.a (mon.0) 959 : audit [INF] from='client.25369 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi163-83845-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.635189+0000 mon.c (mon.2) 80 : audit [INF] from='client.? v1:172.21.15.163:0/856035894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.636068+0000 mon.a (mon.0) 960 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi163-83976-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:54.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:52.636641+0000 mon.a (mon.0) 961 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:53.346713+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? v1:172.21.15.163:0/3521777757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:04:54.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:53.397835+0000 mon.c (mon.2) 82 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:54.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:53.511521+0000 mon.a (mon.0) 962 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:04:54.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:53.511873+0000 mon.a (mon.0) 963 : cluster [WRN] pool 'PoolQuotaPP_smithi163-83854-3' is full (reached quota's max_bytes: 4 KiB) 2024-02-19T21:04:54.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:53.513996+0000 mon.a (mon.0) 964 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-19T21:04:54.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:53.558549+0000 mon.a (mon.0) 965 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi163-83904-10"}]': finished 2024-02-19T21:04:54.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:53.558706+0000 mon.a (mon.0) 966 : audit [INF] from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]': finished 2024-02-19T21:04:54.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:53.558819+0000 mon.a (mon.0) 967 : audit [INF] from='client.25381 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:54.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:53.558901+0000 mon.a (mon.0) 968 : audit [INF] from='client.25369 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi163-83845-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:54.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:53.953431+0000 mon.a (mon.0) 969 : audit [INF] from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:54.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:53.953598+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-19T21:04:54.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:51.819464+0000 osd.2 (osd.2) 61 : cluster [DBG] 5.b scrub starts 2024-02-19T21:04:54.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:51.822841+0000 osd.2 (osd.2) 62 : cluster [DBG] 5.b scrub ok 2024-02-19T21:04:54.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:52.024447+0000 osd.3 (osd.3) 81 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:04:54.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:52.026994+0000 osd.3 (osd.3) 82 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:04:54.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:52.562358+0000 mon.a (mon.0) 949 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:04:54.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.573528+0000 mon.a (mon.0) 950 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi163-83904-10"}]': finished 2024-02-19T21:04:54.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.573606+0000 mon.a (mon.0) 951 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "mode": "writeback"}]': finished 2024-02-19T21:04:54.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.573733+0000 mon.a (mon.0) 952 : audit [INF] from='client.35258 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi163-83944-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:54.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.573888+0000 mon.a (mon.0) 953 : audit [INF] from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi163-83896-10"}]': finished 2024-02-19T21:04:54.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.574102+0000 mon.a (mon.0) 954 : audit [INF] from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:04:54.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:52.574171+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-02-19T21:04:54.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.588717+0000 mon.b (mon.1) 161 : audit [INF] from='client.? v1:172.21.15.163:0/2922168330' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:54.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.592252+0000 mon.b (mon.1) 162 : audit [INF] from='client.? v1:172.21.15.163:0/1555920557' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.592397+0000 mon.b (mon.1) 163 : audit [INF] from='client.? v1:172.21.15.163:0/3689611843' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi163-83845-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.597531+0000 mon.c (mon.2) 79 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi163-83976-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:54.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.598954+0000 mon.a (mon.0) 956 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi163-83904-10"}]: dispatch 2024-02-19T21:04:54.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.620272+0000 mon.a (mon.0) 957 : audit [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]: dispatch 2024-02-19T21:04:54.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.620739+0000 mon.a (mon.0) 958 : audit [INF] from='client.25381 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.620860+0000 mon.a (mon.0) 959 : audit [INF] from='client.25369 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi163-83845-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.635189+0000 mon.c (mon.2) 80 : audit [INF] from='client.? v1:172.21.15.163:0/856035894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.636068+0000 mon.a (mon.0) 960 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi163-83976-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:54.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:52.636641+0000 mon.a (mon.0) 961 : audit [INF] from='client.35273 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:54.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:53.346713+0000 mon.c (mon.2) 81 : audit [DBG] from='client.? v1:172.21.15.163:0/3521777757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:04:54.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:53.397835+0000 mon.c (mon.2) 82 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:54.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:53.511521+0000 mon.a (mon.0) 962 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:04:54.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:53.511873+0000 mon.a (mon.0) 963 : cluster [WRN] pool 'PoolQuotaPP_smithi163-83854-3' is full (reached quota's max_bytes: 4 KiB) 2024-02-19T21:04:54.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:53.513996+0000 mon.a (mon.0) 964 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-02-19T21:04:54.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:53.558549+0000 mon.a (mon.0) 965 : audit [INF] from='client.? v1:172.21.15.163:0/676446879' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi163-83904-10"}]': finished 2024-02-19T21:04:54.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:53.558706+0000 mon.a (mon.0) 966 : audit [INF] from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi163-83896-10"}]': finished 2024-02-19T21:04:54.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:53.558819+0000 mon.a (mon.0) 967 : audit [INF] from='client.25381 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:54.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:53.558901+0000 mon.a (mon.0) 968 : audit [INF] from='client.25369 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi163-83845-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:54.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:53.953431+0000 mon.a (mon.0) 969 : audit [INF] from='client.35273 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:54.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:53.953598+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-02-19T21:04:55.124 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:55.141 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:55.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:55.243 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:55.269 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:55.287 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:55.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:55.391 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:55.420 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:55.624 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:55.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:55.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:55.734 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:04:55.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:52.789073+0000 osd.2 (osd.2) 63 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:04:55.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:52.792926+0000 osd.2 (osd.2) 64 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:04:55.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:52.969918+0000 mgr.y (mgr.14550) 157 : cluster [DBG] pgmap v141: 520 pgs: 128 unknown, 4 creating+peering, 1 active+clean+snaptrim, 387 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-02-19T21:04:55.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:52.988015+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:04:55.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:52.990308+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:04:55.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:53.131544+0000 osd.5 (osd.5) 64 : cluster [DBG] 28.9 deep-scrub starts 2024-02-19T21:04:55.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:53.134420+0000 osd.5 (osd.5) 65 : cluster [DBG] 28.9 deep-scrub ok 2024-02-19T21:04:55.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:53.958673+0000 mon.b (mon.1) 164 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:55.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:53.972548+0000 mon.a (mon.0) 971 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:55.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:54.083913+0000 mon.a (mon.0) 972 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:04:55.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:54.321396+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:04:55.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:54.322188+0000 osd.0 (osd.0) 70 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:04:55.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:54.399040+0000 mon.c (mon.2) 83 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:55.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:54.562335+0000 mon.a (mon.0) 973 : audit [INF] from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi163-83976-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]': finished 2024-02-19T21:04:55.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:54.562543+0000 mon.a (mon.0) 974 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:55.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:54.562624+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-02-19T21:04:55.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:52.789073+0000 osd.2 (osd.2) 63 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:04:55.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:52.792926+0000 osd.2 (osd.2) 64 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:04:55.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:52.969918+0000 mgr.y (mgr.14550) 157 : cluster [DBG] pgmap v141: 520 pgs: 128 unknown, 4 creating+peering, 1 active+clean+snaptrim, 387 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-02-19T21:04:55.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:52.988015+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:04:55.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:52.990308+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:04:55.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:53.131544+0000 osd.5 (osd.5) 64 : cluster [DBG] 28.9 deep-scrub starts 2024-02-19T21:04:55.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:53.134420+0000 osd.5 (osd.5) 65 : cluster [DBG] 28.9 deep-scrub ok 2024-02-19T21:04:55.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:53.958673+0000 mon.b (mon.1) 164 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:55.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:53.972548+0000 mon.a (mon.0) 971 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:55.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:54.083913+0000 mon.a (mon.0) 972 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:04:55.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:54.321396+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:04:55.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:54.322188+0000 osd.0 (osd.0) 70 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:04:55.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:54.399040+0000 mon.c (mon.2) 83 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:55.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:54.562335+0000 mon.a (mon.0) 973 : audit [INF] from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi163-83976-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]': finished 2024-02-19T21:04:55.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:54.562543+0000 mon.a (mon.0) 974 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:55.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:54.562624+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-02-19T21:04:55.760 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-02-19T21:04:55.760 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-02-19T21:04:55.760 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-02-19T21:04:55.760 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-02-19T21:04:55.760 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-02-19T21:04:55.760 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: notify 2024-02-19T21:04:55.761 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1132 ms) 2024-02-19T21:04:55.761 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-02-19T21:04:55.761 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (4 ms) 2024-02-19T21:04:55.761 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1136 ms total) 2024-02-19T21:04:55.761 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: 2024-02-19T21:04:55.761 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-02-19T21:04:55.761 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-02-19T21:04:55.761 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: notify 2024-02-19T21:04:55.761 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (72 ms) 2024-02-19T21:04:55.761 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-02-19T21:04:55.761 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: notify 2024-02-19T21:04:55.761 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (4070 ms) 2024-02-19T21:04:55.761 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-02-19T21:04:55.761 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (13 ms) 2024-02-19T21:04:55.761 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (18 ms) 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: handle_notify cookie 94487086340384 notify_id 485331304453 notifier_gid 35237 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (14 ms) 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: handle_notify cookie 94487086340384 notify_id 485331304449 notifier_gid 35237 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (13 ms) 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: handle_notify cookie 94487086340384 notify_id 485331304450 notifier_gid 35237 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (19 ms) 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: handle_notify cookie 94487086340384 notify_id 485331304451 notifier_gid 35237 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (7 ms) 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: handle_notify cookie 94487086340384 notify_id 485331304451 notifier_gid 35237 2024-02-19T21:04:55.762 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (12 ms) 2024-02-19T21:04:55.763 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-02-19T21:04:55.763 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: handle_notify cookie 94487086340384 notify_id 485331304452 notifier_gid 35237 2024-02-19T21:04:55.763 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (5 ms) 2024-02-19T21:04:55.763 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-02-19T21:04:55.763 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: trying... 2024-02-19T21:04:55.763 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: handle_notify cookie 94487086340384 notify_id 485331304452 notifier_gid 35237 2024-02-19T21:04:55.763 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: timed out 2024-02-19T21:04:55.763 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: flushing 2024-02-19T21:04:55.802 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:56.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:52.789073+0000 osd.2 (osd.2) 63 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:04:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:52.792926+0000 osd.2 (osd.2) 64 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:04:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:52.969918+0000 mgr.y (mgr.14550) 157 : cluster [DBG] pgmap v141: 520 pgs: 128 unknown, 4 creating+peering, 1 active+clean+snaptrim, 387 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-02-19T21:04:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:52.988015+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:04:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:52.990308+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:04:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:53.131544+0000 osd.5 (osd.5) 64 : cluster [DBG] 28.9 deep-scrub starts 2024-02-19T21:04:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:53.134420+0000 osd.5 (osd.5) 65 : cluster [DBG] 28.9 deep-scrub ok 2024-02-19T21:04:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:53.958673+0000 mon.b (mon.1) 164 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:53.972548+0000 mon.a (mon.0) 971 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:54.083913+0000 mon.a (mon.0) 972 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:04:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:54.321396+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:04:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:54.322188+0000 osd.0 (osd.0) 70 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:04:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:54.399040+0000 mon.c (mon.2) 83 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:54.562335+0000 mon.a (mon.0) 973 : audit [INF] from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi163-83976-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]': finished 2024-02-19T21:04:56.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:54.562543+0000 mon.a (mon.0) 974 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:56.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:54.562624+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-02-19T21:04:56.151 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:04:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:56.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:04:56.261 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:04:56.329 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:56.690 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:04:56.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:56.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:04:56.807 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:04:56.834 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:56.853 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:04:56.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:56.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:04:56.957 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:56.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:55.400303+0000 mon.c (mon.2) 84 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:56.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:55.647309+0000 mon.a (mon.0) 976 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-02-19T21:04:56.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:55.670002+0000 mon.b (mon.1) 165 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:56.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:55.670255+0000 mon.b (mon.1) 166 : audit [INF] from='client.? v1:172.21.15.163:0/2924448120' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi163-83952-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:56.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:55.670521+0000 mon.b (mon.1) 167 : audit [INF] from='client.? v1:172.21.15.163:0/1145664497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:56.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:55.400303+0000 mon.c (mon.2) 84 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:56.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:55.647309+0000 mon.a (mon.0) 976 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-02-19T21:04:56.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:55.670002+0000 mon.b (mon.1) 165 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:56.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:55.670255+0000 mon.b (mon.1) 166 : audit [INF] from='client.? v1:172.21.15.163:0/2924448120' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi163-83952-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:56.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:55.670521+0000 mon.b (mon.1) 167 : audit [INF] from='client.? v1:172.21.15.163:0/1145664497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:57.007 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:04:57.007 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:04:57.007 INFO:tasks.thrashosds.thrasher:Growing pool .rgw.root 2024-02-19T21:04:57.007 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:04:57.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:55.400303+0000 mon.c (mon.2) 84 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:57.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:55.647309+0000 mon.a (mon.0) 976 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-02-19T21:04:57.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:55.670002+0000 mon.b (mon.1) 165 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:57.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:55.670255+0000 mon.b (mon.1) 166 : audit [INF] from='client.? v1:172.21.15.163:0/2924448120' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi163-83952-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:57.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:55.670521+0000 mon.b (mon.1) 167 : audit [INF] from='client.? v1:172.21.15.163:0/1145664497' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:57.090 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:57.359 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:57.365 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:57.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:57.466 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:57.492 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:57.511 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:57.513 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:57.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:57.550 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:04:57.550 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:04:57.615 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:04:57.631 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:04:57.642 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:57.661 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:04:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:57.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:04:57.765 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:57.816 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:57.846 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:57.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:57.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:57.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.828+0000 7f2915c3f700 1 -- 172.21.15.163:0/1192678327 >> v1:172.21.15.163:6790/0 conn(0x7f2910113250 legacy=0x7f2910115640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:57.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.829+0000 7f2915c3f700 1 -- 172.21.15.163:0/1192678327 shutdown_connections 2024-02-19T21:04:57.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.829+0000 7f2915c3f700 1 -- 172.21.15.163:0/1192678327 >> 172.21.15.163:0/1192678327 conn(0x7f2910063b20 msgr2=0x7f2910067230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:57.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.829+0000 7f2915c3f700 1 -- 172.21.15.163:0/1192678327 shutdown_connections 2024-02-19T21:04:57.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.829+0000 7f2915c3f700 1 -- 172.21.15.163:0/1192678327 wait complete. 2024-02-19T21:04:57.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.830+0000 7f2915c3f700 1 librados: starting msgr at 2024-02-19T21:04:57.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.830+0000 7f2915c3f700 1 librados: starting objecter 2024-02-19T21:04:57.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.830+0000 7f2915c3f700 1 Processor -- start 2024-02-19T21:04:57.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.830+0000 7f2915c3f700 1 -- start start 2024-02-19T21:04:57.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.830+0000 7f2915c3f700 1 librados: setting wanted keys 2024-02-19T21:04:57.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.830+0000 7f2915c3f700 1 librados: calling monclient init 2024-02-19T21:04:57.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.831+0000 7f2915c3f700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2910112d10 con 0x7f2910069ee0 2024-02-19T21:04:57.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.831+0000 7f2915c3f700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2910112ea0 con 0x7f29100693f0 2024-02-19T21:04:57.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.831+0000 7f2915c3f700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2910113030 con 0x7f2910113250 2024-02-19T21:04:57.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.832+0000 7f290f7fe700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f29100693f0 0x7f291010b7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:53230/0 (socket says 172.21.15.163:53230) 2024-02-19T21:04:57.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.832+0000 7f290f7fe700 1 -- 172.21.15.163:0/2178055801 learned_addr learned my addr 172.21.15.163:0/2178055801 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:04:57.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.836+0000 7f290cff9700 10 client.?.objecter ms_handle_connect 0x7f29100693f0 2024-02-19T21:04:57.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.836+0000 7f290cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:57.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.837+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722428679 0 0) 0x7f2910112ea0 con 0x7f29100693f0 2024-02-19T21:04:57.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.837+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28f8003630 con 0x7f29100693f0 2024-02-19T21:04:57.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.839+0000 7f290cff9700 10 client.?.objecter ms_handle_connect 0x7f2910069ee0 2024-02-19T21:04:57.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.839+0000 7f290cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:04:57.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.839+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1653241017 0 0) 0x7f28f8003630 con 0x7f29100693f0 2024-02-19T21:04:57.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.839+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2910112ea0 con 0x7f29100693f0 2024-02-19T21:04:57.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.839+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f29000028d0 con 0x7f29100693f0 2024-02-19T21:04:57.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.841+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1373720128 0 0) 0x7f2910112ea0 con 0x7f29100693f0 2024-02-19T21:04:57.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.841+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 >> v1:172.21.15.163:6790/0 conn(0x7f2910113250 legacy=0x7f291012d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:57.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.841+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 >> v1:172.21.15.163:6789/0 conn(0x7f2910069ee0 legacy=0x7f29101115f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:57.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.841+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2910131e50 con 0x7f29100693f0 2024-02-19T21:04:57.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.841+0000 7f2915c3f700 1 -- 172.21.15.163:0/2178055801 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291012ee80 con 0x7f29100693f0 2024-02-19T21:04:57.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.841+0000 7f2915c3f700 10 client.29173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:57.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.841+0000 7f2915c3f700 1 -- 172.21.15.163:0/2178055801 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f291012f2f0 con 0x7f29100693f0 2024-02-19T21:04:57.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.841+0000 7f2915c3f700 1 librados: init done 2024-02-19T21:04:57.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.843+0000 7f2915c3f700 1 -- 172.21.15.163:0/2178055801 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2910108380 con 0x7f29100693f0 2024-02-19T21:04:57.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.847+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f29000013f0 con 0x7f29100693f0 2024-02-19T21:04:57.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.847+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2900005b10 con 0x7f29100693f0 2024-02-19T21:04:57.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.847+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f2900018aa0 con 0x7f29100693f0 2024-02-19T21:04:57.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.847+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(118..118 src has 43..118) v4 ==== 12663+0+0 (unknown 1314235764 0 0) 0x7f290007af00 con 0x7f29100693f0 2024-02-19T21:04:57.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.847+0000 7f290cff9700 10 client.29173.objecter ms_dispatch 0x7f2910001550 osd_map(118..118 src has 43..118) v4 2024-02-19T21:04:57.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.847+0000 7f290cff9700 3 client.29173.objecter handle_osd_map got epochs [118,118] > 0 2024-02-19T21:04:57.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.847+0000 7f290cff9700 3 client.29173.objecter handle_osd_map decoding full epoch 118 2024-02-19T21:04:57.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.847+0000 7f290cff9700 10 client.29173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:57.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.847+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=119}) v3 -- 0x7f2910112ea0 con 0x7f29100693f0 2024-02-19T21:04:57.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.847+0000 7f290cff9700 20 client.29173.objecter dump_active .. 0 homeless 2024-02-19T21:04:57.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.849+0000 7f290cff9700 10 client.29173.objecter ms_handle_connect 0x7f28f8063c30 2024-02-19T21:04:57.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:57.855+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f290004d180 con 0x7f29100693f0 2024-02-19T21:04:57.949 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:04:58.027 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:54.971408+0000 mgr.y (mgr.14550) 158 : cluster [DBG] pgmap v144: 432 pgs: 8 creating+peering, 424 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:04:58.027 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:54.971634+0000 osd.3 (osd.3) 85 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:04:58.027 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:54.972825+0000 osd.3 (osd.3) 86 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:04:58.027 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:56.119593+0000 osd.5 (osd.5) 66 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:04:58.027 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:56.123089+0000 osd.5 (osd.5) 67 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:04:58.027 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:56.260829+0000 osd.0 (osd.0) 71 : cluster [DBG] 4.b scrub starts 2024-02-19T21:04:58.027 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:56.262922+0000 osd.0 (osd.0) 72 : cluster [DBG] 4.b scrub ok 2024-02-19T21:04:58.028 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:56.401837+0000 mon.c (mon.2) 85 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:58.028 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:56.717915+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:04:58.028 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:56.720600+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:04:58.028 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:56.747217+0000 mon.a (mon.0) 977 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.028 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:56.747529+0000 mon.a (mon.0) 978 : audit [INF] from='client.26962 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi163-83952-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.028 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:56.747937+0000 mon.a (mon.0) 979 : audit [INF] from='client.26965 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.028 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:56.757033+0000 mon.a (mon.0) 980 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-02-19T21:04:58.028 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:56.774468+0000 mon.b (mon.1) 168 : audit [INF] from='client.? v1:172.21.15.163:0/2922312863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.028 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:56.775933+0000 mon.b (mon.1) 169 : audit [INF] from='client.? v1:172.21.15.163:0/2968371302' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi163-83845-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.028 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:56.790806+0000 mon.a (mon.0) 981 : audit [INF] from='client.26980 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.028 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:56.791117+0000 mon.a (mon.0) 982 : audit [INF] from='client.26974 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi163-83845-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.028 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:56.985123+0000 mgr.y (mgr.14550) 159 : cluster [DBG] pgmap v147: 592 pgs: 160 unknown, 8 creating+peering, 424 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1020 B/s rd, 5.5 KiB/s wr, 5 op/s 2024-02-19T21:04:58.028 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:57.168108+0000 osd.5 (osd.5) 68 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:04:58.029 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:57.188660+0000 osd.5 (osd.5) 69 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:04:58.029 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:57.406624+0000 mon.c (mon.2) 86 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:58.029 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:54.971408+0000 mgr.y (mgr.14550) 158 : cluster [DBG] pgmap v144: 432 pgs: 8 creating+peering, 424 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:04:58.029 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:54.971634+0000 osd.3 (osd.3) 85 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:04:58.029 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:54.972825+0000 osd.3 (osd.3) 86 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:04:58.029 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:56.119593+0000 osd.5 (osd.5) 66 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:04:58.029 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:56.123089+0000 osd.5 (osd.5) 67 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:04:58.029 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:56.260829+0000 osd.0 (osd.0) 71 : cluster [DBG] 4.b scrub starts 2024-02-19T21:04:58.029 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:56.262922+0000 osd.0 (osd.0) 72 : cluster [DBG] 4.b scrub ok 2024-02-19T21:04:58.029 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:56.401837+0000 mon.c (mon.2) 85 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:58.029 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:56.717915+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:04:58.029 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:56.720600+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:04:58.030 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:56.747217+0000 mon.a (mon.0) 977 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.030 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:56.747529+0000 mon.a (mon.0) 978 : audit [INF] from='client.26962 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi163-83952-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.030 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:56.747937+0000 mon.a (mon.0) 979 : audit [INF] from='client.26965 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.030 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:56.757033+0000 mon.a (mon.0) 980 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-02-19T21:04:58.030 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:56.774468+0000 mon.b (mon.1) 168 : audit [INF] from='client.? v1:172.21.15.163:0/2922312863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.030 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:56.775933+0000 mon.b (mon.1) 169 : audit [INF] from='client.? v1:172.21.15.163:0/2968371302' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi163-83845-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.030 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:56.790806+0000 mon.a (mon.0) 981 : audit [INF] from='client.26980 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.030 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:56.791117+0000 mon.a (mon.0) 982 : audit [INF] from='client.26974 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi163-83845-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.030 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:56.985123+0000 mgr.y (mgr.14550) 159 : cluster [DBG] pgmap v147: 592 pgs: 160 unknown, 8 creating+peering, 424 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1020 B/s rd, 5.5 KiB/s wr, 5 op/s 2024-02-19T21:04:58.030 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:57.168108+0000 osd.5 (osd.5) 68 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:04:58.030 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:57.188660+0000 osd.5 (osd.5) 69 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:04:58.030 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:57.406624+0000 mon.c (mon.2) 86 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:58.064 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:04:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:04:58.064 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:54.971408+0000 mgr.y (mgr.14550) 158 : cluster [DBG] pgmap v144: 432 pgs: 8 creating+peering, 424 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:04:58.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:54.971634+0000 osd.3 (osd.3) 85 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:04:58.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:54.972825+0000 osd.3 (osd.3) 86 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:04:58.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:56.119593+0000 osd.5 (osd.5) 66 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:04:58.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:56.123089+0000 osd.5 (osd.5) 67 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:04:58.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:56.260829+0000 osd.0 (osd.0) 71 : cluster [DBG] 4.b scrub starts 2024-02-19T21:04:58.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:56.262922+0000 osd.0 (osd.0) 72 : cluster [DBG] 4.b scrub ok 2024-02-19T21:04:58.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:56.401837+0000 mon.c (mon.2) 85 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:58.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:56.717915+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:04:58.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:56.720600+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:04:58.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:56.747217+0000 mon.a (mon.0) 977 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:56.747529+0000 mon.a (mon.0) 978 : audit [INF] from='client.26962 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi163-83952-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:56.747937+0000 mon.a (mon.0) 979 : audit [INF] from='client.26965 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:56.757033+0000 mon.a (mon.0) 980 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-02-19T21:04:58.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:56.774468+0000 mon.b (mon.1) 168 : audit [INF] from='client.? v1:172.21.15.163:0/2922312863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:56.775933+0000 mon.b (mon.1) 169 : audit [INF] from='client.? v1:172.21.15.163:0/2968371302' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi163-83845-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:56.790806+0000 mon.a (mon.0) 981 : audit [INF] from='client.26980 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:56.791117+0000 mon.a (mon.0) 982 : audit [INF] from='client.26974 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi163-83845-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:04:58.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:56.985123+0000 mgr.y (mgr.14550) 159 : cluster [DBG] pgmap v147: 592 pgs: 160 unknown, 8 creating+peering, 424 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1020 B/s rd, 5.5 KiB/s wr, 5 op/s 2024-02-19T21:04:58.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:57.168108+0000 osd.5 (osd.5) 68 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:04:58.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:57.188660+0000 osd.5 (osd.5) 69 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:04:58.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:57.406624+0000 mon.c (mon.2) 86 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:58.134 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:58.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.303+0000 7f2915c3f700 1 -- 172.21.15.163:0/2178055801 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f291006a8c0 con 0x7f29100693f0 2024-02-19T21:04:58.315 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:04:58.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:58.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:04:58.450 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:58.509 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:58.683 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:04:58.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:58.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:04:58.731 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-19T21:04:58.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.776+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 <== mon.1 v1:172.21.15.193:6789/0 10 ==== osd_map(119..119 src has 43..119) v4 ==== 1728+0+0 (unknown 1364095504 0 0) 0x7f290004d3b0 con 0x7f29100693f0 2024-02-19T21:04:58.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.776+0000 7f290cff9700 10 client.29173.objecter ms_dispatch 0x7f2910001550 osd_map(119..119 src has 43..119) v4 2024-02-19T21:04:58.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.776+0000 7f290cff9700 3 client.29173.objecter handle_osd_map got epochs [119,119] > 118 2024-02-19T21:04:58.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.776+0000 7f290cff9700 3 client.29173.objecter handle_osd_map decoding incremental epoch 119 2024-02-19T21:04:58.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.776+0000 7f290cff9700 10 client.29173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:04:58.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.776+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=120}) v3 -- 0x7f2910113030 con 0x7f29100693f0 2024-02-19T21:04:58.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.776+0000 7f290cff9700 20 client.29173.objecter dump_active .. 0 homeless 2024-02-19T21:04:58.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.796+0000 7f290cff9700 1 -- 172.21.15.163:0/2178055801 <== mon.1 v1:172.21.15.193:6789/0 11 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v119) v1 ==== 101+0+0 (unknown 327193665 0 0) 0x7f290004b590 con 0x7f29100693f0 2024-02-19T21:04:58.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.800+0000 7f2915c3f700 10 librados: watch_flush enter 2024-02-19T21:04:58.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.800+0000 7f2915c3f700 10 librados: watch_flush exit 2024-02-19T21:04:58.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.800+0000 7f2915c3f700 20 client.29173.objecter shutdown clearing up homeless session... 2024-02-19T21:04:58.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.800+0000 7f2915c3f700 10 client.29173.objecter successfully canceled tick 2024-02-19T21:04:58.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.801+0000 7f2915c3f700 1 -- 172.21.15.163:0/2178055801 >> v1:172.21.15.163:6800/1552514681 conn(0x7f28f8063c30 legacy=0x7f28f80660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:58.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.801+0000 7f2915c3f700 1 -- 172.21.15.163:0/2178055801 >> v1:172.21.15.193:6789/0 conn(0x7f29100693f0 legacy=0x7f291010b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:04:58.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.801+0000 7f2915c3f700 1 -- 172.21.15.163:0/2178055801 shutdown_connections 2024-02-19T21:04:58.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.802+0000 7f2915c3f700 1 -- 172.21.15.163:0/2178055801 >> 172.21.15.163:0/2178055801 conn(0x7f2910063b20 msgr2=0x7f2910064720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:04:58.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.802+0000 7f2915c3f700 1 -- 172.21.15.163:0/2178055801 shutdown_connections 2024-02-19T21:04:58.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.802+0000 7f2915c3f700 1 -- 172.21.15.163:0/2178055801 wait complete. 2024-02-19T21:04:58.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:04:58.802+0000 7f2915c3f700 1 librados: shutdown 2024-02-19T21:04:58.816 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:58.817 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:58.817 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:04:58.876 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:04:58.963 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:59.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:56.826972+0000 osd.2 (osd.2) 65 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:04:59.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:56.846615+0000 osd.2 (osd.2) 66 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:04:59.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:56.992400+0000 osd.3 (osd.3) 87 : cluster [DBG] 5.13 scrub starts 2024-02-19T21:04:59.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:56.999134+0000 osd.3 (osd.3) 88 : cluster [DBG] 5.13 scrub ok 2024-02-19T21:04:59.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:57.778267+0000 mon.a (mon.0) 983 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:59.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:57.778346+0000 mon.a (mon.0) 984 : audit [INF] from='client.26962 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi163-83952-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:59.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:57.778408+0000 mon.a (mon.0) 985 : audit [INF] from='client.26965 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:59.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:57.786185+0000 mon.a (mon.0) 986 : audit [INF] from='client.26980 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:59.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:57.786274+0000 mon.c (mon.2) 87 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:59.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:57.786286+0000 mon.a (mon.0) 987 : audit [INF] from='client.26974 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi163-83845-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:59.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:57.786373+0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-02-19T21:04:59.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:57.822083+0000 mon.a (mon.0) 989 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:59.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:57.914371+0000 mon.b (mon.1) 170 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:04:59.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:57.915458+0000 mon.a (mon.0) 990 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:04:59.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:58.260586+0000 osd.7 (osd.7) 71 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:04:59.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:58.273110+0000 osd.7 (osd.7) 72 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:04:59.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:58.304929+0000 mon.b (mon.1) 171 : audit [INF] from='client.? v1:172.21.15.163:0/2178055801' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:59.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:58.306278+0000 mon.a (mon.0) 991 : audit [INF] from='client.29173 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:59.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:58.409348+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:59.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:04:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:58.515550+0000 mon.a (mon.0) 992 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:04:59.116 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:59.120 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:59.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:59.124 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:56.826972+0000 osd.2 (osd.2) 65 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:04:59.124 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:56.846615+0000 osd.2 (osd.2) 66 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:04:59.124 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:56.992400+0000 osd.3 (osd.3) 87 : cluster [DBG] 5.13 scrub starts 2024-02-19T21:04:59.124 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:56.999134+0000 osd.3 (osd.3) 88 : cluster [DBG] 5.13 scrub ok 2024-02-19T21:04:59.124 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:57.778267+0000 mon.a (mon.0) 983 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:59.124 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:57.778346+0000 mon.a (mon.0) 984 : audit [INF] from='client.26962 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi163-83952-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:59.124 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:57.778408+0000 mon.a (mon.0) 985 : audit [INF] from='client.26965 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:59.124 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:57.786185+0000 mon.a (mon.0) 986 : audit [INF] from='client.26980 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:59.124 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:57.786274+0000 mon.c (mon.2) 87 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:59.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:57.786286+0000 mon.a (mon.0) 987 : audit [INF] from='client.26974 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi163-83845-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:59.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:57.786373+0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-02-19T21:04:59.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:57.822083+0000 mon.a (mon.0) 989 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:59.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:57.914371+0000 mon.b (mon.1) 170 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:04:59.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:57.915458+0000 mon.a (mon.0) 990 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:04:59.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:58.260586+0000 osd.7 (osd.7) 71 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:04:59.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:58.273110+0000 osd.7 (osd.7) 72 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:04:59.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:58.304929+0000 mon.b (mon.1) 171 : audit [INF] from='client.? v1:172.21.15.163:0/2178055801' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:59.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:58.306278+0000 mon.a (mon.0) 991 : audit [INF] from='client.29173 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:59.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:58.409348+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:59.125 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:58.515550+0000 mon.a (mon.0) 992 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:04:59.125 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:56.826972+0000 osd.2 (osd.2) 65 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:04:59.126 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:56.846615+0000 osd.2 (osd.2) 66 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:04:59.126 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:56.992400+0000 osd.3 (osd.3) 87 : cluster [DBG] 5.13 scrub starts 2024-02-19T21:04:59.126 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:56.999134+0000 osd.3 (osd.3) 88 : cluster [DBG] 5.13 scrub ok 2024-02-19T21:04:59.126 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:57.778267+0000 mon.a (mon.0) 983 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:59.126 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:57.778346+0000 mon.a (mon.0) 984 : audit [INF] from='client.26962 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi163-83952-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:59.126 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:57.778408+0000 mon.a (mon.0) 985 : audit [INF] from='client.26965 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:59.126 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:57.786185+0000 mon.a (mon.0) 986 : audit [INF] from='client.26980 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:59.126 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:57.786274+0000 mon.c (mon.2) 87 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:59.126 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:57.786286+0000 mon.a (mon.0) 987 : audit [INF] from='client.26974 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi163-83845-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:04:59.126 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:57.786373+0000 mon.a (mon.0) 988 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-02-19T21:04:59.126 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:57.822083+0000 mon.a (mon.0) 989 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:04:59.127 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:57.914371+0000 mon.b (mon.1) 170 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:04:59.127 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:57.915458+0000 mon.a (mon.0) 990 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:04:59.127 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:58.260586+0000 osd.7 (osd.7) 71 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:04:59.127 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:58.273110+0000 osd.7 (osd.7) 72 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:04:59.127 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:58.304929+0000 mon.b (mon.1) 171 : audit [INF] from='client.? v1:172.21.15.163:0/2178055801' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:59.127 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:58.306278+0000 mon.a (mon.0) 991 : audit [INF] from='client.29173 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:04:59.127 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:58.409348+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:04:59.127 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:04:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:58.515550+0000 mon.a (mon.0) 992 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:04:59.222 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:59.249 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:59.269 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:59.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:59.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:59.372 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:04:59.399 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:59.417 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:04:59.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:59.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:04:59.521 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:04:59.548 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:59.568 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:04:59.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:59.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:04:59.671 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:04:59.769 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:04:59.807 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:04:59.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:04:59.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:04:59.912 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:04:59.979 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:00.078 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:58.772555+0000 mon.a (mon.0) 993 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:05:00.078 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:58.777447+0000 mon.a (mon.0) 994 : audit [INF] from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]': finished 2024-02-19T21:05:00.078 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:58.777544+0000 mon.a (mon.0) 995 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:05:00.078 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:58.777623+0000 mon.a (mon.0) 996 : audit [INF] from='client.29173 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:05:00.078 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:58.777660+0000 mon.a (mon.0) 997 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-02-19T21:05:00.078 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:58.798649+0000 mon.b (mon.1) 172 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-7"}]: dispatch 2024-02-19T21:05:00.078 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:58.799453+0000 mon.c (mon.2) 89 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:05:00.078 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:58.803302+0000 mon.a (mon.0) 998 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-7"}]: dispatch 2024-02-19T21:05:00.078 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:59.217937+0000 mon.a (mon.0) 999 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:05:00.078 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:04:59.410198+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:00.105 INFO:tasks.workunit.client.0.smithi163.stdout: api api_watch_notify: Running main() from gmock_main.cc 2024-02-19T21:05:00.105 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-02-19T21:05:00.105 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: watch_notify_test_cb 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (307 ms) 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93925786816720 err -107 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1026 ms) 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93925786832848 err -107 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1009 ms) 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: watch_notify2_test_cb from 24922 notify_id 438086664192 cookie 93925786838848 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (22 ms) 2024-02-19T21:05:00.106 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: watch_notify2_test_cb from 24922 notify_id 438086664193 cookie 93925786888384 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (22 ms) 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: watch_notify2_test_cb from 24922 notify_id 438086664193 cookie 93925786926336 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (27 ms) 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: watch_notify2_test_cb from 24922 notify_id 438086664194 cookie 93925786902992 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: watch_notify2_test_cb from 24922 notify_id 438086664194 cookie 93925786926336 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (20 ms) 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: watch_notify2_test_cb from 24922 notify_id 438086664193 cookie 93925786926336 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: watch_notify2_test_cb from 24922 notify_id 442381631491 cookie 93925786926336 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3007 ms) 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-02-19T21:05:00.107 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93925786926336 err -107 2024-02-19T21:05:00.108 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: watch_notify2_test_cb from 24922 notify_id 472446402564 cookie 93925786926336 2024-02-19T21:05:00.108 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5009 ms) 2024-02-19T21:05:00.108 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-02-19T21:05:00.108 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-02-19T21:05:00.108 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93925786926336 err -107 2024-02-19T21:05:00.108 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1004 ms) 2024-02-19T21:05:00.108 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11454 ms total) 2024-02-19T21:05:00.108 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: 2024-02-19T21:05:00.108 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-02-19T21:05:00.108 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-02-19T21:05:00.108 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: watch_notify_test_cb 2024-02-19T21:05:00.108 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1153 ms) 2024-02-19T21:05:00.108 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1153 ms total) 2024-02-19T21:05:00.109 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: 2024-02-19T21:05:00.109 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-02-19T21:05:00.109 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (27287 ms total) 2024-02-19T21:05:00.109 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-02-19T21:05:00.123 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:00.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:00.299 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-02-19T21:05:00.299 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: [----------] Global test environment set-up. 2024-02-19T21:05:00.299 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: [----------] 4 tests from LibRadosService 2024-02-19T21:05:00.299 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-02-19T21:05:00.299 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5084 ms) 2024-02-19T21:05:00.299 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-02-19T21:05:00.300 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: [ OK ] LibRadosService.RegisterLate (38 ms) 2024-02-19T21:05:00.300 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-02-19T21:05:00.300 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: cluster: 2024-02-19T21:05:00.300 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: id: 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T21:05:00.300 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: health: HEALTH_WARN 2024-02-19T21:05:00.300 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: noscrub flag(s) set 2024-02-19T21:05:00.300 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: 2024-02-19T21:05:00.300 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: services: 2024-02-19T21:05:00.303 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-02-19T21:05:00.303 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-02-19T21:05:00.303 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-02-19T21:05:00.303 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: flags noscrub 2024-02-19T21:05:00.303 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-02-19T21:05:00.303 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: 2024-02-19T21:05:00.304 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: data: 2024-02-19T21:05:00.304 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: pools: 6 pools, 132 pgs 2024-02-19T21:05:00.304 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: objects: 192 objects, 5.7 KiB 2024-02-19T21:05:00.304 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:05:00.304 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: pgs: 132 active+clean 2024-02-19T21:05:00.304 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: 2024-02-19T21:05:00.304 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: io: 2024-02-19T21:05:00.304 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: client: 1023 B/s rd, 0 op/s rd, 0 op/s wr 2024-02-19T21:05:00.305 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: 2024-02-19T21:05:00.305 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: 2024-02-19T21:05:00.305 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: cluster: 2024-02-19T21:05:00.305 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: id: 2591809a-cf69-11ee-95bb-87774f69a715 2024-02-19T21:05:00.305 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: health: HEALTH_WARN 2024-02-19T21:05:00.305 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: noscrub,nodeep-scrub flag(s) set 2024-02-19T21:05:00.305 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: 2024-02-19T21:05:00.305 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: services: 2024-02-19T21:05:00.306 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-02-19T21:05:00.306 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: mgr: y(active, since 3m), standbys: x 2024-02-19T21:05:00.306 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-02-19T21:05:00.306 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: flags noscrub,nodeep-scrub 2024-02-19T21:05:00.306 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-02-19T21:05:00.306 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-02-19T21:05:00.306 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-02-19T21:05:00.307 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: 2024-02-19T21:05:00.307 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: data: 2024-02-19T21:05:00.307 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: pools: 29 pools, 868 pgs 2024-02-19T21:05:00.307 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: objects: 192 objects, 5.7 KiB 2024-02-19T21:05:00.307 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:05:00.307 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: pgs: 79.954% pgs unknown 2024-02-19T21:05:00.307 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: 4.839% pgs not active 2024-02-19T21:05:00.308 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: 694 unknown 2024-02-19T21:05:00.308 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: 132 active+clean 2024-02-19T21:05:00.308 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: 42 creating+peering 2024-02-19T21:05:00.308 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: 2024-02-19T21:05:00.308 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: io: 2024-02-19T21:05:00.308 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: client: 639 B/s rd, 0 op/s rd, 0 op/s wr 2024-02-19T21:05:00.308 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: 2024-02-19T21:05:00.308 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: 2024-02-19T21:05:00.309 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: [ OK ] LibRadosService.StatusFormat (2230 ms) 2024-02-19T21:05:00.309 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: [ RUN ] LibRadosService.Status 2024-02-19T21:05:00.309 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: [ OK ] LibRadosService.Status (20048 ms) 2024-02-19T21:05:00.309 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: [----------] 4 tests from LibRadosService (27400 ms total) 2024-02-19T21:05:00.309 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: 2024-02-19T21:05:00.309 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: [----------] Global test environment tear-down 2024-02-19T21:05:00.309 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: [==========] 4 tests from 1 test suite ran. (27400 ms total) 2024-02-19T21:05:00.309 INFO:tasks.workunit.client.0.smithi163.stdout: api_service: [ PASSED ] 4 tests. 2024-02-19T21:05:00.310 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:00.328 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:58.772555+0000 mon.a (mon.0) 993 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:05:00.328 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:58.777447+0000 mon.a (mon.0) 994 : audit [INF] from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]': finished 2024-02-19T21:05:00.328 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:58.777544+0000 mon.a (mon.0) 995 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:05:00.328 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:58.777623+0000 mon.a (mon.0) 996 : audit [INF] from='client.29173 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:05:00.328 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:58.777660+0000 mon.a (mon.0) 997 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-02-19T21:05:00.328 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:58.798649+0000 mon.b (mon.1) 172 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-7"}]: dispatch 2024-02-19T21:05:00.328 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:58.799453+0000 mon.c (mon.2) 89 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:05:00.328 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:58.803302+0000 mon.a (mon.0) 998 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-7"}]: dispatch 2024-02-19T21:05:00.329 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:59.217937+0000 mon.a (mon.0) 999 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:05:00.329 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:04:59.410198+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:00.334 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:05:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:05:00.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:58.772555+0000 mon.a (mon.0) 993 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:05:00.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:58.777447+0000 mon.a (mon.0) 994 : audit [INF] from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi163-83976-12"}]': finished 2024-02-19T21:05:00.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:58.777544+0000 mon.a (mon.0) 995 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:05:00.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:58.777623+0000 mon.a (mon.0) 996 : audit [INF] from='client.29173 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:05:00.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:58.777660+0000 mon.a (mon.0) 997 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-02-19T21:05:00.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:58.798649+0000 mon.b (mon.1) 172 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-7"}]: dispatch 2024-02-19T21:05:00.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:58.799453+0000 mon.c (mon.2) 89 : audit [INF] from='client.? v1:172.21.15.163:0/319640210' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:05:00.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:58.803302+0000 mon.a (mon.0) 998 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-7"}]: dispatch 2024-02-19T21:05:00.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:59.217937+0000 mon.a (mon.0) 999 : audit [INF] from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi163-83976-12"}]: dispatch 2024-02-19T21:05:00.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:04:59.410198+0000 mon.c (mon.2) 90 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:00.336 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:00.356 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:00.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:00.459 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:00.491 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:01.062 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:01.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:01.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:01.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:05:01.202 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:01.224 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:05:01.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:01.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:05:01.327 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:01.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:58.987381+0000 mgr.y (mgr.14550) 160 : cluster [DBG] pgmap v150: 488 pgs: 64 unknown, 424 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:01.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:59.247523+0000 osd.7 (osd.7) 73 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:05:01.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:59.262392+0000 osd.7 (osd.7) 74 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:05:01.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:59.301772+0000 osd.4 (osd.4) 73 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:05:01.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:04:59.356917+0000 osd.4 (osd.4) 74 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:05:01.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:00.089563+0000 mgr.y (mgr.14550) 161 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:01.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:00.095281+0000 mon.a (mon.0) 1000 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-7"}]': finished 2024-02-19T21:05:01.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:00.095406+0000 mon.a (mon.0) 1001 : audit [INF] from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi163-83976-12"}]': finished 2024-02-19T21:05:01.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:00.095580+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-02-19T21:05:01.444 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:00.111305+0000 mon.b (mon.1) 173 : audit [INF] from='client.? v1:172.21.15.163:0/1305961758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:01.444 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:00.112264+0000 mon.b (mon.1) 174 : audit [INF] from='client.? v1:172.21.15.163:0/3429581000' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi163-83845-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:01.444 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:00.115346+0000 mon.a (mon.0) 1003 : audit [INF] from='client.29869 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:01.444 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:00.115684+0000 mon.a (mon.0) 1004 : audit [INF] from='client.29866 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi163-83845-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:01.444 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:00.281748+0000 osd.7 (osd.7) 75 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:05:01.444 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:00.284619+0000 osd.7 (osd.7) 76 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:05:01.444 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:00.330914+0000 mon.b (mon.1) 175 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:01.444 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:00.331350+0000 mon.a (mon.0) 1005 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:01.444 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:00.419562+0000 mon.c (mon.2) 91 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:01.444 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:58.987381+0000 mgr.y (mgr.14550) 160 : cluster [DBG] pgmap v150: 488 pgs: 64 unknown, 424 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:01.444 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:59.247523+0000 osd.7 (osd.7) 73 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:05:01.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:59.262392+0000 osd.7 (osd.7) 74 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:05:01.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:59.301772+0000 osd.4 (osd.4) 73 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:05:01.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:04:59.356917+0000 osd.4 (osd.4) 74 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:05:01.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:00.089563+0000 mgr.y (mgr.14550) 161 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:01.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:00.095281+0000 mon.a (mon.0) 1000 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-7"}]': finished 2024-02-19T21:05:01.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:00.095406+0000 mon.a (mon.0) 1001 : audit [INF] from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi163-83976-12"}]': finished 2024-02-19T21:05:01.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:00.095580+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-02-19T21:05:01.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:00.111305+0000 mon.b (mon.1) 173 : audit [INF] from='client.? v1:172.21.15.163:0/1305961758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:01.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:00.112264+0000 mon.b (mon.1) 174 : audit [INF] from='client.? v1:172.21.15.163:0/3429581000' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi163-83845-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:01.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:00.115346+0000 mon.a (mon.0) 1003 : audit [INF] from='client.29869 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:01.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:00.115684+0000 mon.a (mon.0) 1004 : audit [INF] from='client.29866 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi163-83845-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:01.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:00.281748+0000 osd.7 (osd.7) 75 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:05:01.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:00.284619+0000 osd.7 (osd.7) 76 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:05:01.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:00.330914+0000 mon.b (mon.1) 175 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:01.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:00.331350+0000 mon.a (mon.0) 1005 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:01.446 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:00.419562+0000 mon.c (mon.2) 91 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:01.456 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:01.503 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:01.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:01.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:01.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:58.987381+0000 mgr.y (mgr.14550) 160 : cluster [DBG] pgmap v150: 488 pgs: 64 unknown, 424 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:01.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:59.247523+0000 osd.7 (osd.7) 73 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:05:01.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:59.262392+0000 osd.7 (osd.7) 74 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:05:01.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:59.301772+0000 osd.4 (osd.4) 73 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:05:01.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:04:59.356917+0000 osd.4 (osd.4) 74 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:05:01.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:00.089563+0000 mgr.y (mgr.14550) 161 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:01.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:00.095281+0000 mon.a (mon.0) 1000 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-7"}]': finished 2024-02-19T21:05:01.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:00.095406+0000 mon.a (mon.0) 1001 : audit [INF] from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi163-83976-12"}]': finished 2024-02-19T21:05:01.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:00.095580+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-02-19T21:05:01.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:00.111305+0000 mon.b (mon.1) 173 : audit [INF] from='client.? v1:172.21.15.163:0/1305961758' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:00.112264+0000 mon.b (mon.1) 174 : audit [INF] from='client.? v1:172.21.15.163:0/3429581000' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi163-83845-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:00.115346+0000 mon.a (mon.0) 1003 : audit [INF] from='client.29869 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:00.115684+0000 mon.a (mon.0) 1004 : audit [INF] from='client.29866 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi163-83845-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:00.281748+0000 osd.7 (osd.7) 75 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:05:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:00.284619+0000 osd.7 (osd.7) 76 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:05:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:00.330914+0000 mon.b (mon.1) 175 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:00.331350+0000 mon.a (mon.0) 1005 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:00.419562+0000 mon.c (mon.2) 91 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:01.607 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:05:01.686 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:02.063 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:05:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:02.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:05:02.159 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-02-19T21:05:02.159 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-02-19T21:05:02.159 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-02-19T21:05:02.159 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-02-19T21:05:02.159 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-02-19T21:05:02.159 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-02-19T21:05:02.159 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-02-19T21:05:02.159 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: 2024-02-19T21:05:02.159 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-02-19T21:05:02.159 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-02-19T21:05:02.159 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (7982 ms) 2024-02-19T21:05:02.159 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-02-19T21:05:02.159 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (4865 ms) 2024-02-19T21:05:02.160 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-02-19T21:05:02.160 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (2872 ms) 2024-02-19T21:05:02.160 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-02-19T21:05:02.160 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3060 ms) 2024-02-19T21:05:02.160 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-02-19T21:05:02.160 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (2969 ms) 2024-02-19T21:05:02.160 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-02-19T21:05:02.160 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (4223 ms) 2024-02-19T21:05:02.160 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-02-19T21:05:02.160 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (3365 ms) 2024-02-19T21:05:02.160 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (29337 ms total) 2024-02-19T21:05:02.160 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: 2024-02-19T21:05:02.160 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-02-19T21:05:02.161 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (29337 ms total) 2024-02-19T21:05:02.161 INFO:tasks.workunit.client.0.smithi163.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-02-19T21:05:02.169 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:05:02.215 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:02.234 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:05:02.237 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:02.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:05:02.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:00.987878+0000 mgr.y (mgr.14550) 162 : cluster [DBG] pgmap v152: 584 pgs: 96 unknown, 488 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:05:02.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:01.105728+0000 mon.a (mon.0) 1006 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-19T21:05:02.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:01.142387+0000 mon.a (mon.0) 1007 : audit [INF] from='client.29869 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:02.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:01.142472+0000 mon.a (mon.0) 1008 : audit [INF] from='client.29866 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi163-83845-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:02.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:01.142570+0000 mon.a (mon.0) 1009 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:05:02.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:01.142615+0000 mon.a (mon.0) 1010 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-02-19T21:05:02.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:01.148288+0000 mon.b (mon.1) 176 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7"}]: dispatch 2024-02-19T21:05:02.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:01.161702+0000 mon.a (mon.0) 1011 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7"}]: dispatch 2024-02-19T21:05:02.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:01.367288+0000 osd.0 (osd.0) 73 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:05:02.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:01.368504+0000 osd.0 (osd.0) 74 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:05:02.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:01.420639+0000 mon.c (mon.2) 92 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:02.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:02.005813+0000 osd.3 (osd.3) 89 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:05:02.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:02.007645+0000 osd.3 (osd.3) 90 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:05:02.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:00.987878+0000 mgr.y (mgr.14550) 162 : cluster [DBG] pgmap v152: 584 pgs: 96 unknown, 488 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:05:02.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:01.105728+0000 mon.a (mon.0) 1006 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-19T21:05:02.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:01.142387+0000 mon.a (mon.0) 1007 : audit [INF] from='client.29869 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:02.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:01.142472+0000 mon.a (mon.0) 1008 : audit [INF] from='client.29866 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi163-83845-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:02.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:01.142570+0000 mon.a (mon.0) 1009 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:05:02.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:01.142615+0000 mon.a (mon.0) 1010 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-02-19T21:05:02.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:01.148288+0000 mon.b (mon.1) 176 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7"}]: dispatch 2024-02-19T21:05:02.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:01.161702+0000 mon.a (mon.0) 1011 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7"}]: dispatch 2024-02-19T21:05:02.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:01.367288+0000 osd.0 (osd.0) 73 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:05:02.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:01.368504+0000 osd.0 (osd.0) 74 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:05:02.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:01.420639+0000 mon.c (mon.2) 92 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:02.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:02.005813+0000 osd.3 (osd.3) 89 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:05:02.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:02.007645+0000 osd.3 (osd.3) 90 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:05:02.338 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:05:02.402 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:02.507 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:02.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:00.987878+0000 mgr.y (mgr.14550) 162 : cluster [DBG] pgmap v152: 584 pgs: 96 unknown, 488 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:05:02.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:01.105728+0000 mon.a (mon.0) 1006 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-02-19T21:05:02.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:01.142387+0000 mon.a (mon.0) 1007 : audit [INF] from='client.29869 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-84024-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:02.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:01.142472+0000 mon.a (mon.0) 1008 : audit [INF] from='client.29866 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi163-83845-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:02.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:01.142570+0000 mon.a (mon.0) 1009 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:05:02.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:01.142615+0000 mon.a (mon.0) 1010 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-02-19T21:05:02.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:01.148288+0000 mon.b (mon.1) 176 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7"}]: dispatch 2024-02-19T21:05:02.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:01.161702+0000 mon.a (mon.0) 1011 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7"}]: dispatch 2024-02-19T21:05:02.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:01.367288+0000 osd.0 (osd.0) 73 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:05:02.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:01.368504+0000 osd.0 (osd.0) 74 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:05:02.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:01.420639+0000 mon.c (mon.2) 92 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:02.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:02.005813+0000 osd.3 (osd.3) 89 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:05:02.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:02.007645+0000 osd.3 (osd.3) 90 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:05:02.977 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:05:03.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:03.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:05:03.119 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:05:03.151 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:05:03.264 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-02-19T21:05:03.265 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: [----------] Global test environment set-up. 2024-02-19T21:05:03.265 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-02-19T21:05:03.265 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-02-19T21:05:03.265 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5072 ms) 2024-02-19T21:05:03.265 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-02-19T21:05:03.265 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (30 ms) 2024-02-19T21:05:03.265 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-02-19T21:05:03.265 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20043 ms) 2024-02-19T21:05:03.265 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-02-19T21:05:03.265 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: attempt 0 of 20 2024-02-19T21:05:03.265 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: failed to deregister: 2024-02-19T21:05:03.265 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: { 2024-02-19T21:05:03.265 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "epoch": 18, 2024-02-19T21:05:03.265 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "modified": "2024-02-19T21:05:00.987579+0000", 2024-02-19T21:05:03.265 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "services": { 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "laundry": { 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "daemons": { 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "summary": "", 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "close-test-pid84016": { 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "start_epoch": 17, 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "start_stamp": "2024-02-19T21:04:58.130435+0000", 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "gid": 29374, 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "addr": "172.21.15.163:0/1919640627", 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "metadata": { 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "arch": "x86_64", 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "ceph_release": "pacific", 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "ceph_version": "ceph version 16.2.14-548-g81bd20d6 (81bd20d634209c7cb82c18be12b4b5a05643ebf1) pacific (stable)", 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "ceph_version_short": "16.2.14-548-g81bd20d6", 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "distro": "centos", 2024-02-19T21:05:03.266 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "distro_description": "CentOS Stream 8", 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "distro_version": "8", 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "foo": "bar", 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "hostname": "smithi163", 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "kernel_description": "#1 SMP Mon Feb 5 18:34:56 UTC 2024", 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "kernel_version": "4.18.0-539.el8.x86_64", 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "mem_swap_kb": "0", 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "mem_total_kb": "32537628", 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "os": "Linux", 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "this": "that" 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: }, 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "task_status": {} 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: } 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: } 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: }, 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "rgw": { 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "daemons": { 2024-02-19T21:05:03.267 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "summary": "", 2024-02-19T21:05:03.268 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "24400": { 2024-02-19T21:05:03.268 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "start_epoch": 12, 2024-02-19T21:05:03.268 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "start_stamp": "2024-02-19T21:01:23.376647+0000", 2024-02-19T21:05:03.268 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "gid": 24400, 2024-02-19T21:05:03.268 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "addr": "172.21.15.163:0/3997262145", 2024-02-19T21:05:03.268 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "metadata": { 2024-02-19T21:05:03.268 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "arch": "x86_64", 2024-02-19T21:05:03.268 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "ceph_release": "pacific", 2024-02-19T21:05:03.304 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:03.331 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:05:03.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:03.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:05:03.365 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:02.157154+0000 mon.a (mon.0) 1012 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7"}]': finished 2024-02-19T21:05:03.365 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:02.157312+0000 mon.a (mon.0) 1013 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-02-19T21:05:03.366 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:02.176255+0000 mon.b (mon.1) 177 : audit [INF] from='client.? v1:172.21.15.163:0/2722908877' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:03.366 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:02.178107+0000 mon.a (mon.0) 1014 : audit [INF] from='client.31900 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:03.366 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:02.202221+0000 mon.b (mon.1) 178 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:03.366 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:02.204411+0000 mon.a (mon.0) 1015 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:03.366 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:02.206028+0000 mon.b (mon.1) 179 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:03.366 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:02.219734+0000 mon.a (mon.0) 1016 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:03.366 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:02.220278+0000 mon.b (mon.1) 180 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:03.366 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:02.220759+0000 mon.a (mon.0) 1017 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:03.366 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:02.425728+0000 mon.c (mon.2) 93 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:03.436 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:05:03.601 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:03.626 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:05:03] "GET /metrics HTTP/1.1" 200 210092 "" "Prometheus/2.33.4" 2024-02-19T21:05:03.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:02.157154+0000 mon.a (mon.0) 1012 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7"}]': finished 2024-02-19T21:05:03.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:02.157312+0000 mon.a (mon.0) 1013 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-02-19T21:05:03.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:02.176255+0000 mon.b (mon.1) 177 : audit [INF] from='client.? v1:172.21.15.163:0/2722908877' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:03.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:02.178107+0000 mon.a (mon.0) 1014 : audit [INF] from='client.31900 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:03.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:02.202221+0000 mon.b (mon.1) 178 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:03.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:02.204411+0000 mon.a (mon.0) 1015 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:03.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:02.206028+0000 mon.b (mon.1) 179 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:03.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:02.219734+0000 mon.a (mon.0) 1016 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:03.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:02.220278+0000 mon.b (mon.1) 180 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:03.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:02.220759+0000 mon.a (mon.0) 1017 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:03.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:02.425728+0000 mon.c (mon.2) 93 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:03.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:02.157154+0000 mon.a (mon.0) 1012 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-7"}]': finished 2024-02-19T21:05:03.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:02.157312+0000 mon.a (mon.0) 1013 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-02-19T21:05:03.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:02.176255+0000 mon.b (mon.1) 177 : audit [INF] from='client.? v1:172.21.15.163:0/2722908877' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:03.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:02.178107+0000 mon.a (mon.0) 1014 : audit [INF] from='client.31900 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:03.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:02.202221+0000 mon.b (mon.1) 178 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:03.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:02.204411+0000 mon.a (mon.0) 1015 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:03.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:02.206028+0000 mon.b (mon.1) 179 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:03.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:02.219734+0000 mon.a (mon.0) 1016 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:03.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:02.220278+0000 mon.b (mon.1) 180 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:03.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:02.220759+0000 mon.a (mon.0) 1017 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:03.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:02.425728+0000 mon.c (mon.2) 93 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:03.960 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:05:03.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:03.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:05:04.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.049+0000 7f0a0490d700 1 -- 172.21.15.163:0/3393375204 >> v1:172.21.15.193:6789/0 conn(0x7f0a001031b0 legacy=0x7f0a00105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.049+0000 7f0a0490d700 1 -- 172.21.15.163:0/3393375204 shutdown_connections 2024-02-19T21:05:04.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.049+0000 7f0a0490d700 1 -- 172.21.15.163:0/3393375204 >> 172.21.15.163:0/3393375204 conn(0x7f0a000f2470 msgr2=0x7f0a000f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:04.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.049+0000 7f0a0490d700 1 -- 172.21.15.163:0/3393375204 shutdown_connections 2024-02-19T21:05:04.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.049+0000 7f0a0490d700 1 -- 172.21.15.163:0/3393375204 wait complete. 2024-02-19T21:05:04.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.050+0000 7f0a0490d700 1 librados: starting msgr at 2024-02-19T21:05:04.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.050+0000 7f0a0490d700 1 librados: starting objecter 2024-02-19T21:05:04.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.050+0000 7f0a0490d700 1 Processor -- start 2024-02-19T21:05:04.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.050+0000 7f0a0490d700 1 -- start start 2024-02-19T21:05:04.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.050+0000 7f0a0490d700 1 librados: setting wanted keys 2024-02-19T21:05:04.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.050+0000 7f0a0490d700 1 librados: calling monclient init 2024-02-19T21:05:04.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f0a0490d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a001195b0 con 0x7f0a001031b0 2024-02-19T21:05:04.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f0a0490d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a0011a790 con 0x7f0a000fc870 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f0a0490d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0a0011b990 con 0x7f0a000ff670 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f09fed9d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f0a001031b0 0x7f0a00117c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:42224/0 (socket says 172.21.15.163:42224) 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f09fed9d700 1 -- 172.21.15.163:0/2032652544 learned_addr learned my addr 172.21.15.163:0/2032652544 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f09f77fe700 10 client.?.objecter ms_handle_connect 0x7f0a001031b0 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f09f77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f09f77fe700 10 client.?.objecter ms_handle_connect 0x7f0a000fc870 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f09f77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f09f77fe700 1 -- 172.21.15.163:0/2032652544 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811817627 0 0) 0x7f0a0011a790 con 0x7f0a000fc870 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f09f77fe700 1 -- 172.21.15.163:0/2032652544 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09ec002220 con 0x7f0a000fc870 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f09f77fe700 1 -- 172.21.15.163:0/2032652544 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3833494409 0 0) 0x7f09ec002220 con 0x7f0a000fc870 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f09f77fe700 1 -- 172.21.15.163:0/2032652544 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a0011a790 con 0x7f0a000fc870 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f09f77fe700 1 -- 172.21.15.163:0/2032652544 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f09e8001880 con 0x7f0a000fc870 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f09f77fe700 1 -- 172.21.15.163:0/2032652544 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 404784734 0 0) 0x7f0a0011a790 con 0x7f0a000fc870 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f09f77fe700 1 -- 172.21.15.163:0/2032652544 >> v1:172.21.15.163:6790/0 conn(0x7f0a000ff670 legacy=0x7f0a000610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.051+0000 7f09f77fe700 1 -- 172.21.15.163:0/2032652544 >> v1:172.21.15.163:6789/0 conn(0x7f0a001031b0 legacy=0x7f0a00117c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.052+0000 7f09f77fe700 1 -- 172.21.15.163:0/2032652544 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0011cb90 con 0x7f0a000fc870 2024-02-19T21:05:04.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.052+0000 7f0a0490d700 1 -- 172.21.15.163:0/2032652544 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0011a980 con 0x7f0a000fc870 2024-02-19T21:05:04.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.052+0000 7f0a0490d700 10 client.33955.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:04.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.052+0000 7f0a0490d700 1 -- 172.21.15.163:0/2032652544 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0011adf0 con 0x7f0a000fc870 2024-02-19T21:05:04.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.052+0000 7f0a0490d700 1 librados: init done 2024-02-19T21:05:04.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.052+0000 7f09f77fe700 1 -- 172.21.15.163:0/2032652544 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f09e8001f40 con 0x7f0a000fc870 2024-02-19T21:05:04.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.052+0000 7f09f77fe700 1 -- 172.21.15.163:0/2032652544 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f09e80032f0 con 0x7f0a000fc870 2024-02-19T21:05:04.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.053+0000 7f0a0490d700 1 -- 172.21.15.163:0/2032652544 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e0004f50 con 0x7f0a000fc870 2024-02-19T21:05:04.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.053+0000 7f09f77fe700 1 -- 172.21.15.163:0/2032652544 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f09e80026e0 con 0x7f0a000fc870 2024-02-19T21:05:04.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.054+0000 7f09f77fe700 1 -- 172.21.15.163:0/2032652544 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(124..124 src has 43..124) v4 ==== 11420+0+0 (unknown 2322591081 0 0) 0x7f09e8077330 con 0x7f0a000fc870 2024-02-19T21:05:04.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.054+0000 7f09f77fe700 10 client.33955.objecter ms_dispatch 0x7f0a00001550 osd_map(124..124 src has 43..124) v4 2024-02-19T21:05:04.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.054+0000 7f09f77fe700 3 client.33955.objecter handle_osd_map got epochs [124,124] > 0 2024-02-19T21:05:04.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.054+0000 7f09f77fe700 3 client.33955.objecter handle_osd_map decoding full epoch 124 2024-02-19T21:05:04.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.054+0000 7f09f77fe700 20 client.33955.objecter dump_active .. 0 homeless 2024-02-19T21:05:04.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.054+0000 7f09f77fe700 10 client.33955.objecter ms_handle_connect 0x7f09ec063b30 2024-02-19T21:05:04.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.058+0000 7f09f77fe700 1 -- 172.21.15.163:0/2032652544 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f09e8049a00 con 0x7f0a000fc870 2024-02-19T21:05:04.081 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:04.107 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:04.127 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:04.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:04.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:04.230 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:04.257 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:04.277 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:04.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:04.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:04.381 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:04.409 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:04.429 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:04.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:04.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:04.533 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:05:04.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.542+0000 7f0a0490d700 1 -- 172.21.15.163:0/2032652544 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f09e0000bc0 con 0x7f09ec063b30 2024-02-19T21:05:04.596 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:01.237613+0000 osd.4 (osd.4) 75 : cluster [DBG] 2.f scrub starts 2024-02-19T21:05:04.596 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:01.259307+0000 osd.4 (osd.4) 76 : cluster [DBG] 2.f scrub ok 2024-02-19T21:05:04.596 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:01.684023+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.14 deep-scrub starts 2024-02-19T21:05:04.596 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:01.684951+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.14 deep-scrub ok 2024-02-19T21:05:04.596 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:01.852079+0000 osd.1 (osd.1) 77 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:05:04.596 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:01.858914+0000 osd.1 (osd.1) 78 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:05:04.596 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:03.092132+0000 mgr.y (mgr.14550) 163 : cluster [DBG] pgmap v155: 424 pgs: 32 unknown, 392 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 949 B/s rd, 1 op/s 2024-02-19T21:05:04.596 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:03.096168+0000 osd.1 (osd.1) 79 : cluster [DBG] 4.f scrub starts 2024-02-19T21:05:04.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:03.209867+0000 osd.1 (osd.1) 80 : cluster [DBG] 4.f scrub ok 2024-02-19T21:05:04.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:03.245009+0000 osd.7 (osd.7) 77 : cluster [DBG] 6.3 deep-scrub starts 2024-02-19T21:05:04.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:03.248502+0000 osd.7 (osd.7) 78 : cluster [DBG] 6.3 deep-scrub ok 2024-02-19T21:05:04.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:03.251858+0000 mon.c (mon.2) 94 : audit [DBG] from='client.? v1:172.21.15.163:0/3575209332' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-19T21:05:04.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:03.253059+0000 mgr.y (mgr.14550) 164 : audit [DBG] from='mon.? v1:172.21.15.163:0/3010617610' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-19T21:05:04.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:03.361859+0000 mon.a (mon.0) 1018 : audit [INF] from='client.31900 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:04.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:03.361972+0000 mon.a (mon.0) 1019 : audit [INF] from='client.32629 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:05:04.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:03.362173+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-02-19T21:05:04.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:03.373574+0000 mon.b (mon.1) 181 : audit [INF] from='client.? v1:172.21.15.163:0/1301771977' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi163-83854-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:04.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:03.373857+0000 mon.b (mon.1) 182 : audit [INF] from='client.? v1:172.21.15.163:0/1979745060' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi163-83845-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:04.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:03.375005+0000 mon.a (mon.0) 1021 : audit [INF] from='client.32662 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi163-83854-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:04.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:03.375293+0000 mon.a (mon.0) 1022 : audit [INF] from='client.32650 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi163-83845-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:04.598 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:03.391299+0000 mon.b (mon.1) 183 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi163-83944-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:04.598 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:03.435963+0000 mon.c (mon.2) 95 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:04.598 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:03.731929+0000 mon.a (mon.0) 1023 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi163-83944-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:04.598 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:03.814331+0000 mon.a (mon.0) 1024 : audit [INF] from='client.32662 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi163-83854-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:04.598 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:03.814426+0000 mon.a (mon.0) 1025 : audit [INF] from='client.32650 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi163-83845-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:04.598 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:03.815876+0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-02-19T21:05:04.598 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:04.181076+0000 mon.b (mon.1) 184 : audit [DBG] from='client.? v1:172.21.15.163:0/1173853636' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:04.598 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:01.237613+0000 osd.4 (osd.4) 75 : cluster [DBG] 2.f scrub starts 2024-02-19T21:05:04.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:01.259307+0000 osd.4 (osd.4) 76 : cluster [DBG] 2.f scrub ok 2024-02-19T21:05:04.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:01.684023+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.14 deep-scrub starts 2024-02-19T21:05:04.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:01.684951+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.14 deep-scrub ok 2024-02-19T21:05:04.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:01.852079+0000 osd.1 (osd.1) 77 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:05:04.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:01.858914+0000 osd.1 (osd.1) 78 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:05:04.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:03.092132+0000 mgr.y (mgr.14550) 163 : cluster [DBG] pgmap v155: 424 pgs: 32 unknown, 392 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 949 B/s rd, 1 op/s 2024-02-19T21:05:04.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:03.096168+0000 osd.1 (osd.1) 79 : cluster [DBG] 4.f scrub starts 2024-02-19T21:05:04.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:03.209867+0000 osd.1 (osd.1) 80 : cluster [DBG] 4.f scrub ok 2024-02-19T21:05:04.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:03.245009+0000 osd.7 (osd.7) 77 : cluster [DBG] 6.3 deep-scrub starts 2024-02-19T21:05:04.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:03.248502+0000 osd.7 (osd.7) 78 : cluster [DBG] 6.3 deep-scrub ok 2024-02-19T21:05:04.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:03.251858+0000 mon.c (mon.2) 94 : audit [DBG] from='client.? v1:172.21.15.163:0/3575209332' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-19T21:05:04.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:03.253059+0000 mgr.y (mgr.14550) 164 : audit [DBG] from='mon.? v1:172.21.15.163:0/3010617610' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-19T21:05:04.600 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:03.361859+0000 mon.a (mon.0) 1018 : audit [INF] from='client.31900 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:04.600 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:03.361972+0000 mon.a (mon.0) 1019 : audit [INF] from='client.32629 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:05:04.600 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:03.362173+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-02-19T21:05:04.600 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:03.373574+0000 mon.b (mon.1) 181 : audit [INF] from='client.? v1:172.21.15.163:0/1301771977' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi163-83854-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:04.600 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:03.373857+0000 mon.b (mon.1) 182 : audit [INF] from='client.? v1:172.21.15.163:0/1979745060' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi163-83845-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:04.600 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:03.375005+0000 mon.a (mon.0) 1021 : audit [INF] from='client.32662 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi163-83854-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:04.600 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:03.375293+0000 mon.a (mon.0) 1022 : audit [INF] from='client.32650 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi163-83845-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:04.600 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:03.391299+0000 mon.b (mon.1) 183 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi163-83944-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:04.600 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:03.435963+0000 mon.c (mon.2) 95 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:04.600 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:03.731929+0000 mon.a (mon.0) 1023 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi163-83944-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:04.600 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:03.814331+0000 mon.a (mon.0) 1024 : audit [INF] from='client.32662 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi163-83854-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:04.600 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:03.814426+0000 mon.a (mon.0) 1025 : audit [INF] from='client.32650 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi163-83845-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:04.600 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:03.815876+0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-02-19T21:05:04.601 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:04.181076+0000 mon.b (mon.1) 184 : audit [DBG] from='client.? v1:172.21.15.163:0/1173853636' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:04.612 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:04.635 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:05:04.636 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:05:04.636 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:05:04.636 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:05:04.636 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:05:04.636 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:05:04.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.174+0000 7eff5dd63700 1 -- 172.21.15.163:0/3575193125 >> v1:172.21.15.193:6789/0 conn(0x7eff580693f0 legacy=0x7eff580697d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.174+0000 7eff5dd63700 1 -- 172.21.15.163:0/3575193125 shutdown_connections 2024-02-19T21:05:04.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.174+0000 7eff5dd63700 1 -- 172.21.15.163:0/3575193125 >> 172.21.15.163:0/3575193125 conn(0x7eff58063b20 msgr2=0x7eff58067230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:04.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.174+0000 7eff5dd63700 1 -- 172.21.15.163:0/3575193125 shutdown_connections 2024-02-19T21:05:04.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.174+0000 7eff5dd63700 1 -- 172.21.15.163:0/3575193125 wait complete. 2024-02-19T21:05:04.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.175+0000 7eff5dd63700 1 librados: starting msgr at 2024-02-19T21:05:04.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.175+0000 7eff5dd63700 1 librados: starting objecter 2024-02-19T21:05:04.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.175+0000 7eff5dd63700 1 Processor -- start 2024-02-19T21:05:04.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.175+0000 7eff5dd63700 1 -- start start 2024-02-19T21:05:04.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.175+0000 7eff5dd63700 1 librados: setting wanted keys 2024-02-19T21:05:04.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.175+0000 7eff5dd63700 1 librados: calling monclient init 2024-02-19T21:05:04.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.176+0000 7eff5dd63700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7eff5810c6c0 con 0x7eff58113250 2024-02-19T21:05:04.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.176+0000 7eff5dd63700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7eff5812fe30 con 0x7eff580693f0 2024-02-19T21:05:04.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.176+0000 7eff5dd63700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7eff58130fd0 con 0x7eff58069ee0 2024-02-19T21:05:04.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.176+0000 7eff577fe700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7eff580693f0 0x7eff5810bc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:41212/0 (socket says 172.21.15.163:41212) 2024-02-19T21:05:04.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.176+0000 7eff577fe700 1 -- 172.21.15.163:0/1173853636 learned_addr learned my addr 172.21.15.163:0/1173853636 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:04.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.176+0000 7eff54ff9700 10 client.?.objecter ms_handle_connect 0x7eff580693f0 2024-02-19T21:05:04.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.176+0000 7eff54ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:04.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.176+0000 7eff54ff9700 10 client.?.objecter ms_handle_connect 0x7eff58069ee0 2024-02-19T21:05:04.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.176+0000 7eff54ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:04.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.176+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719092703 0 0) 0x7eff5812fe30 con 0x7eff580693f0 2024-02-19T21:05:04.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.176+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff40002220 con 0x7eff580693f0 2024-02-19T21:05:04.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.177+0000 7eff54ff9700 10 client.?.objecter ms_handle_connect 0x7eff58113250 2024-02-19T21:05:04.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.177+0000 7eff54ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:04.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.177+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3603844568 0 0) 0x7eff40002220 con 0x7eff580693f0 2024-02-19T21:05:04.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.177+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff5812fe30 con 0x7eff580693f0 2024-02-19T21:05:04.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.177+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7eff48001720 con 0x7eff580693f0 2024-02-19T21:05:04.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.177+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1937030646 0 0) 0x7eff5812fe30 con 0x7eff580693f0 2024-02-19T21:05:04.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.177+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 >> v1:172.21.15.163:6790/0 conn(0x7eff58069ee0 legacy=0x7eff581119f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.177+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 >> v1:172.21.15.163:6789/0 conn(0x7eff58113250 legacy=0x7eff5812d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.177+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff581321b0 con 0x7eff580693f0 2024-02-19T21:05:04.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.177+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff581311c0 con 0x7eff580693f0 2024-02-19T21:05:04.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.177+0000 7eff5dd63700 10 client.34084.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:04.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.177+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff58131670 con 0x7eff580693f0 2024-02-19T21:05:04.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.177+0000 7eff5dd63700 1 librados: init done 2024-02-19T21:05:04.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.177+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7eff48002310 con 0x7eff580693f0 2024-02-19T21:05:04.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.177+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7eff480037f0 con 0x7eff580693f0 2024-02-19T21:05:04.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.179+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7eff44004f30 con 0x7eff580693f0 2024-02-19T21:05:04.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.179+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7eff48003a50 con 0x7eff580693f0 2024-02-19T21:05:04.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(124..124 src has 43..124) v4 ==== 11420+0+0 (unknown 2322591081 0 0) 0x7eff48079e10 con 0x7eff580693f0 2024-02-19T21:05:04.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 osd_map(124..124 src has 43..124) v4 2024-02-19T21:05:04.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff54ff9700 3 client.34084.objecter handle_osd_map got epochs [124,124] > 0 2024-02-19T21:05:04.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff54ff9700 3 client.34084.objecter handle_osd_map decoding full epoch 124 2024-02-19T21:05:04.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff54ff9700 20 client.34084.objecter dump_active .. 0 homeless 2024-02-19T21:05:04.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v124) v1 ==== 54+0+16 (unknown 1125300689 0 2420000192) 0x7eff4804d2f0 con 0x7eff580693f0 2024-02-19T21:05:04.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff5dd63700 10 client.34084.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:04.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 1 2024-02-19T21:05:04.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff440051d0 osd=0 3 2024-02-19T21:05:04.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff440051d0 osd=0 3 2024-02-19T21:05:04.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff440051d0 osd=0 3 2024-02-19T21:05:04.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 1 2024-02-19T21:05:04.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff440051d0 osd=0 3 2024-02-19T21:05:04.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 0 1 2024-02-19T21:05:04.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff440051d0 osd=0 4 2024-02-19T21:05:04.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff5dd63700 10 client.34084.objecter _send_command 1 2024-02-19T21:05:04.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff44004980 con 0x7eff44004560 2024-02-19T21:05:04.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.180+0000 7eff54ff9700 10 client.34084.objecter ms_handle_connect 0x7eff40063b30 2024-02-19T21:05:04.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.181+0000 7eff54ff9700 10 client.34084.objecter ms_handle_connect 0x7eff44004560 2024-02-19T21:05:04.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.182+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7eff44004980 con 0x7eff44004560 2024-02-19T21:05:04.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.182+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:05:04.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.182+0000 7eff54ff9700 10 client.34084.objecter _finish_command 1 = system:0 2024-02-19T21:05:04.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.182+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff440051d0 osd=0 4 2024-02-19T21:05:04.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.182+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 0 1 2024-02-19T21:05:04.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.204+0000 7eff5dd63700 10 client.34084.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:04.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.204+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 2 2024-02-19T21:05:04.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.204+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff440051d0 osd=0 3 2024-02-19T21:05:04.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.204+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff440051d0 osd=0 3 2024-02-19T21:05:04.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.204+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff440051d0 osd=0 3 2024-02-19T21:05:04.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.204+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 2 2024-02-19T21:05:04.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.204+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff440051d0 osd=0 3 2024-02-19T21:05:04.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.204+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 0 2 2024-02-19T21:05:04.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.204+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff440051d0 osd=0 4 2024-02-19T21:05:04.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.204+0000 7eff5dd63700 10 client.34084.objecter _send_command 2 2024-02-19T21:05:04.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.204+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7eff44004d60 con 0x7eff44004560 2024-02-19T21:05:04.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.206+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7eff44004d60 con 0x7eff44004560 2024-02-19T21:05:04.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.206+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:04.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.206+0000 7eff54ff9700 10 client.34084.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.206+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff440051d0 osd=0 4 2024-02-19T21:05:04.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.206+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 0 2 2024-02-19T21:05:04.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.207+0000 7eff5dd63700 10 client.34084.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:04.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.207+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 3 2024-02-19T21:05:04.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.207+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44004010 osd=1 3 2024-02-19T21:05:04.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.207+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44004010 osd=1 3 2024-02-19T21:05:04.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.207+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44004010 osd=1 3 2024-02-19T21:05:04.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.207+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 3 2024-02-19T21:05:04.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.207+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff44004010 osd=1 3 2024-02-19T21:05:04.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.207+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 1 3 2024-02-19T21:05:04.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.207+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44004010 osd=1 4 2024-02-19T21:05:04.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.207+0000 7eff5dd63700 10 client.34084.objecter _send_command 3 2024-02-19T21:05:04.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.207+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff4400c030 con 0x7eff44004180 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.210+0000 7eff54ff9700 10 client.34084.objecter ms_handle_connect 0x7eff44004180 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.212+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7eff4400c030 con 0x7eff44004180 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.212+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.212+0000 7eff54ff9700 10 client.34084.objecter _finish_command 3 = system:0 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.212+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff44004010 osd=1 4 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.212+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 1 3 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.292+0000 7eff5dd63700 10 client.34084.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.292+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 4 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.292+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44004010 osd=1 3 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.292+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44004010 osd=1 3 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.292+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44004010 osd=1 3 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.292+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 4 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.292+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff44004010 osd=1 3 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.292+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 1 4 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.292+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44004010 osd=1 4 2024-02-19T21:05:04.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.292+0000 7eff5dd63700 10 client.34084.objecter _send_command 4 2024-02-19T21:05:04.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.292+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7eff44001880 con 0x7eff44004180 2024-02-19T21:05:04.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.312+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7eff44001880 con 0x7eff44004180 2024-02-19T21:05:04.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.312+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:04.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.312+0000 7eff54ff9700 10 client.34084.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.312+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff44004010 osd=1 4 2024-02-19T21:05:04.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.312+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 1 4 2024-02-19T21:05:04.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.316+0000 7eff5dd63700 10 client.34084.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:04.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.316+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 5 2024-02-19T21:05:04.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.316+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44003660 osd=2 3 2024-02-19T21:05:04.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.316+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44003660 osd=2 3 2024-02-19T21:05:04.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.316+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44003660 osd=2 3 2024-02-19T21:05:04.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.316+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 5 2024-02-19T21:05:04.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.316+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff44003660 osd=2 3 2024-02-19T21:05:04.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.316+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 2 5 2024-02-19T21:05:04.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.316+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44003660 osd=2 4 2024-02-19T21:05:04.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.316+0000 7eff5dd63700 10 client.34084.objecter _send_command 5 2024-02-19T21:05:04.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.316+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff44003ea0 con 0x7eff4400d320 2024-02-19T21:05:04.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.317+0000 7eff54ff9700 10 client.34084.objecter ms_handle_connect 0x7eff4400d320 2024-02-19T21:05:04.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.318+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7eff44003ea0 con 0x7eff4400d320 2024-02-19T21:05:04.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.318+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:05:04.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.318+0000 7eff54ff9700 10 client.34084.objecter _finish_command 5 = system:0 2024-02-19T21:05:04.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.318+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff44003660 osd=2 4 2024-02-19T21:05:04.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.318+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 2 5 2024-02-19T21:05:04.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.346+0000 7eff5dd63700 10 client.34084.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:04.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.346+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 6 2024-02-19T21:05:04.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.346+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44003660 osd=2 3 2024-02-19T21:05:04.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.346+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44003660 osd=2 3 2024-02-19T21:05:04.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.346+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44003660 osd=2 3 2024-02-19T21:05:04.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.346+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 6 2024-02-19T21:05:04.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.346+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff44003660 osd=2 3 2024-02-19T21:05:04.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.346+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 2 6 2024-02-19T21:05:04.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.346+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44003660 osd=2 4 2024-02-19T21:05:04.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.346+0000 7eff5dd63700 10 client.34084.objecter _send_command 6 2024-02-19T21:05:04.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.346+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7eff4400e780 con 0x7eff4400d320 2024-02-19T21:05:04.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.348+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7eff4400e780 con 0x7eff4400d320 2024-02-19T21:05:04.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.348+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:04.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.348+0000 7eff54ff9700 10 client.34084.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.348+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff44003660 osd=2 4 2024-02-19T21:05:04.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.348+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 2 6 2024-02-19T21:05:04.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.349+0000 7eff5dd63700 10 client.34084.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:04.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.349+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 7 2024-02-19T21:05:04.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.349+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44003440 osd=3 3 2024-02-19T21:05:04.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.349+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44003440 osd=3 3 2024-02-19T21:05:04.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.349+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44003440 osd=3 3 2024-02-19T21:05:04.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.349+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 7 2024-02-19T21:05:04.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.349+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff44003440 osd=3 3 2024-02-19T21:05:04.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.349+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 3 7 2024-02-19T21:05:04.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.349+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44003440 osd=3 4 2024-02-19T21:05:04.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.349+0000 7eff5dd63700 10 client.34084.objecter _send_command 7 2024-02-19T21:05:04.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.349+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff44015b30 con 0x7eff44011eb0 2024-02-19T21:05:04.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.349+0000 7eff54ff9700 10 client.34084.objecter ms_handle_connect 0x7eff44011eb0 2024-02-19T21:05:04.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.350+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7eff44015b30 con 0x7eff44011eb0 2024-02-19T21:05:04.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.350+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:05:04.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.350+0000 7eff54ff9700 10 client.34084.objecter _finish_command 7 = system:0 2024-02-19T21:05:04.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.350+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff44003440 osd=3 4 2024-02-19T21:05:04.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.350+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 3 7 2024-02-19T21:05:04.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.384+0000 7eff5dd63700 10 client.34084.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:04.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.384+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 8 2024-02-19T21:05:04.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.384+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44003440 osd=3 3 2024-02-19T21:05:04.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.384+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44003440 osd=3 3 2024-02-19T21:05:04.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.384+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44003440 osd=3 3 2024-02-19T21:05:04.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.384+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 8 2024-02-19T21:05:04.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.384+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff44003440 osd=3 3 2024-02-19T21:05:04.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.384+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 3 8 2024-02-19T21:05:04.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.384+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44003440 osd=3 4 2024-02-19T21:05:04.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.384+0000 7eff5dd63700 10 client.34084.objecter _send_command 8 2024-02-19T21:05:04.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.384+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7eff4400e1f0 con 0x7eff44011eb0 2024-02-19T21:05:04.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.399+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7eff4400e1f0 con 0x7eff44011eb0 2024-02-19T21:05:04.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.399+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:04.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.399+0000 7eff54ff9700 10 client.34084.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.399+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff44003440 osd=3 4 2024-02-19T21:05:04.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.399+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 3 8 2024-02-19T21:05:04.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.412+0000 7eff5dd63700 10 client.34084.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:04.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.412+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 9 2024-02-19T21:05:04.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.412+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff4400e1f0 osd=4 3 2024-02-19T21:05:04.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.412+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff4400e1f0 osd=4 3 2024-02-19T21:05:04.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.412+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff4400e1f0 osd=4 3 2024-02-19T21:05:04.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.412+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 9 2024-02-19T21:05:04.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.412+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff4400e1f0 osd=4 3 2024-02-19T21:05:04.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.412+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 4 9 2024-02-19T21:05:04.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.412+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff4400e1f0 osd=4 4 2024-02-19T21:05:04.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.412+0000 7eff5dd63700 10 client.34084.objecter _send_command 9 2024-02-19T21:05:04.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.412+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff4400e520 con 0x7eff4400ce80 2024-02-19T21:05:04.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.416+0000 7eff54ff9700 10 client.34084.objecter ms_handle_connect 0x7eff4400ce80 2024-02-19T21:05:04.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.416+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7eff4400e520 con 0x7eff4400ce80 2024-02-19T21:05:04.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.416+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:05:04.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.416+0000 7eff54ff9700 10 client.34084.objecter _finish_command 9 = system:0 2024-02-19T21:05:04.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.416+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff4400e1f0 osd=4 4 2024-02-19T21:05:04.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.416+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 4 9 2024-02-19T21:05:04.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.523+0000 7eff5dd63700 10 client.34084.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:04.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.523+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 10 2024-02-19T21:05:04.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.523+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff4400e1f0 osd=4 3 2024-02-19T21:05:04.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.523+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff4400e1f0 osd=4 3 2024-02-19T21:05:04.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.523+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff4400e1f0 osd=4 3 2024-02-19T21:05:04.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.523+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 10 2024-02-19T21:05:04.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.523+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff4400e1f0 osd=4 3 2024-02-19T21:05:04.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.523+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 4 10 2024-02-19T21:05:04.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.523+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff4400e1f0 osd=4 4 2024-02-19T21:05:04.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.523+0000 7eff5dd63700 10 client.34084.objecter _send_command 10 2024-02-19T21:05:04.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.523+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7eff4400e520 con 0x7eff4400ce80 2024-02-19T21:05:04.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.539+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7eff4400e520 con 0x7eff4400ce80 2024-02-19T21:05:04.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.539+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:04.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.539+0000 7eff54ff9700 10 client.34084.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.539+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff4400e1f0 osd=4 4 2024-02-19T21:05:04.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.539+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 4 10 2024-02-19T21:05:04.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.540+0000 7eff5dd63700 10 client.34084.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:04.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.540+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 11 2024-02-19T21:05:04.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.540+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff4400e520 osd=5 3 2024-02-19T21:05:04.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.540+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff4400e520 osd=5 3 2024-02-19T21:05:04.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.540+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff4400e520 osd=5 3 2024-02-19T21:05:04.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.540+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 11 2024-02-19T21:05:04.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.540+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff4400e520 osd=5 3 2024-02-19T21:05:04.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.540+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 5 11 2024-02-19T21:05:04.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.540+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff4400e520 osd=5 4 2024-02-19T21:05:04.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.540+0000 7eff5dd63700 10 client.34084.objecter _send_command 11 2024-02-19T21:05:04.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.540+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff44018080 con 0x7eff44017ab0 2024-02-19T21:05:04.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.541+0000 7eff54ff9700 10 client.34084.objecter ms_handle_connect 0x7eff44017ab0 2024-02-19T21:05:04.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.541+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7eff44018080 con 0x7eff44017ab0 2024-02-19T21:05:04.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.541+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:05:04.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.541+0000 7eff54ff9700 10 client.34084.objecter _finish_command 11 = system:0 2024-02-19T21:05:04.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.541+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff4400e520 osd=5 4 2024-02-19T21:05:04.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.541+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 5 11 2024-02-19T21:05:04.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.596+0000 7eff5dd63700 10 client.34084.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:04.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.596+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 12 2024-02-19T21:05:04.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.596+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff4400e520 osd=5 3 2024-02-19T21:05:04.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.596+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff4400e520 osd=5 3 2024-02-19T21:05:04.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.596+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff4400e520 osd=5 3 2024-02-19T21:05:04.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.596+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 12 2024-02-19T21:05:04.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.596+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff4400e520 osd=5 3 2024-02-19T21:05:04.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.596+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 5 12 2024-02-19T21:05:04.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.596+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff4400e520 osd=5 4 2024-02-19T21:05:04.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.596+0000 7eff5dd63700 10 client.34084.objecter _send_command 12 2024-02-19T21:05:04.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.596+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7eff44018080 con 0x7eff44017ab0 2024-02-19T21:05:04.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.608+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7eff44018080 con 0x7eff44017ab0 2024-02-19T21:05:04.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.609+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:04.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.609+0000 7eff54ff9700 10 client.34084.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.609+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff4400e520 osd=5 4 2024-02-19T21:05:04.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.609+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 5 12 2024-02-19T21:05:04.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.620+0000 7eff5dd63700 10 client.34084.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:04.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.620+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 13 2024-02-19T21:05:04.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.620+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44018080 osd=6 3 2024-02-19T21:05:04.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.620+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44018080 osd=6 3 2024-02-19T21:05:04.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.620+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44018080 osd=6 3 2024-02-19T21:05:04.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.620+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 13 2024-02-19T21:05:04.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.620+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff44018080 osd=6 3 2024-02-19T21:05:04.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.620+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 6 13 2024-02-19T21:05:04.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.620+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44018080 osd=6 4 2024-02-19T21:05:04.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.620+0000 7eff5dd63700 10 client.34084.objecter _send_command 13 2024-02-19T21:05:04.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.620+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff44022ce0 con 0x7eff4401f130 2024-02-19T21:05:04.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.652+0000 7eff54ff9700 10 client.34084.objecter ms_handle_connect 0x7eff4401f130 2024-02-19T21:05:04.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:04.655 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:05:04.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:05:04.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.653+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7eff44022ce0 con 0x7eff4401f130 2024-02-19T21:05:04.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.653+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:05:04.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.653+0000 7eff54ff9700 10 client.34084.objecter _finish_command 13 = system:0 2024-02-19T21:05:04.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.653+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff44018080 osd=6 4 2024-02-19T21:05:04.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.653+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 6 13 2024-02-19T21:05:04.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.668+0000 7f09f77fe700 1 -- 172.21.15.163:0/2032652544 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+1170708 (unknown 2965378022 0 2998735271) 0x7f09e0000bc0 con 0x7f09ec063b30 2024-02-19T21:05:04.677 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:05:04.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.679+0000 7eff5dd63700 10 client.34084.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:04.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.679+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 14 2024-02-19T21:05:04.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.679+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44018080 osd=6 3 2024-02-19T21:05:04.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.679+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44018080 osd=6 3 2024-02-19T21:05:04.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.679+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44018080 osd=6 3 2024-02-19T21:05:04.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.679+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 14 2024-02-19T21:05:04.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.679+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff44018080 osd=6 3 2024-02-19T21:05:04.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.679+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 6 14 2024-02-19T21:05:04.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.679+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44018080 osd=6 4 2024-02-19T21:05:04.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.679+0000 7eff5dd63700 10 client.34084.objecter _send_command 14 2024-02-19T21:05:04.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.679+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7eff44022ce0 con 0x7eff4401f130 2024-02-19T21:05:04.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.683+0000 7f0a0490d700 10 librados: watch_flush enter 2024-02-19T21:05:04.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.683+0000 7f0a0490d700 10 librados: watch_flush exit 2024-02-19T21:05:04.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.683+0000 7f0a0490d700 20 client.33955.objecter shutdown clearing up homeless session... 2024-02-19T21:05:04.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.683+0000 7f0a0490d700 10 client.33955.objecter successfully canceled tick 2024-02-19T21:05:04.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.683+0000 7f0a0490d700 1 -- 172.21.15.163:0/2032652544 >> v1:172.21.15.163:6800/1552514681 conn(0x7f09ec063b30 legacy=0x7f09ec065ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.683+0000 7f0a0490d700 1 -- 172.21.15.163:0/2032652544 >> v1:172.21.15.193:6789/0 conn(0x7f0a000fc870 legacy=0x7f0a000609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.700 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:05:04.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7eff44022ce0 con 0x7eff4401f130 2024-02-19T21:05:04.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:04.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff54ff9700 10 client.34084.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff44018080 osd=6 4 2024-02-19T21:05:04.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 6 14 2024-02-19T21:05:04.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff5dd63700 10 client.34084.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:04.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 15 2024-02-19T21:05:04.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44022ce0 osd=7 3 2024-02-19T21:05:04.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44022ce0 osd=7 3 2024-02-19T21:05:04.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44022ce0 osd=7 3 2024-02-19T21:05:04.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 15 2024-02-19T21:05:04.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff44022ce0 osd=7 3 2024-02-19T21:05:04.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 7 15 2024-02-19T21:05:04.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44022ce0 osd=7 4 2024-02-19T21:05:04.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff5dd63700 10 client.34084.objecter _send_command 15 2024-02-19T21:05:04.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.692+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff44023f60 con 0x7eff440249e0 2024-02-19T21:05:04.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.696+0000 7f0a0490d700 1 -- 172.21.15.163:0/2032652544 shutdown_connections 2024-02-19T21:05:04.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.696+0000 7f0a0490d700 1 -- 172.21.15.163:0/2032652544 >> 172.21.15.163:0/2032652544 conn(0x7f0a000f2470 msgr2=0x7f0a00106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:04.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.700+0000 7eff54ff9700 10 client.34084.objecter ms_handle_connect 0x7eff440249e0 2024-02-19T21:05:04.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.700+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7eff44023f60 con 0x7eff440249e0 2024-02-19T21:05:04.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.700+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:05:04.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.700+0000 7eff54ff9700 10 client.34084.objecter _finish_command 15 = system:0 2024-02-19T21:05:04.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.700+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff44022ce0 osd=7 4 2024-02-19T21:05:04.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.700+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 7 15 2024-02-19T21:05:04.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.702+0000 7f0a0490d700 1 -- 172.21.15.163:0/2032652544 shutdown_connections 2024-02-19T21:05:04.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.717+0000 7f0a0490d700 1 -- 172.21.15.163:0/2032652544 wait complete. 2024-02-19T21:05:04.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.717+0000 7f0a0490d700 1 librados: shutdown 2024-02-19T21:05:04.730 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:05:04.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.736+0000 7eff5dd63700 10 client.34084.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:04.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.736+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign -1 16 2024-02-19T21:05:04.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.736+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44022ce0 osd=7 3 2024-02-19T21:05:04.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.736+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44022ce0 osd=7 3 2024-02-19T21:05:04.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.736+0000 7eff5dd63700 20 client.34084.objecter _get_session s=0x7eff44022ce0 osd=7 3 2024-02-19T21:05:04.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.737+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_remove -1 16 2024-02-19T21:05:04.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.737+0000 7eff5dd63700 20 client.34084.objecter get_session s=0x7eff44022ce0 osd=7 3 2024-02-19T21:05:04.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.737+0000 7eff5dd63700 15 client.34084.objecter _session_command_op_assign 7 16 2024-02-19T21:05:04.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.737+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44022ce0 osd=7 4 2024-02-19T21:05:04.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.737+0000 7eff5dd63700 10 client.34084.objecter _send_command 16 2024-02-19T21:05:04.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.737+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7eff44023f60 con 0x7eff440249e0 2024-02-19T21:05:04.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff54ff9700 1 -- 172.21.15.163:0/1173853636 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7eff44023f60 con 0x7eff440249e0 2024-02-19T21:05:04.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff54ff9700 10 client.34084.objecter ms_dispatch 0x7eff58001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:04.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff54ff9700 10 client.34084.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff54ff9700 20 client.34084.objecter put_session s=0x7eff44022ce0 osd=7 4 2024-02-19T21:05:04.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff54ff9700 15 client.34084.objecter _session_command_op_remove 7 16 2024-02-19T21:05:04.741 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:05:04.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 10 librados: watch_flush enter 2024-02-19T21:05:04.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 10 librados: watch_flush exit 2024-02-19T21:05:04.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 10 client.34084.objecter close_session for osd.0 2024-02-19T21:05:04.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 >> v1:172.21.15.163:6801/1559418115 conn(0x7eff44004560 legacy=0x7eff44002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff440051d0 osd=0 1 2024-02-19T21:05:04.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 10 client.34084.objecter close_session for osd.1 2024-02-19T21:05:04.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 >> v1:172.21.15.163:6805/3125047432 conn(0x7eff44004180 legacy=0x7eff4400a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44004010 osd=1 1 2024-02-19T21:05:04.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 10 client.34084.objecter close_session for osd.2 2024-02-19T21:05:04.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 >> v1:172.21.15.163:6809/1563158487 conn(0x7eff4400d320 legacy=0x7eff4400d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44003660 osd=2 1 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 10 client.34084.objecter close_session for osd.3 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 >> v1:172.21.15.163:6813/2112279769 conn(0x7eff44011eb0 legacy=0x7eff440142a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44003440 osd=3 1 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 10 client.34084.objecter close_session for osd.4 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 >> v1:172.21.15.193:6800/3129716965 conn(0x7eff4400ce80 legacy=0x7eff44015cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff4400e1f0 osd=4 1 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 10 client.34084.objecter close_session for osd.5 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 >> v1:172.21.15.193:6804/805161991 conn(0x7eff44017ab0 legacy=0x7eff4401d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff4400e520 osd=5 1 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 10 client.34084.objecter close_session for osd.6 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 >> v1:172.21.15.193:6808/787604615 conn(0x7eff4401f130 legacy=0x7eff44021520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44018080 osd=6 1 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 10 client.34084.objecter close_session for osd.7 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 >> v1:172.21.15.193:6812/157927432 conn(0x7eff440249e0 legacy=0x7eff44024e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 20 client.34084.objecter put_session s=0x7eff44022ce0 osd=7 1 2024-02-19T21:05:04.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 20 client.34084.objecter shutdown clearing up homeless session... 2024-02-19T21:05:04.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff5dd63700 10 client.34084.objecter successfully canceled tick 2024-02-19T21:05:04.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff57fff700 1 -- 172.21.15.163:0/1173853636 reap_dead start 2024-02-19T21:05:04.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff57fff700 1 -- 172.21.15.163:0/1173853636 reap_dead start 2024-02-19T21:05:04.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff57fff700 1 -- 172.21.15.163:0/1173853636 reap_dead start 2024-02-19T21:05:04.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff57fff700 1 -- 172.21.15.163:0/1173853636 reap_dead start 2024-02-19T21:05:04.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.746+0000 7eff57fff700 1 -- 172.21.15.163:0/1173853636 reap_dead start 2024-02-19T21:05:04.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.747+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 >> v1:172.21.15.163:6800/1552514681 conn(0x7eff40063b30 legacy=0x7eff40065ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.747+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 >> v1:172.21.15.193:6789/0 conn(0x7eff580693f0 legacy=0x7eff5810bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:04.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.747+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 shutdown_connections 2024-02-19T21:05:04.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.747+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 >> 172.21.15.163:0/1173853636 conn(0x7eff58063b20 msgr2=0x7eff58064720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:04.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.747+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 shutdown_connections 2024-02-19T21:05:04.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.747+0000 7eff5dd63700 1 -- 172.21.15.163:0/1173853636 wait complete. 2024-02-19T21:05:04.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:04.748+0000 7eff5dd63700 1 librados: shutdown 2024-02-19T21:05:04.748 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.748 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.748 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.748 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.748 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.748 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.748 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.749 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:05:04.756 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:05:04.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:01.237613+0000 osd.4 (osd.4) 75 : cluster [DBG] 2.f scrub starts 2024-02-19T21:05:04.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:01.259307+0000 osd.4 (osd.4) 76 : cluster [DBG] 2.f scrub ok 2024-02-19T21:05:04.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:01.684023+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.14 deep-scrub starts 2024-02-19T21:05:04.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:01.684951+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.14 deep-scrub ok 2024-02-19T21:05:04.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:01.852079+0000 osd.1 (osd.1) 77 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:05:04.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:01.858914+0000 osd.1 (osd.1) 78 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:05:04.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:03.092132+0000 mgr.y (mgr.14550) 163 : cluster [DBG] pgmap v155: 424 pgs: 32 unknown, 392 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 949 B/s rd, 1 op/s 2024-02-19T21:05:04.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:03.096168+0000 osd.1 (osd.1) 79 : cluster [DBG] 4.f scrub starts 2024-02-19T21:05:04.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:03.209867+0000 osd.1 (osd.1) 80 : cluster [DBG] 4.f scrub ok 2024-02-19T21:05:04.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:03.245009+0000 osd.7 (osd.7) 77 : cluster [DBG] 6.3 deep-scrub starts 2024-02-19T21:05:04.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:03.248502+0000 osd.7 (osd.7) 78 : cluster [DBG] 6.3 deep-scrub ok 2024-02-19T21:05:04.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:03.251858+0000 mon.c (mon.2) 94 : audit [DBG] from='client.? v1:172.21.15.163:0/3575209332' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-19T21:05:04.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:03.253059+0000 mgr.y (mgr.14550) 164 : audit [DBG] from='mon.? v1:172.21.15.163:0/3010617610' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-19T21:05:04.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:03.361859+0000 mon.a (mon.0) 1018 : audit [INF] from='client.31900 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:04.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:03.361972+0000 mon.a (mon.0) 1019 : audit [INF] from='client.32629 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:05:04.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:03.362173+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-02-19T21:05:04.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:03.373574+0000 mon.b (mon.1) 181 : audit [INF] from='client.? v1:172.21.15.163:0/1301771977' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi163-83854-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:04.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:03.373857+0000 mon.b (mon.1) 182 : audit [INF] from='client.? v1:172.21.15.163:0/1979745060' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi163-83845-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:04.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:03.375005+0000 mon.a (mon.0) 1021 : audit [INF] from='client.32662 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi163-83854-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:04.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:03.375293+0000 mon.a (mon.0) 1022 : audit [INF] from='client.32650 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi163-83845-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:04.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:03.391299+0000 mon.b (mon.1) 183 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi163-83944-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:04.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:03.435963+0000 mon.c (mon.2) 95 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:04.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:03.731929+0000 mon.a (mon.0) 1023 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi163-83944-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:04.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:03.814331+0000 mon.a (mon.0) 1024 : audit [INF] from='client.32662 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi163-83854-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:04.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:03.814426+0000 mon.a (mon.0) 1025 : audit [INF] from='client.32650 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi163-83845-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:04.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:03.815876+0000 mon.a (mon.0) 1026 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-02-19T21:05:04.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:04.181076+0000 mon.b (mon.1) 184 : audit [DBG] from='client.? v1:172.21.15.163:0/1173853636' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:04.823 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:05.430 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:05:05.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:05.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:05:05.547 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:05:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:05.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:03.901687+0000 osd.1 (osd.1) 81 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:05:05.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:03.922045+0000 osd.1 (osd.1) 82 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:05:05.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:04.136003+0000 osd.5 (osd.5) 70 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:05:05.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:04.173894+0000 osd.5 (osd.5) 71 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:05:05.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:04.502033+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:05.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:04.549930+0000 mgr.y (mgr.14550) 165 : audit [DBG] from='client.33955 v1:172.21.15.163:0/2032652544' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:05:05.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:04.983772+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4"}]: dispatch 2024-02-19T21:05:05.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:05.183886+0000 osd.7 (osd.7) 79 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:05:05.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:05.187469+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:05:05.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:05.358870+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-02-19T21:05:05.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:05.359844+0000 mon.b (mon.1) 185 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:05.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:05.365113+0000 mon.a (mon.0) 1028 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4"}]: dispatch 2024-02-19T21:05:05.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:05.368627+0000 mon.a (mon.0) 1029 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:05.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:03.901687+0000 osd.1 (osd.1) 81 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:05:05.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:03.922045+0000 osd.1 (osd.1) 82 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:05:05.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:04.136003+0000 osd.5 (osd.5) 70 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:05:05.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:04.173894+0000 osd.5 (osd.5) 71 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:05:05.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:04.502033+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:05.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:04.549930+0000 mgr.y (mgr.14550) 165 : audit [DBG] from='client.33955 v1:172.21.15.163:0/2032652544' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:05:05.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:04.983772+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4"}]: dispatch 2024-02-19T21:05:05.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:05.183886+0000 osd.7 (osd.7) 79 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:05:05.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:05.187469+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:05:05.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:05.358870+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-02-19T21:05:05.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:05.359844+0000 mon.b (mon.1) 185 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:05.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:05.365113+0000 mon.a (mon.0) 1028 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4"}]: dispatch 2024-02-19T21:05:05.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:05.368627+0000 mon.a (mon.0) 1029 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:05.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:03.901687+0000 osd.1 (osd.1) 81 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:05:05.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:03.922045+0000 osd.1 (osd.1) 82 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:05:05.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:04.136003+0000 osd.5 (osd.5) 70 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:05:05.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:04.173894+0000 osd.5 (osd.5) 71 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:05:05.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:04.502033+0000 mon.c (mon.2) 96 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:05.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:04.549930+0000 mgr.y (mgr.14550) 165 : audit [DBG] from='client.33955 v1:172.21.15.163:0/2032652544' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:05:05.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:04.983772+0000 mon.c (mon.2) 97 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4"}]: dispatch 2024-02-19T21:05:05.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:05.183886+0000 osd.7 (osd.7) 79 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:05:05.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:05.187469+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:05:05.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:05.358870+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-02-19T21:05:05.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:05.359844+0000 mon.b (mon.1) 185 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:05.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:05.365113+0000 mon.a (mon.0) 1028 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4"}]: dispatch 2024-02-19T21:05:05.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:05.368627+0000 mon.a (mon.0) 1029 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:07.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:05.092727+0000 mgr.y (mgr.14550) 166 : cluster [DBG] pgmap v159: 456 pgs: 10 creating+peering, 32 unknown, 414 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:05:07.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:05.531878+0000 mon.c (mon.2) 98 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:07.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:05.998736+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.b scrub starts 2024-02-19T21:05:07.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:06.000893+0000 osd.3 (osd.3) 92 : cluster [DBG] 6.b scrub ok 2024-02-19T21:05:07.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:06.010654+0000 mon.a (mon.0) 1030 : audit [INF] from='client.32629 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi163-83944-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]': finished 2024-02-19T21:05:07.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:06.010731+0000 mon.a (mon.0) 1031 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4"}]': finished 2024-02-19T21:05:07.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:06.010862+0000 mon.a (mon.0) 1032 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:07.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:06.010944+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-02-19T21:05:07.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:06.015823+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]: dispatch 2024-02-19T21:05:07.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:06.051030+0000 mon.b (mon.1) 186 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:07.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:06.062303+0000 mon.a (mon.0) 1034 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]: dispatch 2024-02-19T21:05:07.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:06.063650+0000 mon.a (mon.0) 1035 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:07.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:06.075547+0000 mon.a (mon.0) 1036 : audit [INF] from='client.? v1:172.21.15.163:0/1116752732' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi163-83854-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:07.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:06.129753+0000 osd.5 (osd.5) 72 : cluster [DBG] 2.2 scrub starts 2024-02-19T21:05:07.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:06.145407+0000 osd.5 (osd.5) 73 : cluster [DBG] 2.2 scrub ok 2024-02-19T21:05:07.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:06.532819+0000 mon.c (mon.2) 100 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:07.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:05.092727+0000 mgr.y (mgr.14550) 166 : cluster [DBG] pgmap v159: 456 pgs: 10 creating+peering, 32 unknown, 414 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:05:07.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:05.531878+0000 mon.c (mon.2) 98 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:07.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:05.998736+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.b scrub starts 2024-02-19T21:05:07.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:06.000893+0000 osd.3 (osd.3) 92 : cluster [DBG] 6.b scrub ok 2024-02-19T21:05:07.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:06.010654+0000 mon.a (mon.0) 1030 : audit [INF] from='client.32629 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi163-83944-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]': finished 2024-02-19T21:05:07.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:06.010731+0000 mon.a (mon.0) 1031 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4"}]': finished 2024-02-19T21:05:07.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:06.010862+0000 mon.a (mon.0) 1032 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:07.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:06.010944+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-02-19T21:05:07.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:06.015823+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]: dispatch 2024-02-19T21:05:07.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:06.051030+0000 mon.b (mon.1) 186 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:07.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:06.062303+0000 mon.a (mon.0) 1034 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]: dispatch 2024-02-19T21:05:07.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:06.063650+0000 mon.a (mon.0) 1035 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:07.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:06.075547+0000 mon.a (mon.0) 1036 : audit [INF] from='client.? v1:172.21.15.163:0/1116752732' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi163-83854-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:07.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:06.129753+0000 osd.5 (osd.5) 72 : cluster [DBG] 2.2 scrub starts 2024-02-19T21:05:07.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:06.145407+0000 osd.5 (osd.5) 73 : cluster [DBG] 2.2 scrub ok 2024-02-19T21:05:07.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:06.532819+0000 mon.c (mon.2) 100 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:07.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:05.092727+0000 mgr.y (mgr.14550) 166 : cluster [DBG] pgmap v159: 456 pgs: 10 creating+peering, 32 unknown, 414 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:05:07.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:05.531878+0000 mon.c (mon.2) 98 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:07.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:05.998736+0000 osd.3 (osd.3) 91 : cluster [DBG] 6.b scrub starts 2024-02-19T21:05:07.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:06.000893+0000 osd.3 (osd.3) 92 : cluster [DBG] 6.b scrub ok 2024-02-19T21:05:07.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:06.010654+0000 mon.a (mon.0) 1030 : audit [INF] from='client.32629 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi163-83944-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]': finished 2024-02-19T21:05:07.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:06.010731+0000 mon.a (mon.0) 1031 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4"}]': finished 2024-02-19T21:05:07.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:06.010862+0000 mon.a (mon.0) 1032 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:07.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:06.010944+0000 mon.a (mon.0) 1033 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-02-19T21:05:07.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:06.015823+0000 mon.c (mon.2) 99 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]: dispatch 2024-02-19T21:05:07.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:06.051030+0000 mon.b (mon.1) 186 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:07.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:06.062303+0000 mon.a (mon.0) 1034 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]: dispatch 2024-02-19T21:05:07.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:06.063650+0000 mon.a (mon.0) 1035 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:07.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:06.075547+0000 mon.a (mon.0) 1036 : audit [INF] from='client.? v1:172.21.15.163:0/1116752732' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi163-83854-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:07.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:06.129753+0000 osd.5 (osd.5) 72 : cluster [DBG] 2.2 scrub starts 2024-02-19T21:05:07.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:06.145407+0000 osd.5 (osd.5) 73 : cluster [DBG] 2.2 scrub ok 2024-02-19T21:05:07.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:06.532819+0000 mon.c (mon.2) 100 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:08.064 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:05:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:05:08.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:07.027383+0000 mon.a (mon.0) 1037 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:05:08.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:07.038318+0000 mon.a (mon.0) 1038 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]': finished 2024-02-19T21:05:08.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:07.038429+0000 mon.a (mon.0) 1039 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:05:08.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:07.038514+0000 mon.a (mon.0) 1040 : audit [INF] from='client.? v1:172.21.15.163:0/1116752732' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi163-83854-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:08.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:07.038595+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-02-19T21:05:08.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:07.040639+0000 mon.b (mon.1) 187 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-9"}]: dispatch 2024-02-19T21:05:08.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:07.041787+0000 mon.b (mon.1) 188 : audit [INF] from='client.? v1:172.21.15.163:0/2336517267' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:07.059371+0000 mon.c (mon.2) 101 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:07.067187+0000 mon.b (mon.1) 189 : audit [INF] from='client.? v1:172.21.15.163:0/4170288922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi163-83845-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:07.072894+0000 mon.a (mon.0) 1042 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-9"}]: dispatch 2024-02-19T21:05:08.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:07.073343+0000 mon.a (mon.0) 1043 : audit [INF] from='client.35593 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:07.074898+0000 mon.a (mon.0) 1044 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:07.079979+0000 mon.a (mon.0) 1045 : audit [INF] from='client.35590 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi163-83845-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:07.093844+0000 mgr.y (mgr.14550) 167 : cluster [DBG] pgmap v162: 496 pgs: 136 unknown, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-02-19T21:05:08.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:07.115774+0000 osd.5 (osd.5) 74 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:05:08.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:07.118662+0000 osd.5 (osd.5) 75 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:05:08.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:07.541090+0000 mon.c (mon.2) 102 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:08.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:07.656662+0000 osd.0 (osd.0) 75 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:05:08.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:07.660426+0000 osd.0 (osd.0) 76 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:05:08.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:07.027383+0000 mon.a (mon.0) 1037 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:05:08.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:07.038318+0000 mon.a (mon.0) 1038 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]': finished 2024-02-19T21:05:08.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:07.038429+0000 mon.a (mon.0) 1039 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:05:08.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:07.038514+0000 mon.a (mon.0) 1040 : audit [INF] from='client.? v1:172.21.15.163:0/1116752732' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi163-83854-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:08.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:07.038595+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-02-19T21:05:08.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:07.040639+0000 mon.b (mon.1) 187 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-9"}]: dispatch 2024-02-19T21:05:08.150 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:07.041787+0000 mon.b (mon.1) 188 : audit [INF] from='client.? v1:172.21.15.163:0/2336517267' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:07.059371+0000 mon.c (mon.2) 101 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:07.067187+0000 mon.b (mon.1) 189 : audit [INF] from='client.? v1:172.21.15.163:0/4170288922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi163-83845-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:07.072894+0000 mon.a (mon.0) 1042 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-9"}]: dispatch 2024-02-19T21:05:08.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:07.073343+0000 mon.a (mon.0) 1043 : audit [INF] from='client.35593 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:07.074898+0000 mon.a (mon.0) 1044 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:07.079979+0000 mon.a (mon.0) 1045 : audit [INF] from='client.35590 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi163-83845-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:07.093844+0000 mgr.y (mgr.14550) 167 : cluster [DBG] pgmap v162: 496 pgs: 136 unknown, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-02-19T21:05:08.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:07.115774+0000 osd.5 (osd.5) 74 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:05:08.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:07.118662+0000 osd.5 (osd.5) 75 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:05:08.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:07.541090+0000 mon.c (mon.2) 102 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:08.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:07.656662+0000 osd.0 (osd.0) 75 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:05:08.151 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:07.660426+0000 osd.0 (osd.0) 76 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:05:08.151 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:07.027383+0000 mon.a (mon.0) 1037 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:05:08.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:07.038318+0000 mon.a (mon.0) 1038 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6"}]': finished 2024-02-19T21:05:08.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:07.038429+0000 mon.a (mon.0) 1039 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:05:08.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:07.038514+0000 mon.a (mon.0) 1040 : audit [INF] from='client.? v1:172.21.15.163:0/1116752732' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi163-83854-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:08.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:07.038595+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-02-19T21:05:08.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:07.040639+0000 mon.b (mon.1) 187 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-9"}]: dispatch 2024-02-19T21:05:08.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:07.041787+0000 mon.b (mon.1) 188 : audit [INF] from='client.? v1:172.21.15.163:0/2336517267' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:07.059371+0000 mon.c (mon.2) 101 : audit [INF] from='client.? v1:172.21.15.163:0/1081998331' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:07.067187+0000 mon.b (mon.1) 189 : audit [INF] from='client.? v1:172.21.15.163:0/4170288922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi163-83845-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:07.072894+0000 mon.a (mon.0) 1042 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-9"}]: dispatch 2024-02-19T21:05:08.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:07.073343+0000 mon.a (mon.0) 1043 : audit [INF] from='client.35593 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:07.074898+0000 mon.a (mon.0) 1044 : audit [INF] from='client.35237 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:07.079979+0000 mon.a (mon.0) 1045 : audit [INF] from='client.35590 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi163-83845-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:08.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:07.093844+0000 mgr.y (mgr.14550) 167 : cluster [DBG] pgmap v162: 496 pgs: 136 unknown, 360 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-02-19T21:05:08.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:07.115774+0000 osd.5 (osd.5) 74 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:05:08.152 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:07.118662+0000 osd.5 (osd.5) 75 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:05:08.153 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:07.541090+0000 mon.c (mon.2) 102 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:08.153 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:07.656662+0000 osd.0 (osd.0) 75 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:05:08.153 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:07.660426+0000 osd.0 (osd.0) 76 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:05:08.366 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "ceph_version": "ceph version 16.2.14-548-g81bd20d6 (81bd20d634209c7cb82c18be12b4b5a05643ebf1) pacific (stable)", 2024-02-19T21:05:08.366 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "ceph_version_short": "16.2.14-548-g81bd20d6", 2024-02-19T21:05:08.366 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "container_hostname": "smithi163", 2024-02-19T21:05:08.366 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:7fa249cc0191869e2bcc743303b0da933d04d152b2397edb19ec992e48553ab9", 2024-02-19T21:05:08.366 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-02-19T21:05:08.366 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "distro": "centos", 2024-02-19T21:05:08.366 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "distro_description": "CentOS Stream 8", 2024-02-19T21:05:08.366 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "distro_version": "8", 2024-02-19T21:05:08.366 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "frontend_config#0": "beast port=80", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "frontend_type#0": "beast", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "hostname": "smithi163", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "id": "foo.a", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "kernel_description": "#1 SMP Mon Feb 5 18:34:56 UTC 2024", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "kernel_version": "4.18.0-539.el8.x86_64", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "mem_swap_kb": "0", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "mem_total_kb": "32537628", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "num_handles": "1", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "os": "Linux", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "pid": "7", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "realm_id": "", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "realm_name": "", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "zone_id": "66abb82d-cdc7-4da9-af39-78ae2ae2585a", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "zone_name": "default", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "zonegroup_id": "e57e57e8-029e-47bf-93a3-0326a8530e42", 2024-02-19T21:05:08.367 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "zonegroup_name": "default" 2024-02-19T21:05:08.368 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: }, 2024-02-19T21:05:08.368 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: "task_status": {} 2024-02-19T21:05:08.368 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: } 2024-02-19T21:05:08.368 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: } 2024-02-19T21:05:08.368 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: } 2024-02-19T21:05:08.368 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: } 2024-02-19T21:05:08.368 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: } 2024-02-19T21:05:08.368 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: 2024-02-19T21:05:08.368 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: attempt 1 of 20 2024-02-19T21:05:08.368 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10337 ms) 2024-02-19T21:05:08.368 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (35482 ms total) 2024-02-19T21:05:08.368 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: 2024-02-19T21:05:08.368 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: [----------] Global test environment tear-down 2024-02-19T21:05:08.368 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (35482 ms total) 2024-02-19T21:05:08.369 INFO:tasks.workunit.client.0.smithi163.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-02-19T21:05:09.009 INFO:tasks.workunit.client.0.smithi163.stdout:_watch_notify_pp: flushed 2024-02-19T21:05:09.010 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3005 ms) 2024-02-19T21:05:09.010 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-02-19T21:05:09.010 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: trying... 2024-02-19T21:05:09.010 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: handle_notify cookie 94487086349072 notify_id 498216206337 notifier_gid 35237 2024-02-19T21:05:09.010 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: timed out 2024-02-19T21:05:09.010 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: flushing 2024-02-19T21:05:09.010 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: flushed 2024-02-19T21:05:09.010 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3046 ms) 2024-02-19T21:05:09.010 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-02-19T21:05:09.010 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: List watches 2024-02-19T21:05:09.010 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: notify2 2024-02-19T21:05:09.010 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: handle_notify cookie 94487086349072 notify_id 511101108226 notifier_gid 35237 2024-02-19T21:05:09.010 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: notify2 done 2024-02-19T21:05:09.010 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: watch_check 2024-02-19T21:05:09.010 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: unwatch2 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: flushing 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: done 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3136 ms) 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: List watches 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: notify2 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: handle_notify cookie 94487086349072 notify_id 519691042821 notifier_gid 35237 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: notify2 done 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: watch_check 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: unwatch2 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: flushing 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: done 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3033 ms) 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16463 ms total) 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: 2024-02-19T21:05:09.011 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-02-19T21:05:09.012 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (36189 ms total) 2024-02-19T21:05:09.012 INFO:tasks.workunit.client.0.smithi163.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-02-19T21:05:09.492 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:07.768963+0000 osd.6 (osd.6) 63 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:05:09.492 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:07.770723+0000 osd.6 (osd.6) 64 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:05:09.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:08.070289+0000 mon.a (mon.0) 1046 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-9"}]': finished 2024-02-19T21:05:09.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:08.070363+0000 mon.a (mon.0) 1047 : audit [INF] from='client.35593 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:09.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:08.070481+0000 mon.a (mon.0) 1048 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "yes_i_really_really_mean_it": true}]': finished 2024-02-19T21:05:09.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:08.070543+0000 mon.a (mon.0) 1049 : audit [INF] from='client.35590 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi163-83845-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:09.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:08.070694+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-02-19T21:05:09.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:08.135969+0000 mon.b (mon.1) 190 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-9", "mode": "writeback"}]: dispatch 2024-02-19T21:05:09.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:08.357208+0000 mon.b (mon.1) 191 : audit [DBG] from='client.? v1:172.21.15.163:0/1927055629' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-19T21:05:09.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:08.358211+0000 mgr.y (mgr.14550) 168 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-19T21:05:09.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:08.542769+0000 mon.c (mon.2) 103 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:09.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:08.598009+0000 osd.0 (osd.0) 77 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:05:09.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:08.599001+0000 osd.0 (osd.0) 78 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:05:09.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:08.599172+0000 mon.a (mon.0) 1051 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-9", "mode": "writeback"}]: dispatch 2024-02-19T21:05:09.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:08.988125+0000 mon.a (mon.0) 1052 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:05:09.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:08.997647+0000 mon.a (mon.0) 1053 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-9", "mode": "writeback"}]': finished 2024-02-19T21:05:09.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:08.998023+0000 mon.a (mon.0) 1054 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-02-19T21:05:09.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:09.010117+0000 mon.b (mon.1) 192 : audit [INF] from='client.? v1:172.21.15.163:0/2227392415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi163-83854-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:09.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:09.028266+0000 mon.a (mon.0) 1055 : audit [INF] from='client.37339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi163-83854-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:09.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:07.768963+0000 osd.6 (osd.6) 63 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:05:09.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:07.770723+0000 osd.6 (osd.6) 64 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:05:09.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:08.070289+0000 mon.a (mon.0) 1046 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-9"}]': finished 2024-02-19T21:05:09.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:08.070363+0000 mon.a (mon.0) 1047 : audit [INF] from='client.35593 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:09.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:08.070481+0000 mon.a (mon.0) 1048 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "yes_i_really_really_mean_it": true}]': finished 2024-02-19T21:05:09.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:08.070543+0000 mon.a (mon.0) 1049 : audit [INF] from='client.35590 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi163-83845-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:09.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:08.070694+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-02-19T21:05:09.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:08.135969+0000 mon.b (mon.1) 190 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-9", "mode": "writeback"}]: dispatch 2024-02-19T21:05:09.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:08.357208+0000 mon.b (mon.1) 191 : audit [DBG] from='client.? v1:172.21.15.163:0/1927055629' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-19T21:05:09.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:08.358211+0000 mgr.y (mgr.14550) 168 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-19T21:05:09.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:08.542769+0000 mon.c (mon.2) 103 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:09.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:08.598009+0000 osd.0 (osd.0) 77 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:05:09.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:08.599001+0000 osd.0 (osd.0) 78 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:05:09.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:08.599172+0000 mon.a (mon.0) 1051 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-9", "mode": "writeback"}]: dispatch 2024-02-19T21:05:09.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:08.988125+0000 mon.a (mon.0) 1052 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:05:09.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:08.997647+0000 mon.a (mon.0) 1053 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-9", "mode": "writeback"}]': finished 2024-02-19T21:05:09.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:08.998023+0000 mon.a (mon.0) 1054 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-02-19T21:05:09.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:09.010117+0000 mon.b (mon.1) 192 : audit [INF] from='client.? v1:172.21.15.163:0/2227392415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi163-83854-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:09.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:09.028266+0000 mon.a (mon.0) 1055 : audit [INF] from='client.37339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi163-83854-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:09.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:07.768963+0000 osd.6 (osd.6) 63 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:05:09.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:07.770723+0000 osd.6 (osd.6) 64 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:05:09.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:08.070289+0000 mon.a (mon.0) 1046 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-9"}]': finished 2024-02-19T21:05:09.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:08.070363+0000 mon.a (mon.0) 1047 : audit [INF] from='client.35593 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:09.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:08.070481+0000 mon.a (mon.0) 1048 : audit [INF] from='client.35237 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi163-83984-6", "yes_i_really_really_mean_it": true}]': finished 2024-02-19T21:05:09.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:08.070543+0000 mon.a (mon.0) 1049 : audit [INF] from='client.35590 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi163-83845-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:09.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:08.070694+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-02-19T21:05:09.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:08.135969+0000 mon.b (mon.1) 190 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-9", "mode": "writeback"}]: dispatch 2024-02-19T21:05:09.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:08.357208+0000 mon.b (mon.1) 191 : audit [DBG] from='client.? v1:172.21.15.163:0/1927055629' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-19T21:05:09.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:08.358211+0000 mgr.y (mgr.14550) 168 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-02-19T21:05:09.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:08.542769+0000 mon.c (mon.2) 103 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:09.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:08.598009+0000 osd.0 (osd.0) 77 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:05:09.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:08.599001+0000 osd.0 (osd.0) 78 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:05:09.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:08.599172+0000 mon.a (mon.0) 1051 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-9", "mode": "writeback"}]: dispatch 2024-02-19T21:05:09.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:08.988125+0000 mon.a (mon.0) 1052 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:05:09.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:08.997647+0000 mon.a (mon.0) 1053 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-9", "mode": "writeback"}]': finished 2024-02-19T21:05:09.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:08.998023+0000 mon.a (mon.0) 1054 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-02-19T21:05:09.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:09.010117+0000 mon.b (mon.1) 192 : audit [INF] from='client.? v1:172.21.15.163:0/2227392415' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi163-83854-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:09.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:09.028266+0000 mon.a (mon.0) 1055 : audit [INF] from='client.37339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi163-83854-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:10.565 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:05:10.923 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:05:10.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:10.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:05:10.940 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:08.132183+0000 osd.3 (osd.3) 93 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:05:10.940 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:08.226136+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:05:10.940 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:08.792752+0000 osd.1 (osd.1) 83 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:05:10.940 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:08.825478+0000 osd.1 (osd.1) 84 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:05:10.940 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:09.094420+0000 mgr.y (mgr.14550) 169 : cluster [DBG] pgmap v165: 492 pgs: 168 unknown, 324 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:10.940 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:09.250496+0000 mon.a (mon.0) 1056 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:05:10.940 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:09.295598+0000 mon.b (mon.1) 193 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:10.940 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:09.300029+0000 mon.a (mon.0) 1057 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:10.940 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:09.544820+0000 mon.c (mon.2) 104 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:10.940 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:09.643840+0000 osd.0 (osd.0) 79 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:05:10.940 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:09.646440+0000 osd.0 (osd.0) 80 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:05:10.941 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:09.754482+0000 osd.1 (osd.1) 85 : cluster [DBG] 2.15 deep-scrub starts 2024-02-19T21:05:10.941 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:09.756090+0000 osd.1 (osd.1) 86 : cluster [DBG] 2.15 deep-scrub ok 2024-02-19T21:05:10.941 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:10.009490+0000 mon.a (mon.0) 1058 : audit [INF] from='client.37339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi163-83854-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:10.941 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:10.009619+0000 mon.a (mon.0) 1059 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:05:10.941 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:10.009663+0000 mon.a (mon.0) 1060 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-02-19T21:05:10.941 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:10.022520+0000 mon.b (mon.1) 194 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9"}]: dispatch 2024-02-19T21:05:10.941 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:10.024438+0000 mon.a (mon.0) 1061 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9"}]: dispatch 2024-02-19T21:05:10.941 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:10.560890+0000 mon.c (mon.2) 105 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:10.941 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:08.132183+0000 osd.3 (osd.3) 93 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:05:10.941 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:08.226136+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:05:10.941 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:08.792752+0000 osd.1 (osd.1) 83 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:05:10.941 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:08.825478+0000 osd.1 (osd.1) 84 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:05:10.941 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:09.094420+0000 mgr.y (mgr.14550) 169 : cluster [DBG] pgmap v165: 492 pgs: 168 unknown, 324 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:10.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:09.250496+0000 mon.a (mon.0) 1056 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:05:10.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:09.295598+0000 mon.b (mon.1) 193 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:10.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:09.300029+0000 mon.a (mon.0) 1057 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:10.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:09.544820+0000 mon.c (mon.2) 104 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:10.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:09.643840+0000 osd.0 (osd.0) 79 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:05:10.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:09.646440+0000 osd.0 (osd.0) 80 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:05:10.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:09.754482+0000 osd.1 (osd.1) 85 : cluster [DBG] 2.15 deep-scrub starts 2024-02-19T21:05:10.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:09.756090+0000 osd.1 (osd.1) 86 : cluster [DBG] 2.15 deep-scrub ok 2024-02-19T21:05:10.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:10.009490+0000 mon.a (mon.0) 1058 : audit [INF] from='client.37339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi163-83854-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:10.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:10.009619+0000 mon.a (mon.0) 1059 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:05:10.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:10.009663+0000 mon.a (mon.0) 1060 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-02-19T21:05:10.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:10.022520+0000 mon.b (mon.1) 194 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9"}]: dispatch 2024-02-19T21:05:10.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:10.024438+0000 mon.a (mon.0) 1061 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9"}]: dispatch 2024-02-19T21:05:10.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:10.560890+0000 mon.c (mon.2) 105 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:11.028 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:11.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:08.132183+0000 osd.3 (osd.3) 93 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:05:11.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:08.226136+0000 osd.3 (osd.3) 94 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:05:11.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:08.792752+0000 osd.1 (osd.1) 83 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:05:11.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:08.825478+0000 osd.1 (osd.1) 84 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:05:11.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:09.094420+0000 mgr.y (mgr.14550) 169 : cluster [DBG] pgmap v165: 492 pgs: 168 unknown, 324 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:11.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:09.250496+0000 mon.a (mon.0) 1056 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:05:11.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:09.295598+0000 mon.b (mon.1) 193 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:11.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:09.300029+0000 mon.a (mon.0) 1057 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:11.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:09.544820+0000 mon.c (mon.2) 104 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:11.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:09.643840+0000 osd.0 (osd.0) 79 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:05:11.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:09.646440+0000 osd.0 (osd.0) 80 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:05:11.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:09.754482+0000 osd.1 (osd.1) 85 : cluster [DBG] 2.15 deep-scrub starts 2024-02-19T21:05:11.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:09.756090+0000 osd.1 (osd.1) 86 : cluster [DBG] 2.15 deep-scrub ok 2024-02-19T21:05:11.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:10.009490+0000 mon.a (mon.0) 1058 : audit [INF] from='client.37339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi163-83854-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:11.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:10.009619+0000 mon.a (mon.0) 1059 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:05:11.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:10.009663+0000 mon.a (mon.0) 1060 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-02-19T21:05:11.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:10.022520+0000 mon.b (mon.1) 194 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9"}]: dispatch 2024-02-19T21:05:11.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:10.024438+0000 mon.a (mon.0) 1061 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9"}]: dispatch 2024-02-19T21:05:11.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:10.560890+0000 mon.c (mon.2) 105 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:11.061 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:11.080 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:11.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:11.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:11.183 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:05:11.210 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:11.229 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:05:11.232 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:11.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:05:11.333 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:05:11.374 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:11.618 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-02-19T21:05:03.811911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":151009274,"num_objects":256,"num_object_clones":0,"num_object_copies":768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":1965,"num_read_kb":49856,"num_write":1598,"num_write_kb":153104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1701,"ondisk_log_size":1701,"up":1080,"acting":1080,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1491,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2908160,"kb_used_data":453568,"kb_used_omap":0,"kb_used_meta":2454528,"kb_avail":747053056,"statfs":{"total":767960285184,"available":764982329344,"internally_reserved":0,"allocated":464453632,"data_stored":458792228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2513436672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28933,"num_objects":-10,"num_object_clones":-1,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":-13,"num_write_kb":-33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.715802"},"pg_stats":[{"pgid":"74.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.1f","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.886123+0000","last_change":"2024-02-19T21:04:56.829409+0000","last_active":"2024-02-19T21:04:57.886123+0000","last_peered":"2024-02-19T21:04:57.886123+0000","last_clean":"2024-02-19T21:04:57.886123+0000","last_became_active":"2024-02-19T21:04:56.829189+0000","last_became_peered":"2024-02-19T21:04:56.829189+0000","last_unstale":"2024-02-19T21:04:57.886123+0000","last_undegraded":"2024-02-19T21:04:57.886123+0000","last_fullsized":"2024-02-19T21:04:57.886123+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"74.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.1e","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.883910+0000","last_change":"2024-02-19T21:04:56.823198+0000","last_active":"2024-02-19T21:04:57.883910+0000","last_peered":"2024-02-19T21:04:57.883910+0000","last_clean":"2024-02-19T21:04:57.883910+0000","last_became_active":"2024-02-19T21:04:56.815603+0000","last_became_peered":"2024-02-19T21:04:56.815603+0000","last_unstale":"2024-02-19T21:04:57.883910+0000","last_undegraded":"2024-02-19T21:04:57.883910+0000","last_fullsized":"2024-02-19T21:04:57.883910+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"74.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.1d","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.832364+0000","last_change":"2024-02-19T21:04:57.830589+0000","last_active":"2024-02-19T21:04:57.832364+0000","last_peered":"2024-02-19T21:04:57.832364+0000","last_clean":"2024-02-19T21:04:57.832364+0000","last_became_active":"2024-02-19T21:04:57.830281+0000","last_became_peered":"2024-02-19T21:04:57.830281+0000","last_unstale":"2024-02-19T21:04:57.832364+0000","last_undegraded":"2024-02-19T21:04:57.832364+0000","last_fullsized":"2024-02-19T21:04:57.832364+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"74.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.1c","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.796579+0000","last_change":"2024-02-19T21:04:56.811917+0000","last_active":"2024-02-19T21:04:57.796579+0000","last_peered":"2024-02-19T21:04:57.796579+0000","last_clean":"2024-02-19T21:04:57.796579+0000","last_became_active":"2024-02-19T21:04:56.805020+0000","last_became_peered":"2024-02-19T21:04:56.805020+0000","last_unstale":"2024-02-19T21:04:57.796579+0000","last_undegraded":"2024-02-19T21:04:57.796579+0000","last_fullsized":"2024-02-19T21:04:57.796579+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"74.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.1b","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.799804+0000","last_change":"2024-02-19T21:04:56.811773+0000","last_active":"2024-02-19T21:04:57.799804+0000","last_peered":"2024-02-19T21:04:57.799804+0000","last_clean":"2024-02-19T21:04:57.799804+0000","last_became_active":"2024-02-19T21:04:56.811490+0000","last_became_peered":"2024-02-19T21:04:56.811490+0000","last_unstale":"2024-02-19T21:04:57.799804+0000","last_undegraded":"2024-02-19T21:04:57.799804+0000","last_fullsized":"2024-02-19T21:04:57.799804+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"74.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.1a","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.815672+0000","last_change":"2024-02-19T21:04:57.815072+0000","last_active":"2024-02-19T21:04:57.815672+0000","last_peered":"2024-02-19T21:04:57.815672+0000","last_clean":"2024-02-19T21:04:57.815672+0000","last_became_active":"2024-02-19T21:04:57.814548+0000","last_became_peered":"2024-02-19T21:04:57.814548+0000","last_unstale":"2024-02-19T21:04:57.815672+0000","last_undegraded":"2024-02-19T21:04:57.815672+0000","last_fullsized":"2024-02-19T21:04:57.815672+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"74.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.19","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.892599+0000","last_change":"2024-02-19T21:04:57.876409+0000","last_active":"2024-02-19T21:04:57.892599+0000","last_peered":"2024-02-19T21:04:57.892599+0000","last_clean":"2024-02-19T21:04:57.892599+0000","last_became_active":"2024-02-19T21:04:57.876034+0000","last_became_peered":"2024-02-19T21:04:57.876034+0000","last_unstale":"2024-02-19T21:04:57.892599+0000","last_undegraded":"2024-02-19T21:04:57.892599+0000","last_fullsized":"2024-02-19T21:04:57.892599+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"74.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.18","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.853718+0000","last_change":"2024-02-19T21:04:57.852924+0000","last_active":"2024-02-19T21:04:57.853718+0000","last_peered":"2024-02-19T21:04:57.853718+0000","last_clean":"2024-02-19T21:04:57.853718+0000","last_became_active":"2024-02-19T21:04:57.852648+0000","last_became_peered":"2024-02-19T21:04:57.852648+0000","last_unstale":"2024-02-19T21:04:57.853718+0000","last_undegraded":"2024-02-19T21:04:57.853718+0000","last_fullsized":"2024-02-19T21:04:57.853718+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"74.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.17","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.876099+0000","last_change":"2024-02-19T21:04:57.875684+0000","last_active":"2024-02-19T21:04:57.876099+0000","last_peered":"2024-02-19T21:04:57.876099+0000","last_clean":"2024-02-19T21:04:57.876099+0000","last_became_active":"2024-02-19T21:04:57.875388+0000","last_became_peered":"2024-02-19T21:04:57.875388+0000","last_unstale":"2024-02-19T21:04:57.876099+0000","last_undegraded":"2024-02-19T21:04:57.876099+0000","last_fullsized":"2024-02-19T21:04:57.876099+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"74.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.16","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.802488+0000","last_change":"2024-02-19T21:04:56.789287+0000","last_active":"2024-02-19T21:04:57.802488+0000","last_peered":"2024-02-19T21:04:57.802488+0000","last_clean":"2024-02-19T21:04:57.802488+0000","last_became_active":"2024-02-19T21:04:56.788989+0000","last_became_peered":"2024-02-19T21:04:56.788989+0000","last_unstale":"2024-02-19T21:04:57.802488+0000","last_undegraded":"2024-02-19T21:04:57.802488+0000","last_fullsized":"2024-02-19T21:04:57.802488+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"74.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.15","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.852716+0000","last_change":"2024-02-19T21:04:56.846347+0000","last_active":"2024-02-19T21:04:57.852716+0000","last_peered":"2024-02-19T21:04:57.852716+0000","last_clean":"2024-02-19T21:04:57.852716+0000","last_became_active":"2024-02-19T21:04:56.846059+0000","last_became_peered":"2024-02-19T21:04:56.846059+0000","last_unstale":"2024-02-19T21:04:57.852716+0000","last_undegraded":"2024-02-19T21:04:57.852716+0000","last_fullsized":"2024-02-19T21:04:57.852716+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"74.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.14","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.796666+0000","last_change":"2024-02-19T21:04:56.813025+0000","last_active":"2024-02-19T21:04:57.796666+0000","last_peered":"2024-02-19T21:04:57.796666+0000","last_clean":"2024-02-19T21:04:57.796666+0000","last_became_active":"2024-02-19T21:04:56.812598+0000","last_became_peered":"2024-02-19T21:04:56.812598+0000","last_unstale":"2024-02-19T21:04:57.796666+0000","last_undegraded":"2024-02-19T21:04:57.796666+0000","last_fullsized":"2024-02-19T21:04:57.796666+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"74.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.13","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.816248+0000","last_change":"2024-02-19T21:04:56.825339+0000","last_active":"2024-02-19T21:04:57.816248+0000","last_peered":"2024-02-19T21:04:57.816248+0000","last_clean":"2024-02-19T21:04:57.816248+0000","last_became_active":"2024-02-19T21:04:56.825057+0000","last_became_peered":"2024-02-19T21:04:56.825057+0000","last_unstale":"2024-02-19T21:04:57.816248+0000","last_undegraded":"2024-02-19T21:04:57.816248+0000","last_fullsized":"2024-02-19T21:04:57.816248+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"74.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.12","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.838907+0000","last_change":"2024-02-19T21:04:57.834143+0000","last_active":"2024-02-19T21:04:57.838907+0000","last_peered":"2024-02-19T21:04:57.838907+0000","last_clean":"2024-02-19T21:04:57.838907+0000","last_became_active":"2024-02-19T21:04:57.833886+0000","last_became_peered":"2024-02-19T21:04:57.833886+0000","last_unstale":"2024-02-19T21:04:57.838907+0000","last_undegraded":"2024-02-19T21:04:57.838907+0000","last_fullsized":"2024-02-19T21:04:57.838907+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"74.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.11","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.884900+0000","last_change":"2024-02-19T21:04:57.884463+0000","last_active":"2024-02-19T21:04:57.884900+0000","last_peered":"2024-02-19T21:04:57.884900+0000","last_clean":"2024-02-19T21:04:57.884900+0000","last_became_active":"2024-02-19T21:04:57.883650+0000","last_became_peered":"2024-02-19T21:04:57.883650+0000","last_unstale":"2024-02-19T21:04:57.884900+0000","last_undegraded":"2024-02-19T21:04:57.884900+0000","last_fullsized":"2024-02-19T21:04:57.884900+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"74.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.10","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.892216+0000","last_change":"2024-02-19T21:04:57.884472+0000","last_active":"2024-02-19T21:04:57.892216+0000","last_peered":"2024-02-19T21:04:57.892216+0000","last_clean":"2024-02-19T21:04:57.892216+0000","last_became_active":"2024-02-19T21:04:57.883813+0000","last_became_peered":"2024-02-19T21:04:57.883813+0000","last_unstale":"2024-02-19T21:04:57.892216+0000","last_undegraded":"2024-02-19T21:04:57.892216+0000","last_fullsized":"2024-02-19T21:04:57.892216+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"74.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.f","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.853749+0000","last_change":"2024-02-19T21:04:57.852997+0000","last_active":"2024-02-19T21:04:57.853749+0000","last_peered":"2024-02-19T21:04:57.853749+0000","last_clean":"2024-02-19T21:04:57.853749+0000","last_became_active":"2024-02-19T21:04:57.852708+0000","last_became_peered":"2024-02-19T21:04:57.852708+0000","last_unstale":"2024-02-19T21:04:57.853749+0000","last_undegraded":"2024-02-19T21:04:57.853749+0000","last_fullsized":"2024-02-19T21:04:57.853749+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"74.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.e","version":"119'1","reported_seq":12,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-19T21:04:58.799510+0000","last_change":"2024-02-19T21:04:57.875700+0000","last_active":"2024-02-19T21:04:58.799510+0000","last_peered":"2024-02-19T21:04:58.799510+0000","last_clean":"2024-02-19T21:04:58.799510+0000","last_became_active":"2024-02-19T21:04:57.875396+0000","last_became_peered":"2024-02-19T21:04:57.875396+0000","last_unstale":"2024-02-19T21:04:58.799510+0000","last_undegraded":"2024-02-19T21:04:58.799510+0000","last_fullsized":"2024-02-19T21:04:58.799510+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"74.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.d","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.814368+0000","last_change":"2024-02-19T21:04:57.811934+0000","last_active":"2024-02-19T21:04:57.814368+0000","last_peered":"2024-02-19T21:04:57.814368+0000","last_clean":"2024-02-19T21:04:57.814368+0000","last_became_active":"2024-02-19T21:04:57.811652+0000","last_became_peered":"2024-02-19T21:04:57.811652+0000","last_unstale":"2024-02-19T21:04:57.814368+0000","last_undegraded":"2024-02-19T21:04:57.814368+0000","last_fullsized":"2024-02-19T21:04:57.814368+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"74.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.c","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.831326+0000","last_change":"2024-02-19T21:04:57.827628+0000","last_active":"2024-02-19T21:04:57.831326+0000","last_peered":"2024-02-19T21:04:57.831326+0000","last_clean":"2024-02-19T21:04:57.831326+0000","last_became_active":"2024-02-19T21:04:57.827332+0000","last_became_peered":"2024-02-19T21:04:57.827332+0000","last_unstale":"2024-02-19T21:04:57.831326+0000","last_undegraded":"2024-02-19T21:04:57.831326+0000","last_fullsized":"2024-02-19T21:04:57.831326+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"74.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.b","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.799904+0000","last_change":"2024-02-19T21:04:56.822136+0000","last_active":"2024-02-19T21:04:57.799904+0000","last_peered":"2024-02-19T21:04:57.799904+0000","last_clean":"2024-02-19T21:04:57.799904+0000","last_became_active":"2024-02-19T21:04:56.821857+0000","last_became_peered":"2024-02-19T21:04:56.821857+0000","last_unstale":"2024-02-19T21:04:57.799904+0000","last_undegraded":"2024-02-19T21:04:57.799904+0000","last_fullsized":"2024-02-19T21:04:57.799904+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"74.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.a","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.805789+0000","last_change":"2024-02-19T21:04:56.826167+0000","last_active":"2024-02-19T21:04:57.805789+0000","last_peered":"2024-02-19T21:04:57.805789+0000","last_clean":"2024-02-19T21:04:57.805789+0000","last_became_active":"2024-02-19T21:04:56.825866+0000","last_became_peered":"2024-02-19T21:04:56.825866+0000","last_unstale":"2024-02-19T21:04:57.805789+0000","last_undegraded":"2024-02-19T21:04:57.805789+0000","last_fullsized":"2024-02-19T21:04:57.805789+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"74.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.9","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.802679+0000","last_change":"2024-02-19T21:04:56.836802+0000","last_active":"2024-02-19T21:04:57.802679+0000","last_peered":"2024-02-19T21:04:57.802679+0000","last_clean":"2024-02-19T21:04:57.802679+0000","last_became_active":"2024-02-19T21:04:56.836513+0000","last_became_peered":"2024-02-19T21:04:56.836513+0000","last_unstale":"2024-02-19T21:04:57.802679+0000","last_undegraded":"2024-02-19T21:04:57.802679+0000","last_fullsized":"2024-02-19T21:04:57.802679+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"74.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.8","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.817435+0000","last_change":"2024-02-19T21:04:57.815106+0000","last_active":"2024-02-19T21:04:57.817435+0000","last_peered":"2024-02-19T21:04:57.817435+0000","last_clean":"2024-02-19T21:04:57.817435+0000","last_became_active":"2024-02-19T21:04:57.814817+0000","last_became_peered":"2024-02-19T21:04:57.814817+0000","last_unstale":"2024-02-19T21:04:57.817435+0000","last_undegraded":"2024-02-19T21:04:57.817435+0000","last_fullsized":"2024-02-19T21:04:57.817435+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"74.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.7","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.851633+0000","last_change":"2024-02-19T21:04:56.834363+0000","last_active":"2024-02-19T21:04:57.851633+0000","last_peered":"2024-02-19T21:04:57.851633+0000","last_clean":"2024-02-19T21:04:57.851633+0000","last_became_active":"2024-02-19T21:04:56.834071+0000","last_became_peered":"2024-02-19T21:04:56.834071+0000","last_unstale":"2024-02-19T21:04:57.851633+0000","last_undegraded":"2024-02-19T21:04:57.851633+0000","last_fullsized":"2024-02-19T21:04:57.851633+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"74.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.6","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.841874+0000","last_change":"2024-02-19T21:04:57.840517+0000","last_active":"2024-02-19T21:04:57.841874+0000","last_peered":"2024-02-19T21:04:57.841874+0000","last_clean":"2024-02-19T21:04:57.841874+0000","last_became_active":"2024-02-19T21:04:57.839254+0000","last_became_peered":"2024-02-19T21:04:57.839254+0000","last_unstale":"2024-02-19T21:04:57.841874+0000","last_undegraded":"2024-02-19T21:04:57.841874+0000","last_fullsized":"2024-02-19T21:04:57.841874+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"74.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.5","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.820706+0000","last_change":"2024-02-19T21:04:57.820410+0000","last_active":"2024-02-19T21:04:57.820706+0000","last_peered":"2024-02-19T21:04:57.820706+0000","last_clean":"2024-02-19T21:04:57.820706+0000","last_became_active":"2024-02-19T21:04:57.820149+0000","last_became_peered":"2024-02-19T21:04:57.820149+0000","last_unstale":"2024-02-19T21:04:57.820706+0000","last_undegraded":"2024-02-19T21:04:57.820706+0000","last_fullsized":"2024-02-19T21:04:57.820706+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"74.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.4","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.832338+0000","last_change":"2024-02-19T21:04:57.829703+0000","last_active":"2024-02-19T21:04:57.832338+0000","last_peered":"2024-02-19T21:04:57.832338+0000","last_clean":"2024-02-19T21:04:57.832338+0000","last_became_active":"2024-02-19T21:04:57.829267+0000","last_became_peered":"2024-02-19T21:04:57.829267+0000","last_unstale":"2024-02-19T21:04:57.832338+0000","last_undegraded":"2024-02-19T21:04:57.832338+0000","last_fullsized":"2024-02-19T21:04:57.832338+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"74.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.3","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.871310+0000","last_change":"2024-02-19T21:04:57.856370+0000","last_active":"2024-02-19T21:04:57.871310+0000","last_peered":"2024-02-19T21:04:57.871310+0000","last_clean":"2024-02-19T21:04:57.871310+0000","last_became_active":"2024-02-19T21:04:57.856066+0000","last_became_peered":"2024-02-19T21:04:57.856066+0000","last_unstale":"2024-02-19T21:04:57.871310+0000","last_undegraded":"2024-02-19T21:04:57.871310+0000","last_fullsized":"2024-02-19T21:04:57.871310+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"74.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.2","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.830871+0000","last_change":"2024-02-19T21:04:57.829612+0000","last_active":"2024-02-19T21:04:57.830871+0000","last_peered":"2024-02-19T21:04:57.830871+0000","last_clean":"2024-02-19T21:04:57.830871+0000","last_became_active":"2024-02-19T21:04:57.822605+0000","last_became_peered":"2024-02-19T21:04:57.822605+0000","last_unstale":"2024-02-19T21:04:57.830871+0000","last_undegraded":"2024-02-19T21:04:57.830871+0000","last_fullsized":"2024-02-19T21:04:57.830871+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"74.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.1","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.811380+0000","last_change":"2024-02-19T21:04:57.809315+0000","last_active":"2024-02-19T21:04:57.811380+0000","last_peered":"2024-02-19T21:04:57.811380+0000","last_clean":"2024-02-19T21:04:57.811380+0000","last_became_active":"2024-02-19T21:04:57.808699+0000","last_became_peered":"2024-02-19T21:04:57.808699+0000","last_unstale":"2024-02-19T21:04:57.811380+0000","last_undegraded":"2024-02-19T21:04:57.811380+0000","last_fullsized":"2024-02-19T21:04:57.811380+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"74.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.728245+0000","last_change":"2024-02-19T21:05:03.728245+0000","last_active":"2024-02-19T21:05:03.728245+0000","last_peered":"2024-02-19T21:05:03.728245+0000","last_clean":"2024-02-19T21:05:03.728245+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.728245+0000","last_undegraded":"2024-02-19T21:05:03.728245+0000","last_fullsized":"2024-02-19T21:05:03.728245+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.728245+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.728245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"73.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"72.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:03.337139+0000","last_change":"2024-02-19T21:05:03.337139+0000","last_active":"2024-02-19T21:05:03.337139+0000","last_peered":"2024-02-19T21:05:03.337139+0000","last_clean":"2024-02-19T21:05:03.337139+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:03.337139+0000","last_undegraded":"2024-02-19T21:05:03.337139+0000","last_fullsized":"2024-02-19T21:05:03.337139+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.337139+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.337139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"71.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:02.140310+0000","last_change":"2024-02-19T21:05:02.140310+0000","last_active":"2024-02-19T21:05:02.140310+0000","last_peered":"2024-02-19T21:05:02.140310+0000","last_clean":"2024-02-19T21:05:02.140310+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:02.140310+0000","last_undegraded":"2024-02-19T21:05:02.140310+0000","last_fullsized":"2024-02-19T21:05:02.140310+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:02.140310+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.140310+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"64.0","version":"0'0","reported_seq":11,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.835263+0000","last_change":"2024-02-19T21:04:56.826719+0000","last_active":"2024-02-19T21:04:57.835263+0000","last_peered":"2024-02-19T21:04:57.835263+0000","last_clean":"2024-02-19T21:04:57.835263+0000","last_became_active":"2024-02-19T21:04:56.826453+0000","last_became_peered":"2024-02-19T21:04:56.826453+0000","last_unstale":"2024-02-19T21:04:57.835263+0000","last_undegraded":"2024-02-19T21:04:57.835263+0000","last_fullsized":"2024-02-19T21:04:57.835263+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.2","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.853946+0000","last_change":"2024-02-19T21:04:54.609372+0000","last_active":"2024-02-19T21:04:57.853946+0000","last_peered":"2024-02-19T21:04:57.853946+0000","last_clean":"2024-02-19T21:04:57.853946+0000","last_became_active":"2024-02-19T21:04:54.609091+0000","last_became_peered":"2024-02-19T21:04:54.609091+0000","last_unstale":"2024-02-19T21:04:57.853946+0000","last_undegraded":"2024-02-19T21:04:57.853946+0000","last_fullsized":"2024-02-19T21:04:57.853946+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"52.b","version":"109'3","reported_seq":25,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.793038+0000","last_change":"2024-02-19T21:04:48.564461+0000","last_active":"2024-02-19T21:04:57.793038+0000","last_peered":"2024-02-19T21:04:57.793038+0000","last_clean":"2024-02-19T21:04:57.793038+0000","last_became_active":"2024-02-19T21:04:48.563636+0000","last_became_peered":"2024-02-19T21:04:48.563636+0000","last_unstale":"2024-02-19T21:04:57.793038+0000","last_undegraded":"2024-02-19T21:04:57.793038+0000","last_fullsized":"2024-02-19T21:04:57.793038+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"46.11","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.811493+0000","last_change":"2024-02-19T21:04:46.989016+0000","last_active":"2024-02-19T21:04:57.811493+0000","last_peered":"2024-02-19T21:04:57.811493+0000","last_clean":"2024-02-19T21:04:57.811493+0000","last_became_active":"2024-02-19T21:04:46.988712+0000","last_became_peered":"2024-02-19T21:04:46.988712+0000","last_unstale":"2024-02-19T21:04:57.811493+0000","last_undegraded":"2024-02-19T21:04:57.811493+0000","last_fullsized":"2024-02-19T21:04:57.811493+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.3","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.800155+0000","last_change":"2024-02-19T21:04:54.642346+0000","last_active":"2024-02-19T21:04:57.800155+0000","last_peered":"2024-02-19T21:04:57.800155+0000","last_clean":"2024-02-19T21:04:57.800155+0000","last_became_active":"2024-02-19T21:04:54.641884+0000","last_became_peered":"2024-02-19T21:04:54.641884+0000","last_unstale":"2024-02-19T21:04:57.800155+0000","last_undegraded":"2024-02-19T21:04:57.800155+0000","last_fullsized":"2024-02-19T21:04:57.800155+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"52.a","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.790667+0000","last_change":"2024-02-19T21:04:48.588262+0000","last_active":"2024-02-19T21:04:57.790667+0000","last_peered":"2024-02-19T21:04:57.790667+0000","last_clean":"2024-02-19T21:04:57.790667+0000","last_became_active":"2024-02-19T21:04:48.587976+0000","last_became_peered":"2024-02-19T21:04:48.587976+0000","last_unstale":"2024-02-19T21:04:57.790667+0000","last_undegraded":"2024-02-19T21:04:57.790667+0000","last_fullsized":"2024-02-19T21:04:57.790667+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"46.10","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.830941+0000","last_change":"2024-02-19T21:04:46.992325+0000","last_active":"2024-02-19T21:04:57.830941+0000","last_peered":"2024-02-19T21:04:57.830941+0000","last_clean":"2024-02-19T21:04:57.830941+0000","last_became_active":"2024-02-19T21:04:46.991862+0000","last_became_peered":"2024-02-19T21:04:46.991862+0000","last_unstale":"2024-02-19T21:04:57.830941+0000","last_undegraded":"2024-02-19T21:04:57.830941+0000","last_fullsized":"2024-02-19T21:04:57.830941+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.0","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.830616+0000","last_change":"2024-02-19T21:04:54.656726+0000","last_active":"2024-02-19T21:04:57.830616+0000","last_peered":"2024-02-19T21:04:57.830616+0000","last_clean":"2024-02-19T21:04:57.830616+0000","last_became_active":"2024-02-19T21:04:54.656450+0000","last_became_peered":"2024-02-19T21:04:54.656450+0000","last_unstale":"2024-02-19T21:04:57.830616+0000","last_undegraded":"2024-02-19T21:04:57.830616+0000","last_fullsized":"2024-02-19T21:04:57.830616+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"52.9","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.776746+0000","last_change":"2024-02-19T21:04:48.562772+0000","last_active":"2024-02-19T21:04:57.776746+0000","last_peered":"2024-02-19T21:04:57.776746+0000","last_clean":"2024-02-19T21:04:57.776746+0000","last_became_active":"2024-02-19T21:04:48.562478+0000","last_became_peered":"2024-02-19T21:04:48.562478+0000","last_unstale":"2024-02-19T21:04:57.776746+0000","last_undegraded":"2024-02-19T21:04:57.776746+0000","last_fullsized":"2024-02-19T21:04:57.776746+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"46.13","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.817152+0000","last_change":"2024-02-19T21:04:46.978841+0000","last_active":"2024-02-19T21:04:57.817152+0000","last_peered":"2024-02-19T21:04:57.817152+0000","last_clean":"2024-02-19T21:04:57.817152+0000","last_became_active":"2024-02-19T21:04:46.978258+0000","last_became_peered":"2024-02-19T21:04:46.978258+0000","last_unstale":"2024-02-19T21:04:57.817152+0000","last_undegraded":"2024-02-19T21:04:57.817152+0000","last_fullsized":"2024-02-19T21:04:57.817152+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.1","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.806763+0000","last_change":"2024-02-19T21:04:54.655147+0000","last_active":"2024-02-19T21:04:57.806763+0000","last_peered":"2024-02-19T21:04:57.806763+0000","last_clean":"2024-02-19T21:04:57.806763+0000","last_became_active":"2024-02-19T21:04:54.654642+0000","last_became_peered":"2024-02-19T21:04:54.654642+0000","last_unstale":"2024-02-19T21:04:57.806763+0000","last_undegraded":"2024-02-19T21:04:57.806763+0000","last_fullsized":"2024-02-19T21:04:57.806763+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"52.8","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.830818+0000","last_change":"2024-02-19T21:04:48.558192+0000","last_active":"2024-02-19T21:04:57.830818+0000","last_peered":"2024-02-19T21:04:57.830818+0000","last_clean":"2024-02-19T21:04:57.830818+0000","last_became_active":"2024-02-19T21:04:48.557907+0000","last_became_peered":"2024-02-19T21:04:48.557907+0000","last_unstale":"2024-02-19T21:04:57.830818+0000","last_undegraded":"2024-02-19T21:04:57.830818+0000","last_fullsized":"2024-02-19T21:04:57.830818+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"46.12","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.799471+0000","last_change":"2024-02-19T21:04:46.975304+0000","last_active":"2024-02-19T21:04:57.799471+0000","last_peered":"2024-02-19T21:04:57.799471+0000","last_clean":"2024-02-19T21:04:57.799471+0000","last_became_active":"2024-02-19T21:04:46.974553+0000","last_became_peered":"2024-02-19T21:04:46.974553+0000","last_unstale":"2024-02-19T21:04:57.799471+0000","last_undegraded":"2024-02-19T21:04:57.799471+0000","last_fullsized":"2024-02-19T21:04:57.799471+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.6","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.860483+0000","last_change":"2024-02-19T21:04:54.658805+0000","last_active":"2024-02-19T21:04:57.860483+0000","last_peered":"2024-02-19T21:04:57.860483+0000","last_clean":"2024-02-19T21:04:57.860483+0000","last_became_active":"2024-02-19T21:04:54.658542+0000","last_became_peered":"2024-02-19T21:04:54.658542+0000","last_unstale":"2024-02-19T21:04:57.860483+0000","last_undegraded":"2024-02-19T21:04:57.860483+0000","last_fullsized":"2024-02-19T21:04:57.860483+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"52.f","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.801282+0000","last_change":"2024-02-19T21:04:48.559954+0000","last_active":"2024-02-19T21:04:57.801282+0000","last_peered":"2024-02-19T21:04:57.801282+0000","last_clean":"2024-02-19T21:04:57.801282+0000","last_became_active":"2024-02-19T21:04:48.559674+0000","last_became_peered":"2024-02-19T21:04:48.559674+0000","last_unstale":"2024-02-19T21:04:57.801282+0000","last_undegraded":"2024-02-19T21:04:57.801282+0000","last_fullsized":"2024-02-19T21:04:57.801282+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"46.15","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.794678+0000","last_change":"2024-02-19T21:04:46.972215+0000","last_active":"2024-02-19T21:04:57.794678+0000","last_peered":"2024-02-19T21:04:57.794678+0000","last_clean":"2024-02-19T21:04:57.794678+0000","last_became_active":"2024-02-19T21:04:46.971876+0000","last_became_peered":"2024-02-19T21:04:46.971876+0000","last_unstale":"2024-02-19T21:04:57.794678+0000","last_undegraded":"2024-02-19T21:04:57.794678+0000","last_fullsized":"2024-02-19T21:04:57.794678+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.7","version":"118'1","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.891562+0000","last_change":"2024-02-19T21:04:54.598336+0000","last_active":"2024-02-19T21:04:57.891562+0000","last_peered":"2024-02-19T21:04:57.891562+0000","last_clean":"2024-02-19T21:04:57.891562+0000","last_became_active":"2024-02-19T21:04:54.598061+0000","last_became_peered":"2024-02-19T21:04:54.598061+0000","last_unstale":"2024-02-19T21:04:57.891562+0000","last_undegraded":"2024-02-19T21:04:57.891562+0000","last_fullsized":"2024-02-19T21:04:57.891562+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"52.e","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.797556+0000","last_change":"2024-02-19T21:04:48.545100+0000","last_active":"2024-02-19T21:04:57.797556+0000","last_peered":"2024-02-19T21:04:57.797556+0000","last_clean":"2024-02-19T21:04:57.797556+0000","last_became_active":"2024-02-19T21:04:48.544799+0000","last_became_peered":"2024-02-19T21:04:48.544799+0000","last_unstale":"2024-02-19T21:04:57.797556+0000","last_undegraded":"2024-02-19T21:04:57.797556+0000","last_fullsized":"2024-02-19T21:04:57.797556+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"46.14","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.798371+0000","last_change":"2024-02-19T21:04:46.975222+0000","last_active":"2024-02-19T21:04:57.798371+0000","last_peered":"2024-02-19T21:04:57.798371+0000","last_clean":"2024-02-19T21:04:57.798371+0000","last_became_active":"2024-02-19T21:04:46.974280+0000","last_became_peered":"2024-02-19T21:04:46.974280+0000","last_unstale":"2024-02-19T21:04:57.798371+0000","last_undegraded":"2024-02-19T21:04:57.798371+0000","last_fullsized":"2024-02-19T21:04:57.798371+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.4","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.806540+0000","last_change":"2024-02-19T21:04:54.639823+0000","last_active":"2024-02-19T21:04:57.806540+0000","last_peered":"2024-02-19T21:04:57.806540+0000","last_clean":"2024-02-19T21:04:57.806540+0000","last_became_active":"2024-02-19T21:04:54.639335+0000","last_became_peered":"2024-02-19T21:04:54.639335+0000","last_unstale":"2024-02-19T21:04:57.806540+0000","last_undegraded":"2024-02-19T21:04:57.806540+0000","last_fullsized":"2024-02-19T21:04:57.806540+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"52.d","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.852492+0000","last_change":"2024-02-19T21:04:48.573203+0000","last_active":"2024-02-19T21:04:57.852492+0000","last_peered":"2024-02-19T21:04:57.852492+0000","last_clean":"2024-02-19T21:04:57.852492+0000","last_became_active":"2024-02-19T21:04:48.564848+0000","last_became_peered":"2024-02-19T21:04:48.564848+0000","last_unstale":"2024-02-19T21:04:57.852492+0000","last_undegraded":"2024-02-19T21:04:57.852492+0000","last_fullsized":"2024-02-19T21:04:57.852492+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"46.17","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.794728+0000","last_change":"2024-02-19T21:04:46.980605+0000","last_active":"2024-02-19T21:04:57.794728+0000","last_peered":"2024-02-19T21:04:57.794728+0000","last_clean":"2024-02-19T21:04:57.794728+0000","last_became_active":"2024-02-19T21:04:46.980322+0000","last_became_peered":"2024-02-19T21:04:46.980322+0000","last_unstale":"2024-02-19T21:04:57.794728+0000","last_undegraded":"2024-02-19T21:04:57.794728+0000","last_fullsized":"2024-02-19T21:04:57.794728+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"61.5","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.794949+0000","last_change":"2024-02-19T21:04:54.656184+0000","last_active":"2024-02-19T21:04:57.794949+0000","last_peered":"2024-02-19T21:04:57.794949+0000","last_clean":"2024-02-19T21:04:57.794949+0000","last_became_active":"2024-02-19T21:04:54.646842+0000","last_became_peered":"2024-02-19T21:04:54.646842+0000","last_unstale":"2024-02-19T21:04:57.794949+0000","last_undegraded":"2024-02-19T21:04:57.794949+0000","last_fullsized":"2024-02-19T21:04:57.794949+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"52.c","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.870895+0000","last_change":"2024-02-19T21:04:48.564874+0000","last_active":"2024-02-19T21:04:57.870895+0000","last_peered":"2024-02-19T21:04:57.870895+0000","last_clean":"2024-02-19T21:04:57.870895+0000","last_became_active":"2024-02-19T21:04:48.564337+0000","last_became_peered":"2024-02-19T21:04:48.564337+0000","last_unstale":"2024-02-19T21:04:57.870895+0000","last_undegraded":"2024-02-19T21:04:57.870895+0000","last_fullsized":"2024-02-19T21:04:57.870895+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.16","version":"107'1","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.802813+0000","last_change":"2024-02-19T21:04:46.991791+0000","last_active":"2024-02-19T21:04:57.802813+0000","last_peered":"2024-02-19T21:04:57.802813+0000","last_clean":"2024-02-19T21:04:57.802813+0000","last_became_active":"2024-02-19T21:04:46.991498+0000","last_became_peered":"2024-02-19T21:04:46.991498+0000","last_unstale":"2024-02-19T21:04:57.802813+0000","last_undegraded":"2024-02-19T21:04:57.802813+0000","last_fullsized":"2024-02-19T21:04:57.802813+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.a","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.853854+0000","last_change":"2024-02-19T21:04:54.612097+0000","last_active":"2024-02-19T21:04:57.853854+0000","last_peered":"2024-02-19T21:04:57.853854+0000","last_clean":"2024-02-19T21:04:57.853854+0000","last_became_active":"2024-02-19T21:04:54.611831+0000","last_became_peered":"2024-02-19T21:04:54.611831+0000","last_unstale":"2024-02-19T21:04:57.853854+0000","last_undegraded":"2024-02-19T21:04:57.853854+0000","last_fullsized":"2024-02-19T21:04:57.853854+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"54.1","version":"114'16","reported_seq":45,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.786239+0000","last_change":"2024-02-19T21:04:50.621434+0000","last_active":"2024-02-19T21:04:57.786239+0000","last_peered":"2024-02-19T21:04:57.786239+0000","last_clean":"2024-02-19T21:04:57.786239+0000","last_became_active":"2024-02-19T21:04:50.621165+0000","last_became_peered":"2024-02-19T21:04:50.621165+0000","last_unstale":"2024-02-19T21:04:57.786239+0000","last_undegraded":"2024-02-19T21:04:57.786239+0000","last_fullsized":"2024-02-19T21:04:57.786239+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":512,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"52.3","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.793094+0000","last_change":"2024-02-19T21:04:48.553094+0000","last_active":"2024-02-19T21:04:57.793094+0000","last_peered":"2024-02-19T21:04:57.793094+0000","last_clean":"2024-02-19T21:04:57.793094+0000","last_became_active":"2024-02-19T21:04:48.552797+0000","last_became_peered":"2024-02-19T21:04:48.552797+0000","last_unstale":"2024-02-19T21:04:57.793094+0000","last_undegraded":"2024-02-19T21:04:57.793094+0000","last_fullsized":"2024-02-19T21:04:57.793094+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"46.19","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.828631+0000","last_change":"2024-02-19T21:04:46.987587+0000","last_active":"2024-02-19T21:04:57.828631+0000","last_peered":"2024-02-19T21:04:57.828631+0000","last_clean":"2024-02-19T21:04:57.828631+0000","last_became_active":"2024-02-19T21:04:46.983228+0000","last_became_peered":"2024-02-19T21:04:46.983228+0000","last_unstale":"2024-02-19T21:04:57.828631+0000","last_undegraded":"2024-02-19T21:04:57.828631+0000","last_fullsized":"2024-02-19T21:04:57.828631+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.b","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.800060+0000","last_change":"2024-02-19T21:04:54.642162+0000","last_active":"2024-02-19T21:04:57.800060+0000","last_peered":"2024-02-19T21:04:57.800060+0000","last_clean":"2024-02-19T21:04:57.800060+0000","last_became_active":"2024-02-19T21:04:54.640024+0000","last_became_peered":"2024-02-19T21:04:54.640024+0000","last_unstale":"2024-02-19T21:04:57.800060+0000","last_undegraded":"2024-02-19T21:04:57.800060+0000","last_fullsized":"2024-02-19T21:04:57.800060+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.0","version":"113'16","reported_seq":42,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.787764+0000","last_change":"2024-02-19T21:04:50.617397+0000","last_active":"2024-02-19T21:04:57.787764+0000","last_peered":"2024-02-19T21:04:57.787764+0000","last_clean":"2024-02-19T21:04:57.787764+0000","last_became_active":"2024-02-19T21:04:50.617152+0000","last_became_peered":"2024-02-19T21:04:50.617152+0000","last_unstale":"2024-02-19T21:04:57.787764+0000","last_undegraded":"2024-02-19T21:04:57.787764+0000","last_fullsized":"2024-02-19T21:04:57.787764+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":512,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"52.2","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.793734+0000","last_change":"2024-02-19T21:04:48.560813+0000","last_active":"2024-02-19T21:04:57.793734+0000","last_peered":"2024-02-19T21:04:57.793734+0000","last_clean":"2024-02-19T21:04:57.793734+0000","last_became_active":"2024-02-19T21:04:48.560536+0000","last_became_peered":"2024-02-19T21:04:48.560536+0000","last_unstale":"2024-02-19T21:04:57.793734+0000","last_undegraded":"2024-02-19T21:04:57.793734+0000","last_fullsized":"2024-02-19T21:04:57.793734+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"46.18","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.787852+0000","last_change":"2024-02-19T21:04:46.969724+0000","last_active":"2024-02-19T21:04:57.787852+0000","last_peered":"2024-02-19T21:04:57.787852+0000","last_clean":"2024-02-19T21:04:57.787852+0000","last_became_active":"2024-02-19T21:04:46.969210+0000","last_became_peered":"2024-02-19T21:04:46.969210+0000","last_unstale":"2024-02-19T21:04:57.787852+0000","last_undegraded":"2024-02-19T21:04:57.787852+0000","last_fullsized":"2024-02-19T21:04:57.787852+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.8","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.787645+0000","last_change":"2024-02-19T21:04:54.650290+0000","last_active":"2024-02-19T21:04:57.787645+0000","last_peered":"2024-02-19T21:04:57.787645+0000","last_clean":"2024-02-19T21:04:57.787645+0000","last_became_active":"2024-02-19T21:04:54.650062+0000","last_became_peered":"2024-02-19T21:04:54.650062+0000","last_unstale":"2024-02-19T21:04:57.787645+0000","last_undegraded":"2024-02-19T21:04:57.787645+0000","last_fullsized":"2024-02-19T21:04:57.787645+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"54.3","version":"113'8","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.800243+0000","last_change":"2024-02-19T21:04:50.612641+0000","last_active":"2024-02-19T21:04:57.800243+0000","last_peered":"2024-02-19T21:04:57.800243+0000","last_clean":"2024-02-19T21:04:57.800243+0000","last_became_active":"2024-02-19T21:04:50.612369+0000","last_became_peered":"2024-02-19T21:04:50.612369+0000","last_unstale":"2024-02-19T21:04:57.800243+0000","last_undegraded":"2024-02-19T21:04:57.800243+0000","last_fullsized":"2024-02-19T21:04:57.800243+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"52.1","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.828521+0000","last_change":"2024-02-19T21:04:48.554397+0000","last_active":"2024-02-19T21:04:57.828521+0000","last_peered":"2024-02-19T21:04:57.828521+0000","last_clean":"2024-02-19T21:04:57.828521+0000","last_became_active":"2024-02-19T21:04:48.554037+0000","last_became_peered":"2024-02-19T21:04:48.554037+0000","last_unstale":"2024-02-19T21:04:57.828521+0000","last_undegraded":"2024-02-19T21:04:57.828521+0000","last_fullsized":"2024-02-19T21:04:57.828521+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.1b","version":"107'1","reported_seq":23,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.796530+0000","last_change":"2024-02-19T21:04:46.989093+0000","last_active":"2024-02-19T21:04:57.796530+0000","last_peered":"2024-02-19T21:04:57.796530+0000","last_clean":"2024-02-19T21:04:57.796530+0000","last_became_active":"2024-02-19T21:04:46.988741+0000","last_became_peered":"2024-02-19T21:04:46.988741+0000","last_unstale":"2024-02-19T21:04:57.796530+0000","last_undegraded":"2024-02-19T21:04:57.796530+0000","last_fullsized":"2024-02-19T21:04:57.796530+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.9","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.806681+0000","last_change":"2024-02-19T21:04:54.672014+0000","last_active":"2024-02-19T21:04:57.806681+0000","last_peered":"2024-02-19T21:04:57.806681+0000","last_clean":"2024-02-19T21:04:57.806681+0000","last_became_active":"2024-02-19T21:04:54.671646+0000","last_became_peered":"2024-02-19T21:04:54.671646+0000","last_unstale":"2024-02-19T21:04:57.806681+0000","last_undegraded":"2024-02-19T21:04:57.806681+0000","last_fullsized":"2024-02-19T21:04:57.806681+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"54.2","version":"119'7","reported_seq":29,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-19T21:04:58.907751+0000","last_change":"2024-02-19T21:04:50.624946+0000","last_active":"2024-02-19T21:04:58.907751+0000","last_peered":"2024-02-19T21:04:58.907751+0000","last_clean":"2024-02-19T21:04:58.907751+0000","last_became_active":"2024-02-19T21:04:50.623676+0000","last_became_peered":"2024-02-19T21:04:50.623676+0000","last_unstale":"2024-02-19T21:04:58.907751+0000","last_undegraded":"2024-02-19T21:04:58.907751+0000","last_fullsized":"2024-02-19T21:04:58.907751+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:49.548372+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.548372+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"52.0","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.776185+0000","last_change":"2024-02-19T21:04:48.556985+0000","last_active":"2024-02-19T21:04:57.776185+0000","last_peered":"2024-02-19T21:04:57.776185+0000","last_clean":"2024-02-19T21:04:57.776185+0000","last_became_active":"2024-02-19T21:04:48.556687+0000","last_became_peered":"2024-02-19T21:04:48.556687+0000","last_unstale":"2024-02-19T21:04:57.776185+0000","last_undegraded":"2024-02-19T21:04:57.776185+0000","last_fullsized":"2024-02-19T21:04:57.776185+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"46.1a","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.806033+0000","last_change":"2024-02-19T21:04:46.977415+0000","last_active":"2024-02-19T21:04:57.806033+0000","last_peered":"2024-02-19T21:04:57.806033+0000","last_clean":"2024-02-19T21:04:57.806033+0000","last_became_active":"2024-02-19T21:04:46.976788+0000","last_became_peered":"2024-02-19T21:04:46.976788+0000","last_unstale":"2024-02-19T21:04:57.806033+0000","last_undegraded":"2024-02-19T21:04:57.806033+0000","last_fullsized":"2024-02-19T21:04:57.806033+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.e","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.802396+0000","last_change":"2024-02-19T21:04:54.603561+0000","last_active":"2024-02-19T21:04:57.802396+0000","last_peered":"2024-02-19T21:04:57.802396+0000","last_clean":"2024-02-19T21:04:57.802396+0000","last_became_active":"2024-02-19T21:04:54.603276+0000","last_became_peered":"2024-02-19T21:04:54.603276+0000","last_unstale":"2024-02-19T21:04:57.802396+0000","last_undegraded":"2024-02-19T21:04:57.802396+0000","last_fullsized":"2024-02-19T21:04:57.802396+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"52.7","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.779130+0000","last_change":"2024-02-19T21:04:48.546015+0000","last_active":"2024-02-19T21:04:57.779130+0000","last_peered":"2024-02-19T21:04:57.779130+0000","last_clean":"2024-02-19T21:04:57.779130+0000","last_became_active":"2024-02-19T21:04:48.545737+0000","last_became_peered":"2024-02-19T21:04:48.545737+0000","last_unstale":"2024-02-19T21:04:57.779130+0000","last_undegraded":"2024-02-19T21:04:57.779130+0000","last_fullsized":"2024-02-19T21:04:57.779130+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"46.1d","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.865128+0000","last_change":"2024-02-19T21:04:46.995482+0000","last_active":"2024-02-19T21:04:57.865128+0000","last_peered":"2024-02-19T21:04:57.865128+0000","last_clean":"2024-02-19T21:04:57.865128+0000","last_became_active":"2024-02-19T21:04:46.995001+0000","last_became_peered":"2024-02-19T21:04:46.995001+0000","last_unstale":"2024-02-19T21:04:57.865128+0000","last_undegraded":"2024-02-19T21:04:57.865128+0000","last_fullsized":"2024-02-19T21:04:57.865128+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.f","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.778882+0000","last_change":"2024-02-19T21:04:54.610675+0000","last_active":"2024-02-19T21:04:57.778882+0000","last_peered":"2024-02-19T21:04:57.778882+0000","last_clean":"2024-02-19T21:04:57.778882+0000","last_became_active":"2024-02-19T21:04:54.610312+0000","last_became_peered":"2024-02-19T21:04:54.610312+0000","last_unstale":"2024-02-19T21:04:57.778882+0000","last_undegraded":"2024-02-19T21:04:57.778882+0000","last_fullsized":"2024-02-19T21:04:57.778882+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"52.6","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.785328+0000","last_change":"2024-02-19T21:04:48.560539+0000","last_active":"2024-02-19T21:04:57.785328+0000","last_peered":"2024-02-19T21:04:57.785328+0000","last_clean":"2024-02-19T21:04:57.785328+0000","last_became_active":"2024-02-19T21:04:48.560244+0000","last_became_peered":"2024-02-19T21:04:48.560244+0000","last_unstale":"2024-02-19T21:04:57.785328+0000","last_undegraded":"2024-02-19T21:04:57.785328+0000","last_fullsized":"2024-02-19T21:04:57.785328+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"46.1c","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.781934+0000","last_change":"2024-02-19T21:04:46.981776+0000","last_active":"2024-02-19T21:04:57.781934+0000","last_peered":"2024-02-19T21:04:57.781934+0000","last_clean":"2024-02-19T21:04:57.781934+0000","last_became_active":"2024-02-19T21:04:46.981476+0000","last_became_peered":"2024-02-19T21:04:46.981476+0000","last_unstale":"2024-02-19T21:04:57.781934+0000","last_undegraded":"2024-02-19T21:04:57.781934+0000","last_fullsized":"2024-02-19T21:04:57.781934+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.c","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.797049+0000","last_change":"2024-02-19T21:04:54.635114+0000","last_active":"2024-02-19T21:04:57.797049+0000","last_peered":"2024-02-19T21:04:57.797049+0000","last_clean":"2024-02-19T21:04:57.797049+0000","last_became_active":"2024-02-19T21:04:54.634803+0000","last_became_peered":"2024-02-19T21:04:54.634803+0000","last_unstale":"2024-02-19T21:04:57.797049+0000","last_undegraded":"2024-02-19T21:04:57.797049+0000","last_fullsized":"2024-02-19T21:04:57.797049+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"52.5","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.865053+0000","last_change":"2024-02-19T21:04:48.558796+0000","last_active":"2024-02-19T21:04:57.865053+0000","last_peered":"2024-02-19T21:04:57.865053+0000","last_clean":"2024-02-19T21:04:57.865053+0000","last_became_active":"2024-02-19T21:04:48.556820+0000","last_became_peered":"2024-02-19T21:04:48.556820+0000","last_unstale":"2024-02-19T21:04:57.865053+0000","last_undegraded":"2024-02-19T21:04:57.865053+0000","last_fullsized":"2024-02-19T21:04:57.865053+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.1f","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.805974+0000","last_change":"2024-02-19T21:04:46.989715+0000","last_active":"2024-02-19T21:04:57.805974+0000","last_peered":"2024-02-19T21:04:57.805974+0000","last_clean":"2024-02-19T21:04:57.805974+0000","last_became_active":"2024-02-19T21:04:46.989474+0000","last_became_peered":"2024-02-19T21:04:46.989474+0000","last_unstale":"2024-02-19T21:04:57.805974+0000","last_undegraded":"2024-02-19T21:04:57.805974+0000","last_fullsized":"2024-02-19T21:04:57.805974+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.d","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.794379+0000","last_change":"2024-02-19T21:04:54.644946+0000","last_active":"2024-02-19T21:04:57.794379+0000","last_peered":"2024-02-19T21:04:57.794379+0000","last_clean":"2024-02-19T21:04:57.794379+0000","last_became_active":"2024-02-19T21:04:54.644749+0000","last_became_peered":"2024-02-19T21:04:54.644749+0000","last_unstale":"2024-02-19T21:04:57.794379+0000","last_undegraded":"2024-02-19T21:04:57.794379+0000","last_fullsized":"2024-02-19T21:04:57.794379+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"52.4","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.798297+0000","last_change":"2024-02-19T21:04:48.578081+0000","last_active":"2024-02-19T21:04:57.798297+0000","last_peered":"2024-02-19T21:04:57.798297+0000","last_clean":"2024-02-19T21:04:57.798297+0000","last_became_active":"2024-02-19T21:04:48.577869+0000","last_became_peered":"2024-02-19T21:04:48.577869+0000","last_unstale":"2024-02-19T21:04:57.798297+0000","last_undegraded":"2024-02-19T21:04:57.798297+0000","last_fullsized":"2024-02-19T21:04:57.798297+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"46.1e","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.860628+0000","last_change":"2024-02-19T21:04:46.976499+0000","last_active":"2024-02-19T21:04:57.860628+0000","last_peered":"2024-02-19T21:04:57.860628+0000","last_clean":"2024-02-19T21:04:57.860628+0000","last_became_active":"2024-02-19T21:04:46.976253+0000","last_became_peered":"2024-02-19T21:04:46.976253+0000","last_unstale":"2024-02-19T21:04:57.860628+0000","last_undegraded":"2024-02-19T21:04:57.860628+0000","last_fullsized":"2024-02-19T21:04:57.860628+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.12","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.805879+0000","last_change":"2024-02-19T21:04:54.671358+0000","last_active":"2024-02-19T21:04:57.805879+0000","last_peered":"2024-02-19T21:04:57.805879+0000","last_clean":"2024-02-19T21:04:57.805879+0000","last_became_active":"2024-02-19T21:04:54.670998+0000","last_became_peered":"2024-02-19T21:04:54.670998+0000","last_unstale":"2024-02-19T21:04:57.805879+0000","last_undegraded":"2024-02-19T21:04:57.805879+0000","last_fullsized":"2024-02-19T21:04:57.805879+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"52.1b","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.801596+0000","last_change":"2024-02-19T21:04:48.568860+0000","last_active":"2024-02-19T21:04:57.801596+0000","last_peered":"2024-02-19T21:04:57.801596+0000","last_clean":"2024-02-19T21:04:57.801596+0000","last_became_active":"2024-02-19T21:04:48.568631+0000","last_became_peered":"2024-02-19T21:04:48.568631+0000","last_unstale":"2024-02-19T21:04:57.801596+0000","last_undegraded":"2024-02-19T21:04:57.801596+0000","last_fullsized":"2024-02-19T21:04:57.801596+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"46.1","version":"107'3","reported_seq":25,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.789150+0000","last_change":"2024-02-19T21:04:46.967362+0000","last_active":"2024-02-19T21:04:57.789150+0000","last_peered":"2024-02-19T21:04:57.789150+0000","last_clean":"2024-02-19T21:04:57.789150+0000","last_became_active":"2024-02-19T21:04:46.966031+0000","last_became_peered":"2024-02-19T21:04:46.966031+0000","last_unstale":"2024-02-19T21:04:57.789150+0000","last_undegraded":"2024-02-19T21:04:57.789150+0000","last_fullsized":"2024-02-19T21:04:57.789150+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":512,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.13","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.782610+0000","last_change":"2024-02-19T21:04:54.609029+0000","last_active":"2024-02-19T21:04:57.782610+0000","last_peered":"2024-02-19T21:04:57.782610+0000","last_clean":"2024-02-19T21:04:57.782610+0000","last_became_active":"2024-02-19T21:04:54.608802+0000","last_became_peered":"2024-02-19T21:04:54.608802+0000","last_unstale":"2024-02-19T21:04:57.782610+0000","last_undegraded":"2024-02-19T21:04:57.782610+0000","last_fullsized":"2024-02-19T21:04:57.782610+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"52.1a","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.854220+0000","last_change":"2024-02-19T21:04:48.556458+0000","last_active":"2024-02-19T21:04:57.854220+0000","last_peered":"2024-02-19T21:04:57.854220+0000","last_clean":"2024-02-19T21:04:57.854220+0000","last_became_active":"2024-02-19T21:04:48.556189+0000","last_became_peered":"2024-02-19T21:04:48.556189+0000","last_unstale":"2024-02-19T21:04:57.854220+0000","last_undegraded":"2024-02-19T21:04:57.854220+0000","last_fullsized":"2024-02-19T21:04:57.854220+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.0","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.836070+0000","last_change":"2024-02-19T21:04:46.969483+0000","last_active":"2024-02-19T21:04:57.836070+0000","last_peered":"2024-02-19T21:04:57.836070+0000","last_clean":"2024-02-19T21:04:57.836070+0000","last_became_active":"2024-02-19T21:04:46.969182+0000","last_became_peered":"2024-02-19T21:04:46.969182+0000","last_unstale":"2024-02-19T21:04:57.836070+0000","last_undegraded":"2024-02-19T21:04:57.836070+0000","last_fullsized":"2024-02-19T21:04:57.836070+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.10","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.778220+0000","last_change":"2024-02-19T21:04:54.626693+0000","last_active":"2024-02-19T21:04:57.778220+0000","last_peered":"2024-02-19T21:04:57.778220+0000","last_clean":"2024-02-19T21:04:57.778220+0000","last_became_active":"2024-02-19T21:04:54.626393+0000","last_became_peered":"2024-02-19T21:04:54.626393+0000","last_unstale":"2024-02-19T21:04:57.778220+0000","last_undegraded":"2024-02-19T21:04:57.778220+0000","last_fullsized":"2024-02-19T21:04:57.778220+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"52.19","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.789034+0000","last_change":"2024-02-19T21:04:48.564817+0000","last_active":"2024-02-19T21:04:57.789034+0000","last_peered":"2024-02-19T21:04:57.789034+0000","last_clean":"2024-02-19T21:04:57.789034+0000","last_became_active":"2024-02-19T21:04:48.564520+0000","last_became_peered":"2024-02-19T21:04:48.564520+0000","last_unstale":"2024-02-19T21:04:57.789034+0000","last_undegraded":"2024-02-19T21:04:57.789034+0000","last_fullsized":"2024-02-19T21:04:57.789034+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"46.3","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.801682+0000","last_change":"2024-02-19T21:04:46.985303+0000","last_active":"2024-02-19T21:04:57.801682+0000","last_peered":"2024-02-19T21:04:57.801682+0000","last_clean":"2024-02-19T21:04:57.801682+0000","last_became_active":"2024-02-19T21:04:46.984629+0000","last_became_peered":"2024-02-19T21:04:46.984629+0000","last_unstale":"2024-02-19T21:04:57.801682+0000","last_undegraded":"2024-02-19T21:04:57.801682+0000","last_fullsized":"2024-02-19T21:04:57.801682+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.11","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.802779+0000","last_change":"2024-02-19T21:04:54.659390+0000","last_active":"2024-02-19T21:04:57.802779+0000","last_peered":"2024-02-19T21:04:57.802779+0000","last_clean":"2024-02-19T21:04:57.802779+0000","last_became_active":"2024-02-19T21:04:54.659096+0000","last_became_peered":"2024-02-19T21:04:54.659096+0000","last_unstale":"2024-02-19T21:04:57.802779+0000","last_undegraded":"2024-02-19T21:04:57.802779+0000","last_fullsized":"2024-02-19T21:04:57.802779+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"52.18","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.830749+0000","last_change":"2024-02-19T21:04:48.561722+0000","last_active":"2024-02-19T21:04:57.830749+0000","last_peered":"2024-02-19T21:04:57.830749+0000","last_clean":"2024-02-19T21:04:57.830749+0000","last_became_active":"2024-02-19T21:04:48.561290+0000","last_became_peered":"2024-02-19T21:04:48.561290+0000","last_unstale":"2024-02-19T21:04:57.830749+0000","last_undegraded":"2024-02-19T21:04:57.830749+0000","last_fullsized":"2024-02-19T21:04:57.830749+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"46.2","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.841238+0000","last_change":"2024-02-19T21:04:46.981151+0000","last_active":"2024-02-19T21:04:57.841238+0000","last_peered":"2024-02-19T21:04:57.841238+0000","last_clean":"2024-02-19T21:04:57.841238+0000","last_became_active":"2024-02-19T21:04:46.980850+0000","last_became_peered":"2024-02-19T21:04:46.980850+0000","last_unstale":"2024-02-19T21:04:57.841238+0000","last_undegraded":"2024-02-19T21:04:57.841238+0000","last_fullsized":"2024-02-19T21:04:57.841238+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.16","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.796384+0000","last_change":"2024-02-19T21:04:54.608351+0000","last_active":"2024-02-19T21:04:57.796384+0000","last_peered":"2024-02-19T21:04:57.796384+0000","last_clean":"2024-02-19T21:04:57.796384+0000","last_became_active":"2024-02-19T21:04:54.608059+0000","last_became_peered":"2024-02-19T21:04:54.608059+0000","last_unstale":"2024-02-19T21:04:57.796384+0000","last_undegraded":"2024-02-19T21:04:57.796384+0000","last_fullsized":"2024-02-19T21:04:57.796384+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"52.1f","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.856352+0000","last_change":"2024-02-19T21:04:48.588646+0000","last_active":"2024-02-19T21:04:57.856352+0000","last_peered":"2024-02-19T21:04:57.856352+0000","last_clean":"2024-02-19T21:04:57.856352+0000","last_became_active":"2024-02-19T21:04:48.588390+0000","last_became_peered":"2024-02-19T21:04:48.588390+0000","last_unstale":"2024-02-19T21:04:57.856352+0000","last_undegraded":"2024-02-19T21:04:57.856352+0000","last_fullsized":"2024-02-19T21:04:57.856352+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"46.5","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.865201+0000","last_change":"2024-02-19T21:04:46.982655+0000","last_active":"2024-02-19T21:04:57.865201+0000","last_peered":"2024-02-19T21:04:57.865201+0000","last_clean":"2024-02-19T21:04:57.865201+0000","last_became_active":"2024-02-19T21:04:46.982387+0000","last_became_peered":"2024-02-19T21:04:46.982387+0000","last_unstale":"2024-02-19T21:04:57.865201+0000","last_undegraded":"2024-02-19T21:04:57.865201+0000","last_fullsized":"2024-02-19T21:04:57.865201+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.17","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.886243+0000","last_change":"2024-02-19T21:04:54.646677+0000","last_active":"2024-02-19T21:04:57.886243+0000","last_peered":"2024-02-19T21:04:57.886243+0000","last_clean":"2024-02-19T21:04:57.886243+0000","last_became_active":"2024-02-19T21:04:54.646395+0000","last_became_peered":"2024-02-19T21:04:54.646395+0000","last_unstale":"2024-02-19T21:04:57.886243+0000","last_undegraded":"2024-02-19T21:04:57.886243+0000","last_fullsized":"2024-02-19T21:04:57.886243+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"52.1e","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.800806+0000","last_change":"2024-02-19T21:04:48.556249+0000","last_active":"2024-02-19T21:04:57.800806+0000","last_peered":"2024-02-19T21:04:57.800806+0000","last_clean":"2024-02-19T21:04:57.800806+0000","last_became_active":"2024-02-19T21:04:48.554366+0000","last_became_peered":"2024-02-19T21:04:48.554366+0000","last_unstale":"2024-02-19T21:04:57.800806+0000","last_undegraded":"2024-02-19T21:04:57.800806+0000","last_fullsized":"2024-02-19T21:04:57.800806+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"46.4","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.781537+0000","last_change":"2024-02-19T21:04:46.978759+0000","last_active":"2024-02-19T21:04:57.781537+0000","last_peered":"2024-02-19T21:04:57.781537+0000","last_clean":"2024-02-19T21:04:57.781537+0000","last_became_active":"2024-02-19T21:04:46.978441+0000","last_became_peered":"2024-02-19T21:04:46.978441+0000","last_unstale":"2024-02-19T21:04:57.781537+0000","last_undegraded":"2024-02-19T21:04:57.781537+0000","last_fullsized":"2024-02-19T21:04:57.781537+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.14","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.867233+0000","last_change":"2024-02-19T21:04:54.648789+0000","last_active":"2024-02-19T21:04:57.867233+0000","last_peered":"2024-02-19T21:04:57.867233+0000","last_clean":"2024-02-19T21:04:57.867233+0000","last_became_active":"2024-02-19T21:04:54.648513+0000","last_became_peered":"2024-02-19T21:04:54.648513+0000","last_unstale":"2024-02-19T21:04:57.867233+0000","last_undegraded":"2024-02-19T21:04:57.867233+0000","last_fullsized":"2024-02-19T21:04:57.867233+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"52.1d","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.880116+0000","last_change":"2024-02-19T21:04:48.565665+0000","last_active":"2024-02-19T21:04:57.880116+0000","last_peered":"2024-02-19T21:04:57.880116+0000","last_clean":"2024-02-19T21:04:57.880116+0000","last_became_active":"2024-02-19T21:04:48.565373+0000","last_became_peered":"2024-02-19T21:04:48.565373+0000","last_unstale":"2024-02-19T21:04:57.880116+0000","last_undegraded":"2024-02-19T21:04:57.880116+0000","last_fullsized":"2024-02-19T21:04:57.880116+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"46.7","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.856431+0000","last_change":"2024-02-19T21:04:46.982144+0000","last_active":"2024-02-19T21:04:57.856431+0000","last_peered":"2024-02-19T21:04:57.856431+0000","last_clean":"2024-02-19T21:04:57.856431+0000","last_became_active":"2024-02-19T21:04:46.979671+0000","last_became_peered":"2024-02-19T21:04:46.979671+0000","last_unstale":"2024-02-19T21:04:57.856431+0000","last_undegraded":"2024-02-19T21:04:57.856431+0000","last_fullsized":"2024-02-19T21:04:57.856431+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.15","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.782051+0000","last_change":"2024-02-19T21:04:54.627052+0000","last_active":"2024-02-19T21:04:57.782051+0000","last_peered":"2024-02-19T21:04:57.782051+0000","last_clean":"2024-02-19T21:04:57.782051+0000","last_became_active":"2024-02-19T21:04:54.626719+0000","last_became_peered":"2024-02-19T21:04:54.626719+0000","last_unstale":"2024-02-19T21:04:57.782051+0000","last_undegraded":"2024-02-19T21:04:57.782051+0000","last_fullsized":"2024-02-19T21:04:57.782051+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"52.1c","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.798272+0000","last_change":"2024-02-19T21:04:48.559638+0000","last_active":"2024-02-19T21:04:57.798272+0000","last_peered":"2024-02-19T21:04:57.798272+0000","last_clean":"2024-02-19T21:04:57.798272+0000","last_became_active":"2024-02-19T21:04:48.559348+0000","last_became_peered":"2024-02-19T21:04:48.559348+0000","last_unstale":"2024-02-19T21:04:57.798272+0000","last_undegraded":"2024-02-19T21:04:57.798272+0000","last_fullsized":"2024-02-19T21:04:57.798272+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"46.6","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.797891+0000","last_change":"2024-02-19T21:04:46.982073+0000","last_active":"2024-02-19T21:04:57.797891+0000","last_peered":"2024-02-19T21:04:57.797891+0000","last_clean":"2024-02-19T21:04:57.797891+0000","last_became_active":"2024-02-19T21:04:46.981803+0000","last_became_peered":"2024-02-19T21:04:46.981803+0000","last_unstale":"2024-02-19T21:04:57.797891+0000","last_undegraded":"2024-02-19T21:04:57.797891+0000","last_fullsized":"2024-02-19T21:04:57.797891+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.1a","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.779815+0000","last_change":"2024-02-19T21:04:54.635189+0000","last_active":"2024-02-19T21:04:57.779815+0000","last_peered":"2024-02-19T21:04:57.779815+0000","last_clean":"2024-02-19T21:04:57.779815+0000","last_became_active":"2024-02-19T21:04:54.634861+0000","last_became_peered":"2024-02-19T21:04:54.634861+0000","last_unstale":"2024-02-19T21:04:57.779815+0000","last_undegraded":"2024-02-19T21:04:57.779815+0000","last_fullsized":"2024-02-19T21:04:57.779815+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"52.13","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.796440+0000","last_change":"2024-02-19T21:04:48.585695+0000","last_active":"2024-02-19T21:04:57.796440+0000","last_peered":"2024-02-19T21:04:57.796440+0000","last_clean":"2024-02-19T21:04:57.796440+0000","last_became_active":"2024-02-19T21:04:48.585480+0000","last_became_peered":"2024-02-19T21:04:48.585480+0000","last_unstale":"2024-02-19T21:04:57.796440+0000","last_undegraded":"2024-02-19T21:04:57.796440+0000","last_fullsized":"2024-02-19T21:04:57.796440+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"46.9","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.835912+0000","last_change":"2024-02-19T21:04:46.993801+0000","last_active":"2024-02-19T21:04:57.835912+0000","last_peered":"2024-02-19T21:04:57.835912+0000","last_clean":"2024-02-19T21:04:57.835912+0000","last_became_active":"2024-02-19T21:04:46.993645+0000","last_became_peered":"2024-02-19T21:04:46.993645+0000","last_unstale":"2024-02-19T21:04:57.835912+0000","last_undegraded":"2024-02-19T21:04:57.835912+0000","last_fullsized":"2024-02-19T21:04:57.835912+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.1b","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.862415+0000","last_change":"2024-02-19T21:04:54.659000+0000","last_active":"2024-02-19T21:04:57.862415+0000","last_peered":"2024-02-19T21:04:57.862415+0000","last_clean":"2024-02-19T21:04:57.862415+0000","last_became_active":"2024-02-19T21:04:54.658733+0000","last_became_peered":"2024-02-19T21:04:54.658733+0000","last_unstale":"2024-02-19T21:04:57.862415+0000","last_undegraded":"2024-02-19T21:04:57.862415+0000","last_fullsized":"2024-02-19T21:04:57.862415+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"52.12","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.793663+0000","last_change":"2024-02-19T21:04:48.600368+0000","last_active":"2024-02-19T21:04:57.793663+0000","last_peered":"2024-02-19T21:04:57.793663+0000","last_clean":"2024-02-19T21:04:57.793663+0000","last_became_active":"2024-02-19T21:04:48.600047+0000","last_became_peered":"2024-02-19T21:04:48.600047+0000","last_unstale":"2024-02-19T21:04:57.793663+0000","last_undegraded":"2024-02-19T21:04:57.793663+0000","last_fullsized":"2024-02-19T21:04:57.793663+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"46.8","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.792848+0000","last_change":"2024-02-19T21:04:46.985573+0000","last_active":"2024-02-19T21:04:57.792848+0000","last_peered":"2024-02-19T21:04:57.792848+0000","last_clean":"2024-02-19T21:04:57.792848+0000","last_became_active":"2024-02-19T21:04:46.984897+0000","last_became_peered":"2024-02-19T21:04:46.984897+0000","last_unstale":"2024-02-19T21:04:57.792848+0000","last_undegraded":"2024-02-19T21:04:57.792848+0000","last_fullsized":"2024-02-19T21:04:57.792848+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.18","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.814302+0000","last_change":"2024-02-19T21:04:54.610877+0000","last_active":"2024-02-19T21:04:57.814302+0000","last_peered":"2024-02-19T21:04:57.814302+0000","last_clean":"2024-02-19T21:04:57.814302+0000","last_became_active":"2024-02-19T21:04:54.610608+0000","last_became_peered":"2024-02-19T21:04:54.610608+0000","last_unstale":"2024-02-19T21:04:57.814302+0000","last_undegraded":"2024-02-19T21:04:57.814302+0000","last_fullsized":"2024-02-19T21:04:57.814302+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"52.11","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.828190+0000","last_change":"2024-02-19T21:04:48.556367+0000","last_active":"2024-02-19T21:04:57.828190+0000","last_peered":"2024-02-19T21:04:57.828190+0000","last_clean":"2024-02-19T21:04:57.828190+0000","last_became_active":"2024-02-19T21:04:48.556044+0000","last_became_peered":"2024-02-19T21:04:48.556044+0000","last_unstale":"2024-02-19T21:04:57.828190+0000","last_undegraded":"2024-02-19T21:04:57.828190+0000","last_fullsized":"2024-02-19T21:04:57.828190+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.b","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.843146+0000","last_change":"2024-02-19T21:04:46.991729+0000","last_active":"2024-02-19T21:04:57.843146+0000","last_peered":"2024-02-19T21:04:57.843146+0000","last_clean":"2024-02-19T21:04:57.843146+0000","last_became_active":"2024-02-19T21:04:46.991447+0000","last_became_peered":"2024-02-19T21:04:46.991447+0000","last_unstale":"2024-02-19T21:04:57.843146+0000","last_undegraded":"2024-02-19T21:04:57.843146+0000","last_fullsized":"2024-02-19T21:04:57.843146+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.19","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.783032+0000","last_change":"2024-02-19T21:04:54.608992+0000","last_active":"2024-02-19T21:04:57.783032+0000","last_peered":"2024-02-19T21:04:57.783032+0000","last_clean":"2024-02-19T21:04:57.783032+0000","last_became_active":"2024-02-19T21:04:54.608702+0000","last_became_peered":"2024-02-19T21:04:54.608702+0000","last_unstale":"2024-02-19T21:04:57.783032+0000","last_undegraded":"2024-02-19T21:04:57.783032+0000","last_fullsized":"2024-02-19T21:04:57.783032+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"52.10","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.787724+0000","last_change":"2024-02-19T21:04:48.546860+0000","last_active":"2024-02-19T21:04:57.787724+0000","last_peered":"2024-02-19T21:04:57.787724+0000","last_clean":"2024-02-19T21:04:57.787724+0000","last_became_active":"2024-02-19T21:04:48.546515+0000","last_became_peered":"2024-02-19T21:04:48.546515+0000","last_unstale":"2024-02-19T21:04:57.787724+0000","last_undegraded":"2024-02-19T21:04:57.787724+0000","last_fullsized":"2024-02-19T21:04:57.787724+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"46.a","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.780457+0000","last_change":"2024-02-19T21:04:46.981935+0000","last_active":"2024-02-19T21:04:57.780457+0000","last_peered":"2024-02-19T21:04:57.780457+0000","last_clean":"2024-02-19T21:04:57.780457+0000","last_became_active":"2024-02-19T21:04:46.981497+0000","last_became_peered":"2024-02-19T21:04:46.981497+0000","last_unstale":"2024-02-19T21:04:57.780457+0000","last_undegraded":"2024-02-19T21:04:57.780457+0000","last_fullsized":"2024-02-19T21:04:57.780457+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1e","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.785022+0000","last_change":"2024-02-19T21:04:54.629724+0000","last_active":"2024-02-19T21:04:57.785022+0000","last_peered":"2024-02-19T21:04:57.785022+0000","last_clean":"2024-02-19T21:04:57.785022+0000","last_became_active":"2024-02-19T21:04:54.629433+0000","last_became_peered":"2024-02-19T21:04:54.629433+0000","last_unstale":"2024-02-19T21:04:57.785022+0000","last_undegraded":"2024-02-19T21:04:57.785022+0000","last_fullsized":"2024-02-19T21:04:57.785022+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"52.17","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.796025+0000","last_change":"2024-02-19T21:04:48.557710+0000","last_active":"2024-02-19T21:04:57.796025+0000","last_peered":"2024-02-19T21:04:57.796025+0000","last_clean":"2024-02-19T21:04:57.796025+0000","last_became_active":"2024-02-19T21:04:48.557386+0000","last_became_peered":"2024-02-19T21:04:48.557386+0000","last_unstale":"2024-02-19T21:04:57.796025+0000","last_undegraded":"2024-02-19T21:04:57.796025+0000","last_fullsized":"2024-02-19T21:04:57.796025+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"46.d","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.880158+0000","last_change":"2024-02-19T21:04:46.993128+0000","last_active":"2024-02-19T21:04:57.880158+0000","last_peered":"2024-02-19T21:04:57.880158+0000","last_clean":"2024-02-19T21:04:57.880158+0000","last_became_active":"2024-02-19T21:04:46.992869+0000","last_became_peered":"2024-02-19T21:04:46.992869+0000","last_unstale":"2024-02-19T21:04:57.880158+0000","last_undegraded":"2024-02-19T21:04:57.880158+0000","last_fullsized":"2024-02-19T21:04:57.880158+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.1f","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.855573+0000","last_change":"2024-02-19T21:04:54.672244+0000","last_active":"2024-02-19T21:04:57.855573+0000","last_peered":"2024-02-19T21:04:57.855573+0000","last_clean":"2024-02-19T21:04:57.855573+0000","last_became_active":"2024-02-19T21:04:54.671969+0000","last_became_peered":"2024-02-19T21:04:54.671969+0000","last_unstale":"2024-02-19T21:04:57.855573+0000","last_undegraded":"2024-02-19T21:04:57.855573+0000","last_fullsized":"2024-02-19T21:04:57.855573+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"52.16","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.866292+0000","last_change":"2024-02-19T21:04:48.569978+0000","last_active":"2024-02-19T21:04:57.866292+0000","last_peered":"2024-02-19T21:04:57.866292+0000","last_clean":"2024-02-19T21:04:57.866292+0000","last_became_active":"2024-02-19T21:04:48.569752+0000","last_became_peered":"2024-02-19T21:04:48.569752+0000","last_unstale":"2024-02-19T21:04:57.866292+0000","last_undegraded":"2024-02-19T21:04:57.866292+0000","last_fullsized":"2024-02-19T21:04:57.866292+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.c","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.798446+0000","last_change":"2024-02-19T21:04:46.973707+0000","last_active":"2024-02-19T21:04:57.798446+0000","last_peered":"2024-02-19T21:04:57.798446+0000","last_clean":"2024-02-19T21:04:57.798446+0000","last_became_active":"2024-02-19T21:04:46.973128+0000","last_became_peered":"2024-02-19T21:04:46.973128+0000","last_unstale":"2024-02-19T21:04:57.798446+0000","last_undegraded":"2024-02-19T21:04:57.798446+0000","last_fullsized":"2024-02-19T21:04:57.798446+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1c","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.794334+0000","last_change":"2024-02-19T21:04:54.606199+0000","last_active":"2024-02-19T21:04:57.794334+0000","last_peered":"2024-02-19T21:04:57.794334+0000","last_clean":"2024-02-19T21:04:57.794334+0000","last_became_active":"2024-02-19T21:04:54.605898+0000","last_became_peered":"2024-02-19T21:04:54.605898+0000","last_unstale":"2024-02-19T21:04:57.794334+0000","last_undegraded":"2024-02-19T21:04:57.794334+0000","last_fullsized":"2024-02-19T21:04:57.794334+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"52.15","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.782341+0000","last_change":"2024-02-19T21:04:48.565207+0000","last_active":"2024-02-19T21:04:57.782341+0000","last_peered":"2024-02-19T21:04:57.782341+0000","last_clean":"2024-02-19T21:04:57.782341+0000","last_became_active":"2024-02-19T21:04:48.564919+0000","last_became_peered":"2024-02-19T21:04:48.564919+0000","last_unstale":"2024-02-19T21:04:57.782341+0000","last_undegraded":"2024-02-19T21:04:57.782341+0000","last_fullsized":"2024-02-19T21:04:57.782341+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"46.f","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.801349+0000","last_change":"2024-02-19T21:04:46.981761+0000","last_active":"2024-02-19T21:04:57.801349+0000","last_peered":"2024-02-19T21:04:57.801349+0000","last_clean":"2024-02-19T21:04:57.801349+0000","last_became_active":"2024-02-19T21:04:46.981454+0000","last_became_peered":"2024-02-19T21:04:46.981454+0000","last_unstale":"2024-02-19T21:04:57.801349+0000","last_undegraded":"2024-02-19T21:04:57.801349+0000","last_fullsized":"2024-02-19T21:04:57.801349+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1d","version":"0'0","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.794866+0000","last_change":"2024-02-19T21:04:54.656782+0000","last_active":"2024-02-19T21:04:57.794866+0000","last_peered":"2024-02-19T21:04:57.794866+0000","last_clean":"2024-02-19T21:04:57.794866+0000","last_became_active":"2024-02-19T21:04:54.655665+0000","last_became_peered":"2024-02-19T21:04:54.655665+0000","last_unstale":"2024-02-19T21:04:57.794866+0000","last_undegraded":"2024-02-19T21:04:57.794866+0000","last_fullsized":"2024-02-19T21:04:57.794866+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"52.14","version":"0'0","reported_seq":19,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.870838+0000","last_change":"2024-02-19T21:04:48.564606+0000","last_active":"2024-02-19T21:04:57.870838+0000","last_peered":"2024-02-19T21:04:57.870838+0000","last_clean":"2024-02-19T21:04:57.870838+0000","last_became_active":"2024-02-19T21:04:48.559882+0000","last_became_peered":"2024-02-19T21:04:48.559882+0000","last_unstale":"2024-02-19T21:04:57.870838+0000","last_undegraded":"2024-02-19T21:04:57.870838+0000","last_fullsized":"2024-02-19T21:04:57.870838+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:47.936107+0000","last_clean_scrub_stamp":"2024-02-19T21:04:47.936107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.e","version":"0'0","reported_seq":21,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.860704+0000","last_change":"2024-02-19T21:04:46.960885+0000","last_active":"2024-02-19T21:04:57.860704+0000","last_peered":"2024-02-19T21:04:57.860704+0000","last_clean":"2024-02-19T21:04:57.860704+0000","last_became_active":"2024-02-19T21:04:46.960545+0000","last_became_peered":"2024-02-19T21:04:46.960545+0000","last_unstale":"2024-02-19T21:04:57.860704+0000","last_undegraded":"2024-02-19T21:04:57.860704+0000","last_fullsized":"2024-02-19T21:04:57.860704+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.920915+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.920915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"100'1","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.790261+0000","last_change":"2024-02-19T21:04:39.739732+0000","last_active":"2024-02-19T21:04:57.790261+0000","last_peered":"2024-02-19T21:04:57.790261+0000","last_clean":"2024-02-19T21:04:57.790261+0000","last_became_active":"2024-02-19T21:04:39.739003+0000","last_became_peered":"2024-02-19T21:04:39.739003+0000","last_unstale":"2024-02-19T21:04:57.790261+0000","last_undegraded":"2024-02-19T21:04:57.790261+0000","last_fullsized":"2024-02-19T21:04:57.790261+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.797200+0000","last_change":"2024-02-19T21:04:39.640053+0000","last_active":"2024-02-19T21:04:57.797200+0000","last_peered":"2024-02-19T21:04:57.797200+0000","last_clean":"2024-02-19T21:04:57.797200+0000","last_became_active":"2024-02-19T21:04:39.639772+0000","last_became_peered":"2024-02-19T21:04:39.639772+0000","last_unstale":"2024-02-19T21:04:57.797200+0000","last_undegraded":"2024-02-19T21:04:57.797200+0000","last_fullsized":"2024-02-19T21:04:57.797200+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"97'9","reported_seq":129,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-19T21:04:58.785374+0000","last_change":"2024-02-19T21:04:24.046111+0000","last_active":"2024-02-19T21:04:58.785374+0000","last_peered":"2024-02-19T21:04:58.785374+0000","last_clean":"2024-02-19T21:04:58.785374+0000","last_became_active":"2024-02-19T21:01:20.288720+0000","last_became_peered":"2024-02-19T21:01:20.288720+0000","last_unstale":"2024-02-19T21:04:58.785374+0000","last_undegraded":"2024-02-19T21:04:58.785374+0000","last_fullsized":"2024-02-19T21:04:58.785374+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:04:24.046034+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":106,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.788629+0000","last_change":"2024-02-19T21:04:56.123170+0000","last_active":"2024-02-19T21:04:57.788629+0000","last_peered":"2024-02-19T21:04:57.788629+0000","last_clean":"2024-02-19T21:04:57.788629+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:04:57.788629+0000","last_undegraded":"2024-02-19T21:04:57.788629+0000","last_fullsized":"2024-02-19T21:04:57.788629+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:04:56.123096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.123096+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.861224+0000","last_change":"2024-02-19T21:04:11.889869+0000","last_active":"2024-02-19T21:04:57.861224+0000","last_peered":"2024-02-19T21:04:57.861224+0000","last_clean":"2024-02-19T21:04:57.861224+0000","last_became_active":"2024-02-19T21:01:16.271911+0000","last_became_peered":"2024-02-19T21:01:16.271911+0000","last_unstale":"2024-02-19T21:04:57.861224+0000","last_undegraded":"2024-02-19T21:04:57.861224+0000","last_fullsized":"2024-02-19T21:04:57.861224+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.889778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:37.258580+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.889778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":69,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.791390+0000","last_change":"2024-02-19T21:03:10.017843+0000","last_active":"2024-02-19T21:04:57.791390+0000","last_peered":"2024-02-19T21:04:57.791390+0000","last_clean":"2024-02-19T21:04:57.791390+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:04:57.791390+0000","last_undegraded":"2024-02-19T21:04:57.791390+0000","last_fullsized":"2024-02-19T21:04:57.791390+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:10.017764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:10.017764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.864153+0000","last_change":"2024-02-19T21:04:39.703479+0000","last_active":"2024-02-19T21:04:57.864153+0000","last_peered":"2024-02-19T21:04:57.864153+0000","last_clean":"2024-02-19T21:04:57.864153+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:04:57.864153+0000","last_undegraded":"2024-02-19T21:04:57.864153+0000","last_fullsized":"2024-02-19T21:04:57.864153+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.13","version":"102'3","reported_seq":34,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.817708+0000","last_change":"2024-02-19T21:04:39.785733+0000","last_active":"2024-02-19T21:04:57.817708+0000","last_peered":"2024-02-19T21:04:57.817708+0000","last_clean":"2024-02-19T21:04:57.817708+0000","last_became_active":"2024-02-19T21:04:39.784009+0000","last_became_peered":"2024-02-19T21:04:39.784009+0000","last_unstale":"2024-02-19T21:04:57.817708+0000","last_undegraded":"2024-02-19T21:04:57.817708+0000","last_fullsized":"2024-02-19T21:04:57.817708+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165843,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.805604+0000","last_change":"2024-02-19T21:04:39.772957+0000","last_active":"2024-02-19T21:04:57.805604+0000","last_peered":"2024-02-19T21:04:57.805604+0000","last_clean":"2024-02-19T21:04:57.805604+0000","last_became_active":"2024-02-19T21:04:39.771391+0000","last_became_peered":"2024-02-19T21:04:39.771391+0000","last_unstale":"2024-02-19T21:04:57.805604+0000","last_undegraded":"2024-02-19T21:04:57.805604+0000","last_fullsized":"2024-02-19T21:04:57.805604+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.843879+0000","last_change":"2024-02-19T21:04:56.263016+0000","last_active":"2024-02-19T21:04:57.843879+0000","last_peered":"2024-02-19T21:04:57.843879+0000","last_clean":"2024-02-19T21:04:57.843879+0000","last_became_active":"2024-02-19T21:01:18.288322+0000","last_became_peered":"2024-02-19T21:01:18.288322+0000","last_unstale":"2024-02-19T21:04:57.843879+0000","last_undegraded":"2024-02-19T21:04:57.843879+0000","last_fullsized":"2024-02-19T21:04:57.843879+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:56.262927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.262927+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":89,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.801871+0000","last_change":"2024-02-19T21:04:12.927016+0000","last_active":"2024-02-19T21:04:57.801871+0000","last_peered":"2024-02-19T21:04:57.801871+0000","last_clean":"2024-02-19T21:04:57.801871+0000","last_became_active":"2024-02-19T21:04:12.926645+0000","last_became_peered":"2024-02-19T21:04:12.926645+0000","last_unstale":"2024-02-19T21:04:57.801871+0000","last_undegraded":"2024-02-19T21:04:57.801871+0000","last_fullsized":"2024-02-19T21:04:57.801871+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:30.462868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:30.462868+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.806509+0000","last_change":"2024-02-19T21:04:17.831164+0000","last_active":"2024-02-19T21:04:57.806509+0000","last_peered":"2024-02-19T21:04:57.806509+0000","last_clean":"2024-02-19T21:04:57.806509+0000","last_became_active":"2024-02-19T21:01:20.290566+0000","last_became_peered":"2024-02-19T21:01:20.290566+0000","last_unstale":"2024-02-19T21:04:57.806509+0000","last_undegraded":"2024-02-19T21:04:57.806509+0000","last_fullsized":"2024-02-19T21:04:57.806509+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:17.831065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:17.831065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.836329+0000","last_change":"2024-02-19T21:04:21.659690+0000","last_active":"2024-02-19T21:04:57.836329+0000","last_peered":"2024-02-19T21:04:57.836329+0000","last_clean":"2024-02-19T21:04:57.836329+0000","last_became_active":"2024-02-19T21:01:22.297998+0000","last_became_peered":"2024-02-19T21:01:22.297998+0000","last_unstale":"2024-02-19T21:04:57.836329+0000","last_undegraded":"2024-02-19T21:04:57.836329+0000","last_fullsized":"2024-02-19T21:04:57.836329+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:21.659616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.659616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.796565+0000","last_change":"2024-02-19T21:04:39.765776+0000","last_active":"2024-02-19T21:04:57.796565+0000","last_peered":"2024-02-19T21:04:57.796565+0000","last_clean":"2024-02-19T21:04:57.796565+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:04:57.796565+0000","last_undegraded":"2024-02-19T21:04:57.796565+0000","last_fullsized":"2024-02-19T21:04:57.796565+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.12","version":"101'2","reported_seq":32,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.844366+0000","last_change":"2024-02-19T21:04:39.785340+0000","last_active":"2024-02-19T21:04:57.844366+0000","last_peered":"2024-02-19T21:04:57.844366+0000","last_clean":"2024-02-19T21:04:57.844366+0000","last_became_active":"2024-02-19T21:04:39.784930+0000","last_became_peered":"2024-02-19T21:04:39.784930+0000","last_unstale":"2024-02-19T21:04:57.844366+0000","last_undegraded":"2024-02-19T21:04:57.844366+0000","last_fullsized":"2024-02-19T21:04:57.844366+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"100'9","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.884529+0000","last_change":"2024-02-19T21:04:39.731654+0000","last_active":"2024-02-19T21:04:57.884529+0000","last_peered":"2024-02-19T21:04:57.884529+0000","last_clean":"2024-02-19T21:04:57.884529+0000","last_became_active":"2024-02-19T21:04:39.731439+0000","last_became_peered":"2024-02-19T21:04:39.731439+0000","last_unstale":"2024-02-19T21:04:57.884529+0000","last_undegraded":"2024-02-19T21:04:57.884529+0000","last_fullsized":"2024-02-19T21:04:57.884529+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":109,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.799938+0000","last_change":"2024-02-19T21:04:20.836693+0000","last_active":"2024-02-19T21:04:57.799938+0000","last_peered":"2024-02-19T21:04:57.799938+0000","last_clean":"2024-02-19T21:04:57.799938+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:04:57.799938+0000","last_undegraded":"2024-02-19T21:04:57.799938+0000","last_fullsized":"2024-02-19T21:04:57.799938+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:04:20.836609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:20.836609+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.863776+0000","last_change":"2024-02-19T21:04:03.767826+0000","last_active":"2024-02-19T21:04:57.863776+0000","last_peered":"2024-02-19T21:04:57.863776+0000","last_clean":"2024-02-19T21:04:57.863776+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:04:57.863776+0000","last_undegraded":"2024-02-19T21:04:57.863776+0000","last_fullsized":"2024-02-19T21:04:57.863776+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:03.767713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:03.767713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.817525+0000","last_change":"2024-02-19T21:04:51.822914+0000","last_active":"2024-02-19T21:04:57.817525+0000","last_peered":"2024-02-19T21:04:57.817525+0000","last_clean":"2024-02-19T21:04:57.817525+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:04:57.817525+0000","last_undegraded":"2024-02-19T21:04:57.817525+0000","last_fullsized":"2024-02-19T21:04:57.817525+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:51.822845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:51.822845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":68,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.778994+0000","last_change":"2024-02-19T21:03:03.500049+0000","last_active":"2024-02-19T21:04:57.778994+0000","last_peered":"2024-02-19T21:04:57.778994+0000","last_clean":"2024-02-19T21:04:57.778994+0000","last_became_active":"2024-02-19T21:01:22.291759+0000","last_became_peered":"2024-02-19T21:01:22.291759+0000","last_unstale":"2024-02-19T21:04:57.778994+0000","last_undegraded":"2024-02-19T21:04:57.778994+0000","last_fullsized":"2024-02-19T21:04:57.778994+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:03.499960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:03.499960+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.785766+0000","last_change":"2024-02-19T21:04:39.620073+0000","last_active":"2024-02-19T21:04:57.785766+0000","last_peered":"2024-02-19T21:04:57.785766+0000","last_clean":"2024-02-19T21:04:57.785766+0000","last_became_active":"2024-02-19T21:04:39.618614+0000","last_became_peered":"2024-02-19T21:04:39.618614+0000","last_unstale":"2024-02-19T21:04:57.785766+0000","last_undegraded":"2024-02-19T21:04:57.785766+0000","last_fullsized":"2024-02-19T21:04:57.785766+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.804976+0000","last_change":"2024-02-19T21:04:39.690722+0000","last_active":"2024-02-19T21:04:57.804976+0000","last_peered":"2024-02-19T21:04:57.804976+0000","last_clean":"2024-02-19T21:04:57.804976+0000","last_became_active":"2024-02-19T21:04:39.689794+0000","last_became_peered":"2024-02-19T21:04:39.689794+0000","last_unstale":"2024-02-19T21:04:57.804976+0000","last_undegraded":"2024-02-19T21:04:57.804976+0000","last_fullsized":"2024-02-19T21:04:57.804976+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":133,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.802872+0000","last_change":"2024-02-19T21:04:03.152903+0000","last_active":"2024-02-19T21:04:57.802872+0000","last_peered":"2024-02-19T21:04:57.802872+0000","last_clean":"2024-02-19T21:04:57.802872+0000","last_became_active":"2024-02-19T21:01:30.923214+0000","last_became_peered":"2024-02-19T21:01:30.923214+0000","last_unstale":"2024-02-19T21:04:57.802872+0000","last_undegraded":"2024-02-19T21:04:57.802872+0000","last_fullsized":"2024-02-19T21:04:57.802872+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:04:03.152817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:03.152817+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":85,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.783422+0000","last_change":"2024-02-19T21:04:21.952701+0000","last_active":"2024-02-19T21:04:57.783422+0000","last_peered":"2024-02-19T21:04:57.783422+0000","last_clean":"2024-02-19T21:04:57.783422+0000","last_became_active":"2024-02-19T21:01:16.274937+0000","last_became_peered":"2024-02-19T21:01:16.274937+0000","last_unstale":"2024-02-19T21:04:57.783422+0000","last_undegraded":"2024-02-19T21:04:57.783422+0000","last_fullsized":"2024-02-19T21:04:57.783422+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:04:21.952586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.952586+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.799006+0000","last_change":"2024-02-19T21:04:04.386811+0000","last_active":"2024-02-19T21:04:57.799006+0000","last_peered":"2024-02-19T21:04:57.799006+0000","last_clean":"2024-02-19T21:04:57.799006+0000","last_became_active":"2024-02-19T21:01:20.290687+0000","last_became_peered":"2024-02-19T21:01:20.290687+0000","last_unstale":"2024-02-19T21:04:57.799006+0000","last_undegraded":"2024-02-19T21:04:57.799006+0000","last_fullsized":"2024-02-19T21:04:57.799006+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:04.386735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:04.386735+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.857164+0000","last_change":"2024-02-19T21:04:19.845711+0000","last_active":"2024-02-19T21:04:57.857164+0000","last_peered":"2024-02-19T21:04:57.857164+0000","last_clean":"2024-02-19T21:04:57.857164+0000","last_became_active":"2024-02-19T21:01:22.298941+0000","last_became_peered":"2024-02-19T21:01:22.298941+0000","last_unstale":"2024-02-19T21:04:57.857164+0000","last_undegraded":"2024-02-19T21:04:57.857164+0000","last_fullsized":"2024-02-19T21:04:57.857164+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.845609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"100'5","reported_seq":34,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.811758+0000","last_change":"2024-02-19T21:04:39.781921+0000","last_active":"2024-02-19T21:04:57.811758+0000","last_peered":"2024-02-19T21:04:57.811758+0000","last_clean":"2024-02-19T21:04:57.811758+0000","last_became_active":"2024-02-19T21:04:39.778460+0000","last_became_peered":"2024-02-19T21:04:39.778460+0000","last_unstale":"2024-02-19T21:04:57.811758+0000","last_undegraded":"2024-02-19T21:04:57.811758+0000","last_fullsized":"2024-02-19T21:04:57.811758+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.811866+0000","last_change":"2024-02-19T21:04:39.699479+0000","last_active":"2024-02-19T21:04:57.811866+0000","last_peered":"2024-02-19T21:04:57.811866+0000","last_clean":"2024-02-19T21:04:57.811866+0000","last_became_active":"2024-02-19T21:04:39.698684+0000","last_became_peered":"2024-02-19T21:04:39.698684+0000","last_unstale":"2024-02-19T21:04:57.811866+0000","last_undegraded":"2024-02-19T21:04:57.811866+0000","last_fullsized":"2024-02-19T21:04:57.811866+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.14","version":"100'6","reported_seq":35,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.874586+0000","last_change":"2024-02-19T21:04:39.757025+0000","last_active":"2024-02-19T21:04:57.874586+0000","last_peered":"2024-02-19T21:04:57.874586+0000","last_clean":"2024-02-19T21:04:57.874586+0000","last_became_active":"2024-02-19T21:04:39.755747+0000","last_became_peered":"2024-02-19T21:04:39.755747+0000","last_unstale":"2024-02-19T21:04:57.874586+0000","last_undegraded":"2024-02-19T21:04:57.874586+0000","last_fullsized":"2024-02-19T21:04:57.874586+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1024,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"82'15","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.790940+0000","last_change":"2024-02-19T21:04:49.285676+0000","last_active":"2024-02-19T21:04:57.790940+0000","last_peered":"2024-02-19T21:04:57.790940+0000","last_clean":"2024-02-19T21:04:57.790940+0000","last_became_active":"2024-02-19T21:01:18.282805+0000","last_became_peered":"2024-02-19T21:01:18.282805+0000","last_unstale":"2024-02-19T21:04:57.790940+0000","last_undegraded":"2024-02-19T21:04:57.790940+0000","last_fullsized":"2024-02-19T21:04:57.790940+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:04:49.285599+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.285599+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.794398+0000","last_change":"2024-02-19T21:04:11.303670+0000","last_active":"2024-02-19T21:04:57.794398+0000","last_peered":"2024-02-19T21:04:57.794398+0000","last_clean":"2024-02-19T21:04:57.794398+0000","last_became_active":"2024-02-19T21:01:16.274086+0000","last_became_peered":"2024-02-19T21:01:16.274086+0000","last_unstale":"2024-02-19T21:04:57.794398+0000","last_undegraded":"2024-02-19T21:04:57.794398+0000","last_fullsized":"2024-02-19T21:04:57.794398+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.303572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.303572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":129,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-19T21:04:58.809731+0000","last_change":"2024-02-19T21:04:07.809065+0000","last_active":"2024-02-19T21:04:58.809731+0000","last_peered":"2024-02-19T21:04:58.809731+0000","last_clean":"2024-02-19T21:04:58.809731+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:04:58.809731+0000","last_undegraded":"2024-02-19T21:04:58.809731+0000","last_fullsized":"2024-02-19T21:04:58.809731+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:07.808944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:07.808944+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.797385+0000","last_change":"2024-02-19T21:04:21.031646+0000","last_active":"2024-02-19T21:04:57.797385+0000","last_peered":"2024-02-19T21:04:57.797385+0000","last_clean":"2024-02-19T21:04:57.797385+0000","last_became_active":"2024-02-19T21:01:22.299993+0000","last_became_peered":"2024-02-19T21:01:22.299993+0000","last_unstale":"2024-02-19T21:04:57.797385+0000","last_undegraded":"2024-02-19T21:04:57.797385+0000","last_fullsized":"2024-02-19T21:04:57.797385+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.031577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"100'5","reported_seq":34,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.821715+0000","last_change":"2024-02-19T21:04:39.746708+0000","last_active":"2024-02-19T21:04:57.821715+0000","last_peered":"2024-02-19T21:04:57.821715+0000","last_clean":"2024-02-19T21:04:57.821715+0000","last_became_active":"2024-02-19T21:04:39.742728+0000","last_became_peered":"2024-02-19T21:04:39.742728+0000","last_unstale":"2024-02-19T21:04:57.821715+0000","last_undegraded":"2024-02-19T21:04:57.821715+0000","last_fullsized":"2024-02-19T21:04:57.821715+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.832085+0000","last_change":"2024-02-19T21:04:39.766043+0000","last_active":"2024-02-19T21:04:57.832085+0000","last_peered":"2024-02-19T21:04:57.832085+0000","last_clean":"2024-02-19T21:04:57.832085+0000","last_became_active":"2024-02-19T21:04:39.765782+0000","last_became_peered":"2024-02-19T21:04:39.765782+0000","last_unstale":"2024-02-19T21:04:57.832085+0000","last_undegraded":"2024-02-19T21:04:57.832085+0000","last_fullsized":"2024-02-19T21:04:57.832085+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"102'7","reported_seq":43,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.780171+0000","last_change":"2024-02-19T21:04:39.659021+0000","last_active":"2024-02-19T21:04:57.780171+0000","last_peered":"2024-02-19T21:04:57.780171+0000","last_clean":"2024-02-19T21:04:57.780171+0000","last_became_active":"2024-02-19T21:04:39.658450+0000","last_became_peered":"2024-02-19T21:04:39.658450+0000","last_unstale":"2024-02-19T21:04:57.780171+0000","last_undegraded":"2024-02-19T21:04:57.780171+0000","last_fullsized":"2024-02-19T21:04:57.780171+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165900,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"100'5","reported_seq":34,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.802040+0000","last_change":"2024-02-19T21:04:39.676464+0000","last_active":"2024-02-19T21:04:57.802040+0000","last_peered":"2024-02-19T21:04:57.802040+0000","last_clean":"2024-02-19T21:04:57.802040+0000","last_became_active":"2024-02-19T21:04:39.675580+0000","last_became_peered":"2024-02-19T21:04:39.675580+0000","last_unstale":"2024-02-19T21:04:57.802040+0000","last_undegraded":"2024-02-19T21:04:57.802040+0000","last_fullsized":"2024-02-19T21:04:57.802040+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":98,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.805681+0000","last_change":"2024-02-19T21:02:53.526211+0000","last_active":"2024-02-19T21:04:57.805681+0000","last_peered":"2024-02-19T21:04:57.805681+0000","last_clean":"2024-02-19T21:04:57.805681+0000","last_became_active":"2024-02-19T21:01:18.285534+0000","last_became_peered":"2024-02-19T21:01:18.285534+0000","last_unstale":"2024-02-19T21:04:57.805681+0000","last_undegraded":"2024-02-19T21:04:57.805681+0000","last_fullsized":"2024-02-19T21:04:57.805681+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:02:53.526141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:53.526141+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.812030+0000","last_change":"2024-02-19T21:04:12.305976+0000","last_active":"2024-02-19T21:04:57.812030+0000","last_peered":"2024-02-19T21:04:57.812030+0000","last_clean":"2024-02-19T21:04:57.812030+0000","last_became_active":"2024-02-19T21:01:16.274939+0000","last_became_peered":"2024-02-19T21:01:16.274939+0000","last_unstale":"2024-02-19T21:04:57.812030+0000","last_undegraded":"2024-02-19T21:04:57.812030+0000","last_fullsized":"2024-02-19T21:04:57.812030+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:12.305886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:12.305886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":129,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-19T21:04:58.788231+0000","last_change":"2024-02-19T21:04:16.059715+0000","last_active":"2024-02-19T21:04:58.788231+0000","last_peered":"2024-02-19T21:04:58.788231+0000","last_clean":"2024-02-19T21:04:58.788231+0000","last_became_active":"2024-02-19T21:01:20.293341+0000","last_became_peered":"2024-02-19T21:01:20.293341+0000","last_unstale":"2024-02-19T21:04:58.788231+0000","last_undegraded":"2024-02-19T21:04:58.788231+0000","last_fullsized":"2024-02-19T21:04:58.788231+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:16.059648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.059648+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.795896+0000","last_change":"2024-02-19T21:04:08.185818+0000","last_active":"2024-02-19T21:04:57.795896+0000","last_peered":"2024-02-19T21:04:57.795896+0000","last_clean":"2024-02-19T21:04:57.795896+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:04:57.795896+0000","last_undegraded":"2024-02-19T21:04:57.795896+0000","last_fullsized":"2024-02-19T21:04:57.795896+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.185717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.185717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.793369+0000","last_change":"2024-02-19T21:04:39.657326+0000","last_active":"2024-02-19T21:04:57.793369+0000","last_peered":"2024-02-19T21:04:57.793369+0000","last_clean":"2024-02-19T21:04:57.793369+0000","last_became_active":"2024-02-19T21:04:39.657052+0000","last_became_peered":"2024-02-19T21:04:39.657052+0000","last_unstale":"2024-02-19T21:04:57.793369+0000","last_undegraded":"2024-02-19T21:04:57.793369+0000","last_fullsized":"2024-02-19T21:04:57.793369+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.885168+0000","last_change":"2024-02-19T21:04:39.800157+0000","last_active":"2024-02-19T21:04:57.885168+0000","last_peered":"2024-02-19T21:04:57.885168+0000","last_clean":"2024-02-19T21:04:57.885168+0000","last_became_active":"2024-02-19T21:04:39.798967+0000","last_became_peered":"2024-02-19T21:04:39.798967+0000","last_unstale":"2024-02-19T21:04:57.885168+0000","last_undegraded":"2024-02-19T21:04:57.885168+0000","last_fullsized":"2024-02-19T21:04:57.885168+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.781347+0000","last_change":"2024-02-19T21:04:39.662708+0000","last_active":"2024-02-19T21:04:57.781347+0000","last_peered":"2024-02-19T21:04:57.781347+0000","last_clean":"2024-02-19T21:04:57.781347+0000","last_became_active":"2024-02-19T21:04:39.662412+0000","last_became_peered":"2024-02-19T21:04:39.662412+0000","last_unstale":"2024-02-19T21:04:57.781347+0000","last_undegraded":"2024-02-19T21:04:57.781347+0000","last_fullsized":"2024-02-19T21:04:57.781347+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.797096+0000","last_change":"2024-02-19T21:03:35.086667+0000","last_active":"2024-02-19T21:04:57.797096+0000","last_peered":"2024-02-19T21:04:57.797096+0000","last_clean":"2024-02-19T21:04:57.797096+0000","last_became_active":"2024-02-19T21:01:18.285670+0000","last_became_peered":"2024-02-19T21:01:18.285670+0000","last_unstale":"2024-02-19T21:04:57.797096+0000","last_undegraded":"2024-02-19T21:04:57.797096+0000","last_fullsized":"2024-02-19T21:04:57.797096+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:03:35.086579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:35.086579+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.779250+0000","last_change":"2024-02-19T21:04:08.858046+0000","last_active":"2024-02-19T21:04:57.779250+0000","last_peered":"2024-02-19T21:04:57.779250+0000","last_clean":"2024-02-19T21:04:57.779250+0000","last_became_active":"2024-02-19T21:01:16.275520+0000","last_became_peered":"2024-02-19T21:01:16.275520+0000","last_unstale":"2024-02-19T21:04:57.779250+0000","last_undegraded":"2024-02-19T21:04:57.779250+0000","last_fullsized":"2024-02-19T21:04:57.779250+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.857973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.857973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.796855+0000","last_change":"2024-02-19T21:04:16.217810+0000","last_active":"2024-02-19T21:04:57.796855+0000","last_peered":"2024-02-19T21:04:57.796855+0000","last_clean":"2024-02-19T21:04:57.796855+0000","last_became_active":"2024-02-19T21:01:20.289375+0000","last_became_peered":"2024-02-19T21:01:20.289375+0000","last_unstale":"2024-02-19T21:04:57.796855+0000","last_undegraded":"2024-02-19T21:04:57.796855+0000","last_fullsized":"2024-02-19T21:04:57.796855+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:16.217744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.217744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":69,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.871264+0000","last_change":"2024-02-19T21:03:23.656785+0000","last_active":"2024-02-19T21:04:57.871264+0000","last_peered":"2024-02-19T21:04:57.871264+0000","last_clean":"2024-02-19T21:04:57.871264+0000","last_became_active":"2024-02-19T21:01:22.607007+0000","last_became_peered":"2024-02-19T21:01:22.607007+0000","last_unstale":"2024-02-19T21:04:57.871264+0000","last_undegraded":"2024-02-19T21:04:57.871264+0000","last_fullsized":"2024-02-19T21:04:57.871264+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:23.656688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:23.656688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.854427+0000","last_change":"2024-02-19T21:04:39.744505+0000","last_active":"2024-02-19T21:04:57.854427+0000","last_peered":"2024-02-19T21:04:57.854427+0000","last_clean":"2024-02-19T21:04:57.854427+0000","last_became_active":"2024-02-19T21:04:39.744231+0000","last_became_peered":"2024-02-19T21:04:39.744231+0000","last_unstale":"2024-02-19T21:04:57.854427+0000","last_undegraded":"2024-02-19T21:04:57.854427+0000","last_fullsized":"2024-02-19T21:04:57.854427+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.791424+0000","last_change":"2024-02-19T21:04:39.640944+0000","last_active":"2024-02-19T21:04:57.791424+0000","last_peered":"2024-02-19T21:04:57.791424+0000","last_clean":"2024-02-19T21:04:57.791424+0000","last_became_active":"2024-02-19T21:04:39.640667+0000","last_became_peered":"2024-02-19T21:04:39.640667+0000","last_unstale":"2024-02-19T21:04:57.791424+0000","last_undegraded":"2024-02-19T21:04:57.791424+0000","last_fullsized":"2024-02-19T21:04:57.791424+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.d","version":"100'5","reported_seq":34,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.795481+0000","last_change":"2024-02-19T21:04:39.631816+0000","last_active":"2024-02-19T21:04:57.795481+0000","last_peered":"2024-02-19T21:04:57.795481+0000","last_clean":"2024-02-19T21:04:57.795481+0000","last_became_active":"2024-02-19T21:04:39.631449+0000","last_became_peered":"2024-02-19T21:04:39.631449+0000","last_unstale":"2024-02-19T21:04:57.795481+0000","last_undegraded":"2024-02-19T21:04:57.795481+0000","last_fullsized":"2024-02-19T21:04:57.795481+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":129,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-19T21:04:58.825272+0000","last_change":"2024-02-19T21:03:33.736454+0000","last_active":"2024-02-19T21:04:58.825272+0000","last_peered":"2024-02-19T21:04:58.825272+0000","last_clean":"2024-02-19T21:04:58.825272+0000","last_became_active":"2024-02-19T21:01:20.286505+0000","last_became_peered":"2024-02-19T21:01:20.286505+0000","last_unstale":"2024-02-19T21:04:58.825272+0000","last_undegraded":"2024-02-19T21:04:58.825272+0000","last_fullsized":"2024-02-19T21:04:58.825272+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:33.736366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:33.736366+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":99,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.812313+0000","last_change":"2024-02-19T21:04:50.266262+0000","last_active":"2024-02-19T21:04:57.812313+0000","last_peered":"2024-02-19T21:04:57.812313+0000","last_clean":"2024-02-19T21:04:57.812313+0000","last_became_active":"2024-02-19T21:01:18.282702+0000","last_became_peered":"2024-02-19T21:01:18.282702+0000","last_unstale":"2024-02-19T21:04:57.812313+0000","last_undegraded":"2024-02-19T21:04:57.812313+0000","last_fullsized":"2024-02-19T21:04:57.812313+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:04:50.266152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:50.266152+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.801887+0000","last_change":"2024-02-19T21:04:22.800646+0000","last_active":"2024-02-19T21:04:57.801887+0000","last_peered":"2024-02-19T21:04:57.801887+0000","last_clean":"2024-02-19T21:04:57.801887+0000","last_became_active":"2024-02-19T21:01:16.274462+0000","last_became_peered":"2024-02-19T21:01:16.274462+0000","last_unstale":"2024-02-19T21:04:57.801887+0000","last_undegraded":"2024-02-19T21:04:57.801887+0000","last_fullsized":"2024-02-19T21:04:57.801887+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:22.800578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:22.800578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":68,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.783811+0000","last_change":"2024-02-19T21:03:01.496926+0000","last_active":"2024-02-19T21:04:57.783811+0000","last_peered":"2024-02-19T21:04:57.783811+0000","last_clean":"2024-02-19T21:04:57.783811+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:04:57.783811+0000","last_undegraded":"2024-02-19T21:04:57.783811+0000","last_fullsized":"2024-02-19T21:04:57.783811+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:01.496833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:01.496833+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.884001+0000","last_change":"2024-02-19T21:04:39.659690+0000","last_active":"2024-02-19T21:04:57.884001+0000","last_peered":"2024-02-19T21:04:57.884001+0000","last_clean":"2024-02-19T21:04:57.884001+0000","last_became_active":"2024-02-19T21:04:39.658536+0000","last_became_peered":"2024-02-19T21:04:39.658536+0000","last_unstale":"2024-02-19T21:04:57.884001+0000","last_undegraded":"2024-02-19T21:04:57.884001+0000","last_fullsized":"2024-02-19T21:04:57.884001+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.18","version":"102'4","reported_seq":36,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.822006+0000","last_change":"2024-02-19T21:04:39.748257+0000","last_active":"2024-02-19T21:04:57.822006+0000","last_peered":"2024-02-19T21:04:57.822006+0000","last_clean":"2024-02-19T21:04:57.822006+0000","last_became_active":"2024-02-19T21:04:39.747589+0000","last_became_peered":"2024-02-19T21:04:39.747589+0000","last_unstale":"2024-02-19T21:04:57.822006+0000","last_undegraded":"2024-02-19T21:04:57.822006+0000","last_fullsized":"2024-02-19T21:04:57.822006+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50331664,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":4,"num_write_kb":51902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.868121+0000","last_change":"2024-02-19T21:04:39.769987+0000","last_active":"2024-02-19T21:04:57.868121+0000","last_peered":"2024-02-19T21:04:57.868121+0000","last_clean":"2024-02-19T21:04:57.868121+0000","last_became_active":"2024-02-19T21:04:39.767371+0000","last_became_peered":"2024-02-19T21:04:39.767371+0000","last_unstale":"2024-02-19T21:04:57.868121+0000","last_undegraded":"2024-02-19T21:04:57.868121+0000","last_fullsized":"2024-02-19T21:04:57.868121+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":71,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.812437+0000","last_change":"2024-02-19T21:02:57.997535+0000","last_active":"2024-02-19T21:04:57.812437+0000","last_peered":"2024-02-19T21:04:57.812437+0000","last_clean":"2024-02-19T21:04:57.812437+0000","last_became_active":"2024-02-19T21:01:20.291079+0000","last_became_peered":"2024-02-19T21:01:20.291079+0000","last_unstale":"2024-02-19T21:04:57.812437+0000","last_undegraded":"2024-02-19T21:04:57.812437+0000","last_fullsized":"2024-02-19T21:04:57.812437+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.997427+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.997427+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":110,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.822078+0000","last_change":"2024-02-19T21:04:08.837761+0000","last_active":"2024-02-19T21:04:57.822078+0000","last_peered":"2024-02-19T21:04:57.822078+0000","last_clean":"2024-02-19T21:04:57.822078+0000","last_became_active":"2024-02-19T21:01:18.293812+0000","last_became_peered":"2024-02-19T21:01:18.293812+0000","last_unstale":"2024-02-19T21:04:57.822078+0000","last_undegraded":"2024-02-19T21:04:57.822078+0000","last_fullsized":"2024-02-19T21:04:57.822078+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:04:08.837652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.837652+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.804588+0000","last_change":"2024-02-19T21:04:21.205943+0000","last_active":"2024-02-19T21:04:57.804588+0000","last_peered":"2024-02-19T21:04:57.804588+0000","last_clean":"2024-02-19T21:04:57.804588+0000","last_became_active":"2024-02-19T21:01:16.271340+0000","last_became_peered":"2024-02-19T21:01:16.271340+0000","last_unstale":"2024-02-19T21:04:57.804588+0000","last_undegraded":"2024-02-19T21:04:57.804588+0000","last_fullsized":"2024-02-19T21:04:57.804588+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:21.205829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.205829+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":69,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.801625+0000","last_change":"2024-02-19T21:03:17.088777+0000","last_active":"2024-02-19T21:04:57.801625+0000","last_peered":"2024-02-19T21:04:57.801625+0000","last_clean":"2024-02-19T21:04:57.801625+0000","last_became_active":"2024-02-19T21:01:22.297707+0000","last_became_peered":"2024-02-19T21:01:22.297707+0000","last_unstale":"2024-02-19T21:04:57.801625+0000","last_undegraded":"2024-02-19T21:04:57.801625+0000","last_fullsized":"2024-02-19T21:04:57.801625+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:17.088695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:17.088695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.868195+0000","last_change":"2024-02-19T21:04:39.770138+0000","last_active":"2024-02-19T21:04:57.868195+0000","last_peered":"2024-02-19T21:04:57.868195+0000","last_clean":"2024-02-19T21:04:57.868195+0000","last_became_active":"2024-02-19T21:04:39.767723+0000","last_became_peered":"2024-02-19T21:04:39.767723+0000","last_unstale":"2024-02-19T21:04:57.868195+0000","last_undegraded":"2024-02-19T21:04:57.868195+0000","last_fullsized":"2024-02-19T21:04:57.868195+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"100'1","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.817450+0000","last_change":"2024-02-19T21:04:39.786049+0000","last_active":"2024-02-19T21:04:57.817450+0000","last_peered":"2024-02-19T21:04:57.817450+0000","last_clean":"2024-02-19T21:04:57.817450+0000","last_became_active":"2024-02-19T21:04:39.784742+0000","last_became_peered":"2024-02-19T21:04:39.784742+0000","last_unstale":"2024-02-19T21:04:57.817450+0000","last_undegraded":"2024-02-19T21:04:57.817450+0000","last_fullsized":"2024-02-19T21:04:57.817450+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.869961+0000","last_change":"2024-02-19T21:04:39.722429+0000","last_active":"2024-02-19T21:04:57.869961+0000","last_peered":"2024-02-19T21:04:57.869961+0000","last_clean":"2024-02-19T21:04:57.869961+0000","last_became_active":"2024-02-19T21:04:39.711377+0000","last_became_peered":"2024-02-19T21:04:39.711377+0000","last_unstale":"2024-02-19T21:04:57.869961+0000","last_undegraded":"2024-02-19T21:04:57.869961+0000","last_fullsized":"2024-02-19T21:04:57.869961+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":81,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.800012+0000","last_change":"2024-02-19T21:04:56.720699+0000","last_active":"2024-02-19T21:04:57.800012+0000","last_peered":"2024-02-19T21:04:57.800012+0000","last_clean":"2024-02-19T21:04:57.800012+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:04:57.800012+0000","last_undegraded":"2024-02-19T21:04:57.800012+0000","last_fullsized":"2024-02-19T21:04:57.800012+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:56.720607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.720607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.843727+0000","last_change":"2024-02-19T21:04:07.592349+0000","last_active":"2024-02-19T21:04:57.843727+0000","last_peered":"2024-02-19T21:04:57.843727+0000","last_clean":"2024-02-19T21:04:57.843727+0000","last_became_active":"2024-02-19T21:01:18.293791+0000","last_became_peered":"2024-02-19T21:01:18.293791+0000","last_unstale":"2024-02-19T21:04:57.843727+0000","last_undegraded":"2024-02-19T21:04:57.843727+0000","last_fullsized":"2024-02-19T21:04:57.843727+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:04:07.592209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:07.592209+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":81,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:58.273215+0000","last_change":"2024-02-19T21:04:58.273181+0000","last_active":"2024-02-19T21:04:58.273215+0000","last_peered":"2024-02-19T21:04:58.273215+0000","last_clean":"2024-02-19T21:04:58.273215+0000","last_became_active":"2024-02-19T21:01:16.274215+0000","last_became_peered":"2024-02-19T21:01:16.274215+0000","last_unstale":"2024-02-19T21:04:58.273215+0000","last_undegraded":"2024-02-19T21:04:58.273215+0000","last_fullsized":"2024-02-19T21:04:58.273215+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:58.273113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:58.273113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":69,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.814859+0000","last_change":"2024-02-19T21:03:07.473156+0000","last_active":"2024-02-19T21:04:57.814859+0000","last_peered":"2024-02-19T21:04:57.814859+0000","last_clean":"2024-02-19T21:04:57.814859+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:04:57.814859+0000","last_undegraded":"2024-02-19T21:04:57.814859+0000","last_fullsized":"2024-02-19T21:04:57.814859+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:03:07.473070+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.857445+0000","last_change":"2024-02-19T21:04:39.719113+0000","last_active":"2024-02-19T21:04:57.857445+0000","last_peered":"2024-02-19T21:04:57.857445+0000","last_clean":"2024-02-19T21:04:57.857445+0000","last_became_active":"2024-02-19T21:04:39.718926+0000","last_became_peered":"2024-02-19T21:04:39.718926+0000","last_unstale":"2024-02-19T21:04:57.857445+0000","last_undegraded":"2024-02-19T21:04:57.857445+0000","last_fullsized":"2024-02-19T21:04:57.857445+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"100'1","reported_seq":30,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.797521+0000","last_change":"2024-02-19T21:04:39.765598+0000","last_active":"2024-02-19T21:04:57.797521+0000","last_peered":"2024-02-19T21:04:57.797521+0000","last_clean":"2024-02-19T21:04:57.797521+0000","last_became_active":"2024-02-19T21:04:39.765165+0000","last_became_peered":"2024-02-19T21:04:39.765165+0000","last_unstale":"2024-02-19T21:04:57.797521+0000","last_undegraded":"2024-02-19T21:04:57.797521+0000","last_fullsized":"2024-02-19T21:04:57.797521+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"100'10","reported_seq":39,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.794095+0000","last_change":"2024-02-19T21:04:39.794115+0000","last_active":"2024-02-19T21:04:57.794095+0000","last_peered":"2024-02-19T21:04:57.794095+0000","last_clean":"2024-02-19T21:04:57.794095+0000","last_became_active":"2024-02-19T21:04:39.792916+0000","last_became_peered":"2024-02-19T21:04:39.792916+0000","last_unstale":"2024-02-19T21:04:57.794095+0000","last_undegraded":"2024-02-19T21:04:57.794095+0000","last_fullsized":"2024-02-19T21:04:57.794095+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.797258+0000","last_change":"2024-02-19T21:04:57.188742+0000","last_active":"2024-02-19T21:04:57.797258+0000","last_peered":"2024-02-19T21:04:57.797258+0000","last_clean":"2024-02-19T21:04:57.797258+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:04:57.797258+0000","last_undegraded":"2024-02-19T21:04:57.797258+0000","last_fullsized":"2024-02-19T21:04:57.797258+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:57.188664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:57.188664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":96,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.884775+0000","last_change":"2024-02-19T21:03:22.495828+0000","last_active":"2024-02-19T21:04:57.884775+0000","last_peered":"2024-02-19T21:04:57.884775+0000","last_clean":"2024-02-19T21:04:57.884775+0000","last_became_active":"2024-02-19T21:01:18.287139+0000","last_became_peered":"2024-02-19T21:01:18.287139+0000","last_unstale":"2024-02-19T21:04:57.884775+0000","last_undegraded":"2024-02-19T21:04:57.884775+0000","last_fullsized":"2024-02-19T21:04:57.884775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:03:22.495754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:22.495754+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":86,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.779078+0000","last_change":"2024-02-19T21:04:48.298552+0000","last_active":"2024-02-19T21:04:57.779078+0000","last_peered":"2024-02-19T21:04:57.779078+0000","last_clean":"2024-02-19T21:04:57.779078+0000","last_became_active":"2024-02-19T21:01:16.275904+0000","last_became_peered":"2024-02-19T21:01:16.275904+0000","last_unstale":"2024-02-19T21:04:57.779078+0000","last_undegraded":"2024-02-19T21:04:57.779078+0000","last_fullsized":"2024-02-19T21:04:57.779078+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:04:48.298487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:48.298487+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.836102+0000","last_change":"2024-02-19T21:02:57.505093+0000","last_active":"2024-02-19T21:04:57.836102+0000","last_peered":"2024-02-19T21:04:57.836102+0000","last_clean":"2024-02-19T21:04:57.836102+0000","last_became_active":"2024-02-19T21:01:16.274398+0000","last_became_peered":"2024-02-19T21:01:16.274398+0000","last_unstale":"2024-02-19T21:04:57.836102+0000","last_undegraded":"2024-02-19T21:04:57.836102+0000","last_fullsized":"2024-02-19T21:04:57.836102+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:02:57.505029+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.799092+0000","last_change":"2024-02-19T21:04:22.168700+0000","last_active":"2024-02-19T21:04:57.799092+0000","last_peered":"2024-02-19T21:04:57.799092+0000","last_clean":"2024-02-19T21:04:57.799092+0000","last_became_active":"2024-02-19T21:01:22.298477+0000","last_became_peered":"2024-02-19T21:01:22.298477+0000","last_unstale":"2024-02-19T21:04:57.799092+0000","last_undegraded":"2024-02-19T21:04:57.799092+0000","last_fullsized":"2024-02-19T21:04:57.799092+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:22.168596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:22.168596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.799777+0000","last_change":"2024-02-19T21:04:39.665830+0000","last_active":"2024-02-19T21:04:57.799777+0000","last_peered":"2024-02-19T21:04:57.799777+0000","last_clean":"2024-02-19T21:04:57.799777+0000","last_became_active":"2024-02-19T21:04:39.662125+0000","last_became_peered":"2024-02-19T21:04:39.662125+0000","last_unstale":"2024-02-19T21:04:57.799777+0000","last_undegraded":"2024-02-19T21:04:57.799777+0000","last_fullsized":"2024-02-19T21:04:57.799777+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.796028+0000","last_change":"2024-02-19T21:04:39.656608+0000","last_active":"2024-02-19T21:04:57.796028+0000","last_peered":"2024-02-19T21:04:57.796028+0000","last_clean":"2024-02-19T21:04:57.796028+0000","last_became_active":"2024-02-19T21:04:39.655971+0000","last_became_peered":"2024-02-19T21:04:39.655971+0000","last_unstale":"2024-02-19T21:04:57.796028+0000","last_undegraded":"2024-02-19T21:04:57.796028+0000","last_fullsized":"2024-02-19T21:04:57.796028+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.789533+0000","last_change":"2024-02-19T21:04:39.633236+0000","last_active":"2024-02-19T21:04:57.789533+0000","last_peered":"2024-02-19T21:04:57.789533+0000","last_clean":"2024-02-19T21:04:57.789533+0000","last_became_active":"2024-02-19T21:04:39.632936+0000","last_became_peered":"2024-02-19T21:04:39.632936+0000","last_unstale":"2024-02-19T21:04:57.789533+0000","last_undegraded":"2024-02-19T21:04:57.789533+0000","last_fullsized":"2024-02-19T21:04:57.789533+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.782463+0000","last_change":"2024-02-19T21:04:19.858793+0000","last_active":"2024-02-19T21:04:57.782463+0000","last_peered":"2024-02-19T21:04:57.782463+0000","last_clean":"2024-02-19T21:04:57.782463+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:04:57.782463+0000","last_undegraded":"2024-02-19T21:04:57.782463+0000","last_fullsized":"2024-02-19T21:04:57.782463+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:19.858688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.858688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":100,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.801022+0000","last_change":"2024-02-19T21:04:19.885862+0000","last_active":"2024-02-19T21:04:57.801022+0000","last_peered":"2024-02-19T21:04:57.801022+0000","last_clean":"2024-02-19T21:04:57.801022+0000","last_became_active":"2024-02-19T21:01:18.293784+0000","last_became_peered":"2024-02-19T21:01:18.293784+0000","last_unstale":"2024-02-19T21:04:57.801022+0000","last_undegraded":"2024-02-19T21:04:57.801022+0000","last_fullsized":"2024-02-19T21:04:57.801022+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:04:19.885797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.885797+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":83,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.779169+0000","last_change":"2024-02-19T21:03:11.598713+0000","last_active":"2024-02-19T21:04:57.779169+0000","last_peered":"2024-02-19T21:04:57.779169+0000","last_clean":"2024-02-19T21:04:57.779169+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:04:57.779169+0000","last_undegraded":"2024-02-19T21:04:57.779169+0000","last_fullsized":"2024-02-19T21:04:57.779169+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:11.598603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:03:11.598603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.793569+0000","last_change":"2024-02-19T21:04:14.215108+0000","last_active":"2024-02-19T21:04:57.793569+0000","last_peered":"2024-02-19T21:04:57.793569+0000","last_clean":"2024-02-19T21:04:57.793569+0000","last_became_active":"2024-02-19T21:01:16.276807+0000","last_became_peered":"2024-02-19T21:01:16.276807+0000","last_unstale":"2024-02-19T21:04:57.793569+0000","last_undegraded":"2024-02-19T21:04:57.793569+0000","last_fullsized":"2024-02-19T21:04:57.793569+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:14.215016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:14.215016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.854622+0000","last_change":"2024-02-19T21:04:54.972925+0000","last_active":"2024-02-19T21:04:57.854622+0000","last_peered":"2024-02-19T21:04:57.854622+0000","last_clean":"2024-02-19T21:04:57.854622+0000","last_became_active":"2024-02-19T21:01:16.274290+0000","last_became_peered":"2024-02-19T21:01:16.274290+0000","last_unstale":"2024-02-19T21:04:57.854622+0000","last_undegraded":"2024-02-19T21:04:57.854622+0000","last_fullsized":"2024-02-19T21:04:57.854622+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:54.972832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:54.972832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.796961+0000","last_change":"2024-02-19T21:04:05.136254+0000","last_active":"2024-02-19T21:04:57.796961+0000","last_peered":"2024-02-19T21:04:57.796961+0000","last_clean":"2024-02-19T21:04:57.796961+0000","last_became_active":"2024-02-19T21:01:22.292299+0000","last_became_peered":"2024-02-19T21:01:22.292299+0000","last_unstale":"2024-02-19T21:04:57.796961+0000","last_undegraded":"2024-02-19T21:04:57.796961+0000","last_fullsized":"2024-02-19T21:04:57.796961+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:05.136186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:05.136186+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.791222+0000","last_change":"2024-02-19T21:04:39.667403+0000","last_active":"2024-02-19T21:04:57.791222+0000","last_peered":"2024-02-19T21:04:57.791222+0000","last_clean":"2024-02-19T21:04:57.791222+0000","last_became_active":"2024-02-19T21:04:39.666739+0000","last_became_peered":"2024-02-19T21:04:39.666739+0000","last_unstale":"2024-02-19T21:04:57.791222+0000","last_undegraded":"2024-02-19T21:04:57.791222+0000","last_fullsized":"2024-02-19T21:04:57.791222+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1f","version":"100'6","reported_seq":34,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.886893+0000","last_change":"2024-02-19T21:04:39.743491+0000","last_active":"2024-02-19T21:04:57.886893+0000","last_peered":"2024-02-19T21:04:57.886893+0000","last_clean":"2024-02-19T21:04:57.886893+0000","last_became_active":"2024-02-19T21:04:39.742131+0000","last_became_peered":"2024-02-19T21:04:39.742131+0000","last_unstale":"2024-02-19T21:04:57.886893+0000","last_undegraded":"2024-02-19T21:04:57.886893+0000","last_fullsized":"2024-02-19T21:04:57.886893+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"100'2","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.863024+0000","last_change":"2024-02-19T21:04:39.723860+0000","last_active":"2024-02-19T21:04:57.863024+0000","last_peered":"2024-02-19T21:04:57.863024+0000","last_clean":"2024-02-19T21:04:57.863024+0000","last_became_active":"2024-02-19T21:04:39.715390+0000","last_became_peered":"2024-02-19T21:04:39.715390+0000","last_unstale":"2024-02-19T21:04:57.863024+0000","last_undegraded":"2024-02-19T21:04:57.863024+0000","last_fullsized":"2024-02-19T21:04:57.863024+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.861470+0000","last_change":"2024-02-19T21:04:15.868824+0000","last_active":"2024-02-19T21:04:57.861470+0000","last_peered":"2024-02-19T21:04:57.861470+0000","last_clean":"2024-02-19T21:04:57.861470+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:04:57.861470+0000","last_undegraded":"2024-02-19T21:04:57.861470+0000","last_fullsized":"2024-02-19T21:04:57.861470+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:15.868726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:15.868726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.857316+0000","last_change":"2024-02-19T21:04:56.846706+0000","last_active":"2024-02-19T21:04:57.857316+0000","last_peered":"2024-02-19T21:04:57.857316+0000","last_clean":"2024-02-19T21:04:57.857316+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:04:57.857316+0000","last_undegraded":"2024-02-19T21:04:57.857316+0000","last_fullsized":"2024-02-19T21:04:57.857316+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:04:56.846622+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.846622+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":85,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.805596+0000","last_change":"2024-02-19T21:04:52.793040+0000","last_active":"2024-02-19T21:04:57.805596+0000","last_peered":"2024-02-19T21:04:57.805596+0000","last_clean":"2024-02-19T21:04:57.805596+0000","last_became_active":"2024-02-19T21:01:16.276032+0000","last_became_peered":"2024-02-19T21:01:16.276032+0000","last_unstale":"2024-02-19T21:04:57.805596+0000","last_undegraded":"2024-02-19T21:04:57.805596+0000","last_fullsized":"2024-02-19T21:04:57.805596+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:52.792935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:52.792935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":299,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-19T21:04:58.786438+0000","last_change":"2024-02-19T21:04:20.241440+0000","last_active":"2024-02-19T21:04:58.786438+0000","last_peered":"2024-02-19T21:04:58.786438+0000","last_clean":"2024-02-19T21:04:58.786438+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:04:58.786438+0000","last_undegraded":"2024-02-19T21:04:58.786438+0000","last_fullsized":"2024-02-19T21:04:58.786438+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:04:20.241327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:20.241327+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":216,"num_read_kb":211,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":73,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.784307+0000","last_change":"2024-02-19T21:04:11.900738+0000","last_active":"2024-02-19T21:04:57.784307+0000","last_peered":"2024-02-19T21:04:57.784307+0000","last_clean":"2024-02-19T21:04:57.784307+0000","last_became_active":"2024-02-19T21:01:22.607883+0000","last_became_peered":"2024-02-19T21:01:22.607883+0000","last_unstale":"2024-02-19T21:04:57.784307+0000","last_undegraded":"2024-02-19T21:04:57.784307+0000","last_fullsized":"2024-02-19T21:04:57.784307+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.900625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.900625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.802538+0000","last_change":"2024-02-19T21:04:39.692210+0000","last_active":"2024-02-19T21:04:57.802538+0000","last_peered":"2024-02-19T21:04:57.802538+0000","last_clean":"2024-02-19T21:04:57.802538+0000","last_became_active":"2024-02-19T21:04:39.691434+0000","last_became_peered":"2024-02-19T21:04:39.691434+0000","last_unstale":"2024-02-19T21:04:57.802538+0000","last_undegraded":"2024-02-19T21:04:57.802538+0000","last_fullsized":"2024-02-19T21:04:57.802538+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.789994+0000","last_change":"2024-02-19T21:04:39.701428+0000","last_active":"2024-02-19T21:04:57.789994+0000","last_peered":"2024-02-19T21:04:57.789994+0000","last_clean":"2024-02-19T21:04:57.789994+0000","last_became_active":"2024-02-19T21:04:39.698760+0000","last_became_peered":"2024-02-19T21:04:39.698760+0000","last_unstale":"2024-02-19T21:04:57.789994+0000","last_undegraded":"2024-02-19T21:04:57.789994+0000","last_fullsized":"2024-02-19T21:04:57.789994+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.8","version":"100'2","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.788844+0000","last_change":"2024-02-19T21:04:39.773441+0000","last_active":"2024-02-19T21:04:57.788844+0000","last_peered":"2024-02-19T21:04:57.788844+0000","last_clean":"2024-02-19T21:04:57.788844+0000","last_became_active":"2024-02-19T21:04:39.771376+0000","last_became_peered":"2024-02-19T21:04:39.771376+0000","last_unstale":"2024-02-19T21:04:57.788844+0000","last_undegraded":"2024-02-19T21:04:57.788844+0000","last_fullsized":"2024-02-19T21:04:57.788844+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":75,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-19T21:04:59.286495+0000","last_change":"2024-02-19T21:04:59.287433+0000","last_active":"2024-02-19T21:04:59.286495+0000","last_peered":"2024-02-19T21:04:59.286495+0000","last_clean":"2024-02-19T21:04:59.286495+0000","last_became_active":"2024-02-19T21:01:20.290758+0000","last_became_peered":"2024-02-19T21:01:20.290758+0000","last_unstale":"2024-02-19T21:04:59.286495+0000","last_undegraded":"2024-02-19T21:04:59.286495+0000","last_fullsized":"2024-02-19T21:04:59.286495+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.510474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":171,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-19T21:04:58.795309+0000","last_change":"2024-02-19T21:04:06.383697+0000","last_active":"2024-02-19T21:04:58.795309+0000","last_peered":"2024-02-19T21:04:58.795309+0000","last_clean":"2024-02-19T21:04:58.795309+0000","last_became_active":"2024-02-19T21:01:18.289076+0000","last_became_peered":"2024-02-19T21:01:18.289076+0000","last_unstale":"2024-02-19T21:04:58.795309+0000","last_undegraded":"2024-02-19T21:04:58.795309+0000","last_fullsized":"2024-02-19T21:04:58.795309+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-19T21:04:06.383559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.383559+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.791552+0000","last_change":"2024-02-19T21:02:58.986556+0000","last_active":"2024-02-19T21:04:57.791552+0000","last_peered":"2024-02-19T21:04:57.791552+0000","last_clean":"2024-02-19T21:04:57.791552+0000","last_became_active":"2024-02-19T21:01:16.270159+0000","last_became_peered":"2024-02-19T21:01:16.270159+0000","last_unstale":"2024-02-19T21:04:57.791552+0000","last_undegraded":"2024-02-19T21:04:57.791552+0000","last_fullsized":"2024-02-19T21:04:57.791552+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:02:58.986458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:02:58.986458+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.870491+0000","last_change":"2024-02-19T21:04:49.004211+0000","last_active":"2024-02-19T21:04:57.870491+0000","last_peered":"2024-02-19T21:04:57.870491+0000","last_clean":"2024-02-19T21:04:57.870491+0000","last_became_active":"2024-02-19T21:01:22.291396+0000","last_became_peered":"2024-02-19T21:01:22.291396+0000","last_unstale":"2024-02-19T21:04:57.870491+0000","last_undegraded":"2024-02-19T21:04:57.870491+0000","last_fullsized":"2024-02-19T21:04:57.870491+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.004148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.004148+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.836605+0000","last_change":"2024-02-19T21:04:39.721663+0000","last_active":"2024-02-19T21:04:57.836605+0000","last_peered":"2024-02-19T21:04:57.836605+0000","last_clean":"2024-02-19T21:04:57.836605+0000","last_became_active":"2024-02-19T21:04:39.721416+0000","last_became_peered":"2024-02-19T21:04:39.721416+0000","last_unstale":"2024-02-19T21:04:57.836605+0000","last_undegraded":"2024-02-19T21:04:57.836605+0000","last_fullsized":"2024-02-19T21:04:57.836605+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.800298+0000","last_change":"2024-02-19T21:04:39.678110+0000","last_active":"2024-02-19T21:04:57.800298+0000","last_peered":"2024-02-19T21:04:57.800298+0000","last_clean":"2024-02-19T21:04:57.800298+0000","last_became_active":"2024-02-19T21:04:39.674697+0000","last_became_peered":"2024-02-19T21:04:39.674697+0000","last_unstale":"2024-02-19T21:04:57.800298+0000","last_undegraded":"2024-02-19T21:04:57.800298+0000","last_fullsized":"2024-02-19T21:04:57.800298+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.884443+0000","last_change":"2024-02-19T21:04:39.674303+0000","last_active":"2024-02-19T21:04:57.884443+0000","last_peered":"2024-02-19T21:04:57.884443+0000","last_clean":"2024-02-19T21:04:57.884443+0000","last_became_active":"2024-02-19T21:04:39.674039+0000","last_became_peered":"2024-02-19T21:04:39.674039+0000","last_unstale":"2024-02-19T21:04:57.884443+0000","last_undegraded":"2024-02-19T21:04:57.884443+0000","last_fullsized":"2024-02-19T21:04:57.884443+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":132,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-19T21:04:58.844913+0000","last_change":"2024-02-19T21:04:50.353975+0000","last_active":"2024-02-19T21:04:58.844913+0000","last_peered":"2024-02-19T21:04:58.844913+0000","last_clean":"2024-02-19T21:04:58.844913+0000","last_became_active":"2024-02-19T21:01:20.288918+0000","last_became_peered":"2024-02-19T21:01:20.288918+0000","last_unstale":"2024-02-19T21:04:58.844913+0000","last_undegraded":"2024-02-19T21:04:58.844913+0000","last_fullsized":"2024-02-19T21:04:58.844913+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:50.353869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:50.353869+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":88,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.794532+0000","last_change":"2024-02-19T21:02:56.539408+0000","last_active":"2024-02-19T21:04:57.794532+0000","last_peered":"2024-02-19T21:04:57.794532+0000","last_clean":"2024-02-19T21:04:57.794532+0000","last_became_active":"2024-02-19T21:01:18.285891+0000","last_became_peered":"2024-02-19T21:01:18.285891+0000","last_unstale":"2024-02-19T21:04:57.794532+0000","last_undegraded":"2024-02-19T21:04:57.794532+0000","last_fullsized":"2024-02-19T21:04:57.794532+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:02:56.539303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:02:56.539303+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":89,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.798661+0000","last_change":"2024-02-19T21:04:18.242620+0000","last_active":"2024-02-19T21:04:57.798661+0000","last_peered":"2024-02-19T21:04:57.798661+0000","last_clean":"2024-02-19T21:04:57.798661+0000","last_became_active":"2024-02-19T21:01:30.926676+0000","last_became_peered":"2024-02-19T21:01:30.926676+0000","last_unstale":"2024-02-19T21:04:57.798661+0000","last_undegraded":"2024-02-19T21:04:57.798661+0000","last_fullsized":"2024-02-19T21:04:57.798661+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:18.242556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:04:18.242556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.831337+0000","last_change":"2024-02-19T21:04:06.599565+0000","last_active":"2024-02-19T21:04:57.831337+0000","last_peered":"2024-02-19T21:04:57.831337+0000","last_clean":"2024-02-19T21:04:57.831337+0000","last_became_active":"2024-02-19T21:01:22.292133+0000","last_became_peered":"2024-02-19T21:01:22.292133+0000","last_unstale":"2024-02-19T21:04:57.831337+0000","last_undegraded":"2024-02-19T21:04:57.831337+0000","last_fullsized":"2024-02-19T21:04:57.831337+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:06.599439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.599439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.866623+0000","last_change":"2024-02-19T21:04:39.767350+0000","last_active":"2024-02-19T21:04:57.866623+0000","last_peered":"2024-02-19T21:04:57.866623+0000","last_clean":"2024-02-19T21:04:57.866623+0000","last_became_active":"2024-02-19T21:04:39.766156+0000","last_became_peered":"2024-02-19T21:04:39.766156+0000","last_unstale":"2024-02-19T21:04:57.866623+0000","last_undegraded":"2024-02-19T21:04:57.866623+0000","last_fullsized":"2024-02-19T21:04:57.866623+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.11","version":"100'1","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.811952+0000","last_change":"2024-02-19T21:04:39.781019+0000","last_active":"2024-02-19T21:04:57.811952+0000","last_peered":"2024-02-19T21:04:57.811952+0000","last_clean":"2024-02-19T21:04:57.811952+0000","last_became_active":"2024-02-19T21:04:39.776326+0000","last_became_peered":"2024-02-19T21:04:39.776326+0000","last_unstale":"2024-02-19T21:04:57.811952+0000","last_undegraded":"2024-02-19T21:04:57.811952+0000","last_fullsized":"2024-02-19T21:04:57.811952+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.801375+0000","last_change":"2024-02-19T21:04:39.656217+0000","last_active":"2024-02-19T21:04:57.801375+0000","last_peered":"2024-02-19T21:04:57.801375+0000","last_clean":"2024-02-19T21:04:57.801375+0000","last_became_active":"2024-02-19T21:04:39.654865+0000","last_became_peered":"2024-02-19T21:04:39.654865+0000","last_unstale":"2024-02-19T21:04:57.801375+0000","last_undegraded":"2024-02-19T21:04:57.801375+0000","last_fullsized":"2024-02-19T21:04:57.801375+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.836417+0000","last_change":"2024-02-19T21:04:16.866220+0000","last_active":"2024-02-19T21:04:57.836417+0000","last_peered":"2024-02-19T21:04:57.836417+0000","last_clean":"2024-02-19T21:04:57.836417+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:04:57.836417+0000","last_undegraded":"2024-02-19T21:04:57.836417+0000","last_fullsized":"2024-02-19T21:04:57.836417+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.866116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":101,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-19T21:04:59.357001+0000","last_change":"2024-02-19T21:04:59.357001+0000","last_active":"2024-02-19T21:04:59.357001+0000","last_peered":"2024-02-19T21:04:59.357001+0000","last_clean":"2024-02-19T21:04:59.357001+0000","last_became_active":"2024-02-19T21:01:18.285672+0000","last_became_peered":"2024-02-19T21:01:18.285672+0000","last_unstale":"2024-02-19T21:04:59.357001+0000","last_undegraded":"2024-02-19T21:04:59.357001+0000","last_fullsized":"2024-02-19T21:04:59.357001+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:04:59.356922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:59.356922+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":95,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.795154+0000","last_change":"2024-02-19T21:03:21.164366+0000","last_active":"2024-02-19T21:04:57.795154+0000","last_peered":"2024-02-19T21:04:57.795154+0000","last_clean":"2024-02-19T21:04:57.795154+0000","last_became_active":"2024-02-19T21:01:16.273545+0000","last_became_peered":"2024-02-19T21:01:16.273545+0000","last_unstale":"2024-02-19T21:04:57.795154+0000","last_undegraded":"2024-02-19T21:04:57.795154+0000","last_fullsized":"2024-02-19T21:04:57.795154+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:03:21.164286+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":69,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.863244+0000","last_change":"2024-02-19T21:03:17.625775+0000","last_active":"2024-02-19T21:04:57.863244+0000","last_peered":"2024-02-19T21:04:57.863244+0000","last_clean":"2024-02-19T21:04:57.863244+0000","last_became_active":"2024-02-19T21:01:22.295912+0000","last_became_peered":"2024-02-19T21:01:22.295912+0000","last_unstale":"2024-02-19T21:04:57.863244+0000","last_undegraded":"2024-02-19T21:04:57.863244+0000","last_fullsized":"2024-02-19T21:04:57.863244+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:03:17.625711+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.853036+0000","last_change":"2024-02-19T21:04:39.696886+0000","last_active":"2024-02-19T21:04:57.853036+0000","last_peered":"2024-02-19T21:04:57.853036+0000","last_clean":"2024-02-19T21:04:57.853036+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:04:57.853036+0000","last_undegraded":"2024-02-19T21:04:57.853036+0000","last_fullsized":"2024-02-19T21:04:57.853036+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.884938+0000","last_change":"2024-02-19T21:04:39.741185+0000","last_active":"2024-02-19T21:04:57.884938+0000","last_peered":"2024-02-19T21:04:57.884938+0000","last_clean":"2024-02-19T21:04:57.884938+0000","last_became_active":"2024-02-19T21:04:39.740396+0000","last_became_peered":"2024-02-19T21:04:39.740396+0000","last_unstale":"2024-02-19T21:04:57.884938+0000","last_undegraded":"2024-02-19T21:04:57.884938+0000","last_fullsized":"2024-02-19T21:04:57.884938+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"100'5","reported_seq":34,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.844283+0000","last_change":"2024-02-19T21:04:39.736543+0000","last_active":"2024-02-19T21:04:57.844283+0000","last_peered":"2024-02-19T21:04:57.844283+0000","last_clean":"2024-02-19T21:04:57.844283+0000","last_became_active":"2024-02-19T21:04:39.731233+0000","last_became_peered":"2024-02-19T21:04:39.731233+0000","last_unstale":"2024-02-19T21:04:57.844283+0000","last_undegraded":"2024-02-19T21:04:57.844283+0000","last_fullsized":"2024-02-19T21:04:57.844283+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.884859+0000","last_change":"2024-02-19T21:04:09.571081+0000","last_active":"2024-02-19T21:04:57.884859+0000","last_peered":"2024-02-19T21:04:57.884859+0000","last_clean":"2024-02-19T21:04:57.884859+0000","last_became_active":"2024-02-19T21:01:18.293773+0000","last_became_peered":"2024-02-19T21:01:18.293773+0000","last_unstale":"2024-02-19T21:04:57.884859+0000","last_undegraded":"2024-02-19T21:04:57.884859+0000","last_fullsized":"2024-02-19T21:04:57.884859+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:09.570972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:09.570972+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.836343+0000","last_change":"2024-02-19T21:04:49.819984+0000","last_active":"2024-02-19T21:04:57.836343+0000","last_peered":"2024-02-19T21:04:57.836343+0000","last_clean":"2024-02-19T21:04:57.836343+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:04:57.836343+0000","last_undegraded":"2024-02-19T21:04:57.836343+0000","last_fullsized":"2024-02-19T21:04:57.836343+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.819921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.819921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.870140+0000","last_change":"2024-02-19T21:04:11.841348+0000","last_active":"2024-02-19T21:04:57.870140+0000","last_peered":"2024-02-19T21:04:57.870140+0000","last_clean":"2024-02-19T21:04:57.870140+0000","last_became_active":"2024-02-19T21:01:20.285298+0000","last_became_peered":"2024-02-19T21:01:20.285298+0000","last_unstale":"2024-02-19T21:04:57.870140+0000","last_undegraded":"2024-02-19T21:04:57.870140+0000","last_fullsized":"2024-02-19T21:04:57.870140+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.841184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.841184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.863573+0000","last_change":"2024-02-19T21:04:08.831108+0000","last_active":"2024-02-19T21:04:57.863573+0000","last_peered":"2024-02-19T21:04:57.863573+0000","last_clean":"2024-02-19T21:04:57.863573+0000","last_became_active":"2024-02-19T21:01:22.297086+0000","last_became_peered":"2024-02-19T21:01:22.297086+0000","last_unstale":"2024-02-19T21:04:57.863573+0000","last_undegraded":"2024-02-19T21:04:57.863573+0000","last_fullsized":"2024-02-19T21:04:57.863573+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.830994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.830994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.781260+0000","last_change":"2024-02-19T21:04:39.639826+0000","last_active":"2024-02-19T21:04:57.781260+0000","last_peered":"2024-02-19T21:04:57.781260+0000","last_clean":"2024-02-19T21:04:57.781260+0000","last_became_active":"2024-02-19T21:04:39.639518+0000","last_became_peered":"2024-02-19T21:04:39.639518+0000","last_unstale":"2024-02-19T21:04:57.781260+0000","last_undegraded":"2024-02-19T21:04:57.781260+0000","last_fullsized":"2024-02-19T21:04:57.781260+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.801738+0000","last_change":"2024-02-19T21:04:39.679134+0000","last_active":"2024-02-19T21:04:57.801738+0000","last_peered":"2024-02-19T21:04:57.801738+0000","last_clean":"2024-02-19T21:04:57.801738+0000","last_became_active":"2024-02-19T21:04:39.678877+0000","last_became_peered":"2024-02-19T21:04:39.678877+0000","last_unstale":"2024-02-19T21:04:57.801738+0000","last_undegraded":"2024-02-19T21:04:57.801738+0000","last_fullsized":"2024-02-19T21:04:57.801738+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1b","version":"100'10","reported_seq":39,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.803489+0000","last_change":"2024-02-19T21:04:39.642209+0000","last_active":"2024-02-19T21:04:57.803489+0000","last_peered":"2024-02-19T21:04:57.803489+0000","last_clean":"2024-02-19T21:04:57.803489+0000","last_became_active":"2024-02-19T21:04:39.638760+0000","last_became_peered":"2024-02-19T21:04:39.638760+0000","last_unstale":"2024-02-19T21:04:57.803489+0000","last_undegraded":"2024-02-19T21:04:57.803489+0000","last_fullsized":"2024-02-19T21:04:57.803489+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":90,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.870100+0000","last_change":"2024-02-19T21:04:06.807310+0000","last_active":"2024-02-19T21:04:57.870100+0000","last_peered":"2024-02-19T21:04:57.870100+0000","last_clean":"2024-02-19T21:04:57.870100+0000","last_became_active":"2024-02-19T21:01:18.293913+0000","last_became_peered":"2024-02-19T21:01:18.293913+0000","last_unstale":"2024-02-19T21:04:57.870100+0000","last_undegraded":"2024-02-19T21:04:57.870100+0000","last_fullsized":"2024-02-19T21:04:57.870100+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:04:06.807190+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.807190+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.789764+0000","last_change":"2024-02-19T21:04:08.757186+0000","last_active":"2024-02-19T21:04:57.789764+0000","last_peered":"2024-02-19T21:04:57.789764+0000","last_clean":"2024-02-19T21:04:57.789764+0000","last_became_active":"2024-02-19T21:01:16.271210+0000","last_became_peered":"2024-02-19T21:01:16.271210+0000","last_unstale":"2024-02-19T21:04:57.789764+0000","last_undegraded":"2024-02-19T21:04:57.789764+0000","last_fullsized":"2024-02-19T21:04:57.789764+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.757098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.757098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":71,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.798324+0000","last_change":"2024-02-19T21:03:08.984824+0000","last_active":"2024-02-19T21:04:57.798324+0000","last_peered":"2024-02-19T21:04:57.798324+0000","last_clean":"2024-02-19T21:04:57.798324+0000","last_became_active":"2024-02-19T21:01:20.288589+0000","last_became_peered":"2024-02-19T21:01:20.288589+0000","last_unstale":"2024-02-19T21:04:57.798324+0000","last_undegraded":"2024-02-19T21:04:57.798324+0000","last_fullsized":"2024-02-19T21:04:57.798324+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:08.984720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:08.984720+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":73,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.783635+0000","last_change":"2024-02-19T21:04:06.880972+0000","last_active":"2024-02-19T21:04:57.783635+0000","last_peered":"2024-02-19T21:04:57.783635+0000","last_clean":"2024-02-19T21:04:57.783635+0000","last_became_active":"2024-02-19T21:01:22.608285+0000","last_became_peered":"2024-02-19T21:01:22.608285+0000","last_unstale":"2024-02-19T21:04:57.783635+0000","last_undegraded":"2024-02-19T21:04:57.783635+0000","last_fullsized":"2024-02-19T21:04:57.783635+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:06.880899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.880899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.793321+0000","last_change":"2024-02-19T21:04:39.641073+0000","last_active":"2024-02-19T21:04:57.793321+0000","last_peered":"2024-02-19T21:04:57.793321+0000","last_clean":"2024-02-19T21:04:57.793321+0000","last_became_active":"2024-02-19T21:04:39.639775+0000","last_became_peered":"2024-02-19T21:04:39.639775+0000","last_unstale":"2024-02-19T21:04:57.793321+0000","last_undegraded":"2024-02-19T21:04:57.793321+0000","last_fullsized":"2024-02-19T21:04:57.793321+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.c","version":"100'1","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.871448+0000","last_change":"2024-02-19T21:04:39.753081+0000","last_active":"2024-02-19T21:04:57.871448+0000","last_peered":"2024-02-19T21:04:57.871448+0000","last_clean":"2024-02-19T21:04:57.871448+0000","last_became_active":"2024-02-19T21:04:39.752718+0000","last_became_peered":"2024-02-19T21:04:39.752718+0000","last_unstale":"2024-02-19T21:04:57.871448+0000","last_undegraded":"2024-02-19T21:04:57.871448+0000","last_fullsized":"2024-02-19T21:04:57.871448+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"100'2","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.790354+0000","last_change":"2024-02-19T21:04:39.635873+0000","last_active":"2024-02-19T21:04:57.790354+0000","last_peered":"2024-02-19T21:04:57.790354+0000","last_clean":"2024-02-19T21:04:57.790354+0000","last_became_active":"2024-02-19T21:04:39.631391+0000","last_became_peered":"2024-02-19T21:04:39.631391+0000","last_unstale":"2024-02-19T21:04:57.790354+0000","last_undegraded":"2024-02-19T21:04:57.790354+0000","last_fullsized":"2024-02-19T21:04:57.790354+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":93,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.871366+0000","last_change":"2024-02-19T21:04:22.763995+0000","last_active":"2024-02-19T21:04:57.871366+0000","last_peered":"2024-02-19T21:04:57.871366+0000","last_clean":"2024-02-19T21:04:57.871366+0000","last_became_active":"2024-02-19T21:01:18.294369+0000","last_became_peered":"2024-02-19T21:01:18.294369+0000","last_unstale":"2024-02-19T21:04:57.871366+0000","last_undegraded":"2024-02-19T21:04:57.871366+0000","last_fullsized":"2024-02-19T21:04:57.871366+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:04:22.763926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:22.763926+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.791472+0000","last_change":"2024-02-19T21:04:09.180232+0000","last_active":"2024-02-19T21:04:57.791472+0000","last_peered":"2024-02-19T21:04:57.791472+0000","last_clean":"2024-02-19T21:04:57.791472+0000","last_became_active":"2024-02-19T21:01:16.271608+0000","last_became_peered":"2024-02-19T21:01:16.271608+0000","last_unstale":"2024-02-19T21:04:57.791472+0000","last_undegraded":"2024-02-19T21:04:57.791472+0000","last_fullsized":"2024-02-19T21:04:57.791472+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:09.180159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:09.180159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-19T21:04:58.790986+0000","last_change":"2024-02-19T21:03:11.063302+0000","last_active":"2024-02-19T21:04:58.790986+0000","last_peered":"2024-02-19T21:04:58.790986+0000","last_clean":"2024-02-19T21:04:58.790986+0000","last_became_active":"2024-02-19T21:01:20.288799+0000","last_became_peered":"2024-02-19T21:01:20.288799+0000","last_unstale":"2024-02-19T21:04:58.790986+0000","last_undegraded":"2024-02-19T21:04:58.790986+0000","last_fullsized":"2024-02-19T21:04:58.790986+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:11.063222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:11.063222+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.884688+0000","last_change":"2024-02-19T21:04:05.558879+0000","last_active":"2024-02-19T21:04:57.884688+0000","last_peered":"2024-02-19T21:04:57.884688+0000","last_clean":"2024-02-19T21:04:57.884688+0000","last_became_active":"2024-02-19T21:01:22.297188+0000","last_became_peered":"2024-02-19T21:01:22.297188+0000","last_unstale":"2024-02-19T21:04:57.884688+0000","last_undegraded":"2024-02-19T21:04:57.884688+0000","last_fullsized":"2024-02-19T21:04:57.884688+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:04:05.558765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:05.558765+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.831949+0000","last_change":"2024-02-19T21:04:39.688410+0000","last_active":"2024-02-19T21:04:57.831949+0000","last_peered":"2024-02-19T21:04:57.831949+0000","last_clean":"2024-02-19T21:04:57.831949+0000","last_became_active":"2024-02-19T21:04:39.687668+0000","last_became_peered":"2024-02-19T21:04:39.687668+0000","last_unstale":"2024-02-19T21:04:57.831949+0000","last_undegraded":"2024-02-19T21:04:57.831949+0000","last_fullsized":"2024-02-19T21:04:57.831949+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.784221+0000","last_change":"2024-02-19T21:04:39.749884+0000","last_active":"2024-02-19T21:04:57.784221+0000","last_peered":"2024-02-19T21:04:57.784221+0000","last_clean":"2024-02-19T21:04:57.784221+0000","last_became_active":"2024-02-19T21:04:39.748211+0000","last_became_peered":"2024-02-19T21:04:39.748211+0000","last_unstale":"2024-02-19T21:04:57.784221+0000","last_undegraded":"2024-02-19T21:04:57.784221+0000","last_fullsized":"2024-02-19T21:04:57.784221+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.791590+0000","last_change":"2024-02-19T21:04:39.667189+0000","last_active":"2024-02-19T21:04:57.791590+0000","last_peered":"2024-02-19T21:04:57.791590+0000","last_clean":"2024-02-19T21:04:57.791590+0000","last_became_active":"2024-02-19T21:04:39.666360+0000","last_became_peered":"2024-02-19T21:04:39.666360+0000","last_unstale":"2024-02-19T21:04:57.791590+0000","last_undegraded":"2024-02-19T21:04:57.791590+0000","last_fullsized":"2024-02-19T21:04:57.791590+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.795645+0000","last_change":"2024-02-19T21:04:15.187687+0000","last_active":"2024-02-19T21:04:57.795645+0000","last_peered":"2024-02-19T21:04:57.795645+0000","last_clean":"2024-02-19T21:04:57.795645+0000","last_became_active":"2024-02-19T21:01:18.293786+0000","last_became_peered":"2024-02-19T21:01:18.293786+0000","last_unstale":"2024-02-19T21:04:57.795645+0000","last_undegraded":"2024-02-19T21:04:57.795645+0000","last_fullsized":"2024-02-19T21:04:57.795645+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:15.187587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:15.187587+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.843806+0000","last_change":"2024-02-19T21:03:39.467124+0000","last_active":"2024-02-19T21:04:57.843806+0000","last_peered":"2024-02-19T21:04:57.843806+0000","last_clean":"2024-02-19T21:04:57.843806+0000","last_became_active":"2024-02-19T21:01:16.273181+0000","last_became_peered":"2024-02-19T21:01:16.273181+0000","last_unstale":"2024-02-19T21:04:57.843806+0000","last_undegraded":"2024-02-19T21:04:57.843806+0000","last_fullsized":"2024-02-19T21:04:57.843806+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:39.467059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:39.467059+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-19T21:04:58.870634+0000","last_change":"2024-02-19T21:03:26.669863+0000","last_active":"2024-02-19T21:04:58.870634+0000","last_peered":"2024-02-19T21:04:58.870634+0000","last_clean":"2024-02-19T21:04:58.870634+0000","last_became_active":"2024-02-19T21:01:20.283943+0000","last_became_peered":"2024-02-19T21:01:20.283943+0000","last_unstale":"2024-02-19T21:04:58.870634+0000","last_undegraded":"2024-02-19T21:04:58.870634+0000","last_fullsized":"2024-02-19T21:04:58.870634+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:26.669794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:26.669794+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.795352+0000","last_change":"2024-02-19T21:04:15.099340+0000","last_active":"2024-02-19T21:04:57.795352+0000","last_peered":"2024-02-19T21:04:57.795352+0000","last_clean":"2024-02-19T21:04:57.795352+0000","last_became_active":"2024-02-19T21:01:22.299969+0000","last_became_peered":"2024-02-19T21:01:22.299969+0000","last_unstale":"2024-02-19T21:04:57.795352+0000","last_undegraded":"2024-02-19T21:04:57.795352+0000","last_fullsized":"2024-02-19T21:04:57.795352+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:15.099208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_clean_scrub_stamp":"2024-02-19T21:04:15.099208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":41,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.805414+0000","last_change":"2024-02-19T21:04:39.767396+0000","last_active":"2024-02-19T21:04:57.805414+0000","last_peered":"2024-02-19T21:04:57.805414+0000","last_clean":"2024-02-19T21:04:57.805414+0000","last_became_active":"2024-02-19T21:04:39.760167+0000","last_became_peered":"2024-02-19T21:04:39.760167+0000","last_unstale":"2024-02-19T21:04:57.805414+0000","last_undegraded":"2024-02-19T21:04:57.805414+0000","last_fullsized":"2024-02-19T21:04:57.805414+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.a","version":"102'5","reported_seq":37,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.791248+0000","last_change":"2024-02-19T21:04:39.751144+0000","last_active":"2024-02-19T21:04:57.791248+0000","last_peered":"2024-02-19T21:04:57.791248+0000","last_clean":"2024-02-19T21:04:57.791248+0000","last_became_active":"2024-02-19T21:04:39.750165+0000","last_became_peered":"2024-02-19T21:04:39.750165+0000","last_unstale":"2024-02-19T21:04:57.791248+0000","last_undegraded":"2024-02-19T21:04:57.791248+0000","last_fullsized":"2024-02-19T21:04:57.791248+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165843,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":2,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.866695+0000","last_change":"2024-02-19T21:04:39.704782+0000","last_active":"2024-02-19T21:04:57.866695+0000","last_peered":"2024-02-19T21:04:57.866695+0000","last_clean":"2024-02-19T21:04:57.866695+0000","last_became_active":"2024-02-19T21:04:39.704532+0000","last_became_peered":"2024-02-19T21:04:39.704532+0000","last_unstale":"2024-02-19T21:04:57.866695+0000","last_undegraded":"2024-02-19T21:04:57.866695+0000","last_fullsized":"2024-02-19T21:04:57.866695+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":77,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.791159+0000","last_change":"2024-02-19T21:03:22.984626+0000","last_active":"2024-02-19T21:04:57.791159+0000","last_peered":"2024-02-19T21:04:57.791159+0000","last_clean":"2024-02-19T21:04:57.791159+0000","last_became_active":"2024-02-19T21:01:30.922577+0000","last_became_peered":"2024-02-19T21:01:30.922577+0000","last_unstale":"2024-02-19T21:04:57.791159+0000","last_undegraded":"2024-02-19T21:04:57.791159+0000","last_fullsized":"2024-02-19T21:04:57.791159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:22.984547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:22.984547+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.798847+0000","last_change":"2024-02-19T21:03:33.049004+0000","last_active":"2024-02-19T21:04:57.798847+0000","last_peered":"2024-02-19T21:04:57.798847+0000","last_clean":"2024-02-19T21:04:57.798847+0000","last_became_active":"2024-02-19T21:01:16.271316+0000","last_became_peered":"2024-02-19T21:01:16.271316+0000","last_unstale":"2024-02-19T21:04:57.798847+0000","last_undegraded":"2024-02-19T21:04:57.798847+0000","last_fullsized":"2024-02-19T21:04:57.798847+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:33.048928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:33.048928+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.863205+0000","last_change":"2024-02-19T21:04:56.999269+0000","last_active":"2024-02-19T21:04:57.863205+0000","last_peered":"2024-02-19T21:04:57.863205+0000","last_clean":"2024-02-19T21:04:57.863205+0000","last_became_active":"2024-02-19T21:01:20.289691+0000","last_became_peered":"2024-02-19T21:01:20.289691+0000","last_unstale":"2024-02-19T21:04:57.863205+0000","last_undegraded":"2024-02-19T21:04:57.863205+0000","last_fullsized":"2024-02-19T21:04:57.863205+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:56.999142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.999142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.831266+0000","last_change":"2024-02-19T21:04:08.616495+0000","last_active":"2024-02-19T21:04:57.831266+0000","last_peered":"2024-02-19T21:04:57.831266+0000","last_clean":"2024-02-19T21:04:57.831266+0000","last_became_active":"2024-02-19T21:01:22.293093+0000","last_became_peered":"2024-02-19T21:01:22.293093+0000","last_unstale":"2024-02-19T21:04:57.831266+0000","last_undegraded":"2024-02-19T21:04:57.831266+0000","last_fullsized":"2024-02-19T21:04:57.831266+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.616396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.616396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.803000+0000","last_change":"2024-02-19T21:04:39.674336+0000","last_active":"2024-02-19T21:04:57.803000+0000","last_peered":"2024-02-19T21:04:57.803000+0000","last_clean":"2024-02-19T21:04:57.803000+0000","last_became_active":"2024-02-19T21:04:39.672228+0000","last_became_peered":"2024-02-19T21:04:39.672228+0000","last_unstale":"2024-02-19T21:04:57.803000+0000","last_undegraded":"2024-02-19T21:04:57.803000+0000","last_fullsized":"2024-02-19T21:04:57.803000+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.b","version":"102'3","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.797128+0000","last_change":"2024-02-19T21:04:39.657768+0000","last_active":"2024-02-19T21:04:57.797128+0000","last_peered":"2024-02-19T21:04:57.797128+0000","last_clean":"2024-02-19T21:04:57.797128+0000","last_became_active":"2024-02-19T21:04:39.655157+0000","last_became_peered":"2024-02-19T21:04:39.655157+0000","last_unstale":"2024-02-19T21:04:57.797128+0000","last_undegraded":"2024-02-19T21:04:57.797128+0000","last_fullsized":"2024-02-19T21:04:57.797128+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2807,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1f","version":"100'5","reported_seq":34,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.886706+0000","last_change":"2024-02-19T21:04:39.743813+0000","last_active":"2024-02-19T21:04:57.886706+0000","last_peered":"2024-02-19T21:04:57.886706+0000","last_clean":"2024-02-19T21:04:57.886706+0000","last_became_active":"2024-02-19T21:04:39.742840+0000","last_became_peered":"2024-02-19T21:04:39.742840+0000","last_unstale":"2024-02-19T21:04:57.886706+0000","last_undegraded":"2024-02-19T21:04:57.886706+0000","last_fullsized":"2024-02-19T21:04:57.886706+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.779950+0000","last_change":"2024-02-19T21:04:09.139464+0000","last_active":"2024-02-19T21:04:57.779950+0000","last_peered":"2024-02-19T21:04:57.779950+0000","last_clean":"2024-02-19T21:04:57.779950+0000","last_became_active":"2024-02-19T21:01:18.286128+0000","last_became_peered":"2024-02-19T21:01:18.286128+0000","last_unstale":"2024-02-19T21:04:57.779950+0000","last_undegraded":"2024-02-19T21:04:57.779950+0000","last_fullsized":"2024-02-19T21:04:57.779950+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:04:09.139344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:09.139344+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.801786+0000","last_change":"2024-02-19T21:03:20.445543+0000","last_active":"2024-02-19T21:04:57.801786+0000","last_peered":"2024-02-19T21:04:57.801786+0000","last_clean":"2024-02-19T21:04:57.801786+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:04:57.801786+0000","last_undegraded":"2024-02-19T21:04:57.801786+0000","last_fullsized":"2024-02-19T21:04:57.801786+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:20.445477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:20.445477+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.794607+0000","last_change":"2024-02-19T21:04:13.337608+0000","last_active":"2024-02-19T21:04:57.794607+0000","last_peered":"2024-02-19T21:04:57.794607+0000","last_clean":"2024-02-19T21:04:57.794607+0000","last_became_active":"2024-02-19T21:01:20.289020+0000","last_became_peered":"2024-02-19T21:01:20.289020+0000","last_unstale":"2024-02-19T21:04:57.794607+0000","last_undegraded":"2024-02-19T21:04:57.794607+0000","last_fullsized":"2024-02-19T21:04:57.794607+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.337498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.337498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.855588+0000","last_change":"2024-02-19T21:04:51.014323+0000","last_active":"2024-02-19T21:04:57.855588+0000","last_peered":"2024-02-19T21:04:57.855588+0000","last_clean":"2024-02-19T21:04:57.855588+0000","last_became_active":"2024-02-19T21:01:22.292704+0000","last_became_peered":"2024-02-19T21:01:22.292704+0000","last_unstale":"2024-02-19T21:04:57.855588+0000","last_undegraded":"2024-02-19T21:04:57.855588+0000","last_fullsized":"2024-02-19T21:04:57.855588+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:51.014245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:51.014245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.869910+0000","last_change":"2024-02-19T21:04:39.672911+0000","last_active":"2024-02-19T21:04:57.869910+0000","last_peered":"2024-02-19T21:04:57.869910+0000","last_clean":"2024-02-19T21:04:57.869910+0000","last_became_active":"2024-02-19T21:04:39.672672+0000","last_became_peered":"2024-02-19T21:04:39.672672+0000","last_unstale":"2024-02-19T21:04:57.869910+0000","last_undegraded":"2024-02-19T21:04:57.869910+0000","last_fullsized":"2024-02-19T21:04:57.869910+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.8","version":"100'2","reported_seq":35,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.788336+0000","last_change":"2024-02-19T21:04:49.651643+0000","last_active":"2024-02-19T21:04:57.788336+0000","last_peered":"2024-02-19T21:04:57.788336+0000","last_clean":"2024-02-19T21:04:57.788336+0000","last_became_active":"2024-02-19T21:04:39.725742+0000","last_became_peered":"2024-02-19T21:04:39.725742+0000","last_unstale":"2024-02-19T21:04:57.788336+0000","last_undegraded":"2024-02-19T21:04:57.788336+0000","last_fullsized":"2024-02-19T21:04:57.788336+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'2","last_scrub_stamp":"2024-02-19T21:04:49.651571+0000","last_deep_scrub":"100'2","last_deep_scrub_stamp":"2024-02-19T21:04:49.651571+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.651571+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.797276+0000","last_change":"2024-02-19T21:04:39.639950+0000","last_active":"2024-02-19T21:04:57.797276+0000","last_peered":"2024-02-19T21:04:57.797276+0000","last_clean":"2024-02-19T21:04:57.797276+0000","last_became_active":"2024-02-19T21:04:39.639552+0000","last_became_peered":"2024-02-19T21:04:39.639552+0000","last_unstale":"2024-02-19T21:04:57.797276+0000","last_undegraded":"2024-02-19T21:04:57.797276+0000","last_fullsized":"2024-02-19T21:04:57.797276+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":84,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.821860+0000","last_change":"2024-02-19T21:03:32.722001+0000","last_active":"2024-02-19T21:04:57.821860+0000","last_peered":"2024-02-19T21:04:57.821860+0000","last_clean":"2024-02-19T21:04:57.821860+0000","last_became_active":"2024-02-19T21:01:18.287874+0000","last_became_peered":"2024-02-19T21:01:18.287874+0000","last_unstale":"2024-02-19T21:04:57.821860+0000","last_undegraded":"2024-02-19T21:04:57.821860+0000","last_fullsized":"2024-02-19T21:04:57.821860+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:03:32.721877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:32.721877+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.798449+0000","last_change":"2024-02-19T21:04:13.205667+0000","last_active":"2024-02-19T21:04:57.798449+0000","last_peered":"2024-02-19T21:04:57.798449+0000","last_clean":"2024-02-19T21:04:57.798449+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:04:57.798449+0000","last_undegraded":"2024-02-19T21:04:57.798449+0000","last_fullsized":"2024-02-19T21:04:57.798449+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.205596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.205596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.789691+0000","last_change":"2024-02-19T21:04:11.754594+0000","last_active":"2024-02-19T21:04:57.789691+0000","last_peered":"2024-02-19T21:04:57.789691+0000","last_clean":"2024-02-19T21:04:57.789691+0000","last_became_active":"2024-02-19T21:01:20.290962+0000","last_became_peered":"2024-02-19T21:01:20.290962+0000","last_unstale":"2024-02-19T21:04:57.789691+0000","last_undegraded":"2024-02-19T21:04:57.789691+0000","last_fullsized":"2024-02-19T21:04:57.789691+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.754525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.754525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":73,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.780877+0000","last_change":"2024-02-19T21:04:13.941207+0000","last_active":"2024-02-19T21:04:57.780877+0000","last_peered":"2024-02-19T21:04:57.780877+0000","last_clean":"2024-02-19T21:04:57.780877+0000","last_became_active":"2024-02-19T21:01:22.297693+0000","last_became_peered":"2024-02-19T21:01:22.297693+0000","last_unstale":"2024-02-19T21:04:57.780877+0000","last_undegraded":"2024-02-19T21:04:57.780877+0000","last_fullsized":"2024-02-19T21:04:57.780877+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.941132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.941132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.871129+0000","last_change":"2024-02-19T21:04:39.717968+0000","last_active":"2024-02-19T21:04:57.871129+0000","last_peered":"2024-02-19T21:04:57.871129+0000","last_clean":"2024-02-19T21:04:57.871129+0000","last_became_active":"2024-02-19T21:04:39.716606+0000","last_became_peered":"2024-02-19T21:04:39.716606+0000","last_unstale":"2024-02-19T21:04:57.871129+0000","last_undegraded":"2024-02-19T21:04:57.871129+0000","last_fullsized":"2024-02-19T21:04:57.871129+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.791826+0000","last_change":"2024-02-19T21:04:53.134500+0000","last_active":"2024-02-19T21:04:57.791826+0000","last_peered":"2024-02-19T21:04:57.791826+0000","last_clean":"2024-02-19T21:04:57.791826+0000","last_became_active":"2024-02-19T21:04:39.658671+0000","last_became_peered":"2024-02-19T21:04:39.658671+0000","last_unstale":"2024-02-19T21:04:57.791826+0000","last_undegraded":"2024-02-19T21:04:57.791826+0000","last_fullsized":"2024-02-19T21:04:57.791826+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.134425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.134425+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.134425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"100'7","reported_seq":36,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.880597+0000","last_change":"2024-02-19T21:04:39.782279+0000","last_active":"2024-02-19T21:04:57.880597+0000","last_peered":"2024-02-19T21:04:57.880597+0000","last_clean":"2024-02-19T21:04:57.880597+0000","last_became_active":"2024-02-19T21:04:39.781203+0000","last_became_peered":"2024-02-19T21:04:39.781203+0000","last_unstale":"2024-02-19T21:04:57.880597+0000","last_undegraded":"2024-02-19T21:04:57.880597+0000","last_fullsized":"2024-02-19T21:04:57.880597+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.856034+0000","last_change":"2024-02-19T21:04:52.027084+0000","last_active":"2024-02-19T21:04:57.856034+0000","last_peered":"2024-02-19T21:04:57.856034+0000","last_clean":"2024-02-19T21:04:57.856034+0000","last_became_active":"2024-02-19T21:01:18.293793+0000","last_became_peered":"2024-02-19T21:01:18.293793+0000","last_unstale":"2024-02-19T21:04:57.856034+0000","last_undegraded":"2024-02-19T21:04:57.856034+0000","last_fullsized":"2024-02-19T21:04:57.856034+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:04:52.027000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:52.027000+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.801814+0000","last_change":"2024-02-19T21:04:13.816617+0000","last_active":"2024-02-19T21:04:57.801814+0000","last_peered":"2024-02-19T21:04:57.801814+0000","last_clean":"2024-02-19T21:04:57.801814+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:04:57.801814+0000","last_undegraded":"2024-02-19T21:04:57.801814+0000","last_fullsized":"2024-02-19T21:04:57.801814+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.816545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.816545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":70,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.779442+0000","last_change":"2024-02-19T21:03:04.486168+0000","last_active":"2024-02-19T21:04:57.779442+0000","last_peered":"2024-02-19T21:04:57.779442+0000","last_clean":"2024-02-19T21:04:57.779442+0000","last_became_active":"2024-02-19T21:01:20.287961+0000","last_became_peered":"2024-02-19T21:01:20.287961+0000","last_unstale":"2024-02-19T21:04:57.779442+0000","last_undegraded":"2024-02-19T21:04:57.779442+0000","last_fullsized":"2024-02-19T21:04:57.779442+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.486067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.486067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.863326+0000","last_change":"2024-02-19T21:04:10.863028+0000","last_active":"2024-02-19T21:04:57.863326+0000","last_peered":"2024-02-19T21:04:57.863326+0000","last_clean":"2024-02-19T21:04:57.863326+0000","last_became_active":"2024-02-19T21:01:22.608997+0000","last_became_peered":"2024-02-19T21:01:22.608997+0000","last_unstale":"2024-02-19T21:04:57.863326+0000","last_undegraded":"2024-02-19T21:04:57.863326+0000","last_fullsized":"2024-02-19T21:04:57.863326+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:10.862926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:10.862926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.880681+0000","last_change":"2024-02-19T21:04:39.686091+0000","last_active":"2024-02-19T21:04:57.880681+0000","last_peered":"2024-02-19T21:04:57.880681+0000","last_clean":"2024-02-19T21:04:57.880681+0000","last_became_active":"2024-02-19T21:04:39.685620+0000","last_became_peered":"2024-02-19T21:04:39.685620+0000","last_unstale":"2024-02-19T21:04:57.880681+0000","last_undegraded":"2024-02-19T21:04:57.880681+0000","last_fullsized":"2024-02-19T21:04:57.880681+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.786183+0000","last_change":"2024-02-19T21:04:45.355883+0000","last_active":"2024-02-19T21:04:57.786183+0000","last_peered":"2024-02-19T21:04:57.786183+0000","last_clean":"2024-02-19T21:04:57.786183+0000","last_became_active":"2024-02-19T21:04:39.755561+0000","last_became_peered":"2024-02-19T21:04:39.755561+0000","last_unstale":"2024-02-19T21:04:57.786183+0000","last_undegraded":"2024-02-19T21:04:57.786183+0000","last_fullsized":"2024-02-19T21:04:57.786183+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:45.355820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:45.355820+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.355820+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.799786+0000","last_change":"2024-02-19T21:04:39.665998+0000","last_active":"2024-02-19T21:04:57.799786+0000","last_peered":"2024-02-19T21:04:57.799786+0000","last_clean":"2024-02-19T21:04:57.799786+0000","last_became_active":"2024-02-19T21:04:39.662519+0000","last_became_peered":"2024-02-19T21:04:39.662519+0000","last_unstale":"2024-02-19T21:04:57.799786+0000","last_undegraded":"2024-02-19T21:04:57.799786+0000","last_fullsized":"2024-02-19T21:04:57.799786+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":93,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.885135+0000","last_change":"2024-02-19T21:04:04.545245+0000","last_active":"2024-02-19T21:04:57.885135+0000","last_peered":"2024-02-19T21:04:57.885135+0000","last_clean":"2024-02-19T21:04:57.885135+0000","last_became_active":"2024-02-19T21:01:18.293865+0000","last_became_peered":"2024-02-19T21:01:18.293865+0000","last_unstale":"2024-02-19T21:04:57.885135+0000","last_undegraded":"2024-02-19T21:04:57.885135+0000","last_fullsized":"2024-02-19T21:04:57.885135+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:04:04.545131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:04.545131+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.788706+0000","last_change":"2024-02-19T21:03:06.000805+0000","last_active":"2024-02-19T21:04:57.788706+0000","last_peered":"2024-02-19T21:04:57.788706+0000","last_clean":"2024-02-19T21:04:57.788706+0000","last_became_active":"2024-02-19T21:01:16.272284+0000","last_became_peered":"2024-02-19T21:01:16.272284+0000","last_unstale":"2024-02-19T21:04:57.788706+0000","last_undegraded":"2024-02-19T21:04:57.788706+0000","last_fullsized":"2024-02-19T21:04:57.788706+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.000732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.000732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":132,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-19T21:04:58.805470+0000","last_change":"2024-02-19T21:04:05.787496+0000","last_active":"2024-02-19T21:04:58.805470+0000","last_peered":"2024-02-19T21:04:58.805470+0000","last_clean":"2024-02-19T21:04:58.805470+0000","last_became_active":"2024-02-19T21:01:20.290567+0000","last_became_peered":"2024-02-19T21:01:20.290567+0000","last_unstale":"2024-02-19T21:04:58.805470+0000","last_undegraded":"2024-02-19T21:04:58.805470+0000","last_fullsized":"2024-02-19T21:04:58.805470+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:05.787415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:05.787415+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.782657+0000","last_change":"2024-02-19T21:04:18.877455+0000","last_active":"2024-02-19T21:04:57.782657+0000","last_peered":"2024-02-19T21:04:57.782657+0000","last_clean":"2024-02-19T21:04:57.782657+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:04:57.782657+0000","last_undegraded":"2024-02-19T21:04:57.782657+0000","last_fullsized":"2024-02-19T21:04:57.782657+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:04:18.877385+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.806873+0000","last_change":"2024-02-19T21:04:39.700717+0000","last_active":"2024-02-19T21:04:57.806873+0000","last_peered":"2024-02-19T21:04:57.806873+0000","last_clean":"2024-02-19T21:04:57.806873+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:04:57.806873+0000","last_undegraded":"2024-02-19T21:04:57.806873+0000","last_fullsized":"2024-02-19T21:04:57.806873+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"102'5","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.805794+0000","last_change":"2024-02-19T21:04:43.961401+0000","last_active":"2024-02-19T21:04:57.805794+0000","last_peered":"2024-02-19T21:04:57.805794+0000","last_clean":"2024-02-19T21:04:57.805794+0000","last_became_active":"2024-02-19T21:04:39.668876+0000","last_became_peered":"2024-02-19T21:04:39.668876+0000","last_unstale":"2024-02-19T21:04:57.805794+0000","last_undegraded":"2024-02-19T21:04:57.805794+0000","last_fullsized":"2024-02-19T21:04:57.805794+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'5","last_scrub_stamp":"2024-02-19T21:04:43.961315+0000","last_deep_scrub":"102'5","last_deep_scrub_stamp":"2024-02-19T21:04:43.961315+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.961315+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165832,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":27326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"100'9","reported_seq":38,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.862982+0000","last_change":"2024-02-19T21:04:39.724745+0000","last_active":"2024-02-19T21:04:57.862982+0000","last_peered":"2024-02-19T21:04:57.862982+0000","last_clean":"2024-02-19T21:04:57.862982+0000","last_became_active":"2024-02-19T21:04:39.715739+0000","last_became_peered":"2024-02-19T21:04:39.715739+0000","last_unstale":"2024-02-19T21:04:57.862982+0000","last_undegraded":"2024-02-19T21:04:57.862982+0000","last_fullsized":"2024-02-19T21:04:57.862982+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":97,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.802129+0000","last_change":"2024-02-19T21:04:04.812216+0000","last_active":"2024-02-19T21:04:57.802129+0000","last_peered":"2024-02-19T21:04:57.802129+0000","last_clean":"2024-02-19T21:04:57.802129+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:04:57.802129+0000","last_undegraded":"2024-02-19T21:04:57.802129+0000","last_fullsized":"2024-02-19T21:04:57.802129+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:04:04.812144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:04.812144+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.855777+0000","last_change":"2024-02-19T21:04:52.990442+0000","last_active":"2024-02-19T21:04:57.855777+0000","last_peered":"2024-02-19T21:04:57.855777+0000","last_clean":"2024-02-19T21:04:57.855777+0000","last_became_active":"2024-02-19T21:01:16.277826+0000","last_became_peered":"2024-02-19T21:01:16.277826+0000","last_unstale":"2024-02-19T21:04:57.855777+0000","last_undegraded":"2024-02-19T21:04:57.855777+0000","last_fullsized":"2024-02-19T21:04:57.855777+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:52.990315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:52.990315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.885608+0000","last_change":"2024-02-19T21:04:54.322261+0000","last_active":"2024-02-19T21:04:57.885608+0000","last_peered":"2024-02-19T21:04:57.885608+0000","last_clean":"2024-02-19T21:04:57.885608+0000","last_became_active":"2024-02-19T21:01:20.288016+0000","last_became_peered":"2024-02-19T21:01:20.288016+0000","last_unstale":"2024-02-19T21:04:57.885608+0000","last_undegraded":"2024-02-19T21:04:57.885608+0000","last_fullsized":"2024-02-19T21:04:57.885608+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:54.322192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:54.322192+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":69,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.795772+0000","last_change":"2024-02-19T21:03:06.456310+0000","last_active":"2024-02-19T21:04:57.795772+0000","last_peered":"2024-02-19T21:04:57.795772+0000","last_clean":"2024-02-19T21:04:57.795772+0000","last_became_active":"2024-02-19T21:01:22.293230+0000","last_became_peered":"2024-02-19T21:01:22.293230+0000","last_unstale":"2024-02-19T21:04:57.795772+0000","last_undegraded":"2024-02-19T21:04:57.795772+0000","last_fullsized":"2024-02-19T21:04:57.795772+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:06.456212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:06.456212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.802404+0000","last_change":"2024-02-19T21:04:39.801737+0000","last_active":"2024-02-19T21:04:57.802404+0000","last_peered":"2024-02-19T21:04:57.802404+0000","last_clean":"2024-02-19T21:04:57.802404+0000","last_became_active":"2024-02-19T21:04:39.797018+0000","last_became_peered":"2024-02-19T21:04:39.797018+0000","last_unstale":"2024-02-19T21:04:57.802404+0000","last_undegraded":"2024-02-19T21:04:57.802404+0000","last_fullsized":"2024-02-19T21:04:57.802404+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.798713+0000","last_change":"2024-02-19T21:04:44.678319+0000","last_active":"2024-02-19T21:04:57.798713+0000","last_peered":"2024-02-19T21:04:57.798713+0000","last_clean":"2024-02-19T21:04:57.798713+0000","last_became_active":"2024-02-19T21:04:39.676208+0000","last_became_peered":"2024-02-19T21:04:39.676208+0000","last_unstale":"2024-02-19T21:04:57.798713+0000","last_undegraded":"2024-02-19T21:04:57.798713+0000","last_fullsized":"2024-02-19T21:04:57.798713+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:44.678223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:44.678223+0000","last_clean_scrub_stamp":"2024-02-19T21:04:44.678223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.778838+0000","last_change":"2024-02-19T21:04:39.613875+0000","last_active":"2024-02-19T21:04:57.778838+0000","last_peered":"2024-02-19T21:04:57.778838+0000","last_clean":"2024-02-19T21:04:57.778838+0000","last_became_active":"2024-02-19T21:04:39.613576+0000","last_became_peered":"2024-02-19T21:04:39.613576+0000","last_unstale":"2024-02-19T21:04:57.778838+0000","last_undegraded":"2024-02-19T21:04:57.778838+0000","last_fullsized":"2024-02-19T21:04:57.778838+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":103,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.863909+0000","last_change":"2024-02-19T21:04:20.874434+0000","last_active":"2024-02-19T21:04:57.863909+0000","last_peered":"2024-02-19T21:04:57.863909+0000","last_clean":"2024-02-19T21:04:57.863909+0000","last_became_active":"2024-02-19T21:01:18.289712+0000","last_became_peered":"2024-02-19T21:01:18.289712+0000","last_unstale":"2024-02-19T21:04:57.863909+0000","last_undegraded":"2024-02-19T21:04:57.863909+0000","last_fullsized":"2024-02-19T21:04:57.863909+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:04:20.874350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:20.874350+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.799933+0000","last_change":"2024-02-19T21:04:21.828958+0000","last_active":"2024-02-19T21:04:57.799933+0000","last_peered":"2024-02-19T21:04:57.799933+0000","last_clean":"2024-02-19T21:04:57.799933+0000","last_became_active":"2024-02-19T21:01:16.274524+0000","last_became_peered":"2024-02-19T21:01:16.274524+0000","last_unstale":"2024-02-19T21:04:57.799933+0000","last_undegraded":"2024-02-19T21:04:57.799933+0000","last_fullsized":"2024-02-19T21:04:57.799933+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:21.828876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.828876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.804929+0000","last_change":"2024-02-19T21:04:49.851482+0000","last_active":"2024-02-19T21:04:57.804929+0000","last_peered":"2024-02-19T21:04:57.804929+0000","last_clean":"2024-02-19T21:04:57.804929+0000","last_became_active":"2024-02-19T21:01:20.290705+0000","last_became_peered":"2024-02-19T21:01:20.290705+0000","last_unstale":"2024-02-19T21:04:57.804929+0000","last_undegraded":"2024-02-19T21:04:57.804929+0000","last_fullsized":"2024-02-19T21:04:57.804929+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.851392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.851392+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":69,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.797177+0000","last_change":"2024-02-19T21:03:12.081314+0000","last_active":"2024-02-19T21:04:57.797177+0000","last_peered":"2024-02-19T21:04:57.797177+0000","last_clean":"2024-02-19T21:04:57.797177+0000","last_became_active":"2024-02-19T21:01:22.607435+0000","last_became_peered":"2024-02-19T21:01:22.607435+0000","last_unstale":"2024-02-19T21:04:57.797177+0000","last_undegraded":"2024-02-19T21:04:57.797177+0000","last_fullsized":"2024-02-19T21:04:57.797177+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:03:12.081227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.832017+0000","last_change":"2024-02-19T21:04:39.688234+0000","last_active":"2024-02-19T21:04:57.832017+0000","last_peered":"2024-02-19T21:04:57.832017+0000","last_clean":"2024-02-19T21:04:57.832017+0000","last_became_active":"2024-02-19T21:04:39.687307+0000","last_became_peered":"2024-02-19T21:04:39.687307+0000","last_unstale":"2024-02-19T21:04:57.832017+0000","last_undegraded":"2024-02-19T21:04:57.832017+0000","last_fullsized":"2024-02-19T21:04:57.832017+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.784944+0000","last_change":"2024-02-19T21:04:43.320971+0000","last_active":"2024-02-19T21:04:57.784944+0000","last_peered":"2024-02-19T21:04:57.784944+0000","last_clean":"2024-02-19T21:04:57.784944+0000","last_became_active":"2024-02-19T21:04:39.690799+0000","last_became_peered":"2024-02-19T21:04:39.690799+0000","last_unstale":"2024-02-19T21:04:57.784944+0000","last_undegraded":"2024-02-19T21:04:57.784944+0000","last_fullsized":"2024-02-19T21:04:57.784944+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.320871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.320871+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.320871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.11","version":"100'7","reported_seq":36,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.791739+0000","last_change":"2024-02-19T21:04:39.612171+0000","last_active":"2024-02-19T21:04:57.791739+0000","last_peered":"2024-02-19T21:04:57.791739+0000","last_clean":"2024-02-19T21:04:57.791739+0000","last_became_active":"2024-02-19T21:04:39.610484+0000","last_became_peered":"2024-02-19T21:04:39.610484+0000","last_unstale":"2024-02-19T21:04:57.791739+0000","last_undegraded":"2024-02-19T21:04:57.791739+0000","last_fullsized":"2024-02-19T21:04:57.791739+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":101,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.865611+0000","last_change":"2024-02-19T21:04:50.027760+0000","last_active":"2024-02-19T21:04:57.865611+0000","last_peered":"2024-02-19T21:04:57.865611+0000","last_clean":"2024-02-19T21:04:57.865611+0000","last_became_active":"2024-02-19T21:01:18.285926+0000","last_became_peered":"2024-02-19T21:01:18.285926+0000","last_unstale":"2024-02-19T21:04:57.865611+0000","last_undegraded":"2024-02-19T21:04:57.865611+0000","last_fullsized":"2024-02-19T21:04:57.865611+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:04:50.027690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:50.027690+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.863167+0000","last_change":"2024-02-19T21:03:20.637528+0000","last_active":"2024-02-19T21:04:57.863167+0000","last_peered":"2024-02-19T21:04:57.863167+0000","last_clean":"2024-02-19T21:04:57.863167+0000","last_became_active":"2024-02-19T21:01:16.274811+0000","last_became_peered":"2024-02-19T21:01:16.274811+0000","last_unstale":"2024-02-19T21:04:57.863167+0000","last_undegraded":"2024-02-19T21:04:57.863167+0000","last_fullsized":"2024-02-19T21:04:57.863167+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:20.637465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:20.637465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":71,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.791019+0000","last_change":"2024-02-19T21:03:09.010420+0000","last_active":"2024-02-19T21:04:57.791019+0000","last_peered":"2024-02-19T21:04:57.791019+0000","last_clean":"2024-02-19T21:04:57.791019+0000","last_became_active":"2024-02-19T21:01:20.291672+0000","last_became_peered":"2024-02-19T21:01:20.291672+0000","last_unstale":"2024-02-19T21:04:57.791019+0000","last_undegraded":"2024-02-19T21:04:57.791019+0000","last_fullsized":"2024-02-19T21:04:57.791019+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:09.010341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:09.010341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.870057+0000","last_change":"2024-02-19T21:04:03.806998+0000","last_active":"2024-02-19T21:04:57.870057+0000","last_peered":"2024-02-19T21:04:57.870057+0000","last_clean":"2024-02-19T21:04:57.870057+0000","last_became_active":"2024-02-19T21:01:22.607099+0000","last_became_peered":"2024-02-19T21:01:22.607099+0000","last_unstale":"2024-02-19T21:04:57.870057+0000","last_undegraded":"2024-02-19T21:04:57.870057+0000","last_fullsized":"2024-02-19T21:04:57.870057+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:03.806903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:03.806903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.813050+0000","last_change":"2024-02-19T21:04:39.781223+0000","last_active":"2024-02-19T21:04:57.813050+0000","last_peered":"2024-02-19T21:04:57.813050+0000","last_clean":"2024-02-19T21:04:57.813050+0000","last_became_active":"2024-02-19T21:04:39.777442+0000","last_became_peered":"2024-02-19T21:04:39.777442+0000","last_unstale":"2024-02-19T21:04:57.813050+0000","last_undegraded":"2024-02-19T21:04:57.813050+0000","last_fullsized":"2024-02-19T21:04:57.813050+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.844116+0000","last_change":"2024-02-19T21:04:42.535665+0000","last_active":"2024-02-19T21:04:57.844116+0000","last_peered":"2024-02-19T21:04:57.844116+0000","last_clean":"2024-02-19T21:04:57.844116+0000","last_became_active":"2024-02-19T21:04:39.716211+0000","last_became_peered":"2024-02-19T21:04:39.716211+0000","last_unstale":"2024-02-19T21:04:57.844116+0000","last_undegraded":"2024-02-19T21:04:57.844116+0000","last_fullsized":"2024-02-19T21:04:57.844116+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:42.535597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:42.535597+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.535597+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.16","version":"100'5","reported_seq":34,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.866772+0000","last_change":"2024-02-19T21:04:39.742946+0000","last_active":"2024-02-19T21:04:57.866772+0000","last_peered":"2024-02-19T21:04:57.866772+0000","last_clean":"2024-02-19T21:04:57.866772+0000","last_became_active":"2024-02-19T21:04:39.741473+0000","last_became_peered":"2024-02-19T21:04:39.741473+0000","last_unstale":"2024-02-19T21:04:57.866772+0000","last_undegraded":"2024-02-19T21:04:57.866772+0000","last_fullsized":"2024-02-19T21:04:57.866772+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":94,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.799684+0000","last_change":"2024-02-19T21:04:11.165275+0000","last_active":"2024-02-19T21:04:57.799684+0000","last_peered":"2024-02-19T21:04:57.799684+0000","last_clean":"2024-02-19T21:04:57.799684+0000","last_became_active":"2024-02-19T21:01:18.285269+0000","last_became_peered":"2024-02-19T21:01:18.285269+0000","last_unstale":"2024-02-19T21:04:57.799684+0000","last_undegraded":"2024-02-19T21:04:57.799684+0000","last_fullsized":"2024-02-19T21:04:57.799684+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:11.165197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.165197+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":79,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-02-19T21:04:59.262530+0000","last_change":"2024-02-19T21:04:59.262530+0000","last_active":"2024-02-19T21:04:59.262530+0000","last_peered":"2024-02-19T21:04:59.262530+0000","last_clean":"2024-02-19T21:04:59.262530+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:04:59.262530+0000","last_undegraded":"2024-02-19T21:04:59.262530+0000","last_fullsized":"2024-02-19T21:04:59.262530+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:59.262400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:59.262400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":71,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.793519+0000","last_change":"2024-02-19T21:03:26.922649+0000","last_active":"2024-02-19T21:04:57.793519+0000","last_peered":"2024-02-19T21:04:57.793519+0000","last_clean":"2024-02-19T21:04:57.793519+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:04:57.793519+0000","last_undegraded":"2024-02-19T21:04:57.793519+0000","last_fullsized":"2024-02-19T21:04:57.793519+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:26.922572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:26.922572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.831788+0000","last_change":"2024-02-19T21:04:49.382806+0000","last_active":"2024-02-19T21:04:57.831788+0000","last_peered":"2024-02-19T21:04:57.831788+0000","last_clean":"2024-02-19T21:04:57.831788+0000","last_became_active":"2024-02-19T21:01:22.292714+0000","last_became_peered":"2024-02-19T21:01:22.292714+0000","last_unstale":"2024-02-19T21:04:57.831788+0000","last_undegraded":"2024-02-19T21:04:57.831788+0000","last_fullsized":"2024-02-19T21:04:57.831788+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.382712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.382712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.861722+0000","last_change":"2024-02-19T21:04:39.737500+0000","last_active":"2024-02-19T21:04:57.861722+0000","last_peered":"2024-02-19T21:04:57.861722+0000","last_clean":"2024-02-19T21:04:57.861722+0000","last_became_active":"2024-02-19T21:04:39.736990+0000","last_became_peered":"2024-02-19T21:04:39.736990+0000","last_unstale":"2024-02-19T21:04:57.861722+0000","last_undegraded":"2024-02-19T21:04:57.861722+0000","last_fullsized":"2024-02-19T21:04:57.861722+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3","version":"101'1003","reported_seq":2037,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.793794+0000","last_change":"2024-02-19T21:04:45.203149+0000","last_active":"2024-02-19T21:04:57.793794+0000","last_peered":"2024-02-19T21:04:57.793794+0000","last_clean":"2024-02-19T21:04:57.793794+0000","last_became_active":"2024-02-19T21:04:39.663589+0000","last_became_peered":"2024-02-19T21:04:39.663589+0000","last_unstale":"2024-02-19T21:04:57.793794+0000","last_undegraded":"2024-02-19T21:04:57.793794+0000","last_fullsized":"2024-02-19T21:04:57.793794+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'1003","last_scrub_stamp":"2024-02-19T21:04:45.203066+0000","last_deep_scrub":"101'1003","last_deep_scrub_stamp":"2024-02-19T21:04:45.203066+0000","last_clean_scrub_stamp":"2024-02-19T21:04:45.203066+0000","log_size":1003,"ondisk_log_size":1003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1001,"num_read_kb":1,"num_write":1002,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.17","version":"100'5","reported_seq":34,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.779714+0000","last_change":"2024-02-19T21:04:39.746785+0000","last_active":"2024-02-19T21:04:57.779714+0000","last_peered":"2024-02-19T21:04:57.779714+0000","last_clean":"2024-02-19T21:04:57.779714+0000","last_became_active":"2024-02-19T21:04:39.746470+0000","last_became_peered":"2024-02-19T21:04:39.746470+0000","last_unstale":"2024-02-19T21:04:57.779714+0000","last_undegraded":"2024-02-19T21:04:57.779714+0000","last_fullsized":"2024-02-19T21:04:57.779714+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":88,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.779796+0000","last_change":"2024-02-19T21:04:12.133700+0000","last_active":"2024-02-19T21:04:57.779796+0000","last_peered":"2024-02-19T21:04:57.779796+0000","last_clean":"2024-02-19T21:04:57.779796+0000","last_became_active":"2024-02-19T21:01:18.285735+0000","last_became_peered":"2024-02-19T21:01:18.285735+0000","last_unstale":"2024-02-19T21:04:57.779796+0000","last_undegraded":"2024-02-19T21:04:57.779796+0000","last_fullsized":"2024-02-19T21:04:57.779796+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:04:12.133628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:12.133628+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":79,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.784049+0000","last_change":"2024-02-19T21:03:26.793343+0000","last_active":"2024-02-19T21:04:57.784049+0000","last_peered":"2024-02-19T21:04:57.784049+0000","last_clean":"2024-02-19T21:04:57.784049+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:04:57.784049+0000","last_undegraded":"2024-02-19T21:04:57.784049+0000","last_fullsized":"2024-02-19T21:04:57.784049+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:26.793263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:26.793263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.781081+0000","last_change":"2024-02-19T21:04:10.891192+0000","last_active":"2024-02-19T21:04:57.781081+0000","last_peered":"2024-02-19T21:04:57.781081+0000","last_clean":"2024-02-19T21:04:57.781081+0000","last_became_active":"2024-02-19T21:01:20.287940+0000","last_became_peered":"2024-02-19T21:01:20.287940+0000","last_unstale":"2024-02-19T21:04:57.781081+0000","last_undegraded":"2024-02-19T21:04:57.781081+0000","last_fullsized":"2024-02-19T21:04:57.781081+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:10.891113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:10.891113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.791511+0000","last_change":"2024-02-19T21:04:19.314335+0000","last_active":"2024-02-19T21:04:57.791511+0000","last_peered":"2024-02-19T21:04:57.791511+0000","last_clean":"2024-02-19T21:04:57.791511+0000","last_became_active":"2024-02-19T21:01:22.293282+0000","last_became_peered":"2024-02-19T21:01:22.293282+0000","last_unstale":"2024-02-19T21:04:57.791511+0000","last_undegraded":"2024-02-19T21:04:57.791511+0000","last_fullsized":"2024-02-19T21:04:57.791511+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:19.314241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.314241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.886967+0000","last_change":"2024-02-19T21:04:39.714784+0000","last_active":"2024-02-19T21:04:57.886967+0000","last_peered":"2024-02-19T21:04:57.886967+0000","last_clean":"2024-02-19T21:04:57.886967+0000","last_became_active":"2024-02-19T21:04:39.714495+0000","last_became_peered":"2024-02-19T21:04:39.714495+0000","last_unstale":"2024-02-19T21:04:57.886967+0000","last_undegraded":"2024-02-19T21:04:57.886967+0000","last_fullsized":"2024-02-19T21:04:57.886967+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":33,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.789148+0000","last_change":"2024-02-19T21:04:43.646643+0000","last_active":"2024-02-19T21:04:57.789148+0000","last_peered":"2024-02-19T21:04:57.789148+0000","last_clean":"2024-02-19T21:04:57.789148+0000","last_became_active":"2024-02-19T21:04:39.726486+0000","last_became_peered":"2024-02-19T21:04:39.726486+0000","last_unstale":"2024-02-19T21:04:57.789148+0000","last_undegraded":"2024-02-19T21:04:57.789148+0000","last_fullsized":"2024-02-19T21:04:57.789148+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:43.646543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:43.646543+0000","last_clean_scrub_stamp":"2024-02-19T21:04:43.646543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.14","version":"100'2","reported_seq":31,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.791218+0000","last_change":"2024-02-19T21:04:39.628834+0000","last_active":"2024-02-19T21:04:57.791218+0000","last_peered":"2024-02-19T21:04:57.791218+0000","last_clean":"2024-02-19T21:04:57.791218+0000","last_became_active":"2024-02-19T21:04:39.628525+0000","last_became_peered":"2024-02-19T21:04:39.628525+0000","last_unstale":"2024-02-19T21:04:57.791218+0000","last_undegraded":"2024-02-19T21:04:57.791218+0000","last_fullsized":"2024-02-19T21:04:57.791218+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":89,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.789860+0000","last_change":"2024-02-19T21:03:16.047345+0000","last_active":"2024-02-19T21:04:57.789860+0000","last_peered":"2024-02-19T21:04:57.789860+0000","last_clean":"2024-02-19T21:04:57.789860+0000","last_became_active":"2024-02-19T21:01:18.287645+0000","last_became_peered":"2024-02-19T21:01:18.287645+0000","last_unstale":"2024-02-19T21:04:57.789860+0000","last_undegraded":"2024-02-19T21:04:57.789860+0000","last_fullsized":"2024-02-19T21:04:57.789860+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:16.047258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:16.047258+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":80,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.870417+0000","last_change":"2024-02-19T21:04:02.781654+0000","last_active":"2024-02-19T21:04:57.870417+0000","last_peered":"2024-02-19T21:04:57.870417+0000","last_clean":"2024-02-19T21:04:57.870417+0000","last_became_active":"2024-02-19T21:01:16.272132+0000","last_became_peered":"2024-02-19T21:01:16.272132+0000","last_unstale":"2024-02-19T21:04:57.870417+0000","last_undegraded":"2024-02-19T21:04:57.870417+0000","last_fullsized":"2024-02-19T21:04:57.870417+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:02.781556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:02.781556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":76,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.812100+0000","last_change":"2024-02-19T21:04:16.276617+0000","last_active":"2024-02-19T21:04:57.812100+0000","last_peered":"2024-02-19T21:04:57.812100+0000","last_clean":"2024-02-19T21:04:57.812100+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:04:57.812100+0000","last_undegraded":"2024-02-19T21:04:57.812100+0000","last_fullsized":"2024-02-19T21:04:57.812100+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:16.276518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.276518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":74,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.799655+0000","last_change":"2024-02-19T21:04:08.178302+0000","last_active":"2024-02-19T21:04:57.799655+0000","last_peered":"2024-02-19T21:04:57.799655+0000","last_clean":"2024-02-19T21:04:57.799655+0000","last_became_active":"2024-02-19T21:01:22.293432+0000","last_became_peered":"2024-02-19T21:01:22.293432+0000","last_unstale":"2024-02-19T21:04:57.799655+0000","last_undegraded":"2024-02-19T21:04:57.799655+0000","last_fullsized":"2024-02-19T21:04:57.799655+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.178225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.178225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.797352+0000","last_change":"2024-02-19T21:04:39.662781+0000","last_active":"2024-02-19T21:04:57.797352+0000","last_peered":"2024-02-19T21:04:57.797352+0000","last_clean":"2024-02-19T21:04:57.797352+0000","last_became_active":"2024-02-19T21:04:39.661810+0000","last_became_peered":"2024-02-19T21:04:39.661810+0000","last_unstale":"2024-02-19T21:04:57.797352+0000","last_undegraded":"2024-02-19T21:04:57.797352+0000","last_fullsized":"2024-02-19T21:04:57.797352+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1","version":"103'3","reported_seq":39,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.813138+0000","last_change":"2024-02-19T21:04:42.918366+0000","last_active":"2024-02-19T21:04:57.813138+0000","last_peered":"2024-02-19T21:04:57.813138+0000","last_clean":"2024-02-19T21:04:57.813138+0000","last_became_active":"2024-02-19T21:04:39.683344+0000","last_became_peered":"2024-02-19T21:04:39.683344+0000","last_unstale":"2024-02-19T21:04:57.813138+0000","last_undegraded":"2024-02-19T21:04:57.813138+0000","last_fullsized":"2024-02-19T21:04:57.813138+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'3","last_scrub_stamp":"2024-02-19T21:04:42.918299+0000","last_deep_scrub":"103'3","last_deep_scrub_stamp":"2024-02-19T21:04:42.918299+0000","last_clean_scrub_stamp":"2024-02-19T21:04:42.918299+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":29,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.783555+0000","last_change":"2024-02-19T21:04:39.691058+0000","last_active":"2024-02-19T21:04:57.783555+0000","last_peered":"2024-02-19T21:04:57.783555+0000","last_clean":"2024-02-19T21:04:57.783555+0000","last_became_active":"2024-02-19T21:04:39.687677+0000","last_became_peered":"2024-02-19T21:04:39.687677+0000","last_unstale":"2024-02-19T21:04:57.783555+0000","last_undegraded":"2024-02-19T21:04:57.783555+0000","last_fullsized":"2024-02-19T21:04:57.783555+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":103,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.856119+0000","last_change":"2024-02-19T21:04:14.826385+0000","last_active":"2024-02-19T21:04:57.856119+0000","last_peered":"2024-02-19T21:04:57.856119+0000","last_clean":"2024-02-19T21:04:57.856119+0000","last_became_active":"2024-02-19T21:01:18.283222+0000","last_became_peered":"2024-02-19T21:01:18.283222+0000","last_unstale":"2024-02-19T21:04:57.856119+0000","last_undegraded":"2024-02-19T21:04:57.856119+0000","last_fullsized":"2024-02-19T21:04:57.856119+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:04:14.826273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:14.826273+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":75,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.886780+0000","last_change":"2024-02-19T21:03:04.644754+0000","last_active":"2024-02-19T21:04:57.886780+0000","last_peered":"2024-02-19T21:04:57.886780+0000","last_clean":"2024-02-19T21:04:57.886780+0000","last_became_active":"2024-02-19T21:01:16.278753+0000","last_became_peered":"2024-02-19T21:01:16.278753+0000","last_unstale":"2024-02-19T21:04:57.886780+0000","last_undegraded":"2024-02-19T21:04:57.886780+0000","last_fullsized":"2024-02-19T21:04:57.886780+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.644689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.644689+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":71,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.790164+0000","last_change":"2024-02-19T21:03:05.073746+0000","last_active":"2024-02-19T21:04:57.790164+0000","last_peered":"2024-02-19T21:04:57.790164+0000","last_clean":"2024-02-19T21:04:57.790164+0000","last_became_active":"2024-02-19T21:01:20.289208+0000","last_became_peered":"2024-02-19T21:01:20.289208+0000","last_unstale":"2024-02-19T21:04:57.790164+0000","last_undegraded":"2024-02-19T21:04:57.790164+0000","last_fullsized":"2024-02-19T21:04:57.790164+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.073664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.073664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":69,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.863070+0000","last_change":"2024-02-19T21:03:27.633572+0000","last_active":"2024-02-19T21:04:57.863070+0000","last_peered":"2024-02-19T21:04:57.863070+0000","last_clean":"2024-02-19T21:04:57.863070+0000","last_became_active":"2024-02-19T21:01:22.609059+0000","last_became_peered":"2024-02-19T21:01:22.609059+0000","last_unstale":"2024-02-19T21:04:57.863070+0000","last_undegraded":"2024-02-19T21:04:57.863070+0000","last_fullsized":"2024-02-19T21:04:57.863070+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:27.633492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:27.633492+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":28,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-02-19T21:04:57.800871+0000","last_change":"2024-02-19T21:04:39.726477+0000","last_active":"2024-02-19T21:04:57.800871+0000","last_peered":"2024-02-19T21:04:57.800871+0000","last_clean":"2024-02-19T21:04:57.800871+0000","last_became_active":"2024-02-19T21:04:39.726195+0000","last_became_peered":"2024-02-19T21:04:39.726195+0000","last_unstale":"2024-02-19T21:04:57.800871+0000","last_undegraded":"2024-02-19T21:04:57.800871+0000","last_fullsized":"2024-02-19T21:04:57.800871+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":74,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":73,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":72,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":71,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":96,"acting":96,"num_store_stats":8},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":96,"acting":96,"num_store_stats":8},{"poolid":54,"num_pg":4,"stat_sum":{"num_bytes":1536,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":10,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":4608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47,"ondisk_log_size":47,"up":12,"acting":12,"num_store_stats":7},{"poolid":52,"num_pg":32,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":46,"num_pg":32,"stat_sum":{"num_bytes":651,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":150998978,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":1011,"num_read_kb":49161,"num_write":1043,"num_write_kb":152978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":453267456,"data_stored":452996934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1054,"ondisk_log_size":1054,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":216,"num_read_kb":211,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266735,"num_pgs":181,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":359908,"kb_used_data":50524,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93385244,"statfs":{"total":95995035648,"available":95626489856,"internally_reserved":0,"allocated":51736576,"data_stored":51048039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.49299999999999999,"15min":0.504},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.77400000000000002,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.78200000000000003,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.31900000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50900000000000001,"15min":0.51900000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.67400000000000004,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51200000000000001,"15min":0.51800000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.43099999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.49399999999999999,"15min":0.498},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":2.4980000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.53000000000000003,"15min":0.52400000000000002},"min":{"1min":0.39700000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73999999999999999,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.48099999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.71199999999999997,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.504,"15min":0.497},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.49399999999999999,"15min":0.501},"min":{"1min":0.33900000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73799999999999999,"5min":1.373,"15min":1.373},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.496,"15min":0.48999999999999999},"min":{"1min":0.32000000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.81100000000000005,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.33600000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.47299999999999998,"15min":0.46500000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.64900000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52400000000000002,"15min":0.51600000000000001},"min":{"1min":0.38800000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80800000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.49399999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.54000000000000004,"15min":0.54100000000000004},"min":{"1min":0.34899999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82299999999999995,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.57899999999999996,"15min":0.57999999999999996},"min":{"1min":0.44500000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83799999999999997,"5min":1.262,"15min":1.262},"last":0.46000000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528373,"num_pgs":167,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":380428,"kb_used_data":75140,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93364724,"statfs":{"total":95995035648,"available":95605477376,"internally_reserved":0,"allocated":76943360,"data_stored":76218081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.55800000000000005,"15min":0.51500000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":3.9689999999999999,"5min":3.9689999999999999,"15min":3.9689999999999999},"last":0.505},{"interface":"front","average":{"1min":1.246,"5min":0.64200000000000002,"15min":0.54000000000000004},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":10.119,"5min":10.119,"15min":10.119},"last":0.51200000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":0.61099999999999999,"15min":0.53100000000000003},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":8.0310000000000006,"5min":8.0310000000000006,"15min":8.0310000000000006},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.56100000000000005,"15min":0.52000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.3520000000000003,"5min":4.3520000000000003,"15min":4.3520000000000003},"last":0.45300000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.56599999999999995,"15min":0.53500000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":4.1529999999999996,"5min":4.1529999999999996,"15min":4.1529999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.59399999999999997,"15min":0.52400000000000002},"min":{"1min":0.33400000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":8.6430000000000007,"5min":8.6430000000000007,"15min":8.6430000000000007},"last":0.46400000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":0.65800000000000003,"15min":0.56000000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":11.927,"5min":11.927,"15min":11.927},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.56799999999999995,"15min":0.51800000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.7670000000000003,"5min":4.7670000000000003,"15min":4.7670000000000003},"last":0.54400000000000004}]},{"osd":4,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.41699999999999998,"15min":0.38300000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.58099999999999996,"5min":1.363,"15min":1.363},"last":0.443},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.436,"15min":0.40999999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69899999999999995,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.53600000000000003}]},{"osd":5,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45700000000000002,"15min":0.42099999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.59799999999999998,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48999999999999999,"15min":0.45600000000000002},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71699999999999997,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.55100000000000005}]},{"osd":7,"last update":"Mon Feb 19 21:04:19 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.35599999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68899999999999995,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52600000000000002,"15min":0.53300000000000003},"min":{"1min":0.39600000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65900000000000003,"5min":1.534,"15min":1.534},"last":0.48899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":185,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":355856,"kb_used_data":50568,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93389296,"statfs":{"total":95995035648,"available":95630639104,"internally_reserved":0,"allocated":51781632,"data_stored":51054733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:59 2024","interfaces":[{"interface":"back","average":{"1min":2.2589999999999999,"5min":0.75,"15min":0.47799999999999998},"min":{"1min":0.245,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.777000000000001,"5min":27.777000000000001,"15min":27.777000000000001},"last":27.777000000000001},{"interface":"front","average":{"1min":2.4969999999999999,"5min":0.81200000000000006,"15min":0.504},"min":{"1min":0.29899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":27.631,"5min":27.631,"15min":27.631},"last":27.631}]},{"osd":2,"last update":"Mon Feb 19 21:04:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.41499999999999998,"15min":0.40999999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":27.798999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29199999999999998,"5min":0.255,"15min":0.255},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":27.606999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:04:45 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.41099999999999998,"15min":0.42299999999999999},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.92000000000000004,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":8.9730000000000008},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.46200000000000002,"15min":0.46500000000000002},"min":{"1min":0.33600000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":27.82}]},{"osd":4,"last update":"Mon Feb 19 21:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.56599999999999995,"15min":0.56100000000000005},"min":{"1min":0.42599999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":8.8930000000000007},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57599999999999996,"15min":0.56599999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":27.707000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:04:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.47299999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":27.529},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60399999999999998,"15min":0.61199999999999999},"min":{"1min":0.438,"5min":0.377,"15min":0.377},"max":{"1min":0.93999999999999995,"5min":1.052,"15min":1.052},"last":8.952}]},{"osd":6,"last update":"Mon Feb 19 21:04:50 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40699999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89900000000000002,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":8.9260000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.63500000000000001,"15min":0.64500000000000002},"min":{"1min":0.45400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":27.733000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:04:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.60299999999999998,"15min":0.59199999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":27.576000000000001},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.61799999999999999,"15min":0.60699999999999998},"min":{"1min":0.441,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":27.756}]}]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":192,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":359960,"kb_used_data":50576,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93385192,"statfs":{"total":95995035648,"available":95626436608,"internally_reserved":0,"allocated":51789824,"data_stored":51069291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.819,"5min":0.69899999999999995,"15min":0.42399999999999999},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.16200000000000001},"max":{"1min":12.720000000000001,"5min":12.720000000000001,"15min":12.720000000000001},"last":8.9250000000000007},{"interface":"front","average":{"1min":2.2690000000000001,"5min":0.74199999999999999,"15min":0.44400000000000001},"min":{"1min":0.26300000000000001,"5min":0.20899999999999999,"15min":0.188},"max":{"1min":18.193000000000001,"5min":18.193000000000001,"15min":18.193000000000001},"last":10.131}]},{"osd":2,"last update":"Mon Feb 19 21:04:20 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40999999999999998,"15min":0.38200000000000001},"min":{"1min":0.26900000000000002,"5min":0.219,"15min":0.219},"max":{"1min":1.079,"5min":3.121,"15min":3.121},"last":8.8550000000000004},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.44500000000000001,"15min":0.39500000000000002},"min":{"1min":0.25700000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.3720000000000001,"5min":4.2569999999999997,"15min":4.2569999999999997},"last":0.439}]},{"osd":3,"last update":"Mon Feb 19 21:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.47399999999999998,"15min":0.45200000000000001},"min":{"1min":0.28100000000000003,"5min":0.249,"15min":0.249},"max":{"1min":1.238,"5min":4.4470000000000001,"15min":4.4470000000000001},"last":8.8900000000000006},{"interface":"front","average":{"1min":0.502,"5min":0.48499999999999999,"15min":0.46100000000000002},"min":{"1min":0.315,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.97099999999999997,"5min":4.3070000000000004,"15min":4.3070000000000004},"last":9.9429999999999996}]},{"osd":4,"last update":"Mon Feb 19 21:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":0.76700000000000002,"15min":0.59899999999999998},"min":{"1min":0.36599999999999999,"5min":0.314,"15min":0.314},"max":{"1min":10.217000000000001,"5min":10.217000000000001,"15min":10.217000000000001},"last":10.217000000000001},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.64100000000000001,"15min":0.56899999999999995},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.4409999999999998,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":3.4409999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:04:30 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60899999999999999,"15min":0.63500000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95599999999999996,"5min":1.581,"15min":1.581},"last":3.407},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.70899999999999996,"15min":0.80200000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88900000000000001,"5min":4.1859999999999999,"15min":4.1859999999999999},"last":8.2720000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:04:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.39200000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81999999999999995,"5min":1.381,"15min":1.381},"last":10.186999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61199999999999999,"15min":0.64600000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.98999999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":3.4670000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:04:16 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.73199999999999998,"15min":0.78600000000000003},"min":{"1min":0.51100000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.875,"5min":4.226,"15min":4.226},"last":8.2189999999999994},{"interface":"front","average":{"1min":0.628,"5min":0.623,"15min":0.63100000000000001},"min":{"1min":0.45300000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.82499999999999996,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":10.246}]}]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":169,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":331256,"kb_used_data":25968,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93413896,"statfs":{"total":95995035648,"available":95655829504,"internally_reserved":0,"allocated":26591232,"data_stored":25901100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.41399999999999998,"15min":0.371},"min":{"1min":0.23000000000000001,"5min":0.222,"15min":0.222},"max":{"1min":4.6150000000000002,"5min":4.6150000000000002,"15min":4.6150000000000002},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.42599999999999999,"15min":0.37},"min":{"1min":0.254,"5min":0.214,"15min":0.214},"max":{"1min":5.3609999999999998,"5min":5.3609999999999998,"15min":5.3609999999999998},"last":2.8069999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.42699999999999999,"15min":0.373},"min":{"1min":0.26500000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":4.6989999999999998,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":6.2510000000000003},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.46100000000000002,"15min":0.41499999999999998},"min":{"1min":0.29899999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.782,"5min":4.782,"15min":4.782},"last":4.5449999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:04:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.441,"15min":0.45400000000000001},"min":{"1min":0.29599999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.78900000000000003,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":4.4880000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.49099999999999999,"15min":0.48299999999999998},"min":{"1min":0.27600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.8399999999999999,"5min":4.8399999999999999,"15min":4.8399999999999999},"last":2.6920000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.628,"15min":0.61599999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.2880000000000003,"5min":5.2880000000000003,"15min":5.2880000000000003},"last":2.7330000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.59799999999999998,"15min":0.64000000000000001},"min":{"1min":0.373,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79700000000000004,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":4.6079999999999997}]},{"osd":5,"last update":"Mon Feb 19 21:04:28 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60799999999999998,"15min":0.64600000000000002},"min":{"1min":0.45000000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83999999999999997,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":4.569},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.67900000000000005,"15min":0.66600000000000004},"min":{"1min":0.48299999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":4.7320000000000002,"5min":4.7320000000000002,"15min":4.7320000000000002},"last":0.52900000000000003}]},{"osd":6,"last update":"Mon Feb 19 21:03:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.623,"15min":0.63},"min":{"1min":0.40000000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":2.7570000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61699999999999999,"15min":0.63100000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90000000000000002,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":4.5250000000000004}]},{"osd":7,"last update":"Mon Feb 19 21:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.65000000000000002,"15min":0.60799999999999998},"min":{"1min":0.437,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.758,"5min":4.758,"15min":4.758},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.67200000000000004,"15min":0.628},"min":{"1min":0.51100000000000001,"5min":0.379,"15min":0.379},"max":{"1min":5.2430000000000003,"5min":5.2430000000000003,"15min":5.2430000000000003},"last":2.7789999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182467,"num_pgs":222,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":405012,"kb_used_data":99724,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93340140,"statfs":{"total":95995035648,"available":95580303360,"internally_reserved":0,"allocated":102117376,"data_stored":101385860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:45 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":1.05,"15min":0.58599999999999997},"min":{"1min":0.26700000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":77.662000000000006,"5min":77.662000000000006,"15min":77.662000000000006},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.1679999999999999,"5min":0.52800000000000002,"15min":0.44800000000000001},"min":{"1min":0.23699999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":17.456,"5min":17.456,"15min":17.456},"last":1.5760000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:04:45 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":0.51900000000000002,"15min":0.441},"min":{"1min":0.23000000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":17.327999999999999,"5min":17.327999999999999,"15min":17.327999999999999},"last":7.3929999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.39600000000000002,"15min":0.40000000000000002},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.73799999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:04:45 2024","interfaces":[{"interface":"back","average":{"1min":2.9950000000000001,"5min":0.91800000000000004,"15min":0.58999999999999997},"min":{"1min":0.30599999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":56.890999999999998,"5min":56.890999999999998,"15min":56.890999999999998},"last":0.753},{"interface":"front","average":{"1min":2.4489999999999998,"5min":0.79900000000000004,"15min":0.54900000000000004},"min":{"1min":0.318,"5min":0.22,"15min":0.22},"max":{"1min":44.600999999999999,"5min":44.600999999999999,"15min":44.600999999999999},"last":7.4340000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:04:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54800000000000004,"15min":0.54700000000000004},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.84299999999999997,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55300000000000005,"15min":0.56499999999999995},"min":{"1min":0.41999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82499999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.56899999999999995}]},{"osd":5,"last update":"Mon Feb 19 21:04:25 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56999999999999995,"15min":0.57799999999999996},"min":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.56299999999999994,"15min":0.56399999999999995},"min":{"1min":0.45900000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.70899999999999996}]},{"osd":6,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":0.70999999999999996,"15min":0.61799999999999999},"min":{"1min":0.371,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":17.367000000000001,"5min":17.367000000000001,"15min":17.367000000000001},"last":0.498},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57899999999999996,"15min":0.59099999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79200000000000004,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.72499999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:04:13 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.56899999999999995,"15min":0.56299999999999994},"min":{"1min":0.43099999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.57999999999999996,"15min":0.57499999999999996},"min":{"1min":0.43099999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.52600000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986239,"num_pgs":175,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306644,"kb_used_data":1356,"kb_used_omap":0,"kb_used_meta":305280,"kb_avail":93438508,"statfs":{"total":95995035648,"available":95681032192,"internally_reserved":0,"allocated":1388544,"data_stored":726209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312606720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:04 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51300000000000001,"15min":0.54400000000000004},"min":{"1min":0.33700000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.505,"15min":0.52200000000000002},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.67400000000000004}]},{"osd":1,"last update":"Mon Feb 19 21:04:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.40999999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.78900000000000003,"5min":2.149,"15min":2.149},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.499,"15min":0.51800000000000002},"min":{"1min":0.35899999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.42299999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:04:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52500000000000002,"15min":0.52800000000000002},"min":{"1min":0.38600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.55400000000000005},"min":{"1min":0.36599999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70699999999999996,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.57999999999999996}]},{"osd":3,"last update":"Mon Feb 19 21:04:04 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.50900000000000001,"15min":0.52300000000000002},"min":{"1min":0.33800000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63700000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.308},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56499999999999995,"15min":0.56999999999999995},"min":{"1min":0.40600000000000003,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68300000000000005,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":0.51700000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:04:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.43099999999999999,"15min":0.432},"min":{"1min":0.33900000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.75700000000000001,"5min":2.0859999999999999,"15min":2.0859999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.45600000000000002,"15min":0.44},"min":{"1min":0.378,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.82699999999999996,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.36499999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.43099999999999999,"15min":0.38700000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63400000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.47799999999999998,"15min":0.438},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64600000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.34999999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:04:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.46800000000000003,"15min":0.44},"min":{"1min":0.38600000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.376},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.49299999999999999,"15min":0.46700000000000003},"min":{"1min":0.41399999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.38300000000000001}]}]},{"osd":5,"up_from":37,"seq":158913790010,"num_pgs":200,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":409096,"kb_used_data":99712,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93336056,"statfs":{"total":95995035648,"available":95576121344,"internally_reserved":0,"allocated":102105088,"data_stored":101388915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.47899999999999998,"15min":0.505},"min":{"1min":0.23899999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.66600000000000004,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":2.9420000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.52600000000000002,"15min":0.54000000000000004},"min":{"1min":0.27300000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.702,"5min":2.702,"15min":2.702},"last":2.8599999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.49399999999999999,"15min":0.51600000000000001},"min":{"1min":0.308,"5min":0.248,"15min":0.248},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.53300000000000003,"15min":0.53500000000000003},"min":{"1min":0.31900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":2.9430000000000001,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":3.7999999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.50700000000000001,"15min":0.53100000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":11.734999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.51200000000000001,"15min":0.52900000000000003},"min":{"1min":0.34499999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.71399999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.40200000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.36399999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.53500000000000003,"15min":0.54800000000000004},"min":{"1min":0.39200000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":1.113}]},{"osd":4,"last update":"Mon Feb 19 21:04:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.41799999999999998,"15min":0.39500000000000002},"min":{"1min":0.28000000000000003,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.39800000000000002},"min":{"1min":0.35999999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.35299999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:04:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45700000000000002,"15min":0.42799999999999999},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.47099999999999997,"15min":0.442},"min":{"1min":0.33300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.318}]},{"osd":7,"last update":"Mon Feb 19 21:04:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.50700000000000001,"15min":0.503},"min":{"1min":0.36799999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.38},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.51900000000000002,"15min":0.50900000000000001},"min":{"1min":0.38600000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.41599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50372608,"data_stored":50334515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50360320,"data_stored":50331754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25186304,"data_stored":25166874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":100712448,"data_stored":100664403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":100700160,"data_stored":100663429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":75554816,"data_stored":75500361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":50368512,"data_stored":50334560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":52,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":54,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:05:11.636 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-19T21:05:11.636 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool set .rgw.root pg_num 42 2024-02-19T21:05:12.235 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:05:12.235 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:05:12.235 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-02-19T21:05:12.235 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-02-19T21:05:12.235 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:05:12.254 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:10.101865+0000 mgr.y (mgr.14550) 170 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:12.254 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:11.006730+0000 mon.a (mon.0) 1062 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:05:12.254 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:11.014752+0000 mon.a (mon.0) 1063 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9"}]': finished 2024-02-19T21:05:12.254 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:11.014917+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-02-19T21:05:12.254 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:11.021854+0000 mon.b (mon.1) 195 : audit [INF] from='client.? v1:172.21.15.163:0/3840798168' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi163-83845-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:12.254 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:11.026552+0000 mon.a (mon.0) 1065 : audit [INF] from='client.38953 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi163-83845-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:12.255 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:10.101865+0000 mgr.y (mgr.14550) 170 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:12.255 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:11.006730+0000 mon.a (mon.0) 1062 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:05:12.255 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:11.014752+0000 mon.a (mon.0) 1063 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9"}]': finished 2024-02-19T21:05:12.255 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:11.014917+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-02-19T21:05:12.255 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:11.021854+0000 mon.b (mon.1) 195 : audit [INF] from='client.? v1:172.21.15.163:0/3840798168' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi163-83845-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:12.255 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:11.026552+0000 mon.a (mon.0) 1065 : audit [INF] from='client.38953 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi163-83845-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:12.314 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:10.101865+0000 mgr.y (mgr.14550) 170 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:12.314 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:11.006730+0000 mon.a (mon.0) 1062 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:05:12.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:11.014752+0000 mon.a (mon.0) 1063 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-9"}]': finished 2024-02-19T21:05:12.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:11.014917+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-02-19T21:05:12.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:11.021854+0000 mon.b (mon.1) 195 : audit [INF] from='client.? v1:172.21.15.163:0/3840798168' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi163-83845-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:12.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:11.026552+0000 mon.a (mon.0) 1065 : audit [INF] from='client.38953 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi163-83845-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:12.477 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:05:12.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:12.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:05:12.583 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:12.606 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:12.613 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:12.632 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:12.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:12.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:12.684 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:12.735 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:12.811 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:13.210 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:10.704681+0000 osd.0 (osd.0) 81 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:05:13.210 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:10.715430+0000 osd.0 (osd.0) 82 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:05:13.210 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:10.762806+0000 osd.1 (osd.1) 87 : cluster [DBG] 2.d scrub starts 2024-02-19T21:05:13.210 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:10.763883+0000 osd.1 (osd.1) 88 : cluster [DBG] 2.d scrub ok 2024-02-19T21:05:13.210 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:11.098898+0000 mgr.y (mgr.14550) 171 : cluster [DBG] pgmap v168: 396 pgs: 32 unknown, 364 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 op/s 2024-02-19T21:05:13.210 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:11.601790+0000 mon.c (mon.2) 106 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:13.210 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:11.804728+0000 osd.2 (osd.2) 67 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:05:13.210 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:11.826638+0000 osd.2 (osd.2) 68 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:05:13.210 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:12.009957+0000 mon.a (mon.0) 1066 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:05:13.211 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.077005+0000 mon.a (mon.0) 1067 : audit [INF] from='client.38953 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi163-83845-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:13.211 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:12.077278+0000 mon.a (mon.0) 1068 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-02-19T21:05:13.211 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.080335+0000 mon.b (mon.1) 196 : audit [INF] from='client.? v1:172.21.15.163:0/3505054706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi163-83854-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:13.211 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.090103+0000 mon.a (mon.0) 1069 : audit [INF] from='client.39709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi163-83854-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:13.211 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.102698+0000 mon.c (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.163:0/195719787' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:13.211 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.584726+0000 mon.a (mon.0) 1070 : audit [INF] from='client.44138 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:13.211 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.607163+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:13.211 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:10.704681+0000 osd.0 (osd.0) 81 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:05:13.211 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:10.715430+0000 osd.0 (osd.0) 82 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:05:13.211 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:10.762806+0000 osd.1 (osd.1) 87 : cluster [DBG] 2.d scrub starts 2024-02-19T21:05:13.211 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:10.763883+0000 osd.1 (osd.1) 88 : cluster [DBG] 2.d scrub ok 2024-02-19T21:05:13.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:11.098898+0000 mgr.y (mgr.14550) 171 : cluster [DBG] pgmap v168: 396 pgs: 32 unknown, 364 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 op/s 2024-02-19T21:05:13.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:11.601790+0000 mon.c (mon.2) 106 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:13.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:11.804728+0000 osd.2 (osd.2) 67 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:05:13.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:11.826638+0000 osd.2 (osd.2) 68 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:05:13.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:12.009957+0000 mon.a (mon.0) 1066 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:05:13.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.077005+0000 mon.a (mon.0) 1067 : audit [INF] from='client.38953 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi163-83845-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:13.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:12.077278+0000 mon.a (mon.0) 1068 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-02-19T21:05:13.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.080335+0000 mon.b (mon.1) 196 : audit [INF] from='client.? v1:172.21.15.163:0/3505054706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi163-83854-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:13.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.090103+0000 mon.a (mon.0) 1069 : audit [INF] from='client.39709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi163-83854-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:13.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.102698+0000 mon.c (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.163:0/195719787' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:13.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.584726+0000 mon.a (mon.0) 1070 : audit [INF] from='client.44138 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:13.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.607163+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:13.212 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.848358+0000 mon.b (mon.1) 197 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.0"}]: dispatch 2024-02-19T21:05:13.213 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.848656+0000 mgr.y (mgr.14550) 172 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.0"}]: dispatch 2024-02-19T21:05:13.213 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.849000+0000 mon.b (mon.1) 198 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.1"}]: dispatch 2024-02-19T21:05:13.213 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.849144+0000 mgr.y (mgr.14550) 173 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.1"}]: dispatch 2024-02-19T21:05:13.213 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.851487+0000 mon.b (mon.1) 199 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.2"}]: dispatch 2024-02-19T21:05:13.213 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.852314+0000 mgr.y (mgr.14550) 174 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.2"}]: dispatch 2024-02-19T21:05:13.213 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.854220+0000 mon.b (mon.1) 200 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.3"}]: dispatch 2024-02-19T21:05:13.213 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.854457+0000 mgr.y (mgr.14550) 175 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.3"}]: dispatch 2024-02-19T21:05:13.213 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.854876+0000 mon.b (mon.1) 201 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.4"}]: dispatch 2024-02-19T21:05:13.213 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.855019+0000 mgr.y (mgr.14550) 176 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.4"}]: dispatch 2024-02-19T21:05:13.213 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.855494+0000 mon.b (mon.1) 202 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.5"}]: dispatch 2024-02-19T21:05:13.213 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.855766+0000 mgr.y (mgr.14550) 177 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.5"}]: dispatch 2024-02-19T21:05:13.213 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.856172+0000 mon.b (mon.1) 203 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.6"}]: dispatch 2024-02-19T21:05:13.213 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.862018+0000 mon.b (mon.1) 204 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.7"}]: dispatch 2024-02-19T21:05:13.214 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.865915+0000 mon.b (mon.1) 205 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.8"}]: dispatch 2024-02-19T21:05:13.214 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.867824+0000 mon.b (mon.1) 206 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.9"}]: dispatch 2024-02-19T21:05:13.214 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.848358+0000 mon.b (mon.1) 197 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.0"}]: dispatch 2024-02-19T21:05:13.214 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.848656+0000 mgr.y (mgr.14550) 172 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.0"}]: dispatch 2024-02-19T21:05:13.214 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.849000+0000 mon.b (mon.1) 198 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.1"}]: dispatch 2024-02-19T21:05:13.214 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.849144+0000 mgr.y (mgr.14550) 173 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.1"}]: dispatch 2024-02-19T21:05:13.214 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.851487+0000 mon.b (mon.1) 199 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.2"}]: dispatch 2024-02-19T21:05:13.214 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.852314+0000 mgr.y (mgr.14550) 174 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.2"}]: dispatch 2024-02-19T21:05:13.214 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.854220+0000 mon.b (mon.1) 200 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.3"}]: dispatch 2024-02-19T21:05:13.214 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.854457+0000 mgr.y (mgr.14550) 175 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.3"}]: dispatch 2024-02-19T21:05:13.214 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.854876+0000 mon.b (mon.1) 201 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.4"}]: dispatch 2024-02-19T21:05:13.215 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.855019+0000 mgr.y (mgr.14550) 176 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.4"}]: dispatch 2024-02-19T21:05:13.215 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.855494+0000 mon.b (mon.1) 202 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.5"}]: dispatch 2024-02-19T21:05:13.215 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.855766+0000 mgr.y (mgr.14550) 177 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.5"}]: dispatch 2024-02-19T21:05:13.215 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.856172+0000 mon.b (mon.1) 203 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.6"}]: dispatch 2024-02-19T21:05:13.215 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.862018+0000 mon.b (mon.1) 204 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.7"}]: dispatch 2024-02-19T21:05:13.215 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.865915+0000 mon.b (mon.1) 205 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.8"}]: dispatch 2024-02-19T21:05:13.215 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.867824+0000 mon.b (mon.1) 206 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.9"}]: dispatch 2024-02-19T21:05:13.239 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-19T21:05:13.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:10.704681+0000 osd.0 (osd.0) 81 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:05:13.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:10.715430+0000 osd.0 (osd.0) 82 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:05:13.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:10.762806+0000 osd.1 (osd.1) 87 : cluster [DBG] 2.d scrub starts 2024-02-19T21:05:13.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:10.763883+0000 osd.1 (osd.1) 88 : cluster [DBG] 2.d scrub ok 2024-02-19T21:05:13.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:11.098898+0000 mgr.y (mgr.14550) 171 : cluster [DBG] pgmap v168: 396 pgs: 32 unknown, 364 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 op/s 2024-02-19T21:05:13.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:11.601790+0000 mon.c (mon.2) 106 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:13.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:11.804728+0000 osd.2 (osd.2) 67 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:05:13.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:11.826638+0000 osd.2 (osd.2) 68 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:05:13.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:12.009957+0000 mon.a (mon.0) 1066 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:05:13.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.077005+0000 mon.a (mon.0) 1067 : audit [INF] from='client.38953 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi163-83845-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:13.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:12.077278+0000 mon.a (mon.0) 1068 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-02-19T21:05:13.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.080335+0000 mon.b (mon.1) 196 : audit [INF] from='client.? v1:172.21.15.163:0/3505054706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi163-83854-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:13.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.090103+0000 mon.a (mon.0) 1069 : audit [INF] from='client.39709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi163-83854-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:13.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.102698+0000 mon.c (mon.2) 107 : audit [INF] from='client.? v1:172.21.15.163:0/195719787' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:13.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.584726+0000 mon.a (mon.0) 1070 : audit [INF] from='client.44138 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:13.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.607163+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:13.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.848358+0000 mon.b (mon.1) 197 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.0"}]: dispatch 2024-02-19T21:05:13.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.848656+0000 mgr.y (mgr.14550) 172 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.0"}]: dispatch 2024-02-19T21:05:13.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.849000+0000 mon.b (mon.1) 198 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.1"}]: dispatch 2024-02-19T21:05:13.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.849144+0000 mgr.y (mgr.14550) 173 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.1"}]: dispatch 2024-02-19T21:05:13.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.851487+0000 mon.b (mon.1) 199 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.2"}]: dispatch 2024-02-19T21:05:13.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.852314+0000 mgr.y (mgr.14550) 174 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.2"}]: dispatch 2024-02-19T21:05:13.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.854220+0000 mon.b (mon.1) 200 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.3"}]: dispatch 2024-02-19T21:05:13.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.854457+0000 mgr.y (mgr.14550) 175 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.3"}]: dispatch 2024-02-19T21:05:13.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.854876+0000 mon.b (mon.1) 201 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.4"}]: dispatch 2024-02-19T21:05:13.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.855019+0000 mgr.y (mgr.14550) 176 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.4"}]: dispatch 2024-02-19T21:05:13.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.855494+0000 mon.b (mon.1) 202 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.5"}]: dispatch 2024-02-19T21:05:13.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.855766+0000 mgr.y (mgr.14550) 177 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.5"}]: dispatch 2024-02-19T21:05:13.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.856172+0000 mon.b (mon.1) 203 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.6"}]: dispatch 2024-02-19T21:05:13.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.862018+0000 mon.b (mon.1) 204 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.7"}]: dispatch 2024-02-19T21:05:13.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.865915+0000 mon.b (mon.1) 205 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.8"}]: dispatch 2024-02-19T21:05:13.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.867824+0000 mon.b (mon.1) 206 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.9"}]: dispatch 2024-02-19T21:05:13.513 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:05:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:05:13] "GET /metrics HTTP/1.1" 200 206117 "" "Prometheus/2.33.4" 2024-02-19T21:05:14.187 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:05:14.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:12.316955+0000 osd.4 (osd.4) 77 : cluster [DBG] 5.1 deep-scrub starts 2024-02-19T21:05:14.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:12.318385+0000 osd.4 (osd.4) 78 : cluster [DBG] 5.1 deep-scrub ok 2024-02-19T21:05:14.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.859500+0000 mgr.y (mgr.14550) 178 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.6"}]: dispatch 2024-02-19T21:05:14.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.864403+0000 mgr.y (mgr.14550) 179 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.7"}]: dispatch 2024-02-19T21:05:14.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.866095+0000 mgr.y (mgr.14550) 180 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.8"}]: dispatch 2024-02-19T21:05:14.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:12.868058+0000 mgr.y (mgr.14550) 181 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.9"}]: dispatch 2024-02-19T21:05:14.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:13.080768+0000 mon.a (mon.0) 1071 : audit [INF] from='client.39709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi163-83854-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:14.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:13.080963+0000 mon.a (mon.0) 1072 : audit [INF] from='client.44138 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:14.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:13.081153+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-02-19T21:05:14.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:13.100751+0000 mon.b (mon.1) 207 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:14.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:13.114426+0000 mgr.y (mgr.14550) 182 : cluster [DBG] pgmap v171: 428 pgs: 96 unknown, 332 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-02-19T21:05:14.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:13.588765+0000 mon.a (mon.0) 1074 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:14.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:13.621471+0000 mon.c (mon.2) 109 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:14.508 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:12.316955+0000 osd.4 (osd.4) 77 : cluster [DBG] 5.1 deep-scrub starts 2024-02-19T21:05:14.508 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:12.318385+0000 osd.4 (osd.4) 78 : cluster [DBG] 5.1 deep-scrub ok 2024-02-19T21:05:14.508 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.859500+0000 mgr.y (mgr.14550) 178 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.6"}]: dispatch 2024-02-19T21:05:14.508 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.864403+0000 mgr.y (mgr.14550) 179 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.7"}]: dispatch 2024-02-19T21:05:14.508 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.866095+0000 mgr.y (mgr.14550) 180 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.8"}]: dispatch 2024-02-19T21:05:14.508 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:12.868058+0000 mgr.y (mgr.14550) 181 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.9"}]: dispatch 2024-02-19T21:05:14.508 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:13.080768+0000 mon.a (mon.0) 1071 : audit [INF] from='client.39709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi163-83854-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:14.508 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:13.080963+0000 mon.a (mon.0) 1072 : audit [INF] from='client.44138 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:14.509 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:13.081153+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-02-19T21:05:14.509 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:13.100751+0000 mon.b (mon.1) 207 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:14.509 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:13.114426+0000 mgr.y (mgr.14550) 182 : cluster [DBG] pgmap v171: 428 pgs: 96 unknown, 332 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-02-19T21:05:14.509 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:13.588765+0000 mon.a (mon.0) 1074 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:14.509 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:13.621471+0000 mon.c (mon.2) 109 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:14.546 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:14.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:14.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:14.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:12.316955+0000 osd.4 (osd.4) 77 : cluster [DBG] 5.1 deep-scrub starts 2024-02-19T21:05:14.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:12.318385+0000 osd.4 (osd.4) 78 : cluster [DBG] 5.1 deep-scrub ok 2024-02-19T21:05:14.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.859500+0000 mgr.y (mgr.14550) 178 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.6"}]: dispatch 2024-02-19T21:05:14.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.864403+0000 mgr.y (mgr.14550) 179 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.7"}]: dispatch 2024-02-19T21:05:14.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.866095+0000 mgr.y (mgr.14550) 180 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.8"}]: dispatch 2024-02-19T21:05:14.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:12.868058+0000 mgr.y (mgr.14550) 181 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "28.9"}]: dispatch 2024-02-19T21:05:14.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:13.080768+0000 mon.a (mon.0) 1071 : audit [INF] from='client.39709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi163-83854-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:14.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:13.080963+0000 mon.a (mon.0) 1072 : audit [INF] from='client.44138 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83936-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:14.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:13.081153+0000 mon.a (mon.0) 1073 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-02-19T21:05:14.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:13.100751+0000 mon.b (mon.1) 207 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:14.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:13.114426+0000 mgr.y (mgr.14550) 182 : cluster [DBG] pgmap v171: 428 pgs: 96 unknown, 332 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-02-19T21:05:14.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:13.588765+0000 mon.a (mon.0) 1074 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:14.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:13.621471+0000 mon.c (mon.2) 109 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:14.652 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:14.835 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:15.150 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:15.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:15.258 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:15.284 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:15.305 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:15.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:15.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:15.409 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:05:15.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:13.738525+0000 osd.0 (osd.0) 83 : cluster [DBG] 28.2 deep-scrub starts 2024-02-19T21:05:15.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:13.790829+0000 osd.0 (osd.0) 84 : cluster [DBG] 28.2 deep-scrub ok 2024-02-19T21:05:15.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:13.801352+0000 osd.6 (osd.6) 65 : cluster [DBG] 28.0 deep-scrub starts 2024-02-19T21:05:15.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:13.806005+0000 osd.6 (osd.6) 66 : cluster [DBG] 28.0 deep-scrub ok 2024-02-19T21:05:15.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:14.166262+0000 mon.a (mon.0) 1075 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:15.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:14.166436+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-02-19T21:05:15.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:14.176735+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.163:0/195719787' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache", "force_nonempty":""}]: dispatch 2024-02-19T21:05:15.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:14.185885+0000 mon.b (mon.1) 208 : audit [INF] from='client.? v1:172.21.15.163:0/2273804006' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi163-83845-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:15.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:14.195556+0000 mon.a (mon.0) 1077 : audit [INF] from='client.44138 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache", "force_nonempty":""}]: dispatch 2024-02-19T21:05:15.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:14.236531+0000 mon.a (mon.0) 1078 : audit [INF] from='client.41221 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi163-83845-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:15.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:14.622823+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:15.452 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:15.168520+0000 mon.a (mon.0) 1079 : audit [INF] from='client.44138 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache", "force_nonempty":""}]': finished 2024-02-19T21:05:15.452 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:15.168712+0000 mon.a (mon.0) 1080 : audit [INF] from='client.41221 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi163-83845-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:15.452 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:15.168854+0000 mon.a (mon.0) 1081 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-02-19T21:05:15.452 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:15.179412+0000 mon.b (mon.1) 209 : audit [INF] from='client.? v1:172.21.15.163:0/2527193429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi163-83854-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:15.452 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:15.201632+0000 mon.a (mon.0) 1082 : audit [INF] from='client.42214 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi163-83854-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:15.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:13.738525+0000 osd.0 (osd.0) 83 : cluster [DBG] 28.2 deep-scrub starts 2024-02-19T21:05:15.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:13.790829+0000 osd.0 (osd.0) 84 : cluster [DBG] 28.2 deep-scrub ok 2024-02-19T21:05:15.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:13.801352+0000 osd.6 (osd.6) 65 : cluster [DBG] 28.0 deep-scrub starts 2024-02-19T21:05:15.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:13.806005+0000 osd.6 (osd.6) 66 : cluster [DBG] 28.0 deep-scrub ok 2024-02-19T21:05:15.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:14.166262+0000 mon.a (mon.0) 1075 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:15.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:14.166436+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-02-19T21:05:15.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:14.176735+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.163:0/195719787' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache", "force_nonempty":""}]: dispatch 2024-02-19T21:05:15.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:14.185885+0000 mon.b (mon.1) 208 : audit [INF] from='client.? v1:172.21.15.163:0/2273804006' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi163-83845-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:15.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:14.195556+0000 mon.a (mon.0) 1077 : audit [INF] from='client.44138 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache", "force_nonempty":""}]: dispatch 2024-02-19T21:05:15.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:14.236531+0000 mon.a (mon.0) 1078 : audit [INF] from='client.41221 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi163-83845-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:15.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:14.622823+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:15.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:15.168520+0000 mon.a (mon.0) 1079 : audit [INF] from='client.44138 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache", "force_nonempty":""}]': finished 2024-02-19T21:05:15.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:15.168712+0000 mon.a (mon.0) 1080 : audit [INF] from='client.41221 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi163-83845-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:15.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:15.168854+0000 mon.a (mon.0) 1081 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-02-19T21:05:15.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:15.179412+0000 mon.b (mon.1) 209 : audit [INF] from='client.? v1:172.21.15.163:0/2527193429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi163-83854-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:15.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:15.201632+0000 mon.a (mon.0) 1082 : audit [INF] from='client.42214 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi163-83854-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:15.454 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:15.497 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:05:15.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:15.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:05:15.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:13.738525+0000 osd.0 (osd.0) 83 : cluster [DBG] 28.2 deep-scrub starts 2024-02-19T21:05:15.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:13.790829+0000 osd.0 (osd.0) 84 : cluster [DBG] 28.2 deep-scrub ok 2024-02-19T21:05:15.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:13.801352+0000 osd.6 (osd.6) 65 : cluster [DBG] 28.0 deep-scrub starts 2024-02-19T21:05:15.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:13.806005+0000 osd.6 (osd.6) 66 : cluster [DBG] 28.0 deep-scrub ok 2024-02-19T21:05:15.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:14.166262+0000 mon.a (mon.0) 1075 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:15.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:14.166436+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-02-19T21:05:15.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:14.176735+0000 mon.c (mon.2) 110 : audit [INF] from='client.? v1:172.21.15.163:0/195719787' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache", "force_nonempty":""}]: dispatch 2024-02-19T21:05:15.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:14.185885+0000 mon.b (mon.1) 208 : audit [INF] from='client.? v1:172.21.15.163:0/2273804006' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi163-83845-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:15.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:14.195556+0000 mon.a (mon.0) 1077 : audit [INF] from='client.44138 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache", "force_nonempty":""}]: dispatch 2024-02-19T21:05:15.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:14.236531+0000 mon.a (mon.0) 1078 : audit [INF] from='client.41221 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi163-83845-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:15.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:14.622823+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:15.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:15.168520+0000 mon.a (mon.0) 1079 : audit [INF] from='client.44138 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache", "force_nonempty":""}]': finished 2024-02-19T21:05:15.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:15.168712+0000 mon.a (mon.0) 1080 : audit [INF] from='client.41221 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi163-83845-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:15.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:15.168854+0000 mon.a (mon.0) 1081 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-02-19T21:05:15.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:15.179412+0000 mon.b (mon.1) 209 : audit [INF] from='client.? v1:172.21.15.163:0/2527193429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi163-83854-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:15.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:15.201632+0000 mon.a (mon.0) 1082 : audit [INF] from='client.42214 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi163-83854-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:15.601 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:15.627 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:15.645 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:15.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:15.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:15.749 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:05:15.775 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:15.794 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:05:15.797 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:15.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:05:15.898 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:05:16.032 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:16.095 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:05:16.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:16.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:05:16.295 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:05:16.321 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:16.342 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:05:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:16.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:05:16.445 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:16.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.475+0000 7f959c28d700 1 -- 172.21.15.163:0/1370909074 >> v1:172.21.15.163:6790/0 conn(0x7f95941031b0 legacy=0x7f9594105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:16.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.475+0000 7f959c28d700 1 -- 172.21.15.163:0/1370909074 shutdown_connections 2024-02-19T21:05:16.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.475+0000 7f959c28d700 1 -- 172.21.15.163:0/1370909074 >> 172.21.15.163:0/1370909074 conn(0x7f95940f2470 msgr2=0x7f95940f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:16.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.475+0000 7f959c28d700 1 -- 172.21.15.163:0/1370909074 shutdown_connections 2024-02-19T21:05:16.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.475+0000 7f959c28d700 1 -- 172.21.15.163:0/1370909074 wait complete. 2024-02-19T21:05:16.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.475+0000 7f959c28d700 1 librados: starting msgr at 2024-02-19T21:05:16.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.475+0000 7f959c28d700 1 librados: starting objecter 2024-02-19T21:05:16.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.475+0000 7f959c28d700 1 Processor -- start 2024-02-19T21:05:16.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.476+0000 7f959c28d700 1 -- start start 2024-02-19T21:05:16.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.476+0000 7f959c28d700 1 librados: setting wanted keys 2024-02-19T21:05:16.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.476+0000 7f959c28d700 1 librados: calling monclient init 2024-02-19T21:05:16.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.476+0000 7f959c28d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9594115160 con 0x7f95940fc870 2024-02-19T21:05:16.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.476+0000 7f959c28d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9594116340 con 0x7f95940ff670 2024-02-19T21:05:16.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.476+0000 7f959c28d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9594116530 con 0x7f95941031b0 2024-02-19T21:05:16.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.476+0000 7f9599828700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f95940ff670 0x7f95940f8b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:59082/0 (socket says 172.21.15.163:59082) 2024-02-19T21:05:16.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.476+0000 7f9599828700 1 -- 172.21.15.163:0/2680502209 learned_addr learned my addr 172.21.15.163:0/2680502209 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:16.471 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:16.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 10 client.?.objecter ms_handle_connect 0x7f95940ff670 2024-02-19T21:05:16.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:16.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 10 client.?.objecter ms_handle_connect 0x7f95940fc870 2024-02-19T21:05:16.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:16.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 10 client.?.objecter ms_handle_connect 0x7f95941031b0 2024-02-19T21:05:16.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:16.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915730201 0 0) 0x7f9594116340 con 0x7f95940ff670 2024-02-19T21:05:16.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9580002220 con 0x7f95940ff670 2024-02-19T21:05:16.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700562824 0 0) 0x7f9594116530 con 0x7f95941031b0 2024-02-19T21:05:16.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9594116340 con 0x7f95941031b0 2024-02-19T21:05:16.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1060108695 0 0) 0x7f9580002220 con 0x7f95940ff670 2024-02-19T21:05:16.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9594116530 con 0x7f95940ff670 2024-02-19T21:05:16.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9584001820 con 0x7f95940ff670 2024-02-19T21:05:16.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1497815289 0 0) 0x7f9594116340 con 0x7f95941031b0 2024-02-19T21:05:16.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9580002220 con 0x7f95941031b0 2024-02-19T21:05:16.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f958c001b70 con 0x7f95941031b0 2024-02-19T21:05:16.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 618368966 0 0) 0x7f9580002220 con 0x7f95941031b0 2024-02-19T21:05:16.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 >> v1:172.21.15.193:6789/0 conn(0x7f95940ff670 legacy=0x7f95940f8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:16.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 >> v1:172.21.15.163:6789/0 conn(0x7f95940fc870 legacy=0x7f95940f8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:16.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9594116720 con 0x7f95941031b0 2024-02-19T21:05:16.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f959c28d700 1 -- 172.21.15.163:0/2680502209 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95941169d0 con 0x7f95941031b0 2024-02-19T21:05:16.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f959c28d700 10 client.46559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:16.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f959c28d700 1 -- 172.21.15.163:0/2680502209 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9594116e40 con 0x7f95941031b0 2024-02-19T21:05:16.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.482+0000 7f959c28d700 1 librados: init done 2024-02-19T21:05:16.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.483+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f958c001e10 con 0x7f95941031b0 2024-02-19T21:05:16.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.483+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f958c003990 con 0x7f95941031b0 2024-02-19T21:05:16.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.484+0000 7f959c28d700 1 -- 172.21.15.163:0/2680502209 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9578004f20 con 0x7f95941031b0 2024-02-19T21:05:16.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.485+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f958c002310 con 0x7f95941031b0 2024-02-19T21:05:16.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.485+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(136..136 src has 43..136) v4 ==== 10784+0+0 (unknown 3285936802 0 0) 0x7f958c077330 con 0x7f95941031b0 2024-02-19T21:05:16.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.485+0000 7f958b779700 10 client.46559.objecter ms_dispatch 0x7f9594001550 osd_map(136..136 src has 43..136) v4 2024-02-19T21:05:16.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.485+0000 7f958b779700 3 client.46559.objecter handle_osd_map got epochs [136,136] > 0 2024-02-19T21:05:16.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.485+0000 7f958b779700 3 client.46559.objecter handle_osd_map decoding full epoch 136 2024-02-19T21:05:16.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.485+0000 7f958b779700 20 client.46559.objecter dump_active .. 0 homeless 2024-02-19T21:05:16.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.488+0000 7f958b779700 10 client.46559.objecter ms_handle_connect 0x7f9580063c60 2024-02-19T21:05:16.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.491+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f958c049d40 con 0x7f95941031b0 2024-02-19T21:05:16.490 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:16.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:15.115984+0000 mgr.y (mgr.14550) 183 : cluster [DBG] pgmap v173: 460 pgs: 64 creating+peering, 2 active+clean+snaptrim, 394 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-02-19T21:05:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:15.234701+0000 mon.c (mon.2) 112 : audit [INF] from='client.? v1:172.21.15.163:0/195719787' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83936-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:15.261444+0000 mon.a (mon.0) 1083 : audit [INF] from='client.44138 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83936-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:15.306076+0000 mon.b (mon.1) 210 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:15.306711+0000 mon.a (mon.0) 1084 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:15.629331+0000 mon.c (mon.2) 113 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:16.259835+0000 mon.a (mon.0) 1085 : audit [INF] from='client.42214 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi163-83854-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:16.259978+0000 mon.a (mon.0) 1086 : audit [INF] from='client.44138 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83936-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:16.260048+0000 mon.a (mon.0) 1087 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:05:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:16.260106+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-02-19T21:05:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:16.266535+0000 mon.b (mon.1) 211 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:16.275798+0000 mon.b (mon.1) 212 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-11"}]: dispatch 2024-02-19T21:05:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:16.276984+0000 mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.163:0/195719787' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache"}]: dispatch 2024-02-19T21:05:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:16.277905+0000 mon.a (mon.0) 1089 : audit [INF] from='client.44138 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache"}]: dispatch 2024-02-19T21:05:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:16.278143+0000 mon.a (mon.0) 1090 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:16.278346+0000 mon.a (mon.0) 1091 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-11"}]: dispatch 2024-02-19T21:05:16.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:15.115984+0000 mgr.y (mgr.14550) 183 : cluster [DBG] pgmap v173: 460 pgs: 64 creating+peering, 2 active+clean+snaptrim, 394 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-02-19T21:05:16.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:15.234701+0000 mon.c (mon.2) 112 : audit [INF] from='client.? v1:172.21.15.163:0/195719787' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83936-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:16.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:15.261444+0000 mon.a (mon.0) 1083 : audit [INF] from='client.44138 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83936-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:16.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:15.306076+0000 mon.b (mon.1) 210 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:16.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:15.306711+0000 mon.a (mon.0) 1084 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:16.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:15.629331+0000 mon.c (mon.2) 113 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:16.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:16.259835+0000 mon.a (mon.0) 1085 : audit [INF] from='client.42214 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi163-83854-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:16.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:16.259978+0000 mon.a (mon.0) 1086 : audit [INF] from='client.44138 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83936-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:16.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:16.260048+0000 mon.a (mon.0) 1087 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:05:16.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:16.260106+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-02-19T21:05:16.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:16.266535+0000 mon.b (mon.1) 211 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:16.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:16.275798+0000 mon.b (mon.1) 212 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-11"}]: dispatch 2024-02-19T21:05:16.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:16.276984+0000 mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.163:0/195719787' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache"}]: dispatch 2024-02-19T21:05:16.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:16.277905+0000 mon.a (mon.0) 1089 : audit [INF] from='client.44138 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache"}]: dispatch 2024-02-19T21:05:16.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:16.278143+0000 mon.a (mon.0) 1090 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:16.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:16.278346+0000 mon.a (mon.0) 1091 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-11"}]: dispatch 2024-02-19T21:05:16.594 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:16.618 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:15.115984+0000 mgr.y (mgr.14550) 183 : cluster [DBG] pgmap v173: 460 pgs: 64 creating+peering, 2 active+clean+snaptrim, 394 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 6 op/s 2024-02-19T21:05:16.618 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:15.234701+0000 mon.c (mon.2) 112 : audit [INF] from='client.? v1:172.21.15.163:0/195719787' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83936-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:16.618 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:15.261444+0000 mon.a (mon.0) 1083 : audit [INF] from='client.44138 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83936-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:16.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:15.306076+0000 mon.b (mon.1) 210 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:16.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:15.306711+0000 mon.a (mon.0) 1084 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:16.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:15.629331+0000 mon.c (mon.2) 113 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:16.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:16.259835+0000 mon.a (mon.0) 1085 : audit [INF] from='client.42214 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi163-83854-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:16.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:16.259978+0000 mon.a (mon.0) 1086 : audit [INF] from='client.44138 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83936-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:16.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:16.260048+0000 mon.a (mon.0) 1087 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:05:16.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:16.260106+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-02-19T21:05:16.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:16.266535+0000 mon.b (mon.1) 211 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:16.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:16.275798+0000 mon.b (mon.1) 212 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-11"}]: dispatch 2024-02-19T21:05:16.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:16.276984+0000 mon.c (mon.2) 114 : audit [INF] from='client.? v1:172.21.15.163:0/195719787' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache"}]: dispatch 2024-02-19T21:05:16.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:16.277905+0000 mon.a (mon.0) 1089 : audit [INF] from='client.44138 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache"}]: dispatch 2024-02-19T21:05:16.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:16.278143+0000 mon.a (mon.0) 1090 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:16.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:16.278346+0000 mon.a (mon.0) 1091 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-11"}]: dispatch 2024-02-19T21:05:16.620 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:16.640 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:16.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:16.744 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:16.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.760+0000 7fc39c5bd700 1 -- 172.21.15.163:0/4053020501 >> v1:172.21.15.193:6789/0 conn(0x7fc3940f94d0 legacy=0x7fc3941029a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:16.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.761+0000 7fc39c5bd700 1 -- 172.21.15.163:0/4053020501 shutdown_connections 2024-02-19T21:05:16.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.761+0000 7fc39c5bd700 1 -- 172.21.15.163:0/4053020501 >> 172.21.15.163:0/4053020501 conn(0x7fc3940f2470 msgr2=0x7fc3940f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:16.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.761+0000 7fc39c5bd700 1 -- 172.21.15.163:0/4053020501 shutdown_connections 2024-02-19T21:05:16.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.761+0000 7fc39c5bd700 1 -- 172.21.15.163:0/4053020501 wait complete. 2024-02-19T21:05:16.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.762+0000 7fc39c5bd700 1 librados: starting msgr at 2024-02-19T21:05:16.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.762+0000 7fc39c5bd700 1 librados: starting objecter 2024-02-19T21:05:16.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.762+0000 7fc39c5bd700 1 Processor -- start 2024-02-19T21:05:16.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.762+0000 7fc39c5bd700 1 -- start start 2024-02-19T21:05:16.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.762+0000 7fc39c5bd700 1 librados: setting wanted keys 2024-02-19T21:05:16.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.762+0000 7fc39c5bd700 1 librados: calling monclient init 2024-02-19T21:05:16.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.762+0000 7fc39c5bd700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc394126250 con 0x7fc3941042b0 2024-02-19T21:05:16.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.762+0000 7fc39c5bd700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc394127430 con 0x7fc3940f8920 2024-02-19T21:05:16.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.762+0000 7fc39c5bd700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc394128630 con 0x7fc3940f94d0 2024-02-19T21:05:16.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.762+0000 7fc399b58700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fc3940f94d0 0x7fc39410fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:43882/0 (socket says 172.21.15.163:43882) 2024-02-19T21:05:16.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc399b58700 1 -- 172.21.15.163:0/2597427537 learned_addr learned my addr 172.21.15.163:0/2597427537 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:16.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc39a359700 1 --1- 172.21.15.163:0/2597427537 >> v1:172.21.15.193:6789/0 conn(0x7fc3940f8920 0x7fc39410f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:59988/0 (socket says 172.21.15.163:59988) 2024-02-19T21:05:16.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc38b7fe700 10 client.?.objecter ms_handle_connect 0x7fc3940f8920 2024-02-19T21:05:16.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc38b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:16.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc38b7fe700 10 client.?.objecter ms_handle_connect 0x7fc3940f94d0 2024-02-19T21:05:16.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc38b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:16.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079720837 0 0) 0x7fc394127430 con 0x7fc3940f8920 2024-02-19T21:05:16.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc380002220 con 0x7fc3940f8920 2024-02-19T21:05:16.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426612305 0 0) 0x7fc394128630 con 0x7fc3940f94d0 2024-02-19T21:05:16.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc394127430 con 0x7fc3940f94d0 2024-02-19T21:05:16.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 219636529 0 0) 0x7fc380002220 con 0x7fc3940f8920 2024-02-19T21:05:16.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc394128630 con 0x7fc3940f8920 2024-02-19T21:05:16.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc3900018c0 con 0x7fc3940f8920 2024-02-19T21:05:16.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4130288018 0 0) 0x7fc394128630 con 0x7fc3940f8920 2024-02-19T21:05:16.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 >> v1:172.21.15.163:6790/0 conn(0x7fc3940f94d0 legacy=0x7fc39410fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:16.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 >> v1:172.21.15.163:6789/0 conn(0x7fc3941042b0 legacy=0x7fc394102990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:16.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.763+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc394129830 con 0x7fc3940f8920 2024-02-19T21:05:16.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.764+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fc390002420 con 0x7fc3940f8920 2024-02-19T21:05:16.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.764+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc390005b20 con 0x7fc3940f8920 2024-02-19T21:05:16.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.766+0000 7fc39c5bd700 1 -- 172.21.15.163:0/2597427537 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc394128820 con 0x7fc3940f8920 2024-02-19T21:05:16.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.766+0000 7fc39c5bd700 10 client.44038.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:16.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.766+0000 7fc39c5bd700 1 -- 172.21.15.163:0/2597427537 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc394128c90 con 0x7fc3940f8920 2024-02-19T21:05:16.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.766+0000 7fc39c5bd700 1 librados: init done 2024-02-19T21:05:16.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.767+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fc390001f80 con 0x7fc3940f8920 2024-02-19T21:05:16.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.767+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(136..136 src has 43..136) v4 ==== 10784+0+0 (unknown 3285936802 0 0) 0x7fc390079790 con 0x7fc3940f8920 2024-02-19T21:05:16.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.767+0000 7fc38b7fe700 10 client.44038.objecter ms_dispatch 0x7fc394001550 osd_map(136..136 src has 43..136) v4 2024-02-19T21:05:16.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.767+0000 7fc38b7fe700 3 client.44038.objecter handle_osd_map got epochs [136,136] > 0 2024-02-19T21:05:16.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.767+0000 7fc38b7fe700 3 client.44038.objecter handle_osd_map decoding full epoch 136 2024-02-19T21:05:16.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.768+0000 7fc38b7fe700 20 client.44038.objecter dump_active .. 0 homeless 2024-02-19T21:05:16.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.768+0000 7fc38b7fe700 10 client.44038.objecter ms_handle_connect 0x7fc380063ba0 2024-02-19T21:05:16.765 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:16.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.772+0000 7fc39c5bd700 1 -- 172.21.15.163:0/2597427537 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc39405f700 con 0x7fc3940f8920 2024-02-19T21:05:16.771 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:16.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.783+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc39004c0e0 con 0x7fc3940f8920 2024-02-19T21:05:16.790 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:16.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:16.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:16.834 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:16.894 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:05:16.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:16.912+0000 7f959c28d700 1 -- 172.21.15.163:0/2680502209 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f95780051d0 con 0x7f95941031b0 2024-02-19T21:05:16.996 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:17.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.089+0000 7fc39c5bd700 1 -- 172.21.15.163:0/2597427537 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"} v 0) v1 -- 0x7fc394108dd0 con 0x7fc3940f8920 2024-02-19T21:05:17.122 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:05:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:17.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:05:17.228 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:05:17.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.322+0000 7f958b779700 1 -- 172.21.15.163:0/2680502209 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v137) v1 ==== 87+0+0 (unknown 3483337043 0 0) 0x7f958c0481b0 con 0x7f95941031b0 2024-02-19T21:05:17.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.322+0000 7fc38b7fe700 1 -- 172.21.15.163:0/2597427537 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]=0 set pool 2 pg_num to 42 v137) v1 ==== 134+0+0 (unknown 740549385 0 0) 0x7fc39004a4f0 con 0x7fc3940f8920 2024-02-19T21:05:17.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.324+0000 7f959c28d700 10 librados: watch_flush enter 2024-02-19T21:05:17.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.324+0000 7f959c28d700 10 librados: watch_flush exit 2024-02-19T21:05:17.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.325+0000 7f959c28d700 20 client.46559.objecter shutdown clearing up homeless session... 2024-02-19T21:05:17.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.325+0000 7f959c28d700 10 client.46559.objecter successfully canceled tick 2024-02-19T21:05:17.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.325+0000 7f959c28d700 1 -- 172.21.15.163:0/2680502209 >> v1:172.21.15.163:6800/1552514681 conn(0x7f9580063c60 legacy=0x7f9580066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:17.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.325+0000 7f959c28d700 1 -- 172.21.15.163:0/2680502209 >> v1:172.21.15.163:6790/0 conn(0x7f95941031b0 legacy=0x7f9594113820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:17.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.326+0000 7fc39c5bd700 10 librados: watch_flush enter 2024-02-19T21:05:17.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.333+0000 7fc39c5bd700 10 librados: watch_flush exit 2024-02-19T21:05:17.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.333+0000 7fc39c5bd700 20 client.44038.objecter shutdown clearing up homeless session... 2024-02-19T21:05:17.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.333+0000 7fc39c5bd700 10 client.44038.objecter successfully canceled tick 2024-02-19T21:05:17.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.325+0000 7f959c28d700 1 -- 172.21.15.163:0/2680502209 shutdown_connections 2024-02-19T21:05:17.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.325+0000 7f959c28d700 1 -- 172.21.15.163:0/2680502209 >> 172.21.15.163:0/2680502209 conn(0x7f95940f2470 msgr2=0x7f95941067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:17.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.347+0000 7f959c28d700 1 -- 172.21.15.163:0/2680502209 shutdown_connections 2024-02-19T21:05:17.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.347+0000 7fc39c5bd700 1 -- 172.21.15.163:0/2597427537 >> v1:172.21.15.163:6800/1552514681 conn(0x7fc380063ba0 legacy=0x7fc380066060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:17.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.347+0000 7fc39c5bd700 1 -- 172.21.15.163:0/2597427537 >> v1:172.21.15.193:6789/0 conn(0x7fc3940f8920 legacy=0x7fc39410f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:17.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.349+0000 7f959c28d700 1 -- 172.21.15.163:0/2680502209 wait complete. 2024-02-19T21:05:17.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.349+0000 7f959c28d700 1 librados: shutdown 2024-02-19T21:05:17.345 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:17.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.352+0000 7fc39c5bd700 1 -- 172.21.15.163:0/2597427537 shutdown_connections 2024-02-19T21:05:17.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.352+0000 7fc39c5bd700 1 -- 172.21.15.163:0/2597427537 >> 172.21.15.163:0/2597427537 conn(0x7fc3940f2470 msgr2=0x7fc3941077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:17.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.359+0000 7fc39c5bd700 1 -- 172.21.15.163:0/2597427537 shutdown_connections 2024-02-19T21:05:17.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.359+0000 7fc39c5bd700 1 -- 172.21.15.163:0/2597427537 wait complete. 2024-02-19T21:05:17.354 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:05:17.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:17.359+0000 7fc39c5bd700 1 librados: shutdown 2024-02-19T21:05:17.387 INFO:teuthology.orchestra.run.smithi163.stderr:set pool 2 pg_num to 42 2024-02-19T21:05:17.633 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:05:17.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:17.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:05:17.754 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:17.780 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:17.798 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:17.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:17.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:17.903 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:17.904 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:15.804603+0000 osd.0 (osd.0) 85 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:05:17.904 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:15.807728+0000 osd.0 (osd.0) 86 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:05:17.931 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:17.951 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:17.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:17.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:18.055 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:18.065 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:05:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:05:18.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:15.804603+0000 osd.0 (osd.0) 85 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:05:18.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:15.807728+0000 osd.0 (osd.0) 86 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:05:18.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:15.830443+0000 osd.1 (osd.1) 89 : cluster [DBG] 28.1 deep-scrub starts 2024-02-19T21:05:18.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:15.923354+0000 osd.1 (osd.1) 90 : cluster [DBG] 28.1 deep-scrub ok 2024-02-19T21:05:18.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:16.630100+0000 mon.c (mon.2) 115 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:18.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:16.915701+0000 mon.c (mon.2) 116 : audit [INF] from='client.? v1:172.21.15.163:0/2680502209' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:05:18.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:16.916697+0000 mon.a (mon.0) 1092 : audit [INF] from='client.46559 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:05:18.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:17.090603+0000 mon.b (mon.1) 213 : audit [INF] from='client.? v1:172.21.15.163:0/2597427537' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2024-02-19T21:05:18.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:17.091473+0000 mon.a (mon.0) 1093 : audit [INF] from='client.44038 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2024-02-19T21:05:18.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:17.260092+0000 mon.a (mon.0) 1094 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:05:18.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:17.313111+0000 mon.a (mon.0) 1095 : audit [INF] from='client.44138 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache"}]': finished 2024-02-19T21:05:18.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:17.313190+0000 mon.a (mon.0) 1096 : audit [INF] from='client.32629 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]': finished 2024-02-19T21:05:18.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:17.313245+0000 mon.a (mon.0) 1097 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-11"}]': finished 2024-02-19T21:05:18.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:17.313354+0000 mon.a (mon.0) 1098 : audit [INF] from='client.46559 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:05:18.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:17.313405+0000 mon.a (mon.0) 1099 : audit [INF] from='client.44038 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]': finished 2024-02-19T21:05:18.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:17.313451+0000 mon.a (mon.0) 1100 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-02-19T21:05:18.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:17.323739+0000 mon.b (mon.1) 214 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:18.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:17.324988+0000 mon.b (mon.1) 215 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-11", "mode": "writeback"}]: dispatch 2024-02-19T21:05:18.239 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:15.804603+0000 osd.0 (osd.0) 85 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:05:18.240 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:15.807728+0000 osd.0 (osd.0) 86 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:05:18.240 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:15.830443+0000 osd.1 (osd.1) 89 : cluster [DBG] 28.1 deep-scrub starts 2024-02-19T21:05:18.240 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:15.923354+0000 osd.1 (osd.1) 90 : cluster [DBG] 28.1 deep-scrub ok 2024-02-19T21:05:18.240 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:16.630100+0000 mon.c (mon.2) 115 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:18.240 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:16.915701+0000 mon.c (mon.2) 116 : audit [INF] from='client.? v1:172.21.15.163:0/2680502209' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:05:18.240 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:16.916697+0000 mon.a (mon.0) 1092 : audit [INF] from='client.46559 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:05:18.240 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:17.090603+0000 mon.b (mon.1) 213 : audit [INF] from='client.? v1:172.21.15.163:0/2597427537' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2024-02-19T21:05:18.240 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:17.091473+0000 mon.a (mon.0) 1093 : audit [INF] from='client.44038 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2024-02-19T21:05:18.240 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:17.260092+0000 mon.a (mon.0) 1094 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:05:18.240 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:17.313111+0000 mon.a (mon.0) 1095 : audit [INF] from='client.44138 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache"}]': finished 2024-02-19T21:05:18.240 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:17.313190+0000 mon.a (mon.0) 1096 : audit [INF] from='client.32629 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]': finished 2024-02-19T21:05:18.241 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:17.313245+0000 mon.a (mon.0) 1097 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-11"}]': finished 2024-02-19T21:05:18.241 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:17.313354+0000 mon.a (mon.0) 1098 : audit [INF] from='client.46559 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:05:18.241 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:17.313405+0000 mon.a (mon.0) 1099 : audit [INF] from='client.44038 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]': finished 2024-02-19T21:05:18.241 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:17.313451+0000 mon.a (mon.0) 1100 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-02-19T21:05:18.241 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:17.323739+0000 mon.b (mon.1) 214 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:18.241 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:17.324988+0000 mon.b (mon.1) 215 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-11", "mode": "writeback"}]: dispatch 2024-02-19T21:05:18.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:15.830443+0000 osd.1 (osd.1) 89 : cluster [DBG] 28.1 deep-scrub starts 2024-02-19T21:05:18.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:15.923354+0000 osd.1 (osd.1) 90 : cluster [DBG] 28.1 deep-scrub ok 2024-02-19T21:05:18.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:16.630100+0000 mon.c (mon.2) 115 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:18.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:16.915701+0000 mon.c (mon.2) 116 : audit [INF] from='client.? v1:172.21.15.163:0/2680502209' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:05:18.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:16.916697+0000 mon.a (mon.0) 1092 : audit [INF] from='client.46559 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:05:18.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:17.090603+0000 mon.b (mon.1) 213 : audit [INF] from='client.? v1:172.21.15.163:0/2597427537' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2024-02-19T21:05:18.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:17.091473+0000 mon.a (mon.0) 1093 : audit [INF] from='client.44038 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2024-02-19T21:05:18.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:17.260092+0000 mon.a (mon.0) 1094 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:05:18.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:17.313111+0000 mon.a (mon.0) 1095 : audit [INF] from='client.44138 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83936-10", "tierpool":"test-rados-api-smithi163-83936-10-cache"}]': finished 2024-02-19T21:05:18.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:17.313190+0000 mon.a (mon.0) 1096 : audit [INF] from='client.32629 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi163-83944-9"}]': finished 2024-02-19T21:05:18.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:17.313245+0000 mon.a (mon.0) 1097 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-11"}]': finished 2024-02-19T21:05:18.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:17.313354+0000 mon.a (mon.0) 1098 : audit [INF] from='client.46559 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:05:18.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:17.313405+0000 mon.a (mon.0) 1099 : audit [INF] from='client.44038 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]': finished 2024-02-19T21:05:18.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:17.313451+0000 mon.a (mon.0) 1100 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-02-19T21:05:18.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:17.323739+0000 mon.b (mon.1) 214 : audit [INF] from='client.? v1:172.21.15.163:0/1643190421' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:18.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:17.324988+0000 mon.b (mon.1) 215 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-11", "mode": "writeback"}]: dispatch 2024-02-19T21:05:18.265 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:18.873 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:18.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:18.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:18.980 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:19.006 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:19.025 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:19.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:19.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:19.129 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:05:19.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: Running main() from gmock_main.cc 2024-02-19T21:05:19.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-02-19T21:05:19.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [----------] Global test environment set-up. 2024-02-19T21:05:19.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-02-19T21:05:19.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-02-19T21:05:19.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ OK ] LibRadosPools.PoolList (7980 ms) 2024-02-19T21:05:19.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-02-19T21:05:19.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (4860 ms) 2024-02-19T21:05:19.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-02-19T21:05:19.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (2862 ms) 2024-02-19T21:05:19.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-02-19T21:05:19.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3059 ms) 2024-02-19T21:05:19.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-02-19T21:05:19.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3014 ms) 2024-02-19T21:05:19.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-02-19T21:05:19.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6561 ms) 2024-02-19T21:05:19.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-02-19T21:05:19.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (4854 ms) 2024-02-19T21:05:19.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-02-19T21:05:19.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5011 ms) 2024-02-19T21:05:19.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-02-19T21:05:19.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8179 ms) 2024-02-19T21:05:19.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [----------] 9 tests from LibRadosPools (46380 ms total) 2024-02-19T21:05:19.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: 2024-02-19T21:05:19.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [----------] Global test environment tear-down 2024-02-19T21:05:19.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (46380 ms total) 2024-02-19T21:05:19.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_pool: [ PASSED ] 9 tests. 2024-02-19T21:05:19.228 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:16.791565+0000 osd.6 (osd.6) 67 : cluster [DBG] 28.4 deep-scrub starts 2024-02-19T21:05:19.228 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:16.792655+0000 osd.6 (osd.6) 68 : cluster [DBG] 28.4 deep-scrub ok 2024-02-19T21:05:19.228 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:17.019647+0000 osd.5 (osd.5) 76 : cluster [DBG] 28.3 deep-scrub starts 2024-02-19T21:05:19.228 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:17.044643+0000 osd.5 (osd.5) 77 : cluster [DBG] 28.3 deep-scrub ok 2024-02-19T21:05:19.228 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:17.120646+0000 mgr.y (mgr.14550) 184 : cluster [DBG] pgmap v176: 452 pgs: 32 unknown, 32 creating+peering, 1 active+clean+snaptrim, 387 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 5 op/s 2024-02-19T21:05:19.228 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:17.357141+0000 mon.a (mon.0) 1101 : audit [INF] from='client.? v1:172.21.15.163:0/1523762372' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi163-83845-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:19.228 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:17.357592+0000 mon.a (mon.0) 1102 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:19.228 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:17.357914+0000 mon.a (mon.0) 1103 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-11", "mode": "writeback"}]: dispatch 2024-02-19T21:05:19.229 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:17.378822+0000 mon.a (mon.0) 1104 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:05:19.229 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:17.637403+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:19.229 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:18.311566+0000 mon.a (mon.0) 1105 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:05:19.229 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:18.328243+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? v1:172.21.15.163:0/1523762372' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi163-83845-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:19.229 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:18.328607+0000 mon.a (mon.0) 1107 : audit [INF] from='client.32629 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi163-83944-9"}]': finished 2024-02-19T21:05:19.229 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:18.328688+0000 mon.a (mon.0) 1108 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-11", "mode": "writeback"}]': finished 2024-02-19T21:05:19.229 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:18.328828+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-02-19T21:05:19.229 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:16.791565+0000 osd.6 (osd.6) 67 : cluster [DBG] 28.4 deep-scrub starts 2024-02-19T21:05:19.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:16.792655+0000 osd.6 (osd.6) 68 : cluster [DBG] 28.4 deep-scrub ok 2024-02-19T21:05:19.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:17.019647+0000 osd.5 (osd.5) 76 : cluster [DBG] 28.3 deep-scrub starts 2024-02-19T21:05:19.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:17.044643+0000 osd.5 (osd.5) 77 : cluster [DBG] 28.3 deep-scrub ok 2024-02-19T21:05:19.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:17.120646+0000 mgr.y (mgr.14550) 184 : cluster [DBG] pgmap v176: 452 pgs: 32 unknown, 32 creating+peering, 1 active+clean+snaptrim, 387 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 5 op/s 2024-02-19T21:05:19.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:17.357141+0000 mon.a (mon.0) 1101 : audit [INF] from='client.? v1:172.21.15.163:0/1523762372' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi163-83845-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:19.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:17.357592+0000 mon.a (mon.0) 1102 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:19.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:17.357914+0000 mon.a (mon.0) 1103 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-11", "mode": "writeback"}]: dispatch 2024-02-19T21:05:19.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:17.378822+0000 mon.a (mon.0) 1104 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:05:19.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:17.637403+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:19.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:18.311566+0000 mon.a (mon.0) 1105 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:05:19.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:18.328243+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? v1:172.21.15.163:0/1523762372' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi163-83845-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:19.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:18.328607+0000 mon.a (mon.0) 1107 : audit [INF] from='client.32629 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi163-83944-9"}]': finished 2024-02-19T21:05:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:18.328688+0000 mon.a (mon.0) 1108 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-11", "mode": "writeback"}]': finished 2024-02-19T21:05:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:18.328828+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-02-19T21:05:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:18.403970+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.163:0/3219046901' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:18.406743+0000 mon.a (mon.0) 1110 : audit [INF] from='client.47072 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:18.502743+0000 mon.b (mon.1) 216 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:18.573807+0000 mon.a (mon.0) 1111 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:18.576816+0000 mon.b (mon.1) 217 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:18.601594+0000 mon.a (mon.0) 1112 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:18.612433+0000 mon.b (mon.1) 218 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:18.639402+0000 mon.c (mon.2) 119 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:19.231 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:18.648353+0000 mon.a (mon.0) 1113 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:19.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:18.403970+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.163:0/3219046901' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:19.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:18.406743+0000 mon.a (mon.0) 1110 : audit [INF] from='client.47072 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:19.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:18.502743+0000 mon.b (mon.1) 216 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:19.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:18.573807+0000 mon.a (mon.0) 1111 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:19.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:18.576816+0000 mon.b (mon.1) 217 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:19.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:18.601594+0000 mon.a (mon.0) 1112 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:19.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:18.612433+0000 mon.b (mon.1) 218 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:19.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:18.639402+0000 mon.c (mon.2) 119 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:19.232 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:18.648353+0000 mon.a (mon.0) 1113 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:19.233 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:19.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:16.791565+0000 osd.6 (osd.6) 67 : cluster [DBG] 28.4 deep-scrub starts 2024-02-19T21:05:19.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:16.792655+0000 osd.6 (osd.6) 68 : cluster [DBG] 28.4 deep-scrub ok 2024-02-19T21:05:19.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:17.019647+0000 osd.5 (osd.5) 76 : cluster [DBG] 28.3 deep-scrub starts 2024-02-19T21:05:19.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:17.044643+0000 osd.5 (osd.5) 77 : cluster [DBG] 28.3 deep-scrub ok 2024-02-19T21:05:19.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:17.120646+0000 mgr.y (mgr.14550) 184 : cluster [DBG] pgmap v176: 452 pgs: 32 unknown, 32 creating+peering, 1 active+clean+snaptrim, 387 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 5 op/s 2024-02-19T21:05:19.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:17.357141+0000 mon.a (mon.0) 1101 : audit [INF] from='client.? v1:172.21.15.163:0/1523762372' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi163-83845-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:19.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:17.357592+0000 mon.a (mon.0) 1102 : audit [INF] from='client.32629 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi163-83944-9"}]: dispatch 2024-02-19T21:05:19.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:17.357914+0000 mon.a (mon.0) 1103 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-11", "mode": "writeback"}]: dispatch 2024-02-19T21:05:19.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:17.378822+0000 mon.a (mon.0) 1104 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:05:19.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:17.637403+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:19.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:18.311566+0000 mon.a (mon.0) 1105 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:05:19.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:18.328243+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? v1:172.21.15.163:0/1523762372' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi163-83845-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:19.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:18.328607+0000 mon.a (mon.0) 1107 : audit [INF] from='client.32629 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi163-83944-9"}]': finished 2024-02-19T21:05:19.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:18.328688+0000 mon.a (mon.0) 1108 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-11", "mode": "writeback"}]': finished 2024-02-19T21:05:19.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:18.328828+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-02-19T21:05:19.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:18.403970+0000 mon.c (mon.2) 118 : audit [INF] from='client.? v1:172.21.15.163:0/3219046901' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:19.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:18.406743+0000 mon.a (mon.0) 1110 : audit [INF] from='client.47072 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:19.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:18.502743+0000 mon.b (mon.1) 216 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:19.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:18.573807+0000 mon.a (mon.0) 1111 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:19.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:18.576816+0000 mon.b (mon.1) 217 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:19.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:18.601594+0000 mon.a (mon.0) 1112 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:19.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:18.612433+0000 mon.b (mon.1) 218 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:19.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:18.639402+0000 mon.c (mon.2) 119 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:19.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:18.648353+0000 mon.a (mon.0) 1113 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:19.844 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:05:19.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:19.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:05:19.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:19.121625+0000 mon.b (mon.1) 219 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-19T21:05:19.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:19.123799+0000 mon.a (mon.0) 1114 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-19T21:05:19.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:19.163456+0000 mon.a (mon.0) 1115 : audit [INF] from='client.47072 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:19.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:19.163579+0000 mon.a (mon.0) 1116 : audit [INF] from='client.45484 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:05:19.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:19.163682+0000 mon.a (mon.0) 1117 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]': finished 2024-02-19T21:05:19.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:19.163735+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-02-19T21:05:19.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:19.181105+0000 mon.b (mon.1) 220 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:19.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:19.182691+0000 mon.a (mon.0) 1119 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:19.967 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:05:20.016 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:20.245 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:19.121625+0000 mon.b (mon.1) 219 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-19T21:05:20.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:19.123799+0000 mon.a (mon.0) 1114 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-19T21:05:20.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:19.163456+0000 mon.a (mon.0) 1115 : audit [INF] from='client.47072 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:20.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:19.163579+0000 mon.a (mon.0) 1116 : audit [INF] from='client.45484 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:05:20.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:19.163682+0000 mon.a (mon.0) 1117 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]': finished 2024-02-19T21:05:20.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:19.163735+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-02-19T21:05:20.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:19.181105+0000 mon.b (mon.1) 220 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:20.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:19.182691+0000 mon.a (mon.0) 1119 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:20.315 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:05:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:05:20.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:19.121625+0000 mon.b (mon.1) 219 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-19T21:05:20.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:19.123799+0000 mon.a (mon.0) 1114 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-19T21:05:20.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:19.163456+0000 mon.a (mon.0) 1115 : audit [INF] from='client.47072 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:20.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:19.163579+0000 mon.a (mon.0) 1116 : audit [INF] from='client.45484 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:05:20.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:19.163682+0000 mon.a (mon.0) 1117 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]': finished 2024-02-19T21:05:20.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:19.163735+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-02-19T21:05:20.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:19.181105+0000 mon.b (mon.1) 220 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:20.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:19.182691+0000 mon.a (mon.0) 1119 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:20.666 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:05:20.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:20.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:05:20.772 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:05:20.800 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:21.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-02-19T21:05:21.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [----------] Global test environment set-up. 2024-02-19T21:05:21.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-02-19T21:05:21.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-02-19T21:05:21.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-02-19T21:05:21.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-02-19T21:05:21.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: 2024-02-19T21:05:21.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-02-19T21:05:21.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-02-19T21:05:21.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-02-19T21:05:21.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: 2024-02-19T21:04:32.965+0000 7f4143cca700 -1 failed for service _ceph-mon._tcp 2024-02-19T21:05:21.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: 2024-02-19T21:04:32.965+0000 7f4143cca700 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-02-19T21:05:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (90 ms) 2024-02-19T21:05:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-02-19T21:05:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2011 ms) 2024-02-19T21:05:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2101 ms total) 2024-02-19T21:05:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: 2024-02-19T21:05:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-02-19T21:05:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-02-19T21:05:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: started 0x55928e1bff70 2024-02-19T21:05:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: started 0x55928e1c9450 2024-02-19T21:05:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: started 2 aios 2024-02-19T21:05:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: waiting 0x55928e1bff70 2024-02-19T21:05:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: waiting 0x55928e1c9450 2024-02-19T21:05:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: done. 2024-02-19T21:05:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (9742 ms) 2024-02-19T21:05:21.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (9742 ms total) 2024-02-19T21:05:21.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: 2024-02-19T21:05:21.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-02-19T21:05:21.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-02-19T21:05:21.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-02-19T21:05:21.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-02-19T21:05:21.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ OK ] LibRadosMisc.Exec (4 ms) 2024-02-19T21:05:21.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-02-19T21:05:21.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (6 ms) 2024-02-19T21:05:21.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-02-19T21:05:21.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (4 ms) 2024-02-19T21:05:21.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-02-19T21:05:21.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4116 ms) 2024-02-19T21:05:21.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-02-19T21:05:21.205 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-02-19T21:05:21.205 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-02-19T21:05:21.205 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-02-19T21:05:21.205 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-02-19T21:05:21.205 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (29197 ms) 2024-02-19T21:05:21.205 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [----------] 8 tests from LibRadosMisc (33328 ms total) 2024-02-19T21:05:21.205 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: 2024-02-19T21:05:21.205 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [----------] Global test environment tear-down 2024-02-19T21:05:21.205 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (48314 ms total) 2024-02-19T21:05:21.205 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc: [ PASSED ] 12 tests. 2024-02-19T21:05:21.226 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:05:21.226 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-02-19T21:05:21.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-02-19T21:05:21.226 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-02-19T21:05:21.226 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.226 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24919.0:2074 28.18 28:19125b2f:test-rados-api-smithi163-83920-17::big3:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected e102)", 2024-02-19T21:05:21.226 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:42.228484+0000", 2024-02-19T21:05:21.226 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 39.001707138, 2024-02-19T21:05:21.226 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.066405645999999999, 2024-02-19T21:05:21.226 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.226 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:21.226 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:05:21.227 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24919", 2024-02-19T21:05:21.227 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/4089460683", 2024-02-19T21:05:21.227 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2074 2024-02-19T21:05:21.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.227 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.227 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.227 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.228484+0000", 2024-02-19T21:05:21.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.227 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.227 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.227 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.227 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.228484+0000", 2024-02-19T21:05:21.227 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999967 2024-02-19T21:05:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.228481+0000", 2024-02-19T21:05:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0067282870000000003 2024-02-19T21:05:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235209+0000", 2024-02-19T21:05:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8715999999999998e-05 2024-02-19T21:05:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.228 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235228+0000", 2024-02-19T21:05:21.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7010000000000002e-06 2024-02-19T21:05:21.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.229 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.229 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235237+0000", 2024-02-19T21:05:21.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5971000000000003e-05 2024-02-19T21:05:21.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.229 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.229 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235292+0000", 2024-02-19T21:05:21.229 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000144649 2024-02-19T21:05:21.229 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.229 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.229 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235437+0000", 2024-02-19T21:05:21.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000111014 2024-02-19T21:05:21.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,5", 2024-02-19T21:05:21.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235548+0000", 2024-02-19T21:05:21.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.050371437999999998 2024-02-19T21:05:21.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.230 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.285920+0000", 2024-02-19T21:05:21.230 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084572139999999994 2024-02-19T21:05:21.230 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.230 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.230 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.294377+0000", 2024-02-19T21:05:21.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040791500000000001 2024-02-19T21:05:21.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:05:21.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.294785+0000", 2024-02-19T21:05:21.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6798000000000001e-05 2024-02-19T21:05:21.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:05:21.231 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.294842+0000", 2024-02-19T21:05:21.231 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8254000000000003e-05 2024-02-19T21:05:21.231 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.231 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.231 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.232 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.294890+0000", 2024-02-19T21:05:21.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.066405645999999999 2024-02-19T21:05:21.232 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.232 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.232 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.232 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.232 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.232 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24919.0:2076 28.18 28:19125b2f:test-rados-api-smithi163-83920-17::big3:head [omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected e102)", 2024-02-19T21:05:21.232 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:42.297620+0000", 2024-02-19T21:05:21.232 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 38.932571476, 2024-02-19T21:05:21.232 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019708486000000001, 2024-02-19T21:05:21.232 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.232 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24919", 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/4089460683", 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2076 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.297620+0000", 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.297620+0000", 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:05:21.233 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.297617+0000", 2024-02-19T21:05:21.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00064993500000000003 2024-02-19T21:05:21.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.234 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.234 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.298267+0000", 2024-02-19T21:05:21.234 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9818000000000001e-05 2024-02-19T21:05:21.234 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.234 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.298297+0000", 2024-02-19T21:05:21.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3176999999999999e-05 2024-02-19T21:05:21.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.298310+0000", 2024-02-19T21:05:21.235 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6193000000000006e-05 2024-02-19T21:05:21.235 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.235 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.235 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.235 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.298396+0000", 2024-02-19T21:05:21.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2559999999999995e-05 2024-02-19T21:05:21.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.236 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.298489+0000", 2024-02-19T21:05:21.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0059358900000000001 2024-02-19T21:05:21.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.236 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,5", 2024-02-19T21:05:21.236 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.304425+0000", 2024-02-19T21:05:21.236 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0071163659999999998 2024-02-19T21:05:21.236 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.236 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.236 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:05:21.237 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.311541+0000", 2024-02-19T21:05:21.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0051423989999999998 2024-02-19T21:05:21.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.237 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.237 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.316684+0000", 2024-02-19T21:05:21.237 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00053349099999999998 2024-02-19T21:05:21.237 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.237 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.317217+0000", 2024-02-19T21:05:21.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6847999999999994e-05 2024-02-19T21:05:21.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:05:21.238 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.317284+0000", 2024-02-19T21:05:21.238 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4279000000000001e-05 2024-02-19T21:05:21.238 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.238 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.238 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.317328+0000", 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.019708486000000001 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.35189.0:4 36.16 e104/102 36:681f27cd:::2.obj:head v 104'1, mlcod=0'0)", 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:43.771810+0000", 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 37.458380898999998, 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020257964, 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.239 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.771810+0000", 2024-02-19T21:05:21.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.771810+0000", 2024-02-19T21:05:21.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:05:21.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.240 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.240 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.771810+0000", 2024-02-19T21:05:21.240 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7360000000000004e-06 2024-02-19T21:05:21.240 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.240 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.771817+0000", 2024-02-19T21:05:21.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3720000000000004e-06 2024-02-19T21:05:21.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.771822+0000", 2024-02-19T21:05:21.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5220000000000004e-06 2024-02-19T21:05:21.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.241 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.241 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.771829+0000", 2024-02-19T21:05:21.241 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014034539999999999 2024-02-19T21:05:21.241 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.241 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.773232+0000", 2024-02-19T21:05:21.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1633000000000001e-05 2024-02-19T21:05:21.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.773254+0000", 2024-02-19T21:05:21.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018794551999999999 2024-02-19T21:05:21.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.242 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.242 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:05:21.242 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.792048+0000", 2024-02-19T21:05:21.242 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.003e-05 2024-02-19T21:05:21.242 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.243 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.243 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.792068+0000", 2024-02-19T21:05:21.243 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020257964 2024-02-19T21:05:21.243 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.243 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.243 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.243 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.243 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.243 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25258.0:1 45.6s0 45:6dd63e69:test-rados-api-smithi163-83864-16::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e106)", 2024-02-19T21:05:21.244 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:45.971389+0000", 2024-02-19T21:05:21.244 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 35.258802588999998, 2024-02-19T21:05:21.244 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0304329539999999, 2024-02-19T21:05:21.244 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.244 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:21.244 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:05:21.244 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25258", 2024-02-19T21:05:21.244 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3498182538", 2024-02-19T21:05:21.244 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:05:21.244 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.244 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.244 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.245 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.245 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971389+0000", 2024-02-19T21:05:21.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.245 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.245 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971389+0000", 2024-02-19T21:05:21.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999967 2024-02-19T21:05:21.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.245 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.245 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.245 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971386+0000", 2024-02-19T21:05:21.245 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4117000000000001e-05 2024-02-19T21:05:21.245 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971400+0000", 2024-02-19T21:05:21.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.344e-05 2024-02-19T21:05:21.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971413+0000", 2024-02-19T21:05:21.246 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8219999999999996e-06 2024-02-19T21:05:21.246 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.246 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.246 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.246 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971422+0000", 2024-02-19T21:05:21.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2419999999999997e-05 2024-02-19T21:05:21.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971464+0000", 2024-02-19T21:05:21.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3889999999999998e-06 2024-02-19T21:05:21.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:05:21.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971469+0000", 2024-02-19T21:05:21.247 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0243061849999999 2024-02-19T21:05:21.247 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.247 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.247 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.247 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.995775+0000", 2024-02-19T21:05:21.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012886000000000001 2024-02-19T21:05:21.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.995904+0000", 2024-02-19T21:05:21.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023266189999999998 2024-02-19T21:05:21.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.248 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:05:21.248 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.998230+0000", 2024-02-19T21:05:21.248 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033170790000000001 2024-02-19T21:05:21.248 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.248 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:05:21.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.001547+0000", 2024-02-19T21:05:21.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9880000000000002e-05 2024-02-19T21:05:21.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:05:21.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.001617+0000", 2024-02-19T21:05:21.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000204291 2024-02-19T21:05:21.249 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.249 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.249 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.249 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.001822+0000", 2024-02-19T21:05:21.249 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0304329539999999 2024-02-19T21:05:21.249 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.249 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.250 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.250 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.250 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25318.0:2 48.6 48:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:05:21.250 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:47.974844+0000", 2024-02-19T21:05:21.250 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.255347624000002, 2024-02-19T21:05:21.250 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.048850169999999998, 2024-02-19T21:05:21.250 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.250 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:21.250 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:05:21.250 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25318", 2024-02-19T21:05:21.250 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/4284884290", 2024-02-19T21:05:21.250 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:05:21.250 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.251 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974844+0000", 2024-02-19T21:05:21.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974844+0000", 2024-02-19T21:05:21.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:05:21.251 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.251 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.251 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.251 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974841+0000", 2024-02-19T21:05:21.251 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2949999999999999e-06 2024-02-19T21:05:21.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974850+0000", 2024-02-19T21:05:21.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1317e-05 2024-02-19T21:05:21.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974862+0000", 2024-02-19T21:05:21.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6129999999999998e-06 2024-02-19T21:05:21.252 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.252 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.252 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.252 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974869+0000", 2024-02-19T21:05:21.252 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018860630000000001 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.976755+0000", 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2200000000000003e-06 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.976762+0000", 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043147748 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.019910+0000", 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9196000000000006e-05 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.253 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.019979+0000", 2024-02-19T21:05:21.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141478 2024-02-19T21:05:21.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,5", 2024-02-19T21:05:21.254 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.020121+0000", 2024-02-19T21:05:21.254 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00053344600000000003 2024-02-19T21:05:21.254 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.254 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.254 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:05:21.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.020654+0000", 2024-02-19T21:05:21.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001117311 2024-02-19T21:05:21.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.021772+0000", 2024-02-19T21:05:21.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001839909 2024-02-19T21:05:21.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.255 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.255 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.255 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.023612+0000", 2024-02-19T21:05:21.255 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2648000000000002e-05 2024-02-19T21:05:21.255 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:05:21.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.023654+0000", 2024-02-19T21:05:21.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9617999999999999e-05 2024-02-19T21:05:21.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.256 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.256 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.256 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.023694+0000", 2024-02-19T21:05:21.256 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.048850169999999998 2024-02-19T21:05:21.256 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.256 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.256 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.256 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.257 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25294.0:1 51.3s0 51:d77cf283:test-rados-api-smithi163-83896-11::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:05:21.257 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:47.988394+0000", 2024-02-19T21:05:21.257 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.241796997999998, 2024-02-19T21:05:21.257 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.593213569, 2024-02-19T21:05:21.257 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.257 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:21.257 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:05:21.257 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25294", 2024-02-19T21:05:21.257 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2922168330", 2024-02-19T21:05:21.257 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:05:21.257 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.257 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.257 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988394+0000", 2024-02-19T21:05:21.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988394+0000", 2024-02-19T21:05:21.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:05:21.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.258 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.258 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.258 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988392+0000", 2024-02-19T21:05:21.258 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9700000000000002e-06 2024-02-19T21:05:21.258 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988399+0000", 2024-02-19T21:05:21.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3440000000000001e-06 2024-02-19T21:05:21.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988408+0000", 2024-02-19T21:05:21.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2559999999999996e-06 2024-02-19T21:05:21.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.259 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.259 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.259 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988415+0000", 2024-02-19T21:05:21.259 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013816699999999999 2024-02-19T21:05:21.259 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988553+0000", 2024-02-19T21:05:21.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.056e-06 2024-02-19T21:05:21.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:05:21.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988556+0000", 2024-02-19T21:05:21.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.57455238200000003 2024-02-19T21:05:21.260 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.260 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.260 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.260 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.563108+0000", 2024-02-19T21:05:21.260 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000100029 2024-02-19T21:05:21.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.563208+0000", 2024-02-19T21:05:21.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00041083999999999999 2024-02-19T21:05:21.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:05:21.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.563619+0000", 2024-02-19T21:05:21.261 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0070195659999999997 2024-02-19T21:05:21.261 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.261 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.261 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:05:21.261 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.570639+0000", 2024-02-19T21:05:21.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.01090327 2024-02-19T21:05:21.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:05:21.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.581542+0000", 2024-02-19T21:05:21.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5691999999999994e-05 2024-02-19T21:05:21.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.262 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.262 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.262 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.581608+0000", 2024-02-19T21:05:21.262 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.593213569 2024-02-19T21:05:21.262 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.262 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.262 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.262 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.263 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24865.0:19 11.8 11:1381d4a2:test-rados-api-smithi163-83952-6::foo:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e111)", 2024-02-19T21:05:21.263 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:50.587909+0000", 2024-02-19T21:05:21.263 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 30.642282414, 2024-02-19T21:05:21.263 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023535916, 2024-02-19T21:05:21.263 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.263 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:21.263 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:05:21.263 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24865", 2024-02-19T21:05:21.263 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3187889786", 2024-02-19T21:05:21.263 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 19 2024-02-19T21:05:21.263 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.263 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.263 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.263 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.587909+0000", 2024-02-19T21:05:21.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.587909+0000", 2024-02-19T21:05:21.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:05:21.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.264 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.264 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.264 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.587907+0000", 2024-02-19T21:05:21.264 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6520000000000003e-06 2024-02-19T21:05:21.264 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.587913+0000", 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9639999999999998e-06 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.587923+0000", 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0039999999999999e-06 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.587931+0000", 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014025973000000001 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.265 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.601957+0000", 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3761e-05 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.602040+0000", 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020066699999999999 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.602241+0000", 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0048605849999999997 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:05:21.266 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.607102+0000", 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002572122 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.609674+0000", 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001682794 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.611357+0000", 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5219000000000002e-05 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.611402+0000", 2024-02-19T21:05:21.267 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2987e-05 2024-02-19T21:05:21.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.268 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.268 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.611445+0000", 2024-02-19T21:05:21.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023535916 2024-02-19T21:05:21.268 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.268 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.268 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.268 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.268 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.268 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14952.0:2 53.6 53:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-19T21:05:21.268 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:50.605965+0000", 2024-02-19T21:05:21.268 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 30.624226376999999, 2024-02-19T21:05:21.268 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036561746999999999, 2024-02-19T21:05:21.268 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:05:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14952", 2024-02-19T21:05:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2459019407", 2024-02-19T21:05:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:05:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605965+0000", 2024-02-19T21:05:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605965+0000", 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605963+0000", 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1330000000000004e-06 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605969+0000", 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5689999999999999e-06 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605977+0000", 2024-02-19T21:05:21.270 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4559999999999999e-06 2024-02-19T21:05:21.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605982+0000", 2024-02-19T21:05:21.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003383039 2024-02-19T21:05:21.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.609365+0000", 2024-02-19T21:05:21.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3409999999999997e-06 2024-02-19T21:05:21.271 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.271 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.271 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:05:21.271 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.609373+0000", 2024-02-19T21:05:21.271 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026694262 2024-02-19T21:05:21.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.636067+0000", 2024-02-19T21:05:21.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000103708 2024-02-19T21:05:21.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.636171+0000", 2024-02-19T21:05:21.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118288 2024-02-19T21:05:21.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.272 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.272 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4", 2024-02-19T21:05:21.272 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.636289+0000", 2024-02-19T21:05:21.272 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0017480130000000001 2024-02-19T21:05:21.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.638037+0000", 2024-02-19T21:05:21.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00049063000000000004 2024-02-19T21:05:21.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:05:21.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.638528+0000", 2024-02-19T21:05:21.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039461189999999997 2024-02-19T21:05:21.273 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.273 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.273 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.273 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.642474+0000", 2024-02-19T21:05:21.273 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1164000000000002e-05 2024-02-19T21:05:21.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.274 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:05:21.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.642505+0000", 2024-02-19T21:05:21.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1668999999999999e-05 2024-02-19T21:05:21.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.274 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.274 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.274 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.642527+0000", 2024-02-19T21:05:21.274 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036561746999999999 2024-02-19T21:05:21.274 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.274 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24865.0:21 11.8 11:1381d4a2:test-rados-api-smithi163-83952-6::foo:head [create,delete] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-19T21:05:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:51.573042+0000", 2024-02-19T21:05:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 29.657149311000001, 2024-02-19T21:05:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.070965300999999995, 2024-02-19T21:05:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:05:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24865", 2024-02-19T21:05:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3187889786", 2024-02-19T21:05:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 21 2024-02-19T21:05:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.275 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573042+0000", 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573042+0000", 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573040+0000", 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2414e-05 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573052+0000", 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00026839099999999999 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573321+0000", 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0720000000000001e-05 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573331+0000", 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012845001 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.277 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.586176+0000", 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5872999999999995e-05 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.586272+0000", 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018172699999999999 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.586454+0000", 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034122739999999999 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.589866+0000", 2024-02-19T21:05:21.278 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047893630999999999 2024-02-19T21:05:21.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.637760+0000", 2024-02-19T21:05:21.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061464759999999997 2024-02-19T21:05:21.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.643906+0000", 2024-02-19T21:05:21.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4803999999999999e-05 2024-02-19T21:05:21.279 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.279 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.279 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:05:21.279 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.643961+0000", 2024-02-19T21:05:21.279 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6133e-05 2024-02-19T21:05:21.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.280 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.280 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.644007+0000", 2024-02-19T21:05:21.280 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.070965300999999995 2024-02-19T21:05:21.280 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.280 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.280 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.280 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.280 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.280 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.35590.0:2 78.6 e128/127 78:602f83fe:::foo:head v 128'1, mlcod=0'0)", 2024-02-19T21:05:21.280 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:08.266769+0000", 2024-02-19T21:05:21.280 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.963422639000001, 2024-02-19T21:05:21.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057433619999999998, 2024-02-19T21:05:21.281 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.281 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:21.281 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266769+0000", 2024-02-19T21:05:21.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.281 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.281 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266769+0000", 2024-02-19T21:05:21.281 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:05:21.281 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.281 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266768+0000", 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2753e-05 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266781+0000", 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6710000000000004e-06 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266787+0000", 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8820000000000001e-06 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.282 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266795+0000", 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001516304 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.268311+0000", 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7995e-05 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.268329+0000", 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055833582999999999 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:05:21.283 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.324162+0000", 2024-02-19T21:05:21.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9811000000000002e-05 2024-02-19T21:05:21.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.284 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.284 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.324202+0000", 2024-02-19T21:05:21.284 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057433619999999998 2024-02-19T21:05:21.284 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.284 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.284 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.284 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.284 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.284 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.26962.0:28 64.1e e128/116 64:7e799bab:test-rados-api-smithi163-83952-10::foo:head v 128'1, mlcod=0'0)", 2024-02-19T21:05:21.284 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:08.266799+0000", 2024-02-19T21:05:21.285 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.963392111999999, 2024-02-19T21:05:21.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057427275999999999, 2024-02-19T21:05:21.285 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.285 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:21.285 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266799+0000", 2024-02-19T21:05:21.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.285 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.285 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.285 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.285 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266799+0000", 2024-02-19T21:05:21.285 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:05:21.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266799+0000", 2024-02-19T21:05:21.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4129999999999999e-06 2024-02-19T21:05:21.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266803+0000", 2024-02-19T21:05:21.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6290000000000001e-06 2024-02-19T21:05:21.286 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.286 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.286 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.286 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266808+0000", 2024-02-19T21:05:21.286 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4929999999999998e-06 2024-02-19T21:05:21.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266811+0000", 2024-02-19T21:05:21.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001716396 2024-02-19T21:05:21.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.268527+0000", 2024-02-19T21:05:21.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.647e-05 2024-02-19T21:05:21.287 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.287 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.287 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.287 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.268544+0000", 2024-02-19T21:05:21.287 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055662087999999998 2024-02-19T21:05:21.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:05:21.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.324206+0000", 2024-02-19T21:05:21.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0389000000000001e-05 2024-02-19T21:05:21.288 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.288 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.288 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.288 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.324226+0000", 2024-02-19T21:05:21.288 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057427275999999999 2024-02-19T21:05:21.288 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.288 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.288 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.289 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:110 75.1e 75:7b0234b0:test-rados-api-smithi163-83928-10::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e129)", 2024-02-19T21:05:21.289 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:09.003796+0000", 2024-02-19T21:05:21.289 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.226395694000001, 2024-02-19T21:05:21.289 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.093054726000000004, 2024-02-19T21:05:21.289 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.289 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:05:21.289 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:05:21.289 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:05:21.289 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:05:21.289 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 110 2024-02-19T21:05:21.289 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.289 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.289 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003796+0000", 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003796+0000", 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003793+0000", 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6710000000000005e-06 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.290 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003802+0000", 2024-02-19T21:05:21.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2583e-05 2024-02-19T21:05:21.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003814+0000", 2024-02-19T21:05:21.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8140000000000007e-06 2024-02-19T21:05:21.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.291 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003823+0000", 2024-02-19T21:05:21.291 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.905e-05 2024-02-19T21:05:21.291 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.291 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.291 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003872+0000", 2024-02-19T21:05:21.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1829999999999997e-06 2024-02-19T21:05:21.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:05:21.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003880+0000", 2024-02-19T21:05:21.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010142976999999999 2024-02-19T21:05:21.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.292 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.014023+0000", 2024-02-19T21:05:21.292 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000209595 2024-02-19T21:05:21.292 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.292 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.292 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.014232+0000", 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017728766999999999 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.031961+0000", 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0221000000000003e-05 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032041+0000", 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.059847310000000001 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.293 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.091888+0000", 2024-02-19T21:05:21.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118545 2024-02-19T21:05:21.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.092007+0000", 2024-02-19T21:05:21.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0048433310000000002 2024-02-19T21:05:21.294 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.294 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.294 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.294 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.096850+0000", 2024-02-19T21:05:21.294 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.093054726000000004 2024-02-19T21:05:21.294 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.294 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.294 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(osd.3.0:301 75.1e e129/125 ondisk, result = 0)", 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:09.032674+0000", 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.197516853, 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036648739999999999, 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032674+0000", 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.295 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032674+0000", 2024-02-19T21:05:21.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:05:21.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032673+0000", 2024-02-19T21:05:21.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2400000000000001e-06 2024-02-19T21:05:21.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.296 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.296 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032680+0000", 2024-02-19T21:05:21.296 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5481e-05 2024-02-19T21:05:21.296 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.296 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032696+0000", 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5610000000000001e-06 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032703+0000", 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036602245999999998 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.069305+0000", 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4330000000000004e-06 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.297 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.069314+0000", 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3780000000000007e-06 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.069323+0000", 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036648739999999999 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:111 75.5 75:a23b9a9e:test-rados-api-smithi163-83928-10::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e129)", 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:09.097539+0000", 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.13265191, 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.13010182100000001, 2024-02-19T21:05:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 111 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097539+0000", 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097539+0000", 2024-02-19T21:05:21.299 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999962 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097536+0000", 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4768e-05 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097550+0000", 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8895000000000001e-05 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097569+0000", 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6809999999999993e-06 2024-02-19T21:05:21.300 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097579+0000", 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001749121 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.099328+0000", 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00027112000000000002 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.099599+0000", 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002897892 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.301 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.102497+0000", 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6007000000000001e-05 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.102553+0000", 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.12498295099999999 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.227536+0000", 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105049 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.302 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.227641+0000", 2024-02-19T21:05:21.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.13010182100000001 2024-02-19T21:05:21.303 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.303 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.303 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.303 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.303 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.303 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(76.3s1 133/126 ECSubWrite(tid=279, reqid=osd.4.0:279, at_version=133'6, trim_to=0'0, roll_forward_to=132'4))", 2024-02-19T21:05:21.303 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:13.106749+0000", 2024-02-19T21:05:21.303 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.1234427100000008, 2024-02-19T21:05:21.303 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039746143999999997, 2024-02-19T21:05:21.304 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.304 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:05:21.304 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.304 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.106749+0000", 2024-02-19T21:05:21.304 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.304 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.304 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.304 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.106749+0000", 2024-02-19T21:05:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:05:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.106747+0000", 2024-02-19T21:05:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3730999999999999e-05 2024-02-19T21:05:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.106781+0000", 2024-02-19T21:05:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015052899999999999 2024-02-19T21:05:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.305 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.121834+0000", 2024-02-19T21:05:21.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7899999999999994e-06 2024-02-19T21:05:21.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.121844+0000", 2024-02-19T21:05:21.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001291719 2024-02-19T21:05:21.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.306 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.123135+0000", 2024-02-19T21:05:21.306 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0396e-05 2024-02-19T21:05:21.306 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.306 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.306 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.123146+0000", 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023338983000000001 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.146485+0000", 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0078000000000001e-05 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.146495+0000", 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039746143999999997 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.307 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.308 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.39709.0:2 84.3 84:c656db43:::oid_RoundTripPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-02-19T21:05:21.308 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:13.135107+0000", 2024-02-19T21:05:21.308 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 8.0950841649999994, 2024-02-19T21:05:21.308 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037126581999999998, 2024-02-19T21:05:21.308 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.308 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:21.308 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:05:21.308 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.39709", 2024-02-19T21:05:21.308 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3505054706", 2024-02-19T21:05:21.308 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:05:21.308 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.308 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.308 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.308 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135107+0000", 2024-02-19T21:05:21.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135107+0000", 2024-02-19T21:05:21.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:05:21.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.309 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.309 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135104+0000", 2024-02-19T21:05:21.309 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8099999999999992e-06 2024-02-19T21:05:21.309 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.309 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135114+0000", 2024-02-19T21:05:21.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1422e-05 2024-02-19T21:05:21.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135125+0000", 2024-02-19T21:05:21.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5689999999999999e-06 2024-02-19T21:05:21.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.310 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.310 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135133+0000", 2024-02-19T21:05:21.310 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8407e-05 2024-02-19T21:05:21.310 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.310 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135171+0000", 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2189999999999998e-06 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135179+0000", 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034870916000000002 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.170050+0000", 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011553900000000001 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.311 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.170165+0000", 2024-02-19T21:05:21.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019117400000000001 2024-02-19T21:05:21.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,7", 2024-02-19T21:05:21.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.170356+0000", 2024-02-19T21:05:21.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015803569999999999 2024-02-19T21:05:21.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.312 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.312 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:05:21.312 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.171937+0000", 2024-02-19T21:05:21.312 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000119746 2024-02-19T21:05:21.312 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.172056+0000", 2024-02-19T21:05:21.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104515 2024-02-19T21:05:21.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.172161+0000", 2024-02-19T21:05:21.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2716000000000003e-05 2024-02-19T21:05:21.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.313 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:05:21.313 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.172194+0000", 2024-02-19T21:05:21.313 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-02-19T21:05:21.313 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.314 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.314 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.172234+0000", 2024-02-19T21:05:21.314 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037126581999999998 2024-02-19T21:05:21.314 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.314 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.314 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.314 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.314 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.314 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.0.0:267 64.1e e134/116 64:7e799bab:test-rados-api-smithi163-83952-10::foo:b v 134'20, mlcod=131'16)", 2024-02-19T21:05:21.315 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:14.266716+0000", 2024-02-19T21:05:21.315 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 6.9634754010000002, 2024-02-19T21:05:21.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.083584948000000006, 2024-02-19T21:05:21.315 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.315 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:21.315 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266716+0000", 2024-02-19T21:05:21.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.315 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.315 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.315 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266716+0000", 2024-02-19T21:05:21.315 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:05:21.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266715+0000", 2024-02-19T21:05:21.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3909000000000001e-05 2024-02-19T21:05:21.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266739+0000", 2024-02-19T21:05:21.316 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.178e-06 2024-02-19T21:05:21.316 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.316 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.316 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.316 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266745+0000", 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2620000000000004e-06 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266752+0000", 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015674777000000001 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.282426+0000", 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4134999999999998e-05 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.282450+0000", 2024-02-19T21:05:21.317 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067824522999999998 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.350275+0000", 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5811000000000001e-05 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.350301+0000", 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.083584948000000006 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.318 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDScrubReserve(6.0 REQUEST e135)", 2024-02-19T21:05:21.319 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:15.781138+0000", 2024-02-19T21:05:21.319 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 5.4490531280000001, 2024-02-19T21:05:21.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023234604999999998, 2024-02-19T21:05:21.319 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.319 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:05:21.319 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.781138+0000", 2024-02-19T21:05:21.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.319 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.319 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.319 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.319 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.781138+0000", 2024-02-19T21:05:21.319 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999962 2024-02-19T21:05:21.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.781134+0000", 2024-02-19T21:05:21.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.403e-05 2024-02-19T21:05:21.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.320 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.320 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.781148+0000", 2024-02-19T21:05:21.320 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2660999999999999e-05 2024-02-19T21:05:21.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.320 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.781171+0000", 2024-02-19T21:05:21.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.025e-05 2024-02-19T21:05:21.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.781181+0000", 2024-02-19T21:05:21.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023144954999999998 2024-02-19T21:05:21.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.321 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.321 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.804326+0000", 2024-02-19T21:05:21.321 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6294e-05 2024-02-19T21:05:21.321 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.321 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.804342+0000", 2024-02-19T21:05:21.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0343000000000001e-05 2024-02-19T21:05:21.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.322 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.322 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.804373+0000", 2024-02-19T21:05:21.322 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023234604999999998 2024-02-19T21:05:21.322 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.322 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.322 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.322 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.322 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:257 85.2 85:43403dbe:test-rados-api-smithi163-83928-12::bam:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e138)", 2024-02-19T21:05:21.322 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:18.587348+0000", 2024-02-19T21:05:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 2.642843058, 2024-02-19T21:05:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.069587300000000005, 2024-02-19T21:05:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:05:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:05:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:05:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:05:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 257 2024-02-19T21:05:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587348+0000", 2024-02-19T21:05:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587348+0000", 2024-02-19T21:05:21.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999957 2024-02-19T21:05:21.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.324 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587344+0000", 2024-02-19T21:05:21.324 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6024000000000001e-05 2024-02-19T21:05:21.324 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.324 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.324 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587360+0000", 2024-02-19T21:05:21.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5661999999999999e-05 2024-02-19T21:05:21.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587386+0000", 2024-02-19T21:05:21.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0662999999999999e-05 2024-02-19T21:05:21.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.325 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587396+0000", 2024-02-19T21:05:21.325 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4304e-05 2024-02-19T21:05:21.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.325 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.325 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587450+0000", 2024-02-19T21:05:21.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023577900000000001 2024-02-19T21:05:21.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", 2024-02-19T21:05:21.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587686+0000", 2024-02-19T21:05:21.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0062748650000000001 2024-02-19T21:05:21.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.326 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.326 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.593961+0000", 2024-02-19T21:05:21.326 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8112000000000001e-05 2024-02-19T21:05:21.326 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.326 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-02-19T21:05:21.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.594029+0000", 2024-02-19T21:05:21.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.059753043999999998 2024-02-19T21:05:21.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.653782+0000", 2024-02-19T21:05:21.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7813999999999996e-05 2024-02-19T21:05:21.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.327 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.327 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.653880+0000", 2024-02-19T21:05:21.327 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003055412 2024-02-19T21:05:21.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.327 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.328 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.328 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.656935+0000", 2024-02-19T21:05:21.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.069587300000000005 2024-02-19T21:05:21.328 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.328 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.328 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.328 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.26962.0:57 64.3 64:c6c3d28d:test-rados-api-smithi163-83952-12::foo:head [write 0~128] snapc 14=[14] ondisk+write+ordersnap+known_if_redirected e139)", 2024-02-19T21:05:21.328 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:19.170619+0000", 2024-02-19T21:05:21.328 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 2.0595717680000001, 2024-02-19T21:05:21.328 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.10751445900000001, 2024-02-19T21:05:21.328 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:05:21.329 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:21.329 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:05:21.329 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.26962", 2024-02-19T21:05:21.329 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2924448120", 2024-02-19T21:05:21.329 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 57 2024-02-19T21:05:21.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.329 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:05:21.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:05:21.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170619+0000", 2024-02-19T21:05:21.329 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:05:21.329 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.329 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.329 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:05:21.329 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170619+0000", 2024-02-19T21:05:21.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:05:21.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:05:21.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170617+0000", 2024-02-19T21:05:21.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0165e-05 2024-02-19T21:05:21.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:05:21.330 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170627+0000", 2024-02-19T21:05:21.330 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2262999999999999e-05 2024-02-19T21:05:21.330 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.330 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.330 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:05:21.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170640+0000", 2024-02-19T21:05:21.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2870000000000004e-06 2024-02-19T21:05:21.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:05:21.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170648+0000", 2024-02-19T21:05:21.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0032999999999999e-05 2024-02-19T21:05:21.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.331 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170708+0000", 2024-02-19T21:05:21.331 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7159999999999997e-06 2024-02-19T21:05:21.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.331 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.331 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:05:21.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170715+0000", 2024-02-19T21:05:21.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047117838000000002 2024-02-19T21:05:21.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:05:21.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.217833+0000", 2024-02-19T21:05:21.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012449 2024-02-19T21:05:21.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.332 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:05:21.332 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.217957+0000", 2024-02-19T21:05:21.332 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015379599999999999 2024-02-19T21:05:21.332 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.332 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,6", 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.218111+0000", 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014134015999999999 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.232245+0000", 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039615919999999999 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.271861+0000", 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061818860000000002 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.333 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.278043+0000", 2024-02-19T21:05:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8167999999999997e-05 2024-02-19T21:05:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:05:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.278091+0000", 2024-02-19T21:05:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3086e-05 2024-02-19T21:05:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:05:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.278134+0000", 2024-02-19T21:05:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.10751445900000001 2024-02-19T21:05:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.334 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.335 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:21.335 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:21.335 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:05:21.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:19.121231+0000 mgr.y (mgr.14550) 185 : cluster [DBG] pgmap v179: 452 pgs: 64 unknown, 1 active+clean+snaptrim, 387 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:21.335 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:19.358659+0000 mon.b (mon.1) 221 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:19.646213+0000 mon.c (mon.2) 120 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:19.964532+0000 mon.a (mon.0) 1120 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:20.064766+0000 osd.5 (osd.5) 78 : cluster [DBG] 28.9 deep-scrub starts 2024-02-19T21:05:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:20.120912+0000 mgr.y (mgr.14550) 186 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:20.156470+0000 osd.5 (osd.5) 79 : cluster [DBG] 28.9 deep-scrub ok 2024-02-19T21:05:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:20.624924+0000 mon.a (mon.0) 1121 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:05:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:20.624997+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-02-19T21:05:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:20.629577+0000 mon.b (mon.1) 222 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11"}]: dispatch 2024-02-19T21:05:21.336 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:20.635224+0000 mon.a (mon.0) 1123 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11"}]: dispatch 2024-02-19T21:05:21.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:20.638053+0000 mon.b (mon.1) 223 : audit [INF] from='client.? v1:172.21.15.163:0/872449564' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi163-83845-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:21.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:20.638664+0000 mon.a (mon.0) 1124 : audit [INF] from='client.46153 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi163-83845-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:21.337 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:20.661296+0000 mon.c (mon.2) 121 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:21.337 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:19.121231+0000 mgr.y (mgr.14550) 185 : cluster [DBG] pgmap v179: 452 pgs: 64 unknown, 1 active+clean+snaptrim, 387 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:21.337 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:19.358659+0000 mon.b (mon.1) 221 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:21.337 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:19.646213+0000 mon.c (mon.2) 120 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:21.337 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:19.964532+0000 mon.a (mon.0) 1120 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:21.337 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:20.064766+0000 osd.5 (osd.5) 78 : cluster [DBG] 28.9 deep-scrub starts 2024-02-19T21:05:21.337 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:20.120912+0000 mgr.y (mgr.14550) 186 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:21.337 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:20.156470+0000 osd.5 (osd.5) 79 : cluster [DBG] 28.9 deep-scrub ok 2024-02-19T21:05:21.337 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:20.624924+0000 mon.a (mon.0) 1121 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:05:21.337 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:20.624997+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-02-19T21:05:21.338 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:20.629577+0000 mon.b (mon.1) 222 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11"}]: dispatch 2024-02-19T21:05:21.338 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:20.635224+0000 mon.a (mon.0) 1123 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11"}]: dispatch 2024-02-19T21:05:21.338 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:20.638053+0000 mon.b (mon.1) 223 : audit [INF] from='client.? v1:172.21.15.163:0/872449564' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi163-83845-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:21.338 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:20.638664+0000 mon.a (mon.0) 1124 : audit [INF] from='client.46153 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi163-83845-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:21.338 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:20.661296+0000 mon.c (mon.2) 121 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:21.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:19.121231+0000 mgr.y (mgr.14550) 185 : cluster [DBG] pgmap v179: 452 pgs: 64 unknown, 1 active+clean+snaptrim, 387 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:21.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:19.358659+0000 mon.b (mon.1) 221 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:21.338 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:19.646213+0000 mon.c (mon.2) 120 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:21.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:19.964532+0000 mon.a (mon.0) 1120 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:05:21.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:20.064766+0000 osd.5 (osd.5) 78 : cluster [DBG] 28.9 deep-scrub starts 2024-02-19T21:05:21.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:20.120912+0000 mgr.y (mgr.14550) 186 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:21.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:20.156470+0000 osd.5 (osd.5) 79 : cluster [DBG] 28.9 deep-scrub ok 2024-02-19T21:05:21.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:20.624924+0000 mon.a (mon.0) 1121 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:05:21.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:20.624997+0000 mon.a (mon.0) 1122 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-02-19T21:05:21.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:20.629577+0000 mon.b (mon.1) 222 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11"}]: dispatch 2024-02-19T21:05:21.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:20.635224+0000 mon.a (mon.0) 1123 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11"}]: dispatch 2024-02-19T21:05:21.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:20.638053+0000 mon.b (mon.1) 223 : audit [INF] from='client.? v1:172.21.15.163:0/872449564' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi163-83845-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:21.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:20.638664+0000 mon.a (mon.0) 1124 : audit [INF] from='client.46153 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi163-83845-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:21.339 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:20.661296+0000 mon.c (mon.2) 121 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:22.101 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:21.121737+0000 mgr.y (mgr.14550) 187 : cluster [DBG] pgmap v182: 398 pgs: 10 peering, 6 active+clean+snaptrim_wait, 42 unknown, 6 active+clean+snaptrim, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 257 KiB/s wr, 5 op/s 2024-02-19T21:05:22.101 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:21.122164+0000 mon.b (mon.1) 224 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-19T21:05:22.101 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:21.123332+0000 mon.a (mon.0) 1125 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-19T21:05:22.101 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:21.160942+0000 mon.a (mon.0) 1126 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:05:22.101 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:21.168076+0000 mon.a (mon.0) 1127 : audit [INF] from='client.45484 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]': finished 2024-02-19T21:05:22.101 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:21.168314+0000 mon.a (mon.0) 1128 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11"}]': finished 2024-02-19T21:05:22.101 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:21.168387+0000 mon.a (mon.0) 1129 : audit [INF] from='client.46153 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi163-83845-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:22.101 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:21.168542+0000 mon.a (mon.0) 1130 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]': finished 2024-02-19T21:05:22.101 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:21.168603+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-02-19T21:05:22.102 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:21.208522+0000 mon.b (mon.1) 225 : audit [INF] from='client.? v1:172.21.15.163:0/1485143480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi163-83854-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:22.102 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:21.208925+0000 mon.a (mon.0) 1132 : audit [INF] from='client.47503 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi163-83854-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:22.102 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:21.671007+0000 mon.c (mon.2) 122 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:22.102 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:21.121737+0000 mgr.y (mgr.14550) 187 : cluster [DBG] pgmap v182: 398 pgs: 10 peering, 6 active+clean+snaptrim_wait, 42 unknown, 6 active+clean+snaptrim, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 257 KiB/s wr, 5 op/s 2024-02-19T21:05:22.102 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:21.122164+0000 mon.b (mon.1) 224 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-19T21:05:22.102 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:21.123332+0000 mon.a (mon.0) 1125 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-19T21:05:22.102 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:21.160942+0000 mon.a (mon.0) 1126 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:05:22.102 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:21.168076+0000 mon.a (mon.0) 1127 : audit [INF] from='client.45484 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]': finished 2024-02-19T21:05:22.102 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:21.168314+0000 mon.a (mon.0) 1128 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11"}]': finished 2024-02-19T21:05:22.102 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:21.168387+0000 mon.a (mon.0) 1129 : audit [INF] from='client.46153 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi163-83845-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:22.102 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:21.168542+0000 mon.a (mon.0) 1130 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]': finished 2024-02-19T21:05:22.102 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:21.168603+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-02-19T21:05:22.102 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:21.208522+0000 mon.b (mon.1) 225 : audit [INF] from='client.? v1:172.21.15.163:0/1485143480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi163-83854-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:22.103 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:21.208925+0000 mon.a (mon.0) 1132 : audit [INF] from='client.47503 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi163-83854-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:22.103 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:21.671007+0000 mon.c (mon.2) 122 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:22.120 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-19T21:05:22.149 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:05:22.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:22.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:05:22.253 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:22.280 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:22.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:21.121737+0000 mgr.y (mgr.14550) 187 : cluster [DBG] pgmap v182: 398 pgs: 10 peering, 6 active+clean+snaptrim_wait, 42 unknown, 6 active+clean+snaptrim, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.2 KiB/s rd, 257 KiB/s wr, 5 op/s 2024-02-19T21:05:22.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:21.122164+0000 mon.b (mon.1) 224 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-19T21:05:22.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:21.123332+0000 mon.a (mon.0) 1125 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-19T21:05:22.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:21.160942+0000 mon.a (mon.0) 1126 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:05:22.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:21.168076+0000 mon.a (mon.0) 1127 : audit [INF] from='client.45484 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi163-83944-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]': finished 2024-02-19T21:05:22.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:21.168314+0000 mon.a (mon.0) 1128 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-11"}]': finished 2024-02-19T21:05:22.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:21.168387+0000 mon.a (mon.0) 1129 : audit [INF] from='client.46153 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi163-83845-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:22.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:21.168542+0000 mon.a (mon.0) 1130 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]': finished 2024-02-19T21:05:22.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:21.168603+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-02-19T21:05:22.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:21.208522+0000 mon.b (mon.1) 225 : audit [INF] from='client.? v1:172.21.15.163:0/1485143480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi163-83854-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:22.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:21.208925+0000 mon.a (mon.0) 1132 : audit [INF] from='client.47503 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi163-83854-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:22.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:21.671007+0000 mon.c (mon.2) 122 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:22.423 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:05:22.484 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:22.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:22.587 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:22.618 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:22.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.695+0000 7f9b1277e700 1 -- 172.21.15.163:0/1685202241 >> v1:172.21.15.163:6789/0 conn(0x7f9b0c0f9470 legacy=0x7f9b0c102a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:22.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.695+0000 7f9b1277e700 1 -- 172.21.15.163:0/1685202241 shutdown_connections 2024-02-19T21:05:22.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.695+0000 7f9b1277e700 1 -- 172.21.15.163:0/1685202241 >> 172.21.15.163:0/1685202241 conn(0x7f9b0c0f2470 msgr2=0x7f9b0c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:22.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.695+0000 7f9b1277e700 1 -- 172.21.15.163:0/1685202241 shutdown_connections 2024-02-19T21:05:22.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.695+0000 7f9b1277e700 1 -- 172.21.15.163:0/1685202241 wait complete. 2024-02-19T21:05:22.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.695+0000 7f9b1277e700 1 librados: starting msgr at 2024-02-19T21:05:22.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.695+0000 7f9b1277e700 1 librados: starting objecter 2024-02-19T21:05:22.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b1277e700 1 Processor -- start 2024-02-19T21:05:22.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b1277e700 1 -- start start 2024-02-19T21:05:22.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b1277e700 1 librados: setting wanted keys 2024-02-19T21:05:22.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b1277e700 1 librados: calling monclient init 2024-02-19T21:05:22.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b1277e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9b0c102ba0 con 0x7f9b0c0f8980 2024-02-19T21:05:22.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b1277e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9b0c102d70 con 0x7f9b0c104280 2024-02-19T21:05:22.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b1277e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9b0c102f60 con 0x7f9b0c0f9470 2024-02-19T21:05:22.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b0b7fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f9b0c0f9470 0x7f9b0c114cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:55376/0 (socket says 172.21.15.163:55376) 2024-02-19T21:05:22.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b0b7fe700 1 -- 172.21.15.163:0/2116044021 learned_addr learned my addr 172.21.15.163:0/2116044021 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:22.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b097fa700 10 client.?.objecter ms_handle_connect 0x7f9b0c0f9470 2024-02-19T21:05:22.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b097fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:22.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b097fa700 10 client.?.objecter ms_handle_connect 0x7f9b0c0f8980 2024-02-19T21:05:22.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b097fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:22.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b097fa700 10 client.?.objecter ms_handle_connect 0x7f9b0c104280 2024-02-19T21:05:22.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b097fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:22.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616441741 0 0) 0x7f9b0c102f60 con 0x7f9b0c0f9470 2024-02-19T21:05:22.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af80033f0 con 0x7f9b0c0f9470 2024-02-19T21:05:22.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721937336 0 0) 0x7f9b0c102ba0 con 0x7f9b0c0f8980 2024-02-19T21:05:22.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b0c102f60 con 0x7f9b0c0f8980 2024-02-19T21:05:22.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846873599 0 0) 0x7f9b0c102d70 con 0x7f9b0c104280 2024-02-19T21:05:22.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.696+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b0c102ba0 con 0x7f9b0c104280 2024-02-19T21:05:22.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1013473543 0 0) 0x7f9af80033f0 con 0x7f9b0c0f9470 2024-02-19T21:05:22.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b0c102d70 con 0x7f9b0c0f9470 2024-02-19T21:05:22.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2588911072 0 0) 0x7f9b0c102ba0 con 0x7f9b0c104280 2024-02-19T21:05:22.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9af80033f0 con 0x7f9b0c104280 2024-02-19T21:05:22.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9af4002cb0 con 0x7f9b0c0f9470 2024-02-19T21:05:22.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9b04001c30 con 0x7f9b0c104280 2024-02-19T21:05:22.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1795994105 0 0) 0x7f9b0c102f60 con 0x7f9b0c0f8980 2024-02-19T21:05:22.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b0c102ba0 con 0x7f9b0c0f8980 2024-02-19T21:05:22.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9afc001840 con 0x7f9b0c0f8980 2024-02-19T21:05:22.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 436111062 0 0) 0x7f9b0c102d70 con 0x7f9b0c0f9470 2024-02-19T21:05:22.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 >> v1:172.21.15.193:6789/0 conn(0x7f9b0c104280 legacy=0x7f9b0c118400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:22.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 >> v1:172.21.15.163:6789/0 conn(0x7f9b0c0f8980 legacy=0x7f9b0c102390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:22.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0c11cb90 con 0x7f9b0c0f9470 2024-02-19T21:05:22.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b0c119b80 con 0x7f9b0c0f9470 2024-02-19T21:05:22.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b1277e700 10 client.48803.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:22.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b0c119ff0 con 0x7f9b0c0f9470 2024-02-19T21:05:22.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b1277e700 1 librados: init done 2024-02-19T21:05:22.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9af4001b30 con 0x7f9b0c0f9470 2024-02-19T21:05:22.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.697+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9af4003830 con 0x7f9b0c0f9470 2024-02-19T21:05:22.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.698+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9aec004f00 con 0x7f9b0c0f9470 2024-02-19T21:05:22.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.698+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9af4003b20 con 0x7f9b0c0f9470 2024-02-19T21:05:22.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(142..142 src has 43..142) v4 ==== 9738+0+0 (unknown 466293249 0 0) 0x7f9af40797a0 con 0x7f9b0c0f9470 2024-02-19T21:05:22.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 osd_map(142..142 src has 43..142) v4 2024-02-19T21:05:22.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b097fa700 3 client.48803.objecter handle_osd_map got epochs [142,142] > 0 2024-02-19T21:05:22.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b097fa700 3 client.48803.objecter handle_osd_map decoding full epoch 142 2024-02-19T21:05:22.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b097fa700 20 client.48803.objecter dump_active .. 0 homeless 2024-02-19T21:05:22.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v142) v1 ==== 54+0+16 (unknown 1954259925 0 2420000192) 0x7f9af407bbb0 con 0x7f9b0c0f9470 2024-02-19T21:05:22.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b097fa700 10 client.48803.objecter ms_handle_connect 0x7f9af8063cb0 2024-02-19T21:05:22.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b1277e700 10 client.48803.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:22.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 1 2024-02-19T21:05:22.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec0051a0 osd=0 3 2024-02-19T21:05:22.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec0051a0 osd=0 3 2024-02-19T21:05:22.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec0051a0 osd=0 3 2024-02-19T21:05:22.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 1 2024-02-19T21:05:22.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec0051a0 osd=0 3 2024-02-19T21:05:22.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 0 1 2024-02-19T21:05:22.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec0051a0 osd=0 4 2024-02-19T21:05:22.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b1277e700 10 client.48803.objecter _send_command 1 2024-02-19T21:05:22.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9aec004950 con 0x7f9aec004530 2024-02-19T21:05:22.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.699+0000 7f9b097fa700 10 client.48803.objecter ms_handle_connect 0x7f9aec004530 2024-02-19T21:05:22.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.700+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9aec004950 con 0x7f9aec004530 2024-02-19T21:05:22.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.700+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:05:22.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.700+0000 7f9b097fa700 10 client.48803.objecter _finish_command 1 = system:0 2024-02-19T21:05:22.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.700+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec0051a0 osd=0 4 2024-02-19T21:05:22.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.700+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 0 1 2024-02-19T21:05:22.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:22.701 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:22.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:22.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.715+0000 7f9b1277e700 10 client.48803.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:22.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.715+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 2 2024-02-19T21:05:22.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.715+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec0051a0 osd=0 3 2024-02-19T21:05:22.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.715+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec0051a0 osd=0 3 2024-02-19T21:05:22.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.715+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec0051a0 osd=0 3 2024-02-19T21:05:22.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.715+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 2 2024-02-19T21:05:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.715+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec0051a0 osd=0 3 2024-02-19T21:05:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.715+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 0 2 2024-02-19T21:05:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.715+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec0051a0 osd=0 4 2024-02-19T21:05:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.715+0000 7f9b1277e700 10 client.48803.objecter _send_command 2 2024-02-19T21:05:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.715+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9aec004d30 con 0x7f9aec004530 2024-02-19T21:05:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9aec004d30 con 0x7f9aec004530 2024-02-19T21:05:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b097fa700 10 client.48803.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec0051a0 osd=0 4 2024-02-19T21:05:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 0 2 2024-02-19T21:05:22.711 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:05:22.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b1277e700 10 client.48803.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:22.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 3 2024-02-19T21:05:22.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec003fe0 osd=1 3 2024-02-19T21:05:22.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec003fe0 osd=1 3 2024-02-19T21:05:22.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec003fe0 osd=1 3 2024-02-19T21:05:22.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 3 2024-02-19T21:05:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec003fe0 osd=1 3 2024-02-19T21:05:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 1 3 2024-02-19T21:05:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec003fe0 osd=1 4 2024-02-19T21:05:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b1277e700 10 client.48803.objecter _send_command 3 2024-02-19T21:05:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9aec00c000 con 0x7f9aec004150 2024-02-19T21:05:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.717+0000 7f9b097fa700 10 client.48803.objecter ms_handle_connect 0x7f9aec004150 2024-02-19T21:05:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.718+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9aec00c000 con 0x7f9aec004150 2024-02-19T21:05:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.718+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:05:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.718+0000 7f9b097fa700 10 client.48803.objecter _finish_command 3 = system:0 2024-02-19T21:05:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.718+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec003fe0 osd=1 4 2024-02-19T21:05:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.718+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 1 3 2024-02-19T21:05:22.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.732+0000 7f9b1277e700 10 client.48803.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:22.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.732+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 4 2024-02-19T21:05:22.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.732+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec003fe0 osd=1 3 2024-02-19T21:05:22.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.732+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec003fe0 osd=1 3 2024-02-19T21:05:22.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.732+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec003fe0 osd=1 3 2024-02-19T21:05:22.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.732+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 4 2024-02-19T21:05:22.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.732+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec003fe0 osd=1 3 2024-02-19T21:05:22.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.732+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 1 4 2024-02-19T21:05:22.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.732+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec003fe0 osd=1 4 2024-02-19T21:05:22.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.732+0000 7f9b1277e700 10 client.48803.objecter _send_command 4 2024-02-19T21:05:22.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.732+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9aec0018b0 con 0x7f9aec004150 2024-02-19T21:05:22.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9aec0018b0 con 0x7f9aec004150 2024-02-19T21:05:22.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:22.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b097fa700 10 client.48803.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec003fe0 osd=1 4 2024-02-19T21:05:22.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 1 4 2024-02-19T21:05:22.729 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:05:22.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b1277e700 10 client.48803.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:22.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 5 2024-02-19T21:05:22.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec003630 osd=2 3 2024-02-19T21:05:22.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec003630 osd=2 3 2024-02-19T21:05:22.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec003630 osd=2 3 2024-02-19T21:05:22.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 5 2024-02-19T21:05:22.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec003630 osd=2 3 2024-02-19T21:05:22.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 2 5 2024-02-19T21:05:22.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec003630 osd=2 4 2024-02-19T21:05:22.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b1277e700 10 client.48803.objecter _send_command 5 2024-02-19T21:05:22.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9aec00de90 con 0x7f9aec00d2f0 2024-02-19T21:05:22.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.734+0000 7f9b097fa700 10 client.48803.objecter ms_handle_connect 0x7f9aec00d2f0 2024-02-19T21:05:22.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.735+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9aec00de90 con 0x7f9aec00d2f0 2024-02-19T21:05:22.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.735+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:05:22.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.735+0000 7f9b097fa700 10 client.48803.objecter _finish_command 5 = system:0 2024-02-19T21:05:22.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.735+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec003630 osd=2 4 2024-02-19T21:05:22.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.735+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 2 5 2024-02-19T21:05:22.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.749+0000 7f9b1277e700 10 client.48803.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:22.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.749+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 6 2024-02-19T21:05:22.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.749+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec003630 osd=2 3 2024-02-19T21:05:22.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.749+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec003630 osd=2 3 2024-02-19T21:05:22.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.749+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec003630 osd=2 3 2024-02-19T21:05:22.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.749+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 6 2024-02-19T21:05:22.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.749+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec003630 osd=2 3 2024-02-19T21:05:22.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.749+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 2 6 2024-02-19T21:05:22.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.749+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec003630 osd=2 4 2024-02-19T21:05:22.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.749+0000 7f9b1277e700 10 client.48803.objecter _send_command 6 2024-02-19T21:05:22.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.749+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9aec0030a0 con 0x7f9aec00d2f0 2024-02-19T21:05:22.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9aec0030a0 con 0x7f9aec00d2f0 2024-02-19T21:05:22.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:22.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b097fa700 10 client.48803.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec003630 osd=2 4 2024-02-19T21:05:22.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 2 6 2024-02-19T21:05:22.746 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:05:22.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b1277e700 10 client.48803.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:22.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 7 2024-02-19T21:05:22.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec011970 osd=3 3 2024-02-19T21:05:22.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec011970 osd=3 3 2024-02-19T21:05:22.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec011970 osd=3 3 2024-02-19T21:05:22.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 7 2024-02-19T21:05:22.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec011970 osd=3 3 2024-02-19T21:05:22.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 3 7 2024-02-19T21:05:22.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec011970 osd=3 4 2024-02-19T21:05:22.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b1277e700 10 client.48803.objecter _send_command 7 2024-02-19T21:05:22.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9aec015b00 con 0x7f9aec011ff0 2024-02-19T21:05:22.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.751+0000 7f9b097fa700 10 client.48803.objecter ms_handle_connect 0x7f9aec011ff0 2024-02-19T21:05:22.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.752+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9aec015b00 con 0x7f9aec011ff0 2024-02-19T21:05:22.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.752+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:05:22.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.752+0000 7f9b097fa700 10 client.48803.objecter _finish_command 7 = system:0 2024-02-19T21:05:22.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.752+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec011970 osd=3 4 2024-02-19T21:05:22.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.752+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 3 7 2024-02-19T21:05:22.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.766+0000 7f9b1277e700 10 client.48803.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:22.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.766+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 8 2024-02-19T21:05:22.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.766+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec011970 osd=3 3 2024-02-19T21:05:22.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.766+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec011970 osd=3 3 2024-02-19T21:05:22.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.766+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec011970 osd=3 3 2024-02-19T21:05:22.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.766+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 8 2024-02-19T21:05:22.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.766+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec011970 osd=3 3 2024-02-19T21:05:22.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.766+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 3 8 2024-02-19T21:05:22.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.766+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec011970 osd=3 4 2024-02-19T21:05:22.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.766+0000 7f9b1277e700 10 client.48803.objecter _send_command 8 2024-02-19T21:05:22.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.766+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9aec00e240 con 0x7f9aec011ff0 2024-02-19T21:05:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9aec00e240 con 0x7f9aec011ff0 2024-02-19T21:05:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b097fa700 10 client.48803.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec011970 osd=3 4 2024-02-19T21:05:22.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 3 8 2024-02-19T21:05:22.762 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:05:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b1277e700 10 client.48803.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 9 2024-02-19T21:05:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec00e000 osd=4 3 2024-02-19T21:05:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec00e000 osd=4 3 2024-02-19T21:05:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec00e000 osd=4 3 2024-02-19T21:05:22.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 9 2024-02-19T21:05:22.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec00e000 osd=4 3 2024-02-19T21:05:22.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 4 9 2024-02-19T21:05:22.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec00e000 osd=4 4 2024-02-19T21:05:22.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b1277e700 10 client.48803.objecter _send_command 9 2024-02-19T21:05:22.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9aec00e1a0 con 0x7f9aec00ce50 2024-02-19T21:05:22.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.768+0000 7f9b097fa700 10 client.48803.objecter ms_handle_connect 0x7f9aec00ce50 2024-02-19T21:05:22.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.769+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9aec00e1a0 con 0x7f9aec00ce50 2024-02-19T21:05:22.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.769+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:05:22.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.769+0000 7f9b097fa700 10 client.48803.objecter _finish_command 9 = system:0 2024-02-19T21:05:22.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.769+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec00e000 osd=4 4 2024-02-19T21:05:22.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.769+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 4 9 2024-02-19T21:05:22.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.784+0000 7f9b1277e700 10 client.48803.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:22.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.784+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 10 2024-02-19T21:05:22.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.784+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec00e000 osd=4 3 2024-02-19T21:05:22.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.784+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec00e000 osd=4 3 2024-02-19T21:05:22.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.784+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec00e000 osd=4 3 2024-02-19T21:05:22.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.784+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 10 2024-02-19T21:05:22.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.784+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec00e000 osd=4 3 2024-02-19T21:05:22.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.784+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 4 10 2024-02-19T21:05:22.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.784+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec00e000 osd=4 4 2024-02-19T21:05:22.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.784+0000 7f9b1277e700 10 client.48803.objecter _send_command 10 2024-02-19T21:05:22.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.784+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9aec00e1a0 con 0x7f9aec00ce50 2024-02-19T21:05:22.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9aec00e1a0 con 0x7f9aec00ce50 2024-02-19T21:05:22.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:22.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b097fa700 10 client.48803.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec00e000 osd=4 4 2024-02-19T21:05:22.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 4 10 2024-02-19T21:05:22.781 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:05:22.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b1277e700 10 client.48803.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:22.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 11 2024-02-19T21:05:22.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec00e1a0 osd=5 3 2024-02-19T21:05:22.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec00e1a0 osd=5 3 2024-02-19T21:05:22.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec00e1a0 osd=5 3 2024-02-19T21:05:22.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 11 2024-02-19T21:05:22.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec00e1a0 osd=5 3 2024-02-19T21:05:22.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 5 11 2024-02-19T21:05:22.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec00e1a0 osd=5 4 2024-02-19T21:05:22.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b1277e700 10 client.48803.objecter _send_command 11 2024-02-19T21:05:22.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.786+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9aec017fc0 con 0x7f9aec017a60 2024-02-19T21:05:22.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.787+0000 7f9b097fa700 10 client.48803.objecter ms_handle_connect 0x7f9aec017a60 2024-02-19T21:05:22.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.787+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9aec017fc0 con 0x7f9aec017a60 2024-02-19T21:05:22.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.787+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:05:22.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.787+0000 7f9b097fa700 10 client.48803.objecter _finish_command 11 = system:0 2024-02-19T21:05:22.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.787+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec00e1a0 osd=5 4 2024-02-19T21:05:22.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.787+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 5 11 2024-02-19T21:05:22.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.802+0000 7f9b1277e700 10 client.48803.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:22.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.802+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 12 2024-02-19T21:05:22.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.802+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec00e1a0 osd=5 3 2024-02-19T21:05:22.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.802+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec00e1a0 osd=5 3 2024-02-19T21:05:22.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.802+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec00e1a0 osd=5 3 2024-02-19T21:05:22.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.802+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 12 2024-02-19T21:05:22.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.802+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec00e1a0 osd=5 3 2024-02-19T21:05:22.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.802+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 5 12 2024-02-19T21:05:22.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.802+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec00e1a0 osd=5 4 2024-02-19T21:05:22.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.802+0000 7f9b1277e700 10 client.48803.objecter _send_command 12 2024-02-19T21:05:22.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.802+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9aec017fc0 con 0x7f9aec017a60 2024-02-19T21:05:22.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.803+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9aec017fc0 con 0x7f9aec017a60 2024-02-19T21:05:22.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.803+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:22.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.803+0000 7f9b097fa700 10 client.48803.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.803+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec00e1a0 osd=5 4 2024-02-19T21:05:22.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.803+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 5 12 2024-02-19T21:05:22.798 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:05:22.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.804+0000 7f9b1277e700 10 client.48803.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:22.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.804+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 13 2024-02-19T21:05:22.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.804+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec017fc0 osd=6 3 2024-02-19T21:05:22.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.804+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec017fc0 osd=6 3 2024-02-19T21:05:22.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.804+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec017fc0 osd=6 3 2024-02-19T21:05:22.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.804+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 13 2024-02-19T21:05:22.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.804+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec017fc0 osd=6 3 2024-02-19T21:05:22.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.804+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 6 13 2024-02-19T21:05:22.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.804+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec017fc0 osd=6 4 2024-02-19T21:05:22.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.804+0000 7f9b1277e700 10 client.48803.objecter _send_command 13 2024-02-19T21:05:22.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.804+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9aec022c70 con 0x7f9aec01f080 2024-02-19T21:05:22.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.804+0000 7f9b097fa700 10 client.48803.objecter ms_handle_connect 0x7f9aec01f080 2024-02-19T21:05:22.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.805+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9aec022c70 con 0x7f9aec01f080 2024-02-19T21:05:22.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.805+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:05:22.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.805+0000 7f9b097fa700 10 client.48803.objecter _finish_command 13 = system:0 2024-02-19T21:05:22.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.805+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec017fc0 osd=6 4 2024-02-19T21:05:22.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.805+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 6 13 2024-02-19T21:05:22.802 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:05:22.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.819+0000 7f9b1277e700 10 client.48803.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:22.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.819+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 14 2024-02-19T21:05:22.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.819+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec017fc0 osd=6 3 2024-02-19T21:05:22.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.819+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec017fc0 osd=6 3 2024-02-19T21:05:22.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.819+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec017fc0 osd=6 3 2024-02-19T21:05:22.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.819+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 14 2024-02-19T21:05:22.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.819+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec017fc0 osd=6 3 2024-02-19T21:05:22.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.819+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 6 14 2024-02-19T21:05:22.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.819+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec017fc0 osd=6 4 2024-02-19T21:05:22.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.819+0000 7f9b1277e700 10 client.48803.objecter _send_command 14 2024-02-19T21:05:22.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.819+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9aec022c30 con 0x7f9aec01f080 2024-02-19T21:05:22.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.820+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9aec022c30 con 0x7f9aec01f080 2024-02-19T21:05:22.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.820+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:22.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.820+0000 7f9b097fa700 10 client.48803.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.820+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec017fc0 osd=6 4 2024-02-19T21:05:22.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.820+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 6 14 2024-02-19T21:05:22.815 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:05:22.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.821+0000 7f9b1277e700 10 client.48803.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:22.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.821+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 15 2024-02-19T21:05:22.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.821+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec022c30 osd=7 3 2024-02-19T21:05:22.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.821+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec022c30 osd=7 3 2024-02-19T21:05:22.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.821+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec022c30 osd=7 3 2024-02-19T21:05:22.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.821+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 15 2024-02-19T21:05:22.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.821+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec022c30 osd=7 3 2024-02-19T21:05:22.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.821+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 7 15 2024-02-19T21:05:22.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.821+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec022c30 osd=7 4 2024-02-19T21:05:22.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.821+0000 7f9b1277e700 10 client.48803.objecter _send_command 15 2024-02-19T21:05:22.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.821+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9aec024090 con 0x7f9aec023350 2024-02-19T21:05:22.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.821+0000 7f9b097fa700 10 client.48803.objecter ms_handle_connect 0x7f9aec023350 2024-02-19T21:05:22.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.822+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9aec024090 con 0x7f9aec023350 2024-02-19T21:05:22.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.822+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:05:22.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.822+0000 7f9b097fa700 10 client.48803.objecter _finish_command 15 = system:0 2024-02-19T21:05:22.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.822+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec022c30 osd=7 4 2024-02-19T21:05:22.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.822+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 7 15 2024-02-19T21:05:22.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.836+0000 7f9b1277e700 10 client.48803.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:22.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.836+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign -1 16 2024-02-19T21:05:22.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.836+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec022c30 osd=7 3 2024-02-19T21:05:22.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.836+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec022c30 osd=7 3 2024-02-19T21:05:22.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.836+0000 7f9b1277e700 20 client.48803.objecter _get_session s=0x7f9aec022c30 osd=7 3 2024-02-19T21:05:22.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.836+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_remove -1 16 2024-02-19T21:05:22.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.836+0000 7f9b1277e700 20 client.48803.objecter get_session s=0x7f9aec022c30 osd=7 3 2024-02-19T21:05:22.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.836+0000 7f9b1277e700 15 client.48803.objecter _session_command_op_assign 7 16 2024-02-19T21:05:22.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.836+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec022c30 osd=7 4 2024-02-19T21:05:22.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.836+0000 7f9b1277e700 10 client.48803.objecter _send_command 16 2024-02-19T21:05:22.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.836+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9aec024050 con 0x7f9aec023350 2024-02-19T21:05:22.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.837+0000 7f9b097fa700 1 -- 172.21.15.163:0/2116044021 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9aec024050 con 0x7f9aec023350 2024-02-19T21:05:22.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.837+0000 7f9b097fa700 10 client.48803.objecter ms_dispatch 0x7f9b0c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:22.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.837+0000 7f9b097fa700 10 client.48803.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.837+0000 7f9b097fa700 20 client.48803.objecter put_session s=0x7f9aec022c30 osd=7 4 2024-02-19T21:05:22.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.837+0000 7f9b097fa700 15 client.48803.objecter _session_command_op_remove 7 16 2024-02-19T21:05:22.832 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:05:22.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 10 librados: watch_flush enter 2024-02-19T21:05:22.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 10 librados: watch_flush exit 2024-02-19T21:05:22.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 10 client.48803.objecter close_session for osd.0 2024-02-19T21:05:22.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 >> v1:172.21.15.163:6801/1559418115 conn(0x7f9aec004530 legacy=0x7f9aec002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:22.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec0051a0 osd=0 1 2024-02-19T21:05:22.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 10 client.48803.objecter close_session for osd.1 2024-02-19T21:05:22.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 >> v1:172.21.15.163:6805/3125047432 conn(0x7f9aec004150 legacy=0x7f9aec00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:22.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec003fe0 osd=1 1 2024-02-19T21:05:22.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 10 client.48803.objecter close_session for osd.2 2024-02-19T21:05:22.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 >> v1:172.21.15.163:6809/1563158487 conn(0x7f9aec00d2f0 legacy=0x7f9aec00d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:22.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec003630 osd=2 1 2024-02-19T21:05:22.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 10 client.48803.objecter close_session for osd.3 2024-02-19T21:05:22.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 >> v1:172.21.15.163:6813/2112279769 conn(0x7f9aec011ff0 legacy=0x7f9aec0143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:22.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec011970 osd=3 1 2024-02-19T21:05:22.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 10 client.48803.objecter close_session for osd.4 2024-02-19T21:05:22.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 >> v1:172.21.15.193:6800/3129716965 conn(0x7f9aec00ce50 legacy=0x7f9aec016df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:22.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b10d1b700 1 -- 172.21.15.163:0/2116044021 reap_dead start 2024-02-19T21:05:22.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec00e000 osd=4 1 2024-02-19T21:05:22.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 10 client.48803.objecter close_session for osd.5 2024-02-19T21:05:22.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 >> v1:172.21.15.193:6804/805161991 conn(0x7f9aec017a60 legacy=0x7f9aec01d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:22.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec00e1a0 osd=5 1 2024-02-19T21:05:22.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 10 client.48803.objecter close_session for osd.6 2024-02-19T21:05:22.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b10d1b700 1 -- 172.21.15.163:0/2116044021 reap_dead start 2024-02-19T21:05:22.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 >> v1:172.21.15.193:6808/787604615 conn(0x7f9aec01f080 legacy=0x7f9aec021470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:22.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec017fc0 osd=6 1 2024-02-19T21:05:22.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 10 client.48803.objecter close_session for osd.7 2024-02-19T21:05:22.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 >> v1:172.21.15.193:6812/157927432 conn(0x7f9aec023350 legacy=0x7f9aec0237b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:22.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 20 client.48803.objecter put_session s=0x7f9aec022c30 osd=7 1 2024-02-19T21:05:22.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 20 client.48803.objecter shutdown clearing up homeless session... 2024-02-19T21:05:22.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 10 client.48803.objecter successfully canceled tick 2024-02-19T21:05:22.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b10d1b700 1 -- 172.21.15.163:0/2116044021 reap_dead start 2024-02-19T21:05:22.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 >> v1:172.21.15.163:6800/1552514681 conn(0x7f9af8063cb0 legacy=0x7f9af8066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:22.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 >> v1:172.21.15.163:6790/0 conn(0x7f9b0c0f9470 legacy=0x7f9b0c114cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:22.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 shutdown_connections 2024-02-19T21:05:22.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 >> 172.21.15.163:0/2116044021 conn(0x7f9b0c0f2470 msgr2=0x7f9b0c107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:22.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 shutdown_connections 2024-02-19T21:05:22.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 1 -- 172.21.15.163:0/2116044021 wait complete. 2024-02-19T21:05:22.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:22.838+0000 7f9b1277e700 1 librados: shutdown 2024-02-19T21:05:22.836 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.836 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.836 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.836 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.836 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.836 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.836 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.836 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:05:22.837 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:22.910 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:05:23.184 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:05:23.191 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:21.353144+0000 osd.7 (osd.7) 81 : cluster [DBG] 28.5 deep-scrub starts 2024-02-19T21:05:23.191 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:21.356216+0000 osd.7 (osd.7) 82 : cluster [DBG] 28.5 deep-scrub ok 2024-02-19T21:05:23.191 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:21.775325+0000 osd.6 (osd.6) 69 : cluster [DBG] 28.8 deep-scrub starts 2024-02-19T21:05:23.191 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:21.780078+0000 osd.6 (osd.6) 70 : cluster [DBG] 28.8 deep-scrub ok 2024-02-19T21:05:23.191 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:22.137554+0000 osd.2 (osd.2) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:23.192 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:22.137626+0000 osd.2 (osd.2) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:23.192 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:22.167619+0000 mon.a (mon.0) 1133 : audit [INF] from='client.47503 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi163-83854-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:23.192 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:22.167744+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-02-19T21:05:23.192 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:22.671734+0000 mon.c (mon.2) 123 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:23.192 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:22.700068+0000 mon.c (mon.2) 124 : audit [DBG] from='client.? v1:172.21.15.163:0/2116044021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:23.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:23.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:05:23.294 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:23.321 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:23.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:21.353144+0000 osd.7 (osd.7) 81 : cluster [DBG] 28.5 deep-scrub starts 2024-02-19T21:05:23.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:21.356216+0000 osd.7 (osd.7) 82 : cluster [DBG] 28.5 deep-scrub ok 2024-02-19T21:05:23.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:21.775325+0000 osd.6 (osd.6) 69 : cluster [DBG] 28.8 deep-scrub starts 2024-02-19T21:05:23.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:21.780078+0000 osd.6 (osd.6) 70 : cluster [DBG] 28.8 deep-scrub ok 2024-02-19T21:05:23.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:22.137554+0000 osd.2 (osd.2) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:23.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:22.137626+0000 osd.2 (osd.2) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:23.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:22.167619+0000 mon.a (mon.0) 1133 : audit [INF] from='client.47503 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi163-83854-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:23.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:22.167744+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-02-19T21:05:23.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:22.671734+0000 mon.c (mon.2) 123 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:23.442 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:22.700068+0000 mon.c (mon.2) 124 : audit [DBG] from='client.? v1:172.21.15.163:0/2116044021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:23.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:21.353144+0000 osd.7 (osd.7) 81 : cluster [DBG] 28.5 deep-scrub starts 2024-02-19T21:05:23.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:21.356216+0000 osd.7 (osd.7) 82 : cluster [DBG] 28.5 deep-scrub ok 2024-02-19T21:05:23.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:21.775325+0000 osd.6 (osd.6) 69 : cluster [DBG] 28.8 deep-scrub starts 2024-02-19T21:05:23.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:21.780078+0000 osd.6 (osd.6) 70 : cluster [DBG] 28.8 deep-scrub ok 2024-02-19T21:05:23.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:22.137554+0000 osd.2 (osd.2) 69 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:23.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:22.137626+0000 osd.2 (osd.2) 70 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:23.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:22.167619+0000 mon.a (mon.0) 1133 : audit [INF] from='client.47503 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi163-83854-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:23.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:22.167744+0000 mon.a (mon.0) 1134 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-02-19T21:05:23.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:22.671734+0000 mon.c (mon.2) 123 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:23.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:22.700068+0000 mon.c (mon.2) 124 : audit [DBG] from='client.? v1:172.21.15.163:0/2116044021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:23.518 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:23.741 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:05:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:05:23] "GET /metrics HTTP/1.1" 200 206117 "" "Prometheus/2.33.4" 2024-02-19T21:05:23.770 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:23.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:23.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:23.874 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:23.903 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:24.002 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:24.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:24.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:24.132 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:24.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:23.122179+0000 mgr.y (mgr.14550) 188 : cluster [DBG] pgmap v185: 342 pgs: 10 peering, 2 active+clean+snaptrim_wait, 50 unknown, 3 active+clean+snaptrim, 277 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 255 KiB/s wr, 1 op/s 2024-02-19T21:05:24.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:23.173707+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-02-19T21:05:24.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:23.182274+0000 mon.b (mon.1) 226 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:24.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:23.184612+0000 mon.c (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.163:0/983764721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi163-83845-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:24.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:23.185055+0000 mon.a (mon.0) 1136 : audit [INF] from='client.48797 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi163-83845-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:24.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:23.198666+0000 mon.a (mon.0) 1137 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:24.315 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:23.637946+0000 mon.b (mon.1) 227 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:05:24.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:23.672615+0000 mon.c (mon.2) 126 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:24.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:24.091056+0000 mon.a (mon.0) 1138 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:05:24.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:24.151569+0000 mon.b (mon.1) 228 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:05:24.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:24.152288+0000 mon.b (mon.1) 229 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:05:24.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:24.158895+0000 mon.a (mon.0) 1139 : audit [INF] from='client.48797 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi163-83845-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:24.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:24.159023+0000 mon.a (mon.0) 1140 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:24.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:24.159140+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-02-19T21:05:24.316 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:24.163483+0000 mon.a (mon.0) 1142 : audit [INF] from='client.? v1:172.21.15.163:0/3271369549' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi163-83854-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:24.316 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:05:24.317 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:05:24.450 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:23.122179+0000 mgr.y (mgr.14550) 188 : cluster [DBG] pgmap v185: 342 pgs: 10 peering, 2 active+clean+snaptrim_wait, 50 unknown, 3 active+clean+snaptrim, 277 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 255 KiB/s wr, 1 op/s 2024-02-19T21:05:24.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:23.173707+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-02-19T21:05:24.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:23.182274+0000 mon.b (mon.1) 226 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:24.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:23.184612+0000 mon.c (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.163:0/983764721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi163-83845-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:24.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:23.185055+0000 mon.a (mon.0) 1136 : audit [INF] from='client.48797 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi163-83845-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:24.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:23.198666+0000 mon.a (mon.0) 1137 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:24.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:23.637946+0000 mon.b (mon.1) 227 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:05:24.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:23.672615+0000 mon.c (mon.2) 126 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:24.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:24.091056+0000 mon.a (mon.0) 1138 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:05:24.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:24.151569+0000 mon.b (mon.1) 228 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:05:24.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:24.152288+0000 mon.b (mon.1) 229 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:05:24.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:24.158895+0000 mon.a (mon.0) 1139 : audit [INF] from='client.48797 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi163-83845-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:24.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:24.159023+0000 mon.a (mon.0) 1140 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:24.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:24.159140+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-02-19T21:05:24.452 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:24.163483+0000 mon.a (mon.0) 1142 : audit [INF] from='client.? v1:172.21.15.163:0/3271369549' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi163-83854-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:24.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:23.122179+0000 mgr.y (mgr.14550) 188 : cluster [DBG] pgmap v185: 342 pgs: 10 peering, 2 active+clean+snaptrim_wait, 50 unknown, 3 active+clean+snaptrim, 277 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 255 KiB/s wr, 1 op/s 2024-02-19T21:05:24.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:23.173707+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-02-19T21:05:24.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:23.182274+0000 mon.b (mon.1) 226 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:24.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:23.184612+0000 mon.c (mon.2) 125 : audit [INF] from='client.? v1:172.21.15.163:0/983764721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi163-83845-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:24.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:23.185055+0000 mon.a (mon.0) 1136 : audit [INF] from='client.48797 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi163-83845-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:24.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:23.198666+0000 mon.a (mon.0) 1137 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:24.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:23.637946+0000 mon.b (mon.1) 227 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:05:24.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:23.672615+0000 mon.c (mon.2) 126 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:24.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:24.091056+0000 mon.a (mon.0) 1138 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:05:24.452 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:24.151569+0000 mon.b (mon.1) 228 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:05:24.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:24.152288+0000 mon.b (mon.1) 229 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:05:24.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:24.158895+0000 mon.a (mon.0) 1139 : audit [INF] from='client.48797 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi163-83845-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:24.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:24.159023+0000 mon.a (mon.0) 1140 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:24.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:24.159140+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-02-19T21:05:24.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:24.163483+0000 mon.a (mon.0) 1142 : audit [INF] from='client.? v1:172.21.15.163:0/3271369549' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi163-83854-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:24.783 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:24.785 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:24.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:24.887 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:24.919 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:24.994 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:24.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:25.000 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-19T21:05:25.097 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:25.123 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:25.143 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:25.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:25.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:25.196 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:24.173877+0000 mon.a (mon.0) 1143 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:05:25.196 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:24.673490+0000 mon.c (mon.2) 127 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:25.196 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:25.159980+0000 mon.a (mon.0) 1144 : audit [INF] from='client.? v1:172.21.15.163:0/3271369549' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi163-83854-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:25.196 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:25.160212+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-02-19T21:05:25.247 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:05:25.282 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:25.316 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:05:25.318 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:24.173877+0000 mon.a (mon.0) 1143 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:05:25.318 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:24.673490+0000 mon.c (mon.2) 127 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:25.318 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:25.159980+0000 mon.a (mon.0) 1144 : audit [INF] from='client.? v1:172.21.15.163:0/3271369549' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi163-83854-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:25.318 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:25.160212+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-02-19T21:05:25.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:25.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:05:25.319 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:24.173877+0000 mon.a (mon.0) 1143 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:05:25.319 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:24.673490+0000 mon.c (mon.2) 127 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:25.320 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:25.159980+0000 mon.a (mon.0) 1144 : audit [INF] from='client.? v1:172.21.15.163:0/3271369549' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi163-83854-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:25.320 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:25.160212+0000 mon.a (mon.0) 1145 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-02-19T21:05:25.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.344+0000 7f1d4adb6700 1 -- 172.21.15.163:0/3728203482 >> v1:172.21.15.163:6790/0 conn(0x7f1d44100780 legacy=0x7f1d44102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:25.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.344+0000 7f1d4adb6700 1 -- 172.21.15.163:0/3728203482 shutdown_connections 2024-02-19T21:05:25.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.344+0000 7f1d4adb6700 1 -- 172.21.15.163:0/3728203482 >> 172.21.15.163:0/3728203482 conn(0x7f1d440f35a0 msgr2=0x7f1d440f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:25.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.344+0000 7f1d4adb6700 1 -- 172.21.15.163:0/3728203482 shutdown_connections 2024-02-19T21:05:25.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.344+0000 7f1d4adb6700 1 -- 172.21.15.163:0/3728203482 wait complete. 2024-02-19T21:05:25.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d4adb6700 1 librados: starting msgr at 2024-02-19T21:05:25.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d4adb6700 1 librados: starting objecter 2024-02-19T21:05:25.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d4adb6700 1 Processor -- start 2024-02-19T21:05:25.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d4adb6700 1 -- start start 2024-02-19T21:05:25.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d4adb6700 1 librados: setting wanted keys 2024-02-19T21:05:25.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d4adb6700 1 librados: calling monclient init 2024-02-19T21:05:25.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d4adb6700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1d441174b0 con 0x7f1d441042c0 2024-02-19T21:05:25.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d4adb6700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1d44118690 con 0x7f1d44100780 2024-02-19T21:05:25.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d4adb6700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1d44119890 con 0x7f1d440fd980 2024-02-19T21:05:25.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d48b52700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f1d440fd980 0x7f1d440f9c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:40398/0 (socket says 172.21.15.163:40398) 2024-02-19T21:05:25.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d48b52700 1 -- 172.21.15.163:0/618255354 learned_addr learned my addr 172.21.15.163:0/618255354 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:25.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d41ffb700 10 client.?.objecter ms_handle_connect 0x7f1d440fd980 2024-02-19T21:05:25.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d41ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:25.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d41ffb700 10 client.?.objecter ms_handle_connect 0x7f1d44100780 2024-02-19T21:05:25.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d41ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:25.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d41ffb700 10 client.?.objecter ms_handle_connect 0x7f1d441042c0 2024-02-19T21:05:25.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.345+0000 7f1d41ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:25.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894733232 0 0) 0x7f1d44118690 con 0x7f1d44100780 2024-02-19T21:05:25.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d34003630 con 0x7f1d44100780 2024-02-19T21:05:25.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457558564 0 0) 0x7f1d44119890 con 0x7f1d440fd980 2024-02-19T21:05:25.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d44118690 con 0x7f1d440fd980 2024-02-19T21:05:25.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 942500553 0 0) 0x7f1d44118690 con 0x7f1d440fd980 2024-02-19T21:05:25.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d44119890 con 0x7f1d440fd980 2024-02-19T21:05:25.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4093382498 0 0) 0x7f1d34003630 con 0x7f1d44100780 2024-02-19T21:05:25.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d44118690 con 0x7f1d44100780 2024-02-19T21:05:25.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1d38001940 con 0x7f1d440fd980 2024-02-19T21:05:25.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1d30002f00 con 0x7f1d44100780 2024-02-19T21:05:25.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1833591041 0 0) 0x7f1d44119890 con 0x7f1d440fd980 2024-02-19T21:05:25.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 >> v1:172.21.15.193:6789/0 conn(0x7f1d44100780 legacy=0x7f1d440fa350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:25.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 >> v1:172.21.15.163:6789/0 conn(0x7f1d441042c0 legacy=0x7f1d44115b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:25.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d4411aa90 con 0x7f1d440fd980 2024-02-19T21:05:25.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d4adb6700 1 -- 172.21.15.163:0/618255354 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d44118880 con 0x7f1d440fd980 2024-02-19T21:05:25.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f1d38002d20 con 0x7f1d440fd980 2024-02-19T21:05:25.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1d38005bb0 con 0x7f1d440fd980 2024-02-19T21:05:25.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d4adb6700 10 client.48818.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:25.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d4adb6700 1 -- 172.21.15.163:0/618255354 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d44118cf0 con 0x7f1d440fd980 2024-02-19T21:05:25.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.346+0000 7f1d4adb6700 1 librados: init done 2024-02-19T21:05:25.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.347+0000 7f1d4adb6700 1 -- 172.21.15.163:0/618255354 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4405f660 con 0x7f1d440fd980 2024-02-19T21:05:25.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.347+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f1d38001ea0 con 0x7f1d440fd980 2024-02-19T21:05:25.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.348+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(145..145 src has 43..145) v4 ==== 10154+0+0 (unknown 3534433380 0 0) 0x7f1d38079660 con 0x7f1d440fd980 2024-02-19T21:05:25.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.348+0000 7f1d41ffb700 10 client.48818.objecter ms_dispatch 0x7f1d44001550 osd_map(145..145 src has 43..145) v4 2024-02-19T21:05:25.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.348+0000 7f1d41ffb700 3 client.48818.objecter handle_osd_map got epochs [145,145] > 0 2024-02-19T21:05:25.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.348+0000 7f1d41ffb700 3 client.48818.objecter handle_osd_map decoding full epoch 145 2024-02-19T21:05:25.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.348+0000 7f1d41ffb700 20 client.48818.objecter dump_active .. 0 homeless 2024-02-19T21:05:25.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.348+0000 7f1d41ffb700 10 client.48818.objecter ms_handle_connect 0x7f1d34063ad0 2024-02-19T21:05:25.346 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:05:25.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.352+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1d3804c240 con 0x7f1d440fd980 2024-02-19T21:05:25.420 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:05:25.447 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:25.459 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:05:25.467 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:05:25.469 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:25.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:05:25.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:25.537+0000 7f1d4adb6700 1 -- 172.21.15.163:0/618255354 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f1d44047540 con 0x7f1d440fd980 2024-02-19T21:05:25.570 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:05:25.596 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:25.614 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:05:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:25.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:05:25.702 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:05:25.702 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:05:25.702 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-19T21:05:25.703 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-19T21:05:25.718 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:25.733 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:25.761 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:25.967 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:25.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:25.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:26.071 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:05:26.099 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:26.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:26.173+0000 7f1d41ffb700 1 -- 172.21.15.163:0/618255354 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v146) v1 ==== 97+0+0 (unknown 3500042103 0 0) 0x7f1d3804a650 con 0x7f1d440fd980 2024-02-19T21:05:26.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:26.177+0000 7f1d4adb6700 10 librados: watch_flush enter 2024-02-19T21:05:26.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:26.177+0000 7f1d4adb6700 10 librados: watch_flush exit 2024-02-19T21:05:26.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:26.177+0000 7f1d4adb6700 20 client.48818.objecter shutdown clearing up homeless session... 2024-02-19T21:05:26.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:26.177+0000 7f1d4adb6700 10 client.48818.objecter successfully canceled tick 2024-02-19T21:05:26.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:26.180+0000 7f1d4adb6700 1 -- 172.21.15.163:0/618255354 >> v1:172.21.15.163:6800/1552514681 conn(0x7f1d34063ad0 legacy=0x7f1d34065f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:26.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:26.180+0000 7f1d4adb6700 1 -- 172.21.15.163:0/618255354 >> v1:172.21.15.163:6790/0 conn(0x7f1d440fd980 legacy=0x7f1d440f9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:26.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:26.180+0000 7f1d4adb6700 1 -- 172.21.15.163:0/618255354 shutdown_connections 2024-02-19T21:05:26.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:26.180+0000 7f1d4adb6700 1 -- 172.21.15.163:0/618255354 >> 172.21.15.163:0/618255354 conn(0x7f1d440f35a0 msgr2=0x7f1d44107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:26.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:26.185+0000 7f1d4adb6700 1 -- 172.21.15.163:0/618255354 shutdown_connections 2024-02-19T21:05:26.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:26.186+0000 7f1d4adb6700 1 -- 172.21.15.163:0/618255354 wait complete. 2024-02-19T21:05:26.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:26.187+0000 7f1d4adb6700 1 librados: shutdown 2024-02-19T21:05:26.186 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:05:26.198 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:05:26.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:26.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:05:26.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:24.906035+0000 osd.1 (osd.1) 91 : cluster [DBG] 28.7 deep-scrub starts 2024-02-19T21:05:26.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:24.947287+0000 osd.1 (osd.1) 92 : cluster [DBG] 28.7 deep-scrub ok 2024-02-19T21:05:26.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:25.123538+0000 mgr.y (mgr.14550) 189 : cluster [DBG] pgmap v188: 406 pgs: 32 creating+peering, 1 active+clean+snaptrim, 1 active+clean+scrubbing+deep, 372 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s wr, 24 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:05:26.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:25.186973+0000 mon.a (mon.0) 1146 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:05:26.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:25.538615+0000 mon.c (mon.2) 128 : audit [INF] from='client.? v1:172.21.15.163:0/618255354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:26.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:25.538955+0000 mon.a (mon.0) 1147 : audit [INF] from='client.48818 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:26.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:25.674184+0000 mon.c (mon.2) 129 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:26.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:26.157527+0000 mon.a (mon.0) 1148 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:05:26.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:26.167764+0000 mon.a (mon.0) 1149 : audit [INF] from='client.48818 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:05:26.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:26.167831+0000 mon.a (mon.0) 1150 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-02-19T21:05:26.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:26.178276+0000 mon.b (mon.1) 230 : audit [INF] from='client.? v1:172.21.15.163:0/921219709' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi163-83845-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:26.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:26.179853+0000 mon.a (mon.0) 1151 : audit [INF] from='client.47605 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi163-83845-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:26.242 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:24.906035+0000 osd.1 (osd.1) 91 : cluster [DBG] 28.7 deep-scrub starts 2024-02-19T21:05:26.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:24.947287+0000 osd.1 (osd.1) 92 : cluster [DBG] 28.7 deep-scrub ok 2024-02-19T21:05:26.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:25.123538+0000 mgr.y (mgr.14550) 189 : cluster [DBG] pgmap v188: 406 pgs: 32 creating+peering, 1 active+clean+snaptrim, 1 active+clean+scrubbing+deep, 372 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s wr, 24 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:05:26.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:25.186973+0000 mon.a (mon.0) 1146 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:05:26.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:25.538615+0000 mon.c (mon.2) 128 : audit [INF] from='client.? v1:172.21.15.163:0/618255354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:26.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:25.538955+0000 mon.a (mon.0) 1147 : audit [INF] from='client.48818 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:26.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:25.674184+0000 mon.c (mon.2) 129 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:26.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:26.157527+0000 mon.a (mon.0) 1148 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:05:26.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:26.167764+0000 mon.a (mon.0) 1149 : audit [INF] from='client.48818 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:05:26.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:26.167831+0000 mon.a (mon.0) 1150 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-02-19T21:05:26.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:26.178276+0000 mon.b (mon.1) 230 : audit [INF] from='client.? v1:172.21.15.163:0/921219709' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi163-83845-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:26.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:26.179853+0000 mon.a (mon.0) 1151 : audit [INF] from='client.47605 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi163-83845-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:26.302 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:26.328 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:26.377 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:24.906035+0000 osd.1 (osd.1) 91 : cluster [DBG] 28.7 deep-scrub starts 2024-02-19T21:05:26.377 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:24.947287+0000 osd.1 (osd.1) 92 : cluster [DBG] 28.7 deep-scrub ok 2024-02-19T21:05:26.377 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:25.123538+0000 mgr.y (mgr.14550) 189 : cluster [DBG] pgmap v188: 406 pgs: 32 creating+peering, 1 active+clean+snaptrim, 1 active+clean+scrubbing+deep, 372 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s wr, 24 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:05:26.377 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:25.186973+0000 mon.a (mon.0) 1146 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:05:26.377 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:25.538615+0000 mon.c (mon.2) 128 : audit [INF] from='client.? v1:172.21.15.163:0/618255354' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:26.377 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:25.538955+0000 mon.a (mon.0) 1147 : audit [INF] from='client.48818 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:26.377 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:25.674184+0000 mon.c (mon.2) 129 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:26.377 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:26.157527+0000 mon.a (mon.0) 1148 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:05:26.377 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:26.167764+0000 mon.a (mon.0) 1149 : audit [INF] from='client.48818 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:05:26.377 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:26.167831+0000 mon.a (mon.0) 1150 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-02-19T21:05:26.377 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:26.178276+0000 mon.b (mon.1) 230 : audit [INF] from='client.? v1:172.21.15.163:0/921219709' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi163-83845-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:26.377 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:26.179853+0000 mon.a (mon.0) 1151 : audit [INF] from='client.47605 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi163-83845-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:26.380 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:26.387 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:26.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:26.489 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:26.515 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:26.762 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:26.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:26.865 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:26.912 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:27.036 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:27.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:27.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:27.140 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:27.183 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:27.287 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:27.289 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:26.244566+0000 osd.7 (osd.7) 83 : cluster [DBG] 28.6 deep-scrub starts 2024-02-19T21:05:27.289 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:26.248619+0000 osd.7 (osd.7) 84 : cluster [DBG] 28.6 deep-scrub ok 2024-02-19T21:05:27.289 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:26.674973+0000 mon.c (mon.2) 130 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:27.289 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:27.164584+0000 mon.a (mon.0) 1152 : audit [INF] from='client.47605 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi163-83845-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:27.289 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:27.164795+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-02-19T21:05:27.289 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:27.179097+0000 mon.b (mon.1) 231 : audit [INF] from='client.? v1:172.21.15.163:0/736896345' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi163-83854-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:27.289 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:27.180148+0000 mon.a (mon.0) 1154 : audit [INF] from='client.47617 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi163-83854-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:27.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:27.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:27.311 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:05:27.311 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:05:27.391 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:27.420 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:27.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:26.244566+0000 osd.7 (osd.7) 83 : cluster [DBG] 28.6 deep-scrub starts 2024-02-19T21:05:27.492 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:26.248619+0000 osd.7 (osd.7) 84 : cluster [DBG] 28.6 deep-scrub ok 2024-02-19T21:05:27.492 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:26.674973+0000 mon.c (mon.2) 130 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:27.492 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:27.164584+0000 mon.a (mon.0) 1152 : audit [INF] from='client.47605 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi163-83845-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:27.492 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:27.164795+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-02-19T21:05:27.492 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:27.179097+0000 mon.b (mon.1) 231 : audit [INF] from='client.? v1:172.21.15.163:0/736896345' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi163-83854-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:27.492 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:27.180148+0000 mon.a (mon.0) 1154 : audit [INF] from='client.47617 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi163-83854-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:27.492 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:26.244566+0000 osd.7 (osd.7) 83 : cluster [DBG] 28.6 deep-scrub starts 2024-02-19T21:05:27.492 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:26.248619+0000 osd.7 (osd.7) 84 : cluster [DBG] 28.6 deep-scrub ok 2024-02-19T21:05:27.492 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:26.674973+0000 mon.c (mon.2) 130 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:27.492 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:27.164584+0000 mon.a (mon.0) 1152 : audit [INF] from='client.47605 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi163-83845-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:27.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:27.164795+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-02-19T21:05:27.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:27.179097+0000 mon.b (mon.1) 231 : audit [INF] from='client.? v1:172.21.15.163:0/736896345' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi163-83854-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:27.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:27.180148+0000 mon.a (mon.0) 1154 : audit [INF] from='client.47617 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi163-83854-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:27.493 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:27.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:27.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:27.596 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:27.623 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:27.674 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:27.907 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:05:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:05:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:05:27.925 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-19T21:05:27.971 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:27.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:28.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:05:28.105 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:28.158 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:05:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:28.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:05:28.209 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:27.124032+0000 mgr.y (mgr.14550) 190 : cluster [DBG] pgmap v191: 374 pgs: 32 unknown, 1 active+clean+snaptrim, 1 active+clean+scrubbing+deep, 340 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 30 KiB/s wr, 28 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:05:28.209 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:27.675753+0000 mon.c (mon.2) 131 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:28.209 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:28.170464+0000 mon.a (mon.0) 1155 : audit [INF] from='client.47617 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi163-83854-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:28.209 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:28.170602+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-02-19T21:05:28.209 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:27.124032+0000 mgr.y (mgr.14550) 190 : cluster [DBG] pgmap v191: 374 pgs: 32 unknown, 1 active+clean+snaptrim, 1 active+clean+scrubbing+deep, 340 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 30 KiB/s wr, 28 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:05:28.209 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:27.675753+0000 mon.c (mon.2) 131 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:28.209 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:28.170464+0000 mon.a (mon.0) 1155 : audit [INF] from='client.47617 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi163-83854-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:28.209 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:28.170602+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-02-19T21:05:28.259 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:05:28.264 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:28.296 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:28.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.423+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/248192052 >> v1:172.21.15.163:6790/0 conn(0x7ff0f8100780 legacy=0x7ff0f8102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:28.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.423+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/248192052 shutdown_connections 2024-02-19T21:05:28.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.423+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/248192052 >> 172.21.15.163:0/248192052 conn(0x7ff0f80f35a0 msgr2=0x7ff0f80f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:28.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.423+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/248192052 shutdown_connections 2024-02-19T21:05:28.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.423+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/248192052 wait complete. 2024-02-19T21:05:28.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.423+0000 7ff0fdcc8700 1 librados: starting msgr at 2024-02-19T21:05:28.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.423+0000 7ff0fdcc8700 1 librados: starting objecter 2024-02-19T21:05:28.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.423+0000 7ff0fdcc8700 1 Processor -- start 2024-02-19T21:05:28.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0fdcc8700 1 -- start start 2024-02-19T21:05:28.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0fdcc8700 1 librados: setting wanted keys 2024-02-19T21:05:28.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0fdcc8700 1 librados: calling monclient init 2024-02-19T21:05:28.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0fdcc8700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0f8117500 con 0x7ff0f8100780 2024-02-19T21:05:28.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0fdcc8700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0f81186e0 con 0x7ff0f80fd980 2024-02-19T21:05:28.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0fdcc8700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0f81198e0 con 0x7ff0f81042c0 2024-02-19T21:05:28.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0f6ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ff0f8100780 0x7ff0f80fa410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:36038/0 (socket says 172.21.15.163:36038) 2024-02-19T21:05:28.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0f6ffd700 1 -- 172.21.15.163:0/4094168549 learned_addr learned my addr 172.21.15.163:0/4094168549 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:28.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0f7fff700 1 --1- 172.21.15.163:0/4094168549 >> v1:172.21.15.163:6790/0 conn(0x7ff0f81042c0 0x7ff0f8115bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:40452/0 (socket says 172.21.15.163:40452) 2024-02-19T21:05:28.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0f4ff9700 10 client.?.objecter ms_handle_connect 0x7ff0f80fd980 2024-02-19T21:05:28.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0f4ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:28.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0f4ff9700 10 client.?.objecter ms_handle_connect 0x7ff0f81042c0 2024-02-19T21:05:28.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0f4ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:28.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0f4ff9700 10 client.?.objecter ms_handle_connect 0x7ff0f8100780 2024-02-19T21:05:28.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0f4ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:28.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84294265 0 0) 0x7ff0f81186e0 con 0x7ff0f80fd980 2024-02-19T21:05:28.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0e0003030 con 0x7ff0f80fd980 2024-02-19T21:05:28.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474472134 0 0) 0x7ff0f81198e0 con 0x7ff0f81042c0 2024-02-19T21:05:28.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.424+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0f81186e0 con 0x7ff0f81042c0 2024-02-19T21:05:28.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2835240960 0 0) 0x7ff0e0003030 con 0x7ff0f80fd980 2024-02-19T21:05:28.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0f81198e0 con 0x7ff0f80fd980 2024-02-19T21:05:28.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff0e80017f0 con 0x7ff0f80fd980 2024-02-19T21:05:28.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4274145742 0 0) 0x7ff0f81186e0 con 0x7ff0f81042c0 2024-02-19T21:05:28.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0e0003030 con 0x7ff0f81042c0 2024-02-19T21:05:28.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff0f0001cd0 con 0x7ff0f81042c0 2024-02-19T21:05:28.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 29283520 0 0) 0x7ff0f81198e0 con 0x7ff0f80fd980 2024-02-19T21:05:28.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 >> v1:172.21.15.163:6790/0 conn(0x7ff0f81042c0 legacy=0x7ff0f8115bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:28.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 >> v1:172.21.15.163:6789/0 conn(0x7ff0f8100780 legacy=0x7ff0f80fa410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:28.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f811aae0 con 0x7ff0f80fd980 2024-02-19T21:05:28.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f8119ad0 con 0x7ff0f80fd980 2024-02-19T21:05:28.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0fdcc8700 10 client.47626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:28.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f8119f80 con 0x7ff0f80fd980 2024-02-19T21:05:28.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0fdcc8700 1 librados: init done 2024-02-19T21:05:28.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff0e8001eb0 con 0x7ff0f80fd980 2024-02-19T21:05:28.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.425+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff0e8003300 con 0x7ff0f80fd980 2024-02-19T21:05:28.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.426+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff0e4004f00 con 0x7ff0f80fd980 2024-02-19T21:05:28.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ff0e8001a10 con 0x7ff0f80fd980 2024-02-19T21:05:28.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(148..148 src has 43..148) v4 ==== 10166+0+0 (unknown 2480130053 0 0) 0x7ff0e8076bc0 con 0x7ff0f80fd980 2024-02-19T21:05:28.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 osd_map(148..148 src has 43..148) v4 2024-02-19T21:05:28.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0f4ff9700 3 client.47626.objecter handle_osd_map got epochs [148,148] > 0 2024-02-19T21:05:28.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0f4ff9700 3 client.47626.objecter handle_osd_map decoding full epoch 148 2024-02-19T21:05:28.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0f4ff9700 20 client.47626.objecter dump_active .. 0 homeless 2024-02-19T21:05:28.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v148) v1 ==== 54+0+16 (unknown 2551583071 0 2420000192) 0x7ff0e8077060 con 0x7ff0f80fd980 2024-02-19T21:05:28.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0f4ff9700 10 client.47626.objecter ms_handle_connect 0x7ff0e0063ce0 2024-02-19T21:05:28.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:28.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 1 2024-02-19T21:05:28.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e40051a0 osd=0 3 2024-02-19T21:05:28.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e40051a0 osd=0 3 2024-02-19T21:05:28.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e40051a0 osd=0 3 2024-02-19T21:05:28.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 1 2024-02-19T21:05:28.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e40051a0 osd=0 3 2024-02-19T21:05:28.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 0 1 2024-02-19T21:05:28.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e40051a0 osd=0 4 2024-02-19T21:05:28.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 1 2024-02-19T21:05:28.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.427+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e4004950 con 0x7ff0e4004530 2024-02-19T21:05:28.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.428+0000 7ff0f4ff9700 10 client.47626.objecter ms_handle_connect 0x7ff0e4004530 2024-02-19T21:05:28.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.429+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff0e4004950 con 0x7ff0e4004530 2024-02-19T21:05:28.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.429+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:05:28.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.429+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 1 = system:0 2024-02-19T21:05:28.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.429+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e40051a0 osd=0 4 2024-02-19T21:05:28.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.429+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 0 1 2024-02-19T21:05:28.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.444+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:28.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.444+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 2 2024-02-19T21:05:28.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.444+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e40051a0 osd=0 3 2024-02-19T21:05:28.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.444+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e40051a0 osd=0 3 2024-02-19T21:05:28.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.444+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e40051a0 osd=0 3 2024-02-19T21:05:28.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.444+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 2 2024-02-19T21:05:28.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.444+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e40051a0 osd=0 3 2024-02-19T21:05:28.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.444+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 0 2 2024-02-19T21:05:28.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.444+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e40051a0 osd=0 4 2024-02-19T21:05:28.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.444+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 2 2024-02-19T21:05:28.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.444+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff0e4004d30 con 0x7ff0e4004530 2024-02-19T21:05:28.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff0e4004d30 con 0x7ff0e4004530 2024-02-19T21:05:28.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:28.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:28.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e40051a0 osd=0 4 2024-02-19T21:05:28.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 0 2 2024-02-19T21:05:28.440 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:05:28.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:28.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 3 2024-02-19T21:05:28.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e4004270 osd=1 3 2024-02-19T21:05:28.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e4004270 osd=1 3 2024-02-19T21:05:28.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e4004270 osd=1 3 2024-02-19T21:05:28.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 3 2024-02-19T21:05:28.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e4004270 osd=1 3 2024-02-19T21:05:28.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 1 3 2024-02-19T21:05:28.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e4004270 osd=1 4 2024-02-19T21:05:28.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 3 2024-02-19T21:05:28.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e400bf60 con 0x7ff0e4003b40 2024-02-19T21:05:28.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.446+0000 7ff0f4ff9700 10 client.47626.objecter ms_handle_connect 0x7ff0e4003b40 2024-02-19T21:05:28.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.447+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff0e400bf60 con 0x7ff0e4003b40 2024-02-19T21:05:28.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.447+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:05:28.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.447+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 3 = system:0 2024-02-19T21:05:28.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.447+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e4004270 osd=1 4 2024-02-19T21:05:28.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.447+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 1 3 2024-02-19T21:05:28.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.461+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:28.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.461+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 4 2024-02-19T21:05:28.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.461+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e4004270 osd=1 3 2024-02-19T21:05:28.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.461+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e4004270 osd=1 3 2024-02-19T21:05:28.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.461+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e4004270 osd=1 3 2024-02-19T21:05:28.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.461+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 4 2024-02-19T21:05:28.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.461+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e4004270 osd=1 3 2024-02-19T21:05:28.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.461+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 1 4 2024-02-19T21:05:28.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.461+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e4004270 osd=1 4 2024-02-19T21:05:28.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.461+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 4 2024-02-19T21:05:28.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.461+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff0e400d280 con 0x7ff0e4003b40 2024-02-19T21:05:28.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff0e400d280 con 0x7ff0e4003b40 2024-02-19T21:05:28.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:28.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:28.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e4004270 osd=1 4 2024-02-19T21:05:28.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 1 4 2024-02-19T21:05:28.457 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:05:28.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:28.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 5 2024-02-19T21:05:28.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e40033a0 osd=2 3 2024-02-19T21:05:28.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e40033a0 osd=2 3 2024-02-19T21:05:28.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e40033a0 osd=2 3 2024-02-19T21:05:28.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 5 2024-02-19T21:05:28.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e40033a0 osd=2 3 2024-02-19T21:05:28.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 2 5 2024-02-19T21:05:28.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e40033a0 osd=2 4 2024-02-19T21:05:28.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 5 2024-02-19T21:05:28.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e4010d30 con 0x7ff0e4002070 2024-02-19T21:05:28.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.463+0000 7ff0f4ff9700 10 client.47626.objecter ms_handle_connect 0x7ff0e4002070 2024-02-19T21:05:28.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.464+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff0e4010d30 con 0x7ff0e4002070 2024-02-19T21:05:28.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.464+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:05:28.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.464+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 5 = system:0 2024-02-19T21:05:28.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.464+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e40033a0 osd=2 4 2024-02-19T21:05:28.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.464+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 2 5 2024-02-19T21:05:28.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.478+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:28.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.478+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 6 2024-02-19T21:05:28.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.478+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e40033a0 osd=2 3 2024-02-19T21:05:28.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.479+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e40033a0 osd=2 3 2024-02-19T21:05:28.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.479+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e40033a0 osd=2 3 2024-02-19T21:05:28.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.479+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 6 2024-02-19T21:05:28.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.479+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e40033a0 osd=2 3 2024-02-19T21:05:28.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.479+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 2 6 2024-02-19T21:05:28.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.479+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e40033a0 osd=2 4 2024-02-19T21:05:28.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.479+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 6 2024-02-19T21:05:28.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.479+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff0e4001630 con 0x7ff0e4002070 2024-02-19T21:05:28.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff0e4001630 con 0x7ff0e4002070 2024-02-19T21:05:28.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:28.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:28.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e40033a0 osd=2 4 2024-02-19T21:05:28.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 2 6 2024-02-19T21:05:28.474 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:05:28.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:28.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 7 2024-02-19T21:05:28.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e400c8d0 osd=3 3 2024-02-19T21:05:28.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e400c8d0 osd=3 3 2024-02-19T21:05:28.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e400c8d0 osd=3 3 2024-02-19T21:05:28.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 7 2024-02-19T21:05:28.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e400c8d0 osd=3 3 2024-02-19T21:05:28.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 3 7 2024-02-19T21:05:28.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e400c8d0 osd=3 4 2024-02-19T21:05:28.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 7 2024-02-19T21:05:28.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.480+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e400d0b0 con 0x7ff0e4010ed0 2024-02-19T21:05:28.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.481+0000 7ff0f4ff9700 10 client.47626.objecter ms_handle_connect 0x7ff0e4010ed0 2024-02-19T21:05:28.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.481+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff0e400d0b0 con 0x7ff0e4010ed0 2024-02-19T21:05:28.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.481+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:05:28.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.481+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 7 = system:0 2024-02-19T21:05:28.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.481+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e400c8d0 osd=3 4 2024-02-19T21:05:28.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.481+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 3 7 2024-02-19T21:05:28.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.495+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:28.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.495+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 8 2024-02-19T21:05:28.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.495+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e400c8d0 osd=3 3 2024-02-19T21:05:28.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.495+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e400c8d0 osd=3 3 2024-02-19T21:05:28.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.495+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e400c8d0 osd=3 3 2024-02-19T21:05:28.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.495+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 8 2024-02-19T21:05:28.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.495+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e400c8d0 osd=3 3 2024-02-19T21:05:28.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.495+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 3 8 2024-02-19T21:05:28.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.495+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e400c8d0 osd=3 4 2024-02-19T21:05:28.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.495+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 8 2024-02-19T21:05:28.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.495+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff0e4012260 con 0x7ff0e4010ed0 2024-02-19T21:05:28.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff0e4012260 con 0x7ff0e4010ed0 2024-02-19T21:05:28.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e400c8d0 osd=3 4 2024-02-19T21:05:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 3 8 2024-02-19T21:05:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 9 2024-02-19T21:05:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e4012260 osd=4 3 2024-02-19T21:05:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e4012260 osd=4 3 2024-02-19T21:05:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e4012260 osd=4 3 2024-02-19T21:05:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 9 2024-02-19T21:05:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e4012260 osd=4 3 2024-02-19T21:05:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 4 9 2024-02-19T21:05:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e4012260 osd=4 4 2024-02-19T21:05:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 9 2024-02-19T21:05:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.497+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e401e6e0 con 0x7ff0e401ab00 2024-02-19T21:05:28.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.498+0000 7ff0f4ff9700 10 client.47626.objecter ms_handle_connect 0x7ff0e401ab00 2024-02-19T21:05:28.493 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:05:28.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:27.124032+0000 mgr.y (mgr.14550) 190 : cluster [DBG] pgmap v191: 374 pgs: 32 unknown, 1 active+clean+snaptrim, 1 active+clean+scrubbing+deep, 340 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 30 KiB/s wr, 28 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:05:28.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:27.675753+0000 mon.c (mon.2) 131 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:28.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:28.170464+0000 mon.a (mon.0) 1155 : audit [INF] from='client.47617 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi163-83854-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:28.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:28.170602+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-02-19T21:05:28.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.498+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff0e401e6e0 con 0x7ff0e401ab00 2024-02-19T21:05:28.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.498+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:05:28.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.498+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 9 = system:0 2024-02-19T21:05:28.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.498+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e4012260 osd=4 4 2024-02-19T21:05:28.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.498+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 4 9 2024-02-19T21:05:28.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.513+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:28.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.513+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 10 2024-02-19T21:05:28.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.513+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e4012260 osd=4 3 2024-02-19T21:05:28.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.513+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e4012260 osd=4 3 2024-02-19T21:05:28.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.513+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e4012260 osd=4 3 2024-02-19T21:05:28.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.513+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 10 2024-02-19T21:05:28.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.513+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e4012260 osd=4 3 2024-02-19T21:05:28.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.513+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 4 10 2024-02-19T21:05:28.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.513+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e4012260 osd=4 4 2024-02-19T21:05:28.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.513+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 10 2024-02-19T21:05:28.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.513+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff0e401e6e0 con 0x7ff0e401ab00 2024-02-19T21:05:28.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff0e401e6e0 con 0x7ff0e401ab00 2024-02-19T21:05:28.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:28.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:28.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e4012260 osd=4 4 2024-02-19T21:05:28.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 4 10 2024-02-19T21:05:28.509 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:05:28.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:28.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 11 2024-02-19T21:05:28.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e401e6e0 osd=5 3 2024-02-19T21:05:28.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e401e6e0 osd=5 3 2024-02-19T21:05:28.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e401e6e0 osd=5 3 2024-02-19T21:05:28.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 11 2024-02-19T21:05:28.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e401e6e0 osd=5 3 2024-02-19T21:05:28.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 5 11 2024-02-19T21:05:28.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e401e6e0 osd=5 4 2024-02-19T21:05:28.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 11 2024-02-19T21:05:28.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e401f930 con 0x7ff0e40203e0 2024-02-19T21:05:28.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.515+0000 7ff0f4ff9700 10 client.47626.objecter ms_handle_connect 0x7ff0e40203e0 2024-02-19T21:05:28.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.516+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff0e401f930 con 0x7ff0e40203e0 2024-02-19T21:05:28.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.516+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:05:28.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.516+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 11 = system:0 2024-02-19T21:05:28.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.516+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e401e6e0 osd=5 4 2024-02-19T21:05:28.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.516+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 5 11 2024-02-19T21:05:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.530+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.530+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 12 2024-02-19T21:05:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.530+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e401e6e0 osd=5 3 2024-02-19T21:05:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.530+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e401e6e0 osd=5 3 2024-02-19T21:05:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.530+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e401e6e0 osd=5 3 2024-02-19T21:05:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.530+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 12 2024-02-19T21:05:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.530+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e401e6e0 osd=5 3 2024-02-19T21:05:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.530+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 5 12 2024-02-19T21:05:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.530+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e401e6e0 osd=5 4 2024-02-19T21:05:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.530+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 12 2024-02-19T21:05:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.530+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff0e401f930 con 0x7ff0e40203e0 2024-02-19T21:05:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff0e401f930 con 0x7ff0e40203e0 2024-02-19T21:05:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e401e6e0 osd=5 4 2024-02-19T21:05:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 5 12 2024-02-19T21:05:28.526 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:05:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 13 2024-02-19T21:05:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e401f930 osd=6 3 2024-02-19T21:05:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e401f930 osd=6 3 2024-02-19T21:05:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e401f930 osd=6 3 2024-02-19T21:05:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 13 2024-02-19T21:05:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e401f930 osd=6 3 2024-02-19T21:05:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 6 13 2024-02-19T21:05:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e401f930 osd=6 4 2024-02-19T21:05:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 13 2024-02-19T21:05:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.532+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e40270b0 con 0x7ff0e40235a0 2024-02-19T21:05:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.533+0000 7ff0f4ff9700 10 client.47626.objecter ms_handle_connect 0x7ff0e40235a0 2024-02-19T21:05:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.534+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff0e40270b0 con 0x7ff0e40235a0 2024-02-19T21:05:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.534+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:05:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.534+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 13 = system:0 2024-02-19T21:05:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.534+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e401f930 osd=6 4 2024-02-19T21:05:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.534+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 6 13 2024-02-19T21:05:28.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.548+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:28.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.548+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 14 2024-02-19T21:05:28.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.548+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e401f930 osd=6 3 2024-02-19T21:05:28.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.548+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e401f930 osd=6 3 2024-02-19T21:05:28.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.548+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e401f930 osd=6 3 2024-02-19T21:05:28.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.548+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 14 2024-02-19T21:05:28.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.548+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e401f930 osd=6 3 2024-02-19T21:05:28.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.548+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 6 14 2024-02-19T21:05:28.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.548+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e401f930 osd=6 4 2024-02-19T21:05:28.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.548+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 14 2024-02-19T21:05:28.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.548+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff0e40270b0 con 0x7ff0e40235a0 2024-02-19T21:05:28.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff0e40270b0 con 0x7ff0e40235a0 2024-02-19T21:05:28.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:28.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:28.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e401f930 osd=6 4 2024-02-19T21:05:28.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 6 14 2024-02-19T21:05:28.543 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:05:28.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:28.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 15 2024-02-19T21:05:28.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e40270b0 osd=7 3 2024-02-19T21:05:28.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e40270b0 osd=7 3 2024-02-19T21:05:28.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e40270b0 osd=7 3 2024-02-19T21:05:28.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 15 2024-02-19T21:05:28.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e40270b0 osd=7 3 2024-02-19T21:05:28.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 7 15 2024-02-19T21:05:28.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e40270b0 osd=7 4 2024-02-19T21:05:28.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 15 2024-02-19T21:05:28.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0e402b400 con 0x7ff0e4027730 2024-02-19T21:05:28.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.550+0000 7ff0f4ff9700 10 client.47626.objecter ms_handle_connect 0x7ff0e4027730 2024-02-19T21:05:28.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.551+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff0e402b400 con 0x7ff0e4027730 2024-02-19T21:05:28.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.551+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:05:28.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.551+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 15 = system:0 2024-02-19T21:05:28.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.551+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e40270b0 osd=7 4 2024-02-19T21:05:28.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.551+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 7 15 2024-02-19T21:05:28.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.565+0000 7ff0fdcc8700 10 client.47626.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:28.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.565+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign -1 16 2024-02-19T21:05:28.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.565+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e40270b0 osd=7 3 2024-02-19T21:05:28.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.565+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e40270b0 osd=7 3 2024-02-19T21:05:28.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.565+0000 7ff0fdcc8700 20 client.47626.objecter _get_session s=0x7ff0e40270b0 osd=7 3 2024-02-19T21:05:28.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.565+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_remove -1 16 2024-02-19T21:05:28.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.565+0000 7ff0fdcc8700 20 client.47626.objecter get_session s=0x7ff0e40270b0 osd=7 3 2024-02-19T21:05:28.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.565+0000 7ff0fdcc8700 15 client.47626.objecter _session_command_op_assign 7 16 2024-02-19T21:05:28.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.565+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e40270b0 osd=7 4 2024-02-19T21:05:28.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.565+0000 7ff0fdcc8700 10 client.47626.objecter _send_command 16 2024-02-19T21:05:28.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.566+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff0e402cbb0 con 0x7ff0e4027730 2024-02-19T21:05:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0f4ff9700 1 -- 172.21.15.163:0/4094168549 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff0e402cbb0 con 0x7ff0e4027730 2024-02-19T21:05:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0f4ff9700 10 client.47626.objecter ms_dispatch 0x7ff0f8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0f4ff9700 10 client.47626.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0f4ff9700 20 client.47626.objecter put_session s=0x7ff0e40270b0 osd=7 4 2024-02-19T21:05:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0f4ff9700 15 client.47626.objecter _session_command_op_remove 7 16 2024-02-19T21:05:28.561 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:05:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 10 librados: watch_flush enter 2024-02-19T21:05:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 10 librados: watch_flush exit 2024-02-19T21:05:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 10 client.47626.objecter close_session for osd.0 2024-02-19T21:05:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 >> v1:172.21.15.163:6801/1559418115 conn(0x7ff0e4004530 legacy=0x7ff0e4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e40051a0 osd=0 1 2024-02-19T21:05:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 10 client.47626.objecter close_session for osd.1 2024-02-19T21:05:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 >> v1:172.21.15.163:6805/3125047432 conn(0x7ff0e4003b40 legacy=0x7ff0e400a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e4004270 osd=1 1 2024-02-19T21:05:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 10 client.47626.objecter close_session for osd.2 2024-02-19T21:05:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 >> v1:172.21.15.163:6809/1563158487 conn(0x7ff0e4002070 legacy=0x7ff0e400f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e40033a0 osd=2 1 2024-02-19T21:05:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 10 client.47626.objecter close_session for osd.3 2024-02-19T21:05:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 >> v1:172.21.15.163:6813/2112279769 conn(0x7ff0e4010ed0 legacy=0x7ff0e4011310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e400c8d0 osd=3 1 2024-02-19T21:05:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 10 client.47626.objecter close_session for osd.4 2024-02-19T21:05:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0f7fff700 1 -- 172.21.15.163:0/4094168549 reap_dead start 2024-02-19T21:05:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 >> v1:172.21.15.193:6800/3129716965 conn(0x7ff0e401ab00 legacy=0x7ff0e401cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.567+0000 7ff0f7fff700 1 -- 172.21.15.163:0/4094168549 reap_dead start 2024-02-19T21:05:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e4012260 osd=4 1 2024-02-19T21:05:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 10 client.47626.objecter close_session for osd.5 2024-02-19T21:05:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 >> v1:172.21.15.193:6804/805161991 conn(0x7ff0e40203e0 legacy=0x7ff0e4020840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e401e6e0 osd=5 1 2024-02-19T21:05:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 10 client.47626.objecter close_session for osd.6 2024-02-19T21:05:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 >> v1:172.21.15.193:6808/787604615 conn(0x7ff0e40235a0 legacy=0x7ff0e4025990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e401f930 osd=6 1 2024-02-19T21:05:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 10 client.47626.objecter close_session for osd.7 2024-02-19T21:05:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 >> v1:172.21.15.193:6812/157927432 conn(0x7ff0e4027730 legacy=0x7ff0e4029b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 20 client.47626.objecter put_session s=0x7ff0e40270b0 osd=7 1 2024-02-19T21:05:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 20 client.47626.objecter shutdown clearing up homeless session... 2024-02-19T21:05:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 10 client.47626.objecter successfully canceled tick 2024-02-19T21:05:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 >> v1:172.21.15.163:6800/1552514681 conn(0x7ff0e0063ce0 legacy=0x7ff0e00661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 >> v1:172.21.15.193:6789/0 conn(0x7ff0f80fd980 legacy=0x7ff0f80f9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0f7fff700 1 -- 172.21.15.163:0/4094168549 reap_dead start 2024-02-19T21:05:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 shutdown_connections 2024-02-19T21:05:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 >> 172.21.15.163:0/4094168549 conn(0x7ff0f80f35a0 msgr2=0x7ff0f8107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 shutdown_connections 2024-02-19T21:05:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 1 -- 172.21.15.163:0/4094168549 wait complete. 2024-02-19T21:05:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:28.568+0000 7ff0fdcc8700 1 librados: shutdown 2024-02-19T21:05:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:05:28.565 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:05:28.565 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:05:28.565 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:05:28.565 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:05:28.565 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:05:28.565 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:05:28.565 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:05:29.610 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:28.365989+0000 mon.b (mon.1) 232 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:29.610 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:28.426605+0000 mon.b (mon.1) 233 : audit [DBG] from='client.? v1:172.21.15.163:0/4094168549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:29.610 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:28.477842+0000 mon.a (mon.0) 1157 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:29.610 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:28.676725+0000 mon.c (mon.2) 132 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:29.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:29.164364+0000 mon.a (mon.0) 1158 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:05:29.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:29.164447+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-02-19T21:05:29.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:29.174632+0000 mon.b (mon.1) 234 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-13"}]: dispatch 2024-02-19T21:05:29.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:29.180936+0000 mon.a (mon.0) 1160 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-13"}]: dispatch 2024-02-19T21:05:29.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:29.181161+0000 mon.c (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.163:0/1284807366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi163-83845-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:29.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:29.181815+0000 mon.a (mon.0) 1161 : audit [INF] from='client.48827 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi163-83845-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:29.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:28.365989+0000 mon.b (mon.1) 232 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:29.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:28.426605+0000 mon.b (mon.1) 233 : audit [DBG] from='client.? v1:172.21.15.163:0/4094168549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:29.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:28.477842+0000 mon.a (mon.0) 1157 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:29.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:28.676725+0000 mon.c (mon.2) 132 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:29.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:29.164364+0000 mon.a (mon.0) 1158 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:05:29.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:29.164447+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-02-19T21:05:29.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:29.174632+0000 mon.b (mon.1) 234 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-13"}]: dispatch 2024-02-19T21:05:29.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:29.180936+0000 mon.a (mon.0) 1160 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-13"}]: dispatch 2024-02-19T21:05:29.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:29.181161+0000 mon.c (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.163:0/1284807366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi163-83845-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:29.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:29.181815+0000 mon.a (mon.0) 1161 : audit [INF] from='client.48827 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi163-83845-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:29.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:28.365989+0000 mon.b (mon.1) 232 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:29.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:28.426605+0000 mon.b (mon.1) 233 : audit [DBG] from='client.? v1:172.21.15.163:0/4094168549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:29.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:28.477842+0000 mon.a (mon.0) 1157 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:05:29.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:28.676725+0000 mon.c (mon.2) 132 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:29.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:29.164364+0000 mon.a (mon.0) 1158 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:05:29.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:29.164447+0000 mon.a (mon.0) 1159 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-02-19T21:05:29.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:29.174632+0000 mon.b (mon.1) 234 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-13"}]: dispatch 2024-02-19T21:05:29.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:29.180936+0000 mon.a (mon.0) 1160 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-13"}]: dispatch 2024-02-19T21:05:29.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:29.181161+0000 mon.c (mon.2) 133 : audit [INF] from='client.? v1:172.21.15.163:0/1284807366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi163-83845-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:29.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:29.181815+0000 mon.a (mon.0) 1161 : audit [INF] from='client.48827 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi163-83845-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:29.681 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:29.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:29.699 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:05:29.785 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:05:29.815 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:29.835 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:05:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:29.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:05:29.851 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:05:29.851 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-02-19T21:05:29.851 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-02-19T21:05:29.852 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-02-19T21:05:29.852 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.852 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.24400.0:294 4.1a 4:5ae4284b:::obj_delete_at_hint.0000000032:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2024-02-19T21:05:29.852 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:01:23.445378+0000", 2024-02-19T21:05:29.852 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 246.41036133700001, 2024-02-19T21:05:29.852 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0036833230000000001, 2024-02-19T21:05:29.852 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.852 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:29.852 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.852 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.24400", 2024-02-19T21:05:29.852 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:05:29.852 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 294 2024-02-19T21:05:29.852 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.853 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.853 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.853 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.853 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:23.445378+0000", 2024-02-19T21:05:29.853 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.853 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.853 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.853 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.853 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:23.445378+0000", 2024-02-19T21:05:29.853 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:05:29.853 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.853 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.853 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.853 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:23.445377+0000", 2024-02-19T21:05:29.853 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2459999999999997e-06 2024-02-19T21:05:29.854 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.854 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.854 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.854 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:23.445382+0000", 2024-02-19T21:05:29.854 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6329999999999999e-06 2024-02-19T21:05:29.854 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.854 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.854 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.854 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:23.445388+0000", 2024-02-19T21:05:29.854 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2219999999999998e-06 2024-02-19T21:05:29.854 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.854 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.854 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.855 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:23.445391+0000", 2024-02-19T21:05:29.855 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7662999999999999e-05 2024-02-19T21:05:29.855 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.855 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.855 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.855 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:23.445419+0000", 2024-02-19T21:05:29.855 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3866000000000001e-05 2024-02-19T21:05:29.855 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.855 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.855 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.855 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:23.445493+0000", 2024-02-19T21:05:29.855 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000112447 2024-02-19T21:05:29.855 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.856 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.856 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-02-19T21:05:29.856 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:23.445605+0000", 2024-02-19T21:05:29.856 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00026338499999999999 2024-02-19T21:05:29.856 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.856 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.856 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:05:29.856 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:23.445869+0000", 2024-02-19T21:05:29.856 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00035310199999999999 2024-02-19T21:05:29.856 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.856 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.856 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.856 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:23.446222+0000", 2024-02-19T21:05:29.856 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0027928279999999998 2024-02-19T21:05:29.856 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:23.449015+0000", 2024-02-19T21:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7064e-05 2024-02-19T21:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:23.449042+0000", 2024-02-19T21:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9775999999999998e-05 2024-02-19T21:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:23.449062+0000", 2024-02-19T21:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0036833230000000001 2024-02-19T21:05:29.857 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.858 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.858 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.858 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.858 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.858 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35189.0:4 36.16 36:681f27cd:::2.obj:head [write 0~105] snapc 0=[] ondisk+write+known_if_redirected e103)", 2024-02-19T21:05:29.858 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:42.759848+0000", 2024-02-19T21:05:29.858 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 47.095891584999997, 2024-02-19T21:05:29.858 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.034936716, 2024-02-19T21:05:29.858 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.858 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:29.858 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.858 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35189", 2024-02-19T21:05:29.858 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3441459623", 2024-02-19T21:05:29.858 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 4 2024-02-19T21:05:29.858 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.859 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.859 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.859 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.859 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759848+0000", 2024-02-19T21:05:29.859 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.859 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.859 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.859 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.859 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759848+0000", 2024-02-19T21:05:29.859 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:05:29.859 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.859 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.859 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.859 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759846+0000", 2024-02-19T21:05:29.859 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8869999999999998e-06 2024-02-19T21:05:29.860 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.860 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.860 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.860 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759855+0000", 2024-02-19T21:05:29.860 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2349999999999999e-06 2024-02-19T21:05:29.860 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.860 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.860 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.860 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759863+0000", 2024-02-19T21:05:29.860 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2473999999999999e-05 2024-02-19T21:05:29.860 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.860 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.860 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759876+0000", 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6888000000000003e-05 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759913+0000", 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5630000000000003e-06 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759920+0000", 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.99619900800000005 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.861 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.756119+0000", 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000116972 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.756236+0000", 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015304199999999999 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,3", 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.756389+0000", 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011827809999999999 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.768217+0000", 2024-02-19T21:05:29.862 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011764907 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.779982+0000", 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014710997999999999 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.794693+0000", 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1137999999999997e-05 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.794744+0000", 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0725000000000001e-05 2024-02-19T21:05:29.863 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.864 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.864 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.864 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.794785+0000", 2024-02-19T21:05:29.864 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.034936716 2024-02-19T21:05:29.864 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.864 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.864 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.864 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.864 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.864 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25213.0:3 34.9 34:9294e480:nspace::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e103)", 2024-02-19T21:05:29.864 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:42.759875+0000", 2024-02-19T21:05:29.864 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 47.095864536999997, 2024-02-19T21:05:29.864 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.053943584, 2024-02-19T21:05:29.865 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.865 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:29.865 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.865 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25213", 2024-02-19T21:05:29.865 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2094902030", 2024-02-19T21:05:29.865 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 3 2024-02-19T21:05:29.865 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.865 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.865 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.865 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.865 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759875+0000", 2024-02-19T21:05:29.865 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.865 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.865 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.865 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759875+0000", 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759874+0000", 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4439999999999999e-06 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759879+0000", 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8599999999999999e-06 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759888+0000", 2024-02-19T21:05:29.866 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7299999999999996e-06 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759893+0000", 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1069000000000003e-05 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759934+0000", 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.422e-06 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759937+0000", 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0133646970000001 2024-02-19T21:05:29.867 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.868 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.868 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.868 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.773302+0000", 2024-02-19T21:05:29.868 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012570699999999999 2024-02-19T21:05:29.868 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.868 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.868 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.868 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.773427+0000", 2024-02-19T21:05:29.868 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000139443 2024-02-19T21:05:29.868 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.868 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.868 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-02-19T21:05:29.868 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.773567+0000", 2024-02-19T21:05:29.868 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0052079140000000001 2024-02-19T21:05:29.869 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.869 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.869 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:05:29.869 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.778775+0000", 2024-02-19T21:05:29.869 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.030842405999999999 2024-02-19T21:05:29.869 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.869 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.869 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.869 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.809617+0000", 2024-02-19T21:05:29.869 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0041308889999999996 2024-02-19T21:05:29.869 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.869 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.869 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.869 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.813748+0000", 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3467999999999999e-05 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.813782+0000", 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6962999999999998e-05 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.813819+0000", 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.053943584 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.870 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.871 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "MOSDECSubOpWrite(43.6s1 107/106 ECSubWrite(tid=1186, reqid=client.25267.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=107'1))", 2024-02-19T21:05:29.871 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:47.030901+0000", 2024-02-19T21:05:29.871 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 42.824838063999998, 2024-02-19T21:05:29.871 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0032785480000000001, 2024-02-19T21:05:29.871 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.871 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-02-19T21:05:29.871 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.871 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.871 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.871 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.030901+0000", 2024-02-19T21:05:29.871 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.871 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.872 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.872 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.872 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.030901+0000", 2024-02-19T21:05:29.872 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:05:29.872 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.872 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.872 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.872 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.030900+0000", 2024-02-19T21:05:29.872 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1549000000000001e-05 2024-02-19T21:05:29.872 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.872 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.872 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.872 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.030912+0000", 2024-02-19T21:05:29.872 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0336000000000001e-05 2024-02-19T21:05:29.873 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.873 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.873 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.873 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.030922+0000", 2024-02-19T21:05:29.873 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2110000000000001e-06 2024-02-19T21:05:29.873 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.873 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.873 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.873 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.030929+0000", 2024-02-19T21:05:29.873 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1001999999999995e-05 2024-02-19T21:05:29.873 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.873 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.873 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.031000+0000", 2024-02-19T21:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6760000000000002e-06 2024-02-19T21:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.031010+0000", 2024-02-19T21:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0031627629999999999 2024-02-19T21:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.034173+0000", 2024-02-19T21:05:29.874 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1319999999999996e-06 2024-02-19T21:05:29.875 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.875 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.875 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.875 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.034180+0000", 2024-02-19T21:05:29.875 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0032785480000000001 2024-02-19T21:05:29.875 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.875 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.875 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.875 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.875 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.875 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25105.0:3 40.6 40:602f83fe:::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:05:29.875 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:47.968492+0000", 2024-02-19T21:05:29.875 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 41.887247868999999, 2024-02-19T21:05:29.875 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.042153208999999997, 2024-02-19T21:05:29.875 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.876 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:29.876 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.876 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25105", 2024-02-19T21:05:29.876 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3624732563", 2024-02-19T21:05:29.876 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 3 2024-02-19T21:05:29.876 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.876 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.876 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.876 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.876 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968492+0000", 2024-02-19T21:05:29.876 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.876 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.876 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.876 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.877 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968492+0000", 2024-02-19T21:05:29.877 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:05:29.877 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.877 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.877 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.877 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968490+0000", 2024-02-19T21:05:29.877 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4610000000000008e-06 2024-02-19T21:05:29.877 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.877 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.877 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.877 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968498+0000", 2024-02-19T21:05:29.877 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4069999999999993e-06 2024-02-19T21:05:29.877 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.877 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.877 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.878 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968506+0000", 2024-02-19T21:05:29.878 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0049999999999998e-06 2024-02-19T21:05:29.878 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.878 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.878 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.878 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968513+0000", 2024-02-19T21:05:29.878 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001534941 2024-02-19T21:05:29.878 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.878 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.878 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.878 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.970048+0000", 2024-02-19T21:05:29.878 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0030718680000000002 2024-02-19T21:05:29.878 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.879 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.879 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:05:29.879 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.973120+0000", 2024-02-19T21:05:29.879 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.023296391 2024-02-19T21:05:29.879 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.879 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.879 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.879 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.996417+0000", 2024-02-19T21:05:29.879 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5544000000000006e-05 2024-02-19T21:05:29.879 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.879 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.879 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.879 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.996502+0000", 2024-02-19T21:05:29.879 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000144523 2024-02-19T21:05:29.880 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.880 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.880 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-02-19T21:05:29.880 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.996647+0000", 2024-02-19T21:05:29.880 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00039452900000000002 2024-02-19T21:05:29.880 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.880 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.880 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:05:29.880 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.997041+0000", 2024-02-19T21:05:29.880 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0016365290000000001 2024-02-19T21:05:29.880 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.880 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.880 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.880 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.998678+0000", 2024-02-19T21:05:29.880 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011897256 2024-02-19T21:05:29.881 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.881 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.881 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.881 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010575+0000", 2024-02-19T21:05:29.881 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0837000000000003e-05 2024-02-19T21:05:29.881 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.881 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.881 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:29.881 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010606+0000", 2024-02-19T21:05:29.881 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9029000000000002e-05 2024-02-19T21:05:29.881 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.881 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.881 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.881 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010645+0000", 2024-02-19T21:05:29.881 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.042153208999999997 2024-02-19T21:05:29.882 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.882 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.882 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.882 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.882 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.882 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25321.0:2 49.15 49:ad551702:::test:head [create,setxattr key (5) in=3b] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:05:29.882 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:47.976371+0000", 2024-02-19T21:05:29.882 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 41.879368808000002, 2024-02-19T21:05:29.882 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037843445000000003, 2024-02-19T21:05:29.882 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.882 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:29.882 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.882 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25321", 2024-02-19T21:05:29.882 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/826688492", 2024-02-19T21:05:29.882 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976371+0000", 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976371+0000", 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976369+0000", 2024-02-19T21:05:29.883 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4760000000000004e-06 2024-02-19T21:05:29.884 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.884 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.884 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.884 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976378+0000", 2024-02-19T21:05:29.884 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1931000000000001e-05 2024-02-19T21:05:29.884 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.884 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.884 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.884 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976400+0000", 2024-02-19T21:05:29.884 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.422e-06 2024-02-19T21:05:29.884 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.884 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.884 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.884 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976408+0000", 2024-02-19T21:05:29.885 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7206000000000001e-05 2024-02-19T21:05:29.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.885 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.885 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.885 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976466+0000", 2024-02-19T21:05:29.885 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3210000000000005e-06 2024-02-19T21:05:29.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.885 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.885 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:05:29.885 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976474+0000", 2024-02-19T21:05:29.885 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.028525920999999999 2024-02-19T21:05:29.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.885 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.885 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.886 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.005000+0000", 2024-02-19T21:05:29.886 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011637399999999999 2024-02-19T21:05:29.886 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.886 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.886 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.886 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.005116+0000", 2024-02-19T21:05:29.886 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013954899999999999 2024-02-19T21:05:29.886 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.886 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.886 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-02-19T21:05:29.886 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.005256+0000", 2024-02-19T21:05:29.886 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00050348099999999996 2024-02-19T21:05:29.886 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.886 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.886 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:05:29.887 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.005759+0000", 2024-02-19T21:05:29.887 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0074804399999999997 2024-02-19T21:05:29.887 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.887 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.887 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.887 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.013240+0000", 2024-02-19T21:05:29.887 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00090275499999999999 2024-02-19T21:05:29.887 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.887 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.887 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.887 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014143+0000", 2024-02-19T21:05:29.887 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3145999999999997e-05 2024-02-19T21:05:29.887 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.888 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.888 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:29.888 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014176+0000", 2024-02-19T21:05:29.888 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8489999999999999e-05 2024-02-19T21:05:29.888 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.888 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.888 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.888 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014214+0000", 2024-02-19T21:05:29.888 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037843445000000003 2024-02-19T21:05:29.888 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.888 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.888 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.888 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.888 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.889 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.2.0:220 11.d e108/99 11:b2497998:test-rados-api-smithi163-83952-4::foo:5 v 108'6, mlcod=107'3)", 2024-02-19T21:05:29.889 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:47.991817+0000", 2024-02-19T21:05:29.889 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 41.863922932999998, 2024-02-19T21:05:29.889 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0042156260000000001, 2024-02-19T21:05:29.889 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.889 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:29.889 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.889 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.889 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.889 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991817+0000", 2024-02-19T21:05:29.889 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.889 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.889 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.889 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.889 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991817+0000", 2024-02-19T21:05:29.890 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:05:29.890 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.890 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.890 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.890 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991816+0000", 2024-02-19T21:05:29.890 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9602999999999998e-05 2024-02-19T21:05:29.890 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.890 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.890 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.890 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991836+0000", 2024-02-19T21:05:29.890 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5769999999999999e-06 2024-02-19T21:05:29.890 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.890 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.890 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.890 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991842+0000", 2024-02-19T21:05:29.891 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2580000000000001e-06 2024-02-19T21:05:29.891 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.891 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.891 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.891 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991849+0000", 2024-02-19T21:05:29.891 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0012432529999999999 2024-02-19T21:05:29.891 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.891 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.891 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.891 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.993092+0000", 2024-02-19T21:05:29.891 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9272000000000001e-05 2024-02-19T21:05:29.891 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.891 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.891 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.892 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.993111+0000", 2024-02-19T21:05:29.892 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0028920130000000001 2024-02-19T21:05:29.892 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.892 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.892 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:29.892 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.996003+0000", 2024-02-19T21:05:29.892 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.8989e-05 2024-02-19T21:05:29.892 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.892 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.892 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.892 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.996032+0000", 2024-02-19T21:05:29.892 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0042156260000000001 2024-02-19T21:05:29.892 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.892 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.893 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.893 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.893 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.893 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.24865.0:14 11.18 e108/99 11:1d8f7fe2:test-rados-api-smithi163-83952-5::foo:head v 108'1, mlcod=0'0)", 2024-02-19T21:05:29.893 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:47.991853+0000", 2024-02-19T21:05:29.893 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 41.863886116000003, 2024-02-19T21:05:29.893 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0047438740000000004, 2024-02-19T21:05:29.893 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.893 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:29.893 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.893 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.893 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.893 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991853+0000", 2024-02-19T21:05:29.894 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.894 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.894 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.894 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.894 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991853+0000", 2024-02-19T21:05:29.894 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:05:29.894 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.894 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.894 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.894 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991853+0000", 2024-02-19T21:05:29.894 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0939999999999998e-06 2024-02-19T21:05:29.894 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.894 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.894 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.895 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991858+0000", 2024-02-19T21:05:29.895 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8879999999999996e-06 2024-02-19T21:05:29.895 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.895 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.895 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.895 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991863+0000", 2024-02-19T21:05:29.895 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1400000000000002e-06 2024-02-19T21:05:29.895 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.895 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.895 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.895 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991867+0000", 2024-02-19T21:05:29.895 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00050798900000000003 2024-02-19T21:05:29.895 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.895 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.895 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.992375+0000", 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8471999999999999e-05 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.992393+0000", 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0041760570000000004 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.996569+0000", 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7812000000000001e-05 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.896 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.996597+0000", 2024-02-19T21:05:29.897 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0047438740000000004 2024-02-19T21:05:29.897 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.897 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.897 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.897 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.897 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.897 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25105.0:4 40.6 40:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108)", 2024-02-19T21:05:29.897 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:48.010891+0000", 2024-02-19T21:05:29.897 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 41.844848831, 2024-02-19T21:05:29.897 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010449923999999999, 2024-02-19T21:05:29.897 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.897 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:05:29.897 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.897 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25105", 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3624732563", 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 4 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010891+0000", 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010891+0000", 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.898 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.899 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010889+0000", 2024-02-19T21:05:29.899 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3759999999999999e-06 2024-02-19T21:05:29.899 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.899 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.899 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.899 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010894+0000", 2024-02-19T21:05:29.899 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7290000000000006e-06 2024-02-19T21:05:29.899 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.899 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.899 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.899 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010903+0000", 2024-02-19T21:05:29.899 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9740000000000001e-06 2024-02-19T21:05:29.899 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.899 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.899 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.900 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010908+0000", 2024-02-19T21:05:29.900 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6906000000000002e-05 2024-02-19T21:05:29.900 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.900 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.900 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.900 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010945+0000", 2024-02-19T21:05:29.900 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5568000000000006e-05 2024-02-19T21:05:29.900 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.900 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.900 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.900 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.011011+0000", 2024-02-19T21:05:29.900 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010224851 2024-02-19T21:05:29.900 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.900 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.900 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021235+0000", 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8914e-05 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021284+0000", 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6223e-05 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021341+0000", 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010449923999999999 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.901 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.902 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.902 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25105.0:5 40.6 40:602f83fe:::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:05:29.902 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:48.010924+0000", 2024-02-19T21:05:29.902 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 41.844815662000002, 2024-02-19T21:05:29.902 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011507269000000001, 2024-02-19T21:05:29.902 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.902 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:29.902 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.902 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25105", 2024-02-19T21:05:29.902 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3624732563", 2024-02-19T21:05:29.902 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 5 2024-02-19T21:05:29.902 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.902 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.902 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.902 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010924+0000", 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010924+0000", 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010923+0000", 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.438e-06 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.903 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010929+0000", 2024-02-19T21:05:29.904 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0610000000000001e-06 2024-02-19T21:05:29.904 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.904 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.904 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.904 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010935+0000", 2024-02-19T21:05:29.904 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1400000000000002e-06 2024-02-19T21:05:29.904 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.904 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.904 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.904 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010940+0000", 2024-02-19T21:05:29.904 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00023434000000000001 2024-02-19T21:05:29.904 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.904 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.904 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.011174+0000", 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3954e-05 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.011218+0000", 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010153373 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021371+0000", 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9161999999999998e-05 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.905 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021460+0000", 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013853699999999999 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021599+0000", 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000369594 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021969+0000", 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000122967 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.022092+0000", 2024-02-19T21:05:29.906 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000293484 2024-02-19T21:05:29.907 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.907 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.907 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.907 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.022385+0000", 2024-02-19T21:05:29.907 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.586e-05 2024-02-19T21:05:29.907 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.907 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.907 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:29.907 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.022411+0000", 2024-02-19T21:05:29.907 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0248e-05 2024-02-19T21:05:29.907 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.907 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.907 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.907 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.022431+0000", 2024-02-19T21:05:29.908 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011507269000000001 2024-02-19T21:05:29.908 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.908 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.908 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.908 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.908 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.908 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25321.0:3 49.15 49:ad551702:::test:head [create,cmpxattr key (5) op 1 mode 1 in=3b,rmxattr key in=3b] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:05:29.908 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:48.014851+0000", 2024-02-19T21:05:29.908 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 41.840888751000001, 2024-02-19T21:05:29.908 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0034778330000000001, 2024-02-19T21:05:29.908 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.908 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:29.908 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.908 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25321", 2024-02-19T21:05:29.908 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/826688492", 2024-02-19T21:05:29.909 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 3 2024-02-19T21:05:29.909 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.909 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.909 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.909 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.909 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014851+0000", 2024-02-19T21:05:29.909 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.909 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.909 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.909 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.909 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014851+0000", 2024-02-19T21:05:29.909 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:05:29.909 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.909 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.909 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.910 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014849+0000", 2024-02-19T21:05:29.910 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7989999999999999e-06 2024-02-19T21:05:29.910 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.910 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.910 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.910 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014855+0000", 2024-02-19T21:05:29.910 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6710000000000005e-06 2024-02-19T21:05:29.910 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.910 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.910 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.910 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014864+0000", 2024-02-19T21:05:29.910 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6160000000000004e-06 2024-02-19T21:05:29.910 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.910 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.910 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014869+0000", 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5794000000000003e-05 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014904+0000", 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9505000000000001e-05 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014954+0000", 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000105592 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-02-19T21:05:29.911 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.015059+0000", 2024-02-19T21:05:29.912 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000239003 2024-02-19T21:05:29.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.912 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.912 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:05:29.912 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.015298+0000", 2024-02-19T21:05:29.912 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002923414 2024-02-19T21:05:29.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.912 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.912 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.912 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.018222+0000", 2024-02-19T21:05:29.912 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5448000000000002e-05 2024-02-19T21:05:29.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.912 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.912 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.912 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.018277+0000", 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0131999999999999e-05 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.018307+0000", 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1160000000000001e-05 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.018329+0000", 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0034778330000000001 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.913 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.32629.0:7 76.3s0 76:dfb9de0f:test-rados-api-smithi163-83944-12::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:11.017799+0000", 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 18.837940543999999, 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0068870470000000003, 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.32629", 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1643190421", 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 7 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017799+0000", 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.914 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017799+0000", 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017797+0000", 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3829999999999992e-06 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017805+0000", 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3729999999999999e-06 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.915 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.916 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.916 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017815+0000", 2024-02-19T21:05:29.916 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0319999999999998e-06 2024-02-19T21:05:29.916 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.916 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.916 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.916 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017822+0000", 2024-02-19T21:05:29.916 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3600999999999999e-05 2024-02-19T21:05:29.916 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.916 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.916 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.916 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017855+0000", 2024-02-19T21:05:29.916 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000114827 2024-02-19T21:05:29.916 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017970+0000", 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00040942799999999998 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.018379+0000", 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00058837500000000003 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.018968+0000", 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0056430810000000003 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.917 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.918 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:29.918 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.024611+0000", 2024-02-19T21:05:29.918 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5131999999999998e-05 2024-02-19T21:05:29.918 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.918 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.918 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.918 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.024686+0000", 2024-02-19T21:05:29.918 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0068870470000000003 2024-02-19T21:05:29.918 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.918 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.918 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.918 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.918 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.918 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.32629.0:9 76.3s0 76:dfb9de0f:test-rados-api-smithi163-83944-12::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.093636+0000", 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 17.762103869000001, 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.084125466999999995, 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.32629", 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1643190421", 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 9 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093636+0000", 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.920 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.920 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.920 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093636+0000", 2024-02-19T21:05:29.920 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:05:29.920 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.920 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.920 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.920 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093633+0000", 2024-02-19T21:05:29.920 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4586000000000001e-05 2024-02-19T21:05:29.920 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.920 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.920 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.920 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093648+0000", 2024-02-19T21:05:29.920 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4055e-05 2024-02-19T21:05:29.921 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.921 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.921 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.921 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093662+0000", 2024-02-19T21:05:29.921 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7350000000000002e-06 2024-02-19T21:05:29.921 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.921 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.921 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.921 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093670+0000", 2024-02-19T21:05:29.921 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1807000000000001e-05 2024-02-19T21:05:29.921 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.921 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.921 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.921 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093722+0000", 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0189000000000001e-05 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093802+0000", 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00059241199999999995 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.094395+0000", 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0013219200000000001 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.095717+0000", 2024-02-19T21:05:29.922 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.081990402000000004 2024-02-19T21:05:29.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.923 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.923 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:29.923 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.177707+0000", 2024-02-19T21:05:29.923 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4097000000000001e-05 2024-02-19T21:05:29.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.923 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.923 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.923 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.177761+0000", 2024-02-19T21:05:29.923 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.084125466999999995 2024-02-19T21:05:29.923 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.923 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.923 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.923 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.924 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.32629.0:10 76.3s0 76:dfb9de0f:test-rados-api-smithi163-83944-12::foo:head [rollback 5] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-02-19T21:05:29.924 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.178134+0000", 2024-02-19T21:05:29.924 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 17.677605080999999, 2024-02-19T21:05:29.924 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015877571, 2024-02-19T21:05:29.924 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.924 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:29.924 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.924 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.32629", 2024-02-19T21:05:29.924 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1643190421", 2024-02-19T21:05:29.924 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 10 2024-02-19T21:05:29.924 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.924 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.924 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.925 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.925 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178134+0000", 2024-02-19T21:05:29.925 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.925 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.925 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.925 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178134+0000", 2024-02-19T21:05:29.925 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:05:29.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.925 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.925 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.925 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178133+0000", 2024-02-19T21:05:29.925 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6609999999999997e-06 2024-02-19T21:05:29.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.925 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178138+0000", 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7979999999999995e-06 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178147+0000", 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7389999999999999e-06 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178152+0000", 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7440000000000002e-05 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.926 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.927 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178179+0000", 2024-02-19T21:05:29.927 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4633000000000003e-05 2024-02-19T21:05:29.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.927 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.927 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178244+0000", 2024-02-19T21:05:29.927 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00033892500000000001 2024-02-19T21:05:29.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.927 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:05:29.927 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178583+0000", 2024-02-19T21:05:29.927 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025176799999999998 2024-02-19T21:05:29.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178835+0000", 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015138621999999999 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.193973+0000", 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8704000000000001e-05 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.194012+0000", 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015877571 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.928 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.929 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.929 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.32629.0:11 76.3s0 76:dfb9de0f:test-rados-api-smithi163-83944-12::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e132)", 2024-02-19T21:05:29.929 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.201158+0000", 2024-02-19T21:05:29.929 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 17.654581297, 2024-02-19T21:05:29.929 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011048186, 2024-02-19T21:05:29.929 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.929 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:05:29.929 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.929 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.32629", 2024-02-19T21:05:29.929 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1643190421", 2024-02-19T21:05:29.929 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 11 2024-02-19T21:05:29.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.929 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.929 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.930 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.930 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201158+0000", 2024-02-19T21:05:29.930 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.930 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.930 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.930 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201158+0000", 2024-02-19T21:05:29.930 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:05:29.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.930 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.930 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.930 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201157+0000", 2024-02-19T21:05:29.930 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0796e-05 2024-02-19T21:05:29.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.930 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201168+0000", 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9330000000000007e-06 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201176+0000", 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.229e-06 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201183+0000", 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9822999999999999e-05 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.932 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201213+0000", 2024-02-19T21:05:29.932 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0374000000000001e-05 2024-02-19T21:05:29.932 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.932 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.932 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.932 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201283+0000", 2024-02-19T21:05:29.932 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010923510000000001 2024-02-19T21:05:29.932 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.932 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.932 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.932 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.212206+0000", 2024-02-19T21:05:29.932 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011048186 2024-02-19T21:05:29.932 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.932 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.932 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:252 61.16 61:68726f76:test-rados-api-smithi163-83928-12::foo:head [writefull 0~5] snapc 2=[2] ondisk+write+known_if_redirected e135)", 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:15.184366+0000", 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 14.671373689999999, 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0086795729999999995, 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 252 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.933 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184366+0000", 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184366+0000", 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184363+0000", 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1565e-05 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.934 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184375+0000", 2024-02-19T21:05:29.935 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.38e-05 2024-02-19T21:05:29.935 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.935 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.935 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.935 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184388+0000", 2024-02-19T21:05:29.935 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.366e-05 2024-02-19T21:05:29.935 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.935 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.935 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.935 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184402+0000", 2024-02-19T21:05:29.935 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0022266690000000001 2024-02-19T21:05:29.935 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.935 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.935 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.935 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.186629+0000", 2024-02-19T21:05:29.936 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010571800000000001 2024-02-19T21:05:29.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.936 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.936 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.936 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.186734+0000", 2024-02-19T21:05:29.936 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00043829699999999998 2024-02-19T21:05:29.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.936 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.936 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 5,7", 2024-02-19T21:05:29.936 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.187173+0000", 2024-02-19T21:05:29.936 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0040489430000000002 2024-02-19T21:05:29.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.936 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.936 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.936 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.191222+0000", 2024-02-19T21:05:29.937 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0153000000000002e-05 2024-02-19T21:05:29.937 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.937 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.937 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:05:29.937 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.191312+0000", 2024-02-19T21:05:29.937 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001633362 2024-02-19T21:05:29.937 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.937 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.937 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.937 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.192945+0000", 2024-02-19T21:05:29.937 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9512000000000003e-05 2024-02-19T21:05:29.937 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.937 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.937 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:29.938 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.192995+0000", 2024-02-19T21:05:29.938 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0753e-05 2024-02-19T21:05:29.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.938 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.938 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193045+0000", 2024-02-19T21:05:29.938 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0086795729999999995 2024-02-19T21:05:29.938 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.938 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.938 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.938 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.26962.0:51 64.16 e135/116 64:6f72f007:test-rados-api-smithi163-83952-11::foo:head v 135'1, mlcod=0'0)", 2024-02-19T21:05:29.939 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:15.292795+0000", 2024-02-19T21:05:29.939 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 14.562944631000001, 2024-02-19T21:05:29.939 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.004000221, 2024-02-19T21:05:29.939 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.939 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:29.939 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.939 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.939 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.939 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.292795+0000", 2024-02-19T21:05:29.939 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.939 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.939 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.939 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.939 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.292795+0000", 2024-02-19T21:05:29.939 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:05:29.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.940 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.940 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.292794+0000", 2024-02-19T21:05:29.940 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0030854039999999999 2024-02-19T21:05:29.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.940 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.940 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.295880+0000", 2024-02-19T21:05:29.940 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3908e-05 2024-02-19T21:05:29.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.940 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.941 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.295894+0000", 2024-02-19T21:05:29.941 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3810000000000001e-06 2024-02-19T21:05:29.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.941 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.941 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.295901+0000", 2024-02-19T21:05:29.941 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6004000000000002e-05 2024-02-19T21:05:29.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.941 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.941 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.295947+0000", 2024-02-19T21:05:29.941 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9066e-05 2024-02-19T21:05:29.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.942 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.942 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.295966+0000", 2024-02-19T21:05:29.942 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00079886900000000001 2024-02-19T21:05:29.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.942 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.942 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:29.942 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.296765+0000", 2024-02-19T21:05:29.942 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0266000000000001e-05 2024-02-19T21:05:29.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.942 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.942 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.942 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.296795+0000", 2024-02-19T21:05:29.942 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.004000221 2024-02-19T21:05:29.942 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.943 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.943 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.943 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.943 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:259 85.1e 85:780b1894:test-rados-api-smithi163-83928-12::bar:2 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e138)", 2024-02-19T21:05:29.943 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:18.696167+0000", 2024-02-19T21:05:29.943 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 11.15957249, 2024-02-19T21:05:29.943 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0052543879999999996, 2024-02-19T21:05:29.943 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.943 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:05:29.943 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.943 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:05:29.943 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:05:29.943 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 259 2024-02-19T21:05:29.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.944 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.944 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.944 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696167+0000", 2024-02-19T21:05:29.944 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.944 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.944 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696167+0000", 2024-02-19T21:05:29.944 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:05:29.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696165+0000", 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3890000000000002e-06 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696172+0000", 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0154e-05 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696182+0000", 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9739999999999999e-06 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.945 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.946 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696188+0000", 2024-02-19T21:05:29.946 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6316999999999998e-05 2024-02-19T21:05:29.946 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.947 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.947 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696224+0000", 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015598700000000001 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696380+0000", 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00111491 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.697495+0000", 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014420899999999999 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:05:29.948 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.697639+0000", 2024-02-19T21:05:29.949 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00075171600000000004 2024-02-19T21:05:29.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.949 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.949 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.949 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.698391+0000", 2024-02-19T21:05:29.949 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4733999999999994e-05 2024-02-19T21:05:29.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.949 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.949 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:05:29.949 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.698456+0000", 2024-02-19T21:05:29.949 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001527849 2024-02-19T21:05:29.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.949 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.949 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.949 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.699983+0000", 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6997000000000003e-05 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.700040+0000", 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0013809650000000001 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.701421+0000", 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0052543879999999996 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.950 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:260 85.10 85:0d6787e5:test-rados-api-smithi163-83928-12::baz:2 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e138)", 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:18.703035+0000", 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 11.152704669, 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0050535320000000003, 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 260 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703035+0000", 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.951 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.952 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.952 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.952 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703035+0000", 2024-02-19T21:05:29.952 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:05:29.952 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.952 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.952 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.952 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703031+0000", 2024-02-19T21:05:29.952 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4063000000000001e-05 2024-02-19T21:05:29.952 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.952 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.952 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.952 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703045+0000", 2024-02-19T21:05:29.952 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5846e-05 2024-02-19T21:05:29.952 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703071+0000", 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8336e-05 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703110+0000", 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5061999999999995e-05 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703175+0000", 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00024473799999999999 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.953 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703419+0000", 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00071855499999999995 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.704138+0000", 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015597300000000001 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.704294+0000", 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00081258100000000005 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.954 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.705107+0000", 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8016999999999997e-05 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.705165+0000", 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001633406 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.706798+0000", 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2812999999999997e-05 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.955 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.706851+0000", 2024-02-19T21:05:29.956 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0012376080000000001 2024-02-19T21:05:29.956 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.956 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.956 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.956 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.708088+0000", 2024-02-19T21:05:29.956 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0050535320000000003 2024-02-19T21:05:29.956 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.956 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.956 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.956 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.956 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.956 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.47503.0:2 93.18 93:1e0071ee:::oid_RoundTripSparseReadPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-02-19T21:05:29.956 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:22.198653+0000", 2024-02-19T21:05:29.956 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 7.6570860920000001, 2024-02-19T21:05:29.956 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025958305000000001, 2024-02-19T21:05:29.957 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:29.957 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:29.957 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:29.957 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.47503", 2024-02-19T21:05:29.957 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1485143480", 2024-02-19T21:05:29.957 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:05:29.957 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.957 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:29.957 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.957 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:29.957 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198653+0000", 2024-02-19T21:05:29.957 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:29.957 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198653+0000", 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198651+0000", 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2534000000000001e-05 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198663+0000", 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8508999999999998e-05 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.958 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198682+0000", 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2040000000000003e-06 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198691+0000", 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6539999999999997e-05 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198747+0000", 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0387000000000001e-05 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.959 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:05:29.960 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198758+0000", 2024-02-19T21:05:29.960 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015289329000000001 2024-02-19T21:05:29.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.960 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.960 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:29.960 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.214047+0000", 2024-02-19T21:05:29.960 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000131958 2024-02-19T21:05:29.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.960 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.960 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:29.960 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.214179+0000", 2024-02-19T21:05:29.960 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000165496 2024-02-19T21:05:29.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.960 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.960 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,5", 2024-02-19T21:05:29.961 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.214345+0000", 2024-02-19T21:05:29.961 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00067452899999999999 2024-02-19T21:05:29.961 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.961 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.961 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:05:29.961 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.215019+0000", 2024-02-19T21:05:29.961 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0069561409999999999 2024-02-19T21:05:29.961 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.961 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.961 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.961 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.221975+0000", 2024-02-19T21:05:29.961 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002534653 2024-02-19T21:05:29.961 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.961 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.961 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.224510+0000", 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7840000000000002e-05 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.224568+0000", 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4063999999999997e-05 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.224612+0000", 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025958305000000001 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:29.962 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:29.963 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:05:29.963 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:29.993 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:30.090 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:30.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:30.194 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:05:30.220 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:30.274 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:05:30.278 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:30.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:05:30.379 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:30.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.059+0000 7fdf0d09c700 1 -- 172.21.15.163:0/2314545714 >> v1:172.21.15.163:6790/0 conn(0x7fdf081031b0 legacy=0x7fdf08105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:30.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.059+0000 7fdf0d09c700 1 -- 172.21.15.163:0/2314545714 shutdown_connections 2024-02-19T21:05:30.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.059+0000 7fdf0d09c700 1 -- 172.21.15.163:0/2314545714 >> 172.21.15.163:0/2314545714 conn(0x7fdf080f2470 msgr2=0x7fdf080f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:30.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.059+0000 7fdf0d09c700 1 -- 172.21.15.163:0/2314545714 shutdown_connections 2024-02-19T21:05:30.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.059+0000 7fdf0d09c700 1 -- 172.21.15.163:0/2314545714 wait complete. 2024-02-19T21:05:30.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.060+0000 7fdf0d09c700 1 librados: starting msgr at 2024-02-19T21:05:30.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.060+0000 7fdf0d09c700 1 librados: starting objecter 2024-02-19T21:05:30.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.060+0000 7fdf0d09c700 1 Processor -- start 2024-02-19T21:05:30.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.060+0000 7fdf0d09c700 1 -- start start 2024-02-19T21:05:30.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.060+0000 7fdf0d09c700 1 librados: setting wanted keys 2024-02-19T21:05:30.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.060+0000 7fdf0d09c700 1 librados: calling monclient init 2024-02-19T21:05:30.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.060+0000 7fdf0d09c700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf080fbee0 con 0x7fdf080fc870 2024-02-19T21:05:30.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdf0d09c700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf080fc0b0 con 0x7fdf080ff670 2024-02-19T21:05:30.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdf0d09c700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf080fc2a0 con 0x7fdf081031b0 2024-02-19T21:05:30.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdf06d9d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fdf080fc870 0x7fdf080fb740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:36076/0 (socket says 172.21.15.163:36076) 2024-02-19T21:05:30.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdf06d9d700 1 -- 172.21.15.163:0/3640601682 learned_addr learned my addr 172.21.15.163:0/3640601682 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:30.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdf0759e700 1 --1- 172.21.15.163:0/3640601682 >> v1:172.21.15.163:6790/0 conn(0x7fdf081031b0 0x7fdf08118340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:40512/0 (socket says 172.21.15.163:40512) 2024-02-19T21:05:30.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 10 client.?.objecter ms_handle_connect 0x7fdf080fc870 2024-02-19T21:05:30.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:30.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 10 client.?.objecter ms_handle_connect 0x7fdf081031b0 2024-02-19T21:05:30.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:30.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 10 client.?.objecter ms_handle_connect 0x7fdf080ff670 2024-02-19T21:05:30.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:30.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507682477 0 0) 0x7fdf0810a570 con 0x7fdf081031b0 2024-02-19T21:05:30.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef00033f0 con 0x7fdf081031b0 2024-02-19T21:05:30.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130150269 0 0) 0x7fdf080fbee0 con 0x7fdf080fc870 2024-02-19T21:05:30.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf0810a570 con 0x7fdf080fc870 2024-02-19T21:05:30.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915547462 0 0) 0x7fdf080fc0b0 con 0x7fdf080ff670 2024-02-19T21:05:30.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf080fbee0 con 0x7fdf080ff670 2024-02-19T21:05:30.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1291872574 0 0) 0x7fdef00033f0 con 0x7fdf081031b0 2024-02-19T21:05:30.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf080fc0b0 con 0x7fdf081031b0 2024-02-19T21:05:30.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdf00001310 con 0x7fdf081031b0 2024-02-19T21:05:30.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4230794087 0 0) 0x7fdf080fbee0 con 0x7fdf080ff670 2024-02-19T21:05:30.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdef00033f0 con 0x7fdf080ff670 2024-02-19T21:05:30.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1796590249 0 0) 0x7fdf0810a570 con 0x7fdf080fc870 2024-02-19T21:05:30.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf080fbee0 con 0x7fdf080fc870 2024-02-19T21:05:30.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdefc001840 con 0x7fdf080ff670 2024-02-19T21:05:30.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdef80019d0 con 0x7fdf080fc870 2024-02-19T21:05:30.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1390782953 0 0) 0x7fdf080fc0b0 con 0x7fdf081031b0 2024-02-19T21:05:30.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 >> v1:172.21.15.193:6789/0 conn(0x7fdf080ff670 legacy=0x7fdf08114c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:30.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 >> v1:172.21.15.163:6789/0 conn(0x7fdf080fc870 legacy=0x7fdf080fb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:30.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0811caf0 con 0x7fdf081031b0 2024-02-19T21:05:30.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdf0d09c700 1 -- 172.21.15.163:0/3640601682 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf08119ac0 con 0x7fdf081031b0 2024-02-19T21:05:30.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdf0d09c700 10 client.48842.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:30.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdf0d09c700 1 -- 172.21.15.163:0/3640601682 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf08119f30 con 0x7fdf081031b0 2024-02-19T21:05:30.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdf0d09c700 1 librados: init done 2024-02-19T21:05:30.393 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:05:30.393 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:05:30.393 INFO:teuthology.orchestra.run.smithi163.stdout: "epoch": 150, 2024-02-19T21:05:30.393 INFO:teuthology.orchestra.run.smithi163.stdout: "fsid": "2591809a-cf69-11ee-95bb-87774f69a715", 2024-02-19T21:05:30.393 INFO:teuthology.orchestra.run.smithi163.stdout: "created": "2024-02-19T20:55:37.794885+0000", 2024-02-19T21:05:30.393 INFO:teuthology.orchestra.run.smithi163.stdout: "modified": "2024-02-19T21:05:30.157979+0000", 2024-02-19T21:05:30.393 INFO:teuthology.orchestra.run.smithi163.stdout: "last_up_change": "2024-02-19T21:01:09.214134+0000", 2024-02-19T21:05:30.393 INFO:teuthology.orchestra.run.smithi163.stdout: "last_in_change": "2024-02-19T21:00:51.699017+0000", 2024-02-19T21:05:30.393 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-19T21:05:30.393 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_num": 5806080, 2024-02-19T21:05:30.393 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_set": [ 2024-02-19T21:05:30.393 INFO:teuthology.orchestra.run.smithi163.stdout: "nodeep-scrub", 2024-02-19T21:05:30.393 INFO:teuthology.orchestra.run.smithi163.stdout: "noscrub", 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: "pglog_hardlimit", 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: "purged_snapdirs", 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_deletes", 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: "sortbitwise" 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_version": 27, 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: "full_ratio": 0.94999998807907104, 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_snapshot": "", 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_max": 100, 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: "max_osd": 8, 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: "require_min_compat_client": "luminous", 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: "min_compat_client": "luminous", 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: "require_osd_release": "pacific", 2024-02-19T21:05:30.394 INFO:teuthology.orchestra.run.smithi163.stdout: "pools": [ 2024-02-19T21:05:30.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.395 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 1, 2024-02-19T21:05:30.395 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "device_health_metrics", 2024-02-19T21:05:30.395 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T20:59:09.425169+0000", 2024-02-19T21:05:30.395 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:05:30.395 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:05:30.395 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:05:30.395 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:05:30.395 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:05:30.395 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:05:30.395 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:05:30.395 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:05:30.395 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:05:30.395 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:05:30.396 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:05:30.396 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:05:30.396 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 1, 2024-02-19T21:05:30.396 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 1, 2024-02-19T21:05:30.396 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 1, 2024-02-19T21:05:30.396 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 1, 2024-02-19T21:05:30.396 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 1, 2024-02-19T21:05:30.396 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:05:30.396 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:05:30.396 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:05:30.396 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:05:30.396 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:05:30.396 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:05:30.396 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:05:30.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.397 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "74", 2024-02-19T21:05:30.397 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:05:30.397 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:05:30.397 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:05:30.397 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:05:30.397 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:05:30.397 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:05:30.397 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:05:30.397 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:05:30.397 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:05:30.397 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:05:30.397 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:05:30.397 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:05:30.397 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:05:30.398 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:05:30.398 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:05:30.398 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:05:30.398 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:05:30.398 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:05:30.398 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:05:30.398 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:05:30.398 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:05:30.398 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:05:30.398 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:05:30.398 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:05:30.398 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:05:30.398 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:05:30.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.399 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:05:30.399 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:05:30.399 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:05:30.399 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:05:30.399 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:05:30.399 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:05:30.399 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:05:30.399 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:05:30.399 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:05:30.399 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:05:30.399 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:05:30.399 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:05:30.399 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_max": 32, 2024-02-19T21:05:30.399 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_min": 1 2024-02-19T21:05:30.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.400 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:05:30.400 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_devicehealth": {} 2024-02-19T21:05:30.400 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.400 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 2, 2024-02-19T21:05:30.400 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".rgw.root", 2024-02-19T21:05:30.400 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:14.658945+0000", 2024-02-19T21:05:30.400 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:05:30.400 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:05:30.400 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:05:30.400 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:05:30.400 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:05:30.400 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:05:30.400 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:05:30.401 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:05:30.401 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:05:30.401 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:05:30.401 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:05:30.401 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:05:30.401 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 42, 2024-02-19T21:05:30.401 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 42, 2024-02-19T21:05:30.401 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 42, 2024-02-19T21:05:30.401 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 42, 2024-02-19T21:05:30.401 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 42, 2024-02-19T21:05:30.401 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:05:30.401 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:05:30.401 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:05:30.401 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:05:30.402 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:05:30.402 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:05:30.402 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:05:30.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.402 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "141", 2024-02-19T21:05:30.402 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:05:30.402 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:05:30.402 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "139", 2024-02-19T21:05:30.402 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:05:30.402 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:05:30.402 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:05:30.402 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:05:30.402 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:05:30.402 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:05:30.402 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:05:30.403 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:05:30.403 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:05:30.403 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:05:30.403 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:05:30.403 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:05:30.403 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:05:30.403 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:05:30.403 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:05:30.403 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:05:30.403 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:05:30.403 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:05:30.403 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:05:30.403 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:05:30.404 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:05:30.404 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:05:30.404 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:05:30.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.404 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:05:30.404 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:05:30.404 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:05:30.404 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:05:30.404 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:05:30.404 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:05:30.404 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:05:30.405 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:05:30.405 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:05:30.405 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:05:30.405 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:05:30.405 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:05:30.405 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:05:30.405 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:05:30.405 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.405 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 3, 2024-02-19T21:05:30.405 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "iscsi", 2024-02-19T21:05:30.405 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:14.958800+0000", 2024-02-19T21:05:30.405 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:05:30.405 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 3, 2024-02-19T21:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 3, 2024-02-19T21:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 3, 2024-02-19T21:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 3, 2024-02-19T21:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 3, 2024-02-19T21:05:30.407 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:05:30.407 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:05:30.407 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:05:30.407 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:05:30.407 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:05:30.407 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:05:30.407 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:05:30.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.407 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "100", 2024-02-19T21:05:30.407 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:05:30.407 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:05:30.407 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:05:30.407 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:05:30.408 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:05:30.408 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:05:30.408 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:05:30.408 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:05:30.408 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:05:30.408 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:05:30.408 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:05:30.408 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:05:30.408 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:05:30.408 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:05:30.408 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:05:30.408 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:05:30.408 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:05:30.408 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:05:30.409 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:05:30.409 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:05:30.409 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:05:30.409 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:05:30.409 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:05:30.409 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:05:30.409 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:05:30.409 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:05:30.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.409 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:05:30.409 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:05:30.409 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:05:30.409 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:05:30.409 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:05:30.409 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:05:30.410 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:05:30.410 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:05:30.410 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:05:30.410 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:05:30.410 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:05:30.410 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:05:30.410 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:05:30.410 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd": {} 2024-02-19T21:05:30.410 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.410 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 4, 2024-02-19T21:05:30.410 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.log", 2024-02-19T21:05:30.411 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:16.543816+0000", 2024-02-19T21:05:30.411 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:05:30.411 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:05:30.411 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:05:30.411 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:05:30.411 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:05:30.411 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:05:30.411 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:05:30.411 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:05:30.411 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:05:30.411 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:05:30.411 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:05:30.411 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:05:30.412 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:05:30.412 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:05:30.412 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:05:30.412 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:05:30.412 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:05:30.412 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:05:30.412 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:05:30.412 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:05:30.412 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:05:30.412 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:05:30.412 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:05:30.412 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:05:30.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.413 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "61", 2024-02-19T21:05:30.413 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:05:30.413 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:05:30.413 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:05:30.413 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:05:30.413 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:05:30.413 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:05:30.413 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:05:30.413 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:05:30.413 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:05:30.413 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:05:30.413 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:05:30.413 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:05:30.413 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:05:30.414 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:05:30.414 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:05:30.414 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:05:30.414 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:05:30.414 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:05:30.414 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:05:30.414 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:05:30.414 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:05:30.414 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:05:30.414 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:05:30.414 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:05:30.414 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:05:30.414 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:05:30.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.416 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.416 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 5, 2024-02-19T21:05:30.416 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.control", 2024-02-19T21:05:30.416 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:18.521482+0000", 2024-02-19T21:05:30.416 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:05:30.416 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:05:30.416 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:05:30.416 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:05:30.416 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:05:30.416 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:05:30.416 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:05:30.416 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:05:30.416 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:05:30.416 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.417 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "63", 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:05:30.418 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:05:30.419 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 6, 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.meta", 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:20.586564+0000", 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:05:30.420 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:05:30.421 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:05:30.422 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:05:30.422 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:05:30.422 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:05:30.422 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:05:30.422 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:05:30.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.422 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "65", 2024-02-19T21:05:30.422 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:05:30.422 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:05:30.422 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:05:30.422 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:05:30.422 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:05:30.422 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:05:30.422 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:05:30.423 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:05:30.423 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:05:30.423 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:05:30.423 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:05:30.423 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:05:30.423 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:05:30.423 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:05:30.423 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:05:30.423 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:05:30.423 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:05:30.423 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:05:30.423 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:05:30.423 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:05:30.423 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:05:30.423 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:05:30.424 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:05:30.424 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:05:30.424 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:05:30.424 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:05:30.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.424 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:05:30.424 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:05:30.424 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:05:30.424 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:05:30.424 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:05:30.424 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:05:30.424 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:05:30.424 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:05:30.425 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:05:30.425 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:05:30.425 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:05:30.425 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:05:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fdf00002fe0 con 0x7fdf081031b0 2024-02-19T21:05:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.061+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdf00003890 con 0x7fdf081031b0 2024-02-19T21:05:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.062+0000 7fdf0d09c700 1 -- 172.21.15.163:0/3640601682 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef4004f20 con 0x7fdf081031b0 2024-02-19T21:05:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.063+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fdf00003ad0 con 0x7fdf081031b0 2024-02-19T21:05:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.063+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(149..149 src has 43..149) v4 ==== 10163+0+0 (unknown 1484555491 0 0) 0x7fdf00079780 con 0x7fdf081031b0 2024-02-19T21:05:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.063+0000 7fdeeffff700 10 client.48842.objecter ms_dispatch 0x7fdf08001550 osd_map(149..149 src has 43..149) v4 2024-02-19T21:05:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.063+0000 7fdeeffff700 3 client.48842.objecter handle_osd_map got epochs [149,149] > 0 2024-02-19T21:05:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.063+0000 7fdeeffff700 3 client.48842.objecter handle_osd_map decoding full epoch 149 2024-02-19T21:05:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.063+0000 7fdeeffff700 20 client.48842.objecter dump_active .. 0 homeless 2024-02-19T21:05:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.063+0000 7fdeeffff700 10 client.48842.objecter ms_handle_connect 0x7fdef0063e70 2024-02-19T21:05:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.067+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fdf0004c550 con 0x7fdf081031b0 2024-02-19T21:05:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.257+0000 7fdf0d09c700 1 -- 172.21.15.163:0/3640601682 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fdef40051d0 con 0x7fdf081031b0 2024-02-19T21:05:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.259+0000 7fdeeffff700 1 -- 172.21.15.163:0/3640601682 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v150) v1 ==== 81+0+59333 (unknown 3545803895 0 1766169087) 0x7fdf0004a9c0 con 0x7fdf081031b0 2024-02-19T21:05:30.426 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_bias": 4 2024-02-19T21:05:30.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.426 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:05:30.426 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:05:30.426 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 8, 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "LibRadosList_smithi163-83888-1", 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:33.426193+0000", 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:05:30.427 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "101", 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:05:30.428 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:05:30.429 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:05:30.429 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:05:30.429 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:05:30.429 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:05:30.429 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:05:30.429 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:05:30.429 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:05:30.429 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:05:30.429 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:05:30.429 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:05:30.429 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:05:30.429 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:05:30.429 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:05:30.429 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:05:30.429 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:05:30.430 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 16, 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:35.333192+0000", 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:05:30.431 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:05:30.432 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "101", 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:05:30.433 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:05:30.434 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:05:30.435 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:05:30.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.435 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:05:30.435 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:05:30.435 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:05:30.435 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:05:30.435 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:05:30.435 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:05:30.435 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:05:30.435 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:05:30.435 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:05:30.435 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:05:30.435 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:05:30.435 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:05:30.435 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": {} 2024-02-19T21:05:30.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.436 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 28, 2024-02-19T21:05:30.436 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "LibRadosMiscPP_smithi163-83920-1", 2024-02-19T21:05:30.436 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:38.251419+0000", 2024-02-19T21:05:30.436 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:05:30.436 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:05:30.436 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:05:30.436 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:05:30.436 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:05:30.436 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:05:30.436 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:05:30.436 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:05:30.436 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:05:30.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.438 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "101", 2024-02-19T21:05:30.438 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:05:30.438 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:05:30.438 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:05:30.438 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:05:30.438 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:05:30.438 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:05:30.438 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:05:30.438 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:05:30.438 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:05:30.438 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:05:30.438 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:05:30.438 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:05:30.438 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:05:30.438 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:05:30.439 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:05:30.439 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:05:30.439 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:05:30.439 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:05:30.439 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:05:30.439 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:05:30.439 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:05:30.439 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:05:30.439 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:05:30.439 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:05:30.439 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:05:30.439 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:05:30.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.439 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:05:30.440 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:05:30.440 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:05:30.440 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:05:30.440 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:05:30.440 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:05:30.440 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:05:30.440 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:05:30.440 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:05:30.440 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:05:30.440 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:05:30.440 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:05:30.440 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:05:30.440 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-02-19T21:05:30.440 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.441 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 61, 2024-02-19T21:05:30.441 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "test-rados-api-smithi163-83928-6", 2024-02-19T21:05:30.441 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:52.991355+0000", 2024-02-19T21:05:30.441 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8193, 2024-02-19T21:05:30.441 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-19T21:05:30.441 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:05:30.441 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:05:30.441 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:05:30.441 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:05:30.441 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:05:30.441 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:05:30.442 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:05:30.442 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:05:30.442 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:05:30.442 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:05:30.442 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:05:30.442 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:05:30.442 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:05:30.442 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:05:30.442 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:05:30.442 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:05:30.442 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:05:30.442 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:05:30.442 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:05:30.442 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:05:30.443 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:05:30.443 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:05:30.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.443 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "150", 2024-02-19T21:05:30.443 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "150", 2024-02-19T21:05:30.443 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "150", 2024-02-19T21:05:30.443 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "150", 2024-02-19T21:05:30.443 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:05:30.443 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:05:30.443 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 7, 2024-02-19T21:05:30.443 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 149, 2024-02-19T21:05:30.443 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:05:30.443 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:05:30.443 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:05:30.443 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:05:30.444 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [ 2024-02-19T21:05:30.444 INFO:teuthology.orchestra.run.smithi163.stdout: 94 2024-02-19T21:05:30.444 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:05:30.444 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:05:30.444 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": 94, 2024-02-19T21:05:30.444 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": 94, 2024-02-19T21:05:30.444 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:05:30.444 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:05:30.444 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:05:30.444 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:05:30.444 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:05:30.444 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:05:30.444 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:05:30.445 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:05:30.445 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:05:30.445 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:05:30.445 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:05:30.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.445 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:05:30.445 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:05:30.445 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:05:30.445 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:05:30.445 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:05:30.445 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:05:30.445 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:05:30.445 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:05:30.445 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:05:30.445 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 64, 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "LibRadosSnapshotsSelfManagedPP_smithi163-83952-7", 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:55.373357+0000", 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8193, 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:05:30.446 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:05:30.447 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "143", 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 24, 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 143, 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:05:30.448 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.449 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.450 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 94, 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "test-rados-api-smithi163-83928-13", 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:05:22.433403+0000", 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8193, 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:05:30.451 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "150", 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "150", 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "150", 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "150", 2024-02-19T21:05:30.452 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 7, 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 149, 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": 61, 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:05:30.453 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:05:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:05:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:05:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:05:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:05:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:05:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:05:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:05:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:05:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:05:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:05:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:05:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:05:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:05:30.454 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 99, 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "FlushAsync_smithi163-83845-16", 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:05:28.477448+0000", 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 32769, 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,creating", 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:05:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:05:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:05:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:05:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:05:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:05:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:05:30.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.260+0000 7fdf0d09c700 10 librados: watch_flush enter 2024-02-19T21:05:30.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.260+0000 7fdf0d09c700 10 librados: watch_flush exit 2024-02-19T21:05:30.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.260+0000 7fdf0d09c700 20 client.48842.objecter shutdown clearing up homeless session... 2024-02-19T21:05:30.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.260+0000 7fdf0d09c700 10 client.48842.objecter successfully canceled tick 2024-02-19T21:05:30.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.260+0000 7fdf0d09c700 1 -- 172.21.15.163:0/3640601682 >> v1:172.21.15.163:6800/1552514681 conn(0x7fdef0063e70 legacy=0x7fdef0066330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:30.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.260+0000 7fdf0d09c700 1 -- 172.21.15.163:0/3640601682 >> v1:172.21.15.163:6790/0 conn(0x7fdf081031b0 legacy=0x7fdf08118340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:30.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.261+0000 7fdf0d09c700 1 -- 172.21.15.163:0/3640601682 shutdown_connections 2024-02-19T21:05:30.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.261+0000 7fdf0d09c700 1 -- 172.21.15.163:0/3640601682 >> 172.21.15.163:0/3640601682 conn(0x7fdf080f2470 msgr2=0x7fdf08106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:30.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.261+0000 7fdf0d09c700 1 -- 172.21.15.163:0/3640601682 shutdown_connections 2024-02-19T21:05:30.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.261+0000 7fdf0d09c700 1 -- 172.21.15.163:0/3640601682 wait complete. 2024-02-19T21:05:30.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:30.261+0000 7fdf0d09c700 1 librados: shutdown 2024-02-19T21:05:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:05:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:05:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:05:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:05:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:05:30.457 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:05:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:05:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:05:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:05:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:05:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:05:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:05:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:05:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "150", 2024-02-19T21:05:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:05:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:05:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:05:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:05:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:05:30.458 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:05:30.459 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:05:30.460 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 100, 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "ReturnValuePP_smithi163-83854-13", 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:05:29.444938+0000", 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 32769, 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,creating", 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:05:30.461 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:05:30.462 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "150", 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:05:30.463 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:05:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:05:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:05:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:05:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:05:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:05:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:05:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:05:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:05:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:05:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:05:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:05:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:05:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:05:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.464 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": {} 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-02-19T21:05:30.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088", 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 8, 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 149, 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.466 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6801", 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6802", 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:05:30.467 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6804", 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6803", 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.468 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6801/1559418115", 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6802/1559418115", 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6804/1559418115", 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6803/1559418115", 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "e74a65fa-cb9c-42c3-9293-602cf7e9dc67", 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:05:30.469 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 13, 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 149, 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6805", 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6806", 2024-02-19T21:05:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:05:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:05:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6808", 2024-02-19T21:05:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:05:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.471 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:05:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6807", 2024-02-19T21:05:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:05:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6805/3125047432", 2024-02-19T21:05:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6806/3125047432", 2024-02-19T21:05:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6808/3125047432", 2024-02-19T21:05:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6807/3125047432", 2024-02-19T21:05:30.472 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "6980a625-cdc9-487e-8f8c-7997ba716a97", 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 18, 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 149, 2024-02-19T21:05:30.473 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:05:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:05:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:05:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6809", 2024-02-19T21:05:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:05:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:05:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.474 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6810", 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6812", 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:05:30.475 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6811", 2024-02-19T21:05:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:05:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6809/1563158487", 2024-02-19T21:05:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6810/1563158487", 2024-02-19T21:05:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6812/1563158487", 2024-02-19T21:05:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6811/1563158487", 2024-02-19T21:05:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:05:30.476 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:05:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:05:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-02-19T21:05:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5", 2024-02-19T21:05:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:05:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:05:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:05:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:05:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:05:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:05:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 25, 2024-02-19T21:05:30.477 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 149, 2024-02-19T21:05:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:05:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:05:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:05:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6813", 2024-02-19T21:05:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:05:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:05:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.478 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6814", 2024-02-19T21:05:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:05:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:05:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6816", 2024-02-19T21:05:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:05:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.479 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:05:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6815", 2024-02-19T21:05:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:05:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6813/2112279769", 2024-02-19T21:05:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6814/2112279769", 2024-02-19T21:05:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6816/2112279769", 2024-02-19T21:05:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6815/2112279769", 2024-02-19T21:05:30.480 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:05:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:05:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:05:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-02-19T21:05:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0", 2024-02-19T21:05:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:05:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:05:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:05:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:05:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:05:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:05:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 31, 2024-02-19T21:05:30.481 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 149, 2024-02-19T21:05:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:05:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:05:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:05:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6800", 2024-02-19T21:05:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:05:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:05:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.482 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6801", 2024-02-19T21:05:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:05:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:05:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6803", 2024-02-19T21:05:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:05:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.483 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:05:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6802", 2024-02-19T21:05:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:05:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6800/3129716965", 2024-02-19T21:05:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6801/3129716965", 2024-02-19T21:05:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6803/3129716965", 2024-02-19T21:05:30.484 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6802/3129716965", 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f", 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 37, 2024-02-19T21:05:30.485 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 149, 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6804", 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.486 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6805", 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6807", 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6806", 2024-02-19T21:05:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:05:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6804/805161991", 2024-02-19T21:05:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6805/805161991", 2024-02-19T21:05:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6807/805161991", 2024-02-19T21:05:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6806/805161991", 2024-02-19T21:05:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:05:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:05:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:05:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-02-19T21:05:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "302c95df-3842-4818-8afd-d79d8184684f", 2024-02-19T21:05:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:05:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:05:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:05:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:05:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:05:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:05:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 45, 2024-02-19T21:05:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 149, 2024-02-19T21:05:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:05:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:05:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:05:30.489 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6808", 2024-02-19T21:05:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:05:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:05:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6809", 2024-02-19T21:05:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:05:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.490 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:05:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6811", 2024-02-19T21:05:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:05:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:05:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.491 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6810", 2024-02-19T21:05:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:05:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6808/787604615", 2024-02-19T21:05:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6809/787604615", 2024-02-19T21:05:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6811/787604615", 2024-02-19T21:05:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6810/787604615", 2024-02-19T21:05:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:05:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:05:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:05:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-02-19T21:05:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f", 2024-02-19T21:05:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:05:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:05:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:05:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:05:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:05:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:05:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 53, 2024-02-19T21:05:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 149, 2024-02-19T21:05:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:05:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:05:30.493 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:05:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6812", 2024-02-19T21:05:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:05:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:05:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6813", 2024-02-19T21:05:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:05:30.494 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6815", 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:05:30.495 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6814", 2024-02-19T21:05:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:05:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6812/157927432", 2024-02-19T21:05:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6813/157927432", 2024-02-19T21:05:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6815/157927432", 2024-02-19T21:05:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6814/157927432", 2024-02-19T21:05:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:05:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:05:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:05:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.496 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:05:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_xinfo": [ 2024-02-19T21:05:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-02-19T21:05:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:05:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:05:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:05:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:05:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:05:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:04:41.038878+0000", 2024-02-19T21:05:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:05:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-02-19T21:05:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:05:30.497 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:04:40.274356+0000", 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:05:22.137628+0000", 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:05:30.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:04:19.405976+0000", 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:05:30.499 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:04:36.495035+0000", 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:04:03.077388+0000", 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.500 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-02-19T21:05:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:05:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:05:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:05:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:05:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:05:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:04:35.069603+0000", 2024-02-19T21:05:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:05:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-02-19T21:05:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:05:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:05:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:05:30.501 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:05:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:05:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:04:43.864482+0000", 2024-02-19T21:05:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:05:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:05:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap": [ 2024-02-19T21:05:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "2.d", 2024-02-19T21:05:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-02-19T21:05:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: 1, 2024-02-19T21:05:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: 4, 2024-02-19T21:05:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: 2 2024-02-19T21:05:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.502 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:05:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_items": [ 2024-02-19T21:05:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "2.4", 2024-02-19T21:05:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:05:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 7, 2024-02-19T21:05:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-02-19T21:05:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.7", 2024-02-19T21:05:30.503 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:05:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:05:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-02-19T21:05:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.d", 2024-02-19T21:05:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:05:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:05:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 5 2024-02-19T21:05:30.504 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.12", 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 5 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_temp": [], 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_temp": [], 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "blocklist": { 2024-02-19T21:05:30.505 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1427774891": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:05:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3282102947": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:05:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2735092549": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:05:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1235588846": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:05:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2731225015": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:05:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/506491651": "2024-02-20T20:56:02.185409+0000", 2024-02-19T21:05:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/168117826": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:05:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1458009465": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:05:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/541631472": "2024-02-20T20:56:02.185409+0000", 2024-02-19T21:05:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2248999541": "2024-02-20T20:56:02.185409+0000" 2024-02-19T21:05:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "range_blocklist": {}, 2024-02-19T21:05:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profiles": { 2024-02-19T21:05:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "default": { 2024-02-19T21:05:30.506 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-02-19T21:05:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-02-19T21:05:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-02-19T21:05:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-02-19T21:05:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "ruleset-failure-domain": "osd", 2024-02-19T21:05:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van" 2024-02-19T21:05:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14": { 2024-02-19T21:05:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-device-class": "", 2024-02-19T21:05:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-02-19T21:05:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-root": "default", 2024-02-19T21:05:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-19T21:05:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-02-19T21:05:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-02-19T21:05:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-02-19T21:05:30.507 INFO:teuthology.orchestra.run.smithi163.stdout: "ruleset-failure-domain": "osd", 2024-02-19T21:05:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van", 2024-02-19T21:05:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "w": "8" 2024-02-19T21:05:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps_queue": [ 2024-02-19T21:05:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 11, 2024-02-19T21:05:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:05:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:05:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:05:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:05:30.508 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 9, 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 25, 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.509 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:05:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:05:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:05:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:05:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 57, 2024-02-19T21:05:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:05:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:05:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:05:30.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:05:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:05:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 76, 2024-02-19T21:05:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:05:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:05:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:05:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.511 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:05:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:05:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:05:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:05:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 85, 2024-02-19T21:05:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:05:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:05:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:05:30.512 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 92, 2024-02-19T21:05:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:05:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:05:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:05:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:05:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:05:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:05:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:05:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.513 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:05:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:05:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "new_removed_snaps": [], 2024-02-19T21:05:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "new_purged_snaps": [], 2024-02-19T21:05:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_node_flags": {}, 2024-02-19T21:05:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "device_class_flags": {}, 2024-02-19T21:05:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode": { 2024-02-19T21:05:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_enabled": false, 2024-02-19T21:05:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_bucket_count": 0, 2024-02-19T21:05:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "degraded_stretch_mode": 0, 2024-02-19T21:05:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "recovering_stretch_mode": 0, 2024-02-19T21:05:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_bucket": 0 2024-02-19T21:05:30.514 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:05:30.514 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:05:30.515 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:05:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:05:30.515 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:29.124637+0000 mgr.y (mgr.14550) 191 : cluster [DBG] pgmap v194: 374 pgs: 32 unknown, 1 active+clean+snaptrim, 1 active+clean+scrubbing+deep, 340 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:30.515 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:29.677476+0000 mon.c (mon.2) 134 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:30.515 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:30.164072+0000 mon.a (mon.0) 1162 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-13"}]': finished 2024-02-19T21:05:30.515 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:30.164180+0000 mon.a (mon.0) 1163 : audit [INF] from='client.48827 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi163-83845-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:30.516 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:30.164335+0000 mon.a (mon.0) 1164 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-02-19T21:05:30.516 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:30.173738+0000 mon.b (mon.1) 235 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-13", "mode": "writeback"}]: dispatch 2024-02-19T21:05:30.516 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:30.174808+0000 mon.a (mon.0) 1165 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-13", "mode": "writeback"}]: dispatch 2024-02-19T21:05:30.516 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:30.175306+0000 mon.b (mon.1) 236 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:30.516 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:30.175801+0000 mon.a (mon.0) 1166 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:30.516 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:30.177594+0000 mon.b (mon.1) 237 : audit [INF] from='client.? v1:172.21.15.163:0/94307050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi163-83854-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:30.516 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:30.179426+0000 mon.a (mon.0) 1167 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi163-83854-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:30.516 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:30.257995+0000 mon.c (mon.2) 135 : audit [DBG] from='client.? v1:172.21.15.163:0/3640601682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:05:30.516 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:30.516 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:29.124637+0000 mgr.y (mgr.14550) 191 : cluster [DBG] pgmap v194: 374 pgs: 32 unknown, 1 active+clean+snaptrim, 1 active+clean+scrubbing+deep, 340 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:30.516 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:29.677476+0000 mon.c (mon.2) 134 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:30.517 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:30.164072+0000 mon.a (mon.0) 1162 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-13"}]': finished 2024-02-19T21:05:30.517 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:30.164180+0000 mon.a (mon.0) 1163 : audit [INF] from='client.48827 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi163-83845-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:30.517 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:30.164335+0000 mon.a (mon.0) 1164 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-02-19T21:05:30.517 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:30.173738+0000 mon.b (mon.1) 235 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-13", "mode": "writeback"}]: dispatch 2024-02-19T21:05:30.517 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:30.174808+0000 mon.a (mon.0) 1165 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-13", "mode": "writeback"}]: dispatch 2024-02-19T21:05:30.517 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:30.175306+0000 mon.b (mon.1) 236 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:30.517 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:30.175801+0000 mon.a (mon.0) 1166 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:30.517 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:30.177594+0000 mon.b (mon.1) 237 : audit [INF] from='client.? v1:172.21.15.163:0/94307050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi163-83854-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:30.517 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:30.179426+0000 mon.a (mon.0) 1167 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi163-83854-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:30.517 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:30.257995+0000 mon.c (mon.2) 135 : audit [DBG] from='client.? v1:172.21.15.163:0/3640601682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:05:30.605 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:30.607 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-19T21:05:30.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:30.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:30.655 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:30.673 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:29.124637+0000 mgr.y (mgr.14550) 191 : cluster [DBG] pgmap v194: 374 pgs: 32 unknown, 1 active+clean+snaptrim, 1 active+clean+scrubbing+deep, 340 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:30.673 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:29.677476+0000 mon.c (mon.2) 134 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:30.673 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:30.164072+0000 mon.a (mon.0) 1162 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-13"}]': finished 2024-02-19T21:05:30.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:30.164180+0000 mon.a (mon.0) 1163 : audit [INF] from='client.48827 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi163-83845-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:30.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:30.164335+0000 mon.a (mon.0) 1164 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-02-19T21:05:30.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:30.173738+0000 mon.b (mon.1) 235 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-13", "mode": "writeback"}]: dispatch 2024-02-19T21:05:30.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:30.174808+0000 mon.a (mon.0) 1165 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-13", "mode": "writeback"}]: dispatch 2024-02-19T21:05:30.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:30.175306+0000 mon.b (mon.1) 236 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:30.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:30.175801+0000 mon.a (mon.0) 1166 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:30.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:30.177594+0000 mon.b (mon.1) 237 : audit [INF] from='client.? v1:172.21.15.163:0/94307050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi163-83854-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:30.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:30.179426+0000 mon.a (mon.0) 1167 : audit [INF] from='client.47632 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi163-83854-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:30.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:30.257995+0000 mon.c (mon.2) 135 : audit [DBG] from='client.? v1:172.21.15.163:0/3640601682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:05:30.709 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:30.737 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:30.757 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:30.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:30.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:30.860 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:30.887 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:30.909 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:30.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:30.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:30.926 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:05:30.991 INFO:tasks.thrashosds.thrasher:j is {'epoch': 150, 'fsid': '2591809a-cf69-11ee-95bb-87774f69a715', 'created': '2024-02-19T20:55:37.794885+0000', 'modified': '2024-02-19T21:05:30.157979+0000', 'last_up_change': '2024-02-19T21:01:09.214134+0000', 'last_in_change': '2024-02-19T21:00:51.699017+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 27, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 100, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-19T20:59:09.425169+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-19T21:01:14.658945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '139', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-19T21:01:14.958800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-19T21:01:16.543816+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-19T21:01:18.521482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-19T21:01:20.586564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 8, 'pool_name': 'LibRadosList_smithi163-83888-1', 'create_time': '2024-02-19T21:04:33.426193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 16, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-19T21:04:35.333192+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 28, 'pool_name': 'LibRadosMiscPP_smithi163-83920-1', 'create_time': '2024-02-19T21:04:38.251419+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 61, 'pool_name': 'test-rados-api-smithi163-83928-6', 'create_time': '2024-02-19T21:04:52.991355+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '150', 'last_force_op_resend_prenautilus': '150', 'last_force_op_resend_preluminous': '150', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 149, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [94], 'tier_of': -1, 'read_tier': 94, 'write_tier': 94, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 64, 'pool_name': 'LibRadosSnapshotsSelfManagedPP_smithi163-83952-7', 'create_time': '2024-02-19T21:04:55.373357+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '143', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 24, 'snap_epoch': 143, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 94, 'pool_name': 'test-rados-api-smithi163-83928-13', 'create_time': '2024-02-19T21:05:22.433403+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '150', 'last_force_op_resend_prenautilus': '150', 'last_force_op_resend_preluminous': '150', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 149, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 61, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 99, 'pool_name': 'FlushAsync_smithi163-83845-16', 'create_time': '2024-02-19T21:05:28.477448+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 100, 'pool_name': 'ReturnValuePP_smithi163-83854-13', 'create_time': '2024-02-19T21:05:29.444938+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '4a14b406-7c21-40f3-8b12-8cf9df811088', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6801', 'nonce': 1559418115}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6802', 'nonce': 1559418115}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6804', 'nonce': 1559418115}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6803', 'nonce': 1559418115}]}, 'public_addr': '172.21.15.163:6801/1559418115', 'cluster_addr': '172.21.15.163:6802/1559418115', 'heartbeat_back_addr': '172.21.15.163:6804/1559418115', 'heartbeat_front_addr': '172.21.15.163:6803/1559418115', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e74a65fa-cb9c-42c3-9293-602cf7e9dc67', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6805', 'nonce': 3125047432}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6806', 'nonce': 3125047432}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6808', 'nonce': 3125047432}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6807', 'nonce': 3125047432}]}, 'public_addr': '172.21.15.163:6805/3125047432', 'cluster_addr': '172.21.15.163:6806/3125047432', 'heartbeat_back_addr': '172.21.15.163:6808/3125047432', 'heartbeat_front_addr': '172.21.15.163:6807/3125047432', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6980a625-cdc9-487e-8f8c-7997ba716a97', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6809', 'nonce': 1563158487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6810', 'nonce': 1563158487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6812', 'nonce': 1563158487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6811', 'nonce': 1563158487}]}, 'public_addr': '172.21.15.163:6809/1563158487', 'cluster_addr': '172.21.15.163:6810/1563158487', 'heartbeat_back_addr': '172.21.15.163:6812/1563158487', 'heartbeat_front_addr': '172.21.15.163:6811/1563158487', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '359e2ee0-9d1d-4074-8119-2d69bc2ca3a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6813', 'nonce': 2112279769}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6814', 'nonce': 2112279769}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6816', 'nonce': 2112279769}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6815', 'nonce': 2112279769}]}, 'public_addr': '172.21.15.163:6813/2112279769', 'cluster_addr': '172.21.15.163:6814/2112279769', 'heartbeat_back_addr': '172.21.15.163:6816/2112279769', 'heartbeat_front_addr': '172.21.15.163:6815/2112279769', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2bf5ef05-292b-4693-85a9-97991f2091e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6800', 'nonce': 3129716965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6801', 'nonce': 3129716965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6803', 'nonce': 3129716965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6802', 'nonce': 3129716965}]}, 'public_addr': '172.21.15.193:6800/3129716965', 'cluster_addr': '172.21.15.193:6801/3129716965', 'heartbeat_back_addr': '172.21.15.193:6803/3129716965', 'heartbeat_front_addr': '172.21.15.193:6802/3129716965', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '59065f9b-9240-414b-9c91-e91cda4aed6f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6804', 'nonce': 805161991}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6805', 'nonce': 805161991}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6807', 'nonce': 805161991}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6806', 'nonce': 805161991}]}, 'public_addr': '172.21.15.193:6804/805161991', 'cluster_addr': '172.21.15.193:6805/805161991', 'heartbeat_back_addr': '172.21.15.193:6807/805161991', 'heartbeat_front_addr': '172.21.15.193:6806/805161991', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '302c95df-3842-4818-8afd-d79d8184684f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6808', 'nonce': 787604615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6809', 'nonce': 787604615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6811', 'nonce': 787604615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6810', 'nonce': 787604615}]}, 'public_addr': '172.21.15.193:6808/787604615', 'cluster_addr': '172.21.15.193:6809/787604615', 'heartbeat_back_addr': '172.21.15.193:6811/787604615', 'heartbeat_front_addr': '172.21.15.193:6810/787604615', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '9ebeff27-c2f8-4a66-923f-a0d7573ddd3f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6812', 'nonce': 157927432}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6813', 'nonce': 157927432}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6815', 'nonce': 157927432}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6814', 'nonce': 157927432}]}, 'public_addr': '172.21.15.193:6812/157927432', 'cluster_addr': '172.21.15.193:6813/157927432', 'heartbeat_back_addr': '172.21.15.193:6815/157927432', 'heartbeat_front_addr': '172.21.15.193:6814/157927432', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:04:41.038878+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:04:40.274356+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:05:22.137628+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:04:19.405976+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:04:36.495035+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:04:03.077388+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:04:35.069603+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:04:43.864482+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.d', 'osds': [1, 4, 2]}], 'pg_upmap_items': [{'pgid': '2.4', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.163:0/1427774891': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/3282102947': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:6800/2735092549': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/1235588846': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:6800/2731225015': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:0/506491651': '2024-02-20T20:56:02.185409+0000', '172.21.15.163:0/168117826': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/1458009465': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:0/541631472': '2024-02-20T20:56:02.185409+0000', '172.21.15.163:6800/2248999541': '2024-02-20T20:56:02.185409+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 11, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 25, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 57, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 76, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 85, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 92, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-19T21:05:30.992 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:05:31.014 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:05:31.061 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:31.458 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:30.131578+0000 mgr.y (mgr.14550) 192 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:31.458 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:30.292164+0000 osd.5 (osd.5) 80 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:31.458 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:30.292758+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:31.458 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:30.678474+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:31.459 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:31.160282+0000 mon.a (mon.0) 1168 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:05:31.459 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.167419+0000 mon.a (mon.0) 1169 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-13", "mode": "writeback"}]': finished 2024-02-19T21:05:31.459 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.167509+0000 mon.a (mon.0) 1170 : audit [INF] from='client.45484 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]': finished 2024-02-19T21:05:31.459 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.167567+0000 mon.a (mon.0) 1171 : audit [INF] from='client.47632 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi163-83854-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:31.459 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:31.167688+0000 mon.a (mon.0) 1172 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-02-19T21:05:31.459 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.181454+0000 mon.b (mon.1) 238 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:31.459 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.184053+0000 mon.a (mon.0) 1173 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:31.459 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:30.131578+0000 mgr.y (mgr.14550) 192 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:31.460 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:30.292164+0000 osd.5 (osd.5) 80 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:31.460 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:30.292758+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:31.460 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:30.678474+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:31.460 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:31.160282+0000 mon.a (mon.0) 1168 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:05:31.460 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.167419+0000 mon.a (mon.0) 1169 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-13", "mode": "writeback"}]': finished 2024-02-19T21:05:31.460 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.167509+0000 mon.a (mon.0) 1170 : audit [INF] from='client.45484 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]': finished 2024-02-19T21:05:31.460 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.167567+0000 mon.a (mon.0) 1171 : audit [INF] from='client.47632 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi163-83854-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:31.460 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:31.167688+0000 mon.a (mon.0) 1172 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-02-19T21:05:31.460 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.181454+0000 mon.b (mon.1) 238 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:31.460 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.184053+0000 mon.a (mon.0) 1173 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:31.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:30.131578+0000 mgr.y (mgr.14550) 192 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:31.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:30.292164+0000 osd.5 (osd.5) 80 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:31.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:30.292758+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:31.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:30.678474+0000 mon.c (mon.2) 136 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:31.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:31.160282+0000 mon.a (mon.0) 1168 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:05:31.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.167419+0000 mon.a (mon.0) 1169 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-13", "mode": "writeback"}]': finished 2024-02-19T21:05:31.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.167509+0000 mon.a (mon.0) 1170 : audit [INF] from='client.45484 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]': finished 2024-02-19T21:05:31.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.167567+0000 mon.a (mon.0) 1171 : audit [INF] from='client.47632 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi163-83854-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:31.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:31.167688+0000 mon.a (mon.0) 1172 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-02-19T21:05:31.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.181454+0000 mon.b (mon.1) 238 : audit [INF] from='client.? v1:172.21.15.163:0/4096098867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:31.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.184053+0000 mon.a (mon.0) 1173 : audit [INF] from='client.45484 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]: dispatch 2024-02-19T21:05:31.911 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:05:31.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:31.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:05:32.015 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:05:32.042 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:32.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: Running main() from gmock_main.cc 2024-02-19T21:05:32.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [==========] Running 12 tests from 4 test suites. 2024-02-19T21:05:32.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [----------] Global test environment set-up. 2024-02-19T21:05:32.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-02-19T21:05:32.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-02-19T21:05:32.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2073 ms) 2024-02-19T21:05:32.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-02-19T21:05:32.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (4029 ms) 2024-02-19T21:05:32.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-02-19T21:05:32.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2292 ms) 2024-02-19T21:05:32.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-02-19T21:05:32.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (1618 ms) 2024-02-19T21:05:32.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (10012 ms total) 2024-02-19T21:05:32.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: 2024-02-19T21:05:32.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged 2024-02-19T21:05:32.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-02-19T21:05:32.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4161 ms) 2024-02-19T21:05:32.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-02-19T21:05:32.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4417 ms) 2024-02-19T21:05:32.183 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged (8578 ms total) 2024-02-19T21:05:32.183 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: 2024-02-19T21:05:32.183 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-02-19T21:05:32.183 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-02-19T21:05:32.183 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2989 ms) 2024-02-19T21:05:32.183 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-02-19T21:05:32.183 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2007 ms) 2024-02-19T21:05:32.183 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-02-19T21:05:32.183 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2082 ms) 2024-02-19T21:05:32.183 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-02-19T21:05:32.183 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2086 ms) 2024-02-19T21:05:32.183 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9164 ms total) 2024-02-19T21:05:32.183 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: 2024-02-19T21:05:32.183 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-02-19T21:05:32.183 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-02-19T21:05:32.184 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (3996 ms) 2024-02-19T21:05:32.184 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-02-19T21:05:32.184 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (3993 ms) 2024-02-19T21:05:32.184 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (7989 ms total) 2024-02-19T21:05:32.184 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: 2024-02-19T21:05:32.184 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [----------] Global test environment tear-down 2024-02-19T21:05:32.184 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [==========] 12 tests from 4 test suites ran. (59339 ms total) 2024-02-19T21:05:32.184 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots: [ PASSED ] 12 tests. 2024-02-19T21:05:32.184 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:05:32.184 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=84019 2024-02-19T21:05:32.185 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 84019 2024-02-19T21:05:32.185 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:05:32.185 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83875 2024-02-19T21:05:32.185 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83875 2024-02-19T21:05:32.185 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:05:32.185 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83979 2024-02-19T21:05:32.185 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83979 2024-02-19T21:05:32.185 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:05:32.185 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83947 2024-02-19T21:05:32.185 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83947 2024-02-19T21:05:32.289 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:05:32.460 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:31.125093+0000 mgr.y (mgr.14550) 193 : cluster [DBG] pgmap v197: 398 pgs: 32 unknown, 32 creating+peering, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 96 KiB/s wr, 100 op/s 2024-02-19T21:05:32.460 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:31.459123+0000 mon.a (mon.0) 1174 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:05:32.460 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.679199+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:32.460 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.824960+0000 mon.b (mon.1) 239 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.0"}]: dispatch 2024-02-19T21:05:32.460 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.825203+0000 mgr.y (mgr.14550) 194 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.0"}]: dispatch 2024-02-19T21:05:32.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:31.125093+0000 mgr.y (mgr.14550) 193 : cluster [DBG] pgmap v197: 398 pgs: 32 unknown, 32 creating+peering, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 96 KiB/s wr, 100 op/s 2024-02-19T21:05:32.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:31.459123+0000 mon.a (mon.0) 1174 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:05:32.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.679199+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:32.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.824960+0000 mon.b (mon.1) 239 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.0"}]: dispatch 2024-02-19T21:05:32.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.825203+0000 mgr.y (mgr.14550) 194 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.0"}]: dispatch 2024-02-19T21:05:32.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.825571+0000 mon.b (mon.1) 240 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.1"}]: dispatch 2024-02-19T21:05:32.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.825716+0000 mgr.y (mgr.14550) 195 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.1"}]: dispatch 2024-02-19T21:05:32.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.826036+0000 mon.b (mon.1) 241 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.2"}]: dispatch 2024-02-19T21:05:32.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.826148+0000 mgr.y (mgr.14550) 196 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.2"}]: dispatch 2024-02-19T21:05:32.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.826449+0000 mon.b (mon.1) 242 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.3"}]: dispatch 2024-02-19T21:05:32.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.826580+0000 mgr.y (mgr.14550) 197 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.3"}]: dispatch 2024-02-19T21:05:32.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.827061+0000 mon.b (mon.1) 243 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.4"}]: dispatch 2024-02-19T21:05:32.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.827219+0000 mgr.y (mgr.14550) 198 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.4"}]: dispatch 2024-02-19T21:05:32.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.827495+0000 mon.b (mon.1) 244 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.5"}]: dispatch 2024-02-19T21:05:32.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.827616+0000 mgr.y (mgr.14550) 199 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.5"}]: dispatch 2024-02-19T21:05:32.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.827876+0000 mon.b (mon.1) 245 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.6"}]: dispatch 2024-02-19T21:05:32.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.828003+0000 mgr.y (mgr.14550) 200 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.6"}]: dispatch 2024-02-19T21:05:32.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.828288+0000 mon.b (mon.1) 246 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.7"}]: dispatch 2024-02-19T21:05:32.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.828414+0000 mgr.y (mgr.14550) 201 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.7"}]: dispatch 2024-02-19T21:05:32.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.828690+0000 mon.b (mon.1) 247 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.8"}]: dispatch 2024-02-19T21:05:32.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.828805+0000 mgr.y (mgr.14550) 202 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.8"}]: dispatch 2024-02-19T21:05:32.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.829078+0000 mon.b (mon.1) 248 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.9"}]: dispatch 2024-02-19T21:05:32.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:31.829198+0000 mgr.y (mgr.14550) 203 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.9"}]: dispatch 2024-02-19T21:05:32.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:32.172658+0000 mon.a (mon.0) 1175 : audit [INF] from='client.45484 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]': finished 2024-02-19T21:05:32.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:32.172798+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-02-19T21:05:32.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:32.180431+0000 mon.c (mon.2) 138 : audit [INF] from='client.? v1:172.21.15.163:0/2989707866' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi163-83845-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:32.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:32.181053+0000 mon.a (mon.0) 1177 : audit [INF] from='client.48848 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi163-83845-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:32.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:32.222903+0000 osd.7 (osd.7) 85 : cluster [DBG] 94.1 scrub starts 2024-02-19T21:05:32.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:32.226378+0000 osd.7 (osd.7) 86 : cluster [DBG] 94.1 scrub ok 2024-02-19T21:05:32.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.825571+0000 mon.b (mon.1) 240 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.1"}]: dispatch 2024-02-19T21:05:32.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.825716+0000 mgr.y (mgr.14550) 195 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.1"}]: dispatch 2024-02-19T21:05:32.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.826036+0000 mon.b (mon.1) 241 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.2"}]: dispatch 2024-02-19T21:05:32.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.826148+0000 mgr.y (mgr.14550) 196 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.2"}]: dispatch 2024-02-19T21:05:32.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.826449+0000 mon.b (mon.1) 242 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.3"}]: dispatch 2024-02-19T21:05:32.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.826580+0000 mgr.y (mgr.14550) 197 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.3"}]: dispatch 2024-02-19T21:05:32.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.827061+0000 mon.b (mon.1) 243 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.4"}]: dispatch 2024-02-19T21:05:32.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.827219+0000 mgr.y (mgr.14550) 198 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.4"}]: dispatch 2024-02-19T21:05:32.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.827495+0000 mon.b (mon.1) 244 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.5"}]: dispatch 2024-02-19T21:05:32.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.827616+0000 mgr.y (mgr.14550) 199 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.5"}]: dispatch 2024-02-19T21:05:32.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.827876+0000 mon.b (mon.1) 245 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.6"}]: dispatch 2024-02-19T21:05:32.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.828003+0000 mgr.y (mgr.14550) 200 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.6"}]: dispatch 2024-02-19T21:05:32.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.828288+0000 mon.b (mon.1) 246 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.7"}]: dispatch 2024-02-19T21:05:32.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.828414+0000 mgr.y (mgr.14550) 201 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.7"}]: dispatch 2024-02-19T21:05:32.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.828690+0000 mon.b (mon.1) 247 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.8"}]: dispatch 2024-02-19T21:05:32.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:31.125093+0000 mgr.y (mgr.14550) 193 : cluster [DBG] pgmap v197: 398 pgs: 32 unknown, 32 creating+peering, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 96 KiB/s wr, 100 op/s 2024-02-19T21:05:32.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:31.459123+0000 mon.a (mon.0) 1174 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:05:32.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.679199+0000 mon.c (mon.2) 137 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:32.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.824960+0000 mon.b (mon.1) 239 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.0"}]: dispatch 2024-02-19T21:05:32.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.825203+0000 mgr.y (mgr.14550) 194 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.0"}]: dispatch 2024-02-19T21:05:32.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.825571+0000 mon.b (mon.1) 240 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.1"}]: dispatch 2024-02-19T21:05:32.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.825716+0000 mgr.y (mgr.14550) 195 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.1"}]: dispatch 2024-02-19T21:05:32.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.826036+0000 mon.b (mon.1) 241 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.2"}]: dispatch 2024-02-19T21:05:32.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.826148+0000 mgr.y (mgr.14550) 196 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.2"}]: dispatch 2024-02-19T21:05:32.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.826449+0000 mon.b (mon.1) 242 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.3"}]: dispatch 2024-02-19T21:05:32.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.826580+0000 mgr.y (mgr.14550) 197 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.3"}]: dispatch 2024-02-19T21:05:32.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.827061+0000 mon.b (mon.1) 243 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.4"}]: dispatch 2024-02-19T21:05:32.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.827219+0000 mgr.y (mgr.14550) 198 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.4"}]: dispatch 2024-02-19T21:05:32.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.827495+0000 mon.b (mon.1) 244 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.5"}]: dispatch 2024-02-19T21:05:32.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.827616+0000 mgr.y (mgr.14550) 199 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.5"}]: dispatch 2024-02-19T21:05:32.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.828805+0000 mgr.y (mgr.14550) 202 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.8"}]: dispatch 2024-02-19T21:05:32.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.829078+0000 mon.b (mon.1) 248 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.9"}]: dispatch 2024-02-19T21:05:32.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:31.829198+0000 mgr.y (mgr.14550) 203 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.9"}]: dispatch 2024-02-19T21:05:32.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:32.172658+0000 mon.a (mon.0) 1175 : audit [INF] from='client.45484 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]': finished 2024-02-19T21:05:32.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:32.172798+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-02-19T21:05:32.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:32.180431+0000 mon.c (mon.2) 138 : audit [INF] from='client.? v1:172.21.15.163:0/2989707866' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi163-83845-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:32.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:32.181053+0000 mon.a (mon.0) 1177 : audit [INF] from='client.48848 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi163-83845-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:32.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:32.222903+0000 osd.7 (osd.7) 85 : cluster [DBG] 94.1 scrub starts 2024-02-19T21:05:32.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:32.226378+0000 osd.7 (osd.7) 86 : cluster [DBG] 94.1 scrub ok 2024-02-19T21:05:32.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.827876+0000 mon.b (mon.1) 245 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.6"}]: dispatch 2024-02-19T21:05:32.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.828003+0000 mgr.y (mgr.14550) 200 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.6"}]: dispatch 2024-02-19T21:05:32.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.828288+0000 mon.b (mon.1) 246 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.7"}]: dispatch 2024-02-19T21:05:32.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.828414+0000 mgr.y (mgr.14550) 201 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.7"}]: dispatch 2024-02-19T21:05:32.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.828690+0000 mon.b (mon.1) 247 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.8"}]: dispatch 2024-02-19T21:05:32.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.828805+0000 mgr.y (mgr.14550) 202 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.8"}]: dispatch 2024-02-19T21:05:32.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.829078+0000 mon.b (mon.1) 248 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "94.9"}]: dispatch 2024-02-19T21:05:32.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:31.829198+0000 mgr.y (mgr.14550) 203 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "94.9"}]: dispatch 2024-02-19T21:05:32.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:32.172658+0000 mon.a (mon.0) 1175 : audit [INF] from='client.45484 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi163-83944-14"}]': finished 2024-02-19T21:05:32.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:32.172798+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-02-19T21:05:32.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:32.180431+0000 mon.c (mon.2) 138 : audit [INF] from='client.? v1:172.21.15.163:0/2989707866' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi163-83845-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:32.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:32.181053+0000 mon.a (mon.0) 1177 : audit [INF] from='client.48848 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi163-83845-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:32.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:32.222903+0000 osd.7 (osd.7) 85 : cluster [DBG] 94.1 scrub starts 2024-02-19T21:05:32.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:32.226378+0000 osd.7 (osd.7) 86 : cluster [DBG] 94.1 scrub ok 2024-02-19T21:05:32.852 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:32.912 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:05:32.912 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:05:32.912 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-02-19T21:05:32.912 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:05:33.091 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:05:33.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:33.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:05:33.195 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:33.225 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:33.277 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:33.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:33.391 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:33.417 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:33.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:32.680412+0000 mon.c (mon.2) 139 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:33.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:33.168782+0000 mon.a (mon.0) 1178 : audit [INF] from='client.48848 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi163-83845-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:33.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:33.168933+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-02-19T21:05:33.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:33.179047+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? v1:172.21.15.163:0/1800034971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi163-83854-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:33.722 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-19T21:05:33.741 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:05:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:05:33] "GET /metrics HTTP/1.1" 200 197066 "" "Prometheus/2.33.4" 2024-02-19T21:05:33.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:32.680412+0000 mon.c (mon.2) 139 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:33.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:33.168782+0000 mon.a (mon.0) 1178 : audit [INF] from='client.48848 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi163-83845-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:33.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:33.168933+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-02-19T21:05:33.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:33.179047+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? v1:172.21.15.163:0/1800034971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi163-83854-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:33.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:32.680412+0000 mon.c (mon.2) 139 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:33.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:33.168782+0000 mon.a (mon.0) 1178 : audit [INF] from='client.48848 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi163-83845-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:33.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:33.168933+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-02-19T21:05:33.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:33.179047+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? v1:172.21.15.163:0/1800034971' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi163-83854-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:33.765 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:33.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:33.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:33.869 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:33.895 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:33.914 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:33.916 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:33.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:34.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:05:34.036 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:05:34.046 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:34.067 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:05:34.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:34.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:05:34.172 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:05:34.219 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:34.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.685+0000 7f9eb2366700 1 -- 172.21.15.163:0/968550629 >> v1:172.21.15.163:6790/0 conn(0x7f9eac1042c0 legacy=0x7f9eac106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:34.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.685+0000 7f9eb2366700 1 -- 172.21.15.163:0/968550629 shutdown_connections 2024-02-19T21:05:34.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.685+0000 7f9eb2366700 1 -- 172.21.15.163:0/968550629 >> 172.21.15.163:0/968550629 conn(0x7f9eac0f35a0 msgr2=0x7f9eac0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:34.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.685+0000 7f9eb2366700 1 -- 172.21.15.163:0/968550629 shutdown_connections 2024-02-19T21:05:34.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.685+0000 7f9eb2366700 1 -- 172.21.15.163:0/968550629 wait complete. 2024-02-19T21:05:34.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.685+0000 7f9eb2366700 1 librados: starting msgr at 2024-02-19T21:05:34.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.685+0000 7f9eb2366700 1 librados: starting objecter 2024-02-19T21:05:34.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.685+0000 7f9eb2366700 1 Processor -- start 2024-02-19T21:05:34.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.685+0000 7f9eb2366700 1 -- start start 2024-02-19T21:05:34.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.685+0000 7f9eb2366700 1 librados: setting wanted keys 2024-02-19T21:05:34.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.685+0000 7f9eb2366700 1 librados: calling monclient init 2024-02-19T21:05:34.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9eb2366700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9eac0fcf30 con 0x7f9eac0fd980 2024-02-19T21:05:34.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9eb2366700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9eac0fd0c0 con 0x7f9eac1042c0 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9eb2366700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9eac0fd290 con 0x7f9eac100780 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9eabfff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f9eac0fd980 0x7f9eac0fc790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:36128/0 (socket says 172.21.15.163:36128) 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9eabfff700 1 -- 172.21.15.163:0/3337436897 learned_addr learned my addr 172.21.15.163:0/3337436897 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9eab7fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f9eac100780 0x7f9eac114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:40580/0 (socket says 172.21.15.163:40580) 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 10 client.?.objecter ms_handle_connect 0x7f9eac100780 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 10 client.?.objecter ms_handle_connect 0x7f9eac0fd980 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264484131 0 0) 0x7f9eac0fd290 con 0x7f9eac100780 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e940033f0 con 0x7f9eac100780 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572448189 0 0) 0x7f9eac0fcf30 con 0x7f9eac0fd980 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9eac0fd290 con 0x7f9eac0fd980 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 10 client.?.objecter ms_handle_connect 0x7f9eac1042c0 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549125934 0 0) 0x7f9eac0fd0c0 con 0x7f9eac1042c0 2024-02-19T21:05:34.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9eac0fcf30 con 0x7f9eac1042c0 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3918437826 0 0) 0x7f9eac0fd290 con 0x7f9eac0fd980 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9eac0fd0c0 con 0x7f9eac0fd980 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9ea00019d0 con 0x7f9eac0fd980 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3616193797 0 0) 0x7f9e940033f0 con 0x7f9eac100780 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9eac0fd290 con 0x7f9eac100780 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9e9c0028f0 con 0x7f9eac100780 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.686+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2781783101 0 0) 0x7f9eac0fcf30 con 0x7f9eac1042c0 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.687+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e940033f0 con 0x7f9eac1042c0 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.687+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9ea40023d0 con 0x7f9eac1042c0 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.687+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1146186317 0 0) 0x7f9eac0fd0c0 con 0x7f9eac0fd980 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.687+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 >> v1:172.21.15.163:6790/0 conn(0x7f9eac100780 legacy=0x7f9eac114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.687+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 >> v1:172.21.15.193:6789/0 conn(0x7f9eac1042c0 legacy=0x7f9eac118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.687+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eac11cb90 con 0x7f9eac0fd980 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.687+0000 7f9eb2366700 1 -- 172.21.15.163:0/3337436897 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eac119bc0 con 0x7f9eac0fd980 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.687+0000 7f9eb2366700 10 client.22236.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:34.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.687+0000 7f9eb2366700 1 -- 172.21.15.163:0/3337436897 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eac11a030 con 0x7f9eac0fd980 2024-02-19T21:05:34.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.687+0000 7f9eb2366700 1 librados: init done 2024-02-19T21:05:34.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.687+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9ea00024b0 con 0x7f9eac0fd980 2024-02-19T21:05:34.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.687+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9ea0005b40 con 0x7f9eac0fd980 2024-02-19T21:05:34.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.688+0000 7f9eb2366700 1 -- 172.21.15.163:0/3337436897 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e98004f20 con 0x7f9eac0fd980 2024-02-19T21:05:34.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.688+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9ea0002010 con 0x7f9eac0fd980 2024-02-19T21:05:34.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.689+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(153..153 src has 43..153) v4 ==== 9585+0+0 (unknown 3485181117 0 0) 0x7f9ea00793d0 con 0x7f9eac0fd980 2024-02-19T21:05:34.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.689+0000 7f9ea97fa700 10 client.22236.objecter ms_dispatch 0x7f9eac001550 osd_map(153..153 src has 43..153) v4 2024-02-19T21:05:34.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.689+0000 7f9ea97fa700 3 client.22236.objecter handle_osd_map got epochs [153,153] > 0 2024-02-19T21:05:34.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.689+0000 7f9ea97fa700 3 client.22236.objecter handle_osd_map decoding full epoch 153 2024-02-19T21:05:34.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.689+0000 7f9ea97fa700 20 client.22236.objecter dump_active .. 0 homeless 2024-02-19T21:05:34.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.689+0000 7f9ea97fa700 10 client.22236.objecter ms_handle_connect 0x7f9e94063c20 2024-02-19T21:05:34.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.693+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9ea004c1d0 con 0x7f9eac0fd980 2024-02-19T21:05:34.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:33.872+0000 7f9eb2366700 1 -- 172.21.15.163:0/3337436897 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f9e980051d0 con 0x7f9eac0fd980 2024-02-19T21:05:34.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:34.167+0000 7f9ea97fa700 1 -- 172.21.15.163:0/3337436897 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v154) v1 ==== 91+0+0 (unknown 3108069113 0 0) 0x7f9ea004a5e0 con 0x7f9eac0fd980 2024-02-19T21:05:34.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:34.172+0000 7f9eb2366700 10 librados: watch_flush enter 2024-02-19T21:05:34.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:34.173+0000 7f9eb2366700 10 librados: watch_flush exit 2024-02-19T21:05:34.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:34.173+0000 7f9eb2366700 20 client.22236.objecter shutdown clearing up homeless session... 2024-02-19T21:05:34.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:34.173+0000 7f9eb2366700 10 client.22236.objecter successfully canceled tick 2024-02-19T21:05:34.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:34.173+0000 7f9eb2366700 1 -- 172.21.15.163:0/3337436897 >> v1:172.21.15.163:6800/1552514681 conn(0x7f9e94063c20 legacy=0x7f9e940660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:34.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:34.173+0000 7f9eb2366700 1 -- 172.21.15.163:0/3337436897 >> v1:172.21.15.163:6789/0 conn(0x7f9eac0fd980 legacy=0x7f9eac0fc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:34.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:34.178+0000 7f9eb2366700 1 -- 172.21.15.163:0/3337436897 shutdown_connections 2024-02-19T21:05:34.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:34.178+0000 7f9eb2366700 1 -- 172.21.15.163:0/3337436897 >> 172.21.15.163:0/3337436897 conn(0x7f9eac0f35a0 msgr2=0x7f9eac107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:34.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:34.182+0000 7f9eb2366700 1 -- 172.21.15.163:0/3337436897 shutdown_connections 2024-02-19T21:05:34.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:34.194+0000 7f9eb2366700 1 -- 172.21.15.163:0/3337436897 wait complete. 2024-02-19T21:05:34.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:34.198+0000 7f9eb2366700 1 librados: shutdown 2024-02-19T21:05:34.329 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:05:34.563 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:34.570 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:05:34.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:34.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:05:34.674 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:34.702 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:34.728 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:34.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:32.908363+0000 osd.1 (osd.1) 93 : cluster [DBG] 94.0 scrub starts 2024-02-19T21:05:34.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:32.911845+0000 osd.1 (osd.1) 94 : cluster [DBG] 94.0 scrub ok 2024-02-19T21:05:34.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:33.125629+0000 mgr.y (mgr.14550) 204 : cluster [DBG] pgmap v200: 366 pgs: 32 unknown, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 96 KiB/s wr, 100 op/s 2024-02-19T21:05:34.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:33.681221+0000 mon.c (mon.2) 140 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:34.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:33.723791+0000 osd.2 (osd.2) 71 : cluster [DBG] 94.2 scrub starts 2024-02-19T21:05:34.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:33.726044+0000 osd.2 (osd.2) 72 : cluster [DBG] 94.2 scrub ok 2024-02-19T21:05:34.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:33.872584+0000 mon.a (mon.0) 1181 : audit [INF] from='client.? v1:172.21.15.163:0/3337436897' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:05:34.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:34.158230+0000 mon.a (mon.0) 1182 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:05:34.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:34.165255+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? v1:172.21.15.163:0/1800034971' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi163-83854-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:34.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:34.165388+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? v1:172.21.15.163:0/3337436897' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:05:34.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:34.165431+0000 mon.a (mon.0) 1185 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-02-19T21:05:34.732 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:32.908363+0000 osd.1 (osd.1) 93 : cluster [DBG] 94.0 scrub starts 2024-02-19T21:05:34.732 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:32.911845+0000 osd.1 (osd.1) 94 : cluster [DBG] 94.0 scrub ok 2024-02-19T21:05:34.732 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:33.125629+0000 mgr.y (mgr.14550) 204 : cluster [DBG] pgmap v200: 366 pgs: 32 unknown, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 96 KiB/s wr, 100 op/s 2024-02-19T21:05:34.732 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:33.681221+0000 mon.c (mon.2) 140 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:34.732 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:33.723791+0000 osd.2 (osd.2) 71 : cluster [DBG] 94.2 scrub starts 2024-02-19T21:05:34.732 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:33.726044+0000 osd.2 (osd.2) 72 : cluster [DBG] 94.2 scrub ok 2024-02-19T21:05:34.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:33.872584+0000 mon.a (mon.0) 1181 : audit [INF] from='client.? v1:172.21.15.163:0/3337436897' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:05:34.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:34.158230+0000 mon.a (mon.0) 1182 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:05:34.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:34.165255+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? v1:172.21.15.163:0/1800034971' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi163-83854-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:34.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:34.165388+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? v1:172.21.15.163:0/3337436897' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:05:34.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:34.165431+0000 mon.a (mon.0) 1185 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-02-19T21:05:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:34.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:34.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:32.908363+0000 osd.1 (osd.1) 93 : cluster [DBG] 94.0 scrub starts 2024-02-19T21:05:34.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:32.911845+0000 osd.1 (osd.1) 94 : cluster [DBG] 94.0 scrub ok 2024-02-19T21:05:34.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:33.125629+0000 mgr.y (mgr.14550) 204 : cluster [DBG] pgmap v200: 366 pgs: 32 unknown, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s rd, 96 KiB/s wr, 100 op/s 2024-02-19T21:05:34.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:33.681221+0000 mon.c (mon.2) 140 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:34.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:33.723791+0000 osd.2 (osd.2) 71 : cluster [DBG] 94.2 scrub starts 2024-02-19T21:05:34.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:33.726044+0000 osd.2 (osd.2) 72 : cluster [DBG] 94.2 scrub ok 2024-02-19T21:05:34.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:33.872584+0000 mon.a (mon.0) 1181 : audit [INF] from='client.? v1:172.21.15.163:0/3337436897' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:05:34.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:34.158230+0000 mon.a (mon.0) 1182 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:05:34.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:34.165255+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? v1:172.21.15.163:0/1800034971' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi163-83854-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:34.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:34.165388+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? v1:172.21.15.163:0/3337436897' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:05:34.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:34.165431+0000 mon.a (mon.0) 1185 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-02-19T21:05:34.834 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:34.863 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:35.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.099+0000 7f5ccf462700 1 -- 172.21.15.163:0/2960512534 >> v1:172.21.15.163:6789/0 conn(0x7f5cc80ff680 legacy=0x7f5cc8101a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:35.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.099+0000 7f5ccf462700 1 -- 172.21.15.163:0/2960512534 shutdown_connections 2024-02-19T21:05:35.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.099+0000 7f5ccf462700 1 -- 172.21.15.163:0/2960512534 >> 172.21.15.163:0/2960512534 conn(0x7f5cc80f24a0 msgr2=0x7f5cc80f48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:35.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.099+0000 7f5ccf462700 1 -- 172.21.15.163:0/2960512534 shutdown_connections 2024-02-19T21:05:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.099+0000 7f5ccf462700 1 -- 172.21.15.163:0/2960512534 wait complete. 2024-02-19T21:05:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.099+0000 7f5ccf462700 1 librados: starting msgr at 2024-02-19T21:05:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.099+0000 7f5ccf462700 1 librados: starting objecter 2024-02-19T21:05:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.099+0000 7f5ccf462700 1 Processor -- start 2024-02-19T21:05:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.099+0000 7f5ccf462700 1 -- start start 2024-02-19T21:05:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.099+0000 7f5ccf462700 1 librados: setting wanted keys 2024-02-19T21:05:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.099+0000 7f5ccf462700 1 librados: calling monclient init 2024-02-19T21:05:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5ccf462700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5cc80618e0 con 0x7f5cc80ff680 2024-02-19T21:05:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5ccf462700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5cc8118690 con 0x7f5cc80fc880 2024-02-19T21:05:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5ccf462700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5cc8119830 con 0x7f5cc81031c0 2024-02-19T21:05:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5ccd9ff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f5cc81031c0 0x7f5cc8115f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:59542/0 (socket says 172.21.15.163:59542) 2024-02-19T21:05:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5ccd9ff700 1 -- 172.21.15.163:0/3397044972 learned_addr learned my addr 172.21.15.163:0/3397044972 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5ccc9fd700 1 --1- 172.21.15.163:0/3397044972 >> v1:172.21.15.163:6789/0 conn(0x7f5cc80ff680 0x7f5cc8112830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:36992/0 (socket says 172.21.15.163:36992) 2024-02-19T21:05:35.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 10 client.?.objecter ms_handle_connect 0x7f5cc81031c0 2024-02-19T21:05:35.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:35.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 10 client.?.objecter ms_handle_connect 0x7f5cc80ff680 2024-02-19T21:05:35.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:35.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 10 client.?.objecter ms_handle_connect 0x7f5cc80fc880 2024-02-19T21:05:35.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:35.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072293082 0 0) 0x7f5cc8118690 con 0x7f5cc80fc880 2024-02-19T21:05:35.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cb40038e0 con 0x7f5cc80fc880 2024-02-19T21:05:35.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989280244 0 0) 0x7f5cc8119830 con 0x7f5cc81031c0 2024-02-19T21:05:35.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cc8118690 con 0x7f5cc81031c0 2024-02-19T21:05:35.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517789500 0 0) 0x7f5cc80618e0 con 0x7f5cc80ff680 2024-02-19T21:05:35.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cc8119830 con 0x7f5cc80ff680 2024-02-19T21:05:35.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 110086397 0 0) 0x7f5cb40038e0 con 0x7f5cc80fc880 2024-02-19T21:05:35.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cc80618e0 con 0x7f5cc80fc880 2024-02-19T21:05:35.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5cc4001800 con 0x7f5cc80fc880 2024-02-19T21:05:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3108269841 0 0) 0x7f5cc8119830 con 0x7f5cc80ff680 2024-02-19T21:05:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cb40038e0 con 0x7f5cc80ff680 2024-02-19T21:05:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 572261144 0 0) 0x7f5cc8118690 con 0x7f5cc81031c0 2024-02-19T21:05:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cc8119830 con 0x7f5cc81031c0 2024-02-19T21:05:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5cb80015d0 con 0x7f5cc80ff680 2024-02-19T21:05:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5cc0001c50 con 0x7f5cc81031c0 2024-02-19T21:05:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.100+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 551551160 0 0) 0x7f5cc80618e0 con 0x7f5cc80fc880 2024-02-19T21:05:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.101+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 >> v1:172.21.15.163:6790/0 conn(0x7f5cc81031c0 legacy=0x7f5cc8115f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.101+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 >> v1:172.21.15.163:6789/0 conn(0x7f5cc80ff680 legacy=0x7f5cc8112830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.101+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc811a9d0 con 0x7f5cc80fc880 2024-02-19T21:05:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.101+0000 7f5ccf462700 1 -- 172.21.15.163:0/3397044972 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cc81176e0 con 0x7f5cc80fc880 2024-02-19T21:05:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.101+0000 7f5ccf462700 10 client.47668.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.101+0000 7f5ccf462700 1 -- 172.21.15.163:0/3397044972 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cc8117b50 con 0x7f5cc80fc880 2024-02-19T21:05:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.101+0000 7f5ccf462700 1 librados: init done 2024-02-19T21:05:35.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.101+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f5cc4001ec0 con 0x7f5cc80fc880 2024-02-19T21:05:35.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.101+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5cc40032a0 con 0x7f5cc80fc880 2024-02-19T21:05:35.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.102+0000 7f5ccf462700 1 -- 172.21.15.163:0/3397044972 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cac004f20 con 0x7f5cc80fc880 2024-02-19T21:05:35.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.102+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f5cc4001a20 con 0x7f5cc80fc880 2024-02-19T21:05:35.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.102+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(154..154 src has 43..154) v4 ==== 9209+0+0 (unknown 695253494 0 0) 0x7f5cc4076910 con 0x7f5cc80fc880 2024-02-19T21:05:35.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.102+0000 7f5cbe7fc700 10 client.47668.objecter ms_dispatch 0x7f5cc8001550 osd_map(154..154 src has 43..154) v4 2024-02-19T21:05:35.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.102+0000 7f5cbe7fc700 3 client.47668.objecter handle_osd_map got epochs [154,154] > 0 2024-02-19T21:05:35.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.102+0000 7f5cbe7fc700 3 client.47668.objecter handle_osd_map decoding full epoch 154 2024-02-19T21:05:35.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.102+0000 7f5cbe7fc700 20 client.47668.objecter dump_active .. 0 homeless 2024-02-19T21:05:35.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.102+0000 7f5cbe7fc700 10 client.47668.objecter ms_handle_connect 0x7f5cb4063e80 2024-02-19T21:05:35.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.107+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5cc4049880 con 0x7f5cc80fc880 2024-02-19T21:05:35.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.305+0000 7f5ccf462700 1 -- 172.21.15.163:0/3397044972 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5cac000b90 con 0x7f5cb4063e80 2024-02-19T21:05:35.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.328+0000 7f5cbe7fc700 1 -- 172.21.15.163:0/3397044972 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+896240 (unknown 2965378022 0 3569305059) 0x7f5cac000b90 con 0x7f5cb4063e80 2024-02-19T21:05:35.333 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:05:35.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.333+0000 7f5ccf462700 10 librados: watch_flush enter 2024-02-19T21:05:35.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.333+0000 7f5ccf462700 10 librados: watch_flush exit 2024-02-19T21:05:35.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.333+0000 7f5ccf462700 20 client.47668.objecter shutdown clearing up homeless session... 2024-02-19T21:05:35.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.333+0000 7f5ccf462700 10 client.47668.objecter successfully canceled tick 2024-02-19T21:05:35.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.333+0000 7f5ccf462700 1 -- 172.21.15.163:0/3397044972 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5cb4063e80 legacy=0x7f5cb4066340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:35.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.333+0000 7f5ccf462700 1 -- 172.21.15.163:0/3397044972 >> v1:172.21.15.193:6789/0 conn(0x7f5cc80fc880 legacy=0x7f5cc8060ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:35.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.333+0000 7f5ccf462700 1 -- 172.21.15.163:0/3397044972 shutdown_connections 2024-02-19T21:05:35.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.333+0000 7f5ccf462700 1 -- 172.21.15.163:0/3397044972 >> 172.21.15.163:0/3397044972 conn(0x7f5cc80f24a0 msgr2=0x7f5cc8106630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:35.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.333+0000 7f5ccf462700 1 -- 172.21.15.163:0/3397044972 shutdown_connections 2024-02-19T21:05:35.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.333+0000 7f5ccf462700 1 -- 172.21.15.163:0/3397044972 wait complete. 2024-02-19T21:05:35.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:35.333+0000 7f5ccf462700 1 librados: shutdown 2024-02-19T21:05:35.345 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:05:35.513 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:35.516 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:33.924474+0000 osd.6 (osd.6) 71 : cluster [DBG] 94.3 scrub starts 2024-02-19T21:05:35.516 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:33.926448+0000 osd.6 (osd.6) 72 : cluster [DBG] 94.3 scrub ok 2024-02-19T21:05:35.516 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:34.682360+0000 mon.c (mon.2) 141 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:35.516 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:35.167417+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-02-19T21:05:35.516 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:35.188875+0000 mon.c (mon.2) 142 : audit [INF] from='client.? v1:172.21.15.163:0/2565608143' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi163-83845-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:35.516 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:35.190144+0000 mon.a (mon.0) 1187 : audit [INF] from='client.48866 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi163-83845-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:35.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:35.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:35.618 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:35.649 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:35.652 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:33.924474+0000 osd.6 (osd.6) 71 : cluster [DBG] 94.3 scrub starts 2024-02-19T21:05:35.652 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:33.926448+0000 osd.6 (osd.6) 72 : cluster [DBG] 94.3 scrub ok 2024-02-19T21:05:35.652 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:34.682360+0000 mon.c (mon.2) 141 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:35.652 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:35.167417+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-02-19T21:05:35.652 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:35.188875+0000 mon.c (mon.2) 142 : audit [INF] from='client.? v1:172.21.15.163:0/2565608143' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi163-83845-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:35.652 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:35.190144+0000 mon.a (mon.0) 1187 : audit [INF] from='client.48866 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi163-83845-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:35.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:33.924474+0000 osd.6 (osd.6) 71 : cluster [DBG] 94.3 scrub starts 2024-02-19T21:05:35.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:33.926448+0000 osd.6 (osd.6) 72 : cluster [DBG] 94.3 scrub ok 2024-02-19T21:05:35.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:34.682360+0000 mon.c (mon.2) 141 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:35.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:35.167417+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-02-19T21:05:35.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:35.188875+0000 mon.c (mon.2) 142 : audit [INF] from='client.? v1:172.21.15.163:0/2565608143' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi163-83845-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:35.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:35.190144+0000 mon.a (mon.0) 1187 : audit [INF] from='client.48866 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi163-83845-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:35.748 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:35.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:35.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:35.764 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:05:35.764 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:05:35.764 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-02-19T21:05:35.764 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-02-19T21:05:35.764 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:05:35.851 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:35.879 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:36.078 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:36.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:36.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:36.182 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:36.211 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:36.640 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-19T21:05:36.660 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:36.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:36.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:36.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:34.860433+0000 osd.0 (osd.0) 87 : cluster [DBG] 94.5 scrub starts 2024-02-19T21:05:36.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:34.862570+0000 osd.0 (osd.0) 88 : cluster [DBG] 94.5 scrub ok 2024-02-19T21:05:36.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:34.885458+0000 osd.5 (osd.5) 82 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:05:36.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:34.888939+0000 osd.5 (osd.5) 83 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:05:36.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:35.126840+0000 mgr.y (mgr.14550) 205 : cluster [DBG] pgmap v203: 366 pgs: 366 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 255 B/s wr, 77 op/s 2024-02-19T21:05:36.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:35.305838+0000 mgr.y (mgr.14550) 206 : audit [DBG] from='client.47668 v1:172.21.15.163:0/3397044972' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:05:36.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:35.683081+0000 mon.c (mon.2) 143 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:36.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:35.683701+0000 mon.c (mon.2) 144 : audit [INF] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]: dispatch 2024-02-19T21:05:36.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:35.683979+0000 mon.a (mon.0) 1188 : audit [INF] from='client.34832 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]: dispatch 2024-02-19T21:05:36.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:35.760101+0000 osd.2 (osd.2) 73 : cluster [DBG] 94.6 scrub starts 2024-02-19T21:05:36.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:35.761282+0000 osd.2 (osd.2) 74 : cluster [DBG] 94.6 scrub ok 2024-02-19T21:05:36.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:36.167741+0000 mon.a (mon.0) 1189 : audit [INF] from='client.48866 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi163-83845-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:36.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:36.167850+0000 mon.a (mon.0) 1190 : audit [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]': finished 2024-02-19T21:05:36.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:36.167911+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-02-19T21:05:36.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:36.171396+0000 mon.a (mon.0) 1192 : audit [INF] from='client.? v1:172.21.15.163:0/2521895994' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi163-83854-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:36.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:36.171623+0000 mon.c (mon.2) 145 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:36.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:36.172452+0000 mon.c (mon.2) 146 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:36.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:36.173508+0000 mon.c (mon.2) 147 : audit [INF] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-19T21:05:36.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:36.173913+0000 mon.a (mon.0) 1193 : audit [INF] from='client.34832 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-19T21:05:36.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:34.860433+0000 osd.0 (osd.0) 87 : cluster [DBG] 94.5 scrub starts 2024-02-19T21:05:36.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:34.862570+0000 osd.0 (osd.0) 88 : cluster [DBG] 94.5 scrub ok 2024-02-19T21:05:36.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:34.885458+0000 osd.5 (osd.5) 82 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:05:36.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:34.888939+0000 osd.5 (osd.5) 83 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:05:36.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:35.126840+0000 mgr.y (mgr.14550) 205 : cluster [DBG] pgmap v203: 366 pgs: 366 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 255 B/s wr, 77 op/s 2024-02-19T21:05:36.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:35.305838+0000 mgr.y (mgr.14550) 206 : audit [DBG] from='client.47668 v1:172.21.15.163:0/3397044972' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:05:36.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:35.683081+0000 mon.c (mon.2) 143 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:36.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:35.683701+0000 mon.c (mon.2) 144 : audit [INF] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]: dispatch 2024-02-19T21:05:36.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:35.683979+0000 mon.a (mon.0) 1188 : audit [INF] from='client.34832 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]: dispatch 2024-02-19T21:05:36.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:35.760101+0000 osd.2 (osd.2) 73 : cluster [DBG] 94.6 scrub starts 2024-02-19T21:05:36.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:35.761282+0000 osd.2 (osd.2) 74 : cluster [DBG] 94.6 scrub ok 2024-02-19T21:05:36.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:36.167741+0000 mon.a (mon.0) 1189 : audit [INF] from='client.48866 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi163-83845-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:36.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:36.167850+0000 mon.a (mon.0) 1190 : audit [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]': finished 2024-02-19T21:05:36.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:36.167911+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-02-19T21:05:36.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:36.171396+0000 mon.a (mon.0) 1192 : audit [INF] from='client.? v1:172.21.15.163:0/2521895994' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi163-83854-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:36.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:36.171623+0000 mon.c (mon.2) 145 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:36.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:36.172452+0000 mon.c (mon.2) 146 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:36.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:36.173508+0000 mon.c (mon.2) 147 : audit [INF] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-19T21:05:36.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:36.173913+0000 mon.a (mon.0) 1193 : audit [INF] from='client.34832 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-19T21:05:36.764 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:36.788 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:34.860433+0000 osd.0 (osd.0) 87 : cluster [DBG] 94.5 scrub starts 2024-02-19T21:05:36.788 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:34.862570+0000 osd.0 (osd.0) 88 : cluster [DBG] 94.5 scrub ok 2024-02-19T21:05:36.788 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:34.885458+0000 osd.5 (osd.5) 82 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:05:36.788 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:34.888939+0000 osd.5 (osd.5) 83 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:05:36.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:35.126840+0000 mgr.y (mgr.14550) 205 : cluster [DBG] pgmap v203: 366 pgs: 366 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 255 B/s wr, 77 op/s 2024-02-19T21:05:36.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:35.305838+0000 mgr.y (mgr.14550) 206 : audit [DBG] from='client.47668 v1:172.21.15.163:0/3397044972' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:05:36.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:35.683081+0000 mon.c (mon.2) 143 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:36.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:35.683701+0000 mon.c (mon.2) 144 : audit [INF] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]: dispatch 2024-02-19T21:05:36.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:35.683979+0000 mon.a (mon.0) 1188 : audit [INF] from='client.34832 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]: dispatch 2024-02-19T21:05:36.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:35.760101+0000 osd.2 (osd.2) 73 : cluster [DBG] 94.6 scrub starts 2024-02-19T21:05:36.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:35.761282+0000 osd.2 (osd.2) 74 : cluster [DBG] 94.6 scrub ok 2024-02-19T21:05:36.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:36.167741+0000 mon.a (mon.0) 1189 : audit [INF] from='client.48866 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi163-83845-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:36.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:36.167850+0000 mon.a (mon.0) 1190 : audit [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]': finished 2024-02-19T21:05:36.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:36.167911+0000 mon.a (mon.0) 1191 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-02-19T21:05:36.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:36.171396+0000 mon.a (mon.0) 1192 : audit [INF] from='client.? v1:172.21.15.163:0/2521895994' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi163-83854-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:36.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:36.171623+0000 mon.c (mon.2) 145 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:36.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:36.172452+0000 mon.c (mon.2) 146 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:36.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:36.173508+0000 mon.c (mon.2) 147 : audit [INF] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-19T21:05:36.790 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:36.173913+0000 mon.a (mon.0) 1193 : audit [INF] from='client.34832 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-19T21:05:36.790 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:36.808 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:36.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:36.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:36.912 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:36.930 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:05:36.941 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:37.140 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:37.142 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:37.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:37.243 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:37.272 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:37.294 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:37.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:37.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:37.398 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:37.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.424+0000 7f1271ca1700 1 -- 172.21.15.163:0/1412992273 >> v1:172.21.15.163:6790/0 conn(0x7f126c1042b0 legacy=0x7f126c106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:37.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.425+0000 7f1271ca1700 1 -- 172.21.15.163:0/1412992273 shutdown_connections 2024-02-19T21:05:37.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.425+0000 7f1271ca1700 1 -- 172.21.15.163:0/1412992273 >> 172.21.15.163:0/1412992273 conn(0x7f126c0f2470 msgr2=0x7f126c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:37.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.425+0000 7f1271ca1700 1 -- 172.21.15.163:0/1412992273 shutdown_connections 2024-02-19T21:05:37.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.425+0000 7f1271ca1700 1 -- 172.21.15.163:0/1412992273 wait complete. 2024-02-19T21:05:37.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.425+0000 7f1271ca1700 1 librados: starting msgr at 2024-02-19T21:05:37.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.425+0000 7f1271ca1700 1 librados: starting objecter 2024-02-19T21:05:37.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.425+0000 7f1271ca1700 1 Processor -- start 2024-02-19T21:05:37.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.425+0000 7f1271ca1700 1 -- start start 2024-02-19T21:05:37.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.425+0000 7f1271ca1700 1 librados: setting wanted keys 2024-02-19T21:05:37.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.425+0000 7f1271ca1700 1 librados: calling monclient init 2024-02-19T21:05:37.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1271ca1700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f126c100440 con 0x7f126c0f8920 2024-02-19T21:05:37.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1271ca1700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f126c1187e0 con 0x7f126c1042b0 2024-02-19T21:05:37.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1271ca1700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f126c1199c0 con 0x7f126c0f94d0 2024-02-19T21:05:37.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f126affd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f126c0f94d0 0x7f126c112980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:59586/0 (socket says 172.21.15.163:59586) 2024-02-19T21:05:37.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f126affd700 1 -- 172.21.15.163:0/2551310881 learned_addr learned my addr 172.21.15.163:0/2551310881 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:37.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f126b7fe700 1 --1- 172.21.15.163:0/2551310881 >> v1:172.21.15.163:6789/0 conn(0x7f126c0f8920 0x7f126c0ff7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:37048/0 (socket says 172.21.15.163:37048) 2024-02-19T21:05:37.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 10 client.?.objecter ms_handle_connect 0x7f126c0f94d0 2024-02-19T21:05:37.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:37.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 10 client.?.objecter ms_handle_connect 0x7f126c0f8920 2024-02-19T21:05:37.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:37.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 10 client.?.objecter ms_handle_connect 0x7f126c1042b0 2024-02-19T21:05:37.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:37.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730201310 0 0) 0x7f126c1199c0 con 0x7f126c0f94d0 2024-02-19T21:05:37.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1254003630 con 0x7f126c0f94d0 2024-02-19T21:05:37.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206746192 0 0) 0x7f126c1187e0 con 0x7f126c1042b0 2024-02-19T21:05:37.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f126c1199c0 con 0x7f126c1042b0 2024-02-19T21:05:37.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 535810707 0 0) 0x7f1254003630 con 0x7f126c0f94d0 2024-02-19T21:05:37.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f126c1187e0 con 0x7f126c0f94d0 2024-02-19T21:05:37.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1260001850 con 0x7f126c0f94d0 2024-02-19T21:05:37.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 551787262 0 0) 0x7f126c1199c0 con 0x7f126c1042b0 2024-02-19T21:05:37.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1254003630 con 0x7f126c1042b0 2024-02-19T21:05:37.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1264003280 con 0x7f126c1042b0 2024-02-19T21:05:37.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 849438483 0 0) 0x7f126c1187e0 con 0x7f126c0f94d0 2024-02-19T21:05:37.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 >> v1:172.21.15.193:6789/0 conn(0x7f126c1042b0 legacy=0x7f126c1160b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:37.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 >> v1:172.21.15.163:6789/0 conn(0x7f126c0f8920 legacy=0x7f126c0ff7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:37.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126c11abe0 con 0x7f126c0f94d0 2024-02-19T21:05:37.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.426+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126c117830 con 0x7f126c0f94d0 2024-02-19T21:05:37.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.427+0000 7f1271ca1700 10 client.48887.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:37.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.427+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f126c117ce0 con 0x7f126c0f94d0 2024-02-19T21:05:37.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.427+0000 7f1271ca1700 1 librados: init done 2024-02-19T21:05:37.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.427+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f126c05f700 con 0x7f126c0f94d0 2024-02-19T21:05:37.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.427+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f1260002340 con 0x7f126c0f94d0 2024-02-19T21:05:37.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.427+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1260005a20 con 0x7f126c0f94d0 2024-02-19T21:05:37.425 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:37.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.428+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f12600188d0 con 0x7f126c0f94d0 2024-02-19T21:05:37.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.428+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(157..157 src has 43..157) v4 ==== 9214+0+0 (unknown 3392113750 0 0) 0x7f1260079100 con 0x7f126c0f94d0 2024-02-19T21:05:37.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.428+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 osd_map(157..157 src has 43..157) v4 2024-02-19T21:05:37.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.428+0000 7f1268ff9700 3 client.48887.objecter handle_osd_map got epochs [157,157] > 0 2024-02-19T21:05:37.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.428+0000 7f1268ff9700 3 client.48887.objecter handle_osd_map decoding full epoch 157 2024-02-19T21:05:37.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.429+0000 7f1268ff9700 20 client.48887.objecter dump_active .. 0 homeless 2024-02-19T21:05:37.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.429+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v157) v1 ==== 54+0+16 (unknown 3314946750 0 2420000192) 0x7f12600794e0 con 0x7f126c0f94d0 2024-02-19T21:05:37.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.429+0000 7f1268ff9700 10 client.48887.objecter ms_handle_connect 0x7f1254063b70 2024-02-19T21:05:37.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.429+0000 7f1271ca1700 10 client.48887.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:37.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.429+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 1 2024-02-19T21:05:37.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.429+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c047540 osd=0 3 2024-02-19T21:05:37.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.429+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c047540 osd=0 3 2024-02-19T21:05:37.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.429+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c047540 osd=0 3 2024-02-19T21:05:37.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.429+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 1 2024-02-19T21:05:37.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.429+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c047540 osd=0 3 2024-02-19T21:05:37.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.429+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 0 1 2024-02-19T21:05:37.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.429+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c047540 osd=0 4 2024-02-19T21:05:37.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.429+0000 7f1271ca1700 10 client.48887.objecter _send_command 1 2024-02-19T21:05:37.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.429+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f126c061360 con 0x7f126c118970 2024-02-19T21:05:37.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.429+0000 7f1268ff9700 10 client.48887.objecter ms_handle_connect 0x7f126c118970 2024-02-19T21:05:37.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.430+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f126c061360 con 0x7f126c118970 2024-02-19T21:05:37.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.430+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:05:37.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.430+0000 7f1268ff9700 10 client.48887.objecter _finish_command 1 = system:0 2024-02-19T21:05:37.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.430+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c047540 osd=0 4 2024-02-19T21:05:37.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.430+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 0 1 2024-02-19T21:05:37.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.444+0000 7f1271ca1700 10 client.48887.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:37.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.444+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 2 2024-02-19T21:05:37.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.444+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c047540 osd=0 3 2024-02-19T21:05:37.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.444+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c047540 osd=0 3 2024-02-19T21:05:37.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.444+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c047540 osd=0 3 2024-02-19T21:05:37.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.444+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 2 2024-02-19T21:05:37.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.444+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c047540 osd=0 3 2024-02-19T21:05:37.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.444+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 0 2 2024-02-19T21:05:37.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.444+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c047540 osd=0 4 2024-02-19T21:05:37.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.444+0000 7f1271ca1700 10 client.48887.objecter _send_command 2 2024-02-19T21:05:37.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.444+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f126c109fd0 con 0x7f126c118970 2024-02-19T21:05:37.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f126c109fd0 con 0x7f126c118970 2024-02-19T21:05:37.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:37.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1268ff9700 10 client.48887.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c047540 osd=0 4 2024-02-19T21:05:37.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 0 2 2024-02-19T21:05:37.441 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:05:37.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1271ca1700 10 client.48887.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:37.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 3 2024-02-19T21:05:37.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c0f9930 osd=1 3 2024-02-19T21:05:37.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c0f9930 osd=1 3 2024-02-19T21:05:37.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c0f9930 osd=1 3 2024-02-19T21:05:37.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 3 2024-02-19T21:05:37.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c0f9930 osd=1 3 2024-02-19T21:05:37.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 1 3 2024-02-19T21:05:37.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c0f9930 osd=1 4 2024-02-19T21:05:37.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1271ca1700 10 client.48887.objecter _send_command 3 2024-02-19T21:05:37.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.446+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f126c0fa3f0 con 0x7f126c0f9fb0 2024-02-19T21:05:37.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.447+0000 7f1268ff9700 10 client.48887.objecter ms_handle_connect 0x7f126c0f9fb0 2024-02-19T21:05:37.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.447+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f126c0fa3f0 con 0x7f126c0f9fb0 2024-02-19T21:05:37.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.447+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:05:37.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.447+0000 7f1268ff9700 10 client.48887.objecter _finish_command 3 = system:0 2024-02-19T21:05:37.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.447+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c0f9930 osd=1 4 2024-02-19T21:05:37.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.447+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 1 3 2024-02-19T21:05:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.462+0000 7f1271ca1700 10 client.48887.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.462+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 4 2024-02-19T21:05:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.462+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c0f9930 osd=1 3 2024-02-19T21:05:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.462+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c0f9930 osd=1 3 2024-02-19T21:05:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.462+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c0f9930 osd=1 3 2024-02-19T21:05:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.462+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 4 2024-02-19T21:05:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.462+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c0f9930 osd=1 3 2024-02-19T21:05:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.462+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 1 4 2024-02-19T21:05:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.462+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c0f9930 osd=1 4 2024-02-19T21:05:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.462+0000 7f1271ca1700 10 client.48887.objecter _send_command 4 2024-02-19T21:05:37.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.462+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f126c0fa7d0 con 0x7f126c0f9fb0 2024-02-19T21:05:37.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f126c0fa7d0 con 0x7f126c0f9fb0 2024-02-19T21:05:37.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:37.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1268ff9700 10 client.48887.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c0f9930 osd=1 4 2024-02-19T21:05:37.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 1 4 2024-02-19T21:05:37.459 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:05:37.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1271ca1700 10 client.48887.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 5 2024-02-19T21:05:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c118210 osd=2 3 2024-02-19T21:05:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c118210 osd=2 3 2024-02-19T21:05:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c118210 osd=2 3 2024-02-19T21:05:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 5 2024-02-19T21:05:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c118210 osd=2 3 2024-02-19T21:05:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 2 5 2024-02-19T21:05:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c118210 osd=2 4 2024-02-19T21:05:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1271ca1700 10 client.48887.objecter _send_command 5 2024-02-19T21:05:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f126c0287b0 con 0x7f126c118380 2024-02-19T21:05:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.464+0000 7f1268ff9700 10 client.48887.objecter ms_handle_connect 0x7f126c118380 2024-02-19T21:05:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.465+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f126c0287b0 con 0x7f126c118380 2024-02-19T21:05:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.465+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:05:37.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.465+0000 7f1268ff9700 10 client.48887.objecter _finish_command 5 = system:0 2024-02-19T21:05:37.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.465+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c118210 osd=2 4 2024-02-19T21:05:37.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.465+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 2 5 2024-02-19T21:05:37.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.480+0000 7f1271ca1700 10 client.48887.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:37.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.480+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 6 2024-02-19T21:05:37.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.480+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c118210 osd=2 3 2024-02-19T21:05:37.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.480+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c118210 osd=2 3 2024-02-19T21:05:37.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.480+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c118210 osd=2 3 2024-02-19T21:05:37.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.480+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 6 2024-02-19T21:05:37.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.480+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c118210 osd=2 3 2024-02-19T21:05:37.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.480+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 2 6 2024-02-19T21:05:37.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.480+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c118210 osd=2 4 2024-02-19T21:05:37.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.480+0000 7f1271ca1700 10 client.48887.objecter _send_command 6 2024-02-19T21:05:37.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.480+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f126c027d10 con 0x7f126c118380 2024-02-19T21:05:37.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f126c027d10 con 0x7f126c118380 2024-02-19T21:05:37.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:37.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1268ff9700 10 client.48887.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c118210 osd=2 4 2024-02-19T21:05:37.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 2 6 2024-02-19T21:05:37.476 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:05:37.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1271ca1700 10 client.48887.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:37.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 7 2024-02-19T21:05:37.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c0280c0 osd=3 3 2024-02-19T21:05:37.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c0280c0 osd=3 3 2024-02-19T21:05:37.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c0280c0 osd=3 3 2024-02-19T21:05:37.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 7 2024-02-19T21:05:37.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c0280c0 osd=3 3 2024-02-19T21:05:37.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 3 7 2024-02-19T21:05:37.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c0280c0 osd=3 4 2024-02-19T21:05:37.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1271ca1700 10 client.48887.objecter _send_command 7 2024-02-19T21:05:37.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.481+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f126c127e40 con 0x7f126c059e00 2024-02-19T21:05:37.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.482+0000 7f1268ff9700 10 client.48887.objecter ms_handle_connect 0x7f126c059e00 2024-02-19T21:05:37.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.482+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f126c127e40 con 0x7f126c059e00 2024-02-19T21:05:37.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.482+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:05:37.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.482+0000 7f1268ff9700 10 client.48887.objecter _finish_command 7 = system:0 2024-02-19T21:05:37.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.482+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c0280c0 osd=3 4 2024-02-19T21:05:37.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.482+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 3 7 2024-02-19T21:05:37.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:35.864456+0000 osd.0 (osd.0) 89 : cluster [DBG] 94.8 scrub starts 2024-02-19T21:05:37.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:35.867349+0000 osd.0 (osd.0) 90 : cluster [DBG] 94.8 scrub ok 2024-02-19T21:05:37.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:35.950702+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:05:37.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:35.951588+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:05:37.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:36.349419+0000 osd.7 (osd.7) 87 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:05:37.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:36.352576+0000 osd.7 (osd.7) 88 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:05:37.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:36.525749+0000 osd.4 (osd.4) 79 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:05:37.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:36.527434+0000 osd.4 (osd.4) 80 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:05:37.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:36.919184+0000 osd.3 (osd.3) 95 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:05:37.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:36.920111+0000 osd.3 (osd.3) 96 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:05:37.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:37.128038+0000 mon.b (mon.1) 249 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-19T21:05:37.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:37.129143+0000 mon.a (mon.0) 1194 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-19T21:05:37.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:37.169954+0000 mon.a (mon.0) 1195 : audit [INF] from='client.? v1:172.21.15.163:0/2521895994' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi163-83854-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:37.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:37.170046+0000 mon.a (mon.0) 1196 : audit [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]': finished 2024-02-19T21:05:37.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:37.170234+0000 mon.a (mon.0) 1197 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-19T21:05:37.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:37.170342+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-02-19T21:05:37.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:37.179649+0000 mon.c (mon.2) 148 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:37.483 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:37.429350+0000 mon.c (mon.2) 149 : audit [DBG] from='client.? v1:172.21.15.163:0/2551310881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:37.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.498+0000 7f1271ca1700 10 client.48887.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:37.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.498+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 8 2024-02-19T21:05:37.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.498+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c0280c0 osd=3 3 2024-02-19T21:05:37.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.498+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c0280c0 osd=3 3 2024-02-19T21:05:37.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.498+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c0280c0 osd=3 3 2024-02-19T21:05:37.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.498+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 8 2024-02-19T21:05:37.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.498+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c0280c0 osd=3 3 2024-02-19T21:05:37.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.498+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 3 8 2024-02-19T21:05:37.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.498+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c0280c0 osd=3 4 2024-02-19T21:05:37.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.498+0000 7f1271ca1700 10 client.48887.objecter _send_command 8 2024-02-19T21:05:37.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.498+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f126c059360 con 0x7f126c059e00 2024-02-19T21:05:37.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f126c059360 con 0x7f126c059e00 2024-02-19T21:05:37.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:37.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1268ff9700 10 client.48887.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c0280c0 osd=3 4 2024-02-19T21:05:37.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 3 8 2024-02-19T21:05:37.494 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:05:37.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1271ca1700 10 client.48887.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:37.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 9 2024-02-19T21:05:37.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c059360 osd=4 3 2024-02-19T21:05:37.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c059360 osd=4 3 2024-02-19T21:05:37.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c059360 osd=4 3 2024-02-19T21:05:37.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 9 2024-02-19T21:05:37.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c059360 osd=4 3 2024-02-19T21:05:37.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 4 9 2024-02-19T21:05:37.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c059360 osd=4 4 2024-02-19T21:05:37.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1271ca1700 10 client.48887.objecter _send_command 9 2024-02-19T21:05:37.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f126c124390 con 0x7f126c0594d0 2024-02-19T21:05:37.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.500+0000 7f1268ff9700 10 client.48887.objecter ms_handle_connect 0x7f126c0594d0 2024-02-19T21:05:37.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.501+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f126c124390 con 0x7f126c0594d0 2024-02-19T21:05:37.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.501+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:05:37.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.501+0000 7f1268ff9700 10 client.48887.objecter _finish_command 9 = system:0 2024-02-19T21:05:37.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.501+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c059360 osd=4 4 2024-02-19T21:05:37.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.501+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 4 9 2024-02-19T21:05:37.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.516+0000 7f1271ca1700 10 client.48887.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:37.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.516+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 10 2024-02-19T21:05:37.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.516+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c059360 osd=4 3 2024-02-19T21:05:37.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.516+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c059360 osd=4 3 2024-02-19T21:05:37.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.516+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c059360 osd=4 3 2024-02-19T21:05:37.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.516+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 10 2024-02-19T21:05:37.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.516+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c059360 osd=4 3 2024-02-19T21:05:37.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.516+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 4 10 2024-02-19T21:05:37.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.516+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c059360 osd=4 4 2024-02-19T21:05:37.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.516+0000 7f1271ca1700 10 client.48887.objecter _send_command 10 2024-02-19T21:05:37.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.516+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f126c124390 con 0x7f126c0594d0 2024-02-19T21:05:37.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.518+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f126c124390 con 0x7f126c0594d0 2024-02-19T21:05:37.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.518+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:37.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.518+0000 7f1268ff9700 10 client.48887.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.518+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c059360 osd=4 4 2024-02-19T21:05:37.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.518+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 4 10 2024-02-19T21:05:37.513 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:05:37.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.519+0000 7f1271ca1700 10 client.48887.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:37.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.519+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 11 2024-02-19T21:05:37.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.519+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c124390 osd=5 3 2024-02-19T21:05:37.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.519+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c124390 osd=5 3 2024-02-19T21:05:37.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.519+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c124390 osd=5 3 2024-02-19T21:05:37.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.519+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 11 2024-02-19T21:05:37.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.519+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c124390 osd=5 3 2024-02-19T21:05:37.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.519+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 5 11 2024-02-19T21:05:37.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.519+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c124390 osd=5 4 2024-02-19T21:05:37.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.519+0000 7f1271ca1700 10 client.48887.objecter _send_command 11 2024-02-19T21:05:37.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.519+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f126c130970 con 0x7f126c12cd70 2024-02-19T21:05:37.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.519+0000 7f1268ff9700 10 client.48887.objecter ms_handle_connect 0x7f126c12cd70 2024-02-19T21:05:37.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.520+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f126c130970 con 0x7f126c12cd70 2024-02-19T21:05:37.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.520+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:05:37.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.520+0000 7f1268ff9700 10 client.48887.objecter _finish_command 11 = system:0 2024-02-19T21:05:37.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.520+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c124390 osd=5 4 2024-02-19T21:05:37.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.520+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 5 11 2024-02-19T21:05:37.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.534+0000 7f1271ca1700 10 client.48887.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:37.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.534+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 12 2024-02-19T21:05:37.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.534+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c124390 osd=5 3 2024-02-19T21:05:37.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.534+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c124390 osd=5 3 2024-02-19T21:05:37.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.534+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c124390 osd=5 3 2024-02-19T21:05:37.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.534+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 12 2024-02-19T21:05:37.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.534+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c124390 osd=5 3 2024-02-19T21:05:37.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.534+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 5 12 2024-02-19T21:05:37.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.534+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c124390 osd=5 4 2024-02-19T21:05:37.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.534+0000 7f1271ca1700 10 client.48887.objecter _send_command 12 2024-02-19T21:05:37.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.534+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f126c132120 con 0x7f126c12cd70 2024-02-19T21:05:37.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.535+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f126c132120 con 0x7f126c12cd70 2024-02-19T21:05:37.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.535+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:37.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.535+0000 7f1268ff9700 10 client.48887.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.535+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c124390 osd=5 4 2024-02-19T21:05:37.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.535+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 5 12 2024-02-19T21:05:37.530 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:05:37.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.536+0000 7f1271ca1700 10 client.48887.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:37.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.536+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 13 2024-02-19T21:05:37.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.536+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c132120 osd=6 3 2024-02-19T21:05:37.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.536+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c132120 osd=6 3 2024-02-19T21:05:37.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.536+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c132120 osd=6 3 2024-02-19T21:05:37.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.536+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 13 2024-02-19T21:05:37.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.536+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c132120 osd=6 3 2024-02-19T21:05:37.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.536+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 6 13 2024-02-19T21:05:37.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.536+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c132120 osd=6 4 2024-02-19T21:05:37.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.536+0000 7f1271ca1700 10 client.48887.objecter _send_command 13 2024-02-19T21:05:37.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.536+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f126c132d90 con 0x7f126c1327e0 2024-02-19T21:05:37.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.536+0000 7f1268ff9700 10 client.48887.objecter ms_handle_connect 0x7f126c1327e0 2024-02-19T21:05:37.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.537+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f126c132d90 con 0x7f126c1327e0 2024-02-19T21:05:37.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.537+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:05:37.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.537+0000 7f1268ff9700 10 client.48887.objecter _finish_command 13 = system:0 2024-02-19T21:05:37.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.537+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c132120 osd=6 4 2024-02-19T21:05:37.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.537+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 6 13 2024-02-19T21:05:37.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.551+0000 7f1271ca1700 10 client.48887.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:37.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.551+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 14 2024-02-19T21:05:37.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.551+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c132120 osd=6 3 2024-02-19T21:05:37.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.551+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c132120 osd=6 3 2024-02-19T21:05:37.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.551+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c132120 osd=6 3 2024-02-19T21:05:37.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.551+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 14 2024-02-19T21:05:37.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.551+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c132120 osd=6 3 2024-02-19T21:05:37.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.551+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 6 14 2024-02-19T21:05:37.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.551+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c132120 osd=6 4 2024-02-19T21:05:37.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.551+0000 7f1271ca1700 10 client.48887.objecter _send_command 14 2024-02-19T21:05:37.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.551+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f126c132d90 con 0x7f126c1327e0 2024-02-19T21:05:37.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f126c132d90 con 0x7f126c1327e0 2024-02-19T21:05:37.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:37.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1268ff9700 10 client.48887.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c132120 osd=6 4 2024-02-19T21:05:37.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 6 14 2024-02-19T21:05:37.548 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:05:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1271ca1700 10 client.48887.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 15 2024-02-19T21:05:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c132d90 osd=7 3 2024-02-19T21:05:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c132d90 osd=7 3 2024-02-19T21:05:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c132d90 osd=7 3 2024-02-19T21:05:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 15 2024-02-19T21:05:37.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c132d90 osd=7 3 2024-02-19T21:05:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 7 15 2024-02-19T21:05:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c132d90 osd=7 4 2024-02-19T21:05:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1271ca1700 10 client.48887.objecter _send_command 15 2024-02-19T21:05:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f126c139340 con 0x7f126c135830 2024-02-19T21:05:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.553+0000 7f1268ff9700 10 client.48887.objecter ms_handle_connect 0x7f126c135830 2024-02-19T21:05:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.554+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f126c139340 con 0x7f126c135830 2024-02-19T21:05:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.554+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:05:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.554+0000 7f1268ff9700 10 client.48887.objecter _finish_command 15 = system:0 2024-02-19T21:05:37.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.554+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c132d90 osd=7 4 2024-02-19T21:05:37.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.554+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 7 15 2024-02-19T21:05:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.568+0000 7f1271ca1700 10 client.48887.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.568+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign -1 16 2024-02-19T21:05:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.568+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c132d90 osd=7 3 2024-02-19T21:05:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.568+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c132d90 osd=7 3 2024-02-19T21:05:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.568+0000 7f1271ca1700 20 client.48887.objecter _get_session s=0x7f126c132d90 osd=7 3 2024-02-19T21:05:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.568+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_remove -1 16 2024-02-19T21:05:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.568+0000 7f1271ca1700 20 client.48887.objecter get_session s=0x7f126c132d90 osd=7 3 2024-02-19T21:05:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.568+0000 7f1271ca1700 15 client.48887.objecter _session_command_op_assign 7 16 2024-02-19T21:05:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.568+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c132d90 osd=7 4 2024-02-19T21:05:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.568+0000 7f1271ca1700 10 client.48887.objecter _send_command 16 2024-02-19T21:05:37.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.568+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f126c139340 con 0x7f126c135830 2024-02-19T21:05:37.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1268ff9700 1 -- 172.21.15.163:0/2551310881 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f126c139340 con 0x7f126c135830 2024-02-19T21:05:37.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1268ff9700 10 client.48887.objecter ms_dispatch 0x7f126c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:37.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1268ff9700 10 client.48887.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1268ff9700 20 client.48887.objecter put_session s=0x7f126c132d90 osd=7 4 2024-02-19T21:05:37.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1268ff9700 15 client.48887.objecter _session_command_op_remove 7 16 2024-02-19T21:05:37.565 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:05:37.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 10 librados: watch_flush enter 2024-02-19T21:05:37.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 10 librados: watch_flush exit 2024-02-19T21:05:37.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 10 client.48887.objecter close_session for osd.0 2024-02-19T21:05:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 >> v1:172.21.15.163:6801/1559418115 conn(0x7f126c118970 legacy=0x7f126c118d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c047540 osd=0 1 2024-02-19T21:05:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 10 client.48887.objecter close_session for osd.1 2024-02-19T21:05:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 >> v1:172.21.15.163:6805/3125047432 conn(0x7f126c0f9fb0 legacy=0x7f126c0273c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c0f9930 osd=1 1 2024-02-19T21:05:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 10 client.48887.objecter close_session for osd.2 2024-02-19T21:05:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 >> v1:172.21.15.163:6809/1563158487 conn(0x7f126c118380 legacy=0x7f126c121e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c118210 osd=2 1 2024-02-19T21:05:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 10 client.48887.objecter close_session for osd.3 2024-02-19T21:05:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 >> v1:172.21.15.163:6813/2112279769 conn(0x7f126c059e00 legacy=0x7f126c126720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c0280c0 osd=3 1 2024-02-19T21:05:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 10 client.48887.objecter close_session for osd.4 2024-02-19T21:05:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f126bfff700 1 -- 172.21.15.163:0/2551310881 reap_dead start 2024-02-19T21:05:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 >> v1:172.21.15.193:6800/3129716965 conn(0x7f126c0594d0 legacy=0x7f126c12b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f126bfff700 1 -- 172.21.15.163:0/2551310881 reap_dead start 2024-02-19T21:05:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c059360 osd=4 1 2024-02-19T21:05:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 10 client.48887.objecter close_session for osd.5 2024-02-19T21:05:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 >> v1:172.21.15.193:6804/805161991 conn(0x7f126c12cd70 legacy=0x7f126c12f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c124390 osd=5 1 2024-02-19T21:05:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 10 client.48887.objecter close_session for osd.6 2024-02-19T21:05:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 >> v1:172.21.15.193:6808/787604615 conn(0x7f126c1327e0 legacy=0x7f126c130970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c132120 osd=6 1 2024-02-19T21:05:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 10 client.48887.objecter close_session for osd.7 2024-02-19T21:05:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 >> v1:172.21.15.193:6812/157927432 conn(0x7f126c135830 legacy=0x7f126c137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 20 client.48887.objecter put_session s=0x7f126c132d90 osd=7 1 2024-02-19T21:05:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 20 client.48887.objecter shutdown clearing up homeless session... 2024-02-19T21:05:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.570+0000 7f1271ca1700 10 client.48887.objecter successfully canceled tick 2024-02-19T21:05:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.571+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 >> v1:172.21.15.163:6800/1552514681 conn(0x7f1254063b70 legacy=0x7f1254066030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.571+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 >> v1:172.21.15.163:6790/0 conn(0x7f126c0f94d0 legacy=0x7f126c112980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.571+0000 7f126bfff700 1 -- 172.21.15.163:0/2551310881 reap_dead start 2024-02-19T21:05:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.571+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 shutdown_connections 2024-02-19T21:05:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.571+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 >> 172.21.15.163:0/2551310881 conn(0x7f126c0f2470 msgr2=0x7f126c107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.571+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 shutdown_connections 2024-02-19T21:05:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.571+0000 7f1271ca1700 1 -- 172.21.15.163:0/2551310881 wait complete. 2024-02-19T21:05:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:37.571+0000 7f1271ca1700 1 librados: shutdown 2024-02-19T21:05:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:05:37.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:35.864456+0000 osd.0 (osd.0) 89 : cluster [DBG] 94.8 scrub starts 2024-02-19T21:05:37.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:35.867349+0000 osd.0 (osd.0) 90 : cluster [DBG] 94.8 scrub ok 2024-02-19T21:05:37.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:35.950702+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:05:37.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:35.951588+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:05:37.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:36.349419+0000 osd.7 (osd.7) 87 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:05:37.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:36.352576+0000 osd.7 (osd.7) 88 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:05:37.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:36.525749+0000 osd.4 (osd.4) 79 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:05:37.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:36.527434+0000 osd.4 (osd.4) 80 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:05:37.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:36.919184+0000 osd.3 (osd.3) 95 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:05:37.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:36.920111+0000 osd.3 (osd.3) 96 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:05:37.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:37.128038+0000 mon.b (mon.1) 249 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-19T21:05:37.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:37.129143+0000 mon.a (mon.0) 1194 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-19T21:05:37.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:37.169954+0000 mon.a (mon.0) 1195 : audit [INF] from='client.? v1:172.21.15.163:0/2521895994' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi163-83854-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:37.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:37.170046+0000 mon.a (mon.0) 1196 : audit [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]': finished 2024-02-19T21:05:37.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:37.170234+0000 mon.a (mon.0) 1197 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-19T21:05:37.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:37.170342+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-02-19T21:05:37.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:37.179649+0000 mon.c (mon.2) 148 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:37.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:37.429350+0000 mon.c (mon.2) 149 : audit [DBG] from='client.? v1:172.21.15.163:0/2551310881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:37.766 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:35.864456+0000 osd.0 (osd.0) 89 : cluster [DBG] 94.8 scrub starts 2024-02-19T21:05:37.766 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:35.867349+0000 osd.0 (osd.0) 90 : cluster [DBG] 94.8 scrub ok 2024-02-19T21:05:37.766 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:35.950702+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:05:37.766 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:35.951588+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:05:37.766 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:36.349419+0000 osd.7 (osd.7) 87 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:05:37.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:36.352576+0000 osd.7 (osd.7) 88 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:05:37.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:36.525749+0000 osd.4 (osd.4) 79 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:05:37.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:36.527434+0000 osd.4 (osd.4) 80 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:05:37.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:36.919184+0000 osd.3 (osd.3) 95 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:05:37.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:36.920111+0000 osd.3 (osd.3) 96 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:05:37.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:37.128038+0000 mon.b (mon.1) 249 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-19T21:05:37.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:37.129143+0000 mon.a (mon.0) 1194 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-19T21:05:37.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:37.169954+0000 mon.a (mon.0) 1195 : audit [INF] from='client.? v1:172.21.15.163:0/2521895994' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi163-83854-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:37.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:37.170046+0000 mon.a (mon.0) 1196 : audit [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]': finished 2024-02-19T21:05:37.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:37.170234+0000 mon.a (mon.0) 1197 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-19T21:05:37.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:37.170342+0000 mon.a (mon.0) 1198 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-02-19T21:05:37.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:37.179649+0000 mon.c (mon.2) 148 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:37.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:37.429350+0000 mon.c (mon.2) 149 : audit [DBG] from='client.? v1:172.21.15.163:0/2551310881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:37.868 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-02-19T21:05:35.164663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":226516667,"num_objects":973,"num_object_clones":542,"num_object_copies":2919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":752,"num_whiteouts":0,"num_read":2306,"num_read_kb":50116,"num_write":2120,"num_write_kb":228376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1423,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":221,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8400000,"num_omap_keys":3000,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2847,"ondisk_log_size":2847,"up":1002,"acting":1002,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1098,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3127820,"kb_used_data":689092,"kb_used_omap":0,"kb_used_meta":2438656,"kb_avail":746833396,"statfs":{"total":767960285184,"available":764757397504,"internally_reserved":0,"allocated":705630208,"data_stored":688898168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2497183744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2281,"num_objects":221,"num_object_clones":141,"num_object_copies":663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":307,"num_read_kb":227,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":221,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001981"},"pg_stats":[{"pgid":"103.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"103.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:05:35.161837+0000","last_change":"2024-02-19T21:05:35.161837+0000","last_active":"2024-02-19T21:05:35.161837+0000","last_peered":"2024-02-19T21:05:35.161837+0000","last_clean":"2024-02-19T21:05:35.161837+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:05:35.161837+0000","last_undegraded":"2024-02-19T21:05:35.161837+0000","last_fullsized":"2024-02-19T21:05:35.161837+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:35.161837+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.161837+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"94.1","version":"151'11","reported_seq":45,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.175334+0000","last_change":"2024-02-19T21:05:32.226445+0000","last_active":"2024-02-19T21:05:33.175334+0000","last_peered":"2024-02-19T21:05:33.175334+0000","last_clean":"2024-02-19T21:05:33.175334+0000","last_became_active":"2024-02-19T21:05:24.198029+0000","last_became_peered":"2024-02-19T21:05:24.198029+0000","last_unstale":"2024-02-19T21:05:33.175334+0000","last_undegraded":"2024-02-19T21:05:33.175334+0000","last_fullsized":"2024-02-19T21:05:33.175334+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'11","last_scrub_stamp":"2024-02-19T21:05:32.226382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:32.226382+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":112,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.1f","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181689+0000","last_change":"2024-02-19T21:05:23.189401+0000","last_active":"2024-02-19T21:05:33.181689+0000","last_peered":"2024-02-19T21:05:33.181689+0000","last_clean":"2024-02-19T21:05:33.181689+0000","last_became_active":"2024-02-19T21:04:56.829189+0000","last_became_peered":"2024-02-19T21:04:56.829189+0000","last_unstale":"2024-02-19T21:05:33.181689+0000","last_undegraded":"2024-02-19T21:05:33.181689+0000","last_fullsized":"2024-02-19T21:05:33.181689+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"94.0","version":"151'5","reported_seq":33,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.180421+0000","last_change":"2024-02-19T21:05:32.911920+0000","last_active":"2024-02-19T21:05:33.180421+0000","last_peered":"2024-02-19T21:05:33.180421+0000","last_clean":"2024-02-19T21:05:33.180421+0000","last_became_active":"2024-02-19T21:05:24.207336+0000","last_became_peered":"2024-02-19T21:05:24.207336+0000","last_unstale":"2024-02-19T21:05:33.180421+0000","last_undegraded":"2024-02-19T21:05:33.180421+0000","last_fullsized":"2024-02-19T21:05:33.180421+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'5","last_scrub_stamp":"2024-02-19T21:05:32.911849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:32.911849+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1e","version":"134'20","reported_seq":112,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.180347+0000","last_change":"2024-02-19T21:05:23.191569+0000","last_active":"2024-02-19T21:05:33.180347+0000","last_peered":"2024-02-19T21:05:33.180347+0000","last_clean":"2024-02-19T21:05:33.180347+0000","last_became_active":"2024-02-19T21:04:56.815603+0000","last_became_peered":"2024-02-19T21:04:56.815603+0000","last_unstale":"2024-02-19T21:05:33.180347+0000","last_undegraded":"2024-02-19T21:05:33.180347+0000","last_fullsized":"2024-02-19T21:05:33.180347+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1280,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"94.3","version":"151'7","reported_seq":35,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.926622+0000","last_change":"2024-02-19T21:05:33.926576+0000","last_active":"2024-02-19T21:05:33.926622+0000","last_peered":"2024-02-19T21:05:33.926622+0000","last_clean":"2024-02-19T21:05:33.926622+0000","last_became_active":"2024-02-19T21:05:24.188295+0000","last_became_peered":"2024-02-19T21:05:24.188295+0000","last_unstale":"2024-02-19T21:05:33.926622+0000","last_undegraded":"2024-02-19T21:05:33.926622+0000","last_fullsized":"2024-02-19T21:05:33.926622+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'7","last_scrub_stamp":"2024-02-19T21:05:33.926458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:33.926458+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.1d","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.178200+0000","last_change":"2024-02-19T21:05:23.193437+0000","last_active":"2024-02-19T21:05:33.178200+0000","last_peered":"2024-02-19T21:05:33.178200+0000","last_clean":"2024-02-19T21:05:33.178200+0000","last_became_active":"2024-02-19T21:04:57.830281+0000","last_became_peered":"2024-02-19T21:04:57.830281+0000","last_unstale":"2024-02-19T21:05:33.178200+0000","last_undegraded":"2024-02-19T21:05:33.178200+0000","last_fullsized":"2024-02-19T21:05:33.178200+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"94.2","version":"151'13","reported_seq":47,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.726143+0000","last_change":"2024-02-19T21:05:33.726117+0000","last_active":"2024-02-19T21:05:33.726143+0000","last_peered":"2024-02-19T21:05:33.726143+0000","last_clean":"2024-02-19T21:05:33.726143+0000","last_became_active":"2024-02-19T21:05:24.207872+0000","last_became_peered":"2024-02-19T21:05:24.207872+0000","last_unstale":"2024-02-19T21:05:33.726143+0000","last_undegraded":"2024-02-19T21:05:33.726143+0000","last_fullsized":"2024-02-19T21:05:33.726143+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'13","last_scrub_stamp":"2024-02-19T21:05:33.726050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:33.726050+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":131,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":13,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.1c","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.184651+0000","last_change":"2024-02-19T21:05:23.197706+0000","last_active":"2024-02-19T21:05:33.184651+0000","last_peered":"2024-02-19T21:05:33.184651+0000","last_clean":"2024-02-19T21:05:33.184651+0000","last_became_active":"2024-02-19T21:04:56.805020+0000","last_became_peered":"2024-02-19T21:04:56.805020+0000","last_unstale":"2024-02-19T21:05:33.184651+0000","last_undegraded":"2024-02-19T21:05:33.184651+0000","last_fullsized":"2024-02-19T21:05:33.184651+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"94.5","version":"151'12","reported_seq":46,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-02-19T21:05:34.862640+0000","last_change":"2024-02-19T21:05:34.862640+0000","last_active":"2024-02-19T21:05:34.862640+0000","last_peered":"2024-02-19T21:05:34.862640+0000","last_clean":"2024-02-19T21:05:34.862640+0000","last_became_active":"2024-02-19T21:05:24.212054+0000","last_became_peered":"2024-02-19T21:05:24.212054+0000","last_unstale":"2024-02-19T21:05:34.862640+0000","last_undegraded":"2024-02-19T21:05:34.862640+0000","last_fullsized":"2024-02-19T21:05:34.862640+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'12","last_scrub_stamp":"2024-02-19T21:05:34.862574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:34.862574+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":126,"num_objects":12,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1b","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185048+0000","last_change":"2024-02-19T21:05:23.197050+0000","last_active":"2024-02-19T21:05:33.185048+0000","last_peered":"2024-02-19T21:05:33.185048+0000","last_clean":"2024-02-19T21:05:33.185048+0000","last_became_active":"2024-02-19T21:04:56.811490+0000","last_became_peered":"2024-02-19T21:04:56.811490+0000","last_unstale":"2024-02-19T21:05:33.185048+0000","last_undegraded":"2024-02-19T21:05:33.185048+0000","last_fullsized":"2024-02-19T21:05:33.185048+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"94.4","version":"151'8","reported_seq":33,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.184462+0000","last_change":"2024-02-19T21:05:24.204725+0000","last_active":"2024-02-19T21:05:33.184462+0000","last_peered":"2024-02-19T21:05:33.184462+0000","last_clean":"2024-02-19T21:05:33.184462+0000","last_became_active":"2024-02-19T21:05:24.204240+0000","last_became_peered":"2024-02-19T21:05:24.204240+0000","last_unstale":"2024-02-19T21:05:33.184462+0000","last_undegraded":"2024-02-19T21:05:33.184462+0000","last_fullsized":"2024-02-19T21:05:33.184462+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":8,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.1a","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.176630+0000","last_change":"2024-02-19T21:05:23.192465+0000","last_active":"2024-02-19T21:05:33.176630+0000","last_peered":"2024-02-19T21:05:33.176630+0000","last_clean":"2024-02-19T21:05:33.176630+0000","last_became_active":"2024-02-19T21:04:57.814548+0000","last_became_peered":"2024-02-19T21:04:57.814548+0000","last_unstale":"2024-02-19T21:05:33.176630+0000","last_undegraded":"2024-02-19T21:05:33.176630+0000","last_fullsized":"2024-02-19T21:05:33.176630+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"94.7","version":"151'2","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172516+0000","last_change":"2024-02-19T21:05:24.211560+0000","last_active":"2024-02-19T21:05:33.172516+0000","last_peered":"2024-02-19T21:05:33.172516+0000","last_clean":"2024-02-19T21:05:33.172516+0000","last_became_active":"2024-02-19T21:05:24.210711+0000","last_became_peered":"2024-02-19T21:05:24.210711+0000","last_unstale":"2024-02-19T21:05:33.172516+0000","last_undegraded":"2024-02-19T21:05:33.172516+0000","last_fullsized":"2024-02-19T21:05:33.172516+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.19","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170443+0000","last_change":"2024-02-19T21:05:23.177281+0000","last_active":"2024-02-19T21:05:33.170443+0000","last_peered":"2024-02-19T21:05:33.170443+0000","last_clean":"2024-02-19T21:05:33.170443+0000","last_became_active":"2024-02-19T21:04:57.876034+0000","last_became_peered":"2024-02-19T21:04:57.876034+0000","last_unstale":"2024-02-19T21:05:33.170443+0000","last_undegraded":"2024-02-19T21:05:33.170443+0000","last_fullsized":"2024-02-19T21:05:33.170443+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"94.6","version":"151'3","reported_seq":24,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170828+0000","last_change":"2024-02-19T21:05:24.211279+0000","last_active":"2024-02-19T21:05:33.170828+0000","last_peered":"2024-02-19T21:05:33.170828+0000","last_clean":"2024-02-19T21:05:33.170828+0000","last_became_active":"2024-02-19T21:05:24.210977+0000","last_became_peered":"2024-02-19T21:05:24.210977+0000","last_unstale":"2024-02-19T21:05:33.170828+0000","last_undegraded":"2024-02-19T21:05:33.170828+0000","last_fullsized":"2024-02-19T21:05:33.170828+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.18","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.174393+0000","last_change":"2024-02-19T21:05:23.198111+0000","last_active":"2024-02-19T21:05:33.174393+0000","last_peered":"2024-02-19T21:05:33.174393+0000","last_clean":"2024-02-19T21:05:33.174393+0000","last_became_active":"2024-02-19T21:04:57.852648+0000","last_became_peered":"2024-02-19T21:04:57.852648+0000","last_unstale":"2024-02-19T21:05:33.174393+0000","last_undegraded":"2024-02-19T21:05:33.174393+0000","last_fullsized":"2024-02-19T21:05:33.174393+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"94.9","version":"151'11","reported_seq":37,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.176896+0000","last_change":"2024-02-19T21:05:24.182513+0000","last_active":"2024-02-19T21:05:33.176896+0000","last_peered":"2024-02-19T21:05:33.176896+0000","last_clean":"2024-02-19T21:05:33.176896+0000","last_became_active":"2024-02-19T21:05:24.182235+0000","last_became_peered":"2024-02-19T21:05:24.182235+0000","last_unstale":"2024-02-19T21:05:33.176896+0000","last_undegraded":"2024-02-19T21:05:33.176896+0000","last_fullsized":"2024-02-19T21:05:33.176896+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":112,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":11,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.17","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.189830+0000","last_change":"2024-02-19T21:05:23.193754+0000","last_active":"2024-02-19T21:05:33.189830+0000","last_peered":"2024-02-19T21:05:33.189830+0000","last_clean":"2024-02-19T21:05:33.189830+0000","last_became_active":"2024-02-19T21:04:57.875388+0000","last_became_peered":"2024-02-19T21:04:57.875388+0000","last_unstale":"2024-02-19T21:05:33.189830+0000","last_undegraded":"2024-02-19T21:05:33.189830+0000","last_fullsized":"2024-02-19T21:05:33.189830+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"94.8","version":"0'0","reported_seq":19,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185427+0000","last_change":"2024-02-19T21:05:24.211451+0000","last_active":"2024-02-19T21:05:33.185427+0000","last_peered":"2024-02-19T21:05:33.185427+0000","last_clean":"2024-02-19T21:05:33.185427+0000","last_became_active":"2024-02-19T21:05:24.210981+0000","last_became_peered":"2024-02-19T21:05:24.210981+0000","last_unstale":"2024-02-19T21:05:33.185427+0000","last_undegraded":"2024-02-19T21:05:33.185427+0000","last_fullsized":"2024-02-19T21:05:33.185427+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.16","version":"137'5","reported_seq":95,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183647+0000","last_change":"2024-02-19T21:05:23.199315+0000","last_active":"2024-02-19T21:05:33.183647+0000","last_peered":"2024-02-19T21:05:33.183647+0000","last_clean":"2024-02-19T21:05:33.183647+0000","last_became_active":"2024-02-19T21:04:56.788989+0000","last_became_peered":"2024-02-19T21:04:56.788989+0000","last_unstale":"2024-02-19T21:05:33.183647+0000","last_undegraded":"2024-02-19T21:05:33.183647+0000","last_fullsized":"2024-02-19T21:05:33.183647+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"94.b","version":"151'7","reported_seq":30,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177676+0000","last_change":"2024-02-19T21:05:24.191838+0000","last_active":"2024-02-19T21:05:33.177676+0000","last_peered":"2024-02-19T21:05:33.177676+0000","last_clean":"2024-02-19T21:05:33.177676+0000","last_became_active":"2024-02-19T21:05:24.191550+0000","last_became_peered":"2024-02-19T21:05:24.191550+0000","last_unstale":"2024-02-19T21:05:33.177676+0000","last_undegraded":"2024-02-19T21:05:33.177676+0000","last_fullsized":"2024-02-19T21:05:33.177676+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.15","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170285+0000","last_change":"2024-02-19T21:05:23.179573+0000","last_active":"2024-02-19T21:05:33.170285+0000","last_peered":"2024-02-19T21:05:33.170285+0000","last_clean":"2024-02-19T21:05:33.170285+0000","last_became_active":"2024-02-19T21:04:56.846059+0000","last_became_peered":"2024-02-19T21:04:56.846059+0000","last_unstale":"2024-02-19T21:05:33.170285+0000","last_undegraded":"2024-02-19T21:05:33.170285+0000","last_fullsized":"2024-02-19T21:05:33.170285+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"94.a","version":"151'2","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.182756+0000","last_change":"2024-02-19T21:05:24.169990+0000","last_active":"2024-02-19T21:05:33.182756+0000","last_peered":"2024-02-19T21:05:33.182756+0000","last_clean":"2024-02-19T21:05:33.182756+0000","last_became_active":"2024-02-19T21:05:24.169693+0000","last_became_peered":"2024-02-19T21:05:24.169693+0000","last_unstale":"2024-02-19T21:05:33.182756+0000","last_undegraded":"2024-02-19T21:05:33.182756+0000","last_fullsized":"2024-02-19T21:05:33.182756+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.14","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.184727+0000","last_change":"2024-02-19T21:05:23.197815+0000","last_active":"2024-02-19T21:05:33.184727+0000","last_peered":"2024-02-19T21:05:33.184727+0000","last_clean":"2024-02-19T21:05:33.184727+0000","last_became_active":"2024-02-19T21:04:56.812598+0000","last_became_peered":"2024-02-19T21:04:56.812598+0000","last_unstale":"2024-02-19T21:05:33.184727+0000","last_undegraded":"2024-02-19T21:05:33.184727+0000","last_fullsized":"2024-02-19T21:05:33.184727+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"94.d","version":"151'4","reported_seq":26,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.178368+0000","last_change":"2024-02-19T21:05:24.214985+0000","last_active":"2024-02-19T21:05:33.178368+0000","last_peered":"2024-02-19T21:05:33.178368+0000","last_clean":"2024-02-19T21:05:33.178368+0000","last_became_active":"2024-02-19T21:05:24.213384+0000","last_became_peered":"2024-02-19T21:05:24.213384+0000","last_unstale":"2024-02-19T21:05:33.178368+0000","last_undegraded":"2024-02-19T21:05:33.178368+0000","last_fullsized":"2024-02-19T21:05:33.178368+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.13","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168559+0000","last_change":"2024-02-19T21:05:23.183446+0000","last_active":"2024-02-19T21:05:33.168559+0000","last_peered":"2024-02-19T21:05:33.168559+0000","last_clean":"2024-02-19T21:05:33.168559+0000","last_became_active":"2024-02-19T21:04:56.825057+0000","last_became_peered":"2024-02-19T21:04:56.825057+0000","last_unstale":"2024-02-19T21:05:33.168559+0000","last_undegraded":"2024-02-19T21:05:33.168559+0000","last_fullsized":"2024-02-19T21:05:33.168559+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"94.c","version":"151'7","reported_seq":33,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.188574+0000","last_change":"2024-02-19T21:05:24.188700+0000","last_active":"2024-02-19T21:05:33.188574+0000","last_peered":"2024-02-19T21:05:33.188574+0000","last_clean":"2024-02-19T21:05:33.188574+0000","last_became_active":"2024-02-19T21:05:24.188350+0000","last_became_peered":"2024-02-19T21:05:24.188350+0000","last_unstale":"2024-02-19T21:05:33.188574+0000","last_undegraded":"2024-02-19T21:05:33.188574+0000","last_fullsized":"2024-02-19T21:05:33.188574+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.12","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181143+0000","last_change":"2024-02-19T21:05:23.181376+0000","last_active":"2024-02-19T21:05:33.181143+0000","last_peered":"2024-02-19T21:05:33.181143+0000","last_clean":"2024-02-19T21:05:33.181143+0000","last_became_active":"2024-02-19T21:04:57.833886+0000","last_became_peered":"2024-02-19T21:04:57.833886+0000","last_unstale":"2024-02-19T21:05:33.181143+0000","last_undegraded":"2024-02-19T21:05:33.181143+0000","last_fullsized":"2024-02-19T21:05:33.181143+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"94.f","version":"151'6","reported_seq":29,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171930+0000","last_change":"2024-02-19T21:05:24.181352+0000","last_active":"2024-02-19T21:05:33.171930+0000","last_peered":"2024-02-19T21:05:33.171930+0000","last_clean":"2024-02-19T21:05:33.171930+0000","last_became_active":"2024-02-19T21:05:24.181048+0000","last_became_peered":"2024-02-19T21:05:24.181048+0000","last_unstale":"2024-02-19T21:05:33.171930+0000","last_undegraded":"2024-02-19T21:05:33.171930+0000","last_fullsized":"2024-02-19T21:05:33.171930+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.11","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170535+0000","last_change":"2024-02-19T21:05:23.178833+0000","last_active":"2024-02-19T21:05:33.170535+0000","last_peered":"2024-02-19T21:05:33.170535+0000","last_clean":"2024-02-19T21:05:33.170535+0000","last_became_active":"2024-02-19T21:04:57.883650+0000","last_became_peered":"2024-02-19T21:04:57.883650+0000","last_unstale":"2024-02-19T21:05:33.170535+0000","last_undegraded":"2024-02-19T21:05:33.170535+0000","last_fullsized":"2024-02-19T21:05:33.170535+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"94.e","version":"151'8","reported_seq":85,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172143+0000","last_change":"2024-02-19T21:05:24.181380+0000","last_active":"2024-02-19T21:05:33.172143+0000","last_peered":"2024-02-19T21:05:33.172143+0000","last_clean":"2024-02-19T21:05:33.172143+0000","last_became_active":"2024-02-19T21:05:24.181100+0000","last_became_peered":"2024-02-19T21:05:24.181100+0000","last_unstale":"2024-02-19T21:05:33.172143+0000","last_undegraded":"2024-02-19T21:05:33.172143+0000","last_fullsized":"2024-02-19T21:05:33.172143+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":58,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":8,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.10","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.167988+0000","last_change":"2024-02-19T21:05:23.178729+0000","last_active":"2024-02-19T21:05:33.167988+0000","last_peered":"2024-02-19T21:05:33.167988+0000","last_clean":"2024-02-19T21:05:33.167988+0000","last_became_active":"2024-02-19T21:04:57.883813+0000","last_became_peered":"2024-02-19T21:04:57.883813+0000","last_unstale":"2024-02-19T21:05:33.167988+0000","last_undegraded":"2024-02-19T21:05:33.167988+0000","last_fullsized":"2024-02-19T21:05:33.167988+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"94.11","version":"151'7","reported_seq":31,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.175483+0000","last_change":"2024-02-19T21:05:24.197535+0000","last_active":"2024-02-19T21:05:33.175483+0000","last_peered":"2024-02-19T21:05:33.175483+0000","last_clean":"2024-02-19T21:05:33.175483+0000","last_became_active":"2024-02-19T21:05:24.196656+0000","last_became_peered":"2024-02-19T21:05:24.196656+0000","last_unstale":"2024-02-19T21:05:33.175483+0000","last_undegraded":"2024-02-19T21:05:33.175483+0000","last_fullsized":"2024-02-19T21:05:33.175483+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.f","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.184852+0000","last_change":"2024-02-19T21:05:23.202404+0000","last_active":"2024-02-19T21:05:33.184852+0000","last_peered":"2024-02-19T21:05:33.184852+0000","last_clean":"2024-02-19T21:05:33.184852+0000","last_became_active":"2024-02-19T21:04:57.852708+0000","last_became_peered":"2024-02-19T21:04:57.852708+0000","last_unstale":"2024-02-19T21:05:33.184852+0000","last_undegraded":"2024-02-19T21:05:33.184852+0000","last_fullsized":"2024-02-19T21:05:33.184852+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"94.10","version":"151'4","reported_seq":26,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185376+0000","last_change":"2024-02-19T21:05:24.211264+0000","last_active":"2024-02-19T21:05:33.185376+0000","last_peered":"2024-02-19T21:05:33.185376+0000","last_clean":"2024-02-19T21:05:33.185376+0000","last_became_active":"2024-02-19T21:05:24.210605+0000","last_became_peered":"2024-02-19T21:05:24.210605+0000","last_unstale":"2024-02-19T21:05:33.185376+0000","last_undegraded":"2024-02-19T21:05:33.185376+0000","last_fullsized":"2024-02-19T21:05:33.185376+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.e","version":"122'6","reported_seq":97,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.191697+0000","last_change":"2024-02-19T21:05:23.193379+0000","last_active":"2024-02-19T21:05:33.191697+0000","last_peered":"2024-02-19T21:05:33.191697+0000","last_clean":"2024-02-19T21:05:33.191697+0000","last_became_active":"2024-02-19T21:04:57.875396+0000","last_became_peered":"2024-02-19T21:04:57.875396+0000","last_unstale":"2024-02-19T21:05:33.191697+0000","last_undegraded":"2024-02-19T21:05:33.191697+0000","last_fullsized":"2024-02-19T21:05:33.191697+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"94.13","version":"151'10","reported_seq":36,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177927+0000","last_change":"2024-02-19T21:05:24.198995+0000","last_active":"2024-02-19T21:05:33.177927+0000","last_peered":"2024-02-19T21:05:33.177927+0000","last_clean":"2024-02-19T21:05:33.177927+0000","last_became_active":"2024-02-19T21:05:24.198509+0000","last_became_peered":"2024-02-19T21:05:24.198509+0000","last_unstale":"2024-02-19T21:05:33.177927+0000","last_undegraded":"2024-02-19T21:05:33.177927+0000","last_fullsized":"2024-02-19T21:05:33.177927+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":10,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.d","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171905+0000","last_change":"2024-02-19T21:05:23.182594+0000","last_active":"2024-02-19T21:05:33.171905+0000","last_peered":"2024-02-19T21:05:33.171905+0000","last_clean":"2024-02-19T21:05:33.171905+0000","last_became_active":"2024-02-19T21:04:57.811652+0000","last_became_peered":"2024-02-19T21:04:57.811652+0000","last_unstale":"2024-02-19T21:05:33.171905+0000","last_undegraded":"2024-02-19T21:05:33.171905+0000","last_fullsized":"2024-02-19T21:05:33.171905+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"94.12","version":"151'4","reported_seq":28,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170721+0000","last_change":"2024-02-19T21:05:24.194024+0000","last_active":"2024-02-19T21:05:33.170721+0000","last_peered":"2024-02-19T21:05:33.170721+0000","last_clean":"2024-02-19T21:05:33.170721+0000","last_became_active":"2024-02-19T21:05:24.193766+0000","last_became_peered":"2024-02-19T21:05:24.193766+0000","last_unstale":"2024-02-19T21:05:33.170721+0000","last_undegraded":"2024-02-19T21:05:33.170721+0000","last_fullsized":"2024-02-19T21:05:33.170721+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.c","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.187487+0000","last_change":"2024-02-19T21:05:23.197721+0000","last_active":"2024-02-19T21:05:33.187487+0000","last_peered":"2024-02-19T21:05:33.187487+0000","last_clean":"2024-02-19T21:05:33.187487+0000","last_became_active":"2024-02-19T21:04:57.827332+0000","last_became_peered":"2024-02-19T21:04:57.827332+0000","last_unstale":"2024-02-19T21:05:33.187487+0000","last_undegraded":"2024-02-19T21:05:33.187487+0000","last_fullsized":"2024-02-19T21:05:33.187487+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"94.15","version":"151'5","reported_seq":28,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177562+0000","last_change":"2024-02-19T21:05:24.199668+0000","last_active":"2024-02-19T21:05:33.177562+0000","last_peered":"2024-02-19T21:05:33.177562+0000","last_clean":"2024-02-19T21:05:33.177562+0000","last_became_active":"2024-02-19T21:05:24.199404+0000","last_became_peered":"2024-02-19T21:05:24.199404+0000","last_unstale":"2024-02-19T21:05:33.177562+0000","last_undegraded":"2024-02-19T21:05:33.177562+0000","last_fullsized":"2024-02-19T21:05:33.177562+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.b","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185125+0000","last_change":"2024-02-19T21:05:23.197237+0000","last_active":"2024-02-19T21:05:33.185125+0000","last_peered":"2024-02-19T21:05:33.185125+0000","last_clean":"2024-02-19T21:05:33.185125+0000","last_became_active":"2024-02-19T21:04:56.821857+0000","last_became_peered":"2024-02-19T21:04:56.821857+0000","last_unstale":"2024-02-19T21:05:33.185125+0000","last_undegraded":"2024-02-19T21:05:33.185125+0000","last_fullsized":"2024-02-19T21:05:33.185125+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"94.14","version":"151'5","reported_seq":28,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.182760+0000","last_change":"2024-02-19T21:05:24.196883+0000","last_active":"2024-02-19T21:05:33.182760+0000","last_peered":"2024-02-19T21:05:33.182760+0000","last_clean":"2024-02-19T21:05:33.182760+0000","last_became_active":"2024-02-19T21:05:24.196588+0000","last_became_peered":"2024-02-19T21:05:24.196588+0000","last_unstale":"2024-02-19T21:05:33.182760+0000","last_undegraded":"2024-02-19T21:05:33.182760+0000","last_fullsized":"2024-02-19T21:05:33.182760+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"64.a","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170644+0000","last_change":"2024-02-19T21:05:23.178625+0000","last_active":"2024-02-19T21:05:33.170644+0000","last_peered":"2024-02-19T21:05:33.170644+0000","last_clean":"2024-02-19T21:05:33.170644+0000","last_became_active":"2024-02-19T21:04:56.825866+0000","last_became_peered":"2024-02-19T21:04:56.825866+0000","last_unstale":"2024-02-19T21:05:33.170644+0000","last_undegraded":"2024-02-19T21:05:33.170644+0000","last_fullsized":"2024-02-19T21:05:33.170644+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"94.17","version":"151'2","reported_seq":23,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.189744+0000","last_change":"2024-02-19T21:05:24.193025+0000","last_active":"2024-02-19T21:05:33.189744+0000","last_peered":"2024-02-19T21:05:33.189744+0000","last_clean":"2024-02-19T21:05:33.189744+0000","last_became_active":"2024-02-19T21:05:24.192743+0000","last_became_peered":"2024-02-19T21:05:24.192743+0000","last_unstale":"2024-02-19T21:05:33.189744+0000","last_undegraded":"2024-02-19T21:05:33.189744+0000","last_fullsized":"2024-02-19T21:05:33.189744+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.9","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169345+0000","last_change":"2024-02-19T21:05:23.178404+0000","last_active":"2024-02-19T21:05:33.169345+0000","last_peered":"2024-02-19T21:05:33.169345+0000","last_clean":"2024-02-19T21:05:33.169345+0000","last_became_active":"2024-02-19T21:04:56.836513+0000","last_became_peered":"2024-02-19T21:04:56.836513+0000","last_unstale":"2024-02-19T21:05:33.169345+0000","last_undegraded":"2024-02-19T21:05:33.169345+0000","last_fullsized":"2024-02-19T21:05:33.169345+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"94.16","version":"0'0","reported_seq":19,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172055+0000","last_change":"2024-02-19T21:05:24.194861+0000","last_active":"2024-02-19T21:05:33.172055+0000","last_peered":"2024-02-19T21:05:33.172055+0000","last_clean":"2024-02-19T21:05:33.172055+0000","last_became_active":"2024-02-19T21:05:24.194581+0000","last_became_peered":"2024-02-19T21:05:24.194581+0000","last_unstale":"2024-02-19T21:05:33.172055+0000","last_undegraded":"2024-02-19T21:05:33.172055+0000","last_fullsized":"2024-02-19T21:05:33.172055+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.8","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183688+0000","last_change":"2024-02-19T21:05:23.181129+0000","last_active":"2024-02-19T21:05:33.183688+0000","last_peered":"2024-02-19T21:05:33.183688+0000","last_clean":"2024-02-19T21:05:33.183688+0000","last_became_active":"2024-02-19T21:04:57.814817+0000","last_became_peered":"2024-02-19T21:04:57.814817+0000","last_unstale":"2024-02-19T21:05:33.183688+0000","last_undegraded":"2024-02-19T21:05:33.183688+0000","last_fullsized":"2024-02-19T21:05:33.183688+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"94.19","version":"151'12","reported_seq":40,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.180918+0000","last_change":"2024-02-19T21:05:24.208854+0000","last_active":"2024-02-19T21:05:33.180918+0000","last_peered":"2024-02-19T21:05:33.180918+0000","last_clean":"2024-02-19T21:05:33.180918+0000","last_became_active":"2024-02-19T21:05:24.208539+0000","last_became_peered":"2024-02-19T21:05:24.208539+0000","last_unstale":"2024-02-19T21:05:33.180918+0000","last_undegraded":"2024-02-19T21:05:33.180918+0000","last_fullsized":"2024-02-19T21:05:33.180918+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":120,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"64.7","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172618+0000","last_change":"2024-02-19T21:05:23.183000+0000","last_active":"2024-02-19T21:05:33.172618+0000","last_peered":"2024-02-19T21:05:33.172618+0000","last_clean":"2024-02-19T21:05:33.172618+0000","last_became_active":"2024-02-19T21:04:56.834071+0000","last_became_peered":"2024-02-19T21:04:56.834071+0000","last_unstale":"2024-02-19T21:05:33.172618+0000","last_undegraded":"2024-02-19T21:05:33.172618+0000","last_fullsized":"2024-02-19T21:05:33.172618+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"94.18","version":"151'12","reported_seq":40,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170243+0000","last_change":"2024-02-19T21:05:24.181969+0000","last_active":"2024-02-19T21:05:33.170243+0000","last_peered":"2024-02-19T21:05:33.170243+0000","last_clean":"2024-02-19T21:05:33.170243+0000","last_became_active":"2024-02-19T21:05:24.181631+0000","last_became_peered":"2024-02-19T21:05:24.181631+0000","last_unstale":"2024-02-19T21:05:33.170243+0000","last_undegraded":"2024-02-19T21:05:33.170243+0000","last_fullsized":"2024-02-19T21:05:33.170243+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":129,"num_objects":12,"num_object_clones":7,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":12,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.6","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177622+0000","last_change":"2024-02-19T21:05:23.198799+0000","last_active":"2024-02-19T21:05:33.177622+0000","last_peered":"2024-02-19T21:05:33.177622+0000","last_clean":"2024-02-19T21:05:33.177622+0000","last_became_active":"2024-02-19T21:04:57.839254+0000","last_became_peered":"2024-02-19T21:04:57.839254+0000","last_unstale":"2024-02-19T21:05:33.177622+0000","last_undegraded":"2024-02-19T21:05:33.177622+0000","last_fullsized":"2024-02-19T21:05:33.177622+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"94.1b","version":"151'3","reported_seq":24,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170308+0000","last_change":"2024-02-19T21:05:24.193697+0000","last_active":"2024-02-19T21:05:33.170308+0000","last_peered":"2024-02-19T21:05:33.170308+0000","last_clean":"2024-02-19T21:05:33.170308+0000","last_became_active":"2024-02-19T21:05:24.193393+0000","last_became_peered":"2024-02-19T21:05:24.193393+0000","last_unstale":"2024-02-19T21:05:33.170308+0000","last_undegraded":"2024-02-19T21:05:33.170308+0000","last_fullsized":"2024-02-19T21:05:33.170308+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.5","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183189+0000","last_change":"2024-02-19T21:05:23.196246+0000","last_active":"2024-02-19T21:05:33.183189+0000","last_peered":"2024-02-19T21:05:33.183189+0000","last_clean":"2024-02-19T21:05:33.183189+0000","last_became_active":"2024-02-19T21:04:57.820149+0000","last_became_peered":"2024-02-19T21:04:57.820149+0000","last_unstale":"2024-02-19T21:05:33.183189+0000","last_undegraded":"2024-02-19T21:05:33.183189+0000","last_fullsized":"2024-02-19T21:05:33.183189+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"94.1a","version":"151'14","reported_seq":45,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170466+0000","last_change":"2024-02-19T21:05:24.210764+0000","last_active":"2024-02-19T21:05:33.170466+0000","last_peered":"2024-02-19T21:05:33.170466+0000","last_clean":"2024-02-19T21:05:33.170466+0000","last_became_active":"2024-02-19T21:05:24.210480+0000","last_became_peered":"2024-02-19T21:05:24.210480+0000","last_unstale":"2024-02-19T21:05:33.170466+0000","last_undegraded":"2024-02-19T21:05:33.170466+0000","last_fullsized":"2024-02-19T21:05:33.170466+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":148,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":14,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"64.4","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177351+0000","last_change":"2024-02-19T21:05:23.191801+0000","last_active":"2024-02-19T21:05:33.177351+0000","last_peered":"2024-02-19T21:05:33.177351+0000","last_clean":"2024-02-19T21:05:33.177351+0000","last_became_active":"2024-02-19T21:04:57.829267+0000","last_became_peered":"2024-02-19T21:04:57.829267+0000","last_unstale":"2024-02-19T21:05:33.177351+0000","last_undegraded":"2024-02-19T21:05:33.177351+0000","last_fullsized":"2024-02-19T21:05:33.177351+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"94.1d","version":"151'6","reported_seq":29,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171770+0000","last_change":"2024-02-19T21:05:24.181423+0000","last_active":"2024-02-19T21:05:33.171770+0000","last_peered":"2024-02-19T21:05:33.171770+0000","last_clean":"2024-02-19T21:05:33.171770+0000","last_became_active":"2024-02-19T21:05:24.181065+0000","last_became_peered":"2024-02-19T21:05:24.181065+0000","last_unstale":"2024-02-19T21:05:33.171770+0000","last_undegraded":"2024-02-19T21:05:33.171770+0000","last_fullsized":"2024-02-19T21:05:33.171770+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"64.3","version":"140'4","reported_seq":94,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168382+0000","last_change":"2024-02-19T21:05:23.179848+0000","last_active":"2024-02-19T21:05:33.168382+0000","last_peered":"2024-02-19T21:05:33.168382+0000","last_clean":"2024-02-19T21:05:33.168382+0000","last_became_active":"2024-02-19T21:04:57.856066+0000","last_became_peered":"2024-02-19T21:04:57.856066+0000","last_unstale":"2024-02-19T21:05:33.168382+0000","last_undegraded":"2024-02-19T21:05:33.168382+0000","last_fullsized":"2024-02-19T21:05:33.168382+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"94.1c","version":"151'20","reported_seq":54,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.188476+0000","last_change":"2024-02-19T21:05:24.187518+0000","last_active":"2024-02-19T21:05:33.188476+0000","last_peered":"2024-02-19T21:05:33.188476+0000","last_clean":"2024-02-19T21:05:33.188476+0000","last_became_active":"2024-02-19T21:05:24.187188+0000","last_became_peered":"2024-02-19T21:05:24.187188+0000","last_unstale":"2024-02-19T21:05:33.188476+0000","last_undegraded":"2024-02-19T21:05:33.188476+0000","last_fullsized":"2024-02-19T21:05:33.188476+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":12,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":20,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"64.2","version":"0'0","reported_seq":91,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185501+0000","last_change":"2024-02-19T21:05:23.189965+0000","last_active":"2024-02-19T21:05:33.185501+0000","last_peered":"2024-02-19T21:05:33.185501+0000","last_clean":"2024-02-19T21:05:33.185501+0000","last_became_active":"2024-02-19T21:04:57.822605+0000","last_became_peered":"2024-02-19T21:04:57.822605+0000","last_unstale":"2024-02-19T21:05:33.185501+0000","last_undegraded":"2024-02-19T21:05:33.185501+0000","last_fullsized":"2024-02-19T21:05:33.185501+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"94.1f","version":"151'7","reported_seq":32,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181605+0000","last_change":"2024-02-19T21:05:24.209741+0000","last_active":"2024-02-19T21:05:33.181605+0000","last_peered":"2024-02-19T21:05:33.181605+0000","last_clean":"2024-02-19T21:05:33.181605+0000","last_became_active":"2024-02-19T21:05:24.209502+0000","last_became_peered":"2024-02-19T21:05:24.209502+0000","last_unstale":"2024-02-19T21:05:33.181605+0000","last_undegraded":"2024-02-19T21:05:33.181605+0000","last_fullsized":"2024-02-19T21:05:33.181605+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":71,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":7,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"64.1","version":"126'9","reported_seq":105,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.175580+0000","last_change":"2024-02-19T21:05:23.199719+0000","last_active":"2024-02-19T21:05:33.175580+0000","last_peered":"2024-02-19T21:05:33.175580+0000","last_clean":"2024-02-19T21:05:33.175580+0000","last_became_active":"2024-02-19T21:04:57.808699+0000","last_became_peered":"2024-02-19T21:04:57.808699+0000","last_unstale":"2024-02-19T21:05:33.175580+0000","last_undegraded":"2024-02-19T21:05:33.175580+0000","last_fullsized":"2024-02-19T21:05:33.175580+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":384,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"94.1e","version":"151'4","reported_seq":27,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177388+0000","last_change":"2024-02-19T21:05:24.203896+0000","last_active":"2024-02-19T21:05:33.177388+0000","last_peered":"2024-02-19T21:05:33.177388+0000","last_clean":"2024-02-19T21:05:33.177388+0000","last_became_active":"2024-02-19T21:05:24.203573+0000","last_became_peered":"2024-02-19T21:05:24.203573+0000","last_unstale":"2024-02-19T21:05:33.177388+0000","last_undegraded":"2024-02-19T21:05:33.177388+0000","last_fullsized":"2024-02-19T21:05:33.177388+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":143,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:23.164578+0000","last_clean_scrub_stamp":"2024-02-19T21:05:23.164578+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"64.0","version":"141'1","reported_seq":92,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169733+0000","last_change":"2024-02-19T21:05:23.175176+0000","last_active":"2024-02-19T21:05:33.169733+0000","last_peered":"2024-02-19T21:05:33.169733+0000","last_clean":"2024-02-19T21:05:33.169733+0000","last_became_active":"2024-02-19T21:04:56.826453+0000","last_became_peered":"2024-02-19T21:04:56.826453+0000","last_unstale":"2024-02-19T21:05:33.169733+0000","last_undegraded":"2024-02-19T21:05:33.169733+0000","last_fullsized":"2024-02-19T21:05:33.169733+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:55.616535+0000","last_clean_scrub_stamp":"2024-02-19T21:04:55.616535+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.2","version":"148'41","reported_seq":96,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.173652+0000","last_change":"2024-02-19T21:05:21.286457+0000","last_active":"2024-02-19T21:05:33.173652+0000","last_peered":"2024-02-19T21:05:33.173652+0000","last_clean":"2024-02-19T21:05:33.173652+0000","last_became_active":"2024-02-19T21:04:54.609091+0000","last_became_peered":"2024-02-19T21:04:54.609091+0000","last_unstale":"2024-02-19T21:05:33.173652+0000","last_undegraded":"2024-02-19T21:05:33.173652+0000","last_fullsized":"2024-02-19T21:05:33.173652+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":213,"num_objects":21,"num_object_clones":16,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.3","version":"148'36","reported_seq":83,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185405+0000","last_change":"2024-02-19T21:05:19.314353+0000","last_active":"2024-02-19T21:05:33.185405+0000","last_peered":"2024-02-19T21:05:33.185405+0000","last_clean":"2024-02-19T21:05:33.185405+0000","last_became_active":"2024-02-19T21:04:54.641884+0000","last_became_peered":"2024-02-19T21:04:54.641884+0000","last_unstale":"2024-02-19T21:05:33.185405+0000","last_undegraded":"2024-02-19T21:05:33.185405+0000","last_fullsized":"2024-02-19T21:05:33.185405+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.0","version":"148'18","reported_seq":70,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185549+0000","last_change":"2024-02-19T21:05:19.225070+0000","last_active":"2024-02-19T21:05:33.185549+0000","last_peered":"2024-02-19T21:05:33.185549+0000","last_clean":"2024-02-19T21:05:33.185549+0000","last_became_active":"2024-02-19T21:04:54.656450+0000","last_became_peered":"2024-02-19T21:04:54.656450+0000","last_unstale":"2024-02-19T21:05:33.185549+0000","last_undegraded":"2024-02-19T21:05:33.185549+0000","last_fullsized":"2024-02-19T21:05:33.185549+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.1","version":"148'36","reported_seq":88,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181211+0000","last_change":"2024-02-19T21:05:19.261616+0000","last_active":"2024-02-19T21:05:33.181211+0000","last_peered":"2024-02-19T21:05:33.181211+0000","last_clean":"2024-02-19T21:05:33.181211+0000","last_became_active":"2024-02-19T21:04:54.654642+0000","last_became_peered":"2024-02-19T21:04:54.654642+0000","last_unstale":"2024-02-19T21:05:33.181211+0000","last_undegraded":"2024-02-19T21:05:33.181211+0000","last_fullsized":"2024-02-19T21:05:33.181211+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.6","version":"148'18","reported_seq":67,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171166+0000","last_change":"2024-02-19T21:05:19.313683+0000","last_active":"2024-02-19T21:05:33.171166+0000","last_peered":"2024-02-19T21:05:33.171166+0000","last_clean":"2024-02-19T21:05:33.171166+0000","last_became_active":"2024-02-19T21:04:54.658542+0000","last_became_peered":"2024-02-19T21:04:54.658542+0000","last_unstale":"2024-02-19T21:05:33.171166+0000","last_undegraded":"2024-02-19T21:05:33.171166+0000","last_fullsized":"2024-02-19T21:05:33.171166+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.7","version":"148'10","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183052+0000","last_change":"2024-02-19T21:05:19.172202+0000","last_active":"2024-02-19T21:05:33.183052+0000","last_peered":"2024-02-19T21:05:33.183052+0000","last_clean":"2024-02-19T21:05:33.183052+0000","last_became_active":"2024-02-19T21:04:54.598061+0000","last_became_peered":"2024-02-19T21:04:54.598061+0000","last_unstale":"2024-02-19T21:05:33.183052+0000","last_undegraded":"2024-02-19T21:05:33.183052+0000","last_fullsized":"2024-02-19T21:05:33.183052+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":56,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.4","version":"148'36","reported_seq":84,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169677+0000","last_change":"2024-02-19T21:05:19.256837+0000","last_active":"2024-02-19T21:05:33.169677+0000","last_peered":"2024-02-19T21:05:33.169677+0000","last_clean":"2024-02-19T21:05:33.169677+0000","last_became_active":"2024-02-19T21:04:54.639335+0000","last_became_peered":"2024-02-19T21:04:54.639335+0000","last_unstale":"2024-02-19T21:05:33.169677+0000","last_undegraded":"2024-02-19T21:05:33.169677+0000","last_fullsized":"2024-02-19T21:05:33.169677+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.5","version":"148'63","reported_seq":106,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183943+0000","last_change":"2024-02-19T21:05:19.320366+0000","last_active":"2024-02-19T21:05:33.183943+0000","last_peered":"2024-02-19T21:05:33.183943+0000","last_clean":"2024-02-19T21:05:33.183943+0000","last_became_active":"2024-02-19T21:04:54.646842+0000","last_became_peered":"2024-02-19T21:04:54.646842+0000","last_unstale":"2024-02-19T21:05:33.183943+0000","last_undegraded":"2024-02-19T21:05:33.183943+0000","last_fullsized":"2024-02-19T21:05:33.183943+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":63,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":364,"num_objects":35,"num_object_clones":28,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.a","version":"148'9","reported_seq":61,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.173564+0000","last_change":"2024-02-19T21:05:19.247008+0000","last_active":"2024-02-19T21:05:33.173564+0000","last_peered":"2024-02-19T21:05:33.173564+0000","last_clean":"2024-02-19T21:05:33.173564+0000","last_became_active":"2024-02-19T21:04:54.611831+0000","last_became_peered":"2024-02-19T21:04:54.611831+0000","last_unstale":"2024-02-19T21:05:33.173564+0000","last_undegraded":"2024-02-19T21:05:33.173564+0000","last_fullsized":"2024-02-19T21:05:33.173564+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.b","version":"148'45","reported_seq":88,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185364+0000","last_change":"2024-02-19T21:05:19.305146+0000","last_active":"2024-02-19T21:05:33.185364+0000","last_peered":"2024-02-19T21:05:33.185364+0000","last_clean":"2024-02-19T21:05:33.185364+0000","last_became_active":"2024-02-19T21:04:54.640024+0000","last_became_peered":"2024-02-19T21:04:54.640024+0000","last_unstale":"2024-02-19T21:05:33.185364+0000","last_undegraded":"2024-02-19T21:05:33.185364+0000","last_fullsized":"2024-02-19T21:05:33.185364+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.8","version":"148'18","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.174662+0000","last_change":"2024-02-19T21:05:19.179008+0000","last_active":"2024-02-19T21:05:33.174662+0000","last_peered":"2024-02-19T21:05:33.174662+0000","last_clean":"2024-02-19T21:05:33.174662+0000","last_became_active":"2024-02-19T21:04:54.650062+0000","last_became_peered":"2024-02-19T21:04:54.650062+0000","last_unstale":"2024-02-19T21:05:33.174662+0000","last_undegraded":"2024-02-19T21:05:33.174662+0000","last_fullsized":"2024-02-19T21:05:33.174662+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.9","version":"148'36","reported_seq":88,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181128+0000","last_change":"2024-02-19T21:05:19.233556+0000","last_active":"2024-02-19T21:05:33.181128+0000","last_peered":"2024-02-19T21:05:33.181128+0000","last_clean":"2024-02-19T21:05:33.181128+0000","last_became_active":"2024-02-19T21:04:54.671646+0000","last_became_peered":"2024-02-19T21:04:54.671646+0000","last_unstale":"2024-02-19T21:05:33.181128+0000","last_undegraded":"2024-02-19T21:05:33.181128+0000","last_fullsized":"2024-02-19T21:05:33.181128+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.e","version":"148'18","reported_seq":73,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.191361+0000","last_change":"2024-02-19T21:05:19.184335+0000","last_active":"2024-02-19T21:05:33.191361+0000","last_peered":"2024-02-19T21:05:33.191361+0000","last_clean":"2024-02-19T21:05:33.191361+0000","last_became_active":"2024-02-19T21:04:54.603276+0000","last_became_peered":"2024-02-19T21:04:54.603276+0000","last_unstale":"2024-02-19T21:05:33.191361+0000","last_undegraded":"2024-02-19T21:05:33.191361+0000","last_fullsized":"2024-02-19T21:05:33.191361+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.f","version":"148'18","reported_seq":71,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177510+0000","last_change":"2024-02-19T21:05:19.192554+0000","last_active":"2024-02-19T21:05:33.177510+0000","last_peered":"2024-02-19T21:05:33.177510+0000","last_clean":"2024-02-19T21:05:33.177510+0000","last_became_active":"2024-02-19T21:04:54.610312+0000","last_became_peered":"2024-02-19T21:04:54.610312+0000","last_unstale":"2024-02-19T21:05:33.177510+0000","last_undegraded":"2024-02-19T21:05:33.177510+0000","last_fullsized":"2024-02-19T21:05:33.177510+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.c","version":"148'36","reported_seq":82,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.187661+0000","last_change":"2024-02-19T21:05:19.194391+0000","last_active":"2024-02-19T21:05:33.187661+0000","last_peered":"2024-02-19T21:05:33.187661+0000","last_clean":"2024-02-19T21:05:33.187661+0000","last_became_active":"2024-02-19T21:04:54.634803+0000","last_became_peered":"2024-02-19T21:04:54.634803+0000","last_unstale":"2024-02-19T21:05:33.187661+0000","last_undegraded":"2024-02-19T21:05:33.187661+0000","last_fullsized":"2024-02-19T21:05:33.187661+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.d","version":"148'18","reported_seq":69,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183284+0000","last_change":"2024-02-19T21:05:19.194415+0000","last_active":"2024-02-19T21:05:33.183284+0000","last_peered":"2024-02-19T21:05:33.183284+0000","last_clean":"2024-02-19T21:05:33.183284+0000","last_became_active":"2024-02-19T21:04:54.644749+0000","last_became_peered":"2024-02-19T21:04:54.644749+0000","last_unstale":"2024-02-19T21:05:33.183284+0000","last_undegraded":"2024-02-19T21:05:33.183284+0000","last_fullsized":"2024-02-19T21:05:33.183284+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.12","version":"148'18","reported_seq":68,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170729+0000","last_change":"2024-02-19T21:05:19.225502+0000","last_active":"2024-02-19T21:05:33.170729+0000","last_peered":"2024-02-19T21:05:33.170729+0000","last_clean":"2024-02-19T21:05:33.170729+0000","last_became_active":"2024-02-19T21:04:54.670998+0000","last_became_peered":"2024-02-19T21:04:54.670998+0000","last_unstale":"2024-02-19T21:05:33.170729+0000","last_undegraded":"2024-02-19T21:05:33.170729+0000","last_fullsized":"2024-02-19T21:05:33.170729+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.13","version":"148'36","reported_seq":86,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177945+0000","last_change":"2024-02-19T21:05:21.209659+0000","last_active":"2024-02-19T21:05:33.177945+0000","last_peered":"2024-02-19T21:05:33.177945+0000","last_clean":"2024-02-19T21:05:33.177945+0000","last_became_active":"2024-02-19T21:04:54.608802+0000","last_became_peered":"2024-02-19T21:04:54.608802+0000","last_unstale":"2024-02-19T21:05:33.177945+0000","last_undegraded":"2024-02-19T21:05:33.177945+0000","last_fullsized":"2024-02-19T21:05:33.177945+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.10","version":"148'23","reported_seq":74,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.174777+0000","last_change":"2024-02-19T21:05:21.213421+0000","last_active":"2024-02-19T21:05:33.174777+0000","last_peered":"2024-02-19T21:05:33.174777+0000","last_clean":"2024-02-19T21:05:33.174777+0000","last_became_active":"2024-02-19T21:04:54.626393+0000","last_became_peered":"2024-02-19T21:04:54.626393+0000","last_unstale":"2024-02-19T21:05:33.174777+0000","last_undegraded":"2024-02-19T21:05:33.174777+0000","last_fullsized":"2024-02-19T21:05:33.174777+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.11","version":"148'18","reported_seq":72,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169409+0000","last_change":"2024-02-19T21:05:19.251781+0000","last_active":"2024-02-19T21:05:33.169409+0000","last_peered":"2024-02-19T21:05:33.169409+0000","last_clean":"2024-02-19T21:05:33.169409+0000","last_became_active":"2024-02-19T21:04:54.659096+0000","last_became_peered":"2024-02-19T21:04:54.659096+0000","last_unstale":"2024-02-19T21:05:33.169409+0000","last_undegraded":"2024-02-19T21:05:33.169409+0000","last_fullsized":"2024-02-19T21:05:33.169409+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.184598+0000","last_change":"2024-02-19T21:05:22.208403+0000","last_active":"2024-02-19T21:05:33.184598+0000","last_peered":"2024-02-19T21:05:33.184598+0000","last_clean":"2024-02-19T21:05:33.184598+0000","last_became_active":"2024-02-19T21:05:22.208034+0000","last_became_peered":"2024-02-19T21:05:22.208034+0000","last_unstale":"2024-02-19T21:05:33.184598+0000","last_undegraded":"2024-02-19T21:05:33.184598+0000","last_fullsized":"2024-02-19T21:05:33.184598+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:12.305886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:12.305886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.16","version":"148'14","reported_seq":65,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172512+0000","last_change":"2024-02-19T21:05:23.202521+0000","last_active":"2024-02-19T21:05:33.172512+0000","last_peered":"2024-02-19T21:05:33.172512+0000","last_clean":"2024-02-19T21:05:33.172512+0000","last_became_active":"2024-02-19T21:04:54.608059+0000","last_became_peered":"2024-02-19T21:04:54.608059+0000","last_unstale":"2024-02-19T21:05:33.172512+0000","last_undegraded":"2024-02-19T21:05:33.172512+0000","last_fullsized":"2024-02-19T21:05:33.172512+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.167902+0000","last_change":"2024-02-19T21:05:22.220592+0000","last_active":"2024-02-19T21:05:33.167902+0000","last_peered":"2024-02-19T21:05:33.167902+0000","last_clean":"2024-02-19T21:05:33.167902+0000","last_became_active":"2024-02-19T21:05:22.220291+0000","last_became_peered":"2024-02-19T21:05:22.220291+0000","last_unstale":"2024-02-19T21:05:33.167902+0000","last_undegraded":"2024-02-19T21:05:33.167902+0000","last_fullsized":"2024-02-19T21:05:33.167902+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.857973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.857973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.17","version":"148'9","reported_seq":61,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181847+0000","last_change":"2024-02-19T21:05:19.238158+0000","last_active":"2024-02-19T21:05:33.181847+0000","last_peered":"2024-02-19T21:05:33.181847+0000","last_clean":"2024-02-19T21:05:33.181847+0000","last_became_active":"2024-02-19T21:04:54.646395+0000","last_became_peered":"2024-02-19T21:04:54.646395+0000","last_unstale":"2024-02-19T21:05:33.181847+0000","last_undegraded":"2024-02-19T21:05:33.181847+0000","last_fullsized":"2024-02-19T21:05:33.181847+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.14","version":"148'27","reported_seq":75,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.180685+0000","last_change":"2024-02-19T21:05:19.238575+0000","last_active":"2024-02-19T21:05:33.180685+0000","last_peered":"2024-02-19T21:05:33.180685+0000","last_clean":"2024-02-19T21:05:33.180685+0000","last_became_active":"2024-02-19T21:04:54.648513+0000","last_became_peered":"2024-02-19T21:04:54.648513+0000","last_unstale":"2024-02-19T21:05:33.180685+0000","last_undegraded":"2024-02-19T21:05:33.180685+0000","last_fullsized":"2024-02-19T21:05:33.180685+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"61.15","version":"148'27","reported_seq":76,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177725+0000","last_change":"2024-02-19T21:05:19.192425+0000","last_active":"2024-02-19T21:05:33.177725+0000","last_peered":"2024-02-19T21:05:33.177725+0000","last_clean":"2024-02-19T21:05:33.177725+0000","last_became_active":"2024-02-19T21:04:54.626719+0000","last_became_peered":"2024-02-19T21:04:54.626719+0000","last_unstale":"2024-02-19T21:05:33.177725+0000","last_undegraded":"2024-02-19T21:05:33.177725+0000","last_fullsized":"2024-02-19T21:05:33.177725+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"0'0","reported_seq":125,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171833+0000","last_change":"2024-02-19T21:05:22.210164+0000","last_active":"2024-02-19T21:05:33.171833+0000","last_peered":"2024-02-19T21:05:33.171833+0000","last_clean":"2024-02-19T21:05:33.171833+0000","last_became_active":"2024-02-19T21:05:22.209781+0000","last_became_peered":"2024-02-19T21:05:22.209781+0000","last_unstale":"2024-02-19T21:05:33.171833+0000","last_undegraded":"2024-02-19T21:05:33.171833+0000","last_fullsized":"2024-02-19T21:05:33.171833+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:58.273113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:58.273113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"61.1a","version":"148'45","reported_seq":96,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.175439+0000","last_change":"2024-02-19T21:05:19.189713+0000","last_active":"2024-02-19T21:05:33.175439+0000","last_peered":"2024-02-19T21:05:33.175439+0000","last_clean":"2024-02-19T21:05:33.175439+0000","last_became_active":"2024-02-19T21:04:54.634861+0000","last_became_peered":"2024-02-19T21:04:54.634861+0000","last_unstale":"2024-02-19T21:05:33.175439+0000","last_undegraded":"2024-02-19T21:05:33.175439+0000","last_fullsized":"2024-02-19T21:05:33.175439+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":260,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.188655+0000","last_change":"2024-02-19T21:05:22.208965+0000","last_active":"2024-02-19T21:05:33.188655+0000","last_peered":"2024-02-19T21:05:33.188655+0000","last_clean":"2024-02-19T21:05:33.188655+0000","last_became_active":"2024-02-19T21:05:22.208647+0000","last_became_peered":"2024-02-19T21:05:22.208647+0000","last_unstale":"2024-02-19T21:05:33.188655+0000","last_undegraded":"2024-02-19T21:05:33.188655+0000","last_fullsized":"2024-02-19T21:05:33.188655+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:18.242556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:04:18.242556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1b","version":"148'27","reported_seq":72,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168468+0000","last_change":"2024-02-19T21:05:19.286897+0000","last_active":"2024-02-19T21:05:33.168468+0000","last_peered":"2024-02-19T21:05:33.168468+0000","last_clean":"2024-02-19T21:05:33.168468+0000","last_became_active":"2024-02-19T21:04:54.658733+0000","last_became_peered":"2024-02-19T21:04:54.658733+0000","last_unstale":"2024-02-19T21:05:33.168468+0000","last_undegraded":"2024-02-19T21:05:33.168468+0000","last_fullsized":"2024-02-19T21:05:33.168468+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":124,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177352+0000","last_change":"2024-02-19T21:05:22.217801+0000","last_active":"2024-02-19T21:05:33.177352+0000","last_peered":"2024-02-19T21:05:33.177352+0000","last_clean":"2024-02-19T21:05:33.177352+0000","last_became_active":"2024-02-19T21:05:22.217505+0000","last_became_peered":"2024-02-19T21:05:22.217505+0000","last_unstale":"2024-02-19T21:05:33.177352+0000","last_undegraded":"2024-02-19T21:05:33.177352+0000","last_fullsized":"2024-02-19T21:05:33.177352+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:22.800578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:22.800578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.18","version":"148'54","reported_seq":100,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168863+0000","last_change":"2024-02-19T21:05:19.171216+0000","last_active":"2024-02-19T21:05:33.168863+0000","last_peered":"2024-02-19T21:05:33.168863+0000","last_clean":"2024-02-19T21:05:33.168863+0000","last_became_active":"2024-02-19T21:04:54.610608+0000","last_became_peered":"2024-02-19T21:04:54.610608+0000","last_unstale":"2024-02-19T21:05:33.168863+0000","last_undegraded":"2024-02-19T21:05:33.168863+0000","last_fullsized":"2024-02-19T21:05:33.168863+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":312,"num_objects":30,"num_object_clones":24,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.26","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177475+0000","last_change":"2024-02-19T21:05:22.217945+0000","last_active":"2024-02-19T21:05:33.177475+0000","last_peered":"2024-02-19T21:05:33.177475+0000","last_clean":"2024-02-19T21:05:33.177475+0000","last_became_active":"2024-02-19T21:05:22.216623+0000","last_became_peered":"2024-02-19T21:05:22.216623+0000","last_unstale":"2024-02-19T21:05:33.177475+0000","last_undegraded":"2024-02-19T21:05:33.177475+0000","last_fullsized":"2024-02-19T21:05:33.177475+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:21.205829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.205829+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.19","version":"148'36","reported_seq":89,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.176036+0000","last_change":"2024-02-19T21:05:19.172663+0000","last_active":"2024-02-19T21:05:33.176036+0000","last_peered":"2024-02-19T21:05:33.176036+0000","last_clean":"2024-02-19T21:05:33.176036+0000","last_became_active":"2024-02-19T21:04:54.608702+0000","last_became_peered":"2024-02-19T21:04:54.608702+0000","last_unstale":"2024-02-19T21:05:33.176036+0000","last_undegraded":"2024-02-19T21:05:33.176036+0000","last_fullsized":"2024-02-19T21:05:33.176036+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":208,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":137,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169269+0000","last_change":"2024-02-19T21:05:22.217241+0000","last_active":"2024-02-19T21:05:33.169269+0000","last_peered":"2024-02-19T21:05:33.169269+0000","last_clean":"2024-02-19T21:05:33.169269+0000","last_became_active":"2024-02-19T21:05:22.216738+0000","last_became_peered":"2024-02-19T21:05:22.216738+0000","last_unstale":"2024-02-19T21:05:33.169269+0000","last_undegraded":"2024-02-19T21:05:33.169269+0000","last_fullsized":"2024-02-19T21:05:33.169269+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:52.792935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:52.792935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.1e","version":"148'24","reported_seq":78,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177777+0000","last_change":"2024-02-19T21:05:21.208877+0000","last_active":"2024-02-19T21:05:33.177777+0000","last_peered":"2024-02-19T21:05:33.177777+0000","last_clean":"2024-02-19T21:05:33.177777+0000","last_became_active":"2024-02-19T21:04:54.629433+0000","last_became_peered":"2024-02-19T21:04:54.629433+0000","last_unstale":"2024-02-19T21:05:33.177777+0000","last_undegraded":"2024-02-19T21:05:33.177777+0000","last_fullsized":"2024-02-19T21:05:33.177777+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":117,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.20","version":"58'1","reported_seq":144,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.174579+0000","last_change":"2024-02-19T21:05:22.313264+0000","last_active":"2024-02-19T21:05:33.174579+0000","last_peered":"2024-02-19T21:05:33.174579+0000","last_clean":"2024-02-19T21:05:33.174579+0000","last_became_active":"2024-02-19T21:05:22.223725+0000","last_became_peered":"2024-02-19T21:05:22.223725+0000","last_unstale":"2024-02-19T21:05:33.174579+0000","last_undegraded":"2024-02-19T21:05:33.174579+0000","last_fullsized":"2024-02-19T21:05:33.174579+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:04:48.298487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:48.298487+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":437,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1f","version":"148'18","reported_seq":72,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172712+0000","last_change":"2024-02-19T21:05:19.308955+0000","last_active":"2024-02-19T21:05:33.172712+0000","last_peered":"2024-02-19T21:05:33.172712+0000","last_clean":"2024-02-19T21:05:33.172712+0000","last_became_active":"2024-02-19T21:04:54.671969+0000","last_became_peered":"2024-02-19T21:04:54.671969+0000","last_unstale":"2024-02-19T21:05:33.172712+0000","last_undegraded":"2024-02-19T21:05:33.172712+0000","last_fullsized":"2024-02-19T21:05:33.172712+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168239+0000","last_change":"2024-02-19T21:05:22.219273+0000","last_active":"2024-02-19T21:05:33.168239+0000","last_peered":"2024-02-19T21:05:33.168239+0000","last_clean":"2024-02-19T21:05:33.168239+0000","last_became_active":"2024-02-19T21:05:22.218880+0000","last_became_peered":"2024-02-19T21:05:22.218880+0000","last_unstale":"2024-02-19T21:05:33.168239+0000","last_undegraded":"2024-02-19T21:05:33.168239+0000","last_fullsized":"2024-02-19T21:05:33.168239+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:14.215016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:14.215016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.1c","version":"148'72","reported_seq":121,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.186901+0000","last_change":"2024-02-19T21:05:19.194170+0000","last_active":"2024-02-19T21:05:33.186901+0000","last_peered":"2024-02-19T21:05:33.186901+0000","last_clean":"2024-02-19T21:05:33.186901+0000","last_became_active":"2024-02-19T21:04:54.605898+0000","last_became_peered":"2024-02-19T21:04:54.605898+0000","last_unstale":"2024-02-19T21:05:33.186901+0000","last_undegraded":"2024-02-19T21:05:33.186901+0000","last_fullsized":"2024-02-19T21:05:33.186901+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":72,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":416,"num_objects":40,"num_object_clones":32,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":40,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":124,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.186555+0000","last_change":"2024-02-19T21:05:22.197328+0000","last_active":"2024-02-19T21:05:33.186555+0000","last_peered":"2024-02-19T21:05:33.186555+0000","last_clean":"2024-02-19T21:05:33.186555+0000","last_became_active":"2024-02-19T21:05:22.196948+0000","last_became_peered":"2024-02-19T21:05:22.196948+0000","last_unstale":"2024-02-19T21:05:33.186555+0000","last_undegraded":"2024-02-19T21:05:33.186555+0000","last_fullsized":"2024-02-19T21:05:33.186555+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:06.145411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:06.145411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1d","version":"148'18","reported_seq":71,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183856+0000","last_change":"2024-02-19T21:05:19.320001+0000","last_active":"2024-02-19T21:05:33.183856+0000","last_peered":"2024-02-19T21:05:33.183856+0000","last_clean":"2024-02-19T21:05:33.183856+0000","last_became_active":"2024-02-19T21:04:54.655665+0000","last_became_peered":"2024-02-19T21:04:54.655665+0000","last_unstale":"2024-02-19T21:05:33.183856+0000","last_undegraded":"2024-02-19T21:05:33.183856+0000","last_fullsized":"2024-02-19T21:05:33.183856+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":104,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"100'1","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170562+0000","last_change":"2024-02-19T21:04:39.739732+0000","last_active":"2024-02-19T21:05:33.170562+0000","last_peered":"2024-02-19T21:05:33.170562+0000","last_clean":"2024-02-19T21:05:33.170562+0000","last_became_active":"2024-02-19T21:04:39.739003+0000","last_became_peered":"2024-02-19T21:04:39.739003+0000","last_unstale":"2024-02-19T21:05:33.170562+0000","last_undegraded":"2024-02-19T21:05:33.170562+0000","last_fullsized":"2024-02-19T21:05:33.170562+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.187766+0000","last_change":"2024-02-19T21:04:39.640053+0000","last_active":"2024-02-19T21:05:33.187766+0000","last_peered":"2024-02-19T21:05:33.187766+0000","last_clean":"2024-02-19T21:05:33.187766+0000","last_became_active":"2024-02-19T21:04:39.639772+0000","last_became_peered":"2024-02-19T21:04:39.639772+0000","last_unstale":"2024-02-19T21:05:33.187766+0000","last_undegraded":"2024-02-19T21:05:33.187766+0000","last_fullsized":"2024-02-19T21:05:33.187766+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"97'9","reported_seq":171,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-02-19T21:05:34.172010+0000","last_change":"2024-02-19T21:04:24.046111+0000","last_active":"2024-02-19T21:05:34.172010+0000","last_peered":"2024-02-19T21:05:34.172010+0000","last_clean":"2024-02-19T21:05:34.172010+0000","last_became_active":"2024-02-19T21:01:20.288720+0000","last_became_peered":"2024-02-19T21:01:20.288720+0000","last_unstale":"2024-02-19T21:05:34.172010+0000","last_undegraded":"2024-02-19T21:05:34.172010+0000","last_fullsized":"2024-02-19T21:05:34.172010+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:04:24.046034+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":141,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.174799+0000","last_change":"2024-02-19T21:04:56.123170+0000","last_active":"2024-02-19T21:05:33.174799+0000","last_peered":"2024-02-19T21:05:33.174799+0000","last_clean":"2024-02-19T21:05:33.174799+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:05:33.174799+0000","last_undegraded":"2024-02-19T21:05:33.174799+0000","last_fullsized":"2024-02-19T21:05:33.174799+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:04:56.123096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.123096+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171486+0000","last_change":"2024-02-19T21:04:11.889869+0000","last_active":"2024-02-19T21:05:33.171486+0000","last_peered":"2024-02-19T21:05:33.171486+0000","last_clean":"2024-02-19T21:05:33.171486+0000","last_became_active":"2024-02-19T21:01:16.271911+0000","last_became_peered":"2024-02-19T21:01:16.271911+0000","last_unstale":"2024-02-19T21:05:33.171486+0000","last_undegraded":"2024-02-19T21:05:33.171486+0000","last_fullsized":"2024-02-19T21:05:33.171486+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.889778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:37.258580+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.889778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":104,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181829+0000","last_change":"2024-02-19T21:03:10.017843+0000","last_active":"2024-02-19T21:05:33.181829+0000","last_peered":"2024-02-19T21:05:33.181829+0000","last_clean":"2024-02-19T21:05:33.181829+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:05:33.181829+0000","last_undegraded":"2024-02-19T21:05:33.181829+0000","last_fullsized":"2024-02-19T21:05:33.181829+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:10.017764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:10.017764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171473+0000","last_change":"2024-02-19T21:04:39.703479+0000","last_active":"2024-02-19T21:05:33.171473+0000","last_peered":"2024-02-19T21:05:33.171473+0000","last_clean":"2024-02-19T21:05:33.171473+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:05:33.171473+0000","last_undegraded":"2024-02-19T21:05:33.171473+0000","last_fullsized":"2024-02-19T21:05:33.171473+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.13","version":"102'3","reported_seq":73,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168965+0000","last_change":"2024-02-19T21:04:39.785733+0000","last_active":"2024-02-19T21:05:33.168965+0000","last_peered":"2024-02-19T21:05:33.168965+0000","last_clean":"2024-02-19T21:05:33.168965+0000","last_became_active":"2024-02-19T21:04:39.784009+0000","last_became_peered":"2024-02-19T21:04:39.784009+0000","last_unstale":"2024-02-19T21:05:33.168965+0000","last_undegraded":"2024-02-19T21:05:33.168965+0000","last_fullsized":"2024-02-19T21:05:33.168965+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165843,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":24579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185031+0000","last_change":"2024-02-19T21:04:39.772957+0000","last_active":"2024-02-19T21:05:33.185031+0000","last_peered":"2024-02-19T21:05:33.185031+0000","last_clean":"2024-02-19T21:05:33.185031+0000","last_became_active":"2024-02-19T21:04:39.771391+0000","last_became_peered":"2024-02-19T21:04:39.771391+0000","last_unstale":"2024-02-19T21:05:33.185031+0000","last_undegraded":"2024-02-19T21:05:33.185031+0000","last_fullsized":"2024-02-19T21:05:33.185031+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":129,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181053+0000","last_change":"2024-02-19T21:04:56.263016+0000","last_active":"2024-02-19T21:05:33.181053+0000","last_peered":"2024-02-19T21:05:33.181053+0000","last_clean":"2024-02-19T21:05:33.181053+0000","last_became_active":"2024-02-19T21:01:18.288322+0000","last_became_peered":"2024-02-19T21:01:18.288322+0000","last_unstale":"2024-02-19T21:05:33.181053+0000","last_undegraded":"2024-02-19T21:05:33.181053+0000","last_fullsized":"2024-02-19T21:05:33.181053+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:56.262927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.262927+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":129,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.184491+0000","last_change":"2024-02-19T21:05:10.764055+0000","last_active":"2024-02-19T21:05:33.184491+0000","last_peered":"2024-02-19T21:05:33.184491+0000","last_clean":"2024-02-19T21:05:33.184491+0000","last_became_active":"2024-02-19T21:04:12.926645+0000","last_became_peered":"2024-02-19T21:04:12.926645+0000","last_unstale":"2024-02-19T21:05:33.184491+0000","last_undegraded":"2024-02-19T21:05:33.184491+0000","last_fullsized":"2024-02-19T21:05:33.184491+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:10.763889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:10.763889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170986+0000","last_change":"2024-02-19T21:04:17.831164+0000","last_active":"2024-02-19T21:05:33.170986+0000","last_peered":"2024-02-19T21:05:33.170986+0000","last_clean":"2024-02-19T21:05:33.170986+0000","last_became_active":"2024-02-19T21:01:20.290566+0000","last_became_peered":"2024-02-19T21:01:20.290566+0000","last_unstale":"2024-02-19T21:05:33.170986+0000","last_undegraded":"2024-02-19T21:05:33.170986+0000","last_fullsized":"2024-02-19T21:05:33.170986+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:17.831065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:17.831065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177550+0000","last_change":"2024-02-19T21:04:21.659690+0000","last_active":"2024-02-19T21:05:33.177550+0000","last_peered":"2024-02-19T21:05:33.177550+0000","last_clean":"2024-02-19T21:05:33.177550+0000","last_became_active":"2024-02-19T21:01:22.297998+0000","last_became_peered":"2024-02-19T21:01:22.297998+0000","last_unstale":"2024-02-19T21:05:33.177550+0000","last_undegraded":"2024-02-19T21:05:33.177550+0000","last_fullsized":"2024-02-19T21:05:33.177550+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:21.659616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.659616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183342+0000","last_change":"2024-02-19T21:04:39.765776+0000","last_active":"2024-02-19T21:05:33.183342+0000","last_peered":"2024-02-19T21:05:33.183342+0000","last_clean":"2024-02-19T21:05:33.183342+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:05:33.183342+0000","last_undegraded":"2024-02-19T21:05:33.183342+0000","last_fullsized":"2024-02-19T21:05:33.183342+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.12","version":"101'2","reported_seq":67,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.176691+0000","last_change":"2024-02-19T21:04:39.785340+0000","last_active":"2024-02-19T21:05:33.176691+0000","last_peered":"2024-02-19T21:05:33.176691+0000","last_clean":"2024-02-19T21:05:33.176691+0000","last_became_active":"2024-02-19T21:04:39.784930+0000","last_became_peered":"2024-02-19T21:04:39.784930+0000","last_unstale":"2024-02-19T21:05:33.176691+0000","last_undegraded":"2024-02-19T21:05:33.176691+0000","last_fullsized":"2024-02-19T21:05:33.176691+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"100'9","reported_seq":73,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181602+0000","last_change":"2024-02-19T21:04:39.731654+0000","last_active":"2024-02-19T21:05:33.181602+0000","last_peered":"2024-02-19T21:05:33.181602+0000","last_clean":"2024-02-19T21:05:33.181602+0000","last_became_active":"2024-02-19T21:04:39.731439+0000","last_became_peered":"2024-02-19T21:04:39.731439+0000","last_unstale":"2024-02-19T21:05:33.181602+0000","last_undegraded":"2024-02-19T21:05:33.181602+0000","last_fullsized":"2024-02-19T21:05:33.181602+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":144,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185977+0000","last_change":"2024-02-19T21:04:20.836693+0000","last_active":"2024-02-19T21:05:33.185977+0000","last_peered":"2024-02-19T21:05:33.185977+0000","last_clean":"2024-02-19T21:05:33.185977+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:05:33.185977+0000","last_undegraded":"2024-02-19T21:05:33.185977+0000","last_fullsized":"2024-02-19T21:05:33.185977+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:04:20.836609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:20.836609+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171066+0000","last_change":"2024-02-19T21:04:03.767826+0000","last_active":"2024-02-19T21:05:33.171066+0000","last_peered":"2024-02-19T21:05:33.171066+0000","last_clean":"2024-02-19T21:05:33.171066+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:05:33.171066+0000","last_undegraded":"2024-02-19T21:05:33.171066+0000","last_fullsized":"2024-02-19T21:05:33.171066+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:03.767713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:03.767713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168785+0000","last_change":"2024-02-19T21:04:51.822914+0000","last_active":"2024-02-19T21:05:33.168785+0000","last_peered":"2024-02-19T21:05:33.168785+0000","last_clean":"2024-02-19T21:05:33.168785+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:05:33.168785+0000","last_undegraded":"2024-02-19T21:05:33.168785+0000","last_fullsized":"2024-02-19T21:05:33.168785+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:51.822845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:51.822845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":108,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.175346+0000","last_change":"2024-02-19T21:05:05.187540+0000","last_active":"2024-02-19T21:05:33.175346+0000","last_peered":"2024-02-19T21:05:33.175346+0000","last_clean":"2024-02-19T21:05:33.175346+0000","last_became_active":"2024-02-19T21:01:22.291759+0000","last_became_peered":"2024-02-19T21:01:22.291759+0000","last_unstale":"2024-02-19T21:05:33.175346+0000","last_undegraded":"2024-02-19T21:05:33.175346+0000","last_fullsized":"2024-02-19T21:05:33.175346+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:05.187474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:05.187474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.178312+0000","last_change":"2024-02-19T21:04:39.620073+0000","last_active":"2024-02-19T21:05:33.178312+0000","last_peered":"2024-02-19T21:05:33.178312+0000","last_clean":"2024-02-19T21:05:33.178312+0000","last_became_active":"2024-02-19T21:04:39.618614+0000","last_became_peered":"2024-02-19T21:04:39.618614+0000","last_unstale":"2024-02-19T21:05:33.178312+0000","last_undegraded":"2024-02-19T21:05:33.178312+0000","last_fullsized":"2024-02-19T21:05:33.178312+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.15","version":"132'1","reported_seq":67,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.184579+0000","last_change":"2024-02-19T21:04:39.690722+0000","last_active":"2024-02-19T21:05:33.184579+0000","last_peered":"2024-02-19T21:05:33.184579+0000","last_clean":"2024-02-19T21:05:33.184579+0000","last_became_active":"2024-02-19T21:04:39.689794+0000","last_became_peered":"2024-02-19T21:04:39.689794+0000","last_unstale":"2024-02-19T21:05:33.184579+0000","last_undegraded":"2024-02-19T21:05:33.184579+0000","last_fullsized":"2024-02-19T21:05:33.184579+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165828,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":168,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172534+0000","last_change":"2024-02-19T21:04:03.152903+0000","last_active":"2024-02-19T21:05:33.172534+0000","last_peered":"2024-02-19T21:05:33.172534+0000","last_clean":"2024-02-19T21:05:33.172534+0000","last_became_active":"2024-02-19T21:01:30.923214+0000","last_became_peered":"2024-02-19T21:01:30.923214+0000","last_unstale":"2024-02-19T21:05:33.172534+0000","last_undegraded":"2024-02-19T21:05:33.172534+0000","last_fullsized":"2024-02-19T21:05:33.172534+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:04:03.152817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:03.152817+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":120,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.178282+0000","last_change":"2024-02-19T21:04:21.952701+0000","last_active":"2024-02-19T21:05:33.178282+0000","last_peered":"2024-02-19T21:05:33.178282+0000","last_clean":"2024-02-19T21:05:33.178282+0000","last_became_active":"2024-02-19T21:01:16.274937+0000","last_became_peered":"2024-02-19T21:01:16.274937+0000","last_unstale":"2024-02-19T21:05:33.178282+0000","last_undegraded":"2024-02-19T21:05:33.178282+0000","last_fullsized":"2024-02-19T21:05:33.178282+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:04:21.952586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.952586+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185192+0000","last_change":"2024-02-19T21:04:04.386811+0000","last_active":"2024-02-19T21:05:33.185192+0000","last_peered":"2024-02-19T21:05:33.185192+0000","last_clean":"2024-02-19T21:05:33.185192+0000","last_became_active":"2024-02-19T21:01:20.290687+0000","last_became_peered":"2024-02-19T21:01:20.290687+0000","last_unstale":"2024-02-19T21:05:33.185192+0000","last_undegraded":"2024-02-19T21:05:33.185192+0000","last_fullsized":"2024-02-19T21:05:33.185192+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:04.386735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:04.386735+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.173394+0000","last_change":"2024-02-19T21:04:19.845711+0000","last_active":"2024-02-19T21:05:33.173394+0000","last_peered":"2024-02-19T21:05:33.173394+0000","last_clean":"2024-02-19T21:05:33.173394+0000","last_became_active":"2024-02-19T21:01:22.298941+0000","last_became_peered":"2024-02-19T21:01:22.298941+0000","last_unstale":"2024-02-19T21:05:33.173394+0000","last_undegraded":"2024-02-19T21:05:33.173394+0000","last_fullsized":"2024-02-19T21:05:33.173394+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.845609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"100'5","reported_seq":69,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181469+0000","last_change":"2024-02-19T21:04:39.781921+0000","last_active":"2024-02-19T21:05:33.181469+0000","last_peered":"2024-02-19T21:05:33.181469+0000","last_clean":"2024-02-19T21:05:33.181469+0000","last_became_active":"2024-02-19T21:04:39.778460+0000","last_became_peered":"2024-02-19T21:04:39.778460+0000","last_unstale":"2024-02-19T21:05:33.181469+0000","last_undegraded":"2024-02-19T21:05:33.181469+0000","last_fullsized":"2024-02-19T21:05:33.181469+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170405+0000","last_change":"2024-02-19T21:04:39.699479+0000","last_active":"2024-02-19T21:05:33.170405+0000","last_peered":"2024-02-19T21:05:33.170405+0000","last_clean":"2024-02-19T21:05:33.170405+0000","last_became_active":"2024-02-19T21:04:39.698684+0000","last_became_peered":"2024-02-19T21:04:39.698684+0000","last_unstale":"2024-02-19T21:05:33.170405+0000","last_undegraded":"2024-02-19T21:05:33.170405+0000","last_fullsized":"2024-02-19T21:05:33.170405+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.14","version":"100'6","reported_seq":70,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169782+0000","last_change":"2024-02-19T21:04:39.757025+0000","last_active":"2024-02-19T21:05:33.169782+0000","last_peered":"2024-02-19T21:05:33.169782+0000","last_clean":"2024-02-19T21:05:33.169782+0000","last_became_active":"2024-02-19T21:04:39.755747+0000","last_became_peered":"2024-02-19T21:04:39.755747+0000","last_unstale":"2024-02-19T21:05:33.169782+0000","last_undegraded":"2024-02-19T21:05:33.169782+0000","last_fullsized":"2024-02-19T21:05:33.169782+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1024,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"82'15","reported_seq":133,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171932+0000","last_change":"2024-02-19T21:04:49.285676+0000","last_active":"2024-02-19T21:05:33.171932+0000","last_peered":"2024-02-19T21:05:33.171932+0000","last_clean":"2024-02-19T21:05:33.171932+0000","last_became_active":"2024-02-19T21:01:18.282805+0000","last_became_peered":"2024-02-19T21:01:18.282805+0000","last_unstale":"2024-02-19T21:05:33.171932+0000","last_undegraded":"2024-02-19T21:05:33.171932+0000","last_fullsized":"2024-02-19T21:05:33.171932+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:04:49.285599+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.285599+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183206+0000","last_change":"2024-02-19T21:04:11.303670+0000","last_active":"2024-02-19T21:05:33.183206+0000","last_peered":"2024-02-19T21:05:33.183206+0000","last_clean":"2024-02-19T21:05:33.183206+0000","last_became_active":"2024-02-19T21:01:16.274086+0000","last_became_peered":"2024-02-19T21:01:16.274086+0000","last_unstale":"2024-02-19T21:05:33.183206+0000","last_undegraded":"2024-02-19T21:05:33.183206+0000","last_fullsized":"2024-02-19T21:05:33.183206+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.303572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.303572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":171,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-02-19T21:05:34.198074+0000","last_change":"2024-02-19T21:04:07.809065+0000","last_active":"2024-02-19T21:05:34.198074+0000","last_peered":"2024-02-19T21:05:34.198074+0000","last_clean":"2024-02-19T21:05:34.198074+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:05:34.198074+0000","last_undegraded":"2024-02-19T21:05:34.198074+0000","last_fullsized":"2024-02-19T21:05:34.198074+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:07.808944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:07.808944+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.173128+0000","last_change":"2024-02-19T21:04:21.031646+0000","last_active":"2024-02-19T21:05:33.173128+0000","last_peered":"2024-02-19T21:05:33.173128+0000","last_clean":"2024-02-19T21:05:33.173128+0000","last_became_active":"2024-02-19T21:01:22.299993+0000","last_became_peered":"2024-02-19T21:01:22.299993+0000","last_unstale":"2024-02-19T21:05:33.173128+0000","last_undegraded":"2024-02-19T21:05:33.173128+0000","last_fullsized":"2024-02-19T21:05:33.173128+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.031577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"100'5","reported_seq":69,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170102+0000","last_change":"2024-02-19T21:04:39.746708+0000","last_active":"2024-02-19T21:05:33.170102+0000","last_peered":"2024-02-19T21:05:33.170102+0000","last_clean":"2024-02-19T21:05:33.170102+0000","last_became_active":"2024-02-19T21:04:39.742728+0000","last_became_peered":"2024-02-19T21:04:39.742728+0000","last_unstale":"2024-02-19T21:05:33.170102+0000","last_undegraded":"2024-02-19T21:05:33.170102+0000","last_fullsized":"2024-02-19T21:05:33.170102+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.186168+0000","last_change":"2024-02-19T21:04:39.766043+0000","last_active":"2024-02-19T21:05:33.186168+0000","last_peered":"2024-02-19T21:05:33.186168+0000","last_clean":"2024-02-19T21:05:33.186168+0000","last_became_active":"2024-02-19T21:04:39.765782+0000","last_became_peered":"2024-02-19T21:04:39.765782+0000","last_unstale":"2024-02-19T21:05:33.186168+0000","last_undegraded":"2024-02-19T21:05:33.186168+0000","last_fullsized":"2024-02-19T21:05:33.186168+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"102'7","reported_seq":78,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.178285+0000","last_change":"2024-02-19T21:04:39.659021+0000","last_active":"2024-02-19T21:05:33.178285+0000","last_peered":"2024-02-19T21:05:33.178285+0000","last_clean":"2024-02-19T21:05:33.178285+0000","last_became_active":"2024-02-19T21:04:39.658450+0000","last_became_peered":"2024-02-19T21:04:39.658450+0000","last_unstale":"2024-02-19T21:05:33.178285+0000","last_undegraded":"2024-02-19T21:05:33.178285+0000","last_fullsized":"2024-02-19T21:05:33.178285+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165900,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"100'5","reported_seq":69,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.186788+0000","last_change":"2024-02-19T21:04:39.676464+0000","last_active":"2024-02-19T21:05:33.186788+0000","last_peered":"2024-02-19T21:05:33.186788+0000","last_clean":"2024-02-19T21:05:33.186788+0000","last_became_active":"2024-02-19T21:04:39.675580+0000","last_became_peered":"2024-02-19T21:04:39.675580+0000","last_unstale":"2024-02-19T21:05:33.186788+0000","last_undegraded":"2024-02-19T21:05:33.186788+0000","last_fullsized":"2024-02-19T21:05:33.186788+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":138,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185118+0000","last_change":"2024-02-19T21:05:03.210019+0000","last_active":"2024-02-19T21:05:33.185118+0000","last_peered":"2024-02-19T21:05:33.185118+0000","last_clean":"2024-02-19T21:05:33.185118+0000","last_became_active":"2024-02-19T21:01:18.285534+0000","last_became_peered":"2024-02-19T21:01:18.285534+0000","last_unstale":"2024-02-19T21:05:33.185118+0000","last_undegraded":"2024-02-19T21:05:33.185118+0000","last_fullsized":"2024-02-19T21:05:33.185118+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:05:03.209876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.209876+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":124,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181750+0000","last_change":"2024-02-19T21:05:20.189104+0000","last_active":"2024-02-19T21:05:33.181750+0000","last_peered":"2024-02-19T21:05:33.181750+0000","last_clean":"2024-02-19T21:05:33.181750+0000","last_became_active":"2024-02-19T21:05:20.188821+0000","last_became_peered":"2024-02-19T21:05:20.188821+0000","last_unstale":"2024-02-19T21:05:33.181750+0000","last_undegraded":"2024-02-19T21:05:33.181750+0000","last_fullsized":"2024-02-19T21:05:33.181750+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:12.305886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:12.305886+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":171,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-02-19T21:05:34.169886+0000","last_change":"2024-02-19T21:04:16.059715+0000","last_active":"2024-02-19T21:05:34.169886+0000","last_peered":"2024-02-19T21:05:34.169886+0000","last_clean":"2024-02-19T21:05:34.169886+0000","last_became_active":"2024-02-19T21:01:20.293341+0000","last_became_peered":"2024-02-19T21:01:20.293341+0000","last_unstale":"2024-02-19T21:05:34.169886+0000","last_undegraded":"2024-02-19T21:05:34.169886+0000","last_fullsized":"2024-02-19T21:05:34.169886+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:16.059648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.059648+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183792+0000","last_change":"2024-02-19T21:04:08.185818+0000","last_active":"2024-02-19T21:05:33.183792+0000","last_peered":"2024-02-19T21:05:33.183792+0000","last_clean":"2024-02-19T21:05:33.183792+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:05:33.183792+0000","last_undegraded":"2024-02-19T21:05:33.183792+0000","last_fullsized":"2024-02-19T21:05:33.183792+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.185717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.185717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.182475+0000","last_change":"2024-02-19T21:04:39.657326+0000","last_active":"2024-02-19T21:05:33.182475+0000","last_peered":"2024-02-19T21:05:33.182475+0000","last_clean":"2024-02-19T21:05:33.182475+0000","last_became_active":"2024-02-19T21:04:39.657052+0000","last_became_peered":"2024-02-19T21:04:39.657052+0000","last_unstale":"2024-02-19T21:05:33.182475+0000","last_undegraded":"2024-02-19T21:05:33.182475+0000","last_fullsized":"2024-02-19T21:05:33.182475+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.182204+0000","last_change":"2024-02-19T21:04:39.800157+0000","last_active":"2024-02-19T21:05:33.182204+0000","last_peered":"2024-02-19T21:05:33.182204+0000","last_clean":"2024-02-19T21:05:33.182204+0000","last_became_active":"2024-02-19T21:04:39.798967+0000","last_became_peered":"2024-02-19T21:04:39.798967+0000","last_unstale":"2024-02-19T21:05:33.182204+0000","last_undegraded":"2024-02-19T21:05:33.182204+0000","last_fullsized":"2024-02-19T21:05:33.182204+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170988+0000","last_change":"2024-02-19T21:04:39.662708+0000","last_active":"2024-02-19T21:05:33.170988+0000","last_peered":"2024-02-19T21:05:33.170988+0000","last_clean":"2024-02-19T21:05:33.170988+0000","last_became_active":"2024-02-19T21:04:39.662412+0000","last_became_peered":"2024-02-19T21:04:39.662412+0000","last_unstale":"2024-02-19T21:05:33.170988+0000","last_undegraded":"2024-02-19T21:05:33.170988+0000","last_fullsized":"2024-02-19T21:05:33.170988+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":132,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172809+0000","last_change":"2024-02-19T21:03:35.086667+0000","last_active":"2024-02-19T21:05:33.172809+0000","last_peered":"2024-02-19T21:05:33.172809+0000","last_clean":"2024-02-19T21:05:33.172809+0000","last_became_active":"2024-02-19T21:01:18.285670+0000","last_became_peered":"2024-02-19T21:01:18.285670+0000","last_unstale":"2024-02-19T21:05:33.172809+0000","last_undegraded":"2024-02-19T21:05:33.172809+0000","last_fullsized":"2024-02-19T21:05:33.172809+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:03:35.086579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:35.086579+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":123,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.175567+0000","last_change":"2024-02-19T21:05:20.211931+0000","last_active":"2024-02-19T21:05:33.175567+0000","last_peered":"2024-02-19T21:05:33.175567+0000","last_clean":"2024-02-19T21:05:33.175567+0000","last_became_active":"2024-02-19T21:05:20.211711+0000","last_became_peered":"2024-02-19T21:05:20.211711+0000","last_unstale":"2024-02-19T21:05:33.175567+0000","last_undegraded":"2024-02-19T21:05:33.175567+0000","last_fullsized":"2024-02-19T21:05:33.175567+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.857973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.857973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183717+0000","last_change":"2024-02-19T21:04:16.217810+0000","last_active":"2024-02-19T21:05:33.183717+0000","last_peered":"2024-02-19T21:05:33.183717+0000","last_clean":"2024-02-19T21:05:33.183717+0000","last_became_active":"2024-02-19T21:01:20.289375+0000","last_became_peered":"2024-02-19T21:01:20.289375+0000","last_unstale":"2024-02-19T21:05:33.183717+0000","last_undegraded":"2024-02-19T21:05:33.183717+0000","last_fullsized":"2024-02-19T21:05:33.183717+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:16.217744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.217744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":104,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169464+0000","last_change":"2024-02-19T21:03:23.656785+0000","last_active":"2024-02-19T21:05:33.169464+0000","last_peered":"2024-02-19T21:05:33.169464+0000","last_clean":"2024-02-19T21:05:33.169464+0000","last_became_active":"2024-02-19T21:01:22.607007+0000","last_became_peered":"2024-02-19T21:01:22.607007+0000","last_unstale":"2024-02-19T21:05:33.169464+0000","last_undegraded":"2024-02-19T21:05:33.169464+0000","last_fullsized":"2024-02-19T21:05:33.169464+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:23.656688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:23.656688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.173727+0000","last_change":"2024-02-19T21:04:39.744505+0000","last_active":"2024-02-19T21:05:33.173727+0000","last_peered":"2024-02-19T21:05:33.173727+0000","last_clean":"2024-02-19T21:05:33.173727+0000","last_became_active":"2024-02-19T21:04:39.744231+0000","last_became_peered":"2024-02-19T21:04:39.744231+0000","last_unstale":"2024-02-19T21:05:33.173727+0000","last_undegraded":"2024-02-19T21:05:33.173727+0000","last_fullsized":"2024-02-19T21:05:33.173727+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177533+0000","last_change":"2024-02-19T21:04:39.640944+0000","last_active":"2024-02-19T21:05:33.177533+0000","last_peered":"2024-02-19T21:05:33.177533+0000","last_clean":"2024-02-19T21:05:33.177533+0000","last_became_active":"2024-02-19T21:04:39.640667+0000","last_became_peered":"2024-02-19T21:04:39.640667+0000","last_unstale":"2024-02-19T21:05:33.177533+0000","last_undegraded":"2024-02-19T21:05:33.177533+0000","last_fullsized":"2024-02-19T21:05:33.177533+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.d","version":"100'5","reported_seq":69,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183374+0000","last_change":"2024-02-19T21:04:39.631816+0000","last_active":"2024-02-19T21:05:33.183374+0000","last_peered":"2024-02-19T21:05:33.183374+0000","last_clean":"2024-02-19T21:05:33.183374+0000","last_became_active":"2024-02-19T21:04:39.631449+0000","last_became_peered":"2024-02-19T21:04:39.631449+0000","last_unstale":"2024-02-19T21:05:33.183374+0000","last_undegraded":"2024-02-19T21:05:33.183374+0000","last_fullsized":"2024-02-19T21:05:33.183374+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":171,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-02-19T21:05:34.171837+0000","last_change":"2024-02-19T21:03:33.736454+0000","last_active":"2024-02-19T21:05:34.171837+0000","last_peered":"2024-02-19T21:05:34.171837+0000","last_clean":"2024-02-19T21:05:34.171837+0000","last_became_active":"2024-02-19T21:01:20.286505+0000","last_became_peered":"2024-02-19T21:01:20.286505+0000","last_unstale":"2024-02-19T21:05:34.171837+0000","last_undegraded":"2024-02-19T21:05:34.171837+0000","last_fullsized":"2024-02-19T21:05:34.171837+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:03:33.736366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:33.736366+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":134,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170883+0000","last_change":"2024-02-19T21:04:50.266262+0000","last_active":"2024-02-19T21:05:33.170883+0000","last_peered":"2024-02-19T21:05:33.170883+0000","last_clean":"2024-02-19T21:05:33.170883+0000","last_became_active":"2024-02-19T21:01:18.282702+0000","last_became_peered":"2024-02-19T21:01:18.282702+0000","last_unstale":"2024-02-19T21:05:33.170883+0000","last_undegraded":"2024-02-19T21:05:33.170883+0000","last_fullsized":"2024-02-19T21:05:33.170883+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:04:50.266152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:50.266152+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":124,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.190259+0000","last_change":"2024-02-19T21:05:20.224978+0000","last_active":"2024-02-19T21:05:33.190259+0000","last_peered":"2024-02-19T21:05:33.190259+0000","last_clean":"2024-02-19T21:05:33.190259+0000","last_became_active":"2024-02-19T21:05:20.224706+0000","last_became_peered":"2024-02-19T21:05:20.224706+0000","last_unstale":"2024-02-19T21:05:33.190259+0000","last_undegraded":"2024-02-19T21:05:33.190259+0000","last_fullsized":"2024-02-19T21:05:33.190259+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:22.800578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:22.800578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":110,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.178610+0000","last_change":"2024-02-19T21:05:03.248574+0000","last_active":"2024-02-19T21:05:33.178610+0000","last_peered":"2024-02-19T21:05:33.178610+0000","last_clean":"2024-02-19T21:05:33.178610+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:05:33.178610+0000","last_undegraded":"2024-02-19T21:05:33.178610+0000","last_fullsized":"2024-02-19T21:05:33.178610+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.248506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183710+0000","last_change":"2024-02-19T21:04:39.659690+0000","last_active":"2024-02-19T21:05:33.183710+0000","last_peered":"2024-02-19T21:05:33.183710+0000","last_clean":"2024-02-19T21:05:33.183710+0000","last_became_active":"2024-02-19T21:04:39.658536+0000","last_became_peered":"2024-02-19T21:04:39.658536+0000","last_unstale":"2024-02-19T21:05:33.183710+0000","last_undegraded":"2024-02-19T21:05:33.183710+0000","last_fullsized":"2024-02-19T21:05:33.183710+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.18","version":"102'4","reported_seq":75,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169316+0000","last_change":"2024-02-19T21:04:39.748257+0000","last_active":"2024-02-19T21:05:33.169316+0000","last_peered":"2024-02-19T21:05:33.169316+0000","last_clean":"2024-02-19T21:05:33.169316+0000","last_became_active":"2024-02-19T21:04:39.747589+0000","last_became_peered":"2024-02-19T21:04:39.747589+0000","last_unstale":"2024-02-19T21:05:33.169316+0000","last_undegraded":"2024-02-19T21:05:33.169316+0000","last_fullsized":"2024-02-19T21:05:33.169316+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50331664,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":4,"num_write_kb":51902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181003+0000","last_change":"2024-02-19T21:04:39.769987+0000","last_active":"2024-02-19T21:05:33.181003+0000","last_peered":"2024-02-19T21:05:33.181003+0000","last_clean":"2024-02-19T21:05:33.181003+0000","last_became_active":"2024-02-19T21:04:39.767371+0000","last_became_peered":"2024-02-19T21:04:39.767371+0000","last_unstale":"2024-02-19T21:05:33.181003+0000","last_undegraded":"2024-02-19T21:05:33.181003+0000","last_fullsized":"2024-02-19T21:05:33.181003+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171036+0000","last_change":"2024-02-19T21:05:12.318453+0000","last_active":"2024-02-19T21:05:33.171036+0000","last_peered":"2024-02-19T21:05:33.171036+0000","last_clean":"2024-02-19T21:05:33.171036+0000","last_became_active":"2024-02-19T21:01:20.291079+0000","last_became_peered":"2024-02-19T21:01:20.291079+0000","last_unstale":"2024-02-19T21:05:33.171036+0000","last_undegraded":"2024-02-19T21:05:33.171036+0000","last_fullsized":"2024-02-19T21:05:33.171036+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:05:12.318389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":145,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169402+0000","last_change":"2024-02-19T21:04:08.837761+0000","last_active":"2024-02-19T21:05:33.169402+0000","last_peered":"2024-02-19T21:05:33.169402+0000","last_clean":"2024-02-19T21:05:33.169402+0000","last_became_active":"2024-02-19T21:01:18.293812+0000","last_became_peered":"2024-02-19T21:01:18.293812+0000","last_unstale":"2024-02-19T21:05:33.169402+0000","last_undegraded":"2024-02-19T21:05:33.169402+0000","last_fullsized":"2024-02-19T21:05:33.169402+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:04:08.837652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.837652+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":124,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183851+0000","last_change":"2024-02-19T21:05:20.189334+0000","last_active":"2024-02-19T21:05:33.183851+0000","last_peered":"2024-02-19T21:05:33.183851+0000","last_clean":"2024-02-19T21:05:33.183851+0000","last_became_active":"2024-02-19T21:05:20.189051+0000","last_became_peered":"2024-02-19T21:05:20.189051+0000","last_unstale":"2024-02-19T21:05:33.183851+0000","last_undegraded":"2024-02-19T21:05:33.183851+0000","last_fullsized":"2024-02-19T21:05:33.183851+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:21.205829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.205829+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":104,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171524+0000","last_change":"2024-02-19T21:03:17.088777+0000","last_active":"2024-02-19T21:05:33.171524+0000","last_peered":"2024-02-19T21:05:33.171524+0000","last_clean":"2024-02-19T21:05:33.171524+0000","last_became_active":"2024-02-19T21:01:22.297707+0000","last_became_peered":"2024-02-19T21:01:22.297707+0000","last_unstale":"2024-02-19T21:05:33.171524+0000","last_undegraded":"2024-02-19T21:05:33.171524+0000","last_fullsized":"2024-02-19T21:05:33.171524+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:17.088695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:17.088695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181077+0000","last_change":"2024-02-19T21:04:39.770138+0000","last_active":"2024-02-19T21:05:33.181077+0000","last_peered":"2024-02-19T21:05:33.181077+0000","last_clean":"2024-02-19T21:05:33.181077+0000","last_became_active":"2024-02-19T21:04:39.767723+0000","last_became_peered":"2024-02-19T21:04:39.767723+0000","last_unstale":"2024-02-19T21:05:33.181077+0000","last_undegraded":"2024-02-19T21:05:33.181077+0000","last_fullsized":"2024-02-19T21:05:33.181077+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"100'1","reported_seq":65,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168711+0000","last_change":"2024-02-19T21:04:39.786049+0000","last_active":"2024-02-19T21:05:33.168711+0000","last_peered":"2024-02-19T21:05:33.168711+0000","last_clean":"2024-02-19T21:05:33.168711+0000","last_became_active":"2024-02-19T21:04:39.784742+0000","last_became_peered":"2024-02-19T21:04:39.784742+0000","last_unstale":"2024-02-19T21:05:33.168711+0000","last_undegraded":"2024-02-19T21:05:33.168711+0000","last_fullsized":"2024-02-19T21:05:33.168711+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168581+0000","last_change":"2024-02-19T21:04:39.722429+0000","last_active":"2024-02-19T21:05:33.168581+0000","last_peered":"2024-02-19T21:05:33.168581+0000","last_clean":"2024-02-19T21:05:33.168581+0000","last_became_active":"2024-02-19T21:04:39.711377+0000","last_became_peered":"2024-02-19T21:04:39.711377+0000","last_unstale":"2024-02-19T21:05:33.168581+0000","last_undegraded":"2024-02-19T21:05:33.168581+0000","last_fullsized":"2024-02-19T21:05:33.168581+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":116,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.186134+0000","last_change":"2024-02-19T21:04:56.720699+0000","last_active":"2024-02-19T21:05:33.186134+0000","last_peered":"2024-02-19T21:05:33.186134+0000","last_clean":"2024-02-19T21:05:33.186134+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:05:33.186134+0000","last_undegraded":"2024-02-19T21:05:33.186134+0000","last_fullsized":"2024-02-19T21:05:33.186134+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:56.720607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.720607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":149,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.180889+0000","last_change":"2024-02-19T21:04:07.592349+0000","last_active":"2024-02-19T21:05:33.180889+0000","last_peered":"2024-02-19T21:05:33.180889+0000","last_clean":"2024-02-19T21:05:33.180889+0000","last_became_active":"2024-02-19T21:01:18.293791+0000","last_became_peered":"2024-02-19T21:01:18.293791+0000","last_unstale":"2024-02-19T21:05:33.180889+0000","last_undegraded":"2024-02-19T21:05:33.180889+0000","last_fullsized":"2024-02-19T21:05:33.180889+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:04:07.592209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:07.592209+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":125,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.179046+0000","last_change":"2024-02-19T21:05:20.211488+0000","last_active":"2024-02-19T21:05:33.179046+0000","last_peered":"2024-02-19T21:05:33.179046+0000","last_clean":"2024-02-19T21:05:33.179046+0000","last_became_active":"2024-02-19T21:05:20.211241+0000","last_became_peered":"2024-02-19T21:05:20.211241+0000","last_unstale":"2024-02-19T21:05:33.179046+0000","last_undegraded":"2024-02-19T21:05:33.179046+0000","last_fullsized":"2024-02-19T21:05:33.179046+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:58.273113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:58.273113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181906+0000","last_change":"2024-02-19T21:05:01.859040+0000","last_active":"2024-02-19T21:05:33.181906+0000","last_peered":"2024-02-19T21:05:33.181906+0000","last_clean":"2024-02-19T21:05:33.181906+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:05:33.181906+0000","last_undegraded":"2024-02-19T21:05:33.181906+0000","last_fullsized":"2024-02-19T21:05:33.181906+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:01.858924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.858924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.173677+0000","last_change":"2024-02-19T21:04:39.719113+0000","last_active":"2024-02-19T21:05:33.173677+0000","last_peered":"2024-02-19T21:05:33.173677+0000","last_clean":"2024-02-19T21:05:33.173677+0000","last_became_active":"2024-02-19T21:04:39.718926+0000","last_became_peered":"2024-02-19T21:04:39.718926+0000","last_unstale":"2024-02-19T21:05:33.173677+0000","last_undegraded":"2024-02-19T21:05:33.173677+0000","last_fullsized":"2024-02-19T21:05:33.173677+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"100'1","reported_seq":65,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.173200+0000","last_change":"2024-02-19T21:04:39.765598+0000","last_active":"2024-02-19T21:05:33.173200+0000","last_peered":"2024-02-19T21:05:33.173200+0000","last_clean":"2024-02-19T21:05:33.173200+0000","last_became_active":"2024-02-19T21:04:39.765165+0000","last_became_peered":"2024-02-19T21:04:39.765165+0000","last_unstale":"2024-02-19T21:05:33.173200+0000","last_undegraded":"2024-02-19T21:05:33.173200+0000","last_fullsized":"2024-02-19T21:05:33.173200+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.a","version":"100'10","reported_seq":74,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.182946+0000","last_change":"2024-02-19T21:04:39.794115+0000","last_active":"2024-02-19T21:05:33.182946+0000","last_peered":"2024-02-19T21:05:33.182946+0000","last_clean":"2024-02-19T21:05:33.182946+0000","last_became_active":"2024-02-19T21:04:39.792916+0000","last_became_peered":"2024-02-19T21:04:39.792916+0000","last_unstale":"2024-02-19T21:05:33.182946+0000","last_undegraded":"2024-02-19T21:05:33.182946+0000","last_fullsized":"2024-02-19T21:05:33.182946+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183926+0000","last_change":"2024-02-19T21:04:57.188742+0000","last_active":"2024-02-19T21:05:33.183926+0000","last_peered":"2024-02-19T21:05:33.183926+0000","last_clean":"2024-02-19T21:05:33.183926+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:05:33.183926+0000","last_undegraded":"2024-02-19T21:05:33.183926+0000","last_fullsized":"2024-02-19T21:05:33.183926+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:57.188664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:57.188664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":136,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181775+0000","last_change":"2024-02-19T21:05:09.646541+0000","last_active":"2024-02-19T21:05:33.181775+0000","last_peered":"2024-02-19T21:05:33.181775+0000","last_clean":"2024-02-19T21:05:33.181775+0000","last_became_active":"2024-02-19T21:01:18.287139+0000","last_became_peered":"2024-02-19T21:01:18.287139+0000","last_unstale":"2024-02-19T21:05:33.181775+0000","last_undegraded":"2024-02-19T21:05:33.181775+0000","last_fullsized":"2024-02-19T21:05:33.181775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:05:09.646445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:09.646445+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":130,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.175423+0000","last_change":"2024-02-19T21:05:20.212043+0000","last_active":"2024-02-19T21:05:33.175423+0000","last_peered":"2024-02-19T21:05:33.175423+0000","last_clean":"2024-02-19T21:05:33.175423+0000","last_became_active":"2024-02-19T21:05:20.211701+0000","last_became_peered":"2024-02-19T21:05:20.211701+0000","last_unstale":"2024-02-19T21:05:33.175423+0000","last_undegraded":"2024-02-19T21:05:33.175423+0000","last_fullsized":"2024-02-19T21:05:33.175423+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:04:48.298487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:48.298487+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":436,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177189+0000","last_change":"2024-02-19T21:05:07.661094+0000","last_active":"2024-02-19T21:05:33.177189+0000","last_peered":"2024-02-19T21:05:33.177189+0000","last_clean":"2024-02-19T21:05:33.177189+0000","last_became_active":"2024-02-19T21:01:16.274398+0000","last_became_peered":"2024-02-19T21:01:16.274398+0000","last_unstale":"2024-02-19T21:05:33.177189+0000","last_undegraded":"2024-02-19T21:05:33.177189+0000","last_fullsized":"2024-02-19T21:05:33.177189+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:07.660435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:05:07.660435+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185284+0000","last_change":"2024-02-19T21:04:22.168700+0000","last_active":"2024-02-19T21:05:33.185284+0000","last_peered":"2024-02-19T21:05:33.185284+0000","last_clean":"2024-02-19T21:05:33.185284+0000","last_became_active":"2024-02-19T21:01:22.298477+0000","last_became_peered":"2024-02-19T21:01:22.298477+0000","last_unstale":"2024-02-19T21:05:33.185284+0000","last_undegraded":"2024-02-19T21:05:33.185284+0000","last_fullsized":"2024-02-19T21:05:33.185284+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:22.168596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:22.168596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185710+0000","last_change":"2024-02-19T21:04:39.665830+0000","last_active":"2024-02-19T21:05:33.185710+0000","last_peered":"2024-02-19T21:05:33.185710+0000","last_clean":"2024-02-19T21:05:33.185710+0000","last_became_active":"2024-02-19T21:04:39.662125+0000","last_became_peered":"2024-02-19T21:04:39.662125+0000","last_unstale":"2024-02-19T21:05:33.185710+0000","last_undegraded":"2024-02-19T21:05:33.185710+0000","last_fullsized":"2024-02-19T21:05:33.185710+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183850+0000","last_change":"2024-02-19T21:04:39.656608+0000","last_active":"2024-02-19T21:05:33.183850+0000","last_peered":"2024-02-19T21:05:33.183850+0000","last_clean":"2024-02-19T21:05:33.183850+0000","last_became_active":"2024-02-19T21:04:39.655971+0000","last_became_peered":"2024-02-19T21:04:39.655971+0000","last_unstale":"2024-02-19T21:05:33.183850+0000","last_undegraded":"2024-02-19T21:05:33.183850+0000","last_fullsized":"2024-02-19T21:05:33.183850+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185057+0000","last_change":"2024-02-19T21:04:39.633236+0000","last_active":"2024-02-19T21:05:33.185057+0000","last_peered":"2024-02-19T21:05:33.185057+0000","last_clean":"2024-02-19T21:05:33.185057+0000","last_became_active":"2024-02-19T21:04:39.632936+0000","last_became_peered":"2024-02-19T21:04:39.632936+0000","last_unstale":"2024-02-19T21:05:33.185057+0000","last_undegraded":"2024-02-19T21:05:33.185057+0000","last_fullsized":"2024-02-19T21:05:33.185057+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":110,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.178957+0000","last_change":"2024-02-19T21:04:19.858793+0000","last_active":"2024-02-19T21:05:33.178957+0000","last_peered":"2024-02-19T21:05:33.178957+0000","last_clean":"2024-02-19T21:05:33.178957+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:05:33.178957+0000","last_undegraded":"2024-02-19T21:05:33.178957+0000","last_fullsized":"2024-02-19T21:05:33.178957+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:19.858688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.858688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":135,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.188912+0000","last_change":"2024-02-19T21:04:19.885862+0000","last_active":"2024-02-19T21:05:33.188912+0000","last_peered":"2024-02-19T21:05:33.188912+0000","last_clean":"2024-02-19T21:05:33.188912+0000","last_became_active":"2024-02-19T21:01:18.293784+0000","last_became_peered":"2024-02-19T21:01:18.293784+0000","last_unstale":"2024-02-19T21:05:33.188912+0000","last_undegraded":"2024-02-19T21:05:33.188912+0000","last_fullsized":"2024-02-19T21:05:33.188912+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:04:19.885797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.885797+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":123,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.175499+0000","last_change":"2024-02-19T21:05:00.284714+0000","last_active":"2024-02-19T21:05:33.175499+0000","last_peered":"2024-02-19T21:05:33.175499+0000","last_clean":"2024-02-19T21:05:33.175499+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:05:33.175499+0000","last_undegraded":"2024-02-19T21:05:33.175499+0000","last_fullsized":"2024-02-19T21:05:33.175499+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:00.284626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:05:00.284626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":124,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.182779+0000","last_change":"2024-02-19T21:05:20.196022+0000","last_active":"2024-02-19T21:05:33.182779+0000","last_peered":"2024-02-19T21:05:33.182779+0000","last_clean":"2024-02-19T21:05:33.182779+0000","last_became_active":"2024-02-19T21:05:20.195763+0000","last_became_peered":"2024-02-19T21:05:20.195763+0000","last_unstale":"2024-02-19T21:05:33.182779+0000","last_undegraded":"2024-02-19T21:05:33.182779+0000","last_fullsized":"2024-02-19T21:05:33.182779+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:14.215016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:14.215016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.173863+0000","last_change":"2024-02-19T21:04:54.972925+0000","last_active":"2024-02-19T21:05:33.173863+0000","last_peered":"2024-02-19T21:05:33.173863+0000","last_clean":"2024-02-19T21:05:33.173863+0000","last_became_active":"2024-02-19T21:01:16.274290+0000","last_became_peered":"2024-02-19T21:01:16.274290+0000","last_unstale":"2024-02-19T21:05:33.173863+0000","last_undegraded":"2024-02-19T21:05:33.173863+0000","last_fullsized":"2024-02-19T21:05:33.173863+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:54.972832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:54.972832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.184010+0000","last_change":"2024-02-19T21:04:05.136254+0000","last_active":"2024-02-19T21:05:33.184010+0000","last_peered":"2024-02-19T21:05:33.184010+0000","last_clean":"2024-02-19T21:05:33.184010+0000","last_became_active":"2024-02-19T21:01:22.292299+0000","last_became_peered":"2024-02-19T21:01:22.292299+0000","last_unstale":"2024-02-19T21:05:33.184010+0000","last_undegraded":"2024-02-19T21:05:33.184010+0000","last_fullsized":"2024-02-19T21:05:33.184010+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:05.136186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:05.136186+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177342+0000","last_change":"2024-02-19T21:04:39.667403+0000","last_active":"2024-02-19T21:05:33.177342+0000","last_peered":"2024-02-19T21:05:33.177342+0000","last_clean":"2024-02-19T21:05:33.177342+0000","last_became_active":"2024-02-19T21:04:39.666739+0000","last_became_peered":"2024-02-19T21:04:39.666739+0000","last_unstale":"2024-02-19T21:05:33.177342+0000","last_undegraded":"2024-02-19T21:05:33.177342+0000","last_fullsized":"2024-02-19T21:05:33.177342+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1f","version":"100'6","reported_seq":69,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.182327+0000","last_change":"2024-02-19T21:04:39.743491+0000","last_active":"2024-02-19T21:05:33.182327+0000","last_peered":"2024-02-19T21:05:33.182327+0000","last_clean":"2024-02-19T21:05:33.182327+0000","last_became_active":"2024-02-19T21:04:39.742131+0000","last_became_peered":"2024-02-19T21:04:39.742131+0000","last_unstale":"2024-02-19T21:05:33.182327+0000","last_undegraded":"2024-02-19T21:05:33.182327+0000","last_fullsized":"2024-02-19T21:05:33.182327+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"100'2","reported_seq":66,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168756+0000","last_change":"2024-02-19T21:04:39.723860+0000","last_active":"2024-02-19T21:05:33.168756+0000","last_peered":"2024-02-19T21:05:33.168756+0000","last_clean":"2024-02-19T21:05:33.168756+0000","last_became_active":"2024-02-19T21:04:39.715390+0000","last_became_peered":"2024-02-19T21:04:39.715390+0000","last_unstale":"2024-02-19T21:05:33.168756+0000","last_undegraded":"2024-02-19T21:05:33.168756+0000","last_fullsized":"2024-02-19T21:05:33.168756+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.175486+0000","last_change":"2024-02-19T21:04:15.868824+0000","last_active":"2024-02-19T21:05:33.175486+0000","last_peered":"2024-02-19T21:05:33.175486+0000","last_clean":"2024-02-19T21:05:33.175486+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:05:33.175486+0000","last_undegraded":"2024-02-19T21:05:33.175486+0000","last_fullsized":"2024-02-19T21:05:33.175486+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:15.868726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:15.868726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":129,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.173551+0000","last_change":"2024-02-19T21:04:56.846706+0000","last_active":"2024-02-19T21:05:33.173551+0000","last_peered":"2024-02-19T21:05:33.173551+0000","last_clean":"2024-02-19T21:05:33.173551+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:05:33.173551+0000","last_undegraded":"2024-02-19T21:05:33.173551+0000","last_fullsized":"2024-02-19T21:05:33.173551+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:04:56.846622+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.846622+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":129,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169726+0000","last_change":"2024-02-19T21:05:20.220281+0000","last_active":"2024-02-19T21:05:33.169726+0000","last_peered":"2024-02-19T21:05:33.169726+0000","last_clean":"2024-02-19T21:05:33.169726+0000","last_became_active":"2024-02-19T21:05:20.219851+0000","last_became_peered":"2024-02-19T21:05:20.219851+0000","last_unstale":"2024-02-19T21:05:33.169726+0000","last_undegraded":"2024-02-19T21:05:33.169726+0000","last_fullsized":"2024-02-19T21:05:33.169726+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:52.792935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:52.792935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":369,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-02-19T21:05:34.174677+0000","last_change":"2024-02-19T21:04:20.241440+0000","last_active":"2024-02-19T21:05:34.174677+0000","last_peered":"2024-02-19T21:05:34.174677+0000","last_clean":"2024-02-19T21:05:34.174677+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:05:34.174677+0000","last_undegraded":"2024-02-19T21:05:34.174677+0000","last_fullsized":"2024-02-19T21:05:34.174677+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:04:20.241327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:20.241327+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":251,"num_read_kb":246,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":108,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.178970+0000","last_change":"2024-02-19T21:04:11.900738+0000","last_active":"2024-02-19T21:05:33.178970+0000","last_peered":"2024-02-19T21:05:33.178970+0000","last_clean":"2024-02-19T21:05:33.178970+0000","last_became_active":"2024-02-19T21:01:22.607883+0000","last_became_peered":"2024-02-19T21:01:22.607883+0000","last_unstale":"2024-02-19T21:05:33.178970+0000","last_undegraded":"2024-02-19T21:05:33.178970+0000","last_fullsized":"2024-02-19T21:05:33.178970+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.900625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.900625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.187154+0000","last_change":"2024-02-19T21:04:39.692210+0000","last_active":"2024-02-19T21:05:33.187154+0000","last_peered":"2024-02-19T21:05:33.187154+0000","last_clean":"2024-02-19T21:05:33.187154+0000","last_became_active":"2024-02-19T21:04:39.691434+0000","last_became_peered":"2024-02-19T21:04:39.691434+0000","last_unstale":"2024-02-19T21:05:33.187154+0000","last_undegraded":"2024-02-19T21:05:33.187154+0000","last_fullsized":"2024-02-19T21:05:33.187154+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171665+0000","last_change":"2024-02-19T21:04:39.701428+0000","last_active":"2024-02-19T21:05:33.171665+0000","last_peered":"2024-02-19T21:05:33.171665+0000","last_clean":"2024-02-19T21:05:33.171665+0000","last_became_active":"2024-02-19T21:04:39.698760+0000","last_became_peered":"2024-02-19T21:04:39.698760+0000","last_unstale":"2024-02-19T21:05:33.171665+0000","last_undegraded":"2024-02-19T21:05:33.171665+0000","last_fullsized":"2024-02-19T21:05:33.171665+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.8","version":"100'2","reported_seq":66,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.175031+0000","last_change":"2024-02-19T21:04:39.773441+0000","last_active":"2024-02-19T21:05:33.175031+0000","last_peered":"2024-02-19T21:05:33.175031+0000","last_clean":"2024-02-19T21:05:33.175031+0000","last_became_active":"2024-02-19T21:04:39.771376+0000","last_became_peered":"2024-02-19T21:04:39.771376+0000","last_unstale":"2024-02-19T21:05:33.175031+0000","last_undegraded":"2024-02-19T21:05:33.175031+0000","last_fullsized":"2024-02-19T21:05:33.175031+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183636+0000","last_change":"2024-02-19T21:05:01.368595+0000","last_active":"2024-02-19T21:05:33.183636+0000","last_peered":"2024-02-19T21:05:33.183636+0000","last_clean":"2024-02-19T21:05:33.183636+0000","last_became_active":"2024-02-19T21:01:20.290758+0000","last_became_peered":"2024-02-19T21:01:20.290758+0000","last_unstale":"2024-02-19T21:05:33.183636+0000","last_undegraded":"2024-02-19T21:05:33.183636+0000","last_fullsized":"2024-02-19T21:05:33.183636+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:01.368509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.368509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":213,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-02-19T21:05:34.173013+0000","last_change":"2024-02-19T21:04:06.383697+0000","last_active":"2024-02-19T21:05:34.173013+0000","last_peered":"2024-02-19T21:05:34.173013+0000","last_clean":"2024-02-19T21:05:34.173013+0000","last_became_active":"2024-02-19T21:01:18.289076+0000","last_became_peered":"2024-02-19T21:01:18.289076+0000","last_unstale":"2024-02-19T21:05:34.173013+0000","last_undegraded":"2024-02-19T21:05:34.173013+0000","last_fullsized":"2024-02-19T21:05:34.173013+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-19T21:04:06.383559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.383559+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":124,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181998+0000","last_change":"2024-02-19T21:05:20.181793+0000","last_active":"2024-02-19T21:05:33.181998+0000","last_peered":"2024-02-19T21:05:33.181998+0000","last_clean":"2024-02-19T21:05:33.181998+0000","last_became_active":"2024-02-19T21:05:20.181450+0000","last_became_peered":"2024-02-19T21:05:20.181450+0000","last_unstale":"2024-02-19T21:05:33.181998+0000","last_undegraded":"2024-02-19T21:05:33.181998+0000","last_fullsized":"2024-02-19T21:05:33.181998+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:06.145411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:06.145411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172209+0000","last_change":"2024-02-19T21:04:49.004211+0000","last_active":"2024-02-19T21:05:33.172209+0000","last_peered":"2024-02-19T21:05:33.172209+0000","last_clean":"2024-02-19T21:05:33.172209+0000","last_became_active":"2024-02-19T21:01:22.291396+0000","last_became_peered":"2024-02-19T21:01:22.291396+0000","last_unstale":"2024-02-19T21:05:33.172209+0000","last_undegraded":"2024-02-19T21:05:33.172209+0000","last_fullsized":"2024-02-19T21:05:33.172209+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.004148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.004148+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172213+0000","last_change":"2024-02-19T21:04:39.721663+0000","last_active":"2024-02-19T21:05:33.172213+0000","last_peered":"2024-02-19T21:05:33.172213+0000","last_clean":"2024-02-19T21:05:33.172213+0000","last_became_active":"2024-02-19T21:04:39.721416+0000","last_became_peered":"2024-02-19T21:04:39.721416+0000","last_unstale":"2024-02-19T21:05:33.172213+0000","last_undegraded":"2024-02-19T21:05:33.172213+0000","last_fullsized":"2024-02-19T21:05:33.172213+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.186352+0000","last_change":"2024-02-19T21:04:39.678110+0000","last_active":"2024-02-19T21:05:33.186352+0000","last_peered":"2024-02-19T21:05:33.186352+0000","last_clean":"2024-02-19T21:05:33.186352+0000","last_became_active":"2024-02-19T21:04:39.674697+0000","last_became_peered":"2024-02-19T21:04:39.674697+0000","last_unstale":"2024-02-19T21:05:33.186352+0000","last_undegraded":"2024-02-19T21:05:33.186352+0000","last_fullsized":"2024-02-19T21:05:33.186352+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.180485+0000","last_change":"2024-02-19T21:04:39.674303+0000","last_active":"2024-02-19T21:05:33.180485+0000","last_peered":"2024-02-19T21:05:33.180485+0000","last_clean":"2024-02-19T21:05:33.180485+0000","last_became_active":"2024-02-19T21:04:39.674039+0000","last_became_peered":"2024-02-19T21:04:39.674039+0000","last_unstale":"2024-02-19T21:05:33.180485+0000","last_undegraded":"2024-02-19T21:05:33.180485+0000","last_fullsized":"2024-02-19T21:05:33.180485+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":174,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-02-19T21:05:34.167698+0000","last_change":"2024-02-19T21:04:50.353975+0000","last_active":"2024-02-19T21:05:34.167698+0000","last_peered":"2024-02-19T21:05:34.167698+0000","last_clean":"2024-02-19T21:05:34.167698+0000","last_became_active":"2024-02-19T21:01:20.288918+0000","last_became_peered":"2024-02-19T21:01:20.288918+0000","last_unstale":"2024-02-19T21:05:34.167698+0000","last_undegraded":"2024-02-19T21:05:34.167698+0000","last_fullsized":"2024-02-19T21:05:34.167698+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:50.353869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:50.353869+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":128,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183315+0000","last_change":"2024-02-19T21:05:03.922155+0000","last_active":"2024-02-19T21:05:33.183315+0000","last_peered":"2024-02-19T21:05:33.183315+0000","last_clean":"2024-02-19T21:05:33.183315+0000","last_became_active":"2024-02-19T21:01:18.285891+0000","last_became_peered":"2024-02-19T21:01:18.285891+0000","last_unstale":"2024-02-19T21:05:33.183315+0000","last_undegraded":"2024-02-19T21:05:33.183315+0000","last_fullsized":"2024-02-19T21:05:33.183315+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:05:03.922055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.922055+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":133,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185212+0000","last_change":"2024-02-19T21:05:20.223218+0000","last_active":"2024-02-19T21:05:33.185212+0000","last_peered":"2024-02-19T21:05:33.185212+0000","last_clean":"2024-02-19T21:05:33.185212+0000","last_became_active":"2024-02-19T21:05:20.222948+0000","last_became_peered":"2024-02-19T21:05:20.222948+0000","last_unstale":"2024-02-19T21:05:33.185212+0000","last_undegraded":"2024-02-19T21:05:33.185212+0000","last_fullsized":"2024-02-19T21:05:33.185212+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:18.242556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:04:18.242556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":114,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185696+0000","last_change":"2024-02-19T21:05:15.807832+0000","last_active":"2024-02-19T21:05:33.185696+0000","last_peered":"2024-02-19T21:05:33.185696+0000","last_clean":"2024-02-19T21:05:33.185696+0000","last_became_active":"2024-02-19T21:01:22.292133+0000","last_became_peered":"2024-02-19T21:01:22.292133+0000","last_unstale":"2024-02-19T21:05:33.185696+0000","last_undegraded":"2024-02-19T21:05:33.185696+0000","last_fullsized":"2024-02-19T21:05:33.185696+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:15.807734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:15.807734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171610+0000","last_change":"2024-02-19T21:04:39.767350+0000","last_active":"2024-02-19T21:05:33.171610+0000","last_peered":"2024-02-19T21:05:33.171610+0000","last_clean":"2024-02-19T21:05:33.171610+0000","last_became_active":"2024-02-19T21:04:39.766156+0000","last_became_peered":"2024-02-19T21:04:39.766156+0000","last_unstale":"2024-02-19T21:05:33.171610+0000","last_undegraded":"2024-02-19T21:05:33.171610+0000","last_fullsized":"2024-02-19T21:05:33.171610+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.11","version":"100'1","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181668+0000","last_change":"2024-02-19T21:04:39.781019+0000","last_active":"2024-02-19T21:05:33.181668+0000","last_peered":"2024-02-19T21:05:33.181668+0000","last_clean":"2024-02-19T21:05:33.181668+0000","last_became_active":"2024-02-19T21:04:39.776326+0000","last_became_peered":"2024-02-19T21:04:39.776326+0000","last_unstale":"2024-02-19T21:05:33.181668+0000","last_undegraded":"2024-02-19T21:05:33.181668+0000","last_fullsized":"2024-02-19T21:05:33.181668+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.189585+0000","last_change":"2024-02-19T21:04:39.656217+0000","last_active":"2024-02-19T21:05:33.189585+0000","last_peered":"2024-02-19T21:05:33.189585+0000","last_clean":"2024-02-19T21:05:33.189585+0000","last_became_active":"2024-02-19T21:04:39.654865+0000","last_became_peered":"2024-02-19T21:04:39.654865+0000","last_unstale":"2024-02-19T21:05:33.189585+0000","last_undegraded":"2024-02-19T21:05:33.189585+0000","last_fullsized":"2024-02-19T21:05:33.189585+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172035+0000","last_change":"2024-02-19T21:04:16.866220+0000","last_active":"2024-02-19T21:05:33.172035+0000","last_peered":"2024-02-19T21:05:33.172035+0000","last_clean":"2024-02-19T21:05:33.172035+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:05:33.172035+0000","last_undegraded":"2024-02-19T21:05:33.172035+0000","last_fullsized":"2024-02-19T21:05:33.172035+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.866116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":136,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170970+0000","last_change":"2024-02-19T21:04:59.357001+0000","last_active":"2024-02-19T21:05:33.170970+0000","last_peered":"2024-02-19T21:05:33.170970+0000","last_clean":"2024-02-19T21:05:33.170970+0000","last_became_active":"2024-02-19T21:01:18.285672+0000","last_became_peered":"2024-02-19T21:01:18.285672+0000","last_unstale":"2024-02-19T21:05:33.170970+0000","last_undegraded":"2024-02-19T21:05:33.170970+0000","last_fullsized":"2024-02-19T21:05:33.170970+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:04:59.356922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:59.356922+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":135,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172899+0000","last_change":"2024-02-19T21:05:01.259381+0000","last_active":"2024-02-19T21:05:33.172899+0000","last_peered":"2024-02-19T21:05:33.172899+0000","last_clean":"2024-02-19T21:05:33.172899+0000","last_became_active":"2024-02-19T21:01:16.273545+0000","last_became_peered":"2024-02-19T21:01:16.273545+0000","last_unstale":"2024-02-19T21:05:33.172899+0000","last_undegraded":"2024-02-19T21:05:33.172899+0000","last_fullsized":"2024-02-19T21:05:33.172899+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:05:01.259313+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.259313+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169092+0000","last_change":"2024-02-19T21:05:06.001027+0000","last_active":"2024-02-19T21:05:33.169092+0000","last_peered":"2024-02-19T21:05:33.169092+0000","last_clean":"2024-02-19T21:05:33.169092+0000","last_became_active":"2024-02-19T21:01:22.295912+0000","last_became_peered":"2024-02-19T21:01:22.295912+0000","last_unstale":"2024-02-19T21:05:33.169092+0000","last_undegraded":"2024-02-19T21:05:33.169092+0000","last_fullsized":"2024-02-19T21:05:33.169092+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:06.000903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:05:06.000903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170636+0000","last_change":"2024-02-19T21:04:39.696886+0000","last_active":"2024-02-19T21:05:33.170636+0000","last_peered":"2024-02-19T21:05:33.170636+0000","last_clean":"2024-02-19T21:05:33.170636+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:05:33.170636+0000","last_undegraded":"2024-02-19T21:05:33.170636+0000","last_fullsized":"2024-02-19T21:05:33.170636+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181933+0000","last_change":"2024-02-19T21:04:39.741185+0000","last_active":"2024-02-19T21:05:33.181933+0000","last_peered":"2024-02-19T21:05:33.181933+0000","last_clean":"2024-02-19T21:05:33.181933+0000","last_became_active":"2024-02-19T21:04:39.740396+0000","last_became_peered":"2024-02-19T21:04:39.740396+0000","last_unstale":"2024-02-19T21:05:33.181933+0000","last_undegraded":"2024-02-19T21:05:33.181933+0000","last_fullsized":"2024-02-19T21:05:33.181933+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1a","version":"100'5","reported_seq":69,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.176610+0000","last_change":"2024-02-19T21:04:39.736543+0000","last_active":"2024-02-19T21:05:33.176610+0000","last_peered":"2024-02-19T21:05:33.176610+0000","last_clean":"2024-02-19T21:05:33.176610+0000","last_became_active":"2024-02-19T21:04:39.731233+0000","last_became_peered":"2024-02-19T21:04:39.731233+0000","last_unstale":"2024-02-19T21:05:33.176610+0000","last_undegraded":"2024-02-19T21:05:33.176610+0000","last_fullsized":"2024-02-19T21:05:33.176610+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":129,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181859+0000","last_change":"2024-02-19T21:04:09.571081+0000","last_active":"2024-02-19T21:05:33.181859+0000","last_peered":"2024-02-19T21:05:33.181859+0000","last_clean":"2024-02-19T21:05:33.181859+0000","last_became_active":"2024-02-19T21:01:18.293773+0000","last_became_peered":"2024-02-19T21:01:18.293773+0000","last_unstale":"2024-02-19T21:05:33.181859+0000","last_undegraded":"2024-02-19T21:05:33.181859+0000","last_fullsized":"2024-02-19T21:05:33.181859+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:09.570972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:09.570972+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171893+0000","last_change":"2024-02-19T21:04:49.819984+0000","last_active":"2024-02-19T21:05:33.171893+0000","last_peered":"2024-02-19T21:05:33.171893+0000","last_clean":"2024-02-19T21:05:33.171893+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:05:33.171893+0000","last_undegraded":"2024-02-19T21:05:33.171893+0000","last_fullsized":"2024-02-19T21:05:33.171893+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.819921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.819921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168901+0000","last_change":"2024-02-19T21:04:11.841348+0000","last_active":"2024-02-19T21:05:33.168901+0000","last_peered":"2024-02-19T21:05:33.168901+0000","last_clean":"2024-02-19T21:05:33.168901+0000","last_became_active":"2024-02-19T21:01:20.285298+0000","last_became_peered":"2024-02-19T21:01:20.285298+0000","last_unstale":"2024-02-19T21:05:33.168901+0000","last_undegraded":"2024-02-19T21:05:33.168901+0000","last_fullsized":"2024-02-19T21:05:33.168901+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.841184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.841184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":114,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170310+0000","last_change":"2024-02-19T21:05:11.826779+0000","last_active":"2024-02-19T21:05:33.170310+0000","last_peered":"2024-02-19T21:05:33.170310+0000","last_clean":"2024-02-19T21:05:33.170310+0000","last_became_active":"2024-02-19T21:01:22.297086+0000","last_became_peered":"2024-02-19T21:01:22.297086+0000","last_unstale":"2024-02-19T21:05:33.170310+0000","last_undegraded":"2024-02-19T21:05:33.170310+0000","last_fullsized":"2024-02-19T21:05:33.170310+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:11.826647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:11.826647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170916+0000","last_change":"2024-02-19T21:04:39.639826+0000","last_active":"2024-02-19T21:05:33.170916+0000","last_peered":"2024-02-19T21:05:33.170916+0000","last_clean":"2024-02-19T21:05:33.170916+0000","last_became_active":"2024-02-19T21:04:39.639518+0000","last_became_peered":"2024-02-19T21:04:39.639518+0000","last_unstale":"2024-02-19T21:05:33.170916+0000","last_undegraded":"2024-02-19T21:05:33.170916+0000","last_fullsized":"2024-02-19T21:05:33.170916+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.190078+0000","last_change":"2024-02-19T21:04:39.679134+0000","last_active":"2024-02-19T21:05:33.190078+0000","last_peered":"2024-02-19T21:05:33.190078+0000","last_clean":"2024-02-19T21:05:33.190078+0000","last_became_active":"2024-02-19T21:04:39.678877+0000","last_became_peered":"2024-02-19T21:04:39.678877+0000","last_unstale":"2024-02-19T21:05:33.190078+0000","last_undegraded":"2024-02-19T21:05:33.190078+0000","last_fullsized":"2024-02-19T21:05:33.190078+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1b","version":"100'10","reported_seq":74,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170992+0000","last_change":"2024-02-19T21:04:39.642209+0000","last_active":"2024-02-19T21:05:33.170992+0000","last_peered":"2024-02-19T21:05:33.170992+0000","last_clean":"2024-02-19T21:05:33.170992+0000","last_became_active":"2024-02-19T21:04:39.638760+0000","last_became_peered":"2024-02-19T21:04:39.638760+0000","last_unstale":"2024-02-19T21:05:33.170992+0000","last_undegraded":"2024-02-19T21:05:33.170992+0000","last_fullsized":"2024-02-19T21:05:33.170992+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":125,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168836+0000","last_change":"2024-02-19T21:04:06.807310+0000","last_active":"2024-02-19T21:05:33.168836+0000","last_peered":"2024-02-19T21:05:33.168836+0000","last_clean":"2024-02-19T21:05:33.168836+0000","last_became_active":"2024-02-19T21:01:18.293913+0000","last_became_peered":"2024-02-19T21:01:18.293913+0000","last_unstale":"2024-02-19T21:05:33.168836+0000","last_undegraded":"2024-02-19T21:05:33.168836+0000","last_fullsized":"2024-02-19T21:05:33.168836+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:04:06.807190+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.807190+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185187+0000","last_change":"2024-02-19T21:04:08.757186+0000","last_active":"2024-02-19T21:05:33.185187+0000","last_peered":"2024-02-19T21:05:33.185187+0000","last_clean":"2024-02-19T21:05:33.185187+0000","last_became_active":"2024-02-19T21:01:16.271210+0000","last_became_peered":"2024-02-19T21:01:16.271210+0000","last_unstale":"2024-02-19T21:05:33.185187+0000","last_undegraded":"2024-02-19T21:05:33.185187+0000","last_fullsized":"2024-02-19T21:05:33.185187+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.757098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.757098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":113,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.184560+0000","last_change":"2024-02-19T21:05:04.174022+0000","last_active":"2024-02-19T21:05:33.184560+0000","last_peered":"2024-02-19T21:05:33.184560+0000","last_clean":"2024-02-19T21:05:33.184560+0000","last_became_active":"2024-02-19T21:01:20.288589+0000","last_became_peered":"2024-02-19T21:01:20.288589+0000","last_unstale":"2024-02-19T21:05:33.184560+0000","last_undegraded":"2024-02-19T21:05:33.184560+0000","last_fullsized":"2024-02-19T21:05:33.184560+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:04.173902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:04.173902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":108,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.178105+0000","last_change":"2024-02-19T21:04:06.880972+0000","last_active":"2024-02-19T21:05:33.178105+0000","last_peered":"2024-02-19T21:05:33.178105+0000","last_clean":"2024-02-19T21:05:33.178105+0000","last_became_active":"2024-02-19T21:01:22.608285+0000","last_became_peered":"2024-02-19T21:01:22.608285+0000","last_unstale":"2024-02-19T21:05:33.178105+0000","last_undegraded":"2024-02-19T21:05:33.178105+0000","last_fullsized":"2024-02-19T21:05:33.178105+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:06.880899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.880899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.180975+0000","last_change":"2024-02-19T21:04:39.641073+0000","last_active":"2024-02-19T21:05:33.180975+0000","last_peered":"2024-02-19T21:05:33.180975+0000","last_clean":"2024-02-19T21:05:33.180975+0000","last_became_active":"2024-02-19T21:04:39.639775+0000","last_became_peered":"2024-02-19T21:04:39.639775+0000","last_unstale":"2024-02-19T21:05:33.180975+0000","last_undegraded":"2024-02-19T21:05:33.180975+0000","last_fullsized":"2024-02-19T21:05:33.180975+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.c","version":"100'1","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169554+0000","last_change":"2024-02-19T21:04:39.753081+0000","last_active":"2024-02-19T21:05:33.169554+0000","last_peered":"2024-02-19T21:05:33.169554+0000","last_clean":"2024-02-19T21:05:33.169554+0000","last_became_active":"2024-02-19T21:04:39.752718+0000","last_became_peered":"2024-02-19T21:04:39.752718+0000","last_unstale":"2024-02-19T21:05:33.169554+0000","last_undegraded":"2024-02-19T21:05:33.169554+0000","last_fullsized":"2024-02-19T21:05:33.169554+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"100'2","reported_seq":66,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170676+0000","last_change":"2024-02-19T21:04:39.635873+0000","last_active":"2024-02-19T21:05:33.170676+0000","last_peered":"2024-02-19T21:05:33.170676+0000","last_clean":"2024-02-19T21:05:33.170676+0000","last_became_active":"2024-02-19T21:04:39.631391+0000","last_became_peered":"2024-02-19T21:04:39.631391+0000","last_unstale":"2024-02-19T21:05:33.170676+0000","last_undegraded":"2024-02-19T21:05:33.170676+0000","last_fullsized":"2024-02-19T21:05:33.170676+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":128,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169507+0000","last_change":"2024-02-19T21:04:22.763995+0000","last_active":"2024-02-19T21:05:33.169507+0000","last_peered":"2024-02-19T21:05:33.169507+0000","last_clean":"2024-02-19T21:05:33.169507+0000","last_became_active":"2024-02-19T21:01:18.294369+0000","last_became_peered":"2024-02-19T21:01:18.294369+0000","last_unstale":"2024-02-19T21:05:33.169507+0000","last_undegraded":"2024-02-19T21:05:33.169507+0000","last_fullsized":"2024-02-19T21:05:33.169507+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:04:22.763926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:22.763926+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181907+0000","last_change":"2024-02-19T21:04:09.180232+0000","last_active":"2024-02-19T21:05:33.181907+0000","last_peered":"2024-02-19T21:05:33.181907+0000","last_clean":"2024-02-19T21:05:33.181907+0000","last_became_active":"2024-02-19T21:01:16.271608+0000","last_became_peered":"2024-02-19T21:01:16.271608+0000","last_unstale":"2024-02-19T21:05:33.181907+0000","last_undegraded":"2024-02-19T21:05:33.181907+0000","last_fullsized":"2024-02-19T21:05:33.181907+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:09.180159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:09.180159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":171,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-02-19T21:05:34.889050+0000","last_change":"2024-02-19T21:05:34.889050+0000","last_active":"2024-02-19T21:05:34.889050+0000","last_peered":"2024-02-19T21:05:34.889050+0000","last_clean":"2024-02-19T21:05:34.889050+0000","last_became_active":"2024-02-19T21:01:20.288799+0000","last_became_peered":"2024-02-19T21:01:20.288799+0000","last_unstale":"2024-02-19T21:05:34.889050+0000","last_undegraded":"2024-02-19T21:05:34.889050+0000","last_fullsized":"2024-02-19T21:05:34.889050+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:05:34.888948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:34.888948+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":110,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181690+0000","last_change":"2024-02-19T21:04:05.558879+0000","last_active":"2024-02-19T21:05:33.181690+0000","last_peered":"2024-02-19T21:05:33.181690+0000","last_clean":"2024-02-19T21:05:33.181690+0000","last_became_active":"2024-02-19T21:01:22.297188+0000","last_became_peered":"2024-02-19T21:01:22.297188+0000","last_unstale":"2024-02-19T21:05:33.181690+0000","last_undegraded":"2024-02-19T21:05:33.181690+0000","last_fullsized":"2024-02-19T21:05:33.181690+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:04:05.558765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:05.558765+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.186030+0000","last_change":"2024-02-19T21:04:39.688410+0000","last_active":"2024-02-19T21:05:33.186030+0000","last_peered":"2024-02-19T21:05:33.186030+0000","last_clean":"2024-02-19T21:05:33.186030+0000","last_became_active":"2024-02-19T21:04:39.687668+0000","last_became_peered":"2024-02-19T21:04:39.687668+0000","last_unstale":"2024-02-19T21:05:33.186030+0000","last_undegraded":"2024-02-19T21:05:33.186030+0000","last_fullsized":"2024-02-19T21:05:33.186030+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.178882+0000","last_change":"2024-02-19T21:04:39.749884+0000","last_active":"2024-02-19T21:05:33.178882+0000","last_peered":"2024-02-19T21:05:33.178882+0000","last_clean":"2024-02-19T21:05:33.178882+0000","last_became_active":"2024-02-19T21:04:39.748211+0000","last_became_peered":"2024-02-19T21:04:39.748211+0000","last_unstale":"2024-02-19T21:05:33.178882+0000","last_undegraded":"2024-02-19T21:05:33.178882+0000","last_fullsized":"2024-02-19T21:05:33.178882+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177685+0000","last_change":"2024-02-19T21:04:39.667189+0000","last_active":"2024-02-19T21:05:33.177685+0000","last_peered":"2024-02-19T21:05:33.177685+0000","last_clean":"2024-02-19T21:05:33.177685+0000","last_became_active":"2024-02-19T21:04:39.666360+0000","last_became_peered":"2024-02-19T21:04:39.666360+0000","last_unstale":"2024-02-19T21:05:33.177685+0000","last_undegraded":"2024-02-19T21:05:33.177685+0000","last_fullsized":"2024-02-19T21:05:33.177685+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":129,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183571+0000","last_change":"2024-02-19T21:04:15.187687+0000","last_active":"2024-02-19T21:05:33.183571+0000","last_peered":"2024-02-19T21:05:33.183571+0000","last_clean":"2024-02-19T21:05:33.183571+0000","last_became_active":"2024-02-19T21:01:18.293786+0000","last_became_peered":"2024-02-19T21:01:18.293786+0000","last_unstale":"2024-02-19T21:05:33.183571+0000","last_undegraded":"2024-02-19T21:05:33.183571+0000","last_fullsized":"2024-02-19T21:05:33.183571+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:15.187587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:15.187587+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":120,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.180973+0000","last_change":"2024-02-19T21:05:10.715545+0000","last_active":"2024-02-19T21:05:33.180973+0000","last_peered":"2024-02-19T21:05:33.180973+0000","last_clean":"2024-02-19T21:05:33.180973+0000","last_became_active":"2024-02-19T21:01:16.273181+0000","last_became_peered":"2024-02-19T21:01:16.273181+0000","last_unstale":"2024-02-19T21:05:33.180973+0000","last_undegraded":"2024-02-19T21:05:33.180973+0000","last_fullsized":"2024-02-19T21:05:33.180973+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:10.715442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:10.715442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":171,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-02-19T21:05:34.167326+0000","last_change":"2024-02-19T21:05:02.007769+0000","last_active":"2024-02-19T21:05:34.167326+0000","last_peered":"2024-02-19T21:05:34.167326+0000","last_clean":"2024-02-19T21:05:34.167326+0000","last_became_active":"2024-02-19T21:01:20.283943+0000","last_became_peered":"2024-02-19T21:01:20.283943+0000","last_unstale":"2024-02-19T21:05:34.167326+0000","last_undegraded":"2024-02-19T21:05:34.167326+0000","last_fullsized":"2024-02-19T21:05:34.167326+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:05:02.007654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.007654+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172594+0000","last_change":"2024-02-19T21:04:15.099340+0000","last_active":"2024-02-19T21:05:33.172594+0000","last_peered":"2024-02-19T21:05:33.172594+0000","last_clean":"2024-02-19T21:05:33.172594+0000","last_became_active":"2024-02-19T21:01:22.299969+0000","last_became_peered":"2024-02-19T21:01:22.299969+0000","last_unstale":"2024-02-19T21:05:33.172594+0000","last_undegraded":"2024-02-19T21:05:33.172594+0000","last_fullsized":"2024-02-19T21:05:33.172594+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:15.099208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_clean_scrub_stamp":"2024-02-19T21:04:15.099208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":76,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169515+0000","last_change":"2024-02-19T21:04:39.767396+0000","last_active":"2024-02-19T21:05:33.169515+0000","last_peered":"2024-02-19T21:05:33.169515+0000","last_clean":"2024-02-19T21:05:33.169515+0000","last_became_active":"2024-02-19T21:04:39.760167+0000","last_became_peered":"2024-02-19T21:04:39.760167+0000","last_unstale":"2024-02-19T21:05:33.169515+0000","last_undegraded":"2024-02-19T21:05:33.169515+0000","last_fullsized":"2024-02-19T21:05:33.169515+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.a","version":"102'5","reported_seq":72,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181658+0000","last_change":"2024-02-19T21:04:39.751144+0000","last_active":"2024-02-19T21:05:33.181658+0000","last_peered":"2024-02-19T21:05:33.181658+0000","last_clean":"2024-02-19T21:05:33.181658+0000","last_became_active":"2024-02-19T21:04:39.750165+0000","last_became_peered":"2024-02-19T21:04:39.750165+0000","last_unstale":"2024-02-19T21:05:33.181658+0000","last_undegraded":"2024-02-19T21:05:33.181658+0000","last_fullsized":"2024-02-19T21:05:33.181658+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165843,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":24577,"num_write":2,"num_write_kb":24578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171688+0000","last_change":"2024-02-19T21:04:39.704782+0000","last_active":"2024-02-19T21:05:33.171688+0000","last_peered":"2024-02-19T21:05:33.171688+0000","last_clean":"2024-02-19T21:05:33.171688+0000","last_became_active":"2024-02-19T21:04:39.704532+0000","last_became_peered":"2024-02-19T21:04:39.704532+0000","last_unstale":"2024-02-19T21:05:33.171688+0000","last_undegraded":"2024-02-19T21:05:33.171688+0000","last_fullsized":"2024-02-19T21:05:33.171688+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":112,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181571+0000","last_change":"2024-02-19T21:03:22.984626+0000","last_active":"2024-02-19T21:05:33.181571+0000","last_peered":"2024-02-19T21:05:33.181571+0000","last_clean":"2024-02-19T21:05:33.181571+0000","last_became_active":"2024-02-19T21:01:30.922577+0000","last_became_peered":"2024-02-19T21:01:30.922577+0000","last_unstale":"2024-02-19T21:05:33.181571+0000","last_undegraded":"2024-02-19T21:05:33.181571+0000","last_fullsized":"2024-02-19T21:05:33.181571+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:22.984547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:22.984547+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":120,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.188945+0000","last_change":"2024-02-19T21:05:01.685058+0000","last_active":"2024-02-19T21:05:33.188945+0000","last_peered":"2024-02-19T21:05:33.188945+0000","last_clean":"2024-02-19T21:05:33.188945+0000","last_became_active":"2024-02-19T21:01:16.271316+0000","last_became_peered":"2024-02-19T21:01:16.271316+0000","last_unstale":"2024-02-19T21:05:33.188945+0000","last_undegraded":"2024-02-19T21:05:33.188945+0000","last_fullsized":"2024-02-19T21:05:33.188945+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.684958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169017+0000","last_change":"2024-02-19T21:04:56.999269+0000","last_active":"2024-02-19T21:05:33.169017+0000","last_peered":"2024-02-19T21:05:33.169017+0000","last_clean":"2024-02-19T21:05:33.169017+0000","last_became_active":"2024-02-19T21:01:20.289691+0000","last_became_peered":"2024-02-19T21:01:20.289691+0000","last_unstale":"2024-02-19T21:05:33.169017+0000","last_undegraded":"2024-02-19T21:05:33.169017+0000","last_fullsized":"2024-02-19T21:05:33.169017+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:56.999142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.999142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185669+0000","last_change":"2024-02-19T21:04:08.616495+0000","last_active":"2024-02-19T21:05:33.185669+0000","last_peered":"2024-02-19T21:05:33.185669+0000","last_clean":"2024-02-19T21:05:33.185669+0000","last_became_active":"2024-02-19T21:01:22.293093+0000","last_became_peered":"2024-02-19T21:01:22.293093+0000","last_unstale":"2024-02-19T21:05:33.185669+0000","last_undegraded":"2024-02-19T21:05:33.185669+0000","last_fullsized":"2024-02-19T21:05:33.185669+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.616396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.616396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.191821+0000","last_change":"2024-02-19T21:04:39.674336+0000","last_active":"2024-02-19T21:05:33.191821+0000","last_peered":"2024-02-19T21:05:33.191821+0000","last_clean":"2024-02-19T21:05:33.191821+0000","last_became_active":"2024-02-19T21:04:39.672228+0000","last_became_peered":"2024-02-19T21:04:39.672228+0000","last_unstale":"2024-02-19T21:05:33.191821+0000","last_undegraded":"2024-02-19T21:05:33.191821+0000","last_fullsized":"2024-02-19T21:05:33.191821+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.b","version":"102'3","reported_seq":67,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.189461+0000","last_change":"2024-02-19T21:04:39.657768+0000","last_active":"2024-02-19T21:05:33.189461+0000","last_peered":"2024-02-19T21:05:33.189461+0000","last_clean":"2024-02-19T21:05:33.189461+0000","last_became_active":"2024-02-19T21:04:39.655157+0000","last_became_peered":"2024-02-19T21:04:39.655157+0000","last_unstale":"2024-02-19T21:05:33.189461+0000","last_undegraded":"2024-02-19T21:05:33.189461+0000","last_fullsized":"2024-02-19T21:05:33.189461+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2807,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1f","version":"100'5","reported_seq":69,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181998+0000","last_change":"2024-02-19T21:04:39.743813+0000","last_active":"2024-02-19T21:05:33.181998+0000","last_peered":"2024-02-19T21:05:33.181998+0000","last_clean":"2024-02-19T21:05:33.181998+0000","last_became_active":"2024-02-19T21:04:39.742840+0000","last_became_peered":"2024-02-19T21:04:39.742840+0000","last_unstale":"2024-02-19T21:05:33.181998+0000","last_undegraded":"2024-02-19T21:05:33.181998+0000","last_fullsized":"2024-02-19T21:05:33.181998+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":132,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170798+0000","last_change":"2024-02-19T21:04:09.139464+0000","last_active":"2024-02-19T21:05:33.170798+0000","last_peered":"2024-02-19T21:05:33.170798+0000","last_clean":"2024-02-19T21:05:33.170798+0000","last_became_active":"2024-02-19T21:01:18.286128+0000","last_became_peered":"2024-02-19T21:01:18.286128+0000","last_unstale":"2024-02-19T21:05:33.170798+0000","last_undegraded":"2024-02-19T21:05:33.170798+0000","last_fullsized":"2024-02-19T21:05:33.170798+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:04:09.139344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:09.139344+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":120,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.184422+0000","last_change":"2024-02-19T21:05:09.756185+0000","last_active":"2024-02-19T21:05:33.184422+0000","last_peered":"2024-02-19T21:05:33.184422+0000","last_clean":"2024-02-19T21:05:33.184422+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:05:33.184422+0000","last_undegraded":"2024-02-19T21:05:33.184422+0000","last_fullsized":"2024-02-19T21:05:33.184422+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:05:09.756096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183397+0000","last_change":"2024-02-19T21:04:13.337608+0000","last_active":"2024-02-19T21:05:33.183397+0000","last_peered":"2024-02-19T21:05:33.183397+0000","last_clean":"2024-02-19T21:05:33.183397+0000","last_became_active":"2024-02-19T21:01:20.289020+0000","last_became_peered":"2024-02-19T21:01:20.289020+0000","last_unstale":"2024-02-19T21:05:33.183397+0000","last_undegraded":"2024-02-19T21:05:33.183397+0000","last_fullsized":"2024-02-19T21:05:33.183397+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.337498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.337498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170575+0000","last_change":"2024-02-19T21:04:51.014323+0000","last_active":"2024-02-19T21:05:33.170575+0000","last_peered":"2024-02-19T21:05:33.170575+0000","last_clean":"2024-02-19T21:05:33.170575+0000","last_became_active":"2024-02-19T21:01:22.292704+0000","last_became_peered":"2024-02-19T21:01:22.292704+0000","last_unstale":"2024-02-19T21:05:33.170575+0000","last_undegraded":"2024-02-19T21:05:33.170575+0000","last_fullsized":"2024-02-19T21:05:33.170575+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:51.014245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:51.014245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168499+0000","last_change":"2024-02-19T21:04:39.672911+0000","last_active":"2024-02-19T21:05:33.168499+0000","last_peered":"2024-02-19T21:05:33.168499+0000","last_clean":"2024-02-19T21:05:33.168499+0000","last_became_active":"2024-02-19T21:04:39.672672+0000","last_became_peered":"2024-02-19T21:04:39.672672+0000","last_unstale":"2024-02-19T21:05:33.168499+0000","last_undegraded":"2024-02-19T21:05:33.168499+0000","last_fullsized":"2024-02-19T21:05:33.168499+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.8","version":"141'4","reported_seq":77,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.184100+0000","last_change":"2024-02-19T21:05:21.780159+0000","last_active":"2024-02-19T21:05:33.184100+0000","last_peered":"2024-02-19T21:05:33.184100+0000","last_clean":"2024-02-19T21:05:33.184100+0000","last_became_active":"2024-02-19T21:04:39.725742+0000","last_became_peered":"2024-02-19T21:04:39.725742+0000","last_unstale":"2024-02-19T21:05:33.184100+0000","last_undegraded":"2024-02-19T21:05:33.184100+0000","last_fullsized":"2024-02-19T21:05:33.184100+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'4","last_scrub_stamp":"2024-02-19T21:05:21.780088+0000","last_deep_scrub":"141'4","last_deep_scrub_stamp":"2024-02-19T21:05:21.780088+0000","last_clean_scrub_stamp":"2024-02-19T21:05:21.780088+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.187782+0000","last_change":"2024-02-19T21:04:39.639950+0000","last_active":"2024-02-19T21:05:33.187782+0000","last_peered":"2024-02-19T21:05:33.187782+0000","last_clean":"2024-02-19T21:05:33.187782+0000","last_became_active":"2024-02-19T21:04:39.639552+0000","last_became_peered":"2024-02-19T21:04:39.639552+0000","last_unstale":"2024-02-19T21:05:33.187782+0000","last_undegraded":"2024-02-19T21:05:33.187782+0000","last_fullsized":"2024-02-19T21:05:33.187782+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":124,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169177+0000","last_change":"2024-02-19T21:05:08.226255+0000","last_active":"2024-02-19T21:05:33.169177+0000","last_peered":"2024-02-19T21:05:33.169177+0000","last_clean":"2024-02-19T21:05:33.169177+0000","last_became_active":"2024-02-19T21:01:18.287874+0000","last_became_peered":"2024-02-19T21:01:18.287874+0000","last_unstale":"2024-02-19T21:05:33.169177+0000","last_undegraded":"2024-02-19T21:05:33.169177+0000","last_fullsized":"2024-02-19T21:05:33.169177+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:05:08.226144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:08.226144+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.184964+0000","last_change":"2024-02-19T21:04:13.205667+0000","last_active":"2024-02-19T21:05:33.184964+0000","last_peered":"2024-02-19T21:05:33.184964+0000","last_clean":"2024-02-19T21:05:33.184964+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:05:33.184964+0000","last_undegraded":"2024-02-19T21:05:33.184964+0000","last_fullsized":"2024-02-19T21:05:33.184964+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.205596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.205596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185156+0000","last_change":"2024-02-19T21:04:11.754594+0000","last_active":"2024-02-19T21:05:33.185156+0000","last_peered":"2024-02-19T21:05:33.185156+0000","last_clean":"2024-02-19T21:05:33.185156+0000","last_became_active":"2024-02-19T21:01:20.290962+0000","last_became_peered":"2024-02-19T21:01:20.290962+0000","last_unstale":"2024-02-19T21:05:33.185156+0000","last_undegraded":"2024-02-19T21:05:33.185156+0000","last_fullsized":"2024-02-19T21:05:33.185156+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.754525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.754525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":108,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.176433+0000","last_change":"2024-02-19T21:04:13.941207+0000","last_active":"2024-02-19T21:05:33.176433+0000","last_peered":"2024-02-19T21:05:33.176433+0000","last_clean":"2024-02-19T21:05:33.176433+0000","last_became_active":"2024-02-19T21:01:22.297693+0000","last_became_peered":"2024-02-19T21:01:22.297693+0000","last_unstale":"2024-02-19T21:05:33.176433+0000","last_undegraded":"2024-02-19T21:05:33.176433+0000","last_fullsized":"2024-02-19T21:05:33.176433+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.941132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.941132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169387+0000","last_change":"2024-02-19T21:04:39.717968+0000","last_active":"2024-02-19T21:05:33.169387+0000","last_peered":"2024-02-19T21:05:33.169387+0000","last_clean":"2024-02-19T21:05:33.169387+0000","last_became_active":"2024-02-19T21:04:39.716606+0000","last_became_peered":"2024-02-19T21:04:39.716606+0000","last_unstale":"2024-02-19T21:05:33.169387+0000","last_undegraded":"2024-02-19T21:05:33.169387+0000","last_fullsized":"2024-02-19T21:05:33.169387+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.9","version":"139'2","reported_seq":78,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177902+0000","last_change":"2024-02-19T21:05:20.156552+0000","last_active":"2024-02-19T21:05:33.177902+0000","last_peered":"2024-02-19T21:05:33.177902+0000","last_clean":"2024-02-19T21:05:33.177902+0000","last_became_active":"2024-02-19T21:04:39.658671+0000","last_became_peered":"2024-02-19T21:04:39.658671+0000","last_unstale":"2024-02-19T21:05:33.177902+0000","last_undegraded":"2024-02-19T21:05:33.177902+0000","last_fullsized":"2024-02-19T21:05:33.177902+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'2","last_scrub_stamp":"2024-02-19T21:05:20.156478+0000","last_deep_scrub":"139'2","last_deep_scrub_stamp":"2024-02-19T21:05:20.156478+0000","last_clean_scrub_stamp":"2024-02-19T21:05:20.156478+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165828,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1d","version":"100'7","reported_seq":71,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183173+0000","last_change":"2024-02-19T21:04:39.782279+0000","last_active":"2024-02-19T21:05:33.183173+0000","last_peered":"2024-02-19T21:05:33.183173+0000","last_clean":"2024-02-19T21:05:33.183173+0000","last_became_active":"2024-02-19T21:04:39.781203+0000","last_became_peered":"2024-02-19T21:04:39.781203+0000","last_unstale":"2024-02-19T21:05:33.183173+0000","last_undegraded":"2024-02-19T21:05:33.183173+0000","last_fullsized":"2024-02-19T21:05:33.183173+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":132,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171031+0000","last_change":"2024-02-19T21:04:52.027084+0000","last_active":"2024-02-19T21:05:33.171031+0000","last_peered":"2024-02-19T21:05:33.171031+0000","last_clean":"2024-02-19T21:05:33.171031+0000","last_became_active":"2024-02-19T21:01:18.293793+0000","last_became_peered":"2024-02-19T21:01:18.293793+0000","last_unstale":"2024-02-19T21:05:33.171031+0000","last_undegraded":"2024-02-19T21:05:33.171031+0000","last_fullsized":"2024-02-19T21:05:33.171031+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:04:52.027000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:52.027000+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.190173+0000","last_change":"2024-02-19T21:04:13.816617+0000","last_active":"2024-02-19T21:05:33.190173+0000","last_peered":"2024-02-19T21:05:33.190173+0000","last_clean":"2024-02-19T21:05:33.190173+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:05:33.190173+0000","last_undegraded":"2024-02-19T21:05:33.190173+0000","last_fullsized":"2024-02-19T21:05:33.190173+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.816545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.816545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":105,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.175743+0000","last_change":"2024-02-19T21:03:04.486168+0000","last_active":"2024-02-19T21:05:33.175743+0000","last_peered":"2024-02-19T21:05:33.175743+0000","last_clean":"2024-02-19T21:05:33.175743+0000","last_became_active":"2024-02-19T21:01:20.287961+0000","last_became_peered":"2024-02-19T21:01:20.287961+0000","last_unstale":"2024-02-19T21:05:33.175743+0000","last_undegraded":"2024-02-19T21:05:33.175743+0000","last_fullsized":"2024-02-19T21:05:33.175743+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:04.486067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:04.486067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169171+0000","last_change":"2024-02-19T21:04:10.863028+0000","last_active":"2024-02-19T21:05:33.169171+0000","last_peered":"2024-02-19T21:05:33.169171+0000","last_clean":"2024-02-19T21:05:33.169171+0000","last_became_active":"2024-02-19T21:01:22.608997+0000","last_became_peered":"2024-02-19T21:01:22.608997+0000","last_unstale":"2024-02-19T21:05:33.169171+0000","last_undegraded":"2024-02-19T21:05:33.169171+0000","last_fullsized":"2024-02-19T21:05:33.169171+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:10.862926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:10.862926+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183257+0000","last_change":"2024-02-19T21:04:39.686091+0000","last_active":"2024-02-19T21:05:33.183257+0000","last_peered":"2024-02-19T21:05:33.183257+0000","last_clean":"2024-02-19T21:05:33.183257+0000","last_became_active":"2024-02-19T21:04:39.685620+0000","last_became_peered":"2024-02-19T21:04:39.685620+0000","last_unstale":"2024-02-19T21:05:33.183257+0000","last_undegraded":"2024-02-19T21:05:33.183257+0000","last_fullsized":"2024-02-19T21:05:33.183257+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"146'2","reported_seq":75,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.178724+0000","last_change":"2024-02-19T21:05:26.248715+0000","last_active":"2024-02-19T21:05:33.178724+0000","last_peered":"2024-02-19T21:05:33.178724+0000","last_clean":"2024-02-19T21:05:33.178724+0000","last_became_active":"2024-02-19T21:04:39.755561+0000","last_became_peered":"2024-02-19T21:04:39.755561+0000","last_unstale":"2024-02-19T21:05:33.178724+0000","last_undegraded":"2024-02-19T21:05:33.178724+0000","last_fullsized":"2024-02-19T21:05:33.178724+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'2","last_scrub_stamp":"2024-02-19T21:05:26.248627+0000","last_deep_scrub":"146'2","last_deep_scrub_stamp":"2024-02-19T21:05:26.248627+0000","last_clean_scrub_stamp":"2024-02-19T21:05:26.248627+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2800,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185785+0000","last_change":"2024-02-19T21:04:39.665998+0000","last_active":"2024-02-19T21:05:33.185785+0000","last_peered":"2024-02-19T21:05:33.185785+0000","last_clean":"2024-02-19T21:05:33.185785+0000","last_became_active":"2024-02-19T21:04:39.662519+0000","last_became_peered":"2024-02-19T21:04:39.662519+0000","last_unstale":"2024-02-19T21:05:33.185785+0000","last_undegraded":"2024-02-19T21:05:33.185785+0000","last_fullsized":"2024-02-19T21:05:33.185785+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":128,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.182125+0000","last_change":"2024-02-19T21:04:04.545245+0000","last_active":"2024-02-19T21:05:33.182125+0000","last_peered":"2024-02-19T21:05:33.182125+0000","last_clean":"2024-02-19T21:05:33.182125+0000","last_became_active":"2024-02-19T21:01:18.293865+0000","last_became_peered":"2024-02-19T21:01:18.293865+0000","last_unstale":"2024-02-19T21:05:33.182125+0000","last_undegraded":"2024-02-19T21:05:33.182125+0000","last_fullsized":"2024-02-19T21:05:33.182125+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:04:04.545131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:04.545131+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.174877+0000","last_change":"2024-02-19T21:05:07.118727+0000","last_active":"2024-02-19T21:05:33.174877+0000","last_peered":"2024-02-19T21:05:33.174877+0000","last_clean":"2024-02-19T21:05:33.174877+0000","last_became_active":"2024-02-19T21:01:16.272284+0000","last_became_peered":"2024-02-19T21:01:16.272284+0000","last_unstale":"2024-02-19T21:05:33.174877+0000","last_undegraded":"2024-02-19T21:05:33.174877+0000","last_fullsized":"2024-02-19T21:05:33.174877+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:07.118667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:07.118667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":174,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-02-19T21:05:34.179657+0000","last_change":"2024-02-19T21:04:05.787496+0000","last_active":"2024-02-19T21:05:34.179657+0000","last_peered":"2024-02-19T21:05:34.179657+0000","last_clean":"2024-02-19T21:05:34.179657+0000","last_became_active":"2024-02-19T21:01:20.290567+0000","last_became_peered":"2024-02-19T21:01:20.290567+0000","last_unstale":"2024-02-19T21:05:34.179657+0000","last_undegraded":"2024-02-19T21:05:34.179657+0000","last_fullsized":"2024-02-19T21:05:34.179657+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:05.787415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:05.787415+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":110,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.179079+0000","last_change":"2024-02-19T21:04:18.877455+0000","last_active":"2024-02-19T21:05:33.179079+0000","last_peered":"2024-02-19T21:05:33.179079+0000","last_clean":"2024-02-19T21:05:33.179079+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:05:33.179079+0000","last_undegraded":"2024-02-19T21:05:33.179079+0000","last_fullsized":"2024-02-19T21:05:33.179079+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:04:18.877385+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171355+0000","last_change":"2024-02-19T21:04:39.700717+0000","last_active":"2024-02-19T21:05:33.171355+0000","last_peered":"2024-02-19T21:05:33.171355+0000","last_clean":"2024-02-19T21:05:33.171355+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:05:33.171355+0000","last_undegraded":"2024-02-19T21:05:33.171355+0000","last_fullsized":"2024-02-19T21:05:33.171355+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"144'7","reported_seq":84,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185406+0000","last_change":"2024-02-19T21:05:24.947358+0000","last_active":"2024-02-19T21:05:33.185406+0000","last_peered":"2024-02-19T21:05:33.185406+0000","last_clean":"2024-02-19T21:05:33.185406+0000","last_became_active":"2024-02-19T21:04:39.668876+0000","last_became_peered":"2024-02-19T21:04:39.668876+0000","last_unstale":"2024-02-19T21:05:33.185406+0000","last_undegraded":"2024-02-19T21:05:33.185406+0000","last_fullsized":"2024-02-19T21:05:33.185406+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'7","last_scrub_stamp":"2024-02-19T21:05:24.947294+0000","last_deep_scrub":"144'7","last_deep_scrub_stamp":"2024-02-19T21:05:24.947294+0000","last_clean_scrub_stamp":"2024-02-19T21:05:24.947294+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165832,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":27326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"100'9","reported_seq":73,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168671+0000","last_change":"2024-02-19T21:04:39.724745+0000","last_active":"2024-02-19T21:05:33.168671+0000","last_peered":"2024-02-19T21:05:33.168671+0000","last_clean":"2024-02-19T21:05:33.168671+0000","last_became_active":"2024-02-19T21:04:39.715739+0000","last_became_peered":"2024-02-19T21:04:39.715739+0000","last_unstale":"2024-02-19T21:05:33.168671+0000","last_undegraded":"2024-02-19T21:05:33.168671+0000","last_fullsized":"2024-02-19T21:05:33.168671+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":137,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.190471+0000","last_change":"2024-02-19T21:05:07.770793+0000","last_active":"2024-02-19T21:05:33.190471+0000","last_peered":"2024-02-19T21:05:33.190471+0000","last_clean":"2024-02-19T21:05:33.190471+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:05:33.190471+0000","last_undegraded":"2024-02-19T21:05:33.190471+0000","last_fullsized":"2024-02-19T21:05:33.190471+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:05:07.770727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:07.770727+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170757+0000","last_change":"2024-02-19T21:04:52.990442+0000","last_active":"2024-02-19T21:05:33.170757+0000","last_peered":"2024-02-19T21:05:33.170757+0000","last_clean":"2024-02-19T21:05:33.170757+0000","last_became_active":"2024-02-19T21:01:16.277826+0000","last_became_peered":"2024-02-19T21:01:16.277826+0000","last_unstale":"2024-02-19T21:05:33.170757+0000","last_undegraded":"2024-02-19T21:05:33.170757+0000","last_fullsized":"2024-02-19T21:05:33.170757+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:52.990315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:52.990315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.182944+0000","last_change":"2024-02-19T21:04:54.322261+0000","last_active":"2024-02-19T21:05:33.182944+0000","last_peered":"2024-02-19T21:05:33.182944+0000","last_clean":"2024-02-19T21:05:33.182944+0000","last_became_active":"2024-02-19T21:01:20.288016+0000","last_became_peered":"2024-02-19T21:01:20.288016+0000","last_unstale":"2024-02-19T21:05:33.182944+0000","last_undegraded":"2024-02-19T21:05:33.182944+0000","last_fullsized":"2024-02-19T21:05:33.182944+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:54.322192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:54.322192+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.184278+0000","last_change":"2024-02-19T21:05:08.825596+0000","last_active":"2024-02-19T21:05:33.184278+0000","last_peered":"2024-02-19T21:05:33.184278+0000","last_clean":"2024-02-19T21:05:33.184278+0000","last_became_active":"2024-02-19T21:01:22.293230+0000","last_became_peered":"2024-02-19T21:01:22.293230+0000","last_unstale":"2024-02-19T21:05:33.184278+0000","last_undegraded":"2024-02-19T21:05:33.184278+0000","last_fullsized":"2024-02-19T21:05:33.184278+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:08.825485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:08.825485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.187032+0000","last_change":"2024-02-19T21:04:39.801737+0000","last_active":"2024-02-19T21:05:33.187032+0000","last_peered":"2024-02-19T21:05:33.187032+0000","last_clean":"2024-02-19T21:05:33.187032+0000","last_became_active":"2024-02-19T21:04:39.797018+0000","last_became_peered":"2024-02-19T21:04:39.797018+0000","last_unstale":"2024-02-19T21:05:33.187032+0000","last_undegraded":"2024-02-19T21:05:33.187032+0000","last_fullsized":"2024-02-19T21:05:33.187032+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":75,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.188799+0000","last_change":"2024-02-19T21:05:16.792740+0000","last_active":"2024-02-19T21:05:33.188799+0000","last_peered":"2024-02-19T21:05:33.188799+0000","last_clean":"2024-02-19T21:05:33.188799+0000","last_became_active":"2024-02-19T21:04:39.676208+0000","last_became_peered":"2024-02-19T21:04:39.676208+0000","last_unstale":"2024-02-19T21:05:33.188799+0000","last_undegraded":"2024-02-19T21:05:33.188799+0000","last_fullsized":"2024-02-19T21:05:33.188799+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:16.792664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:16.792664+0000","last_clean_scrub_stamp":"2024-02-19T21:05:16.792664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.174857+0000","last_change":"2024-02-19T21:04:39.613875+0000","last_active":"2024-02-19T21:05:33.174857+0000","last_peered":"2024-02-19T21:05:33.174857+0000","last_clean":"2024-02-19T21:05:33.174857+0000","last_became_active":"2024-02-19T21:04:39.613576+0000","last_became_peered":"2024-02-19T21:04:39.613576+0000","last_unstale":"2024-02-19T21:05:33.174857+0000","last_undegraded":"2024-02-19T21:05:33.174857+0000","last_fullsized":"2024-02-19T21:05:33.174857+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":138,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171207+0000","last_change":"2024-02-19T21:04:20.874434+0000","last_active":"2024-02-19T21:05:33.171207+0000","last_peered":"2024-02-19T21:05:33.171207+0000","last_clean":"2024-02-19T21:05:33.171207+0000","last_became_active":"2024-02-19T21:01:18.289712+0000","last_became_peered":"2024-02-19T21:01:18.289712+0000","last_unstale":"2024-02-19T21:05:33.171207+0000","last_undegraded":"2024-02-19T21:05:33.171207+0000","last_fullsized":"2024-02-19T21:05:33.171207+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:04:20.874350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:20.874350+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.186055+0000","last_change":"2024-02-19T21:04:21.828958+0000","last_active":"2024-02-19T21:05:33.186055+0000","last_peered":"2024-02-19T21:05:33.186055+0000","last_clean":"2024-02-19T21:05:33.186055+0000","last_became_active":"2024-02-19T21:01:16.274524+0000","last_became_peered":"2024-02-19T21:01:16.274524+0000","last_unstale":"2024-02-19T21:05:33.186055+0000","last_undegraded":"2024-02-19T21:05:33.186055+0000","last_fullsized":"2024-02-19T21:05:33.186055+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:21.828876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.828876+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.184511+0000","last_change":"2024-02-19T21:04:49.851482+0000","last_active":"2024-02-19T21:05:33.184511+0000","last_peered":"2024-02-19T21:05:33.184511+0000","last_clean":"2024-02-19T21:05:33.184511+0000","last_became_active":"2024-02-19T21:01:20.290705+0000","last_became_peered":"2024-02-19T21:01:20.290705+0000","last_unstale":"2024-02-19T21:05:33.184511+0000","last_undegraded":"2024-02-19T21:05:33.184511+0000","last_fullsized":"2024-02-19T21:05:33.184511+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.851392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.851392+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":104,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172890+0000","last_change":"2024-02-19T21:03:12.081314+0000","last_active":"2024-02-19T21:05:33.172890+0000","last_peered":"2024-02-19T21:05:33.172890+0000","last_clean":"2024-02-19T21:05:33.172890+0000","last_became_active":"2024-02-19T21:01:22.607435+0000","last_became_peered":"2024-02-19T21:01:22.607435+0000","last_unstale":"2024-02-19T21:05:33.172890+0000","last_undegraded":"2024-02-19T21:05:33.172890+0000","last_fullsized":"2024-02-19T21:05:33.172890+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:03:12.081227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.186103+0000","last_change":"2024-02-19T21:04:39.688234+0000","last_active":"2024-02-19T21:05:33.186103+0000","last_peered":"2024-02-19T21:05:33.186103+0000","last_clean":"2024-02-19T21:05:33.186103+0000","last_became_active":"2024-02-19T21:04:39.687307+0000","last_became_peered":"2024-02-19T21:04:39.687307+0000","last_unstale":"2024-02-19T21:05:33.186103+0000","last_undegraded":"2024-02-19T21:05:33.186103+0000","last_fullsized":"2024-02-19T21:05:33.186103+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":73,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.179265+0000","last_change":"2024-02-19T21:05:21.356284+0000","last_active":"2024-02-19T21:05:33.179265+0000","last_peered":"2024-02-19T21:05:33.179265+0000","last_clean":"2024-02-19T21:05:33.179265+0000","last_became_active":"2024-02-19T21:04:39.690799+0000","last_became_peered":"2024-02-19T21:04:39.690799+0000","last_unstale":"2024-02-19T21:05:33.179265+0000","last_undegraded":"2024-02-19T21:05:33.179265+0000","last_fullsized":"2024-02-19T21:05:33.179265+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:21.356220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:21.356220+0000","last_clean_scrub_stamp":"2024-02-19T21:05:21.356220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.11","version":"100'7","reported_seq":71,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177819+0000","last_change":"2024-02-19T21:04:39.612171+0000","last_active":"2024-02-19T21:05:33.177819+0000","last_peered":"2024-02-19T21:05:33.177819+0000","last_clean":"2024-02-19T21:05:33.177819+0000","last_became_active":"2024-02-19T21:04:39.610484+0000","last_became_peered":"2024-02-19T21:04:39.610484+0000","last_unstale":"2024-02-19T21:05:33.177819+0000","last_undegraded":"2024-02-19T21:05:33.177819+0000","last_fullsized":"2024-02-19T21:05:33.177819+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":136,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169626+0000","last_change":"2024-02-19T21:04:50.027760+0000","last_active":"2024-02-19T21:05:33.169626+0000","last_peered":"2024-02-19T21:05:33.169626+0000","last_clean":"2024-02-19T21:05:33.169626+0000","last_became_active":"2024-02-19T21:01:18.285926+0000","last_became_peered":"2024-02-19T21:01:18.285926+0000","last_unstale":"2024-02-19T21:05:33.169626+0000","last_undegraded":"2024-02-19T21:05:33.169626+0000","last_fullsized":"2024-02-19T21:05:33.169626+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:04:50.027690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:50.027690+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":110,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.169867+0000","last_change":"2024-02-19T21:03:20.637528+0000","last_active":"2024-02-19T21:05:33.169867+0000","last_peered":"2024-02-19T21:05:33.169867+0000","last_clean":"2024-02-19T21:05:33.169867+0000","last_became_active":"2024-02-19T21:01:16.274811+0000","last_became_peered":"2024-02-19T21:01:16.274811+0000","last_unstale":"2024-02-19T21:05:33.169867+0000","last_undegraded":"2024-02-19T21:05:33.169867+0000","last_fullsized":"2024-02-19T21:05:33.169867+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:20.637465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:20.637465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":106,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172001+0000","last_change":"2024-02-19T21:03:09.010420+0000","last_active":"2024-02-19T21:05:33.172001+0000","last_peered":"2024-02-19T21:05:33.172001+0000","last_clean":"2024-02-19T21:05:33.172001+0000","last_became_active":"2024-02-19T21:01:20.291672+0000","last_became_peered":"2024-02-19T21:01:20.291672+0000","last_unstale":"2024-02-19T21:05:33.172001+0000","last_undegraded":"2024-02-19T21:05:33.172001+0000","last_fullsized":"2024-02-19T21:05:33.172001+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:09.010341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:09.010341+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168745+0000","last_change":"2024-02-19T21:04:03.806998+0000","last_active":"2024-02-19T21:05:33.168745+0000","last_peered":"2024-02-19T21:05:33.168745+0000","last_clean":"2024-02-19T21:05:33.168745+0000","last_became_active":"2024-02-19T21:01:22.607099+0000","last_became_peered":"2024-02-19T21:01:22.607099+0000","last_unstale":"2024-02-19T21:05:33.168745+0000","last_undegraded":"2024-02-19T21:05:33.168745+0000","last_fullsized":"2024-02-19T21:05:33.168745+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:03.806903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:03.806903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.182042+0000","last_change":"2024-02-19T21:04:39.781223+0000","last_active":"2024-02-19T21:05:33.182042+0000","last_peered":"2024-02-19T21:05:33.182042+0000","last_clean":"2024-02-19T21:05:33.182042+0000","last_became_active":"2024-02-19T21:04:39.777442+0000","last_became_peered":"2024-02-19T21:04:39.777442+0000","last_unstale":"2024-02-19T21:05:33.182042+0000","last_undegraded":"2024-02-19T21:05:33.182042+0000","last_fullsized":"2024-02-19T21:05:33.182042+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2","version":"133'2","reported_seq":78,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.176471+0000","last_change":"2024-02-19T21:05:13.790930+0000","last_active":"2024-02-19T21:05:33.176471+0000","last_peered":"2024-02-19T21:05:33.176471+0000","last_clean":"2024-02-19T21:05:33.176471+0000","last_became_active":"2024-02-19T21:04:39.716211+0000","last_became_peered":"2024-02-19T21:04:39.716211+0000","last_unstale":"2024-02-19T21:05:33.176471+0000","last_undegraded":"2024-02-19T21:05:33.176471+0000","last_fullsized":"2024-02-19T21:05:33.176471+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'2","last_scrub_stamp":"2024-02-19T21:05:13.790836+0000","last_deep_scrub":"133'2","last_deep_scrub_stamp":"2024-02-19T21:05:13.790836+0000","last_clean_scrub_stamp":"2024-02-19T21:05:13.790836+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25165828,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":24577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2800000,"num_omap_keys":1000,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.16","version":"100'5","reported_seq":69,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171770+0000","last_change":"2024-02-19T21:04:39.742946+0000","last_active":"2024-02-19T21:05:33.171770+0000","last_peered":"2024-02-19T21:05:33.171770+0000","last_clean":"2024-02-19T21:05:33.171770+0000","last_became_active":"2024-02-19T21:04:39.741473+0000","last_became_peered":"2024-02-19T21:04:39.741473+0000","last_unstale":"2024-02-19T21:05:33.171770+0000","last_undegraded":"2024-02-19T21:05:33.171770+0000","last_fullsized":"2024-02-19T21:05:33.171770+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"65'9","reported_seq":129,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171224+0000","last_change":"2024-02-19T21:04:11.165275+0000","last_active":"2024-02-19T21:05:33.171224+0000","last_peered":"2024-02-19T21:05:33.171224+0000","last_clean":"2024-02-19T21:05:33.171224+0000","last_became_active":"2024-02-19T21:01:18.285269+0000","last_became_peered":"2024-02-19T21:01:18.285269+0000","last_unstale":"2024-02-19T21:05:33.171224+0000","last_undegraded":"2024-02-19T21:05:33.171224+0000","last_fullsized":"2024-02-19T21:05:33.171224+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:11.165197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.165197+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":114,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.178820+0000","last_change":"2024-02-19T21:04:59.262530+0000","last_active":"2024-02-19T21:05:33.178820+0000","last_peered":"2024-02-19T21:05:33.178820+0000","last_clean":"2024-02-19T21:05:33.178820+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:05:33.178820+0000","last_undegraded":"2024-02-19T21:05:33.178820+0000","last_fullsized":"2024-02-19T21:05:33.178820+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:59.262400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:59.262400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":106,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.182705+0000","last_change":"2024-02-19T21:03:26.922649+0000","last_active":"2024-02-19T21:05:33.182705+0000","last_peered":"2024-02-19T21:05:33.182705+0000","last_clean":"2024-02-19T21:05:33.182705+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:05:33.182705+0000","last_undegraded":"2024-02-19T21:05:33.182705+0000","last_fullsized":"2024-02-19T21:05:33.182705+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:26.922572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:26.922572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.185926+0000","last_change":"2024-02-19T21:04:49.382806+0000","last_active":"2024-02-19T21:05:33.185926+0000","last_peered":"2024-02-19T21:05:33.185926+0000","last_clean":"2024-02-19T21:05:33.185926+0000","last_became_active":"2024-02-19T21:01:22.292714+0000","last_became_peered":"2024-02-19T21:01:22.292714+0000","last_unstale":"2024-02-19T21:05:33.185926+0000","last_undegraded":"2024-02-19T21:05:33.185926+0000","last_fullsized":"2024-02-19T21:05:33.185926+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.382712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.382712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.175739+0000","last_change":"2024-02-19T21:04:39.737500+0000","last_active":"2024-02-19T21:05:33.175739+0000","last_peered":"2024-02-19T21:05:33.175739+0000","last_clean":"2024-02-19T21:05:33.175739+0000","last_became_active":"2024-02-19T21:04:39.736990+0000","last_became_peered":"2024-02-19T21:04:39.736990+0000","last_unstale":"2024-02-19T21:05:33.175739+0000","last_undegraded":"2024-02-19T21:05:33.175739+0000","last_fullsized":"2024-02-19T21:05:33.175739+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3","version":"136'1005","reported_seq":2079,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.183258+0000","last_change":"2024-02-19T21:05:17.044717+0000","last_active":"2024-02-19T21:05:33.183258+0000","last_peered":"2024-02-19T21:05:33.183258+0000","last_clean":"2024-02-19T21:05:33.183258+0000","last_became_active":"2024-02-19T21:04:39.663589+0000","last_became_peered":"2024-02-19T21:04:39.663589+0000","last_unstale":"2024-02-19T21:05:33.183258+0000","last_undegraded":"2024-02-19T21:05:33.183258+0000","last_fullsized":"2024-02-19T21:05:33.183258+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"136'1005","last_scrub_stamp":"2024-02-19T21:05:17.044649+0000","last_deep_scrub":"136'1005","last_deep_scrub_stamp":"2024-02-19T21:05:17.044649+0000","last_clean_scrub_stamp":"2024-02-19T21:05:17.044649+0000","log_size":1005,"ondisk_log_size":1005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1001,"num_read_kb":1,"num_write":1002,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.17","version":"100'5","reported_seq":69,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177840+0000","last_change":"2024-02-19T21:04:39.746785+0000","last_active":"2024-02-19T21:05:33.177840+0000","last_peered":"2024-02-19T21:05:33.177840+0000","last_clean":"2024-02-19T21:05:33.177840+0000","last_became_active":"2024-02-19T21:04:39.746470+0000","last_became_peered":"2024-02-19T21:04:39.746470+0000","last_unstale":"2024-02-19T21:05:33.177840+0000","last_undegraded":"2024-02-19T21:05:33.177840+0000","last_fullsized":"2024-02-19T21:05:33.177840+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":123,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170596+0000","last_change":"2024-02-19T21:04:12.133700+0000","last_active":"2024-02-19T21:05:33.170596+0000","last_peered":"2024-02-19T21:05:33.170596+0000","last_clean":"2024-02-19T21:05:33.170596+0000","last_became_active":"2024-02-19T21:01:18.285735+0000","last_became_peered":"2024-02-19T21:01:18.285735+0000","last_unstale":"2024-02-19T21:05:33.170596+0000","last_undegraded":"2024-02-19T21:05:33.170596+0000","last_fullsized":"2024-02-19T21:05:33.170596+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:04:12.133628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:12.133628+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":114,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.178226+0000","last_change":"2024-02-19T21:03:26.793343+0000","last_active":"2024-02-19T21:05:33.178226+0000","last_peered":"2024-02-19T21:05:33.178226+0000","last_clean":"2024-02-19T21:05:33.178226+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:05:33.178226+0000","last_undegraded":"2024-02-19T21:05:33.178226+0000","last_fullsized":"2024-02-19T21:05:33.178226+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:26.793263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:03:26.793263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":110,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.176558+0000","last_change":"2024-02-19T21:04:10.891192+0000","last_active":"2024-02-19T21:05:33.176558+0000","last_peered":"2024-02-19T21:05:33.176558+0000","last_clean":"2024-02-19T21:05:33.176558+0000","last_became_active":"2024-02-19T21:01:20.287940+0000","last_became_peered":"2024-02-19T21:01:20.287940+0000","last_unstale":"2024-02-19T21:05:33.176558+0000","last_undegraded":"2024-02-19T21:05:33.176558+0000","last_fullsized":"2024-02-19T21:05:33.176558+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:10.891113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:10.891113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.177612+0000","last_change":"2024-02-19T21:04:19.314335+0000","last_active":"2024-02-19T21:05:33.177612+0000","last_peered":"2024-02-19T21:05:33.177612+0000","last_clean":"2024-02-19T21:05:33.177612+0000","last_became_active":"2024-02-19T21:01:22.293282+0000","last_became_peered":"2024-02-19T21:01:22.293282+0000","last_unstale":"2024-02-19T21:05:33.177612+0000","last_undegraded":"2024-02-19T21:05:33.177612+0000","last_fullsized":"2024-02-19T21:05:33.177612+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:19.314241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.314241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.182415+0000","last_change":"2024-02-19T21:04:39.714784+0000","last_active":"2024-02-19T21:05:33.182415+0000","last_peered":"2024-02-19T21:05:33.182415+0000","last_clean":"2024-02-19T21:05:33.182415+0000","last_became_active":"2024-02-19T21:04:39.714495+0000","last_became_peered":"2024-02-19T21:04:39.714495+0000","last_unstale":"2024-02-19T21:05:33.182415+0000","last_undegraded":"2024-02-19T21:05:33.182415+0000","last_fullsized":"2024-02-19T21:05:33.182415+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":73,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.184427+0000","last_change":"2024-02-19T21:05:13.806121+0000","last_active":"2024-02-19T21:05:33.184427+0000","last_peered":"2024-02-19T21:05:33.184427+0000","last_clean":"2024-02-19T21:05:33.184427+0000","last_became_active":"2024-02-19T21:04:39.726486+0000","last_became_peered":"2024-02-19T21:04:39.726486+0000","last_unstale":"2024-02-19T21:05:33.184427+0000","last_undegraded":"2024-02-19T21:05:33.184427+0000","last_fullsized":"2024-02-19T21:05:33.184427+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:13.806012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:13.806012+0000","last_clean_scrub_stamp":"2024-02-19T21:05:13.806012+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.14","version":"100'2","reported_seq":66,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172210+0000","last_change":"2024-02-19T21:04:39.628834+0000","last_active":"2024-02-19T21:05:33.172210+0000","last_peered":"2024-02-19T21:05:33.172210+0000","last_clean":"2024-02-19T21:05:33.172210+0000","last_became_active":"2024-02-19T21:04:39.628525+0000","last_became_peered":"2024-02-19T21:04:39.628525+0000","last_unstale":"2024-02-19T21:05:33.172210+0000","last_undegraded":"2024-02-19T21:05:33.172210+0000","last_fullsized":"2024-02-19T21:05:33.172210+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":124,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170407+0000","last_change":"2024-02-19T21:03:16.047345+0000","last_active":"2024-02-19T21:05:33.170407+0000","last_peered":"2024-02-19T21:05:33.170407+0000","last_clean":"2024-02-19T21:05:33.170407+0000","last_became_active":"2024-02-19T21:01:18.287645+0000","last_became_peered":"2024-02-19T21:01:18.287645+0000","last_unstale":"2024-02-19T21:05:33.170407+0000","last_undegraded":"2024-02-19T21:05:33.170407+0000","last_fullsized":"2024-02-19T21:05:33.170407+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:03:16.047258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:03:16.047258+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.172133+0000","last_change":"2024-02-19T21:04:02.781654+0000","last_active":"2024-02-19T21:05:33.172133+0000","last_peered":"2024-02-19T21:05:33.172133+0000","last_clean":"2024-02-19T21:05:33.172133+0000","last_became_active":"2024-02-19T21:01:16.272132+0000","last_became_peered":"2024-02-19T21:01:16.272132+0000","last_unstale":"2024-02-19T21:05:33.172133+0000","last_undegraded":"2024-02-19T21:05:33.172133+0000","last_fullsized":"2024-02-19T21:05:33.172133+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:02.781556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:02.781556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":111,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.181826+0000","last_change":"2024-02-19T21:04:16.276617+0000","last_active":"2024-02-19T21:05:33.181826+0000","last_peered":"2024-02-19T21:05:33.181826+0000","last_clean":"2024-02-19T21:05:33.181826+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:05:33.181826+0000","last_undegraded":"2024-02-19T21:05:33.181826+0000","last_fullsized":"2024-02-19T21:05:33.181826+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:16.276518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.276518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":109,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171143+0000","last_change":"2024-02-19T21:04:08.178302+0000","last_active":"2024-02-19T21:05:33.171143+0000","last_peered":"2024-02-19T21:05:33.171143+0000","last_clean":"2024-02-19T21:05:33.171143+0000","last_became_active":"2024-02-19T21:01:22.293432+0000","last_became_peered":"2024-02-19T21:01:22.293432+0000","last_unstale":"2024-02-19T21:05:33.171143+0000","last_undegraded":"2024-02-19T21:05:33.171143+0000","last_fullsized":"2024-02-19T21:05:33.171143+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.178225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.178225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.188682+0000","last_change":"2024-02-19T21:04:39.662781+0000","last_active":"2024-02-19T21:05:33.188682+0000","last_peered":"2024-02-19T21:05:33.188682+0000","last_clean":"2024-02-19T21:05:33.188682+0000","last_became_active":"2024-02-19T21:04:39.661810+0000","last_became_peered":"2024-02-19T21:04:39.661810+0000","last_unstale":"2024-02-19T21:05:33.188682+0000","last_undegraded":"2024-02-19T21:05:33.188682+0000","last_fullsized":"2024-02-19T21:05:33.188682+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1","version":"135'4","reported_seq":80,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.182454+0000","last_change":"2024-02-19T21:05:15.923492+0000","last_active":"2024-02-19T21:05:33.182454+0000","last_peered":"2024-02-19T21:05:33.182454+0000","last_clean":"2024-02-19T21:05:33.182454+0000","last_became_active":"2024-02-19T21:04:39.683344+0000","last_became_peered":"2024-02-19T21:04:39.683344+0000","last_unstale":"2024-02-19T21:05:33.182454+0000","last_undegraded":"2024-02-19T21:05:33.182454+0000","last_fullsized":"2024-02-19T21:05:33.182454+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'4","last_scrub_stamp":"2024-02-19T21:05:15.923364+0000","last_deep_scrub":"135'4","last_deep_scrub_stamp":"2024-02-19T21:05:15.923364+0000","last_clean_scrub_stamp":"2024-02-19T21:05:15.923364+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":64,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.178001+0000","last_change":"2024-02-19T21:04:39.691058+0000","last_active":"2024-02-19T21:05:33.178001+0000","last_peered":"2024-02-19T21:05:33.178001+0000","last_clean":"2024-02-19T21:05:33.178001+0000","last_became_active":"2024-02-19T21:04:39.687677+0000","last_became_peered":"2024-02-19T21:04:39.687677+0000","last_unstale":"2024-02-19T21:05:33.178001+0000","last_undegraded":"2024-02-19T21:05:33.178001+0000","last_fullsized":"2024-02-19T21:05:33.178001+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":138,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.171082+0000","last_change":"2024-02-19T21:04:14.826385+0000","last_active":"2024-02-19T21:05:33.171082+0000","last_peered":"2024-02-19T21:05:33.171082+0000","last_clean":"2024-02-19T21:05:33.171082+0000","last_became_active":"2024-02-19T21:01:18.283222+0000","last_became_peered":"2024-02-19T21:01:18.283222+0000","last_unstale":"2024-02-19T21:05:33.171082+0000","last_undegraded":"2024-02-19T21:05:33.171082+0000","last_fullsized":"2024-02-19T21:05:33.171082+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:04:14.826273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:14.826273+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":115,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.182130+0000","last_change":"2024-02-19T21:05:08.599108+0000","last_active":"2024-02-19T21:05:33.182130+0000","last_peered":"2024-02-19T21:05:33.182130+0000","last_clean":"2024-02-19T21:05:33.182130+0000","last_became_active":"2024-02-19T21:01:16.278753+0000","last_became_peered":"2024-02-19T21:01:16.278753+0000","last_unstale":"2024-02-19T21:05:33.182130+0000","last_undegraded":"2024-02-19T21:05:33.182130+0000","last_fullsized":"2024-02-19T21:05:33.182130+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:08.599008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:08.599008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":106,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.170634+0000","last_change":"2024-02-19T21:03:05.073746+0000","last_active":"2024-02-19T21:05:33.170634+0000","last_peered":"2024-02-19T21:05:33.170634+0000","last_clean":"2024-02-19T21:05:33.170634+0000","last_became_active":"2024-02-19T21:01:20.289208+0000","last_became_peered":"2024-02-19T21:01:20.289208+0000","last_unstale":"2024-02-19T21:05:33.170634+0000","last_undegraded":"2024-02-19T21:05:33.170634+0000","last_fullsized":"2024-02-19T21:05:33.170634+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:05.073664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:03:05.073664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":104,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.168836+0000","last_change":"2024-02-19T21:03:27.633572+0000","last_active":"2024-02-19T21:05:33.168836+0000","last_peered":"2024-02-19T21:05:33.168836+0000","last_clean":"2024-02-19T21:05:33.168836+0000","last_became_active":"2024-02-19T21:01:22.609059+0000","last_became_peered":"2024-02-19T21:01:22.609059+0000","last_unstale":"2024-02-19T21:05:33.168836+0000","last_undegraded":"2024-02-19T21:05:33.168836+0000","last_fullsized":"2024-02-19T21:05:33.168836+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:03:27.633492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:03:27.633492+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":63,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-19T21:05:33.188775+0000","last_change":"2024-02-19T21:04:39.726477+0000","last_active":"2024-02-19T21:05:33.188775+0000","last_peered":"2024-02-19T21:05:33.188775+0000","last_clean":"2024-02-19T21:05:33.188775+0000","last_became_active":"2024-02-19T21:04:39.726195+0000","last_became_peered":"2024-02-19T21:04:39.726195+0000","last_unstale":"2024-02-19T21:05:33.188775+0000","last_undegraded":"2024-02-19T21:05:33.188775+0000","last_fullsized":"2024-02-19T21:05:33.188775+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":103,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":94,"num_pg":32,"stat_sum":{"num_bytes":2281,"num_objects":221,"num_object_clones":141,"num_object_copies":663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":222,"num_read_kb":222,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":221,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2715648,"data_stored":6843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":96,"acting":96,"num_store_stats":8},{"poolid":64,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":4,"num_write":28,"num_write_kb":1049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":96,"acting":96,"num_store_stats":8},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":5227,"num_objects":505,"num_object_clones":400,"num_object_copies":1515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":505,"num_whiteouts":0,"num_read":86,"num_read_kb":1,"num_write":510,"num_write_kb":509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6205440,"data_stored":15681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":922,"ondisk_log_size":922,"up":96,"acting":96,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":226499262,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":0,"num_read":1011,"num_read_kb":49161,"num_write":1047,"num_write_kb":226712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8400000,"num_omap_keys":3000,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679809024,"data_stored":679497786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1068,"ondisk_log_size":1068,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":251,"num_read_kb":246,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":22,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":126,"acting":126,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":58,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266742,"num_pgs":136,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":353556,"kb_used_data":52232,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93391596,"statfs":{"total":95995035648,"available":95632994304,"internally_reserved":0,"allocated":53485568,"data_stored":51512642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":0.63700000000000001,"15min":0.55200000000000005},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":8.5640000000000001,"5min":8.5640000000000001,"15min":8.5640000000000001},"last":0.215},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.621,"15min":0.53400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.7110000000000003,"5min":4.7110000000000003,"15min":4.7110000000000003},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.71999999999999997,"15min":0.58899999999999997},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":8.1669999999999998,"5min":8.1669999999999998,"15min":8.1669999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":2.1600000000000001,"5min":0.83999999999999997,"15min":0.628},"min":{"1min":0.35799999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":17.318999999999999,"5min":17.318999999999999,"15min":17.318999999999999},"last":0.254}]},{"osd":2,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":0.69699999999999995,"15min":0.56599999999999995},"min":{"1min":0.35199999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":8.4149999999999991,"5min":8.4149999999999991,"15min":8.4149999999999991},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.2749999999999999,"5min":0.68100000000000005,"15min":0.57399999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":6.875,"5min":6.875,"15min":6.875},"last":0.35699999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":0.65700000000000003,"15min":0.57399999999999995},"min":{"1min":0.32200000000000001,"5min":0.254,"15min":0.254},"max":{"1min":3.8260000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":1.2490000000000001,"5min":0.65500000000000003,"15min":0.54700000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":6.7380000000000004,"5min":6.7380000000000004,"15min":6.7380000000000004},"last":0.32700000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.47299999999999998,"15min":0.49399999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63800000000000001,"5min":1.373,"15min":1.373},"last":0.308},{"interface":"front","average":{"1min":0.437,"5min":0.48599999999999999,"15min":0.48699999999999999},"min":{"1min":0.26600000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63800000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.379}]},{"osd":5,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.46300000000000002,"15min":0.46200000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.51700000000000002,"15min":0.51300000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.70299999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.372}]},{"osd":6,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.52200000000000002,"15min":0.53500000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59399999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.56399999999999995,"15min":0.57499999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73099999999999998,"5min":1.262,"15min":1.262},"last":0.33700000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528380,"num_pgs":135,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":464132,"kb_used_data":150652,"kb_used_omap":0,"kb_used_meta":313472,"kb_avail":93281020,"statfs":{"total":95995035648,"available":95519764480,"internally_reserved":0,"allocated":154267648,"data_stored":152172992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":320995328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.55800000000000005,"15min":0.51500000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":3.9689999999999999,"5min":3.9689999999999999,"15min":3.9689999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.246,"5min":0.64200000000000002,"15min":0.54000000000000004},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":10.119,"5min":10.119,"15min":10.119},"last":0.36899999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":0.61099999999999999,"15min":0.53100000000000003},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":8.0310000000000006,"5min":8.0310000000000006,"15min":8.0310000000000006},"last":0.316},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.56100000000000005,"15min":0.52000000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.3520000000000003,"5min":4.3520000000000003,"15min":4.3520000000000003},"last":0.40500000000000003}]},{"osd":2,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.56599999999999995,"15min":0.53500000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":4.1529999999999996,"5min":4.1529999999999996,"15min":4.1529999999999996},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.59399999999999997,"15min":0.52400000000000002},"min":{"1min":0.33400000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":8.6430000000000007,"5min":8.6430000000000007,"15min":8.6430000000000007},"last":0.33800000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":0.65800000000000003,"15min":0.56000000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":11.927,"5min":11.927,"15min":11.927},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.56799999999999995,"15min":0.51800000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.7670000000000003,"5min":4.7670000000000003,"15min":4.7670000000000003},"last":0.42599999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.41699999999999998,"15min":0.38300000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.58099999999999996,"5min":1.363,"15min":1.363},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.436,"15min":0.40999999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69899999999999995,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.38300000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45700000000000002,"15min":0.42099999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.59799999999999998,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48999999999999999,"15min":0.45600000000000002},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71699999999999997,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.45600000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:05:22 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.497,"15min":0.51300000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.63500000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.439},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.504,"15min":0.52600000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.56799999999999995,"5min":1.534,"15min":1.534},"last":0.44500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":144,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":410808,"kb_used_data":101424,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93334344,"statfs":{"total":95995035648,"available":95574368256,"internally_reserved":0,"allocated":103858176,"data_stored":101837841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:59 2024","interfaces":[{"interface":"back","average":{"1min":2.2589999999999999,"5min":0.75,"15min":0.47799999999999998},"min":{"1min":0.245,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.777000000000001,"5min":27.777000000000001,"15min":27.777000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":2.4969999999999999,"5min":0.81200000000000006,"15min":0.504},"min":{"1min":0.29899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":27.631,"5min":27.631,"15min":27.631},"last":0.315}]},{"osd":2,"last update":"Mon Feb 19 21:05:17 2024","interfaces":[{"interface":"back","average":{"1min":5.1449999999999996,"5min":1.3620000000000001,"15min":0.72599999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":27.798999999999999,"5min":27.798999999999999,"15min":27.798999999999999},"last":0.30199999999999999},{"interface":"front","average":{"1min":5.2469999999999999,"5min":1.3799999999999999,"15min":0.73999999999999999},"min":{"1min":0.28100000000000003,"5min":0.255,"15min":0.255},"max":{"1min":27.606999999999999,"5min":27.606999999999999,"15min":27.606999999999999},"last":0.29299999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:04:45 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.41099999999999998,"15min":0.42299999999999999},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.92000000000000004,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.46200000000000002,"15min":0.46500000000000002},"min":{"1min":0.33600000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.35599999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:05:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":0.77300000000000002,"15min":0.63},"min":{"1min":0.40100000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":8.8930000000000007,"5min":8.8930000000000007,"15min":8.8930000000000007},"last":0.434},{"interface":"front","average":{"1min":2.7400000000000002,"5min":1.012,"15min":0.71199999999999997},"min":{"1min":0.33600000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":27.707000000000001,"5min":27.707000000000001,"15min":27.707000000000001},"last":0.52900000000000003}]},{"osd":5,"last update":"Mon Feb 19 21:04:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58899999999999997,"15min":0.58799999999999997},"min":{"1min":0.47299999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60399999999999998,"15min":0.61199999999999999},"min":{"1min":0.438,"5min":0.377,"15min":0.377},"max":{"1min":0.93999999999999995,"5min":1.052,"15min":1.052},"last":0.501}]},{"osd":6,"last update":"Mon Feb 19 21:04:50 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40699999999999997,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.89900000000000002,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.63500000000000001,"15min":0.64500000000000002},"min":{"1min":0.45400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.429,"5min":1.429,"15min":1.429},"last":0.54900000000000004}]},{"osd":7,"last update":"Mon Feb 19 21:05:17 2024","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":1.1759999999999999,"15min":0.78300000000000003},"min":{"1min":0.40400000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":27.576000000000001,"5min":27.576000000000001,"15min":27.576000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":3.722,"5min":1.242,"15min":0.81499999999999995},"min":{"1min":0.41399999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":27.756,"5min":27.756,"15min":27.756},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":128,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":398532,"kb_used_data":101308,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93346620,"statfs":{"total":95995035648,"available":95586938880,"internally_reserved":0,"allocated":103739392,"data_stored":101840994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.819,"5min":0.69899999999999995,"15min":0.42399999999999999},"min":{"1min":0.223,"5min":0.21299999999999999,"15min":0.16200000000000001},"max":{"1min":12.720000000000001,"5min":12.720000000000001,"15min":12.720000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":2.2690000000000001,"5min":0.74199999999999999,"15min":0.44400000000000001},"min":{"1min":0.26300000000000001,"5min":0.20899999999999999,"15min":0.188},"max":{"1min":18.193000000000001,"5min":18.193000000000001,"15min":18.193000000000001},"last":0.44400000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:05:23 2024","interfaces":[{"interface":"back","average":{"1min":4.8140000000000001,"5min":1.2989999999999999,"15min":0.67900000000000005},"min":{"1min":0.245,"5min":0.219,"15min":0.219},"max":{"1min":27.916,"5min":27.916,"15min":27.916},"last":0.40600000000000003},{"interface":"front","average":{"1min":3.6320000000000001,"5min":1.097,"15min":0.61299999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":22.004000000000001,"5min":22.004000000000001,"15min":22.004000000000001},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:04:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.47399999999999998,"15min":0.45200000000000001},"min":{"1min":0.28100000000000003,"5min":0.249,"15min":0.249},"max":{"1min":1.238,"5min":4.4470000000000001,"15min":4.4470000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.502,"5min":0.48499999999999999,"15min":0.46100000000000002},"min":{"1min":0.315,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.97099999999999997,"5min":4.3070000000000004,"15min":4.3070000000000004},"last":0.504}]},{"osd":4,"last update":"Mon Feb 19 21:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":0.76700000000000002,"15min":0.59899999999999998},"min":{"1min":0.36599999999999999,"5min":0.314,"15min":0.314},"max":{"1min":10.217000000000001,"5min":10.217000000000001,"15min":10.217000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.64100000000000001,"15min":0.56899999999999995},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.4409999999999998,"5min":3.4409999999999998,"15min":3.4409999999999998},"last":0.871}]},{"osd":5,"last update":"Mon Feb 19 21:05:31 2024","interfaces":[{"interface":"back","average":{"1min":3.7320000000000002,"5min":1.226,"15min":0.83999999999999997},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":20.021999999999998,"5min":20.021999999999998,"15min":20.021999999999998},"last":0.83399999999999996},{"interface":"front","average":{"1min":3.137,"5min":1.167,"15min":0.95499999999999996},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":20.908999999999999,"5min":20.908999999999999,"15min":20.908999999999999},"last":0.86199999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:04:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.39200000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81999999999999995,"5min":1.381,"15min":1.381},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.61199999999999999,"15min":0.64600000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.98999999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.90300000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":3.0059999999999998,"5min":1.1699999999999999,"15min":0.93200000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":20.940999999999999,"5min":20.940999999999999,"15min":20.940999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":3.548,"5min":1.206,"15min":0.82499999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":20.471,"5min":20.471,"15min":20.471},"last":0.88500000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":135,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":361692,"kb_used_data":52308,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93383460,"statfs":{"total":95995035648,"available":95624663040,"internally_reserved":0,"allocated":53563392,"data_stored":51510154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:05:22 2024","interfaces":[{"interface":"back","average":{"1min":2.5449999999999999,"5min":0.85299999999999998,"15min":0.51700000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.222},"max":{"1min":26.690999999999999,"5min":26.690999999999999,"15min":26.690999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":2.9300000000000002,"5min":0.94299999999999995,"15min":0.54200000000000004},"min":{"1min":0.23799999999999999,"5min":0.214,"15min":0.214},"max":{"1min":24.315000000000001,"5min":24.315000000000001,"15min":24.315000000000001},"last":0.30299999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:05:22 2024","interfaces":[{"interface":"back","average":{"1min":3.6960000000000002,"5min":1.0960000000000001,"15min":0.59599999999999997},"min":{"1min":0.251,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":21.891999999999999,"5min":21.891999999999999,"15min":21.891999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":2.9470000000000001,"5min":0.97199999999999998,"15min":0.58499999999999996},"min":{"1min":0.28899999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":26.748999999999999,"5min":26.748999999999999,"15min":26.748999999999999},"last":0.39800000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:04:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.441,"15min":0.45400000000000001},"min":{"1min":0.29599999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.78900000000000003,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.49099999999999999,"15min":0.48299999999999998},"min":{"1min":0.27600000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.8399999999999999,"5min":4.8399999999999999,"15min":4.8399999999999999},"last":0.42399999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:05:07 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.66100000000000003,"15min":0.627},"min":{"1min":0.309,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.7330000000000001,"5min":5.2880000000000003,"15min":5.2880000000000003},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.65900000000000003,"15min":0.66000000000000003},"min":{"1min":0.433,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":4.6079999999999997,"5min":4.6079999999999997,"15min":4.6079999999999997},"last":0.41299999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:05:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":0.68899999999999995,"15min":0.67300000000000004},"min":{"1min":0.46200000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":4.569,"5min":4.569,"15min":4.569},"last":0.42999999999999999},{"interface":"front","average":{"1min":2.589,"5min":1.0649999999999999,"15min":0.79500000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":26.606000000000002,"5min":26.606000000000002,"15min":26.606000000000002},"last":0.443}]},{"osd":6,"last update":"Mon Feb 19 21:05:01 2024","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.68700000000000006,"15min":0.65100000000000002},"min":{"1min":0.44500000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":5.3170000000000002,"5min":5.3170000000000002,"15min":5.3170000000000002},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.66900000000000004,"15min":0.64800000000000002},"min":{"1min":0.48399999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":4.5250000000000004,"5min":4.5250000000000004,"15min":4.5250000000000004},"last":0.56200000000000006}]},{"osd":7,"last update":"Mon Feb 19 21:05:22 2024","interfaces":[{"interface":"back","average":{"1min":2.6480000000000001,"5min":1.0620000000000001,"15min":0.745},"min":{"1min":0.46200000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":26.658000000000001,"5min":26.658000000000001,"15min":26.658000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":2.109,"5min":0.97199999999999998,"15min":0.72799999999999998},"min":{"1min":0.41899999999999998,"5min":0.379,"15min":0.379},"max":{"1min":24.402000000000001,"5min":24.402000000000001,"15min":24.402000000000001},"last":0.51000000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":135,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":410640,"kb_used_data":101256,"kb_used_omap":0,"kb_used_meta":309376,"kb_avail":93334512,"statfs":{"total":95995035648,"available":95574540288,"internally_reserved":0,"allocated":103686144,"data_stored":101841964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316801024},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:04:45 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":1.05,"15min":0.58599999999999997},"min":{"1min":0.26700000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":77.662000000000006,"5min":77.662000000000006,"15min":77.662000000000006},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.1679999999999999,"5min":0.52800000000000002,"15min":0.44800000000000001},"min":{"1min":0.23699999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":17.456,"5min":17.456,"15min":17.456},"last":0.28799999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:04:45 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":0.51900000000000002,"15min":0.441},"min":{"1min":0.23000000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":17.327999999999999,"5min":17.327999999999999,"15min":17.327999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.39600000000000002,"15min":0.40000000000000002},"min":{"1min":0.252,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.312}]},{"osd":2,"last update":"Mon Feb 19 21:04:45 2024","interfaces":[{"interface":"back","average":{"1min":2.9950000000000001,"5min":0.91800000000000004,"15min":0.58999999999999997},"min":{"1min":0.30599999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":56.890999999999998,"5min":56.890999999999998,"15min":56.890999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":2.4489999999999998,"5min":0.79900000000000004,"15min":0.54900000000000004},"min":{"1min":0.318,"5min":0.22,"15min":0.22},"max":{"1min":44.600999999999999,"5min":44.600999999999999,"15min":44.600999999999999},"last":0.30199999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:05:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":0.67100000000000004,"15min":0.58799999999999997},"min":{"1min":0.41199999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":7.1909999999999998,"5min":7.1909999999999998,"15min":7.1909999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":2.8809999999999998,"5min":1.0149999999999999,"15min":0.71899999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":26.393999999999998,"5min":26.393999999999998,"15min":26.393999999999998},"last":0.46999999999999997}]},{"osd":5,"last update":"Mon Feb 19 21:05:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0990000000000002,"5min":0.873,"15min":0.67900000000000005},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":17.401,"5min":17.401,"15min":17.401},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.891,"5min":0.82799999999999996,"15min":0.65200000000000002},"min":{"1min":0.36199999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":12.882,"5min":12.882,"15min":12.882},"last":0.45900000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":0.70999999999999996,"15min":0.61799999999999999},"min":{"1min":0.371,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":17.367000000000001,"5min":17.367000000000001,"15min":17.367000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57899999999999996,"15min":0.59099999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.79200000000000004,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.435}]},{"osd":7,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":1.976,"5min":0.85199999999999998,"15min":0.65800000000000003},"min":{"1min":0.38800000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":17.428999999999998,"5min":17.428999999999998,"15min":17.428999999999998},"last":0.45000000000000001},{"interface":"front","average":{"1min":3.956,"5min":1.2569999999999999,"15min":0.80100000000000005},"min":{"1min":0.437,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":44.378,"5min":44.378,"15min":44.378},"last":0.53900000000000003}]}]},{"osd":4,"up_from":31,"seq":133143986246,"num_pgs":136,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300644,"kb_used_data":3416,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444508,"statfs":{"total":95995035648,"available":95687176192,"internally_reserved":0,"allocated":3497984,"data_stored":1176688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:05:05 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":0.63600000000000001,"15min":0.58499999999999996},"min":{"1min":0.249,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.9630000000000001,"5min":6.9630000000000001,"15min":6.9630000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":3.802,"5min":1.1599999999999999,"15min":0.73999999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":70.063000000000002,"5min":70.063000000000002,"15min":70.063000000000002},"last":0.441}]},{"osd":1,"last update":"Mon Feb 19 21:05:05 2024","interfaces":[{"interface":"back","average":{"1min":1.869,"5min":0.80100000000000005,"15min":0.629},"min":{"1min":0.32200000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":26.510999999999999,"5min":26.510999999999999,"15min":26.510999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.143,"5min":0.622,"15min":0.55900000000000005},"min":{"1min":0.32000000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":13.635999999999999,"5min":13.635999999999999,"15min":13.635999999999999},"last":0.40500000000000003}]},{"osd":2,"last update":"Mon Feb 19 21:05:05 2024","interfaces":[{"interface":"back","average":{"1min":2.617,"5min":0.94299999999999995,"15min":0.66700000000000004},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":37.320999999999998,"5min":37.320999999999998,"15min":37.320999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":4.1840000000000002,"5min":1.2470000000000001,"15min":0.79500000000000004},"min":{"1min":0.314,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":66.840999999999994,"5min":66.840999999999994,"15min":66.840999999999994},"last":0.52100000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:05:05 2024","interfaces":[{"interface":"back","average":{"1min":1.9570000000000001,"5min":0.79400000000000004,"15min":0.61799999999999999},"min":{"1min":0.308,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":13.49,"5min":13.49,"15min":13.49},"last":0.48699999999999999},{"interface":"front","average":{"1min":2.379,"5min":0.92600000000000005,"15min":0.68999999999999995},"min":{"1min":0.29599999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":13.709,"5min":13.709,"15min":13.709},"last":0.47099999999999997}]},{"osd":5,"last update":"Mon Feb 19 21:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.438,"15min":0.434},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.218},"max":{"1min":3.2799999999999998,"5min":3.2799999999999998,"15min":3.2799999999999998},"last":0.499},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.48399999999999999,"15min":0.45000000000000001},"min":{"1min":0.26600000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.4340000000000002,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":0.52700000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.43099999999999999,"15min":0.38700000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63400000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.47799999999999998,"15min":0.438},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.64600000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Feb 19 21:05:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.52100000000000002,"15min":0.45800000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.0730000000000004,"5min":6.0730000000000004,"15min":6.0730000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.51800000000000002,"15min":0.47599999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.202,"5min":3.202,"15min":3.202},"last":0.57399999999999995}]}]},{"osd":5,"up_from":37,"seq":158913790017,"num_pgs":149,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":427816,"kb_used_data":126496,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93317336,"statfs":{"total":95995035648,"available":95556952064,"internally_reserved":0,"allocated":129531904,"data_stored":127004893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:05:26 2024","interfaces":[{"interface":"back","average":{"1min":21.559999999999999,"5min":4.6870000000000003,"15min":1.907},"min":{"1min":0.27100000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":276.47899999999998,"5min":276.47899999999998,"15min":276.47899999999998},"last":0.374},{"interface":"front","average":{"1min":6.4080000000000004,"5min":1.698,"15min":0.93100000000000005},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":52.887,"5min":52.887,"15min":52.887},"last":0.437}]},{"osd":1,"last update":"Mon Feb 19 21:05:26 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":2.7999999999999998,"15min":1.2849999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":120.011,"5min":120.011,"15min":120.011},"last":0.47899999999999998},{"interface":"front","average":{"1min":8.0280000000000005,"5min":2.0310000000000001,"15min":1.0349999999999999},"min":{"1min":0.317,"5min":0.307,"15min":0.307},"max":{"1min":118.105,"5min":118.105,"15min":118.105},"last":0.39300000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:05:26 2024","interfaces":[{"interface":"back","average":{"1min":7.6929999999999996,"5min":1.9370000000000001,"15min":1.008},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":82.242000000000004,"5min":82.242000000000004,"15min":82.242000000000004},"last":0.40500000000000003},{"interface":"front","average":{"1min":15.124000000000001,"5min":3.4289999999999998,"15min":1.5009999999999999},"min":{"1min":0.32000000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":232.84999999999999,"5min":232.84999999999999,"15min":232.84999999999999},"last":0.44600000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:05:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5739999999999998,"5min":2.1269999999999998,"15min":1.0680000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":109.70099999999999,"5min":109.70099999999999,"15min":109.70099999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":6.1449999999999996,"5min":1.653,"15min":0.92100000000000004},"min":{"1min":0.376,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":65.519000000000005,"5min":65.519000000000005,"15min":65.519000000000005},"last":0.41499999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.41499999999999998,"15min":0.39400000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.622,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.437,"15min":0.39800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.59799999999999998,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.42699999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:04:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45700000000000002,"15min":0.42799999999999999},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.495},{"interface":"front","average":{"1min":0.503,"5min":0.47099999999999997,"15min":0.442},"min":{"1min":0.33300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.35399999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:05:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.496,"15min":0.5},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.62,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.44,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63800000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.45700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100712448,"data_stored":100666171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":100700160,"data_stored":100663410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50360320,"data_stored":50335502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":100716544,"data_stored":100667203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":125870080,"data_stored":125829257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":151064576,"data_stored":150997845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":50372608,"data_stored":50337360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":520192,"data_stored":1309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":798720,"data_stored":2028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":684032,"data_stored":1729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":618496,"data_stored":1565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":905216,"data_stored":2293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1011712,"data_stored":2557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":872448,"data_stored":2202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":794624,"data_stored":1998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":64,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":94,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":372736,"data_stored":950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":94,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":94,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":1045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":94,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":94,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":1190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":94,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":1397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":94,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":94,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:05:37.881 INFO:tasks.thrashosds.thrasher:Setting 94.1c to [4, 2, 0] 2024-02-19T21:05:37.881 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '94.1c', '4', '2', '0'] 2024-02-19T21:05:37.881 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pg-upmap 94.1c 4 2 0 2024-02-19T21:05:38.065 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:05:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:05:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:05:38.273 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:38.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:38.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:38.374 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:05:38.377 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:38.404 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:38.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:37.127573+0000 mgr.y (mgr.14550) 207 : cluster [DBG] pgmap v206: 398 pgs: 64 unknown, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 76 op/s 2024-02-19T21:05:38.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:37.571904+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:05:38.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:37.572691+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:05:38.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:38.173836+0000 mon.a (mon.0) 1199 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-02-19T21:05:38.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:38.185650+0000 mon.c (mon.2) 150 : audit [INF] from='client.? v1:172.21.15.163:0/2126190265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi163-83845-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:38.527 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:38.529 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:37.127573+0000 mgr.y (mgr.14550) 207 : cluster [DBG] pgmap v206: 398 pgs: 64 unknown, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 76 op/s 2024-02-19T21:05:38.529 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:37.571904+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:05:38.529 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:37.572691+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:05:38.530 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:38.173836+0000 mon.a (mon.0) 1199 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-02-19T21:05:38.530 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:38.185650+0000 mon.c (mon.2) 150 : audit [INF] from='client.? v1:172.21.15.163:0/2126190265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi163-83845-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:38.530 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:38.186130+0000 mon.a (mon.0) 1200 : audit [INF] from='client.48881 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi163-83845-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:38.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:38.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:38.595 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:38.631 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:05:38.663 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:38.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:38.186130+0000 mon.a (mon.0) 1200 : audit [INF] from='client.48881 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi163-83845-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:38.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:37.127573+0000 mgr.y (mgr.14550) 207 : cluster [DBG] pgmap v206: 398 pgs: 64 unknown, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 57 KiB/s rd, 76 op/s 2024-02-19T21:05:38.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:37.571904+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:05:38.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:37.572691+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:05:38.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:38.173836+0000 mon.a (mon.0) 1199 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-02-19T21:05:38.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:38.185650+0000 mon.c (mon.2) 150 : audit [INF] from='client.? v1:172.21.15.163:0/2126190265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi163-83845-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:38.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:38.186130+0000 mon.a (mon.0) 1200 : audit [INF] from='client.48881 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi163-83845-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:38.802 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:05:38.803 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:05:39.018 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:05:39.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:39.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:05:39.122 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:39.152 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:39.186 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:39.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:39.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:39.291 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:39.320 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:39.502 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:39.525 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:39.527 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:39.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:39.629 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:39.650 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-19T21:05:39.659 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:39.681 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:39.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:39.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:39.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:37.955409+0000 osd.1 (osd.1) 95 : cluster [DBG] 94.4 scrub starts 2024-02-19T21:05:39.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:37.958000+0000 osd.1 (osd.1) 96 : cluster [DBG] 94.4 scrub ok 2024-02-19T21:05:39.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:38.563568+0000 osd.4 (osd.4) 83 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:05:39.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:38.564464+0000 osd.4 (osd.4) 84 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:05:39.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:39.095640+0000 mon.a (mon.0) 1201 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:05:39.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:39.173703+0000 mon.a (mon.0) 1202 : audit [INF] from='client.48881 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi163-83845-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:39.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:39.173918+0000 mon.a (mon.0) 1203 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-02-19T21:05:39.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:39.191758+0000 mon.c (mon.2) 151 : audit [INF] from='client.? v1:172.21.15.163:0/415394340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi163-83854-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:39.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:39.192353+0000 mon.a (mon.0) 1204 : audit [INF] from='client.48893 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi163-83854-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:39.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:37.955409+0000 osd.1 (osd.1) 95 : cluster [DBG] 94.4 scrub starts 2024-02-19T21:05:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:37.958000+0000 osd.1 (osd.1) 96 : cluster [DBG] 94.4 scrub ok 2024-02-19T21:05:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:38.563568+0000 osd.4 (osd.4) 83 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:05:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:38.564464+0000 osd.4 (osd.4) 84 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:05:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:39.095640+0000 mon.a (mon.0) 1201 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:05:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:39.173703+0000 mon.a (mon.0) 1202 : audit [INF] from='client.48881 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi163-83845-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:39.173918+0000 mon.a (mon.0) 1203 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-02-19T21:05:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:39.191758+0000 mon.c (mon.2) 151 : audit [INF] from='client.? v1:172.21.15.163:0/415394340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi163-83854-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:39.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:39.192353+0000 mon.a (mon.0) 1204 : audit [INF] from='client.48893 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi163-83854-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:39.786 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:05:39.813 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:37.955409+0000 osd.1 (osd.1) 95 : cluster [DBG] 94.4 scrub starts 2024-02-19T21:05:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:37.958000+0000 osd.1 (osd.1) 96 : cluster [DBG] 94.4 scrub ok 2024-02-19T21:05:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:38.563568+0000 osd.4 (osd.4) 83 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:05:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:38.564464+0000 osd.4 (osd.4) 84 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:05:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:39.095640+0000 mon.a (mon.0) 1201 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:05:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:39.173703+0000 mon.a (mon.0) 1202 : audit [INF] from='client.48881 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi163-83845-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:39.173918+0000 mon.a (mon.0) 1203 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-02-19T21:05:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:39.191758+0000 mon.c (mon.2) 151 : audit [INF] from='client.? v1:172.21.15.163:0/415394340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi163-83854-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:39.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:39.192353+0000 mon.a (mon.0) 1204 : audit [INF] from='client.48893 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi163-83854-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:39.926 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:05:39.963 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:05:39.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:39.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:05:40.067 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:40.093 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:40.166 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:40.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:40.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:40.269 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:05:40.296 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:40.418 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:05:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:05:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:05:40.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:40.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:05:40.517 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:05:40.527 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:05:40.592 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:40.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:39.128080+0000 mgr.y (mgr.14550) 208 : cluster [DBG] pgmap v209: 366 pgs: 32 unknown, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:40.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:39.786524+0000 osd.2 (osd.2) 75 : cluster [DBG] 94.7 scrub starts 2024-02-19T21:05:40.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:39.788170+0000 osd.2 (osd.2) 76 : cluster [DBG] 94.7 scrub ok 2024-02-19T21:05:40.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:40.176565+0000 mon.a (mon.0) 1205 : audit [INF] from='client.48893 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi163-83854-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:40.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:40.176712+0000 mon.a (mon.0) 1206 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-02-19T21:05:40.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:40.193392+0000 mon.b (mon.1) 250 : audit [INF] from='client.? v1:172.21.15.163:0/1100318564' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83952-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:40.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:40.195770+0000 mon.a (mon.0) 1207 : audit [INF] from='client.47698 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83952-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:39.128080+0000 mgr.y (mgr.14550) 208 : cluster [DBG] pgmap v209: 366 pgs: 32 unknown, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:39.786524+0000 osd.2 (osd.2) 75 : cluster [DBG] 94.7 scrub starts 2024-02-19T21:05:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:39.788170+0000 osd.2 (osd.2) 76 : cluster [DBG] 94.7 scrub ok 2024-02-19T21:05:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:40.176565+0000 mon.a (mon.0) 1205 : audit [INF] from='client.48893 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi163-83854-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:40.176712+0000 mon.a (mon.0) 1206 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-02-19T21:05:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:40.193392+0000 mon.b (mon.1) 250 : audit [INF] from='client.? v1:172.21.15.163:0/1100318564' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83952-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:40.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:40.195770+0000 mon.a (mon.0) 1207 : audit [INF] from='client.47698 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83952-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:40.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:39.128080+0000 mgr.y (mgr.14550) 208 : cluster [DBG] pgmap v209: 366 pgs: 32 unknown, 334 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:40.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:39.786524+0000 osd.2 (osd.2) 75 : cluster [DBG] 94.7 scrub starts 2024-02-19T21:05:40.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:39.788170+0000 osd.2 (osd.2) 76 : cluster [DBG] 94.7 scrub ok 2024-02-19T21:05:40.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:40.176565+0000 mon.a (mon.0) 1205 : audit [INF] from='client.48893 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi163-83854-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:40.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:40.176712+0000 mon.a (mon.0) 1206 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-02-19T21:05:40.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:40.193392+0000 mon.b (mon.1) 250 : audit [INF] from='client.? v1:172.21.15.163:0/1100318564' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83952-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:40.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:40.195770+0000 mon.a (mon.0) 1207 : audit [INF] from='client.47698 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83952-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:40.840 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:05:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:40.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:05:40.944 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:40.971 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:41.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.145+0000 7f0393856700 1 -- 172.21.15.163:0/1164284759 >> v1:172.21.15.163:6789/0 conn(0x7f038c0f89c0 legacy=0x7f038c0f8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:41.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.145+0000 7f0393856700 1 -- 172.21.15.163:0/1164284759 shutdown_connections 2024-02-19T21:05:41.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.145+0000 7f0393856700 1 -- 172.21.15.163:0/1164284759 >> 172.21.15.163:0/1164284759 conn(0x7f038c0f2470 msgr2=0x7f038c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:41.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.146+0000 7f0393856700 1 -- 172.21.15.163:0/1164284759 shutdown_connections 2024-02-19T21:05:41.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.146+0000 7f0393856700 1 -- 172.21.15.163:0/1164284759 wait complete. 2024-02-19T21:05:41.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.146+0000 7f0393856700 1 librados: starting msgr at 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.146+0000 7f0393856700 1 librados: starting objecter 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.146+0000 7f0393856700 1 Processor -- start 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.146+0000 7f0393856700 1 -- start start 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.146+0000 7f0393856700 1 librados: setting wanted keys 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.146+0000 7f0393856700 1 librados: calling monclient init 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.146+0000 7f0393856700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f038c117190 con 0x7f038c0f94b0 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.146+0000 7f0393856700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f038c118370 con 0x7f038c104290 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.146+0000 7f0393856700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f038c119550 con 0x7f038c0f89c0 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f0390df1700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f038c0f94b0 0x7f038c0ffa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:37090/0 (socket says 172.21.15.163:37090) 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f0390df1700 1 -- 172.21.15.163:0/3544451220 learned_addr learned my addr 172.21.15.163:0/3544451220 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 10 client.?.objecter ms_handle_connect 0x7f038c0f94b0 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 10 client.?.objecter ms_handle_connect 0x7f038c0f89c0 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 10 client.?.objecter ms_handle_connect 0x7f038c104290 2024-02-19T21:05:41.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:41.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800119643 0 0) 0x7f038c117190 con 0x7f038c0f94b0 2024-02-19T21:05:41.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0378003030 con 0x7f038c0f94b0 2024-02-19T21:05:41.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194371879 0 0) 0x7f038c119550 con 0x7f038c0f89c0 2024-02-19T21:05:41.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f038c117190 con 0x7f038c0f89c0 2024-02-19T21:05:41.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612863624 0 0) 0x7f038c118370 con 0x7f038c104290 2024-02-19T21:05:41.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f038c119550 con 0x7f038c104290 2024-02-19T21:05:41.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1560065788 0 0) 0x7f0378003030 con 0x7f038c0f94b0 2024-02-19T21:05:41.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f038c118370 con 0x7f038c0f94b0 2024-02-19T21:05:41.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1593627144 0 0) 0x7f038c117190 con 0x7f038c0f89c0 2024-02-19T21:05:41.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0378003030 con 0x7f038c0f89c0 2024-02-19T21:05:41.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f037c001990 con 0x7f038c0f94b0 2024-02-19T21:05:41.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0388000f10 con 0x7f038c0f89c0 2024-02-19T21:05:41.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 659608703 0 0) 0x7f038c119550 con 0x7f038c104290 2024-02-19T21:05:41.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f038c117190 con 0x7f038c104290 2024-02-19T21:05:41.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0384001b30 con 0x7f038c104290 2024-02-19T21:05:41.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 226597764 0 0) 0x7f038c118370 con 0x7f038c0f94b0 2024-02-19T21:05:41.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 >> v1:172.21.15.163:6790/0 conn(0x7f038c0f89c0 legacy=0x7f038c0ff370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:41.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 >> v1:172.21.15.193:6789/0 conn(0x7f038c104290 legacy=0x7f038c115910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:41.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038c11a750 con 0x7f038c0f94b0 2024-02-19T21:05:41.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f0393856700 1 -- 172.21.15.163:0/3544451220 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f038c119740 con 0x7f038c0f94b0 2024-02-19T21:05:41.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f0393856700 10 client.22278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:41.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f0393856700 1 -- 172.21.15.163:0/3544451220 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f038c119bb0 con 0x7f038c0f94b0 2024-02-19T21:05:41.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.147+0000 7f0393856700 1 librados: init done 2024-02-19T21:05:41.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.148+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f037c001b80 con 0x7f038c0f94b0 2024-02-19T21:05:41.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.148+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f037c005bc0 con 0x7f038c0f94b0 2024-02-19T21:05:41.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.148+0000 7f0393856700 1 -- 172.21.15.163:0/3544451220 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0370004f20 con 0x7f038c0f94b0 2024-02-19T21:05:41.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.149+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f037c002230 con 0x7f038c0f94b0 2024-02-19T21:05:41.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.150+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(160..160 src has 43..160) v4 ==== 9194+0+0 (unknown 3436134252 0 0) 0x7f037c0791d0 con 0x7f038c0f94b0 2024-02-19T21:05:41.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.150+0000 7f03827fc700 10 client.22278.objecter ms_dispatch 0x7f038c001550 osd_map(160..160 src has 43..160) v4 2024-02-19T21:05:41.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.150+0000 7f03827fc700 3 client.22278.objecter handle_osd_map got epochs [160,160] > 0 2024-02-19T21:05:41.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.150+0000 7f03827fc700 3 client.22278.objecter handle_osd_map decoding full epoch 160 2024-02-19T21:05:41.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.153+0000 7f03827fc700 20 client.22278.objecter dump_active .. 0 homeless 2024-02-19T21:05:41.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.153+0000 7f03827fc700 10 client.22278.objecter ms_handle_connect 0x7f0378063c40 2024-02-19T21:05:41.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.155+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f037c04c170 con 0x7f038c0f94b0 2024-02-19T21:05:41.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:41.379+0000 7f0393856700 1 -- 172.21.15.163:0/3544451220 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd pg-upmap", "pgid": "94.1c", "id": [4, 2, 0]} v 0) v1 -- 0x7f03700059e0 con 0x7f038c0f94b0 2024-02-19T21:05:41.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:39.888821+0000 osd.5 (osd.5) 84 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:05:41.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:39.892291+0000 osd.5 (osd.5) 85 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:05:41.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:39.946641+0000 osd.1 (osd.1) 97 : cluster [DBG] 5.c scrub starts 2024-02-19T21:05:41.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:39.947518+0000 osd.1 (osd.1) 98 : cluster [DBG] 5.c scrub ok 2024-02-19T21:05:41.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:40.136595+0000 mgr.y (mgr.14550) 209 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:41.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:41.129157+0000 mon.b (mon.1) 251 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-19T21:05:41.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:41.131167+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-19T21:05:41.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:41.177244+0000 mon.a (mon.0) 1209 : audit [INF] from='client.47698 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83952-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:41.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:41.177379+0000 mon.a (mon.0) 1210 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-19T21:05:41.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:41.177422+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-02-19T21:05:41.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:41.185081+0000 mon.b (mon.1) 252 : audit [INF] from='client.? v1:172.21.15.163:0/2621662847' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi163-83845-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:41.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:41.190804+0000 mon.a (mon.0) 1212 : audit [INF] from='client.47704 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi163-83845-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:41.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:41.469008+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? v1:172.21.15.163:0/3544451220' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "94.1c", "id": [4, 2, 0]}]: dispatch 2024-02-19T21:05:41.578 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:41.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:39.888821+0000 osd.5 (osd.5) 84 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:05:41.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:39.892291+0000 osd.5 (osd.5) 85 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:05:41.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:39.946641+0000 osd.1 (osd.1) 97 : cluster [DBG] 5.c scrub starts 2024-02-19T21:05:41.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:39.947518+0000 osd.1 (osd.1) 98 : cluster [DBG] 5.c scrub ok 2024-02-19T21:05:41.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:40.136595+0000 mgr.y (mgr.14550) 209 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:41.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:41.129157+0000 mon.b (mon.1) 251 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-19T21:05:41.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:41.131167+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-19T21:05:41.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:41.177244+0000 mon.a (mon.0) 1209 : audit [INF] from='client.47698 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83952-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:41.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:41.177379+0000 mon.a (mon.0) 1210 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-19T21:05:41.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:41.177422+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-02-19T21:05:41.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:41.185081+0000 mon.b (mon.1) 252 : audit [INF] from='client.? v1:172.21.15.163:0/2621662847' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi163-83845-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:41.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:41.190804+0000 mon.a (mon.0) 1212 : audit [INF] from='client.47704 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi163-83845-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:41.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:41.469008+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? v1:172.21.15.163:0/3544451220' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "94.1c", "id": [4, 2, 0]}]: dispatch 2024-02-19T21:05:41.703 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:05:41.704 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:05:41.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:39.888821+0000 osd.5 (osd.5) 84 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:05:41.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:39.892291+0000 osd.5 (osd.5) 85 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:05:41.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:39.946641+0000 osd.1 (osd.1) 97 : cluster [DBG] 5.c scrub starts 2024-02-19T21:05:41.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:39.947518+0000 osd.1 (osd.1) 98 : cluster [DBG] 5.c scrub ok 2024-02-19T21:05:41.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:40.136595+0000 mgr.y (mgr.14550) 209 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:41.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:41.129157+0000 mon.b (mon.1) 251 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-19T21:05:41.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:41.131167+0000 mon.a (mon.0) 1208 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-02-19T21:05:41.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:41.177244+0000 mon.a (mon.0) 1209 : audit [INF] from='client.47698 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83952-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:41.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:41.177379+0000 mon.a (mon.0) 1210 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-02-19T21:05:41.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:41.177422+0000 mon.a (mon.0) 1211 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-02-19T21:05:41.726 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:41.185081+0000 mon.b (mon.1) 252 : audit [INF] from='client.? v1:172.21.15.163:0/2621662847' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi163-83845-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:41.726 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:41.190804+0000 mon.a (mon.0) 1212 : audit [INF] from='client.47704 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi163-83845-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:41.726 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:41.469008+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? v1:172.21.15.163:0/3544451220' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "94.1c", "id": [4, 2, 0]}]: dispatch 2024-02-19T21:05:41.829 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:41.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:41.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:41.933 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:41.964 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:42.088 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:42.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:42.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:42.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.181+0000 7f03827fc700 1 -- 172.21.15.163:0/3544451220 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "94.1c", "id": [4, 2, 0]}]=0 set 94.1c pg_upmap mapping to [4,2,0] v162) v1 ==== 131+0+0 (unknown 2465207181 0 0) 0x7f037c04a580 con 0x7f038c0f94b0 2024-02-19T21:05:42.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.184+0000 7f0393856700 10 librados: watch_flush enter 2024-02-19T21:05:42.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.190+0000 7f0393856700 10 librados: watch_flush exit 2024-02-19T21:05:42.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.190+0000 7f0393856700 20 client.22278.objecter shutdown clearing up homeless session... 2024-02-19T21:05:42.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.190+0000 7f0393856700 10 client.22278.objecter successfully canceled tick 2024-02-19T21:05:42.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.190+0000 7f0393856700 1 -- 172.21.15.163:0/3544451220 >> v1:172.21.15.163:6800/1552514681 conn(0x7f0378063c40 legacy=0x7f0378066100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:42.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.190+0000 7f0393856700 1 -- 172.21.15.163:0/3544451220 >> v1:172.21.15.163:6789/0 conn(0x7f038c0f94b0 legacy=0x7f038c0ffa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:42.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.195+0000 7f0393856700 1 -- 172.21.15.163:0/3544451220 shutdown_connections 2024-02-19T21:05:42.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.195+0000 7f0393856700 1 -- 172.21.15.163:0/3544451220 >> 172.21.15.163:0/3544451220 conn(0x7f038c0f2470 msgr2=0x7f038c107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:42.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.204+0000 7f0393856700 1 -- 172.21.15.163:0/3544451220 shutdown_connections 2024-02-19T21:05:42.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.205+0000 7f0393856700 1 -- 172.21.15.163:0/3544451220 wait complete. 2024-02-19T21:05:42.203 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:05:42.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.208+0000 7f0393856700 1 librados: shutdown 2024-02-19T21:05:42.206 INFO:teuthology.orchestra.run.smithi163.stderr:set 94.1c pg_upmap mapping to [4,2,0] 2024-02-19T21:05:42.231 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:42.492 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:40.916332+0000 osd.6 (osd.6) 75 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:05:42.512 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-19T21:05:42.579 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:05:42.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:42.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:05:42.683 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:42.712 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:42.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:40.916332+0000 osd.6 (osd.6) 75 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:05:42.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:40.919256+0000 osd.6 (osd.6) 76 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:05:42.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:41.128535+0000 mgr.y (mgr.14550) 210 : cluster [DBG] pgmap v212: 366 pgs: 32 creating+peering, 1 active+clean+scrubbing, 32 unknown, 301 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 319 B/s, 4 objects/s recovering 2024-02-19T21:05:42.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:41.532154+0000 osd.4 (osd.4) 85 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:05:42.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:41.532838+0000 osd.4 (osd.4) 86 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:05:42.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:42.179903+0000 mon.a (mon.0) 1214 : audit [INF] from='client.47704 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi163-83845-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:42.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:42.180208+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? v1:172.21.15.163:0/3544451220' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "94.1c", "id": [4, 2, 0]}]': finished 2024-02-19T21:05:42.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:42.180269+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-02-19T21:05:42.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:42.195690+0000 mon.b (mon.1) 253 : audit [INF] from='client.? v1:172.21.15.163:0/811617816' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:42.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:42.196471+0000 mon.a (mon.0) 1217 : audit [INF] from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:42.822 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:05:42.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.903+0000 7f656cc3a700 1 -- 172.21.15.163:0/3162712726 >> v1:172.21.15.163:6790/0 conn(0x7f65681031b0 legacy=0x7f6568105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:42.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.903+0000 7f656cc3a700 1 -- 172.21.15.163:0/3162712726 shutdown_connections 2024-02-19T21:05:42.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.903+0000 7f656cc3a700 1 -- 172.21.15.163:0/3162712726 >> 172.21.15.163:0/3162712726 conn(0x7f65680f2470 msgr2=0x7f65680f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:42.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.903+0000 7f656cc3a700 1 -- 172.21.15.163:0/3162712726 shutdown_connections 2024-02-19T21:05:42.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.903+0000 7f656cc3a700 1 -- 172.21.15.163:0/3162712726 wait complete. 2024-02-19T21:05:42.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.903+0000 7f656cc3a700 1 librados: starting msgr at 2024-02-19T21:05:42.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.903+0000 7f656cc3a700 1 librados: starting objecter 2024-02-19T21:05:42.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f656cc3a700 1 Processor -- start 2024-02-19T21:05:42.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f656cc3a700 1 -- start start 2024-02-19T21:05:42.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f656cc3a700 1 librados: setting wanted keys 2024-02-19T21:05:42.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f656cc3a700 1 librados: calling monclient init 2024-02-19T21:05:42.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f656cc3a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f65681163a0 con 0x7f65681031b0 2024-02-19T21:05:42.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f656cc3a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6568117580 con 0x7f65680fc870 2024-02-19T21:05:42.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f656cc3a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6568118780 con 0x7f65680ff670 2024-02-19T21:05:42.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f6565d9b700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f65680ff670 0x7f65680f92a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:59736/0 (socket says 172.21.15.163:59736) 2024-02-19T21:05:42.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f6565d9b700 1 -- 172.21.15.163:0/1634422608 learned_addr learned my addr 172.21.15.163:0/1634422608 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:42.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f6566d9d700 1 --1- 172.21.15.163:0/1634422608 >> v1:172.21.15.163:6789/0 conn(0x7f65681031b0 0x7f6568114a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:37166/0 (socket says 172.21.15.163:37166) 2024-02-19T21:05:42.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f65577fe700 10 client.?.objecter ms_handle_connect 0x7f65681031b0 2024-02-19T21:05:42.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f65577fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:42.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f65577fe700 10 client.?.objecter ms_handle_connect 0x7f65680ff670 2024-02-19T21:05:42.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f65577fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:42.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f65577fe700 10 client.?.objecter ms_handle_connect 0x7f65680fc870 2024-02-19T21:05:42.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f65577fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:42.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430339858 0 0) 0x7f65681163a0 con 0x7f65681031b0 2024-02-19T21:05:42.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65500033f0 con 0x7f65681031b0 2024-02-19T21:05:42.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.904+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648491354 0 0) 0x7f6568118780 con 0x7f65680ff670 2024-02-19T21:05:42.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65681163a0 con 0x7f65680ff670 2024-02-19T21:05:42.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063083531 0 0) 0x7f6568117580 con 0x7f65680fc870 2024-02-19T21:05:42.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6568118780 con 0x7f65680fc870 2024-02-19T21:05:42.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 647314887 0 0) 0x7f65500033f0 con 0x7f65681031b0 2024-02-19T21:05:42.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6568117580 con 0x7f65681031b0 2024-02-19T21:05:42.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2247670298 0 0) 0x7f6568118780 con 0x7f65680fc870 2024-02-19T21:05:42.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65500033f0 con 0x7f65680fc870 2024-02-19T21:05:42.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2173243306 0 0) 0x7f65681163a0 con 0x7f65680ff670 2024-02-19T21:05:42.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6568118780 con 0x7f65680ff670 2024-02-19T21:05:42.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f65600030e0 con 0x7f65681031b0 2024-02-19T21:05:42.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6558001730 con 0x7f65680fc870 2024-02-19T21:05:42.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f655c0018c0 con 0x7f65680ff670 2024-02-19T21:05:42.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 814742127 0 0) 0x7f65500033f0 con 0x7f65680fc870 2024-02-19T21:05:42.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 >> v1:172.21.15.163:6790/0 conn(0x7f65680ff670 legacy=0x7f65680f92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:42.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 >> v1:172.21.15.163:6789/0 conn(0x7f65681031b0 legacy=0x7f6568114a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:42.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6568119980 con 0x7f65680fc870 2024-02-19T21:05:42.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f656cc3a700 1 -- 172.21.15.163:0/1634422608 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6568118970 con 0x7f65680fc870 2024-02-19T21:05:42.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f656cc3a700 10 client.47731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:42.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f656cc3a700 1 -- 172.21.15.163:0/1634422608 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6568118de0 con 0x7f65680fc870 2024-02-19T21:05:42.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f656cc3a700 1 librados: init done 2024-02-19T21:05:42.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6558002120 con 0x7f65680fc870 2024-02-19T21:05:42.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.905+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f65580031a0 con 0x7f65680fc870 2024-02-19T21:05:42.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.906+0000 7f656cc3a700 1 -- 172.21.15.163:0/1634422608 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6548004f20 con 0x7f65680fc870 2024-02-19T21:05:42.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.906+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6558001c80 con 0x7f65680fc870 2024-02-19T21:05:42.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.907+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(162..162 src has 43..162) v4 ==== 9611+0+0 (unknown 3871360327 0 0) 0x7f6558076840 con 0x7f65680fc870 2024-02-19T21:05:42.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.907+0000 7f65577fe700 10 client.47731.objecter ms_dispatch 0x7f6568001550 osd_map(162..162 src has 43..162) v4 2024-02-19T21:05:42.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.907+0000 7f65577fe700 3 client.47731.objecter handle_osd_map got epochs [162,162] > 0 2024-02-19T21:05:42.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.907+0000 7f65577fe700 3 client.47731.objecter handle_osd_map decoding full epoch 162 2024-02-19T21:05:42.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.907+0000 7f65577fe700 20 client.47731.objecter dump_active .. 0 homeless 2024-02-19T21:05:42.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.907+0000 7f65577fe700 10 client.47731.objecter ms_handle_connect 0x7f6550063c80 2024-02-19T21:05:42.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:42.911+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6558049620 con 0x7f65680fc870 2024-02-19T21:05:42.911 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:42.913 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:40.916332+0000 osd.6 (osd.6) 75 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:05:42.913 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:40.919256+0000 osd.6 (osd.6) 76 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:05:42.913 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:41.128535+0000 mgr.y (mgr.14550) 210 : cluster [DBG] pgmap v212: 366 pgs: 32 creating+peering, 1 active+clean+scrubbing, 32 unknown, 301 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 319 B/s, 4 objects/s recovering 2024-02-19T21:05:42.914 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:41.532154+0000 osd.4 (osd.4) 85 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:05:42.914 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:41.532838+0000 osd.4 (osd.4) 86 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:05:42.914 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:42.179903+0000 mon.a (mon.0) 1214 : audit [INF] from='client.47704 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi163-83845-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:42.914 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:42.180208+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? v1:172.21.15.163:0/3544451220' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "94.1c", "id": [4, 2, 0]}]': finished 2024-02-19T21:05:42.914 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:42.180269+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-02-19T21:05:42.914 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:42.195690+0000 mon.b (mon.1) 253 : audit [INF] from='client.? v1:172.21.15.163:0/811617816' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:42.914 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:42.196471+0000 mon.a (mon.0) 1217 : audit [INF] from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:42.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:40.919256+0000 osd.6 (osd.6) 76 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:05:42.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:41.128535+0000 mgr.y (mgr.14550) 210 : cluster [DBG] pgmap v212: 366 pgs: 32 creating+peering, 1 active+clean+scrubbing, 32 unknown, 301 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 319 B/s, 4 objects/s recovering 2024-02-19T21:05:42.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:41.532154+0000 osd.4 (osd.4) 85 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:05:42.915 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:41.532838+0000 osd.4 (osd.4) 86 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:05:42.915 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:42.179903+0000 mon.a (mon.0) 1214 : audit [INF] from='client.47704 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi163-83845-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:42.915 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:42.180208+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? v1:172.21.15.163:0/3544451220' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "94.1c", "id": [4, 2, 0]}]': finished 2024-02-19T21:05:42.915 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:42.180269+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-02-19T21:05:42.915 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:42.195690+0000 mon.b (mon.1) 253 : audit [INF] from='client.? v1:172.21.15.163:0/811617816' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:42.915 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:42.196471+0000 mon.a (mon.0) 1217 : audit [INF] from='client.47719 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:42.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:42.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:43.017 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:05:43.044 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:43.064 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:05:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:43.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:05:43.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:43.090+0000 7f656cc3a700 1 -- 172.21.15.163:0/1634422608 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f65480051d0 con 0x7f65680fc870 2024-02-19T21:05:43.168 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:05:43.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:43.183+0000 7f65577fe700 1 -- 172.21.15.163:0/1634422608 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v163) v1 ==== 101+0+0 (unknown 2915950811 0 0) 0x7f6558047a30 con 0x7f65680fc870 2024-02-19T21:05:43.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:43.190+0000 7f656cc3a700 10 librados: watch_flush enter 2024-02-19T21:05:43.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:43.191+0000 7f656cc3a700 10 librados: watch_flush exit 2024-02-19T21:05:43.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:43.191+0000 7f656cc3a700 20 client.47731.objecter shutdown clearing up homeless session... 2024-02-19T21:05:43.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:43.191+0000 7f656cc3a700 10 client.47731.objecter successfully canceled tick 2024-02-19T21:05:43.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:43.195+0000 7f656cc3a700 1 -- 172.21.15.163:0/1634422608 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6550063c80 legacy=0x7f6550066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:43.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:43.195+0000 7f656cc3a700 1 -- 172.21.15.163:0/1634422608 >> v1:172.21.15.193:6789/0 conn(0x7f65680fc870 legacy=0x7f65680f8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:43.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:43.206+0000 7f656cc3a700 1 -- 172.21.15.163:0/1634422608 shutdown_connections 2024-02-19T21:05:43.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:43.206+0000 7f656cc3a700 1 -- 172.21.15.163:0/1634422608 >> 172.21.15.163:0/1634422608 conn(0x7f65680f2470 msgr2=0x7f6568106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:43.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:43.213+0000 7f656cc3a700 1 -- 172.21.15.163:0/1634422608 shutdown_connections 2024-02-19T21:05:43.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:43.214+0000 7f656cc3a700 1 -- 172.21.15.163:0/1634422608 wait complete. 2024-02-19T21:05:43.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:43.215+0000 7f656cc3a700 1 librados: shutdown 2024-02-19T21:05:43.214 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:05:43.241 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:43.341 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:05:43.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:43.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:05:43.445 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:43.473 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:43.491 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:05:43] "GET /metrics HTTP/1.1" 200 194990 "" "Prometheus/2.33.4" 2024-02-19T21:05:43.595 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:43.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:43.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:43.699 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:05:43.724 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:41.908784+0000 osd.6 (osd.6) 77 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:05:43.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:41.909790+0000 osd.6 (osd.6) 78 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:05:43.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:42.876107+0000 mon.b (mon.1) 254 : audit [DBG] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-19T21:05:43.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:42.876702+0000 mon.b (mon.1) 255 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:43.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:42.876989+0000 mon.a (mon.0) 1218 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:43.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:43.092173+0000 mon.b (mon.1) 256 : audit [INF] from='client.? v1:172.21.15.163:0/1634422608' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:43.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:43.092910+0000 mon.a (mon.0) 1219 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:43.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:43.176568+0000 mon.a (mon.0) 1220 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:05:43.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:43.180699+0000 mon.a (mon.0) 1221 : audit [INF] from='client.47719 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:43.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:43.180819+0000 mon.a (mon.0) 1222 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:43.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:43.180878+0000 mon.a (mon.0) 1223 : audit [INF] from='client.47731 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:05:43.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:43.180925+0000 mon.a (mon.0) 1224 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-02-19T21:05:43.725 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:43.195503+0000 mon.b (mon.1) 257 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app2"}]: dispatch 2024-02-19T21:05:43.726 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:43.197241+0000 mon.b (mon.1) 258 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:43.726 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:43.209119+0000 mon.a (mon.0) 1225 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:43.726 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:43.799 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:05:43.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:43.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:05:43.908 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:05:43.934 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:43.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:41.908784+0000 osd.6 (osd.6) 77 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:05:43.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:41.909790+0000 osd.6 (osd.6) 78 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:05:43.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:42.876107+0000 mon.b (mon.1) 254 : audit [DBG] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-19T21:05:43.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:42.876702+0000 mon.b (mon.1) 255 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:43.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:42.876989+0000 mon.a (mon.0) 1218 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:43.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:43.092173+0000 mon.b (mon.1) 256 : audit [INF] from='client.? v1:172.21.15.163:0/1634422608' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:43.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:43.092910+0000 mon.a (mon.0) 1219 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:43.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:43.176568+0000 mon.a (mon.0) 1220 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:05:43.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:43.180699+0000 mon.a (mon.0) 1221 : audit [INF] from='client.47719 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:43.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:43.180819+0000 mon.a (mon.0) 1222 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:43.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:43.180878+0000 mon.a (mon.0) 1223 : audit [INF] from='client.47731 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:05:43.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:43.180925+0000 mon.a (mon.0) 1224 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-02-19T21:05:43.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:43.195503+0000 mon.b (mon.1) 257 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app2"}]: dispatch 2024-02-19T21:05:43.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:43.197241+0000 mon.b (mon.1) 258 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:43.984 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:43.209119+0000 mon.a (mon.0) 1225 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:43.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:41.908784+0000 osd.6 (osd.6) 77 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:05:43.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:41.909790+0000 osd.6 (osd.6) 78 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:05:43.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:42.876107+0000 mon.b (mon.1) 254 : audit [DBG] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-19T21:05:43.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:42.876702+0000 mon.b (mon.1) 255 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:43.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:42.876989+0000 mon.a (mon.0) 1218 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:43.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:43.092173+0000 mon.b (mon.1) 256 : audit [INF] from='client.? v1:172.21.15.163:0/1634422608' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:43.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:43.092910+0000 mon.a (mon.0) 1219 : audit [INF] from='client.47731 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:43.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:43.176568+0000 mon.a (mon.0) 1220 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:05:43.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:43.180699+0000 mon.a (mon.0) 1221 : audit [INF] from='client.47719 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:43.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:43.180819+0000 mon.a (mon.0) 1222 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:43.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:43.180878+0000 mon.a (mon.0) 1223 : audit [INF] from='client.47731 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:05:43.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:43.180925+0000 mon.a (mon.0) 1224 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-02-19T21:05:43.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:43.195503+0000 mon.b (mon.1) 257 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app2"}]: dispatch 2024-02-19T21:05:43.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:43.197241+0000 mon.b (mon.1) 258 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:43.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:43.209119+0000 mon.a (mon.0) 1225 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:44.483 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:05:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:44.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:05:44.588 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:05:44.617 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:44.718 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:05:44.718 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:05:44.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:42.774680+0000 osd.0 (osd.0) 91 : cluster [DBG] 94.9 scrub starts 2024-02-19T21:05:44.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:42.778411+0000 osd.0 (osd.0) 92 : cluster [DBG] 94.9 scrub ok 2024-02-19T21:05:44.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:42.926069+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:05:44.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:42.927635+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:05:44.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:43.128998+0000 mgr.y (mgr.14550) 211 : cluster [DBG] pgmap v215: 398 pgs: 1 active+clean+scrubbing, 96 unknown, 301 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 319 B/s, 4 objects/s recovering 2024-02-19T21:05:44.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:43.358871+0000 osd.7 (osd.7) 89 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:05:44.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:43.361773+0000 osd.7 (osd.7) 90 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:05:44.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:44.166181+0000 mon.a (mon.0) 1226 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:44.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:44.166319+0000 mon.a (mon.0) 1227 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-02-19T21:05:44.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:44.186308+0000 mon.b (mon.1) 259 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-02-19T21:05:44.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:44.186917+0000 mon.b (mon.1) 260 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-19T21:05:44.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:44.190174+0000 mon.a (mon.0) 1228 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-19T21:05:44.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:44.193242+0000 mon.b (mon.1) 261 : audit [INF] from='client.? v1:172.21.15.163:0/1822655164' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi163-83845-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:44.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:44.195032+0000 mon.a (mon.0) 1229 : audit [INF] from='client.47740 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi163-83845-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:44.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:44.225535+0000 mon.c (mon.2) 152 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:44.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:42.774680+0000 osd.0 (osd.0) 91 : cluster [DBG] 94.9 scrub starts 2024-02-19T21:05:44.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:42.778411+0000 osd.0 (osd.0) 92 : cluster [DBG] 94.9 scrub ok 2024-02-19T21:05:44.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:42.926069+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:05:44.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:42.927635+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:05:44.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:43.128998+0000 mgr.y (mgr.14550) 211 : cluster [DBG] pgmap v215: 398 pgs: 1 active+clean+scrubbing, 96 unknown, 301 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 319 B/s, 4 objects/s recovering 2024-02-19T21:05:44.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:43.358871+0000 osd.7 (osd.7) 89 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:05:44.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:43.361773+0000 osd.7 (osd.7) 90 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:05:44.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:44.166181+0000 mon.a (mon.0) 1226 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:44.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:44.166319+0000 mon.a (mon.0) 1227 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-02-19T21:05:44.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:44.186308+0000 mon.b (mon.1) 259 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-02-19T21:05:44.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:44.186917+0000 mon.b (mon.1) 260 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-19T21:05:44.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:44.190174+0000 mon.a (mon.0) 1228 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-19T21:05:44.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:44.193242+0000 mon.b (mon.1) 261 : audit [INF] from='client.? v1:172.21.15.163:0/1822655164' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi163-83845-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:44.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:44.195032+0000 mon.a (mon.0) 1229 : audit [INF] from='client.47740 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi163-83845-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:44.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:44.225535+0000 mon.c (mon.2) 152 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:44.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:44.328635+0000 osd.7 (osd.7) 91 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:05:44.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:44.331543+0000 osd.7 (osd.7) 92 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:05:44.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:44.454819+0000 mon.a (mon.0) 1230 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:44.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:44.455410+0000 mon.c (mon.2) 153 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:44.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:44.455652+0000 mon.a (mon.0) 1231 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:44.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:44.456006+0000 mon.c (mon.2) 154 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:44.456236+0000 mon.a (mon.0) 1232 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:44.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:44.328635+0000 osd.7 (osd.7) 91 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:05:44.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:44.331543+0000 osd.7 (osd.7) 92 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:05:44.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:44.454819+0000 mon.a (mon.0) 1230 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:44.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:44.455410+0000 mon.c (mon.2) 153 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:44.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:44.455652+0000 mon.a (mon.0) 1231 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:44.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:44.456006+0000 mon.c (mon.2) 154 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:44.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:44.456236+0000 mon.a (mon.0) 1232 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:44.746 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:42.774680+0000 osd.0 (osd.0) 91 : cluster [DBG] 94.9 scrub starts 2024-02-19T21:05:44.746 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:42.778411+0000 osd.0 (osd.0) 92 : cluster [DBG] 94.9 scrub ok 2024-02-19T21:05:44.746 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:42.926069+0000 osd.1 (osd.1) 99 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:05:44.746 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:42.927635+0000 osd.1 (osd.1) 100 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:05:44.746 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:43.128998+0000 mgr.y (mgr.14550) 211 : cluster [DBG] pgmap v215: 398 pgs: 1 active+clean+scrubbing, 96 unknown, 301 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 319 B/s, 4 objects/s recovering 2024-02-19T21:05:44.746 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:43.358871+0000 osd.7 (osd.7) 89 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:05:44.746 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:43.361773+0000 osd.7 (osd.7) 90 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:05:44.746 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:44.166181+0000 mon.a (mon.0) 1226 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi163-83920-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:44.746 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:44.166319+0000 mon.a (mon.0) 1227 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-02-19T21:05:44.746 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:44.186308+0000 mon.b (mon.1) 259 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-02-19T21:05:44.747 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:44.186917+0000 mon.b (mon.1) 260 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-19T21:05:44.747 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:44.190174+0000 mon.a (mon.0) 1228 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-02-19T21:05:44.747 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:44.193242+0000 mon.b (mon.1) 261 : audit [INF] from='client.? v1:172.21.15.163:0/1822655164' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi163-83845-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:44.747 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:44.195032+0000 mon.a (mon.0) 1229 : audit [INF] from='client.47740 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi163-83845-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:44.747 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:44.225535+0000 mon.c (mon.2) 152 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:44.747 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:44.328635+0000 osd.7 (osd.7) 91 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:05:44.747 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:44.331543+0000 osd.7 (osd.7) 92 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:05:44.747 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:44.454819+0000 mon.a (mon.0) 1230 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:44.747 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:44.455410+0000 mon.c (mon.2) 153 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:44.747 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:44.455652+0000 mon.a (mon.0) 1231 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:44.747 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:44.456006+0000 mon.c (mon.2) 154 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:44.748 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:44.456236+0000 mon.a (mon.0) 1232 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:44.816 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:05:44.818 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:44.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:05:44.919 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:44.949 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:44.968 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:44.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:44.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:45.071 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:45.097 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:45.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.420+0000 7fbf7168a700 1 -- 172.21.15.163:0/3411135609 >> v1:172.21.15.163:6790/0 conn(0x7fbf6c0f7270 legacy=0x7fbf6c104e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:45.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.420+0000 7fbf7168a700 1 -- 172.21.15.163:0/3411135609 shutdown_connections 2024-02-19T21:05:45.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.420+0000 7fbf7168a700 1 -- 172.21.15.163:0/3411135609 >> 172.21.15.163:0/3411135609 conn(0x7fbf6c0f24a0 msgr2=0x7fbf6c0f48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:45.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.420+0000 7fbf7168a700 1 -- 172.21.15.163:0/3411135609 shutdown_connections 2024-02-19T21:05:45.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.420+0000 7fbf7168a700 1 -- 172.21.15.163:0/3411135609 wait complete. 2024-02-19T21:05:45.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.421+0000 7fbf7168a700 1 librados: starting msgr at 2024-02-19T21:05:45.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.421+0000 7fbf7168a700 1 librados: starting objecter 2024-02-19T21:05:45.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.421+0000 7fbf7168a700 1 Processor -- start 2024-02-19T21:05:45.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.421+0000 7fbf7168a700 1 -- start start 2024-02-19T21:05:45.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.421+0000 7fbf7168a700 1 librados: setting wanted keys 2024-02-19T21:05:45.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.421+0000 7fbf7168a700 1 librados: calling monclient init 2024-02-19T21:05:45.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.421+0000 7fbf7168a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbf6c119630 con 0x7fbf6c0f7270 2024-02-19T21:05:45.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.421+0000 7fbf7168a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbf6c11a810 con 0x7fbf6c0f6780 2024-02-19T21:05:45.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.421+0000 7fbf7168a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbf6c11ba10 con 0x7fbf6c1065b0 2024-02-19T21:05:45.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.421+0000 7fbf6b7fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fbf6c1065b0 0x7fbf6c117cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38706/0 (socket says 172.21.15.163:38706) 2024-02-19T21:05:45.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.421+0000 7fbf6a7fc700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fbf6c0f7270 0x7fbf6c103320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:42074/0 (socket says 172.21.15.163:42074) 2024-02-19T21:05:45.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.421+0000 7fbf6b7fe700 1 -- 172.21.15.163:0/1240681493 learned_addr learned my addr 172.21.15.163:0/1240681493 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:45.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 10 client.?.objecter ms_handle_connect 0x7fbf6c1065b0 2024-02-19T21:05:45.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:45.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 10 client.?.objecter ms_handle_connect 0x7fbf6c0f7270 2024-02-19T21:05:45.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:45.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 10 client.?.objecter ms_handle_connect 0x7fbf6c0f6780 2024-02-19T21:05:45.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:45.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030426970 0 0) 0x7fbf6c11a810 con 0x7fbf6c0f6780 2024-02-19T21:05:45.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf54003030 con 0x7fbf6c0f6780 2024-02-19T21:05:45.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928816052 0 0) 0x7fbf6c119630 con 0x7fbf6c0f7270 2024-02-19T21:05:45.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf6c11a810 con 0x7fbf6c0f7270 2024-02-19T21:05:45.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648779744 0 0) 0x7fbf6c11ba10 con 0x7fbf6c1065b0 2024-02-19T21:05:45.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf6c119630 con 0x7fbf6c1065b0 2024-02-19T21:05:45.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2940340050 0 0) 0x7fbf54003030 con 0x7fbf6c0f6780 2024-02-19T21:05:45.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf6c11ba10 con 0x7fbf6c0f6780 2024-02-19T21:05:45.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 814881501 0 0) 0x7fbf6c11a810 con 0x7fbf6c0f7270 2024-02-19T21:05:45.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf54003030 con 0x7fbf6c0f7270 2024-02-19T21:05:45.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1207333664 0 0) 0x7fbf6c119630 con 0x7fbf6c1065b0 2024-02-19T21:05:45.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf6c11a810 con 0x7fbf6c1065b0 2024-02-19T21:05:45.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbf5c001800 con 0x7fbf6c0f6780 2024-02-19T21:05:45.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbf60002b90 con 0x7fbf6c0f7270 2024-02-19T21:05:45.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbf64001be0 con 0x7fbf6c1065b0 2024-02-19T21:05:45.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4294248738 0 0) 0x7fbf54003030 con 0x7fbf6c0f7270 2024-02-19T21:05:45.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 >> v1:172.21.15.163:6790/0 conn(0x7fbf6c1065b0 legacy=0x7fbf6c117cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:45.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 >> v1:172.21.15.193:6789/0 conn(0x7fbf6c0f6780 legacy=0x7fbf6c102c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:45.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c11cc10 con 0x7fbf6c0f7270 2024-02-19T21:05:45.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6c11aa00 con 0x7fbf6c0f7270 2024-02-19T21:05:45.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf7168a700 10 client.22323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:45.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6c11aef0 con 0x7fbf6c0f7270 2024-02-19T21:05:45.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf7168a700 1 librados: init done 2024-02-19T21:05:45.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.422+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fbf60001aa0 con 0x7fbf6c0f7270 2024-02-19T21:05:45.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.423+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbf600035e0 con 0x7fbf6c0f7270 2024-02-19T21:05:45.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.423+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fbf58004f00 con 0x7fbf6c0f7270 2024-02-19T21:05:45.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.424+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fbf60001eb0 con 0x7fbf6c0f7270 2024-02-19T21:05:45.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.424+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(165..165 src has 43..165) v4 ==== 9592+0+0 (unknown 4191456400 0 0) 0x7fbf60079390 con 0x7fbf6c0f7270 2024-02-19T21:05:45.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.424+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 osd_map(165..165 src has 43..165) v4 2024-02-19T21:05:45.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.424+0000 7fbf53fff700 3 client.22323.objecter handle_osd_map got epochs [165,165] > 0 2024-02-19T21:05:45.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.424+0000 7fbf53fff700 3 client.22323.objecter handle_osd_map decoding full epoch 165 2024-02-19T21:05:45.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.424+0000 7fbf53fff700 20 client.22323.objecter dump_active .. 0 homeless 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.424+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v165) v1 ==== 54+0+16 (unknown 1685745335 0 2420000192) 0x7fbf6007b7a0 con 0x7fbf6c0f7270 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.424+0000 7fbf53fff700 10 client.22323.objecter ms_handle_connect 0x7fbf54063ff0 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf7168a700 10 client.22323.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 1 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf580051a0 osd=0 3 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf580051a0 osd=0 3 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf580051a0 osd=0 3 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 1 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf580051a0 osd=0 3 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 0 1 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf580051a0 osd=0 4 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf7168a700 10 client.22323.objecter _send_command 1 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf58004950 con 0x7fbf58004530 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf53fff700 10 client.22323.objecter ms_handle_connect 0x7fbf58004530 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf58004950 con 0x7fbf58004530 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:05:45.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf53fff700 10 client.22323.objecter _finish_command 1 = system:0 2024-02-19T21:05:45.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf580051a0 osd=0 4 2024-02-19T21:05:45.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.425+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 0 1 2024-02-19T21:05:45.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.440+0000 7fbf7168a700 10 client.22323.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:45.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.440+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 2 2024-02-19T21:05:45.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.440+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf580051a0 osd=0 3 2024-02-19T21:05:45.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.440+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf580051a0 osd=0 3 2024-02-19T21:05:45.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.440+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf580051a0 osd=0 3 2024-02-19T21:05:45.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.440+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 2 2024-02-19T21:05:45.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.440+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf580051a0 osd=0 3 2024-02-19T21:05:45.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.440+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 0 2 2024-02-19T21:05:45.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.440+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf580051a0 osd=0 4 2024-02-19T21:05:45.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.440+0000 7fbf7168a700 10 client.22323.objecter _send_command 2 2024-02-19T21:05:45.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.440+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf58004d30 con 0x7fbf58004530 2024-02-19T21:05:45.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.441+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf58004d30 con 0x7fbf58004530 2024-02-19T21:05:45.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.441+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:45.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.441+0000 7fbf53fff700 10 client.22323.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.441+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf580051a0 osd=0 4 2024-02-19T21:05:45.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.441+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 0 2 2024-02-19T21:05:45.437 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:05:45.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf7168a700 10 client.22323.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:45.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 3 2024-02-19T21:05:45.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58003fe0 osd=1 3 2024-02-19T21:05:45.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58003fe0 osd=1 3 2024-02-19T21:05:45.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58003fe0 osd=1 3 2024-02-19T21:05:45.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 3 2024-02-19T21:05:45.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf58003fe0 osd=1 3 2024-02-19T21:05:45.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 1 3 2024-02-19T21:05:45.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58003fe0 osd=1 4 2024-02-19T21:05:45.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf7168a700 10 client.22323.objecter _send_command 3 2024-02-19T21:05:45.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf5800c000 con 0x7fbf58004150 2024-02-19T21:05:45.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf53fff700 10 client.22323.objecter ms_handle_connect 0x7fbf58004150 2024-02-19T21:05:45.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf5800c000 con 0x7fbf58004150 2024-02-19T21:05:45.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:05:45.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf53fff700 10 client.22323.objecter _finish_command 3 = system:0 2024-02-19T21:05:45.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf58003fe0 osd=1 4 2024-02-19T21:05:45.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.442+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 1 3 2024-02-19T21:05:45.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.457+0000 7fbf7168a700 10 client.22323.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:45.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.457+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 4 2024-02-19T21:05:45.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.457+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58003fe0 osd=1 3 2024-02-19T21:05:45.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.457+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58003fe0 osd=1 3 2024-02-19T21:05:45.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.457+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58003fe0 osd=1 3 2024-02-19T21:05:45.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.457+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 4 2024-02-19T21:05:45.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.457+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf58003fe0 osd=1 3 2024-02-19T21:05:45.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.457+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 1 4 2024-02-19T21:05:45.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.457+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58003fe0 osd=1 4 2024-02-19T21:05:45.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.457+0000 7fbf7168a700 10 client.22323.objecter _send_command 4 2024-02-19T21:05:45.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.457+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf580018b0 con 0x7fbf58004150 2024-02-19T21:05:45.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf580018b0 con 0x7fbf58004150 2024-02-19T21:05:45.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:45.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf53fff700 10 client.22323.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf58003fe0 osd=1 4 2024-02-19T21:05:45.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 1 4 2024-02-19T21:05:45.453 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:05:45.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf7168a700 10 client.22323.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:45.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 5 2024-02-19T21:05:45.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58003630 osd=2 3 2024-02-19T21:05:45.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58003630 osd=2 3 2024-02-19T21:05:45.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58003630 osd=2 3 2024-02-19T21:05:45.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 5 2024-02-19T21:05:45.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf58003630 osd=2 3 2024-02-19T21:05:45.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 2 5 2024-02-19T21:05:45.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58003630 osd=2 4 2024-02-19T21:05:45.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf7168a700 10 client.22323.objecter _send_command 5 2024-02-19T21:05:45.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.458+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf5800de90 con 0x7fbf5800d2f0 2024-02-19T21:05:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.459+0000 7fbf53fff700 10 client.22323.objecter ms_handle_connect 0x7fbf5800d2f0 2024-02-19T21:05:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.459+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf5800de90 con 0x7fbf5800d2f0 2024-02-19T21:05:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.459+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:05:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.459+0000 7fbf53fff700 10 client.22323.objecter _finish_command 5 = system:0 2024-02-19T21:05:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.459+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf58003630 osd=2 4 2024-02-19T21:05:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.459+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 2 5 2024-02-19T21:05:45.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.474+0000 7fbf7168a700 10 client.22323.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:45.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.474+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 6 2024-02-19T21:05:45.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.474+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58003630 osd=2 3 2024-02-19T21:05:45.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.474+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58003630 osd=2 3 2024-02-19T21:05:45.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.474+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58003630 osd=2 3 2024-02-19T21:05:45.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.474+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 6 2024-02-19T21:05:45.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.474+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf58003630 osd=2 3 2024-02-19T21:05:45.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.474+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 2 6 2024-02-19T21:05:45.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.474+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58003630 osd=2 4 2024-02-19T21:05:45.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.474+0000 7fbf7168a700 10 client.22323.objecter _send_command 6 2024-02-19T21:05:45.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.474+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf580030a0 con 0x7fbf5800d2f0 2024-02-19T21:05:45.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf580030a0 con 0x7fbf5800d2f0 2024-02-19T21:05:45.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:45.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf53fff700 10 client.22323.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf58003630 osd=2 4 2024-02-19T21:05:45.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 2 6 2024-02-19T21:05:45.470 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:05:45.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf7168a700 10 client.22323.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:45.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 7 2024-02-19T21:05:45.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58011970 osd=3 3 2024-02-19T21:05:45.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58011970 osd=3 3 2024-02-19T21:05:45.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58011970 osd=3 3 2024-02-19T21:05:45.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 7 2024-02-19T21:05:45.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf58011970 osd=3 3 2024-02-19T21:05:45.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 3 7 2024-02-19T21:05:45.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58011970 osd=3 4 2024-02-19T21:05:45.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf7168a700 10 client.22323.objecter _send_command 7 2024-02-19T21:05:45.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf58015b00 con 0x7fbf58011ff0 2024-02-19T21:05:45.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.475+0000 7fbf53fff700 10 client.22323.objecter ms_handle_connect 0x7fbf58011ff0 2024-02-19T21:05:45.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.476+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf58015b00 con 0x7fbf58011ff0 2024-02-19T21:05:45.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.476+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:05:45.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.476+0000 7fbf53fff700 10 client.22323.objecter _finish_command 7 = system:0 2024-02-19T21:05:45.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.476+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf58011970 osd=3 4 2024-02-19T21:05:45.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.476+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 3 7 2024-02-19T21:05:45.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.490+0000 7fbf7168a700 10 client.22323.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:45.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.490+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 8 2024-02-19T21:05:45.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.490+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58011970 osd=3 3 2024-02-19T21:05:45.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.490+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58011970 osd=3 3 2024-02-19T21:05:45.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.490+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58011970 osd=3 3 2024-02-19T21:05:45.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.490+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 8 2024-02-19T21:05:45.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.490+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf58011970 osd=3 3 2024-02-19T21:05:45.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.490+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 3 8 2024-02-19T21:05:45.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.490+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58011970 osd=3 4 2024-02-19T21:05:45.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.490+0000 7fbf7168a700 10 client.22323.objecter _send_command 8 2024-02-19T21:05:45.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.490+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf5800e240 con 0x7fbf58011ff0 2024-02-19T21:05:45.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf5800e240 con 0x7fbf58011ff0 2024-02-19T21:05:45.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:45.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf53fff700 10 client.22323.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf58011970 osd=3 4 2024-02-19T21:05:45.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 3 8 2024-02-19T21:05:45.487 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:05:45.488 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-19T21:05:45.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf7168a700 10 client.22323.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:45.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 9 2024-02-19T21:05:45.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf5800e000 osd=4 3 2024-02-19T21:05:45.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf5800e000 osd=4 3 2024-02-19T21:05:45.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf5800e000 osd=4 3 2024-02-19T21:05:45.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 9 2024-02-19T21:05:45.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf5800e000 osd=4 3 2024-02-19T21:05:45.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 4 9 2024-02-19T21:05:45.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf5800e000 osd=4 4 2024-02-19T21:05:45.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf7168a700 10 client.22323.objecter _send_command 9 2024-02-19T21:05:45.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf5800e1b0 con 0x7fbf5800ce50 2024-02-19T21:05:45.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.492+0000 7fbf53fff700 10 client.22323.objecter ms_handle_connect 0x7fbf5800ce50 2024-02-19T21:05:45.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.493+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf5800e1b0 con 0x7fbf5800ce50 2024-02-19T21:05:45.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.493+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:05:45.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.493+0000 7fbf53fff700 10 client.22323.objecter _finish_command 9 = system:0 2024-02-19T21:05:45.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.493+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf5800e000 osd=4 4 2024-02-19T21:05:45.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.493+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 4 9 2024-02-19T21:05:45.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.512+0000 7fbf7168a700 10 client.22323.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:45.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.512+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 10 2024-02-19T21:05:45.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.512+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf5800e000 osd=4 3 2024-02-19T21:05:45.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.512+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf5800e000 osd=4 3 2024-02-19T21:05:45.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.512+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf5800e000 osd=4 3 2024-02-19T21:05:45.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.512+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 10 2024-02-19T21:05:45.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.512+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf5800e000 osd=4 3 2024-02-19T21:05:45.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.512+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 4 10 2024-02-19T21:05:45.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.512+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf5800e000 osd=4 4 2024-02-19T21:05:45.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.512+0000 7fbf7168a700 10 client.22323.objecter _send_command 10 2024-02-19T21:05:45.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.512+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf5800e1b0 con 0x7fbf5800ce50 2024-02-19T21:05:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.513+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf5800e1b0 con 0x7fbf5800ce50 2024-02-19T21:05:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.513+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.513+0000 7fbf53fff700 10 client.22323.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.513+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf5800e000 osd=4 4 2024-02-19T21:05:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.513+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 4 10 2024-02-19T21:05:45.509 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:05:45.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.514+0000 7fbf7168a700 10 client.22323.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:45.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.514+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 11 2024-02-19T21:05:45.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.514+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf5800e1b0 osd=5 3 2024-02-19T21:05:45.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.514+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf5800e1b0 osd=5 3 2024-02-19T21:05:45.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.514+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf5800e1b0 osd=5 3 2024-02-19T21:05:45.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.514+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 11 2024-02-19T21:05:45.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.514+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf5800e1b0 osd=5 3 2024-02-19T21:05:45.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.514+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 5 11 2024-02-19T21:05:45.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.514+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf5800e1b0 osd=5 4 2024-02-19T21:05:45.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.514+0000 7fbf7168a700 10 client.22323.objecter _send_command 11 2024-02-19T21:05:45.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.514+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf58018070 con 0x7fbf58017a60 2024-02-19T21:05:45.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.514+0000 7fbf53fff700 10 client.22323.objecter ms_handle_connect 0x7fbf58017a60 2024-02-19T21:05:45.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.515+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf58018070 con 0x7fbf58017a60 2024-02-19T21:05:45.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.515+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:05:45.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.515+0000 7fbf53fff700 10 client.22323.objecter _finish_command 11 = system:0 2024-02-19T21:05:45.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.515+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf5800e1b0 osd=5 4 2024-02-19T21:05:45.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.515+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 5 11 2024-02-19T21:05:45.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.529+0000 7fbf7168a700 10 client.22323.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:45.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.529+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 12 2024-02-19T21:05:45.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.529+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf5800e1b0 osd=5 3 2024-02-19T21:05:45.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.529+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf5800e1b0 osd=5 3 2024-02-19T21:05:45.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.529+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf5800e1b0 osd=5 3 2024-02-19T21:05:45.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.529+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 12 2024-02-19T21:05:45.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.529+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf5800e1b0 osd=5 3 2024-02-19T21:05:45.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.529+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 5 12 2024-02-19T21:05:45.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.529+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf5800e1b0 osd=5 4 2024-02-19T21:05:45.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.529+0000 7fbf7168a700 10 client.22323.objecter _send_command 12 2024-02-19T21:05:45.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.529+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf58018070 con 0x7fbf58017a60 2024-02-19T21:05:45.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf58018070 con 0x7fbf58017a60 2024-02-19T21:05:45.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:45.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf53fff700 10 client.22323.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf5800e1b0 osd=5 4 2024-02-19T21:05:45.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 5 12 2024-02-19T21:05:45.526 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:05:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf7168a700 10 client.22323.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 13 2024-02-19T21:05:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58018070 osd=6 3 2024-02-19T21:05:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58018070 osd=6 3 2024-02-19T21:05:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58018070 osd=6 3 2024-02-19T21:05:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 13 2024-02-19T21:05:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf58018070 osd=6 3 2024-02-19T21:05:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 6 13 2024-02-19T21:05:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58018070 osd=6 4 2024-02-19T21:05:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf7168a700 10 client.22323.objecter _send_command 13 2024-02-19T21:05:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf58022c50 con 0x7fbf5801f080 2024-02-19T21:05:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.531+0000 7fbf53fff700 10 client.22323.objecter ms_handle_connect 0x7fbf5801f080 2024-02-19T21:05:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.532+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf58022c50 con 0x7fbf5801f080 2024-02-19T21:05:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.532+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:05:45.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.532+0000 7fbf53fff700 10 client.22323.objecter _finish_command 13 = system:0 2024-02-19T21:05:45.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.532+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf58018070 osd=6 4 2024-02-19T21:05:45.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.532+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 6 13 2024-02-19T21:05:45.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.546+0000 7fbf7168a700 10 client.22323.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:45.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.546+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 14 2024-02-19T21:05:45.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.546+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58018070 osd=6 3 2024-02-19T21:05:45.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.546+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58018070 osd=6 3 2024-02-19T21:05:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.546+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58018070 osd=6 3 2024-02-19T21:05:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.546+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 14 2024-02-19T21:05:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.546+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf58018070 osd=6 3 2024-02-19T21:05:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.546+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 6 14 2024-02-19T21:05:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.546+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58018070 osd=6 4 2024-02-19T21:05:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.546+0000 7fbf7168a700 10 client.22323.objecter _send_command 14 2024-02-19T21:05:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.546+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf58022c50 con 0x7fbf5801f080 2024-02-19T21:05:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf58022c50 con 0x7fbf5801f080 2024-02-19T21:05:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf53fff700 10 client.22323.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf58018070 osd=6 4 2024-02-19T21:05:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 6 14 2024-02-19T21:05:45.543 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:05:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf7168a700 10 client.22323.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 15 2024-02-19T21:05:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58022c50 osd=7 3 2024-02-19T21:05:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58022c50 osd=7 3 2024-02-19T21:05:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58022c50 osd=7 3 2024-02-19T21:05:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 15 2024-02-19T21:05:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf58022c50 osd=7 3 2024-02-19T21:05:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 7 15 2024-02-19T21:05:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58022c50 osd=7 4 2024-02-19T21:05:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf7168a700 10 client.22323.objecter _send_command 15 2024-02-19T21:05:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf58023e70 con 0x7fbf58024910 2024-02-19T21:05:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.548+0000 7fbf53fff700 10 client.22323.objecter ms_handle_connect 0x7fbf58024910 2024-02-19T21:05:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.549+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fbf58023e70 con 0x7fbf58024910 2024-02-19T21:05:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.549+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:05:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.549+0000 7fbf53fff700 10 client.22323.objecter _finish_command 15 = system:0 2024-02-19T21:05:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.549+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf58022c50 osd=7 4 2024-02-19T21:05:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.549+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 7 15 2024-02-19T21:05:45.553 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:45.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:45.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:45.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.563+0000 7fbf7168a700 10 client.22323.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:45.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.563+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign -1 16 2024-02-19T21:05:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.563+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58022c50 osd=7 3 2024-02-19T21:05:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.563+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58022c50 osd=7 3 2024-02-19T21:05:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.563+0000 7fbf7168a700 20 client.22323.objecter _get_session s=0x7fbf58022c50 osd=7 3 2024-02-19T21:05:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.563+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_remove -1 16 2024-02-19T21:05:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.563+0000 7fbf7168a700 20 client.22323.objecter get_session s=0x7fbf58022c50 osd=7 3 2024-02-19T21:05:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.563+0000 7fbf7168a700 15 client.22323.objecter _session_command_op_assign 7 16 2024-02-19T21:05:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.563+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58022c50 osd=7 4 2024-02-19T21:05:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.563+0000 7fbf7168a700 10 client.22323.objecter _send_command 16 2024-02-19T21:05:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.563+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fbf58023e70 con 0x7fbf58024910 2024-02-19T21:05:45.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf53fff700 1 -- 172.21.15.163:0/1240681493 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fbf58023e70 con 0x7fbf58024910 2024-02-19T21:05:45.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf53fff700 10 client.22323.objecter ms_dispatch 0x7fbf6c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:45.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf53fff700 10 client.22323.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf53fff700 20 client.22323.objecter put_session s=0x7fbf58022c50 osd=7 4 2024-02-19T21:05:45.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf53fff700 15 client.22323.objecter _session_command_op_remove 7 16 2024-02-19T21:05:45.560 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:05:45.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 10 librados: watch_flush enter 2024-02-19T21:05:45.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 10 librados: watch_flush exit 2024-02-19T21:05:45.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 10 client.22323.objecter close_session for osd.0 2024-02-19T21:05:45.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 >> v1:172.21.15.163:6801/1559418115 conn(0x7fbf58004530 legacy=0x7fbf58002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:45.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf580051a0 osd=0 1 2024-02-19T21:05:45.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 10 client.22323.objecter close_session for osd.1 2024-02-19T21:05:45.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 >> v1:172.21.15.163:6805/3125047432 conn(0x7fbf58004150 legacy=0x7fbf5800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:45.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58003fe0 osd=1 1 2024-02-19T21:05:45.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 10 client.22323.objecter close_session for osd.2 2024-02-19T21:05:45.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 >> v1:172.21.15.163:6809/1563158487 conn(0x7fbf5800d2f0 legacy=0x7fbf5800d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:45.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58003630 osd=2 1 2024-02-19T21:05:45.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 10 client.22323.objecter close_session for osd.3 2024-02-19T21:05:45.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 >> v1:172.21.15.163:6813/2112279769 conn(0x7fbf58011ff0 legacy=0x7fbf580143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:45.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58011970 osd=3 1 2024-02-19T21:05:45.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 10 client.22323.objecter close_session for osd.4 2024-02-19T21:05:45.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 >> v1:172.21.15.193:6800/3129716965 conn(0x7fbf5800ce50 legacy=0x7fbf58016df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:45.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf5800e000 osd=4 1 2024-02-19T21:05:45.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf6b7fe700 1 -- 172.21.15.163:0/1240681493 reap_dead start 2024-02-19T21:05:45.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 10 client.22323.objecter close_session for osd.5 2024-02-19T21:05:45.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 >> v1:172.21.15.193:6804/805161991 conn(0x7fbf58017a60 legacy=0x7fbf5801d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:45.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf6b7fe700 1 -- 172.21.15.163:0/1240681493 reap_dead start 2024-02-19T21:05:45.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf5800e1b0 osd=5 1 2024-02-19T21:05:45.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 10 client.22323.objecter close_session for osd.6 2024-02-19T21:05:45.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf6b7fe700 1 -- 172.21.15.163:0/1240681493 reap_dead start 2024-02-19T21:05:45.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 >> v1:172.21.15.193:6808/787604615 conn(0x7fbf5801f080 legacy=0x7fbf58021470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:45.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58018070 osd=6 1 2024-02-19T21:05:45.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 10 client.22323.objecter close_session for osd.7 2024-02-19T21:05:45.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 >> v1:172.21.15.193:6812/157927432 conn(0x7fbf58024910 legacy=0x7fbf58024d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:45.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 20 client.22323.objecter put_session s=0x7fbf58022c50 osd=7 1 2024-02-19T21:05:45.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 20 client.22323.objecter shutdown clearing up homeless session... 2024-02-19T21:05:45.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 10 client.22323.objecter successfully canceled tick 2024-02-19T21:05:45.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 >> v1:172.21.15.163:6800/1552514681 conn(0x7fbf54063ff0 legacy=0x7fbf540664b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:45.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 >> v1:172.21.15.163:6789/0 conn(0x7fbf6c0f7270 legacy=0x7fbf6c103320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:45.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 shutdown_connections 2024-02-19T21:05:45.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 >> 172.21.15.163:0/1240681493 conn(0x7fbf6c0f24a0 msgr2=0x7fbf6c109970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:45.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 shutdown_connections 2024-02-19T21:05:45.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 1 -- 172.21.15.163:0/1240681493 wait complete. 2024-02-19T21:05:45.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:45.565+0000 7fbf7168a700 1 librados: shutdown 2024-02-19T21:05:45.564 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.564 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.564 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.564 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.564 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.564 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.564 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.564 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:05:45.657 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:45.685 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:45.783 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:45.786 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:43.788687+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:05:45.786 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:43.790218+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:05:45.786 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:43.901963+0000 osd.1 (osd.1) 101 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:05:45.786 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:43.904866+0000 osd.1 (osd.1) 102 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:05:45.786 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:43.944905+0000 osd.5 (osd.5) 86 : cluster [DBG] 6.a scrub starts 2024-02-19T21:05:45.786 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:43.947821+0000 osd.5 (osd.5) 87 : cluster [DBG] 6.a scrub ok 2024-02-19T21:05:45.786 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:43.957437+0000 osd.6 (osd.6) 79 : cluster [DBG] 2.22 scrub starts 2024-02-19T21:05:45.786 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:43.958541+0000 osd.6 (osd.6) 80 : cluster [DBG] 2.22 scrub ok 2024-02-19T21:05:45.786 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:45.130571+0000 mon.b (mon.1) 262 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-19T21:05:45.786 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:45.131706+0000 mon.a (mon.0) 1233 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-19T21:05:45.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:45.170402+0000 mon.a (mon.0) 1234 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-19T21:05:45.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:45.170464+0000 mon.a (mon.0) 1235 : audit [INF] from='client.47740 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi163-83845-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:45.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:45.170540+0000 mon.a (mon.0) 1236 : audit [INF] from='client.48947 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:05:45.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:45.170590+0000 mon.a (mon.0) 1237 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-19T21:05:45.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:45.170630+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-02-19T21:05:45.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:45.179294+0000 mon.c (mon.2) 155 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi163-83952-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:45.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:45.179761+0000 mon.b (mon.1) 263 : audit [INF] from='client.? v1:172.21.15.163:0/4174416569' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi163-83854-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:45.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:45.179779+0000 mon.a (mon.0) 1239 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi163-83952-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:45.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:45.184750+0000 mon.a (mon.0) 1240 : audit [INF] from='client.47755 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi163-83854-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:45.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:45.185256+0000 mon.b (mon.1) 264 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-19T21:05:45.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:45.186819+0000 mon.a (mon.0) 1241 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-19T21:05:45.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:45.425647+0000 mon.a (mon.0) 1242 : audit [DBG] from='client.? v1:172.21.15.163:0/1240681493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:45.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:43.788687+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:05:45.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:43.790218+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:05:45.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:43.901963+0000 osd.1 (osd.1) 101 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:05:45.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:43.904866+0000 osd.1 (osd.1) 102 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:05:45.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:43.944905+0000 osd.5 (osd.5) 86 : cluster [DBG] 6.a scrub starts 2024-02-19T21:05:45.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:43.947821+0000 osd.5 (osd.5) 87 : cluster [DBG] 6.a scrub ok 2024-02-19T21:05:45.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:43.957437+0000 osd.6 (osd.6) 79 : cluster [DBG] 2.22 scrub starts 2024-02-19T21:05:45.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:43.958541+0000 osd.6 (osd.6) 80 : cluster [DBG] 2.22 scrub ok 2024-02-19T21:05:45.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:45.130571+0000 mon.b (mon.1) 262 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-19T21:05:45.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:45.131706+0000 mon.a (mon.0) 1233 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-19T21:05:45.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:45.170402+0000 mon.a (mon.0) 1234 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-19T21:05:45.789 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:45.170464+0000 mon.a (mon.0) 1235 : audit [INF] from='client.47740 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi163-83845-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:45.789 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:45.170540+0000 mon.a (mon.0) 1236 : audit [INF] from='client.48947 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:05:45.789 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:45.170590+0000 mon.a (mon.0) 1237 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-19T21:05:45.789 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:45.170630+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-02-19T21:05:45.789 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:45.179294+0000 mon.c (mon.2) 155 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi163-83952-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:45.789 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:45.179761+0000 mon.b (mon.1) 263 : audit [INF] from='client.? v1:172.21.15.163:0/4174416569' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi163-83854-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:45.789 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:45.179779+0000 mon.a (mon.0) 1239 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi163-83952-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:45.789 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:45.184750+0000 mon.a (mon.0) 1240 : audit [INF] from='client.47755 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi163-83854-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:45.789 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:45.185256+0000 mon.b (mon.1) 264 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-19T21:05:45.789 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:45.186819+0000 mon.a (mon.0) 1241 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-19T21:05:45.789 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:45.425647+0000 mon.a (mon.0) 1242 : audit [DBG] from='client.? v1:172.21.15.163:0/1240681493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:45.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:45.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:45.792 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:05:45.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:43.788687+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:05:45.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:43.790218+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:05:45.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:43.901963+0000 osd.1 (osd.1) 101 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:05:45.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:43.904866+0000 osd.1 (osd.1) 102 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:05:45.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:43.944905+0000 osd.5 (osd.5) 86 : cluster [DBG] 6.a scrub starts 2024-02-19T21:05:45.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:43.947821+0000 osd.5 (osd.5) 87 : cluster [DBG] 6.a scrub ok 2024-02-19T21:05:45.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:43.957437+0000 osd.6 (osd.6) 79 : cluster [DBG] 2.22 scrub starts 2024-02-19T21:05:45.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:43.958541+0000 osd.6 (osd.6) 80 : cluster [DBG] 2.22 scrub ok 2024-02-19T21:05:45.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:45.130571+0000 mon.b (mon.1) 262 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-19T21:05:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:45.131706+0000 mon.a (mon.0) 1233 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-02-19T21:05:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:45.170402+0000 mon.a (mon.0) 1234 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-02-19T21:05:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:45.170464+0000 mon.a (mon.0) 1235 : audit [INF] from='client.47740 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi163-83845-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:45.170540+0000 mon.a (mon.0) 1236 : audit [INF] from='client.48947 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:05:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:45.170590+0000 mon.a (mon.0) 1237 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-02-19T21:05:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:45.170630+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-02-19T21:05:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:45.179294+0000 mon.c (mon.2) 155 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi163-83952-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:45.179761+0000 mon.b (mon.1) 263 : audit [INF] from='client.? v1:172.21.15.163:0/4174416569' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi163-83854-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:45.179779+0000 mon.a (mon.0) 1239 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi163-83952-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:45.184750+0000 mon.a (mon.0) 1240 : audit [INF] from='client.47755 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi163-83854-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:45.185256+0000 mon.b (mon.1) 264 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-19T21:05:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:45.186819+0000 mon.a (mon.0) 1241 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-02-19T21:05:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:45.425647+0000 mon.a (mon.0) 1242 : audit [DBG] from='client.? v1:172.21.15.163:0/1240681493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:45.891 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:45.919 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:46.004 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:05:46.267 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:46.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:46.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:46.295 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:46.371 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:05:46.397 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:46.546 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:05:46.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:46.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:05:46.650 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:46.678 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:46.809 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:44.875436+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:05:46.809 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:44.876421+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:05:46.809 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:44.983920+0000 osd.6 (osd.6) 81 : cluster [DBG] 2.29 scrub starts 2024-02-19T21:05:46.809 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:44.986867+0000 osd.6 (osd.6) 82 : cluster [DBG] 2.29 scrub ok 2024-02-19T21:05:46.809 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:45.010976+0000 osd.2 (osd.2) 77 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:05:46.809 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:45.013939+0000 osd.2 (osd.2) 78 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:05:46.809 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:45.130134+0000 mgr.y (mgr.14550) 212 : cluster [DBG] pgmap v218: 333 pgs: 32 creating+peering, 301 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 51 B/s, 4 objects/s recovering 2024-02-19T21:05:46.809 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:46.425680+0000 mon.a (mon.0) 1243 : audit [INF] from='client.47755 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi163-83854-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:46.809 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:46.425794+0000 mon.a (mon.0) 1244 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-19T21:05:46.809 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:46.425929+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-02-19T21:05:46.809 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:46.427236+0000 mon.b (mon.1) 265 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1"}]: dispatch 2024-02-19T21:05:46.810 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:46.427692+0000 mon.a (mon.0) 1246 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1"}]: dispatch 2024-02-19T21:05:46.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:44.875436+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:05:46.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:44.876421+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:05:46.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:44.983920+0000 osd.6 (osd.6) 81 : cluster [DBG] 2.29 scrub starts 2024-02-19T21:05:46.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:44.986867+0000 osd.6 (osd.6) 82 : cluster [DBG] 2.29 scrub ok 2024-02-19T21:05:46.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:45.010976+0000 osd.2 (osd.2) 77 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:05:46.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:45.013939+0000 osd.2 (osd.2) 78 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:05:46.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:45.130134+0000 mgr.y (mgr.14550) 212 : cluster [DBG] pgmap v218: 333 pgs: 32 creating+peering, 301 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 51 B/s, 4 objects/s recovering 2024-02-19T21:05:46.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:46.425680+0000 mon.a (mon.0) 1243 : audit [INF] from='client.47755 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi163-83854-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:46.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:46.425794+0000 mon.a (mon.0) 1244 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-19T21:05:46.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:46.425929+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-02-19T21:05:46.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:46.427236+0000 mon.b (mon.1) 265 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1"}]: dispatch 2024-02-19T21:05:46.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:46.427692+0000 mon.a (mon.0) 1246 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1"}]: dispatch 2024-02-19T21:05:47.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:44.875436+0000 osd.1 (osd.1) 103 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:05:47.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:44.876421+0000 osd.1 (osd.1) 104 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:05:47.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:44.983920+0000 osd.6 (osd.6) 81 : cluster [DBG] 2.29 scrub starts 2024-02-19T21:05:47.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:44.986867+0000 osd.6 (osd.6) 82 : cluster [DBG] 2.29 scrub ok 2024-02-19T21:05:47.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:45.010976+0000 osd.2 (osd.2) 77 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:05:47.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:45.013939+0000 osd.2 (osd.2) 78 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:05:47.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:45.130134+0000 mgr.y (mgr.14550) 212 : cluster [DBG] pgmap v218: 333 pgs: 32 creating+peering, 301 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 51 B/s, 4 objects/s recovering 2024-02-19T21:05:47.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:46.425680+0000 mon.a (mon.0) 1243 : audit [INF] from='client.47755 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi163-83854-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:47.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:46.425794+0000 mon.a (mon.0) 1244 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-02-19T21:05:47.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:46.425929+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-02-19T21:05:47.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:46.427236+0000 mon.b (mon.1) 265 : audit [INF] from='client.? v1:172.21.15.163:0/4089460683' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1"}]: dispatch 2024-02-19T21:05:47.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:46.427692+0000 mon.a (mon.0) 1246 : audit [INF] from='client.24919 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1"}]: dispatch 2024-02-19T21:05:47.227 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:47.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:47.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:47.332 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:47.360 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:47.458 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:47.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:47.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:47.564 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:47.594 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:47.717 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:05:47.717 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-02-19T21:05:47.717 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-02-19T21:05:47.717 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-02-19T21:05:47.718 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.718 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34382.0:53 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-19T21:05:47.718 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:01:20.279139+0000", 2024-02-19T21:05:47.718 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 267.42693916899998, 2024-02-19T21:05:47.718 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.016593475, 2024-02-19T21:05:47.718 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.718 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.718 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:47.718 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34382", 2024-02-19T21:05:47.718 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2202902387", 2024-02-19T21:05:47.718 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 53 2024-02-19T21:05:47.718 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.718 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.718 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.718 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.719 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279139+0000", 2024-02-19T21:05:47.719 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.719 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.719 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.719 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.719 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279139+0000", 2024-02-19T21:05:47.719 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:05:47.719 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.719 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.719 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.719 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279136+0000", 2024-02-19T21:05:47.719 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4159999999999993e-06 2024-02-19T21:05:47.719 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.719 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.720 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.720 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279145+0000", 2024-02-19T21:05:47.720 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5599000000000001e-05 2024-02-19T21:05:47.720 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.720 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.720 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.720 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279161+0000", 2024-02-19T21:05:47.720 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3079999999999997e-06 2024-02-19T21:05:47.720 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.720 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.720 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.720 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279170+0000", 2024-02-19T21:05:47.720 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9360000000000001e-05 2024-02-19T21:05:47.720 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.721 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.721 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.721 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279230+0000", 2024-02-19T21:05:47.721 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4889999999999998e-06 2024-02-19T21:05:47.721 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.721 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.721 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:05:47.721 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279237+0000", 2024-02-19T21:05:47.721 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003363923 2024-02-19T21:05:47.721 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.721 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.721 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.721 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.282601+0000", 2024-02-19T21:05:47.721 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2699999999999993e-06 2024-02-19T21:05:47.721 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.722 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.722 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:05:47.722 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.282610+0000", 2024-02-19T21:05:47.722 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0088474929999999997 2024-02-19T21:05:47.722 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.722 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.722 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.722 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.291458+0000", 2024-02-19T21:05:47.722 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9685999999999997e-05 2024-02-19T21:05:47.722 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.722 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.722 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.722 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.291528+0000", 2024-02-19T21:05:47.722 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015815 2024-02-19T21:05:47.722 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.723 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.723 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 4,7", 2024-02-19T21:05:47.723 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.291686+0000", 2024-02-19T21:05:47.723 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0032840980000000001 2024-02-19T21:05:47.723 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.723 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.723 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:47.723 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.294970+0000", 2024-02-19T21:05:47.723 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0307000000000001e-05 2024-02-19T21:05:47.723 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.723 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.723 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:05:47.723 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.294990+0000", 2024-02-19T21:05:47.723 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00066387399999999995 2024-02-19T21:05:47.723 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.724 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.724 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:47.724 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.295654+0000", 2024-02-19T21:05:47.724 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1844000000000001e-05 2024-02-19T21:05:47.724 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.724 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.724 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.724 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.295696+0000", 2024-02-19T21:05:47.724 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6424000000000002e-05 2024-02-19T21:05:47.724 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.724 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.724 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.725 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.295732+0000", 2024-02-19T21:05:47.725 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.016593475 2024-02-19T21:05:47.725 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.725 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.725 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.725 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.725 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.725 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:91 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-19T21:05:47.725 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:01:20.281408+0000", 2024-02-19T21:05:47.725 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 267.42467007200003, 2024-02-19T21:05:47.725 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014198916000000001, 2024-02-19T21:05:47.725 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.725 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.725 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:47.725 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:05:47.726 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:05:47.726 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 91 2024-02-19T21:05:47.726 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.726 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.726 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.726 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.726 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.281408+0000", 2024-02-19T21:05:47.726 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.726 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.726 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.726 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.726 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.281408+0000", 2024-02-19T21:05:47.726 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:05:47.726 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.726 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.727 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.727 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.281405+0000", 2024-02-19T21:05:47.727 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7749999999999993e-06 2024-02-19T21:05:47.727 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.727 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.727 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.727 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.281413+0000", 2024-02-19T21:05:47.727 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1241e-05 2024-02-19T21:05:47.727 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.727 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.727 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.727 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.281424+0000", 2024-02-19T21:05:47.727 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5439999999999999e-06 2024-02-19T21:05:47.727 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.728 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.728 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.728 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.281433+0000", 2024-02-19T21:05:47.728 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001918274 2024-02-19T21:05:47.728 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.728 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.728 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.728 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.283351+0000", 2024-02-19T21:05:47.728 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5009999999999998e-06 2024-02-19T21:05:47.728 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.728 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.728 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:05:47.728 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.283359+0000", 2024-02-19T21:05:47.729 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0075975360000000002 2024-02-19T21:05:47.729 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.729 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.729 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.729 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.290956+0000", 2024-02-19T21:05:47.729 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000121327 2024-02-19T21:05:47.729 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.729 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.729 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.729 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.291078+0000", 2024-02-19T21:05:47.729 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000152733 2024-02-19T21:05:47.729 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.729 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.729 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 4,7", 2024-02-19T21:05:47.729 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.291230+0000", 2024-02-19T21:05:47.730 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0036645950000000001 2024-02-19T21:05:47.730 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.730 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.730 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:47.730 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.294895+0000", 2024-02-19T21:05:47.730 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0357000000000002e-05 2024-02-19T21:05:47.730 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.730 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.730 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:05:47.730 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.294985+0000", 2024-02-19T21:05:47.730 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00054387100000000002 2024-02-19T21:05:47.730 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.730 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.730 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:47.730 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.295529+0000", 2024-02-19T21:05:47.731 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5608000000000001e-05 2024-02-19T21:05:47.731 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.731 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.731 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.731 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.295575+0000", 2024-02-19T21:05:47.731 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1906000000000003e-05 2024-02-19T21:05:47.731 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.731 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.731 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.731 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.295607+0000", 2024-02-19T21:05:47.731 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014198916000000001 2024-02-19T21:05:47.731 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.731 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.731 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.731 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.732 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.732 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.3.0:299 11.8 e113/99 11:1381d4a2:test-rados-api-smithi163-83952-6::foo:9 v 113'6, mlcod=111'3)", 2024-02-19T21:05:47.732 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:52.583933+0000", 2024-02-19T21:05:47.732 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 55.122144695000003, 2024-02-19T21:05:47.732 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0114605, 2024-02-19T21:05:47.732 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.732 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.732 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.732 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.732 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.732 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583933+0000", 2024-02-19T21:05:47.732 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.732 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.732 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.732 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.733 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583933+0000", 2024-02-19T21:05:47.733 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:05:47.733 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.733 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.733 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.733 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583933+0000", 2024-02-19T21:05:47.733 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9749999999999998e-06 2024-02-19T21:05:47.733 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.733 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.733 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.733 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583938+0000", 2024-02-19T21:05:47.733 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2570000000000004e-06 2024-02-19T21:05:47.733 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.733 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.734 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.734 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583944+0000", 2024-02-19T21:05:47.734 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.657e-06 2024-02-19T21:05:47.734 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.734 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.734 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.734 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583951+0000", 2024-02-19T21:05:47.734 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00042205900000000002 2024-02-19T21:05:47.734 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.734 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.734 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.734 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.584373+0000", 2024-02-19T21:05:47.734 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3680999999999999e-05 2024-02-19T21:05:47.734 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.735 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.735 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.735 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.584396+0000", 2024-02-19T21:05:47.735 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010962929 2024-02-19T21:05:47.735 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.735 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.735 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.735 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.595359+0000", 2024-02-19T21:05:47.735 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4283999999999999e-05 2024-02-19T21:05:47.735 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.735 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.735 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.735 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.595394+0000", 2024-02-19T21:05:47.735 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0114605 2024-02-19T21:05:47.735 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.736 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.736 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.736 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.736 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.736 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35237.0:8 54.0 54:2698d9fa:test-rados-api-smithi163-83984-7::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-19T21:05:47.736 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:52.638465+0000", 2024-02-19T21:05:47.736 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 55.067613119999997, 2024-02-19T21:05:47.736 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0094088230000000002, 2024-02-19T21:05:47.736 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.736 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.736 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:47.736 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35237", 2024-02-19T21:05:47.736 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1081998331", 2024-02-19T21:05:47.736 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 8 2024-02-19T21:05:47.736 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.737 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.737 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.737 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.737 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638465+0000", 2024-02-19T21:05:47.737 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.737 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.737 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.737 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.737 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638465+0000", 2024-02-19T21:05:47.737 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:05:47.737 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.737 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.737 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.737 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638463+0000", 2024-02-19T21:05:47.738 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9859999999999994e-06 2024-02-19T21:05:47.738 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.738 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.738 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.738 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638471+0000", 2024-02-19T21:05:47.738 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8330000000000006e-06 2024-02-19T21:05:47.738 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.738 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.738 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.738 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638481+0000", 2024-02-19T21:05:47.738 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2889999999999997e-06 2024-02-19T21:05:47.738 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.738 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.738 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.738 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638486+0000", 2024-02-19T21:05:47.739 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7515999999999998e-05 2024-02-19T21:05:47.739 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.739 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.739 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.739 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638523+0000", 2024-02-19T21:05:47.739 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000171239 2024-02-19T21:05:47.739 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.739 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.739 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:05:47.739 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638695+0000", 2024-02-19T21:05:47.739 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00090101799999999996 2024-02-19T21:05:47.739 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.739 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.739 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.740 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.639596+0000", 2024-02-19T21:05:47.740 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3301999999999996e-05 2024-02-19T21:05:47.740 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.740 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.740 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.740 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.639679+0000", 2024-02-19T21:05:47.740 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013513300000000001 2024-02-19T21:05:47.740 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.740 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.740 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,4", 2024-02-19T21:05:47.740 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.639814+0000", 2024-02-19T21:05:47.740 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00034915400000000001 2024-02-19T21:05:47.740 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.740 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.740 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:05:47.741 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.640163+0000", 2024-02-19T21:05:47.741 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014084300000000001 2024-02-19T21:05:47.741 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.741 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.741 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:47.741 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.640304+0000", 2024-02-19T21:05:47.741 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0075044070000000003 2024-02-19T21:05:47.741 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.741 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.741 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:47.741 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.647809+0000", 2024-02-19T21:05:47.741 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0991000000000002e-05 2024-02-19T21:05:47.741 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.741 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.742 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.742 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.647840+0000", 2024-02-19T21:05:47.742 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4032000000000002e-05 2024-02-19T21:05:47.742 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.742 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.742 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.742 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.647874+0000", 2024-02-19T21:05:47.742 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0094088230000000002 2024-02-19T21:05:47.742 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.742 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.742 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.742 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.742 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.742 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35237.0:9 54.0 54:2698d9fa:test-rados-api-smithi163-83984-7::foo:head [watch watch cookie 94487086298256] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-19T21:05:47.743 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:52.648180+0000", 2024-02-19T21:05:47.743 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 55.057897834000002, 2024-02-19T21:05:47.743 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0066472850000000002, 2024-02-19T21:05:47.743 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.743 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.743 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:47.743 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35237", 2024-02-19T21:05:47.743 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1081998331", 2024-02-19T21:05:47.743 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 9 2024-02-19T21:05:47.743 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.743 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.743 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.743 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.743 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.648180+0000", 2024-02-19T21:05:47.743 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.744 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.744 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.744 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.744 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.648180+0000", 2024-02-19T21:05:47.744 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:05:47.744 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.744 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.744 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.744 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.648179+0000", 2024-02-19T21:05:47.744 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3200000000000001e-06 2024-02-19T21:05:47.744 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.744 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.744 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.744 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.648183+0000", 2024-02-19T21:05:47.745 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2960000000000001e-06 2024-02-19T21:05:47.745 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.745 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.745 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.745 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.648190+0000", 2024-02-19T21:05:47.745 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7469999999999999e-06 2024-02-19T21:05:47.745 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.745 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.745 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.745 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.648194+0000", 2024-02-19T21:05:47.745 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5545999999999999e-05 2024-02-19T21:05:47.745 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.745 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.745 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.745 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.648220+0000", 2024-02-19T21:05:47.746 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9559999999999999e-05 2024-02-19T21:05:47.746 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.746 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.746 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.746 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.648279+0000", 2024-02-19T21:05:47.746 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2807999999999996e-05 2024-02-19T21:05:47.746 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.746 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.746 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,4", 2024-02-19T21:05:47.746 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.648372+0000", 2024-02-19T21:05:47.746 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000241843 2024-02-19T21:05:47.746 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.746 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.746 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:05:47.747 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.648614+0000", 2024-02-19T21:05:47.747 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00019323699999999999 2024-02-19T21:05:47.747 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.747 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.747 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:47.747 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.648807+0000", 2024-02-19T21:05:47.747 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0059061249999999999 2024-02-19T21:05:47.747 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.747 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.747 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:47.747 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.654713+0000", 2024-02-19T21:05:47.747 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6647999999999998e-05 2024-02-19T21:05:47.747 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.747 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.747 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.748 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.654760+0000", 2024-02-19T21:05:47.748 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7366000000000005e-05 2024-02-19T21:05:47.748 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.748 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.748 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.748 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.654827+0000", 2024-02-19T21:05:47.748 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0066472850000000002 2024-02-19T21:05:47.748 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.748 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.748 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.748 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.748 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.748 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35237.0:17 54.0 54:2016bb24:test-rados-api-smithi163-83984-9::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-19T21:05:47.748 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:52.675344+0000", 2024-02-19T21:05:47.749 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 55.030734275999997, 2024-02-19T21:05:47.749 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0075634550000000002, 2024-02-19T21:05:47.749 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.749 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.749 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:05:47.749 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35237", 2024-02-19T21:05:47.749 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1081998331", 2024-02-19T21:05:47.749 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 17 2024-02-19T21:05:47.749 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.749 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.749 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.749 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.749 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.675344+0000", 2024-02-19T21:05:47.749 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.675344+0000", 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.675340+0000", 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4884e-05 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.675355+0000", 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5582999999999999e-05 2024-02-19T21:05:47.750 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.675381+0000", 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2228999999999999e-05 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.675393+0000", 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5155999999999996e-05 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.675478+0000", 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00028893099999999998 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.751 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.752 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:05:47.752 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.675767+0000", 2024-02-19T21:05:47.752 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00072265899999999995 2024-02-19T21:05:47.752 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.752 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.752 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.752 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.676490+0000", 2024-02-19T21:05:47.752 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1937e-05 2024-02-19T21:05:47.752 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.752 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.752 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.752 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.676552+0000", 2024-02-19T21:05:47.752 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4288000000000003e-05 2024-02-19T21:05:47.752 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.753 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.753 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,4", 2024-02-19T21:05:47.753 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.676646+0000", 2024-02-19T21:05:47.753 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00028980299999999999 2024-02-19T21:05:47.753 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.753 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.753 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:05:47.753 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.676936+0000", 2024-02-19T21:05:47.753 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00028186299999999998 2024-02-19T21:05:47.753 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.753 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.753 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:47.753 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.677218+0000", 2024-02-19T21:05:47.753 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0056381410000000002 2024-02-19T21:05:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:05:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.682856+0000", 2024-02-19T21:05:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.074e-05 2024-02-19T21:05:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.682886+0000", 2024-02-19T21:05:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0665e-05 2024-02-19T21:05:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.682907+0000", 2024-02-19T21:05:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0075634550000000002 2024-02-19T21:05:47.755 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.755 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.755 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.755 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.755 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.755 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "replica_scrub(pg: 4.10,from:0'0,to:65'4,epoch:127/59,start:4:08000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-02-19T21:05:47.755 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:08.142968+0000", 2024-02-19T21:05:47.755 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 39.563110385000002, 2024-02-19T21:05:47.755 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010775658, 2024-02-19T21:05:47.755 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.755 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:05:47.755 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.755 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.755 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.755 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.142968+0000", 2024-02-19T21:05:47.756 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.756 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.756 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.756 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.756 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.142968+0000", 2024-02-19T21:05:47.756 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:05:47.759 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.759 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.759 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.759 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.142966+0000", 2024-02-19T21:05:47.759 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2630000000000003e-06 2024-02-19T21:05:47.759 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.759 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.759 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.760 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.142974+0000", 2024-02-19T21:05:47.760 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0028049339999999998 2024-02-19T21:05:47.760 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.760 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.760 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.760 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.145779+0000", 2024-02-19T21:05:47.760 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3135999999999999e-05 2024-02-19T21:05:47.760 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.760 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.760 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.760 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.145792+0000", 2024-02-19T21:05:47.760 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0078870490000000001 2024-02-19T21:05:47.760 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.153679+0000", 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5184000000000001e-05 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.153695+0000", 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8640999999999997e-05 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.153743+0000", 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010775658 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.761 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.762 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.762 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.762 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.762 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.1.0:241 28.15 e132/99 -30:ad087405:::temp_28.15_0_14259_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:05:47.762 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.401838+0000", 2024-02-19T21:05:47.762 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 35.304239514999999, 2024-02-19T21:05:47.762 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025390421, 2024-02-19T21:05:47.762 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.762 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.762 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.762 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.762 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.762 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.401838+0000", 2024-02-19T21:05:47.762 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.762 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.763 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.763 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.763 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.401838+0000", 2024-02-19T21:05:47.763 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:05:47.763 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.763 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.763 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.763 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.401837+0000", 2024-02-19T21:05:47.763 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.016854948000000002 2024-02-19T21:05:47.763 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.763 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.763 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.763 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.418692+0000", 2024-02-19T21:05:47.763 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8876e-05 2024-02-19T21:05:47.764 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.764 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.764 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.764 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.418710+0000", 2024-02-19T21:05:47.764 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4498000000000002e-05 2024-02-19T21:05:47.764 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.764 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.764 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.764 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.418735+0000", 2024-02-19T21:05:47.764 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7998000000000003e-05 2024-02-19T21:05:47.764 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.764 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.764 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.764 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.418793+0000", 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6917e-05 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.418820+0000", 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0083708069999999992 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.427191+0000", 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8194000000000001e-05 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.427229+0000", 2024-02-19T21:05:47.765 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025390421 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.1.0:242 28.15 e132/99 -30:ad087405:::temp_28.15_0_14259_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.454353+0000", 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 35.251725362999998, 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.029471536999999999, 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.454353+0000", 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.766 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.767 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.767 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.767 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.454353+0000", 2024-02-19T21:05:47.767 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:05:47.767 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.767 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.767 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.767 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.454350+0000", 2024-02-19T21:05:47.767 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.021044232999999999 2024-02-19T21:05:47.767 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.767 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.767 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.767 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.475394+0000", 2024-02-19T21:05:47.767 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7527999999999999e-05 2024-02-19T21:05:47.767 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.768 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.768 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.768 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.475412+0000", 2024-02-19T21:05:47.768 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1040000000000007e-06 2024-02-19T21:05:47.768 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.768 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.768 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.768 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.475420+0000", 2024-02-19T21:05:47.768 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3225000000000005e-05 2024-02-19T21:05:47.768 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.768 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.768 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.768 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.475483+0000", 2024-02-19T21:05:47.768 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4318e-05 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.475508+0000", 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0082776299999999994 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.483785+0000", 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8816999999999998e-05 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.483824+0000", 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.029471536999999999 2024-02-19T21:05:47.769 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.770 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.770 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.770 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.770 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.770 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.1.0:243 28.15 e132/99 -30:ad087405:::temp_28.15_0_14259_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:05:47.770 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.477725+0000", 2024-02-19T21:05:47.770 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 35.228352958999999, 2024-02-19T21:05:47.770 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.020820749, 2024-02-19T21:05:47.770 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.770 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.770 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.770 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.770 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.770 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.477725+0000", 2024-02-19T21:05:47.770 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.477725+0000", 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.477723+0000", 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.012259177 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.489982+0000", 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6534000000000001e-05 2024-02-19T21:05:47.771 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.772 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.772 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.772 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.489999+0000", 2024-02-19T21:05:47.772 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8809999999999996e-06 2024-02-19T21:05:47.772 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.772 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.772 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.772 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.490006+0000", 2024-02-19T21:05:47.772 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2170999999999997e-05 2024-02-19T21:05:47.772 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.772 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.772 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.772 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.490079+0000", 2024-02-19T21:05:47.772 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3287e-05 2024-02-19T21:05:47.772 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.773 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.773 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.773 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.490132+0000", 2024-02-19T21:05:47.773 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0083842740000000006 2024-02-19T21:05:47.773 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.773 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.773 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.773 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.498516+0000", 2024-02-19T21:05:47.773 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9481000000000001e-05 2024-02-19T21:05:47.773 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.773 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.773 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.773 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.498546+0000", 2024-02-19T21:05:47.773 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.020820749 2024-02-19T21:05:47.773 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.774 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.774 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.774 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.774 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.774 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.24919.0:2078 28.15 e132/99 28:ad087405:test-rados-api-smithi163-83920-17::big.copy:head v 132'1, mlcod=0'0)", 2024-02-19T21:05:47.774 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.490024+0000", 2024-02-19T21:05:47.774 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 35.216054243999999, 2024-02-19T21:05:47.774 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0085769699999999997, 2024-02-19T21:05:47.774 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.774 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.774 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.774 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.774 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.774 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.490024+0000", 2024-02-19T21:05:47.774 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.775 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.775 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.775 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.775 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.490024+0000", 2024-02-19T21:05:47.775 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:05:47.775 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.775 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.775 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.775 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.490022+0000", 2024-02-19T21:05:47.775 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1821000000000001e-05 2024-02-19T21:05:47.775 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.775 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.775 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.775 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.490034+0000", 2024-02-19T21:05:47.775 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8679999999999998e-06 2024-02-19T21:05:47.776 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.776 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.776 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.776 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.490038+0000", 2024-02-19T21:05:47.776 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0890000000000002e-06 2024-02-19T21:05:47.776 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.776 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.776 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.776 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.490042+0000", 2024-02-19T21:05:47.776 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0023538069999999999 2024-02-19T21:05:47.776 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.776 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.776 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.776 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.492396+0000", 2024-02-19T21:05:47.777 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8417999999999999e-05 2024-02-19T21:05:47.777 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.777 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.777 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.777 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.492415+0000", 2024-02-19T21:05:47.777 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0061699859999999997 2024-02-19T21:05:47.777 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.777 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.777 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.777 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.498585+0000", 2024-02-19T21:05:47.777 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5940999999999999e-05 2024-02-19T21:05:47.777 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.777 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.777 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.778 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.498601+0000", 2024-02-19T21:05:47.778 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0085769699999999997 2024-02-19T21:05:47.778 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.778 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.778 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.778 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.778 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.778 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.5.0:1270 28.9 e132/99 -30:90852524:::temp_28.9_0_24280_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:05:47.778 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.578695+0000", 2024-02-19T21:05:47.778 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 35.127383364000003, 2024-02-19T21:05:47.778 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017938424000000001, 2024-02-19T21:05:47.778 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.778 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.778 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.779 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.779 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.779 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.578695+0000", 2024-02-19T21:05:47.779 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.779 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.779 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.779 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.779 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.578695+0000", 2024-02-19T21:05:47.779 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:05:47.779 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.779 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.779 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.779 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.578691+0000", 2024-02-19T21:05:47.779 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0030402350000000001 2024-02-19T21:05:47.779 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.780 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.780 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.780 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.581731+0000", 2024-02-19T21:05:47.780 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2955e-05 2024-02-19T21:05:47.780 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.780 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.780 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.780 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.581754+0000", 2024-02-19T21:05:47.780 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0519999999999999e-05 2024-02-19T21:05:47.780 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.780 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.780 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.780 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.581765+0000", 2024-02-19T21:05:47.780 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7444e-05 2024-02-19T21:05:47.780 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.781 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.781 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.781 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.581832+0000", 2024-02-19T21:05:47.781 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9349000000000001e-05 2024-02-19T21:05:47.781 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.781 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.781 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.781 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.581851+0000", 2024-02-19T21:05:47.781 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014735747 2024-02-19T21:05:47.781 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.781 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.781 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.781 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.596587+0000", 2024-02-19T21:05:47.781 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5918000000000003e-05 2024-02-19T21:05:47.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.782 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.782 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.782 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.596633+0000", 2024-02-19T21:05:47.782 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017938424000000001 2024-02-19T21:05:47.782 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.782 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.782 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.782 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.782 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.782 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.5.0:1271 28.9 e132/99 -30:90852524:::temp_28.9_0_24280_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:05:47.782 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.608168+0000", 2024-02-19T21:05:47.782 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 35.097910128000002, 2024-02-19T21:05:47.782 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017383068000000002, 2024-02-19T21:05:47.783 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.783 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.783 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.783 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.783 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.783 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.608168+0000", 2024-02-19T21:05:47.783 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.783 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.783 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.783 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.783 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.608168+0000", 2024-02-19T21:05:47.783 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:05:47.783 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.783 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.783 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.784 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.608164+0000", 2024-02-19T21:05:47.784 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003639705 2024-02-19T21:05:47.784 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.784 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.784 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.784 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.611804+0000", 2024-02-19T21:05:47.784 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6741e-05 2024-02-19T21:05:47.784 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.784 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.784 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.784 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.611830+0000", 2024-02-19T21:05:47.784 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0243e-05 2024-02-19T21:05:47.784 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.784 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.785 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.785 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.611841+0000", 2024-02-19T21:05:47.785 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8799999999999999e-05 2024-02-19T21:05:47.785 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.785 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.785 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.785 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.611899+0000", 2024-02-19T21:05:47.785 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7115e-05 2024-02-19T21:05:47.785 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.785 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.785 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.785 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.611927+0000", 2024-02-19T21:05:47.785 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013572864 2024-02-19T21:05:47.785 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.785 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.786 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.786 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.625499+0000", 2024-02-19T21:05:47.786 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1449999999999997e-05 2024-02-19T21:05:47.786 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.786 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.786 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.786 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.625551+0000", 2024-02-19T21:05:47.786 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017383068000000002 2024-02-19T21:05:47.786 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.786 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.786 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.786 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.786 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.786 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.5.0:1272 28.9 e132/99 -30:90852524:::temp_28.9_0_24280_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:05:47.786 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.668531+0000", 2024-02-19T21:05:47.787 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 35.037546532999997, 2024-02-19T21:05:47.787 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017253208999999999, 2024-02-19T21:05:47.787 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.787 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.787 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.787 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.787 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.787 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.668531+0000", 2024-02-19T21:05:47.787 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.787 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.787 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.787 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.787 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.668531+0000", 2024-02-19T21:05:47.787 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999957 2024-02-19T21:05:47.788 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.788 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.788 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.788 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.668527+0000", 2024-02-19T21:05:47.788 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0031829409999999999 2024-02-19T21:05:47.788 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.788 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.788 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.788 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.671710+0000", 2024-02-19T21:05:47.788 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5936999999999999e-05 2024-02-19T21:05:47.788 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.788 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.788 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.789 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.671736+0000", 2024-02-19T21:05:47.789 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1466e-05 2024-02-19T21:05:47.789 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.789 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.789 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.789 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.671747+0000", 2024-02-19T21:05:47.789 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2371e-05 2024-02-19T21:05:47.789 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.789 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.789 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.789 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.671840+0000", 2024-02-19T21:05:47.789 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0371000000000001e-05 2024-02-19T21:05:47.789 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.789 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.671870+0000", 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013870054 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.685740+0000", 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4508000000000001e-05 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.685785+0000", 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017253208999999999 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.790 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.791 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.791 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.791 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.24919.0:2079 28.9 e132/99 28:90852524:test-rados-api-smithi163-83920-17::big2.copy:head v 132'1, mlcod=0'0)", 2024-02-19T21:05:47.791 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.686869+0000", 2024-02-19T21:05:47.791 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 35.019209359999998, 2024-02-19T21:05:47.791 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0096879210000000004, 2024-02-19T21:05:47.791 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.791 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.791 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.791 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.791 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.791 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.686869+0000", 2024-02-19T21:05:47.791 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.791 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.791 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.792 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.792 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.686869+0000", 2024-02-19T21:05:47.792 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:05:47.792 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.792 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.792 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.792 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.686865+0000", 2024-02-19T21:05:47.792 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00082622800000000003 2024-02-19T21:05:47.792 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.792 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.792 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.792 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.687691+0000", 2024-02-19T21:05:47.792 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7241e-05 2024-02-19T21:05:47.792 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.793 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.793 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.793 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.687709+0000", 2024-02-19T21:05:47.793 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3516e-05 2024-02-19T21:05:47.793 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.793 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.793 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.793 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.687732+0000", 2024-02-19T21:05:47.793 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4459000000000002e-05 2024-02-19T21:05:47.793 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.793 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.793 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.793 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.687787+0000", 2024-02-19T21:05:47.793 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.618e-05 2024-02-19T21:05:47.793 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.794 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.794 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.794 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.687813+0000", 2024-02-19T21:05:47.794 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0086885639999999993 2024-02-19T21:05:47.794 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.794 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.794 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.794 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.696501+0000", 2024-02-19T21:05:47.794 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5009000000000001e-05 2024-02-19T21:05:47.794 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.794 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.794 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.794 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.696556+0000", 2024-02-19T21:05:47.794 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0096879210000000004 2024-02-19T21:05:47.795 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.795 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.795 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.795 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.795 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.795 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.0.0:262 28.2 e132/99 -30:4327d3d8:::temp_28.2_0_14232_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:05:47.795 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.757694+0000", 2024-02-19T21:05:47.795 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 34.948383473, 2024-02-19T21:05:47.795 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037835989, 2024-02-19T21:05:47.795 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.795 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.795 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.795 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.795 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.795 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.757694+0000", 2024-02-19T21:05:47.796 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.796 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.796 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.796 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.796 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.757694+0000", 2024-02-19T21:05:47.796 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:05:47.796 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.796 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.796 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.796 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.757692+0000", 2024-02-19T21:05:47.796 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.029339015 2024-02-19T21:05:47.796 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.796 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.796 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.796 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787031+0000", 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0139000000000001e-05 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787052+0000", 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0044000000000001e-05 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787062+0000", 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8369000000000003e-05 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787120+0000", 2024-02-19T21:05:47.797 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7753000000000001e-05 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787138+0000", 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0083446010000000001 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.795482+0000", 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8127000000000002e-05 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.795530+0000", 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037835989 2024-02-19T21:05:47.798 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.0.0:263 28.2 e132/99 -30:4327d3d8:::temp_28.2_0_14232_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.787076+0000", 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 34.919001926999996, 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.019684811, 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787076+0000", 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.799 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.800 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.800 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.800 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787076+0000", 2024-02-19T21:05:47.800 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:05:47.800 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.800 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.800 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.800 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787074+0000", 2024-02-19T21:05:47.800 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011113661 2024-02-19T21:05:47.800 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.800 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.800 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.800 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.798187+0000", 2024-02-19T21:05:47.800 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2937e-05 2024-02-19T21:05:47.800 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.801 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.801 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.801 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.798200+0000", 2024-02-19T21:05:47.801 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1439999999999997e-06 2024-02-19T21:05:47.801 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.801 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.801 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.801 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.798207+0000", 2024-02-19T21:05:47.801 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7419000000000002e-05 2024-02-19T21:05:47.801 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.801 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.801 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.801 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.798245+0000", 2024-02-19T21:05:47.801 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7419999999999999e-05 2024-02-19T21:05:47.801 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.802 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.802 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.802 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.798262+0000", 2024-02-19T21:05:47.802 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0084319070000000006 2024-02-19T21:05:47.802 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.802 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.802 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.802 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.806694+0000", 2024-02-19T21:05:47.802 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6605000000000004e-05 2024-02-19T21:05:47.802 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.802 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.802 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.802 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.806761+0000", 2024-02-19T21:05:47.802 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.019684811 2024-02-19T21:05:47.802 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.803 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.803 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.803 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.803 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.803 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.0.0:264 28.2 e132/99 -30:4327d3d8:::temp_28.2_0_14232_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:05:47.803 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.801755+0000", 2024-02-19T21:05:47.803 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 34.904322721, 2024-02-19T21:05:47.803 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022050645000000001, 2024-02-19T21:05:47.803 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.803 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.803 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.803 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.803 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.803 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.801755+0000", 2024-02-19T21:05:47.804 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.804 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.804 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.804 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.804 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.801755+0000", 2024-02-19T21:05:47.804 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:05:47.804 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.804 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.804 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.804 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.801753+0000", 2024-02-19T21:05:47.804 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013699945999999999 2024-02-19T21:05:47.804 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.804 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.804 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.804 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.815453+0000", 2024-02-19T21:05:47.805 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7495000000000001e-05 2024-02-19T21:05:47.805 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.805 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.805 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.805 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.815471+0000", 2024-02-19T21:05:47.805 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2979999999999992e-06 2024-02-19T21:05:47.805 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.805 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.805 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.805 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.815479+0000", 2024-02-19T21:05:47.805 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8300999999999997e-05 2024-02-19T21:05:47.805 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.805 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.805 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.805 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.815527+0000", 2024-02-19T21:05:47.806 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7969999999999999e-05 2024-02-19T21:05:47.806 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.806 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.806 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.806 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.815545+0000", 2024-02-19T21:05:47.806 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0082266249999999996 2024-02-19T21:05:47.806 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.806 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.806 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.806 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.823772+0000", 2024-02-19T21:05:47.806 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4051000000000003e-05 2024-02-19T21:05:47.806 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.806 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.806 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.806 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.823806+0000", 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022050645000000001 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.24919.0:2080 28.2 e132/99 28:4327d3d8:test-rados-api-smithi163-83920-17::big3.copy:head v 132'1, mlcod=0'0)", 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.823445+0000", 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 34.882633343000002, 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0097812209999999997, 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.807 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.823445+0000", 2024-02-19T21:05:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.823445+0000", 2024-02-19T21:05:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:05:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.823442+0000", 2024-02-19T21:05:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0033991920000000001 2024-02-19T21:05:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.808 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.826841+0000", 2024-02-19T21:05:47.809 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3720000000000001e-05 2024-02-19T21:05:47.809 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.809 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.809 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.809 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.826855+0000", 2024-02-19T21:05:47.809 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2879999999999993e-06 2024-02-19T21:05:47.809 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.809 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.809 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.809 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.826863+0000", 2024-02-19T21:05:47.809 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1155000000000002e-05 2024-02-19T21:05:47.809 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.809 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.809 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.809 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.826904+0000", 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9006e-05 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.826933+0000", 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0062467809999999999 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.833180+0000", 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5667999999999997e-05 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.833226+0000", 2024-02-19T21:05:47.810 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0097812209999999997 2024-02-19T21:05:47.811 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.811 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.811 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.811 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.811 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.811 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.4.0:287 85.10 e139/133 85:0d6787e5:test-rados-api-smithi163-83928-12::baz:2 v 139'4, mlcod=138'1)", 2024-02-19T21:05:47.811 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:19.187930+0000", 2024-02-19T21:05:47.811 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 28.518148002, 2024-02-19T21:05:47.811 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.006453365, 2024-02-19T21:05:47.811 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:05:47.811 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:05:47.811 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:05:47.811 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.811 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:05:47.811 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.187930+0000", 2024-02-19T21:05:47.812 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:05:47.812 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.812 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.812 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:05:47.812 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.187930+0000", 2024-02-19T21:05:47.812 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:05:47.812 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.812 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.812 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:05:47.812 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.187930+0000", 2024-02-19T21:05:47.812 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8489999999999995e-06 2024-02-19T21:05:47.812 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.812 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.812 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:05:47.812 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.187938+0000", 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9660000000000008e-06 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.187948+0000", 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0369999999999995e-06 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.187956+0000", 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00031348500000000001 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.188270+0000", 2024-02-19T21:05:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.959e-05 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.188289+0000", 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0060582190000000001 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.194348+0000", 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5559999999999998e-05 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.194383+0000", 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.006453365 2024-02-19T21:05:47.814 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.815 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.815 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.815 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:05:47.815 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:05:47.815 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:05:47.815 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:47.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:46.046844+0000 osd.2 (osd.2) 79 : cluster [DBG] 2.21 deep-scrub starts 2024-02-19T21:05:47.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:46.049870+0000 osd.2 (osd.2) 80 : cluster [DBG] 2.21 deep-scrub ok 2024-02-19T21:05:47.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:47.175090+0000 mon.a (mon.0) 1247 : audit [INF] from='client.48947 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi163-83952-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]': finished 2024-02-19T21:05:47.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:47.175194+0000 mon.a (mon.0) 1248 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1"}]': finished 2024-02-19T21:05:47.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:47.175318+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-02-19T21:05:47.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:47.193015+0000 mon.b (mon.1) 266 : audit [INF] from='client.? v1:172.21.15.163:0/1441559216' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi163-83845-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:47.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:47.194370+0000 mon.a (mon.0) 1250 : audit [INF] from='client.47773 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi163-83845-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:47.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:47.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:47.890 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:46.046844+0000 osd.2 (osd.2) 79 : cluster [DBG] 2.21 deep-scrub starts 2024-02-19T21:05:47.890 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:46.049870+0000 osd.2 (osd.2) 80 : cluster [DBG] 2.21 deep-scrub ok 2024-02-19T21:05:47.890 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:47.175090+0000 mon.a (mon.0) 1247 : audit [INF] from='client.48947 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi163-83952-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]': finished 2024-02-19T21:05:47.890 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:47.175194+0000 mon.a (mon.0) 1248 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1"}]': finished 2024-02-19T21:05:47.890 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:47.175318+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-02-19T21:05:47.890 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:47.193015+0000 mon.b (mon.1) 266 : audit [INF] from='client.? v1:172.21.15.163:0/1441559216' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi163-83845-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:47.890 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:47.194370+0000 mon.a (mon.0) 1250 : audit [INF] from='client.47773 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi163-83845-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:47.890 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:46.046844+0000 osd.2 (osd.2) 79 : cluster [DBG] 2.21 deep-scrub starts 2024-02-19T21:05:47.890 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:46.049870+0000 osd.2 (osd.2) 80 : cluster [DBG] 2.21 deep-scrub ok 2024-02-19T21:05:47.890 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:47.175090+0000 mon.a (mon.0) 1247 : audit [INF] from='client.48947 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi163-83952-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]': finished 2024-02-19T21:05:47.891 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:47.175194+0000 mon.a (mon.0) 1248 : audit [INF] from='client.24919 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi163-83920-1","app":"app1","key":"key1"}]': finished 2024-02-19T21:05:47.891 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:47.175318+0000 mon.a (mon.0) 1249 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-02-19T21:05:47.891 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:47.193015+0000 mon.b (mon.1) 266 : audit [INF] from='client.? v1:172.21.15.163:0/1441559216' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi163-83845-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:47.891 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:47.194370+0000 mon.a (mon.0) 1250 : audit [INF] from='client.47773 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi163-83845-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:47.918 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:05:47.948 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:48.062 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:05:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:05:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:05:48.146 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:05:48.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:48.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:05:48.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.164+0000 7fb52d5d1700 1 -- 172.21.15.163:0/1255606344 >> v1:172.21.15.163:6790/0 conn(0x7fb5281042c0 legacy=0x7fb528106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:48.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.164+0000 7fb52d5d1700 1 -- 172.21.15.163:0/1255606344 shutdown_connections 2024-02-19T21:05:48.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.164+0000 7fb52d5d1700 1 -- 172.21.15.163:0/1255606344 >> 172.21.15.163:0/1255606344 conn(0x7fb52806cc00 msgr2=0x7fb5280f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:48.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.164+0000 7fb52d5d1700 1 -- 172.21.15.163:0/1255606344 shutdown_connections 2024-02-19T21:05:48.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.164+0000 7fb52d5d1700 1 -- 172.21.15.163:0/1255606344 wait complete. 2024-02-19T21:05:48.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.165+0000 7fb52d5d1700 1 librados: starting msgr at 2024-02-19T21:05:48.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.165+0000 7fb52d5d1700 1 librados: starting objecter 2024-02-19T21:05:48.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.165+0000 7fb52d5d1700 1 Processor -- start 2024-02-19T21:05:48.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.165+0000 7fb52d5d1700 1 -- start start 2024-02-19T21:05:48.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.165+0000 7fb52d5d1700 1 librados: setting wanted keys 2024-02-19T21:05:48.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.165+0000 7fb52d5d1700 1 librados: calling monclient init 2024-02-19T21:05:48.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.165+0000 7fb52d5d1700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb528117270 con 0x7fb5281007b0 2024-02-19T21:05:48.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.165+0000 7fb52d5d1700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb528118450 con 0x7fb5281042c0 2024-02-19T21:05:48.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.165+0000 7fb52d5d1700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb528119650 con 0x7fb5280fd960 2024-02-19T21:05:48.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.165+0000 7fb526ffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fb5280fd960 0x7fb5280f9a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38760/0 (socket says 172.21.15.163:38760) 2024-02-19T21:05:48.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.165+0000 7fb526ffd700 1 -- 172.21.15.163:0/946402823 learned_addr learned my addr 172.21.15.163:0/946402823 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:48.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.165+0000 7fb50ffff700 10 client.?.objecter ms_handle_connect 0x7fb5280fd960 2024-02-19T21:05:48.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.165+0000 7fb50ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:48.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 10 client.?.objecter ms_handle_connect 0x7fb5281042c0 2024-02-19T21:05:48.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:48.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797152476 0 0) 0x7fb528119650 con 0x7fb5280fd960 2024-02-19T21:05:48.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb510003800 con 0x7fb5280fd960 2024-02-19T21:05:48.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 10 client.?.objecter ms_handle_connect 0x7fb5281007b0 2024-02-19T21:05:48.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:48.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359361065 0 0) 0x7fb528118450 con 0x7fb5281042c0 2024-02-19T21:05:48.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb528119650 con 0x7fb5281042c0 2024-02-19T21:05:48.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097894111 0 0) 0x7fb528117270 con 0x7fb5281007b0 2024-02-19T21:05:48.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb528118450 con 0x7fb5281007b0 2024-02-19T21:05:48.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4077075233 0 0) 0x7fb510003800 con 0x7fb5280fd960 2024-02-19T21:05:48.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb528117270 con 0x7fb5280fd960 2024-02-19T21:05:48.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb518001980 con 0x7fb5280fd960 2024-02-19T21:05:48.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2500044793 0 0) 0x7fb528119650 con 0x7fb5281042c0 2024-02-19T21:05:48.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb510003800 con 0x7fb5281042c0 2024-02-19T21:05:48.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb520001930 con 0x7fb5281042c0 2024-02-19T21:05:48.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1573728266 0 0) 0x7fb528118450 con 0x7fb5281007b0 2024-02-19T21:05:48.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb528119650 con 0x7fb5281007b0 2024-02-19T21:05:48.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb51c001620 con 0x7fb5281007b0 2024-02-19T21:05:48.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 931313383 0 0) 0x7fb528117270 con 0x7fb5280fd960 2024-02-19T21:05:48.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 >> v1:172.21.15.193:6789/0 conn(0x7fb5281042c0 legacy=0x7fb528115950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:48.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 >> v1:172.21.15.163:6789/0 conn(0x7fb5281007b0 legacy=0x7fb5280fa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:48.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52811a850 con 0x7fb5280fd960 2024-02-19T21:05:48.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb52d5d1700 1 -- 172.21.15.163:0/946402823 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb528118640 con 0x7fb5280fd960 2024-02-19T21:05:48.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb52d5d1700 10 client.48971.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:48.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb52d5d1700 1 -- 172.21.15.163:0/946402823 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb528118b30 con 0x7fb5280fd960 2024-02-19T21:05:48.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb52d5d1700 1 librados: init done 2024-02-19T21:05:48.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb518002d00 con 0x7fb5280fd960 2024-02-19T21:05:48.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.166+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb518005ae0 con 0x7fb5280fd960 2024-02-19T21:05:48.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.167+0000 7fb52d5d1700 1 -- 172.21.15.163:0/946402823 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb514004f20 con 0x7fb5280fd960 2024-02-19T21:05:48.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.168+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fb518018970 con 0x7fb5280fd960 2024-02-19T21:05:48.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.168+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(167..167 src has 43..167) v4 ==== 9746+0+0 (unknown 782033756 0 0) 0x7fb518079330 con 0x7fb5280fd960 2024-02-19T21:05:48.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.168+0000 7fb50ffff700 10 client.48971.objecter ms_dispatch 0x7fb528001550 osd_map(167..167 src has 43..167) v4 2024-02-19T21:05:48.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.168+0000 7fb50ffff700 3 client.48971.objecter handle_osd_map got epochs [167,167] > 0 2024-02-19T21:05:48.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.168+0000 7fb50ffff700 3 client.48971.objecter handle_osd_map decoding full epoch 167 2024-02-19T21:05:48.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.168+0000 7fb50ffff700 20 client.48971.objecter dump_active .. 0 homeless 2024-02-19T21:05:48.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.168+0000 7fb50ffff700 10 client.48971.objecter ms_handle_connect 0x7fb510063c20 2024-02-19T21:05:48.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.172+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb51804c200 con 0x7fb5280fd960 2024-02-19T21:05:48.192 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-02-19T21:05:48.192 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-02-19T21:05:48.192 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-02-19T21:05:48.192 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-02-19T21:05:48.192 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-02-19T21:05:48.192 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-02-19T21:05:48.192 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: 2024-02-19T21:05:48.192 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-02-19T21:05:48.192 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: seed 83920 2024-02-19T21:05:48.192 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-02-19T21:05:48.193 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (11 ms) 2024-02-19T21:05:48.193 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-02-19T21:05:48.193 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (263 ms) 2024-02-19T21:05:48.193 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-02-19T21:05:48.193 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (34 ms) 2024-02-19T21:05:48.193 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-02-19T21:05:48.193 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (11 ms) 2024-02-19T21:05:48.193 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-02-19T21:05:48.193 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (25 ms) 2024-02-19T21:05:48.193 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-02-19T21:05:48.193 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (8 ms) 2024-02-19T21:05:48.193 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-02-19T21:05:48.193 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (13 ms) 2024-02-19T21:05:48.193 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-02-19T21:05:48.194 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (6 ms) 2024-02-19T21:05:48.194 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-02-19T21:05:48.194 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (4 ms) 2024-02-19T21:05:48.194 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-02-19T21:05:48.194 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (14 ms) 2024-02-19T21:05:48.194 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-02-19T21:05:48.194 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (4 ms) 2024-02-19T21:05:48.194 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-02-19T21:05:48.194 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (5 ms) 2024-02-19T21:05:48.194 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-02-19T21:05:48.194 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (9 ms) 2024-02-19T21:05:48.194 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-02-19T21:05:48.194 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: osd_max_attr_size = 0 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1593 ms) 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (450 ms) 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: done waiting, doing copies 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: waiting for final deep scrubs... 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: done waiting 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60849 ms) 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (5 ms) 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (2 ms) 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-02-19T21:05:48.195 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4318 ms) 2024-02-19T21:05:48.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-02-19T21:05:48.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-02-19T21:05:48.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-02-19T21:05:48.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-02-19T21:05:48.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-02-19T21:05:48.196 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-02-19T21:05:48.250 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:05:48.251 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:05:48.282 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:48.305 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:05:48.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:48.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:05:48.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:48.397+0000 7fb52d5d1700 1 -- 172.21.15.163:0/946402823 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fb5140051d0 con 0x7fb5280fd960 2024-02-19T21:05:48.409 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:05:48.437 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:48.526 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:48.535 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:05:48.536 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-19T21:05:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:48.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:05:48.639 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:48.666 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:48.685 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:48.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:48.789 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:05:48.817 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:48.836 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:05:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:48.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:05:48.852 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:05:48.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:05:48.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:47.130632+0000 mgr.y (mgr.14550) 213 : cluster [DBG] pgmap v221: 333 pgs: 32 unknown, 301 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 51 B/s, 4 objects/s recovering 2024-02-19T21:05:48.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:47.538108+0000 osd.4 (osd.4) 87 : cluster [DBG] 6.2 deep-scrub starts 2024-02-19T21:05:48.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:47.538913+0000 osd.4 (osd.4) 88 : cluster [DBG] 6.2 deep-scrub ok 2024-02-19T21:05:48.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:47.884298+0000 osd.1 (osd.1) 105 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:05:48.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:47.885021+0000 osd.1 (osd.1) 106 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:05:48.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:48.180636+0000 mon.a (mon.0) 1251 : audit [INF] from='client.47773 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi163-83845-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:48.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:48.180764+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-02-19T21:05:48.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:48.189375+0000 mon.b (mon.1) 267 : audit [INF] from='client.? v1:172.21.15.163:0/4153636191' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi163-83854-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:48.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:48.189975+0000 mon.a (mon.0) 1253 : audit [INF] from='client.47779 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi163-83854-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:48.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:48.233023+0000 mon.b (mon.1) 268 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:48.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:48.233633+0000 mon.a (mon.0) 1254 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:48.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:48.234595+0000 mon.b (mon.1) 269 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:48.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:48.234954+0000 mon.a (mon.0) 1255 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:48.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:48.235432+0000 mon.b (mon.1) 270 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83920-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:48.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:48.235711+0000 mon.a (mon.0) 1256 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83920-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:48.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:48.277004+0000 mon.c (mon.2) 156 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:48.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:48.398526+0000 mon.c (mon.2) 157 : audit [INF] from='client.? v1:172.21.15.163:0/946402823' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:05:48.968 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:48.398886+0000 mon.a (mon.0) 1257 : audit [INF] from='client.48971 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:05:48.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:47.130632+0000 mgr.y (mgr.14550) 213 : cluster [DBG] pgmap v221: 333 pgs: 32 unknown, 301 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 51 B/s, 4 objects/s recovering 2024-02-19T21:05:48.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:47.538108+0000 osd.4 (osd.4) 87 : cluster [DBG] 6.2 deep-scrub starts 2024-02-19T21:05:48.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:47.538913+0000 osd.4 (osd.4) 88 : cluster [DBG] 6.2 deep-scrub ok 2024-02-19T21:05:48.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:47.884298+0000 osd.1 (osd.1) 105 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:05:48.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:47.885021+0000 osd.1 (osd.1) 106 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:05:48.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:48.180636+0000 mon.a (mon.0) 1251 : audit [INF] from='client.47773 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi163-83845-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:48.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:48.180764+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-02-19T21:05:48.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:48.189375+0000 mon.b (mon.1) 267 : audit [INF] from='client.? v1:172.21.15.163:0/4153636191' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi163-83854-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:48.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:48.189975+0000 mon.a (mon.0) 1253 : audit [INF] from='client.47779 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi163-83854-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:48.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:48.233023+0000 mon.b (mon.1) 268 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:48.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:48.233633+0000 mon.a (mon.0) 1254 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:48.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:48.234595+0000 mon.b (mon.1) 269 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:48.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:48.234954+0000 mon.a (mon.0) 1255 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:48.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:48.235432+0000 mon.b (mon.1) 270 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83920-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:48.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:48.235711+0000 mon.a (mon.0) 1256 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83920-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:48.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:48.277004+0000 mon.c (mon.2) 156 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:48.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:48.398526+0000 mon.c (mon.2) 157 : audit [INF] from='client.? v1:172.21.15.163:0/946402823' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:05:48.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:48.398886+0000 mon.a (mon.0) 1257 : audit [INF] from='client.48971 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:05:48.970 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:49.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:47.130632+0000 mgr.y (mgr.14550) 213 : cluster [DBG] pgmap v221: 333 pgs: 32 unknown, 301 active+clean; 216 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 51 B/s, 4 objects/s recovering 2024-02-19T21:05:49.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:47.538108+0000 osd.4 (osd.4) 87 : cluster [DBG] 6.2 deep-scrub starts 2024-02-19T21:05:49.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:47.538913+0000 osd.4 (osd.4) 88 : cluster [DBG] 6.2 deep-scrub ok 2024-02-19T21:05:49.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:47.884298+0000 osd.1 (osd.1) 105 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:05:49.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:47.885021+0000 osd.1 (osd.1) 106 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:05:49.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:48.180636+0000 mon.a (mon.0) 1251 : audit [INF] from='client.47773 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi163-83845-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:49.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:48.180764+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-02-19T21:05:49.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:48.189375+0000 mon.b (mon.1) 267 : audit [INF] from='client.? v1:172.21.15.163:0/4153636191' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi163-83854-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:49.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:48.189975+0000 mon.a (mon.0) 1253 : audit [INF] from='client.47779 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi163-83854-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:49.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:48.233023+0000 mon.b (mon.1) 268 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:49.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:48.233633+0000 mon.a (mon.0) 1254 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:49.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:48.234595+0000 mon.b (mon.1) 269 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:49.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:48.234954+0000 mon.a (mon.0) 1255 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:49.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:48.235432+0000 mon.b (mon.1) 270 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83920-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:49.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:48.235711+0000 mon.a (mon.0) 1256 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83920-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:49.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:48.277004+0000 mon.c (mon.2) 156 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:49.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:48.398526+0000 mon.c (mon.2) 157 : audit [INF] from='client.? v1:172.21.15.163:0/946402823' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:05:49.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:48.398886+0000 mon.a (mon.0) 1257 : audit [INF] from='client.48971 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:05:49.067 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:05:49.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:49.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:05:49.171 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:05:49.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:49.180+0000 7fb50ffff700 1 -- 172.21.15.163:0/946402823 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v169) v1 ==== 87+0+0 (unknown 1125223746 0 0) 0x7fb51804a610 con 0x7fb5280fd960 2024-02-19T21:05:49.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:49.191+0000 7fb52d5d1700 10 librados: watch_flush enter 2024-02-19T21:05:49.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:49.193+0000 7fb52d5d1700 10 librados: watch_flush exit 2024-02-19T21:05:49.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:49.193+0000 7fb52d5d1700 20 client.48971.objecter shutdown clearing up homeless session... 2024-02-19T21:05:49.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:49.193+0000 7fb52d5d1700 10 client.48971.objecter successfully canceled tick 2024-02-19T21:05:49.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:49.197+0000 7fb52d5d1700 1 -- 172.21.15.163:0/946402823 >> v1:172.21.15.163:6800/1552514681 conn(0x7fb510063c20 legacy=0x7fb5100660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:49.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:49.197+0000 7fb52d5d1700 1 -- 172.21.15.163:0/946402823 >> v1:172.21.15.163:6790/0 conn(0x7fb5280fd960 legacy=0x7fb5280f9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:49.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:49.205+0000 7fb52d5d1700 1 -- 172.21.15.163:0/946402823 shutdown_connections 2024-02-19T21:05:49.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:49.205+0000 7fb52d5d1700 1 -- 172.21.15.163:0/946402823 >> 172.21.15.163:0/946402823 conn(0x7fb52806cc00 msgr2=0x7fb528107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:49.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:49.214+0000 7fb52d5d1700 1 -- 172.21.15.163:0/946402823 shutdown_connections 2024-02-19T21:05:49.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:49.214+0000 7fb52d5d1700 1 -- 172.21.15.163:0/946402823 wait complete. 2024-02-19T21:05:49.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:49.215+0000 7fb52d5d1700 1 librados: shutdown 2024-02-19T21:05:49.218 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:05:49.234 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:49.406 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:05:49.407 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:05:49.407 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-02-19T21:05:49.407 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-19T21:05:49.407 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-19T21:05:49.407 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-02-19T21:05:49.586 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:05:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:49.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:05:49.690 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:49.717 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:49.754 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:49.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:49.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:49.862 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:49.888 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:49.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:47.803992+0000 osd.0 (osd.0) 95 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:05:49.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:47.805753+0000 osd.0 (osd.0) 96 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:05:49.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:49.131657+0000 mon.b (mon.1) 271 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-19T21:05:49.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:49.132754+0000 mon.a (mon.0) 1258 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-19T21:05:49.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:49.174852+0000 mon.a (mon.0) 1259 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:05:49.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:49.179193+0000 mon.a (mon.0) 1260 : audit [INF] from='client.47779 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi163-83854-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:49.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:49.179434+0000 mon.a (mon.0) 1261 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83920-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:05:49.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:49.179495+0000 mon.a (mon.0) 1262 : audit [INF] from='client.48971 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:05:49.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:49.179528+0000 mon.a (mon.0) 1263 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-19T21:05:49.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:49.179561+0000 mon.a (mon.0) 1264 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-02-19T21:05:49.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:49.187383+0000 mon.b (mon.1) 272 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83920-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:49.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:49.207148+0000 mon.a (mon.0) 1265 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83920-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:49.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:49.277832+0000 mon.c (mon.2) 158 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:49.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:47.803992+0000 osd.0 (osd.0) 95 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:05:49.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:47.805753+0000 osd.0 (osd.0) 96 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:05:49.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:49.131657+0000 mon.b (mon.1) 271 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-19T21:05:49.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:49.132754+0000 mon.a (mon.0) 1258 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-19T21:05:49.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:49.174852+0000 mon.a (mon.0) 1259 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:05:49.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:49.179193+0000 mon.a (mon.0) 1260 : audit [INF] from='client.47779 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi163-83854-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:49.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:49.179434+0000 mon.a (mon.0) 1261 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83920-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:05:49.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:49.179495+0000 mon.a (mon.0) 1262 : audit [INF] from='client.48971 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:05:49.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:49.179528+0000 mon.a (mon.0) 1263 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-19T21:05:49.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:49.179561+0000 mon.a (mon.0) 1264 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-02-19T21:05:49.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:49.187383+0000 mon.b (mon.1) 272 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83920-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:49.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:49.207148+0000 mon.a (mon.0) 1265 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83920-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:49.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:49.277832+0000 mon.c (mon.2) 158 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:50.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:47.803992+0000 osd.0 (osd.0) 95 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:05:50.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:47.805753+0000 osd.0 (osd.0) 96 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:05:50.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:49.131657+0000 mon.b (mon.1) 271 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-19T21:05:50.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:49.132754+0000 mon.a (mon.0) 1258 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-02-19T21:05:50.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:49.174852+0000 mon.a (mon.0) 1259 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:05:50.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:49.179193+0000 mon.a (mon.0) 1260 : audit [INF] from='client.47779 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi163-83854-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:50.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:49.179434+0000 mon.a (mon.0) 1261 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83920-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:05:50.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:49.179495+0000 mon.a (mon.0) 1262 : audit [INF] from='client.48971 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:05:50.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:49.179528+0000 mon.a (mon.0) 1263 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-02-19T21:05:50.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:49.179561+0000 mon.a (mon.0) 1264 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-02-19T21:05:50.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:49.187383+0000 mon.b (mon.1) 272 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83920-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:50.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:49.207148+0000 mon.a (mon.0) 1265 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83920-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:50.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:49.277832+0000 mon.c (mon.2) 158 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:50.086 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:50.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:50.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:50.120 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:05:50.191 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:50.219 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:50.319 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:50.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:50.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:50.348 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:05:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:05:50.426 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:50.455 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:50.578 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:50.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:50.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:50.682 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:50.704 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:49.050858+0000 osd.2 (osd.2) 81 : cluster [DBG] 2.c deep-scrub starts 2024-02-19T21:05:50.704 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:49.051726+0000 osd.2 (osd.2) 82 : cluster [DBG] 2.c deep-scrub ok 2024-02-19T21:05:50.704 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:49.131087+0000 mgr.y (mgr.14550) 214 : cluster [DBG] pgmap v224: 340 pgs: 72 unknown, 268 active+clean; 15 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:50.704 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:49.588499+0000 osd.3 (osd.3) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:50.704 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:49.588901+0000 osd.3 (osd.3) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:50.704 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:50.278527+0000 mon.c (mon.2) 159 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:50.704 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:50.422129+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-02-19T21:05:50.704 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:50.425636+0000 mon.b (mon.1) 273 : audit [INF] from='client.? v1:172.21.15.163:0/3639409363' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi163-83845-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:50.711 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:50.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:49.050858+0000 osd.2 (osd.2) 81 : cluster [DBG] 2.c deep-scrub starts 2024-02-19T21:05:50.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:49.051726+0000 osd.2 (osd.2) 82 : cluster [DBG] 2.c deep-scrub ok 2024-02-19T21:05:50.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:49.131087+0000 mgr.y (mgr.14550) 214 : cluster [DBG] pgmap v224: 340 pgs: 72 unknown, 268 active+clean; 15 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:50.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:49.588499+0000 osd.3 (osd.3) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:50.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:49.588901+0000 osd.3 (osd.3) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:50.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:50.278527+0000 mon.c (mon.2) 159 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:50.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:50.422129+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-02-19T21:05:50.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:50.425636+0000 mon.b (mon.1) 273 : audit [INF] from='client.? v1:172.21.15.163:0/3639409363' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi163-83845-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:50.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:50.426115+0000 mon.a (mon.0) 1267 : audit [INF] from='client.47797 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi163-83845-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:50.728 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:49.050858+0000 osd.2 (osd.2) 81 : cluster [DBG] 2.c deep-scrub starts 2024-02-19T21:05:50.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:49.051726+0000 osd.2 (osd.2) 82 : cluster [DBG] 2.c deep-scrub ok 2024-02-19T21:05:50.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:49.131087+0000 mgr.y (mgr.14550) 214 : cluster [DBG] pgmap v224: 340 pgs: 72 unknown, 268 active+clean; 15 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-02-19T21:05:50.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:49.588499+0000 osd.3 (osd.3) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:50.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:49.588901+0000 osd.3 (osd.3) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:50.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:50.278527+0000 mon.c (mon.2) 159 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:50.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:50.422129+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-02-19T21:05:50.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:50.425636+0000 mon.b (mon.1) 273 : audit [INF] from='client.? v1:172.21.15.163:0/3639409363' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi163-83845-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:50.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:50.426115+0000 mon.a (mon.0) 1267 : audit [INF] from='client.47797 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi163-83845-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:50.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.978+0000 7fa2594a3700 1 -- 172.21.15.163:0/393330165 >> v1:172.21.15.163:6789/0 conn(0x7fa2540fd980 legacy=0x7fa2540fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:50.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.978+0000 7fa2594a3700 1 -- 172.21.15.163:0/393330165 shutdown_connections 2024-02-19T21:05:50.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.978+0000 7fa2594a3700 1 -- 172.21.15.163:0/393330165 >> 172.21.15.163:0/393330165 conn(0x7fa2540f35a0 msgr2=0x7fa2540f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:50.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.979+0000 7fa2594a3700 1 -- 172.21.15.163:0/393330165 shutdown_connections 2024-02-19T21:05:50.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.979+0000 7fa2594a3700 1 -- 172.21.15.163:0/393330165 wait complete. 2024-02-19T21:05:50.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.979+0000 7fa2594a3700 1 librados: starting msgr at 2024-02-19T21:05:50.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.979+0000 7fa2594a3700 1 librados: starting objecter 2024-02-19T21:05:50.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.979+0000 7fa2594a3700 1 Processor -- start 2024-02-19T21:05:50.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.979+0000 7fa2594a3700 1 -- start start 2024-02-19T21:05:50.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.979+0000 7fa2594a3700 1 librados: setting wanted keys 2024-02-19T21:05:50.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.979+0000 7fa2594a3700 1 librados: calling monclient init 2024-02-19T21:05:50.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.979+0000 7fa2594a3700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa254119450 con 0x7fa2541042c0 2024-02-19T21:05:50.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.979+0000 7fa2594a3700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa25411a5f0 con 0x7fa254100780 2024-02-19T21:05:50.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.979+0000 7fa2594a3700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa25411b790 con 0x7fa2540fd980 2024-02-19T21:05:50.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa252ffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fa2540fd980 0x7fa2540fc5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38844/0 (socket says 172.21.15.163:38844) 2024-02-19T21:05:50.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa252ffd700 1 -- 172.21.15.163:0/1117161042 learned_addr learned my addr 172.21.15.163:0/1117161042 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:50.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 10 client.?.objecter ms_handle_connect 0x7fa2540fd980 2024-02-19T21:05:50.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:50.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 10 client.?.objecter ms_handle_connect 0x7fa2541042c0 2024-02-19T21:05:50.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:50.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 10 client.?.objecter ms_handle_connect 0x7fa254100780 2024-02-19T21:05:50.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:50.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075624988 0 0) 0x7fa254119450 con 0x7fa2541042c0 2024-02-19T21:05:50.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa23c0038e0 con 0x7fa2541042c0 2024-02-19T21:05:50.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828206635 0 0) 0x7fa25411b790 con 0x7fa2540fd980 2024-02-19T21:05:50.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa254119450 con 0x7fa2540fd980 2024-02-19T21:05:50.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863683540 0 0) 0x7fa25411a5f0 con 0x7fa254100780 2024-02-19T21:05:50.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa25411b790 con 0x7fa254100780 2024-02-19T21:05:50.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2419086712 0 0) 0x7fa254119450 con 0x7fa2540fd980 2024-02-19T21:05:50.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa25411a5f0 con 0x7fa2540fd980 2024-02-19T21:05:50.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa244002070 con 0x7fa2540fd980 2024-02-19T21:05:50.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2538843049 0 0) 0x7fa23c0038e0 con 0x7fa2541042c0 2024-02-19T21:05:50.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa254119450 con 0x7fa2541042c0 2024-02-19T21:05:50.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4106065564 0 0) 0x7fa25411b790 con 0x7fa254100780 2024-02-19T21:05:50.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa23c0038e0 con 0x7fa254100780 2024-02-19T21:05:50.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa24c001b40 con 0x7fa2541042c0 2024-02-19T21:05:50.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa2480019f0 con 0x7fa254100780 2024-02-19T21:05:50.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2281669506 0 0) 0x7fa25411a5f0 con 0x7fa2540fd980 2024-02-19T21:05:50.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 >> v1:172.21.15.193:6789/0 conn(0x7fa254100780 legacy=0x7fa2540fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:50.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 >> v1:172.21.15.163:6789/0 conn(0x7fa2541042c0 legacy=0x7fa254117d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:50.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25411c930 con 0x7fa2540fd980 2024-02-19T21:05:50.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa254119640 con 0x7fa2540fd980 2024-02-19T21:05:50.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa2594a3700 10 client.48989.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:50.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa254119b30 con 0x7fa2540fd980 2024-02-19T21:05:50.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa2594a3700 1 librados: init done 2024-02-19T21:05:50.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fa2440026b0 con 0x7fa2540fd980 2024-02-19T21:05:50.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.980+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa244003880 con 0x7fa2540fd980 2024-02-19T21:05:50.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.981+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa240004f00 con 0x7fa2540fd980 2024-02-19T21:05:50.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fa244002210 con 0x7fa2540fd980 2024-02-19T21:05:50.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(170..170 src has 43..170) v4 ==== 9777+0+0 (unknown 3592836877 0 0) 0x7fa244079600 con 0x7fa2540fd980 2024-02-19T21:05:50.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 osd_map(170..170 src has 43..170) v4 2024-02-19T21:05:50.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa23bfff700 3 client.48989.objecter handle_osd_map got epochs [170,170] > 0 2024-02-19T21:05:50.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa23bfff700 3 client.48989.objecter handle_osd_map decoding full epoch 170 2024-02-19T21:05:50.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa23bfff700 20 client.48989.objecter dump_active .. 0 homeless 2024-02-19T21:05:50.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v170) v1 ==== 54+0+16 (unknown 127392416 0 2420000192) 0x7fa24407ba10 con 0x7fa2540fd980 2024-02-19T21:05:50.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa23bfff700 10 client.48989.objecter ms_handle_connect 0x7fa23c063d10 2024-02-19T21:05:50.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa2594a3700 10 client.48989.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:50.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 1 2024-02-19T21:05:50.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa2400051a0 osd=0 3 2024-02-19T21:05:50.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa2400051a0 osd=0 3 2024-02-19T21:05:50.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa2400051a0 osd=0 3 2024-02-19T21:05:50.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 1 2024-02-19T21:05:50.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa2400051a0 osd=0 3 2024-02-19T21:05:50.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 0 1 2024-02-19T21:05:50.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa2400051a0 osd=0 4 2024-02-19T21:05:50.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa2594a3700 10 client.48989.objecter _send_command 1 2024-02-19T21:05:50.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.982+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa240004950 con 0x7fa240004530 2024-02-19T21:05:50.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.983+0000 7fa23bfff700 10 client.48989.objecter ms_handle_connect 0x7fa240004530 2024-02-19T21:05:50.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.983+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa240004950 con 0x7fa240004530 2024-02-19T21:05:50.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.983+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:05:50.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.983+0000 7fa23bfff700 10 client.48989.objecter _finish_command 1 = system:0 2024-02-19T21:05:50.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.983+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa2400051a0 osd=0 4 2024-02-19T21:05:50.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.983+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 0 1 2024-02-19T21:05:50.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.998+0000 7fa2594a3700 10 client.48989.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:50.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.998+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 2 2024-02-19T21:05:50.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.998+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa2400051a0 osd=0 3 2024-02-19T21:05:50.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.998+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa2400051a0 osd=0 3 2024-02-19T21:05:50.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.998+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa2400051a0 osd=0 3 2024-02-19T21:05:50.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.998+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 2 2024-02-19T21:05:50.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.998+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa2400051a0 osd=0 3 2024-02-19T21:05:50.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.998+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 0 2 2024-02-19T21:05:50.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.998+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa2400051a0 osd=0 4 2024-02-19T21:05:50.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.998+0000 7fa2594a3700 10 client.48989.objecter _send_command 2 2024-02-19T21:05:50.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.998+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa240004d30 con 0x7fa240004530 2024-02-19T21:05:50.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.999+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa240004d30 con 0x7fa240004530 2024-02-19T21:05:50.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.999+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:50.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.999+0000 7fa23bfff700 10 client.48989.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:50.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.999+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa2400051a0 osd=0 4 2024-02-19T21:05:50.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:50.999+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 0 2 2024-02-19T21:05:50.994 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:05:50.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa2594a3700 10 client.48989.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:50.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 3 2024-02-19T21:05:50.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240003fe0 osd=1 3 2024-02-19T21:05:50.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240003fe0 osd=1 3 2024-02-19T21:05:50.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240003fe0 osd=1 3 2024-02-19T21:05:50.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 3 2024-02-19T21:05:50.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa240003fe0 osd=1 3 2024-02-19T21:05:50.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 1 3 2024-02-19T21:05:50.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240003fe0 osd=1 4 2024-02-19T21:05:50.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa2594a3700 10 client.48989.objecter _send_command 3 2024-02-19T21:05:50.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa24000c000 con 0x7fa240004150 2024-02-19T21:05:50.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa23bfff700 10 client.48989.objecter ms_handle_connect 0x7fa240004150 2024-02-19T21:05:50.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa24000c000 con 0x7fa240004150 2024-02-19T21:05:50.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:05:50.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa23bfff700 10 client.48989.objecter _finish_command 3 = system:0 2024-02-19T21:05:50.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa240003fe0 osd=1 4 2024-02-19T21:05:50.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.000+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 1 3 2024-02-19T21:05:51.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.015+0000 7fa2594a3700 10 client.48989.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:51.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.015+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 4 2024-02-19T21:05:51.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.015+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240003fe0 osd=1 3 2024-02-19T21:05:51.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.015+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240003fe0 osd=1 3 2024-02-19T21:05:51.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.015+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240003fe0 osd=1 3 2024-02-19T21:05:51.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.015+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 4 2024-02-19T21:05:51.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.015+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa240003fe0 osd=1 3 2024-02-19T21:05:51.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.015+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 1 4 2024-02-19T21:05:51.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.015+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240003fe0 osd=1 4 2024-02-19T21:05:51.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.015+0000 7fa2594a3700 10 client.48989.objecter _send_command 4 2024-02-19T21:05:51.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.015+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa240001850 con 0x7fa240004150 2024-02-19T21:05:51.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.016+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa240001850 con 0x7fa240004150 2024-02-19T21:05:51.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.016+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:51.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.016+0000 7fa23bfff700 10 client.48989.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:51.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.016+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa240003fe0 osd=1 4 2024-02-19T21:05:51.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.016+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 1 4 2024-02-19T21:05:51.012 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:05:51.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa2594a3700 10 client.48989.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:51.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 5 2024-02-19T21:05:51.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240003630 osd=2 3 2024-02-19T21:05:51.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240003630 osd=2 3 2024-02-19T21:05:51.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240003630 osd=2 3 2024-02-19T21:05:51.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 5 2024-02-19T21:05:51.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa240003630 osd=2 3 2024-02-19T21:05:51.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 2 5 2024-02-19T21:05:51.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240003630 osd=2 4 2024-02-19T21:05:51.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa2594a3700 10 client.48989.objecter _send_command 5 2024-02-19T21:05:51.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa240003e70 con 0x7fa24000d2f0 2024-02-19T21:05:51.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa23bfff700 10 client.48989.objecter ms_handle_connect 0x7fa24000d2f0 2024-02-19T21:05:51.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa240003e70 con 0x7fa24000d2f0 2024-02-19T21:05:51.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:05:51.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa23bfff700 10 client.48989.objecter _finish_command 5 = system:0 2024-02-19T21:05:51.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa240003630 osd=2 4 2024-02-19T21:05:51.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.017+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 2 5 2024-02-19T21:05:51.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.031+0000 7fa2594a3700 10 client.48989.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:51.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.031+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 6 2024-02-19T21:05:51.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.031+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240003630 osd=2 3 2024-02-19T21:05:51.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.031+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240003630 osd=2 3 2024-02-19T21:05:51.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.031+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240003630 osd=2 3 2024-02-19T21:05:51.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.031+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 6 2024-02-19T21:05:51.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.031+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa240003630 osd=2 3 2024-02-19T21:05:51.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.031+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 2 6 2024-02-19T21:05:51.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.031+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240003630 osd=2 4 2024-02-19T21:05:51.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.031+0000 7fa2594a3700 10 client.48989.objecter _send_command 6 2024-02-19T21:05:51.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.031+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa24000e6c0 con 0x7fa24000d2f0 2024-02-19T21:05:51.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa24000e6c0 con 0x7fa24000d2f0 2024-02-19T21:05:51.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:51.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa23bfff700 10 client.48989.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:51.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa240003630 osd=2 4 2024-02-19T21:05:51.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 2 6 2024-02-19T21:05:51.028 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:05:51.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa2594a3700 10 client.48989.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:51.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 7 2024-02-19T21:05:51.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240003390 osd=3 3 2024-02-19T21:05:51.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240003390 osd=3 3 2024-02-19T21:05:51.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240003390 osd=3 3 2024-02-19T21:05:51.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 7 2024-02-19T21:05:51.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa240003390 osd=3 3 2024-02-19T21:05:51.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 3 7 2024-02-19T21:05:51.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240003390 osd=3 4 2024-02-19T21:05:51.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa2594a3700 10 client.48989.objecter _send_command 7 2024-02-19T21:05:51.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa240015a60 con 0x7fa240011e80 2024-02-19T21:05:51.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.033+0000 7fa23bfff700 10 client.48989.objecter ms_handle_connect 0x7fa240011e80 2024-02-19T21:05:51.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.034+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa240015a60 con 0x7fa240011e80 2024-02-19T21:05:51.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.034+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:05:51.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.034+0000 7fa23bfff700 10 client.48989.objecter _finish_command 7 = system:0 2024-02-19T21:05:51.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.034+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa240003390 osd=3 4 2024-02-19T21:05:51.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.034+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 3 7 2024-02-19T21:05:51.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.048+0000 7fa2594a3700 10 client.48989.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:51.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.048+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 8 2024-02-19T21:05:51.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.048+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240003390 osd=3 3 2024-02-19T21:05:51.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.048+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240003390 osd=3 3 2024-02-19T21:05:51.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.048+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240003390 osd=3 3 2024-02-19T21:05:51.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.048+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 8 2024-02-19T21:05:51.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.048+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa240003390 osd=3 3 2024-02-19T21:05:51.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.048+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 3 8 2024-02-19T21:05:51.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.048+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240003390 osd=3 4 2024-02-19T21:05:51.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.048+0000 7fa2594a3700 10 client.48989.objecter _send_command 8 2024-02-19T21:05:51.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.048+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa240015a60 con 0x7fa240011e80 2024-02-19T21:05:51.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.049+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa240015a60 con 0x7fa240011e80 2024-02-19T21:05:51.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.049+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:51.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.049+0000 7fa23bfff700 10 client.48989.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:51.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.049+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa240003390 osd=3 4 2024-02-19T21:05:51.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.049+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 3 8 2024-02-19T21:05:51.044 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:05:51.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.049+0000 7fa2594a3700 10 client.48989.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:51.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.049+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 9 2024-02-19T21:05:51.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.049+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240015a60 osd=4 3 2024-02-19T21:05:51.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.049+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240015a60 osd=4 3 2024-02-19T21:05:51.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.049+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240015a60 osd=4 3 2024-02-19T21:05:51.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.049+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 9 2024-02-19T21:05:51.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.049+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa240015a60 osd=4 3 2024-02-19T21:05:51.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.050+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 4 9 2024-02-19T21:05:51.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.050+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240015a60 osd=4 4 2024-02-19T21:05:51.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.050+0000 7fa2594a3700 10 client.48989.objecter _send_command 9 2024-02-19T21:05:51.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.050+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa24000cec0 con 0x7fa24000c940 2024-02-19T21:05:51.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.050+0000 7fa23bfff700 10 client.48989.objecter ms_handle_connect 0x7fa24000c940 2024-02-19T21:05:51.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.051+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa24000cec0 con 0x7fa24000c940 2024-02-19T21:05:51.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.051+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:05:51.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.051+0000 7fa23bfff700 10 client.48989.objecter _finish_command 9 = system:0 2024-02-19T21:05:51.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.051+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa240015a60 osd=4 4 2024-02-19T21:05:51.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.051+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 4 9 2024-02-19T21:05:51.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.065+0000 7fa2594a3700 10 client.48989.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:51.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.065+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 10 2024-02-19T21:05:51.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.065+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240015a60 osd=4 3 2024-02-19T21:05:51.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.065+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240015a60 osd=4 3 2024-02-19T21:05:51.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.065+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240015a60 osd=4 3 2024-02-19T21:05:51.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.065+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 10 2024-02-19T21:05:51.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.065+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa240015a60 osd=4 3 2024-02-19T21:05:51.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.065+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 4 10 2024-02-19T21:05:51.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.065+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240015a60 osd=4 4 2024-02-19T21:05:51.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.065+0000 7fa2594a3700 10 client.48989.objecter _send_command 10 2024-02-19T21:05:51.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.065+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa24000cec0 con 0x7fa24000c940 2024-02-19T21:05:51.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa24000cec0 con 0x7fa24000c940 2024-02-19T21:05:51.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:51.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa23bfff700 10 client.48989.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:51.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa240015a60 osd=4 4 2024-02-19T21:05:51.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 4 10 2024-02-19T21:05:51.062 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:05:51.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa2594a3700 10 client.48989.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:51.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 11 2024-02-19T21:05:51.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa24000cec0 osd=5 3 2024-02-19T21:05:51.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa24000cec0 osd=5 3 2024-02-19T21:05:51.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa24000cec0 osd=5 3 2024-02-19T21:05:51.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 11 2024-02-19T21:05:51.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa24000cec0 osd=5 3 2024-02-19T21:05:51.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 5 11 2024-02-19T21:05:51.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa24000cec0 osd=5 4 2024-02-19T21:05:51.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa2594a3700 10 client.48989.objecter _send_command 11 2024-02-19T21:05:51.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.067+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa240017de0 con 0x7fa2400179a0 2024-02-19T21:05:51.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.068+0000 7fa23bfff700 10 client.48989.objecter ms_handle_connect 0x7fa2400179a0 2024-02-19T21:05:51.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.068+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa240017de0 con 0x7fa2400179a0 2024-02-19T21:05:51.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.068+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:05:51.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.068+0000 7fa23bfff700 10 client.48989.objecter _finish_command 11 = system:0 2024-02-19T21:05:51.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.068+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa24000cec0 osd=5 4 2024-02-19T21:05:51.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.068+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 5 11 2024-02-19T21:05:51.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:50.426115+0000 mon.a (mon.0) 1267 : audit [INF] from='client.47797 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi163-83845-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:51.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.082+0000 7fa2594a3700 10 client.48989.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:51.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.082+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 12 2024-02-19T21:05:51.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.082+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa24000cec0 osd=5 3 2024-02-19T21:05:51.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.082+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa24000cec0 osd=5 3 2024-02-19T21:05:51.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.082+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa24000cec0 osd=5 3 2024-02-19T21:05:51.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.082+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 12 2024-02-19T21:05:51.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.082+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa24000cec0 osd=5 3 2024-02-19T21:05:51.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.082+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 5 12 2024-02-19T21:05:51.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.082+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa24000cec0 osd=5 4 2024-02-19T21:05:51.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.082+0000 7fa2594a3700 10 client.48989.objecter _send_command 12 2024-02-19T21:05:51.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.082+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa240017de0 con 0x7fa2400179a0 2024-02-19T21:05:51.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa240017de0 con 0x7fa2400179a0 2024-02-19T21:05:51.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:51.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa23bfff700 10 client.48989.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:51.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa24000cec0 osd=5 4 2024-02-19T21:05:51.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 5 12 2024-02-19T21:05:51.079 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:05:51.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa2594a3700 10 client.48989.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:51.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 13 2024-02-19T21:05:51.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240017de0 osd=6 3 2024-02-19T21:05:51.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240017de0 osd=6 3 2024-02-19T21:05:51.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240017de0 osd=6 3 2024-02-19T21:05:51.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 13 2024-02-19T21:05:51.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa240017de0 osd=6 3 2024-02-19T21:05:51.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 6 13 2024-02-19T21:05:51.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240017de0 osd=6 4 2024-02-19T21:05:51.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa2594a3700 10 client.48989.objecter _send_command 13 2024-02-19T21:05:51.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.084+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2400181b0 con 0x7fa24001f060 2024-02-19T21:05:51.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.085+0000 7fa23bfff700 10 client.48989.objecter ms_handle_connect 0x7fa24001f060 2024-02-19T21:05:51.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.086+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa2400181b0 con 0x7fa24001f060 2024-02-19T21:05:51.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.086+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:05:51.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.086+0000 7fa23bfff700 10 client.48989.objecter _finish_command 13 = system:0 2024-02-19T21:05:51.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.086+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa240017de0 osd=6 4 2024-02-19T21:05:51.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.086+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 6 13 2024-02-19T21:05:51.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.099+0000 7fa2594a3700 10 client.48989.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:51.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.099+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 14 2024-02-19T21:05:51.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.099+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240017de0 osd=6 3 2024-02-19T21:05:51.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.099+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240017de0 osd=6 3 2024-02-19T21:05:51.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.099+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa240017de0 osd=6 3 2024-02-19T21:05:51.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.099+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 14 2024-02-19T21:05:51.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.099+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa240017de0 osd=6 3 2024-02-19T21:05:51.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.099+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 6 14 2024-02-19T21:05:51.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.099+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240017de0 osd=6 4 2024-02-19T21:05:51.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.099+0000 7fa2594a3700 10 client.48989.objecter _send_command 14 2024-02-19T21:05:51.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.099+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa2400181b0 con 0x7fa24001f060 2024-02-19T21:05:51.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa2400181b0 con 0x7fa24001f060 2024-02-19T21:05:51.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:51.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa23bfff700 10 client.48989.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:51.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa240017de0 osd=6 4 2024-02-19T21:05:51.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 6 14 2024-02-19T21:05:51.096 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:05:51.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa2594a3700 10 client.48989.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:51.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 15 2024-02-19T21:05:51.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa2400181b0 osd=7 3 2024-02-19T21:05:51.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa2400181b0 osd=7 3 2024-02-19T21:05:51.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa2400181b0 osd=7 3 2024-02-19T21:05:51.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 15 2024-02-19T21:05:51.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa2400181b0 osd=7 3 2024-02-19T21:05:51.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 7 15 2024-02-19T21:05:51.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa2400181b0 osd=7 4 2024-02-19T21:05:51.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa2594a3700 10 client.48989.objecter _send_command 15 2024-02-19T21:05:51.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.101+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa240024d60 con 0x7fa240023080 2024-02-19T21:05:51.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.102+0000 7fa23bfff700 10 client.48989.objecter ms_handle_connect 0x7fa240023080 2024-02-19T21:05:51.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.102+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa240024d60 con 0x7fa240023080 2024-02-19T21:05:51.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.102+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:05:51.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.102+0000 7fa23bfff700 10 client.48989.objecter _finish_command 15 = system:0 2024-02-19T21:05:51.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.102+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa2400181b0 osd=7 4 2024-02-19T21:05:51.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.102+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 7 15 2024-02-19T21:05:51.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.116+0000 7fa2594a3700 10 client.48989.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:05:51.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.116+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign -1 16 2024-02-19T21:05:51.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.116+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa2400181b0 osd=7 3 2024-02-19T21:05:51.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.116+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa2400181b0 osd=7 3 2024-02-19T21:05:51.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.116+0000 7fa2594a3700 20 client.48989.objecter _get_session s=0x7fa2400181b0 osd=7 3 2024-02-19T21:05:51.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.116+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_remove -1 16 2024-02-19T21:05:51.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.116+0000 7fa2594a3700 20 client.48989.objecter get_session s=0x7fa2400181b0 osd=7 3 2024-02-19T21:05:51.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.116+0000 7fa2594a3700 15 client.48989.objecter _session_command_op_assign 7 16 2024-02-19T21:05:51.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.116+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa2400181b0 osd=7 4 2024-02-19T21:05:51.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.116+0000 7fa2594a3700 10 client.48989.objecter _send_command 16 2024-02-19T21:05:51.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.116+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa240024d60 con 0x7fa240023080 2024-02-19T21:05:51.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa23bfff700 1 -- 172.21.15.163:0/1117161042 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa240024d60 con 0x7fa240023080 2024-02-19T21:05:51.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa23bfff700 10 client.48989.objecter ms_dispatch 0x7fa254001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:05:51.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa23bfff700 10 client.48989.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:05:51.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa23bfff700 20 client.48989.objecter put_session s=0x7fa2400181b0 osd=7 4 2024-02-19T21:05:51.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa23bfff700 15 client.48989.objecter _session_command_op_remove 7 16 2024-02-19T21:05:51.113 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:05:51.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 10 librados: watch_flush enter 2024-02-19T21:05:51.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 10 librados: watch_flush exit 2024-02-19T21:05:51.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 10 client.48989.objecter close_session for osd.0 2024-02-19T21:05:51.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 >> v1:172.21.15.163:6801/1559418115 conn(0x7fa240004530 legacy=0x7fa240002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa2400051a0 osd=0 1 2024-02-19T21:05:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 10 client.48989.objecter close_session for osd.1 2024-02-19T21:05:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 >> v1:172.21.15.163:6805/3125047432 conn(0x7fa240004150 legacy=0x7fa24000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240003fe0 osd=1 1 2024-02-19T21:05:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 10 client.48989.objecter close_session for osd.2 2024-02-19T21:05:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 >> v1:172.21.15.163:6809/1563158487 conn(0x7fa24000d2f0 legacy=0x7fa24000d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240003630 osd=2 1 2024-02-19T21:05:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 10 client.48989.objecter close_session for osd.3 2024-02-19T21:05:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 >> v1:172.21.15.163:6813/2112279769 conn(0x7fa240011e80 legacy=0x7fa240014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240003390 osd=3 1 2024-02-19T21:05:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 10 client.48989.objecter close_session for osd.4 2024-02-19T21:05:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 >> v1:172.21.15.193:6800/3129716965 conn(0x7fa24000c940 legacy=0x7fa240015c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240015a60 osd=4 1 2024-02-19T21:05:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 10 client.48989.objecter close_session for osd.5 2024-02-19T21:05:51.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 >> v1:172.21.15.193:6804/805161991 conn(0x7fa2400179a0 legacy=0x7fa24001d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:51.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2537fe700 1 -- 172.21.15.163:0/1117161042 reap_dead start 2024-02-19T21:05:51.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa24000cec0 osd=5 1 2024-02-19T21:05:51.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 10 client.48989.objecter close_session for osd.6 2024-02-19T21:05:51.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 >> v1:172.21.15.193:6808/787604615 conn(0x7fa24001f060 legacy=0x7fa240021450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:51.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa240017de0 osd=6 1 2024-02-19T21:05:51.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 10 client.48989.objecter close_session for osd.7 2024-02-19T21:05:51.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 >> v1:172.21.15.193:6812/157927432 conn(0x7fa240023080 legacy=0x7fa2400234e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:51.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 20 client.48989.objecter put_session s=0x7fa2400181b0 osd=7 1 2024-02-19T21:05:51.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 20 client.48989.objecter shutdown clearing up homeless session... 2024-02-19T21:05:51.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 10 client.48989.objecter successfully canceled tick 2024-02-19T21:05:51.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2537fe700 1 -- 172.21.15.163:0/1117161042 reap_dead start 2024-02-19T21:05:51.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 >> v1:172.21.15.163:6800/1552514681 conn(0x7fa23c063d10 legacy=0x7fa23c0661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:51.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2537fe700 1 -- 172.21.15.163:0/1117161042 reap_dead start 2024-02-19T21:05:51.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 >> v1:172.21.15.163:6790/0 conn(0x7fa2540fd980 legacy=0x7fa2540fc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:51.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2537fe700 1 -- 172.21.15.163:0/1117161042 reap_dead start 2024-02-19T21:05:51.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2537fe700 1 -- 172.21.15.163:0/1117161042 reap_dead start 2024-02-19T21:05:51.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2537fe700 1 -- 172.21.15.163:0/1117161042 reap_dead start 2024-02-19T21:05:51.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 shutdown_connections 2024-02-19T21:05:51.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.118+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 >> 172.21.15.163:0/1117161042 conn(0x7fa2540f35a0 msgr2=0x7fa254107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:51.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.119+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 shutdown_connections 2024-02-19T21:05:51.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.119+0000 7fa2594a3700 1 -- 172.21.15.163:0/1117161042 wait complete. 2024-02-19T21:05:51.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:51.119+0000 7fa2594a3700 1 librados: shutdown 2024-02-19T21:05:51.117 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:05:51.117 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:05:51.117 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:05:51.117 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:05:51.117 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:05:51.117 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:05:51.117 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:05:51.117 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:05:51.709 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:50.147591+0000 mgr.y (mgr.14550) 215 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:51.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:50.983507+0000 mon.c (mon.2) 160 : audit [DBG] from='client.? v1:172.21.15.163:0/1117161042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:51.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:51.188436+0000 mon.a (mon.0) 1268 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83920-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83920-24"}]': finished 2024-02-19T21:05:51.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:51.188699+0000 mon.a (mon.0) 1269 : audit [INF] from='client.47797 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi163-83845-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:51.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:51.188783+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-02-19T21:05:51.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:51.197651+0000 mon.c (mon.2) 161 : audit [INF] from='client.? v1:172.21.15.163:0/121887391' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi163-83854-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:51.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:51.198130+0000 mon.a (mon.0) 1271 : audit [INF] from='client.48983 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi163-83854-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:51.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:51.279262+0000 mon.c (mon.2) 162 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:51.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:51.547268+0000 osd.6 (osd.6) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:51.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:51.547825+0000 osd.6 (osd.6) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:51.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:50.147591+0000 mgr.y (mgr.14550) 215 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:51.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:50.983507+0000 mon.c (mon.2) 160 : audit [DBG] from='client.? v1:172.21.15.163:0/1117161042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:51.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:51.188436+0000 mon.a (mon.0) 1268 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83920-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83920-24"}]': finished 2024-02-19T21:05:51.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:51.188699+0000 mon.a (mon.0) 1269 : audit [INF] from='client.47797 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi163-83845-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:51.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:51.188783+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-02-19T21:05:51.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:51.197651+0000 mon.c (mon.2) 161 : audit [INF] from='client.? v1:172.21.15.163:0/121887391' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi163-83854-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:51.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:51.198130+0000 mon.a (mon.0) 1271 : audit [INF] from='client.48983 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi163-83854-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:51.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:51.279262+0000 mon.c (mon.2) 162 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:51.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:51.547268+0000 osd.6 (osd.6) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:51.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:51.547825+0000 osd.6 (osd.6) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:52.028 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:05:52.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:50.147591+0000 mgr.y (mgr.14550) 215 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:05:52.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:50.983507+0000 mon.c (mon.2) 160 : audit [DBG] from='client.? v1:172.21.15.163:0/1117161042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:52.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:51.188436+0000 mon.a (mon.0) 1268 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83920-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83920-24"}]': finished 2024-02-19T21:05:52.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:51.188699+0000 mon.a (mon.0) 1269 : audit [INF] from='client.47797 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi163-83845-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:52.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:51.188783+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-02-19T21:05:52.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:51.197651+0000 mon.c (mon.2) 161 : audit [INF] from='client.? v1:172.21.15.163:0/121887391' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi163-83854-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:52.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:51.198130+0000 mon.a (mon.0) 1271 : audit [INF] from='client.48983 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi163-83854-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:52.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:51.279262+0000 mon.c (mon.2) 162 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:52.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:51.547268+0000 osd.6 (osd.6) 83 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:52.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:51.547825+0000 osd.6 (osd.6) 84 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:52.322 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:52.833 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:51.131751+0000 mgr.y (mgr.14550) 216 : cluster [DBG] pgmap v227: 308 pgs: 2 peering, 32 unknown, 274 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:05:52.833 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:51.709543+0000 mon.a (mon.0) 1272 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:05:52.833 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:51.951018+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:52.833 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:51.951654+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:52.833 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:52.189046+0000 mon.a (mon.0) 1273 : audit [INF] from='client.48983 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi163-83854-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:52.833 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:52.189280+0000 mon.a (mon.0) 1274 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-02-19T21:05:52.833 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:52.218118+0000 mon.b (mon.1) 274 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:52.833 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:52.223565+0000 mon.a (mon.0) 1275 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:52.833 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:52.280042+0000 mon.c (mon.2) 163 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:52.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:51.131751+0000 mgr.y (mgr.14550) 216 : cluster [DBG] pgmap v227: 308 pgs: 2 peering, 32 unknown, 274 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:05:52.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:51.709543+0000 mon.a (mon.0) 1272 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:05:52.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:51.951018+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:52.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:51.951654+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:52.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:52.189046+0000 mon.a (mon.0) 1273 : audit [INF] from='client.48983 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi163-83854-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:52.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:52.189280+0000 mon.a (mon.0) 1274 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-02-19T21:05:52.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:52.218118+0000 mon.b (mon.1) 274 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:52.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:52.223565+0000 mon.a (mon.0) 1275 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:52.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:52.280042+0000 mon.c (mon.2) 163 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:53.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:51.131751+0000 mgr.y (mgr.14550) 216 : cluster [DBG] pgmap v227: 308 pgs: 2 peering, 32 unknown, 274 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:05:53.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:51.709543+0000 mon.a (mon.0) 1272 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:05:53.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:51.951018+0000 osd.7 (osd.7) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:53.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:51.951654+0000 osd.7 (osd.7) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:53.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:52.189046+0000 mon.a (mon.0) 1273 : audit [INF] from='client.48983 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi163-83854-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:53.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:52.189280+0000 mon.a (mon.0) 1274 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-02-19T21:05:53.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:52.218118+0000 mon.b (mon.1) 274 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:53.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:52.223565+0000 mon.a (mon.0) 1275 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:53.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:52.280042+0000 mon.c (mon.2) 163 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:53.263 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:05:53.263 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:05:53.264 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-02-19T21:05:53.264 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:05:53.544 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:05:53] "GET /metrics HTTP/1.1" 200 194990 "" "Prometheus/2.33.4" 2024-02-19T21:05:53.781 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:05:53.801 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:53.132631+0000 mon.b (mon.1) 275 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-19T21:05:53.801 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:53.134144+0000 mon.a (mon.0) 1276 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-19T21:05:53.801 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:53.189681+0000 mon.a (mon.0) 1277 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:53.801 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:53.189838+0000 mon.a (mon.0) 1278 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-19T21:05:53.801 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:53.189899+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-02-19T21:05:53.801 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:53.199867+0000 mon.a (mon.0) 1280 : audit [INF] from='client.? v1:172.21.15.163:0/654122024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi163-83845-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:53.801 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:53.202106+0000 mon.b (mon.1) 276 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:53.801 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:53.204762+0000 mon.a (mon.0) 1281 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:53.801 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:53.280748+0000 mon.c (mon.2) 164 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:53.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:53.132631+0000 mon.b (mon.1) 275 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-19T21:05:53.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:53.134144+0000 mon.a (mon.0) 1276 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-19T21:05:53.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:53.189681+0000 mon.a (mon.0) 1277 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:53.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:53.189838+0000 mon.a (mon.0) 1278 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-19T21:05:53.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:53.189899+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-02-19T21:05:53.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:53.199867+0000 mon.a (mon.0) 1280 : audit [INF] from='client.? v1:172.21.15.163:0/654122024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi163-83845-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:53.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:53.202106+0000 mon.b (mon.1) 276 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:53.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:53.204762+0000 mon.a (mon.0) 1281 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:53.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:53.280748+0000 mon.c (mon.2) 164 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:53.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:53.132631+0000 mon.b (mon.1) 275 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-19T21:05:53.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:53.134144+0000 mon.a (mon.0) 1276 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-02-19T21:05:53.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:53.189681+0000 mon.a (mon.0) 1277 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:53.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:53.189838+0000 mon.a (mon.0) 1278 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-02-19T21:05:53.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:53.189899+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-02-19T21:05:53.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:53.199867+0000 mon.a (mon.0) 1280 : audit [INF] from='client.? v1:172.21.15.163:0/654122024' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi163-83845-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:53.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:53.202106+0000 mon.b (mon.1) 276 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:53.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:53.204762+0000 mon.a (mon.0) 1281 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:53.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:53.280748+0000 mon.c (mon.2) 164 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:53.967 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:53.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:53.991 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-19T21:05:54.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.067+0000 7f27f08f4700 1 -- 172.21.15.163:0/1300586069 >> v1:172.21.15.163:6790/0 conn(0x7f27ec0fd9b0 legacy=0x7f27ec0fdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:54.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.067+0000 7f27f08f4700 1 -- 172.21.15.163:0/1300586069 shutdown_connections 2024-02-19T21:05:54.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.067+0000 7f27f08f4700 1 -- 172.21.15.163:0/1300586069 >> 172.21.15.163:0/1300586069 conn(0x7f27ec06fdc0 msgr2=0x7f27ec0f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:54.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.067+0000 7f27f08f4700 1 -- 172.21.15.163:0/1300586069 shutdown_connections 2024-02-19T21:05:54.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.067+0000 7f27f08f4700 1 -- 172.21.15.163:0/1300586069 wait complete. 2024-02-19T21:05:54.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.067+0000 7f27f08f4700 1 librados: starting msgr at 2024-02-19T21:05:54.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.067+0000 7f27f08f4700 1 librados: starting objecter 2024-02-19T21:05:54.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27f08f4700 1 Processor -- start 2024-02-19T21:05:54.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27f08f4700 1 -- start start 2024-02-19T21:05:54.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27f08f4700 1 librados: setting wanted keys 2024-02-19T21:05:54.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27f08f4700 1 librados: calling monclient init 2024-02-19T21:05:54.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27f08f4700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f27ec0fcf90 con 0x7f27ec0fd9b0 2024-02-19T21:05:54.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27f08f4700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f27ec0fd120 con 0x7f27ec1007a0 2024-02-19T21:05:54.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27f08f4700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f27ec0fd310 con 0x7f27ec104310 2024-02-19T21:05:54.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27ead9d700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f27ec104310 0x7f27ec1184b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38908/0 (socket says 172.21.15.163:38908) 2024-02-19T21:05:54.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27ead9d700 1 -- 172.21.15.163:0/3586749806 learned_addr learned my addr 172.21.15.163:0/3586749806 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:54.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27db7fe700 10 client.?.objecter ms_handle_connect 0x7f27ec104310 2024-02-19T21:05:54.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27db7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:54.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27db7fe700 10 client.?.objecter ms_handle_connect 0x7f27ec0fd9b0 2024-02-19T21:05:54.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27db7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27db7fe700 10 client.?.objecter ms_handle_connect 0x7f27ec1007a0 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27db7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262400213 0 0) 0x7f27ec0fd310 con 0x7f27ec104310 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27d40033f0 con 0x7f27ec104310 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132051980 0 0) 0x7f27ec0fcf90 con 0x7f27ec0fd9b0 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27ec0fd310 con 0x7f27ec0fd9b0 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280680182 0 0) 0x7f27ec0fd120 con 0x7f27ec1007a0 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.068+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27ec0fcf90 con 0x7f27ec1007a0 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3436808664 0 0) 0x7f27d40033f0 con 0x7f27ec104310 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27ec0fd120 con 0x7f27ec104310 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f27e4001d50 con 0x7f27ec104310 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3948650141 0 0) 0x7f27ec0fd310 con 0x7f27ec0fd9b0 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27d40033f0 con 0x7f27ec0fd9b0 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f27dc002050 con 0x7f27ec0fd9b0 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2361687089 0 0) 0x7f27ec0fcf90 con 0x7f27ec1007a0 2024-02-19T21:05:54.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27ec0fd310 con 0x7f27ec1007a0 2024-02-19T21:05:54.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f27e0001840 con 0x7f27ec1007a0 2024-02-19T21:05:54.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 515732899 0 0) 0x7f27ec0fd120 con 0x7f27ec104310 2024-02-19T21:05:54.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 >> v1:172.21.15.193:6789/0 conn(0x7f27ec1007a0 legacy=0x7f27ec114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:54.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 >> v1:172.21.15.163:6789/0 conn(0x7f27ec0fd9b0 legacy=0x7f27ec0fc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:54.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec11cc00 con 0x7f27ec104310 2024-02-19T21:05:54.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27f08f4700 1 -- 172.21.15.163:0/3586749806 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ec119c30 con 0x7f27ec104310 2024-02-19T21:05:54.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27f08f4700 10 client.49007.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:54.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27f08f4700 1 -- 172.21.15.163:0/3586749806 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ec11a0a0 con 0x7f27ec104310 2024-02-19T21:05:54.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27f08f4700 1 librados: init done 2024-02-19T21:05:54.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f27e4002850 con 0x7f27ec104310 2024-02-19T21:05:54.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.069+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f27e4005ed0 con 0x7f27ec104310 2024-02-19T21:05:54.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.070+0000 7f27f08f4700 1 -- 172.21.15.163:0/3586749806 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27cc004f20 con 0x7f27ec104310 2024-02-19T21:05:54.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.070+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f27e4018d60 con 0x7f27ec104310 2024-02-19T21:05:54.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.071+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(173..173 src has 43..173) v4 ==== 10604+0+0 (unknown 1259554991 0 0) 0x7f27e4079a00 con 0x7f27ec104310 2024-02-19T21:05:54.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.071+0000 7f27db7fe700 10 client.49007.objecter ms_dispatch 0x7f27ec001550 osd_map(173..173 src has 43..173) v4 2024-02-19T21:05:54.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.071+0000 7f27db7fe700 3 client.49007.objecter handle_osd_map got epochs [173,173] > 0 2024-02-19T21:05:54.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.071+0000 7f27db7fe700 3 client.49007.objecter handle_osd_map decoding full epoch 173 2024-02-19T21:05:54.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.071+0000 7f27db7fe700 20 client.49007.objecter dump_active .. 0 homeless 2024-02-19T21:05:54.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.071+0000 7f27db7fe700 10 client.49007.objecter ms_handle_connect 0x7f27d4063c80 2024-02-19T21:05:54.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.075+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f27e404c570 con 0x7f27ec104310 2024-02-19T21:05:54.072 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:54.103 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:54.147 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:54.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:54.296+0000 7f27f08f4700 1 -- 172.21.15.163:0/3586749806 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f27cc0051d0 con 0x7f27ec104310 2024-02-19T21:05:54.555 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:54.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:54.659 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:05:54.689 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:54.784 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:53.132220+0000 mgr.y (mgr.14550) 217 : cluster [DBG] pgmap v230: 347 pgs: 1 peering, 72 unknown, 274 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-02-19T21:05:54.784 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:54.174755+0000 mon.a (mon.0) 1282 : audit [INF] from='client.? v1:172.21.15.163:0/654122024' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi163-83845-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:54.784 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:54.174930+0000 mon.a (mon.0) 1283 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:54.784 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:54.175072+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-02-19T21:05:54.784 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:54.193980+0000 mon.a (mon.0) 1285 : audit [INF] from='client.? v1:172.21.15.163:0/129334150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi163-83854-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:54.785 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:54.281436+0000 mon.c (mon.2) 165 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:54.785 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:54.298159+0000 mon.c (mon.2) 166 : audit [INF] from='client.? v1:172.21.15.163:0/3586749806' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:54.785 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:54.298528+0000 mon.a (mon.0) 1286 : audit [INF] from='client.49007 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:54.887 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:05:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:54.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:05:54.991 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:54.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:53.132220+0000 mgr.y (mgr.14550) 217 : cluster [DBG] pgmap v230: 347 pgs: 1 peering, 72 unknown, 274 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-02-19T21:05:54.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:54.174755+0000 mon.a (mon.0) 1282 : audit [INF] from='client.? v1:172.21.15.163:0/654122024' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi163-83845-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:54.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:54.174930+0000 mon.a (mon.0) 1283 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:54.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:54.175072+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-02-19T21:05:54.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:54.193980+0000 mon.a (mon.0) 1285 : audit [INF] from='client.? v1:172.21.15.163:0/129334150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi163-83854-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:54.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:54.281436+0000 mon.c (mon.2) 165 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:54.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:54.298159+0000 mon.c (mon.2) 166 : audit [INF] from='client.? v1:172.21.15.163:0/3586749806' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:54.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:54.298528+0000 mon.a (mon.0) 1286 : audit [INF] from='client.49007 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:54.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:53.132220+0000 mgr.y (mgr.14550) 217 : cluster [DBG] pgmap v230: 347 pgs: 1 peering, 72 unknown, 274 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-02-19T21:05:54.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:54.174755+0000 mon.a (mon.0) 1282 : audit [INF] from='client.? v1:172.21.15.163:0/654122024' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi163-83845-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:54.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:54.174930+0000 mon.a (mon.0) 1283 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83920-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:54.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:54.175072+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-02-19T21:05:54.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:54.193980+0000 mon.a (mon.0) 1285 : audit [INF] from='client.? v1:172.21.15.163:0/129334150' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi163-83854-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:54.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:54.281436+0000 mon.c (mon.2) 165 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:54.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:54.298159+0000 mon.c (mon.2) 166 : audit [INF] from='client.? v1:172.21.15.163:0/3586749806' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:54.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:54.298528+0000 mon.a (mon.0) 1286 : audit [INF] from='client.49007 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:05:55.023 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:55.044 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:55.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:55.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:55.064 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:05:55.065 INFO:teuthology.orchestra.run.smithi193.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-19T21:05:55.065 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:05:55.102 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:05:55.148 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:05:55.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:55.182+0000 7f27db7fe700 1 -- 172.21.15.163:0/3586749806 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v175) v1 ==== 97+0+0 (unknown 2210614083 0 0) 0x7f27e404a980 con 0x7f27ec104310 2024-02-19T21:05:55.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:55.186+0000 7f27f08f4700 10 librados: watch_flush enter 2024-02-19T21:05:55.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:55.186+0000 7f27f08f4700 10 librados: watch_flush exit 2024-02-19T21:05:55.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:55.186+0000 7f27f08f4700 20 client.49007.objecter shutdown clearing up homeless session... 2024-02-19T21:05:55.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:55.186+0000 7f27f08f4700 10 client.49007.objecter successfully canceled tick 2024-02-19T21:05:55.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:55.186+0000 7f27f08f4700 1 -- 172.21.15.163:0/3586749806 >> v1:172.21.15.163:6800/1552514681 conn(0x7f27d4063c80 legacy=0x7f27d4066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:55.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:55.186+0000 7f27f08f4700 1 -- 172.21.15.163:0/3586749806 >> v1:172.21.15.163:6790/0 conn(0x7f27ec104310 legacy=0x7f27ec1184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:55.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:55.190+0000 7f27f08f4700 1 -- 172.21.15.163:0/3586749806 shutdown_connections 2024-02-19T21:05:55.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:55.190+0000 7f27f08f4700 1 -- 172.21.15.163:0/3586749806 >> 172.21.15.163:0/3586749806 conn(0x7f27ec06fdc0 msgr2=0x7f27ec0f4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:55.186 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:55.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:55.200+0000 7f27f08f4700 1 -- 172.21.15.163:0/3586749806 shutdown_connections 2024-02-19T21:05:55.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:55.204+0000 7f27f08f4700 1 -- 172.21.15.163:0/3586749806 wait complete. 2024-02-19T21:05:55.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:55.205+0000 7f27f08f4700 1 librados: shutdown 2024-02-19T21:05:55.218 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:05:55.325 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:05:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:55.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:05:55.430 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:55.457 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:55.906 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:55.908 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:55.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:55.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:55.169580+0000 mon.a (mon.0) 1287 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:05:55.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:55.181404+0000 mon.a (mon.0) 1288 : audit [INF] from='client.? v1:172.21.15.163:0/129334150' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi163-83854-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:55.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:55.181606+0000 mon.a (mon.0) 1289 : audit [INF] from='client.49007 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:05:55.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:55.181667+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-02-19T21:05:55.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:55.282245+0000 mon.c (mon.2) 167 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:55.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:55.169580+0000 mon.a (mon.0) 1287 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:05:55.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:55.181404+0000 mon.a (mon.0) 1288 : audit [INF] from='client.? v1:172.21.15.163:0/129334150' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi163-83854-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:55.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:55.181606+0000 mon.a (mon.0) 1289 : audit [INF] from='client.49007 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:05:55.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:55.181667+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-02-19T21:05:55.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:55.282245+0000 mon.c (mon.2) 167 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:56.009 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:05:56.040 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:55.169580+0000 mon.a (mon.0) 1287 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:05:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:55.181404+0000 mon.a (mon.0) 1288 : audit [INF] from='client.? v1:172.21.15.163:0/129334150' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi163-83854-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:55.181606+0000 mon.a (mon.0) 1289 : audit [INF] from='client.49007 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:05:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:55.181667+0000 mon.a (mon.0) 1290 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-02-19T21:05:56.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:55.282245+0000 mon.c (mon.2) 167 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:56.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.035+0000 7f04921ba700 1 -- 172.21.15.163:0/3956800327 >> v1:172.21.15.163:6790/0 conn(0x7f048c0f9530 legacy=0x7f048c102a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:56.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.035+0000 7f04921ba700 1 -- 172.21.15.163:0/3956800327 shutdown_connections 2024-02-19T21:05:56.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.035+0000 7f04921ba700 1 -- 172.21.15.163:0/3956800327 >> 172.21.15.163:0/3956800327 conn(0x7f048c0f2470 msgr2=0x7f048c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:56.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.035+0000 7f04921ba700 1 -- 172.21.15.163:0/3956800327 shutdown_connections 2024-02-19T21:05:56.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.035+0000 7f04921ba700 1 -- 172.21.15.163:0/3956800327 wait complete. 2024-02-19T21:05:56.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f04921ba700 1 librados: starting msgr at 2024-02-19T21:05:56.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f04921ba700 1 librados: starting objecter 2024-02-19T21:05:56.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f04921ba700 1 Processor -- start 2024-02-19T21:05:56.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f04921ba700 1 -- start start 2024-02-19T21:05:56.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f04921ba700 1 librados: setting wanted keys 2024-02-19T21:05:56.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f04921ba700 1 librados: calling monclient init 2024-02-19T21:05:56.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f04921ba700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f048c117150 con 0x7f048c1042b0 2024-02-19T21:05:56.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f04921ba700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f048c118330 con 0x7f048c0f8980 2024-02-19T21:05:56.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f04921ba700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f048c119510 con 0x7f048c0f9530 2024-02-19T21:05:56.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f0483fff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f048c0f9530 0x7f048c0ffa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:49332/0 (socket says 172.21.15.163:49332) 2024-02-19T21:05:56.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f0483fff700 1 -- 172.21.15.163:0/3457809448 learned_addr learned my addr 172.21.15.163:0/3457809448 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:05:56.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f048bfff700 1 --1- 172.21.15.163:0/3457809448 >> v1:172.21.15.163:6789/0 conn(0x7f048c1042b0 0x7f048c115930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46576/0 (socket says 172.21.15.163:46576) 2024-02-19T21:05:56.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f04897fa700 10 client.?.objecter ms_handle_connect 0x7f048c1042b0 2024-02-19T21:05:56.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f04897fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:56.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f04897fa700 10 client.?.objecter ms_handle_connect 0x7f048c0f9530 2024-02-19T21:05:56.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.036+0000 7f04897fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:56.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 10 client.?.objecter ms_handle_connect 0x7f048c0f8980 2024-02-19T21:05:56.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:05:56.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210903463 0 0) 0x7f048c119510 con 0x7f048c0f9530 2024-02-19T21:05:56.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0478003030 con 0x7f048c0f9530 2024-02-19T21:05:56.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962958554 0 0) 0x7f048c117150 con 0x7f048c1042b0 2024-02-19T21:05:56.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048c119510 con 0x7f048c1042b0 2024-02-19T21:05:56.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524993684 0 0) 0x7f048c118330 con 0x7f048c0f8980 2024-02-19T21:05:56.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048c117150 con 0x7f048c0f8980 2024-02-19T21:05:56.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2384076653 0 0) 0x7f0478003030 con 0x7f048c0f9530 2024-02-19T21:05:56.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f048c118330 con 0x7f048c0f9530 2024-02-19T21:05:56.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0474002e10 con 0x7f048c0f9530 2024-02-19T21:05:56.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4065733587 0 0) 0x7f048c117150 con 0x7f048c0f8980 2024-02-19T21:05:56.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0478003030 con 0x7f048c0f8980 2024-02-19T21:05:56.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f047c0019f0 con 0x7f048c0f8980 2024-02-19T21:05:56.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 426889425 0 0) 0x7f048c119510 con 0x7f048c1042b0 2024-02-19T21:05:56.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f048c117150 con 0x7f048c1042b0 2024-02-19T21:05:56.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0484001bc0 con 0x7f048c1042b0 2024-02-19T21:05:56.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1516698257 0 0) 0x7f048c118330 con 0x7f048c0f9530 2024-02-19T21:05:56.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 >> v1:172.21.15.193:6789/0 conn(0x7f048c0f8980 legacy=0x7f048c0ff340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:56.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 >> v1:172.21.15.163:6789/0 conn(0x7f048c1042b0 legacy=0x7f048c115930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:56.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c11a710 con 0x7f048c0f9530 2024-02-19T21:05:56.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048c119700 con 0x7f048c0f9530 2024-02-19T21:05:56.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04921ba700 10 client.49019.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:05:56.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f048c119b70 con 0x7f048c0f9530 2024-02-19T21:05:56.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04921ba700 1 librados: init done 2024-02-19T21:05:56.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f0474000fe0 con 0x7f048c0f9530 2024-02-19T21:05:56.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.037+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0474003750 con 0x7f048c0f9530 2024-02-19T21:05:56.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.038+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f046c004f00 con 0x7f048c0f9530 2024-02-19T21:05:56.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f0474003950 con 0x7f048c0f9530 2024-02-19T21:05:56.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(175..175 src has 43..175) v4 ==== 10265+0+0 (unknown 1364507462 0 0) 0x7f0474079660 con 0x7f048c0f9530 2024-02-19T21:05:56.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 osd_map(175..175 src has 43..175) v4 2024-02-19T21:05:56.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04897fa700 3 client.49019.objecter handle_osd_map got epochs [175,175] > 0 2024-02-19T21:05:56.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04897fa700 3 client.49019.objecter handle_osd_map decoding full epoch 175 2024-02-19T21:05:56.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04897fa700 20 client.49019.objecter dump_active .. 0 homeless 2024-02-19T21:05:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v175) v1 ==== 54+0+16 (unknown 650950317 0 2420000192) 0x7f047407ba70 con 0x7f048c0f9530 2024-02-19T21:05:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04897fa700 10 client.49019.objecter ms_handle_connect 0x7f0478063cc0 2024-02-19T21:05:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04921ba700 10 client.49019.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 1 2024-02-19T21:05:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c0051a0 osd=0 3 2024-02-19T21:05:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c0051a0 osd=0 3 2024-02-19T21:05:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c0051a0 osd=0 3 2024-02-19T21:05:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 1 2024-02-19T21:05:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c0051a0 osd=0 3 2024-02-19T21:05:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 0 1 2024-02-19T21:05:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c0051a0 osd=0 4 2024-02-19T21:05:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04921ba700 10 client.49019.objecter _send_command 1 2024-02-19T21:05:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.039+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f046c004950 con 0x7f046c004530 2024-02-19T21:05:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.040+0000 7f04897fa700 10 client.49019.objecter ms_handle_connect 0x7f046c004530 2024-02-19T21:05:56.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.040+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f046c004950 con 0x7f046c004530 2024-02-19T21:05:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.040+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:05:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.040+0000 7f04897fa700 10 client.49019.objecter _finish_command 1 = system:0 2024-02-19T21:05:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.040+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c0051a0 osd=0 4 2024-02-19T21:05:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.040+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 0 1 2024-02-19T21:05:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.056+0000 7f04921ba700 10 client.49019.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.056+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 2 2024-02-19T21:05:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.056+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c0051a0 osd=0 3 2024-02-19T21:05:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.056+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c0051a0 osd=0 3 2024-02-19T21:05:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.056+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c0051a0 osd=0 3 2024-02-19T21:05:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.056+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 2 2024-02-19T21:05:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.056+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c0051a0 osd=0 3 2024-02-19T21:05:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.056+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 0 2 2024-02-19T21:05:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.056+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c0051a0 osd=0 4 2024-02-19T21:05:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.056+0000 7f04921ba700 10 client.49019.objecter _send_command 2 2024-02-19T21:05:56.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.056+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f046c004d30 con 0x7f046c004530 2024-02-19T21:05:56.367 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:05:56.367 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:05:56.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f046c004d30 con 0x7f046c004530 2024-02-19T21:05:56.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:56.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04897fa700 10 client.49019.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c0051a0 osd=0 4 2024-02-19T21:05:56.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 0 2 2024-02-19T21:05:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04921ba700 10 client.49019.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 3 2024-02-19T21:05:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c003fe0 osd=1 3 2024-02-19T21:05:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c003fe0 osd=1 3 2024-02-19T21:05:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c003fe0 osd=1 3 2024-02-19T21:05:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 3 2024-02-19T21:05:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c003fe0 osd=1 3 2024-02-19T21:05:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 1 3 2024-02-19T21:05:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c003fe0 osd=1 4 2024-02-19T21:05:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04921ba700 10 client.49019.objecter _send_command 3 2024-02-19T21:05:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.058+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f046c00c000 con 0x7f046c004150 2024-02-19T21:05:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.059+0000 7f04897fa700 10 client.49019.objecter ms_handle_connect 0x7f046c004150 2024-02-19T21:05:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.059+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f046c00c000 con 0x7f046c004150 2024-02-19T21:05:56.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.059+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.059+0000 7f04897fa700 10 client.49019.objecter _finish_command 3 = system:0 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.059+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c003fe0 osd=1 4 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.059+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 1 3 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.074+0000 7f04921ba700 10 client.49019.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.074+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 4 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.074+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c003fe0 osd=1 3 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.074+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c003fe0 osd=1 3 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.074+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c003fe0 osd=1 3 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.074+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 4 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.074+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c003fe0 osd=1 3 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.074+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 1 4 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.074+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c003fe0 osd=1 4 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.074+0000 7f04921ba700 10 client.49019.objecter _send_command 4 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.074+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f046c0018b0 con 0x7f046c004150 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f046c0018b0 con 0x7f046c004150 2024-02-19T21:05:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04897fa700 10 client.49019.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c003fe0 osd=1 4 2024-02-19T21:05:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 1 4 2024-02-19T21:05:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04921ba700 10 client.49019.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 5 2024-02-19T21:05:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c003630 osd=2 3 2024-02-19T21:05:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c003630 osd=2 3 2024-02-19T21:05:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c003630 osd=2 3 2024-02-19T21:05:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 5 2024-02-19T21:05:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c003630 osd=2 3 2024-02-19T21:05:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 2 5 2024-02-19T21:05:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c003630 osd=2 4 2024-02-19T21:05:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04921ba700 10 client.49019.objecter _send_command 5 2024-02-19T21:05:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f046c00ccd0 con 0x7f046c00c170 2024-02-19T21:05:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.076+0000 7f04897fa700 10 client.49019.objecter ms_handle_connect 0x7f046c00c170 2024-02-19T21:05:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.077+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f046c00ccd0 con 0x7f046c00c170 2024-02-19T21:05:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.077+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:05:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.077+0000 7f04897fa700 10 client.49019.objecter _finish_command 5 = system:0 2024-02-19T21:05:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.077+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c003630 osd=2 4 2024-02-19T21:05:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.077+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 2 5 2024-02-19T21:05:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.091+0000 7f04921ba700 10 client.49019.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.091+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 6 2024-02-19T21:05:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.091+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c003630 osd=2 3 2024-02-19T21:05:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.091+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c003630 osd=2 3 2024-02-19T21:05:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.091+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c003630 osd=2 3 2024-02-19T21:05:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.091+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 6 2024-02-19T21:05:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.091+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c003630 osd=2 3 2024-02-19T21:05:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.091+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 2 6 2024-02-19T21:05:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.091+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c003630 osd=2 4 2024-02-19T21:05:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.091+0000 7f04921ba700 10 client.49019.objecter _send_command 6 2024-02-19T21:05:56.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.091+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f046c00e790 con 0x7f046c00c170 2024-02-19T21:05:56.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f046c00e790 con 0x7f046c00c170 2024-02-19T21:05:56.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:56.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04897fa700 10 client.49019.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c003630 osd=2 4 2024-02-19T21:05:56.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 2 6 2024-02-19T21:05:56.372 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:05:56.372 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:05:56.372 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:05:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04921ba700 10 client.49019.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 7 2024-02-19T21:05:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c003380 osd=3 3 2024-02-19T21:05:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c003380 osd=3 3 2024-02-19T21:05:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c003380 osd=3 3 2024-02-19T21:05:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 7 2024-02-19T21:05:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c003380 osd=3 3 2024-02-19T21:05:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 3 7 2024-02-19T21:05:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c003380 osd=3 4 2024-02-19T21:05:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04921ba700 10 client.49019.objecter _send_command 7 2024-02-19T21:05:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f046c019e90 con 0x7f046c016290 2024-02-19T21:05:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.093+0000 7f04897fa700 10 client.49019.objecter ms_handle_connect 0x7f046c016290 2024-02-19T21:05:56.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.094+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f046c019e90 con 0x7f046c016290 2024-02-19T21:05:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.094+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:05:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.094+0000 7f04897fa700 10 client.49019.objecter _finish_command 7 = system:0 2024-02-19T21:05:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.094+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c003380 osd=3 4 2024-02-19T21:05:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.094+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 3 7 2024-02-19T21:05:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.108+0000 7f04921ba700 10 client.49019.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.108+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 8 2024-02-19T21:05:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.108+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c003380 osd=3 3 2024-02-19T21:05:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.108+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c003380 osd=3 3 2024-02-19T21:05:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.108+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c003380 osd=3 3 2024-02-19T21:05:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.108+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 8 2024-02-19T21:05:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.108+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c003380 osd=3 3 2024-02-19T21:05:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.108+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 3 8 2024-02-19T21:05:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.108+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c003380 osd=3 4 2024-02-19T21:05:56.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.108+0000 7f04921ba700 10 client.49019.objecter _send_command 8 2024-02-19T21:05:56.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.108+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f046c01b1b0 con 0x7f046c016290 2024-02-19T21:05:56.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.109+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f046c01b1b0 con 0x7f046c016290 2024-02-19T21:05:56.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:56.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04897fa700 10 client.49019.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c003380 osd=3 4 2024-02-19T21:05:56.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 3 8 2024-02-19T21:05:56.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04921ba700 10 client.49019.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:56.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 9 2024-02-19T21:05:56.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c01b1b0 osd=4 3 2024-02-19T21:05:56.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c01b1b0 osd=4 3 2024-02-19T21:05:56.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c01b1b0 osd=4 3 2024-02-19T21:05:56.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 9 2024-02-19T21:05:56.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c01b1b0 osd=4 3 2024-02-19T21:05:56.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 4 9 2024-02-19T21:05:56.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c01b1b0 osd=4 4 2024-02-19T21:05:56.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04921ba700 10 client.49019.objecter _send_command 9 2024-02-19T21:05:56.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f046c01e840 con 0x7f046c01c360 2024-02-19T21:05:56.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.110+0000 7f04897fa700 10 client.49019.objecter ms_handle_connect 0x7f046c01c360 2024-02-19T21:05:56.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.111+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f046c01e840 con 0x7f046c01c360 2024-02-19T21:05:56.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.111+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:05:56.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.111+0000 7f04897fa700 10 client.49019.objecter _finish_command 9 = system:0 2024-02-19T21:05:56.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.111+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c01b1b0 osd=4 4 2024-02-19T21:05:56.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.111+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 4 9 2024-02-19T21:05:56.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.125+0000 7f04921ba700 10 client.49019.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:56.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.125+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 10 2024-02-19T21:05:56.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.125+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c01b1b0 osd=4 3 2024-02-19T21:05:56.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.125+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c01b1b0 osd=4 3 2024-02-19T21:05:56.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.125+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c01b1b0 osd=4 3 2024-02-19T21:05:56.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.125+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 10 2024-02-19T21:05:56.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.125+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c01b1b0 osd=4 3 2024-02-19T21:05:56.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.125+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 4 10 2024-02-19T21:05:56.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.125+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c01b1b0 osd=4 4 2024-02-19T21:05:56.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.125+0000 7f04921ba700 10 client.49019.objecter _send_command 10 2024-02-19T21:05:56.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.125+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f046c01e840 con 0x7f046c01c360 2024-02-19T21:05:56.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.127+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f046c01e840 con 0x7f046c01c360 2024-02-19T21:05:56.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.127+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:56.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.127+0000 7f04897fa700 10 client.49019.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.127+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c01b1b0 osd=4 4 2024-02-19T21:05:56.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.127+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 4 10 2024-02-19T21:05:56.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.128+0000 7f04921ba700 10 client.49019.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:56.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.128+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 11 2024-02-19T21:05:56.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.128+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c01e840 osd=5 3 2024-02-19T21:05:56.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.128+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c01e840 osd=5 3 2024-02-19T21:05:56.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.128+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c01e840 osd=5 3 2024-02-19T21:05:56.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.128+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 11 2024-02-19T21:05:56.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.128+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c01e840 osd=5 3 2024-02-19T21:05:56.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.128+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 5 11 2024-02-19T21:05:56.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.128+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c01e840 osd=5 4 2024-02-19T21:05:56.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.128+0000 7f04921ba700 10 client.49019.objecter _send_command 11 2024-02-19T21:05:56.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.128+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f046c020bf0 con 0x7f046c01eef0 2024-02-19T21:05:56.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.128+0000 7f04897fa700 10 client.49019.objecter ms_handle_connect 0x7f046c01eef0 2024-02-19T21:05:56.378 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:05:56.378 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:05:56.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.129+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f046c020bf0 con 0x7f046c01eef0 2024-02-19T21:05:56.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.129+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:05:56.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.129+0000 7f04897fa700 10 client.49019.objecter _finish_command 11 = system:0 2024-02-19T21:05:56.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.129+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c01e840 osd=5 4 2024-02-19T21:05:56.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.129+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 5 11 2024-02-19T21:05:56.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.143+0000 7f04921ba700 10 client.49019.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:56.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.143+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 12 2024-02-19T21:05:56.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.143+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c01e840 osd=5 3 2024-02-19T21:05:56.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.143+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c01e840 osd=5 3 2024-02-19T21:05:56.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.143+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c01e840 osd=5 3 2024-02-19T21:05:56.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.143+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 12 2024-02-19T21:05:56.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.143+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c01e840 osd=5 3 2024-02-19T21:05:56.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.143+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 5 12 2024-02-19T21:05:56.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.143+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c01e840 osd=5 4 2024-02-19T21:05:56.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.143+0000 7f04921ba700 10 client.49019.objecter _send_command 12 2024-02-19T21:05:56.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.143+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f046c020bf0 con 0x7f046c01eef0 2024-02-19T21:05:56.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f046c020bf0 con 0x7f046c01eef0 2024-02-19T21:05:56.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:56.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04897fa700 10 client.49019.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c01e840 osd=5 4 2024-02-19T21:05:56.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 5 12 2024-02-19T21:05:56.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04921ba700 10 client.49019.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:56.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 13 2024-02-19T21:05:56.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c020bf0 osd=6 3 2024-02-19T21:05:56.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c020bf0 osd=6 3 2024-02-19T21:05:56.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c020bf0 osd=6 3 2024-02-19T21:05:56.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 13 2024-02-19T21:05:56.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c020bf0 osd=6 3 2024-02-19T21:05:56.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 6 13 2024-02-19T21:05:56.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c020bf0 osd=6 4 2024-02-19T21:05:56.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04921ba700 10 client.49019.objecter _send_command 13 2024-02-19T21:05:56.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.145+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f046c027190 con 0x7f046c023680 2024-02-19T21:05:56.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.146+0000 7f04897fa700 10 client.49019.objecter ms_handle_connect 0x7f046c023680 2024-02-19T21:05:56.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.146+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f046c027190 con 0x7f046c023680 2024-02-19T21:05:56.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.146+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:05:56.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.146+0000 7f04897fa700 10 client.49019.objecter _finish_command 13 = system:0 2024-02-19T21:05:56.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.146+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c020bf0 osd=6 4 2024-02-19T21:05:56.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.146+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 6 13 2024-02-19T21:05:56.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.160+0000 7f04921ba700 10 client.49019.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:56.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.160+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 14 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.160+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c020bf0 osd=6 3 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.160+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c020bf0 osd=6 3 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.160+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c020bf0 osd=6 3 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.160+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 14 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.160+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c020bf0 osd=6 3 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.160+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 6 14 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.160+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c020bf0 osd=6 4 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.160+0000 7f04921ba700 10 client.49019.objecter _send_command 14 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.160+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f046c027190 con 0x7f046c023680 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.162+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f046c027190 con 0x7f046c023680 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.162+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.162+0000 7f04897fa700 10 client.49019.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.162+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c020bf0 osd=6 4 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.162+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 6 14 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.163+0000 7f04921ba700 10 client.49019.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:05:56.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.163+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 15 2024-02-19T21:05:56.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.163+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c027190 osd=7 3 2024-02-19T21:05:56.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.163+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c027190 osd=7 3 2024-02-19T21:05:56.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.163+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c027190 osd=7 3 2024-02-19T21:05:56.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.163+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 15 2024-02-19T21:05:56.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.163+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c027190 osd=7 3 2024-02-19T21:05:56.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.163+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 7 15 2024-02-19T21:05:56.383 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:05:56.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.163+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c027190 osd=7 4 2024-02-19T21:05:56.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.163+0000 7f04921ba700 10 client.49019.objecter _send_command 15 2024-02-19T21:05:56.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.163+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f046c02b4e0 con 0x7f046c027840 2024-02-19T21:05:56.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.163+0000 7f04897fa700 10 client.49019.objecter ms_handle_connect 0x7f046c027840 2024-02-19T21:05:56.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.164+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f046c02b4e0 con 0x7f046c027840 2024-02-19T21:05:56.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.164+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:05:56.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.164+0000 7f04897fa700 10 client.49019.objecter _finish_command 15 = system:0 2024-02-19T21:05:56.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.164+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c027190 osd=7 4 2024-02-19T21:05:56.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.164+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 7 15 2024-02-19T21:05:56.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.184+0000 7f04921ba700 10 client.49019.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:05:56.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.184+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign -1 16 2024-02-19T21:05:56.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.184+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c027190 osd=7 3 2024-02-19T21:05:56.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.184+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c027190 osd=7 3 2024-02-19T21:05:56.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.184+0000 7f04921ba700 20 client.49019.objecter _get_session s=0x7f046c027190 osd=7 3 2024-02-19T21:05:56.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.184+0000 7f04921ba700 15 client.49019.objecter _session_command_op_remove -1 16 2024-02-19T21:05:56.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.184+0000 7f04921ba700 20 client.49019.objecter get_session s=0x7f046c027190 osd=7 3 2024-02-19T21:05:56.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.184+0000 7f04921ba700 15 client.49019.objecter _session_command_op_assign 7 16 2024-02-19T21:05:56.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.184+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c027190 osd=7 4 2024-02-19T21:05:56.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.184+0000 7f04921ba700 10 client.49019.objecter _send_command 16 2024-02-19T21:05:56.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.184+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f046c02cc90 con 0x7f046c027840 2024-02-19T21:05:56.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04897fa700 1 -- 172.21.15.163:0/3457809448 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f046c02cc90 con 0x7f046c027840 2024-02-19T21:05:56.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04897fa700 10 client.49019.objecter ms_dispatch 0x7f048c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:05:56.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04897fa700 10 client.49019.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04897fa700 20 client.49019.objecter put_session s=0x7f046c027190 osd=7 4 2024-02-19T21:05:56.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04897fa700 15 client.49019.objecter _session_command_op_remove 7 16 2024-02-19T21:05:56.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 10 librados: watch_flush enter 2024-02-19T21:05:56.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 10 librados: watch_flush exit 2024-02-19T21:05:56.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 10 client.49019.objecter close_session for osd.0 2024-02-19T21:05:56.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 >> v1:172.21.15.163:6801/1559418115 conn(0x7f046c004530 legacy=0x7f046c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:56.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c0051a0 osd=0 1 2024-02-19T21:05:56.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 10 client.49019.objecter close_session for osd.1 2024-02-19T21:05:56.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 >> v1:172.21.15.163:6805/3125047432 conn(0x7f046c004150 legacy=0x7f046c00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:56.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c003fe0 osd=1 1 2024-02-19T21:05:56.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 10 client.49019.objecter close_session for osd.2 2024-02-19T21:05:56.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 >> v1:172.21.15.163:6809/1563158487 conn(0x7f046c00c170 legacy=0x7f046c00c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:56.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c003630 osd=2 1 2024-02-19T21:05:56.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 10 client.49019.objecter close_session for osd.3 2024-02-19T21:05:56.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 >> v1:172.21.15.163:6813/2112279769 conn(0x7f046c016290 legacy=0x7f046c018680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:56.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c003380 osd=3 1 2024-02-19T21:05:56.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 10 client.49019.objecter close_session for osd.4 2024-02-19T21:05:56.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 >> v1:172.21.15.193:6800/3129716965 conn(0x7f046c01c360 legacy=0x7f046c00ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:56.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f048bfff700 1 -- 172.21.15.163:0/3457809448 reap_dead start 2024-02-19T21:05:56.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f048bfff700 1 -- 172.21.15.163:0/3457809448 reap_dead start 2024-02-19T21:05:56.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c01b1b0 osd=4 1 2024-02-19T21:05:56.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 10 client.49019.objecter close_session for osd.5 2024-02-19T21:05:56.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 >> v1:172.21.15.193:6804/805161991 conn(0x7f046c01eef0 legacy=0x7f046c01f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:56.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c01e840 osd=5 1 2024-02-19T21:05:56.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 10 client.49019.objecter close_session for osd.6 2024-02-19T21:05:56.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 >> v1:172.21.15.193:6808/787604615 conn(0x7f046c023680 legacy=0x7f046c025a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:56.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c020bf0 osd=6 1 2024-02-19T21:05:56.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 10 client.49019.objecter close_session for osd.7 2024-02-19T21:05:56.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 >> v1:172.21.15.193:6812/157927432 conn(0x7f046c027840 legacy=0x7f046c029c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:56.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 20 client.49019.objecter put_session s=0x7f046c027190 osd=7 1 2024-02-19T21:05:56.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 20 client.49019.objecter shutdown clearing up homeless session... 2024-02-19T21:05:56.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.205+0000 7f04921ba700 10 client.49019.objecter successfully canceled tick 2024-02-19T21:05:56.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.206+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 >> v1:172.21.15.163:6800/1552514681 conn(0x7f0478063cc0 legacy=0x7f0478066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:56.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.206+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 >> v1:172.21.15.163:6790/0 conn(0x7f048c0f9530 legacy=0x7f048c0ffa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:05:56.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.206+0000 7f048bfff700 1 -- 172.21.15.163:0/3457809448 reap_dead start 2024-02-19T21:05:56.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.206+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 shutdown_connections 2024-02-19T21:05:56.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.206+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 >> 172.21.15.163:0/3457809448 conn(0x7f048c0f2470 msgr2=0x7f048c106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:05:56.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.206+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 shutdown_connections 2024-02-19T21:05:56.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.206+0000 7f04921ba700 1 -- 172.21.15.163:0/3457809448 wait complete. 2024-02-19T21:05:56.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:05:56.206+0000 7f04921ba700 1 librados: shutdown 2024-02-19T21:05:56.388 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.388 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.388 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.388 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.388 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.388 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.388 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.388 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:05:56.591 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:05:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:56.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:05:56.694 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:56.722 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:56.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:55.133440+0000 mgr.y (mgr.14550) 218 : cluster [DBG] pgmap v233: 379 pgs: 32 creating+peering, 1 active+clean+snaptrim, 2 clean+premerge+peered, 344 active+clean; 16 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 MiB/s wr, 4 op/s 2024-02-19T21:05:56.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:55.719247+0000 osd.4 (osd.4) 89 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:56.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:55.719926+0000 osd.4 (osd.4) 90 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:56.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:56.040418+0000 mon.c (mon.2) 168 : audit [DBG] from='client.? v1:172.21.15.163:0/3457809448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:56.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:56.178069+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-02-19T21:05:56.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:56.184848+0000 mon.c (mon.2) 169 : audit [INF] from='client.? v1:172.21.15.163:0/3219076896' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi163-83845-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:56.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:56.193188+0000 mon.b (mon.1) 277 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:56.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:56.194296+0000 mon.a (mon.0) 1292 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:56.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:56.198371+0000 mon.a (mon.0) 1293 : audit [INF] from='client.49013 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi163-83845-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:56.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:56.282999+0000 mon.c (mon.2) 170 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:56.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:56.376312+0000 osd.1 (osd.1) 107 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:56.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:56.376786+0000 osd.1 (osd.1) 108 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:56.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:55.133440+0000 mgr.y (mgr.14550) 218 : cluster [DBG] pgmap v233: 379 pgs: 32 creating+peering, 1 active+clean+snaptrim, 2 clean+premerge+peered, 344 active+clean; 16 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 MiB/s wr, 4 op/s 2024-02-19T21:05:56.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:55.719247+0000 osd.4 (osd.4) 89 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:56.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:55.719926+0000 osd.4 (osd.4) 90 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:56.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:56.040418+0000 mon.c (mon.2) 168 : audit [DBG] from='client.? v1:172.21.15.163:0/3457809448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:56.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:56.178069+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-02-19T21:05:56.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:56.184848+0000 mon.c (mon.2) 169 : audit [INF] from='client.? v1:172.21.15.163:0/3219076896' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi163-83845-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:56.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:56.193188+0000 mon.b (mon.1) 277 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:56.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:56.194296+0000 mon.a (mon.0) 1292 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:56.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:56.198371+0000 mon.a (mon.0) 1293 : audit [INF] from='client.49013 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi163-83845-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:56.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:56.282999+0000 mon.c (mon.2) 170 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:56.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:56.376312+0000 osd.1 (osd.1) 107 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:56.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:56.376786+0000 osd.1 (osd.1) 108 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:57.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:55.133440+0000 mgr.y (mgr.14550) 218 : cluster [DBG] pgmap v233: 379 pgs: 32 creating+peering, 1 active+clean+snaptrim, 2 clean+premerge+peered, 344 active+clean; 16 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.0 MiB/s wr, 4 op/s 2024-02-19T21:05:57.065 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:55.719247+0000 osd.4 (osd.4) 89 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:57.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:55.719926+0000 osd.4 (osd.4) 90 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:57.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:56.040418+0000 mon.c (mon.2) 168 : audit [DBG] from='client.? v1:172.21.15.163:0/3457809448' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:05:57.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:56.178069+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-02-19T21:05:57.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:56.184848+0000 mon.c (mon.2) 169 : audit [INF] from='client.? v1:172.21.15.163:0/3219076896' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi163-83845-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:57.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:56.193188+0000 mon.b (mon.1) 277 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:57.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:56.194296+0000 mon.a (mon.0) 1292 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:57.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:56.198371+0000 mon.a (mon.0) 1293 : audit [INF] from='client.49013 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi163-83845-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:57.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:56.282999+0000 mon.c (mon.2) 170 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:57.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:56.376312+0000 osd.1 (osd.1) 107 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:57.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:56.376786+0000 osd.1 (osd.1) 108 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:57.172 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:57.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:57.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:57.277 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:57.304 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:57.452 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:57.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:57.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:57.556 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:57.584 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:57.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:57.627 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:57.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:57.715 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:05:57.715 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:05:57.715 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-02-19T21:05:57.715 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-02-19T21:05:57.716 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:05:57.728 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:05:57.762 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:57.784 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:05:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:57.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:05:57.888 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:57.917 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:57.935 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:57.938 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:57.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:57.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:57.134425+0000 mon.b (mon.1) 278 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-19T21:05:57.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:57.135571+0000 mon.a (mon.0) 1294 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-19T21:05:57.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:57.180522+0000 mon.a (mon.0) 1295 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83920-24"}]': finished 2024-02-19T21:05:57.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:57.180741+0000 mon.a (mon.0) 1296 : audit [INF] from='client.49013 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi163-83845-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:57.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:57.180849+0000 mon.a (mon.0) 1297 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-19T21:05:57.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:57.180911+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-02-19T21:05:57.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:57.185434+0000 mon.a (mon.0) 1299 : audit [INF] from='client.? v1:172.21.15.163:0/3786926911' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi163-83854-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:57.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:57.185814+0000 mon.c (mon.2) 171 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:57.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:57.186072+0000 mon.b (mon.1) 279 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:57.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:57.186410+0000 mon.a (mon.0) 1300 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:57.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:57.187207+0000 mon.a (mon.0) 1301 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:57.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:57.283743+0000 mon.c (mon.2) 172 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:57.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:57.520031+0000 osd.0 (osd.0) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:57.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:57.520393+0000 osd.0 (osd.0) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:57.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:57.134425+0000 mon.b (mon.1) 278 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-19T21:05:57.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:57.135571+0000 mon.a (mon.0) 1294 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-19T21:05:57.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:57.180522+0000 mon.a (mon.0) 1295 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83920-24"}]': finished 2024-02-19T21:05:57.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:57.180741+0000 mon.a (mon.0) 1296 : audit [INF] from='client.49013 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi163-83845-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:57.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:57.180849+0000 mon.a (mon.0) 1297 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-19T21:05:57.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:57.180911+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-02-19T21:05:57.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:57.185434+0000 mon.a (mon.0) 1299 : audit [INF] from='client.? v1:172.21.15.163:0/3786926911' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi163-83854-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:57.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:57.185814+0000 mon.c (mon.2) 171 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:57.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:57.186072+0000 mon.b (mon.1) 279 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:57.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:57.186410+0000 mon.a (mon.0) 1300 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:57.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:57.187207+0000 mon.a (mon.0) 1301 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:57.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:57.283743+0000 mon.c (mon.2) 172 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:57.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:57.520031+0000 osd.0 (osd.0) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:57.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:57.520393+0000 osd.0 (osd.0) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:58.039 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:05:58.060 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:05:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:05:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:05:58.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:57.134425+0000 mon.b (mon.1) 278 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-19T21:05:58.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:57.135571+0000 mon.a (mon.0) 1294 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-02-19T21:05:58.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:57.180522+0000 mon.a (mon.0) 1295 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83920-24"}]': finished 2024-02-19T21:05:58.060 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:57.180741+0000 mon.a (mon.0) 1296 : audit [INF] from='client.49013 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi163-83845-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:58.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:57.180849+0000 mon.a (mon.0) 1297 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-02-19T21:05:58.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:57.180911+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-02-19T21:05:58.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:57.185434+0000 mon.a (mon.0) 1299 : audit [INF] from='client.? v1:172.21.15.163:0/3786926911' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi163-83854-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:05:58.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:57.185814+0000 mon.c (mon.2) 171 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:58.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:57.186072+0000 mon.b (mon.1) 279 : audit [INF] from='client.? v1:172.21.15.163:0/518414052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:58.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:57.186410+0000 mon.a (mon.0) 1300 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:58.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:57.187207+0000 mon.a (mon.0) 1301 : audit [INF] from='client.47791 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83920-24"}]: dispatch 2024-02-19T21:05:58.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:57.283743+0000 mon.c (mon.2) 172 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:58.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:57.520031+0000 osd.0 (osd.0) 97 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:05:58.061 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:57.520393+0000 osd.0 (osd.0) 98 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:05:58.070 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:58.093 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:05:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:58.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:05:58.197 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:58.232 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:58.276 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:05:58.461 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-19T21:05:58.480 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:58.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:58.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:58.548 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:58.583 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:05:58.609 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:58.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:57.133873+0000 mgr.y (mgr.14550) 219 : cluster [DBG] pgmap v236: 306 pgs: 32 unknown, 1 active+clean+snaptrim, 1 clean+premerge+peered, 272 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-02-19T21:05:58.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:58.183100+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? v1:172.21.15.163:0/3786926911' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi163-83854-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:58.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:58.183202+0000 mon.a (mon.0) 1303 : audit [INF] from='client.48947 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]': finished 2024-02-19T21:05:58.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:58.183279+0000 mon.a (mon.0) 1304 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83920-24"}]': finished 2024-02-19T21:05:58.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:58.183423+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-02-19T21:05:58.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:58.186343+0000 mon.c (mon.2) 173 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:58.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:58.187103+0000 mon.a (mon.0) 1306 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:58.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:58.228351+0000 mon.c (mon.2) 174 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:05:58.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:58.228872+0000 mon.a (mon.0) 1307 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:05:58.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:58.229662+0000 mon.c (mon.2) 175 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:05:58.747 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:05:58.808 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:05:58.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:58.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:05:58.912 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:58.938 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:58.958 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:58.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:05:58.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:57.133873+0000 mgr.y (mgr.14550) 219 : cluster [DBG] pgmap v236: 306 pgs: 32 unknown, 1 active+clean+snaptrim, 1 clean+premerge+peered, 272 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-02-19T21:05:58.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:58.183100+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? v1:172.21.15.163:0/3786926911' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi163-83854-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:58.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:58.183202+0000 mon.a (mon.0) 1303 : audit [INF] from='client.48947 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]': finished 2024-02-19T21:05:58.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:58.183279+0000 mon.a (mon.0) 1304 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83920-24"}]': finished 2024-02-19T21:05:58.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:58.183423+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-02-19T21:05:58.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:58.186343+0000 mon.c (mon.2) 173 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:58.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:58.187103+0000 mon.a (mon.0) 1306 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:58.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:58.228351+0000 mon.c (mon.2) 174 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:05:58.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:58.228872+0000 mon.a (mon.0) 1307 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:05:58.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:58.229662+0000 mon.c (mon.2) 175 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:05:58.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:58.284449+0000 mon.c (mon.2) 176 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:58.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:58.472385+0000 mon.a (mon.0) 1308 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:05:58.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:58.473092+0000 mon.c (mon.2) 177 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi163-83845-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:58.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:58.473392+0000 mon.a (mon.0) 1309 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi163-83845-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:58.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:57.133873+0000 mgr.y (mgr.14550) 219 : cluster [DBG] pgmap v236: 306 pgs: 32 unknown, 1 active+clean+snaptrim, 1 clean+premerge+peered, 272 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-02-19T21:05:58.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:58.183100+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? v1:172.21.15.163:0/3786926911' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi163-83854-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:05:58.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:58.183202+0000 mon.a (mon.0) 1303 : audit [INF] from='client.48947 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi163-83952-15"}]': finished 2024-02-19T21:05:58.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:58.183279+0000 mon.a (mon.0) 1304 : audit [INF] from='client.47791 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83920-24"}]': finished 2024-02-19T21:05:58.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:58.183423+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-02-19T21:05:58.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:58.186343+0000 mon.c (mon.2) 173 : audit [INF] from='client.? v1:172.21.15.163:0/3754083491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:58.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:58.187103+0000 mon.a (mon.0) 1306 : audit [INF] from='client.48947 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi163-83952-15"}]: dispatch 2024-02-19T21:05:58.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:58.228351+0000 mon.c (mon.2) 174 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:05:58.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:58.228872+0000 mon.a (mon.0) 1307 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:05:58.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:58.229662+0000 mon.c (mon.2) 175 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:05:58.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:58.284449+0000 mon.c (mon.2) 176 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:58.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:58.472385+0000 mon.a (mon.0) 1308 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:05:58.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:58.473092+0000 mon.c (mon.2) 177 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi163-83845-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:58.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:05:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:58.473392+0000 mon.a (mon.0) 1309 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi163-83845-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:59.009 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:58.284449+0000 mon.c (mon.2) 176 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:05:59.009 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:58.472385+0000 mon.a (mon.0) 1308 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:05:59.009 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:58.473092+0000 mon.c (mon.2) 177 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi163-83845-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:59.010 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:05:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:58.473392+0000 mon.a (mon.0) 1309 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi163-83845-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:05:59.062 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:05:59.064 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:05:59.088 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:59.188 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] 22 tests from LibRadosMisc api_snapshots_pp: Running main() from gmock_main.cc 2024-02-19T21:05:59.188 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [==========] Running 20 tests from 5 test suites. 2024-02-19T21:05:59.188 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-02-19T21:05:59.188 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-02-19T21:05:59.188 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: seed 83952 2024-02-19T21:05:59.188 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-02-19T21:05:59.188 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2095 ms) 2024-02-19T21:05:59.188 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-02-19T21:05:59.188 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (3792 ms) 2024-02-19T21:05:59.188 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-02-19T21:05:59.188 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2538 ms) 2024-02-19T21:05:59.188 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-02-19T21:05:59.189 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (1618 ms) 2024-02-19T21:05:59.189 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-02-19T21:05:59.189 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (4007 ms) 2024-02-19T21:05:59.189 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (14050 ms total) 2024-02-19T21:05:59.189 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: 2024-02-19T21:05:59.189 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP 2024-02-19T21:05:59.189 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-02-19T21:05:59.189 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4398 ms) 2024-02-19T21:05:59.189 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-02-19T21:05:59.189 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4813 ms) 2024-02-19T21:05:59.189 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-02-19T21:05:59.189 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (7140 ms) 2024-02-19T21:05:59.189 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-02-19T21:05:59.189 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4162 ms) 2024-02-19T21:05:59.189 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-02-19T21:05:59.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2294 ms) 2024-02-19T21:05:59.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-02-19T21:05:59.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi163-83952-7 2024-02-19T21:05:59.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: waiting for snaps to purge 2024-02-19T21:05:59.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17570 ms) 2024-02-19T21:05:59.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP (40377 ms total) 2024-02-19T21:05:59.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: 2024-02-19T21:05:59.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-02-19T21:05:59.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-02-19T21:05:59.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (8 ms) 2024-02-19T21:05:59.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-02-19T21:05:59.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5012 ms) 2024-02-19T21:05:59.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5020 ms total) 2024-02-19T21:05:59.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: 2024-02-19T21:05:59.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-02-19T21:05:59.191 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-02-19T21:05:59.191 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3002 ms) 2024-02-19T21:05:59.191 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-02-19T21:05:59.191 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2005 ms) 2024-02-19T21:05:59.191 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-02-19T21:05:59.191 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2013 ms) 2024-02-19T21:05:59.191 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-02-19T21:05:59.191 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1975 ms) 2024-02-19T21:05:59.191 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8995 ms total) 2024-02-19T21:05:59.191 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: 2024-02-19T21:05:59.236 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:05:59.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:59.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:05:59.340 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:05:59.369 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:59.393 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:05:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:59.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:05:59.497 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:05:59.525 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:59.599 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:05:59.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:59.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:05:59.704 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:05:59.731 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:05:59.815 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:05:59.980 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:05:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:05:59.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:00.083 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:06:00.110 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:00.210 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:06:00.212 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:06:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:06:00.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:00.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:06:00.314 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:06:00.343 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:00.445 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:06:00.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:00.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:06:00.467 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:06:00.467 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-02-19T21:06:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-02-19T21:06:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-02-19T21:06:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:109 61.1e 61:7b0234b0:test-rados-api-smithi163-83928-10::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e126)", 2024-02-19T21:06:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:06.034266+0000", 2024-02-19T21:06:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 54.437128663000003, 2024-02-19T21:06:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013927448, 2024-02-19T21:06:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:06:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:06:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:06:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 109 2024-02-19T21:06:00.468 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034266+0000", 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034266+0000", 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034264+0000", 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4800000000000001e-06 2024-02-19T21:06:00.469 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034272+0000", 2024-02-19T21:06:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8009999999999993e-06 2024-02-19T21:06:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034281+0000", 2024-02-19T21:06:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9020000000000002e-06 2024-02-19T21:06:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034288+0000", 2024-02-19T21:06:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018518399999999999 2024-02-19T21:06:00.470 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034473+0000", 2024-02-19T21:06:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012265400000000001 2024-02-19T21:06:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034596+0000", 2024-02-19T21:06:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000145477 2024-02-19T21:06:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,6", 2024-02-19T21:06:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034742+0000", 2024-02-19T21:06:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001663596 2024-02-19T21:06:00.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:06:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.036405+0000", 2024-02-19T21:06:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0045168939999999996 2024-02-19T21:06:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.040922+0000", 2024-02-19T21:06:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0072134110000000003 2024-02-19T21:06:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.048135+0000", 2024-02-19T21:06:00.472 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4739000000000001e-05 2024-02-19T21:06:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:06:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.048170+0000", 2024-02-19T21:06:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3264e-05 2024-02-19T21:06:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.048193+0000", 2024-02-19T21:06:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013927448 2024-02-19T21:06:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.38953.0:6 82.c 82:3197012c:::foo_rmxattr:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.230127+0000", 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 48.241268007000002, 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.044067013000000002, 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.38953", 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3840798168", 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 6 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.474 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230127+0000", 2024-02-19T21:06:00.475 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.475 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.475 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.475 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.475 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230127+0000", 2024-02-19T21:06:00.475 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:06:00.475 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.475 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.475 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.475 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230125+0000", 2024-02-19T21:06:00.475 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5880000000000004e-06 2024-02-19T21:06:00.475 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.475 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.475 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.476 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230134+0000", 2024-02-19T21:06:00.476 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1489999999999994e-06 2024-02-19T21:06:00.476 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.476 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.476 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.476 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230142+0000", 2024-02-19T21:06:00.476 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9089999999999999e-06 2024-02-19T21:06:00.476 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.476 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.476 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.476 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230147+0000", 2024-02-19T21:06:00.476 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5340999999999997e-05 2024-02-19T21:06:00.476 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.476 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.476 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230213+0000", 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000120505 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230333+0000", 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015051999999999999 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,4", 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230484+0000", 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00033057799999999999 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:06:00.477 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230814+0000", 2024-02-19T21:06:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000223568 2024-02-19T21:06:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.231038+0000", 2024-02-19T21:06:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.043097783000000001 2024-02-19T21:06:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.274136+0000", 2024-02-19T21:06:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2132999999999999e-05 2024-02-19T21:06:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:06:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.274168+0000", 2024-02-19T21:06:00.479 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5828000000000001e-05 2024-02-19T21:06:00.479 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.479 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.479 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.479 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.274194+0000", 2024-02-19T21:06:00.479 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.044067013000000002 2024-02-19T21:06:00.479 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.479 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.479 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.479 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.479 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.479 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.38953.0:7 82.c 82:3197012c:::foo_rmxattr:head [setxattr attr2 (12) in=5b] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-02-19T21:06:00.479 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.277467+0000", 2024-02-19T21:06:00.479 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 48.193928034000002, 2024-02-19T21:06:00.479 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0054803059999999999, 2024-02-19T21:06:00.480 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.480 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:06:00.480 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.480 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.38953", 2024-02-19T21:06:00.480 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3840798168", 2024-02-19T21:06:00.480 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 7 2024-02-19T21:06:00.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.480 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.480 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.480 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.277467+0000", 2024-02-19T21:06:00.480 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.480 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.480 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.277467+0000", 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.277463+0000", 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6711999999999999e-05 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.277480+0000", 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6146999999999999e-05 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.481 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.482 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.277506+0000", 2024-02-19T21:06:00.482 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2262999999999999e-05 2024-02-19T21:06:00.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.482 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.482 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.482 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.277518+0000", 2024-02-19T21:06:00.482 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7418999999999998e-05 2024-02-19T21:06:00.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.482 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.482 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.482 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.277605+0000", 2024-02-19T21:06:00.482 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000127601 2024-02-19T21:06:00.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.482 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.482 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.277733+0000", 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000199731 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,4", 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.277933+0000", 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000270592 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.278203+0000", 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00033046200000000001 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.483 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.278534+0000", 2024-02-19T21:06:00.484 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0042128620000000004 2024-02-19T21:06:00.484 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.484 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.484 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.484 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.282747+0000", 2024-02-19T21:06:00.484 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013421799999999999 2024-02-19T21:06:00.484 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.484 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.484 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:06:00.484 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.282881+0000", 2024-02-19T21:06:00.484 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5955000000000001e-05 2024-02-19T21:06:00.484 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.484 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.484 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.484 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.282947+0000", 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0054803059999999999 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.38953.0:8 82.c 82:3197012c:::foo_rmxattr:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e132)", 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.288784+0000", 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 48.182610553000004, 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.027473164000000001, 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.38953", 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3840798168", 2024-02-19T21:06:00.485 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 8 2024-02-19T21:06:00.486 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.486 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.486 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.486 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.486 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288784+0000", 2024-02-19T21:06:00.486 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.486 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.486 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.486 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.486 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288784+0000", 2024-02-19T21:06:00.486 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:06:00.486 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.486 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.486 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.486 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288781+0000", 2024-02-19T21:06:00.487 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2977e-05 2024-02-19T21:06:00.487 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.487 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.487 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.487 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288794+0000", 2024-02-19T21:06:00.487 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6616000000000001e-05 2024-02-19T21:06:00.487 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.487 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.487 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.487 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288820+0000", 2024-02-19T21:06:00.487 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2593e-05 2024-02-19T21:06:00.487 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.487 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.487 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.487 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288833+0000", 2024-02-19T21:06:00.488 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9002000000000001e-05 2024-02-19T21:06:00.488 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.488 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.488 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.488 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288902+0000", 2024-02-19T21:06:00.488 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4057000000000002e-05 2024-02-19T21:06:00.488 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.488 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.488 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.488 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288946+0000", 2024-02-19T21:06:00.488 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2805999999999994e-05 2024-02-19T21:06:00.488 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.488 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.488 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,4", 2024-02-19T21:06:00.488 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.289019+0000", 2024-02-19T21:06:00.489 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025527699999999999 2024-02-19T21:06:00.489 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.489 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.489 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:06:00.489 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.289274+0000", 2024-02-19T21:06:00.489 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000221688 2024-02-19T21:06:00.489 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.489 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.489 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.489 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.289496+0000", 2024-02-19T21:06:00.489 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026693499999999998 2024-02-19T21:06:00.489 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.489 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.489 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.489 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.316189+0000", 2024-02-19T21:06:00.490 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.042e-05 2024-02-19T21:06:00.490 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.490 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.490 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:06:00.490 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.316220+0000", 2024-02-19T21:06:00.490 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7688000000000002e-05 2024-02-19T21:06:00.490 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.490 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.490 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.490 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.316257+0000", 2024-02-19T21:06:00.490 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.027473164000000001 2024-02-19T21:06:00.490 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.490 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.490 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.24919.0:2077 28.6 28:6320398f:test-rados-api-smithi163-83920-17::small.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected e102)", 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.322491+0000", 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 48.148903148000002, 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0065386519999999998, 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.24919", 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/4089460683", 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2077 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.491 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.492 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322491+0000", 2024-02-19T21:06:00.492 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.492 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.492 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.492 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322491+0000", 2024-02-19T21:06:00.492 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:06:00.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.492 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.492 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.492 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322489+0000", 2024-02-19T21:06:00.492 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6159999999999996e-06 2024-02-19T21:06:00.492 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.492 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.492 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.493 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322497+0000", 2024-02-19T21:06:00.493 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1027e-05 2024-02-19T21:06:00.493 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.493 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.493 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.493 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322508+0000", 2024-02-19T21:06:00.493 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3990000000000001e-06 2024-02-19T21:06:00.493 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.493 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.493 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.493 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322514+0000", 2024-02-19T21:06:00.493 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3918999999999995e-05 2024-02-19T21:06:00.493 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.493 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.493 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322588+0000", 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3276999999999996e-05 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322672+0000", 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00092435900000000003 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,3", 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.323596+0000", 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00035623600000000001 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:06:00.494 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.323952+0000", 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001907939 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.325860+0000", 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0031191090000000001 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.328979+0000", 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.8789000000000002e-05 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.329008+0000", 2024-02-19T21:06:00.495 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2078000000000001e-05 2024-02-19T21:06:00.496 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.496 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.496 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.496 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.329030+0000", 2024-02-19T21:06:00.496 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0065386519999999998 2024-02-19T21:06:00.496 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.496 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.496 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.496 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.496 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.496 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:248 61.1e 61:780b1894:test-rados-api-smithi163-83928-12::bar:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-02-19T21:06:00.496 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:14.241429+0000", 2024-02-19T21:06:00.496 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 46.229965419999999, 2024-02-19T21:06:00.496 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.036279570999999997, 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 248 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241429+0000", 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.497 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241429+0000", 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241427+0000", 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1410000000000005e-06 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241435+0000", 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7259999999999997e-06 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241445+0000", 2024-02-19T21:06:00.498 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3310000000000003e-06 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241450+0000", 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5508999999999999e-05 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241516+0000", 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012163399999999999 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241638+0000", 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000150047 2024-02-19T21:06:00.499 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.500 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.500 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,6", 2024-02-19T21:06:00.500 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241788+0000", 2024-02-19T21:06:00.500 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00038934700000000001 2024-02-19T21:06:00.500 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.500 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.500 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:06:00.500 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.242177+0000", 2024-02-19T21:06:00.500 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018429899999999999 2024-02-19T21:06:00.500 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.500 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.500 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.500 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.242361+0000", 2024-02-19T21:06:00.500 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.035272228000000003 2024-02-19T21:06:00.500 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.277633+0000", 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8115000000000003e-05 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.277682+0000", 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7237e-05 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.277709+0000", 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.036279570999999997 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.501 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:249 61.10 61:0d6787e5:test-rados-api-smithi163-83928-12::baz:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:14.291021+0000", 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 46.180373582999998, 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.031923083999999997, 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 249 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.502 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.503 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.503 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291021+0000", 2024-02-19T21:06:00.503 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.503 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.503 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.503 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291021+0000", 2024-02-19T21:06:00.503 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:06:00.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.503 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.503 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.503 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291019+0000", 2024-02-19T21:06:00.503 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1210000000000001e-06 2024-02-19T21:06:00.503 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.503 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.504 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.504 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291026+0000", 2024-02-19T21:06:00.504 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8940000000000005e-06 2024-02-19T21:06:00.504 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.504 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.504 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.504 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291036+0000", 2024-02-19T21:06:00.504 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8369999999999996e-06 2024-02-19T21:06:00.504 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.504 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.504 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.504 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291041+0000", 2024-02-19T21:06:00.504 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8821999999999999e-05 2024-02-19T21:06:00.504 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.505 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.505 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.505 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291109+0000", 2024-02-19T21:06:00.505 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4403999999999999e-05 2024-02-19T21:06:00.505 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.505 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.505 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.505 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291204+0000", 2024-02-19T21:06:00.505 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2696999999999997e-05 2024-02-19T21:06:00.505 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.505 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.505 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-02-19T21:06:00.505 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291297+0000", 2024-02-19T21:06:00.505 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00037753299999999999 2024-02-19T21:06:00.505 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291674+0000", 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000206999 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291881+0000", 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.030980852999999999 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.322862+0000", 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5098000000000001e-05 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.506 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.322907+0000", 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7076000000000002e-05 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.322944+0000", 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.031923083999999997 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:253 61.1e 61:780b1894:test-rados-api-smithi163-83928-12::bar:head [writefull 0~5] snapc 2=[2] ondisk+write+known_if_redirected e135)", 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:15.193696+0000", 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 45.277698676999997, 2024-02-19T21:06:00.507 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.104949372, 2024-02-19T21:06:00.508 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.508 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:06:00.508 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.508 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:06:00.508 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:06:00.508 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 253 2024-02-19T21:06:00.508 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.508 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.508 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.508 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.508 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193696+0000", 2024-02-19T21:06:00.508 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.508 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.508 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193696+0000", 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193694+0000", 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0650000000000001e-06 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193701+0000", 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.431e-06 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.509 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193711+0000", 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0629999999999996e-06 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193716+0000", 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0915000000000003e-05 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193767+0000", 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9301999999999995e-05 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193836+0000", 2024-02-19T21:06:00.510 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012302 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,6", 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193959+0000", 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00041847100000000001 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.194377+0000", 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00021071800000000001 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.194588+0000", 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.103953399 2024-02-19T21:06:00.511 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.512 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.512 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.512 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.298541+0000", 2024-02-19T21:06:00.512 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9375e-05 2024-02-19T21:06:00.512 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.512 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.512 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:06:00.512 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.298601+0000", 2024-02-19T21:06:00.512 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4469999999999999e-05 2024-02-19T21:06:00.512 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.512 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.512 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.512 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.298645+0000", 2024-02-19T21:06:00.512 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.104949372 2024-02-19T21:06:00.512 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.513 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.513 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.513 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.513 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.513 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:878 94.15 94:afdf704a:test-rados-api-smithi163-83928-14::foo25:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-19T21:06:00.513 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:31.221744+0000", 2024-02-19T21:06:00.513 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 29.249651087, 2024-02-19T21:06:00.513 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0055046590000000003, 2024-02-19T21:06:00.513 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.513 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:06:00.513 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.513 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:06:00.513 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:06:00.513 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 878 2024-02-19T21:06:00.513 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.221744+0000", 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.221744+0000", 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.221740+0000", 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0838e-05 2024-02-19T21:06:00.514 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.221751+0000", 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6963e-05 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.221768+0000", 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4870000000000008e-06 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.221778+0000", 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3424999999999997e-05 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.515 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.516 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.516 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.221841+0000", 2024-02-19T21:06:00.516 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00023138600000000001 2024-02-19T21:06:00.516 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.516 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.516 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:06:00.516 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.222072+0000", 2024-02-19T21:06:00.516 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0013751600000000001 2024-02-19T21:06:00.516 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.516 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.516 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.516 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.223448+0000", 2024-02-19T21:06:00.516 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9454999999999999e-05 2024-02-19T21:06:00.516 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:06:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.223507+0000", 2024-02-19T21:06:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000798104 2024-02-19T21:06:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.224305+0000", 2024-02-19T21:06:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8437999999999999e-05 2024-02-19T21:06:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.224354+0000", 2024-02-19T21:06:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0028946459999999999 2024-02-19T21:06:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.518 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.518 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.518 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.227248+0000", 2024-02-19T21:06:00.518 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0055046590000000003 2024-02-19T21:06:00.518 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.518 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.518 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.518 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.518 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.518 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:880 94.1 94:877aab9c:test-rados-api-smithi163-83928-14::foo35:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-19T21:06:00.518 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:31.232804+0000", 2024-02-19T21:06:00.518 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 29.238590640999998, 2024-02-19T21:06:00.518 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.004991568, 2024-02-19T21:06:00.518 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.518 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 880 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.232804+0000", 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.232804+0000", 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:06:00.519 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.520 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.520 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.520 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.232800+0000", 2024-02-19T21:06:00.520 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4514e-05 2024-02-19T21:06:00.520 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.520 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.520 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.520 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.232815+0000", 2024-02-19T21:06:00.520 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6045e-05 2024-02-19T21:06:00.520 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.520 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.520 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.520 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.232831+0000", 2024-02-19T21:06:00.520 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6000000000000001e-06 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.232838+0000", 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4240999999999999e-05 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.232883+0000", 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000190562 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.233073+0000", 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014795730000000001 2024-02-19T21:06:00.521 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.522 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.522 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.522 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.234553+0000", 2024-02-19T21:06:00.522 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4618999999999997e-05 2024-02-19T21:06:00.522 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.522 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.522 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:06:00.522 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.234647+0000", 2024-02-19T21:06:00.522 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00029817599999999999 2024-02-19T21:06:00.522 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.522 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.522 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.522 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.234946+0000", 2024-02-19T21:06:00.522 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7767000000000002e-05 2024-02-19T21:06:00.523 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.523 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.523 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.523 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.235033+0000", 2024-02-19T21:06:00.523 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0027622179999999999 2024-02-19T21:06:00.523 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.523 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.523 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.523 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.237796+0000", 2024-02-19T21:06:00.523 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.004991568 2024-02-19T21:06:00.523 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.523 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.523 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.523 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.523 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.524 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:1022 94.15 94:ad6f9ae3:test-rados-api-smithi163-83928-14::foo38:5 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-19T21:06:00.524 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:31.582158+0000", 2024-02-19T21:06:00.524 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 28.889236814, 2024-02-19T21:06:00.524 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0050855010000000001, 2024-02-19T21:06:00.524 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.524 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:06:00.524 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.524 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:06:00.524 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:06:00.524 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1022 2024-02-19T21:06:00.524 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.524 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.524 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.524 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.525 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.582158+0000", 2024-02-19T21:06:00.525 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.525 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.525 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.525 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.525 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.582158+0000", 2024-02-19T21:06:00.525 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:06:00.525 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.525 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.525 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.525 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.582154+0000", 2024-02-19T21:06:00.525 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4126e-05 2024-02-19T21:06:00.525 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.525 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.582168+0000", 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6424e-05 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.582195+0000", 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2391999999999999e-05 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.582207+0000", 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0109000000000001e-05 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.526 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.582297+0000", 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022300299999999999 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.582520+0000", 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000850719 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.583371+0000", 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013638300000000001 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:06:00.527 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.583507+0000", 2024-02-19T21:06:00.528 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00081293800000000003 2024-02-19T21:06:00.528 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.528 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.528 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.528 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.584320+0000", 2024-02-19T21:06:00.528 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000102052 2024-02-19T21:06:00.528 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.528 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.528 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:06:00.528 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.584422+0000", 2024-02-19T21:06:00.528 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00021477000000000001 2024-02-19T21:06:00.528 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.528 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.528 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.528 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.584637+0000", 2024-02-19T21:06:00.529 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9803000000000003e-05 2024-02-19T21:06:00.529 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.529 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.529 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.529 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.584697+0000", 2024-02-19T21:06:00.529 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025464680000000001 2024-02-19T21:06:00.529 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.529 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.529 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.529 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.587243+0000", 2024-02-19T21:06:00.529 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0050855010000000001 2024-02-19T21:06:00.529 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.529 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.529 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.529 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.530 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.530 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:1030 94.1 94:854d9d95:test-rados-api-smithi163-83928-14::foo70:5 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-19T21:06:00.530 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:31.611365+0000", 2024-02-19T21:06:00.530 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 28.860029738000001, 2024-02-19T21:06:00.530 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0043171980000000004, 2024-02-19T21:06:00.530 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.530 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:06:00.530 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.530 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:06:00.530 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:06:00.530 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1030 2024-02-19T21:06:00.530 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.530 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.530 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.531 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.531 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.611365+0000", 2024-02-19T21:06:00.531 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.531 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.531 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.531 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.531 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.611365+0000", 2024-02-19T21:06:00.531 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:06:00.531 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.531 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.531 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.531 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.611363+0000", 2024-02-19T21:06:00.531 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0820000000000004e-06 2024-02-19T21:06:00.531 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.531 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.532 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.532 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.611369+0000", 2024-02-19T21:06:00.532 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0737000000000001e-05 2024-02-19T21:06:00.532 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.532 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.532 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.532 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.611380+0000", 2024-02-19T21:06:00.532 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.079e-06 2024-02-19T21:06:00.532 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.532 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.532 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.532 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.611385+0000", 2024-02-19T21:06:00.532 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8708999999999998e-05 2024-02-19T21:06:00.532 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.533 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.533 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.533 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.611424+0000", 2024-02-19T21:06:00.533 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014783600000000001 2024-02-19T21:06:00.533 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.533 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.533 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:06:00.533 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.611572+0000", 2024-02-19T21:06:00.533 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00079314900000000005 2024-02-19T21:06:00.533 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.533 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.533 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.533 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.612365+0000", 2024-02-19T21:06:00.533 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5051999999999999e-05 2024-02-19T21:06:00.534 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.534 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.534 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:06:00.534 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.612440+0000", 2024-02-19T21:06:00.534 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000283107 2024-02-19T21:06:00.534 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.534 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.534 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.534 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.612723+0000", 2024-02-19T21:06:00.534 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9998999999999999e-05 2024-02-19T21:06:00.534 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.534 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.534 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.534 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.612783+0000", 2024-02-19T21:06:00.534 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0028992340000000001 2024-02-19T21:06:00.535 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.535 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.535 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.535 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.615682+0000", 2024-02-19T21:06:00.535 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0043171980000000004 2024-02-19T21:06:00.535 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.535 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.535 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.535 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.535 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.535 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:1031 94.b 94:d40ead83:test-rados-api-smithi163-83928-14::foo24:5 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-19T21:06:00.535 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:31.616149+0000", 2024-02-19T21:06:00.535 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 28.855246073, 2024-02-19T21:06:00.535 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0051042250000000004, 2024-02-19T21:06:00.536 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.536 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:06:00.536 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.536 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:06:00.536 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:06:00.536 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1031 2024-02-19T21:06:00.536 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.536 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.536 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.536 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.536 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.616149+0000", 2024-02-19T21:06:00.536 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.536 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.536 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.537 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.537 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.616149+0000", 2024-02-19T21:06:00.537 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:06:00.537 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.537 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.537 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.537 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.616147+0000", 2024-02-19T21:06:00.537 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9049999999999996e-06 2024-02-19T21:06:00.537 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.537 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.537 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.537 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.616152+0000", 2024-02-19T21:06:00.537 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5640000000000002e-06 2024-02-19T21:06:00.537 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.537 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.616162+0000", 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6870000000000002e-06 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.616166+0000", 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3958999999999998e-05 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.616200+0000", 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000119752 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.538 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:06:00.539 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.616320+0000", 2024-02-19T21:06:00.539 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000945772 2024-02-19T21:06:00.539 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.539 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.539 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.539 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.617266+0000", 2024-02-19T21:06:00.539 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00019301299999999999 2024-02-19T21:06:00.539 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.539 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.539 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:06:00.539 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.617459+0000", 2024-02-19T21:06:00.539 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00094729099999999995 2024-02-19T21:06:00.539 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.539 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.540 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.540 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.618406+0000", 2024-02-19T21:06:00.540 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000100141 2024-02-19T21:06:00.540 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.540 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.540 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:06:00.540 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.618506+0000", 2024-02-19T21:06:00.540 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000272086 2024-02-19T21:06:00.540 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.540 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.540 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.540 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.618778+0000", 2024-02-19T21:06:00.540 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9778999999999998e-05 2024-02-19T21:06:00.540 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.540 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.541 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.541 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.618838+0000", 2024-02-19T21:06:00.541 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024146060000000001 2024-02-19T21:06:00.541 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.541 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.541 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.541 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.621253+0000", 2024-02-19T21:06:00.541 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0051042250000000004 2024-02-19T21:06:00.541 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.541 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.541 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.541 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.541 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.541 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:1063 94.15 94:ad6f9ae3:test-rados-api-smithi163-83928-14::foo38:4 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-19T21:06:00.541 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:31.713144+0000", 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 28.758250626999999, 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0041202890000000001, 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1063 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.713144+0000", 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.542 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.543 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.713144+0000", 2024-02-19T21:06:00.543 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:06:00.543 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.543 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.543 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.543 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.713142+0000", 2024-02-19T21:06:00.543 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4339999999999997e-06 2024-02-19T21:06:00.543 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.543 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.543 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.543 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.713148+0000", 2024-02-19T21:06:00.543 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0019e-05 2024-02-19T21:06:00.543 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.543 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.543 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.544 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.713158+0000", 2024-02-19T21:06:00.544 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7910000000000004e-06 2024-02-19T21:06:00.544 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.544 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.544 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.544 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.713163+0000", 2024-02-19T21:06:00.544 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6152000000000002e-05 2024-02-19T21:06:00.544 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.544 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.544 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.544 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.713199+0000", 2024-02-19T21:06:00.544 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011734399999999999 2024-02-19T21:06:00.544 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.544 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.545 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:06:00.545 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.713317+0000", 2024-02-19T21:06:00.545 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00078580700000000002 2024-02-19T21:06:00.545 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.545 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.545 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.545 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.714103+0000", 2024-02-19T21:06:00.545 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8808999999999998e-05 2024-02-19T21:06:00.545 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.545 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.545 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:06:00.545 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.714171+0000", 2024-02-19T21:06:00.545 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000309862 2024-02-19T21:06:00.545 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.545 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.714481+0000", 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8551000000000002e-05 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.714540+0000", 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002724499 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.717264+0000", 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0041202890000000001 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.546 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:1064 94.b 94:d3ba4d38:test-rados-api-smithi163-83928-14::foo49:4 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:31.717412+0000", 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 28.753982732000001, 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0051778750000000002, 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1064 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.547 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.548 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.717412+0000", 2024-02-19T21:06:00.548 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.548 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.548 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.548 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.548 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.717412+0000", 2024-02-19T21:06:00.548 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:06:00.548 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.548 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.548 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.548 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.717411+0000", 2024-02-19T21:06:00.548 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3699999999999999e-06 2024-02-19T21:06:00.548 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.548 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.717414+0000", 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7900000000000002e-06 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.717421+0000", 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.743e-06 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.717425+0000", 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9947e-05 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.549 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.717455+0000", 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000118202 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.717573+0000", 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00096911600000000001 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.718542+0000", 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000155878 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.718698+0000", 2024-02-19T21:06:00.550 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001057868 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.719756+0000", 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0473000000000002e-05 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.719816+0000", 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00034043499999999999 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.720157+0000", 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0340999999999997e-05 2024-02-19T21:06:00.551 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.552 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.552 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.552 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.720217+0000", 2024-02-19T21:06:00.552 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0023726799999999998 2024-02-19T21:06:00.552 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.552 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.552 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.552 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.722590+0000", 2024-02-19T21:06:00.552 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0051778750000000002 2024-02-19T21:06:00.552 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.552 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.552 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.552 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.552 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.552 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:1070 94.1 94:877aab9c:test-rados-api-smithi163-83928-14::foo35:4 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:31.743616+0000", 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 28.727778968999999, 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0048200669999999999, 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1070 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.743616+0000", 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.553 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.554 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.554 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.554 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.743616+0000", 2024-02-19T21:06:00.554 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:06:00.554 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.554 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.554 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.554 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.743612+0000", 2024-02-19T21:06:00.554 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3190000000000001e-05 2024-02-19T21:06:00.554 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.554 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.554 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.554 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.743625+0000", 2024-02-19T21:06:00.554 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5513000000000001e-05 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.743651+0000", 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.201e-05 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.743663+0000", 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012386799999999999 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.743787+0000", 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000301365 2024-02-19T21:06:00.555 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.556 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.556 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:06:00.556 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.744088+0000", 2024-02-19T21:06:00.556 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0012381919999999999 2024-02-19T21:06:00.556 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.556 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.556 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.556 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.745326+0000", 2024-02-19T21:06:00.556 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4293000000000001e-05 2024-02-19T21:06:00.556 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.556 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.556 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:06:00.556 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.745411+0000", 2024-02-19T21:06:00.556 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00030168600000000001 2024-02-19T21:06:00.556 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.745712+0000", 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6591000000000001e-05 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.745769+0000", 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0026667919999999999 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.748436+0000", 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0048200669999999999 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.557 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.558 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.558 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.558 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22230.0:2 102.d 102:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e154)", 2024-02-19T21:06:00.558 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:34.185436+0000", 2024-02-19T21:06:00.558 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 26.285958337, 2024-02-19T21:06:00.558 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015189054, 2024-02-19T21:06:00.558 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.558 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:06:00.558 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.558 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22230", 2024-02-19T21:06:00.558 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1800034971", 2024-02-19T21:06:00.558 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:06:00.558 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.559 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.559 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.559 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.559 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185436+0000", 2024-02-19T21:06:00.559 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.559 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.559 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.559 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.559 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185436+0000", 2024-02-19T21:06:00.559 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:06:00.559 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.559 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.559 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.559 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185434+0000", 2024-02-19T21:06:00.559 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.024e-06 2024-02-19T21:06:00.560 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.560 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.560 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.560 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185441+0000", 2024-02-19T21:06:00.560 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6830000000000003e-06 2024-02-19T21:06:00.560 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.560 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.560 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.560 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185451+0000", 2024-02-19T21:06:00.560 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6630000000000004e-06 2024-02-19T21:06:00.560 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.560 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.560 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.560 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185458+0000", 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00056233499999999996 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.186020+0000", 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1280000000000008e-06 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.186028+0000", 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.012894070000000001 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.198922+0000", 2024-02-19T21:06:00.561 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012016800000000001 2024-02-19T21:06:00.562 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.562 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.562 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.562 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.199042+0000", 2024-02-19T21:06:00.562 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000105312 2024-02-19T21:06:00.562 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.562 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.562 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,6", 2024-02-19T21:06:00.562 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.199148+0000", 2024-02-19T21:06:00.562 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00055710799999999995 2024-02-19T21:06:00.562 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.562 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.562 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:06:00.562 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.199705+0000", 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7489999999999999e-05 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.199732+0000", 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00081238399999999998 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.200545+0000", 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8712999999999999e-05 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.200593+0000", 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1940999999999998e-05 2024-02-19T21:06:00.563 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.564 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.564 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.564 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.200625+0000", 2024-02-19T21:06:00.564 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015189054 2024-02-19T21:06:00.564 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.564 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.564 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.564 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.564 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.564 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22254.0:2 104.9 104:90ca3134:::oid_FlushAsyncPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e157)", 2024-02-19T21:06:00.564 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:37.200018+0000", 2024-02-19T21:06:00.564 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 23.271377115, 2024-02-19T21:06:00.564 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0068597529999999997, 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22254", 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2521895994", 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200018+0000", 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.565 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200018+0000", 2024-02-19T21:06:00.566 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:06:00.566 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.566 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.566 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.566 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200015+0000", 2024-02-19T21:06:00.566 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8359999999999992e-06 2024-02-19T21:06:00.566 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.566 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.566 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.566 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200023+0000", 2024-02-19T21:06:00.566 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9250000000000007e-06 2024-02-19T21:06:00.566 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.566 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.566 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.566 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200033+0000", 2024-02-19T21:06:00.567 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7489999999999998e-06 2024-02-19T21:06:00.567 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.567 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.567 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.567 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200041+0000", 2024-02-19T21:06:00.567 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015734399999999999 2024-02-19T21:06:00.567 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.567 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.567 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.567 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200198+0000", 2024-02-19T21:06:00.567 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8997000000000002e-05 2024-02-19T21:06:00.567 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.567 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.567 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.568 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200287+0000", 2024-02-19T21:06:00.568 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000121697 2024-02-19T21:06:00.568 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.568 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.568 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 5,6", 2024-02-19T21:06:00.568 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200409+0000", 2024-02-19T21:06:00.568 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0042214710000000001 2024-02-19T21:06:00.568 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.568 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.568 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:06:00.568 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.204630+0000", 2024-02-19T21:06:00.568 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000147804 2024-02-19T21:06:00.568 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.568 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.568 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.569 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.204778+0000", 2024-02-19T21:06:00.569 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020029409999999998 2024-02-19T21:06:00.569 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.569 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.569 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.569 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.206781+0000", 2024-02-19T21:06:00.569 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5789999999999999e-05 2024-02-19T21:06:00.569 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.569 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.569 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:06:00.569 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.206837+0000", 2024-02-19T21:06:00.569 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0491999999999998e-05 2024-02-19T21:06:00.569 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.569 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.569 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.570 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.206877+0000", 2024-02-19T21:06:00.570 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0068597529999999997 2024-02-19T21:06:00.570 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.570 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.570 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.570 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.570 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.570 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.47755.0:2 111.16 111:6eb9e51e:::oid_RoundTripWriteSamePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e166)", 2024-02-19T21:06:00.570 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:46.429271+0000", 2024-02-19T21:06:00.570 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 14.042123339, 2024-02-19T21:06:00.570 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0054490270000000004, 2024-02-19T21:06:00.570 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.570 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:06:00.570 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.570 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.47755", 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/4174416569", 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:46.429271+0000", 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:46.429271+0000", 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.571 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.572 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:46.429268+0000", 2024-02-19T21:06:00.572 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6039999999999999e-06 2024-02-19T21:06:00.572 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.572 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.572 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.572 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:46.429278+0000", 2024-02-19T21:06:00.572 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.505e-05 2024-02-19T21:06:00.572 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.572 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.572 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.572 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:46.429293+0000", 2024-02-19T21:06:00.572 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4040000000000001e-06 2024-02-19T21:06:00.572 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.572 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.572 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.573 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:46.429303+0000", 2024-02-19T21:06:00.573 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6998999999999998e-05 2024-02-19T21:06:00.573 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.573 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.573 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.573 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:46.429390+0000", 2024-02-19T21:06:00.573 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013020800000000001 2024-02-19T21:06:00.573 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.573 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.573 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.573 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:46.429520+0000", 2024-02-19T21:06:00.573 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000180106 2024-02-19T21:06:00.573 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.573 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.573 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,5", 2024-02-19T21:06:00.574 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:46.429700+0000", 2024-02-19T21:06:00.574 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002026651 2024-02-19T21:06:00.574 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.574 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.574 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.574 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:46.431726+0000", 2024-02-19T21:06:00.574 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0023703280000000001 2024-02-19T21:06:00.574 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.574 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.574 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:06:00.574 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:46.434097+0000", 2024-02-19T21:06:00.574 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00056533199999999999 2024-02-19T21:06:00.574 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.574 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.575 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.575 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:46.434662+0000", 2024-02-19T21:06:00.575 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3083000000000002e-05 2024-02-19T21:06:00.575 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.575 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.575 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:06:00.575 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:46.434695+0000", 2024-02-19T21:06:00.575 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5099000000000001e-05 2024-02-19T21:06:00.575 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.575 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.575 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.575 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:46.434720+0000", 2024-02-19T21:06:00.575 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0054490270000000004 2024-02-19T21:06:00.575 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.575 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.576 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.576 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.576 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.576 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.47779.0:2 114.13 114:cbb9903e:::test_obj:head [write-same] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-02-19T21:06:00.576 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:49.194644+0000", 2024-02-19T21:06:00.576 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 11.276750169, 2024-02-19T21:06:00.576 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.061315005999999998, 2024-02-19T21:06:00.576 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:06:00.576 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:06:00.576 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:06:00.576 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.47779", 2024-02-19T21:06:00.576 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/4153636191", 2024-02-19T21:06:00.576 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:06:00.576 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.577 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:06:00.577 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.577 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:06:00.577 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194644+0000", 2024-02-19T21:06:00.577 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:06:00.577 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.577 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.577 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:06:00.577 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194644+0000", 2024-02-19T21:06:00.577 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:06:00.577 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.577 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.577 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:06:00.577 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194643+0000", 2024-02-19T21:06:00.577 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8570000000000007e-06 2024-02-19T21:06:00.578 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.578 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.578 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:06:00.578 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194653+0000", 2024-02-19T21:06:00.578 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6340000000000007e-06 2024-02-19T21:06:00.578 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.578 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.578 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:06:00.578 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194661+0000", 2024-02-19T21:06:00.578 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7300000000000002e-06 2024-02-19T21:06:00.578 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.578 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.578 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:06:00.578 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194667+0000", 2024-02-19T21:06:00.578 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0865999999999998e-05 2024-02-19T21:06:00.579 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.579 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.579 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.579 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194708+0000", 2024-02-19T21:06:00.579 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3140000000000003e-06 2024-02-19T21:06:00.579 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.579 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.579 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:06:00.579 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194713+0000", 2024-02-19T21:06:00.579 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.020878119000000001 2024-02-19T21:06:00.579 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.579 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.579 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:06:00.579 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.215591+0000", 2024-02-19T21:06:00.579 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011031699999999999 2024-02-19T21:06:00.580 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.580 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.580 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:06:00.580 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.215702+0000", 2024-02-19T21:06:00.580 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000156618 2024-02-19T21:06:00.580 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.580 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.580 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,2", 2024-02-19T21:06:00.580 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.215858+0000", 2024-02-19T21:06:00.580 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0048208690000000002 2024-02-19T21:06:00.580 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.580 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.580 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:06:00.580 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.220679+0000", 2024-02-19T21:06:00.580 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.029705588000000002 2024-02-19T21:06:00.581 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.581 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.581 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.581 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.250385+0000", 2024-02-19T21:06:00.581 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0055153939999999999 2024-02-19T21:06:00.581 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.581 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.581 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:06:00.581 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.255900+0000", 2024-02-19T21:06:00.581 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1894999999999999e-05 2024-02-19T21:06:00.581 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.581 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.581 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:06:00.581 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.255932+0000", 2024-02-19T21:06:00.582 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7478e-05 2024-02-19T21:06:00.582 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:06:00.582 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:06:00.582 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:06:00.582 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.255959+0000", 2024-02-19T21:06:00.582 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.061315005999999998 2024-02-19T21:06:00.582 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.582 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.582 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.582 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:06:00.582 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:06:00.582 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:06:00.582 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:00.583 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:59.134367+0000 mgr.y (mgr.14550) 220 : cluster [DBG] pgmap v239: 298 pgs: 32 unknown, 1 clean+premerge+peered, 265 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:00.583 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:59.184716+0000 mon.a (mon.0) 1310 : audit [INF] from='client.48947 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi163-83952-15"}]': finished 2024-02-19T21:06:00.583 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:59.184896+0000 mon.a (mon.0) 1311 : audit [INF] from='client.49037 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi163-83845-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:00.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:59.184965+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-02-19T21:06:00.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:59.186872+0000 mon.c (mon.2) 178 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi163-83845-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:00.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:59.191474+0000 mon.b (mon.1) 280 : audit [INF] from='client.? v1:172.21.15.163:0/751814193' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi163-83920-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:00.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:59.191998+0000 mon.a (mon.0) 1313 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi163-83845-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:00.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:59.194686+0000 mon.a (mon.0) 1314 : audit [INF] from='client.47848 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi163-83920-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:00.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:59.217926+0000 mon.a (mon.0) 1315 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]: dispatch 2024-02-19T21:06:00.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:59.219935+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]: dispatch 2024-02-19T21:06:00.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:59.220238+0000 mon.a (mon.0) 1317 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:00.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:05:59.285171+0000 mon.c (mon.2) 179 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:00.584 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:05:59.740410+0000 mon.a (mon.0) 1318 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:06:00.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:59.134367+0000 mgr.y (mgr.14550) 220 : cluster [DBG] pgmap v239: 298 pgs: 32 unknown, 1 clean+premerge+peered, 265 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:00.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:59.184716+0000 mon.a (mon.0) 1310 : audit [INF] from='client.48947 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi163-83952-15"}]': finished 2024-02-19T21:06:00.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:59.184896+0000 mon.a (mon.0) 1311 : audit [INF] from='client.49037 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi163-83845-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:00.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:59.184965+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-02-19T21:06:00.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:59.186872+0000 mon.c (mon.2) 178 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi163-83845-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:00.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:59.191474+0000 mon.b (mon.1) 280 : audit [INF] from='client.? v1:172.21.15.163:0/751814193' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi163-83920-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:00.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:59.191998+0000 mon.a (mon.0) 1313 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi163-83845-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:00.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:59.194686+0000 mon.a (mon.0) 1314 : audit [INF] from='client.47848 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi163-83920-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:00.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:59.217926+0000 mon.a (mon.0) 1315 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]: dispatch 2024-02-19T21:06:00.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:59.219935+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]: dispatch 2024-02-19T21:06:00.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:59.220238+0000 mon.a (mon.0) 1317 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:00.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:05:59.285171+0000 mon.c (mon.2) 179 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:00.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:05:59.740410+0000 mon.a (mon.0) 1318 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:06:00.612 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:59.134367+0000 mgr.y (mgr.14550) 220 : cluster [DBG] pgmap v239: 298 pgs: 32 unknown, 1 clean+premerge+peered, 265 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:00.612 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:59.184716+0000 mon.a (mon.0) 1310 : audit [INF] from='client.48947 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi163-83952-15"}]': finished 2024-02-19T21:06:00.612 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:59.184896+0000 mon.a (mon.0) 1311 : audit [INF] from='client.49037 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi163-83845-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:00.612 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:59.184965+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-02-19T21:06:00.612 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:59.186872+0000 mon.c (mon.2) 178 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi163-83845-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:00.612 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:59.191474+0000 mon.b (mon.1) 280 : audit [INF] from='client.? v1:172.21.15.163:0/751814193' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi163-83920-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:00.612 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:59.191998+0000 mon.a (mon.0) 1313 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi163-83845-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:00.612 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:59.194686+0000 mon.a (mon.0) 1314 : audit [INF] from='client.47848 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi163-83920-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:00.612 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:59.217926+0000 mon.a (mon.0) 1315 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]: dispatch 2024-02-19T21:06:00.612 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:59.219935+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]: dispatch 2024-02-19T21:06:00.612 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:59.220238+0000 mon.a (mon.0) 1317 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:00.612 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:05:59.285171+0000 mon.c (mon.2) 179 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:00.612 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:05:59.740410+0000 mon.a (mon.0) 1318 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:06:00.613 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:00.790 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:00.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:00.887 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:06:00.887 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:06:00.888 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-19T21:06:00.888 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-19T21:06:00.894 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:00.959 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:01.290 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-19T21:06:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:00.150018+0000 mgr.y (mgr.14550) 221 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:00.285925+0000 mon.c (mon.2) 180 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:00.436006+0000 mon.a (mon.0) 1319 : audit [INF] from='client.47848 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi163-83920-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:00.436157+0000 mon.a (mon.0) 1320 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:00.436281+0000 mon.a (mon.0) 1321 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-02-19T21:06:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:00.437703+0000 mon.a (mon.0) 1322 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]: dispatch 2024-02-19T21:06:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:00.438977+0000 mon.b (mon.1) 281 : audit [INF] from='client.? v1:172.21.15.163:0/2577922117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi163-83854-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:00.440166+0000 mon.a (mon.0) 1323 : audit [INF] from='client.47863 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi163-83854-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:01.135323+0000 mon.b (mon.1) 282 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-19T21:06:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:01.136400+0000 mon.a (mon.0) 1324 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-19T21:06:01.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:01.191114+0000 mon.a (mon.0) 1325 : audit [INF] from='client.49037 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi163-83845-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi163-83845-26"}]': finished 2024-02-19T21:06:01.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.372+0000 7f7c77e59700 1 -- 172.21.15.163:0/996231710 >> v1:172.21.15.163:6790/0 conn(0x7f7c701042b0 legacy=0x7f7c70106730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.372+0000 7f7c77e59700 1 -- 172.21.15.163:0/996231710 shutdown_connections 2024-02-19T21:06:01.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.372+0000 7f7c77e59700 1 -- 172.21.15.163:0/996231710 >> 172.21.15.163:0/996231710 conn(0x7f7c700f3590 msgr2=0x7f7c700f59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:01.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.372+0000 7f7c77e59700 1 -- 172.21.15.163:0/996231710 shutdown_connections 2024-02-19T21:06:01.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.372+0000 7f7c77e59700 1 -- 172.21.15.163:0/996231710 wait complete. 2024-02-19T21:06:01.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.372+0000 7f7c77e59700 1 librados: starting msgr at 2024-02-19T21:06:01.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.372+0000 7f7c77e59700 1 librados: starting objecter 2024-02-19T21:06:01.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.372+0000 7f7c77e59700 1 Processor -- start 2024-02-19T21:06:01.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c77e59700 1 -- start start 2024-02-19T21:06:01.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c77e59700 1 librados: setting wanted keys 2024-02-19T21:06:01.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c77e59700 1 librados: calling monclient init 2024-02-19T21:06:01.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c77e59700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7c70117450 con 0x7f7c701042b0 2024-02-19T21:06:01.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c77e59700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7c70118630 con 0x7f7c70100770 2024-02-19T21:06:01.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c77e59700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7c70119830 con 0x7f7c700fd970 2024-02-19T21:06:01.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c763f6700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f7c701042b0 0x7f7c70115b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46680/0 (socket says 172.21.15.163:46680) 2024-02-19T21:06:01.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c763f6700 1 -- 172.21.15.163:0/987528065 learned_addr learned my addr 172.21.15.163:0/987528065 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:01.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c66ffd700 10 client.?.objecter ms_handle_connect 0x7f7c700fd970 2024-02-19T21:06:01.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c66ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:01.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c66ffd700 10 client.?.objecter ms_handle_connect 0x7f7c701042b0 2024-02-19T21:06:01.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c66ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:01.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c66ffd700 10 client.?.objecter ms_handle_connect 0x7f7c70100770 2024-02-19T21:06:01.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c66ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:01.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199177728 0 0) 0x7f7c70119830 con 0x7f7c700fd970 2024-02-19T21:06:01.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c5c002220 con 0x7f7c700fd970 2024-02-19T21:06:01.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657595025 0 0) 0x7f7c70118630 con 0x7f7c70100770 2024-02-19T21:06:01.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c70119830 con 0x7f7c70100770 2024-02-19T21:06:01.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1515660660 0 0) 0x7f7c5c002220 con 0x7f7c700fd970 2024-02-19T21:06:01.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c70118630 con 0x7f7c700fd970 2024-02-19T21:06:01.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7c6c001860 con 0x7f7c700fd970 2024-02-19T21:06:01.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3083513322 0 0) 0x7f7c70119830 con 0x7f7c70100770 2024-02-19T21:06:01.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.373+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c5c002220 con 0x7f7c70100770 2024-02-19T21:06:01.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.374+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7c600017f0 con 0x7f7c70100770 2024-02-19T21:06:01.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.374+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1088331238 0 0) 0x7f7c70118630 con 0x7f7c700fd970 2024-02-19T21:06:01.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.374+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 >> v1:172.21.15.193:6789/0 conn(0x7f7c70100770 legacy=0x7f7c700fa3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.374+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 >> v1:172.21.15.163:6789/0 conn(0x7f7c701042b0 legacy=0x7f7c70115b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.374+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7011aa30 con 0x7f7c700fd970 2024-02-19T21:06:01.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.374+0000 7f7c77e59700 1 -- 172.21.15.163:0/987528065 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c70117640 con 0x7f7c700fd970 2024-02-19T21:06:01.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.374+0000 7f7c77e59700 10 client.49052.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:01.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.374+0000 7f7c77e59700 1 -- 172.21.15.163:0/987528065 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c70117b30 con 0x7f7c700fd970 2024-02-19T21:06:01.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.374+0000 7f7c77e59700 1 librados: init done 2024-02-19T21:06:01.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.374+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f7c6c001ff0 con 0x7f7c700fd970 2024-02-19T21:06:01.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.374+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7c6c0033e0 con 0x7f7c700fd970 2024-02-19T21:06:01.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.375+0000 7f7c77e59700 1 -- 172.21.15.163:0/987528065 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c54004f20 con 0x7f7c700fd970 2024-02-19T21:06:01.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.375+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f7c6c001b50 con 0x7f7c700fd970 2024-02-19T21:06:01.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.376+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(181..181 src has 43..181) v4 ==== 9807+0+0 (unknown 3930536000 0 0) 0x7f7c6c076b40 con 0x7f7c700fd970 2024-02-19T21:06:01.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.376+0000 7f7c66ffd700 10 client.49052.objecter ms_dispatch 0x7f7c70001550 osd_map(181..181 src has 43..181) v4 2024-02-19T21:06:01.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.376+0000 7f7c66ffd700 3 client.49052.objecter handle_osd_map got epochs [181,181] > 0 2024-02-19T21:06:01.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.376+0000 7f7c66ffd700 3 client.49052.objecter handle_osd_map decoding full epoch 181 2024-02-19T21:06:01.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.376+0000 7f7c66ffd700 20 client.49052.objecter dump_active .. 0 homeless 2024-02-19T21:06:01.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.376+0000 7f7c66ffd700 10 client.49052.objecter ms_handle_connect 0x7f7c5c063c00 2024-02-19T21:06:01.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.380+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f7c6c049860 con 0x7f7c700fd970 2024-02-19T21:06:01.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.574+0000 7f7c77e59700 1 -- 172.21.15.163:0/987528065 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f7c540051d0 con 0x7f7c700fd970 2024-02-19T21:06:01.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:00.150018+0000 mgr.y (mgr.14550) 221 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:01.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:00.285925+0000 mon.c (mon.2) 180 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:01.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:00.436006+0000 mon.a (mon.0) 1319 : audit [INF] from='client.47848 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi163-83920-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:01.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:00.436157+0000 mon.a (mon.0) 1320 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:01.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:00.436281+0000 mon.a (mon.0) 1321 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-02-19T21:06:01.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:00.437703+0000 mon.a (mon.0) 1322 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]: dispatch 2024-02-19T21:06:01.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:00.438977+0000 mon.b (mon.1) 281 : audit [INF] from='client.? v1:172.21.15.163:0/2577922117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi163-83854-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:01.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:00.440166+0000 mon.a (mon.0) 1323 : audit [INF] from='client.47863 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi163-83854-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:01.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:01.135323+0000 mon.b (mon.1) 282 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-19T21:06:01.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:01.136400+0000 mon.a (mon.0) 1324 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-19T21:06:01.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:01.191114+0000 mon.a (mon.0) 1325 : audit [INF] from='client.49037 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi163-83845-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi163-83845-26"}]': finished 2024-02-19T21:06:01.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:00.150018+0000 mgr.y (mgr.14550) 221 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:01.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:00.285925+0000 mon.c (mon.2) 180 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:01.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:00.436006+0000 mon.a (mon.0) 1319 : audit [INF] from='client.47848 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi163-83920-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:01.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:00.436157+0000 mon.a (mon.0) 1320 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:01.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:00.436281+0000 mon.a (mon.0) 1321 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-02-19T21:06:01.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:00.437703+0000 mon.a (mon.0) 1322 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]: dispatch 2024-02-19T21:06:01.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:00.438977+0000 mon.b (mon.1) 281 : audit [INF] from='client.? v1:172.21.15.163:0/2577922117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi163-83854-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:01.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:00.440166+0000 mon.a (mon.0) 1323 : audit [INF] from='client.47863 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi163-83854-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:01.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:01.135323+0000 mon.b (mon.1) 282 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-19T21:06:01.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:01.136400+0000 mon.a (mon.0) 1324 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-19T21:06:01.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:01.191114+0000 mon.a (mon.0) 1325 : audit [INF] from='client.49037 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi163-83845-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi163-83845-26"}]': finished 2024-02-19T21:06:01.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.777+0000 7f112ca1b700 1 -- 172.21.15.163:0/559767541 >> v1:172.21.15.163:6790/0 conn(0x7f11280fc870 legacy=0x7f11280fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.777+0000 7f112ca1b700 1 -- 172.21.15.163:0/559767541 shutdown_connections 2024-02-19T21:06:01.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.777+0000 7f112ca1b700 1 -- 172.21.15.163:0/559767541 >> 172.21.15.163:0/559767541 conn(0x7f11280f2470 msgr2=0x7f11280f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:01.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.777+0000 7f112ca1b700 1 -- 172.21.15.163:0/559767541 shutdown_connections 2024-02-19T21:06:01.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.777+0000 7f112ca1b700 1 -- 172.21.15.163:0/559767541 wait complete. 2024-02-19T21:06:01.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.778+0000 7f112ca1b700 1 librados: starting msgr at 2024-02-19T21:06:01.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.778+0000 7f112ca1b700 1 librados: starting objecter 2024-02-19T21:06:01.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.778+0000 7f112ca1b700 1 Processor -- start 2024-02-19T21:06:01.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.778+0000 7f112ca1b700 1 -- start start 2024-02-19T21:06:01.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.778+0000 7f112ca1b700 1 librados: setting wanted keys 2024-02-19T21:06:01.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.778+0000 7f112ca1b700 1 librados: calling monclient init 2024-02-19T21:06:01.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.778+0000 7f112ca1b700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1128117350 con 0x7f11280ff670 2024-02-19T21:06:01.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.778+0000 7f112ca1b700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1128118530 con 0x7f11281031b0 2024-02-19T21:06:01.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.778+0000 7f112ca1b700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1128119730 con 0x7f11280fc870 2024-02-19T21:06:01.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.778+0000 7f112659c700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f11280fc870 0x7f11280f9ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:49484/0 (socket says 172.21.15.163:49484) 2024-02-19T21:06:01.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.778+0000 7f112659c700 1 -- 172.21.15.163:0/797099915 learned_addr learned my addr 172.21.15.163:0/797099915 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:01.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.778+0000 7f11177fe700 10 client.?.objecter ms_handle_connect 0x7f11280fc870 2024-02-19T21:06:01.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.778+0000 7f11177fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:01.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.778+0000 7f11177fe700 10 client.?.objecter ms_handle_connect 0x7f11280ff670 2024-02-19T21:06:01.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.778+0000 7f11177fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:01.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 10 client.?.objecter ms_handle_connect 0x7f11281031b0 2024-02-19T21:06:01.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:01.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737238254 0 0) 0x7f1128119730 con 0x7f11280fc870 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1110003030 con 0x7f11280fc870 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077717597 0 0) 0x7f1128117350 con 0x7f11280ff670 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1128119730 con 0x7f11280ff670 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302863192 0 0) 0x7f1128118530 con 0x7f11281031b0 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1128117350 con 0x7f11281031b0 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3379431860 0 0) 0x7f1110003030 con 0x7f11280fc870 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1128118530 con 0x7f11280fc870 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f111c0015d0 con 0x7f11280fc870 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3396096369 0 0) 0x7f1128119730 con 0x7f11280ff670 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1110003030 con 0x7f11280ff670 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f11180018c0 con 0x7f11280ff670 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2365682044 0 0) 0x7f1128117350 con 0x7f11281031b0 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1128119730 con 0x7f11281031b0 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1120001a20 con 0x7f11281031b0 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3279331442 0 0) 0x7f1110003030 con 0x7f11280ff670 2024-02-19T21:06:01.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 >> v1:172.21.15.163:6790/0 conn(0x7f11280fc870 legacy=0x7f11280f9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 >> v1:172.21.15.193:6789/0 conn(0x7f11281031b0 legacy=0x7f1128115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112811a930 con 0x7f11280ff670 2024-02-19T21:06:01.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1128117540 con 0x7f11280ff670 2024-02-19T21:06:01.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f112ca1b700 10 client.22422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:01.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11281179b0 con 0x7f11280ff670 2024-02-19T21:06:01.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f112ca1b700 1 librados: init done 2024-02-19T21:06:01.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f1118001f50 con 0x7f11280ff670 2024-02-19T21:06:01.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.779+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1118003220 con 0x7f11280ff670 2024-02-19T21:06:01.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.780+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1108004f00 con 0x7f11280ff670 2024-02-19T21:06:01.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.781+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f1118001ab0 con 0x7f11280ff670 2024-02-19T21:06:01.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.781+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(181..181 src has 43..181) v4 ==== 9807+0+0 (unknown 3930536000 0 0) 0x7f11180769b0 con 0x7f11280ff670 2024-02-19T21:06:01.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.781+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 osd_map(181..181 src has 43..181) v4 2024-02-19T21:06:01.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.781+0000 7f11177fe700 3 client.22422.objecter handle_osd_map got epochs [181,181] > 0 2024-02-19T21:06:01.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.781+0000 7f11177fe700 3 client.22422.objecter handle_osd_map decoding full epoch 181 2024-02-19T21:06:01.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.781+0000 7f11177fe700 20 client.22422.objecter dump_active .. 0 homeless 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.781+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v181) v1 ==== 54+0+16 (unknown 3399549991 0 2420000192) 0x7f1118077060 con 0x7f11280ff670 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.782+0000 7f11177fe700 10 client.22422.objecter ms_handle_connect 0x7f1110068150 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.782+0000 7f112ca1b700 10 client.22422.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.782+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 1 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.782+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f11080051a0 osd=0 3 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.782+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f11080051a0 osd=0 3 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.782+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f11080051a0 osd=0 3 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.782+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 1 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.782+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f11080051a0 osd=0 3 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.782+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 0 1 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.782+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f11080051a0 osd=0 4 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.782+0000 7f112ca1b700 10 client.22422.objecter _send_command 1 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.782+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1108004950 con 0x7f1108004530 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.782+0000 7f11177fe700 10 client.22422.objecter ms_handle_connect 0x7f1108004530 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.783+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1108004950 con 0x7f1108004530 2024-02-19T21:06:01.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.783+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:06:01.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.783+0000 7f11177fe700 10 client.22422.objecter _finish_command 1 = system:0 2024-02-19T21:06:01.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.783+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f11080051a0 osd=0 4 2024-02-19T21:06:01.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.783+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 0 1 2024-02-19T21:06:01.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.797+0000 7f112ca1b700 10 client.22422.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:01.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.797+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 2 2024-02-19T21:06:01.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.797+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f11080051a0 osd=0 3 2024-02-19T21:06:01.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.797+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f11080051a0 osd=0 3 2024-02-19T21:06:01.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.797+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f11080051a0 osd=0 3 2024-02-19T21:06:01.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.797+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 2 2024-02-19T21:06:01.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.797+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f11080051a0 osd=0 3 2024-02-19T21:06:01.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.797+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 0 2 2024-02-19T21:06:01.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.797+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f11080051a0 osd=0 4 2024-02-19T21:06:01.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.797+0000 7f112ca1b700 10 client.22422.objecter _send_command 2 2024-02-19T21:06:01.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.797+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1108004d30 con 0x7f1108004530 2024-02-19T21:06:01.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1108004d30 con 0x7f1108004530 2024-02-19T21:06:01.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:01.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f11177fe700 10 client.22422.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f11080051a0 osd=0 4 2024-02-19T21:06:01.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 0 2 2024-02-19T21:06:01.795 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:06:01.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f112ca1b700 10 client.22422.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:01.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 3 2024-02-19T21:06:01.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f1108004270 osd=1 3 2024-02-19T21:06:01.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f1108004270 osd=1 3 2024-02-19T21:06:01.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f1108004270 osd=1 3 2024-02-19T21:06:01.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 3 2024-02-19T21:06:01.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f1108004270 osd=1 3 2024-02-19T21:06:01.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 1 3 2024-02-19T21:06:01.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f1108004270 osd=1 4 2024-02-19T21:06:01.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f112ca1b700 10 client.22422.objecter _send_command 3 2024-02-19T21:06:01.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f110800bf60 con 0x7f1108003b40 2024-02-19T21:06:01.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.799+0000 7f11177fe700 10 client.22422.objecter ms_handle_connect 0x7f1108003b40 2024-02-19T21:06:01.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.800+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f110800bf60 con 0x7f1108003b40 2024-02-19T21:06:01.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.800+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:06:01.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.800+0000 7f11177fe700 10 client.22422.objecter _finish_command 3 = system:0 2024-02-19T21:06:01.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.800+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f1108004270 osd=1 4 2024-02-19T21:06:01.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.800+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 1 3 2024-02-19T21:06:01.808 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:01.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.816+0000 7f112ca1b700 10 client.22422.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:01.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.816+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 4 2024-02-19T21:06:01.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.816+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f1108004270 osd=1 3 2024-02-19T21:06:01.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.816+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f1108004270 osd=1 3 2024-02-19T21:06:01.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.816+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f1108004270 osd=1 3 2024-02-19T21:06:01.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.816+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 4 2024-02-19T21:06:01.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.816+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f1108004270 osd=1 3 2024-02-19T21:06:01.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.816+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 1 4 2024-02-19T21:06:01.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.816+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f1108004270 osd=1 4 2024-02-19T21:06:01.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.816+0000 7f112ca1b700 10 client.22422.objecter _send_command 4 2024-02-19T21:06:01.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.816+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f110800d280 con 0x7f1108003b40 2024-02-19T21:06:01.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f110800d280 con 0x7f1108003b40 2024-02-19T21:06:01.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:01.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f11177fe700 10 client.22422.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f1108004270 osd=1 4 2024-02-19T21:06:01.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 1 4 2024-02-19T21:06:01.813 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:06:01.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f112ca1b700 10 client.22422.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:01.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 5 2024-02-19T21:06:01.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f11080033a0 osd=2 3 2024-02-19T21:06:01.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f11080033a0 osd=2 3 2024-02-19T21:06:01.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f11080033a0 osd=2 3 2024-02-19T21:06:01.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 5 2024-02-19T21:06:01.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f11080033a0 osd=2 3 2024-02-19T21:06:01.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 2 5 2024-02-19T21:06:01.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f11080033a0 osd=2 4 2024-02-19T21:06:01.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f112ca1b700 10 client.22422.objecter _send_command 5 2024-02-19T21:06:01.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1108010d30 con 0x7f1108002070 2024-02-19T21:06:01.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.818+0000 7f11177fe700 10 client.22422.objecter ms_handle_connect 0x7f1108002070 2024-02-19T21:06:01.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.819+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1108010d30 con 0x7f1108002070 2024-02-19T21:06:01.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.819+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:06:01.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.819+0000 7f11177fe700 10 client.22422.objecter _finish_command 5 = system:0 2024-02-19T21:06:01.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.819+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f11080033a0 osd=2 4 2024-02-19T21:06:01.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.819+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 2 5 2024-02-19T21:06:01.817 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:01.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:01.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.835+0000 7f112ca1b700 10 client.22422.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:01.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.835+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 6 2024-02-19T21:06:01.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.835+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f11080033a0 osd=2 3 2024-02-19T21:06:01.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.835+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f11080033a0 osd=2 3 2024-02-19T21:06:01.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.835+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f11080033a0 osd=2 3 2024-02-19T21:06:01.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.835+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 6 2024-02-19T21:06:01.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.835+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f11080033a0 osd=2 3 2024-02-19T21:06:01.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.835+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 2 6 2024-02-19T21:06:01.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.835+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f11080033a0 osd=2 4 2024-02-19T21:06:01.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.835+0000 7f112ca1b700 10 client.22422.objecter _send_command 6 2024-02-19T21:06:01.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.835+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1108001630 con 0x7f1108002070 2024-02-19T21:06:01.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1108001630 con 0x7f1108002070 2024-02-19T21:06:01.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:01.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f11177fe700 10 client.22422.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f11080033a0 osd=2 4 2024-02-19T21:06:01.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 2 6 2024-02-19T21:06:01.832 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:06:01.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f112ca1b700 10 client.22422.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:01.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 7 2024-02-19T21:06:01.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f110800c8d0 osd=3 3 2024-02-19T21:06:01.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f110800c8d0 osd=3 3 2024-02-19T21:06:01.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f110800c8d0 osd=3 3 2024-02-19T21:06:01.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 7 2024-02-19T21:06:01.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f110800c8d0 osd=3 3 2024-02-19T21:06:01.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 3 7 2024-02-19T21:06:01.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f110800c8d0 osd=3 4 2024-02-19T21:06:01.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f112ca1b700 10 client.22422.objecter _send_command 7 2024-02-19T21:06:01.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f110800d0b0 con 0x7f1108010ed0 2024-02-19T21:06:01.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.837+0000 7f11177fe700 10 client.22422.objecter ms_handle_connect 0x7f1108010ed0 2024-02-19T21:06:01.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.838+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f110800d0b0 con 0x7f1108010ed0 2024-02-19T21:06:01.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.838+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:06:01.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.838+0000 7f11177fe700 10 client.22422.objecter _finish_command 7 = system:0 2024-02-19T21:06:01.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.838+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f110800c8d0 osd=3 4 2024-02-19T21:06:01.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.838+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 3 7 2024-02-19T21:06:01.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.856+0000 7f112ca1b700 10 client.22422.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:01.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.856+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 8 2024-02-19T21:06:01.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.856+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f110800c8d0 osd=3 3 2024-02-19T21:06:01.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.856+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f110800c8d0 osd=3 3 2024-02-19T21:06:01.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.856+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f110800c8d0 osd=3 3 2024-02-19T21:06:01.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.856+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 8 2024-02-19T21:06:01.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.856+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f110800c8d0 osd=3 3 2024-02-19T21:06:01.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.856+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 3 8 2024-02-19T21:06:01.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.856+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f110800c8d0 osd=3 4 2024-02-19T21:06:01.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.856+0000 7f112ca1b700 10 client.22422.objecter _send_command 8 2024-02-19T21:06:01.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.856+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1108012260 con 0x7f1108010ed0 2024-02-19T21:06:01.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1108012260 con 0x7f1108010ed0 2024-02-19T21:06:01.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:01.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f11177fe700 10 client.22422.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f110800c8d0 osd=3 4 2024-02-19T21:06:01.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 3 8 2024-02-19T21:06:01.853 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:06:01.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f112ca1b700 10 client.22422.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:01.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 9 2024-02-19T21:06:01.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f1108012260 osd=4 3 2024-02-19T21:06:01.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f1108012260 osd=4 3 2024-02-19T21:06:01.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f1108012260 osd=4 3 2024-02-19T21:06:01.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 9 2024-02-19T21:06:01.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f1108012260 osd=4 3 2024-02-19T21:06:01.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 4 9 2024-02-19T21:06:01.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f1108012260 osd=4 4 2024-02-19T21:06:01.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f112ca1b700 10 client.22422.objecter _send_command 9 2024-02-19T21:06:01.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.858+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f110801a2a0 con 0x7f11080166f0 2024-02-19T21:06:01.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.859+0000 7f11177fe700 10 client.22422.objecter ms_handle_connect 0x7f11080166f0 2024-02-19T21:06:01.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.860+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f110801a2a0 con 0x7f11080166f0 2024-02-19T21:06:01.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.860+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:06:01.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.860+0000 7f11177fe700 10 client.22422.objecter _finish_command 9 = system:0 2024-02-19T21:06:01.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.860+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f1108012260 osd=4 4 2024-02-19T21:06:01.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.860+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 4 9 2024-02-19T21:06:01.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.875+0000 7f112ca1b700 10 client.22422.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:01.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.875+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 10 2024-02-19T21:06:01.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.875+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f1108012260 osd=4 3 2024-02-19T21:06:01.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.875+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f1108012260 osd=4 3 2024-02-19T21:06:01.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.875+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f1108012260 osd=4 3 2024-02-19T21:06:01.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.875+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 10 2024-02-19T21:06:01.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.875+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f1108012260 osd=4 3 2024-02-19T21:06:01.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.875+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 4 10 2024-02-19T21:06:01.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.875+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f1108012260 osd=4 4 2024-02-19T21:06:01.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.875+0000 7f112ca1b700 10 client.22422.objecter _send_command 10 2024-02-19T21:06:01.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.875+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f110801a2a0 con 0x7f11080166f0 2024-02-19T21:06:01.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.877+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f110801a2a0 con 0x7f11080166f0 2024-02-19T21:06:01.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.877+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:01.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.877+0000 7f11177fe700 10 client.22422.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.877+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f1108012260 osd=4 4 2024-02-19T21:06:01.872 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.877+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 4 10 2024-02-19T21:06:01.873 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:06:01.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.878+0000 7f112ca1b700 10 client.22422.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:01.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.878+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 11 2024-02-19T21:06:01.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.878+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f110801a2a0 osd=5 3 2024-02-19T21:06:01.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.878+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f110801a2a0 osd=5 3 2024-02-19T21:06:01.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.878+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f110801a2a0 osd=5 3 2024-02-19T21:06:01.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.878+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 11 2024-02-19T21:06:01.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.878+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f110801a2a0 osd=5 3 2024-02-19T21:06:01.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.878+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 5 11 2024-02-19T21:06:01.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.878+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f110801a2a0 osd=5 4 2024-02-19T21:06:01.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.878+0000 7f112ca1b700 10 client.22422.objecter _send_command 11 2024-02-19T21:06:01.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.878+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f110801b4f0 con 0x7f110801bfa0 2024-02-19T21:06:01.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.878+0000 7f11177fe700 10 client.22422.objecter ms_handle_connect 0x7f110801bfa0 2024-02-19T21:06:01.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.879+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f110801b4f0 con 0x7f110801bfa0 2024-02-19T21:06:01.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.879+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:06:01.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.879+0000 7f11177fe700 10 client.22422.objecter _finish_command 11 = system:0 2024-02-19T21:06:01.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.879+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f110801a2a0 osd=5 4 2024-02-19T21:06:01.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.879+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 5 11 2024-02-19T21:06:01.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.893+0000 7f112ca1b700 10 client.22422.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:01.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.893+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 12 2024-02-19T21:06:01.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.893+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f110801a2a0 osd=5 3 2024-02-19T21:06:01.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.893+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f110801a2a0 osd=5 3 2024-02-19T21:06:01.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.893+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f110801a2a0 osd=5 3 2024-02-19T21:06:01.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.893+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 12 2024-02-19T21:06:01.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.893+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f110801a2a0 osd=5 3 2024-02-19T21:06:01.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.893+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 5 12 2024-02-19T21:06:01.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.893+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f110801a2a0 osd=5 4 2024-02-19T21:06:01.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.893+0000 7f112ca1b700 10 client.22422.objecter _send_command 12 2024-02-19T21:06:01.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.893+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f110801b4f0 con 0x7f110801bfa0 2024-02-19T21:06:01.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f110801b4f0 con 0x7f110801bfa0 2024-02-19T21:06:01.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:01.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f11177fe700 10 client.22422.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f110801a2a0 osd=5 4 2024-02-19T21:06:01.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 5 12 2024-02-19T21:06:01.891 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:06:01.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f112ca1b700 10 client.22422.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:01.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 13 2024-02-19T21:06:01.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f110801b4f0 osd=6 3 2024-02-19T21:06:01.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f110801b4f0 osd=6 3 2024-02-19T21:06:01.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f110801b4f0 osd=6 3 2024-02-19T21:06:01.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 13 2024-02-19T21:06:01.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f110801b4f0 osd=6 3 2024-02-19T21:06:01.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 6 13 2024-02-19T21:06:01.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f110801b4f0 osd=6 4 2024-02-19T21:06:01.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f112ca1b700 10 client.22422.objecter _send_command 13 2024-02-19T21:06:01.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.895+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1108022c70 con 0x7f110801f160 2024-02-19T21:06:01.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.896+0000 7f11177fe700 10 client.22422.objecter ms_handle_connect 0x7f110801f160 2024-02-19T21:06:01.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.897+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1108022c70 con 0x7f110801f160 2024-02-19T21:06:01.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.897+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:06:01.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.897+0000 7f11177fe700 10 client.22422.objecter _finish_command 13 = system:0 2024-02-19T21:06:01.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.897+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f110801b4f0 osd=6 4 2024-02-19T21:06:01.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.897+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 6 13 2024-02-19T21:06:01.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.911+0000 7f112ca1b700 10 client.22422.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.911+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 14 2024-02-19T21:06:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.911+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f110801b4f0 osd=6 3 2024-02-19T21:06:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.911+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f110801b4f0 osd=6 3 2024-02-19T21:06:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.911+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f110801b4f0 osd=6 3 2024-02-19T21:06:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.911+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 14 2024-02-19T21:06:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.911+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f110801b4f0 osd=6 3 2024-02-19T21:06:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.911+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 6 14 2024-02-19T21:06:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.911+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f110801b4f0 osd=6 4 2024-02-19T21:06:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.911+0000 7f112ca1b700 10 client.22422.objecter _send_command 14 2024-02-19T21:06:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.911+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1108022c70 con 0x7f110801f160 2024-02-19T21:06:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1108022c70 con 0x7f110801f160 2024-02-19T21:06:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f11177fe700 10 client.22422.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f110801b4f0 osd=6 4 2024-02-19T21:06:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 6 14 2024-02-19T21:06:01.908 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:06:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f112ca1b700 10 client.22422.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 15 2024-02-19T21:06:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f1108022c70 osd=7 3 2024-02-19T21:06:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f1108022c70 osd=7 3 2024-02-19T21:06:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f1108022c70 osd=7 3 2024-02-19T21:06:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 15 2024-02-19T21:06:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f1108022c70 osd=7 3 2024-02-19T21:06:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 7 15 2024-02-19T21:06:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f1108022c70 osd=7 4 2024-02-19T21:06:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f112ca1b700 10 client.22422.objecter _send_command 15 2024-02-19T21:06:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.913+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1108026f80 con 0x7f11080232f0 2024-02-19T21:06:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.914+0000 7f11177fe700 10 client.22422.objecter ms_handle_connect 0x7f11080232f0 2024-02-19T21:06:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.914+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1108026f80 con 0x7f11080232f0 2024-02-19T21:06:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.914+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:06:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.914+0000 7f11177fe700 10 client.22422.objecter _finish_command 15 = system:0 2024-02-19T21:06:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.914+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f1108022c70 osd=7 4 2024-02-19T21:06:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.914+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 7 15 2024-02-19T21:06:01.922 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:01.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.929+0000 7f112ca1b700 10 client.22422.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:01.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.929+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign -1 16 2024-02-19T21:06:01.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.929+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f1108022c70 osd=7 3 2024-02-19T21:06:01.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.929+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f1108022c70 osd=7 3 2024-02-19T21:06:01.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.929+0000 7f112ca1b700 20 client.22422.objecter _get_session s=0x7f1108022c70 osd=7 3 2024-02-19T21:06:01.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.929+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_remove -1 16 2024-02-19T21:06:01.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.929+0000 7f112ca1b700 20 client.22422.objecter get_session s=0x7f1108022c70 osd=7 3 2024-02-19T21:06:01.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.929+0000 7f112ca1b700 15 client.22422.objecter _session_command_op_assign 7 16 2024-02-19T21:06:01.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.929+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f1108022c70 osd=7 4 2024-02-19T21:06:01.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.929+0000 7f112ca1b700 10 client.22422.objecter _send_command 16 2024-02-19T21:06:01.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.929+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f1108028730 con 0x7f11080232f0 2024-02-19T21:06:01.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.930+0000 7f11177fe700 1 -- 172.21.15.163:0/797099915 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f1108028730 con 0x7f11080232f0 2024-02-19T21:06:01.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.930+0000 7f11177fe700 10 client.22422.objecter ms_dispatch 0x7f1128001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:01.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.930+0000 7f11177fe700 10 client.22422.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.930+0000 7f11177fe700 20 client.22422.objecter put_session s=0x7f1108022c70 osd=7 4 2024-02-19T21:06:01.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.930+0000 7f11177fe700 15 client.22422.objecter _session_command_op_remove 7 16 2024-02-19T21:06:01.926 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:06:01.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 10 librados: watch_flush enter 2024-02-19T21:06:01.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 10 librados: watch_flush exit 2024-02-19T21:06:01.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 10 client.22422.objecter close_session for osd.0 2024-02-19T21:06:01.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 >> v1:172.21.15.163:6801/1559418115 conn(0x7f1108004530 legacy=0x7f1108002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f11080051a0 osd=0 1 2024-02-19T21:06:01.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 10 client.22422.objecter close_session for osd.1 2024-02-19T21:06:01.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 >> v1:172.21.15.163:6805/3125047432 conn(0x7f1108003b40 legacy=0x7f110800a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f1108004270 osd=1 1 2024-02-19T21:06:01.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 10 client.22422.objecter close_session for osd.2 2024-02-19T21:06:01.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 >> v1:172.21.15.163:6809/1563158487 conn(0x7f1108002070 legacy=0x7f110800f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f11080033a0 osd=2 1 2024-02-19T21:06:01.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 10 client.22422.objecter close_session for osd.3 2024-02-19T21:06:01.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 >> v1:172.21.15.163:6813/2112279769 conn(0x7f1108010ed0 legacy=0x7f1108011310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f110800c8d0 osd=3 1 2024-02-19T21:06:01.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 10 client.22422.objecter close_session for osd.4 2024-02-19T21:06:01.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 >> v1:172.21.15.193:6800/3129716965 conn(0x7f11080166f0 legacy=0x7f1108018ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f1126d9d700 1 -- 172.21.15.163:0/797099915 reap_dead start 2024-02-19T21:06:01.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f1126d9d700 1 -- 172.21.15.163:0/797099915 reap_dead start 2024-02-19T21:06:01.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f1108012260 osd=4 1 2024-02-19T21:06:01.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 10 client.22422.objecter close_session for osd.5 2024-02-19T21:06:01.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 >> v1:172.21.15.193:6804/805161991 conn(0x7f110801bfa0 legacy=0x7f110801c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f110801a2a0 osd=5 1 2024-02-19T21:06:01.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 10 client.22422.objecter close_session for osd.6 2024-02-19T21:06:01.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 >> v1:172.21.15.193:6808/787604615 conn(0x7f110801f160 legacy=0x7f1108021550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f110801b4f0 osd=6 1 2024-02-19T21:06:01.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 10 client.22422.objecter close_session for osd.7 2024-02-19T21:06:01.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 >> v1:172.21.15.193:6812/157927432 conn(0x7f11080232f0 legacy=0x7f1108025700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 20 client.22422.objecter put_session s=0x7f1108022c70 osd=7 1 2024-02-19T21:06:01.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 20 client.22422.objecter shutdown clearing up homeless session... 2024-02-19T21:06:01.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 10 client.22422.objecter successfully canceled tick 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 >> v1:172.21.15.163:6800/1552514681 conn(0x7f1110068150 legacy=0x7f111006a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 >> v1:172.21.15.163:6789/0 conn(0x7f11280ff670 legacy=0x7f11280fa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f1126d9d700 1 -- 172.21.15.163:0/797099915 reap_dead start 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 shutdown_connections 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 >> 172.21.15.163:0/797099915 conn(0x7f11280f2470 msgr2=0x7f11280611e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 shutdown_connections 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 1 -- 172.21.15.163:0/797099915 wait complete. 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:01.931+0000 7f112ca1b700 1 librados: shutdown 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.930 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:06:01.957 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:01.979 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:01.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:01.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:02.083 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:06:02.113 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:02.133 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:06:02.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:02.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:02.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:02.197+0000 7f7c66ffd700 1 -- 172.21.15.163:0/987528065 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v182) v1 ==== 91+0+0 (unknown 2487514314 0 0) 0x7f7c6c047c70 con 0x7f7c700fd970 2024-02-19T21:06:02.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:02.200+0000 7f7c77e59700 10 librados: watch_flush enter 2024-02-19T21:06:02.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:02.204+0000 7f7c77e59700 10 librados: watch_flush exit 2024-02-19T21:06:02.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:02.204+0000 7f7c77e59700 20 client.49052.objecter shutdown clearing up homeless session... 2024-02-19T21:06:02.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:02.204+0000 7f7c77e59700 10 client.49052.objecter successfully canceled tick 2024-02-19T21:06:02.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:02.205+0000 7f7c77e59700 1 -- 172.21.15.163:0/987528065 >> v1:172.21.15.163:6800/1552514681 conn(0x7f7c5c063c00 legacy=0x7f7c5c0660c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:02.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:02.205+0000 7f7c77e59700 1 -- 172.21.15.163:0/987528065 >> v1:172.21.15.163:6790/0 conn(0x7f7c700fd970 legacy=0x7f7c700f9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:02.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:02.207+0000 7f7c77e59700 1 -- 172.21.15.163:0/987528065 shutdown_connections 2024-02-19T21:06:02.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:02.207+0000 7f7c77e59700 1 -- 172.21.15.163:0/987528065 >> 172.21.15.163:0/987528065 conn(0x7f7c700f3590 msgr2=0x7f7c70107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:02.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:02.208+0000 7f7c77e59700 1 -- 172.21.15.163:0/987528065 shutdown_connections 2024-02-19T21:06:02.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:02.208+0000 7f7c77e59700 1 -- 172.21.15.163:0/987528065 wait complete. 2024-02-19T21:06:02.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:02.210+0000 7f7c77e59700 1 librados: shutdown 2024-02-19T21:06:02.216 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:06:02.237 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:02.266 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:02.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:01.134796+0000 mgr.y (mgr.14550) 222 : cluster [DBG] pgmap v242: 329 pgs: 32 unknown, 32 creating+peering, 1 peering, 264 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:06:02.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:01.286658+0000 mon.c (mon.2) 181 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:02.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:01.441672+0000 mon.a (mon.0) 1326 : audit [INF] from='client.47863 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi163-83854-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:02.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:01.441931+0000 mon.a (mon.0) 1327 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-19T21:06:02.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:01.442023+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-02-19T21:06:02.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:01.576070+0000 mon.c (mon.2) 182 : audit [INF] from='client.? v1:172.21.15.163:0/987528065' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:06:02.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:01.714690+0000 mon.a (mon.0) 1329 : audit [INF] from='client.49052 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:06:02.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:01.782215+0000 mon.a (mon.0) 1330 : audit [DBG] from='client.? v1:172.21.15.163:0/797099915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:02.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:02.188910+0000 mon.a (mon.0) 1331 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:06:02.492 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:02.195100+0000 mon.a (mon.0) 1332 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]': finished 2024-02-19T21:06:02.492 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:02.195326+0000 mon.a (mon.0) 1333 : audit [INF] from='client.49052 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:06:02.492 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:02.195404+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-02-19T21:06:02.492 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:02.202347+0000 mon.c (mon.2) 183 : audit [INF] from='client.? v1:172.21.15.163:0/3351446378' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi163-83920-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:02.492 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:01.134796+0000 mgr.y (mgr.14550) 222 : cluster [DBG] pgmap v242: 329 pgs: 32 unknown, 32 creating+peering, 1 peering, 264 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:06:02.492 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:01.286658+0000 mon.c (mon.2) 181 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:02.492 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:01.441672+0000 mon.a (mon.0) 1326 : audit [INF] from='client.47863 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi163-83854-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:02.492 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:01.441931+0000 mon.a (mon.0) 1327 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-19T21:06:02.492 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:01.442023+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-02-19T21:06:02.492 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:01.576070+0000 mon.c (mon.2) 182 : audit [INF] from='client.? v1:172.21.15.163:0/987528065' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:06:02.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:01.714690+0000 mon.a (mon.0) 1329 : audit [INF] from='client.49052 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:06:02.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:01.782215+0000 mon.a (mon.0) 1330 : audit [DBG] from='client.? v1:172.21.15.163:0/797099915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:02.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:02.188910+0000 mon.a (mon.0) 1331 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:06:02.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:02.195100+0000 mon.a (mon.0) 1332 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]': finished 2024-02-19T21:06:02.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:02.195326+0000 mon.a (mon.0) 1333 : audit [INF] from='client.49052 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:06:02.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:02.195404+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-02-19T21:06:02.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:02.202347+0000 mon.c (mon.2) 183 : audit [INF] from='client.? v1:172.21.15.163:0/3351446378' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi163-83920-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:02.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:02.203841+0000 mon.a (mon.0) 1335 : audit [INF] from='client.49058 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi163-83920-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:02.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:02.287501+0000 mon.c (mon.2) 184 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:02.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:02.203841+0000 mon.a (mon.0) 1335 : audit [INF] from='client.49058 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi163-83920-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:02.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:02.287501+0000 mon.c (mon.2) 184 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:02.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:01.134796+0000 mgr.y (mgr.14550) 222 : cluster [DBG] pgmap v242: 329 pgs: 32 unknown, 32 creating+peering, 1 peering, 264 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:06:02.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:01.286658+0000 mon.c (mon.2) 181 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:02.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:01.441672+0000 mon.a (mon.0) 1326 : audit [INF] from='client.47863 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi163-83854-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:02.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:01.441931+0000 mon.a (mon.0) 1327 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-19T21:06:02.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:01.442023+0000 mon.a (mon.0) 1328 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-02-19T21:06:02.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:01.576070+0000 mon.c (mon.2) 182 : audit [INF] from='client.? v1:172.21.15.163:0/987528065' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:06:02.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:01.714690+0000 mon.a (mon.0) 1329 : audit [INF] from='client.49052 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:06:02.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:01.782215+0000 mon.a (mon.0) 1330 : audit [DBG] from='client.? v1:172.21.15.163:0/797099915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:02.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:02.188910+0000 mon.a (mon.0) 1331 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:06:02.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:02.195100+0000 mon.a (mon.0) 1332 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]': finished 2024-02-19T21:06:02.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:02.195326+0000 mon.a (mon.0) 1333 : audit [INF] from='client.49052 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:06:02.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:02.195404+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-02-19T21:06:02.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:02.202347+0000 mon.c (mon.2) 183 : audit [INF] from='client.? v1:172.21.15.163:0/3351446378' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi163-83920-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:02.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:02.203841+0000 mon.a (mon.0) 1335 : audit [INF] from='client.49058 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi163-83920-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:02.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:02.287501+0000 mon.c (mon.2) 184 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:03.742 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:06:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:06:03] "GET /metrics HTTP/1.1" 200 197962 "" "Prometheus/2.33.4" 2024-02-19T21:06:03.816 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:03.819 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:03.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:03.920 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:06:03.948 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:03.967 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:06:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:03.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:06:04.077 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:06:04.103 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:04.274 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:04.306 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:06:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:04.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:06:04.412 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:04.440 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:02.593462+0000 osd.4 (osd.4) 91 : cluster [DBG] 4.e scrub starts 2024-02-19T21:06:04.440 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:02.595259+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.e scrub ok 2024-02-19T21:06:04.440 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:02.691853+0000 osd.3 (osd.3) 99 : cluster [DBG] 2.1b scrub starts 2024-02-19T21:06:04.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:02.693020+0000 osd.3 (osd.3) 100 : cluster [DBG] 2.1b scrub ok 2024-02-19T21:06:04.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:03.135257+0000 mgr.y (mgr.14550) 223 : cluster [DBG] pgmap v245: 313 pgs: 48 unknown, 1 peering, 264 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:06:04.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:03.198080+0000 mon.a (mon.0) 1336 : audit [INF] from='client.49058 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi163-83920-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:04.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:03.198325+0000 mon.a (mon.0) 1337 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-02-19T21:06:04.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:03.203963+0000 mon.c (mon.2) 185 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:04.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:03.205891+0000 mon.a (mon.0) 1338 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:04.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:03.206039+0000 mon.b (mon.1) 283 : audit [INF] from='client.? v1:172.21.15.163:0/2821212368' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi163-83854-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:04.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:03.209938+0000 mon.a (mon.0) 1339 : audit [INF] from='client.47887 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi163-83854-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:04.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:03.288235+0000 mon.c (mon.2) 186 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:04.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:04.175473+0000 mon.a (mon.0) 1340 : audit [INF] from='client.49037 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi163-83845-26"}]': finished 2024-02-19T21:06:04.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:04.175624+0000 mon.a (mon.0) 1341 : audit [INF] from='client.47887 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi163-83854-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:04.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:04.175742+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-02-19T21:06:04.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:04.185312+0000 mon.a (mon.0) 1343 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:04.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:02.593462+0000 osd.4 (osd.4) 91 : cluster [DBG] 4.e scrub starts 2024-02-19T21:06:04.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:02.595259+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.e scrub ok 2024-02-19T21:06:04.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:02.691853+0000 osd.3 (osd.3) 99 : cluster [DBG] 2.1b scrub starts 2024-02-19T21:06:04.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:02.693020+0000 osd.3 (osd.3) 100 : cluster [DBG] 2.1b scrub ok 2024-02-19T21:06:04.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:03.135257+0000 mgr.y (mgr.14550) 223 : cluster [DBG] pgmap v245: 313 pgs: 48 unknown, 1 peering, 264 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:06:04.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:03.198080+0000 mon.a (mon.0) 1336 : audit [INF] from='client.49058 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi163-83920-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:04.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:03.198325+0000 mon.a (mon.0) 1337 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-02-19T21:06:04.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:03.203963+0000 mon.c (mon.2) 185 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:04.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:03.205891+0000 mon.a (mon.0) 1338 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:04.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:03.206039+0000 mon.b (mon.1) 283 : audit [INF] from='client.? v1:172.21.15.163:0/2821212368' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi163-83854-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:04.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:03.209938+0000 mon.a (mon.0) 1339 : audit [INF] from='client.47887 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi163-83854-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:04.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:03.288235+0000 mon.c (mon.2) 186 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:04.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:04.175473+0000 mon.a (mon.0) 1340 : audit [INF] from='client.49037 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi163-83845-26"}]': finished 2024-02-19T21:06:04.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:04.175624+0000 mon.a (mon.0) 1341 : audit [INF] from='client.47887 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi163-83854-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:04.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:04.175742+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-02-19T21:06:04.443 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:04.185312+0000 mon.a (mon.0) 1343 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:04.443 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:04.540 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:04.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:04.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:04.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:02.593462+0000 osd.4 (osd.4) 91 : cluster [DBG] 4.e scrub starts 2024-02-19T21:06:04.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:02.595259+0000 osd.4 (osd.4) 92 : cluster [DBG] 4.e scrub ok 2024-02-19T21:06:04.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:02.691853+0000 osd.3 (osd.3) 99 : cluster [DBG] 2.1b scrub starts 2024-02-19T21:06:04.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:02.693020+0000 osd.3 (osd.3) 100 : cluster [DBG] 2.1b scrub ok 2024-02-19T21:06:04.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:03.135257+0000 mgr.y (mgr.14550) 223 : cluster [DBG] pgmap v245: 313 pgs: 48 unknown, 1 peering, 264 active+clean; 15 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:06:04.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:03.198080+0000 mon.a (mon.0) 1336 : audit [INF] from='client.49058 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi163-83920-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:04.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:03.198325+0000 mon.a (mon.0) 1337 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-02-19T21:06:04.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:03.203963+0000 mon.c (mon.2) 185 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:04.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:03.205891+0000 mon.a (mon.0) 1338 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:04.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:03.206039+0000 mon.b (mon.1) 283 : audit [INF] from='client.? v1:172.21.15.163:0/2821212368' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi163-83854-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:04.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:03.209938+0000 mon.a (mon.0) 1339 : audit [INF] from='client.47887 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi163-83854-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:04.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:03.288235+0000 mon.c (mon.2) 186 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:04.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:04.175473+0000 mon.a (mon.0) 1340 : audit [INF] from='client.49037 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi163-83845-26"}]': finished 2024-02-19T21:06:04.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:04.175624+0000 mon.a (mon.0) 1341 : audit [INF] from='client.47887 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi163-83854-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:04.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:04.175742+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-02-19T21:06:04.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:04.185312+0000 mon.a (mon.0) 1343 : audit [INF] from='client.49037 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:04.644 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:06:04.673 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:04.695 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:06:04.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:04.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:06:04.798 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:04.829 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:04.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:06:05.078 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:05.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:05.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.101+0000 7f2dd5c34700 1 -- 172.21.15.163:0/2041521753 >> v1:172.21.15.163:6790/0 conn(0x7f2dd0100780 legacy=0x7f2dd0102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:05.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.101+0000 7f2dd5c34700 1 -- 172.21.15.163:0/2041521753 shutdown_connections 2024-02-19T21:06:05.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.101+0000 7f2dd5c34700 1 -- 172.21.15.163:0/2041521753 >> 172.21.15.163:0/2041521753 conn(0x7f2dd00f35a0 msgr2=0x7f2dd00f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:05.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.101+0000 7f2dd5c34700 1 -- 172.21.15.163:0/2041521753 shutdown_connections 2024-02-19T21:06:05.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.101+0000 7f2dd5c34700 1 -- 172.21.15.163:0/2041521753 wait complete. 2024-02-19T21:06:05.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dd5c34700 1 librados: starting msgr at 2024-02-19T21:06:05.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dd5c34700 1 librados: starting objecter 2024-02-19T21:06:05.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dd5c34700 1 Processor -- start 2024-02-19T21:06:05.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dd5c34700 1 -- start start 2024-02-19T21:06:05.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dd5c34700 1 librados: setting wanted keys 2024-02-19T21:06:05.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dd5c34700 1 librados: calling monclient init 2024-02-19T21:06:05.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dd5c34700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2dd00fcee0 con 0x7f2dd01042c0 2024-02-19T21:06:05.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dd5c34700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2dd00fd070 con 0x7f2dd0100780 2024-02-19T21:06:05.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dd5c34700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2dd00fd260 con 0x7f2dd00fd980 2024-02-19T21:06:05.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dcffff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f2dd01042c0 0x7f2dd0118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:51946/0 (socket says 172.21.15.163:51946) 2024-02-19T21:06:05.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dcffff700 1 -- 172.21.15.163:0/784388871 learned_addr learned my addr 172.21.15.163:0/784388871 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:05.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dccff9700 10 client.?.objecter ms_handle_connect 0x7f2dd01042c0 2024-02-19T21:06:05.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dccff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:05.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dccff9700 10 client.?.objecter ms_handle_connect 0x7f2dd00fd980 2024-02-19T21:06:05.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dccff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:05.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dccff9700 10 client.?.objecter ms_handle_connect 0x7f2dd0100780 2024-02-19T21:06:05.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.102+0000 7f2dccff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:05.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505999867 0 0) 0x7f2dd00fd070 con 0x7f2dd0100780 2024-02-19T21:06:05.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db80033f0 con 0x7f2dd0100780 2024-02-19T21:06:05.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924676027 0 0) 0x7f2dd00fd260 con 0x7f2dd00fd980 2024-02-19T21:06:05.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd00fd070 con 0x7f2dd00fd980 2024-02-19T21:06:05.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968270580 0 0) 0x7f2dd00fcee0 con 0x7f2dd01042c0 2024-02-19T21:06:05.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd00fd260 con 0x7f2dd01042c0 2024-02-19T21:06:05.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2952959089 0 0) 0x7f2db80033f0 con 0x7f2dd0100780 2024-02-19T21:06:05.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dd00fcee0 con 0x7f2dd0100780 2024-02-19T21:06:05.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2dc4001fa0 con 0x7f2dd0100780 2024-02-19T21:06:05.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3409611304 0 0) 0x7f2dd00fd070 con 0x7f2dd00fd980 2024-02-19T21:06:05.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2db80033f0 con 0x7f2dd00fd980 2024-02-19T21:06:05.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2dc0001580 con 0x7f2dd00fd980 2024-02-19T21:06:05.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2042940474 0 0) 0x7f2dd00fd260 con 0x7f2dd01042c0 2024-02-19T21:06:05.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dd00fd070 con 0x7f2dd01042c0 2024-02-19T21:06:05.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2dc8001d10 con 0x7f2dd01042c0 2024-02-19T21:06:05.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1827687249 0 0) 0x7f2dd00fcee0 con 0x7f2dd0100780 2024-02-19T21:06:05.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 >> v1:172.21.15.163:6790/0 conn(0x7f2dd00fd980 legacy=0x7f2dd00fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:05.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 >> v1:172.21.15.163:6789/0 conn(0x7f2dd01042c0 legacy=0x7f2dd0118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:05.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd011cb90 con 0x7f2dd0100780 2024-02-19T21:06:05.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dd5c34700 1 -- 172.21.15.163:0/784388871 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd0119bc0 con 0x7f2dd0100780 2024-02-19T21:06:05.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dd5c34700 10 client.47899.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:05.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dd5c34700 1 -- 172.21.15.163:0/784388871 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd011a030 con 0x7f2dd0100780 2024-02-19T21:06:05.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dd5c34700 1 librados: init done 2024-02-19T21:06:05.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f2dc4001100 con 0x7f2dd0100780 2024-02-19T21:06:05.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.103+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2dc4003620 con 0x7f2dd0100780 2024-02-19T21:06:05.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.104+0000 7f2dd5c34700 1 -- 172.21.15.163:0/784388871 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dbc004f20 con 0x7f2dd0100780 2024-02-19T21:06:05.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.105+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f2dc4001720 con 0x7f2dd0100780 2024-02-19T21:06:05.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.105+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(184..184 src has 43..184) v4 ==== 9673+0+0 (unknown 1904477654 0 0) 0x7f2dc4076da0 con 0x7f2dd0100780 2024-02-19T21:06:05.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.105+0000 7f2dccff9700 10 client.47899.objecter ms_dispatch 0x7f2dd0001550 osd_map(184..184 src has 43..184) v4 2024-02-19T21:06:05.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.105+0000 7f2dccff9700 3 client.47899.objecter handle_osd_map got epochs [184,184] > 0 2024-02-19T21:06:05.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.105+0000 7f2dccff9700 3 client.47899.objecter handle_osd_map decoding full epoch 184 2024-02-19T21:06:05.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.106+0000 7f2dccff9700 20 client.47899.objecter dump_active .. 0 homeless 2024-02-19T21:06:05.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.106+0000 7f2dccff9700 10 client.47899.objecter ms_handle_connect 0x7f2db8063dc0 2024-02-19T21:06:05.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.110+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f2dc4049b60 con 0x7f2dd0100780 2024-02-19T21:06:05.177 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp api_aio: Running main() from gmock_main.cc 2024-02-19T21:06:05.178 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [==========] Running 41 tests from 2 test suites. 2024-02-19T21:06:05.178 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [----------] Global test environment set-up. 2024-02-19T21:06:05.178 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [----------] 25 tests from LibRadosAio 2024-02-19T21:06:05.178 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-02-19T21:06:05.178 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.TooBig (7967 ms) 2024-02-19T21:06:05.178 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-02-19T21:06:05.178 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (4861 ms) 2024-02-19T21:06:05.178 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-02-19T21:06:05.178 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (2872 ms) 2024-02-19T21:06:05.178 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-02-19T21:06:05.178 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3050 ms) 2024-02-19T21:06:05.178 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-02-19T21:06:05.178 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3001 ms) 2024-02-19T21:06:05.179 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-02-19T21:06:05.179 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (4214 ms) 2024-02-19T21:06:05.179 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-02-19T21:06:05.179 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3360 ms) 2024-02-19T21:06:05.179 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-02-19T21:06:05.179 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3854 ms) 2024-02-19T21:06:05.179 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-02-19T21:06:05.179 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (4017 ms) 2024-02-19T21:06:05.179 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-02-19T21:06:05.179 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3062 ms) 2024-02-19T21:06:05.179 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-02-19T21:06:05.179 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3173 ms) 2024-02-19T21:06:05.179 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-02-19T21:06:05.179 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2904 ms) 2024-02-19T21:06:05.179 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-02-19T21:06:05.180 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3019 ms) 2024-02-19T21:06:05.180 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-02-19T21:06:05.180 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2983 ms) 2024-02-19T21:06:05.180 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-02-19T21:06:05.180 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.Flush (3020 ms) 2024-02-19T21:06:05.180 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-02-19T21:06:05.180 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2979 ms) 2024-02-19T21:06:05.180 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-02-19T21:06:05.180 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3006 ms) 2024-02-19T21:06:05.180 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-02-19T21:06:05.180 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3015 ms) 2024-02-19T21:06:05.180 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-02-19T21:06:05.180 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (2995 ms) 2024-02-19T21:06:05.180 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-02-19T21:06:05.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3011 ms) 2024-02-19T21:06:05.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-02-19T21:06:05.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3227 ms) 2024-02-19T21:06:05.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-02-19T21:06:05.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2778 ms) 2024-02-19T21:06:05.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-02-19T21:06:05.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3006 ms) 2024-02-19T21:06:05.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-02-19T21:06:05.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (2980 ms) 2024-02-19T21:06:05.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-02-19T21:06:05.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3011 ms) 2024-02-19T21:06:05.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [----------] 25 tests from LibRadosAio (85367 ms total) 2024-02-19T21:06:05.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: 2024-02-19T21:06:05.181 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-02-19T21:06:05.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-02-19T21:06:05.182 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6980 ms) 2024-02-19T21:06:05.182 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:05.216 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:05.237 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:05.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:05.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:05.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.313+0000 7f2dd5c34700 1 -- 172.21.15.163:0/784388871 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f2dbc0051d0 con 0x7f2dd0100780 2024-02-19T21:06:05.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.314+0000 7f2dccff9700 1 -- 172.21.15.163:0/784388871 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v185) v1 ==== 81+0+56086 (unknown 1954189736 0 1022627992) 0x7f2dc4047fd0 con 0x7f2dd0100780 2024-02-19T21:06:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:06:05.311 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:06:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "epoch": 185, 2024-02-19T21:06:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "fsid": "2591809a-cf69-11ee-95bb-87774f69a715", 2024-02-19T21:06:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "created": "2024-02-19T20:55:37.794885+0000", 2024-02-19T21:06:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "modified": "2024-02-19T21:06:05.170048+0000", 2024-02-19T21:06:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "last_up_change": "2024-02-19T21:01:09.214134+0000", 2024-02-19T21:06:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "last_in_change": "2024-02-19T21:00:51.699017+0000", 2024-02-19T21:06:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-19T21:06:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_num": 5804032, 2024-02-19T21:06:05.311 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_set": [ 2024-02-19T21:06:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "nodeep-scrub", 2024-02-19T21:06:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "pglog_hardlimit", 2024-02-19T21:06:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "purged_snapdirs", 2024-02-19T21:06:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_deletes", 2024-02-19T21:06:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "sortbitwise" 2024-02-19T21:06:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:06:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_version": 35, 2024-02-19T21:06:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "full_ratio": 0.94999998807907104, 2024-02-19T21:06:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-19T21:06:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-19T21:06:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_snapshot": "", 2024-02-19T21:06:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_max": 129, 2024-02-19T21:06:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "max_osd": 8, 2024-02-19T21:06:05.312 INFO:teuthology.orchestra.run.smithi163.stdout: "require_min_compat_client": "luminous", 2024-02-19T21:06:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "min_compat_client": "luminous", 2024-02-19T21:06:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "require_osd_release": "pacific", 2024-02-19T21:06:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "pools": [ 2024-02-19T21:06:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 1, 2024-02-19T21:06:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "device_health_metrics", 2024-02-19T21:06:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T20:59:09.425169+0000", 2024-02-19T21:06:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:06:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:06:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:06:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:06:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:06:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:06:05.313 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:06:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:06:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:06:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:06:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:06:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:06:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 1, 2024-02-19T21:06:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 1, 2024-02-19T21:06:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 1, 2024-02-19T21:06:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 1, 2024-02-19T21:06:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 1, 2024-02-19T21:06:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:06:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:06:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:06:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:06:05.314 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:06:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:06:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:06:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "74", 2024-02-19T21:06:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:06:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:06:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:06:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:06:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:06:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:06:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:06:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:06:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:06:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:06:05.315 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:06:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:06:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:06:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:06:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:06:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:06:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:06:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:06:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:06:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:06:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:06:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:06:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:06:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:06:05.316 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:06:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:06:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:06:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:06:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:06:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:06:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:06:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:06:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:06:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:06:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:06:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:06:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:06:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:06:05.317 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_max": 32, 2024-02-19T21:06:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_min": 1 2024-02-19T21:06:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:06:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_devicehealth": {} 2024-02-19T21:06:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 2, 2024-02-19T21:06:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".rgw.root", 2024-02-19T21:06:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:14.658945+0000", 2024-02-19T21:06:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:06:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:06:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:06:05.318 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:06:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:06:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:06:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:06:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:06:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:06:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:06:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:06:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:06:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 42, 2024-02-19T21:06:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 42, 2024-02-19T21:06:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 42, 2024-02-19T21:06:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 42, 2024-02-19T21:06:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 42, 2024-02-19T21:06:05.319 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:06:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:06:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:06:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:06:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:06:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:06:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:06:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "141", 2024-02-19T21:06:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:06:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:06:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "139", 2024-02-19T21:06:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:06:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:06:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:06:05.320 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:06:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:06:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:06:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:06:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:06:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:06:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:06:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:06:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:06:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:06:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:06:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:06:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:06:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:06:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:06:05.321 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:06:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:06:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:06:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:06:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:06:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:06:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:06:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:06:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:06:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:06:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:06:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:06:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:06:05.322 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:06:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:06:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:06:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:06:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:06:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:06:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 3, 2024-02-19T21:06:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "iscsi", 2024-02-19T21:06:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:14.958800+0000", 2024-02-19T21:06:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:06:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:06:05.323 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:06:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:06:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:06:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:06:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:06:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:06:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:06:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:06:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:06:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:06:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 3, 2024-02-19T21:06:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 3, 2024-02-19T21:06:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 3, 2024-02-19T21:06:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 3, 2024-02-19T21:06:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 3, 2024-02-19T21:06:05.324 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:06:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:06:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:06:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:06:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:06:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:06:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:06:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "100", 2024-02-19T21:06:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:06:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:06:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:06:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:06:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:06:05.325 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:06:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:06:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:06:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:06:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:06:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:06:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:06:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:06:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:06:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:06:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:06:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:06:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:06:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:06:05.326 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:06:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:06:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:06:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:06:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:06:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:06:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:06:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:06:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:06:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:06:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:06:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:06:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:06:05.327 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:06:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:06:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:06:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:06:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:06:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:06:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:06:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd": {} 2024-02-19T21:06:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 4, 2024-02-19T21:06:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.log", 2024-02-19T21:06:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:16.543816+0000", 2024-02-19T21:06:05.328 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:06:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:06:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:06:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:06:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:06:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:06:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:06:05.329 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:06:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:06:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:06:05.330 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "61", 2024-02-19T21:06:05.331 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:06:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:06:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:06:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:06:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:06:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:06:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:06:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:06:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:06:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:06:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:06:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:06:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:06:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:06:05.332 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:06:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:06:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:06:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:06:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:06:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:06:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:06:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:06:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:06:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:06:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:06:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:06:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:06:05.333 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.334 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 5, 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.control", 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:18.521482+0000", 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:06:05.335 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:06:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:06:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:06:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:06:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:06:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:06:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:06:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:06:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:06:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:06:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:06:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "63", 2024-02-19T21:06:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:06:05.336 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:06:05.337 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:06:05.338 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:06:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:06:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:06:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:06:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:06:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:06:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:06:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:06:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 6, 2024-02-19T21:06:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.meta", 2024-02-19T21:06:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:20.586564+0000", 2024-02-19T21:06:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:06:05.339 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:06:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:06:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:06:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:06:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:06:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:06:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:06:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:06:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:06:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:06:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:06:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:06:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:06:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:06:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:06:05.340 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "65", 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:06:05.341 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:06:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:06:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:06:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:06:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:06:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:06:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:06:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:06:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:06:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:06:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:06:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:06:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:06:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:06:05.342 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:06:05.343 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:06:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:06:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:06:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_bias": 4 2024-02-19T21:06:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:06:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:06:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 8, 2024-02-19T21:06:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "LibRadosList_smithi163-83888-1", 2024-02-19T21:06:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:33.426193+0000", 2024-02-19T21:06:05.344 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 26, 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 11, 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 11, 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 11, 2024-02-19T21:06:05.345 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 25, 2024-02-19T21:06:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:06:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "8.1a", 2024-02-19T21:06:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 183, 2024-02-19T21:06:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 182, 2024-02-19T21:06:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 182, 2024-02-19T21:06:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "168'7", 2024-02-19T21:06:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "168'14" 2024-02-19T21:06:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "185", 2024-02-19T21:06:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:06:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "185", 2024-02-19T21:06:05.346 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "185", 2024-02-19T21:06:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:06:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:06:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:06:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:06:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:06:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:06:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:06:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:06:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:06:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:06:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:06:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:06:05.347 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:06:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:06:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:06:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:06:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:06:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:06:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:06:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:06:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:06:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:06:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:06:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:06:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:06:05.348 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:06:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:06:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:06:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:06:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:06:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:06:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:06:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:06:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:06:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:06:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:06:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-02-19T21:06:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.349 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 16, 2024-02-19T21:06:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-19T21:06:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:35.333192+0000", 2024-02-19T21:06:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:06:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:06:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:06:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:06:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:06:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:06:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:06:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:06:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:06:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:06:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:06:05.350 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:06:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:06:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:06:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:06:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:06:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:06:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:06:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:06:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:06:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:06:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:06:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:06:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:06:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "101", 2024-02-19T21:06:05.351 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:06:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:06:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:06:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:06:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:06:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:06:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:06:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:06:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:06:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:06:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:06:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:06:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:06:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:06:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:06:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:06:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:06:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:06:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:06:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:06:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:06:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:06:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:06:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:06:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:06:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:06:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:06:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:06:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:06:05.353 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:06:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:06:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:06:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:06:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:06:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:06:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:06:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:06:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:06:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": {} 2024-02-19T21:06:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.354 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 61, 2024-02-19T21:06:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "test-rados-api-smithi163-83928-6", 2024-02-19T21:06:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:52.991355+0000", 2024-02-19T21:06:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8193, 2024-02-19T21:06:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-19T21:06:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:06:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:06:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:06:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:06:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:06:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:06:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:06:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:06:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:06:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:06:05.355 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:06:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:06:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:06:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:06:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:06:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:06:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:06:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:06:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:06:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:06:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:06:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:06:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "185", 2024-02-19T21:06:05.356 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "150", 2024-02-19T21:06:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "150", 2024-02-19T21:06:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "150", 2024-02-19T21:06:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:06:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:06:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 11, 2024-02-19T21:06:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 185, 2024-02-19T21:06:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:06:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:06:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:06:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:06:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [ 2024-02-19T21:06:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: 94 2024-02-19T21:06:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:06:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:06:05.357 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": 94, 2024-02-19T21:06:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": 94, 2024-02-19T21:06:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:06:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:06:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:06:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:06:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:06:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:06:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:06:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:06:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:06:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:06:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:06:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:06:05.358 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:06:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:06:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:06:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:06:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:06:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:06:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:06:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:06:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:06:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:06:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:06:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:06:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-02-19T21:06:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.359 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 94, 2024-02-19T21:06:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "test-rados-api-smithi163-83928-13", 2024-02-19T21:06:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:05:22.433403+0000", 2024-02-19T21:06:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8201, 2024-02-19T21:06:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-02-19T21:06:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:06:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:06:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:06:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:06:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:06:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:06:05.360 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:06:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:06:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:06:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:06:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:06:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:06:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:06:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:06:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:06:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:06:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:06:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:06:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:06:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:06:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:06:05.361 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:06:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "185", 2024-02-19T21:06:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "150", 2024-02-19T21:06:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "150", 2024-02-19T21:06:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "150", 2024-02-19T21:06:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:06:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:06:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 11, 2024-02-19T21:06:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 185, 2024-02-19T21:06:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:06:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:06:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:06:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:06:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:06:05.362 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": 61, 2024-02-19T21:06:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:06:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:06:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "writeback", 2024-02-19T21:06:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:06:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:06:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:06:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:06:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:06:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:06:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:06:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:06:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:06:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:06:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.363 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:06:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:06:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:06:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:06:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:06:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:06:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:06:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:06:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:06:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:06:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:06:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:06:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:06:05.364 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-02-19T21:06:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 126, 2024-02-19T21:06:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20", 2024-02-19T21:06:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:06:01.441441+0000", 2024-02-19T21:06:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8193, 2024-02-19T21:06:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-19T21:06:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 3, 2024-02-19T21:06:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:06:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:06:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 2, 2024-02-19T21:06:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:06:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:06:05.365 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:06:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:06:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:06:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:06:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 8, 2024-02-19T21:06:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 8, 2024-02-19T21:06:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 8, 2024-02-19T21:06:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 8, 2024-02-19T21:06:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 8, 2024-02-19T21:06:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:06:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:06:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:06:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:06:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:06:05.366 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:06:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:06:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "185", 2024-02-19T21:06:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:06:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:06:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:06:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:06:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:06:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 4, 2024-02-19T21:06:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 185, 2024-02-19T21:06:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:06:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:06:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:06:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:06:05.367 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:06:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:06:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:06:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:06:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:06:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:06:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:06:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:06:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:06:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:06:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:06:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:06:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20", 2024-02-19T21:06:05.368 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:06:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:06:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:06:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:06:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:06:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:06:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:06:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:06:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:06:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:06:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 8192, 2024-02-19T21:06:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:06:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:06:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:06:05.369 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": {} 2024-02-19T21:06:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 129, 2024-02-19T21:06:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "LibRadosChecksum/2_smithi163-83920-33", 2024-02-19T21:06:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:06:04.469313+0000", 2024-02-19T21:06:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 32769, 2024-02-19T21:06:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,creating", 2024-02-19T21:06:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:06:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:06:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:06:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:06:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:06:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:06:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:06:05.370 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:06:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:06:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:06:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:06:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:06:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:06:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:06:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:06:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:06:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:06:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:06:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:06:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:06:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:06:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:06:05.371 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "185", 2024-02-19T21:06:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:06:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:06:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:06:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:06:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:06:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:06:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:06:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:06:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:06:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:06:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:06:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:06:05.372 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:06:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:06:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:06:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:06:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:06:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:06:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:06:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:06:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:06:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:06:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:06:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:06:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:06:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:06:05.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:06:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:06:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:06:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:06:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:06:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:06:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:06:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:06:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:06:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:06:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:06:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:06:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": {} 2024-02-19T21:06:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.374 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088", 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 8, 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 183, 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.375 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6801", 2024-02-19T21:06:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:06:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:06:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6802", 2024-02-19T21:06:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:06:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.376 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:06:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6804", 2024-02-19T21:06:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:06:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:06:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.377 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6803", 2024-02-19T21:06:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:06:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6801/1559418115", 2024-02-19T21:06:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6802/1559418115", 2024-02-19T21:06:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6804/1559418115", 2024-02-19T21:06:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6803/1559418115", 2024-02-19T21:06:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:06:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:06:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:06:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.378 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-02-19T21:06:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "e74a65fa-cb9c-42c3-9293-602cf7e9dc67", 2024-02-19T21:06:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:06:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:06:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:06:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:06:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:06:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:06:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 13, 2024-02-19T21:06:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 183, 2024-02-19T21:06:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:06:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:06:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:06:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.379 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6805", 2024-02-19T21:06:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:06:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:06:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6806", 2024-02-19T21:06:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:06:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.380 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:06:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6808", 2024-02-19T21:06:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:06:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:06:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6807", 2024-02-19T21:06:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:06:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6805/3125047432", 2024-02-19T21:06:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6806/3125047432", 2024-02-19T21:06:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6808/3125047432", 2024-02-19T21:06:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6807/3125047432", 2024-02-19T21:06:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:06:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:06:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:06:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "6980a625-cdc9-487e-8f8c-7997ba716a97", 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 18, 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 183, 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.383 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6809", 2024-02-19T21:06:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:06:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:06:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6810", 2024-02-19T21:06:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:06:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.384 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6812", 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6811", 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.385 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6809/1563158487", 2024-02-19T21:06:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6810/1563158487", 2024-02-19T21:06:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6812/1563158487", 2024-02-19T21:06:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6811/1563158487", 2024-02-19T21:06:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:06:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:06:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:06:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-02-19T21:06:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5", 2024-02-19T21:06:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:06:05.386 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:06:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:06:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:06:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:06:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:06:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 25, 2024-02-19T21:06:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 183, 2024-02-19T21:06:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:06:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:06:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:06:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6813", 2024-02-19T21:06:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:06:05.387 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:06:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6814", 2024-02-19T21:06:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:06:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:06:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.388 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6816", 2024-02-19T21:06:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:06:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:06:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6815", 2024-02-19T21:06:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:06:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.389 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6813/2112279769", 2024-02-19T21:06:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6814/2112279769", 2024-02-19T21:06:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6816/2112279769", 2024-02-19T21:06:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6815/2112279769", 2024-02-19T21:06:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:06:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:06:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:06:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-02-19T21:06:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0", 2024-02-19T21:06:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:06:05.390 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 31, 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 183, 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6800", 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.391 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:06:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6801", 2024-02-19T21:06:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:06:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:06:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.392 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6803", 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6802", 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.393 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6800/3129716965", 2024-02-19T21:06:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6801/3129716965", 2024-02-19T21:06:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6803/3129716965", 2024-02-19T21:06:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6802/3129716965", 2024-02-19T21:06:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:06:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:06:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:06:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-02-19T21:06:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f", 2024-02-19T21:06:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:06:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:06:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:06:05.394 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 37, 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 183, 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6804", 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.395 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:06:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6805", 2024-02-19T21:06:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:06:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:06:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6807", 2024-02-19T21:06:05.396 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6806", 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6804/805161991", 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6805/805161991", 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6807/805161991", 2024-02-19T21:06:05.397 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6806/805161991", 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "302c95df-3842-4818-8afd-d79d8184684f", 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 45, 2024-02-19T21:06:05.398 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 183, 2024-02-19T21:06:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:06:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:06:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:06:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6808", 2024-02-19T21:06:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:06:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:06:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.399 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6809", 2024-02-19T21:06:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:06:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:06:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6811", 2024-02-19T21:06:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:06:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.400 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:06:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6810", 2024-02-19T21:06:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:06:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6808/787604615", 2024-02-19T21:06:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6809/787604615", 2024-02-19T21:06:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6811/787604615", 2024-02-19T21:06:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6810/787604615", 2024-02-19T21:06:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:06:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:06:05.401 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:06:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-02-19T21:06:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f", 2024-02-19T21:06:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:06:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:06:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:06:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:06:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:06:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:06:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 53, 2024-02-19T21:06:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 183, 2024-02-19T21:06:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:06:05.402 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:06:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:06:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6812", 2024-02-19T21:06:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:06:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:06:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6813", 2024-02-19T21:06:05.403 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:06:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:06:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6815", 2024-02-19T21:06:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:06:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:06:05.404 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:06:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:06:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6814", 2024-02-19T21:06:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:06:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6812/157927432", 2024-02-19T21:06:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6813/157927432", 2024-02-19T21:06:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6815/157927432", 2024-02-19T21:06:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6814/157927432", 2024-02-19T21:06:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:06:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:06:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:06:05.405 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:06:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_xinfo": [ 2024-02-19T21:06:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-02-19T21:06:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:06:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:06:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:06:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:06:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:06:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:05:57.520396+0000", 2024-02-19T21:06:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:06:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.406 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-02-19T21:06:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:06:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:06:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:06:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:06:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:06:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:05:56.376788+0000", 2024-02-19T21:06:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:06:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-02-19T21:06:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:06:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:06:05.407 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:06:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:06:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:06:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:05:22.137628+0000", 2024-02-19T21:06:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:06:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-02-19T21:06:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:06:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:06:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:06:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:06:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:06:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:05:49.588903+0000", 2024-02-19T21:06:05.408 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:06:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-02-19T21:06:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:06:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:06:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:06:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:06:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:06:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:05:55.719928+0000", 2024-02-19T21:06:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:06:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-02-19T21:06:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:06:05.409 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:06:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:06:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:06:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:06:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:05:30.292761+0000", 2024-02-19T21:06:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:06:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-02-19T21:06:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:06:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:06:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:06:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:06:05.410 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:06:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:05:51.547827+0000", 2024-02-19T21:06:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:06:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-02-19T21:06:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:06:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:06:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:06:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:06:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:06:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:05:51.951656+0000", 2024-02-19T21:06:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:06:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.411 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:06:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap": [ 2024-02-19T21:06:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "2.d", 2024-02-19T21:06:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-02-19T21:06:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: 1, 2024-02-19T21:06:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: 4, 2024-02-19T21:06:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: 2 2024-02-19T21:06:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "94.1c", 2024-02-19T21:06:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-02-19T21:06:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: 4, 2024-02-19T21:06:05.412 INFO:teuthology.orchestra.run.smithi163.stdout: 2, 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: 0 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_items": [ 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "2.4", 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 7, 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.413 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.7", 2024-02-19T21:06:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:06:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:06:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-02-19T21:06:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.d", 2024-02-19T21:06:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:06:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:06:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 5 2024-02-19T21:06:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.414 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.12", 2024-02-19T21:06:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:06:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:06:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 5 2024-02-19T21:06:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:06:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_temp": [], 2024-02-19T21:06:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_temp": [], 2024-02-19T21:06:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "blocklist": { 2024-02-19T21:06:05.415 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1427774891": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:06:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3282102947": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:06:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2735092549": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:06:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1235588846": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:06:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2731225015": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:06:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/506491651": "2024-02-20T20:56:02.185409+0000", 2024-02-19T21:06:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/168117826": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:06:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1458009465": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:06:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/541631472": "2024-02-20T20:56:02.185409+0000", 2024-02-19T21:06:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2248999541": "2024-02-20T20:56:02.185409+0000" 2024-02-19T21:06:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "range_blocklist": {}, 2024-02-19T21:06:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profiles": { 2024-02-19T21:06:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "default": { 2024-02-19T21:06:05.416 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-02-19T21:06:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-02-19T21:06:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-02-19T21:06:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-02-19T21:06:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "ruleset-failure-domain": "osd", 2024-02-19T21:06:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van" 2024-02-19T21:06:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20": { 2024-02-19T21:06:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-device-class": "", 2024-02-19T21:06:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-02-19T21:06:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-root": "default", 2024-02-19T21:06:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-19T21:06:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-02-19T21:06:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-02-19T21:06:05.417 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-02-19T21:06:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "ruleset-failure-domain": "osd", 2024-02-19T21:06:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van", 2024-02-19T21:06:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "w": "8" 2024-02-19T21:06:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps_queue": [ 2024-02-19T21:06:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 11, 2024-02-19T21:06:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:06:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:06:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.418 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:06:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 9, 2024-02-19T21:06:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:06:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 25, 2024-02-19T21:06:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:06:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:06:05.419 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:06:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:06:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 57, 2024-02-19T21:06:05.420 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 61, 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 4, 2024-02-19T21:06:05.421 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 4 2024-02-19T21:06:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 76, 2024-02-19T21:06:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:06:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:06:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:06:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.422 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:06:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 85, 2024-02-19T21:06:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:06:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:06:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 92, 2024-02-19T21:06:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:06:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:06:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:06:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:06:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:06:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.424 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 94, 2024-02-19T21:06:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:06:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 4, 2024-02-19T21:06:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 4 2024-02-19T21:06:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 107, 2024-02-19T21:06:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:06:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:06:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.425 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 112, 2024-02-19T21:06:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:06:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 1, 2024-02-19T21:06:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:06:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:06:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:06:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 126, 2024-02-19T21:06:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:06:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:06:05.427 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:06:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: "new_removed_snaps": [ 2024-02-19T21:06:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 61, 2024-02-19T21:06:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:06:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 4, 2024-02-19T21:06:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 94, 2024-02-19T21:06:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:06:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 4, 2024-02-19T21:06:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:06:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 126, 2024-02-19T21:06:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:06:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:06:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:06:05.429 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:06:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:06:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:06:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: "new_purged_snaps": [], 2024-02-19T21:06:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_node_flags": {}, 2024-02-19T21:06:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: "device_class_flags": {}, 2024-02-19T21:06:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode": { 2024-02-19T21:06:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_enabled": false, 2024-02-19T21:06:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_bucket_count": 0, 2024-02-19T21:06:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: "degraded_stretch_mode": 0, 2024-02-19T21:06:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: "recovering_stretch_mode": 0, 2024-02-19T21:06:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_bucket": 0 2024-02-19T21:06:05.430 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:06:05.430 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:06:05.431 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:05.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.316+0000 7f2dd5c34700 10 librados: watch_flush enter 2024-02-19T21:06:05.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.316+0000 7f2dd5c34700 10 librados: watch_flush exit 2024-02-19T21:06:05.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.316+0000 7f2dd5c34700 20 client.47899.objecter shutdown clearing up homeless session... 2024-02-19T21:06:05.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.316+0000 7f2dd5c34700 10 client.47899.objecter successfully canceled tick 2024-02-19T21:06:05.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.317+0000 7f2dd5c34700 1 -- 172.21.15.163:0/784388871 >> v1:172.21.15.163:6800/1552514681 conn(0x7f2db8063dc0 legacy=0x7f2db8066280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:05.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.317+0000 7f2dd5c34700 1 -- 172.21.15.163:0/784388871 >> v1:172.21.15.193:6789/0 conn(0x7f2dd0100780 legacy=0x7f2dd0114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:05.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.317+0000 7f2dd5c34700 1 -- 172.21.15.163:0/784388871 shutdown_connections 2024-02-19T21:06:05.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.317+0000 7f2dd5c34700 1 -- 172.21.15.163:0/784388871 >> 172.21.15.163:0/784388871 conn(0x7f2dd00f35a0 msgr2=0x7f2dd0107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:05.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.317+0000 7f2dd5c34700 1 -- 172.21.15.163:0/784388871 shutdown_connections 2024-02-19T21:06:05.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.317+0000 7f2dd5c34700 1 -- 172.21.15.163:0/784388871 wait complete. 2024-02-19T21:06:05.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:05.317+0000 7f2dd5c34700 1 librados: shutdown 2024-02-19T21:06:05.433 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:04.184649+0000 mon.c (mon.2) 187 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:05.433 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:04.288957+0000 mon.c (mon.2) 188 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:05.433 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.136687+0000 mon.b (mon.1) 284 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-19T21:06:05.433 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.138642+0000 mon.a (mon.0) 1344 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-19T21:06:05.433 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.176312+0000 mon.a (mon.0) 1345 : audit [INF] from='client.49037 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi163-83845-26"}]': finished 2024-02-19T21:06:05.434 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.176463+0000 mon.a (mon.0) 1346 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-19T21:06:05.434 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:05.176529+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-02-19T21:06:05.434 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.186103+0000 mon.c (mon.2) 189 : audit [INF] from='client.? v1:172.21.15.163:0/1897681723' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi163-83920-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:05.434 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.186507+0000 mon.a (mon.0) 1348 : audit [INF] from='client.49076 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi163-83920-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:05.434 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:04.184649+0000 mon.c (mon.2) 187 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:05.434 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:04.288957+0000 mon.c (mon.2) 188 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:05.434 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.136687+0000 mon.b (mon.1) 284 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-19T21:06:05.434 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.138642+0000 mon.a (mon.0) 1344 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-19T21:06:05.434 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.176312+0000 mon.a (mon.0) 1345 : audit [INF] from='client.49037 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi163-83845-26"}]': finished 2024-02-19T21:06:05.434 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.176463+0000 mon.a (mon.0) 1346 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-19T21:06:05.435 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:05.176529+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-02-19T21:06:05.435 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.186103+0000 mon.c (mon.2) 189 : audit [INF] from='client.? v1:172.21.15.163:0/1897681723' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi163-83920-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:05.435 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.186507+0000 mon.a (mon.0) 1348 : audit [INF] from='client.49076 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi163-83920-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:05.460 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:05.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:04.184649+0000 mon.c (mon.2) 187 : audit [INF] from='client.? v1:172.21.15.163:0/1740076176' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi163-83845-26"}]: dispatch 2024-02-19T21:06:05.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:04.288957+0000 mon.c (mon.2) 188 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:05.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.136687+0000 mon.b (mon.1) 284 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-19T21:06:05.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.138642+0000 mon.a (mon.0) 1344 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-02-19T21:06:05.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.176312+0000 mon.a (mon.0) 1345 : audit [INF] from='client.49037 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi163-83845-26"}]': finished 2024-02-19T21:06:05.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.176463+0000 mon.a (mon.0) 1346 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-02-19T21:06:05.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:05.176529+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-02-19T21:06:05.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.186103+0000 mon.c (mon.2) 189 : audit [INF] from='client.? v1:172.21.15.163:0/1897681723' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi163-83920-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:05.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.186507+0000 mon.a (mon.0) 1348 : audit [INF] from='client.49076 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi163-83920-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:05.805 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:06:05.938 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:05.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:06.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:06:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:06.194 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:06:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:06.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:06:06.299 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:06.328 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:06.348 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:06.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:06.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:06.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:06.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:04.600148+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:06:06.483 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:04.600969+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:06:06.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:04.865426+0000 osd.6 (osd.6) 85 : cluster [DBG] 2.24 scrub starts 2024-02-19T21:06:06.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:04.866390+0000 osd.6 (osd.6) 86 : cluster [DBG] 2.24 scrub ok 2024-02-19T21:06:06.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:05.136288+0000 mgr.y (mgr.14550) 224 : cluster [DBG] pgmap v248: 304 pgs: 16 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 240 active+clean; 30 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.2 KiB/s wr, 2 op/s 2024-02-19T21:06:06.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:05.197051+0000 mon.a (mon.0) 1349 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:06:06.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.217825+0000 mon.b (mon.1) 285 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.219291+0000 mon.a (mon.0) 1350 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.220036+0000 mon.b (mon.1) 286 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.286409+0000 mon.b (mon.1) 287 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:06.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.289699+0000 mon.c (mon.2) 190 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:06.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.314874+0000 mon.b (mon.1) 288 : audit [DBG] from='client.? v1:172.21.15.163:0/784388871' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:06:06.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.471609+0000 mon.a (mon.0) 1351 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.471905+0000 mon.a (mon.0) 1352 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:06.484 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.472601+0000 mon.b (mon.1) 289 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi163-83845-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:06.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:04.600148+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:06:06.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:04.600969+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:06:06.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:04.865426+0000 osd.6 (osd.6) 85 : cluster [DBG] 2.24 scrub starts 2024-02-19T21:06:06.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:04.866390+0000 osd.6 (osd.6) 86 : cluster [DBG] 2.24 scrub ok 2024-02-19T21:06:06.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:05.136288+0000 mgr.y (mgr.14550) 224 : cluster [DBG] pgmap v248: 304 pgs: 16 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 240 active+clean; 30 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.2 KiB/s wr, 2 op/s 2024-02-19T21:06:06.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:05.197051+0000 mon.a (mon.0) 1349 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:06:06.485 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.217825+0000 mon.b (mon.1) 285 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.219291+0000 mon.a (mon.0) 1350 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.220036+0000 mon.b (mon.1) 286 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.286409+0000 mon.b (mon.1) 287 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:06.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.289699+0000 mon.c (mon.2) 190 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:06.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.314874+0000 mon.b (mon.1) 288 : audit [DBG] from='client.? v1:172.21.15.163:0/784388871' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:06:06.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.471609+0000 mon.a (mon.0) 1351 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.471905+0000 mon.a (mon.0) 1352 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:06.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.472601+0000 mon.b (mon.1) 289 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi163-83845-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:06.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:05.472913+0000 mon.a (mon.0) 1353 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi163-83845-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:06.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:06.180422+0000 mon.a (mon.0) 1354 : audit [INF] from='client.49076 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi163-83920-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:06.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:06.180557+0000 mon.a (mon.0) 1355 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:06.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:06.180624+0000 mon.a (mon.0) 1356 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi163-83845-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:06.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:06.180689+0000 mon.a (mon.0) 1357 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-02-19T21:06:06.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:06.185599+0000 mon.c (mon.2) 191 : audit [INF] from='client.? v1:172.21.15.163:0/2401115706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi163-83854-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:06.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:06.186034+0000 mon.a (mon.0) 1358 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi163-83854-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:06.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:06.187787+0000 mon.b (mon.1) 290 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13"}]: dispatch 2024-02-19T21:06:06.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:06.188829+0000 mon.a (mon.0) 1359 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13"}]: dispatch 2024-02-19T21:06:06.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:06.188877+0000 mon.b (mon.1) 291 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi163-83845-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:06.193568+0000 mon.a (mon.0) 1360 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi163-83845-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:05.472913+0000 mon.a (mon.0) 1353 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi163-83845-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:06.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:06.180422+0000 mon.a (mon.0) 1354 : audit [INF] from='client.49076 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi163-83920-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:06.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:06.180557+0000 mon.a (mon.0) 1355 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:06.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:06.180624+0000 mon.a (mon.0) 1356 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi163-83845-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:06.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:06.180689+0000 mon.a (mon.0) 1357 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-02-19T21:06:06.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:06.185599+0000 mon.c (mon.2) 191 : audit [INF] from='client.? v1:172.21.15.163:0/2401115706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi163-83854-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:06.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:06.186034+0000 mon.a (mon.0) 1358 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi163-83854-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:06.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:06.187787+0000 mon.b (mon.1) 290 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13"}]: dispatch 2024-02-19T21:06:06.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:06.188829+0000 mon.a (mon.0) 1359 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13"}]: dispatch 2024-02-19T21:06:06.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:06.188877+0000 mon.b (mon.1) 291 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi163-83845-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:06.193568+0000 mon.a (mon.0) 1360 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi163-83845-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.488 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:06.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:04.600148+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:06:06.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:04.600969+0000 osd.4 (osd.4) 94 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:06:06.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:04.865426+0000 osd.6 (osd.6) 85 : cluster [DBG] 2.24 scrub starts 2024-02-19T21:06:06.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:04.866390+0000 osd.6 (osd.6) 86 : cluster [DBG] 2.24 scrub ok 2024-02-19T21:06:06.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:05.136288+0000 mgr.y (mgr.14550) 224 : cluster [DBG] pgmap v248: 304 pgs: 16 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 240 active+clean; 30 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.2 KiB/s wr, 2 op/s 2024-02-19T21:06:06.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:05.197051+0000 mon.a (mon.0) 1349 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:06:06.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.217825+0000 mon.b (mon.1) 285 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.219291+0000 mon.a (mon.0) 1350 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.220036+0000 mon.b (mon.1) 286 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.286409+0000 mon.b (mon.1) 287 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:06.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.289699+0000 mon.c (mon.2) 190 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:06.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.314874+0000 mon.b (mon.1) 288 : audit [DBG] from='client.? v1:172.21.15.163:0/784388871' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:06:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.471609+0000 mon.a (mon.0) 1351 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.471905+0000 mon.a (mon.0) 1352 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.472601+0000 mon.b (mon.1) 289 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi163-83845-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:05.472913+0000 mon.a (mon.0) 1353 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi163-83845-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:06.180422+0000 mon.a (mon.0) 1354 : audit [INF] from='client.49076 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi163-83920-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:06.180557+0000 mon.a (mon.0) 1355 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:06.180624+0000 mon.a (mon.0) 1356 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi163-83845-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:06.180689+0000 mon.a (mon.0) 1357 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-02-19T21:06:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:06.185599+0000 mon.c (mon.2) 191 : audit [INF] from='client.? v1:172.21.15.163:0/2401115706' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi163-83854-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:06.186034+0000 mon.a (mon.0) 1358 : audit [INF] from='client.49091 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi163-83854-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:06.187787+0000 mon.b (mon.1) 290 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13"}]: dispatch 2024-02-19T21:06:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:06.188829+0000 mon.a (mon.0) 1359 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13"}]: dispatch 2024-02-19T21:06:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:06.188877+0000 mon.b (mon.1) 291 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi163-83845-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:06.193568+0000 mon.a (mon.0) 1360 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi163-83845-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:06.705 INFO:tasks.thrashosds.thrasher:j is {'epoch': 185, 'fsid': '2591809a-cf69-11ee-95bb-87774f69a715', 'created': '2024-02-19T20:55:37.794885+0000', 'modified': '2024-02-19T21:06:05.170048+0000', 'last_up_change': '2024-02-19T21:01:09.214134+0000', 'last_in_change': '2024-02-19T21:00:51.699017+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 35, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 129, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-19T20:59:09.425169+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-19T21:01:14.658945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '139', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-19T21:01:14.958800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-19T21:01:16.543816+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-19T21:01:18.521482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-19T21:01:20.586564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 8, 'pool_name': 'LibRadosList_smithi163-83888-1', 'create_time': '2024-02-19T21:04:33.426193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 25, 'last_pg_merge_meta': {'source_pgid': '8.1a', 'ready_epoch': 183, 'last_epoch_started': 182, 'last_epoch_clean': 182, 'source_version': "168'7", 'target_version': "168'14"}, 'last_change': '185', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '185', 'last_force_op_resend_preluminous': '185', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 16, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-19T21:04:35.333192+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 61, 'pool_name': 'test-rados-api-smithi163-83928-6', 'create_time': '2024-02-19T21:04:52.991355+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '185', 'last_force_op_resend': '150', 'last_force_op_resend_prenautilus': '150', 'last_force_op_resend_preluminous': '150', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 11, 'snap_epoch': 185, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [94], 'tier_of': -1, 'read_tier': 94, 'write_tier': 94, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 94, 'pool_name': 'test-rados-api-smithi163-83928-13', 'create_time': '2024-02-19T21:05:22.433403+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '185', 'last_force_op_resend': '150', 'last_force_op_resend_prenautilus': '150', 'last_force_op_resend_preluminous': '150', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 11, 'snap_epoch': 185, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 61, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 126, 'pool_name': 'LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20', 'create_time': '2024-02-19T21:06:01.441441+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 2, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '185', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 185, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 129, 'pool_name': 'LibRadosChecksum/2_smithi163-83920-33', 'create_time': '2024-02-19T21:06:04.469313+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '185', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '4a14b406-7c21-40f3-8b12-8cf9df811088', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6801', 'nonce': 1559418115}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6802', 'nonce': 1559418115}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6804', 'nonce': 1559418115}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6803', 'nonce': 1559418115}]}, 'public_addr': '172.21.15.163:6801/1559418115', 'cluster_addr': '172.21.15.163:6802/1559418115', 'heartbeat_back_addr': '172.21.15.163:6804/1559418115', 'heartbeat_front_addr': '172.21.15.163:6803/1559418115', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e74a65fa-cb9c-42c3-9293-602cf7e9dc67', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6805', 'nonce': 3125047432}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6806', 'nonce': 3125047432}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6808', 'nonce': 3125047432}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6807', 'nonce': 3125047432}]}, 'public_addr': '172.21.15.163:6805/3125047432', 'cluster_addr': '172.21.15.163:6806/3125047432', 'heartbeat_back_addr': '172.21.15.163:6808/3125047432', 'heartbeat_front_addr': '172.21.15.163:6807/3125047432', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6980a625-cdc9-487e-8f8c-7997ba716a97', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6809', 'nonce': 1563158487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6810', 'nonce': 1563158487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6812', 'nonce': 1563158487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6811', 'nonce': 1563158487}]}, 'public_addr': '172.21.15.163:6809/1563158487', 'cluster_addr': '172.21.15.163:6810/1563158487', 'heartbeat_back_addr': '172.21.15.163:6812/1563158487', 'heartbeat_front_addr': '172.21.15.163:6811/1563158487', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '359e2ee0-9d1d-4074-8119-2d69bc2ca3a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6813', 'nonce': 2112279769}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6814', 'nonce': 2112279769}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6816', 'nonce': 2112279769}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6815', 'nonce': 2112279769}]}, 'public_addr': '172.21.15.163:6813/2112279769', 'cluster_addr': '172.21.15.163:6814/2112279769', 'heartbeat_back_addr': '172.21.15.163:6816/2112279769', 'heartbeat_front_addr': '172.21.15.163:6815/2112279769', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2bf5ef05-292b-4693-85a9-97991f2091e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6800', 'nonce': 3129716965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6801', 'nonce': 3129716965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6803', 'nonce': 3129716965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6802', 'nonce': 3129716965}]}, 'public_addr': '172.21.15.193:6800/3129716965', 'cluster_addr': '172.21.15.193:6801/3129716965', 'heartbeat_back_addr': '172.21.15.193:6803/3129716965', 'heartbeat_front_addr': '172.21.15.193:6802/3129716965', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '59065f9b-9240-414b-9c91-e91cda4aed6f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6804', 'nonce': 805161991}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6805', 'nonce': 805161991}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6807', 'nonce': 805161991}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6806', 'nonce': 805161991}]}, 'public_addr': '172.21.15.193:6804/805161991', 'cluster_addr': '172.21.15.193:6805/805161991', 'heartbeat_back_addr': '172.21.15.193:6807/805161991', 'heartbeat_front_addr': '172.21.15.193:6806/805161991', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '302c95df-3842-4818-8afd-d79d8184684f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6808', 'nonce': 787604615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6809', 'nonce': 787604615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6811', 'nonce': 787604615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6810', 'nonce': 787604615}]}, 'public_addr': '172.21.15.193:6808/787604615', 'cluster_addr': '172.21.15.193:6809/787604615', 'heartbeat_back_addr': '172.21.15.193:6811/787604615', 'heartbeat_front_addr': '172.21.15.193:6810/787604615', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '9ebeff27-c2f8-4a66-923f-a0d7573ddd3f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 183, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6812', 'nonce': 157927432}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6813', 'nonce': 157927432}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6815', 'nonce': 157927432}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6814', 'nonce': 157927432}]}, 'public_addr': '172.21.15.193:6812/157927432', 'cluster_addr': '172.21.15.193:6813/157927432', 'heartbeat_back_addr': '172.21.15.193:6815/157927432', 'heartbeat_front_addr': '172.21.15.193:6814/157927432', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:05:57.520396+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:05:56.376788+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:05:22.137628+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:05:49.588903+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:05:55.719928+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:05:30.292761+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:05:51.547827+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:05:51.951656+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.d', 'osds': [1, 4, 2]}, {'pgid': '94.1c', 'osds': [4, 2, 0]}], 'pg_upmap_items': [{'pgid': '2.4', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.163:0/1427774891': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/3282102947': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:6800/2735092549': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/1235588846': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:6800/2731225015': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:0/506491651': '2024-02-20T20:56:02.185409+0000', '172.21.15.163:0/168117826': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/1458009465': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:0/541631472': '2024-02-20T20:56:02.185409+0000', '172.21.15.163:6800/2248999541': '2024-02-20T20:56:02.185409+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 11, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 25, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 57, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 61, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 76, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 85, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 92, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 94, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 126, 'snaps': [{'begin': 3, 'length': 1}]}], 'new_removed_snaps': [{'pool': 61, 'snaps': [{'begin': 4, 'length': 1}]}, {'pool': 94, 'snaps': [{'begin': 4, 'length': 1}]}, {'pool': 126, 'snaps': [{'begin': 3, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-19T21:06:06.706 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:06:06.888 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:06.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:06.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:06.919 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:06.927 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:06.943 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:06:06.943 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:06:06.993 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:07.025 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:07.045 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:07.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:07.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:07.149 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:07.182 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:07.224 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:07.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:07.328 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:07.354 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:07.504 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:07.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:07.607 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:07.634 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:07.707 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:07.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:05.428118+0000 osd.7 (osd.7) 95 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:06:07.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:05.431013+0000 osd.7 (osd.7) 96 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:06:07.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:05.720283+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.c scrub starts 2024-02-19T21:06:07.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:05.721171+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.c scrub ok 2024-02-19T21:06:07.710 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:05.819481+0000 osd.0 (osd.0) 99 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:06:07.710 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:05.822633+0000 osd.0 (osd.0) 100 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:06:07.710 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:06.290350+0000 mon.c (mon.2) 192 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:07.710 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:07.177065+0000 mon.a (mon.0) 1361 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:07.710 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:07.291005+0000 mon.c (mon.2) 193 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:07.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:05.428118+0000 osd.7 (osd.7) 95 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:06:07.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:05.431013+0000 osd.7 (osd.7) 96 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:06:07.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:05.720283+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.c scrub starts 2024-02-19T21:06:07.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:05.721171+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.c scrub ok 2024-02-19T21:06:07.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:05.819481+0000 osd.0 (osd.0) 99 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:06:07.710 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:05.822633+0000 osd.0 (osd.0) 100 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:06:07.711 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:06.290350+0000 mon.c (mon.2) 192 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:07.711 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:07.177065+0000 mon.a (mon.0) 1361 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:07.711 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:07.291005+0000 mon.c (mon.2) 193 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:07.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:07.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:07.766 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:05.428118+0000 osd.7 (osd.7) 95 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:06:07.766 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:05.431013+0000 osd.7 (osd.7) 96 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:06:07.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:05.720283+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.c scrub starts 2024-02-19T21:06:07.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:05.721171+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.c scrub ok 2024-02-19T21:06:07.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:05.819481+0000 osd.0 (osd.0) 99 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:06:07.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:05.822633+0000 osd.0 (osd.0) 100 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:06:07.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:06.290350+0000 mon.c (mon.2) 192 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:07.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:07.177065+0000 mon.a (mon.0) 1361 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:07.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:07.291005+0000 mon.c (mon.2) 193 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:07.813 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:06:07.840 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:07.858 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:06:07.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:07.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:07.962 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:07.988 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:08.006 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:08.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:08.065 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:06:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:06:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:06:08.110 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:08.135 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:08.153 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:08.156 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:08.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:08.257 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:08.283 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:08.302 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:08.304 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:08.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:08.406 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:08.433 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:08.582 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:08.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:08.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:08.680 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:06.455547+0000 osd.7 (osd.7) 97 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:06:08.685 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:08.711 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:08.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:06.458535+0000 osd.7 (osd.7) 98 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:06:08.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:06.628677+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.d scrub starts 2024-02-19T21:06:08.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:06.630818+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.d scrub ok 2024-02-19T21:06:08.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:06.905671+0000 osd.6 (osd.6) 87 : cluster [DBG] 4.a scrub starts 2024-02-19T21:06:08.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:06.909878+0000 osd.6 (osd.6) 88 : cluster [DBG] 4.a scrub ok 2024-02-19T21:06:08.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:07.136749+0000 mgr.y (mgr.14550) 225 : cluster [DBG] pgmap v251: 336 pgs: 16 active+clean+snaptrim, 64 unknown, 48 active+clean+snaptrim_wait, 208 active+clean; 30 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-19T21:06:08.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:07.181926+0000 mon.a (mon.0) 1362 : audit [INF] from='client.49091 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi163-83854-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:08.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:07.182053+0000 mon.a (mon.0) 1363 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13"}]': finished 2024-02-19T21:06:08.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:07.440109+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-02-19T21:06:08.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:07.459373+0000 mon.a (mon.0) 1365 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi163-83920-36"}]: dispatch 2024-02-19T21:06:08.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:07.460408+0000 mon.a (mon.0) 1366 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi163-83920-36"}]: dispatch 2024-02-19T21:06:08.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:07.460667+0000 mon.a (mon.0) 1367 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi163-83920-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:08.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:08.182592+0000 mon.a (mon.0) 1368 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi163-83845-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi163-83845-27"}]': finished 2024-02-19T21:06:08.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:08.182758+0000 mon.a (mon.0) 1369 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi163-83920-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:08.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:08.182817+0000 mon.a (mon.0) 1370 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-02-19T21:06:08.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:08.191893+0000 mon.a (mon.0) 1371 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi163-83920-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi163-83920-36"}]: dispatch 2024-02-19T21:06:08.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:06.455547+0000 osd.7 (osd.7) 97 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:06:08.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:06.458535+0000 osd.7 (osd.7) 98 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:06:08.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:06.628677+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.d scrub starts 2024-02-19T21:06:08.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:06.630818+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.d scrub ok 2024-02-19T21:06:08.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:06.905671+0000 osd.6 (osd.6) 87 : cluster [DBG] 4.a scrub starts 2024-02-19T21:06:08.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:06.909878+0000 osd.6 (osd.6) 88 : cluster [DBG] 4.a scrub ok 2024-02-19T21:06:08.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:07.136749+0000 mgr.y (mgr.14550) 225 : cluster [DBG] pgmap v251: 336 pgs: 16 active+clean+snaptrim, 64 unknown, 48 active+clean+snaptrim_wait, 208 active+clean; 30 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-19T21:06:08.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:07.181926+0000 mon.a (mon.0) 1362 : audit [INF] from='client.49091 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi163-83854-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:08.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:07.182053+0000 mon.a (mon.0) 1363 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13"}]': finished 2024-02-19T21:06:08.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:07.440109+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-02-19T21:06:08.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:07.459373+0000 mon.a (mon.0) 1365 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi163-83920-36"}]: dispatch 2024-02-19T21:06:08.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:07.460408+0000 mon.a (mon.0) 1366 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi163-83920-36"}]: dispatch 2024-02-19T21:06:08.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:07.460667+0000 mon.a (mon.0) 1367 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi163-83920-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:08.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:08.182592+0000 mon.a (mon.0) 1368 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi163-83845-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi163-83845-27"}]': finished 2024-02-19T21:06:08.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:08.182758+0000 mon.a (mon.0) 1369 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi163-83920-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:08.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:08.182817+0000 mon.a (mon.0) 1370 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-02-19T21:06:08.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:08.191893+0000 mon.a (mon.0) 1371 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi163-83920-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi163-83920-36"}]: dispatch 2024-02-19T21:06:09.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:06.455547+0000 osd.7 (osd.7) 97 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:06:09.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:06.458535+0000 osd.7 (osd.7) 98 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:06:09.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:06.628677+0000 osd.4 (osd.4) 95 : cluster [DBG] 4.d scrub starts 2024-02-19T21:06:09.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:06.630818+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.d scrub ok 2024-02-19T21:06:09.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:06.905671+0000 osd.6 (osd.6) 87 : cluster [DBG] 4.a scrub starts 2024-02-19T21:06:09.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:06.909878+0000 osd.6 (osd.6) 88 : cluster [DBG] 4.a scrub ok 2024-02-19T21:06:09.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:07.136749+0000 mgr.y (mgr.14550) 225 : cluster [DBG] pgmap v251: 336 pgs: 16 active+clean+snaptrim, 64 unknown, 48 active+clean+snaptrim_wait, 208 active+clean; 30 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-02-19T21:06:09.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:07.181926+0000 mon.a (mon.0) 1362 : audit [INF] from='client.49091 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi163-83854-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:09.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:07.182053+0000 mon.a (mon.0) 1363 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-13"}]': finished 2024-02-19T21:06:09.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:07.440109+0000 mon.a (mon.0) 1364 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-02-19T21:06:09.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:07.459373+0000 mon.a (mon.0) 1365 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi163-83920-36"}]: dispatch 2024-02-19T21:06:09.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:07.460408+0000 mon.a (mon.0) 1366 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi163-83920-36"}]: dispatch 2024-02-19T21:06:09.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:07.460667+0000 mon.a (mon.0) 1367 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi163-83920-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:09.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:08.182592+0000 mon.a (mon.0) 1368 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi163-83845-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi163-83845-27"}]': finished 2024-02-19T21:06:09.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:08.182758+0000 mon.a (mon.0) 1369 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi163-83920-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:09.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:08.182817+0000 mon.a (mon.0) 1370 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-02-19T21:06:09.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:08.191893+0000 mon.a (mon.0) 1371 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi163-83920-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi163-83920-36"}]: dispatch 2024-02-19T21:06:09.659 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:09.662 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:09.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:09.763 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:09.796 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:09.817 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:09.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:09.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:09.921 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:06:09.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:07.408752+0000 osd.7 (osd.7) 99 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:06:09.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:07.411659+0000 osd.7 (osd.7) 100 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:06:09.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:07.759449+0000 osd.3 (osd.3) 103 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:06:09.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:07.760930+0000 osd.3 (osd.3) 104 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:06:09.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:07.871448+0000 osd.1 (osd.1) 109 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:06:09.942 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:07.874358+0000 osd.1 (osd.1) 110 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:06:09.943 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:08.291715+0000 mon.c (mon.2) 194 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:09.943 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:08.756358+0000 osd.3 (osd.3) 105 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:06:09.943 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:08.757194+0000 osd.3 (osd.3) 106 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:06:09.943 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:09.099236+0000 mon.a (mon.0) 1372 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:06:09.943 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:09.137792+0000 mon.b (mon.1) 292 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-19T21:06:09.943 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:09.138816+0000 mon.a (mon.0) 1373 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-19T21:06:09.943 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:09.292448+0000 mon.c (mon.2) 195 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:09.943 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:09.430627+0000 mon.a (mon.0) 1374 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-19T21:06:09.943 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:09.430727+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-02-19T21:06:09.943 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:09.431377+0000 mon.c (mon.2) 196 : audit [INF] from='client.? v1:172.21.15.163:0/2106617923' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi163-83854-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:09.943 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:09.431589+0000 mon.b (mon.1) 293 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:09.943 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:09.432722+0000 mon.a (mon.0) 1376 : audit [INF] from='client.49100 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi163-83854-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:09.943 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:09.433162+0000 mon.a (mon.0) 1377 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:07.408752+0000 osd.7 (osd.7) 99 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:06:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:07.411659+0000 osd.7 (osd.7) 100 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:06:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:07.759449+0000 osd.3 (osd.3) 103 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:06:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:07.760930+0000 osd.3 (osd.3) 104 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:06:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:07.871448+0000 osd.1 (osd.1) 109 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:06:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:07.874358+0000 osd.1 (osd.1) 110 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:06:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:08.291715+0000 mon.c (mon.2) 194 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:08.756358+0000 osd.3 (osd.3) 105 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:06:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:08.757194+0000 osd.3 (osd.3) 106 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:06:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:09.099236+0000 mon.a (mon.0) 1372 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:06:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:09.137792+0000 mon.b (mon.1) 292 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-19T21:06:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:09.138816+0000 mon.a (mon.0) 1373 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-19T21:06:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:09.292448+0000 mon.c (mon.2) 195 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:09.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:09.430627+0000 mon.a (mon.0) 1374 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-19T21:06:09.945 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:09.430727+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-02-19T21:06:09.945 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:09.431377+0000 mon.c (mon.2) 196 : audit [INF] from='client.? v1:172.21.15.163:0/2106617923' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi163-83854-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:09.945 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:09.431589+0000 mon.b (mon.1) 293 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:09.945 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:09.432722+0000 mon.a (mon.0) 1376 : audit [INF] from='client.49100 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi163-83854-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:09.945 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:09.433162+0000 mon.a (mon.0) 1377 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:09.949 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:07.408752+0000 osd.7 (osd.7) 99 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:06:09.949 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:07.411659+0000 osd.7 (osd.7) 100 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:06:09.949 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:07.759449+0000 osd.3 (osd.3) 103 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:06:09.949 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:07.760930+0000 osd.3 (osd.3) 104 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:06:09.949 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:07.871448+0000 osd.1 (osd.1) 109 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:06:09.949 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:07.874358+0000 osd.1 (osd.1) 110 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:06:09.949 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:08.291715+0000 mon.c (mon.2) 194 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:09.949 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:08.756358+0000 osd.3 (osd.3) 105 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:06:09.950 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:08.757194+0000 osd.3 (osd.3) 106 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:06:09.950 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:09.099236+0000 mon.a (mon.0) 1372 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:06:09.950 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:09.137792+0000 mon.b (mon.1) 292 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-19T21:06:09.950 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:09.138816+0000 mon.a (mon.0) 1373 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-02-19T21:06:09.950 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:09.292448+0000 mon.c (mon.2) 195 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:09.950 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:09.430627+0000 mon.a (mon.0) 1374 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-02-19T21:06:09.950 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:09.430727+0000 mon.a (mon.0) 1375 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-02-19T21:06:09.950 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:09.431377+0000 mon.c (mon.2) 196 : audit [INF] from='client.? v1:172.21.15.163:0/2106617923' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi163-83854-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:09.950 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:09.431589+0000 mon.b (mon.1) 293 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:09.950 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:09.432722+0000 mon.a (mon.0) 1376 : audit [INF] from='client.49100 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi163-83854-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:09.950 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:09.433162+0000 mon.a (mon.0) 1377 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:09.951 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:09.971 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:06:09.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:09.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:06:10.075 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:06:10.102 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:10.121 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:06:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:10.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:06:10.225 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:10.250 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:06:10.256 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:10.270 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:10.273 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:10.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:10.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.328+0000 7f21d4f4d700 1 -- 172.21.15.163:0/2541525768 >> v1:172.21.15.163:6790/0 conn(0x7f21d00fd980 legacy=0x7f21d00fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:10.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.328+0000 7f21d4f4d700 1 -- 172.21.15.163:0/2541525768 shutdown_connections 2024-02-19T21:06:10.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.328+0000 7f21d4f4d700 1 -- 172.21.15.163:0/2541525768 >> 172.21.15.163:0/2541525768 conn(0x7f21d00f35a0 msgr2=0x7f21d00f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:10.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.328+0000 7f21d4f4d700 1 -- 172.21.15.163:0/2541525768 shutdown_connections 2024-02-19T21:06:10.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.328+0000 7f21d4f4d700 1 -- 172.21.15.163:0/2541525768 wait complete. 2024-02-19T21:06:10.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.329+0000 7f21d4f4d700 1 librados: starting msgr at 2024-02-19T21:06:10.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.329+0000 7f21d4f4d700 1 librados: starting objecter 2024-02-19T21:06:10.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.329+0000 7f21d4f4d700 1 Processor -- start 2024-02-19T21:06:10.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.329+0000 7f21d4f4d700 1 -- start start 2024-02-19T21:06:10.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.329+0000 7f21d4f4d700 1 librados: setting wanted keys 2024-02-19T21:06:10.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.329+0000 7f21d4f4d700 1 librados: calling monclient init 2024-02-19T21:06:10.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.329+0000 7f21d4f4d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f21d0117380 con 0x7f21d01042c0 2024-02-19T21:06:10.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.329+0000 7f21d4f4d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f21d0118540 con 0x7f21d00fd980 2024-02-19T21:06:10.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.329+0000 7f21d4f4d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f21d0119740 con 0x7f21d0100780 2024-02-19T21:06:10.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21cdd9b700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f21d0100780 0x7f21d00fa0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38124/0 (socket says 172.21.15.163:38124) 2024-02-19T21:06:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21cdd9b700 1 -- 172.21.15.163:0/1513816879 learned_addr learned my addr 172.21.15.163:0/1513816879 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21ced9d700 1 --1- 172.21.15.163:0/1513816879 >> v1:172.21.15.163:6789/0 conn(0x7f21d01042c0 0x7f21d0115a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:52050/0 (socket says 172.21.15.163:52050) 2024-02-19T21:06:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21ce59c700 1 --1- 172.21.15.163:0/1513816879 >> v1:172.21.15.193:6789/0 conn(0x7f21d00fd980 0x7f21d00f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:38812/0 (socket says 172.21.15.163:38812) 2024-02-19T21:06:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 10 client.?.objecter ms_handle_connect 0x7f21d0100780 2024-02-19T21:06:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 10 client.?.objecter ms_handle_connect 0x7f21d01042c0 2024-02-19T21:06:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 10 client.?.objecter ms_handle_connect 0x7f21d00fd980 2024-02-19T21:06:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706313626 0 0) 0x7f21d0118540 con 0x7f21d00fd980 2024-02-19T21:06:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b80033f0 con 0x7f21d00fd980 2024-02-19T21:06:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959824008 0 0) 0x7f21d0119740 con 0x7f21d0100780 2024-02-19T21:06:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21d0118540 con 0x7f21d0100780 2024-02-19T21:06:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298403548 0 0) 0x7f21d0117380 con 0x7f21d01042c0 2024-02-19T21:06:10.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21d0119740 con 0x7f21d01042c0 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3357990117 0 0) 0x7f21b80033f0 con 0x7f21d00fd980 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21d0117380 con 0x7f21d00fd980 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f21c0001360 con 0x7f21d00fd980 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1398810361 0 0) 0x7f21d0118540 con 0x7f21d0100780 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21b80033f0 con 0x7f21d0100780 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 273726802 0 0) 0x7f21d0119740 con 0x7f21d01042c0 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21d0118540 con 0x7f21d01042c0 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f21c4001770 con 0x7f21d0100780 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f21c8001940 con 0x7f21d01042c0 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3386601646 0 0) 0x7f21d0117380 con 0x7f21d00fd980 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 >> v1:172.21.15.163:6790/0 conn(0x7f21d0100780 legacy=0x7f21d00fa0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 >> v1:172.21.15.163:6789/0 conn(0x7f21d01042c0 legacy=0x7f21d0115a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d011a940 con 0x7f21d00fd980 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d0117570 con 0x7f21d00fd980 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21d4f4d700 10 client.47923.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:10.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d01179e0 con 0x7f21d00fd980 2024-02-19T21:06:10.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21d4f4d700 1 librados: init done 2024-02-19T21:06:10.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f21c00026a0 con 0x7f21d00fd980 2024-02-19T21:06:10.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.330+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f21c0003b20 con 0x7f21d00fd980 2024-02-19T21:06:10.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.331+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f21b0004f00 con 0x7f21d00fd980 2024-02-19T21:06:10.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f21c0001b80 con 0x7f21d00fd980 2024-02-19T21:06:10.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(190..190 src has 43..190) v4 ==== 10761+0+0 (unknown 1054489016 0 0) 0x7f21c0079d60 con 0x7f21d00fd980 2024-02-19T21:06:10.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 osd_map(190..190 src has 43..190) v4 2024-02-19T21:06:10.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21bf7fe700 3 client.47923.objecter handle_osd_map got epochs [190,190] > 0 2024-02-19T21:06:10.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21bf7fe700 3 client.47923.objecter handle_osd_map decoding full epoch 190 2024-02-19T21:06:10.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21bf7fe700 20 client.47923.objecter dump_active .. 0 homeless 2024-02-19T21:06:10.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v190) v1 ==== 54+0+16 (unknown 2197449364 0 2420000192) 0x7f21c007c150 con 0x7f21d00fd980 2024-02-19T21:06:10.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21bf7fe700 10 client.47923.objecter ms_handle_connect 0x7f21b8063ec0 2024-02-19T21:06:10.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 1 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b00051a0 osd=0 3 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b00051a0 osd=0 3 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b00051a0 osd=0 3 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 1 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b00051a0 osd=0 3 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 0 1 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b00051a0 osd=0 4 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21d4f4d700 10 client.47923.objecter _send_command 1 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.332+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b0004950 con 0x7f21b0004530 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.333+0000 7f21bf7fe700 10 client.47923.objecter ms_handle_connect 0x7f21b0004530 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.333+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f21b0004950 con 0x7f21b0004530 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.333+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.333+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 1 = system:0 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.333+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b00051a0 osd=0 4 2024-02-19T21:06:10.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.333+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 0 1 2024-02-19T21:06:10.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.351+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:10.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.351+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 2 2024-02-19T21:06:10.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.351+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b00051a0 osd=0 3 2024-02-19T21:06:10.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.351+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b00051a0 osd=0 3 2024-02-19T21:06:10.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.351+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b00051a0 osd=0 3 2024-02-19T21:06:10.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.351+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 2 2024-02-19T21:06:10.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.351+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b00051a0 osd=0 3 2024-02-19T21:06:10.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.351+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 0 2 2024-02-19T21:06:10.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.351+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b00051a0 osd=0 4 2024-02-19T21:06:10.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.351+0000 7f21d4f4d700 10 client.47923.objecter _send_command 2 2024-02-19T21:06:10.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.351+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21b0004d30 con 0x7f21b0004530 2024-02-19T21:06:10.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f21b0004d30 con 0x7f21b0004530 2024-02-19T21:06:10.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:10.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b00051a0 osd=0 4 2024-02-19T21:06:10.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 0 2 2024-02-19T21:06:10.347 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:06:10.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:10.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 3 2024-02-19T21:06:10.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0003fe0 osd=1 3 2024-02-19T21:06:10.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0003fe0 osd=1 3 2024-02-19T21:06:10.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0003fe0 osd=1 3 2024-02-19T21:06:10.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 3 2024-02-19T21:06:10.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b0003fe0 osd=1 3 2024-02-19T21:06:10.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 1 3 2024-02-19T21:06:10.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0003fe0 osd=1 4 2024-02-19T21:06:10.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21d4f4d700 10 client.47923.objecter _send_command 3 2024-02-19T21:06:10.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b000c000 con 0x7f21b0004150 2024-02-19T21:06:10.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.353+0000 7f21bf7fe700 10 client.47923.objecter ms_handle_connect 0x7f21b0004150 2024-02-19T21:06:10.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.354+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f21b000c000 con 0x7f21b0004150 2024-02-19T21:06:10.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.354+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:06:10.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.354+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 3 = system:0 2024-02-19T21:06:10.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.354+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b0003fe0 osd=1 4 2024-02-19T21:06:10.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.354+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 1 3 2024-02-19T21:06:10.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.369+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:10.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.369+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 4 2024-02-19T21:06:10.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.369+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0003fe0 osd=1 3 2024-02-19T21:06:10.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.369+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0003fe0 osd=1 3 2024-02-19T21:06:10.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.369+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0003fe0 osd=1 3 2024-02-19T21:06:10.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.369+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 4 2024-02-19T21:06:10.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.369+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b0003fe0 osd=1 3 2024-02-19T21:06:10.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.369+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 1 4 2024-02-19T21:06:10.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.369+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0003fe0 osd=1 4 2024-02-19T21:06:10.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.369+0000 7f21d4f4d700 10 client.47923.objecter _send_command 4 2024-02-19T21:06:10.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.369+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21b0001850 con 0x7f21b0004150 2024-02-19T21:06:10.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f21b0001850 con 0x7f21b0004150 2024-02-19T21:06:10.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:10.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b0003fe0 osd=1 4 2024-02-19T21:06:10.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 1 4 2024-02-19T21:06:10.366 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:06:10.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:10.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 5 2024-02-19T21:06:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0003630 osd=2 3 2024-02-19T21:06:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0003630 osd=2 3 2024-02-19T21:06:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0003630 osd=2 3 2024-02-19T21:06:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 5 2024-02-19T21:06:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b0003630 osd=2 3 2024-02-19T21:06:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 2 5 2024-02-19T21:06:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0003630 osd=2 4 2024-02-19T21:06:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21d4f4d700 10 client.47923.objecter _send_command 5 2024-02-19T21:06:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.371+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b0003e70 con 0x7f21b000d2f0 2024-02-19T21:06:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.372+0000 7f21bf7fe700 10 client.47923.objecter ms_handle_connect 0x7f21b000d2f0 2024-02-19T21:06:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.372+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f21b0003e70 con 0x7f21b000d2f0 2024-02-19T21:06:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.372+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:06:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.372+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 5 = system:0 2024-02-19T21:06:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.372+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b0003630 osd=2 4 2024-02-19T21:06:10.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.372+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 2 5 2024-02-19T21:06:10.374 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:10.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.387+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:10.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.387+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 6 2024-02-19T21:06:10.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.387+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0003630 osd=2 3 2024-02-19T21:06:10.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.387+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0003630 osd=2 3 2024-02-19T21:06:10.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.387+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0003630 osd=2 3 2024-02-19T21:06:10.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.387+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 6 2024-02-19T21:06:10.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.387+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b0003630 osd=2 3 2024-02-19T21:06:10.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.387+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 2 6 2024-02-19T21:06:10.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.387+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0003630 osd=2 4 2024-02-19T21:06:10.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.387+0000 7f21d4f4d700 10 client.47923.objecter _send_command 6 2024-02-19T21:06:10.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.387+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21b000e6e0 con 0x7f21b000d2f0 2024-02-19T21:06:10.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f21b000e6e0 con 0x7f21b000d2f0 2024-02-19T21:06:10.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:10.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b0003630 osd=2 4 2024-02-19T21:06:10.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 2 6 2024-02-19T21:06:10.383 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:06:10.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:10.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 7 2024-02-19T21:06:10.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b00033e0 osd=3 3 2024-02-19T21:06:10.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b00033e0 osd=3 3 2024-02-19T21:06:10.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b00033e0 osd=3 3 2024-02-19T21:06:10.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 7 2024-02-19T21:06:10.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b00033e0 osd=3 3 2024-02-19T21:06:10.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 3 7 2024-02-19T21:06:10.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b00033e0 osd=3 4 2024-02-19T21:06:10.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21d4f4d700 10 client.47923.objecter _send_command 7 2024-02-19T21:06:10.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.388+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b0015b00 con 0x7f21b0011e80 2024-02-19T21:06:10.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.389+0000 7f21bf7fe700 10 client.47923.objecter ms_handle_connect 0x7f21b0011e80 2024-02-19T21:06:10.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.389+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f21b0015b00 con 0x7f21b0011e80 2024-02-19T21:06:10.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.389+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:06:10.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.389+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 7 = system:0 2024-02-19T21:06:10.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.389+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b00033e0 osd=3 4 2024-02-19T21:06:10.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.389+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 3 7 2024-02-19T21:06:10.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.404+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:10.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.404+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 8 2024-02-19T21:06:10.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.404+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b00033e0 osd=3 3 2024-02-19T21:06:10.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.404+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b00033e0 osd=3 3 2024-02-19T21:06:10.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.404+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b00033e0 osd=3 3 2024-02-19T21:06:10.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.404+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 8 2024-02-19T21:06:10.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.404+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b00033e0 osd=3 3 2024-02-19T21:06:10.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.404+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 3 8 2024-02-19T21:06:10.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.404+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b00033e0 osd=3 4 2024-02-19T21:06:10.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.404+0000 7f21d4f4d700 10 client.47923.objecter _send_command 8 2024-02-19T21:06:10.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.404+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21b0015b00 con 0x7f21b0011e80 2024-02-19T21:06:10.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f21b0015b00 con 0x7f21b0011e80 2024-02-19T21:06:10.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:10.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b00033e0 osd=3 4 2024-02-19T21:06:10.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 3 8 2024-02-19T21:06:10.399 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:06:10.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:10.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 9 2024-02-19T21:06:10.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0015b00 osd=4 3 2024-02-19T21:06:10.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0015b00 osd=4 3 2024-02-19T21:06:10.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0015b00 osd=4 3 2024-02-19T21:06:10.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 9 2024-02-19T21:06:10.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b0015b00 osd=4 3 2024-02-19T21:06:10.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 4 9 2024-02-19T21:06:10.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0015b00 osd=4 4 2024-02-19T21:06:10.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21d4f4d700 10 client.47923.objecter _send_command 9 2024-02-19T21:06:10.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.405+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b000cec0 con 0x7f21b000c940 2024-02-19T21:06:10.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.406+0000 7f21bf7fe700 10 client.47923.objecter ms_handle_connect 0x7f21b000c940 2024-02-19T21:06:10.401 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:10.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.406+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f21b000cec0 con 0x7f21b000c940 2024-02-19T21:06:10.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.406+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:06:10.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.406+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 9 = system:0 2024-02-19T21:06:10.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.406+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b0015b00 osd=4 4 2024-02-19T21:06:10.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.406+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 4 9 2024-02-19T21:06:10.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.421+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:10.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.421+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 10 2024-02-19T21:06:10.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.421+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0015b00 osd=4 3 2024-02-19T21:06:10.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.421+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0015b00 osd=4 3 2024-02-19T21:06:10.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.421+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0015b00 osd=4 3 2024-02-19T21:06:10.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.421+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 10 2024-02-19T21:06:10.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.421+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b0015b00 osd=4 3 2024-02-19T21:06:10.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.421+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 4 10 2024-02-19T21:06:10.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.421+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0015b00 osd=4 4 2024-02-19T21:06:10.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.421+0000 7f21d4f4d700 10 client.47923.objecter _send_command 10 2024-02-19T21:06:10.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.421+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21b000cec0 con 0x7f21b000c940 2024-02-19T21:06:10.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f21b000cec0 con 0x7f21b000c940 2024-02-19T21:06:10.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:10.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b0015b00 osd=4 4 2024-02-19T21:06:10.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 4 10 2024-02-19T21:06:10.417 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:06:10.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:10.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 11 2024-02-19T21:06:10.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b000cec0 osd=5 3 2024-02-19T21:06:10.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b000cec0 osd=5 3 2024-02-19T21:06:10.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b000cec0 osd=5 3 2024-02-19T21:06:10.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 11 2024-02-19T21:06:10.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b000cec0 osd=5 3 2024-02-19T21:06:10.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 5 11 2024-02-19T21:06:10.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b000cec0 osd=5 4 2024-02-19T21:06:10.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21d4f4d700 10 client.47923.objecter _send_command 11 2024-02-19T21:06:10.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.423+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b0017ea0 con 0x7f21b0017a40 2024-02-19T21:06:10.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.424+0000 7f21bf7fe700 10 client.47923.objecter ms_handle_connect 0x7f21b0017a40 2024-02-19T21:06:10.419 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:10.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.424+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f21b0017ea0 con 0x7f21b0017a40 2024-02-19T21:06:10.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.424+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:06:10.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.424+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 11 = system:0 2024-02-19T21:06:10.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.424+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b000cec0 osd=5 4 2024-02-19T21:06:10.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.424+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 5 11 2024-02-19T21:06:10.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:10.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:10.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.438+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:10.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.438+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 12 2024-02-19T21:06:10.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.438+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b000cec0 osd=5 3 2024-02-19T21:06:10.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.438+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b000cec0 osd=5 3 2024-02-19T21:06:10.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.438+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b000cec0 osd=5 3 2024-02-19T21:06:10.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.438+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 12 2024-02-19T21:06:10.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.438+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b000cec0 osd=5 3 2024-02-19T21:06:10.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.438+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 5 12 2024-02-19T21:06:10.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.438+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b000cec0 osd=5 4 2024-02-19T21:06:10.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.438+0000 7f21d4f4d700 10 client.47923.objecter _send_command 12 2024-02-19T21:06:10.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.438+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21b0017ea0 con 0x7f21b0017a40 2024-02-19T21:06:10.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f21b0017ea0 con 0x7f21b0017a40 2024-02-19T21:06:10.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:10.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b000cec0 osd=5 4 2024-02-19T21:06:10.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 5 12 2024-02-19T21:06:10.435 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:06:10.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:10.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 13 2024-02-19T21:06:10.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0017ea0 osd=6 3 2024-02-19T21:06:10.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0017ea0 osd=6 3 2024-02-19T21:06:10.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0017ea0 osd=6 3 2024-02-19T21:06:10.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 13 2024-02-19T21:06:10.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b0017ea0 osd=6 3 2024-02-19T21:06:10.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 6 13 2024-02-19T21:06:10.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0017ea0 osd=6 4 2024-02-19T21:06:10.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21d4f4d700 10 client.47923.objecter _send_command 13 2024-02-19T21:06:10.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.440+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b0027000 con 0x7f21b00234f0 2024-02-19T21:06:10.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.441+0000 7f21bf7fe700 10 client.47923.objecter ms_handle_connect 0x7f21b00234f0 2024-02-19T21:06:10.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.441+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f21b0027000 con 0x7f21b00234f0 2024-02-19T21:06:10.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.441+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:06:10.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.441+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 13 = system:0 2024-02-19T21:06:10.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.441+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b0017ea0 osd=6 4 2024-02-19T21:06:10.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.441+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 6 13 2024-02-19T21:06:10.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.456+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:10.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.456+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 14 2024-02-19T21:06:10.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.456+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0017ea0 osd=6 3 2024-02-19T21:06:10.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.456+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0017ea0 osd=6 3 2024-02-19T21:06:10.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.456+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0017ea0 osd=6 3 2024-02-19T21:06:10.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.456+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 14 2024-02-19T21:06:10.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.456+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b0017ea0 osd=6 3 2024-02-19T21:06:10.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.456+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 6 14 2024-02-19T21:06:10.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.456+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0017ea0 osd=6 4 2024-02-19T21:06:10.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.456+0000 7f21d4f4d700 10 client.47923.objecter _send_command 14 2024-02-19T21:06:10.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.456+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21b0027000 con 0x7f21b00234f0 2024-02-19T21:06:10.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f21b0027000 con 0x7f21b00234f0 2024-02-19T21:06:10.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:10.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b0017ea0 osd=6 4 2024-02-19T21:06:10.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 6 14 2024-02-19T21:06:10.452 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:06:10.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:10.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 15 2024-02-19T21:06:10.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0027000 osd=7 3 2024-02-19T21:06:10.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0027000 osd=7 3 2024-02-19T21:06:10.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0027000 osd=7 3 2024-02-19T21:06:10.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 15 2024-02-19T21:06:10.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b0027000 osd=7 3 2024-02-19T21:06:10.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 7 15 2024-02-19T21:06:10.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0027000 osd=7 4 2024-02-19T21:06:10.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21d4f4d700 10 client.47923.objecter _send_command 15 2024-02-19T21:06:10.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b002b370 con 0x7f21b00276b0 2024-02-19T21:06:10.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.458+0000 7f21bf7fe700 10 client.47923.objecter ms_handle_connect 0x7f21b00276b0 2024-02-19T21:06:10.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.459+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f21b002b370 con 0x7f21b00276b0 2024-02-19T21:06:10.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.459+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:06:10.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.459+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 15 = system:0 2024-02-19T21:06:10.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.459+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b0027000 osd=7 4 2024-02-19T21:06:10.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.459+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 7 15 2024-02-19T21:06:10.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.473+0000 7f21d4f4d700 10 client.47923.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:10.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.473+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign -1 16 2024-02-19T21:06:10.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.473+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0027000 osd=7 3 2024-02-19T21:06:10.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.473+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0027000 osd=7 3 2024-02-19T21:06:10.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.473+0000 7f21d4f4d700 20 client.47923.objecter _get_session s=0x7f21b0027000 osd=7 3 2024-02-19T21:06:10.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.473+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_remove -1 16 2024-02-19T21:06:10.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.473+0000 7f21d4f4d700 20 client.47923.objecter get_session s=0x7f21b0027000 osd=7 3 2024-02-19T21:06:10.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.473+0000 7f21d4f4d700 15 client.47923.objecter _session_command_op_assign 7 16 2024-02-19T21:06:10.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.473+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0027000 osd=7 4 2024-02-19T21:06:10.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.473+0000 7f21d4f4d700 10 client.47923.objecter _send_command 16 2024-02-19T21:06:10.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.473+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f21b002cb20 con 0x7f21b00276b0 2024-02-19T21:06:10.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21bf7fe700 1 -- 172.21.15.163:0/1513816879 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f21b002cb20 con 0x7f21b00276b0 2024-02-19T21:06:10.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21bf7fe700 10 client.47923.objecter ms_dispatch 0x7f21d0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:10.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21bf7fe700 10 client.47923.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21bf7fe700 20 client.47923.objecter put_session s=0x7f21b0027000 osd=7 4 2024-02-19T21:06:10.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21bf7fe700 15 client.47923.objecter _session_command_op_remove 7 16 2024-02-19T21:06:10.469 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:06:10.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 10 librados: watch_flush enter 2024-02-19T21:06:10.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 10 librados: watch_flush exit 2024-02-19T21:06:10.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 10 client.47923.objecter close_session for osd.0 2024-02-19T21:06:10.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 >> v1:172.21.15.163:6801/1559418115 conn(0x7f21b0004530 legacy=0x7f21b0002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:10.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b00051a0 osd=0 1 2024-02-19T21:06:10.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 10 client.47923.objecter close_session for osd.1 2024-02-19T21:06:10.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 >> v1:172.21.15.163:6805/3125047432 conn(0x7f21b0004150 legacy=0x7f21b000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:10.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0003fe0 osd=1 1 2024-02-19T21:06:10.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 10 client.47923.objecter close_session for osd.2 2024-02-19T21:06:10.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 >> v1:172.21.15.163:6809/1563158487 conn(0x7f21b000d2f0 legacy=0x7f21b000d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:10.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0003630 osd=2 1 2024-02-19T21:06:10.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 10 client.47923.objecter close_session for osd.3 2024-02-19T21:06:10.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 >> v1:172.21.15.163:6813/2112279769 conn(0x7f21b0011e80 legacy=0x7f21b0014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:10.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b00033e0 osd=3 1 2024-02-19T21:06:10.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 10 client.47923.objecter close_session for osd.4 2024-02-19T21:06:10.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 >> v1:172.21.15.193:6800/3129716965 conn(0x7f21b000c940 legacy=0x7f21b0015ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:10.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0015b00 osd=4 1 2024-02-19T21:06:10.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 10 client.47923.objecter close_session for osd.5 2024-02-19T21:06:10.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 >> v1:172.21.15.193:6804/805161991 conn(0x7f21b0017a40 legacy=0x7f21b00218c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:10.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21ced9d700 1 -- 172.21.15.163:0/1513816879 reap_dead start 2024-02-19T21:06:10.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21ced9d700 1 -- 172.21.15.163:0/1513816879 reap_dead start 2024-02-19T21:06:10.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b000cec0 osd=5 1 2024-02-19T21:06:10.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 10 client.47923.objecter close_session for osd.6 2024-02-19T21:06:10.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 >> v1:172.21.15.193:6808/787604615 conn(0x7f21b00234f0 legacy=0x7f21b00258e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:10.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0017ea0 osd=6 1 2024-02-19T21:06:10.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21ced9d700 1 -- 172.21.15.163:0/1513816879 reap_dead start 2024-02-19T21:06:10.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 10 client.47923.objecter close_session for osd.7 2024-02-19T21:06:10.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 >> v1:172.21.15.193:6812/157927432 conn(0x7f21b00276b0 legacy=0x7f21b0029ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:10.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 20 client.47923.objecter put_session s=0x7f21b0027000 osd=7 1 2024-02-19T21:06:10.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 20 client.47923.objecter shutdown clearing up homeless session... 2024-02-19T21:06:10.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.475+0000 7f21d4f4d700 10 client.47923.objecter successfully canceled tick 2024-02-19T21:06:10.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.476+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 >> v1:172.21.15.163:6800/1552514681 conn(0x7f21b8063ec0 legacy=0x7f21b8066380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:10.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.476+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 >> v1:172.21.15.193:6789/0 conn(0x7f21d00fd980 legacy=0x7f21d00f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:10.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.476+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 shutdown_connections 2024-02-19T21:06:10.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.476+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 >> 172.21.15.163:0/1513816879 conn(0x7f21d00f35a0 msgr2=0x7f21d0107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:10.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.476+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 shutdown_connections 2024-02-19T21:06:10.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.476+0000 7f21d4f4d700 1 -- 172.21.15.163:0/1513816879 wait complete. 2024-02-19T21:06:10.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:10.476+0000 7f21d4f4d700 1 librados: shutdown 2024-02-19T21:06:10.473 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.473 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.473 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.473 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.473 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.473 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.474 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.474 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:06:10.522 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:10.548 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:10.567 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:10.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:10.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:10.671 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:10.697 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:10.716 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:10.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:10.719 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-19T21:06:10.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:09.137149+0000 mgr.y (mgr.14550) 226 : cluster [DBG] pgmap v254: 247 pgs: 10 active+clean+snaptrim, 8 unknown, 22 active+clean+snaptrim_wait, 207 active+clean; 28 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:10.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:09.418956+0000 osd.7 (osd.7) 101 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:06:10.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:09.422003+0000 osd.7 (osd.7) 102 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:06:10.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:09.593829+0000 osd.4 (osd.4) 97 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:06:10.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:09.595747+0000 osd.4 (osd.4) 98 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:06:10.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:09.761586+0000 osd.3 (osd.3) 107 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:06:10.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:09.763477+0000 osd.3 (osd.3) 108 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:06:10.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:09.850032+0000 osd.5 (osd.5) 88 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:06:10.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:09.852993+0000 osd.5 (osd.5) 89 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:06:10.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:10.191023+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi163-83920-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi163-83920-36"}]': finished 2024-02-19T21:06:10.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:10.191190+0000 mon.a (mon.0) 1379 : audit [INF] from='client.49100 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi163-83854-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:10.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:10.191310+0000 mon.a (mon.0) 1380 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:10.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:10.191456+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-02-19T21:06:10.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:10.204491+0000 mon.b (mon.1) 294 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:10.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:10.208857+0000 mon.a (mon.0) 1382 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:10.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:10.293363+0000 mon.c (mon.2) 197 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:10.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:10.332481+0000 mon.b (mon.1) 295 : audit [DBG] from='client.? v1:172.21.15.163:0/1513816879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:10.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:09.137149+0000 mgr.y (mgr.14550) 226 : cluster [DBG] pgmap v254: 247 pgs: 10 active+clean+snaptrim, 8 unknown, 22 active+clean+snaptrim_wait, 207 active+clean; 28 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:10.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:09.418956+0000 osd.7 (osd.7) 101 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:06:10.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:09.422003+0000 osd.7 (osd.7) 102 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:06:10.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:09.593829+0000 osd.4 (osd.4) 97 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:06:10.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:09.595747+0000 osd.4 (osd.4) 98 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:06:10.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:09.761586+0000 osd.3 (osd.3) 107 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:06:10.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:09.763477+0000 osd.3 (osd.3) 108 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:06:10.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:09.850032+0000 osd.5 (osd.5) 88 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:06:10.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:09.852993+0000 osd.5 (osd.5) 89 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:06:10.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:10.191023+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi163-83920-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi163-83920-36"}]': finished 2024-02-19T21:06:10.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:10.191190+0000 mon.a (mon.0) 1379 : audit [INF] from='client.49100 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi163-83854-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:10.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:10.191310+0000 mon.a (mon.0) 1380 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:10.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:10.191456+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-02-19T21:06:10.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:10.204491+0000 mon.b (mon.1) 294 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:10.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:10.208857+0000 mon.a (mon.0) 1382 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:10.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:10.293363+0000 mon.c (mon.2) 197 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:10.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:10.332481+0000 mon.b (mon.1) 295 : audit [DBG] from='client.? v1:172.21.15.163:0/1513816879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:10.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:09.137149+0000 mgr.y (mgr.14550) 226 : cluster [DBG] pgmap v254: 247 pgs: 10 active+clean+snaptrim, 8 unknown, 22 active+clean+snaptrim_wait, 207 active+clean; 28 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:10.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:09.418956+0000 osd.7 (osd.7) 101 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:06:10.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:09.422003+0000 osd.7 (osd.7) 102 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:06:10.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:09.593829+0000 osd.4 (osd.4) 97 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:06:10.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:09.595747+0000 osd.4 (osd.4) 98 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:06:10.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:09.761586+0000 osd.3 (osd.3) 107 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:06:10.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:09.763477+0000 osd.3 (osd.3) 108 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:06:10.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:09.850032+0000 osd.5 (osd.5) 88 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:06:10.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:09.852993+0000 osd.5 (osd.5) 89 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:06:10.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:10.191023+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi163-83920-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi163-83920-36"}]': finished 2024-02-19T21:06:10.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:10.191190+0000 mon.a (mon.0) 1379 : audit [INF] from='client.49100 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi163-83854-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:10.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:10.191310+0000 mon.a (mon.0) 1380 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:10.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:10.191456+0000 mon.a (mon.0) 1381 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-02-19T21:06:10.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:10.204491+0000 mon.b (mon.1) 294 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:10.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:10.208857+0000 mon.a (mon.0) 1382 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:10.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:10.293363+0000 mon.c (mon.2) 197 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:10.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:10.332481+0000 mon.b (mon.1) 295 : audit [DBG] from='client.? v1:172.21.15.163:0/1513816879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:10.819 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:06:10.846 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:10.919 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:06:10.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:10.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:06:11.023 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:06:11.050 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:11.070 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:06:11.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:11.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:06:11.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.147+0000 7fd15dce3700 1 -- 172.21.15.163:0/142901138 >> v1:172.21.15.163:6789/0 conn(0x7fd1580fc870 legacy=0x7fd1580fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:11.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.147+0000 7fd15dce3700 1 -- 172.21.15.163:0/142901138 shutdown_connections 2024-02-19T21:06:11.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.147+0000 7fd15dce3700 1 -- 172.21.15.163:0/142901138 >> 172.21.15.163:0/142901138 conn(0x7fd1580f2470 msgr2=0x7fd1580f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:11.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.147+0000 7fd15dce3700 1 -- 172.21.15.163:0/142901138 shutdown_connections 2024-02-19T21:06:11.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.147+0000 7fd15dce3700 1 -- 172.21.15.163:0/142901138 wait complete. 2024-02-19T21:06:11.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.148+0000 7fd15dce3700 1 librados: starting msgr at 2024-02-19T21:06:11.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.148+0000 7fd15dce3700 1 librados: starting objecter 2024-02-19T21:06:11.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.148+0000 7fd15dce3700 1 Processor -- start 2024-02-19T21:06:11.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.148+0000 7fd15dce3700 1 -- start start 2024-02-19T21:06:11.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.148+0000 7fd15dce3700 1 librados: setting wanted keys 2024-02-19T21:06:11.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.148+0000 7fd15dce3700 1 librados: calling monclient init 2024-02-19T21:06:11.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.148+0000 7fd15dce3700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd1580fbf60 con 0x7fd1580ff670 2024-02-19T21:06:11.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.148+0000 7fd15dce3700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd1580fc130 con 0x7fd1581031b0 2024-02-19T21:06:11.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.148+0000 7fd15dce3700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd15811ba80 con 0x7fd1580fc870 2024-02-19T21:06:11.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd156ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fd1580ff670 0x7fd158114c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:52074/0 (socket says 172.21.15.163:52074) 2024-02-19T21:06:11.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd1577fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fd1580fc870 0x7fd1580fb7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38142/0 (socket says 172.21.15.163:38142) 2024-02-19T21:06:11.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd156ffd700 1 -- 172.21.15.163:0/1952741809 learned_addr learned my addr 172.21.15.163:0/1952741809 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:11.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 10 client.?.objecter ms_handle_connect 0x7fd1580ff670 2024-02-19T21:06:11.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:11.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 10 client.?.objecter ms_handle_connect 0x7fd1580fc870 2024-02-19T21:06:11.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:11.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 10 client.?.objecter ms_handle_connect 0x7fd1581031b0 2024-02-19T21:06:11.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:11.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824675676 0 0) 0x7fd1580fbf60 con 0x7fd1580ff670 2024-02-19T21:06:11.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd140003800 con 0x7fd1580ff670 2024-02-19T21:06:11.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791285557 0 0) 0x7fd15811ba80 con 0x7fd1580fc870 2024-02-19T21:06:11.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1580fbf60 con 0x7fd1580fc870 2024-02-19T21:06:11.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1200772369 0 0) 0x7fd1580fc130 con 0x7fd1581031b0 2024-02-19T21:06:11.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd15811ba80 con 0x7fd1581031b0 2024-02-19T21:06:11.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2740601716 0 0) 0x7fd140003800 con 0x7fd1580ff670 2024-02-19T21:06:11.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1580fc130 con 0x7fd1580ff670 2024-02-19T21:06:11.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd14c002910 con 0x7fd1580ff670 2024-02-19T21:06:11.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1163268249 0 0) 0x7fd1580fbf60 con 0x7fd1580fc870 2024-02-19T21:06:11.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd140003800 con 0x7fd1580fc870 2024-02-19T21:06:11.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1053969017 0 0) 0x7fd15811ba80 con 0x7fd1581031b0 2024-02-19T21:06:11.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1580fbf60 con 0x7fd1581031b0 2024-02-19T21:06:11.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd148002f60 con 0x7fd1580fc870 2024-02-19T21:06:11.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd150001c50 con 0x7fd1581031b0 2024-02-19T21:06:11.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3293799755 0 0) 0x7fd1580fc130 con 0x7fd1580ff670 2024-02-19T21:06:11.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 >> v1:172.21.15.163:6790/0 conn(0x7fd1580fc870 legacy=0x7fd1580fb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:11.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 >> v1:172.21.15.193:6789/0 conn(0x7fd1581031b0 legacy=0x7fd158118340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:11.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15811cc20 con 0x7fd1580ff670 2024-02-19T21:06:11.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd15dce3700 1 -- 172.21.15.163:0/1952741809 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15811bc70 con 0x7fd1580ff670 2024-02-19T21:06:11.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd15dce3700 10 client.22464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:11.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd15dce3700 1 -- 172.21.15.163:0/1952741809 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15811c0e0 con 0x7fd1580ff670 2024-02-19T21:06:11.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.149+0000 7fd15dce3700 1 librados: init done 2024-02-19T21:06:11.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.150+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fd14c002ca0 con 0x7fd1580ff670 2024-02-19T21:06:11.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.150+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd14c005bc0 con 0x7fd1580ff670 2024-02-19T21:06:11.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.150+0000 7fd15dce3700 1 -- 172.21.15.163:0/1952741809 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd144004f20 con 0x7fd1580ff670 2024-02-19T21:06:11.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.151+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fd14c001d20 con 0x7fd1580ff670 2024-02-19T21:06:11.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.152+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(190..190 src has 43..190) v4 ==== 10761+0+0 (unknown 1054489016 0 0) 0x7fd14c079800 con 0x7fd1580ff670 2024-02-19T21:06:11.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.152+0000 7fd154ff9700 10 client.22464.objecter ms_dispatch 0x7fd158001550 osd_map(190..190 src has 43..190) v4 2024-02-19T21:06:11.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.152+0000 7fd154ff9700 3 client.22464.objecter handle_osd_map got epochs [190,190] > 0 2024-02-19T21:06:11.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.152+0000 7fd154ff9700 3 client.22464.objecter handle_osd_map decoding full epoch 190 2024-02-19T21:06:11.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.152+0000 7fd154ff9700 20 client.22464.objecter dump_active .. 0 homeless 2024-02-19T21:06:11.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.152+0000 7fd154ff9700 10 client.22464.objecter ms_handle_connect 0x7fd140063ce0 2024-02-19T21:06:11.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.156+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd14c04c160 con 0x7fd1580ff670 2024-02-19T21:06:11.162 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:11.174 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:11.215 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:11.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:11.378+0000 7fd15dce3700 1 -- 172.21.15.163:0/1952741809 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fd1440051d0 con 0x7fd1580ff670 2024-02-19T21:06:11.421 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:11.424 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:11.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:11.529 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:11.556 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:11.583 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:11.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:11.705 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:11.734 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:11.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:09.791583+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:06:11.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:09.793204+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:06:11.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:10.153735+0000 mgr.y (mgr.14550) 227 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:11.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:10.787446+0000 osd.1 (osd.1) 111 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:06:11.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:10.791030+0000 osd.1 (osd.1) 112 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:06:11.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:10.922146+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:06:11.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:10.922910+0000 osd.6 (osd.6) 90 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:06:11.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:11.193914+0000 mon.a (mon.0) 1383 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi163-83845-27"}]': finished 2024-02-19T21:06:11.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:11.194192+0000 mon.a (mon.0) 1384 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-02-19T21:06:11.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:11.207101+0000 mon.b (mon.1) 296 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:11.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:11.220089+0000 mon.a (mon.0) 1385 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:11.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:11.252588+0000 mon.b (mon.1) 297 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:11.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:11.253670+0000 mon.a (mon.0) 1386 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:11.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:11.294193+0000 mon.c (mon.2) 198 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:11.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:11.500547+0000 mon.a (mon.0) 1387 : audit [INF] from='client.? v1:172.21.15.163:0/1952741809' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:06:11.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:09.791583+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:06:11.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:09.793204+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:06:11.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:10.153735+0000 mgr.y (mgr.14550) 227 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:11.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:10.787446+0000 osd.1 (osd.1) 111 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:06:11.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:10.791030+0000 osd.1 (osd.1) 112 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:06:11.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:10.922146+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:06:11.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:10.922910+0000 osd.6 (osd.6) 90 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:06:11.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:11.193914+0000 mon.a (mon.0) 1383 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi163-83845-27"}]': finished 2024-02-19T21:06:11.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:11.194192+0000 mon.a (mon.0) 1384 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-02-19T21:06:11.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:11.207101+0000 mon.b (mon.1) 296 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:11.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:11.220089+0000 mon.a (mon.0) 1385 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:11.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:11.252588+0000 mon.b (mon.1) 297 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:11.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:11.253670+0000 mon.a (mon.0) 1386 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:11.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:11.294193+0000 mon.c (mon.2) 198 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:11.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:11.500547+0000 mon.a (mon.0) 1387 : audit [INF] from='client.? v1:172.21.15.163:0/1952741809' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:06:12.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:09.791583+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:06:12.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:09.793204+0000 osd.0 (osd.0) 102 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:06:12.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:10.153735+0000 mgr.y (mgr.14550) 227 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:12.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:10.787446+0000 osd.1 (osd.1) 111 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:06:12.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:10.791030+0000 osd.1 (osd.1) 112 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:06:12.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:10.922146+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:06:12.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:10.922910+0000 osd.6 (osd.6) 90 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:06:12.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:11.193914+0000 mon.a (mon.0) 1383 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi163-83845-27"}]': finished 2024-02-19T21:06:12.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:11.194192+0000 mon.a (mon.0) 1384 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-02-19T21:06:12.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:11.207101+0000 mon.b (mon.1) 296 : audit [INF] from='client.? v1:172.21.15.163:0/979817491' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:12.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:11.220089+0000 mon.a (mon.0) 1385 : audit [INF] from='client.47902 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi163-83845-27"}]: dispatch 2024-02-19T21:06:12.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:11.252588+0000 mon.b (mon.1) 297 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:12.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:11.253670+0000 mon.a (mon.0) 1386 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:12.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:11.294193+0000 mon.c (mon.2) 198 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:12.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:11.500547+0000 mon.a (mon.0) 1387 : audit [INF] from='client.? v1:172.21.15.163:0/1952741809' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:06:12.183 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:12.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:12.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:12.201+0000 7fd154ff9700 1 -- 172.21.15.163:0/1952741809 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v192) v1 ==== 101+0+0 (unknown 581612997 0 0) 0x7fd14c04a570 con 0x7fd1580ff670 2024-02-19T21:06:12.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:12.206+0000 7fd15dce3700 10 librados: watch_flush enter 2024-02-19T21:06:12.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:12.209+0000 7fd15dce3700 10 librados: watch_flush exit 2024-02-19T21:06:12.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:12.209+0000 7fd15dce3700 20 client.22464.objecter shutdown clearing up homeless session... 2024-02-19T21:06:12.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:12.209+0000 7fd15dce3700 10 client.22464.objecter successfully canceled tick 2024-02-19T21:06:12.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:12.209+0000 7fd15dce3700 1 -- 172.21.15.163:0/1952741809 >> v1:172.21.15.163:6800/1552514681 conn(0x7fd140063ce0 legacy=0x7fd1400661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:12.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:12.209+0000 7fd15dce3700 1 -- 172.21.15.163:0/1952741809 >> v1:172.21.15.163:6789/0 conn(0x7fd1580ff670 legacy=0x7fd158114c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:12.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:12.212+0000 7fd15dce3700 1 -- 172.21.15.163:0/1952741809 shutdown_connections 2024-02-19T21:06:12.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:12.212+0000 7fd15dce3700 1 -- 172.21.15.163:0/1952741809 >> 172.21.15.163:0/1952741809 conn(0x7fd1580f2470 msgr2=0x7fd158061200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:12.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:12.213+0000 7fd15dce3700 1 -- 172.21.15.163:0/1952741809 shutdown_connections 2024-02-19T21:06:12.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:12.213+0000 7fd15dce3700 1 -- 172.21.15.163:0/1952741809 wait complete. 2024-02-19T21:06:12.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:12.213+0000 7fd15dce3700 1 librados: shutdown 2024-02-19T21:06:12.211 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:06:12.292 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:12.321 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:12.342 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:12.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:12.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:12.446 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:12.475 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:12.623 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:12.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:12.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:12.727 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:12.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:10.824815+0000 osd.0 (osd.0) 103 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:06:12.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:10.826883+0000 osd.0 (osd.0) 104 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:06:12.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:11.137666+0000 mgr.y (mgr.14550) 228 : cluster [DBG] pgmap v257: 311 pgs: 64 creating+peering, 2 peering, 14 active+clean+snaptrim, 8 unknown, 24 active+clean+snaptrim_wait, 199 active+clean; 44 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-02-19T21:06:12.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:11.638799+0000 osd.4 (osd.4) 99 : cluster [DBG] 2.25 scrub starts 2024-02-19T21:06:12.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:11.639676+0000 osd.4 (osd.4) 100 : cluster [DBG] 2.25 scrub ok 2024-02-19T21:06:12.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:11.886290+0000 osd.6 (osd.6) 91 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:06:12.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:11.887135+0000 osd.6 (osd.6) 92 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:06:12.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:12.190703+0000 mon.a (mon.0) 1388 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:06:12.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:12.195049+0000 mon.a (mon.0) 1389 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi163-83845-27"}]': finished 2024-02-19T21:06:12.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:12.195151+0000 mon.a (mon.0) 1390 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:12.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:12.195247+0000 mon.a (mon.0) 1391 : audit [INF] from='client.? v1:172.21.15.163:0/1952741809' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:06:12.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:12.195347+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-02-19T21:06:12.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:12.199957+0000 mon.b (mon.1) 298 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-15"}]: dispatch 2024-02-19T21:06:12.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:12.200449+0000 mon.a (mon.0) 1393 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-15"}]: dispatch 2024-02-19T21:06:12.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:12.201598+0000 mon.a (mon.0) 1394 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi163-83920-36"}]: dispatch 2024-02-19T21:06:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:10.824815+0000 osd.0 (osd.0) 103 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:06:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:10.826883+0000 osd.0 (osd.0) 104 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:06:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:11.137666+0000 mgr.y (mgr.14550) 228 : cluster [DBG] pgmap v257: 311 pgs: 64 creating+peering, 2 peering, 14 active+clean+snaptrim, 8 unknown, 24 active+clean+snaptrim_wait, 199 active+clean; 44 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-02-19T21:06:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:11.638799+0000 osd.4 (osd.4) 99 : cluster [DBG] 2.25 scrub starts 2024-02-19T21:06:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:11.639676+0000 osd.4 (osd.4) 100 : cluster [DBG] 2.25 scrub ok 2024-02-19T21:06:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:11.886290+0000 osd.6 (osd.6) 91 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:06:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:11.887135+0000 osd.6 (osd.6) 92 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:06:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:12.190703+0000 mon.a (mon.0) 1388 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:06:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:12.195049+0000 mon.a (mon.0) 1389 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi163-83845-27"}]': finished 2024-02-19T21:06:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:12.195151+0000 mon.a (mon.0) 1390 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:12.195247+0000 mon.a (mon.0) 1391 : audit [INF] from='client.? v1:172.21.15.163:0/1952741809' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:06:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:12.195347+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-02-19T21:06:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:12.199957+0000 mon.b (mon.1) 298 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-15"}]: dispatch 2024-02-19T21:06:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:12.200449+0000 mon.a (mon.0) 1393 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-15"}]: dispatch 2024-02-19T21:06:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:12.201598+0000 mon.a (mon.0) 1394 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi163-83920-36"}]: dispatch 2024-02-19T21:06:12.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:12.208788+0000 mon.b (mon.1) 299 : audit [INF] from='client.? v1:172.21.15.163:0/1595134664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi163-83854-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:12.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:12.209658+0000 mon.a (mon.0) 1395 : audit [INF] from='client.47935 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi163-83854-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:12.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:12.228654+0000 mon.b (mon.1) 300 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:12.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:12.229352+0000 mon.a (mon.0) 1396 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:12.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:12.230652+0000 mon.b (mon.1) 301 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:12.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:12.231683+0000 mon.a (mon.0) 1397 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:12.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:12.232252+0000 mon.b (mon.1) 302 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi163-83845-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:12.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:12.232571+0000 mon.a (mon.0) 1398 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi163-83845-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:12.294899+0000 mon.c (mon.2) 199 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:12.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:12.208788+0000 mon.b (mon.1) 299 : audit [INF] from='client.? v1:172.21.15.163:0/1595134664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi163-83854-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:12.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:12.209658+0000 mon.a (mon.0) 1395 : audit [INF] from='client.47935 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi163-83854-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:12.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:12.228654+0000 mon.b (mon.1) 300 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:12.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:12.229352+0000 mon.a (mon.0) 1396 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:12.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:12.230652+0000 mon.b (mon.1) 301 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:12.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:12.231683+0000 mon.a (mon.0) 1397 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:12.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:12.232252+0000 mon.b (mon.1) 302 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi163-83845-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:12.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:12.232571+0000 mon.a (mon.0) 1398 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi163-83845-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:12.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:12.294899+0000 mon.c (mon.2) 199 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:12.756 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:13.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:10.824815+0000 osd.0 (osd.0) 103 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:06:13.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:10.826883+0000 osd.0 (osd.0) 104 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:06:13.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:11.137666+0000 mgr.y (mgr.14550) 228 : cluster [DBG] pgmap v257: 311 pgs: 64 creating+peering, 2 peering, 14 active+clean+snaptrim, 8 unknown, 24 active+clean+snaptrim_wait, 199 active+clean; 44 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-02-19T21:06:13.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:11.638799+0000 osd.4 (osd.4) 99 : cluster [DBG] 2.25 scrub starts 2024-02-19T21:06:13.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:11.639676+0000 osd.4 (osd.4) 100 : cluster [DBG] 2.25 scrub ok 2024-02-19T21:06:13.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:11.886290+0000 osd.6 (osd.6) 91 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:06:13.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:11.887135+0000 osd.6 (osd.6) 92 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:12.190703+0000 mon.a (mon.0) 1388 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:12.195049+0000 mon.a (mon.0) 1389 : audit [INF] from='client.47902 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi163-83845-27"}]': finished 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:12.195151+0000 mon.a (mon.0) 1390 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:12.195247+0000 mon.a (mon.0) 1391 : audit [INF] from='client.? v1:172.21.15.163:0/1952741809' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:12.195347+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:12.199957+0000 mon.b (mon.1) 298 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-15"}]: dispatch 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:12.200449+0000 mon.a (mon.0) 1393 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-15"}]: dispatch 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:12.201598+0000 mon.a (mon.0) 1394 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi163-83920-36"}]: dispatch 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:12.208788+0000 mon.b (mon.1) 299 : audit [INF] from='client.? v1:172.21.15.163:0/1595134664' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi163-83854-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:12.209658+0000 mon.a (mon.0) 1395 : audit [INF] from='client.47935 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi163-83854-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:12.228654+0000 mon.b (mon.1) 300 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:12.229352+0000 mon.a (mon.0) 1396 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:12.230652+0000 mon.b (mon.1) 301 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:12.231683+0000 mon.a (mon.0) 1397 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:12.232252+0000 mon.b (mon.1) 302 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi163-83845-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:13.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:12.232571+0000 mon.a (mon.0) 1398 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi163-83845-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:13.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:12.294899+0000 mon.c (mon.2) 199 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:13.742 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:06:13] "GET /metrics HTTP/1.1" 200 191002 "" "Prometheus/2.33.4" 2024-02-19T21:06:13.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:12.175954+0000 osd.2 (osd.2) 83 : cluster [DBG] 5.d scrub starts 2024-02-19T21:06:13.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:12.177519+0000 osd.2 (osd.2) 84 : cluster [DBG] 5.d scrub ok 2024-02-19T21:06:13.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:13.138474+0000 mon.b (mon.1) 303 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-19T21:06:13.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:13.139463+0000 mon.a (mon.0) 1399 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-19T21:06:13.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:13.197045+0000 mon.a (mon.0) 1400 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-15"}]': finished 2024-02-19T21:06:13.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:13.197123+0000 mon.a (mon.0) 1401 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi163-83920-36"}]': finished 2024-02-19T21:06:13.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:13.197282+0000 mon.a (mon.0) 1402 : audit [INF] from='client.47935 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi163-83854-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:13.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:13.197371+0000 mon.a (mon.0) 1403 : audit [INF] from='client.47941 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi163-83845-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:13.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:13.197445+0000 mon.a (mon.0) 1404 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-19T21:06:13.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:13.197502+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-02-19T21:06:13.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:13.202812+0000 mon.a (mon.0) 1406 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi163-83920-36"}]: dispatch 2024-02-19T21:06:13.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:13.208141+0000 mon.b (mon.1) 304 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-15", "mode": "writeback"}]: dispatch 2024-02-19T21:06:13.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:13.208476+0000 mon.b (mon.1) 305 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi163-83845-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:13.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:13.212441+0000 mon.a (mon.0) 1407 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-15", "mode": "writeback"}]: dispatch 2024-02-19T21:06:13.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:13.212780+0000 mon.a (mon.0) 1408 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi163-83845-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:13.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:13.295587+0000 mon.c (mon.2) 200 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:13.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:12.175954+0000 osd.2 (osd.2) 83 : cluster [DBG] 5.d scrub starts 2024-02-19T21:06:13.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:12.177519+0000 osd.2 (osd.2) 84 : cluster [DBG] 5.d scrub ok 2024-02-19T21:06:13.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:13.138474+0000 mon.b (mon.1) 303 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-19T21:06:13.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:13.139463+0000 mon.a (mon.0) 1399 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-19T21:06:13.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:13.197045+0000 mon.a (mon.0) 1400 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-15"}]': finished 2024-02-19T21:06:13.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:13.197123+0000 mon.a (mon.0) 1401 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi163-83920-36"}]': finished 2024-02-19T21:06:13.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:13.197282+0000 mon.a (mon.0) 1402 : audit [INF] from='client.47935 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi163-83854-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:13.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:13.197371+0000 mon.a (mon.0) 1403 : audit [INF] from='client.47941 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi163-83845-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:13.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:13.197445+0000 mon.a (mon.0) 1404 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-19T21:06:13.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:13.197502+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-02-19T21:06:13.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:13.202812+0000 mon.a (mon.0) 1406 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi163-83920-36"}]: dispatch 2024-02-19T21:06:13.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:13.208141+0000 mon.b (mon.1) 304 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-15", "mode": "writeback"}]: dispatch 2024-02-19T21:06:13.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:13.208476+0000 mon.b (mon.1) 305 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi163-83845-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:13.212441+0000 mon.a (mon.0) 1407 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-15", "mode": "writeback"}]: dispatch 2024-02-19T21:06:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:13.212780+0000 mon.a (mon.0) 1408 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi163-83845-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:13.295587+0000 mon.c (mon.2) 200 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:14.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:12.175954+0000 osd.2 (osd.2) 83 : cluster [DBG] 5.d scrub starts 2024-02-19T21:06:14.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:12.177519+0000 osd.2 (osd.2) 84 : cluster [DBG] 5.d scrub ok 2024-02-19T21:06:14.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:13.138474+0000 mon.b (mon.1) 303 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-19T21:06:14.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:13.139463+0000 mon.a (mon.0) 1399 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-02-19T21:06:14.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:13.197045+0000 mon.a (mon.0) 1400 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-15"}]': finished 2024-02-19T21:06:14.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:13.197123+0000 mon.a (mon.0) 1401 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi163-83920-36"}]': finished 2024-02-19T21:06:14.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:13.197282+0000 mon.a (mon.0) 1402 : audit [INF] from='client.47935 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi163-83854-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:14.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:13.197371+0000 mon.a (mon.0) 1403 : audit [INF] from='client.47941 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi163-83845-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:14.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:13.197445+0000 mon.a (mon.0) 1404 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-02-19T21:06:14.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:13.197502+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-02-19T21:06:14.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:13.202812+0000 mon.a (mon.0) 1406 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi163-83920-36"}]: dispatch 2024-02-19T21:06:14.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:13.208141+0000 mon.b (mon.1) 304 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-15", "mode": "writeback"}]: dispatch 2024-02-19T21:06:14.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:13.208476+0000 mon.b (mon.1) 305 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi163-83845-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:14.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:13.212441+0000 mon.a (mon.0) 1407 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-15", "mode": "writeback"}]: dispatch 2024-02-19T21:06:14.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:13.212780+0000 mon.a (mon.0) 1408 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi163-83845-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:14.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:13.295587+0000 mon.c (mon.2) 200 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:14.417 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplePP (67624 ms total) 2024-02-19T21:06:14.417 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: 2024-02-19T21:06:14.417 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-02-19T21:06:14.417 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-02-19T21:06:14.417 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (105 ms) 2024-02-19T21:06:14.417 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (105 ms total) 2024-02-19T21:06:14.417 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: 2024-02-19T21:06:14.417 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-02-19T21:06:14.417 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-02-19T21:06:14.418 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (8 ms) 2024-02-19T21:06:14.418 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-02-19T21:06:14.418 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (3 ms) 2024-02-19T21:06:14.418 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (11 ms total) 2024-02-19T21:06:14.418 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: 2024-02-19T21:06:14.418 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-02-19T21:06:14.418 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-02-19T21:06:14.418 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (22 ms) 2024-02-19T21:06:14.418 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-02-19T21:06:14.418 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (2 ms) 2024-02-19T21:06:14.418 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (24 ms total) 2024-02-19T21:06:14.418 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: 2024-02-19T21:06:14.418 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-02-19T21:06:14.418 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-02-19T21:06:14.419 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (28 ms) 2024-02-19T21:06:14.419 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-02-19T21:06:14.419 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (4 ms) 2024-02-19T21:06:14.419 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (32 ms total) 2024-02-19T21:06:14.419 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: 2024-02-19T21:06:14.419 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-02-19T21:06:14.419 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-02-19T21:06:14.419 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1020 ms) 2024-02-19T21:06:14.419 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1020 ms total) 2024-02-19T21:06:14.419 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: 2024-02-19T21:06:14.419 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-02-19T21:06:14.419 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (101545 ms total) 2024-02-19T21:06:14.419 INFO:tasks.workunit.client.0.smithi163.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-02-19T21:06:14.643 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:06:14.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:12.812699+0000 osd.0 (osd.0) 105 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:06:14.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:12.815721+0000 osd.0 (osd.0) 106 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:06:14.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:13.135616+0000 osd.2 (osd.2) 85 : cluster [DBG] 6.f scrub starts 2024-02-19T21:06:14.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:13.136528+0000 osd.2 (osd.2) 86 : cluster [DBG] 6.f scrub ok 2024-02-19T21:06:14.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:13.138081+0000 mgr.y (mgr.14550) 229 : cluster [DBG] pgmap v260: 302 pgs: 32 creating+peering, 1 peering, 14 active+clean+snaptrim, 32 unknown, 24 active+clean+snaptrim_wait, 199 active+clean; 44 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 7.7 KiB/s wr, 2 op/s 2024-02-19T21:06:14.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:13.712822+0000 osd.1 (osd.1) 113 : cluster [DBG] 8.11 scrub starts 2024-02-19T21:06:14.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:13.716386+0000 osd.1 (osd.1) 114 : cluster [DBG] 8.11 scrub ok 2024-02-19T21:06:14.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:14.172465+0000 mon.a (mon.0) 1409 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:14.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:14.179462+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi163-83920-36"}]': finished 2024-02-19T21:06:14.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:14.179577+0000 mon.a (mon.0) 1411 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-15", "mode": "writeback"}]': finished 2024-02-19T21:06:14.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:14.296310+0000 mon.c (mon.2) 201 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:14.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:14.419073+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-02-19T21:06:14.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:12.812699+0000 osd.0 (osd.0) 105 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:06:14.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:12.815721+0000 osd.0 (osd.0) 106 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:06:14.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:13.135616+0000 osd.2 (osd.2) 85 : cluster [DBG] 6.f scrub starts 2024-02-19T21:06:14.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:13.136528+0000 osd.2 (osd.2) 86 : cluster [DBG] 6.f scrub ok 2024-02-19T21:06:14.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:13.138081+0000 mgr.y (mgr.14550) 229 : cluster [DBG] pgmap v260: 302 pgs: 32 creating+peering, 1 peering, 14 active+clean+snaptrim, 32 unknown, 24 active+clean+snaptrim_wait, 199 active+clean; 44 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 7.7 KiB/s wr, 2 op/s 2024-02-19T21:06:14.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:13.712822+0000 osd.1 (osd.1) 113 : cluster [DBG] 8.11 scrub starts 2024-02-19T21:06:14.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:13.716386+0000 osd.1 (osd.1) 114 : cluster [DBG] 8.11 scrub ok 2024-02-19T21:06:14.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:14.172465+0000 mon.a (mon.0) 1409 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:14.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:14.179462+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi163-83920-36"}]': finished 2024-02-19T21:06:14.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:14.179577+0000 mon.a (mon.0) 1411 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-15", "mode": "writeback"}]': finished 2024-02-19T21:06:14.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:14.296310+0000 mon.c (mon.2) 201 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:14.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:14.419073+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-02-19T21:06:15.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:12.812699+0000 osd.0 (osd.0) 105 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:06:15.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:12.815721+0000 osd.0 (osd.0) 106 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:06:15.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:13.135616+0000 osd.2 (osd.2) 85 : cluster [DBG] 6.f scrub starts 2024-02-19T21:06:15.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:13.136528+0000 osd.2 (osd.2) 86 : cluster [DBG] 6.f scrub ok 2024-02-19T21:06:15.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:13.138081+0000 mgr.y (mgr.14550) 229 : cluster [DBG] pgmap v260: 302 pgs: 32 creating+peering, 1 peering, 14 active+clean+snaptrim, 32 unknown, 24 active+clean+snaptrim_wait, 199 active+clean; 44 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 7.7 KiB/s wr, 2 op/s 2024-02-19T21:06:15.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:13.712822+0000 osd.1 (osd.1) 113 : cluster [DBG] 8.11 scrub starts 2024-02-19T21:06:15.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:13.716386+0000 osd.1 (osd.1) 114 : cluster [DBG] 8.11 scrub ok 2024-02-19T21:06:15.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:14.172465+0000 mon.a (mon.0) 1409 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:15.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:14.179462+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? v1:172.21.15.163:0/1222385254' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi163-83920-36"}]': finished 2024-02-19T21:06:15.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:14.179577+0000 mon.a (mon.0) 1411 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-15", "mode": "writeback"}]': finished 2024-02-19T21:06:15.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:14.296310+0000 mon.c (mon.2) 201 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:15.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:14.419073+0000 mon.a (mon.0) 1412 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-02-19T21:06:16.198 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:14.444037+0000 osd.7 (osd.7) 103 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:06:16.199 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:14.446927+0000 osd.7 (osd.7) 104 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:06:16.199 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:14.685042+0000 osd.1 (osd.1) 115 : cluster [DBG] 2.a scrub starts 2024-02-19T21:06:16.199 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:14.687934+0000 osd.1 (osd.1) 116 : cluster [DBG] 2.a scrub ok 2024-02-19T21:06:16.199 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:14.739415+0000 osd.0 (osd.0) 107 : cluster [DBG] 8.e scrub starts 2024-02-19T21:06:16.199 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:14.740521+0000 osd.0 (osd.0) 108 : cluster [DBG] 8.e scrub ok 2024-02-19T21:06:16.199 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:14.768876+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.1a deep-scrub starts 2024-02-19T21:06:16.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:14.772411+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.1a deep-scrub ok 2024-02-19T21:06:16.200 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:14.810907+0000 osd.6 (osd.6) 93 : cluster [DBG] 8.d scrub starts 2024-02-19T21:06:16.492 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:14.444037+0000 osd.7 (osd.7) 103 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:06:16.492 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:14.446927+0000 osd.7 (osd.7) 104 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:06:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:14.685042+0000 osd.1 (osd.1) 115 : cluster [DBG] 2.a scrub starts 2024-02-19T21:06:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:14.687934+0000 osd.1 (osd.1) 116 : cluster [DBG] 2.a scrub ok 2024-02-19T21:06:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:14.739415+0000 osd.0 (osd.0) 107 : cluster [DBG] 8.e scrub starts 2024-02-19T21:06:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:14.740521+0000 osd.0 (osd.0) 108 : cluster [DBG] 8.e scrub ok 2024-02-19T21:06:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:14.768876+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.1a deep-scrub starts 2024-02-19T21:06:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:14.772411+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.1a deep-scrub ok 2024-02-19T21:06:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:14.810907+0000 osd.6 (osd.6) 93 : cluster [DBG] 8.d scrub starts 2024-02-19T21:06:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:14.812467+0000 osd.6 (osd.6) 94 : cluster [DBG] 8.d scrub ok 2024-02-19T21:06:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:15.139119+0000 mgr.y (mgr.14550) 230 : cluster [DBG] pgmap v263: 270 pgs: 2 clean+premerge+peered, 12 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 232 active+clean; 43 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-02-19T21:06:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:15.181537+0000 mon.a (mon.0) 1413 : audit [INF] from='client.47941 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi163-83845-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi163-83845-28"}]': finished 2024-02-19T21:06:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:15.181727+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-02-19T21:06:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:15.189399+0000 mon.a (mon.0) 1415 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]: dispatch 2024-02-19T21:06:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:15.190134+0000 mon.a (mon.0) 1416 : audit [INF] from='client.? v1:172.21.15.163:0/4230570240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi163-83854-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:16.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:15.209857+0000 mon.b (mon.1) 306 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:16.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:15.210287+0000 mon.a (mon.0) 1417 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:16.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:15.297110+0000 mon.c (mon.2) 202 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:16.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:14.444037+0000 osd.7 (osd.7) 103 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:06:16.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:14.446927+0000 osd.7 (osd.7) 104 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:06:16.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:14.685042+0000 osd.1 (osd.1) 115 : cluster [DBG] 2.a scrub starts 2024-02-19T21:06:16.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:14.687934+0000 osd.1 (osd.1) 116 : cluster [DBG] 2.a scrub ok 2024-02-19T21:06:16.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:14.739415+0000 osd.0 (osd.0) 107 : cluster [DBG] 8.e scrub starts 2024-02-19T21:06:16.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:14.740521+0000 osd.0 (osd.0) 108 : cluster [DBG] 8.e scrub ok 2024-02-19T21:06:16.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:14.768876+0000 osd.4 (osd.4) 101 : cluster [DBG] 4.1a deep-scrub starts 2024-02-19T21:06:16.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:14.772411+0000 osd.4 (osd.4) 102 : cluster [DBG] 4.1a deep-scrub ok 2024-02-19T21:06:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:14.810907+0000 osd.6 (osd.6) 93 : cluster [DBG] 8.d scrub starts 2024-02-19T21:06:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:14.812467+0000 osd.6 (osd.6) 94 : cluster [DBG] 8.d scrub ok 2024-02-19T21:06:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:15.139119+0000 mgr.y (mgr.14550) 230 : cluster [DBG] pgmap v263: 270 pgs: 2 clean+premerge+peered, 12 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 232 active+clean; 43 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-02-19T21:06:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:15.181537+0000 mon.a (mon.0) 1413 : audit [INF] from='client.47941 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi163-83845-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi163-83845-28"}]': finished 2024-02-19T21:06:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:15.181727+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-02-19T21:06:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:15.189399+0000 mon.a (mon.0) 1415 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]: dispatch 2024-02-19T21:06:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:15.190134+0000 mon.a (mon.0) 1416 : audit [INF] from='client.? v1:172.21.15.163:0/4230570240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi163-83854-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:15.209857+0000 mon.b (mon.1) 306 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:15.210287+0000 mon.a (mon.0) 1417 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:15.297110+0000 mon.c (mon.2) 202 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:16.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:14.812467+0000 osd.6 (osd.6) 94 : cluster [DBG] 8.d scrub ok 2024-02-19T21:06:16.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:15.139119+0000 mgr.y (mgr.14550) 230 : cluster [DBG] pgmap v263: 270 pgs: 2 clean+premerge+peered, 12 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 232 active+clean; 43 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 2 op/s 2024-02-19T21:06:16.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:15.181537+0000 mon.a (mon.0) 1413 : audit [INF] from='client.47941 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi163-83845-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi163-83845-28"}]': finished 2024-02-19T21:06:16.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:15.181727+0000 mon.a (mon.0) 1414 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-02-19T21:06:16.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:15.189399+0000 mon.a (mon.0) 1415 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]: dispatch 2024-02-19T21:06:16.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:15.190134+0000 mon.a (mon.0) 1416 : audit [INF] from='client.? v1:172.21.15.163:0/4230570240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi163-83854-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:16.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:15.209857+0000 mon.b (mon.1) 306 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:16.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:15.210287+0000 mon.a (mon.0) 1417 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:16.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:15.297110+0000 mon.c (mon.2) 202 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:17.190 INFO:tasks.workunit.client.0.smithi163.stdout:: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-02-19T21:06:17.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-02-19T21:06:17.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (3991 ms) 2024-02-19T21:06:17.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-02-19T21:06:17.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3997 ms) 2024-02-19T21:06:17.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-02-19T21:06:17.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4223 ms) 2024-02-19T21:06:17.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12211 ms total) 2024-02-19T21:06:17.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: 2024-02-19T21:06:17.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-02-19T21:06:17.190 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [==========] 20 tests from 5 test suites ran. (104337 ms total) 2024-02-19T21:06:17.191 INFO:tasks.workunit.client.0.smithi163.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-02-19T21:06:17.191 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:06:17.191 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83907 2024-02-19T21:06:17.191 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83907 2024-02-19T21:06:17.191 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:06:17.191 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83899 2024-02-19T21:06:17.191 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83899 2024-02-19T21:06:17.191 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:06:17.191 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83883 2024-02-19T21:06:17.191 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83883 2024-02-19T21:06:17.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:15.752607+0000 osd.4 (osd.4) 103 : cluster [DBG] 6.17 scrub starts 2024-02-19T21:06:17.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:15.753591+0000 osd.4 (osd.4) 104 : cluster [DBG] 6.17 scrub ok 2024-02-19T21:06:17.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:15.790252+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:06:17.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:15.793320+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:06:17.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:15.890025+0000 osd.5 (osd.5) 90 : cluster [DBG] 2.12 deep-scrub starts 2024-02-19T21:06:17.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:15.892920+0000 osd.5 (osd.5) 91 : cluster [DBG] 2.12 deep-scrub ok 2024-02-19T21:06:17.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:16.104475+0000 osd.2 (osd.2) 87 : cluster [DBG] 2.e deep-scrub starts 2024-02-19T21:06:17.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:16.105465+0000 osd.2 (osd.2) 88 : cluster [DBG] 2.e deep-scrub ok 2024-02-19T21:06:17.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:16.181060+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]': finished 2024-02-19T21:06:17.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:16.181134+0000 mon.a (mon.0) 1419 : audit [INF] from='client.? v1:172.21.15.163:0/4230570240' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi163-83854-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:17.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:16.181304+0000 mon.a (mon.0) 1420 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:17.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:16.181390+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-02-19T21:06:17.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:16.183788+0000 mon.b (mon.1) 307 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15"}]: dispatch 2024-02-19T21:06:17.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:16.191176+0000 mon.a (mon.0) 1422 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]: dispatch 2024-02-19T21:06:17.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:16.192369+0000 mon.a (mon.0) 1423 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15"}]: dispatch 2024-02-19T21:06:17.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:16.297823+0000 mon.c (mon.2) 203 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:17.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:17.140182+0000 mon.b (mon.1) 308 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-19T21:06:17.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:17.141332+0000 mon.a (mon.0) 1424 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-19T21:06:17.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:17.178081+0000 mon.a (mon.0) 1425 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:17.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:17.183538+0000 mon.a (mon.0) 1426 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]': finished 2024-02-19T21:06:17.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:17.183633+0000 mon.a (mon.0) 1427 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15"}]': finished 2024-02-19T21:06:17.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:17.183770+0000 mon.a (mon.0) 1428 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-19T21:06:17.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:17.183839+0000 mon.a (mon.0) 1429 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-02-19T21:06:17.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:17.189044+0000 mon.b (mon.1) 309 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:17.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:17.189685+0000 mon.a (mon.0) 1430 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:17.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:15.752607+0000 osd.4 (osd.4) 103 : cluster [DBG] 6.17 scrub starts 2024-02-19T21:06:17.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:15.753591+0000 osd.4 (osd.4) 104 : cluster [DBG] 6.17 scrub ok 2024-02-19T21:06:17.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:15.790252+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:06:17.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:15.793320+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:06:17.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:15.890025+0000 osd.5 (osd.5) 90 : cluster [DBG] 2.12 deep-scrub starts 2024-02-19T21:06:17.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:15.892920+0000 osd.5 (osd.5) 91 : cluster [DBG] 2.12 deep-scrub ok 2024-02-19T21:06:17.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:16.104475+0000 osd.2 (osd.2) 87 : cluster [DBG] 2.e deep-scrub starts 2024-02-19T21:06:17.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:16.105465+0000 osd.2 (osd.2) 88 : cluster [DBG] 2.e deep-scrub ok 2024-02-19T21:06:17.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:16.181060+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]': finished 2024-02-19T21:06:17.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:16.181134+0000 mon.a (mon.0) 1419 : audit [INF] from='client.? v1:172.21.15.163:0/4230570240' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi163-83854-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:17.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:16.181304+0000 mon.a (mon.0) 1420 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:17.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:16.181390+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-02-19T21:06:17.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:16.183788+0000 mon.b (mon.1) 307 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15"}]: dispatch 2024-02-19T21:06:17.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:16.191176+0000 mon.a (mon.0) 1422 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]: dispatch 2024-02-19T21:06:17.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:16.192369+0000 mon.a (mon.0) 1423 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15"}]: dispatch 2024-02-19T21:06:17.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:16.297823+0000 mon.c (mon.2) 203 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:17.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:17.140182+0000 mon.b (mon.1) 308 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-19T21:06:17.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:17.141332+0000 mon.a (mon.0) 1424 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-19T21:06:17.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:17.178081+0000 mon.a (mon.0) 1425 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:17.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:17.183538+0000 mon.a (mon.0) 1426 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]': finished 2024-02-19T21:06:17.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:17.183633+0000 mon.a (mon.0) 1427 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15"}]': finished 2024-02-19T21:06:17.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:17.183770+0000 mon.a (mon.0) 1428 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-19T21:06:17.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:17.183839+0000 mon.a (mon.0) 1429 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-02-19T21:06:17.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:17.189044+0000 mon.b (mon.1) 309 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:17.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:17.189685+0000 mon.a (mon.0) 1430 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:17.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:15.752607+0000 osd.4 (osd.4) 103 : cluster [DBG] 6.17 scrub starts 2024-02-19T21:06:17.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:15.753591+0000 osd.4 (osd.4) 104 : cluster [DBG] 6.17 scrub ok 2024-02-19T21:06:17.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:15.790252+0000 osd.6 (osd.6) 95 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:06:17.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:15.793320+0000 osd.6 (osd.6) 96 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:06:17.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:15.890025+0000 osd.5 (osd.5) 90 : cluster [DBG] 2.12 deep-scrub starts 2024-02-19T21:06:17.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:15.892920+0000 osd.5 (osd.5) 91 : cluster [DBG] 2.12 deep-scrub ok 2024-02-19T21:06:17.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:16.104475+0000 osd.2 (osd.2) 87 : cluster [DBG] 2.e deep-scrub starts 2024-02-19T21:06:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:16.105465+0000 osd.2 (osd.2) 88 : cluster [DBG] 2.e deep-scrub ok 2024-02-19T21:06:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:16.181060+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]': finished 2024-02-19T21:06:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:16.181134+0000 mon.a (mon.0) 1419 : audit [INF] from='client.? v1:172.21.15.163:0/4230570240' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi163-83854-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:16.181304+0000 mon.a (mon.0) 1420 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:16.181390+0000 mon.a (mon.0) 1421 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-02-19T21:06:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:16.183788+0000 mon.b (mon.1) 307 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15"}]: dispatch 2024-02-19T21:06:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:16.191176+0000 mon.a (mon.0) 1422 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]: dispatch 2024-02-19T21:06:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:16.192369+0000 mon.a (mon.0) 1423 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15"}]: dispatch 2024-02-19T21:06:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:16.297823+0000 mon.c (mon.2) 203 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:17.140182+0000 mon.b (mon.1) 308 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-19T21:06:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:17.141332+0000 mon.a (mon.0) 1424 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-02-19T21:06:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:17.178081+0000 mon.a (mon.0) 1425 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:17.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:17.183538+0000 mon.a (mon.0) 1426 : audit [INF] from='client.? v1:172.21.15.163:0/4166546335' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi163-83952-20"}]': finished 2024-02-19T21:06:17.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:17.183633+0000 mon.a (mon.0) 1427 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-15"}]': finished 2024-02-19T21:06:17.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:17.183770+0000 mon.a (mon.0) 1428 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-02-19T21:06:17.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:17.183839+0000 mon.a (mon.0) 1429 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-02-19T21:06:17.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:17.189044+0000 mon.b (mon.1) 309 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:17.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:17.189685+0000 mon.a (mon.0) 1430 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:17.708 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:17.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:17.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:17.812 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:06:17.838 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:17.988 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:06:17.990 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:17.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:06:18.066 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:06:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:06:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:06:18.091 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:18.124 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:18.145 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:18.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:18.249 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:18.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.257+0000 7f52da0d4700 1 -- 172.21.15.163:0/2065290454 >> v1:172.21.15.193:6789/0 conn(0x7f52d4060a10 legacy=0x7f52d4060e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:18.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.257+0000 7f52da0d4700 1 -- 172.21.15.163:0/2065290454 shutdown_connections 2024-02-19T21:06:18.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.257+0000 7f52da0d4700 1 -- 172.21.15.163:0/2065290454 >> 172.21.15.163:0/2065290454 conn(0x7f52d40f2470 msgr2=0x7f52d40f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:18.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.257+0000 7f52da0d4700 1 -- 172.21.15.163:0/2065290454 shutdown_connections 2024-02-19T21:06:18.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.257+0000 7f52da0d4700 1 -- 172.21.15.163:0/2065290454 wait complete. 2024-02-19T21:06:18.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.258+0000 7f52da0d4700 1 librados: starting msgr at 2024-02-19T21:06:18.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.258+0000 7f52da0d4700 1 librados: starting objecter 2024-02-19T21:06:18.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.258+0000 7f52da0d4700 1 Processor -- start 2024-02-19T21:06:18.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.258+0000 7f52da0d4700 1 -- start start 2024-02-19T21:06:18.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.258+0000 7f52da0d4700 1 librados: setting wanted keys 2024-02-19T21:06:18.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.258+0000 7f52da0d4700 1 librados: calling monclient init 2024-02-19T21:06:18.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.258+0000 7f52da0d4700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f52d4117540 con 0x7f52d40fc870 2024-02-19T21:06:18.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.258+0000 7f52da0d4700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f52d4118740 con 0x7f52d4060a10 2024-02-19T21:06:18.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.258+0000 7f52da0d4700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f52d4119940 con 0x7f52d40ff670 2024-02-19T21:06:18.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.258+0000 7f52d2ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f52d40fc870 0x7f52d40f92c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:52256/0 (socket says 172.21.15.163:52256) 2024-02-19T21:06:18.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.258+0000 7f52d2ffd700 1 -- 172.21.15.163:0/2979342619 learned_addr learned my addr 172.21.15.163:0/2979342619 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:18.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 10 client.?.objecter ms_handle_connect 0x7f52d40ff670 2024-02-19T21:06:18.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:18.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 10 client.?.objecter ms_handle_connect 0x7f52d40fc870 2024-02-19T21:06:18.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:18.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 10 client.?.objecter ms_handle_connect 0x7f52d4060a10 2024-02-19T21:06:18.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:18.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391665492 0 0) 0x7f52d4119940 con 0x7f52d40ff670 2024-02-19T21:06:18.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52bc003630 con 0x7f52d40ff670 2024-02-19T21:06:18.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260680767 0 0) 0x7f52d4118740 con 0x7f52d4060a10 2024-02-19T21:06:18.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d4119940 con 0x7f52d4060a10 2024-02-19T21:06:18.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4235619840 0 0) 0x7f52bc003630 con 0x7f52d40ff670 2024-02-19T21:06:18.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52d4118740 con 0x7f52d40ff670 2024-02-19T21:06:18.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f52cc001d10 con 0x7f52d40ff670 2024-02-19T21:06:18.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 551042048 0 0) 0x7f52d4119940 con 0x7f52d4060a10 2024-02-19T21:06:18.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52bc003630 con 0x7f52d4060a10 2024-02-19T21:06:18.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f52c40017b0 con 0x7f52d4060a10 2024-02-19T21:06:18.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1651540072 0 0) 0x7f52d4118740 con 0x7f52d40ff670 2024-02-19T21:06:18.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 >> v1:172.21.15.193:6789/0 conn(0x7f52d4060a10 legacy=0x7f52d40f8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:18.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 >> v1:172.21.15.163:6789/0 conn(0x7f52d40fc870 legacy=0x7f52d40f92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:18.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d411ab40 con 0x7f52d40ff670 2024-02-19T21:06:18.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52da0d4700 1 -- 172.21.15.163:0/2979342619 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d4117730 con 0x7f52d40ff670 2024-02-19T21:06:18.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52da0d4700 10 client.49136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:18.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52da0d4700 1 -- 172.21.15.163:0/2979342619 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d4117ba0 con 0x7f52d40ff670 2024-02-19T21:06:18.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.259+0000 7f52da0d4700 1 librados: init done 2024-02-19T21:06:18.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.260+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f52cc0032e0 con 0x7f52d40ff670 2024-02-19T21:06:18.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.260+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f52cc003b00 con 0x7f52d40ff670 2024-02-19T21:06:18.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.261+0000 7f52da0d4700 1 -- 172.21.15.163:0/2979342619 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c0004f20 con 0x7f52d40ff670 2024-02-19T21:06:18.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.261+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f52cc016990 con 0x7f52d40ff670 2024-02-19T21:06:18.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.261+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(198..198 src has 43..198) v4 ==== 8397+0+0 (unknown 2068492508 0 0) 0x7f52cc076c00 con 0x7f52d40ff670 2024-02-19T21:06:18.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.261+0000 7f52d0ff9700 10 client.49136.objecter ms_dispatch 0x7f52d4001550 osd_map(198..198 src has 43..198) v4 2024-02-19T21:06:18.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.261+0000 7f52d0ff9700 3 client.49136.objecter handle_osd_map got epochs [198,198] > 0 2024-02-19T21:06:18.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.261+0000 7f52d0ff9700 3 client.49136.objecter handle_osd_map decoding full epoch 198 2024-02-19T21:06:18.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.262+0000 7f52d0ff9700 20 client.49136.objecter dump_active .. 0 homeless 2024-02-19T21:06:18.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.262+0000 7f52d0ff9700 10 client.49136.objecter ms_handle_connect 0x7f52bc063ad0 2024-02-19T21:06:18.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.266+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f52cc049f80 con 0x7f52d40ff670 2024-02-19T21:06:18.278 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:18.377 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:18.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:18.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:18.412 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:18.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.467+0000 7f52da0d4700 1 -- 172.21.15.163:0/2979342619 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f52c0000b90 con 0x7f52bc063ad0 2024-02-19T21:06:18.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.485+0000 7f52d0ff9700 1 -- 172.21.15.163:0/2979342619 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+653253 (unknown 2965378022 0 3135166608) 0x7f52c0000b90 con 0x7f52bc063ad0 2024-02-19T21:06:18.486 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:18.489 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:06:18.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.490+0000 7f52da0d4700 10 librados: watch_flush enter 2024-02-19T21:06:18.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.490+0000 7f52da0d4700 10 librados: watch_flush exit 2024-02-19T21:06:18.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.490+0000 7f52da0d4700 20 client.49136.objecter shutdown clearing up homeless session... 2024-02-19T21:06:18.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.490+0000 7f52da0d4700 10 client.49136.objecter successfully canceled tick 2024-02-19T21:06:18.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.490+0000 7f52da0d4700 1 -- 172.21.15.163:0/2979342619 >> v1:172.21.15.163:6800/1552514681 conn(0x7f52bc063ad0 legacy=0x7f52bc065f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:18.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.490+0000 7f52da0d4700 1 -- 172.21.15.163:0/2979342619 >> v1:172.21.15.163:6790/0 conn(0x7f52d40ff670 legacy=0x7f52d4115c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:18.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.490+0000 7f52da0d4700 1 -- 172.21.15.163:0/2979342619 shutdown_connections 2024-02-19T21:06:18.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.490+0000 7f52da0d4700 1 -- 172.21.15.163:0/2979342619 >> 172.21.15.163:0/2979342619 conn(0x7f52d40f2470 msgr2=0x7f52d40f4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:18.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.490+0000 7f52da0d4700 1 -- 172.21.15.163:0/2979342619 shutdown_connections 2024-02-19T21:06:18.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.490+0000 7f52da0d4700 1 -- 172.21.15.163:0/2979342619 wait complete. 2024-02-19T21:06:18.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:18.490+0000 7f52da0d4700 1 librados: shutdown 2024-02-19T21:06:18.496 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:06:18.524 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:18.543 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:18.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:18.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:18.642 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:16.716250+0000 osd.4 (osd.4) 105 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:06:18.642 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:16.717934+0000 osd.4 (osd.4) 106 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:06:18.642 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:16.779271+0000 osd.0 (osd.0) 109 : cluster [DBG] 16.15 scrub starts 2024-02-19T21:06:18.642 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:16.780597+0000 osd.0 (osd.0) 110 : cluster [DBG] 16.15 scrub ok 2024-02-19T21:06:18.642 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:16.826610+0000 osd.6 (osd.6) 97 : cluster [DBG] 8.5 scrub starts 2024-02-19T21:06:18.642 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:16.827558+0000 osd.6 (osd.6) 98 : cluster [DBG] 8.5 scrub ok 2024-02-19T21:06:18.642 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:16.860426+0000 osd.5 (osd.5) 92 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:06:18.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:16.863394+0000 osd.5 (osd.5) 93 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:06:18.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:17.139660+0000 mgr.y (mgr.14550) 231 : cluster [DBG] pgmap v266: 301 pgs: 1 clean+premerge+peered, 11 active+clean+snaptrim, 40 unknown, 19 active+clean+snaptrim_wait, 230 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:18.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:17.200435+0000 mon.a (mon.0) 1431 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:06:18.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:17.298559+0000 mon.c (mon.2) 204 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:18.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:18.185601+0000 mon.a (mon.0) 1432 : audit [INF] from='client.47941 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi163-83845-28"}]': finished 2024-02-19T21:06:18.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:18.185753+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-02-19T21:06:18.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:18.192240+0000 mon.b (mon.1) 310 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:18.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:18.197147+0000 mon.a (mon.0) 1434 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:18.643 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:16.716250+0000 osd.4 (osd.4) 105 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:06:18.643 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:16.717934+0000 osd.4 (osd.4) 106 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:06:18.643 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:16.779271+0000 osd.0 (osd.0) 109 : cluster [DBG] 16.15 scrub starts 2024-02-19T21:06:18.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:16.780597+0000 osd.0 (osd.0) 110 : cluster [DBG] 16.15 scrub ok 2024-02-19T21:06:18.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:16.826610+0000 osd.6 (osd.6) 97 : cluster [DBG] 8.5 scrub starts 2024-02-19T21:06:18.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:16.827558+0000 osd.6 (osd.6) 98 : cluster [DBG] 8.5 scrub ok 2024-02-19T21:06:18.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:16.860426+0000 osd.5 (osd.5) 92 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:06:18.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:16.863394+0000 osd.5 (osd.5) 93 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:06:18.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:17.139660+0000 mgr.y (mgr.14550) 231 : cluster [DBG] pgmap v266: 301 pgs: 1 clean+premerge+peered, 11 active+clean+snaptrim, 40 unknown, 19 active+clean+snaptrim_wait, 230 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:18.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:17.200435+0000 mon.a (mon.0) 1431 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:06:18.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:17.298559+0000 mon.c (mon.2) 204 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:18.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:18.185601+0000 mon.a (mon.0) 1432 : audit [INF] from='client.47941 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi163-83845-28"}]': finished 2024-02-19T21:06:18.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:18.185753+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-02-19T21:06:18.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:18.192240+0000 mon.b (mon.1) 310 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:18.644 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:18.197147+0000 mon.a (mon.0) 1434 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:18.647 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:18.679 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:18.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:16.716250+0000 osd.4 (osd.4) 105 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:06:18.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:16.717934+0000 osd.4 (osd.4) 106 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:06:18.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:16.779271+0000 osd.0 (osd.0) 109 : cluster [DBG] 16.15 scrub starts 2024-02-19T21:06:18.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:16.780597+0000 osd.0 (osd.0) 110 : cluster [DBG] 16.15 scrub ok 2024-02-19T21:06:18.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:16.826610+0000 osd.6 (osd.6) 97 : cluster [DBG] 8.5 scrub starts 2024-02-19T21:06:18.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:16.827558+0000 osd.6 (osd.6) 98 : cluster [DBG] 8.5 scrub ok 2024-02-19T21:06:18.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:16.860426+0000 osd.5 (osd.5) 92 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:06:18.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:16.863394+0000 osd.5 (osd.5) 93 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:06:18.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:17.139660+0000 mgr.y (mgr.14550) 231 : cluster [DBG] pgmap v266: 301 pgs: 1 clean+premerge+peered, 11 active+clean+snaptrim, 40 unknown, 19 active+clean+snaptrim_wait, 230 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:18.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:17.200435+0000 mon.a (mon.0) 1431 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:06:18.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:17.298559+0000 mon.c (mon.2) 204 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:18.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:18.185601+0000 mon.a (mon.0) 1432 : audit [INF] from='client.47941 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi163-83845-28"}]': finished 2024-02-19T21:06:18.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:18.185753+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-02-19T21:06:18.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:18.192240+0000 mon.b (mon.1) 310 : audit [INF] from='client.? v1:172.21.15.163:0/1153528047' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:18.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:18.197147+0000 mon.a (mon.0) 1434 : audit [INF] from='client.47941 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi163-83845-28"}]: dispatch 2024-02-19T21:06:18.826 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:18.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:18.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:18.930 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:18.959 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:19.061 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:06:19.061 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:06:19.061 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-02-19T21:06:19.062 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-02-19T21:06:19.062 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:06:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:19.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:19.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:19.199 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:19.237 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:19.440 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:19.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:19.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:19.492 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:17.471937+0000 osd.7 (osd.7) 105 : cluster [DBG] 2.26 scrub starts 2024-02-19T21:06:19.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:17.474947+0000 osd.7 (osd.7) 106 : cluster [DBG] 2.26 scrub ok 2024-02-19T21:06:19.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:17.699700+0000 osd.1 (osd.1) 117 : cluster [DBG] 8.7 scrub starts 2024-02-19T21:06:19.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:17.703182+0000 osd.1 (osd.1) 118 : cluster [DBG] 8.7 scrub ok 2024-02-19T21:06:19.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:17.773165+0000 osd.0 (osd.0) 111 : cluster [DBG] 16.0 deep-scrub starts 2024-02-19T21:06:19.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:17.776053+0000 osd.0 (osd.0) 112 : cluster [DBG] 16.0 deep-scrub ok 2024-02-19T21:06:19.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:17.904374+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:06:19.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:17.907313+0000 osd.5 (osd.5) 95 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:06:19.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:18.299346+0000 mon.c (mon.2) 205 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:19.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:18.455912+0000 mon.a (mon.0) 1435 : audit [INF] from='client.? v1:172.21.15.163:0/3656220656' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi163-83854-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:19.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:18.461547+0000 osd.7 (osd.7) 107 : cluster [DBG] 2.20 scrub starts 2024-02-19T21:06:19.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:18.465057+0000 osd.7 (osd.7) 108 : cluster [DBG] 2.20 scrub ok 2024-02-19T21:06:19.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:18.468186+0000 mgr.y (mgr.14550) 232 : audit [DBG] from='client.49136 v1:172.21.15.163:0/2979342619' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:06:19.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:18.703648+0000 osd.4 (osd.4) 107 : cluster [DBG] 6.e scrub starts 2024-02-19T21:06:19.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:18.704443+0000 osd.4 (osd.4) 108 : cluster [DBG] 6.e scrub ok 2024-02-19T21:06:19.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:19.187702+0000 mon.a (mon.0) 1436 : audit [INF] from='client.47941 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi163-83845-28"}]': finished 2024-02-19T21:06:19.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:19.187832+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? v1:172.21.15.163:0/3656220656' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi163-83854-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:19.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:19.187914+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-02-19T21:06:19.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:17.471937+0000 osd.7 (osd.7) 105 : cluster [DBG] 2.26 scrub starts 2024-02-19T21:06:19.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:17.474947+0000 osd.7 (osd.7) 106 : cluster [DBG] 2.26 scrub ok 2024-02-19T21:06:19.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:17.699700+0000 osd.1 (osd.1) 117 : cluster [DBG] 8.7 scrub starts 2024-02-19T21:06:19.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:17.703182+0000 osd.1 (osd.1) 118 : cluster [DBG] 8.7 scrub ok 2024-02-19T21:06:19.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:17.773165+0000 osd.0 (osd.0) 111 : cluster [DBG] 16.0 deep-scrub starts 2024-02-19T21:06:19.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:17.776053+0000 osd.0 (osd.0) 112 : cluster [DBG] 16.0 deep-scrub ok 2024-02-19T21:06:19.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:17.904374+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:06:19.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:17.907313+0000 osd.5 (osd.5) 95 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:06:19.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:18.299346+0000 mon.c (mon.2) 205 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:19.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:18.455912+0000 mon.a (mon.0) 1435 : audit [INF] from='client.? v1:172.21.15.163:0/3656220656' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi163-83854-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:19.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:18.461547+0000 osd.7 (osd.7) 107 : cluster [DBG] 2.20 scrub starts 2024-02-19T21:06:19.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:18.465057+0000 osd.7 (osd.7) 108 : cluster [DBG] 2.20 scrub ok 2024-02-19T21:06:19.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:18.468186+0000 mgr.y (mgr.14550) 232 : audit [DBG] from='client.49136 v1:172.21.15.163:0/2979342619' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:06:19.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:18.703648+0000 osd.4 (osd.4) 107 : cluster [DBG] 6.e scrub starts 2024-02-19T21:06:19.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:18.704443+0000 osd.4 (osd.4) 108 : cluster [DBG] 6.e scrub ok 2024-02-19T21:06:19.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:19.187702+0000 mon.a (mon.0) 1436 : audit [INF] from='client.47941 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi163-83845-28"}]': finished 2024-02-19T21:06:19.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:19.187832+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? v1:172.21.15.163:0/3656220656' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi163-83854-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:19.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:19.187914+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-02-19T21:06:19.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:19.190863+0000 mon.b (mon.1) 311 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:19.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:19.197120+0000 mon.a (mon.0) 1439 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:19.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:19.224345+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi163-83845-29"}]: dispatch 2024-02-19T21:06:19.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:19.225500+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi163-83845-29"}]: dispatch 2024-02-19T21:06:19.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:19.225926+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi163-83845-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:19.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:19.300586+0000 mon.c (mon.2) 206 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:19.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:19.190863+0000 mon.b (mon.1) 311 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:19.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:19.197120+0000 mon.a (mon.0) 1439 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:19.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:19.224345+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi163-83845-29"}]: dispatch 2024-02-19T21:06:19.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:19.225500+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi163-83845-29"}]: dispatch 2024-02-19T21:06:19.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:19.225926+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi163-83845-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:19.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:19.300586+0000 mon.c (mon.2) 206 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:19.555 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:19.583 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:19.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:17.471937+0000 osd.7 (osd.7) 105 : cluster [DBG] 2.26 scrub starts 2024-02-19T21:06:19.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:17.474947+0000 osd.7 (osd.7) 106 : cluster [DBG] 2.26 scrub ok 2024-02-19T21:06:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:17.699700+0000 osd.1 (osd.1) 117 : cluster [DBG] 8.7 scrub starts 2024-02-19T21:06:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:17.703182+0000 osd.1 (osd.1) 118 : cluster [DBG] 8.7 scrub ok 2024-02-19T21:06:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:17.773165+0000 osd.0 (osd.0) 111 : cluster [DBG] 16.0 deep-scrub starts 2024-02-19T21:06:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:17.776053+0000 osd.0 (osd.0) 112 : cluster [DBG] 16.0 deep-scrub ok 2024-02-19T21:06:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:17.904374+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:06:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:17.907313+0000 osd.5 (osd.5) 95 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:06:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:18.299346+0000 mon.c (mon.2) 205 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:18.455912+0000 mon.a (mon.0) 1435 : audit [INF] from='client.? v1:172.21.15.163:0/3656220656' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi163-83854-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:18.461547+0000 osd.7 (osd.7) 107 : cluster [DBG] 2.20 scrub starts 2024-02-19T21:06:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:18.465057+0000 osd.7 (osd.7) 108 : cluster [DBG] 2.20 scrub ok 2024-02-19T21:06:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:18.468186+0000 mgr.y (mgr.14550) 232 : audit [DBG] from='client.49136 v1:172.21.15.163:0/2979342619' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:06:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:18.703648+0000 osd.4 (osd.4) 107 : cluster [DBG] 6.e scrub starts 2024-02-19T21:06:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:18.704443+0000 osd.4 (osd.4) 108 : cluster [DBG] 6.e scrub ok 2024-02-19T21:06:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:19.187702+0000 mon.a (mon.0) 1436 : audit [INF] from='client.47941 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi163-83845-28"}]': finished 2024-02-19T21:06:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:19.187832+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? v1:172.21.15.163:0/3656220656' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi163-83854-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:19.187914+0000 mon.a (mon.0) 1438 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-02-19T21:06:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:19.190863+0000 mon.b (mon.1) 311 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:19.197120+0000 mon.a (mon.0) 1439 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:19.224345+0000 mon.a (mon.0) 1440 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi163-83845-29"}]: dispatch 2024-02-19T21:06:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:19.225500+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi163-83845-29"}]: dispatch 2024-02-19T21:06:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:19.225926+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi163-83845-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:19.300586+0000 mon.c (mon.2) 206 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:20.200 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-02-19T21:06:20.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [==========] Running 56 tests from 4 test suites. 2024-02-19T21:06:20.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-02-19T21:06:20.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [----------] 31 tests from LibRadosAio 2024-02-19T21:06:20.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-02-19T21:06:20.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (7970 ms) 2024-02-19T21:06:20.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-02-19T21:06:20.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (21361 ms) 2024-02-19T21:06:20.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-02-19T21:06:20.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (5999 ms) 2024-02-19T21:06:20.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-02-19T21:06:20.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2853 ms) 2024-02-19T21:06:20.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-02-19T21:06:20.201 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3171 ms) 2024-02-19T21:06:20.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-02-19T21:06:20.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3136 ms) 2024-02-19T21:06:20.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-02-19T21:06:20.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3344 ms) 2024-02-19T21:06:20.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-02-19T21:06:20.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2515 ms) 2024-02-19T21:06:20.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-02-19T21:06:20.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2999 ms) 2024-02-19T21:06:20.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-02-19T21:06:20.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2996 ms) 2024-02-19T21:06:20.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-02-19T21:06:20.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3010 ms) 2024-02-19T21:06:20.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-02-19T21:06:20.202 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3001 ms) 2024-02-19T21:06:20.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-02-19T21:06:20.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3003 ms) 2024-02-19T21:06:20.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-02-19T21:06:20.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2992 ms) 2024-02-19T21:06:20.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-02-19T21:06:20.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3003 ms) 2024-02-19T21:06:20.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-02-19T21:06:20.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3005 ms) 2024-02-19T21:06:20.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-02-19T21:06:20.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3231 ms) 2024-02-19T21:06:20.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-02-19T21:06:20.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2786 ms) 2024-02-19T21:06:20.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-02-19T21:06:20.203 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2986 ms) 2024-02-19T21:06:20.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-02-19T21:06:20.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3013 ms) 2024-02-19T21:06:20.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-02-19T21:06:20.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3000 ms) 2024-02-19T21:06:20.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-02-19T21:06:20.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2974 ms) 2024-02-19T21:06:20.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-02-19T21:06:20.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3012 ms) 2024-02-19T21:06:20.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-02-19T21:06:20.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3021 ms) 2024-02-19T21:06:20.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-02-19T21:06:20.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3201 ms) 2024-02-19T21:06:20.204 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-02-19T21:06:20.205 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (2766 ms) 2024-02-19T21:06:20.205 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-02-19T21:06:20.233 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:20.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:20.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:20.342 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:20.369 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:20.479 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:06:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:06:20.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:18.794281+0000 osd.0 (osd.0) 113 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:06:20.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:18.797427+0000 osd.0 (osd.0) 114 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:06:20.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:19.140152+0000 mgr.y (mgr.14550) 233 : cluster [DBG] pgmap v269: 261 pgs: 1 clean+premerge+peered, 11 active+clean+snaptrim, 32 unknown, 19 active+clean+snaptrim_wait, 198 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:20.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:20.192413+0000 mon.a (mon.0) 1443 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:20.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:20.192568+0000 mon.a (mon.0) 1444 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi163-83845-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:20.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:20.192627+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-02-19T21:06:20.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:20.198951+0000 mon.a (mon.0) 1446 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi163-83845-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi163-83845-29"}]: dispatch 2024-02-19T21:06:20.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:20.231896+0000 mon.b (mon.1) 312 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:20.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:20.232779+0000 mon.a (mon.0) 1447 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:20.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:18.794281+0000 osd.0 (osd.0) 113 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:06:20.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:18.797427+0000 osd.0 (osd.0) 114 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:06:20.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:19.140152+0000 mgr.y (mgr.14550) 233 : cluster [DBG] pgmap v269: 261 pgs: 1 clean+premerge+peered, 11 active+clean+snaptrim, 32 unknown, 19 active+clean+snaptrim_wait, 198 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:20.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:20.192413+0000 mon.a (mon.0) 1443 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:20.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:20.192568+0000 mon.a (mon.0) 1444 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi163-83845-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:20.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:20.192627+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-02-19T21:06:20.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:20.198951+0000 mon.a (mon.0) 1446 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi163-83845-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi163-83845-29"}]: dispatch 2024-02-19T21:06:20.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:20.231896+0000 mon.b (mon.1) 312 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:20.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:20.232779+0000 mon.a (mon.0) 1447 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:20.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:18.794281+0000 osd.0 (osd.0) 113 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:06:20.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:18.797427+0000 osd.0 (osd.0) 114 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:06:20.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:19.140152+0000 mgr.y (mgr.14550) 233 : cluster [DBG] pgmap v269: 261 pgs: 1 clean+premerge+peered, 11 active+clean+snaptrim, 32 unknown, 19 active+clean+snaptrim_wait, 198 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:20.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:20.192413+0000 mon.a (mon.0) 1443 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:20.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:20.192568+0000 mon.a (mon.0) 1444 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi163-83845-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:20.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:20.192627+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-02-19T21:06:20.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:20.198951+0000 mon.a (mon.0) 1446 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi163-83845-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi163-83845-29"}]: dispatch 2024-02-19T21:06:20.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:20.231896+0000 mon.b (mon.1) 312 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:20.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:20.232779+0000 mon.a (mon.0) 1447 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:20.949 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-19T21:06:21.656 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:19.712945+0000 osd.1 (osd.1) 119 : cluster [DBG] 8.b deep-scrub starts 2024-02-19T21:06:21.656 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:19.717592+0000 osd.1 (osd.1) 120 : cluster [DBG] 8.b deep-scrub ok 2024-02-19T21:06:21.656 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:19.842500+0000 osd.5 (osd.5) 96 : cluster [DBG] 2.2 deep-scrub starts 2024-02-19T21:06:21.656 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:19.845389+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.2 deep-scrub ok 2024-02-19T21:06:21.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:20.154758+0000 mgr.y (mgr.14550) 234 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:21.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:20.187918+0000 osd.2 (osd.2) 89 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:06:21.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:20.194010+0000 osd.2 (osd.2) 90 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:06:21.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:20.301338+0000 mon.c (mon.2) 207 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:21.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:20.709127+0000 osd.4 (osd.4) 109 : cluster [DBG] 5.e scrub starts 2024-02-19T21:06:21.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:20.710725+0000 osd.4 (osd.4) 110 : cluster [DBG] 5.e scrub ok 2024-02-19T21:06:21.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:21.141142+0000 mon.b (mon.1) 313 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-19T21:06:21.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:21.142106+0000 mon.a (mon.0) 1448 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-19T21:06:21.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:21.302169+0000 mon.c (mon.2) 208 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:21.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:21.437073+0000 mon.a (mon.0) 1449 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:21.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:21.437176+0000 mon.a (mon.0) 1450 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-19T21:06:21.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:21.437226+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-02-19T21:06:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:19.712945+0000 osd.1 (osd.1) 119 : cluster [DBG] 8.b deep-scrub starts 2024-02-19T21:06:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:19.717592+0000 osd.1 (osd.1) 120 : cluster [DBG] 8.b deep-scrub ok 2024-02-19T21:06:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:19.842500+0000 osd.5 (osd.5) 96 : cluster [DBG] 2.2 deep-scrub starts 2024-02-19T21:06:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:19.845389+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.2 deep-scrub ok 2024-02-19T21:06:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:20.154758+0000 mgr.y (mgr.14550) 234 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:20.187918+0000 osd.2 (osd.2) 89 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:06:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:20.194010+0000 osd.2 (osd.2) 90 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:06:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:20.301338+0000 mon.c (mon.2) 207 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:20.709127+0000 osd.4 (osd.4) 109 : cluster [DBG] 5.e scrub starts 2024-02-19T21:06:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:20.710725+0000 osd.4 (osd.4) 110 : cluster [DBG] 5.e scrub ok 2024-02-19T21:06:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:21.141142+0000 mon.b (mon.1) 313 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-19T21:06:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:21.142106+0000 mon.a (mon.0) 1448 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-19T21:06:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:21.302169+0000 mon.c (mon.2) 208 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:21.437073+0000 mon.a (mon.0) 1449 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:21.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:21.437176+0000 mon.a (mon.0) 1450 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-19T21:06:21.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:21.437226+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-02-19T21:06:21.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:21.437958+0000 mon.b (mon.1) 314 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-17"}]: dispatch 2024-02-19T21:06:21.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:21.439405+0000 mon.b (mon.1) 315 : audit [INF] from='client.? v1:172.21.15.163:0/3616630607' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi163-83854-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:21.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:21.439622+0000 mon.a (mon.0) 1452 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-17"}]: dispatch 2024-02-19T21:06:21.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:21.440379+0000 mon.a (mon.0) 1453 : audit [INF] from='client.47968 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi163-83854-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:21.659 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:21.437958+0000 mon.b (mon.1) 314 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-17"}]: dispatch 2024-02-19T21:06:21.659 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:21.439405+0000 mon.b (mon.1) 315 : audit [INF] from='client.? v1:172.21.15.163:0/3616630607' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi163-83854-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:21.659 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:21.439622+0000 mon.a (mon.0) 1452 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-17"}]: dispatch 2024-02-19T21:06:21.659 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:21.440379+0000 mon.a (mon.0) 1453 : audit [INF] from='client.47968 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi163-83854-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:21.675 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-02-19T21:06:18.183124+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11116,"num_objects":564,"num_object_clones":250,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":564,"num_whiteouts":0,"num_read":1099,"num_read_kb":766,"num_write":1072,"num_write_kb":629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":2575,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1785,"ondisk_log_size":1785,"up":687,"acting":687,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":869,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2437600,"kb_used_data":26972,"kb_used_omap":0,"kb_used_meta":2410496,"kb_avail":747523616,"statfs":{"total":767960285184,"available":765464182784,"internally_reserved":0,"allocated":27619328,"data_stored":13281212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2468347904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33430,"num_objects":-65,"num_object_clones":-63,"num_object_copies":-195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-65,"num_whiteouts":-1,"num_read":-2,"num_read_kb":-27,"num_write":-10,"num_write_kb":-49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008399"},"pg_stats":[{"pgid":"138.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"138.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:06:18.179834+0000","last_change":"2024-02-19T21:06:18.179834+0000","last_active":"2024-02-19T21:06:18.179834+0000","last_peered":"2024-02-19T21:06:18.179834+0000","last_clean":"2024-02-19T21:06:18.179834+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:18.179834+0000","last_undegraded":"2024-02-19T21:06:18.179834+0000","last_fullsized":"2024-02-19T21:06:18.179834+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:18.179834+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.179834+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.2","version":"194'53","reported_seq":151,"reported_epoch":194,"state":"active+clean+snaptrim","last_fresh":"2024-02-19T21:06:14.244921+0000","last_change":"2024-02-19T21:06:14.243778+0000","last_active":"2024-02-19T21:06:14.244921+0000","last_peered":"2024-02-19T21:06:14.244921+0000","last_clean":"2024-02-19T21:06:14.244921+0000","last_became_active":"2024-02-19T21:04:54.609091+0000","last_became_peered":"2024-02-19T21:04:54.609091+0000","last_unstale":"2024-02-19T21:06:14.244921+0000","last_undegraded":"2024-02-19T21:06:14.244921+0000","last_fullsized":"2024-02-19T21:06:14.244921+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":53,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":159,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"61.3","version":"188'44","reported_seq":134,"reported_epoch":193,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:13.202650+0000","last_change":"2024-02-19T21:06:10.245185+0000","last_active":"2024-02-19T21:06:13.202650+0000","last_peered":"2024-02-19T21:06:13.202650+0000","last_clean":"2024-02-19T21:06:13.202650+0000","last_became_active":"2024-02-19T21:04:54.641884+0000","last_became_peered":"2024-02-19T21:04:54.641884+0000","last_unstale":"2024-02-19T21:06:13.202650+0000","last_undegraded":"2024-02-19T21:06:13.202650+0000","last_fullsized":"2024-02-19T21:06:13.202650+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":176,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"61.0","version":"192'30","reported_seq":132,"reported_epoch":194,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:14.252224+0000","last_change":"2024-02-19T21:06:14.252274+0000","last_active":"2024-02-19T21:06:14.252224+0000","last_peered":"2024-02-19T21:06:14.252224+0000","last_clean":"2024-02-19T21:06:14.252224+0000","last_became_active":"2024-02-19T21:04:54.656450+0000","last_became_peered":"2024-02-19T21:04:54.656450+0000","last_unstale":"2024-02-19T21:06:14.252224+0000","last_undegraded":"2024-02-19T21:06:14.252224+0000","last_fullsized":"2024-02-19T21:06:14.252224+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"3"}]},{"pgid":"61.1","version":"184'44","reported_seq":139,"reported_epoch":193,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:13.202601+0000","last_change":"2024-02-19T21:06:06.235786+0000","last_active":"2024-02-19T21:06:13.202601+0000","last_peered":"2024-02-19T21:06:13.202601+0000","last_clean":"2024-02-19T21:06:13.202601+0000","last_became_active":"2024-02-19T21:04:54.654642+0000","last_became_peered":"2024-02-19T21:04:54.654642+0000","last_unstale":"2024-02-19T21:06:13.202601+0000","last_undegraded":"2024-02-19T21:06:13.202601+0000","last_fullsized":"2024-02-19T21:06:13.202601+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":164,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"61.6","version":"190'26","reported_seq":122,"reported_epoch":193,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:13.200702+0000","last_change":"2024-02-19T21:06:12.247782+0000","last_active":"2024-02-19T21:06:13.200702+0000","last_peered":"2024-02-19T21:06:13.200702+0000","last_clean":"2024-02-19T21:06:13.200702+0000","last_became_active":"2024-02-19T21:04:54.658542+0000","last_became_peered":"2024-02-19T21:04:54.658542+0000","last_unstale":"2024-02-19T21:06:13.200702+0000","last_undegraded":"2024-02-19T21:06:13.200702+0000","last_fullsized":"2024-02-19T21:06:13.200702+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"2"}]},{"pgid":"61.7","version":"192'12","reported_seq":110,"reported_epoch":194,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:14.236311+0000","last_change":"2024-02-19T21:06:14.236350+0000","last_active":"2024-02-19T21:06:14.236311+0000","last_peered":"2024-02-19T21:06:14.236311+0000","last_clean":"2024-02-19T21:06:14.236311+0000","last_became_active":"2024-02-19T21:04:54.598061+0000","last_became_peered":"2024-02-19T21:04:54.598061+0000","last_unstale":"2024-02-19T21:06:14.236311+0000","last_undegraded":"2024-02-19T21:06:14.236311+0000","last_fullsized":"2024-02-19T21:06:14.236311+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":48,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"61.4","version":"192'52","reported_seq":145,"reported_epoch":194,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:14.252543+0000","last_change":"2024-02-19T21:06:14.252599+0000","last_active":"2024-02-19T21:06:14.252543+0000","last_peered":"2024-02-19T21:06:14.252543+0000","last_clean":"2024-02-19T21:06:14.252543+0000","last_became_active":"2024-02-19T21:04:54.639335+0000","last_became_peered":"2024-02-19T21:04:54.639335+0000","last_unstale":"2024-02-19T21:06:14.252543+0000","last_undegraded":"2024-02-19T21:06:14.252543+0000","last_fullsized":"2024-02-19T21:06:14.252543+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":132,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"2"}]},{"pgid":"61.5","version":"194'77","reported_seq":159,"reported_epoch":194,"state":"active+clean+snaptrim","last_fresh":"2024-02-19T21:06:14.250855+0000","last_change":"2024-02-19T21:06:08.245430+0000","last_active":"2024-02-19T21:06:14.250855+0000","last_peered":"2024-02-19T21:06:14.250855+0000","last_clean":"2024-02-19T21:06:14.250855+0000","last_became_active":"2024-02-19T21:04:54.646842+0000","last_became_peered":"2024-02-19T21:04:54.646842+0000","last_unstale":"2024-02-19T21:06:14.250855+0000","last_undegraded":"2024-02-19T21:06:14.250855+0000","last_fullsized":"2024-02-19T21:06:14.250855+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":308,"num_objects":28,"num_object_clones":21,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.a","version":"192'13","reported_seq":114,"reported_epoch":194,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:14.243235+0000","last_change":"2024-02-19T21:06:14.243283+0000","last_active":"2024-02-19T21:06:14.243235+0000","last_peered":"2024-02-19T21:06:14.243235+0000","last_clean":"2024-02-19T21:06:14.243235+0000","last_became_active":"2024-02-19T21:04:54.611831+0000","last_became_peered":"2024-02-19T21:04:54.611831+0000","last_unstale":"2024-02-19T21:06:14.243235+0000","last_undegraded":"2024-02-19T21:06:14.243235+0000","last_fullsized":"2024-02-19T21:06:14.243235+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"2"}]},{"pgid":"61.b","version":"194'55","reported_seq":138,"reported_epoch":194,"state":"active+clean+snaptrim","last_fresh":"2024-02-19T21:06:14.248300+0000","last_change":"2024-02-19T21:06:10.245207+0000","last_active":"2024-02-19T21:06:14.248300+0000","last_peered":"2024-02-19T21:06:14.248300+0000","last_clean":"2024-02-19T21:06:14.248300+0000","last_became_active":"2024-02-19T21:04:54.640024+0000","last_became_peered":"2024-02-19T21:04:54.640024+0000","last_unstale":"2024-02-19T21:06:14.248300+0000","last_undegraded":"2024-02-19T21:06:14.248300+0000","last_fullsized":"2024-02-19T21:06:14.248300+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":220,"num_objects":20,"num_object_clones":15,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.8","version":"194'26","reported_seq":115,"reported_epoch":194,"state":"active+clean+snaptrim","last_fresh":"2024-02-19T21:06:14.239996+0000","last_change":"2024-02-19T21:06:14.236436+0000","last_active":"2024-02-19T21:06:14.239996+0000","last_peered":"2024-02-19T21:06:14.239996+0000","last_clean":"2024-02-19T21:06:14.239996+0000","last_became_active":"2024-02-19T21:04:54.650062+0000","last_became_peered":"2024-02-19T21:04:54.650062+0000","last_unstale":"2024-02-19T21:06:14.239996+0000","last_undegraded":"2024-02-19T21:06:14.239996+0000","last_fullsized":"2024-02-19T21:06:14.239996+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"61.9","version":"184'44","reported_seq":139,"reported_epoch":193,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:13.203150+0000","last_change":"2024-02-19T21:06:06.236030+0000","last_active":"2024-02-19T21:06:13.203150+0000","last_peered":"2024-02-19T21:06:13.203150+0000","last_clean":"2024-02-19T21:06:13.203150+0000","last_became_active":"2024-02-19T21:04:54.671646+0000","last_became_peered":"2024-02-19T21:04:54.671646+0000","last_unstale":"2024-02-19T21:06:13.203150+0000","last_undegraded":"2024-02-19T21:06:13.203150+0000","last_fullsized":"2024-02-19T21:06:13.203150+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":164,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"61.e","version":"189'34","reported_seq":141,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.199894+0000","last_change":"2024-02-19T21:06:11.235589+0000","last_active":"2024-02-19T21:06:13.199894+0000","last_peered":"2024-02-19T21:06:13.199894+0000","last_clean":"2024-02-19T21:06:13.199894+0000","last_became_active":"2024-02-19T21:04:54.603276+0000","last_became_peered":"2024-02-19T21:04:54.603276+0000","last_unstale":"2024-02-19T21:06:13.199894+0000","last_undegraded":"2024-02-19T21:06:13.199894+0000","last_fullsized":"2024-02-19T21:06:13.199894+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"61.f","version":"192'22","reported_seq":119,"reported_epoch":193,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:14.235778+0000","last_change":"2024-02-19T21:06:14.235778+0000","last_active":"2024-02-19T21:06:14.235778+0000","last_peered":"2024-02-19T21:06:14.235778+0000","last_clean":"2024-02-19T21:06:14.235778+0000","last_became_active":"2024-02-19T21:04:54.610312+0000","last_became_peered":"2024-02-19T21:04:54.610312+0000","last_unstale":"2024-02-19T21:06:14.235778+0000","last_undegraded":"2024-02-19T21:06:14.235778+0000","last_fullsized":"2024-02-19T21:06:14.235778+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":88,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"61.c","version":"190'44","reported_seq":133,"reported_epoch":193,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:13.203926+0000","last_change":"2024-02-19T21:06:12.234105+0000","last_active":"2024-02-19T21:06:13.203926+0000","last_peered":"2024-02-19T21:06:13.203926+0000","last_clean":"2024-02-19T21:06:13.203926+0000","last_became_active":"2024-02-19T21:04:54.634803+0000","last_became_peered":"2024-02-19T21:04:54.634803+0000","last_unstale":"2024-02-19T21:06:13.203926+0000","last_undegraded":"2024-02-19T21:06:13.203926+0000","last_fullsized":"2024-02-19T21:06:13.203926+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":176,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"61.d","version":"186'22","reported_seq":117,"reported_epoch":193,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:13.201800+0000","last_change":"2024-02-19T21:06:08.230565+0000","last_active":"2024-02-19T21:06:13.201800+0000","last_peered":"2024-02-19T21:06:13.201800+0000","last_clean":"2024-02-19T21:06:13.201800+0000","last_became_active":"2024-02-19T21:04:54.644749+0000","last_became_peered":"2024-02-19T21:04:54.644749+0000","last_unstale":"2024-02-19T21:06:13.201800+0000","last_undegraded":"2024-02-19T21:06:13.201800+0000","last_fullsized":"2024-02-19T21:06:13.201800+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":88,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"61.12","version":"191'29","reported_seq":125,"reported_epoch":193,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:13.201309+0000","last_change":"2024-02-19T21:06:10.248674+0000","last_active":"2024-02-19T21:06:13.201309+0000","last_peered":"2024-02-19T21:06:13.201309+0000","last_clean":"2024-02-19T21:06:13.201309+0000","last_became_active":"2024-02-19T21:04:54.670998+0000","last_became_peered":"2024-02-19T21:04:54.670998+0000","last_unstale":"2024-02-19T21:06:13.201309+0000","last_undegraded":"2024-02-19T21:06:13.201309+0000","last_fullsized":"2024-02-19T21:06:13.201309+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":79,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"61.13","version":"193'48","reported_seq":141,"reported_epoch":194,"state":"active+clean+snaptrim","last_fresh":"2024-02-19T21:06:14.242579+0000","last_change":"2024-02-19T21:06:14.236709+0000","last_active":"2024-02-19T21:06:14.242579+0000","last_peered":"2024-02-19T21:06:14.242579+0000","last_clean":"2024-02-19T21:06:14.242579+0000","last_became_active":"2024-02-19T21:04:54.608802+0000","last_became_peered":"2024-02-19T21:04:54.608802+0000","last_unstale":"2024-02-19T21:06:14.242579+0000","last_undegraded":"2024-02-19T21:06:14.242579+0000","last_fullsized":"2024-02-19T21:06:14.242579+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":148,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"61.10","version":"193'31","reported_seq":126,"reported_epoch":194,"state":"active+clean+snaptrim","last_fresh":"2024-02-19T21:06:14.237408+0000","last_change":"2024-02-19T21:06:14.233947+0000","last_active":"2024-02-19T21:06:14.237408+0000","last_peered":"2024-02-19T21:06:14.237408+0000","last_clean":"2024-02-19T21:06:14.237408+0000","last_became_active":"2024-02-19T21:04:54.626393+0000","last_became_peered":"2024-02-19T21:04:54.626393+0000","last_unstale":"2024-02-19T21:06:14.237408+0000","last_undegraded":"2024-02-19T21:06:14.237408+0000","last_fullsized":"2024-02-19T21:06:14.237408+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"61.11","version":"194'30","reported_seq":131,"reported_epoch":194,"state":"active+clean+snaptrim","last_fresh":"2024-02-19T21:06:14.251100+0000","last_change":"2024-02-19T21:06:14.249545+0000","last_active":"2024-02-19T21:06:14.251100+0000","last_peered":"2024-02-19T21:06:14.251100+0000","last_clean":"2024-02-19T21:06:14.251100+0000","last_became_active":"2024-02-19T21:04:54.659096+0000","last_became_peered":"2024-02-19T21:04:54.659096+0000","last_unstale":"2024-02-19T21:06:14.251100+0000","last_undegraded":"2024-02-19T21:06:14.251100+0000","last_fullsized":"2024-02-19T21:06:14.251100+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.29","version":"0'0","reported_seq":67,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.197347+0000","last_change":"2024-02-19T21:05:44.986979+0000","last_active":"2024-02-19T21:06:13.197347+0000","last_peered":"2024-02-19T21:06:13.197347+0000","last_clean":"2024-02-19T21:06:13.197347+0000","last_became_active":"2024-02-19T21:05:22.208034+0000","last_became_peered":"2024-02-19T21:05:22.208034+0000","last_unstale":"2024-02-19T21:06:13.197347+0000","last_undegraded":"2024-02-19T21:06:13.197347+0000","last_fullsized":"2024-02-19T21:06:13.197347+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.986874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.986874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.16","version":"192'22","reported_seq":129,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.225507+0000","last_change":"2024-02-19T21:06:14.225780+0000","last_active":"2024-02-19T21:06:14.225507+0000","last_peered":"2024-02-19T21:06:14.225507+0000","last_clean":"2024-02-19T21:06:14.225507+0000","last_became_active":"2024-02-19T21:04:54.608059+0000","last_became_peered":"2024-02-19T21:04:54.608059+0000","last_unstale":"2024-02-19T21:06:14.225507+0000","last_undegraded":"2024-02-19T21:06:14.225507+0000","last_fullsized":"2024-02-19T21:06:14.225507+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"2.28","version":"0'0","reported_seq":62,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.196498+0000","last_change":"2024-02-19T21:05:22.220592+0000","last_active":"2024-02-19T21:06:13.196498+0000","last_peered":"2024-02-19T21:06:13.196498+0000","last_clean":"2024-02-19T21:06:13.196498+0000","last_became_active":"2024-02-19T21:05:22.220291+0000","last_became_peered":"2024-02-19T21:05:22.220291+0000","last_unstale":"2024-02-19T21:06:13.196498+0000","last_undegraded":"2024-02-19T21:06:13.196498+0000","last_fullsized":"2024-02-19T21:06:13.196498+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.857973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.857973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.17","version":"194'17","reported_seq":123,"reported_epoch":194,"state":"active+clean+snaptrim","last_fresh":"2024-02-19T21:06:14.252349+0000","last_change":"2024-02-19T21:06:14.252349+0000","last_active":"2024-02-19T21:06:14.252349+0000","last_peered":"2024-02-19T21:06:14.252349+0000","last_clean":"2024-02-19T21:06:14.252349+0000","last_became_active":"2024-02-19T21:04:54.646395+0000","last_became_peered":"2024-02-19T21:04:54.646395+0000","last_unstale":"2024-02-19T21:06:14.252349+0000","last_undegraded":"2024-02-19T21:06:14.252349+0000","last_fullsized":"2024-02-19T21:06:14.252349+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"2"},{"start":"7","length":"1"}]},{"pgid":"61.14","version":"194'43","reported_seq":138,"reported_epoch":194,"state":"active+clean+snaptrim","last_fresh":"2024-02-19T21:06:14.234532+0000","last_change":"2024-02-19T21:06:14.231949+0000","last_active":"2024-02-19T21:06:14.234532+0000","last_peered":"2024-02-19T21:06:14.234532+0000","last_clean":"2024-02-19T21:06:14.234532+0000","last_became_active":"2024-02-19T21:04:54.648513+0000","last_became_peered":"2024-02-19T21:04:54.648513+0000","last_unstale":"2024-02-19T21:06:14.234532+0000","last_undegraded":"2024-02-19T21:06:14.234532+0000","last_fullsized":"2024-02-19T21:06:14.234532+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"61.15","version":"190'33","reported_seq":126,"reported_epoch":193,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:13.214245+0000","last_change":"2024-02-19T21:06:12.229518+0000","last_active":"2024-02-19T21:06:13.214245+0000","last_peered":"2024-02-19T21:06:13.214245+0000","last_clean":"2024-02-19T21:06:13.214245+0000","last_became_active":"2024-02-19T21:04:54.626719+0000","last_became_peered":"2024-02-19T21:04:54.626719+0000","last_unstale":"2024-02-19T21:06:13.214245+0000","last_undegraded":"2024-02-19T21:06:13.214245+0000","last_fullsized":"2024-02-19T21:06:13.214245+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":132,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.25","version":"0'0","reported_seq":170,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.201753+0000","last_change":"2024-02-19T21:06:11.639779+0000","last_active":"2024-02-19T21:06:13.201753+0000","last_peered":"2024-02-19T21:06:13.201753+0000","last_clean":"2024-02-19T21:06:13.201753+0000","last_became_active":"2024-02-19T21:05:22.209781+0000","last_became_peered":"2024-02-19T21:05:22.209781+0000","last_unstale":"2024-02-19T21:06:13.201753+0000","last_undegraded":"2024-02-19T21:06:13.201753+0000","last_fullsized":"2024-02-19T21:06:13.201753+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:11.639684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:11.639684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"61.1a","version":"190'55","reported_seq":149,"reported_epoch":193,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:13.209710+0000","last_change":"2024-02-19T21:06:12.233496+0000","last_active":"2024-02-19T21:06:13.209710+0000","last_peered":"2024-02-19T21:06:13.209710+0000","last_clean":"2024-02-19T21:06:13.209710+0000","last_became_active":"2024-02-19T21:04:54.634861+0000","last_became_peered":"2024-02-19T21:04:54.634861+0000","last_unstale":"2024-02-19T21:06:13.209710+0000","last_undegraded":"2024-02-19T21:06:13.209710+0000","last_fullsized":"2024-02-19T21:06:13.209710+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":220,"num_objects":20,"num_object_clones":15,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.24","version":"0'0","reported_seq":67,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.197956+0000","last_change":"2024-02-19T21:06:04.866459+0000","last_active":"2024-02-19T21:06:13.197956+0000","last_peered":"2024-02-19T21:06:13.197956+0000","last_clean":"2024-02-19T21:06:13.197956+0000","last_became_active":"2024-02-19T21:05:22.208647+0000","last_became_peered":"2024-02-19T21:05:22.208647+0000","last_unstale":"2024-02-19T21:06:13.197956+0000","last_undegraded":"2024-02-19T21:06:13.197956+0000","last_fullsized":"2024-02-19T21:06:13.197956+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:04.866394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:06:04.866394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1b","version":"190'39","reported_seq":131,"reported_epoch":193,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:13.198323+0000","last_change":"2024-02-19T21:06:12.241874+0000","last_active":"2024-02-19T21:06:13.198323+0000","last_peered":"2024-02-19T21:06:13.198323+0000","last_clean":"2024-02-19T21:06:13.198323+0000","last_became_active":"2024-02-19T21:04:54.658733+0000","last_became_peered":"2024-02-19T21:04:54.658733+0000","last_unstale":"2024-02-19T21:06:13.198323+0000","last_undegraded":"2024-02-19T21:06:13.198323+0000","last_fullsized":"2024-02-19T21:06:13.198323+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":99,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":164,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.210064+0000","last_change":"2024-02-19T21:05:22.217801+0000","last_active":"2024-02-19T21:06:13.210064+0000","last_peered":"2024-02-19T21:06:13.210064+0000","last_clean":"2024-02-19T21:06:13.210064+0000","last_became_active":"2024-02-19T21:05:22.217505+0000","last_became_peered":"2024-02-19T21:05:22.217505+0000","last_unstale":"2024-02-19T21:06:13.210064+0000","last_undegraded":"2024-02-19T21:06:13.210064+0000","last_fullsized":"2024-02-19T21:06:13.210064+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:22.800578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:22.800578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.18","version":"194'74","reported_seq":161,"reported_epoch":194,"state":"active+clean+snaptrim","last_fresh":"2024-02-19T21:06:14.245385+0000","last_change":"2024-02-19T21:06:12.240714+0000","last_active":"2024-02-19T21:06:14.245385+0000","last_peered":"2024-02-19T21:06:14.245385+0000","last_clean":"2024-02-19T21:06:14.245385+0000","last_became_active":"2024-02-19T21:04:54.610608+0000","last_became_peered":"2024-02-19T21:04:54.610608+0000","last_unstale":"2024-02-19T21:06:14.245385+0000","last_undegraded":"2024-02-19T21:06:14.245385+0000","last_fullsized":"2024-02-19T21:06:14.245385+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":214,"num_objects":20,"num_object_clones":14,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"2.26","version":"0'0","reported_seq":62,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.208569+0000","last_change":"2024-02-19T21:05:22.217945+0000","last_active":"2024-02-19T21:06:13.208569+0000","last_peered":"2024-02-19T21:06:13.208569+0000","last_clean":"2024-02-19T21:06:13.208569+0000","last_became_active":"2024-02-19T21:05:22.216623+0000","last_became_peered":"2024-02-19T21:05:22.216623+0000","last_unstale":"2024-02-19T21:06:13.208569+0000","last_undegraded":"2024-02-19T21:06:13.208569+0000","last_fullsized":"2024-02-19T21:06:13.208569+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:21.205829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.205829+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.19","version":"184'44","reported_seq":140,"reported_epoch":193,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:13.200413+0000","last_change":"2024-02-19T21:06:06.227723+0000","last_active":"2024-02-19T21:06:13.200413+0000","last_peered":"2024-02-19T21:06:13.200413+0000","last_clean":"2024-02-19T21:06:13.200413+0000","last_became_active":"2024-02-19T21:04:54.608702+0000","last_became_peered":"2024-02-19T21:04:54.608702+0000","last_unstale":"2024-02-19T21:06:13.200413+0000","last_undegraded":"2024-02-19T21:06:13.200413+0000","last_fullsized":"2024-02-19T21:06:13.200413+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":164,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"}]},{"pgid":"2.21","version":"0'0","reported_seq":182,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.199700+0000","last_change":"2024-02-19T21:05:46.049986+0000","last_active":"2024-02-19T21:06:13.199700+0000","last_peered":"2024-02-19T21:06:13.199700+0000","last_clean":"2024-02-19T21:06:13.199700+0000","last_became_active":"2024-02-19T21:05:22.216738+0000","last_became_peered":"2024-02-19T21:05:22.216738+0000","last_unstale":"2024-02-19T21:06:13.199700+0000","last_undegraded":"2024-02-19T21:06:13.199700+0000","last_fullsized":"2024-02-19T21:06:13.199700+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:46.049878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:46.049878+0000","last_clean_scrub_stamp":"2024-02-19T21:05:46.049878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.1e","version":"192'28","reported_seq":126,"reported_epoch":193,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:14.232542+0000","last_change":"2024-02-19T21:06:14.232542+0000","last_active":"2024-02-19T21:06:14.232542+0000","last_peered":"2024-02-19T21:06:14.232542+0000","last_clean":"2024-02-19T21:06:14.232542+0000","last_became_active":"2024-02-19T21:04:54.629433+0000","last_became_peered":"2024-02-19T21:04:54.629433+0000","last_unstale":"2024-02-19T21:06:14.232542+0000","last_undegraded":"2024-02-19T21:06:14.232542+0000","last_fullsized":"2024-02-19T21:06:14.232542+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":101,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.20","version":"58'1","reported_seq":184,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.207616+0000","last_change":"2024-02-19T21:05:22.313264+0000","last_active":"2024-02-19T21:06:13.207616+0000","last_peered":"2024-02-19T21:06:13.207616+0000","last_clean":"2024-02-19T21:06:13.207616+0000","last_became_active":"2024-02-19T21:05:22.223725+0000","last_became_peered":"2024-02-19T21:05:22.223725+0000","last_unstale":"2024-02-19T21:06:13.207616+0000","last_undegraded":"2024-02-19T21:06:13.207616+0000","last_fullsized":"2024-02-19T21:06:13.207616+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:04:48.298487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:48.298487+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":437,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1f","version":"192'26","reported_seq":127,"reported_epoch":194,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:14.249483+0000","last_change":"2024-02-19T21:06:14.249522+0000","last_active":"2024-02-19T21:06:14.249483+0000","last_peered":"2024-02-19T21:06:14.249483+0000","last_clean":"2024-02-19T21:06:14.249483+0000","last_became_active":"2024-02-19T21:04:54.671969+0000","last_became_peered":"2024-02-19T21:04:54.671969+0000","last_unstale":"2024-02-19T21:06:14.249483+0000","last_undegraded":"2024-02-19T21:06:14.249483+0000","last_fullsized":"2024-02-19T21:06:14.249483+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"2"}]},{"pgid":"2.23","version":"0'0","reported_seq":64,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.198192+0000","last_change":"2024-02-19T21:06:10.790355+0000","last_active":"2024-02-19T21:06:13.198192+0000","last_peered":"2024-02-19T21:06:13.198192+0000","last_clean":"2024-02-19T21:06:13.198192+0000","last_became_active":"2024-02-19T21:05:22.218880+0000","last_became_peered":"2024-02-19T21:05:22.218880+0000","last_unstale":"2024-02-19T21:06:13.198192+0000","last_undegraded":"2024-02-19T21:06:13.198192+0000","last_fullsized":"2024-02-19T21:06:13.198192+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:14.215016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:14.215016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.1c","version":"193'88","reported_seq":175,"reported_epoch":194,"state":"active+clean+snaptrim","last_fresh":"2024-02-19T21:06:14.229870+0000","last_change":"2024-02-19T21:06:08.204037+0000","last_active":"2024-02-19T21:06:14.229870+0000","last_peered":"2024-02-19T21:06:14.229870+0000","last_clean":"2024-02-19T21:06:14.229870+0000","last_became_active":"2024-02-19T21:04:54.605898+0000","last_became_peered":"2024-02-19T21:04:54.605898+0000","last_unstale":"2024-02-19T21:06:14.229870+0000","last_undegraded":"2024-02-19T21:06:14.229870+0000","last_fullsized":"2024-02-19T21:06:14.229870+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":88,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"stat_sum":{"num_bytes":352,"num_objects":32,"num_object_clones":24,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":40,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":169,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.197260+0000","last_change":"2024-02-19T21:05:43.958644+0000","last_active":"2024-02-19T21:06:13.197260+0000","last_peered":"2024-02-19T21:06:13.197260+0000","last_clean":"2024-02-19T21:06:13.197260+0000","last_became_active":"2024-02-19T21:05:22.196948+0000","last_became_peered":"2024-02-19T21:05:22.196948+0000","last_unstale":"2024-02-19T21:06:13.197260+0000","last_undegraded":"2024-02-19T21:06:13.197260+0000","last_fullsized":"2024-02-19T21:06:13.197260+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:43.958548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:43.958548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1d","version":"186'22","reported_seq":119,"reported_epoch":193,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:13.207444+0000","last_change":"2024-02-19T21:06:08.245417+0000","last_active":"2024-02-19T21:06:13.207444+0000","last_peered":"2024-02-19T21:06:13.207444+0000","last_clean":"2024-02-19T21:06:13.207444+0000","last_became_active":"2024-02-19T21:04:54.655665+0000","last_became_peered":"2024-02-19T21:04:54.655665+0000","last_unstale":"2024-02-19T21:06:13.207444+0000","last_undegraded":"2024-02-19T21:06:13.207444+0000","last_fullsized":"2024-02-19T21:06:13.207444+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":88,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"8.4","version":"0'0","reported_seq":104,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204104+0000","last_change":"2024-02-19T21:04:39.640053+0000","last_active":"2024-02-19T21:06:13.204104+0000","last_peered":"2024-02-19T21:06:13.204104+0000","last_clean":"2024-02-19T21:06:13.204104+0000","last_became_active":"2024-02-19T21:04:39.639772+0000","last_became_peered":"2024-02-19T21:04:39.639772+0000","last_unstale":"2024-02-19T21:06:13.204104+0000","last_undegraded":"2024-02-19T21:06:13.204104+0000","last_fullsized":"2024-02-19T21:06:13.204104+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"97'9","reported_seq":219,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.242524+0000","last_change":"2024-02-19T21:04:24.046111+0000","last_active":"2024-02-19T21:06:14.242524+0000","last_peered":"2024-02-19T21:06:14.242524+0000","last_clean":"2024-02-19T21:06:14.242524+0000","last_became_active":"2024-02-19T21:01:20.288720+0000","last_became_peered":"2024-02-19T21:01:20.288720+0000","last_unstale":"2024-02-19T21:06:14.242524+0000","last_undegraded":"2024-02-19T21:06:14.242524+0000","last_fullsized":"2024-02-19T21:06:14.242524+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:04:24.046034+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":181,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.208193+0000","last_change":"2024-02-19T21:04:56.123170+0000","last_active":"2024-02-19T21:06:13.208193+0000","last_peered":"2024-02-19T21:06:13.208193+0000","last_clean":"2024-02-19T21:06:13.208193+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:06:13.208193+0000","last_undegraded":"2024-02-19T21:06:13.208193+0000","last_fullsized":"2024-02-19T21:06:13.208193+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:04:56.123096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.123096+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":155,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.200757+0000","last_change":"2024-02-19T21:04:11.889869+0000","last_active":"2024-02-19T21:06:13.200757+0000","last_peered":"2024-02-19T21:06:13.200757+0000","last_clean":"2024-02-19T21:06:13.200757+0000","last_became_active":"2024-02-19T21:01:16.271911+0000","last_became_peered":"2024-02-19T21:01:16.271911+0000","last_unstale":"2024-02-19T21:06:13.200757+0000","last_undegraded":"2024-02-19T21:06:13.200757+0000","last_fullsized":"2024-02-19T21:06:13.200757+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.889778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:37.258580+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.889778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.213789+0000","last_change":"2024-02-19T21:05:43.947943+0000","last_active":"2024-02-19T21:06:13.213789+0000","last_peered":"2024-02-19T21:06:13.213789+0000","last_clean":"2024-02-19T21:06:13.213789+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:06:13.213789+0000","last_undegraded":"2024-02-19T21:06:13.213789+0000","last_fullsized":"2024-02-19T21:06:13.213789+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:43.947829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:43.947829+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.202132+0000","last_change":"2024-02-19T21:04:39.703479+0000","last_active":"2024-02-19T21:06:13.202132+0000","last_peered":"2024-02-19T21:06:13.202132+0000","last_clean":"2024-02-19T21:06:13.202132+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:06:13.202132+0000","last_undegraded":"2024-02-19T21:06:13.202132+0000","last_fullsized":"2024-02-19T21:06:13.202132+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":113,"reported_epoch":194,"state":"clean+premerge+peered","last_fresh":"2024-02-19T21:06:14.193243+0000","last_change":"2024-02-19T21:06:14.193243+0000","last_active":"2024-02-19T21:06:13.201714+0000","last_peered":"2024-02-19T21:06:14.193243+0000","last_clean":"2024-02-19T21:06:14.193243+0000","last_became_active":"2024-02-19T21:04:39.771391+0000","last_became_peered":"2024-02-19T21:06:14.192428+0000","last_unstale":"2024-02-19T21:06:14.193243+0000","last_undegraded":"2024-02-19T21:06:14.193243+0000","last_fullsized":"2024-02-19T21:06:14.193243+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":169,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206906+0000","last_change":"2024-02-19T21:04:56.263016+0000","last_active":"2024-02-19T21:06:13.206906+0000","last_peered":"2024-02-19T21:06:13.206906+0000","last_clean":"2024-02-19T21:06:13.206906+0000","last_became_active":"2024-02-19T21:01:18.288322+0000","last_became_peered":"2024-02-19T21:01:18.288322+0000","last_unstale":"2024-02-19T21:06:13.206906+0000","last_undegraded":"2024-02-19T21:06:13.206906+0000","last_fullsized":"2024-02-19T21:06:13.206906+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:56.262927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.262927+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":169,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.208109+0000","last_change":"2024-02-19T21:05:10.764055+0000","last_active":"2024-02-19T21:06:13.208109+0000","last_peered":"2024-02-19T21:06:13.208109+0000","last_clean":"2024-02-19T21:06:13.208109+0000","last_became_active":"2024-02-19T21:04:12.926645+0000","last_became_peered":"2024-02-19T21:04:12.926645+0000","last_unstale":"2024-02-19T21:06:13.208109+0000","last_undegraded":"2024-02-19T21:06:13.208109+0000","last_fullsized":"2024-02-19T21:06:13.208109+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:10.763889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:10.763889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":151,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.201469+0000","last_change":"2024-02-19T21:04:17.831164+0000","last_active":"2024-02-19T21:06:13.201469+0000","last_peered":"2024-02-19T21:06:13.201469+0000","last_clean":"2024-02-19T21:06:13.201469+0000","last_became_active":"2024-02-19T21:01:20.290566+0000","last_became_peered":"2024-02-19T21:01:20.290566+0000","last_unstale":"2024-02-19T21:06:13.201469+0000","last_undegraded":"2024-02-19T21:06:13.201469+0000","last_fullsized":"2024-02-19T21:06:13.201469+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:17.831065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:17.831065+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":154,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.207972+0000","last_change":"2024-02-19T21:06:05.822750+0000","last_active":"2024-02-19T21:06:13.207972+0000","last_peered":"2024-02-19T21:06:13.207972+0000","last_clean":"2024-02-19T21:06:13.207972+0000","last_became_active":"2024-02-19T21:01:22.297998+0000","last_became_peered":"2024-02-19T21:01:22.297998+0000","last_unstale":"2024-02-19T21:06:13.207972+0000","last_undegraded":"2024-02-19T21:06:13.207972+0000","last_fullsized":"2024-02-19T21:06:13.207972+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:05.822641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:05.822641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.215836+0000","last_change":"2024-02-19T21:04:39.765776+0000","last_active":"2024-02-19T21:06:13.215836+0000","last_peered":"2024-02-19T21:06:13.215836+0000","last_clean":"2024-02-19T21:06:13.215836+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:06:13.215836+0000","last_undegraded":"2024-02-19T21:06:13.215836+0000","last_fullsized":"2024-02-19T21:06:13.215836+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"168'11","reported_seq":115,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206123+0000","last_change":"2024-02-19T21:04:39.731654+0000","last_active":"2024-02-19T21:06:13.206123+0000","last_peered":"2024-02-19T21:06:13.206123+0000","last_clean":"2024-02-19T21:06:13.206123+0000","last_became_active":"2024-02-19T21:04:39.731439+0000","last_became_peered":"2024-02-19T21:04:39.731439+0000","last_unstale":"2024-02-19T21:06:13.206123+0000","last_undegraded":"2024-02-19T21:06:13.206123+0000","last_fullsized":"2024-02-19T21:06:13.206123+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":189,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.199741+0000","last_change":"2024-02-19T21:06:06.909964+0000","last_active":"2024-02-19T21:06:13.199741+0000","last_peered":"2024-02-19T21:06:13.199741+0000","last_clean":"2024-02-19T21:06:13.199741+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:06:13.199741+0000","last_undegraded":"2024-02-19T21:06:13.199741+0000","last_fullsized":"2024-02-19T21:06:13.199741+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:06:06.909884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:06.909884+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":160,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.201930+0000","last_change":"2024-02-19T21:05:49.051796+0000","last_active":"2024-02-19T21:06:13.201930+0000","last_peered":"2024-02-19T21:06:13.201930+0000","last_clean":"2024-02-19T21:06:13.201930+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:06:13.201930+0000","last_undegraded":"2024-02-19T21:06:13.201930+0000","last_fullsized":"2024-02-19T21:06:13.201930+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:05:49.051732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":151,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.200545+0000","last_change":"2024-02-19T21:04:51.822914+0000","last_active":"2024-02-19T21:06:13.200545+0000","last_peered":"2024-02-19T21:06:13.200545+0000","last_clean":"2024-02-19T21:06:13.200545+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:06:13.200545+0000","last_undegraded":"2024-02-19T21:06:13.200545+0000","last_fullsized":"2024-02-19T21:06:13.200545+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:51.822845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:51.822845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":148,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.207941+0000","last_change":"2024-02-19T21:05:05.187540+0000","last_active":"2024-02-19T21:06:13.207941+0000","last_peered":"2024-02-19T21:06:13.207941+0000","last_clean":"2024-02-19T21:06:13.207941+0000","last_became_active":"2024-02-19T21:01:22.291759+0000","last_became_peered":"2024-02-19T21:01:22.291759+0000","last_unstale":"2024-02-19T21:06:13.207941+0000","last_undegraded":"2024-02-19T21:06:13.207941+0000","last_fullsized":"2024-02-19T21:06:13.207941+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:05.187474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:05.187474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.213530+0000","last_change":"2024-02-19T21:04:39.620073+0000","last_active":"2024-02-19T21:06:13.213530+0000","last_peered":"2024-02-19T21:06:13.213530+0000","last_clean":"2024-02-19T21:06:13.213530+0000","last_became_active":"2024-02-19T21:04:39.618614+0000","last_became_peered":"2024-02-19T21:04:39.618614+0000","last_unstale":"2024-02-19T21:06:13.213530+0000","last_undegraded":"2024-02-19T21:06:13.213530+0000","last_fullsized":"2024-02-19T21:06:13.213530+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":213,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.202881+0000","last_change":"2024-02-19T21:06:06.630925+0000","last_active":"2024-02-19T21:06:13.202881+0000","last_peered":"2024-02-19T21:06:13.202881+0000","last_clean":"2024-02-19T21:06:13.202881+0000","last_became_active":"2024-02-19T21:01:30.923214+0000","last_became_peered":"2024-02-19T21:01:30.923214+0000","last_unstale":"2024-02-19T21:06:13.202881+0000","last_undegraded":"2024-02-19T21:06:13.202881+0000","last_fullsized":"2024-02-19T21:06:13.202881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:06:06.630828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:06.630828+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":160,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.215077+0000","last_change":"2024-02-19T21:04:21.952701+0000","last_active":"2024-02-19T21:06:13.215077+0000","last_peered":"2024-02-19T21:06:13.215077+0000","last_clean":"2024-02-19T21:06:13.215077+0000","last_became_active":"2024-02-19T21:01:16.274937+0000","last_became_peered":"2024-02-19T21:01:16.274937+0000","last_unstale":"2024-02-19T21:06:13.215077+0000","last_undegraded":"2024-02-19T21:06:13.215077+0000","last_fullsized":"2024-02-19T21:06:13.215077+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:04:21.952586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.952586+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":156,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204167+0000","last_change":"2024-02-19T21:05:39.947596+0000","last_active":"2024-02-19T21:06:13.204167+0000","last_peered":"2024-02-19T21:06:13.204167+0000","last_clean":"2024-02-19T21:06:13.204167+0000","last_became_active":"2024-02-19T21:01:20.290687+0000","last_became_peered":"2024-02-19T21:01:20.290687+0000","last_unstale":"2024-02-19T21:06:13.204167+0000","last_undegraded":"2024-02-19T21:06:13.204167+0000","last_fullsized":"2024-02-19T21:06:13.204167+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:39.947521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:39.947521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":154,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.200247+0000","last_change":"2024-02-19T21:06:13.136600+0000","last_active":"2024-02-19T21:06:13.200247+0000","last_peered":"2024-02-19T21:06:13.200247+0000","last_clean":"2024-02-19T21:06:13.200247+0000","last_became_active":"2024-02-19T21:01:22.298941+0000","last_became_peered":"2024-02-19T21:01:22.298941+0000","last_unstale":"2024-02-19T21:06:13.200247+0000","last_undegraded":"2024-02-19T21:06:13.200247+0000","last_fullsized":"2024-02-19T21:06:13.200247+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:13.136532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:06:13.136532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"168'7","reported_seq":111,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.203257+0000","last_change":"2024-02-19T21:04:39.781921+0000","last_active":"2024-02-19T21:06:13.203257+0000","last_peered":"2024-02-19T21:06:13.203257+0000","last_clean":"2024-02-19T21:06:13.203257+0000","last_became_active":"2024-02-19T21:04:39.778460+0000","last_became_peered":"2024-02-19T21:04:39.778460+0000","last_unstale":"2024-02-19T21:06:13.203257+0000","last_undegraded":"2024-02-19T21:06:13.203257+0000","last_fullsized":"2024-02-19T21:06:13.203257+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.205045+0000","last_change":"2024-02-19T21:04:39.699479+0000","last_active":"2024-02-19T21:06:13.205045+0000","last_peered":"2024-02-19T21:06:13.205045+0000","last_clean":"2024-02-19T21:06:13.205045+0000","last_became_active":"2024-02-19T21:04:39.698684+0000","last_became_peered":"2024-02-19T21:04:39.698684+0000","last_unstale":"2024-02-19T21:06:13.205045+0000","last_undegraded":"2024-02-19T21:06:13.205045+0000","last_fullsized":"2024-02-19T21:06:13.205045+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"82'15","reported_seq":173,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.207267+0000","last_change":"2024-02-19T21:04:49.285676+0000","last_active":"2024-02-19T21:06:13.207267+0000","last_peered":"2024-02-19T21:06:13.207267+0000","last_clean":"2024-02-19T21:06:13.207267+0000","last_became_active":"2024-02-19T21:01:18.282805+0000","last_became_peered":"2024-02-19T21:01:18.282805+0000","last_unstale":"2024-02-19T21:06:13.207267+0000","last_undegraded":"2024-02-19T21:06:13.207267+0000","last_fullsized":"2024-02-19T21:06:13.207267+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:04:49.285599+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.285599+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":160,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.688019+0000","last_change":"2024-02-19T21:06:14.688019+0000","last_active":"2024-02-19T21:06:14.688019+0000","last_peered":"2024-02-19T21:06:14.688019+0000","last_clean":"2024-02-19T21:06:14.688019+0000","last_became_active":"2024-02-19T21:01:16.274086+0000","last_became_peered":"2024-02-19T21:01:16.274086+0000","last_unstale":"2024-02-19T21:06:14.688019+0000","last_undegraded":"2024-02-19T21:06:14.688019+0000","last_fullsized":"2024-02-19T21:06:14.688019+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:14.687938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.687938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":224,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.236170+0000","last_change":"2024-02-19T21:06:12.177597+0000","last_active":"2024-02-19T21:06:14.236170+0000","last_peered":"2024-02-19T21:06:14.236170+0000","last_clean":"2024-02-19T21:06:14.236170+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:06:14.236170+0000","last_undegraded":"2024-02-19T21:06:14.236170+0000","last_fullsized":"2024-02-19T21:06:14.236170+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:12.177524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:12.177524+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204712+0000","last_change":"2024-02-19T21:04:21.031646+0000","last_active":"2024-02-19T21:06:13.204712+0000","last_peered":"2024-02-19T21:06:13.204712+0000","last_clean":"2024-02-19T21:06:13.204712+0000","last_became_active":"2024-02-19T21:01:22.299993+0000","last_became_peered":"2024-02-19T21:01:22.299993+0000","last_unstale":"2024-02-19T21:06:13.204712+0000","last_undegraded":"2024-02-19T21:06:13.204712+0000","last_fullsized":"2024-02-19T21:06:13.204712+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:04:21.031577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"168'7","reported_seq":111,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.196939+0000","last_change":"2024-02-19T21:04:39.746708+0000","last_active":"2024-02-19T21:06:13.196939+0000","last_peered":"2024-02-19T21:06:13.196939+0000","last_clean":"2024-02-19T21:06:13.196939+0000","last_became_active":"2024-02-19T21:04:39.742728+0000","last_became_peered":"2024-02-19T21:04:39.742728+0000","last_unstale":"2024-02-19T21:06:13.196939+0000","last_undegraded":"2024-02-19T21:06:13.196939+0000","last_fullsized":"2024-02-19T21:06:13.196939+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206680+0000","last_change":"2024-02-19T21:04:39.766043+0000","last_active":"2024-02-19T21:06:13.206680+0000","last_peered":"2024-02-19T21:06:13.206680+0000","last_clean":"2024-02-19T21:06:13.206680+0000","last_became_active":"2024-02-19T21:04:39.765782+0000","last_became_peered":"2024-02-19T21:04:39.765782+0000","last_unstale":"2024-02-19T21:06:13.206680+0000","last_undegraded":"2024-02-19T21:06:13.206680+0000","last_fullsized":"2024-02-19T21:06:13.206680+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"168'7","reported_seq":111,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.203876+0000","last_change":"2024-02-19T21:04:39.676464+0000","last_active":"2024-02-19T21:06:13.203876+0000","last_peered":"2024-02-19T21:06:13.203876+0000","last_clean":"2024-02-19T21:06:13.203876+0000","last_became_active":"2024-02-19T21:04:39.675580+0000","last_became_peered":"2024-02-19T21:04:39.675580+0000","last_unstale":"2024-02-19T21:06:13.203876+0000","last_undegraded":"2024-02-19T21:06:13.203876+0000","last_fullsized":"2024-02-19T21:06:13.203876+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":178,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206257+0000","last_change":"2024-02-19T21:05:03.210019+0000","last_active":"2024-02-19T21:06:13.206257+0000","last_peered":"2024-02-19T21:06:13.206257+0000","last_clean":"2024-02-19T21:06:13.206257+0000","last_became_active":"2024-02-19T21:01:18.285534+0000","last_became_peered":"2024-02-19T21:01:18.285534+0000","last_unstale":"2024-02-19T21:06:13.206257+0000","last_undegraded":"2024-02-19T21:06:13.206257+0000","last_fullsized":"2024-02-19T21:06:13.206257+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:05:03.209876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.209876+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":169,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.203333+0000","last_change":"2024-02-19T21:06:07.874422+0000","last_active":"2024-02-19T21:06:13.203333+0000","last_peered":"2024-02-19T21:06:13.203333+0000","last_clean":"2024-02-19T21:06:13.203333+0000","last_became_active":"2024-02-19T21:05:20.188821+0000","last_became_peered":"2024-02-19T21:05:20.188821+0000","last_unstale":"2024-02-19T21:06:13.203333+0000","last_undegraded":"2024-02-19T21:06:13.203333+0000","last_fullsized":"2024-02-19T21:06:13.203333+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:07.874362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:07.874362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":219,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.180080+0000","last_change":"2024-02-19T21:04:16.059715+0000","last_active":"2024-02-19T21:06:14.180080+0000","last_peered":"2024-02-19T21:06:14.180080+0000","last_clean":"2024-02-19T21:06:14.180080+0000","last_became_active":"2024-02-19T21:01:20.293341+0000","last_became_peered":"2024-02-19T21:01:20.293341+0000","last_unstale":"2024-02-19T21:06:14.180080+0000","last_undegraded":"2024-02-19T21:06:14.180080+0000","last_fullsized":"2024-02-19T21:06:14.180080+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:16.059648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.059648+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.202385+0000","last_change":"2024-02-19T21:04:08.185818+0000","last_active":"2024-02-19T21:06:13.202385+0000","last_peered":"2024-02-19T21:06:13.202385+0000","last_clean":"2024-02-19T21:06:13.202385+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:06:13.202385+0000","last_undegraded":"2024-02-19T21:06:13.202385+0000","last_fullsized":"2024-02-19T21:06:13.202385+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.185717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.185717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204812+0000","last_change":"2024-02-19T21:04:39.657326+0000","last_active":"2024-02-19T21:06:13.204812+0000","last_peered":"2024-02-19T21:06:13.204812+0000","last_clean":"2024-02-19T21:06:13.204812+0000","last_became_active":"2024-02-19T21:04:39.657052+0000","last_became_peered":"2024-02-19T21:04:39.657052+0000","last_unstale":"2024-02-19T21:06:13.204812+0000","last_undegraded":"2024-02-19T21:06:13.204812+0000","last_fullsized":"2024-02-19T21:06:13.204812+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":104,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.205870+0000","last_change":"2024-02-19T21:04:39.662708+0000","last_active":"2024-02-19T21:06:13.205870+0000","last_peered":"2024-02-19T21:06:13.205870+0000","last_clean":"2024-02-19T21:06:13.205870+0000","last_became_active":"2024-02-19T21:04:39.662412+0000","last_became_peered":"2024-02-19T21:04:39.662412+0000","last_unstale":"2024-02-19T21:06:13.205870+0000","last_undegraded":"2024-02-19T21:06:13.205870+0000","last_fullsized":"2024-02-19T21:06:13.205870+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":177,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204357+0000","last_change":"2024-02-19T21:06:02.595348+0000","last_active":"2024-02-19T21:06:13.204357+0000","last_peered":"2024-02-19T21:06:13.204357+0000","last_clean":"2024-02-19T21:06:13.204357+0000","last_became_active":"2024-02-19T21:01:18.285670+0000","last_became_peered":"2024-02-19T21:01:18.285670+0000","last_unstale":"2024-02-19T21:06:13.204357+0000","last_undegraded":"2024-02-19T21:06:13.204357+0000","last_fullsized":"2024-02-19T21:06:13.204357+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:06:02.595266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:02.595266+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":168,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.210006+0000","last_change":"2024-02-19T21:06:07.411725+0000","last_active":"2024-02-19T21:06:13.210006+0000","last_peered":"2024-02-19T21:06:13.210006+0000","last_clean":"2024-02-19T21:06:13.210006+0000","last_became_active":"2024-02-19T21:05:20.211711+0000","last_became_peered":"2024-02-19T21:05:20.211711+0000","last_unstale":"2024-02-19T21:06:13.210006+0000","last_undegraded":"2024-02-19T21:06:13.210006+0000","last_fullsized":"2024-02-19T21:06:13.210006+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:07.411662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:07.411662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":151,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.216749+0000","last_change":"2024-02-19T21:04:16.217810+0000","last_active":"2024-02-19T21:06:13.216749+0000","last_peered":"2024-02-19T21:06:13.216749+0000","last_clean":"2024-02-19T21:06:13.216749+0000","last_became_active":"2024-02-19T21:01:20.289375+0000","last_became_peered":"2024-02-19T21:01:20.289375+0000","last_unstale":"2024-02-19T21:06:13.216749+0000","last_undegraded":"2024-02-19T21:06:13.216749+0000","last_fullsized":"2024-02-19T21:06:13.216749+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:16.217744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.217744+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.198263+0000","last_change":"2024-02-19T21:06:05.721241+0000","last_active":"2024-02-19T21:06:13.198263+0000","last_peered":"2024-02-19T21:06:13.198263+0000","last_clean":"2024-02-19T21:06:13.198263+0000","last_became_active":"2024-02-19T21:01:22.607007+0000","last_became_peered":"2024-02-19T21:01:22.607007+0000","last_unstale":"2024-02-19T21:06:13.198263+0000","last_undegraded":"2024-02-19T21:06:13.198263+0000","last_fullsized":"2024-02-19T21:06:13.198263+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:05.721175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:05.721175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.197640+0000","last_change":"2024-02-19T21:04:39.744505+0000","last_active":"2024-02-19T21:06:13.197640+0000","last_peered":"2024-02-19T21:06:13.197640+0000","last_clean":"2024-02-19T21:06:13.197640+0000","last_became_active":"2024-02-19T21:04:39.744231+0000","last_became_peered":"2024-02-19T21:04:39.744231+0000","last_unstale":"2024-02-19T21:06:13.197640+0000","last_undegraded":"2024-02-19T21:06:13.197640+0000","last_fullsized":"2024-02-19T21:06:13.197640+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"168'9","reported_seq":76,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.812545+0000","last_change":"2024-02-19T21:06:14.812545+0000","last_active":"2024-02-19T21:06:14.812545+0000","last_peered":"2024-02-19T21:06:14.812545+0000","last_clean":"2024-02-19T21:06:14.812545+0000","last_became_active":"2024-02-19T21:05:52.221957+0000","last_became_peered":"2024-02-19T21:05:52.221957+0000","last_unstale":"2024-02-19T21:06:14.812545+0000","last_undegraded":"2024-02-19T21:06:14.812545+0000","last_fullsized":"2024-02-19T21:06:14.812545+0000","mapping_epoch":157,"log_start":"168'9","ondisk_log_start":"168'9","created":98,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'9","last_scrub_stamp":"2024-02-19T21:06:14.812471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.812471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":224,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.187746+0000","last_change":"2024-02-19T21:06:07.761009+0000","last_active":"2024-02-19T21:06:14.187746+0000","last_peered":"2024-02-19T21:06:14.187746+0000","last_clean":"2024-02-19T21:06:14.187746+0000","last_became_active":"2024-02-19T21:01:20.286505+0000","last_became_peered":"2024-02-19T21:01:20.286505+0000","last_unstale":"2024-02-19T21:06:14.187746+0000","last_undegraded":"2024-02-19T21:06:14.187746+0000","last_fullsized":"2024-02-19T21:06:14.187746+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:07.760934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:07.760934+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":174,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.205374+0000","last_change":"2024-02-19T21:04:50.266262+0000","last_active":"2024-02-19T21:06:13.205374+0000","last_peered":"2024-02-19T21:06:13.205374+0000","last_clean":"2024-02-19T21:06:13.205374+0000","last_became_active":"2024-02-19T21:01:18.282702+0000","last_became_peered":"2024-02-19T21:01:18.282702+0000","last_unstale":"2024-02-19T21:06:13.205374+0000","last_undegraded":"2024-02-19T21:06:13.205374+0000","last_fullsized":"2024-02-19T21:06:13.205374+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:04:50.266152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:50.266152+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":169,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.201248+0000","last_change":"2024-02-19T21:05:35.951650+0000","last_active":"2024-02-19T21:06:13.201248+0000","last_peered":"2024-02-19T21:06:13.201248+0000","last_clean":"2024-02-19T21:06:13.201248+0000","last_became_active":"2024-02-19T21:05:20.224706+0000","last_became_peered":"2024-02-19T21:05:20.224706+0000","last_unstale":"2024-02-19T21:06:13.201248+0000","last_undegraded":"2024-02-19T21:06:13.201248+0000","last_fullsized":"2024-02-19T21:06:13.201248+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.951592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.951592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":150,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.215408+0000","last_change":"2024-02-19T21:05:03.248574+0000","last_active":"2024-02-19T21:06:13.215408+0000","last_peered":"2024-02-19T21:06:13.215408+0000","last_clean":"2024-02-19T21:06:13.215408+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:06:13.215408+0000","last_undegraded":"2024-02-19T21:06:13.215408+0000","last_fullsized":"2024-02-19T21:06:13.215408+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.248506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.207096+0000","last_change":"2024-02-19T21:04:39.659690+0000","last_active":"2024-02-19T21:06:13.207096+0000","last_peered":"2024-02-19T21:06:13.207096+0000","last_clean":"2024-02-19T21:06:13.207096+0000","last_became_active":"2024-02-19T21:04:39.658536+0000","last_became_peered":"2024-02-19T21:04:39.658536+0000","last_unstale":"2024-02-19T21:06:13.207096+0000","last_undegraded":"2024-02-19T21:06:13.207096+0000","last_fullsized":"2024-02-19T21:06:13.207096+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":64,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.203107+0000","last_change":"2024-02-19T21:05:56.204628+0000","last_active":"2024-02-19T21:06:13.203107+0000","last_peered":"2024-02-19T21:06:13.203107+0000","last_clean":"2024-02-19T21:06:13.203107+0000","last_became_active":"2024-02-19T21:05:56.204312+0000","last_became_peered":"2024-02-19T21:05:56.204312+0000","last_unstale":"2024-02-19T21:06:13.203107+0000","last_undegraded":"2024-02-19T21:06:13.203107+0000","last_fullsized":"2024-02-19T21:06:13.203107+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":151,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.205532+0000","last_change":"2024-02-19T21:05:12.318453+0000","last_active":"2024-02-19T21:06:13.205532+0000","last_peered":"2024-02-19T21:06:13.205532+0000","last_clean":"2024-02-19T21:06:13.205532+0000","last_became_active":"2024-02-19T21:01:20.291079+0000","last_became_peered":"2024-02-19T21:01:20.291079+0000","last_unstale":"2024-02-19T21:06:13.205532+0000","last_undegraded":"2024-02-19T21:06:13.205532+0000","last_fullsized":"2024-02-19T21:06:13.205532+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:05:12.318389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":185,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.197061+0000","last_change":"2024-02-19T21:04:08.837761+0000","last_active":"2024-02-19T21:06:13.197061+0000","last_peered":"2024-02-19T21:06:13.197061+0000","last_clean":"2024-02-19T21:06:13.197061+0000","last_became_active":"2024-02-19T21:01:18.293812+0000","last_became_peered":"2024-02-19T21:01:18.293812+0000","last_unstale":"2024-02-19T21:06:13.197061+0000","last_undegraded":"2024-02-19T21:06:13.197061+0000","last_fullsized":"2024-02-19T21:06:13.197061+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:04:08.837652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.837652+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":169,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.207004+0000","last_change":"2024-02-19T21:05:44.876482+0000","last_active":"2024-02-19T21:06:13.207004+0000","last_peered":"2024-02-19T21:06:13.207004+0000","last_clean":"2024-02-19T21:06:13.207004+0000","last_became_active":"2024-02-19T21:05:20.189051+0000","last_became_peered":"2024-02-19T21:05:20.189051+0000","last_unstale":"2024-02-19T21:06:13.207004+0000","last_undegraded":"2024-02-19T21:06:13.207004+0000","last_fullsized":"2024-02-19T21:06:13.207004+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206359+0000","last_change":"2024-02-19T21:05:47.538988+0000","last_active":"2024-02-19T21:06:13.206359+0000","last_peered":"2024-02-19T21:06:13.206359+0000","last_clean":"2024-02-19T21:06:13.206359+0000","last_became_active":"2024-02-19T21:01:22.297707+0000","last_became_peered":"2024-02-19T21:01:22.297707+0000","last_unstale":"2024-02-19T21:06:13.206359+0000","last_undegraded":"2024-02-19T21:06:13.206359+0000","last_fullsized":"2024-02-19T21:06:13.206359+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:05:47.538920+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206364+0000","last_change":"2024-02-19T21:04:39.770138+0000","last_active":"2024-02-19T21:06:13.206364+0000","last_peered":"2024-02-19T21:06:13.206364+0000","last_clean":"2024-02-19T21:06:13.206364+0000","last_became_active":"2024-02-19T21:04:39.767723+0000","last_became_peered":"2024-02-19T21:04:39.767723+0000","last_unstale":"2024-02-19T21:06:13.206364+0000","last_undegraded":"2024-02-19T21:06:13.206364+0000","last_fullsized":"2024-02-19T21:06:13.206364+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"168'7","reported_seq":69,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206573+0000","last_change":"2024-02-19T21:05:44.201533+0000","last_active":"2024-02-19T21:06:13.206573+0000","last_peered":"2024-02-19T21:06:13.206573+0000","last_clean":"2024-02-19T21:06:13.206573+0000","last_became_active":"2024-02-19T21:05:44.201176+0000","last_became_peered":"2024-02-19T21:05:44.201176+0000","last_unstale":"2024-02-19T21:06:13.206573+0000","last_undegraded":"2024-02-19T21:06:13.206573+0000","last_fullsized":"2024-02-19T21:06:13.206573+0000","mapping_epoch":157,"log_start":"100'5","ondisk_log_start":"100'5","created":98,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":156,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.199902+0000","last_change":"2024-02-19T21:04:56.720699+0000","last_active":"2024-02-19T21:06:13.199902+0000","last_peered":"2024-02-19T21:06:13.199902+0000","last_clean":"2024-02-19T21:06:13.199902+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:06:13.199902+0000","last_undegraded":"2024-02-19T21:06:13.199902+0000","last_fullsized":"2024-02-19T21:06:13.199902+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:56.720607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.720607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":194,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206736+0000","last_change":"2024-02-19T21:06:10.826958+0000","last_active":"2024-02-19T21:06:13.206736+0000","last_peered":"2024-02-19T21:06:13.206736+0000","last_clean":"2024-02-19T21:06:13.206736+0000","last_became_active":"2024-02-19T21:01:18.293791+0000","last_became_peered":"2024-02-19T21:01:18.293791+0000","last_unstale":"2024-02-19T21:06:13.206736+0000","last_undegraded":"2024-02-19T21:06:13.206736+0000","last_fullsized":"2024-02-19T21:06:13.206736+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:06:10.826889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:10.826889+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":170,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.208894+0000","last_change":"2024-02-19T21:06:06.458624+0000","last_active":"2024-02-19T21:06:13.208894+0000","last_peered":"2024-02-19T21:06:13.208894+0000","last_clean":"2024-02-19T21:06:13.208894+0000","last_became_active":"2024-02-19T21:05:20.211241+0000","last_became_peered":"2024-02-19T21:05:20.211241+0000","last_unstale":"2024-02-19T21:06:13.208894+0000","last_undegraded":"2024-02-19T21:06:13.208894+0000","last_fullsized":"2024-02-19T21:06:13.208894+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:06.458541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:06.458541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.203410+0000","last_change":"2024-02-19T21:05:01.859040+0000","last_active":"2024-02-19T21:06:13.203410+0000","last_peered":"2024-02-19T21:06:13.203410+0000","last_clean":"2024-02-19T21:06:13.203410+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:06:13.203410+0000","last_undegraded":"2024-02-19T21:06:13.203410+0000","last_fullsized":"2024-02-19T21:06:13.203410+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:01.858924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.858924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.200469+0000","last_change":"2024-02-19T21:04:39.719113+0000","last_active":"2024-02-19T21:06:13.200469+0000","last_peered":"2024-02-19T21:06:13.200469+0000","last_clean":"2024-02-19T21:06:13.200469+0000","last_became_active":"2024-02-19T21:04:39.718926+0000","last_became_peered":"2024-02-19T21:04:39.718926+0000","last_unstale":"2024-02-19T21:06:13.200469+0000","last_undegraded":"2024-02-19T21:06:13.200469+0000","last_fullsized":"2024-02-19T21:06:13.200469+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.a","version":"168'14","reported_seq":136,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.205285+0000","last_change":"2024-02-19T21:06:04.207771+0000","last_active":"2024-02-19T21:06:13.205285+0000","last_peered":"2024-02-19T21:06:13.205285+0000","last_clean":"2024-02-19T21:06:13.205285+0000","last_became_active":"2024-02-19T21:06:04.207446+0000","last_became_peered":"2024-02-19T21:06:04.207446+0000","last_unstale":"2024-02-19T21:06:13.205285+0000","last_undegraded":"2024-02-19T21:06:13.205285+0000","last_fullsized":"2024-02-19T21:06:13.205285+0000","mapping_epoch":98,"log_start":"168'14","ondisk_log_start":"168'14","created":98,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":384,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":151,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.216722+0000","last_change":"2024-02-19T21:04:57.188742+0000","last_active":"2024-02-19T21:06:13.216722+0000","last_peered":"2024-02-19T21:06:13.216722+0000","last_clean":"2024-02-19T21:06:13.216722+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:06:13.216722+0000","last_undegraded":"2024-02-19T21:06:13.216722+0000","last_fullsized":"2024-02-19T21:06:13.216722+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:57.188664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:57.188664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":176,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206221+0000","last_change":"2024-02-19T21:05:09.646541+0000","last_active":"2024-02-19T21:06:13.206221+0000","last_peered":"2024-02-19T21:06:13.206221+0000","last_clean":"2024-02-19T21:06:13.206221+0000","last_became_active":"2024-02-19T21:01:18.287139+0000","last_became_peered":"2024-02-19T21:01:18.287139+0000","last_unstale":"2024-02-19T21:06:13.206221+0000","last_undegraded":"2024-02-19T21:06:13.206221+0000","last_fullsized":"2024-02-19T21:06:13.206221+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:05:09.646445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:09.646445+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":175,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.208788+0000","last_change":"2024-02-19T21:06:09.422099+0000","last_active":"2024-02-19T21:06:13.208788+0000","last_peered":"2024-02-19T21:06:13.208788+0000","last_clean":"2024-02-19T21:06:13.208788+0000","last_became_active":"2024-02-19T21:05:20.211701+0000","last_became_peered":"2024-02-19T21:05:20.211701+0000","last_unstale":"2024-02-19T21:06:13.208788+0000","last_undegraded":"2024-02-19T21:06:13.208788+0000","last_fullsized":"2024-02-19T21:06:13.208788+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:06:09.422007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.422007+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":155,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.207660+0000","last_change":"2024-02-19T21:05:07.661094+0000","last_active":"2024-02-19T21:06:13.207660+0000","last_peered":"2024-02-19T21:06:13.207660+0000","last_clean":"2024-02-19T21:06:13.207660+0000","last_became_active":"2024-02-19T21:01:16.274398+0000","last_became_peered":"2024-02-19T21:01:16.274398+0000","last_unstale":"2024-02-19T21:06:13.207660+0000","last_undegraded":"2024-02-19T21:06:13.207660+0000","last_fullsized":"2024-02-19T21:06:13.207660+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:07.660435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:05:07.660435+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204178+0000","last_change":"2024-02-19T21:04:22.168700+0000","last_active":"2024-02-19T21:06:13.204178+0000","last_peered":"2024-02-19T21:06:13.204178+0000","last_clean":"2024-02-19T21:06:13.204178+0000","last_became_active":"2024-02-19T21:01:22.298477+0000","last_became_peered":"2024-02-19T21:01:22.298477+0000","last_unstale":"2024-02-19T21:06:13.204178+0000","last_undegraded":"2024-02-19T21:06:13.204178+0000","last_fullsized":"2024-02-19T21:06:13.204178+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:22.168596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:22.168596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.197675+0000","last_change":"2024-02-19T21:04:39.665830+0000","last_active":"2024-02-19T21:06:13.197675+0000","last_peered":"2024-02-19T21:06:13.197675+0000","last_clean":"2024-02-19T21:06:13.197675+0000","last_became_active":"2024-02-19T21:04:39.662125+0000","last_became_peered":"2024-02-19T21:04:39.662125+0000","last_unstale":"2024-02-19T21:06:13.197675+0000","last_undegraded":"2024-02-19T21:06:13.197675+0000","last_fullsized":"2024-02-19T21:06:13.197675+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":122,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.197559+0000","last_change":"2024-02-19T21:06:08.207026+0000","last_active":"2024-02-19T21:06:13.197559+0000","last_peered":"2024-02-19T21:06:13.197559+0000","last_clean":"2024-02-19T21:06:13.197559+0000","last_became_active":"2024-02-19T21:06:08.206725+0000","last_became_peered":"2024-02-19T21:06:08.206725+0000","last_unstale":"2024-02-19T21:06:13.197559+0000","last_undegraded":"2024-02-19T21:06:13.197559+0000","last_fullsized":"2024-02-19T21:06:13.197559+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":150,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.211339+0000","last_change":"2024-02-19T21:04:19.858793+0000","last_active":"2024-02-19T21:06:13.211339+0000","last_peered":"2024-02-19T21:06:13.211339+0000","last_clean":"2024-02-19T21:06:13.211339+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:06:13.211339+0000","last_undegraded":"2024-02-19T21:06:13.211339+0000","last_fullsized":"2024-02-19T21:06:13.211339+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:19.858688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.858688+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":175,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.199245+0000","last_change":"2024-02-19T21:04:19.885862+0000","last_active":"2024-02-19T21:06:13.199245+0000","last_peered":"2024-02-19T21:06:13.199245+0000","last_clean":"2024-02-19T21:06:13.199245+0000","last_became_active":"2024-02-19T21:01:18.293784+0000","last_became_peered":"2024-02-19T21:01:18.293784+0000","last_unstale":"2024-02-19T21:06:13.199245+0000","last_undegraded":"2024-02-19T21:06:13.199245+0000","last_fullsized":"2024-02-19T21:06:13.199245+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:04:19.885797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.885797+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":163,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.209993+0000","last_change":"2024-02-19T21:05:00.284714+0000","last_active":"2024-02-19T21:06:13.209993+0000","last_peered":"2024-02-19T21:06:13.209993+0000","last_clean":"2024-02-19T21:06:13.209993+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:06:13.209993+0000","last_undegraded":"2024-02-19T21:06:13.209993+0000","last_fullsized":"2024-02-19T21:06:13.209993+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:00.284626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:05:00.284626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":164,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.205142+0000","last_change":"2024-02-19T21:05:20.196022+0000","last_active":"2024-02-19T21:06:13.205142+0000","last_peered":"2024-02-19T21:06:13.205142+0000","last_clean":"2024-02-19T21:06:13.205142+0000","last_became_active":"2024-02-19T21:05:20.195763+0000","last_became_peered":"2024-02-19T21:05:20.195763+0000","last_unstale":"2024-02-19T21:06:13.205142+0000","last_undegraded":"2024-02-19T21:06:13.205142+0000","last_fullsized":"2024-02-19T21:06:13.205142+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:14.215016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:14.215016+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":155,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.197789+0000","last_change":"2024-02-19T21:04:54.972925+0000","last_active":"2024-02-19T21:06:13.197789+0000","last_peered":"2024-02-19T21:06:13.197789+0000","last_clean":"2024-02-19T21:06:13.197789+0000","last_became_active":"2024-02-19T21:01:16.274290+0000","last_became_peered":"2024-02-19T21:01:16.274290+0000","last_unstale":"2024-02-19T21:06:13.197789+0000","last_undegraded":"2024-02-19T21:06:13.197789+0000","last_fullsized":"2024-02-19T21:06:13.197789+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:54.972832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:54.972832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.216814+0000","last_change":"2024-02-19T21:04:05.136254+0000","last_active":"2024-02-19T21:06:13.216814+0000","last_peered":"2024-02-19T21:06:13.216814+0000","last_clean":"2024-02-19T21:06:13.216814+0000","last_became_active":"2024-02-19T21:01:22.292299+0000","last_became_peered":"2024-02-19T21:01:22.292299+0000","last_unstale":"2024-02-19T21:06:13.216814+0000","last_undegraded":"2024-02-19T21:06:13.216814+0000","last_fullsized":"2024-02-19T21:06:13.216814+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:05.136186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:05.136186+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.200577+0000","last_change":"2024-02-19T21:04:39.667403+0000","last_active":"2024-02-19T21:06:13.200577+0000","last_peered":"2024-02-19T21:06:13.200577+0000","last_clean":"2024-02-19T21:06:13.200577+0000","last_became_active":"2024-02-19T21:04:39.666739+0000","last_became_peered":"2024-02-19T21:04:39.666739+0000","last_unstale":"2024-02-19T21:06:13.200577+0000","last_undegraded":"2024-02-19T21:06:13.200577+0000","last_fullsized":"2024-02-19T21:06:13.200577+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"168'14","reported_seq":81,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204853+0000","last_change":"2024-02-19T21:06:00.211010+0000","last_active":"2024-02-19T21:06:13.204853+0000","last_peered":"2024-02-19T21:06:13.204853+0000","last_clean":"2024-02-19T21:06:13.204853+0000","last_became_active":"2024-02-19T21:06:00.210690+0000","last_became_peered":"2024-02-19T21:06:00.210690+0000","last_unstale":"2024-02-19T21:06:13.204853+0000","last_undegraded":"2024-02-19T21:06:13.204853+0000","last_fullsized":"2024-02-19T21:06:13.204853+0000","mapping_epoch":157,"log_start":"168'14","ondisk_log_start":"168'14","created":98,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":151,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.200980+0000","last_change":"2024-02-19T21:04:15.868824+0000","last_active":"2024-02-19T21:06:13.200980+0000","last_peered":"2024-02-19T21:06:13.200980+0000","last_clean":"2024-02-19T21:06:13.200980+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:06:13.200980+0000","last_undegraded":"2024-02-19T21:06:13.200980+0000","last_fullsized":"2024-02-19T21:06:13.200980+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:15.868726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:15.868726+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":169,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.200389+0000","last_change":"2024-02-19T21:04:56.846706+0000","last_active":"2024-02-19T21:06:13.200389+0000","last_peered":"2024-02-19T21:06:13.200389+0000","last_clean":"2024-02-19T21:06:13.200389+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:06:13.200389+0000","last_undegraded":"2024-02-19T21:06:13.200389+0000","last_fullsized":"2024-02-19T21:06:13.200389+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:04:56.846622+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.846622+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":174,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.200308+0000","last_change":"2024-02-19T21:05:45.014031+0000","last_active":"2024-02-19T21:06:13.200308+0000","last_peered":"2024-02-19T21:06:13.200308+0000","last_clean":"2024-02-19T21:06:13.200308+0000","last_became_active":"2024-02-19T21:05:20.219851+0000","last_became_peered":"2024-02-19T21:05:20.219851+0000","last_unstale":"2024-02-19T21:06:13.200308+0000","last_undegraded":"2024-02-19T21:06:13.200308+0000","last_fullsized":"2024-02-19T21:06:13.200308+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:45.013946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:45.013946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":454,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.187697+0000","last_change":"2024-02-19T21:06:10.791107+0000","last_active":"2024-02-19T21:06:14.187697+0000","last_peered":"2024-02-19T21:06:14.187697+0000","last_clean":"2024-02-19T21:06:14.187697+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:06:14.187697+0000","last_undegraded":"2024-02-19T21:06:14.187697+0000","last_fullsized":"2024-02-19T21:06:14.187697+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:06:10.791036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:10.791036+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":291,"num_read_kb":286,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":148,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.208818+0000","last_change":"2024-02-19T21:04:11.900738+0000","last_active":"2024-02-19T21:06:13.208818+0000","last_peered":"2024-02-19T21:06:13.208818+0000","last_clean":"2024-02-19T21:06:13.208818+0000","last_became_active":"2024-02-19T21:01:22.607883+0000","last_became_peered":"2024-02-19T21:01:22.607883+0000","last_unstale":"2024-02-19T21:06:13.208818+0000","last_undegraded":"2024-02-19T21:06:13.208818+0000","last_fullsized":"2024-02-19T21:06:13.208818+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.900625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.900625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204756+0000","last_change":"2024-02-19T21:04:39.692210+0000","last_active":"2024-02-19T21:06:13.204756+0000","last_peered":"2024-02-19T21:06:13.204756+0000","last_clean":"2024-02-19T21:06:13.204756+0000","last_became_active":"2024-02-19T21:04:39.691434+0000","last_became_peered":"2024-02-19T21:04:39.691434+0000","last_unstale":"2024-02-19T21:06:13.204756+0000","last_undegraded":"2024-02-19T21:06:13.204756+0000","last_fullsized":"2024-02-19T21:06:13.204756+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"100'2","reported_seq":124,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.208435+0000","last_change":"2024-02-19T21:06:12.215535+0000","last_active":"2024-02-19T21:06:13.208435+0000","last_peered":"2024-02-19T21:06:13.208435+0000","last_clean":"2024-02-19T21:06:13.208435+0000","last_became_active":"2024-02-19T21:06:12.215103+0000","last_became_peered":"2024-02-19T21:06:12.215103+0000","last_unstale":"2024-02-19T21:06:13.208435+0000","last_undegraded":"2024-02-19T21:06:13.208435+0000","last_fullsized":"2024-02-19T21:06:13.208435+0000","mapping_epoch":98,"log_start":"100'2","ondisk_log_start":"100'2","created":98,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":155,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.207043+0000","last_change":"2024-02-19T21:05:01.368595+0000","last_active":"2024-02-19T21:06:13.207043+0000","last_peered":"2024-02-19T21:06:13.207043+0000","last_clean":"2024-02-19T21:06:13.207043+0000","last_became_active":"2024-02-19T21:01:20.290758+0000","last_became_peered":"2024-02-19T21:01:20.290758+0000","last_unstale":"2024-02-19T21:06:13.207043+0000","last_undegraded":"2024-02-19T21:06:13.207043+0000","last_fullsized":"2024-02-19T21:06:13.207043+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:01.368509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.368509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":266,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.191976+0000","last_change":"2024-02-19T21:05:42.927711+0000","last_active":"2024-02-19T21:06:14.191976+0000","last_peered":"2024-02-19T21:06:14.191976+0000","last_clean":"2024-02-19T21:06:14.191976+0000","last_became_active":"2024-02-19T21:01:18.289076+0000","last_became_peered":"2024-02-19T21:01:18.289076+0000","last_unstale":"2024-02-19T21:06:14.191976+0000","last_undegraded":"2024-02-19T21:06:14.191976+0000","last_fullsized":"2024-02-19T21:06:14.191976+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-19T21:05:42.927640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:42.927640+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":164,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.214060+0000","last_change":"2024-02-19T21:05:20.181793+0000","last_active":"2024-02-19T21:06:13.214060+0000","last_peered":"2024-02-19T21:06:13.214060+0000","last_clean":"2024-02-19T21:06:13.214060+0000","last_became_active":"2024-02-19T21:05:20.181450+0000","last_became_peered":"2024-02-19T21:05:20.181450+0000","last_unstale":"2024-02-19T21:06:13.214060+0000","last_undegraded":"2024-02-19T21:06:13.214060+0000","last_fullsized":"2024-02-19T21:06:13.214060+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:06.145411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:06.145411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.198514+0000","last_change":"2024-02-19T21:04:49.004211+0000","last_active":"2024-02-19T21:06:13.198514+0000","last_peered":"2024-02-19T21:06:13.198514+0000","last_clean":"2024-02-19T21:06:13.198514+0000","last_became_active":"2024-02-19T21:01:22.291396+0000","last_became_peered":"2024-02-19T21:01:22.291396+0000","last_unstale":"2024-02-19T21:06:13.198514+0000","last_undegraded":"2024-02-19T21:06:13.198514+0000","last_fullsized":"2024-02-19T21:06:13.198514+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.004148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.004148+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.199661+0000","last_change":"2024-02-19T21:04:39.721663+0000","last_active":"2024-02-19T21:06:13.199661+0000","last_peered":"2024-02-19T21:06:13.199661+0000","last_clean":"2024-02-19T21:06:13.199661+0000","last_became_active":"2024-02-19T21:04:39.721416+0000","last_became_peered":"2024-02-19T21:04:39.721416+0000","last_unstale":"2024-02-19T21:06:13.199661+0000","last_undegraded":"2024-02-19T21:06:13.199661+0000","last_fullsized":"2024-02-19T21:06:13.199661+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":136,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.740591+0000","last_change":"2024-02-19T21:06:14.740591+0000","last_active":"2024-02-19T21:06:14.740591+0000","last_peered":"2024-02-19T21:06:14.740591+0000","last_clean":"2024-02-19T21:06:14.740591+0000","last_became_active":"2024-02-19T21:05:48.227489+0000","last_became_peered":"2024-02-19T21:05:48.227489+0000","last_unstale":"2024-02-19T21:06:14.740591+0000","last_undegraded":"2024-02-19T21:06:14.740591+0000","last_fullsized":"2024-02-19T21:06:14.740591+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:14.740525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.740525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"65'8","reported_seq":222,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.182761+0000","last_change":"2024-02-19T21:04:50.353975+0000","last_active":"2024-02-19T21:06:14.182761+0000","last_peered":"2024-02-19T21:06:14.182761+0000","last_clean":"2024-02-19T21:06:14.182761+0000","last_became_active":"2024-02-19T21:01:20.288918+0000","last_became_peered":"2024-02-19T21:01:20.288918+0000","last_unstale":"2024-02-19T21:06:14.182761+0000","last_undegraded":"2024-02-19T21:06:14.182761+0000","last_fullsized":"2024-02-19T21:06:14.182761+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:04:50.353869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:50.353869+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":168,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.205676+0000","last_change":"2024-02-19T21:05:03.922155+0000","last_active":"2024-02-19T21:06:13.205676+0000","last_peered":"2024-02-19T21:06:13.205676+0000","last_clean":"2024-02-19T21:06:13.205676+0000","last_became_active":"2024-02-19T21:01:18.285891+0000","last_became_peered":"2024-02-19T21:01:18.285891+0000","last_unstale":"2024-02-19T21:06:13.205676+0000","last_undegraded":"2024-02-19T21:06:13.205676+0000","last_fullsized":"2024-02-19T21:06:13.205676+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:05:03.922055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.922055+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":178,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.203679+0000","last_change":"2024-02-19T21:05:47.885080+0000","last_active":"2024-02-19T21:06:13.203679+0000","last_peered":"2024-02-19T21:06:13.203679+0000","last_clean":"2024-02-19T21:06:13.203679+0000","last_became_active":"2024-02-19T21:05:20.222948+0000","last_became_peered":"2024-02-19T21:05:20.222948+0000","last_unstale":"2024-02-19T21:06:13.203679+0000","last_undegraded":"2024-02-19T21:06:13.203679+0000","last_fullsized":"2024-02-19T21:06:13.203679+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:47.885024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:05:47.885024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":154,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206125+0000","last_change":"2024-02-19T21:05:15.807832+0000","last_active":"2024-02-19T21:06:13.206125+0000","last_peered":"2024-02-19T21:06:13.206125+0000","last_clean":"2024-02-19T21:06:13.206125+0000","last_became_active":"2024-02-19T21:01:22.292133+0000","last_became_peered":"2024-02-19T21:01:22.292133+0000","last_unstale":"2024-02-19T21:06:13.206125+0000","last_undegraded":"2024-02-19T21:06:13.206125+0000","last_fullsized":"2024-02-19T21:06:13.206125+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:15.807734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:15.807734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.197874+0000","last_change":"2024-02-19T21:04:39.767350+0000","last_active":"2024-02-19T21:06:13.197874+0000","last_peered":"2024-02-19T21:06:13.197874+0000","last_clean":"2024-02-19T21:06:13.197874+0000","last_became_active":"2024-02-19T21:04:39.766156+0000","last_became_peered":"2024-02-19T21:04:39.766156+0000","last_unstale":"2024-02-19T21:06:13.197874+0000","last_undegraded":"2024-02-19T21:06:13.197874+0000","last_fullsized":"2024-02-19T21:06:13.197874+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":104,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.199442+0000","last_change":"2024-02-19T21:04:39.656217+0000","last_active":"2024-02-19T21:06:13.199442+0000","last_peered":"2024-02-19T21:06:13.199442+0000","last_clean":"2024-02-19T21:06:13.199442+0000","last_became_active":"2024-02-19T21:04:39.654865+0000","last_became_peered":"2024-02-19T21:04:39.654865+0000","last_unstale":"2024-02-19T21:06:13.199442+0000","last_undegraded":"2024-02-19T21:06:13.199442+0000","last_fullsized":"2024-02-19T21:06:13.199442+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":151,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.199515+0000","last_change":"2024-02-19T21:04:16.866220+0000","last_active":"2024-02-19T21:06:13.199515+0000","last_peered":"2024-02-19T21:06:13.199515+0000","last_clean":"2024-02-19T21:06:13.199515+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:06:13.199515+0000","last_undegraded":"2024-02-19T21:06:13.199515+0000","last_fullsized":"2024-02-19T21:06:13.199515+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.866116+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":176,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206415+0000","last_change":"2024-02-19T21:04:59.357001+0000","last_active":"2024-02-19T21:06:13.206415+0000","last_peered":"2024-02-19T21:06:13.206415+0000","last_clean":"2024-02-19T21:06:13.206415+0000","last_became_active":"2024-02-19T21:01:18.285672+0000","last_became_peered":"2024-02-19T21:01:18.285672+0000","last_unstale":"2024-02-19T21:06:13.206415+0000","last_undegraded":"2024-02-19T21:06:13.206415+0000","last_fullsized":"2024-02-19T21:06:13.206415+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:04:59.356922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:59.356922+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":175,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.203908+0000","last_change":"2024-02-19T21:05:01.259381+0000","last_active":"2024-02-19T21:06:13.203908+0000","last_peered":"2024-02-19T21:06:13.203908+0000","last_clean":"2024-02-19T21:06:13.203908+0000","last_became_active":"2024-02-19T21:01:16.273545+0000","last_became_peered":"2024-02-19T21:01:16.273545+0000","last_unstale":"2024-02-19T21:06:13.203908+0000","last_undegraded":"2024-02-19T21:06:13.203908+0000","last_fullsized":"2024-02-19T21:06:13.203908+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:05:01.259313+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.259313+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.198798+0000","last_change":"2024-02-19T21:05:06.001027+0000","last_active":"2024-02-19T21:06:13.198798+0000","last_peered":"2024-02-19T21:06:13.198798+0000","last_clean":"2024-02-19T21:06:13.198798+0000","last_became_active":"2024-02-19T21:01:22.295912+0000","last_became_peered":"2024-02-19T21:01:22.295912+0000","last_unstale":"2024-02-19T21:06:13.198798+0000","last_undegraded":"2024-02-19T21:06:13.198798+0000","last_fullsized":"2024-02-19T21:06:13.198798+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:06.000903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:05:06.000903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.201065+0000","last_change":"2024-02-19T21:04:39.696886+0000","last_active":"2024-02-19T21:06:13.201065+0000","last_peered":"2024-02-19T21:06:13.201065+0000","last_clean":"2024-02-19T21:06:13.201065+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:06:13.201065+0000","last_undegraded":"2024-02-19T21:06:13.201065+0000","last_fullsized":"2024-02-19T21:06:13.201065+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":174,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206243+0000","last_change":"2024-02-19T21:06:09.793283+0000","last_active":"2024-02-19T21:06:13.206243+0000","last_peered":"2024-02-19T21:06:13.206243+0000","last_clean":"2024-02-19T21:06:13.206243+0000","last_became_active":"2024-02-19T21:01:18.293773+0000","last_became_peered":"2024-02-19T21:01:18.293773+0000","last_unstale":"2024-02-19T21:06:13.206243+0000","last_undegraded":"2024-02-19T21:06:13.206243+0000","last_fullsized":"2024-02-19T21:06:13.206243+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:06:09.793208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.793208+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":155,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.199387+0000","last_change":"2024-02-19T21:04:49.819984+0000","last_active":"2024-02-19T21:06:13.199387+0000","last_peered":"2024-02-19T21:06:13.199387+0000","last_clean":"2024-02-19T21:06:13.199387+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:06:13.199387+0000","last_undegraded":"2024-02-19T21:06:13.199387+0000","last_fullsized":"2024-02-19T21:06:13.199387+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.819921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.819921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":151,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.200347+0000","last_change":"2024-02-19T21:04:11.841348+0000","last_active":"2024-02-19T21:06:13.200347+0000","last_peered":"2024-02-19T21:06:13.200347+0000","last_clean":"2024-02-19T21:06:13.200347+0000","last_became_active":"2024-02-19T21:01:20.285298+0000","last_became_peered":"2024-02-19T21:01:20.285298+0000","last_unstale":"2024-02-19T21:06:13.200347+0000","last_undegraded":"2024-02-19T21:06:13.200347+0000","last_fullsized":"2024-02-19T21:06:13.200347+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.841184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.841184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":154,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.201830+0000","last_change":"2024-02-19T21:05:11.826779+0000","last_active":"2024-02-19T21:06:13.201830+0000","last_peered":"2024-02-19T21:06:13.201830+0000","last_clean":"2024-02-19T21:06:13.201830+0000","last_became_active":"2024-02-19T21:01:22.297086+0000","last_became_peered":"2024-02-19T21:01:22.297086+0000","last_unstale":"2024-02-19T21:06:13.201830+0000","last_undegraded":"2024-02-19T21:06:13.201830+0000","last_fullsized":"2024-02-19T21:06:13.201830+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:11.826647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:11.826647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.205146+0000","last_change":"2024-02-19T21:04:39.639826+0000","last_active":"2024-02-19T21:06:13.205146+0000","last_peered":"2024-02-19T21:06:13.205146+0000","last_clean":"2024-02-19T21:06:13.205146+0000","last_became_active":"2024-02-19T21:04:39.639518+0000","last_became_peered":"2024-02-19T21:04:39.639518+0000","last_unstale":"2024-02-19T21:06:13.205146+0000","last_undegraded":"2024-02-19T21:06:13.205146+0000","last_fullsized":"2024-02-19T21:06:13.205146+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":165,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.200265+0000","last_change":"2024-02-19T21:04:06.807310+0000","last_active":"2024-02-19T21:06:13.200265+0000","last_peered":"2024-02-19T21:06:13.200265+0000","last_clean":"2024-02-19T21:06:13.200265+0000","last_became_active":"2024-02-19T21:01:18.293913+0000","last_became_peered":"2024-02-19T21:01:18.293913+0000","last_unstale":"2024-02-19T21:06:13.200265+0000","last_undegraded":"2024-02-19T21:06:13.200265+0000","last_fullsized":"2024-02-19T21:06:13.200265+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:04:06.807190+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:04:06.807190+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":155,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.197842+0000","last_change":"2024-02-19T21:04:08.757186+0000","last_active":"2024-02-19T21:06:13.197842+0000","last_peered":"2024-02-19T21:06:13.197842+0000","last_clean":"2024-02-19T21:06:13.197842+0000","last_became_active":"2024-02-19T21:01:16.271210+0000","last_became_peered":"2024-02-19T21:01:16.271210+0000","last_unstale":"2024-02-19T21:06:13.197842+0000","last_undegraded":"2024-02-19T21:06:13.197842+0000","last_fullsized":"2024-02-19T21:06:13.197842+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:08.757098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:08.757098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":153,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.216613+0000","last_change":"2024-02-19T21:05:04.174022+0000","last_active":"2024-02-19T21:06:13.216613+0000","last_peered":"2024-02-19T21:06:13.216613+0000","last_clean":"2024-02-19T21:06:13.216613+0000","last_became_active":"2024-02-19T21:01:20.288589+0000","last_became_peered":"2024-02-19T21:01:20.288589+0000","last_unstale":"2024-02-19T21:06:13.216613+0000","last_undegraded":"2024-02-19T21:06:13.216613+0000","last_fullsized":"2024-02-19T21:06:13.216613+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:04.173902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:04.173902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":153,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.208597+0000","last_change":"2024-02-19T21:06:05.431079+0000","last_active":"2024-02-19T21:06:13.208597+0000","last_peered":"2024-02-19T21:06:13.208597+0000","last_clean":"2024-02-19T21:06:13.208597+0000","last_became_active":"2024-02-19T21:01:22.608285+0000","last_became_peered":"2024-02-19T21:01:22.608285+0000","last_unstale":"2024-02-19T21:06:13.208597+0000","last_undegraded":"2024-02-19T21:06:13.208597+0000","last_fullsized":"2024-02-19T21:06:13.208597+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:05.431017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:05.431017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204697+0000","last_change":"2024-02-19T21:04:39.641073+0000","last_active":"2024-02-19T21:06:13.204697+0000","last_peered":"2024-02-19T21:06:13.204697+0000","last_clean":"2024-02-19T21:06:13.204697+0000","last_became_active":"2024-02-19T21:04:39.639775+0000","last_became_peered":"2024-02-19T21:04:39.639775+0000","last_unstale":"2024-02-19T21:06:13.204697+0000","last_undegraded":"2024-02-19T21:06:13.204697+0000","last_fullsized":"2024-02-19T21:06:13.204697+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":168,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.198343+0000","last_change":"2024-02-19T21:04:22.763995+0000","last_active":"2024-02-19T21:06:13.198343+0000","last_peered":"2024-02-19T21:06:13.198343+0000","last_clean":"2024-02-19T21:06:13.198343+0000","last_became_active":"2024-02-19T21:01:18.294369+0000","last_became_peered":"2024-02-19T21:01:18.294369+0000","last_unstale":"2024-02-19T21:06:13.198343+0000","last_undegraded":"2024-02-19T21:06:13.198343+0000","last_fullsized":"2024-02-19T21:06:13.198343+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:04:22.763926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:22.763926+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":155,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.213876+0000","last_change":"2024-02-19T21:04:09.180232+0000","last_active":"2024-02-19T21:06:13.213876+0000","last_peered":"2024-02-19T21:06:13.213876+0000","last_clean":"2024-02-19T21:06:13.213876+0000","last_became_active":"2024-02-19T21:01:16.271608+0000","last_became_peered":"2024-02-19T21:01:16.271608+0000","last_unstale":"2024-02-19T21:06:13.213876+0000","last_undegraded":"2024-02-19T21:06:13.213876+0000","last_fullsized":"2024-02-19T21:06:13.213876+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:09.180159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:09.180159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":219,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.231411+0000","last_change":"2024-02-19T21:05:34.889050+0000","last_active":"2024-02-19T21:06:14.231411+0000","last_peered":"2024-02-19T21:06:14.231411+0000","last_clean":"2024-02-19T21:06:14.231411+0000","last_became_active":"2024-02-19T21:01:20.288799+0000","last_became_peered":"2024-02-19T21:01:20.288799+0000","last_unstale":"2024-02-19T21:06:14.231411+0000","last_undegraded":"2024-02-19T21:06:14.231411+0000","last_fullsized":"2024-02-19T21:06:14.231411+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:05:34.888948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:34.888948+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":155,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206124+0000","last_change":"2024-02-19T21:05:43.790298+0000","last_active":"2024-02-19T21:06:13.206124+0000","last_peered":"2024-02-19T21:06:13.206124+0000","last_clean":"2024-02-19T21:06:13.206124+0000","last_became_active":"2024-02-19T21:01:22.297188+0000","last_became_peered":"2024-02-19T21:01:22.297188+0000","last_unstale":"2024-02-19T21:06:13.206124+0000","last_undegraded":"2024-02-19T21:06:13.206124+0000","last_fullsized":"2024-02-19T21:06:13.206124+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:05:43.790222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:43.790222+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206586+0000","last_change":"2024-02-19T21:04:39.688410+0000","last_active":"2024-02-19T21:06:13.206586+0000","last_peered":"2024-02-19T21:06:13.206586+0000","last_clean":"2024-02-19T21:06:13.206586+0000","last_became_active":"2024-02-19T21:04:39.687668+0000","last_became_peered":"2024-02-19T21:04:39.687668+0000","last_unstale":"2024-02-19T21:06:13.206586+0000","last_undegraded":"2024-02-19T21:06:13.206586+0000","last_fullsized":"2024-02-19T21:06:13.206586+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":169,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.202052+0000","last_change":"2024-02-19T21:04:15.187687+0000","last_active":"2024-02-19T21:06:13.202052+0000","last_peered":"2024-02-19T21:06:13.202052+0000","last_clean":"2024-02-19T21:06:13.202052+0000","last_became_active":"2024-02-19T21:01:18.293786+0000","last_became_peered":"2024-02-19T21:01:18.293786+0000","last_unstale":"2024-02-19T21:06:13.202052+0000","last_undegraded":"2024-02-19T21:06:13.202052+0000","last_fullsized":"2024-02-19T21:06:13.202052+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:15.187587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:15.187587+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":160,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206813+0000","last_change":"2024-02-19T21:05:10.715545+0000","last_active":"2024-02-19T21:06:13.206813+0000","last_peered":"2024-02-19T21:06:13.206813+0000","last_clean":"2024-02-19T21:06:13.206813+0000","last_became_active":"2024-02-19T21:01:16.273181+0000","last_became_peered":"2024-02-19T21:01:16.273181+0000","last_unstale":"2024-02-19T21:06:13.206813+0000","last_undegraded":"2024-02-19T21:06:13.206813+0000","last_fullsized":"2024-02-19T21:06:13.206813+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:10.715442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:10.715442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":219,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.183771+0000","last_change":"2024-02-19T21:05:02.007769+0000","last_active":"2024-02-19T21:06:14.183771+0000","last_peered":"2024-02-19T21:06:14.183771+0000","last_clean":"2024-02-19T21:06:14.183771+0000","last_became_active":"2024-02-19T21:01:20.283943+0000","last_became_peered":"2024-02-19T21:01:20.283943+0000","last_unstale":"2024-02-19T21:06:14.183771+0000","last_undegraded":"2024-02-19T21:06:14.183771+0000","last_fullsized":"2024-02-19T21:06:14.183771+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:05:02.007654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.007654+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204149+0000","last_change":"2024-02-19T21:04:15.099340+0000","last_active":"2024-02-19T21:06:13.204149+0000","last_peered":"2024-02-19T21:06:13.204149+0000","last_clean":"2024-02-19T21:06:13.204149+0000","last_became_active":"2024-02-19T21:01:22.299969+0000","last_became_peered":"2024-02-19T21:01:22.299969+0000","last_unstale":"2024-02-19T21:06:13.204149+0000","last_undegraded":"2024-02-19T21:06:13.204149+0000","last_fullsized":"2024-02-19T21:06:13.204149+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:15.099208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_clean_scrub_stamp":"2024-02-19T21:04:15.099208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":116,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.200104+0000","last_change":"2024-02-19T21:04:39.767396+0000","last_active":"2024-02-19T21:06:13.200104+0000","last_peered":"2024-02-19T21:06:13.200104+0000","last_clean":"2024-02-19T21:06:13.200104+0000","last_became_active":"2024-02-19T21:04:39.760167+0000","last_became_peered":"2024-02-19T21:04:39.760167+0000","last_unstale":"2024-02-19T21:06:13.200104+0000","last_undegraded":"2024-02-19T21:06:13.200104+0000","last_fullsized":"2024-02-19T21:06:13.200104+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":157,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.213635+0000","last_change":"2024-02-19T21:05:39.892409+0000","last_active":"2024-02-19T21:06:13.213635+0000","last_peered":"2024-02-19T21:06:13.213635+0000","last_clean":"2024-02-19T21:06:13.213635+0000","last_became_active":"2024-02-19T21:01:30.922577+0000","last_became_peered":"2024-02-19T21:01:30.922577+0000","last_unstale":"2024-02-19T21:06:13.213635+0000","last_undegraded":"2024-02-19T21:06:13.213635+0000","last_fullsized":"2024-02-19T21:06:13.213635+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:05:39.892300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:39.892300+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":160,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.198238+0000","last_change":"2024-02-19T21:05:01.685058+0000","last_active":"2024-02-19T21:06:13.198238+0000","last_peered":"2024-02-19T21:06:13.198238+0000","last_clean":"2024-02-19T21:06:13.198238+0000","last_became_active":"2024-02-19T21:01:16.271316+0000","last_became_peered":"2024-02-19T21:01:16.271316+0000","last_unstale":"2024-02-19T21:06:13.198238+0000","last_undegraded":"2024-02-19T21:06:13.198238+0000","last_fullsized":"2024-02-19T21:06:13.198238+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.684958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":151,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.198690+0000","last_change":"2024-02-19T21:04:56.999269+0000","last_active":"2024-02-19T21:06:13.198690+0000","last_peered":"2024-02-19T21:06:13.198690+0000","last_clean":"2024-02-19T21:06:13.198690+0000","last_became_active":"2024-02-19T21:01:20.289691+0000","last_became_peered":"2024-02-19T21:01:20.289691+0000","last_unstale":"2024-02-19T21:06:13.198690+0000","last_undegraded":"2024-02-19T21:06:13.198690+0000","last_fullsized":"2024-02-19T21:06:13.198690+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:56.999142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.999142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":154,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206056+0000","last_change":"2024-02-19T21:06:12.815805+0000","last_active":"2024-02-19T21:06:13.206056+0000","last_peered":"2024-02-19T21:06:13.206056+0000","last_clean":"2024-02-19T21:06:13.206056+0000","last_became_active":"2024-02-19T21:01:22.293093+0000","last_became_peered":"2024-02-19T21:01:22.293093+0000","last_unstale":"2024-02-19T21:06:13.206056+0000","last_undegraded":"2024-02-19T21:06:13.206056+0000","last_fullsized":"2024-02-19T21:06:13.206056+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:12.815725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:12.815725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.200322+0000","last_change":"2024-02-19T21:04:39.674336+0000","last_active":"2024-02-19T21:06:13.200322+0000","last_peered":"2024-02-19T21:06:13.200322+0000","last_clean":"2024-02-19T21:06:13.200322+0000","last_became_active":"2024-02-19T21:04:39.672228+0000","last_became_peered":"2024-02-19T21:04:39.672228+0000","last_unstale":"2024-02-19T21:06:13.200322+0000","last_undegraded":"2024-02-19T21:06:13.200322+0000","last_fullsized":"2024-02-19T21:06:13.200322+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":177,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.201347+0000","last_change":"2024-02-19T21:06:09.595836+0000","last_active":"2024-02-19T21:06:13.201347+0000","last_peered":"2024-02-19T21:06:13.201347+0000","last_clean":"2024-02-19T21:06:13.201347+0000","last_became_active":"2024-02-19T21:01:18.286128+0000","last_became_peered":"2024-02-19T21:01:18.286128+0000","last_unstale":"2024-02-19T21:06:13.201347+0000","last_undegraded":"2024-02-19T21:06:13.201347+0000","last_fullsized":"2024-02-19T21:06:13.201347+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:06:09.595752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.595752+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":160,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.208026+0000","last_change":"2024-02-19T21:05:09.756185+0000","last_active":"2024-02-19T21:06:13.208026+0000","last_peered":"2024-02-19T21:06:13.208026+0000","last_clean":"2024-02-19T21:06:13.208026+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:06:13.208026+0000","last_undegraded":"2024-02-19T21:06:13.208026+0000","last_fullsized":"2024-02-19T21:06:13.208026+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:05:09.756096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":156,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.205755+0000","last_change":"2024-02-19T21:05:43.904934+0000","last_active":"2024-02-19T21:06:13.205755+0000","last_peered":"2024-02-19T21:06:13.205755+0000","last_clean":"2024-02-19T21:06:13.205755+0000","last_became_active":"2024-02-19T21:01:20.289020+0000","last_became_peered":"2024-02-19T21:01:20.289020+0000","last_unstale":"2024-02-19T21:06:13.205755+0000","last_undegraded":"2024-02-19T21:06:13.205755+0000","last_fullsized":"2024-02-19T21:06:13.205755+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:43.904870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:43.904870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.196690+0000","last_change":"2024-02-19T21:04:51.014323+0000","last_active":"2024-02-19T21:06:13.196690+0000","last_peered":"2024-02-19T21:06:13.196690+0000","last_clean":"2024-02-19T21:06:13.196690+0000","last_became_active":"2024-02-19T21:01:22.292704+0000","last_became_peered":"2024-02-19T21:01:22.292704+0000","last_unstale":"2024-02-19T21:06:13.196690+0000","last_undegraded":"2024-02-19T21:06:13.196690+0000","last_fullsized":"2024-02-19T21:06:13.196690+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:51.014245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:51.014245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.199935+0000","last_change":"2024-02-19T21:04:39.672911+0000","last_active":"2024-02-19T21:06:13.199935+0000","last_peered":"2024-02-19T21:06:13.199935+0000","last_clean":"2024-02-19T21:06:13.199935+0000","last_became_active":"2024-02-19T21:04:39.672672+0000","last_became_peered":"2024-02-19T21:04:39.672672+0000","last_unstale":"2024-02-19T21:06:13.199935+0000","last_undegraded":"2024-02-19T21:06:13.199935+0000","last_fullsized":"2024-02-19T21:06:13.199935+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":164,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.196958+0000","last_change":"2024-02-19T21:05:08.226255+0000","last_active":"2024-02-19T21:06:13.196958+0000","last_peered":"2024-02-19T21:06:13.196958+0000","last_clean":"2024-02-19T21:06:13.196958+0000","last_became_active":"2024-02-19T21:01:18.287874+0000","last_became_peered":"2024-02-19T21:01:18.287874+0000","last_unstale":"2024-02-19T21:06:13.196958+0000","last_undegraded":"2024-02-19T21:06:13.196958+0000","last_fullsized":"2024-02-19T21:06:13.196958+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:05:08.226144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:08.226144+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":155,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.217978+0000","last_change":"2024-02-19T21:04:13.205667+0000","last_active":"2024-02-19T21:06:13.217978+0000","last_peered":"2024-02-19T21:06:13.217978+0000","last_clean":"2024-02-19T21:06:13.217978+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:06:13.217978+0000","last_undegraded":"2024-02-19T21:06:13.217978+0000","last_fullsized":"2024-02-19T21:06:13.217978+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:13.205596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:13.205596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":156,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.197742+0000","last_change":"2024-02-19T21:05:40.919364+0000","last_active":"2024-02-19T21:06:13.197742+0000","last_peered":"2024-02-19T21:06:13.197742+0000","last_clean":"2024-02-19T21:06:13.197742+0000","last_became_active":"2024-02-19T21:01:20.290962+0000","last_became_peered":"2024-02-19T21:01:20.290962+0000","last_unstale":"2024-02-19T21:06:13.197742+0000","last_undegraded":"2024-02-19T21:06:13.197742+0000","last_fullsized":"2024-02-19T21:06:13.197742+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:40.919263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:40.919263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":153,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.209763+0000","last_change":"2024-02-19T21:05:44.331615+0000","last_active":"2024-02-19T21:06:13.209763+0000","last_peered":"2024-02-19T21:06:13.209763+0000","last_clean":"2024-02-19T21:06:13.209763+0000","last_became_active":"2024-02-19T21:01:22.297693+0000","last_became_peered":"2024-02-19T21:01:22.297693+0000","last_unstale":"2024-02-19T21:06:13.209763+0000","last_undegraded":"2024-02-19T21:06:13.209763+0000","last_fullsized":"2024-02-19T21:06:13.209763+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.331546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.331546+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.198105+0000","last_change":"2024-02-19T21:04:39.717968+0000","last_active":"2024-02-19T21:06:13.198105+0000","last_peered":"2024-02-19T21:06:13.198105+0000","last_clean":"2024-02-19T21:06:13.198105+0000","last_became_active":"2024-02-19T21:04:39.716606+0000","last_became_peered":"2024-02-19T21:04:39.716606+0000","last_unstale":"2024-02-19T21:06:13.198105+0000","last_undegraded":"2024-02-19T21:06:13.198105+0000","last_fullsized":"2024-02-19T21:06:13.198105+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":172,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.197205+0000","last_change":"2024-02-19T21:04:52.027084+0000","last_active":"2024-02-19T21:06:13.197205+0000","last_peered":"2024-02-19T21:06:13.197205+0000","last_clean":"2024-02-19T21:06:13.197205+0000","last_became_active":"2024-02-19T21:01:18.293793+0000","last_became_peered":"2024-02-19T21:01:18.293793+0000","last_unstale":"2024-02-19T21:06:13.197205+0000","last_undegraded":"2024-02-19T21:06:13.197205+0000","last_fullsized":"2024-02-19T21:06:13.197205+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:04:52.027000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:52.027000+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":160,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.201077+0000","last_change":"2024-02-19T21:06:11.887212+0000","last_active":"2024-02-19T21:06:13.201077+0000","last_peered":"2024-02-19T21:06:13.201077+0000","last_clean":"2024-02-19T21:06:13.201077+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:06:13.201077+0000","last_undegraded":"2024-02-19T21:06:13.201077+0000","last_fullsized":"2024-02-19T21:06:13.201077+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:11.887139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:11.887139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":150,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.210887+0000","last_change":"2024-02-19T21:05:43.361840+0000","last_active":"2024-02-19T21:06:13.210887+0000","last_peered":"2024-02-19T21:06:13.210887+0000","last_clean":"2024-02-19T21:06:13.210887+0000","last_became_active":"2024-02-19T21:01:20.287961+0000","last_became_peered":"2024-02-19T21:01:20.287961+0000","last_unstale":"2024-02-19T21:06:13.210887+0000","last_undegraded":"2024-02-19T21:06:13.210887+0000","last_fullsized":"2024-02-19T21:06:13.210887+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:43.361777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:43.361777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":154,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.198879+0000","last_change":"2024-02-19T21:06:08.757280+0000","last_active":"2024-02-19T21:06:13.198879+0000","last_peered":"2024-02-19T21:06:13.198879+0000","last_clean":"2024-02-19T21:06:13.198879+0000","last_became_active":"2024-02-19T21:01:22.608997+0000","last_became_peered":"2024-02-19T21:01:22.608997+0000","last_unstale":"2024-02-19T21:06:13.198879+0000","last_undegraded":"2024-02-19T21:06:13.198879+0000","last_fullsized":"2024-02-19T21:06:13.198879+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:08.757198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:08.757198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206669+0000","last_change":"2024-02-19T21:04:39.686091+0000","last_active":"2024-02-19T21:06:13.206669+0000","last_peered":"2024-02-19T21:06:13.206669+0000","last_clean":"2024-02-19T21:06:13.206669+0000","last_became_active":"2024-02-19T21:04:39.685620+0000","last_became_peered":"2024-02-19T21:04:39.685620+0000","last_unstale":"2024-02-19T21:06:13.206669+0000","last_undegraded":"2024-02-19T21:06:13.206669+0000","last_fullsized":"2024-02-19T21:06:13.206669+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":46,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204825+0000","last_change":"2024-02-19T21:05:38.203355+0000","last_active":"2024-02-19T21:06:13.204825+0000","last_peered":"2024-02-19T21:06:13.204825+0000","last_clean":"2024-02-19T21:06:13.204825+0000","last_became_active":"2024-02-19T21:05:38.202820+0000","last_became_peered":"2024-02-19T21:05:38.202820+0000","last_unstale":"2024-02-19T21:06:13.204825+0000","last_undegraded":"2024-02-19T21:06:13.204825+0000","last_fullsized":"2024-02-19T21:06:13.204825+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":173,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206380+0000","last_change":"2024-02-19T21:05:47.805827+0000","last_active":"2024-02-19T21:06:13.206380+0000","last_peered":"2024-02-19T21:06:13.206380+0000","last_clean":"2024-02-19T21:06:13.206380+0000","last_became_active":"2024-02-19T21:01:18.293865+0000","last_became_peered":"2024-02-19T21:01:18.293865+0000","last_unstale":"2024-02-19T21:06:13.206380+0000","last_undegraded":"2024-02-19T21:06:13.206380+0000","last_fullsized":"2024-02-19T21:06:13.206380+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:05:47.805757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:47.805757+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":155,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.208276+0000","last_change":"2024-02-19T21:05:07.118727+0000","last_active":"2024-02-19T21:06:13.208276+0000","last_peered":"2024-02-19T21:06:13.208276+0000","last_clean":"2024-02-19T21:06:13.208276+0000","last_became_active":"2024-02-19T21:01:16.272284+0000","last_became_peered":"2024-02-19T21:01:16.272284+0000","last_unstale":"2024-02-19T21:06:13.208276+0000","last_undegraded":"2024-02-19T21:06:13.208276+0000","last_fullsized":"2024-02-19T21:06:13.208276+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:07.118667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:07.118667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":227,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.192750+0000","last_change":"2024-02-19T21:05:41.909864+0000","last_active":"2024-02-19T21:06:14.192750+0000","last_peered":"2024-02-19T21:06:14.192750+0000","last_clean":"2024-02-19T21:06:14.192750+0000","last_became_active":"2024-02-19T21:01:20.290567+0000","last_became_peered":"2024-02-19T21:01:20.290567+0000","last_unstale":"2024-02-19T21:06:14.192750+0000","last_undegraded":"2024-02-19T21:06:14.192750+0000","last_fullsized":"2024-02-19T21:06:14.192750+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:05:41.909796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:41.909796+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":150,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.211584+0000","last_change":"2024-02-19T21:04:18.877455+0000","last_active":"2024-02-19T21:06:13.211584+0000","last_peered":"2024-02-19T21:06:13.211584+0000","last_clean":"2024-02-19T21:06:13.211584+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:06:13.211584+0000","last_undegraded":"2024-02-19T21:06:13.211584+0000","last_fullsized":"2024-02-19T21:06:13.211584+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:04:18.877385+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.201718+0000","last_change":"2024-02-19T21:04:39.700717+0000","last_active":"2024-02-19T21:06:13.201718+0000","last_peered":"2024-02-19T21:06:13.201718+0000","last_clean":"2024-02-19T21:06:13.201718+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:06:13.201718+0000","last_undegraded":"2024-02-19T21:06:13.201718+0000","last_fullsized":"2024-02-19T21:06:13.201718+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.13","version":"168'11","reported_seq":78,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204677+0000","last_change":"2024-02-19T21:05:38.974180+0000","last_active":"2024-02-19T21:06:13.204677+0000","last_peered":"2024-02-19T21:06:13.204677+0000","last_clean":"2024-02-19T21:06:13.204677+0000","last_became_active":"2024-02-19T21:05:38.198047+0000","last_became_peered":"2024-02-19T21:05:38.198047+0000","last_unstale":"2024-02-19T21:06:13.204677+0000","last_undegraded":"2024-02-19T21:06:13.204677+0000","last_fullsized":"2024-02-19T21:06:13.204677+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":177,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.201421+0000","last_change":"2024-02-19T21:05:07.770793+0000","last_active":"2024-02-19T21:06:13.201421+0000","last_peered":"2024-02-19T21:06:13.201421+0000","last_clean":"2024-02-19T21:06:13.201421+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:06:13.201421+0000","last_undegraded":"2024-02-19T21:06:13.201421+0000","last_fullsized":"2024-02-19T21:06:13.201421+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:05:07.770727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:07.770727+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":155,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.196922+0000","last_change":"2024-02-19T21:04:52.990442+0000","last_active":"2024-02-19T21:06:13.196922+0000","last_peered":"2024-02-19T21:06:13.196922+0000","last_clean":"2024-02-19T21:06:13.196922+0000","last_became_active":"2024-02-19T21:01:16.277826+0000","last_became_peered":"2024-02-19T21:01:16.277826+0000","last_unstale":"2024-02-19T21:06:13.196922+0000","last_undegraded":"2024-02-19T21:06:13.196922+0000","last_fullsized":"2024-02-19T21:06:13.196922+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:52.990315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:52.990315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":151,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206421+0000","last_change":"2024-02-19T21:04:54.322261+0000","last_active":"2024-02-19T21:06:13.206421+0000","last_peered":"2024-02-19T21:06:13.206421+0000","last_clean":"2024-02-19T21:06:13.206421+0000","last_became_active":"2024-02-19T21:01:20.288016+0000","last_became_peered":"2024-02-19T21:01:20.288016+0000","last_unstale":"2024-02-19T21:06:13.206421+0000","last_undegraded":"2024-02-19T21:06:13.206421+0000","last_fullsized":"2024-02-19T21:06:13.206421+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:54.322192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:54.322192+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.207834+0000","last_change":"2024-02-19T21:05:08.825596+0000","last_active":"2024-02-19T21:06:13.207834+0000","last_peered":"2024-02-19T21:06:13.207834+0000","last_clean":"2024-02-19T21:06:13.207834+0000","last_became_active":"2024-02-19T21:01:22.293230+0000","last_became_peered":"2024-02-19T21:01:22.293230+0000","last_unstale":"2024-02-19T21:06:13.207834+0000","last_undegraded":"2024-02-19T21:06:13.207834+0000","last_fullsized":"2024-02-19T21:06:13.207834+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:08.825485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:08.825485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204578+0000","last_change":"2024-02-19T21:04:39.801737+0000","last_active":"2024-02-19T21:06:13.204578+0000","last_peered":"2024-02-19T21:06:13.204578+0000","last_clean":"2024-02-19T21:06:13.204578+0000","last_became_active":"2024-02-19T21:04:39.797018+0000","last_became_peered":"2024-02-19T21:04:39.797018+0000","last_unstale":"2024-02-19T21:06:13.204578+0000","last_undegraded":"2024-02-19T21:06:13.204578+0000","last_fullsized":"2024-02-19T21:06:13.204578+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":46,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.196417+0000","last_change":"2024-02-19T21:05:38.200914+0000","last_active":"2024-02-19T21:06:13.196417+0000","last_peered":"2024-02-19T21:06:13.196417+0000","last_clean":"2024-02-19T21:06:13.196417+0000","last_became_active":"2024-02-19T21:05:38.200556+0000","last_became_peered":"2024-02-19T21:05:38.200556+0000","last_unstale":"2024-02-19T21:06:13.196417+0000","last_undegraded":"2024-02-19T21:06:13.196417+0000","last_fullsized":"2024-02-19T21:06:13.196417+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":178,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.201970+0000","last_change":"2024-02-19T21:04:20.874434+0000","last_active":"2024-02-19T21:06:13.201970+0000","last_peered":"2024-02-19T21:06:13.201970+0000","last_clean":"2024-02-19T21:06:13.201970+0000","last_became_active":"2024-02-19T21:01:18.289712+0000","last_became_peered":"2024-02-19T21:01:18.289712+0000","last_unstale":"2024-02-19T21:06:13.201970+0000","last_undegraded":"2024-02-19T21:06:13.201970+0000","last_fullsized":"2024-02-19T21:06:13.201970+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:04:20.874350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:20.874350+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":160,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.199823+0000","last_change":"2024-02-19T21:06:10.922977+0000","last_active":"2024-02-19T21:06:13.199823+0000","last_peered":"2024-02-19T21:06:13.199823+0000","last_clean":"2024-02-19T21:06:13.199823+0000","last_became_active":"2024-02-19T21:01:16.274524+0000","last_became_peered":"2024-02-19T21:01:16.274524+0000","last_unstale":"2024-02-19T21:06:13.199823+0000","last_undegraded":"2024-02-19T21:06:13.199823+0000","last_fullsized":"2024-02-19T21:06:13.199823+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:10.922913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:10.922913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":151,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.208240+0000","last_change":"2024-02-19T21:04:49.851482+0000","last_active":"2024-02-19T21:06:13.208240+0000","last_peered":"2024-02-19T21:06:13.208240+0000","last_clean":"2024-02-19T21:06:13.208240+0000","last_became_active":"2024-02-19T21:01:20.290705+0000","last_became_peered":"2024-02-19T21:01:20.290705+0000","last_unstale":"2024-02-19T21:06:13.208240+0000","last_undegraded":"2024-02-19T21:06:13.208240+0000","last_fullsized":"2024-02-19T21:06:13.208240+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.851392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.851392+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204452+0000","last_change":"2024-02-19T21:05:37.572761+0000","last_active":"2024-02-19T21:06:13.204452+0000","last_peered":"2024-02-19T21:06:13.204452+0000","last_clean":"2024-02-19T21:06:13.204452+0000","last_became_active":"2024-02-19T21:01:22.607435+0000","last_became_peered":"2024-02-19T21:01:22.607435+0000","last_unstale":"2024-02-19T21:06:13.204452+0000","last_undegraded":"2024-02-19T21:06:13.204452+0000","last_fullsized":"2024-02-19T21:06:13.204452+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:37.572695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:05:37.572695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206633+0000","last_change":"2024-02-19T21:04:39.688234+0000","last_active":"2024-02-19T21:06:13.206633+0000","last_peered":"2024-02-19T21:06:13.206633+0000","last_clean":"2024-02-19T21:06:13.206633+0000","last_became_active":"2024-02-19T21:04:39.687307+0000","last_became_peered":"2024-02-19T21:04:39.687307+0000","last_unstale":"2024-02-19T21:06:13.206633+0000","last_undegraded":"2024-02-19T21:06:13.206633+0000","last_fullsized":"2024-02-19T21:06:13.206633+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.11","version":"168'9","reported_seq":64,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.716496+0000","last_change":"2024-02-19T21:06:13.716467+0000","last_active":"2024-02-19T21:06:13.716496+0000","last_peered":"2024-02-19T21:06:13.716496+0000","last_clean":"2024-02-19T21:06:13.716496+0000","last_became_active":"2024-02-19T21:05:38.208537+0000","last_became_peered":"2024-02-19T21:05:38.208537+0000","last_unstale":"2024-02-19T21:06:13.716496+0000","last_undegraded":"2024-02-19T21:06:13.716496+0000","last_fullsized":"2024-02-19T21:06:13.716496+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'9","last_scrub_stamp":"2024-02-19T21:06:13.716391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:13.716391+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":176,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.198022+0000","last_change":"2024-02-19T21:04:50.027760+0000","last_active":"2024-02-19T21:06:13.198022+0000","last_peered":"2024-02-19T21:06:13.198022+0000","last_clean":"2024-02-19T21:06:13.198022+0000","last_became_active":"2024-02-19T21:01:18.285926+0000","last_became_peered":"2024-02-19T21:01:18.285926+0000","last_unstale":"2024-02-19T21:06:13.198022+0000","last_undegraded":"2024-02-19T21:06:13.198022+0000","last_fullsized":"2024-02-19T21:06:13.198022+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:04:50.027690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:50.027690+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":155,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.198615+0000","last_change":"2024-02-19T21:06:02.693096+0000","last_active":"2024-02-19T21:06:13.198615+0000","last_peered":"2024-02-19T21:06:13.198615+0000","last_clean":"2024-02-19T21:06:13.198615+0000","last_became_active":"2024-02-19T21:01:16.274811+0000","last_became_peered":"2024-02-19T21:01:16.274811+0000","last_unstale":"2024-02-19T21:06:13.198615+0000","last_undegraded":"2024-02-19T21:06:13.198615+0000","last_fullsized":"2024-02-19T21:06:13.198615+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:02.693025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:02.693025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":151,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.207290+0000","last_change":"2024-02-19T21:05:38.564563+0000","last_active":"2024-02-19T21:06:13.207290+0000","last_peered":"2024-02-19T21:06:13.207290+0000","last_clean":"2024-02-19T21:06:13.207290+0000","last_became_active":"2024-02-19T21:01:20.291672+0000","last_became_peered":"2024-02-19T21:01:20.291672+0000","last_unstale":"2024-02-19T21:06:13.207290+0000","last_undegraded":"2024-02-19T21:06:13.207290+0000","last_fullsized":"2024-02-19T21:06:13.207290+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:38.564472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:38.564472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.200177+0000","last_change":"2024-02-19T21:04:03.806998+0000","last_active":"2024-02-19T21:06:13.200177+0000","last_peered":"2024-02-19T21:06:13.200177+0000","last_clean":"2024-02-19T21:06:13.200177+0000","last_became_active":"2024-02-19T21:01:22.607099+0000","last_became_peered":"2024-02-19T21:01:22.607099+0000","last_unstale":"2024-02-19T21:06:13.200177+0000","last_undegraded":"2024-02-19T21:06:13.200177+0000","last_fullsized":"2024-02-19T21:06:13.200177+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:03.806903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:03.806903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.203479+0000","last_change":"2024-02-19T21:04:39.781223+0000","last_active":"2024-02-19T21:06:13.203479+0000","last_peered":"2024-02-19T21:06:13.203479+0000","last_clean":"2024-02-19T21:06:13.203479+0000","last_became_active":"2024-02-19T21:04:39.777442+0000","last_became_peered":"2024-02-19T21:04:39.777442+0000","last_unstale":"2024-02-19T21:06:13.203479+0000","last_undegraded":"2024-02-19T21:06:13.203479+0000","last_fullsized":"2024-02-19T21:06:13.203479+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.16","version":"168'7","reported_seq":55,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206493+0000","last_change":"2024-02-19T21:05:38.636968+0000","last_active":"2024-02-19T21:06:13.206493+0000","last_peered":"2024-02-19T21:06:13.206493+0000","last_clean":"2024-02-19T21:06:13.206493+0000","last_became_active":"2024-02-19T21:05:38.198083+0000","last_became_peered":"2024-02-19T21:05:38.198083+0000","last_unstale":"2024-02-19T21:06:13.206493+0000","last_undegraded":"2024-02-19T21:06:13.206493+0000","last_fullsized":"2024-02-19T21:06:13.206493+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"194'13","reported_seq":178,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.772493+0000","last_change":"2024-02-19T21:06:14.772493+0000","last_active":"2024-02-19T21:06:14.772493+0000","last_peered":"2024-02-19T21:06:14.772493+0000","last_clean":"2024-02-19T21:06:14.772493+0000","last_became_active":"2024-02-19T21:01:18.285269+0000","last_became_peered":"2024-02-19T21:01:18.285269+0000","last_unstale":"2024-02-19T21:06:14.772493+0000","last_undegraded":"2024-02-19T21:06:14.772493+0000","last_fullsized":"2024-02-19T21:06:14.772493+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'13","last_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.772417+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":154,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.209961+0000","last_change":"2024-02-19T21:04:59.262530+0000","last_active":"2024-02-19T21:06:13.209961+0000","last_peered":"2024-02-19T21:06:13.209961+0000","last_clean":"2024-02-19T21:06:13.209961+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:06:13.209961+0000","last_undegraded":"2024-02-19T21:06:13.209961+0000","last_fullsized":"2024-02-19T21:06:13.209961+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:59.262400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:59.262400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":151,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.205049+0000","last_change":"2024-02-19T21:06:09.853068+0000","last_active":"2024-02-19T21:06:13.205049+0000","last_peered":"2024-02-19T21:06:13.205049+0000","last_clean":"2024-02-19T21:06:13.205049+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:06:13.205049+0000","last_undegraded":"2024-02-19T21:06:13.205049+0000","last_fullsized":"2024-02-19T21:06:13.205049+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:09.852996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.852996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206495+0000","last_change":"2024-02-19T21:04:49.382806+0000","last_active":"2024-02-19T21:06:13.206495+0000","last_peered":"2024-02-19T21:06:13.206495+0000","last_clean":"2024-02-19T21:06:13.206495+0000","last_became_active":"2024-02-19T21:01:22.292714+0000","last_became_peered":"2024-02-19T21:01:22.292714+0000","last_unstale":"2024-02-19T21:06:13.206495+0000","last_undegraded":"2024-02-19T21:06:13.206495+0000","last_fullsized":"2024-02-19T21:06:13.206495+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.382712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.382712+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.201087+0000","last_change":"2024-02-19T21:04:39.737500+0000","last_active":"2024-02-19T21:06:13.201087+0000","last_peered":"2024-02-19T21:06:13.201087+0000","last_clean":"2024-02-19T21:06:13.201087+0000","last_became_active":"2024-02-19T21:04:39.736990+0000","last_became_peered":"2024-02-19T21:04:39.736990+0000","last_unstale":"2024-02-19T21:06:13.201087+0000","last_undegraded":"2024-02-19T21:06:13.201087+0000","last_fullsized":"2024-02-19T21:06:13.201087+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":163,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.201177+0000","last_change":"2024-02-19T21:04:12.133700+0000","last_active":"2024-02-19T21:06:13.201177+0000","last_peered":"2024-02-19T21:06:13.201177+0000","last_clean":"2024-02-19T21:06:13.201177+0000","last_became_active":"2024-02-19T21:01:18.285735+0000","last_became_peered":"2024-02-19T21:01:18.285735+0000","last_unstale":"2024-02-19T21:06:13.201177+0000","last_undegraded":"2024-02-19T21:06:13.201177+0000","last_fullsized":"2024-02-19T21:06:13.201177+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:04:12.133628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:12.133628+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":159,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.208669+0000","last_change":"2024-02-19T21:05:36.352651+0000","last_active":"2024-02-19T21:06:13.208669+0000","last_peered":"2024-02-19T21:06:13.208669+0000","last_clean":"2024-02-19T21:06:13.208669+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:06:13.208669+0000","last_undegraded":"2024-02-19T21:06:13.208669+0000","last_fullsized":"2024-02-19T21:06:13.208669+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:36.352580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:36.352580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":155,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-02-19T21:06:14.447006+0000","last_change":"2024-02-19T21:06:14.447006+0000","last_active":"2024-02-19T21:06:14.447006+0000","last_peered":"2024-02-19T21:06:14.447006+0000","last_clean":"2024-02-19T21:06:14.447006+0000","last_became_active":"2024-02-19T21:01:20.287940+0000","last_became_peered":"2024-02-19T21:01:20.287940+0000","last_unstale":"2024-02-19T21:06:14.447006+0000","last_undegraded":"2024-02-19T21:06:14.447006+0000","last_fullsized":"2024-02-19T21:06:14.447006+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:14.446932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.446932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.201077+0000","last_change":"2024-02-19T21:04:19.314335+0000","last_active":"2024-02-19T21:06:13.201077+0000","last_peered":"2024-02-19T21:06:13.201077+0000","last_clean":"2024-02-19T21:06:13.201077+0000","last_became_active":"2024-02-19T21:01:22.293282+0000","last_became_peered":"2024-02-19T21:01:22.293282+0000","last_unstale":"2024-02-19T21:06:13.201077+0000","last_undegraded":"2024-02-19T21:06:13.201077+0000","last_fullsized":"2024-02-19T21:06:13.201077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:19.314241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:19.314241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.207442+0000","last_change":"2024-02-19T21:04:39.714784+0000","last_active":"2024-02-19T21:06:13.207442+0000","last_peered":"2024-02-19T21:06:13.207442+0000","last_clean":"2024-02-19T21:06:13.207442+0000","last_became_active":"2024-02-19T21:04:39.714495+0000","last_became_peered":"2024-02-19T21:04:39.714495+0000","last_unstale":"2024-02-19T21:06:13.207442+0000","last_undegraded":"2024-02-19T21:06:13.207442+0000","last_fullsized":"2024-02-19T21:06:13.207442+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.14","version":"100'2","reported_seq":50,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204211+0000","last_change":"2024-02-19T21:05:38.297549+0000","last_active":"2024-02-19T21:06:13.204211+0000","last_peered":"2024-02-19T21:06:13.204211+0000","last_clean":"2024-02-19T21:06:13.204211+0000","last_became_active":"2024-02-19T21:05:38.207362+0000","last_became_peered":"2024-02-19T21:05:38.207362+0000","last_unstale":"2024-02-19T21:06:13.204211+0000","last_undegraded":"2024-02-19T21:06:13.204211+0000","last_fullsized":"2024-02-19T21:06:13.204211+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":169,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.207213+0000","last_change":"2024-02-19T21:05:36.527512+0000","last_active":"2024-02-19T21:06:13.207213+0000","last_peered":"2024-02-19T21:06:13.207213+0000","last_clean":"2024-02-19T21:06:13.207213+0000","last_became_active":"2024-02-19T21:01:18.287645+0000","last_became_peered":"2024-02-19T21:01:18.287645+0000","last_unstale":"2024-02-19T21:06:13.207213+0000","last_undegraded":"2024-02-19T21:06:13.207213+0000","last_fullsized":"2024-02-19T21:06:13.207213+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:05:36.527441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:36.527441+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":155,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.198441+0000","last_change":"2024-02-19T21:04:02.781654+0000","last_active":"2024-02-19T21:06:13.198441+0000","last_peered":"2024-02-19T21:06:13.198441+0000","last_clean":"2024-02-19T21:06:13.198441+0000","last_became_active":"2024-02-19T21:01:16.272132+0000","last_became_peered":"2024-02-19T21:01:16.272132+0000","last_unstale":"2024-02-19T21:06:13.198441+0000","last_undegraded":"2024-02-19T21:06:13.198441+0000","last_fullsized":"2024-02-19T21:06:13.198441+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:02.781556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:02.781556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":151,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.203350+0000","last_change":"2024-02-19T21:04:16.276617+0000","last_active":"2024-02-19T21:06:13.203350+0000","last_peered":"2024-02-19T21:06:13.203350+0000","last_clean":"2024-02-19T21:06:13.203350+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:06:13.203350+0000","last_undegraded":"2024-02-19T21:06:13.203350+0000","last_fullsized":"2024-02-19T21:06:13.203350+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:16.276518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:16.276518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":154,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.206041+0000","last_change":"2024-02-19T21:06:04.601048+0000","last_active":"2024-02-19T21:06:13.206041+0000","last_peered":"2024-02-19T21:06:13.206041+0000","last_clean":"2024-02-19T21:06:13.206041+0000","last_became_active":"2024-02-19T21:01:22.293432+0000","last_became_peered":"2024-02-19T21:01:22.293432+0000","last_unstale":"2024-02-19T21:06:13.206041+0000","last_undegraded":"2024-02-19T21:06:13.206041+0000","last_fullsized":"2024-02-19T21:06:13.206041+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:04.600973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:04.600973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.204302+0000","last_change":"2024-02-19T21:04:39.662781+0000","last_active":"2024-02-19T21:06:13.204302+0000","last_peered":"2024-02-19T21:06:13.204302+0000","last_clean":"2024-02-19T21:06:13.204302+0000","last_became_active":"2024-02-19T21:04:39.661810+0000","last_became_peered":"2024-02-19T21:04:39.661810+0000","last_unstale":"2024-02-19T21:06:13.204302+0000","last_undegraded":"2024-02-19T21:06:13.204302+0000","last_fullsized":"2024-02-19T21:06:13.204302+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":49,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.199195+0000","last_change":"2024-02-19T21:05:38.199899+0000","last_active":"2024-02-19T21:06:13.199195+0000","last_peered":"2024-02-19T21:06:13.199195+0000","last_clean":"2024-02-19T21:06:13.199195+0000","last_became_active":"2024-02-19T21:05:38.199308+0000","last_became_peered":"2024-02-19T21:05:38.199308+0000","last_unstale":"2024-02-19T21:06:13.199195+0000","last_undegraded":"2024-02-19T21:06:13.199195+0000","last_fullsized":"2024-02-19T21:06:13.199195+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":183,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.197294+0000","last_change":"2024-02-19T21:06:09.763553+0000","last_active":"2024-02-19T21:06:13.197294+0000","last_peered":"2024-02-19T21:06:13.197294+0000","last_clean":"2024-02-19T21:06:13.197294+0000","last_became_active":"2024-02-19T21:01:18.283222+0000","last_became_peered":"2024-02-19T21:01:18.283222+0000","last_unstale":"2024-02-19T21:06:13.197294+0000","last_undegraded":"2024-02-19T21:06:13.197294+0000","last_fullsized":"2024-02-19T21:06:13.197294+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:06:09.763482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.763482+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":155,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.207220+0000","last_change":"2024-02-19T21:05:08.599108+0000","last_active":"2024-02-19T21:06:13.207220+0000","last_peered":"2024-02-19T21:06:13.207220+0000","last_clean":"2024-02-19T21:06:13.207220+0000","last_became_active":"2024-02-19T21:01:16.278753+0000","last_became_peered":"2024-02-19T21:01:16.278753+0000","last_unstale":"2024-02-19T21:06:13.207220+0000","last_undegraded":"2024-02-19T21:06:13.207220+0000","last_fullsized":"2024-02-19T21:06:13.207220+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:08.599008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:08.599008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":151,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.207489+0000","last_change":"2024-02-19T21:05:41.532900+0000","last_active":"2024-02-19T21:06:13.207489+0000","last_peered":"2024-02-19T21:06:13.207489+0000","last_clean":"2024-02-19T21:06:13.207489+0000","last_became_active":"2024-02-19T21:01:20.289208+0000","last_became_peered":"2024-02-19T21:01:20.289208+0000","last_unstale":"2024-02-19T21:06:13.207489+0000","last_undegraded":"2024-02-19T21:06:13.207489+0000","last_fullsized":"2024-02-19T21:06:13.207489+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:41.532841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:41.532841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":149,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.198414+0000","last_change":"2024-02-19T21:05:36.920186+0000","last_active":"2024-02-19T21:06:13.198414+0000","last_peered":"2024-02-19T21:06:13.198414+0000","last_clean":"2024-02-19T21:06:13.198414+0000","last_became_active":"2024-02-19T21:01:22.609059+0000","last_became_peered":"2024-02-19T21:01:22.609059+0000","last_unstale":"2024-02-19T21:06:13.198414+0000","last_undegraded":"2024-02-19T21:06:13.198414+0000","last_fullsized":"2024-02-19T21:06:13.198414+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:36.920115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:36.920115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":103,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-02-19T21:06:13.199144+0000","last_change":"2024-02-19T21:04:39.726477+0000","last_active":"2024-02-19T21:06:13.199144+0000","last_peered":"2024-02-19T21:06:13.199144+0000","last_clean":"2024-02-19T21:06:13.199144+0000","last_became_active":"2024-02-19T21:04:39.726195+0000","last_became_peered":"2024-02-19T21:04:39.726195+0000","last_unstale":"2024-02-19T21:06:13.199144+0000","last_undegraded":"2024-02-19T21:06:13.199144+0000","last_fullsized":"2024-02-19T21:06:13.199144+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":138,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":3831,"num_objects":356,"num_object_clones":250,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":356,"num_whiteouts":0,"num_read":86,"num_read_kb":1,"num_write":512,"num_write_kb":511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4374528,"data_stored":11493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1227,"ondisk_log_size":1227,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":291,"num_read_kb":286,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":887,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":126,"acting":126,"num_store_stats":8},{"poolid":8,"num_pg":23,"stat_sum":{"num_bytes":1920,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63,"ondisk_log_size":63,"up":69,"acting":69,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266750,"num_pgs":116,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308840,"kb_used_data":3420,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436312,"statfs":{"total":95995035648,"available":95678783488,"internally_reserved":0,"allocated":3502080,"data_stored":1679593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":0.63700000000000001,"15min":0.55200000000000005},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":8.5640000000000001,"5min":8.5640000000000001,"15min":8.5640000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.1439999999999999,"5min":0.621,"15min":0.53400000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.7110000000000003,"5min":4.7110000000000003,"15min":4.7110000000000003},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":1.5760000000000001,"5min":0.71999999999999997,"15min":0.58899999999999997},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":8.1669999999999998,"5min":8.1669999999999998,"15min":8.1669999999999998},"last":0.378},{"interface":"front","average":{"1min":2.1600000000000001,"5min":0.83999999999999997,"15min":0.628},"min":{"1min":0.35799999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":17.318999999999999,"5min":17.318999999999999,"15min":17.318999999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":0.69699999999999995,"15min":0.56599999999999995},"min":{"1min":0.35199999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":8.4149999999999991,"5min":8.4149999999999991,"15min":8.4149999999999991},"last":0.34799999999999998},{"interface":"front","average":{"1min":1.2749999999999999,"5min":0.68100000000000005,"15min":0.57399999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":6.875,"5min":6.875,"15min":6.875},"last":0.46700000000000003}]},{"osd":3,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":0.65700000000000003,"15min":0.57399999999999995},"min":{"1min":0.32200000000000001,"5min":0.254,"15min":0.254},"max":{"1min":3.8260000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.2490000000000001,"5min":0.65500000000000003,"15min":0.54700000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":6.7380000000000004,"5min":6.7380000000000004,"15min":6.7380000000000004},"last":0.49099999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.47299999999999998,"15min":0.49399999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63800000000000001,"5min":1.373,"15min":1.373},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.437,"5min":0.48599999999999999,"15min":0.48699999999999999},"min":{"1min":0.26600000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.63800000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.55800000000000005}]},{"osd":5,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.46300000000000002,"15min":0.46200000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.433},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.51700000000000002,"15min":0.51300000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.70299999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.52200000000000002,"15min":0.53500000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.59399999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.56399999999999995,"15min":0.57499999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.73099999999999998,"5min":1.262,"15min":1.262},"last":0.57599999999999996}]}]},{"osd":6,"up_from":45,"seq":193273528388,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304776,"kb_used_data":3444,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440376,"statfs":{"total":95995035648,"available":95682945024,"internally_reserved":0,"allocated":3526656,"data_stored":1651161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:05:55 2024","interfaces":[{"interface":"back","average":{"1min":1.99,"5min":0.85699999999999998,"15min":0.61499999999999999},"min":{"1min":0.27800000000000002,"5min":0.222,"15min":0.222},"max":{"1min":17.486999999999998,"5min":17.486999999999998,"15min":17.486999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":2.0030000000000001,"5min":0.94499999999999995,"15min":0.64100000000000001},"min":{"1min":0.26300000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":17.43,"5min":17.43,"15min":17.43},"last":0.69399999999999995}]},{"osd":1,"last update":"Mon Feb 19 21:05:55 2024","interfaces":[{"interface":"back","average":{"1min":1.7390000000000001,"5min":0.85999999999999999,"15min":0.61399999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":14.692,"5min":14.692,"15min":14.692},"last":0.60899999999999999},{"interface":"front","average":{"1min":2.6459999999999999,"5min":0.98999999999999999,"15min":0.66300000000000003},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":14.795,"5min":14.795,"15min":14.795},"last":0.67600000000000005}]},{"osd":2,"last update":"Mon Feb 19 21:05:55 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":0.78400000000000003,"15min":0.60799999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":6.931,"5min":6.931,"15min":6.931},"last":0.63400000000000001},{"interface":"front","average":{"1min":2.681,"5min":1.032,"15min":0.67000000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":14.391,"5min":14.391,"15min":14.391},"last":0.66300000000000003}]},{"osd":3,"last update":"Mon Feb 19 21:05:55 2024","interfaces":[{"interface":"back","average":{"1min":2.4510000000000001,"5min":1.046,"15min":0.68999999999999995},"min":{"1min":0.35299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":14.510999999999999,"5min":14.510999999999999,"15min":14.510999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":2.355,"5min":0.93999999999999995,"15min":0.64200000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":17.585000000000001,"5min":17.585000000000001,"15min":17.585000000000001},"last":0.65400000000000003}]},{"osd":4,"last update":"Mon Feb 19 21:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.41199999999999998,"15min":0.38200000000000001},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.54200000000000004,"5min":1.363,"15min":1.363},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.43099999999999999,"15min":0.40899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.53600000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.48399999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.45200000000000001,"15min":0.41899999999999998},"min":{"1min":0.26900000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56999999999999995,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.48499999999999999,"15min":0.45500000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55100000000000005,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.58299999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:05:22 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.497,"15min":0.51300000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.63500000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.504,"15min":0.52600000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.56799999999999995,"5min":1.534,"15min":1.534},"last":0.57699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":123,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304764,"kb_used_data":3432,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440388,"statfs":{"total":95995035648,"available":95682957312,"internally_reserved":0,"allocated":3514368,"data_stored":1651467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:06:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":0.96199999999999997,"15min":0.55200000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":11.269,"5min":27.777000000000001,"15min":27.777000000000001},"last":0.214},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.0129999999999999,"15min":0.57599999999999996},"min":{"1min":0.24199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":8.843,"5min":27.631,"15min":27.631},"last":0.27000000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:05:17 2024","interfaces":[{"interface":"back","average":{"1min":5.1449999999999996,"5min":1.3620000000000001,"15min":0.72599999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":27.798999999999999,"5min":27.798999999999999,"15min":27.798999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":5.2469999999999999,"5min":1.3799999999999999,"15min":0.73999999999999999},"min":{"1min":0.28100000000000003,"5min":0.255,"15min":0.255},"max":{"1min":27.606999999999999,"5min":27.606999999999999,"15min":27.606999999999999},"last":0.316}]},{"osd":3,"last update":"Mon Feb 19 21:05:45 2024","interfaces":[{"interface":"back","average":{"1min":3.0139999999999998,"5min":0.92800000000000005,"15min":0.59599999999999997},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":15.574999999999999,"5min":15.574999999999999,"15min":15.574999999999999},"last":0.254},{"interface":"front","average":{"1min":3.855,"5min":1.1399999999999999,"15min":0.69099999999999995},"min":{"1min":0.316,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":27.82,"5min":27.82,"15min":27.82},"last":0.32400000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:06:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":0.89800000000000002,"15min":0.67200000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":9.6430000000000007,"5min":9.6430000000000007,"15min":9.6430000000000007},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.333,"5min":1.1659999999999999,"15min":0.76300000000000001},"min":{"1min":0.36399999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":11.119,"5min":27.707000000000001,"15min":27.707000000000001},"last":0.56799999999999995}]},{"osd":5,"last update":"Mon Feb 19 21:05:36 2024","interfaces":[{"interface":"back","average":{"1min":3.262,"5min":1.1240000000000001,"15min":0.76600000000000001},"min":{"1min":0.35699999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":27.529,"5min":27.529,"15min":27.529},"last":0.53800000000000003},{"interface":"front","average":{"1min":2.2389999999999999,"5min":0.92800000000000005,"15min":0.71999999999999997},"min":{"1min":0.379,"5min":0.377,"15min":0.377},"max":{"1min":9.6799999999999997,"5min":9.6799999999999997,"15min":9.6799999999999997},"last":0.46999999999999997}]},{"osd":6,"last update":"Mon Feb 19 21:05:52 2024","interfaces":[{"interface":"back","average":{"1min":2.044,"5min":0.88200000000000001,"15min":0.68799999999999994},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":9.6989999999999998,"5min":9.6989999999999998,"15min":9.6989999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":3.1760000000000002,"5min":1.1399999999999999,"15min":0.81299999999999994},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":27.733000000000001,"5min":27.733000000000001,"15min":27.733000000000001},"last":0.65000000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:05:17 2024","interfaces":[{"interface":"back","average":{"1min":3.456,"5min":1.1759999999999999,"15min":0.78300000000000003},"min":{"1min":0.40400000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":27.576000000000001,"5min":27.576000000000001,"15min":27.576000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":3.722,"5min":1.242,"15min":0.81499999999999995},"min":{"1min":0.41399999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":27.756,"5min":27.756,"15min":27.756},"last":0.54900000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304488,"kb_used_data":3160,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440664,"statfs":{"total":95995035648,"available":95683239936,"internally_reserved":0,"allocated":3235840,"data_stored":1653133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:06:03 2024","interfaces":[{"interface":"back","average":{"1min":2.657,"5min":1.1499999999999999,"15min":0.58199999999999996},"min":{"1min":0.188,"5min":0.188,"15min":0.16200000000000001},"max":{"1min":20.966999999999999,"5min":20.966999999999999,"15min":20.966999999999999},"last":0.32500000000000001},{"interface":"front","average":{"1min":2.3199999999999998,"5min":1.127,"15min":0.57899999999999996},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.188},"max":{"1min":20.366,"5min":20.366,"15min":20.366},"last":0.22600000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:05:23 2024","interfaces":[{"interface":"back","average":{"1min":4.8140000000000001,"5min":1.2989999999999999,"15min":0.67900000000000005},"min":{"1min":0.245,"5min":0.219,"15min":0.219},"max":{"1min":27.916,"5min":27.916,"15min":27.916},"last":0.26100000000000001},{"interface":"front","average":{"1min":3.6320000000000001,"5min":1.097,"15min":0.61299999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":22.004000000000001,"5min":22.004000000000001,"15min":22.004000000000001},"last":0.218}]},{"osd":3,"last update":"Mon Feb 19 21:05:45 2024","interfaces":[{"interface":"back","average":{"1min":4.8940000000000001,"5min":1.365,"15min":0.749},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":25.669,"5min":25.669,"15min":25.669},"last":0.317},{"interface":"front","average":{"1min":3.8879999999999999,"5min":1.173,"15min":0.69099999999999995},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":25.006,"5min":25.006,"15min":25.006},"last":0.28299999999999997}]},{"osd":4,"last update":"Mon Feb 19 21:06:03 2024","interfaces":[{"interface":"back","average":{"1min":2.1680000000000001,"5min":1.097,"15min":0.70899999999999996},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":20.411999999999999,"5min":20.411999999999999,"15min":20.411999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":2.9369999999999998,"5min":1.121,"15min":0.72899999999999998},"min":{"1min":0.32600000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":19.986999999999998,"5min":19.986999999999998,"15min":19.986999999999998},"last":0.496}]},{"osd":5,"last update":"Mon Feb 19 21:05:31 2024","interfaces":[{"interface":"back","average":{"1min":3.7320000000000002,"5min":1.226,"15min":0.83999999999999997},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":20.021999999999998,"5min":20.021999999999998,"15min":20.021999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":3.137,"5min":1.167,"15min":0.95499999999999996},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":20.908999999999999,"5min":20.908999999999999,"15min":20.908999999999999},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:05:49 2024","interfaces":[{"interface":"back","average":{"1min":3.0019999999999998,"5min":1.0700000000000001,"15min":0.76300000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":20.445,"5min":20.445,"15min":20.445},"last":0.52800000000000002},{"interface":"front","average":{"1min":3.1960000000000002,"5min":1.119,"15min":0.81499999999999995},"min":{"1min":0.39000000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":20.047999999999998,"5min":20.047999999999998,"15min":20.047999999999998},"last":0.46600000000000003}]},{"osd":7,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":3.0059999999999998,"5min":1.1699999999999999,"15min":0.93200000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":20.940999999999999,"5min":20.940999999999999,"15min":20.940999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":3.548,"5min":1.206,"15min":0.82499999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":20.471,"5min":20.471,"15min":20.471},"last":0.48099999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300520,"kb_used_data":3292,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444632,"statfs":{"total":95995035648,"available":95687303168,"internally_reserved":0,"allocated":3371008,"data_stored":1646316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:05:22 2024","interfaces":[{"interface":"back","average":{"1min":2.5449999999999999,"5min":0.85299999999999998,"15min":0.51700000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.222},"max":{"1min":26.690999999999999,"5min":26.690999999999999,"15min":26.690999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":2.9300000000000002,"5min":0.94299999999999995,"15min":0.54200000000000004},"min":{"1min":0.23799999999999999,"5min":0.214,"15min":0.214},"max":{"1min":24.315000000000001,"5min":24.315000000000001,"15min":24.315000000000001},"last":0.32000000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:05:22 2024","interfaces":[{"interface":"back","average":{"1min":3.6960000000000002,"5min":1.0960000000000001,"15min":0.59599999999999997},"min":{"1min":0.251,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":21.891999999999999,"5min":21.891999999999999,"15min":21.891999999999999},"last":0.24199999999999999},{"interface":"front","average":{"1min":2.9470000000000001,"5min":0.97199999999999998,"15min":0.58499999999999996},"min":{"1min":0.28899999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":26.748999999999999,"5min":26.748999999999999,"15min":26.748999999999999},"last":0.443}]},{"osd":3,"last update":"Mon Feb 19 21:05:43 2024","interfaces":[{"interface":"back","average":{"1min":2.3610000000000002,"5min":0.82099999999999995,"15min":0.58099999999999996},"min":{"1min":0.27300000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":27.977,"5min":27.977,"15min":27.977},"last":0.35199999999999998},{"interface":"front","average":{"1min":2.9980000000000002,"5min":0.995,"15min":0.65100000000000002},"min":{"1min":0.316,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":26.719999999999999,"5min":26.719999999999999,"15min":26.719999999999999},"last":0.41399999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:06:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":0.89100000000000001,"15min":0.70299999999999996},"min":{"1min":0.34000000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":24.356000000000002,"5min":24.356000000000002,"15min":24.356000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.6240000000000001,"5min":0.85199999999999998,"15min":0.72399999999999998},"min":{"1min":0.38600000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":20.707000000000001,"5min":20.707000000000001,"15min":20.707000000000001},"last":0.32800000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:05:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":0.68899999999999995,"15min":0.67300000000000004},"min":{"1min":0.46200000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":4.569,"5min":4.569,"15min":4.569},"last":0.38100000000000001},{"interface":"front","average":{"1min":2.589,"5min":1.0649999999999999,"15min":0.79500000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":26.606000000000002,"5min":26.606000000000002,"15min":26.606000000000002},"last":0.60499999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:06:01 2024","interfaces":[{"interface":"back","average":{"1min":1.9379999999999999,"5min":0.94799999999999995,"15min":0.73799999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":24.381,"5min":24.381,"15min":24.381},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.69899999999999995,"15min":0.65800000000000003},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.613,"5min":4.5250000000000004,"15min":4.5250000000000004},"last":0.45000000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:05:22 2024","interfaces":[{"interface":"back","average":{"1min":2.6480000000000001,"5min":1.0620000000000001,"15min":0.745},"min":{"1min":0.46200000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":26.658000000000001,"5min":26.658000000000001,"15min":26.658000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":2.109,"5min":0.97199999999999998,"15min":0.72799999999999998},"min":{"1min":0.41899999999999998,"5min":0.379,"15min":0.379},"max":{"1min":24.402000000000001,"5min":24.402000000000001,"15min":24.402000000000001},"last":0.42799999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300512,"kb_used_data":3280,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444640,"statfs":{"total":95995035648,"available":95687311360,"internally_reserved":0,"allocated":3358720,"data_stored":1674271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:05:45 2024","interfaces":[{"interface":"back","average":{"1min":2.427,"5min":1.464,"15min":0.72399999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":28.422000000000001,"5min":77.662000000000006,"15min":77.662000000000006},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.5369999999999999,"5min":0.754,"15min":0.52400000000000002},"min":{"1min":0.25800000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":8.5790000000000006,"5min":17.456,"15min":17.456},"last":0.30499999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:05:45 2024","interfaces":[{"interface":"back","average":{"1min":1.778,"5min":0.79400000000000004,"15min":0.53200000000000003},"min":{"1min":0.26600000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":9.2780000000000005,"5min":17.327999999999999,"15min":17.327999999999999},"last":0.248},{"interface":"front","average":{"1min":1.782,"5min":0.67200000000000004,"15min":0.49199999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":15.004,"5min":15.004,"15min":15.004},"last":0.254}]},{"osd":2,"last update":"Mon Feb 19 21:05:45 2024","interfaces":[{"interface":"back","average":{"1min":1.7050000000000001,"5min":1.173,"15min":0.67500000000000004},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":12.925000000000001,"5min":56.890999999999998,"15min":56.890999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":2.2669999999999999,"5min":1.1679999999999999,"15min":0.67200000000000004},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":17.177,"5min":44.600999999999999,"15min":44.600999999999999},"last":0.36299999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:06:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":0.79400000000000004,"15min":0.629},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":12.847,"5min":12.847,"15min":12.847},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":1.0289999999999999,"15min":0.72299999999999998},"min":{"1min":0.372,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":3.294,"5min":26.393999999999998,"15min":26.393999999999998},"last":0.51500000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:05:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0990000000000002,"5min":0.873,"15min":0.67900000000000005},"min":{"1min":0.40699999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":17.401,"5min":17.401,"15min":17.401},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.891,"5min":0.82799999999999996,"15min":0.65200000000000002},"min":{"1min":0.36199999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":12.882,"5min":12.882,"15min":12.882},"last":0.38400000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:05:56 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":0.83799999999999997,"15min":0.66000000000000003},"min":{"1min":0.376,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":8.5090000000000003,"5min":17.367000000000001,"15min":17.367000000000001},"last":0.441},{"interface":"front","average":{"1min":1.782,"5min":0.81599999999999995,"15min":0.67000000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":12.904,"5min":12.904,"15min":12.904},"last":0.45900000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:05:16 2024","interfaces":[{"interface":"back","average":{"1min":1.976,"5min":0.85199999999999998,"15min":0.65800000000000003},"min":{"1min":0.38800000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":17.428999999999998,"5min":17.428999999999998,"15min":17.428999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":3.956,"5min":1.2569999999999999,"15min":0.80100000000000005},"min":{"1min":0.437,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":44.378,"5min":44.378,"15min":44.378},"last":0.53500000000000003}]}]},{"osd":4,"up_from":31,"seq":133143986254,"num_pgs":119,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304768,"kb_used_data":3444,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440384,"statfs":{"total":95995035648,"available":95682953216,"internally_reserved":0,"allocated":3526656,"data_stored":1674630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:06:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":0.83599999999999997,"15min":0.65200000000000002},"min":{"1min":0.26300000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":19.878,"5min":19.878,"15min":19.878},"last":0.66100000000000003},{"interface":"front","average":{"1min":2.6070000000000002,"5min":1.575,"15min":0.879},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":17.675000000000001,"5min":70.063000000000002,"15min":70.063000000000002},"last":0.63}]},{"osd":1,"last update":"Mon Feb 19 21:06:10 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":0.92000000000000004,"15min":0.66800000000000004},"min":{"1min":0.26300000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":9.1920000000000002,"5min":26.510999999999999,"15min":26.510999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":2.6749999999999998,"5min":1.052,"15min":0.70199999999999996},"min":{"1min":0.28100000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":25.949000000000002,"5min":25.949000000000002,"15min":25.949000000000002},"last":0.57599999999999996}]},{"osd":2,"last update":"Mon Feb 19 21:06:10 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.0629999999999999,"15min":0.70699999999999996},"min":{"1min":0.30299999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":7.516,"5min":37.320999999999998,"15min":37.320999999999998},"last":0.64200000000000002},{"interface":"front","average":{"1min":4.9420000000000002,"5min":2.1219999999999999,"15min":1.0860000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":69.936999999999998,"5min":69.936999999999998,"15min":69.936999999999998},"last":0.65000000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:06:10 2024","interfaces":[{"interface":"back","average":{"1min":4.9660000000000002,"5min":1.681,"15min":0.91300000000000003},"min":{"1min":0.29399999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":69.793000000000006,"5min":69.793000000000006,"15min":69.793000000000006},"last":0.60099999999999998},{"interface":"front","average":{"1min":2.0110000000000001,"5min":1.214,"15min":0.78600000000000003},"min":{"1min":0.29499999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":17.986999999999998,"5min":17.986999999999998,"15min":17.986999999999998},"last":0.70999999999999996}]},{"osd":5,"last update":"Mon Feb 19 21:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.438,"15min":0.434},"min":{"1min":0.23400000000000001,"5min":0.218,"15min":0.218},"max":{"1min":3.2799999999999998,"5min":3.2799999999999998,"15min":3.2799999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.48399999999999999,"15min":0.45000000000000001},"min":{"1min":0.26600000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.4340000000000002,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":0.442}]},{"osd":6,"last update":"Mon Feb 19 21:05:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46200000000000002,"15min":0.39800000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.2330000000000001,"5min":3.2330000000000001,"15min":3.2330000000000001},"last":0.432},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.52600000000000002,"15min":0.45400000000000001},"min":{"1min":0.33100000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.5609999999999999,"5min":4.5609999999999999,"15min":4.5609999999999999},"last":0.48399999999999999}]},{"osd":7,"last update":"Mon Feb 19 21:05:23 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.52100000000000002,"15min":0.45800000000000002},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":6.0730000000000004,"5min":6.0730000000000004,"15min":6.0730000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.51800000000000002,"15min":0.47599999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.202,"5min":3.202,"15min":3.202},"last":0.61099999999999999}]}]},{"osd":5,"up_from":37,"seq":158913790025,"num_pgs":120,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308932,"kb_used_data":3500,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436220,"statfs":{"total":95995035648,"available":95678689280,"internally_reserved":0,"allocated":3584000,"data_stored":1650641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:05:26 2024","interfaces":[{"interface":"back","average":{"1min":21.559999999999999,"5min":4.6870000000000003,"15min":1.907},"min":{"1min":0.27100000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":276.47899999999998,"5min":276.47899999999998,"15min":276.47899999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":6.4080000000000004,"5min":1.698,"15min":0.93100000000000005},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":52.887,"5min":52.887,"15min":52.887},"last":0.434}]},{"osd":1,"last update":"Mon Feb 19 21:05:26 2024","interfaces":[{"interface":"back","average":{"1min":12.054,"5min":2.7999999999999998,"15min":1.2849999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":120.011,"5min":120.011,"15min":120.011},"last":0.42199999999999999},{"interface":"front","average":{"1min":8.0280000000000005,"5min":2.0310000000000001,"15min":1.0349999999999999},"min":{"1min":0.317,"5min":0.307,"15min":0.307},"max":{"1min":118.105,"5min":118.105,"15min":118.105},"last":0.39800000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:05:26 2024","interfaces":[{"interface":"back","average":{"1min":7.6929999999999996,"5min":1.9370000000000001,"15min":1.008},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":82.242000000000004,"5min":82.242000000000004,"15min":82.242000000000004},"last":0.40999999999999998},{"interface":"front","average":{"1min":15.124000000000001,"5min":3.4289999999999998,"15min":1.5009999999999999},"min":{"1min":0.32000000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":232.84999999999999,"5min":232.84999999999999,"15min":232.84999999999999},"last":0.40300000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:05:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5739999999999998,"5min":2.1269999999999998,"15min":1.0680000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":109.70099999999999,"5min":109.70099999999999,"15min":109.70099999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":6.1449999999999996,"5min":1.653,"15min":0.92100000000000004},"min":{"1min":0.376,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":65.519000000000005,"5min":65.519000000000005,"15min":65.519000000000005},"last":0.374}]},{"osd":4,"last update":"Mon Feb 19 21:05:26 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.41499999999999998,"15min":0.39400000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.622,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.437,"15min":0.39800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.59799999999999998,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.28199999999999997}]},{"osd":6,"last update":"Mon Feb 19 21:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.44900000000000001,"15min":0.42499999999999999},"min":{"1min":0.26900000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.495,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.38},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.46400000000000002,"15min":0.44},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.52700000000000002,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.33000000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:05:20 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.496,"15min":0.5},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.62,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63800000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.36599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":344064,"data_stored":878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":1527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":1178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":630784,"data_stored":1652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":692224,"data_stored":1828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":630784,"data_stored":1673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":1543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:06:21.684 INFO:tasks.thrashosds.thrasher:Setting 5.1c to [7, 0, 3] 2024-02-19T21:06:21.684 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.1c', '7', '0', '3'] 2024-02-19T21:06:21.684 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pg-upmap 5.1c 7 0 3 2024-02-19T21:06:21.718 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:21.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:21.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:21.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:19.712945+0000 osd.1 (osd.1) 119 : cluster [DBG] 8.b deep-scrub starts 2024-02-19T21:06:21.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:19.717592+0000 osd.1 (osd.1) 120 : cluster [DBG] 8.b deep-scrub ok 2024-02-19T21:06:21.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:19.842500+0000 osd.5 (osd.5) 96 : cluster [DBG] 2.2 deep-scrub starts 2024-02-19T21:06:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:19.845389+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.2 deep-scrub ok 2024-02-19T21:06:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:20.154758+0000 mgr.y (mgr.14550) 234 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:20.187918+0000 osd.2 (osd.2) 89 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:06:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:20.194010+0000 osd.2 (osd.2) 90 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:06:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:20.301338+0000 mon.c (mon.2) 207 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:20.709127+0000 osd.4 (osd.4) 109 : cluster [DBG] 5.e scrub starts 2024-02-19T21:06:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:20.710725+0000 osd.4 (osd.4) 110 : cluster [DBG] 5.e scrub ok 2024-02-19T21:06:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:21.141142+0000 mon.b (mon.1) 313 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-19T21:06:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:21.142106+0000 mon.a (mon.0) 1448 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-19T21:06:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:21.302169+0000 mon.c (mon.2) 208 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:21.437073+0000 mon.a (mon.0) 1449 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:21.437176+0000 mon.a (mon.0) 1450 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-19T21:06:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:21.437226+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-02-19T21:06:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:21.437958+0000 mon.b (mon.1) 314 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-17"}]: dispatch 2024-02-19T21:06:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:21.439405+0000 mon.b (mon.1) 315 : audit [INF] from='client.? v1:172.21.15.163:0/3616630607' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi163-83854-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:21.439622+0000 mon.a (mon.0) 1452 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-17"}]: dispatch 2024-02-19T21:06:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:21.440379+0000 mon.a (mon.0) 1453 : audit [INF] from='client.47968 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi163-83854-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:21.822 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:06:21.848 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:21.922 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:06:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:21.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:06:22.027 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:06:22.056 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:22.137 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:22.154 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:06:22.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:22.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:06:22.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.177+0000 7f860baa3700 1 -- 172.21.15.163:0/282474235 >> v1:172.21.15.163:6790/0 conn(0x7f8604100780 legacy=0x7f8604102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:22.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.177+0000 7f860baa3700 1 -- 172.21.15.163:0/282474235 shutdown_connections 2024-02-19T21:06:22.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.177+0000 7f860baa3700 1 -- 172.21.15.163:0/282474235 >> 172.21.15.163:0/282474235 conn(0x7f86040f35a0 msgr2=0x7f86040f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:22.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.177+0000 7f860baa3700 1 -- 172.21.15.163:0/282474235 shutdown_connections 2024-02-19T21:06:22.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.177+0000 7f860baa3700 1 -- 172.21.15.163:0/282474235 wait complete. 2024-02-19T21:06:22.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.178+0000 7f860baa3700 1 librados: starting msgr at 2024-02-19T21:06:22.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.178+0000 7f860baa3700 1 librados: starting objecter 2024-02-19T21:06:22.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.178+0000 7f860baa3700 1 Processor -- start 2024-02-19T21:06:22.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.178+0000 7f860baa3700 1 -- start start 2024-02-19T21:06:22.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.178+0000 7f860baa3700 1 librados: setting wanted keys 2024-02-19T21:06:22.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.178+0000 7f860baa3700 1 librados: calling monclient init 2024-02-19T21:06:22.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.178+0000 7f860baa3700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86041172c0 con 0x7f8604100780 2024-02-19T21:06:22.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.178+0000 7f860baa3700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86041184c0 con 0x7f86040fd980 2024-02-19T21:06:22.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.178+0000 7f860baa3700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f86041196c0 con 0x7f86041042c0 2024-02-19T21:06:22.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.178+0000 7f860903e700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8604100780 0x7f86040fa070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:52334/0 (socket says 172.21.15.163:52334) 2024-02-19T21:06:22.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.178+0000 7f860903e700 1 -- 172.21.15.163:0/1702776962 learned_addr learned my addr 172.21.15.163:0/1702776962 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:22.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 10 client.?.objecter ms_handle_connect 0x7f8604100780 2024-02-19T21:06:22.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:22.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 10 client.?.objecter ms_handle_connect 0x7f86041042c0 2024-02-19T21:06:22.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:22.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 10 client.?.objecter ms_handle_connect 0x7f86040fd980 2024-02-19T21:06:22.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:22.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395121732 0 0) 0x7f86041172c0 con 0x7f8604100780 2024-02-19T21:06:22.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f0003030 con 0x7f8604100780 2024-02-19T21:06:22.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504673785 0 0) 0x7f86041184c0 con 0x7f86040fd980 2024-02-19T21:06:22.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86041172c0 con 0x7f86040fd980 2024-02-19T21:06:22.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605875997 0 0) 0x7f86041196c0 con 0x7f86041042c0 2024-02-19T21:06:22.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86041184c0 con 0x7f86041042c0 2024-02-19T21:06:22.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2973741023 0 0) 0x7f85f0003030 con 0x7f8604100780 2024-02-19T21:06:22.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86041196c0 con 0x7f8604100780 2024-02-19T21:06:22.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f85f4000ed0 con 0x7f8604100780 2024-02-19T21:06:22.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3405668860 0 0) 0x7f86041172c0 con 0x7f86040fd980 2024-02-19T21:06:22.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85f0003030 con 0x7f86040fd980 2024-02-19T21:06:22.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f86000019d0 con 0x7f86040fd980 2024-02-19T21:06:22.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 575360889 0 0) 0x7f86041196c0 con 0x7f8604100780 2024-02-19T21:06:22.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 >> v1:172.21.15.163:6790/0 conn(0x7f86041042c0 legacy=0x7f8604115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:22.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 >> v1:172.21.15.193:6789/0 conn(0x7f86040fd980 legacy=0x7f86040f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:22.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860411a8c0 con 0x7f8604100780 2024-02-19T21:06:22.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86041198b0 con 0x7f8604100780 2024-02-19T21:06:22.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f860baa3700 10 client.22500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:22.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8604119d20 con 0x7f8604100780 2024-02-19T21:06:22.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.179+0000 7f860baa3700 1 librados: init done 2024-02-19T21:06:22.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.181+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f85f4001110 con 0x7f8604100780 2024-02-19T21:06:22.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.181+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f85f40037b0 con 0x7f8604100780 2024-02-19T21:06:22.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f85f4003970 con 0x7f8604100780 2024-02-19T21:06:22.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f85e8004f00 con 0x7f8604100780 2024-02-19T21:06:22.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(201..201 src has 43..201) v4 ==== 9074+0+0 (unknown 2178571341 0 0) 0x7f85f407a250 con 0x7f8604100780 2024-02-19T21:06:22.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 osd_map(201..201 src has 43..201) v4 2024-02-19T21:06:22.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f85faf78700 3 client.22500.objecter handle_osd_map got epochs [201,201] > 0 2024-02-19T21:06:22.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f85faf78700 3 client.22500.objecter handle_osd_map decoding full epoch 201 2024-02-19T21:06:22.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f85faf78700 20 client.22500.objecter dump_active .. 0 homeless 2024-02-19T21:06:22.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v201) v1 ==== 54+0+16 (unknown 4031702312 0 2420000192) 0x7f85f404d310 con 0x7f8604100780 2024-02-19T21:06:22.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f85faf78700 10 client.22500.objecter ms_handle_connect 0x7f85f0063b70 2024-02-19T21:06:22.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f860baa3700 10 client.22500.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:22.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 1 2024-02-19T21:06:22.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e80051a0 osd=0 3 2024-02-19T21:06:22.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e80051a0 osd=0 3 2024-02-19T21:06:22.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e80051a0 osd=0 3 2024-02-19T21:06:22.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 1 2024-02-19T21:06:22.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e80051a0 osd=0 3 2024-02-19T21:06:22.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 0 1 2024-02-19T21:06:22.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e80051a0 osd=0 4 2024-02-19T21:06:22.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f860baa3700 10 client.22500.objecter _send_command 1 2024-02-19T21:06:22.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.182+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85e8004950 con 0x7f85e8004530 2024-02-19T21:06:22.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.183+0000 7f85faf78700 10 client.22500.objecter ms_handle_connect 0x7f85e8004530 2024-02-19T21:06:22.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.183+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f85e8004950 con 0x7f85e8004530 2024-02-19T21:06:22.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.183+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:06:22.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.183+0000 7f85faf78700 10 client.22500.objecter _finish_command 1 = system:0 2024-02-19T21:06:22.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.183+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e80051a0 osd=0 4 2024-02-19T21:06:22.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.183+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 0 1 2024-02-19T21:06:22.181 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:22.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.208+0000 7f860baa3700 10 client.22500.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:22.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.208+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 2 2024-02-19T21:06:22.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.208+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e80051a0 osd=0 3 2024-02-19T21:06:22.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.208+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e80051a0 osd=0 3 2024-02-19T21:06:22.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.208+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e80051a0 osd=0 3 2024-02-19T21:06:22.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.208+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 2 2024-02-19T21:06:22.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.208+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e80051a0 osd=0 3 2024-02-19T21:06:22.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.208+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 0 2 2024-02-19T21:06:22.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.208+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e80051a0 osd=0 4 2024-02-19T21:06:22.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.208+0000 7f860baa3700 10 client.22500.objecter _send_command 2 2024-02-19T21:06:22.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.208+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f85e8004d30 con 0x7f85e8004530 2024-02-19T21:06:22.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.216+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f85e8004d30 con 0x7f85e8004530 2024-02-19T21:06:22.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.216+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:22.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.216+0000 7f85faf78700 10 client.22500.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.216+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e80051a0 osd=0 4 2024-02-19T21:06:22.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.216+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 0 2 2024-02-19T21:06:22.215 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:06:22.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.220+0000 7f860baa3700 10 client.22500.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:22.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.220+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 3 2024-02-19T21:06:22.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.220+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8003fe0 osd=1 3 2024-02-19T21:06:22.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.220+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8003fe0 osd=1 3 2024-02-19T21:06:22.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.220+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8003fe0 osd=1 3 2024-02-19T21:06:22.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.220+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 3 2024-02-19T21:06:22.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.220+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e8003fe0 osd=1 3 2024-02-19T21:06:22.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.220+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 1 3 2024-02-19T21:06:22.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.220+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8003fe0 osd=1 4 2024-02-19T21:06:22.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.220+0000 7f860baa3700 10 client.22500.objecter _send_command 3 2024-02-19T21:06:22.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.220+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85e8010410 con 0x7f85e8004150 2024-02-19T21:06:22.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.221+0000 7f85faf78700 10 client.22500.objecter ms_handle_connect 0x7f85e8004150 2024-02-19T21:06:22.219 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:06:22.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.223+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f85e8010410 con 0x7f85e8004150 2024-02-19T21:06:22.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.223+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:06:22.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.223+0000 7f85faf78700 10 client.22500.objecter _finish_command 3 = system:0 2024-02-19T21:06:22.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.223+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e8003fe0 osd=1 4 2024-02-19T21:06:22.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.223+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 1 3 2024-02-19T21:06:22.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.239+0000 7f860baa3700 10 client.22500.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:22.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.239+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 4 2024-02-19T21:06:22.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.239+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8003fe0 osd=1 3 2024-02-19T21:06:22.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.239+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8003fe0 osd=1 3 2024-02-19T21:06:22.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.239+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8003fe0 osd=1 3 2024-02-19T21:06:22.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.239+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 4 2024-02-19T21:06:22.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.239+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e8003fe0 osd=1 3 2024-02-19T21:06:22.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.239+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 1 4 2024-02-19T21:06:22.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.239+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8003fe0 osd=1 4 2024-02-19T21:06:22.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.239+0000 7f860baa3700 10 client.22500.objecter _send_command 4 2024-02-19T21:06:22.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.239+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f85e80018b0 con 0x7f85e8004150 2024-02-19T21:06:22.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f85e80018b0 con 0x7f85e8004150 2024-02-19T21:06:22.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:22.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f85faf78700 10 client.22500.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e8003fe0 osd=1 4 2024-02-19T21:06:22.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 1 4 2024-02-19T21:06:22.236 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:06:22.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f860baa3700 10 client.22500.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:22.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 5 2024-02-19T21:06:22.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8003630 osd=2 3 2024-02-19T21:06:22.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8003630 osd=2 3 2024-02-19T21:06:22.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8003630 osd=2 3 2024-02-19T21:06:22.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 5 2024-02-19T21:06:22.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e8003630 osd=2 3 2024-02-19T21:06:22.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 2 5 2024-02-19T21:06:22.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8003630 osd=2 4 2024-02-19T21:06:22.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f860baa3700 10 client.22500.objecter _send_command 5 2024-02-19T21:06:22.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85e80110e0 con 0x7f85e8010580 2024-02-19T21:06:22.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.241+0000 7f85faf78700 10 client.22500.objecter ms_handle_connect 0x7f85e8010580 2024-02-19T21:06:22.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.242+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f85e80110e0 con 0x7f85e8010580 2024-02-19T21:06:22.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.242+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:06:22.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.242+0000 7f85faf78700 10 client.22500.objecter _finish_command 5 = system:0 2024-02-19T21:06:22.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.242+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e8003630 osd=2 4 2024-02-19T21:06:22.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.242+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 2 5 2024-02-19T21:06:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.260+0000 7f860baa3700 10 client.22500.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.260+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 6 2024-02-19T21:06:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.260+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8003630 osd=2 3 2024-02-19T21:06:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.260+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8003630 osd=2 3 2024-02-19T21:06:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.260+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8003630 osd=2 3 2024-02-19T21:06:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.260+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 6 2024-02-19T21:06:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.260+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e8003630 osd=2 3 2024-02-19T21:06:22.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.260+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 2 6 2024-02-19T21:06:22.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.260+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8003630 osd=2 4 2024-02-19T21:06:22.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.260+0000 7f860baa3700 10 client.22500.objecter _send_command 6 2024-02-19T21:06:22.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.260+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f85e8012ba0 con 0x7f85e8010580 2024-02-19T21:06:22.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.261+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f85e8012ba0 con 0x7f85e8010580 2024-02-19T21:06:22.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.261+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:22.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.261+0000 7f85faf78700 10 client.22500.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.261+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e8003630 osd=2 4 2024-02-19T21:06:22.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.261+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 2 6 2024-02-19T21:06:22.257 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:06:22.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.262+0000 7f860baa3700 10 client.22500.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:22.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.262+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 7 2024-02-19T21:06:22.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.262+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8003330 osd=3 3 2024-02-19T21:06:22.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.262+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8003330 osd=3 3 2024-02-19T21:06:22.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.262+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8003330 osd=3 3 2024-02-19T21:06:22.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.262+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 7 2024-02-19T21:06:22.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.262+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e8003330 osd=3 3 2024-02-19T21:06:22.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.262+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 3 7 2024-02-19T21:06:22.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.262+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8003330 osd=3 4 2024-02-19T21:06:22.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.262+0000 7f860baa3700 10 client.22500.objecter _send_command 7 2024-02-19T21:06:22.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.262+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85e8019e40 con 0x7f85e8016290 2024-02-19T21:06:22.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.262+0000 7f85faf78700 10 client.22500.objecter ms_handle_connect 0x7f85e8016290 2024-02-19T21:06:22.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.263+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f85e8019e40 con 0x7f85e8016290 2024-02-19T21:06:22.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.263+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:06:22.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.263+0000 7f85faf78700 10 client.22500.objecter _finish_command 7 = system:0 2024-02-19T21:06:22.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.263+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e8003330 osd=3 4 2024-02-19T21:06:22.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.263+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 3 7 2024-02-19T21:06:22.261 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:06:22.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.277+0000 7f860baa3700 10 client.22500.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:22.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.278+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 8 2024-02-19T21:06:22.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.278+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8003330 osd=3 3 2024-02-19T21:06:22.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.278+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8003330 osd=3 3 2024-02-19T21:06:22.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.278+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8003330 osd=3 3 2024-02-19T21:06:22.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.278+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 8 2024-02-19T21:06:22.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.278+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e8003330 osd=3 3 2024-02-19T21:06:22.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.278+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 3 8 2024-02-19T21:06:22.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.278+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8003330 osd=3 4 2024-02-19T21:06:22.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.278+0000 7f860baa3700 10 client.22500.objecter _send_command 8 2024-02-19T21:06:22.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.278+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f85e801b160 con 0x7f85e8016290 2024-02-19T21:06:22.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f85e801b160 con 0x7f85e8016290 2024-02-19T21:06:22.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:22.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f85faf78700 10 client.22500.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e8003330 osd=3 4 2024-02-19T21:06:22.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 3 8 2024-02-19T21:06:22.275 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:06:22.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f860baa3700 10 client.22500.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:22.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 9 2024-02-19T21:06:22.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e801b160 osd=4 3 2024-02-19T21:06:22.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e801b160 osd=4 3 2024-02-19T21:06:22.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e801b160 osd=4 3 2024-02-19T21:06:22.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 9 2024-02-19T21:06:22.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e801b160 osd=4 3 2024-02-19T21:06:22.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 4 9 2024-02-19T21:06:22.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e801b160 osd=4 4 2024-02-19T21:06:22.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f860baa3700 10 client.22500.objecter _send_command 9 2024-02-19T21:06:22.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.279+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85e801e7f0 con 0x7f85e801c310 2024-02-19T21:06:22.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.280+0000 7f85faf78700 10 client.22500.objecter ms_handle_connect 0x7f85e801c310 2024-02-19T21:06:22.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.281+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f85e801e7f0 con 0x7f85e801c310 2024-02-19T21:06:22.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.281+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:06:22.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.281+0000 7f85faf78700 10 client.22500.objecter _finish_command 9 = system:0 2024-02-19T21:06:22.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.281+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e801b160 osd=4 4 2024-02-19T21:06:22.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.281+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 4 9 2024-02-19T21:06:22.290 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:22.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.296+0000 7f860baa3700 10 client.22500.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:22.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.296+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 10 2024-02-19T21:06:22.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.296+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e801b160 osd=4 3 2024-02-19T21:06:22.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.296+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e801b160 osd=4 3 2024-02-19T21:06:22.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.296+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e801b160 osd=4 3 2024-02-19T21:06:22.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.296+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 10 2024-02-19T21:06:22.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.296+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e801b160 osd=4 3 2024-02-19T21:06:22.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.296+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 4 10 2024-02-19T21:06:22.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.296+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e801b160 osd=4 4 2024-02-19T21:06:22.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.296+0000 7f860baa3700 10 client.22500.objecter _send_command 10 2024-02-19T21:06:22.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.296+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f85e801e7f0 con 0x7f85e801c310 2024-02-19T21:06:22.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.297+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f85e801e7f0 con 0x7f85e801c310 2024-02-19T21:06:22.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.297+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:22.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.298+0000 7f85faf78700 10 client.22500.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.298+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e801b160 osd=4 4 2024-02-19T21:06:22.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.298+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 4 10 2024-02-19T21:06:22.293 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:06:22.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.298+0000 7f860baa3700 10 client.22500.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:22.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.298+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 11 2024-02-19T21:06:22.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.298+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e801e7f0 osd=5 3 2024-02-19T21:06:22.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.298+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e801e7f0 osd=5 3 2024-02-19T21:06:22.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.298+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e801e7f0 osd=5 3 2024-02-19T21:06:22.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.298+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 11 2024-02-19T21:06:22.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.298+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e801e7f0 osd=5 3 2024-02-19T21:06:22.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.298+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 5 11 2024-02-19T21:06:22.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.298+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e801e7f0 osd=5 4 2024-02-19T21:06:22.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.298+0000 7f860baa3700 10 client.22500.objecter _send_command 11 2024-02-19T21:06:22.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.298+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85e8020ba0 con 0x7f85e801eea0 2024-02-19T21:06:22.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.298+0000 7f85faf78700 10 client.22500.objecter ms_handle_connect 0x7f85e801eea0 2024-02-19T21:06:22.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.299+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f85e8020ba0 con 0x7f85e801eea0 2024-02-19T21:06:22.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.299+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:06:22.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.299+0000 7f85faf78700 10 client.22500.objecter _finish_command 11 = system:0 2024-02-19T21:06:22.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.299+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e801e7f0 osd=5 4 2024-02-19T21:06:22.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.299+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 5 11 2024-02-19T21:06:22.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.315+0000 7f860baa3700 10 client.22500.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:22.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.315+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 12 2024-02-19T21:06:22.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.315+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e801e7f0 osd=5 3 2024-02-19T21:06:22.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.315+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e801e7f0 osd=5 3 2024-02-19T21:06:22.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.315+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e801e7f0 osd=5 3 2024-02-19T21:06:22.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.315+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 12 2024-02-19T21:06:22.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.315+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e801e7f0 osd=5 3 2024-02-19T21:06:22.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.315+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 5 12 2024-02-19T21:06:22.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.315+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e801e7f0 osd=5 4 2024-02-19T21:06:22.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.315+0000 7f860baa3700 10 client.22500.objecter _send_command 12 2024-02-19T21:06:22.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.315+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f85e8020ba0 con 0x7f85e801eea0 2024-02-19T21:06:22.311 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:06:22.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f85e8020ba0 con 0x7f85e801eea0 2024-02-19T21:06:22.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:22.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f85faf78700 10 client.22500.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e801e7f0 osd=5 4 2024-02-19T21:06:22.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 5 12 2024-02-19T21:06:22.312 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:06:22.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f860baa3700 10 client.22500.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:22.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 13 2024-02-19T21:06:22.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8020ba0 osd=6 3 2024-02-19T21:06:22.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8020ba0 osd=6 3 2024-02-19T21:06:22.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8020ba0 osd=6 3 2024-02-19T21:06:22.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 13 2024-02-19T21:06:22.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e8020ba0 osd=6 3 2024-02-19T21:06:22.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 6 13 2024-02-19T21:06:22.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8020ba0 osd=6 4 2024-02-19T21:06:22.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f860baa3700 10 client.22500.objecter _send_command 13 2024-02-19T21:06:22.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.317+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85e8027140 con 0x7f85e8023630 2024-02-19T21:06:22.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.318+0000 7f85faf78700 10 client.22500.objecter ms_handle_connect 0x7f85e8023630 2024-02-19T21:06:22.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.318+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f85e8027140 con 0x7f85e8023630 2024-02-19T21:06:22.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.318+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:06:22.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.318+0000 7f85faf78700 10 client.22500.objecter _finish_command 13 = system:0 2024-02-19T21:06:22.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.318+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e8020ba0 osd=6 4 2024-02-19T21:06:22.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.318+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 6 13 2024-02-19T21:06:22.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:22.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:06:22.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.336+0000 7f860baa3700 10 client.22500.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:22.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.336+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 14 2024-02-19T21:06:22.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.336+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8020ba0 osd=6 3 2024-02-19T21:06:22.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.336+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8020ba0 osd=6 3 2024-02-19T21:06:22.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.336+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8020ba0 osd=6 3 2024-02-19T21:06:22.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.336+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 14 2024-02-19T21:06:22.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.336+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e8020ba0 osd=6 3 2024-02-19T21:06:22.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.336+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 6 14 2024-02-19T21:06:22.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.336+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8020ba0 osd=6 4 2024-02-19T21:06:22.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.336+0000 7f860baa3700 10 client.22500.objecter _send_command 14 2024-02-19T21:06:22.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.336+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f85e8027140 con 0x7f85e8023630 2024-02-19T21:06:22.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f85e8027140 con 0x7f85e8023630 2024-02-19T21:06:22.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:22.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f85faf78700 10 client.22500.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e8020ba0 osd=6 4 2024-02-19T21:06:22.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 6 14 2024-02-19T21:06:22.334 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:06:22.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f860baa3700 10 client.22500.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:22.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 15 2024-02-19T21:06:22.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8027140 osd=7 3 2024-02-19T21:06:22.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8027140 osd=7 3 2024-02-19T21:06:22.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8027140 osd=7 3 2024-02-19T21:06:22.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 15 2024-02-19T21:06:22.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e8027140 osd=7 3 2024-02-19T21:06:22.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 7 15 2024-02-19T21:06:22.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8027140 osd=7 4 2024-02-19T21:06:22.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f860baa3700 10 client.22500.objecter _send_command 15 2024-02-19T21:06:22.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.338+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85e802b430 con 0x7f85e80277c0 2024-02-19T21:06:22.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.339+0000 7f85faf78700 10 client.22500.objecter ms_handle_connect 0x7f85e80277c0 2024-02-19T21:06:22.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.339+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f85e802b430 con 0x7f85e80277c0 2024-02-19T21:06:22.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.339+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:06:22.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.339+0000 7f85faf78700 10 client.22500.objecter _finish_command 15 = system:0 2024-02-19T21:06:22.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.339+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e8027140 osd=7 4 2024-02-19T21:06:22.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.339+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 7 15 2024-02-19T21:06:22.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.354+0000 7f860baa3700 10 client.22500.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:22.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.354+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign -1 16 2024-02-19T21:06:22.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.354+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8027140 osd=7 3 2024-02-19T21:06:22.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.354+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8027140 osd=7 3 2024-02-19T21:06:22.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.354+0000 7f860baa3700 20 client.22500.objecter _get_session s=0x7f85e8027140 osd=7 3 2024-02-19T21:06:22.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.354+0000 7f860baa3700 15 client.22500.objecter _session_command_op_remove -1 16 2024-02-19T21:06:22.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.354+0000 7f860baa3700 20 client.22500.objecter get_session s=0x7f85e8027140 osd=7 3 2024-02-19T21:06:22.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.354+0000 7f860baa3700 15 client.22500.objecter _session_command_op_assign 7 16 2024-02-19T21:06:22.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.354+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8027140 osd=7 4 2024-02-19T21:06:22.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.354+0000 7f860baa3700 10 client.22500.objecter _send_command 16 2024-02-19T21:06:22.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.354+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f85e802cbe0 con 0x7f85e80277c0 2024-02-19T21:06:22.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f85faf78700 1 -- 172.21.15.163:0/1702776962 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f85e802cbe0 con 0x7f85e80277c0 2024-02-19T21:06:22.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f85faf78700 10 client.22500.objecter ms_dispatch 0x7f8604001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:22.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f85faf78700 10 client.22500.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f85faf78700 20 client.22500.objecter put_session s=0x7f85e8027140 osd=7 4 2024-02-19T21:06:22.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f85faf78700 15 client.22500.objecter _session_command_op_remove 7 16 2024-02-19T21:06:22.353 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:06:22.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 10 librados: watch_flush enter 2024-02-19T21:06:22.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 10 librados: watch_flush exit 2024-02-19T21:06:22.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 10 client.22500.objecter close_session for osd.0 2024-02-19T21:06:22.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 >> v1:172.21.15.163:6801/1559418115 conn(0x7f85e8004530 legacy=0x7f85e8002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:22.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e80051a0 osd=0 1 2024-02-19T21:06:22.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 10 client.22500.objecter close_session for osd.1 2024-02-19T21:06:22.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 >> v1:172.21.15.163:6805/3125047432 conn(0x7f85e8004150 legacy=0x7f85e800ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8003fe0 osd=1 1 2024-02-19T21:06:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 10 client.22500.objecter close_session for osd.2 2024-02-19T21:06:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 >> v1:172.21.15.163:6809/1563158487 conn(0x7f85e8010580 legacy=0x7f85e80109d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8003630 osd=2 1 2024-02-19T21:06:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 10 client.22500.objecter close_session for osd.3 2024-02-19T21:06:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 >> v1:172.21.15.163:6813/2112279769 conn(0x7f85e8016290 legacy=0x7f85e8018680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8003330 osd=3 1 2024-02-19T21:06:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 10 client.22500.objecter close_session for osd.4 2024-02-19T21:06:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 >> v1:172.21.15.193:6800/3129716965 conn(0x7f85e801c310 legacy=0x7f85e8011250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860a040700 1 -- 172.21.15.163:0/1702776962 reap_dead start 2024-02-19T21:06:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e801b160 osd=4 1 2024-02-19T21:06:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 10 client.22500.objecter close_session for osd.5 2024-02-19T21:06:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 >> v1:172.21.15.193:6804/805161991 conn(0x7f85e801eea0 legacy=0x7f85e801f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860a040700 1 -- 172.21.15.163:0/1702776962 reap_dead start 2024-02-19T21:06:22.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e801e7f0 osd=5 1 2024-02-19T21:06:22.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 10 client.22500.objecter close_session for osd.6 2024-02-19T21:06:22.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 >> v1:172.21.15.193:6808/787604615 conn(0x7f85e8023630 legacy=0x7f85e8025a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:22.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8020ba0 osd=6 1 2024-02-19T21:06:22.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 10 client.22500.objecter close_session for osd.7 2024-02-19T21:06:22.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 >> v1:172.21.15.193:6812/157927432 conn(0x7f85e80277c0 legacy=0x7f85e8029bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:22.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860a040700 1 -- 172.21.15.163:0/1702776962 reap_dead start 2024-02-19T21:06:22.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 20 client.22500.objecter put_session s=0x7f85e8027140 osd=7 1 2024-02-19T21:06:22.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 20 client.22500.objecter shutdown clearing up homeless session... 2024-02-19T21:06:22.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 10 client.22500.objecter successfully canceled tick 2024-02-19T21:06:22.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.356+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 >> v1:172.21.15.163:6800/1552514681 conn(0x7f85f0063b70 legacy=0x7f85f0066030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:22.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.357+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 >> v1:172.21.15.163:6789/0 conn(0x7f8604100780 legacy=0x7f86040fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:22.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.357+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 shutdown_connections 2024-02-19T21:06:22.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.357+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 >> 172.21.15.163:0/1702776962 conn(0x7f86040f35a0 msgr2=0x7f8604107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:22.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.357+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 shutdown_connections 2024-02-19T21:06:22.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.357+0000 7f860baa3700 1 -- 172.21.15.163:0/1702776962 wait complete. 2024-02-19T21:06:22.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:22.357+0000 7f860baa3700 1 librados: shutdown 2024-02-19T21:06:22.357 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.357 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.357 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.357 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.357 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.357 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.357 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.357 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:06:22.416 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:22.453 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:22.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:20.872475+0000 osd.5 (osd.5) 98 : cluster [DBG] 6.d scrub starts 2024-02-19T21:06:22.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:20.875398+0000 osd.5 (osd.5) 99 : cluster [DBG] 6.d scrub ok 2024-02-19T21:06:22.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:21.140578+0000 mgr.y (mgr.14550) 235 : cluster [DBG] pgmap v272: 260 pgs: 32 creating+peering, 1 peering, 11 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 199 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 767 B/s wr, 2 op/s 2024-02-19T21:06:22.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:21.703472+0000 osd.4 (osd.4) 111 : cluster [DBG] 16.2 scrub starts 2024-02-19T21:06:22.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:21.704712+0000 osd.4 (osd.4) 112 : cluster [DBG] 16.2 scrub ok 2024-02-19T21:06:22.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:22.183640+0000 mon.a (mon.0) 1454 : audit [DBG] from='client.? v1:172.21.15.163:0/1702776962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:22.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:22.195039+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi163-83845-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi163-83845-29"}]': finished 2024-02-19T21:06:22.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:22.195209+0000 mon.a (mon.0) 1456 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-17"}]': finished 2024-02-19T21:06:22.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:22.198173+0000 mon.a (mon.0) 1457 : audit [INF] from='client.47968 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi163-83854-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:22.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:22.198243+0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-02-19T21:06:22.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:22.206029+0000 mon.b (mon.1) 316 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-17", "mode": "writeback"}]: dispatch 2024-02-19T21:06:22.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:22.207598+0000 mon.a (mon.0) 1459 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-17", "mode": "writeback"}]: dispatch 2024-02-19T21:06:22.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:22.302969+0000 mon.c (mon.2) 209 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:22.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:20.872475+0000 osd.5 (osd.5) 98 : cluster [DBG] 6.d scrub starts 2024-02-19T21:06:22.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:20.875398+0000 osd.5 (osd.5) 99 : cluster [DBG] 6.d scrub ok 2024-02-19T21:06:22.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:21.140578+0000 mgr.y (mgr.14550) 235 : cluster [DBG] pgmap v272: 260 pgs: 32 creating+peering, 1 peering, 11 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 199 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 767 B/s wr, 2 op/s 2024-02-19T21:06:22.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:21.703472+0000 osd.4 (osd.4) 111 : cluster [DBG] 16.2 scrub starts 2024-02-19T21:06:22.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:21.704712+0000 osd.4 (osd.4) 112 : cluster [DBG] 16.2 scrub ok 2024-02-19T21:06:22.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:22.183640+0000 mon.a (mon.0) 1454 : audit [DBG] from='client.? v1:172.21.15.163:0/1702776962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:22.195039+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi163-83845-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi163-83845-29"}]': finished 2024-02-19T21:06:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:22.195209+0000 mon.a (mon.0) 1456 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-17"}]': finished 2024-02-19T21:06:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:22.198173+0000 mon.a (mon.0) 1457 : audit [INF] from='client.47968 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi163-83854-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:22.198243+0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-02-19T21:06:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:22.206029+0000 mon.b (mon.1) 316 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-17", "mode": "writeback"}]: dispatch 2024-02-19T21:06:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:22.207598+0000 mon.a (mon.0) 1459 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-17", "mode": "writeback"}]: dispatch 2024-02-19T21:06:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:22.302969+0000 mon.c (mon.2) 209 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:22.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:20.872475+0000 osd.5 (osd.5) 98 : cluster [DBG] 6.d scrub starts 2024-02-19T21:06:22.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:20.875398+0000 osd.5 (osd.5) 99 : cluster [DBG] 6.d scrub ok 2024-02-19T21:06:22.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:21.140578+0000 mgr.y (mgr.14550) 235 : cluster [DBG] pgmap v272: 260 pgs: 32 creating+peering, 1 peering, 11 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 199 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 767 B/s wr, 2 op/s 2024-02-19T21:06:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:21.703472+0000 osd.4 (osd.4) 111 : cluster [DBG] 16.2 scrub starts 2024-02-19T21:06:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:21.704712+0000 osd.4 (osd.4) 112 : cluster [DBG] 16.2 scrub ok 2024-02-19T21:06:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:22.183640+0000 mon.a (mon.0) 1454 : audit [DBG] from='client.? v1:172.21.15.163:0/1702776962' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:22.195039+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi163-83845-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi163-83845-29"}]': finished 2024-02-19T21:06:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:22.195209+0000 mon.a (mon.0) 1456 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-17"}]': finished 2024-02-19T21:06:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:22.198173+0000 mon.a (mon.0) 1457 : audit [INF] from='client.47968 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi163-83854-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:22.198243+0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-02-19T21:06:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:22.206029+0000 mon.b (mon.1) 316 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-17", "mode": "writeback"}]: dispatch 2024-02-19T21:06:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:22.207598+0000 mon.a (mon.0) 1459 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-17", "mode": "writeback"}]: dispatch 2024-02-19T21:06:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:22.302969+0000 mon.c (mon.2) 209 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:22.984 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:22.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:22.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:23.088 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:23.120 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:23.141 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:23.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:23.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:23.245 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:23.276 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:23.492 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:06:23] "GET /metrics HTTP/1.1" 200 191002 "" "Prometheus/2.33.4" 2024-02-19T21:06:23.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:21.817422+0000 osd.0 (osd.0) 115 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:06:23.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:21.819101+0000 osd.0 (osd.0) 116 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:06:23.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:23.189906+0000 mon.a (mon.0) 1460 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:23.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:23.193864+0000 mon.a (mon.0) 1461 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-17", "mode": "writeback"}]': finished 2024-02-19T21:06:23.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:23.193983+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-02-19T21:06:23.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:23.265982+0000 mon.b (mon.1) 317 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:23.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:23.303692+0000 mon.c (mon.2) 210 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:23.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:23.476996+0000 mon.a (mon.0) 1463 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:23.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:21.817422+0000 osd.0 (osd.0) 115 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:06:23.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:21.819101+0000 osd.0 (osd.0) 116 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:06:23.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:23.189906+0000 mon.a (mon.0) 1460 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:23.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:23.193864+0000 mon.a (mon.0) 1461 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-17", "mode": "writeback"}]': finished 2024-02-19T21:06:23.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:23.193983+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-02-19T21:06:23.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:23.265982+0000 mon.b (mon.1) 317 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:23.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:23.303692+0000 mon.c (mon.2) 210 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:23.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:23.476996+0000 mon.a (mon.0) 1463 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:23.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:21.817422+0000 osd.0 (osd.0) 115 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:06:23.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:21.819101+0000 osd.0 (osd.0) 116 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:06:23.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:23.189906+0000 mon.a (mon.0) 1460 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:23.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:23.193864+0000 mon.a (mon.0) 1461 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-17", "mode": "writeback"}]': finished 2024-02-19T21:06:23.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:23.193983+0000 mon.a (mon.0) 1462 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-02-19T21:06:23.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:23.265982+0000 mon.b (mon.1) 317 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:23.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:23.303692+0000 mon.c (mon.2) 210 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:23.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:23.476996+0000 mon.a (mon.0) 1463 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:24.124 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:24.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:24.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:24.159 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:24.228 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:06:24.260 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:24.281 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:06:24.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:24.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:24.384 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:24.412 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:24.433 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:24.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:24.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:24.536 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:24.564 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:24.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:23.141153+0000 mgr.y (mgr.14550) 236 : cluster [DBG] pgmap v275: 300 pgs: 32 creating+peering, 1 peering, 11 active+clean+snaptrim, 40 unknown, 17 active+clean+snaptrim_wait, 199 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 767 B/s wr, 2 op/s 2024-02-19T21:06:24.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:23.272406+0000 osd.2 (osd.2) 91 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:06:24.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:23.273270+0000 osd.2 (osd.2) 92 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:06:24.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:24.097687+0000 mon.a (mon.0) 1464 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:06:24.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:24.182285+0000 mon.a (mon.0) 1465 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:24.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:24.182359+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-02-19T21:06:24.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:24.186189+0000 mon.c (mon.2) 211 : audit [INF] from='client.? v1:172.21.15.163:0/2281084990' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi163-83854-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:24.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:24.186462+0000 mon.b (mon.1) 318 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17"}]: dispatch 2024-02-19T21:06:24.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:24.186883+0000 mon.a (mon.0) 1467 : audit [INF] from='client.49160 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi163-83854-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:24.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:24.186916+0000 mon.b (mon.1) 319 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:06:24.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:24.187598+0000 mon.a (mon.0) 1468 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17"}]: dispatch 2024-02-19T21:06:24.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:24.191916+0000 mon.a (mon.0) 1469 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi163-83845-29"}]: dispatch 2024-02-19T21:06:24.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:24.304453+0000 mon.c (mon.2) 212 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:24.584 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:24.587 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:24.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:24.688 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:06:24.715 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:24.733 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:06:24.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:24.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:06:24.837 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:06:24.855 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:23.141153+0000 mgr.y (mgr.14550) 236 : cluster [DBG] pgmap v275: 300 pgs: 32 creating+peering, 1 peering, 11 active+clean+snaptrim, 40 unknown, 17 active+clean+snaptrim_wait, 199 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 767 B/s wr, 2 op/s 2024-02-19T21:06:24.855 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:23.272406+0000 osd.2 (osd.2) 91 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:06:24.855 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:23.273270+0000 osd.2 (osd.2) 92 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:06:24.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:24.097687+0000 mon.a (mon.0) 1464 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:06:24.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:24.182285+0000 mon.a (mon.0) 1465 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:24.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:24.182359+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-02-19T21:06:24.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:24.186189+0000 mon.c (mon.2) 211 : audit [INF] from='client.? v1:172.21.15.163:0/2281084990' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi163-83854-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:24.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:24.186462+0000 mon.b (mon.1) 318 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17"}]: dispatch 2024-02-19T21:06:24.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:24.186883+0000 mon.a (mon.0) 1467 : audit [INF] from='client.49160 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi163-83854-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:24.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:24.186916+0000 mon.b (mon.1) 319 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:06:24.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:24.187598+0000 mon.a (mon.0) 1468 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17"}]: dispatch 2024-02-19T21:06:24.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:24.191916+0000 mon.a (mon.0) 1469 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi163-83845-29"}]: dispatch 2024-02-19T21:06:24.856 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:24.304453+0000 mon.c (mon.2) 212 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:24.856 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:23.141153+0000 mgr.y (mgr.14550) 236 : cluster [DBG] pgmap v275: 300 pgs: 32 creating+peering, 1 peering, 11 active+clean+snaptrim, 40 unknown, 17 active+clean+snaptrim_wait, 199 active+clean; 11 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 767 B/s wr, 2 op/s 2024-02-19T21:06:24.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:23.272406+0000 osd.2 (osd.2) 91 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:06:24.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:23.273270+0000 osd.2 (osd.2) 92 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:06:24.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:24.097687+0000 mon.a (mon.0) 1464 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:06:24.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:24.182285+0000 mon.a (mon.0) 1465 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:24.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:24.182359+0000 mon.a (mon.0) 1466 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-02-19T21:06:24.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:24.186189+0000 mon.c (mon.2) 211 : audit [INF] from='client.? v1:172.21.15.163:0/2281084990' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi163-83854-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:24.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:24.186462+0000 mon.b (mon.1) 318 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17"}]: dispatch 2024-02-19T21:06:24.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:24.186883+0000 mon.a (mon.0) 1467 : audit [INF] from='client.49160 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi163-83854-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:24.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:24.186916+0000 mon.b (mon.1) 319 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:06:24.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:24.187598+0000 mon.a (mon.0) 1468 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17"}]: dispatch 2024-02-19T21:06:24.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:24.191916+0000 mon.a (mon.0) 1469 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi163-83845-29"}]: dispatch 2024-02-19T21:06:24.857 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:24.304453+0000 mon.c (mon.2) 212 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:24.863 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:24.882 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:06:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:24.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:24.985 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:25.005 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:06:25.014 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:25.094 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:25.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:25.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:25.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.130+0000 7f052b460700 1 -- 172.21.15.163:0/3667658542 >> v1:172.21.15.163:6790/0 conn(0x7f05241042b0 legacy=0x7f0524106730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:25.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.130+0000 7f052b460700 1 -- 172.21.15.163:0/3667658542 shutdown_connections 2024-02-19T21:06:25.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.130+0000 7f052b460700 1 -- 172.21.15.163:0/3667658542 >> 172.21.15.163:0/3667658542 conn(0x7f05240f3590 msgr2=0x7f05240f59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:25.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.131+0000 7f052b460700 1 -- 172.21.15.163:0/3667658542 shutdown_connections 2024-02-19T21:06:25.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.131+0000 7f052b460700 1 -- 172.21.15.163:0/3667658542 wait complete. 2024-02-19T21:06:25.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.131+0000 7f052b460700 1 librados: starting msgr at 2024-02-19T21:06:25.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.131+0000 7f052b460700 1 librados: starting objecter 2024-02-19T21:06:25.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.131+0000 7f052b460700 1 Processor -- start 2024-02-19T21:06:25.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.131+0000 7f052b460700 1 -- start start 2024-02-19T21:06:25.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.131+0000 7f052b460700 1 librados: setting wanted keys 2024-02-19T21:06:25.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.131+0000 7f052b460700 1 librados: calling monclient init 2024-02-19T21:06:25.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f052b460700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f05241173a0 con 0x7f0524100770 2024-02-19T21:06:25.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f052b460700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f05241185a0 con 0x7f05241042b0 2024-02-19T21:06:25.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f052b460700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f05241197a0 con 0x7f05240fd970 2024-02-19T21:06:25.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f05291fc700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f05240fd970 0x7f05240f9950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38872/0 (socket says 172.21.15.163:38872) 2024-02-19T21:06:25.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f05291fc700 1 -- 172.21.15.163:0/3759817161 learned_addr learned my addr 172.21.15.163:0/3759817161 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:25.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 10 client.?.objecter ms_handle_connect 0x7f05240fd970 2024-02-19T21:06:25.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:25.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 10 client.?.objecter ms_handle_connect 0x7f0524100770 2024-02-19T21:06:25.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:25.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 10 client.?.objecter ms_handle_connect 0x7f05241042b0 2024-02-19T21:06:25.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:25.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28244989 0 0) 0x7f05241173a0 con 0x7f0524100770 2024-02-19T21:06:25.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0510003800 con 0x7f0524100770 2024-02-19T21:06:25.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183523894 0 0) 0x7f05241185a0 con 0x7f05241042b0 2024-02-19T21:06:25.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05241173a0 con 0x7f05241042b0 2024-02-19T21:06:25.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1180401272 0 0) 0x7f05241173a0 con 0x7f05241042b0 2024-02-19T21:06:25.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05241185a0 con 0x7f05241042b0 2024-02-19T21:06:25.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f051c003150 con 0x7f05241042b0 2024-02-19T21:06:25.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2599487844 0 0) 0x7f0510003800 con 0x7f0524100770 2024-02-19T21:06:25.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05241173a0 con 0x7f0524100770 2024-02-19T21:06:25.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883297956 0 0) 0x7f05241197a0 con 0x7f05240fd970 2024-02-19T21:06:25.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0510003800 con 0x7f05240fd970 2024-02-19T21:06:25.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.132+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f05140018c0 con 0x7f0524100770 2024-02-19T21:06:25.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.133+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 694564619 0 0) 0x7f05241185a0 con 0x7f05241042b0 2024-02-19T21:06:25.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.133+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 >> v1:172.21.15.163:6790/0 conn(0x7f05240fd970 legacy=0x7f05240f9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:25.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.133+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 >> v1:172.21.15.163:6789/0 conn(0x7f0524100770 legacy=0x7f05240fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:25.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.133+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052411a9a0 con 0x7f05241042b0 2024-02-19T21:06:25.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.133+0000 7f052b460700 1 -- 172.21.15.163:0/3759817161 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0524117590 con 0x7f05241042b0 2024-02-19T21:06:25.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.133+0000 7f052b460700 10 client.47986.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:25.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.133+0000 7f052b460700 1 -- 172.21.15.163:0/3759817161 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0524117a00 con 0x7f05241042b0 2024-02-19T21:06:25.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.133+0000 7f052b460700 1 librados: init done 2024-02-19T21:06:25.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.133+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f051c0014c0 con 0x7f05241042b0 2024-02-19T21:06:25.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.133+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f051c0039f0 con 0x7f05241042b0 2024-02-19T21:06:25.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.133+0000 7f052b460700 1 -- 172.21.15.163:0/3759817161 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0508004f20 con 0x7f05241042b0 2024-02-19T21:06:25.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.134+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f051c0168a0 con 0x7f05241042b0 2024-02-19T21:06:25.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.134+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(204..204 src has 43..204) v4 ==== 9077+0+0 (unknown 3101402489 0 0) 0x7f051c076ed0 con 0x7f05241042b0 2024-02-19T21:06:25.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.134+0000 7f051a7fc700 10 client.47986.objecter ms_dispatch 0x7f0524001550 osd_map(204..204 src has 43..204) v4 2024-02-19T21:06:25.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.134+0000 7f051a7fc700 3 client.47986.objecter handle_osd_map got epochs [204,204] > 0 2024-02-19T21:06:25.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.134+0000 7f051a7fc700 3 client.47986.objecter handle_osd_map decoding full epoch 204 2024-02-19T21:06:25.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.134+0000 7f051a7fc700 20 client.47986.objecter dump_active .. 0 homeless 2024-02-19T21:06:25.133 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.135+0000 7f051a7fc700 10 client.47986.objecter ms_handle_connect 0x7f0510063cf0 2024-02-19T21:06:25.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.139+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f051c04a040 con 0x7f05241042b0 2024-02-19T21:06:25.198 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:25.237 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:25.335 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:25.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:25.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:25.361+0000 7f052b460700 1 -- 172.21.15.163:0/3759817161 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 0, 3]} v 0) v1 -- 0x7f05080059e0 con 0x7f05241042b0 2024-02-19T21:06:25.440 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:06:25.467 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:25.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:23.723010+0000 osd.1 (osd.1) 121 : cluster [DBG] 8.1 scrub starts 2024-02-19T21:06:25.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:23.726476+0000 osd.1 (osd.1) 122 : cluster [DBG] 8.1 scrub ok 2024-02-19T21:06:25.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:24.677203+0000 mon.a (mon.0) 1470 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:06:25.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:24.732462+0000 osd.4 (osd.4) 113 : cluster [DBG] 8.2 scrub starts 2024-02-19T21:06:25.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:24.733293+0000 osd.4 (osd.4) 114 : cluster [DBG] 8.2 scrub ok 2024-02-19T21:06:25.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:24.960796+0000 mon.b (mon.1) 320 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:06:25.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:24.961586+0000 mon.b (mon.1) 321 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:06:25.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:24.968795+0000 mon.a (mon.0) 1471 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:06:25.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:25.142882+0000 mon.b (mon.1) 322 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-19T21:06:25.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:25.143910+0000 mon.a (mon.0) 1472 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-19T21:06:25.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:25.179063+0000 mon.a (mon.0) 1473 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:25.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:25.183404+0000 mon.a (mon.0) 1474 : audit [INF] from='client.49160 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi163-83854-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:25.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:25.183494+0000 mon.a (mon.0) 1475 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17"}]': finished 2024-02-19T21:06:25.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:25.183629+0000 mon.a (mon.0) 1476 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi163-83845-29"}]': finished 2024-02-19T21:06:25.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:25.183743+0000 mon.a (mon.0) 1477 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-19T21:06:25.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:25.183810+0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-02-19T21:06:25.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:25.194828+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi163-83845-29"}]: dispatch 2024-02-19T21:06:25.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:25.305352+0000 mon.c (mon.2) 213 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:25.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:25.362709+0000 mon.b (mon.1) 323 : audit [INF] from='client.? v1:172.21.15.163:0/3759817161' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 0, 3]}]: dispatch 2024-02-19T21:06:25.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:25.363098+0000 mon.a (mon.0) 1480 : audit [INF] from='client.47986 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 0, 3]}]: dispatch 2024-02-19T21:06:25.890 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:23.723010+0000 osd.1 (osd.1) 121 : cluster [DBG] 8.1 scrub starts 2024-02-19T21:06:25.890 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:23.726476+0000 osd.1 (osd.1) 122 : cluster [DBG] 8.1 scrub ok 2024-02-19T21:06:25.890 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:24.677203+0000 mon.a (mon.0) 1470 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:06:25.890 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:24.732462+0000 osd.4 (osd.4) 113 : cluster [DBG] 8.2 scrub starts 2024-02-19T21:06:25.891 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:24.733293+0000 osd.4 (osd.4) 114 : cluster [DBG] 8.2 scrub ok 2024-02-19T21:06:25.891 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:24.960796+0000 mon.b (mon.1) 320 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:06:25.891 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:24.961586+0000 mon.b (mon.1) 321 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:06:25.891 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:24.968795+0000 mon.a (mon.0) 1471 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:06:25.891 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:25.142882+0000 mon.b (mon.1) 322 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-19T21:06:25.891 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:25.143910+0000 mon.a (mon.0) 1472 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-19T21:06:25.891 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:25.179063+0000 mon.a (mon.0) 1473 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:25.891 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:25.183404+0000 mon.a (mon.0) 1474 : audit [INF] from='client.49160 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi163-83854-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:25.891 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:25.183494+0000 mon.a (mon.0) 1475 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17"}]': finished 2024-02-19T21:06:25.891 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:25.183629+0000 mon.a (mon.0) 1476 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi163-83845-29"}]': finished 2024-02-19T21:06:25.891 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:25.183743+0000 mon.a (mon.0) 1477 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-19T21:06:25.891 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:25.183810+0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-02-19T21:06:25.891 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:25.194828+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi163-83845-29"}]: dispatch 2024-02-19T21:06:25.891 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:25.305352+0000 mon.c (mon.2) 213 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:25.892 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:25.362709+0000 mon.b (mon.1) 323 : audit [INF] from='client.? v1:172.21.15.163:0/3759817161' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 0, 3]}]: dispatch 2024-02-19T21:06:25.892 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:25.363098+0000 mon.a (mon.0) 1480 : audit [INF] from='client.47986 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 0, 3]}]: dispatch 2024-02-19T21:06:25.892 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:23.723010+0000 osd.1 (osd.1) 121 : cluster [DBG] 8.1 scrub starts 2024-02-19T21:06:25.892 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:23.726476+0000 osd.1 (osd.1) 122 : cluster [DBG] 8.1 scrub ok 2024-02-19T21:06:25.892 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:24.677203+0000 mon.a (mon.0) 1470 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:06:25.892 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:24.732462+0000 osd.4 (osd.4) 113 : cluster [DBG] 8.2 scrub starts 2024-02-19T21:06:25.892 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:24.733293+0000 osd.4 (osd.4) 114 : cluster [DBG] 8.2 scrub ok 2024-02-19T21:06:25.892 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:24.960796+0000 mon.b (mon.1) 320 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:06:25.892 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:24.961586+0000 mon.b (mon.1) 321 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:06:25.892 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:24.968795+0000 mon.a (mon.0) 1471 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:06:25.892 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:25.142882+0000 mon.b (mon.1) 322 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-19T21:06:25.892 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:25.143910+0000 mon.a (mon.0) 1472 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-02-19T21:06:25.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:25.179063+0000 mon.a (mon.0) 1473 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:25.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:25.183404+0000 mon.a (mon.0) 1474 : audit [INF] from='client.49160 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi163-83854-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:25.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:25.183494+0000 mon.a (mon.0) 1475 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-17"}]': finished 2024-02-19T21:06:25.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:25.183629+0000 mon.a (mon.0) 1476 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi163-83845-29"}]': finished 2024-02-19T21:06:25.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:25.183743+0000 mon.a (mon.0) 1477 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-02-19T21:06:25.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:25.183810+0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-02-19T21:06:25.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:25.194828+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi163-83845-29"}]: dispatch 2024-02-19T21:06:25.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:25.305352+0000 mon.c (mon.2) 213 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:25.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:25.362709+0000 mon.b (mon.1) 323 : audit [INF] from='client.? v1:172.21.15.163:0/3759817161' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 0, 3]}]: dispatch 2024-02-19T21:06:25.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:25.363098+0000 mon.a (mon.0) 1480 : audit [INF] from='client.47986 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 0, 3]}]: dispatch 2024-02-19T21:06:26.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:26.187+0000 7f051a7fc700 1 -- 172.21.15.163:0/3759817161 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 0, 3]}]=0 set 5.1c pg_upmap mapping to [7,0,3] v206) v1 ==== 129+0+0 (unknown 3593762874 0 0) 0x7f051c0484b0 con 0x7f05241042b0 2024-02-19T21:06:26.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:26.191+0000 7f052b460700 10 librados: watch_flush enter 2024-02-19T21:06:26.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:26.191+0000 7f052b460700 10 librados: watch_flush exit 2024-02-19T21:06:26.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:26.191+0000 7f052b460700 20 client.47986.objecter shutdown clearing up homeless session... 2024-02-19T21:06:26.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:26.191+0000 7f052b460700 10 client.47986.objecter successfully canceled tick 2024-02-19T21:06:26.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:26.192+0000 7f052b460700 1 -- 172.21.15.163:0/3759817161 >> v1:172.21.15.163:6800/1552514681 conn(0x7f0510063cf0 legacy=0x7f05100661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:26.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:26.192+0000 7f052b460700 1 -- 172.21.15.163:0/3759817161 >> v1:172.21.15.193:6789/0 conn(0x7f05241042b0 legacy=0x7f0524115a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:26.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:26.192+0000 7f052b460700 1 -- 172.21.15.163:0/3759817161 shutdown_connections 2024-02-19T21:06:26.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:26.192+0000 7f052b460700 1 -- 172.21.15.163:0/3759817161 >> 172.21.15.163:0/3759817161 conn(0x7f05240f3590 msgr2=0x7f0524107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:26.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:26.198+0000 7f052b460700 1 -- 172.21.15.163:0/3759817161 shutdown_connections 2024-02-19T21:06:26.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:26.204+0000 7f052b460700 1 -- 172.21.15.163:0/3759817161 wait complete. 2024-02-19T21:06:26.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:26.212+0000 7f052b460700 1 librados: shutdown 2024-02-19T21:06:26.212 INFO:teuthology.orchestra.run.smithi163.stderr:set 5.1c pg_upmap mapping to [7,0,3] 2024-02-19T21:06:26.220 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:06:26.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:26.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:06:26.247 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:26.329 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:06:26.360 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:26.381 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:06:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:26.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:26.485 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:06:26.513 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:26.661 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:06:26.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:26.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:06:26.765 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:26.791 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:24.699302+0000 osd.1 (osd.1) 123 : cluster [DBG] 5.19 deep-scrub starts 2024-02-19T21:06:26.791 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:24.702193+0000 osd.1 (osd.1) 124 : cluster [DBG] 5.19 deep-scrub ok 2024-02-19T21:06:26.791 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:24.835764+0000 osd.5 (osd.5) 100 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:06:26.791 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:24.838655+0000 osd.5 (osd.5) 101 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:06:26.791 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:25.142481+0000 mgr.y (mgr.14550) 237 : cluster [DBG] pgmap v278: 291 pgs: 32 creating+peering, 10 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 233 active+clean; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-19T21:06:26.791 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:26.186538+0000 mon.a (mon.0) 1481 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi163-83845-29"}]': finished 2024-02-19T21:06:26.792 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:26.186674+0000 mon.a (mon.0) 1482 : audit [INF] from='client.47986 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 0, 3]}]': finished 2024-02-19T21:06:26.792 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:26.186749+0000 mon.a (mon.0) 1483 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-02-19T21:06:26.792 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:26.230642+0000 mon.b (mon.1) 324 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:26.792 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:26.306001+0000 mon.c (mon.2) 214 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:26.792 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:24.699302+0000 osd.1 (osd.1) 123 : cluster [DBG] 5.19 deep-scrub starts 2024-02-19T21:06:26.792 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:24.702193+0000 osd.1 (osd.1) 124 : cluster [DBG] 5.19 deep-scrub ok 2024-02-19T21:06:26.792 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:24.835764+0000 osd.5 (osd.5) 100 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:06:26.792 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:24.838655+0000 osd.5 (osd.5) 101 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:06:26.792 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:25.142481+0000 mgr.y (mgr.14550) 237 : cluster [DBG] pgmap v278: 291 pgs: 32 creating+peering, 10 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 233 active+clean; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-19T21:06:26.792 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:26.186538+0000 mon.a (mon.0) 1481 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi163-83845-29"}]': finished 2024-02-19T21:06:26.793 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:26.186674+0000 mon.a (mon.0) 1482 : audit [INF] from='client.47986 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 0, 3]}]': finished 2024-02-19T21:06:26.793 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:26.186749+0000 mon.a (mon.0) 1483 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-02-19T21:06:26.793 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:26.230642+0000 mon.b (mon.1) 324 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:26.793 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:26.306001+0000 mon.c (mon.2) 214 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:26.794 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:27.021 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:06:27.021 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:06:27.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:24.699302+0000 osd.1 (osd.1) 123 : cluster [DBG] 5.19 deep-scrub starts 2024-02-19T21:06:27.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:24.702193+0000 osd.1 (osd.1) 124 : cluster [DBG] 5.19 deep-scrub ok 2024-02-19T21:06:27.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:24.835764+0000 osd.5 (osd.5) 100 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:06:27.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:24.838655+0000 osd.5 (osd.5) 101 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:06:27.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:25.142481+0000 mgr.y (mgr.14550) 237 : cluster [DBG] pgmap v278: 291 pgs: 32 creating+peering, 10 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 233 active+clean; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-19T21:06:27.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:26.186538+0000 mon.a (mon.0) 1481 : audit [INF] from='client.? v1:172.21.15.163:0/833565757' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi163-83845-29"}]': finished 2024-02-19T21:06:27.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:26.186674+0000 mon.a (mon.0) 1482 : audit [INF] from='client.47986 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1c", "id": [7, 0, 3]}]': finished 2024-02-19T21:06:27.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:26.186749+0000 mon.a (mon.0) 1483 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-02-19T21:06:27.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:26.230642+0000 mon.b (mon.1) 324 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:27.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:26.306001+0000 mon.c (mon.2) 214 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:27.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:25.515471+0000 osd.7 (osd.7) 109 : cluster [DBG] 2.27 scrub starts 2024-02-19T21:06:27.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:25.516523+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.27 scrub ok 2024-02-19T21:06:27.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:25.685544+0000 osd.3 (osd.3) 109 : cluster [DBG] 2.23 scrub starts 2024-02-19T21:06:27.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:25.686506+0000 osd.3 (osd.3) 110 : cluster [DBG] 2.23 scrub ok 2024-02-19T21:06:27.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:25.782466+0000 osd.0 (osd.0) 117 : cluster [DBG] 6.18 deep-scrub starts 2024-02-19T21:06:27.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:25.785361+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.18 deep-scrub ok 2024-02-19T21:06:27.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:26.711223+0000 mon.a (mon.0) 1484 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:27.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:26.712036+0000 mon.b (mon.1) 325 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:27.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:26.718644+0000 mon.a (mon.0) 1485 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:27.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:26.719127+0000 mon.b (mon.1) 326 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi163-83845-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:27.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:26.719371+0000 mon.a (mon.0) 1486 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi163-83845-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:27.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:26.774992+0000 osd.6 (osd.6) 99 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:06:27.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:26.776958+0000 osd.6 (osd.6) 100 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:06:27.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:27.192998+0000 mon.a (mon.0) 1487 : audit [INF] from='client.47998 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi163-83845-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:27.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:27.193069+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-02-19T21:06:27.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:27.200675+0000 mon.c (mon.2) 215 : audit [INF] from='client.? v1:172.21.15.163:0/89904398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi163-83854-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:27.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:27.201753+0000 mon.a (mon.0) 1489 : audit [INF] from='client.49169 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi163-83854-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:27.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:27.201765+0000 mon.b (mon.1) 327 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi163-83845-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:27.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:27.202848+0000 mon.a (mon.0) 1490 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi163-83845-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:27.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:27.202902+0000 mon.b (mon.1) 328 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:27.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:27.203635+0000 mon.a (mon.0) 1491 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:27.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:27.306694+0000 mon.c (mon.2) 216 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:27.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:25.515471+0000 osd.7 (osd.7) 109 : cluster [DBG] 2.27 scrub starts 2024-02-19T21:06:27.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:25.516523+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.27 scrub ok 2024-02-19T21:06:27.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:25.685544+0000 osd.3 (osd.3) 109 : cluster [DBG] 2.23 scrub starts 2024-02-19T21:06:27.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:25.686506+0000 osd.3 (osd.3) 110 : cluster [DBG] 2.23 scrub ok 2024-02-19T21:06:27.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:25.782466+0000 osd.0 (osd.0) 117 : cluster [DBG] 6.18 deep-scrub starts 2024-02-19T21:06:27.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:25.785361+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.18 deep-scrub ok 2024-02-19T21:06:27.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:26.711223+0000 mon.a (mon.0) 1484 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:27.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:26.712036+0000 mon.b (mon.1) 325 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:27.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:26.718644+0000 mon.a (mon.0) 1485 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:27.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:26.719127+0000 mon.b (mon.1) 326 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi163-83845-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:27.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:26.719371+0000 mon.a (mon.0) 1486 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi163-83845-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:27.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:26.774992+0000 osd.6 (osd.6) 99 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:06:27.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:26.776958+0000 osd.6 (osd.6) 100 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:06:27.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:27.192998+0000 mon.a (mon.0) 1487 : audit [INF] from='client.47998 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi163-83845-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:27.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:27.193069+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-02-19T21:06:27.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:27.200675+0000 mon.c (mon.2) 215 : audit [INF] from='client.? v1:172.21.15.163:0/89904398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi163-83854-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:27.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:27.201753+0000 mon.a (mon.0) 1489 : audit [INF] from='client.49169 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi163-83854-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:27.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:27.201765+0000 mon.b (mon.1) 327 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi163-83845-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:27.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:27.202848+0000 mon.a (mon.0) 1490 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi163-83845-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:27.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:27.202902+0000 mon.b (mon.1) 328 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:27.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:27.203635+0000 mon.a (mon.0) 1491 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:27.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:27.306694+0000 mon.c (mon.2) 216 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:28.066 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:06:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:06:28.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:25.515471+0000 osd.7 (osd.7) 109 : cluster [DBG] 2.27 scrub starts 2024-02-19T21:06:28.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:25.516523+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.27 scrub ok 2024-02-19T21:06:28.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:25.685544+0000 osd.3 (osd.3) 109 : cluster [DBG] 2.23 scrub starts 2024-02-19T21:06:28.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:25.686506+0000 osd.3 (osd.3) 110 : cluster [DBG] 2.23 scrub ok 2024-02-19T21:06:28.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:25.782466+0000 osd.0 (osd.0) 117 : cluster [DBG] 6.18 deep-scrub starts 2024-02-19T21:06:28.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:25.785361+0000 osd.0 (osd.0) 118 : cluster [DBG] 6.18 deep-scrub ok 2024-02-19T21:06:28.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:26.711223+0000 mon.a (mon.0) 1484 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:28.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:26.712036+0000 mon.b (mon.1) 325 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:28.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:26.718644+0000 mon.a (mon.0) 1485 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:28.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:26.719127+0000 mon.b (mon.1) 326 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi163-83845-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:28.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:26.719371+0000 mon.a (mon.0) 1486 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi163-83845-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:28.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:26.774992+0000 osd.6 (osd.6) 99 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:06:28.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:26.776958+0000 osd.6 (osd.6) 100 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:06:28.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:27.192998+0000 mon.a (mon.0) 1487 : audit [INF] from='client.47998 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi163-83845-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:28.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:27.193069+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-02-19T21:06:28.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:27.200675+0000 mon.c (mon.2) 215 : audit [INF] from='client.? v1:172.21.15.163:0/89904398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi163-83854-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:28.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:27.201753+0000 mon.a (mon.0) 1489 : audit [INF] from='client.49169 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi163-83854-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:28.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:27.201765+0000 mon.b (mon.1) 327 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi163-83845-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:28.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:27.202848+0000 mon.a (mon.0) 1490 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi163-83845-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:28.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:27.202902+0000 mon.b (mon.1) 328 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:28.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:27.203635+0000 mon.a (mon.0) 1491 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:28.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:27.306694+0000 mon.c (mon.2) 216 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:28.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:26.927064+0000 osd.2 (osd.2) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:06:28.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:26.927276+0000 osd.2 (osd.2) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:06:28.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:27.143047+0000 mgr.y (mgr.14550) 238 : cluster [DBG] pgmap v281: 227 pgs: 10 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 201 active+clean; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:06:28.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:27.302496+0000 osd.2 (osd.2) 95 : cluster [DBG] 5.a scrub starts 2024-02-19T21:06:28.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:27.303303+0000 osd.2 (osd.2) 96 : cluster [DBG] 5.a scrub ok 2024-02-19T21:06:28.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:27.516745+0000 osd.7 (osd.7) 111 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:06:28.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:27.520207+0000 osd.7 (osd.7) 112 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:06:28.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:27.743576+0000 osd.6 (osd.6) 101 : cluster [DBG] 16.d scrub starts 2024-02-19T21:06:28.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:27.744831+0000 osd.6 (osd.6) 102 : cluster [DBG] 16.d scrub ok 2024-02-19T21:06:28.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:28.192398+0000 mon.a (mon.0) 1492 : audit [INF] from='client.49169 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi163-83854-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:28.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:28.307380+0000 mon.c (mon.2) 217 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:28.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:28.436221+0000 mon.a (mon.0) 1493 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:28.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:28.436330+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-02-19T21:06:28.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:28.437933+0000 mon.b (mon.1) 329 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:28.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:28.439674+0000 mon.a (mon.0) 1495 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:28.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:26.927064+0000 osd.2 (osd.2) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:06:28.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:26.927276+0000 osd.2 (osd.2) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:06:28.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:27.143047+0000 mgr.y (mgr.14550) 238 : cluster [DBG] pgmap v281: 227 pgs: 10 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 201 active+clean; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:06:28.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:27.302496+0000 osd.2 (osd.2) 95 : cluster [DBG] 5.a scrub starts 2024-02-19T21:06:28.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:27.303303+0000 osd.2 (osd.2) 96 : cluster [DBG] 5.a scrub ok 2024-02-19T21:06:28.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:27.516745+0000 osd.7 (osd.7) 111 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:06:28.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:27.520207+0000 osd.7 (osd.7) 112 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:06:28.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:27.743576+0000 osd.6 (osd.6) 101 : cluster [DBG] 16.d scrub starts 2024-02-19T21:06:28.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:27.744831+0000 osd.6 (osd.6) 102 : cluster [DBG] 16.d scrub ok 2024-02-19T21:06:28.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:28.192398+0000 mon.a (mon.0) 1492 : audit [INF] from='client.49169 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi163-83854-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:28.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:28.307380+0000 mon.c (mon.2) 217 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:28.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:28.436221+0000 mon.a (mon.0) 1493 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:28.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:28.436330+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-02-19T21:06:28.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:28.437933+0000 mon.b (mon.1) 329 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:28.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:28.439674+0000 mon.a (mon.0) 1495 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:29.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:26.927064+0000 osd.2 (osd.2) 93 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:06:29.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:26.927276+0000 osd.2 (osd.2) 94 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:06:29.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:27.143047+0000 mgr.y (mgr.14550) 238 : cluster [DBG] pgmap v281: 227 pgs: 10 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 201 active+clean; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:06:29.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:27.302496+0000 osd.2 (osd.2) 95 : cluster [DBG] 5.a scrub starts 2024-02-19T21:06:29.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:27.303303+0000 osd.2 (osd.2) 96 : cluster [DBG] 5.a scrub ok 2024-02-19T21:06:29.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:27.516745+0000 osd.7 (osd.7) 111 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:06:29.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:27.520207+0000 osd.7 (osd.7) 112 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:06:29.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:27.743576+0000 osd.6 (osd.6) 101 : cluster [DBG] 16.d scrub starts 2024-02-19T21:06:29.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:27.744831+0000 osd.6 (osd.6) 102 : cluster [DBG] 16.d scrub ok 2024-02-19T21:06:29.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:28.192398+0000 mon.a (mon.0) 1492 : audit [INF] from='client.49169 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi163-83854-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:29.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:28.307380+0000 mon.c (mon.2) 217 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:29.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:28.436221+0000 mon.a (mon.0) 1493 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:29.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:28.436330+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-02-19T21:06:29.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:28.437933+0000 mon.b (mon.1) 329 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:29.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:28.439674+0000 mon.a (mon.0) 1495 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:29.092 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-19T21:06:29.145 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:29.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:29.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:29.249 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:29.280 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:29.379 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:29.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:29.483 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:06:29.515 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:29.536 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:06:29.538 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:29.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:06:29.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.632+0000 7f2acb74f700 1 -- 172.21.15.163:0/1910984747 >> v1:172.21.15.163:6790/0 conn(0x7f2ac41042c0 legacy=0x7f2ac4106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:29.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.633+0000 7f2acb74f700 1 -- 172.21.15.163:0/1910984747 shutdown_connections 2024-02-19T21:06:29.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.633+0000 7f2acb74f700 1 -- 172.21.15.163:0/1910984747 >> 172.21.15.163:0/1910984747 conn(0x7f2ac40f35a0 msgr2=0x7f2ac40f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:29.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.633+0000 7f2acb74f700 1 -- 172.21.15.163:0/1910984747 shutdown_connections 2024-02-19T21:06:29.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.633+0000 7f2acb74f700 1 -- 172.21.15.163:0/1910984747 wait complete. 2024-02-19T21:06:29.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.633+0000 7f2acb74f700 1 librados: starting msgr at 2024-02-19T21:06:29.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.633+0000 7f2acb74f700 1 librados: starting objecter 2024-02-19T21:06:29.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.633+0000 7f2acb74f700 1 Processor -- start 2024-02-19T21:06:29.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.633+0000 7f2acb74f700 1 -- start start 2024-02-19T21:06:29.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.633+0000 7f2acb74f700 1 librados: setting wanted keys 2024-02-19T21:06:29.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.633+0000 7f2acb74f700 1 librados: calling monclient init 2024-02-19T21:06:29.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2acb74f700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ac406cd10 con 0x7f2ac40fd980 2024-02-19T21:06:29.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2acb74f700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ac406cea0 con 0x7f2ac4100780 2024-02-19T21:06:29.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2acb74f700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ac411fc50 con 0x7f2ac41042c0 2024-02-19T21:06:29.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2ac8cea700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f2ac4100780 0x7f2ac40fcd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:55806/0 (socket says 172.21.15.163:55806) 2024-02-19T21:06:29.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2ac8cea700 1 -- 172.21.15.163:0/4199443827 learned_addr learned my addr 172.21.15.163:0/4199443827 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:29.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2aba7fc700 10 client.?.objecter ms_handle_connect 0x7f2ac4100780 2024-02-19T21:06:29.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2aba7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:29.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998448541 0 0) 0x7f2ac406cea0 con 0x7f2ac4100780 2024-02-19T21:06:29.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ab0003030 con 0x7f2ac4100780 2024-02-19T21:06:29.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2aba7fc700 10 client.?.objecter ms_handle_connect 0x7f2ac41042c0 2024-02-19T21:06:29.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2aba7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:29.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2aba7fc700 10 client.?.objecter ms_handle_connect 0x7f2ac40fd980 2024-02-19T21:06:29.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2aba7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:29.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148859818 0 0) 0x7f2ac411fc50 con 0x7f2ac41042c0 2024-02-19T21:06:29.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac406cea0 con 0x7f2ac41042c0 2024-02-19T21:06:29.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134090004 0 0) 0x7f2ac406cd10 con 0x7f2ac40fd980 2024-02-19T21:06:29.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac411fc50 con 0x7f2ac40fd980 2024-02-19T21:06:29.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3414321788 0 0) 0x7f2ab0003030 con 0x7f2ac4100780 2024-02-19T21:06:29.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ac406cd10 con 0x7f2ac4100780 2024-02-19T21:06:29.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.634+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2ab4001940 con 0x7f2ac4100780 2024-02-19T21:06:29.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.635+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 714501476 0 0) 0x7f2ac406cd10 con 0x7f2ac4100780 2024-02-19T21:06:29.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.635+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 >> v1:172.21.15.163:6790/0 conn(0x7f2ac41042c0 legacy=0x7f2ac411c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:29.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.635+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 >> v1:172.21.15.163:6789/0 conn(0x7f2ac40fd980 legacy=0x7f2ac40fc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:29.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.635+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac4120df0 con 0x7f2ac4100780 2024-02-19T21:06:29.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.635+0000 7f2acb74f700 1 -- 172.21.15.163:0/4199443827 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac411fe40 con 0x7f2ac4100780 2024-02-19T21:06:29.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.635+0000 7f2acb74f700 10 client.48010.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:29.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.635+0000 7f2acb74f700 1 -- 172.21.15.163:0/4199443827 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac4120330 con 0x7f2ac4100780 2024-02-19T21:06:29.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.635+0000 7f2acb74f700 1 librados: init done 2024-02-19T21:06:29.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.635+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f2ab4001c40 con 0x7f2ac4100780 2024-02-19T21:06:29.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.635+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2ab4016bd0 con 0x7f2ac4100780 2024-02-19T21:06:29.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.636+0000 7f2acb74f700 1 -- 172.21.15.163:0/4199443827 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa8004f50 con 0x7f2ac4100780 2024-02-19T21:06:29.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.636+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f2ab4002b80 con 0x7f2ac4100780 2024-02-19T21:06:29.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.636+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(209..209 src has 43..209) v4 ==== 9192+0+0 (unknown 2286716164 0 0) 0x7f2ab408a280 con 0x7f2ac4100780 2024-02-19T21:06:29.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.636+0000 7f2aba7fc700 10 client.48010.objecter ms_dispatch 0x7f2ac4001550 osd_map(209..209 src has 43..209) v4 2024-02-19T21:06:29.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.636+0000 7f2aba7fc700 3 client.48010.objecter handle_osd_map got epochs [209,209] > 0 2024-02-19T21:06:29.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.636+0000 7f2aba7fc700 3 client.48010.objecter handle_osd_map decoding full epoch 209 2024-02-19T21:06:29.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.637+0000 7f2aba7fc700 20 client.48010.objecter dump_active .. 0 homeless 2024-02-19T21:06:29.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.637+0000 7f2aba7fc700 10 client.48010.objecter ms_handle_connect 0x7f2ab0063690 2024-02-19T21:06:29.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.641+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f2ab405d210 con 0x7f2ac4100780 2024-02-19T21:06:29.639 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:29.668 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:29.765 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:29.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:29.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:29.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:28.745156+0000 osd.4 (osd.4) 115 : cluster [DBG] 4.c scrub starts 2024-02-19T21:06:29.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:28.746966+0000 osd.4 (osd.4) 116 : cluster [DBG] 4.c scrub ok 2024-02-19T21:06:29.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:29.144012+0000 mon.b (mon.1) 330 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-19T21:06:29.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:29.145056+0000 mon.a (mon.0) 1496 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-19T21:06:29.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:29.181718+0000 mon.a (mon.0) 1497 : audit [INF] from='client.47998 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi163-83845-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi163-83845-30"}]': finished 2024-02-19T21:06:29.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:29.181830+0000 mon.a (mon.0) 1498 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:29.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:29.181888+0000 mon.a (mon.0) 1499 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-19T21:06:29.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:29.181941+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-02-19T21:06:29.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:29.194538+0000 mon.b (mon.1) 331 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-19"}]: dispatch 2024-02-19T21:06:29.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:29.202676+0000 mon.a (mon.0) 1501 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-19"}]: dispatch 2024-02-19T21:06:29.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:29.308118+0000 mon.c (mon.2) 218 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:29.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:29.832+0000 7f2acb74f700 1 -- 172.21.15.163:0/4199443827 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f2aa8005200 con 0x7f2ac4100780 2024-02-19T21:06:29.891 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:06:29.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:28.745156+0000 osd.4 (osd.4) 115 : cluster [DBG] 4.c scrub starts 2024-02-19T21:06:29.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:28.746966+0000 osd.4 (osd.4) 116 : cluster [DBG] 4.c scrub ok 2024-02-19T21:06:29.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:29.144012+0000 mon.b (mon.1) 330 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-19T21:06:29.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:29.145056+0000 mon.a (mon.0) 1496 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-19T21:06:29.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:29.181718+0000 mon.a (mon.0) 1497 : audit [INF] from='client.47998 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi163-83845-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi163-83845-30"}]': finished 2024-02-19T21:06:29.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:29.181830+0000 mon.a (mon.0) 1498 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:29.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:29.181888+0000 mon.a (mon.0) 1499 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-19T21:06:29.919 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:29.181941+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-02-19T21:06:29.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:29.194538+0000 mon.b (mon.1) 331 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-19"}]: dispatch 2024-02-19T21:06:29.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:29.202676+0000 mon.a (mon.0) 1501 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-19"}]: dispatch 2024-02-19T21:06:29.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:29.308118+0000 mon.c (mon.2) 218 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:29.920 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:28.745156+0000 osd.4 (osd.4) 115 : cluster [DBG] 4.c scrub starts 2024-02-19T21:06:29.920 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:28.746966+0000 osd.4 (osd.4) 116 : cluster [DBG] 4.c scrub ok 2024-02-19T21:06:29.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:29.144012+0000 mon.b (mon.1) 330 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-19T21:06:29.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:29.145056+0000 mon.a (mon.0) 1496 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-02-19T21:06:29.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:29.181718+0000 mon.a (mon.0) 1497 : audit [INF] from='client.47998 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi163-83845-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi163-83845-30"}]': finished 2024-02-19T21:06:29.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:29.181830+0000 mon.a (mon.0) 1498 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:29.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:29.181888+0000 mon.a (mon.0) 1499 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-02-19T21:06:29.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:29.181941+0000 mon.a (mon.0) 1500 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-02-19T21:06:29.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:29.194538+0000 mon.b (mon.1) 331 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-19"}]: dispatch 2024-02-19T21:06:29.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:29.202676+0000 mon.a (mon.0) 1501 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-19"}]: dispatch 2024-02-19T21:06:29.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:29.308118+0000 mon.c (mon.2) 218 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:29.921 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:30.066 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:06:30.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:30.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:06:30.101 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:30.171 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:30.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:30.193+0000 7f2aba7fc700 1 -- 172.21.15.163:0/4199443827 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v210) v1 ==== 87+0+0 (unknown 1633384497 0 0) 0x7f2ab405b680 con 0x7f2ac4100780 2024-02-19T21:06:30.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:30.196+0000 7f2acb74f700 10 librados: watch_flush enter 2024-02-19T21:06:30.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:30.196+0000 7f2acb74f700 10 librados: watch_flush exit 2024-02-19T21:06:30.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:30.196+0000 7f2acb74f700 20 client.48010.objecter shutdown clearing up homeless session... 2024-02-19T21:06:30.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:30.196+0000 7f2acb74f700 10 client.48010.objecter successfully canceled tick 2024-02-19T21:06:30.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:30.197+0000 7f2acb74f700 1 -- 172.21.15.163:0/4199443827 >> v1:172.21.15.163:6800/1552514681 conn(0x7f2ab0063690 legacy=0x7f2ab0065b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:30.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:30.197+0000 7f2acb74f700 1 -- 172.21.15.163:0/4199443827 >> v1:172.21.15.193:6789/0 conn(0x7f2ac4100780 legacy=0x7f2ac40fcd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:30.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:30.197+0000 7f2acb74f700 1 -- 172.21.15.163:0/4199443827 shutdown_connections 2024-02-19T21:06:30.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:30.197+0000 7f2acb74f700 1 -- 172.21.15.163:0/4199443827 >> 172.21.15.163:0/4199443827 conn(0x7f2ac40f35a0 msgr2=0x7f2ac40f64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:30.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:30.197+0000 7f2acb74f700 1 -- 172.21.15.163:0/4199443827 shutdown_connections 2024-02-19T21:06:30.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:30.198+0000 7f2acb74f700 1 -- 172.21.15.163:0/4199443827 wait complete. 2024-02-19T21:06:30.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:30.198+0000 7f2acb74f700 1 librados: shutdown 2024-02-19T21:06:30.194 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:06:30.213 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:30.234 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:30.236 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:30.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:30.337 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:06:30.366 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:30.566 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:06:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:06:30.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:29.143572+0000 mgr.y (mgr.14550) 239 : cluster [DBG] pgmap v284: 290 pgs: 10 active+clean+snaptrim, 64 unknown, 16 active+clean+snaptrim_wait, 200 active+clean; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:30.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:29.753789+0000 osd.3 (osd.3) 111 : cluster [DBG] 2.28 scrub starts 2024-02-19T21:06:30.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:29.755026+0000 osd.3 (osd.3) 112 : cluster [DBG] 2.28 scrub ok 2024-02-19T21:06:30.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:29.834135+0000 mon.b (mon.1) 332 : audit [INF] from='client.? v1:172.21.15.163:0/4199443827' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:06:30.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:29.834435+0000 mon.a (mon.0) 1502 : audit [INF] from='client.48010 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:06:30.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:30.178683+0000 mon.a (mon.0) 1503 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:06:30.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:30.185572+0000 mon.a (mon.0) 1504 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-19"}]': finished 2024-02-19T21:06:30.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:30.185724+0000 mon.a (mon.0) 1505 : audit [INF] from='client.48010 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:06:30.986 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:30.185779+0000 mon.a (mon.0) 1506 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-02-19T21:06:30.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:30.195003+0000 mon.b (mon.1) 333 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-19", "mode": "writeback"}]: dispatch 2024-02-19T21:06:30.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:30.195756+0000 mon.a (mon.0) 1507 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-19", "mode": "writeback"}]: dispatch 2024-02-19T21:06:30.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:30.199926+0000 mon.b (mon.1) 334 : audit [INF] from='client.? v1:172.21.15.163:0/83185378' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi163-83854-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:30.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:30.201018+0000 mon.a (mon.0) 1508 : audit [INF] from='client.48004 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi163-83854-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:30.987 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:30.309038+0000 mon.c (mon.2) 219 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:30.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:29.143572+0000 mgr.y (mgr.14550) 239 : cluster [DBG] pgmap v284: 290 pgs: 10 active+clean+snaptrim, 64 unknown, 16 active+clean+snaptrim_wait, 200 active+clean; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:30.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:29.753789+0000 osd.3 (osd.3) 111 : cluster [DBG] 2.28 scrub starts 2024-02-19T21:06:30.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:29.755026+0000 osd.3 (osd.3) 112 : cluster [DBG] 2.28 scrub ok 2024-02-19T21:06:30.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:29.834135+0000 mon.b (mon.1) 332 : audit [INF] from='client.? v1:172.21.15.163:0/4199443827' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:06:30.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:29.834435+0000 mon.a (mon.0) 1502 : audit [INF] from='client.48010 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:06:30.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:30.178683+0000 mon.a (mon.0) 1503 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:06:30.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:30.185572+0000 mon.a (mon.0) 1504 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-19"}]': finished 2024-02-19T21:06:30.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:30.185724+0000 mon.a (mon.0) 1505 : audit [INF] from='client.48010 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:06:30.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:30.185779+0000 mon.a (mon.0) 1506 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-02-19T21:06:30.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:30.195003+0000 mon.b (mon.1) 333 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-19", "mode": "writeback"}]: dispatch 2024-02-19T21:06:30.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:30.195756+0000 mon.a (mon.0) 1507 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-19", "mode": "writeback"}]: dispatch 2024-02-19T21:06:30.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:30.199926+0000 mon.b (mon.1) 334 : audit [INF] from='client.? v1:172.21.15.163:0/83185378' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi163-83854-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:30.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:30.201018+0000 mon.a (mon.0) 1508 : audit [INF] from='client.48004 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi163-83854-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:30.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:30.309038+0000 mon.c (mon.2) 219 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:31.018 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:06:31.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:31.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:06:31.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:29.143572+0000 mgr.y (mgr.14550) 239 : cluster [DBG] pgmap v284: 290 pgs: 10 active+clean+snaptrim, 64 unknown, 16 active+clean+snaptrim_wait, 200 active+clean; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:31.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:29.753789+0000 osd.3 (osd.3) 111 : cluster [DBG] 2.28 scrub starts 2024-02-19T21:06:31.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:29.755026+0000 osd.3 (osd.3) 112 : cluster [DBG] 2.28 scrub ok 2024-02-19T21:06:31.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:29.834135+0000 mon.b (mon.1) 332 : audit [INF] from='client.? v1:172.21.15.163:0/4199443827' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:06:31.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:29.834435+0000 mon.a (mon.0) 1502 : audit [INF] from='client.48010 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:06:31.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:30.178683+0000 mon.a (mon.0) 1503 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:06:31.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:30.185572+0000 mon.a (mon.0) 1504 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-19"}]': finished 2024-02-19T21:06:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:30.185724+0000 mon.a (mon.0) 1505 : audit [INF] from='client.48010 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:06:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:30.185779+0000 mon.a (mon.0) 1506 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-02-19T21:06:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:30.195003+0000 mon.b (mon.1) 333 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-19", "mode": "writeback"}]: dispatch 2024-02-19T21:06:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:30.195756+0000 mon.a (mon.0) 1507 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-19", "mode": "writeback"}]: dispatch 2024-02-19T21:06:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:30.199926+0000 mon.b (mon.1) 334 : audit [INF] from='client.? v1:172.21.15.163:0/83185378' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi163-83854-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:30.201018+0000 mon.a (mon.0) 1508 : audit [INF] from='client.48004 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi163-83854-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:30.309038+0000 mon.c (mon.2) 219 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:31.121 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:31.154 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:31.175 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:31.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:31.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:31.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.259+0000 7f79ab321700 1 -- 172.21.15.163:0/967060117 >> v1:172.21.15.163:6789/0 conn(0x7f79a41031b0 legacy=0x7f79a4105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:31.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.259+0000 7f79ab321700 1 -- 172.21.15.163:0/967060117 shutdown_connections 2024-02-19T21:06:31.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.259+0000 7f79ab321700 1 -- 172.21.15.163:0/967060117 >> 172.21.15.163:0/967060117 conn(0x7f79a40f2470 msgr2=0x7f79a40f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:31.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.259+0000 7f79ab321700 1 -- 172.21.15.163:0/967060117 shutdown_connections 2024-02-19T21:06:31.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.259+0000 7f79ab321700 1 -- 172.21.15.163:0/967060117 wait complete. 2024-02-19T21:06:31.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.259+0000 7f79ab321700 1 librados: starting msgr at 2024-02-19T21:06:31.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.259+0000 7f79ab321700 1 librados: starting objecter 2024-02-19T21:06:31.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f79ab321700 1 Processor -- start 2024-02-19T21:06:31.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f79ab321700 1 -- start start 2024-02-19T21:06:31.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f79ab321700 1 librados: setting wanted keys 2024-02-19T21:06:31.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f79ab321700 1 librados: calling monclient init 2024-02-19T21:06:31.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f79ab321700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f79a41194b0 con 0x7f79a41031b0 2024-02-19T21:06:31.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f79ab321700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f79a411a690 con 0x7f79a40ff670 2024-02-19T21:06:31.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f79ab321700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f79a411b890 con 0x7f79a40fc870 2024-02-19T21:06:31.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f79a90bd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f79a40fc870 0x7f79a40f8400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38958/0 (socket says 172.21.15.163:38958) 2024-02-19T21:06:31.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f79a90bd700 1 -- 172.21.15.163:0/1571310155 learned_addr learned my addr 172.21.15.163:0/1571310155 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:31.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 10 client.?.objecter ms_handle_connect 0x7f79a40fc870 2024-02-19T21:06:31.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:31.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 10 client.?.objecter ms_handle_connect 0x7f79a41031b0 2024-02-19T21:06:31.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:31.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 10 client.?.objecter ms_handle_connect 0x7f79a40ff670 2024-02-19T21:06:31.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:31.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862958467 0 0) 0x7f79a41194b0 con 0x7f79a41031b0 2024-02-19T21:06:31.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79900033f0 con 0x7f79a41031b0 2024-02-19T21:06:31.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490198858 0 0) 0x7f79a411b890 con 0x7f79a40fc870 2024-02-19T21:06:31.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79a41194b0 con 0x7f79a40fc870 2024-02-19T21:06:31.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274916822 0 0) 0x7f79a411a690 con 0x7f79a40ff670 2024-02-19T21:06:31.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79a411b890 con 0x7f79a40ff670 2024-02-19T21:06:31.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 726870449 0 0) 0x7f79900033f0 con 0x7f79a41031b0 2024-02-19T21:06:31.258 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79a411a690 con 0x7f79a41031b0 2024-02-19T21:06:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f799c002310 con 0x7f79a41031b0 2024-02-19T21:06:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2886837570 0 0) 0x7f79a41194b0 con 0x7f79a40fc870 2024-02-19T21:06:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79900033f0 con 0x7f79a40fc870 2024-02-19T21:06:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f79a0001900 con 0x7f79a40fc870 2024-02-19T21:06:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3276808751 0 0) 0x7f79a411b890 con 0x7f79a40ff670 2024-02-19T21:06:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.260+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79a41194b0 con 0x7f79a40ff670 2024-02-19T21:06:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.261+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f79940019d0 con 0x7f79a40ff670 2024-02-19T21:06:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.261+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4283053123 0 0) 0x7f79a411a690 con 0x7f79a41031b0 2024-02-19T21:06:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.261+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 >> v1:172.21.15.163:6790/0 conn(0x7f79a40fc870 legacy=0x7f79a40f8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.261+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 >> v1:172.21.15.193:6789/0 conn(0x7f79a40ff670 legacy=0x7f79a40f8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.261+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a411ca90 con 0x7f79a41031b0 2024-02-19T21:06:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.261+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a41196a0 con 0x7f79a41031b0 2024-02-19T21:06:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.261+0000 7f79ab321700 10 client.22527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.261+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a4119b90 con 0x7f79a41031b0 2024-02-19T21:06:31.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.261+0000 7f79ab321700 1 librados: init done 2024-02-19T21:06:31.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.261+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f799c001eb0 con 0x7f79a41031b0 2024-02-19T21:06:31.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.261+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f799c0038a0 con 0x7f79a41031b0 2024-02-19T21:06:31.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.261+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f7988004f00 con 0x7f79a41031b0 2024-02-19T21:06:31.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.262+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f799c002fe0 con 0x7f79a41031b0 2024-02-19T21:06:31.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(211..211 src has 43..211) v4 ==== 9126+0+0 (unknown 534417753 0 0) 0x7f799c076d90 con 0x7f79a41031b0 2024-02-19T21:06:31.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 osd_map(211..211 src has 43..211) v4 2024-02-19T21:06:31.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f799a7fc700 3 client.22527.objecter handle_osd_map got epochs [211,211] > 0 2024-02-19T21:06:31.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f799a7fc700 3 client.22527.objecter handle_osd_map decoding full epoch 211 2024-02-19T21:06:31.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f799a7fc700 20 client.22527.objecter dump_active .. 0 homeless 2024-02-19T21:06:31.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v211) v1 ==== 54+0+16 (unknown 472078242 0 2420000192) 0x7f799c077140 con 0x7f79a41031b0 2024-02-19T21:06:31.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f799a7fc700 10 client.22527.objecter ms_handle_connect 0x7f7990063c30 2024-02-19T21:06:31.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f79ab321700 10 client.22527.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:31.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 1 2024-02-19T21:06:31.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f79880051a0 osd=0 3 2024-02-19T21:06:31.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f79880051a0 osd=0 3 2024-02-19T21:06:31.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f79880051a0 osd=0 3 2024-02-19T21:06:31.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 1 2024-02-19T21:06:31.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f79880051a0 osd=0 3 2024-02-19T21:06:31.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 0 1 2024-02-19T21:06:31.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f79880051a0 osd=0 4 2024-02-19T21:06:31.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f79ab321700 10 client.22527.objecter _send_command 1 2024-02-19T21:06:31.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7988004950 con 0x7f7988004530 2024-02-19T21:06:31.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.263+0000 7f799a7fc700 10 client.22527.objecter ms_handle_connect 0x7f7988004530 2024-02-19T21:06:31.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.264+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f7988004950 con 0x7f7988004530 2024-02-19T21:06:31.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.264+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:06:31.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.264+0000 7f799a7fc700 10 client.22527.objecter _finish_command 1 = system:0 2024-02-19T21:06:31.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.264+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f79880051a0 osd=0 4 2024-02-19T21:06:31.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.264+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 0 1 2024-02-19T21:06:31.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.278+0000 7f79ab321700 10 client.22527.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:31.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.278+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 2 2024-02-19T21:06:31.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.278+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f79880051a0 osd=0 3 2024-02-19T21:06:31.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.278+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f79880051a0 osd=0 3 2024-02-19T21:06:31.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.278+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f79880051a0 osd=0 3 2024-02-19T21:06:31.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.278+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 2 2024-02-19T21:06:31.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.278+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f79880051a0 osd=0 3 2024-02-19T21:06:31.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.278+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 0 2 2024-02-19T21:06:31.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.278+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f79880051a0 osd=0 4 2024-02-19T21:06:31.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.278+0000 7f79ab321700 10 client.22527.objecter _send_command 2 2024-02-19T21:06:31.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.278+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7988004d30 con 0x7f7988004530 2024-02-19T21:06:31.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f7988004d30 con 0x7f7988004530 2024-02-19T21:06:31.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:31.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f799a7fc700 10 client.22527.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f79880051a0 osd=0 4 2024-02-19T21:06:31.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 0 2 2024-02-19T21:06:31.275 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:06:31.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f79ab321700 10 client.22527.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:31.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 3 2024-02-19T21:06:31.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f7988004270 osd=1 3 2024-02-19T21:06:31.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f7988004270 osd=1 3 2024-02-19T21:06:31.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f7988004270 osd=1 3 2024-02-19T21:06:31.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 3 2024-02-19T21:06:31.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f7988004270 osd=1 3 2024-02-19T21:06:31.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 1 3 2024-02-19T21:06:31.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f7988004270 osd=1 4 2024-02-19T21:06:31.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f79ab321700 10 client.22527.objecter _send_command 3 2024-02-19T21:06:31.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f798800bf30 con 0x7f7988003b40 2024-02-19T21:06:31.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.280+0000 7f799a7fc700 10 client.22527.objecter ms_handle_connect 0x7f7988003b40 2024-02-19T21:06:31.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.281+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f798800bf30 con 0x7f7988003b40 2024-02-19T21:06:31.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.281+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:06:31.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.281+0000 7f799a7fc700 10 client.22527.objecter _finish_command 3 = system:0 2024-02-19T21:06:31.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.281+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f7988004270 osd=1 4 2024-02-19T21:06:31.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.281+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 1 3 2024-02-19T21:06:31.280 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:31.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.296+0000 7f79ab321700 10 client.22527.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:31.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.296+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 4 2024-02-19T21:06:31.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.296+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f7988004270 osd=1 3 2024-02-19T21:06:31.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.296+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f7988004270 osd=1 3 2024-02-19T21:06:31.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.296+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f7988004270 osd=1 3 2024-02-19T21:06:31.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.296+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 4 2024-02-19T21:06:31.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.296+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f7988004270 osd=1 3 2024-02-19T21:06:31.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.296+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 1 4 2024-02-19T21:06:31.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.296+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f7988004270 osd=1 4 2024-02-19T21:06:31.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.296+0000 7f79ab321700 10 client.22527.objecter _send_command 4 2024-02-19T21:06:31.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.296+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f798800d250 con 0x7f7988003b40 2024-02-19T21:06:31.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f798800d250 con 0x7f7988003b40 2024-02-19T21:06:31.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:31.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f799a7fc700 10 client.22527.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f7988004270 osd=1 4 2024-02-19T21:06:31.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 1 4 2024-02-19T21:06:31.293 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:06:31.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f79ab321700 10 client.22527.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:31.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 5 2024-02-19T21:06:31.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f79880033a0 osd=2 3 2024-02-19T21:06:31.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f79880033a0 osd=2 3 2024-02-19T21:06:31.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f79880033a0 osd=2 3 2024-02-19T21:06:31.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 5 2024-02-19T21:06:31.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f79880033a0 osd=2 3 2024-02-19T21:06:31.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 2 5 2024-02-19T21:06:31.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f79880033a0 osd=2 4 2024-02-19T21:06:31.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f79ab321700 10 client.22527.objecter _send_command 5 2024-02-19T21:06:31.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7988010d00 con 0x7f7988002070 2024-02-19T21:06:31.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.298+0000 7f799a7fc700 10 client.22527.objecter ms_handle_connect 0x7f7988002070 2024-02-19T21:06:31.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.299+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f7988010d00 con 0x7f7988002070 2024-02-19T21:06:31.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.299+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:06:31.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.299+0000 7f799a7fc700 10 client.22527.objecter _finish_command 5 = system:0 2024-02-19T21:06:31.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.299+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f79880033a0 osd=2 4 2024-02-19T21:06:31.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.299+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 2 5 2024-02-19T21:06:31.309 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:31.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.314+0000 7f79ab321700 10 client.22527.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:31.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.315+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 6 2024-02-19T21:06:31.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.315+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f79880033a0 osd=2 3 2024-02-19T21:06:31.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.315+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f79880033a0 osd=2 3 2024-02-19T21:06:31.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.315+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f79880033a0 osd=2 3 2024-02-19T21:06:31.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.315+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 6 2024-02-19T21:06:31.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.315+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f79880033a0 osd=2 3 2024-02-19T21:06:31.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.315+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 2 6 2024-02-19T21:06:31.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.315+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f79880033a0 osd=2 4 2024-02-19T21:06:31.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.315+0000 7f79ab321700 10 client.22527.objecter _send_command 6 2024-02-19T21:06:31.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.315+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7988001630 con 0x7f7988002070 2024-02-19T21:06:31.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.316+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f7988001630 con 0x7f7988002070 2024-02-19T21:06:31.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.316+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:31.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.316+0000 7f799a7fc700 10 client.22527.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.316+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f79880033a0 osd=2 4 2024-02-19T21:06:31.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.316+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 2 6 2024-02-19T21:06:31.312 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:06:31.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f79ab321700 10 client.22527.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:31.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 7 2024-02-19T21:06:31.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f798800c8a0 osd=3 3 2024-02-19T21:06:31.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f798800c8a0 osd=3 3 2024-02-19T21:06:31.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f798800c8a0 osd=3 3 2024-02-19T21:06:31.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 7 2024-02-19T21:06:31.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f798800c8a0 osd=3 3 2024-02-19T21:06:31.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 3 7 2024-02-19T21:06:31.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f798800c8a0 osd=3 4 2024-02-19T21:06:31.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f79ab321700 10 client.22527.objecter _send_command 7 2024-02-19T21:06:31.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f798800cf80 con 0x7f7988010ea0 2024-02-19T21:06:31.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f799a7fc700 10 client.22527.objecter ms_handle_connect 0x7f7988010ea0 2024-02-19T21:06:31.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f798800cf80 con 0x7f7988010ea0 2024-02-19T21:06:31.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:06:31.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f799a7fc700 10 client.22527.objecter _finish_command 7 = system:0 2024-02-19T21:06:31.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f798800c8a0 osd=3 4 2024-02-19T21:06:31.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.317+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 3 7 2024-02-19T21:06:31.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.333+0000 7f79ab321700 10 client.22527.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:31.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.333+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 8 2024-02-19T21:06:31.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.333+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f798800c8a0 osd=3 3 2024-02-19T21:06:31.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.333+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f798800c8a0 osd=3 3 2024-02-19T21:06:31.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.333+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f798800c8a0 osd=3 3 2024-02-19T21:06:31.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.333+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 8 2024-02-19T21:06:31.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.333+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f798800c8a0 osd=3 3 2024-02-19T21:06:31.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.333+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 3 8 2024-02-19T21:06:31.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.333+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f798800c8a0 osd=3 4 2024-02-19T21:06:31.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.333+0000 7f79ab321700 10 client.22527.objecter _send_command 8 2024-02-19T21:06:31.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.333+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f79880121c0 con 0x7f7988010ea0 2024-02-19T21:06:31.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f79880121c0 con 0x7f7988010ea0 2024-02-19T21:06:31.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:31.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f799a7fc700 10 client.22527.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f798800c8a0 osd=3 4 2024-02-19T21:06:31.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 3 8 2024-02-19T21:06:31.330 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:06:31.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f79ab321700 10 client.22527.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:31.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 9 2024-02-19T21:06:31.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f79880121c0 osd=4 3 2024-02-19T21:06:31.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f79880121c0 osd=4 3 2024-02-19T21:06:31.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f79880121c0 osd=4 3 2024-02-19T21:06:31.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 9 2024-02-19T21:06:31.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f79880121c0 osd=4 3 2024-02-19T21:06:31.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 4 9 2024-02-19T21:06:31.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f79880121c0 osd=4 4 2024-02-19T21:06:31.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f79ab321700 10 client.22527.objecter _send_command 9 2024-02-19T21:06:31.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f798801a270 con 0x7f79880166c0 2024-02-19T21:06:31.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.335+0000 7f799a7fc700 10 client.22527.objecter ms_handle_connect 0x7f79880166c0 2024-02-19T21:06:31.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.336+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f798801a270 con 0x7f79880166c0 2024-02-19T21:06:31.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.336+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:06:31.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.336+0000 7f799a7fc700 10 client.22527.objecter _finish_command 9 = system:0 2024-02-19T21:06:31.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.336+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f79880121c0 osd=4 4 2024-02-19T21:06:31.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.336+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 4 9 2024-02-19T21:06:31.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.351+0000 7f79ab321700 10 client.22527.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:31.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.351+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 10 2024-02-19T21:06:31.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.351+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f79880121c0 osd=4 3 2024-02-19T21:06:31.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.351+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f79880121c0 osd=4 3 2024-02-19T21:06:31.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.351+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f79880121c0 osd=4 3 2024-02-19T21:06:31.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.351+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 10 2024-02-19T21:06:31.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.351+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f79880121c0 osd=4 3 2024-02-19T21:06:31.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.351+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 4 10 2024-02-19T21:06:31.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.351+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f79880121c0 osd=4 4 2024-02-19T21:06:31.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.351+0000 7f79ab321700 10 client.22527.objecter _send_command 10 2024-02-19T21:06:31.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.351+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f798801a270 con 0x7f79880166c0 2024-02-19T21:06:31.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.352+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f798801a270 con 0x7f79880166c0 2024-02-19T21:06:31.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.352+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:31.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.352+0000 7f799a7fc700 10 client.22527.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.352+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f79880121c0 osd=4 4 2024-02-19T21:06:31.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.352+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 4 10 2024-02-19T21:06:31.348 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:06:31.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.353+0000 7f79ab321700 10 client.22527.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:31.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.353+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 11 2024-02-19T21:06:31.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.353+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f798801a270 osd=5 3 2024-02-19T21:06:31.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.353+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f798801a270 osd=5 3 2024-02-19T21:06:31.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.353+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f798801a270 osd=5 3 2024-02-19T21:06:31.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.353+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 11 2024-02-19T21:06:31.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.353+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f798801a270 osd=5 3 2024-02-19T21:06:31.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.353+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 5 11 2024-02-19T21:06:31.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.353+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f798801a270 osd=5 4 2024-02-19T21:06:31.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.353+0000 7f79ab321700 10 client.22527.objecter _send_command 11 2024-02-19T21:06:31.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.353+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f798801b4f0 con 0x7f798801bf70 2024-02-19T21:06:31.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.353+0000 7f799a7fc700 10 client.22527.objecter ms_handle_connect 0x7f798801bf70 2024-02-19T21:06:31.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.354+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f798801b4f0 con 0x7f798801bf70 2024-02-19T21:06:31.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.354+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:06:31.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.354+0000 7f799a7fc700 10 client.22527.objecter _finish_command 11 = system:0 2024-02-19T21:06:31.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.354+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f798801a270 osd=5 4 2024-02-19T21:06:31.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.354+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 5 11 2024-02-19T21:06:31.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.368+0000 7f79ab321700 10 client.22527.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:31.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.368+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 12 2024-02-19T21:06:31.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.368+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f798801a270 osd=5 3 2024-02-19T21:06:31.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.368+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f798801a270 osd=5 3 2024-02-19T21:06:31.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.368+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f798801a270 osd=5 3 2024-02-19T21:06:31.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.368+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 12 2024-02-19T21:06:31.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.368+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f798801a270 osd=5 3 2024-02-19T21:06:31.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.368+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 5 12 2024-02-19T21:06:31.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.368+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f798801a270 osd=5 4 2024-02-19T21:06:31.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.368+0000 7f79ab321700 10 client.22527.objecter _send_command 12 2024-02-19T21:06:31.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.368+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f798801b4f0 con 0x7f798801bf70 2024-02-19T21:06:31.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f798801b4f0 con 0x7f798801bf70 2024-02-19T21:06:31.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:31.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f799a7fc700 10 client.22527.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f798801a270 osd=5 4 2024-02-19T21:06:31.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 5 12 2024-02-19T21:06:31.366 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:06:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f79ab321700 10 client.22527.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 13 2024-02-19T21:06:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f798801b4f0 osd=6 3 2024-02-19T21:06:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f798801b4f0 osd=6 3 2024-02-19T21:06:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f798801b4f0 osd=6 3 2024-02-19T21:06:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 13 2024-02-19T21:06:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f798801b4f0 osd=6 3 2024-02-19T21:06:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 6 13 2024-02-19T21:06:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f798801b4f0 osd=6 4 2024-02-19T21:06:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f79ab321700 10 client.22527.objecter _send_command 13 2024-02-19T21:06:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.370+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7988022c20 con 0x7f798801f110 2024-02-19T21:06:31.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.371+0000 7f799a7fc700 10 client.22527.objecter ms_handle_connect 0x7f798801f110 2024-02-19T21:06:31.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.371+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f7988022c20 con 0x7f798801f110 2024-02-19T21:06:31.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.371+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:06:31.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.371+0000 7f799a7fc700 10 client.22527.objecter _finish_command 13 = system:0 2024-02-19T21:06:31.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.371+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f798801b4f0 osd=6 4 2024-02-19T21:06:31.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.371+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 6 13 2024-02-19T21:06:31.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.388+0000 7f79ab321700 10 client.22527.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:31.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.388+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 14 2024-02-19T21:06:31.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.388+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f798801b4f0 osd=6 3 2024-02-19T21:06:31.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.388+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f798801b4f0 osd=6 3 2024-02-19T21:06:31.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.388+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f798801b4f0 osd=6 3 2024-02-19T21:06:31.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.388+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 14 2024-02-19T21:06:31.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.388+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f798801b4f0 osd=6 3 2024-02-19T21:06:31.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.388+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 6 14 2024-02-19T21:06:31.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.388+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f798801b4f0 osd=6 4 2024-02-19T21:06:31.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.388+0000 7f79ab321700 10 client.22527.objecter _send_command 14 2024-02-19T21:06:31.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.388+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7988022c20 con 0x7f798801f110 2024-02-19T21:06:31.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f7988022c20 con 0x7f798801f110 2024-02-19T21:06:31.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:31.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f799a7fc700 10 client.22527.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f798801b4f0 osd=6 4 2024-02-19T21:06:31.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 6 14 2024-02-19T21:06:31.386 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:06:31.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f79ab321700 10 client.22527.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:31.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 15 2024-02-19T21:06:31.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f7988022c20 osd=7 3 2024-02-19T21:06:31.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f7988022c20 osd=7 3 2024-02-19T21:06:31.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f7988022c20 osd=7 3 2024-02-19T21:06:31.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 15 2024-02-19T21:06:31.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f7988022c20 osd=7 3 2024-02-19T21:06:31.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 7 15 2024-02-19T21:06:31.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f7988022c20 osd=7 4 2024-02-19T21:06:31.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f79ab321700 10 client.22527.objecter _send_command 15 2024-02-19T21:06:31.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.390+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7988026f90 con 0x7f79880232a0 2024-02-19T21:06:31.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.391+0000 7f799a7fc700 10 client.22527.objecter ms_handle_connect 0x7f79880232a0 2024-02-19T21:06:31.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.392+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f7988026f90 con 0x7f79880232a0 2024-02-19T21:06:31.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.392+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:06:31.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.392+0000 7f799a7fc700 10 client.22527.objecter _finish_command 15 = system:0 2024-02-19T21:06:31.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.392+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f7988022c20 osd=7 4 2024-02-19T21:06:31.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.392+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 7 15 2024-02-19T21:06:31.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.406+0000 7f79ab321700 10 client.22527.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:31.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.406+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign -1 16 2024-02-19T21:06:31.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.406+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f7988022c20 osd=7 3 2024-02-19T21:06:31.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.406+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f7988022c20 osd=7 3 2024-02-19T21:06:31.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.406+0000 7f79ab321700 20 client.22527.objecter _get_session s=0x7f7988022c20 osd=7 3 2024-02-19T21:06:31.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.406+0000 7f79ab321700 15 client.22527.objecter _session_command_op_remove -1 16 2024-02-19T21:06:31.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.406+0000 7f79ab321700 20 client.22527.objecter get_session s=0x7f7988022c20 osd=7 3 2024-02-19T21:06:31.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.406+0000 7f79ab321700 15 client.22527.objecter _session_command_op_assign 7 16 2024-02-19T21:06:31.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.406+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f7988022c20 osd=7 4 2024-02-19T21:06:31.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.406+0000 7f79ab321700 10 client.22527.objecter _send_command 16 2024-02-19T21:06:31.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.406+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f7988026f90 con 0x7f79880232a0 2024-02-19T21:06:31.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.407+0000 7f799a7fc700 1 -- 172.21.15.163:0/1571310155 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f7988026f90 con 0x7f79880232a0 2024-02-19T21:06:31.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.407+0000 7f799a7fc700 10 client.22527.objecter ms_dispatch 0x7f79a4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:31.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.407+0000 7f799a7fc700 10 client.22527.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.407+0000 7f799a7fc700 20 client.22527.objecter put_session s=0x7f7988022c20 osd=7 4 2024-02-19T21:06:31.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.407+0000 7f799a7fc700 15 client.22527.objecter _session_command_op_remove 7 16 2024-02-19T21:06:31.403 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:06:31.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 10 librados: watch_flush enter 2024-02-19T21:06:31.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 10 librados: watch_flush exit 2024-02-19T21:06:31.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 10 client.22527.objecter close_session for osd.0 2024-02-19T21:06:31.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 >> v1:172.21.15.163:6801/1559418115 conn(0x7f7988004530 legacy=0x7f7988002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:31.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f79880051a0 osd=0 1 2024-02-19T21:06:31.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 10 client.22527.objecter close_session for osd.1 2024-02-19T21:06:31.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 >> v1:172.21.15.163:6805/3125047432 conn(0x7f7988003b40 legacy=0x7f798800a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:31.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f7988004270 osd=1 1 2024-02-19T21:06:31.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 10 client.22527.objecter close_session for osd.2 2024-02-19T21:06:31.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 >> v1:172.21.15.163:6809/1563158487 conn(0x7f7988002070 legacy=0x7f798800f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:31.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f79880033a0 osd=2 1 2024-02-19T21:06:31.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 10 client.22527.objecter close_session for osd.3 2024-02-19T21:06:31.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 >> v1:172.21.15.163:6813/2112279769 conn(0x7f7988010ea0 legacy=0x7f79880112e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:31.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f798800c8a0 osd=3 1 2024-02-19T21:06:31.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 10 client.22527.objecter close_session for osd.4 2024-02-19T21:06:31.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 >> v1:172.21.15.193:6800/3129716965 conn(0x7f79880166c0 legacy=0x7f7988018ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:31.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79a98be700 1 -- 172.21.15.163:0/1571310155 reap_dead start 2024-02-19T21:06:31.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79a98be700 1 -- 172.21.15.163:0/1571310155 reap_dead start 2024-02-19T21:06:31.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f79880121c0 osd=4 1 2024-02-19T21:06:31.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 10 client.22527.objecter close_session for osd.5 2024-02-19T21:06:31.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 >> v1:172.21.15.193:6804/805161991 conn(0x7f798801bf70 legacy=0x7f798801c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:31.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f798801a270 osd=5 1 2024-02-19T21:06:31.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 10 client.22527.objecter close_session for osd.6 2024-02-19T21:06:31.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 >> v1:172.21.15.193:6808/787604615 conn(0x7f798801f110 legacy=0x7f7988021500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:31.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f798801b4f0 osd=6 1 2024-02-19T21:06:31.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 10 client.22527.objecter close_session for osd.7 2024-02-19T21:06:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 >> v1:172.21.15.193:6812/157927432 conn(0x7f79880232a0 legacy=0x7f7988025710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 20 client.22527.objecter put_session s=0x7f7988022c20 osd=7 1 2024-02-19T21:06:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 20 client.22527.objecter shutdown clearing up homeless session... 2024-02-19T21:06:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 10 client.22527.objecter successfully canceled tick 2024-02-19T21:06:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 >> v1:172.21.15.163:6800/1552514681 conn(0x7f7990063c30 legacy=0x7f79900660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 >> v1:172.21.15.163:6789/0 conn(0x7f79a41031b0 legacy=0x7f79a4117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79a98be700 1 -- 172.21.15.163:0/1571310155 reap_dead start 2024-02-19T21:06:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79a98be700 1 -- 172.21.15.163:0/1571310155 reap_dead start 2024-02-19T21:06:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 shutdown_connections 2024-02-19T21:06:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 >> 172.21.15.163:0/1571310155 conn(0x7f79a40f2470 msgr2=0x7f79a4106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 shutdown_connections 2024-02-19T21:06:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 1 -- 172.21.15.163:0/1571310155 wait complete. 2024-02-19T21:06:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:31.408+0000 7f79ab321700 1 librados: shutdown 2024-02-19T21:06:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.407 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.408 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.408 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.408 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.408 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.408 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.408 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:06:31.658 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:31.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:31.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:31.761 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:31.789 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:31.992 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:30.165917+0000 mgr.y (mgr.14550) 240 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:31.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:31.180059+0000 mon.a (mon.0) 1509 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:31.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:31.184902+0000 mon.a (mon.0) 1510 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-19", "mode": "writeback"}]': finished 2024-02-19T21:06:31.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:31.185009+0000 mon.a (mon.0) 1511 : audit [INF] from='client.48004 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi163-83854-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:31.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:31.185088+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-02-19T21:06:31.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:31.203325+0000 mon.b (mon.1) 335 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:31.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:31.203804+0000 mon.a (mon.0) 1513 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:31.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:31.229584+0000 mon.b (mon.1) 336 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:31.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:31.229931+0000 mon.a (mon.0) 1514 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:31.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:31.263451+0000 mon.a (mon.0) 1515 : audit [DBG] from='client.? v1:172.21.15.163:0/1571310155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:31.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:31.309811+0000 mon.c (mon.2) 220 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:30.165917+0000 mgr.y (mgr.14550) 240 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:31.180059+0000 mon.a (mon.0) 1509 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:31.184902+0000 mon.a (mon.0) 1510 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-19", "mode": "writeback"}]': finished 2024-02-19T21:06:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:31.185009+0000 mon.a (mon.0) 1511 : audit [INF] from='client.48004 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi163-83854-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:31.185088+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-02-19T21:06:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:31.203325+0000 mon.b (mon.1) 335 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:31.203804+0000 mon.a (mon.0) 1513 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:31.229584+0000 mon.b (mon.1) 336 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:31.229931+0000 mon.a (mon.0) 1514 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:31.263451+0000 mon.a (mon.0) 1515 : audit [DBG] from='client.? v1:172.21.15.163:0/1571310155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:31.309811+0000 mon.c (mon.2) 220 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:32.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:30.165917+0000 mgr.y (mgr.14550) 240 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:32.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:31.180059+0000 mon.a (mon.0) 1509 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:32.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:31.184902+0000 mon.a (mon.0) 1510 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-19", "mode": "writeback"}]': finished 2024-02-19T21:06:32.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:31.185009+0000 mon.a (mon.0) 1511 : audit [INF] from='client.48004 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi163-83854-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:32.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:31.185088+0000 mon.a (mon.0) 1512 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-02-19T21:06:32.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:31.203325+0000 mon.b (mon.1) 335 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:32.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:31.203804+0000 mon.a (mon.0) 1513 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:32.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:31.229584+0000 mon.b (mon.1) 336 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:32.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:31.229931+0000 mon.a (mon.0) 1514 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:32.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:31.263451+0000 mon.a (mon.0) 1515 : audit [DBG] from='client.? v1:172.21.15.163:0/1571310155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:32.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:31.309811+0000 mon.c (mon.2) 220 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:32.238 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:32.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:32.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:32.342 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:32.372 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:32.393 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:32.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:32.496 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:32.508 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:06:32.509 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:06:32.509 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-19T21:06:32.509 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-02-19T21:06:32.509 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd primary-affinity 0 0 2024-02-19T21:06:32.525 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:32.544 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:32.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:32.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:32.648 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:06:32.675 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:32.694 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:06:32.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:32.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:32.797 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:32.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:31.144101+0000 mgr.y (mgr.14550) 241 : cluster [DBG] pgmap v287: 298 pgs: 8 creating+peering, 2 peering, 8 active+clean+snaptrim, 32 unknown, 14 active+clean+snaptrim_wait, 234 active+clean; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:06:32.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:32.186558+0000 mon.a (mon.0) 1516 : audit [INF] from='client.47998 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi163-83845-30"}]': finished 2024-02-19T21:06:32.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:32.186653+0000 mon.a (mon.0) 1517 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:32.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:32.186712+0000 mon.a (mon.0) 1518 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-02-19T21:06:32.823 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:32.189025+0000 mon.b (mon.1) 337 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19"}]: dispatch 2024-02-19T21:06:32.823 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:32.189725+0000 mon.b (mon.1) 338 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:32.823 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:32.190532+0000 mon.a (mon.0) 1519 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19"}]: dispatch 2024-02-19T21:06:32.823 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:32.190757+0000 mon.a (mon.0) 1520 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:32.823 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:32.310532+0000 mon.c (mon.2) 221 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:32.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:31.144101+0000 mgr.y (mgr.14550) 241 : cluster [DBG] pgmap v287: 298 pgs: 8 creating+peering, 2 peering, 8 active+clean+snaptrim, 32 unknown, 14 active+clean+snaptrim_wait, 234 active+clean; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:06:32.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:32.186558+0000 mon.a (mon.0) 1516 : audit [INF] from='client.47998 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi163-83845-30"}]': finished 2024-02-19T21:06:32.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:32.186653+0000 mon.a (mon.0) 1517 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:32.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:32.186712+0000 mon.a (mon.0) 1518 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-02-19T21:06:32.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:32.189025+0000 mon.b (mon.1) 337 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19"}]: dispatch 2024-02-19T21:06:32.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:32.189725+0000 mon.b (mon.1) 338 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:32.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:32.190532+0000 mon.a (mon.0) 1519 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19"}]: dispatch 2024-02-19T21:06:32.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:32.190757+0000 mon.a (mon.0) 1520 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:32.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:32.310532+0000 mon.c (mon.2) 221 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:32.824 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:33.004 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:06:33.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:31.144101+0000 mgr.y (mgr.14550) 241 : cluster [DBG] pgmap v287: 298 pgs: 8 creating+peering, 2 peering, 8 active+clean+snaptrim, 32 unknown, 14 active+clean+snaptrim_wait, 234 active+clean; 10 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:06:33.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:32.186558+0000 mon.a (mon.0) 1516 : audit [INF] from='client.47998 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi163-83845-30"}]': finished 2024-02-19T21:06:33.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:32.186653+0000 mon.a (mon.0) 1517 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:33.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:32.186712+0000 mon.a (mon.0) 1518 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-02-19T21:06:33.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:32.189025+0000 mon.b (mon.1) 337 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19"}]: dispatch 2024-02-19T21:06:33.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:32.189725+0000 mon.b (mon.1) 338 : audit [INF] from='client.? v1:172.21.15.163:0/464781093' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:33.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:32.190532+0000 mon.a (mon.0) 1519 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19"}]: dispatch 2024-02-19T21:06:33.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:32.190757+0000 mon.a (mon.0) 1520 : audit [INF] from='client.47998 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi163-83845-30"}]: dispatch 2024-02-19T21:06:33.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:32.310532+0000 mon.c (mon.2) 221 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:33.351 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:33.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:33.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:33.456 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:06:33.485 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:33.505 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:06:33.507 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:33.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:33.608 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:33.636 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:06:33] "GET /metrics HTTP/1.1" 200 191208 "" "Prometheus/2.33.4" 2024-02-19T21:06:33.639 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:33.653 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:06:33.653 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:06:33.653 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-02-19T21:06:33.653 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:06:33.658 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:33.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:33.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:33.762 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:33.790 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:33.975 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:33.144929+0000 mon.b (mon.1) 339 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-19T21:06:33.975 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:33.146116+0000 mon.a (mon.0) 1521 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-19T21:06:33.975 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:33.183929+0000 mon.a (mon.0) 1522 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:33.975 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:33.189475+0000 mon.a (mon.0) 1523 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19"}]': finished 2024-02-19T21:06:33.975 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:33.189559+0000 mon.a (mon.0) 1524 : audit [INF] from='client.47998 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi163-83845-30"}]': finished 2024-02-19T21:06:33.975 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:33.189651+0000 mon.a (mon.0) 1525 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-02-19T21:06:33.975 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:33.189715+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-02-19T21:06:33.976 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:33.196881+0000 mon.a (mon.0) 1527 : audit [INF] from='client.? v1:172.21.15.163:0/1259453423' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi163-83854-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:33.976 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:33.224992+0000 mon.c (mon.2) 222 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:33.976 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:33.225350+0000 mon.a (mon.0) 1528 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:33.976 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:33.226097+0000 mon.c (mon.2) 223 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:33.976 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:33.226316+0000 mon.a (mon.0) 1529 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:33.976 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:33.226684+0000 mon.c (mon.2) 224 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi163-83845-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:33.976 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:33.226902+0000 mon.a (mon.0) 1530 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi163-83845-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:33.976 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:33.311508+0000 mon.c (mon.2) 225 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:33.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:33.144929+0000 mon.b (mon.1) 339 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-19T21:06:33.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:33.146116+0000 mon.a (mon.0) 1521 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-19T21:06:33.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:33.183929+0000 mon.a (mon.0) 1522 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:33.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:33.189475+0000 mon.a (mon.0) 1523 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19"}]': finished 2024-02-19T21:06:33.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:33.189559+0000 mon.a (mon.0) 1524 : audit [INF] from='client.47998 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi163-83845-30"}]': finished 2024-02-19T21:06:33.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:33.189651+0000 mon.a (mon.0) 1525 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-02-19T21:06:33.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:33.189715+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-02-19T21:06:33.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:33.196881+0000 mon.a (mon.0) 1527 : audit [INF] from='client.? v1:172.21.15.163:0/1259453423' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi163-83854-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:33.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:33.224992+0000 mon.c (mon.2) 222 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:33.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:33.225350+0000 mon.a (mon.0) 1528 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:33.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:33.226097+0000 mon.c (mon.2) 223 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:33.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:33.226316+0000 mon.a (mon.0) 1529 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:33.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:33.226684+0000 mon.c (mon.2) 224 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi163-83845-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:33.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:33.226902+0000 mon.a (mon.0) 1530 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi163-83845-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:33.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:33.311508+0000 mon.c (mon.2) 225 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:34.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:33.144929+0000 mon.b (mon.1) 339 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-19T21:06:34.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:33.146116+0000 mon.a (mon.0) 1521 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-02-19T21:06:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:33.183929+0000 mon.a (mon.0) 1522 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:33.189475+0000 mon.a (mon.0) 1523 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-19"}]': finished 2024-02-19T21:06:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:33.189559+0000 mon.a (mon.0) 1524 : audit [INF] from='client.47998 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi163-83845-30"}]': finished 2024-02-19T21:06:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:33.189651+0000 mon.a (mon.0) 1525 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-02-19T21:06:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:33.189715+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-02-19T21:06:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:33.196881+0000 mon.a (mon.0) 1527 : audit [INF] from='client.? v1:172.21.15.163:0/1259453423' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi163-83854-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:33.224992+0000 mon.c (mon.2) 222 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:33.225350+0000 mon.a (mon.0) 1528 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:33.226097+0000 mon.c (mon.2) 223 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:33.226316+0000 mon.a (mon.0) 1529 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:33.226684+0000 mon.c (mon.2) 224 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi163-83845-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:33.226902+0000 mon.a (mon.0) 1530 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi163-83845-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:33.311508+0000 mon.c (mon.2) 225 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:34.086 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:06:34.375 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-19T21:06:34.428 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:34.444 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:34.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:34.472 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:34.480 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:34.550 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:34.580 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:34.600 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:34.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:34.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:34.704 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:34.736 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:34.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:33.144514+0000 mgr.y (mgr.14550) 242 : cluster [DBG] pgmap v290: 257 pgs: 1 peering, 8 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 234 active+clean; 9.9 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:06:34.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:34.182953+0000 mon.a (mon.0) 1531 : audit [INF] from='client.? v1:172.21.15.163:0/1259453423' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi163-83854-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:34.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:34.183081+0000 mon.a (mon.0) 1532 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi163-83845-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:34.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:34.183149+0000 mon.a (mon.0) 1533 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-02-19T21:06:34.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:34.198417+0000 mon.c (mon.2) 226 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi163-83845-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:34.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:34.198857+0000 mon.a (mon.0) 1534 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi163-83845-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:34.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:34.312285+0000 mon.c (mon.2) 227 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:34.955 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:33.144514+0000 mgr.y (mgr.14550) 242 : cluster [DBG] pgmap v290: 257 pgs: 1 peering, 8 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 234 active+clean; 9.9 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:06:34.955 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:34.182953+0000 mon.a (mon.0) 1531 : audit [INF] from='client.? v1:172.21.15.163:0/1259453423' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi163-83854-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:34.955 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:34.183081+0000 mon.a (mon.0) 1532 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi163-83845-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:34.955 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:34.183149+0000 mon.a (mon.0) 1533 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-02-19T21:06:34.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:34.198417+0000 mon.c (mon.2) 226 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi163-83845-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:34.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:34.198857+0000 mon.a (mon.0) 1534 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi163-83845-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:34.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:34.312285+0000 mon.c (mon.2) 227 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:35.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:33.144514+0000 mgr.y (mgr.14550) 242 : cluster [DBG] pgmap v290: 257 pgs: 1 peering, 8 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 234 active+clean; 9.9 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:06:35.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:34.182953+0000 mon.a (mon.0) 1531 : audit [INF] from='client.? v1:172.21.15.163:0/1259453423' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi163-83854-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:35.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:34.183081+0000 mon.a (mon.0) 1532 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi163-83845-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:35.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:34.183149+0000 mon.a (mon.0) 1533 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-02-19T21:06:35.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:34.198417+0000 mon.c (mon.2) 226 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi163-83845-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:35.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:34.198857+0000 mon.a (mon.0) 1534 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi163-83845-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:35.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:34.312285+0000 mon.c (mon.2) 227 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:36.168 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:36.181 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:35.145612+0000 mgr.y (mgr.14550) 243 : cluster [DBG] pgmap v293: 257 pgs: 2 clean+premerge+peered, 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 236 active+clean; 9.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:06:36.181 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:35.313012+0000 mon.c (mon.2) 228 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:36.181 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:35.425259+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-02-19T21:06:36.181 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:35.427200+0000 mon.b (mon.1) 340 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:36.182 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:35.427809+0000 mon.a (mon.0) 1536 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:36.291 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:36.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:36.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:36.396 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:36.428 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:36.448 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:36.450 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:35.145612+0000 mgr.y (mgr.14550) 243 : cluster [DBG] pgmap v293: 257 pgs: 2 clean+premerge+peered, 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 236 active+clean; 9.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:06:36.450 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:35.313012+0000 mon.c (mon.2) 228 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:36.450 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:35.425259+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-02-19T21:06:36.450 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:35.427200+0000 mon.b (mon.1) 340 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:36.450 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:35.427809+0000 mon.a (mon.0) 1536 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:36.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:36.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:36.458 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:35.145612+0000 mgr.y (mgr.14550) 243 : cluster [DBG] pgmap v293: 257 pgs: 2 clean+premerge+peered, 7 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 236 active+clean; 9.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:06:36.458 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:35.313012+0000 mon.c (mon.2) 228 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:36.458 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:35.425259+0000 mon.a (mon.0) 1535 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-02-19T21:06:36.458 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:35.427200+0000 mon.b (mon.1) 340 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:36.459 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:35.427809+0000 mon.a (mon.0) 1536 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:36.552 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:36.581 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:36.601 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:36.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:36.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:36.704 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:06:36.734 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:36.755 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:06:36.757 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:36.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:06:36.858 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:36.886 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:36.904 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:36.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:36.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:37.009 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:37.036 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:37.054 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:37.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:37.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:37.158 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:06:37.184 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:37.222 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:06:37.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:37.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:37.242 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:36.193588+0000 mon.a (mon.0) 1537 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi163-83845-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi163-83845-31"}]': finished 2024-02-19T21:06:37.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:36.193740+0000 mon.a (mon.0) 1538 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:37.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:36.193895+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-02-19T21:06:37.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:36.213362+0000 mon.a (mon.0) 1540 : audit [INF] from='client.? v1:172.21.15.163:0/3549317140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:37.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:36.216964+0000 mon.b (mon.1) 341 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:37.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:36.219309+0000 mon.a (mon.0) 1541 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:37.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:36.313694+0000 mon.c (mon.2) 229 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:37.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:37.146727+0000 mon.b (mon.1) 342 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-19T21:06:37.243 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:37.147831+0000 mon.a (mon.0) 1542 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-19T21:06:37.243 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:36.193588+0000 mon.a (mon.0) 1537 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi163-83845-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi163-83845-31"}]': finished 2024-02-19T21:06:37.243 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:36.193740+0000 mon.a (mon.0) 1538 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:37.243 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:36.193895+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-02-19T21:06:37.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:36.213362+0000 mon.a (mon.0) 1540 : audit [INF] from='client.? v1:172.21.15.163:0/3549317140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:37.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:36.216964+0000 mon.b (mon.1) 341 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:37.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:36.219309+0000 mon.a (mon.0) 1541 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:37.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:36.313694+0000 mon.c (mon.2) 229 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:37.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:37.146727+0000 mon.b (mon.1) 342 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-19T21:06:37.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:37.147831+0000 mon.a (mon.0) 1542 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-19T21:06:37.326 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:37.356 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:37.434 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:37.438 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:37.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:37.539 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:37.565 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:37.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:36.193588+0000 mon.a (mon.0) 1537 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi163-83845-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi163-83845-31"}]': finished 2024-02-19T21:06:37.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:36.193740+0000 mon.a (mon.0) 1538 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:37.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:36.193895+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-02-19T21:06:37.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:36.213362+0000 mon.a (mon.0) 1540 : audit [INF] from='client.? v1:172.21.15.163:0/3549317140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:37.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:36.216964+0000 mon.b (mon.1) 341 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:37.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:36.219309+0000 mon.a (mon.0) 1541 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:37.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:36.313694+0000 mon.c (mon.2) 229 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:37.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:37.146727+0000 mon.b (mon.1) 342 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-19T21:06:37.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:37.147831+0000 mon.a (mon.0) 1542 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-02-19T21:06:38.066 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:06:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:06:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:06:38.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.282+0000 7ff540085700 1 -- 172.21.15.163:0/4264365397 >> v1:172.21.15.163:6790/0 conn(0x7ff5381065c0 legacy=0x7ff5381089b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:38.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.282+0000 7ff540085700 1 -- 172.21.15.163:0/4264365397 shutdown_connections 2024-02-19T21:06:38.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.282+0000 7ff540085700 1 -- 172.21.15.163:0/4264365397 >> 172.21.15.163:0/4264365397 conn(0x7ff53806fdc0 msgr2=0x7ff5380f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:38.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.282+0000 7ff540085700 1 -- 172.21.15.163:0/4264365397 shutdown_connections 2024-02-19T21:06:38.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.282+0000 7ff540085700 1 -- 172.21.15.163:0/4264365397 wait complete. 2024-02-19T21:06:38.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff540085700 1 librados: starting msgr at 2024-02-19T21:06:38.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff540085700 1 librados: starting objecter 2024-02-19T21:06:38.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff540085700 1 Processor -- start 2024-02-19T21:06:38.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff540085700 1 -- start start 2024-02-19T21:06:38.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff540085700 1 librados: setting wanted keys 2024-02-19T21:06:38.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff540085700 1 librados: calling monclient init 2024-02-19T21:06:38.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff540085700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff538105cc0 con 0x7ff5380f8bb0 2024-02-19T21:06:38.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff540085700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff538105e50 con 0x7ff5380f8740 2024-02-19T21:06:38.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff540085700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff538105fe0 con 0x7ff5381065c0 2024-02-19T21:06:38.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff53d620700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ff5380f8bb0 0x7ff538067570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57756/0 (socket says 172.21.15.163:57756) 2024-02-19T21:06:38.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff53d620700 1 -- 172.21.15.163:0/1530497005 learned_addr learned my addr 172.21.15.163:0/1530497005 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:38.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff52effd700 10 client.?.objecter ms_handle_connect 0x7ff5380f8740 2024-02-19T21:06:38.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff52effd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:38.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff52effd700 10 client.?.objecter ms_handle_connect 0x7ff5381065c0 2024-02-19T21:06:38.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff52effd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:38.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff52effd700 10 client.?.objecter ms_handle_connect 0x7ff5380f8bb0 2024-02-19T21:06:38.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff52effd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:38.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573355558 0 0) 0x7ff538105e50 con 0x7ff5380f8740 2024-02-19T21:06:38.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff524003800 con 0x7ff5380f8740 2024-02-19T21:06:38.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.283+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887381136 0 0) 0x7ff538105fe0 con 0x7ff5381065c0 2024-02-19T21:06:38.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff538105e50 con 0x7ff5381065c0 2024-02-19T21:06:38.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799187895 0 0) 0x7ff538105cc0 con 0x7ff5380f8bb0 2024-02-19T21:06:38.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff538105fe0 con 0x7ff5380f8bb0 2024-02-19T21:06:38.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1505471772 0 0) 0x7ff524003800 con 0x7ff5380f8740 2024-02-19T21:06:38.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff538105cc0 con 0x7ff5380f8740 2024-02-19T21:06:38.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff5340019f0 con 0x7ff5380f8740 2024-02-19T21:06:38.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3791772718 0 0) 0x7ff538105e50 con 0x7ff5381065c0 2024-02-19T21:06:38.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff524003800 con 0x7ff5381065c0 2024-02-19T21:06:38.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff530003220 con 0x7ff5381065c0 2024-02-19T21:06:38.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1607118169 0 0) 0x7ff538105cc0 con 0x7ff5380f8740 2024-02-19T21:06:38.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 >> v1:172.21.15.163:6790/0 conn(0x7ff5381065c0 legacy=0x7ff5381055b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:38.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 >> v1:172.21.15.163:6789/0 conn(0x7ff5380f8bb0 legacy=0x7ff538067570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:38.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff538106170 con 0x7ff5380f8740 2024-02-19T21:06:38.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff540085700 1 -- 172.21.15.163:0/1530497005 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff538106420 con 0x7ff5380f8740 2024-02-19T21:06:38.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff540085700 10 client.48043.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:38.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff540085700 1 -- 172.21.15.163:0/1530497005 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff538126db0 con 0x7ff5380f8740 2024-02-19T21:06:38.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.284+0000 7ff540085700 1 librados: init done 2024-02-19T21:06:38.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.285+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff534002360 con 0x7ff5380f8740 2024-02-19T21:06:38.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.285+0000 7ff540085700 1 -- 172.21.15.163:0/1530497005 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53805f700 con 0x7ff5380f8740 2024-02-19T21:06:38.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.285+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff534003180 con 0x7ff5380f8740 2024-02-19T21:06:38.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.287+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ff534003430 con 0x7ff5380f8740 2024-02-19T21:06:38.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.287+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(218..218 src has 43..218) v4 ==== 9498+0+0 (unknown 3278623557 0 0) 0x7ff534076930 con 0x7ff5380f8740 2024-02-19T21:06:38.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.287+0000 7ff52effd700 10 client.48043.objecter ms_dispatch 0x7ff538001550 osd_map(218..218 src has 43..218) v4 2024-02-19T21:06:38.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.287+0000 7ff52effd700 3 client.48043.objecter handle_osd_map got epochs [218,218] > 0 2024-02-19T21:06:38.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.287+0000 7ff52effd700 3 client.48043.objecter handle_osd_map decoding full epoch 218 2024-02-19T21:06:38.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.287+0000 7ff52effd700 20 client.48043.objecter dump_active .. 0 homeless 2024-02-19T21:06:38.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.287+0000 7ff52effd700 10 client.48043.objecter ms_handle_connect 0x7ff524063980 2024-02-19T21:06:38.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.290+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff5340498d0 con 0x7ff5380f8740 2024-02-19T21:06:38.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:38.476+0000 7ff540085700 1 -- 172.21.15.163:0/1530497005 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 0.0} v 0) v1 -- 0x7ff5380fa4b0 con 0x7ff5380f8740 2024-02-19T21:06:38.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:37.146067+0000 mgr.y (mgr.14550) 244 : cluster [DBG] pgmap v296: 296 pgs: 1 clean+premerge+peered, 7 active+clean+snaptrim, 72 unknown, 12 active+clean+snaptrim_wait, 204 active+clean; 9.6 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-02-19T21:06:38.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:37.201303+0000 mon.a (mon.0) 1543 : audit [INF] from='client.? v1:172.21.15.163:0/3549317140' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:38.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:37.201436+0000 mon.a (mon.0) 1544 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:38.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:37.201575+0000 mon.a (mon.0) 1545 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-02-19T21:06:38.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:37.201644+0000 mon.a (mon.0) 1546 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-02-19T21:06:38.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:37.222128+0000 mon.b (mon.1) 343 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-21"}]: dispatch 2024-02-19T21:06:38.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:37.222971+0000 mon.a (mon.0) 1547 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-21"}]: dispatch 2024-02-19T21:06:38.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:37.314410+0000 mon.c (mon.2) 230 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:38.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:37.146067+0000 mgr.y (mgr.14550) 244 : cluster [DBG] pgmap v296: 296 pgs: 1 clean+premerge+peered, 7 active+clean+snaptrim, 72 unknown, 12 active+clean+snaptrim_wait, 204 active+clean; 9.6 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-02-19T21:06:38.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:37.201303+0000 mon.a (mon.0) 1543 : audit [INF] from='client.? v1:172.21.15.163:0/3549317140' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:38.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:37.201436+0000 mon.a (mon.0) 1544 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:38.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:37.201575+0000 mon.a (mon.0) 1545 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-02-19T21:06:38.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:37.201644+0000 mon.a (mon.0) 1546 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-02-19T21:06:38.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:37.222128+0000 mon.b (mon.1) 343 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-21"}]: dispatch 2024-02-19T21:06:38.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:37.222971+0000 mon.a (mon.0) 1547 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-21"}]: dispatch 2024-02-19T21:06:38.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:37.314410+0000 mon.c (mon.2) 230 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:38.515 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:38.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:38.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:38.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:37.146067+0000 mgr.y (mgr.14550) 244 : cluster [DBG] pgmap v296: 296 pgs: 1 clean+premerge+peered, 7 active+clean+snaptrim, 72 unknown, 12 active+clean+snaptrim_wait, 204 active+clean; 9.6 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 4 op/s; 31 B/s, 0 objects/s recovering 2024-02-19T21:06:38.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:37.201303+0000 mon.a (mon.0) 1543 : audit [INF] from='client.? v1:172.21.15.163:0/3549317140' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:38.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:37.201436+0000 mon.a (mon.0) 1544 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:38.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:37.201575+0000 mon.a (mon.0) 1545 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-02-19T21:06:38.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:37.201644+0000 mon.a (mon.0) 1546 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-02-19T21:06:38.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:37.222128+0000 mon.b (mon.1) 343 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-21"}]: dispatch 2024-02-19T21:06:38.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:37.222971+0000 mon.a (mon.0) 1547 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-21"}]: dispatch 2024-02-19T21:06:38.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:37.314410+0000 mon.c (mon.2) 230 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:38.619 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:38.648 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:38.668 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:38.671 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:38.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:38.772 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:06:38.800 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:38.820 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:06:38.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:38.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:38.923 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:38.952 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:38.979 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:38.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:38.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:39.083 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:39.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.089+0000 7fea9f9bf700 1 -- 172.21.15.163:0/1737168985 >> v1:172.21.15.163:6789/0 conn(0x7fea980fd910 legacy=0x7fea980fdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:39.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.089+0000 7fea9f9bf700 1 -- 172.21.15.163:0/1737168985 shutdown_connections 2024-02-19T21:06:39.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.089+0000 7fea9f9bf700 1 -- 172.21.15.163:0/1737168985 >> 172.21.15.163:0/1737168985 conn(0x7fea980f3530 msgr2=0x7fea980f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:39.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.089+0000 7fea9f9bf700 1 -- 172.21.15.163:0/1737168985 shutdown_connections 2024-02-19T21:06:39.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.089+0000 7fea9f9bf700 1 -- 172.21.15.163:0/1737168985 wait complete. 2024-02-19T21:06:39.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.089+0000 7fea9f9bf700 1 librados: starting msgr at 2024-02-19T21:06:39.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.089+0000 7fea9f9bf700 1 librados: starting objecter 2024-02-19T21:06:39.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.089+0000 7fea9f9bf700 1 Processor -- start 2024-02-19T21:06:39.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.089+0000 7fea9f9bf700 1 -- start start 2024-02-19T21:06:39.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.089+0000 7fea9f9bf700 1 librados: setting wanted keys 2024-02-19T21:06:39.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.089+0000 7fea9f9bf700 1 librados: calling monclient init 2024-02-19T21:06:39.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea9f9bf700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fea980fd030 con 0x7fea98100710 2024-02-19T21:06:39.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea9f9bf700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fea980fd200 con 0x7fea980fd910 2024-02-19T21:06:39.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea9f9bf700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fea9811bb10 con 0x7fea98104250 2024-02-19T21:06:39.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea9cf5a700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fea98100710 0x7fea98114ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57780/0 (socket says 172.21.15.163:57780) 2024-02-19T21:06:39.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea9df5c700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fea98104250 0x7fea981183d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:56142/0 (socket says 172.21.15.163:56142) 2024-02-19T21:06:39.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea9cf5a700 1 -- 172.21.15.163:0/2232672904 learned_addr learned my addr 172.21.15.163:0/2232672904 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:39.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 10 client.?.objecter ms_handle_connect 0x7fea98100710 2024-02-19T21:06:39.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:39.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 10 client.?.objecter ms_handle_connect 0x7fea98104250 2024-02-19T21:06:39.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:39.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 10 client.?.objecter ms_handle_connect 0x7fea980fd910 2024-02-19T21:06:39.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:39.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826860558 0 0) 0x7fea9811bb10 con 0x7fea98104250 2024-02-19T21:06:39.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea84003630 con 0x7fea98104250 2024-02-19T21:06:39.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747518702 0 0) 0x7fea980fd030 con 0x7fea98100710 2024-02-19T21:06:39.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea9811bb10 con 0x7fea98100710 2024-02-19T21:06:39.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994587290 0 0) 0x7fea980fd200 con 0x7fea980fd910 2024-02-19T21:06:39.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea980fd030 con 0x7fea980fd910 2024-02-19T21:06:39.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3571949544 0 0) 0x7fea84003630 con 0x7fea98104250 2024-02-19T21:06:39.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea980fd200 con 0x7fea98104250 2024-02-19T21:06:39.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fea90001cd0 con 0x7fea98104250 2024-02-19T21:06:39.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2632764061 0 0) 0x7fea9811bb10 con 0x7fea98100710 2024-02-19T21:06:39.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea84003630 con 0x7fea98100710 2024-02-19T21:06:39.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fea940018e0 con 0x7fea98100710 2024-02-19T21:06:39.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 981392359 0 0) 0x7fea980fd200 con 0x7fea98104250 2024-02-19T21:06:39.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 >> v1:172.21.15.193:6789/0 conn(0x7fea980fd910 legacy=0x7fea980fc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:39.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 >> v1:172.21.15.163:6789/0 conn(0x7fea98100710 legacy=0x7fea98114ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:39.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea9811ccb0 con 0x7fea98104250 2024-02-19T21:06:39.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea9f9bf700 1 -- 172.21.15.163:0/2232672904 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea9811bd00 con 0x7fea98104250 2024-02-19T21:06:39.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.090+0000 7fea9f9bf700 10 client.49223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:39.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.091+0000 7fea9f9bf700 1 -- 172.21.15.163:0/2232672904 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea9811c170 con 0x7fea98104250 2024-02-19T21:06:39.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.091+0000 7fea9f9bf700 1 librados: init done 2024-02-19T21:06:39.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.091+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fea90002700 con 0x7fea98104250 2024-02-19T21:06:39.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.091+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fea90005d50 con 0x7fea98104250 2024-02-19T21:06:39.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.091+0000 7fea9f9bf700 1 -- 172.21.15.163:0/2232672904 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7c004f20 con 0x7fea98104250 2024-02-19T21:06:39.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.092+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fea90002260 con 0x7fea98104250 2024-02-19T21:06:39.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.093+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(218..218 src has 43..218) v4 ==== 9498+0+0 (unknown 3278623557 0 0) 0x7fea900796b0 con 0x7fea98104250 2024-02-19T21:06:39.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.093+0000 7fea8e7fc700 10 client.49223.objecter ms_dispatch 0x7fea98001550 osd_map(218..218 src has 43..218) v4 2024-02-19T21:06:39.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.093+0000 7fea8e7fc700 3 client.49223.objecter handle_osd_map got epochs [218,218] > 0 2024-02-19T21:06:39.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.093+0000 7fea8e7fc700 3 client.49223.objecter handle_osd_map decoding full epoch 218 2024-02-19T21:06:39.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.093+0000 7fea8e7fc700 20 client.49223.objecter dump_active .. 0 homeless 2024-02-19T21:06:39.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.093+0000 7fea8e7fc700 10 client.49223.objecter ms_handle_connect 0x7fea84063c20 2024-02-19T21:06:39.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.097+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fea9004c500 con 0x7fea98104250 2024-02-19T21:06:39.114 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:39.135 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:39.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:39.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:39.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.217+0000 7ff52effd700 1 -- 172.21.15.163:0/1530497005 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]=0 set osd.0 primary-affinity to 0 (802) v219) v1 ==== 129+0+0 (unknown 938792531 0 0) 0x7ff534047ce0 con 0x7ff5380f8740 2024-02-19T21:06:39.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.220+0000 7ff540085700 10 librados: watch_flush enter 2024-02-19T21:06:39.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.222+0000 7ff540085700 10 librados: watch_flush exit 2024-02-19T21:06:39.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.222+0000 7ff540085700 20 client.48043.objecter shutdown clearing up homeless session... 2024-02-19T21:06:39.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.222+0000 7ff540085700 10 client.48043.objecter successfully canceled tick 2024-02-19T21:06:39.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.222+0000 7ff540085700 1 -- 172.21.15.163:0/1530497005 >> v1:172.21.15.163:6800/1552514681 conn(0x7ff524063980 legacy=0x7ff524065e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:39.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.222+0000 7ff540085700 1 -- 172.21.15.163:0/1530497005 >> v1:172.21.15.193:6789/0 conn(0x7ff5380f8740 legacy=0x7ff538066e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:39.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.225+0000 7ff540085700 1 -- 172.21.15.163:0/1530497005 shutdown_connections 2024-02-19T21:06:39.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.225+0000 7ff540085700 1 -- 172.21.15.163:0/1530497005 >> 172.21.15.163:0/1530497005 conn(0x7ff53806fdc0 msgr2=0x7ff5380f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:39.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.232+0000 7ff540085700 1 -- 172.21.15.163:0/1530497005 shutdown_connections 2024-02-19T21:06:39.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.240+0000 7ff540085700 1 -- 172.21.15.163:0/1530497005 wait complete. 2024-02-19T21:06:39.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.240+0000 7ff540085700 1 librados: shutdown 2024-02-19T21:06:39.236 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.0 primary-affinity to 0 (802) 2024-02-19T21:06:39.238 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:06:39.264 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:39.283 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:06:39.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:39.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:39.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.329+0000 7fea9f9bf700 1 -- 172.21.15.163:0/2232672904 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fea7c0051d0 con 0x7fea98104250 2024-02-19T21:06:39.387 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:39.417 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:39.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:38.206206+0000 mon.a (mon.0) 1548 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-21"}]': finished 2024-02-19T21:06:39.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:38.206345+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-02-19T21:06:39.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:38.208896+0000 mon.c (mon.2) 231 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:39.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:38.209039+0000 mon.c (mon.2) 232 : audit [INF] from='client.? v1:172.21.15.163:0/3202752296' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:39.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:38.219562+0000 mon.b (mon.1) 344 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-21", "mode": "writeback"}]: dispatch 2024-02-19T21:06:39.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:38.220527+0000 mon.a (mon.0) 1550 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:39.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:38.220774+0000 mon.a (mon.0) 1551 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-21", "mode": "writeback"}]: dispatch 2024-02-19T21:06:39.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:38.221036+0000 mon.a (mon.0) 1552 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:39.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:38.315057+0000 mon.c (mon.2) 233 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:39.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:38.477818+0000 mon.b (mon.1) 345 : audit [INF] from='client.? v1:172.21.15.163:0/1530497005' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-02-19T21:06:39.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:38.478440+0000 mon.a (mon.0) 1553 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-02-19T21:06:39.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:39.102569+0000 mon.a (mon.0) 1554 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:06:39.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:39.197972+0000 mon.a (mon.0) 1555 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:39.481 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:38.206206+0000 mon.a (mon.0) 1548 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-21"}]': finished 2024-02-19T21:06:39.481 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:38.206345+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-02-19T21:06:39.481 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:38.208896+0000 mon.c (mon.2) 231 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:39.481 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:38.209039+0000 mon.c (mon.2) 232 : audit [INF] from='client.? v1:172.21.15.163:0/3202752296' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:39.481 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:38.219562+0000 mon.b (mon.1) 344 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-21", "mode": "writeback"}]: dispatch 2024-02-19T21:06:39.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:38.220527+0000 mon.a (mon.0) 1550 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:39.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:38.220774+0000 mon.a (mon.0) 1551 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-21", "mode": "writeback"}]: dispatch 2024-02-19T21:06:39.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:38.221036+0000 mon.a (mon.0) 1552 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:39.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:38.315057+0000 mon.c (mon.2) 233 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:39.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:38.477818+0000 mon.b (mon.1) 345 : audit [INF] from='client.? v1:172.21.15.163:0/1530497005' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-02-19T21:06:39.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:38.478440+0000 mon.a (mon.0) 1553 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-02-19T21:06:39.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:39.102569+0000 mon.a (mon.0) 1554 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:06:39.482 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:39.197972+0000 mon.a (mon.0) 1555 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:39.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:38.206206+0000 mon.a (mon.0) 1548 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-21"}]': finished 2024-02-19T21:06:39.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:38.206345+0000 mon.a (mon.0) 1549 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-02-19T21:06:39.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:38.208896+0000 mon.c (mon.2) 231 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:39.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:38.209039+0000 mon.c (mon.2) 232 : audit [INF] from='client.? v1:172.21.15.163:0/3202752296' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:39.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:38.219562+0000 mon.b (mon.1) 344 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-21", "mode": "writeback"}]: dispatch 2024-02-19T21:06:39.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:38.220527+0000 mon.a (mon.0) 1550 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:39.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:38.220774+0000 mon.a (mon.0) 1551 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-21", "mode": "writeback"}]: dispatch 2024-02-19T21:06:39.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:38.221036+0000 mon.a (mon.0) 1552 : audit [INF] from='client.49211 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:39.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:38.315057+0000 mon.c (mon.2) 233 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:39.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:38.477818+0000 mon.b (mon.1) 345 : audit [INF] from='client.? v1:172.21.15.163:0/1530497005' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-02-19T21:06:39.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:38.478440+0000 mon.a (mon.0) 1553 : audit [INF] from='client.48043 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-02-19T21:06:39.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:39.102569+0000 mon.a (mon.0) 1554 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:06:39.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:39.197972+0000 mon.a (mon.0) 1555 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:40.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.833+0000 7f9834858700 1 -- 172.21.15.163:0/1763364402 >> v1:172.21.15.163:6790/0 conn(0x7f98300fc8b0 legacy=0x7f98300fcc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:40.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.833+0000 7f9834858700 1 -- 172.21.15.163:0/1763364402 shutdown_connections 2024-02-19T21:06:40.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.833+0000 7f9834858700 1 -- 172.21.15.163:0/1763364402 >> 172.21.15.163:0/1763364402 conn(0x7f983006fdc0 msgr2=0x7f98300f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:40.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.833+0000 7f9834858700 1 -- 172.21.15.163:0/1763364402 shutdown_connections 2024-02-19T21:06:40.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.833+0000 7f9834858700 1 -- 172.21.15.163:0/1763364402 wait complete. 2024-02-19T21:06:40.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.834+0000 7f9834858700 1 librados: starting msgr at 2024-02-19T21:06:40.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.834+0000 7f9834858700 1 librados: starting objecter 2024-02-19T21:06:40.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.834+0000 7f9834858700 1 Processor -- start 2024-02-19T21:06:40.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.834+0000 7f9834858700 1 -- start start 2024-02-19T21:06:40.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.834+0000 7f9834858700 1 librados: setting wanted keys 2024-02-19T21:06:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.834+0000 7f9834858700 1 librados: calling monclient init 2024-02-19T21:06:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.834+0000 7f9834858700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9830117290 con 0x7f98300ff650 2024-02-19T21:06:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.834+0000 7f9834858700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9830118470 con 0x7f9830103190 2024-02-19T21:06:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.834+0000 7f9834858700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9830119670 con 0x7f98300fc8b0 2024-02-19T21:06:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.834+0000 7f982e59c700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f98300fc8b0 0x7f98300f60e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:56172/0 (socket says 172.21.15.163:56172) 2024-02-19T21:06:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.834+0000 7f982e59c700 1 -- 172.21.15.163:0/639075907 learned_addr learned my addr 172.21.15.163:0/639075907 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 10 client.?.objecter ms_handle_connect 0x7f98300fc8b0 2024-02-19T21:06:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 10 client.?.objecter ms_handle_connect 0x7f98300ff650 2024-02-19T21:06:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 10 client.?.objecter ms_handle_connect 0x7f9830103190 2024-02-19T21:06:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586730325 0 0) 0x7f9830119670 con 0x7f98300fc8b0 2024-02-19T21:06:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98180033f0 con 0x7f98300fc8b0 2024-02-19T21:06:40.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535334145 0 0) 0x7f9830117290 con 0x7f98300ff650 2024-02-19T21:06:40.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9830119670 con 0x7f98300ff650 2024-02-19T21:06:40.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219749476 0 0) 0x7f9830118470 con 0x7f9830103190 2024-02-19T21:06:40.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9830117290 con 0x7f9830103190 2024-02-19T21:06:40.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2790560257 0 0) 0x7f98180033f0 con 0x7f98300fc8b0 2024-02-19T21:06:40.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9830118470 con 0x7f98300fc8b0 2024-02-19T21:06:40.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f98200010c0 con 0x7f98300fc8b0 2024-02-19T21:06:40.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1223625183 0 0) 0x7f9830117290 con 0x7f9830103190 2024-02-19T21:06:40.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98180033f0 con 0x7f9830103190 2024-02-19T21:06:40.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1091951775 0 0) 0x7f9830119670 con 0x7f98300ff650 2024-02-19T21:06:40.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9830117290 con 0x7f98300ff650 2024-02-19T21:06:40.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f98280019a0 con 0x7f9830103190 2024-02-19T21:06:40.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9824002870 con 0x7f98300ff650 2024-02-19T21:06:40.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1470428838 0 0) 0x7f9830118470 con 0x7f98300fc8b0 2024-02-19T21:06:40.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 >> v1:172.21.15.193:6789/0 conn(0x7f9830103190 legacy=0x7f9830115950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:40.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 >> v1:172.21.15.163:6789/0 conn(0x7f98300ff650 legacy=0x7f98300f67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:40.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983011a870 con 0x7f98300fc8b0 2024-02-19T21:06:40.144 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:06:40.144 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:06:40.144 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:06:40.144 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:06:40.144 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:06:40.145 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:06:40.145 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:06:40.145 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:06:40.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9830117480 con 0x7f98300fc8b0 2024-02-19T21:06:40.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f9834858700 10 client.49235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:40.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9830117970 con 0x7f98300fc8b0 2024-02-19T21:06:40.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f9834858700 1 librados: init done 2024-02-19T21:06:40.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9820002830 con 0x7f98300fc8b0 2024-02-19T21:06:40.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.835+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f98200038d0 con 0x7f98300fc8b0 2024-02-19T21:06:40.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.836+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f9810004f00 con 0x7f98300fc8b0 2024-02-19T21:06:40.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9820001900 con 0x7f98300fc8b0 2024-02-19T21:06:40.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(219..219 src has 43..219) v4 ==== 9257+0+0 (unknown 2151616567 0 0) 0x7f98200795b0 con 0x7f98300fc8b0 2024-02-19T21:06:40.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 osd_map(219..219 src has 43..219) v4 2024-02-19T21:06:40.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f981f7fe700 3 client.49235.objecter handle_osd_map got epochs [219,219] > 0 2024-02-19T21:06:40.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f981f7fe700 3 client.49235.objecter handle_osd_map decoding full epoch 219 2024-02-19T21:06:40.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f981f7fe700 20 client.49235.objecter dump_active .. 0 homeless 2024-02-19T21:06:40.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v219) v1 ==== 54+0+16 (unknown 1263461098 0 2420000192) 0x7f982007b9c0 con 0x7f98300fc8b0 2024-02-19T21:06:40.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f981f7fe700 10 client.49235.objecter ms_handle_connect 0x7f9818063d70 2024-02-19T21:06:40.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f9834858700 10 client.49235.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:40.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 1 2024-02-19T21:06:40.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f98100051a0 osd=0 3 2024-02-19T21:06:40.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f98100051a0 osd=0 3 2024-02-19T21:06:40.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f98100051a0 osd=0 3 2024-02-19T21:06:40.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 1 2024-02-19T21:06:40.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f98100051a0 osd=0 3 2024-02-19T21:06:40.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 0 1 2024-02-19T21:06:40.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f98100051a0 osd=0 4 2024-02-19T21:06:40.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f9834858700 10 client.49235.objecter _send_command 1 2024-02-19T21:06:40.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.837+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9810004950 con 0x7f9810004530 2024-02-19T21:06:40.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.838+0000 7f981f7fe700 10 client.49235.objecter ms_handle_connect 0x7f9810004530 2024-02-19T21:06:40.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.838+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9810004950 con 0x7f9810004530 2024-02-19T21:06:40.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.838+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:06:40.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.838+0000 7f981f7fe700 10 client.49235.objecter _finish_command 1 = system:0 2024-02-19T21:06:40.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.838+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f98100051a0 osd=0 4 2024-02-19T21:06:40.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.838+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 0 1 2024-02-19T21:06:40.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.853+0000 7f9834858700 10 client.49235.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:40.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.853+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 2 2024-02-19T21:06:40.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.853+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f98100051a0 osd=0 3 2024-02-19T21:06:40.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.853+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f98100051a0 osd=0 3 2024-02-19T21:06:40.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.853+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f98100051a0 osd=0 3 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.853+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 2 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.853+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f98100051a0 osd=0 3 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.853+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 0 2 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.853+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f98100051a0 osd=0 4 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.853+0000 7f9834858700 10 client.49235.objecter _send_command 2 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.853+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9810004d30 con 0x7f9810004530 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9810004d30 con 0x7f9810004530 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f981f7fe700 10 client.49235.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f98100051a0 osd=0 4 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 0 2 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f9834858700 10 client.49235.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 3 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810003fe0 osd=1 3 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810003fe0 osd=1 3 2024-02-19T21:06:40.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810003fe0 osd=1 3 2024-02-19T21:06:40.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 3 2024-02-19T21:06:40.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f9810003fe0 osd=1 3 2024-02-19T21:06:40.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 1 3 2024-02-19T21:06:40.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810003fe0 osd=1 4 2024-02-19T21:06:40.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f9834858700 10 client.49235.objecter _send_command 3 2024-02-19T21:06:40.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f981000c000 con 0x7f9810004150 2024-02-19T21:06:40.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.855+0000 7f981f7fe700 10 client.49235.objecter ms_handle_connect 0x7f9810004150 2024-02-19T21:06:40.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.856+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f981000c000 con 0x7f9810004150 2024-02-19T21:06:40.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.856+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:06:40.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.856+0000 7f981f7fe700 10 client.49235.objecter _finish_command 3 = system:0 2024-02-19T21:06:40.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.856+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f9810003fe0 osd=1 4 2024-02-19T21:06:40.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.856+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 1 3 2024-02-19T21:06:40.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.871+0000 7f9834858700 10 client.49235.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:40.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.871+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 4 2024-02-19T21:06:40.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.871+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810003fe0 osd=1 3 2024-02-19T21:06:40.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.871+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810003fe0 osd=1 3 2024-02-19T21:06:40.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.871+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810003fe0 osd=1 3 2024-02-19T21:06:40.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.871+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 4 2024-02-19T21:06:40.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.871+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f9810003fe0 osd=1 3 2024-02-19T21:06:40.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.871+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 1 4 2024-02-19T21:06:40.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.871+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810003fe0 osd=1 4 2024-02-19T21:06:40.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.871+0000 7f9834858700 10 client.49235.objecter _send_command 4 2024-02-19T21:06:40.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.871+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9810001850 con 0x7f9810004150 2024-02-19T21:06:40.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9810001850 con 0x7f9810004150 2024-02-19T21:06:40.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:40.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f981f7fe700 10 client.49235.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f9810003fe0 osd=1 4 2024-02-19T21:06:40.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 1 4 2024-02-19T21:06:40.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f9834858700 10 client.49235.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:40.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 5 2024-02-19T21:06:40.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810003630 osd=2 3 2024-02-19T21:06:40.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810003630 osd=2 3 2024-02-19T21:06:40.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810003630 osd=2 3 2024-02-19T21:06:40.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 5 2024-02-19T21:06:40.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f9810003630 osd=2 3 2024-02-19T21:06:40.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 2 5 2024-02-19T21:06:40.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810003630 osd=2 4 2024-02-19T21:06:40.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f9834858700 10 client.49235.objecter _send_command 5 2024-02-19T21:06:40.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.872+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9810003e70 con 0x7f981000d2f0 2024-02-19T21:06:40.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.873+0000 7f981f7fe700 10 client.49235.objecter ms_handle_connect 0x7f981000d2f0 2024-02-19T21:06:40.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.873+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9810003e70 con 0x7f981000d2f0 2024-02-19T21:06:40.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.873+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:06:40.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.873+0000 7f981f7fe700 10 client.49235.objecter _finish_command 5 = system:0 2024-02-19T21:06:40.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.873+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f9810003630 osd=2 4 2024-02-19T21:06:40.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.873+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 2 5 2024-02-19T21:06:40.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.888+0000 7f9834858700 10 client.49235.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:40.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.888+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 6 2024-02-19T21:06:40.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.888+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810003630 osd=2 3 2024-02-19T21:06:40.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.888+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810003630 osd=2 3 2024-02-19T21:06:40.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.888+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810003630 osd=2 3 2024-02-19T21:06:40.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.888+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 6 2024-02-19T21:06:40.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.888+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f9810003630 osd=2 3 2024-02-19T21:06:40.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.888+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 2 6 2024-02-19T21:06:40.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.888+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810003630 osd=2 4 2024-02-19T21:06:40.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.888+0000 7f9834858700 10 client.49235.objecter _send_command 6 2024-02-19T21:06:40.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.888+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f981000e6e0 con 0x7f981000d2f0 2024-02-19T21:06:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f981000e6e0 con 0x7f981000d2f0 2024-02-19T21:06:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f981f7fe700 10 client.49235.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f9810003630 osd=2 4 2024-02-19T21:06:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 2 6 2024-02-19T21:06:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f9834858700 10 client.49235.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 7 2024-02-19T21:06:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f98100033e0 osd=3 3 2024-02-19T21:06:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f98100033e0 osd=3 3 2024-02-19T21:06:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f98100033e0 osd=3 3 2024-02-19T21:06:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 7 2024-02-19T21:06:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f98100033e0 osd=3 3 2024-02-19T21:06:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 3 7 2024-02-19T21:06:40.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f98100033e0 osd=3 4 2024-02-19T21:06:40.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f9834858700 10 client.49235.objecter _send_command 7 2024-02-19T21:06:40.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.889+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9810015b00 con 0x7f9810011e80 2024-02-19T21:06:40.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.890+0000 7f981f7fe700 10 client.49235.objecter ms_handle_connect 0x7f9810011e80 2024-02-19T21:06:40.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.890+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9810015b00 con 0x7f9810011e80 2024-02-19T21:06:40.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.890+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:06:40.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.890+0000 7f981f7fe700 10 client.49235.objecter _finish_command 7 = system:0 2024-02-19T21:06:40.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.890+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f98100033e0 osd=3 4 2024-02-19T21:06:40.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.890+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 3 7 2024-02-19T21:06:40.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.905+0000 7f9834858700 10 client.49235.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:40.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.905+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 8 2024-02-19T21:06:40.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.905+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f98100033e0 osd=3 3 2024-02-19T21:06:40.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.905+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f98100033e0 osd=3 3 2024-02-19T21:06:40.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.905+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f98100033e0 osd=3 3 2024-02-19T21:06:40.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.905+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 8 2024-02-19T21:06:40.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.905+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f98100033e0 osd=3 3 2024-02-19T21:06:40.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.905+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 3 8 2024-02-19T21:06:40.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.905+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f98100033e0 osd=3 4 2024-02-19T21:06:40.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.905+0000 7f9834858700 10 client.49235.objecter _send_command 8 2024-02-19T21:06:40.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.905+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9810015b00 con 0x7f9810011e80 2024-02-19T21:06:40.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9810015b00 con 0x7f9810011e80 2024-02-19T21:06:40.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:40.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f981f7fe700 10 client.49235.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f98100033e0 osd=3 4 2024-02-19T21:06:40.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 3 8 2024-02-19T21:06:40.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f9834858700 10 client.49235.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:40.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 9 2024-02-19T21:06:40.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810015b00 osd=4 3 2024-02-19T21:06:40.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810015b00 osd=4 3 2024-02-19T21:06:40.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810015b00 osd=4 3 2024-02-19T21:06:40.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 9 2024-02-19T21:06:40.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f9810015b00 osd=4 3 2024-02-19T21:06:40.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 4 9 2024-02-19T21:06:40.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810015b00 osd=4 4 2024-02-19T21:06:40.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f9834858700 10 client.49235.objecter _send_command 9 2024-02-19T21:06:40.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.906+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f981000cf40 con 0x7f981000c980 2024-02-19T21:06:40.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.907+0000 7f981f7fe700 10 client.49235.objecter ms_handle_connect 0x7f981000c980 2024-02-19T21:06:40.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.907+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f981000cf40 con 0x7f981000c980 2024-02-19T21:06:40.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.907+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:06:40.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.907+0000 7f981f7fe700 10 client.49235.objecter _finish_command 9 = system:0 2024-02-19T21:06:40.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.907+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f9810015b00 osd=4 4 2024-02-19T21:06:40.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.907+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 4 9 2024-02-19T21:06:40.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.923+0000 7f9834858700 10 client.49235.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:40.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.923+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 10 2024-02-19T21:06:40.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.923+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810015b00 osd=4 3 2024-02-19T21:06:40.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.923+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810015b00 osd=4 3 2024-02-19T21:06:40.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.923+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810015b00 osd=4 3 2024-02-19T21:06:40.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.923+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 10 2024-02-19T21:06:40.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.923+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f9810015b00 osd=4 3 2024-02-19T21:06:40.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.923+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 4 10 2024-02-19T21:06:40.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.923+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810015b00 osd=4 4 2024-02-19T21:06:40.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.923+0000 7f9834858700 10 client.49235.objecter _send_command 10 2024-02-19T21:06:40.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.923+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f981000d180 con 0x7f981000c980 2024-02-19T21:06:40.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f981000d180 con 0x7f981000c980 2024-02-19T21:06:40.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:40.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f981f7fe700 10 client.49235.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f9810015b00 osd=4 4 2024-02-19T21:06:40.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 4 10 2024-02-19T21:06:40.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f9834858700 10 client.49235.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:40.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 11 2024-02-19T21:06:40.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f981000d180 osd=5 3 2024-02-19T21:06:40.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f981000d180 osd=5 3 2024-02-19T21:06:40.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f981000d180 osd=5 3 2024-02-19T21:06:40.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 11 2024-02-19T21:06:40.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f981000d180 osd=5 3 2024-02-19T21:06:40.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 5 11 2024-02-19T21:06:40.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f981000d180 osd=5 4 2024-02-19T21:06:40.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f9834858700 10 client.49235.objecter _send_command 11 2024-02-19T21:06:40.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9810017e90 con 0x7f9810017a40 2024-02-19T21:06:40.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.925+0000 7f981f7fe700 10 client.49235.objecter ms_handle_connect 0x7f9810017a40 2024-02-19T21:06:40.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.926+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9810017e90 con 0x7f9810017a40 2024-02-19T21:06:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.926+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:06:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.926+0000 7f981f7fe700 10 client.49235.objecter _finish_command 11 = system:0 2024-02-19T21:06:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.926+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f981000d180 osd=5 4 2024-02-19T21:06:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.926+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 5 11 2024-02-19T21:06:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.941+0000 7f9834858700 10 client.49235.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.941+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 12 2024-02-19T21:06:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.941+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f981000d180 osd=5 3 2024-02-19T21:06:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.941+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f981000d180 osd=5 3 2024-02-19T21:06:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.941+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f981000d180 osd=5 3 2024-02-19T21:06:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.941+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 12 2024-02-19T21:06:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.941+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f981000d180 osd=5 3 2024-02-19T21:06:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.941+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 5 12 2024-02-19T21:06:40.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.941+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f981000d180 osd=5 4 2024-02-19T21:06:40.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.941+0000 7f9834858700 10 client.49235.objecter _send_command 12 2024-02-19T21:06:40.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.941+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9810017e90 con 0x7f9810017a40 2024-02-19T21:06:40.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9810017e90 con 0x7f9810017a40 2024-02-19T21:06:40.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:40.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f981f7fe700 10 client.49235.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f981000d180 osd=5 4 2024-02-19T21:06:40.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 5 12 2024-02-19T21:06:40.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f9834858700 10 client.49235.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:40.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 13 2024-02-19T21:06:40.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810017e90 osd=6 3 2024-02-19T21:06:40.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810017e90 osd=6 3 2024-02-19T21:06:40.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810017e90 osd=6 3 2024-02-19T21:06:40.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 13 2024-02-19T21:06:40.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f9810017e90 osd=6 3 2024-02-19T21:06:40.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 6 13 2024-02-19T21:06:40.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810017e90 osd=6 4 2024-02-19T21:06:40.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f9834858700 10 client.49235.objecter _send_command 13 2024-02-19T21:06:40.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9810018290 con 0x7f981001f0e0 2024-02-19T21:06:40.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.943+0000 7f981f7fe700 10 client.49235.objecter ms_handle_connect 0x7f981001f0e0 2024-02-19T21:06:40.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.944+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9810018290 con 0x7f981001f0e0 2024-02-19T21:06:40.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.944+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:06:40.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.944+0000 7f981f7fe700 10 client.49235.objecter _finish_command 13 = system:0 2024-02-19T21:06:40.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.944+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f9810017e90 osd=6 4 2024-02-19T21:06:40.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.944+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 6 13 2024-02-19T21:06:40.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.959+0000 7f9834858700 10 client.49235.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:40.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.959+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 14 2024-02-19T21:06:40.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.959+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810017e90 osd=6 3 2024-02-19T21:06:40.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.959+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810017e90 osd=6 3 2024-02-19T21:06:40.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.959+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810017e90 osd=6 3 2024-02-19T21:06:40.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.959+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 14 2024-02-19T21:06:40.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.959+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f9810017e90 osd=6 3 2024-02-19T21:06:40.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.959+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 6 14 2024-02-19T21:06:40.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.959+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810017e90 osd=6 4 2024-02-19T21:06:40.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.959+0000 7f9834858700 10 client.49235.objecter _send_command 14 2024-02-19T21:06:40.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.959+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9810018290 con 0x7f981001f0e0 2024-02-19T21:06:40.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9810018290 con 0x7f981001f0e0 2024-02-19T21:06:40.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:40.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f981f7fe700 10 client.49235.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f9810017e90 osd=6 4 2024-02-19T21:06:40.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 6 14 2024-02-19T21:06:40.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f9834858700 10 client.49235.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:40.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 15 2024-02-19T21:06:40.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810018290 osd=7 3 2024-02-19T21:06:40.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810018290 osd=7 3 2024-02-19T21:06:40.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810018290 osd=7 3 2024-02-19T21:06:40.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 15 2024-02-19T21:06:40.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f9810018290 osd=7 3 2024-02-19T21:06:40.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 7 15 2024-02-19T21:06:40.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810018290 osd=7 4 2024-02-19T21:06:40.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f9834858700 10 client.49235.objecter _send_command 15 2024-02-19T21:06:40.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9810023cb0 con 0x7f9810024750 2024-02-19T21:06:40.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.961+0000 7f981f7fe700 10 client.49235.objecter ms_handle_connect 0x7f9810024750 2024-02-19T21:06:40.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.962+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9810023cb0 con 0x7f9810024750 2024-02-19T21:06:40.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.962+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:06:40.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.962+0000 7f981f7fe700 10 client.49235.objecter _finish_command 15 = system:0 2024-02-19T21:06:40.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.962+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f9810018290 osd=7 4 2024-02-19T21:06:40.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.962+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 7 15 2024-02-19T21:06:40.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.976+0000 7f9834858700 10 client.49235.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:06:40.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.976+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign -1 16 2024-02-19T21:06:40.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.976+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810018290 osd=7 3 2024-02-19T21:06:40.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.976+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810018290 osd=7 3 2024-02-19T21:06:40.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.976+0000 7f9834858700 20 client.49235.objecter _get_session s=0x7f9810018290 osd=7 3 2024-02-19T21:06:40.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.976+0000 7f9834858700 15 client.49235.objecter _session_command_op_remove -1 16 2024-02-19T21:06:40.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.976+0000 7f9834858700 20 client.49235.objecter get_session s=0x7f9810018290 osd=7 3 2024-02-19T21:06:40.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.976+0000 7f9834858700 15 client.49235.objecter _session_command_op_assign 7 16 2024-02-19T21:06:40.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.976+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810018290 osd=7 4 2024-02-19T21:06:40.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.976+0000 7f9834858700 10 client.49235.objecter _send_command 16 2024-02-19T21:06:40.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.976+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f9810023c70 con 0x7f9810024750 2024-02-19T21:06:40.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.978+0000 7f981f7fe700 1 -- 172.21.15.163:0/639075907 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f9810023c70 con 0x7f9810024750 2024-02-19T21:06:40.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.978+0000 7f981f7fe700 10 client.49235.objecter ms_dispatch 0x7f9830001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:06:40.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.978+0000 7f981f7fe700 10 client.49235.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.978+0000 7f981f7fe700 20 client.49235.objecter put_session s=0x7f9810018290 osd=7 4 2024-02-19T21:06:40.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.978+0000 7f981f7fe700 15 client.49235.objecter _session_command_op_remove 7 16 2024-02-19T21:06:40.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 10 librados: watch_flush enter 2024-02-19T21:06:40.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 10 librados: watch_flush exit 2024-02-19T21:06:40.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 10 client.49235.objecter close_session for osd.0 2024-02-19T21:06:40.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 >> v1:172.21.15.163:6801/1559418115 conn(0x7f9810004530 legacy=0x7f9810002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:40.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f98100051a0 osd=0 1 2024-02-19T21:06:40.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 10 client.49235.objecter close_session for osd.1 2024-02-19T21:06:40.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 >> v1:172.21.15.163:6805/3125047432 conn(0x7f9810004150 legacy=0x7f981000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:40.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810003fe0 osd=1 1 2024-02-19T21:06:40.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 10 client.49235.objecter close_session for osd.2 2024-02-19T21:06:40.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 >> v1:172.21.15.163:6809/1563158487 conn(0x7f981000d2f0 legacy=0x7f981000d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:40.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810003630 osd=2 1 2024-02-19T21:06:40.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 10 client.49235.objecter close_session for osd.3 2024-02-19T21:06:40.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 >> v1:172.21.15.163:6813/2112279769 conn(0x7f9810011e80 legacy=0x7f9810014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:40.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f98100033e0 osd=3 1 2024-02-19T21:06:40.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 10 client.49235.objecter close_session for osd.4 2024-02-19T21:06:40.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 >> v1:172.21.15.193:6800/3129716965 conn(0x7f981000c980 legacy=0x7f9810015ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:40.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810015b00 osd=4 1 2024-02-19T21:06:40.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 10 client.49235.objecter close_session for osd.5 2024-02-19T21:06:40.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 >> v1:172.21.15.193:6804/805161991 conn(0x7f9810017a40 legacy=0x7f981001d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:40.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f982ed9d700 1 -- 172.21.15.163:0/639075907 reap_dead start 2024-02-19T21:06:40.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f981000d180 osd=5 1 2024-02-19T21:06:40.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 10 client.49235.objecter close_session for osd.6 2024-02-19T21:06:40.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 >> v1:172.21.15.193:6808/787604615 conn(0x7f981001f0e0 legacy=0x7f98100214d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:40.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f982ed9d700 1 -- 172.21.15.163:0/639075907 reap_dead start 2024-02-19T21:06:40.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810017e90 osd=6 1 2024-02-19T21:06:40.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 10 client.49235.objecter close_session for osd.7 2024-02-19T21:06:40.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 >> v1:172.21.15.193:6812/157927432 conn(0x7f9810024750 legacy=0x7f9810024bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:40.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 20 client.49235.objecter put_session s=0x7f9810018290 osd=7 1 2024-02-19T21:06:40.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 20 client.49235.objecter shutdown clearing up homeless session... 2024-02-19T21:06:40.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 10 client.49235.objecter successfully canceled tick 2024-02-19T21:06:40.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f982ed9d700 1 -- 172.21.15.163:0/639075907 reap_dead start 2024-02-19T21:06:40.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f982ed9d700 1 -- 172.21.15.163:0/639075907 reap_dead start 2024-02-19T21:06:40.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 >> v1:172.21.15.163:6800/1552514681 conn(0x7f9818063d70 legacy=0x7f9818066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:40.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 >> v1:172.21.15.163:6790/0 conn(0x7f98300fc8b0 legacy=0x7f98300f60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:40.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 shutdown_connections 2024-02-19T21:06:40.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 >> 172.21.15.163:0/639075907 conn(0x7f983006fdc0 msgr2=0x7f98300f4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:40.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 shutdown_connections 2024-02-19T21:06:40.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 1 -- 172.21.15.163:0/639075907 wait complete. 2024-02-19T21:06:40.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:39.979+0000 7f9834858700 1 librados: shutdown 2024-02-19T21:06:40.169 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.169 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.169 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.169 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.169 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.169 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.170 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.170 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:06:40.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:40.215+0000 7fea8e7fc700 1 -- 172.21.15.163:0/2232672904 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v220) v1 ==== 97+0+0 (unknown 749703400 0 0) 0x7fea9004a910 con 0x7fea98104250 2024-02-19T21:06:40.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:40.218+0000 7fea9f9bf700 10 librados: watch_flush enter 2024-02-19T21:06:40.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:40.218+0000 7fea9f9bf700 10 librados: watch_flush exit 2024-02-19T21:06:40.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:40.218+0000 7fea9f9bf700 20 client.49223.objecter shutdown clearing up homeless session... 2024-02-19T21:06:40.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:40.218+0000 7fea9f9bf700 10 client.49223.objecter successfully canceled tick 2024-02-19T21:06:40.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:40.224+0000 7fea9f9bf700 1 -- 172.21.15.163:0/2232672904 >> v1:172.21.15.163:6800/1552514681 conn(0x7fea84063c20 legacy=0x7fea840660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:40.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:40.224+0000 7fea9f9bf700 1 -- 172.21.15.163:0/2232672904 >> v1:172.21.15.163:6790/0 conn(0x7fea98104250 legacy=0x7fea981183d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:40.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:40.224+0000 7fea9f9bf700 1 -- 172.21.15.163:0/2232672904 shutdown_connections 2024-02-19T21:06:40.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:40.224+0000 7fea9f9bf700 1 -- 172.21.15.163:0/2232672904 >> 172.21.15.163:0/2232672904 conn(0x7fea980f3530 msgr2=0x7fea981076a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:40.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:40.231+0000 7fea9f9bf700 1 -- 172.21.15.163:0/2232672904 shutdown_connections 2024-02-19T21:06:40.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:40.235+0000 7fea9f9bf700 1 -- 172.21.15.163:0/2232672904 wait complete. 2024-02-19T21:06:40.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:40.235+0000 7fea9f9bf700 1 librados: shutdown 2024-02-19T21:06:40.243 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:06:40.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:39.146615+0000 mgr.y (mgr.14550) 245 : cluster [DBG] pgmap v299: 320 pgs: 1 clean+premerge+peered, 7 active+clean+snaptrim, 96 unknown, 12 active+clean+snaptrim_wait, 204 active+clean; 9.6 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:40.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:39.201933+0000 mon.a (mon.0) 1556 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi163-83845-31"}]': finished 2024-02-19T21:06:40.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:39.202003+0000 mon.a (mon.0) 1557 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-21", "mode": "writeback"}]': finished 2024-02-19T21:06:40.260 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:39.202072+0000 mon.a (mon.0) 1558 : audit [INF] from='client.49211 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:40.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:39.202218+0000 mon.a (mon.0) 1559 : audit [INF] from='client.48043 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-02-19T21:06:40.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:39.202287+0000 mon.a (mon.0) 1560 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-02-19T21:06:40.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:39.205468+0000 mon.c (mon.2) 234 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:40.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:39.222448+0000 mon.a (mon.0) 1561 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:40.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:39.315805+0000 mon.c (mon.2) 235 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:40.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:39.330553+0000 mon.c (mon.2) 236 : audit [INF] from='client.? v1:172.21.15.163:0/2232672904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:06:40.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:39.507739+0000 mon.a (mon.0) 1562 : audit [INF] from='client.49223 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:06:40.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:39.838397+0000 mon.c (mon.2) 237 : audit [DBG] from='client.? v1:172.21.15.163:0/639075907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:40.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:40.199861+0000 mon.a (mon.0) 1563 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:06:40.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:40.211063+0000 mon.a (mon.0) 1564 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi163-83845-31"}]': finished 2024-02-19T21:06:40.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:40.211295+0000 mon.a (mon.0) 1565 : audit [INF] from='client.49223 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:06:40.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:40.211356+0000 mon.a (mon.0) 1566 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-02-19T21:06:40.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:40.216586+0000 mon.c (mon.2) 238 : audit [INF] from='client.? v1:172.21.15.163:0/1861323793' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:40.261 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:40.219807+0000 mon.a (mon.0) 1567 : audit [INF] from='client.49229 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:40.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:39.146615+0000 mgr.y (mgr.14550) 245 : cluster [DBG] pgmap v299: 320 pgs: 1 clean+premerge+peered, 7 active+clean+snaptrim, 96 unknown, 12 active+clean+snaptrim_wait, 204 active+clean; 9.6 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:40.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:39.201933+0000 mon.a (mon.0) 1556 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi163-83845-31"}]': finished 2024-02-19T21:06:40.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:39.202003+0000 mon.a (mon.0) 1557 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-21", "mode": "writeback"}]': finished 2024-02-19T21:06:40.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:39.202072+0000 mon.a (mon.0) 1558 : audit [INF] from='client.49211 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:40.262 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:39.202218+0000 mon.a (mon.0) 1559 : audit [INF] from='client.48043 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-02-19T21:06:40.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:39.202287+0000 mon.a (mon.0) 1560 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-02-19T21:06:40.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:39.205468+0000 mon.c (mon.2) 234 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:40.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:39.222448+0000 mon.a (mon.0) 1561 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:40.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:39.315805+0000 mon.c (mon.2) 235 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:40.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:39.330553+0000 mon.c (mon.2) 236 : audit [INF] from='client.? v1:172.21.15.163:0/2232672904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:06:40.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:39.507739+0000 mon.a (mon.0) 1562 : audit [INF] from='client.49223 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:06:40.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:39.838397+0000 mon.c (mon.2) 237 : audit [DBG] from='client.? v1:172.21.15.163:0/639075907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:40.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:40.199861+0000 mon.a (mon.0) 1563 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:06:40.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:40.211063+0000 mon.a (mon.0) 1564 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi163-83845-31"}]': finished 2024-02-19T21:06:40.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:40.211295+0000 mon.a (mon.0) 1565 : audit [INF] from='client.49223 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:06:40.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:40.211356+0000 mon.a (mon.0) 1566 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-02-19T21:06:40.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:40.216586+0000 mon.c (mon.2) 238 : audit [INF] from='client.? v1:172.21.15.163:0/1861323793' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:40.263 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:40.219807+0000 mon.a (mon.0) 1567 : audit [INF] from='client.49229 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:40.566 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:06:40.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:39.146615+0000 mgr.y (mgr.14550) 245 : cluster [DBG] pgmap v299: 320 pgs: 1 clean+premerge+peered, 7 active+clean+snaptrim, 96 unknown, 12 active+clean+snaptrim_wait, 204 active+clean; 9.6 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:40.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:39.201933+0000 mon.a (mon.0) 1556 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi163-83845-31"}]': finished 2024-02-19T21:06:40.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:39.202003+0000 mon.a (mon.0) 1557 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-21", "mode": "writeback"}]': finished 2024-02-19T21:06:40.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:39.202072+0000 mon.a (mon.0) 1558 : audit [INF] from='client.49211 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:40.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:39.202218+0000 mon.a (mon.0) 1559 : audit [INF] from='client.48043 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-02-19T21:06:40.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:39.202287+0000 mon.a (mon.0) 1560 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-02-19T21:06:40.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:39.205468+0000 mon.c (mon.2) 234 : audit [INF] from='client.? v1:172.21.15.163:0/2827260261' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:40.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:39.222448+0000 mon.a (mon.0) 1561 : audit [INF] from='client.49199 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi163-83845-31"}]: dispatch 2024-02-19T21:06:40.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:39.315805+0000 mon.c (mon.2) 235 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:40.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:39.330553+0000 mon.c (mon.2) 236 : audit [INF] from='client.? v1:172.21.15.163:0/2232672904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:06:40.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:39.507739+0000 mon.a (mon.0) 1562 : audit [INF] from='client.49223 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:06:40.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:39.838397+0000 mon.c (mon.2) 237 : audit [DBG] from='client.? v1:172.21.15.163:0/639075907' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:40.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:40.199861+0000 mon.a (mon.0) 1563 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:06:40.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:40.211063+0000 mon.a (mon.0) 1564 : audit [INF] from='client.49199 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi163-83845-31"}]': finished 2024-02-19T21:06:40.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:40.211295+0000 mon.a (mon.0) 1565 : audit [INF] from='client.49223 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:06:40.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:40.211356+0000 mon.a (mon.0) 1566 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-02-19T21:06:40.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:40.216586+0000 mon.c (mon.2) 238 : audit [INF] from='client.? v1:172.21.15.163:0/1861323793' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:40.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:40.219807+0000 mon.a (mon.0) 1567 : audit [INF] from='client.49229 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:41.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:40.176498+0000 mgr.y (mgr.14550) 246 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:41.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:40.252033+0000 mon.b (mon.1) 346 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:40.252681+0000 mon.a (mon.0) 1568 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.373 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:41.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:41.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:41.477 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:41.492 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:40.176498+0000 mgr.y (mgr.14550) 246 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:40.252033+0000 mon.b (mon.1) 346 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:40.252681+0000 mon.a (mon.0) 1568 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:40.253687+0000 mon.b (mon.1) 347 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:40.253956+0000 mon.a (mon.0) 1569 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:40.254401+0000 mon.b (mon.1) 348 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi163-83845-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:40.254655+0000 mon.a (mon.0) 1570 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi163-83845-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:40.277024+0000 mon.b (mon.1) 349 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:40.277418+0000 mon.a (mon.0) 1571 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:40.316516+0000 mon.c (mon.2) 239 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:41.147950+0000 mon.b (mon.1) 350 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-19T21:06:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:41.149168+0000 mon.a (mon.0) 1572 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-19T21:06:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:40.253687+0000 mon.b (mon.1) 347 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:40.253956+0000 mon.a (mon.0) 1569 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:40.254401+0000 mon.b (mon.1) 348 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi163-83845-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:40.254655+0000 mon.a (mon.0) 1570 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi163-83845-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:40.277024+0000 mon.b (mon.1) 349 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:40.277418+0000 mon.a (mon.0) 1571 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:40.316516+0000 mon.c (mon.2) 239 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:41.147950+0000 mon.b (mon.1) 350 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-19T21:06:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:41.149168+0000 mon.a (mon.0) 1572 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-19T21:06:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:41.209479+0000 mon.a (mon.0) 1573 : audit [INF] from='client.49229 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:41.209641+0000 mon.a (mon.0) 1574 : audit [INF] from='client.48064 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi163-83845-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:41.209722+0000 mon.a (mon.0) 1575 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:41.209776+0000 mon.a (mon.0) 1576 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-02-19T21:06:41.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:41.209837+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-02-19T21:06:41.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:41.215257+0000 mon.b (mon.1) 351 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi163-83845-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:41.215772+0000 mon.a (mon.0) 1578 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi163-83845-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:41.215966+0000 mon.b (mon.1) 352 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21"}]: dispatch 2024-02-19T21:06:41.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:41.216413+0000 mon.a (mon.0) 1579 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21"}]: dispatch 2024-02-19T21:06:41.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:41.209479+0000 mon.a (mon.0) 1573 : audit [INF] from='client.49229 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:41.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:41.209641+0000 mon.a (mon.0) 1574 : audit [INF] from='client.48064 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi163-83845-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:41.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:41.209722+0000 mon.a (mon.0) 1575 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:41.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:41.209776+0000 mon.a (mon.0) 1576 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-02-19T21:06:41.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:41.209837+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-02-19T21:06:41.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:41.215257+0000 mon.b (mon.1) 351 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi163-83845-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:41.215772+0000 mon.a (mon.0) 1578 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi163-83845-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:41.215966+0000 mon.b (mon.1) 352 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21"}]: dispatch 2024-02-19T21:06:41.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:41.216413+0000 mon.a (mon.0) 1579 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21"}]: dispatch 2024-02-19T21:06:41.507 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:41.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:40.176498+0000 mgr.y (mgr.14550) 246 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:41.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:40.252033+0000 mon.b (mon.1) 346 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:40.252681+0000 mon.a (mon.0) 1568 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:40.253687+0000 mon.b (mon.1) 347 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:40.253956+0000 mon.a (mon.0) 1569 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:40.254401+0000 mon.b (mon.1) 348 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi163-83845-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:41.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:40.254655+0000 mon.a (mon.0) 1570 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi163-83845-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:41.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:40.277024+0000 mon.b (mon.1) 349 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:41.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:40.277418+0000 mon.a (mon.0) 1571 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:41.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:40.316516+0000 mon.c (mon.2) 239 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:41.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:41.147950+0000 mon.b (mon.1) 350 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-19T21:06:41.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:41.149168+0000 mon.a (mon.0) 1572 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-02-19T21:06:41.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:41.209479+0000 mon.a (mon.0) 1573 : audit [INF] from='client.49229 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi163-83854-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:41.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:41.209641+0000 mon.a (mon.0) 1574 : audit [INF] from='client.48064 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi163-83845-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:41.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:41.209722+0000 mon.a (mon.0) 1575 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:41.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:41.209776+0000 mon.a (mon.0) 1576 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-02-19T21:06:41.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:41.209837+0000 mon.a (mon.0) 1577 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-02-19T21:06:41.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:41.215257+0000 mon.b (mon.1) 351 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi163-83845-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:41.215772+0000 mon.a (mon.0) 1578 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi163-83845-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:41.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:41.215966+0000 mon.b (mon.1) 352 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21"}]: dispatch 2024-02-19T21:06:41.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:41.216413+0000 mon.a (mon.0) 1579 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21"}]: dispatch 2024-02-19T21:06:41.708 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:41.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:41.823 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:41.851 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:42.271 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:06:42.271 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:06:42.500 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:42.502 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:42.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:42.604 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:42.636 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:42.657 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:42.659 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:42.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:42.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:41.147358+0000 mgr.y (mgr.14550) 247 : cluster [DBG] pgmap v302: 351 pgs: 6 creating+peering, 38 peering, 6 active+clean+snaptrim, 32 unknown, 12 active+clean+snaptrim_wait, 257 active+clean; 9.5 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-02-19T21:06:42.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:41.227879+0000 mon.a (mon.0) 1580 : cluster [WRN] Health check failed: Reduced data availability: 16 pgs inactive, 23 pgs peering (PG_AVAILABILITY) 2024-02-19T21:06:42.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:41.317477+0000 mon.c (mon.2) 240 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:42.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:42.205445+0000 mon.a (mon.0) 1581 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:42.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:42.318258+0000 mon.c (mon.2) 241 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:42.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:41.147358+0000 mgr.y (mgr.14550) 247 : cluster [DBG] pgmap v302: 351 pgs: 6 creating+peering, 38 peering, 6 active+clean+snaptrim, 32 unknown, 12 active+clean+snaptrim_wait, 257 active+clean; 9.5 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-02-19T21:06:42.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:41.227879+0000 mon.a (mon.0) 1580 : cluster [WRN] Health check failed: Reduced data availability: 16 pgs inactive, 23 pgs peering (PG_AVAILABILITY) 2024-02-19T21:06:42.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:41.317477+0000 mon.c (mon.2) 240 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:42.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:42.205445+0000 mon.a (mon.0) 1581 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:42.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:42.318258+0000 mon.c (mon.2) 241 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:42.760 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:42.788 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:42.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:41.147358+0000 mgr.y (mgr.14550) 247 : cluster [DBG] pgmap v302: 351 pgs: 6 creating+peering, 38 peering, 6 active+clean+snaptrim, 32 unknown, 12 active+clean+snaptrim_wait, 257 active+clean; 9.5 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-02-19T21:06:42.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:41.227879+0000 mon.a (mon.0) 1580 : cluster [WRN] Health check failed: Reduced data availability: 16 pgs inactive, 23 pgs peering (PG_AVAILABILITY) 2024-02-19T21:06:42.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:41.317477+0000 mon.c (mon.2) 240 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:42.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:42.205445+0000 mon.a (mon.0) 1581 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:42.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:42.318258+0000 mon.c (mon.2) 241 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:42.921 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:06:42.937 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:42.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:42.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:42.945 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-19T21:06:43.045 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:43.072 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:43.091 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:43.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:43.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:43.195 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:43.215 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:43.239 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:43.260 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:43.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:43.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:43.364 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:43.393 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:43.415 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:43.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:43.480 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:43.498 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:42.448369+0000 mon.a (mon.0) 1582 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21"}]': finished 2024-02-19T21:06:43.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:42.448497+0000 mon.a (mon.0) 1583 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-02-19T21:06:43.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:43.211952+0000 mon.a (mon.0) 1584 : audit [INF] from='client.48064 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi163-83845-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi163-83845-32"}]': finished 2024-02-19T21:06:43.499 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:43.212097+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-02-19T21:06:43.519 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:43.548 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:43.720 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:06:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:06:43] "GET /metrics HTTP/1.1" 200 194014 "" "Prometheus/2.33.4" 2024-02-19T21:06:43.720 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:42.448369+0000 mon.a (mon.0) 1582 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21"}]': finished 2024-02-19T21:06:43.721 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:42.448497+0000 mon.a (mon.0) 1583 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-02-19T21:06:43.721 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:43.211952+0000 mon.a (mon.0) 1584 : audit [INF] from='client.48064 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi163-83845-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi163-83845-32"}]': finished 2024-02-19T21:06:43.721 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:43.212097+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-02-19T21:06:43.721 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:42.448369+0000 mon.a (mon.0) 1582 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-21"}]': finished 2024-02-19T21:06:43.721 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:42.448497+0000 mon.a (mon.0) 1583 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-02-19T21:06:43.721 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:43.211952+0000 mon.a (mon.0) 1584 : audit [INF] from='client.48064 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi163-83845-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi163-83845-32"}]': finished 2024-02-19T21:06:43.721 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:43.212097+0000 mon.a (mon.0) 1585 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-02-19T21:06:44.161 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:06:44.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:43.147958+0000 mgr.y (mgr.14550) 248 : cluster [DBG] pgmap v305: 319 pgs: 6 creating+peering, 38 peering, 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 257 active+clean; 9.5 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-02-19T21:06:44.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:43.318991+0000 mon.c (mon.2) 242 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:44.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:44.186831+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-02-19T21:06:44.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:44.191240+0000 mon.b (mon.1) 353 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:44.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:44.199701+0000 mon.a (mon.0) 1587 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:44.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:44.319818+0000 mon.c (mon.2) 243 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:44.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:43.147958+0000 mgr.y (mgr.14550) 248 : cluster [DBG] pgmap v305: 319 pgs: 6 creating+peering, 38 peering, 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 257 active+clean; 9.5 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-02-19T21:06:44.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:43.318991+0000 mon.c (mon.2) 242 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:44.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:44.186831+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-02-19T21:06:44.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:44.191240+0000 mon.b (mon.1) 353 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:44.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:44.199701+0000 mon.a (mon.0) 1587 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:44.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:44.319818+0000 mon.c (mon.2) 243 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:44.949 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:43.147958+0000 mgr.y (mgr.14550) 248 : cluster [DBG] pgmap v305: 319 pgs: 6 creating+peering, 38 peering, 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 257 active+clean; 9.5 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-02-19T21:06:44.949 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:43.318991+0000 mon.c (mon.2) 242 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:44.949 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:44.186831+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-02-19T21:06:44.949 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:44.191240+0000 mon.b (mon.1) 353 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:44.949 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:44.199701+0000 mon.a (mon.0) 1587 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:44.949 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:44.319818+0000 mon.c (mon.2) 243 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:45.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.527+0000 7f6456809700 1 -- 172.21.15.163:0/2214625352 >> v1:172.21.15.163:6789/0 conn(0x7f64500fc870 legacy=0x7f64500fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.527+0000 7f6456809700 1 -- 172.21.15.163:0/2214625352 shutdown_connections 2024-02-19T21:06:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.527+0000 7f6456809700 1 -- 172.21.15.163:0/2214625352 >> 172.21.15.163:0/2214625352 conn(0x7f64500f2470 msgr2=0x7f64500f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.527+0000 7f6456809700 1 -- 172.21.15.163:0/2214625352 shutdown_connections 2024-02-19T21:06:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.527+0000 7f6456809700 1 -- 172.21.15.163:0/2214625352 wait complete. 2024-02-19T21:06:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.528+0000 7f6456809700 1 librados: starting msgr at 2024-02-19T21:06:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.528+0000 7f6456809700 1 librados: starting objecter 2024-02-19T21:06:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.528+0000 7f6456809700 1 Processor -- start 2024-02-19T21:06:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.528+0000 7f6456809700 1 -- start start 2024-02-19T21:06:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.528+0000 7f6456809700 1 librados: setting wanted keys 2024-02-19T21:06:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.528+0000 7f6456809700 1 librados: calling monclient init 2024-02-19T21:06:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.528+0000 7f6456809700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f64501172f0 con 0x7f64500fc870 2024-02-19T21:06:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.528+0000 7f6456809700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f64501184d0 con 0x7f64501031b0 2024-02-19T21:06:45.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.528+0000 7f6456809700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f64501196d0 con 0x7f64500ff670 2024-02-19T21:06:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644ffff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f64500fc870 0x7f6450060b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:54146/0 (socket says 172.21.15.163:54146) 2024-02-19T21:06:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644ffff700 1 -- 172.21.15.163:0/799793578 learned_addr learned my addr 172.21.15.163:0/799793578 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 10 client.?.objecter ms_handle_connect 0x7f64500ff670 2024-02-19T21:06:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 10 client.?.objecter ms_handle_connect 0x7f64500fc870 2024-02-19T21:06:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 10 client.?.objecter ms_handle_connect 0x7f64501031b0 2024-02-19T21:06:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247239221 0 0) 0x7f64501196d0 con 0x7f64500ff670 2024-02-19T21:06:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64380033f0 con 0x7f64500ff670 2024-02-19T21:06:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803569528 0 0) 0x7f64501184d0 con 0x7f64501031b0 2024-02-19T21:06:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64501196d0 con 0x7f64501031b0 2024-02-19T21:06:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772619494 0 0) 0x7f64501172f0 con 0x7f64500fc870 2024-02-19T21:06:45.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64501184d0 con 0x7f64500fc870 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 893472624 0 0) 0x7f64501196d0 con 0x7f64501031b0 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64501172f0 con 0x7f64501031b0 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6448001980 con 0x7f64501031b0 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2426583335 0 0) 0x7f64501184d0 con 0x7f64500fc870 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64501196d0 con 0x7f64500fc870 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3915859706 0 0) 0x7f64380033f0 con 0x7f64500ff670 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64501184d0 con 0x7f64500ff670 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6440001fa0 con 0x7f64500fc870 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6444001840 con 0x7f64500ff670 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3079759871 0 0) 0x7f64501172f0 con 0x7f64501031b0 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 >> v1:172.21.15.163:6790/0 conn(0x7f64500ff670 legacy=0x7f64500612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.529+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 >> v1:172.21.15.163:6789/0 conn(0x7f64500fc870 legacy=0x7f6450060b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.530+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645011a8d0 con 0x7f64501031b0 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.530+0000 7f6456809700 1 -- 172.21.15.163:0/799793578 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64501174e0 con 0x7f64501031b0 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.530+0000 7f6456809700 10 client.48076.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:45.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.530+0000 7f6456809700 1 -- 172.21.15.163:0/799793578 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6450117950 con 0x7f64501031b0 2024-02-19T21:06:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.530+0000 7f6456809700 1 librados: init done 2024-02-19T21:06:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.530+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6448002620 con 0x7f64501031b0 2024-02-19T21:06:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.530+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6448003620 con 0x7f64501031b0 2024-02-19T21:06:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.531+0000 7f6456809700 1 -- 172.21.15.163:0/799793578 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f643c004f20 con 0x7f64501031b0 2024-02-19T21:06:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.531+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6448003780 con 0x7f64501031b0 2024-02-19T21:06:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.532+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(225..225 src has 43..225) v4 ==== 9118+0+0 (unknown 3788899014 0 0) 0x7f6448076b40 con 0x7f64501031b0 2024-02-19T21:06:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.532+0000 7f644d7fa700 10 client.48076.objecter ms_dispatch 0x7f6450001550 osd_map(225..225 src has 43..225) v4 2024-02-19T21:06:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.532+0000 7f644d7fa700 3 client.48076.objecter handle_osd_map got epochs [225,225] > 0 2024-02-19T21:06:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.532+0000 7f644d7fa700 3 client.48076.objecter handle_osd_map decoding full epoch 225 2024-02-19T21:06:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.532+0000 7f644d7fa700 20 client.48076.objecter dump_active .. 0 homeless 2024-02-19T21:06:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.532+0000 7f644d7fa700 10 client.48076.objecter ms_handle_connect 0x7f6438063db0 2024-02-19T21:06:45.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.536+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6448049b30 con 0x7f64501031b0 2024-02-19T21:06:45.549 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:45.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:45.723+0000 7f6456809700 1 -- 172.21.15.163:0/799793578 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f643c0051d0 con 0x7f64501031b0 2024-02-19T21:06:45.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:45.149687+0000 mon.b (mon.1) 354 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-19T21:06:45.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:45.150633+0000 mon.a (mon.0) 1588 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-19T21:06:45.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:45.188546+0000 mon.a (mon.0) 1589 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:45.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:45.188742+0000 mon.a (mon.0) 1590 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-02-19T21:06:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:45.188804+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-02-19T21:06:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:45.207082+0000 mon.b (mon.1) 355 : audit [INF] from='client.? v1:172.21.15.163:0/307017721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:45.209060+0000 mon.a (mon.0) 1592 : audit [INF] from='client.48070 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:45.212075+0000 mon.b (mon.1) 356 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:45.213032+0000 mon.a (mon.0) 1593 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:45.320652+0000 mon.c (mon.2) 244 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:45.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:45.149687+0000 mon.b (mon.1) 354 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-19T21:06:45.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:45.150633+0000 mon.a (mon.0) 1588 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-19T21:06:45.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:45.188546+0000 mon.a (mon.0) 1589 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:45.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:45.188742+0000 mon.a (mon.0) 1590 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-02-19T21:06:45.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:45.188804+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-02-19T21:06:45.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:45.207082+0000 mon.b (mon.1) 355 : audit [INF] from='client.? v1:172.21.15.163:0/307017721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:45.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:45.209060+0000 mon.a (mon.0) 1592 : audit [INF] from='client.48070 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:45.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:45.212075+0000 mon.b (mon.1) 356 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:45.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:45.213032+0000 mon.a (mon.0) 1593 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:45.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:45.320652+0000 mon.c (mon.2) 244 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:45.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:45.149687+0000 mon.b (mon.1) 354 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-19T21:06:45.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:45.150633+0000 mon.a (mon.0) 1588 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-02-19T21:06:45.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:45.188546+0000 mon.a (mon.0) 1589 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:45.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:45.188742+0000 mon.a (mon.0) 1590 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-02-19T21:06:45.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:45.188804+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-02-19T21:06:45.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:45.207082+0000 mon.b (mon.1) 355 : audit [INF] from='client.? v1:172.21.15.163:0/307017721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:45.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:45.209060+0000 mon.a (mon.0) 1592 : audit [INF] from='client.48070 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:45.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:45.212075+0000 mon.b (mon.1) 356 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:45.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:45.213032+0000 mon.a (mon.0) 1593 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:45.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:45.320652+0000 mon.c (mon.2) 244 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:45.999 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:46.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:46.104 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:06:46.136 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:46.158 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:06:46.160 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:46.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:46.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:46.192+0000 7f644d7fa700 1 -- 172.21.15.163:0/799793578 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v226) v1 ==== 91+0+0 (unknown 2974291742 0 0) 0x7f6448047f40 con 0x7f64501031b0 2024-02-19T21:06:46.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:46.198+0000 7f6456809700 10 librados: watch_flush enter 2024-02-19T21:06:46.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:46.198+0000 7f6456809700 10 librados: watch_flush exit 2024-02-19T21:06:46.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:46.198+0000 7f6456809700 20 client.48076.objecter shutdown clearing up homeless session... 2024-02-19T21:06:46.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:46.198+0000 7f6456809700 10 client.48076.objecter successfully canceled tick 2024-02-19T21:06:46.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:46.198+0000 7f6456809700 1 -- 172.21.15.163:0/799793578 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6438063db0 legacy=0x7f6438066270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:46.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:46.198+0000 7f6456809700 1 -- 172.21.15.163:0/799793578 >> v1:172.21.15.193:6789/0 conn(0x7f64501031b0 legacy=0x7f64501159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:46.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:46.202+0000 7f6456809700 1 -- 172.21.15.163:0/799793578 shutdown_connections 2024-02-19T21:06:46.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:46.202+0000 7f6456809700 1 -- 172.21.15.163:0/799793578 >> 172.21.15.163:0/799793578 conn(0x7f64500f2470 msgr2=0x7f6450106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:46.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:46.203+0000 7f6456809700 1 -- 172.21.15.163:0/799793578 shutdown_connections 2024-02-19T21:06:46.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:46.204+0000 7f6456809700 1 -- 172.21.15.163:0/799793578 wait complete. 2024-02-19T21:06:46.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:46.204+0000 7f6456809700 1 librados: shutdown 2024-02-19T21:06:46.203 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:06:46.261 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:06:46.291 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:46.310 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:06:46.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:46.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:46.357 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:06:46.357 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:06:46.358 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:06:46.413 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:06:46.441 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:46.460 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:06:46.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:46.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:06:46.564 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:46.593 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:46.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:45.148968+0000 mgr.y (mgr.14550) 249 : cluster [DBG] pgmap v308: 262 pgs: 32 creating+peering, 5 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 221 active+clean; 9.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:06:46.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:45.505779+0000 mon.a (mon.0) 1594 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:06:46.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:45.505809+0000 mon.a (mon.0) 1595 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 16 pgs inactive, 23 pgs peering) 2024-02-19T21:06:46.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:45.725423+0000 mon.b (mon.1) 357 : audit [INF] from='client.? v1:172.21.15.163:0/799793578' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:06:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:45.725765+0000 mon.a (mon.0) 1596 : audit [INF] from='client.48076 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:06:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:46.186034+0000 mon.a (mon.0) 1597 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:06:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:46.192898+0000 mon.a (mon.0) 1598 : audit [INF] from='client.48070 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:46.192987+0000 mon.a (mon.0) 1599 : audit [INF] from='client.48064 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi163-83845-32"}]': finished 2024-02-19T21:06:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:46.193065+0000 mon.a (mon.0) 1600 : audit [INF] from='client.48076 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:06:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:46.193125+0000 mon.a (mon.0) 1601 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-02-19T21:06:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:46.196969+0000 mon.b (mon.1) 358 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:46.198074+0000 mon.a (mon.0) 1602 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:46.229006+0000 mon.b (mon.1) 359 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:46.229538+0000 mon.a (mon.0) 1603 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:46.321443+0000 mon.c (mon.2) 245 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:46.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:45.148968+0000 mgr.y (mgr.14550) 249 : cluster [DBG] pgmap v308: 262 pgs: 32 creating+peering, 5 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 221 active+clean; 9.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:06:46.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:45.505779+0000 mon.a (mon.0) 1594 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:06:46.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:45.505809+0000 mon.a (mon.0) 1595 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 16 pgs inactive, 23 pgs peering) 2024-02-19T21:06:46.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:45.725423+0000 mon.b (mon.1) 357 : audit [INF] from='client.? v1:172.21.15.163:0/799793578' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:06:46.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:45.725765+0000 mon.a (mon.0) 1596 : audit [INF] from='client.48076 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:06:46.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:46.186034+0000 mon.a (mon.0) 1597 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:06:46.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:46.192898+0000 mon.a (mon.0) 1598 : audit [INF] from='client.48070 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:46.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:46.192987+0000 mon.a (mon.0) 1599 : audit [INF] from='client.48064 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi163-83845-32"}]': finished 2024-02-19T21:06:46.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:46.193065+0000 mon.a (mon.0) 1600 : audit [INF] from='client.48076 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:06:46.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:46.193125+0000 mon.a (mon.0) 1601 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-02-19T21:06:46.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:46.196969+0000 mon.b (mon.1) 358 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:46.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:46.198074+0000 mon.a (mon.0) 1602 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:46.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:46.229006+0000 mon.b (mon.1) 359 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:46.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:46.229538+0000 mon.a (mon.0) 1603 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:46.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:46.321443+0000 mon.c (mon.2) 245 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:46.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:45.148968+0000 mgr.y (mgr.14550) 249 : cluster [DBG] pgmap v308: 262 pgs: 32 creating+peering, 5 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 221 active+clean; 9.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:06:46.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:45.505779+0000 mon.a (mon.0) 1594 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:06:46.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:45.505809+0000 mon.a (mon.0) 1595 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 16 pgs inactive, 23 pgs peering) 2024-02-19T21:06:46.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:45.725423+0000 mon.b (mon.1) 357 : audit [INF] from='client.? v1:172.21.15.163:0/799793578' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:06:46.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:45.725765+0000 mon.a (mon.0) 1596 : audit [INF] from='client.48076 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:06:46.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:46.186034+0000 mon.a (mon.0) 1597 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:06:46.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:46.192898+0000 mon.a (mon.0) 1598 : audit [INF] from='client.48070 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:46.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:46.192987+0000 mon.a (mon.0) 1599 : audit [INF] from='client.48064 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi163-83845-32"}]': finished 2024-02-19T21:06:46.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:46.193065+0000 mon.a (mon.0) 1600 : audit [INF] from='client.48076 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:06:46.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:46.193125+0000 mon.a (mon.0) 1601 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-02-19T21:06:46.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:46.196969+0000 mon.b (mon.1) 358 : audit [INF] from='client.? v1:172.21.15.163:0/2088533578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:46.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:46.198074+0000 mon.a (mon.0) 1602 : audit [INF] from='client.48064 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi163-83845-32"}]: dispatch 2024-02-19T21:06:46.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:46.229006+0000 mon.b (mon.1) 359 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:46.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:46.229538+0000 mon.a (mon.0) 1603 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:46.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:46.321443+0000 mon.c (mon.2) 245 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:47.108 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:06:47.108 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:06:47.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:46.311114+0000 osd.2 (osd.2) 97 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:06:47.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:46.312015+0000 osd.2 (osd.2) 98 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:06:47.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:46.811949+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:06:47.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:46.813810+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:06:47.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:46.814199+0000 osd.3 (osd.3) 113 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:06:47.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:46.816116+0000 osd.3 (osd.3) 114 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:06:47.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:47.194383+0000 mon.a (mon.0) 1604 : audit [INF] from='client.48064 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi163-83845-32"}]': finished 2024-02-19T21:06:47.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:47.194491+0000 mon.a (mon.0) 1605 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:47.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:47.194601+0000 mon.a (mon.0) 1606 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-02-19T21:06:47.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:47.199900+0000 mon.b (mon.1) 360 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-23"}]: dispatch 2024-02-19T21:06:47.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:47.200580+0000 mon.c (mon.2) 246 : audit [INF] from='client.? v1:172.21.15.163:0/891163250' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:47.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:47.202548+0000 mon.a (mon.0) 1607 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-23"}]: dispatch 2024-02-19T21:06:47.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:47.203583+0000 mon.a (mon.0) 1608 : audit [INF] from='client.49256 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:47.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:47.220498+0000 mon.c (mon.2) 247 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:47.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:47.220815+0000 mon.a (mon.0) 1609 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:47.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:46.311114+0000 osd.2 (osd.2) 97 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:06:47.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:46.312015+0000 osd.2 (osd.2) 98 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:06:47.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:46.811949+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:06:47.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:46.813810+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:06:47.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:46.814199+0000 osd.3 (osd.3) 113 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:06:47.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:46.816116+0000 osd.3 (osd.3) 114 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:06:47.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:47.194383+0000 mon.a (mon.0) 1604 : audit [INF] from='client.48064 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi163-83845-32"}]': finished 2024-02-19T21:06:47.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:47.194491+0000 mon.a (mon.0) 1605 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:47.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:47.194601+0000 mon.a (mon.0) 1606 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-02-19T21:06:47.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:47.199900+0000 mon.b (mon.1) 360 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-23"}]: dispatch 2024-02-19T21:06:47.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:47.200580+0000 mon.c (mon.2) 246 : audit [INF] from='client.? v1:172.21.15.163:0/891163250' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:47.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:47.202548+0000 mon.a (mon.0) 1607 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-23"}]: dispatch 2024-02-19T21:06:47.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:47.203583+0000 mon.a (mon.0) 1608 : audit [INF] from='client.49256 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:47.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:47.220498+0000 mon.c (mon.2) 247 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:47.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:47.220815+0000 mon.a (mon.0) 1609 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:47.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:47.221512+0000 mon.c (mon.2) 248 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:47.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:47.221746+0000 mon.a (mon.0) 1610 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:47.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:47.222098+0000 mon.c (mon.2) 249 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi163-83845-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:47.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:47.222339+0000 mon.a (mon.0) 1611 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi163-83845-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:47.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:47.322317+0000 mon.c (mon.2) 250 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:47.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:47.221512+0000 mon.c (mon.2) 248 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:47.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:47.221746+0000 mon.a (mon.0) 1610 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:47.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:47.222098+0000 mon.c (mon.2) 249 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi163-83845-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:47.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:47.222339+0000 mon.a (mon.0) 1611 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi163-83845-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:47.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:47.322317+0000 mon.c (mon.2) 250 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:47.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:46.311114+0000 osd.2 (osd.2) 97 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:06:47.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:46.312015+0000 osd.2 (osd.2) 98 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:06:47.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:46.811949+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:06:47.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:46.813810+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:06:47.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:46.814199+0000 osd.3 (osd.3) 113 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:06:47.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:46.816116+0000 osd.3 (osd.3) 114 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:06:47.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:47.194383+0000 mon.a (mon.0) 1604 : audit [INF] from='client.48064 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi163-83845-32"}]': finished 2024-02-19T21:06:47.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:47.194491+0000 mon.a (mon.0) 1605 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:47.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:47.194601+0000 mon.a (mon.0) 1606 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-02-19T21:06:47.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:47.199900+0000 mon.b (mon.1) 360 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-23"}]: dispatch 2024-02-19T21:06:47.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:47.200580+0000 mon.c (mon.2) 246 : audit [INF] from='client.? v1:172.21.15.163:0/891163250' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:47.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:47.202548+0000 mon.a (mon.0) 1607 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-23"}]: dispatch 2024-02-19T21:06:47.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:47.203583+0000 mon.a (mon.0) 1608 : audit [INF] from='client.49256 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:47.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:47.220498+0000 mon.c (mon.2) 247 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:47.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:47.220815+0000 mon.a (mon.0) 1609 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:47.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:47.221512+0000 mon.c (mon.2) 248 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:47.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:47.221746+0000 mon.a (mon.0) 1610 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:47.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:47.222098+0000 mon.c (mon.2) 249 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi163-83845-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:47.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:47.222339+0000 mon.a (mon.0) 1611 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi163-83845-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:47.770 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:47.322317+0000 mon.c (mon.2) 250 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:48.066 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:06:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:06:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:06:48.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:46.621192+0000 osd.1 (osd.1) 125 : cluster [DBG] 8.3 scrub starts 2024-02-19T21:06:48.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:46.624677+0000 osd.1 (osd.1) 126 : cluster [DBG] 8.3 scrub ok 2024-02-19T21:06:48.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:47.101917+0000 osd.5 (osd.5) 102 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:06:48.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:47.102110+0000 osd.5 (osd.5) 103 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:06:48.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:47.149676+0000 mgr.y (mgr.14550) 250 : cluster [DBG] pgmap v311: 286 pgs: 32 creating+peering, 5 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 213 active+clean; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:06:48.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:47.537547+0000 osd.5 (osd.5) 104 : cluster [DBG] 4.15 scrub starts 2024-02-19T21:06:48.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:47.541008+0000 osd.5 (osd.5) 105 : cluster [DBG] 4.15 scrub ok 2024-02-19T21:06:48.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:48.194026+0000 mon.a (mon.0) 1612 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-23"}]': finished 2024-02-19T21:06:48.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:48.194119+0000 mon.a (mon.0) 1613 : audit [INF] from='client.49256 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:48.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:48.194223+0000 mon.a (mon.0) 1614 : audit [INF] from='client.49262 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi163-83845-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:48.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:48.194308+0000 mon.a (mon.0) 1615 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-02-19T21:06:48.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:48.197070+0000 mon.c (mon.2) 251 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi163-83845-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:48.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:48.197566+0000 mon.a (mon.0) 1616 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi163-83845-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:48.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:48.203740+0000 mon.b (mon.1) 361 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-23", "mode": "writeback"}]: dispatch 2024-02-19T21:06:48.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:48.208845+0000 mon.a (mon.0) 1617 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-23", "mode": "writeback"}]: dispatch 2024-02-19T21:06:48.826 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:48.322962+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:48.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:46.621192+0000 osd.1 (osd.1) 125 : cluster [DBG] 8.3 scrub starts 2024-02-19T21:06:48.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:46.624677+0000 osd.1 (osd.1) 126 : cluster [DBG] 8.3 scrub ok 2024-02-19T21:06:48.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:47.101917+0000 osd.5 (osd.5) 102 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:06:48.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:47.102110+0000 osd.5 (osd.5) 103 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:06:48.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:47.149676+0000 mgr.y (mgr.14550) 250 : cluster [DBG] pgmap v311: 286 pgs: 32 creating+peering, 5 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 213 active+clean; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:06:48.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:47.537547+0000 osd.5 (osd.5) 104 : cluster [DBG] 4.15 scrub starts 2024-02-19T21:06:48.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:47.541008+0000 osd.5 (osd.5) 105 : cluster [DBG] 4.15 scrub ok 2024-02-19T21:06:48.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:48.194026+0000 mon.a (mon.0) 1612 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-23"}]': finished 2024-02-19T21:06:48.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:48.194119+0000 mon.a (mon.0) 1613 : audit [INF] from='client.49256 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:48.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:48.194223+0000 mon.a (mon.0) 1614 : audit [INF] from='client.49262 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi163-83845-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:48.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:48.194308+0000 mon.a (mon.0) 1615 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-02-19T21:06:48.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:48.197070+0000 mon.c (mon.2) 251 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi163-83845-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:48.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:48.197566+0000 mon.a (mon.0) 1616 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi163-83845-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:48.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:48.203740+0000 mon.b (mon.1) 361 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-23", "mode": "writeback"}]: dispatch 2024-02-19T21:06:48.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:48.208845+0000 mon.a (mon.0) 1617 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-23", "mode": "writeback"}]: dispatch 2024-02-19T21:06:48.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:48.322962+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:48.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:46.621192+0000 osd.1 (osd.1) 125 : cluster [DBG] 8.3 scrub starts 2024-02-19T21:06:48.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:46.624677+0000 osd.1 (osd.1) 126 : cluster [DBG] 8.3 scrub ok 2024-02-19T21:06:48.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:47.101917+0000 osd.5 (osd.5) 102 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:06:48.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:47.102110+0000 osd.5 (osd.5) 103 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:06:48.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:47.149676+0000 mgr.y (mgr.14550) 250 : cluster [DBG] pgmap v311: 286 pgs: 32 creating+peering, 5 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 213 active+clean; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:06:48.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:47.537547+0000 osd.5 (osd.5) 104 : cluster [DBG] 4.15 scrub starts 2024-02-19T21:06:48.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:47.541008+0000 osd.5 (osd.5) 105 : cluster [DBG] 4.15 scrub ok 2024-02-19T21:06:48.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:48.194026+0000 mon.a (mon.0) 1612 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-23"}]': finished 2024-02-19T21:06:48.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:48.194119+0000 mon.a (mon.0) 1613 : audit [INF] from='client.49256 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:48.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:48.194223+0000 mon.a (mon.0) 1614 : audit [INF] from='client.49262 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi163-83845-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:48.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:48.194308+0000 mon.a (mon.0) 1615 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-02-19T21:06:48.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:48.197070+0000 mon.c (mon.2) 251 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi163-83845-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:48.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:48.197566+0000 mon.a (mon.0) 1616 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi163-83845-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:48.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:48.203740+0000 mon.b (mon.1) 361 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-23", "mode": "writeback"}]: dispatch 2024-02-19T21:06:48.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:48.208845+0000 mon.a (mon.0) 1617 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-23", "mode": "writeback"}]: dispatch 2024-02-19T21:06:48.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:48.322962+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:49.143 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:49.148 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:49.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:49.249 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:49.277 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:49.811 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:06:49.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:48.308779+0000 osd.2 (osd.2) 99 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:06:49.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:48.310452+0000 osd.2 (osd.2) 100 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:06:49.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:48.504089+0000 osd.5 (osd.5) 106 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:06:49.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:48.506961+0000 osd.5 (osd.5) 107 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:06:49.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:48.842203+0000 osd.4 (osd.4) 119 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:06:49.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:48.844294+0000 osd.4 (osd.4) 120 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:06:49.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:48.853788+0000 osd.3 (osd.3) 115 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:06:49.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:48.855264+0000 osd.3 (osd.3) 116 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:06:49.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:49.150844+0000 mon.b (mon.1) 362 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-19T21:06:49.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:49.152333+0000 mon.a (mon.0) 1618 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-19T21:06:49.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:49.191817+0000 mon.a (mon.0) 1619 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:49.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:49.323698+0000 mon.c (mon.2) 253 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:49.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:49.439058+0000 mon.a (mon.0) 1620 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-23", "mode": "writeback"}]': finished 2024-02-19T21:06:49.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:49.439175+0000 mon.a (mon.0) 1621 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-02-19T21:06:49.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:49.439240+0000 mon.a (mon.0) 1622 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-02-19T21:06:49.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:49.439918+0000 mon.c (mon.2) 254 : audit [INF] from='client.? v1:172.21.15.163:0/2437101648' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:49.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:49.441461+0000 mon.a (mon.0) 1623 : audit [INF] from='client.49268 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:49.820 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:48.308779+0000 osd.2 (osd.2) 99 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:06:49.820 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:48.310452+0000 osd.2 (osd.2) 100 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:06:49.820 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:48.504089+0000 osd.5 (osd.5) 106 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:06:49.820 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:48.506961+0000 osd.5 (osd.5) 107 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:06:49.820 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:48.842203+0000 osd.4 (osd.4) 119 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:06:49.820 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:48.844294+0000 osd.4 (osd.4) 120 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:06:49.820 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:48.853788+0000 osd.3 (osd.3) 115 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:06:49.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:48.855264+0000 osd.3 (osd.3) 116 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:06:49.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:49.150844+0000 mon.b (mon.1) 362 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-19T21:06:49.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:49.152333+0000 mon.a (mon.0) 1618 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-19T21:06:49.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:49.191817+0000 mon.a (mon.0) 1619 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:49.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:49.323698+0000 mon.c (mon.2) 253 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:49.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:49.439058+0000 mon.a (mon.0) 1620 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-23", "mode": "writeback"}]': finished 2024-02-19T21:06:49.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:49.439175+0000 mon.a (mon.0) 1621 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-02-19T21:06:49.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:49.439240+0000 mon.a (mon.0) 1622 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-02-19T21:06:49.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:49.439918+0000 mon.c (mon.2) 254 : audit [INF] from='client.? v1:172.21.15.163:0/2437101648' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:49.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:49.441461+0000 mon.a (mon.0) 1623 : audit [INF] from='client.49268 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:49.821 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:48.308779+0000 osd.2 (osd.2) 99 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:06:49.821 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:48.310452+0000 osd.2 (osd.2) 100 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:06:49.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:48.504089+0000 osd.5 (osd.5) 106 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:06:49.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:48.506961+0000 osd.5 (osd.5) 107 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:06:49.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:48.842203+0000 osd.4 (osd.4) 119 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:06:49.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:48.844294+0000 osd.4 (osd.4) 120 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:06:49.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:48.853788+0000 osd.3 (osd.3) 115 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:06:49.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:48.855264+0000 osd.3 (osd.3) 116 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:06:49.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:49.150844+0000 mon.b (mon.1) 362 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-19T21:06:49.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:49.152333+0000 mon.a (mon.0) 1618 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-02-19T21:06:49.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:49.191817+0000 mon.a (mon.0) 1619 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:49.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:49.323698+0000 mon.c (mon.2) 253 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:49.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:49.439058+0000 mon.a (mon.0) 1620 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-23", "mode": "writeback"}]': finished 2024-02-19T21:06:49.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:49.439175+0000 mon.a (mon.0) 1621 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-02-19T21:06:49.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:49.439240+0000 mon.a (mon.0) 1622 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-02-19T21:06:49.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:49.439918+0000 mon.c (mon.2) 254 : audit [INF] from='client.? v1:172.21.15.163:0/2437101648' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:49.823 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:49.441461+0000 mon.a (mon.0) 1623 : audit [INF] from='client.49268 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:49.826 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:49.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:49.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:49.931 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:49.958 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:50.242 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-19T21:06:50.310 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:50.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:50.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:50.415 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:06:50.443 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:06:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:06:50.445 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:50.466 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:06:50.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:50.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:06:50.570 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:50.596 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:49.150339+0000 mgr.y (mgr.14550) 251 : cluster [DBG] pgmap v314: 317 pgs: 32 creating+peering, 5 active+clean+snaptrim, 64 unknown, 4 active+clean+snaptrim_wait, 212 active+clean; 8.9 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:49.301557+0000 osd.2 (osd.2) 101 : cluster [DBG] 16.1 scrub starts 2024-02-19T21:06:50.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:49.305113+0000 osd.2 (osd.2) 102 : cluster [DBG] 16.1 scrub ok 2024-02-19T21:06:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:49.687666+0000 osd.7 (osd.7) 113 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:06:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:49.690592+0000 osd.7 (osd.7) 114 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:06:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:50.198211+0000 mon.a (mon.0) 1624 : audit [INF] from='client.49262 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi163-83845-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi163-83845-33"}]': finished 2024-02-19T21:06:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:50.198413+0000 mon.a (mon.0) 1625 : audit [INF] from='client.49268 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:50.198522+0000 mon.a (mon.0) 1626 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-02-19T21:06:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:50.242097+0000 mon.b (mon.1) 363 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:50.242587+0000 mon.a (mon.0) 1627 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:50.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:50.324409+0000 mon.c (mon.2) 255 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:50.821 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:49.150339+0000 mgr.y (mgr.14550) 251 : cluster [DBG] pgmap v314: 317 pgs: 32 creating+peering, 5 active+clean+snaptrim, 64 unknown, 4 active+clean+snaptrim_wait, 212 active+clean; 8.9 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:50.821 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:49.301557+0000 osd.2 (osd.2) 101 : cluster [DBG] 16.1 scrub starts 2024-02-19T21:06:50.821 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:49.305113+0000 osd.2 (osd.2) 102 : cluster [DBG] 16.1 scrub ok 2024-02-19T21:06:50.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:49.687666+0000 osd.7 (osd.7) 113 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:06:50.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:49.690592+0000 osd.7 (osd.7) 114 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:06:50.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:50.198211+0000 mon.a (mon.0) 1624 : audit [INF] from='client.49262 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi163-83845-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi163-83845-33"}]': finished 2024-02-19T21:06:50.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:50.198413+0000 mon.a (mon.0) 1625 : audit [INF] from='client.49268 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:50.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:50.198522+0000 mon.a (mon.0) 1626 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-02-19T21:06:50.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:50.242097+0000 mon.b (mon.1) 363 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:50.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:50.242587+0000 mon.a (mon.0) 1627 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:50.822 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:50.324409+0000 mon.c (mon.2) 255 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:50.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:49.150339+0000 mgr.y (mgr.14550) 251 : cluster [DBG] pgmap v314: 317 pgs: 32 creating+peering, 5 active+clean+snaptrim, 64 unknown, 4 active+clean+snaptrim_wait, 212 active+clean; 8.9 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:50.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:49.301557+0000 osd.2 (osd.2) 101 : cluster [DBG] 16.1 scrub starts 2024-02-19T21:06:50.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:49.305113+0000 osd.2 (osd.2) 102 : cluster [DBG] 16.1 scrub ok 2024-02-19T21:06:50.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:49.687666+0000 osd.7 (osd.7) 113 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:06:50.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:49.690592+0000 osd.7 (osd.7) 114 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:06:50.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:50.198211+0000 mon.a (mon.0) 1624 : audit [INF] from='client.49262 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi163-83845-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi163-83845-33"}]': finished 2024-02-19T21:06:50.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:50.198413+0000 mon.a (mon.0) 1625 : audit [INF] from='client.49268 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:50.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:50.198522+0000 mon.a (mon.0) 1626 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-02-19T21:06:50.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:50.242097+0000 mon.b (mon.1) 363 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:50.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:50.242587+0000 mon.a (mon.0) 1627 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:50.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:50.324409+0000 mon.c (mon.2) 255 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:50.824 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:50.824 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:50.844 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:50.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:50.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:50.951 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:06:50.982 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:51.003 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:06:51.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:51.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:51.106 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:06:51.135 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:06:51.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:51.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:51.258 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:51.286 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:51.676 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:51.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:50.182572+0000 mgr.y (mgr.14550) 252 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:51.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:50.464380+0000 osd.5 (osd.5) 108 : cluster [DBG] 16.f scrub starts 2024-02-19T21:06:51.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:50.467407+0000 osd.5 (osd.5) 109 : cluster [DBG] 16.f scrub ok 2024-02-19T21:06:51.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:50.690773+0000 osd.7 (osd.7) 115 : cluster [DBG] 2.b scrub starts 2024-02-19T21:06:51.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:50.694242+0000 osd.7 (osd.7) 116 : cluster [DBG] 2.b scrub ok 2024-02-19T21:06:51.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:51.201348+0000 mon.a (mon.0) 1628 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:51.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:51.201442+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-02-19T21:06:51.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:51.214376+0000 mon.b (mon.1) 364 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23"}]: dispatch 2024-02-19T21:06:51.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:51.218584+0000 mon.a (mon.0) 1630 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23"}]: dispatch 2024-02-19T21:06:51.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:51.219613+0000 mon.c (mon.2) 256 : audit [INF] from='client.? v1:172.21.15.163:0/1203751048' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:51.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:51.221602+0000 mon.a (mon.0) 1631 : audit [INF] from='client.49274 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:51.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:51.325216+0000 mon.c (mon.2) 257 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:50.182572+0000 mgr.y (mgr.14550) 252 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:50.464380+0000 osd.5 (osd.5) 108 : cluster [DBG] 16.f scrub starts 2024-02-19T21:06:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:50.467407+0000 osd.5 (osd.5) 109 : cluster [DBG] 16.f scrub ok 2024-02-19T21:06:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:50.690773+0000 osd.7 (osd.7) 115 : cluster [DBG] 2.b scrub starts 2024-02-19T21:06:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:50.694242+0000 osd.7 (osd.7) 116 : cluster [DBG] 2.b scrub ok 2024-02-19T21:06:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:51.201348+0000 mon.a (mon.0) 1628 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:51.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:51.201442+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-02-19T21:06:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:51.214376+0000 mon.b (mon.1) 364 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23"}]: dispatch 2024-02-19T21:06:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:51.218584+0000 mon.a (mon.0) 1630 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23"}]: dispatch 2024-02-19T21:06:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:51.219613+0000 mon.c (mon.2) 256 : audit [INF] from='client.? v1:172.21.15.163:0/1203751048' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:51.221602+0000 mon.a (mon.0) 1631 : audit [INF] from='client.49274 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:51.325216+0000 mon.c (mon.2) 257 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:51.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:50.182572+0000 mgr.y (mgr.14550) 252 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:06:51.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:50.464380+0000 osd.5 (osd.5) 108 : cluster [DBG] 16.f scrub starts 2024-02-19T21:06:51.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:50.467407+0000 osd.5 (osd.5) 109 : cluster [DBG] 16.f scrub ok 2024-02-19T21:06:51.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:50.690773+0000 osd.7 (osd.7) 115 : cluster [DBG] 2.b scrub starts 2024-02-19T21:06:51.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:50.694242+0000 osd.7 (osd.7) 116 : cluster [DBG] 2.b scrub ok 2024-02-19T21:06:51.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:51.201348+0000 mon.a (mon.0) 1628 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:06:51.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:51.201442+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-02-19T21:06:51.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:51.214376+0000 mon.b (mon.1) 364 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23"}]: dispatch 2024-02-19T21:06:51.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:51.218584+0000 mon.a (mon.0) 1630 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23"}]: dispatch 2024-02-19T21:06:51.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:51.219613+0000 mon.c (mon.2) 256 : audit [INF] from='client.? v1:172.21.15.163:0/1203751048' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:51.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:51.221602+0000 mon.a (mon.0) 1631 : audit [INF] from='client.49274 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:51.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:51.325216+0000 mon.c (mon.2) 257 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:52.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:51.150874+0000 mgr.y (mgr.14550) 253 : cluster [DBG] pgmap v317: 357 pgs: 32 creating+peering, 2 peering, 3 active+clean+snaptrim, 8 unknown, 4 active+clean+snaptrim_wait, 308 active+clean; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-02-19T21:06:52.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:51.420717+0000 osd.5 (osd.5) 110 : cluster [DBG] 16.1b scrub starts 2024-02-19T21:06:52.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:51.423765+0000 osd.5 (osd.5) 111 : cluster [DBG] 16.1b scrub ok 2024-02-19T21:06:52.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:51.663598+0000 osd.7 (osd.7) 117 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:06:52.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:51.666704+0000 osd.7 (osd.7) 118 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:06:52.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:51.876648+0000 osd.4 (osd.4) 121 : cluster [DBG] 16.19 scrub starts 2024-02-19T21:06:52.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:51.878055+0000 osd.4 (osd.4) 122 : cluster [DBG] 16.19 scrub ok 2024-02-19T21:06:52.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:52.197268+0000 mon.a (mon.0) 1632 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:52.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:52.201648+0000 mon.a (mon.0) 1633 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23"}]': finished 2024-02-19T21:06:52.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:52.201749+0000 mon.a (mon.0) 1634 : audit [INF] from='client.49274 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:52.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:52.201810+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-02-19T21:06:52.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:52.203735+0000 mon.c (mon.2) 258 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:52.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:52.206960+0000 mon.a (mon.0) 1636 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:52.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:52.326046+0000 mon.c (mon.2) 259 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:52.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:51.150874+0000 mgr.y (mgr.14550) 253 : cluster [DBG] pgmap v317: 357 pgs: 32 creating+peering, 2 peering, 3 active+clean+snaptrim, 8 unknown, 4 active+clean+snaptrim_wait, 308 active+clean; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-02-19T21:06:52.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:51.420717+0000 osd.5 (osd.5) 110 : cluster [DBG] 16.1b scrub starts 2024-02-19T21:06:52.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:51.423765+0000 osd.5 (osd.5) 111 : cluster [DBG] 16.1b scrub ok 2024-02-19T21:06:52.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:51.663598+0000 osd.7 (osd.7) 117 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:06:52.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:51.666704+0000 osd.7 (osd.7) 118 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:06:52.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:51.876648+0000 osd.4 (osd.4) 121 : cluster [DBG] 16.19 scrub starts 2024-02-19T21:06:52.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:51.878055+0000 osd.4 (osd.4) 122 : cluster [DBG] 16.19 scrub ok 2024-02-19T21:06:52.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:52.197268+0000 mon.a (mon.0) 1632 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:52.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:52.201648+0000 mon.a (mon.0) 1633 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23"}]': finished 2024-02-19T21:06:52.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:52.201749+0000 mon.a (mon.0) 1634 : audit [INF] from='client.49274 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:52.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:52.201810+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-02-19T21:06:52.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:52.203735+0000 mon.c (mon.2) 258 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:52.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:52.206960+0000 mon.a (mon.0) 1636 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:52.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:52.326046+0000 mon.c (mon.2) 259 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:52.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:51.150874+0000 mgr.y (mgr.14550) 253 : cluster [DBG] pgmap v317: 357 pgs: 32 creating+peering, 2 peering, 3 active+clean+snaptrim, 8 unknown, 4 active+clean+snaptrim_wait, 308 active+clean; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-02-19T21:06:52.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:51.420717+0000 osd.5 (osd.5) 110 : cluster [DBG] 16.1b scrub starts 2024-02-19T21:06:52.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:51.423765+0000 osd.5 (osd.5) 111 : cluster [DBG] 16.1b scrub ok 2024-02-19T21:06:52.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:51.663598+0000 osd.7 (osd.7) 117 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:06:52.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:51.666704+0000 osd.7 (osd.7) 118 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:06:52.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:51.876648+0000 osd.4 (osd.4) 121 : cluster [DBG] 16.19 scrub starts 2024-02-19T21:06:52.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:51.878055+0000 osd.4 (osd.4) 122 : cluster [DBG] 16.19 scrub ok 2024-02-19T21:06:52.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:52.197268+0000 mon.a (mon.0) 1632 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:06:52.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:52.201648+0000 mon.a (mon.0) 1633 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-23"}]': finished 2024-02-19T21:06:52.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:52.201749+0000 mon.a (mon.0) 1634 : audit [INF] from='client.49274 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:52.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:52.201810+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-02-19T21:06:52.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:52.203735+0000 mon.c (mon.2) 258 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:52.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:52.206960+0000 mon.a (mon.0) 1636 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:52.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:52.326046+0000 mon.c (mon.2) 259 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:53.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.303+0000 7f5c51e1f700 1 -- 172.21.15.163:0/503179161 >> v1:172.21.15.163:6790/0 conn(0x7f5c4c100780 legacy=0x7f5c4c102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.303+0000 7f5c51e1f700 1 -- 172.21.15.163:0/503179161 shutdown_connections 2024-02-19T21:06:53.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.303+0000 7f5c51e1f700 1 -- 172.21.15.163:0/503179161 >> 172.21.15.163:0/503179161 conn(0x7f5c4c0f35a0 msgr2=0x7f5c4c0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:53.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.303+0000 7f5c51e1f700 1 -- 172.21.15.163:0/503179161 shutdown_connections 2024-02-19T21:06:53.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.303+0000 7f5c51e1f700 1 -- 172.21.15.163:0/503179161 wait complete. 2024-02-19T21:06:53.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.304+0000 7f5c51e1f700 1 librados: starting msgr at 2024-02-19T21:06:53.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.304+0000 7f5c51e1f700 1 librados: starting objecter 2024-02-19T21:06:53.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.304+0000 7f5c51e1f700 1 Processor -- start 2024-02-19T21:06:53.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.304+0000 7f5c51e1f700 1 -- start start 2024-02-19T21:06:53.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.304+0000 7f5c51e1f700 1 librados: setting wanted keys 2024-02-19T21:06:53.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.304+0000 7f5c51e1f700 1 librados: calling monclient init 2024-02-19T21:06:53.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.304+0000 7f5c51e1f700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c4c10e840 con 0x7f5c4c100780 2024-02-19T21:06:53.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.304+0000 7f5c51e1f700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c4c118910 con 0x7f5c4c1042c0 2024-02-19T21:06:53.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.304+0000 7f5c51e1f700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c4c119af0 con 0x7f5c4c0fd980 2024-02-19T21:06:53.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.304+0000 7f5c4b7fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f5c4c0fd980 0x7f5c4c10dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:48026/0 (socket says 172.21.15.163:48026) 2024-02-19T21:06:53.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.304+0000 7f5c4b7fe700 1 -- 172.21.15.163:0/1102041958 learned_addr learned my addr 172.21.15.163:0/1102041958 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:53.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 10 client.?.objecter ms_handle_connect 0x7f5c4c0fd980 2024-02-19T21:06:53.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:53.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 10 client.?.objecter ms_handle_connect 0x7f5c4c1042c0 2024-02-19T21:06:53.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:53.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198627729 0 0) 0x7f5c4c119af0 con 0x7f5c4c0fd980 2024-02-19T21:06:53.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c34002220 con 0x7f5c4c0fd980 2024-02-19T21:06:53.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957211769 0 0) 0x7f5c4c118910 con 0x7f5c4c1042c0 2024-02-19T21:06:53.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c4c119af0 con 0x7f5c4c1042c0 2024-02-19T21:06:53.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 10 client.?.objecter ms_handle_connect 0x7f5c4c100780 2024-02-19T21:06:53.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:53.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1714782004 0 0) 0x7f5c4c119af0 con 0x7f5c4c1042c0 2024-02-19T21:06:53.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c4c118910 con 0x7f5c4c1042c0 2024-02-19T21:06:53.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2840922784 0 0) 0x7f5c34002220 con 0x7f5c4c0fd980 2024-02-19T21:06:53.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c4c119af0 con 0x7f5c4c0fd980 2024-02-19T21:06:53.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5c44001b80 con 0x7f5c4c1042c0 2024-02-19T21:06:53.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5c3c0018c0 con 0x7f5c4c0fd980 2024-02-19T21:06:53.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1479498132 0 0) 0x7f5c4c118910 con 0x7f5c4c1042c0 2024-02-19T21:06:53.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 >> v1:172.21.15.163:6790/0 conn(0x7f5c4c0fd980 legacy=0x7f5c4c10dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 >> v1:172.21.15.163:6789/0 conn(0x7f5c4c100780 legacy=0x7f5c4c112ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4c11acf0 con 0x7f5c4c1042c0 2024-02-19T21:06:53.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c51e1f700 1 -- 172.21.15.163:0/1102041958 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c4c118b00 con 0x7f5c4c1042c0 2024-02-19T21:06:53.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c51e1f700 10 client.48112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:53.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c51e1f700 1 -- 172.21.15.163:0/1102041958 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4c119030 con 0x7f5c4c1042c0 2024-02-19T21:06:53.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.305+0000 7f5c51e1f700 1 librados: init done 2024-02-19T21:06:53.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.306+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f5c44002240 con 0x7f5c4c1042c0 2024-02-19T21:06:53.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.306+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5c440035b0 con 0x7f5c4c1042c0 2024-02-19T21:06:53.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.306+0000 7f5c51e1f700 1 -- 172.21.15.163:0/1102041958 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c38004f20 con 0x7f5c4c1042c0 2024-02-19T21:06:53.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.307+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f5c44001df0 con 0x7f5c4c1042c0 2024-02-19T21:06:53.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.308+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(233..233 src has 43..233) v4 ==== 10020+0+0 (unknown 1241887173 0 0) 0x7f5c44077020 con 0x7f5c4c1042c0 2024-02-19T21:06:53.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.308+0000 7f5c48ff9700 10 client.48112.objecter ms_dispatch 0x7f5c4c001550 osd_map(233..233 src has 43..233) v4 2024-02-19T21:06:53.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.308+0000 7f5c48ff9700 3 client.48112.objecter handle_osd_map got epochs [233,233] > 0 2024-02-19T21:06:53.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.308+0000 7f5c48ff9700 3 client.48112.objecter handle_osd_map decoding full epoch 233 2024-02-19T21:06:53.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.308+0000 7f5c48ff9700 20 client.48112.objecter dump_active .. 0 homeless 2024-02-19T21:06:53.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.308+0000 7f5c48ff9700 10 client.48112.objecter ms_handle_connect 0x7f5c34063b80 2024-02-19T21:06:53.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.311+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5c44049c90 con 0x7f5c4c1042c0 2024-02-19T21:06:53.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.385+0000 7f26b2096700 1 -- 172.21.15.163:0/4055859464 >> v1:172.21.15.163:6790/0 conn(0x7f26ac0fc890 legacy=0x7f26ac0fcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.385+0000 7f26b2096700 1 -- 172.21.15.163:0/4055859464 shutdown_connections 2024-02-19T21:06:53.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.385+0000 7f26b2096700 1 -- 172.21.15.163:0/4055859464 >> 172.21.15.163:0/4055859464 conn(0x7f26ac0f2470 msgr2=0x7f26ac0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:53.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.385+0000 7f26b2096700 1 -- 172.21.15.163:0/4055859464 shutdown_connections 2024-02-19T21:06:53.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.385+0000 7f26b2096700 1 -- 172.21.15.163:0/4055859464 wait complete. 2024-02-19T21:06:53.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.386+0000 7f26b2096700 1 librados: starting msgr at 2024-02-19T21:06:53.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.386+0000 7f26b2096700 1 librados: starting objecter 2024-02-19T21:06:53.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.386+0000 7f26b2096700 1 Processor -- start 2024-02-19T21:06:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.386+0000 7f26b2096700 1 -- start start 2024-02-19T21:06:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.386+0000 7f26b2096700 1 librados: setting wanted keys 2024-02-19T21:06:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.386+0000 7f26b2096700 1 librados: calling monclient init 2024-02-19T21:06:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.386+0000 7f26b2096700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f26ac1194a0 con 0x7f26ac0ff690 2024-02-19T21:06:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.386+0000 7f26b2096700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f26ac11a640 con 0x7f26ac1031d0 2024-02-19T21:06:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.386+0000 7f26b2096700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f26ac11b840 con 0x7f26ac0fc890 2024-02-19T21:06:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.386+0000 7f26ab7fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f26ac0fc890 0x7f26ac060b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:48054/0 (socket says 172.21.15.163:48054) 2024-02-19T21:06:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.386+0000 7f26ab7fe700 1 -- 172.21.15.163:0/13380774 learned_addr learned my addr 172.21.15.163:0/13380774 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 10 client.?.objecter ms_handle_connect 0x7f26ac0fc890 2024-02-19T21:06:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369340791 0 0) 0x7f26ac11b840 con 0x7f26ac0fc890 2024-02-19T21:06:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2694003630 con 0x7f26ac0fc890 2024-02-19T21:06:53.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 10 client.?.objecter ms_handle_connect 0x7f26ac1031d0 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848122636 0 0) 0x7f26ac11a640 con 0x7f26ac1031d0 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26ac11b840 con 0x7f26ac1031d0 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2835143123 0 0) 0x7f2694003630 con 0x7f26ac0fc890 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26ac11a640 con 0x7f26ac0fc890 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f269c0010c0 con 0x7f26ac0fc890 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 10 client.?.objecter ms_handle_connect 0x7f26ac0ff690 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1092202398 0 0) 0x7f26ac11b840 con 0x7f26ac1031d0 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2694003630 con 0x7f26ac1031d0 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f26a4001af0 con 0x7f26ac1031d0 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2731467644 0 0) 0x7f26ac11a640 con 0x7f26ac0fc890 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 >> v1:172.21.15.193:6789/0 conn(0x7f26ac1031d0 legacy=0x7f26ac117c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 >> v1:172.21.15.163:6789/0 conn(0x7f26ac0ff690 legacy=0x7f26ac061230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ac11ca40 con 0x7f26ac0fc890 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ac11a830 con 0x7f26ac0fc890 2024-02-19T21:06:53.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26b2096700 10 client.49292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:53.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ac11ad60 con 0x7f26ac0fc890 2024-02-19T21:06:53.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.387+0000 7f26b2096700 1 librados: init done 2024-02-19T21:06:53.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.388+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f269c002390 con 0x7f26ac0fc890 2024-02-19T21:06:53.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.388+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f26ac05f520 con 0x7f26ac0fc890 2024-02-19T21:06:53.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.388+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f269c003800 con 0x7f26ac0fc890 2024-02-19T21:06:53.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.389+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f269c003980 con 0x7f26ac0fc890 2024-02-19T21:06:53.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(233..233 src has 43..233) v4 ==== 10020+0+0 (unknown 1241887173 0 0) 0x7f269c079710 con 0x7f26ac0fc890 2024-02-19T21:06:53.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 osd_map(233..233 src has 43..233) v4 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26a8ff9700 3 client.49292.objecter handle_osd_map got epochs [233,233] > 0 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26a8ff9700 3 client.49292.objecter handle_osd_map decoding full epoch 233 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26a8ff9700 20 client.49292.objecter dump_active .. 0 homeless 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26a8ff9700 10 client.49292.objecter ms_handle_connect 0x7f2694063b60 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v233) v1 ==== 54+0+16 (unknown 2819901177 0 2420000192) 0x7f269c079ad0 con 0x7f26ac0fc890 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26b2096700 10 client.49292.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 1 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac047540 osd=0 3 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac047540 osd=0 3 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac047540 osd=0 3 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 1 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac047540 osd=0 3 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 0 1 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac047540 osd=0 4 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26b2096700 10 client.49292.objecter _send_command 1 2024-02-19T21:06:53.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26ac0f80c0 con 0x7f26ac119630 2024-02-19T21:06:53.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.390+0000 7f26a8ff9700 10 client.49292.objecter ms_handle_connect 0x7f26ac119630 2024-02-19T21:06:53.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.391+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f26ac0f80c0 con 0x7f26ac119630 2024-02-19T21:06:53.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.391+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:06:53.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.391+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 1 = system:0 2024-02-19T21:06:53.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.391+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac047540 osd=0 4 2024-02-19T21:06:53.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.391+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 0 1 2024-02-19T21:06:53.488 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:06:53.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.406+0000 7f26b2096700 10 client.49292.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.406+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 2 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.406+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac047540 osd=0 3 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.406+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac047540 osd=0 3 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.406+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac047540 osd=0 3 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.406+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 2 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.406+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac047540 osd=0 3 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.406+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 0 2 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.406+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac047540 osd=0 4 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.406+0000 7f26b2096700 10 client.49292.objecter _send_command 2 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.406+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f26ac108e80 con 0x7f26ac119630 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.408+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f26ac108e80 con 0x7f26ac119630 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.408+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.408+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.408+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac047540 osd=0 4 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.408+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 0 2 2024-02-19T21:06:53.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.409+0000 7f26b2096700 10 client.49292.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:53.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.409+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 3 2024-02-19T21:06:53.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.409+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac0f2850 osd=1 3 2024-02-19T21:06:53.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.409+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac0f2850 osd=1 3 2024-02-19T21:06:53.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.409+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac0f2850 osd=1 3 2024-02-19T21:06:53.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.409+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 3 2024-02-19T21:06:53.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.409+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac0f2850 osd=1 3 2024-02-19T21:06:53.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.409+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 1 3 2024-02-19T21:06:53.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.409+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac0f2850 osd=1 4 2024-02-19T21:06:53.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.409+0000 7f26b2096700 10 client.49292.objecter _send_command 3 2024-02-19T21:06:53.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.409+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26ac0f32d0 con 0x7f26ac0f2ed0 2024-02-19T21:06:53.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.409+0000 7f26a8ff9700 10 client.49292.objecter ms_handle_connect 0x7f26ac0f2ed0 2024-02-19T21:06:53.491 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:06:53.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.410+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f26ac0f32d0 con 0x7f26ac0f2ed0 2024-02-19T21:06:53.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.410+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:06:53.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.410+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 3 = system:0 2024-02-19T21:06:53.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.410+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac0f2850 osd=1 4 2024-02-19T21:06:53.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.410+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 1 3 2024-02-19T21:06:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.424+0000 7f26b2096700 10 client.49292.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.424+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 4 2024-02-19T21:06:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.424+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac0f2850 osd=1 3 2024-02-19T21:06:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.424+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac0f2850 osd=1 3 2024-02-19T21:06:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.424+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac0f2850 osd=1 3 2024-02-19T21:06:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.424+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 4 2024-02-19T21:06:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.424+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac0f2850 osd=1 3 2024-02-19T21:06:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.424+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 1 4 2024-02-19T21:06:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.424+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac0f2850 osd=1 4 2024-02-19T21:06:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.424+0000 7f26b2096700 10 client.49292.objecter _send_command 4 2024-02-19T21:06:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.424+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f26ac0f36b0 con 0x7f26ac0f2ed0 2024-02-19T21:06:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.426+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f26ac0f36b0 con 0x7f26ac0f2ed0 2024-02-19T21:06:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:53.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac0f2850 osd=1 4 2024-02-19T21:06:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 1 4 2024-02-19T21:06:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26b2096700 10 client.49292.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 5 2024-02-19T21:06:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac0fe8b0 osd=2 3 2024-02-19T21:06:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac0fe8b0 osd=2 3 2024-02-19T21:06:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac0fe8b0 osd=2 3 2024-02-19T21:06:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 5 2024-02-19T21:06:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac0fe8b0 osd=2 3 2024-02-19T21:06:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 2 5 2024-02-19T21:06:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac0fe8b0 osd=2 4 2024-02-19T21:06:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26b2096700 10 client.49292.objecter _send_command 5 2024-02-19T21:06:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26ac0287b0 con 0x7f26ac0fea20 2024-02-19T21:06:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.427+0000 7f26a8ff9700 10 client.49292.objecter ms_handle_connect 0x7f26ac0fea20 2024-02-19T21:06:53.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.428+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f26ac0287b0 con 0x7f26ac0fea20 2024-02-19T21:06:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.428+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:06:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.428+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 5 = system:0 2024-02-19T21:06:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.428+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac0fe8b0 osd=2 4 2024-02-19T21:06:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.428+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 2 5 2024-02-19T21:06:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.442+0000 7f26b2096700 10 client.49292.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.442+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 6 2024-02-19T21:06:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.442+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac0fe8b0 osd=2 3 2024-02-19T21:06:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.442+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac0fe8b0 osd=2 3 2024-02-19T21:06:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.442+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac0fe8b0 osd=2 3 2024-02-19T21:06:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.442+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 6 2024-02-19T21:06:53.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.442+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac0fe8b0 osd=2 3 2024-02-19T21:06:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.442+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 2 6 2024-02-19T21:06:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.442+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac0fe8b0 osd=2 4 2024-02-19T21:06:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.442+0000 7f26b2096700 10 client.49292.objecter _send_command 6 2024-02-19T21:06:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.442+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f26ac027d70 con 0x7f26ac0fea20 2024-02-19T21:06:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f26ac027d70 con 0x7f26ac0fea20 2024-02-19T21:06:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac0fe8b0 osd=2 4 2024-02-19T21:06:53.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 2 6 2024-02-19T21:06:53.495 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:06:53.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26b2096700 10 client.49292.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:53.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 7 2024-02-19T21:06:53.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac0598f0 osd=3 3 2024-02-19T21:06:53.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac0598f0 osd=3 3 2024-02-19T21:06:53.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac0598f0 osd=3 3 2024-02-19T21:06:53.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 7 2024-02-19T21:06:53.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac0598f0 osd=3 3 2024-02-19T21:06:53.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 3 7 2024-02-19T21:06:53.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac0598f0 osd=3 4 2024-02-19T21:06:53.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26b2096700 10 client.49292.objecter _send_command 7 2024-02-19T21:06:53.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.444+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26ac05a040 con 0x7f26ac126570 2024-02-19T21:06:53.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.445+0000 7f26a8ff9700 10 client.49292.objecter ms_handle_connect 0x7f26ac126570 2024-02-19T21:06:53.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.445+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f26ac05a040 con 0x7f26ac126570 2024-02-19T21:06:53.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.445+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:06:53.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.445+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 7 = system:0 2024-02-19T21:06:53.497 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:06:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.445+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac0598f0 osd=3 4 2024-02-19T21:06:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.445+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 3 7 2024-02-19T21:06:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.459+0000 7f26b2096700 10 client.49292.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.459+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 8 2024-02-19T21:06:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.459+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac0598f0 osd=3 3 2024-02-19T21:06:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.459+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac0598f0 osd=3 3 2024-02-19T21:06:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.459+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac0598f0 osd=3 3 2024-02-19T21:06:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.459+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 8 2024-02-19T21:06:53.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.459+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac0598f0 osd=3 3 2024-02-19T21:06:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.459+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 3 8 2024-02-19T21:06:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.459+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac0598f0 osd=3 4 2024-02-19T21:06:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.459+0000 7f26b2096700 10 client.49292.objecter _send_command 8 2024-02-19T21:06:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.459+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f26ac05a040 con 0x7f26ac126570 2024-02-19T21:06:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f26ac05a040 con 0x7f26ac126570 2024-02-19T21:06:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac0598f0 osd=3 4 2024-02-19T21:06:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 3 8 2024-02-19T21:06:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26b2096700 10 client.49292.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 9 2024-02-19T21:06:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac05a040 osd=4 3 2024-02-19T21:06:53.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac05a040 osd=4 3 2024-02-19T21:06:53.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac05a040 osd=4 3 2024-02-19T21:06:53.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 9 2024-02-19T21:06:53.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac05a040 osd=4 3 2024-02-19T21:06:53.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 4 9 2024-02-19T21:06:53.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac05a040 osd=4 4 2024-02-19T21:06:53.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26b2096700 10 client.49292.objecter _send_command 9 2024-02-19T21:06:53.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.461+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26ac126140 con 0x7f26ac125bc0 2024-02-19T21:06:53.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.462+0000 7f26a8ff9700 10 client.49292.objecter ms_handle_connect 0x7f26ac125bc0 2024-02-19T21:06:53.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.462+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f26ac126140 con 0x7f26ac125bc0 2024-02-19T21:06:53.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.462+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:06:53.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.462+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 9 = system:0 2024-02-19T21:06:53.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.462+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac05a040 osd=4 4 2024-02-19T21:06:53.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.462+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 4 9 2024-02-19T21:06:53.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.477+0000 7f26b2096700 10 client.49292.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:53.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.477+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 10 2024-02-19T21:06:53.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.477+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac05a040 osd=4 3 2024-02-19T21:06:53.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.477+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac05a040 osd=4 3 2024-02-19T21:06:53.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.477+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac05a040 osd=4 3 2024-02-19T21:06:53.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.477+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 10 2024-02-19T21:06:53.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.477+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac05a040 osd=4 3 2024-02-19T21:06:53.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.477+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 4 10 2024-02-19T21:06:53.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.477+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac05a040 osd=4 4 2024-02-19T21:06:53.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.477+0000 7f26b2096700 10 client.49292.objecter _send_command 10 2024-02-19T21:06:53.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.477+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f26ac126140 con 0x7f26ac125bc0 2024-02-19T21:06:53.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.479+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f26ac126140 con 0x7f26ac125bc0 2024-02-19T21:06:53.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.479+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:53.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.479+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.479+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac05a040 osd=4 4 2024-02-19T21:06:53.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.479+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 4 10 2024-02-19T21:06:53.502 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:06:53.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.479+0000 7f26b2096700 10 client.49292.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:53.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.479+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 11 2024-02-19T21:06:53.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.480+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac126140 osd=5 3 2024-02-19T21:06:53.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.480+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac126140 osd=5 3 2024-02-19T21:06:53.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.480+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac126140 osd=5 3 2024-02-19T21:06:53.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.480+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 11 2024-02-19T21:06:53.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.480+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac126140 osd=5 3 2024-02-19T21:06:53.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.480+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 5 11 2024-02-19T21:06:53.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.480+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac126140 osd=5 4 2024-02-19T21:06:53.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.480+0000 7f26b2096700 10 client.49292.objecter _send_command 11 2024-02-19T21:06:53.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.480+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26ac132950 con 0x7f26ac12ee40 2024-02-19T21:06:53.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.480+0000 7f26a8ff9700 10 client.49292.objecter ms_handle_connect 0x7f26ac12ee40 2024-02-19T21:06:53.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.481+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f26ac132950 con 0x7f26ac12ee40 2024-02-19T21:06:53.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.481+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:06:53.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.481+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 11 = system:0 2024-02-19T21:06:53.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.481+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac126140 osd=5 4 2024-02-19T21:06:53.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.481+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 5 11 2024-02-19T21:06:53.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.496+0000 7f26b2096700 10 client.49292.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:53.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.496+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 12 2024-02-19T21:06:53.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.496+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac126140 osd=5 3 2024-02-19T21:06:53.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.496+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac126140 osd=5 3 2024-02-19T21:06:53.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.496+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac126140 osd=5 3 2024-02-19T21:06:53.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.496+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 12 2024-02-19T21:06:53.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.496+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac126140 osd=5 3 2024-02-19T21:06:53.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.496+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 5 12 2024-02-19T21:06:53.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.496+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac126140 osd=5 4 2024-02-19T21:06:53.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.496+0000 7f26b2096700 10 client.49292.objecter _send_command 12 2024-02-19T21:06:53.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.496+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f26ac1340d0 con 0x7f26ac12ee40 2024-02-19T21:06:53.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.497+0000 7f5c51e1f700 1 -- 172.21.15.163:0/1102041958 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f5c380051d0 con 0x7f5c4c1042c0 2024-02-19T21:06:53.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f26ac1340d0 con 0x7f26ac12ee40 2024-02-19T21:06:53.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:53.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac126140 osd=5 4 2024-02-19T21:06:53.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 5 12 2024-02-19T21:06:53.505 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:06:53.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26b2096700 10 client.49292.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:53.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 13 2024-02-19T21:06:53.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac1340d0 osd=6 3 2024-02-19T21:06:53.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac1340d0 osd=6 3 2024-02-19T21:06:53.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac1340d0 osd=6 3 2024-02-19T21:06:53.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 13 2024-02-19T21:06:53.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac1340d0 osd=6 3 2024-02-19T21:06:53.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 6 13 2024-02-19T21:06:53.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac1340d0 osd=6 4 2024-02-19T21:06:53.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26b2096700 10 client.49292.objecter _send_command 13 2024-02-19T21:06:53.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.498+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26ac134cf0 con 0x7f26ac134780 2024-02-19T21:06:53.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.499+0000 7f26a8ff9700 10 client.49292.objecter ms_handle_connect 0x7f26ac134780 2024-02-19T21:06:53.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.499+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f26ac134cf0 con 0x7f26ac134780 2024-02-19T21:06:53.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.499+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:06:53.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.499+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 13 = system:0 2024-02-19T21:06:53.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.499+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac1340d0 osd=6 4 2024-02-19T21:06:53.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.499+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 6 13 2024-02-19T21:06:53.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.514+0000 7f26b2096700 10 client.49292.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:53.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.514+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 14 2024-02-19T21:06:53.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.514+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac1340d0 osd=6 3 2024-02-19T21:06:53.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.514+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac1340d0 osd=6 3 2024-02-19T21:06:53.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.514+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac1340d0 osd=6 3 2024-02-19T21:06:53.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.514+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 14 2024-02-19T21:06:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.514+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac1340d0 osd=6 3 2024-02-19T21:06:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.514+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 6 14 2024-02-19T21:06:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.514+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac1340d0 osd=6 4 2024-02-19T21:06:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.514+0000 7f26b2096700 10 client.49292.objecter _send_command 14 2024-02-19T21:06:53.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.514+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f26ac134cf0 con 0x7f26ac134780 2024-02-19T21:06:53.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f26ac134cf0 con 0x7f26ac134780 2024-02-19T21:06:53.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:53.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac1340d0 osd=6 4 2024-02-19T21:06:53.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 6 14 2024-02-19T21:06:53.512 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:06:53.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26b2096700 10 client.49292.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:06:53.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 15 2024-02-19T21:06:53.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac134cf0 osd=7 3 2024-02-19T21:06:53.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac134cf0 osd=7 3 2024-02-19T21:06:53.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac134cf0 osd=7 3 2024-02-19T21:06:53.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 15 2024-02-19T21:06:53.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac134cf0 osd=7 3 2024-02-19T21:06:53.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 7 15 2024-02-19T21:06:53.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac134cf0 osd=7 4 2024-02-19T21:06:53.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26b2096700 10 client.49292.objecter _send_command 15 2024-02-19T21:06:53.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26ac13b2f0 con 0x7f26ac1377e0 2024-02-19T21:06:53.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.516+0000 7f26a8ff9700 10 client.49292.objecter ms_handle_connect 0x7f26ac1377e0 2024-02-19T21:06:53.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.517+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f26ac13b2f0 con 0x7f26ac1377e0 2024-02-19T21:06:53.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.517+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:06:53.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.517+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 15 = system:0 2024-02-19T21:06:53.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.517+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac134cf0 osd=7 4 2024-02-19T21:06:53.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.517+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 7 15 2024-02-19T21:06:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.532+0000 7f26b2096700 10 client.49292.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:06:53.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.532+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign -1 16 2024-02-19T21:06:53.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.532+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac134cf0 osd=7 3 2024-02-19T21:06:53.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.532+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac134cf0 osd=7 3 2024-02-19T21:06:53.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.532+0000 7f26b2096700 20 client.49292.objecter _get_session s=0x7f26ac134cf0 osd=7 3 2024-02-19T21:06:53.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.532+0000 7f26b2096700 15 client.49292.objecter _session_command_op_remove -1 16 2024-02-19T21:06:53.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.532+0000 7f26b2096700 20 client.49292.objecter get_session s=0x7f26ac134cf0 osd=7 3 2024-02-19T21:06:53.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.532+0000 7f26b2096700 15 client.49292.objecter _session_command_op_assign 7 16 2024-02-19T21:06:53.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.532+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac134cf0 osd=7 4 2024-02-19T21:06:53.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.532+0000 7f26b2096700 10 client.49292.objecter _send_command 16 2024-02-19T21:06:53.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.532+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f26ac13b2f0 con 0x7f26ac1377e0 2024-02-19T21:06:53.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26a8ff9700 1 -- 172.21.15.163:0/13380774 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f26ac13b2f0 con 0x7f26ac1377e0 2024-02-19T21:06:53.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26a8ff9700 10 client.49292.objecter ms_dispatch 0x7f26ac001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:06:53.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26a8ff9700 10 client.49292.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26a8ff9700 20 client.49292.objecter put_session s=0x7f26ac134cf0 osd=7 4 2024-02-19T21:06:53.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26a8ff9700 15 client.49292.objecter _session_command_op_remove 7 16 2024-02-19T21:06:53.530 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:06:53.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 10 librados: watch_flush enter 2024-02-19T21:06:53.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 10 librados: watch_flush exit 2024-02-19T21:06:53.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 10 client.49292.objecter close_session for osd.0 2024-02-19T21:06:53.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 >> v1:172.21.15.163:6801/1559418115 conn(0x7f26ac119630 legacy=0x7f26ac119a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac047540 osd=0 1 2024-02-19T21:06:53.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 10 client.49292.objecter close_session for osd.1 2024-02-19T21:06:53.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 >> v1:172.21.15.163:6805/3125047432 conn(0x7f26ac0f2ed0 legacy=0x7f26ac0273c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac0f2850 osd=1 1 2024-02-19T21:06:53.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 10 client.49292.objecter close_session for osd.2 2024-02-19T21:06:53.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 >> v1:172.21.15.163:6809/1563158487 conn(0x7f26ac0fea20 legacy=0x7f26ac123cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac0fe8b0 osd=2 1 2024-02-19T21:06:53.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 10 client.49292.objecter close_session for osd.3 2024-02-19T21:06:53.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 >> v1:172.21.15.163:6813/2112279769 conn(0x7f26ac126570 legacy=0x7f26ac128960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac0598f0 osd=3 1 2024-02-19T21:06:53.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 10 client.49292.objecter close_session for osd.4 2024-02-19T21:06:53.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 >> v1:172.21.15.193:6800/3129716965 conn(0x7f26ac125bc0 legacy=0x7f26ac12d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26abfff700 1 -- 172.21.15.163:0/13380774 reap_dead start 2024-02-19T21:06:53.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac05a040 osd=4 1 2024-02-19T21:06:53.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 10 client.49292.objecter close_session for osd.5 2024-02-19T21:06:53.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 >> v1:172.21.15.193:6804/805161991 conn(0x7f26ac12ee40 legacy=0x7f26ac131230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26abfff700 1 -- 172.21.15.163:0/13380774 reap_dead start 2024-02-19T21:06:53.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac126140 osd=5 1 2024-02-19T21:06:53.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 10 client.49292.objecter close_session for osd.6 2024-02-19T21:06:53.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 >> v1:172.21.15.193:6808/787604615 conn(0x7f26ac134780 legacy=0x7f26ac132950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac1340d0 osd=6 1 2024-02-19T21:06:53.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 10 client.49292.objecter close_session for osd.7 2024-02-19T21:06:53.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 >> v1:172.21.15.193:6812/157927432 conn(0x7f26ac1377e0 legacy=0x7f26ac139bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 20 client.49292.objecter put_session s=0x7f26ac134cf0 osd=7 1 2024-02-19T21:06:53.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 20 client.49292.objecter shutdown clearing up homeless session... 2024-02-19T21:06:53.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 10 client.49292.objecter successfully canceled tick 2024-02-19T21:06:53.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26abfff700 1 -- 172.21.15.163:0/13380774 reap_dead start 2024-02-19T21:06:53.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 >> v1:172.21.15.163:6800/1552514681 conn(0x7f2694063b60 legacy=0x7f2694066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 >> v1:172.21.15.163:6790/0 conn(0x7f26ac0fc890 legacy=0x7f26ac060b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:53.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 shutdown_connections 2024-02-19T21:06:53.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 >> 172.21.15.163:0/13380774 conn(0x7f26ac0f2470 msgr2=0x7f26ac106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:53.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 shutdown_connections 2024-02-19T21:06:53.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 1 -- 172.21.15.163:0/13380774 wait complete. 2024-02-19T21:06:53.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:53.534+0000 7f26b2096700 1 librados: shutdown 2024-02-19T21:06:53.533 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.534 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.534 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.534 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.534 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.534 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.534 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.534 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:06:53.743 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:06:53] "GET /metrics HTTP/1.1" 200 194014 "" "Prometheus/2.33.4" 2024-02-19T21:06:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:51.626641+0000 osd.1 (osd.1) 127 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:06:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:51.627700+0000 osd.1 (osd.1) 128 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:06:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:52.400042+0000 osd.5 (osd.5) 112 : cluster [DBG] 16.11 scrub starts 2024-02-19T21:06:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:52.402936+0000 osd.5 (osd.5) 113 : cluster [DBG] 16.11 scrub ok 2024-02-19T21:06:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:52.654624+0000 osd.7 (osd.7) 119 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:06:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:52.658098+0000 osd.7 (osd.7) 120 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:06:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:53.152007+0000 mon.b (mon.1) 365 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-19T21:06:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:53.153164+0000 mon.a (mon.0) 1637 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-19T21:06:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:53.203644+0000 mon.a (mon.0) 1638 : audit [INF] from='client.49262 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi163-83845-33"}]': finished 2024-02-19T21:06:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:53.203771+0000 mon.a (mon.0) 1639 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-02-19T21:06:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:53.203824+0000 mon.a (mon.0) 1640 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-02-19T21:06:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:53.211172+0000 mon.c (mon.2) 260 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:53.211748+0000 mon.a (mon.0) 1641 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:53.225880+0000 mon.b (mon.1) 366 : audit [INF] from='client.? v1:172.21.15.163:0/381265131' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:53.326913+0000 mon.c (mon.2) 261 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:53.390209+0000 mon.c (mon.2) 262 : audit [DBG] from='client.? v1:172.21.15.163:0/13380774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:53.481234+0000 mon.a (mon.0) 1642 : audit [INF] from='client.48106 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:53.499113+0000 mon.b (mon.1) 367 : audit [INF] from='client.? v1:172.21.15.163:0/1102041958' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:06:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:53.499487+0000 mon.a (mon.0) 1643 : audit [INF] from='client.48112 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:06:53.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:51.626641+0000 osd.1 (osd.1) 127 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:06:53.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:51.627700+0000 osd.1 (osd.1) 128 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:06:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:52.400042+0000 osd.5 (osd.5) 112 : cluster [DBG] 16.11 scrub starts 2024-02-19T21:06:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:52.402936+0000 osd.5 (osd.5) 113 : cluster [DBG] 16.11 scrub ok 2024-02-19T21:06:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:52.654624+0000 osd.7 (osd.7) 119 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:06:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:52.658098+0000 osd.7 (osd.7) 120 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:06:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:53.152007+0000 mon.b (mon.1) 365 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-19T21:06:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:53.153164+0000 mon.a (mon.0) 1637 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-19T21:06:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:53.203644+0000 mon.a (mon.0) 1638 : audit [INF] from='client.49262 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi163-83845-33"}]': finished 2024-02-19T21:06:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:53.203771+0000 mon.a (mon.0) 1639 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-02-19T21:06:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:53.203824+0000 mon.a (mon.0) 1640 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-02-19T21:06:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:53.211172+0000 mon.c (mon.2) 260 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:53.211748+0000 mon.a (mon.0) 1641 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:53.225880+0000 mon.b (mon.1) 366 : audit [INF] from='client.? v1:172.21.15.163:0/381265131' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:53.326913+0000 mon.c (mon.2) 261 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:53.390209+0000 mon.c (mon.2) 262 : audit [DBG] from='client.? v1:172.21.15.163:0/13380774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:53.481234+0000 mon.a (mon.0) 1642 : audit [INF] from='client.48106 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:53.499113+0000 mon.b (mon.1) 367 : audit [INF] from='client.? v1:172.21.15.163:0/1102041958' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:06:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:53.499487+0000 mon.a (mon.0) 1643 : audit [INF] from='client.48112 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:06:53.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:51.626641+0000 osd.1 (osd.1) 127 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:06:53.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:51.627700+0000 osd.1 (osd.1) 128 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:06:53.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:52.400042+0000 osd.5 (osd.5) 112 : cluster [DBG] 16.11 scrub starts 2024-02-19T21:06:53.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:52.402936+0000 osd.5 (osd.5) 113 : cluster [DBG] 16.11 scrub ok 2024-02-19T21:06:53.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:52.654624+0000 osd.7 (osd.7) 119 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:06:53.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:52.658098+0000 osd.7 (osd.7) 120 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:06:53.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:53.152007+0000 mon.b (mon.1) 365 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-19T21:06:53.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:53.153164+0000 mon.a (mon.0) 1637 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-02-19T21:06:53.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:53.203644+0000 mon.a (mon.0) 1638 : audit [INF] from='client.49262 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi163-83845-33"}]': finished 2024-02-19T21:06:53.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:53.203771+0000 mon.a (mon.0) 1639 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-02-19T21:06:53.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:53.203824+0000 mon.a (mon.0) 1640 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-02-19T21:06:53.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:53.211172+0000 mon.c (mon.2) 260 : audit [INF] from='client.? v1:172.21.15.163:0/1168274277' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:53.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:53.211748+0000 mon.a (mon.0) 1641 : audit [INF] from='client.49262 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi163-83845-33"}]: dispatch 2024-02-19T21:06:53.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:53.225880+0000 mon.b (mon.1) 366 : audit [INF] from='client.? v1:172.21.15.163:0/381265131' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:53.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:53.326913+0000 mon.c (mon.2) 261 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:53.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:53.390209+0000 mon.c (mon.2) 262 : audit [DBG] from='client.? v1:172.21.15.163:0/13380774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:06:53.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:53.481234+0000 mon.a (mon.0) 1642 : audit [INF] from='client.48106 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:53.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:53.499113+0000 mon.b (mon.1) 367 : audit [INF] from='client.? v1:172.21.15.163:0/1102041958' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:06:53.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:53.499487+0000 mon.a (mon.0) 1643 : audit [INF] from='client.48112 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:06:53.818 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:53.821 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:53.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:53.922 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:06:53.955 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:53.976 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:06:53.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:53.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:06:54.081 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:54.110 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:54.129 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:54.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:54.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:54.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:54.192+0000 7f5c48ff9700 1 -- 172.21.15.163:0/1102041958 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v234) v1 ==== 101+0+0 (unknown 247264343 0 0) 0x7f5c440480a0 con 0x7f5c4c1042c0 2024-02-19T21:06:54.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:54.196+0000 7f5c51e1f700 10 librados: watch_flush enter 2024-02-19T21:06:54.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:54.198+0000 7f5c51e1f700 10 librados: watch_flush exit 2024-02-19T21:06:54.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:54.198+0000 7f5c51e1f700 20 client.48112.objecter shutdown clearing up homeless session... 2024-02-19T21:06:54.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:54.198+0000 7f5c51e1f700 10 client.48112.objecter successfully canceled tick 2024-02-19T21:06:54.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:54.198+0000 7f5c51e1f700 1 -- 172.21.15.163:0/1102041958 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5c34063b80 legacy=0x7f5c34066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:54.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:54.198+0000 7f5c51e1f700 1 -- 172.21.15.163:0/1102041958 >> v1:172.21.15.193:6789/0 conn(0x7f5c4c1042c0 legacy=0x7f5c4c1161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:54.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:54.202+0000 7f5c51e1f700 1 -- 172.21.15.163:0/1102041958 shutdown_connections 2024-02-19T21:06:54.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:54.202+0000 7f5c51e1f700 1 -- 172.21.15.163:0/1102041958 >> 172.21.15.163:0/1102041958 conn(0x7f5c4c0f35a0 msgr2=0x7f5c4c107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:54.205+0000 7f5c51e1f700 1 -- 172.21.15.163:0/1102041958 shutdown_connections 2024-02-19T21:06:54.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:54.212+0000 7f5c51e1f700 1 -- 172.21.15.163:0/1102041958 wait complete. 2024-02-19T21:06:54.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:54.217+0000 7f5c51e1f700 1 librados: shutdown 2024-02-19T21:06:54.225 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:06:54.233 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:54.275 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:54.804 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:54.806 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:54.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:54.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:52.686778+0000 osd.6 (osd.6) 103 : cluster [DBG] 8.6 scrub starts 2024-02-19T21:06:54.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:52.688353+0000 osd.6 (osd.6) 104 : cluster [DBG] 8.6 scrub ok 2024-02-19T21:06:54.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:53.151392+0000 mgr.y (mgr.14550) 254 : cluster [DBG] pgmap v320: 380 pgs: 32 creating+peering, 1 peering, 3 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 308 active+clean; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-02-19T21:06:54.808 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:53.425178+0000 osd.5 (osd.5) 114 : cluster [DBG] 5.f scrub starts 2024-02-19T21:06:54.808 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:53.428074+0000 osd.5 (osd.5) 115 : cluster [DBG] 5.f scrub ok 2024-02-19T21:06:54.808 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:53.751574+0000 osd.3 (osd.3) 117 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:06:54.808 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:53.752535+0000 osd.3 (osd.3) 118 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:06:54.808 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:54.184419+0000 mon.a (mon.0) 1644 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:06:54.808 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:54.185655+0000 mon.a (mon.0) 1645 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:06:54.808 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:54.190994+0000 mon.a (mon.0) 1646 : audit [INF] from='client.49262 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi163-83845-33"}]': finished 2024-02-19T21:06:54.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:54.191183+0000 mon.a (mon.0) 1647 : audit [INF] from='client.48106 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:54.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:54.191259+0000 mon.a (mon.0) 1648 : audit [INF] from='client.48112 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:06:54.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:54.191317+0000 mon.a (mon.0) 1649 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-02-19T21:06:54.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:54.199203+0000 mon.b (mon.1) 368 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:54.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:54.201733+0000 mon.a (mon.0) 1650 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:54.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:54.234809+0000 mon.c (mon.2) 263 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:54.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:54.239360+0000 mon.a (mon.0) 1651 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:54.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:54.241819+0000 mon.c (mon.2) 264 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:54.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:54.242997+0000 mon.a (mon.0) 1652 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:54.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:54.245574+0000 mon.c (mon.2) 265 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi163-83845-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:54.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:54.245958+0000 mon.a (mon.0) 1653 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi163-83845-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:54.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:54.327979+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:54.810 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:52.686778+0000 osd.6 (osd.6) 103 : cluster [DBG] 8.6 scrub starts 2024-02-19T21:06:54.810 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:52.688353+0000 osd.6 (osd.6) 104 : cluster [DBG] 8.6 scrub ok 2024-02-19T21:06:54.810 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:53.151392+0000 mgr.y (mgr.14550) 254 : cluster [DBG] pgmap v320: 380 pgs: 32 creating+peering, 1 peering, 3 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 308 active+clean; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-02-19T21:06:54.810 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:53.425178+0000 osd.5 (osd.5) 114 : cluster [DBG] 5.f scrub starts 2024-02-19T21:06:54.810 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:53.428074+0000 osd.5 (osd.5) 115 : cluster [DBG] 5.f scrub ok 2024-02-19T21:06:54.810 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:53.751574+0000 osd.3 (osd.3) 117 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:06:54.810 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:53.752535+0000 osd.3 (osd.3) 118 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:06:54.810 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:54.184419+0000 mon.a (mon.0) 1644 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:06:54.810 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:54.185655+0000 mon.a (mon.0) 1645 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:06:54.811 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:54.190994+0000 mon.a (mon.0) 1646 : audit [INF] from='client.49262 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi163-83845-33"}]': finished 2024-02-19T21:06:54.811 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:54.191183+0000 mon.a (mon.0) 1647 : audit [INF] from='client.48106 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:54.811 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:54.191259+0000 mon.a (mon.0) 1648 : audit [INF] from='client.48112 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:06:54.811 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:54.191317+0000 mon.a (mon.0) 1649 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-02-19T21:06:54.811 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:54.199203+0000 mon.b (mon.1) 368 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:54.811 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:54.201733+0000 mon.a (mon.0) 1650 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:54.811 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:54.234809+0000 mon.c (mon.2) 263 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:54.811 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:54.239360+0000 mon.a (mon.0) 1651 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:54.811 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:54.241819+0000 mon.c (mon.2) 264 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:54.811 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:54.242997+0000 mon.a (mon.0) 1652 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:54.811 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:54.245574+0000 mon.c (mon.2) 265 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi163-83845-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:54.811 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:54.245958+0000 mon.a (mon.0) 1653 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi163-83845-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:54.811 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:54.327979+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:54.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:52.686778+0000 osd.6 (osd.6) 103 : cluster [DBG] 8.6 scrub starts 2024-02-19T21:06:54.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:52.688353+0000 osd.6 (osd.6) 104 : cluster [DBG] 8.6 scrub ok 2024-02-19T21:06:54.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:53.151392+0000 mgr.y (mgr.14550) 254 : cluster [DBG] pgmap v320: 380 pgs: 32 creating+peering, 1 peering, 3 active+clean+snaptrim, 32 unknown, 4 active+clean+snaptrim_wait, 308 active+clean; 9.1 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-02-19T21:06:54.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:53.425178+0000 osd.5 (osd.5) 114 : cluster [DBG] 5.f scrub starts 2024-02-19T21:06:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:53.428074+0000 osd.5 (osd.5) 115 : cluster [DBG] 5.f scrub ok 2024-02-19T21:06:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:53.751574+0000 osd.3 (osd.3) 117 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:06:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:53.752535+0000 osd.3 (osd.3) 118 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:06:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:54.184419+0000 mon.a (mon.0) 1644 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:06:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:54.185655+0000 mon.a (mon.0) 1645 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:06:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:54.190994+0000 mon.a (mon.0) 1646 : audit [INF] from='client.49262 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi163-83845-33"}]': finished 2024-02-19T21:06:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:54.191183+0000 mon.a (mon.0) 1647 : audit [INF] from='client.48106 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi163-83854-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:54.191259+0000 mon.a (mon.0) 1648 : audit [INF] from='client.48112 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:06:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:54.191317+0000 mon.a (mon.0) 1649 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-02-19T21:06:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:54.199203+0000 mon.b (mon.1) 368 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:54.201733+0000 mon.a (mon.0) 1650 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:06:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:54.234809+0000 mon.c (mon.2) 263 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:54.239360+0000 mon.a (mon.0) 1651 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:54.241819+0000 mon.c (mon.2) 264 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:54.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:54.242997+0000 mon.a (mon.0) 1652 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:54.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:54.245574+0000 mon.c (mon.2) 265 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi163-83845-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:54.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:54.245958+0000 mon.a (mon.0) 1653 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi163-83845-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:06:54.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:54.327979+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:54.908 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:06:54.935 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:55.284 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:06:55.287 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:55.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:06:55.389 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:55.416 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:55.540 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:55.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:55.644 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:55.676 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:55.697 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:55.699 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:55.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:55.801 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:55.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:53.628571+0000 osd.1 (osd.1) 129 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:06:55.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:53.629471+0000 osd.1 (osd.1) 130 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:06:55.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:54.352901+0000 osd.2 (osd.2) 103 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:06:55.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:54.355805+0000 osd.2 (osd.2) 104 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:06:55.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:54.376187+0000 osd.5 (osd.5) 116 : cluster [DBG] 8.8 scrub starts 2024-02-19T21:06:55.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:54.379081+0000 osd.5 (osd.5) 117 : cluster [DBG] 8.8 scrub ok 2024-02-19T21:06:55.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:54.663837+0000 osd.6 (osd.6) 105 : cluster [DBG] 16.5 deep-scrub starts 2024-02-19T21:06:55.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:54.664809+0000 osd.6 (osd.6) 106 : cluster [DBG] 16.5 deep-scrub ok 2024-02-19T21:06:55.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:54.722238+0000 osd.3 (osd.3) 119 : cluster [DBG] 6.1f deep-scrub starts 2024-02-19T21:06:55.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:54.723148+0000 osd.3 (osd.3) 120 : cluster [DBG] 6.1f deep-scrub ok 2024-02-19T21:06:55.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:55.194801+0000 mon.a (mon.0) 1654 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:55.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:55.194962+0000 mon.a (mon.0) 1655 : audit [INF] from='client.49298 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi163-83845-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:55.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:55.195026+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-02-19T21:06:55.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:55.197823+0000 mon.c (mon.2) 267 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi163-83845-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:55.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:55.200422+0000 mon.a (mon.0) 1657 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi163-83845-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:55.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:55.328843+0000 mon.c (mon.2) 268 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:55.831 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:55.850 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:55.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:55.955 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:55.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:53.628571+0000 osd.1 (osd.1) 129 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:06:55.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:53.629471+0000 osd.1 (osd.1) 130 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:06:55.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:54.352901+0000 osd.2 (osd.2) 103 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:06:55.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:54.355805+0000 osd.2 (osd.2) 104 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:06:55.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:54.376187+0000 osd.5 (osd.5) 116 : cluster [DBG] 8.8 scrub starts 2024-02-19T21:06:55.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:54.379081+0000 osd.5 (osd.5) 117 : cluster [DBG] 8.8 scrub ok 2024-02-19T21:06:55.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:54.663837+0000 osd.6 (osd.6) 105 : cluster [DBG] 16.5 deep-scrub starts 2024-02-19T21:06:55.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:54.664809+0000 osd.6 (osd.6) 106 : cluster [DBG] 16.5 deep-scrub ok 2024-02-19T21:06:55.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:54.722238+0000 osd.3 (osd.3) 119 : cluster [DBG] 6.1f deep-scrub starts 2024-02-19T21:06:55.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:54.723148+0000 osd.3 (osd.3) 120 : cluster [DBG] 6.1f deep-scrub ok 2024-02-19T21:06:55.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:55.194801+0000 mon.a (mon.0) 1654 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:55.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:55.194962+0000 mon.a (mon.0) 1655 : audit [INF] from='client.49298 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi163-83845-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:55.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:55.195026+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-02-19T21:06:55.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:55.197823+0000 mon.c (mon.2) 267 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi163-83845-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:55.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:55.200422+0000 mon.a (mon.0) 1657 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi163-83845-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:55.985 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:55.328843+0000 mon.c (mon.2) 268 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:55.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:53.628571+0000 osd.1 (osd.1) 129 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:06:55.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:53.629471+0000 osd.1 (osd.1) 130 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:06:55.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:54.352901+0000 osd.2 (osd.2) 103 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:06:55.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:54.355805+0000 osd.2 (osd.2) 104 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:06:55.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:54.376187+0000 osd.5 (osd.5) 116 : cluster [DBG] 8.8 scrub starts 2024-02-19T21:06:55.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:54.379081+0000 osd.5 (osd.5) 117 : cluster [DBG] 8.8 scrub ok 2024-02-19T21:06:55.985 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:54.663837+0000 osd.6 (osd.6) 105 : cluster [DBG] 16.5 deep-scrub starts 2024-02-19T21:06:55.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:54.664809+0000 osd.6 (osd.6) 106 : cluster [DBG] 16.5 deep-scrub ok 2024-02-19T21:06:55.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:54.722238+0000 osd.3 (osd.3) 119 : cluster [DBG] 6.1f deep-scrub starts 2024-02-19T21:06:55.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:54.723148+0000 osd.3 (osd.3) 120 : cluster [DBG] 6.1f deep-scrub ok 2024-02-19T21:06:55.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:55.194801+0000 mon.a (mon.0) 1654 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:06:55.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:55.194962+0000 mon.a (mon.0) 1655 : audit [INF] from='client.49298 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi163-83845-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:06:55.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:55.195026+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-02-19T21:06:55.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:55.197823+0000 mon.c (mon.2) 267 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi163-83845-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:55.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:55.200422+0000 mon.a (mon.0) 1657 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi163-83845-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:55.986 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:55.328843+0000 mon.c (mon.2) 268 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:55.986 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:56.059 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:56.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:56.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:56.173 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:06:56.208 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:56.456 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:06:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:56.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:06:56.559 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:56.592 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:56.613 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:56.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:56.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:56.708 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:55.152816+0000 mgr.y (mgr.14550) 255 : cluster [DBG] pgmap v323: 412 pgs: 2 clean+premerge+peered, 372 active+clean, 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 32 creating+peering; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:56.708 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:56.329561+0000 mon.c (mon.2) 269 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:56.708 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:56.438394+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-02-19T21:06:56.708 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:56.441277+0000 mon.b (mon.1) 369 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:56.708 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:56.441626+0000 mon.a (mon.0) 1659 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:56.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:55.152816+0000 mgr.y (mgr.14550) 255 : cluster [DBG] pgmap v323: 412 pgs: 2 clean+premerge+peered, 372 active+clean, 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 32 creating+peering; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:56.709 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:56.329561+0000 mon.c (mon.2) 269 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:56.709 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:56.438394+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-02-19T21:06:56.709 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:56.441277+0000 mon.b (mon.1) 369 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:56.709 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:56.441626+0000 mon.a (mon.0) 1659 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:56.717 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:06:56.746 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:56.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:55.152816+0000 mgr.y (mgr.14550) 255 : cluster [DBG] pgmap v323: 412 pgs: 2 clean+premerge+peered, 372 active+clean, 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 32 creating+peering; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:56.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:56.329561+0000 mon.c (mon.2) 269 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:56.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:56.438394+0000 mon.a (mon.0) 1658 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-02-19T21:06:56.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:56.441277+0000 mon.b (mon.1) 369 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:56.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:56.441626+0000 mon.a (mon.0) 1659 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:06:57.289 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:06:57.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.010+0000 7ff768095700 1 -- 172.21.15.163:0/1934658837 >> v1:172.21.15.163:6790/0 conn(0x7ff7600fd980 legacy=0x7ff7600fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:57.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.010+0000 7ff768095700 1 -- 172.21.15.163:0/1934658837 shutdown_connections 2024-02-19T21:06:57.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.010+0000 7ff768095700 1 -- 172.21.15.163:0/1934658837 >> 172.21.15.163:0/1934658837 conn(0x7ff7600f35a0 msgr2=0x7ff7600f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:57.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.010+0000 7ff768095700 1 -- 172.21.15.163:0/1934658837 shutdown_connections 2024-02-19T21:06:57.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.010+0000 7ff768095700 1 -- 172.21.15.163:0/1934658837 wait complete. 2024-02-19T21:06:57.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff768095700 1 librados: starting msgr at 2024-02-19T21:06:57.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff768095700 1 librados: starting objecter 2024-02-19T21:06:57.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff768095700 1 Processor -- start 2024-02-19T21:06:57.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff768095700 1 -- start start 2024-02-19T21:06:57.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff768095700 1 librados: setting wanted keys 2024-02-19T21:06:57.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff768095700 1 librados: calling monclient init 2024-02-19T21:06:57.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff768095700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff7600fcf70 con 0x7ff7600fd980 2024-02-19T21:06:57.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff768095700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff7600fd160 con 0x7ff7601042c0 2024-02-19T21:06:57.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff768095700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff7600fd350 con 0x7ff760100780 2024-02-19T21:06:57.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff765e31700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ff7600fd980 0x7ff7600fc7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33182/0 (socket says 172.21.15.163:33182) 2024-02-19T21:06:57.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff765e31700 1 -- 172.21.15.163:0/3440348924 learned_addr learned my addr 172.21.15.163:0/3440348924 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:06:57.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff765630700 1 --1- 172.21.15.163:0/3440348924 >> v1:172.21.15.163:6790/0 conn(0x7ff760100780 0x7ff760114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45172/0 (socket says 172.21.15.163:45172) 2024-02-19T21:06:57.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff756ffd700 10 client.?.objecter ms_handle_connect 0x7ff7600fd980 2024-02-19T21:06:57.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff756ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:57.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff756ffd700 10 client.?.objecter ms_handle_connect 0x7ff760100780 2024-02-19T21:06:57.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.011+0000 7ff756ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:57.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 10 client.?.objecter ms_handle_connect 0x7ff7601042c0 2024-02-19T21:06:57.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:06:57.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450393303 0 0) 0x7ff7600fcf70 con 0x7ff7600fd980 2024-02-19T21:06:57.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff74c003030 con 0x7ff7600fd980 2024-02-19T21:06:57.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772868610 0 0) 0x7ff7600fd350 con 0x7ff760100780 2024-02-19T21:06:57.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7600fcf70 con 0x7ff760100780 2024-02-19T21:06:57.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955926431 0 0) 0x7ff7600fd160 con 0x7ff7601042c0 2024-02-19T21:06:57.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7600fd350 con 0x7ff7601042c0 2024-02-19T21:06:57.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1575000652 0 0) 0x7ff7600fcf70 con 0x7ff760100780 2024-02-19T21:06:57.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7600fd160 con 0x7ff760100780 2024-02-19T21:06:57.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff75c001a30 con 0x7ff760100780 2024-02-19T21:06:57.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 217929402 0 0) 0x7ff74c003030 con 0x7ff7600fd980 2024-02-19T21:06:57.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7600fcf70 con 0x7ff7600fd980 2024-02-19T21:06:57.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff750001430 con 0x7ff7600fd980 2024-02-19T21:06:57.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4123276465 0 0) 0x7ff7600fd350 con 0x7ff7601042c0 2024-02-19T21:06:57.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff74c003030 con 0x7ff7601042c0 2024-02-19T21:06:57.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff758001b80 con 0x7ff7601042c0 2024-02-19T21:06:57.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 570739335 0 0) 0x7ff7600fd160 con 0x7ff760100780 2024-02-19T21:06:57.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 >> v1:172.21.15.193:6789/0 conn(0x7ff7601042c0 legacy=0x7ff760118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:57.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 >> v1:172.21.15.163:6789/0 conn(0x7ff7600fd980 legacy=0x7ff7600fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:57.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76011cb90 con 0x7ff760100780 2024-02-19T21:06:57.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff768095700 1 -- 172.21.15.163:0/3440348924 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff760119bc0 con 0x7ff760100780 2024-02-19T21:06:57.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff768095700 10 client.49304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:06:57.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff768095700 1 -- 172.21.15.163:0/3440348924 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff76011a030 con 0x7ff760100780 2024-02-19T21:06:57.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff768095700 1 librados: init done 2024-02-19T21:06:57.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff75c0023a0 con 0x7ff760100780 2024-02-19T21:06:57.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.012+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff75c0059d0 con 0x7ff760100780 2024-02-19T21:06:57.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.013+0000 7ff768095700 1 -- 172.21.15.163:0/3440348924 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff744004f20 con 0x7ff760100780 2024-02-19T21:06:57.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.013+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ff75c018920 con 0x7ff760100780 2024-02-19T21:06:57.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.014+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(236..236 src has 43..236) v4 ==== 9933+0+0 (unknown 3817580877 0 0) 0x7ff75c079370 con 0x7ff760100780 2024-02-19T21:06:57.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.014+0000 7ff756ffd700 10 client.49304.objecter ms_dispatch 0x7ff760001550 osd_map(236..236 src has 43..236) v4 2024-02-19T21:06:57.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.014+0000 7ff756ffd700 3 client.49304.objecter handle_osd_map got epochs [236,236] > 0 2024-02-19T21:06:57.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.014+0000 7ff756ffd700 3 client.49304.objecter handle_osd_map decoding full epoch 236 2024-02-19T21:06:57.338 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:06:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.014+0000 7ff756ffd700 20 client.49304.objecter dump_active .. 0 homeless 2024-02-19T21:06:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.014+0000 7ff756ffd700 10 client.49304.objecter ms_handle_connect 0x7ff74c063d10 2024-02-19T21:06:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.018+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff75c04c030 con 0x7ff760100780 2024-02-19T21:06:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.181+0000 7ff768095700 1 -- 172.21.15.163:0/3440348924 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff744000b90 con 0x7ff74c063d10 2024-02-19T21:06:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.212+0000 7ff756ffd700 1 -- 172.21.15.163:0/3440348924 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+857614 (unknown 2965378022 0 286034066) 0x7ff744000b90 con 0x7ff74c063d10 2024-02-19T21:06:57.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.343+0000 7ff768095700 10 librados: watch_flush enter 2024-02-19T21:06:57.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.343+0000 7ff768095700 10 librados: watch_flush exit 2024-02-19T21:06:57.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.343+0000 7ff768095700 20 client.49304.objecter shutdown clearing up homeless session... 2024-02-19T21:06:57.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.343+0000 7ff768095700 10 client.49304.objecter successfully canceled tick 2024-02-19T21:06:57.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.343+0000 7ff768095700 1 -- 172.21.15.163:0/3440348924 >> v1:172.21.15.163:6800/1552514681 conn(0x7ff74c063d10 legacy=0x7ff74c0661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:57.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.344+0000 7ff768095700 1 -- 172.21.15.163:0/3440348924 >> v1:172.21.15.163:6790/0 conn(0x7ff760100780 legacy=0x7ff760114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:06:57.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.344+0000 7ff768095700 1 -- 172.21.15.163:0/3440348924 shutdown_connections 2024-02-19T21:06:57.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.344+0000 7ff768095700 1 -- 172.21.15.163:0/3440348924 >> 172.21.15.163:0/3440348924 conn(0x7ff7600f35a0 msgr2=0x7ff760107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:06:57.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.344+0000 7ff768095700 1 -- 172.21.15.163:0/3440348924 shutdown_connections 2024-02-19T21:06:57.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.344+0000 7ff768095700 1 -- 172.21.15.163:0/3440348924 wait complete. 2024-02-19T21:06:57.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:06:57.344+0000 7ff768095700 1 librados: shutdown 2024-02-19T21:06:57.348 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:06:57.789 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:06:57.789 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:06:57.790 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-02-19T21:06:57.790 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:06:57.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:56.630656+0000 osd.7 (osd.7) 121 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:06:57.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:56.633617+0000 osd.7 (osd.7) 122 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:06:57.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:56.649523+0000 osd.6 (osd.6) 107 : cluster [DBG] 8.9 scrub starts 2024-02-19T21:06:57.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:56.652412+0000 osd.6 (osd.6) 108 : cluster [DBG] 8.9 scrub ok 2024-02-19T21:06:57.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:56.708628+0000 osd.3 (osd.3) 121 : cluster [DBG] 5.17 scrub starts 2024-02-19T21:06:57.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:56.709745+0000 osd.3 (osd.3) 122 : cluster [DBG] 5.17 scrub ok 2024-02-19T21:06:57.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:57.153874+0000 mon.b (mon.1) 370 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-19T21:06:57.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:57.154981+0000 mon.a (mon.0) 1660 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-19T21:06:57.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:57.195714+0000 mon.a (mon.0) 1661 : audit [INF] from='client.49298 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi163-83845-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi163-83845-34"}]': finished 2024-02-19T21:06:57.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:57.195834+0000 mon.a (mon.0) 1662 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:57.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:57.195903+0000 mon.a (mon.0) 1663 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-02-19T21:06:57.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:57.195961+0000 mon.a (mon.0) 1664 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-02-19T21:06:57.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:57.200186+0000 mon.b (mon.1) 371 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-25"}]: dispatch 2024-02-19T21:06:57.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:57.203130+0000 mon.a (mon.0) 1665 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-25"}]: dispatch 2024-02-19T21:06:57.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:57.330240+0000 mon.c (mon.2) 270 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:57.816 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:56.630656+0000 osd.7 (osd.7) 121 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:06:57.816 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:56.633617+0000 osd.7 (osd.7) 122 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:06:57.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:56.649523+0000 osd.6 (osd.6) 107 : cluster [DBG] 8.9 scrub starts 2024-02-19T21:06:57.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:56.652412+0000 osd.6 (osd.6) 108 : cluster [DBG] 8.9 scrub ok 2024-02-19T21:06:57.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:56.708628+0000 osd.3 (osd.3) 121 : cluster [DBG] 5.17 scrub starts 2024-02-19T21:06:57.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:56.709745+0000 osd.3 (osd.3) 122 : cluster [DBG] 5.17 scrub ok 2024-02-19T21:06:57.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:57.153874+0000 mon.b (mon.1) 370 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-19T21:06:57.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:57.154981+0000 mon.a (mon.0) 1660 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-19T21:06:57.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:57.195714+0000 mon.a (mon.0) 1661 : audit [INF] from='client.49298 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi163-83845-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi163-83845-34"}]': finished 2024-02-19T21:06:57.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:57.195834+0000 mon.a (mon.0) 1662 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:57.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:57.195903+0000 mon.a (mon.0) 1663 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-02-19T21:06:57.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:57.195961+0000 mon.a (mon.0) 1664 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-02-19T21:06:57.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:57.200186+0000 mon.b (mon.1) 371 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-25"}]: dispatch 2024-02-19T21:06:57.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:57.203130+0000 mon.a (mon.0) 1665 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-25"}]: dispatch 2024-02-19T21:06:57.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:57.330240+0000 mon.c (mon.2) 270 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:57.818 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:06:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:06:57.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:56.630656+0000 osd.7 (osd.7) 121 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:06:57.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:56.633617+0000 osd.7 (osd.7) 122 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:06:57.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:56.649523+0000 osd.6 (osd.6) 107 : cluster [DBG] 8.9 scrub starts 2024-02-19T21:06:57.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:56.652412+0000 osd.6 (osd.6) 108 : cluster [DBG] 8.9 scrub ok 2024-02-19T21:06:57.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:56.708628+0000 osd.3 (osd.3) 121 : cluster [DBG] 5.17 scrub starts 2024-02-19T21:06:57.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:56.709745+0000 osd.3 (osd.3) 122 : cluster [DBG] 5.17 scrub ok 2024-02-19T21:06:57.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:57.153874+0000 mon.b (mon.1) 370 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-19T21:06:57.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:57.154981+0000 mon.a (mon.0) 1660 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-02-19T21:06:57.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:57.195714+0000 mon.a (mon.0) 1661 : audit [INF] from='client.49298 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi163-83845-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi163-83845-34"}]': finished 2024-02-19T21:06:57.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:57.195834+0000 mon.a (mon.0) 1662 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:06:57.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:57.195903+0000 mon.a (mon.0) 1663 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-02-19T21:06:57.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:57.195961+0000 mon.a (mon.0) 1664 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-02-19T21:06:57.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:57.200186+0000 mon.b (mon.1) 371 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-25"}]: dispatch 2024-02-19T21:06:57.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:57.203130+0000 mon.a (mon.0) 1665 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-25"}]: dispatch 2024-02-19T21:06:57.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:57.330240+0000 mon.c (mon.2) 270 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:58.098 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:06:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:58.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:06:58.202 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:06:58.238 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:58.259 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:06:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:58.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:06:58.363 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:58.391 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:58.512 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-02-19T21:06:57.152976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8920,"num_objects":324,"num_object_clones":3,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":1137,"num_read_kb":798,"num_write":1082,"num_write_kb":637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":1574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2252,"ondisk_log_size":2252,"up":1041,"acting":1041,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1097,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2473200,"kb_used_data":29816,"kb_used_omap":0,"kb_used_meta":2443264,"kb_avail":747488016,"statfs":{"total":767960285184,"available":765427728384,"internally_reserved":0,"allocated":30531584,"data_stored":16767471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2501902336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-414,"num_objects":-19,"num_object_clones":-15,"num_object_copies":-57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-18,"num_whiteouts":-1,"num_read":-2,"num_read_kb":-2,"num_write":-17,"num_write_kb":-8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":-256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":-5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001876"},"pg_stats":[{"pgid":"161.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.190259+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"161.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.198686+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"161.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.190180+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"161.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.205205+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"161.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.203145+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"161.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.193605+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"161.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.193131+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"161.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.190520+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"161.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.190623+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"161.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.198313+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"161.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.196390+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"161.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.194723+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"161.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.202767+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"161.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.194937+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"161.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.204111+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"161.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.192503+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"161.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.196353+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"161.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.202686+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"161.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.207147+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"161.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.196714+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"161.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.195908+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"161.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.193257+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"161.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.207125+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"161.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.194766+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"161.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.195943+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"161.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.195059+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"161.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.206802+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"161.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.209504+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"161.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.207306+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"161.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.192865+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"161.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.192612+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"161.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:06:54.184850+0000","last_change":"2024-02-19T21:06:54.202232+0000","last_active":"2024-02-19T21:06:54.184850+0000","last_peered":"2024-02-19T21:06:54.184850+0000","last_clean":"2024-02-19T21:06:54.184850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:06:54.184850+0000","last_undegraded":"2024-02-19T21:06:54.184850+0000","last_fullsized":"2024-02-19T21:06:54.184850+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":234,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.184850+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.184850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"157.d","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.210416+0000","last_change":"2024-02-19T21:06:50.225845+0000","last_active":"2024-02-19T21:06:53.210416+0000","last_peered":"2024-02-19T21:06:53.210416+0000","last_clean":"2024-02-19T21:06:53.210416+0000","last_became_active":"2024-02-19T21:06:50.225480+0000","last_became_peered":"2024-02-19T21:06:50.225480+0000","last_unstale":"2024-02-19T21:06:53.210416+0000","last_undegraded":"2024-02-19T21:06:53.210416+0000","last_fullsized":"2024-02-19T21:06:53.210416+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"156.c","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215579+0000","last_change":"2024-02-19T21:06:48.211189+0000","last_active":"2024-02-19T21:06:53.215579+0000","last_peered":"2024-02-19T21:06:53.215579+0000","last_clean":"2024-02-19T21:06:53.215579+0000","last_became_active":"2024-02-19T21:06:48.210859+0000","last_became_peered":"2024-02-19T21:06:48.210859+0000","last_unstale":"2024-02-19T21:06:53.215579+0000","last_undegraded":"2024-02-19T21:06:53.215579+0000","last_fullsized":"2024-02-19T21:06:53.215579+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"155.b","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.218459+0000","last_change":"2024-02-19T21:06:46.212631+0000","last_active":"2024-02-19T21:06:53.218459+0000","last_peered":"2024-02-19T21:06:53.218459+0000","last_clean":"2024-02-19T21:06:53.218459+0000","last_became_active":"2024-02-19T21:06:46.212348+0000","last_became_peered":"2024-02-19T21:06:46.212348+0000","last_unstale":"2024-02-19T21:06:53.218459+0000","last_undegraded":"2024-02-19T21:06:53.218459+0000","last_fullsized":"2024-02-19T21:06:53.218459+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"157.c","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.211968+0000","last_change":"2024-02-19T21:06:50.221995+0000","last_active":"2024-02-19T21:06:53.211968+0000","last_peered":"2024-02-19T21:06:53.211968+0000","last_clean":"2024-02-19T21:06:53.211968+0000","last_became_active":"2024-02-19T21:06:50.221341+0000","last_became_peered":"2024-02-19T21:06:50.221341+0000","last_unstale":"2024-02-19T21:06:53.211968+0000","last_undegraded":"2024-02-19T21:06:53.211968+0000","last_fullsized":"2024-02-19T21:06:53.211968+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"156.d","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.210708+0000","last_change":"2024-02-19T21:06:48.212001+0000","last_active":"2024-02-19T21:06:53.210708+0000","last_peered":"2024-02-19T21:06:53.210708+0000","last_clean":"2024-02-19T21:06:53.210708+0000","last_became_active":"2024-02-19T21:06:48.211693+0000","last_became_peered":"2024-02-19T21:06:48.211693+0000","last_unstale":"2024-02-19T21:06:53.210708+0000","last_undegraded":"2024-02-19T21:06:53.210708+0000","last_fullsized":"2024-02-19T21:06:53.210708+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"155.a","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.212016+0000","last_change":"2024-02-19T21:06:46.212678+0000","last_active":"2024-02-19T21:06:53.212016+0000","last_peered":"2024-02-19T21:06:53.212016+0000","last_clean":"2024-02-19T21:06:53.212016+0000","last_became_active":"2024-02-19T21:06:46.212312+0000","last_became_peered":"2024-02-19T21:06:46.212312+0000","last_unstale":"2024-02-19T21:06:53.212016+0000","last_undegraded":"2024-02-19T21:06:53.212016+0000","last_fullsized":"2024-02-19T21:06:53.212016+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"157.f","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.205997+0000","last_change":"2024-02-19T21:06:50.214950+0000","last_active":"2024-02-19T21:06:53.205997+0000","last_peered":"2024-02-19T21:06:53.205997+0000","last_clean":"2024-02-19T21:06:53.205997+0000","last_became_active":"2024-02-19T21:06:50.214651+0000","last_became_peered":"2024-02-19T21:06:50.214651+0000","last_unstale":"2024-02-19T21:06:53.205997+0000","last_undegraded":"2024-02-19T21:06:53.205997+0000","last_fullsized":"2024-02-19T21:06:53.205997+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"156.e","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.217381+0000","last_change":"2024-02-19T21:06:48.203968+0000","last_active":"2024-02-19T21:06:53.217381+0000","last_peered":"2024-02-19T21:06:53.217381+0000","last_clean":"2024-02-19T21:06:53.217381+0000","last_became_active":"2024-02-19T21:06:48.203719+0000","last_became_peered":"2024-02-19T21:06:48.203719+0000","last_unstale":"2024-02-19T21:06:53.217381+0000","last_undegraded":"2024-02-19T21:06:53.217381+0000","last_fullsized":"2024-02-19T21:06:53.217381+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"155.9","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.220489+0000","last_change":"2024-02-19T21:06:46.209721+0000","last_active":"2024-02-19T21:06:53.220489+0000","last_peered":"2024-02-19T21:06:53.220489+0000","last_clean":"2024-02-19T21:06:53.220489+0000","last_became_active":"2024-02-19T21:06:46.209372+0000","last_became_peered":"2024-02-19T21:06:46.209372+0000","last_unstale":"2024-02-19T21:06:53.220489+0000","last_undegraded":"2024-02-19T21:06:53.220489+0000","last_fullsized":"2024-02-19T21:06:53.220489+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"157.e","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216436+0000","last_change":"2024-02-19T21:06:50.228337+0000","last_active":"2024-02-19T21:06:53.216436+0000","last_peered":"2024-02-19T21:06:53.216436+0000","last_clean":"2024-02-19T21:06:53.216436+0000","last_became_active":"2024-02-19T21:06:50.227955+0000","last_became_peered":"2024-02-19T21:06:50.227955+0000","last_unstale":"2024-02-19T21:06:53.216436+0000","last_undegraded":"2024-02-19T21:06:53.216436+0000","last_fullsized":"2024-02-19T21:06:53.216436+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"156.f","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.223655+0000","last_change":"2024-02-19T21:06:48.213119+0000","last_active":"2024-02-19T21:06:53.223655+0000","last_peered":"2024-02-19T21:06:53.223655+0000","last_clean":"2024-02-19T21:06:53.223655+0000","last_became_active":"2024-02-19T21:06:48.212777+0000","last_became_peered":"2024-02-19T21:06:48.212777+0000","last_unstale":"2024-02-19T21:06:53.223655+0000","last_undegraded":"2024-02-19T21:06:53.223655+0000","last_fullsized":"2024-02-19T21:06:53.223655+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"155.8","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215129+0000","last_change":"2024-02-19T21:06:46.213461+0000","last_active":"2024-02-19T21:06:53.215129+0000","last_peered":"2024-02-19T21:06:53.215129+0000","last_clean":"2024-02-19T21:06:53.215129+0000","last_became_active":"2024-02-19T21:06:46.212913+0000","last_became_peered":"2024-02-19T21:06:46.212913+0000","last_unstale":"2024-02-19T21:06:53.215129+0000","last_undegraded":"2024-02-19T21:06:53.215129+0000","last_fullsized":"2024-02-19T21:06:53.215129+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"157.9","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.219837+0000","last_change":"2024-02-19T21:06:50.225825+0000","last_active":"2024-02-19T21:06:53.219837+0000","last_peered":"2024-02-19T21:06:53.219837+0000","last_clean":"2024-02-19T21:06:53.219837+0000","last_became_active":"2024-02-19T21:06:50.225511+0000","last_became_peered":"2024-02-19T21:06:50.225511+0000","last_unstale":"2024-02-19T21:06:53.219837+0000","last_undegraded":"2024-02-19T21:06:53.219837+0000","last_fullsized":"2024-02-19T21:06:53.219837+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"156.8","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.206207+0000","last_change":"2024-02-19T21:06:48.212327+0000","last_active":"2024-02-19T21:06:53.206207+0000","last_peered":"2024-02-19T21:06:53.206207+0000","last_clean":"2024-02-19T21:06:53.206207+0000","last_became_active":"2024-02-19T21:06:48.211982+0000","last_became_peered":"2024-02-19T21:06:48.211982+0000","last_unstale":"2024-02-19T21:06:53.206207+0000","last_undegraded":"2024-02-19T21:06:53.206207+0000","last_fullsized":"2024-02-19T21:06:53.206207+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"155.f","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.212837+0000","last_change":"2024-02-19T21:06:46.215417+0000","last_active":"2024-02-19T21:06:53.212837+0000","last_peered":"2024-02-19T21:06:53.212837+0000","last_clean":"2024-02-19T21:06:53.212837+0000","last_became_active":"2024-02-19T21:06:46.215159+0000","last_became_peered":"2024-02-19T21:06:46.215159+0000","last_unstale":"2024-02-19T21:06:53.212837+0000","last_undegraded":"2024-02-19T21:06:53.212837+0000","last_fullsized":"2024-02-19T21:06:53.212837+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"157.8","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.218193+0000","last_change":"2024-02-19T21:06:50.230658+0000","last_active":"2024-02-19T21:06:53.218193+0000","last_peered":"2024-02-19T21:06:53.218193+0000","last_clean":"2024-02-19T21:06:53.218193+0000","last_became_active":"2024-02-19T21:06:50.229549+0000","last_became_peered":"2024-02-19T21:06:50.229549+0000","last_unstale":"2024-02-19T21:06:53.218193+0000","last_undegraded":"2024-02-19T21:06:53.218193+0000","last_fullsized":"2024-02-19T21:06:53.218193+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"156.9","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.204177+0000","last_change":"2024-02-19T21:06:48.217150+0000","last_active":"2024-02-19T21:06:53.204177+0000","last_peered":"2024-02-19T21:06:53.204177+0000","last_clean":"2024-02-19T21:06:53.204177+0000","last_became_active":"2024-02-19T21:06:48.216871+0000","last_became_peered":"2024-02-19T21:06:48.216871+0000","last_unstale":"2024-02-19T21:06:53.204177+0000","last_undegraded":"2024-02-19T21:06:53.204177+0000","last_fullsized":"2024-02-19T21:06:53.204177+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"155.e","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.225846+0000","last_change":"2024-02-19T21:06:46.203516+0000","last_active":"2024-02-19T21:06:53.225846+0000","last_peered":"2024-02-19T21:06:53.225846+0000","last_clean":"2024-02-19T21:06:53.225846+0000","last_became_active":"2024-02-19T21:06:46.203180+0000","last_became_peered":"2024-02-19T21:06:46.203180+0000","last_unstale":"2024-02-19T21:06:53.225846+0000","last_undegraded":"2024-02-19T21:06:53.225846+0000","last_fullsized":"2024-02-19T21:06:53.225846+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"157.b","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.209367+0000","last_change":"2024-02-19T21:06:50.226071+0000","last_active":"2024-02-19T21:06:53.209367+0000","last_peered":"2024-02-19T21:06:53.209367+0000","last_clean":"2024-02-19T21:06:53.209367+0000","last_became_active":"2024-02-19T21:06:50.225781+0000","last_became_peered":"2024-02-19T21:06:50.225781+0000","last_unstale":"2024-02-19T21:06:53.209367+0000","last_undegraded":"2024-02-19T21:06:53.209367+0000","last_fullsized":"2024-02-19T21:06:53.209367+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"156.a","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.209664+0000","last_change":"2024-02-19T21:06:48.214427+0000","last_active":"2024-02-19T21:06:53.209664+0000","last_peered":"2024-02-19T21:06:53.209664+0000","last_clean":"2024-02-19T21:06:53.209664+0000","last_became_active":"2024-02-19T21:06:48.214093+0000","last_became_peered":"2024-02-19T21:06:48.214093+0000","last_unstale":"2024-02-19T21:06:53.209664+0000","last_undegraded":"2024-02-19T21:06:53.209664+0000","last_fullsized":"2024-02-19T21:06:53.209664+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"155.d","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224461+0000","last_change":"2024-02-19T21:06:46.215135+0000","last_active":"2024-02-19T21:06:53.224461+0000","last_peered":"2024-02-19T21:06:53.224461+0000","last_clean":"2024-02-19T21:06:53.224461+0000","last_became_active":"2024-02-19T21:06:46.214837+0000","last_became_peered":"2024-02-19T21:06:46.214837+0000","last_unstale":"2024-02-19T21:06:53.224461+0000","last_undegraded":"2024-02-19T21:06:53.224461+0000","last_fullsized":"2024-02-19T21:06:53.224461+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"157.a","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.211724+0000","last_change":"2024-02-19T21:06:50.228111+0000","last_active":"2024-02-19T21:06:53.211724+0000","last_peered":"2024-02-19T21:06:53.211724+0000","last_clean":"2024-02-19T21:06:53.211724+0000","last_became_active":"2024-02-19T21:06:50.227827+0000","last_became_peered":"2024-02-19T21:06:50.227827+0000","last_unstale":"2024-02-19T21:06:53.211724+0000","last_undegraded":"2024-02-19T21:06:53.211724+0000","last_fullsized":"2024-02-19T21:06:53.211724+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"156.b","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.209453+0000","last_change":"2024-02-19T21:06:48.214526+0000","last_active":"2024-02-19T21:06:53.209453+0000","last_peered":"2024-02-19T21:06:53.209453+0000","last_clean":"2024-02-19T21:06:53.209453+0000","last_became_active":"2024-02-19T21:06:48.214197+0000","last_became_peered":"2024-02-19T21:06:48.214197+0000","last_unstale":"2024-02-19T21:06:53.209453+0000","last_undegraded":"2024-02-19T21:06:53.209453+0000","last_fullsized":"2024-02-19T21:06:53.209453+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"155.c","version":"234'4","reported_seq":21,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-19T21:06:54.191939+0000","last_change":"2024-02-19T21:06:46.213095+0000","last_active":"2024-02-19T21:06:54.191939+0000","last_peered":"2024-02-19T21:06:54.191939+0000","last_clean":"2024-02-19T21:06:54.191939+0000","last_became_active":"2024-02-19T21:06:46.212736+0000","last_became_peered":"2024-02-19T21:06:46.212736+0000","last_unstale":"2024-02-19T21:06:54.191939+0000","last_undegraded":"2024-02-19T21:06:54.191939+0000","last_fullsized":"2024-02-19T21:06:54.191939+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"157.5","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.214307+0000","last_change":"2024-02-19T21:06:50.219100+0000","last_active":"2024-02-19T21:06:53.214307+0000","last_peered":"2024-02-19T21:06:53.214307+0000","last_clean":"2024-02-19T21:06:53.214307+0000","last_became_active":"2024-02-19T21:06:50.218563+0000","last_became_peered":"2024-02-19T21:06:50.218563+0000","last_unstale":"2024-02-19T21:06:53.214307+0000","last_undegraded":"2024-02-19T21:06:53.214307+0000","last_fullsized":"2024-02-19T21:06:53.214307+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"156.4","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.212057+0000","last_change":"2024-02-19T21:06:48.215273+0000","last_active":"2024-02-19T21:06:53.212057+0000","last_peered":"2024-02-19T21:06:53.212057+0000","last_clean":"2024-02-19T21:06:53.212057+0000","last_became_active":"2024-02-19T21:06:48.214865+0000","last_became_peered":"2024-02-19T21:06:48.214865+0000","last_unstale":"2024-02-19T21:06:53.212057+0000","last_undegraded":"2024-02-19T21:06:53.212057+0000","last_fullsized":"2024-02-19T21:06:53.212057+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"155.3","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.227765+0000","last_change":"2024-02-19T21:06:46.210133+0000","last_active":"2024-02-19T21:06:53.227765+0000","last_peered":"2024-02-19T21:06:53.227765+0000","last_clean":"2024-02-19T21:06:53.227765+0000","last_became_active":"2024-02-19T21:06:46.209785+0000","last_became_peered":"2024-02-19T21:06:46.209785+0000","last_unstale":"2024-02-19T21:06:53.227765+0000","last_undegraded":"2024-02-19T21:06:53.227765+0000","last_fullsized":"2024-02-19T21:06:53.227765+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"157.4","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.212332+0000","last_change":"2024-02-19T21:06:50.215114+0000","last_active":"2024-02-19T21:06:53.212332+0000","last_peered":"2024-02-19T21:06:53.212332+0000","last_clean":"2024-02-19T21:06:53.212332+0000","last_became_active":"2024-02-19T21:06:50.214804+0000","last_became_peered":"2024-02-19T21:06:50.214804+0000","last_unstale":"2024-02-19T21:06:53.212332+0000","last_undegraded":"2024-02-19T21:06:53.212332+0000","last_fullsized":"2024-02-19T21:06:53.212332+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"156.5","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.222935+0000","last_change":"2024-02-19T21:06:48.206753+0000","last_active":"2024-02-19T21:06:53.222935+0000","last_peered":"2024-02-19T21:06:53.222935+0000","last_clean":"2024-02-19T21:06:53.222935+0000","last_became_active":"2024-02-19T21:06:48.206463+0000","last_became_peered":"2024-02-19T21:06:48.206463+0000","last_unstale":"2024-02-19T21:06:53.222935+0000","last_undegraded":"2024-02-19T21:06:53.222935+0000","last_fullsized":"2024-02-19T21:06:53.222935+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"155.2","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.211616+0000","last_change":"2024-02-19T21:06:46.207348+0000","last_active":"2024-02-19T21:06:53.211616+0000","last_peered":"2024-02-19T21:06:53.211616+0000","last_clean":"2024-02-19T21:06:53.211616+0000","last_became_active":"2024-02-19T21:06:46.206908+0000","last_became_peered":"2024-02-19T21:06:46.206908+0000","last_unstale":"2024-02-19T21:06:53.211616+0000","last_undegraded":"2024-02-19T21:06:53.211616+0000","last_fullsized":"2024-02-19T21:06:53.211616+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"157.7","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.214771+0000","last_change":"2024-02-19T21:06:50.225016+0000","last_active":"2024-02-19T21:06:53.214771+0000","last_peered":"2024-02-19T21:06:53.214771+0000","last_clean":"2024-02-19T21:06:53.214771+0000","last_became_active":"2024-02-19T21:06:50.224733+0000","last_became_peered":"2024-02-19T21:06:50.224733+0000","last_unstale":"2024-02-19T21:06:53.214771+0000","last_undegraded":"2024-02-19T21:06:53.214771+0000","last_fullsized":"2024-02-19T21:06:53.214771+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"156.6","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.209998+0000","last_change":"2024-02-19T21:06:48.216208+0000","last_active":"2024-02-19T21:06:53.209998+0000","last_peered":"2024-02-19T21:06:53.209998+0000","last_clean":"2024-02-19T21:06:53.209998+0000","last_became_active":"2024-02-19T21:06:48.215919+0000","last_became_peered":"2024-02-19T21:06:48.215919+0000","last_unstale":"2024-02-19T21:06:53.209998+0000","last_undegraded":"2024-02-19T21:06:53.209998+0000","last_fullsized":"2024-02-19T21:06:53.209998+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"155.1","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.218711+0000","last_change":"2024-02-19T21:06:46.219974+0000","last_active":"2024-02-19T21:06:53.218711+0000","last_peered":"2024-02-19T21:06:53.218711+0000","last_clean":"2024-02-19T21:06:53.218711+0000","last_became_active":"2024-02-19T21:06:46.219538+0000","last_became_peered":"2024-02-19T21:06:46.219538+0000","last_unstale":"2024-02-19T21:06:53.218711+0000","last_undegraded":"2024-02-19T21:06:53.218711+0000","last_fullsized":"2024-02-19T21:06:53.218711+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"157.6","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.225475+0000","last_change":"2024-02-19T21:06:50.230124+0000","last_active":"2024-02-19T21:06:53.225475+0000","last_peered":"2024-02-19T21:06:53.225475+0000","last_clean":"2024-02-19T21:06:53.225475+0000","last_became_active":"2024-02-19T21:06:50.229525+0000","last_became_peered":"2024-02-19T21:06:50.229525+0000","last_unstale":"2024-02-19T21:06:53.225475+0000","last_undegraded":"2024-02-19T21:06:53.225475+0000","last_fullsized":"2024-02-19T21:06:53.225475+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"156.7","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215055+0000","last_change":"2024-02-19T21:06:48.216973+0000","last_active":"2024-02-19T21:06:53.215055+0000","last_peered":"2024-02-19T21:06:53.215055+0000","last_clean":"2024-02-19T21:06:53.215055+0000","last_became_active":"2024-02-19T21:06:48.216562+0000","last_became_peered":"2024-02-19T21:06:48.216562+0000","last_unstale":"2024-02-19T21:06:53.215055+0000","last_undegraded":"2024-02-19T21:06:53.215055+0000","last_fullsized":"2024-02-19T21:06:53.215055+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"155.0","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.222668+0000","last_change":"2024-02-19T21:06:46.217228+0000","last_active":"2024-02-19T21:06:53.222668+0000","last_peered":"2024-02-19T21:06:53.222668+0000","last_clean":"2024-02-19T21:06:53.222668+0000","last_became_active":"2024-02-19T21:06:46.216799+0000","last_became_peered":"2024-02-19T21:06:46.216799+0000","last_unstale":"2024-02-19T21:06:53.222668+0000","last_undegraded":"2024-02-19T21:06:53.222668+0000","last_fullsized":"2024-02-19T21:06:53.222668+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"157.1","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216921+0000","last_change":"2024-02-19T21:06:50.230551+0000","last_active":"2024-02-19T21:06:53.216921+0000","last_peered":"2024-02-19T21:06:53.216921+0000","last_clean":"2024-02-19T21:06:53.216921+0000","last_became_active":"2024-02-19T21:06:50.229614+0000","last_became_peered":"2024-02-19T21:06:50.229614+0000","last_unstale":"2024-02-19T21:06:53.216921+0000","last_undegraded":"2024-02-19T21:06:53.216921+0000","last_fullsized":"2024-02-19T21:06:53.216921+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"156.0","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215037+0000","last_change":"2024-02-19T21:06:48.205244+0000","last_active":"2024-02-19T21:06:53.215037+0000","last_peered":"2024-02-19T21:06:53.215037+0000","last_clean":"2024-02-19T21:06:53.215037+0000","last_became_active":"2024-02-19T21:06:48.204916+0000","last_became_peered":"2024-02-19T21:06:48.204916+0000","last_unstale":"2024-02-19T21:06:53.215037+0000","last_undegraded":"2024-02-19T21:06:53.215037+0000","last_fullsized":"2024-02-19T21:06:53.215037+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"155.7","version":"228'3","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.223807+0000","last_change":"2024-02-19T21:06:46.212954+0000","last_active":"2024-02-19T21:06:53.223807+0000","last_peered":"2024-02-19T21:06:53.223807+0000","last_clean":"2024-02-19T21:06:53.223807+0000","last_became_active":"2024-02-19T21:06:46.212229+0000","last_became_peered":"2024-02-19T21:06:46.212229+0000","last_unstale":"2024-02-19T21:06:53.223807+0000","last_undegraded":"2024-02-19T21:06:53.223807+0000","last_fullsized":"2024-02-19T21:06:53.223807+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"157.0","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.205964+0000","last_change":"2024-02-19T21:06:50.219678+0000","last_active":"2024-02-19T21:06:53.205964+0000","last_peered":"2024-02-19T21:06:53.205964+0000","last_clean":"2024-02-19T21:06:53.205964+0000","last_became_active":"2024-02-19T21:06:50.219231+0000","last_became_peered":"2024-02-19T21:06:50.219231+0000","last_unstale":"2024-02-19T21:06:53.205964+0000","last_undegraded":"2024-02-19T21:06:53.205964+0000","last_fullsized":"2024-02-19T21:06:53.205964+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"156.1","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.218565+0000","last_change":"2024-02-19T21:06:48.205219+0000","last_active":"2024-02-19T21:06:53.218565+0000","last_peered":"2024-02-19T21:06:53.218565+0000","last_clean":"2024-02-19T21:06:53.218565+0000","last_became_active":"2024-02-19T21:06:48.204913+0000","last_became_peered":"2024-02-19T21:06:48.204913+0000","last_unstale":"2024-02-19T21:06:53.218565+0000","last_undegraded":"2024-02-19T21:06:53.218565+0000","last_fullsized":"2024-02-19T21:06:53.218565+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"155.6","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.227147+0000","last_change":"2024-02-19T21:06:46.217372+0000","last_active":"2024-02-19T21:06:53.227147+0000","last_peered":"2024-02-19T21:06:53.227147+0000","last_clean":"2024-02-19T21:06:53.227147+0000","last_became_active":"2024-02-19T21:06:46.217124+0000","last_became_peered":"2024-02-19T21:06:46.217124+0000","last_unstale":"2024-02-19T21:06:53.227147+0000","last_undegraded":"2024-02-19T21:06:53.227147+0000","last_fullsized":"2024-02-19T21:06:53.227147+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"157.3","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.207382+0000","last_change":"2024-02-19T21:06:50.218544+0000","last_active":"2024-02-19T21:06:53.207382+0000","last_peered":"2024-02-19T21:06:53.207382+0000","last_clean":"2024-02-19T21:06:53.207382+0000","last_became_active":"2024-02-19T21:06:50.217831+0000","last_became_peered":"2024-02-19T21:06:50.217831+0000","last_unstale":"2024-02-19T21:06:53.207382+0000","last_undegraded":"2024-02-19T21:06:53.207382+0000","last_fullsized":"2024-02-19T21:06:53.207382+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"156.2","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.222100+0000","last_change":"2024-02-19T21:06:48.211750+0000","last_active":"2024-02-19T21:06:53.222100+0000","last_peered":"2024-02-19T21:06:53.222100+0000","last_clean":"2024-02-19T21:06:53.222100+0000","last_became_active":"2024-02-19T21:06:48.211458+0000","last_became_peered":"2024-02-19T21:06:48.211458+0000","last_unstale":"2024-02-19T21:06:53.222100+0000","last_undegraded":"2024-02-19T21:06:53.222100+0000","last_fullsized":"2024-02-19T21:06:53.222100+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"155.5","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.225884+0000","last_change":"2024-02-19T21:06:46.210241+0000","last_active":"2024-02-19T21:06:53.225884+0000","last_peered":"2024-02-19T21:06:53.225884+0000","last_clean":"2024-02-19T21:06:53.225884+0000","last_became_active":"2024-02-19T21:06:46.209961+0000","last_became_peered":"2024-02-19T21:06:46.209961+0000","last_unstale":"2024-02-19T21:06:53.225884+0000","last_undegraded":"2024-02-19T21:06:53.225884+0000","last_fullsized":"2024-02-19T21:06:53.225884+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"157.2","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.222554+0000","last_change":"2024-02-19T21:06:50.228083+0000","last_active":"2024-02-19T21:06:53.222554+0000","last_peered":"2024-02-19T21:06:53.222554+0000","last_clean":"2024-02-19T21:06:53.222554+0000","last_became_active":"2024-02-19T21:06:50.227751+0000","last_became_peered":"2024-02-19T21:06:50.227751+0000","last_unstale":"2024-02-19T21:06:53.222554+0000","last_undegraded":"2024-02-19T21:06:53.222554+0000","last_fullsized":"2024-02-19T21:06:53.222554+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"156.3","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.219207+0000","last_change":"2024-02-19T21:06:48.214070+0000","last_active":"2024-02-19T21:06:53.219207+0000","last_peered":"2024-02-19T21:06:53.219207+0000","last_clean":"2024-02-19T21:06:53.219207+0000","last_became_active":"2024-02-19T21:06:48.213775+0000","last_became_peered":"2024-02-19T21:06:48.213775+0000","last_unstale":"2024-02-19T21:06:53.219207+0000","last_undegraded":"2024-02-19T21:06:53.219207+0000","last_fullsized":"2024-02-19T21:06:53.219207+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"155.4","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.220707+0000","last_change":"2024-02-19T21:06:46.212971+0000","last_active":"2024-02-19T21:06:53.220707+0000","last_peered":"2024-02-19T21:06:53.220707+0000","last_clean":"2024-02-19T21:06:53.220707+0000","last_became_active":"2024-02-19T21:06:46.212354+0000","last_became_peered":"2024-02-19T21:06:46.212354+0000","last_unstale":"2024-02-19T21:06:53.220707+0000","last_undegraded":"2024-02-19T21:06:53.220707+0000","last_fullsized":"2024-02-19T21:06:53.220707+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"157.1d","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.209093+0000","last_change":"2024-02-19T21:06:50.218313+0000","last_active":"2024-02-19T21:06:53.209093+0000","last_peered":"2024-02-19T21:06:53.209093+0000","last_clean":"2024-02-19T21:06:53.209093+0000","last_became_active":"2024-02-19T21:06:50.218010+0000","last_became_peered":"2024-02-19T21:06:50.218010+0000","last_unstale":"2024-02-19T21:06:53.209093+0000","last_undegraded":"2024-02-19T21:06:53.209093+0000","last_fullsized":"2024-02-19T21:06:53.209093+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"156.1c","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.214386+0000","last_change":"2024-02-19T21:06:48.205899+0000","last_active":"2024-02-19T21:06:53.214386+0000","last_peered":"2024-02-19T21:06:53.214386+0000","last_clean":"2024-02-19T21:06:53.214386+0000","last_became_active":"2024-02-19T21:06:48.205565+0000","last_became_peered":"2024-02-19T21:06:48.205565+0000","last_unstale":"2024-02-19T21:06:53.214386+0000","last_undegraded":"2024-02-19T21:06:53.214386+0000","last_fullsized":"2024-02-19T21:06:53.214386+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"155.1b","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.217535+0000","last_change":"2024-02-19T21:06:46.206243+0000","last_active":"2024-02-19T21:06:53.217535+0000","last_peered":"2024-02-19T21:06:53.217535+0000","last_clean":"2024-02-19T21:06:53.217535+0000","last_became_active":"2024-02-19T21:06:46.205908+0000","last_became_peered":"2024-02-19T21:06:46.205908+0000","last_unstale":"2024-02-19T21:06:53.217535+0000","last_undegraded":"2024-02-19T21:06:53.217535+0000","last_fullsized":"2024-02-19T21:06:53.217535+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"157.1c","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.211863+0000","last_change":"2024-02-19T21:06:50.231716+0000","last_active":"2024-02-19T21:06:53.211863+0000","last_peered":"2024-02-19T21:06:53.211863+0000","last_clean":"2024-02-19T21:06:53.211863+0000","last_became_active":"2024-02-19T21:06:50.231433+0000","last_became_peered":"2024-02-19T21:06:50.231433+0000","last_unstale":"2024-02-19T21:06:53.211863+0000","last_undegraded":"2024-02-19T21:06:53.211863+0000","last_fullsized":"2024-02-19T21:06:53.211863+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"156.1d","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.211903+0000","last_change":"2024-02-19T21:06:48.215490+0000","last_active":"2024-02-19T21:06:53.211903+0000","last_peered":"2024-02-19T21:06:53.211903+0000","last_clean":"2024-02-19T21:06:53.211903+0000","last_became_active":"2024-02-19T21:06:48.215154+0000","last_became_peered":"2024-02-19T21:06:48.215154+0000","last_unstale":"2024-02-19T21:06:53.211903+0000","last_undegraded":"2024-02-19T21:06:53.211903+0000","last_fullsized":"2024-02-19T21:06:53.211903+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"155.1a","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.211507+0000","last_change":"2024-02-19T21:06:46.215219+0000","last_active":"2024-02-19T21:06:53.211507+0000","last_peered":"2024-02-19T21:06:53.211507+0000","last_clean":"2024-02-19T21:06:53.211507+0000","last_became_active":"2024-02-19T21:06:46.214526+0000","last_became_peered":"2024-02-19T21:06:46.214526+0000","last_unstale":"2024-02-19T21:06:53.211507+0000","last_undegraded":"2024-02-19T21:06:53.211507+0000","last_fullsized":"2024-02-19T21:06:53.211507+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"157.1f","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.205925+0000","last_change":"2024-02-19T21:06:50.225910+0000","last_active":"2024-02-19T21:06:53.205925+0000","last_peered":"2024-02-19T21:06:53.205925+0000","last_clean":"2024-02-19T21:06:53.205925+0000","last_became_active":"2024-02-19T21:06:50.225505+0000","last_became_peered":"2024-02-19T21:06:50.225505+0000","last_unstale":"2024-02-19T21:06:53.205925+0000","last_undegraded":"2024-02-19T21:06:53.205925+0000","last_fullsized":"2024-02-19T21:06:53.205925+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"156.1e","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.225651+0000","last_change":"2024-02-19T21:06:48.210334+0000","last_active":"2024-02-19T21:06:53.225651+0000","last_peered":"2024-02-19T21:06:53.225651+0000","last_clean":"2024-02-19T21:06:53.225651+0000","last_became_active":"2024-02-19T21:06:48.209970+0000","last_became_peered":"2024-02-19T21:06:48.209970+0000","last_unstale":"2024-02-19T21:06:53.225651+0000","last_undegraded":"2024-02-19T21:06:53.225651+0000","last_fullsized":"2024-02-19T21:06:53.225651+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"155.19","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.218638+0000","last_change":"2024-02-19T21:06:46.220156+0000","last_active":"2024-02-19T21:06:53.218638+0000","last_peered":"2024-02-19T21:06:53.218638+0000","last_clean":"2024-02-19T21:06:53.218638+0000","last_became_active":"2024-02-19T21:06:46.219806+0000","last_became_peered":"2024-02-19T21:06:46.219806+0000","last_unstale":"2024-02-19T21:06:53.218638+0000","last_undegraded":"2024-02-19T21:06:53.218638+0000","last_fullsized":"2024-02-19T21:06:53.218638+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"157.1e","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.217172+0000","last_change":"2024-02-19T21:06:50.225766+0000","last_active":"2024-02-19T21:06:53.217172+0000","last_peered":"2024-02-19T21:06:53.217172+0000","last_clean":"2024-02-19T21:06:53.217172+0000","last_became_active":"2024-02-19T21:06:50.225431+0000","last_became_peered":"2024-02-19T21:06:50.225431+0000","last_unstale":"2024-02-19T21:06:53.217172+0000","last_undegraded":"2024-02-19T21:06:53.217172+0000","last_fullsized":"2024-02-19T21:06:53.217172+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"156.1f","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.228413+0000","last_change":"2024-02-19T21:06:48.214181+0000","last_active":"2024-02-19T21:06:53.228413+0000","last_peered":"2024-02-19T21:06:53.228413+0000","last_clean":"2024-02-19T21:06:53.228413+0000","last_became_active":"2024-02-19T21:06:48.213899+0000","last_became_peered":"2024-02-19T21:06:48.213899+0000","last_unstale":"2024-02-19T21:06:53.228413+0000","last_undegraded":"2024-02-19T21:06:53.228413+0000","last_fullsized":"2024-02-19T21:06:53.228413+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"155.18","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.218846+0000","last_change":"2024-02-19T21:06:46.203293+0000","last_active":"2024-02-19T21:06:53.218846+0000","last_peered":"2024-02-19T21:06:53.218846+0000","last_clean":"2024-02-19T21:06:53.218846+0000","last_became_active":"2024-02-19T21:06:46.202974+0000","last_became_peered":"2024-02-19T21:06:46.202974+0000","last_unstale":"2024-02-19T21:06:53.218846+0000","last_undegraded":"2024-02-19T21:06:53.218846+0000","last_fullsized":"2024-02-19T21:06:53.218846+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"157.19","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.204049+0000","last_change":"2024-02-19T21:06:50.221892+0000","last_active":"2024-02-19T21:06:53.204049+0000","last_peered":"2024-02-19T21:06:53.204049+0000","last_clean":"2024-02-19T21:06:53.204049+0000","last_became_active":"2024-02-19T21:06:50.221539+0000","last_became_peered":"2024-02-19T21:06:50.221539+0000","last_unstale":"2024-02-19T21:06:53.204049+0000","last_undegraded":"2024-02-19T21:06:53.204049+0000","last_fullsized":"2024-02-19T21:06:53.204049+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"156.18","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.206119+0000","last_change":"2024-02-19T21:06:48.211679+0000","last_active":"2024-02-19T21:06:53.206119+0000","last_peered":"2024-02-19T21:06:53.206119+0000","last_clean":"2024-02-19T21:06:53.206119+0000","last_became_active":"2024-02-19T21:06:48.211345+0000","last_became_peered":"2024-02-19T21:06:48.211345+0000","last_unstale":"2024-02-19T21:06:53.206119+0000","last_undegraded":"2024-02-19T21:06:53.206119+0000","last_fullsized":"2024-02-19T21:06:53.206119+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"155.1f","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.208897+0000","last_change":"2024-02-19T21:06:46.207537+0000","last_active":"2024-02-19T21:06:53.208897+0000","last_peered":"2024-02-19T21:06:53.208897+0000","last_clean":"2024-02-19T21:06:53.208897+0000","last_became_active":"2024-02-19T21:06:46.207219+0000","last_became_peered":"2024-02-19T21:06:46.207219+0000","last_unstale":"2024-02-19T21:06:53.208897+0000","last_undegraded":"2024-02-19T21:06:53.208897+0000","last_fullsized":"2024-02-19T21:06:53.208897+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"157.18","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.222476+0000","last_change":"2024-02-19T21:06:50.226592+0000","last_active":"2024-02-19T21:06:53.222476+0000","last_peered":"2024-02-19T21:06:53.222476+0000","last_clean":"2024-02-19T21:06:53.222476+0000","last_became_active":"2024-02-19T21:06:50.226276+0000","last_became_peered":"2024-02-19T21:06:50.226276+0000","last_unstale":"2024-02-19T21:06:53.222476+0000","last_undegraded":"2024-02-19T21:06:53.222476+0000","last_fullsized":"2024-02-19T21:06:53.222476+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"156.19","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.217004+0000","last_change":"2024-02-19T21:06:48.207620+0000","last_active":"2024-02-19T21:06:53.217004+0000","last_peered":"2024-02-19T21:06:53.217004+0000","last_clean":"2024-02-19T21:06:53.217004+0000","last_became_active":"2024-02-19T21:06:48.207321+0000","last_became_peered":"2024-02-19T21:06:48.207321+0000","last_unstale":"2024-02-19T21:06:53.217004+0000","last_undegraded":"2024-02-19T21:06:53.217004+0000","last_fullsized":"2024-02-19T21:06:53.217004+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"155.1e","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.218738+0000","last_change":"2024-02-19T21:06:46.212535+0000","last_active":"2024-02-19T21:06:53.218738+0000","last_peered":"2024-02-19T21:06:53.218738+0000","last_clean":"2024-02-19T21:06:53.218738+0000","last_became_active":"2024-02-19T21:06:46.212291+0000","last_became_peered":"2024-02-19T21:06:46.212291+0000","last_unstale":"2024-02-19T21:06:53.218738+0000","last_undegraded":"2024-02-19T21:06:53.218738+0000","last_fullsized":"2024-02-19T21:06:53.218738+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"157.1b","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.225476+0000","last_change":"2024-02-19T21:06:50.223160+0000","last_active":"2024-02-19T21:06:53.225476+0000","last_peered":"2024-02-19T21:06:53.225476+0000","last_clean":"2024-02-19T21:06:53.225476+0000","last_became_active":"2024-02-19T21:06:50.222871+0000","last_became_peered":"2024-02-19T21:06:50.222871+0000","last_unstale":"2024-02-19T21:06:53.225476+0000","last_undegraded":"2024-02-19T21:06:53.225476+0000","last_fullsized":"2024-02-19T21:06:53.225476+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"156.1a","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.222638+0000","last_change":"2024-02-19T21:06:48.211439+0000","last_active":"2024-02-19T21:06:53.222638+0000","last_peered":"2024-02-19T21:06:53.222638+0000","last_clean":"2024-02-19T21:06:53.222638+0000","last_became_active":"2024-02-19T21:06:48.211048+0000","last_became_peered":"2024-02-19T21:06:48.211048+0000","last_unstale":"2024-02-19T21:06:53.222638+0000","last_undegraded":"2024-02-19T21:06:53.222638+0000","last_fullsized":"2024-02-19T21:06:53.222638+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"155.1d","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.210933+0000","last_change":"2024-02-19T21:06:46.206509+0000","last_active":"2024-02-19T21:06:53.210933+0000","last_peered":"2024-02-19T21:06:53.210933+0000","last_clean":"2024-02-19T21:06:53.210933+0000","last_became_active":"2024-02-19T21:06:46.206216+0000","last_became_peered":"2024-02-19T21:06:46.206216+0000","last_unstale":"2024-02-19T21:06:53.210933+0000","last_undegraded":"2024-02-19T21:06:53.210933+0000","last_fullsized":"2024-02-19T21:06:53.210933+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"157.1a","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.211325+0000","last_change":"2024-02-19T21:06:50.227722+0000","last_active":"2024-02-19T21:06:53.211325+0000","last_peered":"2024-02-19T21:06:53.211325+0000","last_clean":"2024-02-19T21:06:53.211325+0000","last_became_active":"2024-02-19T21:06:50.226501+0000","last_became_peered":"2024-02-19T21:06:50.226501+0000","last_unstale":"2024-02-19T21:06:53.211325+0000","last_undegraded":"2024-02-19T21:06:53.211325+0000","last_fullsized":"2024-02-19T21:06:53.211325+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"156.1b","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.217449+0000","last_change":"2024-02-19T21:06:48.214596+0000","last_active":"2024-02-19T21:06:53.217449+0000","last_peered":"2024-02-19T21:06:53.217449+0000","last_clean":"2024-02-19T21:06:53.217449+0000","last_became_active":"2024-02-19T21:06:48.214294+0000","last_became_peered":"2024-02-19T21:06:48.214294+0000","last_unstale":"2024-02-19T21:06:53.217449+0000","last_undegraded":"2024-02-19T21:06:53.217449+0000","last_fullsized":"2024-02-19T21:06:53.217449+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"155.1c","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224309+0000","last_change":"2024-02-19T21:06:46.207562+0000","last_active":"2024-02-19T21:06:53.224309+0000","last_peered":"2024-02-19T21:06:53.224309+0000","last_clean":"2024-02-19T21:06:53.224309+0000","last_became_active":"2024-02-19T21:06:46.207061+0000","last_became_peered":"2024-02-19T21:06:46.207061+0000","last_unstale":"2024-02-19T21:06:53.224309+0000","last_undegraded":"2024-02-19T21:06:53.224309+0000","last_fullsized":"2024-02-19T21:06:53.224309+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"157.15","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.211806+0000","last_change":"2024-02-19T21:06:50.224073+0000","last_active":"2024-02-19T21:06:53.211806+0000","last_peered":"2024-02-19T21:06:53.211806+0000","last_clean":"2024-02-19T21:06:53.211806+0000","last_became_active":"2024-02-19T21:06:50.223778+0000","last_became_peered":"2024-02-19T21:06:50.223778+0000","last_unstale":"2024-02-19T21:06:53.211806+0000","last_undegraded":"2024-02-19T21:06:53.211806+0000","last_fullsized":"2024-02-19T21:06:53.211806+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"156.14","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216183+0000","last_change":"2024-02-19T21:06:48.216114+0000","last_active":"2024-02-19T21:06:53.216183+0000","last_peered":"2024-02-19T21:06:53.216183+0000","last_clean":"2024-02-19T21:06:53.216183+0000","last_became_active":"2024-02-19T21:06:48.215858+0000","last_became_peered":"2024-02-19T21:06:48.215858+0000","last_unstale":"2024-02-19T21:06:53.216183+0000","last_undegraded":"2024-02-19T21:06:53.216183+0000","last_fullsized":"2024-02-19T21:06:53.216183+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"155.13","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.218374+0000","last_change":"2024-02-19T21:06:46.212525+0000","last_active":"2024-02-19T21:06:53.218374+0000","last_peered":"2024-02-19T21:06:53.218374+0000","last_clean":"2024-02-19T21:06:53.218374+0000","last_became_active":"2024-02-19T21:06:46.211939+0000","last_became_peered":"2024-02-19T21:06:46.211939+0000","last_unstale":"2024-02-19T21:06:53.218374+0000","last_undegraded":"2024-02-19T21:06:53.218374+0000","last_fullsized":"2024-02-19T21:06:53.218374+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"157.14","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216112+0000","last_change":"2024-02-19T21:06:50.224105+0000","last_active":"2024-02-19T21:06:53.216112+0000","last_peered":"2024-02-19T21:06:53.216112+0000","last_clean":"2024-02-19T21:06:53.216112+0000","last_became_active":"2024-02-19T21:06:50.223148+0000","last_became_peered":"2024-02-19T21:06:50.223148+0000","last_unstale":"2024-02-19T21:06:53.216112+0000","last_undegraded":"2024-02-19T21:06:53.216112+0000","last_fullsized":"2024-02-19T21:06:53.216112+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"156.15","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.210621+0000","last_change":"2024-02-19T21:06:48.215699+0000","last_active":"2024-02-19T21:06:53.210621+0000","last_peered":"2024-02-19T21:06:53.210621+0000","last_clean":"2024-02-19T21:06:53.210621+0000","last_became_active":"2024-02-19T21:06:48.215395+0000","last_became_peered":"2024-02-19T21:06:48.215395+0000","last_unstale":"2024-02-19T21:06:53.210621+0000","last_undegraded":"2024-02-19T21:06:53.210621+0000","last_fullsized":"2024-02-19T21:06:53.210621+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"155.12","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.222848+0000","last_change":"2024-02-19T21:06:46.210037+0000","last_active":"2024-02-19T21:06:53.222848+0000","last_peered":"2024-02-19T21:06:53.222848+0000","last_clean":"2024-02-19T21:06:53.222848+0000","last_became_active":"2024-02-19T21:06:46.209733+0000","last_became_peered":"2024-02-19T21:06:46.209733+0000","last_unstale":"2024-02-19T21:06:53.222848+0000","last_undegraded":"2024-02-19T21:06:53.222848+0000","last_fullsized":"2024-02-19T21:06:53.222848+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"157.17","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.212510+0000","last_change":"2024-02-19T21:06:50.229078+0000","last_active":"2024-02-19T21:06:53.212510+0000","last_peered":"2024-02-19T21:06:53.212510+0000","last_clean":"2024-02-19T21:06:53.212510+0000","last_became_active":"2024-02-19T21:06:50.228782+0000","last_became_peered":"2024-02-19T21:06:50.228782+0000","last_unstale":"2024-02-19T21:06:53.212510+0000","last_undegraded":"2024-02-19T21:06:53.212510+0000","last_fullsized":"2024-02-19T21:06:53.212510+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"156.16","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215145+0000","last_change":"2024-02-19T21:06:48.214497+0000","last_active":"2024-02-19T21:06:53.215145+0000","last_peered":"2024-02-19T21:06:53.215145+0000","last_clean":"2024-02-19T21:06:53.215145+0000","last_became_active":"2024-02-19T21:06:48.214167+0000","last_became_peered":"2024-02-19T21:06:48.214167+0000","last_unstale":"2024-02-19T21:06:53.215145+0000","last_undegraded":"2024-02-19T21:06:53.215145+0000","last_fullsized":"2024-02-19T21:06:53.215145+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"155.11","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.204756+0000","last_change":"2024-02-19T21:06:46.203919+0000","last_active":"2024-02-19T21:06:53.204756+0000","last_peered":"2024-02-19T21:06:53.204756+0000","last_clean":"2024-02-19T21:06:53.204756+0000","last_became_active":"2024-02-19T21:06:46.203623+0000","last_became_peered":"2024-02-19T21:06:46.203623+0000","last_unstale":"2024-02-19T21:06:53.204756+0000","last_undegraded":"2024-02-19T21:06:53.204756+0000","last_fullsized":"2024-02-19T21:06:53.204756+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"157.16","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.214261+0000","last_change":"2024-02-19T21:06:50.224229+0000","last_active":"2024-02-19T21:06:53.214261+0000","last_peered":"2024-02-19T21:06:53.214261+0000","last_clean":"2024-02-19T21:06:53.214261+0000","last_became_active":"2024-02-19T21:06:50.223982+0000","last_became_peered":"2024-02-19T21:06:50.223982+0000","last_unstale":"2024-02-19T21:06:53.214261+0000","last_undegraded":"2024-02-19T21:06:53.214261+0000","last_fullsized":"2024-02-19T21:06:53.214261+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"156.17","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.226359+0000","last_change":"2024-02-19T21:06:48.213087+0000","last_active":"2024-02-19T21:06:53.226359+0000","last_peered":"2024-02-19T21:06:53.226359+0000","last_clean":"2024-02-19T21:06:53.226359+0000","last_became_active":"2024-02-19T21:06:48.212799+0000","last_became_peered":"2024-02-19T21:06:48.212799+0000","last_unstale":"2024-02-19T21:06:53.226359+0000","last_undegraded":"2024-02-19T21:06:53.226359+0000","last_fullsized":"2024-02-19T21:06:53.226359+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"155.10","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.218931+0000","last_change":"2024-02-19T21:06:46.208898+0000","last_active":"2024-02-19T21:06:53.218931+0000","last_peered":"2024-02-19T21:06:53.218931+0000","last_clean":"2024-02-19T21:06:53.218931+0000","last_became_active":"2024-02-19T21:06:46.208627+0000","last_became_peered":"2024-02-19T21:06:46.208627+0000","last_unstale":"2024-02-19T21:06:53.218931+0000","last_undegraded":"2024-02-19T21:06:53.218931+0000","last_fullsized":"2024-02-19T21:06:53.218931+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"157.11","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216843+0000","last_change":"2024-02-19T21:06:50.230495+0000","last_active":"2024-02-19T21:06:53.216843+0000","last_peered":"2024-02-19T21:06:53.216843+0000","last_clean":"2024-02-19T21:06:53.216843+0000","last_became_active":"2024-02-19T21:06:50.228973+0000","last_became_peered":"2024-02-19T21:06:50.228973+0000","last_unstale":"2024-02-19T21:06:53.216843+0000","last_undegraded":"2024-02-19T21:06:53.216843+0000","last_fullsized":"2024-02-19T21:06:53.216843+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"156.10","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.206199+0000","last_change":"2024-02-19T21:06:48.213095+0000","last_active":"2024-02-19T21:06:53.206199+0000","last_peered":"2024-02-19T21:06:53.206199+0000","last_clean":"2024-02-19T21:06:53.206199+0000","last_became_active":"2024-02-19T21:06:48.212753+0000","last_became_peered":"2024-02-19T21:06:48.212753+0000","last_unstale":"2024-02-19T21:06:53.206199+0000","last_undegraded":"2024-02-19T21:06:53.206199+0000","last_fullsized":"2024-02-19T21:06:53.206199+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"155.17","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.226966+0000","last_change":"2024-02-19T21:06:46.213408+0000","last_active":"2024-02-19T21:06:53.226966+0000","last_peered":"2024-02-19T21:06:53.226966+0000","last_clean":"2024-02-19T21:06:53.226966+0000","last_became_active":"2024-02-19T21:06:46.213083+0000","last_became_peered":"2024-02-19T21:06:46.213083+0000","last_unstale":"2024-02-19T21:06:53.226966+0000","last_undegraded":"2024-02-19T21:06:53.226966+0000","last_fullsized":"2024-02-19T21:06:53.226966+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"157.10","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.210919+0000","last_change":"2024-02-19T21:06:50.226232+0000","last_active":"2024-02-19T21:06:53.210919+0000","last_peered":"2024-02-19T21:06:53.210919+0000","last_clean":"2024-02-19T21:06:53.210919+0000","last_became_active":"2024-02-19T21:06:50.225801+0000","last_became_peered":"2024-02-19T21:06:50.225801+0000","last_unstale":"2024-02-19T21:06:53.210919+0000","last_undegraded":"2024-02-19T21:06:53.210919+0000","last_fullsized":"2024-02-19T21:06:53.210919+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"156.11","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.217082+0000","last_change":"2024-02-19T21:06:48.215470+0000","last_active":"2024-02-19T21:06:53.217082+0000","last_peered":"2024-02-19T21:06:53.217082+0000","last_clean":"2024-02-19T21:06:53.217082+0000","last_became_active":"2024-02-19T21:06:48.215117+0000","last_became_peered":"2024-02-19T21:06:48.215117+0000","last_unstale":"2024-02-19T21:06:53.217082+0000","last_undegraded":"2024-02-19T21:06:53.217082+0000","last_fullsized":"2024-02-19T21:06:53.217082+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"155.16","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.214961+0000","last_change":"2024-02-19T21:06:46.206525+0000","last_active":"2024-02-19T21:06:53.214961+0000","last_peered":"2024-02-19T21:06:53.214961+0000","last_clean":"2024-02-19T21:06:53.214961+0000","last_became_active":"2024-02-19T21:06:46.206092+0000","last_became_peered":"2024-02-19T21:06:46.206092+0000","last_unstale":"2024-02-19T21:06:53.214961+0000","last_undegraded":"2024-02-19T21:06:53.214961+0000","last_fullsized":"2024-02-19T21:06:53.214961+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"157.13","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.217191+0000","last_change":"2024-02-19T21:06:50.226426+0000","last_active":"2024-02-19T21:06:53.217191+0000","last_peered":"2024-02-19T21:06:53.217191+0000","last_clean":"2024-02-19T21:06:53.217191+0000","last_became_active":"2024-02-19T21:06:50.226119+0000","last_became_peered":"2024-02-19T21:06:50.226119+0000","last_unstale":"2024-02-19T21:06:53.217191+0000","last_undegraded":"2024-02-19T21:06:53.217191+0000","last_fullsized":"2024-02-19T21:06:53.217191+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"156.12","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.210954+0000","last_change":"2024-02-19T21:06:48.216517+0000","last_active":"2024-02-19T21:06:53.210954+0000","last_peered":"2024-02-19T21:06:53.210954+0000","last_clean":"2024-02-19T21:06:53.210954+0000","last_became_active":"2024-02-19T21:06:48.215975+0000","last_became_peered":"2024-02-19T21:06:48.215975+0000","last_unstale":"2024-02-19T21:06:53.210954+0000","last_undegraded":"2024-02-19T21:06:53.210954+0000","last_fullsized":"2024-02-19T21:06:53.210954+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"155.15","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224379+0000","last_change":"2024-02-19T21:06:46.209880+0000","last_active":"2024-02-19T21:06:53.224379+0000","last_peered":"2024-02-19T21:06:53.224379+0000","last_clean":"2024-02-19T21:06:53.224379+0000","last_became_active":"2024-02-19T21:06:46.209597+0000","last_became_peered":"2024-02-19T21:06:46.209597+0000","last_unstale":"2024-02-19T21:06:53.224379+0000","last_undegraded":"2024-02-19T21:06:53.224379+0000","last_fullsized":"2024-02-19T21:06:53.224379+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"157.12","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.209569+0000","last_change":"2024-02-19T21:06:50.227009+0000","last_active":"2024-02-19T21:06:53.209569+0000","last_peered":"2024-02-19T21:06:53.209569+0000","last_clean":"2024-02-19T21:06:53.209569+0000","last_became_active":"2024-02-19T21:06:50.226719+0000","last_became_peered":"2024-02-19T21:06:50.226719+0000","last_unstale":"2024-02-19T21:06:53.209569+0000","last_undegraded":"2024-02-19T21:06:53.209569+0000","last_fullsized":"2024-02-19T21:06:53.209569+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:49.191031+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.191031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"156.13","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.225555+0000","last_change":"2024-02-19T21:06:48.214932+0000","last_active":"2024-02-19T21:06:53.225555+0000","last_peered":"2024-02-19T21:06:53.225555+0000","last_clean":"2024-02-19T21:06:53.225555+0000","last_became_active":"2024-02-19T21:06:48.214622+0000","last_became_peered":"2024-02-19T21:06:48.214622+0000","last_unstale":"2024-02-19T21:06:53.225555+0000","last_undegraded":"2024-02-19T21:06:53.225555+0000","last_fullsized":"2024-02-19T21:06:53.225555+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:47.187547+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.187547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"155.14","version":"0'0","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216263+0000","last_change":"2024-02-19T21:06:46.214165+0000","last_active":"2024-02-19T21:06:53.216263+0000","last_peered":"2024-02-19T21:06:53.216263+0000","last_clean":"2024-02-19T21:06:53.216263+0000","last_became_active":"2024-02-19T21:06:46.213899+0000","last_became_peered":"2024-02-19T21:06:46.213899+0000","last_unstale":"2024-02-19T21:06:53.216263+0000","last_undegraded":"2024-02-19T21:06:53.216263+0000","last_fullsized":"2024-02-19T21:06:53.216263+0000","mapping_epoch":225,"log_start":"0'0","ondisk_log_start":"0'0","created":225,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:45.183302+0000","last_clean_scrub_stamp":"2024-02-19T21:06:45.183302+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.2","version":"210'73","reported_seq":226,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215343+0000","last_change":"2024-02-19T21:06:50.207416+0000","last_active":"2024-02-19T21:06:53.215343+0000","last_peered":"2024-02-19T21:06:53.215343+0000","last_clean":"2024-02-19T21:06:53.215343+0000","last_became_active":"2024-02-19T21:04:54.609091+0000","last_became_peered":"2024-02-19T21:04:54.609091+0000","last_unstale":"2024-02-19T21:06:53.215343+0000","last_undegraded":"2024-02-19T21:06:53.215343+0000","last_fullsized":"2024-02-19T21:06:53.215343+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.3","version":"230'68","reported_seq":206,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:53.227979+0000","last_change":"2024-02-19T21:06:52.301970+0000","last_active":"2024-02-19T21:06:53.227979+0000","last_peered":"2024-02-19T21:06:53.227979+0000","last_clean":"2024-02-19T21:06:53.227979+0000","last_became_active":"2024-02-19T21:04:54.641884+0000","last_became_peered":"2024-02-19T21:04:54.641884+0000","last_unstale":"2024-02-19T21:06:53.227979+0000","last_undegraded":"2024-02-19T21:06:53.227979+0000","last_fullsized":"2024-02-19T21:06:53.227979+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"}]},{"pgid":"61.0","version":"196'34","reported_seq":166,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.222931+0000","last_change":"2024-02-19T21:06:52.302761+0000","last_active":"2024-02-19T21:06:53.222931+0000","last_peered":"2024-02-19T21:06:53.222931+0000","last_clean":"2024-02-19T21:06:53.222931+0000","last_became_active":"2024-02-19T21:06:40.245157+0000","last_became_peered":"2024-02-19T21:06:40.245157+0000","last_unstale":"2024-02-19T21:06:53.222931+0000","last_undegraded":"2024-02-19T21:06:53.222931+0000","last_fullsized":"2024-02-19T21:06:53.222931+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.1","version":"220'68","reported_seq":215,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:53.223226+0000","last_change":"2024-02-19T21:06:52.302166+0000","last_active":"2024-02-19T21:06:53.223226+0000","last_peered":"2024-02-19T21:06:53.223226+0000","last_clean":"2024-02-19T21:06:53.223226+0000","last_became_active":"2024-02-19T21:04:54.654642+0000","last_became_peered":"2024-02-19T21:04:54.654642+0000","last_unstale":"2024-02-19T21:06:53.223226+0000","last_undegraded":"2024-02-19T21:06:53.223226+0000","last_fullsized":"2024-02-19T21:06:53.223226+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"61.6","version":"202'34","reported_seq":185,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215457+0000","last_change":"2024-02-19T21:06:50.215874+0000","last_active":"2024-02-19T21:06:53.215457+0000","last_peered":"2024-02-19T21:06:53.215457+0000","last_clean":"2024-02-19T21:06:53.215457+0000","last_became_active":"2024-02-19T21:04:54.658542+0000","last_became_peered":"2024-02-19T21:04:54.658542+0000","last_unstale":"2024-02-19T21:06:53.215457+0000","last_undegraded":"2024-02-19T21:06:53.215457+0000","last_fullsized":"2024-02-19T21:06:53.215457+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.7","version":"220'18","reported_seq":177,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.213107+0000","last_change":"2024-02-19T21:06:50.228504+0000","last_active":"2024-02-19T21:06:53.213107+0000","last_peered":"2024-02-19T21:06:53.213107+0000","last_clean":"2024-02-19T21:06:53.213107+0000","last_became_active":"2024-02-19T21:04:54.598061+0000","last_became_peered":"2024-02-19T21:04:54.598061+0000","last_unstale":"2024-02-19T21:06:53.213107+0000","last_undegraded":"2024-02-19T21:06:53.213107+0000","last_fullsized":"2024-02-19T21:06:53.213107+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.4","version":"206'68","reported_seq":214,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.220535+0000","last_change":"2024-02-19T21:06:50.215570+0000","last_active":"2024-02-19T21:06:53.220535+0000","last_peered":"2024-02-19T21:06:53.220535+0000","last_clean":"2024-02-19T21:06:53.220535+0000","last_became_active":"2024-02-19T21:04:54.639335+0000","last_became_peered":"2024-02-19T21:04:54.639335+0000","last_unstale":"2024-02-19T21:06:53.220535+0000","last_undegraded":"2024-02-19T21:06:53.220535+0000","last_fullsized":"2024-02-19T21:06:53.220535+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.5","version":"234'113","reported_seq":241,"reported_epoch":234,"state":"active+clean+snaptrim","last_fresh":"2024-02-19T21:06:54.307306+0000","last_change":"2024-02-19T21:06:52.304262+0000","last_active":"2024-02-19T21:06:54.307306+0000","last_peered":"2024-02-19T21:06:54.307306+0000","last_clean":"2024-02-19T21:06:54.307306+0000","last_became_active":"2024-02-19T21:04:54.646842+0000","last_became_peered":"2024-02-19T21:04:54.646842+0000","last_unstale":"2024-02-19T21:06:54.307306+0000","last_undegraded":"2024-02-19T21:06:54.307306+0000","last_fullsized":"2024-02-19T21:06:54.307306+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":113,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":3,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"}]},{"pgid":"61.a","version":"204'19","reported_seq":179,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.210864+0000","last_change":"2024-02-19T21:06:50.205603+0000","last_active":"2024-02-19T21:06:53.210864+0000","last_peered":"2024-02-19T21:06:53.210864+0000","last_clean":"2024-02-19T21:06:53.210864+0000","last_became_active":"2024-02-19T21:04:54.611831+0000","last_became_peered":"2024-02-19T21:04:54.611831+0000","last_unstale":"2024-02-19T21:06:53.210864+0000","last_undegraded":"2024-02-19T21:06:53.210864+0000","last_fullsized":"2024-02-19T21:06:53.210864+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.b","version":"234'85","reported_seq":216,"reported_epoch":234,"state":"active+clean+snaptrim","last_fresh":"2024-02-19T21:06:54.301425+0000","last_change":"2024-02-19T21:06:50.297975+0000","last_active":"2024-02-19T21:06:54.301425+0000","last_peered":"2024-02-19T21:06:54.301425+0000","last_clean":"2024-02-19T21:06:54.301425+0000","last_became_active":"2024-02-19T21:04:54.640024+0000","last_became_peered":"2024-02-19T21:04:54.640024+0000","last_unstale":"2024-02-19T21:06:54.301425+0000","last_undegraded":"2024-02-19T21:06:54.301425+0000","last_fullsized":"2024-02-19T21:06:54.301425+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"1"}]},{"pgid":"61.8","version":"212'35","reported_seq":182,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.221531+0000","last_change":"2024-02-19T21:06:50.212051+0000","last_active":"2024-02-19T21:06:53.221531+0000","last_peered":"2024-02-19T21:06:53.221531+0000","last_clean":"2024-02-19T21:06:53.221531+0000","last_became_active":"2024-02-19T21:04:54.650062+0000","last_became_peered":"2024-02-19T21:04:54.650062+0000","last_unstale":"2024-02-19T21:06:53.221531+0000","last_undegraded":"2024-02-19T21:06:53.221531+0000","last_fullsized":"2024-02-19T21:06:53.221531+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.9","version":"226'68","reported_seq":215,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:53.223089+0000","last_change":"2024-02-19T21:06:52.302582+0000","last_active":"2024-02-19T21:06:53.223089+0000","last_peered":"2024-02-19T21:06:53.223089+0000","last_clean":"2024-02-19T21:06:53.223089+0000","last_became_active":"2024-02-19T21:04:54.671646+0000","last_became_peered":"2024-02-19T21:04:54.671646+0000","last_unstale":"2024-02-19T21:06:53.223089+0000","last_undegraded":"2024-02-19T21:06:53.223089+0000","last_fullsized":"2024-02-19T21:06:53.223089+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"61.e","version":"189'34","reported_seq":191,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216815+0000","last_change":"2024-02-19T21:06:50.225144+0000","last_active":"2024-02-19T21:06:53.216815+0000","last_peered":"2024-02-19T21:06:53.216815+0000","last_clean":"2024-02-19T21:06:53.216815+0000","last_became_active":"2024-02-19T21:04:54.603276+0000","last_became_peered":"2024-02-19T21:04:54.603276+0000","last_unstale":"2024-02-19T21:06:53.216815+0000","last_undegraded":"2024-02-19T21:06:53.216815+0000","last_fullsized":"2024-02-19T21:06:53.216815+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.f","version":"220'34","reported_seq":189,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.229753+0000","last_change":"2024-02-19T21:06:50.218315+0000","last_active":"2024-02-19T21:06:53.229753+0000","last_peered":"2024-02-19T21:06:53.229753+0000","last_clean":"2024-02-19T21:06:53.229753+0000","last_became_active":"2024-02-19T21:04:54.610312+0000","last_became_peered":"2024-02-19T21:04:54.610312+0000","last_unstale":"2024-02-19T21:06:53.229753+0000","last_undegraded":"2024-02-19T21:06:53.229753+0000","last_fullsized":"2024-02-19T21:06:53.229753+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.c","version":"224'68","reported_seq":212,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.221390+0000","last_change":"2024-02-19T21:06:50.215781+0000","last_active":"2024-02-19T21:06:53.221390+0000","last_peered":"2024-02-19T21:06:53.221390+0000","last_clean":"2024-02-19T21:06:53.221390+0000","last_became_active":"2024-02-19T21:04:54.634803+0000","last_became_peered":"2024-02-19T21:04:54.634803+0000","last_unstale":"2024-02-19T21:06:53.221390+0000","last_undegraded":"2024-02-19T21:06:53.221390+0000","last_fullsized":"2024-02-19T21:06:53.221390+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.d","version":"214'34","reported_seq":187,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.221107+0000","last_change":"2024-02-19T21:06:50.217568+0000","last_active":"2024-02-19T21:06:53.221107+0000","last_peered":"2024-02-19T21:06:53.221107+0000","last_clean":"2024-02-19T21:06:53.221107+0000","last_became_active":"2024-02-19T21:04:54.644749+0000","last_became_peered":"2024-02-19T21:04:54.644749+0000","last_unstale":"2024-02-19T21:06:53.221107+0000","last_undegraded":"2024-02-19T21:06:53.221107+0000","last_fullsized":"2024-02-19T21:06:53.221107+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.12","version":"230'44","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.211953+0000","last_change":"2024-02-19T21:06:52.228085+0000","last_active":"2024-02-19T21:06:53.211953+0000","last_peered":"2024-02-19T21:06:53.211953+0000","last_clean":"2024-02-19T21:06:53.211953+0000","last_became_active":"2024-02-19T21:04:54.670998+0000","last_became_peered":"2024-02-19T21:04:54.670998+0000","last_unstale":"2024-02-19T21:06:53.211953+0000","last_undegraded":"2024-02-19T21:06:53.211953+0000","last_fullsized":"2024-02-19T21:06:53.211953+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.13","version":"216'68","reported_seq":216,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224293+0000","last_change":"2024-02-19T21:06:50.213218+0000","last_active":"2024-02-19T21:06:53.224293+0000","last_peered":"2024-02-19T21:06:53.224293+0000","last_clean":"2024-02-19T21:06:53.224293+0000","last_became_active":"2024-02-19T21:04:54.608802+0000","last_became_peered":"2024-02-19T21:04:54.608802+0000","last_unstale":"2024-02-19T21:06:53.224293+0000","last_undegraded":"2024-02-19T21:06:53.224293+0000","last_fullsized":"2024-02-19T21:06:53.224293+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.10","version":"230'45","reported_seq":200,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.219183+0000","last_change":"2024-02-19T21:06:52.229556+0000","last_active":"2024-02-19T21:06:53.219183+0000","last_peered":"2024-02-19T21:06:53.219183+0000","last_clean":"2024-02-19T21:06:53.219183+0000","last_became_active":"2024-02-19T21:04:54.626393+0000","last_became_peered":"2024-02-19T21:04:54.626393+0000","last_unstale":"2024-02-19T21:06:53.219183+0000","last_undegraded":"2024-02-19T21:06:53.219183+0000","last_fullsized":"2024-02-19T21:06:53.219183+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.11","version":"199'34","reported_seq":190,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.218885+0000","last_change":"2024-02-19T21:06:50.213281+0000","last_active":"2024-02-19T21:06:53.218885+0000","last_peered":"2024-02-19T21:06:53.218885+0000","last_clean":"2024-02-19T21:06:53.218885+0000","last_became_active":"2024-02-19T21:04:54.659096+0000","last_became_peered":"2024-02-19T21:04:54.659096+0000","last_unstale":"2024-02-19T21:06:53.218885+0000","last_undegraded":"2024-02-19T21:06:53.218885+0000","last_fullsized":"2024-02-19T21:06:53.218885+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.29","version":"0'0","reported_seq":107,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.220474+0000","last_change":"2024-02-19T21:05:44.986979+0000","last_active":"2024-02-19T21:06:53.220474+0000","last_peered":"2024-02-19T21:06:53.220474+0000","last_clean":"2024-02-19T21:06:53.220474+0000","last_became_active":"2024-02-19T21:05:22.208034+0000","last_became_peered":"2024-02-19T21:05:22.208034+0000","last_unstale":"2024-02-19T21:06:53.220474+0000","last_undegraded":"2024-02-19T21:06:53.220474+0000","last_fullsized":"2024-02-19T21:06:53.220474+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.986874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.986874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.16","version":"192'22","reported_seq":179,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.218904+0000","last_change":"2024-02-19T21:06:50.215903+0000","last_active":"2024-02-19T21:06:53.218904+0000","last_peered":"2024-02-19T21:06:53.218904+0000","last_clean":"2024-02-19T21:06:53.218904+0000","last_became_active":"2024-02-19T21:04:54.608059+0000","last_became_peered":"2024-02-19T21:04:54.608059+0000","last_unstale":"2024-02-19T21:06:53.218904+0000","last_undegraded":"2024-02-19T21:06:53.218904+0000","last_fullsized":"2024-02-19T21:06:53.218904+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.28","version":"0'0","reported_seq":107,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215168+0000","last_change":"2024-02-19T21:06:29.755093+0000","last_active":"2024-02-19T21:06:53.215168+0000","last_peered":"2024-02-19T21:06:53.215168+0000","last_clean":"2024-02-19T21:06:53.215168+0000","last_became_active":"2024-02-19T21:05:22.220291+0000","last_became_peered":"2024-02-19T21:05:22.220291+0000","last_unstale":"2024-02-19T21:06:53.215168+0000","last_undegraded":"2024-02-19T21:06:53.215168+0000","last_fullsized":"2024-02-19T21:06:53.215168+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:29.755031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:29.755031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.17","version":"220'19","reported_seq":185,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215807+0000","last_change":"2024-02-19T21:06:50.216652+0000","last_active":"2024-02-19T21:06:53.215807+0000","last_peered":"2024-02-19T21:06:53.215807+0000","last_clean":"2024-02-19T21:06:53.215807+0000","last_became_active":"2024-02-19T21:06:40.228088+0000","last_became_peered":"2024-02-19T21:06:40.228088+0000","last_unstale":"2024-02-19T21:06:53.215807+0000","last_undegraded":"2024-02-19T21:06:53.215807+0000","last_fullsized":"2024-02-19T21:06:53.215807+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.14","version":"200'51","reported_seq":184,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.220426+0000","last_change":"2024-02-19T21:06:50.215135+0000","last_active":"2024-02-19T21:06:53.220426+0000","last_peered":"2024-02-19T21:06:53.220426+0000","last_clean":"2024-02-19T21:06:53.220426+0000","last_became_active":"2024-02-19T21:06:40.227840+0000","last_became_peered":"2024-02-19T21:06:40.227840+0000","last_unstale":"2024-02-19T21:06:53.220426+0000","last_undegraded":"2024-02-19T21:06:53.220426+0000","last_fullsized":"2024-02-19T21:06:53.220426+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.15","version":"218'51","reported_seq":202,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.225908+0000","last_change":"2024-02-19T21:06:50.216161+0000","last_active":"2024-02-19T21:06:53.225908+0000","last_peered":"2024-02-19T21:06:53.225908+0000","last_clean":"2024-02-19T21:06:53.225908+0000","last_became_active":"2024-02-19T21:04:54.626719+0000","last_became_peered":"2024-02-19T21:04:54.626719+0000","last_unstale":"2024-02-19T21:06:53.225908+0000","last_undegraded":"2024-02-19T21:06:53.225908+0000","last_fullsized":"2024-02-19T21:06:53.225908+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.25","version":"0'0","reported_seq":210,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.209866+0000","last_change":"2024-02-19T21:06:11.639779+0000","last_active":"2024-02-19T21:06:53.209866+0000","last_peered":"2024-02-19T21:06:53.209866+0000","last_clean":"2024-02-19T21:06:53.209866+0000","last_became_active":"2024-02-19T21:05:22.209781+0000","last_became_peered":"2024-02-19T21:05:22.209781+0000","last_unstale":"2024-02-19T21:06:53.209866+0000","last_undegraded":"2024-02-19T21:06:53.209866+0000","last_fullsized":"2024-02-19T21:06:53.209866+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:11.639684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:11.639684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"61.1a","version":"224'85","reported_seq":234,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.222704+0000","last_change":"2024-02-19T21:06:50.214607+0000","last_active":"2024-02-19T21:06:53.222704+0000","last_peered":"2024-02-19T21:06:53.222704+0000","last_clean":"2024-02-19T21:06:53.222704+0000","last_became_active":"2024-02-19T21:04:54.634861+0000","last_became_peered":"2024-02-19T21:04:54.634861+0000","last_unstale":"2024-02-19T21:06:53.222704+0000","last_undegraded":"2024-02-19T21:06:53.222704+0000","last_fullsized":"2024-02-19T21:06:53.222704+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.24","version":"0'0","reported_seq":107,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.212768+0000","last_change":"2024-02-19T21:06:04.866459+0000","last_active":"2024-02-19T21:06:53.212768+0000","last_peered":"2024-02-19T21:06:53.212768+0000","last_clean":"2024-02-19T21:06:53.212768+0000","last_became_active":"2024-02-19T21:05:22.208647+0000","last_became_peered":"2024-02-19T21:05:22.208647+0000","last_unstale":"2024-02-19T21:06:53.212768+0000","last_undegraded":"2024-02-19T21:06:53.212768+0000","last_fullsized":"2024-02-19T21:06:53.212768+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:04.866394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:06:04.866394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1b","version":"208'51","reported_seq":198,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.212964+0000","last_change":"2024-02-19T21:06:50.199215+0000","last_active":"2024-02-19T21:06:53.212964+0000","last_peered":"2024-02-19T21:06:53.212964+0000","last_clean":"2024-02-19T21:06:53.212964+0000","last_became_active":"2024-02-19T21:04:54.658733+0000","last_became_peered":"2024-02-19T21:04:54.658733+0000","last_unstale":"2024-02-19T21:06:53.212964+0000","last_undegraded":"2024-02-19T21:06:53.212964+0000","last_fullsized":"2024-02-19T21:06:53.212964+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":209,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.229582+0000","last_change":"2024-02-19T21:06:25.516601+0000","last_active":"2024-02-19T21:06:53.229582+0000","last_peered":"2024-02-19T21:06:53.229582+0000","last_clean":"2024-02-19T21:06:53.229582+0000","last_became_active":"2024-02-19T21:05:22.217505+0000","last_became_peered":"2024-02-19T21:05:22.217505+0000","last_unstale":"2024-02-19T21:06:53.229582+0000","last_undegraded":"2024-02-19T21:06:53.229582+0000","last_fullsized":"2024-02-19T21:06:53.229582+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:25.516527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:25.516527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.18","version":"214'102","reported_seq":242,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215256+0000","last_change":"2024-02-19T21:06:50.207774+0000","last_active":"2024-02-19T21:06:53.215256+0000","last_peered":"2024-02-19T21:06:53.215256+0000","last_clean":"2024-02-19T21:06:53.215256+0000","last_became_active":"2024-02-19T21:04:54.610608+0000","last_became_peered":"2024-02-19T21:04:54.610608+0000","last_unstale":"2024-02-19T21:06:53.215256+0000","last_undegraded":"2024-02-19T21:06:53.215256+0000","last_fullsized":"2024-02-19T21:06:53.215256+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.26","version":"0'0","reported_seq":107,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.225948+0000","last_change":"2024-02-19T21:06:17.475027+0000","last_active":"2024-02-19T21:06:53.225948+0000","last_peered":"2024-02-19T21:06:53.225948+0000","last_clean":"2024-02-19T21:06:53.225948+0000","last_became_active":"2024-02-19T21:05:22.216623+0000","last_became_peered":"2024-02-19T21:05:22.216623+0000","last_unstale":"2024-02-19T21:06:53.225948+0000","last_undegraded":"2024-02-19T21:06:53.225948+0000","last_fullsized":"2024-02-19T21:06:53.225948+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:17.474951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:17.474951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.19","version":"218'68","reported_seq":219,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.209416+0000","last_change":"2024-02-19T21:06:50.210268+0000","last_active":"2024-02-19T21:06:53.209416+0000","last_peered":"2024-02-19T21:06:53.209416+0000","last_clean":"2024-02-19T21:06:53.209416+0000","last_became_active":"2024-02-19T21:04:54.608702+0000","last_became_peered":"2024-02-19T21:04:54.608702+0000","last_unstale":"2024-02-19T21:06:53.209416+0000","last_undegraded":"2024-02-19T21:06:53.209416+0000","last_fullsized":"2024-02-19T21:06:53.209416+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.21","version":"0'0","reported_seq":222,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.218791+0000","last_change":"2024-02-19T21:05:46.049986+0000","last_active":"2024-02-19T21:06:53.218791+0000","last_peered":"2024-02-19T21:06:53.218791+0000","last_clean":"2024-02-19T21:06:53.218791+0000","last_became_active":"2024-02-19T21:05:22.216738+0000","last_became_peered":"2024-02-19T21:05:22.216738+0000","last_unstale":"2024-02-19T21:06:53.218791+0000","last_undegraded":"2024-02-19T21:06:53.218791+0000","last_fullsized":"2024-02-19T21:06:53.218791+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:46.049878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:46.049878+0000","last_clean_scrub_stamp":"2024-02-19T21:05:46.049878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.1e","version":"230'45","reported_seq":203,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.226140+0000","last_change":"2024-02-19T21:06:52.226621+0000","last_active":"2024-02-19T21:06:53.226140+0000","last_peered":"2024-02-19T21:06:53.226140+0000","last_clean":"2024-02-19T21:06:53.226140+0000","last_became_active":"2024-02-19T21:04:54.629433+0000","last_became_peered":"2024-02-19T21:04:54.629433+0000","last_unstale":"2024-02-19T21:06:53.226140+0000","last_undegraded":"2024-02-19T21:06:53.226140+0000","last_fullsized":"2024-02-19T21:06:53.226140+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.20","version":"58'1","reported_seq":229,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.219074+0000","last_change":"2024-02-19T21:06:18.465146+0000","last_active":"2024-02-19T21:06:53.219074+0000","last_peered":"2024-02-19T21:06:53.219074+0000","last_clean":"2024-02-19T21:06:53.219074+0000","last_became_active":"2024-02-19T21:05:22.223725+0000","last_became_peered":"2024-02-19T21:05:22.223725+0000","last_unstale":"2024-02-19T21:06:53.219074+0000","last_undegraded":"2024-02-19T21:06:53.219074+0000","last_fullsized":"2024-02-19T21:06:53.219074+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:06:18.465061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.465061+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1f","version":"230'39","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216162+0000","last_change":"2024-02-19T21:06:52.224079+0000","last_active":"2024-02-19T21:06:53.216162+0000","last_peered":"2024-02-19T21:06:53.216162+0000","last_clean":"2024-02-19T21:06:53.216162+0000","last_became_active":"2024-02-19T21:04:54.671969+0000","last_became_peered":"2024-02-19T21:04:54.671969+0000","last_unstale":"2024-02-19T21:06:53.216162+0000","last_undegraded":"2024-02-19T21:06:53.216162+0000","last_fullsized":"2024-02-19T21:06:53.216162+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.23","version":"0'0","reported_seq":109,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.212870+0000","last_change":"2024-02-19T21:06:25.686571+0000","last_active":"2024-02-19T21:06:53.212870+0000","last_peered":"2024-02-19T21:06:53.212870+0000","last_clean":"2024-02-19T21:06:53.212870+0000","last_became_active":"2024-02-19T21:05:22.218880+0000","last_became_peered":"2024-02-19T21:05:22.218880+0000","last_unstale":"2024-02-19T21:06:53.212870+0000","last_undegraded":"2024-02-19T21:06:53.212870+0000","last_fullsized":"2024-02-19T21:06:53.212870+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:25.686510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:25.686510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.1c","version":"228'136","reported_seq":275,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.220766+0000","last_change":"2024-02-19T21:06:50.271910+0000","last_active":"2024-02-19T21:06:53.220766+0000","last_peered":"2024-02-19T21:06:53.220766+0000","last_clean":"2024-02-19T21:06:53.220766+0000","last_became_active":"2024-02-19T21:04:54.605898+0000","last_became_peered":"2024-02-19T21:04:54.605898+0000","last_unstale":"2024-02-19T21:06:53.220766+0000","last_undegraded":"2024-02-19T21:06:53.220766+0000","last_fullsized":"2024-02-19T21:06:53.220766+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":136,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":88,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":40,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.22","version":"0'0","reported_seq":209,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.212557+0000","last_change":"2024-02-19T21:05:43.958644+0000","last_active":"2024-02-19T21:06:53.212557+0000","last_peered":"2024-02-19T21:06:53.212557+0000","last_clean":"2024-02-19T21:06:53.212557+0000","last_became_active":"2024-02-19T21:05:22.196948+0000","last_became_peered":"2024-02-19T21:05:22.196948+0000","last_unstale":"2024-02-19T21:06:53.212557+0000","last_undegraded":"2024-02-19T21:06:53.212557+0000","last_fullsized":"2024-02-19T21:06:53.212557+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:43.958548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:43.958548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1d","version":"222'34","reported_seq":186,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2024-02-19T21:06:53.225628+0000","last_change":"2024-02-19T21:06:52.302387+0000","last_active":"2024-02-19T21:06:53.225628+0000","last_peered":"2024-02-19T21:06:53.225628+0000","last_clean":"2024-02-19T21:06:53.225628+0000","last_became_active":"2024-02-19T21:04:54.655665+0000","last_became_peered":"2024-02-19T21:04:54.655665+0000","last_unstale":"2024-02-19T21:06:53.225628+0000","last_undegraded":"2024-02-19T21:06:53.225628+0000","last_fullsized":"2024-02-19T21:06:53.225628+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6","length":"2"},{"start":"c","length":"1"}]},{"pgid":"8.4","version":"100'2","reported_seq":162,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.221585+0000","last_change":"2024-02-19T21:06:28.219879+0000","last_active":"2024-02-19T21:06:53.221585+0000","last_peered":"2024-02-19T21:06:53.221585+0000","last_clean":"2024-02-19T21:06:53.221585+0000","last_became_active":"2024-02-19T21:06:28.219512+0000","last_became_peered":"2024-02-19T21:06:28.219512+0000","last_unstale":"2024-02-19T21:06:53.221585+0000","last_undegraded":"2024-02-19T21:06:53.221585+0000","last_fullsized":"2024-02-19T21:06:53.221585+0000","mapping_epoch":98,"log_start":"100'2","ondisk_log_start":"100'2","created":98,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"97'9","reported_seq":272,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-19T21:06:54.207181+0000","last_change":"2024-02-19T21:06:52.658179+0000","last_active":"2024-02-19T21:06:54.207181+0000","last_peered":"2024-02-19T21:06:54.207181+0000","last_clean":"2024-02-19T21:06:54.207181+0000","last_became_active":"2024-02-19T21:01:20.288720+0000","last_became_peered":"2024-02-19T21:01:20.288720+0000","last_unstale":"2024-02-19T21:06:54.207181+0000","last_undegraded":"2024-02-19T21:06:54.207181+0000","last_fullsized":"2024-02-19T21:06:54.207181+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-02-19T21:06:52.658104+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:06:52.658104+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":221,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.221209+0000","last_change":"2024-02-19T21:04:56.123170+0000","last_active":"2024-02-19T21:06:53.221209+0000","last_peered":"2024-02-19T21:06:53.221209+0000","last_clean":"2024-02-19T21:06:53.221209+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:06:53.221209+0000","last_undegraded":"2024-02-19T21:06:53.221209+0000","last_fullsized":"2024-02-19T21:06:53.221209+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:04:56.123096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.123096+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":200,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215588+0000","last_change":"2024-02-19T21:06:16.105542+0000","last_active":"2024-02-19T21:06:53.215588+0000","last_peered":"2024-02-19T21:06:53.215588+0000","last_clean":"2024-02-19T21:06:53.215588+0000","last_became_active":"2024-02-19T21:01:16.271911+0000","last_became_peered":"2024-02-19T21:01:16.271911+0000","last_unstale":"2024-02-19T21:06:53.215588+0000","last_undegraded":"2024-02-19T21:06:53.215588+0000","last_fullsized":"2024-02-19T21:06:53.215588+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.105470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":189,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215244+0000","last_change":"2024-02-19T21:05:43.947943+0000","last_active":"2024-02-19T21:06:53.215244+0000","last_peered":"2024-02-19T21:06:53.215244+0000","last_clean":"2024-02-19T21:06:53.215244+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:06:53.215244+0000","last_undegraded":"2024-02-19T21:06:53.215244+0000","last_fullsized":"2024-02-19T21:06:53.215244+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:43.947829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:43.947829+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":148,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.221301+0000","last_change":"2024-02-19T21:06:46.312127+0000","last_active":"2024-02-19T21:06:53.221301+0000","last_peered":"2024-02-19T21:06:53.221301+0000","last_clean":"2024-02-19T21:06:53.221301+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:06:53.221301+0000","last_undegraded":"2024-02-19T21:06:53.221301+0000","last_fullsized":"2024-02-19T21:06:53.221301+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:46.312025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.312025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"168'7","reported_seq":185,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.227453+0000","last_change":"2024-02-19T21:06:48.203814+0000","last_active":"2024-02-19T21:06:53.227453+0000","last_peered":"2024-02-19T21:06:53.227453+0000","last_clean":"2024-02-19T21:06:53.227453+0000","last_became_active":"2024-02-19T21:06:48.203366+0000","last_became_peered":"2024-02-19T21:06:48.203366+0000","last_unstale":"2024-02-19T21:06:53.227453+0000","last_undegraded":"2024-02-19T21:06:53.227453+0000","last_fullsized":"2024-02-19T21:06:53.227453+0000","mapping_epoch":98,"log_start":"168'7","ondisk_log_start":"168'7","created":98,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'7","last_scrub_stamp":"2024-02-19T21:06:17.703187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:17.703187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":54,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.228621+0000","last_change":"2024-02-19T21:06:40.240963+0000","last_active":"2024-02-19T21:06:53.228621+0000","last_peered":"2024-02-19T21:06:53.228621+0000","last_clean":"2024-02-19T21:06:53.228621+0000","last_became_active":"2024-02-19T21:06:40.240602+0000","last_became_peered":"2024-02-19T21:06:40.240602+0000","last_unstale":"2024-02-19T21:06:53.228621+0000","last_undegraded":"2024-02-19T21:06:53.228621+0000","last_fullsized":"2024-02-19T21:06:53.228621+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:56.262927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.262927+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":209,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.226016+0000","last_change":"2024-02-19T21:05:10.764055+0000","last_active":"2024-02-19T21:06:53.226016+0000","last_peered":"2024-02-19T21:06:53.226016+0000","last_clean":"2024-02-19T21:06:53.226016+0000","last_became_active":"2024-02-19T21:04:12.926645+0000","last_became_peered":"2024-02-19T21:04:12.926645+0000","last_unstale":"2024-02-19T21:06:53.226016+0000","last_undegraded":"2024-02-19T21:06:53.226016+0000","last_fullsized":"2024-02-19T21:06:53.226016+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:10.763889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:10.763889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.212099+0000","last_change":"2024-02-19T21:06:27.303370+0000","last_active":"2024-02-19T21:06:53.212099+0000","last_peered":"2024-02-19T21:06:53.212099+0000","last_clean":"2024-02-19T21:06:53.212099+0000","last_became_active":"2024-02-19T21:01:20.290566+0000","last_became_peered":"2024-02-19T21:01:20.290566+0000","last_unstale":"2024-02-19T21:06:53.212099+0000","last_undegraded":"2024-02-19T21:06:53.212099+0000","last_fullsized":"2024-02-19T21:06:53.212099+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:27.303307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:27.303307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":26,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.217505+0000","last_change":"2024-02-19T21:06:40.228164+0000","last_active":"2024-02-19T21:06:53.217505+0000","last_peered":"2024-02-19T21:06:53.217505+0000","last_clean":"2024-02-19T21:06:53.217505+0000","last_became_active":"2024-02-19T21:06:40.227822+0000","last_became_peered":"2024-02-19T21:06:40.227822+0000","last_unstale":"2024-02-19T21:06:53.217505+0000","last_undegraded":"2024-02-19T21:06:53.217505+0000","last_fullsized":"2024-02-19T21:06:53.217505+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:05.822641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:05.822641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.217466+0000","last_change":"2024-02-19T21:04:39.765776+0000","last_active":"2024-02-19T21:06:53.217466+0000","last_peered":"2024-02-19T21:06:53.217466+0000","last_clean":"2024-02-19T21:06:53.217466+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:06:53.217466+0000","last_undegraded":"2024-02-19T21:06:53.217466+0000","last_fullsized":"2024-02-19T21:06:53.217466+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"168'11","reported_seq":136,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216938+0000","last_change":"2024-02-19T21:06:52.688430+0000","last_active":"2024-02-19T21:06:53.216938+0000","last_peered":"2024-02-19T21:06:53.216938+0000","last_clean":"2024-02-19T21:06:53.216938+0000","last_became_active":"2024-02-19T21:06:52.222861+0000","last_became_peered":"2024-02-19T21:06:52.222861+0000","last_unstale":"2024-02-19T21:06:53.216938+0000","last_undegraded":"2024-02-19T21:06:53.216938+0000","last_fullsized":"2024-02-19T21:06:53.216938+0000","mapping_epoch":219,"log_start":"168'11","ondisk_log_start":"168'11","created":98,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'11","last_scrub_stamp":"2024-02-19T21:06:52.688358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:52.688358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":229,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215915+0000","last_change":"2024-02-19T21:06:06.909964+0000","last_active":"2024-02-19T21:06:53.215915+0000","last_peered":"2024-02-19T21:06:53.215915+0000","last_clean":"2024-02-19T21:06:53.215915+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:06:53.215915+0000","last_undegraded":"2024-02-19T21:06:53.215915+0000","last_fullsized":"2024-02-19T21:06:53.215915+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:06:06.909884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:06.909884+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":200,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.220940+0000","last_change":"2024-02-19T21:05:49.051796+0000","last_active":"2024-02-19T21:06:53.220940+0000","last_peered":"2024-02-19T21:06:53.220940+0000","last_clean":"2024-02-19T21:06:53.220940+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:06:53.220940+0000","last_undegraded":"2024-02-19T21:06:53.220940+0000","last_fullsized":"2024-02-19T21:06:53.220940+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:05:49.051732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":191,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.219133+0000","last_change":"2024-02-19T21:04:51.822914+0000","last_active":"2024-02-19T21:06:53.219133+0000","last_peered":"2024-02-19T21:06:53.219133+0000","last_clean":"2024-02-19T21:06:53.219133+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:06:53.219133+0000","last_undegraded":"2024-02-19T21:06:53.219133+0000","last_fullsized":"2024-02-19T21:06:53.219133+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:51.822845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:51.822845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":188,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.219335+0000","last_change":"2024-02-19T21:05:05.187540+0000","last_active":"2024-02-19T21:06:53.219335+0000","last_peered":"2024-02-19T21:06:53.219335+0000","last_clean":"2024-02-19T21:06:53.219335+0000","last_became_active":"2024-02-19T21:01:22.291759+0000","last_became_peered":"2024-02-19T21:01:22.291759+0000","last_unstale":"2024-02-19T21:06:53.219335+0000","last_undegraded":"2024-02-19T21:06:53.219335+0000","last_fullsized":"2024-02-19T21:06:53.219335+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:05.187474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:05.187474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.226265+0000","last_change":"2024-02-19T21:04:39.620073+0000","last_active":"2024-02-19T21:06:53.226265+0000","last_peered":"2024-02-19T21:06:53.226265+0000","last_clean":"2024-02-19T21:06:53.226265+0000","last_became_active":"2024-02-19T21:04:39.618614+0000","last_became_peered":"2024-02-19T21:04:39.618614+0000","last_unstale":"2024-02-19T21:06:53.226265+0000","last_undegraded":"2024-02-19T21:06:53.226265+0000","last_fullsized":"2024-02-19T21:06:53.226265+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":253,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.210950+0000","last_change":"2024-02-19T21:06:06.630925+0000","last_active":"2024-02-19T21:06:53.210950+0000","last_peered":"2024-02-19T21:06:53.210950+0000","last_clean":"2024-02-19T21:06:53.210950+0000","last_became_active":"2024-02-19T21:01:30.923214+0000","last_became_peered":"2024-02-19T21:01:30.923214+0000","last_unstale":"2024-02-19T21:06:53.210950+0000","last_undegraded":"2024-02-19T21:06:53.210950+0000","last_fullsized":"2024-02-19T21:06:53.210950+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:06:06.630828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:06.630828+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":205,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224526+0000","last_change":"2024-02-19T21:06:50.694322+0000","last_active":"2024-02-19T21:06:53.224526+0000","last_peered":"2024-02-19T21:06:53.224526+0000","last_clean":"2024-02-19T21:06:53.224526+0000","last_became_active":"2024-02-19T21:01:16.274937+0000","last_became_peered":"2024-02-19T21:01:16.274937+0000","last_unstale":"2024-02-19T21:06:53.224526+0000","last_undegraded":"2024-02-19T21:06:53.224526+0000","last_fullsized":"2024-02-19T21:06:53.224526+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:06:50.694248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:50.694248+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224399+0000","last_change":"2024-02-19T21:05:39.947596+0000","last_active":"2024-02-19T21:06:53.224399+0000","last_peered":"2024-02-19T21:06:53.224399+0000","last_clean":"2024-02-19T21:06:53.224399+0000","last_became_active":"2024-02-19T21:01:20.290687+0000","last_became_peered":"2024-02-19T21:01:20.290687+0000","last_unstale":"2024-02-19T21:06:53.224399+0000","last_undegraded":"2024-02-19T21:06:53.224399+0000","last_fullsized":"2024-02-19T21:06:53.224399+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:39.947521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:39.947521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":194,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216369+0000","last_change":"2024-02-19T21:06:13.136600+0000","last_active":"2024-02-19T21:06:53.216369+0000","last_peered":"2024-02-19T21:06:53.216369+0000","last_clean":"2024-02-19T21:06:53.216369+0000","last_became_active":"2024-02-19T21:01:22.298941+0000","last_became_peered":"2024-02-19T21:01:22.298941+0000","last_unstale":"2024-02-19T21:06:53.216369+0000","last_undegraded":"2024-02-19T21:06:53.216369+0000","last_fullsized":"2024-02-19T21:06:53.216369+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:13.136532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:06:13.136532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"168'9","reported_seq":174,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224026+0000","last_change":"2024-02-19T21:06:40.246435+0000","last_active":"2024-02-19T21:06:53.224026+0000","last_peered":"2024-02-19T21:06:53.224026+0000","last_clean":"2024-02-19T21:06:53.224026+0000","last_became_active":"2024-02-19T21:06:40.245243+0000","last_became_peered":"2024-02-19T21:06:40.245243+0000","last_unstale":"2024-02-19T21:06:53.224026+0000","last_undegraded":"2024-02-19T21:06:53.224026+0000","last_fullsized":"2024-02-19T21:06:53.224026+0000","mapping_epoch":98,"log_start":"168'9","ondisk_log_start":"168'9","created":98,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'7","last_scrub_stamp":"2024-02-19T21:06:23.726481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:23.726481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":148,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.208117+0000","last_change":"2024-02-19T21:06:51.878186+0000","last_active":"2024-02-19T21:06:53.208117+0000","last_peered":"2024-02-19T21:06:53.208117+0000","last_clean":"2024-02-19T21:06:53.208117+0000","last_became_active":"2024-02-19T21:04:39.698684+0000","last_became_peered":"2024-02-19T21:04:39.698684+0000","last_unstale":"2024-02-19T21:06:53.208117+0000","last_undegraded":"2024-02-19T21:06:53.208117+0000","last_fullsized":"2024-02-19T21:06:53.208117+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.878060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.878060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"82'15","reported_seq":218,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.214857+0000","last_change":"2024-02-19T21:06:28.747060+0000","last_active":"2024-02-19T21:06:53.214857+0000","last_peered":"2024-02-19T21:06:53.214857+0000","last_clean":"2024-02-19T21:06:53.214857+0000","last_became_active":"2024-02-19T21:01:18.282805+0000","last_became_peered":"2024-02-19T21:01:18.282805+0000","last_unstale":"2024-02-19T21:06:53.214857+0000","last_undegraded":"2024-02-19T21:06:53.214857+0000","last_fullsized":"2024-02-19T21:06:53.214857+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:06:28.746972+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:06:28.746972+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":200,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224045+0000","last_change":"2024-02-19T21:06:14.688019+0000","last_active":"2024-02-19T21:06:53.224045+0000","last_peered":"2024-02-19T21:06:53.224045+0000","last_clean":"2024-02-19T21:06:53.224045+0000","last_became_active":"2024-02-19T21:01:16.274086+0000","last_became_peered":"2024-02-19T21:01:16.274086+0000","last_unstale":"2024-02-19T21:06:53.224045+0000","last_undegraded":"2024-02-19T21:06:53.224045+0000","last_fullsized":"2024-02-19T21:06:53.224045+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:14.687938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.687938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":272,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-19T21:06:54.191536+0000","last_change":"2024-02-19T21:06:12.177597+0000","last_active":"2024-02-19T21:06:54.191536+0000","last_peered":"2024-02-19T21:06:54.191536+0000","last_clean":"2024-02-19T21:06:54.191536+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:06:54.191536+0000","last_undegraded":"2024-02-19T21:06:54.191536+0000","last_fullsized":"2024-02-19T21:06:54.191536+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:12.177524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:12.177524+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":194,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.219450+0000","last_change":"2024-02-19T21:06:18.704518+0000","last_active":"2024-02-19T21:06:53.219450+0000","last_peered":"2024-02-19T21:06:53.219450+0000","last_clean":"2024-02-19T21:06:53.219450+0000","last_became_active":"2024-02-19T21:01:22.299993+0000","last_became_peered":"2024-02-19T21:01:22.299993+0000","last_unstale":"2024-02-19T21:06:53.219450+0000","last_undegraded":"2024-02-19T21:06:53.219450+0000","last_fullsized":"2024-02-19T21:06:53.219450+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.704447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.704447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"168'7","reported_seq":169,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215513+0000","last_change":"2024-02-19T21:06:44.222374+0000","last_active":"2024-02-19T21:06:53.215513+0000","last_peered":"2024-02-19T21:06:53.215513+0000","last_clean":"2024-02-19T21:06:53.215513+0000","last_became_active":"2024-02-19T21:06:44.221999+0000","last_became_peered":"2024-02-19T21:06:44.221999+0000","last_unstale":"2024-02-19T21:06:53.215513+0000","last_undegraded":"2024-02-19T21:06:53.215513+0000","last_fullsized":"2024-02-19T21:06:53.215513+0000","mapping_epoch":98,"log_start":"168'7","ondisk_log_start":"168'7","created":98,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":26,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215427+0000","last_change":"2024-02-19T21:06:40.226578+0000","last_active":"2024-02-19T21:06:53.215427+0000","last_peered":"2024-02-19T21:06:53.215427+0000","last_clean":"2024-02-19T21:06:53.215427+0000","last_became_active":"2024-02-19T21:06:40.226290+0000","last_became_peered":"2024-02-19T21:06:40.226290+0000","last_unstale":"2024-02-19T21:06:53.215427+0000","last_undegraded":"2024-02-19T21:06:53.215427+0000","last_fullsized":"2024-02-19T21:06:53.215427+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"168'11","reported_seq":174,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.228167+0000","last_change":"2024-02-19T21:06:46.624774+0000","last_active":"2024-02-19T21:06:53.228167+0000","last_peered":"2024-02-19T21:06:53.228167+0000","last_clean":"2024-02-19T21:06:53.228167+0000","last_became_active":"2024-02-19T21:06:32.206052+0000","last_became_peered":"2024-02-19T21:06:32.206052+0000","last_unstale":"2024-02-19T21:06:53.228167+0000","last_undegraded":"2024-02-19T21:06:53.228167+0000","last_fullsized":"2024-02-19T21:06:53.228167+0000","mapping_epoch":98,"log_start":"168'11","ondisk_log_start":"168'11","created":98,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'11","last_scrub_stamp":"2024-02-19T21:06:46.624682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.624682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":218,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.227492+0000","last_change":"2024-02-19T21:05:03.210019+0000","last_active":"2024-02-19T21:06:53.227492+0000","last_peered":"2024-02-19T21:06:53.227492+0000","last_clean":"2024-02-19T21:06:53.227492+0000","last_became_active":"2024-02-19T21:01:18.285534+0000","last_became_peered":"2024-02-19T21:01:18.285534+0000","last_unstale":"2024-02-19T21:06:53.227492+0000","last_undegraded":"2024-02-19T21:06:53.227492+0000","last_fullsized":"2024-02-19T21:06:53.227492+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:05:03.209876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.209876+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":209,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224124+0000","last_change":"2024-02-19T21:06:07.874422+0000","last_active":"2024-02-19T21:06:53.224124+0000","last_peered":"2024-02-19T21:06:53.224124+0000","last_clean":"2024-02-19T21:06:53.224124+0000","last_became_active":"2024-02-19T21:05:20.188821+0000","last_became_peered":"2024-02-19T21:05:20.188821+0000","last_unstale":"2024-02-19T21:06:53.224124+0000","last_undegraded":"2024-02-19T21:06:53.224124+0000","last_fullsized":"2024-02-19T21:06:53.224124+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:07.874362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:07.874362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":272,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-19T21:06:54.196977+0000","last_change":"2024-02-19T21:06:20.710839+0000","last_active":"2024-02-19T21:06:54.196977+0000","last_peered":"2024-02-19T21:06:54.196977+0000","last_clean":"2024-02-19T21:06:54.196977+0000","last_became_active":"2024-02-19T21:01:20.293341+0000","last_became_peered":"2024-02-19T21:01:20.293341+0000","last_unstale":"2024-02-19T21:06:54.196977+0000","last_undegraded":"2024-02-19T21:06:54.196977+0000","last_fullsized":"2024-02-19T21:06:54.196977+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:20.710731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:20.710731+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":194,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.222142+0000","last_change":"2024-02-19T21:06:20.875472+0000","last_active":"2024-02-19T21:06:53.222142+0000","last_peered":"2024-02-19T21:06:53.222142+0000","last_clean":"2024-02-19T21:06:53.222142+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:06:53.222142+0000","last_undegraded":"2024-02-19T21:06:53.222142+0000","last_fullsized":"2024-02-19T21:06:53.222142+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:20.875401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:20.875401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":148,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.212773+0000","last_change":"2024-02-19T21:06:51.423833+0000","last_active":"2024-02-19T21:06:53.212773+0000","last_peered":"2024-02-19T21:06:53.212773+0000","last_clean":"2024-02-19T21:06:53.212773+0000","last_became_active":"2024-02-19T21:04:39.657052+0000","last_became_peered":"2024-02-19T21:04:39.657052+0000","last_unstale":"2024-02-19T21:06:53.212773+0000","last_undegraded":"2024-02-19T21:06:53.212773+0000","last_fullsized":"2024-02-19T21:06:53.212773+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.423769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.423769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":167,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.213861+0000","last_change":"2024-02-19T21:06:36.215015+0000","last_active":"2024-02-19T21:06:53.213861+0000","last_peered":"2024-02-19T21:06:53.213861+0000","last_clean":"2024-02-19T21:06:53.213861+0000","last_became_active":"2024-02-19T21:06:36.214661+0000","last_became_peered":"2024-02-19T21:06:36.214661+0000","last_unstale":"2024-02-19T21:06:53.213861+0000","last_undegraded":"2024-02-19T21:06:53.213861+0000","last_fullsized":"2024-02-19T21:06:53.213861+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:24.733300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:24.733300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":217,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.219118+0000","last_change":"2024-02-19T21:06:02.595348+0000","last_active":"2024-02-19T21:06:53.219118+0000","last_peered":"2024-02-19T21:06:53.219118+0000","last_clean":"2024-02-19T21:06:53.219118+0000","last_became_active":"2024-02-19T21:01:18.285670+0000","last_became_peered":"2024-02-19T21:01:18.285670+0000","last_unstale":"2024-02-19T21:06:53.219118+0000","last_undegraded":"2024-02-19T21:06:53.219118+0000","last_fullsized":"2024-02-19T21:06:53.219118+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:06:02.595266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:02.595266+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.219584+0000","last_change":"2024-02-19T21:06:07.411725+0000","last_active":"2024-02-19T21:06:53.219584+0000","last_peered":"2024-02-19T21:06:53.219584+0000","last_clean":"2024-02-19T21:06:53.219584+0000","last_became_active":"2024-02-19T21:05:20.211711+0000","last_became_peered":"2024-02-19T21:05:20.211711+0000","last_unstale":"2024-02-19T21:06:53.219584+0000","last_undegraded":"2024-02-19T21:06:53.219584+0000","last_fullsized":"2024-02-19T21:06:53.219584+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:07.411662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:07.411662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.428252+0000","last_change":"2024-02-19T21:06:53.428205+0000","last_active":"2024-02-19T21:06:53.428252+0000","last_peered":"2024-02-19T21:06:53.428252+0000","last_clean":"2024-02-19T21:06:53.428252+0000","last_became_active":"2024-02-19T21:01:20.289375+0000","last_became_peered":"2024-02-19T21:01:20.289375+0000","last_unstale":"2024-02-19T21:06:53.428252+0000","last_undegraded":"2024-02-19T21:06:53.428252+0000","last_fullsized":"2024-02-19T21:06:53.428252+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:53.428093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:53.428093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":189,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.217042+0000","last_change":"2024-02-19T21:06:05.721241+0000","last_active":"2024-02-19T21:06:53.217042+0000","last_peered":"2024-02-19T21:06:53.217042+0000","last_clean":"2024-02-19T21:06:53.217042+0000","last_became_active":"2024-02-19T21:01:22.607007+0000","last_became_peered":"2024-02-19T21:01:22.607007+0000","last_unstale":"2024-02-19T21:06:53.217042+0000","last_undegraded":"2024-02-19T21:06:53.217042+0000","last_fullsized":"2024-02-19T21:06:53.217042+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:05.721175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:05.721175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215425+0000","last_change":"2024-02-19T21:04:39.744505+0000","last_active":"2024-02-19T21:06:53.215425+0000","last_peered":"2024-02-19T21:06:53.215425+0000","last_clean":"2024-02-19T21:06:53.215425+0000","last_became_active":"2024-02-19T21:04:39.744231+0000","last_became_peered":"2024-02-19T21:04:39.744231+0000","last_unstale":"2024-02-19T21:06:53.215425+0000","last_undegraded":"2024-02-19T21:06:53.215425+0000","last_fullsized":"2024-02-19T21:06:53.215425+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":272,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-19T21:06:54.198234+0000","last_change":"2024-02-19T21:06:07.761009+0000","last_active":"2024-02-19T21:06:54.198234+0000","last_peered":"2024-02-19T21:06:54.198234+0000","last_clean":"2024-02-19T21:06:54.198234+0000","last_became_active":"2024-02-19T21:01:20.286505+0000","last_became_peered":"2024-02-19T21:01:20.286505+0000","last_unstale":"2024-02-19T21:06:54.198234+0000","last_undegraded":"2024-02-19T21:06:54.198234+0000","last_fullsized":"2024-02-19T21:06:54.198234+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:07.760934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:07.760934+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":219,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.208444+0000","last_change":"2024-02-19T21:06:46.813883+0000","last_active":"2024-02-19T21:06:53.208444+0000","last_peered":"2024-02-19T21:06:53.208444+0000","last_clean":"2024-02-19T21:06:53.208444+0000","last_became_active":"2024-02-19T21:01:18.282702+0000","last_became_peered":"2024-02-19T21:01:18.282702+0000","last_unstale":"2024-02-19T21:06:53.208444+0000","last_undegraded":"2024-02-19T21:06:53.208444+0000","last_fullsized":"2024-02-19T21:06:53.208444+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:06:46.813814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.813814+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":209,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.209906+0000","last_change":"2024-02-19T21:05:35.951650+0000","last_active":"2024-02-19T21:06:53.209906+0000","last_peered":"2024-02-19T21:06:53.209906+0000","last_clean":"2024-02-19T21:06:53.209906+0000","last_became_active":"2024-02-19T21:05:20.224706+0000","last_became_peered":"2024-02-19T21:05:20.224706+0000","last_unstale":"2024-02-19T21:06:53.209906+0000","last_undegraded":"2024-02-19T21:06:53.209906+0000","last_fullsized":"2024-02-19T21:06:53.209906+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.951592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.951592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":190,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224907+0000","last_change":"2024-02-19T21:05:03.248574+0000","last_active":"2024-02-19T21:06:53.224907+0000","last_peered":"2024-02-19T21:06:53.224907+0000","last_clean":"2024-02-19T21:06:53.224907+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:06:53.224907+0000","last_undegraded":"2024-02-19T21:06:53.224907+0000","last_fullsized":"2024-02-19T21:06:53.224907+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.248506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":26,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.226147+0000","last_change":"2024-02-19T21:06:40.245744+0000","last_active":"2024-02-19T21:06:53.226147+0000","last_peered":"2024-02-19T21:06:53.226147+0000","last_clean":"2024-02-19T21:06:53.226147+0000","last_became_active":"2024-02-19T21:06:40.245385+0000","last_became_peered":"2024-02-19T21:06:40.245385+0000","last_unstale":"2024-02-19T21:06:53.226147+0000","last_undegraded":"2024-02-19T21:06:53.226147+0000","last_fullsized":"2024-02-19T21:06:53.226147+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:16.780601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.780601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":104,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.220582+0000","last_change":"2024-02-19T21:05:56.204628+0000","last_active":"2024-02-19T21:06:53.220582+0000","last_peered":"2024-02-19T21:06:53.220582+0000","last_clean":"2024-02-19T21:06:53.220582+0000","last_became_active":"2024-02-19T21:05:56.204312+0000","last_became_peered":"2024-02-19T21:05:56.204312+0000","last_unstale":"2024-02-19T21:06:53.220582+0000","last_undegraded":"2024-02-19T21:06:53.220582+0000","last_fullsized":"2024-02-19T21:06:53.220582+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":191,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.208553+0000","last_change":"2024-02-19T21:05:12.318453+0000","last_active":"2024-02-19T21:06:53.208553+0000","last_peered":"2024-02-19T21:06:53.208553+0000","last_clean":"2024-02-19T21:06:53.208553+0000","last_became_active":"2024-02-19T21:01:20.291079+0000","last_became_peered":"2024-02-19T21:01:20.291079+0000","last_unstale":"2024-02-19T21:06:53.208553+0000","last_undegraded":"2024-02-19T21:06:53.208553+0000","last_fullsized":"2024-02-19T21:06:53.208553+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:05:12.318389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":230,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215738+0000","last_change":"2024-02-19T21:06:46.816190+0000","last_active":"2024-02-19T21:06:53.215738+0000","last_peered":"2024-02-19T21:06:53.215738+0000","last_clean":"2024-02-19T21:06:53.215738+0000","last_became_active":"2024-02-19T21:01:18.293812+0000","last_became_peered":"2024-02-19T21:01:18.293812+0000","last_unstale":"2024-02-19T21:06:53.215738+0000","last_undegraded":"2024-02-19T21:06:53.215738+0000","last_fullsized":"2024-02-19T21:06:53.215738+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:06:46.816121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.816121+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":209,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.227427+0000","last_change":"2024-02-19T21:05:44.876482+0000","last_active":"2024-02-19T21:06:53.227427+0000","last_peered":"2024-02-19T21:06:53.227427+0000","last_clean":"2024-02-19T21:06:53.227427+0000","last_became_active":"2024-02-19T21:05:20.189051+0000","last_became_peered":"2024-02-19T21:05:20.189051+0000","last_unstale":"2024-02-19T21:06:53.227427+0000","last_undegraded":"2024-02-19T21:06:53.227427+0000","last_fullsized":"2024-02-19T21:06:53.227427+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":189,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.214685+0000","last_change":"2024-02-19T21:05:47.538988+0000","last_active":"2024-02-19T21:06:53.214685+0000","last_peered":"2024-02-19T21:06:53.214685+0000","last_clean":"2024-02-19T21:06:53.214685+0000","last_became_active":"2024-02-19T21:01:22.297707+0000","last_became_peered":"2024-02-19T21:01:22.297707+0000","last_unstale":"2024-02-19T21:06:53.214685+0000","last_undegraded":"2024-02-19T21:06:53.214685+0000","last_fullsized":"2024-02-19T21:06:53.214685+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:05:47.538920+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":26,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216459+0000","last_change":"2024-02-19T21:06:40.243815+0000","last_active":"2024-02-19T21:06:53.216459+0000","last_peered":"2024-02-19T21:06:53.216459+0000","last_clean":"2024-02-19T21:06:53.216459+0000","last_became_active":"2024-02-19T21:06:40.243492+0000","last_became_peered":"2024-02-19T21:06:40.243492+0000","last_unstale":"2024-02-19T21:06:53.216459+0000","last_undegraded":"2024-02-19T21:06:53.216459+0000","last_fullsized":"2024-02-19T21:06:53.216459+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216044+0000","last_change":"2024-02-19T21:04:56.720699+0000","last_active":"2024-02-19T21:06:53.216044+0000","last_peered":"2024-02-19T21:06:53.216044+0000","last_clean":"2024-02-19T21:06:53.216044+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:06:53.216044+0000","last_undegraded":"2024-02-19T21:06:53.216044+0000","last_fullsized":"2024-02-19T21:06:53.216044+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:56.720607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.720607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":74,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.218103+0000","last_change":"2024-02-19T21:06:40.224213+0000","last_active":"2024-02-19T21:06:53.218103+0000","last_peered":"2024-02-19T21:06:53.218103+0000","last_clean":"2024-02-19T21:06:53.218103+0000","last_became_active":"2024-02-19T21:06:40.223768+0000","last_became_peered":"2024-02-19T21:06:40.223768+0000","last_unstale":"2024-02-19T21:06:53.218103+0000","last_undegraded":"2024-02-19T21:06:53.218103+0000","last_fullsized":"2024-02-19T21:06:53.218103+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:06:10.826889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:10.826889+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":210,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.227473+0000","last_change":"2024-02-19T21:06:06.458624+0000","last_active":"2024-02-19T21:06:53.227473+0000","last_peered":"2024-02-19T21:06:53.227473+0000","last_clean":"2024-02-19T21:06:53.227473+0000","last_became_active":"2024-02-19T21:05:20.211241+0000","last_became_peered":"2024-02-19T21:05:20.211241+0000","last_unstale":"2024-02-19T21:06:53.227473+0000","last_undegraded":"2024-02-19T21:06:53.227473+0000","last_fullsized":"2024-02-19T21:06:53.227473+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:06.458541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:06.458541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":189,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224227+0000","last_change":"2024-02-19T21:05:01.859040+0000","last_active":"2024-02-19T21:06:53.224227+0000","last_peered":"2024-02-19T21:06:53.224227+0000","last_clean":"2024-02-19T21:06:53.224227+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:06:53.224227+0000","last_undegraded":"2024-02-19T21:06:53.224227+0000","last_fullsized":"2024-02-19T21:06:53.224227+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:01.858924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.858924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216625+0000","last_change":"2024-02-19T21:04:39.719113+0000","last_active":"2024-02-19T21:06:53.216625+0000","last_peered":"2024-02-19T21:06:53.216625+0000","last_clean":"2024-02-19T21:06:53.216625+0000","last_became_active":"2024-02-19T21:04:39.718926+0000","last_became_peered":"2024-02-19T21:04:39.718926+0000","last_unstale":"2024-02-19T21:06:53.216625+0000","last_undegraded":"2024-02-19T21:06:53.216625+0000","last_fullsized":"2024-02-19T21:06:53.216625+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.a","version":"168'14","reported_seq":176,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224233+0000","last_change":"2024-02-19T21:06:04.207771+0000","last_active":"2024-02-19T21:06:53.224233+0000","last_peered":"2024-02-19T21:06:53.224233+0000","last_clean":"2024-02-19T21:06:53.224233+0000","last_became_active":"2024-02-19T21:06:04.207446+0000","last_became_peered":"2024-02-19T21:06:04.207446+0000","last_unstale":"2024-02-19T21:06:53.224233+0000","last_undegraded":"2024-02-19T21:06:53.224233+0000","last_fullsized":"2024-02-19T21:06:53.224233+0000","mapping_epoch":98,"log_start":"168'14","ondisk_log_start":"168'14","created":98,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":384,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":191,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.217854+0000","last_change":"2024-02-19T21:04:57.188742+0000","last_active":"2024-02-19T21:06:53.217854+0000","last_peered":"2024-02-19T21:06:53.217854+0000","last_clean":"2024-02-19T21:06:53.217854+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:06:53.217854+0000","last_undegraded":"2024-02-19T21:06:53.217854+0000","last_fullsized":"2024-02-19T21:06:53.217854+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:57.188664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:57.188664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":56,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.227593+0000","last_change":"2024-02-19T21:06:40.240732+0000","last_active":"2024-02-19T21:06:53.227593+0000","last_peered":"2024-02-19T21:06:53.227593+0000","last_clean":"2024-02-19T21:06:53.227593+0000","last_became_active":"2024-02-19T21:06:40.240321+0000","last_became_peered":"2024-02-19T21:06:40.240321+0000","last_unstale":"2024-02-19T21:06:53.227593+0000","last_undegraded":"2024-02-19T21:06:53.227593+0000","last_fullsized":"2024-02-19T21:06:53.227593+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:05:09.646445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:09.646445+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":215,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.219421+0000","last_change":"2024-02-19T21:06:09.422099+0000","last_active":"2024-02-19T21:06:53.219421+0000","last_peered":"2024-02-19T21:06:53.219421+0000","last_clean":"2024-02-19T21:06:53.219421+0000","last_became_active":"2024-02-19T21:05:20.211701+0000","last_became_peered":"2024-02-19T21:05:20.211701+0000","last_unstale":"2024-02-19T21:06:53.219421+0000","last_undegraded":"2024-02-19T21:06:53.219421+0000","last_fullsized":"2024-02-19T21:06:53.219421+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:06:09.422007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.422007+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.208355+0000","last_change":"2024-02-19T21:06:40.221051+0000","last_active":"2024-02-19T21:06:53.208355+0000","last_peered":"2024-02-19T21:06:53.208355+0000","last_clean":"2024-02-19T21:06:53.208355+0000","last_became_active":"2024-02-19T21:06:40.220667+0000","last_became_peered":"2024-02-19T21:06:40.220667+0000","last_unstale":"2024-02-19T21:06:53.208355+0000","last_undegraded":"2024-02-19T21:06:53.208355+0000","last_fullsized":"2024-02-19T21:06:53.208355+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:07.660435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:05:07.660435+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":194,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224445+0000","last_change":"2024-02-19T21:06:51.627839+0000","last_active":"2024-02-19T21:06:53.224445+0000","last_peered":"2024-02-19T21:06:53.224445+0000","last_clean":"2024-02-19T21:06:53.224445+0000","last_became_active":"2024-02-19T21:01:22.298477+0000","last_became_peered":"2024-02-19T21:01:22.298477+0000","last_unstale":"2024-02-19T21:06:53.224445+0000","last_undegraded":"2024-02-19T21:06:53.224445+0000","last_fullsized":"2024-02-19T21:06:53.224445+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.627708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.627708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.212801+0000","last_change":"2024-02-19T21:04:39.665830+0000","last_active":"2024-02-19T21:06:53.212801+0000","last_peered":"2024-02-19T21:06:53.212801+0000","last_clean":"2024-02-19T21:06:53.212801+0000","last_became_active":"2024-02-19T21:04:39.662125+0000","last_became_peered":"2024-02-19T21:04:39.662125+0000","last_unstale":"2024-02-19T21:06:53.212801+0000","last_undegraded":"2024-02-19T21:06:53.212801+0000","last_fullsized":"2024-02-19T21:06:53.212801+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":162,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.220667+0000","last_change":"2024-02-19T21:06:08.207026+0000","last_active":"2024-02-19T21:06:53.220667+0000","last_peered":"2024-02-19T21:06:53.220667+0000","last_clean":"2024-02-19T21:06:53.220667+0000","last_became_active":"2024-02-19T21:06:08.206725+0000","last_became_peered":"2024-02-19T21:06:08.206725+0000","last_unstale":"2024-02-19T21:06:53.220667+0000","last_undegraded":"2024-02-19T21:06:53.220667+0000","last_fullsized":"2024-02-19T21:06:53.220667+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":195,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224951+0000","last_change":"2024-02-19T21:06:51.666781+0000","last_active":"2024-02-19T21:06:53.224951+0000","last_peered":"2024-02-19T21:06:53.224951+0000","last_clean":"2024-02-19T21:06:53.224951+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:06:53.224951+0000","last_undegraded":"2024-02-19T21:06:53.224951+0000","last_fullsized":"2024-02-19T21:06:53.224951+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.666708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.666708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":220,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216643+0000","last_change":"2024-02-19T21:06:26.777041+0000","last_active":"2024-02-19T21:06:53.216643+0000","last_peered":"2024-02-19T21:06:53.216643+0000","last_clean":"2024-02-19T21:06:53.216643+0000","last_became_active":"2024-02-19T21:01:18.293784+0000","last_became_peered":"2024-02-19T21:01:18.293784+0000","last_unstale":"2024-02-19T21:06:53.216643+0000","last_undegraded":"2024-02-19T21:06:53.216643+0000","last_fullsized":"2024-02-19T21:06:53.216643+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:06:26.776963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:26.776963+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":203,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.219507+0000","last_change":"2024-02-19T21:05:00.284714+0000","last_active":"2024-02-19T21:06:53.219507+0000","last_peered":"2024-02-19T21:06:53.219507+0000","last_clean":"2024-02-19T21:06:53.219507+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:06:53.219507+0000","last_undegraded":"2024-02-19T21:06:53.219507+0000","last_fullsized":"2024-02-19T21:06:53.219507+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:00.284626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:05:00.284626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":209,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.218017+0000","last_change":"2024-02-19T21:06:17.907387+0000","last_active":"2024-02-19T21:06:53.218017+0000","last_peered":"2024-02-19T21:06:53.218017+0000","last_clean":"2024-02-19T21:06:53.218017+0000","last_became_active":"2024-02-19T21:05:20.195763+0000","last_became_peered":"2024-02-19T21:05:20.195763+0000","last_unstale":"2024-02-19T21:06:53.218017+0000","last_undegraded":"2024-02-19T21:06:53.218017+0000","last_fullsized":"2024-02-19T21:06:53.218017+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:17.907318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:17.907318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":195,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215549+0000","last_change":"2024-02-19T21:04:54.972925+0000","last_active":"2024-02-19T21:06:53.215549+0000","last_peered":"2024-02-19T21:06:53.215549+0000","last_clean":"2024-02-19T21:06:53.215549+0000","last_became_active":"2024-02-19T21:01:16.274290+0000","last_became_peered":"2024-02-19T21:01:16.274290+0000","last_unstale":"2024-02-19T21:06:53.215549+0000","last_undegraded":"2024-02-19T21:06:53.215549+0000","last_fullsized":"2024-02-19T21:06:53.215549+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:54.972832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:54.972832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":194,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.217918+0000","last_change":"2024-02-19T21:06:16.863491+0000","last_active":"2024-02-19T21:06:53.217918+0000","last_peered":"2024-02-19T21:06:53.217918+0000","last_clean":"2024-02-19T21:06:53.217918+0000","last_became_active":"2024-02-19T21:01:22.292299+0000","last_became_peered":"2024-02-19T21:01:22.292299+0000","last_unstale":"2024-02-19T21:06:53.217918+0000","last_undegraded":"2024-02-19T21:06:53.217918+0000","last_fullsized":"2024-02-19T21:06:53.217918+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:16.863398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.863398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":148,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.209627+0000","last_change":"2024-02-19T21:06:52.403019+0000","last_active":"2024-02-19T21:06:53.209627+0000","last_peered":"2024-02-19T21:06:53.209627+0000","last_clean":"2024-02-19T21:06:53.209627+0000","last_became_active":"2024-02-19T21:04:39.666739+0000","last_became_peered":"2024-02-19T21:04:39.666739+0000","last_unstale":"2024-02-19T21:06:53.209627+0000","last_undegraded":"2024-02-19T21:06:53.209627+0000","last_fullsized":"2024-02-19T21:06:53.209627+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:52.402941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:52.402941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"199'16","reported_seq":128,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.228815+0000","last_change":"2024-02-19T21:06:19.717662+0000","last_active":"2024-02-19T21:06:53.228815+0000","last_peered":"2024-02-19T21:06:53.228815+0000","last_clean":"2024-02-19T21:06:53.228815+0000","last_became_active":"2024-02-19T21:06:00.210690+0000","last_became_peered":"2024-02-19T21:06:00.210690+0000","last_unstale":"2024-02-19T21:06:53.228815+0000","last_undegraded":"2024-02-19T21:06:53.228815+0000","last_fullsized":"2024-02-19T21:06:53.228815+0000","mapping_epoch":157,"log_start":"168'14","ondisk_log_start":"168'14","created":98,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"199'16","last_scrub_stamp":"2024-02-19T21:06:19.717596+0000","last_deep_scrub":"199'16","last_deep_scrub_stamp":"2024-02-19T21:06:19.717596+0000","last_clean_scrub_stamp":"2024-02-19T21:06:19.717596+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215726+0000","last_change":"2024-02-19T21:06:23.273349+0000","last_active":"2024-02-19T21:06:53.215726+0000","last_peered":"2024-02-19T21:06:53.215726+0000","last_clean":"2024-02-19T21:06:53.215726+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:06:53.215726+0000","last_undegraded":"2024-02-19T21:06:53.215726+0000","last_fullsized":"2024-02-19T21:06:53.215726+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:23.273274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:23.273274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":209,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216548+0000","last_change":"2024-02-19T21:04:56.846706+0000","last_active":"2024-02-19T21:06:53.216548+0000","last_peered":"2024-02-19T21:06:53.216548+0000","last_clean":"2024-02-19T21:06:53.216548+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:06:53.216548+0000","last_undegraded":"2024-02-19T21:06:53.216548+0000","last_fullsized":"2024-02-19T21:06:53.216548+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:04:56.846622+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.846622+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":214,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.219311+0000","last_change":"2024-02-19T21:05:45.014031+0000","last_active":"2024-02-19T21:06:53.219311+0000","last_peered":"2024-02-19T21:06:53.219311+0000","last_clean":"2024-02-19T21:06:53.219311+0000","last_became_active":"2024-02-19T21:05:20.219851+0000","last_became_peered":"2024-02-19T21:05:20.219851+0000","last_unstale":"2024-02-19T21:06:53.219311+0000","last_undegraded":"2024-02-19T21:06:53.219311+0000","last_fullsized":"2024-02-19T21:06:53.219311+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:45.013946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:45.013946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":534,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-19T21:06:54.212606+0000","last_change":"2024-02-19T21:06:10.791107+0000","last_active":"2024-02-19T21:06:54.212606+0000","last_peered":"2024-02-19T21:06:54.212606+0000","last_clean":"2024-02-19T21:06:54.212606+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:06:54.212606+0000","last_undegraded":"2024-02-19T21:06:54.212606+0000","last_fullsized":"2024-02-19T21:06:54.212606+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:06:10.791036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:10.791036+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":331,"num_read_kb":326,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":193,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.227396+0000","last_change":"2024-02-19T21:06:49.690671+0000","last_active":"2024-02-19T21:06:53.227396+0000","last_peered":"2024-02-19T21:06:53.227396+0000","last_clean":"2024-02-19T21:06:53.227396+0000","last_became_active":"2024-02-19T21:01:22.607883+0000","last_became_peered":"2024-02-19T21:01:22.607883+0000","last_unstale":"2024-02-19T21:06:53.227396+0000","last_undegraded":"2024-02-19T21:06:53.227396+0000","last_fullsized":"2024-02-19T21:06:53.227396+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.690596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.690596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.228718+0000","last_change":"2024-02-19T21:04:39.692210+0000","last_active":"2024-02-19T21:06:53.228718+0000","last_peered":"2024-02-19T21:06:53.228718+0000","last_clean":"2024-02-19T21:06:53.228718+0000","last_became_active":"2024-02-19T21:04:39.691434+0000","last_became_peered":"2024-02-19T21:04:39.691434+0000","last_unstale":"2024-02-19T21:06:53.228718+0000","last_undegraded":"2024-02-19T21:06:53.228718+0000","last_fullsized":"2024-02-19T21:06:53.228718+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"100'2","reported_seq":169,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-19T21:06:54.379177+0000","last_change":"2024-02-19T21:06:54.379177+0000","last_active":"2024-02-19T21:06:54.379177+0000","last_peered":"2024-02-19T21:06:54.379177+0000","last_clean":"2024-02-19T21:06:54.379177+0000","last_became_active":"2024-02-19T21:06:12.215103+0000","last_became_peered":"2024-02-19T21:06:12.215103+0000","last_unstale":"2024-02-19T21:06:54.379177+0000","last_undegraded":"2024-02-19T21:06:54.379177+0000","last_fullsized":"2024-02-19T21:06:54.379177+0000","mapping_epoch":98,"log_start":"100'2","ondisk_log_start":"100'2","created":98,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'2","last_scrub_stamp":"2024-02-19T21:06:54.379107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.379107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":26,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.225904+0000","last_change":"2024-02-19T21:06:40.245910+0000","last_active":"2024-02-19T21:06:53.225904+0000","last_peered":"2024-02-19T21:06:53.225904+0000","last_clean":"2024-02-19T21:06:53.225904+0000","last_became_active":"2024-02-19T21:06:40.245469+0000","last_became_peered":"2024-02-19T21:06:40.245469+0000","last_unstale":"2024-02-19T21:06:53.225904+0000","last_undegraded":"2024-02-19T21:06:53.225904+0000","last_fullsized":"2024-02-19T21:06:53.225904+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:01.368509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.368509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":314,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-19T21:06:54.212463+0000","last_change":"2024-02-19T21:05:42.927711+0000","last_active":"2024-02-19T21:06:54.212463+0000","last_peered":"2024-02-19T21:06:54.212463+0000","last_clean":"2024-02-19T21:06:54.212463+0000","last_became_active":"2024-02-19T21:01:18.289076+0000","last_became_peered":"2024-02-19T21:01:18.289076+0000","last_unstale":"2024-02-19T21:06:54.212463+0000","last_undegraded":"2024-02-19T21:06:54.212463+0000","last_fullsized":"2024-02-19T21:06:54.212463+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-19T21:05:42.927640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:42.927640+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":209,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215405+0000","last_change":"2024-02-19T21:06:19.845462+0000","last_active":"2024-02-19T21:06:53.215405+0000","last_peered":"2024-02-19T21:06:53.215405+0000","last_clean":"2024-02-19T21:06:53.215405+0000","last_became_active":"2024-02-19T21:05:20.181450+0000","last_became_peered":"2024-02-19T21:05:20.181450+0000","last_unstale":"2024-02-19T21:06:53.215405+0000","last_undegraded":"2024-02-19T21:06:53.215405+0000","last_fullsized":"2024-02-19T21:06:53.215405+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:06:19.845394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":189,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.210879+0000","last_change":"2024-02-19T21:04:49.004211+0000","last_active":"2024-02-19T21:06:53.210879+0000","last_peered":"2024-02-19T21:06:53.210879+0000","last_clean":"2024-02-19T21:06:53.210879+0000","last_became_active":"2024-02-19T21:01:22.291396+0000","last_became_peered":"2024-02-19T21:01:22.291396+0000","last_unstale":"2024-02-19T21:06:53.210879+0000","last_undegraded":"2024-02-19T21:06:53.210879+0000","last_fullsized":"2024-02-19T21:06:53.210879+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.004148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.004148+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":148,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-19T21:06:54.355870+0000","last_change":"2024-02-19T21:06:54.355870+0000","last_active":"2024-02-19T21:06:54.355870+0000","last_peered":"2024-02-19T21:06:54.355870+0000","last_clean":"2024-02-19T21:06:54.355870+0000","last_became_active":"2024-02-19T21:04:39.721416+0000","last_became_peered":"2024-02-19T21:04:39.721416+0000","last_unstale":"2024-02-19T21:06:54.355870+0000","last_undegraded":"2024-02-19T21:06:54.355870+0000","last_fullsized":"2024-02-19T21:06:54.355870+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.355809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.355809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"220'9","reported_seq":52,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-19T21:06:54.193193+0000","last_change":"2024-02-19T21:06:40.228012+0000","last_active":"2024-02-19T21:06:54.193193+0000","last_peered":"2024-02-19T21:06:54.193193+0000","last_clean":"2024-02-19T21:06:54.193193+0000","last_became_active":"2024-02-19T21:06:40.227604+0000","last_became_peered":"2024-02-19T21:06:40.227604+0000","last_unstale":"2024-02-19T21:06:54.193193+0000","last_undegraded":"2024-02-19T21:06:54.193193+0000","last_fullsized":"2024-02-19T21:06:54.193193+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:21.819105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:21.819105+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224126+0000","last_change":"2024-02-19T21:05:03.922155+0000","last_active":"2024-02-19T21:06:53.224126+0000","last_peered":"2024-02-19T21:06:53.224126+0000","last_clean":"2024-02-19T21:06:53.224126+0000","last_became_active":"2024-02-19T21:01:18.285891+0000","last_became_peered":"2024-02-19T21:01:18.285891+0000","last_unstale":"2024-02-19T21:06:53.224126+0000","last_undegraded":"2024-02-19T21:06:53.224126+0000","last_fullsized":"2024-02-19T21:06:53.224126+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:05:03.922055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.922055+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":218,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224057+0000","last_change":"2024-02-19T21:05:47.885080+0000","last_active":"2024-02-19T21:06:53.224057+0000","last_peered":"2024-02-19T21:06:53.224057+0000","last_clean":"2024-02-19T21:06:53.224057+0000","last_became_active":"2024-02-19T21:05:20.222948+0000","last_became_peered":"2024-02-19T21:05:20.222948+0000","last_unstale":"2024-02-19T21:06:53.224057+0000","last_undegraded":"2024-02-19T21:06:53.224057+0000","last_fullsized":"2024-02-19T21:06:53.224057+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:47.885024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:05:47.885024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":26,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215665+0000","last_change":"2024-02-19T21:06:40.226781+0000","last_active":"2024-02-19T21:06:53.215665+0000","last_peered":"2024-02-19T21:06:53.215665+0000","last_clean":"2024-02-19T21:06:53.215665+0000","last_became_active":"2024-02-19T21:06:40.226458+0000","last_became_peered":"2024-02-19T21:06:40.226458+0000","last_unstale":"2024-02-19T21:06:53.215665+0000","last_undegraded":"2024-02-19T21:06:53.215665+0000","last_fullsized":"2024-02-19T21:06:53.215665+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:15.807734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:15.807734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.210639+0000","last_change":"2024-02-19T21:04:39.767350+0000","last_active":"2024-02-19T21:06:53.210639+0000","last_peered":"2024-02-19T21:06:53.210639+0000","last_clean":"2024-02-19T21:06:53.210639+0000","last_became_active":"2024-02-19T21:04:39.766156+0000","last_became_peered":"2024-02-19T21:04:39.766156+0000","last_unstale":"2024-02-19T21:06:53.210639+0000","last_undegraded":"2024-02-19T21:06:53.210639+0000","last_fullsized":"2024-02-19T21:06:53.210639+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":176,"reported_epoch":234,"state":"clean+premerge+peered","last_fresh":"2024-02-19T21:06:54.210116+0000","last_change":"2024-02-19T21:06:54.210116+0000","last_active":"2024-02-19T21:06:53.217544+0000","last_peered":"2024-02-19T21:06:54.210116+0000","last_clean":"2024-02-19T21:06:54.210116+0000","last_became_active":"2024-02-19T21:06:24.196185+0000","last_became_peered":"2024-02-19T21:06:54.209320+0000","last_unstale":"2024-02-19T21:06:54.210116+0000","last_undegraded":"2024-02-19T21:06:54.210116+0000","last_fullsized":"2024-02-19T21:06:54.210116+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:16.827566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.827566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.213126+0000","last_change":"2024-02-19T21:06:20.194149+0000","last_active":"2024-02-19T21:06:53.213126+0000","last_peered":"2024-02-19T21:06:53.213126+0000","last_clean":"2024-02-19T21:06:53.213126+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:06:53.213126+0000","last_undegraded":"2024-02-19T21:06:53.213126+0000","last_fullsized":"2024-02-19T21:06:53.213126+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:20.194018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:06:20.194018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":221,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.208461+0000","last_change":"2024-02-19T21:06:48.844390+0000","last_active":"2024-02-19T21:06:53.208461+0000","last_peered":"2024-02-19T21:06:53.208461+0000","last_clean":"2024-02-19T21:06:53.208461+0000","last_became_active":"2024-02-19T21:01:18.285672+0000","last_became_peered":"2024-02-19T21:01:18.285672+0000","last_unstale":"2024-02-19T21:06:53.208461+0000","last_undegraded":"2024-02-19T21:06:53.208461+0000","last_fullsized":"2024-02-19T21:06:53.208461+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:06:48.844301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.844301+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":215,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224233+0000","last_change":"2024-02-19T21:05:01.259381+0000","last_active":"2024-02-19T21:06:53.224233+0000","last_peered":"2024-02-19T21:06:53.224233+0000","last_clean":"2024-02-19T21:06:53.224233+0000","last_became_active":"2024-02-19T21:01:16.273545+0000","last_became_peered":"2024-02-19T21:01:16.273545+0000","last_unstale":"2024-02-19T21:06:53.224233+0000","last_undegraded":"2024-02-19T21:06:53.224233+0000","last_fullsized":"2024-02-19T21:06:53.224233+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:05:01.259313+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.259313+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":189,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.213488+0000","last_change":"2024-02-19T21:05:06.001027+0000","last_active":"2024-02-19T21:06:53.213488+0000","last_peered":"2024-02-19T21:06:53.213488+0000","last_clean":"2024-02-19T21:06:53.213488+0000","last_became_active":"2024-02-19T21:01:22.295912+0000","last_became_peered":"2024-02-19T21:01:22.295912+0000","last_unstale":"2024-02-19T21:06:53.213488+0000","last_undegraded":"2024-02-19T21:06:53.213488+0000","last_fullsized":"2024-02-19T21:06:53.213488+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:06.000903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:05:06.000903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215038+0000","last_change":"2024-02-19T21:04:39.696886+0000","last_active":"2024-02-19T21:06:53.215038+0000","last_peered":"2024-02-19T21:06:53.215038+0000","last_clean":"2024-02-19T21:06:53.215038+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:06:53.215038+0000","last_undegraded":"2024-02-19T21:06:53.215038+0000","last_fullsized":"2024-02-19T21:06:53.215038+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":54,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.210956+0000","last_change":"2024-02-19T21:06:40.227501+0000","last_active":"2024-02-19T21:06:53.210956+0000","last_peered":"2024-02-19T21:06:53.210956+0000","last_clean":"2024-02-19T21:06:53.210956+0000","last_became_active":"2024-02-19T21:06:40.227142+0000","last_became_peered":"2024-02-19T21:06:40.227142+0000","last_unstale":"2024-02-19T21:06:53.210956+0000","last_undegraded":"2024-02-19T21:06:53.210956+0000","last_fullsized":"2024-02-19T21:06:53.210956+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:06:09.793208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.793208+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":195,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.213022+0000","last_change":"2024-02-19T21:04:49.819984+0000","last_active":"2024-02-19T21:06:53.213022+0000","last_peered":"2024-02-19T21:06:53.213022+0000","last_clean":"2024-02-19T21:06:53.213022+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:06:53.213022+0000","last_undegraded":"2024-02-19T21:06:53.213022+0000","last_fullsized":"2024-02-19T21:06:53.213022+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.819921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.819921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":191,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.207515+0000","last_change":"2024-02-19T21:04:11.841348+0000","last_active":"2024-02-19T21:06:53.207515+0000","last_peered":"2024-02-19T21:06:53.207515+0000","last_clean":"2024-02-19T21:06:53.207515+0000","last_became_active":"2024-02-19T21:01:20.285298+0000","last_became_peered":"2024-02-19T21:01:20.285298+0000","last_unstale":"2024-02-19T21:06:53.207515+0000","last_undegraded":"2024-02-19T21:06:53.207515+0000","last_fullsized":"2024-02-19T21:06:53.207515+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:11.841184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:04:11.841184+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":194,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.220814+0000","last_change":"2024-02-19T21:05:11.826779+0000","last_active":"2024-02-19T21:06:53.220814+0000","last_peered":"2024-02-19T21:06:53.220814+0000","last_clean":"2024-02-19T21:06:53.220814+0000","last_became_active":"2024-02-19T21:01:22.297086+0000","last_became_peered":"2024-02-19T21:01:22.297086+0000","last_unstale":"2024-02-19T21:06:53.220814+0000","last_undegraded":"2024-02-19T21:06:53.220814+0000","last_fullsized":"2024-02-19T21:06:53.220814+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:11.826647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:11.826647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":148,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.213790+0000","last_change":"2024-02-19T21:06:21.704812+0000","last_active":"2024-02-19T21:06:53.213790+0000","last_peered":"2024-02-19T21:06:53.213790+0000","last_clean":"2024-02-19T21:06:53.213790+0000","last_became_active":"2024-02-19T21:04:39.639518+0000","last_became_peered":"2024-02-19T21:04:39.639518+0000","last_unstale":"2024-02-19T21:06:53.213790+0000","last_undegraded":"2024-02-19T21:06:53.213790+0000","last_fullsized":"2024-02-19T21:06:53.213790+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:21.704719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:21.704719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":210,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.207438+0000","last_change":"2024-02-19T21:06:48.855339+0000","last_active":"2024-02-19T21:06:53.207438+0000","last_peered":"2024-02-19T21:06:53.207438+0000","last_clean":"2024-02-19T21:06:53.207438+0000","last_became_active":"2024-02-19T21:01:18.293913+0000","last_became_peered":"2024-02-19T21:01:18.293913+0000","last_unstale":"2024-02-19T21:06:53.207438+0000","last_undegraded":"2024-02-19T21:06:53.207438+0000","last_fullsized":"2024-02-19T21:06:53.207438+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:06:48.855268+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.855268+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":200,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.220836+0000","last_change":"2024-02-19T21:06:15.793461+0000","last_active":"2024-02-19T21:06:53.220836+0000","last_peered":"2024-02-19T21:06:53.220836+0000","last_clean":"2024-02-19T21:06:53.220836+0000","last_became_active":"2024-02-19T21:01:16.271210+0000","last_became_peered":"2024-02-19T21:01:16.271210+0000","last_unstale":"2024-02-19T21:06:53.220836+0000","last_undegraded":"2024-02-19T21:06:53.220836+0000","last_fullsized":"2024-02-19T21:06:53.220836+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:15.793328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:15.793328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":193,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.210909+0000","last_change":"2024-02-19T21:05:04.174022+0000","last_active":"2024-02-19T21:06:53.210909+0000","last_peered":"2024-02-19T21:06:53.210909+0000","last_clean":"2024-02-19T21:06:53.210909+0000","last_became_active":"2024-02-19T21:01:20.288589+0000","last_became_peered":"2024-02-19T21:01:20.288589+0000","last_unstale":"2024-02-19T21:06:53.210909+0000","last_undegraded":"2024-02-19T21:06:53.210909+0000","last_fullsized":"2024-02-19T21:06:53.210909+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:04.173902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:04.173902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":193,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.227174+0000","last_change":"2024-02-19T21:06:05.431079+0000","last_active":"2024-02-19T21:06:53.227174+0000","last_peered":"2024-02-19T21:06:53.227174+0000","last_clean":"2024-02-19T21:06:53.227174+0000","last_became_active":"2024-02-19T21:01:22.608285+0000","last_became_peered":"2024-02-19T21:01:22.608285+0000","last_unstale":"2024-02-19T21:06:53.227174+0000","last_undegraded":"2024-02-19T21:06:53.227174+0000","last_fullsized":"2024-02-19T21:06:53.227174+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:05.431017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:05.431017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.212690+0000","last_change":"2024-02-19T21:04:39.641073+0000","last_active":"2024-02-19T21:06:53.212690+0000","last_peered":"2024-02-19T21:06:53.212690+0000","last_clean":"2024-02-19T21:06:53.212690+0000","last_became_active":"2024-02-19T21:04:39.639775+0000","last_became_peered":"2024-02-19T21:04:39.639775+0000","last_unstale":"2024-02-19T21:06:53.212690+0000","last_undegraded":"2024-02-19T21:06:53.212690+0000","last_fullsized":"2024-02-19T21:06:53.212690+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.217125+0000","last_change":"2024-02-19T21:04:22.763995+0000","last_active":"2024-02-19T21:06:53.217125+0000","last_peered":"2024-02-19T21:06:53.217125+0000","last_clean":"2024-02-19T21:06:53.217125+0000","last_became_active":"2024-02-19T21:01:18.294369+0000","last_became_peered":"2024-02-19T21:01:18.294369+0000","last_unstale":"2024-02-19T21:06:53.217125+0000","last_undegraded":"2024-02-19T21:06:53.217125+0000","last_fullsized":"2024-02-19T21:06:53.217125+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:04:22.763926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:22.763926+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":200,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215326+0000","last_change":"2024-02-19T21:06:15.892999+0000","last_active":"2024-02-19T21:06:53.215326+0000","last_peered":"2024-02-19T21:06:53.215326+0000","last_clean":"2024-02-19T21:06:53.215326+0000","last_became_active":"2024-02-19T21:01:16.271608+0000","last_became_peered":"2024-02-19T21:01:16.271608+0000","last_unstale":"2024-02-19T21:06:53.215326+0000","last_undegraded":"2024-02-19T21:06:53.215326+0000","last_fullsized":"2024-02-19T21:06:53.215326+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:15.892925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:15.892925+0000","last_clean_scrub_stamp":"2024-02-19T21:06:15.892925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":267,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-19T21:06:54.197058+0000","last_change":"2024-02-19T21:05:34.889050+0000","last_active":"2024-02-19T21:06:54.197058+0000","last_peered":"2024-02-19T21:06:54.197058+0000","last_clean":"2024-02-19T21:06:54.197058+0000","last_became_active":"2024-02-19T21:01:20.288799+0000","last_became_peered":"2024-02-19T21:01:20.288799+0000","last_unstale":"2024-02-19T21:06:54.197058+0000","last_undegraded":"2024-02-19T21:06:54.197058+0000","last_fullsized":"2024-02-19T21:06:54.197058+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:05:34.888948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:34.888948+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":35,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.226383+0000","last_change":"2024-02-19T21:06:40.229090+0000","last_active":"2024-02-19T21:06:53.226383+0000","last_peered":"2024-02-19T21:06:53.226383+0000","last_clean":"2024-02-19T21:06:53.226383+0000","last_became_active":"2024-02-19T21:06:40.228391+0000","last_became_peered":"2024-02-19T21:06:40.228391+0000","last_unstale":"2024-02-19T21:06:53.226383+0000","last_undegraded":"2024-02-19T21:06:53.226383+0000","last_fullsized":"2024-02-19T21:06:53.226383+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:05:43.790222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:43.790222+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":26,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.208856+0000","last_change":"2024-02-19T21:06:40.219921+0000","last_active":"2024-02-19T21:06:53.208856+0000","last_peered":"2024-02-19T21:06:53.208856+0000","last_clean":"2024-02-19T21:06:53.208856+0000","last_became_active":"2024-02-19T21:06:40.219590+0000","last_became_peered":"2024-02-19T21:06:40.219590+0000","last_unstale":"2024-02-19T21:06:53.208856+0000","last_undegraded":"2024-02-19T21:06:53.208856+0000","last_fullsized":"2024-02-19T21:06:53.208856+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:17.776057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:17.776057+0000","last_clean_scrub_stamp":"2024-02-19T21:06:17.776057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":214,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.221846+0000","last_change":"2024-02-19T21:06:47.541116+0000","last_active":"2024-02-19T21:06:53.221846+0000","last_peered":"2024-02-19T21:06:53.221846+0000","last_clean":"2024-02-19T21:06:53.221846+0000","last_became_active":"2024-02-19T21:01:18.293786+0000","last_became_peered":"2024-02-19T21:01:18.293786+0000","last_unstale":"2024-02-19T21:06:53.221846+0000","last_undegraded":"2024-02-19T21:06:53.221846+0000","last_fullsized":"2024-02-19T21:06:53.221846+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:06:47.541012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.541012+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":26,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.207541+0000","last_change":"2024-02-19T21:06:40.220653+0000","last_active":"2024-02-19T21:06:53.207541+0000","last_peered":"2024-02-19T21:06:53.207541+0000","last_clean":"2024-02-19T21:06:53.207541+0000","last_became_active":"2024-02-19T21:06:40.220310+0000","last_became_peered":"2024-02-19T21:06:40.220310+0000","last_unstale":"2024-02-19T21:06:53.207541+0000","last_undegraded":"2024-02-19T21:06:53.207541+0000","last_fullsized":"2024-02-19T21:06:53.207541+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:10.715442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:10.715442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":267,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-19T21:06:54.202030+0000","last_change":"2024-02-19T21:05:02.007769+0000","last_active":"2024-02-19T21:06:54.202030+0000","last_peered":"2024-02-19T21:06:54.202030+0000","last_clean":"2024-02-19T21:06:54.202030+0000","last_became_active":"2024-02-19T21:01:20.283943+0000","last_became_peered":"2024-02-19T21:01:20.283943+0000","last_unstale":"2024-02-19T21:06:54.202030+0000","last_undegraded":"2024-02-19T21:06:54.202030+0000","last_fullsized":"2024-02-19T21:06:54.202030+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:05:02.007654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.007654+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":194,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224646+0000","last_change":"2024-02-19T21:06:15.753686+0000","last_active":"2024-02-19T21:06:53.224646+0000","last_peered":"2024-02-19T21:06:53.224646+0000","last_clean":"2024-02-19T21:06:53.224646+0000","last_became_active":"2024-02-19T21:01:22.299969+0000","last_became_peered":"2024-02-19T21:01:22.299969+0000","last_unstale":"2024-02-19T21:06:53.224646+0000","last_undegraded":"2024-02-19T21:06:53.224646+0000","last_fullsized":"2024-02-19T21:06:53.224646+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:15.753595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_clean_scrub_stamp":"2024-02-19T21:06:15.753595+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":161,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.219092+0000","last_change":"2024-02-19T21:06:49.305197+0000","last_active":"2024-02-19T21:06:53.219092+0000","last_peered":"2024-02-19T21:06:53.219092+0000","last_clean":"2024-02-19T21:06:53.219092+0000","last_became_active":"2024-02-19T21:04:39.760167+0000","last_became_peered":"2024-02-19T21:04:39.760167+0000","last_unstale":"2024-02-19T21:06:53.219092+0000","last_undegraded":"2024-02-19T21:06:53.219092+0000","last_fullsized":"2024-02-19T21:06:53.219092+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-02-19T21:06:49.305118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.305118+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":197,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215064+0000","last_change":"2024-02-19T21:05:39.892409+0000","last_active":"2024-02-19T21:06:53.215064+0000","last_peered":"2024-02-19T21:06:53.215064+0000","last_clean":"2024-02-19T21:06:53.215064+0000","last_became_active":"2024-02-19T21:01:30.922577+0000","last_became_peered":"2024-02-19T21:01:30.922577+0000","last_unstale":"2024-02-19T21:06:53.215064+0000","last_undegraded":"2024-02-19T21:06:53.215064+0000","last_fullsized":"2024-02-19T21:06:53.215064+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:05:39.892300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:39.892300+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":200,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215574+0000","last_change":"2024-02-19T21:05:01.685058+0000","last_active":"2024-02-19T21:06:53.215574+0000","last_peered":"2024-02-19T21:06:53.215574+0000","last_clean":"2024-02-19T21:06:53.215574+0000","last_became_active":"2024-02-19T21:01:16.271316+0000","last_became_peered":"2024-02-19T21:01:16.271316+0000","last_unstale":"2024-02-19T21:06:53.215574+0000","last_undegraded":"2024-02-19T21:06:53.215574+0000","last_fullsized":"2024-02-19T21:06:53.215574+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.684958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":191,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.213346+0000","last_change":"2024-02-19T21:04:56.999269+0000","last_active":"2024-02-19T21:06:53.213346+0000","last_peered":"2024-02-19T21:06:53.213346+0000","last_clean":"2024-02-19T21:06:53.213346+0000","last_became_active":"2024-02-19T21:01:20.289691+0000","last_became_peered":"2024-02-19T21:01:20.289691+0000","last_unstale":"2024-02-19T21:06:53.213346+0000","last_undegraded":"2024-02-19T21:06:53.213346+0000","last_fullsized":"2024-02-19T21:06:53.213346+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:56.999142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.999142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":26,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.221393+0000","last_change":"2024-02-19T21:06:40.239397+0000","last_active":"2024-02-19T21:06:53.221393+0000","last_peered":"2024-02-19T21:06:53.221393+0000","last_clean":"2024-02-19T21:06:53.221393+0000","last_became_active":"2024-02-19T21:06:40.239199+0000","last_became_peered":"2024-02-19T21:06:40.239199+0000","last_unstale":"2024-02-19T21:06:53.221393+0000","last_undegraded":"2024-02-19T21:06:53.221393+0000","last_fullsized":"2024-02-19T21:06:53.221393+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:12.815725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:12.815725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.217433+0000","last_change":"2024-02-19T21:04:39.674336+0000","last_active":"2024-02-19T21:06:53.217433+0000","last_peered":"2024-02-19T21:06:53.217433+0000","last_clean":"2024-02-19T21:06:53.217433+0000","last_became_active":"2024-02-19T21:04:39.672228+0000","last_became_peered":"2024-02-19T21:04:39.672228+0000","last_unstale":"2024-02-19T21:06:53.217433+0000","last_undegraded":"2024-02-19T21:06:53.217433+0000","last_fullsized":"2024-02-19T21:06:53.217433+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":217,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.207791+0000","last_change":"2024-02-19T21:06:09.595836+0000","last_active":"2024-02-19T21:06:53.207791+0000","last_peered":"2024-02-19T21:06:53.207791+0000","last_clean":"2024-02-19T21:06:53.207791+0000","last_became_active":"2024-02-19T21:01:18.286128+0000","last_became_peered":"2024-02-19T21:01:18.286128+0000","last_unstale":"2024-02-19T21:06:53.207791+0000","last_undegraded":"2024-02-19T21:06:53.207791+0000","last_fullsized":"2024-02-19T21:06:53.207791+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:06:09.595752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.595752+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":200,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.225981+0000","last_change":"2024-02-19T21:05:09.756185+0000","last_active":"2024-02-19T21:06:53.225981+0000","last_peered":"2024-02-19T21:06:53.225981+0000","last_clean":"2024-02-19T21:06:53.225981+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:06:53.225981+0000","last_undegraded":"2024-02-19T21:06:53.225981+0000","last_fullsized":"2024-02-19T21:06:53.225981+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:05:09.756096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224200+0000","last_change":"2024-02-19T21:05:43.904934+0000","last_active":"2024-02-19T21:06:53.224200+0000","last_peered":"2024-02-19T21:06:53.224200+0000","last_clean":"2024-02-19T21:06:53.224200+0000","last_became_active":"2024-02-19T21:01:20.289020+0000","last_became_peered":"2024-02-19T21:01:20.289020+0000","last_unstale":"2024-02-19T21:06:53.224200+0000","last_undegraded":"2024-02-19T21:06:53.224200+0000","last_fullsized":"2024-02-19T21:06:53.224200+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:43.904870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:43.904870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":189,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.204937+0000","last_change":"2024-02-19T21:04:51.014323+0000","last_active":"2024-02-19T21:06:53.204937+0000","last_peered":"2024-02-19T21:06:53.204937+0000","last_clean":"2024-02-19T21:06:53.204937+0000","last_became_active":"2024-02-19T21:01:22.292704+0000","last_became_peered":"2024-02-19T21:01:22.292704+0000","last_unstale":"2024-02-19T21:06:53.204937+0000","last_undegraded":"2024-02-19T21:06:53.204937+0000","last_fullsized":"2024-02-19T21:06:53.204937+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:51.014245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:51.014245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.207225+0000","last_change":"2024-02-19T21:04:39.672911+0000","last_active":"2024-02-19T21:06:53.207225+0000","last_peered":"2024-02-19T21:06:53.207225+0000","last_clean":"2024-02-19T21:06:53.207225+0000","last_became_active":"2024-02-19T21:04:39.672672+0000","last_became_peered":"2024-02-19T21:04:39.672672+0000","last_unstale":"2024-02-19T21:06:53.207225+0000","last_undegraded":"2024-02-19T21:06:53.207225+0000","last_fullsized":"2024-02-19T21:06:53.207225+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":204,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215591+0000","last_change":"2024-02-19T21:05:08.226255+0000","last_active":"2024-02-19T21:06:53.215591+0000","last_peered":"2024-02-19T21:06:53.215591+0000","last_clean":"2024-02-19T21:06:53.215591+0000","last_became_active":"2024-02-19T21:01:18.287874+0000","last_became_peered":"2024-02-19T21:01:18.287874+0000","last_unstale":"2024-02-19T21:06:53.215591+0000","last_undegraded":"2024-02-19T21:06:53.215591+0000","last_fullsized":"2024-02-19T21:06:53.215591+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:05:08.226144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:08.226144+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":200,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.211071+0000","last_change":"2024-02-19T21:06:24.838727+0000","last_active":"2024-02-19T21:06:53.211071+0000","last_peered":"2024-02-19T21:06:53.211071+0000","last_clean":"2024-02-19T21:06:53.211071+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:06:53.211071+0000","last_undegraded":"2024-02-19T21:06:53.211071+0000","last_fullsized":"2024-02-19T21:06:53.211071+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:24.838658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:24.838658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.220779+0000","last_change":"2024-02-19T21:05:40.919364+0000","last_active":"2024-02-19T21:06:53.220779+0000","last_peered":"2024-02-19T21:06:53.220779+0000","last_clean":"2024-02-19T21:06:53.220779+0000","last_became_active":"2024-02-19T21:01:20.290962+0000","last_became_peered":"2024-02-19T21:01:20.290962+0000","last_unstale":"2024-02-19T21:06:53.220779+0000","last_undegraded":"2024-02-19T21:06:53.220779+0000","last_fullsized":"2024-02-19T21:06:53.220779+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:40.919263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:40.919263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":193,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.223002+0000","last_change":"2024-02-19T21:05:44.331615+0000","last_active":"2024-02-19T21:06:53.223002+0000","last_peered":"2024-02-19T21:06:53.223002+0000","last_clean":"2024-02-19T21:06:53.223002+0000","last_became_active":"2024-02-19T21:01:22.297693+0000","last_became_peered":"2024-02-19T21:01:22.297693+0000","last_unstale":"2024-02-19T21:06:53.223002+0000","last_undegraded":"2024-02-19T21:06:53.223002+0000","last_fullsized":"2024-02-19T21:06:53.223002+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.331546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.331546+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216512+0000","last_change":"2024-02-19T21:04:39.717968+0000","last_active":"2024-02-19T21:06:53.216512+0000","last_peered":"2024-02-19T21:06:53.216512+0000","last_clean":"2024-02-19T21:06:53.216512+0000","last_became_active":"2024-02-19T21:04:39.716606+0000","last_became_peered":"2024-02-19T21:04:39.716606+0000","last_unstale":"2024-02-19T21:06:53.216512+0000","last_undegraded":"2024-02-19T21:06:53.216512+0000","last_fullsized":"2024-02-19T21:06:53.216512+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":212,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.205360+0000","last_change":"2024-02-19T21:04:52.027084+0000","last_active":"2024-02-19T21:06:53.205360+0000","last_peered":"2024-02-19T21:06:53.205360+0000","last_clean":"2024-02-19T21:06:53.205360+0000","last_became_active":"2024-02-19T21:01:18.293793+0000","last_became_peered":"2024-02-19T21:01:18.293793+0000","last_unstale":"2024-02-19T21:06:53.205360+0000","last_undegraded":"2024-02-19T21:06:53.205360+0000","last_fullsized":"2024-02-19T21:06:53.205360+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:04:52.027000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:52.027000+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":200,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.209772+0000","last_change":"2024-02-19T21:06:11.887212+0000","last_active":"2024-02-19T21:06:53.209772+0000","last_peered":"2024-02-19T21:06:53.209772+0000","last_clean":"2024-02-19T21:06:53.209772+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:06:53.209772+0000","last_undegraded":"2024-02-19T21:06:53.209772+0000","last_fullsized":"2024-02-19T21:06:53.209772+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:11.887139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:11.887139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":190,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.219798+0000","last_change":"2024-02-19T21:05:43.361840+0000","last_active":"2024-02-19T21:06:53.219798+0000","last_peered":"2024-02-19T21:06:53.219798+0000","last_clean":"2024-02-19T21:06:53.219798+0000","last_became_active":"2024-02-19T21:01:20.287961+0000","last_became_peered":"2024-02-19T21:01:20.287961+0000","last_unstale":"2024-02-19T21:06:53.219798+0000","last_undegraded":"2024-02-19T21:06:53.219798+0000","last_fullsized":"2024-02-19T21:06:53.219798+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:43.361777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:43.361777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":194,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.213576+0000","last_change":"2024-02-19T21:06:08.757280+0000","last_active":"2024-02-19T21:06:53.213576+0000","last_peered":"2024-02-19T21:06:53.213576+0000","last_clean":"2024-02-19T21:06:53.213576+0000","last_became_active":"2024-02-19T21:01:22.608997+0000","last_became_peered":"2024-02-19T21:01:22.608997+0000","last_unstale":"2024-02-19T21:06:53.213576+0000","last_undegraded":"2024-02-19T21:06:53.213576+0000","last_fullsized":"2024-02-19T21:06:53.213576+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:08.757198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:08.757198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":31,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-19T21:06:54.664879+0000","last_change":"2024-02-19T21:06:54.664879+0000","last_active":"2024-02-19T21:06:54.664879+0000","last_peered":"2024-02-19T21:06:54.664879+0000","last_clean":"2024-02-19T21:06:54.664879+0000","last_became_active":"2024-02-19T21:06:40.226196+0000","last_became_peered":"2024-02-19T21:06:40.226196+0000","last_unstale":"2024-02-19T21:06:54.664879+0000","last_undegraded":"2024-02-19T21:06:54.664879+0000","last_fullsized":"2024-02-19T21:06:54.664879+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.664814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":53,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.226520+0000","last_change":"2024-02-19T21:06:40.234278+0000","last_active":"2024-02-19T21:06:53.226520+0000","last_peered":"2024-02-19T21:06:53.226520+0000","last_clean":"2024-02-19T21:06:53.226520+0000","last_became_active":"2024-02-19T21:06:40.233941+0000","last_became_peered":"2024-02-19T21:06:40.233941+0000","last_unstale":"2024-02-19T21:06:53.226520+0000","last_undegraded":"2024-02-19T21:06:53.226520+0000","last_fullsized":"2024-02-19T21:06:53.226520+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:05:47.805757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:47.805757+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":195,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.221304+0000","last_change":"2024-02-19T21:05:07.118727+0000","last_active":"2024-02-19T21:06:53.221304+0000","last_peered":"2024-02-19T21:06:53.221304+0000","last_clean":"2024-02-19T21:06:53.221304+0000","last_became_active":"2024-02-19T21:01:16.272284+0000","last_became_peered":"2024-02-19T21:01:16.272284+0000","last_unstale":"2024-02-19T21:06:53.221304+0000","last_undegraded":"2024-02-19T21:06:53.221304+0000","last_fullsized":"2024-02-19T21:06:53.221304+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:07.118667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:07.118667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":275,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-19T21:06:54.194544+0000","last_change":"2024-02-19T21:05:41.909864+0000","last_active":"2024-02-19T21:06:54.194544+0000","last_peered":"2024-02-19T21:06:54.194544+0000","last_clean":"2024-02-19T21:06:54.194544+0000","last_became_active":"2024-02-19T21:01:20.290567+0000","last_became_peered":"2024-02-19T21:01:20.290567+0000","last_unstale":"2024-02-19T21:06:54.194544+0000","last_undegraded":"2024-02-19T21:06:54.194544+0000","last_fullsized":"2024-02-19T21:06:54.194544+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:05:41.909796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:41.909796+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":195,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.225202+0000","last_change":"2024-02-19T21:06:27.520296+0000","last_active":"2024-02-19T21:06:53.225202+0000","last_peered":"2024-02-19T21:06:53.225202+0000","last_clean":"2024-02-19T21:06:53.225202+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:06:53.225202+0000","last_undegraded":"2024-02-19T21:06:53.225202+0000","last_fullsized":"2024-02-19T21:06:53.225202+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:06:27.520213+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:06:27.520213+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.212352+0000","last_change":"2024-02-19T21:04:39.700717+0000","last_active":"2024-02-19T21:06:53.212352+0000","last_peered":"2024-02-19T21:06:53.212352+0000","last_clean":"2024-02-19T21:06:53.212352+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:06:53.212352+0000","last_undegraded":"2024-02-19T21:06:53.212352+0000","last_fullsized":"2024-02-19T21:06:53.212352+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":217,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.210452+0000","last_change":"2024-02-19T21:05:07.770793+0000","last_active":"2024-02-19T21:06:53.210452+0000","last_peered":"2024-02-19T21:06:53.210452+0000","last_clean":"2024-02-19T21:06:53.210452+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:06:53.210452+0000","last_undegraded":"2024-02-19T21:06:53.210452+0000","last_fullsized":"2024-02-19T21:06:53.210452+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:05:07.770727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:07.770727+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":195,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.205122+0000","last_change":"2024-02-19T21:04:52.990442+0000","last_active":"2024-02-19T21:06:53.205122+0000","last_peered":"2024-02-19T21:06:53.205122+0000","last_clean":"2024-02-19T21:06:53.205122+0000","last_became_active":"2024-02-19T21:01:16.277826+0000","last_became_peered":"2024-02-19T21:01:16.277826+0000","last_unstale":"2024-02-19T21:06:53.205122+0000","last_undegraded":"2024-02-19T21:06:53.205122+0000","last_fullsized":"2024-02-19T21:06:53.205122+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:52.990315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:52.990315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":26,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.226683+0000","last_change":"2024-02-19T21:06:40.234367+0000","last_active":"2024-02-19T21:06:53.226683+0000","last_peered":"2024-02-19T21:06:53.226683+0000","last_clean":"2024-02-19T21:06:53.226683+0000","last_became_active":"2024-02-19T21:06:40.234026+0000","last_became_peered":"2024-02-19T21:06:40.234026+0000","last_unstale":"2024-02-19T21:06:53.226683+0000","last_undegraded":"2024-02-19T21:06:53.226683+0000","last_fullsized":"2024-02-19T21:06:53.226683+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:54.322192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:54.322192+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":189,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.225831+0000","last_change":"2024-02-19T21:05:08.825596+0000","last_active":"2024-02-19T21:06:53.225831+0000","last_peered":"2024-02-19T21:06:53.225831+0000","last_clean":"2024-02-19T21:06:53.225831+0000","last_became_active":"2024-02-19T21:01:22.293230+0000","last_became_peered":"2024-02-19T21:01:22.293230+0000","last_unstale":"2024-02-19T21:06:53.225831+0000","last_undegraded":"2024-02-19T21:06:53.225831+0000","last_fullsized":"2024-02-19T21:06:53.225831+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:08.825485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:08.825485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.228713+0000","last_change":"2024-02-19T21:04:39.801737+0000","last_active":"2024-02-19T21:06:53.228713+0000","last_peered":"2024-02-19T21:06:53.228713+0000","last_clean":"2024-02-19T21:06:53.228713+0000","last_became_active":"2024-02-19T21:04:39.797018+0000","last_became_peered":"2024-02-19T21:04:39.797018+0000","last_unstale":"2024-02-19T21:06:53.228713+0000","last_undegraded":"2024-02-19T21:06:53.228713+0000","last_fullsized":"2024-02-19T21:06:53.228713+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":223,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.221019+0000","last_change":"2024-02-19T21:06:48.310557+0000","last_active":"2024-02-19T21:06:53.221019+0000","last_peered":"2024-02-19T21:06:53.221019+0000","last_clean":"2024-02-19T21:06:53.221019+0000","last_became_active":"2024-02-19T21:01:18.289712+0000","last_became_peered":"2024-02-19T21:01:18.289712+0000","last_unstale":"2024-02-19T21:06:53.221019+0000","last_undegraded":"2024-02-19T21:06:53.221019+0000","last_fullsized":"2024-02-19T21:06:53.221019+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:06:48.310459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.310459+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":200,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215940+0000","last_change":"2024-02-19T21:06:10.922977+0000","last_active":"2024-02-19T21:06:53.215940+0000","last_peered":"2024-02-19T21:06:53.215940+0000","last_clean":"2024-02-19T21:06:53.215940+0000","last_became_active":"2024-02-19T21:01:16.274524+0000","last_became_peered":"2024-02-19T21:01:16.274524+0000","last_unstale":"2024-02-19T21:06:53.215940+0000","last_undegraded":"2024-02-19T21:06:53.215940+0000","last_fullsized":"2024-02-19T21:06:53.215940+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:10.922913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:10.922913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.629562+0000","last_change":"2024-02-19T21:06:53.629540+0000","last_active":"2024-02-19T21:06:53.629562+0000","last_peered":"2024-02-19T21:06:53.629562+0000","last_clean":"2024-02-19T21:06:53.629562+0000","last_became_active":"2024-02-19T21:01:20.290705+0000","last_became_peered":"2024-02-19T21:01:20.290705+0000","last_unstale":"2024-02-19T21:06:53.629562+0000","last_undegraded":"2024-02-19T21:06:53.629562+0000","last_fullsized":"2024-02-19T21:06:53.629562+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:53.629475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:53.629475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":189,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.219229+0000","last_change":"2024-02-19T21:05:37.572761+0000","last_active":"2024-02-19T21:06:53.219229+0000","last_peered":"2024-02-19T21:06:53.219229+0000","last_clean":"2024-02-19T21:06:53.219229+0000","last_became_active":"2024-02-19T21:01:22.607435+0000","last_became_peered":"2024-02-19T21:01:22.607435+0000","last_unstale":"2024-02-19T21:06:53.219229+0000","last_undegraded":"2024-02-19T21:06:53.219229+0000","last_fullsized":"2024-02-19T21:06:53.219229+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:37.572695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:05:37.572695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":26,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.208775+0000","last_change":"2024-02-19T21:06:40.241189+0000","last_active":"2024-02-19T21:06:53.208775+0000","last_peered":"2024-02-19T21:06:53.208775+0000","last_clean":"2024-02-19T21:06:53.208775+0000","last_became_active":"2024-02-19T21:06:40.240859+0000","last_became_peered":"2024-02-19T21:06:40.240859+0000","last_unstale":"2024-02-19T21:06:53.208775+0000","last_undegraded":"2024-02-19T21:06:53.208775+0000","last_fullsized":"2024-02-19T21:06:53.208775+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.797430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.797430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":216,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.210729+0000","last_change":"2024-02-19T21:04:50.027760+0000","last_active":"2024-02-19T21:06:53.210729+0000","last_peered":"2024-02-19T21:06:53.210729+0000","last_clean":"2024-02-19T21:06:53.210729+0000","last_became_active":"2024-02-19T21:01:18.285926+0000","last_became_peered":"2024-02-19T21:01:18.285926+0000","last_unstale":"2024-02-19T21:06:53.210729+0000","last_undegraded":"2024-02-19T21:06:53.210729+0000","last_fullsized":"2024-02-19T21:06:53.210729+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:04:50.027690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:50.027690+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":195,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.213271+0000","last_change":"2024-02-19T21:06:02.693096+0000","last_active":"2024-02-19T21:06:53.213271+0000","last_peered":"2024-02-19T21:06:53.213271+0000","last_clean":"2024-02-19T21:06:53.213271+0000","last_became_active":"2024-02-19T21:01:16.274811+0000","last_became_peered":"2024-02-19T21:01:16.274811+0000","last_unstale":"2024-02-19T21:06:53.213271+0000","last_undegraded":"2024-02-19T21:06:53.213271+0000","last_fullsized":"2024-02-19T21:06:53.213271+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:02.693025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:02.693025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":37,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224426+0000","last_change":"2024-02-19T21:06:27.201481+0000","last_active":"2024-02-19T21:06:53.224426+0000","last_peered":"2024-02-19T21:06:53.224426+0000","last_clean":"2024-02-19T21:06:53.224426+0000","last_became_active":"2024-02-19T21:06:27.201124+0000","last_became_peered":"2024-02-19T21:06:27.201124+0000","last_unstale":"2024-02-19T21:06:53.224426+0000","last_undegraded":"2024-02-19T21:06:53.224426+0000","last_fullsized":"2024-02-19T21:06:53.224426+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:38.564472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:38.564472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":194,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-19T21:06:54.723259+0000","last_change":"2024-02-19T21:06:54.723259+0000","last_active":"2024-02-19T21:06:54.723259+0000","last_peered":"2024-02-19T21:06:54.723259+0000","last_clean":"2024-02-19T21:06:54.723259+0000","last_became_active":"2024-02-19T21:01:22.607099+0000","last_became_peered":"2024-02-19T21:01:22.607099+0000","last_unstale":"2024-02-19T21:06:54.723259+0000","last_undegraded":"2024-02-19T21:06:54.723259+0000","last_fullsized":"2024-02-19T21:06:54.723259+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.723156+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224335+0000","last_change":"2024-02-19T21:04:39.781223+0000","last_active":"2024-02-19T21:06:53.224335+0000","last_peered":"2024-02-19T21:06:53.224335+0000","last_clean":"2024-02-19T21:06:53.224335+0000","last_became_active":"2024-02-19T21:04:39.777442+0000","last_became_peered":"2024-02-19T21:04:39.777442+0000","last_unstale":"2024-02-19T21:06:53.224335+0000","last_undegraded":"2024-02-19T21:06:53.224335+0000","last_fullsized":"2024-02-19T21:06:53.224335+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"194'13","reported_seq":218,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.214102+0000","last_change":"2024-02-19T21:06:14.772493+0000","last_active":"2024-02-19T21:06:53.214102+0000","last_peered":"2024-02-19T21:06:53.214102+0000","last_clean":"2024-02-19T21:06:53.214102+0000","last_became_active":"2024-02-19T21:01:18.285269+0000","last_became_peered":"2024-02-19T21:01:18.285269+0000","last_unstale":"2024-02-19T21:06:53.214102+0000","last_undegraded":"2024-02-19T21:06:53.214102+0000","last_fullsized":"2024-02-19T21:06:53.214102+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'13","last_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.772417+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":194,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224737+0000","last_change":"2024-02-19T21:04:59.262530+0000","last_active":"2024-02-19T21:06:53.224737+0000","last_peered":"2024-02-19T21:06:53.224737+0000","last_clean":"2024-02-19T21:06:53.224737+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:06:53.224737+0000","last_undegraded":"2024-02-19T21:06:53.224737+0000","last_fullsized":"2024-02-19T21:06:53.224737+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:59.262400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:59.262400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":191,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.215463+0000","last_change":"2024-02-19T21:06:09.853068+0000","last_active":"2024-02-19T21:06:53.215463+0000","last_peered":"2024-02-19T21:06:53.215463+0000","last_clean":"2024-02-19T21:06:53.215463+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:06:53.215463+0000","last_undegraded":"2024-02-19T21:06:53.215463+0000","last_fullsized":"2024-02-19T21:06:53.215463+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:09.852996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.852996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":26,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.223138+0000","last_change":"2024-02-19T21:06:40.239902+0000","last_active":"2024-02-19T21:06:53.223138+0000","last_peered":"2024-02-19T21:06:53.223138+0000","last_clean":"2024-02-19T21:06:53.223138+0000","last_became_active":"2024-02-19T21:06:40.239603+0000","last_became_peered":"2024-02-19T21:06:40.239603+0000","last_unstale":"2024-02-19T21:06:53.223138+0000","last_undegraded":"2024-02-19T21:06:53.223138+0000","last_fullsized":"2024-02-19T21:06:53.223138+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:06:25.785366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216151+0000","last_change":"2024-02-19T21:04:39.737500+0000","last_active":"2024-02-19T21:06:53.216151+0000","last_peered":"2024-02-19T21:06:53.216151+0000","last_clean":"2024-02-19T21:06:53.216151+0000","last_became_active":"2024-02-19T21:04:39.736990+0000","last_became_peered":"2024-02-19T21:04:39.736990+0000","last_unstale":"2024-02-19T21:06:53.216151+0000","last_undegraded":"2024-02-19T21:06:53.216151+0000","last_fullsized":"2024-02-19T21:06:53.216151+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.207634+0000","last_change":"2024-02-19T21:06:16.718025+0000","last_active":"2024-02-19T21:06:53.207634+0000","last_peered":"2024-02-19T21:06:53.207634+0000","last_clean":"2024-02-19T21:06:53.207634+0000","last_became_active":"2024-02-19T21:01:18.285735+0000","last_became_peered":"2024-02-19T21:01:18.285735+0000","last_unstale":"2024-02-19T21:06:53.207634+0000","last_undegraded":"2024-02-19T21:06:53.207634+0000","last_fullsized":"2024-02-19T21:06:53.207634+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:06:16.717940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.717940+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":199,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.227251+0000","last_change":"2024-02-19T21:05:36.352651+0000","last_active":"2024-02-19T21:06:53.227251+0000","last_peered":"2024-02-19T21:06:53.227251+0000","last_clean":"2024-02-19T21:06:53.227251+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:06:53.227251+0000","last_undegraded":"2024-02-19T21:06:53.227251+0000","last_fullsized":"2024-02-19T21:06:53.227251+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:36.352580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:36.352580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":195,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.223163+0000","last_change":"2024-02-19T21:06:14.447006+0000","last_active":"2024-02-19T21:06:53.223163+0000","last_peered":"2024-02-19T21:06:53.223163+0000","last_clean":"2024-02-19T21:06:53.223163+0000","last_became_active":"2024-02-19T21:01:20.287940+0000","last_became_peered":"2024-02-19T21:01:20.287940+0000","last_unstale":"2024-02-19T21:06:53.223163+0000","last_undegraded":"2024-02-19T21:06:53.223163+0000","last_fullsized":"2024-02-19T21:06:53.223163+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:14.446932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.446932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":194,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.210045+0000","last_change":"2024-02-19T21:06:48.507033+0000","last_active":"2024-02-19T21:06:53.210045+0000","last_peered":"2024-02-19T21:06:53.210045+0000","last_clean":"2024-02-19T21:06:53.210045+0000","last_became_active":"2024-02-19T21:01:22.293282+0000","last_became_peered":"2024-02-19T21:01:22.293282+0000","last_unstale":"2024-02-19T21:06:53.210045+0000","last_undegraded":"2024-02-19T21:06:53.210045+0000","last_fullsized":"2024-02-19T21:06:53.210045+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:48.506964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.506964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":31,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.217384+0000","last_change":"2024-02-19T21:06:50.467486+0000","last_active":"2024-02-19T21:06:53.217384+0000","last_peered":"2024-02-19T21:06:53.217384+0000","last_clean":"2024-02-19T21:06:53.217384+0000","last_became_active":"2024-02-19T21:06:40.227713+0000","last_became_peered":"2024-02-19T21:06:40.227713+0000","last_unstale":"2024-02-19T21:06:53.217384+0000","last_undegraded":"2024-02-19T21:06:53.217384+0000","last_fullsized":"2024-02-19T21:06:53.217384+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:50.467411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:50.467411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":209,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.207219+0000","last_change":"2024-02-19T21:05:36.527512+0000","last_active":"2024-02-19T21:06:53.207219+0000","last_peered":"2024-02-19T21:06:53.207219+0000","last_clean":"2024-02-19T21:06:53.207219+0000","last_became_active":"2024-02-19T21:01:18.287645+0000","last_became_peered":"2024-02-19T21:01:18.287645+0000","last_unstale":"2024-02-19T21:06:53.207219+0000","last_undegraded":"2024-02-19T21:06:53.207219+0000","last_fullsized":"2024-02-19T21:06:53.207219+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:05:36.527441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:36.527441+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":200,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.752681+0000","last_change":"2024-02-19T21:06:53.752645+0000","last_active":"2024-02-19T21:06:53.752681+0000","last_peered":"2024-02-19T21:06:53.752681+0000","last_clean":"2024-02-19T21:06:53.752681+0000","last_became_active":"2024-02-19T21:01:16.272132+0000","last_became_peered":"2024-02-19T21:01:16.272132+0000","last_unstale":"2024-02-19T21:06:53.752681+0000","last_undegraded":"2024-02-19T21:06:53.752681+0000","last_fullsized":"2024-02-19T21:06:53.752681+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:53.752543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:53.752543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":196,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.224178+0000","last_change":"2024-02-19T21:06:24.702277+0000","last_active":"2024-02-19T21:06:53.224178+0000","last_peered":"2024-02-19T21:06:53.224178+0000","last_clean":"2024-02-19T21:06:53.224178+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:06:53.224178+0000","last_undegraded":"2024-02-19T21:06:53.224178+0000","last_fullsized":"2024-02-19T21:06:53.224178+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:06:24.702197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":194,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.213984+0000","last_change":"2024-02-19T21:06:04.601048+0000","last_active":"2024-02-19T21:06:53.213984+0000","last_peered":"2024-02-19T21:06:53.213984+0000","last_clean":"2024-02-19T21:06:53.213984+0000","last_became_active":"2024-02-19T21:01:22.293432+0000","last_became_peered":"2024-02-19T21:01:22.293432+0000","last_unstale":"2024-02-19T21:06:53.213984+0000","last_undegraded":"2024-02-19T21:06:53.213984+0000","last_fullsized":"2024-02-19T21:06:53.213984+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:04.600973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:04.600973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":143,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.221697+0000","last_change":"2024-02-19T21:04:39.662781+0000","last_active":"2024-02-19T21:06:53.221697+0000","last_peered":"2024-02-19T21:06:53.221697+0000","last_clean":"2024-02-19T21:06:53.221697+0000","last_became_active":"2024-02-19T21:04:39.661810+0000","last_became_peered":"2024-02-19T21:04:39.661810+0000","last_unstale":"2024-02-19T21:06:53.221697+0000","last_undegraded":"2024-02-19T21:06:53.221697+0000","last_fullsized":"2024-02-19T21:06:53.221697+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":223,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.205671+0000","last_change":"2024-02-19T21:06:09.763553+0000","last_active":"2024-02-19T21:06:53.205671+0000","last_peered":"2024-02-19T21:06:53.205671+0000","last_clean":"2024-02-19T21:06:53.205671+0000","last_became_active":"2024-02-19T21:01:18.283222+0000","last_became_peered":"2024-02-19T21:01:18.283222+0000","last_unstale":"2024-02-19T21:06:53.205671+0000","last_undegraded":"2024-02-19T21:06:53.205671+0000","last_fullsized":"2024-02-19T21:06:53.205671+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:06:09.763482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.763482+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":26,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.230130+0000","last_change":"2024-02-19T21:06:40.228214+0000","last_active":"2024-02-19T21:06:53.230130+0000","last_peered":"2024-02-19T21:06:53.230130+0000","last_clean":"2024-02-19T21:06:53.230130+0000","last_became_active":"2024-02-19T21:06:40.227887+0000","last_became_peered":"2024-02-19T21:06:40.227887+0000","last_unstale":"2024-02-19T21:06:53.230130+0000","last_undegraded":"2024-02-19T21:06:53.230130+0000","last_fullsized":"2024-02-19T21:06:53.230130+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:08.599008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:08.599008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":191,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.208852+0000","last_change":"2024-02-19T21:05:41.532900+0000","last_active":"2024-02-19T21:06:53.208852+0000","last_peered":"2024-02-19T21:06:53.208852+0000","last_clean":"2024-02-19T21:06:53.208852+0000","last_became_active":"2024-02-19T21:01:20.289208+0000","last_became_peered":"2024-02-19T21:01:20.289208+0000","last_unstale":"2024-02-19T21:06:53.208852+0000","last_undegraded":"2024-02-19T21:06:53.208852+0000","last_fullsized":"2024-02-19T21:06:53.208852+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:41.532841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:41.532841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":189,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.213069+0000","last_change":"2024-02-19T21:05:36.920186+0000","last_active":"2024-02-19T21:06:53.213069+0000","last_peered":"2024-02-19T21:06:53.213069+0000","last_clean":"2024-02-19T21:06:53.213069+0000","last_became_active":"2024-02-19T21:01:22.609059+0000","last_became_peered":"2024-02-19T21:01:22.609059+0000","last_unstale":"2024-02-19T21:06:53.213069+0000","last_undegraded":"2024-02-19T21:06:53.213069+0000","last_fullsized":"2024-02-19T21:06:53.213069+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:36.920115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:36.920115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":148,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-02-19T21:06:53.216555+0000","last_change":"2024-02-19T21:06:27.744913+0000","last_active":"2024-02-19T21:06:53.216555+0000","last_peered":"2024-02-19T21:06:53.216555+0000","last_clean":"2024-02-19T21:06:53.216555+0000","last_became_active":"2024-02-19T21:04:39.726195+0000","last_became_peered":"2024-02-19T21:04:39.726195+0000","last_unstale":"2024-02-19T21:06:53.216555+0000","last_undegraded":"2024-02-19T21:06:53.216555+0000","last_fullsized":"2024-02-19T21:06:53.216555+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:27.744835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:27.744835+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":161,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":157,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":155,"num_pg":32,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":1199,"num_objects":116,"num_object_clones":3,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116,"num_whiteouts":0,"num_read":92,"num_read_kb":1,"num_write":526,"num_write_kb":523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413120,"data_stored":3597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1747,"ondisk_log_size":1747,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":692,"num_read_kb":449,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":331,"num_read_kb":326,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":126,"acting":126,"num_store_stats":8},{"poolid":156,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":8,"num_pg":13,"stat_sum":{"num_bytes":1792,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116,"num_write_kb":65,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":39,"acting":39,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266758,"num_pgs":139,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313244,"kb_used_data":3724,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431908,"statfs":{"total":95995035648,"available":95674273792,"internally_reserved":0,"allocated":3813376,"data_stored":2096333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":5.3840000000000003,"5min":1.6120000000000001,"15min":0.877},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":89.456999999999994,"5min":89.456999999999994,"15min":89.456999999999994},"last":0.72199999999999998},{"interface":"front","average":{"1min":4.3209999999999997,"5min":1.387,"15min":0.78900000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":70.451999999999998,"5min":70.451999999999998,"15min":70.451999999999998},"last":0.60499999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":2.3599999999999999,"5min":1.087,"15min":0.71099999999999997},"min":{"1min":0.29299999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":34.273000000000003,"5min":34.273000000000003,"15min":34.273000000000003},"last":0.69099999999999995},{"interface":"front","average":{"1min":5.3840000000000003,"5min":1.8120000000000001,"15min":0.95199999999999996},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":89.594999999999999,"5min":89.594999999999999,"15min":89.594999999999999},"last":0.64100000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":6.2359999999999998,"5min":1.8440000000000001,"15min":0.94799999999999995},"min":{"1min":0.27600000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":104.413,"5min":104.413,"15min":104.413},"last":0.70899999999999996},{"interface":"front","average":{"1min":6.2539999999999996,"5min":1.827,"15min":0.95599999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":104.268,"5min":104.268,"15min":104.268},"last":0.67800000000000005}]},{"osd":3,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":6.2930000000000001,"5min":1.8089999999999999,"15min":0.95799999999999996},"min":{"1min":0.32200000000000001,"5min":0.254,"15min":0.254},"max":{"1min":105.3,"5min":105.3,"15min":105.3},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.1899999999999999,"5min":0.79400000000000004,"15min":0.59399999999999997},"min":{"1min":0.32700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":13.243,"5min":13.243,"15min":13.243},"last":0.749}]},{"osd":4,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.45300000000000001,"15min":0.48699999999999999},"min":{"1min":0.27800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.54200000000000004,"5min":1.373,"15min":1.373},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.47199999999999998,"15min":0.48199999999999998},"min":{"1min":0.29199999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55800000000000005,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.46400000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.45200000000000001,"15min":0.45800000000000002},"min":{"1min":0.29399999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.50700000000000001,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.503,"15min":0.50900000000000001},"min":{"1min":0.33800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.59499999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.58499999999999996}]},{"osd":6,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.50900000000000001,"15min":0.53100000000000003},"min":{"1min":0.32800000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60399999999999998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.54100000000000004,"15min":0.56699999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64200000000000002,"5min":1.262,"15min":1.262},"last":0.61799999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528396,"num_pgs":147,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309168,"kb_used_data":3748,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435984,"statfs":{"total":95995035648,"available":95678447616,"internally_reserved":0,"allocated":3837952,"data_stored":2096511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:05:55 2024","interfaces":[{"interface":"back","average":{"1min":1.99,"5min":0.85699999999999998,"15min":0.61499999999999999},"min":{"1min":0.27800000000000002,"5min":0.222,"15min":0.222},"max":{"1min":17.486999999999998,"5min":17.486999999999998,"15min":17.486999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":2.0030000000000001,"5min":0.94499999999999995,"15min":0.64100000000000001},"min":{"1min":0.26300000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":17.43,"5min":17.43,"15min":17.43},"last":0.52400000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:05:55 2024","interfaces":[{"interface":"back","average":{"1min":1.7390000000000001,"5min":0.85999999999999999,"15min":0.61399999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":14.692,"5min":14.692,"15min":14.692},"last":0.53600000000000003},{"interface":"front","average":{"1min":2.6459999999999999,"5min":0.98999999999999999,"15min":0.66300000000000003},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":14.795,"5min":14.795,"15min":14.795},"last":0.5}]},{"osd":2,"last update":"Mon Feb 19 21:05:55 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":0.78400000000000003,"15min":0.60799999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":6.931,"5min":6.931,"15min":6.931},"last":0.56000000000000005},{"interface":"front","average":{"1min":2.681,"5min":1.032,"15min":0.67000000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":14.391,"5min":14.391,"15min":14.391},"last":0.51900000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:05:55 2024","interfaces":[{"interface":"back","average":{"1min":2.4510000000000001,"5min":1.046,"15min":0.68999999999999995},"min":{"1min":0.35299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":14.510999999999999,"5min":14.510999999999999,"15min":14.510999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":2.355,"5min":0.93999999999999995,"15min":0.64200000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":17.585000000000001,"5min":17.585000000000001,"15min":17.585000000000001},"last":0.45500000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.41199999999999998,"15min":0.38200000000000001},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.54200000000000004,"5min":1.363,"15min":1.363},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.43099999999999999,"15min":0.40899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.53600000000000003,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.32400000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.45200000000000001,"15min":0.41899999999999998},"min":{"1min":0.26900000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.56999999999999995,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.48499999999999999,"15min":0.45500000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55100000000000005,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.36499999999999999}]},{"osd":7,"last update":"Mon Feb 19 21:06:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.48399999999999999,"15min":0.50800000000000001},"min":{"1min":0.28699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70599999999999996,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48799999999999999,"15min":0.52000000000000002},"min":{"1min":0.30099999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.69099999999999995,"5min":1.534,"15min":1.534},"last":0.374}]}]},{"osd":1,"up_from":13,"seq":55834574947,"num_pgs":147,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309180,"kb_used_data":3764,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435972,"statfs":{"total":95995035648,"available":95678435328,"internally_reserved":0,"allocated":3854336,"data_stored":2096912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:06:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":0.96199999999999997,"15min":0.55200000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":11.269,"5min":27.777000000000001,"15min":27.777000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.0129999999999999,"15min":0.57599999999999996},"min":{"1min":0.24199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":8.843,"5min":27.631,"15min":27.631},"last":0.255}]},{"osd":2,"last update":"Mon Feb 19 21:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":1.349,"15min":0.72299999999999998},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.56200000000000006,"5min":27.798999999999999,"15min":27.798999999999999},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.371,"5min":1.369,"15min":0.73699999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.627,"5min":27.606999999999999,"15min":27.606999999999999},"last":0.30499999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.91800000000000004,"15min":0.59099999999999997},"min":{"1min":0.254,"5min":0.218,"15min":0.218},"max":{"1min":0.49399999999999999,"5min":15.574999999999999,"15min":15.574999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":1.129,"15min":0.68700000000000006},"min":{"1min":0.318,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.51900000000000002,"5min":27.82,"15min":27.82},"last":0.29499999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:06:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":0.89800000000000002,"15min":0.67200000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":9.6430000000000007,"5min":9.6430000000000007,"15min":9.6430000000000007},"last":0.42499999999999999},{"interface":"front","average":{"1min":1.333,"5min":1.1659999999999999,"15min":0.76300000000000001},"min":{"1min":0.36399999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":11.119,"5min":27.707000000000001,"15min":27.707000000000001},"last":0.49099999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.121,"15min":0.76500000000000001},"min":{"1min":0.35499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.80600000000000005,"5min":27.529,"15min":27.529},"last":0.443},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.91700000000000004,"15min":0.71699999999999997},"min":{"1min":0.41099999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.84199999999999997,"5min":9.6799999999999997,"15min":9.6799999999999997},"last":0.52100000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.88100000000000001,"15min":0.68700000000000006},"min":{"1min":0.45700000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78800000000000003,"5min":9.6989999999999998,"15min":9.6989999999999998},"last":0.498},{"interface":"front","average":{"1min":0.629,"5min":1.1359999999999999,"15min":0.81200000000000006},"min":{"1min":0.38900000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.874,"5min":27.733000000000001,"15min":27.733000000000001},"last":0.435}]},{"osd":7,"last update":"Mon Feb 19 21:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.179,"15min":0.78400000000000003},"min":{"1min":0.39300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85199999999999998,"5min":27.576000000000001,"15min":27.576000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.238,"15min":0.81399999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.73299999999999998,"5min":27.756,"15min":27.756},"last":0.50800000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309100,"kb_used_data":3672,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436052,"statfs":{"total":95995035648,"available":95678517248,"internally_reserved":0,"allocated":3760128,"data_stored":2095097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:06:03 2024","interfaces":[{"interface":"back","average":{"1min":2.657,"5min":1.1499999999999999,"15min":0.58199999999999996},"min":{"1min":0.188,"5min":0.188,"15min":0.16200000000000001},"max":{"1min":20.966999999999999,"5min":20.966999999999999,"15min":20.966999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":2.3199999999999998,"5min":1.127,"15min":0.57899999999999996},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.188},"max":{"1min":20.366,"5min":20.366,"15min":20.366},"last":0.34699999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.3200000000000001,"15min":0.68899999999999995},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.0890000000000004,"5min":27.916,"15min":27.916},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":1.083,"15min":0.61199999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.59599999999999997,"5min":22.004000000000001,"15min":22.004000000000001},"last":0.38700000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":1.2989999999999999,"15min":0.74199999999999999},"min":{"1min":0.248,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.54000000000000004,"5min":25.669,"15min":25.669},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":1.1040000000000001,"15min":0.68400000000000005},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.50700000000000001,"5min":25.006,"15min":25.006},"last":0.36499999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:06:03 2024","interfaces":[{"interface":"back","average":{"1min":2.1680000000000001,"5min":1.097,"15min":0.70899999999999996},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":20.411999999999999,"5min":20.411999999999999,"15min":20.411999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":2.9369999999999998,"5min":1.121,"15min":0.72899999999999998},"min":{"1min":0.32600000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":19.986999999999998,"5min":19.986999999999998,"15min":19.986999999999998},"last":0.72399999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":1.202,"15min":0.83199999999999996},"min":{"1min":0.36299999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.83399999999999996,"5min":20.021999999999998,"15min":20.021999999999998},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":1.1100000000000001,"15min":0.93600000000000005},"min":{"1min":0.40500000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.86199999999999999,"5min":20.908999999999999,"15min":20.908999999999999},"last":0.71399999999999997}]},{"osd":6,"last update":"Mon Feb 19 21:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":1.085,"15min":0.76800000000000002},"min":{"1min":0.41999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.3279999999999998,"5min":20.445,"15min":20.445},"last":0.626},{"interface":"front","average":{"1min":0.65200000000000002,"5min":1.1160000000000001,"15min":0.81399999999999995},"min":{"1min":0.376,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":2.2389999999999999,"5min":20.047999999999998,"15min":20.047999999999998},"last":0.67800000000000005}]},{"osd":7,"last update":"Mon Feb 19 21:06:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":1.1519999999999999,"15min":0.92600000000000005},"min":{"1min":0.373,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.1479999999999997,"5min":20.940999999999999,"15min":20.940999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":1.2250000000000001,"15min":0.83099999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":3.5030000000000001,"5min":20.471,"15min":20.471},"last":0.65700000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411422,"num_pgs":144,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305040,"kb_used_data":3708,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440112,"statfs":{"total":95995035648,"available":95682674688,"internally_reserved":0,"allocated":3796992,"data_stored":2095790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:06:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.873,"15min":0.52900000000000003},"min":{"1min":0.23699999999999999,"5min":0.22900000000000001,"15min":0.222},"max":{"1min":4.665,"5min":26.690999999999999,"15min":26.690999999999999},"last":0.38},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.96399999999999997,"15min":0.55400000000000005},"min":{"1min":0.254,"5min":0.214,"15min":0.214},"max":{"1min":5.2279999999999998,"5min":24.315000000000001,"15min":24.315000000000001},"last":0.44400000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:06:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.327,"15min":0.67900000000000005},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":28.882000000000001,"5min":28.882000000000001,"15min":28.882000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.7370000000000001,"5min":1.228,"15min":0.67500000000000004},"min":{"1min":0.22900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":28.382999999999999,"5min":28.382999999999999,"15min":28.382999999999999},"last":0.46700000000000003}]},{"osd":3,"last update":"Mon Feb 19 21:06:48 2024","interfaces":[{"interface":"back","average":{"1min":1.8069999999999999,"5min":1.087,"15min":0.67100000000000004},"min":{"1min":0.24199999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":28.742000000000001,"5min":28.742000000000001,"15min":28.742000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.97899999999999998,"15min":0.64600000000000002},"min":{"1min":0.28599999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60399999999999998,"5min":26.719999999999999,"15min":26.719999999999999},"last":0.505}]},{"osd":4,"last update":"Mon Feb 19 21:06:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":0.89100000000000001,"15min":0.70299999999999996},"min":{"1min":0.34000000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":24.356000000000002,"5min":24.356000000000002,"15min":24.356000000000002},"last":0.628},{"interface":"front","average":{"1min":1.6240000000000001,"5min":0.85199999999999998,"15min":0.72399999999999998},"min":{"1min":0.38600000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":20.707000000000001,"5min":20.707000000000001,"15min":20.707000000000001},"last":0.67300000000000004}]},{"osd":5,"last update":"Mon Feb 19 21:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.754,"15min":0.69399999999999995},"min":{"1min":0.38100000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":9.9499999999999993,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.86399999999999999,"5min":1.1060000000000001,"15min":0.80800000000000005},"min":{"1min":0.40899999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":4.9359999999999999,"5min":26.606000000000002,"15min":26.606000000000002},"last":0.65600000000000003}]},{"osd":6,"last update":"Mon Feb 19 21:06:01 2024","interfaces":[{"interface":"back","average":{"1min":1.9379999999999999,"5min":0.94799999999999995,"15min":0.73799999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":24.381,"5min":24.381,"15min":24.381},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.69899999999999995,"15min":0.65800000000000003},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.613,"5min":4.5250000000000004,"15min":4.5250000000000004},"last":0.69599999999999995}]},{"osd":7,"last update":"Mon Feb 19 21:06:22 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.1160000000000001,"15min":0.76300000000000001},"min":{"1min":0.41799999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.9139999999999997,"5min":26.658000000000001,"15min":26.658000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.98999999999999999,"15min":0.73399999999999999},"min":{"1min":0.40100000000000002,"5min":0.379,"15min":0.379},"max":{"1min":4.258,"5min":24.402000000000001,"15min":24.402000000000001},"last":0.66300000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182490,"num_pgs":112,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305024,"kb_used_data":3692,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440128,"statfs":{"total":95995035648,"available":95682691072,"internally_reserved":0,"allocated":3780608,"data_stored":2095373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":1.4510000000000001,"15min":0.72199999999999998},"min":{"1min":0.19400000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.51600000000000001,"5min":77.662000000000006,"15min":77.662000000000006},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.74399999999999999,"15min":0.51900000000000002},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.72999999999999998,"5min":17.456,"15min":17.456},"last":0.32600000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.78000000000000003,"15min":0.52700000000000002},"min":{"1min":0.248,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55300000000000005,"5min":17.327999999999999,"15min":17.327999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.69899999999999995,"15min":0.502},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":4.0579999999999998,"5min":15.004,"15min":15.004},"last":0.56699999999999995}]},{"osd":2,"last update":"Mon Feb 19 21:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.2,"15min":0.68500000000000005},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.9319999999999999,"5min":56.890999999999998,"15min":56.890999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":1.1499999999999999,"15min":0.66600000000000004},"min":{"1min":0.253,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.46800000000000003,"5min":44.600999999999999,"15min":44.600999999999999},"last":0.376}]},{"osd":4,"last update":"Mon Feb 19 21:06:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":0.79400000000000004,"15min":0.629},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":12.847,"5min":12.847,"15min":12.847},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":1.0289999999999999,"15min":0.72299999999999998},"min":{"1min":0.372,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":3.294,"5min":26.393999999999998,"15min":26.393999999999998},"last":0.47499999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:06:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.85399999999999998,"15min":0.67300000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.68400000000000005,"5min":17.401,"15min":17.401},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.85499999999999998,"15min":0.66100000000000003},"min":{"1min":0.36899999999999999,"5min":0.36199999999999999,"15min":0.33800000000000002},"max":{"1min":4.0890000000000004,"5min":12.882,"15min":12.882},"last":0.66700000000000004}]},{"osd":6,"last update":"Mon Feb 19 21:05:56 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":0.83799999999999997,"15min":0.66000000000000003},"min":{"1min":0.376,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":8.5090000000000003,"5min":17.367000000000001,"15min":17.367000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.782,"5min":0.81599999999999995,"15min":0.67000000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":12.904,"5min":12.904,"15min":12.904},"last":0.72499999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.84299999999999997,"15min":0.65500000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.80200000000000005,"5min":17.428999999999998,"15min":17.428999999999998},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":1.252,"15min":0.79900000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73299999999999998,"5min":44.378,"15min":44.378},"last":0.504}]}]},{"osd":4,"up_from":31,"seq":133143986262,"num_pgs":142,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309152,"kb_used_data":3732,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436000,"statfs":{"total":95995035648,"available":95678464000,"internally_reserved":0,"allocated":3821568,"data_stored":2095540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:06:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":0.83599999999999997,"15min":0.65200000000000002},"min":{"1min":0.26300000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":19.878,"5min":19.878,"15min":19.878},"last":0.34100000000000003},{"interface":"front","average":{"1min":2.6070000000000002,"5min":1.575,"15min":0.879},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":17.675000000000001,"5min":70.063000000000002,"15min":70.063000000000002},"last":0.215}]},{"osd":1,"last update":"Mon Feb 19 21:06:10 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":0.92000000000000004,"15min":0.66800000000000004},"min":{"1min":0.26300000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":9.1920000000000002,"5min":26.510999999999999,"15min":26.510999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":2.6749999999999998,"5min":1.052,"15min":0.70199999999999996},"min":{"1min":0.28100000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":25.949000000000002,"5min":25.949000000000002,"15min":25.949000000000002},"last":0.23200000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:06:10 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.0629999999999999,"15min":0.70699999999999996},"min":{"1min":0.30299999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":7.516,"5min":37.320999999999998,"15min":37.320999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":4.9420000000000002,"5min":2.1219999999999999,"15min":1.0860000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":69.936999999999998,"5min":69.936999999999998,"15min":69.936999999999998},"last":0.371}]},{"osd":3,"last update":"Mon Feb 19 21:06:10 2024","interfaces":[{"interface":"back","average":{"1min":4.9660000000000002,"5min":1.681,"15min":0.91300000000000003},"min":{"1min":0.29399999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":69.793000000000006,"5min":69.793000000000006,"15min":69.793000000000006},"last":0.315},{"interface":"front","average":{"1min":2.0110000000000001,"5min":1.214,"15min":0.78600000000000003},"min":{"1min":0.29499999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":17.986999999999998,"5min":17.986999999999998,"15min":17.986999999999998},"last":0.36199999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.42099999999999999,"15min":0.42899999999999999},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.218},"max":{"1min":0.52200000000000002,"5min":3.2799999999999998,"15min":3.2799999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.47599999999999998,"15min":0.44700000000000001},"min":{"1min":0.28799999999999998,"5min":0.26600000000000001,"15min":0.23899999999999999},"max":{"1min":0.55100000000000005,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":0.29699999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:05:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46200000000000002,"15min":0.39800000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.2330000000000001,"5min":3.2330000000000001,"15min":3.2330000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.52600000000000002,"15min":0.45400000000000001},"min":{"1min":0.33100000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.5609999999999999,"5min":4.5609999999999999,"15min":4.5609999999999999},"last":0.40699999999999997}]},{"osd":7,"last update":"Mon Feb 19 21:06:28 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.52900000000000003,"15min":0.46100000000000002},"min":{"1min":0.33300000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.71899999999999997,"5min":6.0730000000000004,"15min":6.0730000000000004},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.52200000000000002,"15min":0.47699999999999998},"min":{"1min":0.375,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.624,"5min":3.202,"15min":3.202},"last":0.38300000000000001}]}]},{"osd":5,"up_from":37,"seq":158913790033,"num_pgs":151,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313292,"kb_used_data":3776,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431860,"statfs":{"total":95995035648,"available":95674224640,"internally_reserved":0,"allocated":3866624,"data_stored":2095915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":4.6669999999999998,"15min":1.901},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.51900000000000002,"5min":276.47899999999998,"15min":276.47899999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":1.681,"15min":0.92500000000000004},"min":{"1min":0.24299999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.95399999999999996,"5min":52.887,"15min":52.887},"last":0.38600000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":2.7799999999999998,"15min":1.278},"min":{"1min":0.28599999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63100000000000001,"5min":120.011,"15min":120.011},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":2.0310000000000001,"15min":1.0349999999999999},"min":{"1min":0.33800000000000002,"5min":0.307,"15min":0.307},"max":{"1min":1.7470000000000001,"5min":118.105,"15min":118.105},"last":0.31900000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":1.9390000000000001,"15min":1.0089999999999999},"min":{"1min":0.34399999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.02,"5min":82.242000000000004,"15min":82.242000000000004},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":3.4180000000000001,"15min":1.498},"min":{"1min":0.34499999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85299999999999998,"5min":232.84999999999999,"15min":232.84999999999999},"last":0.40000000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.1150000000000002,"15min":1.0640000000000001},"min":{"1min":0.33900000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.69999999999999996,"5min":109.70099999999999,"15min":109.70099999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":1.637,"15min":0.91500000000000004},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80800000000000005,"5min":65.519000000000005,"15min":65.519000000000005},"last":0.42499999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.438,"15min":0.40200000000000002},"min":{"1min":0.254,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.861,"5min":1.861,"15min":1.861},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.46400000000000002,"15min":0.40699999999999997},"min":{"1min":0.28199999999999997,"5min":0.248,"15min":0.248},"max":{"1min":1.9730000000000001,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":0.376}]},{"osd":6,"last update":"Mon Feb 19 21:06:52 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.46600000000000003,"15min":0.43099999999999999},"min":{"1min":0.312,"5min":0.26900000000000002,"15min":0.26400000000000001},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.46999999999999997,"15min":0.442},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68999999999999995,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.40899999999999997}]},{"osd":7,"last update":"Mon Feb 19 21:06:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.504,"15min":0.502},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.432},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.503,"15min":0.503},"min":{"1min":0.33000000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.42199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":155,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":155,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":155,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":155,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":155,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":155,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":155,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":155,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":156,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":157,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":161,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":161,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":161,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":161,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":161,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":161,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":161,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:06:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:58.568 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:06:58.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:06:58.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:56.567490+0000 osd.1 (osd.1) 131 : cluster [DBG] 61.1 scrub starts 2024-02-19T21:06:58.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:56.570989+0000 osd.1 (osd.1) 132 : cluster [DBG] 61.1 scrub ok 2024-02-19T21:06:58.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:57.153227+0000 mgr.y (mgr.14550) 256 : cluster [DBG] pgmap v326: 347 pgs: 1 clean+premerge+peered, 308 active+clean, 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 32 creating+peering; 8.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:58.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:57.181764+0000 mgr.y (mgr.14550) 257 : audit [DBG] from='client.49304 v1:172.21.15.163:0/3440348924' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:06:58.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:57.681134+0000 osd.3 (osd.3) 123 : cluster [DBG] 8.0 scrub starts 2024-02-19T21:06:58.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:57.682765+0000 osd.3 (osd.3) 124 : cluster [DBG] 8.0 scrub ok 2024-02-19T21:06:58.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:57.886426+0000 osd.4 (osd.4) 123 : cluster [DBG] 61.16 deep-scrub starts 2024-02-19T21:06:58.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:57.889844+0000 osd.4 (osd.4) 124 : cluster [DBG] 61.16 deep-scrub ok 2024-02-19T21:06:58.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:58.201428+0000 mon.a (mon.0) 1666 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-25"}]': finished 2024-02-19T21:06:58.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:58.201552+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-02-19T21:06:58.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:58.207867+0000 mon.b (mon.1) 372 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-25", "mode": "writeback"}]: dispatch 2024-02-19T21:06:58.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:58.209644+0000 mon.a (mon.0) 1668 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-25", "mode": "writeback"}]: dispatch 2024-02-19T21:06:58.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:58.330940+0000 mon.c (mon.2) 271 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:58.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:56.567490+0000 osd.1 (osd.1) 131 : cluster [DBG] 61.1 scrub starts 2024-02-19T21:06:58.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:56.570989+0000 osd.1 (osd.1) 132 : cluster [DBG] 61.1 scrub ok 2024-02-19T21:06:58.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:57.153227+0000 mgr.y (mgr.14550) 256 : cluster [DBG] pgmap v326: 347 pgs: 1 clean+premerge+peered, 308 active+clean, 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 32 creating+peering; 8.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:58.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:57.181764+0000 mgr.y (mgr.14550) 257 : audit [DBG] from='client.49304 v1:172.21.15.163:0/3440348924' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:06:58.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:57.681134+0000 osd.3 (osd.3) 123 : cluster [DBG] 8.0 scrub starts 2024-02-19T21:06:58.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:57.682765+0000 osd.3 (osd.3) 124 : cluster [DBG] 8.0 scrub ok 2024-02-19T21:06:58.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:57.886426+0000 osd.4 (osd.4) 123 : cluster [DBG] 61.16 deep-scrub starts 2024-02-19T21:06:58.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:57.889844+0000 osd.4 (osd.4) 124 : cluster [DBG] 61.16 deep-scrub ok 2024-02-19T21:06:58.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:58.201428+0000 mon.a (mon.0) 1666 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-25"}]': finished 2024-02-19T21:06:58.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:58.201552+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-02-19T21:06:58.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:58.207867+0000 mon.b (mon.1) 372 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-25", "mode": "writeback"}]: dispatch 2024-02-19T21:06:58.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:58.209644+0000 mon.a (mon.0) 1668 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-25", "mode": "writeback"}]: dispatch 2024-02-19T21:06:58.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:58.330940+0000 mon.c (mon.2) 271 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:58.669 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:58.700 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:58.720 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:58.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:58.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:56.567490+0000 osd.1 (osd.1) 131 : cluster [DBG] 61.1 scrub starts 2024-02-19T21:06:58.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:56.570989+0000 osd.1 (osd.1) 132 : cluster [DBG] 61.1 scrub ok 2024-02-19T21:06:58.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:57.153227+0000 mgr.y (mgr.14550) 256 : cluster [DBG] pgmap v326: 347 pgs: 1 clean+premerge+peered, 308 active+clean, 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 32 creating+peering; 8.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:06:58.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:57.181764+0000 mgr.y (mgr.14550) 257 : audit [DBG] from='client.49304 v1:172.21.15.163:0/3440348924' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:06:58.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:57.681134+0000 osd.3 (osd.3) 123 : cluster [DBG] 8.0 scrub starts 2024-02-19T21:06:58.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:57.682765+0000 osd.3 (osd.3) 124 : cluster [DBG] 8.0 scrub ok 2024-02-19T21:06:58.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:57.886426+0000 osd.4 (osd.4) 123 : cluster [DBG] 61.16 deep-scrub starts 2024-02-19T21:06:58.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:57.889844+0000 osd.4 (osd.4) 124 : cluster [DBG] 61.16 deep-scrub ok 2024-02-19T21:06:58.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:58.201428+0000 mon.a (mon.0) 1666 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-25"}]': finished 2024-02-19T21:06:58.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:58.201552+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-02-19T21:06:58.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:58.207867+0000 mon.b (mon.1) 372 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-25", "mode": "writeback"}]: dispatch 2024-02-19T21:06:58.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:58.209644+0000 mon.a (mon.0) 1668 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-25", "mode": "writeback"}]: dispatch 2024-02-19T21:06:58.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:58.330940+0000 mon.c (mon.2) 271 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:58.825 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:58.852 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:58.872 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:58.874 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:58.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:58.948 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-19T21:06:58.959 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:06:58.975 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:06:59.002 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:59.229 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:06:59.257 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:06:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:59.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:06:59.363 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:06:59.392 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:59.413 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:06:59.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:59.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:06:59.517 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:06:59.545 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:59.565 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:06:59.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:06:59.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:06:59.669 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:06:59.697 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:06:59.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:58.309154+0000 osd.2 (osd.2) 105 : cluster [DBG] 16.e scrub starts 2024-02-19T21:06:59.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:58.310299+0000 osd.2 (osd.2) 106 : cluster [DBG] 16.e scrub ok 2024-02-19T21:06:59.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:58.718038+0000 osd.7 (osd.7) 123 : cluster [DBG] 61.f scrub starts 2024-02-19T21:06:59.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:58.720091+0000 osd.7 (osd.7) 124 : cluster [DBG] 61.f scrub ok 2024-02-19T21:06:59.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:58.725840+0000 osd.3 (osd.3) 125 : cluster [DBG] 16.16 scrub starts 2024-02-19T21:06:59.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:58.726852+0000 osd.3 (osd.3) 126 : cluster [DBG] 16.16 scrub ok 2024-02-19T21:06:59.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:59.187331+0000 mon.a (mon.0) 1669 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:59.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:59.191611+0000 mon.a (mon.0) 1670 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-25", "mode": "writeback"}]': finished 2024-02-19T21:06:59.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:59.191739+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-02-19T21:06:59.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:59.195747+0000 mon.c (mon.2) 272 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:59.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:59.196145+0000 mon.a (mon.0) 1672 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:59.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:59.232609+0000 mon.b (mon.1) 373 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:59.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:59.331683+0000 mon.c (mon.2) 273 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:59.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:06:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:06:59.475051+0000 mon.a (mon.0) 1673 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:59.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:58.309154+0000 osd.2 (osd.2) 105 : cluster [DBG] 16.e scrub starts 2024-02-19T21:06:59.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:58.310299+0000 osd.2 (osd.2) 106 : cluster [DBG] 16.e scrub ok 2024-02-19T21:06:59.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:58.718038+0000 osd.7 (osd.7) 123 : cluster [DBG] 61.f scrub starts 2024-02-19T21:06:59.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:58.720091+0000 osd.7 (osd.7) 124 : cluster [DBG] 61.f scrub ok 2024-02-19T21:06:59.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:58.725840+0000 osd.3 (osd.3) 125 : cluster [DBG] 16.16 scrub starts 2024-02-19T21:06:59.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:58.726852+0000 osd.3 (osd.3) 126 : cluster [DBG] 16.16 scrub ok 2024-02-19T21:06:59.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:59.187331+0000 mon.a (mon.0) 1669 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:59.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:59.191611+0000 mon.a (mon.0) 1670 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-25", "mode": "writeback"}]': finished 2024-02-19T21:06:59.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:59.191739+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-02-19T21:06:59.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:59.195747+0000 mon.c (mon.2) 272 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:59.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:59.196145+0000 mon.a (mon.0) 1672 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:59.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:59.232609+0000 mon.b (mon.1) 373 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:59.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:59.331683+0000 mon.c (mon.2) 273 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:59.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:06:59.475051+0000 mon.a (mon.0) 1673 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:59.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:58.309154+0000 osd.2 (osd.2) 105 : cluster [DBG] 16.e scrub starts 2024-02-19T21:06:59.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:58.310299+0000 osd.2 (osd.2) 106 : cluster [DBG] 16.e scrub ok 2024-02-19T21:06:59.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:58.718038+0000 osd.7 (osd.7) 123 : cluster [DBG] 61.f scrub starts 2024-02-19T21:06:59.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:58.720091+0000 osd.7 (osd.7) 124 : cluster [DBG] 61.f scrub ok 2024-02-19T21:06:59.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:58.725840+0000 osd.3 (osd.3) 125 : cluster [DBG] 16.16 scrub starts 2024-02-19T21:06:59.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:58.726852+0000 osd.3 (osd.3) 126 : cluster [DBG] 16.16 scrub ok 2024-02-19T21:06:59.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:59.187331+0000 mon.a (mon.0) 1669 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:06:59.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:59.191611+0000 mon.a (mon.0) 1670 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-25", "mode": "writeback"}]': finished 2024-02-19T21:06:59.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:59.191739+0000 mon.a (mon.0) 1671 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-02-19T21:06:59.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:59.195747+0000 mon.c (mon.2) 272 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:59.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:59.196145+0000 mon.a (mon.0) 1672 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:06:59.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:59.232609+0000 mon.b (mon.1) 373 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:06:59.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:59.331683+0000 mon.c (mon.2) 273 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:06:59.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:06:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:06:59.475051+0000 mon.a (mon.0) 1673 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:00.416 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:00.446 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:00.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:00.552 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:00.557 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:07:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:07:00.579 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:00.632 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:00.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:00.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:00.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:59.153778+0000 mgr.y (mgr.14550) 258 : cluster [DBG] pgmap v329: 291 pgs: 1 clean+premerge+peered, 244 active+clean, 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 8 unknown, 32 creating+peering; 8.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:07:00.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:00.194349+0000 mon.a (mon.0) 1674 : audit [INF] from='client.49298 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi163-83845-34"}]': finished 2024-02-19T21:07:00.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:00.194476+0000 mon.a (mon.0) 1675 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:00.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:00.194538+0000 mon.a (mon.0) 1676 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-02-19T21:07:00.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:00.196929+0000 mon.c (mon.2) 274 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:07:00.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:00.199073+0000 mon.b (mon.1) 374 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25"}]: dispatch 2024-02-19T21:07:00.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:00.199575+0000 mon.a (mon.0) 1677 : audit [INF] from='client.? v1:172.21.15.163:0/3659250931' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi163-83854-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:00.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:00.200060+0000 mon.a (mon.0) 1678 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:07:00.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:00.200384+0000 mon.a (mon.0) 1679 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25"}]: dispatch 2024-02-19T21:07:00.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:00.332420+0000 mon.c (mon.2) 275 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:00.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:59.153778+0000 mgr.y (mgr.14550) 258 : cluster [DBG] pgmap v329: 291 pgs: 1 clean+premerge+peered, 244 active+clean, 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 8 unknown, 32 creating+peering; 8.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:07:00.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:00.194349+0000 mon.a (mon.0) 1674 : audit [INF] from='client.49298 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi163-83845-34"}]': finished 2024-02-19T21:07:00.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:00.194476+0000 mon.a (mon.0) 1675 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:00.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:00.194538+0000 mon.a (mon.0) 1676 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-02-19T21:07:00.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:00.196929+0000 mon.c (mon.2) 274 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:07:00.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:00.199073+0000 mon.b (mon.1) 374 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25"}]: dispatch 2024-02-19T21:07:00.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:00.199575+0000 mon.a (mon.0) 1677 : audit [INF] from='client.? v1:172.21.15.163:0/3659250931' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi163-83854-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:00.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:00.200060+0000 mon.a (mon.0) 1678 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:07:00.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:00.200384+0000 mon.a (mon.0) 1679 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25"}]: dispatch 2024-02-19T21:07:00.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:00.332420+0000 mon.c (mon.2) 275 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:00.737 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:00.769 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:00.790 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:00.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:00.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:59.153778+0000 mgr.y (mgr.14550) 258 : cluster [DBG] pgmap v329: 291 pgs: 1 clean+premerge+peered, 244 active+clean, 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 8 unknown, 32 creating+peering; 8.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:07:00.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:00.194349+0000 mon.a (mon.0) 1674 : audit [INF] from='client.49298 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi163-83845-34"}]': finished 2024-02-19T21:07:00.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:00.194476+0000 mon.a (mon.0) 1675 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:00.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:00.194538+0000 mon.a (mon.0) 1676 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-02-19T21:07:00.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:00.196929+0000 mon.c (mon.2) 274 : audit [INF] from='client.? v1:172.21.15.163:0/2330262446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:07:00.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:00.199073+0000 mon.b (mon.1) 374 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25"}]: dispatch 2024-02-19T21:07:00.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:00.199575+0000 mon.a (mon.0) 1677 : audit [INF] from='client.? v1:172.21.15.163:0/3659250931' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi163-83854-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:00.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:00.200060+0000 mon.a (mon.0) 1678 : audit [INF] from='client.49298 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi163-83845-34"}]: dispatch 2024-02-19T21:07:00.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:00.200384+0000 mon.a (mon.0) 1679 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25"}]: dispatch 2024-02-19T21:07:00.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:00.332420+0000 mon.c (mon.2) 275 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:00.895 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:00.923 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:01.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.559+0000 7fcd4bfd1700 1 -- 172.21.15.163:0/28846390 >> v1:172.21.15.163:6790/0 conn(0x7fcd441031b0 legacy=0x7fcd44105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:01.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.560+0000 7fcd4bfd1700 1 -- 172.21.15.163:0/28846390 shutdown_connections 2024-02-19T21:07:01.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.560+0000 7fcd4bfd1700 1 -- 172.21.15.163:0/28846390 >> 172.21.15.163:0/28846390 conn(0x7fcd440f2470 msgr2=0x7fcd440f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:01.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.560+0000 7fcd4bfd1700 1 -- 172.21.15.163:0/28846390 shutdown_connections 2024-02-19T21:07:01.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.560+0000 7fcd4bfd1700 1 -- 172.21.15.163:0/28846390 wait complete. 2024-02-19T21:07:01.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.561+0000 7fcd4bfd1700 1 librados: starting msgr at 2024-02-19T21:07:01.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.561+0000 7fcd4bfd1700 1 librados: starting objecter 2024-02-19T21:07:01.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.561+0000 7fcd4bfd1700 1 Processor -- start 2024-02-19T21:07:01.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.561+0000 7fcd4bfd1700 1 -- start start 2024-02-19T21:07:01.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.561+0000 7fcd4bfd1700 1 librados: setting wanted keys 2024-02-19T21:07:01.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.561+0000 7fcd4bfd1700 1 librados: calling monclient init 2024-02-19T21:07:01.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.561+0000 7fcd4bfd1700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcd441193e0 con 0x7fcd440ff670 2024-02-19T21:07:01.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.561+0000 7fcd4bfd1700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcd4411a580 con 0x7fcd441031b0 2024-02-19T21:07:01.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.561+0000 7fcd4bfd1700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcd4411b760 con 0x7fcd440fc870 2024-02-19T21:07:01.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.561+0000 7fcd4956c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fcd440ff670 0x7fcd440610f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33234/0 (socket says 172.21.15.163:33234) 2024-02-19T21:07:01.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.561+0000 7fcd4956c700 1 -- 172.21.15.163:0/593739835 learned_addr learned my addr 172.21.15.163:0/593739835 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:01.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.561+0000 7fcd3affd700 10 client.?.objecter ms_handle_connect 0x7fcd440ff670 2024-02-19T21:07:01.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.561+0000 7fcd3affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:01.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 10 client.?.objecter ms_handle_connect 0x7fcd440fc870 2024-02-19T21:07:01.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:01.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 10 client.?.objecter ms_handle_connect 0x7fcd441031b0 2024-02-19T21:07:01.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:01.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53120121 0 0) 0x7fcd441193e0 con 0x7fcd440ff670 2024-02-19T21:07:01.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd300038e0 con 0x7fcd440ff670 2024-02-19T21:07:01.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433961539 0 0) 0x7fcd4411a580 con 0x7fcd441031b0 2024-02-19T21:07:01.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd441193e0 con 0x7fcd441031b0 2024-02-19T21:07:01.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142271732 0 0) 0x7fcd4411b760 con 0x7fcd440fc870 2024-02-19T21:07:01.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd4411a580 con 0x7fcd440fc870 2024-02-19T21:07:01.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3421847544 0 0) 0x7fcd441193e0 con 0x7fcd441031b0 2024-02-19T21:07:01.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd4411b760 con 0x7fcd441031b0 2024-02-19T21:07:01.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcd3c001360 con 0x7fcd441031b0 2024-02-19T21:07:01.559 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:01.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1973957631 0 0) 0x7fcd4411b760 con 0x7fcd441031b0 2024-02-19T21:07:01.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 >> v1:172.21.15.163:6790/0 conn(0x7fcd440fc870 legacy=0x7fcd440609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:01.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 >> v1:172.21.15.163:6789/0 conn(0x7fcd440ff670 legacy=0x7fcd440610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:01.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4411c940 con 0x7fcd441031b0 2024-02-19T21:07:01.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd4bfd1700 1 -- 172.21.15.163:0/593739835 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd4411b950 con 0x7fcd441031b0 2024-02-19T21:07:01.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd4bfd1700 10 client.48145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:01.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd4bfd1700 1 -- 172.21.15.163:0/593739835 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd4411be40 con 0x7fcd441031b0 2024-02-19T21:07:01.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.562+0000 7fcd4bfd1700 1 librados: init done 2024-02-19T21:07:01.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.563+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fcd3c001550 con 0x7fcd441031b0 2024-02-19T21:07:01.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.563+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcd3c003910 con 0x7fcd441031b0 2024-02-19T21:07:01.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.563+0000 7fcd4bfd1700 1 -- 172.21.15.163:0/593739835 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd4405f700 con 0x7fcd441031b0 2024-02-19T21:07:01.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.564+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fcd3c0167c0 con 0x7fcd441031b0 2024-02-19T21:07:01.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.565+0000 7fcd3affd700 10 client.48145.objecter ms_handle_connect 0x7fcd30063ae0 2024-02-19T21:07:01.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.565+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(241..241 src has 43..241) v4 ==== 8795+0+0 (unknown 1437627831 0 0) 0x7fcd3c04aef0 con 0x7fcd441031b0 2024-02-19T21:07:01.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.565+0000 7fcd3affd700 10 client.48145.objecter ms_dispatch 0x7fcd44001550 osd_map(241..241 src has 43..241) v4 2024-02-19T21:07:01.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.566+0000 7fcd3affd700 3 client.48145.objecter handle_osd_map got epochs [241,241] > 0 2024-02-19T21:07:01.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.566+0000 7fcd3affd700 3 client.48145.objecter handle_osd_map decoding full epoch 241 2024-02-19T21:07:01.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.566+0000 7fcd3affd700 20 client.48145.objecter dump_active .. 0 homeless 2024-02-19T21:07:01.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.570+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fcd3c04aa90 con 0x7fcd441031b0 2024-02-19T21:07:01.574 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:01.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:01.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:01.678 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:01.711 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:01.731 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:01.734 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:01.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:01.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:01.756+0000 7fcd4bfd1700 1 -- 172.21.15.163:0/593739835 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fcd44047540 con 0x7fcd441031b0 2024-02-19T21:07:01.835 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:01.865 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:59.672843+0000 osd.6 (osd.6) 109 : cluster [DBG] 61.e scrub starts 2024-02-19T21:07:01.865 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:06:59.674436+0000 osd.6 (osd.6) 110 : cluster [DBG] 61.e scrub ok 2024-02-19T21:07:01.865 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:00.192388+0000 mgr.y (mgr.14550) 259 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:01.866 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:00.446278+0000 osd.5 (osd.5) 118 : cluster [DBG] 16.1f scrub starts 2024-02-19T21:07:01.866 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:00.449530+0000 osd.5 (osd.5) 119 : cluster [DBG] 16.1f scrub ok 2024-02-19T21:07:01.866 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:00.556586+0000 osd.6 (osd.6) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:01.866 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:00.556811+0000 osd.6 (osd.6) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:01.866 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:01.154973+0000 mon.b (mon.1) 375 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-19T21:07:01.866 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:01.155937+0000 mon.a (mon.0) 1680 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-19T21:07:01.866 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:01.190955+0000 mon.a (mon.0) 1681 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:01.866 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:01.195771+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? v1:172.21.15.163:0/3659250931' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi163-83854-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:01.866 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:01.195848+0000 mon.a (mon.0) 1683 : audit [INF] from='client.49298 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi163-83845-34"}]': finished 2024-02-19T21:07:01.866 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:01.195909+0000 mon.a (mon.0) 1684 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25"}]': finished 2024-02-19T21:07:01.866 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:01.196011+0000 mon.a (mon.0) 1685 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-19T21:07:01.867 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:01.196061+0000 mon.a (mon.0) 1686 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-02-19T21:07:01.867 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:01.232190+0000 mon.c (mon.2) 276 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:01.867 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:01.232658+0000 mon.a (mon.0) 1687 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:01.867 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:01.233483+0000 mon.c (mon.2) 277 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:01.867 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:01.233746+0000 mon.a (mon.0) 1688 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:01.867 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:01.234141+0000 mon.c (mon.2) 278 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi163-83845-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:01.867 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:01.234390+0000 mon.a (mon.0) 1689 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi163-83845-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:01.867 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:01.333244+0000 mon.c (mon.2) 279 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:01.867 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:01.886 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:01.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:01.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:01.990 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:01.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:59.672843+0000 osd.6 (osd.6) 109 : cluster [DBG] 61.e scrub starts 2024-02-19T21:07:01.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:06:59.674436+0000 osd.6 (osd.6) 110 : cluster [DBG] 61.e scrub ok 2024-02-19T21:07:01.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:00.192388+0000 mgr.y (mgr.14550) 259 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:01.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:00.446278+0000 osd.5 (osd.5) 118 : cluster [DBG] 16.1f scrub starts 2024-02-19T21:07:01.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:00.449530+0000 osd.5 (osd.5) 119 : cluster [DBG] 16.1f scrub ok 2024-02-19T21:07:01.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:00.556586+0000 osd.6 (osd.6) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:01.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:00.556811+0000 osd.6 (osd.6) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:01.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:01.154973+0000 mon.b (mon.1) 375 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-19T21:07:01.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:01.155937+0000 mon.a (mon.0) 1680 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-19T21:07:01.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:01.190955+0000 mon.a (mon.0) 1681 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:01.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:01.195771+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? v1:172.21.15.163:0/3659250931' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi163-83854-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:01.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:01.195848+0000 mon.a (mon.0) 1683 : audit [INF] from='client.49298 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi163-83845-34"}]': finished 2024-02-19T21:07:01.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:01.195909+0000 mon.a (mon.0) 1684 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25"}]': finished 2024-02-19T21:07:01.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:01.196011+0000 mon.a (mon.0) 1685 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-19T21:07:01.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:01.196061+0000 mon.a (mon.0) 1686 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-02-19T21:07:01.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:01.232190+0000 mon.c (mon.2) 276 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:01.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:01.232658+0000 mon.a (mon.0) 1687 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:01.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:01.233483+0000 mon.c (mon.2) 277 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:01.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:01.233746+0000 mon.a (mon.0) 1688 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:01.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:01.234141+0000 mon.c (mon.2) 278 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi163-83845-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:01.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:01.234390+0000 mon.a (mon.0) 1689 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi163-83845-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:01.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:01.333244+0000 mon.c (mon.2) 279 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:01.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:59.672843+0000 osd.6 (osd.6) 109 : cluster [DBG] 61.e scrub starts 2024-02-19T21:07:01.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:06:59.674436+0000 osd.6 (osd.6) 110 : cluster [DBG] 61.e scrub ok 2024-02-19T21:07:01.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:00.192388+0000 mgr.y (mgr.14550) 259 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:01.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:00.446278+0000 osd.5 (osd.5) 118 : cluster [DBG] 16.1f scrub starts 2024-02-19T21:07:01.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:00.449530+0000 osd.5 (osd.5) 119 : cluster [DBG] 16.1f scrub ok 2024-02-19T21:07:01.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:00.556586+0000 osd.6 (osd.6) 111 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:01.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:00.556811+0000 osd.6 (osd.6) 112 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:01.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:01.154973+0000 mon.b (mon.1) 375 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-19T21:07:01.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:01.155937+0000 mon.a (mon.0) 1680 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-19T21:07:01.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:01.190955+0000 mon.a (mon.0) 1681 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:01.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:01.195771+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? v1:172.21.15.163:0/3659250931' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi163-83854-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:01.195848+0000 mon.a (mon.0) 1683 : audit [INF] from='client.49298 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi163-83845-34"}]': finished 2024-02-19T21:07:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:01.195909+0000 mon.a (mon.0) 1684 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-25"}]': finished 2024-02-19T21:07:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:01.196011+0000 mon.a (mon.0) 1685 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi163-83888-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-19T21:07:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:01.196061+0000 mon.a (mon.0) 1686 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-02-19T21:07:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:01.232190+0000 mon.c (mon.2) 276 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:01.232658+0000 mon.a (mon.0) 1687 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:01.233483+0000 mon.c (mon.2) 277 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:01.233746+0000 mon.a (mon.0) 1688 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:01.234141+0000 mon.c (mon.2) 278 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi163-83845-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:01.234390+0000 mon.a (mon.0) 1689 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi163-83845-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:01.333244+0000 mon.c (mon.2) 279 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:02.020 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:02.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:02.206+0000 7fcd3affd700 1 -- 172.21.15.163:0/593739835 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v242) v1 ==== 87+0+0 (unknown 3990189088 0 0) 0x7fcd3c049130 con 0x7fcd441031b0 2024-02-19T21:07:02.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:02.209+0000 7fcd4bfd1700 10 librados: watch_flush enter 2024-02-19T21:07:02.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:02.213+0000 7fcd4bfd1700 10 librados: watch_flush exit 2024-02-19T21:07:02.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:02.213+0000 7fcd4bfd1700 20 client.48145.objecter shutdown clearing up homeless session... 2024-02-19T21:07:02.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:02.213+0000 7fcd4bfd1700 10 client.48145.objecter successfully canceled tick 2024-02-19T21:07:02.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:02.215+0000 7fcd4bfd1700 1 -- 172.21.15.163:0/593739835 >> v1:172.21.15.163:6800/1552514681 conn(0x7fcd30063ae0 legacy=0x7fcd30065fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:02.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:02.215+0000 7fcd4bfd1700 1 -- 172.21.15.163:0/593739835 >> v1:172.21.15.193:6789/0 conn(0x7fcd441031b0 legacy=0x7fcd44117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:02.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:02.216+0000 7fcd4bfd1700 1 -- 172.21.15.163:0/593739835 shutdown_connections 2024-02-19T21:07:02.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:02.216+0000 7fcd4bfd1700 1 -- 172.21.15.163:0/593739835 >> 172.21.15.163:0/593739835 conn(0x7fcd440f2470 msgr2=0x7fcd44106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:02.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:02.219+0000 7fcd4bfd1700 1 -- 172.21.15.163:0/593739835 shutdown_connections 2024-02-19T21:07:02.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:02.219+0000 7fcd4bfd1700 1 -- 172.21.15.163:0/593739835 wait complete. 2024-02-19T21:07:02.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:02.219+0000 7fcd4bfd1700 1 librados: shutdown 2024-02-19T21:07:02.214 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:07:02.224 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:02.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:02.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:02.330 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:02.360 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:02.379 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:02.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:02.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:02.483 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:02.510 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:02.781 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:00.675782+0000 osd.6 (osd.6) 113 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:07:02.781 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:00.676628+0000 osd.6 (osd.6) 114 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:07:02.781 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:01.154440+0000 mgr.y (mgr.14550) 260 : cluster [DBG] pgmap v332: 282 pgs: 32 unknown, 1 peering, 248 active+clean, 1 active+clean+snaptrim; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 4 op/s; 63 B/s, 0 objects/s recovering 2024-02-19T21:07:02.781 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:01.261955+0000 osd.2 (osd.2) 107 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:07:02.781 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:01.264085+0000 osd.2 (osd.2) 108 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:07:02.781 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:01.757259+0000 mon.b (mon.1) 376 : audit [INF] from='client.? v1:172.21.15.163:0/593739835' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:07:02.781 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:01.757567+0000 mon.a (mon.0) 1690 : audit [INF] from='client.48145 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:07:02.781 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:01.781038+0000 osd.7 (osd.7) 125 : cluster [DBG] 61.1e deep-scrub starts 2024-02-19T21:07:02.782 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:01.787515+0000 osd.7 (osd.7) 126 : cluster [DBG] 61.1e deep-scrub ok 2024-02-19T21:07:02.782 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:02.192551+0000 mon.a (mon.0) 1691 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:02.782 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:02.197933+0000 mon.a (mon.0) 1692 : audit [INF] from='client.49316 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi163-83845-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:02.782 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:02.198038+0000 mon.a (mon.0) 1693 : audit [INF] from='client.48145 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:07:02.782 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:02.198088+0000 mon.a (mon.0) 1694 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-02-19T21:07:02.782 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:02.205941+0000 mon.c (mon.2) 280 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi163-83845-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:02.782 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:02.206536+0000 mon.a (mon.0) 1695 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi163-83845-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:02.782 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:02.334149+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:02.782 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:00.675782+0000 osd.6 (osd.6) 113 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:07:02.782 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:00.676628+0000 osd.6 (osd.6) 114 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:07:02.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:01.154440+0000 mgr.y (mgr.14550) 260 : cluster [DBG] pgmap v332: 282 pgs: 32 unknown, 1 peering, 248 active+clean, 1 active+clean+snaptrim; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 4 op/s; 63 B/s, 0 objects/s recovering 2024-02-19T21:07:02.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:01.261955+0000 osd.2 (osd.2) 107 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:07:02.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:01.264085+0000 osd.2 (osd.2) 108 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:07:02.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:01.757259+0000 mon.b (mon.1) 376 : audit [INF] from='client.? v1:172.21.15.163:0/593739835' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:07:02.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:01.757567+0000 mon.a (mon.0) 1690 : audit [INF] from='client.48145 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:07:02.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:01.781038+0000 osd.7 (osd.7) 125 : cluster [DBG] 61.1e deep-scrub starts 2024-02-19T21:07:02.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:01.787515+0000 osd.7 (osd.7) 126 : cluster [DBG] 61.1e deep-scrub ok 2024-02-19T21:07:02.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:02.192551+0000 mon.a (mon.0) 1691 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:02.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:02.197933+0000 mon.a (mon.0) 1692 : audit [INF] from='client.49316 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi163-83845-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:02.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:02.198038+0000 mon.a (mon.0) 1693 : audit [INF] from='client.48145 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:07:02.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:02.198088+0000 mon.a (mon.0) 1694 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-02-19T21:07:02.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:02.205941+0000 mon.c (mon.2) 280 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi163-83845-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:02.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:02.206536+0000 mon.a (mon.0) 1695 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi163-83845-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:02.784 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:02.334149+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:03.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:00.675782+0000 osd.6 (osd.6) 113 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:07:03.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:00.676628+0000 osd.6 (osd.6) 114 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:07:03.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:01.154440+0000 mgr.y (mgr.14550) 260 : cluster [DBG] pgmap v332: 282 pgs: 32 unknown, 1 peering, 248 active+clean, 1 active+clean+snaptrim; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 4 op/s; 63 B/s, 0 objects/s recovering 2024-02-19T21:07:03.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:01.261955+0000 osd.2 (osd.2) 107 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:07:03.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:01.264085+0000 osd.2 (osd.2) 108 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:07:03.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:01.757259+0000 mon.b (mon.1) 376 : audit [INF] from='client.? v1:172.21.15.163:0/593739835' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:07:03.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:01.757567+0000 mon.a (mon.0) 1690 : audit [INF] from='client.48145 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:07:03.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:01.781038+0000 osd.7 (osd.7) 125 : cluster [DBG] 61.1e deep-scrub starts 2024-02-19T21:07:03.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:01.787515+0000 osd.7 (osd.7) 126 : cluster [DBG] 61.1e deep-scrub ok 2024-02-19T21:07:03.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:02.192551+0000 mon.a (mon.0) 1691 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:03.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:02.197933+0000 mon.a (mon.0) 1692 : audit [INF] from='client.49316 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi163-83845-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:03.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:02.198038+0000 mon.a (mon.0) 1693 : audit [INF] from='client.48145 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:07:03.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:02.198088+0000 mon.a (mon.0) 1694 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-02-19T21:07:03.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:02.205941+0000 mon.c (mon.2) 280 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi163-83845-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:03.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:02.206536+0000 mon.a (mon.0) 1695 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi163-83845-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:03.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:02.334149+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:03.288 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:03.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:03.387 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:07:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:07:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-02-19T21:07:03.388 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-02-19T21:07:03.388 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:07:03.396 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:03.428 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:03.450 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:03.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:03.554 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:03.564 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:07:03.564 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:07:03.564 INFO:tasks.thrashosds.thrasher:Shrinking pool .rgw.root 2024-02-19T21:07:03.564 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .rgw.root by 10 min 8 2024-02-19T21:07:03.564 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:07:03.582 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:07:03] "GET /metrics HTTP/1.1" 200 199470 "" "Prometheus/2.33.4" 2024-02-19T21:07:03.585 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:03.605 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:03.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:03.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:03.708 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:07:03.736 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:03.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:01.729341+0000 osd.3 (osd.3) 127 : cluster [DBG] 4.14 scrub starts 2024-02-19T21:07:03.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:01.731064+0000 osd.3 (osd.3) 128 : cluster [DBG] 4.14 scrub ok 2024-02-19T21:07:03.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:02.613280+0000 osd.4 (osd.4) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:03.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:02.613491+0000 osd.4 (osd.4) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:03.957 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:03.335377+0000 mon.c (mon.2) 282 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:03.957 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:03.441170+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-02-19T21:07:03.957 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:03.442016+0000 mon.c (mon.2) 283 : audit [INF] from='client.? v1:172.21.15.163:0/4008237253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi163-83854-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:03.957 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:03.442124+0000 mon.b (mon.1) 377 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:03.957 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:03.442774+0000 mon.a (mon.0) 1697 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:03.957 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:03.445107+0000 mon.a (mon.0) 1698 : audit [INF] from='client.49328 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi163-83854-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:03.957 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:01.729341+0000 osd.3 (osd.3) 127 : cluster [DBG] 4.14 scrub starts 2024-02-19T21:07:03.957 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:01.731064+0000 osd.3 (osd.3) 128 : cluster [DBG] 4.14 scrub ok 2024-02-19T21:07:03.957 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:02.613280+0000 osd.4 (osd.4) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:03.957 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:02.613491+0000 osd.4 (osd.4) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:03.957 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:03.335377+0000 mon.c (mon.2) 282 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:03.958 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:03.441170+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-02-19T21:07:03.958 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:03.442016+0000 mon.c (mon.2) 283 : audit [INF] from='client.? v1:172.21.15.163:0/4008237253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi163-83854-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:03.958 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:03.442124+0000 mon.b (mon.1) 377 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:03.958 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:03.442774+0000 mon.a (mon.0) 1697 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:03.958 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:03.445107+0000 mon.a (mon.0) 1698 : audit [INF] from='client.49328 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi163-83854-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:03.964 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:07:03.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:03.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:07:04.066 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:01.729341+0000 osd.3 (osd.3) 127 : cluster [DBG] 4.14 scrub starts 2024-02-19T21:07:04.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:01.731064+0000 osd.3 (osd.3) 128 : cluster [DBG] 4.14 scrub ok 2024-02-19T21:07:04.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:02.613280+0000 osd.4 (osd.4) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:04.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:02.613491+0000 osd.4 (osd.4) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:04.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:03.335377+0000 mon.c (mon.2) 282 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:04.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:03.441170+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-02-19T21:07:04.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:03.442016+0000 mon.c (mon.2) 283 : audit [INF] from='client.? v1:172.21.15.163:0/4008237253' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi163-83854-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:04.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:03.442124+0000 mon.b (mon.1) 377 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:04.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:03.442774+0000 mon.a (mon.0) 1697 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:04.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:03.445107+0000 mon.a (mon.0) 1698 : audit [INF] from='client.49328 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi163-83854-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:04.069 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:04.100 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:04.121 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:04.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:04.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:04.226 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:07:04.256 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:04.275 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:07:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:04.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:07:04.378 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:04.406 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:04.426 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:04.428 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:04.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:04.485 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-19T21:07:04.530 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:04.556 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:04.574 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:04.576 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:04.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:04.678 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:04.704 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:04.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:03.155035+0000 mgr.y (mgr.14550) 261 : cluster [DBG] pgmap v335: 218 pgs: 1 peering, 216 active+clean, 1 active+clean+snaptrim; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 4 op/s; 63 B/s, 0 objects/s recovering 2024-02-19T21:07:04.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:04.198322+0000 mon.a (mon.0) 1699 : audit [INF] from='client.49316 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi163-83845-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi163-83845-35"}]': finished 2024-02-19T21:07:04.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:04.198533+0000 mon.a (mon.0) 1700 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:04.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:04.198601+0000 mon.a (mon.0) 1701 : audit [INF] from='client.49328 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi163-83854-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:04.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:04.198687+0000 mon.a (mon.0) 1702 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-02-19T21:07:04.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:04.336268+0000 mon.c (mon.2) 284 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:04.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:03.155035+0000 mgr.y (mgr.14550) 261 : cluster [DBG] pgmap v335: 218 pgs: 1 peering, 216 active+clean, 1 active+clean+snaptrim; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 4 op/s; 63 B/s, 0 objects/s recovering 2024-02-19T21:07:04.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:04.198322+0000 mon.a (mon.0) 1699 : audit [INF] from='client.49316 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi163-83845-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi163-83845-35"}]': finished 2024-02-19T21:07:04.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:04.198533+0000 mon.a (mon.0) 1700 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:04.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:04.198601+0000 mon.a (mon.0) 1701 : audit [INF] from='client.49328 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi163-83854-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:04.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:04.198687+0000 mon.a (mon.0) 1702 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-02-19T21:07:04.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:04.336268+0000 mon.c (mon.2) 284 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:04.725 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:04.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:04.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:04.835 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:04.862 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:04.881 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:04.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:04.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:04.985 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:05.011 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:05.032 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:05.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:05.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:05.064 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:05.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:03.155035+0000 mgr.y (mgr.14550) 261 : cluster [DBG] pgmap v335: 218 pgs: 1 peering, 216 active+clean, 1 active+clean+snaptrim; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 4 op/s; 63 B/s, 0 objects/s recovering 2024-02-19T21:07:05.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:04.198322+0000 mon.a (mon.0) 1699 : audit [INF] from='client.49316 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi163-83845-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi163-83845-35"}]': finished 2024-02-19T21:07:05.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:04.198533+0000 mon.a (mon.0) 1700 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:05.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:04.198601+0000 mon.a (mon.0) 1701 : audit [INF] from='client.49328 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi163-83854-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:05.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:04.198687+0000 mon.a (mon.0) 1702 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-02-19T21:07:05.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:04.336268+0000 mon.c (mon.2) 284 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:05.136 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:05.164 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:05.183 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:05.185 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:05.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:05.287 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:05.313 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:05.332 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:05.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:05.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:05.436 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:05.462 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:05.481 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:05.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:05.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:05.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:05.601 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:05.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.614+0000 7fba1d08e700 1 -- 172.21.15.163:0/1397974040 >> v1:172.21.15.163:6790/0 conn(0x7fba181042c0 legacy=0x7fba18106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:05.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.614+0000 7fba1d08e700 1 -- 172.21.15.163:0/1397974040 shutdown_connections 2024-02-19T21:07:05.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.614+0000 7fba1d08e700 1 -- 172.21.15.163:0/1397974040 >> 172.21.15.163:0/1397974040 conn(0x7fba1806cc00 msgr2=0x7fba180f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:05.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.614+0000 7fba1d08e700 1 -- 172.21.15.163:0/1397974040 shutdown_connections 2024-02-19T21:07:05.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.614+0000 7fba1d08e700 1 -- 172.21.15.163:0/1397974040 wait complete. 2024-02-19T21:07:05.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.614+0000 7fba1d08e700 1 librados: starting msgr at 2024-02-19T21:07:05.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.614+0000 7fba1d08e700 1 librados: starting objecter 2024-02-19T21:07:05.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.615+0000 7fba1d08e700 1 Processor -- start 2024-02-19T21:07:05.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.615+0000 7fba1d08e700 1 -- start start 2024-02-19T21:07:05.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.615+0000 7fba1d08e700 1 librados: setting wanted keys 2024-02-19T21:07:05.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.615+0000 7fba1d08e700 1 librados: calling monclient init 2024-02-19T21:07:05.612 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:05.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.615+0000 7fba1d08e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba181172f0 con 0x7fba181007b0 2024-02-19T21:07:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.615+0000 7fba1d08e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba181184f0 con 0x7fba180fd960 2024-02-19T21:07:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.615+0000 7fba1d08e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba181196f0 con 0x7fba181042c0 2024-02-19T21:07:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fba16d9d700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fba180fd960 0x7fba180f9c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:38986/0 (socket says 172.21.15.163:38986) 2024-02-19T21:07:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fba1759e700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fba181042c0 0x7fba181159b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45886/0 (socket says 172.21.15.163:45886) 2024-02-19T21:07:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fba16d9d700 1 -- 172.21.15.163:0/2965375004 learned_addr learned my addr 172.21.15.163:0/2965375004 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fba1659c700 1 --1- 172.21.15.163:0/2965375004 >> v1:172.21.15.163:6789/0 conn(0x7fba181007b0 0x7fba180fa310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:42800/0 (socket says 172.21.15.163:42800) 2024-02-19T21:07:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 10 client.?.objecter ms_handle_connect 0x7fba181007b0 2024-02-19T21:07:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 10 client.?.objecter ms_handle_connect 0x7fba181042c0 2024-02-19T21:07:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 10 client.?.objecter ms_handle_connect 0x7fba180fd960 2024-02-19T21:07:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295437041 0 0) 0x7fba181184f0 con 0x7fba180fd960 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba000033f0 con 0x7fba180fd960 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917356576 0 0) 0x7fba181196f0 con 0x7fba181042c0 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba181184f0 con 0x7fba181042c0 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2254734566 0 0) 0x7fba000033f0 con 0x7fba180fd960 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba181196f0 con 0x7fba180fd960 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fba0c0018c0 con 0x7fba180fd960 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2455879237 0 0) 0x7fba181184f0 con 0x7fba181042c0 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba000033f0 con 0x7fba181042c0 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fba10002240 con 0x7fba181042c0 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 821203722 0 0) 0x7fba000033f0 con 0x7fba181042c0 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 >> v1:172.21.15.193:6789/0 conn(0x7fba180fd960 legacy=0x7fba180f9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 >> v1:172.21.15.163:6789/0 conn(0x7fba181007b0 legacy=0x7fba180fa310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1811a8f0 con 0x7fba181042c0 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba181198e0 con 0x7fba181042c0 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fba1d08e700 10 client.49340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:05.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba18119dd0 con 0x7fba181042c0 2024-02-19T21:07:05.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.616+0000 7fba1d08e700 1 librados: init done 2024-02-19T21:07:05.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.617+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fba10001c80 con 0x7fba181042c0 2024-02-19T21:07:05.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.617+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fba10003a60 con 0x7fba181042c0 2024-02-19T21:07:05.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.618+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fba04004f00 con 0x7fba181042c0 2024-02-19T21:07:05.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.618+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fba10003bc0 con 0x7fba181042c0 2024-02-19T21:07:05.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.618+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(245..245 src has 43..245) v4 ==== 9212+0+0 (unknown 344155935 0 0) 0x7fba10079660 con 0x7fba181042c0 2024-02-19T21:07:05.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.618+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 osd_map(245..245 src has 43..245) v4 2024-02-19T21:07:05.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.618+0000 7fb9fffff700 3 client.49340.objecter handle_osd_map got epochs [245,245] > 0 2024-02-19T21:07:05.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.618+0000 7fb9fffff700 3 client.49340.objecter handle_osd_map decoding full epoch 245 2024-02-19T21:07:05.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.618+0000 7fb9fffff700 20 client.49340.objecter dump_active .. 0 homeless 2024-02-19T21:07:05.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.618+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v245) v1 ==== 54+0+16 (unknown 2048148357 0 2420000192) 0x7fba1007ba70 con 0x7fba181042c0 2024-02-19T21:07:05.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fb9fffff700 10 client.49340.objecter ms_handle_connect 0x7fba00063b80 2024-02-19T21:07:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fba1d08e700 10 client.49340.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 1 2024-02-19T21:07:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba040051a0 osd=0 3 2024-02-19T21:07:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba040051a0 osd=0 3 2024-02-19T21:07:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba040051a0 osd=0 3 2024-02-19T21:07:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 1 2024-02-19T21:07:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba040051a0 osd=0 3 2024-02-19T21:07:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 0 1 2024-02-19T21:07:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba040051a0 osd=0 4 2024-02-19T21:07:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fba1d08e700 10 client.49340.objecter _send_command 1 2024-02-19T21:07:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba04004950 con 0x7fba04004530 2024-02-19T21:07:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fb9fffff700 10 client.49340.objecter ms_handle_connect 0x7fba04004530 2024-02-19T21:07:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba04004950 con 0x7fba04004530 2024-02-19T21:07:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:07:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fb9fffff700 10 client.49340.objecter _finish_command 1 = system:0 2024-02-19T21:07:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba040051a0 osd=0 4 2024-02-19T21:07:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.619+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 0 1 2024-02-19T21:07:05.629 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:05.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.635+0000 7fba1d08e700 10 client.49340.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:05.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.635+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 2 2024-02-19T21:07:05.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.635+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba040051a0 osd=0 3 2024-02-19T21:07:05.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.635+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba040051a0 osd=0 3 2024-02-19T21:07:05.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.635+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba040051a0 osd=0 3 2024-02-19T21:07:05.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.635+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 2 2024-02-19T21:07:05.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.635+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba040051a0 osd=0 3 2024-02-19T21:07:05.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.635+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 0 2 2024-02-19T21:07:05.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.635+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba040051a0 osd=0 4 2024-02-19T21:07:05.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.635+0000 7fba1d08e700 10 client.49340.objecter _send_command 2 2024-02-19T21:07:05.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.635+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba04004d30 con 0x7fba04004530 2024-02-19T21:07:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:05.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:05.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba04004d30 con 0x7fba04004530 2024-02-19T21:07:05.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:05.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fb9fffff700 10 client.49340.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba040051a0 osd=0 4 2024-02-19T21:07:05.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 0 2 2024-02-19T21:07:05.634 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:07:05.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fba1d08e700 10 client.49340.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:05.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 3 2024-02-19T21:07:05.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba04003fe0 osd=1 3 2024-02-19T21:07:05.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba04003fe0 osd=1 3 2024-02-19T21:07:05.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba04003fe0 osd=1 3 2024-02-19T21:07:05.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 3 2024-02-19T21:07:05.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba04003fe0 osd=1 3 2024-02-19T21:07:05.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 1 3 2024-02-19T21:07:05.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba04003fe0 osd=1 4 2024-02-19T21:07:05.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fba1d08e700 10 client.49340.objecter _send_command 3 2024-02-19T21:07:05.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.637+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba0400c000 con 0x7fba04004150 2024-02-19T21:07:05.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.638+0000 7fb9fffff700 10 client.49340.objecter ms_handle_connect 0x7fba04004150 2024-02-19T21:07:05.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.638+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba0400c000 con 0x7fba04004150 2024-02-19T21:07:05.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.638+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:07:05.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.638+0000 7fb9fffff700 10 client.49340.objecter _finish_command 3 = system:0 2024-02-19T21:07:05.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.638+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba04003fe0 osd=1 4 2024-02-19T21:07:05.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.638+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 1 3 2024-02-19T21:07:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.655+0000 7fba1d08e700 10 client.49340.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.655+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 4 2024-02-19T21:07:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.655+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba04003fe0 osd=1 3 2024-02-19T21:07:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.655+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba04003fe0 osd=1 3 2024-02-19T21:07:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.655+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba04003fe0 osd=1 3 2024-02-19T21:07:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.655+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 4 2024-02-19T21:07:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.655+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba04003fe0 osd=1 3 2024-02-19T21:07:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.655+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 1 4 2024-02-19T21:07:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.655+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba04003fe0 osd=1 4 2024-02-19T21:07:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.655+0000 7fba1d08e700 10 client.49340.objecter _send_command 4 2024-02-19T21:07:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.655+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba040018b0 con 0x7fba04004150 2024-02-19T21:07:05.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba040018b0 con 0x7fba04004150 2024-02-19T21:07:05.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:05.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fb9fffff700 10 client.49340.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba04003fe0 osd=1 4 2024-02-19T21:07:05.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 1 4 2024-02-19T21:07:05.653 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:07:05.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fba1d08e700 10 client.49340.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:05.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 5 2024-02-19T21:07:05.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba04003630 osd=2 3 2024-02-19T21:07:05.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba04003630 osd=2 3 2024-02-19T21:07:05.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba04003630 osd=2 3 2024-02-19T21:07:05.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 5 2024-02-19T21:07:05.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba04003630 osd=2 3 2024-02-19T21:07:05.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 2 5 2024-02-19T21:07:05.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba04003630 osd=2 4 2024-02-19T21:07:05.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fba1d08e700 10 client.49340.objecter _send_command 5 2024-02-19T21:07:05.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba0400de90 con 0x7fba0400d2f0 2024-02-19T21:07:05.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.657+0000 7fb9fffff700 10 client.49340.objecter ms_handle_connect 0x7fba0400d2f0 2024-02-19T21:07:05.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.658+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba0400de90 con 0x7fba0400d2f0 2024-02-19T21:07:05.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.658+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:07:05.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.658+0000 7fb9fffff700 10 client.49340.objecter _finish_command 5 = system:0 2024-02-19T21:07:05.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.658+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba04003630 osd=2 4 2024-02-19T21:07:05.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.658+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 2 5 2024-02-19T21:07:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.672+0000 7fba1d08e700 10 client.49340.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.672+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 6 2024-02-19T21:07:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.672+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba04003630 osd=2 3 2024-02-19T21:07:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.672+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba04003630 osd=2 3 2024-02-19T21:07:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.672+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba04003630 osd=2 3 2024-02-19T21:07:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.672+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 6 2024-02-19T21:07:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.672+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba04003630 osd=2 3 2024-02-19T21:07:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.672+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 2 6 2024-02-19T21:07:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.672+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba04003630 osd=2 4 2024-02-19T21:07:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.672+0000 7fba1d08e700 10 client.49340.objecter _send_command 6 2024-02-19T21:07:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.672+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba040030a0 con 0x7fba0400d2f0 2024-02-19T21:07:05.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba040030a0 con 0x7fba0400d2f0 2024-02-19T21:07:05.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:05.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fb9fffff700 10 client.49340.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba04003630 osd=2 4 2024-02-19T21:07:05.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 2 6 2024-02-19T21:07:05.670 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:07:05.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fba1d08e700 10 client.49340.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:05.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 7 2024-02-19T21:07:05.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba04003480 osd=3 3 2024-02-19T21:07:05.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba04003480 osd=3 3 2024-02-19T21:07:05.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba04003480 osd=3 3 2024-02-19T21:07:05.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 7 2024-02-19T21:07:05.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba04003480 osd=3 3 2024-02-19T21:07:05.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 3 7 2024-02-19T21:07:05.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba04003480 osd=3 4 2024-02-19T21:07:05.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fba1d08e700 10 client.49340.objecter _send_command 7 2024-02-19T21:07:05.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba04015a60 con 0x7fba04011e80 2024-02-19T21:07:05.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.674+0000 7fb9fffff700 10 client.49340.objecter ms_handle_connect 0x7fba04011e80 2024-02-19T21:07:05.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.675+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba04015a60 con 0x7fba04011e80 2024-02-19T21:07:05.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.675+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:07:05.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.675+0000 7fb9fffff700 10 client.49340.objecter _finish_command 7 = system:0 2024-02-19T21:07:05.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.675+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba04003480 osd=3 4 2024-02-19T21:07:05.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.675+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 3 7 2024-02-19T21:07:05.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.689+0000 7fba1d08e700 10 client.49340.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:05.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.689+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 8 2024-02-19T21:07:05.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.689+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba04003480 osd=3 3 2024-02-19T21:07:05.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.689+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba04003480 osd=3 3 2024-02-19T21:07:05.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.689+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba04003480 osd=3 3 2024-02-19T21:07:05.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.689+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 8 2024-02-19T21:07:05.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.689+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba04003480 osd=3 3 2024-02-19T21:07:05.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.689+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 3 8 2024-02-19T21:07:05.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.689+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba04003480 osd=3 4 2024-02-19T21:07:05.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.689+0000 7fba1d08e700 10 client.49340.objecter _send_command 8 2024-02-19T21:07:05.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.689+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba0400e270 con 0x7fba04011e80 2024-02-19T21:07:05.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba0400e270 con 0x7fba04011e80 2024-02-19T21:07:05.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:05.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fb9fffff700 10 client.49340.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba04003480 osd=3 4 2024-02-19T21:07:05.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 3 8 2024-02-19T21:07:05.687 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:07:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fba1d08e700 10 client.49340.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 9 2024-02-19T21:07:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0400e410 osd=4 3 2024-02-19T21:07:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0400e410 osd=4 3 2024-02-19T21:07:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0400e410 osd=4 3 2024-02-19T21:07:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 9 2024-02-19T21:07:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba0400e410 osd=4 3 2024-02-19T21:07:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 4 9 2024-02-19T21:07:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0400e410 osd=4 4 2024-02-19T21:07:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fba1d08e700 10 client.49340.objecter _send_command 9 2024-02-19T21:07:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.691+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba0400e580 con 0x7fba0400ce50 2024-02-19T21:07:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.692+0000 7fb9fffff700 10 client.49340.objecter ms_handle_connect 0x7fba0400ce50 2024-02-19T21:07:05.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.692+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba0400e580 con 0x7fba0400ce50 2024-02-19T21:07:05.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.692+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:07:05.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.692+0000 7fb9fffff700 10 client.49340.objecter _finish_command 9 = system:0 2024-02-19T21:07:05.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.692+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba0400e410 osd=4 4 2024-02-19T21:07:05.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.692+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 4 9 2024-02-19T21:07:05.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.707+0000 7fba1d08e700 10 client.49340.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:05.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.707+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 10 2024-02-19T21:07:05.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.707+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0400e410 osd=4 3 2024-02-19T21:07:05.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.707+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0400e410 osd=4 3 2024-02-19T21:07:05.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.707+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0400e410 osd=4 3 2024-02-19T21:07:05.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.707+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 10 2024-02-19T21:07:05.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.707+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba0400e410 osd=4 3 2024-02-19T21:07:05.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.707+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 4 10 2024-02-19T21:07:05.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.707+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0400e410 osd=4 4 2024-02-19T21:07:05.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.707+0000 7fba1d08e700 10 client.49340.objecter _send_command 10 2024-02-19T21:07:05.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.707+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba0400e580 con 0x7fba0400ce50 2024-02-19T21:07:05.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba0400e580 con 0x7fba0400ce50 2024-02-19T21:07:05.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:05.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fb9fffff700 10 client.49340.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba0400e410 osd=4 4 2024-02-19T21:07:05.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 4 10 2024-02-19T21:07:05.705 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:07:05.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fba1d08e700 10 client.49340.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:05.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 11 2024-02-19T21:07:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0400e580 osd=5 3 2024-02-19T21:07:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0400e580 osd=5 3 2024-02-19T21:07:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0400e580 osd=5 3 2024-02-19T21:07:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 11 2024-02-19T21:07:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba0400e580 osd=5 3 2024-02-19T21:07:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 5 11 2024-02-19T21:07:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0400e580 osd=5 4 2024-02-19T21:07:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fba1d08e700 10 client.49340.objecter _send_command 11 2024-02-19T21:07:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba0401e660 con 0x7fba0401a960 2024-02-19T21:07:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.709+0000 7fb9fffff700 10 client.49340.objecter ms_handle_connect 0x7fba0401a960 2024-02-19T21:07:05.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.710+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba0401e660 con 0x7fba0401a960 2024-02-19T21:07:05.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.710+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:07:05.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.710+0000 7fb9fffff700 10 client.49340.objecter _finish_command 11 = system:0 2024-02-19T21:07:05.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.710+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba0400e580 osd=5 4 2024-02-19T21:07:05.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.710+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 5 11 2024-02-19T21:07:05.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.725+0000 7fba1d08e700 10 client.49340.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:05.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.725+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 12 2024-02-19T21:07:05.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.725+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0400e580 osd=5 3 2024-02-19T21:07:05.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.725+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0400e580 osd=5 3 2024-02-19T21:07:05.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.725+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0400e580 osd=5 3 2024-02-19T21:07:05.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.725+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 12 2024-02-19T21:07:05.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.725+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba0400e580 osd=5 3 2024-02-19T21:07:05.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.725+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 5 12 2024-02-19T21:07:05.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.725+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0400e580 osd=5 4 2024-02-19T21:07:05.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.725+0000 7fba1d08e700 10 client.49340.objecter _send_command 12 2024-02-19T21:07:05.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.725+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba0401e660 con 0x7fba0401a960 2024-02-19T21:07:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba0401e660 con 0x7fba0401a960 2024-02-19T21:07:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fb9fffff700 10 client.49340.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba0400e580 osd=5 4 2024-02-19T21:07:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 5 12 2024-02-19T21:07:05.723 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:07:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fba1d08e700 10 client.49340.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 13 2024-02-19T21:07:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0401e660 osd=6 3 2024-02-19T21:07:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0401e660 osd=6 3 2024-02-19T21:07:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0401e660 osd=6 3 2024-02-19T21:07:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 13 2024-02-19T21:07:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba0401e660 osd=6 3 2024-02-19T21:07:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 6 13 2024-02-19T21:07:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0401e660 osd=6 4 2024-02-19T21:07:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fba1d08e700 10 client.49340.objecter _send_command 13 2024-02-19T21:07:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba0401f840 con 0x7fba04020320 2024-02-19T21:07:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.727+0000 7fb9fffff700 10 client.49340.objecter ms_handle_connect 0x7fba04020320 2024-02-19T21:07:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.728+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba0401f840 con 0x7fba04020320 2024-02-19T21:07:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.728+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:07:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.728+0000 7fb9fffff700 10 client.49340.objecter _finish_command 13 = system:0 2024-02-19T21:07:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.728+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba0401e660 osd=6 4 2024-02-19T21:07:05.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.728+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 6 13 2024-02-19T21:07:05.734 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:05.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.742+0000 7fba1d08e700 10 client.49340.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:05.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.742+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 14 2024-02-19T21:07:05.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.742+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0401e660 osd=6 3 2024-02-19T21:07:05.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.742+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0401e660 osd=6 3 2024-02-19T21:07:05.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.742+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0401e660 osd=6 3 2024-02-19T21:07:05.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.742+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 14 2024-02-19T21:07:05.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.742+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba0401e660 osd=6 3 2024-02-19T21:07:05.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.742+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 6 14 2024-02-19T21:07:05.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.742+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0401e660 osd=6 4 2024-02-19T21:07:05.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.742+0000 7fba1d08e700 10 client.49340.objecter _send_command 14 2024-02-19T21:07:05.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.742+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba0401f840 con 0x7fba04020320 2024-02-19T21:07:05.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.744+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba0401f840 con 0x7fba04020320 2024-02-19T21:07:05.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.744+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:05.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.744+0000 7fb9fffff700 10 client.49340.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.744+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba0401e660 osd=6 4 2024-02-19T21:07:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.744+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 6 14 2024-02-19T21:07:05.741 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:07:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.745+0000 7fba1d08e700 10 client.49340.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:05.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.745+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 15 2024-02-19T21:07:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.745+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0401f840 osd=7 3 2024-02-19T21:07:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.745+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0401f840 osd=7 3 2024-02-19T21:07:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.745+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0401f840 osd=7 3 2024-02-19T21:07:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.745+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 15 2024-02-19T21:07:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.745+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba0401f840 osd=7 3 2024-02-19T21:07:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.745+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 7 15 2024-02-19T21:07:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.745+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0401f840 osd=7 4 2024-02-19T21:07:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.745+0000 7fba1d08e700 10 client.49340.objecter _send_command 15 2024-02-19T21:07:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.745+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba0401fc70 con 0x7fba04023520 2024-02-19T21:07:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.745+0000 7fb9fffff700 10 client.49340.objecter ms_handle_connect 0x7fba04023520 2024-02-19T21:07:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.746+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba0401fc70 con 0x7fba04023520 2024-02-19T21:07:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.746+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:07:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.746+0000 7fb9fffff700 10 client.49340.objecter _finish_command 15 = system:0 2024-02-19T21:07:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.746+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba0401f840 osd=7 4 2024-02-19T21:07:05.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.746+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 7 15 2024-02-19T21:07:05.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.762+0000 7fba1d08e700 10 client.49340.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:05.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.762+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign -1 16 2024-02-19T21:07:05.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.762+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0401f840 osd=7 3 2024-02-19T21:07:05.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.762+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0401f840 osd=7 3 2024-02-19T21:07:05.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.762+0000 7fba1d08e700 20 client.49340.objecter _get_session s=0x7fba0401f840 osd=7 3 2024-02-19T21:07:05.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.762+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_remove -1 16 2024-02-19T21:07:05.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.762+0000 7fba1d08e700 20 client.49340.objecter get_session s=0x7fba0401f840 osd=7 3 2024-02-19T21:07:05.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.762+0000 7fba1d08e700 15 client.49340.objecter _session_command_op_assign 7 16 2024-02-19T21:07:05.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.762+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0401f840 osd=7 4 2024-02-19T21:07:05.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.762+0000 7fba1d08e700 10 client.49340.objecter _send_command 16 2024-02-19T21:07:05.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.762+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba0401fc70 con 0x7fba04023520 2024-02-19T21:07:05.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.763+0000 7fb9fffff700 1 -- 172.21.15.163:0/2965375004 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba0401fc70 con 0x7fba04023520 2024-02-19T21:07:05.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.763+0000 7fb9fffff700 10 client.49340.objecter ms_dispatch 0x7fba18001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:05.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.763+0000 7fb9fffff700 10 client.49340.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.763+0000 7fb9fffff700 20 client.49340.objecter put_session s=0x7fba0401f840 osd=7 4 2024-02-19T21:07:05.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.763+0000 7fb9fffff700 15 client.49340.objecter _session_command_op_remove 7 16 2024-02-19T21:07:05.760 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:07:05.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 10 librados: watch_flush enter 2024-02-19T21:07:05.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 10 librados: watch_flush exit 2024-02-19T21:07:05.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 10 client.49340.objecter close_session for osd.0 2024-02-19T21:07:05.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 >> v1:172.21.15.163:6801/1559418115 conn(0x7fba04004530 legacy=0x7fba04002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:05.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba040051a0 osd=0 1 2024-02-19T21:07:05.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 10 client.49340.objecter close_session for osd.1 2024-02-19T21:07:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 >> v1:172.21.15.163:6805/3125047432 conn(0x7fba04004150 legacy=0x7fba0400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba04003fe0 osd=1 1 2024-02-19T21:07:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 10 client.49340.objecter close_session for osd.2 2024-02-19T21:07:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 >> v1:172.21.15.163:6809/1563158487 conn(0x7fba0400d2f0 legacy=0x7fba0400d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba04003630 osd=2 1 2024-02-19T21:07:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 10 client.49340.objecter close_session for osd.3 2024-02-19T21:07:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 >> v1:172.21.15.163:6813/2112279769 conn(0x7fba04011e80 legacy=0x7fba04014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba04003480 osd=3 1 2024-02-19T21:07:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 10 client.49340.objecter close_session for osd.4 2024-02-19T21:07:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 >> v1:172.21.15.193:6800/3129716965 conn(0x7fba0400ce50 legacy=0x7fba04015a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1759e700 1 -- 172.21.15.163:0/2965375004 reap_dead start 2024-02-19T21:07:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1759e700 1 -- 172.21.15.163:0/2965375004 reap_dead start 2024-02-19T21:07:05.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0400e410 osd=4 1 2024-02-19T21:07:05.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 10 client.49340.objecter close_session for osd.5 2024-02-19T21:07:05.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 >> v1:172.21.15.193:6804/805161991 conn(0x7fba0401a960 legacy=0x7fba0401cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:05.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0400e580 osd=5 1 2024-02-19T21:07:05.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 10 client.49340.objecter close_session for osd.6 2024-02-19T21:07:05.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 >> v1:172.21.15.193:6808/787604615 conn(0x7fba04020320 legacy=0x7fba040207a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:05.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0401e660 osd=6 1 2024-02-19T21:07:05.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 10 client.49340.objecter close_session for osd.7 2024-02-19T21:07:05.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 >> v1:172.21.15.193:6812/157927432 conn(0x7fba04023520 legacy=0x7fba04025910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:05.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 20 client.49340.objecter put_session s=0x7fba0401f840 osd=7 1 2024-02-19T21:07:05.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 20 client.49340.objecter shutdown clearing up homeless session... 2024-02-19T21:07:05.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 10 client.49340.objecter successfully canceled tick 2024-02-19T21:07:05.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 >> v1:172.21.15.163:6800/1552514681 conn(0x7fba00063b80 legacy=0x7fba00066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:05.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 >> v1:172.21.15.163:6790/0 conn(0x7fba181042c0 legacy=0x7fba181159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:05.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1759e700 1 -- 172.21.15.163:0/2965375004 reap_dead start 2024-02-19T21:07:05.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 shutdown_connections 2024-02-19T21:07:05.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 >> 172.21.15.163:0/2965375004 conn(0x7fba1806cc00 msgr2=0x7fba18107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:05.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 shutdown_connections 2024-02-19T21:07:05.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 1 -- 172.21.15.163:0/2965375004 wait complete. 2024-02-19T21:07:05.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:05.764+0000 7fba1d08e700 1 librados: shutdown 2024-02-19T21:07:05.764 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.764 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.764 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.765 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.765 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.765 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.765 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.765 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:07:05.765 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:05.778 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:05.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:05.882 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:05.911 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:05.978 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:07:06.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:05.156098+0000 mgr.y (mgr.14550) 262 : cluster [DBG] pgmap v338: 289 pgs: 8 creating+peering, 281 active+clean; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-19T21:07:06.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:05.196764+0000 mon.a (mon.0) 1703 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-02-19T21:07:06.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:05.225478+0000 mon.b (mon.1) 378 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:06.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:05.226184+0000 mon.a (mon.0) 1704 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:06.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:05.227212+0000 mon.b (mon.1) 379 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:06.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:05.227789+0000 mon.a (mon.0) 1705 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:06.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:05.228722+0000 mon.b (mon.1) 380 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:06.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:05.228901+0000 mon.a (mon.0) 1706 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:06.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:05.229243+0000 mon.b (mon.1) 381 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:06.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:05.229422+0000 mon.a (mon.0) 1707 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:06.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:05.337059+0000 mon.c (mon.2) 285 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:06.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:05.619668+0000 mon.c (mon.2) 286 : audit [DBG] from='client.? v1:172.21.15.163:0/2965375004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:06.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:05.156098+0000 mgr.y (mgr.14550) 262 : cluster [DBG] pgmap v338: 289 pgs: 8 creating+peering, 281 active+clean; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-19T21:07:06.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:05.196764+0000 mon.a (mon.0) 1703 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-02-19T21:07:06.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:05.225478+0000 mon.b (mon.1) 378 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:06.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:05.226184+0000 mon.a (mon.0) 1704 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:06.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:05.227212+0000 mon.b (mon.1) 379 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:06.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:05.227789+0000 mon.a (mon.0) 1705 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:06.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:05.228722+0000 mon.b (mon.1) 380 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:06.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:05.228901+0000 mon.a (mon.0) 1706 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:06.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:05.229243+0000 mon.b (mon.1) 381 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:06.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:05.229422+0000 mon.a (mon.0) 1707 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:06.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:05.337059+0000 mon.c (mon.2) 285 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:06.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:05.619668+0000 mon.c (mon.2) 286 : audit [DBG] from='client.? v1:172.21.15.163:0/2965375004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:06.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:05.156098+0000 mgr.y (mgr.14550) 262 : cluster [DBG] pgmap v338: 289 pgs: 8 creating+peering, 281 active+clean; 9.0 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-19T21:07:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:05.196764+0000 mon.a (mon.0) 1703 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-02-19T21:07:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:05.225478+0000 mon.b (mon.1) 378 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:05.226184+0000 mon.a (mon.0) 1704 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:05.227212+0000 mon.b (mon.1) 379 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:05.227789+0000 mon.a (mon.0) 1705 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:05.228722+0000 mon.b (mon.1) 380 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:05.228901+0000 mon.a (mon.0) 1706 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:05.229243+0000 mon.b (mon.1) 381 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:05.229422+0000 mon.a (mon.0) 1707 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:05.337059+0000 mon.c (mon.2) 285 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:06.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:05.619668+0000 mon.c (mon.2) 286 : audit [DBG] from='client.? v1:172.21.15.163:0/2965375004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:07.439 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:05.638403+0000 osd.1 (osd.1) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:07.439 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:05.638557+0000 osd.1 (osd.1) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:07.439 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:06.198436+0000 mon.a (mon.0) 1708 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:07.439 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:06.198559+0000 mon.a (mon.0) 1709 : audit [INF] from='client.48151 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:07.439 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:06.198614+0000 mon.a (mon.0) 1710 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-02-19T21:07:07.439 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:06.200990+0000 mon.c (mon.2) 287 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:07.439 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:06.203657+0000 mon.b (mon.1) 382 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:07.439 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:06.203946+0000 mon.b (mon.1) 383 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-27"}]: dispatch 2024-02-19T21:07:07.440 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:06.205538+0000 mon.a (mon.0) 1711 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:07.440 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:06.205798+0000 mon.a (mon.0) 1712 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:07.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:06.206337+0000 mon.a (mon.0) 1713 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-27"}]: dispatch 2024-02-19T21:07:07.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:06.337860+0000 mon.c (mon.2) 288 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:07.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:06.492398+0000 osd.0 (osd.0) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:07.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:06.492522+0000 osd.0 (osd.0) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:07.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:07.202348+0000 mon.a (mon.0) 1714 : audit [INF] from='client.49316 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi163-83845-35"}]': finished 2024-02-19T21:07:07.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:05.638403+0000 osd.1 (osd.1) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:07.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:05.638557+0000 osd.1 (osd.1) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:07.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:06.198436+0000 mon.a (mon.0) 1708 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:07.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:06.198559+0000 mon.a (mon.0) 1709 : audit [INF] from='client.48151 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:07.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:06.198614+0000 mon.a (mon.0) 1710 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-02-19T21:07:07.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:06.200990+0000 mon.c (mon.2) 287 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:07.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:06.203657+0000 mon.b (mon.1) 382 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:07.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:06.203946+0000 mon.b (mon.1) 383 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-27"}]: dispatch 2024-02-19T21:07:07.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:06.205538+0000 mon.a (mon.0) 1711 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:07.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:06.205798+0000 mon.a (mon.0) 1712 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:07.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:06.206337+0000 mon.a (mon.0) 1713 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-27"}]: dispatch 2024-02-19T21:07:07.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:06.337860+0000 mon.c (mon.2) 288 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:07.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:06.492398+0000 osd.0 (osd.0) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:07.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:06.492522+0000 osd.0 (osd.0) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:07.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:07.202348+0000 mon.a (mon.0) 1714 : audit [INF] from='client.49316 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi163-83845-35"}]': finished 2024-02-19T21:07:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:05.638403+0000 osd.1 (osd.1) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:05.638557+0000 osd.1 (osd.1) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:06.198436+0000 mon.a (mon.0) 1708 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:06.198559+0000 mon.a (mon.0) 1709 : audit [INF] from='client.48151 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-45", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:06.198614+0000 mon.a (mon.0) 1710 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-02-19T21:07:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:06.200990+0000 mon.c (mon.2) 287 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:06.203657+0000 mon.b (mon.1) 382 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:06.203946+0000 mon.b (mon.1) 383 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-27"}]: dispatch 2024-02-19T21:07:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:06.205538+0000 mon.a (mon.0) 1711 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:07.770 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:06.205798+0000 mon.a (mon.0) 1712 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:07.770 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:06.206337+0000 mon.a (mon.0) 1713 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-27"}]: dispatch 2024-02-19T21:07:07.770 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:06.337860+0000 mon.c (mon.2) 288 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:07.770 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:06.492398+0000 osd.0 (osd.0) 119 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:07.770 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:06.492522+0000 osd.0 (osd.0) 120 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:07.770 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:07.202348+0000 mon.a (mon.0) 1714 : audit [INF] from='client.49316 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi163-83845-35"}]': finished 2024-02-19T21:07:08.066 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:07:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:07:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:07:08.162 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:08.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:08.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:08.266 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:08.297 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:08.317 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:08.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:08.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:08.421 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:08.450 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:08.470 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:08.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:08.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:08.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:07.156707+0000 mgr.y (mgr.14550) 263 : cluster [DBG] pgmap v341: 249 pgs: 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-19T21:07:08.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:07.205752+0000 mon.c (mon.2) 289 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:08.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:07.338542+0000 mon.c (mon.2) 290 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:08.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:07.390966+0000 osd.3 (osd.3) 129 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:08.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:07.391086+0000 osd.3 (osd.3) 130 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:08.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:07.436097+0000 mon.a (mon.0) 1715 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-27"}]': finished 2024-02-19T21:07:08.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:07.436174+0000 mon.a (mon.0) 1716 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-02-19T21:07:08.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:07.437075+0000 mon.b (mon.1) 384 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-27", "mode": "writeback"}]: dispatch 2024-02-19T21:07:08.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:07.437129+0000 mon.a (mon.0) 1717 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:08.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:07.441596+0000 mon.a (mon.0) 1718 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-27", "mode": "writeback"}]: dispatch 2024-02-19T21:07:08.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:08.197837+0000 mon.a (mon.0) 1719 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:08.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:08.202551+0000 mon.a (mon.0) 1720 : audit [INF] from='client.48151 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-45"}]': finished 2024-02-19T21:07:08.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:08.202646+0000 mon.a (mon.0) 1721 : audit [INF] from='client.49316 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi163-83845-35"}]': finished 2024-02-19T21:07:08.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:08.202708+0000 mon.a (mon.0) 1722 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-27", "mode": "writeback"}]': finished 2024-02-19T21:07:08.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:08.202752+0000 mon.a (mon.0) 1723 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-02-19T21:07:08.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:08.226225+0000 mon.c (mon.2) 291 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:08.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:08.226732+0000 mon.a (mon.0) 1724 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:08.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:08.227447+0000 mon.c (mon.2) 292 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:08.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:08.227683+0000 mon.a (mon.0) 1725 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:08.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:08.228037+0000 mon.c (mon.2) 293 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:08.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:08.228271+0000 mon.a (mon.0) 1726 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:08.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:08.339196+0000 mon.c (mon.2) 294 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:08.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:08.339681+0000 mon.c (mon.2) 295 : audit [INF] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]: dispatch 2024-02-19T21:07:08.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:08.339927+0000 mon.a (mon.0) 1727 : audit [INF] from='client.34832 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]: dispatch 2024-02-19T21:07:08.573 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:08.602 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:08.623 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:08.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:08.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:08.673 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:07.156707+0000 mgr.y (mgr.14550) 263 : cluster [DBG] pgmap v341: 249 pgs: 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-19T21:07:08.673 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:07.205752+0000 mon.c (mon.2) 289 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:08.673 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:07.338542+0000 mon.c (mon.2) 290 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:08.673 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:07.390966+0000 osd.3 (osd.3) 129 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:08.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:07.391086+0000 osd.3 (osd.3) 130 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:08.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:07.436097+0000 mon.a (mon.0) 1715 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-27"}]': finished 2024-02-19T21:07:08.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:07.436174+0000 mon.a (mon.0) 1716 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-02-19T21:07:08.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:07.437075+0000 mon.b (mon.1) 384 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-27", "mode": "writeback"}]: dispatch 2024-02-19T21:07:08.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:07.437129+0000 mon.a (mon.0) 1717 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:08.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:07.441596+0000 mon.a (mon.0) 1718 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-27", "mode": "writeback"}]: dispatch 2024-02-19T21:07:08.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:08.197837+0000 mon.a (mon.0) 1719 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:08.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:08.202551+0000 mon.a (mon.0) 1720 : audit [INF] from='client.48151 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-45"}]': finished 2024-02-19T21:07:08.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:08.202646+0000 mon.a (mon.0) 1721 : audit [INF] from='client.49316 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi163-83845-35"}]': finished 2024-02-19T21:07:08.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:08.202708+0000 mon.a (mon.0) 1722 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-27", "mode": "writeback"}]': finished 2024-02-19T21:07:08.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:08.202752+0000 mon.a (mon.0) 1723 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-02-19T21:07:08.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:08.226225+0000 mon.c (mon.2) 291 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:08.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:08.226732+0000 mon.a (mon.0) 1724 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:08.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:08.227447+0000 mon.c (mon.2) 292 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:08.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:08.227683+0000 mon.a (mon.0) 1725 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:08.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:08.228037+0000 mon.c (mon.2) 293 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:08.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:08.228271+0000 mon.a (mon.0) 1726 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:08.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:08.339196+0000 mon.c (mon.2) 294 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:08.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:08.339681+0000 mon.c (mon.2) 295 : audit [INF] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]: dispatch 2024-02-19T21:07:08.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:08.339927+0000 mon.a (mon.0) 1727 : audit [INF] from='client.34832 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]: dispatch 2024-02-19T21:07:08.675 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:07.156707+0000 mgr.y (mgr.14550) 263 : cluster [DBG] pgmap v341: 249 pgs: 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-02-19T21:07:08.675 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:07.205752+0000 mon.c (mon.2) 289 : audit [INF] from='client.? v1:172.21.15.163:0/89414708' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:08.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:07.338542+0000 mon.c (mon.2) 290 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:08.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:07.390966+0000 osd.3 (osd.3) 129 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:08.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:07.391086+0000 osd.3 (osd.3) 130 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:08.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:07.436097+0000 mon.a (mon.0) 1715 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-27"}]': finished 2024-02-19T21:07:08.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:07.436174+0000 mon.a (mon.0) 1716 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-02-19T21:07:08.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:07.437075+0000 mon.b (mon.1) 384 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-27", "mode": "writeback"}]: dispatch 2024-02-19T21:07:08.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:07.437129+0000 mon.a (mon.0) 1717 : audit [INF] from='client.49316 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi163-83845-35"}]: dispatch 2024-02-19T21:07:08.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:07.441596+0000 mon.a (mon.0) 1718 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-27", "mode": "writeback"}]: dispatch 2024-02-19T21:07:08.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:08.197837+0000 mon.a (mon.0) 1719 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:08.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:08.202551+0000 mon.a (mon.0) 1720 : audit [INF] from='client.48151 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-45", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-45"}]': finished 2024-02-19T21:07:08.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:08.202646+0000 mon.a (mon.0) 1721 : audit [INF] from='client.49316 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi163-83845-35"}]': finished 2024-02-19T21:07:08.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:08.202708+0000 mon.a (mon.0) 1722 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-27", "mode": "writeback"}]': finished 2024-02-19T21:07:08.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:08.202752+0000 mon.a (mon.0) 1723 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-02-19T21:07:08.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:08.226225+0000 mon.c (mon.2) 291 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:08.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:08.226732+0000 mon.a (mon.0) 1724 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:08.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:08.227447+0000 mon.c (mon.2) 292 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:08.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:08.227683+0000 mon.a (mon.0) 1725 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:08.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:08.228037+0000 mon.c (mon.2) 293 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:08.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:08.228271+0000 mon.a (mon.0) 1726 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:08.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:08.339196+0000 mon.c (mon.2) 294 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:08.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:08.339681+0000 mon.c (mon.2) 295 : audit [INF] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]: dispatch 2024-02-19T21:07:08.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:08.339927+0000 mon.a (mon.0) 1727 : audit [INF] from='client.34832 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]: dispatch 2024-02-19T21:07:08.726 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:08.754 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:08.852 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:08.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:08.956 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:08.986 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:09.134 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:09.136 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:09.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:09.238 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:09.267 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:09.286 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:09.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:09.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:09.390 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:09.418 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:09.437 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:09.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:09.447 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:07:09.448 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:07:09.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.963+0000 7f5006d2e700 1 -- 172.21.15.163:0/4159030605 >> v1:172.21.15.163:6790/0 conn(0x7f5000100780 legacy=0x7f5000102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:09.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.963+0000 7f5006d2e700 1 -- 172.21.15.163:0/4159030605 shutdown_connections 2024-02-19T21:07:09.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.963+0000 7f5006d2e700 1 -- 172.21.15.163:0/4159030605 >> 172.21.15.163:0/4159030605 conn(0x7f50000f35a0 msgr2=0x7f50000f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:09.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.963+0000 7f5006d2e700 1 -- 172.21.15.163:0/4159030605 shutdown_connections 2024-02-19T21:07:09.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.963+0000 7f5006d2e700 1 -- 172.21.15.163:0/4159030605 wait complete. 2024-02-19T21:07:09.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.964+0000 7f5006d2e700 1 librados: starting msgr at 2024-02-19T21:07:09.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.964+0000 7f5006d2e700 1 librados: starting objecter 2024-02-19T21:07:09.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.964+0000 7f5006d2e700 1 Processor -- start 2024-02-19T21:07:09.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.964+0000 7f5006d2e700 1 -- start start 2024-02-19T21:07:09.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.964+0000 7f5006d2e700 1 librados: setting wanted keys 2024-02-19T21:07:09.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.964+0000 7f5006d2e700 1 librados: calling monclient init 2024-02-19T21:07:09.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.964+0000 7f5006d2e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f50000fcee0 con 0x7f5000100780 2024-02-19T21:07:09.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.964+0000 7f5006d2e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f50000fd070 con 0x7f50000fd980 2024-02-19T21:07:09.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.964+0000 7f5006d2e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f50000fd260 con 0x7f50001042c0 2024-02-19T21:07:09.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.964+0000 7f4ffffff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f5000100780 0x7f5000114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:42846/0 (socket says 172.21.15.163:42846) 2024-02-19T21:07:09.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.964+0000 7f4ffffff700 1 -- 172.21.15.163:0/2668817751 learned_addr learned my addr 172.21.15.163:0/2668817751 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:09.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.964+0000 7f50052cb700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f50001042c0 0x7f5000118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45930/0 (socket says 172.21.15.163:45930) 2024-02-19T21:07:09.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.964+0000 7f4ffdffb700 10 client.?.objecter ms_handle_connect 0x7f50001042c0 2024-02-19T21:07:09.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:09.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 10 client.?.objecter ms_handle_connect 0x7f5000100780 2024-02-19T21:07:09.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:09.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 10 client.?.objecter ms_handle_connect 0x7f50000fd980 2024-02-19T21:07:09.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:09.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839270369 0 0) 0x7f50000fd260 con 0x7f50001042c0 2024-02-19T21:07:09.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff0003030 con 0x7f50001042c0 2024-02-19T21:07:09.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200994116 0 0) 0x7f50000fcee0 con 0x7f5000100780 2024-02-19T21:07:09.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50000fd260 con 0x7f5000100780 2024-02-19T21:07:09.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277034115 0 0) 0x7f50000fd070 con 0x7f50000fd980 2024-02-19T21:07:09.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50000fcee0 con 0x7f50000fd980 2024-02-19T21:07:09.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1467445398 0 0) 0x7f4ff0003030 con 0x7f50001042c0 2024-02-19T21:07:09.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50000fd070 con 0x7f50001042c0 2024-02-19T21:07:09.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4ff8001bc0 con 0x7f50001042c0 2024-02-19T21:07:09.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3442454470 0 0) 0x7f50000fd260 con 0x7f5000100780 2024-02-19T21:07:09.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ff0003030 con 0x7f5000100780 2024-02-19T21:07:09.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2432247422 0 0) 0x7f50000fcee0 con 0x7f50000fd980 2024-02-19T21:07:09.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50000fd260 con 0x7f50000fd980 2024-02-19T21:07:09.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4fec000fe0 con 0x7f5000100780 2024-02-19T21:07:09.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4ff40017b0 con 0x7f50000fd980 2024-02-19T21:07:09.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 435902418 0 0) 0x7f50000fd070 con 0x7f50001042c0 2024-02-19T21:07:09.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 >> v1:172.21.15.193:6789/0 conn(0x7f50000fd980 legacy=0x7f50000fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:09.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 >> v1:172.21.15.163:6789/0 conn(0x7f5000100780 legacy=0x7f5000114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:09.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500011cb90 con 0x7f50001042c0 2024-02-19T21:07:09.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f5006d2e700 1 -- 172.21.15.163:0/2668817751 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5000119bc0 con 0x7f50001042c0 2024-02-19T21:07:09.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f4ff8002690 con 0x7f50001042c0 2024-02-19T21:07:09.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4ff8005dd0 con 0x7f50001042c0 2024-02-19T21:07:09.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f5006d2e700 10 client.49352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:09.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f5006d2e700 1 -- 172.21.15.163:0/2668817751 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f500011a030 con 0x7f50001042c0 2024-02-19T21:07:09.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.965+0000 7f5006d2e700 1 librados: init done 2024-02-19T21:07:09.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.966+0000 7f5006d2e700 1 -- 172.21.15.163:0/2668817751 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500005f660 con 0x7f50001042c0 2024-02-19T21:07:09.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.967+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f4ff8018d00 con 0x7f50001042c0 2024-02-19T21:07:09.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.967+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(248..248 src has 43..248) v4 ==== 9232+0+0 (unknown 2122498312 0 0) 0x7f4ff8079610 con 0x7f50001042c0 2024-02-19T21:07:09.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.967+0000 7f4ffdffb700 10 client.49352.objecter ms_dispatch 0x7f5000001550 osd_map(248..248 src has 43..248) v4 2024-02-19T21:07:09.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.967+0000 7f4ffdffb700 3 client.49352.objecter handle_osd_map got epochs [248,248] > 0 2024-02-19T21:07:09.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.967+0000 7f4ffdffb700 3 client.49352.objecter handle_osd_map decoding full epoch 248 2024-02-19T21:07:09.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.967+0000 7f4ffdffb700 20 client.49352.objecter dump_active .. 0 homeless 2024-02-19T21:07:09.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.967+0000 7f4ffdffb700 10 client.49352.objecter ms_handle_connect 0x7f4ff0063e80 2024-02-19T21:07:09.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:08.972+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f4ff804c590 con 0x7f50001042c0 2024-02-19T21:07:09.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:09.139+0000 7f5006d2e700 1 -- 172.21.15.163:0/2668817751 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5000109f70 con 0x7f4ff0063e80 2024-02-19T21:07:09.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:09.158+0000 7f4ffdffb700 1 -- 172.21.15.163:0/2668817751 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+650532 (unknown 2965378022 0 2886820031) 0x7f5000109f70 con 0x7f4ff0063e80 2024-02-19T21:07:09.464 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:07:09.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:09.462+0000 7f5006d2e700 10 librados: watch_flush enter 2024-02-19T21:07:09.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:09.462+0000 7f5006d2e700 10 librados: watch_flush exit 2024-02-19T21:07:09.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:09.462+0000 7f5006d2e700 20 client.49352.objecter shutdown clearing up homeless session... 2024-02-19T21:07:09.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:09.462+0000 7f5006d2e700 10 client.49352.objecter successfully canceled tick 2024-02-19T21:07:09.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:09.462+0000 7f5006d2e700 1 -- 172.21.15.163:0/2668817751 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4ff0063e80 legacy=0x7f4ff0066340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:09.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:09.462+0000 7f5006d2e700 1 -- 172.21.15.163:0/2668817751 >> v1:172.21.15.163:6790/0 conn(0x7f50001042c0 legacy=0x7f5000118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:09.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:09.462+0000 7f5006d2e700 1 -- 172.21.15.163:0/2668817751 shutdown_connections 2024-02-19T21:07:09.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:09.462+0000 7f5006d2e700 1 -- 172.21.15.163:0/2668817751 >> 172.21.15.163:0/2668817751 conn(0x7f50000f35a0 msgr2=0x7f5000107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:09.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:09.462+0000 7f5006d2e700 1 -- 172.21.15.163:0/2668817751 shutdown_connections 2024-02-19T21:07:09.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:09.462+0000 7f5006d2e700 1 -- 172.21.15.163:0/2668817751 wait complete. 2024-02-19T21:07:09.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:09.462+0000 7f5006d2e700 1 librados: shutdown 2024-02-19T21:07:09.471 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:07:09.480 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:09.540 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:09.570 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:09.919 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:09.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:10.022 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:10.051 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:10.243 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:09.098011+0000 mon.a (mon.0) 1728 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:10.243 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:09.141382+0000 mgr.y (mgr.14550) 264 : audit [DBG] from='client.49352 v1:172.21.15.163:0/2668817751' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:07:10.243 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:09.159761+0000 mgr.y (mgr.14550) 265 : cluster [DBG] pgmap v344: 257 pgs: 8 unknown, 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:07:10.243 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:09.196045+0000 mon.a (mon.0) 1729 : audit [INF] from='client.49346 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:10.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:09.196133+0000 mon.a (mon.0) 1730 : audit [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]': finished 2024-02-19T21:07:10.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:09.196194+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-02-19T21:07:10.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:09.198868+0000 mon.c (mon.2) 296 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:10.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:09.200452+0000 mon.c (mon.2) 297 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:10.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:09.200746+0000 mon.c (mon.2) 298 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi163-83845-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:10.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:09.203437+0000 mon.c (mon.2) 299 : audit [INF] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-19T21:07:10.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:09.209728+0000 mon.a (mon.0) 1732 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi163-83845-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:10.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:09.210159+0000 mon.a (mon.0) 1733 : audit [INF] from='client.34832 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-19T21:07:10.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:09.231796+0000 mon.b (mon.1) 385 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:10.244 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:09.232156+0000 mon.a (mon.0) 1734 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:10.244 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:09.098011+0000 mon.a (mon.0) 1728 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:10.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:09.141382+0000 mgr.y (mgr.14550) 264 : audit [DBG] from='client.49352 v1:172.21.15.163:0/2668817751' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:07:10.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:09.159761+0000 mgr.y (mgr.14550) 265 : cluster [DBG] pgmap v344: 257 pgs: 8 unknown, 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:07:10.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:09.196045+0000 mon.a (mon.0) 1729 : audit [INF] from='client.49346 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:10.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:09.196133+0000 mon.a (mon.0) 1730 : audit [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]': finished 2024-02-19T21:07:10.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:09.196194+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-02-19T21:07:10.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:09.198868+0000 mon.c (mon.2) 296 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:10.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:09.200452+0000 mon.c (mon.2) 297 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:10.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:09.200746+0000 mon.c (mon.2) 298 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi163-83845-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:10.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:09.203437+0000 mon.c (mon.2) 299 : audit [INF] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-19T21:07:10.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:09.209728+0000 mon.a (mon.0) 1732 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi163-83845-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:10.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:09.210159+0000 mon.a (mon.0) 1733 : audit [INF] from='client.34832 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-19T21:07:10.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:09.231796+0000 mon.b (mon.1) 385 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:10.245 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:09.232156+0000 mon.a (mon.0) 1734 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:10.391 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-02-19T21:07:08.200327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":9060,"num_objects":328,"num_object_clones":1,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":328,"num_whiteouts":0,"num_read":1148,"num_read_kb":808,"num_write":1098,"num_write_kb":650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":1830,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2262,"ondisk_log_size":2262,"up":747,"acting":747,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":777,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2474756,"kb_used_data":31364,"kb_used_omap":0,"kb_used_meta":2443264,"kb_avail":747486460,"statfs":{"total":767960285184,"available":765426135040,"internally_reserved":0,"allocated":32116736,"data_stored":17648308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2501902336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":288,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":20,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.006965"},"pg_stats":[{"pgid":"164.1f","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.219268+0000","last_change":"2024-02-19T21:07:04.219268+0000","last_active":"2024-02-19T21:07:04.219268+0000","last_peered":"2024-02-19T21:07:04.219268+0000","last_clean":"2024-02-19T21:07:04.219268+0000","last_became_active":"2024-02-19T21:07:04.218906+0000","last_became_peered":"2024-02-19T21:07:04.218906+0000","last_unstale":"2024-02-19T21:07:04.219268+0000","last_undegraded":"2024-02-19T21:07:04.219268+0000","last_fullsized":"2024-02-19T21:07:04.219268+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"164.1e","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.215648+0000","last_change":"2024-02-19T21:07:04.215648+0000","last_active":"2024-02-19T21:07:04.215648+0000","last_peered":"2024-02-19T21:07:04.215648+0000","last_clean":"2024-02-19T21:07:04.215648+0000","last_became_active":"2024-02-19T21:07:04.215354+0000","last_became_peered":"2024-02-19T21:07:04.215354+0000","last_unstale":"2024-02-19T21:07:04.215648+0000","last_undegraded":"2024-02-19T21:07:04.215648+0000","last_fullsized":"2024-02-19T21:07:04.215648+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"164.1d","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.215386+0000","last_change":"2024-02-19T21:07:04.215386+0000","last_active":"2024-02-19T21:07:04.215386+0000","last_peered":"2024-02-19T21:07:04.215386+0000","last_clean":"2024-02-19T21:07:04.215386+0000","last_became_active":"2024-02-19T21:07:04.215082+0000","last_became_peered":"2024-02-19T21:07:04.215082+0000","last_unstale":"2024-02-19T21:07:04.215386+0000","last_undegraded":"2024-02-19T21:07:04.215386+0000","last_fullsized":"2024-02-19T21:07:04.215386+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"164.1c","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.223292+0000","last_change":"2024-02-19T21:07:04.223292+0000","last_active":"2024-02-19T21:07:04.223292+0000","last_peered":"2024-02-19T21:07:04.223292+0000","last_clean":"2024-02-19T21:07:04.223292+0000","last_became_active":"2024-02-19T21:07:04.223008+0000","last_became_peered":"2024-02-19T21:07:04.223008+0000","last_unstale":"2024-02-19T21:07:04.223292+0000","last_undegraded":"2024-02-19T21:07:04.223292+0000","last_fullsized":"2024-02-19T21:07:04.223292+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"164.1b","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.223625+0000","last_change":"2024-02-19T21:07:04.223625+0000","last_active":"2024-02-19T21:07:04.223625+0000","last_peered":"2024-02-19T21:07:04.223625+0000","last_clean":"2024-02-19T21:07:04.223625+0000","last_became_active":"2024-02-19T21:07:04.223340+0000","last_became_peered":"2024-02-19T21:07:04.223340+0000","last_unstale":"2024-02-19T21:07:04.223625+0000","last_undegraded":"2024-02-19T21:07:04.223625+0000","last_fullsized":"2024-02-19T21:07:04.223625+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"164.1a","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.234756+0000","last_change":"2024-02-19T21:07:04.234756+0000","last_active":"2024-02-19T21:07:04.234756+0000","last_peered":"2024-02-19T21:07:04.234756+0000","last_clean":"2024-02-19T21:07:04.234756+0000","last_became_active":"2024-02-19T21:07:04.234276+0000","last_became_peered":"2024-02-19T21:07:04.234276+0000","last_unstale":"2024-02-19T21:07:04.234756+0000","last_undegraded":"2024-02-19T21:07:04.234756+0000","last_fullsized":"2024-02-19T21:07:04.234756+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"164.19","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.222830+0000","last_change":"2024-02-19T21:07:04.222830+0000","last_active":"2024-02-19T21:07:04.222830+0000","last_peered":"2024-02-19T21:07:04.222830+0000","last_clean":"2024-02-19T21:07:04.222830+0000","last_became_active":"2024-02-19T21:07:04.222499+0000","last_became_peered":"2024-02-19T21:07:04.222499+0000","last_unstale":"2024-02-19T21:07:04.222830+0000","last_undegraded":"2024-02-19T21:07:04.222830+0000","last_fullsized":"2024-02-19T21:07:04.222830+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"164.18","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.225501+0000","last_change":"2024-02-19T21:07:04.225501+0000","last_active":"2024-02-19T21:07:04.225501+0000","last_peered":"2024-02-19T21:07:04.225501+0000","last_clean":"2024-02-19T21:07:04.225501+0000","last_became_active":"2024-02-19T21:07:04.225179+0000","last_became_peered":"2024-02-19T21:07:04.225179+0000","last_unstale":"2024-02-19T21:07:04.225501+0000","last_undegraded":"2024-02-19T21:07:04.225501+0000","last_fullsized":"2024-02-19T21:07:04.225501+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"164.17","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.230662+0000","last_change":"2024-02-19T21:07:04.230662+0000","last_active":"2024-02-19T21:07:04.230662+0000","last_peered":"2024-02-19T21:07:04.230662+0000","last_clean":"2024-02-19T21:07:04.230662+0000","last_became_active":"2024-02-19T21:07:04.230192+0000","last_became_peered":"2024-02-19T21:07:04.230192+0000","last_unstale":"2024-02-19T21:07:04.230662+0000","last_undegraded":"2024-02-19T21:07:04.230662+0000","last_fullsized":"2024-02-19T21:07:04.230662+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"164.16","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.218476+0000","last_change":"2024-02-19T21:07:04.218476+0000","last_active":"2024-02-19T21:07:04.218476+0000","last_peered":"2024-02-19T21:07:04.218476+0000","last_clean":"2024-02-19T21:07:04.218476+0000","last_became_active":"2024-02-19T21:07:04.218170+0000","last_became_peered":"2024-02-19T21:07:04.218170+0000","last_unstale":"2024-02-19T21:07:04.218476+0000","last_undegraded":"2024-02-19T21:07:04.218476+0000","last_fullsized":"2024-02-19T21:07:04.218476+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"164.15","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.219430+0000","last_change":"2024-02-19T21:07:04.219430+0000","last_active":"2024-02-19T21:07:04.219430+0000","last_peered":"2024-02-19T21:07:04.219430+0000","last_clean":"2024-02-19T21:07:04.219430+0000","last_became_active":"2024-02-19T21:07:04.219119+0000","last_became_peered":"2024-02-19T21:07:04.219119+0000","last_unstale":"2024-02-19T21:07:04.219430+0000","last_undegraded":"2024-02-19T21:07:04.219430+0000","last_fullsized":"2024-02-19T21:07:04.219430+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"164.14","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.218719+0000","last_change":"2024-02-19T21:07:04.218719+0000","last_active":"2024-02-19T21:07:04.218719+0000","last_peered":"2024-02-19T21:07:04.218719+0000","last_clean":"2024-02-19T21:07:04.218719+0000","last_became_active":"2024-02-19T21:07:04.218279+0000","last_became_peered":"2024-02-19T21:07:04.218279+0000","last_unstale":"2024-02-19T21:07:04.218719+0000","last_undegraded":"2024-02-19T21:07:04.218719+0000","last_fullsized":"2024-02-19T21:07:04.218719+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"164.13","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.223960+0000","last_change":"2024-02-19T21:07:04.223960+0000","last_active":"2024-02-19T21:07:04.223960+0000","last_peered":"2024-02-19T21:07:04.223960+0000","last_clean":"2024-02-19T21:07:04.223960+0000","last_became_active":"2024-02-19T21:07:04.223151+0000","last_became_peered":"2024-02-19T21:07:04.223151+0000","last_unstale":"2024-02-19T21:07:04.223960+0000","last_undegraded":"2024-02-19T21:07:04.223960+0000","last_fullsized":"2024-02-19T21:07:04.223960+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"164.12","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.226627+0000","last_change":"2024-02-19T21:07:04.226627+0000","last_active":"2024-02-19T21:07:04.226627+0000","last_peered":"2024-02-19T21:07:04.226627+0000","last_clean":"2024-02-19T21:07:04.226627+0000","last_became_active":"2024-02-19T21:07:04.226384+0000","last_became_peered":"2024-02-19T21:07:04.226384+0000","last_unstale":"2024-02-19T21:07:04.226627+0000","last_undegraded":"2024-02-19T21:07:04.226627+0000","last_fullsized":"2024-02-19T21:07:04.226627+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"164.11","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.213545+0000","last_change":"2024-02-19T21:07:04.213545+0000","last_active":"2024-02-19T21:07:04.213545+0000","last_peered":"2024-02-19T21:07:04.213545+0000","last_clean":"2024-02-19T21:07:04.213545+0000","last_became_active":"2024-02-19T21:07:04.212952+0000","last_became_peered":"2024-02-19T21:07:04.212952+0000","last_unstale":"2024-02-19T21:07:04.213545+0000","last_undegraded":"2024-02-19T21:07:04.213545+0000","last_fullsized":"2024-02-19T21:07:04.213545+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"164.10","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.219453+0000","last_change":"2024-02-19T21:07:04.219453+0000","last_active":"2024-02-19T21:07:04.219453+0000","last_peered":"2024-02-19T21:07:04.219453+0000","last_clean":"2024-02-19T21:07:04.219453+0000","last_became_active":"2024-02-19T21:07:04.219152+0000","last_became_peered":"2024-02-19T21:07:04.219152+0000","last_unstale":"2024-02-19T21:07:04.219453+0000","last_undegraded":"2024-02-19T21:07:04.219453+0000","last_fullsized":"2024-02-19T21:07:04.219453+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"164.f","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.223188+0000","last_change":"2024-02-19T21:07:04.223188+0000","last_active":"2024-02-19T21:07:04.223188+0000","last_peered":"2024-02-19T21:07:04.223188+0000","last_clean":"2024-02-19T21:07:04.223188+0000","last_became_active":"2024-02-19T21:07:04.222510+0000","last_became_peered":"2024-02-19T21:07:04.222510+0000","last_unstale":"2024-02-19T21:07:04.223188+0000","last_undegraded":"2024-02-19T21:07:04.223188+0000","last_fullsized":"2024-02-19T21:07:04.223188+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"164.e","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.235657+0000","last_change":"2024-02-19T21:07:04.235657+0000","last_active":"2024-02-19T21:07:04.235657+0000","last_peered":"2024-02-19T21:07:04.235657+0000","last_clean":"2024-02-19T21:07:04.235657+0000","last_became_active":"2024-02-19T21:07:04.235350+0000","last_became_peered":"2024-02-19T21:07:04.235350+0000","last_unstale":"2024-02-19T21:07:04.235657+0000","last_undegraded":"2024-02-19T21:07:04.235657+0000","last_fullsized":"2024-02-19T21:07:04.235657+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"164.d","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.222647+0000","last_change":"2024-02-19T21:07:04.222647+0000","last_active":"2024-02-19T21:07:04.222647+0000","last_peered":"2024-02-19T21:07:04.222647+0000","last_clean":"2024-02-19T21:07:04.222647+0000","last_became_active":"2024-02-19T21:07:04.222337+0000","last_became_peered":"2024-02-19T21:07:04.222337+0000","last_unstale":"2024-02-19T21:07:04.222647+0000","last_undegraded":"2024-02-19T21:07:04.222647+0000","last_fullsized":"2024-02-19T21:07:04.222647+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"164.c","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.216798+0000","last_change":"2024-02-19T21:07:04.216798+0000","last_active":"2024-02-19T21:07:04.216798+0000","last_peered":"2024-02-19T21:07:04.216798+0000","last_clean":"2024-02-19T21:07:04.216798+0000","last_became_active":"2024-02-19T21:07:04.216510+0000","last_became_peered":"2024-02-19T21:07:04.216510+0000","last_unstale":"2024-02-19T21:07:04.216798+0000","last_undegraded":"2024-02-19T21:07:04.216798+0000","last_fullsized":"2024-02-19T21:07:04.216798+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"164.b","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.218375+0000","last_change":"2024-02-19T21:07:04.218375+0000","last_active":"2024-02-19T21:07:04.218375+0000","last_peered":"2024-02-19T21:07:04.218375+0000","last_clean":"2024-02-19T21:07:04.218375+0000","last_became_active":"2024-02-19T21:07:04.218021+0000","last_became_peered":"2024-02-19T21:07:04.218021+0000","last_unstale":"2024-02-19T21:07:04.218375+0000","last_undegraded":"2024-02-19T21:07:04.218375+0000","last_fullsized":"2024-02-19T21:07:04.218375+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"164.a","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.213881+0000","last_change":"2024-02-19T21:07:04.213881+0000","last_active":"2024-02-19T21:07:04.213881+0000","last_peered":"2024-02-19T21:07:04.213881+0000","last_clean":"2024-02-19T21:07:04.213881+0000","last_became_active":"2024-02-19T21:07:04.213593+0000","last_became_peered":"2024-02-19T21:07:04.213593+0000","last_unstale":"2024-02-19T21:07:04.213881+0000","last_undegraded":"2024-02-19T21:07:04.213881+0000","last_fullsized":"2024-02-19T21:07:04.213881+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"164.9","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.224039+0000","last_change":"2024-02-19T21:07:04.224039+0000","last_active":"2024-02-19T21:07:04.224039+0000","last_peered":"2024-02-19T21:07:04.224039+0000","last_clean":"2024-02-19T21:07:04.224039+0000","last_became_active":"2024-02-19T21:07:04.223752+0000","last_became_peered":"2024-02-19T21:07:04.223752+0000","last_unstale":"2024-02-19T21:07:04.224039+0000","last_undegraded":"2024-02-19T21:07:04.224039+0000","last_fullsized":"2024-02-19T21:07:04.224039+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"164.8","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.229827+0000","last_change":"2024-02-19T21:07:04.229827+0000","last_active":"2024-02-19T21:07:04.229827+0000","last_peered":"2024-02-19T21:07:04.229827+0000","last_clean":"2024-02-19T21:07:04.229827+0000","last_became_active":"2024-02-19T21:07:04.229378+0000","last_became_peered":"2024-02-19T21:07:04.229378+0000","last_unstale":"2024-02-19T21:07:04.229827+0000","last_undegraded":"2024-02-19T21:07:04.229827+0000","last_fullsized":"2024-02-19T21:07:04.229827+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"167.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:07:08.197268+0000","last_change":"2024-02-19T21:07:08.197268+0000","last_active":"2024-02-19T21:07:08.197268+0000","last_peered":"2024-02-19T21:07:08.197268+0000","last_clean":"2024-02-19T21:07:08.197268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:07:08.197268+0000","last_undegraded":"2024-02-19T21:07:08.197268+0000","last_fullsized":"2024-02-19T21:07:08.197268+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_clean_scrub_stamp":"2024-02-19T21:07:08.197268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.7","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.223359+0000","last_change":"2024-02-19T21:07:04.223359+0000","last_active":"2024-02-19T21:07:04.223359+0000","last_peered":"2024-02-19T21:07:04.223359+0000","last_clean":"2024-02-19T21:07:04.223359+0000","last_became_active":"2024-02-19T21:07:04.222893+0000","last_became_peered":"2024-02-19T21:07:04.222893+0000","last_unstale":"2024-02-19T21:07:04.223359+0000","last_undegraded":"2024-02-19T21:07:04.223359+0000","last_fullsized":"2024-02-19T21:07:04.223359+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"167.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:07:08.197268+0000","last_change":"2024-02-19T21:07:08.197268+0000","last_active":"2024-02-19T21:07:08.197268+0000","last_peered":"2024-02-19T21:07:08.197268+0000","last_clean":"2024-02-19T21:07:08.197268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:07:08.197268+0000","last_undegraded":"2024-02-19T21:07:08.197268+0000","last_fullsized":"2024-02-19T21:07:08.197268+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_clean_scrub_stamp":"2024-02-19T21:07:08.197268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.6","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.229308+0000","last_change":"2024-02-19T21:07:04.229308+0000","last_active":"2024-02-19T21:07:04.229308+0000","last_peered":"2024-02-19T21:07:04.229308+0000","last_clean":"2024-02-19T21:07:04.229308+0000","last_became_active":"2024-02-19T21:07:04.229011+0000","last_became_peered":"2024-02-19T21:07:04.229011+0000","last_unstale":"2024-02-19T21:07:04.229308+0000","last_undegraded":"2024-02-19T21:07:04.229308+0000","last_fullsized":"2024-02-19T21:07:04.229308+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"167.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:07:08.197268+0000","last_change":"2024-02-19T21:07:08.197268+0000","last_active":"2024-02-19T21:07:08.197268+0000","last_peered":"2024-02-19T21:07:08.197268+0000","last_clean":"2024-02-19T21:07:08.197268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:07:08.197268+0000","last_undegraded":"2024-02-19T21:07:08.197268+0000","last_fullsized":"2024-02-19T21:07:08.197268+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_clean_scrub_stamp":"2024-02-19T21:07:08.197268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.5","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.215754+0000","last_change":"2024-02-19T21:07:04.215754+0000","last_active":"2024-02-19T21:07:04.215754+0000","last_peered":"2024-02-19T21:07:04.215754+0000","last_clean":"2024-02-19T21:07:04.215754+0000","last_became_active":"2024-02-19T21:07:04.215446+0000","last_became_peered":"2024-02-19T21:07:04.215446+0000","last_unstale":"2024-02-19T21:07:04.215754+0000","last_undegraded":"2024-02-19T21:07:04.215754+0000","last_fullsized":"2024-02-19T21:07:04.215754+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"167.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:07:08.197268+0000","last_change":"2024-02-19T21:07:08.197268+0000","last_active":"2024-02-19T21:07:08.197268+0000","last_peered":"2024-02-19T21:07:08.197268+0000","last_clean":"2024-02-19T21:07:08.197268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:07:08.197268+0000","last_undegraded":"2024-02-19T21:07:08.197268+0000","last_fullsized":"2024-02-19T21:07:08.197268+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_clean_scrub_stamp":"2024-02-19T21:07:08.197268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.4","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.216692+0000","last_change":"2024-02-19T21:07:04.216692+0000","last_active":"2024-02-19T21:07:04.216692+0000","last_peered":"2024-02-19T21:07:04.216692+0000","last_clean":"2024-02-19T21:07:04.216692+0000","last_became_active":"2024-02-19T21:07:04.216133+0000","last_became_peered":"2024-02-19T21:07:04.216133+0000","last_unstale":"2024-02-19T21:07:04.216692+0000","last_undegraded":"2024-02-19T21:07:04.216692+0000","last_fullsized":"2024-02-19T21:07:04.216692+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"167.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:07:08.197268+0000","last_change":"2024-02-19T21:07:08.197268+0000","last_active":"2024-02-19T21:07:08.197268+0000","last_peered":"2024-02-19T21:07:08.197268+0000","last_clean":"2024-02-19T21:07:08.197268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:07:08.197268+0000","last_undegraded":"2024-02-19T21:07:08.197268+0000","last_fullsized":"2024-02-19T21:07:08.197268+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_clean_scrub_stamp":"2024-02-19T21:07:08.197268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.3","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.219519+0000","last_change":"2024-02-19T21:07:04.219519+0000","last_active":"2024-02-19T21:07:04.219519+0000","last_peered":"2024-02-19T21:07:04.219519+0000","last_clean":"2024-02-19T21:07:04.219519+0000","last_became_active":"2024-02-19T21:07:04.219214+0000","last_became_peered":"2024-02-19T21:07:04.219214+0000","last_unstale":"2024-02-19T21:07:04.219519+0000","last_undegraded":"2024-02-19T21:07:04.219519+0000","last_fullsized":"2024-02-19T21:07:04.219519+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"167.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:07:08.197268+0000","last_change":"2024-02-19T21:07:08.197268+0000","last_active":"2024-02-19T21:07:08.197268+0000","last_peered":"2024-02-19T21:07:08.197268+0000","last_clean":"2024-02-19T21:07:08.197268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:07:08.197268+0000","last_undegraded":"2024-02-19T21:07:08.197268+0000","last_fullsized":"2024-02-19T21:07:08.197268+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_clean_scrub_stamp":"2024-02-19T21:07:08.197268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.2","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.221242+0000","last_change":"2024-02-19T21:07:04.221242+0000","last_active":"2024-02-19T21:07:04.221242+0000","last_peered":"2024-02-19T21:07:04.221242+0000","last_clean":"2024-02-19T21:07:04.221242+0000","last_became_active":"2024-02-19T21:07:04.220527+0000","last_became_peered":"2024-02-19T21:07:04.220527+0000","last_unstale":"2024-02-19T21:07:04.221242+0000","last_undegraded":"2024-02-19T21:07:04.221242+0000","last_fullsized":"2024-02-19T21:07:04.221242+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"167.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:07:08.197268+0000","last_change":"2024-02-19T21:07:08.197268+0000","last_active":"2024-02-19T21:07:08.197268+0000","last_peered":"2024-02-19T21:07:08.197268+0000","last_clean":"2024-02-19T21:07:08.197268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:07:08.197268+0000","last_undegraded":"2024-02-19T21:07:08.197268+0000","last_fullsized":"2024-02-19T21:07:08.197268+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_clean_scrub_stamp":"2024-02-19T21:07:08.197268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.1","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.221768+0000","last_change":"2024-02-19T21:07:04.221768+0000","last_active":"2024-02-19T21:07:04.221768+0000","last_peered":"2024-02-19T21:07:04.221768+0000","last_clean":"2024-02-19T21:07:04.221768+0000","last_became_active":"2024-02-19T21:07:04.221482+0000","last_became_peered":"2024-02-19T21:07:04.221482+0000","last_unstale":"2024-02-19T21:07:04.221768+0000","last_undegraded":"2024-02-19T21:07:04.221768+0000","last_fullsized":"2024-02-19T21:07:04.221768+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"167.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:07:08.197268+0000","last_change":"2024-02-19T21:07:08.197268+0000","last_active":"2024-02-19T21:07:08.197268+0000","last_peered":"2024-02-19T21:07:08.197268+0000","last_clean":"2024-02-19T21:07:08.197268+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:07:08.197268+0000","last_undegraded":"2024-02-19T21:07:08.197268+0000","last_fullsized":"2024-02-19T21:07:08.197268+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:08.197268+0000","last_clean_scrub_stamp":"2024-02-19T21:07:08.197268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"164.0","version":"0'0","reported_seq":9,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.211737+0000","last_change":"2024-02-19T21:07:04.211737+0000","last_active":"2024-02-19T21:07:04.211737+0000","last_peered":"2024-02-19T21:07:04.211737+0000","last_clean":"2024-02-19T21:07:04.211737+0000","last_became_active":"2024-02-19T21:07:04.211454+0000","last_became_peered":"2024-02-19T21:07:04.211454+0000","last_unstale":"2024-02-19T21:07:04.211737+0000","last_undegraded":"2024-02-19T21:07:04.211737+0000","last_fullsized":"2024-02-19T21:07:04.211737+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":243,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:07:03.193804+0000","last_clean_scrub_stamp":"2024-02-19T21:07:03.193804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.2","version":"244'74","reported_seq":238,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.200548+0000","last_change":"2024-02-19T21:06:50.207416+0000","last_active":"2024-02-19T21:07:04.200548+0000","last_peered":"2024-02-19T21:07:04.200548+0000","last_clean":"2024-02-19T21:07:04.200548+0000","last_became_active":"2024-02-19T21:04:54.609091+0000","last_became_peered":"2024-02-19T21:04:54.609091+0000","last_unstale":"2024-02-19T21:07:04.200548+0000","last_undegraded":"2024-02-19T21:07:04.200548+0000","last_fullsized":"2024-02-19T21:07:04.200548+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":74,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.3","version":"230'68","reported_seq":224,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.206998+0000","last_change":"2024-02-19T21:06:56.304731+0000","last_active":"2024-02-19T21:07:03.206998+0000","last_peered":"2024-02-19T21:07:03.206998+0000","last_clean":"2024-02-19T21:07:03.206998+0000","last_became_active":"2024-02-19T21:04:54.641884+0000","last_became_peered":"2024-02-19T21:04:54.641884+0000","last_unstale":"2024-02-19T21:07:03.206998+0000","last_undegraded":"2024-02-19T21:07:03.206998+0000","last_fullsized":"2024-02-19T21:07:03.206998+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.0","version":"196'34","reported_seq":177,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.204934+0000","last_change":"2024-02-19T21:06:52.302761+0000","last_active":"2024-02-19T21:07:03.204934+0000","last_peered":"2024-02-19T21:07:03.204934+0000","last_clean":"2024-02-19T21:07:03.204934+0000","last_became_active":"2024-02-19T21:06:40.245157+0000","last_became_peered":"2024-02-19T21:06:40.245157+0000","last_unstale":"2024-02-19T21:07:03.204934+0000","last_undegraded":"2024-02-19T21:07:03.204934+0000","last_fullsized":"2024-02-19T21:07:03.204934+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.1","version":"220'68","reported_seq":234,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205176+0000","last_change":"2024-02-19T21:06:56.571077+0000","last_active":"2024-02-19T21:07:03.205176+0000","last_peered":"2024-02-19T21:07:03.205176+0000","last_clean":"2024-02-19T21:07:03.205176+0000","last_became_active":"2024-02-19T21:04:54.654642+0000","last_became_peered":"2024-02-19T21:04:54.654642+0000","last_unstale":"2024-02-19T21:07:03.205176+0000","last_undegraded":"2024-02-19T21:07:03.205176+0000","last_fullsized":"2024-02-19T21:07:03.205176+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'68","last_scrub_stamp":"2024-02-19T21:06:56.570995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:06:56.570995+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.6","version":"202'34","reported_seq":196,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.203644+0000","last_change":"2024-02-19T21:06:50.215874+0000","last_active":"2024-02-19T21:07:03.203644+0000","last_peered":"2024-02-19T21:07:03.203644+0000","last_clean":"2024-02-19T21:07:03.203644+0000","last_became_active":"2024-02-19T21:04:54.658542+0000","last_became_peered":"2024-02-19T21:04:54.658542+0000","last_unstale":"2024-02-19T21:07:03.203644+0000","last_undegraded":"2024-02-19T21:07:03.203644+0000","last_fullsized":"2024-02-19T21:07:03.203644+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.7","version":"220'18","reported_seq":188,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210738+0000","last_change":"2024-02-19T21:06:50.228504+0000","last_active":"2024-02-19T21:07:03.210738+0000","last_peered":"2024-02-19T21:07:03.210738+0000","last_clean":"2024-02-19T21:07:03.210738+0000","last_became_active":"2024-02-19T21:04:54.598061+0000","last_became_peered":"2024-02-19T21:04:54.598061+0000","last_unstale":"2024-02-19T21:07:03.210738+0000","last_undegraded":"2024-02-19T21:07:03.210738+0000","last_fullsized":"2024-02-19T21:07:03.210738+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.4","version":"206'68","reported_seq":225,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.202619+0000","last_change":"2024-02-19T21:06:50.215570+0000","last_active":"2024-02-19T21:07:03.202619+0000","last_peered":"2024-02-19T21:07:03.202619+0000","last_clean":"2024-02-19T21:07:03.202619+0000","last_became_active":"2024-02-19T21:04:54.639335+0000","last_became_peered":"2024-02-19T21:04:54.639335+0000","last_unstale":"2024-02-19T21:07:03.202619+0000","last_undegraded":"2024-02-19T21:07:03.202619+0000","last_fullsized":"2024-02-19T21:07:03.202619+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.5","version":"238'119","reported_seq":267,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207919+0000","last_change":"2024-02-19T21:07:00.311839+0000","last_active":"2024-02-19T21:07:03.207919+0000","last_peered":"2024-02-19T21:07:03.207919+0000","last_clean":"2024-02-19T21:07:03.207919+0000","last_became_active":"2024-02-19T21:04:54.646842+0000","last_became_peered":"2024-02-19T21:04:54.646842+0000","last_unstale":"2024-02-19T21:07:03.207919+0000","last_undegraded":"2024-02-19T21:07:03.207919+0000","last_fullsized":"2024-02-19T21:07:03.207919+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.a","version":"236'22","reported_seq":194,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200233+0000","last_change":"2024-02-19T21:06:50.205603+0000","last_active":"2024-02-19T21:07:03.200233+0000","last_peered":"2024-02-19T21:07:03.200233+0000","last_clean":"2024-02-19T21:07:03.200233+0000","last_became_active":"2024-02-19T21:04:54.611831+0000","last_became_peered":"2024-02-19T21:04:54.611831+0000","last_unstale":"2024-02-19T21:07:03.200233+0000","last_undegraded":"2024-02-19T21:07:03.200233+0000","last_fullsized":"2024-02-19T21:07:03.200233+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.b","version":"234'85","reported_seq":237,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.206902+0000","last_change":"2024-02-19T21:06:56.304982+0000","last_active":"2024-02-19T21:07:03.206902+0000","last_peered":"2024-02-19T21:07:03.206902+0000","last_clean":"2024-02-19T21:07:03.206902+0000","last_became_active":"2024-02-19T21:04:54.640024+0000","last_became_peered":"2024-02-19T21:04:54.640024+0000","last_unstale":"2024-02-19T21:07:03.206902+0000","last_undegraded":"2024-02-19T21:07:03.206902+0000","last_fullsized":"2024-02-19T21:07:03.206902+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.8","version":"212'35","reported_seq":193,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207606+0000","last_change":"2024-02-19T21:06:50.212051+0000","last_active":"2024-02-19T21:07:03.207606+0000","last_peered":"2024-02-19T21:07:03.207606+0000","last_clean":"2024-02-19T21:07:03.207606+0000","last_became_active":"2024-02-19T21:04:54.650062+0000","last_became_peered":"2024-02-19T21:04:54.650062+0000","last_unstale":"2024-02-19T21:07:03.207606+0000","last_undegraded":"2024-02-19T21:07:03.207606+0000","last_fullsized":"2024-02-19T21:07:03.207606+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.9","version":"226'68","reported_seq":229,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205074+0000","last_change":"2024-02-19T21:06:56.304245+0000","last_active":"2024-02-19T21:07:03.205074+0000","last_peered":"2024-02-19T21:07:03.205074+0000","last_clean":"2024-02-19T21:07:03.205074+0000","last_became_active":"2024-02-19T21:04:54.671646+0000","last_became_peered":"2024-02-19T21:04:54.671646+0000","last_unstale":"2024-02-19T21:07:03.205074+0000","last_undegraded":"2024-02-19T21:07:03.205074+0000","last_fullsized":"2024-02-19T21:07:03.205074+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.e","version":"189'34","reported_seq":206,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.215252+0000","last_change":"2024-02-19T21:06:59.674561+0000","last_active":"2024-02-19T21:07:03.215252+0000","last_peered":"2024-02-19T21:07:03.215252+0000","last_clean":"2024-02-19T21:07:03.215252+0000","last_became_active":"2024-02-19T21:04:54.603276+0000","last_became_peered":"2024-02-19T21:04:54.603276+0000","last_unstale":"2024-02-19T21:07:03.215252+0000","last_undegraded":"2024-02-19T21:07:03.215252+0000","last_fullsized":"2024-02-19T21:07:03.215252+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'34","last_scrub_stamp":"2024-02-19T21:06:59.674445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:06:59.674445+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.f","version":"220'34","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.204653+0000","last_change":"2024-02-19T21:06:58.720229+0000","last_active":"2024-02-19T21:07:03.204653+0000","last_peered":"2024-02-19T21:07:03.204653+0000","last_clean":"2024-02-19T21:07:03.204653+0000","last_became_active":"2024-02-19T21:04:54.610312+0000","last_became_peered":"2024-02-19T21:04:54.610312+0000","last_unstale":"2024-02-19T21:07:03.204653+0000","last_undegraded":"2024-02-19T21:07:03.204653+0000","last_fullsized":"2024-02-19T21:07:03.204653+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'34","last_scrub_stamp":"2024-02-19T21:06:58.720103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:06:58.720103+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.c","version":"244'69","reported_seq":224,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.215600+0000","last_change":"2024-02-19T21:06:50.215781+0000","last_active":"2024-02-19T21:07:04.215600+0000","last_peered":"2024-02-19T21:07:04.215600+0000","last_clean":"2024-02-19T21:07:04.215600+0000","last_became_active":"2024-02-19T21:04:54.634803+0000","last_became_peered":"2024-02-19T21:04:54.634803+0000","last_unstale":"2024-02-19T21:07:04.215600+0000","last_undegraded":"2024-02-19T21:07:04.215600+0000","last_fullsized":"2024-02-19T21:07:04.215600+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.d","version":"214'34","reported_seq":198,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.209422+0000","last_change":"2024-02-19T21:06:50.217568+0000","last_active":"2024-02-19T21:07:03.209422+0000","last_peered":"2024-02-19T21:07:03.209422+0000","last_clean":"2024-02-19T21:07:03.209422+0000","last_became_active":"2024-02-19T21:04:54.644749+0000","last_became_peered":"2024-02-19T21:04:54.644749+0000","last_unstale":"2024-02-19T21:07:03.209422+0000","last_undegraded":"2024-02-19T21:07:03.209422+0000","last_fullsized":"2024-02-19T21:07:03.209422+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.12","version":"230'44","reported_seq":208,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.202618+0000","last_change":"2024-02-19T21:06:52.228085+0000","last_active":"2024-02-19T21:07:03.202618+0000","last_peered":"2024-02-19T21:07:03.202618+0000","last_clean":"2024-02-19T21:07:03.202618+0000","last_became_active":"2024-02-19T21:04:54.670998+0000","last_became_peered":"2024-02-19T21:04:54.670998+0000","last_unstale":"2024-02-19T21:07:03.202618+0000","last_undegraded":"2024-02-19T21:07:03.202618+0000","last_fullsized":"2024-02-19T21:07:03.202618+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":14,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.13","version":"216'68","reported_seq":227,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208582+0000","last_change":"2024-02-19T21:06:50.213218+0000","last_active":"2024-02-19T21:07:03.208582+0000","last_peered":"2024-02-19T21:07:03.208582+0000","last_clean":"2024-02-19T21:07:03.208582+0000","last_became_active":"2024-02-19T21:04:54.608802+0000","last_became_peered":"2024-02-19T21:04:54.608802+0000","last_unstale":"2024-02-19T21:07:03.208582+0000","last_undegraded":"2024-02-19T21:07:03.208582+0000","last_fullsized":"2024-02-19T21:07:03.208582+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.10","version":"230'45","reported_seq":211,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205659+0000","last_change":"2024-02-19T21:06:52.229556+0000","last_active":"2024-02-19T21:07:03.205659+0000","last_peered":"2024-02-19T21:07:03.205659+0000","last_clean":"2024-02-19T21:07:03.205659+0000","last_became_active":"2024-02-19T21:04:54.626393+0000","last_became_peered":"2024-02-19T21:04:54.626393+0000","last_unstale":"2024-02-19T21:07:03.205659+0000","last_undegraded":"2024-02-19T21:07:03.205659+0000","last_fullsized":"2024-02-19T21:07:03.205659+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.11","version":"199'34","reported_seq":201,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.203704+0000","last_change":"2024-02-19T21:06:50.213281+0000","last_active":"2024-02-19T21:07:03.203704+0000","last_peered":"2024-02-19T21:07:03.203704+0000","last_clean":"2024-02-19T21:07:03.203704+0000","last_became_active":"2024-02-19T21:04:54.659096+0000","last_became_peered":"2024-02-19T21:04:54.659096+0000","last_unstale":"2024-02-19T21:07:03.203704+0000","last_undegraded":"2024-02-19T21:07:03.203704+0000","last_fullsized":"2024-02-19T21:07:03.203704+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.29","version":"0'0","reported_seq":116,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.213289+0000","last_change":"2024-02-19T21:05:44.986979+0000","last_active":"2024-02-19T21:07:03.213289+0000","last_peered":"2024-02-19T21:07:03.213289+0000","last_clean":"2024-02-19T21:07:03.213289+0000","last_became_active":"2024-02-19T21:05:22.208034+0000","last_became_peered":"2024-02-19T21:05:22.208034+0000","last_unstale":"2024-02-19T21:07:03.213289+0000","last_undegraded":"2024-02-19T21:07:03.213289+0000","last_fullsized":"2024-02-19T21:07:03.213289+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.986874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.986874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.16","version":"237'24","reported_seq":197,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201143+0000","last_change":"2024-02-19T21:06:57.889936+0000","last_active":"2024-02-19T21:07:03.201143+0000","last_peered":"2024-02-19T21:07:03.201143+0000","last_clean":"2024-02-19T21:07:03.201143+0000","last_became_active":"2024-02-19T21:04:54.608059+0000","last_became_peered":"2024-02-19T21:04:54.608059+0000","last_unstale":"2024-02-19T21:07:03.201143+0000","last_undegraded":"2024-02-19T21:07:03.201143+0000","last_fullsized":"2024-02-19T21:07:03.201143+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-02-19T21:06:57.889852+0000","last_deep_scrub":"237'24","last_deep_scrub_stamp":"2024-02-19T21:06:57.889852+0000","last_clean_scrub_stamp":"2024-02-19T21:06:57.889852+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.28","version":"0'0","reported_seq":117,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.198463+0000","last_change":"2024-02-19T21:06:29.755093+0000","last_active":"2024-02-19T21:07:03.198463+0000","last_peered":"2024-02-19T21:07:03.198463+0000","last_clean":"2024-02-19T21:07:03.198463+0000","last_became_active":"2024-02-19T21:05:22.220291+0000","last_became_peered":"2024-02-19T21:05:22.220291+0000","last_unstale":"2024-02-19T21:07:03.198463+0000","last_undegraded":"2024-02-19T21:07:03.198463+0000","last_fullsized":"2024-02-19T21:07:03.198463+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:29.755031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:29.755031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.17","version":"220'19","reported_seq":196,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.203455+0000","last_change":"2024-02-19T21:06:50.216652+0000","last_active":"2024-02-19T21:07:03.203455+0000","last_peered":"2024-02-19T21:07:03.203455+0000","last_clean":"2024-02-19T21:07:03.203455+0000","last_became_active":"2024-02-19T21:06:40.228088+0000","last_became_peered":"2024-02-19T21:06:40.228088+0000","last_unstale":"2024-02-19T21:07:03.203455+0000","last_undegraded":"2024-02-19T21:07:03.203455+0000","last_fullsized":"2024-02-19T21:07:03.203455+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.14","version":"200'51","reported_seq":195,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.202518+0000","last_change":"2024-02-19T21:06:50.215135+0000","last_active":"2024-02-19T21:07:03.202518+0000","last_peered":"2024-02-19T21:07:03.202518+0000","last_clean":"2024-02-19T21:07:03.202518+0000","last_became_active":"2024-02-19T21:06:40.227840+0000","last_became_peered":"2024-02-19T21:06:40.227840+0000","last_unstale":"2024-02-19T21:07:03.202518+0000","last_undegraded":"2024-02-19T21:07:03.202518+0000","last_fullsized":"2024-02-19T21:07:03.202518+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"61.15","version":"218'51","reported_seq":213,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205841+0000","last_change":"2024-02-19T21:06:50.216161+0000","last_active":"2024-02-19T21:07:03.205841+0000","last_peered":"2024-02-19T21:07:03.205841+0000","last_clean":"2024-02-19T21:07:03.205841+0000","last_became_active":"2024-02-19T21:04:54.626719+0000","last_became_peered":"2024-02-19T21:04:54.626719+0000","last_unstale":"2024-02-19T21:07:03.205841+0000","last_undegraded":"2024-02-19T21:07:03.205841+0000","last_fullsized":"2024-02-19T21:07:03.205841+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.25","version":"0'0","reported_seq":220,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200790+0000","last_change":"2024-02-19T21:06:11.639779+0000","last_active":"2024-02-19T21:07:03.200790+0000","last_peered":"2024-02-19T21:07:03.200790+0000","last_clean":"2024-02-19T21:07:03.200790+0000","last_became_active":"2024-02-19T21:05:22.209781+0000","last_became_peered":"2024-02-19T21:05:22.209781+0000","last_unstale":"2024-02-19T21:07:03.200790+0000","last_undegraded":"2024-02-19T21:07:03.200790+0000","last_fullsized":"2024-02-19T21:07:03.200790+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:11.639684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:11.639684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"61.1a","version":"244'86","reported_seq":246,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.200391+0000","last_change":"2024-02-19T21:06:50.214607+0000","last_active":"2024-02-19T21:07:04.200391+0000","last_peered":"2024-02-19T21:07:04.200391+0000","last_clean":"2024-02-19T21:07:04.200391+0000","last_became_active":"2024-02-19T21:04:54.634861+0000","last_became_peered":"2024-02-19T21:04:54.634861+0000","last_unstale":"2024-02-19T21:07:04.200391+0000","last_undegraded":"2024-02-19T21:07:04.200391+0000","last_fullsized":"2024-02-19T21:07:04.200391+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.24","version":"0'0","reported_seq":116,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207682+0000","last_change":"2024-02-19T21:06:04.866459+0000","last_active":"2024-02-19T21:07:03.207682+0000","last_peered":"2024-02-19T21:07:03.207682+0000","last_clean":"2024-02-19T21:07:03.207682+0000","last_became_active":"2024-02-19T21:05:22.208647+0000","last_became_peered":"2024-02-19T21:05:22.208647+0000","last_unstale":"2024-02-19T21:07:03.207682+0000","last_undegraded":"2024-02-19T21:07:03.207682+0000","last_fullsized":"2024-02-19T21:07:03.207682+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:04.866394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:06:04.866394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1b","version":"208'51","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200677+0000","last_change":"2024-02-19T21:06:50.199215+0000","last_active":"2024-02-19T21:07:03.200677+0000","last_peered":"2024-02-19T21:07:03.200677+0000","last_clean":"2024-02-19T21:07:03.200677+0000","last_became_active":"2024-02-19T21:04:54.658733+0000","last_became_peered":"2024-02-19T21:04:54.658733+0000","last_unstale":"2024-02-19T21:07:03.200677+0000","last_undegraded":"2024-02-19T21:07:03.200677+0000","last_fullsized":"2024-02-19T21:07:03.200677+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":51,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.27","version":"0'0","reported_seq":219,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.204454+0000","last_change":"2024-02-19T21:06:25.516601+0000","last_active":"2024-02-19T21:07:03.204454+0000","last_peered":"2024-02-19T21:07:03.204454+0000","last_clean":"2024-02-19T21:07:03.204454+0000","last_became_active":"2024-02-19T21:05:22.217505+0000","last_became_peered":"2024-02-19T21:05:22.217505+0000","last_unstale":"2024-02-19T21:07:03.204454+0000","last_undegraded":"2024-02-19T21:07:03.204454+0000","last_fullsized":"2024-02-19T21:07:03.204454+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:25.516527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:25.516527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.18","version":"214'102","reported_seq":253,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.198635+0000","last_change":"2024-02-19T21:06:50.207774+0000","last_active":"2024-02-19T21:07:03.198635+0000","last_peered":"2024-02-19T21:07:03.198635+0000","last_clean":"2024-02-19T21:07:03.198635+0000","last_became_active":"2024-02-19T21:04:54.610608+0000","last_became_peered":"2024-02-19T21:04:54.610608+0000","last_unstale":"2024-02-19T21:07:03.198635+0000","last_undegraded":"2024-02-19T21:07:03.198635+0000","last_fullsized":"2024-02-19T21:07:03.198635+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.26","version":"0'0","reported_seq":117,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.209710+0000","last_change":"2024-02-19T21:06:17.475027+0000","last_active":"2024-02-19T21:07:03.209710+0000","last_peered":"2024-02-19T21:07:03.209710+0000","last_clean":"2024-02-19T21:07:03.209710+0000","last_became_active":"2024-02-19T21:05:22.216623+0000","last_became_peered":"2024-02-19T21:05:22.216623+0000","last_unstale":"2024-02-19T21:07:03.209710+0000","last_undegraded":"2024-02-19T21:07:03.209710+0000","last_fullsized":"2024-02-19T21:07:03.209710+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:17.474951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:17.474951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.19","version":"218'68","reported_seq":230,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207370+0000","last_change":"2024-02-19T21:06:50.210268+0000","last_active":"2024-02-19T21:07:03.207370+0000","last_peered":"2024-02-19T21:07:03.207370+0000","last_clean":"2024-02-19T21:07:03.207370+0000","last_became_active":"2024-02-19T21:04:54.608702+0000","last_became_peered":"2024-02-19T21:04:54.608702+0000","last_unstale":"2024-02-19T21:07:03.207370+0000","last_undegraded":"2024-02-19T21:07:03.207370+0000","last_fullsized":"2024-02-19T21:07:03.207370+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.21","version":"0'0","reported_seq":232,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.203609+0000","last_change":"2024-02-19T21:05:46.049986+0000","last_active":"2024-02-19T21:07:03.203609+0000","last_peered":"2024-02-19T21:07:03.203609+0000","last_clean":"2024-02-19T21:07:03.203609+0000","last_became_active":"2024-02-19T21:05:22.216738+0000","last_became_peered":"2024-02-19T21:05:22.216738+0000","last_unstale":"2024-02-19T21:07:03.203609+0000","last_undegraded":"2024-02-19T21:07:03.203609+0000","last_fullsized":"2024-02-19T21:07:03.203609+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:46.049878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:46.049878+0000","last_clean_scrub_stamp":"2024-02-19T21:05:46.049878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.1e","version":"241'49","reported_seq":223,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.209893+0000","last_change":"2024-02-19T21:07:01.787605+0000","last_active":"2024-02-19T21:07:03.209893+0000","last_peered":"2024-02-19T21:07:03.209893+0000","last_clean":"2024-02-19T21:07:03.209893+0000","last_became_active":"2024-02-19T21:04:54.629433+0000","last_became_peered":"2024-02-19T21:04:54.629433+0000","last_unstale":"2024-02-19T21:07:03.209893+0000","last_undegraded":"2024-02-19T21:07:03.209893+0000","last_fullsized":"2024-02-19T21:07:03.209893+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'49","last_scrub_stamp":"2024-02-19T21:07:01.787521+0000","last_deep_scrub":"241'49","last_deep_scrub_stamp":"2024-02-19T21:07:01.787521+0000","last_clean_scrub_stamp":"2024-02-19T21:07:01.787521+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.20","version":"58'1","reported_seq":239,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205524+0000","last_change":"2024-02-19T21:06:18.465146+0000","last_active":"2024-02-19T21:07:03.205524+0000","last_peered":"2024-02-19T21:07:03.205524+0000","last_clean":"2024-02-19T21:07:03.205524+0000","last_became_active":"2024-02-19T21:05:22.223725+0000","last_became_peered":"2024-02-19T21:05:22.223725+0000","last_unstale":"2024-02-19T21:07:03.205524+0000","last_undegraded":"2024-02-19T21:07:03.205524+0000","last_fullsized":"2024-02-19T21:07:03.205524+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:06:18.465061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.465061+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1f","version":"244'40","reported_seq":208,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.227972+0000","last_change":"2024-02-19T21:06:52.224079+0000","last_active":"2024-02-19T21:07:04.227972+0000","last_peered":"2024-02-19T21:07:04.227972+0000","last_clean":"2024-02-19T21:07:04.227972+0000","last_became_active":"2024-02-19T21:04:54.671969+0000","last_became_peered":"2024-02-19T21:04:54.671969+0000","last_unstale":"2024-02-19T21:07:04.227972+0000","last_undegraded":"2024-02-19T21:07:04.227972+0000","last_fullsized":"2024-02-19T21:07:04.227972+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.23","version":"0'0","reported_seq":119,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200600+0000","last_change":"2024-02-19T21:06:25.686571+0000","last_active":"2024-02-19T21:07:03.200600+0000","last_peered":"2024-02-19T21:07:03.200600+0000","last_clean":"2024-02-19T21:07:03.200600+0000","last_became_active":"2024-02-19T21:05:22.218880+0000","last_became_peered":"2024-02-19T21:05:22.218880+0000","last_unstale":"2024-02-19T21:07:03.200600+0000","last_undegraded":"2024-02-19T21:07:03.200600+0000","last_fullsized":"2024-02-19T21:07:03.200600+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:25.686510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:25.686510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.1c","version":"228'136","reported_seq":286,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208935+0000","last_change":"2024-02-19T21:06:50.271910+0000","last_active":"2024-02-19T21:07:03.208935+0000","last_peered":"2024-02-19T21:07:03.208935+0000","last_clean":"2024-02-19T21:07:03.208935+0000","last_became_active":"2024-02-19T21:04:54.605898+0000","last_became_peered":"2024-02-19T21:04:54.605898+0000","last_unstale":"2024-02-19T21:07:03.208935+0000","last_undegraded":"2024-02-19T21:07:03.208935+0000","last_fullsized":"2024-02-19T21:07:03.208935+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":136,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":88,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":40,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"2.22","version":"0'0","reported_seq":218,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.213254+0000","last_change":"2024-02-19T21:05:43.958644+0000","last_active":"2024-02-19T21:07:03.213254+0000","last_peered":"2024-02-19T21:07:03.213254+0000","last_clean":"2024-02-19T21:07:03.213254+0000","last_became_active":"2024-02-19T21:05:22.196948+0000","last_became_peered":"2024-02-19T21:05:22.196948+0000","last_unstale":"2024-02-19T21:07:03.213254+0000","last_undegraded":"2024-02-19T21:07:03.213254+0000","last_fullsized":"2024-02-19T21:07:03.213254+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:43.958548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:43.958548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.1d","version":"222'34","reported_seq":200,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207817+0000","last_change":"2024-02-19T21:06:56.303912+0000","last_active":"2024-02-19T21:07:03.207817+0000","last_peered":"2024-02-19T21:07:03.207817+0000","last_clean":"2024-02-19T21:07:03.207817+0000","last_became_active":"2024-02-19T21:04:54.655665+0000","last_became_peered":"2024-02-19T21:04:54.655665+0000","last_unstale":"2024-02-19T21:07:03.207817+0000","last_undegraded":"2024-02-19T21:07:03.207817+0000","last_fullsized":"2024-02-19T21:07:03.207817+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:04:53.511966+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"c","length":"1"},{"start":"e","length":"1"}]},{"pgid":"8.4","version":"100'2","reported_seq":190,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210217+0000","last_change":"2024-02-19T21:07:00.207944+0000","last_active":"2024-02-19T21:07:03.210217+0000","last_peered":"2024-02-19T21:07:03.210217+0000","last_clean":"2024-02-19T21:07:03.210217+0000","last_became_active":"2024-02-19T21:07:00.207626+0000","last_became_peered":"2024-02-19T21:07:00.207626+0000","last_unstale":"2024-02-19T21:07:03.210217+0000","last_undegraded":"2024-02-19T21:07:03.210217+0000","last_fullsized":"2024-02-19T21:07:03.210217+0000","mapping_epoch":98,"log_start":"100'2","ondisk_log_start":"100'2","created":98,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"97'9","reported_seq":284,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.201016+0000","last_change":"2024-02-19T21:06:52.658179+0000","last_active":"2024-02-19T21:07:04.201016+0000","last_peered":"2024-02-19T21:07:04.201016+0000","last_clean":"2024-02-19T21:07:04.201016+0000","last_became_active":"2024-02-19T21:01:20.288720+0000","last_became_peered":"2024-02-19T21:01:20.288720+0000","last_unstale":"2024-02-19T21:07:04.201016+0000","last_undegraded":"2024-02-19T21:07:04.201016+0000","last_fullsized":"2024-02-19T21:07:04.201016+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-02-19T21:06:52.658104+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:06:52.658104+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":231,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207711+0000","last_change":"2024-02-19T21:04:56.123170+0000","last_active":"2024-02-19T21:07:03.207711+0000","last_peered":"2024-02-19T21:07:03.207711+0000","last_clean":"2024-02-19T21:07:03.207711+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:07:03.207711+0000","last_undegraded":"2024-02-19T21:07:03.207711+0000","last_fullsized":"2024-02-19T21:07:03.207711+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:04:56.123096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.123096+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":210,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.204157+0000","last_change":"2024-02-19T21:06:16.105542+0000","last_active":"2024-02-19T21:07:03.204157+0000","last_peered":"2024-02-19T21:07:03.204157+0000","last_clean":"2024-02-19T21:07:03.204157+0000","last_became_active":"2024-02-19T21:01:16.271911+0000","last_became_peered":"2024-02-19T21:01:16.271911+0000","last_unstale":"2024-02-19T21:07:03.204157+0000","last_undegraded":"2024-02-19T21:07:03.204157+0000","last_fullsized":"2024-02-19T21:07:03.204157+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.105470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":199,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208359+0000","last_change":"2024-02-19T21:05:43.947943+0000","last_active":"2024-02-19T21:07:03.208359+0000","last_peered":"2024-02-19T21:07:03.208359+0000","last_clean":"2024-02-19T21:07:03.208359+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:07:03.208359+0000","last_undegraded":"2024-02-19T21:07:03.208359+0000","last_fullsized":"2024-02-19T21:07:03.208359+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:43.947829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:43.947829+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":158,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.204046+0000","last_change":"2024-02-19T21:06:46.312127+0000","last_active":"2024-02-19T21:07:03.204046+0000","last_peered":"2024-02-19T21:07:03.204046+0000","last_clean":"2024-02-19T21:07:03.204046+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:07:03.204046+0000","last_undegraded":"2024-02-19T21:07:03.204046+0000","last_fullsized":"2024-02-19T21:07:03.204046+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:46.312025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.312025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"168'7","reported_seq":195,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.206177+0000","last_change":"2024-02-19T21:06:48.203814+0000","last_active":"2024-02-19T21:07:03.206177+0000","last_peered":"2024-02-19T21:07:03.206177+0000","last_clean":"2024-02-19T21:07:03.206177+0000","last_became_active":"2024-02-19T21:06:48.203366+0000","last_became_peered":"2024-02-19T21:06:48.203366+0000","last_unstale":"2024-02-19T21:07:03.206177+0000","last_undegraded":"2024-02-19T21:07:03.206177+0000","last_fullsized":"2024-02-19T21:07:03.206177+0000","mapping_epoch":98,"log_start":"168'7","ondisk_log_start":"168'7","created":98,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'7","last_scrub_stamp":"2024-02-19T21:06:17.703187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:17.703187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":64,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207281+0000","last_change":"2024-02-19T21:06:40.240963+0000","last_active":"2024-02-19T21:07:03.207281+0000","last_peered":"2024-02-19T21:07:03.207281+0000","last_clean":"2024-02-19T21:07:03.207281+0000","last_became_active":"2024-02-19T21:06:40.240602+0000","last_became_peered":"2024-02-19T21:06:40.240602+0000","last_unstale":"2024-02-19T21:07:03.207281+0000","last_undegraded":"2024-02-19T21:07:03.207281+0000","last_fullsized":"2024-02-19T21:07:03.207281+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:04:56.262927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.262927+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":219,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208484+0000","last_change":"2024-02-19T21:05:10.764055+0000","last_active":"2024-02-19T21:07:03.208484+0000","last_peered":"2024-02-19T21:07:03.208484+0000","last_clean":"2024-02-19T21:07:03.208484+0000","last_became_active":"2024-02-19T21:04:12.926645+0000","last_became_peered":"2024-02-19T21:04:12.926645+0000","last_unstale":"2024-02-19T21:07:03.208484+0000","last_undegraded":"2024-02-19T21:07:03.208484+0000","last_fullsized":"2024-02-19T21:07:03.208484+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:10.763889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:10.763889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":206,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.202802+0000","last_change":"2024-02-19T21:06:27.303370+0000","last_active":"2024-02-19T21:07:03.202802+0000","last_peered":"2024-02-19T21:07:03.202802+0000","last_clean":"2024-02-19T21:07:03.202802+0000","last_became_active":"2024-02-19T21:01:20.290566+0000","last_became_peered":"2024-02-19T21:01:20.290566+0000","last_unstale":"2024-02-19T21:07:03.202802+0000","last_undegraded":"2024-02-19T21:07:03.202802+0000","last_fullsized":"2024-02-19T21:07:03.202802+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:27.303307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:27.303307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":36,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205208+0000","last_change":"2024-02-19T21:06:40.228164+0000","last_active":"2024-02-19T21:07:03.205208+0000","last_peered":"2024-02-19T21:07:03.205208+0000","last_clean":"2024-02-19T21:07:03.205208+0000","last_became_active":"2024-02-19T21:06:40.227822+0000","last_became_peered":"2024-02-19T21:06:40.227822+0000","last_unstale":"2024-02-19T21:07:03.205208+0000","last_undegraded":"2024-02-19T21:07:03.205208+0000","last_fullsized":"2024-02-19T21:07:03.205208+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:05.822641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:05.822641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":158,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210906+0000","last_change":"2024-02-19T21:07:00.449622+0000","last_active":"2024-02-19T21:07:03.210906+0000","last_peered":"2024-02-19T21:07:03.210906+0000","last_clean":"2024-02-19T21:07:03.210906+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:07:03.210906+0000","last_undegraded":"2024-02-19T21:07:03.210906+0000","last_fullsized":"2024-02-19T21:07:03.210906+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:00.449534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:00.449534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"168'11","reported_seq":145,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.215330+0000","last_change":"2024-02-19T21:06:52.688430+0000","last_active":"2024-02-19T21:07:03.215330+0000","last_peered":"2024-02-19T21:07:03.215330+0000","last_clean":"2024-02-19T21:07:03.215330+0000","last_became_active":"2024-02-19T21:06:52.222861+0000","last_became_peered":"2024-02-19T21:06:52.222861+0000","last_unstale":"2024-02-19T21:07:03.215330+0000","last_undegraded":"2024-02-19T21:07:03.215330+0000","last_fullsized":"2024-02-19T21:07:03.215330+0000","mapping_epoch":219,"log_start":"168'11","ondisk_log_start":"168'11","created":98,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'11","last_scrub_stamp":"2024-02-19T21:06:52.688358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:52.688358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":238,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.213630+0000","last_change":"2024-02-19T21:06:06.909964+0000","last_active":"2024-02-19T21:07:03.213630+0000","last_peered":"2024-02-19T21:07:03.213630+0000","last_clean":"2024-02-19T21:07:03.213630+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:07:03.213630+0000","last_undegraded":"2024-02-19T21:07:03.213630+0000","last_fullsized":"2024-02-19T21:07:03.213630+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:06:06.909884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:06.909884+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":210,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.203043+0000","last_change":"2024-02-19T21:05:49.051796+0000","last_active":"2024-02-19T21:07:03.203043+0000","last_peered":"2024-02-19T21:07:03.203043+0000","last_clean":"2024-02-19T21:07:03.203043+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:07:03.203043+0000","last_undegraded":"2024-02-19T21:07:03.203043+0000","last_fullsized":"2024-02-19T21:07:03.203043+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:05:49.051732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":201,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.202703+0000","last_change":"2024-02-19T21:04:51.822914+0000","last_active":"2024-02-19T21:07:03.202703+0000","last_peered":"2024-02-19T21:07:03.202703+0000","last_clean":"2024-02-19T21:07:03.202703+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:07:03.202703+0000","last_undegraded":"2024-02-19T21:07:03.202703+0000","last_fullsized":"2024-02-19T21:07:03.202703+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:51.822845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:51.822845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":198,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205868+0000","last_change":"2024-02-19T21:05:05.187540+0000","last_active":"2024-02-19T21:07:03.205868+0000","last_peered":"2024-02-19T21:07:03.205868+0000","last_clean":"2024-02-19T21:07:03.205868+0000","last_became_active":"2024-02-19T21:01:22.291759+0000","last_became_peered":"2024-02-19T21:01:22.291759+0000","last_unstale":"2024-02-19T21:07:03.205868+0000","last_undegraded":"2024-02-19T21:07:03.205868+0000","last_fullsized":"2024-02-19T21:07:03.205868+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:05.187474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:05.187474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":158,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210081+0000","last_change":"2024-02-19T21:06:56.633700+0000","last_active":"2024-02-19T21:07:03.210081+0000","last_peered":"2024-02-19T21:07:03.210081+0000","last_clean":"2024-02-19T21:07:03.210081+0000","last_became_active":"2024-02-19T21:04:39.618614+0000","last_became_peered":"2024-02-19T21:04:39.618614+0000","last_unstale":"2024-02-19T21:07:03.210081+0000","last_undegraded":"2024-02-19T21:07:03.210081+0000","last_fullsized":"2024-02-19T21:07:03.210081+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:56.633622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:56.633622+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":263,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.202276+0000","last_change":"2024-02-19T21:06:06.630925+0000","last_active":"2024-02-19T21:07:03.202276+0000","last_peered":"2024-02-19T21:07:03.202276+0000","last_clean":"2024-02-19T21:07:03.202276+0000","last_became_active":"2024-02-19T21:01:30.923214+0000","last_became_peered":"2024-02-19T21:01:30.923214+0000","last_unstale":"2024-02-19T21:07:03.202276+0000","last_undegraded":"2024-02-19T21:07:03.202276+0000","last_fullsized":"2024-02-19T21:07:03.202276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:06:06.630828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:06.630828+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":215,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208833+0000","last_change":"2024-02-19T21:06:50.694322+0000","last_active":"2024-02-19T21:07:03.208833+0000","last_peered":"2024-02-19T21:07:03.208833+0000","last_clean":"2024-02-19T21:07:03.208833+0000","last_became_active":"2024-02-19T21:01:16.274937+0000","last_became_peered":"2024-02-19T21:01:16.274937+0000","last_unstale":"2024-02-19T21:07:03.208833+0000","last_undegraded":"2024-02-19T21:07:03.208833+0000","last_fullsized":"2024-02-19T21:07:03.208833+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:06:50.694248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:50.694248+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":206,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205879+0000","last_change":"2024-02-19T21:05:39.947596+0000","last_active":"2024-02-19T21:07:03.205879+0000","last_peered":"2024-02-19T21:07:03.205879+0000","last_clean":"2024-02-19T21:07:03.205879+0000","last_became_active":"2024-02-19T21:01:20.290687+0000","last_became_peered":"2024-02-19T21:01:20.290687+0000","last_unstale":"2024-02-19T21:07:03.205879+0000","last_undegraded":"2024-02-19T21:07:03.205879+0000","last_fullsized":"2024-02-19T21:07:03.205879+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:39.947521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:39.947521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.204085+0000","last_change":"2024-02-19T21:06:13.136600+0000","last_active":"2024-02-19T21:07:03.204085+0000","last_peered":"2024-02-19T21:07:03.204085+0000","last_clean":"2024-02-19T21:07:03.204085+0000","last_became_active":"2024-02-19T21:01:22.298941+0000","last_became_peered":"2024-02-19T21:01:22.298941+0000","last_unstale":"2024-02-19T21:07:03.204085+0000","last_undegraded":"2024-02-19T21:07:03.204085+0000","last_fullsized":"2024-02-19T21:07:03.204085+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:13.136532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:06:13.136532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.1","version":"168'9","reported_seq":184,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205392+0000","last_change":"2024-02-19T21:06:40.246435+0000","last_active":"2024-02-19T21:07:03.205392+0000","last_peered":"2024-02-19T21:07:03.205392+0000","last_clean":"2024-02-19T21:07:03.205392+0000","last_became_active":"2024-02-19T21:06:40.245243+0000","last_became_peered":"2024-02-19T21:06:40.245243+0000","last_unstale":"2024-02-19T21:07:03.205392+0000","last_undegraded":"2024-02-19T21:07:03.205392+0000","last_fullsized":"2024-02-19T21:07:03.205392+0000","mapping_epoch":98,"log_start":"168'9","ondisk_log_start":"168'9","created":98,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'7","last_scrub_stamp":"2024-02-19T21:06:23.726481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:23.726481+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":158,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201067+0000","last_change":"2024-02-19T21:06:51.878186+0000","last_active":"2024-02-19T21:07:03.201067+0000","last_peered":"2024-02-19T21:07:03.201067+0000","last_clean":"2024-02-19T21:07:03.201067+0000","last_became_active":"2024-02-19T21:04:39.698684+0000","last_became_peered":"2024-02-19T21:04:39.698684+0000","last_unstale":"2024-02-19T21:07:03.201067+0000","last_undegraded":"2024-02-19T21:07:03.201067+0000","last_fullsized":"2024-02-19T21:07:03.201067+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.878060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.878060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"82'15","reported_seq":228,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200688+0000","last_change":"2024-02-19T21:06:28.747060+0000","last_active":"2024-02-19T21:07:03.200688+0000","last_peered":"2024-02-19T21:07:03.200688+0000","last_clean":"2024-02-19T21:07:03.200688+0000","last_became_active":"2024-02-19T21:01:18.282805+0000","last_became_peered":"2024-02-19T21:01:18.282805+0000","last_unstale":"2024-02-19T21:07:03.200688+0000","last_undegraded":"2024-02-19T21:07:03.200688+0000","last_fullsized":"2024-02-19T21:07:03.200688+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:06:28.746972+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:06:28.746972+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":210,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.206533+0000","last_change":"2024-02-19T21:06:14.688019+0000","last_active":"2024-02-19T21:07:03.206533+0000","last_peered":"2024-02-19T21:07:03.206533+0000","last_clean":"2024-02-19T21:07:03.206533+0000","last_became_active":"2024-02-19T21:01:16.274086+0000","last_became_peered":"2024-02-19T21:01:16.274086+0000","last_unstale":"2024-02-19T21:07:03.206533+0000","last_undegraded":"2024-02-19T21:07:03.206533+0000","last_fullsized":"2024-02-19T21:07:03.206533+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:14.687938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.687938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":284,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.217589+0000","last_change":"2024-02-19T21:06:12.177597+0000","last_active":"2024-02-19T21:07:04.217589+0000","last_peered":"2024-02-19T21:07:04.217589+0000","last_clean":"2024-02-19T21:07:04.217589+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:07:04.217589+0000","last_undegraded":"2024-02-19T21:07:04.217589+0000","last_fullsized":"2024-02-19T21:07:04.217589+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:12.177524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:12.177524+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201893+0000","last_change":"2024-02-19T21:06:18.704518+0000","last_active":"2024-02-19T21:07:03.201893+0000","last_peered":"2024-02-19T21:07:03.201893+0000","last_clean":"2024-02-19T21:07:03.201893+0000","last_became_active":"2024-02-19T21:01:22.299993+0000","last_became_peered":"2024-02-19T21:01:22.299993+0000","last_unstale":"2024-02-19T21:07:03.201893+0000","last_undegraded":"2024-02-19T21:07:03.201893+0000","last_fullsized":"2024-02-19T21:07:03.201893+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.704447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.704447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"168'7","reported_seq":184,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.198895+0000","last_change":"2024-02-19T21:06:57.682841+0000","last_active":"2024-02-19T21:07:03.198895+0000","last_peered":"2024-02-19T21:07:03.198895+0000","last_clean":"2024-02-19T21:07:03.198895+0000","last_became_active":"2024-02-19T21:06:44.221999+0000","last_became_peered":"2024-02-19T21:06:44.221999+0000","last_unstale":"2024-02-19T21:07:03.198895+0000","last_undegraded":"2024-02-19T21:07:03.198895+0000","last_fullsized":"2024-02-19T21:07:03.198895+0000","mapping_epoch":98,"log_start":"168'7","ondisk_log_start":"168'7","created":98,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'7","last_scrub_stamp":"2024-02-19T21:06:57.682770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:57.682770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":36,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.198797+0000","last_change":"2024-02-19T21:06:40.226578+0000","last_active":"2024-02-19T21:07:03.198797+0000","last_peered":"2024-02-19T21:07:03.198797+0000","last_clean":"2024-02-19T21:07:03.198797+0000","last_became_active":"2024-02-19T21:06:40.226290+0000","last_became_peered":"2024-02-19T21:06:40.226290+0000","last_unstale":"2024-02-19T21:07:03.198797+0000","last_undegraded":"2024-02-19T21:07:03.198797+0000","last_fullsized":"2024-02-19T21:07:03.198797+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"199'16","reported_seq":202,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.218760+0000","last_change":"2024-02-19T21:07:04.218760+0000","last_active":"2024-02-19T21:07:04.218760+0000","last_peered":"2024-02-19T21:07:04.218760+0000","last_clean":"2024-02-19T21:07:04.218760+0000","last_became_active":"2024-02-19T21:07:04.218274+0000","last_became_peered":"2024-02-19T21:07:04.218274+0000","last_unstale":"2024-02-19T21:07:04.218760+0000","last_undegraded":"2024-02-19T21:07:04.218760+0000","last_fullsized":"2024-02-19T21:07:04.218760+0000","mapping_epoch":98,"log_start":"199'16","ondisk_log_start":"199'16","created":98,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'11","last_scrub_stamp":"2024-02-19T21:06:46.624682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.624682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":512,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":228,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.206236+0000","last_change":"2024-02-19T21:05:03.210019+0000","last_active":"2024-02-19T21:07:03.206236+0000","last_peered":"2024-02-19T21:07:03.206236+0000","last_clean":"2024-02-19T21:07:03.206236+0000","last_became_active":"2024-02-19T21:01:18.285534+0000","last_became_peered":"2024-02-19T21:01:18.285534+0000","last_unstale":"2024-02-19T21:07:03.206236+0000","last_undegraded":"2024-02-19T21:07:03.206236+0000","last_fullsized":"2024-02-19T21:07:03.206236+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:05:03.209876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.209876+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":219,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205572+0000","last_change":"2024-02-19T21:06:07.874422+0000","last_active":"2024-02-19T21:07:03.205572+0000","last_peered":"2024-02-19T21:07:03.205572+0000","last_clean":"2024-02-19T21:07:03.205572+0000","last_became_active":"2024-02-19T21:05:20.188821+0000","last_became_peered":"2024-02-19T21:05:20.188821+0000","last_unstale":"2024-02-19T21:07:03.205572+0000","last_undegraded":"2024-02-19T21:07:03.205572+0000","last_fullsized":"2024-02-19T21:07:03.205572+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:07.874362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:07.874362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"65'8","reported_seq":284,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.198344+0000","last_change":"2024-02-19T21:06:20.710839+0000","last_active":"2024-02-19T21:07:04.198344+0000","last_peered":"2024-02-19T21:07:04.198344+0000","last_clean":"2024-02-19T21:07:04.198344+0000","last_became_active":"2024-02-19T21:01:20.293341+0000","last_became_peered":"2024-02-19T21:01:20.293341+0000","last_unstale":"2024-02-19T21:07:04.198344+0000","last_undegraded":"2024-02-19T21:07:04.198344+0000","last_fullsized":"2024-02-19T21:07:04.198344+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:20.710731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:20.710731+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.209965+0000","last_change":"2024-02-19T21:06:20.875472+0000","last_active":"2024-02-19T21:07:03.209965+0000","last_peered":"2024-02-19T21:07:03.209965+0000","last_clean":"2024-02-19T21:07:03.209965+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:07:03.209965+0000","last_undegraded":"2024-02-19T21:07:03.209965+0000","last_fullsized":"2024-02-19T21:07:03.209965+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:20.875401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:20.875401+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":158,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208327+0000","last_change":"2024-02-19T21:06:51.423833+0000","last_active":"2024-02-19T21:07:03.208327+0000","last_peered":"2024-02-19T21:07:03.208327+0000","last_clean":"2024-02-19T21:07:03.208327+0000","last_became_active":"2024-02-19T21:04:39.657052+0000","last_became_peered":"2024-02-19T21:04:39.657052+0000","last_unstale":"2024-02-19T21:07:03.208327+0000","last_undegraded":"2024-02-19T21:07:03.208327+0000","last_fullsized":"2024-02-19T21:07:03.208327+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.423769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.423769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":177,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201509+0000","last_change":"2024-02-19T21:06:36.215015+0000","last_active":"2024-02-19T21:07:03.201509+0000","last_peered":"2024-02-19T21:07:03.201509+0000","last_clean":"2024-02-19T21:07:03.201509+0000","last_became_active":"2024-02-19T21:06:36.214661+0000","last_became_peered":"2024-02-19T21:06:36.214661+0000","last_unstale":"2024-02-19T21:07:03.201509+0000","last_undegraded":"2024-02-19T21:07:03.201509+0000","last_fullsized":"2024-02-19T21:07:03.201509+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:24.733300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:24.733300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":227,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201379+0000","last_change":"2024-02-19T21:06:02.595348+0000","last_active":"2024-02-19T21:07:03.201379+0000","last_peered":"2024-02-19T21:07:03.201379+0000","last_clean":"2024-02-19T21:07:03.201379+0000","last_became_active":"2024-02-19T21:01:18.285670+0000","last_became_peered":"2024-02-19T21:01:18.285670+0000","last_unstale":"2024-02-19T21:07:03.201379+0000","last_undegraded":"2024-02-19T21:07:03.201379+0000","last_fullsized":"2024-02-19T21:07:03.201379+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:06:02.595266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:02.595266+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":218,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.209335+0000","last_change":"2024-02-19T21:06:07.411725+0000","last_active":"2024-02-19T21:07:03.209335+0000","last_peered":"2024-02-19T21:07:03.209335+0000","last_clean":"2024-02-19T21:07:03.209335+0000","last_became_active":"2024-02-19T21:05:20.211711+0000","last_became_peered":"2024-02-19T21:05:20.211711+0000","last_unstale":"2024-02-19T21:07:03.209335+0000","last_undegraded":"2024-02-19T21:07:03.209335+0000","last_fullsized":"2024-02-19T21:07:03.209335+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:07.411662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:07.411662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":206,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.211238+0000","last_change":"2024-02-19T21:06:53.428205+0000","last_active":"2024-02-19T21:07:03.211238+0000","last_peered":"2024-02-19T21:07:03.211238+0000","last_clean":"2024-02-19T21:07:03.211238+0000","last_became_active":"2024-02-19T21:01:20.289375+0000","last_became_peered":"2024-02-19T21:01:20.289375+0000","last_unstale":"2024-02-19T21:07:03.211238+0000","last_undegraded":"2024-02-19T21:07:03.211238+0000","last_fullsized":"2024-02-19T21:07:03.211238+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:53.428093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:53.428093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":199,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201685+0000","last_change":"2024-02-19T21:06:05.721241+0000","last_active":"2024-02-19T21:07:03.201685+0000","last_peered":"2024-02-19T21:07:03.201685+0000","last_clean":"2024-02-19T21:07:03.201685+0000","last_became_active":"2024-02-19T21:01:22.607007+0000","last_became_peered":"2024-02-19T21:01:22.607007+0000","last_unstale":"2024-02-19T21:07:03.201685+0000","last_undegraded":"2024-02-19T21:07:03.201685+0000","last_fullsized":"2024-02-19T21:07:03.201685+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:05.721175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:05.721175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":153,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200447+0000","last_change":"2024-02-19T21:04:39.744505+0000","last_active":"2024-02-19T21:07:03.200447+0000","last_peered":"2024-02-19T21:07:03.200447+0000","last_clean":"2024-02-19T21:07:03.200447+0000","last_became_active":"2024-02-19T21:04:39.744231+0000","last_became_peered":"2024-02-19T21:04:39.744231+0000","last_unstale":"2024-02-19T21:07:03.200447+0000","last_undegraded":"2024-02-19T21:07:03.200447+0000","last_fullsized":"2024-02-19T21:07:03.200447+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"65'8","reported_seq":284,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.199770+0000","last_change":"2024-02-19T21:06:07.761009+0000","last_active":"2024-02-19T21:07:04.199770+0000","last_peered":"2024-02-19T21:07:04.199770+0000","last_clean":"2024-02-19T21:07:04.199770+0000","last_became_active":"2024-02-19T21:01:20.286505+0000","last_became_peered":"2024-02-19T21:01:20.286505+0000","last_unstale":"2024-02-19T21:07:04.199770+0000","last_undegraded":"2024-02-19T21:07:04.199770+0000","last_fullsized":"2024-02-19T21:07:04.199770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:07.760934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:07.760934+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":229,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201450+0000","last_change":"2024-02-19T21:06:46.813883+0000","last_active":"2024-02-19T21:07:03.201450+0000","last_peered":"2024-02-19T21:07:03.201450+0000","last_clean":"2024-02-19T21:07:03.201450+0000","last_became_active":"2024-02-19T21:01:18.282702+0000","last_became_peered":"2024-02-19T21:01:18.282702+0000","last_unstale":"2024-02-19T21:07:03.201450+0000","last_undegraded":"2024-02-19T21:07:03.201450+0000","last_fullsized":"2024-02-19T21:07:03.201450+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:06:46.813814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.813814+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":218,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.206042+0000","last_change":"2024-02-19T21:05:35.951650+0000","last_active":"2024-02-19T21:07:03.206042+0000","last_peered":"2024-02-19T21:07:03.206042+0000","last_clean":"2024-02-19T21:07:03.206042+0000","last_became_active":"2024-02-19T21:05:20.224706+0000","last_became_peered":"2024-02-19T21:05:20.224706+0000","last_unstale":"2024-02-19T21:07:03.206042+0000","last_undegraded":"2024-02-19T21:07:03.206042+0000","last_fullsized":"2024-02-19T21:07:03.206042+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:35.951592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:35.951592+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":200,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.209236+0000","last_change":"2024-02-19T21:05:03.248574+0000","last_active":"2024-02-19T21:07:03.209236+0000","last_peered":"2024-02-19T21:07:03.209236+0000","last_clean":"2024-02-19T21:07:03.209236+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:07:03.209236+0000","last_undegraded":"2024-02-19T21:07:03.209236+0000","last_fullsized":"2024-02-19T21:07:03.209236+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.248506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":36,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208695+0000","last_change":"2024-02-19T21:06:40.245744+0000","last_active":"2024-02-19T21:07:03.208695+0000","last_peered":"2024-02-19T21:07:03.208695+0000","last_clean":"2024-02-19T21:07:03.208695+0000","last_became_active":"2024-02-19T21:06:40.245385+0000","last_became_peered":"2024-02-19T21:06:40.245385+0000","last_unstale":"2024-02-19T21:07:03.208695+0000","last_undegraded":"2024-02-19T21:07:03.208695+0000","last_fullsized":"2024-02-19T21:07:03.208695+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:16.780601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.780601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":201,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.202378+0000","last_change":"2024-02-19T21:05:12.318453+0000","last_active":"2024-02-19T21:07:03.202378+0000","last_peered":"2024-02-19T21:07:03.202378+0000","last_clean":"2024-02-19T21:07:03.202378+0000","last_became_active":"2024-02-19T21:01:20.291079+0000","last_became_peered":"2024-02-19T21:01:20.291079+0000","last_unstale":"2024-02-19T21:07:03.202378+0000","last_undegraded":"2024-02-19T21:07:03.202378+0000","last_fullsized":"2024-02-19T21:07:03.202378+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:05:12.318389+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":240,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.199146+0000","last_change":"2024-02-19T21:06:46.816190+0000","last_active":"2024-02-19T21:07:03.199146+0000","last_peered":"2024-02-19T21:07:03.199146+0000","last_clean":"2024-02-19T21:07:03.199146+0000","last_became_active":"2024-02-19T21:01:18.293812+0000","last_became_peered":"2024-02-19T21:01:18.293812+0000","last_unstale":"2024-02-19T21:07:03.199146+0000","last_undegraded":"2024-02-19T21:07:03.199146+0000","last_fullsized":"2024-02-19T21:07:03.199146+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:06:46.816121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.816121+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":219,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207436+0000","last_change":"2024-02-19T21:05:44.876482+0000","last_active":"2024-02-19T21:07:03.207436+0000","last_peered":"2024-02-19T21:07:03.207436+0000","last_clean":"2024-02-19T21:07:03.207436+0000","last_became_active":"2024-02-19T21:05:20.189051+0000","last_became_peered":"2024-02-19T21:05:20.189051+0000","last_unstale":"2024-02-19T21:07:03.207436+0000","last_undegraded":"2024-02-19T21:07:03.207436+0000","last_fullsized":"2024-02-19T21:07:03.207436+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":199,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.202425+0000","last_change":"2024-02-19T21:05:47.538988+0000","last_active":"2024-02-19T21:07:03.202425+0000","last_peered":"2024-02-19T21:07:03.202425+0000","last_clean":"2024-02-19T21:07:03.202425+0000","last_became_active":"2024-02-19T21:01:22.297707+0000","last_became_peered":"2024-02-19T21:01:22.297707+0000","last_unstale":"2024-02-19T21:07:03.202425+0000","last_undegraded":"2024-02-19T21:07:03.202425+0000","last_fullsized":"2024-02-19T21:07:03.202425+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:05:47.538920+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":36,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201513+0000","last_change":"2024-02-19T21:06:40.243815+0000","last_active":"2024-02-19T21:07:03.201513+0000","last_peered":"2024-02-19T21:07:03.201513+0000","last_clean":"2024-02-19T21:07:03.201513+0000","last_became_active":"2024-02-19T21:06:40.243492+0000","last_became_peered":"2024-02-19T21:06:40.243492+0000","last_unstale":"2024-02-19T21:07:03.201513+0000","last_undegraded":"2024-02-19T21:07:03.201513+0000","last_fullsized":"2024-02-19T21:07:03.201513+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":210,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.213745+0000","last_change":"2024-02-19T21:07:00.676713+0000","last_active":"2024-02-19T21:07:03.213745+0000","last_peered":"2024-02-19T21:07:03.213745+0000","last_clean":"2024-02-19T21:07:03.213745+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:07:03.213745+0000","last_undegraded":"2024-02-19T21:07:03.213745+0000","last_fullsized":"2024-02-19T21:07:03.213745+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:00.676634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:07:00.676634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":84,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210378+0000","last_change":"2024-02-19T21:06:40.224213+0000","last_active":"2024-02-19T21:07:03.210378+0000","last_peered":"2024-02-19T21:07:03.210378+0000","last_clean":"2024-02-19T21:07:03.210378+0000","last_became_active":"2024-02-19T21:06:40.223768+0000","last_became_peered":"2024-02-19T21:06:40.223768+0000","last_unstale":"2024-02-19T21:07:03.210378+0000","last_undegraded":"2024-02-19T21:07:03.210378+0000","last_fullsized":"2024-02-19T21:07:03.210378+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:06:10.826889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:10.826889+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":220,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210901+0000","last_change":"2024-02-19T21:06:06.458624+0000","last_active":"2024-02-19T21:07:03.210901+0000","last_peered":"2024-02-19T21:07:03.210901+0000","last_clean":"2024-02-19T21:07:03.210901+0000","last_became_active":"2024-02-19T21:05:20.211241+0000","last_became_peered":"2024-02-19T21:05:20.211241+0000","last_unstale":"2024-02-19T21:07:03.210901+0000","last_undegraded":"2024-02-19T21:07:03.210901+0000","last_fullsized":"2024-02-19T21:07:03.210901+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:06.458541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:06.458541+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":199,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205704+0000","last_change":"2024-02-19T21:05:01.859040+0000","last_active":"2024-02-19T21:07:03.205704+0000","last_peered":"2024-02-19T21:07:03.205704+0000","last_clean":"2024-02-19T21:07:03.205704+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:07:03.205704+0000","last_undegraded":"2024-02-19T21:07:03.205704+0000","last_fullsized":"2024-02-19T21:07:03.205704+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:01.858924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.858924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":153,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.204059+0000","last_change":"2024-02-19T21:04:39.719113+0000","last_active":"2024-02-19T21:07:03.204059+0000","last_peered":"2024-02-19T21:07:03.204059+0000","last_clean":"2024-02-19T21:07:03.204059+0000","last_became_active":"2024-02-19T21:04:39.718926+0000","last_became_peered":"2024-02-19T21:04:39.718926+0000","last_unstale":"2024-02-19T21:07:03.204059+0000","last_undegraded":"2024-02-19T21:07:03.204059+0000","last_fullsized":"2024-02-19T21:07:03.204059+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.a","version":"168'14","reported_seq":186,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.206207+0000","last_change":"2024-02-19T21:06:04.207771+0000","last_active":"2024-02-19T21:07:03.206207+0000","last_peered":"2024-02-19T21:07:03.206207+0000","last_clean":"2024-02-19T21:07:03.206207+0000","last_became_active":"2024-02-19T21:06:04.207446+0000","last_became_peered":"2024-02-19T21:06:04.207446+0000","last_unstale":"2024-02-19T21:07:03.206207+0000","last_undegraded":"2024-02-19T21:07:03.206207+0000","last_fullsized":"2024-02-19T21:07:03.206207+0000","mapping_epoch":98,"log_start":"168'14","ondisk_log_start":"168'14","created":98,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:04:33.426263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":384,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":201,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.211445+0000","last_change":"2024-02-19T21:04:57.188742+0000","last_active":"2024-02-19T21:07:03.211445+0000","last_peered":"2024-02-19T21:07:03.211445+0000","last_clean":"2024-02-19T21:07:03.211445+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:07:03.211445+0000","last_undegraded":"2024-02-19T21:07:03.211445+0000","last_fullsized":"2024-02-19T21:07:03.211445+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:57.188664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:57.188664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":66,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207537+0000","last_change":"2024-02-19T21:06:40.240732+0000","last_active":"2024-02-19T21:07:03.207537+0000","last_peered":"2024-02-19T21:07:03.207537+0000","last_clean":"2024-02-19T21:07:03.207537+0000","last_became_active":"2024-02-19T21:06:40.240321+0000","last_became_peered":"2024-02-19T21:06:40.240321+0000","last_unstale":"2024-02-19T21:07:03.207537+0000","last_undegraded":"2024-02-19T21:07:03.207537+0000","last_fullsized":"2024-02-19T21:07:03.207537+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:05:09.646445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:09.646445+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"58'1","reported_seq":225,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205972+0000","last_change":"2024-02-19T21:06:09.422099+0000","last_active":"2024-02-19T21:07:03.205972+0000","last_peered":"2024-02-19T21:07:03.205972+0000","last_clean":"2024-02-19T21:07:03.205972+0000","last_became_active":"2024-02-19T21:05:20.211701+0000","last_became_peered":"2024-02-19T21:05:20.211701+0000","last_unstale":"2024-02-19T21:07:03.205972+0000","last_undegraded":"2024-02-19T21:07:03.205972+0000","last_fullsized":"2024-02-19T21:07:03.205972+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:06:09.422007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.422007+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":36,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201433+0000","last_change":"2024-02-19T21:06:40.221051+0000","last_active":"2024-02-19T21:07:03.201433+0000","last_peered":"2024-02-19T21:07:03.201433+0000","last_clean":"2024-02-19T21:07:03.201433+0000","last_became_active":"2024-02-19T21:06:40.220667+0000","last_became_peered":"2024-02-19T21:06:40.220667+0000","last_unstale":"2024-02-19T21:07:03.201433+0000","last_undegraded":"2024-02-19T21:07:03.201433+0000","last_fullsized":"2024-02-19T21:07:03.201433+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:07.660435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:05:07.660435+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205970+0000","last_change":"2024-02-19T21:06:51.627839+0000","last_active":"2024-02-19T21:07:03.205970+0000","last_peered":"2024-02-19T21:07:03.205970+0000","last_clean":"2024-02-19T21:07:03.205970+0000","last_became_active":"2024-02-19T21:01:22.298477+0000","last_became_peered":"2024-02-19T21:01:22.298477+0000","last_unstale":"2024-02-19T21:07:03.205970+0000","last_undegraded":"2024-02-19T21:07:03.205970+0000","last_fullsized":"2024-02-19T21:07:03.205970+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.627708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.627708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":152,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.213477+0000","last_change":"2024-02-19T21:04:39.665830+0000","last_active":"2024-02-19T21:07:03.213477+0000","last_peered":"2024-02-19T21:07:03.213477+0000","last_clean":"2024-02-19T21:07:03.213477+0000","last_became_active":"2024-02-19T21:04:39.662125+0000","last_became_peered":"2024-02-19T21:04:39.662125+0000","last_unstale":"2024-02-19T21:07:03.213477+0000","last_undegraded":"2024-02-19T21:07:03.213477+0000","last_fullsized":"2024-02-19T21:07:03.213477+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":176,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.213715+0000","last_change":"2024-02-19T21:06:56.652488+0000","last_active":"2024-02-19T21:07:03.213715+0000","last_peered":"2024-02-19T21:07:03.213715+0000","last_clean":"2024-02-19T21:07:03.213715+0000","last_became_active":"2024-02-19T21:06:08.206725+0000","last_became_peered":"2024-02-19T21:06:08.206725+0000","last_unstale":"2024-02-19T21:07:03.213715+0000","last_undegraded":"2024-02-19T21:07:03.213715+0000","last_fullsized":"2024-02-19T21:07:03.213715+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":98,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:56.652416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:56.652416+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.204664+0000","last_change":"2024-02-19T21:06:51.666781+0000","last_active":"2024-02-19T21:07:03.204664+0000","last_peered":"2024-02-19T21:07:03.204664+0000","last_clean":"2024-02-19T21:07:03.204664+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:07:03.204664+0000","last_undegraded":"2024-02-19T21:07:03.204664+0000","last_fullsized":"2024-02-19T21:07:03.204664+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.666708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.666708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":229,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.213163+0000","last_change":"2024-02-19T21:06:26.777041+0000","last_active":"2024-02-19T21:07:03.213163+0000","last_peered":"2024-02-19T21:07:03.213163+0000","last_clean":"2024-02-19T21:07:03.213163+0000","last_became_active":"2024-02-19T21:01:18.293784+0000","last_became_peered":"2024-02-19T21:01:18.293784+0000","last_unstale":"2024-02-19T21:07:03.213163+0000","last_undegraded":"2024-02-19T21:07:03.213163+0000","last_fullsized":"2024-02-19T21:07:03.213163+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:06:26.776963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:26.776963+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":213,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.206061+0000","last_change":"2024-02-19T21:05:00.284714+0000","last_active":"2024-02-19T21:07:03.206061+0000","last_peered":"2024-02-19T21:07:03.206061+0000","last_clean":"2024-02-19T21:07:03.206061+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:07:03.206061+0000","last_undegraded":"2024-02-19T21:07:03.206061+0000","last_fullsized":"2024-02-19T21:07:03.206061+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:00.284626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:05:00.284626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":219,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210319+0000","last_change":"2024-02-19T21:06:17.907387+0000","last_active":"2024-02-19T21:07:03.210319+0000","last_peered":"2024-02-19T21:07:03.210319+0000","last_clean":"2024-02-19T21:07:03.210319+0000","last_became_active":"2024-02-19T21:05:20.195763+0000","last_became_peered":"2024-02-19T21:05:20.195763+0000","last_unstale":"2024-02-19T21:07:03.210319+0000","last_undegraded":"2024-02-19T21:07:03.210319+0000","last_fullsized":"2024-02-19T21:07:03.210319+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:17.907318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:17.907318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200578+0000","last_change":"2024-02-19T21:04:54.972925+0000","last_active":"2024-02-19T21:07:03.200578+0000","last_peered":"2024-02-19T21:07:03.200578+0000","last_clean":"2024-02-19T21:07:03.200578+0000","last_became_active":"2024-02-19T21:01:16.274290+0000","last_became_peered":"2024-02-19T21:01:16.274290+0000","last_unstale":"2024-02-19T21:07:03.200578+0000","last_undegraded":"2024-02-19T21:07:03.200578+0000","last_fullsized":"2024-02-19T21:07:03.200578+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:54.972832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:54.972832+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.211545+0000","last_change":"2024-02-19T21:06:16.863491+0000","last_active":"2024-02-19T21:07:03.211545+0000","last_peered":"2024-02-19T21:07:03.211545+0000","last_clean":"2024-02-19T21:07:03.211545+0000","last_became_active":"2024-02-19T21:01:22.292299+0000","last_became_peered":"2024-02-19T21:01:22.292299+0000","last_unstale":"2024-02-19T21:07:03.211545+0000","last_undegraded":"2024-02-19T21:07:03.211545+0000","last_fullsized":"2024-02-19T21:07:03.211545+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:16.863398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.863398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":158,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207550+0000","last_change":"2024-02-19T21:06:52.403019+0000","last_active":"2024-02-19T21:07:03.207550+0000","last_peered":"2024-02-19T21:07:03.207550+0000","last_clean":"2024-02-19T21:07:03.207550+0000","last_became_active":"2024-02-19T21:04:39.666739+0000","last_became_peered":"2024-02-19T21:04:39.666739+0000","last_unstale":"2024-02-19T21:07:03.207550+0000","last_undegraded":"2024-02-19T21:07:03.207550+0000","last_fullsized":"2024-02-19T21:07:03.207550+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:52.402941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:52.402941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":206,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.203911+0000","last_change":"2024-02-19T21:06:23.273349+0000","last_active":"2024-02-19T21:07:03.203911+0000","last_peered":"2024-02-19T21:07:03.203911+0000","last_clean":"2024-02-19T21:07:03.203911+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:07:03.203911+0000","last_undegraded":"2024-02-19T21:07:03.203911+0000","last_fullsized":"2024-02-19T21:07:03.203911+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:23.273274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:23.273274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":224,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.203982+0000","last_change":"2024-02-19T21:07:01.264194+0000","last_active":"2024-02-19T21:07:03.203982+0000","last_peered":"2024-02-19T21:07:03.203982+0000","last_clean":"2024-02-19T21:07:03.203982+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:07:03.203982+0000","last_undegraded":"2024-02-19T21:07:03.203982+0000","last_fullsized":"2024-02-19T21:07:03.203982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:07:01.264093+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:07:01.264093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":224,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.204082+0000","last_change":"2024-02-19T21:05:45.014031+0000","last_active":"2024-02-19T21:07:03.204082+0000","last_peered":"2024-02-19T21:07:03.204082+0000","last_clean":"2024-02-19T21:07:03.204082+0000","last_became_active":"2024-02-19T21:05:20.219851+0000","last_became_peered":"2024-02-19T21:05:20.219851+0000","last_unstale":"2024-02-19T21:07:03.204082+0000","last_undegraded":"2024-02-19T21:07:03.204082+0000","last_fullsized":"2024-02-19T21:07:03.204082+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:45.013946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:45.013946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":554,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.210270+0000","last_change":"2024-02-19T21:06:10.791107+0000","last_active":"2024-02-19T21:07:04.210270+0000","last_peered":"2024-02-19T21:07:04.210270+0000","last_clean":"2024-02-19T21:07:04.210270+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:07:04.210270+0000","last_undegraded":"2024-02-19T21:07:04.210270+0000","last_fullsized":"2024-02-19T21:07:04.210270+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:06:10.791036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:10.791036+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":341,"num_read_kb":336,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":203,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210849+0000","last_change":"2024-02-19T21:06:49.690671+0000","last_active":"2024-02-19T21:07:03.210849+0000","last_peered":"2024-02-19T21:07:03.210849+0000","last_clean":"2024-02-19T21:07:03.210849+0000","last_became_active":"2024-02-19T21:01:22.607883+0000","last_became_peered":"2024-02-19T21:01:22.607883+0000","last_unstale":"2024-02-19T21:07:03.210849+0000","last_undegraded":"2024-02-19T21:07:03.210849+0000","last_fullsized":"2024-02-19T21:07:03.210849+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.690596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.690596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":153,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207483+0000","last_change":"2024-02-19T21:04:39.692210+0000","last_active":"2024-02-19T21:07:03.207483+0000","last_peered":"2024-02-19T21:07:03.207483+0000","last_clean":"2024-02-19T21:07:03.207483+0000","last_became_active":"2024-02-19T21:04:39.691434+0000","last_became_peered":"2024-02-19T21:04:39.691434+0000","last_unstale":"2024-02-19T21:07:03.207483+0000","last_undegraded":"2024-02-19T21:07:03.207483+0000","last_fullsized":"2024-02-19T21:07:03.207483+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"100'2","reported_seq":179,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208003+0000","last_change":"2024-02-19T21:06:54.379177+0000","last_active":"2024-02-19T21:07:03.208003+0000","last_peered":"2024-02-19T21:07:03.208003+0000","last_clean":"2024-02-19T21:07:03.208003+0000","last_became_active":"2024-02-19T21:06:12.215103+0000","last_became_peered":"2024-02-19T21:06:12.215103+0000","last_unstale":"2024-02-19T21:07:03.208003+0000","last_undegraded":"2024-02-19T21:07:03.208003+0000","last_fullsized":"2024-02-19T21:07:03.208003+0000","mapping_epoch":98,"log_start":"100'2","ondisk_log_start":"100'2","created":98,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'2","last_scrub_stamp":"2024-02-19T21:06:54.379107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.379107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":36,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208270+0000","last_change":"2024-02-19T21:06:40.245910+0000","last_active":"2024-02-19T21:07:03.208270+0000","last_peered":"2024-02-19T21:07:03.208270+0000","last_clean":"2024-02-19T21:07:03.208270+0000","last_became_active":"2024-02-19T21:06:40.245469+0000","last_became_peered":"2024-02-19T21:06:40.245469+0000","last_unstale":"2024-02-19T21:07:03.208270+0000","last_undegraded":"2024-02-19T21:07:03.208270+0000","last_fullsized":"2024-02-19T21:07:03.208270+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:01.368509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.368509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"65'28","reported_seq":326,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.210378+0000","last_change":"2024-02-19T21:05:42.927711+0000","last_active":"2024-02-19T21:07:04.210378+0000","last_peered":"2024-02-19T21:07:04.210378+0000","last_clean":"2024-02-19T21:07:04.210378+0000","last_became_active":"2024-02-19T21:01:18.289076+0000","last_became_peered":"2024-02-19T21:01:18.289076+0000","last_unstale":"2024-02-19T21:07:04.210378+0000","last_undegraded":"2024-02-19T21:07:04.210378+0000","last_fullsized":"2024-02-19T21:07:04.210378+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-19T21:05:42.927640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:42.927640+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":219,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208488+0000","last_change":"2024-02-19T21:06:19.845462+0000","last_active":"2024-02-19T21:07:03.208488+0000","last_peered":"2024-02-19T21:07:03.208488+0000","last_clean":"2024-02-19T21:07:03.208488+0000","last_became_active":"2024-02-19T21:05:20.181450+0000","last_became_peered":"2024-02-19T21:05:20.181450+0000","last_unstale":"2024-02-19T21:07:03.208488+0000","last_undegraded":"2024-02-19T21:07:03.208488+0000","last_fullsized":"2024-02-19T21:07:03.208488+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:06:19.845394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":199,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.199978+0000","last_change":"2024-02-19T21:04:49.004211+0000","last_active":"2024-02-19T21:07:03.199978+0000","last_peered":"2024-02-19T21:07:03.199978+0000","last_clean":"2024-02-19T21:07:03.199978+0000","last_became_active":"2024-02-19T21:01:22.291396+0000","last_became_peered":"2024-02-19T21:01:22.291396+0000","last_unstale":"2024-02-19T21:07:03.199978+0000","last_undegraded":"2024-02-19T21:07:03.199978+0000","last_fullsized":"2024-02-19T21:07:03.199978+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.004148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.004148+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":158,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.203392+0000","last_change":"2024-02-19T21:06:54.355870+0000","last_active":"2024-02-19T21:07:03.203392+0000","last_peered":"2024-02-19T21:07:03.203392+0000","last_clean":"2024-02-19T21:07:03.203392+0000","last_became_active":"2024-02-19T21:04:39.721416+0000","last_became_peered":"2024-02-19T21:04:39.721416+0000","last_unstale":"2024-02-19T21:07:03.203392+0000","last_undegraded":"2024-02-19T21:07:03.203392+0000","last_fullsized":"2024-02-19T21:07:03.203392+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.355809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.355809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"220'9","reported_seq":63,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.201964+0000","last_change":"2024-02-19T21:06:40.228012+0000","last_active":"2024-02-19T21:07:04.201964+0000","last_peered":"2024-02-19T21:07:04.201964+0000","last_clean":"2024-02-19T21:07:04.201964+0000","last_became_active":"2024-02-19T21:06:40.227604+0000","last_became_peered":"2024-02-19T21:06:40.227604+0000","last_unstale":"2024-02-19T21:07:04.201964+0000","last_undegraded":"2024-02-19T21:07:04.201964+0000","last_fullsized":"2024-02-19T21:07:04.201964+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:21.819105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:21.819105+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":218,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.206615+0000","last_change":"2024-02-19T21:05:03.922155+0000","last_active":"2024-02-19T21:07:03.206615+0000","last_peered":"2024-02-19T21:07:03.206615+0000","last_clean":"2024-02-19T21:07:03.206615+0000","last_became_active":"2024-02-19T21:01:18.285891+0000","last_became_peered":"2024-02-19T21:01:18.285891+0000","last_unstale":"2024-02-19T21:07:03.206615+0000","last_undegraded":"2024-02-19T21:07:03.206615+0000","last_fullsized":"2024-02-19T21:07:03.206615+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:05:03.922055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:03.922055+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":228,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205538+0000","last_change":"2024-02-19T21:05:47.885080+0000","last_active":"2024-02-19T21:07:03.205538+0000","last_peered":"2024-02-19T21:07:03.205538+0000","last_clean":"2024-02-19T21:07:03.205538+0000","last_became_active":"2024-02-19T21:05:20.222948+0000","last_became_peered":"2024-02-19T21:05:20.222948+0000","last_unstale":"2024-02-19T21:07:03.205538+0000","last_undegraded":"2024-02-19T21:07:03.205538+0000","last_fullsized":"2024-02-19T21:07:03.205538+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:47.885024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:05:47.885024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":36,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.199069+0000","last_change":"2024-02-19T21:06:40.226781+0000","last_active":"2024-02-19T21:07:03.199069+0000","last_peered":"2024-02-19T21:07:03.199069+0000","last_clean":"2024-02-19T21:07:03.199069+0000","last_became_active":"2024-02-19T21:06:40.226458+0000","last_became_peered":"2024-02-19T21:06:40.226458+0000","last_unstale":"2024-02-19T21:07:03.199069+0000","last_undegraded":"2024-02-19T21:07:03.199069+0000","last_fullsized":"2024-02-19T21:07:03.199069+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:15.807734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:15.807734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":158,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.199653+0000","last_change":"2024-02-19T21:06:58.726926+0000","last_active":"2024-02-19T21:07:03.199653+0000","last_peered":"2024-02-19T21:07:03.199653+0000","last_clean":"2024-02-19T21:07:03.199653+0000","last_became_active":"2024-02-19T21:04:39.766156+0000","last_became_peered":"2024-02-19T21:04:39.766156+0000","last_unstale":"2024-02-19T21:07:03.199653+0000","last_undegraded":"2024-02-19T21:07:03.199653+0000","last_fullsized":"2024-02-19T21:07:03.199653+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:58.726858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:58.726858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.5","version":"168'9","reported_seq":194,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.213952+0000","last_change":"2024-02-19T21:06:56.208067+0000","last_active":"2024-02-19T21:07:03.213952+0000","last_peered":"2024-02-19T21:07:03.213952+0000","last_clean":"2024-02-19T21:07:03.213952+0000","last_became_active":"2024-02-19T21:06:56.207595+0000","last_became_peered":"2024-02-19T21:06:56.207595+0000","last_unstale":"2024-02-19T21:07:03.213952+0000","last_undegraded":"2024-02-19T21:07:03.213952+0000","last_fullsized":"2024-02-19T21:07:03.213952+0000","mapping_epoch":98,"log_start":"168'9","ondisk_log_start":"168'9","created":98,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:16.827566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:33.426263+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.827566+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":206,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.203179+0000","last_change":"2024-02-19T21:06:20.194149+0000","last_active":"2024-02-19T21:07:03.203179+0000","last_peered":"2024-02-19T21:07:03.203179+0000","last_clean":"2024-02-19T21:07:03.203179+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:07:03.203179+0000","last_undegraded":"2024-02-19T21:07:03.203179+0000","last_fullsized":"2024-02-19T21:07:03.203179+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:20.194018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:06:20.194018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":231,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.202253+0000","last_change":"2024-02-19T21:06:48.844390+0000","last_active":"2024-02-19T21:07:03.202253+0000","last_peered":"2024-02-19T21:07:03.202253+0000","last_clean":"2024-02-19T21:07:03.202253+0000","last_became_active":"2024-02-19T21:01:18.285672+0000","last_became_peered":"2024-02-19T21:01:18.285672+0000","last_unstale":"2024-02-19T21:07:03.202253+0000","last_undegraded":"2024-02-19T21:07:03.202253+0000","last_fullsized":"2024-02-19T21:07:03.202253+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:06:48.844301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.844301+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":225,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.202484+0000","last_change":"2024-02-19T21:05:01.259381+0000","last_active":"2024-02-19T21:07:03.202484+0000","last_peered":"2024-02-19T21:07:03.202484+0000","last_clean":"2024-02-19T21:07:03.202484+0000","last_became_active":"2024-02-19T21:01:16.273545+0000","last_became_peered":"2024-02-19T21:01:16.273545+0000","last_unstale":"2024-02-19T21:07:03.202484+0000","last_undegraded":"2024-02-19T21:07:03.202484+0000","last_fullsized":"2024-02-19T21:07:03.202484+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:05:01.259313+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.259313+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":199,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201020+0000","last_change":"2024-02-19T21:05:06.001027+0000","last_active":"2024-02-19T21:07:03.201020+0000","last_peered":"2024-02-19T21:07:03.201020+0000","last_clean":"2024-02-19T21:07:03.201020+0000","last_became_active":"2024-02-19T21:01:22.295912+0000","last_became_peered":"2024-02-19T21:01:22.295912+0000","last_unstale":"2024-02-19T21:07:03.201020+0000","last_undegraded":"2024-02-19T21:07:03.201020+0000","last_fullsized":"2024-02-19T21:07:03.201020+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:06.000903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:05:06.000903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":153,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.203483+0000","last_change":"2024-02-19T21:04:39.696886+0000","last_active":"2024-02-19T21:07:03.203483+0000","last_peered":"2024-02-19T21:07:03.203483+0000","last_clean":"2024-02-19T21:07:03.203483+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:07:03.203483+0000","last_undegraded":"2024-02-19T21:07:03.203483+0000","last_fullsized":"2024-02-19T21:07:03.203483+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":64,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200046+0000","last_change":"2024-02-19T21:06:40.227501+0000","last_active":"2024-02-19T21:07:03.200046+0000","last_peered":"2024-02-19T21:07:03.200046+0000","last_clean":"2024-02-19T21:07:03.200046+0000","last_became_active":"2024-02-19T21:06:40.227142+0000","last_became_peered":"2024-02-19T21:06:40.227142+0000","last_unstale":"2024-02-19T21:07:03.200046+0000","last_undegraded":"2024-02-19T21:07:03.200046+0000","last_fullsized":"2024-02-19T21:07:03.200046+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:06:09.793208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.793208+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.203106+0000","last_change":"2024-02-19T21:04:49.819984+0000","last_active":"2024-02-19T21:07:03.203106+0000","last_peered":"2024-02-19T21:07:03.203106+0000","last_clean":"2024-02-19T21:07:03.203106+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:07:03.203106+0000","last_undegraded":"2024-02-19T21:07:03.203106+0000","last_fullsized":"2024-02-19T21:07:03.203106+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:49.819921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:49.819921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":206,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200441+0000","last_change":"2024-02-19T21:06:56.709820+0000","last_active":"2024-02-19T21:07:03.200441+0000","last_peered":"2024-02-19T21:07:03.200441+0000","last_clean":"2024-02-19T21:07:03.200441+0000","last_became_active":"2024-02-19T21:01:20.285298+0000","last_became_peered":"2024-02-19T21:01:20.285298+0000","last_unstale":"2024-02-19T21:07:03.200441+0000","last_undegraded":"2024-02-19T21:07:03.200441+0000","last_fullsized":"2024-02-19T21:07:03.200441+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:56.709750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:06:56.709750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.202919+0000","last_change":"2024-02-19T21:05:11.826779+0000","last_active":"2024-02-19T21:07:03.202919+0000","last_peered":"2024-02-19T21:07:03.202919+0000","last_clean":"2024-02-19T21:07:03.202919+0000","last_became_active":"2024-02-19T21:01:22.297086+0000","last_became_peered":"2024-02-19T21:01:22.297086+0000","last_unstale":"2024-02-19T21:07:03.202919+0000","last_undegraded":"2024-02-19T21:07:03.202919+0000","last_fullsized":"2024-02-19T21:07:03.202919+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:11.826647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:11.826647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":158,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201158+0000","last_change":"2024-02-19T21:06:21.704812+0000","last_active":"2024-02-19T21:07:03.201158+0000","last_peered":"2024-02-19T21:07:03.201158+0000","last_clean":"2024-02-19T21:07:03.201158+0000","last_became_active":"2024-02-19T21:04:39.639518+0000","last_became_peered":"2024-02-19T21:04:39.639518+0000","last_unstale":"2024-02-19T21:07:03.201158+0000","last_undegraded":"2024-02-19T21:07:03.201158+0000","last_fullsized":"2024-02-19T21:07:03.201158+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:21.704719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:21.704719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":220,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200378+0000","last_change":"2024-02-19T21:06:48.855339+0000","last_active":"2024-02-19T21:07:03.200378+0000","last_peered":"2024-02-19T21:07:03.200378+0000","last_clean":"2024-02-19T21:07:03.200378+0000","last_became_active":"2024-02-19T21:01:18.293913+0000","last_became_peered":"2024-02-19T21:01:18.293913+0000","last_unstale":"2024-02-19T21:07:03.200378+0000","last_undegraded":"2024-02-19T21:07:03.200378+0000","last_fullsized":"2024-02-19T21:07:03.200378+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:06:48.855268+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.855268+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.214042+0000","last_change":"2024-02-19T21:06:15.793461+0000","last_active":"2024-02-19T21:07:03.214042+0000","last_peered":"2024-02-19T21:07:03.214042+0000","last_clean":"2024-02-19T21:07:03.214042+0000","last_became_active":"2024-02-19T21:01:16.271210+0000","last_became_peered":"2024-02-19T21:01:16.271210+0000","last_unstale":"2024-02-19T21:07:03.214042+0000","last_undegraded":"2024-02-19T21:07:03.214042+0000","last_fullsized":"2024-02-19T21:07:03.214042+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:15.793328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:15.793328+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":203,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.209365+0000","last_change":"2024-02-19T21:05:04.174022+0000","last_active":"2024-02-19T21:07:03.209365+0000","last_peered":"2024-02-19T21:07:03.209365+0000","last_clean":"2024-02-19T21:07:03.209365+0000","last_became_active":"2024-02-19T21:01:20.288589+0000","last_became_peered":"2024-02-19T21:01:20.288589+0000","last_unstale":"2024-02-19T21:07:03.209365+0000","last_undegraded":"2024-02-19T21:07:03.209365+0000","last_fullsized":"2024-02-19T21:07:03.209365+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:04.173902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:04.173902+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":203,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.206121+0000","last_change":"2024-02-19T21:06:05.431079+0000","last_active":"2024-02-19T21:07:03.206121+0000","last_peered":"2024-02-19T21:07:03.206121+0000","last_clean":"2024-02-19T21:07:03.206121+0000","last_became_active":"2024-02-19T21:01:22.608285+0000","last_became_peered":"2024-02-19T21:01:22.608285+0000","last_unstale":"2024-02-19T21:07:03.206121+0000","last_undegraded":"2024-02-19T21:07:03.206121+0000","last_fullsized":"2024-02-19T21:07:03.206121+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:05.431017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:05.431017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":153,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208221+0000","last_change":"2024-02-19T21:04:39.641073+0000","last_active":"2024-02-19T21:07:03.208221+0000","last_peered":"2024-02-19T21:07:03.208221+0000","last_clean":"2024-02-19T21:07:03.208221+0000","last_became_active":"2024-02-19T21:04:39.639775+0000","last_became_peered":"2024-02-19T21:04:39.639775+0000","last_unstale":"2024-02-19T21:07:03.208221+0000","last_undegraded":"2024-02-19T21:07:03.208221+0000","last_fullsized":"2024-02-19T21:07:03.208221+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":223,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201731+0000","last_change":"2024-02-19T21:07:01.731139+0000","last_active":"2024-02-19T21:07:03.201731+0000","last_peered":"2024-02-19T21:07:03.201731+0000","last_clean":"2024-02-19T21:07:03.201731+0000","last_became_active":"2024-02-19T21:01:18.294369+0000","last_became_peered":"2024-02-19T21:01:18.294369+0000","last_unstale":"2024-02-19T21:07:03.201731+0000","last_undegraded":"2024-02-19T21:07:03.201731+0000","last_fullsized":"2024-02-19T21:07:03.201731+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:07:01.731068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:07:01.731068+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":210,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208482+0000","last_change":"2024-02-19T21:06:15.892999+0000","last_active":"2024-02-19T21:07:03.208482+0000","last_peered":"2024-02-19T21:07:03.208482+0000","last_clean":"2024-02-19T21:07:03.208482+0000","last_became_active":"2024-02-19T21:01:16.271608+0000","last_became_peered":"2024-02-19T21:01:16.271608+0000","last_unstale":"2024-02-19T21:07:03.208482+0000","last_undegraded":"2024-02-19T21:07:03.208482+0000","last_fullsized":"2024-02-19T21:07:03.208482+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:15.892925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:15.892925+0000","last_clean_scrub_stamp":"2024-02-19T21:06:15.892925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":279,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.216532+0000","last_change":"2024-02-19T21:05:34.889050+0000","last_active":"2024-02-19T21:07:04.216532+0000","last_peered":"2024-02-19T21:07:04.216532+0000","last_clean":"2024-02-19T21:07:04.216532+0000","last_became_active":"2024-02-19T21:01:20.288799+0000","last_became_peered":"2024-02-19T21:01:20.288799+0000","last_unstale":"2024-02-19T21:07:04.216532+0000","last_undegraded":"2024-02-19T21:07:04.216532+0000","last_fullsized":"2024-02-19T21:07:04.216532+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:05:34.888948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:34.888948+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":45,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210249+0000","last_change":"2024-02-19T21:06:40.229090+0000","last_active":"2024-02-19T21:07:03.210249+0000","last_peered":"2024-02-19T21:07:03.210249+0000","last_clean":"2024-02-19T21:07:03.210249+0000","last_became_active":"2024-02-19T21:06:40.228391+0000","last_became_peered":"2024-02-19T21:06:40.228391+0000","last_unstale":"2024-02-19T21:07:03.210249+0000","last_undegraded":"2024-02-19T21:07:03.210249+0000","last_fullsized":"2024-02-19T21:07:03.210249+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:05:43.790222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:43.790222+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":35,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.206028+0000","last_change":"2024-02-19T21:06:40.219921+0000","last_active":"2024-02-19T21:07:03.206028+0000","last_peered":"2024-02-19T21:07:03.206028+0000","last_clean":"2024-02-19T21:07:03.206028+0000","last_became_active":"2024-02-19T21:06:40.219590+0000","last_became_peered":"2024-02-19T21:06:40.219590+0000","last_unstale":"2024-02-19T21:07:03.206028+0000","last_undegraded":"2024-02-19T21:07:03.206028+0000","last_fullsized":"2024-02-19T21:07:03.206028+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:17.776057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:17.776057+0000","last_clean_scrub_stamp":"2024-02-19T21:06:17.776057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":224,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.209654+0000","last_change":"2024-02-19T21:06:47.541116+0000","last_active":"2024-02-19T21:07:03.209654+0000","last_peered":"2024-02-19T21:07:03.209654+0000","last_clean":"2024-02-19T21:07:03.209654+0000","last_became_active":"2024-02-19T21:01:18.293786+0000","last_became_peered":"2024-02-19T21:01:18.293786+0000","last_unstale":"2024-02-19T21:07:03.209654+0000","last_undegraded":"2024-02-19T21:07:03.209654+0000","last_fullsized":"2024-02-19T21:07:03.209654+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:06:47.541012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.541012+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":36,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200697+0000","last_change":"2024-02-19T21:06:40.220653+0000","last_active":"2024-02-19T21:07:03.200697+0000","last_peered":"2024-02-19T21:07:03.200697+0000","last_clean":"2024-02-19T21:07:03.200697+0000","last_became_active":"2024-02-19T21:06:40.220310+0000","last_became_peered":"2024-02-19T21:06:40.220310+0000","last_unstale":"2024-02-19T21:07:03.200697+0000","last_undegraded":"2024-02-19T21:07:03.200697+0000","last_fullsized":"2024-02-19T21:07:03.200697+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:10.715442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:10.715442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"65'8","reported_seq":279,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.204999+0000","last_change":"2024-02-19T21:05:02.007769+0000","last_active":"2024-02-19T21:07:04.204999+0000","last_peered":"2024-02-19T21:07:04.204999+0000","last_clean":"2024-02-19T21:07:04.204999+0000","last_became_active":"2024-02-19T21:01:20.283943+0000","last_became_peered":"2024-02-19T21:01:20.283943+0000","last_unstale":"2024-02-19T21:07:04.204999+0000","last_undegraded":"2024-02-19T21:07:04.204999+0000","last_fullsized":"2024-02-19T21:07:04.204999+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:05:02.007654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:02.007654+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.202646+0000","last_change":"2024-02-19T21:06:15.753686+0000","last_active":"2024-02-19T21:07:03.202646+0000","last_peered":"2024-02-19T21:07:03.202646+0000","last_clean":"2024-02-19T21:07:03.202646+0000","last_became_active":"2024-02-19T21:01:22.299969+0000","last_became_peered":"2024-02-19T21:01:22.299969+0000","last_unstale":"2024-02-19T21:07:03.202646+0000","last_undegraded":"2024-02-19T21:07:03.202646+0000","last_fullsized":"2024-02-19T21:07:03.202646+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:15.753595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:39.092154+0000","last_clean_scrub_stamp":"2024-02-19T21:06:15.753595+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":171,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.203904+0000","last_change":"2024-02-19T21:06:49.305197+0000","last_active":"2024-02-19T21:07:03.203904+0000","last_peered":"2024-02-19T21:07:03.203904+0000","last_clean":"2024-02-19T21:07:03.203904+0000","last_became_active":"2024-02-19T21:04:39.760167+0000","last_became_peered":"2024-02-19T21:04:39.760167+0000","last_unstale":"2024-02-19T21:07:03.203904+0000","last_undegraded":"2024-02-19T21:07:03.203904+0000","last_fullsized":"2024-02-19T21:07:03.203904+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-02-19T21:06:49.305118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.305118+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":207,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208373+0000","last_change":"2024-02-19T21:05:39.892409+0000","last_active":"2024-02-19T21:07:03.208373+0000","last_peered":"2024-02-19T21:07:03.208373+0000","last_clean":"2024-02-19T21:07:03.208373+0000","last_became_active":"2024-02-19T21:01:30.922577+0000","last_became_peered":"2024-02-19T21:01:30.922577+0000","last_unstale":"2024-02-19T21:07:03.208373+0000","last_undegraded":"2024-02-19T21:07:03.208373+0000","last_fullsized":"2024-02-19T21:07:03.208373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:05:39.892300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:39.892300+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207803+0000","last_change":"2024-02-19T21:05:01.685058+0000","last_active":"2024-02-19T21:07:03.207803+0000","last_peered":"2024-02-19T21:07:03.207803+0000","last_clean":"2024-02-19T21:07:03.207803+0000","last_became_active":"2024-02-19T21:01:16.271316+0000","last_became_peered":"2024-02-19T21:01:16.271316+0000","last_unstale":"2024-02-19T21:07:03.207803+0000","last_undegraded":"2024-02-19T21:07:03.207803+0000","last_fullsized":"2024-02-19T21:07:03.207803+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:05:01.684958+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":201,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200959+0000","last_change":"2024-02-19T21:04:56.999269+0000","last_active":"2024-02-19T21:07:03.200959+0000","last_peered":"2024-02-19T21:07:03.200959+0000","last_clean":"2024-02-19T21:07:03.200959+0000","last_became_active":"2024-02-19T21:01:20.289691+0000","last_became_peered":"2024-02-19T21:01:20.289691+0000","last_unstale":"2024-02-19T21:07:03.200959+0000","last_undegraded":"2024-02-19T21:07:03.200959+0000","last_fullsized":"2024-02-19T21:07:03.200959+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:56.999142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:03.776397+0000","last_clean_scrub_stamp":"2024-02-19T21:04:56.999142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":36,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207884+0000","last_change":"2024-02-19T21:06:40.239397+0000","last_active":"2024-02-19T21:07:03.207884+0000","last_peered":"2024-02-19T21:07:03.207884+0000","last_clean":"2024-02-19T21:07:03.207884+0000","last_became_active":"2024-02-19T21:06:40.239199+0000","last_became_peered":"2024-02-19T21:06:40.239199+0000","last_unstale":"2024-02-19T21:07:03.207884+0000","last_undegraded":"2024-02-19T21:07:03.207884+0000","last_fullsized":"2024-02-19T21:07:03.207884+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:12.815725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:12.815725+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":152,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.215524+0000","last_change":"2024-02-19T21:04:39.674336+0000","last_active":"2024-02-19T21:07:03.215524+0000","last_peered":"2024-02-19T21:07:03.215524+0000","last_clean":"2024-02-19T21:07:03.215524+0000","last_became_active":"2024-02-19T21:04:39.672228+0000","last_became_peered":"2024-02-19T21:04:39.672228+0000","last_unstale":"2024-02-19T21:07:03.215524+0000","last_undegraded":"2024-02-19T21:07:03.215524+0000","last_fullsized":"2024-02-19T21:07:03.215524+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":227,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201332+0000","last_change":"2024-02-19T21:06:09.595836+0000","last_active":"2024-02-19T21:07:03.201332+0000","last_peered":"2024-02-19T21:07:03.201332+0000","last_clean":"2024-02-19T21:07:03.201332+0000","last_became_active":"2024-02-19T21:01:18.286128+0000","last_became_peered":"2024-02-19T21:01:18.286128+0000","last_unstale":"2024-02-19T21:07:03.201332+0000","last_undegraded":"2024-02-19T21:07:03.201332+0000","last_fullsized":"2024-02-19T21:07:03.201332+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:06:09.595752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.595752+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":210,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208410+0000","last_change":"2024-02-19T21:05:09.756185+0000","last_active":"2024-02-19T21:07:03.208410+0000","last_peered":"2024-02-19T21:07:03.208410+0000","last_clean":"2024-02-19T21:07:03.208410+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:07:03.208410+0000","last_undegraded":"2024-02-19T21:07:03.208410+0000","last_fullsized":"2024-02-19T21:07:03.208410+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:05:09.756096+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":206,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.206689+0000","last_change":"2024-02-19T21:05:43.904934+0000","last_active":"2024-02-19T21:07:03.206689+0000","last_peered":"2024-02-19T21:07:03.206689+0000","last_clean":"2024-02-19T21:07:03.206689+0000","last_became_active":"2024-02-19T21:01:20.289020+0000","last_became_peered":"2024-02-19T21:01:20.289020+0000","last_unstale":"2024-02-19T21:07:03.206689+0000","last_undegraded":"2024-02-19T21:07:03.206689+0000","last_fullsized":"2024-02-19T21:07:03.206689+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:43.904870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:43.904870+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":199,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210458+0000","last_change":"2024-02-19T21:04:51.014323+0000","last_active":"2024-02-19T21:07:03.210458+0000","last_peered":"2024-02-19T21:07:03.210458+0000","last_clean":"2024-02-19T21:07:03.210458+0000","last_became_active":"2024-02-19T21:01:22.292704+0000","last_became_peered":"2024-02-19T21:01:22.292704+0000","last_unstale":"2024-02-19T21:07:03.210458+0000","last_undegraded":"2024-02-19T21:07:03.210458+0000","last_fullsized":"2024-02-19T21:07:03.210458+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:51.014245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:04:51.014245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":153,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200186+0000","last_change":"2024-02-19T21:04:39.672911+0000","last_active":"2024-02-19T21:07:03.200186+0000","last_peered":"2024-02-19T21:07:03.200186+0000","last_clean":"2024-02-19T21:07:03.200186+0000","last_became_active":"2024-02-19T21:04:39.672672+0000","last_became_peered":"2024-02-19T21:04:39.672672+0000","last_unstale":"2024-02-19T21:07:03.200186+0000","last_undegraded":"2024-02-19T21:07:03.200186+0000","last_fullsized":"2024-02-19T21:07:03.200186+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":214,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.198989+0000","last_change":"2024-02-19T21:05:08.226255+0000","last_active":"2024-02-19T21:07:03.198989+0000","last_peered":"2024-02-19T21:07:03.198989+0000","last_clean":"2024-02-19T21:07:03.198989+0000","last_became_active":"2024-02-19T21:01:18.287874+0000","last_became_peered":"2024-02-19T21:01:18.287874+0000","last_unstale":"2024-02-19T21:07:03.198989+0000","last_undegraded":"2024-02-19T21:07:03.198989+0000","last_fullsized":"2024-02-19T21:07:03.198989+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:05:08.226144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:08.226144+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":210,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.209787+0000","last_change":"2024-02-19T21:06:24.838727+0000","last_active":"2024-02-19T21:07:03.209787+0000","last_peered":"2024-02-19T21:07:03.209787+0000","last_clean":"2024-02-19T21:07:03.209787+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:07:03.209787+0000","last_undegraded":"2024-02-19T21:07:03.209787+0000","last_fullsized":"2024-02-19T21:07:03.209787+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:24.838658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:24.838658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.213967+0000","last_change":"2024-02-19T21:05:40.919364+0000","last_active":"2024-02-19T21:07:03.213967+0000","last_peered":"2024-02-19T21:07:03.213967+0000","last_clean":"2024-02-19T21:07:03.213967+0000","last_became_active":"2024-02-19T21:01:20.290962+0000","last_became_peered":"2024-02-19T21:01:20.290962+0000","last_unstale":"2024-02-19T21:07:03.213967+0000","last_undegraded":"2024-02-19T21:07:03.213967+0000","last_fullsized":"2024-02-19T21:07:03.213967+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:40.919263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:40.919263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":203,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.204767+0000","last_change":"2024-02-19T21:05:44.331615+0000","last_active":"2024-02-19T21:07:03.204767+0000","last_peered":"2024-02-19T21:07:03.204767+0000","last_clean":"2024-02-19T21:07:03.204767+0000","last_became_active":"2024-02-19T21:01:22.297693+0000","last_became_peered":"2024-02-19T21:01:22.297693+0000","last_unstale":"2024-02-19T21:07:03.204767+0000","last_undegraded":"2024-02-19T21:07:03.204767+0000","last_fullsized":"2024-02-19T21:07:03.204767+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.331546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.331546+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":153,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201581+0000","last_change":"2024-02-19T21:04:39.717968+0000","last_active":"2024-02-19T21:07:03.201581+0000","last_peered":"2024-02-19T21:07:03.201581+0000","last_clean":"2024-02-19T21:07:03.201581+0000","last_became_active":"2024-02-19T21:04:39.716606+0000","last_became_peered":"2024-02-19T21:04:39.716606+0000","last_unstale":"2024-02-19T21:07:03.201581+0000","last_undegraded":"2024-02-19T21:07:03.201581+0000","last_fullsized":"2024-02-19T21:07:03.201581+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":222,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210846+0000","last_change":"2024-02-19T21:04:52.027084+0000","last_active":"2024-02-19T21:07:03.210846+0000","last_peered":"2024-02-19T21:07:03.210846+0000","last_clean":"2024-02-19T21:07:03.210846+0000","last_became_active":"2024-02-19T21:01:18.293793+0000","last_became_peered":"2024-02-19T21:01:18.293793+0000","last_unstale":"2024-02-19T21:07:03.210846+0000","last_undegraded":"2024-02-19T21:07:03.210846+0000","last_fullsized":"2024-02-19T21:07:03.210846+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:04:52.027000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:52.027000+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205952+0000","last_change":"2024-02-19T21:06:11.887212+0000","last_active":"2024-02-19T21:07:03.205952+0000","last_peered":"2024-02-19T21:07:03.205952+0000","last_clean":"2024-02-19T21:07:03.205952+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:07:03.205952+0000","last_undegraded":"2024-02-19T21:07:03.205952+0000","last_fullsized":"2024-02-19T21:07:03.205952+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:11.887139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:11.887139+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":200,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.209539+0000","last_change":"2024-02-19T21:05:43.361840+0000","last_active":"2024-02-19T21:07:03.209539+0000","last_peered":"2024-02-19T21:07:03.209539+0000","last_clean":"2024-02-19T21:07:03.209539+0000","last_became_active":"2024-02-19T21:01:20.287961+0000","last_became_peered":"2024-02-19T21:01:20.287961+0000","last_unstale":"2024-02-19T21:07:03.209539+0000","last_undegraded":"2024-02-19T21:07:03.209539+0000","last_fullsized":"2024-02-19T21:07:03.209539+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:43.361777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:43.361777+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201084+0000","last_change":"2024-02-19T21:06:08.757280+0000","last_active":"2024-02-19T21:07:03.201084+0000","last_peered":"2024-02-19T21:07:03.201084+0000","last_clean":"2024-02-19T21:07:03.201084+0000","last_became_active":"2024-02-19T21:01:22.608997+0000","last_became_peered":"2024-02-19T21:01:22.608997+0000","last_unstale":"2024-02-19T21:07:03.201084+0000","last_undegraded":"2024-02-19T21:07:03.201084+0000","last_fullsized":"2024-02-19T21:07:03.201084+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:08.757198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:08.757198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":40,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.214040+0000","last_change":"2024-02-19T21:06:54.664879+0000","last_active":"2024-02-19T21:07:03.214040+0000","last_peered":"2024-02-19T21:07:03.214040+0000","last_clean":"2024-02-19T21:07:03.214040+0000","last_became_active":"2024-02-19T21:06:40.226196+0000","last_became_peered":"2024-02-19T21:06:40.226196+0000","last_unstale":"2024-02-19T21:07:03.214040+0000","last_undegraded":"2024-02-19T21:07:03.214040+0000","last_fullsized":"2024-02-19T21:07:03.214040+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.664814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":63,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210444+0000","last_change":"2024-02-19T21:06:40.234278+0000","last_active":"2024-02-19T21:07:03.210444+0000","last_peered":"2024-02-19T21:07:03.210444+0000","last_clean":"2024-02-19T21:07:03.210444+0000","last_became_active":"2024-02-19T21:06:40.233941+0000","last_became_peered":"2024-02-19T21:06:40.233941+0000","last_unstale":"2024-02-19T21:07:03.210444+0000","last_undegraded":"2024-02-19T21:07:03.210444+0000","last_fullsized":"2024-02-19T21:07:03.210444+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:05:47.805757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:47.805757+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207793+0000","last_change":"2024-02-19T21:05:07.118727+0000","last_active":"2024-02-19T21:07:03.207793+0000","last_peered":"2024-02-19T21:07:03.207793+0000","last_clean":"2024-02-19T21:07:03.207793+0000","last_became_active":"2024-02-19T21:01:16.272284+0000","last_became_peered":"2024-02-19T21:01:16.272284+0000","last_unstale":"2024-02-19T21:07:03.207793+0000","last_undegraded":"2024-02-19T21:07:03.207793+0000","last_fullsized":"2024-02-19T21:07:03.207793+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:07.118667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:07.118667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"65'8","reported_seq":286,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-02-19T21:07:04.200637+0000","last_change":"2024-02-19T21:05:41.909864+0000","last_active":"2024-02-19T21:07:04.200637+0000","last_peered":"2024-02-19T21:07:04.200637+0000","last_clean":"2024-02-19T21:07:04.200637+0000","last_became_active":"2024-02-19T21:01:20.290567+0000","last_became_peered":"2024-02-19T21:01:20.290567+0000","last_unstale":"2024-02-19T21:07:04.200637+0000","last_undegraded":"2024-02-19T21:07:04.200637+0000","last_fullsized":"2024-02-19T21:07:04.200637+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:05:41.909796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:41.909796+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.204955+0000","last_change":"2024-02-19T21:06:27.520296+0000","last_active":"2024-02-19T21:07:03.204955+0000","last_peered":"2024-02-19T21:07:03.204955+0000","last_clean":"2024-02-19T21:07:03.204955+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:07:03.204955+0000","last_undegraded":"2024-02-19T21:07:03.204955+0000","last_fullsized":"2024-02-19T21:07:03.204955+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:06:27.520213+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:06:27.520213+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":153,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.203211+0000","last_change":"2024-02-19T21:04:39.700717+0000","last_active":"2024-02-19T21:07:03.203211+0000","last_peered":"2024-02-19T21:07:03.203211+0000","last_clean":"2024-02-19T21:07:03.203211+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:07:03.203211+0000","last_undegraded":"2024-02-19T21:07:03.203211+0000","last_fullsized":"2024-02-19T21:07:03.203211+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":226,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.206283+0000","last_change":"2024-02-19T21:05:07.770793+0000","last_active":"2024-02-19T21:07:03.206283+0000","last_peered":"2024-02-19T21:07:03.206283+0000","last_clean":"2024-02-19T21:07:03.206283+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:07:03.206283+0000","last_undegraded":"2024-02-19T21:07:03.206283+0000","last_fullsized":"2024-02-19T21:07:03.206283+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:05:07.770727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:07.770727+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210626+0000","last_change":"2024-02-19T21:04:52.990442+0000","last_active":"2024-02-19T21:07:03.210626+0000","last_peered":"2024-02-19T21:07:03.210626+0000","last_clean":"2024-02-19T21:07:03.210626+0000","last_became_active":"2024-02-19T21:01:16.277826+0000","last_became_peered":"2024-02-19T21:01:16.277826+0000","last_unstale":"2024-02-19T21:07:03.210626+0000","last_undegraded":"2024-02-19T21:07:03.210626+0000","last_fullsized":"2024-02-19T21:07:03.210626+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:52.990315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:52.990315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":36,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210587+0000","last_change":"2024-02-19T21:06:40.234367+0000","last_active":"2024-02-19T21:07:03.210587+0000","last_peered":"2024-02-19T21:07:03.210587+0000","last_clean":"2024-02-19T21:07:03.210587+0000","last_became_active":"2024-02-19T21:06:40.234026+0000","last_became_peered":"2024-02-19T21:06:40.234026+0000","last_unstale":"2024-02-19T21:07:03.210587+0000","last_undegraded":"2024-02-19T21:07:03.210587+0000","last_fullsized":"2024-02-19T21:07:03.210587+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:54.322192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:04:54.322192+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":199,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208185+0000","last_change":"2024-02-19T21:05:08.825596+0000","last_active":"2024-02-19T21:07:03.208185+0000","last_peered":"2024-02-19T21:07:03.208185+0000","last_clean":"2024-02-19T21:07:03.208185+0000","last_became_active":"2024-02-19T21:01:22.293230+0000","last_became_peered":"2024-02-19T21:01:22.293230+0000","last_unstale":"2024-02-19T21:07:03.208185+0000","last_undegraded":"2024-02-19T21:07:03.208185+0000","last_fullsized":"2024-02-19T21:07:03.208185+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:08.825485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:08.825485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":153,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207430+0000","last_change":"2024-02-19T21:04:39.801737+0000","last_active":"2024-02-19T21:07:03.207430+0000","last_peered":"2024-02-19T21:07:03.207430+0000","last_clean":"2024-02-19T21:07:03.207430+0000","last_became_active":"2024-02-19T21:04:39.797018+0000","last_became_peered":"2024-02-19T21:04:39.797018+0000","last_unstale":"2024-02-19T21:07:03.207430+0000","last_undegraded":"2024-02-19T21:07:03.207430+0000","last_fullsized":"2024-02-19T21:07:03.207430+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":233,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.203847+0000","last_change":"2024-02-19T21:06:48.310557+0000","last_active":"2024-02-19T21:07:03.203847+0000","last_peered":"2024-02-19T21:07:03.203847+0000","last_clean":"2024-02-19T21:07:03.203847+0000","last_became_active":"2024-02-19T21:01:18.289712+0000","last_became_peered":"2024-02-19T21:01:18.289712+0000","last_unstale":"2024-02-19T21:07:03.203847+0000","last_undegraded":"2024-02-19T21:07:03.203847+0000","last_fullsized":"2024-02-19T21:07:03.203847+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:06:48.310459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.310459+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.213673+0000","last_change":"2024-02-19T21:06:10.922977+0000","last_active":"2024-02-19T21:07:03.213673+0000","last_peered":"2024-02-19T21:07:03.213673+0000","last_clean":"2024-02-19T21:07:03.213673+0000","last_became_active":"2024-02-19T21:01:16.274524+0000","last_became_peered":"2024-02-19T21:01:16.274524+0000","last_unstale":"2024-02-19T21:07:03.213673+0000","last_undegraded":"2024-02-19T21:07:03.213673+0000","last_fullsized":"2024-02-19T21:07:03.213673+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:10.922913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:10.922913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":206,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208619+0000","last_change":"2024-02-19T21:06:53.629540+0000","last_active":"2024-02-19T21:07:03.208619+0000","last_peered":"2024-02-19T21:07:03.208619+0000","last_clean":"2024-02-19T21:07:03.208619+0000","last_became_active":"2024-02-19T21:01:20.290705+0000","last_became_peered":"2024-02-19T21:01:20.290705+0000","last_unstale":"2024-02-19T21:07:03.208619+0000","last_undegraded":"2024-02-19T21:07:03.208619+0000","last_fullsized":"2024-02-19T21:07:03.208619+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:53.629475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:53.629475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":199,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201483+0000","last_change":"2024-02-19T21:05:37.572761+0000","last_active":"2024-02-19T21:07:03.201483+0000","last_peered":"2024-02-19T21:07:03.201483+0000","last_clean":"2024-02-19T21:07:03.201483+0000","last_became_active":"2024-02-19T21:01:22.607435+0000","last_became_peered":"2024-02-19T21:01:22.607435+0000","last_unstale":"2024-02-19T21:07:03.201483+0000","last_undegraded":"2024-02-19T21:07:03.201483+0000","last_fullsized":"2024-02-19T21:07:03.201483+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:37.572695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:05:37.572695+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":35,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205953+0000","last_change":"2024-02-19T21:06:40.241189+0000","last_active":"2024-02-19T21:07:03.205953+0000","last_peered":"2024-02-19T21:07:03.205953+0000","last_clean":"2024-02-19T21:07:03.205953+0000","last_became_active":"2024-02-19T21:06:40.240859+0000","last_became_peered":"2024-02-19T21:06:40.240859+0000","last_unstale":"2024-02-19T21:07:03.205953+0000","last_undegraded":"2024-02-19T21:07:03.205953+0000","last_fullsized":"2024-02-19T21:07:03.205953+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.797430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.797430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":226,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201230+0000","last_change":"2024-02-19T21:04:50.027760+0000","last_active":"2024-02-19T21:07:03.201230+0000","last_peered":"2024-02-19T21:07:03.201230+0000","last_clean":"2024-02-19T21:07:03.201230+0000","last_became_active":"2024-02-19T21:01:18.285926+0000","last_became_peered":"2024-02-19T21:01:18.285926+0000","last_unstale":"2024-02-19T21:07:03.201230+0000","last_undegraded":"2024-02-19T21:07:03.201230+0000","last_fullsized":"2024-02-19T21:07:03.201230+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:04:50.027690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:04:50.027690+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200901+0000","last_change":"2024-02-19T21:06:02.693096+0000","last_active":"2024-02-19T21:07:03.200901+0000","last_peered":"2024-02-19T21:07:03.200901+0000","last_clean":"2024-02-19T21:07:03.200901+0000","last_became_active":"2024-02-19T21:01:16.274811+0000","last_became_peered":"2024-02-19T21:01:16.274811+0000","last_unstale":"2024-02-19T21:07:03.200901+0000","last_undegraded":"2024-02-19T21:07:03.200901+0000","last_fullsized":"2024-02-19T21:07:03.200901+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:02.693025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:02.693025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":47,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.204237+0000","last_change":"2024-02-19T21:06:27.201481+0000","last_active":"2024-02-19T21:07:03.204237+0000","last_peered":"2024-02-19T21:07:03.204237+0000","last_clean":"2024-02-19T21:07:03.204237+0000","last_became_active":"2024-02-19T21:06:27.201124+0000","last_became_peered":"2024-02-19T21:06:27.201124+0000","last_unstale":"2024-02-19T21:07:03.204237+0000","last_undegraded":"2024-02-19T21:07:03.204237+0000","last_fullsized":"2024-02-19T21:07:03.204237+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:38.564472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:38.564472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200314+0000","last_change":"2024-02-19T21:06:54.723259+0000","last_active":"2024-02-19T21:07:03.200314+0000","last_peered":"2024-02-19T21:07:03.200314+0000","last_clean":"2024-02-19T21:07:03.200314+0000","last_became_active":"2024-02-19T21:01:22.607099+0000","last_became_peered":"2024-02-19T21:01:22.607099+0000","last_unstale":"2024-02-19T21:07:03.200314+0000","last_undegraded":"2024-02-19T21:07:03.200314+0000","last_fullsized":"2024-02-19T21:07:03.200314+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.723156+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":153,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205803+0000","last_change":"2024-02-19T21:04:39.781223+0000","last_active":"2024-02-19T21:07:03.205803+0000","last_peered":"2024-02-19T21:07:03.205803+0000","last_clean":"2024-02-19T21:07:03.205803+0000","last_became_active":"2024-02-19T21:04:39.777442+0000","last_became_peered":"2024-02-19T21:04:39.777442+0000","last_unstale":"2024-02-19T21:07:03.205803+0000","last_undegraded":"2024-02-19T21:07:03.205803+0000","last_fullsized":"2024-02-19T21:07:03.205803+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"194'13","reported_seq":228,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201824+0000","last_change":"2024-02-19T21:06:14.772493+0000","last_active":"2024-02-19T21:07:03.201824+0000","last_peered":"2024-02-19T21:07:03.201824+0000","last_clean":"2024-02-19T21:07:03.201824+0000","last_became_active":"2024-02-19T21:01:18.285269+0000","last_became_peered":"2024-02-19T21:01:18.285269+0000","last_unstale":"2024-02-19T21:07:03.201824+0000","last_undegraded":"2024-02-19T21:07:03.201824+0000","last_fullsized":"2024-02-19T21:07:03.201824+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'13","last_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.772417+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.204517+0000","last_change":"2024-02-19T21:04:59.262530+0000","last_active":"2024-02-19T21:07:03.204517+0000","last_peered":"2024-02-19T21:07:03.204517+0000","last_clean":"2024-02-19T21:07:03.204517+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:07:03.204517+0000","last_undegraded":"2024-02-19T21:07:03.204517+0000","last_fullsized":"2024-02-19T21:07:03.204517+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:59.262400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:04:59.262400+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":201,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210225+0000","last_change":"2024-02-19T21:06:09.853068+0000","last_active":"2024-02-19T21:07:03.210225+0000","last_peered":"2024-02-19T21:07:03.210225+0000","last_clean":"2024-02-19T21:07:03.210225+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:07:03.210225+0000","last_undegraded":"2024-02-19T21:07:03.210225+0000","last_fullsized":"2024-02-19T21:07:03.210225+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:09.852996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.852996+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":36,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205102+0000","last_change":"2024-02-19T21:06:40.239902+0000","last_active":"2024-02-19T21:07:03.205102+0000","last_peered":"2024-02-19T21:07:03.205102+0000","last_clean":"2024-02-19T21:07:03.205102+0000","last_became_active":"2024-02-19T21:06:40.239603+0000","last_became_peered":"2024-02-19T21:06:40.239603+0000","last_unstale":"2024-02-19T21:07:03.205102+0000","last_undegraded":"2024-02-19T21:07:03.205102+0000","last_fullsized":"2024-02-19T21:07:03.205102+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:06:25.785366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":158,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.204095+0000","last_change":"2024-02-19T21:06:58.310402+0000","last_active":"2024-02-19T21:07:03.204095+0000","last_peered":"2024-02-19T21:07:03.204095+0000","last_clean":"2024-02-19T21:07:03.204095+0000","last_became_active":"2024-02-19T21:04:39.736990+0000","last_became_peered":"2024-02-19T21:04:39.736990+0000","last_unstale":"2024-02-19T21:07:03.204095+0000","last_undegraded":"2024-02-19T21:07:03.204095+0000","last_fullsized":"2024-02-19T21:07:03.204095+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:58.310306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:58.310306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":218,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200733+0000","last_change":"2024-02-19T21:06:16.718025+0000","last_active":"2024-02-19T21:07:03.200733+0000","last_peered":"2024-02-19T21:07:03.200733+0000","last_clean":"2024-02-19T21:07:03.200733+0000","last_became_active":"2024-02-19T21:01:18.285735+0000","last_became_peered":"2024-02-19T21:01:18.285735+0000","last_unstale":"2024-02-19T21:07:03.200733+0000","last_undegraded":"2024-02-19T21:07:03.200733+0000","last_fullsized":"2024-02-19T21:07:03.200733+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:06:16.717940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.717940+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":209,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210737+0000","last_change":"2024-02-19T21:05:36.352651+0000","last_active":"2024-02-19T21:07:03.210737+0000","last_peered":"2024-02-19T21:07:03.210737+0000","last_clean":"2024-02-19T21:07:03.210737+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:07:03.210737+0000","last_undegraded":"2024-02-19T21:07:03.210737+0000","last_fullsized":"2024-02-19T21:07:03.210737+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:36.352580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:36.352580+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":205,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205065+0000","last_change":"2024-02-19T21:06:14.447006+0000","last_active":"2024-02-19T21:07:03.205065+0000","last_peered":"2024-02-19T21:07:03.205065+0000","last_clean":"2024-02-19T21:07:03.205065+0000","last_became_active":"2024-02-19T21:01:20.287940+0000","last_became_peered":"2024-02-19T21:01:20.287940+0000","last_unstale":"2024-02-19T21:07:03.205065+0000","last_undegraded":"2024-02-19T21:07:03.205065+0000","last_fullsized":"2024-02-19T21:07:03.205065+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:14.446932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.446932+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.207781+0000","last_change":"2024-02-19T21:06:48.507033+0000","last_active":"2024-02-19T21:07:03.207781+0000","last_peered":"2024-02-19T21:07:03.207781+0000","last_clean":"2024-02-19T21:07:03.207781+0000","last_became_active":"2024-02-19T21:01:22.293282+0000","last_became_peered":"2024-02-19T21:01:22.293282+0000","last_unstale":"2024-02-19T21:07:03.207781+0000","last_undegraded":"2024-02-19T21:07:03.207781+0000","last_fullsized":"2024-02-19T21:07:03.207781+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:48.506964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.506964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":41,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210829+0000","last_change":"2024-02-19T21:06:50.467486+0000","last_active":"2024-02-19T21:07:03.210829+0000","last_peered":"2024-02-19T21:07:03.210829+0000","last_clean":"2024-02-19T21:07:03.210829+0000","last_became_active":"2024-02-19T21:06:40.227713+0000","last_became_peered":"2024-02-19T21:06:40.227713+0000","last_unstale":"2024-02-19T21:07:03.210829+0000","last_undegraded":"2024-02-19T21:07:03.210829+0000","last_fullsized":"2024-02-19T21:07:03.210829+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:50.467411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:50.467411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":219,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200207+0000","last_change":"2024-02-19T21:05:36.527512+0000","last_active":"2024-02-19T21:07:03.200207+0000","last_peered":"2024-02-19T21:07:03.200207+0000","last_clean":"2024-02-19T21:07:03.200207+0000","last_became_active":"2024-02-19T21:01:18.287645+0000","last_became_peered":"2024-02-19T21:01:18.287645+0000","last_unstale":"2024-02-19T21:07:03.200207+0000","last_undegraded":"2024-02-19T21:07:03.200207+0000","last_fullsized":"2024-02-19T21:07:03.200207+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:05:36.527441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:36.527441+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":210,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.199915+0000","last_change":"2024-02-19T21:06:53.752645+0000","last_active":"2024-02-19T21:07:03.199915+0000","last_peered":"2024-02-19T21:07:03.199915+0000","last_clean":"2024-02-19T21:07:03.199915+0000","last_became_active":"2024-02-19T21:01:16.272132+0000","last_became_peered":"2024-02-19T21:01:16.272132+0000","last_unstale":"2024-02-19T21:07:03.199915+0000","last_undegraded":"2024-02-19T21:07:03.199915+0000","last_fullsized":"2024-02-19T21:07:03.199915+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:53.752543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:53.752543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":206,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.205654+0000","last_change":"2024-02-19T21:06:24.702277+0000","last_active":"2024-02-19T21:07:03.205654+0000","last_peered":"2024-02-19T21:07:03.205654+0000","last_clean":"2024-02-19T21:07:03.205654+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:07:03.205654+0000","last_undegraded":"2024-02-19T21:07:03.205654+0000","last_fullsized":"2024-02-19T21:07:03.205654+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:06:24.702197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":204,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.201724+0000","last_change":"2024-02-19T21:06:04.601048+0000","last_active":"2024-02-19T21:07:03.201724+0000","last_peered":"2024-02-19T21:07:03.201724+0000","last_clean":"2024-02-19T21:07:03.201724+0000","last_became_active":"2024-02-19T21:01:22.293432+0000","last_became_peered":"2024-02-19T21:01:22.293432+0000","last_unstale":"2024-02-19T21:07:03.201724+0000","last_undegraded":"2024-02-19T21:07:03.201724+0000","last_fullsized":"2024-02-19T21:07:03.201724+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:04.600973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:04.600973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":153,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210243+0000","last_change":"2024-02-19T21:04:39.662781+0000","last_active":"2024-02-19T21:07:03.210243+0000","last_peered":"2024-02-19T21:07:03.210243+0000","last_clean":"2024-02-19T21:07:03.210243+0000","last_became_active":"2024-02-19T21:04:39.661810+0000","last_became_peered":"2024-02-19T21:04:39.661810+0000","last_unstale":"2024-02-19T21:07:03.210243+0000","last_undegraded":"2024-02-19T21:07:03.210243+0000","last_fullsized":"2024-02-19T21:07:03.210243+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:04:38.494818+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":233,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.210924+0000","last_change":"2024-02-19T21:06:09.763553+0000","last_active":"2024-02-19T21:07:03.210924+0000","last_peered":"2024-02-19T21:07:03.210924+0000","last_clean":"2024-02-19T21:07:03.210924+0000","last_became_active":"2024-02-19T21:01:18.283222+0000","last_became_peered":"2024-02-19T21:01:18.283222+0000","last_unstale":"2024-02-19T21:07:03.210924+0000","last_undegraded":"2024-02-19T21:07:03.210924+0000","last_fullsized":"2024-02-19T21:07:03.210924+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:06:09.763482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.763482+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":36,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208672+0000","last_change":"2024-02-19T21:06:40.228214+0000","last_active":"2024-02-19T21:07:03.208672+0000","last_peered":"2024-02-19T21:07:03.208672+0000","last_clean":"2024-02-19T21:07:03.208672+0000","last_became_active":"2024-02-19T21:06:40.227887+0000","last_became_peered":"2024-02-19T21:06:40.227887+0000","last_unstale":"2024-02-19T21:07:03.208672+0000","last_undegraded":"2024-02-19T21:07:03.208672+0000","last_fullsized":"2024-02-19T21:07:03.208672+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:08.599008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:08.599008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":201,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200506+0000","last_change":"2024-02-19T21:05:41.532900+0000","last_active":"2024-02-19T21:07:03.200506+0000","last_peered":"2024-02-19T21:07:03.200506+0000","last_clean":"2024-02-19T21:07:03.200506+0000","last_became_active":"2024-02-19T21:01:20.289208+0000","last_became_peered":"2024-02-19T21:01:20.289208+0000","last_unstale":"2024-02-19T21:07:03.200506+0000","last_undegraded":"2024-02-19T21:07:03.200506+0000","last_fullsized":"2024-02-19T21:07:03.200506+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:41.532841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:05:41.532841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":199,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.200747+0000","last_change":"2024-02-19T21:05:36.920186+0000","last_active":"2024-02-19T21:07:03.200747+0000","last_peered":"2024-02-19T21:07:03.200747+0000","last_clean":"2024-02-19T21:07:03.200747+0000","last_became_active":"2024-02-19T21:01:22.609059+0000","last_became_peered":"2024-02-19T21:01:22.609059+0000","last_unstale":"2024-02-19T21:07:03.200747+0000","last_undegraded":"2024-02-19T21:07:03.200747+0000","last_fullsized":"2024-02-19T21:07:03.200747+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:36.920115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:05:36.920115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":157,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-02-19T21:07:03.208873+0000","last_change":"2024-02-19T21:06:27.744913+0000","last_active":"2024-02-19T21:07:03.208873+0000","last_peered":"2024-02-19T21:07:03.208873+0000","last_clean":"2024-02-19T21:07:03.208873+0000","last_became_active":"2024-02-19T21:04:39.726195+0000","last_became_peered":"2024-02-19T21:04:39.726195+0000","last_unstale":"2024-02-19T21:07:03.208873+0000","last_undegraded":"2024-02-19T21:07:03.208873+0000","last_fullsized":"2024-02-19T21:07:03.208873+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:27.744835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:27.744835+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":167,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":1211,"num_objects":119,"num_object_clones":1,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":93,"num_read_kb":1,"num_write":532,"num_write_kb":529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1449984,"data_stored":3633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1766,"ondisk_log_size":1766,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":692,"num_read_kb":449,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":341,"num_read_kb":336,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":126,"acting":126,"num_store_stats":8},{"poolid":8,"num_pg":11,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":6144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":164,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266760,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313444,"kb_used_data":3920,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431708,"statfs":{"total":95995035648,"available":95674068992,"internally_reserved":0,"allocated":4014080,"data_stored":2207812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":5.3840000000000003,"5min":1.6120000000000001,"15min":0.877},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":89.456999999999994,"5min":89.456999999999994,"15min":89.456999999999994},"last":0.33300000000000002},{"interface":"front","average":{"1min":4.3209999999999997,"5min":1.387,"15min":0.78900000000000003},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":70.451999999999998,"5min":70.451999999999998,"15min":70.451999999999998},"last":0.41199999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":2.3599999999999999,"5min":1.087,"15min":0.71099999999999997},"min":{"1min":0.29299999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":34.273000000000003,"5min":34.273000000000003,"15min":34.273000000000003},"last":0.36399999999999999},{"interface":"front","average":{"1min":5.3840000000000003,"5min":1.8120000000000001,"15min":0.95199999999999996},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":89.594999999999999,"5min":89.594999999999999,"15min":89.594999999999999},"last":0.435}]},{"osd":2,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":6.2359999999999998,"5min":1.8440000000000001,"15min":0.94799999999999995},"min":{"1min":0.27600000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":104.413,"5min":104.413,"15min":104.413},"last":0.48299999999999998},{"interface":"front","average":{"1min":6.2539999999999996,"5min":1.827,"15min":0.95599999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":104.268,"5min":104.268,"15min":104.268},"last":0.54600000000000004}]},{"osd":3,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":6.2930000000000001,"5min":1.8089999999999999,"15min":0.95799999999999996},"min":{"1min":0.32200000000000001,"5min":0.254,"15min":0.254},"max":{"1min":105.3,"5min":105.3,"15min":105.3},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.1899999999999999,"5min":0.79400000000000004,"15min":0.59399999999999997},"min":{"1min":0.32700000000000001,"5min":0.315,"15min":0.315},"max":{"1min":13.243,"5min":13.243,"15min":13.243},"last":0.56699999999999995}]},{"osd":4,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.45300000000000001,"15min":0.48699999999999999},"min":{"1min":0.27800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.54200000000000004,"5min":1.373,"15min":1.373},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.47199999999999998,"15min":0.48199999999999998},"min":{"1min":0.29199999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55800000000000005,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.64200000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.45200000000000001,"15min":0.45800000000000002},"min":{"1min":0.29399999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.50700000000000001,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.503,"15min":0.50900000000000001},"min":{"1min":0.33800000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.59499999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.63}]},{"osd":6,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.50900000000000001,"15min":0.53100000000000003},"min":{"1min":0.32800000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60399999999999998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.54100000000000004,"15min":0.56699999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64200000000000002,"5min":1.262,"15min":1.262},"last":0.68100000000000005}]}]},{"osd":6,"up_from":45,"seq":193273528398,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309360,"kb_used_data":3936,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435792,"statfs":{"total":95995035648,"available":95678251008,"internally_reserved":0,"allocated":4030464,"data_stored":2207965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.85199999999999998,"15min":0.61299999999999999},"min":{"1min":0.29499999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.70399999999999996,"5min":17.486999999999998,"15min":17.486999999999998},"last":2.9129999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.95099999999999996,"15min":0.64300000000000002},"min":{"1min":0.31900000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.88500000000000001,"5min":17.43,"15min":17.43},"last":2.8999999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.86299999999999999,"15min":0.61499999999999999},"min":{"1min":0.38100000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.66000000000000003,"5min":14.692,"15min":14.692},"last":2.0270000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.99399999999999999,"15min":0.66400000000000003},"min":{"1min":0.33600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.70499999999999996,"5min":14.795,"15min":14.795},"last":0.39000000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.78400000000000003,"15min":0.60799999999999998},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.68700000000000006,"5min":6.931,"15min":6.931},"last":2.0979999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":1.036,"15min":0.67200000000000004},"min":{"1min":0.36199999999999999,"5min":0.318,"15min":0.318},"max":{"1min":0.67200000000000004,"5min":14.391,"15min":14.391},"last":0.40400000000000003}]},{"osd":3,"last update":"Mon Feb 19 21:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":1.0489999999999999,"15min":0.69099999999999995},"min":{"1min":0.34300000000000003,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.68200000000000005,"5min":14.510999999999999,"15min":14.510999999999999},"last":3.0499999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.94999999999999996,"15min":0.64600000000000002},"min":{"1min":0.376,"5min":0.316,"15min":0.316},"max":{"1min":0.80700000000000005,"5min":17.585000000000001,"15min":17.585000000000001},"last":1.927}]},{"osd":4,"last update":"Mon Feb 19 21:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.38500000000000001},"min":{"1min":0.26700000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.63800000000000001,"5min":1.363,"15min":1.363},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.439,"15min":0.41099999999999998},"min":{"1min":0.29999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.65300000000000002,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.46800000000000003}]},{"osd":5,"last update":"Mon Feb 19 21:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.46600000000000003,"15min":0.42399999999999999},"min":{"1min":0.30299999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.66300000000000003,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.45800000000000002},"min":{"1min":0.34599999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.626,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.51500000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:06:22 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.48399999999999999,"15min":0.50800000000000001},"min":{"1min":0.28699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70599999999999996,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.48799999999999999,"15min":0.52000000000000002},"min":{"1min":0.30099999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.69099999999999995,"5min":1.534,"15min":1.534},"last":0.53400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574949,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309368,"kb_used_data":3948,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435784,"statfs":{"total":95995035648,"available":95678242816,"internally_reserved":0,"allocated":4042752,"data_stored":2208358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:07:03 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.95299999999999996,"15min":0.55100000000000005},"min":{"1min":0.214,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47899999999999998,"5min":27.777000000000001,"15min":27.777000000000001},"last":0.245},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.999,"15min":0.57499999999999996},"min":{"1min":0.255,"5min":0.217,"15min":0.217},"max":{"1min":0.47699999999999998,"5min":27.631,"15min":27.631},"last":0.307}]},{"osd":2,"last update":"Mon Feb 19 21:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":1.349,"15min":0.72299999999999998},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.56200000000000006,"5min":27.798999999999999,"15min":27.798999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.371,"5min":1.369,"15min":0.73699999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.627,"5min":27.606999999999999,"15min":27.606999999999999},"last":0.34699999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.91800000000000004,"15min":0.59099999999999997},"min":{"1min":0.254,"5min":0.218,"15min":0.218},"max":{"1min":0.49399999999999999,"5min":15.574999999999999,"15min":15.574999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":1.129,"15min":0.68700000000000006},"min":{"1min":0.318,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.51900000000000002,"5min":27.82,"15min":27.82},"last":0.35899999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:06:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":0.89800000000000002,"15min":0.67200000000000004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":9.6430000000000007,"5min":9.6430000000000007,"15min":9.6430000000000007},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.333,"5min":1.1659999999999999,"15min":0.76300000000000001},"min":{"1min":0.36399999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":11.119,"5min":27.707000000000001,"15min":27.707000000000001},"last":0.46200000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.121,"15min":0.76500000000000001},"min":{"1min":0.35499999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.80600000000000005,"5min":27.529,"15min":27.529},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.91700000000000004,"15min":0.71699999999999997},"min":{"1min":0.41099999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.84199999999999997,"5min":9.6799999999999997,"15min":9.6799999999999997},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.88100000000000001,"15min":0.68700000000000006},"min":{"1min":0.45700000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78800000000000003,"5min":9.6989999999999998,"15min":9.6989999999999998},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.629,"5min":1.1359999999999999,"15min":0.81200000000000006},"min":{"1min":0.38900000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.874,"5min":27.733000000000001,"15min":27.733000000000001},"last":0.40699999999999997}]},{"osd":7,"last update":"Mon Feb 19 21:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.179,"15min":0.78400000000000003},"min":{"1min":0.39300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85199999999999998,"5min":27.576000000000001,"15min":27.576000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.238,"15min":0.81399999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.73299999999999998,"5min":27.756,"15min":27.756},"last":0.52000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738475,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309272,"kb_used_data":3844,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435880,"statfs":{"total":95995035648,"available":95678341120,"internally_reserved":0,"allocated":3936256,"data_stored":2195489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.27600000000000002,"5min":1.093,"15min":0.58099999999999996},"min":{"1min":0.22,"5min":0.188,"15min":0.16200000000000001},"max":{"1min":0.38300000000000001,"5min":20.966999999999999,"15min":20.966999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.29399999999999998,"5min":1.1160000000000001,"15min":0.57899999999999996},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.188},"max":{"1min":0.41799999999999998,"5min":20.366,"15min":20.366},"last":0.248}]},{"osd":2,"last update":"Mon Feb 19 21:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.3200000000000001,"15min":0.68899999999999995},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":4.0890000000000004,"5min":27.916,"15min":27.916},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":1.083,"15min":0.61199999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.59599999999999997,"5min":22.004000000000001,"15min":22.004000000000001},"last":0.32100000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:06:45 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":1.2989999999999999,"15min":0.74199999999999999},"min":{"1min":0.248,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.54000000000000004,"5min":25.669,"15min":25.669},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":1.1040000000000001,"15min":0.68400000000000005},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.50700000000000001,"5min":25.006,"15min":25.006},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":1.0980000000000001,"15min":0.71099999999999997},"min":{"1min":0.39300000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71999999999999997,"5min":20.411999999999999,"15min":20.411999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":1.103,"15min":0.73199999999999998},"min":{"1min":0.35099999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.2509999999999999,"5min":19.986999999999998,"15min":19.986999999999998},"last":0.51900000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":1.202,"15min":0.83199999999999996},"min":{"1min":0.36299999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.83399999999999996,"5min":20.021999999999998,"15min":20.021999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.56699999999999995,"5min":1.1100000000000001,"15min":0.93600000000000005},"min":{"1min":0.40500000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.86199999999999999,"5min":20.908999999999999,"15min":20.908999999999999},"last":0.60699999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:06:49 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":1.085,"15min":0.76800000000000002},"min":{"1min":0.41999999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.3279999999999998,"5min":20.445,"15min":20.445},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":1.1160000000000001,"15min":0.81399999999999995},"min":{"1min":0.376,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":2.2389999999999999,"5min":20.047999999999998,"15min":20.047999999999998},"last":0.54300000000000004}]},{"osd":7,"last update":"Mon Feb 19 21:06:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":1.1519999999999999,"15min":0.92600000000000005},"min":{"1min":0.373,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":4.1479999999999997,"5min":20.940999999999999,"15min":20.940999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":1.2250000000000001,"15min":0.83099999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":3.5030000000000001,"5min":20.471,"15min":20.471},"last":0.45800000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411424,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305228,"kb_used_data":3904,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93439924,"statfs":{"total":95995035648,"available":95682482176,"internally_reserved":0,"allocated":3997696,"data_stored":2207269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:06:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.873,"15min":0.52900000000000003},"min":{"1min":0.23699999999999999,"5min":0.22900000000000001,"15min":0.222},"max":{"1min":4.665,"5min":26.690999999999999,"15min":26.690999999999999},"last":0.315},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.96399999999999997,"15min":0.55400000000000005},"min":{"1min":0.254,"5min":0.214,"15min":0.214},"max":{"1min":5.2279999999999998,"5min":24.315000000000001,"15min":24.315000000000001},"last":0.28599999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:06:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.327,"15min":0.67900000000000005},"min":{"1min":0.24199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":28.882000000000001,"5min":28.882000000000001,"15min":28.882000000000001},"last":0.248},{"interface":"front","average":{"1min":1.7370000000000001,"5min":1.228,"15min":0.67500000000000004},"min":{"1min":0.22900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":28.382999999999999,"5min":28.382999999999999,"15min":28.382999999999999},"last":0.38}]},{"osd":3,"last update":"Mon Feb 19 21:06:48 2024","interfaces":[{"interface":"back","average":{"1min":1.8069999999999999,"5min":1.087,"15min":0.67100000000000004},"min":{"1min":0.24199999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":28.742000000000001,"5min":28.742000000000001,"15min":28.742000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.97899999999999998,"15min":0.64600000000000002},"min":{"1min":0.28599999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60399999999999998,"5min":26.719999999999999,"15min":26.719999999999999},"last":0.34799999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:06:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":0.89100000000000001,"15min":0.70299999999999996},"min":{"1min":0.34000000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":24.356000000000002,"5min":24.356000000000002,"15min":24.356000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.6240000000000001,"5min":0.85199999999999998,"15min":0.72399999999999998},"min":{"1min":0.38600000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":20.707000000000001,"5min":20.707000000000001,"15min":20.707000000000001},"last":0.376}]},{"osd":5,"last update":"Mon Feb 19 21:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.754,"15min":0.69399999999999995},"min":{"1min":0.38100000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":9.9499999999999993,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.86399999999999999,"5min":1.1060000000000001,"15min":0.80800000000000005},"min":{"1min":0.40899999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":4.9359999999999999,"5min":26.606000000000002,"15min":26.606000000000002},"last":0.41099999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:07:03 2024","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.1419999999999999,"15min":0.80300000000000005},"min":{"1min":0.44900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":20.617000000000001,"5min":24.381,"15min":24.381},"last":0.45400000000000001},{"interface":"front","average":{"1min":2.077,"5min":0.98699999999999999,"15min":0.754},"min":{"1min":0.42099999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":28.706,"5min":28.706,"15min":28.706},"last":0.46899999999999997}]},{"osd":7,"last update":"Mon Feb 19 21:06:22 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.1160000000000001,"15min":0.76300000000000001},"min":{"1min":0.41799999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":4.9139999999999997,"5min":26.658000000000001,"15min":26.658000000000001},"last":0.438},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.98999999999999999,"15min":0.73399999999999999},"min":{"1min":0.40100000000000002,"5min":0.379,"15min":0.379},"max":{"1min":4.258,"5min":24.402000000000001,"15min":24.402000000000001},"last":0.504}]}]},{"osd":3,"up_from":25,"seq":107374182492,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305228,"kb_used_data":3904,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93439924,"statfs":{"total":95995035648,"available":95682482176,"internally_reserved":0,"allocated":3997696,"data_stored":2206881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":1.4510000000000001,"15min":0.72199999999999998},"min":{"1min":0.19400000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.51600000000000001,"5min":77.662000000000006,"15min":77.662000000000006},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.74399999999999999,"15min":0.51900000000000002},"min":{"1min":0.22600000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.72999999999999998,"5min":17.456,"15min":17.456},"last":0.313}]},{"osd":1,"last update":"Mon Feb 19 21:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.78000000000000003,"15min":0.52700000000000002},"min":{"1min":0.248,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.55300000000000005,"5min":17.327999999999999,"15min":17.327999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.69899999999999995,"15min":0.502},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":4.0579999999999998,"5min":15.004,"15min":15.004},"last":0.29999999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:06:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.2,"15min":0.68500000000000005},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.9319999999999999,"5min":56.890999999999998,"15min":56.890999999999998},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.34300000000000003,"5min":1.1499999999999999,"15min":0.66600000000000004},"min":{"1min":0.253,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.46800000000000003,"5min":44.600999999999999,"15min":44.600999999999999},"last":0.45500000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:06:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":0.79400000000000004,"15min":0.629},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":12.847,"5min":12.847,"15min":12.847},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":1.0289999999999999,"15min":0.72299999999999998},"min":{"1min":0.372,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":3.294,"5min":26.393999999999998,"15min":26.393999999999998},"last":0.53000000000000003}]},{"osd":5,"last update":"Mon Feb 19 21:06:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.85399999999999998,"15min":0.67300000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.68400000000000005,"5min":17.401,"15min":17.401},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.85499999999999998,"15min":0.66100000000000003},"min":{"1min":0.36899999999999999,"5min":0.36199999999999999,"15min":0.33800000000000002},"max":{"1min":4.0890000000000004,"5min":12.882,"15min":12.882},"last":0.60799999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:07:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.82799999999999996,"15min":0.65700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.34300000000000003},"max":{"1min":0.65000000000000002,"5min":17.367000000000001,"15min":17.367000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.85099999999999998,"15min":0.68100000000000005},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":4.024,"5min":12.904,"15min":12.904},"last":0.58499999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:06:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.84299999999999997,"15min":0.65500000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.80200000000000005,"5min":17.428999999999998,"15min":17.428999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":1.252,"15min":0.79900000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73299999999999998,"5min":44.378,"15min":44.378},"last":0.59299999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986264,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309364,"kb_used_data":3936,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435788,"statfs":{"total":95995035648,"available":95678246912,"internally_reserved":0,"allocated":4030464,"data_stored":2207032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:06:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":0.83599999999999997,"15min":0.65200000000000002},"min":{"1min":0.26300000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":19.878,"5min":19.878,"15min":19.878},"last":0.45600000000000002},{"interface":"front","average":{"1min":2.6070000000000002,"5min":1.575,"15min":0.879},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":17.675000000000001,"5min":70.063000000000002,"15min":70.063000000000002},"last":0.35599999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:06:10 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":0.92000000000000004,"15min":0.66800000000000004},"min":{"1min":0.26300000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":9.1920000000000002,"5min":26.510999999999999,"15min":26.510999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":2.6749999999999998,"5min":1.052,"15min":0.70199999999999996},"min":{"1min":0.28100000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":25.949000000000002,"5min":25.949000000000002,"15min":25.949000000000002},"last":0.373}]},{"osd":2,"last update":"Mon Feb 19 21:06:10 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.0629999999999999,"15min":0.70699999999999996},"min":{"1min":0.30299999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":7.516,"5min":37.320999999999998,"15min":37.320999999999998},"last":0.436},{"interface":"front","average":{"1min":4.9420000000000002,"5min":2.1219999999999999,"15min":1.0860000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":69.936999999999998,"5min":69.936999999999998,"15min":69.936999999999998},"last":0.41699999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:06:10 2024","interfaces":[{"interface":"back","average":{"1min":4.9660000000000002,"5min":1.681,"15min":0.91300000000000003},"min":{"1min":0.29399999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":69.793000000000006,"5min":69.793000000000006,"15min":69.793000000000006},"last":0.39000000000000001},{"interface":"front","average":{"1min":2.0110000000000001,"5min":1.214,"15min":0.78600000000000003},"min":{"1min":0.29499999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":17.986999999999998,"5min":17.986999999999998,"15min":17.986999999999998},"last":0.40000000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.42099999999999999,"15min":0.42899999999999999},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.218},"max":{"1min":0.52200000000000002,"5min":3.2799999999999998,"15min":3.2799999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.47599999999999998,"15min":0.44700000000000001},"min":{"1min":0.28799999999999998,"5min":0.26600000000000001,"15min":0.23899999999999999},"max":{"1min":0.55100000000000005,"5min":4.4340000000000002,"15min":4.4340000000000002},"last":0.36299999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:06:58 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.47299999999999998,"15min":0.40100000000000002},"min":{"1min":0.27900000000000003,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.623,"5min":3.2330000000000001,"15min":3.2330000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.53600000000000003,"15min":0.45800000000000002},"min":{"1min":0.35999999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.66900000000000004,"5min":4.5609999999999999,"15min":4.5609999999999999},"last":0.44600000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:06:28 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.52900000000000003,"15min":0.46100000000000002},"min":{"1min":0.33300000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.71899999999999997,"5min":6.0730000000000004,"15min":6.0730000000000004},"last":0.433},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.52200000000000002,"15min":0.47699999999999998},"min":{"1min":0.375,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.624,"5min":3.202,"15min":3.202},"last":0.47299999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790035,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313492,"kb_used_data":3972,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431660,"statfs":{"total":95995035648,"available":95674019840,"internally_reserved":0,"allocated":4067328,"data_stored":2207502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":4.6669999999999998,"15min":1.901},"min":{"1min":0.252,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.51900000000000002,"5min":276.47899999999998,"15min":276.47899999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":1.681,"15min":0.92500000000000004},"min":{"1min":0.24299999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.95399999999999996,"5min":52.887,"15min":52.887},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":2.7799999999999998,"15min":1.278},"min":{"1min":0.28599999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63100000000000001,"5min":120.011,"15min":120.011},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":2.0310000000000001,"15min":1.0349999999999999},"min":{"1min":0.33800000000000002,"5min":0.307,"15min":0.307},"max":{"1min":1.7470000000000001,"5min":118.105,"15min":118.105},"last":0.41499999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":1.9390000000000001,"15min":1.0089999999999999},"min":{"1min":0.34399999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.02,"5min":82.242000000000004,"15min":82.242000000000004},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.48199999999999998,"5min":3.4180000000000001,"15min":1.498},"min":{"1min":0.34499999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85299999999999998,"5min":232.84999999999999,"15min":232.84999999999999},"last":0.49199999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":2.1150000000000002,"15min":1.0640000000000001},"min":{"1min":0.33900000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.69999999999999996,"5min":109.70099999999999,"15min":109.70099999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":1.637,"15min":0.91500000000000004},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80800000000000005,"5min":65.519000000000005,"15min":65.519000000000005},"last":0.54200000000000004}]},{"osd":4,"last update":"Mon Feb 19 21:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.438,"15min":0.40200000000000002},"min":{"1min":0.254,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.861,"5min":1.861,"15min":1.861},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.46400000000000002,"15min":0.40699999999999997},"min":{"1min":0.28199999999999997,"5min":0.248,"15min":0.248},"max":{"1min":1.9730000000000001,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":0.505}]},{"osd":6,"last update":"Mon Feb 19 21:06:52 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.46600000000000003,"15min":0.43099999999999999},"min":{"1min":0.312,"5min":0.26900000000000002,"15min":0.26400000000000001},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.46999999999999997,"15min":0.442},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68999999999999995,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.58699999999999997}]},{"osd":7,"last update":"Mon Feb 19 21:06:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.504,"15min":0.502},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.8879999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.503,"15min":0.503},"min":{"1min":0.33000000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.55600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":164,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:07:10.402 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-02-19T21:07:10.402 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool set .rgw.root pg_num 32 2024-02-19T21:07:10.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:10.403 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:10.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:10.505 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:10.534 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:07:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:07:10.534 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:09.098011+0000 mon.a (mon.0) 1728 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:10.534 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:09.141382+0000 mgr.y (mgr.14550) 264 : audit [DBG] from='client.49352 v1:172.21.15.163:0/2668817751' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:07:10.534 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:09.159761+0000 mgr.y (mgr.14550) 265 : cluster [DBG] pgmap v344: 257 pgs: 8 unknown, 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:07:10.534 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:09.196045+0000 mon.a (mon.0) 1729 : audit [INF] from='client.49346 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:10.534 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:09.196133+0000 mon.a (mon.0) 1730 : audit [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pg_num","val":"11"}]': finished 2024-02-19T21:07:10.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:09.196194+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-02-19T21:07:10.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:09.198868+0000 mon.c (mon.2) 296 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:10.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:09.200452+0000 mon.c (mon.2) 297 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:10.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:09.200746+0000 mon.c (mon.2) 298 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi163-83845-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:10.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:09.203437+0000 mon.c (mon.2) 299 : audit [INF] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-19T21:07:10.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:09.209728+0000 mon.a (mon.0) 1732 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi163-83845-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:10.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:09.210159+0000 mon.a (mon.0) 1733 : audit [INF] from='client.34832 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]: dispatch 2024-02-19T21:07:10.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:09.231796+0000 mon.b (mon.1) 385 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:10.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:09.232156+0000 mon.a (mon.0) 1734 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:10.536 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:10.557 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:10.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:10.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:10.661 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:10.689 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:10.709 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:10.711 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:10.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:10.813 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:10.841 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:10.861 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:10.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:10.965 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:07:10.993 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:11.012 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:07:11.014 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:11.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:07:11.115 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:11.143 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:11.153 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:07:11.473 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:10.203264+0000 mgr.y (mgr.14550) 266 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:11.473 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:10.452602+0000 mon.a (mon.0) 1735 : audit [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]': finished 2024-02-19T21:07:11.473 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:10.452723+0000 mon.a (mon.0) 1736 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:11.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:10.452794+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-02-19T21:07:11.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:10.453352+0000 mon.c (mon.2) 300 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:11.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:10.453658+0000 mon.b (mon.1) 386 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27"}]: dispatch 2024-02-19T21:07:11.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:10.453954+0000 mon.b (mon.1) 387 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:11.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:10.453974+0000 mon.a (mon.0) 1738 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27"}]: dispatch 2024-02-19T21:07:11.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:10.454324+0000 mon.a (mon.0) 1739 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:11.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:10.203264+0000 mgr.y (mgr.14550) 266 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:11.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:10.452602+0000 mon.a (mon.0) 1735 : audit [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]': finished 2024-02-19T21:07:11.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:10.452723+0000 mon.a (mon.0) 1736 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:11.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:10.452794+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-02-19T21:07:11.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:10.453352+0000 mon.c (mon.2) 300 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:11.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:10.453658+0000 mon.b (mon.1) 386 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27"}]: dispatch 2024-02-19T21:07:11.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:10.453954+0000 mon.b (mon.1) 387 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:11.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:10.453974+0000 mon.a (mon.0) 1738 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27"}]: dispatch 2024-02-19T21:07:11.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:10.454324+0000 mon.a (mon.0) 1739 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:11.499 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-19T21:07:11.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:10.203264+0000 mgr.y (mgr.14550) 266 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:11.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:10.452602+0000 mon.a (mon.0) 1735 : audit [INF] from='client.34832 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi163-83888-1","var":"pgp_num","val":"11"}]': finished 2024-02-19T21:07:11.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:10.452723+0000 mon.a (mon.0) 1736 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:11.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:10.452794+0000 mon.a (mon.0) 1737 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-02-19T21:07:11.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:10.453352+0000 mon.c (mon.2) 300 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:11.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:10.453658+0000 mon.b (mon.1) 386 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27"}]: dispatch 2024-02-19T21:07:11.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:10.453954+0000 mon.b (mon.1) 387 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:11.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:10.453974+0000 mon.a (mon.0) 1738 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27"}]: dispatch 2024-02-19T21:07:11.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:10.454324+0000 mon.a (mon.0) 1739 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:11.592 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:11.595 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:11.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:11.696 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:11.723 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:11.871 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:11.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:11.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:11.974 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:12.006 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:12.027 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:12.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:12.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:12.131 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:12.161 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:12.181 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:12.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:12.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:12.285 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:12.314 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:12.333 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:12.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:12.409 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:12.413 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:12.436 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:12.464 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:12.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.480+0000 7fb1fe350700 1 -- 172.21.15.163:0/1948963317 >> v1:172.21.15.163:6790/0 conn(0x7fb1f810a8a0 legacy=0x7fb1f810cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:12.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.480+0000 7fb1fe350700 1 -- 172.21.15.163:0/1948963317 shutdown_connections 2024-02-19T21:07:12.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.480+0000 7fb1fe350700 1 -- 172.21.15.163:0/1948963317 >> 172.21.15.163:0/1948963317 conn(0x7fb1f806d7e0 msgr2=0x7fb1f806fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:12.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.480+0000 7fb1fe350700 1 -- 172.21.15.163:0/1948963317 shutdown_connections 2024-02-19T21:07:12.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.480+0000 7fb1fe350700 1 -- 172.21.15.163:0/1948963317 wait complete. 2024-02-19T21:07:12.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.480+0000 7fb1fe350700 1 librados: starting msgr at 2024-02-19T21:07:12.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.480+0000 7fb1fe350700 1 librados: starting objecter 2024-02-19T21:07:12.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.481+0000 7fb1fe350700 1 Processor -- start 2024-02-19T21:07:12.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.481+0000 7fb1fe350700 1 -- start start 2024-02-19T21:07:12.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.481+0000 7fb1fe350700 1 librados: setting wanted keys 2024-02-19T21:07:12.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.481+0000 7fb1fe350700 1 librados: calling monclient init 2024-02-19T21:07:12.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.481+0000 7fb1fe350700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1f811d890 con 0x7fb1f810a8a0 2024-02-19T21:07:12.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.481+0000 7fb1fe350700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1f811ea70 con 0x7fb1f8103f40 2024-02-19T21:07:12.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.481+0000 7fb1fe350700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1f811fc70 con 0x7fb1f8106d90 2024-02-19T21:07:12.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.481+0000 7fb1f77fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fb1f8106d90 0x7fb1f81006e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45938/0 (socket says 172.21.15.163:45938) 2024-02-19T21:07:12.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.481+0000 7fb1f77fe700 1 -- 172.21.15.163:0/3290014652 learned_addr learned my addr 172.21.15.163:0/3290014652 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:12.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.481+0000 7fb1f57fa700 10 client.?.objecter ms_handle_connect 0x7fb1f8106d90 2024-02-19T21:07:12.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.481+0000 7fb1f57fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:12.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.481+0000 7fb1f57fa700 10 client.?.objecter ms_handle_connect 0x7fb1f810a8a0 2024-02-19T21:07:12.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.481+0000 7fb1f57fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:12.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.481+0000 7fb1f57fa700 10 client.?.objecter ms_handle_connect 0x7fb1f8103f40 2024-02-19T21:07:12.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.481+0000 7fb1f57fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:12.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640954730 0 0) 0x7fb1f811d890 con 0x7fb1f810a8a0 2024-02-19T21:07:12.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e0002220 con 0x7fb1f810a8a0 2024-02-19T21:07:12.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932952631 0 0) 0x7fb1f811fc70 con 0x7fb1f8106d90 2024-02-19T21:07:12.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f811d890 con 0x7fb1f8106d90 2024-02-19T21:07:12.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420554747 0 0) 0x7fb1f811ea70 con 0x7fb1f8103f40 2024-02-19T21:07:12.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f811fc70 con 0x7fb1f8103f40 2024-02-19T21:07:12.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1873208566 0 0) 0x7fb1f811d890 con 0x7fb1f8106d90 2024-02-19T21:07:12.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1f811ea70 con 0x7fb1f8106d90 2024-02-19T21:07:12.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3106016240 0 0) 0x7fb1f811fc70 con 0x7fb1f8103f40 2024-02-19T21:07:12.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1f811d890 con 0x7fb1f8103f40 2024-02-19T21:07:12.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb1e8002950 con 0x7fb1f8106d90 2024-02-19T21:07:12.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3983143556 0 0) 0x7fb1e0002220 con 0x7fb1f810a8a0 2024-02-19T21:07:12.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1f811fc70 con 0x7fb1f810a8a0 2024-02-19T21:07:12.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb1ec001620 con 0x7fb1f8103f40 2024-02-19T21:07:12.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb1f00032a0 con 0x7fb1f810a8a0 2024-02-19T21:07:12.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2461324455 0 0) 0x7fb1f811d890 con 0x7fb1f8103f40 2024-02-19T21:07:12.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 >> v1:172.21.15.163:6790/0 conn(0x7fb1f8106d90 legacy=0x7fb1f81006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:12.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 >> v1:172.21.15.163:6789/0 conn(0x7fb1f810a8a0 legacy=0x7fb1f811bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:12.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f8120e70 con 0x7fb1f8103f40 2024-02-19T21:07:12.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1fe350700 1 -- 172.21.15.163:0/3290014652 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f811da80 con 0x7fb1f8103f40 2024-02-19T21:07:12.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1fe350700 10 client.48175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:12.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1fe350700 1 -- 172.21.15.163:0/3290014652 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f811df70 con 0x7fb1f8103f40 2024-02-19T21:07:12.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1fe350700 1 librados: init done 2024-02-19T21:07:12.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb1ec002060 con 0x7fb1f8103f40 2024-02-19T21:07:12.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.482+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb1ec0030b0 con 0x7fb1f8103f40 2024-02-19T21:07:12.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.483+0000 7fb1fe350700 1 -- 172.21.15.163:0/3290014652 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e4004f20 con 0x7fb1f8103f40 2024-02-19T21:07:12.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.484+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fb1ec001bc0 con 0x7fb1f8103f40 2024-02-19T21:07:12.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.484+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(252..252 src has 43..252) v4 ==== 8855+0+0 (unknown 120445429 0 0) 0x7fb1ec0764d0 con 0x7fb1f8103f40 2024-02-19T21:07:12.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.484+0000 7fb1f57fa700 10 client.48175.objecter ms_dispatch 0x7fb1f8001550 osd_map(252..252 src has 43..252) v4 2024-02-19T21:07:12.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.484+0000 7fb1f57fa700 3 client.48175.objecter handle_osd_map got epochs [252,252] > 0 2024-02-19T21:07:12.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.484+0000 7fb1f57fa700 3 client.48175.objecter handle_osd_map decoding full epoch 252 2024-02-19T21:07:12.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.484+0000 7fb1f57fa700 20 client.48175.objecter dump_active .. 0 homeless 2024-02-19T21:07:12.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.484+0000 7fb1f57fa700 10 client.48175.objecter ms_handle_connect 0x7fb1e0063fc0 2024-02-19T21:07:12.484 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:12.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.490+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb1ec0495b0 con 0x7fb1f8103f40 2024-02-19T21:07:12.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:12.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:12.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:11.160141+0000 mgr.y (mgr.14550) 267 : cluster [DBG] pgmap v347: 249 pgs: 4 active+clean+snaptrim, 245 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-19T21:07:12.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:11.208332+0000 mon.a (mon.0) 1740 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:12.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:11.216913+0000 mon.a (mon.0) 1741 : audit [INF] from='client.49346 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi163-83845-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi163-83845-36"}]': finished 2024-02-19T21:07:12.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:11.217002+0000 mon.a (mon.0) 1742 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27"}]': finished 2024-02-19T21:07:12.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:11.217065+0000 mon.a (mon.0) 1743 : audit [INF] from='client.48151 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-45"}]': finished 2024-02-19T21:07:12.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:11.217121+0000 mon.a (mon.0) 1744 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-02-19T21:07:12.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:11.224523+0000 mon.b (mon.1) 388 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:12.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:11.228464+0000 mon.a (mon.0) 1745 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:12.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:11.454094+0000 mon.c (mon.2) 301 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:12.589 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:12.617 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:12.636 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:12.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:12.641 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:12.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:12.672+0000 7fb1fe350700 1 -- 172.21.15.163:0/3290014652 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fb1e40051d0 con 0x7fb1f8103f40 2024-02-19T21:07:12.739 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:07:12.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:11.160141+0000 mgr.y (mgr.14550) 267 : cluster [DBG] pgmap v347: 249 pgs: 4 active+clean+snaptrim, 245 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-19T21:07:12.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:11.208332+0000 mon.a (mon.0) 1740 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:12.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:11.216913+0000 mon.a (mon.0) 1741 : audit [INF] from='client.49346 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi163-83845-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi163-83845-36"}]': finished 2024-02-19T21:07:12.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:11.217002+0000 mon.a (mon.0) 1742 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27"}]': finished 2024-02-19T21:07:12.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:11.217065+0000 mon.a (mon.0) 1743 : audit [INF] from='client.48151 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-45"}]': finished 2024-02-19T21:07:12.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:11.217121+0000 mon.a (mon.0) 1744 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-02-19T21:07:12.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:11.224523+0000 mon.b (mon.1) 388 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:12.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:11.228464+0000 mon.a (mon.0) 1745 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:12.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:11.454094+0000 mon.c (mon.2) 301 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:11.160141+0000 mgr.y (mgr.14550) 267 : cluster [DBG] pgmap v347: 249 pgs: 4 active+clean+snaptrim, 245 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-19T21:07:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:11.208332+0000 mon.a (mon.0) 1740 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:11.216913+0000 mon.a (mon.0) 1741 : audit [INF] from='client.49346 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi163-83845-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi163-83845-36"}]': finished 2024-02-19T21:07:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:11.217002+0000 mon.a (mon.0) 1742 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-27"}]': finished 2024-02-19T21:07:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:11.217065+0000 mon.a (mon.0) 1743 : audit [INF] from='client.48151 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-45"}]': finished 2024-02-19T21:07:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:11.217121+0000 mon.a (mon.0) 1744 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-02-19T21:07:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:11.224523+0000 mon.b (mon.1) 388 : audit [INF] from='client.? v1:172.21.15.163:0/50407966' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:11.228464+0000 mon.a (mon.0) 1745 : audit [INF] from='client.48151 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-45"}]: dispatch 2024-02-19T21:07:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:11.454094+0000 mon.c (mon.2) 301 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:12.773 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:12.848 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:07:12.850 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:12.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:07:12.951 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:12.981 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:13.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:13.219+0000 7fb1f57fa700 1 -- 172.21.15.163:0/3290014652 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v253) v1 ==== 97+0+0 (unknown 1657059800 0 0) 0x7fb1ec0479c0 con 0x7fb1f8103f40 2024-02-19T21:07:13.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:13.222+0000 7fb1fe350700 10 librados: watch_flush enter 2024-02-19T21:07:13.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:13.223+0000 7fb1fe350700 10 librados: watch_flush exit 2024-02-19T21:07:13.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:13.223+0000 7fb1fe350700 20 client.48175.objecter shutdown clearing up homeless session... 2024-02-19T21:07:13.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:13.223+0000 7fb1fe350700 10 client.48175.objecter successfully canceled tick 2024-02-19T21:07:13.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:13.223+0000 7fb1fe350700 1 -- 172.21.15.163:0/3290014652 >> v1:172.21.15.163:6800/1552514681 conn(0x7fb1e0063fc0 legacy=0x7fb1e0066480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:13.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:13.223+0000 7fb1fe350700 1 -- 172.21.15.163:0/3290014652 >> v1:172.21.15.193:6789/0 conn(0x7fb1f8103f40 legacy=0x7fb1f80fffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:13.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:13.227+0000 7fb1fe350700 1 -- 172.21.15.163:0/3290014652 shutdown_connections 2024-02-19T21:07:13.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:13.227+0000 7fb1fe350700 1 -- 172.21.15.163:0/3290014652 >> 172.21.15.163:0/3290014652 conn(0x7fb1f806d7e0 msgr2=0x7fb1f81053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:13.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:13.230+0000 7fb1fe350700 1 -- 172.21.15.163:0/3290014652 shutdown_connections 2024-02-19T21:07:13.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:13.230+0000 7fb1fe350700 1 -- 172.21.15.163:0/3290014652 wait complete. 2024-02-19T21:07:13.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:13.230+0000 7fb1fe350700 1 librados: shutdown 2024-02-19T21:07:13.226 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:07:13.635 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:07:13] "GET /metrics HTTP/1.1" 200 192404 "" "Prometheus/2.33.4" 2024-02-19T21:07:13.636 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:12.217261+0000 mon.a (mon.0) 1746 : audit [INF] from='client.48151 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-45"}]': finished 2024-02-19T21:07:13.636 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:12.217353+0000 mon.a (mon.0) 1747 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-02-19T21:07:13.636 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:12.459156+0000 mon.c (mon.2) 302 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:13.636 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:12.496304+0000 mon.b (mon.1) 389 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.636 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:12.496582+0000 mon.a (mon.0) 1748 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.636 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:12.497588+0000 mon.b (mon.1) 390 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.636 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:12.497801+0000 mon.a (mon.0) 1749 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.636 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:12.498164+0000 mon.b (mon.1) 391 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:13.636 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:12.498375+0000 mon.a (mon.0) 1750 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:13.637 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:12.673926+0000 mon.b (mon.1) 392 : audit [INF] from='client.? v1:172.21.15.163:0/3290014652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:07:13.637 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:12.674646+0000 mon.a (mon.0) 1751 : audit [INF] from='client.48175 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:07:13.637 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:13.214321+0000 mon.a (mon.0) 1752 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:13.637 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:13.218161+0000 mon.a (mon.0) 1753 : audit [INF] from='client.48181 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:13.637 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:12.217261+0000 mon.a (mon.0) 1746 : audit [INF] from='client.48151 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-45"}]': finished 2024-02-19T21:07:13.637 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:12.217353+0000 mon.a (mon.0) 1747 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-02-19T21:07:13.637 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:12.459156+0000 mon.c (mon.2) 302 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:13.637 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:12.496304+0000 mon.b (mon.1) 389 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:12.496582+0000 mon.a (mon.0) 1748 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:12.497588+0000 mon.b (mon.1) 390 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:12.497801+0000 mon.a (mon.0) 1749 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:12.498164+0000 mon.b (mon.1) 391 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:13.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:12.498375+0000 mon.a (mon.0) 1750 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:13.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:12.673926+0000 mon.b (mon.1) 392 : audit [INF] from='client.? v1:172.21.15.163:0/3290014652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:07:13.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:12.674646+0000 mon.a (mon.0) 1751 : audit [INF] from='client.48175 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:07:13.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:13.214321+0000 mon.a (mon.0) 1752 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:13.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:13.218161+0000 mon.a (mon.0) 1753 : audit [INF] from='client.48181 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:13.638 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:13.218284+0000 mon.a (mon.0) 1754 : audit [INF] from='client.48175 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:07:13.638 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:13.218374+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-02-19T21:07:13.639 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:13.220578+0000 mon.c (mon.2) 303 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:13.639 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:13.221523+0000 mon.b (mon.1) 393 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:13.639 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:13.222325+0000 mon.a (mon.0) 1756 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:13.639 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:13.223191+0000 mon.a (mon.0) 1757 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:13.639 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:13.226332+0000 mon.b (mon.1) 394 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.639 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:13.226957+0000 mon.a (mon.0) 1758 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.639 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:13.462599+0000 mon.c (mon.2) 304 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:13.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:13.218284+0000 mon.a (mon.0) 1754 : audit [INF] from='client.48175 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:07:13.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:13.218374+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-02-19T21:07:13.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:13.220578+0000 mon.c (mon.2) 303 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:13.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:13.221523+0000 mon.b (mon.1) 393 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:13.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:13.222325+0000 mon.a (mon.0) 1756 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:13.640 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:13.223191+0000 mon.a (mon.0) 1757 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:13.640 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:13.226332+0000 mon.b (mon.1) 394 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.640 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:13.226957+0000 mon.a (mon.0) 1758 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.640 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:13.462599+0000 mon.c (mon.2) 304 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:13.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:12.217261+0000 mon.a (mon.0) 1746 : audit [INF] from='client.48151 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-45"}]': finished 2024-02-19T21:07:13.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:12.217353+0000 mon.a (mon.0) 1747 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-02-19T21:07:13.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:12.459156+0000 mon.c (mon.2) 302 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:13.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:12.496304+0000 mon.b (mon.1) 389 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:12.496582+0000 mon.a (mon.0) 1748 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:12.497588+0000 mon.b (mon.1) 390 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:12.497801+0000 mon.a (mon.0) 1749 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:12.498164+0000 mon.b (mon.1) 391 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:12.498375+0000 mon.a (mon.0) 1750 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:12.673926+0000 mon.b (mon.1) 392 : audit [INF] from='client.? v1:172.21.15.163:0/3290014652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:07:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:12.674646+0000 mon.a (mon.0) 1751 : audit [INF] from='client.48175 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:07:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:13.214321+0000 mon.a (mon.0) 1752 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:13.218161+0000 mon.a (mon.0) 1753 : audit [INF] from='client.48181 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi163-83854-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:13.218284+0000 mon.a (mon.0) 1754 : audit [INF] from='client.48175 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:07:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:13.218374+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-02-19T21:07:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:13.220578+0000 mon.c (mon.2) 303 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:13.221523+0000 mon.b (mon.1) 393 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:13.222325+0000 mon.a (mon.0) 1756 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:13.223191+0000 mon.a (mon.0) 1757 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:13.226332+0000 mon.b (mon.1) 394 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:13.226957+0000 mon.a (mon.0) 1758 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:13.462599+0000 mon.c (mon.2) 304 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:13.930 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:13.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:13.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:14.034 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:14.062 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:14.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:13.160551+0000 mgr.y (mgr.14550) 268 : cluster [DBG] pgmap v350: 225 pgs: 8 unknown, 4 active+clean+snaptrim, 213 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-19T21:07:14.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:14.197667+0000 mon.a (mon.0) 1759 : audit [INF] from='client.49346 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36"}]': finished 2024-02-19T21:07:14.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:14.197768+0000 mon.a (mon.0) 1760 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:14.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:14.198593+0000 mon.c (mon.2) 305 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:14.520 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:07:14.520 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:07:14.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:14.208093+0000 mon.b (mon.1) 395 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:14.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:14.444175+0000 mon.a (mon.0) 1761 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-02-19T21:07:14.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:14.444811+0000 mon.a (mon.0) 1762 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:14.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:14.445415+0000 mon.a (mon.0) 1763 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:14.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:14.463367+0000 mon.c (mon.2) 306 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:14.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:13.160551+0000 mgr.y (mgr.14550) 268 : cluster [DBG] pgmap v350: 225 pgs: 8 unknown, 4 active+clean+snaptrim, 213 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-19T21:07:14.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:14.197667+0000 mon.a (mon.0) 1759 : audit [INF] from='client.49346 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36"}]': finished 2024-02-19T21:07:14.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:14.197768+0000 mon.a (mon.0) 1760 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:14.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:14.198593+0000 mon.c (mon.2) 305 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:14.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:14.208093+0000 mon.b (mon.1) 395 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:14.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:14.444175+0000 mon.a (mon.0) 1761 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-02-19T21:07:14.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:14.444811+0000 mon.a (mon.0) 1762 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:14.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:14.445415+0000 mon.a (mon.0) 1763 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:14.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:14.463367+0000 mon.c (mon.2) 306 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:14.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:13.160551+0000 mgr.y (mgr.14550) 268 : cluster [DBG] pgmap v350: 225 pgs: 8 unknown, 4 active+clean+snaptrim, 213 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-19T21:07:14.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:14.197667+0000 mon.a (mon.0) 1759 : audit [INF] from='client.49346 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi163-83845-36"}]': finished 2024-02-19T21:07:14.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:14.197768+0000 mon.a (mon.0) 1760 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:14.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:14.198593+0000 mon.c (mon.2) 305 : audit [INF] from='client.? v1:172.21.15.163:0/1615049686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:14.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:14.208093+0000 mon.b (mon.1) 395 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:14.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:14.444175+0000 mon.a (mon.0) 1761 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-02-19T21:07:14.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:14.444811+0000 mon.a (mon.0) 1762 : audit [INF] from='client.49346 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi163-83845-36"}]: dispatch 2024-02-19T21:07:14.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:14.445415+0000 mon.a (mon.0) 1763 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:14.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:14.463367+0000 mon.c (mon.2) 306 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:15.413 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-19T21:07:15.512 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:15.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:15.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:15.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:15.653 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:15.674 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:15.677 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:15.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:15.778 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:15.806 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:15.161424+0000 mgr.y (mgr.14550) 269 : cluster [DBG] pgmap v353: 249 pgs: 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-19T21:07:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:15.198593+0000 mon.a (mon.0) 1764 : audit [INF] from='client.48181 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-46"}]': finished 2024-02-19T21:07:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:15.198680+0000 mon.a (mon.0) 1765 : audit [INF] from='client.49346 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi163-83845-36"}]': finished 2024-02-19T21:07:16.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:15.198736+0000 mon.a (mon.0) 1766 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:16.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:15.198796+0000 mon.a (mon.0) 1767 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-02-19T21:07:16.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:15.200969+0000 mon.b (mon.1) 396 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-29"}]: dispatch 2024-02-19T21:07:16.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:15.205763+0000 mon.a (mon.0) 1768 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-29"}]: dispatch 2024-02-19T21:07:16.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:15.221765+0000 mon.c (mon.2) 307 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:16.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:15.222302+0000 mon.a (mon.0) 1769 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:16.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:15.222916+0000 mon.c (mon.2) 308 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:16.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:15.223109+0000 mon.a (mon.0) 1770 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:16.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:15.223450+0000 mon.c (mon.2) 309 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi163-83845-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:16.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:15.223652+0000 mon.a (mon.0) 1771 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi163-83845-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:16.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:15.464107+0000 mon.c (mon.2) 310 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:15.161424+0000 mgr.y (mgr.14550) 269 : cluster [DBG] pgmap v353: 249 pgs: 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-19T21:07:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:15.198593+0000 mon.a (mon.0) 1764 : audit [INF] from='client.48181 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-46"}]': finished 2024-02-19T21:07:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:15.198680+0000 mon.a (mon.0) 1765 : audit [INF] from='client.49346 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi163-83845-36"}]': finished 2024-02-19T21:07:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:15.198736+0000 mon.a (mon.0) 1766 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:15.198796+0000 mon.a (mon.0) 1767 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-02-19T21:07:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:15.200969+0000 mon.b (mon.1) 396 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-29"}]: dispatch 2024-02-19T21:07:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:15.205763+0000 mon.a (mon.0) 1768 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-29"}]: dispatch 2024-02-19T21:07:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:15.221765+0000 mon.c (mon.2) 307 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:15.222302+0000 mon.a (mon.0) 1769 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:15.222916+0000 mon.c (mon.2) 308 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:15.223109+0000 mon.a (mon.0) 1770 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:15.223450+0000 mon.c (mon.2) 309 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi163-83845-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:15.223652+0000 mon.a (mon.0) 1771 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi163-83845-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:16.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:15.464107+0000 mon.c (mon.2) 310 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:16.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:15.161424+0000 mgr.y (mgr.14550) 269 : cluster [DBG] pgmap v353: 249 pgs: 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-19T21:07:16.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:15.198593+0000 mon.a (mon.0) 1764 : audit [INF] from='client.48181 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi163-83854-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi163-83854-46"}]': finished 2024-02-19T21:07:16.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:15.198680+0000 mon.a (mon.0) 1765 : audit [INF] from='client.49346 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi163-83845-36"}]': finished 2024-02-19T21:07:16.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:15.198736+0000 mon.a (mon.0) 1766 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:16.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:15.198796+0000 mon.a (mon.0) 1767 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-02-19T21:07:16.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:15.200969+0000 mon.b (mon.1) 396 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-29"}]: dispatch 2024-02-19T21:07:16.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:15.205763+0000 mon.a (mon.0) 1768 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-29"}]: dispatch 2024-02-19T21:07:16.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:15.221765+0000 mon.c (mon.2) 307 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:16.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:15.222302+0000 mon.a (mon.0) 1769 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:16.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:15.222916+0000 mon.c (mon.2) 308 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:16.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:15.223109+0000 mon.a (mon.0) 1770 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:16.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:15.223450+0000 mon.c (mon.2) 309 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi163-83845-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:16.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:15.223652+0000 mon.a (mon.0) 1771 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi163-83845-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:16.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:15.464107+0000 mon.c (mon.2) 310 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:17.209 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:16.200736+0000 mon.a (mon.0) 1772 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-29"}]': finished 2024-02-19T21:07:17.209 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:16.200840+0000 mon.a (mon.0) 1773 : audit [INF] from='client.49370 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi163-83845-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:17.209 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:16.200898+0000 mon.a (mon.0) 1774 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-02-19T21:07:17.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:16.209351+0000 mon.b (mon.1) 397 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-29", "mode": "writeback"}]: dispatch 2024-02-19T21:07:17.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:16.210182+0000 mon.c (mon.2) 311 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi163-83845-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:17.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:16.211117+0000 mon.a (mon.0) 1775 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-29", "mode": "writeback"}]: dispatch 2024-02-19T21:07:17.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:16.211368+0000 mon.a (mon.0) 1776 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi163-83845-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:17.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:16.464906+0000 mon.c (mon.2) 312 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:17.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:16.200736+0000 mon.a (mon.0) 1772 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-29"}]': finished 2024-02-19T21:07:17.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:16.200840+0000 mon.a (mon.0) 1773 : audit [INF] from='client.49370 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi163-83845-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:17.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:16.200898+0000 mon.a (mon.0) 1774 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-02-19T21:07:17.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:16.209351+0000 mon.b (mon.1) 397 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-29", "mode": "writeback"}]: dispatch 2024-02-19T21:07:17.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:16.210182+0000 mon.c (mon.2) 311 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi163-83845-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:17.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:16.211117+0000 mon.a (mon.0) 1775 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-29", "mode": "writeback"}]: dispatch 2024-02-19T21:07:17.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:16.211368+0000 mon.a (mon.0) 1776 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi163-83845-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:17.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:16.464906+0000 mon.c (mon.2) 312 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:17.566 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:16.200736+0000 mon.a (mon.0) 1772 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-29"}]': finished 2024-02-19T21:07:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:16.200840+0000 mon.a (mon.0) 1773 : audit [INF] from='client.49370 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi163-83845-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:16.200898+0000 mon.a (mon.0) 1774 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-02-19T21:07:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:16.209351+0000 mon.b (mon.1) 397 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-29", "mode": "writeback"}]: dispatch 2024-02-19T21:07:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:16.210182+0000 mon.c (mon.2) 311 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi163-83845-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:16.211117+0000 mon.a (mon.0) 1775 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-29", "mode": "writeback"}]: dispatch 2024-02-19T21:07:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:16.211368+0000 mon.a (mon.0) 1776 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi163-83845-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:16.464906+0000 mon.c (mon.2) 312 : audit [DBG] from='client.? v1:172.21.15.163:0/21182929' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-02-19T21:07:18.066 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:07:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:07:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:07:18.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.085+0000 7f76cf386700 1 -- 172.21.15.163:0/621933948 >> v1:172.21.15.163:6790/0 conn(0x7f76c8106530 legacy=0x7f76c8108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.085+0000 7f76cf386700 1 -- 172.21.15.163:0/621933948 shutdown_connections 2024-02-19T21:07:18.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.085+0000 7f76cf386700 1 -- 172.21.15.163:0/621933948 >> 172.21.15.163:0/621933948 conn(0x7f76c80f2470 msgr2=0x7f76c80f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:18.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.085+0000 7f76cf386700 1 -- 172.21.15.163:0/621933948 shutdown_connections 2024-02-19T21:07:18.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.085+0000 7f76cf386700 1 -- 172.21.15.163:0/621933948 wait complete. 2024-02-19T21:07:18.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.085+0000 7f76cf386700 1 librados: starting msgr at 2024-02-19T21:07:18.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.085+0000 7f76cf386700 1 librados: starting objecter 2024-02-19T21:07:18.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76cf386700 1 Processor -- start 2024-02-19T21:07:18.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76cf386700 1 -- start start 2024-02-19T21:07:18.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76cf386700 1 librados: setting wanted keys 2024-02-19T21:07:18.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76cf386700 1 librados: calling monclient init 2024-02-19T21:07:18.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76cf386700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76c8119590 con 0x7f76c80f7280 2024-02-19T21:07:18.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76cf386700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76c811a770 con 0x7f76c80f6790 2024-02-19T21:07:18.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76cf386700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76c811b970 con 0x7f76c8106530 2024-02-19T21:07:18.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76cc921700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f76c80f7280 0x7f76c8103220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:58530/0 (socket says 172.21.15.163:58530) 2024-02-19T21:07:18.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76cc921700 1 -- 172.21.15.163:0/1254333461 learned_addr learned my addr 172.21.15.163:0/1254333461 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:18.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76be7fc700 10 client.?.objecter ms_handle_connect 0x7f76c8106530 2024-02-19T21:07:18.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76be7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:18.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76be7fc700 10 client.?.objecter ms_handle_connect 0x7f76c80f7280 2024-02-19T21:07:18.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76be7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:18.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76be7fc700 10 client.?.objecter ms_handle_connect 0x7f76c80f6790 2024-02-19T21:07:18.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76be7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:18.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852270170 0 0) 0x7f76c8119590 con 0x7f76c80f7280 2024-02-19T21:07:18.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b40033f0 con 0x7f76c80f7280 2024-02-19T21:07:18.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568061182 0 0) 0x7f76c811b970 con 0x7f76c8106530 2024-02-19T21:07:18.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c8119590 con 0x7f76c8106530 2024-02-19T21:07:18.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149537276 0 0) 0x7f76c811a770 con 0x7f76c80f6790 2024-02-19T21:07:18.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.086+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c811b970 con 0x7f76c80f6790 2024-02-19T21:07:18.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3641067301 0 0) 0x7f76b40033f0 con 0x7f76c80f7280 2024-02-19T21:07:18.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76c811a770 con 0x7f76c80f7280 2024-02-19T21:07:18.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f76b80019d0 con 0x7f76c80f7280 2024-02-19T21:07:18.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2714125036 0 0) 0x7f76c8119590 con 0x7f76c8106530 2024-02-19T21:07:18.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76b40033f0 con 0x7f76c8106530 2024-02-19T21:07:18.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 778497393 0 0) 0x7f76c811b970 con 0x7f76c80f6790 2024-02-19T21:07:18.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76c8119590 con 0x7f76c80f6790 2024-02-19T21:07:18.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f76c0001360 con 0x7f76c8106530 2024-02-19T21:07:18.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f76c4001840 con 0x7f76c80f6790 2024-02-19T21:07:18.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1246830992 0 0) 0x7f76c811a770 con 0x7f76c80f7280 2024-02-19T21:07:18.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 >> v1:172.21.15.163:6790/0 conn(0x7f76c8106530 legacy=0x7f76c8117c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 >> v1:172.21.15.193:6789/0 conn(0x7f76c80f6790 legacy=0x7f76c8102b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c811cb70 con 0x7f76c80f7280 2024-02-19T21:07:18.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76cf386700 1 -- 172.21.15.163:0/1254333461 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c8119780 con 0x7f76c80f7280 2024-02-19T21:07:18.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76cf386700 10 client.22701.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:18.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76cf386700 1 -- 172.21.15.163:0/1254333461 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c8119bf0 con 0x7f76c80f7280 2024-02-19T21:07:18.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76cf386700 1 librados: init done 2024-02-19T21:07:18.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f76b8002450 con 0x7f76c80f7280 2024-02-19T21:07:18.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.087+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f76b8005b10 con 0x7f76c80f7280 2024-02-19T21:07:18.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.088+0000 7f76cf386700 1 -- 172.21.15.163:0/1254333461 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ac004f20 con 0x7f76c80f7280 2024-02-19T21:07:18.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.089+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f76b8001fb0 con 0x7f76c80f7280 2024-02-19T21:07:18.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.089+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(257..257 src has 43..257) v4 ==== 8822+0+0 (unknown 172581415 0 0) 0x7f76b8078f20 con 0x7f76c80f7280 2024-02-19T21:07:18.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.089+0000 7f76be7fc700 10 client.22701.objecter ms_dispatch 0x7f76c8001550 osd_map(257..257 src has 43..257) v4 2024-02-19T21:07:18.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.089+0000 7f76be7fc700 3 client.22701.objecter handle_osd_map got epochs [257,257] > 0 2024-02-19T21:07:18.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.089+0000 7f76be7fc700 3 client.22701.objecter handle_osd_map decoding full epoch 257 2024-02-19T21:07:18.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.089+0000 7f76be7fc700 20 client.22701.objecter dump_active .. 0 homeless 2024-02-19T21:07:18.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.089+0000 7f76be7fc700 10 client.22701.objecter ms_handle_connect 0x7f76b4063d70 2024-02-19T21:07:18.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.093+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f76b804c040 con 0x7f76c80f7280 2024-02-19T21:07:18.117 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:07:18.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.002+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/3171948274 >> v1:172.21.15.163:6790/0 conn(0x7f5b181042a0 legacy=0x7f5b18106730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.002+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/3171948274 shutdown_connections 2024-02-19T21:07:18.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.002+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/3171948274 >> 172.21.15.163:0/3171948274 conn(0x7f5b1806fdc0 msgr2=0x7f5b180f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:18.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.002+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/3171948274 shutdown_connections 2024-02-19T21:07:18.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.002+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/3171948274 wait complete. 2024-02-19T21:07:18.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.002+0000 7f5b1ea6d700 1 librados: starting msgr at 2024-02-19T21:07:18.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.002+0000 7f5b1ea6d700 1 librados: starting objecter 2024-02-19T21:07:18.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.002+0000 7f5b1ea6d700 1 Processor -- start 2024-02-19T21:07:18.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.002+0000 7f5b1ea6d700 1 -- start start 2024-02-19T21:07:18.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.002+0000 7f5b1ea6d700 1 librados: setting wanted keys 2024-02-19T21:07:18.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.002+0000 7f5b1ea6d700 1 librados: calling monclient init 2024-02-19T21:07:18.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b1ea6d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b18117270 con 0x7f5b18100790 2024-02-19T21:07:18.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b1ea6d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b18118470 con 0x7f5b181042a0 2024-02-19T21:07:18.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b1ea6d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b18119670 con 0x7f5b180fd940 2024-02-19T21:07:18.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b1c809700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f5b180fd940 0x7f5b180f9a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:58498/0 (socket says 172.21.15.163:58498) 2024-02-19T21:07:18.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b1c809700 1 -- 172.21.15.163:0/2684549969 learned_addr learned my addr 172.21.15.163:0/2684549969 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:18.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 10 client.?.objecter ms_handle_connect 0x7f5b18100790 2024-02-19T21:07:18.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:18.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 10 client.?.objecter ms_handle_connect 0x7f5b180fd940 2024-02-19T21:07:18.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:18.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475002602 0 0) 0x7f5b18117270 con 0x7f5b18100790 2024-02-19T21:07:18.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b08003030 con 0x7f5b18100790 2024-02-19T21:07:18.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518586351 0 0) 0x7f5b18119670 con 0x7f5b180fd940 2024-02-19T21:07:18.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b18117270 con 0x7f5b180fd940 2024-02-19T21:07:18.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 10 client.?.objecter ms_handle_connect 0x7f5b181042a0 2024-02-19T21:07:18.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:18.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3104923424 0 0) 0x7f5b18117270 con 0x7f5b180fd940 2024-02-19T21:07:18.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b18119670 con 0x7f5b180fd940 2024-02-19T21:07:18.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3762861100 0 0) 0x7f5b08003030 con 0x7f5b18100790 2024-02-19T21:07:18.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b18117270 con 0x7f5b18100790 2024-02-19T21:07:18.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486968164 0 0) 0x7f5b18118470 con 0x7f5b181042a0 2024-02-19T21:07:18.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b08003030 con 0x7f5b181042a0 2024-02-19T21:07:18.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5b0c001900 con 0x7f5b180fd940 2024-02-19T21:07:18.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5b040017e0 con 0x7f5b18100790 2024-02-19T21:07:18.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.003+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2499315159 0 0) 0x7f5b18119670 con 0x7f5b180fd940 2024-02-19T21:07:18.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.004+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 >> v1:172.21.15.193:6789/0 conn(0x7f5b181042a0 legacy=0x7f5b18115930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.004+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 >> v1:172.21.15.163:6789/0 conn(0x7f5b18100790 legacy=0x7f5b180fa130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.004+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1811a870 con 0x7f5b180fd940 2024-02-19T21:07:18.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.004+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b18119860 con 0x7f5b180fd940 2024-02-19T21:07:18.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.004+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f5b0c002390 con 0x7f5b180fd940 2024-02-19T21:07:18.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.004+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5b0c005a90 con 0x7f5b180fd940 2024-02-19T21:07:18.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.004+0000 7f5b1ea6d700 10 client.49382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:18.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.004+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b18119cd0 con 0x7f5b180fd940 2024-02-19T21:07:18.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.004+0000 7f5b1ea6d700 1 librados: init done 2024-02-19T21:07:18.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.004+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f5b1805f700 con 0x7f5b180fd940 2024-02-19T21:07:18.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.005+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f5b0c018a00 con 0x7f5b180fd940 2024-02-19T21:07:18.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.005+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(257..257 src has 43..257) v4 ==== 8822+0+0 (unknown 172581415 0 0) 0x7f5b0c0790e0 con 0x7f5b180fd940 2024-02-19T21:07:18.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.005+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 osd_map(257..257 src has 43..257) v4 2024-02-19T21:07:18.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.005+0000 7f5b15f76700 3 client.49382.objecter handle_osd_map got epochs [257,257] > 0 2024-02-19T21:07:18.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.005+0000 7f5b15f76700 3 client.49382.objecter handle_osd_map decoding full epoch 257 2024-02-19T21:07:18.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.005+0000 7f5b15f76700 20 client.49382.objecter dump_active .. 0 homeless 2024-02-19T21:07:18.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.005+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v257) v1 ==== 54+0+16 (unknown 3122470173 0 2420000192) 0x7f5b0c07b4f0 con 0x7f5b180fd940 2024-02-19T21:07:18.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.006+0000 7f5b15f76700 10 client.49382.objecter ms_handle_connect 0x7f5b08063e10 2024-02-19T21:07:18.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.006+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:18.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.006+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 1 2024-02-19T21:07:18.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.006+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18047540 osd=0 3 2024-02-19T21:07:18.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.006+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18047540 osd=0 3 2024-02-19T21:07:18.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.006+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18047540 osd=0 3 2024-02-19T21:07:18.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.006+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 1 2024-02-19T21:07:18.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.006+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b18047540 osd=0 3 2024-02-19T21:07:18.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.006+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 0 1 2024-02-19T21:07:18.178 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:07:18.179 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:07:18.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.006+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18047540 osd=0 4 2024-02-19T21:07:18.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.006+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 1 2024-02-19T21:07:18.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.006+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b18109ee0 con 0x7f5b18118600 2024-02-19T21:07:18.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.006+0000 7f5b15f76700 10 client.49382.objecter ms_handle_connect 0x7f5b18118600 2024-02-19T21:07:18.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.007+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b18109ee0 con 0x7f5b18118600 2024-02-19T21:07:18.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.007+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:07:18.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.007+0000 7f5b15f76700 10 client.49382.objecter _finish_command 1 = system:0 2024-02-19T21:07:18.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.007+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b18047540 osd=0 4 2024-02-19T21:07:18.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.007+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 0 1 2024-02-19T21:07:18.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.022+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:18.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.022+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 2 2024-02-19T21:07:18.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.022+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18047540 osd=0 3 2024-02-19T21:07:18.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.022+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18047540 osd=0 3 2024-02-19T21:07:18.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.022+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18047540 osd=0 3 2024-02-19T21:07:18.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.022+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 2 2024-02-19T21:07:18.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.022+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b18047540 osd=0 3 2024-02-19T21:07:18.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.022+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 0 2 2024-02-19T21:07:18.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.022+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18047540 osd=0 4 2024-02-19T21:07:18.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.022+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 2 2024-02-19T21:07:18.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.022+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b18119330 con 0x7f5b18118600 2024-02-19T21:07:18.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b18119330 con 0x7f5b18118600 2024-02-19T21:07:18.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:18.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b15f76700 10 client.49382.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b18047540 osd=0 4 2024-02-19T21:07:18.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 0 2 2024-02-19T21:07:18.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:18.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 3 2024-02-19T21:07:18.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b181194a0 osd=1 3 2024-02-19T21:07:18.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b181194a0 osd=1 3 2024-02-19T21:07:18.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b181194a0 osd=1 3 2024-02-19T21:07:18.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 3 2024-02-19T21:07:18.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b181194a0 osd=1 3 2024-02-19T21:07:18.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 1 3 2024-02-19T21:07:18.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b181194a0 osd=1 4 2024-02-19T21:07:18.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 3 2024-02-19T21:07:18.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.023+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b18059cd0 con 0x7f5b180598f0 2024-02-19T21:07:18.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.024+0000 7f5b15f76700 10 client.49382.objecter ms_handle_connect 0x7f5b180598f0 2024-02-19T21:07:18.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.024+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b18059cd0 con 0x7f5b180598f0 2024-02-19T21:07:18.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.024+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:07:18.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.024+0000 7f5b15f76700 10 client.49382.objecter _finish_command 3 = system:0 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.024+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b181194a0 osd=1 4 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.024+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 1 3 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.039+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.039+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 4 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.039+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b181194a0 osd=1 3 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.039+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b181194a0 osd=1 3 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.039+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b181194a0 osd=1 3 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.039+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 4 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.039+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b181194a0 osd=1 3 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.039+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 1 4 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.039+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b181194a0 osd=1 4 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.039+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 4 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.039+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b1805a080 con 0x7f5b180598f0 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b1805a080 con 0x7f5b180598f0 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:18.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b15f76700 10 client.49382.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b181194a0 osd=1 4 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 1 4 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 5 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18059360 osd=2 3 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18059360 osd=2 3 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18059360 osd=2 3 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 5 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b18059360 osd=2 3 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 2 5 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18059360 osd=2 4 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 5 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b180287b0 con 0x7f5b180594d0 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.041+0000 7f5b15f76700 10 client.49382.objecter ms_handle_connect 0x7f5b180594d0 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.042+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b180287b0 con 0x7f5b180594d0 2024-02-19T21:07:18.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.042+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:07:18.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.042+0000 7f5b15f76700 10 client.49382.objecter _finish_command 5 = system:0 2024-02-19T21:07:18.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.042+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b18059360 osd=2 4 2024-02-19T21:07:18.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.042+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 2 5 2024-02-19T21:07:18.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.056+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:18.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.056+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 6 2024-02-19T21:07:18.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.056+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18059360 osd=2 3 2024-02-19T21:07:18.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.056+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18059360 osd=2 3 2024-02-19T21:07:18.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.056+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18059360 osd=2 3 2024-02-19T21:07:18.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.056+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 6 2024-02-19T21:07:18.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.056+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b18059360 osd=2 3 2024-02-19T21:07:18.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.056+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 2 6 2024-02-19T21:07:18.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.056+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18059360 osd=2 4 2024-02-19T21:07:18.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.056+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 6 2024-02-19T21:07:18.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.056+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b18027d10 con 0x7f5b180594d0 2024-02-19T21:07:18.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b18027d10 con 0x7f5b180594d0 2024-02-19T21:07:18.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:18.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b15f76700 10 client.49382.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b18059360 osd=2 4 2024-02-19T21:07:18.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 2 6 2024-02-19T21:07:18.185 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:07:18.185 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:07:18.185 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:07:18.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:18.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 7 2024-02-19T21:07:18.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18028120 osd=3 3 2024-02-19T21:07:18.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18028120 osd=3 3 2024-02-19T21:07:18.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18028120 osd=3 3 2024-02-19T21:07:18.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 7 2024-02-19T21:07:18.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b18028120 osd=3 3 2024-02-19T21:07:18.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 3 7 2024-02-19T21:07:18.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18028120 osd=3 4 2024-02-19T21:07:18.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 7 2024-02-19T21:07:18.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.058+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b18127b40 con 0x7f5b181053c0 2024-02-19T21:07:18.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.059+0000 7f5b15f76700 10 client.49382.objecter ms_handle_connect 0x7f5b181053c0 2024-02-19T21:07:18.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.059+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b18127b40 con 0x7f5b181053c0 2024-02-19T21:07:18.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.059+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:07:18.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.059+0000 7f5b15f76700 10 client.49382.objecter _finish_command 7 = system:0 2024-02-19T21:07:18.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.059+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b18028120 osd=3 4 2024-02-19T21:07:18.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.059+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 3 7 2024-02-19T21:07:18.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.074+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:18.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.074+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 8 2024-02-19T21:07:18.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.074+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18028120 osd=3 3 2024-02-19T21:07:18.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.074+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18028120 osd=3 3 2024-02-19T21:07:18.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.074+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18028120 osd=3 3 2024-02-19T21:07:18.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.074+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 8 2024-02-19T21:07:18.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.074+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b18028120 osd=3 3 2024-02-19T21:07:18.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.074+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 3 8 2024-02-19T21:07:18.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.074+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18028120 osd=3 4 2024-02-19T21:07:18.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.074+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 8 2024-02-19T21:07:18.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.074+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b18104920 con 0x7f5b181053c0 2024-02-19T21:07:18.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b18104920 con 0x7f5b181053c0 2024-02-19T21:07:18.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:18.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b15f76700 10 client.49382.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b18028120 osd=3 4 2024-02-19T21:07:18.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 3 8 2024-02-19T21:07:18.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:18.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 9 2024-02-19T21:07:18.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18104ac0 osd=4 3 2024-02-19T21:07:18.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18104ac0 osd=4 3 2024-02-19T21:07:18.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18104ac0 osd=4 3 2024-02-19T21:07:18.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 9 2024-02-19T21:07:18.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b18104ac0 osd=4 3 2024-02-19T21:07:18.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 4 9 2024-02-19T21:07:18.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18104ac0 osd=4 4 2024-02-19T21:07:18.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 9 2024-02-19T21:07:18.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b18104c60 con 0x7f5b18123ea0 2024-02-19T21:07:18.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.076+0000 7f5b15f76700 10 client.49382.objecter ms_handle_connect 0x7f5b18123ea0 2024-02-19T21:07:18.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.077+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b18104c60 con 0x7f5b18123ea0 2024-02-19T21:07:18.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.077+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:07:18.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.077+0000 7f5b15f76700 10 client.49382.objecter _finish_command 9 = system:0 2024-02-19T21:07:18.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.077+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b18104ac0 osd=4 4 2024-02-19T21:07:18.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.077+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 4 9 2024-02-19T21:07:18.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.094+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:18.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.094+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 10 2024-02-19T21:07:18.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.094+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18104ac0 osd=4 3 2024-02-19T21:07:18.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.094+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18104ac0 osd=4 3 2024-02-19T21:07:18.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.094+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18104ac0 osd=4 3 2024-02-19T21:07:18.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.094+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 10 2024-02-19T21:07:18.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.094+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b18104ac0 osd=4 3 2024-02-19T21:07:18.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.094+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 4 10 2024-02-19T21:07:18.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.094+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18104ac0 osd=4 4 2024-02-19T21:07:18.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.094+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 10 2024-02-19T21:07:18.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.094+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b18104c60 con 0x7f5b18123ea0 2024-02-19T21:07:18.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b18104c60 con 0x7f5b18123ea0 2024-02-19T21:07:18.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:18.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b15f76700 10 client.49382.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b18104ac0 osd=4 4 2024-02-19T21:07:18.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 4 10 2024-02-19T21:07:18.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:18.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 11 2024-02-19T21:07:18.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18104c60 osd=5 3 2024-02-19T21:07:18.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18104c60 osd=5 3 2024-02-19T21:07:18.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18104c60 osd=5 3 2024-02-19T21:07:18.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 11 2024-02-19T21:07:18.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b18104c60 osd=5 3 2024-02-19T21:07:18.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 5 11 2024-02-19T21:07:18.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18104c60 osd=5 4 2024-02-19T21:07:18.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 11 2024-02-19T21:07:18.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.096+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b1812a020 con 0x7f5b18129a80 2024-02-19T21:07:18.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.097+0000 7f5b15f76700 10 client.49382.objecter ms_handle_connect 0x7f5b18129a80 2024-02-19T21:07:18.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.097+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b1812a020 con 0x7f5b18129a80 2024-02-19T21:07:18.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.097+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:07:18.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.097+0000 7f5b15f76700 10 client.49382.objecter _finish_command 11 = system:0 2024-02-19T21:07:18.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.097+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b18104c60 osd=5 4 2024-02-19T21:07:18.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.097+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 5 11 2024-02-19T21:07:18.193 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:07:18.193 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:07:18.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.112+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:18.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.112+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 12 2024-02-19T21:07:18.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.112+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18104c60 osd=5 3 2024-02-19T21:07:18.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.112+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18104c60 osd=5 3 2024-02-19T21:07:18.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.112+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18104c60 osd=5 3 2024-02-19T21:07:18.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.112+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 12 2024-02-19T21:07:18.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.112+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b18104c60 osd=5 3 2024-02-19T21:07:18.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.112+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 5 12 2024-02-19T21:07:18.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.112+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18104c60 osd=5 4 2024-02-19T21:07:18.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.112+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 12 2024-02-19T21:07:18.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.112+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b1812a020 con 0x7f5b18129a80 2024-02-19T21:07:18.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b1812a020 con 0x7f5b18129a80 2024-02-19T21:07:18.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:18.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b15f76700 10 client.49382.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b18104c60 osd=5 4 2024-02-19T21:07:18.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 5 12 2024-02-19T21:07:18.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:18.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 13 2024-02-19T21:07:18.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b1812a020 osd=6 3 2024-02-19T21:07:18.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b1812a020 osd=6 3 2024-02-19T21:07:18.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b1812a020 osd=6 3 2024-02-19T21:07:18.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 13 2024-02-19T21:07:18.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b1812a020 osd=6 3 2024-02-19T21:07:18.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 6 13 2024-02-19T21:07:18.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b1812a020 osd=6 4 2024-02-19T21:07:18.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 13 2024-02-19T21:07:18.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.114+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b18134c60 con 0x7f5b18131120 2024-02-19T21:07:18.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.115+0000 7f5b15f76700 10 client.49382.objecter ms_handle_connect 0x7f5b18131120 2024-02-19T21:07:18.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.116+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b18134c60 con 0x7f5b18131120 2024-02-19T21:07:18.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.116+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:07:18.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.116+0000 7f5b15f76700 10 client.49382.objecter _finish_command 13 = system:0 2024-02-19T21:07:18.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.116+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b1812a020 osd=6 4 2024-02-19T21:07:18.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.116+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 6 13 2024-02-19T21:07:18.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.132+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:18.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.132+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 14 2024-02-19T21:07:18.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.132+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b1812a020 osd=6 3 2024-02-19T21:07:18.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.132+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b1812a020 osd=6 3 2024-02-19T21:07:18.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.132+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b1812a020 osd=6 3 2024-02-19T21:07:18.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.132+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 14 2024-02-19T21:07:18.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.132+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b1812a020 osd=6 3 2024-02-19T21:07:18.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.132+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 6 14 2024-02-19T21:07:18.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.132+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b1812a020 osd=6 4 2024-02-19T21:07:18.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.132+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 14 2024-02-19T21:07:18.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.132+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b18136410 con 0x7f5b18131120 2024-02-19T21:07:18.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b18136410 con 0x7f5b18131120 2024-02-19T21:07:18.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:18.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b15f76700 10 client.49382.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b1812a020 osd=6 4 2024-02-19T21:07:18.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 6 14 2024-02-19T21:07:18.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:18.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 15 2024-02-19T21:07:18.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18136410 osd=7 3 2024-02-19T21:07:18.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18136410 osd=7 3 2024-02-19T21:07:18.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18136410 osd=7 3 2024-02-19T21:07:18.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 15 2024-02-19T21:07:18.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b18136410 osd=7 3 2024-02-19T21:07:18.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 7 15 2024-02-19T21:07:18.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18136410 osd=7 4 2024-02-19T21:07:18.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 15 2024-02-19T21:07:18.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b18137040 con 0x7f5b18136a90 2024-02-19T21:07:18.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.134+0000 7f5b15f76700 10 client.49382.objecter ms_handle_connect 0x7f5b18136a90 2024-02-19T21:07:18.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.135+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5b18137040 con 0x7f5b18136a90 2024-02-19T21:07:18.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.135+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:07:18.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.135+0000 7f5b15f76700 10 client.49382.objecter _finish_command 15 = system:0 2024-02-19T21:07:18.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.135+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b18136410 osd=7 4 2024-02-19T21:07:18.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.135+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 7 15 2024-02-19T21:07:18.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.151+0000 7f5b1ea6d700 10 client.49382.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:18.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.151+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign -1 16 2024-02-19T21:07:18.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.151+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18136410 osd=7 3 2024-02-19T21:07:18.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.151+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18136410 osd=7 3 2024-02-19T21:07:18.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.151+0000 7f5b1ea6d700 20 client.49382.objecter _get_session s=0x7f5b18136410 osd=7 3 2024-02-19T21:07:18.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.151+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_remove -1 16 2024-02-19T21:07:18.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.151+0000 7f5b1ea6d700 20 client.49382.objecter get_session s=0x7f5b18136410 osd=7 3 2024-02-19T21:07:18.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.151+0000 7f5b1ea6d700 15 client.49382.objecter _session_command_op_assign 7 16 2024-02-19T21:07:18.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.151+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18136410 osd=7 4 2024-02-19T21:07:18.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.151+0000 7f5b1ea6d700 10 client.49382.objecter _send_command 16 2024-02-19T21:07:18.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.151+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5b18137040 con 0x7f5b18136a90 2024-02-19T21:07:18.198 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:07:18.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b15f76700 1 -- 172.21.15.163:0/2684549969 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5b18137040 con 0x7f5b18136a90 2024-02-19T21:07:18.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b15f76700 10 client.49382.objecter ms_dispatch 0x7f5b18001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:18.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b15f76700 10 client.49382.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b15f76700 20 client.49382.objecter put_session s=0x7f5b18136410 osd=7 4 2024-02-19T21:07:18.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b15f76700 15 client.49382.objecter _session_command_op_remove 7 16 2024-02-19T21:07:18.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 10 librados: watch_flush enter 2024-02-19T21:07:18.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 10 librados: watch_flush exit 2024-02-19T21:07:18.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 10 client.49382.objecter close_session for osd.0 2024-02-19T21:07:18.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 >> v1:172.21.15.163:6801/1559418115 conn(0x7f5b18118600 legacy=0x7f5b181189e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18047540 osd=0 1 2024-02-19T21:07:18.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 10 client.49382.objecter close_session for osd.1 2024-02-19T21:07:18.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 >> v1:172.21.15.163:6805/3125047432 conn(0x7f5b180598f0 legacy=0x7f5b180273c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b181194a0 osd=1 1 2024-02-19T21:07:18.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 10 client.49382.objecter close_session for osd.2 2024-02-19T21:07:18.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 >> v1:172.21.15.163:6809/1563158487 conn(0x7f5b180594d0 legacy=0x7f5b18121aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18059360 osd=2 1 2024-02-19T21:07:18.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 10 client.49382.objecter close_session for osd.3 2024-02-19T21:07:18.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 >> v1:172.21.15.163:6813/2112279769 conn(0x7f5b181053c0 legacy=0x7f5b18126350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18028120 osd=3 1 2024-02-19T21:07:18.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 10 client.49382.objecter close_session for osd.4 2024-02-19T21:07:18.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 >> v1:172.21.15.193:6800/3129716965 conn(0x7f5b18123ea0 legacy=0x7f5b18127ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18104ac0 osd=4 1 2024-02-19T21:07:18.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 10 client.49382.objecter close_session for osd.5 2024-02-19T21:07:18.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 >> v1:172.21.15.193:6804/805161991 conn(0x7f5b18129a80 legacy=0x7f5b1812f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1d00a700 1 -- 172.21.15.163:0/2684549969 reap_dead start 2024-02-19T21:07:18.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1d00a700 1 -- 172.21.15.163:0/2684549969 reap_dead start 2024-02-19T21:07:18.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1d00a700 1 -- 172.21.15.163:0/2684549969 reap_dead start 2024-02-19T21:07:18.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18104c60 osd=5 1 2024-02-19T21:07:18.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 10 client.49382.objecter close_session for osd.6 2024-02-19T21:07:18.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 >> v1:172.21.15.193:6808/787604615 conn(0x7f5b18131120 legacy=0x7f5b18133510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b1812a020 osd=6 1 2024-02-19T21:07:18.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 10 client.49382.objecter close_session for osd.7 2024-02-19T21:07:18.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 >> v1:172.21.15.193:6812/157927432 conn(0x7f5b18136a90 legacy=0x7f5b18134c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 20 client.49382.objecter put_session s=0x7f5b18136410 osd=7 1 2024-02-19T21:07:18.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 20 client.49382.objecter shutdown clearing up homeless session... 2024-02-19T21:07:18.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 10 client.49382.objecter successfully canceled tick 2024-02-19T21:07:18.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5b08063e10 legacy=0x7f5b080662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 >> v1:172.21.15.163:6790/0 conn(0x7f5b180fd940 legacy=0x7f5b180f9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:18.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1d00a700 1 -- 172.21.15.163:0/2684549969 reap_dead start 2024-02-19T21:07:18.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 shutdown_connections 2024-02-19T21:07:18.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.153+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 >> 172.21.15.163:0/2684549969 conn(0x7f5b1806fdc0 msgr2=0x7f5b180f4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:18.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.154+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 shutdown_connections 2024-02-19T21:07:18.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.154+0000 7f5b1ea6d700 1 -- 172.21.15.163:0/2684549969 wait complete. 2024-02-19T21:07:18.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.154+0000 7f5b1ea6d700 1 librados: shutdown 2024-02-19T21:07:18.202 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.202 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.202 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.202 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.202 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.202 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.202 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.202 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:07:18.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:18.279+0000 7f76cf386700 1 -- 172.21.15.163:0/1254333461 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "32"} v 0) v1 -- 0x7f76ac005750 con 0x7f76c80f7280 2024-02-19T21:07:18.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:17.161837+0000 mgr.y (mgr.14550) 270 : cluster [DBG] pgmap v356: 257 pgs: 8 unknown, 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:07:18.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:17.206726+0000 mon.a (mon.0) 1777 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:18.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:17.216154+0000 mon.a (mon.0) 1778 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-29", "mode": "writeback"}]': finished 2024-02-19T21:07:18.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:17.306432+0000 mon.b (mon.1) 398 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:18.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:17.460779+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-02-19T21:07:18.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:17.461769+0000 mon.a (mon.0) 1780 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:18.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:17.462249+0000 mon.b (mon.1) 399 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:18.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:17.462525+0000 mon.a (mon.0) 1781 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:18.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:17.477382+0000 mon.a (mon.0) 1782 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi163-83888-2"}]: dispatch 2024-02-19T21:07:18.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:17.478634+0000 mon.a (mon.0) 1783 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi163-83888-2"}]: dispatch 2024-02-19T21:07:18.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:17.478923+0000 mon.a (mon.0) 1784 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi163-83888-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:18.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:18.006755+0000 mon.c (mon.2) 313 : audit [DBG] from='client.? v1:172.21.15.163:0/2684549969' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:18.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:17.161837+0000 mgr.y (mgr.14550) 270 : cluster [DBG] pgmap v356: 257 pgs: 8 unknown, 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:07:18.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:17.206726+0000 mon.a (mon.0) 1777 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:18.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:17.216154+0000 mon.a (mon.0) 1778 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-29", "mode": "writeback"}]': finished 2024-02-19T21:07:18.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:17.306432+0000 mon.b (mon.1) 398 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:18.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:17.460779+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-02-19T21:07:18.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:17.461769+0000 mon.a (mon.0) 1780 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:18.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:17.462249+0000 mon.b (mon.1) 399 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:18.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:17.462525+0000 mon.a (mon.0) 1781 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:18.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:17.477382+0000 mon.a (mon.0) 1782 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi163-83888-2"}]: dispatch 2024-02-19T21:07:18.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:17.478634+0000 mon.a (mon.0) 1783 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi163-83888-2"}]: dispatch 2024-02-19T21:07:18.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:17.478923+0000 mon.a (mon.0) 1784 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi163-83888-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:18.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:18.006755+0000 mon.c (mon.2) 313 : audit [DBG] from='client.? v1:172.21.15.163:0/2684549969' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:18.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:17.161837+0000 mgr.y (mgr.14550) 270 : cluster [DBG] pgmap v356: 257 pgs: 8 unknown, 249 active+clean; 8.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:07:18.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:17.206726+0000 mon.a (mon.0) 1777 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:18.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:17.216154+0000 mon.a (mon.0) 1778 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-29", "mode": "writeback"}]': finished 2024-02-19T21:07:18.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:17.306432+0000 mon.b (mon.1) 398 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:18.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:17.460779+0000 mon.a (mon.0) 1779 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-02-19T21:07:18.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:17.461769+0000 mon.a (mon.0) 1780 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:18.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:17.462249+0000 mon.b (mon.1) 399 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:18.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:17.462525+0000 mon.a (mon.0) 1781 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:18.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:17.477382+0000 mon.a (mon.0) 1782 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi163-83888-2"}]: dispatch 2024-02-19T21:07:18.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:17.478634+0000 mon.a (mon.0) 1783 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi163-83888-2"}]: dispatch 2024-02-19T21:07:18.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:17.478923+0000 mon.a (mon.0) 1784 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi163-83888-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:18.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:18.006755+0000 mon.c (mon.2) 313 : audit [DBG] from='client.? v1:172.21.15.163:0/2684549969' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:18.867 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:18.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:18.914 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:18.924 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:18.971 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:18.998 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:19.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:19.442+0000 7f76be7fc700 1 -- 172.21.15.163:0/1254333461 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "32"}]=0 set pool 2 pg_num to 32 v259) v1 ==== 134+0+0 (unknown 680239399 0 0) 0x7f76b804a450 con 0x7f76c80f7280 2024-02-19T21:07:19.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:19.445+0000 7f76cf386700 10 librados: watch_flush enter 2024-02-19T21:07:19.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:19.445+0000 7f76cf386700 10 librados: watch_flush exit 2024-02-19T21:07:19.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:19.445+0000 7f76cf386700 20 client.22701.objecter shutdown clearing up homeless session... 2024-02-19T21:07:19.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:19.445+0000 7f76cf386700 10 client.22701.objecter successfully canceled tick 2024-02-19T21:07:19.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:19.445+0000 7f76cf386700 1 -- 172.21.15.163:0/1254333461 >> v1:172.21.15.163:6800/1552514681 conn(0x7f76b4063d70 legacy=0x7f76b4066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:19.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:19.445+0000 7f76cf386700 1 -- 172.21.15.163:0/1254333461 >> v1:172.21.15.163:6789/0 conn(0x7f76c80f7280 legacy=0x7f76c8103220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:19.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:19.445+0000 7f76cf386700 1 -- 172.21.15.163:0/1254333461 shutdown_connections 2024-02-19T21:07:19.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:19.445+0000 7f76cf386700 1 -- 172.21.15.163:0/1254333461 >> 172.21.15.163:0/1254333461 conn(0x7f76c80f2470 msgr2=0x7f76c8109930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:19.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:19.445+0000 7f76cf386700 1 -- 172.21.15.163:0/1254333461 shutdown_connections 2024-02-19T21:07:19.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:19.445+0000 7f76cf386700 1 -- 172.21.15.163:0/1254333461 wait complete. 2024-02-19T21:07:19.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:19.445+0000 7f76cf386700 1 librados: shutdown 2024-02-19T21:07:19.443 INFO:teuthology.orchestra.run.smithi163.stderr:set pool 2 pg_num to 32 2024-02-19T21:07:19.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:18.225589+0000 mon.a (mon.0) 1785 : audit [INF] from='client.49370 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi163-83845-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi163-83845-37"}]': finished 2024-02-19T21:07:19.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:18.225709+0000 mon.a (mon.0) 1786 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:19.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:18.225783+0000 mon.a (mon.0) 1787 : audit [INF] from='client.48181 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-46"}]': finished 2024-02-19T21:07:19.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:18.225854+0000 mon.a (mon.0) 1788 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi163-83888-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:19.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:18.225908+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-02-19T21:07:19.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:18.229493+0000 mon.b (mon.1) 400 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:19.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:18.229878+0000 mon.a (mon.0) 1790 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:19.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:18.230531+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi163-83888-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi163-83888-2"}]: dispatch 2024-02-19T21:07:19.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:18.231300+0000 mon.b (mon.1) 401 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29"}]: dispatch 2024-02-19T21:07:19.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:18.231600+0000 mon.a (mon.0) 1792 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29"}]: dispatch 2024-02-19T21:07:19.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:18.281204+0000 mon.a (mon.0) 1793 : audit [INF] from='client.? v1:172.21.15.163:0/1254333461' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "32"}]: dispatch 2024-02-19T21:07:19.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:19.194822+0000 mon.a (mon.0) 1794 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:19.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:19.202443+0000 mon.a (mon.0) 1795 : audit [INF] from='client.48181 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-46"}]': finished 2024-02-19T21:07:19.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:18.225589+0000 mon.a (mon.0) 1785 : audit [INF] from='client.49370 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi163-83845-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi163-83845-37"}]': finished 2024-02-19T21:07:19.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:18.225709+0000 mon.a (mon.0) 1786 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:19.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:18.225783+0000 mon.a (mon.0) 1787 : audit [INF] from='client.48181 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-46"}]': finished 2024-02-19T21:07:19.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:18.225854+0000 mon.a (mon.0) 1788 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi163-83888-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:19.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:18.225908+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-02-19T21:07:19.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:18.229493+0000 mon.b (mon.1) 400 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:19.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:18.229878+0000 mon.a (mon.0) 1790 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:19.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:18.230531+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi163-83888-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi163-83888-2"}]: dispatch 2024-02-19T21:07:19.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:18.231300+0000 mon.b (mon.1) 401 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29"}]: dispatch 2024-02-19T21:07:19.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:18.231600+0000 mon.a (mon.0) 1792 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29"}]: dispatch 2024-02-19T21:07:19.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:18.281204+0000 mon.a (mon.0) 1793 : audit [INF] from='client.? v1:172.21.15.163:0/1254333461' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "32"}]: dispatch 2024-02-19T21:07:19.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:19.194822+0000 mon.a (mon.0) 1794 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:19.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:19.202443+0000 mon.a (mon.0) 1795 : audit [INF] from='client.48181 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-46"}]': finished 2024-02-19T21:07:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:18.225589+0000 mon.a (mon.0) 1785 : audit [INF] from='client.49370 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi163-83845-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi163-83845-37"}]': finished 2024-02-19T21:07:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:18.225709+0000 mon.a (mon.0) 1786 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:18.225783+0000 mon.a (mon.0) 1787 : audit [INF] from='client.48181 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi163-83854-46"}]': finished 2024-02-19T21:07:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:18.225854+0000 mon.a (mon.0) 1788 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi163-83888-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:18.225908+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-02-19T21:07:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:18.229493+0000 mon.b (mon.1) 400 : audit [INF] from='client.? v1:172.21.15.163:0/1670421628' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:18.229878+0000 mon.a (mon.0) 1790 : audit [INF] from='client.48181 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-46"}]: dispatch 2024-02-19T21:07:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:18.230531+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi163-83888-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi163-83888-2"}]: dispatch 2024-02-19T21:07:19.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:18.231300+0000 mon.b (mon.1) 401 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29"}]: dispatch 2024-02-19T21:07:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:18.231600+0000 mon.a (mon.0) 1792 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29"}]: dispatch 2024-02-19T21:07:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:18.281204+0000 mon.a (mon.0) 1793 : audit [INF] from='client.? v1:172.21.15.163:0/1254333461' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "32"}]: dispatch 2024-02-19T21:07:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:19.194822+0000 mon.a (mon.0) 1794 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:19.202443+0000 mon.a (mon.0) 1795 : audit [INF] from='client.48181 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi163-83854-46"}]': finished 2024-02-19T21:07:19.847 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:19.849 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:19.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:19.950 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:19.978 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:20.566 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:07:20.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:18.963162+0000 osd.7 (osd.7) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:20.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:18.963421+0000 osd.7 (osd.7) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:20.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:19.162216+0000 mgr.y (mgr.14550) 271 : cluster [DBG] pgmap v359: 246 pgs: 8 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:07:20.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:19.224162+0000 mon.c (mon.2) 314 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:19.443326+0000 mon.a (mon.0) 1796 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29"}]': finished 2024-02-19T21:07:20.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:19.443423+0000 mon.a (mon.0) 1797 : audit [INF] from='client.? v1:172.21.15.163:0/1254333461' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "32"}]': finished 2024-02-19T21:07:20.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:19.443474+0000 mon.a (mon.0) 1798 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-02-19T21:07:20.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:19.444767+0000 mon.a (mon.0) 1799 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:19.454123+0000 mon.c (mon.2) 315 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:19.462930+0000 mon.a (mon.0) 1800 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:19.463673+0000 mon.c (mon.2) 316 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi163-83854-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:20.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:19.463990+0000 mon.a (mon.0) 1801 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi163-83854-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:20.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:20.202270+0000 mon.a (mon.0) 1802 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi163-83888-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi163-83888-2"}]': finished 2024-02-19T21:07:20.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:20.202362+0000 mon.a (mon.0) 1803 : audit [INF] from='client.49394 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi163-83854-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:20.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:20.202473+0000 mon.a (mon.0) 1804 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-02-19T21:07:20.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:20.205843+0000 mon.c (mon.2) 317 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi163-83854-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:20.206759+0000 mon.a (mon.0) 1805 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi163-83854-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:20.208339+0000 mon.c (mon.2) 318 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:20.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:20.209164+0000 mon.a (mon.0) 1806 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:20.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:18.963162+0000 osd.7 (osd.7) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:20.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:18.963421+0000 osd.7 (osd.7) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:20.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:19.162216+0000 mgr.y (mgr.14550) 271 : cluster [DBG] pgmap v359: 246 pgs: 8 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:07:20.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:19.224162+0000 mon.c (mon.2) 314 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:19.443326+0000 mon.a (mon.0) 1796 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29"}]': finished 2024-02-19T21:07:20.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:19.443423+0000 mon.a (mon.0) 1797 : audit [INF] from='client.? v1:172.21.15.163:0/1254333461' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "32"}]': finished 2024-02-19T21:07:20.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:19.443474+0000 mon.a (mon.0) 1798 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-02-19T21:07:20.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:19.444767+0000 mon.a (mon.0) 1799 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:19.454123+0000 mon.c (mon.2) 315 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:19.462930+0000 mon.a (mon.0) 1800 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:19.463673+0000 mon.c (mon.2) 316 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi163-83854-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:20.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:19.463990+0000 mon.a (mon.0) 1801 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi163-83854-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:20.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:20.202270+0000 mon.a (mon.0) 1802 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi163-83888-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi163-83888-2"}]': finished 2024-02-19T21:07:20.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:18.963162+0000 osd.7 (osd.7) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:20.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:18.963421+0000 osd.7 (osd.7) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:20.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:19.162216+0000 mgr.y (mgr.14550) 271 : cluster [DBG] pgmap v359: 246 pgs: 8 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:07:20.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:19.224162+0000 mon.c (mon.2) 314 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:19.443326+0000 mon.a (mon.0) 1796 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-29"}]': finished 2024-02-19T21:07:20.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:19.443423+0000 mon.a (mon.0) 1797 : audit [INF] from='client.? v1:172.21.15.163:0/1254333461' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "32"}]': finished 2024-02-19T21:07:20.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:19.443474+0000 mon.a (mon.0) 1798 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-02-19T21:07:20.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:19.444767+0000 mon.a (mon.0) 1799 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:19.454123+0000 mon.c (mon.2) 315 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:19.462930+0000 mon.a (mon.0) 1800 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:19.463673+0000 mon.c (mon.2) 316 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi163-83854-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:20.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:19.463990+0000 mon.a (mon.0) 1801 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi163-83854-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:20.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:20.202270+0000 mon.a (mon.0) 1802 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi163-83888-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi163-83888-2"}]': finished 2024-02-19T21:07:20.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:20.202362+0000 mon.a (mon.0) 1803 : audit [INF] from='client.49394 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi163-83854-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:20.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:20.202473+0000 mon.a (mon.0) 1804 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-02-19T21:07:20.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:20.205843+0000 mon.c (mon.2) 317 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi163-83854-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:20.206759+0000 mon.a (mon.0) 1805 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi163-83854-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:20.208339+0000 mon.c (mon.2) 318 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:20.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:20.209164+0000 mon.a (mon.0) 1806 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:20.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:20.202362+0000 mon.a (mon.0) 1803 : audit [INF] from='client.49394 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi163-83854-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:20.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:20.202473+0000 mon.a (mon.0) 1804 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-02-19T21:07:20.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:20.205843+0000 mon.c (mon.2) 317 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi163-83854-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:20.206759+0000 mon.a (mon.0) 1805 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi163-83854-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:20.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:20.208339+0000 mon.c (mon.2) 318 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:20.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:20.209164+0000 mon.a (mon.0) 1806 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:20.853 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:07:21.326 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio_list: : cursor()=8:52ea6a34:::10:head expected=8:52ea6a34:::10:head 2024-02-19T21:07:21.327 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-02-19T21:07:21.327 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=10 expected=10 2024-02-19T21:07:21.327 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-02-19T21:07:21.327 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : cursor()=8:cfc208b3:::3:head expected=8:cfc208b3:::3:head 2024-02-19T21:07:21.327 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-02-19T21:07:21.327 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=3 expected=3 2024-02-19T21:07:21.327 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-02-19T21:07:21.327 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : cursor()=8:c4fdafeb:::6:head expected=8:c4fdafeb:::6:head 2024-02-19T21:07:21.327 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-02-19T21:07:21.327 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=6 expected=6 2024-02-19T21:07:21.327 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-02-19T21:07:21.327 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : cursor()=8:bd63b0f1:::8:head expected=8:bd63b0f1:::8:head 2024-02-19T21:07:21.327 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-02-19T21:07:21.328 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=8 expected=8 2024-02-19T21:07:21.328 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-02-19T21:07:21.328 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : cursor()=8:b29083e3:::5:head expected=8:b29083e3:::5:head 2024-02-19T21:07:21.328 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-02-19T21:07:21.328 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=5 expected=5 2024-02-19T21:07:21.328 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-02-19T21:07:21.328 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : cursor()=8:89d3ae78:::11:head expected=8:89d3ae78:::11:head 2024-02-19T21:07:21.328 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-02-19T21:07:21.328 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=11 expected=11 2024-02-19T21:07:21.328 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:863748b0:::15:head 2024-02-19T21:07:21.328 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : cursor()=8:863748b0:::15:head expected=8:863748b0:::15:head 2024-02-19T21:07:21.328 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-02-19T21:07:21.328 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=15 expected=15 2024-02-19T21:07:21.328 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : cursor()=8:6cac518f:::0:head expected=8:6cac518f:::0:head 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: : entry=0 expected=0 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (245 ms) 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (67897 ms) 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: split 0/5 -> MIN 8:33333333::::head 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: split 1/5 -> 8:33333333::::head 8:66666666::::head 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: split 2/5 -> 8:66666666::::head 8:99999999::::head 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: split 3/5 -> 8:99999999::::head 8:cccccccc::::head 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: split 4/5 -> 8:cccccccc::::head MAX 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (88243 ms) 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [----------] 7 tests from LibRadosList (156948 ms total) 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 2024-02-19T21:07:21.329 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-02-19T21:07:21.330 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-02-19T21:07:21.330 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1070 ms) 2024-02-19T21:07:21.330 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-02-19T21:07:21.330 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: myset foo1,foo2,foo3 2024-02-19T21:07:21.330 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo1 2024-02-19T21:07:21.330 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo2 2024-02-19T21:07:21.330 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo3 2024-02-19T21:07:21.330 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: myset foo1,foo4,foo5 2024-02-19T21:07:21.330 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo4 2024-02-19T21:07:21.330 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo5 2024-02-19T21:07:21.330 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo1 2024-02-19T21:07:21.330 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: myset foo6,foo7 2024-02-19T21:07:21.330 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo7 2024-02-19T21:07:21.330 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: foo6 2024-02-19T21:07:21.330 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-02-19T21:07:21.331 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: ns1:foo4 2024-02-19T21:07:21.331 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: ns1:foo5 2024-02-19T21:07:21.331 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: ns2:foo7 2024-02-19T21:07:21.331 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: ns2:foo6 2024-02-19T21:07:21.331 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: ns1:foo1 2024-02-19T21:07:21.331 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: :foo1 2024-02-19T21:07:21.331 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: :foo2 2024-02-19T21:07:21.331 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: :foo3 2024-02-19T21:07:21.331 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (30 ms) 2024-02-19T21:07:21.331 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-02-19T21:07:21.331 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 1 0 2024-02-19T21:07:21.331 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 10 0 2024-02-19T21:07:21.331 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 13 0 2024-02-19T21:07:21.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:20.218774+0000 mgr.y (mgr.14550) 272 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:21.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:21.163251+0000 mon.b (mon.1) 402 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "32"}]: dispatch 2024-02-19T21:07:21.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:21.164336+0000 mon.a (mon.0) 1807 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "32"}]: dispatch 2024-02-19T21:07:21.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:21.451772+0000 mon.a (mon.0) 1808 : audit [INF] from='client.49370 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi163-83845-37"}]': finished 2024-02-19T21:07:21.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:21.451952+0000 mon.a (mon.0) 1809 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "32"}]': finished 2024-02-19T21:07:21.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:21.452028+0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-02-19T21:07:21.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:21.452379+0000 mon.c (mon.2) 319 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:21.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:21.453006+0000 mon.b (mon.1) 403 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:21.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:21.453240+0000 mon.a (mon.0) 1811 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:21.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:21.454184+0000 mon.a (mon.0) 1812 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:21.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:20.218774+0000 mgr.y (mgr.14550) 272 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:21.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:21.163251+0000 mon.b (mon.1) 402 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "32"}]: dispatch 2024-02-19T21:07:21.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:21.164336+0000 mon.a (mon.0) 1807 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "32"}]: dispatch 2024-02-19T21:07:21.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:21.451772+0000 mon.a (mon.0) 1808 : audit [INF] from='client.49370 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi163-83845-37"}]': finished 2024-02-19T21:07:21.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:21.451952+0000 mon.a (mon.0) 1809 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "32"}]': finished 2024-02-19T21:07:21.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:21.452028+0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-02-19T21:07:21.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:21.452379+0000 mon.c (mon.2) 319 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:21.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:21.453006+0000 mon.b (mon.1) 403 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:21.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:21.453240+0000 mon.a (mon.0) 1811 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:21.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:21.454184+0000 mon.a (mon.0) 1812 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:20.218774+0000 mgr.y (mgr.14550) 272 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:21.163251+0000 mon.b (mon.1) 402 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "32"}]: dispatch 2024-02-19T21:07:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:21.164336+0000 mon.a (mon.0) 1807 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "32"}]: dispatch 2024-02-19T21:07:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:21.451772+0000 mon.a (mon.0) 1808 : audit [INF] from='client.49370 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi163-83845-37"}]': finished 2024-02-19T21:07:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:21.451952+0000 mon.a (mon.0) 1809 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "32"}]': finished 2024-02-19T21:07:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:21.452028+0000 mon.a (mon.0) 1810 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-02-19T21:07:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:21.452379+0000 mon.c (mon.2) 319 : audit [INF] from='client.? v1:172.21.15.163:0/2010916473' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:21.453006+0000 mon.b (mon.1) 403 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:21.453240+0000 mon.a (mon.0) 1811 : audit [INF] from='client.49370 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi163-83845-37"}]: dispatch 2024-02-19T21:07:21.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:21.454184+0000 mon.a (mon.0) 1812 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:22.429 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:22.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:22.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:22.535 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:22.564 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:22.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:21.162557+0000 mgr.y (mgr.14550) 273 : cluster [DBG] pgmap v362: 214 pgs: 8 unknown, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:07:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:22.206239+0000 mon.a (mon.0) 1813 : audit [INF] from='client.49394 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi163-83854-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi163-83854-47"}]': finished 2024-02-19T21:07:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:22.206478+0000 mon.a (mon.0) 1814 : audit [INF] from='client.49370 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi163-83845-37"}]': finished 2024-02-19T21:07:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:22.206592+0000 mon.a (mon.0) 1815 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:22.206664+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-02-19T21:07:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:22.214960+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi163-83888-2"}]: dispatch 2024-02-19T21:07:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:22.216743+0000 mon.b (mon.1) 404 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:22.217609+0000 mon.a (mon.0) 1818 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:22.241556+0000 mon.c (mon.2) 320 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:22.242135+0000 mon.a (mon.0) 1819 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:22.242931+0000 mon.c (mon.2) 321 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:22.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:22.243155+0000 mon.a (mon.0) 1820 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:22.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:22.243515+0000 mon.c (mon.2) 322 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi163-83845-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:22.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:22.243728+0000 mon.a (mon.0) 1821 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi163-83845-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:21.162557+0000 mgr.y (mgr.14550) 273 : cluster [DBG] pgmap v362: 214 pgs: 8 unknown, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:07:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:22.206239+0000 mon.a (mon.0) 1813 : audit [INF] from='client.49394 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi163-83854-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi163-83854-47"}]': finished 2024-02-19T21:07:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:22.206478+0000 mon.a (mon.0) 1814 : audit [INF] from='client.49370 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi163-83845-37"}]': finished 2024-02-19T21:07:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:22.206592+0000 mon.a (mon.0) 1815 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:22.206664+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-02-19T21:07:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:22.214960+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi163-83888-2"}]: dispatch 2024-02-19T21:07:22.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:22.216743+0000 mon.b (mon.1) 404 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:22.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:22.217609+0000 mon.a (mon.0) 1818 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:22.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:22.241556+0000 mon.c (mon.2) 320 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:22.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:22.242135+0000 mon.a (mon.0) 1819 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:22.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:22.242931+0000 mon.c (mon.2) 321 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:22.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:22.243155+0000 mon.a (mon.0) 1820 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:22.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:22.243515+0000 mon.c (mon.2) 322 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi163-83845-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:22.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:22.243728+0000 mon.a (mon.0) 1821 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi163-83845-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:22.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:21.162557+0000 mgr.y (mgr.14550) 273 : cluster [DBG] pgmap v362: 214 pgs: 8 unknown, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:07:22.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:22.206239+0000 mon.a (mon.0) 1813 : audit [INF] from='client.49394 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi163-83854-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi163-83854-47"}]': finished 2024-02-19T21:07:22.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:22.206478+0000 mon.a (mon.0) 1814 : audit [INF] from='client.49370 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi163-83845-37"}]': finished 2024-02-19T21:07:22.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:22.206592+0000 mon.a (mon.0) 1815 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:22.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:22.206664+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-02-19T21:07:22.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:22.214960+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi163-83888-2"}]: dispatch 2024-02-19T21:07:22.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:22.216743+0000 mon.b (mon.1) 404 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:22.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:22.217609+0000 mon.a (mon.0) 1818 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:22.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:22.241556+0000 mon.c (mon.2) 320 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:22.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:22.242135+0000 mon.a (mon.0) 1819 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:22.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:22.242931+0000 mon.c (mon.2) 321 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:22.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:22.243155+0000 mon.a (mon.0) 1820 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:22.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:22.243515+0000 mon.c (mon.2) 322 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi163-83845-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:22.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:22.243728+0000 mon.a (mon.0) 1821 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi163-83845-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:22.912 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:22.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:22.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:23.016 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:23.044 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:23.743 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:07:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:07:23] "GET /metrics HTTP/1.1" 200 192404 "" "Prometheus/2.33.4" 2024-02-19T21:07:23.971 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:23.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:23.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:24.004 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:07:24.004 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:07:24.004 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-02-19T21:07:24.004 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-02-19T21:07:24.004 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:07:24.076 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:24.105 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:24.356 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:24.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:24.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:24.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.230+0000 7f71f69e8700 1 -- 172.21.15.163:0/851009675 >> v1:172.21.15.163:6790/0 conn(0x7f71f00f8920 legacy=0x7f71f00f8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:24.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.230+0000 7f71f69e8700 1 -- 172.21.15.163:0/851009675 shutdown_connections 2024-02-19T21:07:24.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.230+0000 7f71f69e8700 1 -- 172.21.15.163:0/851009675 >> 172.21.15.163:0/851009675 conn(0x7f71f00f2470 msgr2=0x7f71f00f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:24.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.230+0000 7f71f69e8700 1 -- 172.21.15.163:0/851009675 shutdown_connections 2024-02-19T21:07:24.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.230+0000 7f71f69e8700 1 -- 172.21.15.163:0/851009675 wait complete. 2024-02-19T21:07:24.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.231+0000 7f71f69e8700 1 librados: starting msgr at 2024-02-19T21:07:24.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.231+0000 7f71f69e8700 1 librados: starting objecter 2024-02-19T21:07:24.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.231+0000 7f71f69e8700 1 Processor -- start 2024-02-19T21:07:24.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.231+0000 7f71f69e8700 1 -- start start 2024-02-19T21:07:24.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.231+0000 7f71f69e8700 1 librados: setting wanted keys 2024-02-19T21:07:24.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.231+0000 7f71f69e8700 1 librados: calling monclient init 2024-02-19T21:07:24.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.231+0000 7f71f69e8700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71f0102ac0 con 0x7f71f00f94d0 2024-02-19T21:07:24.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.231+0000 7f71f69e8700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71f0102c90 con 0x7f71f00f8920 2024-02-19T21:07:24.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.231+0000 7f71f69e8700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71f0102e80 con 0x7f71f01042b0 2024-02-19T21:07:24.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.231+0000 7f71f4f85700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f71f01042b0 0x7f71f0118430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:58582/0 (socket says 172.21.15.163:58582) 2024-02-19T21:07:24.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71f4f85700 1 -- 172.21.15.163:0/3794000966 learned_addr learned my addr 172.21.15.163:0/3794000966 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:24.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.231+0000 7f71ef7fe700 1 --1- 172.21.15.163:0/3794000966 >> v1:172.21.15.163:6789/0 conn(0x7f71f00f94d0 0x7f71f0114d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:58584/0 (socket says 172.21.15.163:58584) 2024-02-19T21:07:24.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 10 client.?.objecter ms_handle_connect 0x7f71f01042b0 2024-02-19T21:07:24.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:24.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 10 client.?.objecter ms_handle_connect 0x7f71f00f94d0 2024-02-19T21:07:24.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:24.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 10 client.?.objecter ms_handle_connect 0x7f71f00f8920 2024-02-19T21:07:24.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:24.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546406227 0 0) 0x7f71f0102e80 con 0x7f71f01042b0 2024-02-19T21:07:24.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71d8002220 con 0x7f71f01042b0 2024-02-19T21:07:24.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602141689 0 0) 0x7f71f0102c90 con 0x7f71f00f8920 2024-02-19T21:07:24.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71f0102e80 con 0x7f71f00f8920 2024-02-19T21:07:24.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 447063002 0 0) 0x7f71d8002220 con 0x7f71f01042b0 2024-02-19T21:07:24.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71f0102c90 con 0x7f71f01042b0 2024-02-19T21:07:24.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f71e8001c20 con 0x7f71f01042b0 2024-02-19T21:07:24.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3890463932 0 0) 0x7f71f0102e80 con 0x7f71f00f8920 2024-02-19T21:07:24.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71d8002220 con 0x7f71f00f8920 2024-02-19T21:07:24.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f71e0001e50 con 0x7f71f00f8920 2024-02-19T21:07:24.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2053600745 0 0) 0x7f71f0102c90 con 0x7f71f01042b0 2024-02-19T21:07:24.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 >> v1:172.21.15.193:6789/0 conn(0x7f71f00f8920 legacy=0x7f71f0102320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:24.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 >> v1:172.21.15.163:6789/0 conn(0x7f71f00f94d0 legacy=0x7f71f0114d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:24.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f011cb80 con 0x7f71f01042b0 2024-02-19T21:07:24.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71f69e8700 1 -- 172.21.15.163:0/3794000966 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f0119bb0 con 0x7f71f01042b0 2024-02-19T21:07:24.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71f69e8700 10 client.49406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:24.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71f69e8700 1 -- 172.21.15.163:0/3794000966 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f011a020 con 0x7f71f01042b0 2024-02-19T21:07:24.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.232+0000 7f71f69e8700 1 librados: init done 2024-02-19T21:07:24.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.233+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f71e8002700 con 0x7f71f01042b0 2024-02-19T21:07:24.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.233+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f71e8005df0 con 0x7f71f01042b0 2024-02-19T21:07:24.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.234+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f71e8018d00 con 0x7f71f01042b0 2024-02-19T21:07:24.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.234+0000 7f71f69e8700 1 -- 172.21.15.163:0/3794000966 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71dc004f20 con 0x7f71f01042b0 2024-02-19T21:07:24.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.234+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(264..264 src has 43..264) v4 ==== 8798+0+0 (unknown 1753259234 0 0) 0x7f71e80029f0 con 0x7f71f01042b0 2024-02-19T21:07:24.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.234+0000 7f71ed7fa700 10 client.49406.objecter ms_dispatch 0x7f71f0001550 osd_map(264..264 src has 43..264) v4 2024-02-19T21:07:24.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.234+0000 7f71ed7fa700 3 client.49406.objecter handle_osd_map got epochs [264,264] > 0 2024-02-19T21:07:24.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.234+0000 7f71ed7fa700 3 client.49406.objecter handle_osd_map decoding full epoch 264 2024-02-19T21:07:24.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.234+0000 7f71ed7fa700 20 client.49406.objecter dump_active .. 0 homeless 2024-02-19T21:07:24.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.234+0000 7f71ed7fa700 10 client.49406.objecter ms_handle_connect 0x7f71d8063b20 2024-02-19T21:07:24.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.239+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f71e804d300 con 0x7f71f01042b0 2024-02-19T21:07:24.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:24.426+0000 7f71f69e8700 1 -- 172.21.15.163:0/3794000966 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f71dc0051d0 con 0x7f71f01042b0 2024-02-19T21:07:24.462 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:24.465 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:24.491 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:24.710 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:23.163032+0000 mgr.y (mgr.14550) 274 : cluster [DBG] pgmap v365: 246 pgs: 40 unknown, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:07:24.710 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:23.208186+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi163-83888-2"}]': finished 2024-02-19T21:07:24.710 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:23.208299+0000 mon.a (mon.0) 1823 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:24.710 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:23.208413+0000 mon.a (mon.0) 1824 : audit [INF] from='client.49400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi163-83845-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:24.710 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:23.208485+0000 mon.a (mon.0) 1825 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-02-19T21:07:24.711 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:23.209712+0000 mon.c (mon.2) 323 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi163-83845-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:24.711 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:23.211330+0000 mon.b (mon.1) 405 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-31"}]: dispatch 2024-02-19T21:07:24.711 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:23.211528+0000 mon.a (mon.0) 1826 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi163-83845-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:24.711 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:23.212074+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi163-83888-2"}]: dispatch 2024-02-19T21:07:24.711 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:23.212363+0000 mon.a (mon.0) 1828 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-31"}]: dispatch 2024-02-19T21:07:24.711 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:24.098748+0000 mon.a (mon.0) 1829 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:24.711 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:23.163032+0000 mgr.y (mgr.14550) 274 : cluster [DBG] pgmap v365: 246 pgs: 40 unknown, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:07:24.711 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:23.208186+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi163-83888-2"}]': finished 2024-02-19T21:07:24.711 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:23.208299+0000 mon.a (mon.0) 1823 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:24.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:23.208413+0000 mon.a (mon.0) 1824 : audit [INF] from='client.49400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi163-83845-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:24.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:23.208485+0000 mon.a (mon.0) 1825 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-02-19T21:07:24.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:23.209712+0000 mon.c (mon.2) 323 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi163-83845-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:24.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:23.211330+0000 mon.b (mon.1) 405 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-31"}]: dispatch 2024-02-19T21:07:24.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:23.211528+0000 mon.a (mon.0) 1826 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi163-83845-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:24.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:23.212074+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi163-83888-2"}]: dispatch 2024-02-19T21:07:24.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:23.212363+0000 mon.a (mon.0) 1828 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-31"}]: dispatch 2024-02-19T21:07:24.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:24.098748+0000 mon.a (mon.0) 1829 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:24.743 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:24.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:24.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:23.163032+0000 mgr.y (mgr.14550) 274 : cluster [DBG] pgmap v365: 246 pgs: 40 unknown, 206 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:07:24.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:23.208186+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi163-83888-2"}]': finished 2024-02-19T21:07:24.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:23.208299+0000 mon.a (mon.0) 1823 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:24.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:23.208413+0000 mon.a (mon.0) 1824 : audit [INF] from='client.49400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi163-83845-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:24.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:23.208485+0000 mon.a (mon.0) 1825 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-02-19T21:07:24.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:23.209712+0000 mon.c (mon.2) 323 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi163-83845-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:24.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:23.211330+0000 mon.b (mon.1) 405 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-31"}]: dispatch 2024-02-19T21:07:24.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:23.211528+0000 mon.a (mon.0) 1826 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi163-83845-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:24.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:23.212074+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi163-83888-2"}]: dispatch 2024-02-19T21:07:24.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:23.212363+0000 mon.a (mon.0) 1828 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-31"}]: dispatch 2024-02-19T21:07:24.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:24.098748+0000 mon.a (mon.0) 1829 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:24.847 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:24.875 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:24.972 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:24.975 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:24.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:25.077 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:25.104 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:25.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:25.203+0000 7f71ed7fa700 1 -- 172.21.15.163:0/3794000966 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v265) v1 ==== 91+0+0 (unknown 1503747274 0 0) 0x7f71e804b710 con 0x7f71f01042b0 2024-02-19T21:07:25.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:25.208+0000 7f71f69e8700 10 librados: watch_flush enter 2024-02-19T21:07:25.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:25.211+0000 7f71f69e8700 10 librados: watch_flush exit 2024-02-19T21:07:25.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:25.211+0000 7f71f69e8700 20 client.49406.objecter shutdown clearing up homeless session... 2024-02-19T21:07:25.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:25.211+0000 7f71f69e8700 10 client.49406.objecter successfully canceled tick 2024-02-19T21:07:25.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:25.211+0000 7f71f69e8700 1 -- 172.21.15.163:0/3794000966 >> v1:172.21.15.163:6800/1552514681 conn(0x7f71d8063b20 legacy=0x7f71d8065fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:25.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:25.211+0000 7f71f69e8700 1 -- 172.21.15.163:0/3794000966 >> v1:172.21.15.163:6790/0 conn(0x7f71f01042b0 legacy=0x7f71f0118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:25.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:25.212+0000 7f71f69e8700 1 -- 172.21.15.163:0/3794000966 shutdown_connections 2024-02-19T21:07:25.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:25.212+0000 7f71f69e8700 1 -- 172.21.15.163:0/3794000966 >> 172.21.15.163:0/3794000966 conn(0x7f71f00f2470 msgr2=0x7f71f0107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:25.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:25.212+0000 7f71f69e8700 1 -- 172.21.15.163:0/3794000966 shutdown_connections 2024-02-19T21:07:25.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:25.212+0000 7f71f69e8700 1 -- 172.21.15.163:0/3794000966 wait complete. 2024-02-19T21:07:25.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:25.213+0000 7f71f69e8700 1 librados: shutdown 2024-02-19T21:07:25.210 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:07:25.454 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:25.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:25.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:25.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:24.427517+0000 mon.c (mon.2) 324 : audit [INF] from='client.? v1:172.21.15.163:0/3794000966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:07:25.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:24.445639+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi163-83888-2"}]': finished 2024-02-19T21:07:25.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:24.445733+0000 mon.a (mon.0) 1831 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-31"}]': finished 2024-02-19T21:07:25.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:24.445812+0000 mon.a (mon.0) 1832 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-02-19T21:07:25.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:24.446717+0000 mon.b (mon.1) 406 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-31", "mode": "writeback"}]: dispatch 2024-02-19T21:07:25.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:24.446823+0000 mon.a (mon.0) 1833 : audit [INF] from='client.49406 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:07:25.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:24.446870+0000 mon.c (mon.2) 325 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:25.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:24.447081+0000 mon.a (mon.0) 1834 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-31", "mode": "writeback"}]: dispatch 2024-02-19T21:07:25.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:24.453111+0000 mon.a (mon.0) 1835 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:25.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:24.973285+0000 mon.b (mon.1) 407 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:07:25.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.164433+0000 mon.b (mon.1) 408 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "41"}]: dispatch 2024-02-19T21:07:25.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.165314+0000 mon.a (mon.0) 1836 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "41"}]: dispatch 2024-02-19T21:07:25.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:25.198966+0000 mon.a (mon.0) 1837 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:25.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:25.198989+0000 mon.a (mon.0) 1838 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:25.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:24.427517+0000 mon.c (mon.2) 324 : audit [INF] from='client.? v1:172.21.15.163:0/3794000966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:07:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:24.445639+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi163-83888-2"}]': finished 2024-02-19T21:07:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:24.445733+0000 mon.a (mon.0) 1831 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-31"}]': finished 2024-02-19T21:07:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:24.445812+0000 mon.a (mon.0) 1832 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-02-19T21:07:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:24.446717+0000 mon.b (mon.1) 406 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-31", "mode": "writeback"}]: dispatch 2024-02-19T21:07:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:24.446823+0000 mon.a (mon.0) 1833 : audit [INF] from='client.49406 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:07:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:24.446870+0000 mon.c (mon.2) 325 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:24.447081+0000 mon.a (mon.0) 1834 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-31", "mode": "writeback"}]: dispatch 2024-02-19T21:07:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:24.453111+0000 mon.a (mon.0) 1835 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:24.973285+0000 mon.b (mon.1) 407 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:07:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.164433+0000 mon.b (mon.1) 408 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "41"}]: dispatch 2024-02-19T21:07:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.165314+0000 mon.a (mon.0) 1836 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "41"}]: dispatch 2024-02-19T21:07:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:25.198966+0000 mon.a (mon.0) 1837 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:25.198989+0000 mon.a (mon.0) 1838 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:25.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.203837+0000 mon.a (mon.0) 1839 : audit [INF] from='client.49400 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi163-83845-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi163-83845-38"}]': finished 2024-02-19T21:07:25.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.203930+0000 mon.a (mon.0) 1840 : audit [INF] from='client.49406 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:07:25.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.204000+0000 mon.a (mon.0) 1841 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-31", "mode": "writeback"}]': finished 2024-02-19T21:07:25.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.204065+0000 mon.a (mon.0) 1842 : audit [INF] from='client.49394 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi163-83854-47"}]': finished 2024-02-19T21:07:25.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.204154+0000 mon.a (mon.0) 1843 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "41"}]': finished 2024-02-19T21:07:25.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:25.204228+0000 mon.a (mon.0) 1844 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-02-19T21:07:25.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.204987+0000 mon.c (mon.2) 326 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:25.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.207144+0000 mon.a (mon.0) 1845 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:25.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.211168+0000 mon.c (mon.2) 327 : audit [INF] from='client.? v1:172.21.15.163:0/243193800' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83888-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:25.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.211904+0000 mon.a (mon.0) 1846 : audit [INF] from='client.49412 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83888-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:25.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.222844+0000 mon.a (mon.0) 1847 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:25.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.303437+0000 mon.b (mon.1) 409 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:25.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.303767+0000 mon.a (mon.0) 1848 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:25.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.203837+0000 mon.a (mon.0) 1839 : audit [INF] from='client.49400 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi163-83845-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi163-83845-38"}]': finished 2024-02-19T21:07:25.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.203930+0000 mon.a (mon.0) 1840 : audit [INF] from='client.49406 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:07:25.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.204000+0000 mon.a (mon.0) 1841 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-31", "mode": "writeback"}]': finished 2024-02-19T21:07:25.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.204065+0000 mon.a (mon.0) 1842 : audit [INF] from='client.49394 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi163-83854-47"}]': finished 2024-02-19T21:07:25.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.204154+0000 mon.a (mon.0) 1843 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "41"}]': finished 2024-02-19T21:07:25.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:25.204228+0000 mon.a (mon.0) 1844 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-02-19T21:07:25.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.204987+0000 mon.c (mon.2) 326 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:25.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.207144+0000 mon.a (mon.0) 1845 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:25.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.211168+0000 mon.c (mon.2) 327 : audit [INF] from='client.? v1:172.21.15.163:0/243193800' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83888-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:25.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.211904+0000 mon.a (mon.0) 1846 : audit [INF] from='client.49412 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83888-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:25.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.222844+0000 mon.a (mon.0) 1847 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:25.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.303437+0000 mon.b (mon.1) 409 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:25.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.303767+0000 mon.a (mon.0) 1848 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:25.559 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:25.564 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-19T21:07:25.586 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:25.605 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:25.607 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:25.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:25.709 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:07:25.733 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:24.427517+0000 mon.c (mon.2) 324 : audit [INF] from='client.? v1:172.21.15.163:0/3794000966' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:07:25.733 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:24.445639+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? v1:172.21.15.163:0/585802156' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi163-83888-2"}]': finished 2024-02-19T21:07:25.733 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:24.445733+0000 mon.a (mon.0) 1831 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-31"}]': finished 2024-02-19T21:07:25.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:24.445812+0000 mon.a (mon.0) 1832 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-02-19T21:07:25.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:24.446717+0000 mon.b (mon.1) 406 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-31", "mode": "writeback"}]: dispatch 2024-02-19T21:07:25.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:24.446823+0000 mon.a (mon.0) 1833 : audit [INF] from='client.49406 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:07:25.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:24.446870+0000 mon.c (mon.2) 325 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:25.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:24.447081+0000 mon.a (mon.0) 1834 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-31", "mode": "writeback"}]: dispatch 2024-02-19T21:07:25.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:24.453111+0000 mon.a (mon.0) 1835 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:25.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:24.973285+0000 mon.b (mon.1) 407 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:07:25.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.164433+0000 mon.b (mon.1) 408 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "41"}]: dispatch 2024-02-19T21:07:25.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.165314+0000 mon.a (mon.0) 1836 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "41"}]: dispatch 2024-02-19T21:07:25.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:25.198966+0000 mon.a (mon.0) 1837 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:25.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:25.198989+0000 mon.a (mon.0) 1838 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:25.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.203837+0000 mon.a (mon.0) 1839 : audit [INF] from='client.49400 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi163-83845-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi163-83845-38"}]': finished 2024-02-19T21:07:25.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.203930+0000 mon.a (mon.0) 1840 : audit [INF] from='client.49406 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:07:25.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.204000+0000 mon.a (mon.0) 1841 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-31", "mode": "writeback"}]': finished 2024-02-19T21:07:25.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.204065+0000 mon.a (mon.0) 1842 : audit [INF] from='client.49394 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi163-83854-47"}]': finished 2024-02-19T21:07:25.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.204154+0000 mon.a (mon.0) 1843 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "41"}]': finished 2024-02-19T21:07:25.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:25.204228+0000 mon.a (mon.0) 1844 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-02-19T21:07:25.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.204987+0000 mon.c (mon.2) 326 : audit [INF] from='client.? v1:172.21.15.163:0/3062946080' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:25.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.207144+0000 mon.a (mon.0) 1845 : audit [INF] from='client.49394 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi163-83854-47"}]: dispatch 2024-02-19T21:07:25.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.211168+0000 mon.c (mon.2) 327 : audit [INF] from='client.? v1:172.21.15.163:0/243193800' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83888-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:25.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.211904+0000 mon.a (mon.0) 1846 : audit [INF] from='client.49412 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83888-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:25.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.222844+0000 mon.a (mon.0) 1847 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:25.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.303437+0000 mon.b (mon.1) 409 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:25.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.303767+0000 mon.a (mon.0) 1848 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:25.735 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:25.753 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:07:25.756 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:25.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:07:25.857 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:25.884 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:25.921 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:25.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:25.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:26.027 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:26.053 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:26.072 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:26.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:26.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:26.175 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:07:26.203 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:26.233 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:07:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:26.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:07:26.336 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:07:26.363 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:26.495 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:07:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:26.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:07:26.599 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:26.626 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:26.646 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:26.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:26.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:26.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:25.163957+0000 mgr.y (mgr.14550) 275 : cluster [DBG] pgmap v368: 238 pgs: 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:07:26.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.698441+0000 mon.b (mon.1) 410 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:07:26.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.699206+0000 mon.b (mon.1) 411 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:07:26.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:25.703751+0000 mon.a (mon.0) 1849 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:26.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:25.809716+0000 osd.7 (osd.7) 129 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:07:26.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:25.812624+0000 osd.7 (osd.7) 130 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:07:26.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:26.207820+0000 mon.a (mon.0) 1850 : audit [INF] from='client.49394 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi163-83854-47"}]': finished 2024-02-19T21:07:26.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:26.207988+0000 mon.a (mon.0) 1851 : audit [INF] from='client.49412 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83888-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:26.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:26.208072+0000 mon.a (mon.0) 1852 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:26.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:26.208130+0000 mon.a (mon.0) 1853 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-02-19T21:07:26.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:26.210366+0000 mon.b (mon.1) 412 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31"}]: dispatch 2024-02-19T21:07:26.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:26.211452+0000 mon.a (mon.0) 1854 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31"}]: dispatch 2024-02-19T21:07:26.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:26.234092+0000 mon.a (mon.0) 1855 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi163-83854-48"}]: dispatch 2024-02-19T21:07:26.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:26.234849+0000 mon.c (mon.2) 328 : audit [INF] from='client.? v1:172.21.15.163:0/243193800' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi163-83888-3","pool2":"test-rados-api-smithi163-83888-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-19T21:07:26.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:26.235214+0000 mon.a (mon.0) 1856 : audit [INF] from='client.49412 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi163-83888-3","pool2":"test-rados-api-smithi163-83888-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-19T21:07:26.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:26.235851+0000 mon.a (mon.0) 1857 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi163-83854-48"}]: dispatch 2024-02-19T21:07:26.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:26.236116+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi163-83854-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:26.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:25.163957+0000 mgr.y (mgr.14550) 275 : cluster [DBG] pgmap v368: 238 pgs: 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:07:26.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.698441+0000 mon.b (mon.1) 410 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:07:26.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.699206+0000 mon.b (mon.1) 411 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:07:26.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:25.703751+0000 mon.a (mon.0) 1849 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:26.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:25.809716+0000 osd.7 (osd.7) 129 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:07:26.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:25.812624+0000 osd.7 (osd.7) 130 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:07:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:26.207820+0000 mon.a (mon.0) 1850 : audit [INF] from='client.49394 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi163-83854-47"}]': finished 2024-02-19T21:07:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:26.207988+0000 mon.a (mon.0) 1851 : audit [INF] from='client.49412 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83888-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:26.208072+0000 mon.a (mon.0) 1852 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:26.208130+0000 mon.a (mon.0) 1853 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-02-19T21:07:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:26.210366+0000 mon.b (mon.1) 412 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31"}]: dispatch 2024-02-19T21:07:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:26.211452+0000 mon.a (mon.0) 1854 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31"}]: dispatch 2024-02-19T21:07:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:26.234092+0000 mon.a (mon.0) 1855 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi163-83854-48"}]: dispatch 2024-02-19T21:07:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:26.234849+0000 mon.c (mon.2) 328 : audit [INF] from='client.? v1:172.21.15.163:0/243193800' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi163-83888-3","pool2":"test-rados-api-smithi163-83888-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-19T21:07:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:26.235214+0000 mon.a (mon.0) 1856 : audit [INF] from='client.49412 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi163-83888-3","pool2":"test-rados-api-smithi163-83888-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-19T21:07:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:26.235851+0000 mon.a (mon.0) 1857 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi163-83854-48"}]: dispatch 2024-02-19T21:07:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:26.236116+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi163-83854-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:26.749 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:26.775 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:26.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:25.163957+0000 mgr.y (mgr.14550) 275 : cluster [DBG] pgmap v368: 238 pgs: 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:07:26.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.698441+0000 mon.b (mon.1) 410 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:07:26.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.699206+0000 mon.b (mon.1) 411 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:07:26.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:25.703751+0000 mon.a (mon.0) 1849 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:26.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:25.809716+0000 osd.7 (osd.7) 129 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:07:26.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:25.812624+0000 osd.7 (osd.7) 130 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:07:26.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:26.207820+0000 mon.a (mon.0) 1850 : audit [INF] from='client.49394 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi163-83854-47"}]': finished 2024-02-19T21:07:26.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:26.207988+0000 mon.a (mon.0) 1851 : audit [INF] from='client.49412 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83888-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:26.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:26.208072+0000 mon.a (mon.0) 1852 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:26.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:26.208130+0000 mon.a (mon.0) 1853 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-02-19T21:07:26.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:26.210366+0000 mon.b (mon.1) 412 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31"}]: dispatch 2024-02-19T21:07:26.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:26.211452+0000 mon.a (mon.0) 1854 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31"}]: dispatch 2024-02-19T21:07:26.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:26.234092+0000 mon.a (mon.0) 1855 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi163-83854-48"}]: dispatch 2024-02-19T21:07:26.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:26.234849+0000 mon.c (mon.2) 328 : audit [INF] from='client.? v1:172.21.15.163:0/243193800' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi163-83888-3","pool2":"test-rados-api-smithi163-83888-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-19T21:07:26.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:26.235214+0000 mon.a (mon.0) 1856 : audit [INF] from='client.49412 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi163-83888-3","pool2":"test-rados-api-smithi163-83888-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-02-19T21:07:26.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:26.235851+0000 mon.a (mon.0) 1857 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi163-83854-48"}]: dispatch 2024-02-19T21:07:26.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:26.236116+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi163-83854-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:26.974 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:26.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:26.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:27.078 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:27.105 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:27.224 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 7 0 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 14 0 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 0 0 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 15 0 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 11 0 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 5 0 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 8 0 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 6 0 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 3 0 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 4 0 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 12 0 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 9 0 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 2 0 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (43 ms) 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [----------] 3 tests from LibRadosListEC (1143 ms total) 2024-02-19T21:07:27.225 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 2024-02-19T21:07:27.226 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-02-19T21:07:27.226 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-02-19T21:07:27.226 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2779 ms) 2024-02-19T21:07:27.226 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [----------] 1 test from LibRadosListNP (2779 ms total) 2024-02-19T21:07:27.226 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: 2024-02-19T21:07:27.226 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [----------] Global test environment tear-down 2024-02-19T21:07:27.226 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [==========] 11 tests from 3 test suites ran. (174359 ms total) 2024-02-19T21:07:27.226 INFO:tasks.workunit.client.0.smithi163.stdout: api_list: [ PASSED ] 11 tests. 2024-02-19T21:07:27.226 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:07:27.227 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83971 2024-02-19T21:07:27.227 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83971 2024-02-19T21:07:27.227 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:07:27.227 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83963 2024-02-19T21:07:27.227 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83963 2024-02-19T21:07:27.227 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:07:27.227 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=84027 2024-02-19T21:07:27.227 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 84027 2024-02-19T21:07:27.227 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:07:27.227 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83848 2024-02-19T21:07:27.228 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83848 2024-02-19T21:07:27.228 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:27.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:27.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:27.332 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:27.361 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:27.380 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:27.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:27.484 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:27.485 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:27.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.493+0000 7f5697f31700 1 -- 172.21.15.163:0/3846954189 >> v1:172.21.15.163:6789/0 conn(0x7f56901042c0 legacy=0x7f5690106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:27.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.493+0000 7f5697f31700 1 -- 172.21.15.163:0/3846954189 shutdown_connections 2024-02-19T21:07:27.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.493+0000 7f5697f31700 1 -- 172.21.15.163:0/3846954189 >> 172.21.15.163:0/3846954189 conn(0x7f56900f35a0 msgr2=0x7f56900f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:27.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.493+0000 7f5697f31700 1 -- 172.21.15.163:0/3846954189 shutdown_connections 2024-02-19T21:07:27.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.493+0000 7f5697f31700 1 -- 172.21.15.163:0/3846954189 wait complete. 2024-02-19T21:07:27.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.494+0000 7f5697f31700 1 librados: starting msgr at 2024-02-19T21:07:27.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.494+0000 7f5697f31700 1 librados: starting objecter 2024-02-19T21:07:27.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.494+0000 7f5697f31700 1 Processor -- start 2024-02-19T21:07:27.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.494+0000 7f5697f31700 1 -- start start 2024-02-19T21:07:27.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.494+0000 7f5697f31700 1 librados: setting wanted keys 2024-02-19T21:07:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.494+0000 7f5697f31700 1 librados: calling monclient init 2024-02-19T21:07:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.494+0000 7f5697f31700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f56901173e0 con 0x7f5690100780 2024-02-19T21:07:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.494+0000 7f5697f31700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f56901185e0 con 0x7f56900fd980 2024-02-19T21:07:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.494+0000 7f5697f31700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f56901197e0 con 0x7f56901042c0 2024-02-19T21:07:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5695ccd700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f56900fd980 0x7f56900f9b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:39302/0 (socket says 172.21.15.163:39302) 2024-02-19T21:07:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5695ccd700 1 -- 172.21.15.163:0/867658936 learned_addr learned my addr 172.21.15.163:0/867658936 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 10 client.?.objecter ms_handle_connect 0x7f5690100780 2024-02-19T21:07:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 10 client.?.objecter ms_handle_connect 0x7f56900fd980 2024-02-19T21:07:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908424636 0 0) 0x7f56901185e0 con 0x7f56900fd980 2024-02-19T21:07:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f567c0033f0 con 0x7f56900fd980 2024-02-19T21:07:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641055290 0 0) 0x7f56901173e0 con 0x7f5690100780 2024-02-19T21:07:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56901185e0 con 0x7f5690100780 2024-02-19T21:07:27.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2711156516 0 0) 0x7f567c0033f0 con 0x7f56900fd980 2024-02-19T21:07:27.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56901173e0 con 0x7f56900fd980 2024-02-19T21:07:27.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f568c001990 con 0x7f56900fd980 2024-02-19T21:07:27.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4241671191 0 0) 0x7f56901185e0 con 0x7f5690100780 2024-02-19T21:07:27.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f567c0033f0 con 0x7f5690100780 2024-02-19T21:07:27.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f56800018c0 con 0x7f5690100780 2024-02-19T21:07:27.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4194878681 0 0) 0x7f56901173e0 con 0x7f56900fd980 2024-02-19T21:07:27.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 >> v1:172.21.15.163:6790/0 conn(0x7f56901042c0 legacy=0x7f5690115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:27.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 >> v1:172.21.15.163:6789/0 conn(0x7f5690100780 legacy=0x7f56900fa230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:27.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569011a9e0 con 0x7f56900fd980 2024-02-19T21:07:27.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56901187d0 con 0x7f56900fd980 2024-02-19T21:07:27.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5697f31700 10 client.48235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:27.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5690118c40 con 0x7f56900fd980 2024-02-19T21:07:27.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5697f31700 1 librados: init done 2024-02-19T21:07:27.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.495+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f568c002360 con 0x7f56900fd980 2024-02-19T21:07:27.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.496+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f568c005aa0 con 0x7f56900fd980 2024-02-19T21:07:27.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.496+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f5674004f00 con 0x7f56900fd980 2024-02-19T21:07:27.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.497+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f568c018a10 con 0x7f56900fd980 2024-02-19T21:07:27.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.497+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(267..267 src has 43..267) v4 ==== 8680+0+0 (unknown 3606472163 0 0) 0x7f568c079220 con 0x7f56900fd980 2024-02-19T21:07:27.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.497+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 osd_map(267..267 src has 43..267) v4 2024-02-19T21:07:27.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.497+0000 7f5686ffd700 3 client.48235.objecter handle_osd_map got epochs [267,267] > 0 2024-02-19T21:07:27.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.497+0000 7f5686ffd700 3 client.48235.objecter handle_osd_map decoding full epoch 267 2024-02-19T21:07:27.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.497+0000 7f5686ffd700 20 client.48235.objecter dump_active .. 0 homeless 2024-02-19T21:07:27.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.497+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v267) v1 ==== 54+0+16 (unknown 4172006663 0 2420000192) 0x7f568c07b610 con 0x7f56900fd980 2024-02-19T21:07:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.498+0000 7f5686ffd700 10 client.48235.objecter ms_handle_connect 0x7f567c063c80 2024-02-19T21:07:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.498+0000 7f5697f31700 10 client.48235.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.498+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 1 2024-02-19T21:07:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.498+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f56740051a0 osd=0 3 2024-02-19T21:07:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.498+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f56740051a0 osd=0 3 2024-02-19T21:07:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.498+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f56740051a0 osd=0 3 2024-02-19T21:07:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.498+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 1 2024-02-19T21:07:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.498+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f56740051a0 osd=0 3 2024-02-19T21:07:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.498+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 0 1 2024-02-19T21:07:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.498+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f56740051a0 osd=0 4 2024-02-19T21:07:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.498+0000 7f5697f31700 10 client.48235.objecter _send_command 1 2024-02-19T21:07:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.498+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5674004950 con 0x7f5674004530 2024-02-19T21:07:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.498+0000 7f5686ffd700 10 client.48235.objecter ms_handle_connect 0x7f5674004530 2024-02-19T21:07:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.499+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5674004950 con 0x7f5674004530 2024-02-19T21:07:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.499+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:07:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.499+0000 7f5686ffd700 10 client.48235.objecter _finish_command 1 = system:0 2024-02-19T21:07:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.499+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f56740051a0 osd=0 4 2024-02-19T21:07:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.499+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 0 1 2024-02-19T21:07:27.511 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:27.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.515+0000 7f5697f31700 10 client.48235.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:27.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.515+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 2 2024-02-19T21:07:27.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.515+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f56740051a0 osd=0 3 2024-02-19T21:07:27.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.515+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f56740051a0 osd=0 3 2024-02-19T21:07:27.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.515+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f56740051a0 osd=0 3 2024-02-19T21:07:27.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.515+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 2 2024-02-19T21:07:27.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.515+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f56740051a0 osd=0 3 2024-02-19T21:07:27.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.515+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 0 2 2024-02-19T21:07:27.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.515+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f56740051a0 osd=0 4 2024-02-19T21:07:27.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.515+0000 7f5697f31700 10 client.48235.objecter _send_command 2 2024-02-19T21:07:27.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.515+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5674004d30 con 0x7f5674004530 2024-02-19T21:07:27.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5674004d30 con 0x7f5674004530 2024-02-19T21:07:27.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:27.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5686ffd700 10 client.48235.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f56740051a0 osd=0 4 2024-02-19T21:07:27.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 0 2 2024-02-19T21:07:27.513 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:07:27.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5697f31700 10 client.48235.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:27.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 3 2024-02-19T21:07:27.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674003fe0 osd=1 3 2024-02-19T21:07:27.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674003fe0 osd=1 3 2024-02-19T21:07:27.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674003fe0 osd=1 3 2024-02-19T21:07:27.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 3 2024-02-19T21:07:27.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f5674003fe0 osd=1 3 2024-02-19T21:07:27.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 1 3 2024-02-19T21:07:27.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674003fe0 osd=1 4 2024-02-19T21:07:27.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5697f31700 10 client.48235.objecter _send_command 3 2024-02-19T21:07:27.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5674010450 con 0x7f5674004150 2024-02-19T21:07:27.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.517+0000 7f5686ffd700 10 client.48235.objecter ms_handle_connect 0x7f5674004150 2024-02-19T21:07:27.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.518+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5674010450 con 0x7f5674004150 2024-02-19T21:07:27.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.518+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:07:27.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.518+0000 7f5686ffd700 10 client.48235.objecter _finish_command 3 = system:0 2024-02-19T21:07:27.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.518+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f5674003fe0 osd=1 4 2024-02-19T21:07:27.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.518+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 1 3 2024-02-19T21:07:27.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.533+0000 7f5697f31700 10 client.48235.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:27.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.533+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 4 2024-02-19T21:07:27.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.533+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674003fe0 osd=1 3 2024-02-19T21:07:27.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.533+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674003fe0 osd=1 3 2024-02-19T21:07:27.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.533+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674003fe0 osd=1 3 2024-02-19T21:07:27.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.533+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 4 2024-02-19T21:07:27.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.533+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f5674003fe0 osd=1 3 2024-02-19T21:07:27.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.533+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 1 4 2024-02-19T21:07:27.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.533+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674003fe0 osd=1 4 2024-02-19T21:07:27.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.533+0000 7f5697f31700 10 client.48235.objecter _send_command 4 2024-02-19T21:07:27.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.533+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5674001850 con 0x7f5674004150 2024-02-19T21:07:27.530 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:27.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5674001850 con 0x7f5674004150 2024-02-19T21:07:27.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:27.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5686ffd700 10 client.48235.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f5674003fe0 osd=1 4 2024-02-19T21:07:27.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 1 4 2024-02-19T21:07:27.531 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:07:27.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5697f31700 10 client.48235.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:27.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 5 2024-02-19T21:07:27.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674003630 osd=2 3 2024-02-19T21:07:27.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674003630 osd=2 3 2024-02-19T21:07:27.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674003630 osd=2 3 2024-02-19T21:07:27.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 5 2024-02-19T21:07:27.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f5674003630 osd=2 3 2024-02-19T21:07:27.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 2 5 2024-02-19T21:07:27.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674003630 osd=2 4 2024-02-19T21:07:27.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5697f31700 10 client.48235.objecter _send_command 5 2024-02-19T21:07:27.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5674003e70 con 0x7f5674011740 2024-02-19T21:07:27.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.535+0000 7f5686ffd700 10 client.48235.objecter ms_handle_connect 0x7f5674011740 2024-02-19T21:07:27.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.536+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5674003e70 con 0x7f5674011740 2024-02-19T21:07:27.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.536+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:07:27.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.536+0000 7f5686ffd700 10 client.48235.objecter _finish_command 5 = system:0 2024-02-19T21:07:27.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.536+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f5674003630 osd=2 4 2024-02-19T21:07:27.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.536+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 2 5 2024-02-19T21:07:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:27.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:27.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.550+0000 7f5697f31700 10 client.48235.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:27.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.550+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 6 2024-02-19T21:07:27.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.550+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674003630 osd=2 3 2024-02-19T21:07:27.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.550+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674003630 osd=2 3 2024-02-19T21:07:27.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.550+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674003630 osd=2 3 2024-02-19T21:07:27.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.550+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 6 2024-02-19T21:07:27.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.550+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f5674003630 osd=2 3 2024-02-19T21:07:27.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.550+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 2 6 2024-02-19T21:07:27.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.550+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674003630 osd=2 4 2024-02-19T21:07:27.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.550+0000 7f5697f31700 10 client.48235.objecter _send_command 6 2024-02-19T21:07:27.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.550+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5674012b30 con 0x7f5674011740 2024-02-19T21:07:27.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.551+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5674012b30 con 0x7f5674011740 2024-02-19T21:07:27.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.551+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:27.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.551+0000 7f5686ffd700 10 client.48235.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.551+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f5674003630 osd=2 4 2024-02-19T21:07:27.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.551+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 2 6 2024-02-19T21:07:27.548 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:07:27.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.552+0000 7f5697f31700 10 client.48235.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:27.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.552+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 7 2024-02-19T21:07:27.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.552+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f56740033c0 osd=3 3 2024-02-19T21:07:27.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.552+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f56740033c0 osd=3 3 2024-02-19T21:07:27.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.552+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f56740033c0 osd=3 3 2024-02-19T21:07:27.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.552+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 7 2024-02-19T21:07:27.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.552+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f56740033c0 osd=3 3 2024-02-19T21:07:27.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.552+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 3 7 2024-02-19T21:07:27.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.552+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f56740033c0 osd=3 4 2024-02-19T21:07:27.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.552+0000 7f5697f31700 10 client.48235.objecter _send_command 7 2024-02-19T21:07:27.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.552+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5674019ef0 con 0x7f56740162a0 2024-02-19T21:07:27.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.552+0000 7f5686ffd700 10 client.48235.objecter ms_handle_connect 0x7f56740162a0 2024-02-19T21:07:27.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.552+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5674019ef0 con 0x7f56740162a0 2024-02-19T21:07:27.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.552+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:07:27.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.552+0000 7f5686ffd700 10 client.48235.objecter _finish_command 7 = system:0 2024-02-19T21:07:27.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.553+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f56740033c0 osd=3 4 2024-02-19T21:07:27.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.553+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 3 7 2024-02-19T21:07:27.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.567+0000 7f5697f31700 10 client.48235.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:27.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.567+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 8 2024-02-19T21:07:27.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.567+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f56740033c0 osd=3 3 2024-02-19T21:07:27.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.567+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f56740033c0 osd=3 3 2024-02-19T21:07:27.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.567+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f56740033c0 osd=3 3 2024-02-19T21:07:27.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.567+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 8 2024-02-19T21:07:27.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.567+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f56740033c0 osd=3 3 2024-02-19T21:07:27.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.567+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 3 8 2024-02-19T21:07:27.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.567+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f56740033c0 osd=3 4 2024-02-19T21:07:27.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.567+0000 7f5697f31700 10 client.48235.objecter _send_command 8 2024-02-19T21:07:27.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.567+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f56740125a0 con 0x7f56740162a0 2024-02-19T21:07:27.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.568+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f56740125a0 con 0x7f56740162a0 2024-02-19T21:07:27.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.568+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:27.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.568+0000 7f5686ffd700 10 client.48235.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.568+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f56740033c0 osd=3 4 2024-02-19T21:07:27.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.568+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 3 8 2024-02-19T21:07:27.565 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:07:27.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.568+0000 7f5697f31700 10 client.48235.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:27.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.568+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 9 2024-02-19T21:07:27.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.569+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674012360 osd=4 3 2024-02-19T21:07:27.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.569+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674012360 osd=4 3 2024-02-19T21:07:27.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.569+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674012360 osd=4 3 2024-02-19T21:07:27.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.569+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 9 2024-02-19T21:07:27.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.569+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f5674012360 osd=4 3 2024-02-19T21:07:27.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.569+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 4 9 2024-02-19T21:07:27.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.569+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674012360 osd=4 4 2024-02-19T21:07:27.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.569+0000 7f5697f31700 10 client.48235.objecter _send_command 9 2024-02-19T21:07:27.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.569+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f56740124d0 con 0x7f56740112a0 2024-02-19T21:07:27.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.569+0000 7f5686ffd700 10 client.48235.objecter ms_handle_connect 0x7f56740112a0 2024-02-19T21:07:27.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.570+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f56740124d0 con 0x7f56740112a0 2024-02-19T21:07:27.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.570+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:07:27.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.570+0000 7f5686ffd700 10 client.48235.objecter _finish_command 9 = system:0 2024-02-19T21:07:27.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.570+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f5674012360 osd=4 4 2024-02-19T21:07:27.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.570+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 4 9 2024-02-19T21:07:27.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.584+0000 7f5697f31700 10 client.48235.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:27.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.584+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 10 2024-02-19T21:07:27.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.584+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674012360 osd=4 3 2024-02-19T21:07:27.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.584+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674012360 osd=4 3 2024-02-19T21:07:27.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.584+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674012360 osd=4 3 2024-02-19T21:07:27.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.584+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 10 2024-02-19T21:07:27.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.584+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f5674012360 osd=4 3 2024-02-19T21:07:27.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.584+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 4 10 2024-02-19T21:07:27.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.584+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674012360 osd=4 4 2024-02-19T21:07:27.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.584+0000 7f5697f31700 10 client.48235.objecter _send_command 10 2024-02-19T21:07:27.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.584+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f56740124d0 con 0x7f56740112a0 2024-02-19T21:07:27.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f56740124d0 con 0x7f56740112a0 2024-02-19T21:07:27.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:27.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5686ffd700 10 client.48235.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f5674012360 osd=4 4 2024-02-19T21:07:27.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 4 10 2024-02-19T21:07:27.583 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:07:27.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5697f31700 10 client.48235.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:27.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 11 2024-02-19T21:07:27.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f56740124d0 osd=5 3 2024-02-19T21:07:27.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f56740124d0 osd=5 3 2024-02-19T21:07:27.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f56740124d0 osd=5 3 2024-02-19T21:07:27.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 11 2024-02-19T21:07:27.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f56740124d0 osd=5 3 2024-02-19T21:07:27.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 5 11 2024-02-19T21:07:27.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f56740124d0 osd=5 4 2024-02-19T21:07:27.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5697f31700 10 client.48235.objecter _send_command 11 2024-02-19T21:07:27.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.586+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f567401b270 con 0x7f567401acb0 2024-02-19T21:07:27.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.587+0000 7f5686ffd700 10 client.48235.objecter ms_handle_connect 0x7f567401acb0 2024-02-19T21:07:27.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.587+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f567401b270 con 0x7f567401acb0 2024-02-19T21:07:27.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.587+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:07:27.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.587+0000 7f5686ffd700 10 client.48235.objecter _finish_command 11 = system:0 2024-02-19T21:07:27.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.588+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f56740124d0 osd=5 4 2024-02-19T21:07:27.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.588+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 5 11 2024-02-19T21:07:27.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.602+0000 7f5697f31700 10 client.48235.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:27.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.602+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 12 2024-02-19T21:07:27.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.602+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f56740124d0 osd=5 3 2024-02-19T21:07:27.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.602+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f56740124d0 osd=5 3 2024-02-19T21:07:27.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.602+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f56740124d0 osd=5 3 2024-02-19T21:07:27.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.602+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 12 2024-02-19T21:07:27.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.602+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f56740124d0 osd=5 3 2024-02-19T21:07:27.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.602+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 5 12 2024-02-19T21:07:27.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.602+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f56740124d0 osd=5 4 2024-02-19T21:07:27.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.602+0000 7f5697f31700 10 client.48235.objecter _send_command 12 2024-02-19T21:07:27.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.602+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f567401b270 con 0x7f567401acb0 2024-02-19T21:07:27.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f567401b270 con 0x7f567401acb0 2024-02-19T21:07:27.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:27.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5686ffd700 10 client.48235.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f56740124d0 osd=5 4 2024-02-19T21:07:27.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 5 12 2024-02-19T21:07:27.600 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:07:27.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5697f31700 10 client.48235.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:27.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 13 2024-02-19T21:07:27.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f567401b270 osd=6 3 2024-02-19T21:07:27.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f567401b270 osd=6 3 2024-02-19T21:07:27.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f567401b270 osd=6 3 2024-02-19T21:07:27.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 13 2024-02-19T21:07:27.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f567401b270 osd=6 3 2024-02-19T21:07:27.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 6 13 2024-02-19T21:07:27.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f567401b270 osd=6 4 2024-02-19T21:07:27.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5697f31700 10 client.48235.objecter _send_command 13 2024-02-19T21:07:27.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.603+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5674027000 con 0x7f56740234f0 2024-02-19T21:07:27.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.604+0000 7f5686ffd700 10 client.48235.objecter ms_handle_connect 0x7f56740234f0 2024-02-19T21:07:27.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.604+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5674027000 con 0x7f56740234f0 2024-02-19T21:07:27.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.604+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:07:27.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.604+0000 7f5686ffd700 10 client.48235.objecter _finish_command 13 = system:0 2024-02-19T21:07:27.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.604+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f567401b270 osd=6 4 2024-02-19T21:07:27.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.604+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 6 13 2024-02-19T21:07:27.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.619+0000 7f5697f31700 10 client.48235.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:27.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.619+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 14 2024-02-19T21:07:27.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.619+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f567401b270 osd=6 3 2024-02-19T21:07:27.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.619+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f567401b270 osd=6 3 2024-02-19T21:07:27.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.619+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f567401b270 osd=6 3 2024-02-19T21:07:27.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.619+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 14 2024-02-19T21:07:27.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.619+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f567401b270 osd=6 3 2024-02-19T21:07:27.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.619+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 6 14 2024-02-19T21:07:27.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.619+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f567401b270 osd=6 4 2024-02-19T21:07:27.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.619+0000 7f5697f31700 10 client.48235.objecter _send_command 14 2024-02-19T21:07:27.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.619+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f56740287b0 con 0x7f56740234f0 2024-02-19T21:07:27.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f56740287b0 con 0x7f56740234f0 2024-02-19T21:07:27.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:27.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5686ffd700 10 client.48235.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f567401b270 osd=6 4 2024-02-19T21:07:27.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 6 14 2024-02-19T21:07:27.617 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:07:27.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5697f31700 10 client.48235.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:27.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 15 2024-02-19T21:07:27.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674028610 osd=7 3 2024-02-19T21:07:27.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674028610 osd=7 3 2024-02-19T21:07:27.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674028610 osd=7 3 2024-02-19T21:07:27.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 15 2024-02-19T21:07:27.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f5674028610 osd=7 3 2024-02-19T21:07:27.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 7 15 2024-02-19T21:07:27.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674028610 osd=7 4 2024-02-19T21:07:27.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5697f31700 10 client.48235.objecter _send_command 15 2024-02-19T21:07:27.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f56740280c0 con 0x7f5674028d00 2024-02-19T21:07:27.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.621+0000 7f5686ffd700 10 client.48235.objecter ms_handle_connect 0x7f5674028d00 2024-02-19T21:07:27.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.622+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f56740280c0 con 0x7f5674028d00 2024-02-19T21:07:27.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.622+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:07:27.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.622+0000 7f5686ffd700 10 client.48235.objecter _finish_command 15 = system:0 2024-02-19T21:07:27.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.622+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f5674028610 osd=7 4 2024-02-19T21:07:27.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.622+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 7 15 2024-02-19T21:07:27.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.636+0000 7f5697f31700 10 client.48235.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:27.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.636+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign -1 16 2024-02-19T21:07:27.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.636+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674028610 osd=7 3 2024-02-19T21:07:27.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.636+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674028610 osd=7 3 2024-02-19T21:07:27.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.636+0000 7f5697f31700 20 client.48235.objecter _get_session s=0x7f5674028610 osd=7 3 2024-02-19T21:07:27.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.636+0000 7f5697f31700 15 client.48235.objecter _session_command_op_remove -1 16 2024-02-19T21:07:27.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.636+0000 7f5697f31700 20 client.48235.objecter get_session s=0x7f5674028610 osd=7 3 2024-02-19T21:07:27.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.636+0000 7f5697f31700 15 client.48235.objecter _session_command_op_assign 7 16 2024-02-19T21:07:27.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.636+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674028610 osd=7 4 2024-02-19T21:07:27.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.636+0000 7f5697f31700 10 client.48235.objecter _send_command 16 2024-02-19T21:07:27.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.636+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5674028080 con 0x7f5674028d00 2024-02-19T21:07:27.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.637+0000 7f5686ffd700 1 -- 172.21.15.163:0/867658936 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5674028080 con 0x7f5674028d00 2024-02-19T21:07:27.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.637+0000 7f5686ffd700 10 client.48235.objecter ms_dispatch 0x7f5690001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:27.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.637+0000 7f5686ffd700 10 client.48235.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.637+0000 7f5686ffd700 20 client.48235.objecter put_session s=0x7f5674028610 osd=7 4 2024-02-19T21:07:27.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.637+0000 7f5686ffd700 15 client.48235.objecter _session_command_op_remove 7 16 2024-02-19T21:07:27.634 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:07:27.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 10 librados: watch_flush enter 2024-02-19T21:07:27.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 10 librados: watch_flush exit 2024-02-19T21:07:27.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 10 client.48235.objecter close_session for osd.0 2024-02-19T21:07:27.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 >> v1:172.21.15.163:6801/1559418115 conn(0x7f5674004530 legacy=0x7f5674002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:27.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f56740051a0 osd=0 1 2024-02-19T21:07:27.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 10 client.48235.objecter close_session for osd.1 2024-02-19T21:07:27.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 >> v1:172.21.15.163:6805/3125047432 conn(0x7f5674004150 legacy=0x7f567400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:27.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674003fe0 osd=1 1 2024-02-19T21:07:27.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 10 client.48235.objecter close_session for osd.2 2024-02-19T21:07:27.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 >> v1:172.21.15.163:6809/1563158487 conn(0x7f5674011740 legacy=0x7f5674011b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:27.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674003630 osd=2 1 2024-02-19T21:07:27.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 10 client.48235.objecter close_session for osd.3 2024-02-19T21:07:27.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 >> v1:172.21.15.163:6813/2112279769 conn(0x7f56740162a0 legacy=0x7f5674018690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:27.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f56964ce700 1 -- 172.21.15.163:0/867658936 reap_dead start 2024-02-19T21:07:27.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f56740033c0 osd=3 1 2024-02-19T21:07:27.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 10 client.48235.objecter close_session for osd.4 2024-02-19T21:07:27.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 >> v1:172.21.15.193:6800/3129716965 conn(0x7f56740112a0 legacy=0x7f567401a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:27.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f56964ce700 1 -- 172.21.15.163:0/867658936 reap_dead start 2024-02-19T21:07:27.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674012360 osd=4 1 2024-02-19T21:07:27.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 10 client.48235.objecter close_session for osd.5 2024-02-19T21:07:27.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 >> v1:172.21.15.193:6804/805161991 conn(0x7f567401acb0 legacy=0x7f56740218c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:27.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f56740124d0 osd=5 1 2024-02-19T21:07:27.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 10 client.48235.objecter close_session for osd.6 2024-02-19T21:07:27.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 >> v1:172.21.15.193:6808/787604615 conn(0x7f56740234f0 legacy=0x7f56740258e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:27.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f567401b270 osd=6 1 2024-02-19T21:07:27.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 10 client.48235.objecter close_session for osd.7 2024-02-19T21:07:27.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 >> v1:172.21.15.193:6812/157927432 conn(0x7f5674028d00 legacy=0x7f5674029180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:27.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 20 client.48235.objecter put_session s=0x7f5674028610 osd=7 1 2024-02-19T21:07:27.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 20 client.48235.objecter shutdown clearing up homeless session... 2024-02-19T21:07:27.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 10 client.48235.objecter successfully canceled tick 2024-02-19T21:07:27.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 >> v1:172.21.15.163:6800/1552514681 conn(0x7f567c063c80 legacy=0x7f567c066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:27.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 >> v1:172.21.15.193:6789/0 conn(0x7f56900fd980 legacy=0x7f56900f9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:27.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f56964ce700 1 -- 172.21.15.163:0/867658936 reap_dead start 2024-02-19T21:07:27.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 shutdown_connections 2024-02-19T21:07:27.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 >> 172.21.15.163:0/867658936 conn(0x7f56900f35a0 msgr2=0x7f5690107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:27.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 shutdown_connections 2024-02-19T21:07:27.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 1 -- 172.21.15.163:0/867658936 wait complete. 2024-02-19T21:07:27.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:27.638+0000 7f5697f31700 1 librados: shutdown 2024-02-19T21:07:27.638 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.638 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.638 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.638 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.638 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.638 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.638 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.638 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:07:27.639 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:27.663 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:25.732086+0000 osd.1 (osd.1) 135 : cluster [DBG] 16.9 scrub starts 2024-02-19T21:07:27.663 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:25.734983+0000 osd.1 (osd.1) 136 : cluster [DBG] 16.9 scrub ok 2024-02-19T21:07:27.663 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:26.885657+0000 osd.3 (osd.3) 131 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:07:27.663 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:26.886812+0000 osd.3 (osd.3) 132 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:07:27.663 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:27.204054+0000 mon.a (mon.0) 1859 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:27.663 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:27.213468+0000 mon.a (mon.0) 1860 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31"}]': finished 2024-02-19T21:07:27.663 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:27.213591+0000 mon.a (mon.0) 1861 : audit [INF] from='client.49412 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi163-83888-3","pool2":"test-rados-api-smithi163-83888-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-02-19T21:07:27.663 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:27.213670+0000 mon.a (mon.0) 1862 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi163-83854-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:27.663 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:27.213742+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-02-19T21:07:27.664 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:27.220208+0000 mon.a (mon.0) 1864 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi163-83854-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi163-83854-48"}]: dispatch 2024-02-19T21:07:27.664 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:27.226859+0000 mon.c (mon.2) 329 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:27.664 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:27.227337+0000 mon.a (mon.0) 1865 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:27.665 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:27.683 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:27.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:27.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:27.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:25.732086+0000 osd.1 (osd.1) 135 : cluster [DBG] 16.9 scrub starts 2024-02-19T21:07:27.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:25.734983+0000 osd.1 (osd.1) 136 : cluster [DBG] 16.9 scrub ok 2024-02-19T21:07:27.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:26.885657+0000 osd.3 (osd.3) 131 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:07:27.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:26.886812+0000 osd.3 (osd.3) 132 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:07:27.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:27.204054+0000 mon.a (mon.0) 1859 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:27.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:27.213468+0000 mon.a (mon.0) 1860 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31"}]': finished 2024-02-19T21:07:27.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:27.213591+0000 mon.a (mon.0) 1861 : audit [INF] from='client.49412 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi163-83888-3","pool2":"test-rados-api-smithi163-83888-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-02-19T21:07:27.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:27.213670+0000 mon.a (mon.0) 1862 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi163-83854-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:27.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:27.213742+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-02-19T21:07:27.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:27.220208+0000 mon.a (mon.0) 1864 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi163-83854-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi163-83854-48"}]: dispatch 2024-02-19T21:07:27.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:27.226859+0000 mon.c (mon.2) 329 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:27.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:27.227337+0000 mon.a (mon.0) 1865 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:27.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:25.732086+0000 osd.1 (osd.1) 135 : cluster [DBG] 16.9 scrub starts 2024-02-19T21:07:27.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:25.734983+0000 osd.1 (osd.1) 136 : cluster [DBG] 16.9 scrub ok 2024-02-19T21:07:27.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:26.885657+0000 osd.3 (osd.3) 131 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:07:27.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:26.886812+0000 osd.3 (osd.3) 132 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:07:27.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:27.204054+0000 mon.a (mon.0) 1859 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:27.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:27.213468+0000 mon.a (mon.0) 1860 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-31"}]': finished 2024-02-19T21:07:27.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:27.213591+0000 mon.a (mon.0) 1861 : audit [INF] from='client.49412 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi163-83888-3","pool2":"test-rados-api-smithi163-83888-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-02-19T21:07:27.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:27.213670+0000 mon.a (mon.0) 1862 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi163-83854-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:27.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:27.213742+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-02-19T21:07:27.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:27.220208+0000 mon.a (mon.0) 1864 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi163-83854-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi163-83854-48"}]: dispatch 2024-02-19T21:07:27.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:27.226859+0000 mon.c (mon.2) 329 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:27.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:27.227337+0000 mon.a (mon.0) 1865 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:27.787 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:07:27.813 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:27.831 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:07:27.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:27.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:07:27.934 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:27.965 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:28.067 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:07:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:07:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:07:28.514 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:07:28.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:27.164480+0000 mgr.y (mgr.14550) 276 : cluster [DBG] pgmap v371: 278 pgs: 40 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:07:28.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:27.498606+0000 mon.b (mon.1) 413 : audit [DBG] from='client.? v1:172.21.15.163:0/867658936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:28.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:28.449988+0000 mon.a (mon.0) 1866 : audit [INF] from='client.49400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi163-83845-38"}]': finished 2024-02-19T21:07:28.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:28.450101+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-02-19T21:07:28.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:28.450627+0000 mon.c (mon.2) 330 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:28.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:28.450956+0000 mon.a (mon.0) 1868 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:28.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:27.164480+0000 mgr.y (mgr.14550) 276 : cluster [DBG] pgmap v371: 278 pgs: 40 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:07:28.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:27.498606+0000 mon.b (mon.1) 413 : audit [DBG] from='client.? v1:172.21.15.163:0/867658936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:28.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:28.449988+0000 mon.a (mon.0) 1866 : audit [INF] from='client.49400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi163-83845-38"}]': finished 2024-02-19T21:07:28.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:28.450101+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-02-19T21:07:28.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:28.450627+0000 mon.c (mon.2) 330 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:28.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:28.450956+0000 mon.a (mon.0) 1868 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:28.771 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:07:28.772 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:07:28.772 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:07:28.772 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd out 3 2024-02-19T21:07:28.914 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:28.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:28.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:29.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:07:29.047 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:27.164480+0000 mgr.y (mgr.14550) 276 : cluster [DBG] pgmap v371: 278 pgs: 40 unknown, 238 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:07:29.047 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:27.498606+0000 mon.b (mon.1) 413 : audit [DBG] from='client.? v1:172.21.15.163:0/867658936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:29.047 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:28.449988+0000 mon.a (mon.0) 1866 : audit [INF] from='client.49400 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi163-83845-38"}]': finished 2024-02-19T21:07:29.047 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:28.450101+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-02-19T21:07:29.047 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:28.450627+0000 mon.c (mon.2) 330 : audit [INF] from='client.? v1:172.21.15.163:0/896457325' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:29.047 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:28.450956+0000 mon.a (mon.0) 1868 : audit [INF] from='client.49400 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi163-83845-38"}]: dispatch 2024-02-19T21:07:29.049 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:29.070 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:07:29.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:29.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:07:29.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:29.206 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:29.238 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:29.241 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:29.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:29.342 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:29.368 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:29.387 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:29.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:29.490 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:29.505 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:29.517 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:29.535 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:29.537 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:29.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:29.639 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:29.666 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:29.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:28.720857+0000 osd.1 (osd.1) 137 : cluster [DBG] 61.b scrub starts 2024-02-19T21:07:29.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:28.724330+0000 osd.1 (osd.1) 138 : cluster [DBG] 61.b scrub ok 2024-02-19T21:07:29.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:28.856438+0000 osd.7 (osd.7) 131 : cluster [DBG] 61.15 scrub starts 2024-02-19T21:07:29.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:28.859884+0000 osd.7 (osd.7) 132 : cluster [DBG] 61.15 scrub ok 2024-02-19T21:07:29.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:29.165532+0000 mon.b (mon.1) 414 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "40"}]: dispatch 2024-02-19T21:07:29.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:29.166359+0000 mon.a (mon.0) 1869 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "40"}]: dispatch 2024-02-19T21:07:29.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:29.214155+0000 mon.a (mon.0) 1870 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi163-83854-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi163-83854-48"}]': finished 2024-02-19T21:07:29.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:29.214260+0000 mon.a (mon.0) 1871 : audit [INF] from='client.49400 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi163-83845-38"}]': finished 2024-02-19T21:07:29.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:29.214360+0000 mon.a (mon.0) 1872 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "40"}]': finished 2024-02-19T21:07:29.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:29.214414+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-02-19T21:07:29.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:29.232706+0000 mon.b (mon.1) 415 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:29.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:29.233110+0000 mon.a (mon.0) 1874 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:29.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:29.243914+0000 mon.a (mon.0) 1875 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi163-83845-39"}]: dispatch 2024-02-19T21:07:29.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:29.244904+0000 mon.a (mon.0) 1876 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi163-83845-39"}]: dispatch 2024-02-19T21:07:29.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:29.245194+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi163-83845-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:29.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:28.720857+0000 osd.1 (osd.1) 137 : cluster [DBG] 61.b scrub starts 2024-02-19T21:07:29.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:28.724330+0000 osd.1 (osd.1) 138 : cluster [DBG] 61.b scrub ok 2024-02-19T21:07:29.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:28.856438+0000 osd.7 (osd.7) 131 : cluster [DBG] 61.15 scrub starts 2024-02-19T21:07:29.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:28.859884+0000 osd.7 (osd.7) 132 : cluster [DBG] 61.15 scrub ok 2024-02-19T21:07:29.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:29.165532+0000 mon.b (mon.1) 414 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "40"}]: dispatch 2024-02-19T21:07:29.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:29.166359+0000 mon.a (mon.0) 1869 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "40"}]: dispatch 2024-02-19T21:07:29.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:29.214155+0000 mon.a (mon.0) 1870 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi163-83854-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi163-83854-48"}]': finished 2024-02-19T21:07:29.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:29.214260+0000 mon.a (mon.0) 1871 : audit [INF] from='client.49400 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi163-83845-38"}]': finished 2024-02-19T21:07:29.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:29.214360+0000 mon.a (mon.0) 1872 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "40"}]': finished 2024-02-19T21:07:29.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:29.214414+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-02-19T21:07:29.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:29.232706+0000 mon.b (mon.1) 415 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:29.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:29.233110+0000 mon.a (mon.0) 1874 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:29.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:29.243914+0000 mon.a (mon.0) 1875 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi163-83845-39"}]: dispatch 2024-02-19T21:07:29.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:29.244904+0000 mon.a (mon.0) 1876 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi163-83845-39"}]: dispatch 2024-02-19T21:07:29.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:29.245194+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi163-83845-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:29.764 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:29.771 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:29.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:29.872 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:29.901 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:30.024 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:30.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:30.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:28.720857+0000 osd.1 (osd.1) 137 : cluster [DBG] 61.b scrub starts 2024-02-19T21:07:30.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:28.724330+0000 osd.1 (osd.1) 138 : cluster [DBG] 61.b scrub ok 2024-02-19T21:07:30.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:28.856438+0000 osd.7 (osd.7) 131 : cluster [DBG] 61.15 scrub starts 2024-02-19T21:07:30.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:28.859884+0000 osd.7 (osd.7) 132 : cluster [DBG] 61.15 scrub ok 2024-02-19T21:07:30.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:29.165532+0000 mon.b (mon.1) 414 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "40"}]: dispatch 2024-02-19T21:07:30.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:29.166359+0000 mon.a (mon.0) 1869 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "40"}]: dispatch 2024-02-19T21:07:30.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:29.214155+0000 mon.a (mon.0) 1870 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi163-83854-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi163-83854-48"}]': finished 2024-02-19T21:07:30.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:29.214260+0000 mon.a (mon.0) 1871 : audit [INF] from='client.49400 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi163-83845-38"}]': finished 2024-02-19T21:07:30.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:29.214360+0000 mon.a (mon.0) 1872 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "40"}]': finished 2024-02-19T21:07:30.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:29.214414+0000 mon.a (mon.0) 1873 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-02-19T21:07:30.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:29.232706+0000 mon.b (mon.1) 415 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:30.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:29.233110+0000 mon.a (mon.0) 1874 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:30.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:29.243914+0000 mon.a (mon.0) 1875 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi163-83845-39"}]: dispatch 2024-02-19T21:07:30.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:29.244904+0000 mon.a (mon.0) 1876 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi163-83845-39"}]: dispatch 2024-02-19T21:07:30.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:29.245194+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi163-83845-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:30.127 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:30.156 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:30.176 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:30.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:30.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:30.279 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:30.311 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:30.339 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:07:30.339 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:07:30.442 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:07:30.567 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:07:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:07:30.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:29.165018+0000 mgr.y (mgr.14550) 277 : cluster [DBG] pgmap v374: 205 pgs: 205 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:07:30.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:29.944854+0000 osd.3 (osd.3) 133 : cluster [DBG] 16.7 scrub starts 2024-02-19T21:07:30.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:29.946010+0000 osd.3 (osd.3) 134 : cluster [DBG] 16.7 scrub ok 2024-02-19T21:07:30.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:30.215598+0000 mon.a (mon.0) 1878 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:30.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:30.215682+0000 mon.a (mon.0) 1879 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi163-83845-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:30.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:30.215736+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-02-19T21:07:30.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:30.223014+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi163-83845-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi163-83845-39"}]: dispatch 2024-02-19T21:07:30.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:30.225313+0000 mon.b (mon.1) 416 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:30.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:30.225877+0000 mon.a (mon.0) 1882 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:30.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:29.165018+0000 mgr.y (mgr.14550) 277 : cluster [DBG] pgmap v374: 205 pgs: 205 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:07:30.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:29.944854+0000 osd.3 (osd.3) 133 : cluster [DBG] 16.7 scrub starts 2024-02-19T21:07:30.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:29.946010+0000 osd.3 (osd.3) 134 : cluster [DBG] 16.7 scrub ok 2024-02-19T21:07:30.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:30.215598+0000 mon.a (mon.0) 1878 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:30.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:30.215682+0000 mon.a (mon.0) 1879 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi163-83845-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:30.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:30.215736+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-02-19T21:07:30.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:30.223014+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi163-83845-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi163-83845-39"}]: dispatch 2024-02-19T21:07:30.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:30.225313+0000 mon.b (mon.1) 416 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:30.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:30.225877+0000 mon.a (mon.0) 1882 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:29.165018+0000 mgr.y (mgr.14550) 277 : cluster [DBG] pgmap v374: 205 pgs: 205 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:07:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:29.944854+0000 osd.3 (osd.3) 133 : cluster [DBG] 16.7 scrub starts 2024-02-19T21:07:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:29.946010+0000 osd.3 (osd.3) 134 : cluster [DBG] 16.7 scrub ok 2024-02-19T21:07:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:30.215598+0000 mon.a (mon.0) 1878 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:30.215682+0000 mon.a (mon.0) 1879 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi163-83845-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:30.215736+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-02-19T21:07:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:30.223014+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi163-83845-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi163-83845-39"}]: dispatch 2024-02-19T21:07:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:30.225313+0000 mon.b (mon.1) 416 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:31.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:30.225877+0000 mon.a (mon.0) 1882 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:31.259 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:31.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:31.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:31.363 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:31.390 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:31.515 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:31.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:31.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:31.619 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:31.651 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:31.672 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:31.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:31.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:31.776 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:31.805 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:31.825 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:31.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:31.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:31.929 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:31.956 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:30.232165+0000 mgr.y (mgr.14550) 278 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:31.956 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:30.768136+0000 osd.4 (osd.4) 127 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:07:31.956 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:30.769382+0000 osd.4 (osd.4) 128 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:07:31.956 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:30.777408+0000 osd.2 (osd.2) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:31.956 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:30.777566+0000 osd.2 (osd.2) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:31.956 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:31.463817+0000 mon.a (mon.0) 1883 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:31.956 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:31.464190+0000 mon.a (mon.0) 1884 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-02-19T21:07:31.956 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:31.465151+0000 mon.a (mon.0) 1885 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi163-83854-48"}]: dispatch 2024-02-19T21:07:31.956 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:31.465256+0000 mon.b (mon.1) 417 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:31.956 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:31.465541+0000 mon.a (mon.0) 1886 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:31.958 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:31.977 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:31.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:30.232165+0000 mgr.y (mgr.14550) 278 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:30.768136+0000 osd.4 (osd.4) 127 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:07:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:30.769382+0000 osd.4 (osd.4) 128 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:07:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:30.777408+0000 osd.2 (osd.2) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:30.777566+0000 osd.2 (osd.2) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:31.463817+0000 mon.a (mon.0) 1883 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:31.464190+0000 mon.a (mon.0) 1884 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-02-19T21:07:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:31.465151+0000 mon.a (mon.0) 1885 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi163-83854-48"}]: dispatch 2024-02-19T21:07:31.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:31.465256+0000 mon.b (mon.1) 417 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:31.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:31.465541+0000 mon.a (mon.0) 1886 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:31.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:30.232165+0000 mgr.y (mgr.14550) 278 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:31.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:30.768136+0000 osd.4 (osd.4) 127 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:07:31.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:30.769382+0000 osd.4 (osd.4) 128 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:07:31.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:30.777408+0000 osd.2 (osd.2) 109 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:07:31.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:30.777566+0000 osd.2 (osd.2) 110 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:07:31.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:31.463817+0000 mon.a (mon.0) 1883 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:31.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:31.464190+0000 mon.a (mon.0) 1884 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-02-19T21:07:31.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:31.465151+0000 mon.a (mon.0) 1885 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi163-83854-48"}]: dispatch 2024-02-19T21:07:31.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:31.465256+0000 mon.b (mon.1) 417 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:31.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:31.465541+0000 mon.a (mon.0) 1886 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:32.081 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:07:32.110 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:32.182 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-19T21:07:32.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.929+0000 7f7feef71700 1 -- 172.21.15.163:0/744933734 >> v1:172.21.15.163:6789/0 conn(0x7f7fe80ff670 legacy=0x7f7fe8101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:32.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.929+0000 7f7feef71700 1 -- 172.21.15.163:0/744933734 shutdown_connections 2024-02-19T21:07:32.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.929+0000 7f7feef71700 1 -- 172.21.15.163:0/744933734 >> 172.21.15.163:0/744933734 conn(0x7f7fe80f2470 msgr2=0x7f7fe80f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.929+0000 7f7feef71700 1 -- 172.21.15.163:0/744933734 shutdown_connections 2024-02-19T21:07:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.929+0000 7f7feef71700 1 -- 172.21.15.163:0/744933734 wait complete. 2024-02-19T21:07:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.929+0000 7f7feef71700 1 librados: starting msgr at 2024-02-19T21:07:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.929+0000 7f7feef71700 1 librados: starting objecter 2024-02-19T21:07:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.929+0000 7f7feef71700 1 Processor -- start 2024-02-19T21:07:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.929+0000 7f7feef71700 1 -- start start 2024-02-19T21:07:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.929+0000 7f7feef71700 1 librados: setting wanted keys 2024-02-19T21:07:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.929+0000 7f7feef71700 1 librados: calling monclient init 2024-02-19T21:07:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7feef71700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7fe8117280 con 0x7f7fe80ff670 2024-02-19T21:07:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7feef71700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7fe8118460 con 0x7f7fe80fc870 2024-02-19T21:07:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7feef71700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7fe8119660 con 0x7f7fe81031b0 2024-02-19T21:07:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe7fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f7fe80ff670 0x7f7fe80fa120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:41858/0 (socket says 172.21.15.163:41858) 2024-02-19T21:07:32.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe7fff700 1 -- 172.21.15.163:0/3777293487 learned_addr learned my addr 172.21.15.163:0/3777293487 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 10 client.?.objecter ms_handle_connect 0x7f7fe80ff670 2024-02-19T21:07:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 10 client.?.objecter ms_handle_connect 0x7f7fe81031b0 2024-02-19T21:07:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 10 client.?.objecter ms_handle_connect 0x7f7fe80fc870 2024-02-19T21:07:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545982584 0 0) 0x7f7fe8117280 con 0x7f7fe80ff670 2024-02-19T21:07:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd8003800 con 0x7f7fe80ff670 2024-02-19T21:07:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012296287 0 0) 0x7f7fe8119660 con 0x7f7fe81031b0 2024-02-19T21:07:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe8117280 con 0x7f7fe81031b0 2024-02-19T21:07:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536559371 0 0) 0x7f7fe8118460 con 0x7f7fe80fc870 2024-02-19T21:07:32.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe8119660 con 0x7f7fe80fc870 2024-02-19T21:07:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2127976594 0 0) 0x7f7fd8003800 con 0x7f7fe80ff670 2024-02-19T21:07:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fe8118460 con 0x7f7fe80ff670 2024-02-19T21:07:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7fd4002ea0 con 0x7f7fe80ff670 2024-02-19T21:07:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 819966878 0 0) 0x7f7fe8117280 con 0x7f7fe81031b0 2024-02-19T21:07:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fd8003800 con 0x7f7fe81031b0 2024-02-19T21:07:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7fe0001c40 con 0x7f7fe81031b0 2024-02-19T21:07:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1404326081 0 0) 0x7f7fe8119660 con 0x7f7fe80fc870 2024-02-19T21:07:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fe8117280 con 0x7f7fe80fc870 2024-02-19T21:07:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7fdc0018d0 con 0x7f7fe80fc870 2024-02-19T21:07:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2192119831 0 0) 0x7f7fe8118460 con 0x7f7fe80ff670 2024-02-19T21:07:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 >> v1:172.21.15.163:6790/0 conn(0x7f7fe81031b0 legacy=0x7f7fe8115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 >> v1:172.21.15.193:6789/0 conn(0x7f7fe80fc870 legacy=0x7f7fe80f9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.930+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe811a860 con 0x7f7fe80ff670 2024-02-19T21:07:32.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.931+0000 7f7feef71700 1 -- 172.21.15.163:0/3777293487 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe8119850 con 0x7f7fe80ff670 2024-02-19T21:07:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.931+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f7fd4002050 con 0x7f7fe80ff670 2024-02-19T21:07:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.931+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7fd4003b50 con 0x7f7fe80ff670 2024-02-19T21:07:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.931+0000 7f7feef71700 10 client.22734.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.931+0000 7f7feef71700 1 -- 172.21.15.163:0/3777293487 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe8119d40 con 0x7f7fe80ff670 2024-02-19T21:07:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.931+0000 7f7feef71700 1 librados: init done 2024-02-19T21:07:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.932+0000 7f7feef71700 1 -- 172.21.15.163:0/3777293487 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe805f700 con 0x7f7fe80ff670 2024-02-19T21:07:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.932+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f7fd4003140 con 0x7f7fe80ff670 2024-02-19T21:07:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.933+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(271..271 src has 43..271) v4 ==== 8790+0+0 (unknown 1343877319 0 0) 0x7f7fd40794f0 con 0x7f7fe80ff670 2024-02-19T21:07:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.933+0000 7f7fe5ffb700 10 client.22734.objecter ms_dispatch 0x7f7fe8001550 osd_map(271..271 src has 43..271) v4 2024-02-19T21:07:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.933+0000 7f7fe5ffb700 3 client.22734.objecter handle_osd_map got epochs [271,271] > 0 2024-02-19T21:07:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.933+0000 7f7fe5ffb700 3 client.22734.objecter handle_osd_map decoding full epoch 271 2024-02-19T21:07:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.933+0000 7f7fe5ffb700 20 client.22734.objecter dump_active .. 0 homeless 2024-02-19T21:07:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.933+0000 7f7fe5ffb700 10 client.22734.objecter ms_handle_connect 0x7f7fd8063cc0 2024-02-19T21:07:32.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:31.937+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f7fd404c6d0 con 0x7f7fe80ff670 2024-02-19T21:07:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:32.115+0000 7f7feef71700 1 -- 172.21.15.163:0/3777293487 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f7fe8108d90 con 0x7f7fe80ff670 2024-02-19T21:07:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:32.218+0000 7f7fe5ffb700 1 -- 172.21.15.163:0/3777293487 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v272) v1 ==== 101+0+0 (unknown 2710911002 0 0) 0x7f7fd404ab40 con 0x7f7fe80ff670 2024-02-19T21:07:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:32.222+0000 7f7feef71700 10 librados: watch_flush enter 2024-02-19T21:07:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:32.223+0000 7f7feef71700 10 librados: watch_flush exit 2024-02-19T21:07:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:32.223+0000 7f7feef71700 20 client.22734.objecter shutdown clearing up homeless session... 2024-02-19T21:07:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:32.223+0000 7f7feef71700 10 client.22734.objecter successfully canceled tick 2024-02-19T21:07:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:32.223+0000 7f7feef71700 1 -- 172.21.15.163:0/3777293487 >> v1:172.21.15.163:6800/1552514681 conn(0x7f7fd8063cc0 legacy=0x7f7fd8066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:32.223+0000 7f7feef71700 1 -- 172.21.15.163:0/3777293487 >> v1:172.21.15.163:6789/0 conn(0x7f7fe80ff670 legacy=0x7f7fe80fa120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:32.226+0000 7f7feef71700 1 -- 172.21.15.163:0/3777293487 shutdown_connections 2024-02-19T21:07:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:32.226+0000 7f7feef71700 1 -- 172.21.15.163:0/3777293487 >> 172.21.15.163:0/3777293487 conn(0x7f7fe80f2470 msgr2=0x7f7fe8106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:32.226+0000 7f7feef71700 1 -- 172.21.15.163:0/3777293487 shutdown_connections 2024-02-19T21:07:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:32.227+0000 7f7feef71700 1 -- 172.21.15.163:0/3777293487 wait complete. 2024-02-19T21:07:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:32.227+0000 7f7feef71700 1 librados: shutdown 2024-02-19T21:07:32.265 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:07:32.452 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:07:32.488 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:32.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:31.165452+0000 mgr.y (mgr.14550) 279 : cluster [DBG] pgmap v377: 245 pgs: 2 peering, 40 creating+peering, 203 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:07:32.912 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:31.720522+0000 osd.1 (osd.1) 139 : cluster [DBG] 61.9 scrub starts 2024-02-19T21:07:32.912 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:31.724130+0000 osd.1 (osd.1) 140 : cluster [DBG] 61.9 scrub ok 2024-02-19T21:07:32.912 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:32.117091+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? v1:172.21.15.163:0/3777293487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:07:32.912 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:32.214723+0000 mon.a (mon.0) 1888 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:07:32.912 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:32.219066+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi163-83845-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi163-83845-39"}]': finished 2024-02-19T21:07:32.912 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:32.219177+0000 mon.a (mon.0) 1890 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi163-83854-48"}]': finished 2024-02-19T21:07:32.912 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:32.219231+0000 mon.a (mon.0) 1891 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]': finished 2024-02-19T21:07:32.912 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:32.219313+0000 mon.a (mon.0) 1892 : audit [INF] from='client.? v1:172.21.15.163:0/3777293487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:07:32.912 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:32.219363+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-02-19T21:07:32.912 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:32.220705+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi163-83854-48"}]: dispatch 2024-02-19T21:07:32.912 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:32.226365+0000 mon.b (mon.1) 418 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-33", "mode": "writeback"}]: dispatch 2024-02-19T21:07:32.913 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:32.228201+0000 mon.a (mon.0) 1895 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-33", "mode": "writeback"}]: dispatch 2024-02-19T21:07:32.913 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:31.165452+0000 mgr.y (mgr.14550) 279 : cluster [DBG] pgmap v377: 245 pgs: 2 peering, 40 creating+peering, 203 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:07:32.913 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:31.720522+0000 osd.1 (osd.1) 139 : cluster [DBG] 61.9 scrub starts 2024-02-19T21:07:32.913 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:31.724130+0000 osd.1 (osd.1) 140 : cluster [DBG] 61.9 scrub ok 2024-02-19T21:07:32.913 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:32.117091+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? v1:172.21.15.163:0/3777293487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:07:32.913 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:32.214723+0000 mon.a (mon.0) 1888 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:07:32.913 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:32.219066+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi163-83845-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi163-83845-39"}]': finished 2024-02-19T21:07:32.913 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:32.219177+0000 mon.a (mon.0) 1890 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi163-83854-48"}]': finished 2024-02-19T21:07:32.913 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:32.219231+0000 mon.a (mon.0) 1891 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]': finished 2024-02-19T21:07:32.913 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:32.219313+0000 mon.a (mon.0) 1892 : audit [INF] from='client.? v1:172.21.15.163:0/3777293487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:07:32.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:32.219363+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-02-19T21:07:32.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:32.220705+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi163-83854-48"}]: dispatch 2024-02-19T21:07:32.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:32.226365+0000 mon.b (mon.1) 418 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-33", "mode": "writeback"}]: dispatch 2024-02-19T21:07:32.914 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:32.228201+0000 mon.a (mon.0) 1895 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-33", "mode": "writeback"}]: dispatch 2024-02-19T21:07:32.914 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:32.927 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:31.165452+0000 mgr.y (mgr.14550) 279 : cluster [DBG] pgmap v377: 245 pgs: 2 peering, 40 creating+peering, 203 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:07:32.927 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:31.720522+0000 osd.1 (osd.1) 139 : cluster [DBG] 61.9 scrub starts 2024-02-19T21:07:32.927 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:31.724130+0000 osd.1 (osd.1) 140 : cluster [DBG] 61.9 scrub ok 2024-02-19T21:07:32.927 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:32.117091+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? v1:172.21.15.163:0/3777293487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:07:32.927 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:32.214723+0000 mon.a (mon.0) 1888 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:07:32.927 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:32.219066+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi163-83845-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi163-83845-39"}]': finished 2024-02-19T21:07:32.927 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:32.219177+0000 mon.a (mon.0) 1890 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi163-83854-48"}]': finished 2024-02-19T21:07:32.927 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:32.219231+0000 mon.a (mon.0) 1891 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]': finished 2024-02-19T21:07:32.928 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:32.219313+0000 mon.a (mon.0) 1892 : audit [INF] from='client.? v1:172.21.15.163:0/3777293487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:07:32.928 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:32.219363+0000 mon.a (mon.0) 1893 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-02-19T21:07:32.928 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:32.220705+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi163-83854-48"}]: dispatch 2024-02-19T21:07:32.928 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:32.226365+0000 mon.b (mon.1) 418 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-33", "mode": "writeback"}]: dispatch 2024-02-19T21:07:32.928 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:32.228201+0000 mon.a (mon.0) 1895 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-33", "mode": "writeback"}]: dispatch 2024-02-19T21:07:33.159 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:07:33.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:33.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:07:33.263 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:33.293 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:33.313 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:33.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:33.416 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:33.443 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:33.719 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:07:33] "GET /metrics HTTP/1.1" 200 190313 "" "Prometheus/2.33.4" 2024-02-19T21:07:33.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:33.166241+0000 mon.b (mon.1) 419 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "39"}]: dispatch 2024-02-19T21:07:33.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:33.167136+0000 mon.a (mon.0) 1896 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "39"}]: dispatch 2024-02-19T21:07:33.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:33.216685+0000 mon.a (mon.0) 1897 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:33.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:33.221792+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi163-83854-48"}]': finished 2024-02-19T21:07:33.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:33.221935+0000 mon.a (mon.0) 1899 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-33", "mode": "writeback"}]': finished 2024-02-19T21:07:33.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:33.222024+0000 mon.a (mon.0) 1900 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "39"}]': finished 2024-02-19T21:07:33.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:33.222113+0000 mon.a (mon.0) 1901 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-02-19T21:07:33.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:33.244020+0000 mon.b (mon.1) 420 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:33.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:33.244573+0000 mon.a (mon.0) 1902 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:33.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:33.245356+0000 mon.b (mon.1) 421 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:33.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:33.245544+0000 mon.a (mon.0) 1903 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:33.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:33.245890+0000 mon.b (mon.1) 422 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi163-83854-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:33.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:33.246069+0000 mon.a (mon.0) 1904 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi163-83854-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:33.891 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:33.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:33.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:33.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:33.166241+0000 mon.b (mon.1) 419 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "39"}]: dispatch 2024-02-19T21:07:33.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:33.167136+0000 mon.a (mon.0) 1896 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "39"}]: dispatch 2024-02-19T21:07:33.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:33.216685+0000 mon.a (mon.0) 1897 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:33.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:33.221792+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi163-83854-48"}]': finished 2024-02-19T21:07:33.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:33.221935+0000 mon.a (mon.0) 1899 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-33", "mode": "writeback"}]': finished 2024-02-19T21:07:33.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:33.222024+0000 mon.a (mon.0) 1900 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "39"}]': finished 2024-02-19T21:07:33.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:33.222113+0000 mon.a (mon.0) 1901 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-02-19T21:07:33.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:33.244020+0000 mon.b (mon.1) 420 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:33.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:33.244573+0000 mon.a (mon.0) 1902 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:33.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:33.245356+0000 mon.b (mon.1) 421 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:33.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:33.245544+0000 mon.a (mon.0) 1903 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:33.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:33.245890+0000 mon.b (mon.1) 422 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi163-83854-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:33.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:33.246069+0000 mon.a (mon.0) 1904 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi163-83854-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:33.995 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:34.023 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:33.166241+0000 mon.b (mon.1) 419 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "39"}]: dispatch 2024-02-19T21:07:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:33.167136+0000 mon.a (mon.0) 1896 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "39"}]: dispatch 2024-02-19T21:07:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:33.216685+0000 mon.a (mon.0) 1897 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:33.221792+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? v1:172.21.15.163:0/3579143446' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi163-83854-48"}]': finished 2024-02-19T21:07:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:33.221935+0000 mon.a (mon.0) 1899 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-33", "mode": "writeback"}]': finished 2024-02-19T21:07:34.067 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:33.222024+0000 mon.a (mon.0) 1900 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "39"}]': finished 2024-02-19T21:07:34.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:33.222113+0000 mon.a (mon.0) 1901 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-02-19T21:07:34.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:33.244020+0000 mon.b (mon.1) 420 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:34.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:33.244573+0000 mon.a (mon.0) 1902 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:34.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:33.245356+0000 mon.b (mon.1) 421 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:34.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:33.245544+0000 mon.a (mon.0) 1903 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:34.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:33.245890+0000 mon.b (mon.1) 422 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi163-83854-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:34.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:33.246069+0000 mon.a (mon.0) 1904 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi163-83854-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:34.408 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:07:34.408 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:07:34.773 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:34.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:34.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:34.877 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:34.907 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:34.938 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:07:34.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:33.165830+0000 mgr.y (mgr.14550) 280 : cluster [DBG] pgmap v380: 244 pgs: 1 peering, 8 unknown, 32 creating+peering, 203 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:07:34.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:33.683825+0000 osd.1 (osd.1) 141 : cluster [DBG] 16.b scrub starts 2024-02-19T21:07:34.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:33.686913+0000 osd.1 (osd.1) 142 : cluster [DBG] 16.b scrub ok 2024-02-19T21:07:34.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:33.764011+0000 osd.4 (osd.4) 129 : cluster [DBG] 2.f scrub starts 2024-02-19T21:07:34.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:33.765030+0000 osd.4 (osd.4) 130 : cluster [DBG] 2.f scrub ok 2024-02-19T21:07:34.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:34.205779+0000 mon.a (mon.0) 1905 : audit [INF] from='client.48253 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi163-83854-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:34.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:34.205828+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-02-19T21:07:34.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:34.206515+0000 mon.b (mon.1) 423 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi163-83854-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:34.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:34.207381+0000 mon.a (mon.0) 1907 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi163-83845-39"}]: dispatch 2024-02-19T21:07:34.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:34.217563+0000 mon.a (mon.0) 1908 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi163-83854-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:34.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:34.549389+0000 osd.6 (osd.6) 115 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:07:34.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:34.550213+0000 osd.6 (osd.6) 116 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:07:34.971 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:33.165830+0000 mgr.y (mgr.14550) 280 : cluster [DBG] pgmap v380: 244 pgs: 1 peering, 8 unknown, 32 creating+peering, 203 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:07:34.971 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:33.683825+0000 osd.1 (osd.1) 141 : cluster [DBG] 16.b scrub starts 2024-02-19T21:07:34.971 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:33.686913+0000 osd.1 (osd.1) 142 : cluster [DBG] 16.b scrub ok 2024-02-19T21:07:34.972 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:33.764011+0000 osd.4 (osd.4) 129 : cluster [DBG] 2.f scrub starts 2024-02-19T21:07:34.972 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:33.765030+0000 osd.4 (osd.4) 130 : cluster [DBG] 2.f scrub ok 2024-02-19T21:07:34.972 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:34.205779+0000 mon.a (mon.0) 1905 : audit [INF] from='client.48253 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi163-83854-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:34.972 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:34.205828+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-02-19T21:07:34.972 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:34.206515+0000 mon.b (mon.1) 423 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi163-83854-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:34.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:34.207381+0000 mon.a (mon.0) 1907 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi163-83845-39"}]: dispatch 2024-02-19T21:07:34.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:34.217563+0000 mon.a (mon.0) 1908 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi163-83854-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:34.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:34.549389+0000 osd.6 (osd.6) 115 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:07:34.973 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:34.550213+0000 osd.6 (osd.6) 116 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:07:34.986 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:33.165830+0000 mgr.y (mgr.14550) 280 : cluster [DBG] pgmap v380: 244 pgs: 1 peering, 8 unknown, 32 creating+peering, 203 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:07:34.986 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:33.683825+0000 osd.1 (osd.1) 141 : cluster [DBG] 16.b scrub starts 2024-02-19T21:07:34.986 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:33.686913+0000 osd.1 (osd.1) 142 : cluster [DBG] 16.b scrub ok 2024-02-19T21:07:34.986 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:33.764011+0000 osd.4 (osd.4) 129 : cluster [DBG] 2.f scrub starts 2024-02-19T21:07:34.986 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:33.765030+0000 osd.4 (osd.4) 130 : cluster [DBG] 2.f scrub ok 2024-02-19T21:07:34.987 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:34.205779+0000 mon.a (mon.0) 1905 : audit [INF] from='client.48253 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi163-83854-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:34.987 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:34.205828+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-02-19T21:07:34.987 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:34.206515+0000 mon.b (mon.1) 423 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi163-83854-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:34.987 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:34.207381+0000 mon.a (mon.0) 1907 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi163-83845-39"}]: dispatch 2024-02-19T21:07:34.987 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:34.217563+0000 mon.a (mon.0) 1908 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi163-83854-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:34.987 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:34.549389+0000 osd.6 (osd.6) 115 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:07:34.987 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:34.550213+0000 osd.6 (osd.6) 116 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:07:35.004 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-19T21:07:35.064 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:35.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:35.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:35.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:35.204 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:35.226 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:35.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:35.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:35.280 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:07:35.331 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:35.358 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:35.556 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:35.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:35.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.663+0000 7fa477728700 1 -- 172.21.15.163:0/1593197744 >> v1:172.21.15.163:6789/0 conn(0x7fa4701031b0 legacy=0x7fa470105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:35.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.663+0000 7fa477728700 1 -- 172.21.15.163:0/1593197744 shutdown_connections 2024-02-19T21:07:35.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.663+0000 7fa477728700 1 -- 172.21.15.163:0/1593197744 >> 172.21.15.163:0/1593197744 conn(0x7fa4700f2470 msgr2=0x7fa4700f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.663+0000 7fa477728700 1 -- 172.21.15.163:0/1593197744 shutdown_connections 2024-02-19T21:07:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.663+0000 7fa477728700 1 -- 172.21.15.163:0/1593197744 wait complete. 2024-02-19T21:07:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa477728700 1 librados: starting msgr at 2024-02-19T21:07:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa477728700 1 librados: starting objecter 2024-02-19T21:07:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa477728700 1 Processor -- start 2024-02-19T21:07:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa477728700 1 -- start start 2024-02-19T21:07:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa477728700 1 librados: setting wanted keys 2024-02-19T21:07:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa477728700 1 librados: calling monclient init 2024-02-19T21:07:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa477728700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa470114fd0 con 0x7fa4701031b0 2024-02-19T21:07:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa477728700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa470116170 con 0x7fa4700fc870 2024-02-19T21:07:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa477728700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa470117350 con 0x7fa4700ff670 2024-02-19T21:07:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa474cc3700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fa4700ff670 0x7fa4700f8b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:42490/0 (socket says 172.21.15.163:42490) 2024-02-19T21:07:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa474cc3700 1 -- 172.21.15.163:0/2500335453 learned_addr learned my addr 172.21.15.163:0/2500335453 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:35.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa4754c4700 1 --1- 172.21.15.163:0/2500335453 >> v1:172.21.15.193:6789/0 conn(0x7fa4700fc870 0x7fa4700f8400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:59374/0 (socket says 172.21.15.163:59374) 2024-02-19T21:07:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa4627fc700 10 client.?.objecter ms_handle_connect 0x7fa4700ff670 2024-02-19T21:07:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa4627fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa4627fc700 10 client.?.objecter ms_handle_connect 0x7fa4701031b0 2024-02-19T21:07:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa4627fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa4627fc700 10 client.?.objecter ms_handle_connect 0x7fa4700fc870 2024-02-19T21:07:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.664+0000 7fa4627fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006105519 0 0) 0x7fa470116170 con 0x7fa4700fc870 2024-02-19T21:07:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa45c0033f0 con 0x7fa4700fc870 2024-02-19T21:07:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707585563 0 0) 0x7fa470117350 con 0x7fa4700ff670 2024-02-19T21:07:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa470116170 con 0x7fa4700ff670 2024-02-19T21:07:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456720191 0 0) 0x7fa470114fd0 con 0x7fa4701031b0 2024-02-19T21:07:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa470117350 con 0x7fa4701031b0 2024-02-19T21:07:35.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 636967346 0 0) 0x7fa45c0033f0 con 0x7fa4700fc870 2024-02-19T21:07:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa470114fd0 con 0x7fa4700fc870 2024-02-19T21:07:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa464001990 con 0x7fa4700fc870 2024-02-19T21:07:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3868928657 0 0) 0x7fa470116170 con 0x7fa4700ff670 2024-02-19T21:07:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa45c0033f0 con 0x7fa4700ff670 2024-02-19T21:07:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2644142878 0 0) 0x7fa470117350 con 0x7fa4701031b0 2024-02-19T21:07:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa470116170 con 0x7fa4701031b0 2024-02-19T21:07:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa46c0019d0 con 0x7fa4700ff670 2024-02-19T21:07:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa4680023d0 con 0x7fa4701031b0 2024-02-19T21:07:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2518965753 0 0) 0x7fa470114fd0 con 0x7fa4700fc870 2024-02-19T21:07:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 >> v1:172.21.15.163:6790/0 conn(0x7fa4700ff670 legacy=0x7fa4700f8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 >> v1:172.21.15.163:6789/0 conn(0x7fa4701031b0 legacy=0x7fa470113820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa470118550 con 0x7fa4700fc870 2024-02-19T21:07:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa477728700 1 -- 172.21.15.163:0/2500335453 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4701151c0 con 0x7fa4700fc870 2024-02-19T21:07:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa477728700 10 client.48259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:35.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa477728700 1 -- 172.21.15.163:0/2500335453 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4701156b0 con 0x7fa4700fc870 2024-02-19T21:07:35.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa477728700 1 librados: init done 2024-02-19T21:07:35.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fa464002300 con 0x7fa4700fc870 2024-02-19T21:07:35.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.665+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa4640058e0 con 0x7fa4700fc870 2024-02-19T21:07:35.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.666+0000 7fa477728700 1 -- 172.21.15.163:0/2500335453 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa454004f20 con 0x7fa4700fc870 2024-02-19T21:07:35.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.666+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fa464001e60 con 0x7fa4700fc870 2024-02-19T21:07:35.666 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:35.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.667+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(275..275 src has 43..275) v4 ==== 8506+0+0 (unknown 3276311494 0 0) 0x7fa464078ce0 con 0x7fa4700fc870 2024-02-19T21:07:35.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.667+0000 7fa4627fc700 10 client.48259.objecter ms_dispatch 0x7fa470001550 osd_map(275..275 src has 43..275) v4 2024-02-19T21:07:35.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.667+0000 7fa4627fc700 3 client.48259.objecter handle_osd_map got epochs [275,275] > 0 2024-02-19T21:07:35.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.667+0000 7fa4627fc700 3 client.48259.objecter handle_osd_map decoding full epoch 275 2024-02-19T21:07:35.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.667+0000 7fa4627fc700 20 client.48259.objecter dump_active .. 0 homeless 2024-02-19T21:07:35.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.667+0000 7fa4627fc700 10 client.48259.objecter ms_handle_connect 0x7fa45c066070 2024-02-19T21:07:35.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.671+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa46404bf10 con 0x7fa4700fc870 2024-02-19T21:07:35.694 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:35.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:35.853+0000 7fa477728700 1 -- 172.21.15.163:0/2500335453 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd out", "ids": ["3"]} v 0) v1 -- 0x7fa4540051d0 con 0x7fa4700fc870 2024-02-19T21:07:36.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.136+0000 7f6dbf85b700 1 -- 172.21.15.163:0/1297750005 >> v1:172.21.15.163:6789/0 conn(0x7f6db81042b0 legacy=0x7f6db8106730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:36.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.136+0000 7f6dbf85b700 1 -- 172.21.15.163:0/1297750005 shutdown_connections 2024-02-19T21:07:36.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.136+0000 7f6dbf85b700 1 -- 172.21.15.163:0/1297750005 >> 172.21.15.163:0/1297750005 conn(0x7f6db80f3590 msgr2=0x7f6db80f59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:36.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.137+0000 7f6dbf85b700 1 -- 172.21.15.163:0/1297750005 shutdown_connections 2024-02-19T21:07:36.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.137+0000 7f6dbf85b700 1 -- 172.21.15.163:0/1297750005 wait complete. 2024-02-19T21:07:36.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.137+0000 7f6dbf85b700 1 librados: starting msgr at 2024-02-19T21:07:36.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.137+0000 7f6dbf85b700 1 librados: starting objecter 2024-02-19T21:07:36.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.137+0000 7f6dbf85b700 1 Processor -- start 2024-02-19T21:07:36.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.137+0000 7f6dbf85b700 1 -- start start 2024-02-19T21:07:36.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.137+0000 7f6dbf85b700 1 librados: setting wanted keys 2024-02-19T21:07:36.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.137+0000 7f6dbf85b700 1 librados: calling monclient init 2024-02-19T21:07:36.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.137+0000 7f6dbf85b700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6db8117300 con 0x7f6db81042b0 2024-02-19T21:07:36.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.137+0000 7f6dbf85b700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6db81184e0 con 0x7f6db80fd970 2024-02-19T21:07:36.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.137+0000 7f6dbf85b700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6db81196e0 con 0x7f6db8100770 2024-02-19T21:07:36.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.137+0000 7f6dbcdf6700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f6db8100770 0x7f6db80fa220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:42514/0 (socket says 172.21.15.163:42514) 2024-02-19T21:07:36.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.137+0000 7f6dbcdf6700 1 -- 172.21.15.163:0/2363472752 learned_addr learned my addr 172.21.15.163:0/2363472752 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:36.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.137+0000 7f6dae7fc700 10 client.?.objecter ms_handle_connect 0x7f6db8100770 2024-02-19T21:07:36.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.137+0000 7f6dae7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:36.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 10 client.?.objecter ms_handle_connect 0x7f6db81042b0 2024-02-19T21:07:36.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:36.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 10 client.?.objecter ms_handle_connect 0x7f6db80fd970 2024-02-19T21:07:36.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:36.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745026052 0 0) 0x7f6db81196e0 con 0x7f6db8100770 2024-02-19T21:07:36.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6da4003800 con 0x7f6db8100770 2024-02-19T21:07:36.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919826965 0 0) 0x7f6db8117300 con 0x7f6db81042b0 2024-02-19T21:07:36.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db81196e0 con 0x7f6db81042b0 2024-02-19T21:07:36.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183888365 0 0) 0x7f6db81184e0 con 0x7f6db80fd970 2024-02-19T21:07:36.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db8117300 con 0x7f6db80fd970 2024-02-19T21:07:36.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1776191748 0 0) 0x7f6da4003800 con 0x7f6db8100770 2024-02-19T21:07:36.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6db81184e0 con 0x7f6db8100770 2024-02-19T21:07:36.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6da80019a0 con 0x7f6db8100770 2024-02-19T21:07:36.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 34717495 0 0) 0x7f6db81196e0 con 0x7f6db81042b0 2024-02-19T21:07:36.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6da4003800 con 0x7f6db81042b0 2024-02-19T21:07:36.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2018909743 0 0) 0x7f6db8117300 con 0x7f6db80fd970 2024-02-19T21:07:36.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6db81196e0 con 0x7f6db80fd970 2024-02-19T21:07:36.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6db0003220 con 0x7f6db81042b0 2024-02-19T21:07:36.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6db4001840 con 0x7f6db80fd970 2024-02-19T21:07:36.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2985954357 0 0) 0x7f6db81184e0 con 0x7f6db8100770 2024-02-19T21:07:36.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 >> v1:172.21.15.193:6789/0 conn(0x7f6db80fd970 legacy=0x7f6db80f9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:36.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 >> v1:172.21.15.163:6789/0 conn(0x7f6db81042b0 legacy=0x7f6db8115a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:36.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db811a8e0 con 0x7f6db8100770 2024-02-19T21:07:36.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db81174f0 con 0x7f6db8100770 2024-02-19T21:07:36.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dbf85b700 10 client.49448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:36.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db8117960 con 0x7f6db8100770 2024-02-19T21:07:36.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dbf85b700 1 librados: init done 2024-02-19T21:07:36.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6da8002420 con 0x7f6db8100770 2024-02-19T21:07:36.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.138+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6da8005ac0 con 0x7f6db8100770 2024-02-19T21:07:36.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.139+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6d9c004f00 con 0x7f6db8100770 2024-02-19T21:07:36.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6da8001f80 con 0x7f6db8100770 2024-02-19T21:07:36.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(275..275 src has 43..275) v4 ==== 8506+0+0 (unknown 3276311494 0 0) 0x7f6da8078e00 con 0x7f6db8100770 2024-02-19T21:07:36.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 osd_map(275..275 src has 43..275) v4 2024-02-19T21:07:36.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dae7fc700 3 client.49448.objecter handle_osd_map got epochs [275,275] > 0 2024-02-19T21:07:36.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dae7fc700 3 client.49448.objecter handle_osd_map decoding full epoch 275 2024-02-19T21:07:36.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dae7fc700 20 client.49448.objecter dump_active .. 0 homeless 2024-02-19T21:07:36.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v275) v1 ==== 54+0+16 (unknown 18684639 0 2420000192) 0x7f6da8079190 con 0x7f6db8100770 2024-02-19T21:07:36.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dae7fc700 10 client.49448.objecter ms_handle_connect 0x7f6da4063dd0 2024-02-19T21:07:36.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:36.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 1 2024-02-19T21:07:36.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0051a0 osd=0 3 2024-02-19T21:07:36.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0051a0 osd=0 3 2024-02-19T21:07:36.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0051a0 osd=0 3 2024-02-19T21:07:36.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 1 2024-02-19T21:07:36.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c0051a0 osd=0 3 2024-02-19T21:07:36.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 0 1 2024-02-19T21:07:36.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0051a0 osd=0 4 2024-02-19T21:07:36.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dbf85b700 10 client.49448.objecter _send_command 1 2024-02-19T21:07:36.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.140+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d9c004950 con 0x7f6d9c004530 2024-02-19T21:07:36.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.141+0000 7f6dae7fc700 10 client.49448.objecter ms_handle_connect 0x7f6d9c004530 2024-02-19T21:07:36.167 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:07:36.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.141+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d9c004950 con 0x7f6d9c004530 2024-02-19T21:07:36.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.141+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:07:36.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.141+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 1 = system:0 2024-02-19T21:07:36.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.141+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c0051a0 osd=0 4 2024-02-19T21:07:36.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.141+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 0 1 2024-02-19T21:07:36.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.156+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:36.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.156+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 2 2024-02-19T21:07:36.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.156+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0051a0 osd=0 3 2024-02-19T21:07:36.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.156+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0051a0 osd=0 3 2024-02-19T21:07:36.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.156+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0051a0 osd=0 3 2024-02-19T21:07:36.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.156+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 2 2024-02-19T21:07:36.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.156+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c0051a0 osd=0 3 2024-02-19T21:07:36.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.156+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 0 2 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.156+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0051a0 osd=0 4 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.156+0000 7f6dbf85b700 10 client.49448.objecter _send_command 2 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.156+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d9c004d30 con 0x7f6d9c004530 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6d9c004d30 con 0x7f6d9c004530 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c0051a0 osd=0 4 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 0 2 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 3 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c004270 osd=1 3 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c004270 osd=1 3 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c004270 osd=1 3 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 3 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c004270 osd=1 3 2024-02-19T21:07:36.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 1 3 2024-02-19T21:07:36.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c004270 osd=1 4 2024-02-19T21:07:36.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dbf85b700 10 client.49448.objecter _send_command 3 2024-02-19T21:07:36.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d9c00bf30 con 0x7f6d9c003b40 2024-02-19T21:07:36.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.158+0000 7f6dae7fc700 10 client.49448.objecter ms_handle_connect 0x7f6d9c003b40 2024-02-19T21:07:36.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.159+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d9c00bf30 con 0x7f6d9c003b40 2024-02-19T21:07:36.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.159+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:07:36.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.159+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 3 = system:0 2024-02-19T21:07:36.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.159+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c004270 osd=1 4 2024-02-19T21:07:36.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.159+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 1 3 2024-02-19T21:07:36.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.174+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.174+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 4 2024-02-19T21:07:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.174+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c004270 osd=1 3 2024-02-19T21:07:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.174+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c004270 osd=1 3 2024-02-19T21:07:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.174+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c004270 osd=1 3 2024-02-19T21:07:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.174+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 4 2024-02-19T21:07:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.174+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c004270 osd=1 3 2024-02-19T21:07:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.174+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 1 4 2024-02-19T21:07:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.174+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c004270 osd=1 4 2024-02-19T21:07:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.174+0000 7f6dbf85b700 10 client.49448.objecter _send_command 4 2024-02-19T21:07:36.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.174+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d9c00d250 con 0x7f6d9c003b40 2024-02-19T21:07:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.175+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6d9c00d250 con 0x7f6d9c003b40 2024-02-19T21:07:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.175+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.175+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.175+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c004270 osd=1 4 2024-02-19T21:07:36.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.175+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 1 4 2024-02-19T21:07:36.173 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:07:36.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.176+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.176+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 5 2024-02-19T21:07:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.176+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0033a0 osd=2 3 2024-02-19T21:07:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.176+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0033a0 osd=2 3 2024-02-19T21:07:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.176+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0033a0 osd=2 3 2024-02-19T21:07:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.176+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 5 2024-02-19T21:07:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.176+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c0033a0 osd=2 3 2024-02-19T21:07:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.176+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 2 5 2024-02-19T21:07:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.176+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0033a0 osd=2 4 2024-02-19T21:07:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.176+0000 7f6dbf85b700 10 client.49448.objecter _send_command 5 2024-02-19T21:07:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.176+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d9c010ca0 con 0x7f6d9c002070 2024-02-19T21:07:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.176+0000 7f6dae7fc700 10 client.49448.objecter ms_handle_connect 0x7f6d9c002070 2024-02-19T21:07:36.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.177+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d9c010ca0 con 0x7f6d9c002070 2024-02-19T21:07:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.177+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:07:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.177+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 5 = system:0 2024-02-19T21:07:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.177+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c0033a0 osd=2 4 2024-02-19T21:07:36.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.177+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 2 5 2024-02-19T21:07:36.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.192+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:36.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.192+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 6 2024-02-19T21:07:36.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.192+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0033a0 osd=2 3 2024-02-19T21:07:36.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.192+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0033a0 osd=2 3 2024-02-19T21:07:36.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.192+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0033a0 osd=2 3 2024-02-19T21:07:36.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.192+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 6 2024-02-19T21:07:36.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.192+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c0033a0 osd=2 3 2024-02-19T21:07:36.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.192+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 2 6 2024-02-19T21:07:36.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.192+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0033a0 osd=2 4 2024-02-19T21:07:36.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.192+0000 7f6dbf85b700 10 client.49448.objecter _send_command 6 2024-02-19T21:07:36.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.192+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d9c001630 con 0x7f6d9c002070 2024-02-19T21:07:36.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6d9c001630 con 0x7f6d9c002070 2024-02-19T21:07:36.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:36.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c0033a0 osd=2 4 2024-02-19T21:07:36.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 2 6 2024-02-19T21:07:36.190 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:07:36.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:36.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 7 2024-02-19T21:07:36.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c00c870 osd=3 3 2024-02-19T21:07:36.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c00c870 osd=3 3 2024-02-19T21:07:36.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c00c870 osd=3 3 2024-02-19T21:07:36.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 7 2024-02-19T21:07:36.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c00c870 osd=3 3 2024-02-19T21:07:36.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 3 7 2024-02-19T21:07:36.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c00c870 osd=3 4 2024-02-19T21:07:36.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dbf85b700 10 client.49448.objecter _send_command 7 2024-02-19T21:07:36.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d9c00d020 con 0x7f6d9c012450 2024-02-19T21:07:36.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.194+0000 7f6dae7fc700 10 client.49448.objecter ms_handle_connect 0x7f6d9c012450 2024-02-19T21:07:36.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.195+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d9c00d020 con 0x7f6d9c012450 2024-02-19T21:07:36.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.195+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:07:36.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.195+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 7 = system:0 2024-02-19T21:07:36.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.195+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c00c870 osd=3 4 2024-02-19T21:07:36.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.195+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 3 7 2024-02-19T21:07:36.201 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:07:35.853+0000 7fb606608700 -1 mon.a@0(leader).osd e275 definitely_dead 0 2024-02-19T21:07:36.201 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:36.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.213+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:36.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.213+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 8 2024-02-19T21:07:36.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.213+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c00c870 osd=3 3 2024-02-19T21:07:36.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.213+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c00c870 osd=3 3 2024-02-19T21:07:36.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.213+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c00c870 osd=3 3 2024-02-19T21:07:36.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.213+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 8 2024-02-19T21:07:36.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.213+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c00c870 osd=3 3 2024-02-19T21:07:36.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.213+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 3 8 2024-02-19T21:07:36.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.213+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c00c870 osd=3 4 2024-02-19T21:07:36.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.213+0000 7f6dbf85b700 10 client.49448.objecter _send_command 8 2024-02-19T21:07:36.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.213+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d9c0120e0 con 0x7f6d9c012450 2024-02-19T21:07:36.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.214+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6d9c0120e0 con 0x7f6d9c012450 2024-02-19T21:07:36.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.214+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:36.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.214+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.214+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c00c870 osd=3 4 2024-02-19T21:07:36.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.214+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 3 8 2024-02-19T21:07:36.211 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:07:36.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.214+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:36.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.214+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 9 2024-02-19T21:07:36.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.215+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0120e0 osd=4 3 2024-02-19T21:07:36.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.215+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0120e0 osd=4 3 2024-02-19T21:07:36.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.215+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0120e0 osd=4 3 2024-02-19T21:07:36.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.215+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 9 2024-02-19T21:07:36.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.215+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c0120e0 osd=4 3 2024-02-19T21:07:36.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.215+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 4 9 2024-02-19T21:07:36.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.215+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0120e0 osd=4 4 2024-02-19T21:07:36.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.215+0000 7f6dbf85b700 10 client.49448.objecter _send_command 9 2024-02-19T21:07:36.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.215+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d9c01e6f0 con 0x7f6d9c01abe0 2024-02-19T21:07:36.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.215+0000 7f6dae7fc700 10 client.49448.objecter ms_handle_connect 0x7f6d9c01abe0 2024-02-19T21:07:36.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.216+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d9c01e6f0 con 0x7f6d9c01abe0 2024-02-19T21:07:36.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.216+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:07:36.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.216+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 9 = system:0 2024-02-19T21:07:36.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.216+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c0120e0 osd=4 4 2024-02-19T21:07:36.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.216+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 4 9 2024-02-19T21:07:36.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.230+0000 7fa4627fc700 1 -- 172.21.15.163:0/2500335453 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd out", "ids": ["3"]}]=0 marked out osd.3. v276) v1 ==== 87+0+0 (unknown 4071625563 0 0) 0x7fa46404a320 con 0x7fa4700fc870 2024-02-19T21:07:36.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.233+0000 7fa477728700 10 librados: watch_flush enter 2024-02-19T21:07:36.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.233+0000 7fa477728700 10 librados: watch_flush exit 2024-02-19T21:07:36.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.233+0000 7fa477728700 20 client.48259.objecter shutdown clearing up homeless session... 2024-02-19T21:07:36.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.233+0000 7fa477728700 10 client.48259.objecter successfully canceled tick 2024-02-19T21:07:36.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.233+0000 7fa477728700 1 -- 172.21.15.163:0/2500335453 >> v1:172.21.15.163:6800/1552514681 conn(0x7fa45c066070 legacy=0x7fa45c068530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:36.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.233+0000 7fa477728700 1 -- 172.21.15.163:0/2500335453 >> v1:172.21.15.193:6789/0 conn(0x7fa4700fc870 legacy=0x7fa4700f8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:36.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.233+0000 7fa477728700 1 -- 172.21.15.163:0/2500335453 shutdown_connections 2024-02-19T21:07:36.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.233+0000 7fa477728700 1 -- 172.21.15.163:0/2500335453 >> 172.21.15.163:0/2500335453 conn(0x7fa4700f2470 msgr2=0x7fa4700601c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:36.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.236+0000 7fa477728700 1 -- 172.21.15.163:0/2500335453 shutdown_connections 2024-02-19T21:07:36.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.236+0000 7fa477728700 1 -- 172.21.15.163:0/2500335453 wait complete. 2024-02-19T21:07:36.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.236+0000 7fa477728700 1 librados: shutdown 2024-02-19T21:07:36.238 INFO:teuthology.orchestra.run.smithi163.stderr:marked out osd.3. 2024-02-19T21:07:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.247+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.247+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 10 2024-02-19T21:07:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.247+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0120e0 osd=4 3 2024-02-19T21:07:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.247+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0120e0 osd=4 3 2024-02-19T21:07:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.247+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0120e0 osd=4 3 2024-02-19T21:07:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.247+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 10 2024-02-19T21:07:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.247+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c0120e0 osd=4 3 2024-02-19T21:07:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.247+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 4 10 2024-02-19T21:07:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.247+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0120e0 osd=4 4 2024-02-19T21:07:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.247+0000 7f6dbf85b700 10 client.49448.objecter _send_command 10 2024-02-19T21:07:36.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.247+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d9c01e6f0 con 0x7f6d9c01abe0 2024-02-19T21:07:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.249+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6d9c01e6f0 con 0x7f6d9c01abe0 2024-02-19T21:07:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.249+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.249+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.249+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c0120e0 osd=4 4 2024-02-19T21:07:36.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.249+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 4 10 2024-02-19T21:07:36.246 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:07:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.257+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.257+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 11 2024-02-19T21:07:36.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.257+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c01e6f0 osd=5 3 2024-02-19T21:07:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.257+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c01e6f0 osd=5 3 2024-02-19T21:07:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.257+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c01e6f0 osd=5 3 2024-02-19T21:07:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.257+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 11 2024-02-19T21:07:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.257+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c01e6f0 osd=5 3 2024-02-19T21:07:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.257+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 5 11 2024-02-19T21:07:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.257+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c01e6f0 osd=5 4 2024-02-19T21:07:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.257+0000 7f6dbf85b700 10 client.49448.objecter _send_command 11 2024-02-19T21:07:36.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.257+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d9c0229f0 con 0x7f6d9c01ed70 2024-02-19T21:07:36.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.263+0000 7f6dae7fc700 10 client.49448.objecter ms_handle_connect 0x7f6d9c01ed70 2024-02-19T21:07:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.265+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d9c0229f0 con 0x7f6d9c01ed70 2024-02-19T21:07:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.265+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:07:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.265+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 11 = system:0 2024-02-19T21:07:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.265+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c01e6f0 osd=5 4 2024-02-19T21:07:36.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.265+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 5 11 2024-02-19T21:07:36.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.282+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:36.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.282+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 12 2024-02-19T21:07:36.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.282+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c01e6f0 osd=5 3 2024-02-19T21:07:36.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.282+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c01e6f0 osd=5 3 2024-02-19T21:07:36.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.282+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c01e6f0 osd=5 3 2024-02-19T21:07:36.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.282+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 12 2024-02-19T21:07:36.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.282+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c01e6f0 osd=5 3 2024-02-19T21:07:36.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.282+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 5 12 2024-02-19T21:07:36.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.282+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c01e6f0 osd=5 4 2024-02-19T21:07:36.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.282+0000 7f6dbf85b700 10 client.49448.objecter _send_command 12 2024-02-19T21:07:36.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.282+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d9c0229f0 con 0x7f6d9c01ed70 2024-02-19T21:07:36.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6d9c0229f0 con 0x7f6d9c01ed70 2024-02-19T21:07:36.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:36.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c01e6f0 osd=5 4 2024-02-19T21:07:36.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 5 12 2024-02-19T21:07:36.281 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:07:36.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:36.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 13 2024-02-19T21:07:36.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0229f0 osd=6 3 2024-02-19T21:07:36.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0229f0 osd=6 3 2024-02-19T21:07:36.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0229f0 osd=6 3 2024-02-19T21:07:36.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 13 2024-02-19T21:07:36.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c0229f0 osd=6 3 2024-02-19T21:07:36.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 6 13 2024-02-19T21:07:36.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0229f0 osd=6 4 2024-02-19T21:07:36.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dbf85b700 10 client.49448.objecter _send_command 13 2024-02-19T21:07:36.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d9c023c10 con 0x7f6d9c0246f0 2024-02-19T21:07:36.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dae7fc700 10 client.49448.objecter ms_handle_connect 0x7f6d9c0246f0 2024-02-19T21:07:36.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d9c023c10 con 0x7f6d9c0246f0 2024-02-19T21:07:36.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:07:36.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 13 = system:0 2024-02-19T21:07:36.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c0229f0 osd=6 4 2024-02-19T21:07:36.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.285+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 6 13 2024-02-19T21:07:36.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.300+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:36.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.300+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 14 2024-02-19T21:07:36.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.300+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0229f0 osd=6 3 2024-02-19T21:07:36.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.300+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0229f0 osd=6 3 2024-02-19T21:07:36.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.300+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c0229f0 osd=6 3 2024-02-19T21:07:36.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.300+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 14 2024-02-19T21:07:36.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.300+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c0229f0 osd=6 3 2024-02-19T21:07:36.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.300+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 6 14 2024-02-19T21:07:36.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.300+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0229f0 osd=6 4 2024-02-19T21:07:36.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.300+0000 7f6dbf85b700 10 client.49448.objecter _send_command 14 2024-02-19T21:07:36.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.300+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d9c023c10 con 0x7f6d9c0246f0 2024-02-19T21:07:36.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6d9c023c10 con 0x7f6d9c0246f0 2024-02-19T21:07:36.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:36.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c0229f0 osd=6 4 2024-02-19T21:07:36.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 6 14 2024-02-19T21:07:36.299 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:07:36.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:36.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 15 2024-02-19T21:07:36.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c023c10 osd=7 3 2024-02-19T21:07:36.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c023c10 osd=7 3 2024-02-19T21:07:36.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c023c10 osd=7 3 2024-02-19T21:07:36.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 15 2024-02-19T21:07:36.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c023c10 osd=7 3 2024-02-19T21:07:36.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 7 15 2024-02-19T21:07:36.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c023c10 osd=7 4 2024-02-19T21:07:36.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dbf85b700 10 client.49448.objecter _send_command 15 2024-02-19T21:07:36.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d9c02b3c0 con 0x7f6d9c0278b0 2024-02-19T21:07:36.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dae7fc700 10 client.49448.objecter ms_handle_connect 0x7f6d9c0278b0 2024-02-19T21:07:36.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6d9c02b3c0 con 0x7f6d9c0278b0 2024-02-19T21:07:36.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:07:36.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 15 = system:0 2024-02-19T21:07:36.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c023c10 osd=7 4 2024-02-19T21:07:36.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.303+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 7 15 2024-02-19T21:07:36.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.318+0000 7f6dbf85b700 10 client.49448.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:36.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.318+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign -1 16 2024-02-19T21:07:36.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.318+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c023c10 osd=7 3 2024-02-19T21:07:36.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.318+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c023c10 osd=7 3 2024-02-19T21:07:36.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.318+0000 7f6dbf85b700 20 client.49448.objecter _get_session s=0x7f6d9c023c10 osd=7 3 2024-02-19T21:07:36.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.318+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_remove -1 16 2024-02-19T21:07:36.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.318+0000 7f6dbf85b700 20 client.49448.objecter get_session s=0x7f6d9c023c10 osd=7 3 2024-02-19T21:07:36.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.318+0000 7f6dbf85b700 15 client.49448.objecter _session_command_op_assign 7 16 2024-02-19T21:07:36.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.318+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c023c10 osd=7 4 2024-02-19T21:07:36.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.318+0000 7f6dbf85b700 10 client.49448.objecter _send_command 16 2024-02-19T21:07:36.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.318+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6d9c02b3c0 con 0x7f6d9c0278b0 2024-02-19T21:07:36.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dae7fc700 1 -- 172.21.15.163:0/2363472752 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6d9c02b3c0 con 0x7f6d9c0278b0 2024-02-19T21:07:36.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dae7fc700 10 client.49448.objecter ms_dispatch 0x7f6db8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:36.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dae7fc700 10 client.49448.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dae7fc700 20 client.49448.objecter put_session s=0x7f6d9c023c10 osd=7 4 2024-02-19T21:07:36.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dae7fc700 15 client.49448.objecter _session_command_op_remove 7 16 2024-02-19T21:07:36.317 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:07:36.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 10 librados: watch_flush enter 2024-02-19T21:07:36.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 10 librados: watch_flush exit 2024-02-19T21:07:36.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 10 client.49448.objecter close_session for osd.0 2024-02-19T21:07:36.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 >> v1:172.21.15.163:6801/1559418115 conn(0x7f6d9c004530 legacy=0x7f6d9c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:36.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0051a0 osd=0 1 2024-02-19T21:07:36.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 10 client.49448.objecter close_session for osd.1 2024-02-19T21:07:36.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 >> v1:172.21.15.163:6805/3125047432 conn(0x7f6d9c003b40 legacy=0x7f6d9c00a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c004270 osd=1 1 2024-02-19T21:07:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 10 client.49448.objecter close_session for osd.2 2024-02-19T21:07:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 >> v1:172.21.15.163:6809/1563158487 conn(0x7f6d9c002070 legacy=0x7f6d9c00f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0033a0 osd=2 1 2024-02-19T21:07:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 10 client.49448.objecter close_session for osd.3 2024-02-19T21:07:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 >> v1:172.21.15.163:6813/2112279769 conn(0x7f6d9c012450 legacy=0x7f6d9c0128b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c00c870 osd=3 1 2024-02-19T21:07:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 10 client.49448.objecter close_session for osd.4 2024-02-19T21:07:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbddf8700 1 -- 172.21.15.163:0/2363472752 reap_dead start 2024-02-19T21:07:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 >> v1:172.21.15.193:6800/3129716965 conn(0x7f6d9c01abe0 legacy=0x7f6d9c01cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbddf8700 1 -- 172.21.15.163:0/2363472752 reap_dead start 2024-02-19T21:07:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0120e0 osd=4 1 2024-02-19T21:07:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 10 client.49448.objecter close_session for osd.5 2024-02-19T21:07:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 >> v1:172.21.15.193:6804/805161991 conn(0x7f6d9c01ed70 legacy=0x7f6d9c0211a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c01e6f0 osd=5 1 2024-02-19T21:07:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 10 client.49448.objecter close_session for osd.6 2024-02-19T21:07:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 >> v1:172.21.15.193:6808/787604615 conn(0x7f6d9c0246f0 legacy=0x7f6d9c024b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c0229f0 osd=6 1 2024-02-19T21:07:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 10 client.49448.objecter close_session for osd.7 2024-02-19T21:07:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 >> v1:172.21.15.193:6812/157927432 conn(0x7f6d9c0278b0 legacy=0x7f6d9c029ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 20 client.49448.objecter put_session s=0x7f6d9c023c10 osd=7 1 2024-02-19T21:07:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 20 client.49448.objecter shutdown clearing up homeless session... 2024-02-19T21:07:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 10 client.49448.objecter successfully canceled tick 2024-02-19T21:07:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6da4063dd0 legacy=0x7f6da4066290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 >> v1:172.21.15.163:6790/0 conn(0x7f6db8100770 legacy=0x7f6db80fa220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbddf8700 1 -- 172.21.15.163:0/2363472752 reap_dead start 2024-02-19T21:07:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 shutdown_connections 2024-02-19T21:07:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 >> 172.21.15.163:0/2363472752 conn(0x7f6db80f3590 msgr2=0x7f6db8107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 shutdown_connections 2024-02-19T21:07:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 1 -- 172.21.15.163:0/2363472752 wait complete. 2024-02-19T21:07:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:36.321+0000 7f6dbf85b700 1 librados: shutdown 2024-02-19T21:07:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:07:36.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:35.167079+0000 mgr.y (mgr.14550) 281 : cluster [DBG] pgmap v383: 236 pgs: 2 clean+premerge+peered, 234 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:07:36.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:35.202044+0000 osd.2 (osd.2) 111 : cluster [DBG] 61.4 scrub starts 2024-02-19T21:07:36.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:35.203983+0000 osd.2 (osd.2) 112 : cluster [DBG] 61.4 scrub ok 2024-02-19T21:07:36.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:35.206448+0000 mon.a (mon.0) 1909 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi163-83845-39"}]': finished 2024-02-19T21:07:36.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:35.437427+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-02-19T21:07:36.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:35.437957+0000 mon.a (mon.0) 1911 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi163-83845-39"}]: dispatch 2024-02-19T21:07:36.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:35.480819+0000 mon.b (mon.1) 424 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:36.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:35.481222+0000 mon.a (mon.0) 1912 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:36.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:35.504881+0000 osd.6 (osd.6) 117 : cluster [DBG] 16.6 scrub starts 2024-02-19T21:07:36.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:35.505602+0000 osd.6 (osd.6) 118 : cluster [DBG] 16.6 scrub ok 2024-02-19T21:07:36.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:35.854711+0000 mon.b (mon.1) 425 : audit [INF] from='client.? v1:172.21.15.163:0/2500335453' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-19T21:07:36.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:35.855087+0000 mon.a (mon.0) 1913 : audit [INF] from='client.48259 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-19T21:07:36.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:35.855197+0000 mon.a (mon.0) 1914 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-02-19T21:07:36.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:36.140897+0000 mon.c (mon.2) 331 : audit [DBG] from='client.? v1:172.21.15.163:0/2363472752' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:36.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:35.167079+0000 mgr.y (mgr.14550) 281 : cluster [DBG] pgmap v383: 236 pgs: 2 clean+premerge+peered, 234 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:07:36.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:35.202044+0000 osd.2 (osd.2) 111 : cluster [DBG] 61.4 scrub starts 2024-02-19T21:07:36.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:35.203983+0000 osd.2 (osd.2) 112 : cluster [DBG] 61.4 scrub ok 2024-02-19T21:07:36.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:35.206448+0000 mon.a (mon.0) 1909 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi163-83845-39"}]': finished 2024-02-19T21:07:36.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:35.437427+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-02-19T21:07:36.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:35.437957+0000 mon.a (mon.0) 1911 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi163-83845-39"}]: dispatch 2024-02-19T21:07:36.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:35.480819+0000 mon.b (mon.1) 424 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:36.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:35.481222+0000 mon.a (mon.0) 1912 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:36.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:35.504881+0000 osd.6 (osd.6) 117 : cluster [DBG] 16.6 scrub starts 2024-02-19T21:07:36.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:35.505602+0000 osd.6 (osd.6) 118 : cluster [DBG] 16.6 scrub ok 2024-02-19T21:07:36.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:35.854711+0000 mon.b (mon.1) 425 : audit [INF] from='client.? v1:172.21.15.163:0/2500335453' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-19T21:07:36.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:35.855087+0000 mon.a (mon.0) 1913 : audit [INF] from='client.48259 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-19T21:07:36.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:35.855197+0000 mon.a (mon.0) 1914 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-02-19T21:07:36.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:36.140897+0000 mon.c (mon.2) 331 : audit [DBG] from='client.? v1:172.21.15.163:0/2363472752' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:36.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:35.167079+0000 mgr.y (mgr.14550) 281 : cluster [DBG] pgmap v383: 236 pgs: 2 clean+premerge+peered, 234 active+clean; 6.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:07:36.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:35.202044+0000 osd.2 (osd.2) 111 : cluster [DBG] 61.4 scrub starts 2024-02-19T21:07:36.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:35.203983+0000 osd.2 (osd.2) 112 : cluster [DBG] 61.4 scrub ok 2024-02-19T21:07:36.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:35.206448+0000 mon.a (mon.0) 1909 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi163-83845-39"}]': finished 2024-02-19T21:07:36.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:35.437427+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-02-19T21:07:36.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:35.437957+0000 mon.a (mon.0) 1911 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi163-83845-39"}]: dispatch 2024-02-19T21:07:36.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:35.480819+0000 mon.b (mon.1) 424 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:36.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:35.481222+0000 mon.a (mon.0) 1912 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:36.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:35.504881+0000 osd.6 (osd.6) 117 : cluster [DBG] 16.6 scrub starts 2024-02-19T21:07:36.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:35.505602+0000 osd.6 (osd.6) 118 : cluster [DBG] 16.6 scrub ok 2024-02-19T21:07:36.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:35.854711+0000 mon.b (mon.1) 425 : audit [INF] from='client.? v1:172.21.15.163:0/2500335453' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-19T21:07:36.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:35.855087+0000 mon.a (mon.0) 1913 : audit [INF] from='client.48259 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-19T21:07:36.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:35.855197+0000 mon.a (mon.0) 1914 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2024-02-19T21:07:36.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:36.140897+0000 mon.c (mon.2) 331 : audit [DBG] from='client.? v1:172.21.15.163:0/2363472752' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:36.843 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:36.847 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:36.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:36.948 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:07:36.977 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:37.050 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:07:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:37.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:07:37.053 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:07:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:07:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-02-19T21:07:37.053 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-02-19T21:07:37.053 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:07:37.154 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:37.185 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:37.383 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:37.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:37.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:37.479 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:36.221971+0000 mon.a (mon.0) 1915 : audit [INF] from='client.48253 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi163-83854-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi163-83854-49"}]': finished 2024-02-19T21:07:37.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:36.222051+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi163-83845-39"}]': finished 2024-02-19T21:07:37.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:36.222117+0000 mon.a (mon.0) 1917 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:37.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:36.222188+0000 mon.a (mon.0) 1918 : audit [INF] from='client.48259 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-19T21:07:37.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:36.222227+0000 mon.a (mon.0) 1919 : cluster [DBG] osdmap e276: 8 total, 8 up, 7 in 2024-02-19T21:07:37.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:36.264955+0000 mon.b (mon.1) 426 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:37.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:36.265298+0000 mon.a (mon.0) 1920 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:37.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:36.271564+0000 mon.b (mon.1) 427 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:37.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:36.275417+0000 mon.a (mon.0) 1921 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:37.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:36.276031+0000 mon.b (mon.1) 428 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:37.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:36.276242+0000 mon.a (mon.0) 1922 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:37.481 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:36.276717+0000 mon.b (mon.1) 429 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi163-83845-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:37.481 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:36.276933+0000 mon.a (mon.0) 1923 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi163-83845-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:37.481 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:37.168069+0000 mon.b (mon.1) 430 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "38"}]: dispatch 2024-02-19T21:07:37.481 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:37.169101+0000 mon.a (mon.0) 1924 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "38"}]: dispatch 2024-02-19T21:07:37.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:36.221971+0000 mon.a (mon.0) 1915 : audit [INF] from='client.48253 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi163-83854-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi163-83854-49"}]': finished 2024-02-19T21:07:37.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:36.222051+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi163-83845-39"}]': finished 2024-02-19T21:07:37.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:36.222117+0000 mon.a (mon.0) 1917 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:37.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:36.222188+0000 mon.a (mon.0) 1918 : audit [INF] from='client.48259 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-19T21:07:37.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:36.222227+0000 mon.a (mon.0) 1919 : cluster [DBG] osdmap e276: 8 total, 8 up, 7 in 2024-02-19T21:07:37.481 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:36.264955+0000 mon.b (mon.1) 426 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:37.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:36.265298+0000 mon.a (mon.0) 1920 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:37.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:36.271564+0000 mon.b (mon.1) 427 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:37.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:36.275417+0000 mon.a (mon.0) 1921 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:37.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:36.276031+0000 mon.b (mon.1) 428 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:37.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:36.276242+0000 mon.a (mon.0) 1922 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:37.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:36.276717+0000 mon.b (mon.1) 429 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi163-83845-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:37.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:36.276933+0000 mon.a (mon.0) 1923 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi163-83845-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:37.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:37.168069+0000 mon.b (mon.1) 430 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "38"}]: dispatch 2024-02-19T21:07:37.482 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:37.169101+0000 mon.a (mon.0) 1924 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "38"}]: dispatch 2024-02-19T21:07:37.486 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:37.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:36.221971+0000 mon.a (mon.0) 1915 : audit [INF] from='client.48253 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi163-83854-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi163-83854-49"}]': finished 2024-02-19T21:07:37.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:36.222051+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? v1:172.21.15.163:0/2461233870' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi163-83845-39"}]': finished 2024-02-19T21:07:37.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:36.222117+0000 mon.a (mon.0) 1917 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:37.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:36.222188+0000 mon.a (mon.0) 1918 : audit [INF] from='client.48259 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-19T21:07:37.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:36.222227+0000 mon.a (mon.0) 1919 : cluster [DBG] osdmap e276: 8 total, 8 up, 7 in 2024-02-19T21:07:37.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:36.264955+0000 mon.b (mon.1) 426 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:37.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:36.265298+0000 mon.a (mon.0) 1920 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:37.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:36.271564+0000 mon.b (mon.1) 427 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:37.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:36.275417+0000 mon.a (mon.0) 1921 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:37.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:36.276031+0000 mon.b (mon.1) 428 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:37.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:36.276242+0000 mon.a (mon.0) 1922 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:37.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:36.276717+0000 mon.b (mon.1) 429 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi163-83845-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:37.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:36.276933+0000 mon.a (mon.0) 1923 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi163-83845-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:37.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:37.168069+0000 mon.b (mon.1) 430 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "38"}]: dispatch 2024-02-19T21:07:37.488 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:37.169101+0000 mon.a (mon.0) 1924 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "38"}]: dispatch 2024-02-19T21:07:37.514 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:37.815 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-19T21:07:38.067 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:07:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:07:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:07:38.122 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:07:38.363 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:38.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:38.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:38.468 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:38.507 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:38.582 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:38.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:38.608 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:36.722510+0000 osd.4 (osd.4) 131 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:07:38.608 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:36.723483+0000 osd.4 (osd.4) 132 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:07:38.608 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:37.167536+0000 mgr.y (mgr.14550) 282 : cluster [DBG] pgmap v386: 243 pgs: 8 unknown, 1 clean+premerge+peered, 234 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:07:38.608 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:37.221231+0000 mon.a (mon.0) 1925 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]': finished 2024-02-19T21:07:38.608 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:37.221325+0000 mon.a (mon.0) 1926 : audit [INF] from='client.48268 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi163-83845-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:38.609 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:37.221406+0000 mon.a (mon.0) 1927 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "38"}]': finished 2024-02-19T21:07:38.609 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:37.221469+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e277: 8 total, 8 up, 7 in 2024-02-19T21:07:38.609 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:37.232937+0000 mon.b (mon.1) 431 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi163-83845-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:38.609 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:37.235392+0000 mon.a (mon.0) 1929 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi163-83845-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:38.686 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:38.711 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:36.722510+0000 osd.4 (osd.4) 131 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:07:38.711 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:36.723483+0000 osd.4 (osd.4) 132 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:07:38.711 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:37.167536+0000 mgr.y (mgr.14550) 282 : cluster [DBG] pgmap v386: 243 pgs: 8 unknown, 1 clean+premerge+peered, 234 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:07:38.711 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:37.221231+0000 mon.a (mon.0) 1925 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]': finished 2024-02-19T21:07:38.712 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:37.221325+0000 mon.a (mon.0) 1926 : audit [INF] from='client.48268 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi163-83845-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:38.712 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:37.221406+0000 mon.a (mon.0) 1927 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "38"}]': finished 2024-02-19T21:07:38.712 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:37.221469+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e277: 8 total, 8 up, 7 in 2024-02-19T21:07:38.712 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:37.232937+0000 mon.b (mon.1) 431 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi163-83845-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:38.712 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:37.235392+0000 mon.a (mon.0) 1929 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi163-83845-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:38.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:36.722510+0000 osd.4 (osd.4) 131 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:07:38.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:36.723483+0000 osd.4 (osd.4) 132 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:07:38.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:37.167536+0000 mgr.y (mgr.14550) 282 : cluster [DBG] pgmap v386: 243 pgs: 8 unknown, 1 clean+premerge+peered, 234 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:07:38.713 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:37.221231+0000 mon.a (mon.0) 1925 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-33"}]': finished 2024-02-19T21:07:38.713 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:37.221325+0000 mon.a (mon.0) 1926 : audit [INF] from='client.48268 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi163-83845-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:38.713 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:37.221406+0000 mon.a (mon.0) 1927 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "38"}]': finished 2024-02-19T21:07:38.713 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:37.221469+0000 mon.a (mon.0) 1928 : cluster [DBG] osdmap e277: 8 total, 8 up, 7 in 2024-02-19T21:07:38.713 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:37.232937+0000 mon.b (mon.1) 431 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi163-83845-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:38.713 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:37.235392+0000 mon.a (mon.0) 1929 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi163-83845-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:38.713 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:38.811 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:38.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:38.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:38.915 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:07:38.949 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:38.993 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:07:39.004 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:39.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:07:39.105 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:07:39.135 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:39.259 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:07:39.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:39.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:07:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.359+0000 7ff7395ef700 1 -- 172.21.15.163:0/3068389620 >> v1:172.21.15.163:6789/0 conn(0x7ff7341042b0 legacy=0x7ff734106730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.359+0000 7ff7395ef700 1 -- 172.21.15.163:0/3068389620 shutdown_connections 2024-02-19T21:07:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.359+0000 7ff7395ef700 1 -- 172.21.15.163:0/3068389620 >> 172.21.15.163:0/3068389620 conn(0x7ff7340f3590 msgr2=0x7ff7340f59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.359+0000 7ff7395ef700 1 -- 172.21.15.163:0/3068389620 shutdown_connections 2024-02-19T21:07:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.359+0000 7ff7395ef700 1 -- 172.21.15.163:0/3068389620 wait complete. 2024-02-19T21:07:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff7395ef700 1 librados: starting msgr at 2024-02-19T21:07:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff7395ef700 1 librados: starting objecter 2024-02-19T21:07:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff7395ef700 1 Processor -- start 2024-02-19T21:07:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff7395ef700 1 -- start start 2024-02-19T21:07:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff7395ef700 1 librados: setting wanted keys 2024-02-19T21:07:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff7395ef700 1 librados: calling monclient init 2024-02-19T21:07:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff7395ef700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff734117120 con 0x7ff734100770 2024-02-19T21:07:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff7395ef700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff734118300 con 0x7ff7340fd970 2024-02-19T21:07:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff7395ef700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff734119500 con 0x7ff7341042b0 2024-02-19T21:07:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff7327fc700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ff734100770 0x7ff7340f9e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:47864/0 (socket says 172.21.15.163:47864) 2024-02-19T21:07:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff7327fc700 1 -- 172.21.15.163:0/371900368 learned_addr learned my addr 172.21.15.163:0/371900368 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff71bfff700 10 client.?.objecter ms_handle_connect 0x7ff734100770 2024-02-19T21:07:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff71bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff71bfff700 10 client.?.objecter ms_handle_connect 0x7ff7340fd970 2024-02-19T21:07:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff71bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff71bfff700 10 client.?.objecter ms_handle_connect 0x7ff7341042b0 2024-02-19T21:07:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff71bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157856896 0 0) 0x7ff734117120 con 0x7ff734100770 2024-02-19T21:07:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.360+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff71c003800 con 0x7ff734100770 2024-02-19T21:07:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302751866 0 0) 0x7ff734118300 con 0x7ff7340fd970 2024-02-19T21:07:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff734117120 con 0x7ff7340fd970 2024-02-19T21:07:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746110161 0 0) 0x7ff734119500 con 0x7ff7341042b0 2024-02-19T21:07:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff734118300 con 0x7ff7341042b0 2024-02-19T21:07:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3565894095 0 0) 0x7ff734117120 con 0x7ff7340fd970 2024-02-19T21:07:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff734119500 con 0x7ff7340fd970 2024-02-19T21:07:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff7280019f0 con 0x7ff7340fd970 2024-02-19T21:07:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3553263483 0 0) 0x7ff71c003800 con 0x7ff734100770 2024-02-19T21:07:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff734117120 con 0x7ff734100770 2024-02-19T21:07:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2754094130 0 0) 0x7ff734118300 con 0x7ff7341042b0 2024-02-19T21:07:39.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff71c003800 con 0x7ff7341042b0 2024-02-19T21:07:39.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff724001990 con 0x7ff734100770 2024-02-19T21:07:39.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff72c003150 con 0x7ff7341042b0 2024-02-19T21:07:39.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3877799943 0 0) 0x7ff734119500 con 0x7ff7340fd970 2024-02-19T21:07:39.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 >> v1:172.21.15.163:6790/0 conn(0x7ff7341042b0 legacy=0x7ff734115930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:39.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 >> v1:172.21.15.163:6789/0 conn(0x7ff734100770 legacy=0x7ff7340f9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:39.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73411a700 con 0x7ff7340fd970 2024-02-19T21:07:39.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff7395ef700 1 -- 172.21.15.163:0/371900368 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff734117310 con 0x7ff7340fd970 2024-02-19T21:07:39.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff7395ef700 10 client.48274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:39.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff7395ef700 1 -- 172.21.15.163:0/371900368 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff734117800 con 0x7ff7340fd970 2024-02-19T21:07:39.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff7395ef700 1 librados: init done 2024-02-19T21:07:39.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff728002360 con 0x7ff7340fd970 2024-02-19T21:07:39.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.361+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff7280033c0 con 0x7ff7340fd970 2024-02-19T21:07:39.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.362+0000 7ff7395ef700 1 -- 172.21.15.163:0/371900368 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff720004f20 con 0x7ff7340fd970 2024-02-19T21:07:39.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.363+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ff728001ec0 con 0x7ff7340fd970 2024-02-19T21:07:39.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.363+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(279..279 src has 43..279) v4 ==== 8944+0+0 (unknown 3535788686 0 0) 0x7ff728076700 con 0x7ff7340fd970 2024-02-19T21:07:39.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.363+0000 7ff71bfff700 10 client.48274.objecter ms_dispatch 0x7ff734001550 osd_map(279..279 src has 43..279) v4 2024-02-19T21:07:39.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.363+0000 7ff71bfff700 3 client.48274.objecter handle_osd_map got epochs [279,279] > 0 2024-02-19T21:07:39.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.363+0000 7ff71bfff700 3 client.48274.objecter handle_osd_map decoding full epoch 279 2024-02-19T21:07:39.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.363+0000 7ff71bfff700 20 client.48274.objecter dump_active .. 0 homeless 2024-02-19T21:07:39.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.363+0000 7ff71bfff700 10 client.48274.objecter ms_handle_connect 0x7ff71c063ea0 2024-02-19T21:07:39.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.367+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff728049780 con 0x7ff7340fd970 2024-02-19T21:07:39.363 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:39.390 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:39.408 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:39.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:39.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:39.512 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:39.539 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:39.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:39.552+0000 7ff7395ef700 1 -- 172.21.15.163:0/371900368 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7ff720005750 con 0x7ff7340fd970 2024-02-19T21:07:39.567 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:39.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:39.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:39.589 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:38.277424+0000 mon.b (mon.1) 432 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:39.589 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:38.467048+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e278: 8 total, 8 up, 7 in 2024-02-19T21:07:39.590 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:38.467805+0000 mon.a (mon.0) 1931 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:39.590 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:38.467915+0000 mon.b (mon.1) 433 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:39.590 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:38.474638+0000 mon.a (mon.0) 1932 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:39.590 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:39.097884+0000 mon.a (mon.0) 1933 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:39.590 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:39.207324+0000 mon.a (mon.0) 1934 : audit [INF] from='client.48268 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi163-83845-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi163-83845-40"}]': finished 2024-02-19T21:07:39.590 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:39.207435+0000 mon.a (mon.0) 1935 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:39.590 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:39.207510+0000 mon.a (mon.0) 1936 : audit [INF] from='client.48253 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi163-83854-49"}]': finished 2024-02-19T21:07:39.590 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:39.207567+0000 mon.a (mon.0) 1937 : cluster [DBG] osdmap e279: 8 total, 8 up, 7 in 2024-02-19T21:07:39.590 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:39.211099+0000 mon.b (mon.1) 434 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:39.590 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:39.212017+0000 mon.a (mon.0) 1938 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:39.591 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:39.213690+0000 mon.b (mon.1) 435 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:39.591 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:39.223573+0000 mon.a (mon.0) 1939 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:39.672 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:39.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:38.277424+0000 mon.b (mon.1) 432 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:39.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:38.467048+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e278: 8 total, 8 up, 7 in 2024-02-19T21:07:39.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:38.467805+0000 mon.a (mon.0) 1931 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:39.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:38.467915+0000 mon.b (mon.1) 433 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:39.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:38.474638+0000 mon.a (mon.0) 1932 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:39.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:39.097884+0000 mon.a (mon.0) 1933 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:39.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:39.207324+0000 mon.a (mon.0) 1934 : audit [INF] from='client.48268 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi163-83845-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi163-83845-40"}]': finished 2024-02-19T21:07:39.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:39.207435+0000 mon.a (mon.0) 1935 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:39.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:39.207510+0000 mon.a (mon.0) 1936 : audit [INF] from='client.48253 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi163-83854-49"}]': finished 2024-02-19T21:07:39.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:39.207567+0000 mon.a (mon.0) 1937 : cluster [DBG] osdmap e279: 8 total, 8 up, 7 in 2024-02-19T21:07:39.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:39.211099+0000 mon.b (mon.1) 434 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:39.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:39.212017+0000 mon.a (mon.0) 1938 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:39.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:39.213690+0000 mon.b (mon.1) 435 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:39.699 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:39.223573+0000 mon.a (mon.0) 1939 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:39.699 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:38.277424+0000 mon.b (mon.1) 432 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:39.699 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:38.467048+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e278: 8 total, 8 up, 7 in 2024-02-19T21:07:39.699 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:38.467805+0000 mon.a (mon.0) 1931 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:39.699 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:38.467915+0000 mon.b (mon.1) 433 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:39.699 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:38.474638+0000 mon.a (mon.0) 1932 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:39.699 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:39.097884+0000 mon.a (mon.0) 1933 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:39.699 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:39.207324+0000 mon.a (mon.0) 1934 : audit [INF] from='client.48268 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi163-83845-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi163-83845-40"}]': finished 2024-02-19T21:07:39.699 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:39.207435+0000 mon.a (mon.0) 1935 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:39.700 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:39.207510+0000 mon.a (mon.0) 1936 : audit [INF] from='client.48253 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi163-83854-49"}]': finished 2024-02-19T21:07:39.700 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:39.207567+0000 mon.a (mon.0) 1937 : cluster [DBG] osdmap e279: 8 total, 8 up, 7 in 2024-02-19T21:07:39.700 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:39.211099+0000 mon.b (mon.1) 434 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:39.700 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:39.212017+0000 mon.a (mon.0) 1938 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33"}]: dispatch 2024-02-19T21:07:39.701 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:39.213690+0000 mon.b (mon.1) 435 : audit [INF] from='client.? v1:172.21.15.163:0/2746902884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:39.701 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:39.223573+0000 mon.a (mon.0) 1939 : audit [INF] from='client.48253 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi163-83854-49"}]: dispatch 2024-02-19T21:07:39.701 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:39.719 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:39.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:39.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:39.823 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:39.836 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:07:39.836 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-02-19T21:07:39.836 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-02-19T21:07:39.836 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-02-19T21:07:39.837 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.837 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35189.0:4 36.16 36:681f27cd:::2.obj:head [write 0~105] snapc 0=[] ondisk+write+known_if_redirected e103)", 2024-02-19T21:07:39.837 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:42.759848+0000", 2024-02-19T21:07:39.837 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 177.078476588, 2024-02-19T21:07:39.837 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.034936716, 2024-02-19T21:07:39.837 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.837 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:39.837 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.837 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35189", 2024-02-19T21:07:39.837 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3441459623", 2024-02-19T21:07:39.837 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 4 2024-02-19T21:07:39.837 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.837 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.837 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.837 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.838 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759848+0000", 2024-02-19T21:07:39.838 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.838 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.838 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.838 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.838 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759848+0000", 2024-02-19T21:07:39.838 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:07:39.838 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.838 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.838 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.838 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759846+0000", 2024-02-19T21:07:39.838 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8869999999999998e-06 2024-02-19T21:07:39.838 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.838 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.838 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.839 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759855+0000", 2024-02-19T21:07:39.839 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2349999999999999e-06 2024-02-19T21:07:39.839 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.839 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.839 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.839 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759863+0000", 2024-02-19T21:07:39.839 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2473999999999999e-05 2024-02-19T21:07:39.839 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.839 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.839 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.839 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759876+0000", 2024-02-19T21:07:39.839 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6888000000000003e-05 2024-02-19T21:07:39.839 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.839 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.839 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.840 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759913+0000", 2024-02-19T21:07:39.840 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5630000000000003e-06 2024-02-19T21:07:39.840 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.840 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.840 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:07:39.840 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759920+0000", 2024-02-19T21:07:39.840 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.99619900800000005 2024-02-19T21:07:39.840 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.840 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.840 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.840 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.756119+0000", 2024-02-19T21:07:39.840 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000116972 2024-02-19T21:07:39.840 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.840 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.841 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.841 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.756236+0000", 2024-02-19T21:07:39.841 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015304199999999999 2024-02-19T21:07:39.841 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.841 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.841 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,3", 2024-02-19T21:07:39.841 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.756389+0000", 2024-02-19T21:07:39.841 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011827809999999999 2024-02-19T21:07:39.841 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.841 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.841 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:07:39.841 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.768217+0000", 2024-02-19T21:07:39.841 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011764907 2024-02-19T21:07:39.841 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.842 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.842 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:39.842 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.779982+0000", 2024-02-19T21:07:39.842 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014710997999999999 2024-02-19T21:07:39.842 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.842 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.842 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:39.842 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.794693+0000", 2024-02-19T21:07:39.842 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1137999999999997e-05 2024-02-19T21:07:39.842 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.842 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.842 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:39.842 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.794744+0000", 2024-02-19T21:07:39.842 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0725000000000001e-05 2024-02-19T21:07:39.843 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.843 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.843 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.843 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.794785+0000", 2024-02-19T21:07:39.843 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.034936716 2024-02-19T21:07:39.843 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.843 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.843 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.843 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.843 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.843 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25213.0:3 34.9 34:9294e480:nspace::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e103)", 2024-02-19T21:07:39.843 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:42.759875+0000", 2024-02-19T21:07:39.843 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 177.07844954000001, 2024-02-19T21:07:39.843 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.053943584, 2024-02-19T21:07:39.843 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.844 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:39.844 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.844 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25213", 2024-02-19T21:07:39.844 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2094902030", 2024-02-19T21:07:39.844 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 3 2024-02-19T21:07:39.844 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.844 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.844 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.844 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.844 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759875+0000", 2024-02-19T21:07:39.844 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.844 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.844 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.844 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759875+0000", 2024-02-19T21:07:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:07:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759874+0000", 2024-02-19T21:07:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4439999999999999e-06 2024-02-19T21:07:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759879+0000", 2024-02-19T21:07:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8599999999999999e-06 2024-02-19T21:07:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.845 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.846 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759888+0000", 2024-02-19T21:07:39.846 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7299999999999996e-06 2024-02-19T21:07:39.846 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.846 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.846 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.846 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759893+0000", 2024-02-19T21:07:39.846 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1069000000000003e-05 2024-02-19T21:07:39.846 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.846 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.846 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.846 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759934+0000", 2024-02-19T21:07:39.846 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.422e-06 2024-02-19T21:07:39.846 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.846 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.846 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:07:39.847 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759937+0000", 2024-02-19T21:07:39.847 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0133646970000001 2024-02-19T21:07:39.847 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.847 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.847 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.847 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.773302+0000", 2024-02-19T21:07:39.847 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012570699999999999 2024-02-19T21:07:39.847 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.847 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.847 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.847 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.773427+0000", 2024-02-19T21:07:39.847 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000139443 2024-02-19T21:07:39.847 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.847 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.847 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-02-19T21:07:39.848 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.773567+0000", 2024-02-19T21:07:39.848 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0052079140000000001 2024-02-19T21:07:39.848 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.848 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.848 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:07:39.848 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.778775+0000", 2024-02-19T21:07:39.848 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.030842405999999999 2024-02-19T21:07:39.848 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.848 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.848 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:39.848 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.809617+0000", 2024-02-19T21:07:39.848 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0041308889999999996 2024-02-19T21:07:39.848 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.848 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.849 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:39.849 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.813748+0000", 2024-02-19T21:07:39.849 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3467999999999999e-05 2024-02-19T21:07:39.849 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.849 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.849 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:39.849 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.813782+0000", 2024-02-19T21:07:39.849 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6962999999999998e-05 2024-02-19T21:07:39.849 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.849 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.849 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.849 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.813819+0000", 2024-02-19T21:07:39.849 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.053943584 2024-02-19T21:07:39.849 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.850 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.850 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.850 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.850 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.850 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25105.0:3 40.6 40:602f83fe:::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:07:39.850 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:47.968492+0000", 2024-02-19T21:07:39.850 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 171.86983287199999, 2024-02-19T21:07:39.850 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.042153208999999997, 2024-02-19T21:07:39.850 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.850 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:39.850 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.850 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25105", 2024-02-19T21:07:39.850 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3624732563", 2024-02-19T21:07:39.850 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 3 2024-02-19T21:07:39.850 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.851 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.851 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.851 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.851 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968492+0000", 2024-02-19T21:07:39.851 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.851 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.851 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.851 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.851 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968492+0000", 2024-02-19T21:07:39.851 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:07:39.851 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.851 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.851 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.851 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968490+0000", 2024-02-19T21:07:39.851 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4610000000000008e-06 2024-02-19T21:07:39.852 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.852 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.852 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.852 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968498+0000", 2024-02-19T21:07:39.852 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4069999999999993e-06 2024-02-19T21:07:39.852 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.852 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.852 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.852 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968506+0000", 2024-02-19T21:07:39.852 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0049999999999998e-06 2024-02-19T21:07:39.852 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.852 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.852 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968513+0000", 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001534941 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.970048+0000", 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0030718680000000002 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.973120+0000", 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.023296391 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.853 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.996417+0000", 2024-02-19T21:07:39.854 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5544000000000006e-05 2024-02-19T21:07:39.854 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.854 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.854 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.854 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.996502+0000", 2024-02-19T21:07:39.854 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000144523 2024-02-19T21:07:39.854 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.854 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.854 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-02-19T21:07:39.854 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.996647+0000", 2024-02-19T21:07:39.854 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00039452900000000002 2024-02-19T21:07:39.854 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.854 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.854 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:07:39.855 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.997041+0000", 2024-02-19T21:07:39.855 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0016365290000000001 2024-02-19T21:07:39.855 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.855 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.855 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:39.855 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.998678+0000", 2024-02-19T21:07:39.855 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011897256 2024-02-19T21:07:39.855 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.855 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.855 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:39.855 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010575+0000", 2024-02-19T21:07:39.855 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0837000000000003e-05 2024-02-19T21:07:39.855 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.855 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.855 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:39.856 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010606+0000", 2024-02-19T21:07:39.856 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9029000000000002e-05 2024-02-19T21:07:39.856 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.856 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.856 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.856 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010645+0000", 2024-02-19T21:07:39.856 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.042153208999999997 2024-02-19T21:07:39.856 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.856 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.856 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.856 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.856 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.856 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25321.0:2 49.15 49:ad551702:::test:head [create,setxattr key (5) in=3b] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:07:39.856 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:47.976371+0000", 2024-02-19T21:07:39.857 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 171.86195381100001, 2024-02-19T21:07:39.857 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037843445000000003, 2024-02-19T21:07:39.857 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.857 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:39.857 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.857 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25321", 2024-02-19T21:07:39.857 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/826688492", 2024-02-19T21:07:39.857 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:07:39.857 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.857 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.857 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.857 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.857 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976371+0000", 2024-02-19T21:07:39.857 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.857 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.858 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.858 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.858 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976371+0000", 2024-02-19T21:07:39.858 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:07:39.858 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.858 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.858 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.858 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976369+0000", 2024-02-19T21:07:39.858 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4760000000000004e-06 2024-02-19T21:07:39.858 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.858 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.858 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.858 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976378+0000", 2024-02-19T21:07:39.858 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1931000000000001e-05 2024-02-19T21:07:39.859 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.859 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.859 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.859 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976400+0000", 2024-02-19T21:07:39.859 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.422e-06 2024-02-19T21:07:39.859 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.859 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.859 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.859 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976408+0000", 2024-02-19T21:07:39.859 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7206000000000001e-05 2024-02-19T21:07:39.859 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.859 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.859 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.859 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976466+0000", 2024-02-19T21:07:39.859 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3210000000000005e-06 2024-02-19T21:07:39.860 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.860 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.860 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:07:39.860 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976474+0000", 2024-02-19T21:07:39.860 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.028525920999999999 2024-02-19T21:07:39.860 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.860 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.860 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.860 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.005000+0000", 2024-02-19T21:07:39.860 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011637399999999999 2024-02-19T21:07:39.860 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.860 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.860 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.860 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.005116+0000", 2024-02-19T21:07:39.861 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013954899999999999 2024-02-19T21:07:39.861 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.861 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.861 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-02-19T21:07:39.861 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.005256+0000", 2024-02-19T21:07:39.861 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00050348099999999996 2024-02-19T21:07:39.861 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.861 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.861 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:07:39.861 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.005759+0000", 2024-02-19T21:07:39.861 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0074804399999999997 2024-02-19T21:07:39.861 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.861 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.861 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:39.861 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.013240+0000", 2024-02-19T21:07:39.862 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00090275499999999999 2024-02-19T21:07:39.862 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.862 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.862 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:39.862 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014143+0000", 2024-02-19T21:07:39.862 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3145999999999997e-05 2024-02-19T21:07:39.862 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.862 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.862 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:39.862 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014176+0000", 2024-02-19T21:07:39.862 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8489999999999999e-05 2024-02-19T21:07:39.862 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.862 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.862 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.862 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014214+0000", 2024-02-19T21:07:39.863 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037843445000000003 2024-02-19T21:07:39.863 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.863 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.863 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.863 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.863 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.863 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.24865.0:14 11.18 e108/99 11:1d8f7fe2:test-rados-api-smithi163-83952-5::foo:head v 108'1, mlcod=0'0)", 2024-02-19T21:07:39.863 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:47.991853+0000", 2024-02-19T21:07:39.863 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 171.846471119, 2024-02-19T21:07:39.863 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0047438740000000004, 2024-02-19T21:07:39.863 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.863 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:39.863 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.863 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.864 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.864 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991853+0000", 2024-02-19T21:07:39.864 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.864 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.864 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.864 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.864 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991853+0000", 2024-02-19T21:07:39.864 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:07:39.864 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.864 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.864 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.864 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991853+0000", 2024-02-19T21:07:39.864 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0939999999999998e-06 2024-02-19T21:07:39.864 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.865 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.865 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.865 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991858+0000", 2024-02-19T21:07:39.865 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8879999999999996e-06 2024-02-19T21:07:39.865 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.865 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.865 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.865 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991863+0000", 2024-02-19T21:07:39.865 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1400000000000002e-06 2024-02-19T21:07:39.865 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.865 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.865 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.865 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.991867+0000", 2024-02-19T21:07:39.865 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00050798900000000003 2024-02-19T21:07:39.865 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.866 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.866 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.866 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.992375+0000", 2024-02-19T21:07:39.866 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8471999999999999e-05 2024-02-19T21:07:39.866 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.866 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.866 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.866 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.992393+0000", 2024-02-19T21:07:39.866 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0041760570000000004 2024-02-19T21:07:39.866 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.866 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.866 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:39.866 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.996569+0000", 2024-02-19T21:07:39.866 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7812000000000001e-05 2024-02-19T21:07:39.866 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.867 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.867 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.867 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.996597+0000", 2024-02-19T21:07:39.867 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0047438740000000004 2024-02-19T21:07:39.867 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.867 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.867 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.867 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.867 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.867 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25105.0:4 40.6 40:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108)", 2024-02-19T21:07:39.867 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:48.010891+0000", 2024-02-19T21:07:39.867 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 171.827433834, 2024-02-19T21:07:39.867 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010449923999999999, 2024-02-19T21:07:39.867 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.868 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:07:39.868 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.868 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25105", 2024-02-19T21:07:39.868 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3624732563", 2024-02-19T21:07:39.868 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 4 2024-02-19T21:07:39.868 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.868 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.868 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.868 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.868 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010891+0000", 2024-02-19T21:07:39.868 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.868 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.868 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.868 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.869 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010891+0000", 2024-02-19T21:07:39.869 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:07:39.869 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.869 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.869 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.869 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010889+0000", 2024-02-19T21:07:39.869 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3759999999999999e-06 2024-02-19T21:07:39.869 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.869 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.869 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.869 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010894+0000", 2024-02-19T21:07:39.869 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7290000000000006e-06 2024-02-19T21:07:39.869 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.869 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010903+0000", 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9740000000000001e-06 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010908+0000", 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6906000000000002e-05 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010945+0000", 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5568000000000006e-05 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.870 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.871 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.011011+0000", 2024-02-19T21:07:39.871 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010224851 2024-02-19T21:07:39.871 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.871 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.871 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.871 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021235+0000", 2024-02-19T21:07:39.871 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8914e-05 2024-02-19T21:07:39.871 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.871 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.871 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.871 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021284+0000", 2024-02-19T21:07:39.871 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6223e-05 2024-02-19T21:07:39.871 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.871 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.872 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.872 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021341+0000", 2024-02-19T21:07:39.872 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010449923999999999 2024-02-19T21:07:39.872 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.872 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.872 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.872 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.872 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.872 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25105.0:5 40.6 40:602f83fe:::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:07:39.872 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:48.010924+0000", 2024-02-19T21:07:39.872 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 171.827400665, 2024-02-19T21:07:39.872 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011507269000000001, 2024-02-19T21:07:39.872 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.872 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:39.873 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.873 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25105", 2024-02-19T21:07:39.873 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3624732563", 2024-02-19T21:07:39.873 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 5 2024-02-19T21:07:39.873 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.873 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.873 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.873 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.873 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010924+0000", 2024-02-19T21:07:39.873 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.873 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.873 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.873 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.873 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010924+0000", 2024-02-19T21:07:39.873 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:07:39.874 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.874 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.874 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.874 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010923+0000", 2024-02-19T21:07:39.874 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.438e-06 2024-02-19T21:07:39.874 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.874 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.874 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.874 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010929+0000", 2024-02-19T21:07:39.874 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0610000000000001e-06 2024-02-19T21:07:39.874 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.874 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.874 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.874 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010935+0000", 2024-02-19T21:07:39.874 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1400000000000002e-06 2024-02-19T21:07:39.875 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.875 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.875 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.875 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010940+0000", 2024-02-19T21:07:39.875 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00023434000000000001 2024-02-19T21:07:39.875 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.875 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.875 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.875 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.011174+0000", 2024-02-19T21:07:39.875 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3954e-05 2024-02-19T21:07:39.875 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.875 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.875 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:07:39.875 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.011218+0000", 2024-02-19T21:07:39.875 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010153373 2024-02-19T21:07:39.876 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.876 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.876 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.876 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021371+0000", 2024-02-19T21:07:39.876 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9161999999999998e-05 2024-02-19T21:07:39.876 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.876 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.876 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.876 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021460+0000", 2024-02-19T21:07:39.876 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013853699999999999 2024-02-19T21:07:39.876 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.876 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.876 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-02-19T21:07:39.877 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021599+0000", 2024-02-19T21:07:39.877 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000369594 2024-02-19T21:07:39.877 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.877 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.877 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:07:39.877 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021969+0000", 2024-02-19T21:07:39.877 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000122967 2024-02-19T21:07:39.877 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.877 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.877 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:39.877 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.022092+0000", 2024-02-19T21:07:39.877 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000293484 2024-02-19T21:07:39.877 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.877 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.877 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:39.878 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.022385+0000", 2024-02-19T21:07:39.878 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.586e-05 2024-02-19T21:07:39.878 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.878 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.878 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:39.878 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.022411+0000", 2024-02-19T21:07:39.878 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0248e-05 2024-02-19T21:07:39.878 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.878 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.878 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.878 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.022431+0000", 2024-02-19T21:07:39.878 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011507269000000001 2024-02-19T21:07:39.878 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.878 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.879 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.879 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.879 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.879 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.32629.0:7 76.3s0 76:dfb9de0f:test-rados-api-smithi163-83944-12::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-02-19T21:07:39.879 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:11.017799+0000", 2024-02-19T21:07:39.879 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 148.82052554699999, 2024-02-19T21:07:39.879 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0068870470000000003, 2024-02-19T21:07:39.879 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.879 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:39.879 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.879 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.32629", 2024-02-19T21:07:39.879 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1643190421", 2024-02-19T21:07:39.879 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 7 2024-02-19T21:07:39.879 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.879 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.880 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.880 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.880 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017799+0000", 2024-02-19T21:07:39.880 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.880 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.880 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.880 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.880 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017799+0000", 2024-02-19T21:07:39.880 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:07:39.880 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.880 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.880 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.880 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017797+0000", 2024-02-19T21:07:39.880 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3829999999999992e-06 2024-02-19T21:07:39.881 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.881 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.881 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.881 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017805+0000", 2024-02-19T21:07:39.881 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3729999999999999e-06 2024-02-19T21:07:39.881 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.881 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.881 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.881 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017815+0000", 2024-02-19T21:07:39.881 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0319999999999998e-06 2024-02-19T21:07:39.881 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.881 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.881 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.881 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017822+0000", 2024-02-19T21:07:39.881 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3600999999999999e-05 2024-02-19T21:07:39.882 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.882 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.882 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.882 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017855+0000", 2024-02-19T21:07:39.882 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000114827 2024-02-19T21:07:39.882 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.882 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.882 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.882 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017970+0000", 2024-02-19T21:07:39.882 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00040942799999999998 2024-02-19T21:07:39.882 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.882 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.882 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:07:39.882 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.018379+0000", 2024-02-19T21:07:39.882 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00058837500000000003 2024-02-19T21:07:39.883 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.883 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.883 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:07:39.883 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.018968+0000", 2024-02-19T21:07:39.883 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0056430810000000003 2024-02-19T21:07:39.883 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.883 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.883 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:39.883 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.024611+0000", 2024-02-19T21:07:39.883 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5131999999999998e-05 2024-02-19T21:07:39.883 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.883 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.883 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.883 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.024686+0000", 2024-02-19T21:07:39.884 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0068870470000000003 2024-02-19T21:07:39.884 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.884 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.884 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.884 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.884 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.884 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.32629.0:9 76.3s0 76:dfb9de0f:test-rados-api-smithi163-83944-12::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-02-19T21:07:39.884 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.093636+0000", 2024-02-19T21:07:39.884 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.74468887200001, 2024-02-19T21:07:39.884 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.084125466999999995, 2024-02-19T21:07:39.884 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.884 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:39.884 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.885 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.32629", 2024-02-19T21:07:39.885 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1643190421", 2024-02-19T21:07:39.885 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 9 2024-02-19T21:07:39.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.885 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.885 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.885 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.885 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093636+0000", 2024-02-19T21:07:39.885 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.885 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.885 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.885 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093636+0000", 2024-02-19T21:07:39.885 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:07:39.885 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.886 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.886 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.886 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093633+0000", 2024-02-19T21:07:39.886 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4586000000000001e-05 2024-02-19T21:07:39.886 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.886 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.886 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.886 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093648+0000", 2024-02-19T21:07:39.886 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4055e-05 2024-02-19T21:07:39.886 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.886 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.886 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.886 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093662+0000", 2024-02-19T21:07:39.886 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7350000000000002e-06 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093670+0000", 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1807000000000001e-05 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093722+0000", 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0189000000000001e-05 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093802+0000", 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00059241199999999995 2024-02-19T21:07:39.887 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.888 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.888 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:07:39.888 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.094395+0000", 2024-02-19T21:07:39.888 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0013219200000000001 2024-02-19T21:07:39.888 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.888 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.888 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:07:39.888 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.095717+0000", 2024-02-19T21:07:39.888 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.081990402000000004 2024-02-19T21:07:39.888 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.888 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.888 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:39.888 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.177707+0000", 2024-02-19T21:07:39.888 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4097000000000001e-05 2024-02-19T21:07:39.889 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.889 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.889 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.889 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.177761+0000", 2024-02-19T21:07:39.889 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.084125466999999995 2024-02-19T21:07:39.889 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.889 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.889 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.889 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.889 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.889 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.32629.0:10 76.3s0 76:dfb9de0f:test-rados-api-smithi163-83944-12::foo:head [rollback 5] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-02-19T21:07:39.889 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.178134+0000", 2024-02-19T21:07:39.889 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.66019008399999, 2024-02-19T21:07:39.889 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015877571, 2024-02-19T21:07:39.890 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.890 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:39.890 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.890 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.32629", 2024-02-19T21:07:39.890 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1643190421", 2024-02-19T21:07:39.890 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 10 2024-02-19T21:07:39.890 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.890 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.890 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.890 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.890 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178134+0000", 2024-02-19T21:07:39.890 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.890 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.890 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.890 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.891 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178134+0000", 2024-02-19T21:07:39.891 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:07:39.891 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.891 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.891 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.891 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178133+0000", 2024-02-19T21:07:39.891 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6609999999999997e-06 2024-02-19T21:07:39.891 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.891 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.891 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.891 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178138+0000", 2024-02-19T21:07:39.891 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7979999999999995e-06 2024-02-19T21:07:39.891 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.891 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.891 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.892 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178147+0000", 2024-02-19T21:07:39.892 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7389999999999999e-06 2024-02-19T21:07:39.892 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.892 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.892 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.892 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178152+0000", 2024-02-19T21:07:39.892 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7440000000000002e-05 2024-02-19T21:07:39.892 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.892 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.892 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.892 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178179+0000", 2024-02-19T21:07:39.892 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4633000000000003e-05 2024-02-19T21:07:39.892 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.892 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.893 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.893 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178244+0000", 2024-02-19T21:07:39.893 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00033892500000000001 2024-02-19T21:07:39.893 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.893 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.893 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:07:39.893 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178583+0000", 2024-02-19T21:07:39.893 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025176799999999998 2024-02-19T21:07:39.893 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.893 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.893 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:07:39.893 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178835+0000", 2024-02-19T21:07:39.893 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015138621999999999 2024-02-19T21:07:39.893 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.193973+0000", 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8704000000000001e-05 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.194012+0000", 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015877571 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.32629.0:11 76.3s0 76:dfb9de0f:test-rados-api-smithi163-83944-12::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e132)", 2024-02-19T21:07:39.894 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.201158+0000", 2024-02-19T21:07:39.895 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.63716629999999, 2024-02-19T21:07:39.895 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011048186, 2024-02-19T21:07:39.895 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.895 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:07:39.895 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.895 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.32629", 2024-02-19T21:07:39.895 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1643190421", 2024-02-19T21:07:39.895 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 11 2024-02-19T21:07:39.895 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.895 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.895 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.895 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.895 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201158+0000", 2024-02-19T21:07:39.895 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.896 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.896 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.896 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.896 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201158+0000", 2024-02-19T21:07:39.896 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:07:39.896 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.896 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.896 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.896 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201157+0000", 2024-02-19T21:07:39.896 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0796e-05 2024-02-19T21:07:39.896 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.896 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.896 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.896 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201168+0000", 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9330000000000007e-06 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201176+0000", 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.229e-06 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201183+0000", 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9822999999999999e-05 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201213+0000", 2024-02-19T21:07:39.897 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0374000000000001e-05 2024-02-19T21:07:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201283+0000", 2024-02-19T21:07:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010923510000000001 2024-02-19T21:07:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.212206+0000", 2024-02-19T21:07:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011048186 2024-02-19T21:07:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.899 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:252 61.16 61:68726f76:test-rados-api-smithi163-83928-12::foo:head [writefull 0~5] snapc 2=[2] ondisk+write+known_if_redirected e135)", 2024-02-19T21:07:39.899 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:15.184366+0000", 2024-02-19T21:07:39.899 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 144.65395869299999, 2024-02-19T21:07:39.899 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0086795729999999995, 2024-02-19T21:07:39.899 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.899 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:39.899 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.899 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:07:39.899 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:07:39.899 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 252 2024-02-19T21:07:39.899 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.899 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.899 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.899 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.899 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184366+0000", 2024-02-19T21:07:39.900 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.900 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.900 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.900 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.900 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184366+0000", 2024-02-19T21:07:39.900 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:07:39.900 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.900 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.900 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.900 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184363+0000", 2024-02-19T21:07:39.900 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1565e-05 2024-02-19T21:07:39.900 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.900 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.900 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.900 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184375+0000", 2024-02-19T21:07:39.901 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.38e-05 2024-02-19T21:07:39.901 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.901 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.901 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.901 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184388+0000", 2024-02-19T21:07:39.901 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.366e-05 2024-02-19T21:07:39.901 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.901 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.901 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.901 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184402+0000", 2024-02-19T21:07:39.901 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0022266690000000001 2024-02-19T21:07:39.901 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.901 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.901 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.901 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.186629+0000", 2024-02-19T21:07:39.902 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010571800000000001 2024-02-19T21:07:39.902 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.902 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.902 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.902 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.186734+0000", 2024-02-19T21:07:39.902 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00043829699999999998 2024-02-19T21:07:39.902 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.902 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.902 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 5,7", 2024-02-19T21:07:39.902 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.187173+0000", 2024-02-19T21:07:39.902 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0040489430000000002 2024-02-19T21:07:39.902 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.902 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.902 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:39.902 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.191222+0000", 2024-02-19T21:07:39.903 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0153000000000002e-05 2024-02-19T21:07:39.903 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.903 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.903 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:07:39.903 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.191312+0000", 2024-02-19T21:07:39.903 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001633362 2024-02-19T21:07:39.903 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.903 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.903 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:39.903 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.192945+0000", 2024-02-19T21:07:39.903 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9512000000000003e-05 2024-02-19T21:07:39.903 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.903 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.903 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:39.904 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.192995+0000", 2024-02-19T21:07:39.904 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0753e-05 2024-02-19T21:07:39.904 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.904 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.904 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.904 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193045+0000", 2024-02-19T21:07:39.904 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0086795729999999995 2024-02-19T21:07:39.904 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.904 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.904 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.904 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.904 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.904 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:259 85.1e 85:780b1894:test-rados-api-smithi163-83928-12::bar:2 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e138)", 2024-02-19T21:07:39.904 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:18.696167+0000", 2024-02-19T21:07:39.904 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 141.14215749300001, 2024-02-19T21:07:39.905 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0052543879999999996, 2024-02-19T21:07:39.905 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.905 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:07:39.905 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.905 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:07:39.905 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:07:39.905 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 259 2024-02-19T21:07:39.905 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.905 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.905 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.905 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.905 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696167+0000", 2024-02-19T21:07:39.905 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.905 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.905 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.906 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.906 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696167+0000", 2024-02-19T21:07:39.906 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:07:39.906 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.906 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.906 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.906 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696165+0000", 2024-02-19T21:07:39.906 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3890000000000002e-06 2024-02-19T21:07:39.906 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.906 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.906 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.906 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696172+0000", 2024-02-19T21:07:39.906 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0154e-05 2024-02-19T21:07:39.906 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.907 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.907 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.907 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696182+0000", 2024-02-19T21:07:39.907 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9739999999999999e-06 2024-02-19T21:07:39.907 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.907 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.907 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.907 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696188+0000", 2024-02-19T21:07:39.907 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6316999999999998e-05 2024-02-19T21:07:39.907 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.907 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.907 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.907 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696224+0000", 2024-02-19T21:07:39.907 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015598700000000001 2024-02-19T21:07:39.907 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.908 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.908 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:07:39.908 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.696380+0000", 2024-02-19T21:07:39.908 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00111491 2024-02-19T21:07:39.908 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.908 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.908 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.908 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.697495+0000", 2024-02-19T21:07:39.908 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014420899999999999 2024-02-19T21:07:39.908 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.908 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.908 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:07:39.908 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.697639+0000", 2024-02-19T21:07:39.909 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00075171600000000004 2024-02-19T21:07:39.909 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.909 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.909 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.909 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.698391+0000", 2024-02-19T21:07:39.909 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4733999999999994e-05 2024-02-19T21:07:39.909 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.909 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.909 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:07:39.909 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.698456+0000", 2024-02-19T21:07:39.909 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001527849 2024-02-19T21:07:39.909 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.909 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.909 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.910 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.699983+0000", 2024-02-19T21:07:39.910 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6997000000000003e-05 2024-02-19T21:07:39.910 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.910 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.910 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.910 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.700040+0000", 2024-02-19T21:07:39.910 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0013809650000000001 2024-02-19T21:07:39.910 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.910 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.910 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.910 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.701421+0000", 2024-02-19T21:07:39.910 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0052543879999999996 2024-02-19T21:07:39.910 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.910 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.910 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:260 85.10 85:0d6787e5:test-rados-api-smithi163-83928-12::baz:2 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e138)", 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:18.703035+0000", 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 141.135289672, 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0050535320000000003, 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 260 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.911 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.912 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703035+0000", 2024-02-19T21:07:39.912 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.912 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.912 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.912 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703035+0000", 2024-02-19T21:07:39.912 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:07:39.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.912 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.912 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.912 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703031+0000", 2024-02-19T21:07:39.912 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4063000000000001e-05 2024-02-19T21:07:39.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.912 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703045+0000", 2024-02-19T21:07:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5846e-05 2024-02-19T21:07:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703071+0000", 2024-02-19T21:07:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8336e-05 2024-02-19T21:07:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703110+0000", 2024-02-19T21:07:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5061999999999995e-05 2024-02-19T21:07:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.913 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703175+0000", 2024-02-19T21:07:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00024473799999999999 2024-02-19T21:07:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:07:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.703419+0000", 2024-02-19T21:07:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00071855499999999995 2024-02-19T21:07:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.704138+0000", 2024-02-19T21:07:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015597300000000001 2024-02-19T21:07:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.914 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.915 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:07:39.915 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.704294+0000", 2024-02-19T21:07:39.915 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00081258100000000005 2024-02-19T21:07:39.915 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.915 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.915 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.915 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.705107+0000", 2024-02-19T21:07:39.915 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8016999999999997e-05 2024-02-19T21:07:39.915 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.915 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.915 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:07:39.915 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.705165+0000", 2024-02-19T21:07:39.915 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001633406 2024-02-19T21:07:39.915 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.916 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.916 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.916 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.706798+0000", 2024-02-19T21:07:39.916 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2812999999999997e-05 2024-02-19T21:07:39.916 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.916 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.916 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.916 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.706851+0000", 2024-02-19T21:07:39.916 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0012376080000000001 2024-02-19T21:07:39.916 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.916 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.916 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.916 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:18.708088+0000", 2024-02-19T21:07:39.916 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0050535320000000003 2024-02-19T21:07:39.916 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.917 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.917 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.917 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.917 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.917 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.47503.0:2 93.18 93:1e0071ee:::oid_RoundTripSparseReadPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-02-19T21:07:39.917 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:22.198653+0000", 2024-02-19T21:07:39.917 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 137.63967109500001, 2024-02-19T21:07:39.917 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025958305000000001, 2024-02-19T21:07:39.917 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.917 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:39.917 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.917 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.47503", 2024-02-19T21:07:39.917 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1485143480", 2024-02-19T21:07:39.917 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:07:39.917 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.918 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.918 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.918 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.918 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198653+0000", 2024-02-19T21:07:39.918 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.918 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.918 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.918 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.918 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198653+0000", 2024-02-19T21:07:39.918 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:07:39.918 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.918 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.918 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.918 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198651+0000", 2024-02-19T21:07:39.918 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2534000000000001e-05 2024-02-19T21:07:39.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.919 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.919 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.919 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198663+0000", 2024-02-19T21:07:39.919 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8508999999999998e-05 2024-02-19T21:07:39.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.919 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.919 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.919 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198682+0000", 2024-02-19T21:07:39.919 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2040000000000003e-06 2024-02-19T21:07:39.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.919 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.919 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.919 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198691+0000", 2024-02-19T21:07:39.919 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6539999999999997e-05 2024-02-19T21:07:39.920 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.920 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.920 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.920 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198747+0000", 2024-02-19T21:07:39.920 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0387000000000001e-05 2024-02-19T21:07:39.920 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.920 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.920 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:07:39.920 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198758+0000", 2024-02-19T21:07:39.920 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015289329000000001 2024-02-19T21:07:39.920 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.920 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.920 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.920 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.214047+0000", 2024-02-19T21:07:39.921 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000131958 2024-02-19T21:07:39.921 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.921 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.921 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.921 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.214179+0000", 2024-02-19T21:07:39.921 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000165496 2024-02-19T21:07:39.921 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.921 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.921 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,5", 2024-02-19T21:07:39.921 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.214345+0000", 2024-02-19T21:07:39.921 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00067452899999999999 2024-02-19T21:07:39.921 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.921 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.921 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:07:39.922 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.215019+0000", 2024-02-19T21:07:39.922 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0069561409999999999 2024-02-19T21:07:39.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.922 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.922 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:39.922 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.221975+0000", 2024-02-19T21:07:39.922 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002534653 2024-02-19T21:07:39.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.922 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.922 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:39.922 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.224510+0000", 2024-02-19T21:07:39.922 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7840000000000002e-05 2024-02-19T21:07:39.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.922 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.922 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.224568+0000", 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4063999999999997e-05 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.224612+0000", 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025958305000000001 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:873 94.e 94:778c9cf1:test-rados-api-smithi163-83928-14::foo0:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:31.178555+0000", 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 128.65976980100001, 2024-02-19T21:07:39.923 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022321049999999999, 2024-02-19T21:07:39.924 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.924 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:07:39.924 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.924 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:07:39.924 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:07:39.924 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 873 2024-02-19T21:07:39.924 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.924 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.924 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.924 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.924 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178555+0000", 2024-02-19T21:07:39.924 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.924 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.924 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.924 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.925 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178555+0000", 2024-02-19T21:07:39.925 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:07:39.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.925 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.925 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.925 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178552+0000", 2024-02-19T21:07:39.925 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4859999999999997e-06 2024-02-19T21:07:39.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.925 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.925 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.925 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178560+0000", 2024-02-19T21:07:39.925 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0013999999999999e-05 2024-02-19T21:07:39.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.925 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.925 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.926 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178570+0000", 2024-02-19T21:07:39.926 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1750000000000005e-06 2024-02-19T21:07:39.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.926 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.926 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178579+0000", 2024-02-19T21:07:39.926 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5300999999999999e-05 2024-02-19T21:07:39.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.926 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.926 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178624+0000", 2024-02-19T21:07:39.926 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8860000000000001e-06 2024-02-19T21:07:39.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.926 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:07:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178630+0000", 2024-02-19T21:07:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0094360829999999996 2024-02-19T21:07:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.188066+0000", 2024-02-19T21:07:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022991600000000001 2024-02-19T21:07:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:07:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.188296+0000", 2024-02-19T21:07:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0070897570000000004 2024-02-19T21:07:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.927 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.195386+0000", 2024-02-19T21:07:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7982000000000001e-05 2024-02-19T21:07:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:07:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.195454+0000", 2024-02-19T21:07:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0051389959999999998 2024-02-19T21:07:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.200593+0000", 2024-02-19T21:07:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4277999999999996e-05 2024-02-19T21:07:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.928 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.929 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.929 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.200667+0000", 2024-02-19T21:07:39.929 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00020899300000000001 2024-02-19T21:07:39.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.929 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.929 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.929 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.200876+0000", 2024-02-19T21:07:39.929 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022321049999999999 2024-02-19T21:07:39.929 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.929 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.929 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.929 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.929 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:874 94.18 94:1a401a70:test-rados-api-smithi163-83928-14::foo5:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-19T21:07:39.929 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:31.201206+0000", 2024-02-19T21:07:39.930 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 128.63711808400001, 2024-02-19T21:07:39.930 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0051287449999999997, 2024-02-19T21:07:39.930 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.930 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:07:39.930 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.930 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:07:39.930 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:07:39.930 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 874 2024-02-19T21:07:39.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.930 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.930 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.930 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.930 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.201206+0000", 2024-02-19T21:07:39.930 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.201206+0000", 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.201204+0000", 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8189999999999992e-06 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.201212+0000", 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4073999999999999e-05 2024-02-19T21:07:39.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.932 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.932 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.932 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.201226+0000", 2024-02-19T21:07:39.932 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7000000000000008e-06 2024-02-19T21:07:39.932 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.932 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.932 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.932 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.201234+0000", 2024-02-19T21:07:39.932 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9146000000000001e-05 2024-02-19T21:07:39.932 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.932 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.932 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.932 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.201273+0000", 2024-02-19T21:07:39.932 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025459899999999999 2024-02-19T21:07:39.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.933 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.933 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:07:39.933 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.201527+0000", 2024-02-19T21:07:39.933 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0032692210000000001 2024-02-19T21:07:39.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.933 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.933 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.933 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.204796+0000", 2024-02-19T21:07:39.933 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000111468 2024-02-19T21:07:39.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.933 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.933 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:07:39.933 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.204908+0000", 2024-02-19T21:07:39.933 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00036806799999999999 2024-02-19T21:07:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.205276+0000", 2024-02-19T21:07:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1383999999999994e-05 2024-02-19T21:07:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.205337+0000", 2024-02-19T21:07:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00099780799999999998 2024-02-19T21:07:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.206335+0000", 2024-02-19T21:07:39.934 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0051287449999999997 2024-02-19T21:07:39.935 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.935 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.935 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.935 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.935 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.935 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:967 94.1d 94:bb38b02c:test-rados-api-smithi163-83928-14::foo54:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-19T21:07:39.935 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:31.370348+0000", 2024-02-19T21:07:39.935 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 128.46797669200001, 2024-02-19T21:07:39.935 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0052976730000000001, 2024-02-19T21:07:39.935 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.935 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:07:39.935 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:39.935 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:07:39.935 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:07:39.936 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 967 2024-02-19T21:07:39.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.936 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.936 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.936 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.936 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370348+0000", 2024-02-19T21:07:39.936 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.936 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.936 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.936 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370348+0000", 2024-02-19T21:07:39.936 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:07:39.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.936 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.936 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.937 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370347+0000", 2024-02-19T21:07:39.937 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7369999999999997e-06 2024-02-19T21:07:39.937 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.937 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.937 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.937 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370351+0000", 2024-02-19T21:07:39.937 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2239999999999995e-06 2024-02-19T21:07:39.937 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.937 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.937 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.937 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370360+0000", 2024-02-19T21:07:39.937 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6620000000000002e-06 2024-02-19T21:07:39.937 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.937 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.937 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.938 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370365+0000", 2024-02-19T21:07:39.938 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3609999999999998e-05 2024-02-19T21:07:39.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.938 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.938 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370399+0000", 2024-02-19T21:07:39.938 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013085899999999999 2024-02-19T21:07:39.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.938 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:07:39.938 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370530+0000", 2024-02-19T21:07:39.938 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001259874 2024-02-19T21:07:39.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.939 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.939 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.371790+0000", 2024-02-19T21:07:39.939 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00017583099999999999 2024-02-19T21:07:39.939 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.939 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.939 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:07:39.939 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.371966+0000", 2024-02-19T21:07:39.939 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00083226100000000005 2024-02-19T21:07:39.939 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.939 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.939 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.939 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.372798+0000", 2024-02-19T21:07:39.939 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8339999999999998e-05 2024-02-19T21:07:39.939 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.939 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.940 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:07:39.940 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.372886+0000", 2024-02-19T21:07:39.940 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00027153799999999999 2024-02-19T21:07:39.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.940 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.940 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.373158+0000", 2024-02-19T21:07:39.940 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3312000000000006e-05 2024-02-19T21:07:39.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.940 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.940 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.373221+0000", 2024-02-19T21:07:39.940 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002424537 2024-02-19T21:07:39.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.941 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.941 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.375646+0000", 2024-02-19T21:07:39.941 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0052976730000000001 2024-02-19T21:07:39.941 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.941 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.941 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.941 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.22476.0:2 137.a e196/195 137:53c7e431:::oid_AioUnlockPP_:head v 196'1, mlcod=0'0)", 2024-02-19T21:07:39.941 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:06:16.194139+0000", 2024-02-19T21:07:39.941 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 83.644185547000006, 2024-02-19T21:07:39.941 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0057252520000000001, 2024-02-19T21:07:39.941 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.941 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:39.942 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.942 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.942 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.942 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194139+0000", 2024-02-19T21:07:39.942 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.942 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.942 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.942 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194139+0000", 2024-02-19T21:07:39.942 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:07:39.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.942 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.942 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.942 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194138+0000", 2024-02-19T21:07:39.942 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1670000000000002e-06 2024-02-19T21:07:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194145+0000", 2024-02-19T21:07:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8575000000000001e-05 2024-02-19T21:07:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194164+0000", 2024-02-19T21:07:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3140000000000004e-06 2024-02-19T21:07:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194172+0000", 2024-02-19T21:07:39.943 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014666399999999999 2024-02-19T21:07:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194319+0000", 2024-02-19T21:07:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9656e-05 2024-02-19T21:07:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194339+0000", 2024-02-19T21:07:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.005498111 2024-02-19T21:07:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.944 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.199837+0000", 2024-02-19T21:07:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7557999999999999e-05 2024-02-19T21:07:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.199864+0000", 2024-02-19T21:07:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0057252520000000001 2024-02-19T21:07:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "MOSDECSubOpWrite(136.6s2 196/195 ECSubWrite(tid=402, reqid=client.47941.0:1, at_version=196'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-19T21:07:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:06:16.196362+0000", 2024-02-19T21:07:39.945 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 83.641962737, 2024-02-19T21:07:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010287370000000001, 2024-02-19T21:07:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-02-19T21:07:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196362+0000", 2024-02-19T21:07:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196362+0000", 2024-02-19T21:07:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:07:39.946 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196361+0000", 2024-02-19T21:07:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3603999999999999e-05 2024-02-19T21:07:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196395+0000", 2024-02-19T21:07:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.551e-06 2024-02-19T21:07:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196404+0000", 2024-02-19T21:07:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8769999999999993e-06 2024-02-19T21:07:39.947 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196411+0000", 2024-02-19T21:07:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00026643000000000001 2024-02-19T21:07:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196678+0000", 2024-02-19T21:07:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0366e-05 2024-02-19T21:07:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:07:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196688+0000", 2024-02-19T21:07:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0099375339999999996 2024-02-19T21:07:39.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:07:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.206626+0000", 2024-02-19T21:07:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3348000000000001e-05 2024-02-19T21:07:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.206649+0000", 2024-02-19T21:07:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010287370000000001 2024-02-19T21:07:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:39.949 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:39.950 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:07:39.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:39.951 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:39.951 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:39.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:40.052 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:40.082 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:40.101 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:40.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:40.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:40.204 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:07:40.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:40.211+0000 7ff71bfff700 1 -- 172.21.15.163:0/371900368 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v280) v1 ==== 87+0+0 (unknown 1296353566 0 0) 0x7ff728047b90 con 0x7ff7340fd970 2024-02-19T21:07:40.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:40.214+0000 7ff7395ef700 10 librados: watch_flush enter 2024-02-19T21:07:40.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:40.214+0000 7ff7395ef700 10 librados: watch_flush exit 2024-02-19T21:07:40.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:40.214+0000 7ff7395ef700 20 client.48274.objecter shutdown clearing up homeless session... 2024-02-19T21:07:40.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:40.214+0000 7ff7395ef700 10 client.48274.objecter successfully canceled tick 2024-02-19T21:07:40.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:40.214+0000 7ff7395ef700 1 -- 172.21.15.163:0/371900368 >> v1:172.21.15.163:6800/1552514681 conn(0x7ff71c063ea0 legacy=0x7ff71c066360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:40.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:40.215+0000 7ff7395ef700 1 -- 172.21.15.163:0/371900368 >> v1:172.21.15.193:6789/0 conn(0x7ff7340fd970 legacy=0x7ff7340f9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:40.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:40.215+0000 7ff7395ef700 1 -- 172.21.15.163:0/371900368 shutdown_connections 2024-02-19T21:07:40.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:40.215+0000 7ff7395ef700 1 -- 172.21.15.163:0/371900368 >> 172.21.15.163:0/371900368 conn(0x7ff7340f3590 msgr2=0x7ff734107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:40.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:40.215+0000 7ff7395ef700 1 -- 172.21.15.163:0/371900368 shutdown_connections 2024-02-19T21:07:40.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:40.216+0000 7ff7395ef700 1 -- 172.21.15.163:0/371900368 wait complete. 2024-02-19T21:07:40.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:40.216+0000 7ff7395ef700 1 librados: shutdown 2024-02-19T21:07:40.212 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:07:40.242 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:40.374 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:07:40.480 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:07:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:07:40.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:39.168066+0000 mgr.y (mgr.14550) 283 : cluster [DBG] pgmap v389: 235 pgs: 1 clean+premerge+peered, 234 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:07:40.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:39.553533+0000 mon.b (mon.1) 436 : audit [INF] from='client.? v1:172.21.15.163:0/371900368' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:07:40.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:39.554069+0000 mon.a (mon.0) 1940 : audit [INF] from='client.48274 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:07:40.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:40.204913+0000 mon.a (mon.0) 1941 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:40.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:40.204936+0000 mon.a (mon.0) 1942 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:40.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:40.208765+0000 mon.a (mon.0) 1943 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33"}]': finished 2024-02-19T21:07:40.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:40.208836+0000 mon.a (mon.0) 1944 : audit [INF] from='client.48253 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi163-83854-49"}]': finished 2024-02-19T21:07:40.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:40.208920+0000 mon.a (mon.0) 1945 : audit [INF] from='client.48274 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:07:40.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:40.208963+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e280: 8 total, 8 up, 7 in 2024-02-19T21:07:40.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:39.168066+0000 mgr.y (mgr.14550) 283 : cluster [DBG] pgmap v389: 235 pgs: 1 clean+premerge+peered, 234 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:07:40.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:39.553533+0000 mon.b (mon.1) 436 : audit [INF] from='client.? v1:172.21.15.163:0/371900368' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:07:40.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:39.554069+0000 mon.a (mon.0) 1940 : audit [INF] from='client.48274 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:07:40.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:40.204913+0000 mon.a (mon.0) 1941 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:40.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:40.204936+0000 mon.a (mon.0) 1942 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:40.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:40.208765+0000 mon.a (mon.0) 1943 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33"}]': finished 2024-02-19T21:07:40.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:40.208836+0000 mon.a (mon.0) 1944 : audit [INF] from='client.48253 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi163-83854-49"}]': finished 2024-02-19T21:07:40.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:40.208920+0000 mon.a (mon.0) 1945 : audit [INF] from='client.48274 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:07:40.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:40.208963+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e280: 8 total, 8 up, 7 in 2024-02-19T21:07:40.508 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-19T21:07:40.771 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:07:40.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:39.168066+0000 mgr.y (mgr.14550) 283 : cluster [DBG] pgmap v389: 235 pgs: 1 clean+premerge+peered, 234 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:07:40.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:39.553533+0000 mon.b (mon.1) 436 : audit [INF] from='client.? v1:172.21.15.163:0/371900368' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:07:40.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:39.554069+0000 mon.a (mon.0) 1940 : audit [INF] from='client.48274 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:07:40.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:40.204913+0000 mon.a (mon.0) 1941 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:40.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:40.204936+0000 mon.a (mon.0) 1942 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:40.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:40.208765+0000 mon.a (mon.0) 1943 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-33"}]': finished 2024-02-19T21:07:40.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:40.208836+0000 mon.a (mon.0) 1944 : audit [INF] from='client.48253 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi163-83854-49"}]': finished 2024-02-19T21:07:40.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:40.208920+0000 mon.a (mon.0) 1945 : audit [INF] from='client.48274 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:07:40.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:40.208963+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e280: 8 total, 8 up, 7 in 2024-02-19T21:07:41.024 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:07:41.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:41.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:07:41.087 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:41.128 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:07:41.154 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:41.354 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:07:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:41.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:07:41.458 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:41.486 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:41.584 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:41.586 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:40.243474+0000 mgr.y (mgr.14550) 284 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:41.586 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:41.168951+0000 mon.b (mon.1) 437 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "37"}]: dispatch 2024-02-19T21:07:41.586 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:41.170028+0000 mon.a (mon.0) 1947 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "37"}]: dispatch 2024-02-19T21:07:41.586 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:41.210537+0000 mon.a (mon.0) 1948 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "37"}]': finished 2024-02-19T21:07:41.586 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:41.210608+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e281: 8 total, 8 up, 7 in 2024-02-19T21:07:41.587 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:41.217703+0000 mon.c (mon.2) 332 : audit [INF] from='client.? v1:172.21.15.163:0/254556183' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:41.587 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:41.218900+0000 mon.a (mon.0) 1950 : audit [INF] from='client.49463 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:41.587 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:41.228687+0000 mon.b (mon.1) 438 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:41.587 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:41.472433+0000 mon.a (mon.0) 1951 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:41.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:40.243474+0000 mgr.y (mgr.14550) 284 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:41.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:41.168951+0000 mon.b (mon.1) 437 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "37"}]: dispatch 2024-02-19T21:07:41.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:41.170028+0000 mon.a (mon.0) 1947 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "37"}]: dispatch 2024-02-19T21:07:41.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:41.210537+0000 mon.a (mon.0) 1948 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "37"}]': finished 2024-02-19T21:07:41.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:41.210608+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e281: 8 total, 8 up, 7 in 2024-02-19T21:07:41.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:41.217703+0000 mon.c (mon.2) 332 : audit [INF] from='client.? v1:172.21.15.163:0/254556183' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:41.588 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:41.218900+0000 mon.a (mon.0) 1950 : audit [INF] from='client.49463 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:41.588 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:41.228687+0000 mon.b (mon.1) 438 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:41.588 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:41.472433+0000 mon.a (mon.0) 1951 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:41.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:41.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:41.690 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:07:41.718 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:41.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:40.243474+0000 mgr.y (mgr.14550) 284 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:41.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:41.168951+0000 mon.b (mon.1) 437 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "37"}]: dispatch 2024-02-19T21:07:41.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:41.170028+0000 mon.a (mon.0) 1947 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "37"}]: dispatch 2024-02-19T21:07:41.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:41.210537+0000 mon.a (mon.0) 1948 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "37"}]': finished 2024-02-19T21:07:41.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:41.210608+0000 mon.a (mon.0) 1949 : cluster [DBG] osdmap e281: 8 total, 8 up, 7 in 2024-02-19T21:07:41.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:41.217703+0000 mon.c (mon.2) 332 : audit [INF] from='client.? v1:172.21.15.163:0/254556183' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:41.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:41.218900+0000 mon.a (mon.0) 1950 : audit [INF] from='client.49463 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-50","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:41.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:41.228687+0000 mon.b (mon.1) 438 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:41.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:41.472433+0000 mon.a (mon.0) 1951 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:42.367 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:07:42.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:42.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:07:42.471 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:42.502 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:42.525 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:07:42.525 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:07:42.525 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-02-19T21:07:42.525 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:07:42.546 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:41.168499+0000 mgr.y (mgr.14550) 285 : cluster [DBG] pgmap v392: 242 pgs: 1 active+recovering, 2 active+clean+snaptrim, 2 peering, 8 creating+peering, 229 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 1023 B/s wr, 1 op/s; 1/948 objects misplaced (0.105%); 571 B/s, 64 objects/s recovering 2024-02-19T21:07:42.546 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:42.214042+0000 mon.a (mon.0) 1952 : audit [INF] from='client.49463 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-50","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:42.546 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:42.214284+0000 mon.a (mon.0) 1953 : audit [INF] from='client.48268 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi163-83845-40"}]': finished 2024-02-19T21:07:42.546 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:42.214351+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e282: 8 total, 8 up, 7 in 2024-02-19T21:07:42.546 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:42.227620+0000 mon.b (mon.1) 439 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:42.546 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:42.228346+0000 mon.a (mon.0) 1955 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:42.546 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:42.229034+0000 mon.b (mon.1) 440 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:42.546 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:42.229531+0000 mon.a (mon.0) 1956 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:42.576 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:42.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:41.168499+0000 mgr.y (mgr.14550) 285 : cluster [DBG] pgmap v392: 242 pgs: 1 active+recovering, 2 active+clean+snaptrim, 2 peering, 8 creating+peering, 229 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 1023 B/s wr, 1 op/s; 1/948 objects misplaced (0.105%); 571 B/s, 64 objects/s recovering 2024-02-19T21:07:42.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:42.214042+0000 mon.a (mon.0) 1952 : audit [INF] from='client.49463 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-50","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:42.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:42.214284+0000 mon.a (mon.0) 1953 : audit [INF] from='client.48268 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi163-83845-40"}]': finished 2024-02-19T21:07:42.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:42.214351+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e282: 8 total, 8 up, 7 in 2024-02-19T21:07:42.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:42.227620+0000 mon.b (mon.1) 439 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:42.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:42.228346+0000 mon.a (mon.0) 1955 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:42.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:42.229034+0000 mon.b (mon.1) 440 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:42.579 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:42.229531+0000 mon.a (mon.0) 1956 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:42.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:41.168499+0000 mgr.y (mgr.14550) 285 : cluster [DBG] pgmap v392: 242 pgs: 1 active+recovering, 2 active+clean+snaptrim, 2 peering, 8 creating+peering, 229 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 1023 B/s wr, 1 op/s; 1/948 objects misplaced (0.105%); 571 B/s, 64 objects/s recovering 2024-02-19T21:07:42.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:42.214042+0000 mon.a (mon.0) 1952 : audit [INF] from='client.49463 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi163-83854-50","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:42.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:42.214284+0000 mon.a (mon.0) 1953 : audit [INF] from='client.48268 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi163-83845-40"}]': finished 2024-02-19T21:07:42.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:42.214351+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e282: 8 total, 8 up, 7 in 2024-02-19T21:07:42.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:42.227620+0000 mon.b (mon.1) 439 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:42.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:42.228346+0000 mon.a (mon.0) 1955 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:42.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:42.229034+0000 mon.b (mon.1) 440 : audit [INF] from='client.? v1:172.21.15.163:0/748924268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:42.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:42.229531+0000 mon.a (mon.0) 1956 : audit [INF] from='client.48268 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi163-83845-40"}]: dispatch 2024-02-19T21:07:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:42.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:42.681 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:42.711 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:42.784 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:42.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:42.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:42.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.827+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3234237615 >> v1:172.21.15.163:6789/0 conn(0x7f4af81031b0 legacy=0x7f4af8105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:42.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.827+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3234237615 shutdown_connections 2024-02-19T21:07:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.827+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3234237615 >> 172.21.15.163:0/3234237615 conn(0x7f4af80f2470 msgr2=0x7f4af80f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.827+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3234237615 shutdown_connections 2024-02-19T21:07:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.827+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3234237615 wait complete. 2024-02-19T21:07:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4aff8b7700 1 librados: starting msgr at 2024-02-19T21:07:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4aff8b7700 1 librados: starting objecter 2024-02-19T21:07:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4aff8b7700 1 Processor -- start 2024-02-19T21:07:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4aff8b7700 1 -- start start 2024-02-19T21:07:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4aff8b7700 1 librados: setting wanted keys 2024-02-19T21:07:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4aff8b7700 1 librados: calling monclient init 2024-02-19T21:07:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4aff8b7700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4af81194d0 con 0x7f4af80ff670 2024-02-19T21:07:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4aff8b7700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4af811a6b0 con 0x7f4af81031b0 2024-02-19T21:07:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4aff8b7700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4af811b8b0 con 0x7f4af80fc870 2024-02-19T21:07:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4afd653700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f4af80fc870 0x7f4af80609e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:42576/0 (socket says 172.21.15.163:42576) 2024-02-19T21:07:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4afce52700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f4af80ff670 0x7f4af80610f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:47912/0 (socket says 172.21.15.163:47912) 2024-02-19T21:07:42.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4afd653700 1 -- 172.21.15.163:0/3821650192 learned_addr learned my addr 172.21.15.163:0/3821650192 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4aee7fc700 10 client.?.objecter ms_handle_connect 0x7f4af80fc870 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4aee7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4aee7fc700 10 client.?.objecter ms_handle_connect 0x7f4af80ff670 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4aee7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4aee7fc700 10 client.?.objecter ms_handle_connect 0x7f4af81031b0 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.828+0000 7f4aee7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579344442 0 0) 0x7f4af811b8b0 con 0x7f4af80fc870 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae40033f0 con 0x7f4af80fc870 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26361081 0 0) 0x7f4af811a6b0 con 0x7f4af81031b0 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af811b8b0 con 0x7f4af81031b0 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781965466 0 0) 0x7f4af81194d0 con 0x7f4af80ff670 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af811a6b0 con 0x7f4af80ff670 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1963139557 0 0) 0x7f4af811b8b0 con 0x7f4af81031b0 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4af81194d0 con 0x7f4af81031b0 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4af0002360 con 0x7f4af81031b0 2024-02-19T21:07:42.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1463240370 0 0) 0x7f4ae40033f0 con 0x7f4af80fc870 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4af811b8b0 con 0x7f4af80fc870 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4ae8001900 con 0x7f4af80fc870 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3762403380 0 0) 0x7f4af811a6b0 con 0x7f4af80ff670 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ae40033f0 con 0x7f4af80ff670 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4af40011d0 con 0x7f4af80ff670 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1759587286 0 0) 0x7f4af81194d0 con 0x7f4af81031b0 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 >> v1:172.21.15.163:6790/0 conn(0x7f4af80fc870 legacy=0x7f4af80609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 >> v1:172.21.15.163:6789/0 conn(0x7f4af80ff670 legacy=0x7f4af80610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af811cab0 con 0x7f4af81031b0 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af81196c0 con 0x7f4af81031b0 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aff8b7700 10 client.48286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af8119bb0 con 0x7f4af81031b0 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aff8b7700 1 librados: init done 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f4af0003140 con 0x7f4af81031b0 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.829+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4af0003960 con 0x7f4af81031b0 2024-02-19T21:07:42.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.830+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4adc004f00 con 0x7f4af81031b0 2024-02-19T21:07:42.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.830+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f4af0001ab0 con 0x7f4af81031b0 2024-02-19T21:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(282..282 src has 43..282) v4 ==== 8498+0+0 (unknown 3687398482 0 0) 0x7f4af0076c30 con 0x7f4af81031b0 2024-02-19T21:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 osd_map(282..282 src has 43..282) v4 2024-02-19T21:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aee7fc700 3 client.48286.objecter handle_osd_map got epochs [282,282] > 0 2024-02-19T21:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aee7fc700 3 client.48286.objecter handle_osd_map decoding full epoch 282 2024-02-19T21:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aee7fc700 20 client.48286.objecter dump_active .. 0 homeless 2024-02-19T21:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v282) v1 ==== 54+0+16 (unknown 1553799486 0 2420000192) 0x7f4af0076f90 con 0x7f4af81031b0 2024-02-19T21:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aee7fc700 10 client.48286.objecter ms_handle_connect 0x7f4ae4063c60 2024-02-19T21:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 1 2024-02-19T21:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc0051a0 osd=0 3 2024-02-19T21:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc0051a0 osd=0 3 2024-02-19T21:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc0051a0 osd=0 3 2024-02-19T21:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 1 2024-02-19T21:07:42.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc0051a0 osd=0 3 2024-02-19T21:07:42.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 0 1 2024-02-19T21:07:42.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc0051a0 osd=0 4 2024-02-19T21:07:42.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aff8b7700 10 client.48286.objecter _send_command 1 2024-02-19T21:07:42.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4adc004950 con 0x7f4adc004530 2024-02-19T21:07:42.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.831+0000 7f4aee7fc700 10 client.48286.objecter ms_handle_connect 0x7f4adc004530 2024-02-19T21:07:42.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.832+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4adc004950 con 0x7f4adc004530 2024-02-19T21:07:42.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.832+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:07:42.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.832+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 1 = system:0 2024-02-19T21:07:42.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.832+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc0051a0 osd=0 4 2024-02-19T21:07:42.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.832+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 0 1 2024-02-19T21:07:42.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.847+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:42.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.847+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 2 2024-02-19T21:07:42.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.847+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc0051a0 osd=0 3 2024-02-19T21:07:42.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.847+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc0051a0 osd=0 3 2024-02-19T21:07:42.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.847+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc0051a0 osd=0 3 2024-02-19T21:07:42.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.847+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 2 2024-02-19T21:07:42.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.847+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc0051a0 osd=0 3 2024-02-19T21:07:42.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.847+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 0 2 2024-02-19T21:07:42.842 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.847+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc0051a0 osd=0 4 2024-02-19T21:07:42.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.847+0000 7f4aff8b7700 10 client.48286.objecter _send_command 2 2024-02-19T21:07:42.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.847+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4adc004d30 con 0x7f4adc004530 2024-02-19T21:07:42.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4adc004d30 con 0x7f4adc004530 2024-02-19T21:07:42.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:42.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:42.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc0051a0 osd=0 4 2024-02-19T21:07:42.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 0 2 2024-02-19T21:07:42.844 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:07:42.844 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:42.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 3 2024-02-19T21:07:42.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc003fe0 osd=1 3 2024-02-19T21:07:42.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc003fe0 osd=1 3 2024-02-19T21:07:42.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc003fe0 osd=1 3 2024-02-19T21:07:42.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 3 2024-02-19T21:07:42.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc003fe0 osd=1 3 2024-02-19T21:07:42.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 1 3 2024-02-19T21:07:42.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc003fe0 osd=1 4 2024-02-19T21:07:42.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aff8b7700 10 client.48286.objecter _send_command 3 2024-02-19T21:07:42.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.848+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4adc00c000 con 0x7f4adc004150 2024-02-19T21:07:42.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.849+0000 7f4aee7fc700 10 client.48286.objecter ms_handle_connect 0x7f4adc004150 2024-02-19T21:07:42.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.849+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4adc00c000 con 0x7f4adc004150 2024-02-19T21:07:42.845 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.849+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:07:42.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.849+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 3 = system:0 2024-02-19T21:07:42.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.849+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc003fe0 osd=1 4 2024-02-19T21:07:42.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.849+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 1 3 2024-02-19T21:07:42.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.864+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:42.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.864+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 4 2024-02-19T21:07:42.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.864+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc003fe0 osd=1 3 2024-02-19T21:07:42.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.864+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc003fe0 osd=1 3 2024-02-19T21:07:42.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.864+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc003fe0 osd=1 3 2024-02-19T21:07:42.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.864+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 4 2024-02-19T21:07:42.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.864+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc003fe0 osd=1 3 2024-02-19T21:07:42.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.864+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 1 4 2024-02-19T21:07:42.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.864+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc003fe0 osd=1 4 2024-02-19T21:07:42.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.864+0000 7f4aff8b7700 10 client.48286.objecter _send_command 4 2024-02-19T21:07:42.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.864+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4adc0018b0 con 0x7f4adc004150 2024-02-19T21:07:42.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4adc0018b0 con 0x7f4adc004150 2024-02-19T21:07:42.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:42.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:42.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc003fe0 osd=1 4 2024-02-19T21:07:42.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 1 4 2024-02-19T21:07:42.861 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:07:42.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:42.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 5 2024-02-19T21:07:42.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc003630 osd=2 3 2024-02-19T21:07:42.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc003630 osd=2 3 2024-02-19T21:07:42.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc003630 osd=2 3 2024-02-19T21:07:42.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 5 2024-02-19T21:07:42.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc003630 osd=2 3 2024-02-19T21:07:42.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 2 5 2024-02-19T21:07:42.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc003630 osd=2 4 2024-02-19T21:07:42.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aff8b7700 10 client.48286.objecter _send_command 5 2024-02-19T21:07:42.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4adc00de90 con 0x7f4adc00d2f0 2024-02-19T21:07:42.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aee7fc700 10 client.48286.objecter ms_handle_connect 0x7f4adc00d2f0 2024-02-19T21:07:42.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.866+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4adc00de90 con 0x7f4adc00d2f0 2024-02-19T21:07:42.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.867+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:07:42.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.867+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 5 = system:0 2024-02-19T21:07:42.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.867+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc003630 osd=2 4 2024-02-19T21:07:42.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.867+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 2 5 2024-02-19T21:07:42.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.881+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:42.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.881+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 6 2024-02-19T21:07:42.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.881+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc003630 osd=2 3 2024-02-19T21:07:42.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.881+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc003630 osd=2 3 2024-02-19T21:07:42.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.881+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc003630 osd=2 3 2024-02-19T21:07:42.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.881+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 6 2024-02-19T21:07:42.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.881+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc003630 osd=2 3 2024-02-19T21:07:42.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.881+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 2 6 2024-02-19T21:07:42.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.881+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc003630 osd=2 4 2024-02-19T21:07:42.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.881+0000 7f4aff8b7700 10 client.48286.objecter _send_command 6 2024-02-19T21:07:42.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.881+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4adc0030a0 con 0x7f4adc00d2f0 2024-02-19T21:07:42.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4adc0030a0 con 0x7f4adc00d2f0 2024-02-19T21:07:42.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:42.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:42.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc003630 osd=2 4 2024-02-19T21:07:42.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 2 6 2024-02-19T21:07:42.878 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:07:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 7 2024-02-19T21:07:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc003480 osd=3 3 2024-02-19T21:07:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc003480 osd=3 3 2024-02-19T21:07:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc003480 osd=3 3 2024-02-19T21:07:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 7 2024-02-19T21:07:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc003480 osd=3 3 2024-02-19T21:07:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 3 7 2024-02-19T21:07:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc003480 osd=3 4 2024-02-19T21:07:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aff8b7700 10 client.48286.objecter _send_command 7 2024-02-19T21:07:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.882+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4adc015a60 con 0x7f4adc011e80 2024-02-19T21:07:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.883+0000 7f4aee7fc700 10 client.48286.objecter ms_handle_connect 0x7f4adc011e80 2024-02-19T21:07:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.883+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4adc015a60 con 0x7f4adc011e80 2024-02-19T21:07:42.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.883+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:07:42.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.883+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 7 = system:0 2024-02-19T21:07:42.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.883+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc003480 osd=3 4 2024-02-19T21:07:42.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.883+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 3 7 2024-02-19T21:07:42.888 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:42.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.897+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:42.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.897+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 8 2024-02-19T21:07:42.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.897+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc003480 osd=3 3 2024-02-19T21:07:42.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.897+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc003480 osd=3 3 2024-02-19T21:07:42.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.897+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc003480 osd=3 3 2024-02-19T21:07:42.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.897+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 8 2024-02-19T21:07:42.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.897+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc003480 osd=3 3 2024-02-19T21:07:42.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.897+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 3 8 2024-02-19T21:07:42.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.897+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc003480 osd=3 4 2024-02-19T21:07:42.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.897+0000 7f4aff8b7700 10 client.48286.objecter _send_command 8 2024-02-19T21:07:42.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.897+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4adc00e270 con 0x7f4adc011e80 2024-02-19T21:07:42.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4adc00e270 con 0x7f4adc011e80 2024-02-19T21:07:42.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:42.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:42.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc003480 osd=3 4 2024-02-19T21:07:42.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 3 8 2024-02-19T21:07:42.894 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:07:42.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:42.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 9 2024-02-19T21:07:42.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc00e410 osd=4 3 2024-02-19T21:07:42.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc00e410 osd=4 3 2024-02-19T21:07:42.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc00e410 osd=4 3 2024-02-19T21:07:42.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 9 2024-02-19T21:07:42.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc00e410 osd=4 3 2024-02-19T21:07:42.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 4 9 2024-02-19T21:07:42.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc00e410 osd=4 4 2024-02-19T21:07:42.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aff8b7700 10 client.48286.objecter _send_command 9 2024-02-19T21:07:42.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4adc00e580 con 0x7f4adc00ce50 2024-02-19T21:07:42.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.899+0000 7f4aee7fc700 10 client.48286.objecter ms_handle_connect 0x7f4adc00ce50 2024-02-19T21:07:42.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.900+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4adc00e580 con 0x7f4adc00ce50 2024-02-19T21:07:42.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.900+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:07:42.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.900+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 9 = system:0 2024-02-19T21:07:42.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.900+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc00e410 osd=4 4 2024-02-19T21:07:42.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.900+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 4 9 2024-02-19T21:07:42.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.914+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:42.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.914+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 10 2024-02-19T21:07:42.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.914+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc00e410 osd=4 3 2024-02-19T21:07:42.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.914+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc00e410 osd=4 3 2024-02-19T21:07:42.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.914+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc00e410 osd=4 3 2024-02-19T21:07:42.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.914+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 10 2024-02-19T21:07:42.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.914+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc00e410 osd=4 3 2024-02-19T21:07:42.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.914+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 4 10 2024-02-19T21:07:42.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.914+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc00e410 osd=4 4 2024-02-19T21:07:42.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.914+0000 7f4aff8b7700 10 client.48286.objecter _send_command 10 2024-02-19T21:07:42.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.914+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4adc00e580 con 0x7f4adc00ce50 2024-02-19T21:07:42.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4adc00e580 con 0x7f4adc00ce50 2024-02-19T21:07:42.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:42.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:42.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc00e410 osd=4 4 2024-02-19T21:07:42.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 4 10 2024-02-19T21:07:42.911 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:07:42.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:42.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 11 2024-02-19T21:07:42.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc00e580 osd=5 3 2024-02-19T21:07:42.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc00e580 osd=5 3 2024-02-19T21:07:42.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc00e580 osd=5 3 2024-02-19T21:07:42.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 11 2024-02-19T21:07:42.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc00e580 osd=5 3 2024-02-19T21:07:42.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 5 11 2024-02-19T21:07:42.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc00e580 osd=5 4 2024-02-19T21:07:42.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aff8b7700 10 client.48286.objecter _send_command 11 2024-02-19T21:07:42.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.916+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4adc01e660 con 0x7f4adc01a960 2024-02-19T21:07:42.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.917+0000 7f4aee7fc700 10 client.48286.objecter ms_handle_connect 0x7f4adc01a960 2024-02-19T21:07:42.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.917+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4adc01e660 con 0x7f4adc01a960 2024-02-19T21:07:42.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.917+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:07:42.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.917+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 11 = system:0 2024-02-19T21:07:42.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.917+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc00e580 osd=5 4 2024-02-19T21:07:42.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.917+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 5 11 2024-02-19T21:07:42.914 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:42.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.931+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:42.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.931+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 12 2024-02-19T21:07:42.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.931+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc00e580 osd=5 3 2024-02-19T21:07:42.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.931+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc00e580 osd=5 3 2024-02-19T21:07:42.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.931+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc00e580 osd=5 3 2024-02-19T21:07:42.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.931+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 12 2024-02-19T21:07:42.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.931+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc00e580 osd=5 3 2024-02-19T21:07:42.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.931+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 5 12 2024-02-19T21:07:42.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.931+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc00e580 osd=5 4 2024-02-19T21:07:42.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.931+0000 7f4aff8b7700 10 client.48286.objecter _send_command 12 2024-02-19T21:07:42.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.931+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4adc01e660 con 0x7f4adc01a960 2024-02-19T21:07:42.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4adc01e660 con 0x7f4adc01a960 2024-02-19T21:07:42.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:42.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:42.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc00e580 osd=5 4 2024-02-19T21:07:42.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 5 12 2024-02-19T21:07:42.929 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:07:42.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:42.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 13 2024-02-19T21:07:42.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc01e660 osd=6 3 2024-02-19T21:07:42.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc01e660 osd=6 3 2024-02-19T21:07:42.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc01e660 osd=6 3 2024-02-19T21:07:42.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 13 2024-02-19T21:07:42.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc01e660 osd=6 3 2024-02-19T21:07:42.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 6 13 2024-02-19T21:07:42.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc01e660 osd=6 4 2024-02-19T21:07:42.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aff8b7700 10 client.48286.objecter _send_command 13 2024-02-19T21:07:42.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.933+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4adc01f840 con 0x7f4adc020320 2024-02-19T21:07:42.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.934+0000 7f4aee7fc700 10 client.48286.objecter ms_handle_connect 0x7f4adc020320 2024-02-19T21:07:42.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.934+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4adc01f840 con 0x7f4adc020320 2024-02-19T21:07:42.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.934+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:07:42.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.934+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 13 = system:0 2024-02-19T21:07:42.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.934+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc01e660 osd=6 4 2024-02-19T21:07:42.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.934+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 6 13 2024-02-19T21:07:42.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.948+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:42.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.948+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 14 2024-02-19T21:07:42.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.948+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc01e660 osd=6 3 2024-02-19T21:07:42.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.948+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc01e660 osd=6 3 2024-02-19T21:07:42.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.948+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc01e660 osd=6 3 2024-02-19T21:07:42.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.948+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 14 2024-02-19T21:07:42.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.948+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc01e660 osd=6 3 2024-02-19T21:07:42.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.948+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 6 14 2024-02-19T21:07:42.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.948+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc01e660 osd=6 4 2024-02-19T21:07:42.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.948+0000 7f4aff8b7700 10 client.48286.objecter _send_command 14 2024-02-19T21:07:42.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.948+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4adc01f840 con 0x7f4adc020320 2024-02-19T21:07:42.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4adc01f840 con 0x7f4adc020320 2024-02-19T21:07:42.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:42.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:42.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc01e660 osd=6 4 2024-02-19T21:07:42.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 6 14 2024-02-19T21:07:42.946 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:07:42.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:42.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 15 2024-02-19T21:07:42.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc01f840 osd=7 3 2024-02-19T21:07:42.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc01f840 osd=7 3 2024-02-19T21:07:42.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc01f840 osd=7 3 2024-02-19T21:07:42.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 15 2024-02-19T21:07:42.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc01f840 osd=7 3 2024-02-19T21:07:42.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 7 15 2024-02-19T21:07:42.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc01f840 osd=7 4 2024-02-19T21:07:42.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aff8b7700 10 client.48286.objecter _send_command 15 2024-02-19T21:07:42.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.950+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4adc01fc70 con 0x7f4adc023520 2024-02-19T21:07:42.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.951+0000 7f4aee7fc700 10 client.48286.objecter ms_handle_connect 0x7f4adc023520 2024-02-19T21:07:42.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.951+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4adc01fc70 con 0x7f4adc023520 2024-02-19T21:07:42.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.951+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:07:42.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.951+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 15 = system:0 2024-02-19T21:07:42.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.951+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc01f840 osd=7 4 2024-02-19T21:07:42.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.951+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 7 15 2024-02-19T21:07:42.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.966+0000 7f4aff8b7700 10 client.48286.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:42.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.966+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign -1 16 2024-02-19T21:07:42.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.966+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc01f840 osd=7 3 2024-02-19T21:07:42.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.966+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc01f840 osd=7 3 2024-02-19T21:07:42.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.966+0000 7f4aff8b7700 20 client.48286.objecter _get_session s=0x7f4adc01f840 osd=7 3 2024-02-19T21:07:42.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.966+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_remove -1 16 2024-02-19T21:07:42.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.966+0000 7f4aff8b7700 20 client.48286.objecter get_session s=0x7f4adc01f840 osd=7 3 2024-02-19T21:07:42.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.966+0000 7f4aff8b7700 15 client.48286.objecter _session_command_op_assign 7 16 2024-02-19T21:07:42.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.966+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc01f840 osd=7 4 2024-02-19T21:07:42.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.966+0000 7f4aff8b7700 10 client.48286.objecter _send_command 16 2024-02-19T21:07:42.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.966+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4adc01fc70 con 0x7f4adc023520 2024-02-19T21:07:42.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.967+0000 7f4aee7fc700 1 -- 172.21.15.163:0/3821650192 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4adc01fc70 con 0x7f4adc023520 2024-02-19T21:07:42.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.967+0000 7f4aee7fc700 10 client.48286.objecter ms_dispatch 0x7f4af8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:42.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.967+0000 7f4aee7fc700 10 client.48286.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:42.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.967+0000 7f4aee7fc700 20 client.48286.objecter put_session s=0x7f4adc01f840 osd=7 4 2024-02-19T21:07:42.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.967+0000 7f4aee7fc700 15 client.48286.objecter _session_command_op_remove 7 16 2024-02-19T21:07:42.963 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:07:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 10 librados: watch_flush enter 2024-02-19T21:07:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 10 librados: watch_flush exit 2024-02-19T21:07:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 10 client.48286.objecter close_session for osd.0 2024-02-19T21:07:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 >> v1:172.21.15.163:6801/1559418115 conn(0x7f4adc004530 legacy=0x7f4adc002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc0051a0 osd=0 1 2024-02-19T21:07:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 10 client.48286.objecter close_session for osd.1 2024-02-19T21:07:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 >> v1:172.21.15.163:6805/3125047432 conn(0x7f4adc004150 legacy=0x7f4adc00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc003fe0 osd=1 1 2024-02-19T21:07:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 10 client.48286.objecter close_session for osd.2 2024-02-19T21:07:42.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 >> v1:172.21.15.163:6809/1563158487 conn(0x7f4adc00d2f0 legacy=0x7f4adc00d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc003630 osd=2 1 2024-02-19T21:07:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 10 client.48286.objecter close_session for osd.3 2024-02-19T21:07:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 >> v1:172.21.15.163:6813/2112279769 conn(0x7f4adc011e80 legacy=0x7f4adc014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc003480 osd=3 1 2024-02-19T21:07:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4afde54700 1 -- 172.21.15.163:0/3821650192 reap_dead start 2024-02-19T21:07:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 10 client.48286.objecter close_session for osd.4 2024-02-19T21:07:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 >> v1:172.21.15.193:6800/3129716965 conn(0x7f4adc00ce50 legacy=0x7f4adc015a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4afde54700 1 -- 172.21.15.163:0/3821650192 reap_dead start 2024-02-19T21:07:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc00e410 osd=4 1 2024-02-19T21:07:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 10 client.48286.objecter close_session for osd.5 2024-02-19T21:07:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 >> v1:172.21.15.193:6804/805161991 conn(0x7f4adc01a960 legacy=0x7f4adc01cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc00e580 osd=5 1 2024-02-19T21:07:42.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 10 client.48286.objecter close_session for osd.6 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 >> v1:172.21.15.193:6808/787604615 conn(0x7f4adc020320 legacy=0x7f4adc0207a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc01e660 osd=6 1 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 10 client.48286.objecter close_session for osd.7 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 >> v1:172.21.15.193:6812/157927432 conn(0x7f4adc023520 legacy=0x7f4adc025910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 20 client.48286.objecter put_session s=0x7f4adc01f840 osd=7 1 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 20 client.48286.objecter shutdown clearing up homeless session... 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 10 client.48286.objecter successfully canceled tick 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4ae4063c60 legacy=0x7f4ae4066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 >> v1:172.21.15.193:6789/0 conn(0x7f4af81031b0 legacy=0x7f4af8117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4afde54700 1 -- 172.21.15.163:0/3821650192 reap_dead start 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4afde54700 1 -- 172.21.15.163:0/3821650192 reap_dead start 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 shutdown_connections 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 >> 172.21.15.163:0/3821650192 conn(0x7f4af80f2470 msgr2=0x7f4af8106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 shutdown_connections 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 1 -- 172.21.15.163:0/3821650192 wait complete. 2024-02-19T21:07:42.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:42.968+0000 7f4aff8b7700 1 librados: shutdown 2024-02-19T21:07:42.968 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:07:42.968 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:07:42.968 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:07:42.968 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:07:42.968 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:07:42.968 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:07:42.968 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:07:42.968 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:07:43.030 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:07:43.201 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-19T21:07:43.263 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:43.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:43.366 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:43.393 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:43.411 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:43.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:43.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:43.510 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:07:43.516 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:07:43.543 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:43.718 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:07:43] "GET /metrics HTTP/1.1" 200 188035 "" "Prometheus/2.33.4" 2024-02-19T21:07:43.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:42.831211+0000 mon.b (mon.1) 441 : audit [DBG] from='client.? v1:172.21.15.163:0/3821650192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:43.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:43.213973+0000 mon.a (mon.0) 1957 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:43.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:43.214064+0000 mon.a (mon.0) 1958 : audit [INF] from='client.48268 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi163-83845-40"}]': finished 2024-02-19T21:07:43.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:43.214154+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e283: 8 total, 8 up, 7 in 2024-02-19T21:07:43.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:43.227514+0000 mon.b (mon.1) 442 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:43.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:43.231903+0000 mon.a (mon.0) 1960 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:43.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:43.251874+0000 mon.a (mon.0) 1961 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi163-83854-51"}]: dispatch 2024-02-19T21:07:43.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:43.253760+0000 mon.a (mon.0) 1962 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi163-83854-51"}]: dispatch 2024-02-19T21:07:43.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:43.254064+0000 mon.a (mon.0) 1963 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi163-83854-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:43.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:43.260938+0000 mon.a (mon.0) 1964 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi163-83845-41"}]: dispatch 2024-02-19T21:07:43.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:43.261987+0000 mon.a (mon.0) 1965 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi163-83845-41"}]: dispatch 2024-02-19T21:07:43.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:43.262241+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi163-83845-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:43.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:42.831211+0000 mon.b (mon.1) 441 : audit [DBG] from='client.? v1:172.21.15.163:0/3821650192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:43.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:43.213973+0000 mon.a (mon.0) 1957 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:43.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:43.214064+0000 mon.a (mon.0) 1958 : audit [INF] from='client.48268 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi163-83845-40"}]': finished 2024-02-19T21:07:43.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:43.214154+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e283: 8 total, 8 up, 7 in 2024-02-19T21:07:43.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:43.227514+0000 mon.b (mon.1) 442 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:43.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:43.231903+0000 mon.a (mon.0) 1960 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:43.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:43.251874+0000 mon.a (mon.0) 1961 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi163-83854-51"}]: dispatch 2024-02-19T21:07:43.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:43.253760+0000 mon.a (mon.0) 1962 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi163-83854-51"}]: dispatch 2024-02-19T21:07:43.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:43.254064+0000 mon.a (mon.0) 1963 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi163-83854-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:43.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:43.260938+0000 mon.a (mon.0) 1964 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi163-83845-41"}]: dispatch 2024-02-19T21:07:43.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:43.261987+0000 mon.a (mon.0) 1965 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi163-83845-41"}]: dispatch 2024-02-19T21:07:43.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:43.262241+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi163-83845-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:43.725 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:07:43.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:43.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:07:43.747 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:42.831211+0000 mon.b (mon.1) 441 : audit [DBG] from='client.? v1:172.21.15.163:0/3821650192' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:43.748 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:43.213973+0000 mon.a (mon.0) 1957 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:43.749 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:43.214064+0000 mon.a (mon.0) 1958 : audit [INF] from='client.48268 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi163-83845-40"}]': finished 2024-02-19T21:07:43.749 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:43.214154+0000 mon.a (mon.0) 1959 : cluster [DBG] osdmap e283: 8 total, 8 up, 7 in 2024-02-19T21:07:43.749 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:43.227514+0000 mon.b (mon.1) 442 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:43.749 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:43.231903+0000 mon.a (mon.0) 1960 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:43.749 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:43.251874+0000 mon.a (mon.0) 1961 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi163-83854-51"}]: dispatch 2024-02-19T21:07:43.749 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:43.253760+0000 mon.a (mon.0) 1962 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi163-83854-51"}]: dispatch 2024-02-19T21:07:43.749 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:43.254064+0000 mon.a (mon.0) 1963 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi163-83854-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:43.749 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:43.260938+0000 mon.a (mon.0) 1964 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi163-83845-41"}]: dispatch 2024-02-19T21:07:43.750 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:43.261987+0000 mon.a (mon.0) 1965 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi163-83845-41"}]: dispatch 2024-02-19T21:07:43.750 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:43.262241+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi163-83845-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:43.820 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:07:43.820 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:07:43.821 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-19T21:07:43.821 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-19T21:07:43.829 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:43.849 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:43.878 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:44.079 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:44.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:44.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:44.182 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:44.212 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:44.604 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:44.699 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:43.169084+0000 mgr.y (mgr.14550) 286 : cluster [DBG] pgmap v395: 266 pgs: 1 active+recovering, 1 active+clean+snaptrim, 2 peering, 64 unknown, 198 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s; 1/942 objects misplaced (0.106%); 571 B/s, 63 objects/s recovering 2024-02-19T21:07:44.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:44.208680+0000 mon.a (mon.0) 1967 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:44.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:44.208765+0000 mon.a (mon.0) 1968 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi163-83854-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:44.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:44.208825+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi163-83845-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:44.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:44.208928+0000 mon.a (mon.0) 1970 : cluster [DBG] osdmap e284: 8 total, 8 up, 7 in 2024-02-19T21:07:44.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:44.209727+0000 mon.a (mon.0) 1971 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi163-83845-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi163-83845-41"}]: dispatch 2024-02-19T21:07:44.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:44.211055+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi163-83854-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi163-83854-51"}]: dispatch 2024-02-19T21:07:44.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:44.213461+0000 mon.b (mon.1) 443 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-35"}]: dispatch 2024-02-19T21:07:44.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:44.213784+0000 mon.a (mon.0) 1973 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-35"}]: dispatch 2024-02-19T21:07:44.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:43.169084+0000 mgr.y (mgr.14550) 286 : cluster [DBG] pgmap v395: 266 pgs: 1 active+recovering, 1 active+clean+snaptrim, 2 peering, 64 unknown, 198 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s; 1/942 objects misplaced (0.106%); 571 B/s, 63 objects/s recovering 2024-02-19T21:07:44.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:44.208680+0000 mon.a (mon.0) 1967 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:44.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:44.208765+0000 mon.a (mon.0) 1968 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi163-83854-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:44.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:44.208825+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi163-83845-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:44.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:44.208928+0000 mon.a (mon.0) 1970 : cluster [DBG] osdmap e284: 8 total, 8 up, 7 in 2024-02-19T21:07:44.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:44.209727+0000 mon.a (mon.0) 1971 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi163-83845-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi163-83845-41"}]: dispatch 2024-02-19T21:07:44.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:44.211055+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi163-83854-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi163-83854-51"}]: dispatch 2024-02-19T21:07:44.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:44.213461+0000 mon.b (mon.1) 443 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-35"}]: dispatch 2024-02-19T21:07:44.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:44.213784+0000 mon.a (mon.0) 1973 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-35"}]: dispatch 2024-02-19T21:07:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:43.169084+0000 mgr.y (mgr.14550) 286 : cluster [DBG] pgmap v395: 266 pgs: 1 active+recovering, 1 active+clean+snaptrim, 2 peering, 64 unknown, 198 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 255 B/s wr, 0 op/s; 1/942 objects misplaced (0.106%); 571 B/s, 63 objects/s recovering 2024-02-19T21:07:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:44.208680+0000 mon.a (mon.0) 1967 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:44.208765+0000 mon.a (mon.0) 1968 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi163-83854-51", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:44.208825+0000 mon.a (mon.0) 1969 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi163-83845-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:44.208928+0000 mon.a (mon.0) 1970 : cluster [DBG] osdmap e284: 8 total, 8 up, 7 in 2024-02-19T21:07:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:44.209727+0000 mon.a (mon.0) 1971 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi163-83845-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi163-83845-41"}]: dispatch 2024-02-19T21:07:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:44.211055+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi163-83854-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi163-83854-51"}]: dispatch 2024-02-19T21:07:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:44.213461+0000 mon.b (mon.1) 443 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-35"}]: dispatch 2024-02-19T21:07:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:44.213784+0000 mon.a (mon.0) 1973 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-35"}]: dispatch 2024-02-19T21:07:44.862 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:44.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:44.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:44.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:45.004 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:45.202 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:45.204 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:07:45.204 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:07:45.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:45.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:45.306 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:45.335 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:45.482 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:45.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:45.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:45.586 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:45.614 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:45.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.709+0000 7f296ec99700 1 -- 172.21.15.163:0/3197895660 >> v1:172.21.15.163:6790/0 conn(0x7f29680fd980 legacy=0x7f29680fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:45.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.709+0000 7f296ec99700 1 -- 172.21.15.163:0/3197895660 shutdown_connections 2024-02-19T21:07:45.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.709+0000 7f296ec99700 1 -- 172.21.15.163:0/3197895660 >> 172.21.15.163:0/3197895660 conn(0x7f29680f35a0 msgr2=0x7f29680f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:45.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.709+0000 7f296ec99700 1 -- 172.21.15.163:0/3197895660 shutdown_connections 2024-02-19T21:07:45.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.709+0000 7f296ec99700 1 -- 172.21.15.163:0/3197895660 wait complete. 2024-02-19T21:07:45.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.709+0000 7f296ec99700 1 librados: starting msgr at 2024-02-19T21:07:45.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.709+0000 7f296ec99700 1 librados: starting objecter 2024-02-19T21:07:45.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.709+0000 7f296ec99700 1 Processor -- start 2024-02-19T21:07:45.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f296ec99700 1 -- start start 2024-02-19T21:07:45.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f296ec99700 1 librados: setting wanted keys 2024-02-19T21:07:45.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f296ec99700 1 librados: calling monclient init 2024-02-19T21:07:45.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f296ec99700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2968119450 con 0x7f2968100780 2024-02-19T21:07:45.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f296ec99700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f296811a5f0 con 0x7f29681042c0 2024-02-19T21:07:45.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f296ec99700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f296811b790 con 0x7f29680fd980 2024-02-19T21:07:45.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2967fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f2968100780 0x7f29680fcc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33056/0 (socket says 172.21.15.163:33056) 2024-02-19T21:07:45.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2967fff700 1 -- 172.21.15.163:0/3216881432 learned_addr learned my addr 172.21.15.163:0/3216881432 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:45.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f296ca35700 1 --1- 172.21.15.163:0/3216881432 >> v1:172.21.15.163:6790/0 conn(0x7f29680fd980 0x7f29680fc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:41536/0 (socket says 172.21.15.163:41536) 2024-02-19T21:07:45.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2965ffb700 10 client.?.objecter ms_handle_connect 0x7f29680fd980 2024-02-19T21:07:45.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2965ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:45.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2965ffb700 10 client.?.objecter ms_handle_connect 0x7f2968100780 2024-02-19T21:07:45.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2965ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:45.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652944149 0 0) 0x7f296811b790 con 0x7f29680fd980 2024-02-19T21:07:45.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2958003800 con 0x7f29680fd980 2024-02-19T21:07:45.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180789541 0 0) 0x7f2968119450 con 0x7f2968100780 2024-02-19T21:07:45.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f296811b790 con 0x7f2968100780 2024-02-19T21:07:45.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2965ffb700 10 client.?.objecter ms_handle_connect 0x7f29681042c0 2024-02-19T21:07:45.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2965ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:45.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1464946728 0 0) 0x7f2958003800 con 0x7f29680fd980 2024-02-19T21:07:45.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2968119450 con 0x7f29680fd980 2024-02-19T21:07:45.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f295c002f60 con 0x7f29680fd980 2024-02-19T21:07:45.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587882241 0 0) 0x7f296811a5f0 con 0x7f29681042c0 2024-02-19T21:07:45.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.710+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2958003800 con 0x7f29681042c0 2024-02-19T21:07:45.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.711+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3832538667 0 0) 0x7f296811b790 con 0x7f2968100780 2024-02-19T21:07:45.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.711+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f296811a5f0 con 0x7f2968100780 2024-02-19T21:07:45.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.711+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2954001910 con 0x7f2968100780 2024-02-19T21:07:45.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.711+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1107536622 0 0) 0x7f2968119450 con 0x7f29680fd980 2024-02-19T21:07:45.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.711+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 >> v1:172.21.15.193:6789/0 conn(0x7f29681042c0 legacy=0x7f2968117d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:45.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.711+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 >> v1:172.21.15.163:6789/0 conn(0x7f2968100780 legacy=0x7f29680fcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:45.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.711+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296811c930 con 0x7f29680fd980 2024-02-19T21:07:45.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.711+0000 7f296ec99700 1 -- 172.21.15.163:0/3216881432 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2968119640 con 0x7f29680fd980 2024-02-19T21:07:45.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.711+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f295c000fe0 con 0x7f29680fd980 2024-02-19T21:07:45.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.711+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f295c0037d0 con 0x7f29680fd980 2024-02-19T21:07:45.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.711+0000 7f296ec99700 10 client.49484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:45.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.711+0000 7f296ec99700 1 -- 172.21.15.163:0/3216881432 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2968119b30 con 0x7f29680fd980 2024-02-19T21:07:45.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.711+0000 7f296ec99700 1 librados: init done 2024-02-19T21:07:45.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.712+0000 7f296ec99700 1 -- 172.21.15.163:0/3216881432 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296805f660 con 0x7f29680fd980 2024-02-19T21:07:45.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.713+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f295c0039d0 con 0x7f29680fd980 2024-02-19T21:07:45.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.713+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(285..285 src has 43..285) v4 ==== 8804+0+0 (unknown 357227966 0 0) 0x7f295c079120 con 0x7f29680fd980 2024-02-19T21:07:45.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.713+0000 7f2965ffb700 10 client.49484.objecter ms_dispatch 0x7f2968001550 osd_map(285..285 src has 43..285) v4 2024-02-19T21:07:45.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.713+0000 7f2965ffb700 3 client.49484.objecter handle_osd_map got epochs [285,285] > 0 2024-02-19T21:07:45.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.713+0000 7f2965ffb700 3 client.49484.objecter handle_osd_map decoding full epoch 285 2024-02-19T21:07:45.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.713+0000 7f2965ffb700 20 client.49484.objecter dump_active .. 0 homeless 2024-02-19T21:07:45.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.713+0000 7f2965ffb700 10 client.49484.objecter ms_handle_connect 0x7f2958063a10 2024-02-19T21:07:45.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.717+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f295c04c2b0 con 0x7f29680fd980 2024-02-19T21:07:45.740 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:07:45.767 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:45.171029+0000 mon.b (mon.1) 444 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.4"}]: dispatch 2024-02-19T21:07:45.767 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:45.171327+0000 mon.b (mon.1) 445 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.4"}]: dispatch 2024-02-19T21:07:45.767 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:45.171969+0000 mon.a (mon.0) 1974 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.4"}]: dispatch 2024-02-19T21:07:45.767 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:45.172145+0000 mon.a (mon.0) 1975 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.4"}]: dispatch 2024-02-19T21:07:45.768 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:45.171029+0000 mon.b (mon.1) 444 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.4"}]: dispatch 2024-02-19T21:07:45.768 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:45.171327+0000 mon.b (mon.1) 445 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.4"}]: dispatch 2024-02-19T21:07:45.768 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:45.171969+0000 mon.a (mon.0) 1974 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.4"}]: dispatch 2024-02-19T21:07:45.768 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:45.172145+0000 mon.a (mon.0) 1975 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.4"}]: dispatch 2024-02-19T21:07:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:45.171029+0000 mon.b (mon.1) 444 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.4"}]: dispatch 2024-02-19T21:07:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:45.171327+0000 mon.b (mon.1) 445 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.4"}]: dispatch 2024-02-19T21:07:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:45.171969+0000 mon.a (mon.0) 1974 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.4"}]: dispatch 2024-02-19T21:07:45.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:45.172145+0000 mon.a (mon.0) 1975 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.4"}]: dispatch 2024-02-19T21:07:45.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:45.913+0000 7f296ec99700 1 -- 172.21.15.163:0/3216881432 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f2968047540 con 0x7f29680fd980 2024-02-19T21:07:45.948 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-19T21:07:45.963 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:45.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:45.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:46.066 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:46.095 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:46.115 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:46.117 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:46.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:46.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:46.214+0000 7f2965ffb700 1 -- 172.21.15.163:0/3216881432 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v286) v1 ==== 97+0+0 (unknown 2489258276 0 0) 0x7f295c04a720 con 0x7f29680fd980 2024-02-19T21:07:46.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:46.220+0000 7f296ec99700 10 librados: watch_flush enter 2024-02-19T21:07:46.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:46.220+0000 7f296ec99700 10 librados: watch_flush exit 2024-02-19T21:07:46.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:46.220+0000 7f296ec99700 20 client.49484.objecter shutdown clearing up homeless session... 2024-02-19T21:07:46.215 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:46.220+0000 7f296ec99700 10 client.49484.objecter successfully canceled tick 2024-02-19T21:07:46.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:46.220+0000 7f296ec99700 1 -- 172.21.15.163:0/3216881432 >> v1:172.21.15.163:6800/1552514681 conn(0x7f2958063a10 legacy=0x7f2958065ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:46.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:46.220+0000 7f296ec99700 1 -- 172.21.15.163:0/3216881432 >> v1:172.21.15.163:6790/0 conn(0x7f29680fd980 legacy=0x7f29680fc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:46.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:46.220+0000 7f296ec99700 1 -- 172.21.15.163:0/3216881432 shutdown_connections 2024-02-19T21:07:46.216 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:46.221+0000 7f296ec99700 1 -- 172.21.15.163:0/3216881432 >> 172.21.15.163:0/3216881432 conn(0x7f29680f35a0 msgr2=0x7f2968107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:46.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:46.221+0000 7f296ec99700 1 -- 172.21.15.163:0/3216881432 shutdown_connections 2024-02-19T21:07:46.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:46.221+0000 7f296ec99700 1 -- 172.21.15.163:0/3216881432 wait complete. 2024-02-19T21:07:46.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:46.221+0000 7f296ec99700 1 librados: shutdown 2024-02-19T21:07:46.217 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:07:46.219 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:46.233 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:07:46.252 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:46.451 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:46.453 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:46.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:46.555 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:46.581 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:46.679 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:46.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:46.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:46.783 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:46.810 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:46.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:45.170391+0000 mgr.y (mgr.14550) 287 : cluster [DBG] pgmap v398: 233 pgs: 233 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:07:46.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:45.685515+0000 mon.a (mon.0) 1976 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-35"}]': finished 2024-02-19T21:07:46.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:45.685609+0000 mon.a (mon.0) 1977 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.4"}]': finished 2024-02-19T21:07:46.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:45.685683+0000 mon.a (mon.0) 1978 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.4"}]': finished 2024-02-19T21:07:46.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:45.685742+0000 mon.a (mon.0) 1979 : cluster [DBG] osdmap e285: 8 total, 8 up, 7 in 2024-02-19T21:07:46.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:45.686240+0000 mon.b (mon.1) 446 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-35", "mode": "writeback"}]: dispatch 2024-02-19T21:07:46.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:45.686746+0000 mon.a (mon.0) 1980 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-35", "mode": "writeback"}]: dispatch 2024-02-19T21:07:46.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:45.913892+0000 mon.c (mon.2) 333 : audit [INF] from='client.? v1:172.21.15.163:0/3216881432' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:07:46.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:45.914240+0000 mon.a (mon.0) 1981 : audit [INF] from='client.49484 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:07:46.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:46.208540+0000 mon.a (mon.0) 1982 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:46.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:46.208566+0000 mon.a (mon.0) 1983 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:46.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:46.212458+0000 mon.a (mon.0) 1984 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi163-83845-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi163-83845-41"}]': finished 2024-02-19T21:07:46.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:46.212545+0000 mon.a (mon.0) 1985 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi163-83854-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi163-83854-51"}]': finished 2024-02-19T21:07:46.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:46.212635+0000 mon.a (mon.0) 1986 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-35", "mode": "writeback"}]': finished 2024-02-19T21:07:46.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:46.212711+0000 mon.a (mon.0) 1987 : audit [INF] from='client.49484 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:07:46.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:46.212763+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e286: 8 total, 8 up, 7 in 2024-02-19T21:07:46.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:46.253339+0000 mon.b (mon.1) 447 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:46.902 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:46.253744+0000 mon.a (mon.0) 1989 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:46.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:45.170391+0000 mgr.y (mgr.14550) 287 : cluster [DBG] pgmap v398: 233 pgs: 233 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:07:46.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:45.685515+0000 mon.a (mon.0) 1976 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-35"}]': finished 2024-02-19T21:07:46.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:45.685609+0000 mon.a (mon.0) 1977 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.4"}]': finished 2024-02-19T21:07:46.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:45.685683+0000 mon.a (mon.0) 1978 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.4"}]': finished 2024-02-19T21:07:46.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:45.685742+0000 mon.a (mon.0) 1979 : cluster [DBG] osdmap e285: 8 total, 8 up, 7 in 2024-02-19T21:07:46.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:45.686240+0000 mon.b (mon.1) 446 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-35", "mode": "writeback"}]: dispatch 2024-02-19T21:07:46.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:45.686746+0000 mon.a (mon.0) 1980 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-35", "mode": "writeback"}]: dispatch 2024-02-19T21:07:46.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:45.913892+0000 mon.c (mon.2) 333 : audit [INF] from='client.? v1:172.21.15.163:0/3216881432' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:07:46.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:45.914240+0000 mon.a (mon.0) 1981 : audit [INF] from='client.49484 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:07:46.903 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:46.208540+0000 mon.a (mon.0) 1982 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:46.904 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:46.208566+0000 mon.a (mon.0) 1983 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:46.904 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:46.212458+0000 mon.a (mon.0) 1984 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi163-83845-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi163-83845-41"}]': finished 2024-02-19T21:07:46.904 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:46.212545+0000 mon.a (mon.0) 1985 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi163-83854-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi163-83854-51"}]': finished 2024-02-19T21:07:46.904 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:46.212635+0000 mon.a (mon.0) 1986 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-35", "mode": "writeback"}]': finished 2024-02-19T21:07:46.904 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:46.212711+0000 mon.a (mon.0) 1987 : audit [INF] from='client.49484 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:07:46.904 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:46.212763+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e286: 8 total, 8 up, 7 in 2024-02-19T21:07:46.904 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:46.253339+0000 mon.b (mon.1) 447 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:46.904 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:46.253744+0000 mon.a (mon.0) 1989 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:47.007 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:47.009 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:45.170391+0000 mgr.y (mgr.14550) 287 : cluster [DBG] pgmap v398: 233 pgs: 233 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:07:47.009 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:45.685515+0000 mon.a (mon.0) 1976 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-35"}]': finished 2024-02-19T21:07:47.009 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:45.685609+0000 mon.a (mon.0) 1977 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.4"}]': finished 2024-02-19T21:07:47.009 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:45.685683+0000 mon.a (mon.0) 1978 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "2.4"}]': finished 2024-02-19T21:07:47.009 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:45.685742+0000 mon.a (mon.0) 1979 : cluster [DBG] osdmap e285: 8 total, 8 up, 7 in 2024-02-19T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:45.686240+0000 mon.b (mon.1) 446 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-35", "mode": "writeback"}]: dispatch 2024-02-19T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:45.686746+0000 mon.a (mon.0) 1980 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-35", "mode": "writeback"}]: dispatch 2024-02-19T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:45.913892+0000 mon.c (mon.2) 333 : audit [INF] from='client.? v1:172.21.15.163:0/3216881432' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:45.914240+0000 mon.a (mon.0) 1981 : audit [INF] from='client.49484 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:46.208540+0000 mon.a (mon.0) 1982 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:46.208566+0000 mon.a (mon.0) 1983 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:46.212458+0000 mon.a (mon.0) 1984 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi163-83845-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi163-83845-41"}]': finished 2024-02-19T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:46.212545+0000 mon.a (mon.0) 1985 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi163-83854-51", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi163-83854-51"}]': finished 2024-02-19T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:46.212635+0000 mon.a (mon.0) 1986 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-35", "mode": "writeback"}]': finished 2024-02-19T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:46.212711+0000 mon.a (mon.0) 1987 : audit [INF] from='client.49484 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:07:47.010 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:46.212763+0000 mon.a (mon.0) 1988 : cluster [DBG] osdmap e286: 8 total, 8 up, 7 in 2024-02-19T21:07:47.011 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:46.253339+0000 mon.b (mon.1) 447 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:47.011 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:46.253744+0000 mon.a (mon.0) 1989 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:47.011 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:47.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:47.112 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:47.139 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:47.272 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:47.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.281+0000 7f8015590700 1 -- 172.21.15.163:0/3470614891 >> v1:172.21.15.163:6789/0 conn(0x7f80100fc880 legacy=0x7f80100fcc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:47.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.281+0000 7f8015590700 1 -- 172.21.15.163:0/3470614891 shutdown_connections 2024-02-19T21:07:47.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.281+0000 7f8015590700 1 -- 172.21.15.163:0/3470614891 >> 172.21.15.163:0/3470614891 conn(0x7f80100f24a0 msgr2=0x7f80100f48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:47.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.281+0000 7f8015590700 1 -- 172.21.15.163:0/3470614891 shutdown_connections 2024-02-19T21:07:47.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.281+0000 7f8015590700 1 -- 172.21.15.163:0/3470614891 wait complete. 2024-02-19T21:07:47.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.282+0000 7f8015590700 1 librados: starting msgr at 2024-02-19T21:07:47.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.282+0000 7f8015590700 1 librados: starting objecter 2024-02-19T21:07:47.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.282+0000 7f8015590700 1 Processor -- start 2024-02-19T21:07:47.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.282+0000 7f8015590700 1 -- start start 2024-02-19T21:07:47.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.282+0000 7f8015590700 1 librados: setting wanted keys 2024-02-19T21:07:47.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.282+0000 7f8015590700 1 librados: calling monclient init 2024-02-19T21:07:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f8015590700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f80101151d0 con 0x7f80101031c0 2024-02-19T21:07:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f8015590700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f80101163b0 con 0x7f80100fc880 2024-02-19T21:07:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f8015590700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f80101175b0 con 0x7f80100ff680 2024-02-19T21:07:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f800f7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f80101031c0 0x7f8010113890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33060/0 (socket says 172.21.15.163:33060) 2024-02-19T21:07:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f800f7fe700 1 -- 172.21.15.163:0/2590724172 learned_addr learned my addr 172.21.15.163:0/2590724172 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 10 client.?.objecter ms_handle_connect 0x7f80100fc880 2024-02-19T21:07:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 10 client.?.objecter ms_handle_connect 0x7f80100ff680 2024-02-19T21:07:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 10 client.?.objecter ms_handle_connect 0x7f80101031c0 2024-02-19T21:07:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:47.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313612076 0 0) 0x7f80101163b0 con 0x7f80100fc880 2024-02-19T21:07:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff80033f0 con 0x7f80100fc880 2024-02-19T21:07:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474384745 0 0) 0x7f80101175b0 con 0x7f80100ff680 2024-02-19T21:07:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80101163b0 con 0x7f80100ff680 2024-02-19T21:07:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429681292 0 0) 0x7f80101151d0 con 0x7f80101031c0 2024-02-19T21:07:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80101175b0 con 0x7f80101031c0 2024-02-19T21:07:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1733675943 0 0) 0x7f7ff80033f0 con 0x7f80100fc880 2024-02-19T21:07:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80101151d0 con 0x7f80100fc880 2024-02-19T21:07:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8000002130 con 0x7f80100fc880 2024-02-19T21:07:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2752606957 0 0) 0x7f80101175b0 con 0x7f80101031c0 2024-02-19T21:07:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ff80033f0 con 0x7f80101031c0 2024-02-19T21:07:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8008001c90 con 0x7f80101031c0 2024-02-19T21:07:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2059070259 0 0) 0x7f80101163b0 con 0x7f80100ff680 2024-02-19T21:07:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80101175b0 con 0x7f80100ff680 2024-02-19T21:07:47.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2129455782 0 0) 0x7f80101151d0 con 0x7f80100fc880 2024-02-19T21:07:47.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 >> v1:172.21.15.163:6790/0 conn(0x7f80100ff680 legacy=0x7f80100f8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:47.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 >> v1:172.21.15.163:6789/0 conn(0x7f80101031c0 legacy=0x7f8010113890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:47.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.283+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80101187b0 con 0x7f80100fc880 2024-02-19T21:07:47.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.284+0000 7f8015590700 1 -- 172.21.15.163:0/2590724172 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80101177a0 con 0x7f80100fc880 2024-02-19T21:07:47.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.284+0000 7f8015590700 10 client.48307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:47.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.284+0000 7f8015590700 1 -- 172.21.15.163:0/2590724172 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8010117c10 con 0x7f80100fc880 2024-02-19T21:07:47.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.284+0000 7f8015590700 1 librados: init done 2024-02-19T21:07:47.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.284+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f80000022d0 con 0x7f80100fc880 2024-02-19T21:07:47.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.284+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f80000037a0 con 0x7f80100fc880 2024-02-19T21:07:47.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.285+0000 7f8015590700 1 -- 172.21.15.163:0/2590724172 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ffc004f20 con 0x7f80100fc880 2024-02-19T21:07:47.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.285+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f80000166f0 con 0x7f80100fc880 2024-02-19T21:07:47.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.286+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(287..287 src has 43..287) v4 ==== 9644+0+0 (unknown 1266836514 0 0) 0x7f8000076f40 con 0x7f80100fc880 2024-02-19T21:07:47.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.286+0000 7f7ff7fff700 10 client.48307.objecter ms_dispatch 0x7f8010001550 osd_map(287..287 src has 43..287) v4 2024-02-19T21:07:47.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.286+0000 7f7ff7fff700 3 client.48307.objecter handle_osd_map got epochs [287,287] > 0 2024-02-19T21:07:47.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.286+0000 7f7ff7fff700 3 client.48307.objecter handle_osd_map decoding full epoch 287 2024-02-19T21:07:47.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.286+0000 7f7ff7fff700 20 client.48307.objecter dump_active .. 0 homeless 2024-02-19T21:07:47.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.286+0000 7f7ff7fff700 10 client.48307.objecter ms_handle_connect 0x7f7ff8065e60 2024-02-19T21:07:47.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.290+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8000049eb0 con 0x7f80100fc880 2024-02-19T21:07:47.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.460+0000 7f8015590700 1 -- 172.21.15.163:0/2590724172 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f7ffc0051d0 con 0x7f80100fc880 2024-02-19T21:07:47.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.462+0000 7f7ff7fff700 1 -- 172.21.15.163:0/2590724172 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v287) v1 ==== 81+0+53995 (unknown 277080156 0 2332500584) 0x7f8000048320 con 0x7f80100fc880 2024-02-19T21:07:47.459 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:07:47.459 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:07:47.459 INFO:teuthology.orchestra.run.smithi163.stdout: "epoch": 287, 2024-02-19T21:07:47.459 INFO:teuthology.orchestra.run.smithi163.stdout: "fsid": "2591809a-cf69-11ee-95bb-87774f69a715", 2024-02-19T21:07:47.459 INFO:teuthology.orchestra.run.smithi163.stdout: "created": "2024-02-19T20:55:37.794885+0000", 2024-02-19T21:07:47.459 INFO:teuthology.orchestra.run.smithi163.stdout: "modified": "2024-02-19T21:07:47.209742+0000", 2024-02-19T21:07:47.460 INFO:teuthology.orchestra.run.smithi163.stdout: "last_up_change": "2024-02-19T21:01:09.214134+0000", 2024-02-19T21:07:47.460 INFO:teuthology.orchestra.run.smithi163.stdout: "last_in_change": "2024-02-19T21:07:36.204144+0000", 2024-02-19T21:07:47.460 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-19T21:07:47.460 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_num": 5806080, 2024-02-19T21:07:47.460 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_set": [ 2024-02-19T21:07:47.460 INFO:teuthology.orchestra.run.smithi163.stdout: "nodeep-scrub", 2024-02-19T21:07:47.460 INFO:teuthology.orchestra.run.smithi163.stdout: "noscrub", 2024-02-19T21:07:47.460 INFO:teuthology.orchestra.run.smithi163.stdout: "pglog_hardlimit", 2024-02-19T21:07:47.460 INFO:teuthology.orchestra.run.smithi163.stdout: "purged_snapdirs", 2024-02-19T21:07:47.460 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_deletes", 2024-02-19T21:07:47.460 INFO:teuthology.orchestra.run.smithi163.stdout: "sortbitwise" 2024-02-19T21:07:47.460 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:07:47.460 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_version": 76, 2024-02-19T21:07:47.460 INFO:teuthology.orchestra.run.smithi163.stdout: "full_ratio": 0.94999998807907104, 2024-02-19T21:07:47.461 INFO:teuthology.orchestra.run.smithi163.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-19T21:07:47.461 INFO:teuthology.orchestra.run.smithi163.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-19T21:07:47.461 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_snapshot": "", 2024-02-19T21:07:47.461 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_max": 185, 2024-02-19T21:07:47.461 INFO:teuthology.orchestra.run.smithi163.stdout: "max_osd": 8, 2024-02-19T21:07:47.461 INFO:teuthology.orchestra.run.smithi163.stdout: "require_min_compat_client": "luminous", 2024-02-19T21:07:47.461 INFO:teuthology.orchestra.run.smithi163.stdout: "min_compat_client": "luminous", 2024-02-19T21:07:47.461 INFO:teuthology.orchestra.run.smithi163.stdout: "require_osd_release": "pacific", 2024-02-19T21:07:47.461 INFO:teuthology.orchestra.run.smithi163.stdout: "pools": [ 2024-02-19T21:07:47.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.461 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 1, 2024-02-19T21:07:47.461 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "device_health_metrics", 2024-02-19T21:07:47.461 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T20:59:09.425169+0000", 2024-02-19T21:07:47.462 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:07:47.462 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:07:47.462 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:07:47.462 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:07:47.462 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:07:47.462 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:07:47.462 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:07:47.462 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:07:47.462 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:07:47.462 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:07:47.462 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:07:47.462 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:07:47.462 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 1, 2024-02-19T21:07:47.463 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 1, 2024-02-19T21:07:47.463 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 1, 2024-02-19T21:07:47.463 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 1, 2024-02-19T21:07:47.463 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 1, 2024-02-19T21:07:47.463 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:07:47.463 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:07:47.463 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:07:47.463 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:07:47.463 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:07:47.463 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:07:47.463 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:07:47.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.463 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "74", 2024-02-19T21:07:47.464 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:07:47.464 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:07:47.464 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:07:47.464 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:07:47.464 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:07:47.464 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:07:47.464 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:07:47.464 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:07:47.464 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:07:47.465 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:07:47.465 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:07:47.465 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:07:47.465 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:07:47.465 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:07:47.465 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:07:47.465 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:07:47.466 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:07:47.466 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:07:47.466 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:07:47.466 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:07:47.466 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:07:47.466 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:07:47.466 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:07:47.466 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:07:47.466 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:07:47.466 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:07:47.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.467 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:07:47.467 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:07:47.467 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:07:47.467 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:07:47.467 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:07:47.467 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:07:47.467 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:07:47.467 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:07:47.467 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:07:47.467 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:07:47.467 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:07:47.467 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:07:47.467 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_max": 32, 2024-02-19T21:07:47.468 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_min": 1 2024-02-19T21:07:47.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.468 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:07:47.468 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_devicehealth": {} 2024-02-19T21:07:47.468 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.468 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 2, 2024-02-19T21:07:47.468 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".rgw.root", 2024-02-19T21:07:47.468 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:14.658945+0000", 2024-02-19T21:07:47.468 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:07:47.468 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 37, 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 37, 2024-02-19T21:07:47.469 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:07:47.470 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "2.25", 2024-02-19T21:07:47.470 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 283, 2024-02-19T21:07:47.470 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 282, 2024-02-19T21:07:47.470 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 282, 2024-02-19T21:07:47.470 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:07:47.470 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:07:47.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.470 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "283", 2024-02-19T21:07:47.470 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:07:47.470 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "283", 2024-02-19T21:07:47.470 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "281", 2024-02-19T21:07:47.470 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:07:47.471 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:07:47.471 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:07:47.471 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:07:47.471 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:07:47.471 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:07:47.471 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:07:47.471 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:07:47.471 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:07:47.471 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:07:47.471 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:07:47.472 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:07:47.472 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:07:47.472 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:07:47.472 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:07:47.472 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:07:47.472 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:07:47.472 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:07:47.472 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:07:47.472 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:07:47.472 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:07:47.473 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:07:47.473 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:07:47.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.473 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:07:47.473 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:07:47.473 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:07:47.473 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:07:47.473 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:07:47.474 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:07:47.474 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:07:47.474 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:07:47.474 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:07:47.474 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:07:47.474 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:07:47.474 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:07:47.474 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:07:47.475 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:07:47.475 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.475 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 3, 2024-02-19T21:07:47.475 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "iscsi", 2024-02-19T21:07:47.475 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:14.958800+0000", 2024-02-19T21:07:47.476 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:07:47.476 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:07:47.476 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:07:47.476 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:07:47.476 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:07:47.476 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:07:47.476 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:07:47.476 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:07:47.477 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:07:47.477 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:07:47.477 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:07:47.477 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:07:47.477 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 3, 2024-02-19T21:07:47.477 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 3, 2024-02-19T21:07:47.477 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 3, 2024-02-19T21:07:47.477 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 3, 2024-02-19T21:07:47.477 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 3, 2024-02-19T21:07:47.478 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:07:47.478 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:07:47.478 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:07:47.478 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:07:47.478 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:07:47.478 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:07:47.478 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:07:47.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.478 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "100", 2024-02-19T21:07:47.479 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:07:47.479 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:07:47.479 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:07:47.479 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:07:47.479 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:07:47.479 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:07:47.480 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:07:47.480 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:07:47.480 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:07:47.480 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:07:47.480 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:07:47.480 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:07:47.481 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:07:47.481 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:07:47.481 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:07:47.481 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:07:47.481 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:07:47.481 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:07:47.482 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:07:47.482 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:07:47.482 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:07:47.482 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:07:47.482 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:07:47.482 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:07:47.482 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:07:47.483 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:07:47.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.483 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:07:47.483 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:07:47.483 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:07:47.483 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:07:47.484 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:07:47.484 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:07:47.484 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:07:47.484 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:07:47.484 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:07:47.484 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:07:47.484 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:07:47.485 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:07:47.485 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:07:47.485 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd": {} 2024-02-19T21:07:47.485 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.486 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 4, 2024-02-19T21:07:47.486 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.log", 2024-02-19T21:07:47.486 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:16.543816+0000", 2024-02-19T21:07:47.486 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:07:47.486 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:07:47.486 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:07:47.487 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:07:47.487 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:07:47.487 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:07:47.487 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:07:47.487 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:07:47.487 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:07:47.487 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:07:47.488 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:07:47.488 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:07:47.488 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:07:47.488 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:07:47.488 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:07:47.488 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:07:47.488 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:07:47.488 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:07:47.488 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:07:47.488 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:07:47.489 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:07:47.489 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:07:47.489 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:07:47.489 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:07:47.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.489 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "61", 2024-02-19T21:07:47.489 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:07:47.489 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:07:47.489 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:07:47.489 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:07:47.489 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:07:47.489 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:07:47.489 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:07:47.490 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:07:47.490 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:07:47.490 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:07:47.490 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:07:47.490 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:07:47.490 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:07:47.490 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:07:47.490 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:07:47.490 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:07:47.490 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:07:47.490 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:07:47.490 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:07:47.490 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:07:47.491 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:07:47.491 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:07:47.491 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:07:47.491 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:07:47.491 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:07:47.491 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:07:47.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.491 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:07:47.491 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:07:47.491 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:07:47.491 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:07:47.492 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:07:47.492 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:07:47.492 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:07:47.492 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:07:47.492 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:07:47.492 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:07:47.492 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:07:47.492 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:07:47.492 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:07:47.492 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:07:47.492 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.493 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 5, 2024-02-19T21:07:47.493 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.control", 2024-02-19T21:07:47.493 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:18.521482+0000", 2024-02-19T21:07:47.493 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:07:47.493 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:07:47.493 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:07:47.493 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:07:47.493 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:07:47.493 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:07:47.493 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:07:47.493 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:07:47.493 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:07:47.494 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:07:47.494 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:07:47.494 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:07:47.494 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:07:47.494 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:07:47.494 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:07:47.494 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:07:47.494 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:07:47.494 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:07:47.494 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:07:47.494 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:07:47.494 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:07:47.494 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:07:47.494 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:07:47.495 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:07:47.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.495 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "63", 2024-02-19T21:07:47.495 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:07:47.495 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:07:47.495 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:07:47.495 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:07:47.495 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:07:47.495 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:07:47.495 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:07:47.495 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:07:47.495 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:07:47.495 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:07:47.495 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:07:47.496 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:07:47.496 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:07:47.496 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:07:47.496 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:07:47.496 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:07:47.496 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:07:47.496 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:07:47.496 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:07:47.496 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:07:47.496 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:07:47.496 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:07:47.496 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:07:47.496 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:07:47.497 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:07:47.497 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:07:47.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.497 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:07:47.497 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:07:47.497 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:07:47.497 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:07:47.497 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:07:47.497 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:07:47.497 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:07:47.497 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:07:47.497 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:07:47.497 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:07:47.497 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:07:47.497 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:07:47.498 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:07:47.498 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:07:47.498 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.498 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 6, 2024-02-19T21:07:47.498 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.meta", 2024-02-19T21:07:47.498 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:20.586564+0000", 2024-02-19T21:07:47.498 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:07:47.498 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:07:47.498 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:07:47.498 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:07:47.498 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:07:47.499 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:07:47.500 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:07:47.500 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:07:47.500 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:07:47.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.500 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "65", 2024-02-19T21:07:47.500 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:07:47.500 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:07:47.500 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:07:47.500 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:07:47.500 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:07:47.500 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:07:47.500 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:07:47.501 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:07:47.501 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:07:47.501 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:07:47.501 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:07:47.501 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:07:47.501 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:07:47.501 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:07:47.502 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:07:47.502 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:07:47.502 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:07:47.502 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:07:47.502 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:07:47.502 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:07:47.502 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:07:47.502 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:07:47.503 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:07:47.503 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:07:47.503 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:07:47.503 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:07:47.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.503 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:07:47.503 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:07:47.503 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:07:47.504 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:07:47.504 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:07:47.504 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:07:47.504 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:07:47.504 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:07:47.504 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:07:47.504 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:07:47.505 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:07:47.505 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:07:47.505 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_bias": 4 2024-02-19T21:07:47.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.505 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:07:47.505 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:07:47.505 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.506 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 16, 2024-02-19T21:07:47.506 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-19T21:07:47.506 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:35.333192+0000", 2024-02-19T21:07:47.506 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:07:47.506 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:07:47.506 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:07:47.506 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:07:47.507 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:07:47.507 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:07:47.507 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:07:47.507 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:07:47.507 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:07:47.507 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:07:47.507 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:07:47.507 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:07:47.508 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:07:47.508 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:07:47.508 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:07:47.508 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:07:47.508 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:07:47.508 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:07:47.508 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:07:47.508 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:07:47.509 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:07:47.509 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:07:47.509 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:07:47.509 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:07:47.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.509 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "101", 2024-02-19T21:07:47.509 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:07:47.509 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:07:47.509 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:07:47.510 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:07:47.510 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:07:47.510 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:07:47.510 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:07:47.510 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:07:47.510 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:07:47.510 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:07:47.510 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:07:47.510 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:07:47.511 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:07:47.511 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:07:47.511 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:07:47.511 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:07:47.511 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:07:47.511 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:07:47.511 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:07:47.511 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:07:47.511 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:07:47.512 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:07:47.512 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:07:47.512 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:07:47.512 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:07:47.512 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:07:47.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.512 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:07:47.513 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:07:47.513 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:07:47.513 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:07:47.513 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:07:47.513 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:07:47.513 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:07:47.513 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:07:47.513 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:07:47.514 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:07:47.514 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:07:47.514 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:07:47.514 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": {} 2024-02-19T21:07:47.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.514 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 61, 2024-02-19T21:07:47.514 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "test-rados-api-smithi163-83928-6", 2024-02-19T21:07:47.514 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:52.991355+0000", 2024-02-19T21:07:47.515 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8193, 2024-02-19T21:07:47.515 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-19T21:07:47.515 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:07:47.515 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:07:47.515 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:07:47.515 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:07:47.515 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:07:47.515 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:07:47.515 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:07:47.516 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:07:47.516 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:07:47.516 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:07:47.516 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:07:47.516 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:07:47.516 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:07:47.516 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:07:47.516 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:07:47.516 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:07:47.516 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:07:47.517 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:07:47.517 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:07:47.517 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:07:47.517 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:07:47.517 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:07:47.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.518 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "287", 2024-02-19T21:07:47.518 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "287", 2024-02-19T21:07:47.518 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "287", 2024-02-19T21:07:47.518 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "287", 2024-02-19T21:07:47.518 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:07:47.518 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:07:47.518 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 21, 2024-02-19T21:07:47.519 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 284, 2024-02-19T21:07:47.519 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:07:47.519 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:07:47.519 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:07:47.519 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:07:47.519 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [ 2024-02-19T21:07:47.519 INFO:teuthology.orchestra.run.smithi163.stdout: 183 2024-02-19T21:07:47.519 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:07:47.519 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:07:47.519 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:07:47.519 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:07:47.519 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:07:47.519 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:07:47.520 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:07:47.520 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:07:47.520 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:07:47.520 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:07:47.520 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:07:47.520 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:07:47.520 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:07:47.520 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:07:47.520 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:07:47.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.520 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:07:47.520 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:07:47.520 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:07:47.521 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:07:47.521 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:07:47.521 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:07:47.521 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:07:47.521 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:07:47.521 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:07:47.521 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:07:47.521 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:07:47.521 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:07:47.521 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:07:47.521 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-02-19T21:07:47.521 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.522 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 183, 2024-02-19T21:07:47.522 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "test-rados-api-smithi163-83928-35", 2024-02-19T21:07:47.522 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:07:41.472149+0000", 2024-02-19T21:07:47.522 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8201, 2024-02-19T21:07:47.522 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-02-19T21:07:47.522 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:07:47.522 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:07:47.522 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:07:47.522 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:07:47.522 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:07:47.522 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:07:47.523 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:07:47.523 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:07:47.523 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:07:47.523 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:07:47.523 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:07:47.523 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:07:47.523 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:07:47.523 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:07:47.523 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:07:47.523 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:07:47.523 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:07:47.523 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:07:47.523 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:07:47.523 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:07:47.523 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:07:47.524 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:07:47.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.524 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "287", 2024-02-19T21:07:47.524 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "287", 2024-02-19T21:07:47.524 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "287", 2024-02-19T21:07:47.524 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "287", 2024-02-19T21:07:47.524 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:07:47.524 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:07:47.524 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 21, 2024-02-19T21:07:47.524 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 284, 2024-02-19T21:07:47.524 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:07:47.524 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:07:47.524 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:07:47.525 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:07:47.525 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:07:47.525 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": 61, 2024-02-19T21:07:47.525 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:07:47.525 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:07:47.525 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "writeback", 2024-02-19T21:07:47.525 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:07:47.525 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:07:47.525 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:07:47.525 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:07:47.525 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:07:47.525 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:07:47.525 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:07:47.525 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:07:47.525 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:07:47.526 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:07:47.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.526 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:07:47.526 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:07:47.526 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:07:47.526 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:07:47.526 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:07:47.526 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:07:47.526 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:07:47.526 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:07:47.526 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:07:47.526 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:07:47.526 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:07:47.526 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:07:47.526 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:07:47.527 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-02-19T21:07:47.527 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.527 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 184, 2024-02-19T21:07:47.527 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "MultiWrite_smithi163-83845-41", 2024-02-19T21:07:47.527 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:07:45.447887+0000", 2024-02-19T21:07:47.527 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 32769, 2024-02-19T21:07:47.527 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,creating", 2024-02-19T21:07:47.527 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 3, 2024-02-19T21:07:47.527 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:07:47.528 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:07:47.528 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 1, 2024-02-19T21:07:47.528 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:07:47.528 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:07:47.528 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:07:47.528 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:07:47.528 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:07:47.528 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:07:47.528 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 8, 2024-02-19T21:07:47.528 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 8, 2024-02-19T21:07:47.528 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 8, 2024-02-19T21:07:47.528 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 8, 2024-02-19T21:07:47.528 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 8, 2024-02-19T21:07:47.529 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:07:47.529 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:07:47.529 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:07:47.529 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:07:47.529 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:07:47.529 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:07:47.529 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:07:47.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.529 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "286", 2024-02-19T21:07:47.529 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:07:47.529 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:07:47.529 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:07:47.529 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:07:47.529 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:07:47.529 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:07:47.530 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:07:47.530 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:07:47.530 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:07:47.530 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:07:47.530 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:07:47.530 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:07:47.530 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:07:47.530 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:07:47.530 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:07:47.530 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:07:47.530 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:07:47.530 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:07:47.530 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:07:47.530 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:07:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:07:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:07:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:07:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "testprofile-MultiWrite_smithi163-83845-41", 2024-02-19T21:07:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:07:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:07:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:07:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:07:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:07:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:07:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:07:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:07:47.531 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:07:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:07:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 8192, 2024-02-19T21:07:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:07:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:07:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:07:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": {} 2024-02-19T21:07:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 185, 2024-02-19T21:07:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "RoundTripSparseReadPP_smithi163-83854-51", 2024-02-19T21:07:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:07:45.685442+0000", 2024-02-19T21:07:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 32769, 2024-02-19T21:07:47.532 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,creating", 2024-02-19T21:07:47.533 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 3, 2024-02-19T21:07:47.533 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:07:47.533 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:07:47.533 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 2, 2024-02-19T21:07:47.533 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:07:47.533 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:07:47.533 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:07:47.533 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:07:47.533 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:07:47.533 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:07:47.533 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 8, 2024-02-19T21:07:47.533 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 8, 2024-02-19T21:07:47.533 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 8, 2024-02-19T21:07:47.533 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 8, 2024-02-19T21:07:47.534 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 8, 2024-02-19T21:07:47.534 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:07:47.534 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:07:47.534 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:07:47.534 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:07:47.534 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:07:47.534 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:07:47.534 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:07:47.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.534 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "286", 2024-02-19T21:07:47.534 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:07:47.534 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:07:47.534 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:07:47.534 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:07:47.535 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:07:47.535 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:07:47.535 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:07:47.535 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:07:47.535 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:07:47.535 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:07:47.535 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:07:47.535 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:07:47.535 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:07:47.535 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:07:47.535 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:07:47.535 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:07:47.535 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:07:47.536 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:07:47.536 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:07:47.536 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:07:47.536 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:07:47.536 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:07:47.536 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:07:47.536 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "testprofile-RoundTripSparseReadPP_smithi163-83854-51", 2024-02-19T21:07:47.536 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:07:47.536 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:07:47.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.536 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:07:47.536 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:07:47.536 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:07:47.536 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:07:47.537 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:07:47.537 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:07:47.537 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:07:47.537 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:07:47.537 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 8192, 2024-02-19T21:07:47.537 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:07:47.537 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:07:47.537 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:07:47.537 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": {} 2024-02-19T21:07:47.537 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.537 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:07:47.538 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-02-19T21:07:47.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.538 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-02-19T21:07:47.538 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088", 2024-02-19T21:07:47.538 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:07:47.538 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:07:47.538 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:07:47.538 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0, 2024-02-19T21:07:47.538 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:07:47.538 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:07:47.538 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 8, 2024-02-19T21:07:47.539 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 218, 2024-02-19T21:07:47.539 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:07:47.539 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:07:47.539 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:07:47.539 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.539 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.539 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6801", 2024-02-19T21:07:47.539 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:07:47.539 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.539 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.539 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:07:47.539 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.540 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.540 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6802", 2024-02-19T21:07:47.540 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:07:47.540 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.540 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.540 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:07:47.540 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.540 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.540 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6804", 2024-02-19T21:07:47.541 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:07:47.541 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.541 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.541 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:07:47.541 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.541 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.541 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6803", 2024-02-19T21:07:47.541 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:07:47.541 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.542 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.542 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6801/1559418115", 2024-02-19T21:07:47.542 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6802/1559418115", 2024-02-19T21:07:47.542 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6804/1559418115", 2024-02-19T21:07:47.542 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6803/1559418115", 2024-02-19T21:07:47.542 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:07:47.542 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:07:47.542 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:07:47.542 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.542 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-02-19T21:07:47.542 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "e74a65fa-cb9c-42c3-9293-602cf7e9dc67", 2024-02-19T21:07:47.543 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:07:47.543 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:07:47.543 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:07:47.543 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:07:47.543 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:07:47.543 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:07:47.543 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 13, 2024-02-19T21:07:47.543 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 286, 2024-02-19T21:07:47.543 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:07:47.543 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:07:47.543 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:07:47.543 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.543 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.544 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6805", 2024-02-19T21:07:47.544 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:07:47.544 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.544 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.544 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:07:47.544 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.544 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.544 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6806", 2024-02-19T21:07:47.544 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:07:47.544 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.544 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.545 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:07:47.545 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.545 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.545 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.545 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6808", 2024-02-19T21:07:47.545 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:07:47.545 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.545 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.545 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:07:47.545 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.545 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.545 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.545 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6807", 2024-02-19T21:07:47.546 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:07:47.546 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.546 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.546 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6805/3125047432", 2024-02-19T21:07:47.546 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6806/3125047432", 2024-02-19T21:07:47.546 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6808/3125047432", 2024-02-19T21:07:47.547 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6807/3125047432", 2024-02-19T21:07:47.547 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:07:47.547 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:07:47.547 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:07:47.547 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.548 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.548 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-02-19T21:07:47.548 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "6980a625-cdc9-487e-8f8c-7997ba716a97", 2024-02-19T21:07:47.548 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:07:47.548 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:07:47.548 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:07:47.548 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:07:47.548 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:07:47.548 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:07:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 18, 2024-02-19T21:07:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 286, 2024-02-19T21:07:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:07:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:07:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:07:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6809", 2024-02-19T21:07:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:07:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:07:47.549 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.550 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.550 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6810", 2024-02-19T21:07:47.550 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:07:47.550 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.550 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.550 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:07:47.550 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.550 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.550 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6812", 2024-02-19T21:07:47.550 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:07:47.550 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.551 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.551 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.551 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:07:47.551 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.551 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.551 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.551 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6811", 2024-02-19T21:07:47.551 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:07:47.551 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.551 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.551 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.551 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6809/1563158487", 2024-02-19T21:07:47.551 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6810/1563158487", 2024-02-19T21:07:47.551 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6812/1563158487", 2024-02-19T21:07:47.551 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6811/1563158487", 2024-02-19T21:07:47.552 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:07:47.552 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:07:47.552 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:07:47.552 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.552 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.552 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-02-19T21:07:47.552 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5", 2024-02-19T21:07:47.553 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:07:47.553 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 0, 2024-02-19T21:07:47.553 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 0, 2024-02-19T21:07:47.553 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:07:47.553 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:07:47.554 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:07:47.554 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 25, 2024-02-19T21:07:47.554 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 272, 2024-02-19T21:07:47.554 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:07:47.554 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:07:47.554 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:07:47.555 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.555 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.555 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6813", 2024-02-19T21:07:47.555 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:07:47.555 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.555 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.555 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:07:47.556 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.556 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.556 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6814", 2024-02-19T21:07:47.556 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:07:47.556 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.556 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.556 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:07:47.556 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.556 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.556 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6816", 2024-02-19T21:07:47.556 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:07:47.557 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.557 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.557 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:07:47.557 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.557 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.557 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.557 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6815", 2024-02-19T21:07:47.557 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:07:47.557 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.557 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.557 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6813/2112279769", 2024-02-19T21:07:47.557 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6814/2112279769", 2024-02-19T21:07:47.558 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6816/2112279769", 2024-02-19T21:07:47.558 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6815/2112279769", 2024-02-19T21:07:47.558 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:07:47.558 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:07:47.558 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:07:47.558 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.558 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.558 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.558 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-02-19T21:07:47.558 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0", 2024-02-19T21:07:47.558 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:07:47.558 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:07:47.558 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:07:47.558 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:07:47.559 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:07:47.559 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:07:47.559 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 31, 2024-02-19T21:07:47.559 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 286, 2024-02-19T21:07:47.559 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:07:47.559 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:07:47.559 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:07:47.559 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.559 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.559 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.559 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6800", 2024-02-19T21:07:47.559 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:07:47.559 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.559 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.560 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:07:47.560 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.560 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.560 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.560 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6801", 2024-02-19T21:07:47.560 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:07:47.560 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.560 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.560 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:07:47.560 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.560 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.561 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.561 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6803", 2024-02-19T21:07:47.561 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:07:47.561 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.561 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.561 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:07:47.561 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.561 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.561 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.561 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6802", 2024-02-19T21:07:47.561 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:07:47.561 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.561 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.562 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6800/3129716965", 2024-02-19T21:07:47.562 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6801/3129716965", 2024-02-19T21:07:47.562 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6803/3129716965", 2024-02-19T21:07:47.562 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6802/3129716965", 2024-02-19T21:07:47.562 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:07:47.562 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:07:47.562 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:07:47.562 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.562 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-02-19T21:07:47.562 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f", 2024-02-19T21:07:47.563 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:07:47.563 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:07:47.563 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:07:47.563 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:07:47.563 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:07:47.563 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:07:47.563 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 37, 2024-02-19T21:07:47.563 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 286, 2024-02-19T21:07:47.563 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:07:47.563 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:07:47.563 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:07:47.563 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.564 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.564 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6804", 2024-02-19T21:07:47.564 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:07:47.564 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.564 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.564 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:07:47.564 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.564 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.564 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6805", 2024-02-19T21:07:47.564 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:07:47.564 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.564 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.565 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:07:47.565 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.565 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.565 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6807", 2024-02-19T21:07:47.565 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:07:47.565 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.565 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.565 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:07:47.565 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.565 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.565 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6806", 2024-02-19T21:07:47.566 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:07:47.566 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.566 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.566 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.566 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6804/805161991", 2024-02-19T21:07:47.566 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6805/805161991", 2024-02-19T21:07:47.566 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6807/805161991", 2024-02-19T21:07:47.566 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6806/805161991", 2024-02-19T21:07:47.566 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:07:47.566 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:07:47.566 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:07:47.566 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.566 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.567 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-02-19T21:07:47.567 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "302c95df-3842-4818-8afd-d79d8184684f", 2024-02-19T21:07:47.567 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:07:47.567 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:07:47.567 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:07:47.567 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:07:47.567 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:07:47.567 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:07:47.567 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 45, 2024-02-19T21:07:47.567 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 286, 2024-02-19T21:07:47.567 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:07:47.567 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:07:47.567 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:07:47.568 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.568 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.568 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6808", 2024-02-19T21:07:47.568 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:07:47.568 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.568 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.568 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:07:47.568 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.569 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.569 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6809", 2024-02-19T21:07:47.569 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:07:47.569 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.569 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.569 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:07:47.569 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.569 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.569 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6811", 2024-02-19T21:07:47.569 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:07:47.569 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.570 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.570 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:07:47.570 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.570 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.570 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6810", 2024-02-19T21:07:47.570 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:07:47.570 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.570 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.570 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6808/787604615", 2024-02-19T21:07:47.570 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6809/787604615", 2024-02-19T21:07:47.570 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6811/787604615", 2024-02-19T21:07:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6810/787604615", 2024-02-19T21:07:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:07:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:07:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:07:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-02-19T21:07:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f", 2024-02-19T21:07:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:07:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:07:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:07:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:07:47.571 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:07:47.572 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:07:47.572 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 53, 2024-02-19T21:07:47.572 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 283, 2024-02-19T21:07:47.572 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:07:47.572 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:07:47.572 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:07:47.572 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.572 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.572 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6812", 2024-02-19T21:07:47.573 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:07:47.573 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.573 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.573 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:07:47.573 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.573 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.573 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6813", 2024-02-19T21:07:47.573 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:07:47.573 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.573 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.573 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:07:47.574 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.574 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.574 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6815", 2024-02-19T21:07:47.574 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:07:47.574 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.574 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.575 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:07:47.575 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:07:47.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.575 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:07:47.575 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6814", 2024-02-19T21:07:47.576 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:07:47.576 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.576 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.576 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6812/157927432", 2024-02-19T21:07:47.576 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6813/157927432", 2024-02-19T21:07:47.576 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6815/157927432", 2024-02-19T21:07:47.576 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6814/157927432", 2024-02-19T21:07:47.576 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:07:47.576 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:07:47.576 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:07:47.576 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.576 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.577 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:07:47.577 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_xinfo": [ 2024-02-19T21:07:47.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.577 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-02-19T21:07:47.577 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:07:47.577 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:07:47.577 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:07:47.577 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:07:47.577 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:07:47.577 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:07:06.492524+0000", 2024-02-19T21:07:47.577 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:07:47.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.578 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-02-19T21:07:47.578 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:07:47.578 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:07:47.578 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:07:47.578 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:07:47.579 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:07:47.579 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:07:05.638559+0000", 2024-02-19T21:07:47.579 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:07:47.579 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.579 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-02-19T21:07:47.579 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:07:47.579 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:07:47.579 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:07:47.579 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:07:47.579 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:07:47.579 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:07:30.777568+0000", 2024-02-19T21:07:47.580 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:07:47.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.580 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-02-19T21:07:47.580 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:07:47.580 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:07:47.580 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:07:47.580 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:07:47.580 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 65536, 2024-02-19T21:07:47.580 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:07:07.391088+0000", 2024-02-19T21:07:47.580 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:07:47.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.581 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-02-19T21:07:47.581 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:07:47.581 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:07:47.581 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:07:47.581 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:07:47.581 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:07:47.581 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:07:02.613493+0000", 2024-02-19T21:07:47.581 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:07:47.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.581 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-02-19T21:07:47.581 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:07:47.582 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:07:47.582 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:07:47.582 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:07:47.582 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:07:47.582 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:06:47.102112+0000", 2024-02-19T21:07:47.582 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:07:47.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.582 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.582 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-02-19T21:07:47.582 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:07:47.582 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:07:47.582 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:07:47.582 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:07:47.582 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:07:47.583 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:07:00.556813+0000", 2024-02-19T21:07:47.583 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:07:47.583 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.583 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.583 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-02-19T21:07:47.583 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:07:47.583 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:07:47.583 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:07:47.583 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:07:47.583 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:07:47.583 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:07:18.963426+0000", 2024-02-19T21:07:47.583 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:07:47.583 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.583 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:07:47.583 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap": [ 2024-02-19T21:07:47.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.584 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "5.1c", 2024-02-19T21:07:47.584 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-02-19T21:07:47.584 INFO:teuthology.orchestra.run.smithi163.stdout: 7, 2024-02-19T21:07:47.584 INFO:teuthology.orchestra.run.smithi163.stdout: 0, 2024-02-19T21:07:47.584 INFO:teuthology.orchestra.run.smithi163.stdout: 3 2024-02-19T21:07:47.584 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.584 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.584 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:07:47.584 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_items": [ 2024-02-19T21:07:47.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.584 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.7", 2024-02-19T21:07:47.584 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:07:47.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.585 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:07:47.585 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-02-19T21:07:47.585 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.585 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.585 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.585 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.d", 2024-02-19T21:07:47.585 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:07:47.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.585 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:07:47.585 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 5 2024-02-19T21:07:47.585 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.586 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.586 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.12", 2024-02-19T21:07:47.586 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:07:47.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.586 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:07:47.586 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 5 2024-02-19T21:07:47.586 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.586 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.586 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.586 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:07:47.586 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_temp": [], 2024-02-19T21:07:47.587 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_temp": [], 2024-02-19T21:07:47.587 INFO:teuthology.orchestra.run.smithi163.stdout: "blocklist": { 2024-02-19T21:07:47.587 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1427774891": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:07:47.587 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3282102947": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:07:47.587 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2735092549": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:07:47.587 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1235588846": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:07:47.587 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2731225015": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:07:47.587 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/506491651": "2024-02-20T20:56:02.185409+0000", 2024-02-19T21:07:47.587 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/168117826": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:07:47.587 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1458009465": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:07:47.587 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/541631472": "2024-02-20T20:56:02.185409+0000", 2024-02-19T21:07:47.587 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2248999541": "2024-02-20T20:56:02.185409+0000" 2024-02-19T21:07:47.587 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.587 INFO:teuthology.orchestra.run.smithi163.stdout: "range_blocklist": {}, 2024-02-19T21:07:47.588 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profiles": { 2024-02-19T21:07:47.588 INFO:teuthology.orchestra.run.smithi163.stdout: "default": { 2024-02-19T21:07:47.588 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-02-19T21:07:47.588 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-02-19T21:07:47.588 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-02-19T21:07:47.588 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-02-19T21:07:47.588 INFO:teuthology.orchestra.run.smithi163.stdout: "ruleset-failure-domain": "osd", 2024-02-19T21:07:47.588 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van" 2024-02-19T21:07:47.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.588 INFO:teuthology.orchestra.run.smithi163.stdout: "testprofile-MultiWrite_smithi163-83845-41": { 2024-02-19T21:07:47.588 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-device-class": "", 2024-02-19T21:07:47.588 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-02-19T21:07:47.588 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-root": "default", 2024-02-19T21:07:47.588 INFO:teuthology.orchestra.run.smithi163.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-19T21:07:47.589 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-02-19T21:07:47.589 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-02-19T21:07:47.589 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-02-19T21:07:47.589 INFO:teuthology.orchestra.run.smithi163.stdout: "ruleset-failure-domain": "osd", 2024-02-19T21:07:47.589 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van", 2024-02-19T21:07:47.589 INFO:teuthology.orchestra.run.smithi163.stdout: "w": "8" 2024-02-19T21:07:47.589 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.589 INFO:teuthology.orchestra.run.smithi163.stdout: "testprofile-RoundTripSparseReadPP_smithi163-83854-51": { 2024-02-19T21:07:47.589 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-device-class": "", 2024-02-19T21:07:47.589 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-02-19T21:07:47.589 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-root": "default", 2024-02-19T21:07:47.589 INFO:teuthology.orchestra.run.smithi163.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-19T21:07:47.589 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-02-19T21:07:47.589 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-02-19T21:07:47.590 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-02-19T21:07:47.590 INFO:teuthology.orchestra.run.smithi163.stdout: "ruleset-failure-domain": "osd", 2024-02-19T21:07:47.590 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van", 2024-02-19T21:07:47.590 INFO:teuthology.orchestra.run.smithi163.stdout: "w": "8" 2024-02-19T21:07:47.590 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.590 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps_queue": [ 2024-02-19T21:07:47.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.590 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 11, 2024-02-19T21:07:47.590 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:07:47.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.590 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:07:47.590 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.591 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:07:47.591 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.591 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 9, 2024-02-19T21:07:47.591 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:07:47.591 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.591 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.591 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 25, 2024-02-19T21:07:47.592 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:07:47.592 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.592 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:07:47.592 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.592 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.592 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.592 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:07:47.592 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.592 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.592 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.592 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:07:47.592 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.592 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.593 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.593 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.593 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.593 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 57, 2024-02-19T21:07:47.593 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:07:47.593 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.593 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:07:47.593 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:07:47.593 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.593 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.593 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:07:47.593 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:07:47.593 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.593 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.593 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.594 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 61, 2024-02-19T21:07:47.594 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:07:47.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.594 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 20, 2024-02-19T21:07:47.594 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.594 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.594 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.594 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.594 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 76, 2024-02-19T21:07:47.594 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:07:47.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.594 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:07:47.594 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.595 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:07:47.595 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.595 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:07:47.595 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.595 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.595 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.595 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 85, 2024-02-19T21:07:47.595 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:07:47.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.596 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:07:47.596 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.596 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.596 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.596 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.596 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 92, 2024-02-19T21:07:47.596 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:07:47.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.596 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:07:47.596 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:07:47.597 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.597 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.597 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:07:47.597 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:07:47.597 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.597 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.597 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.597 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.597 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 94, 2024-02-19T21:07:47.597 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:07:47.597 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.597 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 4, 2024-02-19T21:07:47.597 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 4 2024-02-19T21:07:47.597 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.597 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.598 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 107, 2024-02-19T21:07:47.598 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:07:47.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.598 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:07:47.598 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.598 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.598 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.598 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 112, 2024-02-19T21:07:47.598 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:07:47.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.599 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 1, 2024-02-19T21:07:47.599 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.599 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:07:47.599 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.599 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:07:47.599 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.599 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:07:47.599 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.599 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.600 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.600 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 126, 2024-02-19T21:07:47.600 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:07:47.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.600 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:07:47.600 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:07:47.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.600 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.600 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:07:47.600 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:07:47.600 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.601 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.601 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 10, 2024-02-19T21:07:47.601 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:07:47.601 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.601 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.601 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.601 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.601 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 132, 2024-02-19T21:07:47.601 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:07:47.601 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.601 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 12, 2024-02-19T21:07:47.601 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.601 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 154, 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 14, 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 164, 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.602 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 17, 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 178, 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 20, 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: "new_removed_snaps": [], 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: "new_purged_snaps": [], 2024-02-19T21:07:47.603 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_node_flags": {}, 2024-02-19T21:07:47.604 INFO:teuthology.orchestra.run.smithi163.stdout: "device_class_flags": {}, 2024-02-19T21:07:47.604 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode": { 2024-02-19T21:07:47.604 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_enabled": false, 2024-02-19T21:07:47.604 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_bucket_count": 0, 2024-02-19T21:07:47.604 INFO:teuthology.orchestra.run.smithi163.stdout: "degraded_stretch_mode": 0, 2024-02-19T21:07:47.604 INFO:teuthology.orchestra.run.smithi163.stdout: "recovering_stretch_mode": 0, 2024-02-19T21:07:47.604 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_bucket": 0 2024-02-19T21:07:47.604 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:07:47.604 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:07:47.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.464+0000 7f8015590700 10 librados: watch_flush enter 2024-02-19T21:07:47.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.464+0000 7f8015590700 10 librados: watch_flush exit 2024-02-19T21:07:47.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.464+0000 7f8015590700 20 client.48307.objecter shutdown clearing up homeless session... 2024-02-19T21:07:47.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.464+0000 7f8015590700 10 client.48307.objecter successfully canceled tick 2024-02-19T21:07:47.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.464+0000 7f8015590700 1 -- 172.21.15.163:0/2590724172 >> v1:172.21.15.163:6800/1552514681 conn(0x7f7ff8065e60 legacy=0x7f7ff8068320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:47.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.464+0000 7f8015590700 1 -- 172.21.15.163:0/2590724172 >> v1:172.21.15.193:6789/0 conn(0x7f80100fc880 legacy=0x7f80100f8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:47.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.464+0000 7f8015590700 1 -- 172.21.15.163:0/2590724172 shutdown_connections 2024-02-19T21:07:47.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.464+0000 7f8015590700 1 -- 172.21.15.163:0/2590724172 >> 172.21.15.163:0/2590724172 conn(0x7f80100f24a0 msgr2=0x7f8010106700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:47.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.464+0000 7f8015590700 1 -- 172.21.15.163:0/2590724172 shutdown_connections 2024-02-19T21:07:47.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.464+0000 7f8015590700 1 -- 172.21.15.163:0/2590724172 wait complete. 2024-02-19T21:07:47.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:47.464+0000 7f8015590700 1 librados: shutdown 2024-02-19T21:07:47.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:47.607 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:47.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:47.707 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:47.737 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:47.993 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:07:47.993 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:07:48.019 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:07:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:07:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:07:48.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:47.170981+0000 mgr.y (mgr.14550) 288 : cluster [DBG] pgmap v401: 249 pgs: 16 unknown, 233 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1 B/s, 0 objects/s recovering 2024-02-19T21:07:48.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:47.219309+0000 mon.a (mon.0) 1990 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:48.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:47.219370+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e287: 8 total, 8 up, 7 in 2024-02-19T21:07:48.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:47.220540+0000 mon.b (mon.1) 448 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35"}]: dispatch 2024-02-19T21:07:48.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:47.221317+0000 mon.a (mon.0) 1992 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35"}]: dispatch 2024-02-19T21:07:48.317 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:47.461378+0000 mon.b (mon.1) 449 : audit [DBG] from='client.? v1:172.21.15.163:0/2590724172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:07:48.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:47.170981+0000 mgr.y (mgr.14550) 288 : cluster [DBG] pgmap v401: 249 pgs: 16 unknown, 233 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1 B/s, 0 objects/s recovering 2024-02-19T21:07:48.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:47.219309+0000 mon.a (mon.0) 1990 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:48.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:47.219370+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e287: 8 total, 8 up, 7 in 2024-02-19T21:07:48.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:47.220540+0000 mon.b (mon.1) 448 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35"}]: dispatch 2024-02-19T21:07:48.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:47.221317+0000 mon.a (mon.0) 1992 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35"}]: dispatch 2024-02-19T21:07:48.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:47.461378+0000 mon.b (mon.1) 449 : audit [DBG] from='client.? v1:172.21.15.163:0/2590724172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:07:48.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:47.170981+0000 mgr.y (mgr.14550) 288 : cluster [DBG] pgmap v401: 249 pgs: 16 unknown, 233 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 1 B/s, 0 objects/s recovering 2024-02-19T21:07:48.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:47.219309+0000 mon.a (mon.0) 1990 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:48.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:47.219370+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e287: 8 total, 8 up, 7 in 2024-02-19T21:07:48.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:47.220540+0000 mon.b (mon.1) 448 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35"}]: dispatch 2024-02-19T21:07:48.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:47.221317+0000 mon.a (mon.0) 1992 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35"}]: dispatch 2024-02-19T21:07:48.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:47.461378+0000 mon.b (mon.1) 449 : audit [DBG] from='client.? v1:172.21.15.163:0/2590724172' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:07:48.586 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:48.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:48.630 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-19T21:07:48.690 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:48.717 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:48.736 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:48.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:48.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:48.841 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:48.868 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:48.889 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:48.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:48.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:48.906 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:07:48.996 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:49.024 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:49.165 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:49.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:49.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:49.269 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:07:49.300 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:49.344 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:07:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:49.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:07:49.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.154+0000 7ffa9466a700 1 -- 172.21.15.163:0/706616925 >> v1:172.21.15.163:6789/0 conn(0x7ffa8c10a8a0 legacy=0x7ffa8c10cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:49.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.154+0000 7ffa9466a700 1 -- 172.21.15.163:0/706616925 shutdown_connections 2024-02-19T21:07:49.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.154+0000 7ffa9466a700 1 -- 172.21.15.163:0/706616925 >> 172.21.15.163:0/706616925 conn(0x7ffa8c06d7e0 msgr2=0x7ffa8c06fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:49.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.154+0000 7ffa9466a700 1 -- 172.21.15.163:0/706616925 shutdown_connections 2024-02-19T21:07:49.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.154+0000 7ffa9466a700 1 -- 172.21.15.163:0/706616925 wait complete. 2024-02-19T21:07:49.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa9466a700 1 librados: starting msgr at 2024-02-19T21:07:49.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa9466a700 1 librados: starting objecter 2024-02-19T21:07:49.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa9466a700 1 Processor -- start 2024-02-19T21:07:49.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa9466a700 1 -- start start 2024-02-19T21:07:49.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa9466a700 1 librados: setting wanted keys 2024-02-19T21:07:49.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa9466a700 1 librados: calling monclient init 2024-02-19T21:07:49.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa9466a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa8c11d890 con 0x7ffa8c103f40 2024-02-19T21:07:49.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa9466a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa8c11ea70 con 0x7ffa8c10a8a0 2024-02-19T21:07:49.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa9466a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa8c11fc70 con 0x7ffa8c106d90 2024-02-19T21:07:49.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa91c05700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7ffa8c106d90 0x7ffa8c1005f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:41580/0 (socket says 172.21.15.163:41580) 2024-02-19T21:07:49.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa91c05700 1 -- 172.21.15.163:0/2094650189 learned_addr learned my addr 172.21.15.163:0/2094650189 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:49.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa837fe700 10 client.?.objecter ms_handle_connect 0x7ffa8c106d90 2024-02-19T21:07:49.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa837fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa837fe700 10 client.?.objecter ms_handle_connect 0x7ffa8c103f40 2024-02-19T21:07:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa837fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa837fe700 10 client.?.objecter ms_handle_connect 0x7ffa8c10a8a0 2024-02-19T21:07:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa837fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229133622 0 0) 0x7ffa8c11fc70 con 0x7ffa8c106d90 2024-02-19T21:07:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa78003800 con 0x7ffa8c106d90 2024-02-19T21:07:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346947141 0 0) 0x7ffa8c11d890 con 0x7ffa8c103f40 2024-02-19T21:07:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa8c11fc70 con 0x7ffa8c103f40 2024-02-19T21:07:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467218946 0 0) 0x7ffa8c11ea70 con 0x7ffa8c10a8a0 2024-02-19T21:07:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa8c11d890 con 0x7ffa8c10a8a0 2024-02-19T21:07:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.155+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 120236465 0 0) 0x7ffa78003800 con 0x7ffa8c106d90 2024-02-19T21:07:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa8c11ea70 con 0x7ffa8c106d90 2024-02-19T21:07:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa88001980 con 0x7ffa8c106d90 2024-02-19T21:07:49.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3607192330 0 0) 0x7ffa8c11fc70 con 0x7ffa8c103f40 2024-02-19T21:07:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa78003800 con 0x7ffa8c103f40 2024-02-19T21:07:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa7c001620 con 0x7ffa8c103f40 2024-02-19T21:07:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3935529303 0 0) 0x7ffa8c11d890 con 0x7ffa8c10a8a0 2024-02-19T21:07:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa8c11fc70 con 0x7ffa8c10a8a0 2024-02-19T21:07:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa84003000 con 0x7ffa8c10a8a0 2024-02-19T21:07:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2453592794 0 0) 0x7ffa8c11ea70 con 0x7ffa8c106d90 2024-02-19T21:07:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 >> v1:172.21.15.193:6789/0 conn(0x7ffa8c10a8a0 legacy=0x7ffa8c11bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 >> v1:172.21.15.163:6789/0 conn(0x7ffa8c103f40 legacy=0x7ffa8c0ffee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa8c120e70 con 0x7ffa8c106d90 2024-02-19T21:07:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8c11da80 con 0x7ffa8c106d90 2024-02-19T21:07:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa9466a700 10 client.49496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa8c11df70 con 0x7ffa8c106d90 2024-02-19T21:07:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa9466a700 1 librados: init done 2024-02-19T21:07:49.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ffa88002100 con 0x7ffa8c106d90 2024-02-19T21:07:49.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.156+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa88005a70 con 0x7ffa8c106d90 2024-02-19T21:07:49.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.157+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ffa70004f00 con 0x7ffa8c106d90 2024-02-19T21:07:49.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.157+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ffa88001c60 con 0x7ffa8c106d90 2024-02-19T21:07:49.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.157+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(288..288 src has 43..288) v4 ==== 8796+0+0 (unknown 2733314910 0 0) 0x7ffa88078ef0 con 0x7ffa8c106d90 2024-02-19T21:07:49.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.157+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 osd_map(288..288 src has 43..288) v4 2024-02-19T21:07:49.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.157+0000 7ffa837fe700 3 client.49496.objecter handle_osd_map got epochs [288,288] > 0 2024-02-19T21:07:49.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.157+0000 7ffa837fe700 3 client.49496.objecter handle_osd_map decoding full epoch 288 2024-02-19T21:07:49.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.158+0000 7ffa837fe700 20 client.49496.objecter dump_active .. 0 homeless 2024-02-19T21:07:49.448 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:07:49.449 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:07:49.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.158+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v288) v1 ==== 54+0+16 (unknown 3903457381 0 2420000192) 0x7ffa8807b2e0 con 0x7ffa8c106d90 2024-02-19T21:07:49.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.158+0000 7ffa837fe700 10 client.49496.objecter ms_handle_connect 0x7ffa78063cb0 2024-02-19T21:07:49.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.158+0000 7ffa9466a700 10 client.49496.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:49.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.158+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 1 2024-02-19T21:07:49.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.158+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa700051a0 osd=0 3 2024-02-19T21:07:49.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.158+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa700051a0 osd=0 3 2024-02-19T21:07:49.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.158+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa700051a0 osd=0 3 2024-02-19T21:07:49.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.158+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 1 2024-02-19T21:07:49.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.158+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa700051a0 osd=0 3 2024-02-19T21:07:49.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.158+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 0 1 2024-02-19T21:07:49.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.158+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa700051a0 osd=0 4 2024-02-19T21:07:49.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.158+0000 7ffa9466a700 10 client.49496.objecter _send_command 1 2024-02-19T21:07:49.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.158+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa70004950 con 0x7ffa70004530 2024-02-19T21:07:49.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.158+0000 7ffa837fe700 10 client.49496.objecter ms_handle_connect 0x7ffa70004530 2024-02-19T21:07:49.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.159+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa70004950 con 0x7ffa70004530 2024-02-19T21:07:49.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.159+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:07:49.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.159+0000 7ffa837fe700 10 client.49496.objecter _finish_command 1 = system:0 2024-02-19T21:07:49.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.159+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa700051a0 osd=0 4 2024-02-19T21:07:49.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.159+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 0 1 2024-02-19T21:07:49.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.174+0000 7ffa9466a700 10 client.49496.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:49.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.174+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 2 2024-02-19T21:07:49.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.174+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa700051a0 osd=0 3 2024-02-19T21:07:49.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.174+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa700051a0 osd=0 3 2024-02-19T21:07:49.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.174+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa700051a0 osd=0 3 2024-02-19T21:07:49.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.174+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 2 2024-02-19T21:07:49.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.174+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa700051a0 osd=0 3 2024-02-19T21:07:49.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.174+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 0 2 2024-02-19T21:07:49.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.174+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa700051a0 osd=0 4 2024-02-19T21:07:49.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.174+0000 7ffa9466a700 10 client.49496.objecter _send_command 2 2024-02-19T21:07:49.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.174+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffa70004d30 con 0x7ffa70004530 2024-02-19T21:07:49.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffa70004d30 con 0x7ffa70004530 2024-02-19T21:07:49.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:49.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa837fe700 10 client.49496.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa700051a0 osd=0 4 2024-02-19T21:07:49.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 0 2 2024-02-19T21:07:49.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa9466a700 10 client.49496.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:49.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 3 2024-02-19T21:07:49.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70003fe0 osd=1 3 2024-02-19T21:07:49.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70003fe0 osd=1 3 2024-02-19T21:07:49.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70003fe0 osd=1 3 2024-02-19T21:07:49.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 3 2024-02-19T21:07:49.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa70003fe0 osd=1 3 2024-02-19T21:07:49.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 1 3 2024-02-19T21:07:49.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70003fe0 osd=1 4 2024-02-19T21:07:49.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa9466a700 10 client.49496.objecter _send_command 3 2024-02-19T21:07:49.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.175+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa7000c000 con 0x7ffa70004150 2024-02-19T21:07:49.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.176+0000 7ffa837fe700 10 client.49496.objecter ms_handle_connect 0x7ffa70004150 2024-02-19T21:07:49.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.176+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa7000c000 con 0x7ffa70004150 2024-02-19T21:07:49.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.176+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:07:49.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.176+0000 7ffa837fe700 10 client.49496.objecter _finish_command 3 = system:0 2024-02-19T21:07:49.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.176+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa70003fe0 osd=1 4 2024-02-19T21:07:49.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.176+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 1 3 2024-02-19T21:07:49.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.191+0000 7ffa9466a700 10 client.49496.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:49.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.191+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 4 2024-02-19T21:07:49.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.191+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70003fe0 osd=1 3 2024-02-19T21:07:49.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.191+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70003fe0 osd=1 3 2024-02-19T21:07:49.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.191+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70003fe0 osd=1 3 2024-02-19T21:07:49.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.191+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 4 2024-02-19T21:07:49.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.191+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa70003fe0 osd=1 3 2024-02-19T21:07:49.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.191+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 1 4 2024-02-19T21:07:49.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.191+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70003fe0 osd=1 4 2024-02-19T21:07:49.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.191+0000 7ffa9466a700 10 client.49496.objecter _send_command 4 2024-02-19T21:07:49.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.191+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffa700018b0 con 0x7ffa70004150 2024-02-19T21:07:49.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffa700018b0 con 0x7ffa70004150 2024-02-19T21:07:49.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:49.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa837fe700 10 client.49496.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa70003fe0 osd=1 4 2024-02-19T21:07:49.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 1 4 2024-02-19T21:07:49.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa9466a700 10 client.49496.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:49.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 5 2024-02-19T21:07:49.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70003630 osd=2 3 2024-02-19T21:07:49.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70003630 osd=2 3 2024-02-19T21:07:49.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70003630 osd=2 3 2024-02-19T21:07:49.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 5 2024-02-19T21:07:49.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa70003630 osd=2 3 2024-02-19T21:07:49.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 2 5 2024-02-19T21:07:49.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70003630 osd=2 4 2024-02-19T21:07:49.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa9466a700 10 client.49496.objecter _send_command 5 2024-02-19T21:07:49.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.192+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa7000de90 con 0x7ffa7000d2f0 2024-02-19T21:07:49.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.193+0000 7ffa837fe700 10 client.49496.objecter ms_handle_connect 0x7ffa7000d2f0 2024-02-19T21:07:49.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.193+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa7000de90 con 0x7ffa7000d2f0 2024-02-19T21:07:49.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.193+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:07:49.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.193+0000 7ffa837fe700 10 client.49496.objecter _finish_command 5 = system:0 2024-02-19T21:07:49.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.193+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa70003630 osd=2 4 2024-02-19T21:07:49.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.193+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 2 5 2024-02-19T21:07:49.455 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:07:49.455 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:07:49.455 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:07:49.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.207+0000 7ffa9466a700 10 client.49496.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:49.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.207+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 6 2024-02-19T21:07:49.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.207+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70003630 osd=2 3 2024-02-19T21:07:49.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.208+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70003630 osd=2 3 2024-02-19T21:07:49.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.208+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70003630 osd=2 3 2024-02-19T21:07:49.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.208+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 6 2024-02-19T21:07:49.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.208+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa70003630 osd=2 3 2024-02-19T21:07:49.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.208+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 2 6 2024-02-19T21:07:49.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.208+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70003630 osd=2 4 2024-02-19T21:07:49.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.208+0000 7ffa9466a700 10 client.49496.objecter _send_command 6 2024-02-19T21:07:49.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.208+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffa700030a0 con 0x7ffa7000d2f0 2024-02-19T21:07:49.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.211+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffa700030a0 con 0x7ffa7000d2f0 2024-02-19T21:07:49.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.211+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:49.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.211+0000 7ffa837fe700 10 client.49496.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.211+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa70003630 osd=2 4 2024-02-19T21:07:49.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.211+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 2 6 2024-02-19T21:07:49.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.212+0000 7ffa9466a700 10 client.49496.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:49.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.212+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 7 2024-02-19T21:07:49.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.212+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70011970 osd=3 3 2024-02-19T21:07:49.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.212+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70011970 osd=3 3 2024-02-19T21:07:49.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.212+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70011970 osd=3 3 2024-02-19T21:07:49.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.212+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 7 2024-02-19T21:07:49.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.212+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa70011970 osd=3 3 2024-02-19T21:07:49.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.212+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 3 7 2024-02-19T21:07:49.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.212+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70011970 osd=3 4 2024-02-19T21:07:49.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.212+0000 7ffa9466a700 10 client.49496.objecter _send_command 7 2024-02-19T21:07:49.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.212+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa70015b00 con 0x7ffa70011ff0 2024-02-19T21:07:49.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.212+0000 7ffa837fe700 10 client.49496.objecter ms_handle_connect 0x7ffa70011ff0 2024-02-19T21:07:49.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.213+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa70015b00 con 0x7ffa70011ff0 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.213+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.213+0000 7ffa837fe700 10 client.49496.objecter _finish_command 7 = system:0 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.213+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa70011970 osd=3 4 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.213+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 3 7 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.230+0000 7ffa9466a700 10 client.49496.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.230+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 8 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.230+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70011970 osd=3 3 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.230+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70011970 osd=3 3 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.230+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70011970 osd=3 3 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.230+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 8 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.230+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa70011970 osd=3 3 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.230+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 3 8 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.230+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70011970 osd=3 4 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.230+0000 7ffa9466a700 10 client.49496.objecter _send_command 8 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.230+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffa7000e240 con 0x7ffa70011ff0 2024-02-19T21:07:49.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffa7000e240 con 0x7ffa70011ff0 2024-02-19T21:07:49.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:49.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa837fe700 10 client.49496.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa70011970 osd=3 4 2024-02-19T21:07:49.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 3 8 2024-02-19T21:07:49.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa9466a700 10 client.49496.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:49.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 9 2024-02-19T21:07:49.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa7000e000 osd=4 3 2024-02-19T21:07:49.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa7000e000 osd=4 3 2024-02-19T21:07:49.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa7000e000 osd=4 3 2024-02-19T21:07:49.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 9 2024-02-19T21:07:49.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa7000e000 osd=4 3 2024-02-19T21:07:49.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 4 9 2024-02-19T21:07:49.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa7000e000 osd=4 4 2024-02-19T21:07:49.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa9466a700 10 client.49496.objecter _send_command 9 2024-02-19T21:07:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.232+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa7000e1a0 con 0x7ffa7000ce50 2024-02-19T21:07:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.233+0000 7ffa837fe700 10 client.49496.objecter ms_handle_connect 0x7ffa7000ce50 2024-02-19T21:07:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.233+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa7000e1a0 con 0x7ffa7000ce50 2024-02-19T21:07:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.233+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:07:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.233+0000 7ffa837fe700 10 client.49496.objecter _finish_command 9 = system:0 2024-02-19T21:07:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.234+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa7000e000 osd=4 4 2024-02-19T21:07:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.234+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 4 9 2024-02-19T21:07:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.249+0000 7ffa9466a700 10 client.49496.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.249+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 10 2024-02-19T21:07:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.249+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa7000e000 osd=4 3 2024-02-19T21:07:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.249+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa7000e000 osd=4 3 2024-02-19T21:07:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.249+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa7000e000 osd=4 3 2024-02-19T21:07:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.249+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 10 2024-02-19T21:07:49.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.249+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa7000e000 osd=4 3 2024-02-19T21:07:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.249+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 4 10 2024-02-19T21:07:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.249+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa7000e000 osd=4 4 2024-02-19T21:07:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.249+0000 7ffa9466a700 10 client.49496.objecter _send_command 10 2024-02-19T21:07:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.249+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffa7000e1a0 con 0x7ffa7000ce50 2024-02-19T21:07:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffa7000e1a0 con 0x7ffa7000ce50 2024-02-19T21:07:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa837fe700 10 client.49496.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa7000e000 osd=4 4 2024-02-19T21:07:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 4 10 2024-02-19T21:07:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa9466a700 10 client.49496.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 11 2024-02-19T21:07:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70017550 osd=5 3 2024-02-19T21:07:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70017550 osd=5 3 2024-02-19T21:07:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70017550 osd=5 3 2024-02-19T21:07:49.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 11 2024-02-19T21:07:49.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa70017550 osd=5 3 2024-02-19T21:07:49.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 5 11 2024-02-19T21:07:49.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70017550 osd=5 4 2024-02-19T21:07:49.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa9466a700 10 client.49496.objecter _send_command 11 2024-02-19T21:07:49.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa70018090 con 0x7ffa70017bd0 2024-02-19T21:07:49.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.251+0000 7ffa837fe700 10 client.49496.objecter ms_handle_connect 0x7ffa70017bd0 2024-02-19T21:07:49.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.252+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa70018090 con 0x7ffa70017bd0 2024-02-19T21:07:49.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.252+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:07:49.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.252+0000 7ffa837fe700 10 client.49496.objecter _finish_command 11 = system:0 2024-02-19T21:07:49.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.252+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa70017550 osd=5 4 2024-02-19T21:07:49.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.252+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 5 11 2024-02-19T21:07:49.463 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:07:49.463 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:07:49.463 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:07:49.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.267+0000 7ffa9466a700 10 client.49496.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:49.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.267+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 12 2024-02-19T21:07:49.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.267+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70017550 osd=5 3 2024-02-19T21:07:49.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.267+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70017550 osd=5 3 2024-02-19T21:07:49.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.267+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70017550 osd=5 3 2024-02-19T21:07:49.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.267+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 12 2024-02-19T21:07:49.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.267+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa70017550 osd=5 3 2024-02-19T21:07:49.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.267+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 5 12 2024-02-19T21:07:49.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.267+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70017550 osd=5 4 2024-02-19T21:07:49.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.267+0000 7ffa9466a700 10 client.49496.objecter _send_command 12 2024-02-19T21:07:49.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.267+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffa70018090 con 0x7ffa70017bd0 2024-02-19T21:07:49.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffa70018090 con 0x7ffa70017bd0 2024-02-19T21:07:49.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:49.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa837fe700 10 client.49496.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa70017550 osd=5 4 2024-02-19T21:07:49.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 5 12 2024-02-19T21:07:49.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa9466a700 10 client.49496.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:49.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 13 2024-02-19T21:07:49.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70018090 osd=6 3 2024-02-19T21:07:49.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70018090 osd=6 3 2024-02-19T21:07:49.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70018090 osd=6 3 2024-02-19T21:07:49.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 13 2024-02-19T21:07:49.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa70018090 osd=6 3 2024-02-19T21:07:49.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 6 13 2024-02-19T21:07:49.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70018090 osd=6 4 2024-02-19T21:07:49.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa9466a700 10 client.49496.objecter _send_command 13 2024-02-19T21:07:49.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.270+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa70022cd0 con 0x7ffa7001f080 2024-02-19T21:07:49.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.272+0000 7ffa837fe700 10 client.49496.objecter ms_handle_connect 0x7ffa7001f080 2024-02-19T21:07:49.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.278+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa70022cd0 con 0x7ffa7001f080 2024-02-19T21:07:49.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.278+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:07:49.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.278+0000 7ffa837fe700 10 client.49496.objecter _finish_command 13 = system:0 2024-02-19T21:07:49.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.278+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa70018090 osd=6 4 2024-02-19T21:07:49.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.278+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 6 13 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.296+0000 7ffa9466a700 10 client.49496.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.296+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 14 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.296+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70018090 osd=6 3 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.296+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70018090 osd=6 3 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.296+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70018090 osd=6 3 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.296+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 14 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.296+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa70018090 osd=6 3 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.296+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 6 14 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.296+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70018090 osd=6 4 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.296+0000 7ffa9466a700 10 client.49496.objecter _send_command 14 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.296+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffa70024480 con 0x7ffa7001f080 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffa70024480 con 0x7ffa7001f080 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa837fe700 10 client.49496.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa70018090 osd=6 4 2024-02-19T21:07:49.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 6 14 2024-02-19T21:07:49.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa9466a700 10 client.49496.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:49.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 15 2024-02-19T21:07:49.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70024480 osd=7 3 2024-02-19T21:07:49.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70024480 osd=7 3 2024-02-19T21:07:49.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70024480 osd=7 3 2024-02-19T21:07:49.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 15 2024-02-19T21:07:49.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa70024480 osd=7 3 2024-02-19T21:07:49.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 7 15 2024-02-19T21:07:49.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70024480 osd=7 4 2024-02-19T21:07:49.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa9466a700 10 client.49496.objecter _send_command 15 2024-02-19T21:07:49.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa700250b0 con 0x7ffa70024b00 2024-02-19T21:07:49.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.298+0000 7ffa837fe700 10 client.49496.objecter ms_handle_connect 0x7ffa70024b00 2024-02-19T21:07:49.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.299+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa700250b0 con 0x7ffa70024b00 2024-02-19T21:07:49.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.299+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:07:49.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.299+0000 7ffa837fe700 10 client.49496.objecter _finish_command 15 = system:0 2024-02-19T21:07:49.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.299+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa70024480 osd=7 4 2024-02-19T21:07:49.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.299+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 7 15 2024-02-19T21:07:49.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.322+0000 7ffa9466a700 10 client.49496.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:07:49.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.322+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign -1 16 2024-02-19T21:07:49.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.322+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70024480 osd=7 3 2024-02-19T21:07:49.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.322+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70024480 osd=7 3 2024-02-19T21:07:49.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.322+0000 7ffa9466a700 20 client.49496.objecter _get_session s=0x7ffa70024480 osd=7 3 2024-02-19T21:07:49.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.322+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_remove -1 16 2024-02-19T21:07:49.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.322+0000 7ffa9466a700 20 client.49496.objecter get_session s=0x7ffa70024480 osd=7 3 2024-02-19T21:07:49.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.322+0000 7ffa9466a700 15 client.49496.objecter _session_command_op_assign 7 16 2024-02-19T21:07:49.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.322+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70024480 osd=7 4 2024-02-19T21:07:49.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.322+0000 7ffa9466a700 10 client.49496.objecter _send_command 16 2024-02-19T21:07:49.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.322+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffa700250b0 con 0x7ffa70024b00 2024-02-19T21:07:49.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa837fe700 1 -- 172.21.15.163:0/2094650189 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffa700250b0 con 0x7ffa70024b00 2024-02-19T21:07:49.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa837fe700 10 client.49496.objecter ms_dispatch 0x7ffa8c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:07:49.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa837fe700 10 client.49496.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa837fe700 20 client.49496.objecter put_session s=0x7ffa70024480 osd=7 4 2024-02-19T21:07:49.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa837fe700 15 client.49496.objecter _session_command_op_remove 7 16 2024-02-19T21:07:49.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 10 librados: watch_flush enter 2024-02-19T21:07:49.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 10 librados: watch_flush exit 2024-02-19T21:07:49.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 10 client.49496.objecter close_session for osd.0 2024-02-19T21:07:49.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 >> v1:172.21.15.163:6801/1559418115 conn(0x7ffa70004530 legacy=0x7ffa70002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:49.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa700051a0 osd=0 1 2024-02-19T21:07:49.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 10 client.49496.objecter close_session for osd.1 2024-02-19T21:07:49.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 >> v1:172.21.15.163:6805/3125047432 conn(0x7ffa70004150 legacy=0x7ffa7000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:49.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70003fe0 osd=1 1 2024-02-19T21:07:49.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 10 client.49496.objecter close_session for osd.2 2024-02-19T21:07:49.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 >> v1:172.21.15.163:6809/1563158487 conn(0x7ffa7000d2f0 legacy=0x7ffa7000d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:49.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70003630 osd=2 1 2024-02-19T21:07:49.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 10 client.49496.objecter close_session for osd.3 2024-02-19T21:07:49.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 >> v1:172.21.15.163:6813/2112279769 conn(0x7ffa70011ff0 legacy=0x7ffa700143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:49.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70011970 osd=3 1 2024-02-19T21:07:49.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 10 client.49496.objecter close_session for osd.4 2024-02-19T21:07:49.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 >> v1:172.21.15.193:6800/3129716965 conn(0x7ffa7000ce50 legacy=0x7ffa70016df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:49.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa7000e000 osd=4 1 2024-02-19T21:07:49.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 10 client.49496.objecter close_session for osd.5 2024-02-19T21:07:49.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 >> v1:172.21.15.193:6804/805161991 conn(0x7ffa70017bd0 legacy=0x7ffa7001d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:49.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70017550 osd=5 1 2024-02-19T21:07:49.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 10 client.49496.objecter close_session for osd.6 2024-02-19T21:07:49.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 >> v1:172.21.15.193:6808/787604615 conn(0x7ffa7001f080 legacy=0x7ffa70021470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:49.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70018090 osd=6 1 2024-02-19T21:07:49.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 10 client.49496.objecter close_session for osd.7 2024-02-19T21:07:49.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 >> v1:172.21.15.193:6812/157927432 conn(0x7ffa70024b00 legacy=0x7ffa70022cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:49.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa92c07700 1 -- 172.21.15.163:0/2094650189 reap_dead start 2024-02-19T21:07:49.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa92c07700 1 -- 172.21.15.163:0/2094650189 reap_dead start 2024-02-19T21:07:49.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 20 client.49496.objecter put_session s=0x7ffa70024480 osd=7 1 2024-02-19T21:07:49.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 20 client.49496.objecter shutdown clearing up homeless session... 2024-02-19T21:07:49.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa9466a700 10 client.49496.objecter successfully canceled tick 2024-02-19T21:07:49.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa92c07700 1 -- 172.21.15.163:0/2094650189 reap_dead start 2024-02-19T21:07:49.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.324+0000 7ffa92c07700 1 -- 172.21.15.163:0/2094650189 reap_dead start 2024-02-19T21:07:49.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.325+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 >> v1:172.21.15.163:6800/1552514681 conn(0x7ffa78063cb0 legacy=0x7ffa78066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:49.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.325+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 >> v1:172.21.15.163:6790/0 conn(0x7ffa8c106d90 legacy=0x7ffa8c1005f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:49.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.325+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 shutdown_connections 2024-02-19T21:07:49.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.325+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 >> 172.21.15.163:0/2094650189 conn(0x7ffa8c06d7e0 msgr2=0x7ffa8c106630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:49.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.325+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 shutdown_connections 2024-02-19T21:07:49.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.325+0000 7ffa9466a700 1 -- 172.21.15.163:0/2094650189 wait complete. 2024-02-19T21:07:49.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:49.325+0000 7ffa9466a700 1 librados: shutdown 2024-02-19T21:07:49.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:07:49.472 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:49.485 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:48.213711+0000 mon.a (mon.0) 1993 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:49.485 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:48.218289+0000 mon.a (mon.0) 1994 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35"}]': finished 2024-02-19T21:07:49.485 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:48.218400+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e288: 8 total, 8 up, 7 in 2024-02-19T21:07:49.485 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:48.219319+0000 mon.a (mon.0) 1996 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi163-83845-41"}]: dispatch 2024-02-19T21:07:49.485 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:48.223662+0000 mon.a (mon.0) 1997 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi163-83854-51"}]: dispatch 2024-02-19T21:07:49.485 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:49.158923+0000 mon.c (mon.2) 334 : audit [DBG] from='client.? v1:172.21.15.163:0/2094650189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:49.485 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:49.212017+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi163-83845-41"}]': finished 2024-02-19T21:07:49.485 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:49.212087+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi163-83854-51"}]': finished 2024-02-19T21:07:49.485 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:49.212153+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e289: 8 total, 8 up, 7 in 2024-02-19T21:07:49.486 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:49.217584+0000 mon.a (mon.0) 2001 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi163-83854-51"}]: dispatch 2024-02-19T21:07:49.486 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:49.217926+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi163-83845-41"}]: dispatch 2024-02-19T21:07:49.499 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:49.706 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:48.213711+0000 mon.a (mon.0) 1993 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:49.706 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:48.218289+0000 mon.a (mon.0) 1994 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35"}]': finished 2024-02-19T21:07:49.706 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:48.218400+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e288: 8 total, 8 up, 7 in 2024-02-19T21:07:49.706 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:48.219319+0000 mon.a (mon.0) 1996 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi163-83845-41"}]: dispatch 2024-02-19T21:07:49.706 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:48.223662+0000 mon.a (mon.0) 1997 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi163-83854-51"}]: dispatch 2024-02-19T21:07:49.706 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:49.158923+0000 mon.c (mon.2) 334 : audit [DBG] from='client.? v1:172.21.15.163:0/2094650189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:49.706 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:49.212017+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi163-83845-41"}]': finished 2024-02-19T21:07:49.707 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:49.212087+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi163-83854-51"}]': finished 2024-02-19T21:07:49.707 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:49.212153+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e289: 8 total, 8 up, 7 in 2024-02-19T21:07:49.707 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:49.217584+0000 mon.a (mon.0) 2001 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi163-83854-51"}]: dispatch 2024-02-19T21:07:49.707 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:49.217926+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi163-83845-41"}]: dispatch 2024-02-19T21:07:49.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:48.213711+0000 mon.a (mon.0) 1993 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:49.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:48.218289+0000 mon.a (mon.0) 1994 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-35"}]': finished 2024-02-19T21:07:49.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:48.218400+0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e288: 8 total, 8 up, 7 in 2024-02-19T21:07:49.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:48.219319+0000 mon.a (mon.0) 1996 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi163-83845-41"}]: dispatch 2024-02-19T21:07:49.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:48.223662+0000 mon.a (mon.0) 1997 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi163-83854-51"}]: dispatch 2024-02-19T21:07:49.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:49.158923+0000 mon.c (mon.2) 334 : audit [DBG] from='client.? v1:172.21.15.163:0/2094650189' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:49.708 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:49.212017+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi163-83845-41"}]': finished 2024-02-19T21:07:49.708 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:49.212087+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi163-83854-51"}]': finished 2024-02-19T21:07:49.708 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:49.212153+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e289: 8 total, 8 up, 7 in 2024-02-19T21:07:49.708 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:49.217584+0000 mon.a (mon.0) 2001 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi163-83854-51"}]: dispatch 2024-02-19T21:07:49.708 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:49.217926+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi163-83845-41"}]: dispatch 2024-02-19T21:07:49.748 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:49.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:49.852 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:49.879 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:49.932 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:49.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:50.041 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:50.071 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:50.111 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:50.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:50.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:50.133 INFO:tasks.thrashosds.thrasher:j is {'epoch': 287, 'fsid': '2591809a-cf69-11ee-95bb-87774f69a715', 'created': '2024-02-19T20:55:37.794885+0000', 'modified': '2024-02-19T21:07:47.209742+0000', 'last_up_change': '2024-02-19T21:01:09.214134+0000', 'last_in_change': '2024-02-19T21:07:36.204144+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 76, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 185, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-19T20:59:09.425169+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-19T21:01:14.658945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 37, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 37, 'last_pg_merge_meta': {'source_pgid': '2.25', 'ready_epoch': 283, 'last_epoch_started': 282, 'last_epoch_clean': 282, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '283', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '283', 'last_force_op_resend_preluminous': '281', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-19T21:01:14.958800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-19T21:01:16.543816+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-19T21:01:18.521482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-19T21:01:20.586564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 16, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-19T21:04:35.333192+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 61, 'pool_name': 'test-rados-api-smithi163-83928-6', 'create_time': '2024-02-19T21:04:52.991355+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '287', 'last_force_op_resend': '287', 'last_force_op_resend_prenautilus': '287', 'last_force_op_resend_preluminous': '287', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 284, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [183], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 183, 'pool_name': 'test-rados-api-smithi163-83928-35', 'create_time': '2024-02-19T21:07:41.472149+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '287', 'last_force_op_resend': '287', 'last_force_op_resend_prenautilus': '287', 'last_force_op_resend_preluminous': '287', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 284, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 61, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 184, 'pool_name': 'MultiWrite_smithi163-83845-41', 'create_time': '2024-02-19T21:07:45.447887+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '286', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-MultiWrite_smithi163-83845-41', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 185, 'pool_name': 'RoundTripSparseReadPP_smithi163-83854-51', 'create_time': '2024-02-19T21:07:45.685442+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 2, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '286', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-RoundTripSparseReadPP_smithi163-83854-51', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '4a14b406-7c21-40f3-8b12-8cf9df811088', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 218, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6801', 'nonce': 1559418115}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6802', 'nonce': 1559418115}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6804', 'nonce': 1559418115}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6803', 'nonce': 1559418115}]}, 'public_addr': '172.21.15.163:6801/1559418115', 'cluster_addr': '172.21.15.163:6802/1559418115', 'heartbeat_back_addr': '172.21.15.163:6804/1559418115', 'heartbeat_front_addr': '172.21.15.163:6803/1559418115', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e74a65fa-cb9c-42c3-9293-602cf7e9dc67', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 286, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6805', 'nonce': 3125047432}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6806', 'nonce': 3125047432}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6808', 'nonce': 3125047432}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6807', 'nonce': 3125047432}]}, 'public_addr': '172.21.15.163:6805/3125047432', 'cluster_addr': '172.21.15.163:6806/3125047432', 'heartbeat_back_addr': '172.21.15.163:6808/3125047432', 'heartbeat_front_addr': '172.21.15.163:6807/3125047432', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6980a625-cdc9-487e-8f8c-7997ba716a97', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 286, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6809', 'nonce': 1563158487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6810', 'nonce': 1563158487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6812', 'nonce': 1563158487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6811', 'nonce': 1563158487}]}, 'public_addr': '172.21.15.163:6809/1563158487', 'cluster_addr': '172.21.15.163:6810/1563158487', 'heartbeat_back_addr': '172.21.15.163:6812/1563158487', 'heartbeat_front_addr': '172.21.15.163:6811/1563158487', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '359e2ee0-9d1d-4074-8119-2d69bc2ca3a5', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 272, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6813', 'nonce': 2112279769}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6814', 'nonce': 2112279769}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6816', 'nonce': 2112279769}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6815', 'nonce': 2112279769}]}, 'public_addr': '172.21.15.163:6813/2112279769', 'cluster_addr': '172.21.15.163:6814/2112279769', 'heartbeat_back_addr': '172.21.15.163:6816/2112279769', 'heartbeat_front_addr': '172.21.15.163:6815/2112279769', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2bf5ef05-292b-4693-85a9-97991f2091e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 286, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6800', 'nonce': 3129716965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6801', 'nonce': 3129716965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6803', 'nonce': 3129716965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6802', 'nonce': 3129716965}]}, 'public_addr': '172.21.15.193:6800/3129716965', 'cluster_addr': '172.21.15.193:6801/3129716965', 'heartbeat_back_addr': '172.21.15.193:6803/3129716965', 'heartbeat_front_addr': '172.21.15.193:6802/3129716965', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '59065f9b-9240-414b-9c91-e91cda4aed6f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 286, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6804', 'nonce': 805161991}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6805', 'nonce': 805161991}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6807', 'nonce': 805161991}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6806', 'nonce': 805161991}]}, 'public_addr': '172.21.15.193:6804/805161991', 'cluster_addr': '172.21.15.193:6805/805161991', 'heartbeat_back_addr': '172.21.15.193:6807/805161991', 'heartbeat_front_addr': '172.21.15.193:6806/805161991', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '302c95df-3842-4818-8afd-d79d8184684f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 286, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6808', 'nonce': 787604615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6809', 'nonce': 787604615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6811', 'nonce': 787604615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6810', 'nonce': 787604615}]}, 'public_addr': '172.21.15.193:6808/787604615', 'cluster_addr': '172.21.15.193:6809/787604615', 'heartbeat_back_addr': '172.21.15.193:6811/787604615', 'heartbeat_front_addr': '172.21.15.193:6810/787604615', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '9ebeff27-c2f8-4a66-923f-a0d7573ddd3f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 283, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6812', 'nonce': 157927432}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6813', 'nonce': 157927432}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6815', 'nonce': 157927432}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6814', 'nonce': 157927432}]}, 'public_addr': '172.21.15.193:6812/157927432', 'cluster_addr': '172.21.15.193:6813/157927432', 'heartbeat_back_addr': '172.21.15.193:6815/157927432', 'heartbeat_front_addr': '172.21.15.193:6814/157927432', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:07:06.492524+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:07:05.638559+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:07:30.777568+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-19T21:07:07.391088+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:07:02.613493+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:06:47.102112+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:07:00.556813+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:07:18.963426+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.1c', 'osds': [7, 0, 3]}], 'pg_upmap_items': [{'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.163:0/1427774891': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/3282102947': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:6800/2735092549': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/1235588846': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:6800/2731225015': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:0/506491651': '2024-02-20T20:56:02.185409+0000', '172.21.15.163:0/168117826': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/1458009465': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:0/541631472': '2024-02-20T20:56:02.185409+0000', '172.21.15.163:6800/2248999541': '2024-02-20T20:56:02.185409+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-MultiWrite_smithi163-83845-41': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}, 'testprofile-RoundTripSparseReadPP_smithi163-83854-51': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 11, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 25, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 57, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 61, 'snaps': [{'begin': 20, 'length': 1}]}, {'pool': 76, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 85, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 92, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 94, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 126, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 132, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 178, 'snaps': [{'begin': 20, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-19T21:07:50.133 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 5.1c 2024-02-19T21:07:50.133 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd rm-pg-upmap 5.1c 2024-02-19T21:07:50.216 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:50.229 INFO:tasks.workunit.client.0.smithi163.stdout:te 2024-02-19T21:07:50.230 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7024 ms) 2024-02-19T21:07:50.230 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-02-19T21:07:50.230 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6994 ms) 2024-02-19T21:07:50.230 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-02-19T21:07:50.230 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7004 ms) 2024-02-19T21:07:50.230 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-02-19T21:07:50.230 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6997 ms) 2024-02-19T21:07:50.230 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-02-19T21:07:50.230 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7017 ms) 2024-02-19T21:07:50.231 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-02-19T21:07:50.231 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6984 ms) 2024-02-19T21:07:50.231 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-02-19T21:07:50.231 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6994 ms) 2024-02-19T21:07:50.231 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-02-19T21:07:50.231 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7017 ms) 2024-02-19T21:07:50.231 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-02-19T21:07:50.231 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6994 ms) 2024-02-19T21:07:50.231 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-02-19T21:07:50.231 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6997 ms) 2024-02-19T21:07:50.231 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-02-19T21:07:50.231 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7011 ms) 2024-02-19T21:07:50.231 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-02-19T21:07:50.232 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7005 ms) 2024-02-19T21:07:50.232 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-02-19T21:07:50.232 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7027 ms) 2024-02-19T21:07:50.232 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-02-19T21:07:50.232 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6994 ms) 2024-02-19T21:07:50.232 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-02-19T21:07:50.232 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6990 ms) 2024-02-19T21:07:50.232 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112029 ms total) 2024-02-19T21:07:50.232 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: 2024-02-19T21:07:50.232 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [----------] Global test environment tear-down 2024-02-19T21:07:50.232 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [==========] 41 tests from 2 test suites ran. (197396 ms total) 2024-02-19T21:07:50.232 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio: [ PASSED ] 41 tests. 2024-02-19T21:07:50.251 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:50.518 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:49.171399+0000 mgr.y (mgr.14550) 289 : cluster [DBG] pgmap v404: 233 pgs: 233 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:07:50.518 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:49.493219+0000 mon.a (mon.0) 2003 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:50.518 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:50.225357+0000 mon.a (mon.0) 2004 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi163-83854-51"}]': finished 2024-02-19T21:07:50.519 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:50.225427+0000 mon.a (mon.0) 2005 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi163-83845-41"}]': finished 2024-02-19T21:07:50.519 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:50.225507+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e290: 8 total, 8 up, 7 in 2024-02-19T21:07:50.519 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:50.232624+0000 mon.b (mon.1) 450 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:50.519 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:50.233039+0000 mon.a (mon.0) 2007 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:50.519 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:50.257066+0000 mon.b (mon.1) 451 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:50.519 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:50.257626+0000 mon.a (mon.0) 2008 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:50.519 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:50.258301+0000 mon.b (mon.1) 452 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:50.519 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:50.258545+0000 mon.a (mon.0) 2009 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:50.519 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:50.258970+0000 mon.b (mon.1) 453 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:50.519 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:50.259234+0000 mon.a (mon.0) 2010 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:50.519 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:07:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:07:50.607 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:50.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:50.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:50.643 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:50.711 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:50.738 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:49.171399+0000 mgr.y (mgr.14550) 289 : cluster [DBG] pgmap v404: 233 pgs: 233 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:07:50.743 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:49.493219+0000 mon.a (mon.0) 2003 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:50.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:50.225357+0000 mon.a (mon.0) 2004 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi163-83854-51"}]': finished 2024-02-19T21:07:50.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:50.225427+0000 mon.a (mon.0) 2005 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi163-83845-41"}]': finished 2024-02-19T21:07:50.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:50.225507+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e290: 8 total, 8 up, 7 in 2024-02-19T21:07:50.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:50.232624+0000 mon.b (mon.1) 450 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:50.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:50.233039+0000 mon.a (mon.0) 2007 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:50.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:50.257066+0000 mon.b (mon.1) 451 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:50.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:50.257626+0000 mon.a (mon.0) 2008 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:50.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:50.258301+0000 mon.b (mon.1) 452 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:50.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:50.258545+0000 mon.a (mon.0) 2009 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:50.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:50.258970+0000 mon.b (mon.1) 453 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:50.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:50.259234+0000 mon.a (mon.0) 2010 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:50.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:49.171399+0000 mgr.y (mgr.14550) 289 : cluster [DBG] pgmap v404: 233 pgs: 233 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:07:50.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:49.493219+0000 mon.a (mon.0) 2003 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:50.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:50.225357+0000 mon.a (mon.0) 2004 : audit [INF] from='client.? v1:172.21.15.163:0/1372319065' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi163-83854-51"}]': finished 2024-02-19T21:07:50.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:50.225427+0000 mon.a (mon.0) 2005 : audit [INF] from='client.? v1:172.21.15.163:0/798881088' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi163-83845-41"}]': finished 2024-02-19T21:07:50.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:50.225507+0000 mon.a (mon.0) 2006 : cluster [DBG] osdmap e290: 8 total, 8 up, 7 in 2024-02-19T21:07:50.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:50.232624+0000 mon.b (mon.1) 450 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:50.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:50.233039+0000 mon.a (mon.0) 2007 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:50.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:50.257066+0000 mon.b (mon.1) 451 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:50.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:50.257626+0000 mon.a (mon.0) 2008 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:50.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:50.258301+0000 mon.b (mon.1) 452 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:50.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:50.258545+0000 mon.a (mon.0) 2009 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:50.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:50.258970+0000 mon.b (mon.1) 453 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:50.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:50.259234+0000 mon.a (mon.0) 2010 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:50.747 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:07:50.747 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:07:50.747 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-02-19T21:07:50.748 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:07:50.937 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:50.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:50.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:51.040 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:51.068 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:51.141 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:51.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:51.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:51.244 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:51.272 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:51.365 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:07:51.371 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-19T21:07:51.471 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:51.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:51.575 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:51.603 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:51.638 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:07:51.675 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:51.680 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:51.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:51.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:50.254009+0000 mgr.y (mgr.14550) 290 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:51.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:51.172284+0000 mon.b (mon.1) 454 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-19T21:07:51.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:51.173173+0000 mon.a (mon.0) 2011 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-19T21:07:51.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:51.224269+0000 mon.a (mon.0) 2012 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:51.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:51.224367+0000 mon.a (mon.0) 2013 : audit [INF] from='client.48319 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:51.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:51.224438+0000 mon.a (mon.0) 2014 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "36"}]': finished 2024-02-19T21:07:51.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:51.224494+0000 mon.a (mon.0) 2015 : cluster [DBG] osdmap e291: 8 total, 8 up, 7 in 2024-02-19T21:07:51.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:51.228192+0000 mon.b (mon.1) 455 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi163-83854-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:51.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:51.229129+0000 mon.a (mon.0) 2016 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi163-83854-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:51.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:51.233987+0000 mon.b (mon.1) 456 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:51.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:51.235412+0000 mon.a (mon.0) 2017 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:50.254009+0000 mgr.y (mgr.14550) 290 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:51.172284+0000 mon.b (mon.1) 454 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-19T21:07:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:51.173173+0000 mon.a (mon.0) 2011 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-19T21:07:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:51.224269+0000 mon.a (mon.0) 2012 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:51.224367+0000 mon.a (mon.0) 2013 : audit [INF] from='client.48319 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:51.224438+0000 mon.a (mon.0) 2014 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "36"}]': finished 2024-02-19T21:07:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:51.224494+0000 mon.a (mon.0) 2015 : cluster [DBG] osdmap e291: 8 total, 8 up, 7 in 2024-02-19T21:07:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:51.228192+0000 mon.b (mon.1) 455 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi163-83854-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:51.229129+0000 mon.a (mon.0) 2016 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi163-83854-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:51.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:51.233987+0000 mon.b (mon.1) 456 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:51.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:51.235412+0000 mon.a (mon.0) 2017 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:51.781 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:51.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:50.254009+0000 mgr.y (mgr.14550) 290 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:07:51.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:51.172284+0000 mon.b (mon.1) 454 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-19T21:07:51.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:51.173173+0000 mon.a (mon.0) 2011 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-02-19T21:07:51.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:51.224269+0000 mon.a (mon.0) 2012 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:07:51.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:51.224367+0000 mon.a (mon.0) 2013 : audit [INF] from='client.48319 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:51.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:51.224438+0000 mon.a (mon.0) 2014 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "36"}]': finished 2024-02-19T21:07:51.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:51.224494+0000 mon.a (mon.0) 2015 : cluster [DBG] osdmap e291: 8 total, 8 up, 7 in 2024-02-19T21:07:51.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:51.228192+0000 mon.b (mon.1) 455 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi163-83854-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:51.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:51.229129+0000 mon.a (mon.0) 2016 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi163-83854-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:51.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:51.233987+0000 mon.b (mon.1) 456 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:51.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:51.235412+0000 mon.a (mon.0) 2017 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:07:51.807 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:51.905 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:51.907 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:51.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:52.009 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:07:52.035 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:52.087 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:07:52.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:52.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:07:52.193 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:52.221 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:52.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.432+0000 7f436dd1d700 1 -- 172.21.15.163:0/4008033753 >> v1:172.21.15.193:6789/0 conn(0x7f43680fa430 legacy=0x7f43680fa8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:52.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.432+0000 7f436dd1d700 1 -- 172.21.15.163:0/4008033753 shutdown_connections 2024-02-19T21:07:52.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.432+0000 7f436dd1d700 1 -- 172.21.15.163:0/4008033753 >> 172.21.15.163:0/4008033753 conn(0x7f436806fdc0 msgr2=0x7f43680f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:52.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.436+0000 7f436dd1d700 1 -- 172.21.15.163:0/4008033753 shutdown_connections 2024-02-19T21:07:52.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.436+0000 7f436dd1d700 1 -- 172.21.15.163:0/4008033753 wait complete. 2024-02-19T21:07:52.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.437+0000 7f436dd1d700 1 librados: starting msgr at 2024-02-19T21:07:52.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.437+0000 7f436dd1d700 1 librados: starting objecter 2024-02-19T21:07:52.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.437+0000 7f436dd1d700 1 Processor -- start 2024-02-19T21:07:52.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.437+0000 7f436dd1d700 1 -- start start 2024-02-19T21:07:52.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.437+0000 7f436dd1d700 1 librados: setting wanted keys 2024-02-19T21:07:52.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.437+0000 7f436dd1d700 1 librados: calling monclient init 2024-02-19T21:07:52.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f436dd1d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4368106570 con 0x7f43680fc780 2024-02-19T21:07:52.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f436dd1d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4368106740 con 0x7f43680faff0 2024-02-19T21:07:52.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f436dd1d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f43681395f0 con 0x7f43680fa430 2024-02-19T21:07:52.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f4367fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f43680fc780 0x7f4368135eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33128/0 (socket says 172.21.15.163:33128) 2024-02-19T21:07:52.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f4367fff700 1 -- 172.21.15.163:0/1671602801 learned_addr learned my addr 172.21.15.163:0/1671602801 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:52.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f4364ff9700 10 client.?.objecter ms_handle_connect 0x7f43680fc780 2024-02-19T21:07:52.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f4364ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:52.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f4364ff9700 10 client.?.objecter ms_handle_connect 0x7f43680fa430 2024-02-19T21:07:52.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f4364ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:52.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f4364ff9700 10 client.?.objecter ms_handle_connect 0x7f43680faff0 2024-02-19T21:07:52.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f4364ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:52.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878658009 0 0) 0x7f43681395f0 con 0x7f43680fa430 2024-02-19T21:07:52.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4350002220 con 0x7f43680fa430 2024-02-19T21:07:52.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024442480 0 0) 0x7f4368106740 con 0x7f43680faff0 2024-02-19T21:07:52.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43681395f0 con 0x7f43680faff0 2024-02-19T21:07:52.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 855286702 0 0) 0x7f4350002220 con 0x7f43680fa430 2024-02-19T21:07:52.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4368106740 con 0x7f43680fa430 2024-02-19T21:07:52.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.438+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f435c0018c0 con 0x7f43680fa430 2024-02-19T21:07:52.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.439+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 933970165 0 0) 0x7f43681395f0 con 0x7f43680faff0 2024-02-19T21:07:52.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.439+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4350002220 con 0x7f43680faff0 2024-02-19T21:07:52.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.439+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4358001e30 con 0x7f43680faff0 2024-02-19T21:07:52.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.439+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2018162122 0 0) 0x7f4368106740 con 0x7f43680fa430 2024-02-19T21:07:52.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.439+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 >> v1:172.21.15.193:6789/0 conn(0x7f43680faff0 legacy=0x7f436810ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:52.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.439+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 >> v1:172.21.15.163:6789/0 conn(0x7f43680fc780 legacy=0x7f4368135eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:52.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.439+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436813a790 con 0x7f43680fa430 2024-02-19T21:07:52.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.439+0000 7f436dd1d700 1 -- 172.21.15.163:0/1671602801 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43681397e0 con 0x7f43680fa430 2024-02-19T21:07:52.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.439+0000 7f436dd1d700 10 client.49508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:52.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.439+0000 7f436dd1d700 1 -- 172.21.15.163:0/1671602801 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4368139c50 con 0x7f43680fa430 2024-02-19T21:07:52.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.439+0000 7f436dd1d700 1 librados: init done 2024-02-19T21:07:52.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.440+0000 7f436dd1d700 1 -- 172.21.15.163:0/1671602801 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43681027f0 con 0x7f43680fa430 2024-02-19T21:07:52.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.440+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f435c001ec0 con 0x7f43680fa430 2024-02-19T21:07:52.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.440+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f435c003310 con 0x7f43680fa430 2024-02-19T21:07:52.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.441+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f435c016220 con 0x7f43680fa430 2024-02-19T21:07:52.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.441+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(292..292 src has 43..292) v4 ==== 8377+0+0 (unknown 1645934877 0 0) 0x7f435c0775d0 con 0x7f43680fa430 2024-02-19T21:07:52.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.441+0000 7f4364ff9700 10 client.49508.objecter ms_dispatch 0x7f4368001550 osd_map(292..292 src has 43..292) v4 2024-02-19T21:07:52.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.441+0000 7f4364ff9700 3 client.49508.objecter handle_osd_map got epochs [292,292] > 0 2024-02-19T21:07:52.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.441+0000 7f4364ff9700 3 client.49508.objecter handle_osd_map decoding full epoch 292 2024-02-19T21:07:52.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.441+0000 7f4364ff9700 20 client.49508.objecter dump_active .. 0 homeless 2024-02-19T21:07:52.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.442+0000 7f4364ff9700 10 client.49508.objecter ms_handle_connect 0x7f4350063c20 2024-02-19T21:07:52.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.447+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f435c04a900 con 0x7f43680fa430 2024-02-19T21:07:52.454 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:52.470 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:52.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:52.575 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:52.586 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:07:52.607 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:52.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:52.652+0000 7f436dd1d700 1 -- 172.21.15.163:0/1671602801 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd rm-pg-upmap", "pgid": "5.1c"} v 0) v1 -- 0x7f43680f7040 con 0x7f43680fa430 2024-02-19T21:07:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:51.171845+0000 mgr.y (mgr.14550) 291 : cluster [DBG] pgmap v407: 233 pgs: 32 unknown, 201 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-19T21:07:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:52.468855+0000 mon.a (mon.0) 2018 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:52.468943+0000 mon.a (mon.0) 2019 : cluster [DBG] osdmap e292: 8 total, 8 up, 7 in 2024-02-19T21:07:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:52.469658+0000 mon.b (mon.1) 457 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-37"}]: dispatch 2024-02-19T21:07:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:52.469967+0000 mon.a (mon.0) 2020 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-37"}]: dispatch 2024-02-19T21:07:52.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:51.171845+0000 mgr.y (mgr.14550) 291 : cluster [DBG] pgmap v407: 233 pgs: 32 unknown, 201 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-19T21:07:52.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:52.468855+0000 mon.a (mon.0) 2018 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:52.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:52.468943+0000 mon.a (mon.0) 2019 : cluster [DBG] osdmap e292: 8 total, 8 up, 7 in 2024-02-19T21:07:52.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:52.469658+0000 mon.b (mon.1) 457 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-37"}]: dispatch 2024-02-19T21:07:52.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:52.469967+0000 mon.a (mon.0) 2020 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-37"}]: dispatch 2024-02-19T21:07:52.760 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:52.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:52.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:52.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:51.171845+0000 mgr.y (mgr.14550) 291 : cluster [DBG] pgmap v407: 233 pgs: 32 unknown, 201 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-19T21:07:52.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:52.468855+0000 mon.a (mon.0) 2018 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:07:52.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:52.468943+0000 mon.a (mon.0) 2019 : cluster [DBG] osdmap e292: 8 total, 8 up, 7 in 2024-02-19T21:07:52.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:52.469658+0000 mon.b (mon.1) 457 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-37"}]: dispatch 2024-02-19T21:07:52.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:52.469967+0000 mon.a (mon.0) 2020 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-37"}]: dispatch 2024-02-19T21:07:52.865 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:52.893 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:52.945 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:52.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:52.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:53.049 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:07:53.076 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:53.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:53.226+0000 7f4364ff9700 1 -- 172.21.15.163:0/1671602801 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap", "pgid": "5.1c"}]=0 clear 5.1c pg_upmap mapping v293) v1 ==== 106+0+0 (unknown 3326954377 0 0) 0x7f435c048d10 con 0x7f43680fa430 2024-02-19T21:07:53.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:53.229+0000 7f436dd1d700 10 librados: watch_flush enter 2024-02-19T21:07:53.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:53.230+0000 7f436dd1d700 10 librados: watch_flush exit 2024-02-19T21:07:53.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:53.230+0000 7f436dd1d700 20 client.49508.objecter shutdown clearing up homeless session... 2024-02-19T21:07:53.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:53.230+0000 7f436dd1d700 10 client.49508.objecter successfully canceled tick 2024-02-19T21:07:53.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:53.230+0000 7f436dd1d700 1 -- 172.21.15.163:0/1671602801 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4350063c20 legacy=0x7f43500660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:53.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:53.230+0000 7f436dd1d700 1 -- 172.21.15.163:0/1671602801 >> v1:172.21.15.163:6790/0 conn(0x7f43680fa430 legacy=0x7f4368105c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:53.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:53.231+0000 7f436dd1d700 1 -- 172.21.15.163:0/1671602801 shutdown_connections 2024-02-19T21:07:53.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:53.231+0000 7f436dd1d700 1 -- 172.21.15.163:0/1671602801 >> 172.21.15.163:0/1671602801 conn(0x7f436806fdc0 msgr2=0x7f436810dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:53.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:53.232+0000 7f436dd1d700 1 -- 172.21.15.163:0/1671602801 shutdown_connections 2024-02-19T21:07:53.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:53.233+0000 7f436dd1d700 1 -- 172.21.15.163:0/1671602801 wait complete. 2024-02-19T21:07:53.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:53.233+0000 7f436dd1d700 1 librados: shutdown 2024-02-19T21:07:53.230 INFO:teuthology.orchestra.run.smithi163.stderr:clear 5.1c pg_upmap mapping 2024-02-19T21:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-02-19T21:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-02-19T21:07:53.475 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:07:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:52.653615+0000 mon.c (mon.2) 335 : audit [INF] from='client.? v1:172.21.15.163:0/1671602801' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.1c"}]: dispatch 2024-02-19T21:07:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:52.654032+0000 mon.a (mon.0) 2021 : audit [INF] from='client.49508 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.1c"}]: dispatch 2024-02-19T21:07:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:53.225689+0000 mon.a (mon.0) 2022 : audit [INF] from='client.48319 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi163-83854-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi163-83854-52"}]': finished 2024-02-19T21:07:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:53.225791+0000 mon.a (mon.0) 2023 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-37"}]': finished 2024-02-19T21:07:53.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:53.225894+0000 mon.a (mon.0) 2024 : audit [INF] from='client.49508 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.1c"}]': finished 2024-02-19T21:07:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:53.225960+0000 mon.a (mon.0) 2025 : cluster [DBG] osdmap e293: 8 total, 8 up, 7 in 2024-02-19T21:07:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:53.227555+0000 mon.b (mon.1) 458 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-37", "mode": "writeback"}]: dispatch 2024-02-19T21:07:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:53.228786+0000 mon.a (mon.0) 2026 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-37", "mode": "writeback"}]: dispatch 2024-02-19T21:07:53.745 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:07:53] "GET /metrics HTTP/1.1" 200 188035 "" "Prometheus/2.33.4" 2024-02-19T21:07:53.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:52.653615+0000 mon.c (mon.2) 335 : audit [INF] from='client.? v1:172.21.15.163:0/1671602801' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.1c"}]: dispatch 2024-02-19T21:07:53.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:52.654032+0000 mon.a (mon.0) 2021 : audit [INF] from='client.49508 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.1c"}]: dispatch 2024-02-19T21:07:53.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:53.225689+0000 mon.a (mon.0) 2022 : audit [INF] from='client.48319 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi163-83854-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi163-83854-52"}]': finished 2024-02-19T21:07:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:53.225791+0000 mon.a (mon.0) 2023 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-37"}]': finished 2024-02-19T21:07:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:53.225894+0000 mon.a (mon.0) 2024 : audit [INF] from='client.49508 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.1c"}]': finished 2024-02-19T21:07:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:53.225960+0000 mon.a (mon.0) 2025 : cluster [DBG] osdmap e293: 8 total, 8 up, 7 in 2024-02-19T21:07:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:53.227555+0000 mon.b (mon.1) 458 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-37", "mode": "writeback"}]: dispatch 2024-02-19T21:07:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:53.228786+0000 mon.a (mon.0) 2026 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-37", "mode": "writeback"}]: dispatch 2024-02-19T21:07:53.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:52.653615+0000 mon.c (mon.2) 335 : audit [INF] from='client.? v1:172.21.15.163:0/1671602801' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.1c"}]: dispatch 2024-02-19T21:07:53.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:52.654032+0000 mon.a (mon.0) 2021 : audit [INF] from='client.49508 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "5.1c"}]: dispatch 2024-02-19T21:07:53.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:53.225689+0000 mon.a (mon.0) 2022 : audit [INF] from='client.48319 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi163-83854-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi163-83854-52"}]': finished 2024-02-19T21:07:53.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:53.225791+0000 mon.a (mon.0) 2023 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-37"}]': finished 2024-02-19T21:07:53.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:53.225894+0000 mon.a (mon.0) 2024 : audit [INF] from='client.49508 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "5.1c"}]': finished 2024-02-19T21:07:53.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:53.225960+0000 mon.a (mon.0) 2025 : cluster [DBG] osdmap e293: 8 total, 8 up, 7 in 2024-02-19T21:07:53.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:53.227555+0000 mon.b (mon.1) 458 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-37", "mode": "writeback"}]: dispatch 2024-02-19T21:07:53.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:53.228786+0000 mon.a (mon.0) 2026 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-37", "mode": "writeback"}]: dispatch 2024-02-19T21:07:54.077 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-19T21:07:54.355 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:07:54.429 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:07:54.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:54.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:07:54.534 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:54.562 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:54.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:53.172326+0000 mgr.y (mgr.14550) 292 : cluster [DBG] pgmap v410: 233 pgs: 32 unknown, 201 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-19T21:07:54.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:54.100453+0000 mon.a (mon.0) 2027 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:54.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:54.208834+0000 mon.a (mon.0) 2028 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:54.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:54.216341+0000 mon.a (mon.0) 2029 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-37", "mode": "writeback"}]': finished 2024-02-19T21:07:54.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:54.216465+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e294: 8 total, 8 up, 7 in 2024-02-19T21:07:54.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:54.299521+0000 mon.b (mon.1) 459 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:54.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:54.299880+0000 mon.a (mon.0) 2031 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:54.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:54.500965+0000 mon.a (mon.0) 2032 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:54.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:53.172326+0000 mgr.y (mgr.14550) 292 : cluster [DBG] pgmap v410: 233 pgs: 32 unknown, 201 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-19T21:07:54.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:54.100453+0000 mon.a (mon.0) 2027 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:54.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:54.208834+0000 mon.a (mon.0) 2028 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:54.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:54.216341+0000 mon.a (mon.0) 2029 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-37", "mode": "writeback"}]': finished 2024-02-19T21:07:54.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:54.216465+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e294: 8 total, 8 up, 7 in 2024-02-19T21:07:54.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:54.299521+0000 mon.b (mon.1) 459 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:54.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:54.299880+0000 mon.a (mon.0) 2031 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:54.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:54.500965+0000 mon.a (mon.0) 2032 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:53.172326+0000 mgr.y (mgr.14550) 292 : cluster [DBG] pgmap v410: 233 pgs: 32 unknown, 201 active+clean; 6.8 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-19T21:07:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:54.100453+0000 mon.a (mon.0) 2027 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:54.208834+0000 mon.a (mon.0) 2028 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:07:54.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:54.216341+0000 mon.a (mon.0) 2029 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-37", "mode": "writeback"}]': finished 2024-02-19T21:07:54.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:54.216465+0000 mon.a (mon.0) 2030 : cluster [DBG] osdmap e294: 8 total, 8 up, 7 in 2024-02-19T21:07:54.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:54.299521+0000 mon.b (mon.1) 459 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:54.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:54.299880+0000 mon.a (mon.0) 2031 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:07:54.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:54.500965+0000 mon.a (mon.0) 2032 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:07:55.740 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:55.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:55.754 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:07:55.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:55.174106+0000 mon.b (mon.1) 460 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "35"}]: dispatch 2024-02-19T21:07:55.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:55.175032+0000 mon.a (mon.0) 2033 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "35"}]: dispatch 2024-02-19T21:07:55.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:55.219849+0000 mon.a (mon.0) 2034 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:55.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:55.219927+0000 mon.a (mon.0) 2035 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "35"}]': finished 2024-02-19T21:07:55.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:55.219991+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e295: 8 total, 8 up, 7 in 2024-02-19T21:07:55.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:55.220802+0000 mon.b (mon.1) 461 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37"}]: dispatch 2024-02-19T21:07:55.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:55.221064+0000 mon.b (mon.1) 462 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:55.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:55.221580+0000 mon.a (mon.0) 2037 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37"}]: dispatch 2024-02-19T21:07:55.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:55.221867+0000 mon.a (mon.0) 2038 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:55.844 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:55.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:55.174106+0000 mon.b (mon.1) 460 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "35"}]: dispatch 2024-02-19T21:07:55.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:55.175032+0000 mon.a (mon.0) 2033 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "35"}]: dispatch 2024-02-19T21:07:55.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:55.219849+0000 mon.a (mon.0) 2034 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:55.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:55.219927+0000 mon.a (mon.0) 2035 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "35"}]': finished 2024-02-19T21:07:55.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:55.219991+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e295: 8 total, 8 up, 7 in 2024-02-19T21:07:55.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:55.220802+0000 mon.b (mon.1) 461 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37"}]: dispatch 2024-02-19T21:07:55.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:55.221064+0000 mon.b (mon.1) 462 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:55.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:55.221580+0000 mon.a (mon.0) 2037 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37"}]: dispatch 2024-02-19T21:07:55.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:55.221867+0000 mon.a (mon.0) 2038 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:55.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:55.174106+0000 mon.b (mon.1) 460 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "35"}]: dispatch 2024-02-19T21:07:55.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:55.175032+0000 mon.a (mon.0) 2033 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "35"}]: dispatch 2024-02-19T21:07:55.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:55.219849+0000 mon.a (mon.0) 2034 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:07:55.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:55.219927+0000 mon.a (mon.0) 2035 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "35"}]': finished 2024-02-19T21:07:55.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:55.219991+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e295: 8 total, 8 up, 7 in 2024-02-19T21:07:55.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:55.220802+0000 mon.b (mon.1) 461 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37"}]: dispatch 2024-02-19T21:07:55.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:55.221064+0000 mon.b (mon.1) 462 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:55.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:55.221580+0000 mon.a (mon.0) 2037 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37"}]: dispatch 2024-02-19T21:07:55.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:55.221867+0000 mon.a (mon.0) 2038 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:55.873 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:56.021 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:56.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:56.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:56.126 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:07:56.161 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:56.183 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:07:56.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:56.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:07:56.287 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:07:56.317 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:56.516 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:07:56.519 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:56.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:07:56.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:55.173390+0000 mgr.y (mgr.14550) 293 : cluster [DBG] pgmap v413: 240 pgs: 240 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 4.2 KiB/s rd, 1.9 MiB/s wr, 72 op/s 2024-02-19T21:07:56.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:55.509639+0000 mon.a (mon.0) 2039 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:07:56.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:56.212003+0000 mon.a (mon.0) 2040 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:56.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:56.216849+0000 mon.a (mon.0) 2041 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37"}]': finished 2024-02-19T21:07:56.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:56.216939+0000 mon.a (mon.0) 2042 : audit [INF] from='client.48319 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52"}]': finished 2024-02-19T21:07:56.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:56.216983+0000 mon.a (mon.0) 2043 : cluster [DBG] osdmap e296: 8 total, 8 up, 7 in 2024-02-19T21:07:56.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:56.218923+0000 mon.b (mon.1) 463 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:56.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:56.219245+0000 mon.a (mon.0) 2044 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:56.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:55.173390+0000 mgr.y (mgr.14550) 293 : cluster [DBG] pgmap v413: 240 pgs: 240 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 4.2 KiB/s rd, 1.9 MiB/s wr, 72 op/s 2024-02-19T21:07:56.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:55.509639+0000 mon.a (mon.0) 2039 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:07:56.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:56.212003+0000 mon.a (mon.0) 2040 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:56.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:56.216849+0000 mon.a (mon.0) 2041 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37"}]': finished 2024-02-19T21:07:56.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:56.216939+0000 mon.a (mon.0) 2042 : audit [INF] from='client.48319 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52"}]': finished 2024-02-19T21:07:56.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:56.216983+0000 mon.a (mon.0) 2043 : cluster [DBG] osdmap e296: 8 total, 8 up, 7 in 2024-02-19T21:07:56.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:56.218923+0000 mon.b (mon.1) 463 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:56.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:56.219245+0000 mon.a (mon.0) 2044 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:56.620 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:07:56.646 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:56.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:55.173390+0000 mgr.y (mgr.14550) 293 : cluster [DBG] pgmap v413: 240 pgs: 240 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 4.2 KiB/s rd, 1.9 MiB/s wr, 72 op/s 2024-02-19T21:07:56.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:55.509639+0000 mon.a (mon.0) 2039 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:07:56.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:56.212003+0000 mon.a (mon.0) 2040 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:07:56.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:56.216849+0000 mon.a (mon.0) 2041 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-37"}]': finished 2024-02-19T21:07:56.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:56.216939+0000 mon.a (mon.0) 2042 : audit [INF] from='client.48319 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi163-83854-52"}]': finished 2024-02-19T21:07:56.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:56.216983+0000 mon.a (mon.0) 2043 : cluster [DBG] osdmap e296: 8 total, 8 up, 7 in 2024-02-19T21:07:56.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:56.218923+0000 mon.b (mon.1) 463 : audit [INF] from='client.? v1:172.21.15.163:0/1695432171' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:56.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:56.219245+0000 mon.a (mon.0) 2044 : audit [INF] from='client.48319 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi163-83854-52"}]: dispatch 2024-02-19T21:07:57.167 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:07:57.168 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-02-19T21:07:57.168 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-02-19T21:07:57.168 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-02-19T21:07:57.168 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.168 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34382.0:53 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-19T21:07:57.168 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:01:20.279139+0000", 2024-02-19T21:07:57.168 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 396.89075848200002, 2024-02-19T21:07:57.168 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.016593475, 2024-02-19T21:07:57.168 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.168 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.169 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:57.169 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34382", 2024-02-19T21:07:57.169 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2202902387", 2024-02-19T21:07:57.169 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 53 2024-02-19T21:07:57.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.169 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.169 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.169 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.169 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279139+0000", 2024-02-19T21:07:57.169 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.169 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.169 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.169 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279139+0000", 2024-02-19T21:07:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:07:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279136+0000", 2024-02-19T21:07:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4159999999999993e-06 2024-02-19T21:07:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279145+0000", 2024-02-19T21:07:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5599000000000001e-05 2024-02-19T21:07:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279161+0000", 2024-02-19T21:07:57.171 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3079999999999997e-06 2024-02-19T21:07:57.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.171 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.171 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.171 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279170+0000", 2024-02-19T21:07:57.171 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9360000000000001e-05 2024-02-19T21:07:57.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.171 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.171 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.171 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279230+0000", 2024-02-19T21:07:57.171 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4889999999999998e-06 2024-02-19T21:07:57.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.171 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.171 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:07:57.172 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.279237+0000", 2024-02-19T21:07:57.172 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003363923 2024-02-19T21:07:57.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.172 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.172 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.172 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.282601+0000", 2024-02-19T21:07:57.172 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2699999999999993e-06 2024-02-19T21:07:57.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.172 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.172 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:07:57.172 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.282610+0000", 2024-02-19T21:07:57.172 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0088474929999999997 2024-02-19T21:07:57.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.173 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.173 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.173 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.291458+0000", 2024-02-19T21:07:57.173 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9685999999999997e-05 2024-02-19T21:07:57.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.173 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.173 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.173 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.291528+0000", 2024-02-19T21:07:57.173 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015815 2024-02-19T21:07:57.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.173 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.173 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 4,7", 2024-02-19T21:07:57.173 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.291686+0000", 2024-02-19T21:07:57.174 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0032840980000000001 2024-02-19T21:07:57.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.174 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.174 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:57.174 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.294970+0000", 2024-02-19T21:07:57.174 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0307000000000001e-05 2024-02-19T21:07:57.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.174 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.174 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:07:57.174 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.294990+0000", 2024-02-19T21:07:57.174 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00066387399999999995 2024-02-19T21:07:57.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.174 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.174 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:57.174 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.295654+0000", 2024-02-19T21:07:57.175 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1844000000000001e-05 2024-02-19T21:07:57.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.175 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.175 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.175 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.295696+0000", 2024-02-19T21:07:57.175 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6424000000000002e-05 2024-02-19T21:07:57.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.175 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.175 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.175 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.295732+0000", 2024-02-19T21:07:57.175 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.016593475 2024-02-19T21:07:57.175 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.175 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.175 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.176 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.176 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:91 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected e62)", 2024-02-19T21:07:57.176 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:01:20.281408+0000", 2024-02-19T21:07:57.176 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 396.88848938500001, 2024-02-19T21:07:57.176 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014198916000000001, 2024-02-19T21:07:57.176 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.176 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.176 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:57.176 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:07:57.176 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:07:57.176 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 91 2024-02-19T21:07:57.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.176 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.176 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.177 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.177 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.281408+0000", 2024-02-19T21:07:57.177 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.177 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.177 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.177 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.281408+0000", 2024-02-19T21:07:57.177 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:07:57.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.177 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.177 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.177 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.281405+0000", 2024-02-19T21:07:57.177 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7749999999999993e-06 2024-02-19T21:07:57.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.178 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.178 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.178 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.281413+0000", 2024-02-19T21:07:57.178 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1241e-05 2024-02-19T21:07:57.178 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.178 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.178 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.178 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.281424+0000", 2024-02-19T21:07:57.178 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5439999999999999e-06 2024-02-19T21:07:57.178 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.178 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.178 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.178 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.281433+0000", 2024-02-19T21:07:57.178 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001918274 2024-02-19T21:07:57.178 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.179 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.179 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.179 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.283351+0000", 2024-02-19T21:07:57.179 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5009999999999998e-06 2024-02-19T21:07:57.179 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.179 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.179 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:07:57.179 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.283359+0000", 2024-02-19T21:07:57.179 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0075975360000000002 2024-02-19T21:07:57.179 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.179 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.179 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.179 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.290956+0000", 2024-02-19T21:07:57.179 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000121327 2024-02-19T21:07:57.179 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.180 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.180 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.291078+0000", 2024-02-19T21:07:57.180 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000152733 2024-02-19T21:07:57.180 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.180 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 4,7", 2024-02-19T21:07:57.180 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.291230+0000", 2024-02-19T21:07:57.180 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0036645950000000001 2024-02-19T21:07:57.180 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.180 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:57.180 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.294895+0000", 2024-02-19T21:07:57.180 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0357000000000002e-05 2024-02-19T21:07:57.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.181 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:07:57.181 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.294985+0000", 2024-02-19T21:07:57.181 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00054387100000000002 2024-02-19T21:07:57.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.181 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:57.181 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.295529+0000", 2024-02-19T21:07:57.181 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5608000000000001e-05 2024-02-19T21:07:57.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.181 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.181 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.295575+0000", 2024-02-19T21:07:57.182 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1906000000000003e-05 2024-02-19T21:07:57.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.182 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.182 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.182 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:20.295607+0000", 2024-02-19T21:07:57.182 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014198916000000001 2024-02-19T21:07:57.182 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.182 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.182 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.182 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.182 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.3.0:299 11.8 e113/99 11:1381d4a2:test-rados-api-smithi163-83952-6::foo:9 v 113'6, mlcod=111'3)", 2024-02-19T21:07:57.182 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:52.583933+0000", 2024-02-19T21:07:57.182 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 184.58596400799999, 2024-02-19T21:07:57.182 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0114605, 2024-02-19T21:07:57.183 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.183 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.183 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.183 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.183 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.183 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583933+0000", 2024-02-19T21:07:57.183 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.183 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.183 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.183 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.183 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583933+0000", 2024-02-19T21:07:57.183 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:07:57.183 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.183 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.184 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.184 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583933+0000", 2024-02-19T21:07:57.184 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9749999999999998e-06 2024-02-19T21:07:57.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.184 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.184 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.184 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583938+0000", 2024-02-19T21:07:57.184 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2570000000000004e-06 2024-02-19T21:07:57.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.184 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.184 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.184 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583944+0000", 2024-02-19T21:07:57.184 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.657e-06 2024-02-19T21:07:57.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.185 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.185 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.185 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583951+0000", 2024-02-19T21:07:57.185 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00042205900000000002 2024-02-19T21:07:57.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.185 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.185 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.185 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.584373+0000", 2024-02-19T21:07:57.185 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3680999999999999e-05 2024-02-19T21:07:57.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.185 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.185 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.185 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.584396+0000", 2024-02-19T21:07:57.185 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010962929 2024-02-19T21:07:57.186 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.186 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.186 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.186 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.595359+0000", 2024-02-19T21:07:57.186 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4283999999999999e-05 2024-02-19T21:07:57.186 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.186 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.186 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.186 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.595394+0000", 2024-02-19T21:07:57.186 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0114605 2024-02-19T21:07:57.186 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.186 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.186 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.186 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.187 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35237.0:8 54.0 54:2698d9fa:test-rados-api-smithi163-83984-7::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e113)", 2024-02-19T21:07:57.187 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:52.638465+0000", 2024-02-19T21:07:57.187 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 184.53143243299999, 2024-02-19T21:07:57.187 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0094088230000000002, 2024-02-19T21:07:57.187 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.187 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.187 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:57.187 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35237", 2024-02-19T21:07:57.187 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1081998331", 2024-02-19T21:07:57.187 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 8 2024-02-19T21:07:57.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.187 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.188 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.188 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638465+0000", 2024-02-19T21:07:57.188 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.188 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.188 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.188 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638465+0000", 2024-02-19T21:07:57.188 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:07:57.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.188 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.188 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.188 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638463+0000", 2024-02-19T21:07:57.188 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9859999999999994e-06 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638471+0000", 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8330000000000006e-06 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638481+0000", 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2889999999999997e-06 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638486+0000", 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7515999999999998e-05 2024-02-19T21:07:57.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.190 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.190 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638523+0000", 2024-02-19T21:07:57.190 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000171239 2024-02-19T21:07:57.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.190 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:07:57.190 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.638695+0000", 2024-02-19T21:07:57.190 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00090101799999999996 2024-02-19T21:07:57.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.190 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.190 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.639596+0000", 2024-02-19T21:07:57.190 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3301999999999996e-05 2024-02-19T21:07:57.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.191 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.191 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.639679+0000", 2024-02-19T21:07:57.191 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013513300000000001 2024-02-19T21:07:57.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.191 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,4", 2024-02-19T21:07:57.191 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.639814+0000", 2024-02-19T21:07:57.191 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00034915400000000001 2024-02-19T21:07:57.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.191 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:07:57.191 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.640163+0000", 2024-02-19T21:07:57.191 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014084300000000001 2024-02-19T21:07:57.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.192 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:57.192 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.640304+0000", 2024-02-19T21:07:57.192 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0075044070000000003 2024-02-19T21:07:57.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.192 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:07:57.192 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.647809+0000", 2024-02-19T21:07:57.192 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0991000000000002e-05 2024-02-19T21:07:57.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.192 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.193 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.647840+0000", 2024-02-19T21:07:57.193 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4032000000000002e-05 2024-02-19T21:07:57.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.193 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.193 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.193 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.647874+0000", 2024-02-19T21:07:57.193 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0094088230000000002 2024-02-19T21:07:57.193 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.193 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.193 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.193 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.193 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "replica_scrub(pg: 4.10,from:0'0,to:65'4,epoch:127/59,start:4:08000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-02-19T21:07:57.194 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:08.142968+0000", 2024-02-19T21:07:57.194 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 169.026929698, 2024-02-19T21:07:57.194 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010775658, 2024-02-19T21:07:57.194 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.194 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:07:57.194 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.194 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.194 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.194 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.142968+0000", 2024-02-19T21:07:57.194 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.194 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.194 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.194 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.142968+0000", 2024-02-19T21:07:57.195 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:07:57.195 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.195 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.195 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.195 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.142966+0000", 2024-02-19T21:07:57.195 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2630000000000003e-06 2024-02-19T21:07:57.195 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.195 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.195 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.195 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.142974+0000", 2024-02-19T21:07:57.195 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0028049339999999998 2024-02-19T21:07:57.195 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.195 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.195 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.196 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.145779+0000", 2024-02-19T21:07:57.196 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3135999999999999e-05 2024-02-19T21:07:57.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.196 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.196 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.196 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.145792+0000", 2024-02-19T21:07:57.196 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0078870490000000001 2024-02-19T21:07:57.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.196 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.196 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.196 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.153679+0000", 2024-02-19T21:07:57.196 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5184000000000001e-05 2024-02-19T21:07:57.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.197 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.197 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.197 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.153695+0000", 2024-02-19T21:07:57.197 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8640999999999997e-05 2024-02-19T21:07:57.197 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.197 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.197 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.197 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.153743+0000", 2024-02-19T21:07:57.197 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010775658 2024-02-19T21:07:57.197 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.197 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.197 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.197 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.197 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.198 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.1.0:241 28.15 e132/99 -30:ad087405:::temp_28.15_0_14259_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:07:57.198 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.401838+0000", 2024-02-19T21:07:57.198 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 164.76805882799999, 2024-02-19T21:07:57.198 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025390421, 2024-02-19T21:07:57.198 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.198 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.198 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.198 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.198 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.198 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.401838+0000", 2024-02-19T21:07:57.198 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.198 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.198 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.198 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.199 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.401838+0000", 2024-02-19T21:07:57.199 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:07:57.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.199 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.199 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.401837+0000", 2024-02-19T21:07:57.199 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.016854948000000002 2024-02-19T21:07:57.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.199 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.199 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.418692+0000", 2024-02-19T21:07:57.199 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8876e-05 2024-02-19T21:07:57.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.199 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.200 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.418710+0000", 2024-02-19T21:07:57.200 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4498000000000002e-05 2024-02-19T21:07:57.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.200 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.200 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.200 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.418735+0000", 2024-02-19T21:07:57.200 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7998000000000003e-05 2024-02-19T21:07:57.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.200 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.200 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.200 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.418793+0000", 2024-02-19T21:07:57.200 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6917e-05 2024-02-19T21:07:57.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.201 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.201 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.201 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.418820+0000", 2024-02-19T21:07:57.201 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0083708069999999992 2024-02-19T21:07:57.201 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.201 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.201 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.201 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.427191+0000", 2024-02-19T21:07:57.201 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8194000000000001e-05 2024-02-19T21:07:57.201 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.201 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.201 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.201 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.427229+0000", 2024-02-19T21:07:57.201 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025390421 2024-02-19T21:07:57.202 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.202 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.202 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.202 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.202 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.202 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.1.0:242 28.15 e132/99 -30:ad087405:::temp_28.15_0_14259_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:07:57.202 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.454353+0000", 2024-02-19T21:07:57.202 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 164.71554467600001, 2024-02-19T21:07:57.202 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.029471536999999999, 2024-02-19T21:07:57.202 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.202 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.202 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.202 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.202 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.203 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.454353+0000", 2024-02-19T21:07:57.203 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.203 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.203 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.203 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.203 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.454353+0000", 2024-02-19T21:07:57.203 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:07:57.203 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.203 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.203 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.203 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.454350+0000", 2024-02-19T21:07:57.203 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.021044232999999999 2024-02-19T21:07:57.203 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.203 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.204 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.204 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.475394+0000", 2024-02-19T21:07:57.204 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7527999999999999e-05 2024-02-19T21:07:57.204 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.204 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.204 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.204 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.475412+0000", 2024-02-19T21:07:57.204 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1040000000000007e-06 2024-02-19T21:07:57.204 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.204 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.204 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.204 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.475420+0000", 2024-02-19T21:07:57.204 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3225000000000005e-05 2024-02-19T21:07:57.204 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.205 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.205 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.205 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.475483+0000", 2024-02-19T21:07:57.205 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4318e-05 2024-02-19T21:07:57.205 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.205 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.205 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.205 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.475508+0000", 2024-02-19T21:07:57.205 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0082776299999999994 2024-02-19T21:07:57.205 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.205 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.205 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.205 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.483785+0000", 2024-02-19T21:07:57.205 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8816999999999998e-05 2024-02-19T21:07:57.206 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.206 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.206 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.206 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.483824+0000", 2024-02-19T21:07:57.206 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.029471536999999999 2024-02-19T21:07:57.206 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.206 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.206 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.206 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.206 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.206 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.1.0:243 28.15 e132/99 -30:ad087405:::temp_28.15_0_14259_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:07:57.206 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.477725+0000", 2024-02-19T21:07:57.206 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 164.69217227199999, 2024-02-19T21:07:57.206 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.020820749, 2024-02-19T21:07:57.206 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.207 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.207 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.207 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.207 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.207 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.477725+0000", 2024-02-19T21:07:57.207 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.207 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.207 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.207 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.207 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.477725+0000", 2024-02-19T21:07:57.207 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:07:57.207 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.207 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.207 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.208 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.477723+0000", 2024-02-19T21:07:57.208 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.012259177 2024-02-19T21:07:57.208 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.208 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.208 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.208 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.489982+0000", 2024-02-19T21:07:57.208 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6534000000000001e-05 2024-02-19T21:07:57.208 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.208 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.208 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.208 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.489999+0000", 2024-02-19T21:07:57.208 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8809999999999996e-06 2024-02-19T21:07:57.208 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.208 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.209 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.209 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.490006+0000", 2024-02-19T21:07:57.209 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2170999999999997e-05 2024-02-19T21:07:57.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.209 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.209 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.209 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.490079+0000", 2024-02-19T21:07:57.209 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3287e-05 2024-02-19T21:07:57.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.209 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.209 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.209 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.490132+0000", 2024-02-19T21:07:57.209 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0083842740000000006 2024-02-19T21:07:57.210 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.210 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.210 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.210 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.498516+0000", 2024-02-19T21:07:57.210 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9481000000000001e-05 2024-02-19T21:07:57.210 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.210 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.210 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.210 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.498546+0000", 2024-02-19T21:07:57.210 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.020820749 2024-02-19T21:07:57.210 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.210 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.210 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.210 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.211 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.211 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.5.0:1270 28.9 e132/99 -30:90852524:::temp_28.9_0_24280_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:07:57.211 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.578695+0000", 2024-02-19T21:07:57.211 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 164.59120267700001, 2024-02-19T21:07:57.211 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017938424000000001, 2024-02-19T21:07:57.211 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.211 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.211 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.211 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.211 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.211 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.578695+0000", 2024-02-19T21:07:57.211 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.211 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.211 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.211 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.212 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.578695+0000", 2024-02-19T21:07:57.212 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:07:57.212 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.212 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.212 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.212 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.578691+0000", 2024-02-19T21:07:57.212 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0030402350000000001 2024-02-19T21:07:57.212 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.212 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.212 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.212 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.581731+0000", 2024-02-19T21:07:57.212 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2955e-05 2024-02-19T21:07:57.212 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.213 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.213 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.213 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.581754+0000", 2024-02-19T21:07:57.213 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0519999999999999e-05 2024-02-19T21:07:57.213 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.213 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.213 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.213 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.581765+0000", 2024-02-19T21:07:57.213 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7444e-05 2024-02-19T21:07:57.213 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.213 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.213 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.213 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.581832+0000", 2024-02-19T21:07:57.213 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9349000000000001e-05 2024-02-19T21:07:57.213 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.581851+0000", 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014735747 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.596587+0000", 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5918000000000003e-05 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.596633+0000", 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017938424000000001 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.214 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.215 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.215 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.215 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.215 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.5.0:1271 28.9 e132/99 -30:90852524:::temp_28.9_0_24280_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:07:57.215 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.608168+0000", 2024-02-19T21:07:57.215 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 164.56172944100001, 2024-02-19T21:07:57.215 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017383068000000002, 2024-02-19T21:07:57.215 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.215 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.215 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.215 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.215 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.215 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.608168+0000", 2024-02-19T21:07:57.215 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.215 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.216 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.216 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.216 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.608168+0000", 2024-02-19T21:07:57.216 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:07:57.216 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.216 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.216 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.216 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.608164+0000", 2024-02-19T21:07:57.216 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003639705 2024-02-19T21:07:57.216 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.216 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.216 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.216 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.611804+0000", 2024-02-19T21:07:57.216 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6741e-05 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.611830+0000", 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0243e-05 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.611841+0000", 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8799999999999999e-05 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.611899+0000", 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7115e-05 2024-02-19T21:07:57.217 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.611927+0000", 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013572864 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.625499+0000", 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1449999999999997e-05 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.625551+0000", 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017383068000000002 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.218 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.219 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.219 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.219 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.219 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.5.0:1272 28.9 e132/99 -30:90852524:::temp_28.9_0_24280_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:07:57.219 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.668531+0000", 2024-02-19T21:07:57.219 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 164.501365846, 2024-02-19T21:07:57.219 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017253208999999999, 2024-02-19T21:07:57.219 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.219 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.219 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.219 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.219 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.219 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.668531+0000", 2024-02-19T21:07:57.219 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.219 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.220 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.220 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.220 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.668531+0000", 2024-02-19T21:07:57.220 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999957 2024-02-19T21:07:57.220 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.220 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.220 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.220 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.668527+0000", 2024-02-19T21:07:57.220 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0031829409999999999 2024-02-19T21:07:57.220 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.220 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.220 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.220 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.671710+0000", 2024-02-19T21:07:57.220 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5936999999999999e-05 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.671736+0000", 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1466e-05 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.671747+0000", 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2371e-05 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.671840+0000", 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0371000000000001e-05 2024-02-19T21:07:57.221 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.222 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.222 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.222 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.671870+0000", 2024-02-19T21:07:57.222 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013870054 2024-02-19T21:07:57.222 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.222 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.222 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.222 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.685740+0000", 2024-02-19T21:07:57.222 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4508000000000001e-05 2024-02-19T21:07:57.222 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.222 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.222 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.222 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.685785+0000", 2024-02-19T21:07:57.222 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017253208999999999 2024-02-19T21:07:57.222 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.223 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.223 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.223 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.223 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.223 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.24919.0:2079 28.9 e132/99 28:90852524:test-rados-api-smithi163-83920-17::big2.copy:head v 132'1, mlcod=0'0)", 2024-02-19T21:07:57.223 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.686869+0000", 2024-02-19T21:07:57.223 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 164.48302867300001, 2024-02-19T21:07:57.223 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0096879210000000004, 2024-02-19T21:07:57.223 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.223 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.223 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.223 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.223 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.223 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.686869+0000", 2024-02-19T21:07:57.224 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.224 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.224 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.224 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.686869+0000", 2024-02-19T21:07:57.224 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:07:57.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.224 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.224 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.224 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.686865+0000", 2024-02-19T21:07:57.224 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00082622800000000003 2024-02-19T21:07:57.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.224 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.224 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.687691+0000", 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7241e-05 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.687709+0000", 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3516e-05 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.687732+0000", 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4459000000000002e-05 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.225 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.687787+0000", 2024-02-19T21:07:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.618e-05 2024-02-19T21:07:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.687813+0000", 2024-02-19T21:07:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0086885639999999993 2024-02-19T21:07:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.696501+0000", 2024-02-19T21:07:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5009000000000001e-05 2024-02-19T21:07:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.226 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.696556+0000", 2024-02-19T21:07:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0096879210000000004 2024-02-19T21:07:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.0.0:262 28.2 e132/99 -30:4327d3d8:::temp_28.2_0_14232_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:07:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.757694+0000", 2024-02-19T21:07:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 164.41220278599999, 2024-02-19T21:07:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037835989, 2024-02-19T21:07:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.227 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.757694+0000", 2024-02-19T21:07:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.757694+0000", 2024-02-19T21:07:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:07:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.757692+0000", 2024-02-19T21:07:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.029339015 2024-02-19T21:07:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.228 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787031+0000", 2024-02-19T21:07:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0139000000000001e-05 2024-02-19T21:07:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787052+0000", 2024-02-19T21:07:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0044000000000001e-05 2024-02-19T21:07:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787062+0000", 2024-02-19T21:07:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8369000000000003e-05 2024-02-19T21:07:57.229 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787120+0000", 2024-02-19T21:07:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7753000000000001e-05 2024-02-19T21:07:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787138+0000", 2024-02-19T21:07:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0083446010000000001 2024-02-19T21:07:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.795482+0000", 2024-02-19T21:07:57.230 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8127000000000002e-05 2024-02-19T21:07:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.795530+0000", 2024-02-19T21:07:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037835989 2024-02-19T21:07:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.0.0:263 28.2 e132/99 -30:4327d3d8:::temp_28.2_0_14232_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:07:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.787076+0000", 2024-02-19T21:07:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 164.38282124, 2024-02-19T21:07:57.231 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.019684811, 2024-02-19T21:07:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787076+0000", 2024-02-19T21:07:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787076+0000", 2024-02-19T21:07:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:07:57.232 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787074+0000", 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011113661 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.798187+0000", 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2937e-05 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.798200+0000", 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1439999999999997e-06 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.233 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.798207+0000", 2024-02-19T21:07:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7419000000000002e-05 2024-02-19T21:07:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.798245+0000", 2024-02-19T21:07:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7419999999999999e-05 2024-02-19T21:07:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.798262+0000", 2024-02-19T21:07:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0084319070000000006 2024-02-19T21:07:57.234 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.806694+0000", 2024-02-19T21:07:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6605000000000004e-05 2024-02-19T21:07:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.806761+0000", 2024-02-19T21:07:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.019684811 2024-02-19T21:07:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.235 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.0.0:264 28.2 e132/99 -30:4327d3d8:::temp_28.2_0_14232_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:07:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.801755+0000", 2024-02-19T21:07:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 164.36814203399999, 2024-02-19T21:07:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022050645000000001, 2024-02-19T21:07:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.801755+0000", 2024-02-19T21:07:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.236 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.801755+0000", 2024-02-19T21:07:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:07:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.801753+0000", 2024-02-19T21:07:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013699945999999999 2024-02-19T21:07:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.815453+0000", 2024-02-19T21:07:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7495000000000001e-05 2024-02-19T21:07:57.237 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.815471+0000", 2024-02-19T21:07:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2979999999999992e-06 2024-02-19T21:07:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.815479+0000", 2024-02-19T21:07:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8300999999999997e-05 2024-02-19T21:07:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.815527+0000", 2024-02-19T21:07:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7969999999999999e-05 2024-02-19T21:07:57.238 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.815545+0000", 2024-02-19T21:07:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0082266249999999996 2024-02-19T21:07:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.823772+0000", 2024-02-19T21:07:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4051000000000003e-05 2024-02-19T21:07:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.239 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.823806+0000", 2024-02-19T21:07:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022050645000000001 2024-02-19T21:07:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.24919.0:2080 28.2 e132/99 28:4327d3d8:test-rados-api-smithi163-83920-17::big3.copy:head v 132'1, mlcod=0'0)", 2024-02-19T21:07:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.823445+0000", 2024-02-19T21:07:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 164.346452656, 2024-02-19T21:07:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0097812209999999997, 2024-02-19T21:07:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.240 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.823445+0000", 2024-02-19T21:07:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.823445+0000", 2024-02-19T21:07:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:07:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.823442+0000", 2024-02-19T21:07:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0033991920000000001 2024-02-19T21:07:57.241 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.826841+0000", 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3720000000000001e-05 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.826855+0000", 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2879999999999993e-06 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.826863+0000", 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1155000000000002e-05 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.242 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.826904+0000", 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9006e-05 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.826933+0000", 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0062467809999999999 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.833180+0000", 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5667999999999997e-05 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.243 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.833226+0000", 2024-02-19T21:07:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0097812209999999997 2024-02-19T21:07:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.48947.0:1 112.0s0 112:0f35bed6:test-rados-api-smithi163-83952-16::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-02-19T21:07:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:47.190422+0000", 2024-02-19T21:07:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 129.97947498100001, 2024-02-19T21:07:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0380564720000001, 2024-02-19T21:07:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:57.244 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.48947", 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3754083491", 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190422+0000", 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190422+0000", 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.245 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190420+0000", 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4949999999999997e-06 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190430+0000", 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0844e-05 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190440+0000", 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3660000000000003e-06 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.246 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190448+0000", 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.083e-05 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190499+0000", 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0529999999999998e-06 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190502+0000", 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.030012302 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:48.220514+0000", 2024-02-19T21:07:57.247 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000132175 2024-02-19T21:07:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:48.220646+0000", 2024-02-19T21:07:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00040324399999999998 2024-02-19T21:07:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:07:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:48.221049+0000", 2024-02-19T21:07:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00034994699999999998 2024-02-19T21:07:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:07:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:48.221399+0000", 2024-02-19T21:07:57.248 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.007007498 2024-02-19T21:07:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:48.228407+0000", 2024-02-19T21:07:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1938000000000001e-05 2024-02-19T21:07:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:48.228479+0000", 2024-02-19T21:07:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0380564720000001 2024-02-19T21:07:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.249 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.49199.0:1 149.6s0 149:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e216)", 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:06:36.212055+0000", 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 80.957842411000001, 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.008338194, 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.49199", 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2827260261", 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212055+0000", 2024-02-19T21:07:57.250 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212055+0000", 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212052+0000", 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0199000000000001e-05 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212073+0000", 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1531000000000001e-05 2024-02-19T21:07:57.251 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212094+0000", 2024-02-19T21:07:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6829999999999997e-06 2024-02-19T21:07:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212103+0000", 2024-02-19T21:07:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00054734599999999999 2024-02-19T21:07:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212650+0000", 2024-02-19T21:07:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8489999999999999e-06 2024-02-19T21:07:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:07:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212654+0000", 2024-02-19T21:07:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0043875600000001 2024-02-19T21:07:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.217042+0000", 2024-02-19T21:07:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012985000000000001 2024-02-19T21:07:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.253 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.217171+0000", 2024-02-19T21:07:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00037497799999999999 2024-02-19T21:07:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:07:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.217546+0000", 2024-02-19T21:07:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020797239999999998 2024-02-19T21:07:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:07:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.219626+0000", 2024-02-19T21:07:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00069722600000000001 2024-02-19T21:07:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.254 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.220323+0000", 2024-02-19T21:07:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9684000000000006e-05 2024-02-19T21:07:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.220393+0000", 2024-02-19T21:07:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.008338194 2024-02-19T21:07:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22689.0:1 172.6s0 172:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-02-19T21:07:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:20.216387+0000", 2024-02-19T21:07:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 36.953510481999999, 2024-02-19T21:07:57.255 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0565913250000001, 2024-02-19T21:07:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:07:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22689", 2024-02-19T21:07:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/585802156", 2024-02-19T21:07:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:07:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216387+0000", 2024-02-19T21:07:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.256 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216387+0000", 2024-02-19T21:07:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:07:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216384+0000", 2024-02-19T21:07:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2157000000000001e-05 2024-02-19T21:07:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216396+0000", 2024-02-19T21:07:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.003e-05 2024-02-19T21:07:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.257 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216416+0000", 2024-02-19T21:07:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7499999999999998e-06 2024-02-19T21:07:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216426+0000", 2024-02-19T21:07:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00066776200000000002 2024-02-19T21:07:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.217093+0000", 2024-02-19T21:07:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6430000000000003e-06 2024-02-19T21:07:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.258 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.217098+0000", 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.051702409 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.268800+0000", 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018998999999999999 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.268990+0000", 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00037534100000000001 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.259 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:07:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.269366+0000", 2024-02-19T21:07:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001076088 2024-02-19T21:07:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:07:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.270442+0000", 2024-02-19T21:07:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024701089999999998 2024-02-19T21:07:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.272912+0000", 2024-02-19T21:07:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6147000000000002e-05 2024-02-19T21:07:57.260 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.272978+0000", 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0565913250000001 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.1.0:634 61.b e278/114 61:d461b0a5:test-rados-api-smithi163-83928-34::foo:14 v 278'95, mlcod=275'91)", 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:38.232659+0000", 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 18.937237725999999, 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0151769, 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:07:57.261 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.232659+0000", 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.232659+0000", 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.232659+0000", 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3270000000000001e-06 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.262 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.232667+0000", 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00080230000000000004 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.233470+0000", 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7861999999999999e-05 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.233488+0000", 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3637000000000003e-05 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.263 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:07:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.233531+0000", 2024-02-19T21:07:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1175999999999999e-05 2024-02-19T21:07:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:07:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.233552+0000", 2024-02-19T21:07:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014260111000000001 2024-02-19T21:07:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:07:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.247812+0000", 2024-02-19T21:07:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3896999999999998e-05 2024-02-19T21:07:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:07:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:07:57.264 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:07:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.247836+0000", 2024-02-19T21:07:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0151769 2024-02-19T21:07:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:07:57.265 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:07:57.265 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:07:57.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.472+0000 7febd49f3700 1 -- 172.21.15.163:0/1692149511 >> v1:172.21.15.193:6789/0 conn(0x7febd0100780 legacy=0x7febd0102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.472+0000 7febd49f3700 1 -- 172.21.15.163:0/1692149511 shutdown_connections 2024-02-19T21:07:57.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.472+0000 7febd49f3700 1 -- 172.21.15.163:0/1692149511 >> 172.21.15.163:0/1692149511 conn(0x7febd006fdc0 msgr2=0x7febd00f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:57.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.472+0000 7febd49f3700 1 -- 172.21.15.163:0/1692149511 shutdown_connections 2024-02-19T21:07:57.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.472+0000 7febd49f3700 1 -- 172.21.15.163:0/1692149511 wait complete. 2024-02-19T21:07:57.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.472+0000 7febd49f3700 1 librados: starting msgr at 2024-02-19T21:07:57.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.472+0000 7febd49f3700 1 librados: starting objecter 2024-02-19T21:07:57.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febd49f3700 1 Processor -- start 2024-02-19T21:07:57.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febd49f3700 1 -- start start 2024-02-19T21:07:57.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febd49f3700 1 librados: setting wanted keys 2024-02-19T21:07:57.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febd49f3700 1 librados: calling monclient init 2024-02-19T21:07:57.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febd49f3700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7febd0119500 con 0x7febd0100780 2024-02-19T21:07:57.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febd49f3700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7febd011a6e0 con 0x7febd01042c0 2024-02-19T21:07:57.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febd49f3700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7febd011b8e0 con 0x7febd00fd9a0 2024-02-19T21:07:57.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febce59c700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7febd00fd9a0 0x7febd0102480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:46948/0 (socket says 172.21.15.163:46948) 2024-02-19T21:07:57.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febce59c700 1 -- 172.21.15.163:0/1895902540 learned_addr learned my addr 172.21.15.163:0/1895902540 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:57.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febbf7fe700 10 client.?.objecter ms_handle_connect 0x7febd00fd9a0 2024-02-19T21:07:57.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febbf7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:57.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febbf7fe700 10 client.?.objecter ms_handle_connect 0x7febd0100780 2024-02-19T21:07:57.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febbf7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:57.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febbf7fe700 10 client.?.objecter ms_handle_connect 0x7febd01042c0 2024-02-19T21:07:57.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febbf7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:57.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635360441 0 0) 0x7febd011b8e0 con 0x7febd00fd9a0 2024-02-19T21:07:57.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febb8004010 con 0x7febd00fd9a0 2024-02-19T21:07:57.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095086080 0 0) 0x7febd0119500 con 0x7febd0100780 2024-02-19T21:07:57.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febd011b8e0 con 0x7febd0100780 2024-02-19T21:07:57.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602171173 0 0) 0x7febd011a6e0 con 0x7febd01042c0 2024-02-19T21:07:57.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febd0119500 con 0x7febd01042c0 2024-02-19T21:07:57.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3193856235 0 0) 0x7febb8004010 con 0x7febd00fd9a0 2024-02-19T21:07:57.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febd011a6e0 con 0x7febd00fd9a0 2024-02-19T21:07:57.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.473+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7febc0002920 con 0x7febd00fd9a0 2024-02-19T21:07:57.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2978752781 0 0) 0x7febd0119500 con 0x7febd01042c0 2024-02-19T21:07:57.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febb8004010 con 0x7febd01042c0 2024-02-19T21:07:57.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1015125550 0 0) 0x7febd011b8e0 con 0x7febd0100780 2024-02-19T21:07:57.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febd0119500 con 0x7febd0100780 2024-02-19T21:07:57.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7febc8001ba0 con 0x7febd01042c0 2024-02-19T21:07:57.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7febc40013d0 con 0x7febd0100780 2024-02-19T21:07:57.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2700927264 0 0) 0x7febd011a6e0 con 0x7febd00fd9a0 2024-02-19T21:07:57.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 >> v1:172.21.15.193:6789/0 conn(0x7febd01042c0 legacy=0x7febd0117d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 >> v1:172.21.15.163:6789/0 conn(0x7febd0100780 legacy=0x7febd0102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd011cae0 con 0x7febd00fd9a0 2024-02-19T21:07:57.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd01196f0 con 0x7febd00fd9a0 2024-02-19T21:07:57.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febd49f3700 10 client.49520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:57.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febd0119be0 con 0x7febd00fd9a0 2024-02-19T21:07:57.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febd49f3700 1 librados: init done 2024-02-19T21:07:57.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7febc0002cb0 con 0x7febd00fd9a0 2024-02-19T21:07:57.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.474+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7febc0005bc0 con 0x7febd00fd9a0 2024-02-19T21:07:57.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.475+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7febb0004f00 con 0x7febd00fd9a0 2024-02-19T21:07:57.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.475+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7febc0018ad0 con 0x7febd00fd9a0 2024-02-19T21:07:57.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(297..297 src has 43..297) v4 ==== 7546+0+0 (unknown 1448518825 0 0) 0x7febc0078b80 con 0x7febd00fd9a0 2024-02-19T21:07:57.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 osd_map(297..297 src has 43..297) v4 2024-02-19T21:07:57.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febbf7fe700 3 client.49520.objecter handle_osd_map got epochs [297,297] > 0 2024-02-19T21:07:57.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febbf7fe700 3 client.49520.objecter handle_osd_map decoding full epoch 297 2024-02-19T21:07:57.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febbf7fe700 20 client.49520.objecter dump_active .. 0 homeless 2024-02-19T21:07:57.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v297) v1 ==== 54+0+16 (unknown 3039431556 0 2420000192) 0x7febc0078f40 con 0x7febd00fd9a0 2024-02-19T21:07:57.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febbf7fe700 10 client.49520.objecter ms_handle_connect 0x7febb8063e30 2024-02-19T21:07:57.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febd49f3700 10 client.49520.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:57.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 1 2024-02-19T21:07:57.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb00051a0 osd=0 3 2024-02-19T21:07:57.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb00051a0 osd=0 3 2024-02-19T21:07:57.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb00051a0 osd=0 3 2024-02-19T21:07:57.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 1 2024-02-19T21:07:57.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb00051a0 osd=0 3 2024-02-19T21:07:57.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 0 1 2024-02-19T21:07:57.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb00051a0 osd=0 4 2024-02-19T21:07:57.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febd49f3700 10 client.49520.objecter _send_command 1 2024-02-19T21:07:57.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7febb0004950 con 0x7febb0004530 2024-02-19T21:07:57.547 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:07:57.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.476+0000 7febbf7fe700 10 client.49520.objecter ms_handle_connect 0x7febb0004530 2024-02-19T21:07:57.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.477+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febb0004950 con 0x7febb0004530 2024-02-19T21:07:57.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.477+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:07:57.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.477+0000 7febbf7fe700 10 client.49520.objecter _finish_command 1 = system:0 2024-02-19T21:07:57.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.477+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb00051a0 osd=0 4 2024-02-19T21:07:57.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.477+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 0 1 2024-02-19T21:07:57.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.492+0000 7febd49f3700 10 client.49520.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:57.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.492+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 2 2024-02-19T21:07:57.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.492+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb00051a0 osd=0 3 2024-02-19T21:07:57.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.492+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb00051a0 osd=0 3 2024-02-19T21:07:57.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.492+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb00051a0 osd=0 3 2024-02-19T21:07:57.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.492+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 2 2024-02-19T21:07:57.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.492+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb00051a0 osd=0 3 2024-02-19T21:07:57.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.492+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 0 2 2024-02-19T21:07:57.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.492+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb00051a0 osd=0 4 2024-02-19T21:07:57.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.492+0000 7febd49f3700 10 client.49520.objecter _send_command 2 2024-02-19T21:07:57.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.492+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febb0004d30 con 0x7febb0004530 2024-02-19T21:07:57.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.493+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febb0004d30 con 0x7febb0004530 2024-02-19T21:07:57.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.493+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:57.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.493+0000 7febbf7fe700 10 client.49520.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.493+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb00051a0 osd=0 4 2024-02-19T21:07:57.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.493+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 0 2 2024-02-19T21:07:57.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.494+0000 7febd49f3700 10 client.49520.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:57.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.494+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 3 2024-02-19T21:07:57.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.494+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0003fe0 osd=1 3 2024-02-19T21:07:57.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.494+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0003fe0 osd=1 3 2024-02-19T21:07:57.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.494+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0003fe0 osd=1 3 2024-02-19T21:07:57.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.494+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 3 2024-02-19T21:07:57.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.494+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb0003fe0 osd=1 3 2024-02-19T21:07:57.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.494+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 1 3 2024-02-19T21:07:57.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.494+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0003fe0 osd=1 4 2024-02-19T21:07:57.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.494+0000 7febd49f3700 10 client.49520.objecter _send_command 3 2024-02-19T21:07:57.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.494+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7febb000c000 con 0x7febb0004150 2024-02-19T21:07:57.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.494+0000 7febbf7fe700 10 client.49520.objecter ms_handle_connect 0x7febb0004150 2024-02-19T21:07:57.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.495+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febb000c000 con 0x7febb0004150 2024-02-19T21:07:57.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.495+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:07:57.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.495+0000 7febbf7fe700 10 client.49520.objecter _finish_command 3 = system:0 2024-02-19T21:07:57.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.495+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb0003fe0 osd=1 4 2024-02-19T21:07:57.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.495+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 1 3 2024-02-19T21:07:57.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.509+0000 7febd49f3700 10 client.49520.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:57.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.509+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 4 2024-02-19T21:07:57.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.509+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0003fe0 osd=1 3 2024-02-19T21:07:57.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.509+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0003fe0 osd=1 3 2024-02-19T21:07:57.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.509+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0003fe0 osd=1 3 2024-02-19T21:07:57.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.509+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 4 2024-02-19T21:07:57.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.509+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb0003fe0 osd=1 3 2024-02-19T21:07:57.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.509+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 1 4 2024-02-19T21:07:57.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.509+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0003fe0 osd=1 4 2024-02-19T21:07:57.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.509+0000 7febd49f3700 10 client.49520.objecter _send_command 4 2024-02-19T21:07:57.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.509+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febb00018b0 con 0x7febb0004150 2024-02-19T21:07:57.551 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:07:57.551 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:07:57.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febb00018b0 con 0x7febb0004150 2024-02-19T21:07:57.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:57.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febbf7fe700 10 client.49520.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb0003fe0 osd=1 4 2024-02-19T21:07:57.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 1 4 2024-02-19T21:07:57.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febd49f3700 10 client.49520.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:57.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 5 2024-02-19T21:07:57.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0003630 osd=2 3 2024-02-19T21:07:57.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0003630 osd=2 3 2024-02-19T21:07:57.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0003630 osd=2 3 2024-02-19T21:07:57.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 5 2024-02-19T21:07:57.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb0003630 osd=2 3 2024-02-19T21:07:57.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 2 5 2024-02-19T21:07:57.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0003630 osd=2 4 2024-02-19T21:07:57.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febd49f3700 10 client.49520.objecter _send_command 5 2024-02-19T21:07:57.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7febb000de90 con 0x7febb000d2f0 2024-02-19T21:07:57.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.511+0000 7febbf7fe700 10 client.49520.objecter ms_handle_connect 0x7febb000d2f0 2024-02-19T21:07:57.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.512+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febb000de90 con 0x7febb000d2f0 2024-02-19T21:07:57.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.512+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:07:57.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.512+0000 7febbf7fe700 10 client.49520.objecter _finish_command 5 = system:0 2024-02-19T21:07:57.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.512+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb0003630 osd=2 4 2024-02-19T21:07:57.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.512+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 2 5 2024-02-19T21:07:57.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.527+0000 7febd49f3700 10 client.49520.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:57.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.527+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 6 2024-02-19T21:07:57.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.527+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0003630 osd=2 3 2024-02-19T21:07:57.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.527+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0003630 osd=2 3 2024-02-19T21:07:57.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.527+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0003630 osd=2 3 2024-02-19T21:07:57.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.527+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 6 2024-02-19T21:07:57.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.527+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb0003630 osd=2 3 2024-02-19T21:07:57.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.527+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 2 6 2024-02-19T21:07:57.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.527+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0003630 osd=2 4 2024-02-19T21:07:57.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.527+0000 7febd49f3700 10 client.49520.objecter _send_command 6 2024-02-19T21:07:57.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.527+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febb00030a0 con 0x7febb000d2f0 2024-02-19T21:07:57.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febb00030a0 con 0x7febb000d2f0 2024-02-19T21:07:57.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:57.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febbf7fe700 10 client.49520.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb0003630 osd=2 4 2024-02-19T21:07:57.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 2 6 2024-02-19T21:07:57.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febd49f3700 10 client.49520.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:57.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 7 2024-02-19T21:07:57.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0011970 osd=3 3 2024-02-19T21:07:57.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0011970 osd=3 3 2024-02-19T21:07:57.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0011970 osd=3 3 2024-02-19T21:07:57.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 7 2024-02-19T21:07:57.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb0011970 osd=3 3 2024-02-19T21:07:57.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 3 7 2024-02-19T21:07:57.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0011970 osd=3 4 2024-02-19T21:07:57.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febd49f3700 10 client.49520.objecter _send_command 7 2024-02-19T21:07:57.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7febb0015b00 con 0x7febb0011ff0 2024-02-19T21:07:57.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.529+0000 7febbf7fe700 10 client.49520.objecter ms_handle_connect 0x7febb0011ff0 2024-02-19T21:07:57.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.530+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febb0015b00 con 0x7febb0011ff0 2024-02-19T21:07:57.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.530+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:07:57.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.530+0000 7febbf7fe700 10 client.49520.objecter _finish_command 7 = system:0 2024-02-19T21:07:57.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.530+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb0011970 osd=3 4 2024-02-19T21:07:57.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.530+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 3 7 2024-02-19T21:07:57.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.545+0000 7febd49f3700 10 client.49520.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:57.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.545+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 8 2024-02-19T21:07:57.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.545+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0011970 osd=3 3 2024-02-19T21:07:57.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.545+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0011970 osd=3 3 2024-02-19T21:07:57.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.545+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0011970 osd=3 3 2024-02-19T21:07:57.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.545+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 8 2024-02-19T21:07:57.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.545+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb0011970 osd=3 3 2024-02-19T21:07:57.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.545+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 3 8 2024-02-19T21:07:57.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.545+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0011970 osd=3 4 2024-02-19T21:07:57.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.545+0000 7febd49f3700 10 client.49520.objecter _send_command 8 2024-02-19T21:07:57.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.545+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febb000e240 con 0x7febb0011ff0 2024-02-19T21:07:57.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.547+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febb000e240 con 0x7febb0011ff0 2024-02-19T21:07:57.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.547+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:57.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.547+0000 7febbf7fe700 10 client.49520.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.547+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb0011970 osd=3 4 2024-02-19T21:07:57.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.547+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 3 8 2024-02-19T21:07:57.557 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:07:57.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.548+0000 7febd49f3700 10 client.49520.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:57.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.548+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 9 2024-02-19T21:07:57.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.548+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb000e000 osd=4 3 2024-02-19T21:07:57.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.548+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb000e000 osd=4 3 2024-02-19T21:07:57.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.548+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb000e000 osd=4 3 2024-02-19T21:07:57.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.548+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 9 2024-02-19T21:07:57.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.548+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb000e000 osd=4 3 2024-02-19T21:07:57.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.548+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 4 9 2024-02-19T21:07:57.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.548+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb000e000 osd=4 4 2024-02-19T21:07:57.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.548+0000 7febd49f3700 10 client.49520.objecter _send_command 9 2024-02-19T21:07:57.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.548+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7febb000e170 con 0x7febb000ce50 2024-02-19T21:07:57.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.548+0000 7febbf7fe700 10 client.49520.objecter ms_handle_connect 0x7febb000ce50 2024-02-19T21:07:57.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.549+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febb000e170 con 0x7febb000ce50 2024-02-19T21:07:57.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.549+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:07:57.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.549+0000 7febbf7fe700 10 client.49520.objecter _finish_command 9 = system:0 2024-02-19T21:07:57.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.549+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb000e000 osd=4 4 2024-02-19T21:07:57.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.549+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 4 9 2024-02-19T21:07:57.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.565+0000 7febd49f3700 10 client.49520.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:57.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.565+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 10 2024-02-19T21:07:57.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.565+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb000e000 osd=4 3 2024-02-19T21:07:57.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.565+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb000e000 osd=4 3 2024-02-19T21:07:57.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.565+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb000e000 osd=4 3 2024-02-19T21:07:57.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.565+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 10 2024-02-19T21:07:57.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.565+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb000e000 osd=4 3 2024-02-19T21:07:57.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.565+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 4 10 2024-02-19T21:07:57.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.565+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb000e000 osd=4 4 2024-02-19T21:07:57.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.565+0000 7febd49f3700 10 client.49520.objecter _send_command 10 2024-02-19T21:07:57.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.565+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febb000e170 con 0x7febb000ce50 2024-02-19T21:07:57.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febb000e170 con 0x7febb000ce50 2024-02-19T21:07:57.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:57.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febbf7fe700 10 client.49520.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb000e000 osd=4 4 2024-02-19T21:07:57.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 4 10 2024-02-19T21:07:57.564 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:07:57.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febd49f3700 10 client.49520.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:57.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 11 2024-02-19T21:07:57.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb000e170 osd=5 3 2024-02-19T21:07:57.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb000e170 osd=5 3 2024-02-19T21:07:57.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb000e170 osd=5 3 2024-02-19T21:07:57.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 11 2024-02-19T21:07:57.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb000e170 osd=5 3 2024-02-19T21:07:57.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 5 11 2024-02-19T21:07:57.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb000e170 osd=5 4 2024-02-19T21:07:57.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febd49f3700 10 client.49520.objecter _send_command 11 2024-02-19T21:07:57.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7febb0018040 con 0x7febb0017a80 2024-02-19T21:07:57.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7febbf7fe700 10 client.49520.objecter ms_handle_connect 0x7febb0017a80 2024-02-19T21:07:57.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.568+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febb0018040 con 0x7febb0017a80 2024-02-19T21:07:57.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.568+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:07:57.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.568+0000 7febbf7fe700 10 client.49520.objecter _finish_command 11 = system:0 2024-02-19T21:07:57.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.568+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb000e170 osd=5 4 2024-02-19T21:07:57.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.568+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 5 11 2024-02-19T21:07:57.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.583+0000 7febd49f3700 10 client.49520.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:57.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.583+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 12 2024-02-19T21:07:57.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.583+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb000e170 osd=5 3 2024-02-19T21:07:57.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.583+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb000e170 osd=5 3 2024-02-19T21:07:57.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.583+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb000e170 osd=5 3 2024-02-19T21:07:57.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.583+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 12 2024-02-19T21:07:57.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.583+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb000e170 osd=5 3 2024-02-19T21:07:57.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.583+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 5 12 2024-02-19T21:07:57.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.583+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb000e170 osd=5 4 2024-02-19T21:07:57.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.583+0000 7febd49f3700 10 client.49520.objecter _send_command 12 2024-02-19T21:07:57.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.583+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febb0018040 con 0x7febb0017a80 2024-02-19T21:07:57.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febb0018040 con 0x7febb0017a80 2024-02-19T21:07:57.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:57.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febbf7fe700 10 client.49520.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb000e170 osd=5 4 2024-02-19T21:07:57.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 5 12 2024-02-19T21:07:57.582 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:07:57.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febd49f3700 10 client.49520.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:57.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 13 2024-02-19T21:07:57.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0018040 osd=6 3 2024-02-19T21:07:57.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0018040 osd=6 3 2024-02-19T21:07:57.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0018040 osd=6 3 2024-02-19T21:07:57.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 13 2024-02-19T21:07:57.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb0018040 osd=6 3 2024-02-19T21:07:57.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 6 13 2024-02-19T21:07:57.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0018040 osd=6 4 2024-02-19T21:07:57.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febd49f3700 10 client.49520.objecter _send_command 13 2024-02-19T21:07:57.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7febb0022c40 con 0x7febb001f100 2024-02-19T21:07:57.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.585+0000 7febbf7fe700 10 client.49520.objecter ms_handle_connect 0x7febb001f100 2024-02-19T21:07:57.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.586+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febb0022c40 con 0x7febb001f100 2024-02-19T21:07:57.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.586+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:07:57.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.586+0000 7febbf7fe700 10 client.49520.objecter _finish_command 13 = system:0 2024-02-19T21:07:57.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.586+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb0018040 osd=6 4 2024-02-19T21:07:57.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.586+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 6 13 2024-02-19T21:07:57.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.600+0000 7febd49f3700 10 client.49520.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:57.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.600+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 14 2024-02-19T21:07:57.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.600+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0018040 osd=6 3 2024-02-19T21:07:57.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.600+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0018040 osd=6 3 2024-02-19T21:07:57.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.600+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb0018040 osd=6 3 2024-02-19T21:07:57.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.600+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 14 2024-02-19T21:07:57.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.600+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb0018040 osd=6 3 2024-02-19T21:07:57.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.600+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 6 14 2024-02-19T21:07:57.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.600+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0018040 osd=6 4 2024-02-19T21:07:57.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.600+0000 7febd49f3700 10 client.49520.objecter _send_command 14 2024-02-19T21:07:57.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.600+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febb00243f0 con 0x7febb001f100 2024-02-19T21:07:57.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febb00243f0 con 0x7febb001f100 2024-02-19T21:07:57.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:57.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febbf7fe700 10 client.49520.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb0018040 osd=6 4 2024-02-19T21:07:57.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 6 14 2024-02-19T21:07:57.599 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:07:57.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febd49f3700 10 client.49520.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:07:57.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 15 2024-02-19T21:07:57.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb00243f0 osd=7 3 2024-02-19T21:07:57.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb00243f0 osd=7 3 2024-02-19T21:07:57.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb00243f0 osd=7 3 2024-02-19T21:07:57.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 15 2024-02-19T21:07:57.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb00243f0 osd=7 3 2024-02-19T21:07:57.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 7 15 2024-02-19T21:07:57.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb00243f0 osd=7 4 2024-02-19T21:07:57.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febd49f3700 10 client.49520.objecter _send_command 15 2024-02-19T21:07:57.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.602+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7febb0025010 con 0x7febb0024ad0 2024-02-19T21:07:57.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.603+0000 7febbf7fe700 10 client.49520.objecter ms_handle_connect 0x7febb0024ad0 2024-02-19T21:07:57.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.603+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febb0025010 con 0x7febb0024ad0 2024-02-19T21:07:57.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.603+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:07:57.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.603+0000 7febbf7fe700 10 client.49520.objecter _finish_command 15 = system:0 2024-02-19T21:07:57.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.603+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb00243f0 osd=7 4 2024-02-19T21:07:57.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.603+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 7 15 2024-02-19T21:07:57.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.617+0000 7febd49f3700 10 client.49520.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:07:57.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.617+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign -1 16 2024-02-19T21:07:57.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.617+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb00243f0 osd=7 3 2024-02-19T21:07:57.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.617+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb00243f0 osd=7 3 2024-02-19T21:07:57.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.617+0000 7febd49f3700 20 client.49520.objecter _get_session s=0x7febb00243f0 osd=7 3 2024-02-19T21:07:57.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.617+0000 7febd49f3700 15 client.49520.objecter _session_command_op_remove -1 16 2024-02-19T21:07:57.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.617+0000 7febd49f3700 20 client.49520.objecter get_session s=0x7febb00243f0 osd=7 3 2024-02-19T21:07:57.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.617+0000 7febd49f3700 15 client.49520.objecter _session_command_op_assign 7 16 2024-02-19T21:07:57.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.617+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb00243f0 osd=7 4 2024-02-19T21:07:57.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.617+0000 7febd49f3700 10 client.49520.objecter _send_command 16 2024-02-19T21:07:57.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.617+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febb0025010 con 0x7febb0024ad0 2024-02-19T21:07:57.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febbf7fe700 1 -- 172.21.15.163:0/1895902540 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febb0025010 con 0x7febb0024ad0 2024-02-19T21:07:57.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febbf7fe700 10 client.49520.objecter ms_dispatch 0x7febd0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:07:57.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febbf7fe700 10 client.49520.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febbf7fe700 20 client.49520.objecter put_session s=0x7febb00243f0 osd=7 4 2024-02-19T21:07:57.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febbf7fe700 15 client.49520.objecter _session_command_op_remove 7 16 2024-02-19T21:07:57.616 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:07:57.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 10 librados: watch_flush enter 2024-02-19T21:07:57.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 10 librados: watch_flush exit 2024-02-19T21:07:57.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 10 client.49520.objecter close_session for osd.0 2024-02-19T21:07:57.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 >> v1:172.21.15.163:6801/1559418115 conn(0x7febb0004530 legacy=0x7febb0002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb00051a0 osd=0 1 2024-02-19T21:07:57.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 10 client.49520.objecter close_session for osd.1 2024-02-19T21:07:57.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 >> v1:172.21.15.163:6805/3125047432 conn(0x7febb0004150 legacy=0x7febb000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0003fe0 osd=1 1 2024-02-19T21:07:57.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 10 client.49520.objecter close_session for osd.2 2024-02-19T21:07:57.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 >> v1:172.21.15.163:6809/1563158487 conn(0x7febb000d2f0 legacy=0x7febb000d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0003630 osd=2 1 2024-02-19T21:07:57.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febced9d700 1 -- 172.21.15.163:0/1895902540 reap_dead start 2024-02-19T21:07:57.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 10 client.49520.objecter close_session for osd.3 2024-02-19T21:07:57.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 >> v1:172.21.15.163:6813/2112279769 conn(0x7febb0011ff0 legacy=0x7febb00143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0011970 osd=3 1 2024-02-19T21:07:57.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 10 client.49520.objecter close_session for osd.4 2024-02-19T21:07:57.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 >> v1:172.21.15.193:6800/3129716965 conn(0x7febb000ce50 legacy=0x7febb0015ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb000e000 osd=4 1 2024-02-19T21:07:57.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 10 client.49520.objecter close_session for osd.5 2024-02-19T21:07:57.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 >> v1:172.21.15.193:6804/805161991 conn(0x7febb0017a80 legacy=0x7febb001d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb000e170 osd=5 1 2024-02-19T21:07:57.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 10 client.49520.objecter close_session for osd.6 2024-02-19T21:07:57.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 >> v1:172.21.15.193:6808/787604615 conn(0x7febb001f100 legacy=0x7febb00214f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb0018040 osd=6 1 2024-02-19T21:07:57.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 10 client.49520.objecter close_session for osd.7 2024-02-19T21:07:57.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 >> v1:172.21.15.193:6812/157927432 conn(0x7febb0024ad0 legacy=0x7febb0022c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 20 client.49520.objecter put_session s=0x7febb00243f0 osd=7 1 2024-02-19T21:07:57.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 20 client.49520.objecter shutdown clearing up homeless session... 2024-02-19T21:07:57.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febd49f3700 10 client.49520.objecter successfully canceled tick 2024-02-19T21:07:57.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.619+0000 7febced9d700 1 -- 172.21.15.163:0/1895902540 reap_dead start 2024-02-19T21:07:57.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.620+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 >> v1:172.21.15.163:6800/1552514681 conn(0x7febb8063e30 legacy=0x7febb80662f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.620+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 >> v1:172.21.15.163:6790/0 conn(0x7febd00fd9a0 legacy=0x7febd0102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.620+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 shutdown_connections 2024-02-19T21:07:57.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.620+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 >> 172.21.15.163:0/1895902540 conn(0x7febd006fdc0 msgr2=0x7febd00f4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:57.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.620+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 shutdown_connections 2024-02-19T21:07:57.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.620+0000 7febd49f3700 1 -- 172.21.15.163:0/1895902540 wait complete. 2024-02-19T21:07:57.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.620+0000 7febd49f3700 1 librados: shutdown 2024-02-19T21:07:57.620 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.620 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.620 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.621 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.621 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.621 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.621 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.621 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:07:57.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.558+0000 7f46f20a4700 1 -- 172.21.15.163:0/3399757699 >> v1:172.21.15.163:6789/0 conn(0x7f46ec0fc870 legacy=0x7f46ec0fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.558+0000 7f46f20a4700 1 -- 172.21.15.163:0/3399757699 shutdown_connections 2024-02-19T21:07:57.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.558+0000 7f46f20a4700 1 -- 172.21.15.163:0/3399757699 >> 172.21.15.163:0/3399757699 conn(0x7f46ec0f2470 msgr2=0x7f46ec0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:57.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.558+0000 7f46f20a4700 1 -- 172.21.15.163:0/3399757699 shutdown_connections 2024-02-19T21:07:57.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.558+0000 7f46f20a4700 1 -- 172.21.15.163:0/3399757699 wait complete. 2024-02-19T21:07:57.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.559+0000 7f46f20a4700 1 librados: starting msgr at 2024-02-19T21:07:57.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.559+0000 7f46f20a4700 1 librados: starting objecter 2024-02-19T21:07:57.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.559+0000 7f46f20a4700 1 Processor -- start 2024-02-19T21:07:57.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.559+0000 7f46f20a4700 1 -- start start 2024-02-19T21:07:57.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.559+0000 7f46f20a4700 1 librados: setting wanted keys 2024-02-19T21:07:57.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.559+0000 7f46f20a4700 1 librados: calling monclient init 2024-02-19T21:07:57.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.559+0000 7f46f20a4700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f46ec117320 con 0x7f46ec1031b0 2024-02-19T21:07:57.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.559+0000 7f46f20a4700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f46ec1184e0 con 0x7f46ec0fc870 2024-02-19T21:07:57.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.559+0000 7f46f20a4700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f46ec1196e0 con 0x7f46ec0ff670 2024-02-19T21:07:57.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46ebfff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f46ec1031b0 0x7f46ec115a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:60240/0 (socket says 172.21.15.163:60240) 2024-02-19T21:07:57.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46eaffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f46ec0ff670 0x7f46ec0613c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:46974/0 (socket says 172.21.15.163:46974) 2024-02-19T21:07:57.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46ebfff700 1 -- 172.21.15.163:0/4256069018 learned_addr learned my addr 172.21.15.163:0/4256069018 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:07:57.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 10 client.?.objecter ms_handle_connect 0x7f46ec0ff670 2024-02-19T21:07:57.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:57.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 10 client.?.objecter ms_handle_connect 0x7f46ec1031b0 2024-02-19T21:07:57.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:57.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 10 client.?.objecter ms_handle_connect 0x7f46ec0fc870 2024-02-19T21:07:57.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:07:57.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971437641 0 0) 0x7f46ec1196e0 con 0x7f46ec0ff670 2024-02-19T21:07:57.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d4003800 con 0x7f46ec0ff670 2024-02-19T21:07:57.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277680558 0 0) 0x7f46ec117320 con 0x7f46ec1031b0 2024-02-19T21:07:57.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46ec1196e0 con 0x7f46ec1031b0 2024-02-19T21:07:57.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928938289 0 0) 0x7f46ec1184e0 con 0x7f46ec0fc870 2024-02-19T21:07:57.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46ec117320 con 0x7f46ec0fc870 2024-02-19T21:07:57.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1644519072 0 0) 0x7f46ec117320 con 0x7f46ec0fc870 2024-02-19T21:07:57.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46ec1184e0 con 0x7f46ec0fc870 2024-02-19T21:07:57.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f46dc002de0 con 0x7f46ec0fc870 2024-02-19T21:07:57.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2826499426 0 0) 0x7f46ec1196e0 con 0x7f46ec1031b0 2024-02-19T21:07:57.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46ec117320 con 0x7f46ec1031b0 2024-02-19T21:07:57.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 411222997 0 0) 0x7f46d4003800 con 0x7f46ec0ff670 2024-02-19T21:07:57.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46ec1196e0 con 0x7f46ec0ff670 2024-02-19T21:07:57.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f46e4001c50 con 0x7f46ec1031b0 2024-02-19T21:07:57.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f46e0001840 con 0x7f46ec0ff670 2024-02-19T21:07:57.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3431141550 0 0) 0x7f46ec1184e0 con 0x7f46ec0fc870 2024-02-19T21:07:57.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 >> v1:172.21.15.163:6790/0 conn(0x7f46ec0ff670 legacy=0x7f46ec0613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 >> v1:172.21.15.163:6789/0 conn(0x7f46ec1031b0 legacy=0x7f46ec115a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:57.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ec11a8e0 con 0x7f46ec0fc870 2024-02-19T21:07:57.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46f20a4700 1 -- 172.21.15.163:0/4256069018 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ec117510 con 0x7f46ec0fc870 2024-02-19T21:07:57.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46f20a4700 10 client.48343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:07:57.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46f20a4700 1 -- 172.21.15.163:0/4256069018 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ec117980 con 0x7f46ec0fc870 2024-02-19T21:07:57.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46f20a4700 1 librados: init done 2024-02-19T21:07:57.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f46dc001120 con 0x7f46ec0fc870 2024-02-19T21:07:57.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.560+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f46dc0036b0 con 0x7f46ec0fc870 2024-02-19T21:07:57.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.561+0000 7f46f20a4700 1 -- 172.21.15.163:0/4256069018 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d8004f20 con 0x7f46ec0fc870 2024-02-19T21:07:57.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.561+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f46dc016540 con 0x7f46ec0fc870 2024-02-19T21:07:57.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.562+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(297..297 src has 43..297) v4 ==== 7546+0+0 (unknown 1448518825 0 0) 0x7f46dc076640 con 0x7f46ec0fc870 2024-02-19T21:07:57.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.562+0000 7f46e8ff9700 10 client.48343.objecter ms_dispatch 0x7f46ec001550 osd_map(297..297 src has 43..297) v4 2024-02-19T21:07:57.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.562+0000 7f46e8ff9700 3 client.48343.objecter handle_osd_map got epochs [297,297] > 0 2024-02-19T21:07:57.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.562+0000 7f46e8ff9700 3 client.48343.objecter handle_osd_map decoding full epoch 297 2024-02-19T21:07:57.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.562+0000 7f46e8ff9700 20 client.48343.objecter dump_active .. 0 homeless 2024-02-19T21:07:57.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.562+0000 7f46e8ff9700 10 client.48343.objecter ms_handle_connect 0x7f46d4063da0 2024-02-19T21:07:57.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.567+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f46dc049c30 con 0x7f46ec0fc870 2024-02-19T21:07:57.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:57.757+0000 7f46f20a4700 1 -- 172.21.15.163:0/4256069018 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f46d80051d0 con 0x7f46ec0fc870 2024-02-19T21:07:57.817 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:07:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:07:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:07:57.918 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-19T21:07:57.924 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:07:57.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:57.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:07:58.028 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:07:58.057 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:58.183 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:07:58.217 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:57.173841+0000 mgr.y (mgr.14550) 294 : cluster [DBG] pgmap v416: 232 pgs: 232 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 71 op/s 2024-02-19T21:07:58.217 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:57.217733+0000 mon.a (mon.0) 2045 : audit [INF] from='client.48319 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi163-83854-52"}]': finished 2024-02-19T21:07:58.217 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:57.217837+0000 mon.a (mon.0) 2046 : cluster [DBG] osdmap e297: 8 total, 8 up, 7 in 2024-02-19T21:07:58.217 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:57.239885+0000 mon.c (mon.2) 336 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:58.217 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:57.469779+0000 mon.a (mon.0) 2047 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:58.217 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:57.471018+0000 mon.c (mon.2) 337 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:58.217 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:57.471344+0000 mon.a (mon.0) 2048 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:58.218 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:57.173841+0000 mgr.y (mgr.14550) 294 : cluster [DBG] pgmap v416: 232 pgs: 232 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 71 op/s 2024-02-19T21:07:58.218 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:57.217733+0000 mon.a (mon.0) 2045 : audit [INF] from='client.48319 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi163-83854-52"}]': finished 2024-02-19T21:07:58.218 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:57.217837+0000 mon.a (mon.0) 2046 : cluster [DBG] osdmap e297: 8 total, 8 up, 7 in 2024-02-19T21:07:58.218 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:57.239885+0000 mon.c (mon.2) 336 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:58.218 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:57.469779+0000 mon.a (mon.0) 2047 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:58.218 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:57.471018+0000 mon.c (mon.2) 337 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:58.218 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:57.471344+0000 mon.a (mon.0) 2048 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:58.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:58.228+0000 7f46e8ff9700 1 -- 172.21.15.163:0/4256069018 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v298) v1 ==== 91+0+0 (unknown 2526410719 0 0) 0x7f46dc0480a0 con 0x7f46ec0fc870 2024-02-19T21:07:58.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:58.237+0000 7f46f20a4700 10 librados: watch_flush enter 2024-02-19T21:07:58.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:58.237+0000 7f46f20a4700 10 librados: watch_flush exit 2024-02-19T21:07:58.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:58.237+0000 7f46f20a4700 20 client.48343.objecter shutdown clearing up homeless session... 2024-02-19T21:07:58.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:58.237+0000 7f46f20a4700 10 client.48343.objecter successfully canceled tick 2024-02-19T21:07:58.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:58.238+0000 7f46f20a4700 1 -- 172.21.15.163:0/4256069018 >> v1:172.21.15.163:6800/1552514681 conn(0x7f46d4063da0 legacy=0x7f46d4066260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:58.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:58.238+0000 7f46f20a4700 1 -- 172.21.15.163:0/4256069018 >> v1:172.21.15.193:6789/0 conn(0x7f46ec0fc870 legacy=0x7f46ec060cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:07:58.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:58.238+0000 7f46f20a4700 1 -- 172.21.15.163:0/4256069018 shutdown_connections 2024-02-19T21:07:58.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:58.238+0000 7f46f20a4700 1 -- 172.21.15.163:0/4256069018 >> 172.21.15.163:0/4256069018 conn(0x7f46ec0f2470 msgr2=0x7f46ec106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:07:58.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:58.239+0000 7f46f20a4700 1 -- 172.21.15.163:0/4256069018 shutdown_connections 2024-02-19T21:07:58.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:58.239+0000 7f46f20a4700 1 -- 172.21.15.163:0/4256069018 wait complete. 2024-02-19T21:07:58.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:07:58.239+0000 7f46f20a4700 1 librados: shutdown 2024-02-19T21:07:58.236 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:07:58.489 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:57.173841+0000 mgr.y (mgr.14550) 294 : cluster [DBG] pgmap v416: 232 pgs: 232 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 71 op/s 2024-02-19T21:07:58.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:57.217733+0000 mon.a (mon.0) 2045 : audit [INF] from='client.48319 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi163-83854-52"}]': finished 2024-02-19T21:07:58.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:57.217837+0000 mon.a (mon.0) 2046 : cluster [DBG] osdmap e297: 8 total, 8 up, 7 in 2024-02-19T21:07:58.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:57.239885+0000 mon.c (mon.2) 336 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:58.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:57.469779+0000 mon.a (mon.0) 2047 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:58.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:57.471018+0000 mon.c (mon.2) 337 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:58.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:57.471344+0000 mon.a (mon.0) 2048 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:58.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:57.472155+0000 mon.c (mon.2) 338 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi163-83854-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:58.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:57.472514+0000 mon.a (mon.0) 2049 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi163-83854-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:58.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:57.476910+0000 mon.c (mon.2) 339 : audit [DBG] from='client.? v1:172.21.15.163:0/1895902540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:58.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:57.759245+0000 mon.b (mon.1) 464 : audit [INF] from='client.? v1:172.21.15.163:0/4256069018' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:07:58.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:57.759596+0000 mon.a (mon.0) 2050 : audit [INF] from='client.48343 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:07:58.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:57.472155+0000 mon.c (mon.2) 338 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi163-83854-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:58.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:57.472514+0000 mon.a (mon.0) 2049 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi163-83854-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:58.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:57.476910+0000 mon.c (mon.2) 339 : audit [DBG] from='client.? v1:172.21.15.163:0/1895902540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:58.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:57.759245+0000 mon.b (mon.1) 464 : audit [INF] from='client.? v1:172.21.15.163:0/4256069018' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:07:58.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:57.759596+0000 mon.a (mon.0) 2050 : audit [INF] from='client.48343 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:07:58.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:57.472155+0000 mon.c (mon.2) 338 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi163-83854-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:58.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:57.472514+0000 mon.a (mon.0) 2049 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi163-83854-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:07:58.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:57.476910+0000 mon.c (mon.2) 339 : audit [DBG] from='client.? v1:172.21.15.163:0/1895902540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:07:58.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:57.759245+0000 mon.b (mon.1) 464 : audit [INF] from='client.? v1:172.21.15.163:0/4256069018' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:07:58.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:57.759596+0000 mon.a (mon.0) 2050 : audit [INF] from='client.48343 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:07:59.137 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:07:59.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:59.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:07:59.240 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:07:59.269 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:59.456 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:58.215591+0000 mon.a (mon.0) 2051 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:59.456 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:58.228492+0000 mon.a (mon.0) 2052 : audit [INF] from='client.49514 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi163-83854-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:59.456 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:58.228572+0000 mon.a (mon.0) 2053 : audit [INF] from='client.48343 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:07:59.457 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:58.228625+0000 mon.a (mon.0) 2054 : cluster [DBG] osdmap e298: 8 total, 8 up, 7 in 2024-02-19T21:07:59.457 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:58.229203+0000 mon.c (mon.2) 340 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi163-83854-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:59.457 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:58.229884+0000 mon.a (mon.0) 2055 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi163-83854-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:59.457 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:58.236111+0000 mon.b (mon.1) 465 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:59.457 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:58.236821+0000 mon.a (mon.0) 2056 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:59.457 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:58.465948+0000 osd.5 (osd.5) 120 : cluster [DBG] 16.c scrub starts 2024-02-19T21:07:59.457 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:58.468836+0000 osd.5 (osd.5) 121 : cluster [DBG] 16.c scrub ok 2024-02-19T21:07:59.457 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:58.499139+0000 osd.6 (osd.6) 119 : cluster [DBG] 16.1a scrub starts 2024-02-19T21:07:59.457 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:58.499837+0000 osd.6 (osd.6) 120 : cluster [DBG] 16.1a scrub ok 2024-02-19T21:07:59.457 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:59.174741+0000 mon.b (mon.1) 466 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "34"}]: dispatch 2024-02-19T21:07:59.458 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:07:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:59.175657+0000 mon.a (mon.0) 2057 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "34"}]: dispatch 2024-02-19T21:07:59.470 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:07:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:59.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:07:59.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:58.215591+0000 mon.a (mon.0) 2051 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:59.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:58.228492+0000 mon.a (mon.0) 2052 : audit [INF] from='client.49514 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi163-83854-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:59.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:58.228572+0000 mon.a (mon.0) 2053 : audit [INF] from='client.48343 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:07:59.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:58.228625+0000 mon.a (mon.0) 2054 : cluster [DBG] osdmap e298: 8 total, 8 up, 7 in 2024-02-19T21:07:59.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:58.229203+0000 mon.c (mon.2) 340 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi163-83854-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:59.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:58.229884+0000 mon.a (mon.0) 2055 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi163-83854-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:59.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:58.236111+0000 mon.b (mon.1) 465 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:59.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:58.236821+0000 mon.a (mon.0) 2056 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:59.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:58.465948+0000 osd.5 (osd.5) 120 : cluster [DBG] 16.c scrub starts 2024-02-19T21:07:59.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:58.468836+0000 osd.5 (osd.5) 121 : cluster [DBG] 16.c scrub ok 2024-02-19T21:07:59.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:58.499139+0000 osd.6 (osd.6) 119 : cluster [DBG] 16.1a scrub starts 2024-02-19T21:07:59.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:58.499837+0000 osd.6 (osd.6) 120 : cluster [DBG] 16.1a scrub ok 2024-02-19T21:07:59.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:59.174741+0000 mon.b (mon.1) 466 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "34"}]: dispatch 2024-02-19T21:07:59.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:59.175657+0000 mon.a (mon.0) 2057 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "34"}]: dispatch 2024-02-19T21:07:59.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:58.215591+0000 mon.a (mon.0) 2051 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:07:59.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:58.228492+0000 mon.a (mon.0) 2052 : audit [INF] from='client.49514 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi163-83854-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:07:59.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:58.228572+0000 mon.a (mon.0) 2053 : audit [INF] from='client.48343 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:07:59.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:58.228625+0000 mon.a (mon.0) 2054 : cluster [DBG] osdmap e298: 8 total, 8 up, 7 in 2024-02-19T21:07:59.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:58.229203+0000 mon.c (mon.2) 340 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi163-83854-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:59.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:58.229884+0000 mon.a (mon.0) 2055 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi163-83854-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:07:59.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:58.236111+0000 mon.b (mon.1) 465 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:59.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:58.236821+0000 mon.a (mon.0) 2056 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:07:59.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:58.465948+0000 osd.5 (osd.5) 120 : cluster [DBG] 16.c scrub starts 2024-02-19T21:07:59.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:58.468836+0000 osd.5 (osd.5) 121 : cluster [DBG] 16.c scrub ok 2024-02-19T21:07:59.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:58.499139+0000 osd.6 (osd.6) 119 : cluster [DBG] 16.1a scrub starts 2024-02-19T21:07:59.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:58.499837+0000 osd.6 (osd.6) 120 : cluster [DBG] 16.1a scrub ok 2024-02-19T21:07:59.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:59.174741+0000 mon.b (mon.1) 466 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "34"}]: dispatch 2024-02-19T21:07:59.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:07:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:59.175657+0000 mon.a (mon.0) 2057 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "34"}]: dispatch 2024-02-19T21:07:59.574 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:07:59.600 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:59.621 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:07:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:59.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:07:59.726 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:07:59.755 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:07:59.932 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:07:59.935 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:07:59.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:07:59.964 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:07:59.964 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:08:00.036 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:08:00.067 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:00.086 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:08:00.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:00.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:08:00.190 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:00.218 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:00.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:58.846921+0000 osd.1 (osd.1) 143 : cluster [DBG] 16.14 scrub starts 2024-02-19T21:08:00.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:58.847910+0000 osd.1 (osd.1) 144 : cluster [DBG] 16.14 scrub ok 2024-02-19T21:08:00.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:59.174228+0000 mgr.y (mgr.14550) 295 : cluster [DBG] pgmap v419: 231 pgs: 32 unknown, 199 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:08:00.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:59.442850+0000 osd.5 (osd.5) 122 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:08:00.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:59.446259+0000 osd.5 (osd.5) 123 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:08:00.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:59.467326+0000 mon.a (mon.0) 2058 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:00.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:59.467469+0000 mon.a (mon.0) 2059 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "34"}]': finished 2024-02-19T21:08:00.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:59.467546+0000 mon.a (mon.0) 2060 : cluster [DBG] osdmap e299: 8 total, 8 up, 7 in 2024-02-19T21:08:00.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:59.468655+0000 mon.b (mon.1) 467 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:00.261 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:07:59.469001+0000 mon.a (mon.0) 2061 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:00.266 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:08:00.266 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:08:00.266 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-02-19T21:08:00.267 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd in 3 2024-02-19T21:08:00.350 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:00.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:00.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:00.485 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:00.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:58.846921+0000 osd.1 (osd.1) 143 : cluster [DBG] 16.14 scrub starts 2024-02-19T21:08:00.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:58.847910+0000 osd.1 (osd.1) 144 : cluster [DBG] 16.14 scrub ok 2024-02-19T21:08:00.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:59.174228+0000 mgr.y (mgr.14550) 295 : cluster [DBG] pgmap v419: 231 pgs: 32 unknown, 199 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:08:00.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:59.442850+0000 osd.5 (osd.5) 122 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:08:00.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:59.446259+0000 osd.5 (osd.5) 123 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:08:00.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:59.467326+0000 mon.a (mon.0) 2058 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:00.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:59.467469+0000 mon.a (mon.0) 2059 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "34"}]': finished 2024-02-19T21:08:00.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:59.467546+0000 mon.a (mon.0) 2060 : cluster [DBG] osdmap e299: 8 total, 8 up, 7 in 2024-02-19T21:08:00.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:59.468655+0000 mon.b (mon.1) 467 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:00.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:07:59.469001+0000 mon.a (mon.0) 2061 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:00.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:58.846921+0000 osd.1 (osd.1) 143 : cluster [DBG] 16.14 scrub starts 2024-02-19T21:08:00.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:58.847910+0000 osd.1 (osd.1) 144 : cluster [DBG] 16.14 scrub ok 2024-02-19T21:08:00.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:59.174228+0000 mgr.y (mgr.14550) 295 : cluster [DBG] pgmap v419: 231 pgs: 32 unknown, 199 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:08:00.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:59.442850+0000 osd.5 (osd.5) 122 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:08:00.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:59.446259+0000 osd.5 (osd.5) 123 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:08:00.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:59.467326+0000 mon.a (mon.0) 2058 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:00.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:59.467469+0000 mon.a (mon.0) 2059 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "34"}]': finished 2024-02-19T21:08:00.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:59.467546+0000 mon.a (mon.0) 2060 : cluster [DBG] osdmap e299: 8 total, 8 up, 7 in 2024-02-19T21:08:00.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:59.468655+0000 mon.b (mon.1) 467 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:00.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:07:59.469001+0000 mon.a (mon.0) 2061 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:00.532 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:00.567 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:08:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:08:00.654 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:08:00.683 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:00.685 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:00.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:00.700 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-19T21:08:00.787 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:00.813 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:00.832 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:00.835 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:00.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:00.936 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:00.964 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:00.989 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:00.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:00.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:01.019 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:08:01.097 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:01.108 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:08:01.129 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:01.382 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:01.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:59.830796+0000 osd.1 (osd.1) 145 : cluster [DBG] 4.f scrub starts 2024-02-19T21:08:01.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:07:59.834275+0000 osd.1 (osd.1) 146 : cluster [DBG] 4.f scrub ok 2024-02-19T21:08:01.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:00.241151+0000 mon.a (mon.0) 2062 : audit [INF] from='client.49514 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi163-83854-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi163-83854-53"}]': finished 2024-02-19T21:08:01.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:00.241257+0000 mon.a (mon.0) 2063 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:08:01.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:00.241313+0000 mon.a (mon.0) 2064 : cluster [DBG] osdmap e300: 8 total, 8 up, 7 in 2024-02-19T21:08:01.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:00.248781+0000 mon.b (mon.1) 468 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:08:01.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:00.249141+0000 mon.a (mon.0) 2065 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:08:01.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:00.264308+0000 mgr.y (mgr.14550) 296 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:01.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:00.483110+0000 osd.6 (osd.6) 121 : cluster [DBG] 61.1b scrub starts 2024-02-19T21:08:01.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:00.485082+0000 osd.6 (osd.6) 122 : cluster [DBG] 61.1b scrub ok 2024-02-19T21:08:01.386 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:01.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:01.387 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:59.830796+0000 osd.1 (osd.1) 145 : cluster [DBG] 4.f scrub starts 2024-02-19T21:08:01.387 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:07:59.834275+0000 osd.1 (osd.1) 146 : cluster [DBG] 4.f scrub ok 2024-02-19T21:08:01.387 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:00.241151+0000 mon.a (mon.0) 2062 : audit [INF] from='client.49514 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi163-83854-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi163-83854-53"}]': finished 2024-02-19T21:08:01.387 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:00.241257+0000 mon.a (mon.0) 2063 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:08:01.387 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:00.241313+0000 mon.a (mon.0) 2064 : cluster [DBG] osdmap e300: 8 total, 8 up, 7 in 2024-02-19T21:08:01.387 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:00.248781+0000 mon.b (mon.1) 468 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:08:01.387 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:00.249141+0000 mon.a (mon.0) 2065 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:08:01.387 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:00.264308+0000 mgr.y (mgr.14550) 296 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:01.387 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:00.483110+0000 osd.6 (osd.6) 121 : cluster [DBG] 61.1b scrub starts 2024-02-19T21:08:01.387 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:00.485082+0000 osd.6 (osd.6) 122 : cluster [DBG] 61.1b scrub ok 2024-02-19T21:08:01.487 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:01.514 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:01.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:59.830796+0000 osd.1 (osd.1) 145 : cluster [DBG] 4.f scrub starts 2024-02-19T21:08:01.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:07:59.834275+0000 osd.1 (osd.1) 146 : cluster [DBG] 4.f scrub ok 2024-02-19T21:08:01.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:00.241151+0000 mon.a (mon.0) 2062 : audit [INF] from='client.49514 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi163-83854-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi163-83854-53"}]': finished 2024-02-19T21:08:01.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:00.241257+0000 mon.a (mon.0) 2063 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:08:01.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:00.241313+0000 mon.a (mon.0) 2064 : cluster [DBG] osdmap e300: 8 total, 8 up, 7 in 2024-02-19T21:08:01.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:00.248781+0000 mon.b (mon.1) 468 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:08:01.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:00.249141+0000 mon.a (mon.0) 2065 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:08:01.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:00.264308+0000 mgr.y (mgr.14550) 296 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:01.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:00.483110+0000 osd.6 (osd.6) 121 : cluster [DBG] 61.1b scrub starts 2024-02-19T21:08:01.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:00.485082+0000 osd.6 (osd.6) 122 : cluster [DBG] 61.1b scrub ok 2024-02-19T21:08:02.129 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:02.253 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:00.863713+0000 osd.1 (osd.1) 147 : cluster [DBG] 61.3 scrub starts 2024-02-19T21:08:02.253 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:00.867232+0000 osd.1 (osd.1) 148 : cluster [DBG] 61.3 scrub ok 2024-02-19T21:08:02.253 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:01.174649+0000 mgr.y (mgr.14550) 297 : cluster [DBG] pgmap v422: 239 pgs: 8 unknown, 2 peering, 229 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:08:02.253 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:01.244849+0000 mon.a (mon.0) 2066 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:08:02.253 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:01.244939+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e301: 8 total, 8 up, 7 in 2024-02-19T21:08:02.253 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:01.251033+0000 mon.b (mon.1) 469 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:02.253 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:01.251455+0000 mon.a (mon.0) 2068 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:02.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.350+0000 7f2469253700 1 -- 172.21.15.163:0/4211146246 >> v1:172.21.15.163:6789/0 conn(0x7f24641042c0 legacy=0x7f2464106760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:02.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.350+0000 7f2469253700 1 -- 172.21.15.163:0/4211146246 shutdown_connections 2024-02-19T21:08:02.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.350+0000 7f2469253700 1 -- 172.21.15.163:0/4211146246 >> 172.21.15.163:0/4211146246 conn(0x7f24640f24a0 msgr2=0x7f24640f48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:02.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.350+0000 7f2469253700 1 -- 172.21.15.163:0/4211146246 shutdown_connections 2024-02-19T21:08:02.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.350+0000 7f2469253700 1 -- 172.21.15.163:0/4211146246 wait complete. 2024-02-19T21:08:02.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.350+0000 7f2469253700 1 librados: starting msgr at 2024-02-19T21:08:02.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.350+0000 7f2469253700 1 librados: starting objecter 2024-02-19T21:08:02.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.351+0000 7f2469253700 1 Processor -- start 2024-02-19T21:08:02.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.351+0000 7f2469253700 1 -- start start 2024-02-19T21:08:02.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.351+0000 7f2469253700 1 librados: setting wanted keys 2024-02-19T21:08:02.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.351+0000 7f2469253700 1 librados: calling monclient init 2024-02-19T21:08:02.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.351+0000 7f2469253700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2464117280 con 0x7f24641042c0 2024-02-19T21:08:02.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.351+0000 7f2469253700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2464118460 con 0x7f24640f8930 2024-02-19T21:08:02.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.351+0000 7f2469253700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2464119660 con 0x7f24640f94e0 2024-02-19T21:08:02.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.351+0000 7f246359e700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f24641042c0 0x7f2464115940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:60260/0 (socket says 172.21.15.163:60260) 2024-02-19T21:08:02.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.351+0000 7f246359e700 1 -- 172.21.15.163:0/3202755741 learned_addr learned my addr 172.21.15.163:0/3202755741 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:02.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.351+0000 7f246259c700 1 --1- 172.21.15.163:0/3202755741 >> v1:172.21.15.163:6790/0 conn(0x7f24640f94e0 0x7f24640ffab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:46992/0 (socket says 172.21.15.163:46992) 2024-02-19T21:08:02.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 10 client.?.objecter ms_handle_connect 0x7f24641042c0 2024-02-19T21:08:02.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:02.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 10 client.?.objecter ms_handle_connect 0x7f24640f94e0 2024-02-19T21:08:02.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:02.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367670155 0 0) 0x7f2464117280 con 0x7f24641042c0 2024-02-19T21:08:02.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f244c0033f0 con 0x7f24641042c0 2024-02-19T21:08:02.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391345485 0 0) 0x7f2464119660 con 0x7f24640f94e0 2024-02-19T21:08:02.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2464117280 con 0x7f24640f94e0 2024-02-19T21:08:02.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 10 client.?.objecter ms_handle_connect 0x7f24640f8930 2024-02-19T21:08:02.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:02.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519169975 0 0) 0x7f2464118460 con 0x7f24640f8930 2024-02-19T21:08:02.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2464119660 con 0x7f24640f8930 2024-02-19T21:08:02.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 864159718 0 0) 0x7f2464117280 con 0x7f24640f94e0 2024-02-19T21:08:02.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2464118460 con 0x7f24640f94e0 2024-02-19T21:08:02.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4130736672 0 0) 0x7f244c0033f0 con 0x7f24641042c0 2024-02-19T21:08:02.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2464117280 con 0x7f24641042c0 2024-02-19T21:08:02.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f24580019d0 con 0x7f24640f94e0 2024-02-19T21:08:02.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f245c002430 con 0x7f24641042c0 2024-02-19T21:08:02.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1404627511 0 0) 0x7f2464119660 con 0x7f24640f8930 2024-02-19T21:08:02.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f244c0033f0 con 0x7f24640f8930 2024-02-19T21:08:02.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2454001870 con 0x7f24640f8930 2024-02-19T21:08:02.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3380508174 0 0) 0x7f2464118460 con 0x7f24640f94e0 2024-02-19T21:08:02.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 >> v1:172.21.15.193:6789/0 conn(0x7f24640f8930 legacy=0x7f24640ff3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:02.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 >> v1:172.21.15.163:6789/0 conn(0x7f24641042c0 legacy=0x7f2464115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:02.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246411a860 con 0x7f24640f94e0 2024-02-19T21:08:02.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f2469253700 1 -- 172.21.15.163:0/3202755741 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2464119850 con 0x7f24640f94e0 2024-02-19T21:08:02.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f2469253700 10 client.49532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:02.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f2469253700 1 -- 172.21.15.163:0/3202755741 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2464119cc0 con 0x7f24640f94e0 2024-02-19T21:08:02.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.352+0000 7f2469253700 1 librados: init done 2024-02-19T21:08:02.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.353+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f2458002420 con 0x7f24640f94e0 2024-02-19T21:08:02.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.353+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f24580059e0 con 0x7f24640f94e0 2024-02-19T21:08:02.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.354+0000 7f2469253700 1 -- 172.21.15.163:0/3202755741 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2450004f20 con 0x7f24640f94e0 2024-02-19T21:08:02.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.354+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f2458018950 con 0x7f24640f94e0 2024-02-19T21:08:02.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.355+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(302..302 src has 43..302) v4 ==== 8334+0+0 (unknown 126628612 0 0) 0x7f2458078df0 con 0x7f24640f94e0 2024-02-19T21:08:02.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.355+0000 7f244bfff700 10 client.49532.objecter ms_dispatch 0x7f2464001550 osd_map(302..302 src has 43..302) v4 2024-02-19T21:08:02.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.355+0000 7f244bfff700 3 client.49532.objecter handle_osd_map got epochs [302,302] > 0 2024-02-19T21:08:02.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.355+0000 7f244bfff700 3 client.49532.objecter handle_osd_map decoding full epoch 302 2024-02-19T21:08:02.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.355+0000 7f244bfff700 20 client.49532.objecter dump_active .. 0 homeless 2024-02-19T21:08:02.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.355+0000 7f244bfff700 10 client.49532.objecter ms_handle_connect 0x7f244c063d30 2024-02-19T21:08:02.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.360+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f245804c0f0 con 0x7f24640f94e0 2024-02-19T21:08:02.362 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:02.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:02.384 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:02.467 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:02.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:00.863713+0000 osd.1 (osd.1) 147 : cluster [DBG] 61.3 scrub starts 2024-02-19T21:08:02.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:00.867232+0000 osd.1 (osd.1) 148 : cluster [DBG] 61.3 scrub ok 2024-02-19T21:08:02.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:01.174649+0000 mgr.y (mgr.14550) 297 : cluster [DBG] pgmap v422: 239 pgs: 8 unknown, 2 peering, 229 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:08:02.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:01.244849+0000 mon.a (mon.0) 2066 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:08:02.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:01.244939+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e301: 8 total, 8 up, 7 in 2024-02-19T21:08:02.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:01.251033+0000 mon.b (mon.1) 469 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:02.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:01.251455+0000 mon.a (mon.0) 2068 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:02.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:01.490781+0000 osd.5 (osd.5) 124 : cluster [DBG] 61.8 scrub starts 2024-02-19T21:08:02.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:01.494477+0000 osd.5 (osd.5) 125 : cluster [DBG] 61.8 scrub ok 2024-02-19T21:08:02.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:01.527723+0000 osd.6 (osd.6) 123 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:08:02.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:01.530614+0000 osd.6 (osd.6) 124 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:08:02.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:00.863713+0000 osd.1 (osd.1) 147 : cluster [DBG] 61.3 scrub starts 2024-02-19T21:08:02.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:00.867232+0000 osd.1 (osd.1) 148 : cluster [DBG] 61.3 scrub ok 2024-02-19T21:08:02.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:01.174649+0000 mgr.y (mgr.14550) 297 : cluster [DBG] pgmap v422: 239 pgs: 8 unknown, 2 peering, 229 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:08:02.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:01.244849+0000 mon.a (mon.0) 2066 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:08:02.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:01.244939+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e301: 8 total, 8 up, 7 in 2024-02-19T21:08:02.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:01.251033+0000 mon.b (mon.1) 469 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:02.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:01.251455+0000 mon.a (mon.0) 2068 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:02.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:01.490781+0000 osd.5 (osd.5) 124 : cluster [DBG] 61.8 scrub starts 2024-02-19T21:08:02.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:01.494477+0000 osd.5 (osd.5) 125 : cluster [DBG] 61.8 scrub ok 2024-02-19T21:08:02.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:01.527723+0000 osd.6 (osd.6) 123 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:08:02.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:01.530614+0000 osd.6 (osd.6) 124 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:08:02.496 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:02.505 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:01.490781+0000 osd.5 (osd.5) 124 : cluster [DBG] 61.8 scrub starts 2024-02-19T21:08:02.505 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:01.494477+0000 osd.5 (osd.5) 125 : cluster [DBG] 61.8 scrub ok 2024-02-19T21:08:02.505 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:01.527723+0000 osd.6 (osd.6) 123 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:08:02.505 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:01.530614+0000 osd.6 (osd.6) 124 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:08:02.518 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:02.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:02.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:02.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:02.552+0000 7f2469253700 1 -- 172.21.15.163:0/3202755741 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd in", "ids": ["3"]} v 0) v1 -- 0x7f24500051d0 con 0x7f24640f94e0 2024-02-19T21:08:02.622 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:08:02.652 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:02.770 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:08:02.770 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:08:02.834 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:08:02.552+0000 7fb606608700 -1 mon.a@0(leader).osd e302 definitely_dead 0 2024-02-19T21:08:03.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:03.264+0000 7f244bfff700 1 -- 172.21.15.163:0/3202755741 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd in", "ids": ["3"]}]=0 marked in osd.3. v303) v1 ==== 85+0+0 (unknown 1156804471 0 0) 0x7f245804a500 con 0x7f24640f94e0 2024-02-19T21:08:03.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:03.269+0000 7f2469253700 10 librados: watch_flush enter 2024-02-19T21:08:03.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:03.269+0000 7f2469253700 10 librados: watch_flush exit 2024-02-19T21:08:03.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:03.269+0000 7f2469253700 20 client.49532.objecter shutdown clearing up homeless session... 2024-02-19T21:08:03.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:03.269+0000 7f2469253700 10 client.49532.objecter successfully canceled tick 2024-02-19T21:08:03.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:03.273+0000 7f2469253700 1 -- 172.21.15.163:0/3202755741 >> v1:172.21.15.163:6800/1552514681 conn(0x7f244c063d30 legacy=0x7f244c0661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:03.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:03.273+0000 7f2469253700 1 -- 172.21.15.163:0/3202755741 >> v1:172.21.15.163:6790/0 conn(0x7f24640f94e0 legacy=0x7f24640ffab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:03.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:03.276+0000 7f2469253700 1 -- 172.21.15.163:0/3202755741 shutdown_connections 2024-02-19T21:08:03.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:03.276+0000 7f2469253700 1 -- 172.21.15.163:0/3202755741 >> 172.21.15.163:0/3202755741 conn(0x7f24640f24a0 msgr2=0x7f2464103980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:03.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:03.279+0000 7f2469253700 1 -- 172.21.15.163:0/3202755741 shutdown_connections 2024-02-19T21:08:03.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:03.281+0000 7f2469253700 1 -- 172.21.15.163:0/3202755741 wait complete. 2024-02-19T21:08:03.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:03.282+0000 7f2469253700 1 librados: shutdown 2024-02-19T21:08:03.282 INFO:teuthology.orchestra.run.smithi163.stderr:marked in osd.3. 2024-02-19T21:08:03.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:01.605464+0000 osd.4 (osd.4) 133 : cluster [DBG] 61.a scrub starts 2024-02-19T21:08:03.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:01.607112+0000 osd.4 (osd.4) 134 : cluster [DBG] 61.a scrub ok 2024-02-19T21:08:03.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:02.251106+0000 mon.a (mon.0) 2069 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:08:03.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:02.251200+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e302: 8 total, 8 up, 7 in 2024-02-19T21:08:03.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:02.254092+0000 mon.c (mon.2) 341 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:08:03.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:02.254682+0000 mon.b (mon.1) 470 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:08:03.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:02.259888+0000 mon.a (mon.0) 2071 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:08:03.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:02.260611+0000 mon.a (mon.0) 2072 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:08:03.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:02.553466+0000 mon.c (mon.2) 342 : audit [DBG] from='client.? v1:172.21.15.163:0/3202755741' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-19T21:08:03.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:02.553757+0000 mon.a (mon.0) 2073 : audit [DBG] from='client.49532 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-19T21:08:03.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:03.175702+0000 mon.b (mon.1) 471 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "33"}]: dispatch 2024-02-19T21:08:03.371 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:03.176704+0000 mon.a (mon.0) 2074 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "33"}]: dispatch 2024-02-19T21:08:03.390 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-19T21:08:03.648 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:08:03] "GET /metrics HTTP/1.1" 200 190662 "" "Prometheus/2.33.4" 2024-02-19T21:08:03.648 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:01.605464+0000 osd.4 (osd.4) 133 : cluster [DBG] 61.a scrub starts 2024-02-19T21:08:03.648 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:01.607112+0000 osd.4 (osd.4) 134 : cluster [DBG] 61.a scrub ok 2024-02-19T21:08:03.648 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:02.251106+0000 mon.a (mon.0) 2069 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:08:03.648 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:02.251200+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e302: 8 total, 8 up, 7 in 2024-02-19T21:08:03.648 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:02.254092+0000 mon.c (mon.2) 341 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:08:03.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:02.254682+0000 mon.b (mon.1) 470 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:08:03.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:02.259888+0000 mon.a (mon.0) 2071 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:08:03.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:02.260611+0000 mon.a (mon.0) 2072 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:08:03.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:02.553466+0000 mon.c (mon.2) 342 : audit [DBG] from='client.? v1:172.21.15.163:0/3202755741' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-19T21:08:03.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:02.553757+0000 mon.a (mon.0) 2073 : audit [DBG] from='client.49532 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-19T21:08:03.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:03.175702+0000 mon.b (mon.1) 471 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "33"}]: dispatch 2024-02-19T21:08:03.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:03.176704+0000 mon.a (mon.0) 2074 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "33"}]: dispatch 2024-02-19T21:08:03.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:01.605464+0000 osd.4 (osd.4) 133 : cluster [DBG] 61.a scrub starts 2024-02-19T21:08:03.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:01.607112+0000 osd.4 (osd.4) 134 : cluster [DBG] 61.a scrub ok 2024-02-19T21:08:03.650 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:02.251106+0000 mon.a (mon.0) 2069 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:08:03.650 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:02.251200+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e302: 8 total, 8 up, 7 in 2024-02-19T21:08:03.650 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:02.254092+0000 mon.c (mon.2) 341 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:08:03.650 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:02.254682+0000 mon.b (mon.1) 470 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:08:03.650 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:02.259888+0000 mon.a (mon.0) 2071 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:08:03.650 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:02.260611+0000 mon.a (mon.0) 2072 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:08:03.650 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:02.553466+0000 mon.c (mon.2) 342 : audit [DBG] from='client.? v1:172.21.15.163:0/3202755741' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-19T21:08:03.650 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:02.553757+0000 mon.a (mon.0) 2073 : audit [DBG] from='client.49532 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-19T21:08:03.650 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:03.175702+0000 mon.b (mon.1) 471 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "33"}]: dispatch 2024-02-19T21:08:03.650 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:03.176704+0000 mon.a (mon.0) 2074 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "33"}]: dispatch 2024-02-19T21:08:03.659 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:08:04.003 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:08:04.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:04.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:08:04.016 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-02-19T21:08:04.106 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:08:04.134 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:04.162 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:08:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:04.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:08:04.268 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:04.302 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:04.451 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:04.454 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:04.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:04.555 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:04.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:02.857985+0000 osd.7 (osd.7) 133 : cluster [DBG] 61.1a scrub starts 2024-02-19T21:08:04.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:02.861463+0000 osd.7 (osd.7) 134 : cluster [DBG] 61.1a scrub ok 2024-02-19T21:08:04.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:03.175223+0000 mgr.y (mgr.14550) 298 : cluster [DBG] pgmap v425: 230 pgs: 1 peering, 229 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:08:04.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:03.176888+0000 osd.2 (osd.2) 113 : cluster [DBG] 16.1d scrub starts 2024-02-19T21:08:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:03.177863+0000 osd.2 (osd.2) 114 : cluster [DBG] 16.1d scrub ok 2024-02-19T21:08:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:03.263406+0000 mon.a (mon.0) 2075 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-19T21:08:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:03.263485+0000 mon.a (mon.0) 2076 : audit [INF] from='client.49514 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi163-83854-53"}]': finished 2024-02-19T21:08:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:03.263547+0000 mon.a (mon.0) 2077 : audit [INF] from='client.49532 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-19T21:08:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:03.263603+0000 mon.a (mon.0) 2078 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "33"}]': finished 2024-02-19T21:08:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:03.263652+0000 mon.a (mon.0) 2079 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-02-19T21:08:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:03.266265+0000 mon.c (mon.2) 343 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:08:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:03.266678+0000 mon.a (mon.0) 2080 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:08:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:03.271829+0000 osd.6 (osd.6) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:03.272060+0000 osd.6 (osd.6) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:03.456004+0000 osd.5 (osd.5) 126 : cluster [DBG] 61.7 scrub starts 2024-02-19T21:08:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:03.459475+0000 osd.5 (osd.5) 127 : cluster [DBG] 61.7 scrub ok 2024-02-19T21:08:04.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:03.559401+0000 osd.6 (osd.6) 127 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:08:04.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:03.561219+0000 osd.6 (osd.6) 128 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:08:04.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:04.211046+0000 mon.a (mon.0) 2081 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:08:04.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:04.219369+0000 mon.a (mon.0) 2082 : audit [INF] from='client.49514 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi163-83854-53"}]': finished 2024-02-19T21:08:04.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:04.219509+0000 mon.a (mon.0) 2083 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-02-19T21:08:04.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:02.857985+0000 osd.7 (osd.7) 133 : cluster [DBG] 61.1a scrub starts 2024-02-19T21:08:04.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:02.861463+0000 osd.7 (osd.7) 134 : cluster [DBG] 61.1a scrub ok 2024-02-19T21:08:04.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:03.175223+0000 mgr.y (mgr.14550) 298 : cluster [DBG] pgmap v425: 230 pgs: 1 peering, 229 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:08:04.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:03.176888+0000 osd.2 (osd.2) 113 : cluster [DBG] 16.1d scrub starts 2024-02-19T21:08:04.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:03.177863+0000 osd.2 (osd.2) 114 : cluster [DBG] 16.1d scrub ok 2024-02-19T21:08:04.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:03.263406+0000 mon.a (mon.0) 2075 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-19T21:08:04.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:03.263485+0000 mon.a (mon.0) 2076 : audit [INF] from='client.49514 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi163-83854-53"}]': finished 2024-02-19T21:08:04.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:03.263547+0000 mon.a (mon.0) 2077 : audit [INF] from='client.49532 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-19T21:08:04.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:03.263603+0000 mon.a (mon.0) 2078 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "33"}]': finished 2024-02-19T21:08:04.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:03.263652+0000 mon.a (mon.0) 2079 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-02-19T21:08:04.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:03.266265+0000 mon.c (mon.2) 343 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:08:04.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:03.266678+0000 mon.a (mon.0) 2080 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:08:04.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:03.271829+0000 osd.6 (osd.6) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:04.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:03.272060+0000 osd.6 (osd.6) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:04.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:03.456004+0000 osd.5 (osd.5) 126 : cluster [DBG] 61.7 scrub starts 2024-02-19T21:08:04.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:03.459475+0000 osd.5 (osd.5) 127 : cluster [DBG] 61.7 scrub ok 2024-02-19T21:08:04.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:03.559401+0000 osd.6 (osd.6) 127 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:08:04.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:03.561219+0000 osd.6 (osd.6) 128 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:08:04.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:04.211046+0000 mon.a (mon.0) 2081 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:08:04.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:04.219369+0000 mon.a (mon.0) 2082 : audit [INF] from='client.49514 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi163-83854-53"}]': finished 2024-02-19T21:08:04.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:04.219509+0000 mon.a (mon.0) 2083 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-02-19T21:08:04.582 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:02.857985+0000 osd.7 (osd.7) 133 : cluster [DBG] 61.1a scrub starts 2024-02-19T21:08:04.582 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:02.861463+0000 osd.7 (osd.7) 134 : cluster [DBG] 61.1a scrub ok 2024-02-19T21:08:04.582 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:03.175223+0000 mgr.y (mgr.14550) 298 : cluster [DBG] pgmap v425: 230 pgs: 1 peering, 229 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:08:04.582 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:03.176888+0000 osd.2 (osd.2) 113 : cluster [DBG] 16.1d scrub starts 2024-02-19T21:08:04.582 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:03.177863+0000 osd.2 (osd.2) 114 : cluster [DBG] 16.1d scrub ok 2024-02-19T21:08:04.582 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:03.263406+0000 mon.a (mon.0) 2075 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-39","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-19T21:08:04.582 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:03.263485+0000 mon.a (mon.0) 2076 : audit [INF] from='client.49514 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi163-83854-53"}]': finished 2024-02-19T21:08:04.582 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:03.263547+0000 mon.a (mon.0) 2077 : audit [INF] from='client.49532 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-19T21:08:04.582 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:03.263603+0000 mon.a (mon.0) 2078 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "33"}]': finished 2024-02-19T21:08:04.582 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:03.263652+0000 mon.a (mon.0) 2079 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-02-19T21:08:04.582 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:03.266265+0000 mon.c (mon.2) 343 : audit [INF] from='client.? v1:172.21.15.163:0/1357696765' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:08:04.582 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:03.266678+0000 mon.a (mon.0) 2080 : audit [INF] from='client.49514 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi163-83854-53"}]: dispatch 2024-02-19T21:08:04.582 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:03.271829+0000 osd.6 (osd.6) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:04.583 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:03.272060+0000 osd.6 (osd.6) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:04.583 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:03.456004+0000 osd.5 (osd.5) 126 : cluster [DBG] 61.7 scrub starts 2024-02-19T21:08:04.583 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:03.459475+0000 osd.5 (osd.5) 127 : cluster [DBG] 61.7 scrub ok 2024-02-19T21:08:04.583 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:03.559401+0000 osd.6 (osd.6) 127 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:08:04.583 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:03.561219+0000 osd.6 (osd.6) 128 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:08:04.583 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:04.211046+0000 mon.a (mon.0) 2081 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:08:04.584 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:04.219369+0000 mon.a (mon.0) 2082 : audit [INF] from='client.49514 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi163-83854-53"}]': finished 2024-02-19T21:08:04.584 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:04.219509+0000 mon.a (mon.0) 2083 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-02-19T21:08:04.584 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:04.604 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:04.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:04.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:04.707 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:04.734 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:04.932 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:04.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:04.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:05.035 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:05.061 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:05.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.152+0000 7fd7b84be700 1 -- 172.21.15.163:0/26545758 >> v1:172.21.15.163:6790/0 conn(0x7fd7b01042b0 legacy=0x7fd7b0106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:05.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.152+0000 7fd7b84be700 1 -- 172.21.15.163:0/26545758 shutdown_connections 2024-02-19T21:08:05.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.152+0000 7fd7b84be700 1 -- 172.21.15.163:0/26545758 >> 172.21.15.163:0/26545758 conn(0x7fd7b00f2470 msgr2=0x7fd7b00f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:05.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.152+0000 7fd7b84be700 1 -- 172.21.15.163:0/26545758 shutdown_connections 2024-02-19T21:08:05.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.152+0000 7fd7b84be700 1 -- 172.21.15.163:0/26545758 wait complete. 2024-02-19T21:08:05.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7b84be700 1 librados: starting msgr at 2024-02-19T21:08:05.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7b84be700 1 librados: starting objecter 2024-02-19T21:08:05.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7b84be700 1 Processor -- start 2024-02-19T21:08:05.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7b84be700 1 -- start start 2024-02-19T21:08:05.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7b84be700 1 librados: setting wanted keys 2024-02-19T21:08:05.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7b84be700 1 librados: calling monclient init 2024-02-19T21:08:05.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7b84be700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd7b01173a0 con 0x7fd7b00f94d0 2024-02-19T21:08:05.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7b84be700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd7b0118580 con 0x7fd7b00f8920 2024-02-19T21:08:05.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7b84be700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd7b0119780 con 0x7fd7b01042b0 2024-02-19T21:08:05.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7b6a5b700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fd7b01042b0 0x7fd7b0115a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:39086/0 (socket says 172.21.15.163:39086) 2024-02-19T21:08:05.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7b6a5b700 1 -- 172.21.15.163:0/3109614039 learned_addr learned my addr 172.21.15.163:0/3109614039 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:05.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7a77fe700 10 client.?.objecter ms_handle_connect 0x7fd7b00f94d0 2024-02-19T21:08:05.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7a77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:05.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7a77fe700 10 client.?.objecter ms_handle_connect 0x7fd7b01042b0 2024-02-19T21:08:05.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7a77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:05.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7a77fe700 10 client.?.objecter ms_handle_connect 0x7fd7b00f8920 2024-02-19T21:08:05.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.153+0000 7fd7a77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:05.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738673777 0 0) 0x7fd7b0118580 con 0x7fd7b00f8920 2024-02-19T21:08:05.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd79c0033f0 con 0x7fd7b00f8920 2024-02-19T21:08:05.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109235740 0 0) 0x7fd7b01173a0 con 0x7fd7b00f94d0 2024-02-19T21:08:05.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7b0118580 con 0x7fd7b00f94d0 2024-02-19T21:08:05.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999545724 0 0) 0x7fd7b0119780 con 0x7fd7b01042b0 2024-02-19T21:08:05.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7b01173a0 con 0x7fd7b01042b0 2024-02-19T21:08:05.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 499902744 0 0) 0x7fd79c0033f0 con 0x7fd7b00f8920 2024-02-19T21:08:05.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7b0119780 con 0x7fd7b00f8920 2024-02-19T21:08:05.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd7ac0019d0 con 0x7fd7b00f8920 2024-02-19T21:08:05.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2137050155 0 0) 0x7fd7b01173a0 con 0x7fd7b01042b0 2024-02-19T21:08:05.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd79c0033f0 con 0x7fd7b01042b0 2024-02-19T21:08:05.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2514969984 0 0) 0x7fd7b0118580 con 0x7fd7b00f94d0 2024-02-19T21:08:05.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7b01173a0 con 0x7fd7b00f94d0 2024-02-19T21:08:05.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd7a8002310 con 0x7fd7b01042b0 2024-02-19T21:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd7a0001840 con 0x7fd7b00f94d0 2024-02-19T21:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4010133302 0 0) 0x7fd7b0119780 con 0x7fd7b00f8920 2024-02-19T21:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 >> v1:172.21.15.163:6790/0 conn(0x7fd7b01042b0 legacy=0x7fd7b0115a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 >> v1:172.21.15.163:6789/0 conn(0x7fd7b00f94d0 legacy=0x7fd7b00ffda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b011a980 con 0x7fd7b00f8920 2024-02-19T21:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7b84be700 1 -- 172.21.15.163:0/3109614039 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b0119970 con 0x7fd7b00f8920 2024-02-19T21:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7b84be700 10 client.48355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7b84be700 1 -- 172.21.15.163:0/3109614039 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b0119de0 con 0x7fd7b00f8920 2024-02-19T21:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7b84be700 1 librados: init done 2024-02-19T21:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fd7ac002340 con 0x7fd7b00f8920 2024-02-19T21:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.154+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd7ac001600 con 0x7fd7b00f8920 2024-02-19T21:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.155+0000 7fd7b84be700 1 -- 172.21.15.163:0/3109614039 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd794004f20 con 0x7fd7b00f8920 2024-02-19T21:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.156+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fd7ac001ea0 con 0x7fd7b00f8920 2024-02-19T21:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.156+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(304..304 src has 43..304) v4 ==== 7972+0+0 (unknown 3488927609 0 0) 0x7fd7ac076260 con 0x7fd7b00f8920 2024-02-19T21:08:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.156+0000 7fd7a77fe700 10 client.48355.objecter ms_dispatch 0x7fd7b0001550 osd_map(304..304 src has 43..304) v4 2024-02-19T21:08:05.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.156+0000 7fd7a77fe700 3 client.48355.objecter handle_osd_map got epochs [304,304] > 0 2024-02-19T21:08:05.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.156+0000 7fd7a77fe700 3 client.48355.objecter handle_osd_map decoding full epoch 304 2024-02-19T21:08:05.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.156+0000 7fd7a77fe700 20 client.48355.objecter dump_active .. 0 homeless 2024-02-19T21:08:05.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.156+0000 7fd7a77fe700 10 client.48355.objecter ms_handle_connect 0x7fd79c063e20 2024-02-19T21:08:05.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.160+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd7ac0496f0 con 0x7fd7b00f8920 2024-02-19T21:08:05.259 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:05.262 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:05.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:05.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:05.355+0000 7fd7b84be700 1 -- 172.21.15.163:0/3109614039 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fd7940051d0 con 0x7fd7b00f8920 2024-02-19T21:08:05.364 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:05.392 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:05.494 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:05.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:05.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:05.539 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:08:05.540 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:08:05.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:04.277037+0000 mon.a (mon.0) 2084 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi163-83854-54"}]: dispatch 2024-02-19T21:08:05.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:04.279135+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi163-83854-54"}]: dispatch 2024-02-19T21:08:05.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:04.279513+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi163-83854-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:05.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:04.496054+0000 osd.4 (osd.4) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:05.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:04.496293+0000 osd.4 (osd.4) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:05.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:05.240560+0000 mon.a (mon.0) 2087 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi163-83854-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:05.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:05.240631+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-02-19T21:08:05.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:05.242260+0000 mon.a (mon.0) 2089 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi163-83854-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi163-83854-54"}]: dispatch 2024-02-19T21:08:05.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:05.296714+0000 mon.b (mon.1) 472 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:05.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:05.297086+0000 mon.a (mon.0) 2090 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:05.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:05.297604+0000 mon.b (mon.1) 473 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39"}]: dispatch 2024-02-19T21:08:05.561 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:05.297839+0000 mon.a (mon.0) 2091 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39"}]: dispatch 2024-02-19T21:08:05.587 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:04.277037+0000 mon.a (mon.0) 2084 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi163-83854-54"}]: dispatch 2024-02-19T21:08:05.588 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:04.279135+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi163-83854-54"}]: dispatch 2024-02-19T21:08:05.588 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:04.279513+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi163-83854-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:05.588 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:04.496054+0000 osd.4 (osd.4) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:05.588 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:04.496293+0000 osd.4 (osd.4) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:05.588 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:05.240560+0000 mon.a (mon.0) 2087 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi163-83854-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:05.589 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:05.240631+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-02-19T21:08:05.589 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:05.242260+0000 mon.a (mon.0) 2089 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi163-83854-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi163-83854-54"}]: dispatch 2024-02-19T21:08:05.589 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:05.296714+0000 mon.b (mon.1) 472 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:05.589 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:05.297086+0000 mon.a (mon.0) 2090 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:05.589 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:05.297604+0000 mon.b (mon.1) 473 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39"}]: dispatch 2024-02-19T21:08:05.589 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:05.297839+0000 mon.a (mon.0) 2091 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39"}]: dispatch 2024-02-19T21:08:05.590 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:04.277037+0000 mon.a (mon.0) 2084 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi163-83854-54"}]: dispatch 2024-02-19T21:08:05.590 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:04.279135+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi163-83854-54"}]: dispatch 2024-02-19T21:08:05.590 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:04.279513+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi163-83854-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:05.590 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:04.496054+0000 osd.4 (osd.4) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:05.591 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:04.496293+0000 osd.4 (osd.4) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:05.591 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:05.240560+0000 mon.a (mon.0) 2087 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi163-83854-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:05.591 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:05.240631+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-02-19T21:08:05.591 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:05.242260+0000 mon.a (mon.0) 2089 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi163-83854-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi163-83854-54"}]: dispatch 2024-02-19T21:08:05.591 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:05.296714+0000 mon.b (mon.1) 472 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:05.591 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:05.297086+0000 mon.a (mon.0) 2090 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:05.592 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:05.297604+0000 mon.b (mon.1) 473 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39"}]: dispatch 2024-02-19T21:08:05.592 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:05.297839+0000 mon.a (mon.0) 2091 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39"}]: dispatch 2024-02-19T21:08:05.600 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:08:05.629 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:06.280 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:08:06.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:06.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:08:06.332 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-19T21:08:06.385 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:06.411 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:06.435 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:06.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:06.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:06.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.487+0000 7fd7a77fe700 1 -- 172.21.15.163:0/3109614039 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v306) v1 ==== 101+0+0 (unknown 1624624873 0 0) 0x7fd7ac047b00 con 0x7fd7b00f8920 2024-02-19T21:08:06.487 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.491+0000 7fd7b84be700 10 librados: watch_flush enter 2024-02-19T21:08:06.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.491+0000 7fd7b84be700 10 librados: watch_flush exit 2024-02-19T21:08:06.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.491+0000 7fd7b84be700 20 client.48355.objecter shutdown clearing up homeless session... 2024-02-19T21:08:06.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.491+0000 7fd7b84be700 10 client.48355.objecter successfully canceled tick 2024-02-19T21:08:06.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.491+0000 7fd7b84be700 1 -- 172.21.15.163:0/3109614039 >> v1:172.21.15.163:6800/1552514681 conn(0x7fd79c063e20 legacy=0x7fd79c0662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:06.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.491+0000 7fd7b84be700 1 -- 172.21.15.163:0/3109614039 >> v1:172.21.15.193:6789/0 conn(0x7fd7b00f8920 legacy=0x7fd7b00ff690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:06.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.491+0000 7fd7b84be700 1 -- 172.21.15.163:0/3109614039 shutdown_connections 2024-02-19T21:08:06.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.491+0000 7fd7b84be700 1 -- 172.21.15.163:0/3109614039 >> 172.21.15.163:0/3109614039 conn(0x7fd7b00f2470 msgr2=0x7fd7b0107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:06.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.491+0000 7fd7b84be700 1 -- 172.21.15.163:0/3109614039 shutdown_connections 2024-02-19T21:08:06.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.491+0000 7fd7b84be700 1 -- 172.21.15.163:0/3109614039 wait complete. 2024-02-19T21:08:06.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.491+0000 7fd7b84be700 1 librados: shutdown 2024-02-19T21:08:06.489 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:08:06.538 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.543+0000 7f181fe18700 1 -- 172.21.15.163:0/3743800162 >> v1:172.21.15.193:6789/0 conn(0x7f18180f94d0 legacy=0x7f18181029a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.543+0000 7f181fe18700 1 -- 172.21.15.163:0/3743800162 shutdown_connections 2024-02-19T21:08:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.543+0000 7f181fe18700 1 -- 172.21.15.163:0/3743800162 >> 172.21.15.163:0/3743800162 conn(0x7f18180f2450 msgr2=0x7f18180f4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.543+0000 7f181fe18700 1 -- 172.21.15.163:0/3743800162 shutdown_connections 2024-02-19T21:08:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.543+0000 7f181fe18700 1 -- 172.21.15.163:0/3743800162 wait complete. 2024-02-19T21:08:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.544+0000 7f181fe18700 1 librados: starting msgr at 2024-02-19T21:08:06.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.544+0000 7f181fe18700 1 librados: starting objecter 2024-02-19T21:08:06.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.544+0000 7f181fe18700 1 Processor -- start 2024-02-19T21:08:06.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.544+0000 7f181fe18700 1 -- start start 2024-02-19T21:08:06.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.544+0000 7f181fe18700 1 librados: setting wanted keys 2024-02-19T21:08:06.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.544+0000 7f181fe18700 1 librados: calling monclient init 2024-02-19T21:08:06.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.544+0000 7f181fe18700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f181810f980 con 0x7f18181042b0 2024-02-19T21:08:06.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.544+0000 7f181fe18700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f181811ec30 con 0x7f18180f8920 2024-02-19T21:08:06.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.544+0000 7f181fe18700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f181811fdd0 con 0x7f18180f94d0 2024-02-19T21:08:06.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.544+0000 7f181dbb4700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f18180f8920 0x7f1818102270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:59300/0 (socket says 172.21.15.163:59300) 2024-02-19T21:08:06.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.544+0000 7f181dbb4700 1 -- 172.21.15.163:0/2856516292 learned_addr learned my addr 172.21.15.163:0/2856516292 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:06.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f180effd700 10 client.?.objecter ms_handle_connect 0x7f18180f8920 2024-02-19T21:08:06.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f180effd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:06.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f180effd700 10 client.?.objecter ms_handle_connect 0x7f18181042b0 2024-02-19T21:08:06.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f180effd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:06.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110565261 0 0) 0x7f181811ec30 con 0x7f18180f8920 2024-02-19T21:08:06.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18040038e0 con 0x7f18180f8920 2024-02-19T21:08:06.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4149409364 0 0) 0x7f18040038e0 con 0x7f18180f8920 2024-02-19T21:08:06.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f181811ec30 con 0x7f18180f8920 2024-02-19T21:08:06.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1814001a30 con 0x7f18180f8920 2024-02-19T21:08:06.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803172451 0 0) 0x7f181810f980 con 0x7f18181042b0 2024-02-19T21:08:06.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18040038e0 con 0x7f18181042b0 2024-02-19T21:08:06.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 292009833 0 0) 0x7f181811ec30 con 0x7f18180f8920 2024-02-19T21:08:06.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 >> v1:172.21.15.163:6790/0 conn(0x7f18180f94d0 legacy=0x7f1818102980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:06.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 >> v1:172.21.15.163:6789/0 conn(0x7f18181042b0 legacy=0x7f181811c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:06.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181810fc10 con 0x7f18180f8920 2024-02-19T21:08:06.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181811dc80 con 0x7f18180f8920 2024-02-19T21:08:06.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f181fe18700 10 client.48361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:06.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f181811e0f0 con 0x7f18180f8920 2024-02-19T21:08:06.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.545+0000 7f181fe18700 1 librados: init done 2024-02-19T21:08:06.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.546+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f181805f540 con 0x7f18180f8920 2024-02-19T21:08:06.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.546+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f1814002500 con 0x7f18180f8920 2024-02-19T21:08:06.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.546+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1814005bf0 con 0x7f18180f8920 2024-02-19T21:08:06.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.547+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f1814018b80 con 0x7f18180f8920 2024-02-19T21:08:06.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.548+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(306..306 src has 43..306) v4 ==== 8351+0+0 (unknown 2874169179 0 0) 0x7f1814079e90 con 0x7f18180f8920 2024-02-19T21:08:06.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.548+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 osd_map(306..306 src has 43..306) v4 2024-02-19T21:08:06.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.548+0000 7f180effd700 3 client.48361.objecter handle_osd_map got epochs [306,306] > 0 2024-02-19T21:08:06.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.548+0000 7f180effd700 3 client.48361.objecter handle_osd_map decoding full epoch 306 2024-02-19T21:08:06.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.548+0000 7f180effd700 20 client.48361.objecter dump_active .. 0 homeless 2024-02-19T21:08:06.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.548+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v306) v1 ==== 54+0+16 (unknown 1403654055 0 2420000192) 0x7f18140020c0 con 0x7f18180f8920 2024-02-19T21:08:06.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.548+0000 7f180effd700 10 client.48361.objecter ms_handle_connect 0x7f1804063c50 2024-02-19T21:08:06.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.549+0000 7f181fe18700 10 client.48361.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:06.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.549+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 1 2024-02-19T21:08:06.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.549+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818047540 osd=0 3 2024-02-19T21:08:06.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.549+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818047540 osd=0 3 2024-02-19T21:08:06.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.549+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818047540 osd=0 3 2024-02-19T21:08:06.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.549+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 1 2024-02-19T21:08:06.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.549+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f1818047540 osd=0 3 2024-02-19T21:08:06.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.549+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 0 1 2024-02-19T21:08:06.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.549+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818047540 osd=0 4 2024-02-19T21:08:06.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.549+0000 7f181fe18700 10 client.48361.objecter _send_command 1 2024-02-19T21:08:06.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.549+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1818109f30 con 0x7f181811ff60 2024-02-19T21:08:06.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.549+0000 7f180effd700 10 client.48361.objecter ms_handle_connect 0x7f181811ff60 2024-02-19T21:08:06.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.550+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1818109f30 con 0x7f181811ff60 2024-02-19T21:08:06.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.550+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:08:06.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.550+0000 7f180effd700 10 client.48361.objecter _finish_command 1 = system:0 2024-02-19T21:08:06.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.550+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f1818047540 osd=0 4 2024-02-19T21:08:06.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.550+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 0 1 2024-02-19T21:08:06.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:05.176156+0000 mgr.y (mgr.14550) 299 : cluster [DBG] pgmap v428: 230 pgs: 1 remapped+peering, 9 active+recovery_wait+degraded, 3 active+recovering, 2 clean+premerge+peered, 215 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 68/948 objects degraded (7.173%); 14/948 objects misplaced (1.477%); 182 B/s, 3 objects/s recovering 2024-02-19T21:08:06.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:05.314781+0000 mon.a (mon.0) 2092 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-19T21:08:06.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:05.314801+0000 mon.a (mon.0) 2093 : cluster [WRN] Health check failed: Degraded data redundancy: 68/948 objects degraded (7.173%), 9 pgs degraded (PG_DEGRADED) 2024-02-19T21:08:06.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:05.357262+0000 mon.b (mon.1) 474 : audit [INF] from='client.? v1:172.21.15.163:0/3109614039' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:06.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:05.357544+0000 mon.a (mon.0) 2094 : audit [INF] from='client.48355 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:06.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:06.235159+0000 mon.a (mon.0) 2095 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:08:06.564 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:06.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.576+0000 7f181fe18700 10 client.48361.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:06.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.576+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 2 2024-02-19T21:08:06.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.576+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818047540 osd=0 3 2024-02-19T21:08:06.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.576+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818047540 osd=0 3 2024-02-19T21:08:06.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.576+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818047540 osd=0 3 2024-02-19T21:08:06.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.576+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 2 2024-02-19T21:08:06.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.576+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f1818047540 osd=0 3 2024-02-19T21:08:06.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.576+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 0 2 2024-02-19T21:08:06.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.576+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818047540 osd=0 4 2024-02-19T21:08:06.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.576+0000 7f181fe18700 10 client.48361.objecter _send_command 2 2024-02-19T21:08:06.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.576+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1818120c90 con 0x7f181811ff60 2024-02-19T21:08:06.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.577+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1818120c90 con 0x7f181811ff60 2024-02-19T21:08:06.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.577+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:06.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.577+0000 7f180effd700 10 client.48361.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.577+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f1818047540 osd=0 4 2024-02-19T21:08:06.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.577+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 0 2 2024-02-19T21:08:06.575 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:08:06.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.578+0000 7f181fe18700 10 client.48361.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:06.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.578+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 3 2024-02-19T21:08:06.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.578+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818120e00 osd=1 3 2024-02-19T21:08:06.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.578+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818120e00 osd=1 3 2024-02-19T21:08:06.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.578+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818120e00 osd=1 3 2024-02-19T21:08:06.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.578+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 3 2024-02-19T21:08:06.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.578+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f1818120e00 osd=1 3 2024-02-19T21:08:06.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.578+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 1 3 2024-02-19T21:08:06.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.578+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818120e00 osd=1 4 2024-02-19T21:08:06.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.578+0000 7f181fe18700 10 client.48361.objecter _send_command 3 2024-02-19T21:08:06.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.578+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1818059cd0 con 0x7f18180598f0 2024-02-19T21:08:06.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.578+0000 7f180effd700 10 client.48361.objecter ms_handle_connect 0x7f18180598f0 2024-02-19T21:08:06.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.579+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1818059cd0 con 0x7f18180598f0 2024-02-19T21:08:06.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.579+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:08:06.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.579+0000 7f180effd700 10 client.48361.objecter _finish_command 3 = system:0 2024-02-19T21:08:06.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.579+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f1818120e00 osd=1 4 2024-02-19T21:08:06.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.579+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 1 3 2024-02-19T21:08:06.582 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:06.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:06.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.606+0000 7f181fe18700 10 client.48361.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:06.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.606+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 4 2024-02-19T21:08:06.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.606+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818120e00 osd=1 3 2024-02-19T21:08:06.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.606+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818120e00 osd=1 3 2024-02-19T21:08:06.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.607+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818120e00 osd=1 3 2024-02-19T21:08:06.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.607+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 4 2024-02-19T21:08:06.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.607+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f1818120e00 osd=1 3 2024-02-19T21:08:06.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.607+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 1 4 2024-02-19T21:08:06.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.607+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818120e00 osd=1 4 2024-02-19T21:08:06.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.607+0000 7f181fe18700 10 client.48361.objecter _send_command 4 2024-02-19T21:08:06.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.607+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f181805a0b0 con 0x7f18180598f0 2024-02-19T21:08:06.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f181805a0b0 con 0x7f18180598f0 2024-02-19T21:08:06.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:06.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f180effd700 10 client.48361.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f1818120e00 osd=1 4 2024-02-19T21:08:06.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 1 4 2024-02-19T21:08:06.605 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:08:06.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f181fe18700 10 client.48361.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:06.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 5 2024-02-19T21:08:06.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818059360 osd=2 3 2024-02-19T21:08:06.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818059360 osd=2 3 2024-02-19T21:08:06.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818059360 osd=2 3 2024-02-19T21:08:06.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 5 2024-02-19T21:08:06.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f1818059360 osd=2 3 2024-02-19T21:08:06.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 2 5 2024-02-19T21:08:06.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818059360 osd=2 4 2024-02-19T21:08:06.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f181fe18700 10 client.48361.objecter _send_command 5 2024-02-19T21:08:06.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.608+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18180287b0 con 0x7f18180594d0 2024-02-19T21:08:06.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.609+0000 7f180effd700 10 client.48361.objecter ms_handle_connect 0x7f18180594d0 2024-02-19T21:08:06.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.609+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f18180287b0 con 0x7f18180594d0 2024-02-19T21:08:06.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.609+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:08:06.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.609+0000 7f180effd700 10 client.48361.objecter _finish_command 5 = system:0 2024-02-19T21:08:06.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.609+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f1818059360 osd=2 4 2024-02-19T21:08:06.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.609+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 2 5 2024-02-19T21:08:06.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.624+0000 7f181fe18700 10 client.48361.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:06.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.624+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 6 2024-02-19T21:08:06.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.624+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818059360 osd=2 3 2024-02-19T21:08:06.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.624+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818059360 osd=2 3 2024-02-19T21:08:06.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.624+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818059360 osd=2 3 2024-02-19T21:08:06.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.624+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 6 2024-02-19T21:08:06.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.624+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f1818059360 osd=2 3 2024-02-19T21:08:06.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.624+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 2 6 2024-02-19T21:08:06.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.624+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818059360 osd=2 4 2024-02-19T21:08:06.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.624+0000 7f181fe18700 10 client.48361.objecter _send_command 6 2024-02-19T21:08:06.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.624+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1818027d10 con 0x7f18180594d0 2024-02-19T21:08:06.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1818027d10 con 0x7f18180594d0 2024-02-19T21:08:06.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:06.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f180effd700 10 client.48361.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f1818059360 osd=2 4 2024-02-19T21:08:06.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 2 6 2024-02-19T21:08:06.622 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:08:06.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f181fe18700 10 client.48361.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:06.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 7 2024-02-19T21:08:06.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818028120 osd=3 3 2024-02-19T21:08:06.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818028120 osd=3 3 2024-02-19T21:08:06.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818028120 osd=3 3 2024-02-19T21:08:06.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 7 2024-02-19T21:08:06.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f1818028120 osd=3 3 2024-02-19T21:08:06.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 3 7 2024-02-19T21:08:06.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818028120 osd=3 4 2024-02-19T21:08:06.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f181fe18700 10 client.48361.objecter _send_command 7 2024-02-19T21:08:06.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.625+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18181182d0 con 0x7f1818132f90 2024-02-19T21:08:06.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.626+0000 7f180effd700 10 client.48361.objecter ms_handle_connect 0x7f1818132f90 2024-02-19T21:08:06.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.626+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f18181182d0 con 0x7f1818132f90 2024-02-19T21:08:06.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.626+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:08:06.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.626+0000 7f180effd700 10 client.48361.objecter _finish_command 7 = system:0 2024-02-19T21:08:06.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.626+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f1818028120 osd=3 4 2024-02-19T21:08:06.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.626+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 3 7 2024-02-19T21:08:06.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.641+0000 7f181fe18700 10 client.48361.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:06.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.641+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 8 2024-02-19T21:08:06.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.641+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818028120 osd=3 3 2024-02-19T21:08:06.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.641+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818028120 osd=3 3 2024-02-19T21:08:06.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.641+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818028120 osd=3 3 2024-02-19T21:08:06.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.641+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 8 2024-02-19T21:08:06.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.641+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f1818028120 osd=3 3 2024-02-19T21:08:06.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.641+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 3 8 2024-02-19T21:08:06.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.641+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818028120 osd=3 4 2024-02-19T21:08:06.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.641+0000 7f181fe18700 10 client.48361.objecter _send_command 8 2024-02-19T21:08:06.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.641+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f18181177f0 con 0x7f1818132f90 2024-02-19T21:08:06.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f18181177f0 con 0x7f1818132f90 2024-02-19T21:08:06.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:06.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f180effd700 10 client.48361.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f1818028120 osd=3 4 2024-02-19T21:08:06.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 3 8 2024-02-19T21:08:06.639 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:08:06.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f181fe18700 10 client.48361.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:06.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 9 2024-02-19T21:08:06.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f18181179b0 osd=4 3 2024-02-19T21:08:06.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f18181179b0 osd=4 3 2024-02-19T21:08:06.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f18181179b0 osd=4 3 2024-02-19T21:08:06.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 9 2024-02-19T21:08:06.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f18181179b0 osd=4 3 2024-02-19T21:08:06.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 4 9 2024-02-19T21:08:06.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f18181179b0 osd=4 4 2024-02-19T21:08:06.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f181fe18700 10 client.48361.objecter _send_command 9 2024-02-19T21:08:06.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.642+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1818117b20 con 0x7f18180fa610 2024-02-19T21:08:06.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.643+0000 7f180effd700 10 client.48361.objecter ms_handle_connect 0x7f18180fa610 2024-02-19T21:08:06.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.643+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1818117b20 con 0x7f18180fa610 2024-02-19T21:08:06.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.643+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:08:06.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.643+0000 7f180effd700 10 client.48361.objecter _finish_command 9 = system:0 2024-02-19T21:08:06.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.643+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f18181179b0 osd=4 4 2024-02-19T21:08:06.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.643+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 4 9 2024-02-19T21:08:06.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.659+0000 7f181fe18700 10 client.48361.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:06.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.659+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 10 2024-02-19T21:08:06.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.659+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f18181179b0 osd=4 3 2024-02-19T21:08:06.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.659+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f18181179b0 osd=4 3 2024-02-19T21:08:06.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.659+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f18181179b0 osd=4 3 2024-02-19T21:08:06.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.659+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 10 2024-02-19T21:08:06.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.659+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f18181179b0 osd=4 3 2024-02-19T21:08:06.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.659+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 4 10 2024-02-19T21:08:06.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.659+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f18181179b0 osd=4 4 2024-02-19T21:08:06.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.659+0000 7f181fe18700 10 client.48361.objecter _send_command 10 2024-02-19T21:08:06.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.659+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1818117b20 con 0x7f18180fa610 2024-02-19T21:08:06.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1818117b20 con 0x7f18180fa610 2024-02-19T21:08:06.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:06.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f180effd700 10 client.48361.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f18181179b0 osd=4 4 2024-02-19T21:08:06.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 4 10 2024-02-19T21:08:06.657 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:08:06.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f181fe18700 10 client.48361.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:06.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 11 2024-02-19T21:08:06.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818117b20 osd=5 3 2024-02-19T21:08:06.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818117b20 osd=5 3 2024-02-19T21:08:06.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818117b20 osd=5 3 2024-02-19T21:08:06.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 11 2024-02-19T21:08:06.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f1818117b20 osd=5 3 2024-02-19T21:08:06.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 5 11 2024-02-19T21:08:06.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818117b20 osd=5 4 2024-02-19T21:08:06.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f181fe18700 10 client.48361.objecter _send_command 11 2024-02-19T21:08:06.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.660+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1818138dd0 con 0x7f1818138840 2024-02-19T21:08:06.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.661+0000 7f180effd700 10 client.48361.objecter ms_handle_connect 0x7f1818138840 2024-02-19T21:08:06.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.662+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1818138dd0 con 0x7f1818138840 2024-02-19T21:08:06.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.662+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:08:06.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.662+0000 7f180effd700 10 client.48361.objecter _finish_command 11 = system:0 2024-02-19T21:08:06.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.662+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f1818117b20 osd=5 4 2024-02-19T21:08:06.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.662+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 5 11 2024-02-19T21:08:06.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.689+0000 7f181fe18700 10 client.48361.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:06.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.689+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 12 2024-02-19T21:08:06.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.689+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818117b20 osd=5 3 2024-02-19T21:08:06.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.689+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818117b20 osd=5 3 2024-02-19T21:08:06.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.689+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818117b20 osd=5 3 2024-02-19T21:08:06.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.689+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 12 2024-02-19T21:08:06.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.689+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f1818117b20 osd=5 3 2024-02-19T21:08:06.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.689+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 5 12 2024-02-19T21:08:06.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.689+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818117b20 osd=5 4 2024-02-19T21:08:06.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.689+0000 7f181fe18700 10 client.48361.objecter _send_command 12 2024-02-19T21:08:06.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.689+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1818138dd0 con 0x7f1818138840 2024-02-19T21:08:06.686 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:06.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.691+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1818138dd0 con 0x7f1818138840 2024-02-19T21:08:06.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.691+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:06.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.691+0000 7f180effd700 10 client.48361.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.691+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f1818117b20 osd=5 4 2024-02-19T21:08:06.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.691+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 5 12 2024-02-19T21:08:06.689 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:08:06.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.692+0000 7f181fe18700 10 client.48361.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:06.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.692+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 13 2024-02-19T21:08:06.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.692+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818138dd0 osd=6 3 2024-02-19T21:08:06.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.692+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818138dd0 osd=6 3 2024-02-19T21:08:06.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.692+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818138dd0 osd=6 3 2024-02-19T21:08:06.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.692+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 13 2024-02-19T21:08:06.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.692+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f1818138dd0 osd=6 3 2024-02-19T21:08:06.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.692+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 6 13 2024-02-19T21:08:06.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.692+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818138dd0 osd=6 4 2024-02-19T21:08:06.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.692+0000 7f181fe18700 10 client.48361.objecter _send_command 13 2024-02-19T21:08:06.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.692+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1818143a00 con 0x7f181813feb0 2024-02-19T21:08:06.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.692+0000 7f180effd700 10 client.48361.objecter ms_handle_connect 0x7f181813feb0 2024-02-19T21:08:06.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.693+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1818143a00 con 0x7f181813feb0 2024-02-19T21:08:06.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.693+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:08:06.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.693+0000 7f180effd700 10 client.48361.objecter _finish_command 13 = system:0 2024-02-19T21:08:06.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.693+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f1818138dd0 osd=6 4 2024-02-19T21:08:06.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.693+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 6 13 2024-02-19T21:08:06.711 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:06.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.720+0000 7f181fe18700 10 client.48361.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:06.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.720+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 14 2024-02-19T21:08:06.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.720+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818138dd0 osd=6 3 2024-02-19T21:08:06.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.720+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818138dd0 osd=6 3 2024-02-19T21:08:06.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.720+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818138dd0 osd=6 3 2024-02-19T21:08:06.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.720+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 14 2024-02-19T21:08:06.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.720+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f1818138dd0 osd=6 3 2024-02-19T21:08:06.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.720+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 6 14 2024-02-19T21:08:06.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.720+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818138dd0 osd=6 4 2024-02-19T21:08:06.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.720+0000 7f181fe18700 10 client.48361.objecter _send_command 14 2024-02-19T21:08:06.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.720+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1818143a00 con 0x7f181813feb0 2024-02-19T21:08:06.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1818143a00 con 0x7f181813feb0 2024-02-19T21:08:06.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:06.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f180effd700 10 client.48361.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f1818138dd0 osd=6 4 2024-02-19T21:08:06.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 6 14 2024-02-19T21:08:06.719 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:08:06.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f181fe18700 10 client.48361.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:06.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 15 2024-02-19T21:08:06.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818143a00 osd=7 3 2024-02-19T21:08:06.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818143a00 osd=7 3 2024-02-19T21:08:06.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818143a00 osd=7 3 2024-02-19T21:08:06.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 15 2024-02-19T21:08:06.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f1818143a00 osd=7 3 2024-02-19T21:08:06.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 7 15 2024-02-19T21:08:06.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818143a00 osd=7 4 2024-02-19T21:08:06.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f181fe18700 10 client.48361.objecter _send_command 15 2024-02-19T21:08:06.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1818144c20 con 0x7f18181456c0 2024-02-19T21:08:06.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.722+0000 7f180effd700 10 client.48361.objecter ms_handle_connect 0x7f18181456c0 2024-02-19T21:08:06.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.723+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1818144c20 con 0x7f18181456c0 2024-02-19T21:08:06.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.723+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:08:06.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.723+0000 7f180effd700 10 client.48361.objecter _finish_command 15 = system:0 2024-02-19T21:08:06.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.723+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f1818143a00 osd=7 4 2024-02-19T21:08:06.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.723+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 7 15 2024-02-19T21:08:06.735 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:06.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:06.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:05.176156+0000 mgr.y (mgr.14550) 299 : cluster [DBG] pgmap v428: 230 pgs: 1 remapped+peering, 9 active+recovery_wait+degraded, 3 active+recovering, 2 clean+premerge+peered, 215 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 68/948 objects degraded (7.173%); 14/948 objects misplaced (1.477%); 182 B/s, 3 objects/s recovering 2024-02-19T21:08:06.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:05.314781+0000 mon.a (mon.0) 2092 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-19T21:08:06.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:05.314801+0000 mon.a (mon.0) 2093 : cluster [WRN] Health check failed: Degraded data redundancy: 68/948 objects degraded (7.173%), 9 pgs degraded (PG_DEGRADED) 2024-02-19T21:08:06.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:05.357262+0000 mon.b (mon.1) 474 : audit [INF] from='client.? v1:172.21.15.163:0/3109614039' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:06.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:05.357544+0000 mon.a (mon.0) 2094 : audit [INF] from='client.48355 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:06.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:06.235159+0000 mon.a (mon.0) 2095 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:08:06.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:05.176156+0000 mgr.y (mgr.14550) 299 : cluster [DBG] pgmap v428: 230 pgs: 1 remapped+peering, 9 active+recovery_wait+degraded, 3 active+recovering, 2 clean+premerge+peered, 215 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 68/948 objects degraded (7.173%); 14/948 objects misplaced (1.477%); 182 B/s, 3 objects/s recovering 2024-02-19T21:08:06.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:05.314781+0000 mon.a (mon.0) 2092 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-19T21:08:06.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:05.314801+0000 mon.a (mon.0) 2093 : cluster [WRN] Health check failed: Degraded data redundancy: 68/948 objects degraded (7.173%), 9 pgs degraded (PG_DEGRADED) 2024-02-19T21:08:06.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:05.357262+0000 mon.b (mon.1) 474 : audit [INF] from='client.? v1:172.21.15.163:0/3109614039' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:06.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:05.357544+0000 mon.a (mon.0) 2094 : audit [INF] from='client.48355 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:06.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:06.235159+0000 mon.a (mon.0) 2095 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:08:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.749+0000 7f181fe18700 10 client.48361.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.749+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign -1 16 2024-02-19T21:08:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.749+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818143a00 osd=7 3 2024-02-19T21:08:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.749+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818143a00 osd=7 3 2024-02-19T21:08:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.749+0000 7f181fe18700 20 client.48361.objecter _get_session s=0x7f1818143a00 osd=7 3 2024-02-19T21:08:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.749+0000 7f181fe18700 15 client.48361.objecter _session_command_op_remove -1 16 2024-02-19T21:08:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.749+0000 7f181fe18700 20 client.48361.objecter get_session s=0x7f1818143a00 osd=7 3 2024-02-19T21:08:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.749+0000 7f181fe18700 15 client.48361.objecter _session_command_op_assign 7 16 2024-02-19T21:08:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.749+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818143a00 osd=7 4 2024-02-19T21:08:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.749+0000 7f181fe18700 10 client.48361.objecter _send_command 16 2024-02-19T21:08:06.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.749+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1818144c20 con 0x7f18181456c0 2024-02-19T21:08:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f180effd700 1 -- 172.21.15.163:0/2856516292 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1818144c20 con 0x7f18181456c0 2024-02-19T21:08:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f180effd700 10 client.48361.objecter ms_dispatch 0x7f1818001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f180effd700 10 client.48361.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f180effd700 20 client.48361.objecter put_session s=0x7f1818143a00 osd=7 4 2024-02-19T21:08:06.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f180effd700 15 client.48361.objecter _session_command_op_remove 7 16 2024-02-19T21:08:06.748 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:08:06.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 10 librados: watch_flush enter 2024-02-19T21:08:06.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 10 librados: watch_flush exit 2024-02-19T21:08:06.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 10 client.48361.objecter close_session for osd.0 2024-02-19T21:08:06.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 >> v1:172.21.15.163:6801/1559418115 conn(0x7f181811ff60 legacy=0x7f1818120340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:06.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818047540 osd=0 1 2024-02-19T21:08:06.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 10 client.48361.objecter close_session for osd.1 2024-02-19T21:08:06.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 >> v1:172.21.15.163:6805/3125047432 conn(0x7f18180598f0 legacy=0x7f18180273c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:06.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818120e00 osd=1 1 2024-02-19T21:08:06.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 10 client.48361.objecter close_session for osd.2 2024-02-19T21:08:06.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 >> v1:172.21.15.163:6809/1563158487 conn(0x7f18180594d0 legacy=0x7f1818116ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:06.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818059360 osd=2 1 2024-02-19T21:08:06.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 10 client.48361.objecter close_session for osd.3 2024-02-19T21:08:06.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 >> v1:172.21.15.163:6813/2112279769 conn(0x7f1818132f90 legacy=0x7f1818135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:06.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818028120 osd=3 1 2024-02-19T21:08:06.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 10 client.48361.objecter close_session for osd.4 2024-02-19T21:08:06.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 >> v1:172.21.15.193:6800/3129716965 conn(0x7f18180fa610 legacy=0x7f1818137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:06.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f18181179b0 osd=4 1 2024-02-19T21:08:06.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181e3b5700 1 -- 172.21.15.163:0/2856516292 reap_dead start 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 10 client.48361.objecter close_session for osd.5 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 >> v1:172.21.15.193:6804/805161991 conn(0x7f1818138840 legacy=0x7f181813e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181e3b5700 1 -- 172.21.15.163:0/2856516292 reap_dead start 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818117b20 osd=5 1 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 10 client.48361.objecter close_session for osd.6 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 >> v1:172.21.15.193:6808/787604615 conn(0x7f181813feb0 legacy=0x7f18181422a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818138dd0 osd=6 1 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 10 client.48361.objecter close_session for osd.7 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 >> v1:172.21.15.193:6812/157927432 conn(0x7f18181456c0 legacy=0x7f1818145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 20 client.48361.objecter put_session s=0x7f1818143a00 osd=7 1 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 20 client.48361.objecter shutdown clearing up homeless session... 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 10 client.48361.objecter successfully canceled tick 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181e3b5700 1 -- 172.21.15.163:0/2856516292 reap_dead start 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 >> v1:172.21.15.163:6800/1552514681 conn(0x7f1804063c50 legacy=0x7f1804066110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 >> v1:172.21.15.193:6789/0 conn(0x7f18180f8920 legacy=0x7f1818102270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:06.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 shutdown_connections 2024-02-19T21:08:06.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.751+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 >> 172.21.15.163:0/2856516292 conn(0x7f18180f2450 msgr2=0x7f1818107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:06.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.752+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 shutdown_connections 2024-02-19T21:08:06.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.752+0000 7f181fe18700 1 -- 172.21.15.163:0/2856516292 wait complete. 2024-02-19T21:08:06.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:06.752+0000 7f181fe18700 1 librados: shutdown 2024-02-19T21:08:06.752 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.752 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.752 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.752 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.752 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.752 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.752 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.752 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:08:06.839 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:08:06.866 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:07.013 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:08:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:07.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:08:07.117 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:08:07.143 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:07.245 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [==========] Running 72 tests from 4 test suites. 2024-02-19T21:08:07.245 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-02-19T21:08:07.245 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-02-19T21:08:07.245 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: seed 83928 2024-02-19T21:08:07.245 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-02-19T21:08:07.245 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (294 ms) 2024-02-19T21:08:07.245 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-02-19T21:08:07.245 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (11731 ms) 2024-02-19T21:08:07.246 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-02-19T21:08:07.246 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (10 ms) 2024-02-19T21:08:07.246 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (12035 ms total) 2024-02-19T21:08:07.246 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: 2024-02-19T21:08:07.246 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [----------] 43 tests from LibRadosTwoPoolsPP 2024-02-19T21:08:07.246 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-02-19T21:08:07.246 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (9231 ms) 2024-02-19T21:08:07.246 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-02-19T21:08:07.246 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8270 ms) 2024-02-19T21:08:07.246 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-02-19T21:08:07.246 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10078 ms) 2024-02-19T21:08:07.246 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-02-19T21:08:07.246 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: my_snaps [4] 2024-02-19T21:08:07.246 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: my_snaps [5,4] 2024-02-19T21:08:07.247 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: my_snaps [6,5,4] 2024-02-19T21:08:07.247 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: my_snaps [7,6,5,4] 2024-02-19T21:08:07.247 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: promoting some heads 2024-02-19T21:08:07.247 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: promoting from clones for snap 7 2024-02-19T21:08:07.247 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: promoting from clones for snap 6 2024-02-19T21:08:07.247 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: promoting from clones for snap 5 2024-02-19T21:08:07.247 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: promoting from clones for snap 4 2024-02-19T21:08:07.247 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: waiting for scrubs... 2024-02-19T21:08:07.247 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: done waiting 2024-02-19T21:08:07.247 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46017 ms) 2024-02-19T21:08:07.247 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-02-19T21:08:07.247 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10001 ms) 2024-02-19T21:08:07.247 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-02-19T21:08:07.247 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8011 ms) 2024-02-19T21:08:07.247 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-02-19T21:08:07.248 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7985 ms) 2024-02-19T21:08:07.248 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-02-19T21:08:07.248 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (9032 ms) 2024-02-19T21:08:07.248 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-02-19T21:08:07.248 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (9999 ms) 2024-02-19T21:08:07.248 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-02-19T21:08:07.248 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8985 ms) 2024-02-19T21:08:07.248 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-02-19T21:08:07.248 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10013 ms) 2024-02-19T21:08:07.248 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-02-19T21:08:07.248 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7997 ms) 2024-02-19T21:08:07.248 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-02-19T21:08:07.248 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8234 ms) 2024-02-19T21:08:07.248 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-02-19T21:08:07.248 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (12774 ms) 2024-02-19T21:08:07.249 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-02-19T21:08:07.249 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7996 ms) 2024-02-19T21:08:07.249 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-02-19T21:08:07.249 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8001 ms) 2024-02-19T21:08:07.249 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-02-19T21:08:07.249 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: hmm, no HitSet yet 2024-02-19T21:08:07.249 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: ok, hit_set contains 188:602f83fe:::foo:head 2024-02-19T21:08:07.591 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:08:07.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:07.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:08:07.696 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:07.721 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:07.740 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:07.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:07.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:06.485992+0000 mon.a (mon.0) 2096 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39"}]': finished 2024-02-19T21:08:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:06.486120+0000 mon.a (mon.0) 2097 : audit [INF] from='client.48355 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:08:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:06.486200+0000 mon.a (mon.0) 2098 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-02-19T21:08:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:06.549314+0000 mon.b (mon.1) 475 : audit [DBG] from='client.? v1:172.21.15.163:0/2856516292' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:08:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:07.177062+0000 mon.b (mon.1) 476 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "32"}]: dispatch 2024-02-19T21:08:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:07.178763+0000 mon.a (mon.0) 2099 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "32"}]: dispatch 2024-02-19T21:08:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:07.240097+0000 mon.a (mon.0) 2100 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi163-83854-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi163-83854-54"}]': finished 2024-02-19T21:08:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:07.240202+0000 mon.a (mon.0) 2101 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "32"}]': finished 2024-02-19T21:08:07.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:07.240328+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-02-19T21:08:07.844 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:08:07.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:06.485992+0000 mon.a (mon.0) 2096 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39"}]': finished 2024-02-19T21:08:07.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:06.486120+0000 mon.a (mon.0) 2097 : audit [INF] from='client.48355 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:08:07.872 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:06.486200+0000 mon.a (mon.0) 2098 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-02-19T21:08:07.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:06.549314+0000 mon.b (mon.1) 475 : audit [DBG] from='client.? v1:172.21.15.163:0/2856516292' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:08:07.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:07.177062+0000 mon.b (mon.1) 476 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "32"}]: dispatch 2024-02-19T21:08:07.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:07.178763+0000 mon.a (mon.0) 2099 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "32"}]: dispatch 2024-02-19T21:08:07.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:07.240097+0000 mon.a (mon.0) 2100 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi163-83854-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi163-83854-54"}]': finished 2024-02-19T21:08:07.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:07.240202+0000 mon.a (mon.0) 2101 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "32"}]': finished 2024-02-19T21:08:07.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:07.240328+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-02-19T21:08:07.873 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:06.485992+0000 mon.a (mon.0) 2096 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-39"}]': finished 2024-02-19T21:08:07.873 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:06.486120+0000 mon.a (mon.0) 2097 : audit [INF] from='client.48355 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:08:07.873 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:06.486200+0000 mon.a (mon.0) 2098 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-02-19T21:08:07.873 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:06.549314+0000 mon.b (mon.1) 475 : audit [DBG] from='client.? v1:172.21.15.163:0/2856516292' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:08:07.874 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:07.177062+0000 mon.b (mon.1) 476 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "32"}]: dispatch 2024-02-19T21:08:07.874 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:07.178763+0000 mon.a (mon.0) 2099 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "32"}]: dispatch 2024-02-19T21:08:07.874 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:07.240097+0000 mon.a (mon.0) 2100 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi163-83854-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi163-83854-54"}]': finished 2024-02-19T21:08:07.874 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:07.240202+0000 mon.a (mon.0) 2101 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "32"}]': finished 2024-02-19T21:08:07.874 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:07.240328+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-02-19T21:08:07.874 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:08.067 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:08:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:08:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:08:08.077 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:08:08.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:08.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:08:08.158 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:08.181 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:08.209 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:08.228 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:08.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:08.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:08.332 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:08:08.359 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:08.397 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:08:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:08.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:08:08.502 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:08.532 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:08.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:07.176635+0000 mgr.y (mgr.14550) 300 : cluster [DBG] pgmap v431: 229 pgs: 1 remapped+peering, 9 active+recovery_wait+degraded, 3 active+recovering, 1 clean+premerge+peered, 215 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 68/948 objects degraded (7.173%); 14/948 objects misplaced (1.477%); 182 B/s, 3 objects/s recovering 2024-02-19T21:08:08.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:08.243955+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-02-19T21:08:08.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:08.251071+0000 mon.b (mon.1) 477 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:08.649 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:08.254787+0000 mon.a (mon.0) 2104 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:08.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:07.176635+0000 mgr.y (mgr.14550) 300 : cluster [DBG] pgmap v431: 229 pgs: 1 remapped+peering, 9 active+recovery_wait+degraded, 3 active+recovering, 1 clean+premerge+peered, 215 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 68/948 objects degraded (7.173%); 14/948 objects misplaced (1.477%); 182 B/s, 3 objects/s recovering 2024-02-19T21:08:08.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:08.243955+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-02-19T21:08:08.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:08.251071+0000 mon.b (mon.1) 477 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:08.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:08.254787+0000 mon.a (mon.0) 2104 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:08.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:07.176635+0000 mgr.y (mgr.14550) 300 : cluster [DBG] pgmap v431: 229 pgs: 1 remapped+peering, 9 active+recovery_wait+degraded, 3 active+recovering, 1 clean+premerge+peered, 215 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 68/948 objects degraded (7.173%); 14/948 objects misplaced (1.477%); 182 B/s, 3 objects/s recovering 2024-02-19T21:08:08.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:08.243955+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-02-19T21:08:08.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:08.251071+0000 mon.b (mon.1) 477 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:08.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:08.254787+0000 mon.a (mon.0) 2104 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:08.979 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:08.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:08.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:09.017 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:08:09.018 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:08:09.018 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-19T21:08:09.018 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-19T21:08:09.084 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:08:09.114 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:09.136 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:08:09.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:09.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:08:09.246 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:09.275 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:09.297 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:09.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:09.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:09.400 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:09.428 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:09.447 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:09.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:09.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:09.551 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:09.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:08.410784+0000 osd.1 (osd.1) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:09.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:08.410939+0000 osd.1 (osd.1) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:09.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:08.506536+0000 osd.0 (osd.0) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:09.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:08.506683+0000 osd.0 (osd.0) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:09.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:09.100289+0000 mon.a (mon.0) 2105 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:09.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:09.218668+0000 mon.a (mon.0) 2106 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:09.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:09.218814+0000 mon.a (mon.0) 2107 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-02-19T21:08:09.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:09.222308+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi163-83854-54"}]: dispatch 2024-02-19T21:08:09.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:09.227125+0000 mon.b (mon.1) 478 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi163-83928-6","var": "pg_num","format": "json"}]: dispatch 2024-02-19T21:08:09.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:09.237879+0000 mon.b (mon.1) 479 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:09.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:09.245928+0000 mon.a (mon.0) 2109 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:09.578 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:09.593 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:08:09.596 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:09.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:09.700 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:09.726 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:09.744 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:09.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:09.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:09.807 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:08.410784+0000 osd.1 (osd.1) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:09.808 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:08.410939+0000 osd.1 (osd.1) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:09.808 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:08.506536+0000 osd.0 (osd.0) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:09.808 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:08.506683+0000 osd.0 (osd.0) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:09.808 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:09.100289+0000 mon.a (mon.0) 2105 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:09.808 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:09.218668+0000 mon.a (mon.0) 2106 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:09.808 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:09.218814+0000 mon.a (mon.0) 2107 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-02-19T21:08:09.808 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:09.222308+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi163-83854-54"}]: dispatch 2024-02-19T21:08:09.808 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:09.227125+0000 mon.b (mon.1) 478 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi163-83928-6","var": "pg_num","format": "json"}]: dispatch 2024-02-19T21:08:09.808 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:09.237879+0000 mon.b (mon.1) 479 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:09.808 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:09.245928+0000 mon.a (mon.0) 2109 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:09.808 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:08.410784+0000 osd.1 (osd.1) 149 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:09.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:08.410939+0000 osd.1 (osd.1) 150 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:09.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:08.506536+0000 osd.0 (osd.0) 121 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:09.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:08.506683+0000 osd.0 (osd.0) 122 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:09.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:09.100289+0000 mon.a (mon.0) 2105 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:09.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:09.218668+0000 mon.a (mon.0) 2106 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:09.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:09.218814+0000 mon.a (mon.0) 2107 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-02-19T21:08:09.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:09.222308+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi163-83854-54"}]: dispatch 2024-02-19T21:08:09.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:09.227125+0000 mon.b (mon.1) 478 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi163-83928-6","var": "pg_num","format": "json"}]: dispatch 2024-02-19T21:08:09.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:09.237879+0000 mon.b (mon.1) 479 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:09.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:09.245928+0000 mon.a (mon.0) 2109 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:09.848 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:09.874 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:09.893 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:09.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:09.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:09.996 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:10.023 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:10.041 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:10.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:10.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:10.080 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:08:10.085 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:08:10.085 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:08:10.085 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-02-19T21:08:10.085 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:08:10.106 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:10.145 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:10.171 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:10.190 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:10.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:10.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:10.293 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:10.320 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:10.338 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:10.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:10.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:10.442 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:10.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:08:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:08:10.468 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:10.486 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:10.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:10.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:10.590 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:10.619 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:10.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:09.177113+0000 mgr.y (mgr.14550) 301 : cluster [DBG] pgmap v434: 237 pgs: 40 unknown, 1 remapped+peering, 9 active+recovery_wait+degraded, 3 active+recovering, 1 clean+premerge+peered, 183 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 68/948 objects degraded (7.173%); 14/948 objects misplaced (1.477%) 2024-02-19T21:08:10.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:09.587800+0000 osd.6 (osd.6) 129 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:08:10.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:09.588679+0000 osd.6 (osd.6) 130 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:08:10.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:10.219382+0000 mon.a (mon.0) 2110 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi163-83854-54"}]': finished 2024-02-19T21:08:10.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:10.219476+0000 mon.a (mon.0) 2111 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:08:10.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:10.219536+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-02-19T21:08:10.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:10.222784+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi163-83854-54"}]: dispatch 2024-02-19T21:08:10.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:10.222864+0000 mon.b (mon.1) 480 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-19T21:08:10.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:10.225521+0000 mon.a (mon.0) 2114 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-19T21:08:10.807 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:09.177113+0000 mgr.y (mgr.14550) 301 : cluster [DBG] pgmap v434: 237 pgs: 40 unknown, 1 remapped+peering, 9 active+recovery_wait+degraded, 3 active+recovering, 1 clean+premerge+peered, 183 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 68/948 objects degraded (7.173%); 14/948 objects misplaced (1.477%) 2024-02-19T21:08:10.807 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:09.587800+0000 osd.6 (osd.6) 129 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:08:10.807 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:09.588679+0000 osd.6 (osd.6) 130 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:08:10.807 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:10.219382+0000 mon.a (mon.0) 2110 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi163-83854-54"}]': finished 2024-02-19T21:08:10.807 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:10.219476+0000 mon.a (mon.0) 2111 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:08:10.807 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:10.219536+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-02-19T21:08:10.808 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:10.222784+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi163-83854-54"}]: dispatch 2024-02-19T21:08:10.808 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:10.222864+0000 mon.b (mon.1) 480 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-19T21:08:10.808 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:10.225521+0000 mon.a (mon.0) 2114 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-19T21:08:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:09.177113+0000 mgr.y (mgr.14550) 301 : cluster [DBG] pgmap v434: 237 pgs: 40 unknown, 1 remapped+peering, 9 active+recovery_wait+degraded, 3 active+recovering, 1 clean+premerge+peered, 183 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 68/948 objects degraded (7.173%); 14/948 objects misplaced (1.477%) 2024-02-19T21:08:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:09.587800+0000 osd.6 (osd.6) 129 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:08:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:09.588679+0000 osd.6 (osd.6) 130 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:08:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:10.219382+0000 mon.a (mon.0) 2110 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi163-83854-54"}]': finished 2024-02-19T21:08:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:10.219476+0000 mon.a (mon.0) 2111 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:08:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:10.219536+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-02-19T21:08:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:10.222784+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi163-83854-54"}]: dispatch 2024-02-19T21:08:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:10.222864+0000 mon.b (mon.1) 480 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-19T21:08:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:10.225521+0000 mon.a (mon.0) 2114 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_count","val": "8"}]: dispatch 2024-02-19T21:08:11.203 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-19T21:08:11.469 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:11.470 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:11.472 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:11.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:11.574 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:08:11.601 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:11.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:10.275037+0000 mgr.y (mgr.14550) 302 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:11.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:10.823873+0000 osd.3 (osd.3) 135 : cluster [DBG] 16.4 scrub starts 2024-02-19T21:08:11.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:10.824691+0000 osd.3 (osd.3) 136 : cluster [DBG] 16.4 scrub ok 2024-02-19T21:08:11.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:11.222208+0000 mon.a (mon.0) 2115 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi163-83854-54"}]': finished 2024-02-19T21:08:11.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:11.222306+0000 mon.a (mon.0) 2116 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_count","val": "8"}]': finished 2024-02-19T21:08:11.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:11.222463+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-02-19T21:08:11.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:11.222972+0000 mon.b (mon.1) 481 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:11.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:11.223392+0000 mon.a (mon.0) 2118 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:11.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:11.238402+0000 mon.a (mon.0) 2119 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi163-83854-55"}]: dispatch 2024-02-19T21:08:11.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:11.239524+0000 mon.a (mon.0) 2120 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi163-83854-55"}]: dispatch 2024-02-19T21:08:11.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:11.239776+0000 mon.a (mon.0) 2121 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi163-83854-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:11.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:10.275037+0000 mgr.y (mgr.14550) 302 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:11.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:10.823873+0000 osd.3 (osd.3) 135 : cluster [DBG] 16.4 scrub starts 2024-02-19T21:08:11.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:10.824691+0000 osd.3 (osd.3) 136 : cluster [DBG] 16.4 scrub ok 2024-02-19T21:08:11.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:11.222208+0000 mon.a (mon.0) 2115 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi163-83854-54"}]': finished 2024-02-19T21:08:11.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:11.222306+0000 mon.a (mon.0) 2116 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_count","val": "8"}]': finished 2024-02-19T21:08:11.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:11.222463+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-02-19T21:08:11.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:11.222972+0000 mon.b (mon.1) 481 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:11.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:11.223392+0000 mon.a (mon.0) 2118 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:11.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:11.238402+0000 mon.a (mon.0) 2119 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi163-83854-55"}]: dispatch 2024-02-19T21:08:11.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:11.239524+0000 mon.a (mon.0) 2120 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi163-83854-55"}]: dispatch 2024-02-19T21:08:11.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:11.239776+0000 mon.a (mon.0) 2121 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi163-83854-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:11.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:10.275037+0000 mgr.y (mgr.14550) 302 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:11.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:10.823873+0000 osd.3 (osd.3) 135 : cluster [DBG] 16.4 scrub starts 2024-02-19T21:08:11.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:10.824691+0000 osd.3 (osd.3) 136 : cluster [DBG] 16.4 scrub ok 2024-02-19T21:08:11.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:11.222208+0000 mon.a (mon.0) 2115 : audit [INF] from='client.? v1:172.21.15.163:0/4035662710' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi163-83854-54"}]': finished 2024-02-19T21:08:11.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:11.222306+0000 mon.a (mon.0) 2116 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_count","val": "8"}]': finished 2024-02-19T21:08:11.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:11.222463+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-02-19T21:08:11.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:11.222972+0000 mon.b (mon.1) 481 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:11.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:11.223392+0000 mon.a (mon.0) 2118 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:11.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:11.238402+0000 mon.a (mon.0) 2119 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi163-83854-55"}]: dispatch 2024-02-19T21:08:11.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:11.239524+0000 mon.a (mon.0) 2120 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi163-83854-55"}]: dispatch 2024-02-19T21:08:11.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:11.239776+0000 mon.a (mon.0) 2121 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi163-83854-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:12.043 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:12.552 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:08:12.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:12.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:08:12.656 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:12.689 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:12.710 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:12.712 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:12.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:11.177676+0000 mgr.y (mgr.14550) 303 : cluster [DBG] pgmap v437: 228 pgs: 2 peering, 226 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 189 B/s, 26 objects/s recovering 2024-02-19T21:08:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:11.303234+0000 osd.2 (osd.2) 115 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:08:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:11.307554+0000 osd.2 (osd.2) 116 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:08:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:11.428719+0000 osd.5 (osd.5) 128 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:08:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:11.431632+0000 osd.5 (osd.5) 129 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:08:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:11.515614+0000 mon.a (mon.0) 2122 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 68/948 objects degraded (7.173%), 9 pgs degraded) 2024-02-19T21:08:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:11.590418+0000 osd.6 (osd.6) 131 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:08:12.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:11.593364+0000 osd.6 (osd.6) 132 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:08:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:11.826446+0000 osd.3 (osd.3) 137 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:08:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:11.828530+0000 osd.3 (osd.3) 138 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:08:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:12.222651+0000 mon.a (mon.0) 2123 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:08:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:12.222749+0000 mon.a (mon.0) 2124 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi163-83854-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:12.222802+0000 mon.a (mon.0) 2125 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-02-19T21:08:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:12.224436+0000 mon.a (mon.0) 2126 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi163-83854-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi163-83854-55"}]: dispatch 2024-02-19T21:08:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:12.226650+0000 mon.b (mon.1) 482 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-19T21:08:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:12.226990+0000 mon.a (mon.0) 2127 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-19T21:08:12.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:11.177676+0000 mgr.y (mgr.14550) 303 : cluster [DBG] pgmap v437: 228 pgs: 2 peering, 226 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 189 B/s, 26 objects/s recovering 2024-02-19T21:08:12.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:11.303234+0000 osd.2 (osd.2) 115 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:08:12.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:11.307554+0000 osd.2 (osd.2) 116 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:08:12.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:11.428719+0000 osd.5 (osd.5) 128 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:08:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:11.431632+0000 osd.5 (osd.5) 129 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:08:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:11.515614+0000 mon.a (mon.0) 2122 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 68/948 objects degraded (7.173%), 9 pgs degraded) 2024-02-19T21:08:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:11.590418+0000 osd.6 (osd.6) 131 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:08:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:11.593364+0000 osd.6 (osd.6) 132 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:08:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:11.826446+0000 osd.3 (osd.3) 137 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:08:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:11.828530+0000 osd.3 (osd.3) 138 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:08:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:12.222651+0000 mon.a (mon.0) 2123 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:08:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:12.222749+0000 mon.a (mon.0) 2124 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi163-83854-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:12.222802+0000 mon.a (mon.0) 2125 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-02-19T21:08:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:12.224436+0000 mon.a (mon.0) 2126 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi163-83854-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi163-83854-55"}]: dispatch 2024-02-19T21:08:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:12.226650+0000 mon.b (mon.1) 482 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-19T21:08:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:12.226990+0000 mon.a (mon.0) 2127 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-19T21:08:12.814 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:12.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:11.177676+0000 mgr.y (mgr.14550) 303 : cluster [DBG] pgmap v437: 228 pgs: 2 peering, 226 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 189 B/s, 26 objects/s recovering 2024-02-19T21:08:12.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:11.303234+0000 osd.2 (osd.2) 115 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:08:12.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:11.307554+0000 osd.2 (osd.2) 116 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:08:12.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:11.428719+0000 osd.5 (osd.5) 128 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:08:12.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:11.431632+0000 osd.5 (osd.5) 129 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:08:12.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:11.515614+0000 mon.a (mon.0) 2122 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 68/948 objects degraded (7.173%), 9 pgs degraded) 2024-02-19T21:08:12.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:11.590418+0000 osd.6 (osd.6) 131 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:08:12.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:11.593364+0000 osd.6 (osd.6) 132 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:08:12.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:11.826446+0000 osd.3 (osd.3) 137 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:08:12.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:11.828530+0000 osd.3 (osd.3) 138 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:08:12.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:12.222651+0000 mon.a (mon.0) 2123 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:08:12.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:12.222749+0000 mon.a (mon.0) 2124 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi163-83854-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:12.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:12.222802+0000 mon.a (mon.0) 2125 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-02-19T21:08:12.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:12.224436+0000 mon.a (mon.0) 2126 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi163-83854-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi163-83854-55"}]: dispatch 2024-02-19T21:08:12.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:12.226650+0000 mon.b (mon.1) 482 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-19T21:08:12.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:12.226990+0000 mon.a (mon.0) 2127 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-02-19T21:08:12.845 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:12.865 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:12.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:12.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:12.973 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:13.004 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:13.447 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:13.450 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:13.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:13.551 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:13.581 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:13.602 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:13.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:13.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:13.691 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (10028 ms) 2024-02-19T21:08:13.692 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-02-19T21:08:13.692 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg_num = 32 2024-02-19T21:08:13.692 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 0 ls 1708376894,0 2024-02-19T21:08:13.692 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 1 ls 1708376894,0 2024-02-19T21:08:13.692 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 2 ls 1708376894,0 2024-02-19T21:08:13.692 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 3 ls 1708376894,0 2024-02-19T21:08:13.692 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 4 ls 1708376894,0 2024-02-19T21:08:13.692 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 5 ls 1708376894,0 2024-02-19T21:08:13.693 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 6 ls 1708376894,0 2024-02-19T21:08:13.693 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 7 ls 1708376894,0 2024-02-19T21:08:13.693 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 8 ls 1708376894,0 2024-02-19T21:08:13.693 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 9 ls 1708376894,0 2024-02-19T21:08:13.693 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 10 ls 1708376894,0 2024-02-19T21:08:13.693 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 11 ls 1708376894,0 2024-02-19T21:08:13.693 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 12 ls 1708376894,0 2024-02-19T21:08:13.693 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 13 ls 1708376894,0 2024-02-19T21:08:13.693 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 14 ls 1708376894,0 2024-02-19T21:08:13.693 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 15 ls 1708376894,0 2024-02-19T21:08:13.694 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 16 ls 1708376894,0 2024-02-19T21:08:13.694 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 17 ls 1708376894,0 2024-02-19T21:08:13.694 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 18 ls 1708376894,0 2024-02-19T21:08:13.694 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 19 ls 1708376894,0 2024-02-19T21:08:13.694 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 20 ls 1708376894,0 2024-02-19T21:08:13.694 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 21 ls 1708376894,0 2024-02-19T21:08:13.694 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 22 ls 1708376894,0 2024-02-19T21:08:13.694 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 23 ls 1708376894,0 2024-02-19T21:08:13.694 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 24 ls 1708376894,0 2024-02-19T21:08:13.695 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 25 ls 1708376894,0 2024-02-19T21:08:13.695 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 26 ls 1708376894,0 2024-02-19T21:08:13.695 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 27 ls 1708376894,0 2024-02-19T21:08:13.695 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 28 ls 1708376894,0 2024-02-19T21:08:13.695 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 29 ls 1708376894,0 2024-02-19T21:08:13.695 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 30 ls 1708376894,0 2024-02-19T21:08:13.695 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg 31 ls 1708376894,0 2024-02-19T21:08:13.695 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: pg_num = 32 2024-02-19T21:08:13.695 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:6cac518f:::0:head 2024-02-19T21:08:13.695 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:02547ec2:::1:head 2024-02-19T21:08:13.696 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:f905c69b:::2:head 2024-02-19T21:08:13.696 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:cfc208b3:::3:head 2024-02-19T21:08:13.696 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:d83876eb:::4:head 2024-02-19T21:08:13.696 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:b29083e3:::5:head 2024-02-19T21:08:13.696 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:c4fdafeb:::6:head 2024-02-19T21:08:13.696 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:5c6b0b28:::7:head 2024-02-19T21:08:13.696 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:bd63b0f1:::8:head 2024-02-19T21:08:13.696 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:e960b815:::9:head 2024-02-19T21:08:13.696 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:52ea6a34:::10:head 2024-02-19T21:08:13.696 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:89d3ae78:::11:head 2024-02-19T21:08:13.696 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:de5d7c5f:::12:head 2024-02-19T21:08:13.697 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:566253c9:::13:head 2024-02-19T21:08:13.697 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:62a1935d:::14:head 2024-02-19T21:08:13.697 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:863748b0:::15:head 2024-02-19T21:08:13.697 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:3958e169:::16:head 2024-02-19T21:08:13.697 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:4d4dabf9:::17:head 2024-02-19T21:08:13.697 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:8391935d:::18:head 2024-02-19T21:08:13.697 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:28883081:::19:head 2024-02-19T21:08:13.697 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:69259c59:::20:head 2024-02-19T21:08:13.697 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:4bdb80b7:::21:head 2024-02-19T21:08:13.697 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:a11c5d71:::22:head 2024-02-19T21:08:13.697 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:271af37b:::23:head 2024-02-19T21:08:13.697 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:95b121be:::24:head 2024-02-19T21:08:13.698 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:58d1031b:::25:head 2024-02-19T21:08:13.698 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:0a050783:::26:head 2024-02-19T21:08:13.698 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:c709704c:::27:head 2024-02-19T21:08:13.698 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:cbe56eaf:::28:head 2024-02-19T21:08:13.698 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:86b4b162:::29:head 2024-02-19T21:08:13.698 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:70d89383:::30:head 2024-02-19T21:08:13.698 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:dd450c7c:::31:head 2024-02-19T21:08:13.698 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:6d5729b1:::32:head 2024-02-19T21:08:13.698 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:c388f3fb:::33:head 2024-02-19T21:08:13.698 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:56cfea31:::34:head 2024-02-19T21:08:13.698 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:9dbc1bf7:::35:head 2024-02-19T21:08:13.698 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:40b74ccd:::36:head 2024-02-19T21:08:13.699 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:4d5aaf42:::37:head 2024-02-19T21:08:13.699 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:920f362c:::38:head 2024-02-19T21:08:13.699 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:6cc53222:::39:head 2024-02-19T21:08:13.699 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:9cad833f:::40:head 2024-02-19T21:08:13.699 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:1ea84d41:::41:head 2024-02-19T21:08:13.699 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:c4480ef6:::42:head 2024-02-19T21:08:13.699 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:a694361e:::43:head 2024-02-19T21:08:13.699 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:d1bd33e9:::44:head 2024-02-19T21:08:13.699 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:ddc2cd5d:::45:head 2024-02-19T21:08:13.699 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:2b782207:::46:head 2024-02-19T21:08:13.699 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:7b187fca:::47:head 2024-02-19T21:08:13.700 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:90ecdf6f:::48:head 2024-02-19T21:08:13.700 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:a5ed95fe:::49:head 2024-02-19T21:08:13.700 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:ea0eaa55:::50:head 2024-02-19T21:08:13.700 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:f33ef17b:::51:head 2024-02-19T21:08:13.700 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:a0d1b2f6:::52:head 2024-02-19T21:08:13.700 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:60c5229e:::53:head 2024-02-19T21:08:13.700 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:edcbc575:::54:head 2024-02-19T21:08:13.700 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:102cf253:::55:head 2024-02-19T21:08:13.700 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:efb7fb0b:::56:head 2024-02-19T21:08:13.700 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:50d0a326:::57:head 2024-02-19T21:08:13.700 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:d4dc5daf:::58:head 2024-02-19T21:08:13.701 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:3a130462:::59:head 2024-02-19T21:08:13.701 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:ec87ed71:::60:head 2024-02-19T21:08:13.701 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:d5bc9454:::61:head 2024-02-19T21:08:13.701 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:3ddfe313:::62:head 2024-02-19T21:08:13.701 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:7c2816b9:::63:head 2024-02-19T21:08:13.701 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:47e00e4d:::64:head 2024-02-19T21:08:13.701 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:c6410c18:::65:head 2024-02-19T21:08:13.701 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:b48ed237:::66:head 2024-02-19T21:08:13.701 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:cd63ad31:::67:head 2024-02-19T21:08:13.701 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:b179e92b:::68:head 2024-02-19T21:08:13.701 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:0d9f741a:::69:head 2024-02-19T21:08:13.701 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:6d3352ae:::70:head 2024-02-19T21:08:13.702 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:c6d5c19e:::71:head 2024-02-19T21:08:13.702 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:bc4729c3:::72:head 2024-02-19T21:08:13.702 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:77e930b9:::73:head 2024-02-19T21:08:13.702 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:0abeecfd:::74:head 2024-02-19T21:08:13.702 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:b7c37e15:::75:head 2024-02-19T21:08:13.702 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:b6378398:::76:head 2024-02-19T21:08:13.702 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:02bd68de:::77:head 2024-02-19T21:08:13.702 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:cc795d2d:::78:head 2024-02-19T21:08:13.702 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:630d4fea:::79:head 2024-02-19T21:08:13.703 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:e0d29ef5:::80:head 2024-02-19T21:08:13.703 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:fd6f13d2:::81:head 2024-02-19T21:08:13.703 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:606461d5:::82:head 2024-02-19T21:08:13.703 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:eadbdc43:::83:head 2024-02-19T21:08:13.703 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:8761d0bb:::84:head 2024-02-19T21:08:13.703 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:9ef0186f:::85:head 2024-02-19T21:08:13.703 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:e0d41294:::86:head 2024-02-19T21:08:13.703 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:961de695:::87:head 2024-02-19T21:08:13.703 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:1423148f:::88:head 2024-02-19T21:08:13.703 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:633a8fa2:::89:head 2024-02-19T21:08:13.703 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:a8653809:::90:head 2024-02-19T21:08:13.704 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:3dac8b33:::91:head 2024-02-19T21:08:13.704 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:35aad435:::92:head 2024-02-19T21:08:13.704 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:f6dcc343:::93:head 2024-02-19T21:08:13.704 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:dbbdad87:::94:head 2024-02-19T21:08:13.704 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:1cb48ce0:::95:head 2024-02-19T21:08:13.704 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:03cd461c:::96:head 2024-02-19T21:08:13.704 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:17a4ea99:::97:head 2024-02-19T21:08:13.704 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:9993c9a7:::98:head 2024-02-19T21:08:13.704 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:6394211c:::99:head 2024-02-19T21:08:13.704 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:94c7ae57:::100:head 2024-02-19T21:08:13.704 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:6fdee5bb:::101:head 2024-02-19T21:08:13.705 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:9a477fd1:::102:head 2024-02-19T21:08:13.705 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:eb850916:::103:head 2024-02-19T21:08:13.705 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:affc56b9:::104:head 2024-02-19T21:08:13.705 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:b42dc814:::105:head 2024-02-19T21:08:13.705 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:f319f8f0:::106:head 2024-02-19T21:08:13.705 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:9a40b9de:::107:head 2024-02-19T21:08:13.705 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:8b524f28:::108:head 2024-02-19T21:08:13.705 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:e3de589f:::109:head 2024-02-19T21:08:13.705 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:90f90a5b:::110:head 2024-02-19T21:08:13.706 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:a7b4f1d7:::111:head 2024-02-19T21:08:13.706 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:af51766e:::112:head 2024-02-19T21:08:13.706 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:b6f90bd1:::113:head 2024-02-19T21:08:13.706 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:e0261208:::114:head 2024-02-19T21:08:13.706 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:c9569ef7:::115:head 2024-02-19T21:08:13.706 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:61bebe50:::116:head 2024-02-19T21:08:13.706 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:fe93412b:::117:head 2024-02-19T21:08:13.706 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:d3d38bee:::118:head 2024-02-19T21:08:13.706 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:3100ba0c:::119:head 2024-02-19T21:08:13.706 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:d0560ada:::120:head 2024-02-19T21:08:13.707 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:f0ea8b35:::121:head 2024-02-19T21:08:13.707 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:766f231a:::122:head 2024-02-19T21:08:13.707 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:a07a2582:::123:head 2024-02-19T21:08:13.707 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:bd7c6b3a:::124:head 2024-02-19T21:08:13.707 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:fb2ddaff:::125:head 2024-02-19T21:08:13.707 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:4408e1fe:::126:head 2024-02-19T21:08:13.707 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:ee1df7a7:::127:head 2024-02-19T21:08:13.707 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:c3002909:::128:head 2024-02-19T21:08:13.707 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:4f48ffa9:::129:head 2024-02-19T21:08:13.707 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:edf38733:::130:head 2024-02-19T21:08:13.708 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:c08425c0:::131:head 2024-02-19T21:08:13.708 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:5f902d98:::132:head 2024-02-19T21:08:13.708 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:41ea2c93:::133:head 2024-02-19T21:08:13.708 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:813cee13:::134:head 2024-02-19T21:08:13.709 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:0131818d:::135:head 2024-02-19T21:08:13.709 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:26ba5a85:::136:head 2024-02-19T21:08:13.709 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:381b8a5a:::137:head 2024-02-19T21:08:13.709 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:28797e47:::138:head 2024-02-19T21:08:13.709 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:bfca7f22:::139:head 2024-02-19T21:08:13.709 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:36807075:::140:head 2024-02-19T21:08:13.709 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:80b03975:::141:head 2024-02-19T21:08:13.709 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:5c15709b:::142:head 2024-02-19T21:08:13.710 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:f39ea15e:::143:head 2024-02-19T21:08:13.710 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:ea992956:::144:head 2024-02-19T21:08:13.710 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:48887b1c:::145:head 2024-02-19T21:08:13.710 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:9f24a9dd:::146:head 2024-02-19T21:08:13.710 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:987f100b:::147:head 2024-02-19T21:08:13.710 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:d2dd3581:::148:head 2024-02-19T21:08:13.710 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:7fed1808:::149:head 2024-02-19T21:08:13.710 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:c80b70e9:::150:head 2024-02-19T21:08:13.711 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:85ed90f9:::151:head 2024-02-19T21:08:13.711 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:36428b24:::152:head 2024-02-19T21:08:13.711 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:d044c34a:::153:head 2024-02-19T21:08:13.711 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:7c18bf58:::154:head 2024-02-19T21:08:13.711 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:d1c21232:::155:head 2024-02-19T21:08:13.711 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:a7a3c575:::156:head 2024-02-19T21:08:13.711 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:87da0633:::157:head 2024-02-19T21:08:13.711 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:d5ac3822:::158:head 2024-02-19T21:08:13.712 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:3f20522d:::159:head 2024-02-19T21:08:13.712 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:6ca26563:::160:head 2024-02-19T21:08:13.712 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:532ce135:::161:head 2024-02-19T21:08:13.712 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:c78863e6:::162:head 2024-02-19T21:08:13.712 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:7462ddf6:::163:head 2024-02-19T21:08:13.712 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:5d165639:::164:head 2024-02-19T21:08:13.712 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:f43765fc:::165:head 2024-02-19T21:08:13.713 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:13.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:12.307894+0000 osd.2 (osd.2) 117 : cluster [DBG] 16.17 scrub starts 2024-02-19T21:08:13.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:12.308947+0000 osd.2 (osd.2) 118 : cluster [DBG] 16.17 scrub ok 2024-02-19T21:08:13.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:12.442200+0000 osd.5 (osd.5) 130 : cluster [DBG] 61.d scrub starts 2024-02-19T21:08:13.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:12.445641+0000 osd.5 (osd.5) 131 : cluster [DBG] 61.d scrub ok 2024-02-19T21:08:13.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:12.605606+0000 osd.4 (osd.4) 137 : cluster [DBG] 5.1 scrub starts 2024-02-19T21:08:13.715 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:12.606581+0000 osd.4 (osd.4) 138 : cluster [DBG] 5.1 scrub ok 2024-02-19T21:08:13.715 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:12.761071+0000 osd.7 (osd.7) 135 : cluster [DBG] 61.13 scrub starts 2024-02-19T21:08:13.715 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:12.764537+0000 osd.7 (osd.7) 136 : cluster [DBG] 61.13 scrub ok 2024-02-19T21:08:13.715 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:12.855026+0000 osd.3 (osd.3) 139 : cluster [DBG] 5.13 deep-scrub starts 2024-02-19T21:08:13.715 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:12.856007+0000 osd.3 (osd.3) 140 : cluster [DBG] 5.13 deep-scrub ok 2024-02-19T21:08:13.715 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:12.952962+0000 mon.a (mon.0) 2128 : audit [INF] from='client.? v1:172.21.15.163:0/3961158216' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:08:13.715 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:13.220014+0000 mon.a (mon.0) 2129 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:08:13.715 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:13.459997+0000 mon.a (mon.0) 2130 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-02-19T21:08:13.715 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:13.460089+0000 mon.a (mon.0) 2131 : audit [INF] from='client.? v1:172.21.15.163:0/3961158216' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:08:13.715 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:13.460151+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-02-19T21:08:13.716 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:08:13] "GET /metrics HTTP/1.1" 200 191055 "" "Prometheus/2.33.4" 2024-02-19T21:08:13.716 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:12.307894+0000 osd.2 (osd.2) 117 : cluster [DBG] 16.17 scrub starts 2024-02-19T21:08:13.716 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:12.308947+0000 osd.2 (osd.2) 118 : cluster [DBG] 16.17 scrub ok 2024-02-19T21:08:13.716 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:12.442200+0000 osd.5 (osd.5) 130 : cluster [DBG] 61.d scrub starts 2024-02-19T21:08:13.716 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:12.445641+0000 osd.5 (osd.5) 131 : cluster [DBG] 61.d scrub ok 2024-02-19T21:08:13.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:12.605606+0000 osd.4 (osd.4) 137 : cluster [DBG] 5.1 scrub starts 2024-02-19T21:08:13.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:12.606581+0000 osd.4 (osd.4) 138 : cluster [DBG] 5.1 scrub ok 2024-02-19T21:08:13.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:12.761071+0000 osd.7 (osd.7) 135 : cluster [DBG] 61.13 scrub starts 2024-02-19T21:08:13.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:12.764537+0000 osd.7 (osd.7) 136 : cluster [DBG] 61.13 scrub ok 2024-02-19T21:08:13.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:12.855026+0000 osd.3 (osd.3) 139 : cluster [DBG] 5.13 deep-scrub starts 2024-02-19T21:08:13.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:12.856007+0000 osd.3 (osd.3) 140 : cluster [DBG] 5.13 deep-scrub ok 2024-02-19T21:08:13.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:12.952962+0000 mon.a (mon.0) 2128 : audit [INF] from='client.? v1:172.21.15.163:0/3961158216' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:08:13.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:13.220014+0000 mon.a (mon.0) 2129 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:08:13.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:13.459997+0000 mon.a (mon.0) 2130 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-02-19T21:08:13.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:13.460089+0000 mon.a (mon.0) 2131 : audit [INF] from='client.? v1:172.21.15.163:0/3961158216' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:08:13.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:13.460151+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-02-19T21:08:13.740 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:13.758 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:13.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:13.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:13.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.762+0000 7ff6c089e700 1 -- 172.21.15.163:0/1684775885 >> v1:172.21.15.163:6789/0 conn(0x7ff6bc0fc870 legacy=0x7ff6bc0fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:13.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.762+0000 7ff6c089e700 1 -- 172.21.15.163:0/1684775885 shutdown_connections 2024-02-19T21:08:13.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.762+0000 7ff6c089e700 1 -- 172.21.15.163:0/1684775885 >> 172.21.15.163:0/1684775885 conn(0x7ff6bc0f2470 msgr2=0x7ff6bc0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:13.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.762+0000 7ff6c089e700 1 -- 172.21.15.163:0/1684775885 shutdown_connections 2024-02-19T21:08:13.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.762+0000 7ff6c089e700 1 -- 172.21.15.163:0/1684775885 wait complete. 2024-02-19T21:08:13.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.763+0000 7ff6c089e700 1 librados: starting msgr at 2024-02-19T21:08:13.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.763+0000 7ff6c089e700 1 librados: starting objecter 2024-02-19T21:08:13.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.763+0000 7ff6c089e700 1 Processor -- start 2024-02-19T21:08:13.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.763+0000 7ff6c089e700 1 -- start start 2024-02-19T21:08:13.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.763+0000 7ff6c089e700 1 librados: setting wanted keys 2024-02-19T21:08:13.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.763+0000 7ff6c089e700 1 librados: calling monclient init 2024-02-19T21:08:13.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.763+0000 7ff6c089e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff6bc119450 con 0x7ff6bc0ff670 2024-02-19T21:08:13.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.763+0000 7ff6c089e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff6bc11a650 con 0x7ff6bc1031b0 2024-02-19T21:08:13.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.763+0000 7ff6c089e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff6bc11b850 con 0x7ff6bc0fc870 2024-02-19T21:08:13.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.763+0000 7ff6b9d9b700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ff6bc0ff670 0x7ff6bc0610f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57894/0 (socket says 172.21.15.163:57894) 2024-02-19T21:08:13.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.763+0000 7ff6b9d9b700 1 -- 172.21.15.163:0/3961158216 learned_addr learned my addr 172.21.15.163:0/3961158216 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:13.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 10 client.?.objecter ms_handle_connect 0x7ff6bc0ff670 2024-02-19T21:08:13.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:13.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 10 client.?.objecter ms_handle_connect 0x7ff6bc0fc870 2024-02-19T21:08:13.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:13.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 10 client.?.objecter ms_handle_connect 0x7ff6bc1031b0 2024-02-19T21:08:13.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:13.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933717841 0 0) 0x7ff6bc119450 con 0x7ff6bc0ff670 2024-02-19T21:08:13.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6a4003800 con 0x7ff6bc0ff670 2024-02-19T21:08:13.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149743410 0 0) 0x7ff6bc11b850 con 0x7ff6bc0fc870 2024-02-19T21:08:13.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6bc119450 con 0x7ff6bc0fc870 2024-02-19T21:08:13.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245583515 0 0) 0x7ff6bc11a650 con 0x7ff6bc1031b0 2024-02-19T21:08:13.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6bc11b850 con 0x7ff6bc1031b0 2024-02-19T21:08:13.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4257212951 0 0) 0x7ff6a4003800 con 0x7ff6bc0ff670 2024-02-19T21:08:13.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6bc11a650 con 0x7ff6bc0ff670 2024-02-19T21:08:13.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff6ac002910 con 0x7ff6bc0ff670 2024-02-19T21:08:13.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1667634903 0 0) 0x7ff6bc119450 con 0x7ff6bc0fc870 2024-02-19T21:08:13.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6a4003800 con 0x7ff6bc0fc870 2024-02-19T21:08:13.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1252243795 0 0) 0x7ff6bc11b850 con 0x7ff6bc1031b0 2024-02-19T21:08:13.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6bc119450 con 0x7ff6bc1031b0 2024-02-19T21:08:13.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff6b0002ea0 con 0x7ff6bc0fc870 2024-02-19T21:08:13.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff6b40019a0 con 0x7ff6bc1031b0 2024-02-19T21:08:13.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1108563755 0 0) 0x7ff6bc11a650 con 0x7ff6bc0ff670 2024-02-19T21:08:13.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 >> v1:172.21.15.163:6790/0 conn(0x7ff6bc0fc870 legacy=0x7ff6bc0609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:13.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 >> v1:172.21.15.193:6789/0 conn(0x7ff6bc1031b0 legacy=0x7ff6bc117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:13.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6bc11ca50 con 0x7ff6bc0ff670 2024-02-19T21:08:13.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6c089e700 1 -- 172.21.15.163:0/3961158216 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6bc11ba40 con 0x7ff6bc0ff670 2024-02-19T21:08:13.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6c089e700 10 client.22854.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:13.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6c089e700 1 -- 172.21.15.163:0/3961158216 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6bc11bf30 con 0x7ff6bc0ff670 2024-02-19T21:08:13.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6c089e700 1 librados: init done 2024-02-19T21:08:13.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff6ac0017d0 con 0x7ff6bc0ff670 2024-02-19T21:08:13.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.764+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff6ac005ab0 con 0x7ff6bc0ff670 2024-02-19T21:08:13.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.765+0000 7ff6c089e700 1 -- 172.21.15.163:0/3961158216 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff69c004f20 con 0x7ff6bc0ff670 2024-02-19T21:08:13.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.766+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ff6ac001330 con 0x7ff6bc0ff670 2024-02-19T21:08:13.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.766+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(312..312 src has 43..312) v4 ==== 8228+0+0 (unknown 1668841495 0 0) 0x7ff6ac078d70 con 0x7ff6bc0ff670 2024-02-19T21:08:13.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.766+0000 7ff6ab7fe700 10 client.22854.objecter ms_dispatch 0x7ff6bc001550 osd_map(312..312 src has 43..312) v4 2024-02-19T21:08:13.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.766+0000 7ff6ab7fe700 3 client.22854.objecter handle_osd_map got epochs [312,312] > 0 2024-02-19T21:08:13.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.766+0000 7ff6ab7fe700 3 client.22854.objecter handle_osd_map decoding full epoch 312 2024-02-19T21:08:13.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.766+0000 7ff6ab7fe700 20 client.22854.objecter dump_active .. 0 homeless 2024-02-19T21:08:13.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.766+0000 7ff6ab7fe700 10 client.22854.objecter ms_handle_connect 0x7ff6a4063d80 2024-02-19T21:08:13.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.770+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff6ac04c0e0 con 0x7ff6bc0ff670 2024-02-19T21:08:13.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:12.951+0000 7ff6c089e700 1 -- 172.21.15.163:0/3961158216 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7ff69c0051d0 con 0x7ff6bc0ff670 2024-02-19T21:08:13.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:13.458+0000 7ff6ab7fe700 1 -- 172.21.15.163:0/3961158216 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v313) v1 ==== 87+0+0 (unknown 3447838424 0 0) 0x7ff6ac04a4f0 con 0x7ff6bc0ff670 2024-02-19T21:08:13.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:13.461+0000 7ff6c089e700 10 librados: watch_flush enter 2024-02-19T21:08:13.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:13.461+0000 7ff6c089e700 10 librados: watch_flush exit 2024-02-19T21:08:13.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:13.461+0000 7ff6c089e700 20 client.22854.objecter shutdown clearing up homeless session... 2024-02-19T21:08:13.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:13.461+0000 7ff6c089e700 10 client.22854.objecter successfully canceled tick 2024-02-19T21:08:13.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:13.461+0000 7ff6c089e700 1 -- 172.21.15.163:0/3961158216 >> v1:172.21.15.163:6800/1552514681 conn(0x7ff6a4063d80 legacy=0x7ff6a4066240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:13.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:13.461+0000 7ff6c089e700 1 -- 172.21.15.163:0/3961158216 >> v1:172.21.15.163:6789/0 conn(0x7ff6bc0ff670 legacy=0x7ff6bc0610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:13.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:13.461+0000 7ff6c089e700 1 -- 172.21.15.163:0/3961158216 shutdown_connections 2024-02-19T21:08:13.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:13.461+0000 7ff6c089e700 1 -- 172.21.15.163:0/3961158216 >> 172.21.15.163:0/3961158216 conn(0x7ff6bc0f2470 msgr2=0x7ff6bc106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:13.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:13.461+0000 7ff6c089e700 1 -- 172.21.15.163:0/3961158216 shutdown_connections 2024-02-19T21:08:13.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:13.461+0000 7ff6c089e700 1 -- 172.21.15.163:0/3961158216 wait complete. 2024-02-19T21:08:13.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:13.461+0000 7ff6c089e700 1 librados: shutdown 2024-02-19T21:08:13.795 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:08:13.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:12.307894+0000 osd.2 (osd.2) 117 : cluster [DBG] 16.17 scrub starts 2024-02-19T21:08:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:12.308947+0000 osd.2 (osd.2) 118 : cluster [DBG] 16.17 scrub ok 2024-02-19T21:08:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:12.442200+0000 osd.5 (osd.5) 130 : cluster [DBG] 61.d scrub starts 2024-02-19T21:08:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:12.445641+0000 osd.5 (osd.5) 131 : cluster [DBG] 61.d scrub ok 2024-02-19T21:08:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:12.605606+0000 osd.4 (osd.4) 137 : cluster [DBG] 5.1 scrub starts 2024-02-19T21:08:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:12.606581+0000 osd.4 (osd.4) 138 : cluster [DBG] 5.1 scrub ok 2024-02-19T21:08:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:12.761071+0000 osd.7 (osd.7) 135 : cluster [DBG] 61.13 scrub starts 2024-02-19T21:08:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:12.764537+0000 osd.7 (osd.7) 136 : cluster [DBG] 61.13 scrub ok 2024-02-19T21:08:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:12.855026+0000 osd.3 (osd.3) 139 : cluster [DBG] 5.13 deep-scrub starts 2024-02-19T21:08:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:12.856007+0000 osd.3 (osd.3) 140 : cluster [DBG] 5.13 deep-scrub ok 2024-02-19T21:08:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:12.952962+0000 mon.a (mon.0) 2128 : audit [INF] from='client.? v1:172.21.15.163:0/3961158216' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:08:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:13.220014+0000 mon.a (mon.0) 2129 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:08:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:13.459997+0000 mon.a (mon.0) 2130 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-41","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-02-19T21:08:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:13.460089+0000 mon.a (mon.0) 2131 : audit [INF] from='client.? v1:172.21.15.163:0/3961158216' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:08:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:13.460151+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-02-19T21:08:13.862 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:13.890 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:14.014 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:14.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:14.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:08:14.145 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:14.496 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:08:14.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:14.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:08:14.525 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:12.581641+0000 osd.1 (osd.1) 151 : cluster [DBG] 16.13 scrub starts 2024-02-19T21:08:14.525 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:12.584869+0000 osd.1 (osd.1) 152 : cluster [DBG] 16.13 scrub ok 2024-02-19T21:08:14.525 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:13.178131+0000 mgr.y (mgr.14550) 304 : cluster [DBG] pgmap v440: 228 pgs: 2 peering, 226 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 189 B/s, 26 objects/s recovering 2024-02-19T21:08:14.525 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:13.277958+0000 osd.2 (osd.2) 119 : cluster [DBG] 16.a scrub starts 2024-02-19T21:08:14.599 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:14.625 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:14.644 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:14.647 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:14.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:14.748 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:08:14.776 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:12.581641+0000 osd.1 (osd.1) 151 : cluster [DBG] 16.13 scrub starts 2024-02-19T21:08:14.776 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:12.584869+0000 osd.1 (osd.1) 152 : cluster [DBG] 16.13 scrub ok 2024-02-19T21:08:14.776 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:13.178131+0000 mgr.y (mgr.14550) 304 : cluster [DBG] pgmap v440: 228 pgs: 2 peering, 226 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 189 B/s, 26 objects/s recovering 2024-02-19T21:08:14.776 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:13.277958+0000 osd.2 (osd.2) 119 : cluster [DBG] 16.a scrub starts 2024-02-19T21:08:14.776 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:13.693436+0000 mon.b (mon.1) 483 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi163-83928-41","var": "pg_num","format": "json"}]: dispatch 2024-02-19T21:08:14.776 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:13.802318+0000 mon.b (mon.1) 484 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:14.776 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:13.802663+0000 mon.a (mon.0) 2133 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:14.776 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:13.803160+0000 mon.b (mon.1) 485 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41"}]: dispatch 2024-02-19T21:08:14.776 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:13.803370+0000 mon.a (mon.0) 2134 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41"}]: dispatch 2024-02-19T21:08:14.777 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:14.220216+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi163-83854-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi163-83854-55"}]': finished 2024-02-19T21:08:14.777 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:14.220304+0000 mon.a (mon.0) 2136 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41"}]': finished 2024-02-19T21:08:14.777 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:14.220363+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-02-19T21:08:14.777 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:13.693436+0000 mon.b (mon.1) 483 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi163-83928-41","var": "pg_num","format": "json"}]: dispatch 2024-02-19T21:08:14.777 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:13.802318+0000 mon.b (mon.1) 484 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:14.777 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:13.802663+0000 mon.a (mon.0) 2133 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:14.777 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:13.803160+0000 mon.b (mon.1) 485 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41"}]: dispatch 2024-02-19T21:08:14.777 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:13.803370+0000 mon.a (mon.0) 2134 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41"}]: dispatch 2024-02-19T21:08:14.777 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:14.220216+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi163-83854-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi163-83854-55"}]': finished 2024-02-19T21:08:14.777 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:14.220304+0000 mon.a (mon.0) 2136 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41"}]': finished 2024-02-19T21:08:14.777 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:14.220363+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-02-19T21:08:14.778 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:14.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:12.581641+0000 osd.1 (osd.1) 151 : cluster [DBG] 16.13 scrub starts 2024-02-19T21:08:14.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:12.584869+0000 osd.1 (osd.1) 152 : cluster [DBG] 16.13 scrub ok 2024-02-19T21:08:14.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:13.178131+0000 mgr.y (mgr.14550) 304 : cluster [DBG] pgmap v440: 228 pgs: 2 peering, 226 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 189 B/s, 26 objects/s recovering 2024-02-19T21:08:14.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:13.277958+0000 osd.2 (osd.2) 119 : cluster [DBG] 16.a scrub starts 2024-02-19T21:08:14.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:13.693436+0000 mon.b (mon.1) 483 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi163-83928-41","var": "pg_num","format": "json"}]: dispatch 2024-02-19T21:08:14.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:13.802318+0000 mon.b (mon.1) 484 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:14.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:13.802663+0000 mon.a (mon.0) 2133 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:14.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:13.803160+0000 mon.b (mon.1) 485 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41"}]: dispatch 2024-02-19T21:08:14.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:13.803370+0000 mon.a (mon.0) 2134 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41"}]: dispatch 2024-02-19T21:08:14.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:14.220216+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi163-83854-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi163-83854-55"}]': finished 2024-02-19T21:08:14.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:14.220304+0000 mon.a (mon.0) 2136 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-41"}]': finished 2024-02-19T21:08:14.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:14.220363+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2024-02-19T21:08:15.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.260+0000 7f4f96775700 1 -- 172.21.15.163:0/2152384071 >> v1:172.21.15.163:6790/0 conn(0x7f4f901042a0 legacy=0x7f4f90106730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:15.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.261+0000 7f4f96775700 1 -- 172.21.15.163:0/2152384071 shutdown_connections 2024-02-19T21:08:15.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.261+0000 7f4f96775700 1 -- 172.21.15.163:0/2152384071 >> 172.21.15.163:0/2152384071 conn(0x7f4f9006fdc0 msgr2=0x7f4f900f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:15.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.261+0000 7f4f96775700 1 -- 172.21.15.163:0/2152384071 shutdown_connections 2024-02-19T21:08:15.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.261+0000 7f4f96775700 1 -- 172.21.15.163:0/2152384071 wait complete. 2024-02-19T21:08:15.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.261+0000 7f4f96775700 1 librados: starting msgr at 2024-02-19T21:08:15.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.261+0000 7f4f96775700 1 librados: starting objecter 2024-02-19T21:08:15.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.261+0000 7f4f96775700 1 Processor -- start 2024-02-19T21:08:15.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.261+0000 7f4f96775700 1 -- start start 2024-02-19T21:08:15.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.261+0000 7f4f96775700 1 librados: setting wanted keys 2024-02-19T21:08:15.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.261+0000 7f4f96775700 1 librados: calling monclient init 2024-02-19T21:08:15.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f96775700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4f900faa30 con 0x7f4f901042a0 2024-02-19T21:08:15.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f96775700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4f901187e0 con 0x7f4f90100790 2024-02-19T21:08:15.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f96775700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4f901199c0 con 0x7f4f900fd940 2024-02-19T21:08:15.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8ffff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f4f900fd940 0x7f4f900f9de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:57368/0 (socket says 172.21.15.163:57368) 2024-02-19T21:08:15.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8ffff700 1 -- 172.21.15.163:0/4126991869 learned_addr learned my addr 172.21.15.163:0/4126991869 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:15.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 10 client.?.objecter ms_handle_connect 0x7f4f901042a0 2024-02-19T21:08:15.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:15.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 10 client.?.objecter ms_handle_connect 0x7f4f900fd940 2024-02-19T21:08:15.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:15.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 10 client.?.objecter ms_handle_connect 0x7f4f90100790 2024-02-19T21:08:15.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:15.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857406511 0 0) 0x7f4f901199c0 con 0x7f4f900fd940 2024-02-19T21:08:15.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f78003630 con 0x7f4f900fd940 2024-02-19T21:08:15.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086655157 0 0) 0x7f4f901187e0 con 0x7f4f90100790 2024-02-19T21:08:15.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f901199c0 con 0x7f4f90100790 2024-02-19T21:08:15.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4238781892 0 0) 0x7f4f78003630 con 0x7f4f900fd940 2024-02-19T21:08:15.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f901187e0 con 0x7f4f900fd940 2024-02-19T21:08:15.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4f80001a10 con 0x7f4f900fd940 2024-02-19T21:08:15.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3535608183 0 0) 0x7f4f901199c0 con 0x7f4f90100790 2024-02-19T21:08:15.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f78003630 con 0x7f4f90100790 2024-02-19T21:08:15.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4f84001770 con 0x7f4f90100790 2024-02-19T21:08:15.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1551953780 0 0) 0x7f4f901187e0 con 0x7f4f900fd940 2024-02-19T21:08:15.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 >> v1:172.21.15.193:6789/0 conn(0x7f4f90100790 legacy=0x7f4f90112980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:15.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 >> v1:172.21.15.163:6789/0 conn(0x7f4f901042a0 legacy=0x7f4f901160b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:15.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f9011abe0 con 0x7f4f900fd940 2024-02-19T21:08:15.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f96775700 1 -- 172.21.15.163:0/4126991869 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f90119bb0 con 0x7f4f900fd940 2024-02-19T21:08:15.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f96775700 10 client.49559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:15.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f96775700 1 -- 172.21.15.163:0/4126991869 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f9011a020 con 0x7f4f900fd940 2024-02-19T21:08:15.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.262+0000 7f4f96775700 1 librados: init done 2024-02-19T21:08:15.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.263+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f4f80002d40 con 0x7f4f900fd940 2024-02-19T21:08:15.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.263+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4f80005c10 con 0x7f4f900fd940 2024-02-19T21:08:15.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.264+0000 7f4f96775700 1 -- 172.21.15.163:0/4126991869 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f7c004f20 con 0x7f4f900fd940 2024-02-19T21:08:15.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.264+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f4f80001fa0 con 0x7f4f900fd940 2024-02-19T21:08:15.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.264+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(315..315 src has 43..315) v4 ==== 8358+0+0 (unknown 18397995 0 0) 0x7f4f80078ed0 con 0x7f4f900fd940 2024-02-19T21:08:15.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.264+0000 7f4f8d7fa700 10 client.49559.objecter ms_dispatch 0x7f4f90001550 osd_map(315..315 src has 43..315) v4 2024-02-19T21:08:15.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.264+0000 7f4f8d7fa700 3 client.49559.objecter handle_osd_map got epochs [315,315] > 0 2024-02-19T21:08:15.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.264+0000 7f4f8d7fa700 3 client.49559.objecter handle_osd_map decoding full epoch 315 2024-02-19T21:08:15.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.264+0000 7f4f8d7fa700 20 client.49559.objecter dump_active .. 0 homeless 2024-02-19T21:08:15.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.265+0000 7f4f8d7fa700 10 client.49559.objecter ms_handle_connect 0x7f4f78063c50 2024-02-19T21:08:15.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.268+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f4f8004c1c0 con 0x7f4f900fd940 2024-02-19T21:08:15.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.451+0000 7f4f96775700 1 -- 172.21.15.163:0/4126991869 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f4f7c0051d0 con 0x7f4f900fd940 2024-02-19T21:08:15.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.453+0000 7f4f8d7fa700 1 -- 172.21.15.163:0/4126991869 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v315) v1 ==== 81+0+47895 (unknown 558302603 0 3383910433) 0x7f4f8004a5d0 con 0x7f4f900fd940 2024-02-19T21:08:15.450 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:08:15.451 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:08:15.451 INFO:teuthology.orchestra.run.smithi163.stdout: "epoch": 315, 2024-02-19T21:08:15.451 INFO:teuthology.orchestra.run.smithi163.stdout: "fsid": "2591809a-cf69-11ee-95bb-87774f69a715", 2024-02-19T21:08:15.451 INFO:teuthology.orchestra.run.smithi163.stdout: "created": "2024-02-19T20:55:37.794885+0000", 2024-02-19T21:08:15.451 INFO:teuthology.orchestra.run.smithi163.stdout: "modified": "2024-02-19T21:08:15.217336+0000", 2024-02-19T21:08:15.451 INFO:teuthology.orchestra.run.smithi163.stdout: "last_up_change": "2024-02-19T21:01:09.214134+0000", 2024-02-19T21:08:15.451 INFO:teuthology.orchestra.run.smithi163.stdout: "last_in_change": "2024-02-19T21:08:03.255528+0000", 2024-02-19T21:08:15.451 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-19T21:08:15.451 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_num": 5801984, 2024-02-19T21:08:15.451 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_set": [ 2024-02-19T21:08:15.451 INFO:teuthology.orchestra.run.smithi163.stdout: "noscrub", 2024-02-19T21:08:15.451 INFO:teuthology.orchestra.run.smithi163.stdout: "pglog_hardlimit", 2024-02-19T21:08:15.451 INFO:teuthology.orchestra.run.smithi163.stdout: "purged_snapdirs", 2024-02-19T21:08:15.452 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_deletes", 2024-02-19T21:08:15.452 INFO:teuthology.orchestra.run.smithi163.stdout: "sortbitwise" 2024-02-19T21:08:15.452 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:08:15.452 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_version": 84, 2024-02-19T21:08:15.452 INFO:teuthology.orchestra.run.smithi163.stdout: "full_ratio": 0.94999998807907104, 2024-02-19T21:08:15.452 INFO:teuthology.orchestra.run.smithi163.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-19T21:08:15.452 INFO:teuthology.orchestra.run.smithi163.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-19T21:08:15.452 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_snapshot": "", 2024-02-19T21:08:15.452 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_max": 192, 2024-02-19T21:08:15.452 INFO:teuthology.orchestra.run.smithi163.stdout: "max_osd": 8, 2024-02-19T21:08:15.452 INFO:teuthology.orchestra.run.smithi163.stdout: "require_min_compat_client": "luminous", 2024-02-19T21:08:15.452 INFO:teuthology.orchestra.run.smithi163.stdout: "min_compat_client": "luminous", 2024-02-19T21:08:15.452 INFO:teuthology.orchestra.run.smithi163.stdout: "require_osd_release": "pacific", 2024-02-19T21:08:15.453 INFO:teuthology.orchestra.run.smithi163.stdout: "pools": [ 2024-02-19T21:08:15.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.453 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 1, 2024-02-19T21:08:15.453 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "device_health_metrics", 2024-02-19T21:08:15.453 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T20:59:09.425169+0000", 2024-02-19T21:08:15.453 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:08:15.453 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:08:15.453 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:08:15.453 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:08:15.453 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:08:15.453 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:08:15.453 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:08:15.453 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:08:15.453 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:08:15.454 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:08:15.454 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:08:15.454 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:08:15.454 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 1, 2024-02-19T21:08:15.454 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 1, 2024-02-19T21:08:15.454 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 1, 2024-02-19T21:08:15.454 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 1, 2024-02-19T21:08:15.454 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 1, 2024-02-19T21:08:15.454 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:08:15.454 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:08:15.454 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:08:15.454 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:08:15.454 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:08:15.455 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:08:15.455 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:08:15.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.455 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "74", 2024-02-19T21:08:15.455 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:08:15.455 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:08:15.455 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:08:15.455 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:08:15.455 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:08:15.455 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:08:15.456 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:08:15.456 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:08:15.456 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:08:15.456 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:08:15.456 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:08:15.456 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:08:15.456 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:08:15.456 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:08:15.456 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:08:15.456 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:08:15.456 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:08:15.456 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:08:15.457 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:08:15.457 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:08:15.457 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:08:15.457 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:08:15.457 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:08:15.457 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:08:15.457 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:08:15.457 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:08:15.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.457 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:08:15.457 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:08:15.457 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:08:15.457 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:08:15.457 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:08:15.457 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:08:15.458 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:08:15.458 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:08:15.458 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:08:15.458 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:08:15.458 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:08:15.458 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:08:15.458 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_max": 32, 2024-02-19T21:08:15.458 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_min": 1 2024-02-19T21:08:15.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.458 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:08:15.458 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_devicehealth": {} 2024-02-19T21:08:15.458 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.459 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 2, 2024-02-19T21:08:15.459 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".rgw.root", 2024-02-19T21:08:15.459 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:14.658945+0000", 2024-02-19T21:08:15.459 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:08:15.459 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:08:15.459 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:08:15.459 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:08:15.459 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:08:15.459 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:08:15.459 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:08:15.459 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:08:15.459 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:08:15.459 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:08:15.460 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:08:15.460 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:08:15.460 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:08:15.460 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:08:15.460 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:08:15.460 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:08:15.460 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:08:15.460 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:08:15.460 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "2.20", 2024-02-19T21:08:15.460 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 309, 2024-02-19T21:08:15.460 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 308, 2024-02-19T21:08:15.460 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 308, 2024-02-19T21:08:15.460 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "58'1", 2024-02-19T21:08:15.460 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "58'1" 2024-02-19T21:08:15.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.461 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "309", 2024-02-19T21:08:15.461 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:08:15.461 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "309", 2024-02-19T21:08:15.461 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "307", 2024-02-19T21:08:15.461 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:08:15.461 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:08:15.461 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:08:15.461 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:08:15.461 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:08:15.461 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:08:15.461 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:08:15.461 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:08:15.461 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:08:15.461 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:08:15.462 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:08:15.463 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:08:15.463 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:08:15.463 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:08:15.463 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:08:15.463 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:08:15.463 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:08:15.463 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:08:15.463 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:08:15.463 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:08:15.463 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:08:15.463 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:08:15.463 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:08:15.463 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.464 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 3, 2024-02-19T21:08:15.464 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "iscsi", 2024-02-19T21:08:15.464 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:14.958800+0000", 2024-02-19T21:08:15.464 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:08:15.464 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:08:15.464 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:08:15.464 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:08:15.464 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:08:15.464 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:08:15.464 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:08:15.464 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:08:15.464 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:08:15.464 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:08:15.465 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:08:15.465 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:08:15.465 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 3, 2024-02-19T21:08:15.465 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 3, 2024-02-19T21:08:15.465 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 3, 2024-02-19T21:08:15.465 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 3, 2024-02-19T21:08:15.465 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 3, 2024-02-19T21:08:15.465 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:08:15.465 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:08:15.465 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:08:15.465 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:08:15.465 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:08:15.465 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:08:15.466 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:08:15.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.466 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "100", 2024-02-19T21:08:15.466 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:08:15.466 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:08:15.466 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:08:15.466 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:08:15.466 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:08:15.466 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:08:15.466 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:08:15.467 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:08:15.467 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:08:15.467 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:08:15.467 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:08:15.467 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:08:15.467 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:08:15.467 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:08:15.467 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:08:15.467 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:08:15.467 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:08:15.467 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:08:15.467 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:08:15.467 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:08:15.467 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:08:15.468 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:08:15.468 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:08:15.468 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:08:15.468 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:08:15.468 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:08:15.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.468 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:08:15.468 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:08:15.468 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:08:15.468 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:08:15.468 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:08:15.469 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:08:15.469 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:08:15.469 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:08:15.469 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:08:15.469 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:08:15.469 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:08:15.469 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:08:15.469 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:08:15.469 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd": {} 2024-02-19T21:08:15.469 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.470 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 4, 2024-02-19T21:08:15.470 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.log", 2024-02-19T21:08:15.470 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:16.543816+0000", 2024-02-19T21:08:15.470 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:08:15.470 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:08:15.470 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:08:15.470 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:08:15.470 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:08:15.470 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:08:15.470 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:08:15.470 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:08:15.470 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:08:15.471 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:08:15.471 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:08:15.471 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:08:15.471 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:08:15.471 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:08:15.471 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:08:15.471 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:08:15.471 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:08:15.471 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:08:15.472 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:08:15.472 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:08:15.472 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:08:15.472 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:08:15.472 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:08:15.472 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:08:15.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.472 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "61", 2024-02-19T21:08:15.473 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:08:15.473 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:08:15.473 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:08:15.473 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:08:15.473 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:08:15.473 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:08:15.473 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:08:15.473 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:08:15.473 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:08:15.473 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:08:15.473 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:08:15.473 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:08:15.473 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:08:15.474 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:08:15.474 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:08:15.474 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:08:15.474 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:08:15.474 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:08:15.474 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:08:15.474 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:08:15.474 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:08:15.474 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:08:15.475 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:08:15.475 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:08:15.475 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:08:15.475 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:08:15.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.475 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:08:15.475 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:08:15.475 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:08:15.475 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:08:15.475 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:08:15.476 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:08:15.476 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:08:15.476 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:08:15.476 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:08:15.476 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:08:15.476 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:08:15.476 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:08:15.476 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:08:15.476 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:08:15.476 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.477 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 5, 2024-02-19T21:08:15.477 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.control", 2024-02-19T21:08:15.477 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:18.521482+0000", 2024-02-19T21:08:15.477 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:08:15.477 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:08:15.477 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:08:15.477 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:08:15.477 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:08:15.477 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:08:15.477 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:08:15.478 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:08:15.478 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:08:15.478 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:08:15.478 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:08:15.478 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:08:15.478 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:08:15.478 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:08:15.478 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:08:15.479 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:08:15.479 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:08:15.479 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:08:15.479 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:08:15.479 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:08:15.479 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:08:15.479 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:08:15.479 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:08:15.479 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:08:15.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.479 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "63", 2024-02-19T21:08:15.479 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:08:15.479 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:08:15.479 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:08:15.479 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:08:15.480 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:08:15.480 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:08:15.480 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:08:15.480 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:08:15.480 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:08:15.480 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:08:15.480 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:08:15.480 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:08:15.481 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:08:15.481 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:08:15.481 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:08:15.481 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:08:15.481 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:08:15.481 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:08:15.481 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:08:15.481 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:08:15.481 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:08:15.482 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:08:15.482 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:08:15.482 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:08:15.482 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:08:15.482 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:08:15.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.482 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:08:15.482 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:08:15.482 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:08:15.482 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:08:15.482 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:08:15.482 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:08:15.482 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:08:15.482 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:08:15.483 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:08:15.483 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:08:15.483 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:08:15.483 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:08:15.483 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:08:15.483 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:08:15.483 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.483 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 6, 2024-02-19T21:08:15.483 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.meta", 2024-02-19T21:08:15.483 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:20.586564+0000", 2024-02-19T21:08:15.483 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:08:15.483 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:08:15.483 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:08:15.484 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:08:15.484 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:08:15.484 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:08:15.484 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:08:15.484 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:08:15.484 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:08:15.484 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:08:15.484 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:08:15.484 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:08:15.484 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:08:15.484 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:08:15.484 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:08:15.484 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:08:15.484 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:08:15.485 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:08:15.485 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:08:15.485 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:08:15.485 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:08:15.485 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:08:15.485 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:08:15.485 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:08:15.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.485 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "65", 2024-02-19T21:08:15.485 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:08:15.486 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:08:15.486 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:08:15.486 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:08:15.486 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:08:15.486 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:08:15.486 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:08:15.486 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:08:15.486 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:08:15.486 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:08:15.486 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:08:15.486 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:08:15.486 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:08:15.486 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:08:15.486 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:08:15.487 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:08:15.487 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:08:15.487 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:08:15.487 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:08:15.487 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:08:15.487 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:08:15.487 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:08:15.487 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:08:15.487 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:08:15.487 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:08:15.487 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:08:15.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.487 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:08:15.487 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:08:15.487 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:08:15.488 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:08:15.488 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:08:15.488 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:08:15.488 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:08:15.488 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:08:15.488 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:08:15.488 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:08:15.488 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:08:15.488 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:08:15.489 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_bias": 4 2024-02-19T21:08:15.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.489 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:08:15.489 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:08:15.489 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.489 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 16, 2024-02-19T21:08:15.489 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-19T21:08:15.489 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:35.333192+0000", 2024-02-19T21:08:15.489 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:08:15.490 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:08:15.490 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:08:15.490 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:08:15.490 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:08:15.490 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:08:15.490 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:08:15.490 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:08:15.490 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:08:15.490 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:08:15.490 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:08:15.491 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:08:15.491 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:08:15.491 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:08:15.491 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:08:15.491 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:08:15.491 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:08:15.491 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:08:15.491 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:08:15.492 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:08:15.492 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:08:15.492 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:08:15.492 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:08:15.492 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:08:15.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.492 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "101", 2024-02-19T21:08:15.492 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:08:15.492 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:08:15.492 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:08:15.493 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:08:15.493 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:08:15.493 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:08:15.493 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:08:15.493 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:08:15.493 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:08:15.493 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:08:15.493 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:08:15.493 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:08:15.493 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:08:15.493 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:08:15.493 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:08:15.493 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:08:15.493 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:08:15.494 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:08:15.494 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:08:15.494 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:08:15.494 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:08:15.494 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:08:15.494 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:08:15.494 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:08:15.494 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:08:15.494 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:08:15.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.494 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:08:15.494 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:08:15.494 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:08:15.494 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:08:15.495 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:08:15.495 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:08:15.495 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:08:15.495 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:08:15.495 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:08:15.495 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:08:15.495 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:08:15.495 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:08:15.495 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": {} 2024-02-19T21:08:15.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.495 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 61, 2024-02-19T21:08:15.495 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "test-rados-api-smithi163-83928-6", 2024-02-19T21:08:15.495 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:52.991355+0000", 2024-02-19T21:08:15.496 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8193, 2024-02-19T21:08:15.496 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-19T21:08:15.496 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:08:15.496 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:08:15.496 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:08:15.496 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:08:15.496 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:08:15.496 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:08:15.496 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:08:15.497 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:08:15.497 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:08:15.497 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:08:15.497 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:08:15.497 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:08:15.497 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:08:15.497 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:08:15.497 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:08:15.497 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:08:15.497 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:08:15.497 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:08:15.497 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:08:15.497 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:08:15.497 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:08:15.498 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:08:15.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.498 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "314", 2024-02-19T21:08:15.498 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "295", 2024-02-19T21:08:15.498 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "295", 2024-02-19T21:08:15.498 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "295", 2024-02-19T21:08:15.498 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:08:15.498 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:08:15.498 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 21, 2024-02-19T21:08:15.498 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 310, 2024-02-19T21:08:15.498 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:08:15.498 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:08:15.498 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:08:15.498 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:08:15.499 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:08:15.499 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:08:15.499 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:08:15.499 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:08:15.499 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:08:15.499 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:08:15.499 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:08:15.499 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:08:15.499 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:08:15.499 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:08:15.499 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:08:15.499 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:08:15.499 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:08:15.499 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:08:15.500 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:08:15.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.500 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:08:15.500 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:08:15.500 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:08:15.500 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:08:15.500 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:08:15.500 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:08:15.500 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:08:15.500 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:08:15.500 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:08:15.500 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:08:15.500 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:08:15.500 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:08:15.501 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:08:15.501 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-02-19T21:08:15.501 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.501 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 192, 2024-02-19T21:08:15.501 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "ReturnValuePP_smithi163-83854-55", 2024-02-19T21:08:15.501 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:08:13.459914+0000", 2024-02-19T21:08:15.501 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 32769, 2024-02-19T21:08:15.501 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,creating", 2024-02-19T21:08:15.501 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 3, 2024-02-19T21:08:15.501 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:08:15.502 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:08:15.502 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 1, 2024-02-19T21:08:15.502 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:08:15.502 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:08:15.502 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:08:15.502 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:08:15.502 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:08:15.502 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:08:15.502 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 8, 2024-02-19T21:08:15.503 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 8, 2024-02-19T21:08:15.503 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 8, 2024-02-19T21:08:15.503 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 8, 2024-02-19T21:08:15.503 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 8, 2024-02-19T21:08:15.503 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:08:15.503 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:08:15.503 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:08:15.503 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:08:15.503 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:08:15.503 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:08:15.503 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:08:15.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.504 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "314", 2024-02-19T21:08:15.504 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:08:15.504 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:08:15.504 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:08:15.504 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:08:15.504 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:08:15.504 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:08:15.504 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:08:15.505 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:08:15.505 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:08:15.505 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:08:15.505 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:08:15.505 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:08:15.505 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:08:15.505 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:08:15.505 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:08:15.505 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:08:15.505 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:08:15.505 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:08:15.505 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:08:15.506 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:08:15.506 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:08:15.506 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:08:15.506 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:08:15.506 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "testprofile-ReturnValuePP_smithi163-83854-55", 2024-02-19T21:08:15.506 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:08:15.506 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:08:15.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.506 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:08:15.506 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:08:15.506 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:08:15.506 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:08:15.507 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:08:15.507 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:08:15.507 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:08:15.507 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:08:15.507 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 8192, 2024-02-19T21:08:15.507 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:08:15.507 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:08:15.507 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:08:15.507 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": {} 2024-02-19T21:08:15.507 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.507 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:08:15.508 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-02-19T21:08:15.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.508 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-02-19T21:08:15.508 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088", 2024-02-19T21:08:15.508 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:08:15.508 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:08:15.508 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:08:15.508 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0, 2024-02-19T21:08:15.508 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:08:15.508 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:08:15.508 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 8, 2024-02-19T21:08:15.509 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 218, 2024-02-19T21:08:15.509 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:08:15.509 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:08:15.509 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:08:15.509 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.509 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.509 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6801", 2024-02-19T21:08:15.509 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:08:15.509 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.510 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.510 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:08:15.510 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.510 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.510 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6802", 2024-02-19T21:08:15.510 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:08:15.510 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.510 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.511 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:08:15.511 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.511 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.511 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6804", 2024-02-19T21:08:15.511 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:08:15.511 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.511 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.511 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:08:15.511 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.512 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.512 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6803", 2024-02-19T21:08:15.512 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:08:15.512 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.512 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.512 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6801/1559418115", 2024-02-19T21:08:15.512 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6802/1559418115", 2024-02-19T21:08:15.512 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6804/1559418115", 2024-02-19T21:08:15.513 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6803/1559418115", 2024-02-19T21:08:15.513 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:08:15.513 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:08:15.513 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:08:15.513 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.513 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-02-19T21:08:15.513 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "e74a65fa-cb9c-42c3-9293-602cf7e9dc67", 2024-02-19T21:08:15.513 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:08:15.513 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:08:15.513 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:08:15.514 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:08:15.514 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:08:15.514 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:08:15.514 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 13, 2024-02-19T21:08:15.514 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 314, 2024-02-19T21:08:15.514 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:08:15.514 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:08:15.514 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:08:15.514 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.514 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.515 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6805", 2024-02-19T21:08:15.515 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:08:15.515 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.515 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.515 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:08:15.515 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.515 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.516 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6806", 2024-02-19T21:08:15.516 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:08:15.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.516 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.516 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:08:15.516 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.516 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.516 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6808", 2024-02-19T21:08:15.516 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:08:15.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.516 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.517 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:08:15.517 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.517 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.517 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6807", 2024-02-19T21:08:15.517 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:08:15.517 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.517 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.517 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6805/3125047432", 2024-02-19T21:08:15.517 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6806/3125047432", 2024-02-19T21:08:15.517 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6808/3125047432", 2024-02-19T21:08:15.518 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6807/3125047432", 2024-02-19T21:08:15.518 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:08:15.518 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:08:15.518 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:08:15.518 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.518 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-02-19T21:08:15.518 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "6980a625-cdc9-487e-8f8c-7997ba716a97", 2024-02-19T21:08:15.518 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:08:15.518 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:08:15.518 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:08:15.518 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:08:15.518 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:08:15.519 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:08:15.519 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 18, 2024-02-19T21:08:15.519 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 307, 2024-02-19T21:08:15.519 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:08:15.519 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:08:15.519 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:08:15.519 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.519 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.519 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6809", 2024-02-19T21:08:15.519 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:08:15.519 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.519 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.520 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:08:15.520 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.520 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.520 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6810", 2024-02-19T21:08:15.520 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:08:15.520 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.520 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.520 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:08:15.520 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.521 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.521 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6812", 2024-02-19T21:08:15.521 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:08:15.521 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.521 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.521 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:08:15.521 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.522 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.522 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6811", 2024-02-19T21:08:15.522 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:08:15.522 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.522 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.522 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6809/1563158487", 2024-02-19T21:08:15.522 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6810/1563158487", 2024-02-19T21:08:15.522 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6812/1563158487", 2024-02-19T21:08:15.522 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6811/1563158487", 2024-02-19T21:08:15.522 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:08:15.522 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:08:15.522 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:08:15.523 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.523 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-02-19T21:08:15.523 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5", 2024-02-19T21:08:15.523 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:08:15.523 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:08:15.523 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:08:15.523 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:08:15.523 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:08:15.523 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:08:15.523 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 25, 2024-02-19T21:08:15.523 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 314, 2024-02-19T21:08:15.523 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:08:15.523 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:08:15.524 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:08:15.524 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.524 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.524 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6813", 2024-02-19T21:08:15.524 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:08:15.524 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.524 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.524 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:08:15.524 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.524 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.524 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6814", 2024-02-19T21:08:15.525 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:08:15.525 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.525 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.525 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:08:15.525 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.525 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.525 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6816", 2024-02-19T21:08:15.525 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:08:15.525 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.525 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.525 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:08:15.525 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.526 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.526 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6815", 2024-02-19T21:08:15.526 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:08:15.526 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.526 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.526 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6813/2112279769", 2024-02-19T21:08:15.526 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6814/2112279769", 2024-02-19T21:08:15.526 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6816/2112279769", 2024-02-19T21:08:15.526 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6815/2112279769", 2024-02-19T21:08:15.526 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:08:15.526 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:08:15.526 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:08:15.527 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.527 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-02-19T21:08:15.527 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0", 2024-02-19T21:08:15.527 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:08:15.527 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:08:15.527 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:08:15.527 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:08:15.527 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:08:15.527 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:08:15.527 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 31, 2024-02-19T21:08:15.527 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 314, 2024-02-19T21:08:15.527 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:08:15.527 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:08:15.528 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:08:15.528 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.528 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.528 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6800", 2024-02-19T21:08:15.528 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:08:15.528 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.528 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.528 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:08:15.528 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.528 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.529 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6801", 2024-02-19T21:08:15.529 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:08:15.529 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.529 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.529 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:08:15.529 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.529 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.529 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6803", 2024-02-19T21:08:15.529 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:08:15.530 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.530 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.530 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:08:15.530 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.530 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.530 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6802", 2024-02-19T21:08:15.530 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:08:15.530 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.530 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.530 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6800/3129716965", 2024-02-19T21:08:15.530 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6801/3129716965", 2024-02-19T21:08:15.531 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6803/3129716965", 2024-02-19T21:08:15.531 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6802/3129716965", 2024-02-19T21:08:15.531 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:08:15.531 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:08:15.531 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:08:15.531 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.531 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.531 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-02-19T21:08:15.531 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f", 2024-02-19T21:08:15.531 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:08:15.531 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:08:15.531 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:08:15.531 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:08:15.532 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:08:15.532 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:08:15.532 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 37, 2024-02-19T21:08:15.532 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 314, 2024-02-19T21:08:15.532 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:08:15.532 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:08:15.532 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:08:15.532 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.532 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.532 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6804", 2024-02-19T21:08:15.532 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:08:15.532 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.532 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.533 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:08:15.533 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.533 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.533 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6805", 2024-02-19T21:08:15.533 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:08:15.533 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.533 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.533 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:08:15.533 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.533 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.534 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6807", 2024-02-19T21:08:15.534 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:08:15.534 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.534 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.534 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:08:15.534 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.534 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.534 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6806", 2024-02-19T21:08:15.534 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:08:15.534 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.534 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.535 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6804/805161991", 2024-02-19T21:08:15.535 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6805/805161991", 2024-02-19T21:08:15.535 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6807/805161991", 2024-02-19T21:08:15.535 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6806/805161991", 2024-02-19T21:08:15.535 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:08:15.535 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:08:15.535 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:08:15.535 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.535 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-02-19T21:08:15.535 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "302c95df-3842-4818-8afd-d79d8184684f", 2024-02-19T21:08:15.536 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:08:15.536 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:08:15.536 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:08:15.536 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:08:15.539 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:08:15.539 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:08:15.539 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 45, 2024-02-19T21:08:15.539 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 308, 2024-02-19T21:08:15.540 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:08:15.540 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:08:15.540 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:08:15.540 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.540 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.540 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6808", 2024-02-19T21:08:15.540 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:08:15.540 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.541 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.541 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:08:15.541 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.541 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.541 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6809", 2024-02-19T21:08:15.541 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:08:15.542 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.542 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.542 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:08:15.542 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.542 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.542 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6811", 2024-02-19T21:08:15.542 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:08:15.542 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.542 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.542 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:08:15.543 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.543 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.543 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6810", 2024-02-19T21:08:15.543 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:08:15.543 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.543 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.543 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6808/787604615", 2024-02-19T21:08:15.543 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6809/787604615", 2024-02-19T21:08:15.543 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6811/787604615", 2024-02-19T21:08:15.544 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6810/787604615", 2024-02-19T21:08:15.544 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:08:15.544 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:08:15.544 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:08:15.544 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.544 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.544 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.544 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-02-19T21:08:15.544 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f", 2024-02-19T21:08:15.544 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:08:15.544 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:08:15.544 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:08:15.544 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:08:15.544 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:08:15.544 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:08:15.545 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 53, 2024-02-19T21:08:15.545 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 314, 2024-02-19T21:08:15.545 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:08:15.545 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:08:15.545 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:08:15.545 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.545 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.545 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.545 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6812", 2024-02-19T21:08:15.545 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:08:15.545 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.545 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.546 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:08:15.546 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.546 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.546 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6813", 2024-02-19T21:08:15.546 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:08:15.546 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.546 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.546 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:08:15.546 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.546 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.546 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6815", 2024-02-19T21:08:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:08:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:08:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:08:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:08:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6814", 2024-02-19T21:08:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:08:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.547 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.548 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6812/157927432", 2024-02-19T21:08:15.548 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6813/157927432", 2024-02-19T21:08:15.548 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6815/157927432", 2024-02-19T21:08:15.548 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6814/157927432", 2024-02-19T21:08:15.548 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:08:15.548 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:08:15.548 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:08:15.548 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.548 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.548 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:08:15.548 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_xinfo": [ 2024-02-19T21:08:15.548 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.548 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-02-19T21:08:15.548 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:08:15.549 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:08:15.549 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:08:15.549 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:08:15.549 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:08:15.549 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:08:08.506685+0000", 2024-02-19T21:08:15.549 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:08:15.549 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.549 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.549 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-02-19T21:08:15.549 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:08:15.549 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:08:15.549 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:08:15.549 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:08:15.550 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:08:15.550 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:08:08.410942+0000", 2024-02-19T21:08:15.550 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:08:15.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.550 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.550 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-02-19T21:08:15.550 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:08:15.550 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:08:15.550 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:08:15.550 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:08:15.550 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:08:15.550 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:07:30.777568+0000", 2024-02-19T21:08:15.550 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:08:15.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-02-19T21:08:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:08:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:08:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:08:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:08:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:08:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:08:14.574648+0000", 2024-02-19T21:08:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:08:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-02-19T21:08:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:08:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:08:15.551 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:08:15.552 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:08:15.552 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:08:15.552 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:08:04.496296+0000", 2024-02-19T21:08:15.552 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:08:15.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.552 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.552 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-02-19T21:08:15.552 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:08:15.552 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:08:15.552 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:08:15.552 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:08:15.552 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:08:15.552 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:06:47.102112+0000", 2024-02-19T21:08:15.552 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:08:15.553 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.553 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.553 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-02-19T21:08:15.553 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:08:15.553 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:08:15.553 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:08:15.553 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:08:15.553 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:08:15.553 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:08:03.272062+0000", 2024-02-19T21:08:15.553 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:08:15.553 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.553 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.553 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-02-19T21:08:15.553 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:08:15.554 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:08:15.554 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:08:15.554 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:08:15.554 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:08:15.554 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:07:18.963426+0000", 2024-02-19T21:08:15.554 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:08:15.554 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.554 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:08:15.554 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap": [], 2024-02-19T21:08:15.554 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_items": [ 2024-02-19T21:08:15.554 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.554 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.7", 2024-02-19T21:08:15.555 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:08:15.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.555 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:08:15.555 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-02-19T21:08:15.555 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.555 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.555 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.555 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.555 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.d", 2024-02-19T21:08:15.556 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:08:15.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.556 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:08:15.556 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 5 2024-02-19T21:08:15.556 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.556 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.556 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.556 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.12", 2024-02-19T21:08:15.557 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:08:15.557 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.557 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:08:15.557 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 5 2024-02-19T21:08:15.557 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.557 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.557 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.557 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:08:15.558 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_temp": [], 2024-02-19T21:08:15.558 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_temp": [], 2024-02-19T21:08:15.558 INFO:teuthology.orchestra.run.smithi163.stdout: "blocklist": { 2024-02-19T21:08:15.558 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1427774891": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:08:15.558 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3282102947": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:08:15.558 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2735092549": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:08:15.558 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1235588846": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:08:15.559 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2731225015": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:08:15.559 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/506491651": "2024-02-20T20:56:02.185409+0000", 2024-02-19T21:08:15.559 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/168117826": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:08:15.559 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1458009465": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:08:15.559 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/541631472": "2024-02-20T20:56:02.185409+0000", 2024-02-19T21:08:15.559 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2248999541": "2024-02-20T20:56:02.185409+0000" 2024-02-19T21:08:15.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.559 INFO:teuthology.orchestra.run.smithi163.stdout: "range_blocklist": {}, 2024-02-19T21:08:15.559 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profiles": { 2024-02-19T21:08:15.559 INFO:teuthology.orchestra.run.smithi163.stdout: "default": { 2024-02-19T21:08:15.560 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-02-19T21:08:15.560 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-02-19T21:08:15.560 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-02-19T21:08:15.560 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-02-19T21:08:15.560 INFO:teuthology.orchestra.run.smithi163.stdout: "ruleset-failure-domain": "osd", 2024-02-19T21:08:15.560 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van" 2024-02-19T21:08:15.560 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.560 INFO:teuthology.orchestra.run.smithi163.stdout: "testprofile-ReturnValuePP_smithi163-83854-55": { 2024-02-19T21:08:15.560 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-device-class": "", 2024-02-19T21:08:15.560 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-02-19T21:08:15.560 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-root": "default", 2024-02-19T21:08:15.561 INFO:teuthology.orchestra.run.smithi163.stdout: "jerasure-per-chunk-alignment": "false", 2024-02-19T21:08:15.561 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-02-19T21:08:15.561 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-02-19T21:08:15.561 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-02-19T21:08:15.561 INFO:teuthology.orchestra.run.smithi163.stdout: "ruleset-failure-domain": "osd", 2024-02-19T21:08:15.561 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van", 2024-02-19T21:08:15.561 INFO:teuthology.orchestra.run.smithi163.stdout: "w": "8" 2024-02-19T21:08:15.561 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.561 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps_queue": [ 2024-02-19T21:08:15.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.562 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 11, 2024-02-19T21:08:15.562 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:08:15.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.562 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:08:15.562 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.562 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:08:15.562 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.563 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 9, 2024-02-19T21:08:15.563 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:08:15.563 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.563 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.563 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.563 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 25, 2024-02-19T21:08:15.563 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:08:15.563 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.564 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:08:15.564 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.564 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:08:15.564 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.564 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.564 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.564 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:08:15.564 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.564 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.564 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.565 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 57, 2024-02-19T21:08:15.565 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:08:15.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.565 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:08:15.565 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:08:15.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.565 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.565 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:08:15.565 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:08:15.566 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.566 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.566 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.566 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.566 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 76, 2024-02-19T21:08:15.566 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:08:15.566 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.566 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:08:15.566 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.566 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.566 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.566 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:08:15.566 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.567 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.567 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:08:15.567 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.567 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.567 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.567 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.567 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 85, 2024-02-19T21:08:15.567 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:08:15.567 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.568 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:08:15.568 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.568 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.568 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.568 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 92, 2024-02-19T21:08:15.568 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:08:15.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.568 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:08:15.568 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:08:15.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.569 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:08:15.569 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:08:15.569 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.569 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.569 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.569 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 94, 2024-02-19T21:08:15.569 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:08:15.569 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.569 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 4, 2024-02-19T21:08:15.569 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 4 2024-02-19T21:08:15.569 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.569 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.570 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 107, 2024-02-19T21:08:15.570 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:08:15.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.570 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:08:15.570 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.570 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.570 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.570 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 112, 2024-02-19T21:08:15.570 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:08:15.570 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.571 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 1, 2024-02-19T21:08:15.571 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.571 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:08:15.571 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.571 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:08:15.571 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.571 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.571 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.571 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:08:15.571 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.572 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.572 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.572 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 126, 2024-02-19T21:08:15.572 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:08:15.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.572 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:08:15.572 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:08:15.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.572 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.572 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:08:15.573 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:08:15.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.573 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 10, 2024-02-19T21:08:15.573 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:08:15.573 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.573 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.573 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 132, 2024-02-19T21:08:15.573 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:08:15.573 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.573 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 12, 2024-02-19T21:08:15.574 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.574 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.574 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.574 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 154, 2024-02-19T21:08:15.574 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:08:15.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.574 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 14, 2024-02-19T21:08:15.574 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.574 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.574 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.574 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.575 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 164, 2024-02-19T21:08:15.575 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:08:15.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.575 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 17, 2024-02-19T21:08:15.575 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.575 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.575 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:08:15.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.575 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 178, 2024-02-19T21:08:15.575 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:08:15.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:08:15.575 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 20, 2024-02-19T21:08:15.575 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:08:15.576 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.576 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:08:15.576 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.576 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:08:15.576 INFO:teuthology.orchestra.run.smithi163.stdout: "new_removed_snaps": [], 2024-02-19T21:08:15.576 INFO:teuthology.orchestra.run.smithi163.stdout: "new_purged_snaps": [], 2024-02-19T21:08:15.576 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_node_flags": {}, 2024-02-19T21:08:15.576 INFO:teuthology.orchestra.run.smithi163.stdout: "device_class_flags": {}, 2024-02-19T21:08:15.576 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode": { 2024-02-19T21:08:15.576 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_enabled": false, 2024-02-19T21:08:15.576 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_bucket_count": 0, 2024-02-19T21:08:15.576 INFO:teuthology.orchestra.run.smithi163.stdout: "degraded_stretch_mode": 0, 2024-02-19T21:08:15.576 INFO:teuthology.orchestra.run.smithi163.stdout: "recovering_stretch_mode": 0, 2024-02-19T21:08:15.577 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_bucket": 0 2024-02-19T21:08:15.577 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:08:15.577 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:08:15.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.454+0000 7f4f96775700 10 librados: watch_flush enter 2024-02-19T21:08:15.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.454+0000 7f4f96775700 10 librados: watch_flush exit 2024-02-19T21:08:15.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.454+0000 7f4f96775700 20 client.49559.objecter shutdown clearing up homeless session... 2024-02-19T21:08:15.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.454+0000 7f4f96775700 10 client.49559.objecter successfully canceled tick 2024-02-19T21:08:15.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.455+0000 7f4f96775700 1 -- 172.21.15.163:0/4126991869 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4f78063c50 legacy=0x7f4f78066110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:15.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.455+0000 7f4f96775700 1 -- 172.21.15.163:0/4126991869 >> v1:172.21.15.163:6790/0 conn(0x7f4f900fd940 legacy=0x7f4f900f9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:15.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.455+0000 7f4f96775700 1 -- 172.21.15.163:0/4126991869 shutdown_connections 2024-02-19T21:08:15.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.455+0000 7f4f96775700 1 -- 172.21.15.163:0/4126991869 >> 172.21.15.163:0/4126991869 conn(0x7f4f9006fdc0 msgr2=0x7f4f900f4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:15.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.455+0000 7f4f96775700 1 -- 172.21.15.163:0/4126991869 shutdown_connections 2024-02-19T21:08:15.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.455+0000 7f4f96775700 1 -- 172.21.15.163:0/4126991869 wait complete. 2024-02-19T21:08:15.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:15.455+0000 7f4f96775700 1 librados: shutdown 2024-02-19T21:08:15.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:14.535921+0000 mon.a (mon.0) 2138 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:15.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:14.574519+0000 osd.3 (osd.3) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:15.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:14.574646+0000 osd.3 (osd.3) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:15.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:15.224691+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-02-19T21:08:15.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:15.453192+0000 mon.c (mon.2) 344 : audit [DBG] from='client.? v1:172.21.15.163:0/4126991869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:08:15.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:14.535921+0000 mon.a (mon.0) 2138 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:15.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:14.574519+0000 osd.3 (osd.3) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:15.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:14.574646+0000 osd.3 (osd.3) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:15.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:15.224691+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-02-19T21:08:15.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:15.453192+0000 mon.c (mon.2) 344 : audit [DBG] from='client.? v1:172.21.15.163:0/4126991869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:08:15.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:14.535921+0000 mon.a (mon.0) 2138 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:15.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:14.574519+0000 osd.3 (osd.3) 141 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:15.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:14.574646+0000 osd.3 (osd.3) 142 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:15.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:15.224691+0000 mon.a (mon.0) 2139 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2024-02-19T21:08:15.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:15.453192+0000 mon.c (mon.2) 344 : audit [DBG] from='client.? v1:172.21.15.163:0/4126991869' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:08:16.362 INFO:tasks.thrashosds.thrasher:j is {'epoch': 315, 'fsid': '2591809a-cf69-11ee-95bb-87774f69a715', 'created': '2024-02-19T20:55:37.794885+0000', 'modified': '2024-02-19T21:08:15.217336+0000', 'last_up_change': '2024-02-19T21:01:09.214134+0000', 'last_in_change': '2024-02-19T21:08:03.255528+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 84, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 192, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-19T20:59:09.425169+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-19T21:01:14.658945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '2.20', 'ready_epoch': 309, 'last_epoch_started': 308, 'last_epoch_clean': 308, 'source_version': "58'1", 'target_version': "58'1"}, 'last_change': '309', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '309', 'last_force_op_resend_preluminous': '307', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-19T21:01:14.958800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-19T21:01:16.543816+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-19T21:01:18.521482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-19T21:01:20.586564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 16, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-19T21:04:35.333192+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 61, 'pool_name': 'test-rados-api-smithi163-83928-6', 'create_time': '2024-02-19T21:04:52.991355+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '314', 'last_force_op_resend': '295', 'last_force_op_resend_prenautilus': '295', 'last_force_op_resend_preluminous': '295', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 310, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}, {'pool': 192, 'pool_name': 'ReturnValuePP_smithi163-83854-55', 'create_time': '2024-02-19T21:08:13.459914+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 3, 'size': 3, 'min_size': 2, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '314', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'testprofile-ReturnValuePP_smithi163-83854-55', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}], 'osds': [{'osd': 0, 'uuid': '4a14b406-7c21-40f3-8b12-8cf9df811088', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 218, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6801', 'nonce': 1559418115}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6802', 'nonce': 1559418115}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6804', 'nonce': 1559418115}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6803', 'nonce': 1559418115}]}, 'public_addr': '172.21.15.163:6801/1559418115', 'cluster_addr': '172.21.15.163:6802/1559418115', 'heartbeat_back_addr': '172.21.15.163:6804/1559418115', 'heartbeat_front_addr': '172.21.15.163:6803/1559418115', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e74a65fa-cb9c-42c3-9293-602cf7e9dc67', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 314, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6805', 'nonce': 3125047432}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6806', 'nonce': 3125047432}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6808', 'nonce': 3125047432}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6807', 'nonce': 3125047432}]}, 'public_addr': '172.21.15.163:6805/3125047432', 'cluster_addr': '172.21.15.163:6806/3125047432', 'heartbeat_back_addr': '172.21.15.163:6808/3125047432', 'heartbeat_front_addr': '172.21.15.163:6807/3125047432', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6980a625-cdc9-487e-8f8c-7997ba716a97', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 307, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6809', 'nonce': 1563158487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6810', 'nonce': 1563158487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6812', 'nonce': 1563158487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6811', 'nonce': 1563158487}]}, 'public_addr': '172.21.15.163:6809/1563158487', 'cluster_addr': '172.21.15.163:6810/1563158487', 'heartbeat_back_addr': '172.21.15.163:6812/1563158487', 'heartbeat_front_addr': '172.21.15.163:6811/1563158487', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '359e2ee0-9d1d-4074-8119-2d69bc2ca3a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 314, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6813', 'nonce': 2112279769}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6814', 'nonce': 2112279769}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6816', 'nonce': 2112279769}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6815', 'nonce': 2112279769}]}, 'public_addr': '172.21.15.163:6813/2112279769', 'cluster_addr': '172.21.15.163:6814/2112279769', 'heartbeat_back_addr': '172.21.15.163:6816/2112279769', 'heartbeat_front_addr': '172.21.15.163:6815/2112279769', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2bf5ef05-292b-4693-85a9-97991f2091e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 314, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6800', 'nonce': 3129716965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6801', 'nonce': 3129716965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6803', 'nonce': 3129716965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6802', 'nonce': 3129716965}]}, 'public_addr': '172.21.15.193:6800/3129716965', 'cluster_addr': '172.21.15.193:6801/3129716965', 'heartbeat_back_addr': '172.21.15.193:6803/3129716965', 'heartbeat_front_addr': '172.21.15.193:6802/3129716965', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '59065f9b-9240-414b-9c91-e91cda4aed6f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 314, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6804', 'nonce': 805161991}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6805', 'nonce': 805161991}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6807', 'nonce': 805161991}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6806', 'nonce': 805161991}]}, 'public_addr': '172.21.15.193:6804/805161991', 'cluster_addr': '172.21.15.193:6805/805161991', 'heartbeat_back_addr': '172.21.15.193:6807/805161991', 'heartbeat_front_addr': '172.21.15.193:6806/805161991', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '302c95df-3842-4818-8afd-d79d8184684f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 308, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6808', 'nonce': 787604615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6809', 'nonce': 787604615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6811', 'nonce': 787604615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6810', 'nonce': 787604615}]}, 'public_addr': '172.21.15.193:6808/787604615', 'cluster_addr': '172.21.15.193:6809/787604615', 'heartbeat_back_addr': '172.21.15.193:6811/787604615', 'heartbeat_front_addr': '172.21.15.193:6810/787604615', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '9ebeff27-c2f8-4a66-923f-a0d7573ddd3f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 314, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6812', 'nonce': 157927432}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6813', 'nonce': 157927432}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6815', 'nonce': 157927432}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6814', 'nonce': 157927432}]}, 'public_addr': '172.21.15.193:6812/157927432', 'cluster_addr': '172.21.15.193:6813/157927432', 'heartbeat_back_addr': '172.21.15.193:6815/157927432', 'heartbeat_front_addr': '172.21.15.193:6814/157927432', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:08:08.506685+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:08:08.410942+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:07:30.777568+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:08:14.574648+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:08:04.496296+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:06:47.102112+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:08:03.272062+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:07:18.963426+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.163:0/1427774891': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/3282102947': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:6800/2735092549': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/1235588846': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:6800/2731225015': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:0/506491651': '2024-02-20T20:56:02.185409+0000', '172.21.15.163:0/168117826': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/1458009465': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:0/541631472': '2024-02-20T20:56:02.185409+0000', '172.21.15.163:6800/2248999541': '2024-02-20T20:56:02.185409+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}, 'testprofile-ReturnValuePP_smithi163-83854-55': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 11, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 25, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 57, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 76, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 85, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 92, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 94, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 126, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 132, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 178, 'snaps': [{'begin': 20, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-19T21:08:16.363 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.12 2024-02-19T21:08:16.363 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd rm-pg-upmap-items 4.12 2024-02-19T21:08:16.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:15.179029+0000 mgr.y (mgr.14550) 305 : cluster [DBG] pgmap v443: 236 pgs: 8 creating+peering, 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 101 op/s; 16 B/s, 7 objects/s recovering 2024-02-19T21:08:16.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:15.534381+0000 mon.a (mon.0) 2140 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-19T21:08:16.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:16.223612+0000 mon.a (mon.0) 2141 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-02-19T21:08:16.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:16.234257+0000 mon.a (mon.0) 2142 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi163-83854-55"}]: dispatch 2024-02-19T21:08:16.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:16.235856+0000 mon.b (mon.1) 486 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:16.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:16.236355+0000 mon.a (mon.0) 2143 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:16.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:15.179029+0000 mgr.y (mgr.14550) 305 : cluster [DBG] pgmap v443: 236 pgs: 8 creating+peering, 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 101 op/s; 16 B/s, 7 objects/s recovering 2024-02-19T21:08:16.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:15.534381+0000 mon.a (mon.0) 2140 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-19T21:08:16.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:16.223612+0000 mon.a (mon.0) 2141 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-02-19T21:08:16.723 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:16.234257+0000 mon.a (mon.0) 2142 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi163-83854-55"}]: dispatch 2024-02-19T21:08:16.723 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:16.235856+0000 mon.b (mon.1) 486 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:16.723 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:16.236355+0000 mon.a (mon.0) 2143 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:16.723 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:16.728 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:08:16.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:16.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:08:16.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:15.179029+0000 mgr.y (mgr.14550) 305 : cluster [DBG] pgmap v443: 236 pgs: 8 creating+peering, 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 101 op/s; 16 B/s, 7 objects/s recovering 2024-02-19T21:08:16.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:15.534381+0000 mon.a (mon.0) 2140 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-19T21:08:16.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:16.223612+0000 mon.a (mon.0) 2141 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2024-02-19T21:08:16.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:16.234257+0000 mon.a (mon.0) 2142 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi163-83854-55"}]: dispatch 2024-02-19T21:08:16.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:16.235856+0000 mon.b (mon.1) 486 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:16.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:16.236355+0000 mon.a (mon.0) 2143 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:16.832 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:16.865 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:16.886 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:16.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:16.990 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:08:17.017 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:17.166 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:08:17.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:17.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:08:17.270 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:17.301 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:17.322 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:17.324 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:17.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:17.425 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:17.454 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:17.475 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:17.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:17.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:16.116078+0000 osd.5 (osd.5) 132 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:17.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:16.116380+0000 osd.5 (osd.5) 133 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:17.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:17.230350+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi163-83854-55"}]': finished 2024-02-19T21:08:17.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:17.230466+0000 mon.a (mon.0) 2145 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:17.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:17.230528+0000 mon.a (mon.0) 2146 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-02-19T21:08:17.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:17.234813+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi163-83854-55"}]: dispatch 2024-02-19T21:08:17.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:17.236123+0000 mon.b (mon.1) 487 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:17.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:17.237008+0000 mon.a (mon.0) 2148 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:17.579 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:17.607 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:17.626 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:17.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:17.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:17.667 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:08:17.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:16.116078+0000 osd.5 (osd.5) 132 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:17.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:16.116380+0000 osd.5 (osd.5) 133 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:17.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:17.230350+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi163-83854-55"}]': finished 2024-02-19T21:08:17.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:17.230466+0000 mon.a (mon.0) 2145 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:17.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:17.230528+0000 mon.a (mon.0) 2146 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-02-19T21:08:17.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:17.234813+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi163-83854-55"}]: dispatch 2024-02-19T21:08:17.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:17.236123+0000 mon.b (mon.1) 487 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:17.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:17.237008+0000 mon.a (mon.0) 2148 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:17.696 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:16.116078+0000 osd.5 (osd.5) 132 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:17.696 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:16.116380+0000 osd.5 (osd.5) 133 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:17.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:17.230350+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi163-83854-55"}]': finished 2024-02-19T21:08:17.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:17.230466+0000 mon.a (mon.0) 2145 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:17.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:17.230528+0000 mon.a (mon.0) 2146 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2024-02-19T21:08:17.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:17.234813+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi163-83854-55"}]: dispatch 2024-02-19T21:08:17.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:17.236123+0000 mon.b (mon.1) 487 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:17.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:17.237008+0000 mon.a (mon.0) 2148 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:17.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.711+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/238180383 >> v1:172.21.15.163:6790/0 conn(0x7f2c980fd980 legacy=0x7f2c980fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:17.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.711+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/238180383 shutdown_connections 2024-02-19T21:08:17.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.711+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/238180383 >> 172.21.15.163:0/238180383 conn(0x7f2c980f35a0 msgr2=0x7f2c980f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:17.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.711+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/238180383 shutdown_connections 2024-02-19T21:08:17.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.711+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/238180383 wait complete. 2024-02-19T21:08:17.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.712+0000 7f2c9e3ac700 1 librados: starting msgr at 2024-02-19T21:08:17.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.712+0000 7f2c9e3ac700 1 librados: starting objecter 2024-02-19T21:08:17.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.712+0000 7f2c9e3ac700 1 Processor -- start 2024-02-19T21:08:17.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.712+0000 7f2c9e3ac700 1 -- start start 2024-02-19T21:08:17.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.712+0000 7f2c9e3ac700 1 librados: setting wanted keys 2024-02-19T21:08:17.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.712+0000 7f2c9e3ac700 1 librados: calling monclient init 2024-02-19T21:08:17.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.712+0000 7f2c9e3ac700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2c98117260 con 0x7f2c981042c0 2024-02-19T21:08:17.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.712+0000 7f2c9e3ac700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2c98118440 con 0x7f2c980fd980 2024-02-19T21:08:17.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.712+0000 7f2c9e3ac700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2c98119640 con 0x7f2c98100780 2024-02-19T21:08:17.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.712+0000 7f2c977fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f2c98100780 0x7f2c980fa070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:57374/0 (socket says 172.21.15.163:57374) 2024-02-19T21:08:17.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c977fe700 1 -- 172.21.15.163:0/3180848905 learned_addr learned my addr 172.21.15.163:0/3180848905 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:17.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.712+0000 7f2c9c949700 1 --1- 172.21.15.163:0/3180848905 >> v1:172.21.15.163:6789/0 conn(0x7f2c981042c0 0x7f2c98115940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:37570/0 (socket says 172.21.15.163:37570) 2024-02-19T21:08:17.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 10 client.?.objecter ms_handle_connect 0x7f2c98100780 2024-02-19T21:08:17.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:17.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 10 client.?.objecter ms_handle_connect 0x7f2c981042c0 2024-02-19T21:08:17.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:17.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 10 client.?.objecter ms_handle_connect 0x7f2c980fd980 2024-02-19T21:08:17.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:17.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973889098 0 0) 0x7f2c98118440 con 0x7f2c980fd980 2024-02-19T21:08:17.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c80003800 con 0x7f2c980fd980 2024-02-19T21:08:17.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1217606750 0 0) 0x7f2c98119640 con 0x7f2c98100780 2024-02-19T21:08:17.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c98118440 con 0x7f2c98100780 2024-02-19T21:08:17.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636976856 0 0) 0x7f2c98117260 con 0x7f2c981042c0 2024-02-19T21:08:17.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c98119640 con 0x7f2c981042c0 2024-02-19T21:08:17.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3987994862 0 0) 0x7f2c80003800 con 0x7f2c980fd980 2024-02-19T21:08:17.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c98117260 con 0x7f2c980fd980 2024-02-19T21:08:17.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2c88002e50 con 0x7f2c980fd980 2024-02-19T21:08:17.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1008257820 0 0) 0x7f2c98118440 con 0x7f2c98100780 2024-02-19T21:08:17.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c80003800 con 0x7f2c98100780 2024-02-19T21:08:17.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2c8c0019f0 con 0x7f2c98100780 2024-02-19T21:08:17.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3200452406 0 0) 0x7f2c98119640 con 0x7f2c981042c0 2024-02-19T21:08:17.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c98118440 con 0x7f2c981042c0 2024-02-19T21:08:17.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2c90001b80 con 0x7f2c981042c0 2024-02-19T21:08:17.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2004553066 0 0) 0x7f2c80003800 con 0x7f2c98100780 2024-02-19T21:08:17.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 >> v1:172.21.15.193:6789/0 conn(0x7f2c980fd980 legacy=0x7f2c980f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:17.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 >> v1:172.21.15.163:6789/0 conn(0x7f2c981042c0 legacy=0x7f2c98115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:17.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9811a840 con 0x7f2c98100780 2024-02-19T21:08:17.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c98117450 con 0x7f2c98100780 2024-02-19T21:08:17.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c9e3ac700 10 client.49565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:17.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c98117940 con 0x7f2c98100780 2024-02-19T21:08:17.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c9e3ac700 1 librados: init done 2024-02-19T21:08:17.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f2c8c002430 con 0x7f2c98100780 2024-02-19T21:08:17.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.713+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2c8c005ac0 con 0x7f2c98100780 2024-02-19T21:08:17.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.714+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f2c84004f00 con 0x7f2c98100780 2024-02-19T21:08:17.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.715+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f2c8c018a10 con 0x7f2c98100780 2024-02-19T21:08:17.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.715+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(317..317 src has 43..317) v4 ==== 8039+0+0 (unknown 2867105205 0 0) 0x7f2c8c078d50 con 0x7f2c98100780 2024-02-19T21:08:17.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.715+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 osd_map(317..317 src has 43..317) v4 2024-02-19T21:08:17.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.715+0000 7f2c957fa700 3 client.49565.objecter handle_osd_map got epochs [317,317] > 0 2024-02-19T21:08:17.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.715+0000 7f2c957fa700 3 client.49565.objecter handle_osd_map decoding full epoch 317 2024-02-19T21:08:17.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.715+0000 7f2c957fa700 20 client.49565.objecter dump_active .. 0 homeless 2024-02-19T21:08:17.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.715+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v317) v1 ==== 54+0+16 (unknown 810007472 0 2420000192) 0x7f2c8c07b1a0 con 0x7f2c98100780 2024-02-19T21:08:17.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.716+0000 7f2c957fa700 10 client.49565.objecter ms_handle_connect 0x7f2c800680d0 2024-02-19T21:08:17.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.716+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:17.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.716+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 1 2024-02-19T21:08:17.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.716+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c840051a0 osd=0 3 2024-02-19T21:08:17.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.716+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c840051a0 osd=0 3 2024-02-19T21:08:17.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.716+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c840051a0 osd=0 3 2024-02-19T21:08:17.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.716+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 1 2024-02-19T21:08:17.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.716+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c840051a0 osd=0 3 2024-02-19T21:08:17.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.716+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 0 1 2024-02-19T21:08:17.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.716+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c840051a0 osd=0 4 2024-02-19T21:08:17.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.716+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 1 2024-02-19T21:08:17.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.716+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c84004950 con 0x7f2c84004530 2024-02-19T21:08:17.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.716+0000 7f2c957fa700 10 client.49565.objecter ms_handle_connect 0x7f2c84004530 2024-02-19T21:08:17.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.717+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2c84004950 con 0x7f2c84004530 2024-02-19T21:08:17.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.717+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:08:17.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.717+0000 7f2c957fa700 10 client.49565.objecter _finish_command 1 = system:0 2024-02-19T21:08:17.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.717+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c840051a0 osd=0 4 2024-02-19T21:08:17.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.717+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 0 1 2024-02-19T21:08:17.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.732+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:17.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.732+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 2 2024-02-19T21:08:17.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.732+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c840051a0 osd=0 3 2024-02-19T21:08:17.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.732+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c840051a0 osd=0 3 2024-02-19T21:08:17.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.732+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c840051a0 osd=0 3 2024-02-19T21:08:17.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.732+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 2 2024-02-19T21:08:17.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.732+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c840051a0 osd=0 3 2024-02-19T21:08:17.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.732+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 0 2 2024-02-19T21:08:17.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.732+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c840051a0 osd=0 4 2024-02-19T21:08:17.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.732+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 2 2024-02-19T21:08:17.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.732+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2c84004d30 con 0x7f2c84004530 2024-02-19T21:08:17.730 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:17.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2c84004d30 con 0x7f2c84004530 2024-02-19T21:08:17.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:17.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c957fa700 10 client.49565.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:17.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c840051a0 osd=0 4 2024-02-19T21:08:17.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 0 2 2024-02-19T21:08:17.732 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:08:17.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:17.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 3 2024-02-19T21:08:17.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84003fe0 osd=1 3 2024-02-19T21:08:17.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84003fe0 osd=1 3 2024-02-19T21:08:17.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84003fe0 osd=1 3 2024-02-19T21:08:17.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 3 2024-02-19T21:08:17.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c84003fe0 osd=1 3 2024-02-19T21:08:17.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 1 3 2024-02-19T21:08:17.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84003fe0 osd=1 4 2024-02-19T21:08:17.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 3 2024-02-19T21:08:17.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c8400c000 con 0x7f2c84004150 2024-02-19T21:08:17.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.734+0000 7f2c957fa700 10 client.49565.objecter ms_handle_connect 0x7f2c84004150 2024-02-19T21:08:17.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.735+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2c8400c000 con 0x7f2c84004150 2024-02-19T21:08:17.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.735+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:08:17.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.735+0000 7f2c957fa700 10 client.49565.objecter _finish_command 3 = system:0 2024-02-19T21:08:17.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.735+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c84003fe0 osd=1 4 2024-02-19T21:08:17.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.735+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 1 3 2024-02-19T21:08:17.760 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:17.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.764+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:17.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.764+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 4 2024-02-19T21:08:17.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.764+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84003fe0 osd=1 3 2024-02-19T21:08:17.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.764+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84003fe0 osd=1 3 2024-02-19T21:08:17.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.764+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84003fe0 osd=1 3 2024-02-19T21:08:17.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.764+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 4 2024-02-19T21:08:17.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.764+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c84003fe0 osd=1 3 2024-02-19T21:08:17.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.764+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 1 4 2024-02-19T21:08:17.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.764+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84003fe0 osd=1 4 2024-02-19T21:08:17.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.764+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 4 2024-02-19T21:08:17.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.764+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2c840018b0 con 0x7f2c84004150 2024-02-19T21:08:17.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.766+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2c840018b0 con 0x7f2c84004150 2024-02-19T21:08:17.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.766+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:17.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.766+0000 7f2c957fa700 10 client.49565.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:17.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.766+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c84003fe0 osd=1 4 2024-02-19T21:08:17.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.766+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 1 4 2024-02-19T21:08:17.763 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:08:17.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:17.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 5 2024-02-19T21:08:17.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84003630 osd=2 3 2024-02-19T21:08:17.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84003630 osd=2 3 2024-02-19T21:08:17.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84003630 osd=2 3 2024-02-19T21:08:17.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 5 2024-02-19T21:08:17.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c84003630 osd=2 3 2024-02-19T21:08:17.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 2 5 2024-02-19T21:08:17.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84003630 osd=2 4 2024-02-19T21:08:17.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 5 2024-02-19T21:08:17.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c8400de60 con 0x7f2c8400d2f0 2024-02-19T21:08:17.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c957fa700 10 client.49565.objecter ms_handle_connect 0x7f2c8400d2f0 2024-02-19T21:08:17.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2c8400de60 con 0x7f2c8400d2f0 2024-02-19T21:08:17.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:08:17.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c957fa700 10 client.49565.objecter _finish_command 5 = system:0 2024-02-19T21:08:17.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c84003630 osd=2 4 2024-02-19T21:08:17.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.767+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 2 5 2024-02-19T21:08:17.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.797+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:17.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.797+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 6 2024-02-19T21:08:17.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.797+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84003630 osd=2 3 2024-02-19T21:08:17.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.797+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84003630 osd=2 3 2024-02-19T21:08:17.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.797+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84003630 osd=2 3 2024-02-19T21:08:17.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.797+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 6 2024-02-19T21:08:17.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.797+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c84003630 osd=2 3 2024-02-19T21:08:17.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.797+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 2 6 2024-02-19T21:08:17.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.797+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84003630 osd=2 4 2024-02-19T21:08:17.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.797+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 6 2024-02-19T21:08:17.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.797+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2c840030a0 con 0x7f2c8400d2f0 2024-02-19T21:08:17.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2c840030a0 con 0x7f2c8400d2f0 2024-02-19T21:08:17.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:17.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c957fa700 10 client.49565.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:17.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c84003630 osd=2 4 2024-02-19T21:08:17.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 2 6 2024-02-19T21:08:17.796 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:08:17.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:17.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 7 2024-02-19T21:08:17.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c840034b0 osd=3 3 2024-02-19T21:08:17.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c840034b0 osd=3 3 2024-02-19T21:08:17.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c840034b0 osd=3 3 2024-02-19T21:08:17.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 7 2024-02-19T21:08:17.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c840034b0 osd=3 3 2024-02-19T21:08:17.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 3 7 2024-02-19T21:08:17.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c840034b0 osd=3 4 2024-02-19T21:08:17.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 7 2024-02-19T21:08:17.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.799+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c84015aa0 con 0x7f2c84011e80 2024-02-19T21:08:17.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.800+0000 7f2c957fa700 10 client.49565.objecter ms_handle_connect 0x7f2c84011e80 2024-02-19T21:08:17.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.800+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2c84015aa0 con 0x7f2c84011e80 2024-02-19T21:08:17.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.800+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:08:17.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.800+0000 7f2c957fa700 10 client.49565.objecter _finish_command 7 = system:0 2024-02-19T21:08:17.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.800+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c840034b0 osd=3 4 2024-02-19T21:08:17.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.800+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 3 7 2024-02-19T21:08:17.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.815+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:17.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.815+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 8 2024-02-19T21:08:17.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.815+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c840034b0 osd=3 3 2024-02-19T21:08:17.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.815+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c840034b0 osd=3 3 2024-02-19T21:08:17.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.815+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c840034b0 osd=3 3 2024-02-19T21:08:17.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.815+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 8 2024-02-19T21:08:17.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.815+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c840034b0 osd=3 3 2024-02-19T21:08:17.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.815+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 3 8 2024-02-19T21:08:17.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.815+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c840034b0 osd=3 4 2024-02-19T21:08:17.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.815+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 8 2024-02-19T21:08:17.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.815+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2c8400e210 con 0x7f2c84011e80 2024-02-19T21:08:17.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2c8400e210 con 0x7f2c84011e80 2024-02-19T21:08:17.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:17.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c957fa700 10 client.49565.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:17.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c840034b0 osd=3 4 2024-02-19T21:08:17.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 3 8 2024-02-19T21:08:17.814 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:08:17.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:17.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 9 2024-02-19T21:08:17.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c8400dfd0 osd=4 3 2024-02-19T21:08:17.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c8400dfd0 osd=4 3 2024-02-19T21:08:17.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c8400dfd0 osd=4 3 2024-02-19T21:08:17.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 9 2024-02-19T21:08:17.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c8400dfd0 osd=4 3 2024-02-19T21:08:17.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 4 9 2024-02-19T21:08:17.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c8400dfd0 osd=4 4 2024-02-19T21:08:17.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 9 2024-02-19T21:08:17.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.816+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c8400e180 con 0x7f2c8400ce50 2024-02-19T21:08:17.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.817+0000 7f2c957fa700 10 client.49565.objecter ms_handle_connect 0x7f2c8400ce50 2024-02-19T21:08:17.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.818+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2c8400e180 con 0x7f2c8400ce50 2024-02-19T21:08:17.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.818+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:08:17.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.818+0000 7f2c957fa700 10 client.49565.objecter _finish_command 9 = system:0 2024-02-19T21:08:17.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.818+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c8400dfd0 osd=4 4 2024-02-19T21:08:17.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.818+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 4 9 2024-02-19T21:08:17.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.849+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:17.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.849+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 10 2024-02-19T21:08:17.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.849+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c8400dfd0 osd=4 3 2024-02-19T21:08:17.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.849+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c8400dfd0 osd=4 3 2024-02-19T21:08:17.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.849+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c8400dfd0 osd=4 3 2024-02-19T21:08:17.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.849+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 10 2024-02-19T21:08:17.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.849+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c8400dfd0 osd=4 3 2024-02-19T21:08:17.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.849+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 4 10 2024-02-19T21:08:17.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.849+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c8400dfd0 osd=4 4 2024-02-19T21:08:17.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.849+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 10 2024-02-19T21:08:17.846 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.849+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2c8400e180 con 0x7f2c8400ce50 2024-02-19T21:08:17.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2c8400e180 con 0x7f2c8400ce50 2024-02-19T21:08:17.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:17.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c957fa700 10 client.49565.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:17.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c8400dfd0 osd=4 4 2024-02-19T21:08:17.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 4 10 2024-02-19T21:08:17.848 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:08:17.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:17.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 11 2024-02-19T21:08:17.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c8400e180 osd=5 3 2024-02-19T21:08:17.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c8400e180 osd=5 3 2024-02-19T21:08:17.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c8400e180 osd=5 3 2024-02-19T21:08:17.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 11 2024-02-19T21:08:17.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c8400e180 osd=5 3 2024-02-19T21:08:17.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 5 11 2024-02-19T21:08:17.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c8400e180 osd=5 4 2024-02-19T21:08:17.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 11 2024-02-19T21:08:17.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c84018010 con 0x7f2c84017a30 2024-02-19T21:08:17.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.851+0000 7f2c957fa700 10 client.49565.objecter ms_handle_connect 0x7f2c84017a30 2024-02-19T21:08:17.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.852+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2c84018010 con 0x7f2c84017a30 2024-02-19T21:08:17.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.852+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:08:17.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.852+0000 7f2c957fa700 10 client.49565.objecter _finish_command 11 = system:0 2024-02-19T21:08:17.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.852+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c8400e180 osd=5 4 2024-02-19T21:08:17.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.852+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 5 11 2024-02-19T21:08:17.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.867+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:17.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.867+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 12 2024-02-19T21:08:17.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.867+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c8400e180 osd=5 3 2024-02-19T21:08:17.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.867+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c8400e180 osd=5 3 2024-02-19T21:08:17.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.867+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c8400e180 osd=5 3 2024-02-19T21:08:17.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.867+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 12 2024-02-19T21:08:17.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.867+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c8400e180 osd=5 3 2024-02-19T21:08:17.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.867+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 5 12 2024-02-19T21:08:17.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.867+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c8400e180 osd=5 4 2024-02-19T21:08:17.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.867+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 12 2024-02-19T21:08:17.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.867+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2c84018250 con 0x7f2c84017a30 2024-02-19T21:08:17.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2c8c04c3f0 con 0x7f2c84017a30 2024-02-19T21:08:17.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:17.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c957fa700 10 client.49565.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:17.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c8400e180 osd=5 4 2024-02-19T21:08:17.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 5 12 2024-02-19T21:08:17.866 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:08:17.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:17.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 13 2024-02-19T21:08:17.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84018010 osd=6 3 2024-02-19T21:08:17.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84018010 osd=6 3 2024-02-19T21:08:17.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84018010 osd=6 3 2024-02-19T21:08:17.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 13 2024-02-19T21:08:17.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c84018010 osd=6 3 2024-02-19T21:08:17.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 6 13 2024-02-19T21:08:17.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84018010 osd=6 4 2024-02-19T21:08:17.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 13 2024-02-19T21:08:17.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c84022da0 con 0x7f2c8401f0a0 2024-02-19T21:08:17.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.869+0000 7f2c957fa700 10 client.49565.objecter ms_handle_connect 0x7f2c8401f0a0 2024-02-19T21:08:17.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.870+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2c84022da0 con 0x7f2c8401f0a0 2024-02-19T21:08:17.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.870+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:08:17.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.870+0000 7f2c957fa700 10 client.49565.objecter _finish_command 13 = system:0 2024-02-19T21:08:17.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.870+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c84018010 osd=6 4 2024-02-19T21:08:17.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.870+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 6 13 2024-02-19T21:08:17.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.885+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:17.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.885+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 14 2024-02-19T21:08:17.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.885+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84018010 osd=6 3 2024-02-19T21:08:17.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.885+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84018010 osd=6 3 2024-02-19T21:08:17.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.885+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84018010 osd=6 3 2024-02-19T21:08:17.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.885+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 14 2024-02-19T21:08:17.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.885+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c84018010 osd=6 3 2024-02-19T21:08:17.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.885+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 6 14 2024-02-19T21:08:17.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.885+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84018010 osd=6 4 2024-02-19T21:08:17.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.885+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 14 2024-02-19T21:08:17.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.885+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2c84024550 con 0x7f2c8401f0a0 2024-02-19T21:08:17.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2c84024550 con 0x7f2c8401f0a0 2024-02-19T21:08:17.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:17.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c957fa700 10 client.49565.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:17.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c84018010 osd=6 4 2024-02-19T21:08:17.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 6 14 2024-02-19T21:08:17.884 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:08:17.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:17.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 15 2024-02-19T21:08:17.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84024550 osd=7 3 2024-02-19T21:08:17.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84024550 osd=7 3 2024-02-19T21:08:17.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84024550 osd=7 3 2024-02-19T21:08:17.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 15 2024-02-19T21:08:17.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c84024550 osd=7 3 2024-02-19T21:08:17.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 7 15 2024-02-19T21:08:17.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84024550 osd=7 4 2024-02-19T21:08:17.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 15 2024-02-19T21:08:17.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c84025170 con 0x7f2c84024c30 2024-02-19T21:08:17.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.887+0000 7f2c957fa700 10 client.49565.objecter ms_handle_connect 0x7f2c84024c30 2024-02-19T21:08:17.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.888+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f2c84025170 con 0x7f2c84024c30 2024-02-19T21:08:17.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.888+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:08:17.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.888+0000 7f2c957fa700 10 client.49565.objecter _finish_command 15 = system:0 2024-02-19T21:08:17.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.888+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c84024550 osd=7 4 2024-02-19T21:08:17.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.888+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 7 15 2024-02-19T21:08:17.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.902+0000 7f2c9e3ac700 10 client.49565.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:17.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.902+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign -1 16 2024-02-19T21:08:17.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.903+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84024550 osd=7 3 2024-02-19T21:08:17.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.903+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84024550 osd=7 3 2024-02-19T21:08:17.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.903+0000 7f2c9e3ac700 20 client.49565.objecter _get_session s=0x7f2c84024550 osd=7 3 2024-02-19T21:08:17.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.903+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_remove -1 16 2024-02-19T21:08:17.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.903+0000 7f2c9e3ac700 20 client.49565.objecter get_session s=0x7f2c84024550 osd=7 3 2024-02-19T21:08:17.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.903+0000 7f2c9e3ac700 15 client.49565.objecter _session_command_op_assign 7 16 2024-02-19T21:08:17.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.903+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84024550 osd=7 4 2024-02-19T21:08:17.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.903+0000 7f2c9e3ac700 10 client.49565.objecter _send_command 16 2024-02-19T21:08:17.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.903+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f2c84025170 con 0x7f2c84024c30 2024-02-19T21:08:17.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c957fa700 1 -- 172.21.15.163:0/3180848905 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f2c84025170 con 0x7f2c84024c30 2024-02-19T21:08:17.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c957fa700 10 client.49565.objecter ms_dispatch 0x7f2c98001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:17.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c957fa700 10 client.49565.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:17.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c957fa700 20 client.49565.objecter put_session s=0x7f2c84024550 osd=7 4 2024-02-19T21:08:17.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c957fa700 15 client.49565.objecter _session_command_op_remove 7 16 2024-02-19T21:08:17.901 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:08:17.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 10 librados: watch_flush enter 2024-02-19T21:08:17.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 10 librados: watch_flush exit 2024-02-19T21:08:17.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 10 client.49565.objecter close_session for osd.0 2024-02-19T21:08:17.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 >> v1:172.21.15.163:6801/1559418115 conn(0x7f2c84004530 legacy=0x7f2c84002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:17.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c840051a0 osd=0 1 2024-02-19T21:08:17.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 10 client.49565.objecter close_session for osd.1 2024-02-19T21:08:17.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 >> v1:172.21.15.163:6805/3125047432 conn(0x7f2c84004150 legacy=0x7f2c8400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:17.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84003fe0 osd=1 1 2024-02-19T21:08:17.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 10 client.49565.objecter close_session for osd.2 2024-02-19T21:08:17.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 >> v1:172.21.15.163:6809/1563158487 conn(0x7f2c8400d2f0 legacy=0x7f2c8400d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:17.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84003630 osd=2 1 2024-02-19T21:08:17.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 10 client.49565.objecter close_session for osd.3 2024-02-19T21:08:17.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 >> v1:172.21.15.163:6813/2112279769 conn(0x7f2c84011e80 legacy=0x7f2c84014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:17.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c840034b0 osd=3 1 2024-02-19T21:08:17.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 10 client.49565.objecter close_session for osd.4 2024-02-19T21:08:17.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 >> v1:172.21.15.193:6800/3129716965 conn(0x7f2c8400ce50 legacy=0x7f2c84015c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:17.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c8400dfd0 osd=4 1 2024-02-19T21:08:17.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 10 client.49565.objecter close_session for osd.5 2024-02-19T21:08:17.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9c949700 1 -- 172.21.15.163:0/3180848905 reap_dead start 2024-02-19T21:08:17.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 >> v1:172.21.15.193:6804/805161991 conn(0x7f2c84017a30 legacy=0x7f2c8401d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:17.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.904+0000 7f2c9c949700 1 -- 172.21.15.163:0/3180848905 reap_dead start 2024-02-19T21:08:17.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c8400e180 osd=5 1 2024-02-19T21:08:17.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9c949700 1 -- 172.21.15.163:0/3180848905 reap_dead start 2024-02-19T21:08:17.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 10 client.49565.objecter close_session for osd.6 2024-02-19T21:08:17.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 >> v1:172.21.15.193:6808/787604615 conn(0x7f2c8401f0a0 legacy=0x7f2c840214f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:17.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84018010 osd=6 1 2024-02-19T21:08:17.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 10 client.49565.objecter close_session for osd.7 2024-02-19T21:08:17.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 >> v1:172.21.15.193:6812/157927432 conn(0x7f2c84024c30 legacy=0x7f2c84022da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:17.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 20 client.49565.objecter put_session s=0x7f2c84024550 osd=7 1 2024-02-19T21:08:17.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 20 client.49565.objecter shutdown clearing up homeless session... 2024-02-19T21:08:17.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 10 client.49565.objecter successfully canceled tick 2024-02-19T21:08:17.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 >> v1:172.21.15.163:6800/1552514681 conn(0x7f2c800680d0 legacy=0x7f2c8006a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:17.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 >> v1:172.21.15.163:6790/0 conn(0x7f2c98100780 legacy=0x7f2c980fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:17.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 shutdown_connections 2024-02-19T21:08:17.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 >> 172.21.15.163:0/3180848905 conn(0x7f2c980f35a0 msgr2=0x7f2c98107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:17.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 shutdown_connections 2024-02-19T21:08:17.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 1 -- 172.21.15.163:0/3180848905 wait complete. 2024-02-19T21:08:17.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:17.905+0000 7f2c9e3ac700 1 librados: shutdown 2024-02-19T21:08:17.905 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:08:17.905 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:08:17.905 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:08:17.905 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:08:17.905 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:08:17.906 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:08:17.906 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:08:17.906 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:08:18.067 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:08:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:08:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:08:18.809 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:18.811 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:17.179453+0000 mgr.y (mgr.14550) 306 : cluster [DBG] pgmap v446: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 16 B/s, 7 objects/s recovering 2024-02-19T21:08:18.811 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:17.716661+0000 mon.c (mon.2) 345 : audit [DBG] from='client.? v1:172.21.15.163:0/3180848905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:08:18.812 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:18.229661+0000 mon.a (mon.0) 2149 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi163-83854-55"}]': finished 2024-02-19T21:08:18.812 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:18.229730+0000 mon.a (mon.0) 2150 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:08:18.812 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:18.229791+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-02-19T21:08:18.812 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:18.232657+0000 mon.b (mon.1) 488 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-19T21:08:18.812 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:18.232988+0000 mon.a (mon.0) 2152 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-19T21:08:18.812 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:18.247135+0000 mon.a (mon.0) 2153 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi163-83854-56"}]: dispatch 2024-02-19T21:08:18.812 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:18.248221+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi163-83854-56"}]: dispatch 2024-02-19T21:08:18.812 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:18.248480+0000 mon.a (mon.0) 2155 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi163-83854-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:18.812 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:17.179453+0000 mgr.y (mgr.14550) 306 : cluster [DBG] pgmap v446: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 16 B/s, 7 objects/s recovering 2024-02-19T21:08:18.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:17.716661+0000 mon.c (mon.2) 345 : audit [DBG] from='client.? v1:172.21.15.163:0/3180848905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:08:18.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:18.229661+0000 mon.a (mon.0) 2149 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi163-83854-55"}]': finished 2024-02-19T21:08:18.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:18.229730+0000 mon.a (mon.0) 2150 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:08:18.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:18.229791+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-02-19T21:08:18.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:18.232657+0000 mon.b (mon.1) 488 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-19T21:08:18.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:18.232988+0000 mon.a (mon.0) 2152 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-19T21:08:18.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:18.247135+0000 mon.a (mon.0) 2153 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi163-83854-56"}]: dispatch 2024-02-19T21:08:18.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:18.248221+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi163-83854-56"}]: dispatch 2024-02-19T21:08:18.813 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:18.248480+0000 mon.a (mon.0) 2155 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi163-83854-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:18.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:18.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:18.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:17.179453+0000 mgr.y (mgr.14550) 306 : cluster [DBG] pgmap v446: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 16 B/s, 7 objects/s recovering 2024-02-19T21:08:18.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:17.716661+0000 mon.c (mon.2) 345 : audit [DBG] from='client.? v1:172.21.15.163:0/3180848905' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:08:18.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:18.229661+0000 mon.a (mon.0) 2149 : audit [INF] from='client.? v1:172.21.15.163:0/1659723686' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi163-83854-55"}]': finished 2024-02-19T21:08:18.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:18.229730+0000 mon.a (mon.0) 2150 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:08:18.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:18.229791+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2024-02-19T21:08:18.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:18.232657+0000 mon.b (mon.1) 488 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-19T21:08:18.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:18.232988+0000 mon.a (mon.0) 2152 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_count","val": "3"}]: dispatch 2024-02-19T21:08:18.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:18.247135+0000 mon.a (mon.0) 2153 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi163-83854-56"}]: dispatch 2024-02-19T21:08:18.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:18.248221+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi163-83854-56"}]: dispatch 2024-02-19T21:08:18.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:18.248480+0000 mon.a (mon.0) 2155 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi163-83854-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:18.914 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:08:18.937 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:08:18.938 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:08:18.945 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:19.694 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:08:19.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:19.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:08:19.798 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:19.833 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:19.851 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:19.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:19.955 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:19.987 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:20.008 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:20.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:20.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:20.089 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:20.112 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:20.117 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-19T21:08:20.129 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:20.141 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:20.161 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:20.163 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:20.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:20.265 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:20.293 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:20.311 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:20.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:20.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:20.415 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:20.446 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:08:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:08:20.446 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:20.461 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:20.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:20.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:20.565 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:20.591 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:20.609 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:20.611 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:20.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:20.712 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:20.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:19.180223+0000 mgr.y (mgr.14550) 307 : cluster [DBG] pgmap v449: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:08:20.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:19.230553+0000 mon.a (mon.0) 2156 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_count","val": "3"}]': finished 2024-02-19T21:08:20.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:19.230643+0000 mon.a (mon.0) 2157 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi163-83854-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:20.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:19.230697+0000 mon.a (mon.0) 2158 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-02-19T21:08:20.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:19.231686+0000 mon.b (mon.1) 489 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-19T21:08:20.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:19.232506+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi163-83854-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi163-83854-56"}]: dispatch 2024-02-19T21:08:20.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:19.233234+0000 mon.a (mon.0) 2160 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-19T21:08:20.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:19.549622+0000 mon.a (mon.0) 2161 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:20.738 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:20.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:19.180223+0000 mgr.y (mgr.14550) 307 : cluster [DBG] pgmap v449: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:08:20.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:19.230553+0000 mon.a (mon.0) 2156 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_count","val": "3"}]': finished 2024-02-19T21:08:20.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:19.230643+0000 mon.a (mon.0) 2157 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi163-83854-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:20.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:19.230697+0000 mon.a (mon.0) 2158 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-02-19T21:08:20.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:19.231686+0000 mon.b (mon.1) 489 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-19T21:08:20.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:19.232506+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi163-83854-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi163-83854-56"}]: dispatch 2024-02-19T21:08:20.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:19.233234+0000 mon.a (mon.0) 2160 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-19T21:08:20.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:19.549622+0000 mon.a (mon.0) 2161 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:20.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:19.180223+0000 mgr.y (mgr.14550) 307 : cluster [DBG] pgmap v449: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:08:20.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:19.230553+0000 mon.a (mon.0) 2156 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_count","val": "3"}]': finished 2024-02-19T21:08:20.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:19.230643+0000 mon.a (mon.0) 2157 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi163-83854-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:20.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:19.230697+0000 mon.a (mon.0) 2158 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2024-02-19T21:08:20.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:19.231686+0000 mon.b (mon.1) 489 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-19T21:08:20.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:19.232506+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi163-83854-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi163-83854-56"}]: dispatch 2024-02-19T21:08:20.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:19.233234+0000 mon.a (mon.0) 2160 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_period","val": "3"}]: dispatch 2024-02-19T21:08:20.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:19.549622+0000 mon.a (mon.0) 2161 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:20.757 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:20.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:20.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:20.860 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:08:20.889 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:20.285602+0000 mgr.y (mgr.14550) 308 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:20.476711+0000 mon.a (mon.0) 2162 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_period","val": "3"}]': finished 2024-02-19T21:08:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:20.476799+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-02-19T21:08:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:20.477583+0000 mon.b (mon.1) 490 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:08:21.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:20.483355+0000 mon.a (mon.0) 2164 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:08:21.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:21.233947+0000 mon.a (mon.0) 2165 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi163-83854-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi163-83854-56"}]': finished 2024-02-19T21:08:21.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:21.234037+0000 mon.a (mon.0) 2166 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:08:21.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:21.234102+0000 mon.a (mon.0) 2167 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-02-19T21:08:21.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:21.238747+0000 mon.b (mon.1) 491 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-19T21:08:21.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:21.240515+0000 mon.a (mon.0) 2168 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-19T21:08:21.659 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:20.285602+0000 mgr.y (mgr.14550) 308 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:21.659 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:20.476711+0000 mon.a (mon.0) 2162 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_period","val": "3"}]': finished 2024-02-19T21:08:21.659 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:20.476799+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-02-19T21:08:21.659 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:20.477583+0000 mon.b (mon.1) 490 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:08:21.659 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:20.483355+0000 mon.a (mon.0) 2164 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:08:21.659 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:21.233947+0000 mon.a (mon.0) 2165 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi163-83854-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi163-83854-56"}]': finished 2024-02-19T21:08:21.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:21.234037+0000 mon.a (mon.0) 2166 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:08:21.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:21.234102+0000 mon.a (mon.0) 2167 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-02-19T21:08:21.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:21.238747+0000 mon.b (mon.1) 491 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-19T21:08:21.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:21.240515+0000 mon.a (mon.0) 2168 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-19T21:08:21.693 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:08:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:20.285602+0000 mgr.y (mgr.14550) 308 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:20.476711+0000 mon.a (mon.0) 2162 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_period","val": "3"}]': finished 2024-02-19T21:08:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:20.476799+0000 mon.a (mon.0) 2163 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2024-02-19T21:08:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:20.477583+0000 mon.b (mon.1) 490 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:08:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:20.483355+0000 mon.a (mon.0) 2164 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:08:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:21.233947+0000 mon.a (mon.0) 2165 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi163-83854-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi163-83854-56"}]': finished 2024-02-19T21:08:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:21.234037+0000 mon.a (mon.0) 2166 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:08:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:21.234102+0000 mon.a (mon.0) 2167 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2024-02-19T21:08:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:21.238747+0000 mon.b (mon.1) 491 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-19T21:08:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:21.240515+0000 mon.a (mon.0) 2168 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-19T21:08:22.459 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:08:22.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:22.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:08:22.564 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:22.596 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:22.617 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:22.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:22.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:22.721 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:22.736 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:22.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:21.180603+0000 mgr.y (mgr.14550) 309 : cluster [DBG] pgmap v452: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:08:22.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:22.234614+0000 mon.a (mon.0) 2169 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-19T21:08:22.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:22.234715+0000 mon.a (mon.0) 2170 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-02-19T21:08:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:21.180603+0000 mgr.y (mgr.14550) 309 : cluster [DBG] pgmap v452: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:08:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:22.234614+0000 mon.a (mon.0) 2169 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-19T21:08:22.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:22.234715+0000 mon.a (mon.0) 2170 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-02-19T21:08:22.748 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:21.180603+0000 mgr.y (mgr.14550) 309 : cluster [DBG] pgmap v452: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:08:22.748 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:22.234614+0000 mon.a (mon.0) 2169 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-43","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-19T21:08:22.749 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:22.234715+0000 mon.a (mon.0) 2170 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2024-02-19T21:08:22.750 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:22.770 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:22.773 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:22.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:22.875 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:22.903 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:22.923 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:22.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:22.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:23.027 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:23.055 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:23.075 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:23.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:23.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:23.178 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:23.206 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:23.225 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:23.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:23.328 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:08:23.358 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:23.380 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:08:23.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:23.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:08:23.484 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:08:23.511 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:23.642 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:08:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:08:23] "GET /metrics HTTP/1.1" 200 191055 "" "Prometheus/2.33.4" 2024-02-19T21:08:23.660 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:08:23.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:23.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:08:23.764 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:23.794 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:23.813 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:23.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:23.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:23.920 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:23.947 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:23.966 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:23.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:23.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:24.069 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:08:24.100 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:24.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.126+0000 7f55ae64a700 1 -- 172.21.15.163:0/2857978835 >> v1:172.21.15.163:6789/0 conn(0x7f55a80f8920 legacy=0x7f55a80f8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:24.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.126+0000 7f55ae64a700 1 -- 172.21.15.163:0/2857978835 shutdown_connections 2024-02-19T21:08:24.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.126+0000 7f55ae64a700 1 -- 172.21.15.163:0/2857978835 >> 172.21.15.163:0/2857978835 conn(0x7f55a80f2470 msgr2=0x7f55a80f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:24.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.126+0000 7f55ae64a700 1 -- 172.21.15.163:0/2857978835 shutdown_connections 2024-02-19T21:08:24.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.126+0000 7f55ae64a700 1 -- 172.21.15.163:0/2857978835 wait complete. 2024-02-19T21:08:24.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.127+0000 7f55ae64a700 1 librados: starting msgr at 2024-02-19T21:08:24.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.127+0000 7f55ae64a700 1 librados: starting objecter 2024-02-19T21:08:24.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.127+0000 7f55ae64a700 1 Processor -- start 2024-02-19T21:08:24.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.127+0000 7f55ae64a700 1 -- start start 2024-02-19T21:08:24.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.127+0000 7f55ae64a700 1 librados: setting wanted keys 2024-02-19T21:08:24.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.127+0000 7f55ae64a700 1 librados: calling monclient init 2024-02-19T21:08:24.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.127+0000 7f55ae64a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f55a81172d0 con 0x7f55a81042b0 2024-02-19T21:08:24.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55ae64a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f55a81184b0 con 0x7f55a80f8920 2024-02-19T21:08:24.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55ae64a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f55a81196b0 con 0x7f55a80f94d0 2024-02-19T21:08:24.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a77fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f55a80f94d0 0x7f55a80ffb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:57404/0 (socket says 172.21.15.163:57404) 2024-02-19T21:08:24.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a77fe700 1 -- 172.21.15.163:0/1912779757 learned_addr learned my addr 172.21.15.163:0/1912779757 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:24.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 10 client.?.objecter ms_handle_connect 0x7f55a80f94d0 2024-02-19T21:08:24.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:24.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 10 client.?.objecter ms_handle_connect 0x7f55a80f8920 2024-02-19T21:08:24.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:24.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 10 client.?.objecter ms_handle_connect 0x7f55a81042b0 2024-02-19T21:08:24.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:24.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333390670 0 0) 0x7f55a81184b0 con 0x7f55a80f8920 2024-02-19T21:08:24.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5590003800 con 0x7f55a80f8920 2024-02-19T21:08:24.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981606091 0 0) 0x7f55a81196b0 con 0x7f55a80f94d0 2024-02-19T21:08:24.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a81184b0 con 0x7f55a80f94d0 2024-02-19T21:08:24.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602001669 0 0) 0x7f55a81172d0 con 0x7f55a81042b0 2024-02-19T21:08:24.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a81196b0 con 0x7f55a81042b0 2024-02-19T21:08:24.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 645892461 0 0) 0x7f5590003800 con 0x7f55a80f8920 2024-02-19T21:08:24.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55a81172d0 con 0x7f55a80f8920 2024-02-19T21:08:24.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5598002f20 con 0x7f55a80f8920 2024-02-19T21:08:24.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 458936554 0 0) 0x7f55a81184b0 con 0x7f55a80f94d0 2024-02-19T21:08:24.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5590003800 con 0x7f55a80f94d0 2024-02-19T21:08:24.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f559c0019f0 con 0x7f55a80f94d0 2024-02-19T21:08:24.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 754598106 0 0) 0x7f55a81196b0 con 0x7f55a81042b0 2024-02-19T21:08:24.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55a81184b0 con 0x7f55a81042b0 2024-02-19T21:08:24.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f55a0001c40 con 0x7f55a81042b0 2024-02-19T21:08:24.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 131641872 0 0) 0x7f55a81172d0 con 0x7f55a80f8920 2024-02-19T21:08:24.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 >> v1:172.21.15.163:6790/0 conn(0x7f55a80f94d0 legacy=0x7f55a80ffb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:24.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 >> v1:172.21.15.163:6789/0 conn(0x7f55a81042b0 legacy=0x7f55a8115990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:24.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a811a8b0 con 0x7f55a80f8920 2024-02-19T21:08:24.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55ae64a700 1 -- 172.21.15.163:0/1912779757 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a81186a0 con 0x7f55a80f8920 2024-02-19T21:08:24.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55ae64a700 10 client.48397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:24.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55ae64a700 1 -- 172.21.15.163:0/1912779757 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a8118b10 con 0x7f55a80f8920 2024-02-19T21:08:24.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.128+0000 7f55ae64a700 1 librados: init done 2024-02-19T21:08:24.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.129+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f55980022f0 con 0x7f55a80f8920 2024-02-19T21:08:24.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.129+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f55980037d0 con 0x7f55a80f8920 2024-02-19T21:08:24.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.129+0000 7f55ae64a700 1 -- 172.21.15.163:0/1912779757 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5594004f20 con 0x7f55a80f8920 2024-02-19T21:08:24.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.130+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f5598016660 con 0x7f55a80f8920 2024-02-19T21:08:24.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.130+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(323..323 src has 43..323) v4 ==== 8350+0+0 (unknown 2966782297 0 0) 0x7f55980767a0 con 0x7f55a80f8920 2024-02-19T21:08:24.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.130+0000 7f55a57fa700 10 client.48397.objecter ms_dispatch 0x7f55a8001550 osd_map(323..323 src has 43..323) v4 2024-02-19T21:08:24.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.130+0000 7f55a57fa700 3 client.48397.objecter handle_osd_map got epochs [323,323] > 0 2024-02-19T21:08:24.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.130+0000 7f55a57fa700 3 client.48397.objecter handle_osd_map decoding full epoch 323 2024-02-19T21:08:24.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.130+0000 7f55a57fa700 20 client.48397.objecter dump_active .. 0 homeless 2024-02-19T21:08:24.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.130+0000 7f55a57fa700 10 client.48397.objecter ms_handle_connect 0x7f5590063cd0 2024-02-19T21:08:24.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.134+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5598049a70 con 0x7f55a80f8920 2024-02-19T21:08:24.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.331+0000 7f55ae64a700 1 -- 172.21.15.163:0/1912779757 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd rm-pg-upmap-items", "pgid": "4.12"} v 0) v1 -- 0x7f55940051d0 con 0x7f55a80f8920 2024-02-19T21:08:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:23.181145+0000 mgr.y (mgr.14550) 310 : cluster [DBG] pgmap v455: 236 pgs: 8 unknown, 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:08:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:23.238221+0000 mon.a (mon.0) 2171 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-02-19T21:08:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:23.241117+0000 mon.a (mon.0) 2172 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi163-83854-56"}]: dispatch 2024-02-19T21:08:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:24.099885+0000 mon.a (mon.0) 2173 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:24.224198+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi163-83854-56"}]': finished 2024-02-19T21:08:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:24.224267+0000 mon.a (mon.0) 2175 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-02-19T21:08:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:24.225154+0000 mon.a (mon.0) 2176 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi163-83854-56"}]: dispatch 2024-02-19T21:08:24.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:23.181145+0000 mgr.y (mgr.14550) 310 : cluster [DBG] pgmap v455: 236 pgs: 8 unknown, 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:08:24.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:23.238221+0000 mon.a (mon.0) 2171 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-02-19T21:08:24.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:23.241117+0000 mon.a (mon.0) 2172 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi163-83854-56"}]: dispatch 2024-02-19T21:08:24.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:24.099885+0000 mon.a (mon.0) 2173 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:24.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:24.224198+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi163-83854-56"}]': finished 2024-02-19T21:08:24.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:24.224267+0000 mon.a (mon.0) 2175 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-02-19T21:08:24.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:24.225154+0000 mon.a (mon.0) 2176 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi163-83854-56"}]: dispatch 2024-02-19T21:08:24.567 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:23.181145+0000 mgr.y (mgr.14550) 310 : cluster [DBG] pgmap v455: 236 pgs: 8 unknown, 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:08:24.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:23.238221+0000 mon.a (mon.0) 2171 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2024-02-19T21:08:24.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:23.241117+0000 mon.a (mon.0) 2172 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi163-83854-56"}]: dispatch 2024-02-19T21:08:24.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:24.099885+0000 mon.a (mon.0) 2173 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:24.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:24.224198+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi163-83854-56"}]': finished 2024-02-19T21:08:24.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:24.224267+0000 mon.a (mon.0) 2175 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2024-02-19T21:08:24.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:24.225154+0000 mon.a (mon.0) 2176 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi163-83854-56"}]: dispatch 2024-02-19T21:08:24.649 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:08:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:24.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:08:24.753 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:24.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.430+0000 7f5f5e1d0700 1 -- 172.21.15.163:0/2449124662 >> v1:172.21.15.163:6790/0 conn(0x7f5f58102910 legacy=0x7f5f58104da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:24.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.430+0000 7f5f5e1d0700 1 -- 172.21.15.163:0/2449124662 shutdown_connections 2024-02-19T21:08:24.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.430+0000 7f5f5e1d0700 1 -- 172.21.15.163:0/2449124662 >> 172.21.15.163:0/2449124662 conn(0x7f5f5806fdc0 msgr2=0x7f5f580f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:24.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.430+0000 7f5f5e1d0700 1 -- 172.21.15.163:0/2449124662 shutdown_connections 2024-02-19T21:08:24.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.430+0000 7f5f5e1d0700 1 -- 172.21.15.163:0/2449124662 wait complete. 2024-02-19T21:08:24.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.430+0000 7f5f5e1d0700 1 librados: starting msgr at 2024-02-19T21:08:24.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.430+0000 7f5f5e1d0700 1 librados: starting objecter 2024-02-19T21:08:24.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.430+0000 7f5f5e1d0700 1 Processor -- start 2024-02-19T21:08:24.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f5e1d0700 1 -- start start 2024-02-19T21:08:24.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f5e1d0700 1 librados: setting wanted keys 2024-02-19T21:08:24.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f5e1d0700 1 librados: calling monclient init 2024-02-19T21:08:24.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f5e1d0700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5f581194a0 con 0x7f5f580fbfb0 2024-02-19T21:08:24.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f5e1d0700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5f5811a680 con 0x7f5f58102910 2024-02-19T21:08:24.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f5e1d0700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5f5811b880 con 0x7f5f580fee00 2024-02-19T21:08:24.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f577fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f5f580fbfb0 0x7f5f58101e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:34976/0 (socket says 172.21.15.163:34976) 2024-02-19T21:08:24.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f577fe700 1 -- 172.21.15.163:0/3801601468 learned_addr learned my addr 172.21.15.163:0/3801601468 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:24.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f54ff9700 10 client.?.objecter ms_handle_connect 0x7f5f580fee00 2024-02-19T21:08:24.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f54ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:24.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f54ff9700 10 client.?.objecter ms_handle_connect 0x7f5f580fbfb0 2024-02-19T21:08:24.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f54ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:24.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353626209 0 0) 0x7f5f5811b880 con 0x7f5f580fee00 2024-02-19T21:08:24.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f40004980 con 0x7f5f580fee00 2024-02-19T21:08:24.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f54ff9700 10 client.?.objecter ms_handle_connect 0x7f5f58102910 2024-02-19T21:08:24.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f54ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:24.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432306420 0 0) 0x7f5f581194a0 con 0x7f5f580fbfb0 2024-02-19T21:08:24.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f5811b880 con 0x7f5f580fbfb0 2024-02-19T21:08:24.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637243162 0 0) 0x7f5f5811a680 con 0x7f5f58102910 2024-02-19T21:08:24.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f581194a0 con 0x7f5f58102910 2024-02-19T21:08:24.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 936685350 0 0) 0x7f5f40004980 con 0x7f5f580fee00 2024-02-19T21:08:24.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f5811a680 con 0x7f5f580fee00 2024-02-19T21:08:24.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.431+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5f480028f0 con 0x7f5f580fee00 2024-02-19T21:08:24.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 333800691 0 0) 0x7f5f5811b880 con 0x7f5f580fbfb0 2024-02-19T21:08:24.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f40004980 con 0x7f5f580fbfb0 2024-02-19T21:08:24.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5f4c0019f0 con 0x7f5f580fbfb0 2024-02-19T21:08:24.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3577057497 0 0) 0x7f5f581194a0 con 0x7f5f58102910 2024-02-19T21:08:24.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f5811b880 con 0x7f5f58102910 2024-02-19T21:08:24.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5f50002f60 con 0x7f5f58102910 2024-02-19T21:08:24.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3598375473 0 0) 0x7f5f5811a680 con 0x7f5f580fee00 2024-02-19T21:08:24.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 >> v1:172.21.15.193:6789/0 conn(0x7f5f58102910 legacy=0x7f5f58117c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:24.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 >> v1:172.21.15.163:6789/0 conn(0x7f5f580fbfb0 legacy=0x7f5f58101e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:24.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5811ca80 con 0x7f5f580fee00 2024-02-19T21:08:24.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f5e1d0700 1 -- 172.21.15.163:0/3801601468 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f58119690 con 0x7f5f580fee00 2024-02-19T21:08:24.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f5e1d0700 10 client.49583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:24.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f5e1d0700 1 -- 172.21.15.163:0/3801601468 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f58119b80 con 0x7f5f580fee00 2024-02-19T21:08:24.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f5e1d0700 1 librados: init done 2024-02-19T21:08:24.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f5f480017d0 con 0x7f5f580fee00 2024-02-19T21:08:24.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.432+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5f48003310 con 0x7f5f580fee00 2024-02-19T21:08:24.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.433+0000 7f5f5e1d0700 1 -- 172.21.15.163:0/3801601468 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f44004f20 con 0x7f5f580fee00 2024-02-19T21:08:24.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.433+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f5f48001330 con 0x7f5f580fee00 2024-02-19T21:08:24.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.433+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(324..324 src has 43..324) v4 ==== 8067+0+0 (unknown 4144317709 0 0) 0x7f5f48076640 con 0x7f5f580fee00 2024-02-19T21:08:24.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.433+0000 7f5f54ff9700 10 client.49583.objecter ms_dispatch 0x7f5f58001550 osd_map(324..324 src has 43..324) v4 2024-02-19T21:08:24.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.433+0000 7f5f54ff9700 3 client.49583.objecter handle_osd_map got epochs [324,324] > 0 2024-02-19T21:08:24.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.433+0000 7f5f54ff9700 3 client.49583.objecter handle_osd_map decoding full epoch 324 2024-02-19T21:08:24.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.433+0000 7f5f54ff9700 20 client.49583.objecter dump_active .. 0 homeless 2024-02-19T21:08:24.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.434+0000 7f5f54ff9700 10 client.49583.objecter ms_handle_connect 0x7f5f40063cb0 2024-02-19T21:08:24.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.438+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5f48049a30 con 0x7f5f580fee00 2024-02-19T21:08:24.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:24.618+0000 7f5f5e1d0700 1 -- 172.21.15.163:0/3801601468 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f5f440051d0 con 0x7f5f580fee00 2024-02-19T21:08:24.766 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:24.779 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:24.798 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:24.800 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:24.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:24.901 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:24.929 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:24.967 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:24.971 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:24.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:25.073 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:08:25.101 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:25.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.225+0000 7f55a57fa700 1 -- 172.21.15.163:0/1912779757 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap-items", "pgid": "4.12"}]=0 clear 4.12 pg_upmap_items mapping v325) v1 ==== 118+0+0 (unknown 2288169789 0 0) 0x7f5598047ee0 con 0x7f55a80f8920 2024-02-19T21:08:25.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.226+0000 7f5f54ff9700 1 -- 172.21.15.163:0/3801601468 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v325) v1 ==== 97+0+0 (unknown 2089705596 0 0) 0x7f5f48047e40 con 0x7f5f580fee00 2024-02-19T21:08:25.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.228+0000 7f55ae64a700 10 librados: watch_flush enter 2024-02-19T21:08:25.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.229+0000 7f5f5e1d0700 10 librados: watch_flush enter 2024-02-19T21:08:25.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.229+0000 7f5f5e1d0700 10 librados: watch_flush exit 2024-02-19T21:08:25.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.229+0000 7f5f5e1d0700 20 client.49583.objecter shutdown clearing up homeless session... 2024-02-19T21:08:25.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.229+0000 7f5f5e1d0700 10 client.49583.objecter successfully canceled tick 2024-02-19T21:08:25.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.230+0000 7f55ae64a700 10 librados: watch_flush exit 2024-02-19T21:08:25.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.230+0000 7f55ae64a700 20 client.48397.objecter shutdown clearing up homeless session... 2024-02-19T21:08:25.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.230+0000 7f55ae64a700 10 client.48397.objecter successfully canceled tick 2024-02-19T21:08:25.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.230+0000 7f5f5e1d0700 1 -- 172.21.15.163:0/3801601468 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5f40063cb0 legacy=0x7f5f40066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:25.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.230+0000 7f55ae64a700 1 -- 172.21.15.163:0/1912779757 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5590063cd0 legacy=0x7f5590066190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:25.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.230+0000 7f5f5e1d0700 1 -- 172.21.15.163:0/3801601468 >> v1:172.21.15.163:6790/0 conn(0x7f5f580fee00 legacy=0x7f5f58114550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:25.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.230+0000 7f55ae64a700 1 -- 172.21.15.163:0/1912779757 >> v1:172.21.15.193:6789/0 conn(0x7f55a80f8920 legacy=0x7f55a80ff410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:25.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.230+0000 7f55ae64a700 1 -- 172.21.15.163:0/1912779757 shutdown_connections 2024-02-19T21:08:25.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.230+0000 7f55ae64a700 1 -- 172.21.15.163:0/1912779757 >> 172.21.15.163:0/1912779757 conn(0x7f55a80f2470 msgr2=0x7f55a8107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:25.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.230+0000 7f55ae64a700 1 -- 172.21.15.163:0/1912779757 shutdown_connections 2024-02-19T21:08:25.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.230+0000 7f55ae64a700 1 -- 172.21.15.163:0/1912779757 wait complete. 2024-02-19T21:08:25.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.230+0000 7f55ae64a700 1 librados: shutdown 2024-02-19T21:08:25.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.231+0000 7f5f5e1d0700 1 -- 172.21.15.163:0/3801601468 shutdown_connections 2024-02-19T21:08:25.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.231+0000 7f5f5e1d0700 1 -- 172.21.15.163:0/3801601468 >> 172.21.15.163:0/3801601468 conn(0x7f5f5806fdc0 msgr2=0x7f5f5806c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:25.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.231+0000 7f5f5e1d0700 1 -- 172.21.15.163:0/3801601468 shutdown_connections 2024-02-19T21:08:25.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.231+0000 7f5f5e1d0700 1 -- 172.21.15.163:0/3801601468 wait complete. 2024-02-19T21:08:25.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:25.231+0000 7f5f5e1d0700 1 librados: shutdown 2024-02-19T21:08:25.229 INFO:teuthology.orchestra.run.smithi163.stderr:clear 4.12 pg_upmap_items mapping 2024-02-19T21:08:25.229 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:08:25.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:24.332239+0000 mon.b (mon.1) 492 : audit [INF] from='client.? v1:172.21.15.163:0/1912779757' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.12"}]: dispatch 2024-02-19T21:08:25.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:24.332850+0000 mon.a (mon.0) 2177 : audit [INF] from='client.48397 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.12"}]: dispatch 2024-02-19T21:08:25.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:24.560685+0000 mon.a (mon.0) 2178 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:25.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:24.618660+0000 mon.c (mon.2) 346 : audit [INF] from='client.? v1:172.21.15.163:0/3801601468' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:25.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:24.618979+0000 mon.a (mon.0) 2179 : audit [INF] from='client.49583 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:25.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:25.220907+0000 mon.a (mon.0) 2180 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:08:25.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:25.224897+0000 mon.a (mon.0) 2181 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi163-83854-56"}]': finished 2024-02-19T21:08:25.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:25.224958+0000 mon.a (mon.0) 2182 : audit [INF] from='client.48397 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.12"}]': finished 2024-02-19T21:08:25.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:25.225011+0000 mon.a (mon.0) 2183 : audit [INF] from='client.49583 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:08:25.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:25.225057+0000 mon.a (mon.0) 2184 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-02-19T21:08:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:24.332239+0000 mon.b (mon.1) 492 : audit [INF] from='client.? v1:172.21.15.163:0/1912779757' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.12"}]: dispatch 2024-02-19T21:08:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:24.332850+0000 mon.a (mon.0) 2177 : audit [INF] from='client.48397 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.12"}]: dispatch 2024-02-19T21:08:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:24.560685+0000 mon.a (mon.0) 2178 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:24.618660+0000 mon.c (mon.2) 346 : audit [INF] from='client.? v1:172.21.15.163:0/3801601468' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:25.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:24.618979+0000 mon.a (mon.0) 2179 : audit [INF] from='client.49583 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:25.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:25.220907+0000 mon.a (mon.0) 2180 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:08:25.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:25.224897+0000 mon.a (mon.0) 2181 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi163-83854-56"}]': finished 2024-02-19T21:08:25.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:25.224958+0000 mon.a (mon.0) 2182 : audit [INF] from='client.48397 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.12"}]': finished 2024-02-19T21:08:25.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:25.225011+0000 mon.a (mon.0) 2183 : audit [INF] from='client.49583 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:08:25.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:25.225057+0000 mon.a (mon.0) 2184 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-02-19T21:08:25.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:24.332239+0000 mon.b (mon.1) 492 : audit [INF] from='client.? v1:172.21.15.163:0/1912779757' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.12"}]: dispatch 2024-02-19T21:08:25.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:24.332850+0000 mon.a (mon.0) 2177 : audit [INF] from='client.48397 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.12"}]: dispatch 2024-02-19T21:08:25.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:24.560685+0000 mon.a (mon.0) 2178 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:25.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:24.618660+0000 mon.c (mon.2) 346 : audit [INF] from='client.? v1:172.21.15.163:0/3801601468' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:25.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:24.618979+0000 mon.a (mon.0) 2179 : audit [INF] from='client.49583 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:25.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:25.220907+0000 mon.a (mon.0) 2180 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:08:25.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:25.224897+0000 mon.a (mon.0) 2181 : audit [INF] from='client.? v1:172.21.15.163:0/3022514242' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi163-83854-56"}]': finished 2024-02-19T21:08:25.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:25.224958+0000 mon.a (mon.0) 2182 : audit [INF] from='client.48397 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.12"}]': finished 2024-02-19T21:08:25.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:25.225011+0000 mon.a (mon.0) 2183 : audit [INF] from='client.49583 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:08:25.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:25.225057+0000 mon.a (mon.0) 2184 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2024-02-19T21:08:26.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:25.182156+0000 mgr.y (mgr.14550) 311 : cluster [DBG] pgmap v458: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-02-19T21:08:26.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:25.252213+0000 mon.a (mon.0) 2185 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi163-83854-57"}]: dispatch 2024-02-19T21:08:26.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:25.253509+0000 mon.a (mon.0) 2186 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi163-83854-57"}]: dispatch 2024-02-19T21:08:26.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:25.253840+0000 mon.a (mon.0) 2187 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi163-83854-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:26.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:25.709441+0000 mon.b (mon.1) 493 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:08:26.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:26.226527+0000 mon.a (mon.0) 2188 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi163-83854-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:26.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:26.226589+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-02-19T21:08:26.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:26.227611+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi163-83854-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi163-83854-57"}]: dispatch 2024-02-19T21:08:26.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:25.182156+0000 mgr.y (mgr.14550) 311 : cluster [DBG] pgmap v458: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-02-19T21:08:26.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:25.252213+0000 mon.a (mon.0) 2185 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi163-83854-57"}]: dispatch 2024-02-19T21:08:26.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:25.253509+0000 mon.a (mon.0) 2186 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi163-83854-57"}]: dispatch 2024-02-19T21:08:26.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:25.253840+0000 mon.a (mon.0) 2187 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi163-83854-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:26.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:25.709441+0000 mon.b (mon.1) 493 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:08:26.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:26.226527+0000 mon.a (mon.0) 2188 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi163-83854-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:26.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:26.226589+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-02-19T21:08:26.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:26.227611+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi163-83854-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi163-83854-57"}]: dispatch 2024-02-19T21:08:26.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:25.182156+0000 mgr.y (mgr.14550) 311 : cluster [DBG] pgmap v458: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-02-19T21:08:26.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:25.252213+0000 mon.a (mon.0) 2185 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi163-83854-57"}]: dispatch 2024-02-19T21:08:26.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:25.253509+0000 mon.a (mon.0) 2186 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi163-83854-57"}]: dispatch 2024-02-19T21:08:26.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:25.253840+0000 mon.a (mon.0) 2187 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi163-83854-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:26.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:25.709441+0000 mon.b (mon.1) 493 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:08:26.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:26.226527+0000 mon.a (mon.0) 2188 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi163-83854-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:26.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:26.226589+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2024-02-19T21:08:26.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:26.227611+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi163-83854-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi163-83854-57"}]: dispatch 2024-02-19T21:08:27.556 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:08:27.557 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:08:27.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:26.420391+0000 mon.b (mon.1) 494 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:08:27.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:26.421144+0000 mon.b (mon.1) 495 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:08:27.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:26.425450+0000 mon.a (mon.0) 2191 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:27.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:26.420391+0000 mon.b (mon.1) 494 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:08:27.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:26.421144+0000 mon.b (mon.1) 495 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:08:27.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:26.425450+0000 mon.a (mon.0) 2191 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:27.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:26.420391+0000 mon.b (mon.1) 494 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:08:27.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:26.421144+0000 mon.b (mon.1) 495 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:08:27.770 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:26.425450+0000 mon.a (mon.0) 2191 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:27.854 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:08:27.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:27.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:08:27.959 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:08:27.990 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:28.067 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:08:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:08:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:08:28.682 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:27.182748+0000 mgr.y (mgr.14550) 312 : cluster [DBG] pgmap v461: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-02-19T21:08:28.682 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:27.469235+0000 mon.a (mon.0) 2192 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-02-19T21:08:28.682 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:28.229943+0000 mon.a (mon.0) 2193 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi163-83854-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi163-83854-57"}]': finished 2024-02-19T21:08:28.682 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:28.230024+0000 mon.a (mon.0) 2194 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-02-19T21:08:28.682 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:27.182748+0000 mgr.y (mgr.14550) 312 : cluster [DBG] pgmap v461: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-02-19T21:08:28.682 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:27.469235+0000 mon.a (mon.0) 2192 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-02-19T21:08:28.682 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:28.229943+0000 mon.a (mon.0) 2193 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi163-83854-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi163-83854-57"}]': finished 2024-02-19T21:08:28.682 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:28.230024+0000 mon.a (mon.0) 2194 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-02-19T21:08:28.703 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-19T21:08:28.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:27.182748+0000 mgr.y (mgr.14550) 312 : cluster [DBG] pgmap v461: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-02-19T21:08:28.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:27.469235+0000 mon.a (mon.0) 2192 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2024-02-19T21:08:28.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:28.229943+0000 mon.a (mon.0) 2193 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi163-83854-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi163-83854-57"}]': finished 2024-02-19T21:08:28.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:28.230024+0000 mon.a (mon.0) 2194 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2024-02-19T21:08:29.139 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:08:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:29.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:08:29.244 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:29.278 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:29.298 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:29.300 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:29.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:29.401 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:29.430 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:29.450 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:29.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:29.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:29.554 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:29.581 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:29.601 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:29.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:29.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:29.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.659+0000 7f63d0350700 1 -- 172.21.15.163:0/2458419722 >> v1:172.21.15.163:6789/0 conn(0x7f63c80fc870 legacy=0x7f63c80fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:29.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.659+0000 7f63d0350700 1 -- 172.21.15.163:0/2458419722 shutdown_connections 2024-02-19T21:08:29.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.659+0000 7f63d0350700 1 -- 172.21.15.163:0/2458419722 >> 172.21.15.163:0/2458419722 conn(0x7f63c80f24b0 msgr2=0x7f63c80f48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:29.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.659+0000 7f63d0350700 1 -- 172.21.15.163:0/2458419722 shutdown_connections 2024-02-19T21:08:29.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.659+0000 7f63d0350700 1 -- 172.21.15.163:0/2458419722 wait complete. 2024-02-19T21:08:29.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.659+0000 7f63d0350700 1 librados: starting msgr at 2024-02-19T21:08:29.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.659+0000 7f63d0350700 1 librados: starting objecter 2024-02-19T21:08:29.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.659+0000 7f63d0350700 1 Processor -- start 2024-02-19T21:08:29.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.659+0000 7f63d0350700 1 -- start start 2024-02-19T21:08:29.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.659+0000 7f63d0350700 1 librados: setting wanted keys 2024-02-19T21:08:29.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.659+0000 7f63d0350700 1 librados: calling monclient init 2024-02-19T21:08:29.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63d0350700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f63c81194f0 con 0x7f63c80fc870 2024-02-19T21:08:29.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63d0350700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f63c811a6d0 con 0x7f63c80ff670 2024-02-19T21:08:29.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63d0350700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f63c811b8d0 con 0x7f63c81031b0 2024-02-19T21:08:29.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63ce0ec700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f63c80fc870 0x7f63c8060b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:35024/0 (socket says 172.21.15.163:35024) 2024-02-19T21:08:29.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63ce8ed700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f63c81031b0 0x7f63c8117c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:33962/0 (socket says 172.21.15.163:33962) 2024-02-19T21:08:29.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63ce0ec700 1 -- 172.21.15.163:0/1762601906 learned_addr learned my addr 172.21.15.163:0/1762601906 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:29.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 10 client.?.objecter ms_handle_connect 0x7f63c80fc870 2024-02-19T21:08:29.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:29.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 10 client.?.objecter ms_handle_connect 0x7f63c81031b0 2024-02-19T21:08:29.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:29.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 10 client.?.objecter ms_handle_connect 0x7f63c80ff670 2024-02-19T21:08:29.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:29.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933708358 0 0) 0x7f63c81194f0 con 0x7f63c80fc870 2024-02-19T21:08:29.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b4003030 con 0x7f63c80fc870 2024-02-19T21:08:29.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690927183 0 0) 0x7f63c811b8d0 con 0x7f63c81031b0 2024-02-19T21:08:29.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c81194f0 con 0x7f63c81031b0 2024-02-19T21:08:29.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021433659 0 0) 0x7f63c811a6d0 con 0x7f63c80ff670 2024-02-19T21:08:29.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c811b8d0 con 0x7f63c80ff670 2024-02-19T21:08:29.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3174233355 0 0) 0x7f63b4003030 con 0x7f63c80fc870 2024-02-19T21:08:29.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63c811a6d0 con 0x7f63c80fc870 2024-02-19T21:08:29.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f63c40015d0 con 0x7f63c80fc870 2024-02-19T21:08:29.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3063139669 0 0) 0x7f63c81194f0 con 0x7f63c81031b0 2024-02-19T21:08:29.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63b4003030 con 0x7f63c81031b0 2024-02-19T21:08:29.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2822638400 0 0) 0x7f63c811b8d0 con 0x7f63c80ff670 2024-02-19T21:08:29.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63c81194f0 con 0x7f63c80ff670 2024-02-19T21:08:29.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f63c0001cf0 con 0x7f63c81031b0 2024-02-19T21:08:29.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f63b8001840 con 0x7f63c80ff670 2024-02-19T21:08:29.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4107268796 0 0) 0x7f63c811a6d0 con 0x7f63c80fc870 2024-02-19T21:08:29.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 >> v1:172.21.15.163:6790/0 conn(0x7f63c81031b0 legacy=0x7f63c8117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:29.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 >> v1:172.21.15.193:6789/0 conn(0x7f63c80ff670 legacy=0x7f63c8061250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:29.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c811cad0 con 0x7f63c80fc870 2024-02-19T21:08:29.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c811bac0 con 0x7f63c80fc870 2024-02-19T21:08:29.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63d0350700 10 client.22890.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:29.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c811bfb0 con 0x7f63c80fc870 2024-02-19T21:08:29.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.660+0000 7f63d0350700 1 librados: init done 2024-02-19T21:08:29.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.661+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f63c4002400 con 0x7f63c80fc870 2024-02-19T21:08:29.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.661+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f63c4003850 con 0x7f63c80fc870 2024-02-19T21:08:29.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.661+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f63ac004f00 con 0x7f63c80fc870 2024-02-19T21:08:29.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.662+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f63c4001840 con 0x7f63c80fc870 2024-02-19T21:08:29.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.662+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(329..329 src has 43..329) v4 ==== 8744+0+0 (unknown 2738753141 0 0) 0x7f63c40792e0 con 0x7f63c80fc870 2024-02-19T21:08:29.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.662+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 osd_map(329..329 src has 43..329) v4 2024-02-19T21:08:29.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.662+0000 7f63bf7fe700 3 client.22890.objecter handle_osd_map got epochs [329,329] > 0 2024-02-19T21:08:29.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.662+0000 7f63bf7fe700 3 client.22890.objecter handle_osd_map decoding full epoch 329 2024-02-19T21:08:29.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.662+0000 7f63bf7fe700 20 client.22890.objecter dump_active .. 0 homeless 2024-02-19T21:08:29.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.662+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v329) v1 ==== 54+0+16 (unknown 1573253111 0 2420000192) 0x7f63c40796c0 con 0x7f63c80fc870 2024-02-19T21:08:29.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.662+0000 7f63bf7fe700 10 client.22890.objecter ms_handle_connect 0x7f63b4063dc0 2024-02-19T21:08:29.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63d0350700 10 client.22890.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:29.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 1 2024-02-19T21:08:29.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac0051a0 osd=0 3 2024-02-19T21:08:29.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac0051a0 osd=0 3 2024-02-19T21:08:29.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac0051a0 osd=0 3 2024-02-19T21:08:29.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 1 2024-02-19T21:08:29.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac0051a0 osd=0 3 2024-02-19T21:08:29.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 0 1 2024-02-19T21:08:29.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac0051a0 osd=0 4 2024-02-19T21:08:29.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63d0350700 10 client.22890.objecter _send_command 1 2024-02-19T21:08:29.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63ac004950 con 0x7f63ac004530 2024-02-19T21:08:29.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63bf7fe700 10 client.22890.objecter ms_handle_connect 0x7f63ac004530 2024-02-19T21:08:29.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f63ac004950 con 0x7f63ac004530 2024-02-19T21:08:29.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:08:29.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 1 = system:0 2024-02-19T21:08:29.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac0051a0 osd=0 4 2024-02-19T21:08:29.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.663+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 0 1 2024-02-19T21:08:29.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.678+0000 7f63d0350700 10 client.22890.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:29.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.678+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 2 2024-02-19T21:08:29.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.678+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac0051a0 osd=0 3 2024-02-19T21:08:29.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.678+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac0051a0 osd=0 3 2024-02-19T21:08:29.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.678+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac0051a0 osd=0 3 2024-02-19T21:08:29.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.678+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 2 2024-02-19T21:08:29.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.678+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac0051a0 osd=0 3 2024-02-19T21:08:29.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.678+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 0 2 2024-02-19T21:08:29.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.678+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac0051a0 osd=0 4 2024-02-19T21:08:29.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.678+0000 7f63d0350700 10 client.22890.objecter _send_command 2 2024-02-19T21:08:29.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.678+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63ac004d30 con 0x7f63ac004530 2024-02-19T21:08:29.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f63ac004d30 con 0x7f63ac004530 2024-02-19T21:08:29.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:29.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac0051a0 osd=0 4 2024-02-19T21:08:29.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 0 2 2024-02-19T21:08:29.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63d0350700 10 client.22890.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:29.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 3 2024-02-19T21:08:29.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac004270 osd=1 3 2024-02-19T21:08:29.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac004270 osd=1 3 2024-02-19T21:08:29.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac004270 osd=1 3 2024-02-19T21:08:29.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 3 2024-02-19T21:08:29.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac004270 osd=1 3 2024-02-19T21:08:29.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 1 3 2024-02-19T21:08:29.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac004270 osd=1 4 2024-02-19T21:08:29.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63d0350700 10 client.22890.objecter _send_command 3 2024-02-19T21:08:29.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63ac00bf60 con 0x7f63ac003b40 2024-02-19T21:08:29.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.680+0000 7f63bf7fe700 10 client.22890.objecter ms_handle_connect 0x7f63ac003b40 2024-02-19T21:08:29.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.681+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f63ac00bf60 con 0x7f63ac003b40 2024-02-19T21:08:29.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.681+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:08:29.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.681+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 3 = system:0 2024-02-19T21:08:29.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.681+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac004270 osd=1 4 2024-02-19T21:08:29.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.681+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 1 3 2024-02-19T21:08:29.691 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:08:29.691 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:29.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.696+0000 7f63d0350700 10 client.22890.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:29.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.696+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 4 2024-02-19T21:08:29.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.696+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac004270 osd=1 3 2024-02-19T21:08:29.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.696+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac004270 osd=1 3 2024-02-19T21:08:29.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.696+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac004270 osd=1 3 2024-02-19T21:08:29.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.696+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 4 2024-02-19T21:08:29.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.696+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac004270 osd=1 3 2024-02-19T21:08:29.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.696+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 1 4 2024-02-19T21:08:29.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.696+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac004270 osd=1 4 2024-02-19T21:08:29.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.696+0000 7f63d0350700 10 client.22890.objecter _send_command 4 2024-02-19T21:08:29.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.696+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63ac00d280 con 0x7f63ac003b40 2024-02-19T21:08:29.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f63ac00d280 con 0x7f63ac003b40 2024-02-19T21:08:29.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:29.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac004270 osd=1 4 2024-02-19T21:08:29.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 1 4 2024-02-19T21:08:29.695 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:08:29.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63d0350700 10 client.22890.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:29.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 5 2024-02-19T21:08:29.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac0033a0 osd=2 3 2024-02-19T21:08:29.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac0033a0 osd=2 3 2024-02-19T21:08:29.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac0033a0 osd=2 3 2024-02-19T21:08:29.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 5 2024-02-19T21:08:29.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac0033a0 osd=2 3 2024-02-19T21:08:29.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 2 5 2024-02-19T21:08:29.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac0033a0 osd=2 4 2024-02-19T21:08:29.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63d0350700 10 client.22890.objecter _send_command 5 2024-02-19T21:08:29.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.698+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63ac010d00 con 0x7f63ac002070 2024-02-19T21:08:29.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.699+0000 7f63bf7fe700 10 client.22890.objecter ms_handle_connect 0x7f63ac002070 2024-02-19T21:08:29.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.699+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f63ac010d00 con 0x7f63ac002070 2024-02-19T21:08:29.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.699+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:08:29.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.699+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 5 = system:0 2024-02-19T21:08:29.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.699+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac0033a0 osd=2 4 2024-02-19T21:08:29.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.699+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 2 5 2024-02-19T21:08:29.704 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:29.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.721+0000 7f63d0350700 10 client.22890.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:29.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.721+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 6 2024-02-19T21:08:29.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.721+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac0033a0 osd=2 3 2024-02-19T21:08:29.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.721+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac0033a0 osd=2 3 2024-02-19T21:08:29.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.721+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac0033a0 osd=2 3 2024-02-19T21:08:29.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.721+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 6 2024-02-19T21:08:29.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.721+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac0033a0 osd=2 3 2024-02-19T21:08:29.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.721+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 2 6 2024-02-19T21:08:29.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.721+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac0033a0 osd=2 4 2024-02-19T21:08:29.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.721+0000 7f63d0350700 10 client.22890.objecter _send_command 6 2024-02-19T21:08:29.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.721+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63ac001630 con 0x7f63ac002070 2024-02-19T21:08:29.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.723+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f63ac001630 con 0x7f63ac002070 2024-02-19T21:08:29.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.723+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:29.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.723+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.723+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac0033a0 osd=2 4 2024-02-19T21:08:29.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.723+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 2 6 2024-02-19T21:08:29.720 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:08:29.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63d0350700 10 client.22890.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:29.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 7 2024-02-19T21:08:29.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac00c8d0 osd=3 3 2024-02-19T21:08:29.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac00c8d0 osd=3 3 2024-02-19T21:08:29.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac00c8d0 osd=3 3 2024-02-19T21:08:29.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 7 2024-02-19T21:08:29.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac00c8d0 osd=3 3 2024-02-19T21:08:29.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 3 7 2024-02-19T21:08:29.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac00c8d0 osd=3 4 2024-02-19T21:08:29.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63d0350700 10 client.22890.objecter _send_command 7 2024-02-19T21:08:29.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63ac00d0b0 con 0x7f63ac010ea0 2024-02-19T21:08:29.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63bf7fe700 10 client.22890.objecter ms_handle_connect 0x7f63ac010ea0 2024-02-19T21:08:29.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f63ac00d0b0 con 0x7f63ac010ea0 2024-02-19T21:08:29.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:08:29.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 7 = system:0 2024-02-19T21:08:29.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac00c8d0 osd=3 4 2024-02-19T21:08:29.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.724+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 3 7 2024-02-19T21:08:29.731 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:29.735 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.740+0000 7f63d0350700 10 client.22890.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:29.735 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.740+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 8 2024-02-19T21:08:29.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.740+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac00c8d0 osd=3 3 2024-02-19T21:08:29.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.740+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac00c8d0 osd=3 3 2024-02-19T21:08:29.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.740+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac00c8d0 osd=3 3 2024-02-19T21:08:29.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.740+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 8 2024-02-19T21:08:29.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.740+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac00c8d0 osd=3 3 2024-02-19T21:08:29.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.740+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 3 8 2024-02-19T21:08:29.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.740+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac00c8d0 osd=3 4 2024-02-19T21:08:29.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.740+0000 7f63d0350700 10 client.22890.objecter _send_command 8 2024-02-19T21:08:29.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.740+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63ac012230 con 0x7f63ac010ea0 2024-02-19T21:08:29.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f63ac012230 con 0x7f63ac010ea0 2024-02-19T21:08:29.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:29.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac00c8d0 osd=3 4 2024-02-19T21:08:29.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 3 8 2024-02-19T21:08:29.738 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:08:29.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63d0350700 10 client.22890.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:29.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 9 2024-02-19T21:08:29.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac012230 osd=4 3 2024-02-19T21:08:29.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac012230 osd=4 3 2024-02-19T21:08:29.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac012230 osd=4 3 2024-02-19T21:08:29.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 9 2024-02-19T21:08:29.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac012230 osd=4 3 2024-02-19T21:08:29.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 4 9 2024-02-19T21:08:29.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac012230 osd=4 4 2024-02-19T21:08:29.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63d0350700 10 client.22890.objecter _send_command 9 2024-02-19T21:08:29.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.741+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63ac01a2a0 con 0x7f63ac0166c0 2024-02-19T21:08:29.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.742+0000 7f63bf7fe700 10 client.22890.objecter ms_handle_connect 0x7f63ac0166c0 2024-02-19T21:08:29.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.743+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f63ac01a2a0 con 0x7f63ac0166c0 2024-02-19T21:08:29.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.743+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:08:29.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.743+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 9 = system:0 2024-02-19T21:08:29.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.743+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac012230 osd=4 4 2024-02-19T21:08:29.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.743+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 4 9 2024-02-19T21:08:29.750 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:29.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:29.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:29.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.758+0000 7f63d0350700 10 client.22890.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:29.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.758+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 10 2024-02-19T21:08:29.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.758+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac012230 osd=4 3 2024-02-19T21:08:29.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.758+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac012230 osd=4 3 2024-02-19T21:08:29.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.758+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac012230 osd=4 3 2024-02-19T21:08:29.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.758+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 10 2024-02-19T21:08:29.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.758+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac012230 osd=4 3 2024-02-19T21:08:29.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.758+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 4 10 2024-02-19T21:08:29.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.758+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac012230 osd=4 4 2024-02-19T21:08:29.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.758+0000 7f63d0350700 10 client.22890.objecter _send_command 10 2024-02-19T21:08:29.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.758+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63ac01ba50 con 0x7f63ac0166c0 2024-02-19T21:08:29.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.759+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f63ac01ba50 con 0x7f63ac0166c0 2024-02-19T21:08:29.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.759+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:29.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.759+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.759+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac012230 osd=4 4 2024-02-19T21:08:29.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.759+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 4 10 2024-02-19T21:08:29.756 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:08:29.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.759+0000 7f63d0350700 10 client.22890.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:29.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.760+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 11 2024-02-19T21:08:29.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.760+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac01ba50 osd=5 3 2024-02-19T21:08:29.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.760+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac01ba50 osd=5 3 2024-02-19T21:08:29.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.760+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac01ba50 osd=5 3 2024-02-19T21:08:29.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.760+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 11 2024-02-19T21:08:29.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.760+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac01ba50 osd=5 3 2024-02-19T21:08:29.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.760+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 5 11 2024-02-19T21:08:29.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.760+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac01ba50 osd=5 4 2024-02-19T21:08:29.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.760+0000 7f63d0350700 10 client.22890.objecter _send_command 11 2024-02-19T21:08:29.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.760+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63ac01c680 con 0x7f63ac01c0d0 2024-02-19T21:08:29.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.760+0000 7f63bf7fe700 10 client.22890.objecter ms_handle_connect 0x7f63ac01c0d0 2024-02-19T21:08:29.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.761+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f63ac01c680 con 0x7f63ac01c0d0 2024-02-19T21:08:29.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.761+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:08:29.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.761+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 11 = system:0 2024-02-19T21:08:29.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.761+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac01ba50 osd=5 4 2024-02-19T21:08:29.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.761+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 5 11 2024-02-19T21:08:29.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.775+0000 7f63d0350700 10 client.22890.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:29.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.775+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 12 2024-02-19T21:08:29.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.775+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac01ba50 osd=5 3 2024-02-19T21:08:29.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.775+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac01ba50 osd=5 3 2024-02-19T21:08:29.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.775+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac01ba50 osd=5 3 2024-02-19T21:08:29.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.775+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 12 2024-02-19T21:08:29.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.775+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac01ba50 osd=5 3 2024-02-19T21:08:29.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.775+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 5 12 2024-02-19T21:08:29.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.775+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac01ba50 osd=5 4 2024-02-19T21:08:29.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.775+0000 7f63d0350700 10 client.22890.objecter _send_command 12 2024-02-19T21:08:29.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.775+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63ac01c680 con 0x7f63ac01c0d0 2024-02-19T21:08:29.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.776+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f63ac01c680 con 0x7f63ac01c0d0 2024-02-19T21:08:29.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.776+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:29.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.776+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.776+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac01ba50 osd=5 4 2024-02-19T21:08:29.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.776+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 5 12 2024-02-19T21:08:29.773 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:08:29.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.777+0000 7f63d0350700 10 client.22890.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:29.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.777+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 13 2024-02-19T21:08:29.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.777+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac01c680 osd=6 3 2024-02-19T21:08:29.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.777+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac01c680 osd=6 3 2024-02-19T21:08:29.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.777+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac01c680 osd=6 3 2024-02-19T21:08:29.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.777+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 13 2024-02-19T21:08:29.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.777+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac01c680 osd=6 3 2024-02-19T21:08:29.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.777+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 6 13 2024-02-19T21:08:29.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.777+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac01c680 osd=6 4 2024-02-19T21:08:29.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.777+0000 7f63d0350700 10 client.22890.objecter _send_command 13 2024-02-19T21:08:29.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.777+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63ac022c70 con 0x7f63ac01f160 2024-02-19T21:08:29.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.777+0000 7f63bf7fe700 10 client.22890.objecter ms_handle_connect 0x7f63ac01f160 2024-02-19T21:08:29.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.778+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f63ac022c70 con 0x7f63ac01f160 2024-02-19T21:08:29.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.778+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:08:29.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.778+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 13 = system:0 2024-02-19T21:08:29.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.778+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac01c680 osd=6 4 2024-02-19T21:08:29.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.778+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 6 13 2024-02-19T21:08:29.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.792+0000 7f63d0350700 10 client.22890.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:29.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.792+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 14 2024-02-19T21:08:29.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.792+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac01c680 osd=6 3 2024-02-19T21:08:29.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.792+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac01c680 osd=6 3 2024-02-19T21:08:29.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.792+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac01c680 osd=6 3 2024-02-19T21:08:29.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.792+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 14 2024-02-19T21:08:29.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.792+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac01c680 osd=6 3 2024-02-19T21:08:29.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.792+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 6 14 2024-02-19T21:08:29.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.792+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac01c680 osd=6 4 2024-02-19T21:08:29.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.792+0000 7f63d0350700 10 client.22890.objecter _send_command 14 2024-02-19T21:08:29.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.792+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63ac022c70 con 0x7f63ac01f160 2024-02-19T21:08:29.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.793+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f63ac022c70 con 0x7f63ac01f160 2024-02-19T21:08:29.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.793+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:29.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.793+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.793+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac01c680 osd=6 4 2024-02-19T21:08:29.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.793+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 6 14 2024-02-19T21:08:29.790 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:08:29.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.794+0000 7f63d0350700 10 client.22890.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:29.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.794+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 15 2024-02-19T21:08:29.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.794+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac022c70 osd=7 3 2024-02-19T21:08:29.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.794+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac022c70 osd=7 3 2024-02-19T21:08:29.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.794+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac022c70 osd=7 3 2024-02-19T21:08:29.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.794+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 15 2024-02-19T21:08:29.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.794+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac022c70 osd=7 3 2024-02-19T21:08:29.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.794+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 7 15 2024-02-19T21:08:29.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.794+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac022c70 osd=7 4 2024-02-19T21:08:29.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.794+0000 7f63d0350700 10 client.22890.objecter _send_command 15 2024-02-19T21:08:29.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.794+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63ac026f60 con 0x7f63ac0232f0 2024-02-19T21:08:29.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.794+0000 7f63bf7fe700 10 client.22890.objecter ms_handle_connect 0x7f63ac0232f0 2024-02-19T21:08:29.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.795+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f63ac026f60 con 0x7f63ac0232f0 2024-02-19T21:08:29.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.795+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:08:29.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.795+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 15 = system:0 2024-02-19T21:08:29.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.795+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac022c70 osd=7 4 2024-02-19T21:08:29.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.795+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 7 15 2024-02-19T21:08:29.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.809+0000 7f63d0350700 10 client.22890.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:29.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.809+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign -1 16 2024-02-19T21:08:29.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.809+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac022c70 osd=7 3 2024-02-19T21:08:29.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.809+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac022c70 osd=7 3 2024-02-19T21:08:29.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.809+0000 7f63d0350700 20 client.22890.objecter _get_session s=0x7f63ac022c70 osd=7 3 2024-02-19T21:08:29.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.809+0000 7f63d0350700 15 client.22890.objecter _session_command_op_remove -1 16 2024-02-19T21:08:29.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.809+0000 7f63d0350700 20 client.22890.objecter get_session s=0x7f63ac022c70 osd=7 3 2024-02-19T21:08:29.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.809+0000 7f63d0350700 15 client.22890.objecter _session_command_op_assign 7 16 2024-02-19T21:08:29.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.809+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac022c70 osd=7 4 2024-02-19T21:08:29.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.809+0000 7f63d0350700 10 client.22890.objecter _send_command 16 2024-02-19T21:08:29.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.809+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63ac028710 con 0x7f63ac0232f0 2024-02-19T21:08:29.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63bf7fe700 1 -- 172.21.15.163:0/1762601906 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f63ac028710 con 0x7f63ac0232f0 2024-02-19T21:08:29.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63bf7fe700 10 client.22890.objecter ms_dispatch 0x7f63c8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:29.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63bf7fe700 10 client.22890.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63bf7fe700 20 client.22890.objecter put_session s=0x7f63ac022c70 osd=7 4 2024-02-19T21:08:29.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63bf7fe700 15 client.22890.objecter _session_command_op_remove 7 16 2024-02-19T21:08:29.807 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:08:29.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 10 librados: watch_flush enter 2024-02-19T21:08:29.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 10 librados: watch_flush exit 2024-02-19T21:08:29.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 10 client.22890.objecter close_session for osd.0 2024-02-19T21:08:29.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 >> v1:172.21.15.163:6801/1559418115 conn(0x7f63ac004530 legacy=0x7f63ac002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:29.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac0051a0 osd=0 1 2024-02-19T21:08:29.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 10 client.22890.objecter close_session for osd.1 2024-02-19T21:08:29.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 >> v1:172.21.15.163:6805/3125047432 conn(0x7f63ac003b40 legacy=0x7f63ac00a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:29.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac004270 osd=1 1 2024-02-19T21:08:29.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 10 client.22890.objecter close_session for osd.2 2024-02-19T21:08:29.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 >> v1:172.21.15.163:6809/1563158487 conn(0x7f63ac002070 legacy=0x7f63ac00f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:29.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac0033a0 osd=2 1 2024-02-19T21:08:29.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 10 client.22890.objecter close_session for osd.3 2024-02-19T21:08:29.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 >> v1:172.21.15.163:6813/2112279769 conn(0x7f63ac010ea0 legacy=0x7f63ac0112e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:29.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63ce8ed700 1 -- 172.21.15.163:0/1762601906 reap_dead start 2024-02-19T21:08:29.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac00c8d0 osd=3 1 2024-02-19T21:08:29.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 10 client.22890.objecter close_session for osd.4 2024-02-19T21:08:29.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 >> v1:172.21.15.193:6800/3129716965 conn(0x7f63ac0166c0 legacy=0x7f63ac018ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:29.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63ce8ed700 1 -- 172.21.15.163:0/1762601906 reap_dead start 2024-02-19T21:08:29.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac012230 osd=4 1 2024-02-19T21:08:29.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 10 client.22890.objecter close_session for osd.5 2024-02-19T21:08:29.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 >> v1:172.21.15.193:6804/805161991 conn(0x7f63ac01c0d0 legacy=0x7f63ac01a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:29.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac01ba50 osd=5 1 2024-02-19T21:08:29.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 10 client.22890.objecter close_session for osd.6 2024-02-19T21:08:29.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 >> v1:172.21.15.193:6808/787604615 conn(0x7f63ac01f160 legacy=0x7f63ac021550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:29.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac01c680 osd=6 1 2024-02-19T21:08:29.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 10 client.22890.objecter close_session for osd.7 2024-02-19T21:08:29.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 >> v1:172.21.15.193:6812/157927432 conn(0x7f63ac0232f0 legacy=0x7f63ac0256e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:29.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 20 client.22890.objecter put_session s=0x7f63ac022c70 osd=7 1 2024-02-19T21:08:29.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 20 client.22890.objecter shutdown clearing up homeless session... 2024-02-19T21:08:29.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 10 client.22890.objecter successfully canceled tick 2024-02-19T21:08:29.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 >> v1:172.21.15.163:6800/1552514681 conn(0x7f63b4063dc0 legacy=0x7f63b4066280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:29.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 >> v1:172.21.15.163:6789/0 conn(0x7f63c80fc870 legacy=0x7f63c8060b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:29.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.811+0000 7f63ce8ed700 1 -- 172.21.15.163:0/1762601906 reap_dead start 2024-02-19T21:08:29.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.812+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 shutdown_connections 2024-02-19T21:08:29.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.812+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 >> 172.21.15.163:0/1762601906 conn(0x7f63c80f24b0 msgr2=0x7f63c8106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:29.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.812+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 shutdown_connections 2024-02-19T21:08:29.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.812+0000 7f63d0350700 1 -- 172.21.15.163:0/1762601906 wait complete. 2024-02-19T21:08:29.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:29.812+0000 7f63d0350700 1 librados: shutdown 2024-02-19T21:08:29.812 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.812 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.812 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.812 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.812 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.812 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.812 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.812 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:08:29.854 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:08:29.884 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:30.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:29.183267+0000 mgr.y (mgr.14550) 313 : cluster [DBG] pgmap v464: 236 pgs: 8 unknown, 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:08:30.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:29.242627+0000 mon.a (mon.0) 2195 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2024-02-19T21:08:30.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:29.662152+0000 mon.a (mon.0) 2196 : audit [DBG] from='client.? v1:172.21.15.163:0/1762601906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:08:30.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:29.183267+0000 mgr.y (mgr.14550) 313 : cluster [DBG] pgmap v464: 236 pgs: 8 unknown, 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:08:30.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:29.242627+0000 mon.a (mon.0) 2195 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2024-02-19T21:08:30.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:29.662152+0000 mon.a (mon.0) 2196 : audit [DBG] from='client.? v1:172.21.15.163:0/1762601906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:08:30.568 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:08:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:08:30.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:29.183267+0000 mgr.y (mgr.14550) 313 : cluster [DBG] pgmap v464: 236 pgs: 8 unknown, 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:08:30.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:29.242627+0000 mon.a (mon.0) 2195 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2024-02-19T21:08:30.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:29.662152+0000 mon.a (mon.0) 2196 : audit [DBG] from='client.? v1:172.21.15.163:0/1762601906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:08:31.120 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:08:31.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:30.244445+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2024-02-19T21:08:31.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:30.249221+0000 mon.a (mon.0) 2198 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi163-83854-57"}]: dispatch 2024-02-19T21:08:31.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:30.295328+0000 mgr.y (mgr.14550) 314 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:31.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:30.244445+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2024-02-19T21:08:31.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:30.249221+0000 mon.a (mon.0) 2198 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi163-83854-57"}]: dispatch 2024-02-19T21:08:31.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:30.295328+0000 mgr.y (mgr.14550) 314 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:31.512 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:08:31.516 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:31.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:08:31.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:30.244445+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2024-02-19T21:08:31.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:30.249221+0000 mon.a (mon.0) 2198 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi163-83854-57"}]: dispatch 2024-02-19T21:08:31.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:30.295328+0000 mgr.y (mgr.14550) 314 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:31.618 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:31.644 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:31.786 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:31.789 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:31.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:31.890 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:08:31.919 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:31.938 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:08:31.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:31.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:08:32.041 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:32.048 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:32.063 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:08:32.063 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:08:32.063 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-02-19T21:08:32.063 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:08:32.074 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:32.095 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:32.097 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:32.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:32.182 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:08:32.183 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:08:32.183 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 4, 5, 6, 7, 3] 2024-02-19T21:08:32.183 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd out 4 2024-02-19T21:08:32.198 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:32.256 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:32.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:31.183810+0000 mgr.y (mgr.14550) 315 : cluster [DBG] pgmap v467: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-19T21:08:32.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:31.247439+0000 mon.a (mon.0) 2199 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi163-83854-57"}]': finished 2024-02-19T21:08:32.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:31.247497+0000 mon.a (mon.0) 2200 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2024-02-19T21:08:32.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:31.248773+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi163-83854-57"}]: dispatch 2024-02-19T21:08:32.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:31.183810+0000 mgr.y (mgr.14550) 315 : cluster [DBG] pgmap v467: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-19T21:08:32.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:31.247439+0000 mon.a (mon.0) 2199 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi163-83854-57"}]': finished 2024-02-19T21:08:32.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:31.247497+0000 mon.a (mon.0) 2200 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2024-02-19T21:08:32.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:31.248773+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi163-83854-57"}]: dispatch 2024-02-19T21:08:32.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:31.183810+0000 mgr.y (mgr.14550) 315 : cluster [DBG] pgmap v467: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-19T21:08:32.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:31.247439+0000 mon.a (mon.0) 2199 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi163-83854-57"}]': finished 2024-02-19T21:08:32.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:31.247497+0000 mon.a (mon.0) 2200 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2024-02-19T21:08:32.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:31.248773+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi163-83854-57"}]: dispatch 2024-02-19T21:08:32.913 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:32.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:32.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:33.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:33.051 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:33.072 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:33.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:33.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:33.176 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:33.205 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:33.225 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:33.228 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:33.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:33.254 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:32.247939+0000 mon.a (mon.0) 2202 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi163-83854-57"}]': finished 2024-02-19T21:08:33.254 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:32.248031+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2024-02-19T21:08:33.254 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:32.273914+0000 mon.b (mon.1) 496 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:33.254 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:32.275017+0000 mon.a (mon.0) 2204 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:33.254 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:32.276271+0000 mon.b (mon.1) 497 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:33.254 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:32.276600+0000 mon.a (mon.0) 2205 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:33.254 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:32.279044+0000 mon.b (mon.1) 498 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:33.254 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:32.279539+0000 mon.a (mon.0) 2206 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:33.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:32.247939+0000 mon.a (mon.0) 2202 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi163-83854-57"}]': finished 2024-02-19T21:08:33.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:32.248031+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2024-02-19T21:08:33.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:32.273914+0000 mon.b (mon.1) 496 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:33.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:32.275017+0000 mon.a (mon.0) 2204 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:33.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:32.276271+0000 mon.b (mon.1) 497 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:33.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:32.276600+0000 mon.a (mon.0) 2205 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:33.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:32.279044+0000 mon.b (mon.1) 498 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:33.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:32.279539+0000 mon.a (mon.0) 2206 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:33.330 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:33.357 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:33.379 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:33.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:33.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:33.482 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-19T21:08:33.483 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:08:33.509 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:08:33] "GET /metrics HTTP/1.1" 200 188756 "" "Prometheus/2.33.4" 2024-02-19T21:08:33.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:32.247939+0000 mon.a (mon.0) 2202 : audit [INF] from='client.? v1:172.21.15.163:0/4088364135' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi163-83854-57"}]': finished 2024-02-19T21:08:33.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:32.248031+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2024-02-19T21:08:33.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:32.273914+0000 mon.b (mon.1) 496 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:33.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:32.275017+0000 mon.a (mon.0) 2204 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:33.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:32.276271+0000 mon.b (mon.1) 497 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:33.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:32.276600+0000 mon.a (mon.0) 2205 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:33.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:32.279044+0000 mon.b (mon.1) 498 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:33.510 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:32.279539+0000 mon.a (mon.0) 2206 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:33.511 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:33.531 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:08:33.532 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:08:33.534 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:33.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:08:33.636 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:08:33.664 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:33.870 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:08:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:33.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:08:33.974 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:34.004 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:34.076 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:34.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:34.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:34.186 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:34.220 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:34.242 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:34.245 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:34.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:34.346 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:08:34.347 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:34.347 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:34.347 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:34.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.351+0000 7faaed7e8700 1 -- 172.21.15.163:0/1795824820 >> v1:172.21.15.163:6790/0 conn(0x7faae80fc870 legacy=0x7faae80fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:34.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.351+0000 7faaed7e8700 1 -- 172.21.15.163:0/1795824820 shutdown_connections 2024-02-19T21:08:34.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.351+0000 7faaed7e8700 1 -- 172.21.15.163:0/1795824820 >> 172.21.15.163:0/1795824820 conn(0x7faae80f2470 msgr2=0x7faae80f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:34.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.351+0000 7faaed7e8700 1 -- 172.21.15.163:0/1795824820 shutdown_connections 2024-02-19T21:08:34.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.351+0000 7faaed7e8700 1 -- 172.21.15.163:0/1795824820 wait complete. 2024-02-19T21:08:34.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.352+0000 7faaed7e8700 1 librados: starting msgr at 2024-02-19T21:08:34.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.352+0000 7faaed7e8700 1 librados: starting objecter 2024-02-19T21:08:34.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.352+0000 7faaed7e8700 1 Processor -- start 2024-02-19T21:08:34.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.353+0000 7faaed7e8700 1 -- start start 2024-02-19T21:08:34.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.353+0000 7faaed7e8700 1 librados: setting wanted keys 2024-02-19T21:08:34.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.353+0000 7faaed7e8700 1 librados: calling monclient init 2024-02-19T21:08:34.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.353+0000 7faaed7e8700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faae8115290 con 0x7faae80fc870 2024-02-19T21:08:34.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.353+0000 7faaed7e8700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faae8116470 con 0x7faae81031b0 2024-02-19T21:08:34.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.353+0000 7faaed7e8700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faae8117670 con 0x7faae80ff670 2024-02-19T21:08:34.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.353+0000 7faae6ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7faae80fc870 0x7faae80f85c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:58500/0 (socket says 172.21.15.163:58500) 2024-02-19T21:08:34.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.353+0000 7faae6ffd700 1 -- 172.21.15.163:0/1651124442 learned_addr learned my addr 172.21.15.163:0/1651124442 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:34.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.353+0000 7faacffff700 10 client.?.objecter ms_handle_connect 0x7faae80fc870 2024-02-19T21:08:34.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.353+0000 7faacffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:34.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.354+0000 7faacffff700 10 client.?.objecter ms_handle_connect 0x7faae80ff670 2024-02-19T21:08:34.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.354+0000 7faacffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:34.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.354+0000 7faacffff700 1 -- 172.21.15.163:0/1651124442 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889136919 0 0) 0x7faae8117670 con 0x7faae80ff670 2024-02-19T21:08:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.354+0000 7faacffff700 1 -- 172.21.15.163:0/1651124442 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faad0003030 con 0x7faae80ff670 2024-02-19T21:08:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.354+0000 7faacffff700 1 -- 172.21.15.163:0/1651124442 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2818228641 0 0) 0x7faad0003030 con 0x7faae80ff670 2024-02-19T21:08:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.354+0000 7faacffff700 1 -- 172.21.15.163:0/1651124442 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faae8117670 con 0x7faae80ff670 2024-02-19T21:08:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.354+0000 7faacffff700 1 -- 172.21.15.163:0/1651124442 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7faadc0017b0 con 0x7faae80ff670 2024-02-19T21:08:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.355+0000 7faacffff700 1 -- 172.21.15.163:0/1651124442 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 538472323 0 0) 0x7faae8117670 con 0x7faae80ff670 2024-02-19T21:08:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.355+0000 7faacffff700 1 -- 172.21.15.163:0/1651124442 >> v1:172.21.15.193:6789/0 conn(0x7faae81031b0 legacy=0x7faae8113950 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T21:08:34.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.355+0000 7faacffff700 1 -- 172.21.15.163:0/1651124442 >> v1:172.21.15.163:6789/0 conn(0x7faae80fc870 legacy=0x7faae80f85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:34.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.355+0000 7faacffff700 1 -- 172.21.15.163:0/1651124442 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae8118870 con 0x7faae80ff670 2024-02-19T21:08:34.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.355+0000 7faacffff700 1 -- 172.21.15.163:0/1651124442 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7faadc001e70 con 0x7faae80ff670 2024-02-19T21:08:34.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.355+0000 7faacffff700 1 -- 172.21.15.163:0/1651124442 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7faadc003250 con 0x7faae80ff670 2024-02-19T21:08:34.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.356+0000 7faaed7e8700 1 -- 172.21.15.163:0/1651124442 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae8118b20 con 0x7faae80ff670 2024-02-19T21:08:34.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.356+0000 7faaed7e8700 10 client.49607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:34.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.356+0000 7faaed7e8700 1 -- 172.21.15.163:0/1651124442 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faae8118f90 con 0x7faae80ff670 2024-02-19T21:08:34.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.357+0000 7faaed7e8700 1 librados: init done 2024-02-19T21:08:34.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.358+0000 7faaed7e8700 1 -- 172.21.15.163:0/1651124442 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae805f700 con 0x7faae80ff670 2024-02-19T21:08:34.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.358+0000 7faacffff700 1 -- 172.21.15.163:0/1651124442 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7faadc001990 con 0x7faae80ff670 2024-02-19T21:08:34.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.359+0000 7faacffff700 1 -- 172.21.15.163:0/1651124442 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(334..334 src has 43..334) v4 ==== 8347+0+0 (unknown 2340405719 0 0) 0x7faadc076650 con 0x7faae80ff670 2024-02-19T21:08:34.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.359+0000 7faacffff700 10 client.49607.objecter ms_dispatch 0x7faae8001550 osd_map(334..334 src has 43..334) v4 2024-02-19T21:08:34.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.359+0000 7faacffff700 3 client.49607.objecter handle_osd_map got epochs [334,334] > 0 2024-02-19T21:08:34.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.359+0000 7faacffff700 3 client.49607.objecter handle_osd_map decoding full epoch 334 2024-02-19T21:08:34.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.359+0000 7faacffff700 20 client.49607.objecter dump_active .. 0 homeless 2024-02-19T21:08:34.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.360+0000 7faacffff700 10 client.49607.objecter ms_handle_connect 0x7faad0063af0 2024-02-19T21:08:34.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.365+0000 7faacffff700 1 -- 172.21.15.163:0/1651124442 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7faadc049940 con 0x7faae80ff670 2024-02-19T21:08:34.409 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:34.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:34.562+0000 7faaed7e8700 1 -- 172.21.15.163:0/1651124442 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7faae8108eb0 con 0x7faae80ff670 2024-02-19T21:08:34.699 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:33.184280+0000 mgr.y (mgr.14550) 316 : cluster [DBG] pgmap v470: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-19T21:08:34.699 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:33.259384+0000 mon.a (mon.0) 2207 : audit [INF] from='client.48421 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:34.699 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:33.259439+0000 mon.a (mon.0) 2208 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2024-02-19T21:08:34.699 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:33.262818+0000 mon.b (mon.1) 499 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi163-83854-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:34.699 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:33.263117+0000 mon.a (mon.0) 2209 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi163-83854-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:34.700 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:33.184280+0000 mgr.y (mgr.14550) 316 : cluster [DBG] pgmap v470: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-19T21:08:34.700 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:33.259384+0000 mon.a (mon.0) 2207 : audit [INF] from='client.48421 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:34.700 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:33.259439+0000 mon.a (mon.0) 2208 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2024-02-19T21:08:34.700 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:33.262818+0000 mon.b (mon.1) 499 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi163-83854-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:34.700 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:33.263117+0000 mon.a (mon.0) 2209 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi163-83854-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:34.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:33.184280+0000 mgr.y (mgr.14550) 316 : cluster [DBG] pgmap v470: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-19T21:08:34.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:33.259384+0000 mon.a (mon.0) 2207 : audit [INF] from='client.48421 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:34.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:33.259439+0000 mon.a (mon.0) 2208 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2024-02-19T21:08:34.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:33.262818+0000 mon.b (mon.1) 499 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi163-83854-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:34.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:33.263117+0000 mon.a (mon.0) 2209 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi163-83854-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:35.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:35.232+0000 7faacffff700 1 -- 172.21.15.163:0/1651124442 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v335) v1 ==== 91+0+0 (unknown 3257170961 0 0) 0x7faadc047d50 con 0x7faae80ff670 2024-02-19T21:08:35.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:35.235+0000 7faaed7e8700 10 librados: watch_flush enter 2024-02-19T21:08:35.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:35.235+0000 7faaed7e8700 10 librados: watch_flush exit 2024-02-19T21:08:35.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:35.235+0000 7faaed7e8700 20 client.49607.objecter shutdown clearing up homeless session... 2024-02-19T21:08:35.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:35.235+0000 7faaed7e8700 10 client.49607.objecter successfully canceled tick 2024-02-19T21:08:35.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:35.235+0000 7faaed7e8700 1 -- 172.21.15.163:0/1651124442 >> v1:172.21.15.163:6800/1552514681 conn(0x7faad0063af0 legacy=0x7faad0065fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:35.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:35.235+0000 7faaed7e8700 1 -- 172.21.15.163:0/1651124442 >> v1:172.21.15.163:6790/0 conn(0x7faae80ff670 legacy=0x7faae80f8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:35.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:35.236+0000 7faaed7e8700 1 -- 172.21.15.163:0/1651124442 shutdown_connections 2024-02-19T21:08:35.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:35.236+0000 7faaed7e8700 1 -- 172.21.15.163:0/1651124442 >> 172.21.15.163:0/1651124442 conn(0x7faae80f2470 msgr2=0x7faae80601c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:35.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:35.236+0000 7faaed7e8700 1 -- 172.21.15.163:0/1651124442 shutdown_connections 2024-02-19T21:08:35.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:35.236+0000 7faaed7e8700 1 -- 172.21.15.163:0/1651124442 wait complete. 2024-02-19T21:08:35.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:35.236+0000 7faaed7e8700 1 librados: shutdown 2024-02-19T21:08:35.235 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:08:35.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:34.476105+0000 mon.a (mon.0) 2210 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2024-02-19T21:08:35.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:34.491260+0000 mon.b (mon.1) 500 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:35.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:34.491673+0000 mon.a (mon.0) 2211 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:35.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:34.492282+0000 mon.b (mon.1) 501 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43"}]: dispatch 2024-02-19T21:08:35.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:34.492546+0000 mon.a (mon.0) 2212 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43"}]: dispatch 2024-02-19T21:08:35.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:34.563759+0000 mon.c (mon.2) 347 : audit [INF] from='client.? v1:172.21.15.163:0/1651124442' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:08:35.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:34.564065+0000 mon.a (mon.0) 2213 : audit [INF] from='client.49607 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:08:35.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:35.224741+0000 mon.a (mon.0) 2214 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:08:35.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:35.228258+0000 mon.a (mon.0) 2215 : audit [INF] from='client.48421 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi163-83854-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]': finished 2024-02-19T21:08:35.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:35.228327+0000 mon.a (mon.0) 2216 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43"}]': finished 2024-02-19T21:08:35.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:35.228377+0000 mon.a (mon.0) 2217 : audit [INF] from='client.49607 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:08:35.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:35.228418+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2024-02-19T21:08:35.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:34.476105+0000 mon.a (mon.0) 2210 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2024-02-19T21:08:35.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:34.491260+0000 mon.b (mon.1) 500 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:35.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:34.491673+0000 mon.a (mon.0) 2211 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:35.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:34.492282+0000 mon.b (mon.1) 501 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43"}]: dispatch 2024-02-19T21:08:35.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:34.492546+0000 mon.a (mon.0) 2212 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43"}]: dispatch 2024-02-19T21:08:35.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:34.563759+0000 mon.c (mon.2) 347 : audit [INF] from='client.? v1:172.21.15.163:0/1651124442' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:08:35.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:34.564065+0000 mon.a (mon.0) 2213 : audit [INF] from='client.49607 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:08:35.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:35.224741+0000 mon.a (mon.0) 2214 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:08:35.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:35.228258+0000 mon.a (mon.0) 2215 : audit [INF] from='client.48421 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi163-83854-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]': finished 2024-02-19T21:08:35.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:35.228327+0000 mon.a (mon.0) 2216 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43"}]': finished 2024-02-19T21:08:35.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:35.228377+0000 mon.a (mon.0) 2217 : audit [INF] from='client.49607 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:08:35.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:35.228418+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2024-02-19T21:08:35.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:34.476105+0000 mon.a (mon.0) 2210 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2024-02-19T21:08:35.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:34.491260+0000 mon.b (mon.1) 500 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:35.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:34.491673+0000 mon.a (mon.0) 2211 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:35.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:34.492282+0000 mon.b (mon.1) 501 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43"}]: dispatch 2024-02-19T21:08:35.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:34.492546+0000 mon.a (mon.0) 2212 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43"}]: dispatch 2024-02-19T21:08:35.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:34.563759+0000 mon.c (mon.2) 347 : audit [INF] from='client.? v1:172.21.15.163:0/1651124442' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:08:35.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:34.564065+0000 mon.a (mon.0) 2213 : audit [INF] from='client.49607 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:08:35.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:35.224741+0000 mon.a (mon.0) 2214 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:08:35.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:35.228258+0000 mon.a (mon.0) 2215 : audit [INF] from='client.48421 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi163-83854-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]': finished 2024-02-19T21:08:35.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:35.228327+0000 mon.a (mon.0) 2216 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-43"}]': finished 2024-02-19T21:08:35.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:35.228377+0000 mon.a (mon.0) 2217 : audit [INF] from='client.49607 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:08:35.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:35.228418+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2024-02-19T21:08:36.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:35.185074+0000 mgr.y (mgr.14550) 317 : cluster [DBG] pgmap v473: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-19T21:08:36.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:36.233226+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2024-02-19T21:08:36.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:35.185074+0000 mgr.y (mgr.14550) 317 : cluster [DBG] pgmap v473: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-19T21:08:36.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:36.233226+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2024-02-19T21:08:36.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:35.185074+0000 mgr.y (mgr.14550) 317 : cluster [DBG] pgmap v473: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-02-19T21:08:36.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:36.233226+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2024-02-19T21:08:37.390 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:08:37.393 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:37.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:08:37.494 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:08:37.524 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:37.623 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:08:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:37.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:08:37.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:35.541578+0000 osd.4 (osd.4) 139 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:08:37.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:35.542747+0000 osd.4 (osd.4) 140 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:08:37.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:36.250801+0000 osd.6 (osd.6) 133 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:08:37.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:36.251950+0000 osd.6 (osd.6) 134 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:08:37.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:36.591364+0000 osd.4 (osd.4) 141 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:08:37.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:36.593165+0000 osd.4 (osd.4) 142 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:08:37.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:36.846476+0000 osd.3 (osd.3) 143 : cluster [DBG] 16.18 scrub starts 2024-02-19T21:08:37.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:36.847544+0000 osd.3 (osd.3) 144 : cluster [DBG] 16.18 scrub ok 2024-02-19T21:08:37.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:36.900763+0000 osd.7 (osd.7) 137 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:08:37.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:36.903671+0000 osd.7 (osd.7) 138 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:08:37.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:37.232578+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2024-02-19T21:08:37.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:37.245230+0000 mon.b (mon.1) 502 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:37.654 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:37.245636+0000 mon.b (mon.1) 503 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:37.655 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:37.246154+0000 mon.a (mon.0) 2221 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:37.655 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:37.246401+0000 mon.a (mon.0) 2222 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:37.655 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:35.541578+0000 osd.4 (osd.4) 139 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:08:37.655 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:35.542747+0000 osd.4 (osd.4) 140 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:08:37.655 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:36.250801+0000 osd.6 (osd.6) 133 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:08:37.655 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:36.251950+0000 osd.6 (osd.6) 134 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:08:37.655 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:36.591364+0000 osd.4 (osd.4) 141 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:08:37.655 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:36.593165+0000 osd.4 (osd.4) 142 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:08:37.655 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:36.846476+0000 osd.3 (osd.3) 143 : cluster [DBG] 16.18 scrub starts 2024-02-19T21:08:37.655 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:36.847544+0000 osd.3 (osd.3) 144 : cluster [DBG] 16.18 scrub ok 2024-02-19T21:08:37.655 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:36.900763+0000 osd.7 (osd.7) 137 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:08:37.655 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:36.903671+0000 osd.7 (osd.7) 138 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:08:37.656 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:37.232578+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2024-02-19T21:08:37.656 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:37.245230+0000 mon.b (mon.1) 502 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:37.656 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:37.245636+0000 mon.b (mon.1) 503 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:37.656 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:37.246154+0000 mon.a (mon.0) 2221 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:37.656 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:37.246401+0000 mon.a (mon.0) 2222 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:37.727 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:37.757 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:35.541578+0000 osd.4 (osd.4) 139 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:08:37.757 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:35.542747+0000 osd.4 (osd.4) 140 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:08:37.757 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:36.250801+0000 osd.6 (osd.6) 133 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:08:37.757 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:36.251950+0000 osd.6 (osd.6) 134 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:08:37.758 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:36.591364+0000 osd.4 (osd.4) 141 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:08:37.758 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:36.593165+0000 osd.4 (osd.4) 142 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:08:37.758 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:36.846476+0000 osd.3 (osd.3) 143 : cluster [DBG] 16.18 scrub starts 2024-02-19T21:08:37.758 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:36.847544+0000 osd.3 (osd.3) 144 : cluster [DBG] 16.18 scrub ok 2024-02-19T21:08:37.758 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:36.900763+0000 osd.7 (osd.7) 137 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:08:37.758 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:36.903671+0000 osd.7 (osd.7) 138 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:08:37.758 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:37.232578+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2024-02-19T21:08:37.758 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:37.245230+0000 mon.b (mon.1) 502 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:37.758 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:37.245636+0000 mon.b (mon.1) 503 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:37.758 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:37.246154+0000 mon.a (mon.0) 2221 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:37.759 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:37.246401+0000 mon.a (mon.0) 2222 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:37.760 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:37.782 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:37.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:37.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:37.885 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:08:37.915 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:38.068 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:08:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:08:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:08:38.708 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:08:38.708 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:08:38.708 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-02-19T21:08:38.708 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-02-19T21:08:38.708 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:08:38.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:36.715480+0000 osd.1 (osd.1) 153 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:08:38.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:36.716460+0000 osd.1 (osd.1) 154 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:08:38.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:37.185474+0000 mgr.y (mgr.14550) 318 : cluster [DBG] pgmap v476: 204 pgs: 8 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:08:38.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:37.519617+0000 osd.5 (osd.5) 134 : cluster [DBG] 61.1c scrub starts 2024-02-19T21:08:38.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:37.523293+0000 osd.5 (osd.5) 135 : cluster [DBG] 61.1c scrub ok 2024-02-19T21:08:38.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:37.865772+0000 osd.3 (osd.3) 145 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:08:38.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:37.867399+0000 osd.3 (osd.3) 146 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:08:38.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:37.932281+0000 osd.7 (osd.7) 139 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:08:38.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:37.935171+0000 osd.7 (osd.7) 140 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:08:38.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:38.238120+0000 mon.a (mon.0) 2223 : audit [INF] from='client.48421 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]': finished 2024-02-19T21:08:38.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:38.238212+0000 mon.a (mon.0) 2224 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:38.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:38.238281+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2024-02-19T21:08:38.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:38.245768+0000 mon.b (mon.1) 504 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:38.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:38.246501+0000 mon.a (mon.0) 2226 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:38.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:38.283565+0000 mon.b (mon.1) 505 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:38.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:38.283927+0000 mon.a (mon.0) 2227 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:38.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:36.715480+0000 osd.1 (osd.1) 153 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:08:38.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:36.716460+0000 osd.1 (osd.1) 154 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:08:38.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:37.185474+0000 mgr.y (mgr.14550) 318 : cluster [DBG] pgmap v476: 204 pgs: 8 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:08:38.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:37.519617+0000 osd.5 (osd.5) 134 : cluster [DBG] 61.1c scrub starts 2024-02-19T21:08:38.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:37.523293+0000 osd.5 (osd.5) 135 : cluster [DBG] 61.1c scrub ok 2024-02-19T21:08:38.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:37.865772+0000 osd.3 (osd.3) 145 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:08:38.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:37.867399+0000 osd.3 (osd.3) 146 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:08:38.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:37.932281+0000 osd.7 (osd.7) 139 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:08:38.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:37.935171+0000 osd.7 (osd.7) 140 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:08:38.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:38.238120+0000 mon.a (mon.0) 2223 : audit [INF] from='client.48421 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]': finished 2024-02-19T21:08:38.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:38.238212+0000 mon.a (mon.0) 2224 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:38.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:38.238281+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2024-02-19T21:08:38.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:38.245768+0000 mon.b (mon.1) 504 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:38.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:38.246501+0000 mon.a (mon.0) 2226 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:38.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:38.283565+0000 mon.b (mon.1) 505 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:38.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:38.283927+0000 mon.a (mon.0) 2227 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:38.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:36.715480+0000 osd.1 (osd.1) 153 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:08:38.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:36.716460+0000 osd.1 (osd.1) 154 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:08:38.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:37.185474+0000 mgr.y (mgr.14550) 318 : cluster [DBG] pgmap v476: 204 pgs: 8 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:08:38.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:37.519617+0000 osd.5 (osd.5) 134 : cluster [DBG] 61.1c scrub starts 2024-02-19T21:08:38.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:37.523293+0000 osd.5 (osd.5) 135 : cluster [DBG] 61.1c scrub ok 2024-02-19T21:08:38.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:37.865772+0000 osd.3 (osd.3) 145 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:08:38.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:37.867399+0000 osd.3 (osd.3) 146 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:08:38.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:37.932281+0000 osd.7 (osd.7) 139 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:08:38.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:37.935171+0000 osd.7 (osd.7) 140 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:08:38.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:38.238120+0000 mon.a (mon.0) 2223 : audit [INF] from='client.48421 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi163-83854-58"}]': finished 2024-02-19T21:08:38.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:38.238212+0000 mon.a (mon.0) 2224 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:38.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:38.238281+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2024-02-19T21:08:38.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:38.245768+0000 mon.b (mon.1) 504 : audit [INF] from='client.? v1:172.21.15.163:0/104540366' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:38.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:38.246501+0000 mon.a (mon.0) 2226 : audit [INF] from='client.48421 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi163-83854-58"}]: dispatch 2024-02-19T21:08:38.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:38.283565+0000 mon.b (mon.1) 505 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:38.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:38.283927+0000 mon.a (mon.0) 2227 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:39.168 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:08:39.174 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:39.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:08:39.276 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:39.308 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:39.328 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:39.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:39.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:39.408 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:08:39.433 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:39.464 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:39.484 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:39.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:39.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:39.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:38.042944+0000 osd.2 (osd.2) 120 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:39.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:38.043132+0000 osd.2 (osd.2) 121 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:39.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:39.238354+0000 mon.a (mon.0) 2228 : audit [INF] from='client.48421 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi163-83854-58"}]': finished 2024-02-19T21:08:39.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:39.238431+0000 mon.a (mon.0) 2229 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:08:39.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:39.238497+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2024-02-19T21:08:39.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:39.240625+0000 mon.b (mon.1) 506 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:39.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:39.240908+0000 mon.a (mon.0) 2231 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:39.588 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:39.615 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:39.636 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:39.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:39.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:39.740 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:39.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:38.042944+0000 osd.2 (osd.2) 120 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:39.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:38.043132+0000 osd.2 (osd.2) 121 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:39.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:39.238354+0000 mon.a (mon.0) 2228 : audit [INF] from='client.48421 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi163-83854-58"}]': finished 2024-02-19T21:08:39.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:39.238431+0000 mon.a (mon.0) 2229 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:08:39.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:39.238497+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2024-02-19T21:08:39.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:39.240625+0000 mon.b (mon.1) 506 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:39.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:39.240908+0000 mon.a (mon.0) 2231 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:38.042944+0000 osd.2 (osd.2) 120 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:38.043132+0000 osd.2 (osd.2) 121 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:39.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:39.238354+0000 mon.a (mon.0) 2228 : audit [INF] from='client.48421 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi163-83854-58"}]': finished 2024-02-19T21:08:39.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:39.238431+0000 mon.a (mon.0) 2229 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:08:39.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:39.238497+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2024-02-19T21:08:39.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:39.240625+0000 mon.b (mon.1) 506 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:39.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:39.240908+0000 mon.a (mon.0) 2231 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:39.766 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:39.785 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:39.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:39.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:39.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.715+0000 7fd5c567e700 1 -- 172.21.15.163:0/301250374 >> v1:172.21.15.163:6789/0 conn(0x7fd5c00f7260 legacy=0x7fd5c0104df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:39.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.715+0000 7fd5c567e700 1 -- 172.21.15.163:0/301250374 shutdown_connections 2024-02-19T21:08:39.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.715+0000 7fd5c567e700 1 -- 172.21.15.163:0/301250374 >> 172.21.15.163:0/301250374 conn(0x7fd5c00f2470 msgr2=0x7fd5c00f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:39.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.715+0000 7fd5c567e700 1 -- 172.21.15.163:0/301250374 shutdown_connections 2024-02-19T21:08:39.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.715+0000 7fd5c567e700 1 -- 172.21.15.163:0/301250374 wait complete. 2024-02-19T21:08:39.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.716+0000 7fd5c567e700 1 librados: starting msgr at 2024-02-19T21:08:39.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.716+0000 7fd5c567e700 1 librados: starting objecter 2024-02-19T21:08:39.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.716+0000 7fd5c567e700 1 Processor -- start 2024-02-19T21:08:39.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.716+0000 7fd5c567e700 1 -- start start 2024-02-19T21:08:39.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.716+0000 7fd5c567e700 1 librados: setting wanted keys 2024-02-19T21:08:39.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.716+0000 7fd5c567e700 1 librados: calling monclient init 2024-02-19T21:08:39.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.716+0000 7fd5c567e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd5c01192f0 con 0x7fd5c00f6770 2024-02-19T21:08:39.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.716+0000 7fd5c567e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd5c011a4d0 con 0x7fd5c00f7260 2024-02-19T21:08:39.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.716+0000 7fd5c567e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd5c011b6d0 con 0x7fd5c0106510 2024-02-19T21:08:39.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.716+0000 7fd5beffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fd5c00f6770 0x7fd5c01026a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:58572/0 (socket says 172.21.15.163:58572) 2024-02-19T21:08:39.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.716+0000 7fd5beffd700 1 -- 172.21.15.163:0/706886661 learned_addr learned my addr 172.21.15.163:0/706886661 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:39.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.716+0000 7fd5a7fff700 10 client.?.objecter ms_handle_connect 0x7fd5c00f6770 2024-02-19T21:08:39.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.716+0000 7fd5a7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:39.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.716+0000 7fd5a7fff700 10 client.?.objecter ms_handle_connect 0x7fd5c0106510 2024-02-19T21:08:39.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.716+0000 7fd5a7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:39.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 10 client.?.objecter ms_handle_connect 0x7fd5c00f7260 2024-02-19T21:08:39.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:39.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404245 0 0) 0x7fd5c01192f0 con 0x7fd5c00f6770 2024-02-19T21:08:39.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a80033f0 con 0x7fd5c00f6770 2024-02-19T21:08:39.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149724548 0 0) 0x7fd5c011b6d0 con 0x7fd5c0106510 2024-02-19T21:08:39.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5c01192f0 con 0x7fd5c0106510 2024-02-19T21:08:39.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226926988 0 0) 0x7fd5c011a4d0 con 0x7fd5c00f7260 2024-02-19T21:08:39.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5c011b6d0 con 0x7fd5c00f7260 2024-02-19T21:08:39.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 803088322 0 0) 0x7fd5a80033f0 con 0x7fd5c00f6770 2024-02-19T21:08:39.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5c011a4d0 con 0x7fd5c00f6770 2024-02-19T21:08:39.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd5b00019d0 con 0x7fd5c00f6770 2024-02-19T21:08:39.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2015985182 0 0) 0x7fd5c01192f0 con 0x7fd5c0106510 2024-02-19T21:08:39.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5a80033f0 con 0x7fd5c0106510 2024-02-19T21:08:39.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd5b8001d50 con 0x7fd5c0106510 2024-02-19T21:08:39.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1910081977 0 0) 0x7fd5c011a4d0 con 0x7fd5c00f6770 2024-02-19T21:08:39.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 >> v1:172.21.15.163:6790/0 conn(0x7fd5c0106510 legacy=0x7fd5c0117b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:39.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 >> v1:172.21.15.193:6789/0 conn(0x7fd5c00f7260 legacy=0x7fd5c0102db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:39.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c011c8d0 con 0x7fd5c00f6770 2024-02-19T21:08:39.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5c567e700 1 -- 172.21.15.163:0/706886661 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c011b8c0 con 0x7fd5c00f6770 2024-02-19T21:08:39.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5c567e700 10 client.22914.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:39.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5c567e700 1 -- 172.21.15.163:0/706886661 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c011bdb0 con 0x7fd5c00f6770 2024-02-19T21:08:39.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5c567e700 1 librados: init done 2024-02-19T21:08:39.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fd5b00023e0 con 0x7fd5c00f6770 2024-02-19T21:08:39.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.717+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd5b0005ae0 con 0x7fd5c00f6770 2024-02-19T21:08:39.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.718+0000 7fd5c567e700 1 -- 172.21.15.163:0/706886661 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5ac004f20 con 0x7fd5c00f6770 2024-02-19T21:08:39.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.719+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fd5b0001f40 con 0x7fd5c00f6770 2024-02-19T21:08:39.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.720+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(339..339 src has 43..339) v4 ==== 7910+0+0 (unknown 3857275204 0 0) 0x7fd5b0078bd0 con 0x7fd5c00f6770 2024-02-19T21:08:39.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.720+0000 7fd5a7fff700 10 client.22914.objecter ms_dispatch 0x7fd5c0001550 osd_map(339..339 src has 43..339) v4 2024-02-19T21:08:39.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.720+0000 7fd5a7fff700 3 client.22914.objecter handle_osd_map got epochs [339,339] > 0 2024-02-19T21:08:39.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.720+0000 7fd5a7fff700 3 client.22914.objecter handle_osd_map decoding full epoch 339 2024-02-19T21:08:39.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.720+0000 7fd5a7fff700 20 client.22914.objecter dump_active .. 0 homeless 2024-02-19T21:08:39.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.720+0000 7fd5a7fff700 10 client.22914.objecter ms_handle_connect 0x7fd5a8063970 2024-02-19T21:08:39.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.723+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd5b004c0c0 con 0x7fd5c00f6770 2024-02-19T21:08:39.888 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:08:39.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.912+0000 7fd5c567e700 1 -- 172.21.15.163:0/706886661 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd out", "ids": ["4"]} v 0) v1 -- 0x7fd5ac005750 con 0x7fd5c00f6770 2024-02-19T21:08:39.917 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:40.064 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:08:39.912+0000 7fb606608700 -1 mon.a@0(leader).osd e339 definitely_dead 0 2024-02-19T21:08:40.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:40.244+0000 7fd5a7fff700 1 -- 172.21.15.163:0/706886661 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd out", "ids": ["4"]}]=0 marked out osd.4. v340) v1 ==== 87+0+0 (unknown 853382713 0 0) 0x7fd5b004a4d0 con 0x7fd5c00f6770 2024-02-19T21:08:40.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:40.250+0000 7fd5c567e700 10 librados: watch_flush enter 2024-02-19T21:08:40.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:40.251+0000 7fd5c567e700 10 librados: watch_flush exit 2024-02-19T21:08:40.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:40.251+0000 7fd5c567e700 20 client.22914.objecter shutdown clearing up homeless session... 2024-02-19T21:08:40.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:40.251+0000 7fd5c567e700 10 client.22914.objecter successfully canceled tick 2024-02-19T21:08:40.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:40.256+0000 7fd5c567e700 1 -- 172.21.15.163:0/706886661 >> v1:172.21.15.163:6800/1552514681 conn(0x7fd5a8063970 legacy=0x7fd5a8065e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:40.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:40.256+0000 7fd5c567e700 1 -- 172.21.15.163:0/706886661 >> v1:172.21.15.163:6789/0 conn(0x7fd5c00f6770 legacy=0x7fd5c01026a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:40.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:40.256+0000 7fd5c567e700 1 -- 172.21.15.163:0/706886661 shutdown_connections 2024-02-19T21:08:40.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:40.256+0000 7fd5c567e700 1 -- 172.21.15.163:0/706886661 >> 172.21.15.163:0/706886661 conn(0x7fd5c00f2470 msgr2=0x7fd5c0108900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:40.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:40.260+0000 7fd5c567e700 1 -- 172.21.15.163:0/706886661 shutdown_connections 2024-02-19T21:08:40.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:40.262+0000 7fd5c567e700 1 -- 172.21.15.163:0/706886661 wait complete. 2024-02-19T21:08:40.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:40.262+0000 7fd5c567e700 1 librados: shutdown 2024-02-19T21:08:40.272 INFO:teuthology.orchestra.run.smithi163.stderr:marked out osd.4. 2024-02-19T21:08:40.273 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:08:40.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:40.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:08:40.286 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-19T21:08:40.377 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:08:40.404 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:40.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.595+0000 7fb13e926700 1 -- 172.21.15.163:0/50510446 >> v1:172.21.15.163:6789/0 conn(0x7fb1380fbfb0 legacy=0x7fb1380fc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:40.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.595+0000 7fb13e926700 1 -- 172.21.15.163:0/50510446 shutdown_connections 2024-02-19T21:08:40.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.595+0000 7fb13e926700 1 -- 172.21.15.163:0/50510446 >> 172.21.15.163:0/50510446 conn(0x7fb13806fdc0 msgr2=0x7fb1380f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:40.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.595+0000 7fb13e926700 1 -- 172.21.15.163:0/50510446 shutdown_connections 2024-02-19T21:08:40.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.595+0000 7fb13e926700 1 -- 172.21.15.163:0/50510446 wait complete. 2024-02-19T21:08:40.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.595+0000 7fb13e926700 1 librados: starting msgr at 2024-02-19T21:08:40.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.595+0000 7fb13e926700 1 librados: starting objecter 2024-02-19T21:08:40.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb13e926700 1 Processor -- start 2024-02-19T21:08:40.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb13e926700 1 -- start start 2024-02-19T21:08:40.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb13e926700 1 librados: setting wanted keys 2024-02-19T21:08:40.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb13e926700 1 librados: calling monclient init 2024-02-19T21:08:40.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb13e926700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb138119410 con 0x7fb1380fee00 2024-02-19T21:08:40.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb13e926700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb13811a610 con 0x7fb1380fbfb0 2024-02-19T21:08:40.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb13e926700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb13811b810 con 0x7fb138102910 2024-02-19T21:08:40.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb13cec3700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fb138102910 0x7fb138117ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:34070/0 (socket says 172.21.15.163:34070) 2024-02-19T21:08:40.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb1377fe700 1 --1- 172.21.15.163:0/1525448003 >> v1:172.21.15.163:6789/0 conn(0x7fb1380fee00 0x7fb138114360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:58544/0 (socket says 172.21.15.163:58544) 2024-02-19T21:08:40.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb13cec3700 1 -- 172.21.15.163:0/1525448003 learned_addr learned my addr 172.21.15.163:0/1525448003 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:40.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb1357fa700 10 client.?.objecter ms_handle_connect 0x7fb138102910 2024-02-19T21:08:40.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb1357fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:40.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb1357fa700 10 client.?.objecter ms_handle_connect 0x7fb1380fee00 2024-02-19T21:08:40.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb1357fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:40.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb1357fa700 10 client.?.objecter ms_handle_connect 0x7fb1380fbfb0 2024-02-19T21:08:40.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb1357fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667137120 0 0) 0x7fb138119410 con 0x7fb1380fee00 2024-02-19T21:08:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb120004980 con 0x7fb1380fee00 2024-02-19T21:08:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140342339 0 0) 0x7fb13811b810 con 0x7fb138102910 2024-02-19T21:08:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb138119410 con 0x7fb138102910 2024-02-19T21:08:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926596392 0 0) 0x7fb13811a610 con 0x7fb1380fbfb0 2024-02-19T21:08:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.596+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb13811b810 con 0x7fb1380fbfb0 2024-02-19T21:08:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 492148008 0 0) 0x7fb138119410 con 0x7fb138102910 2024-02-19T21:08:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb13811a610 con 0x7fb138102910 2024-02-19T21:08:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb130001cd0 con 0x7fb138102910 2024-02-19T21:08:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 715022022 0 0) 0x7fb13811b810 con 0x7fb1380fbfb0 2024-02-19T21:08:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb138119410 con 0x7fb1380fbfb0 2024-02-19T21:08:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2804933888 0 0) 0x7fb120004980 con 0x7fb1380fee00 2024-02-19T21:08:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb13811b810 con 0x7fb1380fee00 2024-02-19T21:08:40.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb128000fe0 con 0x7fb1380fbfb0 2024-02-19T21:08:40.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb12c001a80 con 0x7fb1380fee00 2024-02-19T21:08:40.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3209565001 0 0) 0x7fb13811a610 con 0x7fb138102910 2024-02-19T21:08:40.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 >> v1:172.21.15.193:6789/0 conn(0x7fb1380fbfb0 legacy=0x7fb1380fb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:40.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 >> v1:172.21.15.163:6789/0 conn(0x7fb1380fee00 legacy=0x7fb138114360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:40.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13811ca10 con 0x7fb138102910 2024-02-19T21:08:40.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb138119600 con 0x7fb138102910 2024-02-19T21:08:40.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb13e926700 10 client.49619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:40.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb138119a70 con 0x7fb138102910 2024-02-19T21:08:40.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb13e926700 1 librados: init done 2024-02-19T21:08:40.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb130002850 con 0x7fb138102910 2024-02-19T21:08:40.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.597+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb130005ee0 con 0x7fb138102910 2024-02-19T21:08:40.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.598+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb124004f00 con 0x7fb138102910 2024-02-19T21:08:40.425 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:08:40.425 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:08:40.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.598+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fb130018d70 con 0x7fb138102910 2024-02-19T21:08:40.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(339..339 src has 43..339) v4 ==== 7910+0+0 (unknown 3857275204 0 0) 0x7fb130078f00 con 0x7fb138102910 2024-02-19T21:08:40.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 osd_map(339..339 src has 43..339) v4 2024-02-19T21:08:40.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb1357fa700 3 client.49619.objecter handle_osd_map got epochs [339,339] > 0 2024-02-19T21:08:40.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb1357fa700 3 client.49619.objecter handle_osd_map decoding full epoch 339 2024-02-19T21:08:40.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb1357fa700 20 client.49619.objecter dump_active .. 0 homeless 2024-02-19T21:08:40.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v339) v1 ==== 54+0+16 (unknown 2980601213 0 2420000192) 0x7fb13007b350 con 0x7fb138102910 2024-02-19T21:08:40.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb1357fa700 10 client.49619.objecter ms_handle_connect 0x7fb120063c90 2024-02-19T21:08:40.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb13e926700 10 client.49619.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:40.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 1 2024-02-19T21:08:40.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb1240051a0 osd=0 3 2024-02-19T21:08:40.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb1240051a0 osd=0 3 2024-02-19T21:08:40.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb1240051a0 osd=0 3 2024-02-19T21:08:40.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 1 2024-02-19T21:08:40.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb1240051a0 osd=0 3 2024-02-19T21:08:40.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 0 1 2024-02-19T21:08:40.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb1240051a0 osd=0 4 2024-02-19T21:08:40.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb13e926700 10 client.49619.objecter _send_command 1 2024-02-19T21:08:40.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb124004950 con 0x7fb124004530 2024-02-19T21:08:40.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.599+0000 7fb1357fa700 10 client.49619.objecter ms_handle_connect 0x7fb124004530 2024-02-19T21:08:40.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.600+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb124004950 con 0x7fb124004530 2024-02-19T21:08:40.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.600+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:08:40.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.600+0000 7fb1357fa700 10 client.49619.objecter _finish_command 1 = system:0 2024-02-19T21:08:40.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.600+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb1240051a0 osd=0 4 2024-02-19T21:08:40.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.600+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 0 1 2024-02-19T21:08:40.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.616+0000 7fb13e926700 10 client.49619.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:40.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.616+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 2 2024-02-19T21:08:40.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.616+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb1240051a0 osd=0 3 2024-02-19T21:08:40.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.616+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb1240051a0 osd=0 3 2024-02-19T21:08:40.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.616+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb1240051a0 osd=0 3 2024-02-19T21:08:40.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.616+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 2 2024-02-19T21:08:40.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.616+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb1240051a0 osd=0 3 2024-02-19T21:08:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.616+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 0 2 2024-02-19T21:08:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.616+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb1240051a0 osd=0 4 2024-02-19T21:08:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.616+0000 7fb13e926700 10 client.49619.objecter _send_command 2 2024-02-19T21:08:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.616+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb124004d30 con 0x7fb124004530 2024-02-19T21:08:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb124004d30 con 0x7fb124004530 2024-02-19T21:08:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb1357fa700 10 client.49619.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb1240051a0 osd=0 4 2024-02-19T21:08:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 0 2 2024-02-19T21:08:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb13e926700 10 client.49619.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 3 2024-02-19T21:08:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124003fe0 osd=1 3 2024-02-19T21:08:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124003fe0 osd=1 3 2024-02-19T21:08:40.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124003fe0 osd=1 3 2024-02-19T21:08:40.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 3 2024-02-19T21:08:40.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb124003fe0 osd=1 3 2024-02-19T21:08:40.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 1 3 2024-02-19T21:08:40.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124003fe0 osd=1 4 2024-02-19T21:08:40.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb13e926700 10 client.49619.objecter _send_command 3 2024-02-19T21:08:40.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb12400c000 con 0x7fb124004150 2024-02-19T21:08:40.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.618+0000 7fb1357fa700 10 client.49619.objecter ms_handle_connect 0x7fb124004150 2024-02-19T21:08:40.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.619+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb12400c000 con 0x7fb124004150 2024-02-19T21:08:40.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.619+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:08:40.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.619+0000 7fb1357fa700 10 client.49619.objecter _finish_command 3 = system:0 2024-02-19T21:08:40.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.619+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb124003fe0 osd=1 4 2024-02-19T21:08:40.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.619+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 1 3 2024-02-19T21:08:40.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.634+0000 7fb13e926700 10 client.49619.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:40.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.634+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 4 2024-02-19T21:08:40.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.634+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124003fe0 osd=1 3 2024-02-19T21:08:40.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.634+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124003fe0 osd=1 3 2024-02-19T21:08:40.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.634+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124003fe0 osd=1 3 2024-02-19T21:08:40.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.634+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 4 2024-02-19T21:08:40.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.634+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb124003fe0 osd=1 3 2024-02-19T21:08:40.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.634+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 1 4 2024-02-19T21:08:40.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.634+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124003fe0 osd=1 4 2024-02-19T21:08:40.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.634+0000 7fb13e926700 10 client.49619.objecter _send_command 4 2024-02-19T21:08:40.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.634+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb1240018b0 con 0x7fb124004150 2024-02-19T21:08:40.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb1240018b0 con 0x7fb124004150 2024-02-19T21:08:40.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:40.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb1357fa700 10 client.49619.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb124003fe0 osd=1 4 2024-02-19T21:08:40.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 1 4 2024-02-19T21:08:40.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb13e926700 10 client.49619.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:40.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 5 2024-02-19T21:08:40.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124003630 osd=2 3 2024-02-19T21:08:40.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124003630 osd=2 3 2024-02-19T21:08:40.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124003630 osd=2 3 2024-02-19T21:08:40.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 5 2024-02-19T21:08:40.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb124003630 osd=2 3 2024-02-19T21:08:40.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 2 5 2024-02-19T21:08:40.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124003630 osd=2 4 2024-02-19T21:08:40.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb13e926700 10 client.49619.objecter _send_command 5 2024-02-19T21:08:40.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb12400de90 con 0x7fb12400d2f0 2024-02-19T21:08:40.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.636+0000 7fb1357fa700 10 client.49619.objecter ms_handle_connect 0x7fb12400d2f0 2024-02-19T21:08:40.433 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:08:40.433 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:08:40.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.637+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb12400de90 con 0x7fb12400d2f0 2024-02-19T21:08:40.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.637+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:08:40.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.637+0000 7fb1357fa700 10 client.49619.objecter _finish_command 5 = system:0 2024-02-19T21:08:40.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.637+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb124003630 osd=2 4 2024-02-19T21:08:40.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.637+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 2 5 2024-02-19T21:08:40.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.653+0000 7fb13e926700 10 client.49619.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:40.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.653+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 6 2024-02-19T21:08:40.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.653+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124003630 osd=2 3 2024-02-19T21:08:40.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.653+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124003630 osd=2 3 2024-02-19T21:08:40.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.653+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124003630 osd=2 3 2024-02-19T21:08:40.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.653+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 6 2024-02-19T21:08:40.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.653+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb124003630 osd=2 3 2024-02-19T21:08:40.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.653+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 2 6 2024-02-19T21:08:40.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.653+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124003630 osd=2 4 2024-02-19T21:08:40.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.653+0000 7fb13e926700 10 client.49619.objecter _send_command 6 2024-02-19T21:08:40.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.653+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb124003160 con 0x7fb12400d2f0 2024-02-19T21:08:40.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb124003160 con 0x7fb12400d2f0 2024-02-19T21:08:40.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:40.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb1357fa700 10 client.49619.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb124003630 osd=2 4 2024-02-19T21:08:40.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 2 6 2024-02-19T21:08:40.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb13e926700 10 client.49619.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:40.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 7 2024-02-19T21:08:40.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124011970 osd=3 3 2024-02-19T21:08:40.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124011970 osd=3 3 2024-02-19T21:08:40.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124011970 osd=3 3 2024-02-19T21:08:40.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 7 2024-02-19T21:08:40.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb124011970 osd=3 3 2024-02-19T21:08:40.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 3 7 2024-02-19T21:08:40.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124011970 osd=3 4 2024-02-19T21:08:40.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb13e926700 10 client.49619.objecter _send_command 7 2024-02-19T21:08:40.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb124015b90 con 0x7fb124011ff0 2024-02-19T21:08:40.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.655+0000 7fb1357fa700 10 client.49619.objecter ms_handle_connect 0x7fb124011ff0 2024-02-19T21:08:40.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.656+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb124015b90 con 0x7fb124011ff0 2024-02-19T21:08:40.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.656+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:08:40.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.656+0000 7fb1357fa700 10 client.49619.objecter _finish_command 7 = system:0 2024-02-19T21:08:40.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.656+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb124011970 osd=3 4 2024-02-19T21:08:40.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.656+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 3 7 2024-02-19T21:08:40.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.671+0000 7fb13e926700 10 client.49619.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:40.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.671+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 8 2024-02-19T21:08:40.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.671+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124011970 osd=3 3 2024-02-19T21:08:40.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.671+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124011970 osd=3 3 2024-02-19T21:08:40.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.671+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124011970 osd=3 3 2024-02-19T21:08:40.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.671+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 8 2024-02-19T21:08:40.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.671+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb124011970 osd=3 3 2024-02-19T21:08:40.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.671+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 3 8 2024-02-19T21:08:40.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.671+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124011970 osd=3 4 2024-02-19T21:08:40.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.671+0000 7fb13e926700 10 client.49619.objecter _send_command 8 2024-02-19T21:08:40.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.671+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb12400e270 con 0x7fb124011ff0 2024-02-19T21:08:40.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.672+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb12400e270 con 0x7fb124011ff0 2024-02-19T21:08:40.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.672+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:40.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.672+0000 7fb1357fa700 10 client.49619.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.672+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb124011970 osd=3 4 2024-02-19T21:08:40.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.672+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 3 8 2024-02-19T21:08:40.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.672+0000 7fb13e926700 10 client.49619.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:40.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.672+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 9 2024-02-19T21:08:40.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.673+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb12400e270 osd=4 3 2024-02-19T21:08:40.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.673+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb12400e270 osd=4 3 2024-02-19T21:08:40.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.673+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb12400e270 osd=4 3 2024-02-19T21:08:40.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.673+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 9 2024-02-19T21:08:40.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.673+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb12400e270 osd=4 3 2024-02-19T21:08:40.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.673+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 4 9 2024-02-19T21:08:40.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.673+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb12400e270 osd=4 4 2024-02-19T21:08:40.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.673+0000 7fb13e926700 10 client.49619.objecter _send_command 9 2024-02-19T21:08:40.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.673+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb12400cec0 con 0x7fb12400e3e0 2024-02-19T21:08:40.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.673+0000 7fb1357fa700 10 client.49619.objecter ms_handle_connect 0x7fb12400e3e0 2024-02-19T21:08:40.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.674+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb12400cec0 con 0x7fb12400e3e0 2024-02-19T21:08:40.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.674+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:08:40.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.674+0000 7fb1357fa700 10 client.49619.objecter _finish_command 9 = system:0 2024-02-19T21:08:40.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.674+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb12400e270 osd=4 4 2024-02-19T21:08:40.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.674+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 4 9 2024-02-19T21:08:40.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.689+0000 7fb13e926700 10 client.49619.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:40.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.689+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 10 2024-02-19T21:08:40.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.689+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb12400e270 osd=4 3 2024-02-19T21:08:40.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.689+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb12400e270 osd=4 3 2024-02-19T21:08:40.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.689+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb12400e270 osd=4 3 2024-02-19T21:08:40.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.689+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 10 2024-02-19T21:08:40.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.689+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb12400e270 osd=4 3 2024-02-19T21:08:40.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.689+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 4 10 2024-02-19T21:08:40.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.689+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb12400e270 osd=4 4 2024-02-19T21:08:40.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.689+0000 7fb13e926700 10 client.49619.objecter _send_command 10 2024-02-19T21:08:40.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.689+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb12400cec0 con 0x7fb12400e3e0 2024-02-19T21:08:40.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb12400cec0 con 0x7fb12400e3e0 2024-02-19T21:08:40.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:40.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb1357fa700 10 client.49619.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb12400e270 osd=4 4 2024-02-19T21:08:40.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 4 10 2024-02-19T21:08:40.441 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:08:40.441 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:08:40.441 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:08:40.441 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:08:40.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb13e926700 10 client.49619.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:40.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 11 2024-02-19T21:08:40.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb12400cec0 osd=5 3 2024-02-19T21:08:40.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb12400cec0 osd=5 3 2024-02-19T21:08:40.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb12400cec0 osd=5 3 2024-02-19T21:08:40.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 11 2024-02-19T21:08:40.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb12400cec0 osd=5 3 2024-02-19T21:08:40.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 5 11 2024-02-19T21:08:40.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb12400cec0 osd=5 4 2024-02-19T21:08:40.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb13e926700 10 client.49619.objecter _send_command 11 2024-02-19T21:08:40.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb124016bf0 con 0x7fb1240167b0 2024-02-19T21:08:40.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.691+0000 7fb1357fa700 10 client.49619.objecter ms_handle_connect 0x7fb1240167b0 2024-02-19T21:08:40.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.692+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb124016bf0 con 0x7fb1240167b0 2024-02-19T21:08:40.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.692+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:08:40.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.692+0000 7fb1357fa700 10 client.49619.objecter _finish_command 11 = system:0 2024-02-19T21:08:40.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.692+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb12400cec0 osd=5 4 2024-02-19T21:08:40.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.692+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 5 11 2024-02-19T21:08:40.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.707+0000 7fb13e926700 10 client.49619.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:40.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.707+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 12 2024-02-19T21:08:40.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.707+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb12400cec0 osd=5 3 2024-02-19T21:08:40.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.707+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb12400cec0 osd=5 3 2024-02-19T21:08:40.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.707+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb12400cec0 osd=5 3 2024-02-19T21:08:40.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.707+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 12 2024-02-19T21:08:40.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.707+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb12400cec0 osd=5 3 2024-02-19T21:08:40.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.707+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 5 12 2024-02-19T21:08:40.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.707+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb12400cec0 osd=5 4 2024-02-19T21:08:40.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.707+0000 7fb13e926700 10 client.49619.objecter _send_command 12 2024-02-19T21:08:40.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.707+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb124016bf0 con 0x7fb1240167b0 2024-02-19T21:08:40.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb124016bf0 con 0x7fb1240167b0 2024-02-19T21:08:40.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:40.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb1357fa700 10 client.49619.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb12400cec0 osd=5 4 2024-02-19T21:08:40.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 5 12 2024-02-19T21:08:40.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb13e926700 10 client.49619.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:40.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 13 2024-02-19T21:08:40.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124016bf0 osd=6 3 2024-02-19T21:08:40.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124016bf0 osd=6 3 2024-02-19T21:08:40.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124016bf0 osd=6 3 2024-02-19T21:08:40.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 13 2024-02-19T21:08:40.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb124016bf0 osd=6 3 2024-02-19T21:08:40.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 6 13 2024-02-19T21:08:40.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124016bf0 osd=6 4 2024-02-19T21:08:40.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb13e926700 10 client.49619.objecter _send_command 13 2024-02-19T21:08:40.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.709+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb124016fc0 con 0x7fb12401f190 2024-02-19T21:08:40.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.710+0000 7fb1357fa700 10 client.49619.objecter ms_handle_connect 0x7fb12401f190 2024-02-19T21:08:40.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.710+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb124016fc0 con 0x7fb12401f190 2024-02-19T21:08:40.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.710+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:08:40.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.710+0000 7fb1357fa700 10 client.49619.objecter _finish_command 13 = system:0 2024-02-19T21:08:40.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.710+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb124016bf0 osd=6 4 2024-02-19T21:08:40.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.710+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 6 13 2024-02-19T21:08:40.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.727+0000 7fb13e926700 10 client.49619.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:40.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.727+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 14 2024-02-19T21:08:40.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.727+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124016bf0 osd=6 3 2024-02-19T21:08:40.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.727+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124016bf0 osd=6 3 2024-02-19T21:08:40.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.727+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124016bf0 osd=6 3 2024-02-19T21:08:40.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.727+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 14 2024-02-19T21:08:40.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.727+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb124016bf0 osd=6 3 2024-02-19T21:08:40.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.727+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 6 14 2024-02-19T21:08:40.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.727+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124016bf0 osd=6 4 2024-02-19T21:08:40.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.727+0000 7fb13e926700 10 client.49619.objecter _send_command 14 2024-02-19T21:08:40.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.727+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb124016fc0 con 0x7fb12401f190 2024-02-19T21:08:40.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.728+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb124016fc0 con 0x7fb12401f190 2024-02-19T21:08:40.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.728+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:40.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.728+0000 7fb1357fa700 10 client.49619.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.728+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb124016bf0 osd=6 4 2024-02-19T21:08:40.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.728+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 6 14 2024-02-19T21:08:40.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.729+0000 7fb13e926700 10 client.49619.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:40.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.729+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 15 2024-02-19T21:08:40.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.729+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124016fc0 osd=7 3 2024-02-19T21:08:40.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.729+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124016fc0 osd=7 3 2024-02-19T21:08:40.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.729+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124016fc0 osd=7 3 2024-02-19T21:08:40.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.729+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 15 2024-02-19T21:08:40.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.729+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb124016fc0 osd=7 3 2024-02-19T21:08:40.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.729+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 7 15 2024-02-19T21:08:40.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.729+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124016fc0 osd=7 4 2024-02-19T21:08:40.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.729+0000 7fb13e926700 10 client.49619.objecter _send_command 15 2024-02-19T21:08:40.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.729+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb124023d90 con 0x7fb1240247c0 2024-02-19T21:08:40.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.729+0000 7fb1357fa700 10 client.49619.objecter ms_handle_connect 0x7fb1240247c0 2024-02-19T21:08:40.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.730+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb124023d90 con 0x7fb1240247c0 2024-02-19T21:08:40.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.730+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:08:40.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.730+0000 7fb1357fa700 10 client.49619.objecter _finish_command 15 = system:0 2024-02-19T21:08:40.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.730+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb124016fc0 osd=7 4 2024-02-19T21:08:40.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.730+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 7 15 2024-02-19T21:08:40.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.746+0000 7fb13e926700 10 client.49619.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:40.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.746+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign -1 16 2024-02-19T21:08:40.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.746+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124016fc0 osd=7 3 2024-02-19T21:08:40.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.746+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124016fc0 osd=7 3 2024-02-19T21:08:40.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.746+0000 7fb13e926700 20 client.49619.objecter _get_session s=0x7fb124016fc0 osd=7 3 2024-02-19T21:08:40.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.746+0000 7fb13e926700 15 client.49619.objecter _session_command_op_remove -1 16 2024-02-19T21:08:40.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.746+0000 7fb13e926700 20 client.49619.objecter get_session s=0x7fb124016fc0 osd=7 3 2024-02-19T21:08:40.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.746+0000 7fb13e926700 15 client.49619.objecter _session_command_op_assign 7 16 2024-02-19T21:08:40.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.746+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124016fc0 osd=7 4 2024-02-19T21:08:40.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.746+0000 7fb13e926700 10 client.49619.objecter _send_command 16 2024-02-19T21:08:40.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.746+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb124023d50 con 0x7fb1240247c0 2024-02-19T21:08:40.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb1357fa700 1 -- 172.21.15.163:0/1525448003 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb124023d50 con 0x7fb1240247c0 2024-02-19T21:08:40.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb1357fa700 10 client.49619.objecter ms_dispatch 0x7fb138001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:40.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb1357fa700 10 client.49619.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb1357fa700 20 client.49619.objecter put_session s=0x7fb124016fc0 osd=7 4 2024-02-19T21:08:40.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb1357fa700 15 client.49619.objecter _session_command_op_remove 7 16 2024-02-19T21:08:40.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 10 librados: watch_flush enter 2024-02-19T21:08:40.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 10 librados: watch_flush exit 2024-02-19T21:08:40.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 10 client.49619.objecter close_session for osd.0 2024-02-19T21:08:40.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 >> v1:172.21.15.163:6801/1559418115 conn(0x7fb124004530 legacy=0x7fb124002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:40.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb1240051a0 osd=0 1 2024-02-19T21:08:40.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 10 client.49619.objecter close_session for osd.1 2024-02-19T21:08:40.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 >> v1:172.21.15.163:6805/3125047432 conn(0x7fb124004150 legacy=0x7fb12400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:40.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124003fe0 osd=1 1 2024-02-19T21:08:40.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 10 client.49619.objecter close_session for osd.2 2024-02-19T21:08:40.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 >> v1:172.21.15.163:6809/1563158487 conn(0x7fb12400d2f0 legacy=0x7fb12400d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:40.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124003630 osd=2 1 2024-02-19T21:08:40.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 10 client.49619.objecter close_session for osd.3 2024-02-19T21:08:40.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 >> v1:172.21.15.163:6813/2112279769 conn(0x7fb124011ff0 legacy=0x7fb1240143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:40.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124011970 osd=3 1 2024-02-19T21:08:40.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 10 client.49619.objecter close_session for osd.4 2024-02-19T21:08:40.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13cec3700 1 -- 172.21.15.163:0/1525448003 reap_dead start 2024-02-19T21:08:40.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 >> v1:172.21.15.193:6800/3129716965 conn(0x7fb12400e3e0 legacy=0x7fb124015b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:40.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13cec3700 1 -- 172.21.15.163:0/1525448003 reap_dead start 2024-02-19T21:08:40.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb12400e270 osd=4 1 2024-02-19T21:08:40.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 10 client.49619.objecter close_session for osd.5 2024-02-19T21:08:40.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 >> v1:172.21.15.193:6804/805161991 conn(0x7fb1240167b0 legacy=0x7fb12401d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:40.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb12400cec0 osd=5 1 2024-02-19T21:08:40.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 10 client.49619.objecter close_session for osd.6 2024-02-19T21:08:40.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 >> v1:172.21.15.193:6808/787604615 conn(0x7fb12401f190 legacy=0x7fb124021580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:40.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124016bf0 osd=6 1 2024-02-19T21:08:40.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 10 client.49619.objecter close_session for osd.7 2024-02-19T21:08:40.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 >> v1:172.21.15.193:6812/157927432 conn(0x7fb1240247c0 legacy=0x7fb124024c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:40.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 20 client.49619.objecter put_session s=0x7fb124016fc0 osd=7 1 2024-02-19T21:08:40.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 20 client.49619.objecter shutdown clearing up homeless session... 2024-02-19T21:08:40.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 10 client.49619.objecter successfully canceled tick 2024-02-19T21:08:40.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 >> v1:172.21.15.163:6800/1552514681 conn(0x7fb120063c90 legacy=0x7fb120066150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:40.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 >> v1:172.21.15.163:6790/0 conn(0x7fb138102910 legacy=0x7fb138117ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:40.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13cec3700 1 -- 172.21.15.163:0/1525448003 reap_dead start 2024-02-19T21:08:40.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13cec3700 1 -- 172.21.15.163:0/1525448003 reap_dead start 2024-02-19T21:08:40.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 shutdown_connections 2024-02-19T21:08:40.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 >> 172.21.15.163:0/1525448003 conn(0x7fb13806fdc0 msgr2=0x7fb1380f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:40.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 shutdown_connections 2024-02-19T21:08:40.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.748+0000 7fb13e926700 1 -- 172.21.15.163:0/1525448003 wait complete. 2024-02-19T21:08:40.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:39.749+0000 7fb13e926700 1 librados: shutdown 2024-02-19T21:08:40.453 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.453 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.453 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.453 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.453 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.453 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.453 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.453 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:08:40.568 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:08:40.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:39.105943+0000 osd.2 (osd.2) 122 : cluster [DBG] 16.a scrub starts 2024-02-19T21:08:40.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:39.107236+0000 osd.2 (osd.2) 123 : cluster [DBG] 16.a scrub ok 2024-02-19T21:08:40.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:39.185961+0000 mgr.y (mgr.14550) 319 : cluster [DBG] pgmap v479: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:08:40.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:39.462682+0000 osd.5 (osd.5) 136 : cluster [DBG] 61.19 scrub starts 2024-02-19T21:08:40.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:39.466538+0000 osd.5 (osd.5) 137 : cluster [DBG] 61.19 scrub ok 2024-02-19T21:08:40.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:39.573702+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:08:40.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:39.574827+0000 osd.4 (osd.4) 144 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:08:40.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:39.600231+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? v1:172.21.15.163:0/1525448003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:08:40.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:39.913579+0000 mon.a (mon.0) 2232 : audit [INF] from='client.? v1:172.21.15.163:0/706886661' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-19T21:08:40.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:39.913659+0000 mon.a (mon.0) 2233 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-02-19T21:08:40.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:39.938491+0000 osd.7 (osd.7) 141 : cluster [DBG] 61.10 scrub starts 2024-02-19T21:08:40.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:39.942296+0000 osd.7 (osd.7) 142 : cluster [DBG] 61.10 scrub ok 2024-02-19T21:08:40.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:40.238751+0000 mon.a (mon.0) 2234 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-45"}]': finished 2024-02-19T21:08:40.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:40.238872+0000 mon.a (mon.0) 2235 : audit [INF] from='client.? v1:172.21.15.163:0/706886661' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-19T21:08:40.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:40.238935+0000 mon.a (mon.0) 2236 : cluster [DBG] osdmap e340: 8 total, 8 up, 7 in 2024-02-19T21:08:40.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:40.243718+0000 mon.b (mon.1) 507 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-45", "mode": "writeback"}]: dispatch 2024-02-19T21:08:40.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:40.250043+0000 mon.a (mon.0) 2237 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-45", "mode": "writeback"}]: dispatch 2024-02-19T21:08:40.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:40.251675+0000 mon.c (mon.2) 349 : audit [INF] from='client.? v1:172.21.15.163:0/555858056' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:40.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:40.255178+0000 mon.a (mon.0) 2238 : audit [INF] from='client.49613 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:40.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:39.105943+0000 osd.2 (osd.2) 122 : cluster [DBG] 16.a scrub starts 2024-02-19T21:08:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:39.107236+0000 osd.2 (osd.2) 123 : cluster [DBG] 16.a scrub ok 2024-02-19T21:08:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:39.185961+0000 mgr.y (mgr.14550) 319 : cluster [DBG] pgmap v479: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:08:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:39.462682+0000 osd.5 (osd.5) 136 : cluster [DBG] 61.19 scrub starts 2024-02-19T21:08:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:39.466538+0000 osd.5 (osd.5) 137 : cluster [DBG] 61.19 scrub ok 2024-02-19T21:08:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:39.573702+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:08:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:39.574827+0000 osd.4 (osd.4) 144 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:08:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:39.600231+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? v1:172.21.15.163:0/1525448003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:08:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:39.913579+0000 mon.a (mon.0) 2232 : audit [INF] from='client.? v1:172.21.15.163:0/706886661' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-19T21:08:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:39.913659+0000 mon.a (mon.0) 2233 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-02-19T21:08:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:39.938491+0000 osd.7 (osd.7) 141 : cluster [DBG] 61.10 scrub starts 2024-02-19T21:08:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:39.942296+0000 osd.7 (osd.7) 142 : cluster [DBG] 61.10 scrub ok 2024-02-19T21:08:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:40.238751+0000 mon.a (mon.0) 2234 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-45"}]': finished 2024-02-19T21:08:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:40.238872+0000 mon.a (mon.0) 2235 : audit [INF] from='client.? v1:172.21.15.163:0/706886661' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-19T21:08:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:40.238935+0000 mon.a (mon.0) 2236 : cluster [DBG] osdmap e340: 8 total, 8 up, 7 in 2024-02-19T21:08:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:40.243718+0000 mon.b (mon.1) 507 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-45", "mode": "writeback"}]: dispatch 2024-02-19T21:08:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:40.250043+0000 mon.a (mon.0) 2237 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-45", "mode": "writeback"}]: dispatch 2024-02-19T21:08:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:40.251675+0000 mon.c (mon.2) 349 : audit [INF] from='client.? v1:172.21.15.163:0/555858056' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:40.255178+0000 mon.a (mon.0) 2238 : audit [INF] from='client.49613 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:40.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:39.105943+0000 osd.2 (osd.2) 122 : cluster [DBG] 16.a scrub starts 2024-02-19T21:08:40.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:39.107236+0000 osd.2 (osd.2) 123 : cluster [DBG] 16.a scrub ok 2024-02-19T21:08:40.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:39.185961+0000 mgr.y (mgr.14550) 319 : cluster [DBG] pgmap v479: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:08:40.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:39.462682+0000 osd.5 (osd.5) 136 : cluster [DBG] 61.19 scrub starts 2024-02-19T21:08:40.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:39.466538+0000 osd.5 (osd.5) 137 : cluster [DBG] 61.19 scrub ok 2024-02-19T21:08:40.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:39.573702+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:08:40.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:39.574827+0000 osd.4 (osd.4) 144 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:08:40.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:39.600231+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? v1:172.21.15.163:0/1525448003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:08:40.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:39.913579+0000 mon.a (mon.0) 2232 : audit [INF] from='client.? v1:172.21.15.163:0/706886661' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-19T21:08:40.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:39.913659+0000 mon.a (mon.0) 2233 : cluster [INF] Client client.admin marked osd.4 out, while it was still marked up 2024-02-19T21:08:40.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:39.938491+0000 osd.7 (osd.7) 141 : cluster [DBG] 61.10 scrub starts 2024-02-19T21:08:40.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:39.942296+0000 osd.7 (osd.7) 142 : cluster [DBG] 61.10 scrub ok 2024-02-19T21:08:40.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:40.238751+0000 mon.a (mon.0) 2234 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-45"}]': finished 2024-02-19T21:08:40.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:40.238872+0000 mon.a (mon.0) 2235 : audit [INF] from='client.? v1:172.21.15.163:0/706886661' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-19T21:08:40.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:40.238935+0000 mon.a (mon.0) 2236 : cluster [DBG] osdmap e340: 8 total, 8 up, 7 in 2024-02-19T21:08:40.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:40.243718+0000 mon.b (mon.1) 507 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-45", "mode": "writeback"}]: dispatch 2024-02-19T21:08:40.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:40.250043+0000 mon.a (mon.0) 2237 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-45", "mode": "writeback"}]: dispatch 2024-02-19T21:08:40.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:40.251675+0000 mon.c (mon.2) 349 : audit [INF] from='client.? v1:172.21.15.163:0/555858056' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:40.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:40.255178+0000 mon.a (mon.0) 2238 : audit [INF] from='client.49613 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:40.789 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:08:40.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:40.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:08:40.893 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:08:40.920 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:41.168 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:08:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:41.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:08:41.272 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:41.343 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:41.374 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:41.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:41.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:41.478 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:41.503 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:41.528 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:41.530 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:41.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:41.632 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:41.665 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:39.699420+0000 osd.1 (osd.1) 155 : cluster [DBG] 2.d scrub starts 2024-02-19T21:08:41.665 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:39.700306+0000 osd.1 (osd.1) 156 : cluster [DBG] 2.d scrub ok 2024-02-19T21:08:41.665 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:40.268194+0000 osd.6 (osd.6) 135 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:08:41.665 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:40.270751+0000 osd.6 (osd.6) 136 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:08:41.665 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:40.305340+0000 mgr.y (mgr.14550) 320 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:41.665 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:41.234932+0000 mon.a (mon.0) 2239 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:08:41.665 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:41.238767+0000 mon.a (mon.0) 2240 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-45", "mode": "writeback"}]': finished 2024-02-19T21:08:41.665 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:41.238897+0000 mon.a (mon.0) 2241 : audit [INF] from='client.49613 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:41.665 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:41.239019+0000 mon.a (mon.0) 2242 : cluster [DBG] osdmap e341: 8 total, 8 up, 7 in 2024-02-19T21:08:41.665 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:41.241880+0000 mon.b (mon.1) 508 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:08:41.666 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:41.242666+0000 mon.a (mon.0) 2243 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:08:41.667 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:41.694 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:41.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:41.795 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:39.699420+0000 osd.1 (osd.1) 155 : cluster [DBG] 2.d scrub starts 2024-02-19T21:08:41.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:39.700306+0000 osd.1 (osd.1) 156 : cluster [DBG] 2.d scrub ok 2024-02-19T21:08:41.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:40.268194+0000 osd.6 (osd.6) 135 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:08:41.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:40.270751+0000 osd.6 (osd.6) 136 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:08:41.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:40.305340+0000 mgr.y (mgr.14550) 320 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:41.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:41.234932+0000 mon.a (mon.0) 2239 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:08:41.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:41.238767+0000 mon.a (mon.0) 2240 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-45", "mode": "writeback"}]': finished 2024-02-19T21:08:41.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:41.238897+0000 mon.a (mon.0) 2241 : audit [INF] from='client.49613 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:41.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:41.239019+0000 mon.a (mon.0) 2242 : cluster [DBG] osdmap e341: 8 total, 8 up, 7 in 2024-02-19T21:08:41.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:41.241880+0000 mon.b (mon.1) 508 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:08:41.796 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:41.242666+0000 mon.a (mon.0) 2243 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:08:41.797 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:39.699420+0000 osd.1 (osd.1) 155 : cluster [DBG] 2.d scrub starts 2024-02-19T21:08:41.797 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:39.700306+0000 osd.1 (osd.1) 156 : cluster [DBG] 2.d scrub ok 2024-02-19T21:08:41.797 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:40.268194+0000 osd.6 (osd.6) 135 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:08:41.797 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:40.270751+0000 osd.6 (osd.6) 136 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:08:41.797 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:40.305340+0000 mgr.y (mgr.14550) 320 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:41.797 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:41.234932+0000 mon.a (mon.0) 2239 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:08:41.797 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:41.238767+0000 mon.a (mon.0) 2240 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-45", "mode": "writeback"}]': finished 2024-02-19T21:08:41.797 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:41.238897+0000 mon.a (mon.0) 2241 : audit [INF] from='client.49613 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi163-83854-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:41.797 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:41.239019+0000 mon.a (mon.0) 2242 : cluster [DBG] osdmap e341: 8 total, 8 up, 7 in 2024-02-19T21:08:41.797 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:41.241880+0000 mon.b (mon.1) 508 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:08:41.797 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:41.242666+0000 mon.a (mon.0) 2243 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:08:41.798 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:41.798 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:41.798 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:41.832 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:42.282 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:42.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:42.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:42.390 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:08:42.419 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:42.521 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:08:42.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:42.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:08:42.626 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:42.660 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:42.686 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:42.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:42.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:41.141613+0000 osd.2 (osd.2) 124 : cluster [DBG] 61.11 scrub starts 2024-02-19T21:08:42.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:41.145087+0000 osd.2 (osd.2) 125 : cluster [DBG] 61.11 scrub ok 2024-02-19T21:08:42.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:41.186441+0000 mgr.y (mgr.14550) 321 : cluster [DBG] pgmap v482: 260 pgs: 32 unknown, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-19T21:08:42.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:42.241485+0000 mon.a (mon.0) 2244 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:08:42.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:42.241613+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e342: 8 total, 8 up, 7 in 2024-02-19T21:08:42.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:42.245688+0000 mon.b (mon.1) 509 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:42.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:42.246920+0000 mon.a (mon.0) 2246 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:42.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:42.273151+0000 mon.b (mon.1) 510 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:42.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:42.273849+0000 mon.a (mon.0) 2247 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:42.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:42.276691+0000 mon.b (mon.1) 511 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:42.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:42.277032+0000 mon.a (mon.0) 2248 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:42.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:42.278241+0000 mon.b (mon.1) 512 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi163-83854-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:42.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:42.278530+0000 mon.a (mon.0) 2249 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi163-83854-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:42.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:41.141613+0000 osd.2 (osd.2) 124 : cluster [DBG] 61.11 scrub starts 2024-02-19T21:08:42.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:41.145087+0000 osd.2 (osd.2) 125 : cluster [DBG] 61.11 scrub ok 2024-02-19T21:08:42.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:41.186441+0000 mgr.y (mgr.14550) 321 : cluster [DBG] pgmap v482: 260 pgs: 32 unknown, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-19T21:08:42.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:42.241485+0000 mon.a (mon.0) 2244 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:08:42.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:42.241613+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e342: 8 total, 8 up, 7 in 2024-02-19T21:08:42.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:42.245688+0000 mon.b (mon.1) 509 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:42.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:42.246920+0000 mon.a (mon.0) 2246 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:42.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:42.273151+0000 mon.b (mon.1) 510 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:42.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:42.273849+0000 mon.a (mon.0) 2247 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:42.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:42.276691+0000 mon.b (mon.1) 511 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:42.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:42.277032+0000 mon.a (mon.0) 2248 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:42.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:42.278241+0000 mon.b (mon.1) 512 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi163-83854-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:42.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:42.278530+0000 mon.a (mon.0) 2249 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi163-83854-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:42.790 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:08:42.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:41.141613+0000 osd.2 (osd.2) 124 : cluster [DBG] 61.11 scrub starts 2024-02-19T21:08:42.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:41.145087+0000 osd.2 (osd.2) 125 : cluster [DBG] 61.11 scrub ok 2024-02-19T21:08:42.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:41.186441+0000 mgr.y (mgr.14550) 321 : cluster [DBG] pgmap v482: 260 pgs: 32 unknown, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-19T21:08:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:42.241485+0000 mon.a (mon.0) 2244 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:08:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:42.241613+0000 mon.a (mon.0) 2245 : cluster [DBG] osdmap e342: 8 total, 8 up, 7 in 2024-02-19T21:08:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:42.245688+0000 mon.b (mon.1) 509 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:42.246920+0000 mon.a (mon.0) 2246 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:08:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:42.273151+0000 mon.b (mon.1) 510 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:42.273849+0000 mon.a (mon.0) 2247 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:42.276691+0000 mon.b (mon.1) 511 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:42.277032+0000 mon.a (mon.0) 2248 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:42.278241+0000 mon.b (mon.1) 512 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi163-83854-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:42.278530+0000 mon.a (mon.0) 2249 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi163-83854-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:42.820 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:43.187 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:08:43.744 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:08:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:08:43] "GET /metrics HTTP/1.1" 200 188870 "" "Prometheus/2.33.4" 2024-02-19T21:08:44.073 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:08:44.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:44.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:08:44.177 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:44.204 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:44.224 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:44.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:44.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:44.328 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:44.353 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:44.378 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:44.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:44.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:44.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:44.513 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:44.611 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:44.614 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:44.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:44.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.624+0000 7f0545164700 1 -- 172.21.15.163:0/3941041197 >> v1:172.21.15.193:6789/0 conn(0x7f05401042c0 legacy=0x7f0540106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:44.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.624+0000 7f0545164700 1 -- 172.21.15.163:0/3941041197 shutdown_connections 2024-02-19T21:08:44.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.624+0000 7f0545164700 1 -- 172.21.15.163:0/3941041197 >> 172.21.15.163:0/3941041197 conn(0x7f05400f35a0 msgr2=0x7f05400f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:44.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.624+0000 7f0545164700 1 -- 172.21.15.163:0/3941041197 shutdown_connections 2024-02-19T21:08:44.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.624+0000 7f0545164700 1 -- 172.21.15.163:0/3941041197 wait complete. 2024-02-19T21:08:44.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.625+0000 7f0545164700 1 librados: starting msgr at 2024-02-19T21:08:44.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.625+0000 7f0545164700 1 librados: starting objecter 2024-02-19T21:08:44.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.625+0000 7f0545164700 1 Processor -- start 2024-02-19T21:08:44.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.625+0000 7f0545164700 1 -- start start 2024-02-19T21:08:44.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.625+0000 7f0545164700 1 librados: setting wanted keys 2024-02-19T21:08:44.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.625+0000 7f0545164700 1 librados: calling monclient init 2024-02-19T21:08:44.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.625+0000 7f0545164700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f05400fcfa0 con 0x7f05400fd980 2024-02-19T21:08:44.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.625+0000 7f0545164700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f05400fd130 con 0x7f0540100780 2024-02-19T21:08:44.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.625+0000 7f0545164700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f05400fd320 con 0x7f05401042c0 2024-02-19T21:08:44.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f053f59e700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f05401042c0 0x7f0540118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:56610/0 (socket says 172.21.15.163:56610) 2024-02-19T21:08:44.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f053f59e700 1 -- 172.21.15.163:0/282234922 learned_addr learned my addr 172.21.15.163:0/282234922 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:44.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 10 client.?.objecter ms_handle_connect 0x7f0540100780 2024-02-19T21:08:44.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:44.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 10 client.?.objecter ms_handle_connect 0x7f05401042c0 2024-02-19T21:08:44.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:44.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 10 client.?.objecter ms_handle_connect 0x7f05400fd980 2024-02-19T21:08:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623734100 0 0) 0x7f05400fd130 con 0x7f0540100780 2024-02-19T21:08:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0528003800 con 0x7f0540100780 2024-02-19T21:08:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150330921 0 0) 0x7f05400fd320 con 0x7f05401042c0 2024-02-19T21:08:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05400fd130 con 0x7f05401042c0 2024-02-19T21:08:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845451609 0 0) 0x7f05400fcfa0 con 0x7f05400fd980 2024-02-19T21:08:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05400fd320 con 0x7f05400fd980 2024-02-19T21:08:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3954462855 0 0) 0x7f0528003800 con 0x7f0540100780 2024-02-19T21:08:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05400fcfa0 con 0x7f0540100780 2024-02-19T21:08:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f05340019f0 con 0x7f0540100780 2024-02-19T21:08:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 915649254 0 0) 0x7f05400fd320 con 0x7f05400fd980 2024-02-19T21:08:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0528003800 con 0x7f05400fd980 2024-02-19T21:08:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0530001840 con 0x7f05400fd980 2024-02-19T21:08:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1701442195 0 0) 0x7f05400fcfa0 con 0x7f0540100780 2024-02-19T21:08:44.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 >> v1:172.21.15.163:6790/0 conn(0x7f05401042c0 legacy=0x7f0540118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:44.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 >> v1:172.21.15.163:6789/0 conn(0x7f05400fd980 legacy=0x7f05400fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:44.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054011cb90 con 0x7f0540100780 2024-02-19T21:08:44.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0545164700 1 -- 172.21.15.163:0/282234922 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0540119bc0 con 0x7f0540100780 2024-02-19T21:08:44.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0545164700 10 client.48454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:44.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0545164700 1 -- 172.21.15.163:0/282234922 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f054011a030 con 0x7f0540100780 2024-02-19T21:08:44.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0545164700 1 librados: init done 2024-02-19T21:08:44.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f0534002360 con 0x7f0540100780 2024-02-19T21:08:44.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.626+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0534003340 con 0x7f0540100780 2024-02-19T21:08:44.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.628+0000 7f0545164700 1 -- 172.21.15.163:0/282234922 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052c004f20 con 0x7f0540100780 2024-02-19T21:08:44.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.628+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f05340034a0 con 0x7f0540100780 2024-02-19T21:08:44.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.628+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(344..344 src has 43..344) v4 ==== 8360+0+0 (unknown 3817759938 0 0) 0x7f0534076530 con 0x7f0540100780 2024-02-19T21:08:44.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.628+0000 7f0527fff700 10 client.48454.objecter ms_dispatch 0x7f0540001550 osd_map(344..344 src has 43..344) v4 2024-02-19T21:08:44.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.628+0000 7f0527fff700 3 client.48454.objecter handle_osd_map got epochs [344,344] > 0 2024-02-19T21:08:44.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.628+0000 7f0527fff700 3 client.48454.objecter handle_osd_map decoding full epoch 344 2024-02-19T21:08:44.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.628+0000 7f0527fff700 20 client.48454.objecter dump_active .. 0 homeless 2024-02-19T21:08:44.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.628+0000 7f0527fff700 10 client.48454.objecter ms_handle_connect 0x7f0528063a70 2024-02-19T21:08:44.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.632+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0534049800 con 0x7f0540100780 2024-02-19T21:08:44.717 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:44.741 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:43.186793+0000 mgr.y (mgr.14550) 322 : cluster [DBG] pgmap v485: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-19T21:08:44.741 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:43.241208+0000 mon.a (mon.0) 2250 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:08:44.741 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:43.241308+0000 mon.a (mon.0) 2251 : audit [INF] from='client.48448 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi163-83854-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:44.741 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:43.241395+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e343: 8 total, 8 up, 7 in 2024-02-19T21:08:44.741 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:43.247781+0000 mon.b (mon.1) 513 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi163-83854-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:44.742 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:43.249520+0000 mon.a (mon.0) 2253 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi163-83854-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:44.742 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:43.249535+0000 mon.b (mon.1) 514 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:08:44.742 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:43.249936+0000 mon.a (mon.0) 2254 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:08:44.742 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:44.226086+0000 mon.a (mon.0) 2255 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:08:44.743 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:44.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:43.186793+0000 mgr.y (mgr.14550) 322 : cluster [DBG] pgmap v485: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-19T21:08:44.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:43.241208+0000 mon.a (mon.0) 2250 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:08:44.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:43.241308+0000 mon.a (mon.0) 2251 : audit [INF] from='client.48448 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi163-83854-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:44.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:43.241395+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e343: 8 total, 8 up, 7 in 2024-02-19T21:08:44.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:43.247781+0000 mon.b (mon.1) 513 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi163-83854-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:44.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:43.249520+0000 mon.a (mon.0) 2253 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi163-83854-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:44.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:43.249535+0000 mon.b (mon.1) 514 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:08:44.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:43.249936+0000 mon.a (mon.0) 2254 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:08:44.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:44.226086+0000 mon.a (mon.0) 2255 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:08:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:43.186793+0000 mgr.y (mgr.14550) 322 : cluster [DBG] pgmap v485: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-02-19T21:08:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:43.241208+0000 mon.a (mon.0) 2250 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:08:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:43.241308+0000 mon.a (mon.0) 2251 : audit [INF] from='client.48448 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi163-83854-60", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:43.241395+0000 mon.a (mon.0) 2252 : cluster [DBG] osdmap e343: 8 total, 8 up, 7 in 2024-02-19T21:08:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:43.247781+0000 mon.b (mon.1) 513 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi163-83854-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:43.249520+0000 mon.a (mon.0) 2253 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi163-83854-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:44.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:43.249535+0000 mon.b (mon.1) 514 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:08:44.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:43.249936+0000 mon.a (mon.0) 2254 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:08:44.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:44.226086+0000 mon.a (mon.0) 2255 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:08:44.762 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:44.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:44.815+0000 7f0545164700 1 -- 172.21.15.163:0/282234922 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f052c0051d0 con 0x7f0540100780 2024-02-19T21:08:44.866 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:44.891 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:44.910 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:44.913 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:44.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:45.014 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:45.040 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:45.059 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:45.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:45.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:45.162 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:08:45.193 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:45.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:45.232+0000 7f0527fff700 1 -- 172.21.15.163:0/282234922 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v345) v1 ==== 101+0+0 (unknown 48508054 0 0) 0x7f0534047c10 con 0x7f0540100780 2024-02-19T21:08:45.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:45.235+0000 7f0545164700 10 librados: watch_flush enter 2024-02-19T21:08:45.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:45.235+0000 7f0545164700 10 librados: watch_flush exit 2024-02-19T21:08:45.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:45.235+0000 7f0545164700 20 client.48454.objecter shutdown clearing up homeless session... 2024-02-19T21:08:45.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:45.235+0000 7f0545164700 10 client.48454.objecter successfully canceled tick 2024-02-19T21:08:45.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:45.235+0000 7f0545164700 1 -- 172.21.15.163:0/282234922 >> v1:172.21.15.163:6800/1552514681 conn(0x7f0528063a70 legacy=0x7f0528065f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:45.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:45.235+0000 7f0545164700 1 -- 172.21.15.163:0/282234922 >> v1:172.21.15.193:6789/0 conn(0x7f0540100780 legacy=0x7f0540114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:45.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:45.236+0000 7f0545164700 1 -- 172.21.15.163:0/282234922 shutdown_connections 2024-02-19T21:08:45.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:45.236+0000 7f0545164700 1 -- 172.21.15.163:0/282234922 >> 172.21.15.163:0/282234922 conn(0x7f05400f35a0 msgr2=0x7f0540107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:45.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:45.236+0000 7f0545164700 1 -- 172.21.15.163:0/282234922 shutdown_connections 2024-02-19T21:08:45.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:45.236+0000 7f0545164700 1 -- 172.21.15.163:0/282234922 wait complete. 2024-02-19T21:08:45.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:45.237+0000 7f0545164700 1 librados: shutdown 2024-02-19T21:08:45.237 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:08:45.308 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:08:45.308 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:08:45.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:44.475900+0000 mon.a (mon.0) 2256 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:08:45.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:44.475976+0000 mon.a (mon.0) 2257 : cluster [DBG] osdmap e344: 8 total, 8 up, 7 in 2024-02-19T21:08:45.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:44.476601+0000 mon.b (mon.1) 515 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:08:45.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:44.483278+0000 mon.a (mon.0) 2258 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:08:45.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:44.800311+0000 osd.7 (osd.7) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:45.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:44.800454+0000 osd.7 (osd.7) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:45.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:44.817154+0000 mon.b (mon.1) 516 : audit [INF] from='client.? v1:172.21.15.163:0/282234922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:45.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:44.817409+0000 mon.a (mon.0) 2259 : audit [INF] from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:45.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:45.227778+0000 mon.a (mon.0) 2260 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:08:45.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:45.232111+0000 mon.a (mon.0) 2261 : audit [INF] from='client.48448 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi163-83854-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi163-83854-60"}]': finished 2024-02-19T21:08:45.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:45.232217+0000 mon.a (mon.0) 2262 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-19T21:08:45.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:45.232286+0000 mon.a (mon.0) 2263 : audit [INF] from='client.48454 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:08:45.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:45.232314+0000 mon.a (mon.0) 2264 : cluster [DBG] osdmap e345: 8 total, 8 up, 7 in 2024-02-19T21:08:45.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:45.237793+0000 mon.b (mon.1) 517 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-19T21:08:45.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:45.238139+0000 mon.a (mon.0) 2265 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-19T21:08:45.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:44.475900+0000 mon.a (mon.0) 2256 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:08:45.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:44.475976+0000 mon.a (mon.0) 2257 : cluster [DBG] osdmap e344: 8 total, 8 up, 7 in 2024-02-19T21:08:45.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:44.476601+0000 mon.b (mon.1) 515 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:08:45.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:44.483278+0000 mon.a (mon.0) 2258 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:08:45.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:44.800311+0000 osd.7 (osd.7) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:45.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:44.800454+0000 osd.7 (osd.7) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:45.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:44.817154+0000 mon.b (mon.1) 516 : audit [INF] from='client.? v1:172.21.15.163:0/282234922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:45.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:44.817409+0000 mon.a (mon.0) 2259 : audit [INF] from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:45.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:45.227778+0000 mon.a (mon.0) 2260 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:08:45.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:45.232111+0000 mon.a (mon.0) 2261 : audit [INF] from='client.48448 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi163-83854-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi163-83854-60"}]': finished 2024-02-19T21:08:45.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:45.232217+0000 mon.a (mon.0) 2262 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-19T21:08:45.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:45.232286+0000 mon.a (mon.0) 2263 : audit [INF] from='client.48454 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:08:45.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:45.232314+0000 mon.a (mon.0) 2264 : cluster [DBG] osdmap e345: 8 total, 8 up, 7 in 2024-02-19T21:08:45.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:45.237793+0000 mon.b (mon.1) 517 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-19T21:08:45.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:45.238139+0000 mon.a (mon.0) 2265 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-19T21:08:45.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:44.475900+0000 mon.a (mon.0) 2256 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:08:45.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:44.475976+0000 mon.a (mon.0) 2257 : cluster [DBG] osdmap e344: 8 total, 8 up, 7 in 2024-02-19T21:08:45.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:44.476601+0000 mon.b (mon.1) 515 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:08:45.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:44.483278+0000 mon.a (mon.0) 2258 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:08:45.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:44.800311+0000 osd.7 (osd.7) 143 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:08:45.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:44.800454+0000 osd.7 (osd.7) 144 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:08:45.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:44.817154+0000 mon.b (mon.1) 516 : audit [INF] from='client.? v1:172.21.15.163:0/282234922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:45.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:44.817409+0000 mon.a (mon.0) 2259 : audit [INF] from='client.48454 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:08:45.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:45.227778+0000 mon.a (mon.0) 2260 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:08:45.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:45.232111+0000 mon.a (mon.0) 2261 : audit [INF] from='client.48448 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi163-83854-60", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi163-83854-60"}]': finished 2024-02-19T21:08:45.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:45.232217+0000 mon.a (mon.0) 2262 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-19T21:08:45.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:45.232286+0000 mon.a (mon.0) 2263 : audit [INF] from='client.48454 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:08:45.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:45.232314+0000 mon.a (mon.0) 2264 : cluster [DBG] osdmap e345: 8 total, 8 up, 7 in 2024-02-19T21:08:45.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:45.237793+0000 mon.b (mon.1) 517 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-19T21:08:45.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:45.238139+0000 mon.a (mon.0) 2265 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-19T21:08:46.243 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:08:46.247 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:46.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:08:46.348 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:46.476 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:46.481 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:46.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:46.582 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:46.609 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:46.628 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:46.631 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:46.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:46.732 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:08:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:45.187912+0000 mgr.y (mgr.14550) 323 : cluster [DBG] pgmap v488: 228 pgs: 1 active+recovering+undersized+remapped, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9/1008 objects misplaced (0.893%); 411 B/s, 66 objects/s recovering 2024-02-19T21:08:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:46.233500+0000 mon.a (mon.0) 2266 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-19T21:08:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:46.233624+0000 mon.a (mon.0) 2267 : cluster [DBG] osdmap e346: 8 total, 8 up, 7 in 2024-02-19T21:08:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:46.234560+0000 mon.b (mon.1) 518 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-19T21:08:46.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:46.239691+0000 mon.a (mon.0) 2268 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-19T21:08:46.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:45.187912+0000 mgr.y (mgr.14550) 323 : cluster [DBG] pgmap v488: 228 pgs: 1 active+recovering+undersized+remapped, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9/1008 objects misplaced (0.893%); 411 B/s, 66 objects/s recovering 2024-02-19T21:08:46.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:46.233500+0000 mon.a (mon.0) 2266 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-19T21:08:46.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:46.233624+0000 mon.a (mon.0) 2267 : cluster [DBG] osdmap e346: 8 total, 8 up, 7 in 2024-02-19T21:08:46.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:46.234560+0000 mon.b (mon.1) 518 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-19T21:08:46.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:46.239691+0000 mon.a (mon.0) 2268 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-19T21:08:46.761 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:46.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:45.187912+0000 mgr.y (mgr.14550) 323 : cluster [DBG] pgmap v488: 228 pgs: 1 active+recovering+undersized+remapped, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9/1008 objects misplaced (0.893%); 411 B/s, 66 objects/s recovering 2024-02-19T21:08:46.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:46.233500+0000 mon.a (mon.0) 2266 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-19T21:08:46.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:46.233624+0000 mon.a (mon.0) 2267 : cluster [DBG] osdmap e346: 8 total, 8 up, 7 in 2024-02-19T21:08:46.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:46.234560+0000 mon.b (mon.1) 518 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-19T21:08:46.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:46.239691+0000 mon.a (mon.0) 2268 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-19T21:08:46.822 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 3] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:08:46.823 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 0 1 2024-02-19T21:08:47.010 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:08:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:47.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:08:47.115 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:47.141 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:47.159 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:47.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:47.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:47.263 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:47.290 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:47.308 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:47.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:47.411 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:47.438 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:47.456 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:47.458 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:47.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:47.559 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:47.586 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:47.604 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:47.606 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:47.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:47.707 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:47.734 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:47.753 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:47.755 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:47.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:47.856 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:47.882 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:47.900 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:47.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:48.004 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:48.029 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:08:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:08:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:08:48.030 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:48.049 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:48.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:48.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:48.060 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-19T21:08:48.152 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:48.179 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:48.197 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:48.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:48.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:48.300 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:08:48.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:47.188525+0000 mgr.y (mgr.14550) 324 : cluster [DBG] pgmap v491: 236 pgs: 8 unknown, 1 active+recovering+undersized+remapped, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9/1008 objects misplaced (0.893%); 411 B/s, 66 objects/s recovering 2024-02-19T21:08:48.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:47.235471+0000 mon.a (mon.0) 2269 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-19T21:08:48.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:47.235587+0000 mon.a (mon.0) 2270 : cluster [DBG] osdmap e347: 8 total, 8 up, 7 in 2024-02-19T21:08:48.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:47.237841+0000 mon.b (mon.1) 519 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:48.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:47.240960+0000 mon.a (mon.0) 2271 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:48.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:47.274755+0000 mon.b (mon.1) 520 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:48.319 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:47.275135+0000 mon.a (mon.0) 2272 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:48.327 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:47.188525+0000 mgr.y (mgr.14550) 324 : cluster [DBG] pgmap v491: 236 pgs: 8 unknown, 1 active+recovering+undersized+remapped, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9/1008 objects misplaced (0.893%); 411 B/s, 66 objects/s recovering 2024-02-19T21:08:48.327 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:47.235471+0000 mon.a (mon.0) 2269 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-19T21:08:48.327 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:47.235587+0000 mon.a (mon.0) 2270 : cluster [DBG] osdmap e347: 8 total, 8 up, 7 in 2024-02-19T21:08:48.327 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:47.237841+0000 mon.b (mon.1) 519 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:48.327 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:47.240960+0000 mon.a (mon.0) 2271 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:48.327 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:47.274755+0000 mon.b (mon.1) 520 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:48.327 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:47.275135+0000 mon.a (mon.0) 2272 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:48.328 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:47.188525+0000 mgr.y (mgr.14550) 324 : cluster [DBG] pgmap v491: 236 pgs: 8 unknown, 1 active+recovering+undersized+remapped, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9/1008 objects misplaced (0.893%); 411 B/s, 66 objects/s recovering 2024-02-19T21:08:48.328 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:47.235471+0000 mon.a (mon.0) 2269 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-45","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-19T21:08:48.329 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:47.235587+0000 mon.a (mon.0) 2270 : cluster [DBG] osdmap e347: 8 total, 8 up, 7 in 2024-02-19T21:08:48.329 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:47.237841+0000 mon.b (mon.1) 519 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:48.329 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:47.240960+0000 mon.a (mon.0) 2271 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:48.329 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:47.274755+0000 mon.b (mon.1) 520 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:48.329 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:47.275135+0000 mon.a (mon.0) 2272 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:48.330 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:48.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.334+0000 7f5a342fb700 1 -- 172.21.15.163:0/2668993366 >> v1:172.21.15.163:6790/0 conn(0x7f5a2c100780 legacy=0x7f5a2c102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:48.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.334+0000 7f5a342fb700 1 -- 172.21.15.163:0/2668993366 shutdown_connections 2024-02-19T21:08:48.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.334+0000 7f5a342fb700 1 -- 172.21.15.163:0/2668993366 >> 172.21.15.163:0/2668993366 conn(0x7f5a2c0f35a0 msgr2=0x7f5a2c0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:48.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.334+0000 7f5a342fb700 1 -- 172.21.15.163:0/2668993366 shutdown_connections 2024-02-19T21:08:48.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.334+0000 7f5a342fb700 1 -- 172.21.15.163:0/2668993366 wait complete. 2024-02-19T21:08:48.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.335+0000 7f5a342fb700 1 librados: starting msgr at 2024-02-19T21:08:48.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.335+0000 7f5a342fb700 1 librados: starting objecter 2024-02-19T21:08:48.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.335+0000 7f5a342fb700 1 Processor -- start 2024-02-19T21:08:48.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.335+0000 7f5a342fb700 1 -- start start 2024-02-19T21:08:48.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.335+0000 7f5a342fb700 1 librados: setting wanted keys 2024-02-19T21:08:48.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.335+0000 7f5a342fb700 1 librados: calling monclient init 2024-02-19T21:08:48.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.335+0000 7f5a342fb700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5a2c117380 con 0x7f5a2c1042c0 2024-02-19T21:08:48.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.335+0000 7f5a342fb700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5a2c118540 con 0x7f5a2c100780 2024-02-19T21:08:48.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.335+0000 7f5a342fb700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5a2c119740 con 0x7f5a2c0fd980 2024-02-19T21:08:48.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.335+0000 7f5a32097700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f5a2c0fd980 0x7f5a2c0f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:56626/0 (socket says 172.21.15.163:56626) 2024-02-19T21:08:48.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.335+0000 7f5a32097700 1 -- 172.21.15.163:0/3601168909 learned_addr learned my addr 172.21.15.163:0/3601168909 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:48.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 10 client.?.objecter ms_handle_connect 0x7f5a2c0fd980 2024-02-19T21:08:48.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:48.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 10 client.?.objecter ms_handle_connect 0x7f5a2c100780 2024-02-19T21:08:48.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:48.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 10 client.?.objecter ms_handle_connect 0x7f5a2c1042c0 2024-02-19T21:08:48.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:48.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30077243 0 0) 0x7f5a2c118540 con 0x7f5a2c100780 2024-02-19T21:08:48.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a18003800 con 0x7f5a2c100780 2024-02-19T21:08:48.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879655677 0 0) 0x7f5a2c117380 con 0x7f5a2c1042c0 2024-02-19T21:08:48.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a2c118540 con 0x7f5a2c1042c0 2024-02-19T21:08:48.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891622776 0 0) 0x7f5a2c119740 con 0x7f5a2c0fd980 2024-02-19T21:08:48.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a2c117380 con 0x7f5a2c0fd980 2024-02-19T21:08:48.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3443421106 0 0) 0x7f5a18003800 con 0x7f5a2c100780 2024-02-19T21:08:48.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a2c119740 con 0x7f5a2c100780 2024-02-19T21:08:48.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5a1c0015b0 con 0x7f5a2c100780 2024-02-19T21:08:48.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 221661532 0 0) 0x7f5a2c118540 con 0x7f5a2c1042c0 2024-02-19T21:08:48.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a18003800 con 0x7f5a2c1042c0 2024-02-19T21:08:48.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5a24001bc0 con 0x7f5a2c1042c0 2024-02-19T21:08:48.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2043806119 0 0) 0x7f5a2c119740 con 0x7f5a2c100780 2024-02-19T21:08:48.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 >> v1:172.21.15.163:6790/0 conn(0x7f5a2c0fd980 legacy=0x7f5a2c0f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:48.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 >> v1:172.21.15.163:6789/0 conn(0x7f5a2c1042c0 legacy=0x7f5a2c115a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:48.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c11a940 con 0x7f5a2c100780 2024-02-19T21:08:48.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2c118730 con 0x7f5a2c100780 2024-02-19T21:08:48.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a342fb700 10 client.48460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:48.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2c118ba0 con 0x7f5a2c100780 2024-02-19T21:08:48.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a342fb700 1 librados: init done 2024-02-19T21:08:48.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f5a1c001b50 con 0x7f5a2c100780 2024-02-19T21:08:48.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.336+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5a1c0036a0 con 0x7f5a2c100780 2024-02-19T21:08:48.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.337+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f5a10004f00 con 0x7f5a2c100780 2024-02-19T21:08:48.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.337+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f5a1c016530 con 0x7f5a2c100780 2024-02-19T21:08:48.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.338+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(348..348 src has 43..348) v4 ==== 8043+0+0 (unknown 1672174871 0 0) 0x7f5a1c076730 con 0x7f5a2c100780 2024-02-19T21:08:48.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.338+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 osd_map(348..348 src has 43..348) v4 2024-02-19T21:08:48.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.338+0000 7f5a237fe700 3 client.48460.objecter handle_osd_map got epochs [348,348] > 0 2024-02-19T21:08:48.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.338+0000 7f5a237fe700 3 client.48460.objecter handle_osd_map decoding full epoch 348 2024-02-19T21:08:48.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.338+0000 7f5a237fe700 20 client.48460.objecter dump_active .. 0 homeless 2024-02-19T21:08:48.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.338+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v348) v1 ==== 54+0+16 (unknown 3527797098 0 2420000192) 0x7f5a1c078b20 con 0x7f5a2c100780 2024-02-19T21:08:48.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.338+0000 7f5a237fe700 10 client.48460.objecter ms_handle_connect 0x7f5a18063c00 2024-02-19T21:08:48.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.339+0000 7f5a342fb700 10 client.48460.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:48.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.339+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 1 2024-02-19T21:08:48.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.339+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100051a0 osd=0 3 2024-02-19T21:08:48.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.339+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100051a0 osd=0 3 2024-02-19T21:08:48.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.339+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100051a0 osd=0 3 2024-02-19T21:08:48.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.339+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 1 2024-02-19T21:08:48.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.339+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a100051a0 osd=0 3 2024-02-19T21:08:48.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.339+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 0 1 2024-02-19T21:08:48.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.339+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100051a0 osd=0 4 2024-02-19T21:08:48.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.339+0000 7f5a342fb700 10 client.48460.objecter _send_command 1 2024-02-19T21:08:48.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.339+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a10004950 con 0x7f5a10004530 2024-02-19T21:08:48.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.339+0000 7f5a237fe700 10 client.48460.objecter ms_handle_connect 0x7f5a10004530 2024-02-19T21:08:48.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.340+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5a10004950 con 0x7f5a10004530 2024-02-19T21:08:48.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.340+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:08:48.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.340+0000 7f5a237fe700 10 client.48460.objecter _finish_command 1 = system:0 2024-02-19T21:08:48.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.340+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a100051a0 osd=0 4 2024-02-19T21:08:48.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.340+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 0 1 2024-02-19T21:08:48.352 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:08:48.354 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:48.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:08:48.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.358+0000 7f5a342fb700 10 client.48460.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:48.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.358+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 2 2024-02-19T21:08:48.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.358+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100051a0 osd=0 3 2024-02-19T21:08:48.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.358+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100051a0 osd=0 3 2024-02-19T21:08:48.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.358+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100051a0 osd=0 3 2024-02-19T21:08:48.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.358+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 2 2024-02-19T21:08:48.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.358+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a100051a0 osd=0 3 2024-02-19T21:08:48.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.358+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 0 2 2024-02-19T21:08:48.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.358+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100051a0 osd=0 4 2024-02-19T21:08:48.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.358+0000 7f5a342fb700 10 client.48460.objecter _send_command 2 2024-02-19T21:08:48.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.358+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5a10004d30 con 0x7f5a10004530 2024-02-19T21:08:48.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5a10004d30 con 0x7f5a10004530 2024-02-19T21:08:48.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:48.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a237fe700 10 client.48460.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:48.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a100051a0 osd=0 4 2024-02-19T21:08:48.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 0 2 2024-02-19T21:08:48.358 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:08:48.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a342fb700 10 client.48460.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:48.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 3 2024-02-19T21:08:48.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a10003fe0 osd=1 3 2024-02-19T21:08:48.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a10003fe0 osd=1 3 2024-02-19T21:08:48.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a10003fe0 osd=1 3 2024-02-19T21:08:48.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 3 2024-02-19T21:08:48.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a10003fe0 osd=1 3 2024-02-19T21:08:48.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 1 3 2024-02-19T21:08:48.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a10003fe0 osd=1 4 2024-02-19T21:08:48.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a342fb700 10 client.48460.objecter _send_command 3 2024-02-19T21:08:48.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.360+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a10010450 con 0x7f5a10004150 2024-02-19T21:08:48.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.361+0000 7f5a237fe700 10 client.48460.objecter ms_handle_connect 0x7f5a10004150 2024-02-19T21:08:48.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.361+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5a10010450 con 0x7f5a10004150 2024-02-19T21:08:48.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.361+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:08:48.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.361+0000 7f5a237fe700 10 client.48460.objecter _finish_command 3 = system:0 2024-02-19T21:08:48.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.361+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a10003fe0 osd=1 4 2024-02-19T21:08:48.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.361+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 1 3 2024-02-19T21:08:48.374 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:48.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.382+0000 7f5a342fb700 10 client.48460.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:48.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.382+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 4 2024-02-19T21:08:48.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.382+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a10003fe0 osd=1 3 2024-02-19T21:08:48.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.382+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a10003fe0 osd=1 3 2024-02-19T21:08:48.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.382+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a10003fe0 osd=1 3 2024-02-19T21:08:48.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.382+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 4 2024-02-19T21:08:48.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.382+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a10003fe0 osd=1 3 2024-02-19T21:08:48.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.382+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 1 4 2024-02-19T21:08:48.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.382+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a10003fe0 osd=1 4 2024-02-19T21:08:48.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.382+0000 7f5a342fb700 10 client.48460.objecter _send_command 4 2024-02-19T21:08:48.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.382+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5a100018b0 con 0x7f5a10004150 2024-02-19T21:08:48.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5a100018b0 con 0x7f5a10004150 2024-02-19T21:08:48.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:48.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a237fe700 10 client.48460.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:48.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a10003fe0 osd=1 4 2024-02-19T21:08:48.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 1 4 2024-02-19T21:08:48.383 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:08:48.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a342fb700 10 client.48460.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:48.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 5 2024-02-19T21:08:48.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a10003630 osd=2 3 2024-02-19T21:08:48.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a10003630 osd=2 3 2024-02-19T21:08:48.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a10003630 osd=2 3 2024-02-19T21:08:48.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 5 2024-02-19T21:08:48.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a10003630 osd=2 3 2024-02-19T21:08:48.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 2 5 2024-02-19T21:08:48.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a10003630 osd=2 4 2024-02-19T21:08:48.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a342fb700 10 client.48460.objecter _send_command 5 2024-02-19T21:08:48.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.385+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a100122a0 con 0x7f5a10011740 2024-02-19T21:08:48.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.386+0000 7f5a237fe700 10 client.48460.objecter ms_handle_connect 0x7f5a10011740 2024-02-19T21:08:48.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.386+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5a100122a0 con 0x7f5a10011740 2024-02-19T21:08:48.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.386+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:08:48.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.386+0000 7f5a237fe700 10 client.48460.objecter _finish_command 5 = system:0 2024-02-19T21:08:48.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.386+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a10003630 osd=2 4 2024-02-19T21:08:48.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.386+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 2 5 2024-02-19T21:08:48.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.402+0000 7f5a342fb700 10 client.48460.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:48.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.402+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 6 2024-02-19T21:08:48.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.402+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a10003630 osd=2 3 2024-02-19T21:08:48.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.402+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a10003630 osd=2 3 2024-02-19T21:08:48.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.402+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a10003630 osd=2 3 2024-02-19T21:08:48.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.402+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 6 2024-02-19T21:08:48.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.402+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a10003630 osd=2 3 2024-02-19T21:08:48.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.402+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 2 6 2024-02-19T21:08:48.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.402+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a10003630 osd=2 4 2024-02-19T21:08:48.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.402+0000 7f5a342fb700 10 client.48460.objecter _send_command 6 2024-02-19T21:08:48.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.402+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5a100030a0 con 0x7f5a10011740 2024-02-19T21:08:48.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5a100030a0 con 0x7f5a10011740 2024-02-19T21:08:48.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:48.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a237fe700 10 client.48460.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:48.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a10003630 osd=2 4 2024-02-19T21:08:48.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 2 6 2024-02-19T21:08:48.402 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:08:48.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a342fb700 10 client.48460.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:48.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 7 2024-02-19T21:08:48.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100034b0 osd=3 3 2024-02-19T21:08:48.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100034b0 osd=3 3 2024-02-19T21:08:48.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100034b0 osd=3 3 2024-02-19T21:08:48.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 7 2024-02-19T21:08:48.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a100034b0 osd=3 3 2024-02-19T21:08:48.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 3 7 2024-02-19T21:08:48.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100034b0 osd=3 4 2024-02-19T21:08:48.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a342fb700 10 client.48460.objecter _send_command 7 2024-02-19T21:08:48.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a10019f20 con 0x7f5a100162a0 2024-02-19T21:08:48.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.405+0000 7f5a237fe700 10 client.48460.objecter ms_handle_connect 0x7f5a100162a0 2024-02-19T21:08:48.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.406+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5a10019f20 con 0x7f5a100162a0 2024-02-19T21:08:48.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.406+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:08:48.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.406+0000 7f5a237fe700 10 client.48460.objecter _finish_command 7 = system:0 2024-02-19T21:08:48.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.406+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a100034b0 osd=3 4 2024-02-19T21:08:48.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.406+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 3 7 2024-02-19T21:08:48.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.422+0000 7f5a342fb700 10 client.48460.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:48.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.422+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 8 2024-02-19T21:08:48.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.422+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100034b0 osd=3 3 2024-02-19T21:08:48.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.422+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100034b0 osd=3 3 2024-02-19T21:08:48.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.422+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100034b0 osd=3 3 2024-02-19T21:08:48.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.422+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 8 2024-02-19T21:08:48.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.422+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a100034b0 osd=3 3 2024-02-19T21:08:48.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.422+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 3 8 2024-02-19T21:08:48.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.422+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100034b0 osd=3 4 2024-02-19T21:08:48.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.422+0000 7f5a342fb700 10 client.48460.objecter _send_command 8 2024-02-19T21:08:48.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.422+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5a10012680 con 0x7f5a100162a0 2024-02-19T21:08:48.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5a10012680 con 0x7f5a100162a0 2024-02-19T21:08:48.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:48.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a237fe700 10 client.48460.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:48.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a100034b0 osd=3 4 2024-02-19T21:08:48.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 3 8 2024-02-19T21:08:48.421 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:08:48.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a342fb700 10 client.48460.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:48.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 9 2024-02-19T21:08:48.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a10012440 osd=4 3 2024-02-19T21:08:48.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a10012440 osd=4 3 2024-02-19T21:08:48.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a10012440 osd=4 3 2024-02-19T21:08:48.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 9 2024-02-19T21:08:48.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a10012440 osd=4 3 2024-02-19T21:08:48.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 4 9 2024-02-19T21:08:48.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a10012440 osd=4 4 2024-02-19T21:08:48.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a342fb700 10 client.48460.objecter _send_command 9 2024-02-19T21:08:48.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a100125f0 con 0x7f5a100112a0 2024-02-19T21:08:48.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.424+0000 7f5a237fe700 10 client.48460.objecter ms_handle_connect 0x7f5a100112a0 2024-02-19T21:08:48.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.425+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5a100125f0 con 0x7f5a100112a0 2024-02-19T21:08:48.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.425+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:08:48.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.425+0000 7f5a237fe700 10 client.48460.objecter _finish_command 9 = system:0 2024-02-19T21:08:48.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.425+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a10012440 osd=4 4 2024-02-19T21:08:48.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.425+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 4 9 2024-02-19T21:08:48.432 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:48.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.441+0000 7f5a342fb700 10 client.48460.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:48.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.441+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 10 2024-02-19T21:08:48.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.441+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a10012440 osd=4 3 2024-02-19T21:08:48.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.441+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a10012440 osd=4 3 2024-02-19T21:08:48.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.441+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a10012440 osd=4 3 2024-02-19T21:08:48.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.441+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 10 2024-02-19T21:08:48.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.441+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a10012440 osd=4 3 2024-02-19T21:08:48.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.441+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 4 10 2024-02-19T21:08:48.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.441+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a10012440 osd=4 4 2024-02-19T21:08:48.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.441+0000 7f5a342fb700 10 client.48460.objecter _send_command 10 2024-02-19T21:08:48.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.441+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5a100125f0 con 0x7f5a100112a0 2024-02-19T21:08:48.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5a100125f0 con 0x7f5a100112a0 2024-02-19T21:08:48.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:48.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a237fe700 10 client.48460.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:48.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a10012440 osd=4 4 2024-02-19T21:08:48.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 4 10 2024-02-19T21:08:48.440 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:08:48.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a342fb700 10 client.48460.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:48.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 11 2024-02-19T21:08:48.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100125f0 osd=5 3 2024-02-19T21:08:48.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100125f0 osd=5 3 2024-02-19T21:08:48.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100125f0 osd=5 3 2024-02-19T21:08:48.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 11 2024-02-19T21:08:48.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a100125f0 osd=5 3 2024-02-19T21:08:48.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 5 11 2024-02-19T21:08:48.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100125f0 osd=5 4 2024-02-19T21:08:48.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a342fb700 10 client.48460.objecter _send_command 11 2024-02-19T21:08:48.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.442+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a1001b340 con 0x7f5a1001ad30 2024-02-19T21:08:48.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.443+0000 7f5a237fe700 10 client.48460.objecter ms_handle_connect 0x7f5a1001ad30 2024-02-19T21:08:48.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.444+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5a1001b340 con 0x7f5a1001ad30 2024-02-19T21:08:48.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.444+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:08:48.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.444+0000 7f5a237fe700 10 client.48460.objecter _finish_command 11 = system:0 2024-02-19T21:08:48.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.444+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a100125f0 osd=5 4 2024-02-19T21:08:48.442 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.444+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 5 11 2024-02-19T21:08:48.455 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:08:48.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.459+0000 7f5a342fb700 10 client.48460.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:48.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.459+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 12 2024-02-19T21:08:48.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.459+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100125f0 osd=5 3 2024-02-19T21:08:48.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.459+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100125f0 osd=5 3 2024-02-19T21:08:48.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.459+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100125f0 osd=5 3 2024-02-19T21:08:48.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.459+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 12 2024-02-19T21:08:48.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.459+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a100125f0 osd=5 3 2024-02-19T21:08:48.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.459+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 5 12 2024-02-19T21:08:48.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.459+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100125f0 osd=5 4 2024-02-19T21:08:48.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.459+0000 7f5a342fb700 10 client.48460.objecter _send_command 12 2024-02-19T21:08:48.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.459+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5a1001b340 con 0x7f5a1001ad30 2024-02-19T21:08:48.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5a1001b340 con 0x7f5a1001ad30 2024-02-19T21:08:48.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:48.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a237fe700 10 client.48460.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:48.458 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a100125f0 osd=5 4 2024-02-19T21:08:48.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 5 12 2024-02-19T21:08:48.459 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:08:48.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a342fb700 10 client.48460.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:48.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 13 2024-02-19T21:08:48.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a1001b340 osd=6 3 2024-02-19T21:08:48.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a1001b340 osd=6 3 2024-02-19T21:08:48.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a1001b340 osd=6 3 2024-02-19T21:08:48.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 13 2024-02-19T21:08:48.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a1001b340 osd=6 3 2024-02-19T21:08:48.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 6 13 2024-02-19T21:08:48.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a1001b340 osd=6 4 2024-02-19T21:08:48.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a342fb700 10 client.48460.objecter _send_command 13 2024-02-19T21:08:48.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.461+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a100270f0 con 0x7f5a10023520 2024-02-19T21:08:48.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.462+0000 7f5a237fe700 10 client.48460.objecter ms_handle_connect 0x7f5a10023520 2024-02-19T21:08:48.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.462+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5a100270f0 con 0x7f5a10023520 2024-02-19T21:08:48.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.462+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:08:48.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.462+0000 7f5a237fe700 10 client.48460.objecter _finish_command 13 = system:0 2024-02-19T21:08:48.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.462+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a1001b340 osd=6 4 2024-02-19T21:08:48.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.462+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 6 13 2024-02-19T21:08:48.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.477+0000 7f5a342fb700 10 client.48460.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:48.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.477+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 14 2024-02-19T21:08:48.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.477+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a1001b340 osd=6 3 2024-02-19T21:08:48.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.477+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a1001b340 osd=6 3 2024-02-19T21:08:48.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.477+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a1001b340 osd=6 3 2024-02-19T21:08:48.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.477+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 14 2024-02-19T21:08:48.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.477+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a1001b340 osd=6 3 2024-02-19T21:08:48.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.477+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 6 14 2024-02-19T21:08:48.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.477+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a1001b340 osd=6 4 2024-02-19T21:08:48.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.477+0000 7f5a342fb700 10 client.48460.objecter _send_command 14 2024-02-19T21:08:48.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.477+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5a100270f0 con 0x7f5a10023520 2024-02-19T21:08:48.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5a100270f0 con 0x7f5a10023520 2024-02-19T21:08:48.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:48.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a237fe700 10 client.48460.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:48.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a1001b340 osd=6 4 2024-02-19T21:08:48.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 6 14 2024-02-19T21:08:48.476 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:08:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a342fb700 10 client.48460.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 15 2024-02-19T21:08:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100270f0 osd=7 3 2024-02-19T21:08:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100270f0 osd=7 3 2024-02-19T21:08:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100270f0 osd=7 3 2024-02-19T21:08:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 15 2024-02-19T21:08:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a100270f0 osd=7 3 2024-02-19T21:08:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 7 15 2024-02-19T21:08:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100270f0 osd=7 4 2024-02-19T21:08:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a342fb700 10 client.48460.objecter _send_command 15 2024-02-19T21:08:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a10028310 con 0x7f5a10028db0 2024-02-19T21:08:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.479+0000 7f5a237fe700 10 client.48460.objecter ms_handle_connect 0x7f5a10028db0 2024-02-19T21:08:48.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.480+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5a10028310 con 0x7f5a10028db0 2024-02-19T21:08:48.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.480+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:08:48.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.480+0000 7f5a237fe700 10 client.48460.objecter _finish_command 15 = system:0 2024-02-19T21:08:48.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.480+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a100270f0 osd=7 4 2024-02-19T21:08:48.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.480+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 7 15 2024-02-19T21:08:48.482 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:48.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.495+0000 7f5a342fb700 10 client.48460.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:08:48.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.495+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign -1 16 2024-02-19T21:08:48.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.495+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100270f0 osd=7 3 2024-02-19T21:08:48.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.495+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100270f0 osd=7 3 2024-02-19T21:08:48.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.495+0000 7f5a342fb700 20 client.48460.objecter _get_session s=0x7f5a100270f0 osd=7 3 2024-02-19T21:08:48.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.495+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_remove -1 16 2024-02-19T21:08:48.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.495+0000 7f5a342fb700 20 client.48460.objecter get_session s=0x7f5a100270f0 osd=7 3 2024-02-19T21:08:48.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.495+0000 7f5a342fb700 15 client.48460.objecter _session_command_op_assign 7 16 2024-02-19T21:08:48.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.495+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100270f0 osd=7 4 2024-02-19T21:08:48.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.495+0000 7f5a342fb700 10 client.48460.objecter _send_command 16 2024-02-19T21:08:48.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.495+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5a10028310 con 0x7f5a10028db0 2024-02-19T21:08:48.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a237fe700 1 -- 172.21.15.163:0/3601168909 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5a10028310 con 0x7f5a10028db0 2024-02-19T21:08:48.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a237fe700 10 client.48460.objecter ms_dispatch 0x7f5a2c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:08:48.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a237fe700 10 client.48460.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:08:48.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a237fe700 20 client.48460.objecter put_session s=0x7f5a100270f0 osd=7 4 2024-02-19T21:08:48.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a237fe700 15 client.48460.objecter _session_command_op_remove 7 16 2024-02-19T21:08:48.494 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:08:48.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 10 librados: watch_flush enter 2024-02-19T21:08:48.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 10 librados: watch_flush exit 2024-02-19T21:08:48.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 10 client.48460.objecter close_session for osd.0 2024-02-19T21:08:48.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 >> v1:172.21.15.163:6801/1559418115 conn(0x7f5a10004530 legacy=0x7f5a10002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:48.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100051a0 osd=0 1 2024-02-19T21:08:48.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 10 client.48460.objecter close_session for osd.1 2024-02-19T21:08:48.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 >> v1:172.21.15.163:6805/3125047432 conn(0x7f5a10004150 legacy=0x7f5a1000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:48.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a10003fe0 osd=1 1 2024-02-19T21:08:48.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 10 client.48460.objecter close_session for osd.2 2024-02-19T21:08:48.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 >> v1:172.21.15.163:6809/1563158487 conn(0x7f5a10011740 legacy=0x7f5a10011b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:48.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a10003630 osd=2 1 2024-02-19T21:08:48.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 10 client.48460.objecter close_session for osd.3 2024-02-19T21:08:48.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 >> v1:172.21.15.163:6813/2112279769 conn(0x7f5a100162a0 legacy=0x7f5a100186d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:48.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100034b0 osd=3 1 2024-02-19T21:08:48.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 10 client.48460.objecter close_session for osd.4 2024-02-19T21:08:48.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 >> v1:172.21.15.193:6800/3129716965 conn(0x7f5a100112a0 legacy=0x7f5a1001a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:48.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a10012440 osd=4 1 2024-02-19T21:08:48.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a32898700 1 -- 172.21.15.163:0/3601168909 reap_dead start 2024-02-19T21:08:48.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 10 client.48460.objecter close_session for osd.5 2024-02-19T21:08:48.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 >> v1:172.21.15.193:6804/805161991 conn(0x7f5a1001ad30 legacy=0x7f5a100218f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:48.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a32898700 1 -- 172.21.15.163:0/3601168909 reap_dead start 2024-02-19T21:08:48.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a32898700 1 -- 172.21.15.163:0/3601168909 reap_dead start 2024-02-19T21:08:48.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100125f0 osd=5 1 2024-02-19T21:08:48.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 10 client.48460.objecter close_session for osd.6 2024-02-19T21:08:48.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 >> v1:172.21.15.193:6808/787604615 conn(0x7f5a10023520 legacy=0x7f5a10025910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:48.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a1001b340 osd=6 1 2024-02-19T21:08:48.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 10 client.48460.objecter close_session for osd.7 2024-02-19T21:08:48.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 >> v1:172.21.15.193:6812/157927432 conn(0x7f5a10028db0 legacy=0x7f5a10029230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:48.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 20 client.48460.objecter put_session s=0x7f5a100270f0 osd=7 1 2024-02-19T21:08:48.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 20 client.48460.objecter shutdown clearing up homeless session... 2024-02-19T21:08:48.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 10 client.48460.objecter successfully canceled tick 2024-02-19T21:08:48.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5a18063c00 legacy=0x7f5a180660c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:48.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 >> v1:172.21.15.193:6789/0 conn(0x7f5a2c100780 legacy=0x7f5a2c0fa0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:48.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a32898700 1 -- 172.21.15.163:0/3601168909 reap_dead start 2024-02-19T21:08:48.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 shutdown_connections 2024-02-19T21:08:48.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 >> 172.21.15.163:0/3601168909 conn(0x7f5a2c0f35a0 msgr2=0x7f5a2c107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:48.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.497+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 shutdown_connections 2024-02-19T21:08:48.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.498+0000 7f5a342fb700 1 -- 172.21.15.163:0/3601168909 wait complete. 2024-02-19T21:08:48.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:48.498+0000 7f5a342fb700 1 librados: shutdown 2024-02-19T21:08:48.498 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:08:48.498 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:08:48.498 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:08:48.498 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:08:48.498 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:08:48.499 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:08:48.499 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:08:48.499 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:08:49.035 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:08:49.315 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191.RoundTripAppendPP (3016 ms) 2024-02-19T21:08:49.315 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-02-19T21:08:49.315 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3004 ms) 2024-02-19T21:08:49.315 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-02-19T21:08:49.315 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2992 ms) 2024-02-19T21:08:49.315 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-02-19T21:08:49.315 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3001 ms) 2024-02-19T21:08:49.315 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-02-19T21:08:49.315 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (2998 ms) 2024-02-19T21:08:49.315 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [----------] 31 tests from LibRadosAio (119360 ms total) 2024-02-19T21:08:49.315 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: 2024-02-19T21:08:49.315 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-02-19T21:08:49.316 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-02-19T21:08:49.316 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3232 ms) 2024-02-19T21:08:49.316 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-02-19T21:08:49.316 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (8768 ms) 2024-02-19T21:08:49.316 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-02-19T21:08:49.316 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15004 ms) 2024-02-19T21:08:49.316 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-02-19T21:08:49.316 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3016 ms) 2024-02-19T21:08:49.316 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30020 ms total) 2024-02-19T21:08:49.316 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: 2024-02-19T21:08:49.316 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-02-19T21:08:49.316 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-02-19T21:08:49.316 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2989 ms) 2024-02-19T21:08:49.316 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2989 ms total) 2024-02-19T21:08:49.317 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: 2024-02-19T21:08:49.317 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-02-19T21:08:49.317 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-02-19T21:08:49.317 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14003 ms) 2024-02-19T21:08:49.317 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-02-19T21:08:49.317 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7004 ms) 2024-02-19T21:08:49.317 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-02-19T21:08:49.317 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7015 ms) 2024-02-19T21:08:49.317 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-02-19T21:08:49.317 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6988 ms) 2024-02-19T21:08:49.317 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-02-19T21:08:49.317 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3010 ms) 2024-02-19T21:08:49.317 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-02-19T21:08:49.317 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7007 ms) 2024-02-19T21:08:49.317 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-02-19T21:08:49.318 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6989 ms) 2024-02-19T21:08:49.318 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-02-19T21:08:49.318 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7021 ms) 2024-02-19T21:08:49.318 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-02-19T21:08:49.318 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6981 ms) 2024-02-19T21:08:49.318 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-02-19T21:08:49.318 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7009 ms) 2024-02-19T21:08:49.318 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-02-19T21:08:49.318 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6998 ms) 2024-02-19T21:08:49.318 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-02-19T21:08:49.318 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7025 ms) 2024-02-19T21:08:49.318 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-02-19T21:08:49.318 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6986 ms) 2024-02-19T21:08:49.318 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-02-19T21:08:49.319 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3007 ms) 2024-02-19T21:08:49.319 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-02-19T21:08:49.319 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7070 ms) 2024-02-19T21:08:49.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:47.902241+0000 osd.7 (osd.7) 145 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:08:49.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:47.903119+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:08:49.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:48.021368+0000 osd.2 (osd.2) 126 : cluster [DBG] 61.17 scrub starts 2024-02-19T21:08:49.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:48.023074+0000 osd.2 (osd.2) 127 : cluster [DBG] 61.17 scrub ok 2024-02-19T21:08:49.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:48.238474+0000 mon.a (mon.0) 2273 : audit [INF] from='client.48448 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi163-83854-60"}]': finished 2024-02-19T21:08:49.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:48.238567+0000 mon.a (mon.0) 2274 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:08:49.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:48.238615+0000 mon.a (mon.0) 2275 : cluster [DBG] osdmap e348: 8 total, 8 up, 7 in 2024-02-19T21:08:49.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:48.246695+0000 mon.b (mon.1) 521 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:49.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:48.246786+0000 mon.b (mon.1) 522 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:49.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:48.251197+0000 mon.a (mon.0) 2276 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:49.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:48.251387+0000 mon.a (mon.0) 2277 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:49.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:48.339397+0000 mon.b (mon.1) 523 : audit [DBG] from='client.? v1:172.21.15.163:0/3601168909' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:08:49.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:47.902241+0000 osd.7 (osd.7) 145 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:08:49.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:47.903119+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:08:49.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:48.021368+0000 osd.2 (osd.2) 126 : cluster [DBG] 61.17 scrub starts 2024-02-19T21:08:49.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:48.023074+0000 osd.2 (osd.2) 127 : cluster [DBG] 61.17 scrub ok 2024-02-19T21:08:49.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:48.238474+0000 mon.a (mon.0) 2273 : audit [INF] from='client.48448 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi163-83854-60"}]': finished 2024-02-19T21:08:49.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:48.238567+0000 mon.a (mon.0) 2274 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:08:49.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:48.238615+0000 mon.a (mon.0) 2275 : cluster [DBG] osdmap e348: 8 total, 8 up, 7 in 2024-02-19T21:08:49.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:48.246695+0000 mon.b (mon.1) 521 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:49.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:48.246786+0000 mon.b (mon.1) 522 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:49.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:48.251197+0000 mon.a (mon.0) 2276 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:49.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:48.251387+0000 mon.a (mon.0) 2277 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:49.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:48.339397+0000 mon.b (mon.1) 523 : audit [DBG] from='client.? v1:172.21.15.163:0/3601168909' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:08:49.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:47.902241+0000 osd.7 (osd.7) 145 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:08:49.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:47.903119+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:08:49.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:48.021368+0000 osd.2 (osd.2) 126 : cluster [DBG] 61.17 scrub starts 2024-02-19T21:08:49.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:48.023074+0000 osd.2 (osd.2) 127 : cluster [DBG] 61.17 scrub ok 2024-02-19T21:08:49.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:48.238474+0000 mon.a (mon.0) 2273 : audit [INF] from='client.48448 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi163-83854-60"}]': finished 2024-02-19T21:08:49.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:48.238567+0000 mon.a (mon.0) 2274 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:08:49.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:48.238615+0000 mon.a (mon.0) 2275 : cluster [DBG] osdmap e348: 8 total, 8 up, 7 in 2024-02-19T21:08:49.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:48.246695+0000 mon.b (mon.1) 521 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:49.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:48.246786+0000 mon.b (mon.1) 522 : audit [INF] from='client.? v1:172.21.15.163:0/264054107' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:49.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:48.251197+0000 mon.a (mon.0) 2276 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:49.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:48.251387+0000 mon.a (mon.0) 2277 : audit [INF] from='client.48448 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi163-83854-60"}]: dispatch 2024-02-19T21:08:49.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:48.339397+0000 mon.b (mon.1) 523 : audit [DBG] from='client.? v1:172.21.15.163:0/3601168909' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:08:50.060 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:08:50.063 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:50.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:08:50.164 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:50.195 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:50.216 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:50.218 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:50.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:50.319 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:50.346 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:48.759572+0000 osd.1 (osd.1) 157 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:08:50.346 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:48.760621+0000 osd.1 (osd.1) 158 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:08:50.346 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:48.995056+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:08:50.346 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:48.995823+0000 osd.2 (osd.2) 129 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:08:50.346 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:49.179768+0000 osd.6 (osd.6) 137 : cluster [DBG] 4.e scrub starts 2024-02-19T21:08:50.346 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:49.181095+0000 osd.6 (osd.6) 138 : cluster [DBG] 4.e scrub ok 2024-02-19T21:08:50.346 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:49.188917+0000 mgr.y (mgr.14550) 325 : cluster [DBG] pgmap v494: 228 pgs: 1 active+recovering+undersized+remapped, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9/1008 objects misplaced (0.893%) 2024-02-19T21:08:50.346 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:49.312008+0000 mon.a (mon.0) 2278 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45"}]': finished 2024-02-19T21:08:50.346 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:49.312086+0000 mon.a (mon.0) 2279 : audit [INF] from='client.48448 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi163-83854-60"}]': finished 2024-02-19T21:08:50.346 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:49.312130+0000 mon.a (mon.0) 2280 : cluster [DBG] osdmap e349: 8 total, 8 up, 7 in 2024-02-19T21:08:50.347 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:49.333853+0000 mon.b (mon.1) 524 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:50.347 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:49.334091+0000 mon.a (mon.0) 2281 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:50.347 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:49.334662+0000 mon.b (mon.1) 525 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:50.347 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:49.334854+0000 mon.a (mon.0) 2282 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:50.347 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:49.335073+0000 mon.c (mon.2) 350 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:50.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:48.759572+0000 osd.1 (osd.1) 157 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:08:50.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:48.760621+0000 osd.1 (osd.1) 158 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:08:50.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:48.995056+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:08:50.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:48.995823+0000 osd.2 (osd.2) 129 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:08:50.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:49.179768+0000 osd.6 (osd.6) 137 : cluster [DBG] 4.e scrub starts 2024-02-19T21:08:50.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:49.181095+0000 osd.6 (osd.6) 138 : cluster [DBG] 4.e scrub ok 2024-02-19T21:08:50.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:49.188917+0000 mgr.y (mgr.14550) 325 : cluster [DBG] pgmap v494: 228 pgs: 1 active+recovering+undersized+remapped, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9/1008 objects misplaced (0.893%) 2024-02-19T21:08:50.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:49.312008+0000 mon.a (mon.0) 2278 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45"}]': finished 2024-02-19T21:08:50.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:49.312086+0000 mon.a (mon.0) 2279 : audit [INF] from='client.48448 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi163-83854-60"}]': finished 2024-02-19T21:08:50.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:49.312130+0000 mon.a (mon.0) 2280 : cluster [DBG] osdmap e349: 8 total, 8 up, 7 in 2024-02-19T21:08:50.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:49.333853+0000 mon.b (mon.1) 524 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:50.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:49.334091+0000 mon.a (mon.0) 2281 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:50.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:49.334662+0000 mon.b (mon.1) 525 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:50.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:49.334854+0000 mon.a (mon.0) 2282 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:50.348 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:49.335073+0000 mon.c (mon.2) 350 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:50.348 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:49.335375+0000 mon.a (mon.0) 2283 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:50.349 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:49.336394+0000 mon.c (mon.2) 351 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:50.349 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:49.336607+0000 mon.a (mon.0) 2284 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:50.349 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:49.337019+0000 mon.c (mon.2) 352 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:50.349 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:49.337241+0000 mon.a (mon.0) 2285 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:50.349 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:49.609990+0000 osd.5 (osd.5) 138 : cluster [DBG] 61.c scrub starts 2024-02-19T21:08:50.349 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:49.868241+0000 osd.3 (osd.3) 147 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:08:50.349 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:49.869584+0000 osd.3 (osd.3) 148 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:08:50.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:49.335375+0000 mon.a (mon.0) 2283 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:50.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:49.336394+0000 mon.c (mon.2) 351 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:50.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:49.336607+0000 mon.a (mon.0) 2284 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:50.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:49.337019+0000 mon.c (mon.2) 352 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:50.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:49.337241+0000 mon.a (mon.0) 2285 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:50.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:49.609990+0000 osd.5 (osd.5) 138 : cluster [DBG] 61.c scrub starts 2024-02-19T21:08:50.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:49.868241+0000 osd.3 (osd.3) 147 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:08:50.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:49.869584+0000 osd.3 (osd.3) 148 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:08:50.350 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:50.366 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:50.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:50.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:50.470 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:50.494 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:08:50.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:48.759572+0000 osd.1 (osd.1) 157 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:08:50.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:48.760621+0000 osd.1 (osd.1) 158 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:08:50.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:48.995056+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:08:50.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:48.995823+0000 osd.2 (osd.2) 129 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:08:50.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:49.179768+0000 osd.6 (osd.6) 137 : cluster [DBG] 4.e scrub starts 2024-02-19T21:08:50.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:49.181095+0000 osd.6 (osd.6) 138 : cluster [DBG] 4.e scrub ok 2024-02-19T21:08:50.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:49.188917+0000 mgr.y (mgr.14550) 325 : cluster [DBG] pgmap v494: 228 pgs: 1 active+recovering+undersized+remapped, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 9/1008 objects misplaced (0.893%) 2024-02-19T21:08:50.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:49.312008+0000 mon.a (mon.0) 2278 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45"}]': finished 2024-02-19T21:08:50.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:49.312086+0000 mon.a (mon.0) 2279 : audit [INF] from='client.48448 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi163-83854-60"}]': finished 2024-02-19T21:08:50.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:49.312130+0000 mon.a (mon.0) 2280 : cluster [DBG] osdmap e349: 8 total, 8 up, 7 in 2024-02-19T21:08:50.496 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:49.333853+0000 mon.b (mon.1) 524 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:50.496 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:49.334091+0000 mon.a (mon.0) 2281 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:08:50.496 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:49.334662+0000 mon.b (mon.1) 525 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:50.496 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:49.334854+0000 mon.a (mon.0) 2282 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-45"}]: dispatch 2024-02-19T21:08:50.496 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:49.335073+0000 mon.c (mon.2) 350 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:50.496 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:49.335375+0000 mon.a (mon.0) 2283 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:50.496 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:49.336394+0000 mon.c (mon.2) 351 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:50.496 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:49.336607+0000 mon.a (mon.0) 2284 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:50.496 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:49.337019+0000 mon.c (mon.2) 352 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:50.496 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:49.337241+0000 mon.a (mon.0) 2285 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:50.496 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:49.609990+0000 osd.5 (osd.5) 138 : cluster [DBG] 61.c scrub starts 2024-02-19T21:08:50.496 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:49.868241+0000 osd.3 (osd.3) 147 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:08:50.496 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:49.869584+0000 osd.3 (osd.3) 148 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:08:50.497 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:50.515 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:50.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:50.618 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:50.644 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:50.663 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:50.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:50.766 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:50.793 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:50.811 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:50.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:50.916 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:08:50.937 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:50.946 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:51.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:49.613458+0000 osd.5 (osd.5) 139 : cluster [DBG] 61.c scrub ok 2024-02-19T21:08:51.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:49.750467+0000 osd.1 (osd.1) 159 : cluster [DBG] 61.5 scrub starts 2024-02-19T21:08:51.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:49.752413+0000 osd.1 (osd.1) 160 : cluster [DBG] 61.5 scrub ok 2024-02-19T21:08:51.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:49.964373+0000 osd.2 (osd.2) 130 : cluster [DBG] 61.12 scrub starts 2024-02-19T21:08:51.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:49.965985+0000 osd.2 (osd.2) 131 : cluster [DBG] 61.12 scrub ok 2024-02-19T21:08:51.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:50.310427+0000 mon.a (mon.0) 2286 : audit [INF] from='client.49649 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:51.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:50.310524+0000 mon.a (mon.0) 2287 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-02-19T21:08:51.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:50.313581+0000 mon.c (mon.2) 353 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi163-83854-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:51.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:50.313899+0000 mon.a (mon.0) 2288 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi163-83854-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:51.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:50.315615+0000 mgr.y (mgr.14550) 326 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:51.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:50.594340+0000 osd.5 (osd.5) 140 : cluster [DBG] 16.3 scrub starts 2024-02-19T21:08:51.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:50.600651+0000 osd.5 (osd.5) 141 : cluster [DBG] 16.3 scrub ok 2024-02-19T21:08:51.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:50.836032+0000 osd.3 (osd.3) 149 : cluster [DBG] 61.18 scrub starts 2024-02-19T21:08:51.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:50.838358+0000 osd.3 (osd.3) 150 : cluster [DBG] 61.18 scrub ok 2024-02-19T21:08:51.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:49.613458+0000 osd.5 (osd.5) 139 : cluster [DBG] 61.c scrub ok 2024-02-19T21:08:51.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:49.750467+0000 osd.1 (osd.1) 159 : cluster [DBG] 61.5 scrub starts 2024-02-19T21:08:51.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:49.752413+0000 osd.1 (osd.1) 160 : cluster [DBG] 61.5 scrub ok 2024-02-19T21:08:51.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:49.964373+0000 osd.2 (osd.2) 130 : cluster [DBG] 61.12 scrub starts 2024-02-19T21:08:51.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:49.965985+0000 osd.2 (osd.2) 131 : cluster [DBG] 61.12 scrub ok 2024-02-19T21:08:51.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:50.310427+0000 mon.a (mon.0) 2286 : audit [INF] from='client.49649 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:51.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:50.310524+0000 mon.a (mon.0) 2287 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-02-19T21:08:51.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:50.313581+0000 mon.c (mon.2) 353 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi163-83854-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:51.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:50.313899+0000 mon.a (mon.0) 2288 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi163-83854-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:51.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:50.315615+0000 mgr.y (mgr.14550) 326 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:51.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:50.594340+0000 osd.5 (osd.5) 140 : cluster [DBG] 16.3 scrub starts 2024-02-19T21:08:51.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:50.600651+0000 osd.5 (osd.5) 141 : cluster [DBG] 16.3 scrub ok 2024-02-19T21:08:51.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:50.836032+0000 osd.3 (osd.3) 149 : cluster [DBG] 61.18 scrub starts 2024-02-19T21:08:51.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:50.838358+0000 osd.3 (osd.3) 150 : cluster [DBG] 61.18 scrub ok 2024-02-19T21:08:51.645 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:49.613458+0000 osd.5 (osd.5) 139 : cluster [DBG] 61.c scrub ok 2024-02-19T21:08:51.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:49.750467+0000 osd.1 (osd.1) 159 : cluster [DBG] 61.5 scrub starts 2024-02-19T21:08:51.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:49.752413+0000 osd.1 (osd.1) 160 : cluster [DBG] 61.5 scrub ok 2024-02-19T21:08:51.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:49.964373+0000 osd.2 (osd.2) 130 : cluster [DBG] 61.12 scrub starts 2024-02-19T21:08:51.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:49.965985+0000 osd.2 (osd.2) 131 : cluster [DBG] 61.12 scrub ok 2024-02-19T21:08:51.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:50.310427+0000 mon.a (mon.0) 2286 : audit [INF] from='client.49649 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:51.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:50.310524+0000 mon.a (mon.0) 2287 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-02-19T21:08:51.647 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:50.313581+0000 mon.c (mon.2) 353 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi163-83854-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:51.647 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:50.313899+0000 mon.a (mon.0) 2288 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi163-83854-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:51.647 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:50.315615+0000 mgr.y (mgr.14550) 326 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:08:51.647 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:50.594340+0000 osd.5 (osd.5) 140 : cluster [DBG] 16.3 scrub starts 2024-02-19T21:08:51.647 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:50.600651+0000 osd.5 (osd.5) 141 : cluster [DBG] 16.3 scrub ok 2024-02-19T21:08:51.647 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:50.836032+0000 osd.3 (osd.3) 149 : cluster [DBG] 61.18 scrub starts 2024-02-19T21:08:51.648 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:50.838358+0000 osd.3 (osd.3) 150 : cluster [DBG] 61.18 scrub ok 2024-02-19T21:08:51.846 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:08:52.200 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:08:52.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:52.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:08:52.217 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:08:52.217 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:08:52.217 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-02-19T21:08:52.217 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:08:52.308 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:08:52.358 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:52.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:51.189329+0000 mgr.y (mgr.14550) 327 : cluster [DBG] pgmap v497: 196 pgs: 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 23 B/s, 5 objects/s recovering 2024-02-19T21:08:52.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:51.553354+0000 mon.a (mon.0) 2289 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-02-19T21:08:52.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:51.554424+0000 mon.b (mon.1) 526 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:52.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:51.555237+0000 mon.a (mon.0) 2290 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:52.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:51.189329+0000 mgr.y (mgr.14550) 327 : cluster [DBG] pgmap v497: 196 pgs: 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 23 B/s, 5 objects/s recovering 2024-02-19T21:08:52.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:51.553354+0000 mon.a (mon.0) 2289 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-02-19T21:08:52.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:51.554424+0000 mon.b (mon.1) 526 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:52.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:51.555237+0000 mon.a (mon.0) 2290 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:52.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:51.189329+0000 mgr.y (mgr.14550) 327 : cluster [DBG] pgmap v497: 196 pgs: 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 23 B/s, 5 objects/s recovering 2024-02-19T21:08:52.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:51.553354+0000 mon.a (mon.0) 2289 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-02-19T21:08:52.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:51.554424+0000 mon.b (mon.1) 526 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:52.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:51.555237+0000 mon.a (mon.0) 2290 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:08:53.507 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:08:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:53.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:08:53.612 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:08:53.642 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:51.612486+0000 osd.5 (osd.5) 142 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:08:53.642 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:51.616118+0000 osd.5 (osd.5) 143 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:08:53.642 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:51.952844+0000 osd.7 (osd.7) 147 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:08:53.642 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:51.955734+0000 osd.7 (osd.7) 148 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:08:53.642 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:52.200062+0000 osd.6 (osd.6) 139 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:08:53.642 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:52.201968+0000 osd.6 (osd.6) 140 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:08:53.642 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:52.322470+0000 mon.a (mon.0) 2291 : audit [INF] from='client.49649 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi163-83854-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi163-83854-61"}]': finished 2024-02-19T21:08:53.642 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:52.322646+0000 mon.a (mon.0) 2292 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:53.642 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:52.322700+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-02-19T21:08:53.642 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:52.345659+0000 mon.b (mon.1) 527 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:53.642 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:52.346012+0000 mon.a (mon.0) 2294 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:53.642 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:52.576190+0000 osd.5 (osd.5) 144 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:08:53.643 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:52.579173+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:08:53.644 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:53.665 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:08:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:53.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:08:53.744 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:08:53] "GET /metrics HTTP/1.1" 200 188870 "" "Prometheus/2.33.4" 2024-02-19T21:08:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:51.612486+0000 osd.5 (osd.5) 142 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:08:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:51.616118+0000 osd.5 (osd.5) 143 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:08:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:51.952844+0000 osd.7 (osd.7) 147 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:08:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:51.955734+0000 osd.7 (osd.7) 148 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:08:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:52.200062+0000 osd.6 (osd.6) 139 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:08:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:52.201968+0000 osd.6 (osd.6) 140 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:08:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:52.322470+0000 mon.a (mon.0) 2291 : audit [INF] from='client.49649 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi163-83854-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi163-83854-61"}]': finished 2024-02-19T21:08:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:52.322646+0000 mon.a (mon.0) 2292 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:53.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:52.322700+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-02-19T21:08:53.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:52.345659+0000 mon.b (mon.1) 527 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:53.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:52.346012+0000 mon.a (mon.0) 2294 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:53.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:52.576190+0000 osd.5 (osd.5) 144 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:08:53.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:52.579173+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:08:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:51.612486+0000 osd.5 (osd.5) 142 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:08:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:51.616118+0000 osd.5 (osd.5) 143 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:08:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:51.952844+0000 osd.7 (osd.7) 147 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:08:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:51.955734+0000 osd.7 (osd.7) 148 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:08:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:52.200062+0000 osd.6 (osd.6) 139 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:08:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:52.201968+0000 osd.6 (osd.6) 140 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:08:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:52.322470+0000 mon.a (mon.0) 2291 : audit [INF] from='client.49649 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi163-83854-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi163-83854-61"}]': finished 2024-02-19T21:08:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:52.322646+0000 mon.a (mon.0) 2292 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:08:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:52.322700+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-02-19T21:08:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:52.345659+0000 mon.b (mon.1) 527 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:52.346012+0000 mon.a (mon.0) 2294 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:08:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:52.576190+0000 osd.5 (osd.5) 144 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:08:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:52.579173+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:08:53.768 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:08:53.799 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:53.820 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:08:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:53.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:08:53.924 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:08:53.954 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:54.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.965+0000 7ff558427700 1 -- 172.21.15.163:0/2510766299 >> v1:172.21.15.163:6789/0 conn(0x7ff5501031d0 legacy=0x7ff550105650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:54.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.966+0000 7ff558427700 1 -- 172.21.15.163:0/2510766299 shutdown_connections 2024-02-19T21:08:54.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.966+0000 7ff558427700 1 -- 172.21.15.163:0/2510766299 >> 172.21.15.163:0/2510766299 conn(0x7ff5500f2470 msgr2=0x7ff5500f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:54.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.966+0000 7ff558427700 1 -- 172.21.15.163:0/2510766299 shutdown_connections 2024-02-19T21:08:54.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.966+0000 7ff558427700 1 -- 172.21.15.163:0/2510766299 wait complete. 2024-02-19T21:08:54.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.966+0000 7ff558427700 1 librados: starting msgr at 2024-02-19T21:08:54.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.966+0000 7ff558427700 1 librados: starting objecter 2024-02-19T21:08:54.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff558427700 1 Processor -- start 2024-02-19T21:08:54.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff558427700 1 -- start start 2024-02-19T21:08:54.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff558427700 1 librados: setting wanted keys 2024-02-19T21:08:54.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff558427700 1 librados: calling monclient init 2024-02-19T21:08:54.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff558427700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff550117240 con 0x7ff5501031d0 2024-02-19T21:08:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff558427700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff5501183e0 con 0x7ff5500fc890 2024-02-19T21:08:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff558427700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff550119580 con 0x7ff5500ff690 2024-02-19T21:08:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff5569c4700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ff5501031d0 0x7ff550115b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43198/0 (socket says 172.21.15.163:43198) 2024-02-19T21:08:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff5569c4700 1 -- 172.21.15.163:0/1720382424 learned_addr learned my addr 172.21.15.163:0/1720382424 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff5559c2700 1 --1- 172.21.15.163:0/1720382424 >> v1:172.21.15.163:6790/0 conn(0x7ff5500ff690 0x7ff5500f9b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:56674/0 (socket says 172.21.15.163:56674) 2024-02-19T21:08:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff5477fe700 10 client.?.objecter ms_handle_connect 0x7ff5500ff690 2024-02-19T21:08:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff5477fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff5477fe700 10 client.?.objecter ms_handle_connect 0x7ff5501031d0 2024-02-19T21:08:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff5477fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729006299 0 0) 0x7ff550119580 con 0x7ff5500ff690 2024-02-19T21:08:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff53c0033f0 con 0x7ff5500ff690 2024-02-19T21:08:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff5477fe700 10 client.?.objecter ms_handle_connect 0x7ff5500fc890 2024-02-19T21:08:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff5477fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273656528 0 0) 0x7ff550117240 con 0x7ff5501031d0 2024-02-19T21:08:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff550119580 con 0x7ff5501031d0 2024-02-19T21:08:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731082617 0 0) 0x7ff5501183e0 con 0x7ff5500fc890 2024-02-19T21:08:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.967+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff550117240 con 0x7ff5500fc890 2024-02-19T21:08:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3322089483 0 0) 0x7ff53c0033f0 con 0x7ff5500ff690 2024-02-19T21:08:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5501183e0 con 0x7ff5500ff690 2024-02-19T21:08:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2774690562 0 0) 0x7ff550119580 con 0x7ff5501031d0 2024-02-19T21:08:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff53c0033f0 con 0x7ff5501031d0 2024-02-19T21:08:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff5400019d0 con 0x7ff5500ff690 2024-02-19T21:08:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff548001360 con 0x7ff5501031d0 2024-02-19T21:08:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4287251714 0 0) 0x7ff550117240 con 0x7ff5500fc890 2024-02-19T21:08:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff550119580 con 0x7ff5500fc890 2024-02-19T21:08:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff54c001910 con 0x7ff5500fc890 2024-02-19T21:08:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3184581370 0 0) 0x7ff5501183e0 con 0x7ff5500ff690 2024-02-19T21:08:54.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 >> v1:172.21.15.193:6789/0 conn(0x7ff5500fc890 legacy=0x7ff5500f9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:54.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 >> v1:172.21.15.163:6789/0 conn(0x7ff5501031d0 legacy=0x7ff550115b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:54.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55011a760 con 0x7ff5500ff690 2024-02-19T21:08:54.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff558427700 1 -- 172.21.15.163:0/1720382424 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff550117430 con 0x7ff5500ff690 2024-02-19T21:08:54.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff558427700 10 client.49655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:54.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff558427700 1 -- 172.21.15.163:0/1720382424 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5501178a0 con 0x7ff5500ff690 2024-02-19T21:08:54.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff558427700 1 librados: init done 2024-02-19T21:08:54.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff540002420 con 0x7ff5500ff690 2024-02-19T21:08:54.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.968+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff5400059e0 con 0x7ff5500ff690 2024-02-19T21:08:54.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.969+0000 7ff558427700 1 -- 172.21.15.163:0/1720382424 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff534004f20 con 0x7ff5500ff690 2024-02-19T21:08:54.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.969+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ff5400188f0 con 0x7ff5500ff690 2024-02-19T21:08:54.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.970+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(353..353 src has 43..353) v4 ==== 8726+0+0 (unknown 216185966 0 0) 0x7ff540078fe0 con 0x7ff5500ff690 2024-02-19T21:08:54.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.970+0000 7ff5477fe700 10 client.49655.objecter ms_dispatch 0x7ff550001550 osd_map(353..353 src has 43..353) v4 2024-02-19T21:08:54.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.970+0000 7ff5477fe700 3 client.49655.objecter handle_osd_map got epochs [353,353] > 0 2024-02-19T21:08:54.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.970+0000 7ff5477fe700 3 client.49655.objecter handle_osd_map decoding full epoch 353 2024-02-19T21:08:54.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.970+0000 7ff5477fe700 20 client.49655.objecter dump_active .. 0 homeless 2024-02-19T21:08:54.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.970+0000 7ff5477fe700 10 client.49655.objecter ms_handle_connect 0x7ff53c063c60 2024-02-19T21:08:54.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:53.975+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff540002910 con 0x7ff5500ff690 2024-02-19T21:08:54.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:54.156+0000 7ff558427700 1 -- 172.21.15.163:0/1720382424 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7ff5340051d0 con 0x7ff5500ff690 2024-02-19T21:08:54.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:54.234+0000 7ff5477fe700 1 -- 172.21.15.163:0/1720382424 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v354) v1 ==== 113+0+0 (unknown 3506799221 0 0) 0x7ff54004be80 con 0x7ff5500ff690 2024-02-19T21:08:54.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:54.239+0000 7ff558427700 10 librados: watch_flush enter 2024-02-19T21:08:54.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:54.239+0000 7ff558427700 10 librados: watch_flush exit 2024-02-19T21:08:54.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:54.239+0000 7ff558427700 20 client.49655.objecter shutdown clearing up homeless session... 2024-02-19T21:08:54.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:54.239+0000 7ff558427700 10 client.49655.objecter successfully canceled tick 2024-02-19T21:08:54.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:54.239+0000 7ff558427700 1 -- 172.21.15.163:0/1720382424 >> v1:172.21.15.163:6800/1552514681 conn(0x7ff53c063c60 legacy=0x7ff53c066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:54.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:54.239+0000 7ff558427700 1 -- 172.21.15.163:0/1720382424 >> v1:172.21.15.163:6790/0 conn(0x7ff5500ff690 legacy=0x7ff5500f9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:54.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:54.239+0000 7ff558427700 1 -- 172.21.15.163:0/1720382424 shutdown_connections 2024-02-19T21:08:54.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:54.239+0000 7ff558427700 1 -- 172.21.15.163:0/1720382424 >> 172.21.15.163:0/1720382424 conn(0x7ff5500f2470 msgr2=0x7ff5500f4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:54.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:54.239+0000 7ff558427700 1 -- 172.21.15.163:0/1720382424 shutdown_connections 2024-02-19T21:08:54.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:54.239+0000 7ff558427700 1 -- 172.21.15.163:0/1720382424 wait complete. 2024-02-19T21:08:54.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:54.239+0000 7ff558427700 1 librados: shutdown 2024-02-19T21:08:54.238 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.0 to 1 (10000) 2024-02-19T21:08:54.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:53.189761+0000 mgr.y (mgr.14550) 328 : cluster [DBG] pgmap v500: 236 pgs: 40 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 23 B/s, 5 objects/s recovering 2024-02-19T21:08:54.576 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:53.331019+0000 mon.a (mon.0) 2295 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:08:54.576 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:53.331090+0000 mon.a (mon.0) 2296 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-02-19T21:08:54.576 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:53.332296+0000 mon.b (mon.1) 528 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:08:54.576 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:53.334296+0000 mon.a (mon.0) 2297 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:08:54.576 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:53.819107+0000 osd.3 (osd.3) 151 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:08:54.576 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:53.819945+0000 osd.3 (osd.3) 152 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:08:54.576 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:54.101021+0000 mon.a (mon.0) 2298 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:54.576 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:54.157774+0000 mon.c (mon.2) 354 : audit [INF] from='client.? v1:172.21.15.163:0/1720382424' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:08:54.576 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:54.158067+0000 mon.a (mon.0) 2299 : audit [INF] from='client.49655 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:08:54.576 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:54.235247+0000 mon.a (mon.0) 2300 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-47"}]': finished 2024-02-19T21:08:54.576 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:54.235373+0000 mon.a (mon.0) 2301 : audit [INF] from='client.49655 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-19T21:08:54.577 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:54.235431+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-02-19T21:08:54.577 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:54.236875+0000 mon.c (mon.2) 355 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:54.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:53.189761+0000 mgr.y (mgr.14550) 328 : cluster [DBG] pgmap v500: 236 pgs: 40 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 23 B/s, 5 objects/s recovering 2024-02-19T21:08:54.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:53.331019+0000 mon.a (mon.0) 2295 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:08:54.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:53.331090+0000 mon.a (mon.0) 2296 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-02-19T21:08:54.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:53.332296+0000 mon.b (mon.1) 528 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:08:54.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:53.334296+0000 mon.a (mon.0) 2297 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:08:54.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:53.819107+0000 osd.3 (osd.3) 151 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:08:54.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:53.819945+0000 osd.3 (osd.3) 152 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:08:54.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:54.101021+0000 mon.a (mon.0) 2298 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:54.577 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:54.157774+0000 mon.c (mon.2) 354 : audit [INF] from='client.? v1:172.21.15.163:0/1720382424' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:08:54.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:54.158067+0000 mon.a (mon.0) 2299 : audit [INF] from='client.49655 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:08:54.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:54.235247+0000 mon.a (mon.0) 2300 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-47"}]': finished 2024-02-19T21:08:54.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:54.235373+0000 mon.a (mon.0) 2301 : audit [INF] from='client.49655 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-19T21:08:54.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:54.235431+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-02-19T21:08:54.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:54.236875+0000 mon.c (mon.2) 355 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:54.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:54.239673+0000 mon.b (mon.1) 529 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-47", "mode": "readproxy"}]: dispatch 2024-02-19T21:08:54.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:54.239804+0000 mon.a (mon.0) 2303 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:54.578 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:54.240170+0000 mon.a (mon.0) 2304 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-47", "mode": "readproxy"}]: dispatch 2024-02-19T21:08:54.578 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:54.239673+0000 mon.b (mon.1) 529 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-47", "mode": "readproxy"}]: dispatch 2024-02-19T21:08:54.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:54.239804+0000 mon.a (mon.0) 2303 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:54.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:54.240170+0000 mon.a (mon.0) 2304 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-47", "mode": "readproxy"}]: dispatch 2024-02-19T21:08:54.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:53.189761+0000 mgr.y (mgr.14550) 328 : cluster [DBG] pgmap v500: 236 pgs: 40 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 23 B/s, 5 objects/s recovering 2024-02-19T21:08:54.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:53.331019+0000 mon.a (mon.0) 2295 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:08:54.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:53.331090+0000 mon.a (mon.0) 2296 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-02-19T21:08:54.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:53.332296+0000 mon.b (mon.1) 528 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:08:54.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:53.334296+0000 mon.a (mon.0) 2297 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:08:54.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:53.819107+0000 osd.3 (osd.3) 151 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:08:54.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:53.819945+0000 osd.3 (osd.3) 152 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:08:54.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:54.101021+0000 mon.a (mon.0) 2298 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:08:54.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:54.157774+0000 mon.c (mon.2) 354 : audit [INF] from='client.? v1:172.21.15.163:0/1720382424' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:08:54.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:54.158067+0000 mon.a (mon.0) 2299 : audit [INF] from='client.49655 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:08:54.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:54.235247+0000 mon.a (mon.0) 2300 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-47"}]': finished 2024-02-19T21:08:54.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:54.235373+0000 mon.a (mon.0) 2301 : audit [INF] from='client.49655 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-19T21:08:54.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:54.235431+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-02-19T21:08:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:54.236875+0000 mon.c (mon.2) 355 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:54.239673+0000 mon.b (mon.1) 529 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-47", "mode": "readproxy"}]: dispatch 2024-02-19T21:08:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:54.239804+0000 mon.a (mon.0) 2303 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:54.240170+0000 mon.a (mon.0) 2304 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-47", "mode": "readproxy"}]: dispatch 2024-02-19T21:08:54.991 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 1 1 2024-02-19T21:08:55.032 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:08:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:55.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:08:55.137 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:08:55.169 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:55.191 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:08:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:55.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:08:55.294 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:08:55.321 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:55.342 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:08:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:55.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:08:55.446 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:08:55.475 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:55.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:53.779810+0000 osd.1 (osd.1) 161 : cluster [DBG] 4.b scrub starts 2024-02-19T21:08:55.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:53.783287+0000 osd.1 (osd.1) 162 : cluster [DBG] 4.b scrub ok 2024-02-19T21:08:55.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:53.961739+0000 osd.7 (osd.7) 149 : cluster [DBG] 6.8 deep-scrub starts 2024-02-19T21:08:55.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:53.964695+0000 osd.7 (osd.7) 150 : cluster [DBG] 6.8 deep-scrub ok 2024-02-19T21:08:55.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:55.232134+0000 mon.a (mon.0) 2305 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:08:55.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:55.236053+0000 mon.a (mon.0) 2306 : audit [INF] from='client.49649 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61"}]': finished 2024-02-19T21:08:55.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:55.236158+0000 mon.a (mon.0) 2307 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-47", "mode": "readproxy"}]': finished 2024-02-19T21:08:55.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:55.236217+0000 mon.a (mon.0) 2308 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-02-19T21:08:55.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:55.241002+0000 mon.c (mon.2) 356 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:55.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:55.241873+0000 mon.a (mon.0) 2309 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:55.744 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:53.779810+0000 osd.1 (osd.1) 161 : cluster [DBG] 4.b scrub starts 2024-02-19T21:08:55.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:53.783287+0000 osd.1 (osd.1) 162 : cluster [DBG] 4.b scrub ok 2024-02-19T21:08:55.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:53.961739+0000 osd.7 (osd.7) 149 : cluster [DBG] 6.8 deep-scrub starts 2024-02-19T21:08:55.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:53.964695+0000 osd.7 (osd.7) 150 : cluster [DBG] 6.8 deep-scrub ok 2024-02-19T21:08:55.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:55.232134+0000 mon.a (mon.0) 2305 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:08:55.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:55.236053+0000 mon.a (mon.0) 2306 : audit [INF] from='client.49649 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61"}]': finished 2024-02-19T21:08:55.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:55.236158+0000 mon.a (mon.0) 2307 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-47", "mode": "readproxy"}]': finished 2024-02-19T21:08:55.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:55.236217+0000 mon.a (mon.0) 2308 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-02-19T21:08:55.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:55.241002+0000 mon.c (mon.2) 356 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:55.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:55.241873+0000 mon.a (mon.0) 2309 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:55.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:53.779810+0000 osd.1 (osd.1) 161 : cluster [DBG] 4.b scrub starts 2024-02-19T21:08:55.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:53.783287+0000 osd.1 (osd.1) 162 : cluster [DBG] 4.b scrub ok 2024-02-19T21:08:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:53.961739+0000 osd.7 (osd.7) 149 : cluster [DBG] 6.8 deep-scrub starts 2024-02-19T21:08:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:53.964695+0000 osd.7 (osd.7) 150 : cluster [DBG] 6.8 deep-scrub ok 2024-02-19T21:08:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:55.232134+0000 mon.a (mon.0) 2305 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:08:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:55.236053+0000 mon.a (mon.0) 2306 : audit [INF] from='client.49649 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi163-83854-61"}]': finished 2024-02-19T21:08:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:55.236158+0000 mon.a (mon.0) 2307 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-47", "mode": "readproxy"}]': finished 2024-02-19T21:08:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:55.236217+0000 mon.a (mon.0) 2308 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-02-19T21:08:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:55.241002+0000 mon.c (mon.2) 356 : audit [INF] from='client.? v1:172.21.15.163:0/2569144910' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:55.241873+0000 mon.a (mon.0) 2309 : audit [INF] from='client.49649 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi163-83854-61"}]: dispatch 2024-02-19T21:08:55.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-19T21:08:55.800 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:55.816 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:56.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:55.190901+0000 mgr.y (mgr.14550) 329 : cluster [DBG] pgmap v503: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:08:56.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:55.199536+0000 osd.6 (osd.6) 141 : cluster [DBG] 16.0 deep-scrub starts 2024-02-19T21:08:56.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:55.202458+0000 osd.6 (osd.6) 142 : cluster [DBG] 16.0 deep-scrub ok 2024-02-19T21:08:56.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:55.878027+0000 osd.3 (osd.3) 153 : cluster [DBG] 61.2 deep-scrub starts 2024-02-19T21:08:56.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:55.882981+0000 osd.3 (osd.3) 154 : cluster [DBG] 61.2 deep-scrub ok 2024-02-19T21:08:56.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:56.237978+0000 mon.a (mon.0) 2310 : audit [INF] from='client.49649 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi163-83854-61"}]': finished 2024-02-19T21:08:56.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:56.238038+0000 mon.a (mon.0) 2311 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-02-19T21:08:56.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:56.257647+0000 mon.c (mon.2) 357 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:56.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:56.257954+0000 mon.a (mon.0) 2312 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:56.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:56.258761+0000 mon.c (mon.2) 358 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:56.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:56.259034+0000 mon.a (mon.0) 2313 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:56.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:56.259502+0000 mon.c (mon.2) 359 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi163-83854-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:56.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:56.259762+0000 mon.a (mon.0) 2314 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi163-83854-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:56.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:55.190901+0000 mgr.y (mgr.14550) 329 : cluster [DBG] pgmap v503: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:08:56.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:55.199536+0000 osd.6 (osd.6) 141 : cluster [DBG] 16.0 deep-scrub starts 2024-02-19T21:08:56.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:55.202458+0000 osd.6 (osd.6) 142 : cluster [DBG] 16.0 deep-scrub ok 2024-02-19T21:08:56.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:55.878027+0000 osd.3 (osd.3) 153 : cluster [DBG] 61.2 deep-scrub starts 2024-02-19T21:08:56.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:55.882981+0000 osd.3 (osd.3) 154 : cluster [DBG] 61.2 deep-scrub ok 2024-02-19T21:08:56.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:56.237978+0000 mon.a (mon.0) 2310 : audit [INF] from='client.49649 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi163-83854-61"}]': finished 2024-02-19T21:08:56.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:56.238038+0000 mon.a (mon.0) 2311 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-02-19T21:08:56.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:56.257647+0000 mon.c (mon.2) 357 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:56.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:56.257954+0000 mon.a (mon.0) 2312 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:56.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:56.258761+0000 mon.c (mon.2) 358 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:56.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:56.259034+0000 mon.a (mon.0) 2313 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:56.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:56.259502+0000 mon.c (mon.2) 359 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi163-83854-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:56.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:56.259762+0000 mon.a (mon.0) 2314 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi163-83854-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:56.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:55.190901+0000 mgr.y (mgr.14550) 329 : cluster [DBG] pgmap v503: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:08:56.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:55.199536+0000 osd.6 (osd.6) 141 : cluster [DBG] 16.0 deep-scrub starts 2024-02-19T21:08:56.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:55.202458+0000 osd.6 (osd.6) 142 : cluster [DBG] 16.0 deep-scrub ok 2024-02-19T21:08:56.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:55.878027+0000 osd.3 (osd.3) 153 : cluster [DBG] 61.2 deep-scrub starts 2024-02-19T21:08:56.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:55.882981+0000 osd.3 (osd.3) 154 : cluster [DBG] 61.2 deep-scrub ok 2024-02-19T21:08:56.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:56.237978+0000 mon.a (mon.0) 2310 : audit [INF] from='client.49649 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi163-83854-61"}]': finished 2024-02-19T21:08:56.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:56.238038+0000 mon.a (mon.0) 2311 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-02-19T21:08:56.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:56.257647+0000 mon.c (mon.2) 357 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:56.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:56.257954+0000 mon.a (mon.0) 2312 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:56.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:56.258761+0000 mon.c (mon.2) 358 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:56.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:56.259034+0000 mon.a (mon.0) 2313 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:56.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:56.259502+0000 mon.c (mon.2) 359 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi163-83854-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:56.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:56.259762+0000 mon.a (mon.0) 2314 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi163-83854-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:08:57.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.598+0000 7fe4b1798700 1 -- 172.21.15.163:0/2519601181 >> v1:172.21.15.163:6789/0 conn(0x7fe4ac0fc890 legacy=0x7fe4ac0fcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:57.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.598+0000 7fe4b1798700 1 -- 172.21.15.163:0/2519601181 shutdown_connections 2024-02-19T21:08:57.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.599+0000 7fe4b1798700 1 -- 172.21.15.163:0/2519601181 >> 172.21.15.163:0/2519601181 conn(0x7fe4ac0f2470 msgr2=0x7fe4ac0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:57.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.599+0000 7fe4b1798700 1 -- 172.21.15.163:0/2519601181 shutdown_connections 2024-02-19T21:08:57.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.599+0000 7fe4b1798700 1 -- 172.21.15.163:0/2519601181 wait complete. 2024-02-19T21:08:57.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.599+0000 7fe4b1798700 1 librados: starting msgr at 2024-02-19T21:08:57.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.599+0000 7fe4b1798700 1 librados: starting objecter 2024-02-19T21:08:57.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.599+0000 7fe4b1798700 1 Processor -- start 2024-02-19T21:08:57.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.599+0000 7fe4b1798700 1 -- start start 2024-02-19T21:08:57.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.599+0000 7fe4b1798700 1 librados: setting wanted keys 2024-02-19T21:08:57.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.599+0000 7fe4b1798700 1 librados: calling monclient init 2024-02-19T21:08:57.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe4b1798700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe4ac119370 con 0x7fe4ac0ff690 2024-02-19T21:08:57.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe4b1798700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe4ac11a510 con 0x7fe4ac1031d0 2024-02-19T21:08:57.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe4aaffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fe4ac0fc890 0x7fe4ac0fb540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:35490/0 (socket says 172.21.15.163:35490) 2024-02-19T21:08:57.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe4aaffd700 1 -- 172.21.15.163:0/177824702 learned_addr learned my addr 172.21.15.163:0/177824702 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:57.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe4b1798700 1 -- 172.21.15.163:0/177824702 --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe4ac11b6f0 con 0x7fe4ac0fc890 2024-02-19T21:08:57.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 10 client.?.objecter ms_handle_connect 0x7fe4ac0ff690 2024-02-19T21:08:57.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:57.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 10 client.?.objecter ms_handle_connect 0x7fe4ac0fc890 2024-02-19T21:08:57.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:57.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 10 client.?.objecter ms_handle_connect 0x7fe4ac1031d0 2024-02-19T21:08:57.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:57.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648239424 0 0) 0x7fe4ac11b6f0 con 0x7fe4ac0fc890 2024-02-19T21:08:57.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe494003030 con 0x7fe4ac0fc890 2024-02-19T21:08:57.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3674797393 0 0) 0x7fe4ac119370 con 0x7fe4ac0ff690 2024-02-19T21:08:57.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4ac11b6f0 con 0x7fe4ac0ff690 2024-02-19T21:08:57.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708419120 0 0) 0x7fe4ac11a510 con 0x7fe4ac1031d0 2024-02-19T21:08:57.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4ac119370 con 0x7fe4ac1031d0 2024-02-19T21:08:57.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1251046215 0 0) 0x7fe494003030 con 0x7fe4ac0fc890 2024-02-19T21:08:57.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4ac11a510 con 0x7fe4ac0fc890 2024-02-19T21:08:57.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2194059387 0 0) 0x7fe4ac11b6f0 con 0x7fe4ac0ff690 2024-02-19T21:08:57.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe494003030 con 0x7fe4ac0ff690 2024-02-19T21:08:57.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe49c002b50 con 0x7fe4ac0fc890 2024-02-19T21:08:57.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe4a0001800 con 0x7fe4ac0ff690 2024-02-19T21:08:57.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2008847732 0 0) 0x7fe4ac119370 con 0x7fe4ac1031d0 2024-02-19T21:08:57.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4ac11b6f0 con 0x7fe4ac1031d0 2024-02-19T21:08:57.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe4a4001b40 con 0x7fe4ac1031d0 2024-02-19T21:08:57.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1524048067 0 0) 0x7fe4ac11a510 con 0x7fe4ac0fc890 2024-02-19T21:08:57.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 >> v1:172.21.15.193:6789/0 conn(0x7fe4ac1031d0 legacy=0x7fe4ac117c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:57.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 >> v1:172.21.15.163:6789/0 conn(0x7fe4ac0ff690 legacy=0x7fe4ac0fbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:57.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ac11c8f0 con 0x7fe4ac0fc890 2024-02-19T21:08:57.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe4b1798700 1 -- 172.21.15.163:0/177824702 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ac11b8e0 con 0x7fe4ac0fc890 2024-02-19T21:08:57.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe4b1798700 10 client.49667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:57.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe4b1798700 1 -- 172.21.15.163:0/177824702 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ac11bdd0 con 0x7fe4ac0fc890 2024-02-19T21:08:57.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.600+0000 7fe4b1798700 1 librados: init done 2024-02-19T21:08:57.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.601+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fe49c001aa0 con 0x7fe4ac0fc890 2024-02-19T21:08:57.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.601+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe49c003960 con 0x7fe4ac0fc890 2024-02-19T21:08:57.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.601+0000 7fe4b1798700 1 -- 172.21.15.163:0/177824702 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe498004f20 con 0x7fe4ac0fc890 2024-02-19T21:08:57.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.602+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fe49c0021c0 con 0x7fe4ac0fc890 2024-02-19T21:08:57.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.602+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(357..357 src has 43..357) v4 ==== 8198+0+0 (unknown 2725590093 0 0) 0x7fe49c078ff0 con 0x7fe4ac0fc890 2024-02-19T21:08:57.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.602+0000 7fe493fff700 10 client.49667.objecter ms_dispatch 0x7fe4ac001550 osd_map(357..357 src has 43..357) v4 2024-02-19T21:08:57.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.602+0000 7fe493fff700 3 client.49667.objecter handle_osd_map got epochs [357,357] > 0 2024-02-19T21:08:57.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.602+0000 7fe493fff700 3 client.49667.objecter handle_osd_map decoding full epoch 357 2024-02-19T21:08:57.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.602+0000 7fe493fff700 20 client.49667.objecter dump_active .. 0 homeless 2024-02-19T21:08:57.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.602+0000 7fe493fff700 10 client.49667.objecter ms_handle_connect 0x7fe494063c60 2024-02-19T21:08:57.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.606+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe49c04c440 con 0x7fe4ac0fc890 2024-02-19T21:08:57.707 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:55.688799+0000 osd.1 (osd.1) 163 : cluster [DBG] 61.0 scrub starts 2024-02-19T21:08:57.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:55.692310+0000 osd.1 (osd.1) 164 : cluster [DBG] 61.0 scrub ok 2024-02-19T21:08:57.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:56.598055+0000 osd.5 (osd.5) 146 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:08:57.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:56.600945+0000 osd.5 (osd.5) 147 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:08:57.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:57.238937+0000 mon.a (mon.0) 2315 : audit [INF] from='client.49661 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi163-83854-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:57.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:57.239023+0000 mon.a (mon.0) 2316 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2024-02-19T21:08:57.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:57.242157+0000 mon.c (mon.2) 360 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi163-83854-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:57.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:57.242461+0000 mon.a (mon.0) 2317 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi163-83854-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:57.708 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:55.688799+0000 osd.1 (osd.1) 163 : cluster [DBG] 61.0 scrub starts 2024-02-19T21:08:57.708 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:55.692310+0000 osd.1 (osd.1) 164 : cluster [DBG] 61.0 scrub ok 2024-02-19T21:08:57.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:56.598055+0000 osd.5 (osd.5) 146 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:08:57.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:56.600945+0000 osd.5 (osd.5) 147 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:08:57.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:57.238937+0000 mon.a (mon.0) 2315 : audit [INF] from='client.49661 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi163-83854-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:57.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:57.239023+0000 mon.a (mon.0) 2316 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2024-02-19T21:08:57.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:57.242157+0000 mon.c (mon.2) 360 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi163-83854-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:57.709 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:57.242461+0000 mon.a (mon.0) 2317 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi163-83854-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:57.770 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:55.688799+0000 osd.1 (osd.1) 163 : cluster [DBG] 61.0 scrub starts 2024-02-19T21:08:57.770 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:55.692310+0000 osd.1 (osd.1) 164 : cluster [DBG] 61.0 scrub ok 2024-02-19T21:08:57.771 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:56.598055+0000 osd.5 (osd.5) 146 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:08:57.771 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:56.600945+0000 osd.5 (osd.5) 147 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:08:57.771 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:57.238937+0000 mon.a (mon.0) 2315 : audit [INF] from='client.49661 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi163-83854-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:08:57.771 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:57.239023+0000 mon.a (mon.0) 2316 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2024-02-19T21:08:57.771 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:57.242157+0000 mon.c (mon.2) 360 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi163-83854-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:57.771 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:57.242461+0000 mon.a (mon.0) 2317 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi163-83854-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:08:57.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:57.789+0000 7fe4b1798700 1 -- 172.21.15.163:0/177824702 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fe4980051d0 con 0x7fe4ac0fc890 2024-02-19T21:08:58.068 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:08:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:08:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:08:58.356 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:08:58.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:58.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:08:58.469 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:08:58.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:58.476+0000 7fe493fff700 1 -- 172.21.15.163:0/177824702 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v358) v1 ==== 87+0+0 (unknown 1382637798 0 0) 0x7fe49c04a8b0 con 0x7fe4ac0fc890 2024-02-19T21:08:58.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:58.479+0000 7fe4b1798700 10 librados: watch_flush enter 2024-02-19T21:08:58.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:58.479+0000 7fe4b1798700 10 librados: watch_flush exit 2024-02-19T21:08:58.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:58.479+0000 7fe4b1798700 20 client.49667.objecter shutdown clearing up homeless session... 2024-02-19T21:08:58.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:58.479+0000 7fe4b1798700 10 client.49667.objecter successfully canceled tick 2024-02-19T21:08:58.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:58.479+0000 7fe4b1798700 1 -- 172.21.15.163:0/177824702 >> v1:172.21.15.163:6800/1552514681 conn(0x7fe494063c60 legacy=0x7fe494066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:58.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:58.479+0000 7fe4b1798700 1 -- 172.21.15.163:0/177824702 >> v1:172.21.15.163:6790/0 conn(0x7fe4ac0fc890 legacy=0x7fe4ac0fb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:58.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:58.479+0000 7fe4b1798700 1 -- 172.21.15.163:0/177824702 shutdown_connections 2024-02-19T21:08:58.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:58.479+0000 7fe4b1798700 1 -- 172.21.15.163:0/177824702 >> 172.21.15.163:0/177824702 conn(0x7fe4ac0f2470 msgr2=0x7fe4ac106640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:58.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:58.479+0000 7fe4b1798700 1 -- 172.21.15.163:0/177824702 shutdown_connections 2024-02-19T21:08:58.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:58.479+0000 7fe4b1798700 1 -- 172.21.15.163:0/177824702 wait complete. 2024-02-19T21:08:58.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:58.479+0000 7fe4b1798700 1 librados: shutdown 2024-02-19T21:08:58.477 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:08:58.504 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:58.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:56.659516+0000 osd.1 (osd.1) 165 : cluster [DBG] 61.1d scrub starts 2024-02-19T21:08:58.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:56.661291+0000 osd.1 (osd.1) 166 : cluster [DBG] 61.1d scrub ok 2024-02-19T21:08:58.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:57.004357+0000 osd.7 (osd.7) 151 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:08:58.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:57.007810+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:08:58.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:57.191398+0000 mgr.y (mgr.14550) 330 : cluster [DBG] pgmap v506: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:08:58.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:57.790948+0000 mon.c (mon.2) 361 : audit [INF] from='client.? v1:172.21.15.163:0/177824702' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:08:58.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:57.791255+0000 mon.a (mon.0) 2318 : audit [INF] from='client.49667 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:08:58.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:58.236854+0000 mon.a (mon.0) 2319 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:08:58.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:58.477345+0000 mon.a (mon.0) 2320 : audit [INF] from='client.49667 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:08:58.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:56.659516+0000 osd.1 (osd.1) 165 : cluster [DBG] 61.1d scrub starts 2024-02-19T21:08:58.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:56.661291+0000 osd.1 (osd.1) 166 : cluster [DBG] 61.1d scrub ok 2024-02-19T21:08:58.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:57.004357+0000 osd.7 (osd.7) 151 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:08:58.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:57.007810+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:08:58.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:57.191398+0000 mgr.y (mgr.14550) 330 : cluster [DBG] pgmap v506: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:08:58.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:57.790948+0000 mon.c (mon.2) 361 : audit [INF] from='client.? v1:172.21.15.163:0/177824702' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:08:58.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:57.791255+0000 mon.a (mon.0) 2318 : audit [INF] from='client.49667 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:08:58.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:58.236854+0000 mon.a (mon.0) 2319 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:08:58.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:58.477345+0000 mon.a (mon.0) 2320 : audit [INF] from='client.49667 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:08:58.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:56.659516+0000 osd.1 (osd.1) 165 : cluster [DBG] 61.1d scrub starts 2024-02-19T21:08:58.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:56.661291+0000 osd.1 (osd.1) 166 : cluster [DBG] 61.1d scrub ok 2024-02-19T21:08:58.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:57.004357+0000 osd.7 (osd.7) 151 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:08:58.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:57.007810+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:08:58.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:57.191398+0000 mgr.y (mgr.14550) 330 : cluster [DBG] pgmap v506: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:08:58.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:57.790948+0000 mon.c (mon.2) 361 : audit [INF] from='client.? v1:172.21.15.163:0/177824702' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:08:58.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:57.791255+0000 mon.a (mon.0) 2318 : audit [INF] from='client.49667 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:08:58.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:58.236854+0000 mon.a (mon.0) 2319 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:08:58.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:58.477345+0000 mon.a (mon.0) 2320 : audit [INF] from='client.49667 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:08:59.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.192+0000 7efe56137700 1 -- 172.21.15.163:0/250055435 >> v1:172.21.15.163:6790/0 conn(0x7efe501042b0 legacy=0x7efe50106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:59.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.192+0000 7efe56137700 1 -- 172.21.15.163:0/250055435 shutdown_connections 2024-02-19T21:08:59.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.192+0000 7efe56137700 1 -- 172.21.15.163:0/250055435 >> 172.21.15.163:0/250055435 conn(0x7efe500f2470 msgr2=0x7efe500f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:59.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.192+0000 7efe56137700 1 -- 172.21.15.163:0/250055435 shutdown_connections 2024-02-19T21:08:59.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.192+0000 7efe56137700 1 -- 172.21.15.163:0/250055435 wait complete. 2024-02-19T21:08:59.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.192+0000 7efe56137700 1 librados: starting msgr at 2024-02-19T21:08:59.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.192+0000 7efe56137700 1 librados: starting objecter 2024-02-19T21:08:59.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.192+0000 7efe56137700 1 Processor -- start 2024-02-19T21:08:59.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.192+0000 7efe56137700 1 -- start start 2024-02-19T21:08:59.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.192+0000 7efe56137700 1 librados: setting wanted keys 2024-02-19T21:08:59.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.192+0000 7efe56137700 1 librados: calling monclient init 2024-02-19T21:08:59.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe56137700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efe501174a0 con 0x7efe501042b0 2024-02-19T21:08:59.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe56137700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efe50118680 con 0x7efe500f8980 2024-02-19T21:08:59.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe56137700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efe50119880 con 0x7efe500f9530 2024-02-19T21:08:59.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe46ffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7efe500f9530 0x7efe500fff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:35510/0 (socket says 172.21.15.163:35510) 2024-02-19T21:08:59.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe46ffd700 1 -- 172.21.15.163:0/1817000787 learned_addr learned my addr 172.21.15.163:0/1817000787 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:59.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe4ffff700 1 --1- 172.21.15.163:0/1817000787 >> v1:172.21.15.163:6789/0 conn(0x7efe501042b0 0x7efe50115b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:56582/0 (socket says 172.21.15.163:56582) 2024-02-19T21:08:59.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe4d7fa700 10 client.?.objecter ms_handle_connect 0x7efe501042b0 2024-02-19T21:08:59.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe4d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:59.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe4d7fa700 10 client.?.objecter ms_handle_connect 0x7efe500f9530 2024-02-19T21:08:59.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe4d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:59.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe4d7fa700 10 client.?.objecter ms_handle_connect 0x7efe500f8980 2024-02-19T21:08:59.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe4d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:59.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288082070 0 0) 0x7efe501174a0 con 0x7efe501042b0 2024-02-19T21:08:59.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe3c003630 con 0x7efe501042b0 2024-02-19T21:08:59.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211922169 0 0) 0x7efe50119880 con 0x7efe500f9530 2024-02-19T21:08:59.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe501174a0 con 0x7efe500f9530 2024-02-19T21:08:59.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635470384 0 0) 0x7efe50118680 con 0x7efe500f8980 2024-02-19T21:08:59.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.193+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe50119880 con 0x7efe500f8980 2024-02-19T21:08:59.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 799944650 0 0) 0x7efe501174a0 con 0x7efe500f9530 2024-02-19T21:08:59.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe50118680 con 0x7efe500f9530 2024-02-19T21:08:59.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2733691486 0 0) 0x7efe3c003630 con 0x7efe501042b0 2024-02-19T21:08:59.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe501174a0 con 0x7efe501042b0 2024-02-19T21:08:59.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4012638274 0 0) 0x7efe50119880 con 0x7efe500f8980 2024-02-19T21:08:59.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe3c003630 con 0x7efe500f8980 2024-02-19T21:08:59.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7efe380018c0 con 0x7efe500f9530 2024-02-19T21:08:59.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7efe480031c0 con 0x7efe501042b0 2024-02-19T21:08:59.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7efe40001770 con 0x7efe500f8980 2024-02-19T21:08:59.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2476340322 0 0) 0x7efe50118680 con 0x7efe500f9530 2024-02-19T21:08:59.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 >> v1:172.21.15.193:6789/0 conn(0x7efe500f8980 legacy=0x7efe500ff820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:59.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 >> v1:172.21.15.163:6789/0 conn(0x7efe501042b0 legacy=0x7efe50115b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:59.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5011aa80 con 0x7efe500f9530 2024-02-19T21:08:59.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe56137700 1 -- 172.21.15.163:0/1817000787 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe50119a70 con 0x7efe500f9530 2024-02-19T21:08:59.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe56137700 10 client.49673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:59.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe56137700 1 -- 172.21.15.163:0/1817000787 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe50119ee0 con 0x7efe500f9530 2024-02-19T21:08:59.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe56137700 1 librados: init done 2024-02-19T21:08:59.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7efe38001fc0 con 0x7efe500f9530 2024-02-19T21:08:59.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.194+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7efe38005a70 con 0x7efe500f9530 2024-02-19T21:08:59.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.195+0000 7efe56137700 1 -- 172.21.15.163:0/1817000787 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe30004f20 con 0x7efe500f9530 2024-02-19T21:08:59.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.195+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7efe38001b20 con 0x7efe500f9530 2024-02-19T21:08:59.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.196+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(358..358 src has 43..358) v4 ==== 8305+0+0 (unknown 3064570941 0 0) 0x7efe38078ce0 con 0x7efe500f9530 2024-02-19T21:08:59.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.196+0000 7efe4d7fa700 10 client.49673.objecter ms_dispatch 0x7efe50001550 osd_map(358..358 src has 43..358) v4 2024-02-19T21:08:59.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.196+0000 7efe4d7fa700 3 client.49673.objecter handle_osd_map got epochs [358,358] > 0 2024-02-19T21:08:59.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.196+0000 7efe4d7fa700 3 client.49673.objecter handle_osd_map decoding full epoch 358 2024-02-19T21:08:59.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.196+0000 7efe4d7fa700 20 client.49673.objecter dump_active .. 0 homeless 2024-02-19T21:08:59.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.196+0000 7efe4d7fa700 10 client.49673.objecter ms_handle_connect 0x7efe3c068090 2024-02-19T21:08:59.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.200+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7efe3804c000 con 0x7efe500f9530 2024-02-19T21:08:59.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.395+0000 7efe56137700 1 -- 172.21.15.163:0/1817000787 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7efe300051d0 con 0x7efe500f9530 2024-02-19T21:08:59.654 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:08:59.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:59.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:08:59.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:57.650176+0000 osd.1 (osd.1) 167 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:08:59.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:57.653200+0000 osd.1 (osd.1) 168 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:08:59.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:57.879805+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.b scrub starts 2024-02-19T21:08:59.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:57.880930+0000 osd.3 (osd.3) 156 : cluster [DBG] 6.b scrub ok 2024-02-19T21:08:59.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:58.035209+0000 osd.2 (osd.2) 132 : cluster [DBG] 61.6 scrub starts 2024-02-19T21:08:59.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:58.036950+0000 osd.2 (osd.2) 133 : cluster [DBG] 61.6 scrub ok 2024-02-19T21:08:59.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:58.477444+0000 mon.a (mon.0) 2321 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2024-02-19T21:08:59.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:59.242599+0000 mon.a (mon.0) 2322 : audit [INF] from='client.49661 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi163-83854-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi163-83854-62"}]': finished 2024-02-19T21:08:59.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:59.242700+0000 mon.a (mon.0) 2323 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2024-02-19T21:08:59.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:59.397353+0000 mon.c (mon.2) 362 : audit [INF] from='client.? v1:172.21.15.163:0/1817000787' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:08:59.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:59.397668+0000 mon.a (mon.0) 2324 : audit [INF] from='client.49673 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:08:59.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:57.650176+0000 osd.1 (osd.1) 167 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:08:59.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:57.653200+0000 osd.1 (osd.1) 168 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:08:59.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:57.879805+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.b scrub starts 2024-02-19T21:08:59.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:57.880930+0000 osd.3 (osd.3) 156 : cluster [DBG] 6.b scrub ok 2024-02-19T21:08:59.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:58.035209+0000 osd.2 (osd.2) 132 : cluster [DBG] 61.6 scrub starts 2024-02-19T21:08:59.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:58.036950+0000 osd.2 (osd.2) 133 : cluster [DBG] 61.6 scrub ok 2024-02-19T21:08:59.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:58.477444+0000 mon.a (mon.0) 2321 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2024-02-19T21:08:59.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:59.242599+0000 mon.a (mon.0) 2322 : audit [INF] from='client.49661 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi163-83854-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi163-83854-62"}]': finished 2024-02-19T21:08:59.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:59.242700+0000 mon.a (mon.0) 2323 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2024-02-19T21:08:59.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:59.397353+0000 mon.c (mon.2) 362 : audit [INF] from='client.? v1:172.21.15.163:0/1817000787' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:08:59.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:08:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:59.397668+0000 mon.a (mon.0) 2324 : audit [INF] from='client.49673 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:08:59.759 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:08:59.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:57.650176+0000 osd.1 (osd.1) 167 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:08:59.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:57.653200+0000 osd.1 (osd.1) 168 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:08:59.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:57.879805+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.b scrub starts 2024-02-19T21:08:59.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:57.880930+0000 osd.3 (osd.3) 156 : cluster [DBG] 6.b scrub ok 2024-02-19T21:08:59.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:58.035209+0000 osd.2 (osd.2) 132 : cluster [DBG] 61.6 scrub starts 2024-02-19T21:08:59.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:58.036950+0000 osd.2 (osd.2) 133 : cluster [DBG] 61.6 scrub ok 2024-02-19T21:08:59.789 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:58.477444+0000 mon.a (mon.0) 2321 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2024-02-19T21:08:59.790 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:59.242599+0000 mon.a (mon.0) 2322 : audit [INF] from='client.49661 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi163-83854-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi163-83854-62"}]': finished 2024-02-19T21:08:59.790 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:59.242700+0000 mon.a (mon.0) 2323 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2024-02-19T21:08:59.790 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:59.397353+0000 mon.c (mon.2) 362 : audit [INF] from='client.? v1:172.21.15.163:0/1817000787' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:08:59.790 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:08:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:59.397668+0000 mon.a (mon.0) 2324 : audit [INF] from='client.49673 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:08:59.791 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:59.812 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:08:59.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:08:59.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:08:59.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.905+0000 7fb8f007f700 1 -- 172.21.15.163:0/556167138 >> v1:172.21.15.163:6790/0 conn(0x7fb8e81042a0 legacy=0x7fb8e8106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:59.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.905+0000 7fb8f007f700 1 -- 172.21.15.163:0/556167138 shutdown_connections 2024-02-19T21:08:59.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.905+0000 7fb8f007f700 1 -- 172.21.15.163:0/556167138 >> 172.21.15.163:0/556167138 conn(0x7fb8e80f24b0 msgr2=0x7fb8e80f48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:08:59.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.906+0000 7fb8f007f700 1 -- 172.21.15.163:0/556167138 shutdown_connections 2024-02-19T21:08:59.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.906+0000 7fb8f007f700 1 -- 172.21.15.163:0/556167138 wait complete. 2024-02-19T21:08:59.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.906+0000 7fb8f007f700 1 librados: starting msgr at 2024-02-19T21:08:59.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.906+0000 7fb8f007f700 1 librados: starting objecter 2024-02-19T21:08:59.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.906+0000 7fb8f007f700 1 Processor -- start 2024-02-19T21:08:59.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.906+0000 7fb8f007f700 1 -- start start 2024-02-19T21:08:59.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.906+0000 7fb8f007f700 1 librados: setting wanted keys 2024-02-19T21:08:59.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.906+0000 7fb8f007f700 1 librados: calling monclient init 2024-02-19T21:08:59.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8f007f700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb8e81172e0 con 0x7fb8e81042a0 2024-02-19T21:08:59.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8f007f700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb8e81184c0 con 0x7fb8e80f8940 2024-02-19T21:08:59.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8f007f700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb8e81196c0 con 0x7fb8e80f9430 2024-02-19T21:08:59.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8ed61a700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fb8e80f9430 0x7fb8e80ffd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:35520/0 (socket says 172.21.15.163:35520) 2024-02-19T21:08:59.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8ed61a700 1 -- 172.21.15.163:0/2081228583 learned_addr learned my addr 172.21.15.163:0/2081228583 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:08:59.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 10 client.?.objecter ms_handle_connect 0x7fb8e80f9430 2024-02-19T21:08:59.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:59.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 10 client.?.objecter ms_handle_connect 0x7fb8e81042a0 2024-02-19T21:08:59.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:59.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 10 client.?.objecter ms_handle_connect 0x7fb8e80f8940 2024-02-19T21:08:59.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:08:59.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101264374 0 0) 0x7fb8e81196c0 con 0x7fb8e80f9430 2024-02-19T21:08:59.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d40033f0 con 0x7fb8e80f9430 2024-02-19T21:08:59.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869302647 0 0) 0x7fb8e81172e0 con 0x7fb8e81042a0 2024-02-19T21:08:59.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e81196c0 con 0x7fb8e81042a0 2024-02-19T21:08:59.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571139815 0 0) 0x7fb8e81184c0 con 0x7fb8e80f8940 2024-02-19T21:08:59.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e81172e0 con 0x7fb8e80f8940 2024-02-19T21:08:59.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4009425528 0 0) 0x7fb8e81196c0 con 0x7fb8e81042a0 2024-02-19T21:08:59.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8e81184c0 con 0x7fb8e81042a0 2024-02-19T21:08:59.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 315182222 0 0) 0x7fb8d40033f0 con 0x7fb8e80f9430 2024-02-19T21:08:59.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8e81196c0 con 0x7fb8e80f9430 2024-02-19T21:08:59.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb8e0002370 con 0x7fb8e81042a0 2024-02-19T21:08:59.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1743592837 0 0) 0x7fb8e81172e0 con 0x7fb8e80f8940 2024-02-19T21:08:59.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8d40033f0 con 0x7fb8e80f8940 2024-02-19T21:08:59.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb8d80019d0 con 0x7fb8e80f9430 2024-02-19T21:08:59.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb8e4001840 con 0x7fb8e80f8940 2024-02-19T21:08:59.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2856216524 0 0) 0x7fb8e81196c0 con 0x7fb8e80f9430 2024-02-19T21:08:59.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 >> v1:172.21.15.193:6789/0 conn(0x7fb8e80f8940 legacy=0x7fb8e80ff630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:59.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 >> v1:172.21.15.163:6789/0 conn(0x7fb8e81042a0 legacy=0x7fb8e81159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:08:59.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e811a8c0 con 0x7fb8e80f9430 2024-02-19T21:08:59.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.907+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e81174d0 con 0x7fb8e80f9430 2024-02-19T21:08:59.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.908+0000 7fb8f007f700 10 client.49679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:08:59.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.908+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e8117940 con 0x7fb8e80f9430 2024-02-19T21:08:59.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.908+0000 7fb8f007f700 1 librados: init done 2024-02-19T21:08:59.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.908+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb8d80024a0 con 0x7fb8e80f9430 2024-02-19T21:08:59.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.908+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb8d8005ac0 con 0x7fb8e80f9430 2024-02-19T21:08:59.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.908+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb8cc004f00 con 0x7fb8e80f9430 2024-02-19T21:08:59.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.909+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fb8d8002000 con 0x7fb8e80f9430 2024-02-19T21:08:59.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.909+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(359..359 src has 43..359) v4 ==== 8718+0+0 (unknown 1159378182 0 0) 0x7fb8d8078f50 con 0x7fb8e80f9430 2024-02-19T21:08:59.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.909+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 osd_map(359..359 src has 43..359) v4 2024-02-19T21:08:59.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.909+0000 7fb8deffd700 3 client.49679.objecter handle_osd_map got epochs [359,359] > 0 2024-02-19T21:08:59.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.909+0000 7fb8deffd700 3 client.49679.objecter handle_osd_map decoding full epoch 359 2024-02-19T21:08:59.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.910+0000 7fb8deffd700 20 client.49679.objecter dump_active .. 0 homeless 2024-02-19T21:08:59.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.910+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v359) v1 ==== 54+0+16 (unknown 1822379672 0 2420000192) 0x7fb8d8079330 con 0x7fb8e80f9430 2024-02-19T21:08:59.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.910+0000 7fb8deffd700 10 client.49679.objecter ms_handle_connect 0x7fb8d4063c40 2024-02-19T21:08:59.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.910+0000 7fb8f007f700 10 client.49679.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:59.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.910+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 1 2024-02-19T21:08:59.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.910+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc0051a0 osd=0 3 2024-02-19T21:08:59.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.910+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc0051a0 osd=0 3 2024-02-19T21:08:59.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.910+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc0051a0 osd=0 3 2024-02-19T21:08:59.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.910+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 1 2024-02-19T21:08:59.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.910+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc0051a0 osd=0 3 2024-02-19T21:08:59.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.910+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 0 1 2024-02-19T21:08:59.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.910+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc0051a0 osd=0 4 2024-02-19T21:08:59.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.910+0000 7fb8f007f700 10 client.49679.objecter _send_command 1 2024-02-19T21:08:59.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.910+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8cc004950 con 0x7fb8cc004530 2024-02-19T21:08:59.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.911+0000 7fb8deffd700 10 client.49679.objecter ms_handle_connect 0x7fb8cc004530 2024-02-19T21:08:59.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.911+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb8cc004950 con 0x7fb8cc004530 2024-02-19T21:08:59.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.911+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:08:59.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.911+0000 7fb8deffd700 10 client.49679.objecter _finish_command 1 = system:0 2024-02-19T21:08:59.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.911+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc0051a0 osd=0 4 2024-02-19T21:08:59.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.911+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 0 1 2024-02-19T21:08:59.914 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:08:59.917 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:08:59.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.927+0000 7fb8f007f700 10 client.49679.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:59.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.927+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 2 2024-02-19T21:08:59.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.927+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc0051a0 osd=0 3 2024-02-19T21:08:59.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.927+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc0051a0 osd=0 3 2024-02-19T21:08:59.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.927+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc0051a0 osd=0 3 2024-02-19T21:08:59.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.927+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 2 2024-02-19T21:08:59.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.927+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc0051a0 osd=0 3 2024-02-19T21:08:59.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.927+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 0 2 2024-02-19T21:08:59.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.927+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc0051a0 osd=0 4 2024-02-19T21:08:59.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.927+0000 7fb8f007f700 10 client.49679.objecter _send_command 2 2024-02-19T21:08:59.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.927+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb8cc004d30 con 0x7fb8cc004530 2024-02-19T21:08:59.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb8cc004d30 con 0x7fb8cc004530 2024-02-19T21:08:59.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:59.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8deffd700 10 client.49679.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:59.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc0051a0 osd=0 4 2024-02-19T21:08:59.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 0 2 2024-02-19T21:08:59.928 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:08:59.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8f007f700 10 client.49679.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:59.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 3 2024-02-19T21:08:59.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc004270 osd=1 3 2024-02-19T21:08:59.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc004270 osd=1 3 2024-02-19T21:08:59.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc004270 osd=1 3 2024-02-19T21:08:59.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 3 2024-02-19T21:08:59.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc004270 osd=1 3 2024-02-19T21:08:59.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 1 3 2024-02-19T21:08:59.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc004270 osd=1 4 2024-02-19T21:08:59.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8f007f700 10 client.49679.objecter _send_command 3 2024-02-19T21:08:59.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8cc00bf60 con 0x7fb8cc003b40 2024-02-19T21:08:59.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.930+0000 7fb8deffd700 10 client.49679.objecter ms_handle_connect 0x7fb8cc003b40 2024-02-19T21:08:59.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.931+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb8cc00bf60 con 0x7fb8cc003b40 2024-02-19T21:08:59.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.931+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:08:59.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.931+0000 7fb8deffd700 10 client.49679.objecter _finish_command 3 = system:0 2024-02-19T21:08:59.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.931+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc004270 osd=1 4 2024-02-19T21:08:59.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.931+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 1 3 2024-02-19T21:08:59.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.948+0000 7fb8f007f700 10 client.49679.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:59.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.948+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 4 2024-02-19T21:08:59.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.948+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc004270 osd=1 3 2024-02-19T21:08:59.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.948+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc004270 osd=1 3 2024-02-19T21:08:59.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.948+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc004270 osd=1 3 2024-02-19T21:08:59.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.948+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 4 2024-02-19T21:08:59.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.948+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc004270 osd=1 3 2024-02-19T21:08:59.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.948+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 1 4 2024-02-19T21:08:59.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.948+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc004270 osd=1 4 2024-02-19T21:08:59.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.948+0000 7fb8f007f700 10 client.49679.objecter _send_command 4 2024-02-19T21:08:59.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.948+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb8cc00d280 con 0x7fb8cc003b40 2024-02-19T21:08:59.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb8cc00d280 con 0x7fb8cc003b40 2024-02-19T21:08:59.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:59.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8deffd700 10 client.49679.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:59.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc004270 osd=1 4 2024-02-19T21:08:59.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 1 4 2024-02-19T21:08:59.948 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:08:59.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8f007f700 10 client.49679.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:59.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 5 2024-02-19T21:08:59.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc0033a0 osd=2 3 2024-02-19T21:08:59.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc0033a0 osd=2 3 2024-02-19T21:08:59.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc0033a0 osd=2 3 2024-02-19T21:08:59.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 5 2024-02-19T21:08:59.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc0033a0 osd=2 3 2024-02-19T21:08:59.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 2 5 2024-02-19T21:08:59.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc0033a0 osd=2 4 2024-02-19T21:08:59.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8f007f700 10 client.49679.objecter _send_command 5 2024-02-19T21:08:59.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.950+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8cc010d30 con 0x7fb8cc002070 2024-02-19T21:08:59.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.951+0000 7fb8deffd700 10 client.49679.objecter ms_handle_connect 0x7fb8cc002070 2024-02-19T21:08:59.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.952+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb8cc010d30 con 0x7fb8cc002070 2024-02-19T21:08:59.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.952+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:08:59.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.952+0000 7fb8deffd700 10 client.49679.objecter _finish_command 5 = system:0 2024-02-19T21:08:59.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.952+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc0033a0 osd=2 4 2024-02-19T21:08:59.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.952+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 2 5 2024-02-19T21:08:59.950 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:08:59.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.970+0000 7fb8f007f700 10 client.49679.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:59.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.970+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 6 2024-02-19T21:08:59.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.970+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc0033a0 osd=2 3 2024-02-19T21:08:59.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.970+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc0033a0 osd=2 3 2024-02-19T21:08:59.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.970+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc0033a0 osd=2 3 2024-02-19T21:08:59.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.970+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 6 2024-02-19T21:08:59.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.970+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc0033a0 osd=2 3 2024-02-19T21:08:59.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.970+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 2 6 2024-02-19T21:08:59.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.970+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc0033a0 osd=2 4 2024-02-19T21:08:59.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.970+0000 7fb8f007f700 10 client.49679.objecter _send_command 6 2024-02-19T21:08:59.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.970+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb8cc001630 con 0x7fb8cc002070 2024-02-19T21:08:59.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb8cc001630 con 0x7fb8cc002070 2024-02-19T21:08:59.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:59.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8deffd700 10 client.49679.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:59.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc0033a0 osd=2 4 2024-02-19T21:08:59.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 2 6 2024-02-19T21:08:59.970 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:08:59.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8f007f700 10 client.49679.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:59.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 7 2024-02-19T21:08:59.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc00c8d0 osd=3 3 2024-02-19T21:08:59.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc00c8d0 osd=3 3 2024-02-19T21:08:59.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc00c8d0 osd=3 3 2024-02-19T21:08:59.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 7 2024-02-19T21:08:59.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc00c8d0 osd=3 3 2024-02-19T21:08:59.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 3 7 2024-02-19T21:08:59.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc00c8d0 osd=3 4 2024-02-19T21:08:59.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8f007f700 10 client.49679.objecter _send_command 7 2024-02-19T21:08:59.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8cc00d050 con 0x7fb8cc010ea0 2024-02-19T21:08:59.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.972+0000 7fb8deffd700 10 client.49679.objecter ms_handle_connect 0x7fb8cc010ea0 2024-02-19T21:08:59.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.973+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb8cc00d050 con 0x7fb8cc010ea0 2024-02-19T21:08:59.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.973+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:08:59.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.973+0000 7fb8deffd700 10 client.49679.objecter _finish_command 7 = system:0 2024-02-19T21:08:59.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.973+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc00c8d0 osd=3 4 2024-02-19T21:08:59.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.973+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 3 7 2024-02-19T21:08:59.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.987+0000 7fb8f007f700 10 client.49679.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:08:59.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.987+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 8 2024-02-19T21:08:59.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.987+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc00c8d0 osd=3 3 2024-02-19T21:08:59.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.987+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc00c8d0 osd=3 3 2024-02-19T21:08:59.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.987+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc00c8d0 osd=3 3 2024-02-19T21:08:59.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.987+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 8 2024-02-19T21:08:59.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.987+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc00c8d0 osd=3 3 2024-02-19T21:08:59.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.987+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 3 8 2024-02-19T21:08:59.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.987+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc00c8d0 osd=3 4 2024-02-19T21:08:59.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.987+0000 7fb8f007f700 10 client.49679.objecter _send_command 8 2024-02-19T21:08:59.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.987+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb8cc00d050 con 0x7fb8cc010ea0 2024-02-19T21:08:59.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb8cc00d050 con 0x7fb8cc010ea0 2024-02-19T21:08:59.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:08:59.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8deffd700 10 client.49679.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:08:59.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc00c8d0 osd=3 4 2024-02-19T21:08:59.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 3 8 2024-02-19T21:08:59.989 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:08:59.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8f007f700 10 client.49679.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:08:59.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 9 2024-02-19T21:08:59.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc00d050 osd=4 3 2024-02-19T21:08:59.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc00d050 osd=4 3 2024-02-19T21:08:59.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc00d050 osd=4 3 2024-02-19T21:08:59.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 9 2024-02-19T21:08:59.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc00d050 osd=4 3 2024-02-19T21:08:59.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 4 9 2024-02-19T21:08:59.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc00d050 osd=4 4 2024-02-19T21:08:59.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8f007f700 10 client.49679.objecter _send_command 9 2024-02-19T21:08:59.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8cc01a200 con 0x7fb8cc0166f0 2024-02-19T21:08:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.989+0000 7fb8deffd700 10 client.49679.objecter ms_handle_connect 0x7fb8cc0166f0 2024-02-19T21:08:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.990+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb8cc01a200 con 0x7fb8cc0166f0 2024-02-19T21:08:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.990+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:08:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.990+0000 7fb8deffd700 10 client.49679.objecter _finish_command 9 = system:0 2024-02-19T21:08:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.990+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc00d050 osd=4 4 2024-02-19T21:08:59.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:08:59.990+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 4 9 2024-02-19T21:09:00.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.004+0000 7fb8f007f700 10 client.49679.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:00.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.004+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 10 2024-02-19T21:09:00.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.004+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc00d050 osd=4 3 2024-02-19T21:09:00.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.004+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc00d050 osd=4 3 2024-02-19T21:09:00.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.004+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc00d050 osd=4 3 2024-02-19T21:09:00.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.004+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 10 2024-02-19T21:09:00.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.004+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc00d050 osd=4 3 2024-02-19T21:09:00.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.004+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 4 10 2024-02-19T21:09:00.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.004+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc00d050 osd=4 4 2024-02-19T21:09:00.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.004+0000 7fb8f007f700 10 client.49679.objecter _send_command 10 2024-02-19T21:09:00.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.004+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb8cc01e610 con 0x7fb8cc0166f0 2024-02-19T21:09:00.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.006+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb8cc01e610 con 0x7fb8cc0166f0 2024-02-19T21:09:00.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.006+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:00.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.006+0000 7fb8deffd700 10 client.49679.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:00.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.006+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc00d050 osd=4 4 2024-02-19T21:09:00.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.006+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 4 10 2024-02-19T21:09:00.004 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:09:00.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.007+0000 7fb8f007f700 10 client.49679.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:00.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.007+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 11 2024-02-19T21:09:00.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.007+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc01e610 osd=5 3 2024-02-19T21:09:00.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.007+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc01e610 osd=5 3 2024-02-19T21:09:00.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.007+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc01e610 osd=5 3 2024-02-19T21:09:00.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.007+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 11 2024-02-19T21:09:00.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.007+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc01e610 osd=5 3 2024-02-19T21:09:00.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.007+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 5 11 2024-02-19T21:09:00.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.007+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc01e610 osd=5 4 2024-02-19T21:09:00.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.007+0000 7fb8f007f700 10 client.49679.objecter _send_command 11 2024-02-19T21:09:00.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.007+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8cc022950 con 0x7fb8cc01ec90 2024-02-19T21:09:00.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.007+0000 7fb8deffd700 10 client.49679.objecter ms_handle_connect 0x7fb8cc01ec90 2024-02-19T21:09:00.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.008+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb8cc022950 con 0x7fb8cc01ec90 2024-02-19T21:09:00.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.008+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:09:00.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.008+0000 7fb8deffd700 10 client.49679.objecter _finish_command 11 = system:0 2024-02-19T21:09:00.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.008+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc01e610 osd=5 4 2024-02-19T21:09:00.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.008+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 5 11 2024-02-19T21:09:00.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.022+0000 7fb8f007f700 10 client.49679.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:00.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.022+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 12 2024-02-19T21:09:00.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.022+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc01e610 osd=5 3 2024-02-19T21:09:00.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.022+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc01e610 osd=5 3 2024-02-19T21:09:00.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.022+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc01e610 osd=5 3 2024-02-19T21:09:00.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.022+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 12 2024-02-19T21:09:00.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.022+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc01e610 osd=5 3 2024-02-19T21:09:00.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.022+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 5 12 2024-02-19T21:09:00.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.022+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc01e610 osd=5 4 2024-02-19T21:09:00.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.022+0000 7fb8f007f700 10 client.49679.objecter _send_command 12 2024-02-19T21:09:00.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.022+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb8cc022950 con 0x7fb8cc01ec90 2024-02-19T21:09:00.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.024+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb8cc022950 con 0x7fb8cc01ec90 2024-02-19T21:09:00.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.024+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:00.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.024+0000 7fb8deffd700 10 client.49679.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:00.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.024+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc01e610 osd=5 4 2024-02-19T21:09:00.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.024+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 5 12 2024-02-19T21:09:00.022 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:09:00.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.025+0000 7fb8f007f700 10 client.49679.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:00.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.025+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 13 2024-02-19T21:09:00.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.025+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc022950 osd=6 3 2024-02-19T21:09:00.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.025+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc022950 osd=6 3 2024-02-19T21:09:00.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.025+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc022950 osd=6 3 2024-02-19T21:09:00.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.025+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 13 2024-02-19T21:09:00.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.025+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc022950 osd=6 3 2024-02-19T21:09:00.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.025+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 6 13 2024-02-19T21:09:00.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.025+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc022950 osd=6 4 2024-02-19T21:09:00.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.025+0000 7fb8f007f700 10 client.49679.objecter _send_command 13 2024-02-19T21:09:00.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.025+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8cc023b70 con 0x7fb8cc024650 2024-02-19T21:09:00.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.025+0000 7fb8deffd700 10 client.49679.objecter ms_handle_connect 0x7fb8cc024650 2024-02-19T21:09:00.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.026+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb8cc023b70 con 0x7fb8cc024650 2024-02-19T21:09:00.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.026+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:09:00.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.026+0000 7fb8deffd700 10 client.49679.objecter _finish_command 13 = system:0 2024-02-19T21:09:00.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.026+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc022950 osd=6 4 2024-02-19T21:09:00.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.026+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 6 13 2024-02-19T21:09:00.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.040+0000 7fb8f007f700 10 client.49679.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:00.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.040+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 14 2024-02-19T21:09:00.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.040+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc022950 osd=6 3 2024-02-19T21:09:00.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.040+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc022950 osd=6 3 2024-02-19T21:09:00.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.040+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc022950 osd=6 3 2024-02-19T21:09:00.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.040+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 14 2024-02-19T21:09:00.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.040+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc022950 osd=6 3 2024-02-19T21:09:00.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.040+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 6 14 2024-02-19T21:09:00.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.040+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc022950 osd=6 4 2024-02-19T21:09:00.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.040+0000 7fb8f007f700 10 client.49679.objecter _send_command 14 2024-02-19T21:09:00.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.040+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb8cc023b70 con 0x7fb8cc024650 2024-02-19T21:09:00.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb8cc023b70 con 0x7fb8cc024650 2024-02-19T21:09:00.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:00.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8deffd700 10 client.49679.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:00.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc022950 osd=6 4 2024-02-19T21:09:00.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 6 14 2024-02-19T21:09:00.040 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:09:00.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8f007f700 10 client.49679.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:00.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 15 2024-02-19T21:09:00.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc023db0 osd=7 3 2024-02-19T21:09:00.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc023db0 osd=7 3 2024-02-19T21:09:00.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc023db0 osd=7 3 2024-02-19T21:09:00.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 15 2024-02-19T21:09:00.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc023db0 osd=7 3 2024-02-19T21:09:00.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 7 15 2024-02-19T21:09:00.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc023db0 osd=7 4 2024-02-19T21:09:00.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8f007f700 10 client.49679.objecter _send_command 15 2024-02-19T21:09:00.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.042+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8cc02b320 con 0x7fb8cc027810 2024-02-19T21:09:00.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.043+0000 7fb8deffd700 10 client.49679.objecter ms_handle_connect 0x7fb8cc027810 2024-02-19T21:09:00.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.043+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb8cc02b320 con 0x7fb8cc027810 2024-02-19T21:09:00.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.043+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:09:00.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.043+0000 7fb8deffd700 10 client.49679.objecter _finish_command 15 = system:0 2024-02-19T21:09:00.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.043+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc023db0 osd=7 4 2024-02-19T21:09:00.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.043+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 7 15 2024-02-19T21:09:00.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.058+0000 7fb8f007f700 10 client.49679.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:00.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.058+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign -1 16 2024-02-19T21:09:00.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.058+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc023db0 osd=7 3 2024-02-19T21:09:00.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.058+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc023db0 osd=7 3 2024-02-19T21:09:00.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.058+0000 7fb8f007f700 20 client.49679.objecter _get_session s=0x7fb8cc023db0 osd=7 3 2024-02-19T21:09:00.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.058+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_remove -1 16 2024-02-19T21:09:00.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.058+0000 7fb8f007f700 20 client.49679.objecter get_session s=0x7fb8cc023db0 osd=7 3 2024-02-19T21:09:00.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.058+0000 7fb8f007f700 15 client.49679.objecter _session_command_op_assign 7 16 2024-02-19T21:09:00.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.058+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc023db0 osd=7 4 2024-02-19T21:09:00.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.058+0000 7fb8f007f700 10 client.49679.objecter _send_command 16 2024-02-19T21:09:00.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.058+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb8cc02b320 con 0x7fb8cc027810 2024-02-19T21:09:00.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8deffd700 1 -- 172.21.15.163:0/2081228583 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb8cc02b320 con 0x7fb8cc027810 2024-02-19T21:09:00.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8deffd700 10 client.49679.objecter ms_dispatch 0x7fb8e8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:00.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8deffd700 10 client.49679.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:00.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8deffd700 20 client.49679.objecter put_session s=0x7fb8cc023db0 osd=7 4 2024-02-19T21:09:00.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8deffd700 15 client.49679.objecter _session_command_op_remove 7 16 2024-02-19T21:09:00.060 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:09:00.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 10 librados: watch_flush enter 2024-02-19T21:09:00.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 10 librados: watch_flush exit 2024-02-19T21:09:00.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 10 client.49679.objecter close_session for osd.0 2024-02-19T21:09:00.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 >> v1:172.21.15.163:6801/1559418115 conn(0x7fb8cc004530 legacy=0x7fb8cc002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:00.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc0051a0 osd=0 1 2024-02-19T21:09:00.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 10 client.49679.objecter close_session for osd.1 2024-02-19T21:09:00.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 >> v1:172.21.15.163:6805/3125047432 conn(0x7fb8cc003b40 legacy=0x7fb8cc00a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:00.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc004270 osd=1 1 2024-02-19T21:09:00.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 10 client.49679.objecter close_session for osd.2 2024-02-19T21:09:00.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 >> v1:172.21.15.163:6809/1563158487 conn(0x7fb8cc002070 legacy=0x7fb8cc00f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:00.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc0033a0 osd=2 1 2024-02-19T21:09:00.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 10 client.49679.objecter close_session for osd.3 2024-02-19T21:09:00.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 >> v1:172.21.15.163:6813/2112279769 conn(0x7fb8cc010ea0 legacy=0x7fb8cc0112e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:00.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc00c8d0 osd=3 1 2024-02-19T21:09:00.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 10 client.49679.objecter close_session for osd.4 2024-02-19T21:09:00.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8ee61c700 1 -- 172.21.15.163:0/2081228583 reap_dead start 2024-02-19T21:09:00.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 >> v1:172.21.15.193:6800/3129716965 conn(0x7fb8cc0166f0 legacy=0x7fb8cc018ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:00.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8ee61c700 1 -- 172.21.15.163:0/2081228583 reap_dead start 2024-02-19T21:09:00.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc00d050 osd=4 1 2024-02-19T21:09:00.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 10 client.49679.objecter close_session for osd.5 2024-02-19T21:09:00.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 >> v1:172.21.15.193:6804/805161991 conn(0x7fb8cc01ec90 legacy=0x7fb8cc0210a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:00.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc01e610 osd=5 1 2024-02-19T21:09:00.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 10 client.49679.objecter close_session for osd.6 2024-02-19T21:09:00.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 >> v1:172.21.15.193:6808/787604615 conn(0x7fb8cc024650 legacy=0x7fb8cc024ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:00.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc022950 osd=6 1 2024-02-19T21:09:00.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 10 client.49679.objecter close_session for osd.7 2024-02-19T21:09:00.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 >> v1:172.21.15.193:6812/157927432 conn(0x7fb8cc027810 legacy=0x7fb8cc029c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:00.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 20 client.49679.objecter put_session s=0x7fb8cc023db0 osd=7 1 2024-02-19T21:09:00.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 20 client.49679.objecter shutdown clearing up homeless session... 2024-02-19T21:09:00.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 10 client.49679.objecter successfully canceled tick 2024-02-19T21:09:00.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 >> v1:172.21.15.163:6800/1552514681 conn(0x7fb8d4063c40 legacy=0x7fb8d4066100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:00.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 >> v1:172.21.15.163:6790/0 conn(0x7fb8e80f9430 legacy=0x7fb8e80ffd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:00.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8ee61c700 1 -- 172.21.15.163:0/2081228583 reap_dead start 2024-02-19T21:09:00.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 shutdown_connections 2024-02-19T21:09:00.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 >> 172.21.15.163:0/2081228583 conn(0x7fb8e80f24b0 msgr2=0x7fb8e8107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:00.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 shutdown_connections 2024-02-19T21:09:00.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 1 -- 172.21.15.163:0/2081228583 wait complete. 2024-02-19T21:09:00.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.060+0000 7fb8f007f700 1 librados: shutdown 2024-02-19T21:09:00.066 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:09:00.066 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:09:00.066 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:09:00.066 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:09:00.066 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:09:00.066 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:09:00.066 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:09:00.066 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:09:00.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.247+0000 7efe4d7fa700 1 -- 172.21.15.163:0/1817000787 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v360) v1 ==== 113+0+0 (unknown 1426144895 0 0) 0x7efe3804a410 con 0x7efe500f9530 2024-02-19T21:09:00.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.255+0000 7efe56137700 10 librados: watch_flush enter 2024-02-19T21:09:00.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.255+0000 7efe56137700 10 librados: watch_flush exit 2024-02-19T21:09:00.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.255+0000 7efe56137700 20 client.49673.objecter shutdown clearing up homeless session... 2024-02-19T21:09:00.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.255+0000 7efe56137700 10 client.49673.objecter successfully canceled tick 2024-02-19T21:09:00.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.255+0000 7efe56137700 1 -- 172.21.15.163:0/1817000787 >> v1:172.21.15.163:6800/1552514681 conn(0x7efe3c068090 legacy=0x7efe3c06a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:00.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.255+0000 7efe56137700 1 -- 172.21.15.163:0/1817000787 >> v1:172.21.15.163:6790/0 conn(0x7efe500f9530 legacy=0x7efe500fff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:00.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.255+0000 7efe56137700 1 -- 172.21.15.163:0/1817000787 shutdown_connections 2024-02-19T21:09:00.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.255+0000 7efe56137700 1 -- 172.21.15.163:0/1817000787 >> 172.21.15.163:0/1817000787 conn(0x7efe500f2470 msgr2=0x7efe50107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:00.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.255+0000 7efe56137700 1 -- 172.21.15.163:0/1817000787 shutdown_connections 2024-02-19T21:09:00.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.255+0000 7efe56137700 1 -- 172.21.15.163:0/1817000787 wait complete. 2024-02-19T21:09:00.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:00.255+0000 7efe56137700 1 librados: shutdown 2024-02-19T21:09:00.254 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.1 to 1 (10000) 2024-02-19T21:09:00.568 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:09:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:09:00.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:08:59.191751+0000 mgr.y (mgr.14550) 331 : cluster [DBG] pgmap v509: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:00.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:59.599364+0000 mon.a (mon.0) 2325 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:00.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:08:59.910207+0000 mon.c (mon.2) 363 : audit [DBG] from='client.? v1:172.21.15.163:0/2081228583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:00.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:00.245172+0000 mon.a (mon.0) 2326 : audit [INF] from='client.49673 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-19T21:09:00.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:00.245242+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2024-02-19T21:09:00.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:08:59.191751+0000 mgr.y (mgr.14550) 331 : cluster [DBG] pgmap v509: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:00.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:59.599364+0000 mon.a (mon.0) 2325 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:00.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:08:59.910207+0000 mon.c (mon.2) 363 : audit [DBG] from='client.? v1:172.21.15.163:0/2081228583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:00.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:00.245172+0000 mon.a (mon.0) 2326 : audit [INF] from='client.49673 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-19T21:09:00.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:00.245242+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2024-02-19T21:09:01.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:08:59.191751+0000 mgr.y (mgr.14550) 331 : cluster [DBG] pgmap v509: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:01.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:59.599364+0000 mon.a (mon.0) 2325 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:01.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:08:59.910207+0000 mon.c (mon.2) 363 : audit [DBG] from='client.? v1:172.21.15.163:0/2081228583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:01.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:00.245172+0000 mon.a (mon.0) 2326 : audit [INF] from='client.49673 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-19T21:09:01.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:00.245242+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2024-02-19T21:09:01.123 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 2 1 2024-02-19T21:09:01.200 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:09:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:01.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:09:01.307 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:01.333 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:01.540 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:01.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:01.646 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:01.674 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:01.746 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:01.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:00.318224+0000 mgr.y (mgr.14550) 332 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:01.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:01.247390+0000 mon.a (mon.0) 2328 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2024-02-19T21:09:01.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:01.250456+0000 mon.c (mon.2) 364 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:09:01.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:01.251036+0000 mon.a (mon.0) 2329 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:09:01.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:00.318224+0000 mgr.y (mgr.14550) 332 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:01.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:01.247390+0000 mon.a (mon.0) 2328 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2024-02-19T21:09:01.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:01.250456+0000 mon.c (mon.2) 364 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:09:01.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:01.251036+0000 mon.a (mon.0) 2329 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:09:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:01.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:01.851 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:09:01.878 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:02.001 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:09:02.005 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:02.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:09:02.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:00.318224+0000 mgr.y (mgr.14550) 332 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:02.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:01.247390+0000 mon.a (mon.0) 2328 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2024-02-19T21:09:02.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:01.250456+0000 mon.c (mon.2) 364 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:09:02.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:01.251036+0000 mon.a (mon.0) 2329 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:09:02.106 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:02.133 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:02.160 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:02.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:02.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:02.289 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:02.323 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:02.343 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:02.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:02.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:02.447 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:09:02.475 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:02.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:01.192176+0000 mgr.y (mgr.14550) 333 : cluster [DBG] pgmap v512: 236 pgs: 8 creating+peering, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-19T21:09:02.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:02.248970+0000 mon.a (mon.0) 2330 : audit [INF] from='client.49661 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi163-83854-62"}]': finished 2024-02-19T21:09:02.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:02.249039+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2024-02-19T21:09:02.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:02.249865+0000 mon.c (mon.2) 365 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:09:02.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:02.250423+0000 mon.a (mon.0) 2332 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:09:02.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:01.192176+0000 mgr.y (mgr.14550) 333 : cluster [DBG] pgmap v512: 236 pgs: 8 creating+peering, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-19T21:09:02.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:02.248970+0000 mon.a (mon.0) 2330 : audit [INF] from='client.49661 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi163-83854-62"}]': finished 2024-02-19T21:09:02.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:02.249039+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2024-02-19T21:09:02.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:02.249865+0000 mon.c (mon.2) 365 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:09:02.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:02.250423+0000 mon.a (mon.0) 2332 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:09:03.000 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:03.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:01.192176+0000 mgr.y (mgr.14550) 333 : cluster [DBG] pgmap v512: 236 pgs: 8 creating+peering, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-19T21:09:03.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:02.248970+0000 mon.a (mon.0) 2330 : audit [INF] from='client.49661 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi163-83854-62"}]': finished 2024-02-19T21:09:03.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:02.249039+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2024-02-19T21:09:03.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:02.249865+0000 mon.c (mon.2) 365 : audit [INF] from='client.? v1:172.21.15.163:0/1584546626' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:09:03.068 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:02.250423+0000 mon.a (mon.0) 2332 : audit [INF] from='client.49661 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi163-83854-62"}]: dispatch 2024-02-19T21:09:03.494 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:09:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:09:03] "GET /metrics HTTP/1.1" 200 191543 "" "Prometheus/2.33.4" 2024-02-19T21:09:03.515 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:09:03.684 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:09:03.684 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:09:04.020 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:09:04.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:03.192792+0000 mgr.y (mgr.14550) 334 : cluster [DBG] pgmap v515: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-19T21:09:04.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:03.248870+0000 mon.a (mon.0) 2333 : audit [INF] from='client.49661 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi163-83854-62"}]': finished 2024-02-19T21:09:04.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:03.248948+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2024-02-19T21:09:04.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:03.270759+0000 mon.c (mon.2) 366 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:03.271105+0000 mon.a (mon.0) 2335 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:03.272120+0000 mon.c (mon.2) 367 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:03.272385+0000 mon.a (mon.0) 2336 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:03.272764+0000 mon.c (mon.2) 368 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi163-83854-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:09:04.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:03.272964+0000 mon.a (mon.0) 2337 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi163-83854-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:09:04.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:04.241226+0000 mon.a (mon.0) 2338 : audit [INF] from='client.49685 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi163-83854-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:09:04.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:04.241313+0000 mon.a (mon.0) 2339 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2024-02-19T21:09:04.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:04.243276+0000 mon.c (mon.2) 369 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi163-83854-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:04.243602+0000 mon.a (mon.0) 2340 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi163-83854-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:03.192792+0000 mgr.y (mgr.14550) 334 : cluster [DBG] pgmap v515: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-19T21:09:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:03.248870+0000 mon.a (mon.0) 2333 : audit [INF] from='client.49661 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi163-83854-62"}]': finished 2024-02-19T21:09:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:03.248948+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2024-02-19T21:09:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:03.270759+0000 mon.c (mon.2) 366 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:03.271105+0000 mon.a (mon.0) 2335 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:03.272120+0000 mon.c (mon.2) 367 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:03.272385+0000 mon.a (mon.0) 2336 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:03.272764+0000 mon.c (mon.2) 368 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi163-83854-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:09:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:03.272964+0000 mon.a (mon.0) 2337 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi163-83854-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:09:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:04.241226+0000 mon.a (mon.0) 2338 : audit [INF] from='client.49685 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi163-83854-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:09:04.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:04.241313+0000 mon.a (mon.0) 2339 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2024-02-19T21:09:04.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:04.243276+0000 mon.c (mon.2) 369 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi163-83854-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:04.243602+0000 mon.a (mon.0) 2340 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi163-83854-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.520 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-19T21:09:04.528 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:09:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:04.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:09:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:03.192792+0000 mgr.y (mgr.14550) 334 : cluster [DBG] pgmap v515: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-19T21:09:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:03.248870+0000 mon.a (mon.0) 2333 : audit [INF] from='client.49661 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi163-83854-62"}]': finished 2024-02-19T21:09:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:03.248948+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2024-02-19T21:09:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:03.270759+0000 mon.c (mon.2) 366 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:03.271105+0000 mon.a (mon.0) 2335 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:03.272120+0000 mon.c (mon.2) 367 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:03.272385+0000 mon.a (mon.0) 2336 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:03.272764+0000 mon.c (mon.2) 368 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi163-83854-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:09:04.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:03.272964+0000 mon.a (mon.0) 2337 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi163-83854-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:09:04.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:04.241226+0000 mon.a (mon.0) 2338 : audit [INF] from='client.49685 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi163-83854-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:09:04.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:04.241313+0000 mon.a (mon.0) 2339 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2024-02-19T21:09:04.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:04.243276+0000 mon.c (mon.2) 369 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi163-83854-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:04.243602+0000 mon.a (mon.0) 2340 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi163-83854-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:04.632 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:04.658 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:05.307 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:05.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:05.335 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:05.411 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:05.442 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:05.463 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:05.466 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:05.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:05.567 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:05.596 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:05.603 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:05.616 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:05.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:05.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:05.720 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:05.749 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:05.851 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:05.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:05.956 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:05.983 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:06.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:06.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:06.160 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:06.188 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:06.207 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:06.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:06.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.269+0000 7f2fed4cd700 1 -- 172.21.15.163:0/2210908297 >> v1:172.21.15.163:6790/0 conn(0x7f2fe81042c0 legacy=0x7f2fe8106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:06.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.269+0000 7f2fed4cd700 1 -- 172.21.15.163:0/2210908297 shutdown_connections 2024-02-19T21:09:06.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.269+0000 7f2fed4cd700 1 -- 172.21.15.163:0/2210908297 >> 172.21.15.163:0/2210908297 conn(0x7f2fe80f35a0 msgr2=0x7f2fe80f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:06.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.269+0000 7f2fed4cd700 1 -- 172.21.15.163:0/2210908297 shutdown_connections 2024-02-19T21:09:06.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.269+0000 7f2fed4cd700 1 -- 172.21.15.163:0/2210908297 wait complete. 2024-02-19T21:09:06.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.269+0000 7f2fed4cd700 1 librados: starting msgr at 2024-02-19T21:09:06.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fed4cd700 1 librados: starting objecter 2024-02-19T21:09:06.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fed4cd700 1 Processor -- start 2024-02-19T21:09:06.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fed4cd700 1 -- start start 2024-02-19T21:09:06.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fed4cd700 1 librados: setting wanted keys 2024-02-19T21:09:06.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fed4cd700 1 librados: calling monclient init 2024-02-19T21:09:06.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fed4cd700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2fe8117480 con 0x7f2fe8100780 2024-02-19T21:09:06.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fed4cd700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2fe8118680 con 0x7f2fe81042c0 2024-02-19T21:09:06.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fed4cd700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2fe8119880 con 0x7f2fe80fd980 2024-02-19T21:09:06.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fe6ffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f2fe80fd980 0x7f2fe80f9ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:54594/0 (socket says 172.21.15.163:54594) 2024-02-19T21:09:06.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fe6ffd700 1 -- 172.21.15.163:0/379362146 learned_addr learned my addr 172.21.15.163:0/379362146 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:06.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fdffff700 10 client.?.objecter ms_handle_connect 0x7f2fe80fd980 2024-02-19T21:09:06.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fdffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:06.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fdffff700 10 client.?.objecter ms_handle_connect 0x7f2fe8100780 2024-02-19T21:09:06.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fdffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:06.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fdffff700 10 client.?.objecter ms_handle_connect 0x7f2fe81042c0 2024-02-19T21:09:06.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fdffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:06.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063222478 0 0) 0x7f2fe8119880 con 0x7f2fe80fd980 2024-02-19T21:09:06.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd40033f0 con 0x7f2fe80fd980 2024-02-19T21:09:06.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280788436 0 0) 0x7f2fe8117480 con 0x7f2fe8100780 2024-02-19T21:09:06.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fe8119880 con 0x7f2fe8100780 2024-02-19T21:09:06.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322218102 0 0) 0x7f2fe8118680 con 0x7f2fe81042c0 2024-02-19T21:09:06.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.270+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fe8117480 con 0x7f2fe81042c0 2024-02-19T21:09:06.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2226273413 0 0) 0x7f2fe8117480 con 0x7f2fe81042c0 2024-02-19T21:09:06.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fe8118680 con 0x7f2fe81042c0 2024-02-19T21:09:06.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2fe0001ec0 con 0x7f2fe81042c0 2024-02-19T21:09:06.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 803065713 0 0) 0x7f2fd40033f0 con 0x7f2fe80fd980 2024-02-19T21:09:06.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fe8117480 con 0x7f2fe80fd980 2024-02-19T21:09:06.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2fd8001d50 con 0x7f2fe80fd980 2024-02-19T21:09:06.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2606371656 0 0) 0x7f2fe8119880 con 0x7f2fe8100780 2024-02-19T21:09:06.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fd40033f0 con 0x7f2fe8100780 2024-02-19T21:09:06.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2fd0001840 con 0x7f2fe8100780 2024-02-19T21:09:06.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2152993374 0 0) 0x7f2fe8118680 con 0x7f2fe81042c0 2024-02-19T21:09:06.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 >> v1:172.21.15.163:6790/0 conn(0x7f2fe80fd980 legacy=0x7f2fe80f9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:06.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 >> v1:172.21.15.163:6789/0 conn(0x7f2fe8100780 legacy=0x7f2fe80fa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:06.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe811aa80 con 0x7f2fe81042c0 2024-02-19T21:09:06.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fed4cd700 1 -- 172.21.15.163:0/379362146 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe8117670 con 0x7f2fe81042c0 2024-02-19T21:09:06.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fed4cd700 10 client.48508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:06.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fed4cd700 1 -- 172.21.15.163:0/379362146 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe8117ae0 con 0x7f2fe81042c0 2024-02-19T21:09:06.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fed4cd700 1 librados: init done 2024-02-19T21:09:06.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f2fe0002200 con 0x7f2fe81042c0 2024-02-19T21:09:06.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.271+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2fe0003670 con 0x7f2fe81042c0 2024-02-19T21:09:06.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.272+0000 7f2fed4cd700 1 -- 172.21.15.163:0/379362146 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc8004f20 con 0x7f2fe81042c0 2024-02-19T21:09:06.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.272+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f2fe00017b0 con 0x7f2fe81042c0 2024-02-19T21:09:06.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.273+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(366..366 src has 43..366) v4 ==== 8726+0+0 (unknown 3314456812 0 0) 0x7f2fe0076aa0 con 0x7f2fe81042c0 2024-02-19T21:09:06.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.273+0000 7f2fdffff700 10 client.48508.objecter ms_dispatch 0x7f2fe8001550 osd_map(366..366 src has 43..366) v4 2024-02-19T21:09:06.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.273+0000 7f2fdffff700 3 client.48508.objecter handle_osd_map got epochs [366,366] > 0 2024-02-19T21:09:06.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.273+0000 7f2fdffff700 3 client.48508.objecter handle_osd_map decoding full epoch 366 2024-02-19T21:09:06.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.273+0000 7f2fdffff700 20 client.48508.objecter dump_active .. 0 homeless 2024-02-19T21:09:06.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.273+0000 7f2fdffff700 10 client.48508.objecter ms_handle_connect 0x7f2fd4063c70 2024-02-19T21:09:06.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.277+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f2fe0049c00 con 0x7f2fe81042c0 2024-02-19T21:09:06.311 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:06.338 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:06.357 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:06.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:06.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:06.460+0000 7f2fed4cd700 1 -- 172.21.15.163:0/379362146 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7f2fc80051d0 con 0x7f2fe81042c0 2024-02-19T21:09:06.461 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:06.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:05.193688+0000 mgr.y (mgr.14550) 335 : cluster [DBG] pgmap v518: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:09:06.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:05.487982+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2024-02-19T21:09:06.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:05.605490+0000 mon.b (mon.1) 530 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:06.489 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:05.606065+0000 mon.a (mon.0) 2342 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:06.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:05.193688+0000 mgr.y (mgr.14550) 335 : cluster [DBG] pgmap v518: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:09:06.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:05.487982+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2024-02-19T21:09:06.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:05.605490+0000 mon.b (mon.1) 530 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:06.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:05.606065+0000 mon.a (mon.0) 2342 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:06.492 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:06.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:05.193688+0000 mgr.y (mgr.14550) 335 : cluster [DBG] pgmap v518: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:09:06.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:05.487982+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2024-02-19T21:09:06.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:05.605490+0000 mon.b (mon.1) 530 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:06.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:05.606065+0000 mon.a (mon.0) 2342 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:07.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:07.255+0000 7f2fdffff700 1 -- 172.21.15.163:0/379362146 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v367) v1 ==== 113+0+0 (unknown 1831108199 0 0) 0x7f2fe0048070 con 0x7f2fe81042c0 2024-02-19T21:09:07.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:07.259+0000 7f2fed4cd700 10 librados: watch_flush enter 2024-02-19T21:09:07.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:07.259+0000 7f2fed4cd700 10 librados: watch_flush exit 2024-02-19T21:09:07.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:07.259+0000 7f2fed4cd700 20 client.48508.objecter shutdown clearing up homeless session... 2024-02-19T21:09:07.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:07.259+0000 7f2fed4cd700 10 client.48508.objecter successfully canceled tick 2024-02-19T21:09:07.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:07.259+0000 7f2fed4cd700 1 -- 172.21.15.163:0/379362146 >> v1:172.21.15.163:6800/1552514681 conn(0x7f2fd4063c70 legacy=0x7f2fd4066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:07.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:07.259+0000 7f2fed4cd700 1 -- 172.21.15.163:0/379362146 >> v1:172.21.15.193:6789/0 conn(0x7f2fe81042c0 legacy=0x7f2fe8115b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:07.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:07.259+0000 7f2fed4cd700 1 -- 172.21.15.163:0/379362146 shutdown_connections 2024-02-19T21:09:07.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:07.259+0000 7f2fed4cd700 1 -- 172.21.15.163:0/379362146 >> 172.21.15.163:0/379362146 conn(0x7f2fe80f35a0 msgr2=0x7f2fe8107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:07.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:07.259+0000 7f2fed4cd700 1 -- 172.21.15.163:0/379362146 shutdown_connections 2024-02-19T21:09:07.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:07.260+0000 7f2fed4cd700 1 -- 172.21.15.163:0/379362146 wait complete. 2024-02-19T21:09:07.257 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:07.261+0000 7f2fed4cd700 1 librados: shutdown 2024-02-19T21:09:07.259 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.2 to 1 (10000) 2024-02-19T21:09:07.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:06.248288+0000 mon.a (mon.0) 2343 : audit [INF] from='client.49685 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi163-83854-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi163-83854-63"}]': finished 2024-02-19T21:09:07.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:06.248381+0000 mon.a (mon.0) 2344 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:09:07.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:06.248442+0000 mon.a (mon.0) 2345 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2024-02-19T21:09:07.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:06.249627+0000 mon.b (mon.1) 531 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:09:07.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:06.249964+0000 mon.a (mon.0) 2346 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:09:07.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:06.461505+0000 mon.b (mon.1) 532 : audit [INF] from='client.? v1:172.21.15.163:0/379362146' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:09:07.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:06.461807+0000 mon.a (mon.0) 2347 : audit [INF] from='client.48508 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:09:07.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:06.248288+0000 mon.a (mon.0) 2343 : audit [INF] from='client.49685 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi163-83854-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi163-83854-63"}]': finished 2024-02-19T21:09:07.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:06.248381+0000 mon.a (mon.0) 2344 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:09:07.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:06.248442+0000 mon.a (mon.0) 2345 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2024-02-19T21:09:07.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:06.249627+0000 mon.b (mon.1) 531 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:09:07.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:06.249964+0000 mon.a (mon.0) 2346 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:09:07.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:06.461505+0000 mon.b (mon.1) 532 : audit [INF] from='client.? v1:172.21.15.163:0/379362146' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:09:07.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:06.461807+0000 mon.a (mon.0) 2347 : audit [INF] from='client.48508 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:09:07.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:06.248288+0000 mon.a (mon.0) 2343 : audit [INF] from='client.49685 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi163-83854-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi163-83854-63"}]': finished 2024-02-19T21:09:07.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:06.248381+0000 mon.a (mon.0) 2344 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:09:07.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:06.248442+0000 mon.a (mon.0) 2345 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2024-02-19T21:09:07.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:06.249627+0000 mon.b (mon.1) 531 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:09:07.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:06.249964+0000 mon.a (mon.0) 2346 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:09:07.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:06.461505+0000 mon.b (mon.1) 532 : audit [INF] from='client.? v1:172.21.15.163:0/379362146' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:09:07.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:06.461807+0000 mon.a (mon.0) 2347 : audit [INF] from='client.48508 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:09:08.068 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:09:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:09:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:09:08.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:07.194122+0000 mgr.y (mgr.14550) 336 : cluster [DBG] pgmap v521: 236 pgs: 8 unknown, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:09:08.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:07.243395+0000 mon.a (mon.0) 2348 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:09:08.384 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:07.254078+0000 mon.a (mon.0) 2349 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47"}]': finished 2024-02-19T21:09:08.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:07.254269+0000 mon.a (mon.0) 2350 : audit [INF] from='client.48508 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-19T21:09:08.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:07.254337+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2024-02-19T21:09:08.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:07.274958+0000 mon.b (mon.1) 533 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:08.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:07.275306+0000 mon.a (mon.0) 2352 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:08.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:07.275811+0000 mon.b (mon.1) 534 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:09:08.385 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:07.275990+0000 mon.a (mon.0) 2353 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:09:08.385 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:07.194122+0000 mgr.y (mgr.14550) 336 : cluster [DBG] pgmap v521: 236 pgs: 8 unknown, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:09:08.385 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:07.243395+0000 mon.a (mon.0) 2348 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:09:08.385 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:07.254078+0000 mon.a (mon.0) 2349 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47"}]': finished 2024-02-19T21:09:08.385 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:07.254269+0000 mon.a (mon.0) 2350 : audit [INF] from='client.48508 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-19T21:09:08.385 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:07.254337+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2024-02-19T21:09:08.386 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:07.274958+0000 mon.b (mon.1) 533 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:08.386 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:07.275306+0000 mon.a (mon.0) 2352 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:08.386 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:07.275811+0000 mon.b (mon.1) 534 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:09:08.386 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:07.275990+0000 mon.a (mon.0) 2353 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:09:08.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:07.194122+0000 mgr.y (mgr.14550) 336 : cluster [DBG] pgmap v521: 236 pgs: 8 unknown, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:09:08.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:07.243395+0000 mon.a (mon.0) 2348 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:09:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:07.254078+0000 mon.a (mon.0) 2349 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47"}]': finished 2024-02-19T21:09:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:07.254269+0000 mon.a (mon.0) 2350 : audit [INF] from='client.48508 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-19T21:09:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:07.254337+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2024-02-19T21:09:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:07.274958+0000 mon.b (mon.1) 533 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:07.275306+0000 mon.a (mon.0) 2352 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:07.275811+0000 mon.b (mon.1) 534 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:09:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:07.275990+0000 mon.a (mon.0) 2353 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-47"}]: dispatch 2024-02-19T21:09:08.712 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:08.744 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:08.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:08.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:08.850 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:08.880 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:09.065 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:09:09.065 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:09:09.228 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:09.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:09.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:09.333 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:09.361 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:09.381 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:09.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:09.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:09.485 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:09.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:08.257015+0000 mon.a (mon.0) 2354 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2024-02-19T21:09:09.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:08.264601+0000 mon.c (mon.2) 370 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:09.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:08.265118+0000 mon.a (mon.0) 2355 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:09.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:09.101100+0000 mon.a (mon.0) 2356 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:09.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:08.257015+0000 mon.a (mon.0) 2354 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2024-02-19T21:09:09.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:08.264601+0000 mon.c (mon.2) 370 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:09.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:08.265118+0000 mon.a (mon.0) 2355 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:09.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:09.101100+0000 mon.a (mon.0) 2356 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:09.513 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:09.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:08.257015+0000 mon.a (mon.0) 2354 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2024-02-19T21:09:09.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:08.264601+0000 mon.c (mon.2) 370 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:09.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:08.265118+0000 mon.a (mon.0) 2355 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:09.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:09.101100+0000 mon.a (mon.0) 2356 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:10.063 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:10.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:10.167 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:10.198 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:10.220 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:10.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:10.323 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:09.194658+0000 mgr.y (mgr.14550) 337 : cluster [DBG] pgmap v524: 196 pgs: 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:10.323 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:09.258996+0000 mon.a (mon.0) 2357 : audit [INF] from='client.49685 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi163-83854-63"}]': finished 2024-02-19T21:09:10.323 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:09.259101+0000 mon.a (mon.0) 2358 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2024-02-19T21:09:10.324 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:09.262167+0000 mon.b (mon.1) 535 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:10.324 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:09.283595+0000 mon.c (mon.2) 371 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:10.324 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:09.283932+0000 mon.a (mon.0) 2359 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:10.324 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:09.304209+0000 mon.a (mon.0) 2360 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:10.324 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:10.260354+0000 mon.a (mon.0) 2361 : audit [INF] from='client.49685 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi163-83854-63"}]': finished 2024-02-19T21:09:10.324 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:10.260438+0000 mon.a (mon.0) 2362 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:10.324 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:10.260536+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2024-02-19T21:09:10.324 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:09.194658+0000 mgr.y (mgr.14550) 337 : cluster [DBG] pgmap v524: 196 pgs: 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:10.324 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:09.258996+0000 mon.a (mon.0) 2357 : audit [INF] from='client.49685 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi163-83854-63"}]': finished 2024-02-19T21:09:10.324 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:09.259101+0000 mon.a (mon.0) 2358 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2024-02-19T21:09:10.324 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:09.262167+0000 mon.b (mon.1) 535 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:10.325 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:09.283595+0000 mon.c (mon.2) 371 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:10.325 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:09.283932+0000 mon.a (mon.0) 2359 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:10.325 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:09.304209+0000 mon.a (mon.0) 2360 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:10.325 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:10.260354+0000 mon.a (mon.0) 2361 : audit [INF] from='client.49685 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi163-83854-63"}]': finished 2024-02-19T21:09:10.325 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:10.260438+0000 mon.a (mon.0) 2362 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:10.325 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:10.260536+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2024-02-19T21:09:10.325 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:10.352 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:10.371 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:10.373 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:10.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:10.475 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:09:10.500 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:09:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:09:10.500 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:09.194658+0000 mgr.y (mgr.14550) 337 : cluster [DBG] pgmap v524: 196 pgs: 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:10.500 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:09.258996+0000 mon.a (mon.0) 2357 : audit [INF] from='client.49685 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi163-83854-63"}]': finished 2024-02-19T21:09:10.500 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:09.259101+0000 mon.a (mon.0) 2358 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2024-02-19T21:09:10.500 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:09.262167+0000 mon.b (mon.1) 535 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:10.500 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:09.283595+0000 mon.c (mon.2) 371 : audit [INF] from='client.? v1:172.21.15.163:0/2729458884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:10.500 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:09.283932+0000 mon.a (mon.0) 2359 : audit [INF] from='client.49685 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi163-83854-63"}]: dispatch 2024-02-19T21:09:10.501 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:09.304209+0000 mon.a (mon.0) 2360 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:10.501 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:10.260354+0000 mon.a (mon.0) 2361 : audit [INF] from='client.49685 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi163-83854-63"}]': finished 2024-02-19T21:09:10.501 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:10.260438+0000 mon.a (mon.0) 2362 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:10.501 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:10.260536+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2024-02-19T21:09:10.501 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:10.520 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:09:10.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:10.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:09:10.623 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:10.649 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:10.668 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:10.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:10.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:10.772 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:10.800 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:10.949 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:10.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:10.955 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 5 1 2024-02-19T21:09:11.053 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:11.083 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:11.281 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:11.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:11.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:11.347 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:10.287167+0000 mon.b (mon.1) 536 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:09:11.348 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:10.287410+0000 mon.a (mon.0) 2364 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:09:11.348 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:10.291967+0000 mon.a (mon.0) 2365 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi163-83854-64"}]: dispatch 2024-02-19T21:09:11.348 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:10.292971+0000 mon.a (mon.0) 2366 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi163-83854-64"}]: dispatch 2024-02-19T21:09:11.348 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:10.293226+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi163-83854-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:09:11.348 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:10.329331+0000 mgr.y (mgr.14550) 338 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:11.348 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:10.682695+0000 mon.c (mon.2) 372 : audit [DBG] from='client.? v1:172.21.15.163:0/2941242467' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:11.348 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:11.264034+0000 mon.a (mon.0) 2368 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:09:11.348 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:11.264121+0000 mon.a (mon.0) 2369 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi163-83854-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:09:11.348 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:11.264175+0000 mon.a (mon.0) 2370 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2024-02-19T21:09:11.349 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:11.265934+0000 mon.b (mon.1) 537 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:11.349 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:11.266263+0000 mon.a (mon.0) 2371 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:11.349 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:11.266623+0000 mon.a (mon.0) 2372 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi163-83854-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi163-83854-64"}]: dispatch 2024-02-19T21:09:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:10.287167+0000 mon.b (mon.1) 536 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:09:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:10.287410+0000 mon.a (mon.0) 2364 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:09:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:10.291967+0000 mon.a (mon.0) 2365 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi163-83854-64"}]: dispatch 2024-02-19T21:09:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:10.292971+0000 mon.a (mon.0) 2366 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi163-83854-64"}]: dispatch 2024-02-19T21:09:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:10.293226+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi163-83854-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:09:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:10.329331+0000 mgr.y (mgr.14550) 338 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:10.682695+0000 mon.c (mon.2) 372 : audit [DBG] from='client.? v1:172.21.15.163:0/2941242467' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:11.349 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:11.264034+0000 mon.a (mon.0) 2368 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:09:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:11.264121+0000 mon.a (mon.0) 2369 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi163-83854-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:09:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:11.264175+0000 mon.a (mon.0) 2370 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2024-02-19T21:09:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:11.265934+0000 mon.b (mon.1) 537 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:11.266263+0000 mon.a (mon.0) 2371 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:11.350 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:11.266623+0000 mon.a (mon.0) 2372 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi163-83854-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi163-83854-64"}]: dispatch 2024-02-19T21:09:11.385 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:11.415 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:11.434 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:11.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:11.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:11.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.679+0000 7f07abbb8700 1 -- 172.21.15.163:0/3875293324 >> v1:172.21.15.163:6790/0 conn(0x7f07a406b0f0 legacy=0x7f07a406b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:11.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.679+0000 7f07abbb8700 1 -- 172.21.15.163:0/3875293324 shutdown_connections 2024-02-19T21:09:11.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.679+0000 7f07abbb8700 1 -- 172.21.15.163:0/3875293324 >> 172.21.15.163:0/3875293324 conn(0x7f07a40f3590 msgr2=0x7f07a40f59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:11.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.679+0000 7f07abbb8700 1 -- 172.21.15.163:0/3875293324 shutdown_connections 2024-02-19T21:09:11.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.679+0000 7f07abbb8700 1 -- 172.21.15.163:0/3875293324 wait complete. 2024-02-19T21:09:11.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.679+0000 7f07abbb8700 1 librados: starting msgr at 2024-02-19T21:09:11.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.679+0000 7f07abbb8700 1 librados: starting objecter 2024-02-19T21:09:11.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.679+0000 7f07abbb8700 1 Processor -- start 2024-02-19T21:09:11.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.679+0000 7f07abbb8700 1 -- start start 2024-02-19T21:09:11.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.679+0000 7f07abbb8700 1 librados: setting wanted keys 2024-02-19T21:09:11.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.679+0000 7f07abbb8700 1 librados: calling monclient init 2024-02-19T21:09:11.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f07abbb8700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f07a41194c0 con 0x7f07a406b0f0 2024-02-19T21:09:11.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f07abbb8700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f07a411a6a0 con 0x7f07a406bbe0 2024-02-19T21:09:11.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f07abbb8700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f07a411b8a0 con 0x7f07a4106500 2024-02-19T21:09:11.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f07a9954700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f07a406b0f0 0x7f07a4102810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:47416/0 (socket says 172.21.15.163:47416) 2024-02-19T21:09:11.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f07a9954700 1 -- 172.21.15.163:0/2941242467 learned_addr learned my addr 172.21.15.163:0/2941242467 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:11.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 10 client.?.objecter ms_handle_connect 0x7f07a406b0f0 2024-02-19T21:09:11.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:11.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 10 client.?.objecter ms_handle_connect 0x7f07a4106500 2024-02-19T21:09:11.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:11.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 10 client.?.objecter ms_handle_connect 0x7f07a406bbe0 2024-02-19T21:09:11.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:11.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565182715 0 0) 0x7f07a41194c0 con 0x7f07a406b0f0 2024-02-19T21:09:11.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07900033f0 con 0x7f07a406b0f0 2024-02-19T21:09:11.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209043842 0 0) 0x7f07a411b8a0 con 0x7f07a4106500 2024-02-19T21:09:11.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a41194c0 con 0x7f07a4106500 2024-02-19T21:09:11.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707278894 0 0) 0x7f07a411a6a0 con 0x7f07a406bbe0 2024-02-19T21:09:11.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a411b8a0 con 0x7f07a406bbe0 2024-02-19T21:09:11.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4234045270 0 0) 0x7f07a41194c0 con 0x7f07a4106500 2024-02-19T21:09:11.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07a411a6a0 con 0x7f07a4106500 2024-02-19T21:09:11.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1925766664 0 0) 0x7f07900033f0 con 0x7f07a406b0f0 2024-02-19T21:09:11.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07a41194c0 con 0x7f07a406b0f0 2024-02-19T21:09:11.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f079c001c40 con 0x7f07a4106500 2024-02-19T21:09:11.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f07a0000ff0 con 0x7f07a406b0f0 2024-02-19T21:09:11.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 896769286 0 0) 0x7f07a411b8a0 con 0x7f07a406bbe0 2024-02-19T21:09:11.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07900033f0 con 0x7f07a406bbe0 2024-02-19T21:09:11.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0794001600 con 0x7f07a406bbe0 2024-02-19T21:09:11.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 544543841 0 0) 0x7f07a411a6a0 con 0x7f07a4106500 2024-02-19T21:09:11.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 >> v1:172.21.15.193:6789/0 conn(0x7f07a406bbe0 legacy=0x7f07a4102f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:11.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 >> v1:172.21.15.163:6789/0 conn(0x7f07a406b0f0 legacy=0x7f07a4102810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:11.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.680+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a411caa0 con 0x7f07a4106500 2024-02-19T21:09:11.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.681+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a411ba90 con 0x7f07a4106500 2024-02-19T21:09:11.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.681+0000 7f07abbb8700 10 client.49703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:11.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.681+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a411bf00 con 0x7f07a4106500 2024-02-19T21:09:11.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.681+0000 7f07abbb8700 1 librados: init done 2024-02-19T21:09:11.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.681+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f079c002310 con 0x7f07a4106500 2024-02-19T21:09:11.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.681+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f079c003640 con 0x7f07a4106500 2024-02-19T21:09:11.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.681+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0788004f00 con 0x7f07a4106500 2024-02-19T21:09:11.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.682+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f079c001e70 con 0x7f07a4106500 2024-02-19T21:09:11.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.682+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(370..370 src has 43..370) v4 ==== 7902+0+0 (unknown 1878983188 0 0) 0x7f079c0766c0 con 0x7f07a4106500 2024-02-19T21:09:11.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.682+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 osd_map(370..370 src has 43..370) v4 2024-02-19T21:09:11.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.682+0000 7f079affd700 3 client.49703.objecter handle_osd_map got epochs [370,370] > 0 2024-02-19T21:09:11.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.682+0000 7f079affd700 3 client.49703.objecter handle_osd_map decoding full epoch 370 2024-02-19T21:09:11.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.682+0000 7f079affd700 20 client.49703.objecter dump_active .. 0 homeless 2024-02-19T21:09:11.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.682+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v370) v1 ==== 54+0+16 (unknown 3810478085 0 2420000192) 0x7f079c078ac0 con 0x7f07a4106500 2024-02-19T21:09:11.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f079affd700 10 client.49703.objecter ms_handle_connect 0x7f0790063c60 2024-02-19T21:09:11.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f07abbb8700 10 client.49703.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:11.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 1 2024-02-19T21:09:11.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f07880051a0 osd=0 3 2024-02-19T21:09:11.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f07880051a0 osd=0 3 2024-02-19T21:09:11.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f07880051a0 osd=0 3 2024-02-19T21:09:11.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 1 2024-02-19T21:09:11.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f07880051a0 osd=0 3 2024-02-19T21:09:11.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 0 1 2024-02-19T21:09:11.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f07880051a0 osd=0 4 2024-02-19T21:09:11.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f07abbb8700 10 client.49703.objecter _send_command 1 2024-02-19T21:09:11.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0788004950 con 0x7f0788004530 2024-02-19T21:09:11.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f079affd700 10 client.49703.objecter ms_handle_connect 0x7f0788004530 2024-02-19T21:09:11.500 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:09:11.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0788004950 con 0x7f0788004530 2024-02-19T21:09:11.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:09:11.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f079affd700 10 client.49703.objecter _finish_command 1 = system:0 2024-02-19T21:09:11.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f07880051a0 osd=0 4 2024-02-19T21:09:11.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.683+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 0 1 2024-02-19T21:09:11.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.698+0000 7f07abbb8700 10 client.49703.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:11.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.698+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 2 2024-02-19T21:09:11.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.698+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f07880051a0 osd=0 3 2024-02-19T21:09:11.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.698+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f07880051a0 osd=0 3 2024-02-19T21:09:11.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.698+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f07880051a0 osd=0 3 2024-02-19T21:09:11.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.698+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 2 2024-02-19T21:09:11.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.698+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f07880051a0 osd=0 3 2024-02-19T21:09:11.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.698+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 0 2 2024-02-19T21:09:11.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.698+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f07880051a0 osd=0 4 2024-02-19T21:09:11.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.698+0000 7f07abbb8700 10 client.49703.objecter _send_command 2 2024-02-19T21:09:11.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.698+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0788004d30 con 0x7f0788004530 2024-02-19T21:09:11.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f0788004d30 con 0x7f0788004530 2024-02-19T21:09:11.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:11.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f079affd700 10 client.49703.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f07880051a0 osd=0 4 2024-02-19T21:09:11.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 0 2 2024-02-19T21:09:11.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f07abbb8700 10 client.49703.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:11.502 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:09:11.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 3 2024-02-19T21:09:11.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f0788003fe0 osd=1 3 2024-02-19T21:09:11.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f0788003fe0 osd=1 3 2024-02-19T21:09:11.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f0788003fe0 osd=1 3 2024-02-19T21:09:11.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 3 2024-02-19T21:09:11.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f0788003fe0 osd=1 3 2024-02-19T21:09:11.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 1 3 2024-02-19T21:09:11.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f0788003fe0 osd=1 4 2024-02-19T21:09:11.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f07abbb8700 10 client.49703.objecter _send_command 3 2024-02-19T21:09:11.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f078800c000 con 0x7f0788004150 2024-02-19T21:09:11.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.700+0000 7f079affd700 10 client.49703.objecter ms_handle_connect 0x7f0788004150 2024-02-19T21:09:11.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.701+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f078800c000 con 0x7f0788004150 2024-02-19T21:09:11.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.701+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:09:11.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.701+0000 7f079affd700 10 client.49703.objecter _finish_command 3 = system:0 2024-02-19T21:09:11.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.701+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f0788003fe0 osd=1 4 2024-02-19T21:09:11.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.701+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 1 3 2024-02-19T21:09:11.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.715+0000 7f07abbb8700 10 client.49703.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:11.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.715+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 4 2024-02-19T21:09:11.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.715+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f0788003fe0 osd=1 3 2024-02-19T21:09:11.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.715+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f0788003fe0 osd=1 3 2024-02-19T21:09:11.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.715+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f0788003fe0 osd=1 3 2024-02-19T21:09:11.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.715+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 4 2024-02-19T21:09:11.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.715+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f0788003fe0 osd=1 3 2024-02-19T21:09:11.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.715+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 1 4 2024-02-19T21:09:11.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.715+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f0788003fe0 osd=1 4 2024-02-19T21:09:11.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.715+0000 7f07abbb8700 10 client.49703.objecter _send_command 4 2024-02-19T21:09:11.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.715+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f07880018b0 con 0x7f0788004150 2024-02-19T21:09:11.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f07880018b0 con 0x7f0788004150 2024-02-19T21:09:11.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:11.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f079affd700 10 client.49703.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f0788003fe0 osd=1 4 2024-02-19T21:09:11.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 1 4 2024-02-19T21:09:11.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f07abbb8700 10 client.49703.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:11.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 5 2024-02-19T21:09:11.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f0788003630 osd=2 3 2024-02-19T21:09:11.505 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:09:11.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f0788003630 osd=2 3 2024-02-19T21:09:11.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f0788003630 osd=2 3 2024-02-19T21:09:11.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 5 2024-02-19T21:09:11.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f0788003630 osd=2 3 2024-02-19T21:09:11.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 2 5 2024-02-19T21:09:11.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f0788003630 osd=2 4 2024-02-19T21:09:11.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f07abbb8700 10 client.49703.objecter _send_command 5 2024-02-19T21:09:11.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f078800de60 con 0x7f078800d2f0 2024-02-19T21:09:11.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.717+0000 7f079affd700 10 client.49703.objecter ms_handle_connect 0x7f078800d2f0 2024-02-19T21:09:11.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.718+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f078800de60 con 0x7f078800d2f0 2024-02-19T21:09:11.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.718+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:09:11.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.718+0000 7f079affd700 10 client.49703.objecter _finish_command 5 = system:0 2024-02-19T21:09:11.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.718+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f0788003630 osd=2 4 2024-02-19T21:09:11.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.718+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 2 5 2024-02-19T21:09:11.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.732+0000 7f07abbb8700 10 client.49703.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:11.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.732+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 6 2024-02-19T21:09:11.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.732+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f0788003630 osd=2 3 2024-02-19T21:09:11.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.732+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f0788003630 osd=2 3 2024-02-19T21:09:11.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.732+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f0788003630 osd=2 3 2024-02-19T21:09:11.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.732+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 6 2024-02-19T21:09:11.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.732+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f0788003630 osd=2 3 2024-02-19T21:09:11.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.732+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 2 6 2024-02-19T21:09:11.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.732+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f0788003630 osd=2 4 2024-02-19T21:09:11.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.732+0000 7f07abbb8700 10 client.49703.objecter _send_command 6 2024-02-19T21:09:11.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.732+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f07880030a0 con 0x7f078800d2f0 2024-02-19T21:09:11.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.733+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f07880030a0 con 0x7f078800d2f0 2024-02-19T21:09:11.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.733+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:11.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.733+0000 7f079affd700 10 client.49703.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.733+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f0788003630 osd=2 4 2024-02-19T21:09:11.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.733+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 2 6 2024-02-19T21:09:11.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.733+0000 7f07abbb8700 10 client.49703.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:11.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.733+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 7 2024-02-19T21:09:11.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.734+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f07880034b0 osd=3 3 2024-02-19T21:09:11.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.734+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f07880034b0 osd=3 3 2024-02-19T21:09:11.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.734+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f07880034b0 osd=3 3 2024-02-19T21:09:11.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.734+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 7 2024-02-19T21:09:11.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.734+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f07880034b0 osd=3 3 2024-02-19T21:09:11.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.734+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 3 7 2024-02-19T21:09:11.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.734+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f07880034b0 osd=3 4 2024-02-19T21:09:11.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.734+0000 7f07abbb8700 10 client.49703.objecter _send_command 7 2024-02-19T21:09:11.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.734+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0788015b00 con 0x7f0788011e80 2024-02-19T21:09:11.509 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:09:11.509 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:09:11.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.734+0000 7f079affd700 10 client.49703.objecter ms_handle_connect 0x7f0788011e80 2024-02-19T21:09:11.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.734+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0788015b00 con 0x7f0788011e80 2024-02-19T21:09:11.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.734+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:09:11.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.734+0000 7f079affd700 10 client.49703.objecter _finish_command 7 = system:0 2024-02-19T21:09:11.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.734+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f07880034b0 osd=3 4 2024-02-19T21:09:11.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.734+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 3 7 2024-02-19T21:09:11.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.749+0000 7f07abbb8700 10 client.49703.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:11.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.749+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 8 2024-02-19T21:09:11.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.749+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f07880034b0 osd=3 3 2024-02-19T21:09:11.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.749+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f07880034b0 osd=3 3 2024-02-19T21:09:11.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.749+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f07880034b0 osd=3 3 2024-02-19T21:09:11.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.749+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 8 2024-02-19T21:09:11.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.749+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f07880034b0 osd=3 3 2024-02-19T21:09:11.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.749+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 3 8 2024-02-19T21:09:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.749+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f07880034b0 osd=3 4 2024-02-19T21:09:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.749+0000 7f07abbb8700 10 client.49703.objecter _send_command 8 2024-02-19T21:09:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.749+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f078800e240 con 0x7f0788011e80 2024-02-19T21:09:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f078800e240 con 0x7f0788011e80 2024-02-19T21:09:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f079affd700 10 client.49703.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f07880034b0 osd=3 4 2024-02-19T21:09:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 3 8 2024-02-19T21:09:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f07abbb8700 10 client.49703.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 9 2024-02-19T21:09:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f078800e000 osd=4 3 2024-02-19T21:09:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f078800e000 osd=4 3 2024-02-19T21:09:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f078800e000 osd=4 3 2024-02-19T21:09:11.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 9 2024-02-19T21:09:11.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f078800e000 osd=4 3 2024-02-19T21:09:11.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 4 9 2024-02-19T21:09:11.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f078800e000 osd=4 4 2024-02-19T21:09:11.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f07abbb8700 10 client.49703.objecter _send_command 9 2024-02-19T21:09:11.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.750+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f078800e210 con 0x7f078800ce50 2024-02-19T21:09:11.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.751+0000 7f079affd700 10 client.49703.objecter ms_handle_connect 0x7f078800ce50 2024-02-19T21:09:11.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.751+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f078800e210 con 0x7f078800ce50 2024-02-19T21:09:11.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.752+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:09:11.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.752+0000 7f079affd700 10 client.49703.objecter _finish_command 9 = system:0 2024-02-19T21:09:11.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.752+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f078800e000 osd=4 4 2024-02-19T21:09:11.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.752+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 4 9 2024-02-19T21:09:11.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.766+0000 7f07abbb8700 10 client.49703.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:11.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.766+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 10 2024-02-19T21:09:11.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.766+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f078800e000 osd=4 3 2024-02-19T21:09:11.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.766+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f078800e000 osd=4 3 2024-02-19T21:09:11.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.766+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f078800e000 osd=4 3 2024-02-19T21:09:11.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.766+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 10 2024-02-19T21:09:11.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.766+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f078800e000 osd=4 3 2024-02-19T21:09:11.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.766+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 4 10 2024-02-19T21:09:11.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.766+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f078800e000 osd=4 4 2024-02-19T21:09:11.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.766+0000 7f07abbb8700 10 client.49703.objecter _send_command 10 2024-02-19T21:09:11.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.766+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f078800e210 con 0x7f078800ce50 2024-02-19T21:09:11.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.767+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f078800e210 con 0x7f078800ce50 2024-02-19T21:09:11.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.767+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:11.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.767+0000 7f079affd700 10 client.49703.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.767+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f078800e000 osd=4 4 2024-02-19T21:09:11.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.767+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 4 10 2024-02-19T21:09:11.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.768+0000 7f07abbb8700 10 client.49703.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:11.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.768+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 11 2024-02-19T21:09:11.514 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:09:11.514 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:09:11.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.768+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f078800e210 osd=5 3 2024-02-19T21:09:11.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.768+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f078800e210 osd=5 3 2024-02-19T21:09:11.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.768+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f078800e210 osd=5 3 2024-02-19T21:09:11.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.768+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 11 2024-02-19T21:09:11.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.768+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f078800e210 osd=5 3 2024-02-19T21:09:11.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.768+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 5 11 2024-02-19T21:09:11.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.768+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f078800e210 osd=5 4 2024-02-19T21:09:11.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.768+0000 7f07abbb8700 10 client.49703.objecter _send_command 11 2024-02-19T21:09:11.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.768+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f07880180d0 con 0x7f0788017b20 2024-02-19T21:09:11.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.768+0000 7f079affd700 10 client.49703.objecter ms_handle_connect 0x7f0788017b20 2024-02-19T21:09:11.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.769+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f07880180d0 con 0x7f0788017b20 2024-02-19T21:09:11.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.769+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:09:11.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.769+0000 7f079affd700 10 client.49703.objecter _finish_command 11 = system:0 2024-02-19T21:09:11.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.769+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f078800e210 osd=5 4 2024-02-19T21:09:11.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.769+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 5 11 2024-02-19T21:09:11.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.783+0000 7f07abbb8700 10 client.49703.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:11.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.783+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 12 2024-02-19T21:09:11.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.783+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f078800e210 osd=5 3 2024-02-19T21:09:11.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.783+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f078800e210 osd=5 3 2024-02-19T21:09:11.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.783+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f078800e210 osd=5 3 2024-02-19T21:09:11.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.783+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 12 2024-02-19T21:09:11.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.783+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f078800e210 osd=5 3 2024-02-19T21:09:11.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.783+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 5 12 2024-02-19T21:09:11.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.783+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f078800e210 osd=5 4 2024-02-19T21:09:11.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.783+0000 7f07abbb8700 10 client.49703.objecter _send_command 12 2024-02-19T21:09:11.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.783+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f07880180d0 con 0x7f0788017b20 2024-02-19T21:09:11.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f07880180d0 con 0x7f0788017b20 2024-02-19T21:09:11.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:11.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f079affd700 10 client.49703.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f078800e210 osd=5 4 2024-02-19T21:09:11.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 5 12 2024-02-19T21:09:11.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f07abbb8700 10 client.49703.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:11.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 13 2024-02-19T21:09:11.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f07880180d0 osd=6 3 2024-02-19T21:09:11.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f07880180d0 osd=6 3 2024-02-19T21:09:11.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f07880180d0 osd=6 3 2024-02-19T21:09:11.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 13 2024-02-19T21:09:11.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f07880180d0 osd=6 3 2024-02-19T21:09:11.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 6 13 2024-02-19T21:09:11.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f07880180d0 osd=6 4 2024-02-19T21:09:11.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f07abbb8700 10 client.49703.objecter _send_command 13 2024-02-19T21:09:11.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0788022ce0 con 0x7f078801f100 2024-02-19T21:09:11.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.785+0000 7f079affd700 10 client.49703.objecter ms_handle_connect 0x7f078801f100 2024-02-19T21:09:11.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.786+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0788022ce0 con 0x7f078801f100 2024-02-19T21:09:11.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.786+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:09:11.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.786+0000 7f079affd700 10 client.49703.objecter _finish_command 13 = system:0 2024-02-19T21:09:11.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.786+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f07880180d0 osd=6 4 2024-02-19T21:09:11.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.786+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 6 13 2024-02-19T21:09:11.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.800+0000 7f07abbb8700 10 client.49703.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:11.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.800+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 14 2024-02-19T21:09:11.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.800+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f07880180d0 osd=6 3 2024-02-19T21:09:11.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.800+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f07880180d0 osd=6 3 2024-02-19T21:09:11.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.800+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f07880180d0 osd=6 3 2024-02-19T21:09:11.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.800+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 14 2024-02-19T21:09:11.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.800+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f07880180d0 osd=6 3 2024-02-19T21:09:11.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.800+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 6 14 2024-02-19T21:09:11.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.800+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f07880180d0 osd=6 4 2024-02-19T21:09:11.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.800+0000 7f07abbb8700 10 client.49703.objecter _send_command 14 2024-02-19T21:09:11.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.800+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0788024490 con 0x7f078801f100 2024-02-19T21:09:11.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f0788024490 con 0x7f078801f100 2024-02-19T21:09:11.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:11.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f079affd700 10 client.49703.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f07880180d0 osd=6 4 2024-02-19T21:09:11.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 6 14 2024-02-19T21:09:11.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f07abbb8700 10 client.49703.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:11.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 15 2024-02-19T21:09:11.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f0788024490 osd=7 3 2024-02-19T21:09:11.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f0788024490 osd=7 3 2024-02-19T21:09:11.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f0788024490 osd=7 3 2024-02-19T21:09:11.520 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:09:11.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 15 2024-02-19T21:09:11.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f0788024490 osd=7 3 2024-02-19T21:09:11.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 7 15 2024-02-19T21:09:11.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f0788024490 osd=7 4 2024-02-19T21:09:11.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f07abbb8700 10 client.49703.objecter _send_command 15 2024-02-19T21:09:11.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.802+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f07880250b0 con 0x7f0788024b50 2024-02-19T21:09:11.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.803+0000 7f079affd700 10 client.49703.objecter ms_handle_connect 0x7f0788024b50 2024-02-19T21:09:11.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.803+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f07880250b0 con 0x7f0788024b50 2024-02-19T21:09:11.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.803+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:09:11.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.803+0000 7f079affd700 10 client.49703.objecter _finish_command 15 = system:0 2024-02-19T21:09:11.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.803+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f0788024490 osd=7 4 2024-02-19T21:09:11.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.803+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 7 15 2024-02-19T21:09:11.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.819+0000 7f07abbb8700 10 client.49703.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:11.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.819+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign -1 16 2024-02-19T21:09:11.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.819+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f0788024490 osd=7 3 2024-02-19T21:09:11.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.819+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f0788024490 osd=7 3 2024-02-19T21:09:11.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.819+0000 7f07abbb8700 20 client.49703.objecter _get_session s=0x7f0788024490 osd=7 3 2024-02-19T21:09:11.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.819+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_remove -1 16 2024-02-19T21:09:11.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.819+0000 7f07abbb8700 20 client.49703.objecter get_session s=0x7f0788024490 osd=7 3 2024-02-19T21:09:11.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.819+0000 7f07abbb8700 15 client.49703.objecter _session_command_op_assign 7 16 2024-02-19T21:09:11.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.819+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f0788024490 osd=7 4 2024-02-19T21:09:11.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.819+0000 7f07abbb8700 10 client.49703.objecter _send_command 16 2024-02-19T21:09:11.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.819+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f07880250b0 con 0x7f0788024b50 2024-02-19T21:09:11.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.820+0000 7f079affd700 1 -- 172.21.15.163:0/2941242467 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f07880250b0 con 0x7f0788024b50 2024-02-19T21:09:11.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.820+0000 7f079affd700 10 client.49703.objecter ms_dispatch 0x7f07a4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:11.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.820+0000 7f079affd700 10 client.49703.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.820+0000 7f079affd700 20 client.49703.objecter put_session s=0x7f0788024490 osd=7 4 2024-02-19T21:09:11.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.820+0000 7f079affd700 15 client.49703.objecter _session_command_op_remove 7 16 2024-02-19T21:09:11.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 10 librados: watch_flush enter 2024-02-19T21:09:11.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 10 librados: watch_flush exit 2024-02-19T21:09:11.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 10 client.49703.objecter close_session for osd.0 2024-02-19T21:09:11.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 >> v1:172.21.15.163:6801/1559418115 conn(0x7f0788004530 legacy=0x7f0788002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:11.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f07880051a0 osd=0 1 2024-02-19T21:09:11.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 10 client.49703.objecter close_session for osd.1 2024-02-19T21:09:11.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 >> v1:172.21.15.163:6805/3125047432 conn(0x7f0788004150 legacy=0x7f078800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:11.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f0788003fe0 osd=1 1 2024-02-19T21:09:11.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 10 client.49703.objecter close_session for osd.2 2024-02-19T21:09:11.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 >> v1:172.21.15.163:6809/1563158487 conn(0x7f078800d2f0 legacy=0x7f078800d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:11.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f0788003630 osd=2 1 2024-02-19T21:09:11.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 10 client.49703.objecter close_session for osd.3 2024-02-19T21:09:11.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 >> v1:172.21.15.163:6813/2112279769 conn(0x7f0788011e80 legacy=0x7f0788014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:11.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f07880034b0 osd=3 1 2024-02-19T21:09:11.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 10 client.49703.objecter close_session for osd.4 2024-02-19T21:09:11.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 >> v1:172.21.15.193:6800/3129716965 conn(0x7f078800ce50 legacy=0x7f0788015ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:11.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07aa155700 1 -- 172.21.15.163:0/2941242467 reap_dead start 2024-02-19T21:09:11.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f078800e000 osd=4 1 2024-02-19T21:09:11.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 10 client.49703.objecter close_session for osd.5 2024-02-19T21:09:11.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 >> v1:172.21.15.193:6804/805161991 conn(0x7f0788017b20 legacy=0x7f078801d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:11.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f078800e210 osd=5 1 2024-02-19T21:09:11.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 10 client.49703.objecter close_session for osd.6 2024-02-19T21:09:11.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 >> v1:172.21.15.193:6808/787604615 conn(0x7f078801f100 legacy=0x7f07880214f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:11.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f07880180d0 osd=6 1 2024-02-19T21:09:11.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 10 client.49703.objecter close_session for osd.7 2024-02-19T21:09:11.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 >> v1:172.21.15.193:6812/157927432 conn(0x7f0788024b50 legacy=0x7f0788022ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:11.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 20 client.49703.objecter put_session s=0x7f0788024490 osd=7 1 2024-02-19T21:09:11.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 20 client.49703.objecter shutdown clearing up homeless session... 2024-02-19T21:09:11.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 10 client.49703.objecter successfully canceled tick 2024-02-19T21:09:11.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07aa155700 1 -- 172.21.15.163:0/2941242467 reap_dead start 2024-02-19T21:09:11.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 >> v1:172.21.15.163:6800/1552514681 conn(0x7f0790063c60 legacy=0x7f0790066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:11.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07aa155700 1 -- 172.21.15.163:0/2941242467 reap_dead start 2024-02-19T21:09:11.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07aa155700 1 -- 172.21.15.163:0/2941242467 reap_dead start 2024-02-19T21:09:11.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 >> v1:172.21.15.163:6790/0 conn(0x7f07a4106500 legacy=0x7f07a4117b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:11.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07aa155700 1 -- 172.21.15.163:0/2941242467 reap_dead start 2024-02-19T21:09:11.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 shutdown_connections 2024-02-19T21:09:11.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 >> 172.21.15.163:0/2941242467 conn(0x7f07a40f3590 msgr2=0x7f07a40f5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:11.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 shutdown_connections 2024-02-19T21:09:11.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 1 -- 172.21.15.163:0/2941242467 wait complete. 2024-02-19T21:09:11.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:10.821+0000 7f07abbb8700 1 librados: shutdown 2024-02-19T21:09:11.525 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.526 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.526 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.526 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.526 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.526 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.526 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.526 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:09:11.537 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:11.565 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:11.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:10.287167+0000 mon.b (mon.1) 536 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:09:11.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:10.287410+0000 mon.a (mon.0) 2364 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:09:11.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:10.291967+0000 mon.a (mon.0) 2365 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi163-83854-64"}]: dispatch 2024-02-19T21:09:11.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:10.292971+0000 mon.a (mon.0) 2366 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi163-83854-64"}]: dispatch 2024-02-19T21:09:11.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:10.293226+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi163-83854-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:09:11.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:10.329331+0000 mgr.y (mgr.14550) 338 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:11.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:10.682695+0000 mon.c (mon.2) 372 : audit [DBG] from='client.? v1:172.21.15.163:0/2941242467' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:11.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:11.264034+0000 mon.a (mon.0) 2368 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:09:11.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:11.264121+0000 mon.a (mon.0) 2369 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi163-83854-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:09:11.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:11.264175+0000 mon.a (mon.0) 2370 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2024-02-19T21:09:11.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:11.265934+0000 mon.b (mon.1) 537 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:11.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:11.266263+0000 mon.a (mon.0) 2371 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:11.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:11.266623+0000 mon.a (mon.0) 2372 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi163-83854-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi163-83854-64"}]: dispatch 2024-02-19T21:09:11.762 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:11.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:11.879 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:11.907 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:12.106 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:12.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:12.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:12.210 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:12.238 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:12.551 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-19T21:09:12.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.814+0000 7ff421eb9700 1 -- 172.21.15.163:0/3721784706 >> v1:172.21.15.163:6789/0 conn(0x7ff41c1031b0 legacy=0x7ff41c105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:12.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.814+0000 7ff421eb9700 1 -- 172.21.15.163:0/3721784706 shutdown_connections 2024-02-19T21:09:12.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.814+0000 7ff421eb9700 1 -- 172.21.15.163:0/3721784706 >> 172.21.15.163:0/3721784706 conn(0x7ff41c0f2470 msgr2=0x7ff41c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:12.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.814+0000 7ff421eb9700 1 -- 172.21.15.163:0/3721784706 shutdown_connections 2024-02-19T21:09:12.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.814+0000 7ff421eb9700 1 -- 172.21.15.163:0/3721784706 wait complete. 2024-02-19T21:09:12.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.815+0000 7ff421eb9700 1 librados: starting msgr at 2024-02-19T21:09:12.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.815+0000 7ff421eb9700 1 librados: starting objecter 2024-02-19T21:09:12.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.815+0000 7ff421eb9700 1 Processor -- start 2024-02-19T21:09:12.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.815+0000 7ff421eb9700 1 -- start start 2024-02-19T21:09:12.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.815+0000 7ff421eb9700 1 librados: setting wanted keys 2024-02-19T21:09:12.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.815+0000 7ff421eb9700 1 librados: calling monclient init 2024-02-19T21:09:12.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.815+0000 7ff421eb9700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff41c1162b0 con 0x7ff41c1031b0 2024-02-19T21:09:12.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.815+0000 7ff421eb9700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff41c117490 con 0x7ff41c0fc870 2024-02-19T21:09:12.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.815+0000 7ff421eb9700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff41c118690 con 0x7ff41c0ff670 2024-02-19T21:09:12.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.815+0000 7ff41affd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7ff41c0ff670 0x7ff41c0f92e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:54626/0 (socket says 172.21.15.163:54626) 2024-02-19T21:09:12.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.815+0000 7ff41affd700 1 -- 172.21.15.163:0/824730545 learned_addr learned my addr 172.21.15.163:0/824730545 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:12.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.815+0000 7ff41b7fe700 1 --1- 172.21.15.163:0/824730545 >> v1:172.21.15.193:6789/0 conn(0x7ff41c0fc870 0x7ff41c0f8bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:50264/0 (socket says 172.21.15.163:50264) 2024-02-19T21:09:12.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 10 client.?.objecter ms_handle_connect 0x7ff41c0ff670 2024-02-19T21:09:12.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:12.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 10 client.?.objecter ms_handle_connect 0x7ff41c0fc870 2024-02-19T21:09:12.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:12.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 10 client.?.objecter ms_handle_connect 0x7ff41c1031b0 2024-02-19T21:09:12.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:12.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950135950 0 0) 0x7ff41c117490 con 0x7ff41c0fc870 2024-02-19T21:09:12.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff404002220 con 0x7ff41c0fc870 2024-02-19T21:09:12.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724640715 0 0) 0x7ff41c118690 con 0x7ff41c0ff670 2024-02-19T21:09:12.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff41c117490 con 0x7ff41c0ff670 2024-02-19T21:09:12.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345573999 0 0) 0x7ff41c1162b0 con 0x7ff41c1031b0 2024-02-19T21:09:12.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff41c118690 con 0x7ff41c1031b0 2024-02-19T21:09:12.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 338331951 0 0) 0x7ff404002220 con 0x7ff41c0fc870 2024-02-19T21:09:12.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff41c1162b0 con 0x7ff41c0fc870 2024-02-19T21:09:12.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff40c001a30 con 0x7ff41c0fc870 2024-02-19T21:09:12.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2470786431 0 0) 0x7ff41c117490 con 0x7ff41c0ff670 2024-02-19T21:09:12.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff404002220 con 0x7ff41c0ff670 2024-02-19T21:09:12.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff4100018c0 con 0x7ff41c0ff670 2024-02-19T21:09:12.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3679636636 0 0) 0x7ff41c118690 con 0x7ff41c1031b0 2024-02-19T21:09:12.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff41c117490 con 0x7ff41c1031b0 2024-02-19T21:09:12.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff414003020 con 0x7ff41c1031b0 2024-02-19T21:09:12.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1518628518 0 0) 0x7ff41c1162b0 con 0x7ff41c0fc870 2024-02-19T21:09:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 >> v1:172.21.15.163:6790/0 conn(0x7ff41c0ff670 legacy=0x7ff41c0f92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 >> v1:172.21.15.163:6789/0 conn(0x7ff41c1031b0 legacy=0x7ff41c114970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c119890 con 0x7ff41c0fc870 2024-02-19T21:09:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.816+0000 7ff421eb9700 1 -- 172.21.15.163:0/824730545 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41c117680 con 0x7ff41c0fc870 2024-02-19T21:09:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.817+0000 7ff421eb9700 10 client.48526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.817+0000 7ff421eb9700 1 -- 172.21.15.163:0/824730545 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41c117af0 con 0x7ff41c0fc870 2024-02-19T21:09:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.817+0000 7ff421eb9700 1 librados: init done 2024-02-19T21:09:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.817+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff40c0023a0 con 0x7ff41c0fc870 2024-02-19T21:09:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.817+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff40c005a20 con 0x7ff41c0fc870 2024-02-19T21:09:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.817+0000 7ff421eb9700 1 -- 172.21.15.163:0/824730545 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff408004f20 con 0x7ff41c0fc870 2024-02-19T21:09:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.818+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ff40c005b80 con 0x7ff41c0fc870 2024-02-19T21:09:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.819+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(372..372 src has 43..372) v4 ==== 8293+0+0 (unknown 422637723 0 0) 0x7ff40c078cf0 con 0x7ff41c0fc870 2024-02-19T21:09:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.819+0000 7ff418ff9700 10 client.48526.objecter ms_dispatch 0x7ff41c001550 osd_map(372..372 src has 43..372) v4 2024-02-19T21:09:12.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.819+0000 7ff418ff9700 3 client.48526.objecter handle_osd_map got epochs [372,372] > 0 2024-02-19T21:09:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.819+0000 7ff418ff9700 3 client.48526.objecter handle_osd_map decoding full epoch 372 2024-02-19T21:09:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.819+0000 7ff418ff9700 20 client.48526.objecter dump_active .. 0 homeless 2024-02-19T21:09:12.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.819+0000 7ff418ff9700 10 client.48526.objecter ms_handle_connect 0x7ff404068270 2024-02-19T21:09:12.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:11.195024+0000 mgr.y (mgr.14550) 339 : cluster [DBG] pgmap v527: 228 pgs: 32 creating+peering, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:09:12.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:12.263368+0000 mon.a (mon.0) 2373 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-49"}]': finished 2024-02-19T21:09:12.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:12.824+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff40c04c150 con 0x7ff41c0fc870 2024-02-19T21:09:12.844 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:12.888 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:12.890 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:12.890 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:11.195024+0000 mgr.y (mgr.14550) 339 : cluster [DBG] pgmap v527: 228 pgs: 32 creating+peering, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:09:12.890 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:12.263368+0000 mon.a (mon.0) 2373 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-49"}]': finished 2024-02-19T21:09:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:12.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:12.893 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:11.195024+0000 mgr.y (mgr.14550) 339 : cluster [DBG] pgmap v527: 228 pgs: 32 creating+peering, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:09:12.893 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:12.263368+0000 mon.a (mon.0) 2373 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-49"}]': finished 2024-02-19T21:09:12.992 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:13.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:13.019+0000 7ff421eb9700 1 -- 172.21.15.163:0/824730545 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7ff4080051d0 con 0x7ff41c0fc870 2024-02-19T21:09:13.025 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:13.050 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:13.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:13.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:13.152 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:13.180 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:13.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:13.267+0000 7ff418ff9700 1 -- 172.21.15.163:0/824730545 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v373) v1 ==== 97+0+0 (unknown 4210852209 0 0) 0x7ff40c04a560 con 0x7ff41c0fc870 2024-02-19T21:09:13.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:13.272+0000 7ff421eb9700 10 librados: watch_flush enter 2024-02-19T21:09:13.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:13.273+0000 7ff421eb9700 10 librados: watch_flush exit 2024-02-19T21:09:13.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:13.273+0000 7ff421eb9700 20 client.48526.objecter shutdown clearing up homeless session... 2024-02-19T21:09:13.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:13.273+0000 7ff421eb9700 10 client.48526.objecter successfully canceled tick 2024-02-19T21:09:13.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:13.273+0000 7ff421eb9700 1 -- 172.21.15.163:0/824730545 >> v1:172.21.15.163:6800/1552514681 conn(0x7ff404068270 legacy=0x7ff40406a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:13.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:13.273+0000 7ff421eb9700 1 -- 172.21.15.163:0/824730545 >> v1:172.21.15.193:6789/0 conn(0x7ff41c0fc870 legacy=0x7ff41c0f8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:13.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:13.275+0000 7ff421eb9700 1 -- 172.21.15.163:0/824730545 shutdown_connections 2024-02-19T21:09:13.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:13.275+0000 7ff421eb9700 1 -- 172.21.15.163:0/824730545 >> 172.21.15.163:0/824730545 conn(0x7ff41c0f2470 msgr2=0x7ff41c106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:13.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:13.277+0000 7ff421eb9700 1 -- 172.21.15.163:0/824730545 shutdown_connections 2024-02-19T21:09:13.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:13.277+0000 7ff421eb9700 1 -- 172.21.15.163:0/824730545 wait complete. 2024-02-19T21:09:13.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:13.277+0000 7ff421eb9700 1 librados: shutdown 2024-02-19T21:09:13.275 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:09:13.329 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:13.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:13.433 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:13.461 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:13.480 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:13.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:13.584 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:13.611 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:13.630 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:13.632 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:13.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:13.733 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:13.744 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:09:13] "GET /metrics HTTP/1.1" 200 186694 "" "Prometheus/2.33.4" 2024-02-19T21:09:13.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:12.266371+0000 mon.b (mon.1) 538 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-49", "mode": "writeback"}]: dispatch 2024-02-19T21:09:13.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:12.499547+0000 mon.a (mon.0) 2374 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2024-02-19T21:09:13.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:12.500406+0000 mon.a (mon.0) 2375 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-49", "mode": "writeback"}]: dispatch 2024-02-19T21:09:13.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:13.020344+0000 mon.b (mon.1) 539 : audit [INF] from='client.? v1:172.21.15.163:0/824730545' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:13.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:13.020716+0000 mon.a (mon.0) 2376 : audit [INF] from='client.48526 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:13.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:13.260826+0000 mon.a (mon.0) 2377 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:09:13.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:13.260849+0000 mon.a (mon.0) 2378 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:09:13.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:13.267114+0000 mon.a (mon.0) 2379 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi163-83854-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi163-83854-64"}]': finished 2024-02-19T21:09:13.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:13.267185+0000 mon.a (mon.0) 2380 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-49", "mode": "writeback"}]': finished 2024-02-19T21:09:13.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:13.267248+0000 mon.a (mon.0) 2381 : audit [INF] from='client.48526 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:09:13.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:13.267314+0000 mon.a (mon.0) 2382 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2024-02-19T21:09:13.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:13.308689+0000 mon.b (mon.1) 540 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:09:13.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:13.309051+0000 mon.a (mon.0) 2383 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:09:13.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:12.266371+0000 mon.b (mon.1) 538 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-49", "mode": "writeback"}]: dispatch 2024-02-19T21:09:13.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:12.499547+0000 mon.a (mon.0) 2374 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2024-02-19T21:09:13.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:12.500406+0000 mon.a (mon.0) 2375 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-49", "mode": "writeback"}]: dispatch 2024-02-19T21:09:13.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:13.020344+0000 mon.b (mon.1) 539 : audit [INF] from='client.? v1:172.21.15.163:0/824730545' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:13.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:13.020716+0000 mon.a (mon.0) 2376 : audit [INF] from='client.48526 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:13.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:13.260826+0000 mon.a (mon.0) 2377 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:09:13.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:13.260849+0000 mon.a (mon.0) 2378 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:09:13.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:13.267114+0000 mon.a (mon.0) 2379 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi163-83854-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi163-83854-64"}]': finished 2024-02-19T21:09:13.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:13.267185+0000 mon.a (mon.0) 2380 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-49", "mode": "writeback"}]': finished 2024-02-19T21:09:13.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:13.267248+0000 mon.a (mon.0) 2381 : audit [INF] from='client.48526 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:09:13.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:13.267314+0000 mon.a (mon.0) 2382 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2024-02-19T21:09:13.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:13.308689+0000 mon.b (mon.1) 540 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:09:13.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:13.309051+0000 mon.a (mon.0) 2383 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:09:13.762 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:12.266371+0000 mon.b (mon.1) 538 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-49", "mode": "writeback"}]: dispatch 2024-02-19T21:09:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:12.499547+0000 mon.a (mon.0) 2374 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2024-02-19T21:09:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:12.500406+0000 mon.a (mon.0) 2375 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-49", "mode": "writeback"}]: dispatch 2024-02-19T21:09:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:13.020344+0000 mon.b (mon.1) 539 : audit [INF] from='client.? v1:172.21.15.163:0/824730545' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:13.020716+0000 mon.a (mon.0) 2376 : audit [INF] from='client.48526 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:13.260826+0000 mon.a (mon.0) 2377 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:09:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:13.260849+0000 mon.a (mon.0) 2378 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:09:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:13.267114+0000 mon.a (mon.0) 2379 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi163-83854-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi163-83854-64"}]': finished 2024-02-19T21:09:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:13.267185+0000 mon.a (mon.0) 2380 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-49", "mode": "writeback"}]': finished 2024-02-19T21:09:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:13.267248+0000 mon.a (mon.0) 2381 : audit [INF] from='client.48526 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:09:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:13.267314+0000 mon.a (mon.0) 2382 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2024-02-19T21:09:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:13.308689+0000 mon.b (mon.1) 540 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:09:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:13.309051+0000 mon.a (mon.0) 2383 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:09:14.124 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:09:14.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:13.195408+0000 mgr.y (mgr.14550) 340 : cluster [DBG] pgmap v530: 228 pgs: 32 creating+peering, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:09:14.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:14.241910+0000 mon.a (mon.0) 2384 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:09:14.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:14.241990+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2024-02-19T21:09:14.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:14.249058+0000 mon.b (mon.1) 541 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:09:14.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:14.255035+0000 mon.a (mon.0) 2386 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:09:14.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:13.195408+0000 mgr.y (mgr.14550) 340 : cluster [DBG] pgmap v530: 228 pgs: 32 creating+peering, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:09:14.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:14.241910+0000 mon.a (mon.0) 2384 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:09:14.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:14.241990+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2024-02-19T21:09:14.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:14.249058+0000 mon.b (mon.1) 541 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:09:14.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:14.255035+0000 mon.a (mon.0) 2386 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:09:14.832 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:13.195408+0000 mgr.y (mgr.14550) 340 : cluster [DBG] pgmap v530: 228 pgs: 32 creating+peering, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:09:14.832 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:14.241910+0000 mon.a (mon.0) 2384 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:09:14.832 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:14.241990+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2024-02-19T21:09:14.832 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:14.249058+0000 mon.b (mon.1) 541 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:09:14.832 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:14.255035+0000 mon.a (mon.0) 2386 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:09:15.095 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:09:15.095 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:09:15.095 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-02-19T21:09:15.095 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:09:15.812 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:15.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:15.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:15.916 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:15.946 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:16.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:15.196290+0000 mgr.y (mgr.14550) 341 : cluster [DBG] pgmap v533: 236 pgs: 236 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-19T21:09:16.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:15.248500+0000 mon.a (mon.0) 2387 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:09:16.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:15.248588+0000 mon.a (mon.0) 2388 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2024-02-19T21:09:16.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:15.252416+0000 mon.b (mon.1) 542 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:09:16.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:15.252665+0000 mon.a (mon.0) 2389 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi163-83854-64"}]: dispatch 2024-02-19T21:09:16.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:15.253003+0000 mon.a (mon.0) 2390 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:09:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:15.196290+0000 mgr.y (mgr.14550) 341 : cluster [DBG] pgmap v533: 236 pgs: 236 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-19T21:09:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:15.248500+0000 mon.a (mon.0) 2387 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:09:16.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:15.248588+0000 mon.a (mon.0) 2388 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2024-02-19T21:09:16.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:15.252416+0000 mon.b (mon.1) 542 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:09:16.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:15.252665+0000 mon.a (mon.0) 2389 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi163-83854-64"}]: dispatch 2024-02-19T21:09:16.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:15.253003+0000 mon.a (mon.0) 2390 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:09:16.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:15.196290+0000 mgr.y (mgr.14550) 341 : cluster [DBG] pgmap v533: 236 pgs: 236 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-19T21:09:16.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:15.248500+0000 mon.a (mon.0) 2387 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:09:16.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:15.248588+0000 mon.a (mon.0) 2388 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2024-02-19T21:09:16.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:15.252416+0000 mon.b (mon.1) 542 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:09:16.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:15.252665+0000 mon.a (mon.0) 2389 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi163-83854-64"}]: dispatch 2024-02-19T21:09:16.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:15.253003+0000 mon.a (mon.0) 2390 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:09:17.023 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:17.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:17.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:17.031 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-19T21:09:17.043 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:17.128 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:09:17.154 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:17.277 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:09:17.279 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:16.240535+0000 mon.a (mon.0) 2391 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:09:17.279 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:16.246846+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi163-83854-64"}]': finished 2024-02-19T21:09:17.279 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:16.246915+0000 mon.a (mon.0) 2393 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:09:17.279 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:16.246980+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2024-02-19T21:09:17.279 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:16.250202+0000 mon.b (mon.1) 543 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:09:17.279 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:16.251710+0000 mon.a (mon.0) 2395 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi163-83854-64"}]: dispatch 2024-02-19T21:09:17.279 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:16.252043+0000 mon.a (mon.0) 2396 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:09:17.279 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:17.249313+0000 mon.a (mon.0) 2397 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi163-83854-64"}]': finished 2024-02-19T21:09:17.280 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:17.249379+0000 mon.a (mon.0) 2398 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-19T21:09:17.280 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:17.249483+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2024-02-19T21:09:17.280 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:17.250196+0000 mon.b (mon.1) 544 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-19T21:09:17.280 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:17.250567+0000 mon.a (mon.0) 2400 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-19T21:09:17.280 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:16.240535+0000 mon.a (mon.0) 2391 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:09:17.280 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:16.246846+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi163-83854-64"}]': finished 2024-02-19T21:09:17.280 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:16.246915+0000 mon.a (mon.0) 2393 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:09:17.280 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:16.246980+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2024-02-19T21:09:17.280 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:16.250202+0000 mon.b (mon.1) 543 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:09:17.280 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:16.251710+0000 mon.a (mon.0) 2395 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi163-83854-64"}]: dispatch 2024-02-19T21:09:17.281 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:16.252043+0000 mon.a (mon.0) 2396 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:09:17.281 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:17.249313+0000 mon.a (mon.0) 2397 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi163-83854-64"}]': finished 2024-02-19T21:09:17.281 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:17.249379+0000 mon.a (mon.0) 2398 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-19T21:09:17.281 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:17.249483+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2024-02-19T21:09:17.281 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:17.250196+0000 mon.b (mon.1) 544 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-19T21:09:17.281 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:17.250567+0000 mon.a (mon.0) 2400 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-19T21:09:17.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:17.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:09:17.383 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:17.411 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:17.489 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:09:17.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:16.240535+0000 mon.a (mon.0) 2391 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:09:17.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:16.246846+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi163-83854-64"}]': finished 2024-02-19T21:09:17.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:16.246915+0000 mon.a (mon.0) 2393 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:09:17.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:16.246980+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2024-02-19T21:09:17.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:16.250202+0000 mon.b (mon.1) 543 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:09:17.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:16.251710+0000 mon.a (mon.0) 2395 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi163-83854-64"}]: dispatch 2024-02-19T21:09:17.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:16.252043+0000 mon.a (mon.0) 2396 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:09:17.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:17.249313+0000 mon.a (mon.0) 2397 : audit [INF] from='client.? v1:172.21.15.163:0/2118002540' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi163-83854-64"}]': finished 2024-02-19T21:09:17.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:17.249379+0000 mon.a (mon.0) 2398 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-19T21:09:17.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:17.249483+0000 mon.a (mon.0) 2399 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2024-02-19T21:09:17.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:17.250196+0000 mon.b (mon.1) 544 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-19T21:09:17.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:17.250567+0000 mon.a (mon.0) 2400 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "target_max_objects","val": "1"}]: dispatch 2024-02-19T21:09:18.068 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:09:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:09:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:09:18.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.071+0000 7f09647b3700 1 -- 172.21.15.163:0/463647754 >> v1:172.21.15.163:6790/0 conn(0x7f095c1042c0 legacy=0x7f095c106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:18.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.071+0000 7f09647b3700 1 -- 172.21.15.163:0/463647754 shutdown_connections 2024-02-19T21:09:18.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.071+0000 7f09647b3700 1 -- 172.21.15.163:0/463647754 >> 172.21.15.163:0/463647754 conn(0x7f095c0f35a0 msgr2=0x7f095c0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:18.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.071+0000 7f09647b3700 1 -- 172.21.15.163:0/463647754 shutdown_connections 2024-02-19T21:09:18.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.071+0000 7f09647b3700 1 -- 172.21.15.163:0/463647754 wait complete. 2024-02-19T21:09:18.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.071+0000 7f09647b3700 1 librados: starting msgr at 2024-02-19T21:09:18.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.071+0000 7f09647b3700 1 librados: starting objecter 2024-02-19T21:09:18.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.071+0000 7f09647b3700 1 Processor -- start 2024-02-19T21:09:18.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09647b3700 1 -- start start 2024-02-19T21:09:18.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09647b3700 1 librados: setting wanted keys 2024-02-19T21:09:18.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09647b3700 1 librados: calling monclient init 2024-02-19T21:09:18.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09647b3700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f095c117350 con 0x7f095c0fd980 2024-02-19T21:09:18.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09647b3700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f095c118530 con 0x7f095c1042c0 2024-02-19T21:09:18.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09647b3700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f095c119730 con 0x7f095c100780 2024-02-19T21:09:18.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f0961d4e700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f095c100780 0x7f095c0fa070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:51736/0 (socket says 172.21.15.163:51736) 2024-02-19T21:09:18.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f0961d4e700 1 -- 172.21.15.163:0/4273750768 learned_addr learned my addr 172.21.15.163:0/4273750768 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:18.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f096254f700 1 --1- 172.21.15.163:0/4273750768 >> v1:172.21.15.163:6789/0 conn(0x7f095c0fd980 0x7f095c0f9920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44250/0 (socket says 172.21.15.163:44250) 2024-02-19T21:09:18.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 10 client.?.objecter ms_handle_connect 0x7f095c100780 2024-02-19T21:09:18.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:18.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 10 client.?.objecter ms_handle_connect 0x7f095c0fd980 2024-02-19T21:09:18.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:18.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186497517 0 0) 0x7f095c119730 con 0x7f095c100780 2024-02-19T21:09:18.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0948003030 con 0x7f095c100780 2024-02-19T21:09:18.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527574211 0 0) 0x7f095c117350 con 0x7f095c0fd980 2024-02-19T21:09:18.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f095c119730 con 0x7f095c0fd980 2024-02-19T21:09:18.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 10 client.?.objecter ms_handle_connect 0x7f095c1042c0 2024-02-19T21:09:18.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:18.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 562664152 0 0) 0x7f0948003030 con 0x7f095c100780 2024-02-19T21:09:18.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f095c117350 con 0x7f095c100780 2024-02-19T21:09:18.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f094c0018e0 con 0x7f095c100780 2024-02-19T21:09:18.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970302837 0 0) 0x7f095c118530 con 0x7f095c1042c0 2024-02-19T21:09:18.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0948003030 con 0x7f095c1042c0 2024-02-19T21:09:18.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2433658993 0 0) 0x7f095c119730 con 0x7f095c0fd980 2024-02-19T21:09:18.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f095c118530 con 0x7f095c0fd980 2024-02-19T21:09:18.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0958001910 con 0x7f095c0fd980 2024-02-19T21:09:18.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1426219285 0 0) 0x7f095c117350 con 0x7f095c100780 2024-02-19T21:09:18.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 >> v1:172.21.15.193:6789/0 conn(0x7f095c1042c0 legacy=0x7f095c115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:18.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.072+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 >> v1:172.21.15.163:6789/0 conn(0x7f095c0fd980 legacy=0x7f095c0f9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:18.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.073+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c11a930 con 0x7f095c100780 2024-02-19T21:09:18.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.073+0000 7f09647b3700 1 -- 172.21.15.163:0/4273750768 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095c117540 con 0x7f095c100780 2024-02-19T21:09:18.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.073+0000 7f09647b3700 10 client.49721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:18.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.073+0000 7f09647b3700 1 -- 172.21.15.163:0/4273750768 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f095c117a30 con 0x7f095c100780 2024-02-19T21:09:18.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.073+0000 7f09647b3700 1 librados: init done 2024-02-19T21:09:18.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.073+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f094c002420 con 0x7f095c100780 2024-02-19T21:09:18.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.073+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f094c005ae0 con 0x7f095c100780 2024-02-19T21:09:18.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.073+0000 7f09647b3700 1 -- 172.21.15.163:0/4273750768 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0940004f20 con 0x7f095c100780 2024-02-19T21:09:18.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.074+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f094c001f80 con 0x7f095c100780 2024-02-19T21:09:18.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.074+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(377..377 src has 43..377) v4 ==== 7936+0+0 (unknown 2732483874 0 0) 0x7f094c078c40 con 0x7f095c100780 2024-02-19T21:09:18.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.074+0000 7f09537fe700 10 client.49721.objecter ms_dispatch 0x7f095c001550 osd_map(377..377 src has 43..377) v4 2024-02-19T21:09:18.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.074+0000 7f09537fe700 3 client.49721.objecter handle_osd_map got epochs [377,377] > 0 2024-02-19T21:09:18.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.074+0000 7f09537fe700 3 client.49721.objecter handle_osd_map decoding full epoch 377 2024-02-19T21:09:18.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.075+0000 7f09537fe700 20 client.49721.objecter dump_active .. 0 homeless 2024-02-19T21:09:18.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.075+0000 7f09537fe700 10 client.49721.objecter ms_handle_connect 0x7f09480681e0 2024-02-19T21:09:18.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.079+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f094c04c110 con 0x7f095c100780 2024-02-19T21:09:18.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:18.262+0000 7f09647b3700 1 -- 172.21.15.163:0/4273750768 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f09400051d0 con 0x7f095c100780 2024-02-19T21:09:18.560 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:17.196683+0000 mgr.y (mgr.14550) 342 : cluster [DBG] pgmap v536: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-19T21:09:18.560 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:17.271111+0000 mon.c (mon.2) 373 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.561 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:17.271459+0000 mon.a (mon.0) 2401 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.561 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:17.272440+0000 mon.c (mon.2) 374 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.561 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:17.272653+0000 mon.a (mon.0) 2402 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.561 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:17.273020+0000 mon.c (mon.2) 375 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi163-83854-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:09:18.561 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:17.273256+0000 mon.a (mon.0) 2403 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi163-83854-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:09:18.561 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:18.247504+0000 mon.a (mon.0) 2404 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "target_max_objects","val": "1"}]': finished 2024-02-19T21:09:18.561 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:18.247595+0000 mon.a (mon.0) 2405 : audit [INF] from='client.49715 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi163-83854-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:09:18.561 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:18.247672+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2024-02-19T21:09:18.561 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:18.252077+0000 mon.c (mon.2) 376 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi163-83854-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.561 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:18.252429+0000 mon.a (mon.0) 2407 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi163-83854-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.561 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:17.196683+0000 mgr.y (mgr.14550) 342 : cluster [DBG] pgmap v536: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-19T21:09:18.561 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:17.271111+0000 mon.c (mon.2) 373 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.562 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:17.271459+0000 mon.a (mon.0) 2401 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.562 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:17.272440+0000 mon.c (mon.2) 374 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.562 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:17.272653+0000 mon.a (mon.0) 2402 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.562 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:17.273020+0000 mon.c (mon.2) 375 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi163-83854-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:09:18.562 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:17.273256+0000 mon.a (mon.0) 2403 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi163-83854-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:09:18.562 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:18.247504+0000 mon.a (mon.0) 2404 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "target_max_objects","val": "1"}]': finished 2024-02-19T21:09:18.562 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:18.247595+0000 mon.a (mon.0) 2405 : audit [INF] from='client.49715 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi163-83854-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:09:18.562 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:18.247672+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2024-02-19T21:09:18.562 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:18.252077+0000 mon.c (mon.2) 376 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi163-83854-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.562 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:18.252429+0000 mon.a (mon.0) 2407 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi163-83854-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:17.196683+0000 mgr.y (mgr.14550) 342 : cluster [DBG] pgmap v536: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-02-19T21:09:18.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:17.271111+0000 mon.c (mon.2) 373 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:17.271459+0000 mon.a (mon.0) 2401 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:17.272440+0000 mon.c (mon.2) 374 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:17.272653+0000 mon.a (mon.0) 2402 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:17.273020+0000 mon.c (mon.2) 375 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi163-83854-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:09:18.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:17.273256+0000 mon.a (mon.0) 2403 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi163-83854-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:09:18.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:18.247504+0000 mon.a (mon.0) 2404 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-49","var": "target_max_objects","val": "1"}]': finished 2024-02-19T21:09:18.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:18.247595+0000 mon.a (mon.0) 2405 : audit [INF] from='client.49715 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi163-83854-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:09:18.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:18.247672+0000 mon.a (mon.0) 2406 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2024-02-19T21:09:18.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:18.252077+0000 mon.c (mon.2) 376 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi163-83854-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:18.252429+0000 mon.a (mon.0) 2407 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi163-83854-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:18.569 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:18.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:18.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:18.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:18.701 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:18.850 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:18.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:18.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:18.956 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:18.988 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:19.009 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:19.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:19.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:19.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.059+0000 7f02360a8700 1 -- 172.21.15.163:0/1524132112 >> v1:172.21.15.163:6789/0 conn(0x7f02300fbfd0 legacy=0x7f02300fc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:19.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.059+0000 7f02360a8700 1 -- 172.21.15.163:0/1524132112 shutdown_connections 2024-02-19T21:09:19.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.059+0000 7f02360a8700 1 -- 172.21.15.163:0/1524132112 >> 172.21.15.163:0/1524132112 conn(0x7f023006cc00 msgr2=0x7f02300f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:19.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.059+0000 7f02360a8700 1 -- 172.21.15.163:0/1524132112 shutdown_connections 2024-02-19T21:09:19.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.059+0000 7f02360a8700 1 -- 172.21.15.163:0/1524132112 wait complete. 2024-02-19T21:09:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.060+0000 7f02360a8700 1 librados: starting msgr at 2024-02-19T21:09:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.060+0000 7f02360a8700 1 librados: starting objecter 2024-02-19T21:09:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.060+0000 7f02360a8700 1 Processor -- start 2024-02-19T21:09:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.060+0000 7f02360a8700 1 -- start start 2024-02-19T21:09:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.060+0000 7f02360a8700 1 librados: setting wanted keys 2024-02-19T21:09:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.060+0000 7f02360a8700 1 librados: calling monclient init 2024-02-19T21:09:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.060+0000 7f02360a8700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f02301194a0 con 0x7f0230102930 2024-02-19T21:09:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.060+0000 7f02360a8700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f023011a680 con 0x7f02300fee20 2024-02-19T21:09:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.060+0000 7f02360a8700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f023011b880 con 0x7f02300fbfd0 2024-02-19T21:09:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022effd700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f02300fee20 0x7f0230114380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:60882/0 (socket says 172.21.15.163:60882) 2024-02-19T21:09:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022effd700 1 -- 172.21.15.163:0/2623101106 learned_addr learned my addr 172.21.15.163:0/2623101106 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022f7fe700 1 --1- 172.21.15.163:0/2623101106 >> v1:172.21.15.163:6790/0 conn(0x7f02300fbfd0 0x7f0230106fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:51762/0 (socket says 172.21.15.163:51762) 2024-02-19T21:09:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 10 client.?.objecter ms_handle_connect 0x7f02300fee20 2024-02-19T21:09:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 10 client.?.objecter ms_handle_connect 0x7f0230102930 2024-02-19T21:09:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 10 client.?.objecter ms_handle_connect 0x7f02300fbfd0 2024-02-19T21:09:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920394253 0 0) 0x7f023011a680 con 0x7f02300fee20 2024-02-19T21:09:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0218003030 con 0x7f02300fee20 2024-02-19T21:09:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654949136 0 0) 0x7f02301194a0 con 0x7f0230102930 2024-02-19T21:09:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f023011a680 con 0x7f0230102930 2024-02-19T21:09:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734458006 0 0) 0x7f023011b880 con 0x7f02300fbfd0 2024-02-19T21:09:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02301194a0 con 0x7f02300fbfd0 2024-02-19T21:09:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4264941247 0 0) 0x7f0218003030 con 0x7f02300fee20 2024-02-19T21:09:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f023011b880 con 0x7f02300fee20 2024-02-19T21:09:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1535798372 0 0) 0x7f023011a680 con 0x7f0230102930 2024-02-19T21:09:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0218003030 con 0x7f0230102930 2024-02-19T21:09:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3061064055 0 0) 0x7f02301194a0 con 0x7f02300fbfd0 2024-02-19T21:09:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f023011a680 con 0x7f02300fbfd0 2024-02-19T21:09:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0224001980 con 0x7f02300fee20 2024-02-19T21:09:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0228001bb0 con 0x7f0230102930 2024-02-19T21:09:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0220002c60 con 0x7f02300fbfd0 2024-02-19T21:09:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.061+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 277196263 0 0) 0x7f023011a680 con 0x7f02300fbfd0 2024-02-19T21:09:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.062+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 >> v1:172.21.15.193:6789/0 conn(0x7f02300fee20 legacy=0x7f0230114380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.062+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 >> v1:172.21.15.163:6789/0 conn(0x7f0230102930 legacy=0x7f0230117b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.062+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023011ca80 con 0x7f02300fbfd0 2024-02-19T21:09:19.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.062+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0230119690 con 0x7f02300fbfd0 2024-02-19T21:09:19.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.062+0000 7f02360a8700 10 client.49727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:19.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.062+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0230119b00 con 0x7f02300fbfd0 2024-02-19T21:09:19.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.062+0000 7f02360a8700 1 librados: init done 2024-02-19T21:09:19.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.062+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f023005f540 con 0x7f02300fbfd0 2024-02-19T21:09:19.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.062+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f0220001aa0 con 0x7f02300fbfd0 2024-02-19T21:09:19.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.062+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0220003570 con 0x7f02300fbfd0 2024-02-19T21:09:19.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.064+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f02200036d0 con 0x7f02300fbfd0 2024-02-19T21:09:19.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.064+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(378..378 src has 43..378) v4 ==== 8224+0+0 (unknown 117095947 0 0) 0x7f0220078e60 con 0x7f02300fbfd0 2024-02-19T21:09:19.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.064+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 osd_map(378..378 src has 43..378) v4 2024-02-19T21:09:19.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.064+0000 7f022cff9700 3 client.49727.objecter handle_osd_map got epochs [378,378] > 0 2024-02-19T21:09:19.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.064+0000 7f022cff9700 3 client.49727.objecter handle_osd_map decoding full epoch 378 2024-02-19T21:09:19.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.064+0000 7f022cff9700 20 client.49727.objecter dump_active .. 0 homeless 2024-02-19T21:09:19.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.064+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v378) v1 ==== 54+0+16 (unknown 3027434829 0 2420000192) 0x7f0220079240 con 0x7f02300fbfd0 2024-02-19T21:09:19.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.065+0000 7f022cff9700 10 client.49727.objecter ms_handle_connect 0x7f0218063dc0 2024-02-19T21:09:19.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.065+0000 7f02360a8700 10 client.49727.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:19.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.065+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 1 2024-02-19T21:09:19.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.065+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230047540 osd=0 3 2024-02-19T21:09:19.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.065+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230047540 osd=0 3 2024-02-19T21:09:19.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.065+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230047540 osd=0 3 2024-02-19T21:09:19.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.065+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 1 2024-02-19T21:09:19.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.065+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230047540 osd=0 3 2024-02-19T21:09:19.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.065+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 0 1 2024-02-19T21:09:19.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.065+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230047540 osd=0 4 2024-02-19T21:09:19.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.065+0000 7f02360a8700 10 client.49727.objecter _send_command 1 2024-02-19T21:09:19.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.065+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02300f7850 con 0x7f023011a810 2024-02-19T21:09:19.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.065+0000 7f022cff9700 10 client.49727.objecter ms_handle_connect 0x7f023011a810 2024-02-19T21:09:19.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.066+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f02300f7850 con 0x7f023011a810 2024-02-19T21:09:19.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.066+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:09:19.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.066+0000 7f022cff9700 10 client.49727.objecter _finish_command 1 = system:0 2024-02-19T21:09:19.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.066+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230047540 osd=0 4 2024-02-19T21:09:19.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.066+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 0 1 2024-02-19T21:09:19.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.088+0000 7f02360a8700 10 client.49727.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:19.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.088+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 2 2024-02-19T21:09:19.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.088+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230047540 osd=0 3 2024-02-19T21:09:19.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.088+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230047540 osd=0 3 2024-02-19T21:09:19.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.088+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230047540 osd=0 3 2024-02-19T21:09:19.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.088+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 2 2024-02-19T21:09:19.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.088+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230047540 osd=0 3 2024-02-19T21:09:19.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.088+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 0 2 2024-02-19T21:09:19.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.088+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230047540 osd=0 4 2024-02-19T21:09:19.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.088+0000 7f02360a8700 10 client.49727.objecter _send_command 2 2024-02-19T21:09:19.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.088+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0230107cd0 con 0x7f023011a810 2024-02-19T21:09:19.087 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:19.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0230107cd0 con 0x7f023011a810 2024-02-19T21:09:19.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:19.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f022cff9700 10 client.49727.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230047540 osd=0 4 2024-02-19T21:09:19.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 0 2 2024-02-19T21:09:19.088 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:09:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f02360a8700 10 client.49727.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 3 2024-02-19T21:09:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230103d80 osd=1 3 2024-02-19T21:09:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230103d80 osd=1 3 2024-02-19T21:09:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230103d80 osd=1 3 2024-02-19T21:09:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 3 2024-02-19T21:09:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230103d80 osd=1 3 2024-02-19T21:09:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 1 3 2024-02-19T21:09:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230103d80 osd=1 4 2024-02-19T21:09:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f02360a8700 10 client.49727.objecter _send_command 3 2024-02-19T21:09:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.090+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02300283d0 con 0x7f0230104400 2024-02-19T21:09:19.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.091+0000 7f022cff9700 10 client.49727.objecter ms_handle_connect 0x7f0230104400 2024-02-19T21:09:19.090 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:19.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.091+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f02300283d0 con 0x7f0230104400 2024-02-19T21:09:19.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.091+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:09:19.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.091+0000 7f022cff9700 10 client.49727.objecter _finish_command 3 = system:0 2024-02-19T21:09:19.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.091+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230103d80 osd=1 4 2024-02-19T21:09:19.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.091+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 1 3 2024-02-19T21:09:19.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.107+0000 7f02360a8700 10 client.49727.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:19.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.107+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 4 2024-02-19T21:09:19.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.107+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230103d80 osd=1 3 2024-02-19T21:09:19.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.107+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230103d80 osd=1 3 2024-02-19T21:09:19.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.107+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230103d80 osd=1 3 2024-02-19T21:09:19.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.107+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 4 2024-02-19T21:09:19.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.107+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230103d80 osd=1 3 2024-02-19T21:09:19.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.107+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 1 4 2024-02-19T21:09:19.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.107+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230103d80 osd=1 4 2024-02-19T21:09:19.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.107+0000 7f02360a8700 10 client.49727.objecter _send_command 4 2024-02-19T21:09:19.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.107+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0230028780 con 0x7f0230104400 2024-02-19T21:09:19.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0230028780 con 0x7f0230104400 2024-02-19T21:09:19.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:19.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f022cff9700 10 client.49727.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230103d80 osd=1 4 2024-02-19T21:09:19.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 1 4 2024-02-19T21:09:19.107 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:09:19.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f02360a8700 10 client.49727.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:19.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 5 2024-02-19T21:09:19.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230119ff0 osd=2 3 2024-02-19T21:09:19.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230119ff0 osd=2 3 2024-02-19T21:09:19.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230119ff0 osd=2 3 2024-02-19T21:09:19.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 5 2024-02-19T21:09:19.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230119ff0 osd=2 3 2024-02-19T21:09:19.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 2 5 2024-02-19T21:09:19.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230119ff0 osd=2 4 2024-02-19T21:09:19.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f02360a8700 10 client.49727.objecter _send_command 5 2024-02-19T21:09:19.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.109+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f023011a160 con 0x7f0230059e00 2024-02-19T21:09:19.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.110+0000 7f022cff9700 10 client.49727.objecter ms_handle_connect 0x7f0230059e00 2024-02-19T21:09:19.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.110+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f023011a160 con 0x7f0230059e00 2024-02-19T21:09:19.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.110+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:09:19.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.110+0000 7f022cff9700 10 client.49727.objecter _finish_command 5 = system:0 2024-02-19T21:09:19.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.110+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230119ff0 osd=2 4 2024-02-19T21:09:19.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.110+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 2 5 2024-02-19T21:09:19.114 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:09:19.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.131+0000 7f02360a8700 10 client.49727.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:19.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.131+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 6 2024-02-19T21:09:19.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.131+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230119ff0 osd=2 3 2024-02-19T21:09:19.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.131+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230119ff0 osd=2 3 2024-02-19T21:09:19.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.131+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230119ff0 osd=2 3 2024-02-19T21:09:19.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.131+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 6 2024-02-19T21:09:19.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.131+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230119ff0 osd=2 3 2024-02-19T21:09:19.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.131+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 2 6 2024-02-19T21:09:19.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.131+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230119ff0 osd=2 4 2024-02-19T21:09:19.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.131+0000 7f02360a8700 10 client.49727.objecter _send_command 6 2024-02-19T21:09:19.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.131+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f02301255a0 con 0x7f0230059e00 2024-02-19T21:09:19.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f02301255a0 con 0x7f0230059e00 2024-02-19T21:09:19.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:19.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f022cff9700 10 client.49727.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230119ff0 osd=2 4 2024-02-19T21:09:19.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 2 6 2024-02-19T21:09:19.130 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:09:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f02360a8700 10 client.49727.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 7 2024-02-19T21:09:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230059360 osd=3 3 2024-02-19T21:09:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230059360 osd=3 3 2024-02-19T21:09:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230059360 osd=3 3 2024-02-19T21:09:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 7 2024-02-19T21:09:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230059360 osd=3 3 2024-02-19T21:09:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 3 7 2024-02-19T21:09:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230059360 osd=3 4 2024-02-19T21:09:19.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f02360a8700 10 client.49727.objecter _send_command 7 2024-02-19T21:09:19.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.132+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0230121b00 con 0x7f02300594d0 2024-02-19T21:09:19.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.133+0000 7f022cff9700 10 client.49727.objecter ms_handle_connect 0x7f02300594d0 2024-02-19T21:09:19.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.133+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0230121b00 con 0x7f02300594d0 2024-02-19T21:09:19.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.133+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:09:19.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.133+0000 7f022cff9700 10 client.49727.objecter _finish_command 7 = system:0 2024-02-19T21:09:19.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.133+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230059360 osd=3 4 2024-02-19T21:09:19.132 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.133+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 3 7 2024-02-19T21:09:19.143 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:19.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.149+0000 7f02360a8700 10 client.49727.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:19.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.149+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 8 2024-02-19T21:09:19.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.149+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230059360 osd=3 3 2024-02-19T21:09:19.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.149+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230059360 osd=3 3 2024-02-19T21:09:19.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.149+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230059360 osd=3 3 2024-02-19T21:09:19.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.149+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 8 2024-02-19T21:09:19.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.149+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230059360 osd=3 3 2024-02-19T21:09:19.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.149+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 3 8 2024-02-19T21:09:19.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.149+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230059360 osd=3 4 2024-02-19T21:09:19.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.149+0000 7f02360a8700 10 client.49727.objecter _send_command 8 2024-02-19T21:09:19.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.149+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0230120f30 con 0x7f02300594d0 2024-02-19T21:09:19.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.151+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0230120f30 con 0x7f02300594d0 2024-02-19T21:09:19.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.151+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:19.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.151+0000 7f022cff9700 10 client.49727.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.151+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230059360 osd=3 4 2024-02-19T21:09:19.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.151+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 3 8 2024-02-19T21:09:19.149 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:09:19.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.152+0000 7f02360a8700 10 client.49727.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:19.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.152+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 9 2024-02-19T21:09:19.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.152+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230120f30 osd=4 3 2024-02-19T21:09:19.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.152+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230120f30 osd=4 3 2024-02-19T21:09:19.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.152+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230120f30 osd=4 3 2024-02-19T21:09:19.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.152+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 9 2024-02-19T21:09:19.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.152+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230120f30 osd=4 3 2024-02-19T21:09:19.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.152+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 4 9 2024-02-19T21:09:19.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.152+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230120f30 osd=4 4 2024-02-19T21:09:19.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.152+0000 7f02360a8700 10 client.49727.objecter _send_command 9 2024-02-19T21:09:19.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.152+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0230126560 con 0x7f02301210a0 2024-02-19T21:09:19.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.152+0000 7f022cff9700 10 client.49727.objecter ms_handle_connect 0x7f02301210a0 2024-02-19T21:09:19.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.153+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0230126560 con 0x7f02301210a0 2024-02-19T21:09:19.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.153+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:09:19.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.153+0000 7f022cff9700 10 client.49727.objecter _finish_command 9 = system:0 2024-02-19T21:09:19.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.153+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230120f30 osd=4 4 2024-02-19T21:09:19.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.153+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 4 9 2024-02-19T21:09:19.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.169+0000 7f02360a8700 10 client.49727.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:19.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.169+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 10 2024-02-19T21:09:19.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.169+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230120f30 osd=4 3 2024-02-19T21:09:19.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.169+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230120f30 osd=4 3 2024-02-19T21:09:19.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.169+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230120f30 osd=4 3 2024-02-19T21:09:19.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.169+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 10 2024-02-19T21:09:19.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.169+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230120f30 osd=4 3 2024-02-19T21:09:19.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.169+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 4 10 2024-02-19T21:09:19.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.169+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230120f30 osd=4 4 2024-02-19T21:09:19.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.169+0000 7f02360a8700 10 client.49727.objecter _send_command 10 2024-02-19T21:09:19.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.169+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0230126560 con 0x7f02301210a0 2024-02-19T21:09:19.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.171+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0230126560 con 0x7f02301210a0 2024-02-19T21:09:19.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.171+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:19.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.171+0000 7f022cff9700 10 client.49727.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.171+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230120f30 osd=4 4 2024-02-19T21:09:19.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.171+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 4 10 2024-02-19T21:09:19.169 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:09:19.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.172+0000 7f02360a8700 10 client.49727.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:19.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.172+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 11 2024-02-19T21:09:19.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.172+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230126560 osd=5 3 2024-02-19T21:09:19.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.172+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230126560 osd=5 3 2024-02-19T21:09:19.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.172+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230126560 osd=5 3 2024-02-19T21:09:19.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.172+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 11 2024-02-19T21:09:19.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.172+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230126560 osd=5 3 2024-02-19T21:09:19.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.172+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 5 11 2024-02-19T21:09:19.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.172+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230126560 osd=5 4 2024-02-19T21:09:19.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.172+0000 7f02360a8700 10 client.49727.objecter _send_command 11 2024-02-19T21:09:19.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.172+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0230132980 con 0x7f023012ed80 2024-02-19T21:09:19.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.172+0000 7f022cff9700 10 client.49727.objecter ms_handle_connect 0x7f023012ed80 2024-02-19T21:09:19.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.173+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0230132980 con 0x7f023012ed80 2024-02-19T21:09:19.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.173+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:09:19.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.173+0000 7f022cff9700 10 client.49727.objecter _finish_command 11 = system:0 2024-02-19T21:09:19.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.173+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230126560 osd=5 4 2024-02-19T21:09:19.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.173+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 5 11 2024-02-19T21:09:19.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.187+0000 7f02360a8700 10 client.49727.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:19.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.187+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 12 2024-02-19T21:09:19.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.187+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230126560 osd=5 3 2024-02-19T21:09:19.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.187+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230126560 osd=5 3 2024-02-19T21:09:19.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.187+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230126560 osd=5 3 2024-02-19T21:09:19.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.187+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 12 2024-02-19T21:09:19.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.187+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230126560 osd=5 3 2024-02-19T21:09:19.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.187+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 5 12 2024-02-19T21:09:19.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.187+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230126560 osd=5 4 2024-02-19T21:09:19.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.187+0000 7f02360a8700 10 client.49727.objecter _send_command 12 2024-02-19T21:09:19.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.187+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0230132980 con 0x7f023012ed80 2024-02-19T21:09:19.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0230132980 con 0x7f023012ed80 2024-02-19T21:09:19.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:19.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f022cff9700 10 client.49727.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230126560 osd=5 4 2024-02-19T21:09:19.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 5 12 2024-02-19T21:09:19.187 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:09:19.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f02360a8700 10 client.49727.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:19.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 13 2024-02-19T21:09:19.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230132980 osd=6 3 2024-02-19T21:09:19.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230132980 osd=6 3 2024-02-19T21:09:19.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230132980 osd=6 3 2024-02-19T21:09:19.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 13 2024-02-19T21:09:19.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230132980 osd=6 3 2024-02-19T21:09:19.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 6 13 2024-02-19T21:09:19.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230132980 osd=6 4 2024-02-19T21:09:19.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f02360a8700 10 client.49727.objecter _send_command 13 2024-02-19T21:09:19.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.189+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0230133c00 con 0x7f0230134640 2024-02-19T21:09:19.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.190+0000 7f022cff9700 10 client.49727.objecter ms_handle_connect 0x7f0230134640 2024-02-19T21:09:19.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.191+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0230133c00 con 0x7f0230134640 2024-02-19T21:09:19.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.191+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:09:19.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.191+0000 7f022cff9700 10 client.49727.objecter _finish_command 13 = system:0 2024-02-19T21:09:19.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.191+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230132980 osd=6 4 2024-02-19T21:09:19.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.191+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 6 13 2024-02-19T21:09:19.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.205+0000 7f02360a8700 10 client.49727.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:19.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.205+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 14 2024-02-19T21:09:19.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.205+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230132980 osd=6 3 2024-02-19T21:09:19.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.205+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230132980 osd=6 3 2024-02-19T21:09:19.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.205+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230132980 osd=6 3 2024-02-19T21:09:19.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.205+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 14 2024-02-19T21:09:19.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.205+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230132980 osd=6 3 2024-02-19T21:09:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.205+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 6 14 2024-02-19T21:09:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.205+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230132980 osd=6 4 2024-02-19T21:09:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.205+0000 7f02360a8700 10 client.49727.objecter _send_command 14 2024-02-19T21:09:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.205+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0230133c00 con 0x7f0230134640 2024-02-19T21:09:19.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.207+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0230133c00 con 0x7f0230134640 2024-02-19T21:09:19.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.207+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:19.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.207+0000 7f022cff9700 10 client.49727.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.207+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230132980 osd=6 4 2024-02-19T21:09:19.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.207+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 6 14 2024-02-19T21:09:19.206 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:09:19.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.208+0000 7f02360a8700 10 client.49727.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:19.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.208+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 15 2024-02-19T21:09:19.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.208+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230133c00 osd=7 3 2024-02-19T21:09:19.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.208+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230133c00 osd=7 3 2024-02-19T21:09:19.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.208+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230133c00 osd=7 3 2024-02-19T21:09:19.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.208+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 15 2024-02-19T21:09:19.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.208+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230133c00 osd=7 3 2024-02-19T21:09:19.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.208+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 7 15 2024-02-19T21:09:19.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.208+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230133c00 osd=7 4 2024-02-19T21:09:19.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.208+0000 7f02360a8700 10 client.49727.objecter _send_command 15 2024-02-19T21:09:19.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.208+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f023013b350 con 0x7f0230137840 2024-02-19T21:09:19.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.208+0000 7f022cff9700 10 client.49727.objecter ms_handle_connect 0x7f0230137840 2024-02-19T21:09:19.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.209+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f023013b350 con 0x7f0230137840 2024-02-19T21:09:19.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.209+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:09:19.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.209+0000 7f022cff9700 10 client.49727.objecter _finish_command 15 = system:0 2024-02-19T21:09:19.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.209+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230133c00 osd=7 4 2024-02-19T21:09:19.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.209+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 7 15 2024-02-19T21:09:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.223+0000 7f02360a8700 10 client.49727.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.223+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign -1 16 2024-02-19T21:09:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.223+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230133c00 osd=7 3 2024-02-19T21:09:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.223+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230133c00 osd=7 3 2024-02-19T21:09:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.223+0000 7f02360a8700 20 client.49727.objecter _get_session s=0x7f0230133c00 osd=7 3 2024-02-19T21:09:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.223+0000 7f02360a8700 15 client.49727.objecter _session_command_op_remove -1 16 2024-02-19T21:09:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.223+0000 7f02360a8700 20 client.49727.objecter get_session s=0x7f0230133c00 osd=7 3 2024-02-19T21:09:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.223+0000 7f02360a8700 15 client.49727.objecter _session_command_op_assign 7 16 2024-02-19T21:09:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.223+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230133c00 osd=7 4 2024-02-19T21:09:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.223+0000 7f02360a8700 10 client.49727.objecter _send_command 16 2024-02-19T21:09:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.223+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f023013b350 con 0x7f0230137840 2024-02-19T21:09:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.225+0000 7f022cff9700 1 -- 172.21.15.163:0/2623101106 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f023013b350 con 0x7f0230137840 2024-02-19T21:09:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.225+0000 7f022cff9700 10 client.49727.objecter ms_dispatch 0x7f0230001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.225+0000 7f022cff9700 10 client.49727.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.225+0000 7f022cff9700 20 client.49727.objecter put_session s=0x7f0230133c00 osd=7 4 2024-02-19T21:09:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.225+0000 7f022cff9700 15 client.49727.objecter _session_command_op_remove 7 16 2024-02-19T21:09:19.223 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:09:19.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.225+0000 7f02360a8700 10 librados: watch_flush enter 2024-02-19T21:09:19.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 10 librados: watch_flush exit 2024-02-19T21:09:19.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 10 client.49727.objecter close_session for osd.0 2024-02-19T21:09:19.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 >> v1:172.21.15.163:6801/1559418115 conn(0x7f023011a810 legacy=0x7f023011abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:19.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230047540 osd=0 1 2024-02-19T21:09:19.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 10 client.49727.objecter close_session for osd.1 2024-02-19T21:09:19.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 >> v1:172.21.15.163:6805/3125047432 conn(0x7f0230104400 legacy=0x7f02301047e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:19.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230103d80 osd=1 1 2024-02-19T21:09:19.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 10 client.49727.objecter close_session for osd.2 2024-02-19T21:09:19.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 >> v1:172.21.15.163:6809/1563158487 conn(0x7f0230059e00 legacy=0x7f0230123e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:19.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230119ff0 osd=2 1 2024-02-19T21:09:19.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 10 client.49727.objecter close_session for osd.3 2024-02-19T21:09:19.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 >> v1:172.21.15.163:6813/2112279769 conn(0x7f02300594d0 legacy=0x7f02301288a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:19.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230059360 osd=3 1 2024-02-19T21:09:19.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 10 client.49727.objecter close_session for osd.4 2024-02-19T21:09:19.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 >> v1:172.21.15.193:6800/3129716965 conn(0x7f02301210a0 legacy=0x7f023012d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:19.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230120f30 osd=4 1 2024-02-19T21:09:19.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 10 client.49727.objecter close_session for osd.5 2024-02-19T21:09:19.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f022ffff700 1 -- 172.21.15.163:0/2623101106 reap_dead start 2024-02-19T21:09:19.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 >> v1:172.21.15.193:6804/805161991 conn(0x7f023012ed80 legacy=0x7f0230131170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:19.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f022ffff700 1 -- 172.21.15.163:0/2623101106 reap_dead start 2024-02-19T21:09:19.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f022ffff700 1 -- 172.21.15.163:0/2623101106 reap_dead start 2024-02-19T21:09:19.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230126560 osd=5 1 2024-02-19T21:09:19.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 10 client.49727.objecter close_session for osd.6 2024-02-19T21:09:19.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 >> v1:172.21.15.193:6808/787604615 conn(0x7f0230134640 legacy=0x7f0230134af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:19.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230132980 osd=6 1 2024-02-19T21:09:19.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 10 client.49727.objecter close_session for osd.7 2024-02-19T21:09:19.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 >> v1:172.21.15.193:6812/157927432 conn(0x7f0230137840 legacy=0x7f0230139c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:19.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 20 client.49727.objecter put_session s=0x7f0230133c00 osd=7 1 2024-02-19T21:09:19.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 20 client.49727.objecter shutdown clearing up homeless session... 2024-02-19T21:09:19.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 10 client.49727.objecter successfully canceled tick 2024-02-19T21:09:19.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 >> v1:172.21.15.163:6800/1552514681 conn(0x7f0218063dc0 legacy=0x7f0218066280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:19.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 >> v1:172.21.15.163:6790/0 conn(0x7f02300fbfd0 legacy=0x7f0230106fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:19.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f022ffff700 1 -- 172.21.15.163:0/2623101106 reap_dead start 2024-02-19T21:09:19.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 shutdown_connections 2024-02-19T21:09:19.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 >> 172.21.15.163:0/2623101106 conn(0x7f023006cc00 msgr2=0x7f023006d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:19.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 shutdown_connections 2024-02-19T21:09:19.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 1 -- 172.21.15.163:0/2623101106 wait complete. 2024-02-19T21:09:19.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.226+0000 7f02360a8700 1 librados: shutdown 2024-02-19T21:09:19.228 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.229 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.229 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.229 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.229 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.229 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.229 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.229 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:09:19.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.491+0000 7f09537fe700 1 -- 172.21.15.163:0/4273750768 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v379) v1 ==== 113+0+0 (unknown 2613564318 0 0) 0x7f094c04a520 con 0x7f095c100780 2024-02-19T21:09:19.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.493+0000 7f09647b3700 10 librados: watch_flush enter 2024-02-19T21:09:19.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.493+0000 7f09647b3700 10 librados: watch_flush exit 2024-02-19T21:09:19.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.493+0000 7f09647b3700 20 client.49721.objecter shutdown clearing up homeless session... 2024-02-19T21:09:19.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.493+0000 7f09647b3700 10 client.49721.objecter successfully canceled tick 2024-02-19T21:09:19.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.493+0000 7f09647b3700 1 -- 172.21.15.163:0/4273750768 >> v1:172.21.15.163:6800/1552514681 conn(0x7f09480681e0 legacy=0x7f094806a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:19.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.493+0000 7f09647b3700 1 -- 172.21.15.163:0/4273750768 >> v1:172.21.15.163:6790/0 conn(0x7f095c100780 legacy=0x7f095c0fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:19.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.493+0000 7f09647b3700 1 -- 172.21.15.163:0/4273750768 shutdown_connections 2024-02-19T21:09:19.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.493+0000 7f09647b3700 1 -- 172.21.15.163:0/4273750768 >> 172.21.15.163:0/4273750768 conn(0x7f095c0f35a0 msgr2=0x7f095c107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:19.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.493+0000 7f09647b3700 1 -- 172.21.15.163:0/4273750768 shutdown_connections 2024-02-19T21:09:19.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.493+0000 7f09647b3700 1 -- 172.21.15.163:0/4273750768 wait complete. 2024-02-19T21:09:19.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:19.494+0000 7f09647b3700 1 librados: shutdown 2024-02-19T21:09:19.492 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.5 to 1 (10000) 2024-02-19T21:09:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:17.599733+0000 osd.3 (osd.3) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:17.599863+0000 osd.3 (osd.3) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:18.266872+0000 mon.c (mon.2) 377 : audit [INF] from='client.? v1:172.21.15.163:0/4273750768' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:09:19.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:18.267124+0000 mon.a (mon.0) 2408 : audit [INF] from='client.49721 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:09:19.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:19.064679+0000 mon.c (mon.2) 378 : audit [DBG] from='client.? v1:172.21.15.163:0/2623101106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:19.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:17.599733+0000 osd.3 (osd.3) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:19.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:17.599863+0000 osd.3 (osd.3) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:19.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:18.266872+0000 mon.c (mon.2) 377 : audit [INF] from='client.? v1:172.21.15.163:0/4273750768' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:09:19.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:18.267124+0000 mon.a (mon.0) 2408 : audit [INF] from='client.49721 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:09:19.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:19.064679+0000 mon.c (mon.2) 378 : audit [DBG] from='client.? v1:172.21.15.163:0/2623101106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:19.995 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:17.599733+0000 osd.3 (osd.3) 157 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:19.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:17.599863+0000 osd.3 (osd.3) 158 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:19.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:18.266872+0000 mon.c (mon.2) 377 : audit [INF] from='client.? v1:172.21.15.163:0/4273750768' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:09:19.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:18.267124+0000 mon.a (mon.0) 2408 : audit [INF] from='client.49721 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:09:19.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:19.064679+0000 mon.c (mon.2) 378 : audit [DBG] from='client.? v1:172.21.15.163:0/2623101106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:20.092 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:09:20.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:20.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:09:20.197 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:20.226 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:20.575 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:20.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:20.595 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 6 1 2024-02-19T21:09:20.659 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:19.197140+0000 mgr.y (mgr.14550) 343 : cluster [DBG] pgmap v539: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:20.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:19.492111+0000 mon.a (mon.0) 2409 : audit [INF] from='client.49721 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-19T21:09:20.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:19.492182+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2024-02-19T21:09:20.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:19.492417+0000 mon.a (mon.0) 2411 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-19T21:09:20.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:20.256016+0000 mon.a (mon.0) 2412 : audit [INF] from='client.49715 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi163-83854-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi163-83854-65"}]': finished 2024-02-19T21:09:20.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:20.256107+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2024-02-19T21:09:20.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:19.197140+0000 mgr.y (mgr.14550) 343 : cluster [DBG] pgmap v539: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:20.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:19.492111+0000 mon.a (mon.0) 2409 : audit [INF] from='client.49721 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-19T21:09:20.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:19.492182+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2024-02-19T21:09:20.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:19.492417+0000 mon.a (mon.0) 2411 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-19T21:09:20.660 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:20.256016+0000 mon.a (mon.0) 2412 : audit [INF] from='client.49715 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi163-83854-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi163-83854-65"}]': finished 2024-02-19T21:09:20.661 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:20.256107+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2024-02-19T21:09:20.679 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:09:20.705 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:20.759 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:09:20.762 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:20.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:09:20.818 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:09:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:09:20.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:19.197140+0000 mgr.y (mgr.14550) 343 : cluster [DBG] pgmap v539: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:20.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:19.492111+0000 mon.a (mon.0) 2409 : audit [INF] from='client.49721 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-19T21:09:20.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:19.492182+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2024-02-19T21:09:20.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:19.492417+0000 mon.a (mon.0) 2411 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-19T21:09:20.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:20.256016+0000 mon.a (mon.0) 2412 : audit [INF] from='client.49715 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi163-83854-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi163-83854-65"}]': finished 2024-02-19T21:09:20.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:20.256107+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2024-02-19T21:09:20.863 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:09:20.897 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:20.918 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:09:20.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:20.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:09:21.024 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:09:21.052 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:21.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:20.340462+0000 mgr.y (mgr.14550) 344 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:21.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:21.253416+0000 mon.a (mon.0) 2414 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2024-02-19T21:09:21.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:20.340462+0000 mgr.y (mgr.14550) 344 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:21.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:21.253416+0000 mon.a (mon.0) 2414 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2024-02-19T21:09:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:20.340462+0000 mgr.y (mgr.14550) 344 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:21.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:21.253416+0000 mon.a (mon.0) 2414 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2024-02-19T21:09:22.501 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:09:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:22.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:09:22.606 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:22.638 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:22.659 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:22.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:22.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:22.763 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:09:22.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:21.197557+0000 mgr.y (mgr.14550) 345 : cluster [DBG] pgmap v542: 236 pgs: 8 unknown, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:09:22.787 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:22.254205+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2024-02-19T21:09:22.787 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:22.256849+0000 mon.c (mon.2) 379 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:22.787 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:22.257433+0000 mon.a (mon.0) 2416 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:22.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:21.197557+0000 mgr.y (mgr.14550) 345 : cluster [DBG] pgmap v542: 236 pgs: 8 unknown, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:09:22.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:22.254205+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2024-02-19T21:09:22.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:22.256849+0000 mon.c (mon.2) 379 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:22.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:22.257433+0000 mon.a (mon.0) 2416 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:22.796 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:22.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:21.197557+0000 mgr.y (mgr.14550) 345 : cluster [DBG] pgmap v542: 236 pgs: 8 unknown, 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:09:22.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:22.254205+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2024-02-19T21:09:22.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:22.256849+0000 mon.c (mon.2) 379 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:22.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:22.257433+0000 mon.a (mon.0) 2416 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:22.819 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:09:22.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:22.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:09:22.923 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:22.951 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:23.075 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:23.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:23.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:23.180 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:23.209 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:23.307 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:23.309 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:23.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:23.317 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:09:23.317 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:09:23.411 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:23.442 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:23.464 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:23.467 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:23.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:23.569 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:23.594 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:09:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:09:23] "GET /metrics HTTP/1.1" 200 186694 "" "Prometheus/2.33.4" 2024-02-19T21:09:23.597 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:23.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.796+0000 7f7edccc9700 1 -- 172.21.15.163:0/2278285948 >> v1:172.21.15.163:6789/0 conn(0x7f7ed80fc870 legacy=0x7f7ed80fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:23.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.796+0000 7f7edccc9700 1 -- 172.21.15.163:0/2278285948 shutdown_connections 2024-02-19T21:09:23.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.796+0000 7f7edccc9700 1 -- 172.21.15.163:0/2278285948 >> 172.21.15.163:0/2278285948 conn(0x7f7ed80f2470 msgr2=0x7f7ed80f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:23.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.796+0000 7f7edccc9700 1 -- 172.21.15.163:0/2278285948 shutdown_connections 2024-02-19T21:09:23.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.796+0000 7f7edccc9700 1 -- 172.21.15.163:0/2278285948 wait complete. 2024-02-19T21:09:23.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.796+0000 7f7edccc9700 1 librados: starting msgr at 2024-02-19T21:09:23.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.796+0000 7f7edccc9700 1 librados: starting objecter 2024-02-19T21:09:23.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.796+0000 7f7edccc9700 1 Processor -- start 2024-02-19T21:09:23.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.796+0000 7f7edccc9700 1 -- start start 2024-02-19T21:09:23.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.796+0000 7f7edccc9700 1 librados: setting wanted keys 2024-02-19T21:09:23.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.796+0000 7f7edccc9700 1 librados: calling monclient init 2024-02-19T21:09:23.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7edccc9700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7ed80fbec0 con 0x7f7ed80fc870 2024-02-19T21:09:23.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7edccc9700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7ed80fc090 con 0x7f7ed81031b0 2024-02-19T21:09:23.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7edccc9700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7ed80fc280 con 0x7f7ed80ff670 2024-02-19T21:09:23.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ed659c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f7ed80fc870 0x7f7ed80fb720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44278/0 (socket says 172.21.15.163:44278) 2024-02-19T21:09:23.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ed659c700 1 -- 172.21.15.163:0/504451428 learned_addr learned my addr 172.21.15.163:0/504451428 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:23.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ec77fe700 10 client.?.objecter ms_handle_connect 0x7f7ed80ff670 2024-02-19T21:09:23.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ec77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:23.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716329442 0 0) 0x7f7ed80fc280 con 0x7f7ed80ff670 2024-02-19T21:09:23.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec00033f0 con 0x7f7ed80ff670 2024-02-19T21:09:23.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ec77fe700 10 client.?.objecter ms_handle_connect 0x7f7ed81031b0 2024-02-19T21:09:23.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ec77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:23.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113222108 0 0) 0x7f7ed80fc090 con 0x7f7ed81031b0 2024-02-19T21:09:23.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed80fc280 con 0x7f7ed81031b0 2024-02-19T21:09:23.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3846637607 0 0) 0x7f7ec00033f0 con 0x7f7ed80ff670 2024-02-19T21:09:23.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ed80fc090 con 0x7f7ed80ff670 2024-02-19T21:09:23.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7ec8001950 con 0x7f7ed80ff670 2024-02-19T21:09:23.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2106773234 0 0) 0x7f7ed80fc280 con 0x7f7ed81031b0 2024-02-19T21:09:23.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ec00033f0 con 0x7f7ed81031b0 2024-02-19T21:09:23.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3195968345 0 0) 0x7f7ed80fc090 con 0x7f7ed80ff670 2024-02-19T21:09:23.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.797+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 >> v1:172.21.15.193:6789/0 conn(0x7f7ed81031b0 legacy=0x7f7ed8118340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:23.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.798+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 >> v1:172.21.15.163:6789/0 conn(0x7f7ed80fc870 legacy=0x7f7ed80fb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:23.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.798+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed811cad0 con 0x7f7ed80ff670 2024-02-19T21:09:23.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.798+0000 7f7edccc9700 1 -- 172.21.15.163:0/504451428 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed8119ac0 con 0x7f7ed80ff670 2024-02-19T21:09:23.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.798+0000 7f7edccc9700 10 client.49733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:23.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.798+0000 7f7edccc9700 1 -- 172.21.15.163:0/504451428 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed8119f30 con 0x7f7ed80ff670 2024-02-19T21:09:23.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.798+0000 7f7edccc9700 1 librados: init done 2024-02-19T21:09:23.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.798+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f7ec80023a0 con 0x7f7ed80ff670 2024-02-19T21:09:23.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.798+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7ec8003340 con 0x7f7ed80ff670 2024-02-19T21:09:23.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.799+0000 7f7edccc9700 1 -- 172.21.15.163:0/504451428 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb8004f20 con 0x7f7ed80ff670 2024-02-19T21:09:23.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.799+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f7ec8016290 con 0x7f7ed80ff670 2024-02-19T21:09:23.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.799+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(383..383 src has 43..383) v4 ==== 8043+0+0 (unknown 859327231 0 0) 0x7f7ec8076550 con 0x7f7ed80ff670 2024-02-19T21:09:23.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.799+0000 7f7ec77fe700 10 client.49733.objecter ms_dispatch 0x7f7ed8001550 osd_map(383..383 src has 43..383) v4 2024-02-19T21:09:23.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.799+0000 7f7ec77fe700 3 client.49733.objecter handle_osd_map got epochs [383,383] > 0 2024-02-19T21:09:23.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.799+0000 7f7ec77fe700 3 client.49733.objecter handle_osd_map decoding full epoch 383 2024-02-19T21:09:23.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.799+0000 7f7ec77fe700 20 client.49733.objecter dump_active .. 0 homeless 2024-02-19T21:09:23.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.802+0000 7f7ec77fe700 10 client.49733.objecter ms_handle_connect 0x7f7ec0063a70 2024-02-19T21:09:23.801 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:23.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.804+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f7ec80499b0 con 0x7f7ed80ff670 2024-02-19T21:09:23.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:23.984+0000 7f7edccc9700 1 -- 172.21.15.163:0/504451428 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f7eb80051d0 con 0x7f7ed80ff670 2024-02-19T21:09:24.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:24.248+0000 7f7ec77fe700 1 -- 172.21.15.163:0/504451428 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v384) v1 ==== 91+0+0 (unknown 3388196544 0 0) 0x7f7ec8047dc0 con 0x7f7ed80ff670 2024-02-19T21:09:24.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:24.251+0000 7f7edccc9700 10 librados: watch_flush enter 2024-02-19T21:09:24.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:24.251+0000 7f7edccc9700 10 librados: watch_flush exit 2024-02-19T21:09:24.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:24.251+0000 7f7edccc9700 20 client.49733.objecter shutdown clearing up homeless session... 2024-02-19T21:09:24.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:24.251+0000 7f7edccc9700 10 client.49733.objecter successfully canceled tick 2024-02-19T21:09:24.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:24.251+0000 7f7edccc9700 1 -- 172.21.15.163:0/504451428 >> v1:172.21.15.163:6800/1552514681 conn(0x7f7ec0063a70 legacy=0x7f7ec0065f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:24.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:24.251+0000 7f7edccc9700 1 -- 172.21.15.163:0/504451428 >> v1:172.21.15.163:6790/0 conn(0x7f7ed80ff670 legacy=0x7f7ed8114c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:24.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:24.251+0000 7f7edccc9700 1 -- 172.21.15.163:0/504451428 shutdown_connections 2024-02-19T21:09:24.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:24.251+0000 7f7edccc9700 1 -- 172.21.15.163:0/504451428 >> 172.21.15.163:0/504451428 conn(0x7f7ed80f2470 msgr2=0x7f7ed8106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:24.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:24.251+0000 7f7edccc9700 1 -- 172.21.15.163:0/504451428 shutdown_connections 2024-02-19T21:09:24.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:24.251+0000 7f7edccc9700 1 -- 172.21.15.163:0/504451428 wait complete. 2024-02-19T21:09:24.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:24.251+0000 7f7edccc9700 1 librados: shutdown 2024-02-19T21:09:24.250 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-02-19T21:09:24.250 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6923 ms) 2024-02-19T21:09:24.250 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-02-19T21:09:24.250 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7012 ms) 2024-02-19T21:09:24.250 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-02-19T21:09:24.251 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7016 ms) 2024-02-19T21:09:24.251 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-02-19T21:09:24.251 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6982 ms) 2024-02-19T21:09:24.251 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-02-19T21:09:24.251 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (6999 ms) 2024-02-19T21:09:24.251 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139046 ms total) 2024-02-19T21:09:24.251 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: 2024-02-19T21:09:24.251 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-02-19T21:09:24.251 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [==========] 56 tests from 4 test suites ran. (291415 ms total) 2024-02-19T21:09:24.251 INFO:tasks.workunit.client.0.smithi163.stdout: api_aio_pp: [ PASSED ] 56 tests. 2024-02-19T21:09:24.254 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:09:24.254 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83867 2024-02-19T21:09:24.254 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83867 2024-02-19T21:09:24.254 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:09:24.255 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83931 2024-02-19T21:09:24.255 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83931 2024-02-19T21:09:24.255 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:09:24.255 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83915 2024-02-19T21:09:24.255 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83915 2024-02-19T21:09:24.255 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:09:24.255 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83891 2024-02-19T21:09:24.255 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83891 2024-02-19T21:09:24.255 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:09:24.255 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83987 2024-02-19T21:09:24.255 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83987 2024-02-19T21:09:24.255 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:09:24.255 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83859 2024-02-19T21:09:24.255 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83859 2024-02-19T21:09:24.256 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:09:24.256 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=84003 2024-02-19T21:09:24.256 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 84003 2024-02-19T21:09:24.256 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:09:24.256 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83955 2024-02-19T21:09:24.256 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83955 2024-02-19T21:09:24.256 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:09:24.256 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=84035 2024-02-19T21:09:24.256 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 84035 2024-02-19T21:09:24.256 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:09:24.256 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83995 2024-02-19T21:09:24.256 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83995 2024-02-19T21:09:24.256 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:09:24.256 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=84051 2024-02-19T21:09:24.256 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 84051 2024-02-19T21:09:24.257 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:09:24.257 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83840 2024-02-19T21:09:24.257 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83840 2024-02-19T21:09:24.257 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:09:24.257 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=84043 2024-02-19T21:09:24.257 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 84043 2024-02-19T21:09:24.257 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:09:24.257 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=84011 2024-02-19T21:09:24.257 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 84011 2024-02-19T21:09:24.257 INFO:tasks.workunit.client.0.smithi163.stderr:+ for t in "${!pids[@]}" 2024-02-19T21:09:24.257 INFO:tasks.workunit.client.0.smithi163.stderr:+ pid=83923 2024-02-19T21:09:24.257 INFO:tasks.workunit.client.0.smithi163.stderr:+ wait 83923 2024-02-19T21:09:24.257 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:09:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:23.198101+0000 mgr.y (mgr.14550) 346 : cluster [DBG] pgmap v545: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:09:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:23.257353+0000 mon.a (mon.0) 2417 : audit [INF] from='client.49715 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi163-83854-65"}]': finished 2024-02-19T21:09:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:23.257414+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2024-02-19T21:09:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:23.262277+0000 mon.c (mon.2) 380 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:23.262584+0000 mon.a (mon.0) 2419 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:23.985485+0000 mon.c (mon.2) 381 : audit [INF] from='client.? v1:172.21.15.163:0/504451428' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:09:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:23.985754+0000 mon.a (mon.0) 2420 : audit [INF] from='client.49733 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:09:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:24.100962+0000 mon.a (mon.0) 2421 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:24.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:24.240768+0000 mon.a (mon.0) 2422 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:09:24.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:24.249198+0000 mon.a (mon.0) 2423 : audit [INF] from='client.49715 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi163-83854-65"}]': finished 2024-02-19T21:09:24.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:24.249285+0000 mon.a (mon.0) 2424 : audit [INF] from='client.49733 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:09:24.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:24.249342+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2024-02-19T21:09:24.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:23.198101+0000 mgr.y (mgr.14550) 346 : cluster [DBG] pgmap v545: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:09:24.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:23.257353+0000 mon.a (mon.0) 2417 : audit [INF] from='client.49715 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi163-83854-65"}]': finished 2024-02-19T21:09:24.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:23.257414+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2024-02-19T21:09:24.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:23.262277+0000 mon.c (mon.2) 380 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:24.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:23.262584+0000 mon.a (mon.0) 2419 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:24.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:23.985485+0000 mon.c (mon.2) 381 : audit [INF] from='client.? v1:172.21.15.163:0/504451428' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:09:24.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:23.985754+0000 mon.a (mon.0) 2420 : audit [INF] from='client.49733 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:09:24.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:24.100962+0000 mon.a (mon.0) 2421 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:24.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:24.240768+0000 mon.a (mon.0) 2422 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:09:24.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:24.249198+0000 mon.a (mon.0) 2423 : audit [INF] from='client.49715 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi163-83854-65"}]': finished 2024-02-19T21:09:24.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:24.249285+0000 mon.a (mon.0) 2424 : audit [INF] from='client.49733 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:09:24.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:24.249342+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2024-02-19T21:09:24.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:23.198101+0000 mgr.y (mgr.14550) 346 : cluster [DBG] pgmap v545: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:09:24.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:23.257353+0000 mon.a (mon.0) 2417 : audit [INF] from='client.49715 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi163-83854-65"}]': finished 2024-02-19T21:09:24.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:23.257414+0000 mon.a (mon.0) 2418 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2024-02-19T21:09:24.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:23.262277+0000 mon.c (mon.2) 380 : audit [INF] from='client.? v1:172.21.15.163:0/3723307207' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:24.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:23.262584+0000 mon.a (mon.0) 2419 : audit [INF] from='client.49715 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi163-83854-65"}]: dispatch 2024-02-19T21:09:24.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:23.985485+0000 mon.c (mon.2) 381 : audit [INF] from='client.? v1:172.21.15.163:0/504451428' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:09:24.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:23.985754+0000 mon.a (mon.0) 2420 : audit [INF] from='client.49733 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:09:24.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:24.100962+0000 mon.a (mon.0) 2421 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:24.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:24.240768+0000 mon.a (mon.0) 2422 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:09:24.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:24.249198+0000 mon.a (mon.0) 2423 : audit [INF] from='client.49715 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi163-83854-65"}]': finished 2024-02-19T21:09:24.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:24.249285+0000 mon.a (mon.0) 2424 : audit [INF] from='client.49733 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:09:24.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:24.249342+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2024-02-19T21:09:24.669 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-19T21:09:24.813 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:09:26.147 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:26.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:26.252 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:26.284 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:26.306 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:26.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:26.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:26.410 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:09:26.441 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:26.461 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:09:26.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:26.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:09:26.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:25.199218+0000 mgr.y (mgr.14550) 347 : cluster [DBG] pgmap v548: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:09:26.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:25.199218+0000 mgr.y (mgr.14550) 347 : cluster [DBG] pgmap v548: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:09:26.566 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:26.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:25.199218+0000 mgr.y (mgr.14550) 347 : cluster [DBG] pgmap v548: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:09:26.594 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:26.944 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:26.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:26.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:27.048 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:27.076 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:27.095 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:27.100 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:27.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:27.202 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:27.230 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:27.256 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:27.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:27.362 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:27.388 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:25.730065+0000 osd.7 (osd.7) 153 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:09:27.388 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:25.732951+0000 osd.7 (osd.7) 154 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:09:27.388 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:25.986815+0000 osd.2 (osd.2) 134 : cluster [DBG] 61.1f scrub starts 2024-02-19T21:09:27.389 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:25.988969+0000 osd.2 (osd.2) 135 : cluster [DBG] 61.1f scrub ok 2024-02-19T21:09:27.389 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:26.416190+0000 osd.6 (osd.6) 143 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:09:27.389 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:26.417243+0000 osd.6 (osd.6) 144 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:09:27.389 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:26.431613+0000 mon.b (mon.1) 545 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:09:27.389 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:27.067406+0000 mon.b (mon.1) 546 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:09:27.389 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:27.068239+0000 mon.b (mon.1) 547 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:09:27.389 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:27.072692+0000 mon.a (mon.0) 2426 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:27.390 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:27.410 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:27.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:27.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.497+0000 7f372b1f0700 1 -- 172.21.15.163:0/2543355917 >> v1:172.21.15.163:6789/0 conn(0x7f37240fc870 legacy=0x7f37240fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:27.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.497+0000 7f372b1f0700 1 -- 172.21.15.163:0/2543355917 shutdown_connections 2024-02-19T21:09:27.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.497+0000 7f372b1f0700 1 -- 172.21.15.163:0/2543355917 >> 172.21.15.163:0/2543355917 conn(0x7f37240f2470 msgr2=0x7f37240f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:27.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.497+0000 7f372b1f0700 1 -- 172.21.15.163:0/2543355917 shutdown_connections 2024-02-19T21:09:27.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.497+0000 7f372b1f0700 1 -- 172.21.15.163:0/2543355917 wait complete. 2024-02-19T21:09:27.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:25.730065+0000 osd.7 (osd.7) 153 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:09:27.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:25.732951+0000 osd.7 (osd.7) 154 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:09:27.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:25.986815+0000 osd.2 (osd.2) 134 : cluster [DBG] 61.1f scrub starts 2024-02-19T21:09:27.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:25.988969+0000 osd.2 (osd.2) 135 : cluster [DBG] 61.1f scrub ok 2024-02-19T21:09:27.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:26.416190+0000 osd.6 (osd.6) 143 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:09:27.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:26.417243+0000 osd.6 (osd.6) 144 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:09:27.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:26.431613+0000 mon.b (mon.1) 545 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:09:27.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:27.067406+0000 mon.b (mon.1) 546 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:09:27.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:27.068239+0000 mon.b (mon.1) 547 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:09:27.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:27.072692+0000 mon.a (mon.0) 2426 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.497+0000 7f372b1f0700 1 librados: starting msgr at 2024-02-19T21:09:27.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.497+0000 7f372b1f0700 1 librados: starting objecter 2024-02-19T21:09:27.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:25.730065+0000 osd.7 (osd.7) 153 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:09:27.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:25.732951+0000 osd.7 (osd.7) 154 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:09:27.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:25.986815+0000 osd.2 (osd.2) 134 : cluster [DBG] 61.1f scrub starts 2024-02-19T21:09:27.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:25.988969+0000 osd.2 (osd.2) 135 : cluster [DBG] 61.1f scrub ok 2024-02-19T21:09:27.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:26.416190+0000 osd.6 (osd.6) 143 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:09:27.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:26.417243+0000 osd.6 (osd.6) 144 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:09:27.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:26.431613+0000 mon.b (mon.1) 545 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:09:27.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:27.067406+0000 mon.b (mon.1) 546 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:09:27.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:27.068239+0000 mon.b (mon.1) 547 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:09:27.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:27.072692+0000 mon.a (mon.0) 2426 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.497+0000 7f372b1f0700 1 Processor -- start 2024-02-19T21:09:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.497+0000 7f372b1f0700 1 -- start start 2024-02-19T21:09:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.497+0000 7f372b1f0700 1 librados: setting wanted keys 2024-02-19T21:09:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.497+0000 7f372b1f0700 1 librados: calling monclient init 2024-02-19T21:09:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f372b1f0700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f372406ab20 con 0x7f37240ff670 2024-02-19T21:09:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f372b1f0700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f372406acf0 con 0x7f37240fc870 2024-02-19T21:09:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f372b1f0700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3724120f20 con 0x7f37241031b0 2024-02-19T21:09:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3728f8c700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f37240fc870 0x7f37240f8bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:37830/0 (socket says 172.21.15.163:37830) 2024-02-19T21:09:27.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3723fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f37240ff670 0x7f37240f92e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57038/0 (socket says 172.21.15.163:57038) 2024-02-19T21:09:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3728f8c700 1 -- 172.21.15.163:0/3934602217 learned_addr learned my addr 172.21.15.163:0/3934602217 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 10 client.?.objecter ms_handle_connect 0x7f37241031b0 2024-02-19T21:09:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 10 client.?.objecter ms_handle_connect 0x7f37240fc870 2024-02-19T21:09:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 10 client.?.objecter ms_handle_connect 0x7f37240ff670 2024-02-19T21:09:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521539781 0 0) 0x7f372406acf0 con 0x7f37240fc870 2024-02-19T21:09:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3714003030 con 0x7f37240fc870 2024-02-19T21:09:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957892881 0 0) 0x7f3724120f20 con 0x7f37241031b0 2024-02-19T21:09:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f372406acf0 con 0x7f37241031b0 2024-02-19T21:09:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458870694 0 0) 0x7f372406ab20 con 0x7f37240ff670 2024-02-19T21:09:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3724120f20 con 0x7f37240ff670 2024-02-19T21:09:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1326065627 0 0) 0x7f3714003030 con 0x7f37240fc870 2024-02-19T21:09:27.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f372406ab20 con 0x7f37240fc870 2024-02-19T21:09:27.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f37180021f0 con 0x7f37240fc870 2024-02-19T21:09:27.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2722858043 0 0) 0x7f372406acf0 con 0x7f37241031b0 2024-02-19T21:09:27.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3714003030 con 0x7f37241031b0 2024-02-19T21:09:27.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f371c001ba0 con 0x7f37241031b0 2024-02-19T21:09:27.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3159559028 0 0) 0x7f3724120f20 con 0x7f37240ff670 2024-02-19T21:09:27.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f372406acf0 con 0x7f37240ff670 2024-02-19T21:09:27.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3710001800 con 0x7f37240ff670 2024-02-19T21:09:27.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 672125131 0 0) 0x7f372406ab20 con 0x7f37240fc870 2024-02-19T21:09:27.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.498+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 >> v1:172.21.15.163:6790/0 conn(0x7f37241031b0 legacy=0x7f372411d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:27.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.499+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 >> v1:172.21.15.163:6789/0 conn(0x7f37240ff670 legacy=0x7f37240f92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:27.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.499+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3724122100 con 0x7f37240fc870 2024-02-19T21:09:27.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.499+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3718002830 con 0x7f37240fc870 2024-02-19T21:09:27.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.499+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f37180039f0 con 0x7f37240fc870 2024-02-19T21:09:27.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.499+0000 7f372b1f0700 1 -- 172.21.15.163:0/3934602217 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3724121110 con 0x7f37240fc870 2024-02-19T21:09:27.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.499+0000 7f372b1f0700 10 client.48556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:27.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.499+0000 7f372b1f0700 1 -- 172.21.15.163:0/3934602217 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3724121600 con 0x7f37240fc870 2024-02-19T21:09:27.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.499+0000 7f372b1f0700 1 librados: init done 2024-02-19T21:09:27.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.500+0000 7f372b1f0700 1 -- 172.21.15.163:0/3934602217 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f372405f700 con 0x7f37240fc870 2024-02-19T21:09:27.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.500+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f37180018c0 con 0x7f37240fc870 2024-02-19T21:09:27.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.501+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(384..384 src has 43..384) v4 ==== 7936+0+0 (unknown 3410506564 0 0) 0x7f3718079110 con 0x7f37240fc870 2024-02-19T21:09:27.503 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.501+0000 7f3721ffb700 10 client.48556.objecter ms_dispatch 0x7f3724001550 osd_map(384..384 src has 43..384) v4 2024-02-19T21:09:27.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.501+0000 7f3721ffb700 3 client.48556.objecter handle_osd_map got epochs [384,384] > 0 2024-02-19T21:09:27.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.501+0000 7f3721ffb700 3 client.48556.objecter handle_osd_map decoding full epoch 384 2024-02-19T21:09:27.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.501+0000 7f3721ffb700 20 client.48556.objecter dump_active .. 0 homeless 2024-02-19T21:09:27.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.501+0000 7f3721ffb700 10 client.48556.objecter ms_handle_connect 0x7f3714063d20 2024-02-19T21:09:27.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.505+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f371804c5e0 con 0x7f37240fc870 2024-02-19T21:09:27.513 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:27.541 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:27.556 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:27.559 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:27.561 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:27.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:27.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:27.663 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:27.689 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:27.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:27.702+0000 7f372b1f0700 1 -- 172.21.15.163:0/3934602217 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f37240f80a0 con 0x7f37240fc870 2024-02-19T21:09:27.707 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:27.709 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:27.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:27.811 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:09:27.840 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:27.856 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:09:27.858 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:27.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:09:27.959 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:27.989 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:28.068 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:09:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:09:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:09:28.137 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:09:28.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:28.265+0000 7f3721ffb700 1 -- 172.21.15.163:0/3934602217 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v385) v1 ==== 113+0+0 (unknown 3414280535 0 0) 0x7f371804aa50 con 0x7f37240fc870 2024-02-19T21:09:28.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:28.268+0000 7f372b1f0700 10 librados: watch_flush enter 2024-02-19T21:09:28.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:28.268+0000 7f372b1f0700 10 librados: watch_flush exit 2024-02-19T21:09:28.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:28.268+0000 7f372b1f0700 20 client.48556.objecter shutdown clearing up homeless session... 2024-02-19T21:09:28.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:28.268+0000 7f372b1f0700 10 client.48556.objecter successfully canceled tick 2024-02-19T21:09:28.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:28.268+0000 7f372b1f0700 1 -- 172.21.15.163:0/3934602217 >> v1:172.21.15.163:6800/1552514681 conn(0x7f3714063d20 legacy=0x7f37140661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:28.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:28.268+0000 7f372b1f0700 1 -- 172.21.15.163:0/3934602217 >> v1:172.21.15.193:6789/0 conn(0x7f37240fc870 legacy=0x7f37240f8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:28.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:28.270+0000 7f372b1f0700 1 -- 172.21.15.163:0/3934602217 shutdown_connections 2024-02-19T21:09:28.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:28.270+0000 7f372b1f0700 1 -- 172.21.15.163:0/3934602217 >> 172.21.15.163:0/3934602217 conn(0x7f37240f2470 msgr2=0x7f3724106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:28.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:28.270+0000 7f372b1f0700 1 -- 172.21.15.163:0/3934602217 shutdown_connections 2024-02-19T21:09:28.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:28.270+0000 7f372b1f0700 1 -- 172.21.15.163:0/3934602217 wait complete. 2024-02-19T21:09:28.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:28.273+0000 7f372b1f0700 1 librados: shutdown 2024-02-19T21:09:28.272 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.6 to 1 (10000) 2024-02-19T21:09:28.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:26.728722+0000 osd.7 (osd.7) 155 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:09:28.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:26.731857+0000 osd.7 (osd.7) 156 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:09:28.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:26.912544+0000 osd.5 (osd.5) 148 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:09:28.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:26.915998+0000 osd.5 (osd.5) 149 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:09:28.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:27.199699+0000 mgr.y (mgr.14550) 348 : cluster [DBG] pgmap v549: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 860 B/s rd, 0 op/s 2024-02-19T21:09:28.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:27.461072+0000 osd.6 (osd.6) 145 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:09:28.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:27.463961+0000 osd.6 (osd.6) 146 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:09:28.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:27.703952+0000 mon.b (mon.1) 548 : audit [INF] from='client.? v1:172.21.15.163:0/3934602217' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:09:28.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:27.704283+0000 mon.a (mon.0) 2427 : audit [INF] from='client.48556 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:09:28.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:26.728722+0000 osd.7 (osd.7) 155 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:09:28.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:26.731857+0000 osd.7 (osd.7) 156 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:09:28.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:26.912544+0000 osd.5 (osd.5) 148 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:09:28.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:26.915998+0000 osd.5 (osd.5) 149 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:09:28.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:27.199699+0000 mgr.y (mgr.14550) 348 : cluster [DBG] pgmap v549: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 860 B/s rd, 0 op/s 2024-02-19T21:09:28.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:27.461072+0000 osd.6 (osd.6) 145 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:09:28.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:27.463961+0000 osd.6 (osd.6) 146 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:09:28.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:27.703952+0000 mon.b (mon.1) 548 : audit [INF] from='client.? v1:172.21.15.163:0/3934602217' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:09:28.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:27.704283+0000 mon.a (mon.0) 2427 : audit [INF] from='client.48556 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:09:28.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:26.728722+0000 osd.7 (osd.7) 155 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:09:28.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:26.731857+0000 osd.7 (osd.7) 156 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:09:28.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:26.912544+0000 osd.5 (osd.5) 148 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:09:28.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:26.915998+0000 osd.5 (osd.5) 149 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:09:28.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:27.199699+0000 mgr.y (mgr.14550) 348 : cluster [DBG] pgmap v549: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 860 B/s rd, 0 op/s 2024-02-19T21:09:28.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:27.461072+0000 osd.6 (osd.6) 145 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:09:28.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:27.463961+0000 osd.6 (osd.6) 146 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:09:28.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:27.703952+0000 mon.b (mon.1) 548 : audit [INF] from='client.? v1:172.21.15.163:0/3934602217' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:09:28.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:27.704283+0000 mon.a (mon.0) 2427 : audit [INF] from='client.48556 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:09:28.992 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:28.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:29.096 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:29.128 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:29.149 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:29.151 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:29.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:29.253 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:29.286 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:29.307 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:29.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:29.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:29.411 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:29.436 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:27.746918+0000 osd.3 (osd.3) 159 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:09:29.436 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:27.747781+0000 osd.3 (osd.3) 160 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:09:29.436 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:27.767609+0000 osd.7 (osd.7) 157 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:09:29.436 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:27.770497+0000 osd.7 (osd.7) 158 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:09:29.436 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:27.875856+0000 osd.5 (osd.5) 150 : cluster [DBG] 6.a scrub starts 2024-02-19T21:09:29.437 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:27.878753+0000 osd.5 (osd.5) 151 : cluster [DBG] 6.a scrub ok 2024-02-19T21:09:29.437 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:28.030837+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.b scrub starts 2024-02-19T21:09:29.437 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:28.031868+0000 osd.2 (osd.2) 137 : cluster [DBG] 5.b scrub ok 2024-02-19T21:09:29.437 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:28.263892+0000 mon.a (mon.0) 2428 : audit [INF] from='client.48556 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-19T21:09:29.437 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:28.263961+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2024-02-19T21:09:29.437 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:28.282043+0000 mon.b (mon.1) 549 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:29.437 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:28.282452+0000 mon.a (mon.0) 2430 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:29.437 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:29.247699+0000 mon.a (mon.0) 2431 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:09:29.437 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:29.247782+0000 mon.a (mon.0) 2432 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2024-02-19T21:09:29.437 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:29.248659+0000 mon.b (mon.1) 550 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:29.437 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:29.250525+0000 mon.a (mon.0) 2433 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:29.438 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:29.457 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:29.460 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:29.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:29.561 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:29.588 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:29.607 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:29.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:29.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:29.711 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:29.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:27.746918+0000 osd.3 (osd.3) 159 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:09:29.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:27.747781+0000 osd.3 (osd.3) 160 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:09:29.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:27.767609+0000 osd.7 (osd.7) 157 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:09:29.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:27.770497+0000 osd.7 (osd.7) 158 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:09:29.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:27.875856+0000 osd.5 (osd.5) 150 : cluster [DBG] 6.a scrub starts 2024-02-19T21:09:29.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:27.878753+0000 osd.5 (osd.5) 151 : cluster [DBG] 6.a scrub ok 2024-02-19T21:09:29.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:28.030837+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.b scrub starts 2024-02-19T21:09:29.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:28.031868+0000 osd.2 (osd.2) 137 : cluster [DBG] 5.b scrub ok 2024-02-19T21:09:29.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:28.263892+0000 mon.a (mon.0) 2428 : audit [INF] from='client.48556 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-19T21:09:29.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:28.263961+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2024-02-19T21:09:29.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:28.282043+0000 mon.b (mon.1) 549 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:29.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:28.282452+0000 mon.a (mon.0) 2430 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:29.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:29.247699+0000 mon.a (mon.0) 2431 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:09:29.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:29.247782+0000 mon.a (mon.0) 2432 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2024-02-19T21:09:29.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:29.248659+0000 mon.b (mon.1) 550 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:29.739 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:29.250525+0000 mon.a (mon.0) 2433 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:29.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:27.746918+0000 osd.3 (osd.3) 159 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:09:29.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:27.747781+0000 osd.3 (osd.3) 160 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:09:29.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:27.767609+0000 osd.7 (osd.7) 157 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:09:29.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:27.770497+0000 osd.7 (osd.7) 158 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:09:29.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:27.875856+0000 osd.5 (osd.5) 150 : cluster [DBG] 6.a scrub starts 2024-02-19T21:09:29.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:27.878753+0000 osd.5 (osd.5) 151 : cluster [DBG] 6.a scrub ok 2024-02-19T21:09:29.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:28.030837+0000 osd.2 (osd.2) 136 : cluster [DBG] 5.b scrub starts 2024-02-19T21:09:29.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:28.031868+0000 osd.2 (osd.2) 137 : cluster [DBG] 5.b scrub ok 2024-02-19T21:09:29.739 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:28.263892+0000 mon.a (mon.0) 2428 : audit [INF] from='client.48556 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-19T21:09:29.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:28.263961+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2024-02-19T21:09:29.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:28.282043+0000 mon.b (mon.1) 549 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:29.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:28.282452+0000 mon.a (mon.0) 2430 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:29.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:29.247699+0000 mon.a (mon.0) 2431 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:09:29.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:29.247782+0000 mon.a (mon.0) 2432 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2024-02-19T21:09:29.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:29.248659+0000 mon.b (mon.1) 550 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:29.740 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:29.250525+0000 mon.a (mon.0) 2433 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:29.740 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:30.077 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 7 1 2024-02-19T21:09:30.089 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:30.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:30.193 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:30.219 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:30.342 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:28.655993+0000 osd.1 (osd.1) 169 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:09:30.342 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:28.659490+0000 osd.1 (osd.1) 170 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:09:30.342 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:28.710437+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:09:30.342 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:28.712032+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:09:30.342 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:28.995572+0000 osd.2 (osd.2) 138 : cluster [DBG] 61.14 scrub starts 2024-02-19T21:09:30.342 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:28.997589+0000 osd.2 (osd.2) 139 : cluster [DBG] 61.14 scrub ok 2024-02-19T21:09:30.342 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:29.200169+0000 mgr.y (mgr.14550) 349 : cluster [DBG] pgmap v551: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:09:30.343 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:30.251640+0000 mon.a (mon.0) 2434 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49"}]': finished 2024-02-19T21:09:30.343 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:30.251718+0000 mon.a (mon.0) 2435 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2024-02-19T21:09:30.343 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:30.274312+0000 mon.b (mon.1) 551 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:30.343 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:30.274705+0000 mon.a (mon.0) 2436 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:30.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:28.655993+0000 osd.1 (osd.1) 169 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:09:30.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:28.659490+0000 osd.1 (osd.1) 170 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:09:30.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:28.710437+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:09:30.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:28.712032+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:09:30.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:28.995572+0000 osd.2 (osd.2) 138 : cluster [DBG] 61.14 scrub starts 2024-02-19T21:09:30.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:28.997589+0000 osd.2 (osd.2) 139 : cluster [DBG] 61.14 scrub ok 2024-02-19T21:09:30.343 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:29.200169+0000 mgr.y (mgr.14550) 349 : cluster [DBG] pgmap v551: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:09:30.344 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:30.251640+0000 mon.a (mon.0) 2434 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49"}]': finished 2024-02-19T21:09:30.344 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:30.251718+0000 mon.a (mon.0) 2435 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2024-02-19T21:09:30.344 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:30.274312+0000 mon.b (mon.1) 551 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:30.344 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:30.274705+0000 mon.a (mon.0) 2436 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:30.347 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:30.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:30.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:30.452 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:30.482 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:30.568 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:09:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:09:30.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:28.655993+0000 osd.1 (osd.1) 169 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:09:30.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:28.659490+0000 osd.1 (osd.1) 170 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:09:30.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:28.710437+0000 osd.3 (osd.3) 161 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:09:30.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:28.712032+0000 osd.3 (osd.3) 162 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:09:30.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:28.995572+0000 osd.2 (osd.2) 138 : cluster [DBG] 61.14 scrub starts 2024-02-19T21:09:30.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:28.997589+0000 osd.2 (osd.2) 139 : cluster [DBG] 61.14 scrub ok 2024-02-19T21:09:30.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:29.200169+0000 mgr.y (mgr.14550) 349 : cluster [DBG] pgmap v551: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:09:30.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:30.251640+0000 mon.a (mon.0) 2434 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49"}]': finished 2024-02-19T21:09:30.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:30.251718+0000 mon.a (mon.0) 2435 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2024-02-19T21:09:30.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:30.274312+0000 mon.b (mon.1) 551 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:30.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:30.274705+0000 mon.a (mon.0) 2436 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:30.676 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:09:30.676 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:09:30.679 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:30.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:30.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:30.784 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:30.812 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:31.285 INFO:tasks.workunit.client.0.smithi163.stdout::b4c720e9:::166:head 2024-02-19T21:09:31.286 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:e694b040:::167:head 2024-02-19T21:09:31.286 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:afa38db2:::168:head 2024-02-19T21:09:31.286 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:77ba9f53:::169:head 2024-02-19T21:09:31.286 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:87495034:::170:head 2024-02-19T21:09:31.286 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:7c96bf0e:::171:head 2024-02-19T21:09:31.286 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:dbe346cc:::172:head 2024-02-19T21:09:31.286 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:e943ec24:::173:head 2024-02-19T21:09:31.286 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:f97a9c0c:::174:head 2024-02-19T21:09:31.286 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:6f26e74d:::175:head 2024-02-19T21:09:31.286 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:4f95e106:::176:head 2024-02-19T21:09:31.286 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:0e6f2f8f:::177:head 2024-02-19T21:09:31.286 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:05db05f1:::178:head 2024-02-19T21:09:31.286 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:38a78d66:::179:head 2024-02-19T21:09:31.286 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:d095610b:::180:head 2024-02-19T21:09:31.287 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:a1a9d709:::181:head 2024-02-19T21:09:31.287 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:1e5d39db:::182:head 2024-02-19T21:09:31.287 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:f7df4fb9:::183:head 2024-02-19T21:09:31.287 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:03a7f161:::184:head 2024-02-19T21:09:31.287 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:ba70721e:::185:head 2024-02-19T21:09:31.287 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:28e5662d:::186:head 2024-02-19T21:09:31.287 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:973d52de:::187:head 2024-02-19T21:09:31.287 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:4303eb1c:::188:head 2024-02-19T21:09:31.287 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:b990b48e:::189:head 2024-02-19T21:09:31.287 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:29b8165b:::190:head 2024-02-19T21:09:31.287 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:3547f197:::191:head 2024-02-19T21:09:31.287 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:7e260936:::192:head 2024-02-19T21:09:31.287 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:1abec7b1:::193:head 2024-02-19T21:09:31.287 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:10fdda93:::194:head 2024-02-19T21:09:31.287 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:15817eea:::195:head 2024-02-19T21:09:31.288 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:770bab57:::196:head 2024-02-19T21:09:31.288 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:ed9e13e7:::197:head 2024-02-19T21:09:31.288 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:71471a8f:::198:head 2024-02-19T21:09:31.288 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: checking for 191:10fb1d02:::199:head 2024-02-19T21:09:31.288 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (7980 ms) 2024-02-19T21:09:31.288 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-02-19T21:09:31.288 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708376903,0 2024-02-19T21:09:31.288 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: first is 1708376903 2024-02-19T21:09:31.288 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708376903,0 2024-02-19T21:09:31.288 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708376903,0 2024-02-19T21:09:31.288 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708376903,1708376905,1708376906,0 2024-02-19T21:09:31.288 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708376903,1708376905,1708376906,0 2024-02-19T21:09:31.288 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708376903,1708376905,1708376906,0 2024-02-19T21:09:31.288 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708376903,1708376905,1708376906,1708376908,1708376909,0 2024-02-19T21:09:31.289 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708376903,1708376905,1708376906,1708376908,1708376909,0 2024-02-19T21:09:31.289 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708376903,1708376905,1708376906,1708376908,1708376909,0 2024-02-19T21:09:31.289 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708376903,1708376905,1708376906,1708376908,1708376909,1708376911,1708376912,0 2024-02-19T21:09:31.289 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708376903,1708376905,1708376906,1708376908,1708376909,1708376911,1708376912,0 2024-02-19T21:09:31.289 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708376903,1708376905,1708376906,1708376908,1708376909,1708376911,1708376912,0 2024-02-19T21:09:31.289 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708376906,1708376908,1708376909,1708376911,1708376912,1708376914,1708376915,0 2024-02-19T21:09:31.289 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: first now 1708376906, trimmed 2024-02-19T21:09:31.289 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21009 ms) 2024-02-19T21:09:31.289 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-02-19T21:09:31.289 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: foo0 2024-02-19T21:09:31.289 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-02-19T21:09:31.289 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14078 ms) 2024-02-19T21:09:31.289 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-02-19T21:09:31.289 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (17950 ms) 2024-02-19T21:09:31.290 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-02-19T21:09:31.360 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:31.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:31.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:31.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:31.497 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:31.519 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:31.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:31.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:31.554 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:31.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:29.780121+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:09:31.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:29.783605+0000 osd.7 (osd.7) 160 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:09:31.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:29.849821+0000 osd.5 (osd.5) 152 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:09:31.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:29.852784+0000 osd.5 (osd.5) 153 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:09:31.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:30.003541+0000 osd.6 (osd.6) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:31.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:30.003711+0000 osd.6 (osd.6) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:31.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:30.275373+0000 mon.b (mon.1) 552 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:31.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:30.275594+0000 mon.a (mon.0) 2437 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:31.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:30.350669+0000 mgr.y (mgr.14550) 350 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:31.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:30.386180+0000 osd.4 (osd.4) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:31.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:30.386416+0000 osd.4 (osd.4) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:31.623 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:31.653 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:31.674 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:31.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:31.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:29.780121+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:09:31.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:29.783605+0000 osd.7 (osd.7) 160 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:09:31.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:29.849821+0000 osd.5 (osd.5) 152 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:09:31.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:29.852784+0000 osd.5 (osd.5) 153 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:09:31.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:30.003541+0000 osd.6 (osd.6) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:31.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:30.003711+0000 osd.6 (osd.6) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:31.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:30.275373+0000 mon.b (mon.1) 552 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:31.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:30.275594+0000 mon.a (mon.0) 2437 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:31.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:30.350669+0000 mgr.y (mgr.14550) 350 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:31.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:30.386180+0000 osd.4 (osd.4) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:31.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:30.386416+0000 osd.4 (osd.4) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:31.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:29.780121+0000 osd.7 (osd.7) 159 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:09:31.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:29.783605+0000 osd.7 (osd.7) 160 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:09:31.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:29.849821+0000 osd.5 (osd.5) 152 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:09:31.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:29.852784+0000 osd.5 (osd.5) 153 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:09:31.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:30.003541+0000 osd.6 (osd.6) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:31.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:30.003711+0000 osd.6 (osd.6) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:31.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:30.275373+0000 mon.b (mon.1) 552 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:31.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:30.275594+0000 mon.a (mon.0) 2437 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-49"}]: dispatch 2024-02-19T21:09:31.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:30.350669+0000 mgr.y (mgr.14550) 350 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:31.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:30.386180+0000 osd.4 (osd.4) 145 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:31.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:30.386416+0000 osd.4 (osd.4) 146 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:31.778 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:31.807 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:31.827 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:31.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:31.931 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:31.960 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:32.304 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-19T21:09:32.310 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:32.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:32.414 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:32.441 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:32.460 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:32.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:32.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:32.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.550+0000 7f6004e38700 1 -- 172.21.15.163:0/4080985964 >> v1:172.21.15.163:6790/0 conn(0x7f60001042b0 legacy=0x7f6000106730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:32.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.550+0000 7f6004e38700 1 -- 172.21.15.163:0/4080985964 shutdown_connections 2024-02-19T21:09:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.550+0000 7f6004e38700 1 -- 172.21.15.163:0/4080985964 >> 172.21.15.163:0/4080985964 conn(0x7f60000f3570 msgr2=0x7f60000f5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.550+0000 7f6004e38700 1 -- 172.21.15.163:0/4080985964 shutdown_connections 2024-02-19T21:09:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.550+0000 7f6004e38700 1 -- 172.21.15.163:0/4080985964 wait complete. 2024-02-19T21:09:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.551+0000 7f6004e38700 1 librados: starting msgr at 2024-02-19T21:09:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.551+0000 7f6004e38700 1 librados: starting objecter 2024-02-19T21:09:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.551+0000 7f6004e38700 1 Processor -- start 2024-02-19T21:09:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.551+0000 7f6004e38700 1 -- start start 2024-02-19T21:09:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.551+0000 7f6004e38700 1 librados: setting wanted keys 2024-02-19T21:09:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.551+0000 7f6004e38700 1 librados: calling monclient init 2024-02-19T21:09:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.551+0000 7f6004e38700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f600010fb30 con 0x7f6000100770 2024-02-19T21:09:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.551+0000 7f6004e38700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f600011ec70 con 0x7f60001042b0 2024-02-19T21:09:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.551+0000 7f6004e38700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f600011fe70 con 0x7f60000fd970 2024-02-19T21:09:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.551+0000 7f5ffe59c700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f60000fd970 0x7f6000102640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:59730/0 (socket says 172.21.15.163:59730) 2024-02-19T21:09:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.551+0000 7f5ffe59c700 1 -- 172.21.15.163:0/3642090123 learned_addr learned my addr 172.21.15.163:0/3642090123 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.551+0000 7f5fef7fe700 10 client.?.objecter ms_handle_connect 0x7f60000fd970 2024-02-19T21:09:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.551+0000 7f5fef7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.551+0000 7f5fef7fe700 10 client.?.objecter ms_handle_connect 0x7f6000100770 2024-02-19T21:09:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.551+0000 7f5fef7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 10 client.?.objecter ms_handle_connect 0x7f60001042b0 2024-02-19T21:09:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801910705 0 0) 0x7f600011fe70 con 0x7f60000fd970 2024-02-19T21:09:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe80033f0 con 0x7f60000fd970 2024-02-19T21:09:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142838780 0 0) 0x7f600010a470 con 0x7f6000100770 2024-02-19T21:09:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f600011fe70 con 0x7f6000100770 2024-02-19T21:09:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170537803 0 0) 0x7f600011ec70 con 0x7f60001042b0 2024-02-19T21:09:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f600010a470 con 0x7f60001042b0 2024-02-19T21:09:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2374398064 0 0) 0x7f5fe80033f0 con 0x7f60000fd970 2024-02-19T21:09:32.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f600011ec70 con 0x7f60000fd970 2024-02-19T21:09:32.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5ff40029d0 con 0x7f60000fd970 2024-02-19T21:09:32.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1928719986 0 0) 0x7f600011fe70 con 0x7f6000100770 2024-02-19T21:09:32.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fe80033f0 con 0x7f6000100770 2024-02-19T21:09:32.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3818650485 0 0) 0x7f600010a470 con 0x7f60001042b0 2024-02-19T21:09:32.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f600011fe70 con 0x7f60001042b0 2024-02-19T21:09:32.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5ff00015d0 con 0x7f6000100770 2024-02-19T21:09:32.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5ff8002240 con 0x7f60001042b0 2024-02-19T21:09:32.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2420528372 0 0) 0x7f600011ec70 con 0x7f60000fd970 2024-02-19T21:09:32.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 >> v1:172.21.15.193:6789/0 conn(0x7f60001042b0 legacy=0x7f600011c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:32.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 >> v1:172.21.15.163:6789/0 conn(0x7f6000100770 legacy=0x7f6000102d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:32.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6000121070 con 0x7f60000fd970 2024-02-19T21:09:32.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600011dc80 con 0x7f60000fd970 2024-02-19T21:09:32.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f6004e38700 10 client.49745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:32.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f600011e170 con 0x7f60000fd970 2024-02-19T21:09:32.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f6004e38700 1 librados: init done 2024-02-19T21:09:32.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f5ff4002da0 con 0x7f60000fd970 2024-02-19T21:09:32.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.552+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5ff4005c30 con 0x7f60000fd970 2024-02-19T21:09:32.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.553+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f5fe0004f00 con 0x7f60000fd970 2024-02-19T21:09:32.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f5ff4001e80 con 0x7f60000fd970 2024-02-19T21:09:32.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(389..389 src has 43..389) v4 ==== 7889+0+0 (unknown 554259036 0 0) 0x7f5ff4078d70 con 0x7f60000fd970 2024-02-19T21:09:32.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 osd_map(389..389 src has 43..389) v4 2024-02-19T21:09:32.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f5fef7fe700 3 client.49745.objecter handle_osd_map got epochs [389,389] > 0 2024-02-19T21:09:32.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f5fef7fe700 3 client.49745.objecter handle_osd_map decoding full epoch 389 2024-02-19T21:09:32.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f5fef7fe700 20 client.49745.objecter dump_active .. 0 homeless 2024-02-19T21:09:32.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v389) v1 ==== 54+0+16 (unknown 4113382924 0 2420000192) 0x7f5ff407c020 con 0x7f60000fd970 2024-02-19T21:09:32.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f5fef7fe700 10 client.49745.objecter ms_handle_connect 0x7f5fe8063cb0 2024-02-19T21:09:32.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f6004e38700 10 client.49745.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:32.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 1 2024-02-19T21:09:32.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe00051a0 osd=0 3 2024-02-19T21:09:32.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe00051a0 osd=0 3 2024-02-19T21:09:32.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe00051a0 osd=0 3 2024-02-19T21:09:32.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 1 2024-02-19T21:09:32.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe00051a0 osd=0 3 2024-02-19T21:09:32.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 0 1 2024-02-19T21:09:32.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe00051a0 osd=0 4 2024-02-19T21:09:32.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f6004e38700 10 client.49745.objecter _send_command 1 2024-02-19T21:09:32.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.554+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fe0004950 con 0x7f5fe0004530 2024-02-19T21:09:32.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.555+0000 7f5fef7fe700 10 client.49745.objecter ms_handle_connect 0x7f5fe0004530 2024-02-19T21:09:32.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.555+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5fe0004950 con 0x7f5fe0004530 2024-02-19T21:09:32.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.555+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:09:32.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.555+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 1 = system:0 2024-02-19T21:09:32.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.555+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe00051a0 osd=0 4 2024-02-19T21:09:32.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.555+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 0 1 2024-02-19T21:09:32.562 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:32.563 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:32.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:30.703084+0000 osd.1 (osd.1) 171 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:09:32.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:30.704537+0000 osd.1 (osd.1) 172 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:09:32.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:30.778530+0000 osd.7 (osd.7) 161 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:09:32.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:30.781996+0000 osd.7 (osd.7) 162 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:09:32.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:30.949416+0000 osd.2 (osd.2) 140 : cluster [DBG] 2.c scrub starts 2024-02-19T21:09:32.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:30.950622+0000 osd.2 (osd.2) 141 : cluster [DBG] 2.c scrub ok 2024-02-19T21:09:32.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:31.200593+0000 mgr.y (mgr.14550) 351 : cluster [DBG] pgmap v554: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:09:32.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:31.276872+0000 mon.a (mon.0) 2438 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-19T21:09:32.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:31.285278+0000 mon.a (mon.0) 2439 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2024-02-19T21:09:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.574+0000 7f6004e38700 10 client.49745.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.574+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 2 2024-02-19T21:09:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.574+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe00051a0 osd=0 3 2024-02-19T21:09:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.574+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe00051a0 osd=0 3 2024-02-19T21:09:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.574+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe00051a0 osd=0 3 2024-02-19T21:09:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.574+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 2 2024-02-19T21:09:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.574+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe00051a0 osd=0 3 2024-02-19T21:09:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.574+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 0 2 2024-02-19T21:09:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.574+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe00051a0 osd=0 4 2024-02-19T21:09:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.574+0000 7f6004e38700 10 client.49745.objecter _send_command 2 2024-02-19T21:09:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.574+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5fe0004d30 con 0x7f5fe0004530 2024-02-19T21:09:32.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.576+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5fe0004d30 con 0x7f5fe0004530 2024-02-19T21:09:32.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.576+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:32.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.576+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:32.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.576+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe00051a0 osd=0 4 2024-02-19T21:09:32.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.576+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 0 2 2024-02-19T21:09:32.574 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:09:32.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f6004e38700 10 client.49745.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:32.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 3 2024-02-19T21:09:32.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0003fe0 osd=1 3 2024-02-19T21:09:32.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0003fe0 osd=1 3 2024-02-19T21:09:32.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0003fe0 osd=1 3 2024-02-19T21:09:32.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 3 2024-02-19T21:09:32.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe0003fe0 osd=1 3 2024-02-19T21:09:32.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 1 3 2024-02-19T21:09:32.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0003fe0 osd=1 4 2024-02-19T21:09:32.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f6004e38700 10 client.49745.objecter _send_command 3 2024-02-19T21:09:32.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fe000c000 con 0x7f5fe0004150 2024-02-19T21:09:32.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f5fef7fe700 10 client.49745.objecter ms_handle_connect 0x7f5fe0004150 2024-02-19T21:09:32.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5fe000c000 con 0x7f5fe0004150 2024-02-19T21:09:32.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:09:32.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 3 = system:0 2024-02-19T21:09:32.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe0003fe0 osd=1 4 2024-02-19T21:09:32.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.577+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 1 3 2024-02-19T21:09:32.583 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:09:32.590 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:30.703084+0000 osd.1 (osd.1) 171 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:09:32.590 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:30.704537+0000 osd.1 (osd.1) 172 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:09:32.590 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:30.778530+0000 osd.7 (osd.7) 161 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:09:32.590 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:30.781996+0000 osd.7 (osd.7) 162 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:09:32.590 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:30.949416+0000 osd.2 (osd.2) 140 : cluster [DBG] 2.c scrub starts 2024-02-19T21:09:32.590 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:30.950622+0000 osd.2 (osd.2) 141 : cluster [DBG] 2.c scrub ok 2024-02-19T21:09:32.590 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:31.200593+0000 mgr.y (mgr.14550) 351 : cluster [DBG] pgmap v554: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:09:32.591 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:31.276872+0000 mon.a (mon.0) 2438 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-19T21:09:32.591 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:31.285278+0000 mon.a (mon.0) 2439 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2024-02-19T21:09:32.591 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:30.703084+0000 osd.1 (osd.1) 171 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:09:32.591 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:30.704537+0000 osd.1 (osd.1) 172 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:09:32.591 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:30.778530+0000 osd.7 (osd.7) 161 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:09:32.591 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:30.781996+0000 osd.7 (osd.7) 162 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:09:32.591 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:30.949416+0000 osd.2 (osd.2) 140 : cluster [DBG] 2.c scrub starts 2024-02-19T21:09:32.591 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:30.950622+0000 osd.2 (osd.2) 141 : cluster [DBG] 2.c scrub ok 2024-02-19T21:09:32.591 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:31.200593+0000 mgr.y (mgr.14550) 351 : cluster [DBG] pgmap v554: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:09:32.591 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:31.276872+0000 mon.a (mon.0) 2438 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-19T21:09:32.592 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:31.285278+0000 mon.a (mon.0) 2439 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2024-02-19T21:09:32.592 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:32.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.600+0000 7f6004e38700 10 client.49745.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:32.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.600+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 4 2024-02-19T21:09:32.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.600+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0003fe0 osd=1 3 2024-02-19T21:09:32.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.600+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0003fe0 osd=1 3 2024-02-19T21:09:32.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.600+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0003fe0 osd=1 3 2024-02-19T21:09:32.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.600+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 4 2024-02-19T21:09:32.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.600+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe0003fe0 osd=1 3 2024-02-19T21:09:32.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.600+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 1 4 2024-02-19T21:09:32.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.600+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0003fe0 osd=1 4 2024-02-19T21:09:32.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.600+0000 7f6004e38700 10 client.49745.objecter _send_command 4 2024-02-19T21:09:32.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.600+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5fe0001850 con 0x7f5fe0004150 2024-02-19T21:09:32.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5fe0001850 con 0x7f5fe0004150 2024-02-19T21:09:32.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:32.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:32.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe0003fe0 osd=1 4 2024-02-19T21:09:32.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 1 4 2024-02-19T21:09:32.600 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:09:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f6004e38700 10 client.49745.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 5 2024-02-19T21:09:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0003630 osd=2 3 2024-02-19T21:09:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0003630 osd=2 3 2024-02-19T21:09:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0003630 osd=2 3 2024-02-19T21:09:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 5 2024-02-19T21:09:32.601 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe0003630 osd=2 3 2024-02-19T21:09:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 2 5 2024-02-19T21:09:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0003630 osd=2 4 2024-02-19T21:09:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f6004e38700 10 client.49745.objecter _send_command 5 2024-02-19T21:09:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.602+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fe0003e30 con 0x7f5fe000d320 2024-02-19T21:09:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.603+0000 7f5fef7fe700 10 client.49745.objecter ms_handle_connect 0x7f5fe000d320 2024-02-19T21:09:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.603+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5fe0003e30 con 0x7f5fe000d320 2024-02-19T21:09:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.603+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:09:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.603+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 5 = system:0 2024-02-19T21:09:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.603+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe0003630 osd=2 4 2024-02-19T21:09:32.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.603+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 2 5 2024-02-19T21:09:32.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.619+0000 7f6004e38700 10 client.49745.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:32.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.619+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 6 2024-02-19T21:09:32.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.619+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0003630 osd=2 3 2024-02-19T21:09:32.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.619+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0003630 osd=2 3 2024-02-19T21:09:32.617 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.619+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0003630 osd=2 3 2024-02-19T21:09:32.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.619+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 6 2024-02-19T21:09:32.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.619+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe0003630 osd=2 3 2024-02-19T21:09:32.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.619+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 2 6 2024-02-19T21:09:32.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.619+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0003630 osd=2 4 2024-02-19T21:09:32.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.619+0000 7f6004e38700 10 client.49745.objecter _send_command 6 2024-02-19T21:09:32.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.619+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5fe000e710 con 0x7f5fe000d320 2024-02-19T21:09:32.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.621+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5fe000e710 con 0x7f5fe000d320 2024-02-19T21:09:32.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.621+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:32.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.621+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:32.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.621+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe0003630 osd=2 4 2024-02-19T21:09:32.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.621+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 2 6 2024-02-19T21:09:32.619 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:09:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f6004e38700 10 client.49745.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 7 2024-02-19T21:09:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0003390 osd=3 3 2024-02-19T21:09:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0003390 osd=3 3 2024-02-19T21:09:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0003390 osd=3 3 2024-02-19T21:09:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 7 2024-02-19T21:09:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe0003390 osd=3 3 2024-02-19T21:09:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 3 7 2024-02-19T21:09:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0003390 osd=3 4 2024-02-19T21:09:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f6004e38700 10 client.49745.objecter _send_command 7 2024-02-19T21:09:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fe0015a90 con 0x7f5fe0011eb0 2024-02-19T21:09:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f5fef7fe700 10 client.49745.objecter ms_handle_connect 0x7f5fe0011eb0 2024-02-19T21:09:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5fe0015a90 con 0x7f5fe0011eb0 2024-02-19T21:09:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:09:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 7 = system:0 2024-02-19T21:09:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe0003390 osd=3 4 2024-02-19T21:09:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.622+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 3 7 2024-02-19T21:09:32.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.637+0000 7f6004e38700 10 client.49745.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:32.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.637+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 8 2024-02-19T21:09:32.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.637+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0003390 osd=3 3 2024-02-19T21:09:32.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.637+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0003390 osd=3 3 2024-02-19T21:09:32.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.637+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0003390 osd=3 3 2024-02-19T21:09:32.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.637+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 8 2024-02-19T21:09:32.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.637+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe0003390 osd=3 3 2024-02-19T21:09:32.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.637+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 3 8 2024-02-19T21:09:32.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.637+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0003390 osd=3 4 2024-02-19T21:09:32.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.637+0000 7f6004e38700 10 client.49745.objecter _send_command 8 2024-02-19T21:09:32.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.637+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5fe0015a90 con 0x7f5fe0011eb0 2024-02-19T21:09:32.636 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:09:32.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5fe0015a90 con 0x7f5fe0011eb0 2024-02-19T21:09:32.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:32.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:32.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe0003390 osd=3 4 2024-02-19T21:09:32.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 3 8 2024-02-19T21:09:32.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f6004e38700 10 client.49745.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:32.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 9 2024-02-19T21:09:32.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0015a90 osd=4 3 2024-02-19T21:09:32.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0015a90 osd=4 3 2024-02-19T21:09:32.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0015a90 osd=4 3 2024-02-19T21:09:32.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 9 2024-02-19T21:09:32.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe0015a90 osd=4 3 2024-02-19T21:09:32.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 4 9 2024-02-19T21:09:32.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0015a90 osd=4 4 2024-02-19T21:09:32.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f6004e38700 10 client.49745.objecter _send_command 9 2024-02-19T21:09:32.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fe000cf00 con 0x7f5fe000c970 2024-02-19T21:09:32.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.639+0000 7f5fef7fe700 10 client.49745.objecter ms_handle_connect 0x7f5fe000c970 2024-02-19T21:09:32.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.640+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5fe000cf00 con 0x7f5fe000c970 2024-02-19T21:09:32.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.640+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:09:32.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.640+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 9 = system:0 2024-02-19T21:09:32.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.640+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe0015a90 osd=4 4 2024-02-19T21:09:32.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.640+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 4 9 2024-02-19T21:09:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.655+0000 7f6004e38700 10 client.49745.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.655+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 10 2024-02-19T21:09:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.655+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0015a90 osd=4 3 2024-02-19T21:09:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.655+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0015a90 osd=4 3 2024-02-19T21:09:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.655+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0015a90 osd=4 3 2024-02-19T21:09:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.655+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 10 2024-02-19T21:09:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.655+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe0015a90 osd=4 3 2024-02-19T21:09:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.655+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 4 10 2024-02-19T21:09:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.655+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0015a90 osd=4 4 2024-02-19T21:09:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.655+0000 7f6004e38700 10 client.49745.objecter _send_command 10 2024-02-19T21:09:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.655+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5fe000cf00 con 0x7f5fe000c970 2024-02-19T21:09:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5fe000cf00 con 0x7f5fe000c970 2024-02-19T21:09:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe0015a90 osd=4 4 2024-02-19T21:09:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 4 10 2024-02-19T21:09:32.655 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:09:32.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f6004e38700 10 client.49745.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:32.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 11 2024-02-19T21:09:32.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe000cf00 osd=5 3 2024-02-19T21:09:32.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe000cf00 osd=5 3 2024-02-19T21:09:32.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe000cf00 osd=5 3 2024-02-19T21:09:32.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 11 2024-02-19T21:09:32.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe000cf00 osd=5 3 2024-02-19T21:09:32.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 5 11 2024-02-19T21:09:32.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe000cf00 osd=5 4 2024-02-19T21:09:32.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f6004e38700 10 client.49745.objecter _send_command 11 2024-02-19T21:09:32.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fe0017e30 con 0x7f5fe00179d0 2024-02-19T21:09:32.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.657+0000 7f5fef7fe700 10 client.49745.objecter ms_handle_connect 0x7f5fe00179d0 2024-02-19T21:09:32.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.658+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5fe0017e30 con 0x7f5fe00179d0 2024-02-19T21:09:32.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.658+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:09:32.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.658+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 11 = system:0 2024-02-19T21:09:32.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.658+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe000cf00 osd=5 4 2024-02-19T21:09:32.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.658+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 5 11 2024-02-19T21:09:32.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.673+0000 7f6004e38700 10 client.49745.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.673+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 12 2024-02-19T21:09:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.673+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe000cf00 osd=5 3 2024-02-19T21:09:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.673+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe000cf00 osd=5 3 2024-02-19T21:09:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.673+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe000cf00 osd=5 3 2024-02-19T21:09:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.673+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 12 2024-02-19T21:09:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.673+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe000cf00 osd=5 3 2024-02-19T21:09:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.673+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 5 12 2024-02-19T21:09:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.673+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe000cf00 osd=5 4 2024-02-19T21:09:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.673+0000 7f6004e38700 10 client.49745.objecter _send_command 12 2024-02-19T21:09:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.673+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5fe0017e30 con 0x7f5fe00179d0 2024-02-19T21:09:32.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5fe0017e30 con 0x7f5fe00179d0 2024-02-19T21:09:32.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:32.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:32.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe000cf00 osd=5 4 2024-02-19T21:09:32.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 5 12 2024-02-19T21:09:32.673 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:09:32.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f6004e38700 10 client.49745.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:32.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 13 2024-02-19T21:09:32.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0017e30 osd=6 3 2024-02-19T21:09:32.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0017e30 osd=6 3 2024-02-19T21:09:32.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0017e30 osd=6 3 2024-02-19T21:09:32.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 13 2024-02-19T21:09:32.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe0017e30 osd=6 3 2024-02-19T21:09:32.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 6 13 2024-02-19T21:09:32.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0017e30 osd=6 4 2024-02-19T21:09:32.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f6004e38700 10 client.49745.objecter _send_command 13 2024-02-19T21:09:32.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fe0018200 con 0x7f5fe001f090 2024-02-19T21:09:32.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.675+0000 7f5fef7fe700 10 client.49745.objecter ms_handle_connect 0x7f5fe001f090 2024-02-19T21:09:32.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.676+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5fe0018200 con 0x7f5fe001f090 2024-02-19T21:09:32.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.676+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:09:32.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.676+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 13 = system:0 2024-02-19T21:09:32.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.676+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe0017e30 osd=6 4 2024-02-19T21:09:32.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.676+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 6 13 2024-02-19T21:09:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.690+0000 7f6004e38700 10 client.49745.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.690+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 14 2024-02-19T21:09:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.690+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0017e30 osd=6 3 2024-02-19T21:09:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.690+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0017e30 osd=6 3 2024-02-19T21:09:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.690+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0017e30 osd=6 3 2024-02-19T21:09:32.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.690+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 14 2024-02-19T21:09:32.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.690+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe0017e30 osd=6 3 2024-02-19T21:09:32.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.690+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 6 14 2024-02-19T21:09:32.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.690+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0017e30 osd=6 4 2024-02-19T21:09:32.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.690+0000 7f6004e38700 10 client.49745.objecter _send_command 14 2024-02-19T21:09:32.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.690+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5fe0018200 con 0x7f5fe001f090 2024-02-19T21:09:32.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5fe0018200 con 0x7f5fe001f090 2024-02-19T21:09:32.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:32.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:32.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe0017e30 osd=6 4 2024-02-19T21:09:32.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 6 14 2024-02-19T21:09:32.690 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:09:32.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f6004e38700 10 client.49745.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:32.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 15 2024-02-19T21:09:32.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0018200 osd=7 3 2024-02-19T21:09:32.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0018200 osd=7 3 2024-02-19T21:09:32.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0018200 osd=7 3 2024-02-19T21:09:32.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 15 2024-02-19T21:09:32.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe0018200 osd=7 3 2024-02-19T21:09:32.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 7 15 2024-02-19T21:09:32.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0018200 osd=7 4 2024-02-19T21:09:32.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f6004e38700 10 client.49745.objecter _send_command 15 2024-02-19T21:09:32.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.692+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5fe0023c20 con 0x7f5fe00246c0 2024-02-19T21:09:32.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.693+0000 7f5fef7fe700 10 client.49745.objecter ms_handle_connect 0x7f5fe00246c0 2024-02-19T21:09:32.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.693+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5fe0023c20 con 0x7f5fe00246c0 2024-02-19T21:09:32.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.693+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:09:32.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.693+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 15 = system:0 2024-02-19T21:09:32.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.693+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe0018200 osd=7 4 2024-02-19T21:09:32.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.693+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 7 15 2024-02-19T21:09:32.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.710+0000 7f6004e38700 10 client.49745.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:32.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.710+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign -1 16 2024-02-19T21:09:32.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.710+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0018200 osd=7 3 2024-02-19T21:09:32.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.710+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0018200 osd=7 3 2024-02-19T21:09:32.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.710+0000 7f6004e38700 20 client.49745.objecter _get_session s=0x7f5fe0018200 osd=7 3 2024-02-19T21:09:32.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.710+0000 7f6004e38700 15 client.49745.objecter _session_command_op_remove -1 16 2024-02-19T21:09:32.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.710+0000 7f6004e38700 20 client.49745.objecter get_session s=0x7f5fe0018200 osd=7 3 2024-02-19T21:09:32.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.710+0000 7f6004e38700 15 client.49745.objecter _session_command_op_assign 7 16 2024-02-19T21:09:32.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.710+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0018200 osd=7 4 2024-02-19T21:09:32.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.710+0000 7f6004e38700 10 client.49745.objecter _send_command 16 2024-02-19T21:09:32.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.710+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5fe0023c20 con 0x7f5fe00246c0 2024-02-19T21:09:32.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f5fef7fe700 1 -- 172.21.15.163:0/3642090123 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5fe0023c20 con 0x7f5fe00246c0 2024-02-19T21:09:32.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f5fef7fe700 10 client.49745.objecter ms_dispatch 0x7f6000001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:32.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f5fef7fe700 10 client.49745.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:32.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f5fef7fe700 20 client.49745.objecter put_session s=0x7f5fe0018200 osd=7 4 2024-02-19T21:09:32.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f5fef7fe700 15 client.49745.objecter _session_command_op_remove 7 16 2024-02-19T21:09:32.710 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:09:32.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 10 librados: watch_flush enter 2024-02-19T21:09:32.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 10 librados: watch_flush exit 2024-02-19T21:09:32.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 10 client.49745.objecter close_session for osd.0 2024-02-19T21:09:32.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 >> v1:172.21.15.163:6801/1559418115 conn(0x7f5fe0004530 legacy=0x7f5fe0002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:32.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe00051a0 osd=0 1 2024-02-19T21:09:32.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 10 client.49745.objecter close_session for osd.1 2024-02-19T21:09:32.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 >> v1:172.21.15.163:6805/3125047432 conn(0x7f5fe0004150 legacy=0x7f5fe000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:32.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0003fe0 osd=1 1 2024-02-19T21:09:32.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 10 client.49745.objecter close_session for osd.2 2024-02-19T21:09:32.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 >> v1:172.21.15.163:6809/1563158487 conn(0x7f5fe000d320 legacy=0x7f5fe000d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:32.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0003630 osd=2 1 2024-02-19T21:09:32.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 10 client.49745.objecter close_session for osd.3 2024-02-19T21:09:32.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 >> v1:172.21.15.163:6813/2112279769 conn(0x7f5fe0011eb0 legacy=0x7f5fe00142a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:32.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0003390 osd=3 1 2024-02-19T21:09:32.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 10 client.49745.objecter close_session for osd.4 2024-02-19T21:09:32.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 >> v1:172.21.15.193:6800/3129716965 conn(0x7f5fe000c970 legacy=0x7f5fe0015c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:32.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f5ffed9d700 1 -- 172.21.15.163:0/3642090123 reap_dead start 2024-02-19T21:09:32.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0015a90 osd=4 1 2024-02-19T21:09:32.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f5ffed9d700 1 -- 172.21.15.163:0/3642090123 reap_dead start 2024-02-19T21:09:32.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 10 client.49745.objecter close_session for osd.5 2024-02-19T21:09:32.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 >> v1:172.21.15.193:6804/805161991 conn(0x7f5fe00179d0 legacy=0x7f5fe001d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:32.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe000cf00 osd=5 1 2024-02-19T21:09:32.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 10 client.49745.objecter close_session for osd.6 2024-02-19T21:09:32.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 >> v1:172.21.15.193:6808/787604615 conn(0x7f5fe001f090 legacy=0x7f5fe0021480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:32.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0017e30 osd=6 1 2024-02-19T21:09:32.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 10 client.49745.objecter close_session for osd.7 2024-02-19T21:09:32.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 >> v1:172.21.15.193:6812/157927432 conn(0x7f5fe00246c0 legacy=0x7f5fe0024b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:32.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f5ffed9d700 1 -- 172.21.15.163:0/3642090123 reap_dead start 2024-02-19T21:09:32.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 20 client.49745.objecter put_session s=0x7f5fe0018200 osd=7 1 2024-02-19T21:09:32.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 20 client.49745.objecter shutdown clearing up homeless session... 2024-02-19T21:09:32.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.712+0000 7f6004e38700 10 client.49745.objecter successfully canceled tick 2024-02-19T21:09:32.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.713+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5fe8063cb0 legacy=0x7f5fe8066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:32.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.713+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 >> v1:172.21.15.163:6790/0 conn(0x7f60000fd970 legacy=0x7f6000102640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:32.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.713+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 shutdown_connections 2024-02-19T21:09:32.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.713+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 >> 172.21.15.163:0/3642090123 conn(0x7f60000f3570 msgr2=0x7f6000107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:32.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.713+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 shutdown_connections 2024-02-19T21:09:32.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.713+0000 7f6004e38700 1 -- 172.21.15.163:0/3642090123 wait complete. 2024-02-19T21:09:32.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:32.713+0000 7f6004e38700 1 librados: shutdown 2024-02-19T21:09:32.714 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:09:32.714 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:09:32.714 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:09:32.714 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:09:32.714 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:09:32.714 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:09:32.715 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:09:32.715 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:09:33.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:31.853895+0000 osd.5 (osd.5) 154 : cluster [DBG] 6.d scrub starts 2024-02-19T21:09:33.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:31.856780+0000 osd.5 (osd.5) 155 : cluster [DBG] 6.d scrub ok 2024-02-19T21:09:33.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:32.291395+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2024-02-19T21:09:33.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:32.304318+0000 mon.b (mon.1) 553 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:33.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:32.305515+0000 mon.a (mon.0) 2441 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:33.319 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:32.555412+0000 mon.c (mon.2) 382 : audit [DBG] from='client.? v1:172.21.15.163:0/3642090123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:33.467 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:31.853895+0000 osd.5 (osd.5) 154 : cluster [DBG] 6.d scrub starts 2024-02-19T21:09:33.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:31.856780+0000 osd.5 (osd.5) 155 : cluster [DBG] 6.d scrub ok 2024-02-19T21:09:33.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:32.291395+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2024-02-19T21:09:33.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:32.304318+0000 mon.b (mon.1) 553 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:33.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:32.305515+0000 mon.a (mon.0) 2441 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:33.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:32.555412+0000 mon.c (mon.2) 382 : audit [DBG] from='client.? v1:172.21.15.163:0/3642090123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:33.468 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:31.853895+0000 osd.5 (osd.5) 154 : cluster [DBG] 6.d scrub starts 2024-02-19T21:09:33.468 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:31.856780+0000 osd.5 (osd.5) 155 : cluster [DBG] 6.d scrub ok 2024-02-19T21:09:33.468 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:32.291395+0000 mon.a (mon.0) 2440 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2024-02-19T21:09:33.468 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:32.304318+0000 mon.b (mon.1) 553 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:33.468 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:32.305515+0000 mon.a (mon.0) 2441 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:33.468 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:32.555412+0000 mon.c (mon.2) 382 : audit [DBG] from='client.? v1:172.21.15.163:0/3642090123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:33.745 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:09:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:09:33] "GET /metrics HTTP/1.1" 200 189347 "" "Prometheus/2.33.4" 2024-02-19T21:09:33.942 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:33.945 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:33.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:34.046 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:34.076 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:34.208 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:34.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:34.238 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:09:34.239 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:09:34.239 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-02-19T21:09:34.239 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:09:34.311 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:34.340 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:34.414 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:34.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:34.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:34.517 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:34.548 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:32.622100+0000 osd.1 (osd.1) 173 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:09:34.548 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:32.622933+0000 osd.1 (osd.1) 174 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:09:34.548 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:33.201270+0000 mgr.y (mgr.14550) 352 : cluster [DBG] pgmap v557: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-02-19T21:09:34.548 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:33.298176+0000 mon.a (mon.0) 2442 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:34.548 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:33.298316+0000 mon.a (mon.0) 2443 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2024-02-19T21:09:34.548 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:33.332029+0000 mon.b (mon.1) 554 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:34.548 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:33.332312+0000 mon.a (mon.0) 2444 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:34.548 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:33.332777+0000 mon.b (mon.1) 555 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-51"}]: dispatch 2024-02-19T21:09:34.549 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:33.332965+0000 mon.a (mon.0) 2445 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-51"}]: dispatch 2024-02-19T21:09:34.549 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:33.558333+0000 osd.3 (osd.3) 163 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:09:34.549 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:33.559359+0000 osd.3 (osd.3) 164 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:09:34.550 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:34.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:32.622100+0000 osd.1 (osd.1) 173 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:09:34.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:32.622933+0000 osd.1 (osd.1) 174 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:09:34.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:33.201270+0000 mgr.y (mgr.14550) 352 : cluster [DBG] pgmap v557: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-02-19T21:09:34.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:33.298176+0000 mon.a (mon.0) 2442 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:34.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:33.298316+0000 mon.a (mon.0) 2443 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2024-02-19T21:09:34.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:33.332029+0000 mon.b (mon.1) 554 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:34.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:33.332312+0000 mon.a (mon.0) 2444 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:34.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:33.332777+0000 mon.b (mon.1) 555 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-51"}]: dispatch 2024-02-19T21:09:34.581 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:33.332965+0000 mon.a (mon.0) 2445 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-51"}]: dispatch 2024-02-19T21:09:34.581 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:33.558333+0000 osd.3 (osd.3) 163 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:09:34.581 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:33.559359+0000 osd.3 (osd.3) 164 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:09:34.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:32.622100+0000 osd.1 (osd.1) 173 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:09:34.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:32.622933+0000 osd.1 (osd.1) 174 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:09:34.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:33.201270+0000 mgr.y (mgr.14550) 352 : cluster [DBG] pgmap v557: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-02-19T21:09:34.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:33.298176+0000 mon.a (mon.0) 2442 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:34.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:33.298316+0000 mon.a (mon.0) 2443 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2024-02-19T21:09:34.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:33.332029+0000 mon.b (mon.1) 554 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:34.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:33.332312+0000 mon.a (mon.0) 2444 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:34.581 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:33.332777+0000 mon.b (mon.1) 555 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-51"}]: dispatch 2024-02-19T21:09:34.582 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:33.332965+0000 mon.a (mon.0) 2445 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-51"}]: dispatch 2024-02-19T21:09:34.582 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:33.558333+0000 osd.3 (osd.3) 163 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:09:34.582 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:33.559359+0000 osd.3 (osd.3) 164 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:09:34.652 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:34.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:34.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:34.766 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:34.795 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:34.990 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-19T21:09:35.253 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:09:35.306 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:34.302332+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2024-02-19T21:09:35.306 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:34.568732+0000 osd.6 (osd.6) 149 : cluster [DBG] 4.a scrub starts 2024-02-19T21:09:35.307 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:34.573881+0000 osd.6 (osd.6) 150 : cluster [DBG] 4.a scrub ok 2024-02-19T21:09:35.307 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:34.654705+0000 osd.0 (osd.0) 123 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:35.307 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:34.654962+0000 osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:35.444 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:35.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:35.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:35.549 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:35.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:34.302332+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2024-02-19T21:09:35.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:34.568732+0000 osd.6 (osd.6) 149 : cluster [DBG] 4.a scrub starts 2024-02-19T21:09:35.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:34.573881+0000 osd.6 (osd.6) 150 : cluster [DBG] 4.a scrub ok 2024-02-19T21:09:35.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:34.654705+0000 osd.0 (osd.0) 123 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:35.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:34.654962+0000 osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:35.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:34.302332+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2024-02-19T21:09:35.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:34.568732+0000 osd.6 (osd.6) 149 : cluster [DBG] 4.a scrub starts 2024-02-19T21:09:35.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:34.573881+0000 osd.6 (osd.6) 150 : cluster [DBG] 4.a scrub ok 2024-02-19T21:09:35.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:34.654705+0000 osd.0 (osd.0) 123 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:35.576 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:34.654962+0000 osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:35.578 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:35.676 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:35.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:35.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:35.780 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:35.809 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:35.933 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:35.937 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:35.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:36.038 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:09:36.057 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:09:36.069 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:36.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.150+0000 7f7e737bd700 1 -- 172.21.15.163:0/4082764382 >> v1:172.21.15.163:6790/0 conn(0x7f7e6c106530 legacy=0x7f7e6c108960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:36.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.150+0000 7f7e737bd700 1 -- 172.21.15.163:0/4082764382 shutdown_connections 2024-02-19T21:09:36.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.150+0000 7f7e737bd700 1 -- 172.21.15.163:0/4082764382 >> 172.21.15.163:0/4082764382 conn(0x7f7e6c0f35a0 msgr2=0x7f7e6c0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:36.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.150+0000 7f7e737bd700 1 -- 172.21.15.163:0/4082764382 shutdown_connections 2024-02-19T21:09:36.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.150+0000 7f7e737bd700 1 -- 172.21.15.163:0/4082764382 wait complete. 2024-02-19T21:09:36.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.150+0000 7f7e737bd700 1 librados: starting msgr at 2024-02-19T21:09:36.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.150+0000 7f7e737bd700 1 librados: starting objecter 2024-02-19T21:09:36.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e737bd700 1 Processor -- start 2024-02-19T21:09:36.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e737bd700 1 -- start start 2024-02-19T21:09:36.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e737bd700 1 librados: setting wanted keys 2024-02-19T21:09:36.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e737bd700 1 librados: calling monclient init 2024-02-19T21:09:36.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e737bd700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7e6c119540 con 0x7f7e6c06b170 2024-02-19T21:09:36.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e737bd700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7e6c11a720 con 0x7f7e6c106530 2024-02-19T21:09:36.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e737bd700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7e6c11b920 con 0x7f7e6c06bc60 2024-02-19T21:09:36.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e70d58700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f7e6c06bc60 0x7f7e6c101f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:33438/0 (socket says 172.21.15.163:33438) 2024-02-19T21:09:36.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e70d58700 1 -- 172.21.15.163:0/4237921293 learned_addr learned my addr 172.21.15.163:0/4237921293 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:36.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e71559700 1 --1- 172.21.15.163:0/4237921293 >> v1:172.21.15.163:6789/0 conn(0x7f7e6c06b170 0x7f7e6c1017c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:55332/0 (socket says 172.21.15.163:55332) 2024-02-19T21:09:36.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e627fc700 10 client.?.objecter ms_handle_connect 0x7f7e6c06bc60 2024-02-19T21:09:36.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e627fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:36.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e627fc700 10 client.?.objecter ms_handle_connect 0x7f7e6c06b170 2024-02-19T21:09:36.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e627fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:36.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950642526 0 0) 0x7f7e6c11b920 con 0x7f7e6c06bc60 2024-02-19T21:09:36.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e58003800 con 0x7f7e6c06bc60 2024-02-19T21:09:36.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821071593 0 0) 0x7f7e6c119540 con 0x7f7e6c06b170 2024-02-19T21:09:36.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e6c11b920 con 0x7f7e6c06b170 2024-02-19T21:09:36.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e627fc700 10 client.?.objecter ms_handle_connect 0x7f7e6c106530 2024-02-19T21:09:36.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e627fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:36.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.151+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 507168450 0 0) 0x7f7e58003800 con 0x7f7e6c06bc60 2024-02-19T21:09:36.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e6c119540 con 0x7f7e6c06bc60 2024-02-19T21:09:36.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7e5c001980 con 0x7f7e6c06bc60 2024-02-19T21:09:36.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2909234808 0 0) 0x7f7e6c11b920 con 0x7f7e6c06b170 2024-02-19T21:09:36.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e58003800 con 0x7f7e6c06b170 2024-02-19T21:09:36.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7e680018d0 con 0x7f7e6c06b170 2024-02-19T21:09:36.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712114643 0 0) 0x7f7e6c11a720 con 0x7f7e6c106530 2024-02-19T21:09:36.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e6c11b920 con 0x7f7e6c106530 2024-02-19T21:09:36.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 133925993 0 0) 0x7f7e6c119540 con 0x7f7e6c06bc60 2024-02-19T21:09:36.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 >> v1:172.21.15.193:6789/0 conn(0x7f7e6c106530 legacy=0x7f7e6c117c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:36.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 >> v1:172.21.15.163:6789/0 conn(0x7f7e6c06b170 legacy=0x7f7e6c1017c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:36.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6c11cb20 con 0x7f7e6c06bc60 2024-02-19T21:09:36.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e737bd700 1 -- 172.21.15.163:0/4237921293 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e6c119730 con 0x7f7e6c06bc60 2024-02-19T21:09:36.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e737bd700 10 client.49751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:36.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e737bd700 1 -- 172.21.15.163:0/4237921293 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e6c119ba0 con 0x7f7e6c06bc60 2024-02-19T21:09:36.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e737bd700 1 librados: init done 2024-02-19T21:09:36.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f7e5c002100 con 0x7f7e6c06bc60 2024-02-19T21:09:36.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.152+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7e5c005ae0 con 0x7f7e6c06bc60 2024-02-19T21:09:36.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.153+0000 7f7e737bd700 1 -- 172.21.15.163:0/4237921293 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e50004f20 con 0x7f7e6c06bc60 2024-02-19T21:09:36.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.153+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f7e5c018990 con 0x7f7e6c06bc60 2024-02-19T21:09:36.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.154+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(392..392 src has 43..392) v4 ==== 7889+0+0 (unknown 1224511043 0 0) 0x7f7e5c078bf0 con 0x7f7e6c06bc60 2024-02-19T21:09:36.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.154+0000 7f7e627fc700 10 client.49751.objecter ms_dispatch 0x7f7e6c001550 osd_map(392..392 src has 43..392) v4 2024-02-19T21:09:36.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.154+0000 7f7e627fc700 3 client.49751.objecter handle_osd_map got epochs [392,392] > 0 2024-02-19T21:09:36.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.154+0000 7f7e627fc700 3 client.49751.objecter handle_osd_map decoding full epoch 392 2024-02-19T21:09:36.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.154+0000 7f7e627fc700 20 client.49751.objecter dump_active .. 0 homeless 2024-02-19T21:09:36.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.154+0000 7f7e627fc700 10 client.49751.objecter ms_handle_connect 0x7f7e58063df0 2024-02-19T21:09:36.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.158+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f7e5c04c0f0 con 0x7f7e6c06bc60 2024-02-19T21:09:36.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.371+0000 7f7e737bd700 1 -- 172.21.15.163:0/4237921293 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f7e500051d0 con 0x7f7e6c06bc60 2024-02-19T21:09:36.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.386+0000 7f5ff2af4700 1 -- 172.21.15.163:0/617329428 >> v1:172.21.15.193:6789/0 conn(0x7f5fec100780 legacy=0x7f5fec102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.386+0000 7f5ff2af4700 1 -- 172.21.15.163:0/617329428 shutdown_connections 2024-02-19T21:09:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.386+0000 7f5ff2af4700 1 -- 172.21.15.163:0/617329428 >> 172.21.15.163:0/617329428 conn(0x7f5fec0f35a0 msgr2=0x7f5fec0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.386+0000 7f5ff2af4700 1 -- 172.21.15.163:0/617329428 shutdown_connections 2024-02-19T21:09:36.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.386+0000 7f5ff2af4700 1 -- 172.21.15.163:0/617329428 wait complete. 2024-02-19T21:09:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.386+0000 7f5ff2af4700 1 librados: starting msgr at 2024-02-19T21:09:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.386+0000 7f5ff2af4700 1 librados: starting objecter 2024-02-19T21:09:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5ff2af4700 1 Processor -- start 2024-02-19T21:09:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5ff2af4700 1 -- start start 2024-02-19T21:09:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5ff2af4700 1 librados: setting wanted keys 2024-02-19T21:09:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5ff2af4700 1 librados: calling monclient init 2024-02-19T21:09:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5ff2af4700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5fec1171c0 con 0x7f5fec0fd980 2024-02-19T21:09:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5ff2af4700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5fec1183a0 con 0x7f5fec100780 2024-02-19T21:09:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5ff2af4700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5fec119580 con 0x7f5fec1042c0 2024-02-19T21:09:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5ff0890700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f5fec0fd980 0x7f5fec0f9770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:55348/0 (socket says 172.21.15.163:55348) 2024-02-19T21:09:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5ff0890700 1 -- 172.21.15.163:0/6057074 learned_addr learned my addr 172.21.15.163:0/6057074 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5fe9ffb700 10 client.?.objecter ms_handle_connect 0x7f5fec1042c0 2024-02-19T21:09:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5fe9ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5fe9ffb700 10 client.?.objecter ms_handle_connect 0x7f5fec0fd980 2024-02-19T21:09:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5fe9ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5fe9ffb700 10 client.?.objecter ms_handle_connect 0x7f5fec100780 2024-02-19T21:09:36.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5fe9ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956769443 0 0) 0x7f5fec119580 con 0x7f5fec1042c0 2024-02-19T21:09:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fdc0038e0 con 0x7f5fec1042c0 2024-02-19T21:09:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2217016462 0 0) 0x7f5fec1171c0 con 0x7f5fec0fd980 2024-02-19T21:09:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fec119580 con 0x7f5fec0fd980 2024-02-19T21:09:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695668285 0 0) 0x7f5fec1183a0 con 0x7f5fec100780 2024-02-19T21:09:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.387+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fec1171c0 con 0x7f5fec100780 2024-02-19T21:09:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3428439220 0 0) 0x7f5fec1171c0 con 0x7f5fec100780 2024-02-19T21:09:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fec1183a0 con 0x7f5fec100780 2024-02-19T21:09:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5fd8001f30 con 0x7f5fec100780 2024-02-19T21:09:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2495963465 0 0) 0x7f5fdc0038e0 con 0x7f5fec1042c0 2024-02-19T21:09:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fec1171c0 con 0x7f5fec1042c0 2024-02-19T21:09:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5fe4001850 con 0x7f5fec1042c0 2024-02-19T21:09:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 876324921 0 0) 0x7f5fec119580 con 0x7f5fec0fd980 2024-02-19T21:09:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fdc0038e0 con 0x7f5fec0fd980 2024-02-19T21:09:36.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5fe00019f0 con 0x7f5fec0fd980 2024-02-19T21:09:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3200817749 0 0) 0x7f5fec1183a0 con 0x7f5fec100780 2024-02-19T21:09:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 >> v1:172.21.15.163:6790/0 conn(0x7f5fec1042c0 legacy=0x7f5fec115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 >> v1:172.21.15.163:6789/0 conn(0x7f5fec0fd980 legacy=0x7f5fec0f9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec11a780 con 0x7f5fec100780 2024-02-19T21:09:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5ff2af4700 1 -- 172.21.15.163:0/6057074 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fec1173b0 con 0x7f5fec100780 2024-02-19T21:09:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f5fd8002eb0 con 0x7f5fec100780 2024-02-19T21:09:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5fd80036d0 con 0x7f5fec100780 2024-02-19T21:09:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5ff2af4700 10 client.48574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5ff2af4700 1 -- 172.21.15.163:0/6057074 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fec1178a0 con 0x7f5fec100780 2024-02-19T21:09:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.388+0000 7f5ff2af4700 1 librados: init done 2024-02-19T21:09:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.389+0000 7f5ff2af4700 1 -- 172.21.15.163:0/6057074 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fec05f660 con 0x7f5fec100780 2024-02-19T21:09:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.390+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f5fd8001820 con 0x7f5fec100780 2024-02-19T21:09:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.390+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(393..393 src has 43..393) v4 ==== 7902+0+0 (unknown 2896911916 0 0) 0x7f5fd80766b0 con 0x7f5fec100780 2024-02-19T21:09:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.390+0000 7f5fe9ffb700 10 client.48574.objecter ms_dispatch 0x7f5fec001550 osd_map(393..393 src has 43..393) v4 2024-02-19T21:09:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.390+0000 7f5fe9ffb700 3 client.48574.objecter handle_osd_map got epochs [393,393] > 0 2024-02-19T21:09:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.390+0000 7f5fe9ffb700 3 client.48574.objecter handle_osd_map decoding full epoch 393 2024-02-19T21:09:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.390+0000 7f5fe9ffb700 20 client.48574.objecter dump_active .. 0 homeless 2024-02-19T21:09:36.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.390+0000 7f5fe9ffb700 10 client.48574.objecter ms_handle_connect 0x7f5fdc063d30 2024-02-19T21:09:36.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.395+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5fd8049b40 con 0x7f5fec100780 2024-02-19T21:09:36.418 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:09:36.420 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:36.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:09:36.422 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:36.521 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:36.549 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:36.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:36.578+0000 7f5ff2af4700 1 -- 172.21.15.163:0/6057074 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f5fec109f40 con 0x7f5fec100780 2024-02-19T21:09:36.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:34.639534+0000 osd.1 (osd.1) 175 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:09:36.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:34.642426+0000 osd.1 (osd.1) 176 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:09:36.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:34.787957+0000 osd.7 (osd.7) 163 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:09:36.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:34.791056+0000 osd.7 (osd.7) 164 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:09:36.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:35.202151+0000 mgr.y (mgr.14550) 353 : cluster [DBG] pgmap v560: 196 pgs: 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:36.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:35.283746+0000 osd.1 (osd.1) 177 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:36.664 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:35.283894+0000 osd.1 (osd.1) 178 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:36.664 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:35.310425+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2024-02-19T21:09:36.664 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:35.318097+0000 mon.b (mon.1) 556 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:36.664 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:35.320299+0000 mon.a (mon.0) 2448 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:36.664 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:35.625389+0000 osd.3 (osd.3) 165 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:09:36.664 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:35.626543+0000 osd.3 (osd.3) 166 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:09:36.664 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:34.639534+0000 osd.1 (osd.1) 175 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:09:36.664 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:34.642426+0000 osd.1 (osd.1) 176 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:09:36.665 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:34.787957+0000 osd.7 (osd.7) 163 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:09:36.665 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:34.791056+0000 osd.7 (osd.7) 164 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:09:36.665 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:35.202151+0000 mgr.y (mgr.14550) 353 : cluster [DBG] pgmap v560: 196 pgs: 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:36.665 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:35.283746+0000 osd.1 (osd.1) 177 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:36.665 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:35.283894+0000 osd.1 (osd.1) 178 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:36.665 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:35.310425+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2024-02-19T21:09:36.665 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:35.318097+0000 mon.b (mon.1) 556 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:36.666 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:35.320299+0000 mon.a (mon.0) 2448 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:36.666 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:35.625389+0000 osd.3 (osd.3) 165 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:09:36.666 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:35.626543+0000 osd.3 (osd.3) 166 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:09:36.697 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:36.699 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:34.639534+0000 osd.1 (osd.1) 175 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:09:36.699 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:34.642426+0000 osd.1 (osd.1) 176 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:09:36.699 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:34.787957+0000 osd.7 (osd.7) 163 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:09:36.699 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:34.791056+0000 osd.7 (osd.7) 164 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:09:36.699 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:35.202151+0000 mgr.y (mgr.14550) 353 : cluster [DBG] pgmap v560: 196 pgs: 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:36.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:35.283746+0000 osd.1 (osd.1) 177 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:36.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:35.283894+0000 osd.1 (osd.1) 178 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:36.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:35.310425+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2024-02-19T21:09:36.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:35.318097+0000 mon.b (mon.1) 556 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:36.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:35.320299+0000 mon.a (mon.0) 2448 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:36.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:35.625389+0000 osd.3 (osd.3) 165 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:09:36.700 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:35.626543+0000 osd.3 (osd.3) 166 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:09:36.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:36.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:36.803 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:36.831 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:36.936 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:36.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:36.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:36.952 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:09:36.952 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:09:36.952 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-02-19T21:09:36.952 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-02-19T21:09:36.952 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:09:37.040 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:09:37.068 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:37.142 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:09:37.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:37.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:09:37.246 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:37.274 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:37.320 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:35.821411+0000 osd.7 (osd.7) 165 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:09:37.320 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:35.824517+0000 osd.7 (osd.7) 166 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:09:37.321 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:36.309307+0000 mon.a (mon.0) 2449 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:37.321 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:36.309387+0000 mon.a (mon.0) 2450 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2024-02-19T21:09:37.321 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:36.325131+0000 mon.b (mon.1) 557 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:09:37.321 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:36.373051+0000 mon.c (mon.2) 383 : audit [INF] from='client.? v1:172.21.15.163:0/4237921293' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:37.321 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:36.373376+0000 mon.a (mon.0) 2451 : audit [INF] from='client.49751 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:37.321 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:36.579740+0000 mon.b (mon.1) 558 : audit [INF] from='client.? v1:172.21.15.163:0/6057074' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:09:37.322 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:36.580138+0000 mon.a (mon.0) 2452 : audit [INF] from='client.48574 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:09:37.322 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:36.695175+0000 mon.b (mon.1) 559 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:37.322 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:36.695540+0000 mon.a (mon.0) 2453 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:37.322 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:36.696135+0000 mon.b (mon.1) 560 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-53"}]: dispatch 2024-02-19T21:09:37.322 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:36.696351+0000 mon.a (mon.0) 2454 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-53"}]: dispatch 2024-02-19T21:09:37.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.327+0000 7f7e627fc700 1 -- 172.21.15.163:0/4237921293 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v394) v1 ==== 101+0+0 (unknown 265146514 0 0) 0x7f7e5c04a500 con 0x7f7e6c06bc60 2024-02-19T21:09:37.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.330+0000 7f5fe9ffb700 1 -- 172.21.15.163:0/6057074 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v394) v1 ==== 113+0+0 (unknown 2135746032 0 0) 0x7f5fd8047fb0 con 0x7f5fec100780 2024-02-19T21:09:37.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.331+0000 7f7e737bd700 10 librados: watch_flush enter 2024-02-19T21:09:37.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.331+0000 7f7e737bd700 10 librados: watch_flush exit 2024-02-19T21:09:37.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.331+0000 7f7e737bd700 20 client.49751.objecter shutdown clearing up homeless session... 2024-02-19T21:09:37.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.331+0000 7f7e737bd700 10 client.49751.objecter successfully canceled tick 2024-02-19T21:09:37.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.331+0000 7f7e737bd700 1 -- 172.21.15.163:0/4237921293 >> v1:172.21.15.163:6800/1552514681 conn(0x7f7e58063df0 legacy=0x7f7e580662b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:37.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.331+0000 7f7e737bd700 1 -- 172.21.15.163:0/4237921293 >> v1:172.21.15.163:6790/0 conn(0x7f7e6c06bc60 legacy=0x7f7e6c101f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:37.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.332+0000 7f7e737bd700 1 -- 172.21.15.163:0/4237921293 shutdown_connections 2024-02-19T21:09:37.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.332+0000 7f7e737bd700 1 -- 172.21.15.163:0/4237921293 >> 172.21.15.163:0/4237921293 conn(0x7f7e6c0f35a0 msgr2=0x7f7e6c109950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:37.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.332+0000 7f7e737bd700 1 -- 172.21.15.163:0/4237921293 shutdown_connections 2024-02-19T21:09:37.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.332+0000 7f7e737bd700 1 -- 172.21.15.163:0/4237921293 wait complete. 2024-02-19T21:09:37.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.333+0000 7f7e737bd700 1 librados: shutdown 2024-02-19T21:09:37.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.333+0000 7f5ff2af4700 10 librados: watch_flush enter 2024-02-19T21:09:37.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.333+0000 7f5ff2af4700 10 librados: watch_flush exit 2024-02-19T21:09:37.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.333+0000 7f5ff2af4700 20 client.48574.objecter shutdown clearing up homeless session... 2024-02-19T21:09:37.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.333+0000 7f5ff2af4700 10 client.48574.objecter successfully canceled tick 2024-02-19T21:09:37.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.333+0000 7f5ff2af4700 1 -- 172.21.15.163:0/6057074 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5fdc063d30 legacy=0x7f5fdc0661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:37.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.334+0000 7f5ff2af4700 1 -- 172.21.15.163:0/6057074 >> v1:172.21.15.193:6789/0 conn(0x7f5fec100780 legacy=0x7f5fec0f9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:37.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.334+0000 7f5ff2af4700 1 -- 172.21.15.163:0/6057074 shutdown_connections 2024-02-19T21:09:37.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.334+0000 7f5ff2af4700 1 -- 172.21.15.163:0/6057074 >> 172.21.15.163:0/6057074 conn(0x7f5fec0f35a0 msgr2=0x7f5fec107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:37.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.334+0000 7f5ff2af4700 1 -- 172.21.15.163:0/6057074 shutdown_connections 2024-02-19T21:09:37.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.334+0000 7f5ff2af4700 1 -- 172.21.15.163:0/6057074 wait complete. 2024-02-19T21:09:37.333 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:09:37.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:37.334+0000 7f5ff2af4700 1 librados: shutdown 2024-02-19T21:09:37.334 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.7 to 1 (10000) 2024-02-19T21:09:37.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:35.821411+0000 osd.7 (osd.7) 165 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:09:37.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:35.824517+0000 osd.7 (osd.7) 166 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:09:37.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:36.309307+0000 mon.a (mon.0) 2449 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:37.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:36.309387+0000 mon.a (mon.0) 2450 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2024-02-19T21:09:37.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:36.325131+0000 mon.b (mon.1) 557 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:09:37.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:36.373051+0000 mon.c (mon.2) 383 : audit [INF] from='client.? v1:172.21.15.163:0/4237921293' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:37.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:36.373376+0000 mon.a (mon.0) 2451 : audit [INF] from='client.49751 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:37.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:36.579740+0000 mon.b (mon.1) 558 : audit [INF] from='client.? v1:172.21.15.163:0/6057074' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:09:37.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:36.580138+0000 mon.a (mon.0) 2452 : audit [INF] from='client.48574 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:09:37.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:36.695175+0000 mon.b (mon.1) 559 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:37.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:36.695540+0000 mon.a (mon.0) 2453 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:37.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:36.696135+0000 mon.b (mon.1) 560 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-53"}]: dispatch 2024-02-19T21:09:37.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:36.696351+0000 mon.a (mon.0) 2454 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-53"}]: dispatch 2024-02-19T21:09:37.614 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-19T21:09:37.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:35.821411+0000 osd.7 (osd.7) 165 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:09:37.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:35.824517+0000 osd.7 (osd.7) 166 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:09:37.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:36.309307+0000 mon.a (mon.0) 2449 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:37.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:36.309387+0000 mon.a (mon.0) 2450 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2024-02-19T21:09:37.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:36.325131+0000 mon.b (mon.1) 557 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:09:37.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:36.373051+0000 mon.c (mon.2) 383 : audit [INF] from='client.? v1:172.21.15.163:0/4237921293' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:37.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:36.373376+0000 mon.a (mon.0) 2451 : audit [INF] from='client.49751 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:37.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:36.579740+0000 mon.b (mon.1) 558 : audit [INF] from='client.? v1:172.21.15.163:0/6057074' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:09:37.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:36.580138+0000 mon.a (mon.0) 2452 : audit [INF] from='client.48574 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:09:37.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:36.695175+0000 mon.b (mon.1) 559 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:37.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:36.695540+0000 mon.a (mon.0) 2453 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:37.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:36.696135+0000 mon.b (mon.1) 560 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-53"}]: dispatch 2024-02-19T21:09:37.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:36.696351+0000 mon.a (mon.0) 2454 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-53"}]: dispatch 2024-02-19T21:09:37.722 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:37.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:37.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:37.825 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:37.853 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:37.920 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:09:37.951 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:37.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:37.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:38.055 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:38.068 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:09:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:09:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:09:38.081 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:38.180 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:38.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:38.284 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:38.311 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:38.447 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:36.680756+0000 osd.1 (osd.1) 179 : cluster [DBG] 5.c scrub starts 2024-02-19T21:09:38.447 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:36.685055+0000 osd.1 (osd.1) 180 : cluster [DBG] 5.c scrub ok 2024-02-19T21:09:38.447 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:36.859292+0000 osd.7 (osd.7) 167 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:09:38.447 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:36.862293+0000 osd.7 (osd.7) 168 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:09:38.447 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:37.202723+0000 mgr.y (mgr.14550) 354 : cluster [DBG] pgmap v563: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:09:38.447 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:37.317037+0000 mon.a (mon.0) 2455 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:09:38.447 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:37.324849+0000 mon.a (mon.0) 2456 : audit [INF] from='client.49751 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:09:38.447 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:37.324904+0000 mon.a (mon.0) 2457 : audit [INF] from='client.48574 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-19T21:09:38.448 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:37.324951+0000 mon.a (mon.0) 2458 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2024-02-19T21:09:38.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.692+0000 7f38bb718700 1 -- 172.21.15.163:0/2818449713 >> v1:172.21.15.163:6790/0 conn(0x7f38b40fd980 legacy=0x7f38b40fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:38.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.692+0000 7f38bb718700 1 -- 172.21.15.163:0/2818449713 shutdown_connections 2024-02-19T21:09:38.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.692+0000 7f38bb718700 1 -- 172.21.15.163:0/2818449713 >> 172.21.15.163:0/2818449713 conn(0x7f38b40f35a0 msgr2=0x7f38b40f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:38.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38bb718700 1 -- 172.21.15.163:0/2818449713 shutdown_connections 2024-02-19T21:09:38.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38bb718700 1 -- 172.21.15.163:0/2818449713 wait complete. 2024-02-19T21:09:38.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38bb718700 1 librados: starting msgr at 2024-02-19T21:09:38.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38bb718700 1 librados: starting objecter 2024-02-19T21:09:38.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38bb718700 1 Processor -- start 2024-02-19T21:09:38.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38bb718700 1 -- start start 2024-02-19T21:09:38.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38bb718700 1 librados: setting wanted keys 2024-02-19T21:09:38.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38bb718700 1 librados: calling monclient init 2024-02-19T21:09:38.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38bb718700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f38b4117260 con 0x7f38b41042c0 2024-02-19T21:09:38.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38bb718700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f38b4118440 con 0x7f38b4100780 2024-02-19T21:09:38.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38bb718700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f38b4119640 con 0x7f38b40fd980 2024-02-19T21:09:38.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38b94b4700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f38b40fd980 0x7f38b40f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:33470/0 (socket says 172.21.15.163:33470) 2024-02-19T21:09:38.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38b94b4700 1 -- 172.21.15.163:0/2036312708 learned_addr learned my addr 172.21.15.163:0/2036312708 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:38.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38aa7fc700 10 client.?.objecter ms_handle_connect 0x7f38b40fd980 2024-02-19T21:09:38.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38aa7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:38.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38aa7fc700 10 client.?.objecter ms_handle_connect 0x7f38b41042c0 2024-02-19T21:09:38.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.693+0000 7f38aa7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:38.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2818300858 0 0) 0x7f38b4119640 con 0x7f38b40fd980 2024-02-19T21:09:38.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a0003800 con 0x7f38b40fd980 2024-02-19T21:09:38.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295720768 0 0) 0x7f38b4117260 con 0x7f38b41042c0 2024-02-19T21:09:38.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b4119640 con 0x7f38b41042c0 2024-02-19T21:09:38.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 10 client.?.objecter ms_handle_connect 0x7f38b4100780 2024-02-19T21:09:38.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:38.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228196808 0 0) 0x7f38b4118440 con 0x7f38b4100780 2024-02-19T21:09:38.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b4117260 con 0x7f38b4100780 2024-02-19T21:09:38.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1161112810 0 0) 0x7f38a0003800 con 0x7f38b40fd980 2024-02-19T21:09:38.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38b4118440 con 0x7f38b40fd980 2024-02-19T21:09:38.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f38b0002ea0 con 0x7f38b40fd980 2024-02-19T21:09:38.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 962301864 0 0) 0x7f38b4119640 con 0x7f38b41042c0 2024-02-19T21:09:38.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38a0003800 con 0x7f38b41042c0 2024-02-19T21:09:38.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f38ac001bc0 con 0x7f38b41042c0 2024-02-19T21:09:38.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2046408714 0 0) 0x7f38b4117260 con 0x7f38b4100780 2024-02-19T21:09:38.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38b4119640 con 0x7f38b4100780 2024-02-19T21:09:38.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f38a4001620 con 0x7f38b4100780 2024-02-19T21:09:38.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2572751 0 0) 0x7f38b4118440 con 0x7f38b40fd980 2024-02-19T21:09:38.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 >> v1:172.21.15.193:6789/0 conn(0x7f38b4100780 legacy=0x7f38b40fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:38.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 >> v1:172.21.15.163:6789/0 conn(0x7f38b41042c0 legacy=0x7f38b4115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:38.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b411a840 con 0x7f38b40fd980 2024-02-19T21:09:38.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b4117450 con 0x7f38b40fd980 2024-02-19T21:09:38.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38bb718700 10 client.49763.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:38.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b4117940 con 0x7f38b40fd980 2024-02-19T21:09:38.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38bb718700 1 librados: init done 2024-02-19T21:09:38.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f38b0002710 con 0x7f38b40fd980 2024-02-19T21:09:38.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.694+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f38b0003960 con 0x7f38b40fd980 2024-02-19T21:09:38.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.695+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f3898004f00 con 0x7f38b40fd980 2024-02-19T21:09:38.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.695+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f38b0003140 con 0x7f38b40fd980 2024-02-19T21:09:38.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(395..395 src has 43..395) v4 ==== 7889+0+0 (unknown 1351255934 0 0) 0x7f38b0078fe0 con 0x7f38b40fd980 2024-02-19T21:09:38.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 osd_map(395..395 src has 43..395) v4 2024-02-19T21:09:38.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38aa7fc700 3 client.49763.objecter handle_osd_map got epochs [395,395] > 0 2024-02-19T21:09:38.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38aa7fc700 3 client.49763.objecter handle_osd_map decoding full epoch 395 2024-02-19T21:09:38.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38aa7fc700 20 client.49763.objecter dump_active .. 0 homeless 2024-02-19T21:09:38.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v395) v1 ==== 54+0+16 (unknown 2620210354 0 2420000192) 0x7f38b007b3d0 con 0x7f38b40fd980 2024-02-19T21:09:38.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38aa7fc700 10 client.49763.objecter ms_handle_connect 0x7f38a0063d80 2024-02-19T21:09:38.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38bb718700 10 client.49763.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:38.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 1 2024-02-19T21:09:38.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f38980051a0 osd=0 3 2024-02-19T21:09:38.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f38980051a0 osd=0 3 2024-02-19T21:09:38.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f38980051a0 osd=0 3 2024-02-19T21:09:38.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 1 2024-02-19T21:09:38.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f38980051a0 osd=0 3 2024-02-19T21:09:38.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 0 1 2024-02-19T21:09:38.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f38980051a0 osd=0 4 2024-02-19T21:09:38.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38bb718700 10 client.49763.objecter _send_command 1 2024-02-19T21:09:38.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3898004950 con 0x7f3898004530 2024-02-19T21:09:38.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.696+0000 7f38aa7fc700 10 client.49763.objecter ms_handle_connect 0x7f3898004530 2024-02-19T21:09:38.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.697+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3898004950 con 0x7f3898004530 2024-02-19T21:09:38.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.697+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:09:38.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.697+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 1 = system:0 2024-02-19T21:09:38.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.697+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f38980051a0 osd=0 4 2024-02-19T21:09:38.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.697+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 0 1 2024-02-19T21:09:38.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.712+0000 7f38bb718700 10 client.49763.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:38.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.712+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 2 2024-02-19T21:09:38.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.712+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f38980051a0 osd=0 3 2024-02-19T21:09:38.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.712+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f38980051a0 osd=0 3 2024-02-19T21:09:38.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.712+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f38980051a0 osd=0 3 2024-02-19T21:09:38.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.712+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 2 2024-02-19T21:09:38.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.712+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f38980051a0 osd=0 3 2024-02-19T21:09:38.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.712+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 0 2 2024-02-19T21:09:38.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.712+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f38980051a0 osd=0 4 2024-02-19T21:09:38.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.712+0000 7f38bb718700 10 client.49763.objecter _send_command 2 2024-02-19T21:09:38.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.712+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3898004d30 con 0x7f3898004530 2024-02-19T21:09:38.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3898004d30 con 0x7f3898004530 2024-02-19T21:09:38.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:38.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:38.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f38980051a0 osd=0 4 2024-02-19T21:09:38.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 0 2 2024-02-19T21:09:38.718 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:09:38.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38bb718700 10 client.49763.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:38.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 3 2024-02-19T21:09:38.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898003fe0 osd=1 3 2024-02-19T21:09:38.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898003fe0 osd=1 3 2024-02-19T21:09:38.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898003fe0 osd=1 3 2024-02-19T21:09:38.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 3 2024-02-19T21:09:38.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f3898003fe0 osd=1 3 2024-02-19T21:09:38.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 1 3 2024-02-19T21:09:38.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898003fe0 osd=1 4 2024-02-19T21:09:38.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38bb718700 10 client.49763.objecter _send_command 3 2024-02-19T21:09:38.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.713+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f389800c000 con 0x7f3898004150 2024-02-19T21:09:38.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.714+0000 7f38aa7fc700 10 client.49763.objecter ms_handle_connect 0x7f3898004150 2024-02-19T21:09:38.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.714+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f389800c000 con 0x7f3898004150 2024-02-19T21:09:38.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.714+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:09:38.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.714+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 3 = system:0 2024-02-19T21:09:38.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.714+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f3898003fe0 osd=1 4 2024-02-19T21:09:38.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.714+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 1 3 2024-02-19T21:09:38.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.729+0000 7f38bb718700 10 client.49763.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:38.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.729+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 4 2024-02-19T21:09:38.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.729+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898003fe0 osd=1 3 2024-02-19T21:09:38.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.729+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898003fe0 osd=1 3 2024-02-19T21:09:38.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.729+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898003fe0 osd=1 3 2024-02-19T21:09:38.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.729+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 4 2024-02-19T21:09:38.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.729+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f3898003fe0 osd=1 3 2024-02-19T21:09:38.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.729+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 1 4 2024-02-19T21:09:38.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.729+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898003fe0 osd=1 4 2024-02-19T21:09:38.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.729+0000 7f38bb718700 10 client.49763.objecter _send_command 4 2024-02-19T21:09:38.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.729+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3898001850 con 0x7f3898004150 2024-02-19T21:09:38.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3898001850 con 0x7f3898004150 2024-02-19T21:09:38.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:38.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:38.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f3898003fe0 osd=1 4 2024-02-19T21:09:38.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 1 4 2024-02-19T21:09:38.729 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:09:38.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38bb718700 10 client.49763.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:38.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 5 2024-02-19T21:09:38.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898003630 osd=2 3 2024-02-19T21:09:38.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898003630 osd=2 3 2024-02-19T21:09:38.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898003630 osd=2 3 2024-02-19T21:09:38.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 5 2024-02-19T21:09:38.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f3898003630 osd=2 3 2024-02-19T21:09:38.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 2 5 2024-02-19T21:09:38.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898003630 osd=2 4 2024-02-19T21:09:38.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38bb718700 10 client.49763.objecter _send_command 5 2024-02-19T21:09:38.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.730+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3898003e70 con 0x7f389800d2f0 2024-02-19T21:09:38.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.731+0000 7f38aa7fc700 10 client.49763.objecter ms_handle_connect 0x7f389800d2f0 2024-02-19T21:09:38.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.731+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3898003e70 con 0x7f389800d2f0 2024-02-19T21:09:38.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.731+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:09:38.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.731+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 5 = system:0 2024-02-19T21:09:38.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.731+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f3898003630 osd=2 4 2024-02-19T21:09:38.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.731+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 2 5 2024-02-19T21:09:38.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.746+0000 7f38bb718700 10 client.49763.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:38.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.746+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 6 2024-02-19T21:09:38.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.746+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898003630 osd=2 3 2024-02-19T21:09:38.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.746+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898003630 osd=2 3 2024-02-19T21:09:38.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.746+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898003630 osd=2 3 2024-02-19T21:09:38.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.746+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 6 2024-02-19T21:09:38.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.746+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f3898003630 osd=2 3 2024-02-19T21:09:38.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.746+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 2 6 2024-02-19T21:09:38.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.746+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898003630 osd=2 4 2024-02-19T21:09:38.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.746+0000 7f38bb718700 10 client.49763.objecter _send_command 6 2024-02-19T21:09:38.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.746+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f389800e6e0 con 0x7f389800d2f0 2024-02-19T21:09:38.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:36.680756+0000 osd.1 (osd.1) 179 : cluster [DBG] 5.c scrub starts 2024-02-19T21:09:38.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:36.685055+0000 osd.1 (osd.1) 180 : cluster [DBG] 5.c scrub ok 2024-02-19T21:09:38.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:36.859292+0000 osd.7 (osd.7) 167 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:09:38.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:36.862293+0000 osd.7 (osd.7) 168 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:09:38.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:37.202723+0000 mgr.y (mgr.14550) 354 : cluster [DBG] pgmap v563: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:09:38.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:37.317037+0000 mon.a (mon.0) 2455 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:09:38.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:37.324849+0000 mon.a (mon.0) 2456 : audit [INF] from='client.49751 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:09:38.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:37.324904+0000 mon.a (mon.0) 2457 : audit [INF] from='client.48574 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-19T21:09:38.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:37.324951+0000 mon.a (mon.0) 2458 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2024-02-19T21:09:38.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:36.680756+0000 osd.1 (osd.1) 179 : cluster [DBG] 5.c scrub starts 2024-02-19T21:09:38.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:36.685055+0000 osd.1 (osd.1) 180 : cluster [DBG] 5.c scrub ok 2024-02-19T21:09:38.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:36.859292+0000 osd.7 (osd.7) 167 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:09:38.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:36.862293+0000 osd.7 (osd.7) 168 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:09:38.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:37.202723+0000 mgr.y (mgr.14550) 354 : cluster [DBG] pgmap v563: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:09:38.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:37.317037+0000 mon.a (mon.0) 2455 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:09:38.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:37.324849+0000 mon.a (mon.0) 2456 : audit [INF] from='client.49751 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:09:38.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:37.324904+0000 mon.a (mon.0) 2457 : audit [INF] from='client.48574 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-19T21:09:38.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:37.324951+0000 mon.a (mon.0) 2458 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2024-02-19T21:09:38.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f389800e6e0 con 0x7f389800d2f0 2024-02-19T21:09:38.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:38.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:38.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f3898003630 osd=2 4 2024-02-19T21:09:38.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 2 6 2024-02-19T21:09:38.747 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:09:38.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38bb718700 10 client.49763.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:38.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 7 2024-02-19T21:09:38.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898003440 osd=3 3 2024-02-19T21:09:38.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898003440 osd=3 3 2024-02-19T21:09:38.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898003440 osd=3 3 2024-02-19T21:09:38.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 7 2024-02-19T21:09:38.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f3898003440 osd=3 3 2024-02-19T21:09:38.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 3 7 2024-02-19T21:09:38.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898003440 osd=3 4 2024-02-19T21:09:38.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38bb718700 10 client.49763.objecter _send_command 7 2024-02-19T21:09:38.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.747+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3898015b40 con 0x7f3898011e80 2024-02-19T21:09:38.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.748+0000 7f38aa7fc700 10 client.49763.objecter ms_handle_connect 0x7f3898011e80 2024-02-19T21:09:38.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.748+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3898015b40 con 0x7f3898011e80 2024-02-19T21:09:38.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.748+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:09:38.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.748+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 7 = system:0 2024-02-19T21:09:38.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.748+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f3898003440 osd=3 4 2024-02-19T21:09:38.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.748+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 3 7 2024-02-19T21:09:38.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.762+0000 7f38bb718700 10 client.49763.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:38.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.763+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 8 2024-02-19T21:09:38.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.763+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898003440 osd=3 3 2024-02-19T21:09:38.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.763+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898003440 osd=3 3 2024-02-19T21:09:38.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.763+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898003440 osd=3 3 2024-02-19T21:09:38.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.763+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 8 2024-02-19T21:09:38.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.763+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f3898003440 osd=3 3 2024-02-19T21:09:38.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.763+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 3 8 2024-02-19T21:09:38.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.763+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898003440 osd=3 4 2024-02-19T21:09:38.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.763+0000 7f38bb718700 10 client.49763.objecter _send_command 8 2024-02-19T21:09:38.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.763+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f389800e150 con 0x7f3898011e80 2024-02-19T21:09:38.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f389800e150 con 0x7f3898011e80 2024-02-19T21:09:38.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:38.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:38.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f3898003440 osd=3 4 2024-02-19T21:09:38.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 3 8 2024-02-19T21:09:38.762 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:09:38.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38bb718700 10 client.49763.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:38.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 9 2024-02-19T21:09:38.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f389800e150 osd=4 3 2024-02-19T21:09:38.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f389800e150 osd=4 3 2024-02-19T21:09:38.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f389800e150 osd=4 3 2024-02-19T21:09:38.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 9 2024-02-19T21:09:38.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f389800e150 osd=4 3 2024-02-19T21:09:38.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 4 9 2024-02-19T21:09:38.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f389800e150 osd=4 4 2024-02-19T21:09:38.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38bb718700 10 client.49763.objecter _send_command 9 2024-02-19T21:09:38.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.764+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f389800e480 con 0x7f389800ce50 2024-02-19T21:09:38.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.765+0000 7f38aa7fc700 10 client.49763.objecter ms_handle_connect 0x7f389800ce50 2024-02-19T21:09:38.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.765+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f389800e480 con 0x7f389800ce50 2024-02-19T21:09:38.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.765+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:09:38.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.765+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 9 = system:0 2024-02-19T21:09:38.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.765+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f389800e150 osd=4 4 2024-02-19T21:09:38.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.765+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 4 9 2024-02-19T21:09:38.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.780+0000 7f38bb718700 10 client.49763.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:38.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.780+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 10 2024-02-19T21:09:38.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.780+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f389800e150 osd=4 3 2024-02-19T21:09:38.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.780+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f389800e150 osd=4 3 2024-02-19T21:09:38.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.780+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f389800e150 osd=4 3 2024-02-19T21:09:38.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.780+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 10 2024-02-19T21:09:38.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.780+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f389800e150 osd=4 3 2024-02-19T21:09:38.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.780+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 4 10 2024-02-19T21:09:38.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.780+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f389800e150 osd=4 4 2024-02-19T21:09:38.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.780+0000 7f38bb718700 10 client.49763.objecter _send_command 10 2024-02-19T21:09:38.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.780+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f389800e480 con 0x7f389800ce50 2024-02-19T21:09:38.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.781+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f389800e480 con 0x7f389800ce50 2024-02-19T21:09:38.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.781+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:38.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.781+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:38.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.781+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f389800e150 osd=4 4 2024-02-19T21:09:38.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.781+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 4 10 2024-02-19T21:09:38.780 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:09:38.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.782+0000 7f38bb718700 10 client.49763.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:38.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.782+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 11 2024-02-19T21:09:38.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.782+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f389800e480 osd=5 3 2024-02-19T21:09:38.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.782+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f389800e480 osd=5 3 2024-02-19T21:09:38.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.782+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f389800e480 osd=5 3 2024-02-19T21:09:38.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.782+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 11 2024-02-19T21:09:38.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.782+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f389800e480 osd=5 3 2024-02-19T21:09:38.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.782+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 5 11 2024-02-19T21:09:38.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.782+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f389800e480 osd=5 4 2024-02-19T21:09:38.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.782+0000 7f38bb718700 10 client.49763.objecter _send_command 11 2024-02-19T21:09:38.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.782+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3898018050 con 0x7f3898017a80 2024-02-19T21:09:38.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.782+0000 7f38aa7fc700 10 client.49763.objecter ms_handle_connect 0x7f3898017a80 2024-02-19T21:09:38.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.783+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3898018050 con 0x7f3898017a80 2024-02-19T21:09:38.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.783+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:09:38.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.783+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 11 = system:0 2024-02-19T21:09:38.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.783+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f389800e480 osd=5 4 2024-02-19T21:09:38.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.783+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 5 11 2024-02-19T21:09:38.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.797+0000 7f38bb718700 10 client.49763.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:38.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.797+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 12 2024-02-19T21:09:38.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.797+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f389800e480 osd=5 3 2024-02-19T21:09:38.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.797+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f389800e480 osd=5 3 2024-02-19T21:09:38.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.797+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f389800e480 osd=5 3 2024-02-19T21:09:38.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.797+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 12 2024-02-19T21:09:38.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.797+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f389800e480 osd=5 3 2024-02-19T21:09:38.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.797+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 5 12 2024-02-19T21:09:38.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.797+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f389800e480 osd=5 4 2024-02-19T21:09:38.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.797+0000 7f38bb718700 10 client.49763.objecter _send_command 12 2024-02-19T21:09:38.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.797+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3898018050 con 0x7f3898017a80 2024-02-19T21:09:38.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3898018050 con 0x7f3898017a80 2024-02-19T21:09:38.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:38.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:38.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f389800e480 osd=5 4 2024-02-19T21:09:38.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 5 12 2024-02-19T21:09:38.797 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:09:38.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38bb718700 10 client.49763.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:38.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 13 2024-02-19T21:09:38.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898018050 osd=6 3 2024-02-19T21:09:38.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898018050 osd=6 3 2024-02-19T21:09:38.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898018050 osd=6 3 2024-02-19T21:09:38.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 13 2024-02-19T21:09:38.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f3898018050 osd=6 3 2024-02-19T21:09:38.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 6 13 2024-02-19T21:09:38.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898018050 osd=6 4 2024-02-19T21:09:38.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38bb718700 10 client.49763.objecter _send_command 13 2024-02-19T21:09:38.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3898022c50 con 0x7f389801f140 2024-02-19T21:09:38.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.799+0000 7f38aa7fc700 10 client.49763.objecter ms_handle_connect 0x7f389801f140 2024-02-19T21:09:38.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.800+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3898022c50 con 0x7f389801f140 2024-02-19T21:09:38.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.800+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:09:38.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.800+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 13 = system:0 2024-02-19T21:09:38.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.800+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f3898018050 osd=6 4 2024-02-19T21:09:38.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.800+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 6 13 2024-02-19T21:09:38.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.814+0000 7f38bb718700 10 client.49763.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:38.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.814+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 14 2024-02-19T21:09:38.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.814+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898018050 osd=6 3 2024-02-19T21:09:38.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.814+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898018050 osd=6 3 2024-02-19T21:09:38.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.814+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898018050 osd=6 3 2024-02-19T21:09:38.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.814+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 14 2024-02-19T21:09:38.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.814+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f3898018050 osd=6 3 2024-02-19T21:09:38.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.814+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 6 14 2024-02-19T21:09:38.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.814+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898018050 osd=6 4 2024-02-19T21:09:38.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.814+0000 7f38bb718700 10 client.49763.objecter _send_command 14 2024-02-19T21:09:38.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.814+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3898022c50 con 0x7f389801f140 2024-02-19T21:09:38.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3898022c50 con 0x7f389801f140 2024-02-19T21:09:38.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:38.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:38.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f3898018050 osd=6 4 2024-02-19T21:09:38.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 6 14 2024-02-19T21:09:38.814 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:09:38.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38bb718700 10 client.49763.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:38.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 15 2024-02-19T21:09:38.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898022c50 osd=7 3 2024-02-19T21:09:38.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898022c50 osd=7 3 2024-02-19T21:09:38.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898022c50 osd=7 3 2024-02-19T21:09:38.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 15 2024-02-19T21:09:38.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f3898022c50 osd=7 3 2024-02-19T21:09:38.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 7 15 2024-02-19T21:09:38.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898022c50 osd=7 4 2024-02-19T21:09:38.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38bb718700 10 client.49763.objecter _send_command 15 2024-02-19T21:09:38.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3898023eb0 con 0x7f3898024950 2024-02-19T21:09:38.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.816+0000 7f38aa7fc700 10 client.49763.objecter ms_handle_connect 0x7f3898024950 2024-02-19T21:09:38.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.817+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3898023eb0 con 0x7f3898024950 2024-02-19T21:09:38.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.817+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:09:38.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.817+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 15 = system:0 2024-02-19T21:09:38.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.817+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f3898022c50 osd=7 4 2024-02-19T21:09:38.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.817+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 7 15 2024-02-19T21:09:38.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.831+0000 7f38bb718700 10 client.49763.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:38.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.831+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign -1 16 2024-02-19T21:09:38.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.831+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898022c50 osd=7 3 2024-02-19T21:09:38.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.831+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898022c50 osd=7 3 2024-02-19T21:09:38.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.831+0000 7f38bb718700 20 client.49763.objecter _get_session s=0x7f3898022c50 osd=7 3 2024-02-19T21:09:38.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.831+0000 7f38bb718700 15 client.49763.objecter _session_command_op_remove -1 16 2024-02-19T21:09:38.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.831+0000 7f38bb718700 20 client.49763.objecter get_session s=0x7f3898022c50 osd=7 3 2024-02-19T21:09:38.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.831+0000 7f38bb718700 15 client.49763.objecter _session_command_op_assign 7 16 2024-02-19T21:09:38.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.831+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898022c50 osd=7 4 2024-02-19T21:09:38.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.831+0000 7f38bb718700 10 client.49763.objecter _send_command 16 2024-02-19T21:09:38.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.831+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3898023e70 con 0x7f3898024950 2024-02-19T21:09:38.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.832+0000 7f38aa7fc700 1 -- 172.21.15.163:0/2036312708 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3898023e70 con 0x7f3898024950 2024-02-19T21:09:38.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.832+0000 7f38aa7fc700 10 client.49763.objecter ms_dispatch 0x7f38b4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:38.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.832+0000 7f38aa7fc700 10 client.49763.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:38.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.832+0000 7f38aa7fc700 20 client.49763.objecter put_session s=0x7f3898022c50 osd=7 4 2024-02-19T21:09:38.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.832+0000 7f38aa7fc700 15 client.49763.objecter _session_command_op_remove 7 16 2024-02-19T21:09:38.833 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:09:38.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 10 librados: watch_flush enter 2024-02-19T21:09:38.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 10 librados: watch_flush exit 2024-02-19T21:09:38.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 10 client.49763.objecter close_session for osd.0 2024-02-19T21:09:38.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 >> v1:172.21.15.163:6801/1559418115 conn(0x7f3898004530 legacy=0x7f3898002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:38.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f38980051a0 osd=0 1 2024-02-19T21:09:38.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 10 client.49763.objecter close_session for osd.1 2024-02-19T21:09:38.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 >> v1:172.21.15.163:6805/3125047432 conn(0x7f3898004150 legacy=0x7f389800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:38.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898003fe0 osd=1 1 2024-02-19T21:09:38.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 10 client.49763.objecter close_session for osd.2 2024-02-19T21:09:38.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 >> v1:172.21.15.163:6809/1563158487 conn(0x7f389800d2f0 legacy=0x7f389800d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:38.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898003630 osd=2 1 2024-02-19T21:09:38.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 10 client.49763.objecter close_session for osd.3 2024-02-19T21:09:38.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 >> v1:172.21.15.163:6813/2112279769 conn(0x7f3898011e80 legacy=0x7f3898014290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:38.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38b9cb5700 1 -- 172.21.15.163:0/2036312708 reap_dead start 2024-02-19T21:09:38.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898003440 osd=3 1 2024-02-19T21:09:38.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 10 client.49763.objecter close_session for osd.4 2024-02-19T21:09:38.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 >> v1:172.21.15.193:6800/3129716965 conn(0x7f389800ce50 legacy=0x7f3898015ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:38.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f389800e150 osd=4 1 2024-02-19T21:09:38.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 10 client.49763.objecter close_session for osd.5 2024-02-19T21:09:38.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 >> v1:172.21.15.193:6804/805161991 conn(0x7f3898017a80 legacy=0x7f389801d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:38.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f389800e480 osd=5 1 2024-02-19T21:09:38.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 10 client.49763.objecter close_session for osd.6 2024-02-19T21:09:38.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 >> v1:172.21.15.193:6808/787604615 conn(0x7f389801f140 legacy=0x7f3898021530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:38.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898018050 osd=6 1 2024-02-19T21:09:38.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 10 client.49763.objecter close_session for osd.7 2024-02-19T21:09:38.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 >> v1:172.21.15.193:6812/157927432 conn(0x7f3898024950 legacy=0x7f3898024db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:38.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 20 client.49763.objecter put_session s=0x7f3898022c50 osd=7 1 2024-02-19T21:09:38.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 20 client.49763.objecter shutdown clearing up homeless session... 2024-02-19T21:09:38.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 10 client.49763.objecter successfully canceled tick 2024-02-19T21:09:38.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38b9cb5700 1 -- 172.21.15.163:0/2036312708 reap_dead start 2024-02-19T21:09:38.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 >> v1:172.21.15.163:6800/1552514681 conn(0x7f38a0063d80 legacy=0x7f38a0066240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:38.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 >> v1:172.21.15.163:6790/0 conn(0x7f38b40fd980 legacy=0x7f38b40f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:38.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 shutdown_connections 2024-02-19T21:09:38.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 >> 172.21.15.163:0/2036312708 conn(0x7f38b40f35a0 msgr2=0x7f38b4107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:38.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 shutdown_connections 2024-02-19T21:09:38.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 1 -- 172.21.15.163:0/2036312708 wait complete. 2024-02-19T21:09:38.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:38.833+0000 7f38bb718700 1 librados: shutdown 2024-02-19T21:09:38.838 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:09:38.839 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:09:38.839 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:09:38.839 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:09:38.839 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:09:38.839 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:09:38.839 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:09:38.839 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:09:39.148 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 3 1 2024-02-19T21:09:39.165 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:39.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:39.269 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:39.300 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:39.398 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:39.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:39.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:37.864758+0000 osd.7 (osd.7) 169 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:09:39.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:37.870211+0000 osd.7 (osd.7) 170 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:09:39.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:37.894466+0000 osd.5 (osd.5) 156 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:37.894660+0000 osd.5 (osd.5) 157 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:37.952958+0000 osd.2 (osd.2) 142 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:09:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:37.955862+0000 osd.2 (osd.2) 143 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:09:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:38.330206+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2024-02-19T21:09:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:38.336739+0000 mon.b (mon.1) 561 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:38.337596+0000 mon.a (mon.0) 2460 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:38.540542+0000 osd.6 (osd.6) 151 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:09:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:38.541384+0000 osd.6 (osd.6) 152 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:09:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:38.697296+0000 mon.c (mon.2) 384 : audit [DBG] from='client.? v1:172.21.15.163:0/2036312708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:39.100740+0000 mon.a (mon.0) 2461 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:39.250948+0000 mon.a (mon.0) 2462 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:39.251059+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2024-02-19T21:09:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:39.304908+0000 mon.b (mon.1) 562 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:39.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:39.305239+0000 mon.a (mon.0) 2464 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:39.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:39.305765+0000 mon.b (mon.1) 563 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-55"}]: dispatch 2024-02-19T21:09:39.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:39.305952+0000 mon.a (mon.0) 2465 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-55"}]: dispatch 2024-02-19T21:09:39.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:37.864758+0000 osd.7 (osd.7) 169 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:09:39.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:37.870211+0000 osd.7 (osd.7) 170 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:09:39.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:37.894466+0000 osd.5 (osd.5) 156 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:39.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:37.894660+0000 osd.5 (osd.5) 157 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:39.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:37.952958+0000 osd.2 (osd.2) 142 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:09:39.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:37.955862+0000 osd.2 (osd.2) 143 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:09:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:38.330206+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2024-02-19T21:09:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:38.336739+0000 mon.b (mon.1) 561 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:38.337596+0000 mon.a (mon.0) 2460 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:38.540542+0000 osd.6 (osd.6) 151 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:09:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:38.541384+0000 osd.6 (osd.6) 152 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:09:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:38.697296+0000 mon.c (mon.2) 384 : audit [DBG] from='client.? v1:172.21.15.163:0/2036312708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:39.100740+0000 mon.a (mon.0) 2461 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:39.250948+0000 mon.a (mon.0) 2462 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:39.251059+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2024-02-19T21:09:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:39.304908+0000 mon.b (mon.1) 562 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:39.305239+0000 mon.a (mon.0) 2464 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:39.305765+0000 mon.b (mon.1) 563 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-55"}]: dispatch 2024-02-19T21:09:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:39.305952+0000 mon.a (mon.0) 2465 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-55"}]: dispatch 2024-02-19T21:09:39.503 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:09:39.530 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:39.653 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:09:39.653 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:09:39.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:37.864758+0000 osd.7 (osd.7) 169 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:09:39.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:37.870211+0000 osd.7 (osd.7) 170 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:09:39.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:37.894466+0000 osd.5 (osd.5) 156 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:39.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:37.894660+0000 osd.5 (osd.5) 157 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:39.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:37.952958+0000 osd.2 (osd.2) 142 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:09:39.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:37.955862+0000 osd.2 (osd.2) 143 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:09:39.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:38.330206+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2024-02-19T21:09:39.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:38.336739+0000 mon.b (mon.1) 561 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:39.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:38.337596+0000 mon.a (mon.0) 2460 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:39.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:38.540542+0000 osd.6 (osd.6) 151 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:09:39.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:38.541384+0000 osd.6 (osd.6) 152 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:09:39.674 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:38.697296+0000 mon.c (mon.2) 384 : audit [DBG] from='client.? v1:172.21.15.163:0/2036312708' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:39.675 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:39.100740+0000 mon.a (mon.0) 2461 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:39.675 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:39.250948+0000 mon.a (mon.0) 2462 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:39.675 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:39.251059+0000 mon.a (mon.0) 2463 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2024-02-19T21:09:39.675 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:39.304908+0000 mon.b (mon.1) 562 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:39.675 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:39.305239+0000 mon.a (mon.0) 2464 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:39.675 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:39.305765+0000 mon.b (mon.1) 563 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-55"}]: dispatch 2024-02-19T21:09:39.675 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:39.305952+0000 mon.a (mon.0) 2465 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-55"}]: dispatch 2024-02-19T21:09:39.778 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:09:39.781 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:39.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:09:39.882 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:39.911 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:40.298 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-19T21:09:40.360 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:40.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:40.464 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:09:40.490 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:40.528 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:09:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:40.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:09:40.569 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:09:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:09:40.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:38.937813+0000 osd.5 (osd.5) 158 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:09:40.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:38.940930+0000 osd.5 (osd.5) 159 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:09:40.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:39.203097+0000 mgr.y (mgr.14550) 355 : cluster [DBG] pgmap v566: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:40.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:40.252965+0000 mon.a (mon.0) 2466 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2024-02-19T21:09:40.604 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:09:40.633 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:09:40.636 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:40.661 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:38.937813+0000 osd.5 (osd.5) 158 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:09:40.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:38.940930+0000 osd.5 (osd.5) 159 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:09:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:39.203097+0000 mgr.y (mgr.14550) 355 : cluster [DBG] pgmap v566: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:40.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:40.252965+0000 mon.a (mon.0) 2466 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2024-02-19T21:09:40.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:38.937813+0000 osd.5 (osd.5) 158 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:09:40.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:38.940930+0000 osd.5 (osd.5) 159 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:09:40.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:39.203097+0000 mgr.y (mgr.14550) 355 : cluster [DBG] pgmap v566: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:40.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:40.252965+0000 mon.a (mon.0) 2466 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2024-02-19T21:09:40.860 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:09:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:40.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:09:40.964 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:40.992 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:41.089 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:41.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:41.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:41.193 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:41.220 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:41.470 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:41.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:41.571 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:41.597 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:41.635 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:41.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:41.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:41.743 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:41.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:39.595531+0000 osd.3 (osd.3) 167 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:09:41.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:39.596996+0000 osd.3 (osd.3) 168 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:09:41.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:40.361362+0000 mgr.y (mgr.14550) 356 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:41.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:40.548882+0000 osd.3 (osd.3) 169 : cluster [DBG] 6.2 scrub starts 2024-02-19T21:09:41.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:40.549685+0000 osd.3 (osd.3) 170 : cluster [DBG] 6.2 scrub ok 2024-02-19T21:09:41.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:40.930258+0000 osd.7 (osd.7) 171 : cluster [DBG] 2.0 deep-scrub starts 2024-02-19T21:09:41.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:40.934468+0000 osd.7 (osd.7) 172 : cluster [DBG] 2.0 deep-scrub ok 2024-02-19T21:09:41.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:41.253709+0000 mon.a (mon.0) 2467 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2024-02-19T21:09:41.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:41.255221+0000 mon.b (mon.1) 564 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:41.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:41.256625+0000 mon.a (mon.0) 2468 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:41.769 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:41.774 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:09:41.801 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:39.595531+0000 osd.3 (osd.3) 167 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:09:41.801 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:39.596996+0000 osd.3 (osd.3) 168 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:09:41.801 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:40.361362+0000 mgr.y (mgr.14550) 356 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:41.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:40.548882+0000 osd.3 (osd.3) 169 : cluster [DBG] 6.2 scrub starts 2024-02-19T21:09:41.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:40.549685+0000 osd.3 (osd.3) 170 : cluster [DBG] 6.2 scrub ok 2024-02-19T21:09:41.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:40.930258+0000 osd.7 (osd.7) 171 : cluster [DBG] 2.0 deep-scrub starts 2024-02-19T21:09:41.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:40.934468+0000 osd.7 (osd.7) 172 : cluster [DBG] 2.0 deep-scrub ok 2024-02-19T21:09:41.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:41.253709+0000 mon.a (mon.0) 2467 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2024-02-19T21:09:41.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:41.255221+0000 mon.b (mon.1) 564 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:41.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:41.256625+0000 mon.a (mon.0) 2468 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:41.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:39.595531+0000 osd.3 (osd.3) 167 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:09:41.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:39.596996+0000 osd.3 (osd.3) 168 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:09:41.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:40.361362+0000 mgr.y (mgr.14550) 356 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:41.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:40.548882+0000 osd.3 (osd.3) 169 : cluster [DBG] 6.2 scrub starts 2024-02-19T21:09:41.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:40.549685+0000 osd.3 (osd.3) 170 : cluster [DBG] 6.2 scrub ok 2024-02-19T21:09:41.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:40.930258+0000 osd.7 (osd.7) 171 : cluster [DBG] 2.0 deep-scrub starts 2024-02-19T21:09:41.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:40.934468+0000 osd.7 (osd.7) 172 : cluster [DBG] 2.0 deep-scrub ok 2024-02-19T21:09:41.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:41.253709+0000 mon.a (mon.0) 2467 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2024-02-19T21:09:41.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:41.255221+0000 mon.b (mon.1) 564 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:41.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:41.256625+0000 mon.a (mon.0) 2468 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:41.868 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:41.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:41.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:41.972 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:42.000 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:42.099 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:42.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:42.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:42.203 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:09:42.234 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:42.336 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:09:42.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:42.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:09:42.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.400+0000 7f002fe27700 1 -- 172.21.15.163:0/3350698543 >> v1:172.21.15.163:6790/0 conn(0x7f0028100780 legacy=0x7f0028102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:42.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.400+0000 7f002fe27700 1 -- 172.21.15.163:0/3350698543 shutdown_connections 2024-02-19T21:09:42.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.400+0000 7f002fe27700 1 -- 172.21.15.163:0/3350698543 >> 172.21.15.163:0/3350698543 conn(0x7f00280f35a0 msgr2=0x7f00280f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:42.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.400+0000 7f002fe27700 1 -- 172.21.15.163:0/3350698543 shutdown_connections 2024-02-19T21:09:42.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.400+0000 7f002fe27700 1 -- 172.21.15.163:0/3350698543 wait complete. 2024-02-19T21:09:42.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f002fe27700 1 librados: starting msgr at 2024-02-19T21:09:42.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f002fe27700 1 librados: starting objecter 2024-02-19T21:09:42.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f002fe27700 1 Processor -- start 2024-02-19T21:09:42.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f002fe27700 1 -- start start 2024-02-19T21:09:42.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f002fe27700 1 librados: setting wanted keys 2024-02-19T21:09:42.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f002fe27700 1 librados: calling monclient init 2024-02-19T21:09:42.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f002fe27700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0028119450 con 0x7f00280fd980 2024-02-19T21:09:42.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f002fe27700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f002811a5f0 con 0x7f0028100780 2024-02-19T21:09:42.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f002fe27700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f002811b790 con 0x7f00281042c0 2024-02-19T21:09:42.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f002dbc3700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f00280fd980 0x7f00280fc5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:55368/0 (socket says 172.21.15.163:55368) 2024-02-19T21:09:42.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f002dbc3700 1 -- 172.21.15.163:0/3496772249 learned_addr learned my addr 172.21.15.163:0/3496772249 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:42.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 10 client.?.objecter ms_handle_connect 0x7f00281042c0 2024-02-19T21:09:42.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:42.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 10 client.?.objecter ms_handle_connect 0x7f00280fd980 2024-02-19T21:09:42.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:42.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 10 client.?.objecter ms_handle_connect 0x7f0028100780 2024-02-19T21:09:42.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:42.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735503305 0 0) 0x7f002811b790 con 0x7f00281042c0 2024-02-19T21:09:42.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00140033f0 con 0x7f00281042c0 2024-02-19T21:09:42.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032526754 0 0) 0x7f0028119450 con 0x7f00280fd980 2024-02-19T21:09:42.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f002811b790 con 0x7f00280fd980 2024-02-19T21:09:42.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292456793 0 0) 0x7f002811a5f0 con 0x7f0028100780 2024-02-19T21:09:42.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0028119450 con 0x7f0028100780 2024-02-19T21:09:42.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1362271107 0 0) 0x7f00140033f0 con 0x7f00281042c0 2024-02-19T21:09:42.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f002811a5f0 con 0x7f00281042c0 2024-02-19T21:09:42.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0020001920 con 0x7f00281042c0 2024-02-19T21:09:42.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 400236661 0 0) 0x7f002811b790 con 0x7f00280fd980 2024-02-19T21:09:42.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00140033f0 con 0x7f00280fd980 2024-02-19T21:09:42.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.401+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1142615593 0 0) 0x7f0028119450 con 0x7f0028100780 2024-02-19T21:09:42.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.402+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f002811b790 con 0x7f0028100780 2024-02-19T21:09:42.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.402+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0018001570 con 0x7f00280fd980 2024-02-19T21:09:42.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.402+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0024000fe0 con 0x7f0028100780 2024-02-19T21:09:42.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.402+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 307445335 0 0) 0x7f002811a5f0 con 0x7f00281042c0 2024-02-19T21:09:42.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.402+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 >> v1:172.21.15.193:6789/0 conn(0x7f0028100780 legacy=0x7f00280fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:42.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.402+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 >> v1:172.21.15.163:6789/0 conn(0x7f00280fd980 legacy=0x7f00280fc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:42.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.402+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002811c970 con 0x7f00281042c0 2024-02-19T21:09:42.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.402+0000 7f002fe27700 1 -- 172.21.15.163:0/3496772249 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0028119640 con 0x7f00281042c0 2024-02-19T21:09:42.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.402+0000 7f002fe27700 10 client.49769.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:42.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.402+0000 7f002fe27700 1 -- 172.21.15.163:0/3496772249 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0028119b30 con 0x7f00281042c0 2024-02-19T21:09:42.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.402+0000 7f002fe27700 1 librados: init done 2024-02-19T21:09:42.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.402+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f00200025a0 con 0x7f00281042c0 2024-02-19T21:09:42.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.402+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f00200035b0 con 0x7f00281042c0 2024-02-19T21:09:42.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.403+0000 7f002fe27700 1 -- 172.21.15.163:0/3496772249 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000c004f20 con 0x7f00281042c0 2024-02-19T21:09:42.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.403+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f0020002100 con 0x7f00281042c0 2024-02-19T21:09:42.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.404+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(399..399 src has 43..399) v4 ==== 7902+0+0 (unknown 3880020250 0 0) 0x7f0020076640 con 0x7f00281042c0 2024-02-19T21:09:42.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.404+0000 7f001effd700 10 client.49769.objecter ms_dispatch 0x7f0028001550 osd_map(399..399 src has 43..399) v4 2024-02-19T21:09:42.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.404+0000 7f001effd700 3 client.49769.objecter handle_osd_map got epochs [399,399] > 0 2024-02-19T21:09:42.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.404+0000 7f001effd700 3 client.49769.objecter handle_osd_map decoding full epoch 399 2024-02-19T21:09:42.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.404+0000 7f001effd700 20 client.49769.objecter dump_active .. 0 homeless 2024-02-19T21:09:42.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.404+0000 7f001effd700 10 client.49769.objecter ms_handle_connect 0x7f0014063d50 2024-02-19T21:09:42.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.408+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0020049ad0 con 0x7f00281042c0 2024-02-19T21:09:42.440 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:42.454 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:42.462 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:09:42.462 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:09:42.471 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:42.495 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:42.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:42.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:42.550 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:09:42.601 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:42.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:42.628+0000 7f002fe27700 1 -- 172.21.15.163:0/3496772249 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f000c0051d0 con 0x7f00281042c0 2024-02-19T21:09:42.627 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:42.727 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:42.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:40.672962+0000 osd.1 (osd.1) 181 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:09:42.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:40.676434+0000 osd.1 (osd.1) 182 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:09:42.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:40.983757+0000 osd.5 (osd.5) 160 : cluster [DBG] 2.12 deep-scrub starts 2024-02-19T21:09:42.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:40.986631+0000 osd.5 (osd.5) 161 : cluster [DBG] 2.12 deep-scrub ok 2024-02-19T21:09:42.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:41.203636+0000 mgr.y (mgr.14550) 357 : cluster [DBG] pgmap v569: 196 pgs: 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:09:42.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:41.516394+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:09:42.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:41.519286+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:09:42.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:41.948106+0000 osd.7 (osd.7) 173 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:09:42.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:41.951570+0000 osd.7 (osd.7) 174 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:09:42.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:42.260660+0000 mon.a (mon.0) 2469 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:42.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:42.260825+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e399: 8 total, 8 up, 7 in 2024-02-19T21:09:42.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:42.266470+0000 mon.b (mon.1) 565 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:09:42.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:42.300413+0000 mon.b (mon.1) 566 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:42.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:42.300827+0000 mon.a (mon.0) 2471 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:42.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:42.301412+0000 mon.b (mon.1) 567 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-57"}]: dispatch 2024-02-19T21:09:42.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:42.301739+0000 mon.a (mon.0) 2472 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-57"}]: dispatch 2024-02-19T21:09:42.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:42.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:42.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:40.672962+0000 osd.1 (osd.1) 181 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:09:42.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:40.676434+0000 osd.1 (osd.1) 182 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:09:42.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:40.983757+0000 osd.5 (osd.5) 160 : cluster [DBG] 2.12 deep-scrub starts 2024-02-19T21:09:42.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:40.986631+0000 osd.5 (osd.5) 161 : cluster [DBG] 2.12 deep-scrub ok 2024-02-19T21:09:42.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:41.203636+0000 mgr.y (mgr.14550) 357 : cluster [DBG] pgmap v569: 196 pgs: 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:09:42.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:41.516394+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:09:42.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:41.519286+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:09:42.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:41.948106+0000 osd.7 (osd.7) 173 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:09:42.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:41.951570+0000 osd.7 (osd.7) 174 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:09:42.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:42.260660+0000 mon.a (mon.0) 2469 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:42.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:42.260825+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e399: 8 total, 8 up, 7 in 2024-02-19T21:09:42.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:42.266470+0000 mon.b (mon.1) 565 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:09:42.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:42.300413+0000 mon.b (mon.1) 566 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:42.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:42.300827+0000 mon.a (mon.0) 2471 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:42.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:42.301412+0000 mon.b (mon.1) 567 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-57"}]: dispatch 2024-02-19T21:09:42.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:42.301739+0000 mon.a (mon.0) 2472 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-57"}]: dispatch 2024-02-19T21:09:42.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:40.672962+0000 osd.1 (osd.1) 181 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:09:42.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:40.676434+0000 osd.1 (osd.1) 182 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:09:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:40.983757+0000 osd.5 (osd.5) 160 : cluster [DBG] 2.12 deep-scrub starts 2024-02-19T21:09:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:40.986631+0000 osd.5 (osd.5) 161 : cluster [DBG] 2.12 deep-scrub ok 2024-02-19T21:09:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:41.203636+0000 mgr.y (mgr.14550) 357 : cluster [DBG] pgmap v569: 196 pgs: 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:09:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:41.516394+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:09:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:41.519286+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:09:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:41.948106+0000 osd.7 (osd.7) 173 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:09:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:41.951570+0000 osd.7 (osd.7) 174 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:09:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:42.260660+0000 mon.a (mon.0) 2469 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:42.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:42.260825+0000 mon.a (mon.0) 2470 : cluster [DBG] osdmap e399: 8 total, 8 up, 7 in 2024-02-19T21:09:42.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:42.266470+0000 mon.b (mon.1) 565 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:09:42.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:42.300413+0000 mon.b (mon.1) 566 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:42.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:42.300827+0000 mon.a (mon.0) 2471 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:42.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:42.301412+0000 mon.b (mon.1) 567 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-57"}]: dispatch 2024-02-19T21:09:42.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:42.301739+0000 mon.a (mon.0) 2472 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-57"}]: dispatch 2024-02-19T21:09:42.832 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:42.859 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:43.138 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-19T21:09:43.260 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:43.261+0000 7f001effd700 1 -- 172.21.15.163:0/3496772249 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v400) v1 ==== 113+0+0 (unknown 48458533 0 0) 0x7f0020047ee0 con 0x7f00281042c0 2024-02-19T21:09:43.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:43.263+0000 7f002fe27700 10 librados: watch_flush enter 2024-02-19T21:09:43.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:43.264+0000 7f002fe27700 10 librados: watch_flush exit 2024-02-19T21:09:43.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:43.264+0000 7f002fe27700 20 client.49769.objecter shutdown clearing up homeless session... 2024-02-19T21:09:43.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:43.264+0000 7f002fe27700 10 client.49769.objecter successfully canceled tick 2024-02-19T21:09:43.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:43.264+0000 7f002fe27700 1 -- 172.21.15.163:0/3496772249 >> v1:172.21.15.163:6800/1552514681 conn(0x7f0014063d50 legacy=0x7f0014066210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:43.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:43.264+0000 7f002fe27700 1 -- 172.21.15.163:0/3496772249 >> v1:172.21.15.163:6790/0 conn(0x7f00281042c0 legacy=0x7f0028117d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:43.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:43.264+0000 7f002fe27700 1 -- 172.21.15.163:0/3496772249 shutdown_connections 2024-02-19T21:09:43.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:43.264+0000 7f002fe27700 1 -- 172.21.15.163:0/3496772249 >> 172.21.15.163:0/3496772249 conn(0x7f00280f35a0 msgr2=0x7f0028107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:43.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:43.265+0000 7f002fe27700 1 -- 172.21.15.163:0/3496772249 shutdown_connections 2024-02-19T21:09:43.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:43.265+0000 7f002fe27700 1 -- 172.21.15.163:0/3496772249 wait complete. 2024-02-19T21:09:43.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:43.265+0000 7f002fe27700 1 librados: shutdown 2024-02-19T21:09:43.266 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.3 to 1 (10000) 2024-02-19T21:09:43.408 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:09:43.745 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:09:43] "GET /metrics HTTP/1.1" 200 189443 "" "Prometheus/2.33.4" 2024-02-19T21:09:43.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:41.953599+0000 osd.2 (osd.2) 144 : cluster [DBG] 4.d scrub starts 2024-02-19T21:09:43.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:41.955554+0000 osd.2 (osd.2) 145 : cluster [DBG] 4.d scrub ok 2024-02-19T21:09:43.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:41.997944+0000 osd.5 (osd.5) 162 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:09:43.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:42.000847+0000 osd.5 (osd.5) 163 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:09:43.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:42.630041+0000 mon.c (mon.2) 385 : audit [INF] from='client.? v1:172.21.15.163:0/3496772249' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:09:43.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:42.630452+0000 mon.a (mon.0) 2473 : audit [INF] from='client.49769 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:09:43.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:42.964287+0000 osd.7 (osd.7) 175 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:09:43.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:42.967767+0000 osd.7 (osd.7) 176 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:09:43.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:43.256715+0000 mon.a (mon.0) 2474 : audit [INF] from='client.49769 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-19T21:09:43.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:43.256780+0000 mon.a (mon.0) 2475 : cluster [DBG] osdmap e400: 8 total, 8 up, 7 in 2024-02-19T21:09:43.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:41.953599+0000 osd.2 (osd.2) 144 : cluster [DBG] 4.d scrub starts 2024-02-19T21:09:43.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:41.955554+0000 osd.2 (osd.2) 145 : cluster [DBG] 4.d scrub ok 2024-02-19T21:09:43.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:41.997944+0000 osd.5 (osd.5) 162 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:09:43.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:42.000847+0000 osd.5 (osd.5) 163 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:09:43.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:42.630041+0000 mon.c (mon.2) 385 : audit [INF] from='client.? v1:172.21.15.163:0/3496772249' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:09:43.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:42.630452+0000 mon.a (mon.0) 2473 : audit [INF] from='client.49769 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:09:43.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:42.964287+0000 osd.7 (osd.7) 175 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:09:43.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:42.967767+0000 osd.7 (osd.7) 176 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:09:43.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:43.256715+0000 mon.a (mon.0) 2474 : audit [INF] from='client.49769 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-19T21:09:43.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:43.256780+0000 mon.a (mon.0) 2475 : cluster [DBG] osdmap e400: 8 total, 8 up, 7 in 2024-02-19T21:09:43.808 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:43.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:43.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:41.953599+0000 osd.2 (osd.2) 144 : cluster [DBG] 4.d scrub starts 2024-02-19T21:09:43.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:41.955554+0000 osd.2 (osd.2) 145 : cluster [DBG] 4.d scrub ok 2024-02-19T21:09:43.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:41.997944+0000 osd.5 (osd.5) 162 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:09:43.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:42.000847+0000 osd.5 (osd.5) 163 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:09:43.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:42.630041+0000 mon.c (mon.2) 385 : audit [INF] from='client.? v1:172.21.15.163:0/3496772249' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:09:43.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:42.630452+0000 mon.a (mon.0) 2473 : audit [INF] from='client.49769 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:09:43.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:42.964287+0000 osd.7 (osd.7) 175 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:09:43.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:42.967767+0000 osd.7 (osd.7) 176 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:09:43.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:43.256715+0000 mon.a (mon.0) 2474 : audit [INF] from='client.49769 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-19T21:09:43.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:43.256780+0000 mon.a (mon.0) 2475 : cluster [DBG] osdmap e400: 8 total, 8 up, 7 in 2024-02-19T21:09:43.913 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:43.941 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:44.040 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:44.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:44.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:44.144 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:44.173 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:44.210 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:44.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:44.316 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:44.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.318+0000 7f280010d700 1 -- 172.21.15.163:0/3028734684 >> v1:172.21.15.163:6790/0 conn(0x7f27f80ff670 legacy=0x7f27f8101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:44.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.318+0000 7f280010d700 1 -- 172.21.15.163:0/3028734684 shutdown_connections 2024-02-19T21:09:44.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.318+0000 7f280010d700 1 -- 172.21.15.163:0/3028734684 >> 172.21.15.163:0/3028734684 conn(0x7f27f80f2470 msgr2=0x7f27f80f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:44.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.318+0000 7f280010d700 1 -- 172.21.15.163:0/3028734684 shutdown_connections 2024-02-19T21:09:44.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.319+0000 7f280010d700 1 -- 172.21.15.163:0/3028734684 wait complete. 2024-02-19T21:09:44.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.319+0000 7f280010d700 1 librados: starting msgr at 2024-02-19T21:09:44.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.319+0000 7f280010d700 1 librados: starting objecter 2024-02-19T21:09:44.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.319+0000 7f280010d700 1 Processor -- start 2024-02-19T21:09:44.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.319+0000 7f280010d700 1 -- start start 2024-02-19T21:09:44.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.319+0000 7f280010d700 1 librados: setting wanted keys 2024-02-19T21:09:44.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.319+0000 7f280010d700 1 librados: calling monclient init 2024-02-19T21:09:44.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.319+0000 7f280010d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f27f81194b0 con 0x7f27f80ff670 2024-02-19T21:09:44.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.319+0000 7f280010d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f27f811a690 con 0x7f27f81031b0 2024-02-19T21:09:44.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.319+0000 7f280010d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f27f811b870 con 0x7f27f80fc870 2024-02-19T21:09:44.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.319+0000 7f27fd6a8700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f27f80ff670 0x7f27f80610f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:35738/0 (socket says 172.21.15.163:35738) 2024-02-19T21:09:44.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.319+0000 7f27fd6a8700 1 -- 172.21.15.163:0/2557736618 learned_addr learned my addr 172.21.15.163:0/2557736618 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:44.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.319+0000 7f27fdea9700 1 --1- 172.21.15.163:0/2557736618 >> v1:172.21.15.163:6790/0 conn(0x7f27f80fc870 0x7f27f80609e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:33212/0 (socket says 172.21.15.163:33212) 2024-02-19T21:09:44.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 10 client.?.objecter ms_handle_connect 0x7f27f80ff670 2024-02-19T21:09:44.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:44.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 10 client.?.objecter ms_handle_connect 0x7f27f80fc870 2024-02-19T21:09:44.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:44.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 10 client.?.objecter ms_handle_connect 0x7f27f81031b0 2024-02-19T21:09:44.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:44.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361733682 0 0) 0x7f27f81194b0 con 0x7f27f80ff670 2024-02-19T21:09:44.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e4003800 con 0x7f27f80ff670 2024-02-19T21:09:44.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908953603 0 0) 0x7f27f811b870 con 0x7f27f80fc870 2024-02-19T21:09:44.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27f81194b0 con 0x7f27f80fc870 2024-02-19T21:09:44.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150306567 0 0) 0x7f27f811a690 con 0x7f27f81031b0 2024-02-19T21:09:44.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27f811b870 con 0x7f27f81031b0 2024-02-19T21:09:44.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3274052025 0 0) 0x7f27f81194b0 con 0x7f27f80fc870 2024-02-19T21:09:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27f811a690 con 0x7f27f80fc870 2024-02-19T21:09:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f27f40018c0 con 0x7f27f80fc870 2024-02-19T21:09:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2009216879 0 0) 0x7f27e4003800 con 0x7f27f80ff670 2024-02-19T21:09:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27f81194b0 con 0x7f27f80ff670 2024-02-19T21:09:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4081319191 0 0) 0x7f27f811b870 con 0x7f27f81031b0 2024-02-19T21:09:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27e4003800 con 0x7f27f81031b0 2024-02-19T21:09:44.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f27ec002f20 con 0x7f27f80ff670 2024-02-19T21:09:44.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f27f00018d0 con 0x7f27f81031b0 2024-02-19T21:09:44.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4284130409 0 0) 0x7f27f811a690 con 0x7f27f80fc870 2024-02-19T21:09:44.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 >> v1:172.21.15.193:6789/0 conn(0x7f27f81031b0 legacy=0x7f27f8117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:44.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 >> v1:172.21.15.163:6789/0 conn(0x7f27f80ff670 legacy=0x7f27f80610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:44.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f811ca70 con 0x7f27f80fc870 2024-02-19T21:09:44.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f280010d700 1 -- 172.21.15.163:0/2557736618 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f811ba60 con 0x7f27f80fc870 2024-02-19T21:09:44.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f280010d700 10 client.49775.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:44.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f280010d700 1 -- 172.21.15.163:0/2557736618 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f811bf50 con 0x7f27f80fc870 2024-02-19T21:09:44.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f280010d700 1 librados: init done 2024-02-19T21:09:44.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f27f4002050 con 0x7f27f80fc870 2024-02-19T21:09:44.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.320+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f27f4005a70 con 0x7f27f80fc870 2024-02-19T21:09:44.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.321+0000 7f280010d700 1 -- 172.21.15.163:0/2557736618 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27dc004f20 con 0x7f27f80fc870 2024-02-19T21:09:44.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.322+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f27f4001bb0 con 0x7f27f80fc870 2024-02-19T21:09:44.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.322+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(401..401 src has 43..401) v4 ==== 7889+0+0 (unknown 2204188480 0 0) 0x7f27f4078be0 con 0x7f27f80fc870 2024-02-19T21:09:44.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.322+0000 7f27eaffd700 10 client.49775.objecter ms_dispatch 0x7f27f8001550 osd_map(401..401 src has 43..401) v4 2024-02-19T21:09:44.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.322+0000 7f27eaffd700 3 client.49775.objecter handle_osd_map got epochs [401,401] > 0 2024-02-19T21:09:44.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.322+0000 7f27eaffd700 3 client.49775.objecter handle_osd_map decoding full epoch 401 2024-02-19T21:09:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.322+0000 7f27eaffd700 20 client.49775.objecter dump_active .. 0 homeless 2024-02-19T21:09:44.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.322+0000 7f27eaffd700 10 client.49775.objecter ms_handle_connect 0x7f27e4063cf0 2024-02-19T21:09:44.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.326+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f27f404c0e0 con 0x7f27f80fc870 2024-02-19T21:09:44.350 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:44.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:44.502+0000 7f280010d700 1 -- 172.21.15.163:0/2557736618 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f27dc0051d0 con 0x7f27f80fc870 2024-02-19T21:09:44.500 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:44.675 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:43.204027+0000 mgr.y (mgr.14550) 358 : cluster [DBG] pgmap v572: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:09:44.676 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:43.981177+0000 osd.7 (osd.7) 177 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:09:44.676 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:43.984075+0000 osd.7 (osd.7) 178 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:09:44.676 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:44.259717+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e401: 8 total, 8 up, 7 in 2024-02-19T21:09:44.676 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:44.260623+0000 mon.b (mon.1) 568 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:44.676 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:44.260983+0000 mon.a (mon.0) 2477 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:44.676 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:44.503574+0000 mon.c (mon.2) 386 : audit [INF] from='client.? v1:172.21.15.163:0/2557736618' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:09:44.676 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:44.503899+0000 mon.a (mon.0) 2478 : audit [INF] from='client.49775 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:09:44.697 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:44.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:44.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:44.802 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:44.829 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:44.953 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:44.957 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:44.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:44.976 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:43.204027+0000 mgr.y (mgr.14550) 358 : cluster [DBG] pgmap v572: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:09:44.977 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:43.981177+0000 osd.7 (osd.7) 177 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:09:44.977 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:43.984075+0000 osd.7 (osd.7) 178 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:09:44.977 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:44.259717+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e401: 8 total, 8 up, 7 in 2024-02-19T21:09:44.977 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:44.260623+0000 mon.b (mon.1) 568 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:44.977 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:44.260983+0000 mon.a (mon.0) 2477 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:44.977 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:44.503574+0000 mon.c (mon.2) 386 : audit [INF] from='client.? v1:172.21.15.163:0/2557736618' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:09:44.977 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:44.503899+0000 mon.a (mon.0) 2478 : audit [INF] from='client.49775 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:09:44.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:43.204027+0000 mgr.y (mgr.14550) 358 : cluster [DBG] pgmap v572: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:09:44.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:43.981177+0000 osd.7 (osd.7) 177 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:09:44.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:43.984075+0000 osd.7 (osd.7) 178 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:09:44.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:44.259717+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e401: 8 total, 8 up, 7 in 2024-02-19T21:09:44.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:44.260623+0000 mon.b (mon.1) 568 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:44.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:44.260983+0000 mon.a (mon.0) 2477 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:44.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:44.503574+0000 mon.c (mon.2) 386 : audit [INF] from='client.? v1:172.21.15.163:0/2557736618' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:09:44.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:44.503899+0000 mon.a (mon.0) 2478 : audit [INF] from='client.49775 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:09:44.995 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-19T21:09:44.996 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:09:45.059 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:45.088 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:45.210 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:45.212 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:45.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:45.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:45.269+0000 7f27eaffd700 1 -- 172.21.15.163:0/2557736618 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v402) v1 ==== 87+0+0 (unknown 241872302 0 0) 0x7f27f404a4f0 con 0x7f27f80fc870 2024-02-19T21:09:45.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:45.272+0000 7f280010d700 10 librados: watch_flush enter 2024-02-19T21:09:45.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:45.273+0000 7f280010d700 10 librados: watch_flush exit 2024-02-19T21:09:45.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:45.273+0000 7f280010d700 20 client.49775.objecter shutdown clearing up homeless session... 2024-02-19T21:09:45.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:45.273+0000 7f280010d700 10 client.49775.objecter successfully canceled tick 2024-02-19T21:09:45.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:45.273+0000 7f280010d700 1 -- 172.21.15.163:0/2557736618 >> v1:172.21.15.163:6800/1552514681 conn(0x7f27e4063cf0 legacy=0x7f27e40661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:45.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:45.273+0000 7f280010d700 1 -- 172.21.15.163:0/2557736618 >> v1:172.21.15.163:6790/0 conn(0x7f27f80fc870 legacy=0x7f27f80609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:45.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:45.275+0000 7f280010d700 1 -- 172.21.15.163:0/2557736618 shutdown_connections 2024-02-19T21:09:45.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:45.275+0000 7f280010d700 1 -- 172.21.15.163:0/2557736618 >> 172.21.15.163:0/2557736618 conn(0x7f27f80f2470 msgr2=0x7f27f8106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:45.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:45.283+0000 7f280010d700 1 -- 172.21.15.163:0/2557736618 shutdown_connections 2024-02-19T21:09:45.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:45.283+0000 7f280010d700 1 -- 172.21.15.163:0/2557736618 wait complete. 2024-02-19T21:09:45.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:45.283+0000 7f280010d700 1 librados: shutdown 2024-02-19T21:09:45.282 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:09:45.313 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:45.333 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:09:45.334 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:09:45.341 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:45.360 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:45.363 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:45.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:45.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:45.492 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:45.613 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:43.935200+0000 osd.2 (osd.2) 146 : cluster [DBG] 6.17 deep-scrub starts 2024-02-19T21:09:45.613 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:43.936149+0000 osd.2 (osd.2) 147 : cluster [DBG] 6.17 deep-scrub ok 2024-02-19T21:09:45.613 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:44.594849+0000 osd.3 (osd.3) 171 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:09:45.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:44.596322+0000 osd.3 (osd.3) 172 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:09:45.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:44.921340+0000 osd.2 (osd.2) 148 : cluster [DBG] 2.e scrub starts 2024-02-19T21:09:45.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:44.922299+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.e scrub ok 2024-02-19T21:09:45.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:45.255602+0000 mon.a (mon.0) 2479 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:09:45.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:45.265119+0000 mon.a (mon.0) 2480 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:45.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:45.265360+0000 mon.a (mon.0) 2481 : audit [INF] from='client.49775 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:09:45.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:45.265411+0000 mon.a (mon.0) 2482 : cluster [DBG] osdmap e402: 8 total, 8 up, 7 in 2024-02-19T21:09:45.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:45.273221+0000 mon.b (mon.1) 569 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:09:45.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:45.276055+0000 mon.b (mon.1) 570 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:09:45.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:45.278468+0000 mon.a (mon.0) 2483 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:09:45.614 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:43.935200+0000 osd.2 (osd.2) 146 : cluster [DBG] 6.17 deep-scrub starts 2024-02-19T21:09:45.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:43.936149+0000 osd.2 (osd.2) 147 : cluster [DBG] 6.17 deep-scrub ok 2024-02-19T21:09:45.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:44.594849+0000 osd.3 (osd.3) 171 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:09:45.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:44.596322+0000 osd.3 (osd.3) 172 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:09:45.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:44.921340+0000 osd.2 (osd.2) 148 : cluster [DBG] 2.e scrub starts 2024-02-19T21:09:45.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:44.922299+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.e scrub ok 2024-02-19T21:09:45.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:45.255602+0000 mon.a (mon.0) 2479 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:09:45.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:45.265119+0000 mon.a (mon.0) 2480 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:45.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:45.265360+0000 mon.a (mon.0) 2481 : audit [INF] from='client.49775 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:09:45.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:45.265411+0000 mon.a (mon.0) 2482 : cluster [DBG] osdmap e402: 8 total, 8 up, 7 in 2024-02-19T21:09:45.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:45.273221+0000 mon.b (mon.1) 569 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:09:45.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:45.276055+0000 mon.b (mon.1) 570 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:09:45.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:45.278468+0000 mon.a (mon.0) 2483 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:09:45.640 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:45.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:45.743 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:09:45.744 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:43.935200+0000 osd.2 (osd.2) 146 : cluster [DBG] 6.17 deep-scrub starts 2024-02-19T21:09:45.744 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:43.936149+0000 osd.2 (osd.2) 147 : cluster [DBG] 6.17 deep-scrub ok 2024-02-19T21:09:45.744 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:44.594849+0000 osd.3 (osd.3) 171 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:09:45.744 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:44.596322+0000 osd.3 (osd.3) 172 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:09:45.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:44.921340+0000 osd.2 (osd.2) 148 : cluster [DBG] 2.e scrub starts 2024-02-19T21:09:45.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:44.922299+0000 osd.2 (osd.2) 149 : cluster [DBG] 2.e scrub ok 2024-02-19T21:09:45.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:45.255602+0000 mon.a (mon.0) 2479 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:09:45.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:45.265119+0000 mon.a (mon.0) 2480 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:45.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:45.265360+0000 mon.a (mon.0) 2481 : audit [INF] from='client.49775 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:09:45.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:45.265411+0000 mon.a (mon.0) 2482 : cluster [DBG] osdmap e402: 8 total, 8 up, 7 in 2024-02-19T21:09:45.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:45.273221+0000 mon.b (mon.1) 569 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:09:45.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:45.276055+0000 mon.b (mon.1) 570 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:09:45.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:45.278468+0000 mon.a (mon.0) 2483 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:09:45.773 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:45.922 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:09:45.924 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:45.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:09:46.026 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:46.052 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:46.072 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:46.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:46.087 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-19T21:09:46.176 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:46.203 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:46.222 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:46.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:46.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:46.326 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:46.353 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:46.375 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:46.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:46.411 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:09:46.480 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:46.507 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:46.526 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:46.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:46.633 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:46.640 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:45.204876+0000 mgr.y (mgr.14550) 359 : cluster [DBG] pgmap v575: 228 pgs: 32 creating+peering, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-19T21:09:46.640 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:46.263419+0000 mon.a (mon.0) 2484 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:09:46.640 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:46.263495+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e403: 8 total, 8 up, 7 in 2024-02-19T21:09:46.640 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:46.290829+0000 mon.b (mon.1) 571 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:46.640 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:46.291166+0000 mon.a (mon.0) 2486 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:46.640 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:46.291635+0000 mon.b (mon.1) 572 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-59"}]: dispatch 2024-02-19T21:09:46.640 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:46.291796+0000 mon.a (mon.0) 2487 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-59"}]: dispatch 2024-02-19T21:09:46.660 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:46.679 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:46.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:46.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:46.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:45.204876+0000 mgr.y (mgr.14550) 359 : cluster [DBG] pgmap v575: 228 pgs: 32 creating+peering, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-19T21:09:46.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:46.263419+0000 mon.a (mon.0) 2484 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:09:46.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:46.263495+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e403: 8 total, 8 up, 7 in 2024-02-19T21:09:46.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:46.290829+0000 mon.b (mon.1) 571 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:46.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:46.291166+0000 mon.a (mon.0) 2486 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:46.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:46.291635+0000 mon.b (mon.1) 572 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-59"}]: dispatch 2024-02-19T21:09:46.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:46.291796+0000 mon.a (mon.0) 2487 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-59"}]: dispatch 2024-02-19T21:09:46.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:45.204876+0000 mgr.y (mgr.14550) 359 : cluster [DBG] pgmap v575: 228 pgs: 32 creating+peering, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-19T21:09:46.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:46.263419+0000 mon.a (mon.0) 2484 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:09:46.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:46.263495+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e403: 8 total, 8 up, 7 in 2024-02-19T21:09:46.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:46.290829+0000 mon.b (mon.1) 571 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:46.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:46.291166+0000 mon.a (mon.0) 2486 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:09:46.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:46.291635+0000 mon.b (mon.1) 572 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-59"}]: dispatch 2024-02-19T21:09:46.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:46.291796+0000 mon.a (mon.0) 2487 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-59"}]: dispatch 2024-02-19T21:09:46.783 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:09:46.810 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:46.837 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:09:46.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:46.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:09:46.866 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:46.942 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:46.971 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:47.168 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:47.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:47.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:47.272 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:47.302 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:47.906 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:09:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:09:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:09:48.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.567+0000 7f6e5450c700 1 -- 172.21.15.163:0/3624699453 >> v1:172.21.15.163:6790/0 conn(0x7f6e4c1031b0 legacy=0x7f6e4c105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:48.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.567+0000 7f6e5450c700 1 -- 172.21.15.163:0/3624699453 shutdown_connections 2024-02-19T21:09:48.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.567+0000 7f6e5450c700 1 -- 172.21.15.163:0/3624699453 >> 172.21.15.163:0/3624699453 conn(0x7f6e4c0f2470 msgr2=0x7f6e4c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:48.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.567+0000 7f6e5450c700 1 -- 172.21.15.163:0/3624699453 shutdown_connections 2024-02-19T21:09:48.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.567+0000 7f6e5450c700 1 -- 172.21.15.163:0/3624699453 wait complete. 2024-02-19T21:09:48.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e5450c700 1 librados: starting msgr at 2024-02-19T21:09:48.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e5450c700 1 librados: starting objecter 2024-02-19T21:09:48.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e5450c700 1 Processor -- start 2024-02-19T21:09:48.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e5450c700 1 -- start start 2024-02-19T21:09:48.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e5450c700 1 librados: setting wanted keys 2024-02-19T21:09:48.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e5450c700 1 librados: calling monclient init 2024-02-19T21:09:48.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e5450c700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6e4c117670 con 0x7f6e4c1031b0 2024-02-19T21:09:48.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e5450c700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6e4c118810 con 0x7f6e4c0fc870 2024-02-19T21:09:48.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e5450c700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6e4c1199b0 con 0x7f6e4c0ff670 2024-02-19T21:09:48.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e51aa7700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f6e4c0ff670 0x7f6e4c112820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:33232/0 (socket says 172.21.15.163:33232) 2024-02-19T21:09:48.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e51aa7700 1 -- 172.21.15.163:0/2049860212 learned_addr learned my addr 172.21.15.163:0/2049860212 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:48.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e3f7fe700 10 client.?.objecter ms_handle_connect 0x7f6e4c1031b0 2024-02-19T21:09:48.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e3f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:48.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e3f7fe700 10 client.?.objecter ms_handle_connect 0x7f6e4c0ff670 2024-02-19T21:09:48.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e3f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:48.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e3f7fe700 10 client.?.objecter ms_handle_connect 0x7f6e4c0fc870 2024-02-19T21:09:48.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e3f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:48.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721268000 0 0) 0x7f6e4c1199b0 con 0x7f6e4c0ff670 2024-02-19T21:09:48.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e38003030 con 0x7f6e4c0ff670 2024-02-19T21:09:48.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881252535 0 0) 0x7f6e4c117670 con 0x7f6e4c1031b0 2024-02-19T21:09:48.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e4c1199b0 con 0x7f6e4c1031b0 2024-02-19T21:09:48.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425741990 0 0) 0x7f6e4c118810 con 0x7f6e4c0fc870 2024-02-19T21:09:48.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.568+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e4c117670 con 0x7f6e4c0fc870 2024-02-19T21:09:48.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1818305394 0 0) 0x7f6e38003030 con 0x7f6e4c0ff670 2024-02-19T21:09:48.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e4c118810 con 0x7f6e4c0ff670 2024-02-19T21:09:48.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6e480018c0 con 0x7f6e4c0ff670 2024-02-19T21:09:48.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4251542023 0 0) 0x7f6e4c117670 con 0x7f6e4c0fc870 2024-02-19T21:09:48.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e38003030 con 0x7f6e4c0fc870 2024-02-19T21:09:48.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6e40001880 con 0x7f6e4c0fc870 2024-02-19T21:09:48.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2834035761 0 0) 0x7f6e4c1199b0 con 0x7f6e4c1031b0 2024-02-19T21:09:48.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e4c117670 con 0x7f6e4c1031b0 2024-02-19T21:09:48.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6e44002240 con 0x7f6e4c1031b0 2024-02-19T21:09:48.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2347932982 0 0) 0x7f6e4c118810 con 0x7f6e4c0ff670 2024-02-19T21:09:48.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 >> v1:172.21.15.193:6789/0 conn(0x7f6e4c0fc870 legacy=0x7f6e4c0fb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:48.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 >> v1:172.21.15.163:6789/0 conn(0x7f6e4c1031b0 legacy=0x7f6e4c115f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:48.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c11ab50 con 0x7f6e4c0ff670 2024-02-19T21:09:48.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4c119ba0 con 0x7f6e4c0ff670 2024-02-19T21:09:48.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e5450c700 10 client.49781.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:48.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4c11a010 con 0x7f6e4c0ff670 2024-02-19T21:09:48.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e5450c700 1 librados: init done 2024-02-19T21:09:48.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6e48001ec0 con 0x7f6e4c0ff670 2024-02-19T21:09:48.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.569+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6e48005a70 con 0x7f6e4c0ff670 2024-02-19T21:09:48.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.570+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6e30004f00 con 0x7f6e4c0ff670 2024-02-19T21:09:48.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.570+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6e48001a20 con 0x7f6e4c0ff670 2024-02-19T21:09:48.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(404..404 src has 43..404) v4 ==== 7533+0+0 (unknown 135897038 0 0) 0x7f6e48078960 con 0x7f6e4c0ff670 2024-02-19T21:09:48.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 osd_map(404..404 src has 43..404) v4 2024-02-19T21:09:48.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e3f7fe700 3 client.49781.objecter handle_osd_map got epochs [404,404] > 0 2024-02-19T21:09:48.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e3f7fe700 3 client.49781.objecter handle_osd_map decoding full epoch 404 2024-02-19T21:09:48.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e3f7fe700 20 client.49781.objecter dump_active .. 0 homeless 2024-02-19T21:09:48.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v404) v1 ==== 54+0+16 (unknown 1360781877 0 2420000192) 0x7f6e48078d20 con 0x7f6e4c0ff670 2024-02-19T21:09:48.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e3f7fe700 10 client.49781.objecter ms_handle_connect 0x7f6e38063cb0 2024-02-19T21:09:48.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e5450c700 10 client.49781.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:48.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 1 2024-02-19T21:09:48.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e300051a0 osd=0 3 2024-02-19T21:09:48.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e300051a0 osd=0 3 2024-02-19T21:09:48.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e300051a0 osd=0 3 2024-02-19T21:09:48.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 1 2024-02-19T21:09:48.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e300051a0 osd=0 3 2024-02-19T21:09:48.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 0 1 2024-02-19T21:09:48.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e300051a0 osd=0 4 2024-02-19T21:09:48.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e5450c700 10 client.49781.objecter _send_command 1 2024-02-19T21:09:48.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e30004950 con 0x7f6e30004530 2024-02-19T21:09:48.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.571+0000 7f6e3f7fe700 10 client.49781.objecter ms_handle_connect 0x7f6e30004530 2024-02-19T21:09:48.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.572+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e30004950 con 0x7f6e30004530 2024-02-19T21:09:48.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.572+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:09:48.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.572+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 1 = system:0 2024-02-19T21:09:48.010 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:09:48.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.572+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e300051a0 osd=0 4 2024-02-19T21:09:48.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.572+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 0 1 2024-02-19T21:09:48.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.587+0000 7f6e5450c700 10 client.49781.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:48.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.587+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 2 2024-02-19T21:09:48.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.587+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e300051a0 osd=0 3 2024-02-19T21:09:48.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.587+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e300051a0 osd=0 3 2024-02-19T21:09:48.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.587+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e300051a0 osd=0 3 2024-02-19T21:09:48.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.587+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 2 2024-02-19T21:09:48.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.587+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e300051a0 osd=0 3 2024-02-19T21:09:48.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.587+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 0 2 2024-02-19T21:09:48.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.587+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e300051a0 osd=0 4 2024-02-19T21:09:48.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.587+0000 7f6e5450c700 10 client.49781.objecter _send_command 2 2024-02-19T21:09:48.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.587+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6e30004d30 con 0x7f6e30004530 2024-02-19T21:09:48.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.588+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6e30004d30 con 0x7f6e30004530 2024-02-19T21:09:48.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.588+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:48.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.588+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.588+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e300051a0 osd=0 4 2024-02-19T21:09:48.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.588+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 0 2 2024-02-19T21:09:48.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e5450c700 10 client.49781.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:48.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 3 2024-02-19T21:09:48.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30003fe0 osd=1 3 2024-02-19T21:09:48.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30003fe0 osd=1 3 2024-02-19T21:09:48.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30003fe0 osd=1 3 2024-02-19T21:09:48.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 3 2024-02-19T21:09:48.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e30003fe0 osd=1 3 2024-02-19T21:09:48.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 1 3 2024-02-19T21:09:48.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30003fe0 osd=1 4 2024-02-19T21:09:48.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e5450c700 10 client.49781.objecter _send_command 3 2024-02-19T21:09:48.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e3000c000 con 0x7f6e30004150 2024-02-19T21:09:48.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e3f7fe700 10 client.49781.objecter ms_handle_connect 0x7f6e30004150 2024-02-19T21:09:48.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e3000c000 con 0x7f6e30004150 2024-02-19T21:09:48.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:09:48.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 3 = system:0 2024-02-19T21:09:48.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e30003fe0 osd=1 4 2024-02-19T21:09:48.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.589+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 1 3 2024-02-19T21:09:48.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.604+0000 7f6e5450c700 10 client.49781.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:48.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.604+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 4 2024-02-19T21:09:48.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.604+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30003fe0 osd=1 3 2024-02-19T21:09:48.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.604+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30003fe0 osd=1 3 2024-02-19T21:09:48.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.604+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30003fe0 osd=1 3 2024-02-19T21:09:48.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.604+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 4 2024-02-19T21:09:48.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.604+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e30003fe0 osd=1 3 2024-02-19T21:09:48.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.604+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 1 4 2024-02-19T21:09:48.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.604+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30003fe0 osd=1 4 2024-02-19T21:09:48.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.604+0000 7f6e5450c700 10 client.49781.objecter _send_command 4 2024-02-19T21:09:48.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.604+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6e300018b0 con 0x7f6e30004150 2024-02-19T21:09:48.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6e300018b0 con 0x7f6e30004150 2024-02-19T21:09:48.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:48.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e30003fe0 osd=1 4 2024-02-19T21:09:48.015 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:09:48.015 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:09:48.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 1 4 2024-02-19T21:09:48.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e5450c700 10 client.49781.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:48.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 5 2024-02-19T21:09:48.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30003630 osd=2 3 2024-02-19T21:09:48.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30003630 osd=2 3 2024-02-19T21:09:48.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30003630 osd=2 3 2024-02-19T21:09:48.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 5 2024-02-19T21:09:48.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e30003630 osd=2 3 2024-02-19T21:09:48.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 2 5 2024-02-19T21:09:48.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30003630 osd=2 4 2024-02-19T21:09:48.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e5450c700 10 client.49781.objecter _send_command 5 2024-02-19T21:09:48.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e3000de90 con 0x7f6e3000d2f0 2024-02-19T21:09:48.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.606+0000 7f6e3f7fe700 10 client.49781.objecter ms_handle_connect 0x7f6e3000d2f0 2024-02-19T21:09:48.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.607+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e3000de90 con 0x7f6e3000d2f0 2024-02-19T21:09:48.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.607+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:09:48.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.607+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 5 = system:0 2024-02-19T21:09:48.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.607+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e30003630 osd=2 4 2024-02-19T21:09:48.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.607+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 2 5 2024-02-19T21:09:48.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.621+0000 7f6e5450c700 10 client.49781.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:48.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.621+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 6 2024-02-19T21:09:48.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.621+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30003630 osd=2 3 2024-02-19T21:09:48.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.621+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30003630 osd=2 3 2024-02-19T21:09:48.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.621+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30003630 osd=2 3 2024-02-19T21:09:48.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.621+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 6 2024-02-19T21:09:48.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.621+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e30003630 osd=2 3 2024-02-19T21:09:48.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.621+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 2 6 2024-02-19T21:09:48.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.621+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30003630 osd=2 4 2024-02-19T21:09:48.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.621+0000 7f6e5450c700 10 client.49781.objecter _send_command 6 2024-02-19T21:09:48.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.621+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6e30003160 con 0x7f6e3000d2f0 2024-02-19T21:09:48.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6e30003160 con 0x7f6e3000d2f0 2024-02-19T21:09:48.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:48.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e30003630 osd=2 4 2024-02-19T21:09:48.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 2 6 2024-02-19T21:09:48.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e5450c700 10 client.49781.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:48.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 7 2024-02-19T21:09:48.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30011970 osd=3 3 2024-02-19T21:09:48.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30011970 osd=3 3 2024-02-19T21:09:48.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30011970 osd=3 3 2024-02-19T21:09:48.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 7 2024-02-19T21:09:48.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e30011970 osd=3 3 2024-02-19T21:09:48.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 3 7 2024-02-19T21:09:48.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30011970 osd=3 4 2024-02-19T21:09:48.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e5450c700 10 client.49781.objecter _send_command 7 2024-02-19T21:09:48.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.622+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e30015ba0 con 0x7f6e30011ff0 2024-02-19T21:09:48.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.623+0000 7f6e3f7fe700 10 client.49781.objecter ms_handle_connect 0x7f6e30011ff0 2024-02-19T21:09:48.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.623+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e30015ba0 con 0x7f6e30011ff0 2024-02-19T21:09:48.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.623+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:09:48.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.623+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 7 = system:0 2024-02-19T21:09:48.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.623+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e30011970 osd=3 4 2024-02-19T21:09:48.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.623+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 3 7 2024-02-19T21:09:48.019 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:09:48.019 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:09:48.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.638+0000 7f6e5450c700 10 client.49781.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:48.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.638+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 8 2024-02-19T21:09:48.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.638+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30011970 osd=3 3 2024-02-19T21:09:48.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.638+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30011970 osd=3 3 2024-02-19T21:09:48.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.638+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30011970 osd=3 3 2024-02-19T21:09:48.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.638+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 8 2024-02-19T21:09:48.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.638+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e30011970 osd=3 3 2024-02-19T21:09:48.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.638+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 3 8 2024-02-19T21:09:48.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.638+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30011970 osd=3 4 2024-02-19T21:09:48.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.638+0000 7f6e5450c700 10 client.49781.objecter _send_command 8 2024-02-19T21:09:48.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.638+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6e30015ba0 con 0x7f6e30011ff0 2024-02-19T21:09:48.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.640+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6e30015ba0 con 0x7f6e30011ff0 2024-02-19T21:09:48.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.640+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:48.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.640+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.640+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e30011970 osd=3 4 2024-02-19T21:09:48.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.640+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 3 8 2024-02-19T21:09:48.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.641+0000 7f6e5450c700 10 client.49781.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:48.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.641+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 9 2024-02-19T21:09:48.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.641+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30015ba0 osd=4 3 2024-02-19T21:09:48.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.641+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30015ba0 osd=4 3 2024-02-19T21:09:48.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.641+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30015ba0 osd=4 3 2024-02-19T21:09:48.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.641+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 9 2024-02-19T21:09:48.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.641+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e30015ba0 osd=4 3 2024-02-19T21:09:48.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.641+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 4 9 2024-02-19T21:09:48.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.641+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30015ba0 osd=4 4 2024-02-19T21:09:48.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.641+0000 7f6e5450c700 10 client.49781.objecter _send_command 9 2024-02-19T21:09:48.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.641+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e3000ced0 con 0x7f6e3000c940 2024-02-19T21:09:48.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.641+0000 7f6e3f7fe700 10 client.49781.objecter ms_handle_connect 0x7f6e3000c940 2024-02-19T21:09:48.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.642+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e3000ced0 con 0x7f6e3000c940 2024-02-19T21:09:48.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.642+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:09:48.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.642+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 9 = system:0 2024-02-19T21:09:48.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.642+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e30015ba0 osd=4 4 2024-02-19T21:09:48.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.642+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 4 9 2024-02-19T21:09:48.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.657+0000 7f6e5450c700 10 client.49781.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:48.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.657+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 10 2024-02-19T21:09:48.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.657+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30015ba0 osd=4 3 2024-02-19T21:09:48.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.657+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30015ba0 osd=4 3 2024-02-19T21:09:48.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.657+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30015ba0 osd=4 3 2024-02-19T21:09:48.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.657+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 10 2024-02-19T21:09:48.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.657+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e30015ba0 osd=4 3 2024-02-19T21:09:48.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.657+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 4 10 2024-02-19T21:09:48.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.657+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30015ba0 osd=4 4 2024-02-19T21:09:48.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.657+0000 7f6e5450c700 10 client.49781.objecter _send_command 10 2024-02-19T21:09:48.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.657+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6e3000ced0 con 0x7f6e3000c940 2024-02-19T21:09:48.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6e3000ced0 con 0x7f6e3000c940 2024-02-19T21:09:48.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:48.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e30015ba0 osd=4 4 2024-02-19T21:09:48.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 4 10 2024-02-19T21:09:48.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e5450c700 10 client.49781.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:48.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 11 2024-02-19T21:09:48.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e3000ced0 osd=5 3 2024-02-19T21:09:48.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e3000ced0 osd=5 3 2024-02-19T21:09:48.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e3000ced0 osd=5 3 2024-02-19T21:09:48.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 11 2024-02-19T21:09:48.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e3000ced0 osd=5 3 2024-02-19T21:09:48.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 5 11 2024-02-19T21:09:48.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e3000ced0 osd=5 4 2024-02-19T21:09:48.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e5450c700 10 client.49781.objecter _send_command 11 2024-02-19T21:09:48.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.659+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e30016da0 con 0x7f6e30016960 2024-02-19T21:09:48.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.660+0000 7f6e3f7fe700 10 client.49781.objecter ms_handle_connect 0x7f6e30016960 2024-02-19T21:09:48.024 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:09:48.024 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:09:48.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.661+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e30016da0 con 0x7f6e30016960 2024-02-19T21:09:48.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.661+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:09:48.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.661+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 11 = system:0 2024-02-19T21:09:48.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.661+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e3000ced0 osd=5 4 2024-02-19T21:09:48.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.661+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 5 11 2024-02-19T21:09:48.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.675+0000 7f6e5450c700 10 client.49781.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:48.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.675+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 12 2024-02-19T21:09:48.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.675+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e3000ced0 osd=5 3 2024-02-19T21:09:48.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.675+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e3000ced0 osd=5 3 2024-02-19T21:09:48.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.675+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e3000ced0 osd=5 3 2024-02-19T21:09:48.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.675+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 12 2024-02-19T21:09:48.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.675+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e3000ced0 osd=5 3 2024-02-19T21:09:48.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.675+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 5 12 2024-02-19T21:09:48.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.675+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e3000ced0 osd=5 4 2024-02-19T21:09:48.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.675+0000 7f6e5450c700 10 client.49781.objecter _send_command 12 2024-02-19T21:09:48.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.675+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6e30016da0 con 0x7f6e30016960 2024-02-19T21:09:48.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6e30016da0 con 0x7f6e30016960 2024-02-19T21:09:48.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:48.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e3000ced0 osd=5 4 2024-02-19T21:09:48.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 5 12 2024-02-19T21:09:48.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e5450c700 10 client.49781.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:48.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 13 2024-02-19T21:09:48.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30016da0 osd=6 3 2024-02-19T21:09:48.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30016da0 osd=6 3 2024-02-19T21:09:48.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30016da0 osd=6 3 2024-02-19T21:09:48.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 13 2024-02-19T21:09:48.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e30016da0 osd=6 3 2024-02-19T21:09:48.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 6 13 2024-02-19T21:09:48.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30016da0 osd=6 4 2024-02-19T21:09:48.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e5450c700 10 client.49781.objecter _send_command 13 2024-02-19T21:09:48.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e30022cb0 con 0x7f6e3001f1a0 2024-02-19T21:09:48.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.677+0000 7f6e3f7fe700 10 client.49781.objecter ms_handle_connect 0x7f6e3001f1a0 2024-02-19T21:09:48.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.678+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e30022cb0 con 0x7f6e3001f1a0 2024-02-19T21:09:48.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.678+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:09:48.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.678+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 13 = system:0 2024-02-19T21:09:48.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.678+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e30016da0 osd=6 4 2024-02-19T21:09:48.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.678+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 6 13 2024-02-19T21:09:48.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.693+0000 7f6e5450c700 10 client.49781.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:48.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.693+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 14 2024-02-19T21:09:48.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.693+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30016da0 osd=6 3 2024-02-19T21:09:48.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.693+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30016da0 osd=6 3 2024-02-19T21:09:48.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.693+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30016da0 osd=6 3 2024-02-19T21:09:48.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.693+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 14 2024-02-19T21:09:48.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.693+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e30016da0 osd=6 3 2024-02-19T21:09:48.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.693+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 6 14 2024-02-19T21:09:48.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.693+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30016da0 osd=6 4 2024-02-19T21:09:48.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.693+0000 7f6e5450c700 10 client.49781.objecter _send_command 14 2024-02-19T21:09:48.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.693+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6e30022cb0 con 0x7f6e3001f1a0 2024-02-19T21:09:48.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6e30022cb0 con 0x7f6e3001f1a0 2024-02-19T21:09:48.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:48.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e30016da0 osd=6 4 2024-02-19T21:09:48.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 6 14 2024-02-19T21:09:48.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e5450c700 10 client.49781.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:48.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 15 2024-02-19T21:09:48.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30022cb0 osd=7 3 2024-02-19T21:09:48.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30022cb0 osd=7 3 2024-02-19T21:09:48.029 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:09:48.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30022cb0 osd=7 3 2024-02-19T21:09:48.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 15 2024-02-19T21:09:48.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e30022cb0 osd=7 3 2024-02-19T21:09:48.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 7 15 2024-02-19T21:09:48.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30022cb0 osd=7 4 2024-02-19T21:09:48.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e5450c700 10 client.49781.objecter _send_command 15 2024-02-19T21:09:48.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.695+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e30026fa0 con 0x7f6e30023330 2024-02-19T21:09:48.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.696+0000 7f6e3f7fe700 10 client.49781.objecter ms_handle_connect 0x7f6e30023330 2024-02-19T21:09:48.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.696+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6e30026fa0 con 0x7f6e30023330 2024-02-19T21:09:48.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.696+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:09:48.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.696+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 15 = system:0 2024-02-19T21:09:48.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.696+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e30022cb0 osd=7 4 2024-02-19T21:09:48.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.696+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 7 15 2024-02-19T21:09:48.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.711+0000 7f6e5450c700 10 client.49781.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:09:48.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.711+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign -1 16 2024-02-19T21:09:48.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.711+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30022cb0 osd=7 3 2024-02-19T21:09:48.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.711+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30022cb0 osd=7 3 2024-02-19T21:09:48.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.711+0000 7f6e5450c700 20 client.49781.objecter _get_session s=0x7f6e30022cb0 osd=7 3 2024-02-19T21:09:48.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.711+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_remove -1 16 2024-02-19T21:09:48.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.711+0000 7f6e5450c700 20 client.49781.objecter get_session s=0x7f6e30022cb0 osd=7 3 2024-02-19T21:09:48.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.711+0000 7f6e5450c700 15 client.49781.objecter _session_command_op_assign 7 16 2024-02-19T21:09:48.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.711+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30022cb0 osd=7 4 2024-02-19T21:09:48.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.711+0000 7f6e5450c700 10 client.49781.objecter _send_command 16 2024-02-19T21:09:48.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.711+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6e30026fa0 con 0x7f6e30023330 2024-02-19T21:09:48.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e3f7fe700 1 -- 172.21.15.163:0/2049860212 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6e30026fa0 con 0x7f6e30023330 2024-02-19T21:09:48.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e3f7fe700 10 client.49781.objecter ms_dispatch 0x7f6e4c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:09:48.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e3f7fe700 10 client.49781.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e3f7fe700 20 client.49781.objecter put_session s=0x7f6e30022cb0 osd=7 4 2024-02-19T21:09:48.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e3f7fe700 15 client.49781.objecter _session_command_op_remove 7 16 2024-02-19T21:09:48.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 10 librados: watch_flush enter 2024-02-19T21:09:48.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 10 librados: watch_flush exit 2024-02-19T21:09:48.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 10 client.49781.objecter close_session for osd.0 2024-02-19T21:09:48.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 >> v1:172.21.15.163:6801/1559418115 conn(0x7f6e30004530 legacy=0x7f6e30002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:48.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e300051a0 osd=0 1 2024-02-19T21:09:48.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 10 client.49781.objecter close_session for osd.1 2024-02-19T21:09:48.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 >> v1:172.21.15.163:6805/3125047432 conn(0x7f6e30004150 legacy=0x7f6e3000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:48.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30003fe0 osd=1 1 2024-02-19T21:09:48.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 10 client.49781.objecter close_session for osd.2 2024-02-19T21:09:48.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 >> v1:172.21.15.163:6809/1563158487 conn(0x7f6e3000d2f0 legacy=0x7f6e3000d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:48.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30003630 osd=2 1 2024-02-19T21:09:48.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 10 client.49781.objecter close_session for osd.3 2024-02-19T21:09:48.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 >> v1:172.21.15.163:6813/2112279769 conn(0x7f6e30011ff0 legacy=0x7f6e300143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:48.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30011970 osd=3 1 2024-02-19T21:09:48.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 10 client.49781.objecter close_session for osd.4 2024-02-19T21:09:48.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 >> v1:172.21.15.193:6800/3129716965 conn(0x7f6e3000c940 legacy=0x7f6e30015d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:48.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30015ba0 osd=4 1 2024-02-19T21:09:48.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 10 client.49781.objecter close_session for osd.5 2024-02-19T21:09:48.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e52aa9700 1 -- 172.21.15.163:0/2049860212 reap_dead start 2024-02-19T21:09:48.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 >> v1:172.21.15.193:6804/805161991 conn(0x7f6e30016960 legacy=0x7f6e3001d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:48.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e52aa9700 1 -- 172.21.15.163:0/2049860212 reap_dead start 2024-02-19T21:09:48.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e52aa9700 1 -- 172.21.15.163:0/2049860212 reap_dead start 2024-02-19T21:09:48.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e3000ced0 osd=5 1 2024-02-19T21:09:48.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 10 client.49781.objecter close_session for osd.6 2024-02-19T21:09:48.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 >> v1:172.21.15.193:6808/787604615 conn(0x7f6e3001f1a0 legacy=0x7f6e30021590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:48.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30016da0 osd=6 1 2024-02-19T21:09:48.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 10 client.49781.objecter close_session for osd.7 2024-02-19T21:09:48.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 >> v1:172.21.15.193:6812/157927432 conn(0x7f6e30023330 legacy=0x7f6e30025720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:48.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 20 client.49781.objecter put_session s=0x7f6e30022cb0 osd=7 1 2024-02-19T21:09:48.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 20 client.49781.objecter shutdown clearing up homeless session... 2024-02-19T21:09:48.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 10 client.49781.objecter successfully canceled tick 2024-02-19T21:09:48.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6e38063cb0 legacy=0x7f6e38066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:48.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 >> v1:172.21.15.163:6790/0 conn(0x7f6e4c0ff670 legacy=0x7f6e4c112820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:48.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 shutdown_connections 2024-02-19T21:09:48.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 >> 172.21.15.163:0/2049860212 conn(0x7f6e4c0f2470 msgr2=0x7f6e4c0f5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:48.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 shutdown_connections 2024-02-19T21:09:48.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.713+0000 7f6e5450c700 1 -- 172.21.15.163:0/2049860212 wait complete. 2024-02-19T21:09:48.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:47.714+0000 7f6e5450c700 1 librados: shutdown 2024-02-19T21:09:48.035 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.035 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.035 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.035 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.035 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.035 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.035 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.035 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:09:48.110 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:09:48.111 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-02-19T21:09:48.111 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-02-19T21:09:48.111 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-02-19T21:09:48.111 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.111 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.24370.0:1 2.12 2:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e57)", 2024-02-19T21:09:48.111 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:01:16.016370+0000", 2024-02-19T21:09:48.111 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 512.09464638300005, 2024-02-19T21:09:48.111 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.25557504599999997, 2024-02-19T21:09:48.111 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.111 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-02-19T21:09:48.111 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.111 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.24370", 2024-02-19T21:09:48.111 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1964237833", 2024-02-19T21:09:48.111 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:09:48.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.112 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.112 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.112 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:16.016370+0000", 2024-02-19T21:09:48.112 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.112 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.112 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:16.016370+0000", 2024-02-19T21:09:48.112 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:09:48.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.112 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.112 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:16.016368+0000", 2024-02-19T21:09:48.113 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8349999999999998e-06 2024-02-19T21:09:48.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.113 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.113 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.113 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:16.016375+0000", 2024-02-19T21:09:48.113 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7919999999999998e-06 2024-02-19T21:09:48.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.113 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.114 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.114 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:16.016384+0000", 2024-02-19T21:09:48.114 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1269999999999997e-06 2024-02-19T21:09:48.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.114 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.114 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:16.016391+0000", 2024-02-19T21:09:48.114 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6724000000000001e-05 2024-02-19T21:09:48.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.115 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:16.016448+0000", 2024-02-19T21:09:48.115 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8850000000000001e-06 2024-02-19T21:09:48.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:09:48.115 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:16.016454+0000", 2024-02-19T21:09:48.115 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.255397386 2024-02-19T21:09:48.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.116 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:16.271852+0000", 2024-02-19T21:09:48.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3246999999999997e-05 2024-02-19T21:09:48.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.116 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.116 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:01:16.271945+0000", 2024-02-19T21:09:48.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.25557504599999997 2024-02-19T21:09:48.116 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.116 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.116 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.116 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25213.0:2 34.6 34:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e103)", 2024-02-19T21:09:48.117 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:42.715309+0000", 2024-02-19T21:09:48.117 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 305.395707582, 2024-02-19T21:09:48.117 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.043999731, 2024-02-19T21:09:48.117 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.117 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.117 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.117 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25213", 2024-02-19T21:09:48.117 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2094902030", 2024-02-19T21:09:48.117 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:09:48.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.117 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.117 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.117 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.117 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715309+0000", 2024-02-19T21:09:48.118 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.118 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.118 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715309+0000", 2024-02-19T21:09:48.118 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:09:48.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.118 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.118 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715307+0000", 2024-02-19T21:09:48.118 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0489999999999993e-06 2024-02-19T21:09:48.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.118 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.118 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715316+0000", 2024-02-19T21:09:48.119 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2014999999999999e-05 2024-02-19T21:09:48.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.119 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.119 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715328+0000", 2024-02-19T21:09:48.119 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3040000000000005e-06 2024-02-19T21:09:48.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.119 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.119 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715336+0000", 2024-02-19T21:09:48.119 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9018999999999995e-05 2024-02-19T21:09:48.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.119 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.119 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715435+0000", 2024-02-19T21:09:48.120 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9249999999999995e-06 2024-02-19T21:09:48.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.120 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:09:48.120 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715443+0000", 2024-02-19T21:09:48.120 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.033608503999999997 2024-02-19T21:09:48.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.120 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.120 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.749051+0000", 2024-02-19T21:09:48.120 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013195700000000001 2024-02-19T21:09:48.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.120 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.749183+0000", 2024-02-19T21:09:48.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014490099999999999 2024-02-19T21:09:48.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.121 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,2", 2024-02-19T21:09:48.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.749328+0000", 2024-02-19T21:09:48.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000480402 2024-02-19T21:09:48.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.121 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:09:48.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.749809+0000", 2024-02-19T21:09:48.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0069833769999999998 2024-02-19T21:09:48.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.756792+0000", 2024-02-19T21:09:48.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024635049999999999 2024-02-19T21:09:48.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759256+0000", 2024-02-19T21:09:48.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9428999999999999e-05 2024-02-19T21:09:48.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759285+0000", 2024-02-19T21:09:48.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3498000000000001e-05 2024-02-19T21:09:48.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.123 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.123 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759309+0000", 2024-02-19T21:09:48.123 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.043999731 2024-02-19T21:09:48.123 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.123 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.123 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.123 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25255.0:2 39.1a 39:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected e103)", 2024-02-19T21:09:48.123 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:42.715388+0000", 2024-02-19T21:09:48.123 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 305.39562820700002, 2024-02-19T21:09:48.123 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0953346740000001, 2024-02-19T21:09:48.123 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.124 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.124 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.124 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25255", 2024-02-19T21:09:48.124 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1026204543", 2024-02-19T21:09:48.124 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:09:48.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.124 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.124 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.124 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715388+0000", 2024-02-19T21:09:48.124 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.124 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.125 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715388+0000", 2024-02-19T21:09:48.125 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:09:48.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.125 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.125 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715387+0000", 2024-02-19T21:09:48.125 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5220000000000004e-06 2024-02-19T21:09:48.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.125 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.125 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715393+0000", 2024-02-19T21:09:48.125 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1680000000000004e-06 2024-02-19T21:09:48.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.126 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.126 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715402+0000", 2024-02-19T21:09:48.126 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2669999999999997e-06 2024-02-19T21:09:48.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.126 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.126 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715407+0000", 2024-02-19T21:09:48.126 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0061390949999999998 2024-02-19T21:09:48.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.126 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.126 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.721546+0000", 2024-02-19T21:09:48.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.464e-06 2024-02-19T21:09:48.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.127 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:09:48.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.721553+0000", 2024-02-19T21:09:48.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.041467176 2024-02-19T21:09:48.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.127 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.763021+0000", 2024-02-19T21:09:48.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000122223 2024-02-19T21:09:48.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.128 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.128 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.763143+0000", 2024-02-19T21:09:48.128 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000135525 2024-02-19T21:09:48.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.128 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.128 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,7", 2024-02-19T21:09:48.128 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.763278+0000", 2024-02-19T21:09:48.128 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001775127 2024-02-19T21:09:48.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.128 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.128 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:09:48.128 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.765053+0000", 2024-02-19T21:09:48.128 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011170955999999999 2024-02-19T21:09:48.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.128 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.129 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.129 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.776224+0000", 2024-02-19T21:09:48.129 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.034282156000000001 2024-02-19T21:09:48.129 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.129 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.129 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.129 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.810507+0000", 2024-02-19T21:09:48.129 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000115518 2024-02-19T21:09:48.129 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.129 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.129 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.129 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.810622+0000", 2024-02-19T21:09:48.129 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000100735 2024-02-19T21:09:48.130 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.130 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.130 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.130 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.810723+0000", 2024-02-19T21:09:48.130 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0953346740000001 2024-02-19T21:09:48.130 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.130 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.130 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.130 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.130 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.130 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25261.0:2 38.15 38:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-19T21:09:48.130 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:43.718525+0000", 2024-02-19T21:09:48.130 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 304.39249093199999, 2024-02-19T21:09:48.130 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.092407107000000002, 2024-02-19T21:09:48.130 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.131 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.131 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.131 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25261", 2024-02-19T21:09:48.131 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3188847468", 2024-02-19T21:09:48.131 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:09:48.131 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.131 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.131 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.131 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.131 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718525+0000", 2024-02-19T21:09:48.131 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.131 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.131 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.131 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.131 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718525+0000", 2024-02-19T21:09:48.132 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:09:48.132 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.132 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.132 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.132 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718522+0000", 2024-02-19T21:09:48.132 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3332999999999999e-05 2024-02-19T21:09:48.132 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.132 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.132 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.132 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718535+0000", 2024-02-19T21:09:48.132 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2945000000000001e-05 2024-02-19T21:09:48.132 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.132 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.133 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.133 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718548+0000", 2024-02-19T21:09:48.133 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9999999999999996e-06 2024-02-19T21:09:48.133 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.133 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.133 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.133 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718556+0000", 2024-02-19T21:09:48.133 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8600000000000001e-05 2024-02-19T21:09:48.133 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.133 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.133 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.133 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718614+0000", 2024-02-19T21:09:48.133 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4850000000000003e-06 2024-02-19T21:09:48.133 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.134 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.134 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:09:48.134 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718622+0000", 2024-02-19T21:09:48.134 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.080586017999999995 2024-02-19T21:09:48.134 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.134 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.134 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.134 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.799208+0000", 2024-02-19T21:09:48.134 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013370499999999999 2024-02-19T21:09:48.134 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.134 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.134 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.134 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.799342+0000", 2024-02-19T21:09:48.134 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015074000000000001 2024-02-19T21:09:48.134 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.135 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.135 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-02-19T21:09:48.135 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.799492+0000", 2024-02-19T21:09:48.135 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00074646999999999995 2024-02-19T21:09:48.135 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.135 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.135 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:09:48.135 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.800239+0000", 2024-02-19T21:09:48.135 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0041852749999999996 2024-02-19T21:09:48.135 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.135 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.135 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.135 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.804424+0000", 2024-02-19T21:09:48.136 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.006421637 2024-02-19T21:09:48.136 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.136 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.136 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.136 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.810846+0000", 2024-02-19T21:09:48.136 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7818000000000002e-05 2024-02-19T21:09:48.136 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.136 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.136 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.136 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.810894+0000", 2024-02-19T21:09:48.136 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8928999999999999e-05 2024-02-19T21:09:48.136 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.137 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.137 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.137 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.810933+0000", 2024-02-19T21:09:48.137 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.092407107000000002 2024-02-19T21:09:48.137 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.137 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.137 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.137 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.137 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.137 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25243.0:1 44.6s0 44:6ec48444:test-rados-api-smithi163-83968-8::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e106)", 2024-02-19T21:09:48.137 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:45.953526+0000", 2024-02-19T21:09:48.137 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 302.15749001299997, 2024-02-19T21:09:48.137 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0395346679999999, 2024-02-19T21:09:48.137 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.138 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.138 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.138 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25243", 2024-02-19T21:09:48.138 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3720420201", 2024-02-19T21:09:48.138 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:09:48.138 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.138 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.138 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.138 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.138 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953526+0000", 2024-02-19T21:09:48.138 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.138 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.138 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.139 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.139 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953526+0000", 2024-02-19T21:09:48.139 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:09:48.139 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.139 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.139 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.139 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953525+0000", 2024-02-19T21:09:48.139 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8869999999999992e-06 2024-02-19T21:09:48.139 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.139 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.139 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.139 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953532+0000", 2024-02-19T21:09:48.139 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020723920000000002 2024-02-19T21:09:48.139 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.140 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.140 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.955605+0000", 2024-02-19T21:09:48.140 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4469999999999995e-06 2024-02-19T21:09:48.140 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.140 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.140 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.955614+0000", 2024-02-19T21:09:48.140 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9202000000000002e-05 2024-02-19T21:09:48.140 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.140 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.140 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.955654+0000", 2024-02-19T21:09:48.141 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7e-06 2024-02-19T21:09:48.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.141 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.141 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:09:48.141 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.955655+0000", 2024-02-19T21:09:48.141 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0209778119999999 2024-02-19T21:09:48.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.141 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.141 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.141 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.976633+0000", 2024-02-19T21:09:48.141 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013542099999999999 2024-02-19T21:09:48.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.141 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.141 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.141 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.976768+0000", 2024-02-19T21:09:48.142 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00051974200000000001 2024-02-19T21:09:48.142 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.142 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.142 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:09:48.142 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.977288+0000", 2024-02-19T21:09:48.142 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0018645790000000001 2024-02-19T21:09:48.142 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.142 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.142 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:09:48.142 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.979153+0000", 2024-02-19T21:09:48.142 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013854811 2024-02-19T21:09:48.142 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.142 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.142 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.143 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.993008+0000", 2024-02-19T21:09:48.143 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3474999999999999e-05 2024-02-19T21:09:48.143 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.143 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.143 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.143 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.993061+0000", 2024-02-19T21:09:48.143 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0395346679999999 2024-02-19T21:09:48.143 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.143 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.143 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.143 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.143 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.143 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25228.0:1 42.5s0 42:abdeb581:test-rados-api-smithi163-83960-7::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e106)", 2024-02-19T21:09:48.144 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:45.953691+0000", 2024-02-19T21:09:48.144 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 302.15732524100002, 2024-02-19T21:09:48.144 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.03047446, 2024-02-19T21:09:48.144 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.144 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.144 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.144 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25228", 2024-02-19T21:09:48.144 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/516814289", 2024-02-19T21:09:48.144 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:09:48.144 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.144 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.144 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.144 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.144 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953691+0000", 2024-02-19T21:09:48.144 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.145 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.145 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953691+0000", 2024-02-19T21:09:48.145 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:09:48.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.145 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.145 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953690+0000", 2024-02-19T21:09:48.145 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2419999999999997e-06 2024-02-19T21:09:48.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.145 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.145 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953696+0000", 2024-02-19T21:09:48.146 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00023778100000000001 2024-02-19T21:09:48.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.146 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.146 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953934+0000", 2024-02-19T21:09:48.146 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2859999999999999e-06 2024-02-19T21:09:48.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.146 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.146 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953942+0000", 2024-02-19T21:09:48.146 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0016913939999999999 2024-02-19T21:09:48.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.146 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.146 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.955633+0000", 2024-02-19T21:09:48.147 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.225e-06 2024-02-19T21:09:48.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.147 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.147 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:09:48.147 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.955636+0000", 2024-02-19T21:09:48.147 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.008495243 2024-02-19T21:09:48.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.147 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.147 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.147 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.964132+0000", 2024-02-19T21:09:48.147 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000152194 2024-02-19T21:09:48.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.147 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.147 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.147 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.964284+0000", 2024-02-19T21:09:48.148 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00045546399999999999 2024-02-19T21:09:48.148 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.148 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.148 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:09:48.148 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.964739+0000", 2024-02-19T21:09:48.148 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00086119899999999999 2024-02-19T21:09:48.148 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.148 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.148 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:09:48.148 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.965601+0000", 2024-02-19T21:09:48.148 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.018476871999999998 2024-02-19T21:09:48.148 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.148 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.149 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.149 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.984077+0000", 2024-02-19T21:09:48.149 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8187000000000001e-05 2024-02-19T21:09:48.149 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.149 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.149 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.149 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.984166+0000", 2024-02-19T21:09:48.149 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.03047446 2024-02-19T21:09:48.149 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.149 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.149 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.149 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.149 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.149 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.24895.0:14 25.c 25:354b2f79:test-rados-api-smithi163-83944-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:09:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:47.957539+0000", 2024-02-19T21:09:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 300.15347770099999, 2024-02-19T21:09:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.063082621000000005, 2024-02-19T21:09:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.24895", 2024-02-19T21:09:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3326199758", 2024-02-19T21:09:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 14 2024-02-19T21:09:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957539+0000", 2024-02-19T21:09:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.151 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.151 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.151 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.151 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957539+0000", 2024-02-19T21:09:48.151 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:09:48.151 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.151 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.151 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.151 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957536+0000", 2024-02-19T21:09:48.151 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6256000000000002e-05 2024-02-19T21:09:48.151 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.151 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.151 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.151 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957552+0000", 2024-02-19T21:09:48.151 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2793e-05 2024-02-19T21:09:48.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.152 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.152 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957575+0000", 2024-02-19T21:09:48.152 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0999999999999993e-06 2024-02-19T21:09:48.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.152 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.152 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957584+0000", 2024-02-19T21:09:48.152 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6638999999999998e-05 2024-02-19T21:09:48.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.152 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.152 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957640+0000", 2024-02-19T21:09:48.153 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6559999999999992e-06 2024-02-19T21:09:48.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.153 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.153 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:09:48.153 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957648+0000", 2024-02-19T21:09:48.153 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0094927950000000001 2024-02-19T21:09:48.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.153 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.153 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.153 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.967141+0000", 2024-02-19T21:09:48.153 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0094054069999999993 2024-02-19T21:09:48.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.153 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.153 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.154 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976546+0000", 2024-02-19T21:09:48.154 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013502699999999999 2024-02-19T21:09:48.154 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.154 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.154 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-02-19T21:09:48.154 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976681+0000", 2024-02-19T21:09:48.154 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001538427 2024-02-19T21:09:48.154 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.154 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.154 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:09:48.154 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.978220+0000", 2024-02-19T21:09:48.154 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011847066 2024-02-19T21:09:48.154 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.155 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.155 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.155 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.990067+0000", 2024-02-19T21:09:48.155 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.030463536999999999 2024-02-19T21:09:48.155 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.155 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.155 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.155 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.020530+0000", 2024-02-19T21:09:48.155 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6168000000000002e-05 2024-02-19T21:09:48.155 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.155 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.155 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.155 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.020577+0000", 2024-02-19T21:09:48.155 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4790999999999999e-05 2024-02-19T21:09:48.156 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.156 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.156 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.156 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.020621+0000", 2024-02-19T21:09:48.156 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.063082621000000005 2024-02-19T21:09:48.156 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.156 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.156 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.156 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.156 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.156 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.14940.0:1 50.0s0 50:0c5c848e:test-rados-api-smithi163-83904-11::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:09:48.156 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:47.977197+0000", 2024-02-19T21:09:48.156 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 300.13381975999999, 2024-02-19T21:09:48.156 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.62635811600000002, 2024-02-19T21:09:48.157 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.157 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.157 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.157 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.14940", 2024-02-19T21:09:48.157 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/676446879", 2024-02-19T21:09:48.157 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:09:48.157 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.157 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.157 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.157 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.157 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.977197+0000", 2024-02-19T21:09:48.157 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.157 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.157 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.158 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.158 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.977197+0000", 2024-02-19T21:09:48.158 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:09:48.158 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.158 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.158 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.158 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.977194+0000", 2024-02-19T21:09:48.158 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5399999999999996e-06 2024-02-19T21:09:48.158 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.158 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.158 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.158 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.977203+0000", 2024-02-19T21:09:48.158 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6239999999999997e-06 2024-02-19T21:09:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.977213+0000", 2024-02-19T21:09:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5740000000000001e-06 2024-02-19T21:09:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.977219+0000", 2024-02-19T21:09:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001035041 2024-02-19T21:09:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.978254+0000", 2024-02-19T21:09:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3029999999999998e-06 2024-02-19T21:09:48.160 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.160 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.160 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:09:48.160 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.978256+0000", 2024-02-19T21:09:48.160 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.60994813199999998 2024-02-19T21:09:48.160 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.160 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.160 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.160 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.588205+0000", 2024-02-19T21:09:48.160 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.1259000000000002e-05 2024-02-19T21:09:48.160 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.160 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.160 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.160 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.588296+0000", 2024-02-19T21:09:48.161 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00027813000000000002 2024-02-19T21:09:48.161 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.161 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.161 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:09:48.161 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.588574+0000", 2024-02-19T21:09:48.161 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000344502 2024-02-19T21:09:48.161 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.161 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.161 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:09:48.161 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.588918+0000", 2024-02-19T21:09:48.161 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014572966 2024-02-19T21:09:48.161 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.161 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.161 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.161 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.603491+0000", 2024-02-19T21:09:48.162 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3334e-05 2024-02-19T21:09:48.162 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.162 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.162 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.162 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.603555+0000", 2024-02-19T21:09:48.162 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.62635811600000002 2024-02-19T21:09:48.162 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.162 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.162 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.162 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.162 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.162 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35237.0:2 52.b 52:d7043a55:test-rados-api-smithi163-83984-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-19T21:09:48.162 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:48.521853+0000", 2024-02-19T21:09:48.162 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 299.58916346299998, 2024-02-19T21:09:48.163 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.050592887000000003, 2024-02-19T21:09:48.163 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.163 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.163 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.163 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35237", 2024-02-19T21:09:48.163 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1081998331", 2024-02-19T21:09:48.163 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:09:48.163 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.163 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.163 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.163 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.163 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.521853+0000", 2024-02-19T21:09:48.163 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.163 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.163 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.164 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.164 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.521853+0000", 2024-02-19T21:09:48.164 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:09:48.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.164 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.164 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.164 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.521851+0000", 2024-02-19T21:09:48.164 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9590000000000001e-06 2024-02-19T21:09:48.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.164 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.164 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.164 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.521860+0000", 2024-02-19T21:09:48.164 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1649e-05 2024-02-19T21:09:48.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.165 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.165 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.165 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.521871+0000", 2024-02-19T21:09:48.165 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2400000000000007e-06 2024-02-19T21:09:48.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.165 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.165 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.165 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.521880+0000", 2024-02-19T21:09:48.165 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001923971 2024-02-19T21:09:48.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.165 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.165 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.165 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.523804+0000", 2024-02-19T21:09:48.166 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6629999999999999e-06 2024-02-19T21:09:48.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.166 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.166 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:09:48.166 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.523812+0000", 2024-02-19T21:09:48.166 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.040019490999999997 2024-02-19T21:09:48.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.166 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.166 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.166 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.563832+0000", 2024-02-19T21:09:48.166 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5551999999999997e-05 2024-02-19T21:09:48.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.166 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.167 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.167 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.563917+0000", 2024-02-19T21:09:48.167 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012541700000000001 2024-02-19T21:09:48.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.167 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.167 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,7", 2024-02-19T21:09:48.167 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.564043+0000", 2024-02-19T21:09:48.167 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00052241799999999997 2024-02-19T21:09:48.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.167 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.167 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:09:48.167 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.564565+0000", 2024-02-19T21:09:48.167 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001139024 2024-02-19T21:09:48.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.168 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.168 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.168 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.565704+0000", 2024-02-19T21:09:48.168 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0066732509999999998 2024-02-19T21:09:48.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.168 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.168 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.168 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.572377+0000", 2024-02-19T21:09:48.168 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2475000000000003e-05 2024-02-19T21:09:48.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.168 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.168 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.168 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.572410+0000", 2024-02-19T21:09:48.168 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5967000000000003e-05 2024-02-19T21:09:48.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.169 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.169 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.169 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.572446+0000", 2024-02-19T21:09:48.169 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.050592887000000003 2024-02-19T21:09:48.169 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.169 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.169 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.169 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.169 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.14940.0:10 50.0s0 50:0928318f:test-rados-api-smithi163-83904-14::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e110)", 2024-02-19T21:09:48.169 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:50.620751+0000", 2024-02-19T21:09:48.169 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 297.49026573700002, 2024-02-19T21:09:48.169 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.016526605999999999, 2024-02-19T21:09:48.170 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.170 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.170 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.170 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.14940", 2024-02-19T21:09:48.170 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/676446879", 2024-02-19T21:09:48.170 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 10 2024-02-19T21:09:48.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.170 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.170 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.170 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.170 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620751+0000", 2024-02-19T21:09:48.170 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.170 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.171 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.171 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620751+0000", 2024-02-19T21:09:48.171 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:09:48.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.171 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.171 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.171 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620748+0000", 2024-02-19T21:09:48.171 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9029999999999999e-06 2024-02-19T21:09:48.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.171 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.171 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.171 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620758+0000", 2024-02-19T21:09:48.171 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7120000000000001e-05 2024-02-19T21:09:48.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.172 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.172 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.172 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620785+0000", 2024-02-19T21:09:48.172 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6659999999999997e-06 2024-02-19T21:09:48.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.172 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.172 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.172 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620794+0000", 2024-02-19T21:09:48.172 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0709000000000003e-05 2024-02-19T21:09:48.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.172 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.172 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.172 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620874+0000", 2024-02-19T21:09:48.172 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012368899999999999 2024-02-19T21:09:48.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.173 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.173 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.173 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620998+0000", 2024-02-19T21:09:48.173 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000371963 2024-02-19T21:09:48.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.173 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.173 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:09:48.173 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.621370+0000", 2024-02-19T21:09:48.173 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00044841800000000002 2024-02-19T21:09:48.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.173 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.173 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:09:48.173 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.621819+0000", 2024-02-19T21:09:48.174 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015420919999999999 2024-02-19T21:09:48.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.174 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.174 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.174 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.637239+0000", 2024-02-19T21:09:48.174 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7781000000000003e-05 2024-02-19T21:09:48.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.174 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.174 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.174 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.637277+0000", 2024-02-19T21:09:48.174 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.016526605999999999 2024-02-19T21:09:48.174 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.174 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.174 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.175 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.175 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.24865.0:21 11.8 e112/99 11:1381d4a2:test-rados-api-smithi163-83952-6::foo:head v 112'4, mlcod=110'1)", 2024-02-19T21:09:48.175 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:51.587142+0000", 2024-02-19T21:09:48.175 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 296.52387470500003, 2024-02-19T21:09:48.175 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026422297000000001, 2024-02-19T21:09:48.175 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.175 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.175 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.175 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.175 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.175 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.587142+0000", 2024-02-19T21:09:48.175 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.175 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.176 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.176 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.587142+0000", 2024-02-19T21:09:48.176 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:09:48.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.176 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.176 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.176 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.587140+0000", 2024-02-19T21:09:48.176 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3057999999999999e-05 2024-02-19T21:09:48.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.176 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.176 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.176 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.587153+0000", 2024-02-19T21:09:48.176 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00027089900000000001 2024-02-19T21:09:48.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.177 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.177 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.177 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.587424+0000", 2024-02-19T21:09:48.177 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2164000000000001e-05 2024-02-19T21:09:48.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.177 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.177 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.177 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.587436+0000", 2024-02-19T21:09:48.177 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025388634 2024-02-19T21:09:48.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.177 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.177 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.177 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.612825+0000", 2024-02-19T21:09:48.177 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0619e-05 2024-02-19T21:09:48.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.178 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.178 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.178 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.612845+0000", 2024-02-19T21:09:48.178 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00067890799999999999 2024-02-19T21:09:48.178 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.178 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.178 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.178 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.613524+0000", 2024-02-19T21:09:48.178 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9892999999999999e-05 2024-02-19T21:09:48.178 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.178 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.178 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.178 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.613564+0000", 2024-02-19T21:09:48.178 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026422297000000001 2024-02-19T21:09:48.179 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.179 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.179 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.179 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.179 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.179 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.24919.0:2079 28.9 28:90852524:test-rados-api-smithi163-83920-17::big2.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-02-19T21:09:48.179 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.501465+0000", 2024-02-19T21:09:48.179 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 275.60955097800002, 2024-02-19T21:09:48.179 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.23286506000000001, 2024-02-19T21:09:48.179 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.179 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.179 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.24919", 2024-02-19T21:09:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/4089460683", 2024-02-19T21:09:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2079 2024-02-19T21:09:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501465+0000", 2024-02-19T21:09:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.180 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501465+0000", 2024-02-19T21:09:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:09:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501462+0000", 2024-02-19T21:09:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2568000000000001e-05 2024-02-19T21:09:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501475+0000", 2024-02-19T21:09:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6639000000000001e-05 2024-02-19T21:09:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.181 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501491+0000", 2024-02-19T21:09:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9549999999999992e-06 2024-02-19T21:09:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501499+0000", 2024-02-19T21:09:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6508999999999997e-05 2024-02-19T21:09:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501556+0000", 2024-02-19T21:09:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010162699999999999 2024-02-19T21:09:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.182 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501658+0000", 2024-02-19T21:09:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.18445410800000001 2024-02-19T21:09:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,6", 2024-02-19T21:09:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.686112+0000", 2024-02-19T21:09:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010606519 2024-02-19T21:09:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.183 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.696718+0000", 2024-02-19T21:09:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00049238600000000004 2024-02-19T21:09:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:09:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.697211+0000", 2024-02-19T21:09:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037011418999999997 2024-02-19T21:09:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.734222+0000", 2024-02-19T21:09:48.184 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6672000000000002e-05 2024-02-19T21:09:48.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.185 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.185 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.185 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.734279+0000", 2024-02-19T21:09:48.185 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1786000000000002e-05 2024-02-19T21:09:48.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.185 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.185 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.185 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.734330+0000", 2024-02-19T21:09:48.185 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.23286506000000001 2024-02-19T21:09:48.185 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.185 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.185 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.4.0:288 85.1e e139/133 85:780b1894:test-rados-api-smithi163-83928-12::bar:2 v 139'4, mlcod=138'1)", 2024-02-19T21:09:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:19.181043+0000", 2024-02-19T21:09:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 268.92997316399999, 2024-02-19T21:09:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.012983807, 2024-02-19T21:09:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.181043+0000", 2024-02-19T21:09:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.186 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.181043+0000", 2024-02-19T21:09:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:09:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.181043+0000", 2024-02-19T21:09:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1043000000000001e-05 2024-02-19T21:09:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.181054+0000", 2024-02-19T21:09:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.387e-06 2024-02-19T21:09:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.181060+0000", 2024-02-19T21:09:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.401e-06 2024-02-19T21:09:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.181068+0000", 2024-02-19T21:09:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00016220000000000001 2024-02-19T21:09:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.181230+0000", 2024-02-19T21:09:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1056e-05 2024-02-19T21:09:48.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.181251+0000", 2024-02-19T21:09:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.012747283 2024-02-19T21:09:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.193998+0000", 2024-02-19T21:09:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.8738000000000002e-05 2024-02-19T21:09:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.194027+0000", 2024-02-19T21:09:48.189 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.012983807 2024-02-19T21:09:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.47072.0:2 90.13 90:cbb9903e:::test_obj:head [write 0~128 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected e139)", 2024-02-19T21:09:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:19.211019+0000", 2024-02-19T21:09:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 268.89999720700001, 2024-02-19T21:09:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.110542764, 2024-02-19T21:09:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.47072", 2024-02-19T21:09:48.190 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3219046901", 2024-02-19T21:09:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:09:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211019+0000", 2024-02-19T21:09:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211019+0000", 2024-02-19T21:09:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:09:48.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.192 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.192 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211017+0000", 2024-02-19T21:09:48.192 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6989999999999997e-06 2024-02-19T21:09:48.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.192 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.192 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211024+0000", 2024-02-19T21:09:48.192 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9810000000000003e-06 2024-02-19T21:09:48.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.192 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.192 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211032+0000", 2024-02-19T21:09:48.192 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.78e-06 2024-02-19T21:09:48.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.193 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.193 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.193 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211037+0000", 2024-02-19T21:09:48.193 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1912999999999997e-05 2024-02-19T21:09:48.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.193 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.193 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.193 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211079+0000", 2024-02-19T21:09:48.193 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1199000000000001e-05 2024-02-19T21:09:48.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.193 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.193 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:09:48.193 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211090+0000", 2024-02-19T21:09:48.193 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015814228999999999 2024-02-19T21:09:48.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.194 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.194 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.194 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.226904+0000", 2024-02-19T21:09:48.194 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9059000000000003e-05 2024-02-19T21:09:48.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.194 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.194 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.194 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.226983+0000", 2024-02-19T21:09:48.194 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010925399999999999 2024-02-19T21:09:48.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.194 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.194 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,2", 2024-02-19T21:09:48.195 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.227092+0000", 2024-02-19T21:09:48.195 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00033114899999999998 2024-02-19T21:09:48.195 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.195 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.195 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:09:48.195 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.227424+0000", 2024-02-19T21:09:48.195 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.078279331999999993 2024-02-19T21:09:48.195 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.195 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.195 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.195 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.305703+0000", 2024-02-19T21:09:48.195 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015761226999999999 2024-02-19T21:09:48.195 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.195 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.196 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.196 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.321464+0000", 2024-02-19T21:09:48.196 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5554999999999999e-05 2024-02-19T21:09:48.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.196 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.196 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.196 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.321520+0000", 2024-02-19T21:09:48.196 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2196000000000001e-05 2024-02-19T21:09:48.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.196 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.196 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.196 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.321562+0000", 2024-02-19T21:09:48.196 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.110542764 2024-02-19T21:09:48.196 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.197 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.197 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.197 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.197 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.197 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22545.0:4 150.1e 150:7cdf4dc8:::oid_XattrsRoundTripPP_:head [setxattr attr1 (12) in=5b] snapc 0=[] ondisk+write+known_if_redirected e217)", 2024-02-19T21:09:48.197 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:06:39.209473+0000", 2024-02-19T21:09:48.197 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 188.901543299, 2024-02-19T21:09:48.197 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022521461999999999, 2024-02-19T21:09:48.197 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.197 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.197 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.197 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22545", 2024-02-19T21:09:48.197 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3549317140", 2024-02-19T21:09:48.197 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 4 2024-02-19T21:09:48.198 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.198 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.198 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.198 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.198 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209473+0000", 2024-02-19T21:09:48.198 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.198 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.198 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.199 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.199 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209473+0000", 2024-02-19T21:09:48.199 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:09:48.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.199 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.199 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209470+0000", 2024-02-19T21:09:48.199 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1134e-05 2024-02-19T21:09:48.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.199 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.199 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209481+0000", 2024-02-19T21:09:48.199 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2923e-05 2024-02-19T21:09:48.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.200 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.200 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209494+0000", 2024-02-19T21:09:48.200 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9760000000000001e-06 2024-02-19T21:09:48.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.200 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.200 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.200 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209503+0000", 2024-02-19T21:09:48.200 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0747000000000005e-05 2024-02-19T21:09:48.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.200 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.200 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.200 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209584+0000", 2024-02-19T21:09:48.200 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9016000000000002e-05 2024-02-19T21:09:48.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.201 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.201 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.201 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209683+0000", 2024-02-19T21:09:48.201 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015398000000000001 2024-02-19T21:09:48.201 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.201 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.201 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,7", 2024-02-19T21:09:48.201 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209837+0000", 2024-02-19T21:09:48.201 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0071038949999999998 2024-02-19T21:09:48.201 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.201 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.201 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:09:48.201 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.216941+0000", 2024-02-19T21:09:48.201 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001669797 2024-02-19T21:09:48.202 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.202 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.202 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.202 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.218611+0000", 2024-02-19T21:09:48.202 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013307836999999999 2024-02-19T21:09:48.202 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.202 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.202 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.202 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.231919+0000", 2024-02-19T21:09:48.202 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9233000000000001e-05 2024-02-19T21:09:48.202 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.202 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.202 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.203 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.231968+0000", 2024-02-19T21:09:48.203 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6804999999999998e-05 2024-02-19T21:09:48.203 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.203 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.203 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.203 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.231995+0000", 2024-02-19T21:09:48.203 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022521461999999999 2024-02-19T21:09:48.203 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.203 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.203 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.203 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.203 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.203 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.25753.0:2077 61.a e235/114 61:56c6a8b8:test-rados-api-smithi163-83928-26::foo:head v 235'20, mlcod=203'17)", 2024-02-19T21:09:48.203 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:06:55.208201+0000", 2024-02-19T21:09:48.204 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 172.902815301, 2024-02-19T21:09:48.204 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011404020000000001, 2024-02-19T21:09:48.204 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.204 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.204 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.204 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.204 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.204 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:55.208201+0000", 2024-02-19T21:09:48.204 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.204 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.204 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.204 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.204 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:55.208201+0000", 2024-02-19T21:09:48.204 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:09:48.205 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.205 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.205 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.205 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:55.208201+0000", 2024-02-19T21:09:48.205 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1202000000000001e-05 2024-02-19T21:09:48.205 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.205 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.205 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.205 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:55.208222+0000", 2024-02-19T21:09:48.205 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8220000000000002e-06 2024-02-19T21:09:48.205 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.205 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.206 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.206 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:55.208232+0000", 2024-02-19T21:09:48.206 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0180000000000001e-06 2024-02-19T21:09:48.206 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.206 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.206 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.206 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:55.208240+0000", 2024-02-19T21:09:48.206 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0067218909999999998 2024-02-19T21:09:48.206 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.206 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.206 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.206 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:55.214962+0000", 2024-02-19T21:09:48.206 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2639000000000001e-05 2024-02-19T21:09:48.206 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.207 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.207 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.207 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:55.214984+0000", 2024-02-19T21:09:48.207 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0045909999999999996 2024-02-19T21:09:48.207 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.207 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.207 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.207 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:55.219575+0000", 2024-02-19T21:09:48.207 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9912000000000001e-05 2024-02-19T21:09:48.207 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.207 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.207 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.207 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:55.219605+0000", 2024-02-19T21:09:48.207 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011404020000000001 2024-02-19T21:09:48.207 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.208 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.208 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.208 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.208 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.208 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.49328.0:2 165.18 165:1c88de7c:::oid_XattrListPP_:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected e244)", 2024-02-19T21:09:48.208 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:04.204109+0000", 2024-02-19T21:09:48.208 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 163.90690722599999, 2024-02-19T21:09:48.208 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025817819999999998, 2024-02-19T21:09:48.208 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.208 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.208 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.208 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.49328", 2024-02-19T21:09:48.208 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/4008237253", 2024-02-19T21:09:48.209 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:09:48.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.209 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.209 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.209 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.209 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204109+0000", 2024-02-19T21:09:48.209 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.209 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.209 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.209 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204109+0000", 2024-02-19T21:09:48.209 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:09:48.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.209 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.210 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.210 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204106+0000", 2024-02-19T21:09:48.210 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0121e-05 2024-02-19T21:09:48.210 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.210 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.210 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.210 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204117+0000", 2024-02-19T21:09:48.210 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6637999999999999e-05 2024-02-19T21:09:48.210 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.210 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.210 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.210 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204133+0000", 2024-02-19T21:09:48.210 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4710000000000007e-06 2024-02-19T21:09:48.211 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.211 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.211 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.211 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204142+0000", 2024-02-19T21:09:48.211 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5637999999999997e-05 2024-02-19T21:09:48.211 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.211 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.211 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.211 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204207+0000", 2024-02-19T21:09:48.211 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.973e-06 2024-02-19T21:09:48.211 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.211 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.211 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:09:48.211 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204214+0000", 2024-02-19T21:09:48.212 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0045024840000000002 2024-02-19T21:09:48.212 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.212 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.212 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.212 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.208717+0000", 2024-02-19T21:09:48.212 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7860000000000002e-06 2024-02-19T21:09:48.212 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.212 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.212 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:09:48.212 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.208726+0000", 2024-02-19T21:09:48.212 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.016396628 2024-02-19T21:09:48.212 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.212 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.212 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.213 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.225122+0000", 2024-02-19T21:09:48.213 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013968699999999999 2024-02-19T21:09:48.213 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.213 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.213 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.213 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.225262+0000", 2024-02-19T21:09:48.213 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000145799 2024-02-19T21:09:48.213 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.213 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.213 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,4", 2024-02-19T21:09:48.213 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.225408+0000", 2024-02-19T21:09:48.213 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014860590000000001 2024-02-19T21:09:48.213 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.213 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.214 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.214 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.226894+0000", 2024-02-19T21:09:48.214 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0022827239999999999 2024-02-19T21:09:48.214 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.214 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.214 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.214 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.229176+0000", 2024-02-19T21:09:48.214 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00046320699999999999 2024-02-19T21:09:48.214 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.214 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.214 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:09:48.214 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.229640+0000", 2024-02-19T21:09:48.214 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0414000000000002e-05 2024-02-19T21:09:48.214 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.214 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.215 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.215 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.229690+0000", 2024-02-19T21:09:48.215 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000236868 2024-02-19T21:09:48.215 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.215 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.215 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.215 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.229927+0000", 2024-02-19T21:09:48.215 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025817819999999998 2024-02-19T21:09:48.215 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.215 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.215 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.215 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.215 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.216 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.49412.0:2 176.6 176:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e266)", 2024-02-19T21:09:48.216 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:26.220918+0000", 2024-02-19T21:09:48.216 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 141.890098704, 2024-02-19T21:09:48.216 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.012830304000000001, 2024-02-19T21:09:48.216 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.216 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.216 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.216 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.49412", 2024-02-19T21:09:48.216 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/243193800", 2024-02-19T21:09:48.216 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:09:48.216 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.216 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.216 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.217 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.217 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.220918+0000", 2024-02-19T21:09:48.217 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.217 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.217 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.217 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.217 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.220918+0000", 2024-02-19T21:09:48.217 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:09:48.217 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.217 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.217 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.217 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.220915+0000", 2024-02-19T21:09:48.217 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2437e-05 2024-02-19T21:09:48.217 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.217 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.218 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.218 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.220928+0000", 2024-02-19T21:09:48.218 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0311e-05 2024-02-19T21:09:48.218 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.218 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.218 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.218 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.220938+0000", 2024-02-19T21:09:48.218 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9130000000000003e-06 2024-02-19T21:09:48.218 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.218 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.218 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.218 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.220946+0000", 2024-02-19T21:09:48.218 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00044748299999999998 2024-02-19T21:09:48.219 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.219 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.219 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.219 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.221394+0000", 2024-02-19T21:09:48.219 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.994e-06 2024-02-19T21:09:48.219 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.219 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.219 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:09:48.219 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.221402+0000", 2024-02-19T21:09:48.219 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0068278649999999998 2024-02-19T21:09:48.219 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.219 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.219 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.219 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.228229+0000", 2024-02-19T21:09:48.219 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000122028 2024-02-19T21:09:48.220 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.220 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.220 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.220 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.228351+0000", 2024-02-19T21:09:48.220 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014841299999999999 2024-02-19T21:09:48.220 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.220 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.220 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,6", 2024-02-19T21:09:48.220 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.228500+0000", 2024-02-19T21:09:48.220 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025454869999999999 2024-02-19T21:09:48.220 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.220 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.220 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.220 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.231045+0000", 2024-02-19T21:09:48.221 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000867514 2024-02-19T21:09:48.221 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.221 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.221 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:09:48.221 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.231913+0000", 2024-02-19T21:09:48.221 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001751184 2024-02-19T21:09:48.221 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.221 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.221 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:09:48.221 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.233664+0000", 2024-02-19T21:09:48.221 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5961000000000003e-05 2024-02-19T21:09:48.221 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.221 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.221 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.222 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.233710+0000", 2024-02-19T21:09:48.222 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7960999999999998e-05 2024-02-19T21:09:48.222 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.222 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.222 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.222 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:26.233748+0000", 2024-02-19T21:09:48.222 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.012830304000000001 2024-02-19T21:09:48.222 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.222 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.222 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.222 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.222 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.222 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22848.0:1 192.5s0 192:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e314)", 2024-02-19T21:09:48.222 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:08:14.229643+0000", 2024-02-19T21:09:48.223 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 93.881373206999996, 2024-02-19T21:09:48.223 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4029510890000001, 2024-02-19T21:09:48.223 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.223 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-02-19T21:09:48.223 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:09:48.223 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22848", 2024-02-19T21:09:48.223 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1659723686", 2024-02-19T21:09:48.223 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:09:48.223 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.223 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.223 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.223 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.223 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229643+0000", 2024-02-19T21:09:48.224 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.224 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.224 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.224 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229643+0000", 2024-02-19T21:09:48.224 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:09:48.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.224 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.224 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.224 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229641+0000", 2024-02-19T21:09:48.224 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2750000000000001e-06 2024-02-19T21:09:48.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.224 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.225 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.225 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229647+0000", 2024-02-19T21:09:48.225 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4100000000000008e-06 2024-02-19T21:09:48.225 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.225 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.225 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.225 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229656+0000", 2024-02-19T21:09:48.225 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4850000000000003e-06 2024-02-19T21:09:48.225 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.225 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.225 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.225 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229663+0000", 2024-02-19T21:09:48.225 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0076000000000001e-05 2024-02-19T21:09:48.225 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.225 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.226 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.226 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229703+0000", 2024-02-19T21:09:48.226 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0790000000000001e-06 2024-02-19T21:09:48.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.226 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.226 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:09:48.226 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229707+0000", 2024-02-19T21:09:48.226 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4027761860000001 2024-02-19T21:09:48.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.226 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.226 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.226 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:15.632483+0000", 2024-02-19T21:09:48.226 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011157 2024-02-19T21:09:48.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.226 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.227 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.227 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:15.632594+0000", 2024-02-19T21:09:48.227 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4029510890000001 2024-02-19T21:09:48.227 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.227 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.227 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.227 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.227 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.227 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.25753.0:4348 191.8 e314/308 191:10fb1d02:::199:head v 314'8, mlcod=313'6)", 2024-02-19T21:09:48.227 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:08:14.258303+0000", 2024-02-19T21:09:48.227 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 93.852713003999995, 2024-02-19T21:09:48.227 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.023938170000000002, 2024-02-19T21:09:48.227 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:09:48.227 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:09:48.227 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:09:48.228 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.228 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:09:48.228 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258303+0000", 2024-02-19T21:09:48.228 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:09:48.228 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.228 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.228 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:09:48.228 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258303+0000", 2024-02-19T21:09:48.228 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:09:48.228 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.228 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.228 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:09:48.228 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258302+0000", 2024-02-19T21:09:48.228 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8970000000000008e-06 2024-02-19T21:09:48.229 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.229 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.229 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:09:48.229 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258310+0000", 2024-02-19T21:09:48.229 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0559e-05 2024-02-19T21:09:48.229 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.229 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.229 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:09:48.229 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258321+0000", 2024-02-19T21:09:48.229 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2500000000000003e-06 2024-02-19T21:09:48.229 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.229 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.229 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:09:48.229 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258327+0000", 2024-02-19T21:09:48.229 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7713999999999998e-05 2024-02-19T21:09:48.230 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.230 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.230 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:09:48.230 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258365+0000", 2024-02-19T21:09:48.230 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0503999999999999e-05 2024-02-19T21:09:48.230 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.230 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.230 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:09:48.230 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258385+0000", 2024-02-19T21:09:48.230 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.023829350999999999 2024-02-19T21:09:48.230 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.230 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.230 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:09:48.230 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.282214+0000", 2024-02-19T21:09:48.230 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7092000000000001e-05 2024-02-19T21:09:48.231 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:09:48.231 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:09:48.231 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:09:48.231 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.282242+0000", 2024-02-19T21:09:48.231 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.023938170000000002 2024-02-19T21:09:48.231 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.231 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.231 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.231 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:09:48.231 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:09:48.231 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:09:48.250 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:48.264 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:48.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:48.367 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:48.396 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:48.432 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:09:48.517 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:47.205340+0000 mgr.y (mgr.14550) 360 : cluster [DBG] pgmap v578: 228 pgs: 32 creating+peering, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-19T21:09:48.517 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:47.264510+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e404: 8 total, 8 up, 7 in 2024-02-19T21:09:48.517 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:47.572172+0000 mon.c (mon.2) 387 : audit [DBG] from='client.? v1:172.21.15.163:0/2049860212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:48.664 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:47.205340+0000 mgr.y (mgr.14550) 360 : cluster [DBG] pgmap v578: 228 pgs: 32 creating+peering, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-19T21:09:48.665 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:47.264510+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e404: 8 total, 8 up, 7 in 2024-02-19T21:09:48.665 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:47.572172+0000 mon.c (mon.2) 387 : audit [DBG] from='client.? v1:172.21.15.163:0/2049860212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:48.665 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:47.205340+0000 mgr.y (mgr.14550) 360 : cluster [DBG] pgmap v578: 228 pgs: 32 creating+peering, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-02-19T21:09:48.665 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:47.264510+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e404: 8 total, 8 up, 7 in 2024-02-19T21:09:48.665 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:47.572172+0000 mon.c (mon.2) 387 : audit [DBG] from='client.? v1:172.21.15.163:0/2049860212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:48.825 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-19T21:09:49.090 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:09:49.139 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:49.146 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:49.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:49.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:49.251 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:49.293 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:49.314 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:49.316 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:49.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:49.369 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:48.272161+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e405: 8 total, 8 up, 7 in 2024-02-19T21:09:49.370 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:48.281714+0000 mon.b (mon.1) 573 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:49.370 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:48.282076+0000 mon.a (mon.0) 2490 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:49.370 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:49.253720+0000 mon.a (mon.0) 2491 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:49.370 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:49.253817+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e406: 8 total, 8 up, 7 in 2024-02-19T21:09:49.370 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:48.272161+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e405: 8 total, 8 up, 7 in 2024-02-19T21:09:49.371 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:48.281714+0000 mon.b (mon.1) 573 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:49.371 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:48.282076+0000 mon.a (mon.0) 2490 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:49.371 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:49.253720+0000 mon.a (mon.0) 2491 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:49.371 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:49.253817+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e406: 8 total, 8 up, 7 in 2024-02-19T21:09:49.417 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:49.445 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:49.517 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:49.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:49.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:49.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:48.272161+0000 mon.a (mon.0) 2489 : cluster [DBG] osdmap e405: 8 total, 8 up, 7 in 2024-02-19T21:09:49.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:48.281714+0000 mon.b (mon.1) 573 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:49.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:48.282076+0000 mon.a (mon.0) 2490 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:09:49.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:49.253720+0000 mon.a (mon.0) 2491 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:09:49.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:49.253817+0000 mon.a (mon.0) 2492 : cluster [DBG] osdmap e406: 8 total, 8 up, 7 in 2024-02-19T21:09:49.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:49.648 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:49.846 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:49.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:49.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:49.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.798+0000 7f01f4a76700 1 -- 172.21.15.163:0/3555855028 >> v1:172.21.15.163:6790/0 conn(0x7f01f0100780 legacy=0x7f01f0102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:49.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.798+0000 7f01f4a76700 1 -- 172.21.15.163:0/3555855028 shutdown_connections 2024-02-19T21:09:49.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.798+0000 7f01f4a76700 1 -- 172.21.15.163:0/3555855028 >> 172.21.15.163:0/3555855028 conn(0x7f01f00f35a0 msgr2=0x7f01f00f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:49.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.798+0000 7f01f4a76700 1 -- 172.21.15.163:0/3555855028 shutdown_connections 2024-02-19T21:09:49.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.798+0000 7f01f4a76700 1 -- 172.21.15.163:0/3555855028 wait complete. 2024-02-19T21:09:49.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01f4a76700 1 librados: starting msgr at 2024-02-19T21:09:49.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01f4a76700 1 librados: starting objecter 2024-02-19T21:09:49.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01f4a76700 1 Processor -- start 2024-02-19T21:09:49.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01f4a76700 1 -- start start 2024-02-19T21:09:49.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01f4a76700 1 librados: setting wanted keys 2024-02-19T21:09:49.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01f4a76700 1 librados: calling monclient init 2024-02-19T21:09:49.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01f4a76700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f01f0117500 con 0x7f01f0100780 2024-02-19T21:09:49.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01f4a76700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f01f01186e0 con 0x7f01f00fd980 2024-02-19T21:09:49.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01f4a76700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f01f01198e0 con 0x7f01f01042c0 2024-02-19T21:09:49.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01edd9b700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f01f0100780 0x7f01f00fa410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:35762/0 (socket says 172.21.15.163:35762) 2024-02-19T21:09:49.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01eed9d700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f01f01042c0 0x7f01f0115bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:33256/0 (socket says 172.21.15.163:33256) 2024-02-19T21:09:49.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01edd9b700 1 -- 172.21.15.163:0/3008499132 learned_addr learned my addr 172.21.15.163:0/3008499132 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:49.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01e77fe700 10 client.?.objecter ms_handle_connect 0x7f01f0100780 2024-02-19T21:09:49.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01e77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:49.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01e77fe700 10 client.?.objecter ms_handle_connect 0x7f01f01042c0 2024-02-19T21:09:49.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.799+0000 7f01e77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:49.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 10 client.?.objecter ms_handle_connect 0x7f01f00fd980 2024-02-19T21:09:49.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:49.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193992383 0 0) 0x7f01f01198e0 con 0x7f01f01042c0 2024-02-19T21:09:49.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01dc003630 con 0x7f01f01042c0 2024-02-19T21:09:49.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731249271 0 0) 0x7f01f0117500 con 0x7f01f0100780 2024-02-19T21:09:49.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f01198e0 con 0x7f01f0100780 2024-02-19T21:09:49.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895574717 0 0) 0x7f01f01186e0 con 0x7f01f00fd980 2024-02-19T21:09:49.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f0117500 con 0x7f01f00fd980 2024-02-19T21:09:49.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2062023441 0 0) 0x7f01dc003630 con 0x7f01f01042c0 2024-02-19T21:09:49.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01f01186e0 con 0x7f01f01042c0 2024-02-19T21:09:49.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f01e8001950 con 0x7f01f01042c0 2024-02-19T21:09:49.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 185810971 0 0) 0x7f01f01198e0 con 0x7f01f0100780 2024-02-19T21:09:49.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01dc003630 con 0x7f01f0100780 2024-02-19T21:09:49.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f01d8002f00 con 0x7f01f0100780 2024-02-19T21:09:49.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1148030349 0 0) 0x7f01f0117500 con 0x7f01f00fd980 2024-02-19T21:09:49.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01f01198e0 con 0x7f01f00fd980 2024-02-19T21:09:49.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f01e0001b30 con 0x7f01f00fd980 2024-02-19T21:09:49.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3895830745 0 0) 0x7f01f01186e0 con 0x7f01f01042c0 2024-02-19T21:09:49.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 >> v1:172.21.15.193:6789/0 conn(0x7f01f00fd980 legacy=0x7f01f00f9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:49.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 >> v1:172.21.15.163:6789/0 conn(0x7f01f0100780 legacy=0x7f01f00fa410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:49.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f011aae0 con 0x7f01f01042c0 2024-02-19T21:09:49.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01f4a76700 1 -- 172.21.15.163:0/3008499132 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f0119ad0 con 0x7f01f01042c0 2024-02-19T21:09:49.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01f4a76700 10 client.49787.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:49.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01f4a76700 1 -- 172.21.15.163:0/3008499132 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f0119f40 con 0x7f01f01042c0 2024-02-19T21:09:49.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.800+0000 7f01f4a76700 1 librados: init done 2024-02-19T21:09:49.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.801+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f01e80022c0 con 0x7f01f01042c0 2024-02-19T21:09:49.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.801+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f01e8005960 con 0x7f01f01042c0 2024-02-19T21:09:49.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.802+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f01e80188b0 con 0x7f01f01042c0 2024-02-19T21:09:49.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.802+0000 7f01f4a76700 1 -- 172.21.15.163:0/3008499132 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d0004f20 con 0x7f01f01042c0 2024-02-19T21:09:49.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.802+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(406..406 src has 43..406) v4 ==== 7918+0+0 (unknown 1991030176 0 0) 0x7f01e8079980 con 0x7f01f01042c0 2024-02-19T21:09:49.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.802+0000 7f01e77fe700 10 client.49787.objecter ms_dispatch 0x7f01f0001550 osd_map(406..406 src has 43..406) v4 2024-02-19T21:09:49.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.802+0000 7f01e77fe700 3 client.49787.objecter handle_osd_map got epochs [406,406] > 0 2024-02-19T21:09:49.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.802+0000 7f01e77fe700 3 client.49787.objecter handle_osd_map decoding full epoch 406 2024-02-19T21:09:49.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.802+0000 7f01e77fe700 20 client.49787.objecter dump_active .. 0 homeless 2024-02-19T21:09:49.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.802+0000 7f01e77fe700 10 client.49787.objecter ms_handle_connect 0x7f01dc063cd0 2024-02-19T21:09:49.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.807+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f01e804ce60 con 0x7f01f01042c0 2024-02-19T21:09:49.949 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:09:49.975 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:49.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.981+0000 7f01f4a76700 1 -- 172.21.15.163:0/3008499132 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f01d0000b90 con 0x7f01dc063cd0 2024-02-19T21:09:49.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:49.996+0000 7f01e77fe700 1 -- 172.21.15.163:0/3008499132 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+571661 (unknown 2965378022 0 1474425220) 0x7f01d0000b90 con 0x7f01dc063cd0 2024-02-19T21:09:50.006 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:09:50.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:50.000+0000 7f01f4a76700 10 librados: watch_flush enter 2024-02-19T21:09:50.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:50.000+0000 7f01f4a76700 10 librados: watch_flush exit 2024-02-19T21:09:50.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:50.000+0000 7f01f4a76700 20 client.49787.objecter shutdown clearing up homeless session... 2024-02-19T21:09:50.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:50.000+0000 7f01f4a76700 10 client.49787.objecter successfully canceled tick 2024-02-19T21:09:50.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:50.000+0000 7f01f4a76700 1 -- 172.21.15.163:0/3008499132 >> v1:172.21.15.163:6800/1552514681 conn(0x7f01dc063cd0 legacy=0x7f01dc066190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:50.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:50.000+0000 7f01f4a76700 1 -- 172.21.15.163:0/3008499132 >> v1:172.21.15.163:6790/0 conn(0x7f01f01042c0 legacy=0x7f01f0115bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:50.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:50.000+0000 7f01f4a76700 1 -- 172.21.15.163:0/3008499132 shutdown_connections 2024-02-19T21:09:50.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:50.000+0000 7f01f4a76700 1 -- 172.21.15.163:0/3008499132 >> 172.21.15.163:0/3008499132 conn(0x7f01f00f35a0 msgr2=0x7f01f0107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:50.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:50.000+0000 7f01f4a76700 1 -- 172.21.15.163:0/3008499132 shutdown_connections 2024-02-19T21:09:50.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:50.000+0000 7f01f4a76700 1 -- 172.21.15.163:0/3008499132 wait complete. 2024-02-19T21:09:50.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:50.000+0000 7f01f4a76700 1 librados: shutdown 2024-02-19T21:09:50.012 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:09:50.568 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:09:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:09:50.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:49.205745+0000 mgr.y (mgr.14550) 361 : cluster [DBG] pgmap v581: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:50.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:49.271844+0000 mon.b (mon.1) 574 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:09:50.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:49.274757+0000 mon.b (mon.1) 575 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:09:50.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:49.290337+0000 mon.a (mon.0) 2493 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:09:50.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:50.263148+0000 mon.a (mon.0) 2494 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:09:50.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:50.263225+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e407: 8 total, 8 up, 7 in 2024-02-19T21:09:50.624 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:09:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:50.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:09:50.687 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:09:50.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:49.205745+0000 mgr.y (mgr.14550) 361 : cluster [DBG] pgmap v581: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:50.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:49.271844+0000 mon.b (mon.1) 574 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:09:50.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:49.274757+0000 mon.b (mon.1) 575 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:09:50.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:49.290337+0000 mon.a (mon.0) 2493 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:09:50.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:50.263148+0000 mon.a (mon.0) 2494 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:09:50.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:50.263225+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e407: 8 total, 8 up, 7 in 2024-02-19T21:09:50.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:49.205745+0000 mgr.y (mgr.14550) 361 : cluster [DBG] pgmap v581: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:09:50.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:49.271844+0000 mon.b (mon.1) 574 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:09:50.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:49.274757+0000 mon.b (mon.1) 575 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:09:50.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:49.290337+0000 mon.a (mon.0) 2493 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:09:50.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:50.263148+0000 mon.a (mon.0) 2494 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:09:50.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:50.263225+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e407: 8 total, 8 up, 7 in 2024-02-19T21:09:50.729 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:09:50.759 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:50.831 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:09:50.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:50.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:09:50.834 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:09:50.834 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:09:50.834 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-02-19T21:09:50.834 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:09:50.935 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:50.962 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:51.086 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:51.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:51.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:51.191 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:51.217 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:51.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.368+0000 7fda19e7d700 1 -- 172.21.15.163:0/3360098216 >> v1:172.21.15.163:6790/0 conn(0x7fda14100780 legacy=0x7fda14102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:51.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.368+0000 7fda19e7d700 1 -- 172.21.15.163:0/3360098216 shutdown_connections 2024-02-19T21:09:51.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.368+0000 7fda19e7d700 1 -- 172.21.15.163:0/3360098216 >> 172.21.15.163:0/3360098216 conn(0x7fda140f35a0 msgr2=0x7fda140f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:51.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.368+0000 7fda19e7d700 1 -- 172.21.15.163:0/3360098216 shutdown_connections 2024-02-19T21:09:51.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.368+0000 7fda19e7d700 1 -- 172.21.15.163:0/3360098216 wait complete. 2024-02-19T21:09:51.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.369+0000 7fda19e7d700 1 librados: starting msgr at 2024-02-19T21:09:51.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.369+0000 7fda19e7d700 1 librados: starting objecter 2024-02-19T21:09:51.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.369+0000 7fda19e7d700 1 Processor -- start 2024-02-19T21:09:51.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.369+0000 7fda19e7d700 1 -- start start 2024-02-19T21:09:51.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.369+0000 7fda19e7d700 1 librados: setting wanted keys 2024-02-19T21:09:51.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.369+0000 7fda19e7d700 1 librados: calling monclient init 2024-02-19T21:09:51.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.369+0000 7fda19e7d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fda14117320 con 0x7fda140fd980 2024-02-19T21:09:51.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.369+0000 7fda19e7d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fda14118520 con 0x7fda14100780 2024-02-19T21:09:51.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.369+0000 7fda19e7d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fda14119720 con 0x7fda141042c0 2024-02-19T21:09:51.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.369+0000 7fda137fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fda140fd980 0x7fda140f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:35770/0 (socket says 172.21.15.163:35770) 2024-02-19T21:09:51.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda137fe700 1 -- 172.21.15.163:0/1184515762 learned_addr learned my addr 172.21.15.163:0/1184515762 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:51.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 10 client.?.objecter ms_handle_connect 0x7fda140fd980 2024-02-19T21:09:51.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:51.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 10 client.?.objecter ms_handle_connect 0x7fda14100780 2024-02-19T21:09:51.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:51.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 10 client.?.objecter ms_handle_connect 0x7fda141042c0 2024-02-19T21:09:51.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:51.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721148807 0 0) 0x7fda14118520 con 0x7fda14100780 2024-02-19T21:09:51.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9fc003800 con 0x7fda14100780 2024-02-19T21:09:51.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892989459 0 0) 0x7fda14117320 con 0x7fda140fd980 2024-02-19T21:09:51.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda14118520 con 0x7fda140fd980 2024-02-19T21:09:51.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237870091 0 0) 0x7fda14119720 con 0x7fda141042c0 2024-02-19T21:09:51.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda14117320 con 0x7fda141042c0 2024-02-19T21:09:51.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1198981903 0 0) 0x7fd9fc003800 con 0x7fda14100780 2024-02-19T21:09:51.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda14119720 con 0x7fda14100780 2024-02-19T21:09:51.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fda04002e30 con 0x7fda14100780 2024-02-19T21:09:51.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 763851687 0 0) 0x7fda14118520 con 0x7fda140fd980 2024-02-19T21:09:51.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9fc003800 con 0x7fda140fd980 2024-02-19T21:09:51.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fda08001a70 con 0x7fda140fd980 2024-02-19T21:09:51.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1199619501 0 0) 0x7fda14117320 con 0x7fda141042c0 2024-02-19T21:09:51.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda14118520 con 0x7fda141042c0 2024-02-19T21:09:51.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fda0c001c40 con 0x7fda141042c0 2024-02-19T21:09:51.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4029580667 0 0) 0x7fda14119720 con 0x7fda14100780 2024-02-19T21:09:51.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 >> v1:172.21.15.163:6790/0 conn(0x7fda141042c0 legacy=0x7fda141159e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:51.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 >> v1:172.21.15.163:6789/0 conn(0x7fda140fd980 legacy=0x7fda140f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:51.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1411a920 con 0x7fda14100780 2024-02-19T21:09:51.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda19e7d700 1 -- 172.21.15.163:0/1184515762 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda14118710 con 0x7fda14100780 2024-02-19T21:09:51.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda19e7d700 10 client.48610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:51.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda19e7d700 1 -- 172.21.15.163:0/1184515762 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda14118b80 con 0x7fda14100780 2024-02-19T21:09:51.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda19e7d700 1 librados: init done 2024-02-19T21:09:51.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fda04001070 con 0x7fda14100780 2024-02-19T21:09:51.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.370+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fda04003560 con 0x7fda14100780 2024-02-19T21:09:51.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.371+0000 7fda19e7d700 1 -- 172.21.15.163:0/1184515762 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda00004f20 con 0x7fda14100780 2024-02-19T21:09:51.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.371+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fda040164b0 con 0x7fda14100780 2024-02-19T21:09:51.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.372+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(408..408 src has 43..408) v4 ==== 7918+0+0 (unknown 1784351949 0 0) 0x7fda04076700 con 0x7fda14100780 2024-02-19T21:09:51.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.372+0000 7fda10ff9700 10 client.48610.objecter ms_dispatch 0x7fda14001550 osd_map(408..408 src has 43..408) v4 2024-02-19T21:09:51.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.372+0000 7fda10ff9700 3 client.48610.objecter handle_osd_map got epochs [408,408] > 0 2024-02-19T21:09:51.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.372+0000 7fda10ff9700 3 client.48610.objecter handle_osd_map decoding full epoch 408 2024-02-19T21:09:51.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.372+0000 7fda10ff9700 20 client.48610.objecter dump_active .. 0 homeless 2024-02-19T21:09:51.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.372+0000 7fda10ff9700 10 client.48610.objecter ms_handle_connect 0x7fd9fc063da0 2024-02-19T21:09:51.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.376+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fda04049c40 con 0x7fda14100780 2024-02-19T21:09:51.451 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:49.983068+0000 mgr.y (mgr.14550) 362 : audit [DBG] from='client.49787 v1:172.21.15.163:0/3008499132' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:09:51.451 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:50.371740+0000 mgr.y (mgr.14550) 363 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:51.451 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:51.266971+0000 mon.a (mon.0) 2496 : cluster [DBG] osdmap e408: 8 total, 8 up, 7 in 2024-02-19T21:09:51.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:49.983068+0000 mgr.y (mgr.14550) 362 : audit [DBG] from='client.49787 v1:172.21.15.163:0/3008499132' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:09:51.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:50.371740+0000 mgr.y (mgr.14550) 363 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:51.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:51.266971+0000 mon.a (mon.0) 2496 : cluster [DBG] osdmap e408: 8 total, 8 up, 7 in 2024-02-19T21:09:51.471 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:49.983068+0000 mgr.y (mgr.14550) 362 : audit [DBG] from='client.49787 v1:172.21.15.163:0/3008499132' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:09:51.471 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:50.371740+0000 mgr.y (mgr.14550) 363 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:09:51.471 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:51.266971+0000 mon.a (mon.0) 2496 : cluster [DBG] osdmap e408: 8 total, 8 up, 7 in 2024-02-19T21:09:51.488 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-19T21:09:51.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:51.564+0000 7fda19e7d700 1 -- 172.21.15.163:0/1184515762 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fda000051d0 con 0x7fda14100780 2024-02-19T21:09:51.566 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:51.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:51.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:51.670 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:09:51.696 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:51.715 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:09:51.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:51.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:09:51.767 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:51.792 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":582,"stamp":"2024-02-19T21:09:49.255132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007256,"num_objects":348,"num_object_clones":2,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":1307,"num_read_kb":961,"num_write":1021,"num_write_kb":4533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":7009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2355,"ondisk_log_size":2355,"up":588,"acting":588,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":621,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2190740,"kb_used_data":56872,"kb_used_omap":0,"kb_used_meta":2133760,"kb_avail":654025324,"statfs":{"total":671965249536,"available":669721931776,"internally_reserved":0,"allocated":58236928,"data_stored":38114304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2184970240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.995119"},"pg_stats":[{"pgid":"212.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"212.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:09:48.261871+0000","last_change":"2024-02-19T21:09:48.261871+0000","last_active":"2024-02-19T21:09:48.261871+0000","last_peered":"2024-02-19T21:09:48.261871+0000","last_clean":"2024-02-19T21:09:48.261871+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:09:48.261871+0000","last_undegraded":"2024-02-19T21:09:48.261871+0000","last_fullsized":"2024-02-19T21:09:48.261871+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.2","version":"355'85","reported_seq":170,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.261532+0000","last_change":"2024-02-19T21:08:55.883058+0000","last_active":"2024-02-19T21:09:43.261532+0000","last_peered":"2024-02-19T21:09:43.261532+0000","last_clean":"2024-02-19T21:09:43.261532+0000","last_became_active":"2024-02-19T21:08:04.247600+0000","last_became_peered":"2024-02-19T21:08:04.247600+0000","last_unstale":"2024-02-19T21:09:43.261532+0000","last_undegraded":"2024-02-19T21:09:43.261532+0000","last_fullsized":"2024-02-19T21:09:43.261532+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'85","last_scrub_stamp":"2024-02-19T21:08:55.882988+0000","last_deep_scrub":"355'85","last_deep_scrub_stamp":"2024-02-19T21:08:55.882988+0000","last_clean_scrub_stamp":"2024-02-19T21:08:55.882988+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.3","version":"338'71","reported_seq":401,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258438+0000","last_change":"2024-02-19T21:08:00.867356+0000","last_active":"2024-02-19T21:09:43.258438+0000","last_peered":"2024-02-19T21:09:43.258438+0000","last_clean":"2024-02-19T21:09:43.258438+0000","last_became_active":"2024-02-19T21:04:54.641884+0000","last_became_peered":"2024-02-19T21:04:54.641884+0000","last_unstale":"2024-02-19T21:09:43.258438+0000","last_undegraded":"2024-02-19T21:09:43.258438+0000","last_fullsized":"2024-02-19T21:09:43.258438+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'69","last_scrub_stamp":"2024-02-19T21:08:00.867237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:00.867237+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000068,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":23,"num_write_kb":3929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.0","version":"257'35","reported_seq":348,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.256203+0000","last_change":"2024-02-19T21:08:55.692386+0000","last_active":"2024-02-19T21:09:43.256203+0000","last_peered":"2024-02-19T21:09:43.256203+0000","last_clean":"2024-02-19T21:09:43.256203+0000","last_became_active":"2024-02-19T21:06:40.245157+0000","last_became_peered":"2024-02-19T21:06:40.245157+0000","last_unstale":"2024-02-19T21:09:43.256203+0000","last_undegraded":"2024-02-19T21:09:43.256203+0000","last_fullsized":"2024-02-19T21:09:43.256203+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'35","last_scrub_stamp":"2024-02-19T21:08:55.692313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:55.692313+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1","version":"338'69","reported_seq":453,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.262124+0000","last_change":"2024-02-19T21:08:44.405628+0000","last_active":"2024-02-19T21:09:43.262124+0000","last_peered":"2024-02-19T21:09:43.262124+0000","last_clean":"2024-02-19T21:09:43.262124+0000","last_became_active":"2024-02-19T21:08:41.312701+0000","last_became_peered":"2024-02-19T21:08:41.312701+0000","last_unstale":"2024-02-19T21:09:43.262124+0000","last_undegraded":"2024-02-19T21:09:43.262124+0000","last_fullsized":"2024-02-19T21:09:43.262124+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'68","last_scrub_stamp":"2024-02-19T21:06:56.570995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:06:56.570995+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.6","version":"202'34","reported_seq":366,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.583266+0000","last_change":"2024-02-19T21:08:58.037024+0000","last_active":"2024-02-19T21:09:43.583266+0000","last_peered":"2024-02-19T21:09:43.583266+0000","last_clean":"2024-02-19T21:09:43.583266+0000","last_became_active":"2024-02-19T21:04:54.658542+0000","last_became_peered":"2024-02-19T21:04:54.658542+0000","last_unstale":"2024-02-19T21:09:43.583266+0000","last_undegraded":"2024-02-19T21:09:43.583266+0000","last_fullsized":"2024-02-19T21:09:43.583266+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'34","last_scrub_stamp":"2024-02-19T21:08:58.036955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:58.036955+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.7","version":"396'20","reported_seq":361,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579669+0000","last_change":"2024-02-19T21:08:03.459553+0000","last_active":"2024-02-19T21:09:43.579669+0000","last_peered":"2024-02-19T21:09:43.579669+0000","last_clean":"2024-02-19T21:09:43.579669+0000","last_became_active":"2024-02-19T21:04:54.598061+0000","last_became_peered":"2024-02-19T21:04:54.598061+0000","last_unstale":"2024-02-19T21:09:43.579669+0000","last_undegraded":"2024-02-19T21:09:43.579669+0000","last_fullsized":"2024-02-19T21:09:43.579669+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'18","last_scrub_stamp":"2024-02-19T21:08:03.459480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:03.459480+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.4","version":"206'68","reported_seq":448,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578697+0000","last_change":"2024-02-19T21:08:41.388322+0000","last_active":"2024-02-19T21:09:43.578697+0000","last_peered":"2024-02-19T21:09:43.578697+0000","last_clean":"2024-02-19T21:09:43.578697+0000","last_became_active":"2024-02-19T21:08:41.286273+0000","last_became_peered":"2024-02-19T21:08:41.286273+0000","last_unstale":"2024-02-19T21:09:43.578697+0000","last_undegraded":"2024-02-19T21:09:43.578697+0000","last_fullsized":"2024-02-19T21:09:43.578697+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'68","last_scrub_stamp":"2024-02-19T21:07:35.203992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:07:35.203992+0000","log_size":68,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":20,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.5","version":"338'121","reported_seq":439,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258833+0000","last_change":"2024-02-19T21:08:49.752489+0000","last_active":"2024-02-19T21:09:43.258833+0000","last_peered":"2024-02-19T21:09:43.258833+0000","last_clean":"2024-02-19T21:09:43.258833+0000","last_became_active":"2024-02-19T21:04:54.646842+0000","last_became_peered":"2024-02-19T21:04:54.646842+0000","last_unstale":"2024-02-19T21:09:43.258833+0000","last_undegraded":"2024-02-19T21:09:43.258833+0000","last_fullsized":"2024-02-19T21:09:43.258833+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'121","last_scrub_stamp":"2024-02-19T21:08:49.752418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.752418+0000","log_size":121,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.a","version":"236'22","reported_seq":152,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.267595+0000","last_change":"2024-02-19T21:08:42.397580+0000","last_active":"2024-02-19T21:09:43.267595+0000","last_peered":"2024-02-19T21:09:43.267595+0000","last_clean":"2024-02-19T21:09:43.267595+0000","last_became_active":"2024-02-19T21:08:41.297020+0000","last_became_peered":"2024-02-19T21:08:41.297020+0000","last_unstale":"2024-02-19T21:09:43.267595+0000","last_undegraded":"2024-02-19T21:09:43.267595+0000","last_fullsized":"2024-02-19T21:09:43.267595+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-02-19T21:08:01.607121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:01.607121+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":64,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.b","version":"278'95","reported_seq":420,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258349+0000","last_change":"2024-02-19T21:07:40.251032+0000","last_active":"2024-02-19T21:09:43.258349+0000","last_peered":"2024-02-19T21:09:43.258349+0000","last_clean":"2024-02-19T21:09:43.258349+0000","last_became_active":"2024-02-19T21:04:54.640024+0000","last_became_peered":"2024-02-19T21:04:54.640024+0000","last_unstale":"2024-02-19T21:09:43.258349+0000","last_undegraded":"2024-02-19T21:09:43.258349+0000","last_fullsized":"2024-02-19T21:09:43.258349+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'87","last_scrub_stamp":"2024-02-19T21:07:28.724335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:07:28.724335+0000","log_size":95,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.8","version":"399'39","reported_seq":434,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578360+0000","last_change":"2024-02-19T21:08:06.277510+0000","last_active":"2024-02-19T21:09:43.578360+0000","last_peered":"2024-02-19T21:09:43.578360+0000","last_clean":"2024-02-19T21:09:43.578360+0000","last_became_active":"2024-02-19T21:08:04.266021+0000","last_became_peered":"2024-02-19T21:08:04.266021+0000","last_unstale":"2024-02-19T21:09:43.578360+0000","last_undegraded":"2024-02-19T21:09:43.578360+0000","last_fullsized":"2024-02-19T21:09:43.578360+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'35","last_scrub_stamp":"2024-02-19T21:08:01.494484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:01.494484+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":50,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.9","version":"370'69","reported_seq":402,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.262030+0000","last_change":"2024-02-19T21:07:38.229321+0000","last_active":"2024-02-19T21:09:43.262030+0000","last_peered":"2024-02-19T21:09:43.262030+0000","last_clean":"2024-02-19T21:09:43.262030+0000","last_became_active":"2024-02-19T21:04:54.671646+0000","last_became_peered":"2024-02-19T21:04:54.671646+0000","last_unstale":"2024-02-19T21:09:43.262030+0000","last_undegraded":"2024-02-19T21:09:43.262030+0000","last_fullsized":"2024-02-19T21:09:43.262030+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'68","last_scrub_stamp":"2024-02-19T21:07:31.724135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:07:31.724135+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.e","version":"393'40","reported_seq":412,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579916+0000","last_change":"2024-02-19T21:08:41.400322+0000","last_active":"2024-02-19T21:09:43.579916+0000","last_peered":"2024-02-19T21:09:43.579916+0000","last_clean":"2024-02-19T21:09:43.579916+0000","last_became_active":"2024-02-19T21:08:41.280291+0000","last_became_peered":"2024-02-19T21:08:41.280291+0000","last_unstale":"2024-02-19T21:09:43.579916+0000","last_undegraded":"2024-02-19T21:09:43.579916+0000","last_fullsized":"2024-02-19T21:09:43.579916+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'34","last_scrub_stamp":"2024-02-19T21:06:59.674445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:06:59.674445+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.f","version":"399'37","reported_seq":405,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.480528+0000","last_change":"2024-02-19T21:08:42.378520+0000","last_active":"2024-02-19T21:09:43.480528+0000","last_peered":"2024-02-19T21:09:43.480528+0000","last_clean":"2024-02-19T21:09:43.480528+0000","last_became_active":"2024-02-19T21:08:41.303778+0000","last_became_peered":"2024-02-19T21:08:41.303778+0000","last_unstale":"2024-02-19T21:09:43.480528+0000","last_undegraded":"2024-02-19T21:09:43.480528+0000","last_fullsized":"2024-02-19T21:09:43.480528+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'34","last_scrub_stamp":"2024-02-19T21:06:58.720103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:06:58.720103+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.c","version":"249'73","reported_seq":563,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.581673+0000","last_change":"2024-02-19T21:08:49.613543+0000","last_active":"2024-02-19T21:09:43.581673+0000","last_peered":"2024-02-19T21:09:43.581673+0000","last_clean":"2024-02-19T21:09:43.581673+0000","last_became_active":"2024-02-19T21:08:41.280328+0000","last_became_peered":"2024-02-19T21:08:41.280328+0000","last_unstale":"2024-02-19T21:09:43.581673+0000","last_undegraded":"2024-02-19T21:09:43.581673+0000","last_fullsized":"2024-02-19T21:09:43.581673+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'73","last_scrub_stamp":"2024-02-19T21:08:49.613462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.613462+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.d","version":"393'39","reported_seq":375,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.582873+0000","last_change":"2024-02-19T21:08:12.445740+0000","last_active":"2024-02-19T21:09:43.582873+0000","last_peered":"2024-02-19T21:09:43.582873+0000","last_clean":"2024-02-19T21:09:43.582873+0000","last_became_active":"2024-02-19T21:04:54.644749+0000","last_became_peered":"2024-02-19T21:04:54.644749+0000","last_unstale":"2024-02-19T21:09:43.582873+0000","last_undegraded":"2024-02-19T21:09:43.582873+0000","last_fullsized":"2024-02-19T21:09:43.582873+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'34","last_scrub_stamp":"2024-02-19T21:08:12.445646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.445646+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.12","version":"390'47","reported_seq":459,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.581776+0000","last_change":"2024-02-19T21:08:49.966059+0000","last_active":"2024-02-19T21:09:43.581776+0000","last_peered":"2024-02-19T21:09:43.581776+0000","last_clean":"2024-02-19T21:09:43.581776+0000","last_became_active":"2024-02-19T21:08:04.263085+0000","last_became_peered":"2024-02-19T21:08:04.263085+0000","last_unstale":"2024-02-19T21:09:43.581776+0000","last_undegraded":"2024-02-19T21:09:43.581776+0000","last_fullsized":"2024-02-19T21:09:43.581776+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'45","last_scrub_stamp":"2024-02-19T21:08:49.965991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.965991+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.13","version":"370'71","reported_seq":460,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.481618+0000","last_change":"2024-02-19T21:08:43.370293+0000","last_active":"2024-02-19T21:09:43.481618+0000","last_peered":"2024-02-19T21:09:43.481618+0000","last_clean":"2024-02-19T21:09:43.481618+0000","last_became_active":"2024-02-19T21:08:41.264164+0000","last_became_peered":"2024-02-19T21:08:41.264164+0000","last_unstale":"2024-02-19T21:09:43.481618+0000","last_undegraded":"2024-02-19T21:09:43.481618+0000","last_fullsized":"2024-02-19T21:09:43.481618+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'68","last_scrub_stamp":"2024-02-19T21:08:12.764542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.764542+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":60,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.10","version":"230'45","reported_seq":382,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.480302+0000","last_change":"2024-02-19T21:08:39.942370+0000","last_active":"2024-02-19T21:09:43.480302+0000","last_peered":"2024-02-19T21:09:43.480302+0000","last_clean":"2024-02-19T21:09:43.480302+0000","last_became_active":"2024-02-19T21:04:54.626393+0000","last_became_peered":"2024-02-19T21:04:54.626393+0000","last_unstale":"2024-02-19T21:09:43.480302+0000","last_undegraded":"2024-02-19T21:09:43.480302+0000","last_fullsized":"2024-02-19T21:09:43.480302+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'45","last_scrub_stamp":"2024-02-19T21:08:39.942302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.942302+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.11","version":"338'35","reported_seq":372,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578348+0000","last_change":"2024-02-19T21:08:41.145163+0000","last_active":"2024-02-19T21:09:43.578348+0000","last_peered":"2024-02-19T21:09:43.578348+0000","last_clean":"2024-02-19T21:09:43.578348+0000","last_became_active":"2024-02-19T21:04:54.659096+0000","last_became_peered":"2024-02-19T21:04:54.659096+0000","last_unstale":"2024-02-19T21:09:43.578348+0000","last_undegraded":"2024-02-19T21:09:43.578348+0000","last_fullsized":"2024-02-19T21:09:43.578348+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'35","last_scrub_stamp":"2024-02-19T21:08:41.145092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:41.145092+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.16","version":"396'26","reported_seq":272,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.582723+0000","last_change":"2024-02-19T21:08:41.886431+0000","last_active":"2024-02-19T21:09:43.582723+0000","last_peered":"2024-02-19T21:09:43.582723+0000","last_clean":"2024-02-19T21:09:43.582723+0000","last_became_active":"2024-02-19T21:08:41.269638+0000","last_became_peered":"2024-02-19T21:08:41.269638+0000","last_unstale":"2024-02-19T21:09:43.582723+0000","last_undegraded":"2024-02-19T21:09:43.582723+0000","last_fullsized":"2024-02-19T21:09:43.582723+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-02-19T21:06:57.889852+0000","last_deep_scrub":"237'24","last_deep_scrub_stamp":"2024-02-19T21:06:57.889852+0000","last_clean_scrub_stamp":"2024-02-19T21:06:57.889852+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.17","version":"286'24","reported_seq":396,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.583022+0000","last_change":"2024-02-19T21:08:48.023159+0000","last_active":"2024-02-19T21:09:43.583022+0000","last_peered":"2024-02-19T21:09:43.583022+0000","last_clean":"2024-02-19T21:09:43.583022+0000","last_became_active":"2024-02-19T21:08:41.285673+0000","last_became_peered":"2024-02-19T21:08:41.285673+0000","last_unstale":"2024-02-19T21:09:43.583022+0000","last_undegraded":"2024-02-19T21:09:43.583022+0000","last_fullsized":"2024-02-19T21:09:43.583022+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'24","last_scrub_stamp":"2024-02-19T21:08:48.023079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:48.023079+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":27,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.14","version":"338'52","reported_seq":366,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578604+0000","last_change":"2024-02-19T21:09:28.997669+0000","last_active":"2024-02-19T21:09:43.578604+0000","last_peered":"2024-02-19T21:09:43.578604+0000","last_clean":"2024-02-19T21:09:43.578604+0000","last_became_active":"2024-02-19T21:06:40.227840+0000","last_became_peered":"2024-02-19T21:06:40.227840+0000","last_unstale":"2024-02-19T21:09:43.578604+0000","last_undegraded":"2024-02-19T21:09:43.578604+0000","last_fullsized":"2024-02-19T21:09:43.578604+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'52","last_scrub_stamp":"2024-02-19T21:09:28.997593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.997593+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.15","version":"370'52","reported_seq":510,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.480980+0000","last_change":"2024-02-19T21:08:42.570340+0000","last_active":"2024-02-19T21:09:43.480980+0000","last_peered":"2024-02-19T21:09:43.480980+0000","last_clean":"2024-02-19T21:09:43.480980+0000","last_became_active":"2024-02-19T21:08:41.277044+0000","last_became_peered":"2024-02-19T21:08:41.277044+0000","last_unstale":"2024-02-19T21:09:43.480980+0000","last_undegraded":"2024-02-19T21:09:43.480980+0000","last_fullsized":"2024-02-19T21:09:43.480980+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'51","last_scrub_stamp":"2024-02-19T21:07:28.859888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:07:28.859888+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":99,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1a","version":"338'91","reported_seq":551,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.480957+0000","last_change":"2024-02-19T21:08:06.663074+0000","last_active":"2024-02-19T21:09:43.480957+0000","last_peered":"2024-02-19T21:09:43.480957+0000","last_clean":"2024-02-19T21:09:43.480957+0000","last_became_active":"2024-02-19T21:08:04.265588+0000","last_became_peered":"2024-02-19T21:08:04.265588+0000","last_unstale":"2024-02-19T21:09:43.480957+0000","last_undegraded":"2024-02-19T21:09:43.480957+0000","last_fullsized":"2024-02-19T21:09:43.480957+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'90","last_scrub_stamp":"2024-02-19T21:08:02.861467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:02.861467+0000","log_size":91,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1b","version":"338'52","reported_seq":140,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258905+0000","last_change":"2024-02-19T21:08:04.358505+0000","last_active":"2024-02-19T21:09:43.258905+0000","last_peered":"2024-02-19T21:09:43.258905+0000","last_clean":"2024-02-19T21:09:43.258905+0000","last_became_active":"2024-02-19T21:08:04.262932+0000","last_became_peered":"2024-02-19T21:08:04.262932+0000","last_unstale":"2024-02-19T21:09:43.258905+0000","last_undegraded":"2024-02-19T21:09:43.258905+0000","last_fullsized":"2024-02-19T21:09:43.258905+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'51","last_scrub_stamp":"2024-02-19T21:08:00.485092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:00.485092+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.18","version":"214'102","reported_seq":404,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258819+0000","last_change":"2024-02-19T21:08:50.838459+0000","last_active":"2024-02-19T21:09:43.258819+0000","last_peered":"2024-02-19T21:09:43.258819+0000","last_clean":"2024-02-19T21:09:43.258819+0000","last_became_active":"2024-02-19T21:08:44.236717+0000","last_became_peered":"2024-02-19T21:08:44.236717+0000","last_unstale":"2024-02-19T21:09:43.258819+0000","last_undegraded":"2024-02-19T21:09:43.258819+0000","last_fullsized":"2024-02-19T21:09:43.258819+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-19T21:08:50.838363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:50.838363+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.19","version":"338'69","reported_seq":455,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578879+0000","last_change":"2024-02-19T21:08:43.811586+0000","last_active":"2024-02-19T21:09:43.578879+0000","last_peered":"2024-02-19T21:09:43.578879+0000","last_clean":"2024-02-19T21:09:43.578879+0000","last_became_active":"2024-02-19T21:08:41.292658+0000","last_became_peered":"2024-02-19T21:08:41.292658+0000","last_unstale":"2024-02-19T21:09:43.578879+0000","last_undegraded":"2024-02-19T21:09:43.578879+0000","last_fullsized":"2024-02-19T21:09:43.578879+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-19T21:08:39.466543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.466543+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.1e","version":"338'50","reported_seq":390,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.480392+0000","last_change":"2024-02-19T21:07:38.255436+0000","last_active":"2024-02-19T21:09:43.480392+0000","last_peered":"2024-02-19T21:09:43.480392+0000","last_clean":"2024-02-19T21:09:43.480392+0000","last_became_active":"2024-02-19T21:04:54.629433+0000","last_became_peered":"2024-02-19T21:04:54.629433+0000","last_unstale":"2024-02-19T21:09:43.480392+0000","last_undegraded":"2024-02-19T21:09:43.480392+0000","last_fullsized":"2024-02-19T21:09:43.480392+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'49","last_scrub_stamp":"2024-02-19T21:07:01.787521+0000","last_deep_scrub":"241'49","last_deep_scrub_stamp":"2024-02-19T21:07:01.787521+0000","last_clean_scrub_stamp":"2024-02-19T21:07:01.787521+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":48,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1f","version":"249'44","reported_seq":381,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.583113+0000","last_change":"2024-02-19T21:09:25.989101+0000","last_active":"2024-02-19T21:09:43.583113+0000","last_peered":"2024-02-19T21:09:43.583113+0000","last_clean":"2024-02-19T21:09:43.583113+0000","last_became_active":"2024-02-19T21:04:54.671969+0000","last_became_peered":"2024-02-19T21:04:54.671969+0000","last_unstale":"2024-02-19T21:09:43.583113+0000","last_undegraded":"2024-02-19T21:09:43.583113+0000","last_fullsized":"2024-02-19T21:09:43.583113+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'44","last_scrub_stamp":"2024-02-19T21:09:25.988978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:09:25.988978+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.1c","version":"338'137","reported_seq":559,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.582860+0000","last_change":"2024-02-19T21:08:48.248777+0000","last_active":"2024-02-19T21:09:43.582860+0000","last_peered":"2024-02-19T21:09:43.582860+0000","last_clean":"2024-02-19T21:09:43.582860+0000","last_became_active":"2024-02-19T21:08:48.248009+0000","last_became_peered":"2024-02-19T21:08:48.248009+0000","last_unstale":"2024-02-19T21:09:43.582860+0000","last_undegraded":"2024-02-19T21:09:43.582860+0000","last_fullsized":"2024-02-19T21:09:43.582860+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'136","last_scrub_stamp":"2024-02-19T21:08:37.523303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:37.523303+0000","log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":96,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":41,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":96,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.1d","version":"370'36","reported_seq":374,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258745+0000","last_change":"2024-02-19T21:08:56.661364+0000","last_active":"2024-02-19T21:09:43.258745+0000","last_peered":"2024-02-19T21:09:43.258745+0000","last_clean":"2024-02-19T21:09:43.258745+0000","last_became_active":"2024-02-19T21:04:54.655665+0000","last_became_peered":"2024-02-19T21:04:54.655665+0000","last_unstale":"2024-02-19T21:09:43.258745+0000","last_undegraded":"2024-02-19T21:09:43.258745+0000","last_fullsized":"2024-02-19T21:09:43.258745+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'35","last_scrub_stamp":"2024-02-19T21:08:56.661296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:56.661296+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"344'10","reported_seq":489,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-02-19T21:09:44.268868+0000","last_change":"2024-02-19T21:09:42.967840+0000","last_active":"2024-02-19T21:09:44.268868+0000","last_peered":"2024-02-19T21:09:44.268868+0000","last_clean":"2024-02-19T21:09:44.268868+0000","last_became_active":"2024-02-19T21:08:41.307616+0000","last_became_peered":"2024-02-19T21:08:41.307616+0000","last_unstale":"2024-02-19T21:09:44.268868+0000","last_undegraded":"2024-02-19T21:09:44.268868+0000","last_fullsized":"2024-02-19T21:09:44.268868+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'10","last_scrub_stamp":"2024-02-19T21:09:42.967771+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:09:42.967771+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":391,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578405+0000","last_change":"2024-02-19T21:08:51.616197+0000","last_active":"2024-02-19T21:09:43.578405+0000","last_peered":"2024-02-19T21:09:43.578405+0000","last_clean":"2024-02-19T21:09:43.578405+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:09:43.578405+0000","last_undegraded":"2024-02-19T21:09:43.578405+0000","last_fullsized":"2024-02-19T21:09:43.578405+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:08:51.616123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:51.616123+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":387,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-02-19T21:09:44.922372+0000","last_change":"2024-02-19T21:09:44.922372+0000","last_active":"2024-02-19T21:09:44.922372+0000","last_peered":"2024-02-19T21:09:44.922372+0000","last_clean":"2024-02-19T21:09:44.922372+0000","last_became_active":"2024-02-19T21:08:04.239474+0000","last_became_peered":"2024-02-19T21:08:04.239474+0000","last_unstale":"2024-02-19T21:09:44.922372+0000","last_undegraded":"2024-02-19T21:09:44.922372+0000","last_fullsized":"2024-02-19T21:09:44.922372+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:44.922303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_clean_scrub_stamp":"2024-02-19T21:09:44.922303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":359,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.581414+0000","last_change":"2024-02-19T21:09:27.878844+0000","last_active":"2024-02-19T21:09:43.581414+0000","last_peered":"2024-02-19T21:09:43.581414+0000","last_clean":"2024-02-19T21:09:43.581414+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:09:43.581414+0000","last_undegraded":"2024-02-19T21:09:43.581414+0000","last_fullsized":"2024-02-19T21:09:43.581414+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.878760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.878760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":312,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.582588+0000","last_change":"2024-02-19T21:06:46.312127+0000","last_active":"2024-02-19T21:09:43.582588+0000","last_peered":"2024-02-19T21:09:43.582588+0000","last_clean":"2024-02-19T21:09:43.582588+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:09:43.582588+0000","last_undegraded":"2024-02-19T21:09:43.582588+0000","last_fullsized":"2024-02-19T21:09:43.582588+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:46.312025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.312025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":242,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258715+0000","last_change":"2024-02-19T21:08:53.783361+0000","last_active":"2024-02-19T21:09:43.258715+0000","last_peered":"2024-02-19T21:09:43.258715+0000","last_clean":"2024-02-19T21:09:43.258715+0000","last_became_active":"2024-02-19T21:08:41.314093+0000","last_became_peered":"2024-02-19T21:08:41.314093+0000","last_unstale":"2024-02-19T21:09:43.258715+0000","last_undegraded":"2024-02-19T21:09:43.258715+0000","last_fullsized":"2024-02-19T21:09:43.258715+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:08:53.783292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:53.783292+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":396,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.259400+0000","last_change":"2024-02-19T21:08:41.309125+0000","last_active":"2024-02-19T21:09:43.259400+0000","last_peered":"2024-02-19T21:09:43.259400+0000","last_clean":"2024-02-19T21:09:43.259400+0000","last_became_active":"2024-02-19T21:08:41.302799+0000","last_became_peered":"2024-02-19T21:08:41.302799+0000","last_unstale":"2024-02-19T21:09:43.259400+0000","last_undegraded":"2024-02-19T21:09:43.259400+0000","last_fullsized":"2024-02-19T21:09:43.259400+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:39.700315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.700315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":387,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.581860+0000","last_change":"2024-02-19T21:08:41.277873+0000","last_active":"2024-02-19T21:09:43.581860+0000","last_peered":"2024-02-19T21:09:43.581860+0000","last_clean":"2024-02-19T21:09:43.581860+0000","last_became_active":"2024-02-19T21:08:41.276911+0000","last_became_peered":"2024-02-19T21:08:41.276911+0000","last_unstale":"2024-02-19T21:09:43.581860+0000","last_undegraded":"2024-02-19T21:09:43.581860+0000","last_fullsized":"2024-02-19T21:09:43.581860+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:27.303307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:27.303307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":196,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.480485+0000","last_change":"2024-02-19T21:09:34.791174+0000","last_active":"2024-02-19T21:09:43.480485+0000","last_peered":"2024-02-19T21:09:43.480485+0000","last_clean":"2024-02-19T21:09:43.480485+0000","last_became_active":"2024-02-19T21:06:40.227822+0000","last_became_peered":"2024-02-19T21:06:40.227822+0000","last_unstale":"2024-02-19T21:09:43.480485+0000","last_undegraded":"2024-02-19T21:09:43.480485+0000","last_fullsized":"2024-02-19T21:09:43.480485+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:34.791063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:34.791063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":313,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579822+0000","last_change":"2024-02-19T21:07:00.449622+0000","last_active":"2024-02-19T21:09:43.579822+0000","last_peered":"2024-02-19T21:09:43.579822+0000","last_clean":"2024-02-19T21:09:43.579822+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:09:43.579822+0000","last_undegraded":"2024-02-19T21:09:43.579822+0000","last_fullsized":"2024-02-19T21:09:43.579822+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:00.449534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:00.449534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":396,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.580201+0000","last_change":"2024-02-19T21:09:34.573994+0000","last_active":"2024-02-19T21:09:43.580201+0000","last_peered":"2024-02-19T21:09:43.580201+0000","last_clean":"2024-02-19T21:09:43.580201+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:09:43.580201+0000","last_undegraded":"2024-02-19T21:09:43.580201+0000","last_fullsized":"2024-02-19T21:09:43.580201+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:09:34.573889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:34.573889+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":369,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.582301+0000","last_change":"2024-02-19T21:09:30.950696+0000","last_active":"2024-02-19T21:09:43.582301+0000","last_peered":"2024-02-19T21:09:43.582301+0000","last_clean":"2024-02-19T21:09:43.582301+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:09:43.582301+0000","last_undegraded":"2024-02-19T21:09:43.582301+0000","last_fullsized":"2024-02-19T21:09:43.582301+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:30.950626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:09:30.950626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":360,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579384+0000","last_change":"2024-02-19T21:09:28.031934+0000","last_active":"2024-02-19T21:09:43.579384+0000","last_peered":"2024-02-19T21:09:43.579384+0000","last_clean":"2024-02-19T21:09:43.579384+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:09:43.579384+0000","last_undegraded":"2024-02-19T21:09:43.579384+0000","last_fullsized":"2024-02-19T21:09:43.579384+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:28.031872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.031872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":376,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.481454+0000","last_change":"2024-02-19T21:08:53.964809+0000","last_active":"2024-02-19T21:09:43.481454+0000","last_peered":"2024-02-19T21:09:43.481454+0000","last_clean":"2024-02-19T21:09:43.481454+0000","last_became_active":"2024-02-19T21:08:04.265081+0000","last_became_peered":"2024-02-19T21:08:04.265081+0000","last_unstale":"2024-02-19T21:09:43.481454+0000","last_undegraded":"2024-02-19T21:09:43.481454+0000","last_fullsized":"2024-02-19T21:09:43.481454+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_clean_scrub_stamp":"2024-02-19T21:08:53.964703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":327,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.984187+0000","last_change":"2024-02-19T21:09:43.984159+0000","last_active":"2024-02-19T21:09:43.984187+0000","last_peered":"2024-02-19T21:09:43.984187+0000","last_clean":"2024-02-19T21:09:43.984187+0000","last_became_active":"2024-02-19T21:08:41.304395+0000","last_became_peered":"2024-02-19T21:08:41.304395+0000","last_unstale":"2024-02-19T21:09:43.984187+0000","last_undegraded":"2024-02-19T21:09:43.984187+0000","last_fullsized":"2024-02-19T21:09:43.984187+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:43.984079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:09:43.984079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":133,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.580173+0000","last_change":"2024-02-19T21:09:41.955664+0000","last_active":"2024-02-19T21:09:43.580173+0000","last_peered":"2024-02-19T21:09:43.580173+0000","last_clean":"2024-02-19T21:09:43.580173+0000","last_became_active":"2024-02-19T21:08:41.286244+0000","last_became_peered":"2024-02-19T21:08:41.286244+0000","last_unstale":"2024-02-19T21:09:43.580173+0000","last_undegraded":"2024-02-19T21:09:43.580173+0000","last_fullsized":"2024-02-19T21:09:43.580173+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:09:41.955563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:41.955563+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":383,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.481857+0000","last_change":"2024-02-19T21:08:41.367028+0000","last_active":"2024-02-19T21:09:43.481857+0000","last_peered":"2024-02-19T21:09:43.481857+0000","last_clean":"2024-02-19T21:09:43.481857+0000","last_became_active":"2024-02-19T21:08:41.273961+0000","last_became_peered":"2024-02-19T21:08:41.273961+0000","last_unstale":"2024-02-19T21:09:43.481857+0000","last_undegraded":"2024-02-19T21:09:43.481857+0000","last_fullsized":"2024-02-19T21:09:43.481857+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:06:50.694248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:50.694248+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":374,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.263498+0000","last_change":"2024-02-19T21:09:36.685164+0000","last_active":"2024-02-19T21:09:43.263498+0000","last_peered":"2024-02-19T21:09:43.263498+0000","last_clean":"2024-02-19T21:09:43.263498+0000","last_became_active":"2024-02-19T21:08:41.302537+0000","last_became_peered":"2024-02-19T21:08:41.302537+0000","last_unstale":"2024-02-19T21:09:43.263498+0000","last_undegraded":"2024-02-19T21:09:43.263498+0000","last_fullsized":"2024-02-19T21:09:43.263498+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:36.685063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:36.685063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":385,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.583374+0000","last_change":"2024-02-19T21:08:41.278598+0000","last_active":"2024-02-19T21:09:43.583374+0000","last_peered":"2024-02-19T21:09:43.583374+0000","last_clean":"2024-02-19T21:09:43.583374+0000","last_became_active":"2024-02-19T21:08:41.278298+0000","last_became_peered":"2024-02-19T21:08:41.278298+0000","last_unstale":"2024-02-19T21:09:43.583374+0000","last_undegraded":"2024-02-19T21:09:43.583374+0000","last_fullsized":"2024-02-19T21:09:43.583374+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:13.136532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:06:13.136532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":298,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.260510+0000","last_change":"2024-02-19T21:08:41.296517+0000","last_active":"2024-02-19T21:09:43.260510+0000","last_peered":"2024-02-19T21:09:43.260510+0000","last_clean":"2024-02-19T21:09:43.260510+0000","last_became_active":"2024-02-19T21:08:41.296219+0000","last_became_peered":"2024-02-19T21:08:41.296219+0000","last_unstale":"2024-02-19T21:09:43.260510+0000","last_undegraded":"2024-02-19T21:09:43.260510+0000","last_fullsized":"2024-02-19T21:09:43.260510+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.878060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.878060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"82'15","reported_seq":392,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.267932+0000","last_change":"2024-02-19T21:08:42.367657+0000","last_active":"2024-02-19T21:09:43.267932+0000","last_peered":"2024-02-19T21:09:43.267932+0000","last_clean":"2024-02-19T21:09:43.267932+0000","last_became_active":"2024-02-19T21:08:41.295931+0000","last_became_peered":"2024-02-19T21:08:41.295931+0000","last_unstale":"2024-02-19T21:09:43.267932+0000","last_undegraded":"2024-02-19T21:09:43.267932+0000","last_fullsized":"2024-02-19T21:09:43.267932+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:06:28.746972+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:06:28.746972+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":382,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.260600+0000","last_change":"2024-02-19T21:08:04.269175+0000","last_active":"2024-02-19T21:09:43.260600+0000","last_peered":"2024-02-19T21:09:43.260600+0000","last_clean":"2024-02-19T21:09:43.260600+0000","last_became_active":"2024-02-19T21:08:04.268406+0000","last_became_peered":"2024-02-19T21:08:04.268406+0000","last_unstale":"2024-02-19T21:09:43.260600+0000","last_undegraded":"2024-02-19T21:09:43.260600+0000","last_fullsized":"2024-02-19T21:09:43.260600+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:14.687938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.687938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":470,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-02-19T21:09:44.269759+0000","last_change":"2024-02-19T21:06:12.177597+0000","last_active":"2024-02-19T21:09:44.269759+0000","last_peered":"2024-02-19T21:09:44.269759+0000","last_clean":"2024-02-19T21:09:44.269759+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:09:44.269759+0000","last_undegraded":"2024-02-19T21:09:44.269759+0000","last_fullsized":"2024-02-19T21:09:44.269759+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:12.177524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:12.177524+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":71,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.257536+0000","last_change":"2024-02-19T21:08:41.300913+0000","last_active":"2024-02-19T21:09:43.257536+0000","last_peered":"2024-02-19T21:09:43.257536+0000","last_clean":"2024-02-19T21:09:43.257536+0000","last_became_active":"2024-02-19T21:08:41.300707+0000","last_became_peered":"2024-02-19T21:08:41.300707+0000","last_unstale":"2024-02-19T21:09:43.257536+0000","last_undegraded":"2024-02-19T21:09:43.257536+0000","last_fullsized":"2024-02-19T21:09:43.257536+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.704447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.704447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":112,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258461+0000","last_change":"2024-02-19T21:08:36.847613+0000","last_active":"2024-02-19T21:09:43.258461+0000","last_peered":"2024-02-19T21:09:43.258461+0000","last_clean":"2024-02-19T21:09:43.258461+0000","last_became_active":"2024-02-19T21:08:04.242244+0000","last_became_peered":"2024-02-19T21:08:04.242244+0000","last_unstale":"2024-02-19T21:09:43.258461+0000","last_undegraded":"2024-02-19T21:09:43.258461+0000","last_fullsized":"2024-02-19T21:09:43.258461+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:36.847548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.847548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":462,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.256258+0000","last_change":"2024-02-19T21:08:43.178829+0000","last_active":"2024-02-19T21:09:43.256258+0000","last_peered":"2024-02-19T21:09:43.256258+0000","last_clean":"2024-02-19T21:09:43.256258+0000","last_became_active":"2024-02-19T21:08:41.311383+0000","last_became_peered":"2024-02-19T21:08:41.311383+0000","last_unstale":"2024-02-19T21:09:43.256258+0000","last_undegraded":"2024-02-19T21:09:43.256258+0000","last_fullsized":"2024-02-19T21:09:43.256258+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:07:59.834282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:07:59.834282+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":409,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.262488+0000","last_change":"2024-02-19T21:08:04.271314+0000","last_active":"2024-02-19T21:09:43.262488+0000","last_peered":"2024-02-19T21:09:43.262488+0000","last_clean":"2024-02-19T21:09:43.262488+0000","last_became_active":"2024-02-19T21:08:04.270601+0000","last_became_peered":"2024-02-19T21:08:04.270601+0000","last_unstale":"2024-02-19T21:09:43.262488+0000","last_undegraded":"2024-02-19T21:09:43.262488+0000","last_fullsized":"2024-02-19T21:09:43.262488+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:07.874362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:07.874362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"344'9","reported_seq":107,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-02-19T21:09:44.265734+0000","last_change":"2024-02-19T21:08:42.577019+0000","last_active":"2024-02-19T21:09:44.265734+0000","last_peered":"2024-02-19T21:09:44.265734+0000","last_clean":"2024-02-19T21:09:44.265734+0000","last_became_active":"2024-02-19T21:08:41.293491+0000","last_became_peered":"2024-02-19T21:08:41.293491+0000","last_unstale":"2024-02-19T21:09:44.265734+0000","last_undegraded":"2024-02-19T21:09:44.265734+0000","last_fullsized":"2024-02-19T21:09:44.265734+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:20.710731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:20.710731+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":364,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.583372+0000","last_change":"2024-02-19T21:09:31.856861+0000","last_active":"2024-02-19T21:09:43.583372+0000","last_peered":"2024-02-19T21:09:43.583372+0000","last_clean":"2024-02-19T21:09:43.583372+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:09:43.583372+0000","last_undegraded":"2024-02-19T21:09:43.583372+0000","last_fullsized":"2024-02-19T21:09:43.583372+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:31.856784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:31.856784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":313,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579147+0000","last_change":"2024-02-19T21:06:51.423833+0000","last_active":"2024-02-19T21:09:43.579147+0000","last_peered":"2024-02-19T21:09:43.579147+0000","last_clean":"2024-02-19T21:09:43.579147+0000","last_became_active":"2024-02-19T21:04:39.657052+0000","last_became_peered":"2024-02-19T21:04:39.657052+0000","last_unstale":"2024-02-19T21:09:43.579147+0000","last_undegraded":"2024-02-19T21:09:43.579147+0000","last_fullsized":"2024-02-19T21:09:43.579147+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.423769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.423769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":118,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.580113+0000","last_change":"2024-02-19T21:08:49.181165+0000","last_active":"2024-02-19T21:09:43.580113+0000","last_peered":"2024-02-19T21:09:43.580113+0000","last_clean":"2024-02-19T21:09:43.580113+0000","last_became_active":"2024-02-19T21:08:41.278810+0000","last_became_peered":"2024-02-19T21:08:41.278810+0000","last_unstale":"2024-02-19T21:09:43.580113+0000","last_undegraded":"2024-02-19T21:09:43.580113+0000","last_fullsized":"2024-02-19T21:09:43.580113+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:49.181100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.181100+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":396,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.481768+0000","last_change":"2024-02-19T21:09:37.870271+0000","last_active":"2024-02-19T21:09:43.481768+0000","last_peered":"2024-02-19T21:09:43.481768+0000","last_clean":"2024-02-19T21:09:43.481768+0000","last_became_active":"2024-02-19T21:07:32.227557+0000","last_became_peered":"2024-02-19T21:07:32.227557+0000","last_unstale":"2024-02-19T21:09:43.481768+0000","last_undegraded":"2024-02-19T21:09:43.481768+0000","last_fullsized":"2024-02-19T21:09:43.481768+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.870218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.870218+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":370,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.580129+0000","last_change":"2024-02-19T21:08:41.293711+0000","last_active":"2024-02-19T21:09:43.580129+0000","last_peered":"2024-02-19T21:09:43.580129+0000","last_clean":"2024-02-19T21:09:43.580129+0000","last_became_active":"2024-02-19T21:08:41.292701+0000","last_became_peered":"2024-02-19T21:08:41.292701+0000","last_unstale":"2024-02-19T21:09:43.580129+0000","last_undegraded":"2024-02-19T21:09:43.580129+0000","last_fullsized":"2024-02-19T21:09:43.580129+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:53.428093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:53.428093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":107,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.268179+0000","last_change":"2024-02-19T21:08:04.279238+0000","last_active":"2024-02-19T21:09:43.268179+0000","last_peered":"2024-02-19T21:09:43.268179+0000","last_clean":"2024-02-19T21:09:43.268179+0000","last_became_active":"2024-02-19T21:08:04.279008+0000","last_became_peered":"2024-02-19T21:08:04.279008+0000","last_unstale":"2024-02-19T21:09:43.268179+0000","last_undegraded":"2024-02-19T21:09:43.268179+0000","last_fullsized":"2024-02-19T21:09:43.268179+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:05.721175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:05.721175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":107,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.261614+0000","last_change":"2024-02-19T21:08:04.280833+0000","last_active":"2024-02-19T21:09:43.261614+0000","last_peered":"2024-02-19T21:09:43.261614+0000","last_clean":"2024-02-19T21:09:43.261614+0000","last_became_active":"2024-02-19T21:08:04.280514+0000","last_became_peered":"2024-02-19T21:08:04.280514+0000","last_unstale":"2024-02-19T21:09:43.261614+0000","last_undegraded":"2024-02-19T21:09:43.261614+0000","last_fullsized":"2024-02-19T21:09:43.261614+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:58.499841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:58.499841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"344'11","reported_seq":152,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-02-19T21:09:44.264059+0000","last_change":"2024-02-19T21:09:39.597074+0000","last_active":"2024-02-19T21:09:44.264059+0000","last_peered":"2024-02-19T21:09:44.264059+0000","last_clean":"2024-02-19T21:09:44.264059+0000","last_became_active":"2024-02-19T21:08:41.296797+0000","last_became_peered":"2024-02-19T21:08:41.296797+0000","last_unstale":"2024-02-19T21:09:44.264059+0000","last_undegraded":"2024-02-19T21:09:44.264059+0000","last_fullsized":"2024-02-19T21:09:44.264059+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'11","last_scrub_stamp":"2024-02-19T21:09:39.597001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:39.597001+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":120,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579081+0000","last_change":"2024-02-19T21:08:41.921037+0000","last_active":"2024-02-19T21:09:43.579081+0000","last_peered":"2024-02-19T21:09:43.579081+0000","last_clean":"2024-02-19T21:09:43.579081+0000","last_became_active":"2024-02-19T21:08:41.268914+0000","last_became_peered":"2024-02-19T21:08:41.268914+0000","last_unstale":"2024-02-19T21:09:43.579081+0000","last_undegraded":"2024-02-19T21:09:43.579081+0000","last_fullsized":"2024-02-19T21:09:43.579081+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:06:46.813814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.813814+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":394,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.583643+0000","last_change":"2024-02-19T21:09:38.541487+0000","last_active":"2024-02-19T21:09:43.583643+0000","last_peered":"2024-02-19T21:09:43.583643+0000","last_clean":"2024-02-19T21:09:43.583643+0000","last_became_active":"2024-02-19T21:07:36.263299+0000","last_became_peered":"2024-02-19T21:07:36.263299+0000","last_unstale":"2024-02-19T21:09:43.583643+0000","last_undegraded":"2024-02-19T21:09:43.583643+0000","last_fullsized":"2024-02-19T21:09:43.583643+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:38.541390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:38.541390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":360,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.482162+0000","last_change":"2024-02-19T21:07:25.812691+0000","last_active":"2024-02-19T21:09:43.482162+0000","last_peered":"2024-02-19T21:09:43.482162+0000","last_clean":"2024-02-19T21:09:43.482162+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:09:43.482162+0000","last_undegraded":"2024-02-19T21:09:43.482162+0000","last_fullsized":"2024-02-19T21:09:43.482162+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:25.812627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:07:25.812627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":190,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.259618+0000","last_change":"2024-02-19T21:06:40.245744+0000","last_active":"2024-02-19T21:09:43.259618+0000","last_peered":"2024-02-19T21:09:43.259618+0000","last_clean":"2024-02-19T21:09:43.259618+0000","last_became_active":"2024-02-19T21:06:40.245385+0000","last_became_peered":"2024-02-19T21:06:40.245385+0000","last_unstale":"2024-02-19T21:09:43.259618+0000","last_undegraded":"2024-02-19T21:09:43.259618+0000","last_fullsized":"2024-02-19T21:09:43.259618+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:16.780601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.780601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":341,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.259996+0000","last_change":"2024-02-19T21:08:41.270754+0000","last_active":"2024-02-19T21:09:43.259996+0000","last_peered":"2024-02-19T21:09:43.259996+0000","last_clean":"2024-02-19T21:09:43.259996+0000","last_became_active":"2024-02-19T21:08:41.270483+0000","last_became_peered":"2024-02-19T21:08:41.270483+0000","last_unstale":"2024-02-19T21:09:43.259996+0000","last_undegraded":"2024-02-19T21:09:43.259996+0000","last_fullsized":"2024-02-19T21:09:43.259996+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:12.606585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.606585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":125,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258726+0000","last_change":"2024-02-19T21:08:04.343458+0000","last_active":"2024-02-19T21:09:43.258726+0000","last_peered":"2024-02-19T21:09:43.258726+0000","last_clean":"2024-02-19T21:09:43.258726+0000","last_became_active":"2024-02-19T21:08:04.242466+0000","last_became_peered":"2024-02-19T21:08:04.242466+0000","last_unstale":"2024-02-19T21:09:43.258726+0000","last_undegraded":"2024-02-19T21:09:43.258726+0000","last_fullsized":"2024-02-19T21:09:43.258726+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:06:46.816121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.816121+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":400,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.257383+0000","last_change":"2024-02-19T21:08:41.294617+0000","last_active":"2024-02-19T21:09:43.257383+0000","last_peered":"2024-02-19T21:09:43.257383+0000","last_clean":"2024-02-19T21:09:43.257383+0000","last_became_active":"2024-02-19T21:08:41.294281+0000","last_became_peered":"2024-02-19T21:08:41.294281+0000","last_unstale":"2024-02-19T21:09:43.257383+0000","last_undegraded":"2024-02-19T21:09:43.257383+0000","last_fullsized":"2024-02-19T21:09:43.257383+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":344,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.261357+0000","last_change":"2024-02-19T21:09:40.549754+0000","last_active":"2024-02-19T21:09:43.261357+0000","last_peered":"2024-02-19T21:09:43.261357+0000","last_clean":"2024-02-19T21:09:43.261357+0000","last_became_active":"2024-02-19T21:08:41.266934+0000","last_became_peered":"2024-02-19T21:08:41.266934+0000","last_unstale":"2024-02-19T21:09:43.261357+0000","last_undegraded":"2024-02-19T21:09:43.261357+0000","last_fullsized":"2024-02-19T21:09:43.261357+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:40.549690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.549690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":107,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.268427+0000","last_change":"2024-02-19T21:08:04.267005+0000","last_active":"2024-02-19T21:09:43.268427+0000","last_peered":"2024-02-19T21:09:43.268427+0000","last_clean":"2024-02-19T21:09:43.268427+0000","last_became_active":"2024-02-19T21:08:04.266538+0000","last_became_peered":"2024-02-19T21:08:04.266538+0000","last_unstale":"2024-02-19T21:09:43.268427+0000","last_undegraded":"2024-02-19T21:09:43.268427+0000","last_fullsized":"2024-02-19T21:09:43.268427+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:58.847915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:58.847915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":363,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.580388+0000","last_change":"2024-02-19T21:07:00.676713+0000","last_active":"2024-02-19T21:09:43.580388+0000","last_peered":"2024-02-19T21:09:43.580388+0000","last_clean":"2024-02-19T21:09:43.580388+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:09:43.580388+0000","last_undegraded":"2024-02-19T21:09:43.580388+0000","last_fullsized":"2024-02-19T21:09:43.580388+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:00.676634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:07:00.676634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":239,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579431+0000","last_change":"2024-02-19T21:06:40.224213+0000","last_active":"2024-02-19T21:09:43.579431+0000","last_peered":"2024-02-19T21:09:43.579431+0000","last_clean":"2024-02-19T21:09:43.579431+0000","last_became_active":"2024-02-19T21:06:40.223768+0000","last_became_peered":"2024-02-19T21:06:40.223768+0000","last_unstale":"2024-02-19T21:09:43.579431+0000","last_undegraded":"2024-02-19T21:09:43.579431+0000","last_fullsized":"2024-02-19T21:09:43.579431+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:06:10.826889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:10.826889+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":407,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.481886+0000","last_change":"2024-02-19T21:09:35.824601+0000","last_active":"2024-02-19T21:09:43.481886+0000","last_peered":"2024-02-19T21:09:43.481886+0000","last_clean":"2024-02-19T21:09:43.481886+0000","last_became_active":"2024-02-19T21:08:41.303606+0000","last_became_peered":"2024-02-19T21:08:41.303606+0000","last_unstale":"2024-02-19T21:09:43.481886+0000","last_undegraded":"2024-02-19T21:09:43.481886+0000","last_fullsized":"2024-02-19T21:09:43.481886+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:35.824520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:35.824520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":358,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.262658+0000","last_change":"2024-02-19T21:08:57.653280+0000","last_active":"2024-02-19T21:09:43.262658+0000","last_peered":"2024-02-19T21:09:43.262658+0000","last_clean":"2024-02-19T21:09:43.262658+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:09:43.262658+0000","last_undegraded":"2024-02-19T21:09:43.262658+0000","last_fullsized":"2024-02-19T21:09:43.262658+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:57.653204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:08:57.653204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":312,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.583617+0000","last_change":"2024-02-19T21:08:12.309008+0000","last_active":"2024-02-19T21:09:43.583617+0000","last_peered":"2024-02-19T21:09:43.583617+0000","last_clean":"2024-02-19T21:09:43.583617+0000","last_became_active":"2024-02-19T21:04:39.718926+0000","last_became_peered":"2024-02-19T21:04:39.718926+0000","last_unstale":"2024-02-19T21:09:43.583617+0000","last_undegraded":"2024-02-19T21:09:43.583617+0000","last_fullsized":"2024-02-19T21:09:43.583617+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:12.308951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.308951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":361,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.580263+0000","last_change":"2024-02-19T21:08:11.431726+0000","last_active":"2024-02-19T21:09:43.580263+0000","last_peered":"2024-02-19T21:09:43.580263+0000","last_clean":"2024-02-19T21:09:43.580263+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:09:43.580263+0000","last_undegraded":"2024-02-19T21:09:43.580263+0000","last_fullsized":"2024-02-19T21:09:43.580263+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:11.431637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:11.431637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":225,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258111+0000","last_change":"2024-02-19T21:09:30.704618+0000","last_active":"2024-02-19T21:09:43.258111+0000","last_peered":"2024-02-19T21:09:43.258111+0000","last_clean":"2024-02-19T21:09:43.258111+0000","last_became_active":"2024-02-19T21:06:40.240321+0000","last_became_peered":"2024-02-19T21:06:40.240321+0000","last_unstale":"2024-02-19T21:09:43.258111+0000","last_undegraded":"2024-02-19T21:09:43.258111+0000","last_fullsized":"2024-02-19T21:09:43.258111+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:09:30.704543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:30.704543+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"397'2","reported_seq":404,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.481547+0000","last_change":"2024-02-19T21:09:40.934534+0000","last_active":"2024-02-19T21:09:43.481547+0000","last_peered":"2024-02-19T21:09:43.481547+0000","last_clean":"2024-02-19T21:09:43.481547+0000","last_became_active":"2024-02-19T21:08:10.225480+0000","last_became_peered":"2024-02-19T21:08:10.225480+0000","last_unstale":"2024-02-19T21:09:43.481547+0000","last_undegraded":"2024-02-19T21:09:43.481547+0000","last_fullsized":"2024-02-19T21:09:43.481547+0000","mapping_epoch":57,"log_start":"58'1","ondisk_log_start":"58'1","created":57,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.934474+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":176,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.260435+0000","last_change":"2024-02-19T21:08:41.277088+0000","last_active":"2024-02-19T21:09:43.260435+0000","last_peered":"2024-02-19T21:09:43.260435+0000","last_clean":"2024-02-19T21:09:43.260435+0000","last_became_active":"2024-02-19T21:08:41.276642+0000","last_became_peered":"2024-02-19T21:08:41.276642+0000","last_unstale":"2024-02-19T21:09:43.260435+0000","last_undegraded":"2024-02-19T21:09:43.260435+0000","last_fullsized":"2024-02-19T21:09:43.260435+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:35.542752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:08:35.542752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":376,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.263583+0000","last_change":"2024-02-19T21:08:04.282673+0000","last_active":"2024-02-19T21:09:43.263583+0000","last_peered":"2024-02-19T21:09:43.263583+0000","last_clean":"2024-02-19T21:09:43.263583+0000","last_became_active":"2024-02-19T21:08:04.282243+0000","last_became_peered":"2024-02-19T21:08:04.282243+0000","last_unstale":"2024-02-19T21:09:43.263583+0000","last_undegraded":"2024-02-19T21:09:43.263583+0000","last_fullsized":"2024-02-19T21:09:43.263583+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.627708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.627708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":312,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579125+0000","last_change":"2024-02-19T21:07:34.550279+0000","last_active":"2024-02-19T21:09:43.579125+0000","last_peered":"2024-02-19T21:09:43.579125+0000","last_clean":"2024-02-19T21:09:43.579125+0000","last_became_active":"2024-02-19T21:04:39.662125+0000","last_became_peered":"2024-02-19T21:04:39.662125+0000","last_unstale":"2024-02-19T21:09:43.579125+0000","last_undegraded":"2024-02-19T21:09:43.579125+0000","last_fullsized":"2024-02-19T21:09:43.579125+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:34.550217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:34.550217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":360,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.483035+0000","last_change":"2024-02-19T21:06:51.666781+0000","last_active":"2024-02-19T21:09:43.483035+0000","last_peered":"2024-02-19T21:09:43.483035+0000","last_clean":"2024-02-19T21:09:43.483035+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:09:43.483035+0000","last_undegraded":"2024-02-19T21:09:43.483035+0000","last_fullsized":"2024-02-19T21:09:43.483035+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.666708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.666708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":407,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.581758+0000","last_change":"2024-02-19T21:08:41.312655+0000","last_active":"2024-02-19T21:09:43.581758+0000","last_peered":"2024-02-19T21:09:43.581758+0000","last_clean":"2024-02-19T21:09:43.581758+0000","last_became_active":"2024-02-19T21:08:41.263010+0000","last_became_peered":"2024-02-19T21:08:41.263010+0000","last_unstale":"2024-02-19T21:09:43.581758+0000","last_undegraded":"2024-02-19T21:09:43.581758+0000","last_fullsized":"2024-02-19T21:09:43.581758+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:06:26.776963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:26.776963+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":373,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.481636+0000","last_change":"2024-02-19T21:08:37.935245+0000","last_active":"2024-02-19T21:09:43.481636+0000","last_peered":"2024-02-19T21:09:43.481636+0000","last_clean":"2024-02-19T21:09:43.481636+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:09:43.481636+0000","last_undegraded":"2024-02-19T21:09:43.481636+0000","last_fullsized":"2024-02-19T21:09:43.481636+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:37.935175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:08:37.935175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":392,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579351+0000","last_change":"2024-02-19T21:07:58.239278+0000","last_active":"2024-02-19T21:09:43.579351+0000","last_peered":"2024-02-19T21:09:43.579351+0000","last_clean":"2024-02-19T21:09:43.579351+0000","last_became_active":"2024-02-19T21:07:58.238944+0000","last_became_peered":"2024-02-19T21:07:58.238944+0000","last_unstale":"2024-02-19T21:09:43.579351+0000","last_undegraded":"2024-02-19T21:09:43.579351+0000","last_fullsized":"2024-02-19T21:09:43.579351+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:17.907318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:17.907318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":112,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.261690+0000","last_change":"2024-02-19T21:08:53.820016+0000","last_active":"2024-02-19T21:09:43.261690+0000","last_peered":"2024-02-19T21:09:43.261690+0000","last_clean":"2024-02-19T21:09:43.261690+0000","last_became_active":"2024-02-19T21:08:04.278265+0000","last_became_peered":"2024-02-19T21:08:04.278265+0000","last_unstale":"2024-02-19T21:09:43.261690+0000","last_undegraded":"2024-02-19T21:09:43.261690+0000","last_fullsized":"2024-02-19T21:09:43.261690+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:53.819949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:53.819949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":386,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.580758+0000","last_change":"2024-02-19T21:08:41.306530+0000","last_active":"2024-02-19T21:09:43.580758+0000","last_peered":"2024-02-19T21:09:43.580758+0000","last_clean":"2024-02-19T21:09:43.580758+0000","last_became_active":"2024-02-19T21:08:41.305847+0000","last_became_peered":"2024-02-19T21:08:41.305847+0000","last_unstale":"2024-02-19T21:09:43.580758+0000","last_undegraded":"2024-02-19T21:09:43.580758+0000","last_fullsized":"2024-02-19T21:09:43.580758+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:16.863398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.863398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":313,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578955+0000","last_change":"2024-02-19T21:06:52.403019+0000","last_active":"2024-02-19T21:09:43.578955+0000","last_peered":"2024-02-19T21:09:43.578955+0000","last_clean":"2024-02-19T21:09:43.578955+0000","last_became_active":"2024-02-19T21:04:39.666739+0000","last_became_peered":"2024-02-19T21:04:39.666739+0000","last_unstale":"2024-02-19T21:09:43.578955+0000","last_undegraded":"2024-02-19T21:09:43.578955+0000","last_fullsized":"2024-02-19T21:09:43.578955+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:52.402941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:52.402941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":360,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.583542+0000","last_change":"2024-02-19T21:06:23.273349+0000","last_active":"2024-02-19T21:09:43.583542+0000","last_peered":"2024-02-19T21:09:43.583542+0000","last_clean":"2024-02-19T21:09:43.583542+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:09:43.583542+0000","last_undegraded":"2024-02-19T21:09:43.583542+0000","last_fullsized":"2024-02-19T21:09:43.583542+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:23.273274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:23.273274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":378,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.583540+0000","last_change":"2024-02-19T21:07:01.264194+0000","last_active":"2024-02-19T21:09:43.583540+0000","last_peered":"2024-02-19T21:09:43.583540+0000","last_clean":"2024-02-19T21:09:43.583540+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:09:43.583540+0000","last_undegraded":"2024-02-19T21:09:43.583540+0000","last_fullsized":"2024-02-19T21:09:43.583540+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:07:01.264093+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:07:01.264093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":410,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578737+0000","last_change":"2024-02-19T21:09:37.955966+0000","last_active":"2024-02-19T21:09:43.578737+0000","last_peered":"2024-02-19T21:09:43.578737+0000","last_clean":"2024-02-19T21:09:43.578737+0000","last_became_active":"2024-02-19T21:08:06.473207+0000","last_became_peered":"2024-02-19T21:08:06.473207+0000","last_unstale":"2024-02-19T21:09:43.578737+0000","last_undegraded":"2024-02-19T21:09:43.578737+0000","last_fullsized":"2024-02-19T21:09:43.578737+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.955872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.955872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":868,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-02-19T21:09:44.262610+0000","last_change":"2024-02-19T21:06:10.791107+0000","last_active":"2024-02-19T21:09:44.262610+0000","last_peered":"2024-02-19T21:09:44.262610+0000","last_clean":"2024-02-19T21:09:44.262610+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:09:44.262610+0000","last_undegraded":"2024-02-19T21:09:44.262610+0000","last_fullsized":"2024-02-19T21:09:44.262610+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:06:10.791036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:10.791036+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":501,"num_read_kb":496,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":376,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.481880+0000","last_change":"2024-02-19T21:08:04.265242+0000","last_active":"2024-02-19T21:09:43.481880+0000","last_peered":"2024-02-19T21:09:43.481880+0000","last_clean":"2024-02-19T21:09:43.481880+0000","last_became_active":"2024-02-19T21:08:04.264896+0000","last_became_peered":"2024-02-19T21:08:04.264896+0000","last_unstale":"2024-02-19T21:09:43.481880+0000","last_undegraded":"2024-02-19T21:09:43.481880+0000","last_fullsized":"2024-02-19T21:09:43.481880+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.690596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.690596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":312,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258982+0000","last_change":"2024-02-19T21:08:12.584947+0000","last_active":"2024-02-19T21:09:43.258982+0000","last_peered":"2024-02-19T21:09:43.258982+0000","last_clean":"2024-02-19T21:09:43.258982+0000","last_became_active":"2024-02-19T21:04:39.691434+0000","last_became_peered":"2024-02-19T21:04:39.691434+0000","last_unstale":"2024-02-19T21:09:43.258982+0000","last_undegraded":"2024-02-19T21:09:43.258982+0000","last_fullsized":"2024-02-19T21:09:43.258982+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:12.584874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.584874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":204,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.259170+0000","last_change":"2024-02-19T21:08:41.309414+0000","last_active":"2024-02-19T21:09:43.259170+0000","last_peered":"2024-02-19T21:09:43.259170+0000","last_clean":"2024-02-19T21:09:43.259170+0000","last_became_active":"2024-02-19T21:08:41.303115+0000","last_became_peered":"2024-02-19T21:08:41.303115+0000","last_unstale":"2024-02-19T21:09:43.259170+0000","last_undegraded":"2024-02-19T21:09:43.259170+0000","last_fullsized":"2024-02-19T21:09:43.259170+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:36.716464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.716464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"304'30","reported_seq":574,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-02-19T21:09:44.264558+0000","last_change":"2024-02-19T21:08:05.292869+0000","last_active":"2024-02-19T21:09:44.264558+0000","last_peered":"2024-02-19T21:09:44.264558+0000","last_clean":"2024-02-19T21:09:44.264558+0000","last_became_active":"2024-02-19T21:08:04.266322+0000","last_became_peered":"2024-02-19T21:08:04.266322+0000","last_unstale":"2024-02-19T21:09:44.264558+0000","last_undegraded":"2024-02-19T21:09:44.264558+0000","last_fullsized":"2024-02-19T21:09:44.264558+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-19T21:05:42.927640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:42.927640+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":392,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.581551+0000","last_change":"2024-02-19T21:08:02.435986+0000","last_active":"2024-02-19T21:09:43.581551+0000","last_peered":"2024-02-19T21:09:43.581551+0000","last_clean":"2024-02-19T21:09:43.581551+0000","last_became_active":"2024-02-19T21:08:02.435663+0000","last_became_peered":"2024-02-19T21:08:02.435663+0000","last_unstale":"2024-02-19T21:09:43.581551+0000","last_undegraded":"2024-02-19T21:09:43.581551+0000","last_fullsized":"2024-02-19T21:09:43.581551+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:06:19.845394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":116,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.267391+0000","last_change":"2024-02-19T21:08:41.272198+0000","last_active":"2024-02-19T21:09:43.267391+0000","last_peered":"2024-02-19T21:09:43.267391+0000","last_clean":"2024-02-19T21:09:43.267391+0000","last_became_active":"2024-02-19T21:08:41.271603+0000","last_became_peered":"2024-02-19T21:08:41.271603+0000","last_unstale":"2024-02-19T21:09:43.267391+0000","last_undegraded":"2024-02-19T21:09:43.267391+0000","last_fullsized":"2024-02-19T21:09:43.267391+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:26.886817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:07:26.886817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":312,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579102+0000","last_change":"2024-02-19T21:06:54.355870+0000","last_active":"2024-02-19T21:09:43.579102+0000","last_peered":"2024-02-19T21:09:43.579102+0000","last_clean":"2024-02-19T21:09:43.579102+0000","last_became_active":"2024-02-19T21:04:39.721416+0000","last_became_peered":"2024-02-19T21:04:39.721416+0000","last_unstale":"2024-02-19T21:09:43.579102+0000","last_undegraded":"2024-02-19T21:09:43.579102+0000","last_fullsized":"2024-02-19T21:09:43.579102+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.355809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.355809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"220'9","reported_seq":248,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-02-19T21:09:44.264178+0000","last_change":"2024-02-19T21:06:40.228012+0000","last_active":"2024-02-19T21:09:44.264178+0000","last_peered":"2024-02-19T21:09:44.264178+0000","last_clean":"2024-02-19T21:09:44.264178+0000","last_became_active":"2024-02-19T21:06:40.227604+0000","last_became_peered":"2024-02-19T21:06:40.227604+0000","last_unstale":"2024-02-19T21:09:44.264178+0000","last_undegraded":"2024-02-19T21:09:44.264178+0000","last_fullsized":"2024-02-19T21:09:44.264178+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:21.819105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:21.819105+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":398,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.260689+0000","last_change":"2024-02-19T21:09:28.659562+0000","last_active":"2024-02-19T21:09:43.260689+0000","last_peered":"2024-02-19T21:09:43.260689+0000","last_clean":"2024-02-19T21:09:43.260689+0000","last_became_active":"2024-02-19T21:08:41.306070+0000","last_became_peered":"2024-02-19T21:08:41.306070+0000","last_unstale":"2024-02-19T21:09:43.260689+0000","last_undegraded":"2024-02-19T21:09:43.260689+0000","last_fullsized":"2024-02-19T21:09:43.260689+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:09:28.659495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.659495+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":409,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.263205+0000","last_change":"2024-02-19T21:07:54.235439+0000","last_active":"2024-02-19T21:09:43.263205+0000","last_peered":"2024-02-19T21:09:43.263205+0000","last_clean":"2024-02-19T21:09:43.263205+0000","last_became_active":"2024-02-19T21:07:54.235080+0000","last_became_peered":"2024-02-19T21:07:54.235080+0000","last_unstale":"2024-02-19T21:09:43.263205+0000","last_undegraded":"2024-02-19T21:09:43.263205+0000","last_fullsized":"2024-02-19T21:09:43.263205+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:47.885024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:05:47.885024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":112,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258639+0000","last_change":"2024-02-19T21:09:27.747880+0000","last_active":"2024-02-19T21:09:43.258639+0000","last_peered":"2024-02-19T21:09:43.258639+0000","last_clean":"2024-02-19T21:09:43.258639+0000","last_became_active":"2024-02-19T21:08:04.244317+0000","last_became_peered":"2024-02-19T21:08:04.244317+0000","last_unstale":"2024-02-19T21:09:43.258639+0000","last_undegraded":"2024-02-19T21:09:43.258639+0000","last_fullsized":"2024-02-19T21:09:43.258639+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.747789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.747789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":107,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.267529+0000","last_change":"2024-02-19T21:08:04.261805+0000","last_active":"2024-02-19T21:09:43.267529+0000","last_peered":"2024-02-19T21:09:43.267529+0000","last_clean":"2024-02-19T21:09:43.267529+0000","last_became_active":"2024-02-19T21:08:04.261408+0000","last_became_peered":"2024-02-19T21:08:04.261408+0000","last_unstale":"2024-02-19T21:09:43.267529+0000","last_undegraded":"2024-02-19T21:09:43.267529+0000","last_fullsized":"2024-02-19T21:09:43.267529+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:58.726858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:58.726858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":360,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578896+0000","last_change":"2024-02-19T21:06:20.194149+0000","last_active":"2024-02-19T21:09:43.578896+0000","last_peered":"2024-02-19T21:09:43.578896+0000","last_clean":"2024-02-19T21:09:43.578896+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:09:43.578896+0000","last_undegraded":"2024-02-19T21:09:43.578896+0000","last_fullsized":"2024-02-19T21:09:43.578896+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:20.194018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:06:20.194018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":118,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.262398+0000","last_change":"2024-02-19T21:08:42.355747+0000","last_active":"2024-02-19T21:09:43.262398+0000","last_peered":"2024-02-19T21:09:43.262398+0000","last_clean":"2024-02-19T21:09:43.262398+0000","last_became_active":"2024-02-19T21:08:41.294283+0000","last_became_peered":"2024-02-19T21:08:41.294283+0000","last_unstale":"2024-02-19T21:09:43.262398+0000","last_undegraded":"2024-02-19T21:09:43.262398+0000","last_fullsized":"2024-02-19T21:09:43.262398+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:06:48.844301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.844301+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":130,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.480934+0000","last_change":"2024-02-19T21:08:41.279224+0000","last_active":"2024-02-19T21:09:43.480934+0000","last_peered":"2024-02-19T21:09:43.480934+0000","last_clean":"2024-02-19T21:09:43.480934+0000","last_became_active":"2024-02-19T21:08:41.269102+0000","last_became_peered":"2024-02-19T21:08:41.269102+0000","last_unstale":"2024-02-19T21:09:43.480934+0000","last_undegraded":"2024-02-19T21:09:43.480934+0000","last_fullsized":"2024-02-19T21:09:43.480934+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:07:33.765035+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:07:33.765035+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":112,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258984+0000","last_change":"2024-02-19T21:08:57.881002+0000","last_active":"2024-02-19T21:09:43.258984+0000","last_peered":"2024-02-19T21:09:43.258984+0000","last_clean":"2024-02-19T21:09:43.258984+0000","last_became_active":"2024-02-19T21:08:04.264593+0000","last_became_peered":"2024-02-19T21:08:04.264593+0000","last_unstale":"2024-02-19T21:09:43.258984+0000","last_undegraded":"2024-02-19T21:09:43.258984+0000","last_fullsized":"2024-02-19T21:09:43.258984+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:57.880935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:08:57.880935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":312,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.580261+0000","last_change":"2024-02-19T21:08:03.177937+0000","last_active":"2024-02-19T21:09:43.580261+0000","last_peered":"2024-02-19T21:09:43.580261+0000","last_clean":"2024-02-19T21:09:43.580261+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:09:43.580261+0000","last_undegraded":"2024-02-19T21:09:43.580261+0000","last_fullsized":"2024-02-19T21:09:43.580261+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:03.177867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:03.177867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":117,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.267373+0000","last_change":"2024-02-19T21:08:06.478479+0000","last_active":"2024-02-19T21:09:43.267373+0000","last_peered":"2024-02-19T21:09:43.267373+0000","last_clean":"2024-02-19T21:09:43.267373+0000","last_became_active":"2024-02-19T21:08:04.258740+0000","last_became_peered":"2024-02-19T21:08:04.258740+0000","last_unstale":"2024-02-19T21:09:43.267373+0000","last_undegraded":"2024-02-19T21:09:43.267373+0000","last_fullsized":"2024-02-19T21:09:43.267373+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:06:09.793208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.793208+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":364,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578810+0000","last_change":"2024-02-19T21:08:11.307651+0000","last_active":"2024-02-19T21:09:43.578810+0000","last_peered":"2024-02-19T21:09:43.578810+0000","last_clean":"2024-02-19T21:09:43.578810+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:09:43.578810+0000","last_undegraded":"2024-02-19T21:09:43.578810+0000","last_fullsized":"2024-02-19T21:09:43.578810+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:11.307561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:11.307561+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":107,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.261195+0000","last_change":"2024-02-19T21:08:04.265732+0000","last_active":"2024-02-19T21:09:43.261195+0000","last_peered":"2024-02-19T21:09:43.261195+0000","last_clean":"2024-02-19T21:09:43.261195+0000","last_became_active":"2024-02-19T21:08:04.264972+0000","last_became_peered":"2024-02-19T21:08:04.264972+0000","last_unstale":"2024-02-19T21:09:43.261195+0000","last_undegraded":"2024-02-19T21:09:43.261195+0000","last_fullsized":"2024-02-19T21:09:43.261195+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:56.709750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:06:56.709750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":372,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.584277+0000","last_change":"2024-02-19T21:08:48.995892+0000","last_active":"2024-02-19T21:09:43.584277+0000","last_peered":"2024-02-19T21:09:43.584277+0000","last_clean":"2024-02-19T21:09:43.584277+0000","last_became_active":"2024-02-19T21:08:41.286874+0000","last_became_peered":"2024-02-19T21:08:41.286874+0000","last_unstale":"2024-02-19T21:09:43.584277+0000","last_undegraded":"2024-02-19T21:09:43.584277+0000","last_fullsized":"2024-02-19T21:09:43.584277+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:48.995828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:08:48.995828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":71,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.260278+0000","last_change":"2024-02-19T21:08:41.311611+0000","last_active":"2024-02-19T21:09:43.260278+0000","last_peered":"2024-02-19T21:09:43.260278+0000","last_clean":"2024-02-19T21:09:43.260278+0000","last_became_active":"2024-02-19T21:08:41.311298+0000","last_became_peered":"2024-02-19T21:08:41.311298+0000","last_unstale":"2024-02-19T21:09:43.260278+0000","last_undegraded":"2024-02-19T21:09:43.260278+0000","last_fullsized":"2024-02-19T21:09:43.260278+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:21.704719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:21.704719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":115,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.261171+0000","last_change":"2024-02-19T21:08:04.356074+0000","last_active":"2024-02-19T21:09:43.261171+0000","last_peered":"2024-02-19T21:09:43.261171+0000","last_clean":"2024-02-19T21:09:43.261171+0000","last_became_active":"2024-02-19T21:08:04.264472+0000","last_became_peered":"2024-02-19T21:08:04.264472+0000","last_unstale":"2024-02-19T21:09:43.261171+0000","last_undegraded":"2024-02-19T21:09:43.261171+0000","last_fullsized":"2024-02-19T21:09:43.261171+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:06:48.855268+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.855268+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":376,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.582239+0000","last_change":"2024-02-19T21:09:41.519382+0000","last_active":"2024-02-19T21:09:43.582239+0000","last_peered":"2024-02-19T21:09:43.582239+0000","last_clean":"2024-02-19T21:09:43.582239+0000","last_became_active":"2024-02-19T21:08:41.295743+0000","last_became_peered":"2024-02-19T21:08:41.295743+0000","last_unstale":"2024-02-19T21:09:43.582239+0000","last_undegraded":"2024-02-19T21:09:43.582239+0000","last_fullsized":"2024-02-19T21:09:43.582239+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:41.519294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:41.519294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":381,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.582890+0000","last_change":"2024-02-19T21:08:56.601041+0000","last_active":"2024-02-19T21:09:43.582890+0000","last_peered":"2024-02-19T21:09:43.582890+0000","last_clean":"2024-02-19T21:09:43.582890+0000","last_became_active":"2024-02-19T21:08:04.265573+0000","last_became_peered":"2024-02-19T21:08:04.265573+0000","last_unstale":"2024-02-19T21:09:43.582890+0000","last_undegraded":"2024-02-19T21:09:43.582890+0000","last_fullsized":"2024-02-19T21:09:43.582890+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:56.600951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:56.600951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":372,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.481661+0000","last_change":"2024-02-19T21:09:27.770579+0000","last_active":"2024-02-19T21:09:43.481661+0000","last_peered":"2024-02-19T21:09:43.481661+0000","last_clean":"2024-02-19T21:09:43.481661+0000","last_became_active":"2024-02-19T21:08:41.268962+0000","last_became_peered":"2024-02-19T21:08:41.268962+0000","last_unstale":"2024-02-19T21:09:43.481661+0000","last_undegraded":"2024-02-19T21:09:43.481661+0000","last_fullsized":"2024-02-19T21:09:43.481661+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.770501+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":322,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579067+0000","last_change":"2024-02-19T21:08:50.600747+0000","last_active":"2024-02-19T21:09:43.579067+0000","last_peered":"2024-02-19T21:09:43.579067+0000","last_clean":"2024-02-19T21:09:43.579067+0000","last_became_active":"2024-02-19T21:08:41.292646+0000","last_became_peered":"2024-02-19T21:08:41.292646+0000","last_unstale":"2024-02-19T21:09:43.579067+0000","last_undegraded":"2024-02-19T21:09:43.579067+0000","last_fullsized":"2024-02-19T21:09:43.579067+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:50.600659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:50.600659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":119,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.268275+0000","last_change":"2024-02-19T21:08:04.361409+0000","last_active":"2024-02-19T21:09:43.268275+0000","last_peered":"2024-02-19T21:09:43.268275+0000","last_clean":"2024-02-19T21:09:43.268275+0000","last_became_active":"2024-02-19T21:08:04.267216+0000","last_became_peered":"2024-02-19T21:08:04.267216+0000","last_unstale":"2024-02-19T21:09:43.268275+0000","last_undegraded":"2024-02-19T21:09:43.268275+0000","last_fullsized":"2024-02-19T21:09:43.268275+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:07:01.731068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:07:01.731068+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":388,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.583450+0000","last_change":"2024-02-19T21:09:40.986717+0000","last_active":"2024-02-19T21:09:43.583450+0000","last_peered":"2024-02-19T21:09:43.583450+0000","last_clean":"2024-02-19T21:09:43.583450+0000","last_became_active":"2024-02-19T21:08:04.266728+0000","last_became_peered":"2024-02-19T21:08:04.266728+0000","last_unstale":"2024-02-19T21:09:43.583450+0000","last_undegraded":"2024-02-19T21:09:43.583450+0000","last_fullsized":"2024-02-19T21:09:43.583450+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.986635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":471,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-02-19T21:09:44.265873+0000","last_change":"2024-02-19T21:09:26.916067+0000","last_active":"2024-02-19T21:09:44.265873+0000","last_peered":"2024-02-19T21:09:44.265873+0000","last_clean":"2024-02-19T21:09:44.265873+0000","last_became_active":"2024-02-19T21:01:20.288799+0000","last_became_peered":"2024-02-19T21:01:20.288799+0000","last_unstale":"2024-02-19T21:09:44.265873+0000","last_undegraded":"2024-02-19T21:09:44.265873+0000","last_fullsized":"2024-02-19T21:09:44.265873+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:09:26.916002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:26.916002+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":231,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.480554+0000","last_change":"2024-02-19T21:08:57.007906+0000","last_active":"2024-02-19T21:09:43.480554+0000","last_peered":"2024-02-19T21:09:43.480554+0000","last_clean":"2024-02-19T21:09:43.480554+0000","last_became_active":"2024-02-19T21:08:04.266747+0000","last_became_peered":"2024-02-19T21:08:04.266747+0000","last_unstale":"2024-02-19T21:09:43.480554+0000","last_undegraded":"2024-02-19T21:09:43.480554+0000","last_fullsized":"2024-02-19T21:09:43.480554+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:08:57.007815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:08:57.007815+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":193,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578430+0000","last_change":"2024-02-19T21:08:55.202543+0000","last_active":"2024-02-19T21:09:43.578430+0000","last_peered":"2024-02-19T21:09:43.578430+0000","last_clean":"2024-02-19T21:09:43.578430+0000","last_became_active":"2024-02-19T21:06:40.219590+0000","last_became_peered":"2024-02-19T21:06:40.219590+0000","last_unstale":"2024-02-19T21:09:43.578430+0000","last_undegraded":"2024-02-19T21:09:43.578430+0000","last_fullsized":"2024-02-19T21:09:43.578430+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:55.202463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:55.202463+0000","last_clean_scrub_stamp":"2024-02-19T21:08:55.202463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":417,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.583091+0000","last_change":"2024-02-19T21:08:04.681714+0000","last_active":"2024-02-19T21:09:43.583091+0000","last_peered":"2024-02-19T21:09:43.583091+0000","last_clean":"2024-02-19T21:09:43.583091+0000","last_became_active":"2024-02-19T21:08:04.266103+0000","last_became_peered":"2024-02-19T21:08:04.266103+0000","last_unstale":"2024-02-19T21:09:43.583091+0000","last_undegraded":"2024-02-19T21:09:43.583091+0000","last_fullsized":"2024-02-19T21:09:43.583091+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:06:47.541012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.541012+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":181,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258714+0000","last_change":"2024-02-19T21:08:41.280409+0000","last_active":"2024-02-19T21:09:43.258714+0000","last_peered":"2024-02-19T21:09:43.258714+0000","last_clean":"2024-02-19T21:09:43.258714+0000","last_became_active":"2024-02-19T21:08:41.279782+0000","last_became_peered":"2024-02-19T21:08:41.279782+0000","last_unstale":"2024-02-19T21:09:43.258714+0000","last_undegraded":"2024-02-19T21:09:43.258714+0000","last_fullsized":"2024-02-19T21:09:43.258714+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:30.769388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:07:30.769388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":137,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-02-19T21:09:44.267017+0000","last_change":"2024-02-19T21:08:37.867477+0000","last_active":"2024-02-19T21:09:44.267017+0000","last_peered":"2024-02-19T21:09:44.267017+0000","last_clean":"2024-02-19T21:09:44.267017+0000","last_became_active":"2024-02-19T21:08:04.256045+0000","last_became_peered":"2024-02-19T21:08:04.256045+0000","last_unstale":"2024-02-19T21:09:44.267017+0000","last_undegraded":"2024-02-19T21:09:44.267017+0000","last_fullsized":"2024-02-19T21:09:44.267017+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'10","last_scrub_stamp":"2024-02-19T21:08:37.867405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:37.867405+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":75,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.936239+0000","last_change":"2024-02-19T21:09:43.936218+0000","last_active":"2024-02-19T21:09:43.936239+0000","last_peered":"2024-02-19T21:09:43.936239+0000","last_clean":"2024-02-19T21:09:43.936239+0000","last_became_active":"2024-02-19T21:08:41.286185+0000","last_became_peered":"2024-02-19T21:08:41.286185+0000","last_unstale":"2024-02-19T21:09:43.936239+0000","last_undegraded":"2024-02-19T21:09:43.936239+0000","last_fullsized":"2024-02-19T21:09:43.936239+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_clean_scrub_stamp":"2024-02-19T21:09:43.936153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":325,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578515+0000","last_change":"2024-02-19T21:06:49.305197+0000","last_active":"2024-02-19T21:09:43.578515+0000","last_peered":"2024-02-19T21:09:43.578515+0000","last_clean":"2024-02-19T21:09:43.578515+0000","last_became_active":"2024-02-19T21:04:39.760167+0000","last_became_peered":"2024-02-19T21:04:39.760167+0000","last_unstale":"2024-02-19T21:09:43.578515+0000","last_undegraded":"2024-02-19T21:09:43.578515+0000","last_fullsized":"2024-02-19T21:09:43.578515+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-02-19T21:06:49.305118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.305118+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":99,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.260134+0000","last_change":"2024-02-19T21:09:40.676522+0000","last_active":"2024-02-19T21:09:43.260134+0000","last_peered":"2024-02-19T21:09:43.260134+0000","last_clean":"2024-02-19T21:09:43.260134+0000","last_became_active":"2024-02-19T21:08:26.235018+0000","last_became_peered":"2024-02-19T21:08:26.235018+0000","last_unstale":"2024-02-19T21:09:43.260134+0000","last_undegraded":"2024-02-19T21:09:43.260134+0000","last_fullsized":"2024-02-19T21:09:43.260134+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:09:40.676439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.676439+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":385,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579127+0000","last_change":"2024-02-19T21:08:09.588799+0000","last_active":"2024-02-19T21:09:43.579127+0000","last_peered":"2024-02-19T21:09:43.579127+0000","last_clean":"2024-02-19T21:09:43.579127+0000","last_became_active":"2024-02-19T21:08:04.258699+0000","last_became_peered":"2024-02-19T21:08:04.258699+0000","last_unstale":"2024-02-19T21:09:43.579127+0000","last_undegraded":"2024-02-19T21:09:43.579127+0000","last_fullsized":"2024-02-19T21:09:43.579127+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:09.588687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:08:09.588687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":112,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.259137+0000","last_change":"2024-02-19T21:08:12.856127+0000","last_active":"2024-02-19T21:09:43.259137+0000","last_peered":"2024-02-19T21:09:43.259137+0000","last_clean":"2024-02-19T21:09:43.259137+0000","last_became_active":"2024-02-19T21:08:04.264983+0000","last_became_peered":"2024-02-19T21:08:04.264983+0000","last_unstale":"2024-02-19T21:09:43.259137+0000","last_undegraded":"2024-02-19T21:09:43.259137+0000","last_fullsized":"2024-02-19T21:09:43.259137+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.856015+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":196,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578492+0000","last_change":"2024-02-19T21:09:38.941012+0000","last_active":"2024-02-19T21:09:43.578492+0000","last_peered":"2024-02-19T21:09:43.578492+0000","last_clean":"2024-02-19T21:09:43.578492+0000","last_became_active":"2024-02-19T21:06:40.239199+0000","last_became_peered":"2024-02-19T21:06:40.239199+0000","last_unstale":"2024-02-19T21:09:43.578492+0000","last_undegraded":"2024-02-19T21:09:43.578492+0000","last_fullsized":"2024-02-19T21:09:43.578492+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:38.940935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:38.940935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":310,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.580281+0000","last_change":"2024-02-19T21:07:35.505665+0000","last_active":"2024-02-19T21:09:43.580281+0000","last_peered":"2024-02-19T21:09:43.580281+0000","last_clean":"2024-02-19T21:09:43.580281+0000","last_became_active":"2024-02-19T21:04:39.672228+0000","last_became_peered":"2024-02-19T21:04:39.672228+0000","last_unstale":"2024-02-19T21:09:43.580281+0000","last_undegraded":"2024-02-19T21:09:43.580281+0000","last_fullsized":"2024-02-19T21:09:43.580281+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:35.505607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:35.505607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":118,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579303+0000","last_change":"2024-02-19T21:08:52.202056+0000","last_active":"2024-02-19T21:09:43.579303+0000","last_peered":"2024-02-19T21:09:43.579303+0000","last_clean":"2024-02-19T21:09:43.579303+0000","last_became_active":"2024-02-19T21:08:41.295736+0000","last_became_peered":"2024-02-19T21:08:41.295736+0000","last_unstale":"2024-02-19T21:09:43.579303+0000","last_undegraded":"2024-02-19T21:09:43.579303+0000","last_fullsized":"2024-02-19T21:09:43.579303+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:52.201972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:52.201972+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":369,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.259323+0000","last_change":"2024-02-19T21:09:32.623020+0000","last_active":"2024-02-19T21:09:43.259323+0000","last_peered":"2024-02-19T21:09:43.259323+0000","last_clean":"2024-02-19T21:09:43.259323+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:09:43.259323+0000","last_undegraded":"2024-02-19T21:09:43.259323+0000","last_fullsized":"2024-02-19T21:09:43.259323+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:32.622939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:09:32.622939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":383,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.260769+0000","last_change":"2024-02-19T21:09:34.642491+0000","last_active":"2024-02-19T21:09:43.260769+0000","last_peered":"2024-02-19T21:09:43.260769+0000","last_clean":"2024-02-19T21:09:43.260769+0000","last_became_active":"2024-02-19T21:08:04.268847+0000","last_became_peered":"2024-02-19T21:08:04.268847+0000","last_unstale":"2024-02-19T21:09:43.260769+0000","last_undegraded":"2024-02-19T21:09:43.260769+0000","last_fullsized":"2024-02-19T21:09:43.260769+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:34.642432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:34.642432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":107,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.260118+0000","last_change":"2024-02-19T21:08:04.280460+0000","last_active":"2024-02-19T21:09:43.260118+0000","last_peered":"2024-02-19T21:09:43.260118+0000","last_clean":"2024-02-19T21:09:43.260118+0000","last_became_active":"2024-02-19T21:08:04.280233+0000","last_became_peered":"2024-02-19T21:08:04.280233+0000","last_unstale":"2024-02-19T21:09:43.260118+0000","last_undegraded":"2024-02-19T21:09:43.260118+0000","last_fullsized":"2024-02-19T21:09:43.260118+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:59.446267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:07:59.446267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":107,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.261267+0000","last_change":"2024-02-19T21:08:04.258981+0000","last_active":"2024-02-19T21:09:43.261267+0000","last_peered":"2024-02-19T21:09:43.261267+0000","last_clean":"2024-02-19T21:09:43.261267+0000","last_became_active":"2024-02-19T21:08:04.258568+0000","last_became_peered":"2024-02-19T21:08:04.258568+0000","last_unstale":"2024-02-19T21:09:43.261267+0000","last_undegraded":"2024-02-19T21:09:43.261267+0000","last_fullsized":"2024-02-19T21:09:43.261267+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:29.946015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:29.946015+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":118,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258553+0000","last_change":"2024-02-19T21:09:28.712106+0000","last_active":"2024-02-19T21:09:43.258553+0000","last_peered":"2024-02-19T21:09:43.258553+0000","last_clean":"2024-02-19T21:09:43.258553+0000","last_became_active":"2024-02-19T21:08:04.274921+0000","last_became_peered":"2024-02-19T21:08:04.274921+0000","last_unstale":"2024-02-19T21:09:43.258553+0000","last_undegraded":"2024-02-19T21:09:43.258553+0000","last_fullsized":"2024-02-19T21:09:43.258553+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:09:28.712037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.712037+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":370,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.582998+0000","last_change":"2024-02-19T21:09:42.000906+0000","last_active":"2024-02-19T21:09:43.582998+0000","last_peered":"2024-02-19T21:09:43.582998+0000","last_clean":"2024-02-19T21:09:43.582998+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:09:43.582998+0000","last_undegraded":"2024-02-19T21:09:43.582998+0000","last_fullsized":"2024-02-19T21:09:43.582998+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:42.000850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:42.000850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":372,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.580479+0000","last_change":"2024-02-19T21:08:41.261809+0000","last_active":"2024-02-19T21:09:43.580479+0000","last_peered":"2024-02-19T21:09:43.580479+0000","last_clean":"2024-02-19T21:09:43.580479+0000","last_became_active":"2024-02-19T21:08:41.261268+0000","last_became_peered":"2024-02-19T21:08:41.261268+0000","last_unstale":"2024-02-19T21:09:43.580479+0000","last_undegraded":"2024-02-19T21:09:43.580479+0000","last_fullsized":"2024-02-19T21:09:43.580479+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:11.593368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:11.593368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":372,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.482055+0000","last_change":"2024-02-19T21:09:26.731948+0000","last_active":"2024-02-19T21:09:43.482055+0000","last_peered":"2024-02-19T21:09:43.482055+0000","last_clean":"2024-02-19T21:09:43.482055+0000","last_became_active":"2024-02-19T21:08:41.274144+0000","last_became_peered":"2024-02-19T21:08:41.274144+0000","last_unstale":"2024-02-19T21:09:43.482055+0000","last_undegraded":"2024-02-19T21:09:43.482055+0000","last_fullsized":"2024-02-19T21:09:43.482055+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:26.731860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:26.731860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":112,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.268354+0000","last_change":"2024-02-19T21:08:10.824761+0000","last_active":"2024-02-19T21:09:43.268354+0000","last_peered":"2024-02-19T21:09:43.268354+0000","last_clean":"2024-02-19T21:09:43.268354+0000","last_became_active":"2024-02-19T21:08:04.266161+0000","last_became_peered":"2024-02-19T21:08:04.266161+0000","last_unstale":"2024-02-19T21:09:43.268354+0000","last_undegraded":"2024-02-19T21:09:43.268354+0000","last_fullsized":"2024-02-19T21:09:43.268354+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:10.824694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:10.824694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":124,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.260644+0000","last_change":"2024-02-19T21:08:11.828624+0000","last_active":"2024-02-19T21:09:43.260644+0000","last_peered":"2024-02-19T21:09:43.260644+0000","last_clean":"2024-02-19T21:09:43.260644+0000","last_became_active":"2024-02-19T21:08:04.250742+0000","last_became_peered":"2024-02-19T21:08:04.250742+0000","last_unstale":"2024-02-19T21:09:43.260644+0000","last_undegraded":"2024-02-19T21:09:43.260644+0000","last_fullsized":"2024-02-19T21:09:43.260644+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:11.828537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:11.828537+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":367,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.582842+0000","last_change":"2024-02-19T21:08:40.270859+0000","last_active":"2024-02-19T21:09:43.582842+0000","last_peered":"2024-02-19T21:09:43.582842+0000","last_clean":"2024-02-19T21:09:43.582842+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:09:43.582842+0000","last_undegraded":"2024-02-19T21:09:43.582842+0000","last_fullsized":"2024-02-19T21:09:43.582842+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:40.270758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:40.270758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":369,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.481842+0000","last_change":"2024-02-19T21:09:29.783688+0000","last_active":"2024-02-19T21:09:43.481842+0000","last_peered":"2024-02-19T21:09:43.481842+0000","last_clean":"2024-02-19T21:09:43.481842+0000","last_became_active":"2024-02-19T21:08:41.267236+0000","last_became_peered":"2024-02-19T21:08:41.267236+0000","last_unstale":"2024-02-19T21:09:43.481842+0000","last_undegraded":"2024-02-19T21:09:43.481842+0000","last_fullsized":"2024-02-19T21:09:43.481842+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:29.783609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:29.783609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":107,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.259224+0000","last_change":"2024-02-19T21:08:04.254279+0000","last_active":"2024-02-19T21:09:43.259224+0000","last_peered":"2024-02-19T21:09:43.259224+0000","last_clean":"2024-02-19T21:09:43.259224+0000","last_became_active":"2024-02-19T21:08:04.251987+0000","last_became_peered":"2024-02-19T21:08:04.251987+0000","last_unstale":"2024-02-19T21:09:43.259224+0000","last_undegraded":"2024-02-19T21:09:43.259224+0000","last_fullsized":"2024-02-19T21:09:43.259224+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:08.757198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:08.757198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":193,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.581982+0000","last_change":"2024-02-19T21:06:54.664879+0000","last_active":"2024-02-19T21:09:43.581982+0000","last_peered":"2024-02-19T21:09:43.581982+0000","last_clean":"2024-02-19T21:09:43.581982+0000","last_became_active":"2024-02-19T21:06:40.226196+0000","last_became_peered":"2024-02-19T21:06:40.226196+0000","last_unstale":"2024-02-19T21:09:43.581982+0000","last_undegraded":"2024-02-19T21:09:43.581982+0000","last_fullsized":"2024-02-19T21:09:43.581982+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.664814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":265,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.480832+0000","last_change":"2024-02-19T21:09:30.782123+0000","last_active":"2024-02-19T21:09:43.480832+0000","last_peered":"2024-02-19T21:09:43.480832+0000","last_clean":"2024-02-19T21:09:43.480832+0000","last_became_active":"2024-02-19T21:08:04.267222+0000","last_became_peered":"2024-02-19T21:08:04.267222+0000","last_unstale":"2024-02-19T21:09:43.480832+0000","last_undegraded":"2024-02-19T21:09:43.480832+0000","last_fullsized":"2024-02-19T21:09:43.480832+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:09:30.782002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:30.782002+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":374,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578443+0000","last_change":"2024-02-19T21:08:52.579253+0000","last_active":"2024-02-19T21:09:43.578443+0000","last_peered":"2024-02-19T21:09:43.578443+0000","last_clean":"2024-02-19T21:09:43.578443+0000","last_became_active":"2024-02-19T21:08:41.264843+0000","last_became_peered":"2024-02-19T21:08:41.264843+0000","last_unstale":"2024-02-19T21:09:43.578443+0000","last_undegraded":"2024-02-19T21:09:43.578443+0000","last_fullsized":"2024-02-19T21:09:43.578443+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:52.579178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:52.579178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"344'9","reported_seq":489,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-02-19T21:09:44.265408+0000","last_change":"2024-02-19T21:08:41.319305+0000","last_active":"2024-02-19T21:09:44.265408+0000","last_peered":"2024-02-19T21:09:44.265408+0000","last_clean":"2024-02-19T21:09:44.265408+0000","last_became_active":"2024-02-19T21:08:41.281854+0000","last_became_peered":"2024-02-19T21:08:41.281854+0000","last_unstale":"2024-02-19T21:09:44.265408+0000","last_undegraded":"2024-02-19T21:09:44.265408+0000","last_fullsized":"2024-02-19T21:09:44.265408+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:08:36.251958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.251958+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":365,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.483130+0000","last_change":"2024-02-19T21:09:41.951642+0000","last_active":"2024-02-19T21:09:43.483130+0000","last_peered":"2024-02-19T21:09:43.483130+0000","last_clean":"2024-02-19T21:09:43.483130+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:09:43.483130+0000","last_undegraded":"2024-02-19T21:09:43.483130+0000","last_fullsized":"2024-02-19T21:09:43.483130+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:09:41.951576+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:09:41.951576+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":315,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.582115+0000","last_change":"2024-02-19T21:08:39.107325+0000","last_active":"2024-02-19T21:09:43.582115+0000","last_peered":"2024-02-19T21:09:43.582115+0000","last_clean":"2024-02-19T21:09:43.582115+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:09:43.582115+0000","last_undegraded":"2024-02-19T21:09:43.582115+0000","last_fullsized":"2024-02-19T21:09:43.582115+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:39.107240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.107240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":384,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.583814+0000","last_change":"2024-02-19T21:08:03.561292+0000","last_active":"2024-02-19T21:09:43.583814+0000","last_peered":"2024-02-19T21:09:43.583814+0000","last_clean":"2024-02-19T21:09:43.583814+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:09:43.583814+0000","last_undegraded":"2024-02-19T21:09:43.583814+0000","last_fullsized":"2024-02-19T21:09:43.583814+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:03.561223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:03.561223+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":107,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.260730+0000","last_change":"2024-02-19T21:08:04.254214+0000","last_active":"2024-02-19T21:09:43.260730+0000","last_peered":"2024-02-19T21:09:43.260730+0000","last_clean":"2024-02-19T21:09:43.260730+0000","last_became_active":"2024-02-19T21:08:04.251241+0000","last_became_peered":"2024-02-19T21:08:04.251241+0000","last_unstale":"2024-02-19T21:09:43.260730+0000","last_undegraded":"2024-02-19T21:09:43.260730+0000","last_fullsized":"2024-02-19T21:09:43.260730+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:01.530618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:01.530618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":196,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.480774+0000","last_change":"2024-02-19T21:08:36.903762+0000","last_active":"2024-02-19T21:09:43.480774+0000","last_peered":"2024-02-19T21:09:43.480774+0000","last_clean":"2024-02-19T21:09:43.480774+0000","last_became_active":"2024-02-19T21:06:40.234026+0000","last_became_peered":"2024-02-19T21:06:40.234026+0000","last_unstale":"2024-02-19T21:09:43.480774+0000","last_undegraded":"2024-02-19T21:09:43.480774+0000","last_fullsized":"2024-02-19T21:09:43.480774+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:36.903674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.903674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":367,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.259094+0000","last_change":"2024-02-19T21:08:48.760695+0000","last_active":"2024-02-19T21:09:43.259094+0000","last_peered":"2024-02-19T21:09:43.259094+0000","last_clean":"2024-02-19T21:09:43.259094+0000","last_became_active":"2024-02-19T21:08:41.309795+0000","last_became_peered":"2024-02-19T21:08:41.309795+0000","last_unstale":"2024-02-19T21:09:43.259094+0000","last_undegraded":"2024-02-19T21:09:43.259094+0000","last_fullsized":"2024-02-19T21:09:43.259094+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:48.760625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:08:48.760625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":321,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.258907+0000","last_change":"2024-02-19T21:08:41.310765+0000","last_active":"2024-02-19T21:09:43.258907+0000","last_peered":"2024-02-19T21:09:43.258907+0000","last_clean":"2024-02-19T21:09:43.258907+0000","last_became_active":"2024-02-19T21:08:41.310546+0000","last_became_peered":"2024-02-19T21:08:41.310546+0000","last_unstale":"2024-02-19T21:09:43.258907+0000","last_undegraded":"2024-02-19T21:09:43.258907+0000","last_fullsized":"2024-02-19T21:09:43.258907+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:33.686917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:33.686917+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":437,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.582391+0000","last_change":"2024-02-19T21:08:04.337335+0000","last_active":"2024-02-19T21:09:43.582391+0000","last_peered":"2024-02-19T21:09:43.582391+0000","last_clean":"2024-02-19T21:09:43.582391+0000","last_became_active":"2024-02-19T21:08:04.248378+0000","last_became_peered":"2024-02-19T21:08:04.248378+0000","last_unstale":"2024-02-19T21:09:43.582391+0000","last_undegraded":"2024-02-19T21:09:43.582391+0000","last_fullsized":"2024-02-19T21:09:43.582391+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:06:48.310459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.310459+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":376,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.580290+0000","last_change":"2024-02-19T21:09:26.417332+0000","last_active":"2024-02-19T21:09:43.580290+0000","last_peered":"2024-02-19T21:09:43.580290+0000","last_clean":"2024-02-19T21:09:43.580290+0000","last_became_active":"2024-02-19T21:08:41.278751+0000","last_became_peered":"2024-02-19T21:08:41.278751+0000","last_unstale":"2024-02-19T21:09:43.580290+0000","last_undegraded":"2024-02-19T21:09:43.580290+0000","last_fullsized":"2024-02-19T21:09:43.580290+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:26.417247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:26.417247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":369,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.259539+0000","last_change":"2024-02-19T21:08:41.308944+0000","last_active":"2024-02-19T21:09:43.259539+0000","last_peered":"2024-02-19T21:09:43.259539+0000","last_clean":"2024-02-19T21:09:43.259539+0000","last_became_active":"2024-02-19T21:08:41.302364+0000","last_became_peered":"2024-02-19T21:08:41.302364+0000","last_unstale":"2024-02-19T21:09:43.259539+0000","last_undegraded":"2024-02-19T21:09:43.259539+0000","last_fullsized":"2024-02-19T21:09:43.259539+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:53.629475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:53.629475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":70,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.580010+0000","last_change":"2024-02-19T21:08:41.279946+0000","last_active":"2024-02-19T21:09:43.580010+0000","last_peered":"2024-02-19T21:09:43.580010+0000","last_clean":"2024-02-19T21:09:43.580010+0000","last_became_active":"2024-02-19T21:08:41.278358+0000","last_became_peered":"2024-02-19T21:08:41.278358+0000","last_unstale":"2024-02-19T21:09:43.580010+0000","last_undegraded":"2024-02-19T21:09:43.580010+0000","last_fullsized":"2024-02-19T21:09:43.580010+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:36.723492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:07:36.723492+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":193,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578351+0000","last_change":"2024-02-19T21:09:27.464040+0000","last_active":"2024-02-19T21:09:43.578351+0000","last_peered":"2024-02-19T21:09:43.578351+0000","last_clean":"2024-02-19T21:09:43.578351+0000","last_became_active":"2024-02-19T21:06:40.240859+0000","last_became_peered":"2024-02-19T21:06:40.240859+0000","last_unstale":"2024-02-19T21:09:43.578351+0000","last_undegraded":"2024-02-19T21:09:43.578351+0000","last_fullsized":"2024-02-19T21:09:43.578351+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.463966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.463966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":145,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.260736+0000","last_change":"2024-02-19T21:08:49.869657+0000","last_active":"2024-02-19T21:09:43.260736+0000","last_peered":"2024-02-19T21:09:43.260736+0000","last_clean":"2024-02-19T21:09:43.260736+0000","last_became_active":"2024-02-19T21:08:41.295566+0000","last_became_peered":"2024-02-19T21:08:41.295566+0000","last_unstale":"2024-02-19T21:09:43.260736+0000","last_undegraded":"2024-02-19T21:09:43.260736+0000","last_fullsized":"2024-02-19T21:09:43.260736+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:08:49.869588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.869588+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":107,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.259312+0000","last_change":"2024-02-19T21:08:04.254443+0000","last_active":"2024-02-19T21:09:43.259312+0000","last_peered":"2024-02-19T21:09:43.259312+0000","last_clean":"2024-02-19T21:09:43.259312+0000","last_became_active":"2024-02-19T21:08:04.253058+0000","last_became_peered":"2024-02-19T21:08:04.253058+0000","last_unstale":"2024-02-19T21:09:43.259312+0000","last_undegraded":"2024-02-19T21:09:43.259312+0000","last_fullsized":"2024-02-19T21:09:43.259312+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:02.693025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:02.693025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":114,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.268011+0000","last_change":"2024-02-19T21:09:35.626645+0000","last_active":"2024-02-19T21:09:43.268011+0000","last_peered":"2024-02-19T21:09:43.268011+0000","last_clean":"2024-02-19T21:09:43.268011+0000","last_became_active":"2024-02-19T21:08:41.288781+0000","last_became_peered":"2024-02-19T21:08:41.288781+0000","last_unstale":"2024-02-19T21:09:43.268011+0000","last_undegraded":"2024-02-19T21:09:43.268011+0000","last_fullsized":"2024-02-19T21:09:43.268011+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:35.626548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:35.626548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":107,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.261290+0000","last_change":"2024-02-19T21:08:04.278426+0000","last_active":"2024-02-19T21:09:43.261290+0000","last_peered":"2024-02-19T21:09:43.261290+0000","last_clean":"2024-02-19T21:09:43.261290+0000","last_became_active":"2024-02-19T21:08:04.278045+0000","last_became_peered":"2024-02-19T21:08:04.278045+0000","last_unstale":"2024-02-19T21:09:43.261290+0000","last_undegraded":"2024-02-19T21:09:43.261290+0000","last_fullsized":"2024-02-19T21:09:43.261290+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.723156+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":312,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.262805+0000","last_change":"2024-02-19T21:07:25.735056+0000","last_active":"2024-02-19T21:09:43.262805+0000","last_peered":"2024-02-19T21:09:43.262805+0000","last_clean":"2024-02-19T21:09:43.262805+0000","last_became_active":"2024-02-19T21:04:39.777442+0000","last_became_peered":"2024-02-19T21:04:39.777442+0000","last_unstale":"2024-02-19T21:09:43.262805+0000","last_undegraded":"2024-02-19T21:09:43.262805+0000","last_fullsized":"2024-02-19T21:09:43.262805+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:25.734988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:25.734988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"194'13","reported_seq":388,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.261441+0000","last_change":"2024-02-19T21:08:41.890452+0000","last_active":"2024-02-19T21:09:43.261441+0000","last_peered":"2024-02-19T21:09:43.261441+0000","last_clean":"2024-02-19T21:09:43.261441+0000","last_became_active":"2024-02-19T21:08:41.283514+0000","last_became_peered":"2024-02-19T21:08:41.283514+0000","last_unstale":"2024-02-19T21:09:43.261441+0000","last_undegraded":"2024-02-19T21:09:43.261441+0000","last_fullsized":"2024-02-19T21:09:43.261441+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'13","last_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.772417+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":364,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.482956+0000","last_change":"2024-02-19T21:08:51.955830+0000","last_active":"2024-02-19T21:09:43.482956+0000","last_peered":"2024-02-19T21:09:43.482956+0000","last_clean":"2024-02-19T21:09:43.482956+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:09:43.482956+0000","last_undegraded":"2024-02-19T21:09:43.482956+0000","last_fullsized":"2024-02-19T21:09:43.482956+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:51.955738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:51.955738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":361,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579277+0000","last_change":"2024-02-19T21:09:29.852892+0000","last_active":"2024-02-19T21:09:43.579277+0000","last_peered":"2024-02-19T21:09:43.579277+0000","last_clean":"2024-02-19T21:09:43.579277+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:09:43.579277+0000","last_undegraded":"2024-02-19T21:09:43.579277+0000","last_fullsized":"2024-02-19T21:09:43.579277+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:29.852792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:29.852792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":190,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.256417+0000","last_change":"2024-02-19T21:06:40.239902+0000","last_active":"2024-02-19T21:09:43.256417+0000","last_peered":"2024-02-19T21:09:43.256417+0000","last_clean":"2024-02-19T21:09:43.256417+0000","last_became_active":"2024-02-19T21:06:40.239603+0000","last_became_peered":"2024-02-19T21:06:40.239603+0000","last_unstale":"2024-02-19T21:09:43.256417+0000","last_undegraded":"2024-02-19T21:09:43.256417+0000","last_fullsized":"2024-02-19T21:09:43.256417+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:06:25.785366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":321,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.583729+0000","last_change":"2024-02-19T21:08:41.276256+0000","last_active":"2024-02-19T21:09:43.583729+0000","last_peered":"2024-02-19T21:09:43.583729+0000","last_clean":"2024-02-19T21:09:43.583729+0000","last_became_active":"2024-02-19T21:08:41.268425+0000","last_became_peered":"2024-02-19T21:08:41.268425+0000","last_unstale":"2024-02-19T21:09:43.583729+0000","last_undegraded":"2024-02-19T21:09:43.583729+0000","last_fullsized":"2024-02-19T21:09:43.583729+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:58.310306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:58.310306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":375,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-02-19T21:09:44.596395+0000","last_change":"2024-02-19T21:09:44.596395+0000","last_active":"2024-02-19T21:09:44.596395+0000","last_peered":"2024-02-19T21:09:44.596395+0000","last_clean":"2024-02-19T21:09:44.596395+0000","last_became_active":"2024-02-19T21:08:41.298233+0000","last_became_peered":"2024-02-19T21:08:41.298233+0000","last_unstale":"2024-02-19T21:09:44.596395+0000","last_undegraded":"2024-02-19T21:09:44.596395+0000","last_fullsized":"2024-02-19T21:09:44.596395+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:09:44.596326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:44.596326+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":369,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.481745+0000","last_change":"2024-02-19T21:09:25.733055+0000","last_active":"2024-02-19T21:09:43.481745+0000","last_peered":"2024-02-19T21:09:43.481745+0000","last_clean":"2024-02-19T21:09:43.481745+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:09:43.481745+0000","last_undegraded":"2024-02-19T21:09:43.481745+0000","last_fullsized":"2024-02-19T21:09:43.481745+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:25.732955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:25.732955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":374,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.482181+0000","last_change":"2024-02-19T21:09:36.862378+0000","last_active":"2024-02-19T21:09:43.482181+0000","last_peered":"2024-02-19T21:09:43.482181+0000","last_clean":"2024-02-19T21:09:43.482181+0000","last_became_active":"2024-02-19T21:08:41.304581+0000","last_became_peered":"2024-02-19T21:08:41.304581+0000","last_unstale":"2024-02-19T21:09:43.482181+0000","last_undegraded":"2024-02-19T21:09:43.482181+0000","last_fullsized":"2024-02-19T21:09:43.482181+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:36.862297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:36.862297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":377,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579165+0000","last_change":"2024-02-19T21:08:04.266429+0000","last_active":"2024-02-19T21:09:43.579165+0000","last_peered":"2024-02-19T21:09:43.579165+0000","last_clean":"2024-02-19T21:09:43.579165+0000","last_became_active":"2024-02-19T21:08:04.266100+0000","last_became_peered":"2024-02-19T21:08:04.266100+0000","last_unstale":"2024-02-19T21:09:43.579165+0000","last_undegraded":"2024-02-19T21:09:43.579165+0000","last_fullsized":"2024-02-19T21:09:43.579165+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:48.506964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.506964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":196,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.579746+0000","last_change":"2024-02-19T21:06:50.467486+0000","last_active":"2024-02-19T21:09:43.579746+0000","last_peered":"2024-02-19T21:09:43.579746+0000","last_clean":"2024-02-19T21:09:43.579746+0000","last_became_active":"2024-02-19T21:06:40.227713+0000","last_became_peered":"2024-02-19T21:06:40.227713+0000","last_unstale":"2024-02-19T21:09:43.579746+0000","last_undegraded":"2024-02-19T21:09:43.579746+0000","last_fullsized":"2024-02-19T21:09:43.579746+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:50.467411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:50.467411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":108,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578850+0000","last_change":"2024-02-19T21:08:42.700047+0000","last_active":"2024-02-19T21:09:43.578850+0000","last_peered":"2024-02-19T21:09:43.578850+0000","last_clean":"2024-02-19T21:09:43.578850+0000","last_became_active":"2024-02-19T21:08:41.266486+0000","last_became_peered":"2024-02-19T21:08:41.266486+0000","last_unstale":"2024-02-19T21:09:43.578850+0000","last_undegraded":"2024-02-19T21:09:43.578850+0000","last_fullsized":"2024-02-19T21:09:43.578850+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:08:36.593172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.593172+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":107,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.267472+0000","last_change":"2024-02-19T21:08:04.278604+0000","last_active":"2024-02-19T21:09:43.267472+0000","last_peered":"2024-02-19T21:09:43.267472+0000","last_clean":"2024-02-19T21:09:43.267472+0000","last_became_active":"2024-02-19T21:08:04.277576+0000","last_became_peered":"2024-02-19T21:08:04.277576+0000","last_unstale":"2024-02-19T21:09:43.267472+0000","last_undegraded":"2024-02-19T21:09:43.267472+0000","last_fullsized":"2024-02-19T21:09:43.267472+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:53.752543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:53.752543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":360,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.262570+0000","last_change":"2024-02-19T21:06:24.702277+0000","last_active":"2024-02-19T21:09:43.262570+0000","last_peered":"2024-02-19T21:09:43.262570+0000","last_clean":"2024-02-19T21:09:43.262570+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:09:43.262570+0000","last_undegraded":"2024-02-19T21:09:43.262570+0000","last_fullsized":"2024-02-19T21:09:43.262570+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:06:24.702197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":71,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.581325+0000","last_change":"2024-02-19T21:08:41.307487+0000","last_active":"2024-02-19T21:09:43.581325+0000","last_peered":"2024-02-19T21:09:43.581325+0000","last_clean":"2024-02-19T21:09:43.581325+0000","last_became_active":"2024-02-19T21:08:41.306708+0000","last_became_peered":"2024-02-19T21:08:41.306708+0000","last_unstale":"2024-02-19T21:09:43.581325+0000","last_undegraded":"2024-02-19T21:09:43.581325+0000","last_fullsized":"2024-02-19T21:09:43.581325+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:04.600973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:04.600973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":322,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.582955+0000","last_change":"2024-02-19T21:08:41.293045+0000","last_active":"2024-02-19T21:09:43.582955+0000","last_peered":"2024-02-19T21:09:43.582955+0000","last_clean":"2024-02-19T21:09:43.582955+0000","last_became_active":"2024-02-19T21:08:41.292651+0000","last_became_peered":"2024-02-19T21:08:41.292651+0000","last_unstale":"2024-02-19T21:09:43.582955+0000","last_undegraded":"2024-02-19T21:09:43.582955+0000","last_fullsized":"2024-02-19T21:09:43.582955+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:58.468840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:58.468840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":123,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.260065+0000","last_change":"2024-02-19T21:08:04.341196+0000","last_active":"2024-02-19T21:09:43.260065+0000","last_peered":"2024-02-19T21:09:43.260065+0000","last_clean":"2024-02-19T21:09:43.260065+0000","last_became_active":"2024-02-19T21:08:04.249443+0000","last_became_peered":"2024-02-19T21:08:04.249443+0000","last_unstale":"2024-02-19T21:09:43.260065+0000","last_undegraded":"2024-02-19T21:09:43.260065+0000","last_fullsized":"2024-02-19T21:09:43.260065+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:06:09.763482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.763482+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":207,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.481574+0000","last_change":"2024-02-19T21:08:47.903239+0000","last_active":"2024-02-19T21:09:43.481574+0000","last_peered":"2024-02-19T21:09:43.481574+0000","last_clean":"2024-02-19T21:09:43.481574+0000","last_became_active":"2024-02-19T21:08:41.263206+0000","last_became_peered":"2024-02-19T21:08:41.263206+0000","last_unstale":"2024-02-19T21:09:43.481574+0000","last_undegraded":"2024-02-19T21:09:43.481574+0000","last_fullsized":"2024-02-19T21:09:43.481574+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:47.903127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:47.903127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":70,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.578759+0000","last_change":"2024-02-19T21:08:41.263525+0000","last_active":"2024-02-19T21:09:43.578759+0000","last_peered":"2024-02-19T21:09:43.578759+0000","last_clean":"2024-02-19T21:09:43.578759+0000","last_became_active":"2024-02-19T21:08:41.263160+0000","last_became_peered":"2024-02-19T21:08:41.263160+0000","last_unstale":"2024-02-19T21:09:43.578759+0000","last_undegraded":"2024-02-19T21:09:43.578759+0000","last_fullsized":"2024-02-19T21:09:43.578759+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:39.574835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.574835+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":112,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.259061+0000","last_change":"2024-02-19T21:09:33.559448+0000","last_active":"2024-02-19T21:09:43.259061+0000","last_peered":"2024-02-19T21:09:43.259061+0000","last_clean":"2024-02-19T21:09:43.259061+0000","last_became_active":"2024-02-19T21:08:04.252640+0000","last_became_peered":"2024-02-19T21:08:04.252640+0000","last_unstale":"2024-02-19T21:09:43.259061+0000","last_undegraded":"2024-02-19T21:09:43.259061+0000","last_fullsized":"2024-02-19T21:09:43.259061+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:33.559363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:33.559363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":328,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-02-19T21:09:43.581672+0000","last_change":"2024-02-19T21:08:04.251168+0000","last_active":"2024-02-19T21:09:43.581672+0000","last_peered":"2024-02-19T21:09:43.581672+0000","last_clean":"2024-02-19T21:09:43.581672+0000","last_became_active":"2024-02-19T21:08:04.250937+0000","last_became_peered":"2024-02-19T21:08:04.250937+0000","last_unstale":"2024-02-19T21:09:43.581672+0000","last_undegraded":"2024-02-19T21:09:43.581672+0000","last_fullsized":"2024-02-19T21:09:43.581672+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:27.744835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:27.744835+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":212,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":4001455,"num_objects":155,"num_object_clones":2,"num_object_copies":465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":107,"num_read_kb":9,"num_write":587,"num_write_kb":4486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":491,"num_bytes_recovered":1160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13885440,"data_stored":12004461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1850,"ondisk_log_size":1850,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":677,"num_read_kb":434,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":4500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":410,"ondisk_log_size":410,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":501,"num_read_kb":496,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266792,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311656,"kb_used_data":10328,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93433496,"statfs":{"total":95995035648,"available":95675899904,"internally_reserved":0,"allocated":10575872,"data_stored":7726264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":1.603,"15min":0.86899999999999999},"min":{"1min":0.26000000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.65200000000000002,"5min":89.456999999999994,"15min":89.456999999999994},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":1.377,"15min":0.78600000000000003},"min":{"1min":0.28499999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.67900000000000005,"5min":70.451999999999998,"15min":70.451999999999998},"last":0.36199999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.0740000000000001,"15min":0.70199999999999996},"min":{"1min":0.36299999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.85099999999999998,"5min":34.273000000000003,"15min":34.273000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":1.7889999999999999,"15min":0.94099999999999995},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.254},"max":{"1min":0.79200000000000004,"5min":89.594999999999999,"15min":89.594999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":1.837,"15min":0.94399999999999995},"min":{"1min":0.35099999999999998,"5min":0.27600000000000002,"15min":0.24299999999999999},"max":{"1min":0.67100000000000004,"5min":104.413,"15min":104.413},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":1.8009999999999999,"15min":0.94999999999999996},"min":{"1min":0.35599999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70599999999999996,"5min":104.268,"15min":104.268},"last":0.39400000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":1.7969999999999999,"15min":0.95199999999999996},"min":{"1min":0.38800000000000001,"5min":0.29099999999999998,"15min":0.254},"max":{"1min":1.2949999999999999,"5min":105.3,"15min":105.3},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.79700000000000004,"15min":0.59799999999999998},"min":{"1min":0.34300000000000003,"5min":0.315,"15min":0.315},"max":{"1min":0.879,"5min":13.243,"15min":13.243},"last":0.442}]},{"osd":4,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.42799999999999999,"15min":0.47599999999999998},"min":{"1min":0.315,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63200000000000001,"5min":0.69299999999999995,"15min":1.373},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.45700000000000002,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.26600000000000001,"15min":0.25900000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":1.0740000000000001},"last":0.46700000000000003}]},{"osd":5,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.28599999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.65900000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.373},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.48999999999999999,"15min":0.50900000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69399999999999995,"5min":0.77400000000000002,"15min":0.85899999999999999},"last":0.34399999999999997}]},{"osd":6,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.49199999999999999,"15min":0.52400000000000002},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.70199999999999996,"5min":0.76500000000000001,"15min":1.0049999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52300000000000002,"15min":0.56100000000000005},"min":{"1min":0.34100000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72499999999999998,"5min":0.75900000000000001,"15min":1.262},"last":0.41999999999999998}]}]},{"osd":6,"up_from":45,"seq":193273528430,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307796,"kb_used_data":6464,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437356,"statfs":{"total":95995035648,"available":95679852544,"internally_reserved":0,"allocated":6619136,"data_stored":3733722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.86299999999999999,"15min":0.61699999999999999},"min":{"1min":0.22700000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.73099999999999998,"5min":17.486999999999998,"15min":17.486999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.98299999999999998,"15min":0.65400000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64900000000000002,"5min":17.43,"15min":17.43},"last":0.44600000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.873,"15min":0.61799999999999999},"min":{"1min":0.28699999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.71599999999999997,"5min":14.692,"15min":14.692},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.99299999999999999,"15min":0.66400000000000003},"min":{"1min":0.29099999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.74099999999999999,"5min":14.795,"15min":14.795},"last":0.37}]},{"osd":2,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.79100000000000004,"15min":0.60799999999999998},"min":{"1min":0.309,"5min":0.255,"15min":0.255},"max":{"1min":0.61899999999999999,"5min":6.931,"15min":6.931},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":1.0249999999999999,"15min":0.66900000000000004},"min":{"1min":0.34100000000000003,"5min":0.318,"15min":0.318},"max":{"1min":0.71899999999999997,"5min":14.391,"15min":14.391},"last":0.53600000000000003}]},{"osd":3,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":1.077,"15min":0.69999999999999996},"min":{"1min":0.378,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73599999999999999,"5min":14.510999999999999,"15min":14.510999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.95299999999999996,"15min":0.65300000000000002},"min":{"1min":0.35299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.64500000000000002,"5min":17.585000000000001,"15min":17.585000000000001},"last":0.501}]},{"osd":4,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40100000000000002,"15min":0.39200000000000002},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.63800000000000001,"15min":1.363},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.41599999999999998},"min":{"1min":0.29799999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.63500000000000001,"5min":0.69899999999999995,"15min":0.70999999999999996},"last":0.33400000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46100000000000002,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.59699999999999998,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.504,"15min":0.47299999999999998},"min":{"1min":0.35099999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.627,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.442}]},{"osd":7,"last update":"Mon Feb 19 21:09:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47699999999999998,"15min":0.503},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":1.1160000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.46999999999999997,"15min":0.51100000000000001},"min":{"1min":0.32700000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":1.534},"last":0.39700000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574981,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307660,"kb_used_data":10428,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93437492,"statfs":{"total":95995035648,"available":95679991808,"internally_reserved":0,"allocated":10678272,"data_stored":7735345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.92900000000000005,"15min":0.54600000000000004},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.51400000000000001,"5min":27.777000000000001,"15min":27.777000000000001},"last":0.372},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.98799999999999999,"15min":0.57399999999999995},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.217},"max":{"1min":0.70899999999999996,"5min":27.631,"15min":27.631},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":1.3109999999999999,"15min":0.71099999999999997},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.66300000000000003,"5min":27.798999999999999,"15min":27.798999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.372,"5min":1.3420000000000001,"15min":0.72599999999999998},"min":{"1min":0.24299999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.73299999999999998,"5min":27.606999999999999,"15min":27.606999999999999},"last":0.498}]},{"osd":3,"last update":"Mon Feb 19 21:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.90100000000000002,"15min":0.57899999999999996},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.55400000000000005,"5min":15.574999999999999,"15min":15.574999999999999},"last":0.437},{"interface":"front","average":{"1min":0.371,"5min":1.119,"15min":0.67400000000000004},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.54200000000000004,"5min":27.82,"15min":27.82},"last":0.51000000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:09:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.86099999999999999,"15min":0.66200000000000003},"min":{"1min":0.36099999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.80400000000000005,"5min":9.6430000000000007,"15min":9.6430000000000007},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.127,"15min":0.755},"min":{"1min":0.40200000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.93000000000000005,"5min":27.707000000000001,"15min":27.707000000000001},"last":0.55800000000000005}]},{"osd":5,"last update":"Mon Feb 19 21:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":1.075,"15min":0.75},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":0.94199999999999995,"5min":27.529,"15min":27.529},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.86799999999999999,"15min":0.69599999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.84499999999999997,"5min":9.6799999999999997,"15min":9.6799999999999997},"last":0.59599999999999997}]},{"osd":6,"last update":"Mon Feb 19 21:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.84999999999999998,"15min":0.67500000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.67100000000000004,"5min":9.6989999999999998,"15min":9.6989999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":1.115,"15min":0.79500000000000004},"min":{"1min":0.41799999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76600000000000001,"5min":27.733000000000001,"15min":27.733000000000001},"last":0.58499999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.139,"15min":0.77600000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.92000000000000004,"5min":27.576000000000001,"15min":27.576000000000001},"last":0.621},{"interface":"front","average":{"1min":0.66700000000000004,"5min":1.2170000000000001,"15min":0.81200000000000006},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":2.0169999999999999,"5min":27.756,"15min":27.756},"last":0.65000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738507,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307728,"kb_used_data":6404,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437424,"statfs":{"total":95995035648,"available":95679922176,"internally_reserved":0,"allocated":6557696,"data_stored":3725903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":1.119,"15min":0.59899999999999998},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.16200000000000001},"max":{"1min":0.53800000000000003,"5min":20.966999999999999,"15min":20.966999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":1.179,"15min":0.60699999999999998},"min":{"1min":0.216,"5min":0.20399999999999999,"15min":0.188},"max":{"1min":0.68200000000000005,"5min":20.366,"15min":20.366},"last":0.26200000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:09:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":1.369,"15min":0.71599999999999997},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51000000000000001,"5min":27.916,"15min":27.916},"last":0.316},{"interface":"front","average":{"1min":0.39300000000000002,"5min":1.081,"15min":0.63200000000000001},"min":{"1min":0.23999999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.622,"5min":22.004000000000001,"15min":22.004000000000001},"last":0.252}]},{"osd":3,"last update":"Mon Feb 19 21:08:46 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":1.3740000000000001,"15min":0.75800000000000001},"min":{"1min":0.30399999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.53600000000000003,"5min":25.669,"15min":25.669},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":1.1839999999999999,"15min":0.70299999999999996},"min":{"1min":0.29699999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.69599999999999995,"5min":25.006,"15min":25.006},"last":0.33300000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.0960000000000001,"15min":0.71499999999999997},"min":{"1min":0.34399999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.86499999999999999,"5min":20.411999999999999,"15min":20.411999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":1.153,"15min":0.748},"min":{"1min":0.376,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.1160000000000001,"5min":19.986999999999998,"15min":19.986999999999998},"last":0.495}]},{"osd":5,"last update":"Mon Feb 19 21:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":1.2430000000000001,"15min":0.83299999999999996},"min":{"1min":0.40100000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.1890000000000001,"5min":20.021999999999998,"15min":20.021999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.173,"15min":0.91000000000000003},"min":{"1min":0.317,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":6.2859999999999996,"5min":20.908999999999999,"15min":20.908999999999999},"last":0.56599999999999995}]},{"osd":6,"last update":"Mon Feb 19 21:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":1.0900000000000001,"15min":0.76400000000000001},"min":{"1min":0.40100000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.238,"5min":20.445,"15min":20.445},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":1.1679999999999999,"15min":0.81899999999999995},"min":{"1min":0.38500000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.3530000000000002,"5min":20.047999999999998,"15min":20.047999999999998},"last":0.51400000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:09:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.135,"15min":0.89300000000000002},"min":{"1min":0.41099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.95499999999999996,"5min":20.940999999999999,"15min":20.940999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":1.2170000000000001,"15min":0.82499999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.3420000000000001,"5min":20.471,"15min":20.471},"last":0.60699999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411456,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":323968,"kb_used_data":10352,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93421184,"statfs":{"total":95995035648,"available":95663292416,"internally_reserved":0,"allocated":10600448,"data_stored":7734114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:27 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.85499999999999998,"15min":0.53900000000000003},"min":{"1min":0.222,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.54100000000000004,"5min":26.690999999999999,"15min":26.690999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.32200000000000001,"5min":1.0069999999999999,"15min":0.59099999999999997},"min":{"1min":0.247,"5min":0.23200000000000001,"15min":0.214},"max":{"1min":0.48999999999999999,"5min":24.315000000000001,"15min":24.315000000000001},"last":0.31}]},{"osd":1,"last update":"Mon Feb 19 21:09:27 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":1.3480000000000001,"15min":0.70699999999999996},"min":{"1min":0.252,"5min":0.22,"15min":0.20100000000000001},"max":{"1min":0.52300000000000002,"5min":28.882000000000001,"15min":28.882000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":1.2,"15min":0.68400000000000005},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":0.50600000000000001,"5min":28.382999999999999,"15min":28.382999999999999},"last":0.23999999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:08:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.294,"15min":0.72999999999999998},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":12.75,"5min":28.742000000000001,"15min":28.742000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.0249999999999999,"15min":0.65100000000000002},"min":{"1min":0.307,"5min":0.27600000000000002,"15min":0.25700000000000001},"max":{"1min":3.6499999999999999,"5min":26.719999999999999,"15min":26.719999999999999},"last":0.34000000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.89300000000000002,"15min":0.70999999999999996},"min":{"1min":0.34799999999999998,"5min":0.309,"15min":0.30599999999999999},"max":{"1min":3.4039999999999999,"5min":24.356000000000002,"15min":24.356000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.96999999999999997,"15min":0.74299999999999999},"min":{"1min":0.39000000000000001,"5min":0.32800000000000001,"15min":0.32400000000000001},"max":{"1min":3.3919999999999999,"5min":20.707000000000001,"15min":20.707000000000001},"last":0.60299999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:09:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.83999999999999997,"15min":0.70399999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.748,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":1.081,"15min":0.80700000000000005},"min":{"1min":0.41299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85099999999999998,"5min":26.606000000000002,"15min":26.606000000000002},"last":0.61299999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":1.1870000000000001,"15min":0.81499999999999995},"min":{"1min":0.46200000000000002,"5min":0.376,"15min":0.376},"max":{"1min":1.4510000000000001,"5min":24.381,"15min":24.381},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.996,"15min":0.75},"min":{"1min":0.42199999999999999,"5min":0.42099999999999999,"15min":0.39100000000000001},"max":{"1min":0.93600000000000005,"5min":28.706,"15min":28.706},"last":0.64400000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:09:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.099,"15min":0.77800000000000002},"min":{"1min":0.46000000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76200000000000001,"5min":26.658000000000001,"15min":26.658000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.93400000000000005,"15min":0.73699999999999999},"min":{"1min":0.439,"5min":0.39100000000000001,"15min":0.379},"max":{"1min":0.76700000000000002,"5min":24.402000000000001,"15min":24.402000000000001},"last":0.63100000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182524,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320012,"kb_used_data":6400,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93425140,"statfs":{"total":95995035648,"available":95667343360,"internally_reserved":0,"allocated":6553600,"data_stored":3734430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":1.454,"15min":0.71899999999999997},"min":{"1min":0.23000000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.64200000000000002,"5min":77.662000000000006,"15min":77.662000000000006},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.74199999999999999,"15min":0.50900000000000001},"min":{"1min":0.22700000000000001,"5min":0.22600000000000001,"15min":0.19700000000000001},"max":{"1min":0.60999999999999999,"5min":17.456,"15min":17.456},"last":0.313}]},{"osd":1,"last update":"Mon Feb 19 21:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.84499999999999997,"15min":0.54000000000000004},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.22800000000000001},"max":{"1min":6.9299999999999997,"5min":17.327999999999999,"15min":17.327999999999999},"last":0.247},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.77000000000000002,"15min":0.51700000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":7.3620000000000001,"5min":15.004,"15min":15.004},"last":0.33000000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":1.198,"15min":0.67600000000000005},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.54200000000000004,"5min":56.890999999999998,"15min":56.890999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.376,"5min":1.1619999999999999,"15min":0.66000000000000003},"min":{"1min":0.24399999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59199999999999997,"5min":44.600999999999999,"15min":44.600999999999999},"last":0.29299999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:09:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.82099999999999995,"15min":0.63800000000000001},"min":{"1min":0.33100000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.74399999999999999,"5min":12.847,"15min":12.847},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.026,"15min":0.71599999999999997},"min":{"1min":0.34000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.83899999999999997,"5min":26.393999999999998,"15min":26.393999999999998},"last":0.35499999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:09:34 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92600000000000005,"15min":0.69299999999999995},"min":{"1min":0.34699999999999998,"5min":0.313,"15min":0.313},"max":{"1min":9.2360000000000007,"5min":17.401,"15min":17.401},"last":0.373},{"interface":"front","average":{"1min":0.626,"5min":0.85599999999999998,"15min":0.66100000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.9690000000000001,"5min":12.882,"15min":12.882},"last":0.50700000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.84999999999999998,"15min":0.65900000000000003},"min":{"1min":0.39000000000000001,"5min":0.35299999999999998,"15min":0.34300000000000003},"max":{"1min":1.9790000000000001,"5min":17.367000000000001,"15min":17.367000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.84599999999999997,"15min":0.67500000000000004},"min":{"1min":0.35699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.88,"5min":12.904,"15min":12.904},"last":0.44800000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:09:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.83799999999999997,"15min":0.65600000000000003},"min":{"1min":0.38600000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":17.428999999999998,"15min":17.428999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.626,"5min":1.2509999999999999,"15min":0.80100000000000005},"min":{"1min":0.41899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85899999999999999,"5min":44.378,"15min":44.378},"last":0.42099999999999999}]}]},{"osd":4,"up_from":0,"seq":133143986296,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":37,"seq":158913790067,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311920,"kb_used_data":6496,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433232,"statfs":{"total":95995035648,"available":95675629568,"internally_reserved":0,"allocated":6651904,"data_stored":3724526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":4.6790000000000003,"15min":1.8919999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.69999999999999996,"5min":276.47899999999998,"15min":276.47899999999998},"last":0.311},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.6759999999999999,"15min":0.91700000000000004},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.95299999999999996,"5min":52.887,"15min":52.887},"last":0.497}]},{"osd":1,"last update":"Mon Feb 19 21:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":2.8250000000000002,"15min":1.282},"min":{"1min":0.254,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.599,"5min":120.011,"15min":120.011},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.52800000000000002,"5min":2.024,"15min":1.0309999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.97699999999999998,"5min":118.105,"15min":118.105},"last":0.47199999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":1.9690000000000001,"15min":1.006},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66600000000000004,"5min":82.242000000000004,"15min":82.242000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":3.4910000000000001,"15min":1.514},"min":{"1min":0.309,"5min":0.309,"15min":0.30199999999999999},"max":{"1min":4.5019999999999998,"5min":232.84999999999999,"15min":232.84999999999999},"last":0.39100000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":2.1680000000000001,"15min":1.075},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.4640000000000004,"5min":109.70099999999999,"15min":109.70099999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.6599999999999999,"15min":0.91600000000000004},"min":{"1min":0.34799999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72199999999999998,"5min":65.519000000000005,"15min":65.519000000000005},"last":0.437}]},{"osd":4,"last update":"Mon Feb 19 21:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.443,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.0129999999999999,"5min":1.861,"15min":1.861},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45300000000000001,"15min":0.42199999999999999},"min":{"1min":0.308,"5min":0.248,"15min":0.248},"max":{"1min":0.72499999999999998,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":0.45600000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46300000000000002,"15min":0.439},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.746,"5min":1.823,"15min":1.823},"last":0.379},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47299999999999998,"15min":0.45200000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.66300000000000003,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.505}]},{"osd":7,"last update":"Mon Feb 19 21:09:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.36199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.052,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.501,"15min":0.50800000000000001},"min":{"1min":0.38400000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.871,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.53100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4317184,"data_stored":4000776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4259840,"data_stored":4000599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":96,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4243456,"data_stored":4000579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:09:51.803 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:09:51.803 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:09:51.820 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:51.845 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:51.993 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:51.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:52.097 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:52.126 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:52.224 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:52.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:52.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:52.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:52.272+0000 7fda10ff9700 1 -- 172.21.15.163:0/1184515762 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v409) v1 ==== 97+0+0 (unknown 1940589742 0 0) 0x7fda040480b0 con 0x7fda14100780 2024-02-19T21:09:52.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:52.275+0000 7fda19e7d700 10 librados: watch_flush enter 2024-02-19T21:09:52.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:52.275+0000 7fda19e7d700 10 librados: watch_flush exit 2024-02-19T21:09:52.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:52.275+0000 7fda19e7d700 20 client.48610.objecter shutdown clearing up homeless session... 2024-02-19T21:09:52.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:52.275+0000 7fda19e7d700 10 client.48610.objecter successfully canceled tick 2024-02-19T21:09:52.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:52.275+0000 7fda19e7d700 1 -- 172.21.15.163:0/1184515762 >> v1:172.21.15.163:6800/1552514681 conn(0x7fd9fc063da0 legacy=0x7fd9fc066260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:52.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:52.275+0000 7fda19e7d700 1 -- 172.21.15.163:0/1184515762 >> v1:172.21.15.193:6789/0 conn(0x7fda14100780 legacy=0x7fda140fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:52.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:52.275+0000 7fda19e7d700 1 -- 172.21.15.163:0/1184515762 shutdown_connections 2024-02-19T21:09:52.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:52.275+0000 7fda19e7d700 1 -- 172.21.15.163:0/1184515762 >> 172.21.15.163:0/1184515762 conn(0x7fda140f35a0 msgr2=0x7fda14107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:52.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:52.275+0000 7fda19e7d700 1 -- 172.21.15.163:0/1184515762 shutdown_connections 2024-02-19T21:09:52.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:52.275+0000 7fda19e7d700 1 -- 172.21.15.163:0/1184515762 wait complete. 2024-02-19T21:09:52.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:52.275+0000 7fda19e7d700 1 librados: shutdown 2024-02-19T21:09:52.273 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:09:52.297 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:51.206153+0000 mgr.y (mgr.14550) 364 : cluster [DBG] pgmap v584: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:09:52.298 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:51.565014+0000 mon.b (mon.1) 576 : audit [INF] from='client.? v1:172.21.15.163:0/1184515762' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:52.298 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:51.565313+0000 mon.a (mon.0) 2497 : audit [INF] from='client.48610 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:52.298 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:52.261889+0000 mon.a (mon.0) 2498 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:09:52.298 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:52.270555+0000 mon.a (mon.0) 2499 : audit [INF] from='client.48610 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:09:52.298 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:52.270615+0000 mon.a (mon.0) 2500 : cluster [DBG] osdmap e409: 8 total, 8 up, 7 in 2024-02-19T21:09:52.329 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:52.357 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:52.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:51.206153+0000 mgr.y (mgr.14550) 364 : cluster [DBG] pgmap v584: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:09:52.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:51.565014+0000 mon.b (mon.1) 576 : audit [INF] from='client.? v1:172.21.15.163:0/1184515762' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:52.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:51.565313+0000 mon.a (mon.0) 2497 : audit [INF] from='client.48610 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:52.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:52.261889+0000 mon.a (mon.0) 2498 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:09:52.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:52.270555+0000 mon.a (mon.0) 2499 : audit [INF] from='client.48610 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:09:52.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:52.270615+0000 mon.a (mon.0) 2500 : cluster [DBG] osdmap e409: 8 total, 8 up, 7 in 2024-02-19T21:09:52.616 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:51.206153+0000 mgr.y (mgr.14550) 364 : cluster [DBG] pgmap v584: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:09:52.617 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:51.565014+0000 mon.b (mon.1) 576 : audit [INF] from='client.? v1:172.21.15.163:0/1184515762' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:52.617 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:51.565313+0000 mon.a (mon.0) 2497 : audit [INF] from='client.48610 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:09:52.617 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:52.261889+0000 mon.a (mon.0) 2498 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:09:52.617 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:52.270555+0000 mon.a (mon.0) 2499 : audit [INF] from='client.48610 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:09:52.617 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:52.270615+0000 mon.a (mon.0) 2500 : cluster [DBG] osdmap e409: 8 total, 8 up, 7 in 2024-02-19T21:09:53.005 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:53.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:53.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:53.109 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:53.140 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:53.339 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:53.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:53.443 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:53.471 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:53.495 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:09:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:09:53] "GET /metrics HTTP/1.1" 200 189443 "" "Prometheus/2.33.4" 2024-02-19T21:09:53.526 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:53.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:53.581 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:09:53.581 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:09:53.581 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-02-19T21:09:53.581 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-02-19T21:09:53.581 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:09:53.642 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:09:53.668 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:54.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.140+0000 7fba562c4700 1 -- 172.21.15.163:0/3929197499 >> v1:172.21.15.163:6789/0 conn(0x7fba501064e0 legacy=0x7fba50108950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:54.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.140+0000 7fba562c4700 1 -- 172.21.15.163:0/3929197499 shutdown_connections 2024-02-19T21:09:54.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.140+0000 7fba562c4700 1 -- 172.21.15.163:0/3929197499 >> 172.21.15.163:0/3929197499 conn(0x7fba500f3570 msgr2=0x7fba500f5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:54.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.140+0000 7fba562c4700 1 -- 172.21.15.163:0/3929197499 shutdown_connections 2024-02-19T21:09:54.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.140+0000 7fba562c4700 1 -- 172.21.15.163:0/3929197499 wait complete. 2024-02-19T21:09:54.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.140+0000 7fba562c4700 1 librados: starting msgr at 2024-02-19T21:09:54.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.140+0000 7fba562c4700 1 librados: starting objecter 2024-02-19T21:09:54.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.140+0000 7fba562c4700 1 Processor -- start 2024-02-19T21:09:54.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.140+0000 7fba562c4700 1 -- start start 2024-02-19T21:09:54.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.140+0000 7fba562c4700 1 librados: setting wanted keys 2024-02-19T21:09:54.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.140+0000 7fba562c4700 1 librados: calling monclient init 2024-02-19T21:09:54.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba562c4700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba50119590 con 0x7fba5006bc00 2024-02-19T21:09:54.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba562c4700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba5011a730 con 0x7fba501064e0 2024-02-19T21:09:54.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba562c4700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fba5011b930 con 0x7fba5006b110 2024-02-19T21:09:54.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4f7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fba5006bc00 0x7fba501031a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:35792/0 (socket says 172.21.15.163:35792) 2024-02-19T21:09:54.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4f7fe700 1 -- 172.21.15.163:0/2557079264 learned_addr learned my addr 172.21.15.163:0/2557079264 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:54.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4ffff700 1 --1- 172.21.15.163:0/2557079264 >> v1:172.21.15.163:6790/0 conn(0x7fba5006b110 0x7fba50102a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:33296/0 (socket says 172.21.15.163:33296) 2024-02-19T21:09:54.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 10 client.?.objecter ms_handle_connect 0x7fba5006b110 2024-02-19T21:09:54.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:54.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 10 client.?.objecter ms_handle_connect 0x7fba5006bc00 2024-02-19T21:09:54.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:54.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 10 client.?.objecter ms_handle_connect 0x7fba501064e0 2024-02-19T21:09:54.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:54.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006716281 0 0) 0x7fba50119590 con 0x7fba5006bc00 2024-02-19T21:09:54.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba38003800 con 0x7fba5006bc00 2024-02-19T21:09:54.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2099723238 0 0) 0x7fba5011b930 con 0x7fba5006b110 2024-02-19T21:09:54.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba50119590 con 0x7fba5006b110 2024-02-19T21:09:54.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181033264 0 0) 0x7fba5011a730 con 0x7fba501064e0 2024-02-19T21:09:54.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba5011b930 con 0x7fba501064e0 2024-02-19T21:09:54.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3897732100 0 0) 0x7fba38003800 con 0x7fba5006bc00 2024-02-19T21:09:54.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba5011a730 con 0x7fba5006bc00 2024-02-19T21:09:54.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fba440019f0 con 0x7fba5006bc00 2024-02-19T21:09:54.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 295564171 0 0) 0x7fba50119590 con 0x7fba5006b110 2024-02-19T21:09:54.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba38003800 con 0x7fba5006b110 2024-02-19T21:09:54.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2519795558 0 0) 0x7fba5011b930 con 0x7fba501064e0 2024-02-19T21:09:54.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba50119590 con 0x7fba501064e0 2024-02-19T21:09:54.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fba40001840 con 0x7fba5006b110 2024-02-19T21:09:54.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fba48002f60 con 0x7fba501064e0 2024-02-19T21:09:54.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 905724768 0 0) 0x7fba5011a730 con 0x7fba5006bc00 2024-02-19T21:09:54.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 >> v1:172.21.15.163:6790/0 conn(0x7fba5006b110 legacy=0x7fba50102a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:54.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 >> v1:172.21.15.193:6789/0 conn(0x7fba501064e0 legacy=0x7fba50117c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:54.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5011cb30 con 0x7fba5006bc00 2024-02-19T21:09:54.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba50119780 con 0x7fba5006bc00 2024-02-19T21:09:54.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba562c4700 10 client.23079.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:54.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba50119bf0 con 0x7fba5006bc00 2024-02-19T21:09:54.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.141+0000 7fba562c4700 1 librados: init done 2024-02-19T21:09:54.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.142+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fba44002550 con 0x7fba5006bc00 2024-02-19T21:09:54.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.142+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fba44005bf0 con 0x7fba5006bc00 2024-02-19T21:09:54.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.142+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fba3c004f00 con 0x7fba5006bc00 2024-02-19T21:09:54.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.143+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fba44018b40 con 0x7fba5006bc00 2024-02-19T21:09:54.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.143+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(410..410 src has 43..410) v4 ==== 7918+0+0 (unknown 1853679874 0 0) 0x7fba44078ce0 con 0x7fba5006bc00 2024-02-19T21:09:54.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.143+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 osd_map(410..410 src has 43..410) v4 2024-02-19T21:09:54.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.143+0000 7fba4d775700 3 client.23079.objecter handle_osd_map got epochs [410,410] > 0 2024-02-19T21:09:54.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.143+0000 7fba4d775700 3 client.23079.objecter handle_osd_map decoding full epoch 410 2024-02-19T21:09:54.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.143+0000 7fba4d775700 20 client.23079.objecter dump_active .. 0 homeless 2024-02-19T21:09:54.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.143+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v410) v1 ==== 54+0+16 (unknown 941351051 0 2420000192) 0x7fba4407b080 con 0x7fba5006bc00 2024-02-19T21:09:54.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.143+0000 7fba4d775700 10 client.23079.objecter ms_handle_connect 0x7fba38063cc0 2024-02-19T21:09:54.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba562c4700 10 client.23079.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:54.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 1 2024-02-19T21:09:54.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c0051a0 osd=0 3 2024-02-19T21:09:54.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c0051a0 osd=0 3 2024-02-19T21:09:54.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c0051a0 osd=0 3 2024-02-19T21:09:54.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 1 2024-02-19T21:09:54.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c0051a0 osd=0 3 2024-02-19T21:09:54.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 0 1 2024-02-19T21:09:54.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c0051a0 osd=0 4 2024-02-19T21:09:54.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba562c4700 10 client.23079.objecter _send_command 1 2024-02-19T21:09:54.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba3c004950 con 0x7fba3c004530 2024-02-19T21:09:54.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba4d775700 10 client.23079.objecter ms_handle_connect 0x7fba3c004530 2024-02-19T21:09:54.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba3c004950 con 0x7fba3c004530 2024-02-19T21:09:54.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:09:54.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba4d775700 10 client.23079.objecter _finish_command 1 = system:0 2024-02-19T21:09:54.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c0051a0 osd=0 4 2024-02-19T21:09:54.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.144+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 0 1 2024-02-19T21:09:54.150 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:54.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.160+0000 7fba562c4700 10 client.23079.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:54.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.160+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 2 2024-02-19T21:09:54.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.160+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c0051a0 osd=0 3 2024-02-19T21:09:54.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.160+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c0051a0 osd=0 3 2024-02-19T21:09:54.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.160+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c0051a0 osd=0 3 2024-02-19T21:09:54.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.160+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 2 2024-02-19T21:09:54.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.160+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c0051a0 osd=0 3 2024-02-19T21:09:54.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.160+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 0 2 2024-02-19T21:09:54.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.160+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c0051a0 osd=0 4 2024-02-19T21:09:54.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.160+0000 7fba562c4700 10 client.23079.objecter _send_command 2 2024-02-19T21:09:54.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.160+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba3c004d30 con 0x7fba3c004530 2024-02-19T21:09:54.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.161+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba3c004d30 con 0x7fba3c004530 2024-02-19T21:09:54.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.161+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:54.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.161+0000 7fba4d775700 10 client.23079.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.161+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c0051a0 osd=0 4 2024-02-19T21:09:54.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.161+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 0 2 2024-02-19T21:09:54.159 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:09:54.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.162+0000 7fba562c4700 10 client.23079.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:54.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.162+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 3 2024-02-19T21:09:54.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.162+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c003fe0 osd=1 3 2024-02-19T21:09:54.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.162+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c003fe0 osd=1 3 2024-02-19T21:09:54.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.162+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c003fe0 osd=1 3 2024-02-19T21:09:54.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.162+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 3 2024-02-19T21:09:54.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.162+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c003fe0 osd=1 3 2024-02-19T21:09:54.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.162+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 1 3 2024-02-19T21:09:54.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.162+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c003fe0 osd=1 4 2024-02-19T21:09:54.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.162+0000 7fba562c4700 10 client.23079.objecter _send_command 3 2024-02-19T21:09:54.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.162+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba3c00c000 con 0x7fba3c004150 2024-02-19T21:09:54.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.162+0000 7fba4d775700 10 client.23079.objecter ms_handle_connect 0x7fba3c004150 2024-02-19T21:09:54.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.163+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba3c00c000 con 0x7fba3c004150 2024-02-19T21:09:54.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.163+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:09:54.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.163+0000 7fba4d775700 10 client.23079.objecter _finish_command 3 = system:0 2024-02-19T21:09:54.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.163+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c003fe0 osd=1 4 2024-02-19T21:09:54.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.163+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 1 3 2024-02-19T21:09:54.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.178+0000 7fba562c4700 10 client.23079.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:54.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.178+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 4 2024-02-19T21:09:54.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.178+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c003fe0 osd=1 3 2024-02-19T21:09:54.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.178+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c003fe0 osd=1 3 2024-02-19T21:09:54.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.178+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c003fe0 osd=1 3 2024-02-19T21:09:54.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.178+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 4 2024-02-19T21:09:54.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.178+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c003fe0 osd=1 3 2024-02-19T21:09:54.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.178+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 1 4 2024-02-19T21:09:54.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.178+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c003fe0 osd=1 4 2024-02-19T21:09:54.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.178+0000 7fba562c4700 10 client.23079.objecter _send_command 4 2024-02-19T21:09:54.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.178+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba3c0018b0 con 0x7fba3c004150 2024-02-19T21:09:54.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba3c0018b0 con 0x7fba3c004150 2024-02-19T21:09:54.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:54.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba4d775700 10 client.23079.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c003fe0 osd=1 4 2024-02-19T21:09:54.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 1 4 2024-02-19T21:09:54.178 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:09:54.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba562c4700 10 client.23079.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:54.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 5 2024-02-19T21:09:54.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c003630 osd=2 3 2024-02-19T21:09:54.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c003630 osd=2 3 2024-02-19T21:09:54.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c003630 osd=2 3 2024-02-19T21:09:54.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 5 2024-02-19T21:09:54.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c003630 osd=2 3 2024-02-19T21:09:54.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 2 5 2024-02-19T21:09:54.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c003630 osd=2 4 2024-02-19T21:09:54.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba562c4700 10 client.23079.objecter _send_command 5 2024-02-19T21:09:54.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.180+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba3c00dec0 con 0x7fba3c00d320 2024-02-19T21:09:54.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.181+0000 7fba4d775700 10 client.23079.objecter ms_handle_connect 0x7fba3c00d320 2024-02-19T21:09:54.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.181+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba3c00dec0 con 0x7fba3c00d320 2024-02-19T21:09:54.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.181+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:09:54.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.181+0000 7fba4d775700 10 client.23079.objecter _finish_command 5 = system:0 2024-02-19T21:09:54.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.181+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c003630 osd=2 4 2024-02-19T21:09:54.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.181+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 2 5 2024-02-19T21:09:54.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.197+0000 7fba562c4700 10 client.23079.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:54.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.197+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 6 2024-02-19T21:09:54.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.197+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c003630 osd=2 3 2024-02-19T21:09:54.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.197+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c003630 osd=2 3 2024-02-19T21:09:54.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.197+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c003630 osd=2 3 2024-02-19T21:09:54.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.197+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 6 2024-02-19T21:09:54.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.197+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c003630 osd=2 3 2024-02-19T21:09:54.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.197+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 2 6 2024-02-19T21:09:54.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.197+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c003630 osd=2 4 2024-02-19T21:09:54.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.197+0000 7fba562c4700 10 client.23079.objecter _send_command 6 2024-02-19T21:09:54.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.197+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba3c0030a0 con 0x7fba3c00d320 2024-02-19T21:09:54.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba3c0030a0 con 0x7fba3c00d320 2024-02-19T21:09:54.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:54.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba4d775700 10 client.23079.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c003630 osd=2 4 2024-02-19T21:09:54.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 2 6 2024-02-19T21:09:54.195 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:09:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba562c4700 10 client.23079.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 7 2024-02-19T21:09:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c0034b0 osd=3 3 2024-02-19T21:09:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c0034b0 osd=3 3 2024-02-19T21:09:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c0034b0 osd=3 3 2024-02-19T21:09:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 7 2024-02-19T21:09:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c0034b0 osd=3 3 2024-02-19T21:09:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 3 7 2024-02-19T21:09:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c0034b0 osd=3 4 2024-02-19T21:09:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba562c4700 10 client.23079.objecter _send_command 7 2024-02-19T21:09:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.198+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba3c015b10 con 0x7fba3c011eb0 2024-02-19T21:09:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.199+0000 7fba4d775700 10 client.23079.objecter ms_handle_connect 0x7fba3c011eb0 2024-02-19T21:09:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.199+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba3c015b10 con 0x7fba3c011eb0 2024-02-19T21:09:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.199+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:09:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.199+0000 7fba4d775700 10 client.23079.objecter _finish_command 7 = system:0 2024-02-19T21:09:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.199+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c0034b0 osd=3 4 2024-02-19T21:09:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.199+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 3 7 2024-02-19T21:09:54.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.213+0000 7fba562c4700 10 client.23079.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:54.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.213+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 8 2024-02-19T21:09:54.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.213+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c0034b0 osd=3 3 2024-02-19T21:09:54.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.213+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c0034b0 osd=3 3 2024-02-19T21:09:54.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.213+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c0034b0 osd=3 3 2024-02-19T21:09:54.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.213+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 8 2024-02-19T21:09:54.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.213+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c0034b0 osd=3 3 2024-02-19T21:09:54.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.213+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 3 8 2024-02-19T21:09:54.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.213+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c0034b0 osd=3 4 2024-02-19T21:09:54.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.213+0000 7fba562c4700 10 client.23079.objecter _send_command 8 2024-02-19T21:09:54.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.213+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba3c00e270 con 0x7fba3c011eb0 2024-02-19T21:09:54.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba3c00e270 con 0x7fba3c011eb0 2024-02-19T21:09:54.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:54.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba4d775700 10 client.23079.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c0034b0 osd=3 4 2024-02-19T21:09:54.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 3 8 2024-02-19T21:09:54.212 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:09:54.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba562c4700 10 client.23079.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:54.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 9 2024-02-19T21:09:54.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c00e270 osd=4 3 2024-02-19T21:09:54.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c00e270 osd=4 3 2024-02-19T21:09:54.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c00e270 osd=4 3 2024-02-19T21:09:54.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 9 2024-02-19T21:09:54.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c00e270 osd=4 3 2024-02-19T21:09:54.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 4 9 2024-02-19T21:09:54.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c00e270 osd=4 4 2024-02-19T21:09:54.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba562c4700 10 client.23079.objecter _send_command 9 2024-02-19T21:09:54.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.215+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba3c00e560 con 0x7fba3c00ce80 2024-02-19T21:09:54.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.216+0000 7fba4d775700 10 client.23079.objecter ms_handle_connect 0x7fba3c00ce80 2024-02-19T21:09:54.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.216+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba3c00e560 con 0x7fba3c00ce80 2024-02-19T21:09:54.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.216+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:09:54.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.216+0000 7fba4d775700 10 client.23079.objecter _finish_command 9 = system:0 2024-02-19T21:09:54.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.216+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c00e270 osd=4 4 2024-02-19T21:09:54.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.216+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 4 9 2024-02-19T21:09:54.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.231+0000 7fba562c4700 10 client.23079.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:54.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.231+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 10 2024-02-19T21:09:54.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.231+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c00e270 osd=4 3 2024-02-19T21:09:54.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.231+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c00e270 osd=4 3 2024-02-19T21:09:54.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.231+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c00e270 osd=4 3 2024-02-19T21:09:54.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.231+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 10 2024-02-19T21:09:54.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.231+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c00e270 osd=4 3 2024-02-19T21:09:54.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.231+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 4 10 2024-02-19T21:09:54.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.231+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c00e270 osd=4 4 2024-02-19T21:09:54.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.231+0000 7fba562c4700 10 client.23079.objecter _send_command 10 2024-02-19T21:09:54.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.231+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba3c00e560 con 0x7fba3c00ce80 2024-02-19T21:09:54.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba3c00e560 con 0x7fba3c00ce80 2024-02-19T21:09:54.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:54.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba4d775700 10 client.23079.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c00e270 osd=4 4 2024-02-19T21:09:54.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 4 10 2024-02-19T21:09:54.232 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:09:54.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba562c4700 10 client.23079.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:54.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 11 2024-02-19T21:09:54.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c00e560 osd=5 3 2024-02-19T21:09:54.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c00e560 osd=5 3 2024-02-19T21:09:54.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c00e560 osd=5 3 2024-02-19T21:09:54.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 11 2024-02-19T21:09:54.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c00e560 osd=5 3 2024-02-19T21:09:54.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 5 11 2024-02-19T21:09:54.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c00e560 osd=5 4 2024-02-19T21:09:54.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba562c4700 10 client.23079.objecter _send_command 11 2024-02-19T21:09:54.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.233+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba3c018000 con 0x7fba3c017a50 2024-02-19T21:09:54.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.234+0000 7fba4d775700 10 client.23079.objecter ms_handle_connect 0x7fba3c017a50 2024-02-19T21:09:54.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.234+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba3c018000 con 0x7fba3c017a50 2024-02-19T21:09:54.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.234+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:09:54.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.234+0000 7fba4d775700 10 client.23079.objecter _finish_command 11 = system:0 2024-02-19T21:09:54.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.234+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c00e560 osd=5 4 2024-02-19T21:09:54.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.234+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 5 11 2024-02-19T21:09:54.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.249+0000 7fba562c4700 10 client.23079.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:54.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.249+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 12 2024-02-19T21:09:54.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.249+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c00e560 osd=5 3 2024-02-19T21:09:54.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.249+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c00e560 osd=5 3 2024-02-19T21:09:54.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.249+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c00e560 osd=5 3 2024-02-19T21:09:54.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.249+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 12 2024-02-19T21:09:54.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.249+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c00e560 osd=5 3 2024-02-19T21:09:54.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.249+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 5 12 2024-02-19T21:09:54.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.249+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c00e560 osd=5 4 2024-02-19T21:09:54.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.249+0000 7fba562c4700 10 client.23079.objecter _send_command 12 2024-02-19T21:09:54.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.249+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba3c018000 con 0x7fba3c017a50 2024-02-19T21:09:54.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.250+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba3c018000 con 0x7fba3c017a50 2024-02-19T21:09:54.247 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.250+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:54.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.250+0000 7fba4d775700 10 client.23079.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.250+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c00e560 osd=5 4 2024-02-19T21:09:54.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.250+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 5 12 2024-02-19T21:09:54.248 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:09:54.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.251+0000 7fba562c4700 10 client.23079.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:54.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.251+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 13 2024-02-19T21:09:54.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.251+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c018000 osd=6 3 2024-02-19T21:09:54.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.251+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c018000 osd=6 3 2024-02-19T21:09:54.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.251+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c018000 osd=6 3 2024-02-19T21:09:54.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.251+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 13 2024-02-19T21:09:54.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.251+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c018000 osd=6 3 2024-02-19T21:09:54.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.251+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 6 13 2024-02-19T21:09:54.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.251+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c018000 osd=6 4 2024-02-19T21:09:54.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.251+0000 7fba562c4700 10 client.23079.objecter _send_command 13 2024-02-19T21:09:54.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.251+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba3c022c60 con 0x7fba3c01f110 2024-02-19T21:09:54.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.251+0000 7fba4d775700 10 client.23079.objecter ms_handle_connect 0x7fba3c01f110 2024-02-19T21:09:54.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.252+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba3c022c60 con 0x7fba3c01f110 2024-02-19T21:09:54.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.252+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:09:54.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.252+0000 7fba4d775700 10 client.23079.objecter _finish_command 13 = system:0 2024-02-19T21:09:54.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.252+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c018000 osd=6 4 2024-02-19T21:09:54.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.252+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 6 13 2024-02-19T21:09:54.256 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-19T21:09:54.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.270+0000 7fba562c4700 10 client.23079.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:54.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.270+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 14 2024-02-19T21:09:54.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.270+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c018000 osd=6 3 2024-02-19T21:09:54.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.270+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c018000 osd=6 3 2024-02-19T21:09:54.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.270+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c018000 osd=6 3 2024-02-19T21:09:54.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.270+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 14 2024-02-19T21:09:54.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.270+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c018000 osd=6 3 2024-02-19T21:09:54.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.270+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 6 14 2024-02-19T21:09:54.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.270+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c018000 osd=6 4 2024-02-19T21:09:54.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.270+0000 7fba562c4700 10 client.23079.objecter _send_command 14 2024-02-19T21:09:54.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.270+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba3c022c20 con 0x7fba3c01f110 2024-02-19T21:09:54.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.275+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba3c022c20 con 0x7fba3c01f110 2024-02-19T21:09:54.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.275+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:54.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.275+0000 7fba4d775700 10 client.23079.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.275+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c018000 osd=6 4 2024-02-19T21:09:54.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.275+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 6 14 2024-02-19T21:09:54.278 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:09:54.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.281+0000 7fba562c4700 10 client.23079.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:09:54.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.281+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 15 2024-02-19T21:09:54.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.281+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c022c20 osd=7 3 2024-02-19T21:09:54.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.281+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c022c20 osd=7 3 2024-02-19T21:09:54.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.281+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c022c20 osd=7 3 2024-02-19T21:09:54.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.281+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 15 2024-02-19T21:09:54.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.281+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c022c20 osd=7 3 2024-02-19T21:09:54.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.281+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 7 15 2024-02-19T21:09:54.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.281+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c022c20 osd=7 4 2024-02-19T21:09:54.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.281+0000 7fba562c4700 10 client.23079.objecter _send_command 15 2024-02-19T21:09:54.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.281+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba3c023e80 con 0x7fba3c024920 2024-02-19T21:09:54.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.281+0000 7fba4d775700 10 client.23079.objecter ms_handle_connect 0x7fba3c024920 2024-02-19T21:09:54.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.282+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fba3c023e80 con 0x7fba3c024920 2024-02-19T21:09:54.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.282+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:09:54.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.282+0000 7fba4d775700 10 client.23079.objecter _finish_command 15 = system:0 2024-02-19T21:09:54.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.282+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c022c20 osd=7 4 2024-02-19T21:09:54.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.282+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 7 15 2024-02-19T21:09:54.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.298+0000 7fba562c4700 10 client.23079.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:09:54.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.298+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign -1 16 2024-02-19T21:09:54.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.298+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c022c20 osd=7 3 2024-02-19T21:09:54.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.298+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c022c20 osd=7 3 2024-02-19T21:09:54.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.298+0000 7fba562c4700 20 client.23079.objecter _get_session s=0x7fba3c022c20 osd=7 3 2024-02-19T21:09:54.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.298+0000 7fba562c4700 15 client.23079.objecter _session_command_op_remove -1 16 2024-02-19T21:09:54.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.298+0000 7fba562c4700 20 client.23079.objecter get_session s=0x7fba3c022c20 osd=7 3 2024-02-19T21:09:54.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.298+0000 7fba562c4700 15 client.23079.objecter _session_command_op_assign 7 16 2024-02-19T21:09:54.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.298+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c022c20 osd=7 4 2024-02-19T21:09:54.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.298+0000 7fba562c4700 10 client.23079.objecter _send_command 16 2024-02-19T21:09:54.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.298+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fba3c023e40 con 0x7fba3c024920 2024-02-19T21:09:54.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.299+0000 7fba4d775700 1 -- 172.21.15.163:0/2557079264 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fba3c023e40 con 0x7fba3c024920 2024-02-19T21:09:54.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.299+0000 7fba4d775700 10 client.23079.objecter ms_dispatch 0x7fba50001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:09:54.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.299+0000 7fba4d775700 10 client.23079.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.299+0000 7fba4d775700 20 client.23079.objecter put_session s=0x7fba3c022c20 osd=7 4 2024-02-19T21:09:54.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.299+0000 7fba4d775700 15 client.23079.objecter _session_command_op_remove 7 16 2024-02-19T21:09:54.299 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:09:54.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 10 librados: watch_flush enter 2024-02-19T21:09:54.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 10 librados: watch_flush exit 2024-02-19T21:09:54.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 10 client.23079.objecter close_session for osd.0 2024-02-19T21:09:54.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 >> v1:172.21.15.163:6801/1559418115 conn(0x7fba3c004530 legacy=0x7fba3c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:54.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c0051a0 osd=0 1 2024-02-19T21:09:54.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 10 client.23079.objecter close_session for osd.1 2024-02-19T21:09:54.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 >> v1:172.21.15.163:6805/3125047432 conn(0x7fba3c004150 legacy=0x7fba3c00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:54.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c003fe0 osd=1 1 2024-02-19T21:09:54.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 10 client.23079.objecter close_session for osd.2 2024-02-19T21:09:54.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 >> v1:172.21.15.163:6809/1563158487 conn(0x7fba3c00d320 legacy=0x7fba3c00d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:54.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c003630 osd=2 1 2024-02-19T21:09:54.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 10 client.23079.objecter close_session for osd.3 2024-02-19T21:09:54.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 >> v1:172.21.15.163:6813/2112279769 conn(0x7fba3c011eb0 legacy=0x7fba3c0142c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:54.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c0034b0 osd=3 1 2024-02-19T21:09:54.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 10 client.23079.objecter close_session for osd.4 2024-02-19T21:09:54.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 >> v1:172.21.15.193:6800/3129716965 conn(0x7fba3c00ce80 legacy=0x7fba3c015cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:54.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c00e270 osd=4 1 2024-02-19T21:09:54.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 10 client.23079.objecter close_session for osd.5 2024-02-19T21:09:54.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 >> v1:172.21.15.193:6804/805161991 conn(0x7fba3c017a50 legacy=0x7fba3c01d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:54.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c00e560 osd=5 1 2024-02-19T21:09:54.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 10 client.23079.objecter close_session for osd.6 2024-02-19T21:09:54.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 >> v1:172.21.15.193:6808/787604615 conn(0x7fba3c01f110 legacy=0x7fba3c021500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:54.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c018000 osd=6 1 2024-02-19T21:09:54.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 10 client.23079.objecter close_session for osd.7 2024-02-19T21:09:54.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 >> v1:172.21.15.193:6812/157927432 conn(0x7fba3c024920 legacy=0x7fba3c024d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:54.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba54861700 1 -- 172.21.15.163:0/2557079264 reap_dead start 2024-02-19T21:09:54.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 20 client.23079.objecter put_session s=0x7fba3c022c20 osd=7 1 2024-02-19T21:09:54.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 20 client.23079.objecter shutdown clearing up homeless session... 2024-02-19T21:09:54.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 10 client.23079.objecter successfully canceled tick 2024-02-19T21:09:54.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba54861700 1 -- 172.21.15.163:0/2557079264 reap_dead start 2024-02-19T21:09:54.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba54861700 1 -- 172.21.15.163:0/2557079264 reap_dead start 2024-02-19T21:09:54.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba54861700 1 -- 172.21.15.163:0/2557079264 reap_dead start 2024-02-19T21:09:54.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba54861700 1 -- 172.21.15.163:0/2557079264 reap_dead start 2024-02-19T21:09:54.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 >> v1:172.21.15.163:6800/1552514681 conn(0x7fba38063cc0 legacy=0x7fba38066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:54.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 >> v1:172.21.15.163:6789/0 conn(0x7fba5006bc00 legacy=0x7fba501031a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:54.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 shutdown_connections 2024-02-19T21:09:54.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 >> 172.21.15.163:0/2557079264 conn(0x7fba500f3570 msgr2=0x7fba501088f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:54.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 shutdown_connections 2024-02-19T21:09:54.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.300+0000 7fba562c4700 1 -- 172.21.15.163:0/2557079264 wait complete. 2024-02-19T21:09:54.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:54.301+0000 7fba562c4700 1 librados: shutdown 2024-02-19T21:09:54.303 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.303 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.303 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.303 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.303 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.303 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.304 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.304 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:09:54.538 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:09:54.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:53.206617+0000 mgr.y (mgr.14550) 365 : cluster [DBG] pgmap v587: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:09:54.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:53.267155+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e410: 8 total, 8 up, 7 in 2024-02-19T21:09:54.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:54.101206+0000 mon.a (mon.0) 2502 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:54.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:54.143622+0000 mon.a (mon.0) 2503 : audit [DBG] from='client.? v1:172.21.15.163:0/2557079264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:54.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:53.206617+0000 mgr.y (mgr.14550) 365 : cluster [DBG] pgmap v587: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:09:54.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:53.267155+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e410: 8 total, 8 up, 7 in 2024-02-19T21:09:54.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:54.101206+0000 mon.a (mon.0) 2502 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:54.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:54.143622+0000 mon.a (mon.0) 2503 : audit [DBG] from='client.? v1:172.21.15.163:0/2557079264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:54.692 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:53.206617+0000 mgr.y (mgr.14550) 365 : cluster [DBG] pgmap v587: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:09:54.692 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:53.267155+0000 mon.a (mon.0) 2501 : cluster [DBG] osdmap e410: 8 total, 8 up, 7 in 2024-02-19T21:09:54.692 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:54.101206+0000 mon.a (mon.0) 2502 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:09:54.692 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:54.143622+0000 mon.a (mon.0) 2503 : audit [DBG] from='client.? v1:172.21.15.163:0/2557079264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:09:55.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:54.085524+0000 osd.7 (osd.7) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:55.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:54.085749+0000 osd.7 (osd.7) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:55.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:54.269223+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e411: 8 total, 8 up, 7 in 2024-02-19T21:09:55.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:54.085524+0000 osd.7 (osd.7) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:55.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:54.085749+0000 osd.7 (osd.7) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:55.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:54.269223+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e411: 8 total, 8 up, 7 in 2024-02-19T21:09:55.568 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:54.085524+0000 osd.7 (osd.7) 179 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:09:55.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:54.085749+0000 osd.7 (osd.7) 180 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:09:55.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:54.269223+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e411: 8 total, 8 up, 7 in 2024-02-19T21:09:55.907 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:09:55.919 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:09:55.923 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:55.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:09:56.025 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:56.052 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:56.253 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:56.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:56.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:56.316 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:09:56.316 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:09:56.318 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:55.207676+0000 mgr.y (mgr.14550) 366 : cluster [DBG] pgmap v590: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.0 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-02-19T21:09:56.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:56.389 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:56.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:55.207676+0000 mgr.y (mgr.14550) 366 : cluster [DBG] pgmap v590: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.0 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-02-19T21:09:56.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:55.207676+0000 mgr.y (mgr.14550) 366 : cluster [DBG] pgmap v590: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.0 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-02-19T21:09:57.014 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-19T21:09:57.094 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:09:57.283 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:09:57.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.236+0000 7fe2cfb82700 1 -- 172.21.15.163:0/849407333 >> v1:172.21.15.163:6789/0 conn(0x7fe2c8104290 legacy=0x7fe2c8106730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.236+0000 7fe2cfb82700 1 -- 172.21.15.163:0/849407333 shutdown_connections 2024-02-19T21:09:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.236+0000 7fe2cfb82700 1 -- 172.21.15.163:0/849407333 >> 172.21.15.163:0/849407333 conn(0x7fe2c80f2470 msgr2=0x7fe2c80f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.236+0000 7fe2cfb82700 1 -- 172.21.15.163:0/849407333 shutdown_connections 2024-02-19T21:09:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.236+0000 7fe2cfb82700 1 -- 172.21.15.163:0/849407333 wait complete. 2024-02-19T21:09:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.237+0000 7fe2cfb82700 1 librados: starting msgr at 2024-02-19T21:09:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.237+0000 7fe2cfb82700 1 librados: starting objecter 2024-02-19T21:09:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.237+0000 7fe2cfb82700 1 Processor -- start 2024-02-19T21:09:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.237+0000 7fe2cfb82700 1 -- start start 2024-02-19T21:09:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.237+0000 7fe2cfb82700 1 librados: setting wanted keys 2024-02-19T21:09:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.237+0000 7fe2cfb82700 1 librados: calling monclient init 2024-02-19T21:09:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.237+0000 7fe2cfb82700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe2c8102d80 con 0x7fe2c8104290 2024-02-19T21:09:57.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.237+0000 7fe2cfb82700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe2c8102f50 con 0x7fe2c80f89c0 2024-02-19T21:09:57.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.237+0000 7fe2cfb82700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe2c811bb50 con 0x7fe2c80f94b0 2024-02-19T21:09:57.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.237+0000 7fe2cd11d700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fe2c80f94b0 0x7fe2c8114ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:37000/0 (socket says 172.21.15.163:37000) 2024-02-19T21:09:57.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.237+0000 7fe2ce11f700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fe2c8104290 0x7fe2c8118410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:40674/0 (socket says 172.21.15.163:40674) 2024-02-19T21:09:57.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.237+0000 7fe2cd11d700 1 -- 172.21.15.163:0/955938063 learned_addr learned my addr 172.21.15.163:0/955938063 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:57.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.237+0000 7fe2beffd700 10 client.?.objecter ms_handle_connect 0x7fe2c80f94b0 2024-02-19T21:09:57.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:57.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 10 client.?.objecter ms_handle_connect 0x7fe2c8104290 2024-02-19T21:09:57.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:57.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464686013 0 0) 0x7fe2c8102d80 con 0x7fe2c8104290 2024-02-19T21:09:57.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b40033f0 con 0x7fe2c8104290 2024-02-19T21:09:57.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 10 client.?.objecter ms_handle_connect 0x7fe2c80f89c0 2024-02-19T21:09:57.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:57.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121314496 0 0) 0x7fe2c811bb50 con 0x7fe2c80f94b0 2024-02-19T21:09:57.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2c8102d80 con 0x7fe2c80f94b0 2024-02-19T21:09:57.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802744925 0 0) 0x7fe2c8102f50 con 0x7fe2c80f89c0 2024-02-19T21:09:57.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2c811bb50 con 0x7fe2c80f89c0 2024-02-19T21:09:57.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2051599445 0 0) 0x7fe2b40033f0 con 0x7fe2c8104290 2024-02-19T21:09:57.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2c8102f50 con 0x7fe2c8104290 2024-02-19T21:09:57.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe2c00031c0 con 0x7fe2c8104290 2024-02-19T21:09:57.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 556074762 0 0) 0x7fe2c8102d80 con 0x7fe2c80f94b0 2024-02-19T21:09:57.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2b40033f0 con 0x7fe2c80f94b0 2024-02-19T21:09:57.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe2b8002950 con 0x7fe2c80f94b0 2024-02-19T21:09:57.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 995740923 0 0) 0x7fe2c811bb50 con 0x7fe2c80f89c0 2024-02-19T21:09:57.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2c8102d80 con 0x7fe2c80f89c0 2024-02-19T21:09:57.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe2c4001990 con 0x7fe2c80f89c0 2024-02-19T21:09:57.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 778425529 0 0) 0x7fe2c8102f50 con 0x7fe2c8104290 2024-02-19T21:09:57.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 >> v1:172.21.15.163:6790/0 conn(0x7fe2c80f94b0 legacy=0x7fe2c8114ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:57.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 >> v1:172.21.15.193:6789/0 conn(0x7fe2c80f89c0 legacy=0x7fe2c8102570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:57.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c811cd50 con 0x7fe2c8104290 2024-02-19T21:09:57.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2cfb82700 1 -- 172.21.15.163:0/955938063 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2c811bd40 con 0x7fe2c8104290 2024-02-19T21:09:57.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2cfb82700 10 client.23085.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:57.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2cfb82700 1 -- 172.21.15.163:0/955938063 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2c811c1b0 con 0x7fe2c8104290 2024-02-19T21:09:57.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2cfb82700 1 librados: init done 2024-02-19T21:09:57.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fe2c0001550 con 0x7fe2c8104290 2024-02-19T21:09:57.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.238+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe2c0003a70 con 0x7fe2c8104290 2024-02-19T21:09:57.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.239+0000 7fe2cfb82700 1 -- 172.21.15.163:0/955938063 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2ac004f20 con 0x7fe2c8104290 2024-02-19T21:09:57.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.240+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fe2c0003cd0 con 0x7fe2c8104290 2024-02-19T21:09:57.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.240+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(411..411 src has 43..411) v4 ==== 7974+0+0 (unknown 2420631175 0 0) 0x7fe2c0079200 con 0x7fe2c8104290 2024-02-19T21:09:57.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.240+0000 7fe2beffd700 10 client.23085.objecter ms_dispatch 0x7fe2c8001550 osd_map(411..411 src has 43..411) v4 2024-02-19T21:09:57.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.240+0000 7fe2beffd700 3 client.23085.objecter handle_osd_map got epochs [411,411] > 0 2024-02-19T21:09:57.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.240+0000 7fe2beffd700 3 client.23085.objecter handle_osd_map decoding full epoch 411 2024-02-19T21:09:57.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.240+0000 7fe2beffd700 20 client.23085.objecter dump_active .. 0 homeless 2024-02-19T21:09:57.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.240+0000 7fe2beffd700 10 client.23085.objecter ms_handle_connect 0x7fe2b4063d00 2024-02-19T21:09:57.342 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.244+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe2c004c650 con 0x7fe2c8104290 2024-02-19T21:09:57.379 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:57.424 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:09:57.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.432+0000 7fe2cfb82700 1 -- 172.21.15.163:0/955938063 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe2ac000b90 con 0x7fe2b4063d00 2024-02-19T21:09:57.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.449+0000 7fe2beffd700 1 -- 172.21.15.163:0/955938063 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+576283 (unknown 2965378022 0 2826819555) 0x7fe2ac000b90 con 0x7fe2b4063d00 2024-02-19T21:09:57.454 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:09:57.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.453+0000 7fe2cfb82700 10 librados: watch_flush enter 2024-02-19T21:09:57.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.453+0000 7fe2cfb82700 10 librados: watch_flush exit 2024-02-19T21:09:57.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.453+0000 7fe2cfb82700 20 client.23085.objecter shutdown clearing up homeless session... 2024-02-19T21:09:57.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.453+0000 7fe2cfb82700 10 client.23085.objecter successfully canceled tick 2024-02-19T21:09:57.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.453+0000 7fe2cfb82700 1 -- 172.21.15.163:0/955938063 >> v1:172.21.15.163:6800/1552514681 conn(0x7fe2b4063d00 legacy=0x7fe2b40661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:57.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.453+0000 7fe2cfb82700 1 -- 172.21.15.163:0/955938063 >> v1:172.21.15.163:6789/0 conn(0x7fe2c8104290 legacy=0x7fe2c8118410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:57.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.453+0000 7fe2cfb82700 1 -- 172.21.15.163:0/955938063 shutdown_connections 2024-02-19T21:09:57.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.453+0000 7fe2cfb82700 1 -- 172.21.15.163:0/955938063 >> 172.21.15.163:0/955938063 conn(0x7fe2c80f2470 msgr2=0x7fe2c8103700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:57.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.453+0000 7fe2cfb82700 1 -- 172.21.15.163:0/955938063 shutdown_connections 2024-02-19T21:09:57.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.453+0000 7fe2cfb82700 1 -- 172.21.15.163:0/955938063 wait complete. 2024-02-19T21:09:57.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:57.453+0000 7fe2cfb82700 1 librados: shutdown 2024-02-19T21:09:57.461 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:09:57.639 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:57.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:57.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:57.744 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:57.772 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:57.970 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:09:57.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:57.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:09:58.069 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:09:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:09:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:09:58.077 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:58.105 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:58.391 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:57.208062+0000 mgr.y (mgr.14550) 367 : cluster [DBG] pgmap v591: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-02-19T21:09:58.392 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:09:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:57.432927+0000 mgr.y (mgr.14550) 368 : audit [DBG] from='client.23085 v1:172.21.15.163:0/955938063' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:09:58.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:57.208062+0000 mgr.y (mgr.14550) 367 : cluster [DBG] pgmap v591: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-02-19T21:09:58.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:09:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:57.432927+0000 mgr.y (mgr.14550) 368 : audit [DBG] from='client.23085 v1:172.21.15.163:0/955938063' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:09:58.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:57.208062+0000 mgr.y (mgr.14550) 367 : cluster [DBG] pgmap v591: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-02-19T21:09:58.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:09:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:57.432927+0000 mgr.y (mgr.14550) 368 : audit [DBG] from='client.23085 v1:172.21.15.163:0/955938063' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:09:58.855 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:58.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:58.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:58.959 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:09:58.967 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:09:58.967 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:09:58.987 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:59.008 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:09:59.010 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:59.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:09:59.111 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:59.139 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:59.238 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:59.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:59.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:59.252 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":591,"stamp":"2024-02-19T21:09:57.207843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007304,"num_objects":357,"num_object_clones":4,"num_object_copies":1071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":1327,"num_read_kb":978,"num_write":1040,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":7009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2386,"ondisk_log_size":2386,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":684,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2179444,"kb_used_data":57848,"kb_used_omap":0,"kb_used_meta":2121472,"kb_avail":654036620,"statfs":{"total":671965249536,"available":669733498880,"internally_reserved":0,"allocated":59236352,"data_stored":38701888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2172387328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":15,"num_read_kb":12,"num_write":17,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.945640"},"pg_stats":[{"pgid":"212.1f","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273797+0000","last_change":"2024-02-19T21:09:49.280368+0000","last_active":"2024-02-19T21:09:53.273797+0000","last_peered":"2024-02-19T21:09:53.273797+0000","last_clean":"2024-02-19T21:09:53.273797+0000","last_became_active":"2024-02-19T21:09:49.280113+0000","last_became_peered":"2024-02-19T21:09:49.280113+0000","last_unstale":"2024-02-19T21:09:53.273797+0000","last_undegraded":"2024-02-19T21:09:53.273797+0000","last_fullsized":"2024-02-19T21:09:53.273797+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.1e","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273601+0000","last_change":"2024-02-19T21:09:49.283981+0000","last_active":"2024-02-19T21:09:53.273601+0000","last_peered":"2024-02-19T21:09:53.273601+0000","last_clean":"2024-02-19T21:09:53.273601+0000","last_became_active":"2024-02-19T21:09:49.283490+0000","last_became_peered":"2024-02-19T21:09:49.283490+0000","last_unstale":"2024-02-19T21:09:53.273601+0000","last_undegraded":"2024-02-19T21:09:53.273601+0000","last_fullsized":"2024-02-19T21:09:53.273601+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.1d","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.281149+0000","last_change":"2024-02-19T21:09:49.279223+0000","last_active":"2024-02-19T21:09:53.281149+0000","last_peered":"2024-02-19T21:09:53.281149+0000","last_clean":"2024-02-19T21:09:53.281149+0000","last_became_active":"2024-02-19T21:09:49.278912+0000","last_became_peered":"2024-02-19T21:09:49.278912+0000","last_unstale":"2024-02-19T21:09:53.281149+0000","last_undegraded":"2024-02-19T21:09:53.281149+0000","last_fullsized":"2024-02-19T21:09:53.281149+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.1c","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278090+0000","last_change":"2024-02-19T21:09:49.273516+0000","last_active":"2024-02-19T21:09:53.278090+0000","last_peered":"2024-02-19T21:09:53.278090+0000","last_clean":"2024-02-19T21:09:53.278090+0000","last_became_active":"2024-02-19T21:09:49.273302+0000","last_became_peered":"2024-02-19T21:09:49.273302+0000","last_unstale":"2024-02-19T21:09:53.278090+0000","last_undegraded":"2024-02-19T21:09:53.278090+0000","last_fullsized":"2024-02-19T21:09:53.278090+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.1b","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.277682+0000","last_change":"2024-02-19T21:09:49.269667+0000","last_active":"2024-02-19T21:09:53.277682+0000","last_peered":"2024-02-19T21:09:53.277682+0000","last_clean":"2024-02-19T21:09:53.277682+0000","last_became_active":"2024-02-19T21:09:49.269397+0000","last_became_peered":"2024-02-19T21:09:49.269397+0000","last_unstale":"2024-02-19T21:09:53.277682+0000","last_undegraded":"2024-02-19T21:09:53.277682+0000","last_fullsized":"2024-02-19T21:09:53.277682+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.1a","version":"410'3","reported_seq":17,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.279949+0000","last_change":"2024-02-19T21:09:49.268123+0000","last_active":"2024-02-19T21:09:53.279949+0000","last_peered":"2024-02-19T21:09:53.279949+0000","last_clean":"2024-02-19T21:09:53.279949+0000","last_became_active":"2024-02-19T21:09:49.267619+0000","last_became_peered":"2024-02-19T21:09:49.267619+0000","last_unstale":"2024-02-19T21:09:53.279949+0000","last_undegraded":"2024-02-19T21:09:53.279949+0000","last_fullsized":"2024-02-19T21:09:53.279949+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.19","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269629+0000","last_change":"2024-02-19T21:09:49.276718+0000","last_active":"2024-02-19T21:09:53.269629+0000","last_peered":"2024-02-19T21:09:53.269629+0000","last_clean":"2024-02-19T21:09:53.269629+0000","last_became_active":"2024-02-19T21:09:49.276431+0000","last_became_peered":"2024-02-19T21:09:49.276431+0000","last_unstale":"2024-02-19T21:09:53.269629+0000","last_undegraded":"2024-02-19T21:09:53.269629+0000","last_fullsized":"2024-02-19T21:09:53.269629+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"212.18","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.280568+0000","last_change":"2024-02-19T21:09:49.269923+0000","last_active":"2024-02-19T21:09:53.280568+0000","last_peered":"2024-02-19T21:09:53.280568+0000","last_clean":"2024-02-19T21:09:53.280568+0000","last_became_active":"2024-02-19T21:09:49.269452+0000","last_became_peered":"2024-02-19T21:09:49.269452+0000","last_unstale":"2024-02-19T21:09:53.280568+0000","last_undegraded":"2024-02-19T21:09:53.280568+0000","last_fullsized":"2024-02-19T21:09:53.280568+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.17","version":"407'2","reported_seq":17,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.281975+0000","last_change":"2024-02-19T21:09:49.278385+0000","last_active":"2024-02-19T21:09:53.281975+0000","last_peered":"2024-02-19T21:09:53.281975+0000","last_clean":"2024-02-19T21:09:53.281975+0000","last_became_active":"2024-02-19T21:09:49.277932+0000","last_became_peered":"2024-02-19T21:09:49.277932+0000","last_unstale":"2024-02-19T21:09:53.281975+0000","last_undegraded":"2024-02-19T21:09:53.281975+0000","last_fullsized":"2024-02-19T21:09:53.281975+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.16","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270482+0000","last_change":"2024-02-19T21:09:49.276757+0000","last_active":"2024-02-19T21:09:53.270482+0000","last_peered":"2024-02-19T21:09:53.270482+0000","last_clean":"2024-02-19T21:09:53.270482+0000","last_became_active":"2024-02-19T21:09:49.276453+0000","last_became_peered":"2024-02-19T21:09:49.276453+0000","last_unstale":"2024-02-19T21:09:53.270482+0000","last_undegraded":"2024-02-19T21:09:53.270482+0000","last_fullsized":"2024-02-19T21:09:53.270482+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"212.15","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278259+0000","last_change":"2024-02-19T21:09:49.274158+0000","last_active":"2024-02-19T21:09:53.278259+0000","last_peered":"2024-02-19T21:09:53.278259+0000","last_clean":"2024-02-19T21:09:53.278259+0000","last_became_active":"2024-02-19T21:09:49.273893+0000","last_became_peered":"2024-02-19T21:09:49.273893+0000","last_unstale":"2024-02-19T21:09:53.278259+0000","last_undegraded":"2024-02-19T21:09:53.278259+0000","last_fullsized":"2024-02-19T21:09:53.278259+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.14","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271914+0000","last_change":"2024-02-19T21:09:49.287989+0000","last_active":"2024-02-19T21:09:53.271914+0000","last_peered":"2024-02-19T21:09:53.271914+0000","last_clean":"2024-02-19T21:09:53.271914+0000","last_became_active":"2024-02-19T21:09:49.287532+0000","last_became_peered":"2024-02-19T21:09:49.287532+0000","last_unstale":"2024-02-19T21:09:53.271914+0000","last_undegraded":"2024-02-19T21:09:53.271914+0000","last_fullsized":"2024-02-19T21:09:53.271914+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"212.13","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.268691+0000","last_change":"2024-02-19T21:09:49.278147+0000","last_active":"2024-02-19T21:09:53.268691+0000","last_peered":"2024-02-19T21:09:53.268691+0000","last_clean":"2024-02-19T21:09:53.268691+0000","last_became_active":"2024-02-19T21:09:49.277829+0000","last_became_peered":"2024-02-19T21:09:49.277829+0000","last_unstale":"2024-02-19T21:09:53.268691+0000","last_undegraded":"2024-02-19T21:09:53.268691+0000","last_fullsized":"2024-02-19T21:09:53.268691+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"212.12","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.277932+0000","last_change":"2024-02-19T21:09:49.273451+0000","last_active":"2024-02-19T21:09:53.277932+0000","last_peered":"2024-02-19T21:09:53.277932+0000","last_clean":"2024-02-19T21:09:53.277932+0000","last_became_active":"2024-02-19T21:09:49.273142+0000","last_became_peered":"2024-02-19T21:09:49.273142+0000","last_unstale":"2024-02-19T21:09:53.277932+0000","last_undegraded":"2024-02-19T21:09:53.277932+0000","last_fullsized":"2024-02-19T21:09:53.277932+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.11","version":"411'4","reported_seq":21,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.273310+0000","last_change":"2024-02-19T21:09:49.283943+0000","last_active":"2024-02-19T21:09:54.273310+0000","last_peered":"2024-02-19T21:09:54.273310+0000","last_clean":"2024-02-19T21:09:54.273310+0000","last_became_active":"2024-02-19T21:09:49.283453+0000","last_became_peered":"2024-02-19T21:09:49.283453+0000","last_unstale":"2024-02-19T21:09:54.273310+0000","last_undegraded":"2024-02-19T21:09:54.273310+0000","last_fullsized":"2024-02-19T21:09:54.273310+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.10","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.280563+0000","last_change":"2024-02-19T21:09:49.277056+0000","last_active":"2024-02-19T21:09:53.280563+0000","last_peered":"2024-02-19T21:09:53.280563+0000","last_clean":"2024-02-19T21:09:53.280563+0000","last_became_active":"2024-02-19T21:09:49.276708+0000","last_became_peered":"2024-02-19T21:09:49.276708+0000","last_unstale":"2024-02-19T21:09:53.280563+0000","last_undegraded":"2024-02-19T21:09:53.280563+0000","last_fullsized":"2024-02-19T21:09:53.280563+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.f","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.281389+0000","last_change":"2024-02-19T21:09:49.278210+0000","last_active":"2024-02-19T21:09:53.281389+0000","last_peered":"2024-02-19T21:09:53.281389+0000","last_clean":"2024-02-19T21:09:53.281389+0000","last_became_active":"2024-02-19T21:09:49.277549+0000","last_became_peered":"2024-02-19T21:09:49.277549+0000","last_unstale":"2024-02-19T21:09:53.281389+0000","last_undegraded":"2024-02-19T21:09:53.281389+0000","last_fullsized":"2024-02-19T21:09:53.281389+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.e","version":"407'1","reported_seq":15,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273671+0000","last_change":"2024-02-19T21:09:49.283811+0000","last_active":"2024-02-19T21:09:53.273671+0000","last_peered":"2024-02-19T21:09:53.273671+0000","last_clean":"2024-02-19T21:09:53.273671+0000","last_became_active":"2024-02-19T21:09:49.283455+0000","last_became_peered":"2024-02-19T21:09:49.283455+0000","last_unstale":"2024-02-19T21:09:53.273671+0000","last_undegraded":"2024-02-19T21:09:53.273671+0000","last_fullsized":"2024-02-19T21:09:53.273671+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.d","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270491+0000","last_change":"2024-02-19T21:09:49.279144+0000","last_active":"2024-02-19T21:09:53.270491+0000","last_peered":"2024-02-19T21:09:53.270491+0000","last_clean":"2024-02-19T21:09:53.270491+0000","last_became_active":"2024-02-19T21:09:49.278831+0000","last_became_peered":"2024-02-19T21:09:49.278831+0000","last_unstale":"2024-02-19T21:09:53.270491+0000","last_undegraded":"2024-02-19T21:09:53.270491+0000","last_fullsized":"2024-02-19T21:09:53.270491+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"212.c","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273110+0000","last_change":"2024-02-19T21:09:49.268676+0000","last_active":"2024-02-19T21:09:53.273110+0000","last_peered":"2024-02-19T21:09:53.273110+0000","last_clean":"2024-02-19T21:09:53.273110+0000","last_became_active":"2024-02-19T21:09:49.268398+0000","last_became_peered":"2024-02-19T21:09:49.268398+0000","last_unstale":"2024-02-19T21:09:53.273110+0000","last_undegraded":"2024-02-19T21:09:53.273110+0000","last_fullsized":"2024-02-19T21:09:53.273110+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.b","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272818+0000","last_change":"2024-02-19T21:09:49.278115+0000","last_active":"2024-02-19T21:09:53.272818+0000","last_peered":"2024-02-19T21:09:53.272818+0000","last_clean":"2024-02-19T21:09:53.272818+0000","last_became_active":"2024-02-19T21:09:49.277842+0000","last_became_peered":"2024-02-19T21:09:49.277842+0000","last_unstale":"2024-02-19T21:09:53.272818+0000","last_undegraded":"2024-02-19T21:09:53.272818+0000","last_fullsized":"2024-02-19T21:09:53.272818+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.a","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.280788+0000","last_change":"2024-02-19T21:09:49.268726+0000","last_active":"2024-02-19T21:09:53.280788+0000","last_peered":"2024-02-19T21:09:53.280788+0000","last_clean":"2024-02-19T21:09:53.280788+0000","last_became_active":"2024-02-19T21:09:49.268352+0000","last_became_peered":"2024-02-19T21:09:49.268352+0000","last_unstale":"2024-02-19T21:09:53.280788+0000","last_undegraded":"2024-02-19T21:09:53.280788+0000","last_fullsized":"2024-02-19T21:09:53.280788+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.9","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.277548+0000","last_change":"2024-02-19T21:09:49.273414+0000","last_active":"2024-02-19T21:09:53.277548+0000","last_peered":"2024-02-19T21:09:53.277548+0000","last_clean":"2024-02-19T21:09:53.277548+0000","last_became_active":"2024-02-19T21:09:49.273100+0000","last_became_peered":"2024-02-19T21:09:49.273100+0000","last_unstale":"2024-02-19T21:09:53.277548+0000","last_undegraded":"2024-02-19T21:09:53.277548+0000","last_fullsized":"2024-02-19T21:09:53.277548+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.8","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269700+0000","last_change":"2024-02-19T21:09:49.286754+0000","last_active":"2024-02-19T21:09:53.269700+0000","last_peered":"2024-02-19T21:09:53.269700+0000","last_clean":"2024-02-19T21:09:53.269700+0000","last_became_active":"2024-02-19T21:09:49.286432+0000","last_became_peered":"2024-02-19T21:09:49.286432+0000","last_unstale":"2024-02-19T21:09:53.269700+0000","last_undegraded":"2024-02-19T21:09:53.269700+0000","last_fullsized":"2024-02-19T21:09:53.269700+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"212.7","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273801+0000","last_change":"2024-02-19T21:09:49.272287+0000","last_active":"2024-02-19T21:09:53.273801+0000","last_peered":"2024-02-19T21:09:53.273801+0000","last_clean":"2024-02-19T21:09:53.273801+0000","last_became_active":"2024-02-19T21:09:49.271942+0000","last_became_peered":"2024-02-19T21:09:49.271942+0000","last_unstale":"2024-02-19T21:09:53.273801+0000","last_undegraded":"2024-02-19T21:09:53.273801+0000","last_fullsized":"2024-02-19T21:09:53.273801+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.6","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273692+0000","last_change":"2024-02-19T21:09:49.284037+0000","last_active":"2024-02-19T21:09:53.273692+0000","last_peered":"2024-02-19T21:09:53.273692+0000","last_clean":"2024-02-19T21:09:53.273692+0000","last_became_active":"2024-02-19T21:09:49.283823+0000","last_became_peered":"2024-02-19T21:09:49.283823+0000","last_unstale":"2024-02-19T21:09:53.273692+0000","last_undegraded":"2024-02-19T21:09:53.273692+0000","last_fullsized":"2024-02-19T21:09:53.273692+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.5","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272738+0000","last_change":"2024-02-19T21:09:49.283803+0000","last_active":"2024-02-19T21:09:53.272738+0000","last_peered":"2024-02-19T21:09:53.272738+0000","last_clean":"2024-02-19T21:09:53.272738+0000","last_became_active":"2024-02-19T21:09:49.283499+0000","last_became_peered":"2024-02-19T21:09:49.283499+0000","last_unstale":"2024-02-19T21:09:53.272738+0000","last_undegraded":"2024-02-19T21:09:53.272738+0000","last_fullsized":"2024-02-19T21:09:53.272738+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"212.4","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271954+0000","last_change":"2024-02-19T21:09:49.287819+0000","last_active":"2024-02-19T21:09:53.271954+0000","last_peered":"2024-02-19T21:09:53.271954+0000","last_clean":"2024-02-19T21:09:53.271954+0000","last_became_active":"2024-02-19T21:09:49.287128+0000","last_became_peered":"2024-02-19T21:09:49.287128+0000","last_unstale":"2024-02-19T21:09:53.271954+0000","last_undegraded":"2024-02-19T21:09:53.271954+0000","last_fullsized":"2024-02-19T21:09:53.271954+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"212.3","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270125+0000","last_change":"2024-02-19T21:09:49.276495+0000","last_active":"2024-02-19T21:09:53.270125+0000","last_peered":"2024-02-19T21:09:53.270125+0000","last_clean":"2024-02-19T21:09:53.270125+0000","last_became_active":"2024-02-19T21:09:49.276206+0000","last_became_peered":"2024-02-19T21:09:49.276206+0000","last_unstale":"2024-02-19T21:09:53.270125+0000","last_undegraded":"2024-02-19T21:09:53.270125+0000","last_fullsized":"2024-02-19T21:09:53.270125+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"212.2","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.268706+0000","last_change":"2024-02-19T21:09:49.275792+0000","last_active":"2024-02-19T21:09:53.268706+0000","last_peered":"2024-02-19T21:09:53.268706+0000","last_clean":"2024-02-19T21:09:53.268706+0000","last_became_active":"2024-02-19T21:09:49.275500+0000","last_became_peered":"2024-02-19T21:09:49.275500+0000","last_unstale":"2024-02-19T21:09:53.268706+0000","last_undegraded":"2024-02-19T21:09:53.268706+0000","last_fullsized":"2024-02-19T21:09:53.268706+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"212.1","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272821+0000","last_change":"2024-02-19T21:09:49.278031+0000","last_active":"2024-02-19T21:09:53.272821+0000","last_peered":"2024-02-19T21:09:53.272821+0000","last_clean":"2024-02-19T21:09:53.272821+0000","last_became_active":"2024-02-19T21:09:49.277749+0000","last_became_peered":"2024-02-19T21:09:49.277749+0000","last_unstale":"2024-02-19T21:09:53.272821+0000","last_undegraded":"2024-02-19T21:09:53.272821+0000","last_fullsized":"2024-02-19T21:09:53.272821+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.0","version":"0'0","reported_seq":14,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.277461+0000","last_change":"2024-02-19T21:09:49.268926+0000","last_active":"2024-02-19T21:09:53.277461+0000","last_peered":"2024-02-19T21:09:53.277461+0000","last_clean":"2024-02-19T21:09:53.277461+0000","last_became_active":"2024-02-19T21:09:49.268626+0000","last_became_peered":"2024-02-19T21:09:49.268626+0000","last_unstale":"2024-02-19T21:09:53.277461+0000","last_undegraded":"2024-02-19T21:09:53.277461+0000","last_fullsized":"2024-02-19T21:09:53.277461+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.2","version":"355'85","reported_seq":184,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270868+0000","last_change":"2024-02-19T21:09:54.271142+0000","last_active":"2024-02-19T21:09:54.270868+0000","last_peered":"2024-02-19T21:09:54.270868+0000","last_clean":"2024-02-19T21:09:54.270868+0000","last_became_active":"2024-02-19T21:08:04.247600+0000","last_became_peered":"2024-02-19T21:08:04.247600+0000","last_unstale":"2024-02-19T21:09:54.270868+0000","last_undegraded":"2024-02-19T21:09:54.270868+0000","last_fullsized":"2024-02-19T21:09:54.270868+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'85","last_scrub_stamp":"2024-02-19T21:08:55.882988+0000","last_deep_scrub":"355'85","last_deep_scrub_stamp":"2024-02-19T21:08:55.882988+0000","last_clean_scrub_stamp":"2024-02-19T21:08:55.882988+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.3","version":"338'71","reported_seq":415,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447402+0000","last_change":"2024-02-19T21:09:54.447662+0000","last_active":"2024-02-19T21:09:54.447402+0000","last_peered":"2024-02-19T21:09:54.447402+0000","last_clean":"2024-02-19T21:09:54.447402+0000","last_became_active":"2024-02-19T21:04:54.641884+0000","last_became_peered":"2024-02-19T21:04:54.641884+0000","last_unstale":"2024-02-19T21:09:54.447402+0000","last_undegraded":"2024-02-19T21:09:54.447402+0000","last_fullsized":"2024-02-19T21:09:54.447402+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'69","last_scrub_stamp":"2024-02-19T21:08:00.867237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:00.867237+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000068,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":23,"num_write_kb":3929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.0","version":"257'35","reported_seq":362,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446339+0000","last_change":"2024-02-19T21:09:54.448846+0000","last_active":"2024-02-19T21:09:54.446339+0000","last_peered":"2024-02-19T21:09:54.446339+0000","last_clean":"2024-02-19T21:09:54.446339+0000","last_became_active":"2024-02-19T21:06:40.245157+0000","last_became_peered":"2024-02-19T21:06:40.245157+0000","last_unstale":"2024-02-19T21:09:54.446339+0000","last_undegraded":"2024-02-19T21:09:54.446339+0000","last_fullsized":"2024-02-19T21:09:54.446339+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'35","last_scrub_stamp":"2024-02-19T21:08:55.692313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:55.692313+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.1","version":"338'69","reported_seq":467,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446598+0000","last_change":"2024-02-19T21:09:54.447045+0000","last_active":"2024-02-19T21:09:54.446598+0000","last_peered":"2024-02-19T21:09:54.446598+0000","last_clean":"2024-02-19T21:09:54.446598+0000","last_became_active":"2024-02-19T21:08:41.312701+0000","last_became_peered":"2024-02-19T21:08:41.312701+0000","last_unstale":"2024-02-19T21:09:54.446598+0000","last_undegraded":"2024-02-19T21:09:54.446598+0000","last_fullsized":"2024-02-19T21:09:54.446598+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'68","last_scrub_stamp":"2024-02-19T21:06:56.570995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:06:56.570995+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.6","version":"202'34","reported_seq":380,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.284405+0000","last_change":"2024-02-19T21:09:54.284645+0000","last_active":"2024-02-19T21:09:54.284405+0000","last_peered":"2024-02-19T21:09:54.284405+0000","last_clean":"2024-02-19T21:09:54.284405+0000","last_became_active":"2024-02-19T21:04:54.658542+0000","last_became_peered":"2024-02-19T21:04:54.658542+0000","last_unstale":"2024-02-19T21:09:54.284405+0000","last_undegraded":"2024-02-19T21:09:54.284405+0000","last_fullsized":"2024-02-19T21:09:54.284405+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'34","last_scrub_stamp":"2024-02-19T21:08:58.036955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:58.036955+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.7","version":"396'20","reported_seq":375,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269381+0000","last_change":"2024-02-19T21:09:54.270075+0000","last_active":"2024-02-19T21:09:54.269381+0000","last_peered":"2024-02-19T21:09:54.269381+0000","last_clean":"2024-02-19T21:09:54.269381+0000","last_became_active":"2024-02-19T21:04:54.598061+0000","last_became_peered":"2024-02-19T21:04:54.598061+0000","last_unstale":"2024-02-19T21:09:54.269381+0000","last_undegraded":"2024-02-19T21:09:54.269381+0000","last_fullsized":"2024-02-19T21:09:54.269381+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'18","last_scrub_stamp":"2024-02-19T21:08:03.459480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:03.459480+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.4","version":"403'70","reported_seq":464,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.283867+0000","last_change":"2024-02-19T21:09:54.284356+0000","last_active":"2024-02-19T21:09:54.283867+0000","last_peered":"2024-02-19T21:09:54.283867+0000","last_clean":"2024-02-19T21:09:54.283867+0000","last_became_active":"2024-02-19T21:08:41.286273+0000","last_became_peered":"2024-02-19T21:08:41.286273+0000","last_unstale":"2024-02-19T21:09:54.283867+0000","last_undegraded":"2024-02-19T21:09:54.283867+0000","last_fullsized":"2024-02-19T21:09:54.283867+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'68","last_scrub_stamp":"2024-02-19T21:07:35.203992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:07:35.203992+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.5","version":"338'121","reported_seq":453,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.448839+0000","last_change":"2024-02-19T21:09:54.449010+0000","last_active":"2024-02-19T21:09:54.448839+0000","last_peered":"2024-02-19T21:09:54.448839+0000","last_clean":"2024-02-19T21:09:54.448839+0000","last_became_active":"2024-02-19T21:04:54.646842+0000","last_became_peered":"2024-02-19T21:04:54.646842+0000","last_unstale":"2024-02-19T21:09:54.448839+0000","last_undegraded":"2024-02-19T21:09:54.448839+0000","last_fullsized":"2024-02-19T21:09:54.448839+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'121","last_scrub_stamp":"2024-02-19T21:08:49.752418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.752418+0000","log_size":121,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.a","version":"236'22","reported_seq":166,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.271048+0000","last_change":"2024-02-19T21:09:54.271322+0000","last_active":"2024-02-19T21:09:54.271048+0000","last_peered":"2024-02-19T21:09:54.271048+0000","last_clean":"2024-02-19T21:09:54.271048+0000","last_became_active":"2024-02-19T21:08:41.297020+0000","last_became_peered":"2024-02-19T21:08:41.297020+0000","last_unstale":"2024-02-19T21:09:54.271048+0000","last_undegraded":"2024-02-19T21:09:54.271048+0000","last_fullsized":"2024-02-19T21:09:54.271048+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-02-19T21:08:01.607121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:01.607121+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":64,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.b","version":"278'95","reported_seq":434,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447722+0000","last_change":"2024-02-19T21:09:54.447879+0000","last_active":"2024-02-19T21:09:54.447722+0000","last_peered":"2024-02-19T21:09:54.447722+0000","last_clean":"2024-02-19T21:09:54.447722+0000","last_became_active":"2024-02-19T21:04:54.640024+0000","last_became_peered":"2024-02-19T21:04:54.640024+0000","last_unstale":"2024-02-19T21:09:54.447722+0000","last_undegraded":"2024-02-19T21:09:54.447722+0000","last_fullsized":"2024-02-19T21:09:54.447722+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'87","last_scrub_stamp":"2024-02-19T21:07:28.724335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:07:28.724335+0000","log_size":95,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.8","version":"399'39","reported_seq":448,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268356+0000","last_change":"2024-02-19T21:09:54.268765+0000","last_active":"2024-02-19T21:09:54.268356+0000","last_peered":"2024-02-19T21:09:54.268356+0000","last_clean":"2024-02-19T21:09:54.268356+0000","last_became_active":"2024-02-19T21:08:04.266021+0000","last_became_peered":"2024-02-19T21:08:04.266021+0000","last_unstale":"2024-02-19T21:09:54.268356+0000","last_undegraded":"2024-02-19T21:09:54.268356+0000","last_fullsized":"2024-02-19T21:09:54.268356+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'35","last_scrub_stamp":"2024-02-19T21:08:01.494484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:01.494484+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":50,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.9","version":"403'71","reported_seq":418,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447135+0000","last_change":"2024-02-19T21:09:54.447341+0000","last_active":"2024-02-19T21:09:54.447135+0000","last_peered":"2024-02-19T21:09:54.447135+0000","last_clean":"2024-02-19T21:09:54.447135+0000","last_became_active":"2024-02-19T21:04:54.671646+0000","last_became_peered":"2024-02-19T21:04:54.671646+0000","last_unstale":"2024-02-19T21:09:54.447135+0000","last_undegraded":"2024-02-19T21:09:54.447135+0000","last_fullsized":"2024-02-19T21:09:54.447135+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'68","last_scrub_stamp":"2024-02-19T21:07:31.724135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:07:31.724135+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.e","version":"393'40","reported_seq":426,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447862+0000","last_change":"2024-02-19T21:09:54.448037+0000","last_active":"2024-02-19T21:09:54.447862+0000","last_peered":"2024-02-19T21:09:54.447862+0000","last_clean":"2024-02-19T21:09:54.447862+0000","last_became_active":"2024-02-19T21:08:41.280291+0000","last_became_peered":"2024-02-19T21:08:41.280291+0000","last_unstale":"2024-02-19T21:09:54.447862+0000","last_undegraded":"2024-02-19T21:09:54.447862+0000","last_fullsized":"2024-02-19T21:09:54.447862+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'34","last_scrub_stamp":"2024-02-19T21:06:59.674445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:06:59.674445+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.f","version":"399'37","reported_seq":419,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.275142+0000","last_change":"2024-02-19T21:09:54.276366+0000","last_active":"2024-02-19T21:09:54.275142+0000","last_peered":"2024-02-19T21:09:54.275142+0000","last_clean":"2024-02-19T21:09:54.275142+0000","last_became_active":"2024-02-19T21:08:41.303778+0000","last_became_peered":"2024-02-19T21:08:41.303778+0000","last_unstale":"2024-02-19T21:09:54.275142+0000","last_undegraded":"2024-02-19T21:09:54.275142+0000","last_fullsized":"2024-02-19T21:09:54.275142+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'34","last_scrub_stamp":"2024-02-19T21:06:58.720103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:06:58.720103+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.c","version":"249'73","reported_seq":577,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270402+0000","last_change":"2024-02-19T21:09:54.270914+0000","last_active":"2024-02-19T21:09:54.270402+0000","last_peered":"2024-02-19T21:09:54.270402+0000","last_clean":"2024-02-19T21:09:54.270402+0000","last_became_active":"2024-02-19T21:08:41.280328+0000","last_became_peered":"2024-02-19T21:08:41.280328+0000","last_unstale":"2024-02-19T21:09:54.270402+0000","last_undegraded":"2024-02-19T21:09:54.270402+0000","last_fullsized":"2024-02-19T21:09:54.270402+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'73","last_scrub_stamp":"2024-02-19T21:08:49.613462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.613462+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.d","version":"393'39","reported_seq":389,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269718+0000","last_change":"2024-02-19T21:09:54.270298+0000","last_active":"2024-02-19T21:09:54.269718+0000","last_peered":"2024-02-19T21:09:54.269718+0000","last_clean":"2024-02-19T21:09:54.269718+0000","last_became_active":"2024-02-19T21:04:54.644749+0000","last_became_peered":"2024-02-19T21:04:54.644749+0000","last_unstale":"2024-02-19T21:09:54.269718+0000","last_undegraded":"2024-02-19T21:09:54.269718+0000","last_fullsized":"2024-02-19T21:09:54.269718+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'34","last_scrub_stamp":"2024-02-19T21:08:12.445646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.445646+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.12","version":"390'47","reported_seq":473,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.279975+0000","last_change":"2024-02-19T21:09:54.280518+0000","last_active":"2024-02-19T21:09:54.279975+0000","last_peered":"2024-02-19T21:09:54.279975+0000","last_clean":"2024-02-19T21:09:54.279975+0000","last_became_active":"2024-02-19T21:08:04.263085+0000","last_became_peered":"2024-02-19T21:08:04.263085+0000","last_unstale":"2024-02-19T21:09:54.279975+0000","last_undegraded":"2024-02-19T21:09:54.279975+0000","last_fullsized":"2024-02-19T21:09:54.279975+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'45","last_scrub_stamp":"2024-02-19T21:08:49.965991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.965991+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.13","version":"370'71","reported_seq":474,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.274258+0000","last_change":"2024-02-19T21:09:54.274552+0000","last_active":"2024-02-19T21:09:54.274258+0000","last_peered":"2024-02-19T21:09:54.274258+0000","last_clean":"2024-02-19T21:09:54.274258+0000","last_became_active":"2024-02-19T21:08:41.264164+0000","last_became_peered":"2024-02-19T21:08:41.264164+0000","last_unstale":"2024-02-19T21:09:54.274258+0000","last_undegraded":"2024-02-19T21:09:54.274258+0000","last_fullsized":"2024-02-19T21:09:54.274258+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'68","last_scrub_stamp":"2024-02-19T21:08:12.764542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.764542+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":60,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.10","version":"230'45","reported_seq":396,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.273729+0000","last_change":"2024-02-19T21:09:54.274139+0000","last_active":"2024-02-19T21:09:54.273729+0000","last_peered":"2024-02-19T21:09:54.273729+0000","last_clean":"2024-02-19T21:09:54.273729+0000","last_became_active":"2024-02-19T21:04:54.626393+0000","last_became_peered":"2024-02-19T21:04:54.626393+0000","last_unstale":"2024-02-19T21:09:54.273729+0000","last_undegraded":"2024-02-19T21:09:54.273729+0000","last_fullsized":"2024-02-19T21:09:54.273729+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'45","last_scrub_stamp":"2024-02-19T21:08:39.942302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.942302+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.11","version":"338'35","reported_seq":386,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.278844+0000","last_change":"2024-02-19T21:09:54.279500+0000","last_active":"2024-02-19T21:09:54.278844+0000","last_peered":"2024-02-19T21:09:54.278844+0000","last_clean":"2024-02-19T21:09:54.278844+0000","last_became_active":"2024-02-19T21:04:54.659096+0000","last_became_peered":"2024-02-19T21:04:54.659096+0000","last_unstale":"2024-02-19T21:09:54.278844+0000","last_undegraded":"2024-02-19T21:09:54.278844+0000","last_fullsized":"2024-02-19T21:09:54.278844+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'35","last_scrub_stamp":"2024-02-19T21:08:41.145092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:41.145092+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.16","version":"411'43","reported_seq":299,"reported_epoch":411,"state":"active+clean+snaptrim","last_fresh":"2024-02-19T21:09:54.270812+0000","last_change":"2024-02-19T21:09:54.270812+0000","last_active":"2024-02-19T21:09:54.270812+0000","last_peered":"2024-02-19T21:09:54.270812+0000","last_clean":"2024-02-19T21:09:54.270812+0000","last_became_active":"2024-02-19T21:08:41.269638+0000","last_became_peered":"2024-02-19T21:08:41.269638+0000","last_unstale":"2024-02-19T21:09:54.270812+0000","last_undegraded":"2024-02-19T21:09:54.270812+0000","last_fullsized":"2024-02-19T21:09:54.270812+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-02-19T21:06:57.889852+0000","last_deep_scrub":"237'24","last_deep_scrub_stamp":"2024-02-19T21:06:57.889852+0000","last_clean_scrub_stamp":"2024-02-19T21:06:57.889852+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":34,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.17","version":"286'24","reported_seq":410,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.284443+0000","last_change":"2024-02-19T21:09:54.284847+0000","last_active":"2024-02-19T21:09:54.284443+0000","last_peered":"2024-02-19T21:09:54.284443+0000","last_clean":"2024-02-19T21:09:54.284443+0000","last_became_active":"2024-02-19T21:08:41.285673+0000","last_became_peered":"2024-02-19T21:08:41.285673+0000","last_unstale":"2024-02-19T21:09:54.284443+0000","last_undegraded":"2024-02-19T21:09:54.284443+0000","last_fullsized":"2024-02-19T21:09:54.284443+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'24","last_scrub_stamp":"2024-02-19T21:08:48.023079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:48.023079+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":27,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.14","version":"338'52","reported_seq":380,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.281890+0000","last_change":"2024-02-19T21:09:54.284416+0000","last_active":"2024-02-19T21:09:54.281890+0000","last_peered":"2024-02-19T21:09:54.281890+0000","last_clean":"2024-02-19T21:09:54.281890+0000","last_became_active":"2024-02-19T21:06:40.227840+0000","last_became_peered":"2024-02-19T21:06:40.227840+0000","last_unstale":"2024-02-19T21:09:54.281890+0000","last_undegraded":"2024-02-19T21:09:54.281890+0000","last_fullsized":"2024-02-19T21:09:54.281890+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'52","last_scrub_stamp":"2024-02-19T21:09:28.997593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.997593+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.15","version":"370'52","reported_seq":524,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.276335+0000","last_change":"2024-02-19T21:09:54.276777+0000","last_active":"2024-02-19T21:09:54.276335+0000","last_peered":"2024-02-19T21:09:54.276335+0000","last_clean":"2024-02-19T21:09:54.276335+0000","last_became_active":"2024-02-19T21:08:41.277044+0000","last_became_peered":"2024-02-19T21:08:41.277044+0000","last_unstale":"2024-02-19T21:09:54.276335+0000","last_undegraded":"2024-02-19T21:09:54.276335+0000","last_fullsized":"2024-02-19T21:09:54.276335+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'51","last_scrub_stamp":"2024-02-19T21:07:28.859888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:07:28.859888+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":99,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.1a","version":"338'91","reported_seq":565,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.273785+0000","last_change":"2024-02-19T21:09:54.274279+0000","last_active":"2024-02-19T21:09:54.273785+0000","last_peered":"2024-02-19T21:09:54.273785+0000","last_clean":"2024-02-19T21:09:54.273785+0000","last_became_active":"2024-02-19T21:08:04.265588+0000","last_became_peered":"2024-02-19T21:08:04.265588+0000","last_unstale":"2024-02-19T21:09:54.273785+0000","last_undegraded":"2024-02-19T21:09:54.273785+0000","last_fullsized":"2024-02-19T21:09:54.273785+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'90","last_scrub_stamp":"2024-02-19T21:08:02.861467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:02.861467+0000","log_size":91,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.1b","version":"338'52","reported_seq":154,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269600+0000","last_change":"2024-02-19T21:09:54.270086+0000","last_active":"2024-02-19T21:09:54.269600+0000","last_peered":"2024-02-19T21:09:54.269600+0000","last_clean":"2024-02-19T21:09:54.269600+0000","last_became_active":"2024-02-19T21:08:04.262932+0000","last_became_peered":"2024-02-19T21:08:04.262932+0000","last_unstale":"2024-02-19T21:09:54.269600+0000","last_undegraded":"2024-02-19T21:09:54.269600+0000","last_fullsized":"2024-02-19T21:09:54.269600+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'51","last_scrub_stamp":"2024-02-19T21:08:00.485092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:00.485092+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.18","version":"214'102","reported_seq":418,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268546+0000","last_change":"2024-02-19T21:09:54.268817+0000","last_active":"2024-02-19T21:09:54.268546+0000","last_peered":"2024-02-19T21:09:54.268546+0000","last_clean":"2024-02-19T21:09:54.268546+0000","last_became_active":"2024-02-19T21:08:44.236717+0000","last_became_peered":"2024-02-19T21:08:44.236717+0000","last_unstale":"2024-02-19T21:09:54.268546+0000","last_undegraded":"2024-02-19T21:09:54.268546+0000","last_fullsized":"2024-02-19T21:09:54.268546+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-19T21:08:50.838363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:50.838363+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.19","version":"338'69","reported_seq":469,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269329+0000","last_change":"2024-02-19T21:09:54.269653+0000","last_active":"2024-02-19T21:09:54.269329+0000","last_peered":"2024-02-19T21:09:54.269329+0000","last_clean":"2024-02-19T21:09:54.269329+0000","last_became_active":"2024-02-19T21:08:41.292658+0000","last_became_peered":"2024-02-19T21:08:41.292658+0000","last_unstale":"2024-02-19T21:09:54.269329+0000","last_undegraded":"2024-02-19T21:09:54.269329+0000","last_fullsized":"2024-02-19T21:09:54.269329+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-19T21:08:39.466543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.466543+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.1e","version":"338'50","reported_seq":404,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.274371+0000","last_change":"2024-02-19T21:09:54.276257+0000","last_active":"2024-02-19T21:09:54.274371+0000","last_peered":"2024-02-19T21:09:54.274371+0000","last_clean":"2024-02-19T21:09:54.274371+0000","last_became_active":"2024-02-19T21:04:54.629433+0000","last_became_peered":"2024-02-19T21:04:54.629433+0000","last_unstale":"2024-02-19T21:09:54.274371+0000","last_undegraded":"2024-02-19T21:09:54.274371+0000","last_fullsized":"2024-02-19T21:09:54.274371+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'49","last_scrub_stamp":"2024-02-19T21:07:01.787521+0000","last_deep_scrub":"241'49","last_deep_scrub_stamp":"2024-02-19T21:07:01.787521+0000","last_clean_scrub_stamp":"2024-02-19T21:07:01.787521+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":48,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.1f","version":"249'44","reported_seq":395,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.284719+0000","last_change":"2024-02-19T21:09:54.285026+0000","last_active":"2024-02-19T21:09:54.284719+0000","last_peered":"2024-02-19T21:09:54.284719+0000","last_clean":"2024-02-19T21:09:54.284719+0000","last_became_active":"2024-02-19T21:04:54.671969+0000","last_became_peered":"2024-02-19T21:04:54.671969+0000","last_unstale":"2024-02-19T21:09:54.284719+0000","last_undegraded":"2024-02-19T21:09:54.284719+0000","last_fullsized":"2024-02-19T21:09:54.284719+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'44","last_scrub_stamp":"2024-02-19T21:09:25.988978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:09:25.988978+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.1c","version":"338'137","reported_seq":573,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270137+0000","last_change":"2024-02-19T21:09:54.270752+0000","last_active":"2024-02-19T21:09:54.270137+0000","last_peered":"2024-02-19T21:09:54.270137+0000","last_clean":"2024-02-19T21:09:54.270137+0000","last_became_active":"2024-02-19T21:08:48.248009+0000","last_became_peered":"2024-02-19T21:08:48.248009+0000","last_unstale":"2024-02-19T21:09:54.270137+0000","last_undegraded":"2024-02-19T21:09:54.270137+0000","last_fullsized":"2024-02-19T21:09:54.270137+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'136","last_scrub_stamp":"2024-02-19T21:08:37.523303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:37.523303+0000","log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":96,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":41,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":96,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.1d","version":"370'36","reported_seq":388,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447930+0000","last_change":"2024-02-19T21:09:54.448779+0000","last_active":"2024-02-19T21:09:54.447930+0000","last_peered":"2024-02-19T21:09:54.447930+0000","last_clean":"2024-02-19T21:09:54.447930+0000","last_became_active":"2024-02-19T21:04:54.655665+0000","last_became_peered":"2024-02-19T21:04:54.655665+0000","last_unstale":"2024-02-19T21:09:54.447930+0000","last_undegraded":"2024-02-19T21:09:54.447930+0000","last_fullsized":"2024-02-19T21:09:54.447930+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'35","last_scrub_stamp":"2024-02-19T21:08:56.661296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:56.661296+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"5.9","version":"344'10","reported_seq":501,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.275710+0000","last_change":"2024-02-19T21:09:42.967840+0000","last_active":"2024-02-19T21:09:54.275710+0000","last_peered":"2024-02-19T21:09:54.275710+0000","last_clean":"2024-02-19T21:09:54.275710+0000","last_became_active":"2024-02-19T21:08:41.307616+0000","last_became_peered":"2024-02-19T21:08:41.307616+0000","last_unstale":"2024-02-19T21:09:54.275710+0000","last_undegraded":"2024-02-19T21:09:54.275710+0000","last_fullsized":"2024-02-19T21:09:54.275710+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'10","last_scrub_stamp":"2024-02-19T21:09:42.967771+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:09:42.967771+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":401,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269695+0000","last_change":"2024-02-19T21:08:51.616197+0000","last_active":"2024-02-19T21:09:53.269695+0000","last_peered":"2024-02-19T21:09:53.269695+0000","last_clean":"2024-02-19T21:09:53.269695+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:09:53.269695+0000","last_undegraded":"2024-02-19T21:09:53.269695+0000","last_fullsized":"2024-02-19T21:09:53.269695+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:08:51.616123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:51.616123+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":397,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.281672+0000","last_change":"2024-02-19T21:09:44.922372+0000","last_active":"2024-02-19T21:09:53.281672+0000","last_peered":"2024-02-19T21:09:53.281672+0000","last_clean":"2024-02-19T21:09:53.281672+0000","last_became_active":"2024-02-19T21:08:04.239474+0000","last_became_peered":"2024-02-19T21:08:04.239474+0000","last_unstale":"2024-02-19T21:09:53.281672+0000","last_undegraded":"2024-02-19T21:09:53.281672+0000","last_fullsized":"2024-02-19T21:09:53.281672+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:44.922303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_clean_scrub_stamp":"2024-02-19T21:09:44.922303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":369,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270000+0000","last_change":"2024-02-19T21:09:27.878844+0000","last_active":"2024-02-19T21:09:53.270000+0000","last_peered":"2024-02-19T21:09:53.270000+0000","last_clean":"2024-02-19T21:09:53.270000+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:09:53.270000+0000","last_undegraded":"2024-02-19T21:09:53.270000+0000","last_fullsized":"2024-02-19T21:09:53.270000+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.878760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.878760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":322,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.282532+0000","last_change":"2024-02-19T21:06:46.312127+0000","last_active":"2024-02-19T21:09:53.282532+0000","last_peered":"2024-02-19T21:09:53.282532+0000","last_clean":"2024-02-19T21:09:53.282532+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:09:53.282532+0000","last_undegraded":"2024-02-19T21:09:53.282532+0000","last_fullsized":"2024-02-19T21:09:53.282532+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:46.312025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.312025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":252,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269705+0000","last_change":"2024-02-19T21:08:53.783361+0000","last_active":"2024-02-19T21:09:53.269705+0000","last_peered":"2024-02-19T21:09:53.269705+0000","last_clean":"2024-02-19T21:09:53.269705+0000","last_became_active":"2024-02-19T21:08:41.314093+0000","last_became_peered":"2024-02-19T21:08:41.314093+0000","last_unstale":"2024-02-19T21:09:53.269705+0000","last_undegraded":"2024-02-19T21:09:53.269705+0000","last_fullsized":"2024-02-19T21:09:53.269705+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:08:53.783292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:53.783292+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":406,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273238+0000","last_change":"2024-02-19T21:08:41.309125+0000","last_active":"2024-02-19T21:09:53.273238+0000","last_peered":"2024-02-19T21:09:53.273238+0000","last_clean":"2024-02-19T21:09:53.273238+0000","last_became_active":"2024-02-19T21:08:41.302799+0000","last_became_peered":"2024-02-19T21:08:41.302799+0000","last_unstale":"2024-02-19T21:09:53.273238+0000","last_undegraded":"2024-02-19T21:09:53.273238+0000","last_fullsized":"2024-02-19T21:09:53.273238+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:39.700315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.700315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":397,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.280246+0000","last_change":"2024-02-19T21:08:41.277873+0000","last_active":"2024-02-19T21:09:53.280246+0000","last_peered":"2024-02-19T21:09:53.280246+0000","last_clean":"2024-02-19T21:09:53.280246+0000","last_became_active":"2024-02-19T21:08:41.276911+0000","last_became_peered":"2024-02-19T21:08:41.276911+0000","last_unstale":"2024-02-19T21:09:53.280246+0000","last_undegraded":"2024-02-19T21:09:53.280246+0000","last_fullsized":"2024-02-19T21:09:53.280246+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:27.303307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:27.303307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":206,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273002+0000","last_change":"2024-02-19T21:09:34.791174+0000","last_active":"2024-02-19T21:09:53.273002+0000","last_peered":"2024-02-19T21:09:53.273002+0000","last_clean":"2024-02-19T21:09:53.273002+0000","last_became_active":"2024-02-19T21:06:40.227822+0000","last_became_peered":"2024-02-19T21:06:40.227822+0000","last_unstale":"2024-02-19T21:09:53.273002+0000","last_undegraded":"2024-02-19T21:09:53.273002+0000","last_fullsized":"2024-02-19T21:09:53.273002+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:34.791063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:34.791063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":323,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271120+0000","last_change":"2024-02-19T21:07:00.449622+0000","last_active":"2024-02-19T21:09:53.271120+0000","last_peered":"2024-02-19T21:09:53.271120+0000","last_clean":"2024-02-19T21:09:53.271120+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:09:53.271120+0000","last_undegraded":"2024-02-19T21:09:53.271120+0000","last_fullsized":"2024-02-19T21:09:53.271120+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:00.449534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:00.449534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":406,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278850+0000","last_change":"2024-02-19T21:09:34.573994+0000","last_active":"2024-02-19T21:09:53.278850+0000","last_peered":"2024-02-19T21:09:53.278850+0000","last_clean":"2024-02-19T21:09:53.278850+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:09:53.278850+0000","last_undegraded":"2024-02-19T21:09:53.278850+0000","last_fullsized":"2024-02-19T21:09:53.278850+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:09:34.573889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:34.573889+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":379,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.282395+0000","last_change":"2024-02-19T21:09:30.950696+0000","last_active":"2024-02-19T21:09:53.282395+0000","last_peered":"2024-02-19T21:09:53.282395+0000","last_clean":"2024-02-19T21:09:53.282395+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:09:53.282395+0000","last_undegraded":"2024-02-19T21:09:53.282395+0000","last_fullsized":"2024-02-19T21:09:53.282395+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:30.950626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:09:30.950626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":370,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.281262+0000","last_change":"2024-02-19T21:09:28.031934+0000","last_active":"2024-02-19T21:09:53.281262+0000","last_peered":"2024-02-19T21:09:53.281262+0000","last_clean":"2024-02-19T21:09:53.281262+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:09:53.281262+0000","last_undegraded":"2024-02-19T21:09:53.281262+0000","last_fullsized":"2024-02-19T21:09:53.281262+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:28.031872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.031872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":386,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272547+0000","last_change":"2024-02-19T21:08:53.964809+0000","last_active":"2024-02-19T21:09:53.272547+0000","last_peered":"2024-02-19T21:09:53.272547+0000","last_clean":"2024-02-19T21:09:53.272547+0000","last_became_active":"2024-02-19T21:08:04.265081+0000","last_became_peered":"2024-02-19T21:08:04.265081+0000","last_unstale":"2024-02-19T21:09:53.272547+0000","last_undegraded":"2024-02-19T21:09:53.272547+0000","last_fullsized":"2024-02-19T21:09:53.272547+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_clean_scrub_stamp":"2024-02-19T21:08:53.964703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":337,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273866+0000","last_change":"2024-02-19T21:09:43.984159+0000","last_active":"2024-02-19T21:09:53.273866+0000","last_peered":"2024-02-19T21:09:53.273866+0000","last_clean":"2024-02-19T21:09:53.273866+0000","last_became_active":"2024-02-19T21:08:41.304395+0000","last_became_peered":"2024-02-19T21:08:41.304395+0000","last_unstale":"2024-02-19T21:09:53.273866+0000","last_undegraded":"2024-02-19T21:09:53.273866+0000","last_fullsized":"2024-02-19T21:09:53.273866+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:43.984079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:09:43.984079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":143,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.281567+0000","last_change":"2024-02-19T21:09:41.955664+0000","last_active":"2024-02-19T21:09:53.281567+0000","last_peered":"2024-02-19T21:09:53.281567+0000","last_clean":"2024-02-19T21:09:53.281567+0000","last_became_active":"2024-02-19T21:08:41.286244+0000","last_became_peered":"2024-02-19T21:08:41.286244+0000","last_unstale":"2024-02-19T21:09:53.281567+0000","last_undegraded":"2024-02-19T21:09:53.281567+0000","last_fullsized":"2024-02-19T21:09:53.281567+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:09:41.955563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:41.955563+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":393,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273156+0000","last_change":"2024-02-19T21:08:41.367028+0000","last_active":"2024-02-19T21:09:53.273156+0000","last_peered":"2024-02-19T21:09:53.273156+0000","last_clean":"2024-02-19T21:09:53.273156+0000","last_became_active":"2024-02-19T21:08:41.273961+0000","last_became_peered":"2024-02-19T21:08:41.273961+0000","last_unstale":"2024-02-19T21:09:53.273156+0000","last_undegraded":"2024-02-19T21:09:53.273156+0000","last_fullsized":"2024-02-19T21:09:53.273156+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:06:50.694248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:50.694248+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":384,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272224+0000","last_change":"2024-02-19T21:09:36.685164+0000","last_active":"2024-02-19T21:09:53.272224+0000","last_peered":"2024-02-19T21:09:53.272224+0000","last_clean":"2024-02-19T21:09:53.272224+0000","last_became_active":"2024-02-19T21:08:41.302537+0000","last_became_peered":"2024-02-19T21:08:41.302537+0000","last_unstale":"2024-02-19T21:09:53.272224+0000","last_undegraded":"2024-02-19T21:09:53.272224+0000","last_fullsized":"2024-02-19T21:09:53.272224+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:36.685063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:36.685063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":395,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.280340+0000","last_change":"2024-02-19T21:08:41.278598+0000","last_active":"2024-02-19T21:09:53.280340+0000","last_peered":"2024-02-19T21:09:53.280340+0000","last_clean":"2024-02-19T21:09:53.280340+0000","last_became_active":"2024-02-19T21:08:41.278298+0000","last_became_peered":"2024-02-19T21:08:41.278298+0000","last_unstale":"2024-02-19T21:09:53.280340+0000","last_undegraded":"2024-02-19T21:09:53.280340+0000","last_fullsized":"2024-02-19T21:09:53.280340+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:13.136532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:06:13.136532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":308,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270239+0000","last_change":"2024-02-19T21:08:41.296517+0000","last_active":"2024-02-19T21:09:53.270239+0000","last_peered":"2024-02-19T21:09:53.270239+0000","last_clean":"2024-02-19T21:09:53.270239+0000","last_became_active":"2024-02-19T21:08:41.296219+0000","last_became_peered":"2024-02-19T21:08:41.296219+0000","last_unstale":"2024-02-19T21:09:53.270239+0000","last_undegraded":"2024-02-19T21:09:53.270239+0000","last_fullsized":"2024-02-19T21:09:53.270239+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.878060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.878060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"82'15","reported_seq":402,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272244+0000","last_change":"2024-02-19T21:08:42.367657+0000","last_active":"2024-02-19T21:09:53.272244+0000","last_peered":"2024-02-19T21:09:53.272244+0000","last_clean":"2024-02-19T21:09:53.272244+0000","last_became_active":"2024-02-19T21:08:41.295931+0000","last_became_peered":"2024-02-19T21:08:41.295931+0000","last_unstale":"2024-02-19T21:09:53.272244+0000","last_undegraded":"2024-02-19T21:09:53.272244+0000","last_fullsized":"2024-02-19T21:09:53.272244+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:06:28.746972+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:06:28.746972+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":392,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269161+0000","last_change":"2024-02-19T21:08:04.269175+0000","last_active":"2024-02-19T21:09:53.269161+0000","last_peered":"2024-02-19T21:09:53.269161+0000","last_clean":"2024-02-19T21:09:53.269161+0000","last_became_active":"2024-02-19T21:08:04.268406+0000","last_became_peered":"2024-02-19T21:08:04.268406+0000","last_unstale":"2024-02-19T21:09:53.269161+0000","last_undegraded":"2024-02-19T21:09:53.269161+0000","last_fullsized":"2024-02-19T21:09:53.269161+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:14.687938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.687938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":482,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.281143+0000","last_change":"2024-02-19T21:06:12.177597+0000","last_active":"2024-02-19T21:09:54.281143+0000","last_peered":"2024-02-19T21:09:54.281143+0000","last_clean":"2024-02-19T21:09:54.281143+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:09:54.281143+0000","last_undegraded":"2024-02-19T21:09:54.281143+0000","last_fullsized":"2024-02-19T21:09:54.281143+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:12.177524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:12.177524+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":81,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273264+0000","last_change":"2024-02-19T21:08:41.300913+0000","last_active":"2024-02-19T21:09:53.273264+0000","last_peered":"2024-02-19T21:09:53.273264+0000","last_clean":"2024-02-19T21:09:53.273264+0000","last_became_active":"2024-02-19T21:08:41.300707+0000","last_became_peered":"2024-02-19T21:08:41.300707+0000","last_unstale":"2024-02-19T21:09:53.273264+0000","last_undegraded":"2024-02-19T21:09:53.273264+0000","last_fullsized":"2024-02-19T21:09:53.273264+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.704447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.704447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":122,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270019+0000","last_change":"2024-02-19T21:08:36.847613+0000","last_active":"2024-02-19T21:09:53.270019+0000","last_peered":"2024-02-19T21:09:53.270019+0000","last_clean":"2024-02-19T21:09:53.270019+0000","last_became_active":"2024-02-19T21:08:04.242244+0000","last_became_peered":"2024-02-19T21:08:04.242244+0000","last_unstale":"2024-02-19T21:09:53.270019+0000","last_undegraded":"2024-02-19T21:09:53.270019+0000","last_fullsized":"2024-02-19T21:09:53.270019+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:36.847548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.847548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":472,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273099+0000","last_change":"2024-02-19T21:08:43.178829+0000","last_active":"2024-02-19T21:09:53.273099+0000","last_peered":"2024-02-19T21:09:53.273099+0000","last_clean":"2024-02-19T21:09:53.273099+0000","last_became_active":"2024-02-19T21:08:41.311383+0000","last_became_peered":"2024-02-19T21:08:41.311383+0000","last_unstale":"2024-02-19T21:09:53.273099+0000","last_undegraded":"2024-02-19T21:09:53.273099+0000","last_fullsized":"2024-02-19T21:09:53.273099+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:07:59.834282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:07:59.834282+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":419,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269196+0000","last_change":"2024-02-19T21:08:04.271314+0000","last_active":"2024-02-19T21:09:53.269196+0000","last_peered":"2024-02-19T21:09:53.269196+0000","last_clean":"2024-02-19T21:09:53.269196+0000","last_became_active":"2024-02-19T21:08:04.270601+0000","last_became_peered":"2024-02-19T21:08:04.270601+0000","last_unstale":"2024-02-19T21:09:53.269196+0000","last_undegraded":"2024-02-19T21:09:53.269196+0000","last_fullsized":"2024-02-19T21:09:53.269196+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:07.874362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:07.874362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"344'9","reported_seq":119,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270786+0000","last_change":"2024-02-19T21:08:42.577019+0000","last_active":"2024-02-19T21:09:54.270786+0000","last_peered":"2024-02-19T21:09:54.270786+0000","last_clean":"2024-02-19T21:09:54.270786+0000","last_became_active":"2024-02-19T21:08:41.293491+0000","last_became_peered":"2024-02-19T21:08:41.293491+0000","last_unstale":"2024-02-19T21:09:54.270786+0000","last_undegraded":"2024-02-19T21:09:54.270786+0000","last_fullsized":"2024-02-19T21:09:54.270786+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:20.710731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:20.710731+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":374,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270883+0000","last_change":"2024-02-19T21:09:31.856861+0000","last_active":"2024-02-19T21:09:53.270883+0000","last_peered":"2024-02-19T21:09:53.270883+0000","last_clean":"2024-02-19T21:09:53.270883+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:09:53.270883+0000","last_undegraded":"2024-02-19T21:09:53.270883+0000","last_fullsized":"2024-02-19T21:09:53.270883+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:31.856784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:31.856784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":323,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270149+0000","last_change":"2024-02-19T21:06:51.423833+0000","last_active":"2024-02-19T21:09:53.270149+0000","last_peered":"2024-02-19T21:09:53.270149+0000","last_clean":"2024-02-19T21:09:53.270149+0000","last_became_active":"2024-02-19T21:04:39.657052+0000","last_became_peered":"2024-02-19T21:04:39.657052+0000","last_unstale":"2024-02-19T21:09:53.270149+0000","last_undegraded":"2024-02-19T21:09:53.270149+0000","last_fullsized":"2024-02-19T21:09:53.270149+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.423769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.423769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":128,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278791+0000","last_change":"2024-02-19T21:08:49.181165+0000","last_active":"2024-02-19T21:09:53.278791+0000","last_peered":"2024-02-19T21:09:53.278791+0000","last_clean":"2024-02-19T21:09:53.278791+0000","last_became_active":"2024-02-19T21:08:41.278810+0000","last_became_peered":"2024-02-19T21:08:41.278810+0000","last_unstale":"2024-02-19T21:09:53.278791+0000","last_undegraded":"2024-02-19T21:09:53.278791+0000","last_fullsized":"2024-02-19T21:09:53.278791+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:49.181100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.181100+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":406,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272699+0000","last_change":"2024-02-19T21:09:37.870271+0000","last_active":"2024-02-19T21:09:53.272699+0000","last_peered":"2024-02-19T21:09:53.272699+0000","last_clean":"2024-02-19T21:09:53.272699+0000","last_became_active":"2024-02-19T21:07:32.227557+0000","last_became_peered":"2024-02-19T21:07:32.227557+0000","last_unstale":"2024-02-19T21:09:53.272699+0000","last_undegraded":"2024-02-19T21:09:53.272699+0000","last_fullsized":"2024-02-19T21:09:53.272699+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.870218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.870218+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":380,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271275+0000","last_change":"2024-02-19T21:08:41.293711+0000","last_active":"2024-02-19T21:09:53.271275+0000","last_peered":"2024-02-19T21:09:53.271275+0000","last_clean":"2024-02-19T21:09:53.271275+0000","last_became_active":"2024-02-19T21:08:41.292701+0000","last_became_peered":"2024-02-19T21:08:41.292701+0000","last_unstale":"2024-02-19T21:09:53.271275+0000","last_undegraded":"2024-02-19T21:09:53.271275+0000","last_fullsized":"2024-02-19T21:09:53.271275+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:53.428093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:53.428093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":117,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272421+0000","last_change":"2024-02-19T21:08:04.279238+0000","last_active":"2024-02-19T21:09:53.272421+0000","last_peered":"2024-02-19T21:09:53.272421+0000","last_clean":"2024-02-19T21:09:53.272421+0000","last_became_active":"2024-02-19T21:08:04.279008+0000","last_became_peered":"2024-02-19T21:08:04.279008+0000","last_unstale":"2024-02-19T21:09:53.272421+0000","last_undegraded":"2024-02-19T21:09:53.272421+0000","last_fullsized":"2024-02-19T21:09:53.272421+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:05.721175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:05.721175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":117,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271768+0000","last_change":"2024-02-19T21:08:04.280833+0000","last_active":"2024-02-19T21:09:53.271768+0000","last_peered":"2024-02-19T21:09:53.271768+0000","last_clean":"2024-02-19T21:09:53.271768+0000","last_became_active":"2024-02-19T21:08:04.280514+0000","last_became_peered":"2024-02-19T21:08:04.280514+0000","last_unstale":"2024-02-19T21:09:53.271768+0000","last_undegraded":"2024-02-19T21:09:53.271768+0000","last_fullsized":"2024-02-19T21:09:53.271768+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:58.499841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:58.499841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"344'11","reported_seq":164,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268164+0000","last_change":"2024-02-19T21:09:39.597074+0000","last_active":"2024-02-19T21:09:54.268164+0000","last_peered":"2024-02-19T21:09:54.268164+0000","last_clean":"2024-02-19T21:09:54.268164+0000","last_became_active":"2024-02-19T21:08:41.296797+0000","last_became_peered":"2024-02-19T21:08:41.296797+0000","last_unstale":"2024-02-19T21:09:54.268164+0000","last_undegraded":"2024-02-19T21:09:54.268164+0000","last_fullsized":"2024-02-19T21:09:54.268164+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'11","last_scrub_stamp":"2024-02-19T21:09:39.597001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:39.597001+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":130,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269889+0000","last_change":"2024-02-19T21:08:41.921037+0000","last_active":"2024-02-19T21:09:53.269889+0000","last_peered":"2024-02-19T21:09:53.269889+0000","last_clean":"2024-02-19T21:09:53.269889+0000","last_became_active":"2024-02-19T21:08:41.268914+0000","last_became_peered":"2024-02-19T21:08:41.268914+0000","last_unstale":"2024-02-19T21:09:53.269889+0000","last_undegraded":"2024-02-19T21:09:53.269889+0000","last_fullsized":"2024-02-19T21:09:53.269889+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:06:46.813814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.813814+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":404,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278787+0000","last_change":"2024-02-19T21:09:38.541487+0000","last_active":"2024-02-19T21:09:53.278787+0000","last_peered":"2024-02-19T21:09:53.278787+0000","last_clean":"2024-02-19T21:09:53.278787+0000","last_became_active":"2024-02-19T21:07:36.263299+0000","last_became_peered":"2024-02-19T21:07:36.263299+0000","last_unstale":"2024-02-19T21:09:53.278787+0000","last_undegraded":"2024-02-19T21:09:53.278787+0000","last_fullsized":"2024-02-19T21:09:53.278787+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:38.541390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:38.541390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":370,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273477+0000","last_change":"2024-02-19T21:07:25.812691+0000","last_active":"2024-02-19T21:09:53.273477+0000","last_peered":"2024-02-19T21:09:53.273477+0000","last_clean":"2024-02-19T21:09:53.273477+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:09:53.273477+0000","last_undegraded":"2024-02-19T21:09:53.273477+0000","last_fullsized":"2024-02-19T21:09:53.273477+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:25.812627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:07:25.812627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":200,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273325+0000","last_change":"2024-02-19T21:06:40.245744+0000","last_active":"2024-02-19T21:09:53.273325+0000","last_peered":"2024-02-19T21:09:53.273325+0000","last_clean":"2024-02-19T21:09:53.273325+0000","last_became_active":"2024-02-19T21:06:40.245385+0000","last_became_peered":"2024-02-19T21:06:40.245385+0000","last_unstale":"2024-02-19T21:09:53.273325+0000","last_undegraded":"2024-02-19T21:09:53.273325+0000","last_fullsized":"2024-02-19T21:09:53.273325+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:16.780601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.780601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":351,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270070+0000","last_change":"2024-02-19T21:08:41.270754+0000","last_active":"2024-02-19T21:09:53.270070+0000","last_peered":"2024-02-19T21:09:53.270070+0000","last_clean":"2024-02-19T21:09:53.270070+0000","last_became_active":"2024-02-19T21:08:41.270483+0000","last_became_peered":"2024-02-19T21:08:41.270483+0000","last_unstale":"2024-02-19T21:09:53.270070+0000","last_undegraded":"2024-02-19T21:09:53.270070+0000","last_fullsized":"2024-02-19T21:09:53.270070+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:12.606585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.606585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":135,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270272+0000","last_change":"2024-02-19T21:08:04.343458+0000","last_active":"2024-02-19T21:09:53.270272+0000","last_peered":"2024-02-19T21:09:53.270272+0000","last_clean":"2024-02-19T21:09:53.270272+0000","last_became_active":"2024-02-19T21:08:04.242466+0000","last_became_peered":"2024-02-19T21:08:04.242466+0000","last_unstale":"2024-02-19T21:09:53.270272+0000","last_undegraded":"2024-02-19T21:09:53.270272+0000","last_fullsized":"2024-02-19T21:09:53.270272+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:06:46.816121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.816121+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":410,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273078+0000","last_change":"2024-02-19T21:08:41.294617+0000","last_active":"2024-02-19T21:09:53.273078+0000","last_peered":"2024-02-19T21:09:53.273078+0000","last_clean":"2024-02-19T21:09:53.273078+0000","last_became_active":"2024-02-19T21:08:41.294281+0000","last_became_peered":"2024-02-19T21:08:41.294281+0000","last_unstale":"2024-02-19T21:09:53.273078+0000","last_undegraded":"2024-02-19T21:09:53.273078+0000","last_fullsized":"2024-02-19T21:09:53.273078+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":354,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269752+0000","last_change":"2024-02-19T21:09:40.549754+0000","last_active":"2024-02-19T21:09:53.269752+0000","last_peered":"2024-02-19T21:09:53.269752+0000","last_clean":"2024-02-19T21:09:53.269752+0000","last_became_active":"2024-02-19T21:08:41.266934+0000","last_became_peered":"2024-02-19T21:08:41.266934+0000","last_unstale":"2024-02-19T21:09:53.269752+0000","last_undegraded":"2024-02-19T21:09:53.269752+0000","last_fullsized":"2024-02-19T21:09:53.269752+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:40.549690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.549690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":117,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272561+0000","last_change":"2024-02-19T21:08:04.267005+0000","last_active":"2024-02-19T21:09:53.272561+0000","last_peered":"2024-02-19T21:09:53.272561+0000","last_clean":"2024-02-19T21:09:53.272561+0000","last_became_active":"2024-02-19T21:08:04.266538+0000","last_became_peered":"2024-02-19T21:08:04.266538+0000","last_unstale":"2024-02-19T21:09:53.272561+0000","last_undegraded":"2024-02-19T21:09:53.272561+0000","last_fullsized":"2024-02-19T21:09:53.272561+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:58.847915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:58.847915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":373,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278931+0000","last_change":"2024-02-19T21:07:00.676713+0000","last_active":"2024-02-19T21:09:53.278931+0000","last_peered":"2024-02-19T21:09:53.278931+0000","last_clean":"2024-02-19T21:09:53.278931+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:09:53.278931+0000","last_undegraded":"2024-02-19T21:09:53.278931+0000","last_fullsized":"2024-02-19T21:09:53.278931+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:00.676634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:07:00.676634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":249,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270308+0000","last_change":"2024-02-19T21:06:40.224213+0000","last_active":"2024-02-19T21:09:53.270308+0000","last_peered":"2024-02-19T21:09:53.270308+0000","last_clean":"2024-02-19T21:09:53.270308+0000","last_became_active":"2024-02-19T21:06:40.223768+0000","last_became_peered":"2024-02-19T21:06:40.223768+0000","last_unstale":"2024-02-19T21:09:53.270308+0000","last_undegraded":"2024-02-19T21:09:53.270308+0000","last_fullsized":"2024-02-19T21:09:53.270308+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:06:10.826889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:10.826889+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":417,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273538+0000","last_change":"2024-02-19T21:09:35.824601+0000","last_active":"2024-02-19T21:09:53.273538+0000","last_peered":"2024-02-19T21:09:53.273538+0000","last_clean":"2024-02-19T21:09:53.273538+0000","last_became_active":"2024-02-19T21:08:41.303606+0000","last_became_peered":"2024-02-19T21:08:41.303606+0000","last_unstale":"2024-02-19T21:09:53.273538+0000","last_undegraded":"2024-02-19T21:09:53.273538+0000","last_fullsized":"2024-02-19T21:09:53.273538+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:35.824520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:35.824520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":368,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269396+0000","last_change":"2024-02-19T21:08:57.653280+0000","last_active":"2024-02-19T21:09:53.269396+0000","last_peered":"2024-02-19T21:09:53.269396+0000","last_clean":"2024-02-19T21:09:53.269396+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:09:53.269396+0000","last_undegraded":"2024-02-19T21:09:53.269396+0000","last_fullsized":"2024-02-19T21:09:53.269396+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:57.653204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:08:57.653204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":322,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.281970+0000","last_change":"2024-02-19T21:08:12.309008+0000","last_active":"2024-02-19T21:09:53.281970+0000","last_peered":"2024-02-19T21:09:53.281970+0000","last_clean":"2024-02-19T21:09:53.281970+0000","last_became_active":"2024-02-19T21:04:39.718926+0000","last_became_peered":"2024-02-19T21:04:39.718926+0000","last_unstale":"2024-02-19T21:09:53.281970+0000","last_undegraded":"2024-02-19T21:09:53.281970+0000","last_fullsized":"2024-02-19T21:09:53.281970+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:12.308951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.308951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":371,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271321+0000","last_change":"2024-02-19T21:08:11.431726+0000","last_active":"2024-02-19T21:09:53.271321+0000","last_peered":"2024-02-19T21:09:53.271321+0000","last_clean":"2024-02-19T21:09:53.271321+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:09:53.271321+0000","last_undegraded":"2024-02-19T21:09:53.271321+0000","last_fullsized":"2024-02-19T21:09:53.271321+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:11.431637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:11.431637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":235,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273173+0000","last_change":"2024-02-19T21:09:30.704618+0000","last_active":"2024-02-19T21:09:53.273173+0000","last_peered":"2024-02-19T21:09:53.273173+0000","last_clean":"2024-02-19T21:09:53.273173+0000","last_became_active":"2024-02-19T21:06:40.240321+0000","last_became_peered":"2024-02-19T21:06:40.240321+0000","last_unstale":"2024-02-19T21:09:53.273173+0000","last_undegraded":"2024-02-19T21:09:53.273173+0000","last_fullsized":"2024-02-19T21:09:53.273173+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:09:30.704543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:30.704543+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"397'2","reported_seq":414,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272596+0000","last_change":"2024-02-19T21:09:40.934534+0000","last_active":"2024-02-19T21:09:53.272596+0000","last_peered":"2024-02-19T21:09:53.272596+0000","last_clean":"2024-02-19T21:09:53.272596+0000","last_became_active":"2024-02-19T21:08:10.225480+0000","last_became_peered":"2024-02-19T21:08:10.225480+0000","last_unstale":"2024-02-19T21:09:53.272596+0000","last_undegraded":"2024-02-19T21:09:53.272596+0000","last_fullsized":"2024-02-19T21:09:53.272596+0000","mapping_epoch":57,"log_start":"58'1","ondisk_log_start":"58'1","created":57,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.934474+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":186,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270165+0000","last_change":"2024-02-19T21:08:41.277088+0000","last_active":"2024-02-19T21:09:53.270165+0000","last_peered":"2024-02-19T21:09:53.270165+0000","last_clean":"2024-02-19T21:09:53.270165+0000","last_became_active":"2024-02-19T21:08:41.276642+0000","last_became_peered":"2024-02-19T21:08:41.276642+0000","last_unstale":"2024-02-19T21:09:53.270165+0000","last_undegraded":"2024-02-19T21:09:53.270165+0000","last_fullsized":"2024-02-19T21:09:53.270165+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:35.542752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:08:35.542752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":386,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272323+0000","last_change":"2024-02-19T21:08:04.282673+0000","last_active":"2024-02-19T21:09:53.272323+0000","last_peered":"2024-02-19T21:09:53.272323+0000","last_clean":"2024-02-19T21:09:53.272323+0000","last_became_active":"2024-02-19T21:08:04.282243+0000","last_became_peered":"2024-02-19T21:08:04.282243+0000","last_unstale":"2024-02-19T21:09:53.272323+0000","last_undegraded":"2024-02-19T21:09:53.272323+0000","last_fullsized":"2024-02-19T21:09:53.272323+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.627708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.627708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":322,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278360+0000","last_change":"2024-02-19T21:07:34.550279+0000","last_active":"2024-02-19T21:09:53.278360+0000","last_peered":"2024-02-19T21:09:53.278360+0000","last_clean":"2024-02-19T21:09:53.278360+0000","last_became_active":"2024-02-19T21:04:39.662125+0000","last_became_peered":"2024-02-19T21:04:39.662125+0000","last_unstale":"2024-02-19T21:09:53.278360+0000","last_undegraded":"2024-02-19T21:09:53.278360+0000","last_fullsized":"2024-02-19T21:09:53.278360+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:34.550217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:34.550217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":370,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273601+0000","last_change":"2024-02-19T21:06:51.666781+0000","last_active":"2024-02-19T21:09:53.273601+0000","last_peered":"2024-02-19T21:09:53.273601+0000","last_clean":"2024-02-19T21:09:53.273601+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:09:53.273601+0000","last_undegraded":"2024-02-19T21:09:53.273601+0000","last_fullsized":"2024-02-19T21:09:53.273601+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.666708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.666708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":417,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278591+0000","last_change":"2024-02-19T21:08:41.312655+0000","last_active":"2024-02-19T21:09:53.278591+0000","last_peered":"2024-02-19T21:09:53.278591+0000","last_clean":"2024-02-19T21:09:53.278591+0000","last_became_active":"2024-02-19T21:08:41.263010+0000","last_became_peered":"2024-02-19T21:08:41.263010+0000","last_unstale":"2024-02-19T21:09:53.278591+0000","last_undegraded":"2024-02-19T21:09:53.278591+0000","last_fullsized":"2024-02-19T21:09:53.278591+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:06:26.776963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:26.776963+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":383,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272647+0000","last_change":"2024-02-19T21:08:37.935245+0000","last_active":"2024-02-19T21:09:53.272647+0000","last_peered":"2024-02-19T21:09:53.272647+0000","last_clean":"2024-02-19T21:09:53.272647+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:09:53.272647+0000","last_undegraded":"2024-02-19T21:09:53.272647+0000","last_fullsized":"2024-02-19T21:09:53.272647+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:37.935175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:08:37.935175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":402,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270241+0000","last_change":"2024-02-19T21:07:58.239278+0000","last_active":"2024-02-19T21:09:53.270241+0000","last_peered":"2024-02-19T21:09:53.270241+0000","last_clean":"2024-02-19T21:09:53.270241+0000","last_became_active":"2024-02-19T21:07:58.238944+0000","last_became_peered":"2024-02-19T21:07:58.238944+0000","last_unstale":"2024-02-19T21:09:53.270241+0000","last_undegraded":"2024-02-19T21:09:53.270241+0000","last_fullsized":"2024-02-19T21:09:53.270241+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:17.907318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:17.907318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":122,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271806+0000","last_change":"2024-02-19T21:08:53.820016+0000","last_active":"2024-02-19T21:09:53.271806+0000","last_peered":"2024-02-19T21:09:53.271806+0000","last_clean":"2024-02-19T21:09:53.271806+0000","last_became_active":"2024-02-19T21:08:04.278265+0000","last_became_peered":"2024-02-19T21:08:04.278265+0000","last_unstale":"2024-02-19T21:09:53.271806+0000","last_undegraded":"2024-02-19T21:09:53.271806+0000","last_fullsized":"2024-02-19T21:09:53.271806+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:53.819949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:53.819949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":396,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271369+0000","last_change":"2024-02-19T21:08:41.306530+0000","last_active":"2024-02-19T21:09:53.271369+0000","last_peered":"2024-02-19T21:09:53.271369+0000","last_clean":"2024-02-19T21:09:53.271369+0000","last_became_active":"2024-02-19T21:08:41.305847+0000","last_became_peered":"2024-02-19T21:08:41.305847+0000","last_unstale":"2024-02-19T21:09:53.271369+0000","last_undegraded":"2024-02-19T21:09:53.271369+0000","last_fullsized":"2024-02-19T21:09:53.271369+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:16.863398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.863398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":323,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269746+0000","last_change":"2024-02-19T21:06:52.403019+0000","last_active":"2024-02-19T21:09:53.269746+0000","last_peered":"2024-02-19T21:09:53.269746+0000","last_clean":"2024-02-19T21:09:53.269746+0000","last_became_active":"2024-02-19T21:04:39.666739+0000","last_became_peered":"2024-02-19T21:04:39.666739+0000","last_unstale":"2024-02-19T21:09:53.269746+0000","last_undegraded":"2024-02-19T21:09:53.269746+0000","last_fullsized":"2024-02-19T21:09:53.269746+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:52.402941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:52.402941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":370,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.281809+0000","last_change":"2024-02-19T21:06:23.273349+0000","last_active":"2024-02-19T21:09:53.281809+0000","last_peered":"2024-02-19T21:09:53.281809+0000","last_clean":"2024-02-19T21:09:53.281809+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:09:53.281809+0000","last_undegraded":"2024-02-19T21:09:53.281809+0000","last_fullsized":"2024-02-19T21:09:53.281809+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:23.273274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:23.273274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":388,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.281798+0000","last_change":"2024-02-19T21:07:01.264194+0000","last_active":"2024-02-19T21:09:53.281798+0000","last_peered":"2024-02-19T21:09:53.281798+0000","last_clean":"2024-02-19T21:09:53.281798+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:09:53.281798+0000","last_undegraded":"2024-02-19T21:09:53.281798+0000","last_fullsized":"2024-02-19T21:09:53.281798+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:07:01.264093+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:07:01.264093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":420,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.280991+0000","last_change":"2024-02-19T21:09:37.955966+0000","last_active":"2024-02-19T21:09:53.280991+0000","last_peered":"2024-02-19T21:09:53.280991+0000","last_clean":"2024-02-19T21:09:53.280991+0000","last_became_active":"2024-02-19T21:08:06.473207+0000","last_became_peered":"2024-02-19T21:08:06.473207+0000","last_unstale":"2024-02-19T21:09:53.280991+0000","last_undegraded":"2024-02-19T21:09:53.280991+0000","last_fullsized":"2024-02-19T21:09:53.280991+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.955872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.955872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":888,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447932+0000","last_change":"2024-02-19T21:06:10.791107+0000","last_active":"2024-02-19T21:09:54.447932+0000","last_peered":"2024-02-19T21:09:54.447932+0000","last_clean":"2024-02-19T21:09:54.447932+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:09:54.447932+0000","last_undegraded":"2024-02-19T21:09:54.447932+0000","last_fullsized":"2024-02-19T21:09:54.447932+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:06:10.791036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:10.791036+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":511,"num_read_kb":506,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":386,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273436+0000","last_change":"2024-02-19T21:08:04.265242+0000","last_active":"2024-02-19T21:09:53.273436+0000","last_peered":"2024-02-19T21:09:53.273436+0000","last_clean":"2024-02-19T21:09:53.273436+0000","last_became_active":"2024-02-19T21:08:04.264896+0000","last_became_peered":"2024-02-19T21:08:04.264896+0000","last_unstale":"2024-02-19T21:09:53.273436+0000","last_undegraded":"2024-02-19T21:09:53.273436+0000","last_fullsized":"2024-02-19T21:09:53.273436+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.690596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.690596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":322,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272633+0000","last_change":"2024-02-19T21:08:12.584947+0000","last_active":"2024-02-19T21:09:53.272633+0000","last_peered":"2024-02-19T21:09:53.272633+0000","last_clean":"2024-02-19T21:09:53.272633+0000","last_became_active":"2024-02-19T21:04:39.691434+0000","last_became_peered":"2024-02-19T21:04:39.691434+0000","last_unstale":"2024-02-19T21:09:53.272633+0000","last_undegraded":"2024-02-19T21:09:53.272633+0000","last_fullsized":"2024-02-19T21:09:53.272633+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:12.584874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.584874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":214,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273116+0000","last_change":"2024-02-19T21:08:41.309414+0000","last_active":"2024-02-19T21:09:53.273116+0000","last_peered":"2024-02-19T21:09:53.273116+0000","last_clean":"2024-02-19T21:09:53.273116+0000","last_became_active":"2024-02-19T21:08:41.303115+0000","last_became_peered":"2024-02-19T21:08:41.303115+0000","last_unstale":"2024-02-19T21:09:53.273116+0000","last_undegraded":"2024-02-19T21:09:53.273116+0000","last_fullsized":"2024-02-19T21:09:53.273116+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:36.716464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.716464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"304'30","reported_seq":586,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446849+0000","last_change":"2024-02-19T21:08:05.292869+0000","last_active":"2024-02-19T21:09:54.446849+0000","last_peered":"2024-02-19T21:09:54.446849+0000","last_clean":"2024-02-19T21:09:54.446849+0000","last_became_active":"2024-02-19T21:08:04.266322+0000","last_became_peered":"2024-02-19T21:08:04.266322+0000","last_unstale":"2024-02-19T21:09:54.446849+0000","last_undegraded":"2024-02-19T21:09:54.446849+0000","last_fullsized":"2024-02-19T21:09:54.446849+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-19T21:05:42.927640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:42.927640+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":402,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270035+0000","last_change":"2024-02-19T21:08:02.435986+0000","last_active":"2024-02-19T21:09:53.270035+0000","last_peered":"2024-02-19T21:09:53.270035+0000","last_clean":"2024-02-19T21:09:53.270035+0000","last_became_active":"2024-02-19T21:08:02.435663+0000","last_became_peered":"2024-02-19T21:08:02.435663+0000","last_unstale":"2024-02-19T21:09:53.270035+0000","last_undegraded":"2024-02-19T21:09:53.270035+0000","last_fullsized":"2024-02-19T21:09:53.270035+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:06:19.845394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":126,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271133+0000","last_change":"2024-02-19T21:08:41.272198+0000","last_active":"2024-02-19T21:09:53.271133+0000","last_peered":"2024-02-19T21:09:53.271133+0000","last_clean":"2024-02-19T21:09:53.271133+0000","last_became_active":"2024-02-19T21:08:41.271603+0000","last_became_peered":"2024-02-19T21:08:41.271603+0000","last_unstale":"2024-02-19T21:09:53.271133+0000","last_undegraded":"2024-02-19T21:09:53.271133+0000","last_fullsized":"2024-02-19T21:09:53.271133+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:26.886817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:07:26.886817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":322,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.280895+0000","last_change":"2024-02-19T21:06:54.355870+0000","last_active":"2024-02-19T21:09:53.280895+0000","last_peered":"2024-02-19T21:09:53.280895+0000","last_clean":"2024-02-19T21:09:53.280895+0000","last_became_active":"2024-02-19T21:04:39.721416+0000","last_became_peered":"2024-02-19T21:04:39.721416+0000","last_unstale":"2024-02-19T21:09:53.280895+0000","last_undegraded":"2024-02-19T21:09:53.280895+0000","last_fullsized":"2024-02-19T21:09:53.280895+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.355809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.355809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"220'9","reported_seq":260,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446904+0000","last_change":"2024-02-19T21:06:40.228012+0000","last_active":"2024-02-19T21:09:54.446904+0000","last_peered":"2024-02-19T21:09:54.446904+0000","last_clean":"2024-02-19T21:09:54.446904+0000","last_became_active":"2024-02-19T21:06:40.227604+0000","last_became_peered":"2024-02-19T21:06:40.227604+0000","last_unstale":"2024-02-19T21:09:54.446904+0000","last_undegraded":"2024-02-19T21:09:54.446904+0000","last_fullsized":"2024-02-19T21:09:54.446904+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:21.819105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:21.819105+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":408,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269220+0000","last_change":"2024-02-19T21:09:28.659562+0000","last_active":"2024-02-19T21:09:53.269220+0000","last_peered":"2024-02-19T21:09:53.269220+0000","last_clean":"2024-02-19T21:09:53.269220+0000","last_became_active":"2024-02-19T21:08:41.306070+0000","last_became_peered":"2024-02-19T21:08:41.306070+0000","last_unstale":"2024-02-19T21:09:53.269220+0000","last_undegraded":"2024-02-19T21:09:53.269220+0000","last_fullsized":"2024-02-19T21:09:53.269220+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:09:28.659495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.659495+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":419,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271886+0000","last_change":"2024-02-19T21:07:54.235439+0000","last_active":"2024-02-19T21:09:53.271886+0000","last_peered":"2024-02-19T21:09:53.271886+0000","last_clean":"2024-02-19T21:09:53.271886+0000","last_became_active":"2024-02-19T21:07:54.235080+0000","last_became_peered":"2024-02-19T21:07:54.235080+0000","last_unstale":"2024-02-19T21:09:53.271886+0000","last_undegraded":"2024-02-19T21:09:53.271886+0000","last_fullsized":"2024-02-19T21:09:53.271886+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:47.885024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:05:47.885024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":122,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270187+0000","last_change":"2024-02-19T21:09:27.747880+0000","last_active":"2024-02-19T21:09:53.270187+0000","last_peered":"2024-02-19T21:09:53.270187+0000","last_clean":"2024-02-19T21:09:53.270187+0000","last_became_active":"2024-02-19T21:08:04.244317+0000","last_became_peered":"2024-02-19T21:08:04.244317+0000","last_unstale":"2024-02-19T21:09:53.270187+0000","last_undegraded":"2024-02-19T21:09:53.270187+0000","last_fullsized":"2024-02-19T21:09:53.270187+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.747789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.747789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":117,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271174+0000","last_change":"2024-02-19T21:08:04.261805+0000","last_active":"2024-02-19T21:09:53.271174+0000","last_peered":"2024-02-19T21:09:53.271174+0000","last_clean":"2024-02-19T21:09:53.271174+0000","last_became_active":"2024-02-19T21:08:04.261408+0000","last_became_peered":"2024-02-19T21:08:04.261408+0000","last_unstale":"2024-02-19T21:09:53.271174+0000","last_undegraded":"2024-02-19T21:09:53.271174+0000","last_fullsized":"2024-02-19T21:09:53.271174+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:58.726858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:58.726858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":370,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.280802+0000","last_change":"2024-02-19T21:06:20.194149+0000","last_active":"2024-02-19T21:09:53.280802+0000","last_peered":"2024-02-19T21:09:53.280802+0000","last_clean":"2024-02-19T21:09:53.280802+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:09:53.280802+0000","last_undegraded":"2024-02-19T21:09:53.280802+0000","last_fullsized":"2024-02-19T21:09:53.280802+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:20.194018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:06:20.194018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":128,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269113+0000","last_change":"2024-02-19T21:08:42.355747+0000","last_active":"2024-02-19T21:09:53.269113+0000","last_peered":"2024-02-19T21:09:53.269113+0000","last_clean":"2024-02-19T21:09:53.269113+0000","last_became_active":"2024-02-19T21:08:41.294283+0000","last_became_peered":"2024-02-19T21:08:41.294283+0000","last_unstale":"2024-02-19T21:09:53.269113+0000","last_undegraded":"2024-02-19T21:09:53.269113+0000","last_fullsized":"2024-02-19T21:09:53.269113+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:06:48.844301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.844301+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":140,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272398+0000","last_change":"2024-02-19T21:08:41.279224+0000","last_active":"2024-02-19T21:09:53.272398+0000","last_peered":"2024-02-19T21:09:53.272398+0000","last_clean":"2024-02-19T21:09:53.272398+0000","last_became_active":"2024-02-19T21:08:41.269102+0000","last_became_peered":"2024-02-19T21:08:41.269102+0000","last_unstale":"2024-02-19T21:09:53.272398+0000","last_undegraded":"2024-02-19T21:09:53.272398+0000","last_fullsized":"2024-02-19T21:09:53.272398+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:07:33.765035+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:07:33.765035+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":122,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270622+0000","last_change":"2024-02-19T21:08:57.881002+0000","last_active":"2024-02-19T21:09:53.270622+0000","last_peered":"2024-02-19T21:09:53.270622+0000","last_clean":"2024-02-19T21:09:53.270622+0000","last_became_active":"2024-02-19T21:08:04.264593+0000","last_became_peered":"2024-02-19T21:08:04.264593+0000","last_unstale":"2024-02-19T21:09:53.270622+0000","last_undegraded":"2024-02-19T21:09:53.270622+0000","last_fullsized":"2024-02-19T21:09:53.270622+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:57.880935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:08:57.880935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":322,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.281662+0000","last_change":"2024-02-19T21:08:03.177937+0000","last_active":"2024-02-19T21:09:53.281662+0000","last_peered":"2024-02-19T21:09:53.281662+0000","last_clean":"2024-02-19T21:09:53.281662+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:09:53.281662+0000","last_undegraded":"2024-02-19T21:09:53.281662+0000","last_fullsized":"2024-02-19T21:09:53.281662+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:03.177867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:03.177867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":127,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271055+0000","last_change":"2024-02-19T21:08:06.478479+0000","last_active":"2024-02-19T21:09:53.271055+0000","last_peered":"2024-02-19T21:09:53.271055+0000","last_clean":"2024-02-19T21:09:53.271055+0000","last_became_active":"2024-02-19T21:08:04.258740+0000","last_became_peered":"2024-02-19T21:08:04.258740+0000","last_unstale":"2024-02-19T21:09:53.271055+0000","last_undegraded":"2024-02-19T21:09:53.271055+0000","last_fullsized":"2024-02-19T21:09:53.271055+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:06:09.793208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.793208+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":374,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.280752+0000","last_change":"2024-02-19T21:08:11.307651+0000","last_active":"2024-02-19T21:09:53.280752+0000","last_peered":"2024-02-19T21:09:53.280752+0000","last_clean":"2024-02-19T21:09:53.280752+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:09:53.280752+0000","last_undegraded":"2024-02-19T21:09:53.280752+0000","last_fullsized":"2024-02-19T21:09:53.280752+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:11.307561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:11.307561+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":117,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271436+0000","last_change":"2024-02-19T21:08:04.265732+0000","last_active":"2024-02-19T21:09:53.271436+0000","last_peered":"2024-02-19T21:09:53.271436+0000","last_clean":"2024-02-19T21:09:53.271436+0000","last_became_active":"2024-02-19T21:08:04.264972+0000","last_became_peered":"2024-02-19T21:08:04.264972+0000","last_unstale":"2024-02-19T21:09:53.271436+0000","last_undegraded":"2024-02-19T21:09:53.271436+0000","last_fullsized":"2024-02-19T21:09:53.271436+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:56.709750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:06:56.709750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":382,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.282340+0000","last_change":"2024-02-19T21:08:48.995892+0000","last_active":"2024-02-19T21:09:53.282340+0000","last_peered":"2024-02-19T21:09:53.282340+0000","last_clean":"2024-02-19T21:09:53.282340+0000","last_became_active":"2024-02-19T21:08:41.286874+0000","last_became_peered":"2024-02-19T21:08:41.286874+0000","last_unstale":"2024-02-19T21:09:53.282340+0000","last_undegraded":"2024-02-19T21:09:53.282340+0000","last_fullsized":"2024-02-19T21:09:53.282340+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:48.995828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:08:48.995828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":81,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269017+0000","last_change":"2024-02-19T21:08:41.311611+0000","last_active":"2024-02-19T21:09:53.269017+0000","last_peered":"2024-02-19T21:09:53.269017+0000","last_clean":"2024-02-19T21:09:53.269017+0000","last_became_active":"2024-02-19T21:08:41.311298+0000","last_became_peered":"2024-02-19T21:08:41.311298+0000","last_unstale":"2024-02-19T21:09:53.269017+0000","last_undegraded":"2024-02-19T21:09:53.269017+0000","last_fullsized":"2024-02-19T21:09:53.269017+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:21.704719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:21.704719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":125,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271439+0000","last_change":"2024-02-19T21:08:04.356074+0000","last_active":"2024-02-19T21:09:53.271439+0000","last_peered":"2024-02-19T21:09:53.271439+0000","last_clean":"2024-02-19T21:09:53.271439+0000","last_became_active":"2024-02-19T21:08:04.264472+0000","last_became_peered":"2024-02-19T21:08:04.264472+0000","last_unstale":"2024-02-19T21:09:53.271439+0000","last_undegraded":"2024-02-19T21:09:53.271439+0000","last_fullsized":"2024-02-19T21:09:53.271439+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:06:48.855268+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.855268+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":386,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.277955+0000","last_change":"2024-02-19T21:09:41.519382+0000","last_active":"2024-02-19T21:09:53.277955+0000","last_peered":"2024-02-19T21:09:53.277955+0000","last_clean":"2024-02-19T21:09:53.277955+0000","last_became_active":"2024-02-19T21:08:41.295743+0000","last_became_peered":"2024-02-19T21:08:41.295743+0000","last_unstale":"2024-02-19T21:09:53.277955+0000","last_undegraded":"2024-02-19T21:09:53.277955+0000","last_fullsized":"2024-02-19T21:09:53.277955+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:41.519294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:41.519294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":391,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270810+0000","last_change":"2024-02-19T21:08:56.601041+0000","last_active":"2024-02-19T21:09:53.270810+0000","last_peered":"2024-02-19T21:09:53.270810+0000","last_clean":"2024-02-19T21:09:53.270810+0000","last_became_active":"2024-02-19T21:08:04.265573+0000","last_became_peered":"2024-02-19T21:08:04.265573+0000","last_unstale":"2024-02-19T21:09:53.270810+0000","last_undegraded":"2024-02-19T21:09:53.270810+0000","last_fullsized":"2024-02-19T21:09:53.270810+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:56.600951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:56.600951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":382,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272395+0000","last_change":"2024-02-19T21:09:27.770579+0000","last_active":"2024-02-19T21:09:53.272395+0000","last_peered":"2024-02-19T21:09:53.272395+0000","last_clean":"2024-02-19T21:09:53.272395+0000","last_became_active":"2024-02-19T21:08:41.268962+0000","last_became_peered":"2024-02-19T21:08:41.268962+0000","last_unstale":"2024-02-19T21:09:53.272395+0000","last_undegraded":"2024-02-19T21:09:53.272395+0000","last_fullsized":"2024-02-19T21:09:53.272395+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.770501+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":332,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270075+0000","last_change":"2024-02-19T21:08:50.600747+0000","last_active":"2024-02-19T21:09:53.270075+0000","last_peered":"2024-02-19T21:09:53.270075+0000","last_clean":"2024-02-19T21:09:53.270075+0000","last_became_active":"2024-02-19T21:08:41.292646+0000","last_became_peered":"2024-02-19T21:08:41.292646+0000","last_unstale":"2024-02-19T21:09:53.270075+0000","last_undegraded":"2024-02-19T21:09:53.270075+0000","last_fullsized":"2024-02-19T21:09:53.270075+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:50.600659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:50.600659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":129,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272478+0000","last_change":"2024-02-19T21:08:04.361409+0000","last_active":"2024-02-19T21:09:53.272478+0000","last_peered":"2024-02-19T21:09:53.272478+0000","last_clean":"2024-02-19T21:09:53.272478+0000","last_became_active":"2024-02-19T21:08:04.267216+0000","last_became_peered":"2024-02-19T21:08:04.267216+0000","last_unstale":"2024-02-19T21:09:53.272478+0000","last_undegraded":"2024-02-19T21:09:53.272478+0000","last_fullsized":"2024-02-19T21:09:53.272478+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:07:01.731068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:07:01.731068+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":398,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269983+0000","last_change":"2024-02-19T21:09:40.986717+0000","last_active":"2024-02-19T21:09:53.269983+0000","last_peered":"2024-02-19T21:09:53.269983+0000","last_clean":"2024-02-19T21:09:53.269983+0000","last_became_active":"2024-02-19T21:08:04.266728+0000","last_became_peered":"2024-02-19T21:08:04.266728+0000","last_unstale":"2024-02-19T21:09:53.269983+0000","last_undegraded":"2024-02-19T21:09:53.269983+0000","last_fullsized":"2024-02-19T21:09:53.269983+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.986635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":483,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269540+0000","last_change":"2024-02-19T21:09:26.916067+0000","last_active":"2024-02-19T21:09:54.269540+0000","last_peered":"2024-02-19T21:09:54.269540+0000","last_clean":"2024-02-19T21:09:54.269540+0000","last_became_active":"2024-02-19T21:01:20.288799+0000","last_became_peered":"2024-02-19T21:01:20.288799+0000","last_unstale":"2024-02-19T21:09:54.269540+0000","last_undegraded":"2024-02-19T21:09:54.269540+0000","last_fullsized":"2024-02-19T21:09:54.269540+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:09:26.916002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:26.916002+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":241,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273907+0000","last_change":"2024-02-19T21:08:57.007906+0000","last_active":"2024-02-19T21:09:53.273907+0000","last_peered":"2024-02-19T21:09:53.273907+0000","last_clean":"2024-02-19T21:09:53.273907+0000","last_became_active":"2024-02-19T21:08:04.266747+0000","last_became_peered":"2024-02-19T21:08:04.266747+0000","last_unstale":"2024-02-19T21:09:53.273907+0000","last_undegraded":"2024-02-19T21:09:53.273907+0000","last_fullsized":"2024-02-19T21:09:53.273907+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:08:57.007815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:08:57.007815+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":203,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.277552+0000","last_change":"2024-02-19T21:08:55.202543+0000","last_active":"2024-02-19T21:09:53.277552+0000","last_peered":"2024-02-19T21:09:53.277552+0000","last_clean":"2024-02-19T21:09:53.277552+0000","last_became_active":"2024-02-19T21:06:40.219590+0000","last_became_peered":"2024-02-19T21:06:40.219590+0000","last_unstale":"2024-02-19T21:09:53.277552+0000","last_undegraded":"2024-02-19T21:09:53.277552+0000","last_fullsized":"2024-02-19T21:09:53.277552+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:55.202463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:55.202463+0000","last_clean_scrub_stamp":"2024-02-19T21:08:55.202463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":427,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270715+0000","last_change":"2024-02-19T21:08:04.681714+0000","last_active":"2024-02-19T21:09:53.270715+0000","last_peered":"2024-02-19T21:09:53.270715+0000","last_clean":"2024-02-19T21:09:53.270715+0000","last_became_active":"2024-02-19T21:08:04.266103+0000","last_became_peered":"2024-02-19T21:08:04.266103+0000","last_unstale":"2024-02-19T21:09:53.270715+0000","last_undegraded":"2024-02-19T21:09:53.270715+0000","last_fullsized":"2024-02-19T21:09:53.270715+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:06:47.541012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.541012+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":191,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270344+0000","last_change":"2024-02-19T21:08:41.280409+0000","last_active":"2024-02-19T21:09:53.270344+0000","last_peered":"2024-02-19T21:09:53.270344+0000","last_clean":"2024-02-19T21:09:53.270344+0000","last_became_active":"2024-02-19T21:08:41.279782+0000","last_became_peered":"2024-02-19T21:08:41.279782+0000","last_unstale":"2024-02-19T21:09:53.270344+0000","last_undegraded":"2024-02-19T21:09:53.270344+0000","last_fullsized":"2024-02-19T21:09:53.270344+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:30.769388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:07:30.769388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":149,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.271207+0000","last_change":"2024-02-19T21:08:37.867477+0000","last_active":"2024-02-19T21:09:54.271207+0000","last_peered":"2024-02-19T21:09:54.271207+0000","last_clean":"2024-02-19T21:09:54.271207+0000","last_became_active":"2024-02-19T21:08:04.256045+0000","last_became_peered":"2024-02-19T21:08:04.256045+0000","last_unstale":"2024-02-19T21:09:54.271207+0000","last_undegraded":"2024-02-19T21:09:54.271207+0000","last_fullsized":"2024-02-19T21:09:54.271207+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'10","last_scrub_stamp":"2024-02-19T21:08:37.867405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:37.867405+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":85,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.281796+0000","last_change":"2024-02-19T21:09:43.936218+0000","last_active":"2024-02-19T21:09:53.281796+0000","last_peered":"2024-02-19T21:09:53.281796+0000","last_clean":"2024-02-19T21:09:53.281796+0000","last_became_active":"2024-02-19T21:08:41.286185+0000","last_became_peered":"2024-02-19T21:08:41.286185+0000","last_unstale":"2024-02-19T21:09:53.281796+0000","last_undegraded":"2024-02-19T21:09:53.281796+0000","last_fullsized":"2024-02-19T21:09:53.281796+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_clean_scrub_stamp":"2024-02-19T21:09:43.936153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":335,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.280807+0000","last_change":"2024-02-19T21:06:49.305197+0000","last_active":"2024-02-19T21:09:53.280807+0000","last_peered":"2024-02-19T21:09:53.280807+0000","last_clean":"2024-02-19T21:09:53.280807+0000","last_became_active":"2024-02-19T21:04:39.760167+0000","last_became_peered":"2024-02-19T21:04:39.760167+0000","last_unstale":"2024-02-19T21:09:53.280807+0000","last_undegraded":"2024-02-19T21:09:53.280807+0000","last_fullsized":"2024-02-19T21:09:53.280807+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-02-19T21:06:49.305118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.305118+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":109,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.268894+0000","last_change":"2024-02-19T21:09:40.676522+0000","last_active":"2024-02-19T21:09:53.268894+0000","last_peered":"2024-02-19T21:09:53.268894+0000","last_clean":"2024-02-19T21:09:53.268894+0000","last_became_active":"2024-02-19T21:08:26.235018+0000","last_became_peered":"2024-02-19T21:08:26.235018+0000","last_unstale":"2024-02-19T21:09:53.268894+0000","last_undegraded":"2024-02-19T21:09:53.268894+0000","last_fullsized":"2024-02-19T21:09:53.268894+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:09:40.676439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.676439+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":395,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278285+0000","last_change":"2024-02-19T21:08:09.588799+0000","last_active":"2024-02-19T21:09:53.278285+0000","last_peered":"2024-02-19T21:09:53.278285+0000","last_clean":"2024-02-19T21:09:53.278285+0000","last_became_active":"2024-02-19T21:08:04.258699+0000","last_became_peered":"2024-02-19T21:08:04.258699+0000","last_unstale":"2024-02-19T21:09:53.278285+0000","last_undegraded":"2024-02-19T21:09:53.278285+0000","last_fullsized":"2024-02-19T21:09:53.278285+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:09.588687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:08:09.588687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":122,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270777+0000","last_change":"2024-02-19T21:08:12.856127+0000","last_active":"2024-02-19T21:09:53.270777+0000","last_peered":"2024-02-19T21:09:53.270777+0000","last_clean":"2024-02-19T21:09:53.270777+0000","last_became_active":"2024-02-19T21:08:04.264983+0000","last_became_peered":"2024-02-19T21:08:04.264983+0000","last_unstale":"2024-02-19T21:09:53.270777+0000","last_undegraded":"2024-02-19T21:09:53.270777+0000","last_fullsized":"2024-02-19T21:09:53.270777+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.856015+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":206,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269825+0000","last_change":"2024-02-19T21:09:38.941012+0000","last_active":"2024-02-19T21:09:53.269825+0000","last_peered":"2024-02-19T21:09:53.269825+0000","last_clean":"2024-02-19T21:09:53.269825+0000","last_became_active":"2024-02-19T21:06:40.239199+0000","last_became_peered":"2024-02-19T21:06:40.239199+0000","last_unstale":"2024-02-19T21:09:53.269825+0000","last_undegraded":"2024-02-19T21:09:53.269825+0000","last_fullsized":"2024-02-19T21:09:53.269825+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:38.940935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:38.940935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":320,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278837+0000","last_change":"2024-02-19T21:07:35.505665+0000","last_active":"2024-02-19T21:09:53.278837+0000","last_peered":"2024-02-19T21:09:53.278837+0000","last_clean":"2024-02-19T21:09:53.278837+0000","last_became_active":"2024-02-19T21:04:39.672228+0000","last_became_peered":"2024-02-19T21:04:39.672228+0000","last_unstale":"2024-02-19T21:09:53.278837+0000","last_undegraded":"2024-02-19T21:09:53.278837+0000","last_fullsized":"2024-02-19T21:09:53.278837+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:35.505607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:35.505607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":128,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278196+0000","last_change":"2024-02-19T21:08:52.202056+0000","last_active":"2024-02-19T21:09:53.278196+0000","last_peered":"2024-02-19T21:09:53.278196+0000","last_clean":"2024-02-19T21:09:53.278196+0000","last_became_active":"2024-02-19T21:08:41.295736+0000","last_became_peered":"2024-02-19T21:08:41.295736+0000","last_unstale":"2024-02-19T21:09:53.278196+0000","last_undegraded":"2024-02-19T21:09:53.278196+0000","last_fullsized":"2024-02-19T21:09:53.278196+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:52.201972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:52.201972+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":379,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273208+0000","last_change":"2024-02-19T21:09:32.623020+0000","last_active":"2024-02-19T21:09:53.273208+0000","last_peered":"2024-02-19T21:09:53.273208+0000","last_clean":"2024-02-19T21:09:53.273208+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:09:53.273208+0000","last_undegraded":"2024-02-19T21:09:53.273208+0000","last_fullsized":"2024-02-19T21:09:53.273208+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:32.622939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:09:32.622939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":393,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269242+0000","last_change":"2024-02-19T21:09:34.642491+0000","last_active":"2024-02-19T21:09:53.269242+0000","last_peered":"2024-02-19T21:09:53.269242+0000","last_clean":"2024-02-19T21:09:53.269242+0000","last_became_active":"2024-02-19T21:08:04.268847+0000","last_became_peered":"2024-02-19T21:08:04.268847+0000","last_unstale":"2024-02-19T21:09:53.269242+0000","last_undegraded":"2024-02-19T21:09:53.269242+0000","last_fullsized":"2024-02-19T21:09:53.269242+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:34.642432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:34.642432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":117,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270161+0000","last_change":"2024-02-19T21:08:04.280460+0000","last_active":"2024-02-19T21:09:53.270161+0000","last_peered":"2024-02-19T21:09:53.270161+0000","last_clean":"2024-02-19T21:09:53.270161+0000","last_became_active":"2024-02-19T21:08:04.280233+0000","last_became_peered":"2024-02-19T21:08:04.280233+0000","last_unstale":"2024-02-19T21:09:53.270161+0000","last_undegraded":"2024-02-19T21:09:53.270161+0000","last_fullsized":"2024-02-19T21:09:53.270161+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:59.446267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:07:59.446267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":117,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271508+0000","last_change":"2024-02-19T21:08:04.258981+0000","last_active":"2024-02-19T21:09:53.271508+0000","last_peered":"2024-02-19T21:09:53.271508+0000","last_clean":"2024-02-19T21:09:53.271508+0000","last_became_active":"2024-02-19T21:08:04.258568+0000","last_became_peered":"2024-02-19T21:08:04.258568+0000","last_unstale":"2024-02-19T21:09:53.271508+0000","last_undegraded":"2024-02-19T21:09:53.271508+0000","last_fullsized":"2024-02-19T21:09:53.271508+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:29.946015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:29.946015+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":128,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270104+0000","last_change":"2024-02-19T21:09:28.712106+0000","last_active":"2024-02-19T21:09:53.270104+0000","last_peered":"2024-02-19T21:09:53.270104+0000","last_clean":"2024-02-19T21:09:53.270104+0000","last_became_active":"2024-02-19T21:08:04.274921+0000","last_became_peered":"2024-02-19T21:08:04.274921+0000","last_unstale":"2024-02-19T21:09:53.270104+0000","last_undegraded":"2024-02-19T21:09:53.270104+0000","last_fullsized":"2024-02-19T21:09:53.270104+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:09:28.712037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.712037+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":380,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270942+0000","last_change":"2024-02-19T21:09:42.000906+0000","last_active":"2024-02-19T21:09:53.270942+0000","last_peered":"2024-02-19T21:09:53.270942+0000","last_clean":"2024-02-19T21:09:53.270942+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:09:53.270942+0000","last_undegraded":"2024-02-19T21:09:53.270942+0000","last_fullsized":"2024-02-19T21:09:53.270942+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:42.000850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:42.000850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":382,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.277866+0000","last_change":"2024-02-19T21:08:41.261809+0000","last_active":"2024-02-19T21:09:53.277866+0000","last_peered":"2024-02-19T21:09:53.277866+0000","last_clean":"2024-02-19T21:09:53.277866+0000","last_became_active":"2024-02-19T21:08:41.261268+0000","last_became_peered":"2024-02-19T21:08:41.261268+0000","last_unstale":"2024-02-19T21:09:53.277866+0000","last_undegraded":"2024-02-19T21:09:53.277866+0000","last_fullsized":"2024-02-19T21:09:53.277866+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:11.593368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:11.593368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":382,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272771+0000","last_change":"2024-02-19T21:09:26.731948+0000","last_active":"2024-02-19T21:09:53.272771+0000","last_peered":"2024-02-19T21:09:53.272771+0000","last_clean":"2024-02-19T21:09:53.272771+0000","last_became_active":"2024-02-19T21:08:41.274144+0000","last_became_peered":"2024-02-19T21:08:41.274144+0000","last_unstale":"2024-02-19T21:09:53.272771+0000","last_undegraded":"2024-02-19T21:09:53.272771+0000","last_fullsized":"2024-02-19T21:09:53.272771+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:26.731860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:26.731860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":122,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272522+0000","last_change":"2024-02-19T21:08:10.824761+0000","last_active":"2024-02-19T21:09:53.272522+0000","last_peered":"2024-02-19T21:09:53.272522+0000","last_clean":"2024-02-19T21:09:53.272522+0000","last_became_active":"2024-02-19T21:08:04.266161+0000","last_became_peered":"2024-02-19T21:08:04.266161+0000","last_unstale":"2024-02-19T21:09:53.272522+0000","last_undegraded":"2024-02-19T21:09:53.272522+0000","last_fullsized":"2024-02-19T21:09:53.272522+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:10.824694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:10.824694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":134,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270313+0000","last_change":"2024-02-19T21:08:11.828624+0000","last_active":"2024-02-19T21:09:53.270313+0000","last_peered":"2024-02-19T21:09:53.270313+0000","last_clean":"2024-02-19T21:09:53.270313+0000","last_became_active":"2024-02-19T21:08:04.250742+0000","last_became_peered":"2024-02-19T21:08:04.250742+0000","last_unstale":"2024-02-19T21:09:53.270313+0000","last_undegraded":"2024-02-19T21:09:53.270313+0000","last_fullsized":"2024-02-19T21:09:53.270313+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:11.828537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:11.828537+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":377,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278746+0000","last_change":"2024-02-19T21:08:40.270859+0000","last_active":"2024-02-19T21:09:53.278746+0000","last_peered":"2024-02-19T21:09:53.278746+0000","last_clean":"2024-02-19T21:09:53.278746+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:09:53.278746+0000","last_undegraded":"2024-02-19T21:09:53.278746+0000","last_fullsized":"2024-02-19T21:09:53.278746+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:40.270758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:40.270758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":379,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272791+0000","last_change":"2024-02-19T21:09:29.783688+0000","last_active":"2024-02-19T21:09:53.272791+0000","last_peered":"2024-02-19T21:09:53.272791+0000","last_clean":"2024-02-19T21:09:53.272791+0000","last_became_active":"2024-02-19T21:08:41.267236+0000","last_became_peered":"2024-02-19T21:08:41.267236+0000","last_unstale":"2024-02-19T21:09:53.272791+0000","last_undegraded":"2024-02-19T21:09:53.272791+0000","last_fullsized":"2024-02-19T21:09:53.272791+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:29.783609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:29.783609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":117,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270857+0000","last_change":"2024-02-19T21:08:04.254279+0000","last_active":"2024-02-19T21:09:53.270857+0000","last_peered":"2024-02-19T21:09:53.270857+0000","last_clean":"2024-02-19T21:09:53.270857+0000","last_became_active":"2024-02-19T21:08:04.251987+0000","last_became_peered":"2024-02-19T21:08:04.251987+0000","last_unstale":"2024-02-19T21:09:53.270857+0000","last_undegraded":"2024-02-19T21:09:53.270857+0000","last_fullsized":"2024-02-19T21:09:53.270857+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:08.757198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:08.757198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":203,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278793+0000","last_change":"2024-02-19T21:06:54.664879+0000","last_active":"2024-02-19T21:09:53.278793+0000","last_peered":"2024-02-19T21:09:53.278793+0000","last_clean":"2024-02-19T21:09:53.278793+0000","last_became_active":"2024-02-19T21:06:40.226196+0000","last_became_peered":"2024-02-19T21:06:40.226196+0000","last_unstale":"2024-02-19T21:09:53.278793+0000","last_undegraded":"2024-02-19T21:09:53.278793+0000","last_fullsized":"2024-02-19T21:09:53.278793+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.664814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":275,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273988+0000","last_change":"2024-02-19T21:09:30.782123+0000","last_active":"2024-02-19T21:09:53.273988+0000","last_peered":"2024-02-19T21:09:53.273988+0000","last_clean":"2024-02-19T21:09:53.273988+0000","last_became_active":"2024-02-19T21:08:04.267222+0000","last_became_peered":"2024-02-19T21:08:04.267222+0000","last_unstale":"2024-02-19T21:09:53.273988+0000","last_undegraded":"2024-02-19T21:09:53.273988+0000","last_fullsized":"2024-02-19T21:09:53.273988+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:09:30.782002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:30.782002+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":384,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269804+0000","last_change":"2024-02-19T21:08:52.579253+0000","last_active":"2024-02-19T21:09:53.269804+0000","last_peered":"2024-02-19T21:09:53.269804+0000","last_clean":"2024-02-19T21:09:53.269804+0000","last_became_active":"2024-02-19T21:08:41.264843+0000","last_became_peered":"2024-02-19T21:08:41.264843+0000","last_unstale":"2024-02-19T21:09:53.269804+0000","last_undegraded":"2024-02-19T21:09:53.269804+0000","last_fullsized":"2024-02-19T21:09:53.269804+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:52.579178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:52.579178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"344'9","reported_seq":501,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447447+0000","last_change":"2024-02-19T21:08:41.319305+0000","last_active":"2024-02-19T21:09:54.447447+0000","last_peered":"2024-02-19T21:09:54.447447+0000","last_clean":"2024-02-19T21:09:54.447447+0000","last_became_active":"2024-02-19T21:08:41.281854+0000","last_became_peered":"2024-02-19T21:08:41.281854+0000","last_unstale":"2024-02-19T21:09:54.447447+0000","last_undegraded":"2024-02-19T21:09:54.447447+0000","last_fullsized":"2024-02-19T21:09:54.447447+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:08:36.251958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.251958+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":375,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273828+0000","last_change":"2024-02-19T21:09:41.951642+0000","last_active":"2024-02-19T21:09:53.273828+0000","last_peered":"2024-02-19T21:09:53.273828+0000","last_clean":"2024-02-19T21:09:53.273828+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:09:53.273828+0000","last_undegraded":"2024-02-19T21:09:53.273828+0000","last_fullsized":"2024-02-19T21:09:53.273828+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:09:41.951576+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:09:41.951576+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":325,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.281782+0000","last_change":"2024-02-19T21:08:39.107325+0000","last_active":"2024-02-19T21:09:53.281782+0000","last_peered":"2024-02-19T21:09:53.281782+0000","last_clean":"2024-02-19T21:09:53.281782+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:09:53.281782+0000","last_undegraded":"2024-02-19T21:09:53.281782+0000","last_fullsized":"2024-02-19T21:09:53.281782+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:39.107240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.107240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":394,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278879+0000","last_change":"2024-02-19T21:08:03.561292+0000","last_active":"2024-02-19T21:09:53.278879+0000","last_peered":"2024-02-19T21:09:53.278879+0000","last_clean":"2024-02-19T21:09:53.278879+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:09:53.278879+0000","last_undegraded":"2024-02-19T21:09:53.278879+0000","last_fullsized":"2024-02-19T21:09:53.278879+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:03.561223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:03.561223+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":117,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270330+0000","last_change":"2024-02-19T21:08:04.254214+0000","last_active":"2024-02-19T21:09:53.270330+0000","last_peered":"2024-02-19T21:09:53.270330+0000","last_clean":"2024-02-19T21:09:53.270330+0000","last_became_active":"2024-02-19T21:08:04.251241+0000","last_became_peered":"2024-02-19T21:08:04.251241+0000","last_unstale":"2024-02-19T21:09:53.270330+0000","last_undegraded":"2024-02-19T21:09:53.270330+0000","last_fullsized":"2024-02-19T21:09:53.270330+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:01.530618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:01.530618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":206,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.274037+0000","last_change":"2024-02-19T21:08:36.903762+0000","last_active":"2024-02-19T21:09:53.274037+0000","last_peered":"2024-02-19T21:09:53.274037+0000","last_clean":"2024-02-19T21:09:53.274037+0000","last_became_active":"2024-02-19T21:06:40.234026+0000","last_became_peered":"2024-02-19T21:06:40.234026+0000","last_unstale":"2024-02-19T21:09:53.274037+0000","last_undegraded":"2024-02-19T21:09:53.274037+0000","last_fullsized":"2024-02-19T21:09:53.274037+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:36.903674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.903674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":377,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273084+0000","last_change":"2024-02-19T21:08:48.760695+0000","last_active":"2024-02-19T21:09:53.273084+0000","last_peered":"2024-02-19T21:09:53.273084+0000","last_clean":"2024-02-19T21:09:53.273084+0000","last_became_active":"2024-02-19T21:08:41.309795+0000","last_became_peered":"2024-02-19T21:08:41.309795+0000","last_unstale":"2024-02-19T21:09:53.273084+0000","last_undegraded":"2024-02-19T21:09:53.273084+0000","last_fullsized":"2024-02-19T21:09:53.273084+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:48.760625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:08:48.760625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":331,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272547+0000","last_change":"2024-02-19T21:08:41.310765+0000","last_active":"2024-02-19T21:09:53.272547+0000","last_peered":"2024-02-19T21:09:53.272547+0000","last_clean":"2024-02-19T21:09:53.272547+0000","last_became_active":"2024-02-19T21:08:41.310546+0000","last_became_peered":"2024-02-19T21:08:41.310546+0000","last_unstale":"2024-02-19T21:09:53.272547+0000","last_undegraded":"2024-02-19T21:09:53.272547+0000","last_fullsized":"2024-02-19T21:09:53.272547+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:33.686917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:33.686917+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":447,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.282414+0000","last_change":"2024-02-19T21:08:04.337335+0000","last_active":"2024-02-19T21:09:53.282414+0000","last_peered":"2024-02-19T21:09:53.282414+0000","last_clean":"2024-02-19T21:09:53.282414+0000","last_became_active":"2024-02-19T21:08:04.248378+0000","last_became_peered":"2024-02-19T21:08:04.248378+0000","last_unstale":"2024-02-19T21:09:53.282414+0000","last_undegraded":"2024-02-19T21:09:53.282414+0000","last_fullsized":"2024-02-19T21:09:53.282414+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:06:48.310459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.310459+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":386,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278936+0000","last_change":"2024-02-19T21:09:26.417332+0000","last_active":"2024-02-19T21:09:53.278936+0000","last_peered":"2024-02-19T21:09:53.278936+0000","last_clean":"2024-02-19T21:09:53.278936+0000","last_became_active":"2024-02-19T21:08:41.278751+0000","last_became_peered":"2024-02-19T21:08:41.278751+0000","last_unstale":"2024-02-19T21:09:53.278936+0000","last_undegraded":"2024-02-19T21:09:53.278936+0000","last_fullsized":"2024-02-19T21:09:53.278936+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:26.417247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:26.417247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":379,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273327+0000","last_change":"2024-02-19T21:08:41.308944+0000","last_active":"2024-02-19T21:09:53.273327+0000","last_peered":"2024-02-19T21:09:53.273327+0000","last_clean":"2024-02-19T21:09:53.273327+0000","last_became_active":"2024-02-19T21:08:41.302364+0000","last_became_peered":"2024-02-19T21:08:41.302364+0000","last_unstale":"2024-02-19T21:09:53.273327+0000","last_undegraded":"2024-02-19T21:09:53.273327+0000","last_fullsized":"2024-02-19T21:09:53.273327+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:53.629475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:53.629475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":80,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278589+0000","last_change":"2024-02-19T21:08:41.279946+0000","last_active":"2024-02-19T21:09:53.278589+0000","last_peered":"2024-02-19T21:09:53.278589+0000","last_clean":"2024-02-19T21:09:53.278589+0000","last_became_active":"2024-02-19T21:08:41.278358+0000","last_became_peered":"2024-02-19T21:08:41.278358+0000","last_unstale":"2024-02-19T21:09:53.278589+0000","last_undegraded":"2024-02-19T21:09:53.278589+0000","last_fullsized":"2024-02-19T21:09:53.278589+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:36.723492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:07:36.723492+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":203,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.277549+0000","last_change":"2024-02-19T21:09:27.464040+0000","last_active":"2024-02-19T21:09:53.277549+0000","last_peered":"2024-02-19T21:09:53.277549+0000","last_clean":"2024-02-19T21:09:53.277549+0000","last_became_active":"2024-02-19T21:06:40.240859+0000","last_became_peered":"2024-02-19T21:06:40.240859+0000","last_unstale":"2024-02-19T21:09:53.277549+0000","last_undegraded":"2024-02-19T21:09:53.277549+0000","last_fullsized":"2024-02-19T21:09:53.277549+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.463966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.463966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":155,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270701+0000","last_change":"2024-02-19T21:08:49.869657+0000","last_active":"2024-02-19T21:09:53.270701+0000","last_peered":"2024-02-19T21:09:53.270701+0000","last_clean":"2024-02-19T21:09:53.270701+0000","last_became_active":"2024-02-19T21:08:41.295566+0000","last_became_peered":"2024-02-19T21:08:41.295566+0000","last_unstale":"2024-02-19T21:09:53.270701+0000","last_undegraded":"2024-02-19T21:09:53.270701+0000","last_fullsized":"2024-02-19T21:09:53.270701+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:08:49.869588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.869588+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":117,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270932+0000","last_change":"2024-02-19T21:08:04.254443+0000","last_active":"2024-02-19T21:09:53.270932+0000","last_peered":"2024-02-19T21:09:53.270932+0000","last_clean":"2024-02-19T21:09:53.270932+0000","last_became_active":"2024-02-19T21:08:04.253058+0000","last_became_peered":"2024-02-19T21:08:04.253058+0000","last_unstale":"2024-02-19T21:09:53.270932+0000","last_undegraded":"2024-02-19T21:09:53.270932+0000","last_fullsized":"2024-02-19T21:09:53.270932+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:02.693025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:02.693025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":124,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272315+0000","last_change":"2024-02-19T21:09:35.626645+0000","last_active":"2024-02-19T21:09:53.272315+0000","last_peered":"2024-02-19T21:09:53.272315+0000","last_clean":"2024-02-19T21:09:53.272315+0000","last_became_active":"2024-02-19T21:08:41.288781+0000","last_became_peered":"2024-02-19T21:08:41.288781+0000","last_unstale":"2024-02-19T21:09:53.272315+0000","last_undegraded":"2024-02-19T21:09:53.272315+0000","last_fullsized":"2024-02-19T21:09:53.272315+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:35.626548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:35.626548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":117,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271520+0000","last_change":"2024-02-19T21:08:04.278426+0000","last_active":"2024-02-19T21:09:53.271520+0000","last_peered":"2024-02-19T21:09:53.271520+0000","last_clean":"2024-02-19T21:09:53.271520+0000","last_became_active":"2024-02-19T21:08:04.278045+0000","last_became_peered":"2024-02-19T21:08:04.278045+0000","last_unstale":"2024-02-19T21:09:53.271520+0000","last_undegraded":"2024-02-19T21:09:53.271520+0000","last_fullsized":"2024-02-19T21:09:53.271520+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.723156+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":322,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269525+0000","last_change":"2024-02-19T21:07:25.735056+0000","last_active":"2024-02-19T21:09:53.269525+0000","last_peered":"2024-02-19T21:09:53.269525+0000","last_clean":"2024-02-19T21:09:53.269525+0000","last_became_active":"2024-02-19T21:04:39.777442+0000","last_became_peered":"2024-02-19T21:04:39.777442+0000","last_unstale":"2024-02-19T21:09:53.269525+0000","last_undegraded":"2024-02-19T21:09:53.269525+0000","last_fullsized":"2024-02-19T21:09:53.269525+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:25.734988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:25.734988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"194'13","reported_seq":398,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271679+0000","last_change":"2024-02-19T21:08:41.890452+0000","last_active":"2024-02-19T21:09:53.271679+0000","last_peered":"2024-02-19T21:09:53.271679+0000","last_clean":"2024-02-19T21:09:53.271679+0000","last_became_active":"2024-02-19T21:08:41.283514+0000","last_became_peered":"2024-02-19T21:08:41.283514+0000","last_unstale":"2024-02-19T21:09:53.271679+0000","last_undegraded":"2024-02-19T21:09:53.271679+0000","last_fullsized":"2024-02-19T21:09:53.271679+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'13","last_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.772417+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":374,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273426+0000","last_change":"2024-02-19T21:08:51.955830+0000","last_active":"2024-02-19T21:09:53.273426+0000","last_peered":"2024-02-19T21:09:53.273426+0000","last_clean":"2024-02-19T21:09:53.273426+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:09:53.273426+0000","last_undegraded":"2024-02-19T21:09:53.273426+0000","last_fullsized":"2024-02-19T21:09:53.273426+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:51.955738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:51.955738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":371,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270217+0000","last_change":"2024-02-19T21:09:29.852892+0000","last_active":"2024-02-19T21:09:53.270217+0000","last_peered":"2024-02-19T21:09:53.270217+0000","last_clean":"2024-02-19T21:09:53.270217+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:09:53.270217+0000","last_undegraded":"2024-02-19T21:09:53.270217+0000","last_fullsized":"2024-02-19T21:09:53.270217+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:29.852792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:29.852792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":200,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.268610+0000","last_change":"2024-02-19T21:06:40.239902+0000","last_active":"2024-02-19T21:09:53.268610+0000","last_peered":"2024-02-19T21:09:53.268610+0000","last_clean":"2024-02-19T21:09:53.268610+0000","last_became_active":"2024-02-19T21:06:40.239603+0000","last_became_peered":"2024-02-19T21:06:40.239603+0000","last_unstale":"2024-02-19T21:09:53.268610+0000","last_undegraded":"2024-02-19T21:09:53.268610+0000","last_fullsized":"2024-02-19T21:09:53.268610+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:06:25.785366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":331,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.280302+0000","last_change":"2024-02-19T21:08:41.276256+0000","last_active":"2024-02-19T21:09:53.280302+0000","last_peered":"2024-02-19T21:09:53.280302+0000","last_clean":"2024-02-19T21:09:53.280302+0000","last_became_active":"2024-02-19T21:08:41.268425+0000","last_became_peered":"2024-02-19T21:08:41.268425+0000","last_unstale":"2024-02-19T21:09:53.280302+0000","last_undegraded":"2024-02-19T21:09:53.280302+0000","last_fullsized":"2024-02-19T21:09:53.280302+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:58.310306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:58.310306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":385,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270430+0000","last_change":"2024-02-19T21:09:44.596395+0000","last_active":"2024-02-19T21:09:53.270430+0000","last_peered":"2024-02-19T21:09:53.270430+0000","last_clean":"2024-02-19T21:09:53.270430+0000","last_became_active":"2024-02-19T21:08:41.298233+0000","last_became_peered":"2024-02-19T21:08:41.298233+0000","last_unstale":"2024-02-19T21:09:53.270430+0000","last_undegraded":"2024-02-19T21:09:53.270430+0000","last_fullsized":"2024-02-19T21:09:53.270430+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:09:44.596326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:44.596326+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":379,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.273709+0000","last_change":"2024-02-19T21:09:25.733055+0000","last_active":"2024-02-19T21:09:53.273709+0000","last_peered":"2024-02-19T21:09:53.273709+0000","last_clean":"2024-02-19T21:09:53.273709+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:09:53.273709+0000","last_undegraded":"2024-02-19T21:09:53.273709+0000","last_fullsized":"2024-02-19T21:09:53.273709+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:25.732955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:25.732955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":384,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.272880+0000","last_change":"2024-02-19T21:09:36.862378+0000","last_active":"2024-02-19T21:09:53.272880+0000","last_peered":"2024-02-19T21:09:53.272880+0000","last_clean":"2024-02-19T21:09:53.272880+0000","last_became_active":"2024-02-19T21:08:41.304581+0000","last_became_peered":"2024-02-19T21:08:41.304581+0000","last_unstale":"2024-02-19T21:09:53.272880+0000","last_undegraded":"2024-02-19T21:09:53.272880+0000","last_fullsized":"2024-02-19T21:09:53.272880+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:36.862297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:36.862297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":387,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269912+0000","last_change":"2024-02-19T21:08:04.266429+0000","last_active":"2024-02-19T21:09:53.269912+0000","last_peered":"2024-02-19T21:09:53.269912+0000","last_clean":"2024-02-19T21:09:53.269912+0000","last_became_active":"2024-02-19T21:08:04.266100+0000","last_became_peered":"2024-02-19T21:08:04.266100+0000","last_unstale":"2024-02-19T21:09:53.269912+0000","last_undegraded":"2024-02-19T21:09:53.269912+0000","last_fullsized":"2024-02-19T21:09:53.269912+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:48.506964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.506964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":206,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270990+0000","last_change":"2024-02-19T21:06:50.467486+0000","last_active":"2024-02-19T21:09:53.270990+0000","last_peered":"2024-02-19T21:09:53.270990+0000","last_clean":"2024-02-19T21:09:53.270990+0000","last_became_active":"2024-02-19T21:06:40.227713+0000","last_became_peered":"2024-02-19T21:06:40.227713+0000","last_unstale":"2024-02-19T21:09:53.270990+0000","last_undegraded":"2024-02-19T21:09:53.270990+0000","last_fullsized":"2024-02-19T21:09:53.270990+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:50.467411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:50.467411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":118,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.277260+0000","last_change":"2024-02-19T21:08:42.700047+0000","last_active":"2024-02-19T21:09:53.277260+0000","last_peered":"2024-02-19T21:09:53.277260+0000","last_clean":"2024-02-19T21:09:53.277260+0000","last_became_active":"2024-02-19T21:08:41.266486+0000","last_became_peered":"2024-02-19T21:08:41.266486+0000","last_unstale":"2024-02-19T21:09:53.277260+0000","last_undegraded":"2024-02-19T21:09:53.277260+0000","last_fullsized":"2024-02-19T21:09:53.277260+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:08:36.593172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.593172+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":117,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271213+0000","last_change":"2024-02-19T21:08:04.278604+0000","last_active":"2024-02-19T21:09:53.271213+0000","last_peered":"2024-02-19T21:09:53.271213+0000","last_clean":"2024-02-19T21:09:53.271213+0000","last_became_active":"2024-02-19T21:08:04.277576+0000","last_became_peered":"2024-02-19T21:08:04.277576+0000","last_unstale":"2024-02-19T21:09:53.271213+0000","last_undegraded":"2024-02-19T21:09:53.271213+0000","last_fullsized":"2024-02-19T21:09:53.271213+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:53.752543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:53.752543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":370,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269316+0000","last_change":"2024-02-19T21:06:24.702277+0000","last_active":"2024-02-19T21:09:53.269316+0000","last_peered":"2024-02-19T21:09:53.269316+0000","last_clean":"2024-02-19T21:09:53.269316+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:09:53.269316+0000","last_undegraded":"2024-02-19T21:09:53.269316+0000","last_fullsized":"2024-02-19T21:09:53.269316+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:06:24.702197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":81,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.269913+0000","last_change":"2024-02-19T21:08:41.307487+0000","last_active":"2024-02-19T21:09:53.269913+0000","last_peered":"2024-02-19T21:09:53.269913+0000","last_clean":"2024-02-19T21:09:53.269913+0000","last_became_active":"2024-02-19T21:08:41.306708+0000","last_became_peered":"2024-02-19T21:08:41.306708+0000","last_unstale":"2024-02-19T21:09:53.269913+0000","last_undegraded":"2024-02-19T21:09:53.269913+0000","last_fullsized":"2024-02-19T21:09:53.269913+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:04.600973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:04.600973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":332,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.271043+0000","last_change":"2024-02-19T21:08:41.293045+0000","last_active":"2024-02-19T21:09:53.271043+0000","last_peered":"2024-02-19T21:09:53.271043+0000","last_clean":"2024-02-19T21:09:53.271043+0000","last_became_active":"2024-02-19T21:08:41.292651+0000","last_became_peered":"2024-02-19T21:08:41.292651+0000","last_unstale":"2024-02-19T21:09:53.271043+0000","last_undegraded":"2024-02-19T21:09:53.271043+0000","last_fullsized":"2024-02-19T21:09:53.271043+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:58.468840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:58.468840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":133,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270065+0000","last_change":"2024-02-19T21:08:04.341196+0000","last_active":"2024-02-19T21:09:53.270065+0000","last_peered":"2024-02-19T21:09:53.270065+0000","last_clean":"2024-02-19T21:09:53.270065+0000","last_became_active":"2024-02-19T21:08:04.249443+0000","last_became_peered":"2024-02-19T21:08:04.249443+0000","last_unstale":"2024-02-19T21:09:53.270065+0000","last_undegraded":"2024-02-19T21:09:53.270065+0000","last_fullsized":"2024-02-19T21:09:53.270065+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:06:09.763482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.763482+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":217,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.274132+0000","last_change":"2024-02-19T21:08:47.903239+0000","last_active":"2024-02-19T21:09:53.274132+0000","last_peered":"2024-02-19T21:09:53.274132+0000","last_clean":"2024-02-19T21:09:53.274132+0000","last_became_active":"2024-02-19T21:08:41.263206+0000","last_became_peered":"2024-02-19T21:08:41.263206+0000","last_unstale":"2024-02-19T21:09:53.274132+0000","last_undegraded":"2024-02-19T21:09:53.274132+0000","last_fullsized":"2024-02-19T21:09:53.274132+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:47.903127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:47.903127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":80,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.277830+0000","last_change":"2024-02-19T21:08:41.263525+0000","last_active":"2024-02-19T21:09:53.277830+0000","last_peered":"2024-02-19T21:09:53.277830+0000","last_clean":"2024-02-19T21:09:53.277830+0000","last_became_active":"2024-02-19T21:08:41.263160+0000","last_became_peered":"2024-02-19T21:08:41.263160+0000","last_unstale":"2024-02-19T21:09:53.277830+0000","last_undegraded":"2024-02-19T21:09:53.277830+0000","last_fullsized":"2024-02-19T21:09:53.277830+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:39.574835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.574835+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":122,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.270702+0000","last_change":"2024-02-19T21:09:33.559448+0000","last_active":"2024-02-19T21:09:53.270702+0000","last_peered":"2024-02-19T21:09:53.270702+0000","last_clean":"2024-02-19T21:09:53.270702+0000","last_became_active":"2024-02-19T21:08:04.252640+0000","last_became_peered":"2024-02-19T21:08:04.252640+0000","last_unstale":"2024-02-19T21:09:53.270702+0000","last_undegraded":"2024-02-19T21:09:53.270702+0000","last_fullsized":"2024-02-19T21:09:53.270702+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:33.559363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:33.559363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":338,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-02-19T21:09:53.278532+0000","last_change":"2024-02-19T21:08:04.251168+0000","last_active":"2024-02-19T21:09:53.278532+0000","last_peered":"2024-02-19T21:09:53.278532+0000","last_clean":"2024-02-19T21:09:53.278532+0000","last_became_active":"2024-02-19T21:08:04.250937+0000","last_became_peered":"2024-02-19T21:08:04.250937+0000","last_unstale":"2024-02-19T21:09:53.278532+0000","last_undegraded":"2024-02-19T21:09:53.278532+0000","last_fullsized":"2024-02-19T21:09:53.278532+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:27.744835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:27.744835+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":212,"num_pg":32,"stat_sum":{"num_bytes":14,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":7,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":42,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":7},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":4001489,"num_objects":160,"num_object_clones":4,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":107,"num_read_kb":9,"num_write":596,"num_write_kb":4495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":491,"num_bytes_recovered":1160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13959168,"data_stored":12004581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1871,"ondisk_log_size":1871,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":677,"num_read_kb":434,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":4500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":410,"ondisk_log_size":410,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":511,"num_read_kb":506,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266794,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311816,"kb_used_data":10484,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93433336,"statfs":{"total":95995035648,"available":95675736064,"internally_reserved":0,"allocated":10735616,"data_stored":7811392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":1.603,"15min":0.86899999999999999},"min":{"1min":0.26000000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.65200000000000002,"5min":89.456999999999994,"15min":89.456999999999994},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":1.377,"15min":0.78600000000000003},"min":{"1min":0.28499999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.67900000000000005,"5min":70.451999999999998,"15min":70.451999999999998},"last":0.32500000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.0740000000000001,"15min":0.70199999999999996},"min":{"1min":0.36299999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.85099999999999998,"5min":34.273000000000003,"15min":34.273000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":1.7889999999999999,"15min":0.94099999999999995},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.254},"max":{"1min":0.79200000000000004,"5min":89.594999999999999,"15min":89.594999999999999},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":1.837,"15min":0.94399999999999995},"min":{"1min":0.35099999999999998,"5min":0.27600000000000002,"15min":0.24299999999999999},"max":{"1min":0.67100000000000004,"5min":104.413,"15min":104.413},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":1.8009999999999999,"15min":0.94999999999999996},"min":{"1min":0.35599999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70599999999999996,"5min":104.268,"15min":104.268},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":1.7969999999999999,"15min":0.95199999999999996},"min":{"1min":0.38800000000000001,"5min":0.29099999999999998,"15min":0.254},"max":{"1min":1.2949999999999999,"5min":105.3,"15min":105.3},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.79700000000000004,"15min":0.59799999999999998},"min":{"1min":0.34300000000000003,"5min":0.315,"15min":0.315},"max":{"1min":0.879,"5min":13.243,"15min":13.243},"last":0.48299999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.42799999999999999,"15min":0.47599999999999998},"min":{"1min":0.315,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63200000000000001,"5min":0.69299999999999995,"15min":1.373},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.45700000000000002,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.26600000000000001,"15min":0.25900000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":1.0740000000000001},"last":0.36899999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.28599999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.65900000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.48999999999999999,"15min":0.50900000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69399999999999995,"5min":0.77400000000000002,"15min":0.85899999999999999},"last":0.432}]},{"osd":6,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.49199999999999999,"15min":0.52400000000000002},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.70199999999999996,"5min":0.76500000000000001,"15min":1.0049999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52300000000000002,"15min":0.56100000000000005},"min":{"1min":0.34100000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72499999999999998,"5min":0.75900000000000001,"15min":1.262},"last":0.5}]}]},{"osd":6,"up_from":45,"seq":193273528432,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307920,"kb_used_data":6596,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437232,"statfs":{"total":95995035648,"available":95679725568,"internally_reserved":0,"allocated":6754304,"data_stored":3818923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.86299999999999999,"15min":0.61699999999999999},"min":{"1min":0.22700000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.73099999999999998,"5min":17.486999999999998,"15min":17.486999999999998},"last":0.497},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.98299999999999998,"15min":0.65400000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64900000000000002,"5min":17.43,"15min":17.43},"last":0.48499999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.873,"15min":0.61799999999999999},"min":{"1min":0.28699999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.71599999999999997,"5min":14.692,"15min":14.692},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.99299999999999999,"15min":0.66400000000000003},"min":{"1min":0.29099999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.74099999999999999,"5min":14.795,"15min":14.795},"last":0.41599999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.79100000000000004,"15min":0.60799999999999998},"min":{"1min":0.309,"5min":0.255,"15min":0.255},"max":{"1min":0.61899999999999999,"5min":6.931,"15min":6.931},"last":0.436},{"interface":"front","average":{"1min":0.47699999999999998,"5min":1.0249999999999999,"15min":0.66900000000000004},"min":{"1min":0.34100000000000003,"5min":0.318,"15min":0.318},"max":{"1min":0.71899999999999997,"5min":14.391,"15min":14.391},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":1.077,"15min":0.69999999999999996},"min":{"1min":0.378,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73599999999999999,"5min":14.510999999999999,"15min":14.510999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.95299999999999996,"15min":0.65300000000000002},"min":{"1min":0.35299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.64500000000000002,"5min":17.585000000000001,"15min":17.585000000000001},"last":0.46500000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40100000000000002,"15min":0.39200000000000002},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.63800000000000001,"15min":1.363},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.41599999999999998},"min":{"1min":0.29799999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.63500000000000001,"5min":0.69899999999999995,"15min":0.70999999999999996},"last":0.42199999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46100000000000002,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.59699999999999998,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.504,"15min":0.47299999999999998},"min":{"1min":0.35099999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.627,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.38100000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:09:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47699999999999998,"15min":0.503},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":1.1160000000000001},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.46999999999999997,"15min":0.51100000000000001},"min":{"1min":0.32700000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":1.534},"last":0.40300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574983,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311896,"kb_used_data":10560,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93433256,"statfs":{"total":95995035648,"available":95675654144,"internally_reserved":0,"allocated":10813440,"data_stored":7820552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.92900000000000005,"15min":0.54600000000000004},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.51400000000000001,"5min":27.777000000000001,"15min":27.777000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.98799999999999999,"15min":0.57399999999999995},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.217},"max":{"1min":0.70899999999999996,"5min":27.631,"15min":27.631},"last":0.30199999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":1.3109999999999999,"15min":0.71099999999999997},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.66300000000000003,"5min":27.798999999999999,"15min":27.798999999999999},"last":0.312},{"interface":"front","average":{"1min":0.372,"5min":1.3420000000000001,"15min":0.72599999999999998},"min":{"1min":0.24299999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.73299999999999998,"5min":27.606999999999999,"15min":27.606999999999999},"last":0.27000000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.90100000000000002,"15min":0.57899999999999996},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.55400000000000005,"5min":15.574999999999999,"15min":15.574999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.371,"5min":1.119,"15min":0.67400000000000004},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.54200000000000004,"5min":27.82,"15min":27.82},"last":0.28199999999999997}]},{"osd":4,"last update":"Mon Feb 19 21:09:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.86099999999999999,"15min":0.66200000000000003},"min":{"1min":0.36099999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.80400000000000005,"5min":9.6430000000000007,"15min":9.6430000000000007},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.127,"15min":0.755},"min":{"1min":0.40200000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.93000000000000005,"5min":27.707000000000001,"15min":27.707000000000001},"last":0.47799999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":1.075,"15min":0.75},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":0.94199999999999995,"5min":27.529,"15min":27.529},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.86799999999999999,"15min":0.69599999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.84499999999999997,"5min":9.6799999999999997,"15min":9.6799999999999997},"last":0.42899999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.84999999999999998,"15min":0.67500000000000004},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.67100000000000004,"5min":9.6989999999999998,"15min":9.6989999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":1.115,"15min":0.79500000000000004},"min":{"1min":0.41799999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76600000000000001,"5min":27.733000000000001,"15min":27.733000000000001},"last":0.41899999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.139,"15min":0.77600000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.92000000000000004,"5min":27.576000000000001,"15min":27.576000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":1.2170000000000001,"15min":0.81200000000000006},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":2.0169999999999999,"5min":27.756,"15min":27.756},"last":0.439}]}]},{"osd":0,"up_from":8,"seq":34359738509,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307884,"kb_used_data":6556,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437268,"statfs":{"total":95995035648,"available":95679762432,"internally_reserved":0,"allocated":6713344,"data_stored":3811023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":1.119,"15min":0.59899999999999998},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.16200000000000001},"max":{"1min":0.53800000000000003,"5min":20.966999999999999,"15min":20.966999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.34799999999999998,"5min":1.179,"15min":0.60699999999999998},"min":{"1min":0.216,"5min":0.20399999999999999,"15min":0.188},"max":{"1min":0.68200000000000005,"5min":20.366,"15min":20.366},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:09:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":1.369,"15min":0.71599999999999997},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51000000000000001,"5min":27.916,"15min":27.916},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.39300000000000002,"5min":1.081,"15min":0.63200000000000001},"min":{"1min":0.23999999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.622,"5min":22.004000000000001,"15min":22.004000000000001},"last":0.35499999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":1.347,"15min":0.754},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73199999999999998,"5min":25.669,"15min":25.669},"last":0.372},{"interface":"front","average":{"1min":0.38700000000000001,"5min":1.161,"15min":0.69899999999999995},"min":{"1min":0.28899999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.59999999999999998,"5min":25.006,"15min":25.006},"last":0.314}]},{"osd":4,"last update":"Mon Feb 19 21:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.0960000000000001,"15min":0.71499999999999997},"min":{"1min":0.34399999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.86499999999999999,"5min":20.411999999999999,"15min":20.411999999999999},"last":0.5},{"interface":"front","average":{"1min":0.55900000000000005,"5min":1.153,"15min":0.748},"min":{"1min":0.376,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.1160000000000001,"5min":19.986999999999998,"15min":19.986999999999998},"last":0.52200000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":1.2430000000000001,"15min":0.83299999999999996},"min":{"1min":0.40100000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.1890000000000001,"5min":20.021999999999998,"15min":20.021999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.173,"15min":0.91000000000000003},"min":{"1min":0.317,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":6.2859999999999996,"5min":20.908999999999999,"15min":20.908999999999999},"last":0.55200000000000005}]},{"osd":6,"last update":"Mon Feb 19 21:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":1.0900000000000001,"15min":0.76400000000000001},"min":{"1min":0.40100000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.238,"5min":20.445,"15min":20.445},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":1.1679999999999999,"15min":0.81899999999999995},"min":{"1min":0.38500000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.3530000000000002,"5min":20.047999999999998,"15min":20.047999999999998},"last":0.54700000000000004}]},{"osd":7,"last update":"Mon Feb 19 21:09:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.135,"15min":0.89300000000000002},"min":{"1min":0.41099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.95499999999999996,"5min":20.940999999999999,"15min":20.940999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":1.2170000000000001,"15min":0.82499999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.3420000000000001,"5min":20.471,"15min":20.471},"last":0.47699999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411458,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307712,"kb_used_data":10480,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93437440,"statfs":{"total":95995035648,"available":95679938560,"internally_reserved":0,"allocated":10731520,"data_stored":7810725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:27 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.85499999999999998,"15min":0.53900000000000003},"min":{"1min":0.222,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.54100000000000004,"5min":26.690999999999999,"15min":26.690999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.32200000000000001,"5min":1.0069999999999999,"15min":0.59099999999999997},"min":{"1min":0.247,"5min":0.23200000000000001,"15min":0.214},"max":{"1min":0.48999999999999999,"5min":24.315000000000001,"15min":24.315000000000001},"last":0.47599999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:09:27 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":1.3480000000000001,"15min":0.70699999999999996},"min":{"1min":0.252,"5min":0.22,"15min":0.20100000000000001},"max":{"1min":0.52300000000000002,"5min":28.882000000000001,"15min":28.882000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":1.2,"15min":0.68400000000000005},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":0.50600000000000001,"5min":28.382999999999999,"15min":28.382999999999999},"last":0.49299999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:09:54 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":1.276,"15min":0.72599999999999998},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56200000000000006,"5min":28.742000000000001,"15min":28.742000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.96899999999999997,"15min":0.64700000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.25700000000000001},"max":{"1min":0.57299999999999995,"5min":26.719999999999999,"15min":26.719999999999999},"last":0.52700000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.89300000000000002,"15min":0.70999999999999996},"min":{"1min":0.34799999999999998,"5min":0.309,"15min":0.30599999999999999},"max":{"1min":3.4039999999999999,"5min":24.356000000000002,"15min":24.356000000000002},"last":0.63},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.96999999999999997,"15min":0.74299999999999999},"min":{"1min":0.39000000000000001,"5min":0.32800000000000001,"15min":0.32400000000000001},"max":{"1min":3.3919999999999999,"5min":20.707000000000001,"15min":20.707000000000001},"last":0.64600000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:09:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.83999999999999997,"15min":0.70399999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.748,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":1.081,"15min":0.80700000000000005},"min":{"1min":0.41299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85099999999999998,"5min":26.606000000000002,"15min":26.606000000000002},"last":0.68799999999999994}]},{"osd":6,"last update":"Mon Feb 19 21:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":1.1870000000000001,"15min":0.81499999999999995},"min":{"1min":0.46200000000000002,"5min":0.376,"15min":0.376},"max":{"1min":1.4510000000000001,"5min":24.381,"15min":24.381},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.996,"15min":0.75},"min":{"1min":0.42199999999999999,"5min":0.42099999999999999,"15min":0.39100000000000001},"max":{"1min":0.93600000000000005,"5min":28.706,"15min":28.706},"last":0.69699999999999995}]},{"osd":7,"last update":"Mon Feb 19 21:09:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.099,"15min":0.77800000000000002},"min":{"1min":0.46000000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76200000000000001,"5min":26.658000000000001,"15min":26.658000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.93400000000000005,"15min":0.73699999999999999},"min":{"1min":0.439,"5min":0.39100000000000001,"15min":0.379},"max":{"1min":0.76700000000000002,"5min":24.402000000000001,"15min":24.402000000000001},"last":0.66000000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182526,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320148,"kb_used_data":6532,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93425004,"statfs":{"total":95995035648,"available":95667204096,"internally_reserved":0,"allocated":6688768,"data_stored":3819637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":1.454,"15min":0.71899999999999997},"min":{"1min":0.23000000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.64200000000000002,"5min":77.662000000000006,"15min":77.662000000000006},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.74199999999999999,"15min":0.50900000000000001},"min":{"1min":0.22700000000000001,"5min":0.22600000000000001,"15min":0.19700000000000001},"max":{"1min":0.60999999999999999,"5min":17.456,"15min":17.456},"last":0.28699999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.84499999999999997,"15min":0.54000000000000004},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.22800000000000001},"max":{"1min":6.9299999999999997,"5min":17.327999999999999,"15min":17.327999999999999},"last":0.247},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.77000000000000002,"15min":0.51700000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":7.3620000000000001,"5min":15.004,"15min":15.004},"last":0.314}]},{"osd":2,"last update":"Mon Feb 19 21:08:54 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":1.198,"15min":0.67600000000000005},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.54200000000000004,"5min":56.890999999999998,"15min":56.890999999999998},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.376,"5min":1.1619999999999999,"15min":0.66000000000000003},"min":{"1min":0.24399999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59199999999999997,"5min":44.600999999999999,"15min":44.600999999999999},"last":0.32600000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:09:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.82099999999999995,"15min":0.63800000000000001},"min":{"1min":0.33100000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.74399999999999999,"5min":12.847,"15min":12.847},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.026,"15min":0.71599999999999997},"min":{"1min":0.34000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.83899999999999997,"5min":26.393999999999998,"15min":26.393999999999998},"last":0.63700000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:09:34 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92600000000000005,"15min":0.69299999999999995},"min":{"1min":0.34699999999999998,"5min":0.313,"15min":0.313},"max":{"1min":9.2360000000000007,"5min":17.401,"15min":17.401},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.85599999999999998,"15min":0.66100000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.9690000000000001,"5min":12.882,"15min":12.882},"last":0.54000000000000004}]},{"osd":6,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.84999999999999998,"15min":0.65900000000000003},"min":{"1min":0.39000000000000001,"5min":0.35299999999999998,"15min":0.34300000000000003},"max":{"1min":1.9790000000000001,"5min":17.367000000000001,"15min":17.367000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.84599999999999997,"15min":0.67500000000000004},"min":{"1min":0.35699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.88,"5min":12.904,"15min":12.904},"last":0.64500000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:09:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.83799999999999997,"15min":0.65600000000000003},"min":{"1min":0.38600000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":17.428999999999998,"15min":17.428999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.626,"5min":1.2509999999999999,"15min":0.80100000000000005},"min":{"1min":0.41899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85899999999999999,"5min":44.378,"15min":44.378},"last":0.61699999999999999}]}]},{"osd":4,"up_from":0,"seq":133143986298,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":37,"seq":158913790069,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312068,"kb_used_data":6640,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433084,"statfs":{"total":95995035648,"available":95675478016,"internally_reserved":0,"allocated":6799360,"data_stored":3809636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":4.6790000000000003,"15min":1.8919999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.69999999999999996,"5min":276.47899999999998,"15min":276.47899999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.6759999999999999,"15min":0.91700000000000004},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.95299999999999996,"5min":52.887,"15min":52.887},"last":0.47599999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":2.8250000000000002,"15min":1.282},"min":{"1min":0.254,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.599,"5min":120.011,"15min":120.011},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":2.024,"15min":1.0309999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.97699999999999998,"5min":118.105,"15min":118.105},"last":0.503}]},{"osd":2,"last update":"Mon Feb 19 21:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":1.9690000000000001,"15min":1.006},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66600000000000004,"5min":82.242000000000004,"15min":82.242000000000004},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":3.4910000000000001,"15min":1.514},"min":{"1min":0.309,"5min":0.309,"15min":0.30199999999999999},"max":{"1min":4.5019999999999998,"5min":232.84999999999999,"15min":232.84999999999999},"last":0.51100000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":2.1680000000000001,"15min":1.075},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.4640000000000004,"5min":109.70099999999999,"15min":109.70099999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.6599999999999999,"15min":0.91600000000000004},"min":{"1min":0.34799999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72199999999999998,"5min":65.519000000000005,"15min":65.519000000000005},"last":0.40600000000000003}]},{"osd":4,"last update":"Mon Feb 19 21:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.443,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.0129999999999999,"5min":1.861,"15min":1.861},"last":0.374},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45300000000000001,"15min":0.42199999999999999},"min":{"1min":0.308,"5min":0.248,"15min":0.248},"max":{"1min":0.72499999999999998,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":0.438}]},{"osd":6,"last update":"Mon Feb 19 21:08:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46300000000000002,"15min":0.439},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.746,"5min":1.823,"15min":1.823},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47299999999999998,"15min":0.45200000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.66300000000000003,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.39600000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:09:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.36199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.052,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.501,"15min":0.50800000000000001},"min":{"1min":0.38400000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.871,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.49399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4321280,"data_stored":4000784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4263936,"data_stored":4000607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":96,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4268032,"data_stored":4000619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":6,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:09:59.262 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-19T21:09:59.262 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:09:59.262 INFO:tasks.thrashosds.thrasher:fixing pg num pool .rgw.root 2024-02-19T21:09:59.262 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:09:59.344 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:09:59.370 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:59.494 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:09:59.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:59.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:09:59.598 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:09:59.627 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:59.653 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:09:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:59.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:09:59.742 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-19T21:09:59.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.753+0000 7fc83cf4e700 1 -- 172.21.15.163:0/3275899939 >> v1:172.21.15.163:6789/0 conn(0x7fc838106550 legacy=0x7fc838108940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:59.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.753+0000 7fc83cf4e700 1 -- 172.21.15.163:0/3275899939 shutdown_connections 2024-02-19T21:09:59.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.753+0000 7fc83cf4e700 1 -- 172.21.15.163:0/3275899939 >> 172.21.15.163:0/3275899939 conn(0x7fc8380f2470 msgr2=0x7fc8380f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:09:59.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.753+0000 7fc83cf4e700 1 -- 172.21.15.163:0/3275899939 shutdown_connections 2024-02-19T21:09:59.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.753+0000 7fc83cf4e700 1 -- 172.21.15.163:0/3275899939 wait complete. 2024-02-19T21:09:59.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc83cf4e700 1 librados: starting msgr at 2024-02-19T21:09:59.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc83cf4e700 1 librados: starting objecter 2024-02-19T21:09:59.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc83cf4e700 1 Processor -- start 2024-02-19T21:09:59.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc83cf4e700 1 -- start start 2024-02-19T21:09:59.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc83cf4e700 1 librados: setting wanted keys 2024-02-19T21:09:59.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc83cf4e700 1 librados: calling monclient init 2024-02-19T21:09:59.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc83cf4e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc838119490 con 0x7fc8380f72a0 2024-02-19T21:09:59.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc83cf4e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc83811a670 con 0x7fc838106550 2024-02-19T21:09:59.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc83cf4e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc83811b870 con 0x7fc8380f67b0 2024-02-19T21:09:59.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc835d9b700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fc8380f72a0 0x7fc838102fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:40694/0 (socket says 172.21.15.163:40694) 2024-02-19T21:09:59.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc835d9b700 1 -- 172.21.15.163:0/736845126 learned_addr learned my addr 172.21.15.163:0/736845126 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:09:59.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc83659c700 1 --1- 172.21.15.163:0/736845126 >> v1:172.21.15.163:6790/0 conn(0x7fc8380f67b0 0x7fc838102890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:37018/0 (socket says 172.21.15.163:37018) 2024-02-19T21:09:59.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc8277fe700 10 client.?.objecter ms_handle_connect 0x7fc8380f67b0 2024-02-19T21:09:59.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc8277fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:59.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc8277fe700 10 client.?.objecter ms_handle_connect 0x7fc8380f72a0 2024-02-19T21:09:59.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc8277fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:59.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc8277fe700 10 client.?.objecter ms_handle_connect 0x7fc838106550 2024-02-19T21:09:59.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.754+0000 7fc8277fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:09:59.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933857530 0 0) 0x7fc838119490 con 0x7fc8380f72a0 2024-02-19T21:09:59.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8200033f0 con 0x7fc8380f72a0 2024-02-19T21:09:59.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384914413 0 0) 0x7fc83811b870 con 0x7fc8380f67b0 2024-02-19T21:09:59.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc838119490 con 0x7fc8380f67b0 2024-02-19T21:09:59.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339091119 0 0) 0x7fc83811a670 con 0x7fc838106550 2024-02-19T21:09:59.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc83811b870 con 0x7fc838106550 2024-02-19T21:09:59.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2886864729 0 0) 0x7fc838119490 con 0x7fc8380f67b0 2024-02-19T21:09:59.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc83811a670 con 0x7fc8380f67b0 2024-02-19T21:09:59.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc8280018c0 con 0x7fc8380f67b0 2024-02-19T21:09:59.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1794843299 0 0) 0x7fc8200033f0 con 0x7fc8380f72a0 2024-02-19T21:09:59.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc838119490 con 0x7fc8380f72a0 2024-02-19T21:09:59.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 656618239 0 0) 0x7fc83811b870 con 0x7fc838106550 2024-02-19T21:09:59.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8200033f0 con 0x7fc838106550 2024-02-19T21:09:59.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc82c0019d0 con 0x7fc8380f72a0 2024-02-19T21:09:59.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc830001360 con 0x7fc838106550 2024-02-19T21:09:59.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3107771640 0 0) 0x7fc83811a670 con 0x7fc8380f67b0 2024-02-19T21:09:59.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 >> v1:172.21.15.193:6789/0 conn(0x7fc838106550 legacy=0x7fc838117b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:59.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 >> v1:172.21.15.163:6789/0 conn(0x7fc8380f72a0 legacy=0x7fc838102fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:09:59.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83811ca70 con 0x7fc8380f67b0 2024-02-19T21:09:59.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc83cf4e700 1 -- 172.21.15.163:0/736845126 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc838119680 con 0x7fc8380f67b0 2024-02-19T21:09:59.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc83cf4e700 10 client.49811.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:09:59.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc83cf4e700 1 -- 172.21.15.163:0/736845126 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc838119af0 con 0x7fc8380f67b0 2024-02-19T21:09:59.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc83cf4e700 1 librados: init done 2024-02-19T21:09:59.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fc828002090 con 0x7fc8380f67b0 2024-02-19T21:09:59.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.755+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc828005ae0 con 0x7fc8380f67b0 2024-02-19T21:09:59.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.756+0000 7fc83cf4e700 1 -- 172.21.15.163:0/736845126 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc818004f20 con 0x7fc8380f67b0 2024-02-19T21:09:59.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.757+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fc828001bf0 con 0x7fc8380f67b0 2024-02-19T21:09:59.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.757+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(412..412 src has 43..412) v4 ==== 7946+0+0 (unknown 1078123711 0 0) 0x7fc828078bd0 con 0x7fc8380f67b0 2024-02-19T21:09:59.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.757+0000 7fc8277fe700 10 client.49811.objecter ms_dispatch 0x7fc838001550 osd_map(412..412 src has 43..412) v4 2024-02-19T21:09:59.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.757+0000 7fc8277fe700 3 client.49811.objecter handle_osd_map got epochs [412,412] > 0 2024-02-19T21:09:59.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.757+0000 7fc8277fe700 3 client.49811.objecter handle_osd_map decoding full epoch 412 2024-02-19T21:09:59.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.757+0000 7fc8277fe700 20 client.49811.objecter dump_active .. 0 homeless 2024-02-19T21:09:59.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.758+0000 7fc8277fe700 10 client.49811.objecter ms_handle_connect 0x7fc820063d70 2024-02-19T21:09:59.757 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:09:59.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.763+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc82804c090 con 0x7fc8380f67b0 2024-02-19T21:09:59.811 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:59.830 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:09:59.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:09:59.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:09:59.934 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:09:59.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:09:59.944+0000 7fc83cf4e700 1 -- 172.21.15.163:0/736845126 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fc8180051d0 con 0x7fc8380f67b0 2024-02-19T21:09:59.960 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:09:59.998 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:00.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:00.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:00.013 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:10:00.102 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:10:00.132 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:00.280 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:10:00.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:00.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:10:00.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:00.298+0000 7fc8277fe700 1 -- 172.21.15.163:0/736845126 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v413) v1 ==== 91+0+0 (unknown 2504721641 0 0) 0x7fc82804a4a0 con 0x7fc8380f67b0 2024-02-19T21:10:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:00.300+0000 7fc83cf4e700 10 librados: watch_flush enter 2024-02-19T21:10:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:00.300+0000 7fc83cf4e700 10 librados: watch_flush exit 2024-02-19T21:10:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:00.301+0000 7fc83cf4e700 20 client.49811.objecter shutdown clearing up homeless session... 2024-02-19T21:10:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:00.301+0000 7fc83cf4e700 10 client.49811.objecter successfully canceled tick 2024-02-19T21:10:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:00.301+0000 7fc83cf4e700 1 -- 172.21.15.163:0/736845126 >> v1:172.21.15.163:6800/1552514681 conn(0x7fc820063d70 legacy=0x7fc820066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:00.301+0000 7fc83cf4e700 1 -- 172.21.15.163:0/736845126 >> v1:172.21.15.163:6790/0 conn(0x7fc8380f67b0 legacy=0x7fc838102890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:00.301+0000 7fc83cf4e700 1 -- 172.21.15.163:0/736845126 shutdown_connections 2024-02-19T21:10:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:00.301+0000 7fc83cf4e700 1 -- 172.21.15.163:0/736845126 >> 172.21.15.163:0/736845126 conn(0x7fc8380f2470 msgr2=0x7fc838109930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:00.301+0000 7fc83cf4e700 1 -- 172.21.15.163:0/736845126 shutdown_connections 2024-02-19T21:10:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:00.301+0000 7fc83cf4e700 1 -- 172.21.15.163:0/736845126 wait complete. 2024-02-19T21:10:00.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:00.301+0000 7fc83cf4e700 1 librados: shutdown 2024-02-19T21:10:00.299 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:10:00.379 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:59.208485+0000 mgr.y (mgr.14550) 369 : cluster [DBG] pgmap v592: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-02-19T21:10:00.379 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:59.262665+0000 mon.a (mon.0) 2505 : cluster [DBG] osdmap e412: 8 total, 8 up, 7 in 2024-02-19T21:10:00.379 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:59.922820+0000 osd.2 (osd.2) 150 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:10:00.379 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:09:59.922970+0000 osd.2 (osd.2) 151 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:10:00.379 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:59.945011+0000 mon.c (mon.2) 388 : audit [INF] from='client.? v1:172.21.15.163:0/736845126' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:10:00.379 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:09:59.945312+0000 mon.a (mon.0) 2506 : audit [INF] from='client.49811 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:10:00.379 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:00.000111+0000 mon.a (mon.0) 2507 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled 2024-02-19T21:10:00.379 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:00.000144+0000 mon.a (mon.0) 2508 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-19T21:10:00.380 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:00.000159+0000 mon.a (mon.0) 2509 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-19T21:10:00.380 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:00.000181+0000 mon.a (mon.0) 2510 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-19T21:10:00.380 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:00.000194+0000 mon.a (mon.0) 2511 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-19T21:10:00.385 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:00.412 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:00.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:59.208485+0000 mgr.y (mgr.14550) 369 : cluster [DBG] pgmap v592: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-02-19T21:10:00.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:59.262665+0000 mon.a (mon.0) 2505 : cluster [DBG] osdmap e412: 8 total, 8 up, 7 in 2024-02-19T21:10:00.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:59.922820+0000 osd.2 (osd.2) 150 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:10:00.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:09:59.922970+0000 osd.2 (osd.2) 151 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:10:00.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:59.945011+0000 mon.c (mon.2) 388 : audit [INF] from='client.? v1:172.21.15.163:0/736845126' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:10:00.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:09:59.945312+0000 mon.a (mon.0) 2506 : audit [INF] from='client.49811 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:10:00.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:00.000111+0000 mon.a (mon.0) 2507 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled 2024-02-19T21:10:00.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:00.000144+0000 mon.a (mon.0) 2508 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-19T21:10:00.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:00.000159+0000 mon.a (mon.0) 2509 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-19T21:10:00.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:00.000181+0000 mon.a (mon.0) 2510 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-19T21:10:00.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:00.000194+0000 mon.a (mon.0) 2511 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-19T21:10:00.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:59.208485+0000 mgr.y (mgr.14550) 369 : cluster [DBG] pgmap v592: 228 pgs: 1 active+clean+snaptrim, 227 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-02-19T21:10:00.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:59.262665+0000 mon.a (mon.0) 2505 : cluster [DBG] osdmap e412: 8 total, 8 up, 7 in 2024-02-19T21:10:00.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:59.922820+0000 osd.2 (osd.2) 150 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:10:00.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:09:59.922970+0000 osd.2 (osd.2) 151 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:10:00.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:59.945011+0000 mon.c (mon.2) 388 : audit [INF] from='client.? v1:172.21.15.163:0/736845126' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:10:00.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:09:59.945312+0000 mon.a (mon.0) 2506 : audit [INF] from='client.49811 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:10:00.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:00.000111+0000 mon.a (mon.0) 2507 : cluster [WRN] Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled 2024-02-19T21:10:00.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:00.000144+0000 mon.a (mon.0) 2508 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-19T21:10:00.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:00.000159+0000 mon.a (mon.0) 2509 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-19T21:10:00.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:00.000181+0000 mon.a (mon.0) 2510 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-19T21:10:00.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:00.000194+0000 mon.a (mon.0) 2511 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-19T21:10:00.657 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:00.818 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:10:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:10:01.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:00.287608+0000 mon.a (mon.0) 2512 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:10:01.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:00.296393+0000 mon.a (mon.0) 2513 : audit [INF] from='client.49811 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:10:01.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:00.296462+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e413: 8 total, 8 up, 7 in 2024-02-19T21:10:01.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:00.381358+0000 mgr.y (mgr.14550) 370 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:01.420 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:00.287608+0000 mon.a (mon.0) 2512 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:10:01.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:00.296393+0000 mon.a (mon.0) 2513 : audit [INF] from='client.49811 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:10:01.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:00.296462+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e413: 8 total, 8 up, 7 in 2024-02-19T21:10:01.421 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:00.381358+0000 mgr.y (mgr.14550) 370 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:01.462 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:01.464 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:00.287608+0000 mon.a (mon.0) 2512 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:10:01.464 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:00.296393+0000 mon.a (mon.0) 2513 : audit [INF] from='client.49811 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:10:01.464 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:00.296462+0000 mon.a (mon.0) 2514 : cluster [DBG] osdmap e413: 8 total, 8 up, 7 in 2024-02-19T21:10:01.464 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:00.381358+0000 mgr.y (mgr.14550) 370 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:01.468 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:01.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:01.569 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:01.600 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:02.450 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:02.452 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:10:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:10:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-02-19T21:10:02.452 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-02-19T21:10:02.452 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:10:02.453 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:00.546353+0000 osd.3 (osd.3) 173 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:10:02.453 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:00.547224+0000 osd.3 (osd.3) 174 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:10:02.453 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:01.208920+0000 mgr.y (mgr.14550) 371 : cluster [DBG] pgmap v595: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.4 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-02-19T21:10:02.453 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:01.544352+0000 osd.1 (osd.1) 183 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:10:02.453 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:01.545893+0000 osd.1 (osd.1) 184 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:10:02.453 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:01.942210+0000 mon.c (mon.2) 389 : audit [DBG] from='client.? v1:172.21.15.163:0/2192545964' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:02.455 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:02.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:02.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:00.546353+0000 osd.3 (osd.3) 173 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:10:02.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:00.547224+0000 osd.3 (osd.3) 174 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:10:02.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:01.208920+0000 mgr.y (mgr.14550) 371 : cluster [DBG] pgmap v595: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.4 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-02-19T21:10:02.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:01.544352+0000 osd.1 (osd.1) 183 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:10:02.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:01.545893+0000 osd.1 (osd.1) 184 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:10:02.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:01.942210+0000 mon.c (mon.2) 389 : audit [DBG] from='client.? v1:172.21.15.163:0/2192545964' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:02.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:00.546353+0000 osd.3 (osd.3) 173 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:10:02.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:00.547224+0000 osd.3 (osd.3) 174 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:10:02.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:01.208920+0000 mgr.y (mgr.14550) 371 : cluster [DBG] pgmap v595: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.4 KiB/s rd, 1.6 KiB/s wr, 5 op/s 2024-02-19T21:10:02.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:01.544352+0000 osd.1 (osd.1) 183 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:10:02.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:01.545893+0000 osd.1 (osd.1) 184 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:10:02.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:01.942210+0000 mon.c (mon.2) 389 : audit [DBG] from='client.? v1:172.21.15.163:0/2192545964' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:02.556 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:10:02.588 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:03.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.937+0000 7f1ae362b700 1 -- 172.21.15.163:0/2873221467 >> v1:172.21.15.163:6790/0 conn(0x7f1adc100780 legacy=0x7f1adc102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.937+0000 7f1ae362b700 1 -- 172.21.15.163:0/2873221467 shutdown_connections 2024-02-19T21:10:03.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.937+0000 7f1ae362b700 1 -- 172.21.15.163:0/2873221467 >> 172.21.15.163:0/2873221467 conn(0x7f1adc0f35a0 msgr2=0x7f1adc0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:03.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.937+0000 7f1ae362b700 1 -- 172.21.15.163:0/2873221467 shutdown_connections 2024-02-19T21:10:03.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.937+0000 7f1ae362b700 1 -- 172.21.15.163:0/2873221467 wait complete. 2024-02-19T21:10:03.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.938+0000 7f1ae362b700 1 librados: starting msgr at 2024-02-19T21:10:03.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.938+0000 7f1ae362b700 1 librados: starting objecter 2024-02-19T21:10:03.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.938+0000 7f1ae362b700 1 Processor -- start 2024-02-19T21:10:03.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.938+0000 7f1ae362b700 1 -- start start 2024-02-19T21:10:03.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.938+0000 7f1ae362b700 1 librados: setting wanted keys 2024-02-19T21:10:03.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.938+0000 7f1ae362b700 1 librados: calling monclient init 2024-02-19T21:10:03.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.938+0000 7f1ae362b700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1adc117440 con 0x7f1adc100780 2024-02-19T21:10:03.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.938+0000 7f1ae362b700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1adc118620 con 0x7f1adc1042c0 2024-02-19T21:10:03.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.938+0000 7f1ae362b700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1adc119820 con 0x7f1adc0fd980 2024-02-19T21:10:03.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.938+0000 7f1ae13c7700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f1adc0fd980 0x7f1adc0f9ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:37028/0 (socket says 172.21.15.163:37028) 2024-02-19T21:10:03.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.938+0000 7f1ae13c7700 1 -- 172.21.15.163:0/2192545964 learned_addr learned my addr 172.21.15.163:0/2192545964 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:03.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.938+0000 7f1ad27fc700 10 client.?.objecter ms_handle_connect 0x7f1adc0fd980 2024-02-19T21:10:03.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.938+0000 7f1ad27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:03.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.938+0000 7f1ad27fc700 10 client.?.objecter ms_handle_connect 0x7f1adc100780 2024-02-19T21:10:03.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.938+0000 7f1ad27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:03.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 10 client.?.objecter ms_handle_connect 0x7f1adc1042c0 2024-02-19T21:10:03.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:03.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549738927 0 0) 0x7f1adc119820 con 0x7f1adc0fd980 2024-02-19T21:10:03.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac80033f0 con 0x7f1adc0fd980 2024-02-19T21:10:03.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359155525 0 0) 0x7f1adc117440 con 0x7f1adc100780 2024-02-19T21:10:03.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1adc119820 con 0x7f1adc100780 2024-02-19T21:10:03.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109437948 0 0) 0x7f1adc118620 con 0x7f1adc1042c0 2024-02-19T21:10:03.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1adc117440 con 0x7f1adc1042c0 2024-02-19T21:10:03.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3422849594 0 0) 0x7f1ac80033f0 con 0x7f1adc0fd980 2024-02-19T21:10:03.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1adc118620 con 0x7f1adc0fd980 2024-02-19T21:10:03.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1acc0029d0 con 0x7f1adc0fd980 2024-02-19T21:10:03.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1496310990 0 0) 0x7f1adc119820 con 0x7f1adc100780 2024-02-19T21:10:03.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ac80033f0 con 0x7f1adc100780 2024-02-19T21:10:03.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2479176100 0 0) 0x7f1adc117440 con 0x7f1adc1042c0 2024-02-19T21:10:03.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1adc119820 con 0x7f1adc1042c0 2024-02-19T21:10:03.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1ad80010b0 con 0x7f1adc100780 2024-02-19T21:10:03.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1ad4001b80 con 0x7f1adc1042c0 2024-02-19T21:10:03.067 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:10:03.067 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:10:03.067 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:10:03.067 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:10:03.067 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:10:03.067 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:10:03.067 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:10:03.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 937359656 0 0) 0x7f1adc118620 con 0x7f1adc0fd980 2024-02-19T21:10:03.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 >> v1:172.21.15.193:6789/0 conn(0x7f1adc1042c0 legacy=0x7f1adc115b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 >> v1:172.21.15.163:6789/0 conn(0x7f1adc100780 legacy=0x7f1adc0fa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1adc11aa20 con 0x7f1adc0fd980 2024-02-19T21:10:03.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1adc117630 con 0x7f1adc0fd980 2024-02-19T21:10:03.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ae362b700 10 client.49817.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:03.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1adc117b20 con 0x7f1adc0fd980 2024-02-19T21:10:03.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ae362b700 1 librados: init done 2024-02-19T21:10:03.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f1acc002cd0 con 0x7f1adc0fd980 2024-02-19T21:10:03.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.939+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1acc005c00 con 0x7f1adc0fd980 2024-02-19T21:10:03.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.940+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1ac0004f00 con 0x7f1adc0fd980 2024-02-19T21:10:03.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.940+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f1acc005de0 con 0x7f1adc0fd980 2024-02-19T21:10:03.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(413..413 src has 43..413) v4 ==== 7946+0+0 (unknown 882006242 0 0) 0x7f1acc078d80 con 0x7f1adc0fd980 2024-02-19T21:10:03.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 osd_map(413..413 src has 43..413) v4 2024-02-19T21:10:03.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ad27fc700 3 client.49817.objecter handle_osd_map got epochs [413,413] > 0 2024-02-19T21:10:03.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ad27fc700 3 client.49817.objecter handle_osd_map decoding full epoch 413 2024-02-19T21:10:03.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ad27fc700 20 client.49817.objecter dump_active .. 0 homeless 2024-02-19T21:10:03.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v413) v1 ==== 54+0+16 (unknown 211542484 0 2420000192) 0x7f1acc07b1d0 con 0x7f1adc0fd980 2024-02-19T21:10:03.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ad27fc700 10 client.49817.objecter ms_handle_connect 0x7f1ac8063d50 2024-02-19T21:10:03.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ae362b700 10 client.49817.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:03.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 1 2024-02-19T21:10:03.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac00051a0 osd=0 3 2024-02-19T21:10:03.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac00051a0 osd=0 3 2024-02-19T21:10:03.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac00051a0 osd=0 3 2024-02-19T21:10:03.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 1 2024-02-19T21:10:03.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac00051a0 osd=0 3 2024-02-19T21:10:03.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 0 1 2024-02-19T21:10:03.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac00051a0 osd=0 4 2024-02-19T21:10:03.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ae362b700 10 client.49817.objecter _send_command 1 2024-02-19T21:10:03.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ac0004950 con 0x7f1ac0004530 2024-02-19T21:10:03.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.941+0000 7f1ad27fc700 10 client.49817.objecter ms_handle_connect 0x7f1ac0004530 2024-02-19T21:10:03.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.942+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1ac0004950 con 0x7f1ac0004530 2024-02-19T21:10:03.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.942+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:10:03.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.942+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 1 = system:0 2024-02-19T21:10:03.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.942+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac00051a0 osd=0 4 2024-02-19T21:10:03.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.942+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 0 1 2024-02-19T21:10:03.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.957+0000 7f1ae362b700 10 client.49817.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:03.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.957+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 2 2024-02-19T21:10:03.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.957+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac00051a0 osd=0 3 2024-02-19T21:10:03.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.957+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac00051a0 osd=0 3 2024-02-19T21:10:03.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.957+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac00051a0 osd=0 3 2024-02-19T21:10:03.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.957+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 2 2024-02-19T21:10:03.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.957+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac00051a0 osd=0 3 2024-02-19T21:10:03.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.957+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 0 2 2024-02-19T21:10:03.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.957+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac00051a0 osd=0 4 2024-02-19T21:10:03.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.957+0000 7f1ae362b700 10 client.49817.objecter _send_command 2 2024-02-19T21:10:03.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.957+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1ac0004d30 con 0x7f1ac0004530 2024-02-19T21:10:03.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1ac0004d30 con 0x7f1ac0004530 2024-02-19T21:10:03.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:03.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac00051a0 osd=0 4 2024-02-19T21:10:03.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 0 2 2024-02-19T21:10:03.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ae362b700 10 client.49817.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:03.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 3 2024-02-19T21:10:03.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0003fe0 osd=1 3 2024-02-19T21:10:03.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0003fe0 osd=1 3 2024-02-19T21:10:03.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0003fe0 osd=1 3 2024-02-19T21:10:03.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 3 2024-02-19T21:10:03.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac0003fe0 osd=1 3 2024-02-19T21:10:03.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 1 3 2024-02-19T21:10:03.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0003fe0 osd=1 4 2024-02-19T21:10:03.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ae362b700 10 client.49817.objecter _send_command 3 2024-02-19T21:10:03.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.958+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ac000c000 con 0x7f1ac0004150 2024-02-19T21:10:03.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.959+0000 7f1ad27fc700 10 client.49817.objecter ms_handle_connect 0x7f1ac0004150 2024-02-19T21:10:03.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.959+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1ac000c000 con 0x7f1ac0004150 2024-02-19T21:10:03.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.959+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:10:03.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.959+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 3 = system:0 2024-02-19T21:10:03.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.959+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac0003fe0 osd=1 4 2024-02-19T21:10:03.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.959+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 1 3 2024-02-19T21:10:03.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.974+0000 7f1ae362b700 10 client.49817.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:03.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.974+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 4 2024-02-19T21:10:03.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.974+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0003fe0 osd=1 3 2024-02-19T21:10:03.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.974+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0003fe0 osd=1 3 2024-02-19T21:10:03.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.974+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0003fe0 osd=1 3 2024-02-19T21:10:03.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.974+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 4 2024-02-19T21:10:03.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.974+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac0003fe0 osd=1 3 2024-02-19T21:10:03.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.974+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 1 4 2024-02-19T21:10:03.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.974+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0003fe0 osd=1 4 2024-02-19T21:10:03.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.974+0000 7f1ae362b700 10 client.49817.objecter _send_command 4 2024-02-19T21:10:03.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.974+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1ac0001850 con 0x7f1ac0004150 2024-02-19T21:10:03.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1ac0001850 con 0x7f1ac0004150 2024-02-19T21:10:03.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:03.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac0003fe0 osd=1 4 2024-02-19T21:10:03.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 1 4 2024-02-19T21:10:03.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ae362b700 10 client.49817.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:03.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 5 2024-02-19T21:10:03.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0003630 osd=2 3 2024-02-19T21:10:03.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0003630 osd=2 3 2024-02-19T21:10:03.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0003630 osd=2 3 2024-02-19T21:10:03.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 5 2024-02-19T21:10:03.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac0003630 osd=2 3 2024-02-19T21:10:03.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 2 5 2024-02-19T21:10:03.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0003630 osd=2 4 2024-02-19T21:10:03.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ae362b700 10 client.49817.objecter _send_command 5 2024-02-19T21:10:03.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.975+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ac0003e70 con 0x7f1ac000d2f0 2024-02-19T21:10:03.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.976+0000 7f1ad27fc700 10 client.49817.objecter ms_handle_connect 0x7f1ac000d2f0 2024-02-19T21:10:03.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.976+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1ac0003e70 con 0x7f1ac000d2f0 2024-02-19T21:10:03.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.976+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:10:03.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.976+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 5 = system:0 2024-02-19T21:10:03.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.976+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac0003630 osd=2 4 2024-02-19T21:10:03.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.976+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 2 5 2024-02-19T21:10:03.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.990+0000 7f1ae362b700 10 client.49817.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:03.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.990+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 6 2024-02-19T21:10:03.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.990+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0003630 osd=2 3 2024-02-19T21:10:03.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.990+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0003630 osd=2 3 2024-02-19T21:10:03.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.990+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0003630 osd=2 3 2024-02-19T21:10:03.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.990+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 6 2024-02-19T21:10:03.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.990+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac0003630 osd=2 3 2024-02-19T21:10:03.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.990+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 2 6 2024-02-19T21:10:03.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.990+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0003630 osd=2 4 2024-02-19T21:10:03.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.990+0000 7f1ae362b700 10 client.49817.objecter _send_command 6 2024-02-19T21:10:03.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.990+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1ac000e6c0 con 0x7f1ac000d2f0 2024-02-19T21:10:03.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1ac000e6c0 con 0x7f1ac000d2f0 2024-02-19T21:10:03.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:03.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac0003630 osd=2 4 2024-02-19T21:10:03.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 2 6 2024-02-19T21:10:03.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ae362b700 10 client.49817.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:03.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 7 2024-02-19T21:10:03.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0003430 osd=3 3 2024-02-19T21:10:03.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0003430 osd=3 3 2024-02-19T21:10:03.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0003430 osd=3 3 2024-02-19T21:10:03.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 7 2024-02-19T21:10:03.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac0003430 osd=3 3 2024-02-19T21:10:03.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 3 7 2024-02-19T21:10:03.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0003430 osd=3 4 2024-02-19T21:10:03.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ae362b700 10 client.49817.objecter _send_command 7 2024-02-19T21:10:03.077 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:10:03.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ac0019f10 con 0x7f1ac0016290 2024-02-19T21:10:03.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.992+0000 7f1ad27fc700 10 client.49817.objecter ms_handle_connect 0x7f1ac0016290 2024-02-19T21:10:03.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.993+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1ac0019f10 con 0x7f1ac0016290 2024-02-19T21:10:03.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.993+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:10:03.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.993+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 7 = system:0 2024-02-19T21:10:03.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.993+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac0003430 osd=3 4 2024-02-19T21:10:03.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:01.993+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 3 7 2024-02-19T21:10:03.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.007+0000 7f1ae362b700 10 client.49817.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:03.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.007+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 8 2024-02-19T21:10:03.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.007+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0003430 osd=3 3 2024-02-19T21:10:03.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.007+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0003430 osd=3 3 2024-02-19T21:10:03.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.007+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0003430 osd=3 3 2024-02-19T21:10:03.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.007+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 8 2024-02-19T21:10:03.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.007+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac0003430 osd=3 3 2024-02-19T21:10:03.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.007+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 3 8 2024-02-19T21:10:03.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.007+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0003430 osd=3 4 2024-02-19T21:10:03.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.007+0000 7f1ae362b700 10 client.49817.objecter _send_command 8 2024-02-19T21:10:03.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.007+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1ac000e100 con 0x7f1ac0016290 2024-02-19T21:10:03.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.008+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1ac000e100 con 0x7f1ac0016290 2024-02-19T21:10:03.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.008+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:03.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.008+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.008+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac0003430 osd=3 4 2024-02-19T21:10:03.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.008+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 3 8 2024-02-19T21:10:03.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.009+0000 7f1ae362b700 10 client.49817.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:03.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.009+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 9 2024-02-19T21:10:03.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.009+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac000e300 osd=4 3 2024-02-19T21:10:03.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.009+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac000e300 osd=4 3 2024-02-19T21:10:03.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.009+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac000e300 osd=4 3 2024-02-19T21:10:03.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.009+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 9 2024-02-19T21:10:03.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.009+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac000e300 osd=4 3 2024-02-19T21:10:03.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.009+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 4 9 2024-02-19T21:10:03.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.009+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac000e300 osd=4 4 2024-02-19T21:10:03.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.009+0000 7f1ae362b700 10 client.49817.objecter _send_command 9 2024-02-19T21:10:03.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.009+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ac000e4b0 con 0x7f1ac000ce50 2024-02-19T21:10:03.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.009+0000 7f1ad27fc700 10 client.49817.objecter ms_handle_connect 0x7f1ac000ce50 2024-02-19T21:10:03.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.010+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1ac000e4b0 con 0x7f1ac000ce50 2024-02-19T21:10:03.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.010+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:10:03.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.010+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 9 = system:0 2024-02-19T21:10:03.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.010+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac000e300 osd=4 4 2024-02-19T21:10:03.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.010+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 4 9 2024-02-19T21:10:03.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.024+0000 7f1ae362b700 10 client.49817.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:03.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.024+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 10 2024-02-19T21:10:03.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.024+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac000e300 osd=4 3 2024-02-19T21:10:03.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.024+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac000e300 osd=4 3 2024-02-19T21:10:03.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.024+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac000e300 osd=4 3 2024-02-19T21:10:03.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.024+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 10 2024-02-19T21:10:03.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.024+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac000e300 osd=4 3 2024-02-19T21:10:03.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.024+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 4 10 2024-02-19T21:10:03.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.024+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac000e300 osd=4 4 2024-02-19T21:10:03.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.024+0000 7f1ae362b700 10 client.49817.objecter _send_command 10 2024-02-19T21:10:03.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.024+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1ac000e4b0 con 0x7f1ac000ce50 2024-02-19T21:10:03.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1ac000e4b0 con 0x7f1ac000ce50 2024-02-19T21:10:03.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:03.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac000e300 osd=4 4 2024-02-19T21:10:03.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 4 10 2024-02-19T21:10:03.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ae362b700 10 client.49817.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:03.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 11 2024-02-19T21:10:03.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac000e4b0 osd=5 3 2024-02-19T21:10:03.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac000e4b0 osd=5 3 2024-02-19T21:10:03.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac000e4b0 osd=5 3 2024-02-19T21:10:03.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 11 2024-02-19T21:10:03.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac000e4b0 osd=5 3 2024-02-19T21:10:03.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 5 11 2024-02-19T21:10:03.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac000e4b0 osd=5 4 2024-02-19T21:10:03.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ae362b700 10 client.49817.objecter _send_command 11 2024-02-19T21:10:03.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.026+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ac001c480 con 0x7f1ac001bea0 2024-02-19T21:10:03.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.027+0000 7f1ad27fc700 10 client.49817.objecter ms_handle_connect 0x7f1ac001bea0 2024-02-19T21:10:03.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.027+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1ac001c480 con 0x7f1ac001bea0 2024-02-19T21:10:03.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.027+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:10:03.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.027+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 11 = system:0 2024-02-19T21:10:03.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.027+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac000e4b0 osd=5 4 2024-02-19T21:10:03.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.027+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 5 11 2024-02-19T21:10:03.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.041+0000 7f1ae362b700 10 client.49817.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:03.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.041+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 12 2024-02-19T21:10:03.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.041+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac000e4b0 osd=5 3 2024-02-19T21:10:03.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.041+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac000e4b0 osd=5 3 2024-02-19T21:10:03.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.041+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac000e4b0 osd=5 3 2024-02-19T21:10:03.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.041+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 12 2024-02-19T21:10:03.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.041+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac000e4b0 osd=5 3 2024-02-19T21:10:03.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.041+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 5 12 2024-02-19T21:10:03.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.041+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac000e4b0 osd=5 4 2024-02-19T21:10:03.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.041+0000 7f1ae362b700 10 client.49817.objecter _send_command 12 2024-02-19T21:10:03.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.041+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1ac001c480 con 0x7f1ac001bea0 2024-02-19T21:10:03.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1ac001c480 con 0x7f1ac001bea0 2024-02-19T21:10:03.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:03.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac000e4b0 osd=5 4 2024-02-19T21:10:03.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 5 12 2024-02-19T21:10:03.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ae362b700 10 client.49817.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:03.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 13 2024-02-19T21:10:03.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac001c480 osd=6 3 2024-02-19T21:10:03.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac001c480 osd=6 3 2024-02-19T21:10:03.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac001c480 osd=6 3 2024-02-19T21:10:03.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 13 2024-02-19T21:10:03.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac001c480 osd=6 3 2024-02-19T21:10:03.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 6 13 2024-02-19T21:10:03.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac001c480 osd=6 4 2024-02-19T21:10:03.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ae362b700 10 client.49817.objecter _send_command 13 2024-02-19T21:10:03.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.043+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ac0027060 con 0x7f1ac0023510 2024-02-19T21:10:03.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.044+0000 7f1ad27fc700 10 client.49817.objecter ms_handle_connect 0x7f1ac0023510 2024-02-19T21:10:03.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.044+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1ac0027060 con 0x7f1ac0023510 2024-02-19T21:10:03.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.044+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:10:03.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.044+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 13 = system:0 2024-02-19T21:10:03.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.044+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac001c480 osd=6 4 2024-02-19T21:10:03.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.044+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 6 13 2024-02-19T21:10:03.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.058+0000 7f1ae362b700 10 client.49817.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:03.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.058+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 14 2024-02-19T21:10:03.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.058+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac001c480 osd=6 3 2024-02-19T21:10:03.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.058+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac001c480 osd=6 3 2024-02-19T21:10:03.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.058+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac001c480 osd=6 3 2024-02-19T21:10:03.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.058+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 14 2024-02-19T21:10:03.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.058+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac001c480 osd=6 3 2024-02-19T21:10:03.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.058+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 6 14 2024-02-19T21:10:03.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.058+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac001c480 osd=6 4 2024-02-19T21:10:03.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.058+0000 7f1ae362b700 10 client.49817.objecter _send_command 14 2024-02-19T21:10:03.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.058+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1ac0028810 con 0x7f1ac0023510 2024-02-19T21:10:03.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1ac0028810 con 0x7f1ac0023510 2024-02-19T21:10:03.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:03.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac001c480 osd=6 4 2024-02-19T21:10:03.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 6 14 2024-02-19T21:10:03.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ae362b700 10 client.49817.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:03.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 15 2024-02-19T21:10:03.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0028810 osd=7 3 2024-02-19T21:10:03.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0028810 osd=7 3 2024-02-19T21:10:03.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0028810 osd=7 3 2024-02-19T21:10:03.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 15 2024-02-19T21:10:03.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac0028810 osd=7 3 2024-02-19T21:10:03.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 7 15 2024-02-19T21:10:03.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0028810 osd=7 4 2024-02-19T21:10:03.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ae362b700 10 client.49817.objecter _send_command 15 2024-02-19T21:10:03.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.060+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ac0029440 con 0x7f1ac0028e90 2024-02-19T21:10:03.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.061+0000 7f1ad27fc700 10 client.49817.objecter ms_handle_connect 0x7f1ac0028e90 2024-02-19T21:10:03.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.062+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1ac0029440 con 0x7f1ac0028e90 2024-02-19T21:10:03.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.062+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:10:03.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.062+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 15 = system:0 2024-02-19T21:10:03.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.062+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac0028810 osd=7 4 2024-02-19T21:10:03.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.062+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 7 15 2024-02-19T21:10:03.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.076+0000 7f1ae362b700 10 client.49817.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:03.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.076+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign -1 16 2024-02-19T21:10:03.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.076+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0028810 osd=7 3 2024-02-19T21:10:03.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.076+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0028810 osd=7 3 2024-02-19T21:10:03.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.076+0000 7f1ae362b700 20 client.49817.objecter _get_session s=0x7f1ac0028810 osd=7 3 2024-02-19T21:10:03.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.076+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_remove -1 16 2024-02-19T21:10:03.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.076+0000 7f1ae362b700 20 client.49817.objecter get_session s=0x7f1ac0028810 osd=7 3 2024-02-19T21:10:03.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.076+0000 7f1ae362b700 15 client.49817.objecter _session_command_op_assign 7 16 2024-02-19T21:10:03.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.076+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0028810 osd=7 4 2024-02-19T21:10:03.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.076+0000 7f1ae362b700 10 client.49817.objecter _send_command 16 2024-02-19T21:10:03.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.076+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1ac0029440 con 0x7f1ac0028e90 2024-02-19T21:10:03.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ad27fc700 1 -- 172.21.15.163:0/2192545964 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1ac0029440 con 0x7f1ac0028e90 2024-02-19T21:10:03.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ad27fc700 10 client.49817.objecter ms_dispatch 0x7f1adc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:03.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ad27fc700 10 client.49817.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ad27fc700 20 client.49817.objecter put_session s=0x7f1ac0028810 osd=7 4 2024-02-19T21:10:03.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ad27fc700 15 client.49817.objecter _session_command_op_remove 7 16 2024-02-19T21:10:03.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 10 librados: watch_flush enter 2024-02-19T21:10:03.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 10 librados: watch_flush exit 2024-02-19T21:10:03.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 10 client.49817.objecter close_session for osd.0 2024-02-19T21:10:03.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 >> v1:172.21.15.163:6801/1559418115 conn(0x7f1ac0004530 legacy=0x7f1ac0002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac00051a0 osd=0 1 2024-02-19T21:10:03.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 10 client.49817.objecter close_session for osd.1 2024-02-19T21:10:03.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 >> v1:172.21.15.163:6805/3125047432 conn(0x7f1ac0004150 legacy=0x7f1ac000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0003fe0 osd=1 1 2024-02-19T21:10:03.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 10 client.49817.objecter close_session for osd.2 2024-02-19T21:10:03.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 >> v1:172.21.15.163:6809/1563158487 conn(0x7f1ac000d2f0 legacy=0x7f1ac000d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0003630 osd=2 1 2024-02-19T21:10:03.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 10 client.49817.objecter close_session for osd.3 2024-02-19T21:10:03.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 >> v1:172.21.15.163:6813/2112279769 conn(0x7f1ac0016290 legacy=0x7f1ac0018680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0003430 osd=3 1 2024-02-19T21:10:03.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 10 client.49817.objecter close_session for osd.4 2024-02-19T21:10:03.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 >> v1:172.21.15.193:6800/3129716965 conn(0x7f1ac000ce50 legacy=0x7f1ac001a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae1bc8700 1 -- 172.21.15.163:0/2192545964 reap_dead start 2024-02-19T21:10:03.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae1bc8700 1 -- 172.21.15.163:0/2192545964 reap_dead start 2024-02-19T21:10:03.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac000e300 osd=4 1 2024-02-19T21:10:03.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 10 client.49817.objecter close_session for osd.5 2024-02-19T21:10:03.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 >> v1:172.21.15.193:6804/805161991 conn(0x7f1ac001bea0 legacy=0x7f1ac00218e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac000e4b0 osd=5 1 2024-02-19T21:10:03.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 10 client.49817.objecter close_session for osd.6 2024-02-19T21:10:03.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 >> v1:172.21.15.193:6808/787604615 conn(0x7f1ac0023510 legacy=0x7f1ac0025900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac001c480 osd=6 1 2024-02-19T21:10:03.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 10 client.49817.objecter close_session for osd.7 2024-02-19T21:10:03.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 >> v1:172.21.15.193:6812/157927432 conn(0x7f1ac0028e90 legacy=0x7f1ac0027060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 20 client.49817.objecter put_session s=0x7f1ac0028810 osd=7 1 2024-02-19T21:10:03.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 20 client.49817.objecter shutdown clearing up homeless session... 2024-02-19T21:10:03.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 10 client.49817.objecter successfully canceled tick 2024-02-19T21:10:03.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 >> v1:172.21.15.163:6800/1552514681 conn(0x7f1ac8063d50 legacy=0x7f1ac8066210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 >> v1:172.21.15.163:6790/0 conn(0x7f1adc0fd980 legacy=0x7f1adc0f9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.092 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae1bc8700 1 -- 172.21.15.163:0/2192545964 reap_dead start 2024-02-19T21:10:03.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae1bc8700 1 -- 172.21.15.163:0/2192545964 reap_dead start 2024-02-19T21:10:03.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 shutdown_connections 2024-02-19T21:10:03.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 >> 172.21.15.163:0/2192545964 conn(0x7f1adc0f35a0 msgr2=0x7f1adc107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:03.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 shutdown_connections 2024-02-19T21:10:03.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.078+0000 7f1ae362b700 1 -- 172.21.15.163:0/2192545964 wait complete. 2024-02-19T21:10:03.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:02.079+0000 7f1ae362b700 1 librados: shutdown 2024-02-19T21:10:03.093 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.093 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.093 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.093 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.093 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.093 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.093 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.094 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:10:03.238 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:10:03.241 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-19T21:10:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:03.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:10:03.343 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:03.371 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:03.443 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:03.446 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:03.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:03.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:02.518883+0000 osd.1 (osd.1) 185 : cluster [DBG] 6.e scrub starts 2024-02-19T21:10:03.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:02.519637+0000 osd.1 (osd.1) 186 : cluster [DBG] 6.e scrub ok 2024-02-19T21:10:03.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:02.518883+0000 osd.1 (osd.1) 185 : cluster [DBG] 6.e scrub starts 2024-02-19T21:10:03.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:02.519637+0000 osd.1 (osd.1) 186 : cluster [DBG] 6.e scrub ok 2024-02-19T21:10:03.495 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:10:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:10:03] "GET /metrics HTTP/1.1" 200 189563 "" "Prometheus/2.33.4" 2024-02-19T21:10:03.511 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:10:03.547 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:10:03.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:02.518883+0000 osd.1 (osd.1) 185 : cluster [DBG] 6.e scrub starts 2024-02-19T21:10:03.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:02.519637+0000 osd.1 (osd.1) 186 : cluster [DBG] 6.e scrub ok 2024-02-19T21:10:03.574 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:03.696 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:10:03.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.698+0000 7fe358dc9700 1 -- 172.21.15.163:0/3651726995 >> v1:172.21.15.163:6789/0 conn(0x7fe3541042c0 legacy=0x7fe354106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.698+0000 7fe358dc9700 1 -- 172.21.15.163:0/3651726995 shutdown_connections 2024-02-19T21:10:03.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.698+0000 7fe358dc9700 1 -- 172.21.15.163:0/3651726995 >> 172.21.15.163:0/3651726995 conn(0x7fe3540f35a0 msgr2=0x7fe3540f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:03.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.698+0000 7fe358dc9700 1 -- 172.21.15.163:0/3651726995 shutdown_connections 2024-02-19T21:10:03.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.698+0000 7fe358dc9700 1 -- 172.21.15.163:0/3651726995 wait complete. 2024-02-19T21:10:03.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.698+0000 7fe358dc9700 1 librados: starting msgr at 2024-02-19T21:10:03.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.698+0000 7fe358dc9700 1 librados: starting objecter 2024-02-19T21:10:03.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe358dc9700 1 Processor -- start 2024-02-19T21:10:03.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe358dc9700 1 -- start start 2024-02-19T21:10:03.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe358dc9700 1 librados: setting wanted keys 2024-02-19T21:10:03.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe358dc9700 1 librados: calling monclient init 2024-02-19T21:10:03.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe358dc9700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe3540fcf30 con 0x7fe354100780 2024-02-19T21:10:03.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe358dc9700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe3540fd100 con 0x7fe3540fd980 2024-02-19T21:10:03.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe358dc9700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe3540fd2f0 con 0x7fe3541042c0 2024-02-19T21:10:03.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe351d9b700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fe354100780 0x7fe354114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:40730/0 (socket says 172.21.15.163:40730) 2024-02-19T21:10:03.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe351d9b700 1 -- 172.21.15.163:0/673487266 learned_addr learned my addr 172.21.15.163:0/673487266 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:03.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe352d9d700 1 --1- 172.21.15.163:0/673487266 >> v1:172.21.15.163:6790/0 conn(0x7fe3541042c0 0x7fe354118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:37046/0 (socket says 172.21.15.163:37046) 2024-02-19T21:10:03.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe3437fe700 10 client.?.objecter ms_handle_connect 0x7fe354100780 2024-02-19T21:10:03.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe3437fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:03.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe3437fe700 10 client.?.objecter ms_handle_connect 0x7fe3541042c0 2024-02-19T21:10:03.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe3437fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:03.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe3437fe700 10 client.?.objecter ms_handle_connect 0x7fe3540fd980 2024-02-19T21:10:03.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe3437fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:03.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707314702 0 0) 0x7fe3540fcf30 con 0x7fe354100780 2024-02-19T21:10:03.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe33c0033f0 con 0x7fe354100780 2024-02-19T21:10:03.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702613318 0 0) 0x7fe3540fd2f0 con 0x7fe3541042c0 2024-02-19T21:10:03.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3540fcf30 con 0x7fe3541042c0 2024-02-19T21:10:03.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647510471 0 0) 0x7fe3540fd100 con 0x7fe3540fd980 2024-02-19T21:10:03.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3540fd2f0 con 0x7fe3540fd980 2024-02-19T21:10:03.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 763185610 0 0) 0x7fe33c0033f0 con 0x7fe354100780 2024-02-19T21:10:03.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3540fd100 con 0x7fe354100780 2024-02-19T21:10:03.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.699+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe348002910 con 0x7fe354100780 2024-02-19T21:10:03.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3479307617 0 0) 0x7fe3540fcf30 con 0x7fe3541042c0 2024-02-19T21:10:03.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe33c0033f0 con 0x7fe3541042c0 2024-02-19T21:10:03.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe34c002240 con 0x7fe3541042c0 2024-02-19T21:10:03.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2518269511 0 0) 0x7fe3540fd2f0 con 0x7fe3540fd980 2024-02-19T21:10:03.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3540fcf30 con 0x7fe3540fd980 2024-02-19T21:10:03.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe3440015a0 con 0x7fe3540fd980 2024-02-19T21:10:03.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4228342043 0 0) 0x7fe3540fd100 con 0x7fe354100780 2024-02-19T21:10:03.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 >> v1:172.21.15.163:6790/0 conn(0x7fe3541042c0 legacy=0x7fe354118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 >> v1:172.21.15.193:6789/0 conn(0x7fe3540fd980 legacy=0x7fe3540fc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35411cb90 con 0x7fe354100780 2024-02-19T21:10:03.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe358dc9700 1 -- 172.21.15.163:0/673487266 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe354119bc0 con 0x7fe354100780 2024-02-19T21:10:03.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe358dc9700 10 client.23103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:03.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe358dc9700 1 -- 172.21.15.163:0/673487266 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35411a030 con 0x7fe354100780 2024-02-19T21:10:03.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe358dc9700 1 librados: init done 2024-02-19T21:10:03.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fe3480017d0 con 0x7fe354100780 2024-02-19T21:10:03.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.700+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe348005a40 con 0x7fe354100780 2024-02-19T21:10:03.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.701+0000 7fe358dc9700 1 -- 172.21.15.163:0/673487266 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe334004f20 con 0x7fe354100780 2024-02-19T21:10:03.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.701+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fe348001330 con 0x7fe354100780 2024-02-19T21:10:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:03.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.702+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(413..413 src has 43..413) v4 ==== 7946+0+0 (unknown 882006242 0 0) 0x7fe348078d70 con 0x7fe354100780 2024-02-19T21:10:03.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.702+0000 7fe3437fe700 10 client.23103.objecter ms_dispatch 0x7fe354001550 osd_map(413..413 src has 43..413) v4 2024-02-19T21:10:03.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.702+0000 7fe3437fe700 3 client.23103.objecter handle_osd_map got epochs [413,413] > 0 2024-02-19T21:10:03.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.702+0000 7fe3437fe700 3 client.23103.objecter handle_osd_map decoding full epoch 413 2024-02-19T21:10:03.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.702+0000 7fe3437fe700 20 client.23103.objecter dump_active .. 0 homeless 2024-02-19T21:10:03.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.702+0000 7fe3437fe700 10 client.23103.objecter ms_handle_connect 0x7fe33c063e20 2024-02-19T21:10:03.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.706+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe34804c1d0 con 0x7fe354100780 2024-02-19T21:10:03.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:10:03.805 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:03.834 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:03.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.879+0000 7fe358dc9700 1 -- 172.21.15.163:0/673487266 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe334000b90 con 0x7fe33c063e20 2024-02-19T21:10:03.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.894+0000 7fe3437fe700 1 -- 172.21.15.163:0/673487266 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+576377 (unknown 2965378022 0 2562115847) 0x7fe334000b90 con 0x7fe33c063e20 2024-02-19T21:10:03.924 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:10:03.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.898+0000 7fe358dc9700 10 librados: watch_flush enter 2024-02-19T21:10:03.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.898+0000 7fe358dc9700 10 librados: watch_flush exit 2024-02-19T21:10:03.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.898+0000 7fe358dc9700 20 client.23103.objecter shutdown clearing up homeless session... 2024-02-19T21:10:03.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.898+0000 7fe358dc9700 10 client.23103.objecter successfully canceled tick 2024-02-19T21:10:03.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.898+0000 7fe358dc9700 1 -- 172.21.15.163:0/673487266 >> v1:172.21.15.163:6800/1552514681 conn(0x7fe33c063e20 legacy=0x7fe33c0662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.898+0000 7fe358dc9700 1 -- 172.21.15.163:0/673487266 >> v1:172.21.15.163:6789/0 conn(0x7fe354100780 legacy=0x7fe354114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:03.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.898+0000 7fe358dc9700 1 -- 172.21.15.163:0/673487266 shutdown_connections 2024-02-19T21:10:03.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.898+0000 7fe358dc9700 1 -- 172.21.15.163:0/673487266 >> 172.21.15.163:0/673487266 conn(0x7fe3540f35a0 msgr2=0x7fe354107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:03.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.898+0000 7fe358dc9700 1 -- 172.21.15.163:0/673487266 shutdown_connections 2024-02-19T21:10:03.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.898+0000 7fe358dc9700 1 -- 172.21.15.163:0/673487266 wait complete. 2024-02-19T21:10:03.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:03.898+0000 7fe358dc9700 1 librados: shutdown 2024-02-19T21:10:03.933 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:10:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:04.389 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:04.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:04.491 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:04.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:02.530878+0000 osd.3 (osd.3) 175 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:10:04.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:02.532794+0000 osd.3 (osd.3) 176 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:10:04.495 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:03.052816+0000 osd.2 (osd.2) 152 : cluster [DBG] 6.f scrub starts 2024-02-19T21:10:04.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:03.053804+0000 osd.2 (osd.2) 153 : cluster [DBG] 6.f scrub ok 2024-02-19T21:10:04.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:03.209331+0000 mgr.y (mgr.14550) 372 : cluster [DBG] pgmap v596: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:10:04.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:03.549224+0000 osd.1 (osd.1) 187 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:10:04.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:03.552683+0000 osd.1 (osd.1) 188 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:10:04.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:03.880571+0000 mgr.y (mgr.14550) 373 : audit [DBG] from='client.23103 v1:172.21.15.163:0/673487266' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:10:04.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:04.261577+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e414: 8 total, 8 up, 7 in 2024-02-19T21:10:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:02.530878+0000 osd.3 (osd.3) 175 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:10:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:02.532794+0000 osd.3 (osd.3) 176 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:10:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:03.052816+0000 osd.2 (osd.2) 152 : cluster [DBG] 6.f scrub starts 2024-02-19T21:10:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:03.053804+0000 osd.2 (osd.2) 153 : cluster [DBG] 6.f scrub ok 2024-02-19T21:10:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:03.209331+0000 mgr.y (mgr.14550) 372 : cluster [DBG] pgmap v596: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:10:04.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:03.549224+0000 osd.1 (osd.1) 187 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:10:04.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:03.552683+0000 osd.1 (osd.1) 188 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:10:04.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:03.880571+0000 mgr.y (mgr.14550) 373 : audit [DBG] from='client.23103 v1:172.21.15.163:0/673487266' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:10:04.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:04.261577+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e414: 8 total, 8 up, 7 in 2024-02-19T21:10:04.517 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:02.530878+0000 osd.3 (osd.3) 175 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:10:04.517 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:02.532794+0000 osd.3 (osd.3) 176 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:10:04.517 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:03.052816+0000 osd.2 (osd.2) 152 : cluster [DBG] 6.f scrub starts 2024-02-19T21:10:04.517 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:03.053804+0000 osd.2 (osd.2) 153 : cluster [DBG] 6.f scrub ok 2024-02-19T21:10:04.517 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:03.209331+0000 mgr.y (mgr.14550) 372 : cluster [DBG] pgmap v596: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:10:04.517 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:03.549224+0000 osd.1 (osd.1) 187 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:10:04.517 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:03.552683+0000 osd.1 (osd.1) 188 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:10:04.517 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:03.880571+0000 mgr.y (mgr.14550) 373 : audit [DBG] from='client.23103 v1:172.21.15.163:0/673487266' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:10:04.518 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:04.261577+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e414: 8 total, 8 up, 7 in 2024-02-19T21:10:04.520 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:05.270 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:05.274 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:10:05.274 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-02-19T21:10:05.274 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-02-19T21:10:05.274 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-02-19T21:10:05.274 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.274 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:109 61.1e 61:7b0234b0:test-rados-api-smithi163-83928-10::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e126)", 2024-02-19T21:10:05.274 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:06.034266+0000", 2024-02-19T21:10:05.274 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 299.23919161700002, 2024-02-19T21:10:05.274 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013927448, 2024-02-19T21:10:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:10:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:10:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 109 2024-02-19T21:10:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034266+0000", 2024-02-19T21:10:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.275 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.276 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034266+0000", 2024-02-19T21:10:05.276 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:10:05.276 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.276 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.276 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.276 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034264+0000", 2024-02-19T21:10:05.276 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4800000000000001e-06 2024-02-19T21:10:05.276 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.276 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.276 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.276 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034272+0000", 2024-02-19T21:10:05.276 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8009999999999993e-06 2024-02-19T21:10:05.276 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.277 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.277 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.277 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034281+0000", 2024-02-19T21:10:05.277 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9020000000000002e-06 2024-02-19T21:10:05.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.277 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.277 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.277 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034288+0000", 2024-02-19T21:10:05.277 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018518399999999999 2024-02-19T21:10:05.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.277 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.277 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.277 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034473+0000", 2024-02-19T21:10:05.277 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012265400000000001 2024-02-19T21:10:05.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.278 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.278 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.278 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034596+0000", 2024-02-19T21:10:05.278 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000145477 2024-02-19T21:10:05.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.278 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.278 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,6", 2024-02-19T21:10:05.278 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034742+0000", 2024-02-19T21:10:05.278 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001663596 2024-02-19T21:10:05.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.278 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.278 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:10:05.278 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.036405+0000", 2024-02-19T21:10:05.278 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0045168939999999996 2024-02-19T21:10:05.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.279 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.279 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.279 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.040922+0000", 2024-02-19T21:10:05.279 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0072134110000000003 2024-02-19T21:10:05.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.279 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.279 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.279 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.048135+0000", 2024-02-19T21:10:05.279 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4739000000000001e-05 2024-02-19T21:10:05.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.279 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.279 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:10:05.279 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.048170+0000", 2024-02-19T21:10:05.279 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3264e-05 2024-02-19T21:10:05.280 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.280 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.280 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.280 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.048193+0000", 2024-02-19T21:10:05.280 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013927448 2024-02-19T21:10:05.280 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.280 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.280 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.280 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.280 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.280 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.38953.0:6 82.c 82:3197012c:::foo_rmxattr:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-02-19T21:10:05.280 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.230127+0000", 2024-02-19T21:10:05.280 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 293.04333096099998, 2024-02-19T21:10:05.280 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.044067013000000002, 2024-02-19T21:10:05.281 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.281 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:05.281 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.281 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.38953", 2024-02-19T21:10:05.281 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3840798168", 2024-02-19T21:10:05.281 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 6 2024-02-19T21:10:05.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.281 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.281 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.281 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.281 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230127+0000", 2024-02-19T21:10:05.281 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.281 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.281 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.282 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.282 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230127+0000", 2024-02-19T21:10:05.282 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:10:05.282 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.282 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.282 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.282 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230125+0000", 2024-02-19T21:10:05.282 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5880000000000004e-06 2024-02-19T21:10:05.282 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.282 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.282 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.282 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230134+0000", 2024-02-19T21:10:05.282 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1489999999999994e-06 2024-02-19T21:10:05.282 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.282 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.283 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.283 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230142+0000", 2024-02-19T21:10:05.283 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9089999999999999e-06 2024-02-19T21:10:05.283 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.283 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.283 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.283 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230147+0000", 2024-02-19T21:10:05.283 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5340999999999997e-05 2024-02-19T21:10:05.283 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.283 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.283 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.283 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230213+0000", 2024-02-19T21:10:05.283 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000120505 2024-02-19T21:10:05.283 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.283 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.284 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.284 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230333+0000", 2024-02-19T21:10:05.284 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015051999999999999 2024-02-19T21:10:05.284 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.284 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.284 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,4", 2024-02-19T21:10:05.284 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230484+0000", 2024-02-19T21:10:05.284 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00033057799999999999 2024-02-19T21:10:05.284 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.284 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.284 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:10:05.284 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230814+0000", 2024-02-19T21:10:05.284 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000223568 2024-02-19T21:10:05.284 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.285 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.285 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.285 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.231038+0000", 2024-02-19T21:10:05.285 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.043097783000000001 2024-02-19T21:10:05.285 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.285 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.285 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.285 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.274136+0000", 2024-02-19T21:10:05.285 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2132999999999999e-05 2024-02-19T21:10:05.285 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.285 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.285 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:10:05.285 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.274168+0000", 2024-02-19T21:10:05.285 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5828000000000001e-05 2024-02-19T21:10:05.285 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.286 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.286 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.286 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.274194+0000", 2024-02-19T21:10:05.286 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.044067013000000002 2024-02-19T21:10:05.286 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.286 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.286 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.286 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.286 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.286 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.38953.0:8 82.c 82:3197012c:::foo_rmxattr:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e132)", 2024-02-19T21:10:05.286 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.288784+0000", 2024-02-19T21:10:05.286 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 292.98467350700002, 2024-02-19T21:10:05.286 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.027473164000000001, 2024-02-19T21:10:05.286 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.287 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:05.287 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.287 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.38953", 2024-02-19T21:10:05.287 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3840798168", 2024-02-19T21:10:05.287 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 8 2024-02-19T21:10:05.287 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.287 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.287 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.287 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.287 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288784+0000", 2024-02-19T21:10:05.287 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.287 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.287 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.288 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.288 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288784+0000", 2024-02-19T21:10:05.288 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:10:05.288 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.288 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.288 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.288 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288781+0000", 2024-02-19T21:10:05.288 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2977e-05 2024-02-19T21:10:05.288 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.288 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.288 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.288 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288794+0000", 2024-02-19T21:10:05.288 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6616000000000001e-05 2024-02-19T21:10:05.288 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.289 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.289 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.289 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288820+0000", 2024-02-19T21:10:05.289 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2593e-05 2024-02-19T21:10:05.289 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.289 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.289 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.289 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288833+0000", 2024-02-19T21:10:05.289 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9002000000000001e-05 2024-02-19T21:10:05.289 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.289 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.289 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.289 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288902+0000", 2024-02-19T21:10:05.289 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4057000000000002e-05 2024-02-19T21:10:05.290 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.290 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.290 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.290 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288946+0000", 2024-02-19T21:10:05.290 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2805999999999994e-05 2024-02-19T21:10:05.290 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.290 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.290 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,4", 2024-02-19T21:10:05.290 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.289019+0000", 2024-02-19T21:10:05.290 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025527699999999999 2024-02-19T21:10:05.290 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.290 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.290 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:10:05.290 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.289274+0000", 2024-02-19T21:10:05.291 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000221688 2024-02-19T21:10:05.291 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.291 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.291 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.291 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.289496+0000", 2024-02-19T21:10:05.291 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026693499999999998 2024-02-19T21:10:05.291 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.291 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.291 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.291 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.316189+0000", 2024-02-19T21:10:05.291 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.042e-05 2024-02-19T21:10:05.291 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.291 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.291 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:10:05.291 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.316220+0000", 2024-02-19T21:10:05.292 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7688000000000002e-05 2024-02-19T21:10:05.292 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.292 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.292 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.292 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.316257+0000", 2024-02-19T21:10:05.292 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.027473164000000001 2024-02-19T21:10:05.292 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.292 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.292 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.292 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.292 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.292 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.24919.0:2077 28.6 28:6320398f:test-rados-api-smithi163-83920-17::small.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected e102)", 2024-02-19T21:10:05.292 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.322491+0000", 2024-02-19T21:10:05.292 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 292.950966102, 2024-02-19T21:10:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0065386519999999998, 2024-02-19T21:10:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.24919", 2024-02-19T21:10:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/4089460683", 2024-02-19T21:10:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2077 2024-02-19T21:10:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322491+0000", 2024-02-19T21:10:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.293 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322491+0000", 2024-02-19T21:10:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:10:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322489+0000", 2024-02-19T21:10:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6159999999999996e-06 2024-02-19T21:10:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322497+0000", 2024-02-19T21:10:05.294 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1027e-05 2024-02-19T21:10:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322508+0000", 2024-02-19T21:10:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3990000000000001e-06 2024-02-19T21:10:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322514+0000", 2024-02-19T21:10:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3918999999999995e-05 2024-02-19T21:10:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322588+0000", 2024-02-19T21:10:05.295 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3276999999999996e-05 2024-02-19T21:10:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.322672+0000", 2024-02-19T21:10:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00092435900000000003 2024-02-19T21:10:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,3", 2024-02-19T21:10:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.323596+0000", 2024-02-19T21:10:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00035623600000000001 2024-02-19T21:10:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:10:05.296 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.323952+0000", 2024-02-19T21:10:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001907939 2024-02-19T21:10:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.325860+0000", 2024-02-19T21:10:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0031191090000000001 2024-02-19T21:10:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.328979+0000", 2024-02-19T21:10:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.8789000000000002e-05 2024-02-19T21:10:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.297 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:10:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.329008+0000", 2024-02-19T21:10:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2078000000000001e-05 2024-02-19T21:10:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.329030+0000", 2024-02-19T21:10:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0065386519999999998 2024-02-19T21:10:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:248 61.1e 61:780b1894:test-rados-api-smithi163-83928-12::bar:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-02-19T21:10:05.298 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:14.241429+0000", 2024-02-19T21:10:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 291.03202837399999, 2024-02-19T21:10:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.036279570999999997, 2024-02-19T21:10:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:10:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:10:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 248 2024-02-19T21:10:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241429+0000", 2024-02-19T21:10:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.299 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241429+0000", 2024-02-19T21:10:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:10:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241427+0000", 2024-02-19T21:10:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1410000000000005e-06 2024-02-19T21:10:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241435+0000", 2024-02-19T21:10:05.300 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7259999999999997e-06 2024-02-19T21:10:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241445+0000", 2024-02-19T21:10:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3310000000000003e-06 2024-02-19T21:10:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241450+0000", 2024-02-19T21:10:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5508999999999999e-05 2024-02-19T21:10:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.301 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241516+0000", 2024-02-19T21:10:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012163399999999999 2024-02-19T21:10:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241638+0000", 2024-02-19T21:10:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000150047 2024-02-19T21:10:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,6", 2024-02-19T21:10:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241788+0000", 2024-02-19T21:10:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00038934700000000001 2024-02-19T21:10:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.302 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:10:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.242177+0000", 2024-02-19T21:10:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018429899999999999 2024-02-19T21:10:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.242361+0000", 2024-02-19T21:10:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.035272228000000003 2024-02-19T21:10:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.303 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.277633+0000", 2024-02-19T21:10:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8115000000000003e-05 2024-02-19T21:10:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:10:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.277682+0000", 2024-02-19T21:10:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7237e-05 2024-02-19T21:10:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.277709+0000", 2024-02-19T21:10:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.036279570999999997 2024-02-19T21:10:05.304 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:249 61.10 61:0d6787e5:test-rados-api-smithi163-83928-12::baz:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-02-19T21:10:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:14.291021+0000", 2024-02-19T21:10:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 290.98243653700001, 2024-02-19T21:10:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.031923083999999997, 2024-02-19T21:10:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:10:05.305 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:10:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 249 2024-02-19T21:10:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291021+0000", 2024-02-19T21:10:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291021+0000", 2024-02-19T21:10:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:10:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.306 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291019+0000", 2024-02-19T21:10:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1210000000000001e-06 2024-02-19T21:10:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291026+0000", 2024-02-19T21:10:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8940000000000005e-06 2024-02-19T21:10:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291036+0000", 2024-02-19T21:10:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8369999999999996e-06 2024-02-19T21:10:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.307 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291041+0000", 2024-02-19T21:10:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8821999999999999e-05 2024-02-19T21:10:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291109+0000", 2024-02-19T21:10:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4403999999999999e-05 2024-02-19T21:10:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291204+0000", 2024-02-19T21:10:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2696999999999997e-05 2024-02-19T21:10:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.308 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-02-19T21:10:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291297+0000", 2024-02-19T21:10:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00037753299999999999 2024-02-19T21:10:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:10:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291674+0000", 2024-02-19T21:10:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000206999 2024-02-19T21:10:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291881+0000", 2024-02-19T21:10:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.030980852999999999 2024-02-19T21:10:05.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.322862+0000", 2024-02-19T21:10:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5098000000000001e-05 2024-02-19T21:10:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:10:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.322907+0000", 2024-02-19T21:10:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7076000000000002e-05 2024-02-19T21:10:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.322944+0000", 2024-02-19T21:10:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.031923083999999997 2024-02-19T21:10:05.310 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:253 61.1e 61:780b1894:test-rados-api-smithi163-83928-12::bar:head [writefull 0~5] snapc 2=[2] ondisk+write+known_if_redirected e135)", 2024-02-19T21:10:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:15.193696+0000", 2024-02-19T21:10:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 290.079761631, 2024-02-19T21:10:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.104949372, 2024-02-19T21:10:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:10:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:10:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 253 2024-02-19T21:10:05.311 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193696+0000", 2024-02-19T21:10:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193696+0000", 2024-02-19T21:10:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:10:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.312 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193694+0000", 2024-02-19T21:10:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0650000000000001e-06 2024-02-19T21:10:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193701+0000", 2024-02-19T21:10:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.431e-06 2024-02-19T21:10:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193711+0000", 2024-02-19T21:10:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0629999999999996e-06 2024-02-19T21:10:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.313 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193716+0000", 2024-02-19T21:10:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0915000000000003e-05 2024-02-19T21:10:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193767+0000", 2024-02-19T21:10:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9301999999999995e-05 2024-02-19T21:10:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193836+0000", 2024-02-19T21:10:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012302 2024-02-19T21:10:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.314 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,6", 2024-02-19T21:10:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193959+0000", 2024-02-19T21:10:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00041847100000000001 2024-02-19T21:10:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:10:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.194377+0000", 2024-02-19T21:10:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00021071800000000001 2024-02-19T21:10:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.194588+0000", 2024-02-19T21:10:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.103953399 2024-02-19T21:10:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.315 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.298541+0000", 2024-02-19T21:10:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9375e-05 2024-02-19T21:10:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:10:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.298601+0000", 2024-02-19T21:10:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4469999999999999e-05 2024-02-19T21:10:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.298645+0000", 2024-02-19T21:10:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.104949372 2024-02-19T21:10:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.316 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:878 94.15 94:afdf704a:test-rados-api-smithi163-83928-14::foo25:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-19T21:10:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:31.221744+0000", 2024-02-19T21:10:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 274.05171404100003, 2024-02-19T21:10:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0055046590000000003, 2024-02-19T21:10:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:10:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:10:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:10:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 878 2024-02-19T21:10:05.317 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.221744+0000", 2024-02-19T21:10:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.221744+0000", 2024-02-19T21:10:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:10:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.318 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.221740+0000", 2024-02-19T21:10:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0838e-05 2024-02-19T21:10:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.221751+0000", 2024-02-19T21:10:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6963e-05 2024-02-19T21:10:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.221768+0000", 2024-02-19T21:10:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4870000000000008e-06 2024-02-19T21:10:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.319 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.221778+0000", 2024-02-19T21:10:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3424999999999997e-05 2024-02-19T21:10:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.221841+0000", 2024-02-19T21:10:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00023138600000000001 2024-02-19T21:10:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:10:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.222072+0000", 2024-02-19T21:10:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0013751600000000001 2024-02-19T21:10:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.320 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.223448+0000", 2024-02-19T21:10:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9454999999999999e-05 2024-02-19T21:10:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:10:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.223507+0000", 2024-02-19T21:10:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000798104 2024-02-19T21:10:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.224305+0000", 2024-02-19T21:10:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8437999999999999e-05 2024-02-19T21:10:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.321 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.224354+0000", 2024-02-19T21:10:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0028946459999999999 2024-02-19T21:10:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.227248+0000", 2024-02-19T21:10:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0055046590000000003 2024-02-19T21:10:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.322 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22230.0:2 102.d 102:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e154)", 2024-02-19T21:10:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:34.185436+0000", 2024-02-19T21:10:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 271.08802129100002, 2024-02-19T21:10:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015189054, 2024-02-19T21:10:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22230", 2024-02-19T21:10:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1800034971", 2024-02-19T21:10:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:10:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185436+0000", 2024-02-19T21:10:05.323 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185436+0000", 2024-02-19T21:10:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:10:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185434+0000", 2024-02-19T21:10:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.024e-06 2024-02-19T21:10:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.324 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185441+0000", 2024-02-19T21:10:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6830000000000003e-06 2024-02-19T21:10:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185451+0000", 2024-02-19T21:10:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6630000000000004e-06 2024-02-19T21:10:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185458+0000", 2024-02-19T21:10:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00056233499999999996 2024-02-19T21:10:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.325 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.186020+0000", 2024-02-19T21:10:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1280000000000008e-06 2024-02-19T21:10:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:10:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.186028+0000", 2024-02-19T21:10:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.012894070000000001 2024-02-19T21:10:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.198922+0000", 2024-02-19T21:10:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012016800000000001 2024-02-19T21:10:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.326 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.199042+0000", 2024-02-19T21:10:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000105312 2024-02-19T21:10:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,6", 2024-02-19T21:10:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.199148+0000", 2024-02-19T21:10:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00055710799999999995 2024-02-19T21:10:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:10:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.199705+0000", 2024-02-19T21:10:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7489999999999999e-05 2024-02-19T21:10:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.327 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.199732+0000", 2024-02-19T21:10:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00081238399999999998 2024-02-19T21:10:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.200545+0000", 2024-02-19T21:10:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8712999999999999e-05 2024-02-19T21:10:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:10:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.200593+0000", 2024-02-19T21:10:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1940999999999998e-05 2024-02-19T21:10:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.328 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.200625+0000", 2024-02-19T21:10:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015189054 2024-02-19T21:10:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22254.0:2 104.9 104:90ca3134:::oid_FlushAsyncPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e157)", 2024-02-19T21:10:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:37.200018+0000", 2024-02-19T21:10:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 268.07344006900001, 2024-02-19T21:10:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0068597529999999997, 2024-02-19T21:10:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.329 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22254", 2024-02-19T21:10:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2521895994", 2024-02-19T21:10:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:10:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200018+0000", 2024-02-19T21:10:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200018+0000", 2024-02-19T21:10:05.330 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:10:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200015+0000", 2024-02-19T21:10:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8359999999999992e-06 2024-02-19T21:10:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200023+0000", 2024-02-19T21:10:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9250000000000007e-06 2024-02-19T21:10:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200033+0000", 2024-02-19T21:10:05.331 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7489999999999998e-06 2024-02-19T21:10:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200041+0000", 2024-02-19T21:10:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015734399999999999 2024-02-19T21:10:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200198+0000", 2024-02-19T21:10:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8997000000000002e-05 2024-02-19T21:10:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.332 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200287+0000", 2024-02-19T21:10:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000121697 2024-02-19T21:10:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 5,6", 2024-02-19T21:10:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200409+0000", 2024-02-19T21:10:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0042214710000000001 2024-02-19T21:10:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:10:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.204630+0000", 2024-02-19T21:10:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000147804 2024-02-19T21:10:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.333 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.204778+0000", 2024-02-19T21:10:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020029409999999998 2024-02-19T21:10:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.206781+0000", 2024-02-19T21:10:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5789999999999999e-05 2024-02-19T21:10:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:10:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.206837+0000", 2024-02-19T21:10:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0491999999999998e-05 2024-02-19T21:10:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.334 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.206877+0000", 2024-02-19T21:10:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0068597529999999997 2024-02-19T21:10:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.47779.0:2 114.13 114:cbb9903e:::test_obj:head [write-same] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-02-19T21:10:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:49.194644+0000", 2024-02-19T21:10:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 256.07881312299997, 2024-02-19T21:10:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.061315005999999998, 2024-02-19T21:10:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.335 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.47779", 2024-02-19T21:10:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/4153636191", 2024-02-19T21:10:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:10:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194644+0000", 2024-02-19T21:10:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194644+0000", 2024-02-19T21:10:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:10:05.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194643+0000", 2024-02-19T21:10:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8570000000000007e-06 2024-02-19T21:10:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194653+0000", 2024-02-19T21:10:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6340000000000007e-06 2024-02-19T21:10:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194661+0000", 2024-02-19T21:10:05.337 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7300000000000002e-06 2024-02-19T21:10:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194667+0000", 2024-02-19T21:10:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0865999999999998e-05 2024-02-19T21:10:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194708+0000", 2024-02-19T21:10:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3140000000000003e-06 2024-02-19T21:10:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:10:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194713+0000", 2024-02-19T21:10:05.338 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.020878119000000001 2024-02-19T21:10:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.215591+0000", 2024-02-19T21:10:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011031699999999999 2024-02-19T21:10:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.215702+0000", 2024-02-19T21:10:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000156618 2024-02-19T21:10:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,2", 2024-02-19T21:10:05.339 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.215858+0000", 2024-02-19T21:10:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0048208690000000002 2024-02-19T21:10:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:10:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.220679+0000", 2024-02-19T21:10:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.029705588000000002 2024-02-19T21:10:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.250385+0000", 2024-02-19T21:10:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0055153939999999999 2024-02-19T21:10:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.340 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.255900+0000", 2024-02-19T21:10:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1894999999999999e-05 2024-02-19T21:10:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:10:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.255932+0000", 2024-02-19T21:10:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7478e-05 2024-02-19T21:10:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.255959+0000", 2024-02-19T21:10:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.061315005999999998 2024-02-19T21:10:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.341 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.49298.0:1 162.6s0 162:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e237)", 2024-02-19T21:10:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:06:57.212999+0000", 2024-02-19T21:10:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 188.06045823900001, 2024-02-19T21:10:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.009269615, 2024-02-19T21:10:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.49298", 2024-02-19T21:10:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2330262446", 2024-02-19T21:10:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:10:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.342 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.212999+0000", 2024-02-19T21:10:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.212999+0000", 2024-02-19T21:10:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:10:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.212996+0000", 2024-02-19T21:10:05.343 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.08e-05 2024-02-19T21:10:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.213007+0000", 2024-02-19T21:10:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1018e-05 2024-02-19T21:10:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.213018+0000", 2024-02-19T21:10:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7819999999999994e-06 2024-02-19T21:10:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.344 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.213026+0000", 2024-02-19T21:10:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9827999999999996e-05 2024-02-19T21:10:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.213096+0000", 2024-02-19T21:10:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.399e-06 2024-02-19T21:10:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:10:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.213099+0000", 2024-02-19T21:10:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.006276073 2024-02-19T21:10:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.345 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.219375+0000", 2024-02-19T21:10:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012834399999999999 2024-02-19T21:10:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.219504+0000", 2024-02-19T21:10:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00046405399999999999 2024-02-19T21:10:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:10:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.219968+0000", 2024-02-19T21:10:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00075405700000000003 2024-02-19T21:10:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.346 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:10:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.220722+0000", 2024-02-19T21:10:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014760260000000001 2024-02-19T21:10:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:10:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.222198+0000", 2024-02-19T21:10:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1095999999999998e-05 2024-02-19T21:10:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.222269+0000", 2024-02-19T21:10:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.009269615 2024-02-19T21:10:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.347 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:2468 169.4 169:20000000::::head [pgnls start_epoch 259] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e259)", 2024-02-19T21:10:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:19.458185+0000", 2024-02-19T21:10:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 165.81527242300001, 2024-02-19T21:10:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0067992360000000002, 2024-02-19T21:10:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:10:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:10:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:10:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2468 2024-02-19T21:10:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.348 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458185+0000", 2024-02-19T21:10:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458185+0000", 2024-02-19T21:10:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999933 2024-02-19T21:10:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458179+0000", 2024-02-19T21:10:05.349 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1345e-05 2024-02-19T21:10:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458200+0000", 2024-02-19T21:10:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4885999999999999e-05 2024-02-19T21:10:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458225+0000", 2024-02-19T21:10:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0018e-05 2024-02-19T21:10:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458235+0000", 2024-02-19T21:10:05.350 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4640000000000003e-05 2024-02-19T21:10:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458320+0000", 2024-02-19T21:10:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6839999999999996e-06 2024-02-19T21:10:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:10:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458327+0000", 2024-02-19T21:10:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0065004249999999998 2024-02-19T21:10:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.464828+0000", 2024-02-19T21:10:05.351 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8716e-05 2024-02-19T21:10:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.464866+0000", 2024-02-19T21:10:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011796899999999999 2024-02-19T21:10:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.464984+0000", 2024-02-19T21:10:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0067992360000000002 2024-02-19T21:10:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:2918 178.14 178:28000000:%01:::head [pgnls start_epoch 278] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e278)", 2024-02-19T21:10:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:38.255394+0000", 2024-02-19T21:10:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.01806316299999, 2024-02-19T21:10:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.005781263, 2024-02-19T21:10:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:10:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:10:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:10:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2918 2024-02-19T21:10:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.353 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.255394+0000", 2024-02-19T21:10:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.255394+0000", 2024-02-19T21:10:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:10:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.255392+0000", 2024-02-19T21:10:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2310000000000002e-06 2024-02-19T21:10:05.354 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.255401+0000", 2024-02-19T21:10:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1428e-05 2024-02-19T21:10:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.255413+0000", 2024-02-19T21:10:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0010000000000004e-06 2024-02-19T21:10:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.255420+0000", 2024-02-19T21:10:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00019466700000000001 2024-02-19T21:10:05.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.255614+0000", 2024-02-19T21:10:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8669999999999999e-05 2024-02-19T21:10:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.255633+0000", 2024-02-19T21:10:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0055427599999999999 2024-02-19T21:10:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.261176+0000", 2024-02-19T21:10:05.356 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.005781263 2024-02-19T21:10:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.2.18:69 61.3 e294/114 61:c0520479:test-rados-api-smithi163-83928-38::foo:head v 294'69, mlcod=230'66)", 2024-02-19T21:10:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:54.263507+0000", 2024-02-19T21:10:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 131.009950277, 2024-02-19T21:10:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0078694200000000002, 2024-02-19T21:10:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.357 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.263507+0000", 2024-02-19T21:10:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.263507+0000", 2024-02-19T21:10:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967296 2024-02-19T21:10:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.263507+0000", 2024-02-19T21:10:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003686576 2024-02-19T21:10:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.358 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.267194+0000", 2024-02-19T21:10:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4829e-05 2024-02-19T21:10:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.267208+0000", 2024-02-19T21:10:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0240000000000003e-06 2024-02-19T21:10:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.267217+0000", 2024-02-19T21:10:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1632000000000003e-05 2024-02-19T21:10:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.359 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.360 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.360 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.267269+0000", 2024-02-19T21:10:05.360 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0282999999999999e-05 2024-02-19T21:10:05.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.360 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.360 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.360 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.267289+0000", 2024-02-19T21:10:05.360 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0040479779999999998 2024-02-19T21:10:05.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.360 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.360 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:10:05.360 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.271337+0000", 2024-02-19T21:10:05.360 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9335999999999997e-05 2024-02-19T21:10:05.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.361 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.361 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.361 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.271377+0000", 2024-02-19T21:10:05.361 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0078694200000000002 2024-02-19T21:10:05.361 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.361 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.361 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.361 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.361 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.361 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.24400.0:541 4.16 4:6fe6cc47:notif::queues_list_object:head [omap-get-keys] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e297)", 2024-02-19T21:10:05.361 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:57.226039+0000", 2024-02-19T21:10:05.363 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 128.04741852000001, 2024-02-19T21:10:05.363 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0064452540000000001, 2024-02-19T21:10:05.363 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.363 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-02-19T21:10:05.363 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.363 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.24400", 2024-02-19T21:10:05.363 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3997262145", 2024-02-19T21:10:05.363 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 541 2024-02-19T21:10:05.363 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.363 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.363 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.363 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.363 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:57.226039+0000", 2024-02-19T21:10:05.364 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.364 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.364 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.364 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:57.226039+0000", 2024-02-19T21:10:05.364 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:10:05.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.364 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.364 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.364 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:57.226036+0000", 2024-02-19T21:10:05.364 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2422000000000001e-05 2024-02-19T21:10:05.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.364 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.364 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.365 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:57.226048+0000", 2024-02-19T21:10:05.365 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2859e-05 2024-02-19T21:10:05.365 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.365 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.365 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.365 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:57.226061+0000", 2024-02-19T21:10:05.365 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3580000000000003e-06 2024-02-19T21:10:05.365 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.365 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.365 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.365 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:57.226070+0000", 2024-02-19T21:10:05.365 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.428e-05 2024-02-19T21:10:05.365 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.365 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.365 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.366 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:57.226144+0000", 2024-02-19T21:10:05.366 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3109999999999995e-06 2024-02-19T21:10:05.366 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.366 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.366 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:10:05.366 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:57.226153+0000", 2024-02-19T21:10:05.366 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0062423380000000001 2024-02-19T21:10:05.366 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.366 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.366 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.366 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:57.232396+0000", 2024-02-19T21:10:05.366 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8691999999999997e-05 2024-02-19T21:10:05.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.367 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.367 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.367 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:57.232484+0000", 2024-02-19T21:10:05.367 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0064452540000000001 2024-02-19T21:10:05.367 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.367 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.367 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.367 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.367 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:4348 191.8 191:10fb1d02:::199:head [delete] snapc 0=[] ondisk+write+ignore_cache+known_if_redirected e314)", 2024-02-19T21:10:05.367 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:08:14.258045+0000", 2024-02-19T21:10:05.367 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 111.015412617, 2024-02-19T21:10:05.367 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.039591451999999999, 2024-02-19T21:10:05.368 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.368 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:05.368 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.368 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:10:05.368 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:10:05.368 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 4348 2024-02-19T21:10:05.368 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.368 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.368 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.368 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.368 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258045+0000", 2024-02-19T21:10:05.368 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.368 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.368 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.368 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.369 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258045+0000", 2024-02-19T21:10:05.369 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:10:05.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.369 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.369 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258043+0000", 2024-02-19T21:10:05.369 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5199999999999997e-06 2024-02-19T21:10:05.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.369 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.369 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258049+0000", 2024-02-19T21:10:05.369 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6630000000000005e-06 2024-02-19T21:10:05.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.370 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.370 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258059+0000", 2024-02-19T21:10:05.370 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0889999999999999e-06 2024-02-19T21:10:05.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.370 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.370 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258064+0000", 2024-02-19T21:10:05.370 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5559999999999998e-05 2024-02-19T21:10:05.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.370 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.370 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258099+0000", 2024-02-19T21:10:05.370 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4418999999999994e-05 2024-02-19T21:10:05.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.371 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.371 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258164+0000", 2024-02-19T21:10:05.371 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1835000000000006e-05 2024-02-19T21:10:05.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.371 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,5", 2024-02-19T21:10:05.371 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258236+0000", 2024-02-19T21:10:05.371 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.021542664 2024-02-19T21:10:05.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.371 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:10:05.371 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.279778+0000", 2024-02-19T21:10:05.371 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025807920000000002 2024-02-19T21:10:05.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.372 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.372 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.282359+0000", 2024-02-19T21:10:05.372 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015223703 2024-02-19T21:10:05.372 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.372 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.372 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.372 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297583+0000", 2024-02-19T21:10:05.372 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0100999999999999e-05 2024-02-19T21:10:05.372 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.372 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.372 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:10:05.372 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297613+0000", 2024-02-19T21:10:05.372 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3626e-05 2024-02-19T21:10:05.372 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.373 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.373 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.373 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297636+0000", 2024-02-19T21:10:05.373 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.039591451999999999 2024-02-19T21:10:05.373 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.373 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.373 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.373 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.373 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:4349 191.8 191:10fdda93:::194:head [delete] snapc 0=[] ondisk+write+ignore_cache+known_if_redirected e314)", 2024-02-19T21:10:05.373 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:08:14.297868+0000", 2024-02-19T21:10:05.373 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 110.975589744, 2024-02-19T21:10:05.373 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026989016000000001, 2024-02-19T21:10:05.373 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.374 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:05.374 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.374 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:10:05.374 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:10:05.374 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 4349 2024-02-19T21:10:05.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.374 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.374 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.374 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.374 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297868+0000", 2024-02-19T21:10:05.374 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.374 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.374 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.374 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297868+0000", 2024-02-19T21:10:05.375 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:10:05.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.375 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.375 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297866+0000", 2024-02-19T21:10:05.375 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4129999999999997e-06 2024-02-19T21:10:05.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.375 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.375 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297871+0000", 2024-02-19T21:10:05.375 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0733999999999999e-05 2024-02-19T21:10:05.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.375 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.375 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297882+0000", 2024-02-19T21:10:05.376 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5829999999999997e-06 2024-02-19T21:10:05.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.376 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.376 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297888+0000", 2024-02-19T21:10:05.376 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4179e-05 2024-02-19T21:10:05.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.376 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.376 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297912+0000", 2024-02-19T21:10:05.376 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7067000000000002e-05 2024-02-19T21:10:05.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.376 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.377 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297959+0000", 2024-02-19T21:10:05.377 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9258999999999995e-05 2024-02-19T21:10:05.377 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.377 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.377 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,5", 2024-02-19T21:10:05.377 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.298028+0000", 2024-02-19T21:10:05.377 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00027415799999999999 2024-02-19T21:10:05.377 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.377 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.377 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:10:05.377 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.298302+0000", 2024-02-19T21:10:05.377 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018730600000000001 2024-02-19T21:10:05.377 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.377 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.377 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.378 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.298490+0000", 2024-02-19T21:10:05.378 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026316253000000001 2024-02-19T21:10:05.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.378 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.378 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:05.378 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.324806+0000", 2024-02-19T21:10:05.378 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9e-05 2024-02-19T21:10:05.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.378 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.378 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:10:05.378 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.324835+0000", 2024-02-19T21:10:05.378 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1923999999999999e-05 2024-02-19T21:10:05.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.378 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.378 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.379 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.324857+0000", 2024-02-19T21:10:05.379 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026989016000000001 2024-02-19T21:10:05.379 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.379 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.379 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.379 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.379 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22980.0:1 205.3s0 205:cbb9903e:::test_obj:head [omap-set-vals,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected e373)", 2024-02-19T21:10:05.379 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:09:13.274270+0000", 2024-02-19T21:10:05.379 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 51.999187120999999, 2024-02-19T21:10:05.379 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.581927844, 2024-02-19T21:10:05.379 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.379 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:10:05.379 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.379 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22980", 2024-02-19T21:10:05.380 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2118002540", 2024-02-19T21:10:05.380 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:10:05.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.380 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.380 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.380 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.380 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274270+0000", 2024-02-19T21:10:05.380 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.380 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.380 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.380 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274270+0000", 2024-02-19T21:10:05.380 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:10:05.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.381 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.381 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274267+0000", 2024-02-19T21:10:05.381 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1556000000000001e-05 2024-02-19T21:10:05.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.381 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.381 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274279+0000", 2024-02-19T21:10:05.381 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1654000000000001e-05 2024-02-19T21:10:05.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.381 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.381 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274291+0000", 2024-02-19T21:10:05.381 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7020000000000007e-06 2024-02-19T21:10:05.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.382 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.382 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.382 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274299+0000", 2024-02-19T21:10:05.382 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014805800000000001 2024-02-19T21:10:05.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.382 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.382 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.382 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274447+0000", 2024-02-19T21:10:05.382 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0749999999999999e-06 2024-02-19T21:10:05.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.382 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.382 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:10:05.382 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274452+0000", 2024-02-19T21:10:05.382 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5792793839999999 2024-02-19T21:10:05.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.383 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.383 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.383 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:14.853731+0000", 2024-02-19T21:10:05.383 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000125561 2024-02-19T21:10:05.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.383 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.383 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.383 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:14.853856+0000", 2024-02-19T21:10:05.383 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0023418330000000002 2024-02-19T21:10:05.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.383 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.383 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.383 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:14.856198+0000", 2024-02-19T21:10:05.383 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.581927844 2024-02-19T21:10:05.384 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.384 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.384 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.384 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.384 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:5600 208.10 208:08000000:%01:::head [pgnls start_epoch 393] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e393)", 2024-02-19T21:10:05.384 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:09:36.343158+0000", 2024-02-19T21:10:05.384 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 28.930299325, 2024-02-19T21:10:05.384 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.338287899, 2024-02-19T21:10:05.384 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:10:05.384 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:10:05.384 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:10:05.384 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:10:05.384 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:10:05.385 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 5600 2024-02-19T21:10:05.385 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.385 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:10:05.385 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.385 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:10:05.385 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343158+0000", 2024-02-19T21:10:05.385 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:10:05.385 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.385 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.385 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:10:05.385 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343158+0000", 2024-02-19T21:10:05.385 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:10:05.385 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.385 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.386 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:10:05.386 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343157+0000", 2024-02-19T21:10:05.386 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9830000000000001e-06 2024-02-19T21:10:05.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.386 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.386 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:10:05.386 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343163+0000", 2024-02-19T21:10:05.386 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4930000000000004e-06 2024-02-19T21:10:05.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.386 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.386 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:10:05.386 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343172+0000", 2024-02-19T21:10:05.386 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2700000000000004e-06 2024-02-19T21:10:05.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.386 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.387 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:10:05.387 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343178+0000", 2024-02-19T21:10:05.387 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3643000000000002e-05 2024-02-19T21:10:05.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.387 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.387 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343221+0000", 2024-02-19T21:10:05.387 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6779999999999997e-06 2024-02-19T21:10:05.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.387 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:10:05.387 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343227+0000", 2024-02-19T21:10:05.387 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.33795772099999999 2024-02-19T21:10:05.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.388 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:10:05.388 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.681185+0000", 2024-02-19T21:10:05.388 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0336000000000001e-05 2024-02-19T21:10:05.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.388 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.388 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:10:05.388 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.681235+0000", 2024-02-19T21:10:05.388 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00021113999999999999 2024-02-19T21:10:05.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:10:05.388 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:10:05.388 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:10:05.388 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.681446+0000", 2024-02-19T21:10:05.388 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.338287899 2024-02-19T21:10:05.388 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.388 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.389 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.389 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:10:05.389 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:10:05.389 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:10:05.390 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:05.391 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":596,"stamp":"2024-02-19T21:10:03.209110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007304,"num_objects":357,"num_object_clones":4,"num_object_copies":1071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":1332,"num_read_kb":983,"num_write":1040,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":713,"num_bytes_recovered":7009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2386,"ondisk_log_size":2386,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":684,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2179504,"kb_used_data":57932,"kb_used_omap":0,"kb_used_meta":2121472,"kb_avail":654036560,"statfs":{"total":671965249536,"available":669733437440,"internally_reserved":0,"allocated":59322368,"data_stored":38760678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2172387328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002300"},"pg_stats":[{"pgid":"212.1f","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.274184+0000","last_change":"2024-02-19T21:09:49.280368+0000","last_active":"2024-02-19T21:09:54.274184+0000","last_peered":"2024-02-19T21:09:54.274184+0000","last_clean":"2024-02-19T21:09:54.274184+0000","last_became_active":"2024-02-19T21:09:49.280113+0000","last_became_peered":"2024-02-19T21:09:49.280113+0000","last_unstale":"2024-02-19T21:09:54.274184+0000","last_undegraded":"2024-02-19T21:09:54.274184+0000","last_fullsized":"2024-02-19T21:09:54.274184+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.1e","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.273770+0000","last_change":"2024-02-19T21:09:49.283981+0000","last_active":"2024-02-19T21:09:54.273770+0000","last_peered":"2024-02-19T21:09:54.273770+0000","last_clean":"2024-02-19T21:09:54.273770+0000","last_became_active":"2024-02-19T21:09:49.283490+0000","last_became_peered":"2024-02-19T21:09:49.283490+0000","last_unstale":"2024-02-19T21:09:54.273770+0000","last_undegraded":"2024-02-19T21:09:54.273770+0000","last_fullsized":"2024-02-19T21:09:54.273770+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.1d","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.280815+0000","last_change":"2024-02-19T21:09:49.279223+0000","last_active":"2024-02-19T21:09:54.280815+0000","last_peered":"2024-02-19T21:09:54.280815+0000","last_clean":"2024-02-19T21:09:54.280815+0000","last_became_active":"2024-02-19T21:09:49.278912+0000","last_became_peered":"2024-02-19T21:09:49.278912+0000","last_unstale":"2024-02-19T21:09:54.280815+0000","last_undegraded":"2024-02-19T21:09:54.280815+0000","last_fullsized":"2024-02-19T21:09:54.280815+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.1c","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446958+0000","last_change":"2024-02-19T21:09:49.273516+0000","last_active":"2024-02-19T21:09:54.446958+0000","last_peered":"2024-02-19T21:09:54.446958+0000","last_clean":"2024-02-19T21:09:54.446958+0000","last_became_active":"2024-02-19T21:09:49.273302+0000","last_became_peered":"2024-02-19T21:09:49.273302+0000","last_unstale":"2024-02-19T21:09:54.446958+0000","last_undegraded":"2024-02-19T21:09:54.446958+0000","last_fullsized":"2024-02-19T21:09:54.446958+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.1b","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446338+0000","last_change":"2024-02-19T21:09:49.269667+0000","last_active":"2024-02-19T21:09:54.446338+0000","last_peered":"2024-02-19T21:09:54.446338+0000","last_clean":"2024-02-19T21:09:54.446338+0000","last_became_active":"2024-02-19T21:09:49.269397+0000","last_became_peered":"2024-02-19T21:09:49.269397+0000","last_unstale":"2024-02-19T21:09:54.446338+0000","last_undegraded":"2024-02-19T21:09:54.446338+0000","last_fullsized":"2024-02-19T21:09:54.446338+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.1a","version":"410'3","reported_seq":18,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446192+0000","last_change":"2024-02-19T21:09:49.268123+0000","last_active":"2024-02-19T21:09:54.446192+0000","last_peered":"2024-02-19T21:09:54.446192+0000","last_clean":"2024-02-19T21:09:54.446192+0000","last_became_active":"2024-02-19T21:09:49.267619+0000","last_became_peered":"2024-02-19T21:09:49.267619+0000","last_unstale":"2024-02-19T21:09:54.446192+0000","last_undegraded":"2024-02-19T21:09:54.446192+0000","last_fullsized":"2024-02-19T21:09:54.446192+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.19","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268010+0000","last_change":"2024-02-19T21:09:49.276718+0000","last_active":"2024-02-19T21:09:54.268010+0000","last_peered":"2024-02-19T21:09:54.268010+0000","last_clean":"2024-02-19T21:09:54.268010+0000","last_became_active":"2024-02-19T21:09:49.276431+0000","last_became_peered":"2024-02-19T21:09:49.276431+0000","last_unstale":"2024-02-19T21:09:54.268010+0000","last_undegraded":"2024-02-19T21:09:54.268010+0000","last_fullsized":"2024-02-19T21:09:54.268010+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"212.18","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.276337+0000","last_change":"2024-02-19T21:09:49.269923+0000","last_active":"2024-02-19T21:09:54.276337+0000","last_peered":"2024-02-19T21:09:54.276337+0000","last_clean":"2024-02-19T21:09:54.276337+0000","last_became_active":"2024-02-19T21:09:49.269452+0000","last_became_peered":"2024-02-19T21:09:49.269452+0000","last_unstale":"2024-02-19T21:09:54.276337+0000","last_undegraded":"2024-02-19T21:09:54.276337+0000","last_fullsized":"2024-02-19T21:09:54.276337+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.17","version":"407'2","reported_seq":18,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.282212+0000","last_change":"2024-02-19T21:09:49.278385+0000","last_active":"2024-02-19T21:09:54.282212+0000","last_peered":"2024-02-19T21:09:54.282212+0000","last_clean":"2024-02-19T21:09:54.282212+0000","last_became_active":"2024-02-19T21:09:49.277932+0000","last_became_peered":"2024-02-19T21:09:49.277932+0000","last_unstale":"2024-02-19T21:09:54.282212+0000","last_undegraded":"2024-02-19T21:09:54.282212+0000","last_fullsized":"2024-02-19T21:09:54.282212+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.16","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270226+0000","last_change":"2024-02-19T21:09:49.276757+0000","last_active":"2024-02-19T21:09:54.270226+0000","last_peered":"2024-02-19T21:09:54.270226+0000","last_clean":"2024-02-19T21:09:54.270226+0000","last_became_active":"2024-02-19T21:09:49.276453+0000","last_became_peered":"2024-02-19T21:09:49.276453+0000","last_unstale":"2024-02-19T21:09:54.270226+0000","last_undegraded":"2024-02-19T21:09:54.270226+0000","last_fullsized":"2024-02-19T21:09:54.270226+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"212.15","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447036+0000","last_change":"2024-02-19T21:09:49.274158+0000","last_active":"2024-02-19T21:09:54.447036+0000","last_peered":"2024-02-19T21:09:54.447036+0000","last_clean":"2024-02-19T21:09:54.447036+0000","last_became_active":"2024-02-19T21:09:49.273893+0000","last_became_peered":"2024-02-19T21:09:49.273893+0000","last_unstale":"2024-02-19T21:09:54.447036+0000","last_undegraded":"2024-02-19T21:09:54.447036+0000","last_fullsized":"2024-02-19T21:09:54.447036+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.14","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270831+0000","last_change":"2024-02-19T21:09:49.287989+0000","last_active":"2024-02-19T21:09:54.270831+0000","last_peered":"2024-02-19T21:09:54.270831+0000","last_clean":"2024-02-19T21:09:54.270831+0000","last_became_active":"2024-02-19T21:09:49.287532+0000","last_became_peered":"2024-02-19T21:09:49.287532+0000","last_unstale":"2024-02-19T21:09:54.270831+0000","last_undegraded":"2024-02-19T21:09:54.270831+0000","last_fullsized":"2024-02-19T21:09:54.270831+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"212.13","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446167+0000","last_change":"2024-02-19T21:09:49.278147+0000","last_active":"2024-02-19T21:09:54.446167+0000","last_peered":"2024-02-19T21:09:54.446167+0000","last_clean":"2024-02-19T21:09:54.446167+0000","last_became_active":"2024-02-19T21:09:49.277829+0000","last_became_peered":"2024-02-19T21:09:49.277829+0000","last_unstale":"2024-02-19T21:09:54.446167+0000","last_undegraded":"2024-02-19T21:09:54.446167+0000","last_fullsized":"2024-02-19T21:09:54.446167+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"212.12","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446395+0000","last_change":"2024-02-19T21:09:49.273451+0000","last_active":"2024-02-19T21:09:54.446395+0000","last_peered":"2024-02-19T21:09:54.446395+0000","last_clean":"2024-02-19T21:09:54.446395+0000","last_became_active":"2024-02-19T21:09:49.273142+0000","last_became_peered":"2024-02-19T21:09:49.273142+0000","last_unstale":"2024-02-19T21:09:54.446395+0000","last_undegraded":"2024-02-19T21:09:54.446395+0000","last_fullsized":"2024-02-19T21:09:54.446395+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.11","version":"411'4","reported_seq":22,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.276647+0000","last_change":"2024-02-19T21:09:49.283943+0000","last_active":"2024-02-19T21:09:54.276647+0000","last_peered":"2024-02-19T21:09:54.276647+0000","last_clean":"2024-02-19T21:09:54.276647+0000","last_became_active":"2024-02-19T21:09:49.283453+0000","last_became_peered":"2024-02-19T21:09:49.283453+0000","last_unstale":"2024-02-19T21:09:54.276647+0000","last_undegraded":"2024-02-19T21:09:54.276647+0000","last_fullsized":"2024-02-19T21:09:54.276647+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.10","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.276352+0000","last_change":"2024-02-19T21:09:49.277056+0000","last_active":"2024-02-19T21:09:54.276352+0000","last_peered":"2024-02-19T21:09:54.276352+0000","last_clean":"2024-02-19T21:09:54.276352+0000","last_became_active":"2024-02-19T21:09:49.276708+0000","last_became_peered":"2024-02-19T21:09:49.276708+0000","last_unstale":"2024-02-19T21:09:54.276352+0000","last_undegraded":"2024-02-19T21:09:54.276352+0000","last_fullsized":"2024-02-19T21:09:54.276352+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.f","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.282191+0000","last_change":"2024-02-19T21:09:49.278210+0000","last_active":"2024-02-19T21:09:54.282191+0000","last_peered":"2024-02-19T21:09:54.282191+0000","last_clean":"2024-02-19T21:09:54.282191+0000","last_became_active":"2024-02-19T21:09:49.277549+0000","last_became_peered":"2024-02-19T21:09:49.277549+0000","last_unstale":"2024-02-19T21:09:54.282191+0000","last_undegraded":"2024-02-19T21:09:54.282191+0000","last_fullsized":"2024-02-19T21:09:54.282191+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.e","version":"407'1","reported_seq":16,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.273863+0000","last_change":"2024-02-19T21:09:49.283811+0000","last_active":"2024-02-19T21:09:54.273863+0000","last_peered":"2024-02-19T21:09:54.273863+0000","last_clean":"2024-02-19T21:09:54.273863+0000","last_became_active":"2024-02-19T21:09:49.283455+0000","last_became_peered":"2024-02-19T21:09:49.283455+0000","last_unstale":"2024-02-19T21:09:54.273863+0000","last_undegraded":"2024-02-19T21:09:54.273863+0000","last_fullsized":"2024-02-19T21:09:54.273863+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.d","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268783+0000","last_change":"2024-02-19T21:09:49.279144+0000","last_active":"2024-02-19T21:09:54.268783+0000","last_peered":"2024-02-19T21:09:54.268783+0000","last_clean":"2024-02-19T21:09:54.268783+0000","last_became_active":"2024-02-19T21:09:49.278831+0000","last_became_peered":"2024-02-19T21:09:49.278831+0000","last_unstale":"2024-02-19T21:09:54.268783+0000","last_undegraded":"2024-02-19T21:09:54.268783+0000","last_fullsized":"2024-02-19T21:09:54.268783+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"212.c","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.273737+0000","last_change":"2024-02-19T21:09:49.268676+0000","last_active":"2024-02-19T21:09:54.273737+0000","last_peered":"2024-02-19T21:09:54.273737+0000","last_clean":"2024-02-19T21:09:54.273737+0000","last_became_active":"2024-02-19T21:09:49.268398+0000","last_became_peered":"2024-02-19T21:09:49.268398+0000","last_unstale":"2024-02-19T21:09:54.273737+0000","last_undegraded":"2024-02-19T21:09:54.273737+0000","last_fullsized":"2024-02-19T21:09:54.273737+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.b","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.273499+0000","last_change":"2024-02-19T21:09:49.278115+0000","last_active":"2024-02-19T21:09:54.273499+0000","last_peered":"2024-02-19T21:09:54.273499+0000","last_clean":"2024-02-19T21:09:54.273499+0000","last_became_active":"2024-02-19T21:09:49.277842+0000","last_became_peered":"2024-02-19T21:09:49.277842+0000","last_unstale":"2024-02-19T21:09:54.273499+0000","last_undegraded":"2024-02-19T21:09:54.273499+0000","last_fullsized":"2024-02-19T21:09:54.273499+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.a","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.279686+0000","last_change":"2024-02-19T21:09:49.268726+0000","last_active":"2024-02-19T21:09:54.279686+0000","last_peered":"2024-02-19T21:09:54.279686+0000","last_clean":"2024-02-19T21:09:54.279686+0000","last_became_active":"2024-02-19T21:09:49.268352+0000","last_became_peered":"2024-02-19T21:09:49.268352+0000","last_unstale":"2024-02-19T21:09:54.279686+0000","last_undegraded":"2024-02-19T21:09:54.279686+0000","last_fullsized":"2024-02-19T21:09:54.279686+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"212.9","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446117+0000","last_change":"2024-02-19T21:09:49.273414+0000","last_active":"2024-02-19T21:09:54.446117+0000","last_peered":"2024-02-19T21:09:54.446117+0000","last_clean":"2024-02-19T21:09:54.446117+0000","last_became_active":"2024-02-19T21:09:49.273100+0000","last_became_peered":"2024-02-19T21:09:49.273100+0000","last_unstale":"2024-02-19T21:09:54.446117+0000","last_undegraded":"2024-02-19T21:09:54.446117+0000","last_fullsized":"2024-02-19T21:09:54.446117+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"212.8","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.267998+0000","last_change":"2024-02-19T21:09:49.286754+0000","last_active":"2024-02-19T21:09:54.267998+0000","last_peered":"2024-02-19T21:09:54.267998+0000","last_clean":"2024-02-19T21:09:54.267998+0000","last_became_active":"2024-02-19T21:09:49.286432+0000","last_became_peered":"2024-02-19T21:09:49.286432+0000","last_unstale":"2024-02-19T21:09:54.267998+0000","last_undegraded":"2024-02-19T21:09:54.267998+0000","last_fullsized":"2024-02-19T21:09:54.267998+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"212.7","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.274270+0000","last_change":"2024-02-19T21:09:49.272287+0000","last_active":"2024-02-19T21:09:54.274270+0000","last_peered":"2024-02-19T21:09:54.274270+0000","last_clean":"2024-02-19T21:09:54.274270+0000","last_became_active":"2024-02-19T21:09:49.271942+0000","last_became_peered":"2024-02-19T21:09:49.271942+0000","last_unstale":"2024-02-19T21:09:54.274270+0000","last_undegraded":"2024-02-19T21:09:54.274270+0000","last_fullsized":"2024-02-19T21:09:54.274270+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.6","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.273945+0000","last_change":"2024-02-19T21:09:49.284037+0000","last_active":"2024-02-19T21:09:54.273945+0000","last_peered":"2024-02-19T21:09:54.273945+0000","last_clean":"2024-02-19T21:09:54.273945+0000","last_became_active":"2024-02-19T21:09:49.283823+0000","last_became_peered":"2024-02-19T21:09:49.283823+0000","last_unstale":"2024-02-19T21:09:54.273945+0000","last_undegraded":"2024-02-19T21:09:54.273945+0000","last_fullsized":"2024-02-19T21:09:54.273945+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.5","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446891+0000","last_change":"2024-02-19T21:09:49.283803+0000","last_active":"2024-02-19T21:09:54.446891+0000","last_peered":"2024-02-19T21:09:54.446891+0000","last_clean":"2024-02-19T21:09:54.446891+0000","last_became_active":"2024-02-19T21:09:49.283499+0000","last_became_peered":"2024-02-19T21:09:49.283499+0000","last_unstale":"2024-02-19T21:09:54.446891+0000","last_undegraded":"2024-02-19T21:09:54.446891+0000","last_fullsized":"2024-02-19T21:09:54.446891+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"212.4","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270905+0000","last_change":"2024-02-19T21:09:49.287819+0000","last_active":"2024-02-19T21:09:54.270905+0000","last_peered":"2024-02-19T21:09:54.270905+0000","last_clean":"2024-02-19T21:09:54.270905+0000","last_became_active":"2024-02-19T21:09:49.287128+0000","last_became_peered":"2024-02-19T21:09:49.287128+0000","last_unstale":"2024-02-19T21:09:54.270905+0000","last_undegraded":"2024-02-19T21:09:54.270905+0000","last_fullsized":"2024-02-19T21:09:54.270905+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"212.3","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269068+0000","last_change":"2024-02-19T21:09:49.276495+0000","last_active":"2024-02-19T21:09:54.269068+0000","last_peered":"2024-02-19T21:09:54.269068+0000","last_clean":"2024-02-19T21:09:54.269068+0000","last_became_active":"2024-02-19T21:09:49.276206+0000","last_became_peered":"2024-02-19T21:09:49.276206+0000","last_unstale":"2024-02-19T21:09:54.269068+0000","last_undegraded":"2024-02-19T21:09:54.269068+0000","last_fullsized":"2024-02-19T21:09:54.269068+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"212.2","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446032+0000","last_change":"2024-02-19T21:09:49.275792+0000","last_active":"2024-02-19T21:09:54.446032+0000","last_peered":"2024-02-19T21:09:54.446032+0000","last_clean":"2024-02-19T21:09:54.446032+0000","last_became_active":"2024-02-19T21:09:49.275500+0000","last_became_peered":"2024-02-19T21:09:49.275500+0000","last_unstale":"2024-02-19T21:09:54.446032+0000","last_undegraded":"2024-02-19T21:09:54.446032+0000","last_fullsized":"2024-02-19T21:09:54.446032+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"212.1","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.275133+0000","last_change":"2024-02-19T21:09:49.278031+0000","last_active":"2024-02-19T21:09:54.275133+0000","last_peered":"2024-02-19T21:09:54.275133+0000","last_clean":"2024-02-19T21:09:54.275133+0000","last_became_active":"2024-02-19T21:09:49.277749+0000","last_became_peered":"2024-02-19T21:09:49.277749+0000","last_unstale":"2024-02-19T21:09:54.275133+0000","last_undegraded":"2024-02-19T21:09:54.275133+0000","last_fullsized":"2024-02-19T21:09:54.275133+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"212.0","version":"0'0","reported_seq":15,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446465+0000","last_change":"2024-02-19T21:09:49.268926+0000","last_active":"2024-02-19T21:09:54.446465+0000","last_peered":"2024-02-19T21:09:54.446465+0000","last_clean":"2024-02-19T21:09:54.446465+0000","last_became_active":"2024-02-19T21:09:49.268626+0000","last_became_peered":"2024-02-19T21:09:49.268626+0000","last_unstale":"2024-02-19T21:09:54.446465+0000","last_undegraded":"2024-02-19T21:09:54.446465+0000","last_fullsized":"2024-02-19T21:09:54.446465+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:48.261871+0000","last_clean_scrub_stamp":"2024-02-19T21:09:48.261871+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.2","version":"355'85","reported_seq":185,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.271142+0000","last_change":"2024-02-19T21:09:54.271142+0000","last_active":"2024-02-19T21:09:54.271142+0000","last_peered":"2024-02-19T21:09:54.271142+0000","last_clean":"2024-02-19T21:09:54.271142+0000","last_became_active":"2024-02-19T21:08:04.247600+0000","last_became_peered":"2024-02-19T21:08:04.247600+0000","last_unstale":"2024-02-19T21:09:54.271142+0000","last_undegraded":"2024-02-19T21:09:54.271142+0000","last_fullsized":"2024-02-19T21:09:54.271142+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'85","last_scrub_stamp":"2024-02-19T21:08:55.882988+0000","last_deep_scrub":"355'85","last_deep_scrub_stamp":"2024-02-19T21:08:55.882988+0000","last_clean_scrub_stamp":"2024-02-19T21:08:55.882988+0000","log_size":85,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.3","version":"338'71","reported_seq":416,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447662+0000","last_change":"2024-02-19T21:09:54.447662+0000","last_active":"2024-02-19T21:09:54.447662+0000","last_peered":"2024-02-19T21:09:54.447662+0000","last_clean":"2024-02-19T21:09:54.447662+0000","last_became_active":"2024-02-19T21:04:54.641884+0000","last_became_peered":"2024-02-19T21:04:54.641884+0000","last_unstale":"2024-02-19T21:09:54.447662+0000","last_undegraded":"2024-02-19T21:09:54.447662+0000","last_fullsized":"2024-02-19T21:09:54.447662+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'69","last_scrub_stamp":"2024-02-19T21:08:00.867237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:00.867237+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000068,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":23,"num_write_kb":3929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.0","version":"257'35","reported_seq":363,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.448846+0000","last_change":"2024-02-19T21:09:54.448846+0000","last_active":"2024-02-19T21:09:54.448846+0000","last_peered":"2024-02-19T21:09:54.448846+0000","last_clean":"2024-02-19T21:09:54.448846+0000","last_became_active":"2024-02-19T21:06:40.245157+0000","last_became_peered":"2024-02-19T21:06:40.245157+0000","last_unstale":"2024-02-19T21:09:54.448846+0000","last_undegraded":"2024-02-19T21:09:54.448846+0000","last_fullsized":"2024-02-19T21:09:54.448846+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'35","last_scrub_stamp":"2024-02-19T21:08:55.692313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:55.692313+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.1","version":"338'69","reported_seq":468,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447045+0000","last_change":"2024-02-19T21:09:54.447045+0000","last_active":"2024-02-19T21:09:54.447045+0000","last_peered":"2024-02-19T21:09:54.447045+0000","last_clean":"2024-02-19T21:09:54.447045+0000","last_became_active":"2024-02-19T21:08:41.312701+0000","last_became_peered":"2024-02-19T21:08:41.312701+0000","last_unstale":"2024-02-19T21:09:54.447045+0000","last_undegraded":"2024-02-19T21:09:54.447045+0000","last_fullsized":"2024-02-19T21:09:54.447045+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'68","last_scrub_stamp":"2024-02-19T21:06:56.570995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:06:56.570995+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.6","version":"202'34","reported_seq":381,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.284645+0000","last_change":"2024-02-19T21:09:54.284645+0000","last_active":"2024-02-19T21:09:54.284645+0000","last_peered":"2024-02-19T21:09:54.284645+0000","last_clean":"2024-02-19T21:09:54.284645+0000","last_became_active":"2024-02-19T21:04:54.658542+0000","last_became_peered":"2024-02-19T21:04:54.658542+0000","last_unstale":"2024-02-19T21:09:54.284645+0000","last_undegraded":"2024-02-19T21:09:54.284645+0000","last_fullsized":"2024-02-19T21:09:54.284645+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'34","last_scrub_stamp":"2024-02-19T21:08:58.036955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:58.036955+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.7","version":"396'20","reported_seq":376,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270075+0000","last_change":"2024-02-19T21:09:54.270075+0000","last_active":"2024-02-19T21:09:54.270075+0000","last_peered":"2024-02-19T21:09:54.270075+0000","last_clean":"2024-02-19T21:09:54.270075+0000","last_became_active":"2024-02-19T21:04:54.598061+0000","last_became_peered":"2024-02-19T21:04:54.598061+0000","last_unstale":"2024-02-19T21:09:54.270075+0000","last_undegraded":"2024-02-19T21:09:54.270075+0000","last_fullsized":"2024-02-19T21:09:54.270075+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'18","last_scrub_stamp":"2024-02-19T21:08:03.459480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:03.459480+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.4","version":"403'70","reported_seq":465,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.284356+0000","last_change":"2024-02-19T21:09:54.284356+0000","last_active":"2024-02-19T21:09:54.284356+0000","last_peered":"2024-02-19T21:09:54.284356+0000","last_clean":"2024-02-19T21:09:54.284356+0000","last_became_active":"2024-02-19T21:08:41.286273+0000","last_became_peered":"2024-02-19T21:08:41.286273+0000","last_unstale":"2024-02-19T21:09:54.284356+0000","last_undegraded":"2024-02-19T21:09:54.284356+0000","last_fullsized":"2024-02-19T21:09:54.284356+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"206'68","last_scrub_stamp":"2024-02-19T21:07:35.203992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:07:35.203992+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.5","version":"338'121","reported_seq":454,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.449010+0000","last_change":"2024-02-19T21:09:54.449010+0000","last_active":"2024-02-19T21:09:54.449010+0000","last_peered":"2024-02-19T21:09:54.449010+0000","last_clean":"2024-02-19T21:09:54.449010+0000","last_became_active":"2024-02-19T21:04:54.646842+0000","last_became_peered":"2024-02-19T21:04:54.646842+0000","last_unstale":"2024-02-19T21:09:54.449010+0000","last_undegraded":"2024-02-19T21:09:54.449010+0000","last_fullsized":"2024-02-19T21:09:54.449010+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'121","last_scrub_stamp":"2024-02-19T21:08:49.752418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.752418+0000","log_size":121,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.a","version":"236'22","reported_seq":167,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.271322+0000","last_change":"2024-02-19T21:09:54.271322+0000","last_active":"2024-02-19T21:09:54.271322+0000","last_peered":"2024-02-19T21:09:54.271322+0000","last_clean":"2024-02-19T21:09:54.271322+0000","last_became_active":"2024-02-19T21:08:41.297020+0000","last_became_peered":"2024-02-19T21:08:41.297020+0000","last_unstale":"2024-02-19T21:09:54.271322+0000","last_undegraded":"2024-02-19T21:09:54.271322+0000","last_fullsized":"2024-02-19T21:09:54.271322+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-02-19T21:08:01.607121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:01.607121+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":64,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.b","version":"278'95","reported_seq":435,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447879+0000","last_change":"2024-02-19T21:09:54.447879+0000","last_active":"2024-02-19T21:09:54.447879+0000","last_peered":"2024-02-19T21:09:54.447879+0000","last_clean":"2024-02-19T21:09:54.447879+0000","last_became_active":"2024-02-19T21:04:54.640024+0000","last_became_peered":"2024-02-19T21:04:54.640024+0000","last_unstale":"2024-02-19T21:09:54.447879+0000","last_undegraded":"2024-02-19T21:09:54.447879+0000","last_fullsized":"2024-02-19T21:09:54.447879+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'87","last_scrub_stamp":"2024-02-19T21:07:28.724335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:07:28.724335+0000","log_size":95,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.8","version":"399'39","reported_seq":449,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268765+0000","last_change":"2024-02-19T21:09:54.268765+0000","last_active":"2024-02-19T21:09:54.268765+0000","last_peered":"2024-02-19T21:09:54.268765+0000","last_clean":"2024-02-19T21:09:54.268765+0000","last_became_active":"2024-02-19T21:08:04.266021+0000","last_became_peered":"2024-02-19T21:08:04.266021+0000","last_unstale":"2024-02-19T21:09:54.268765+0000","last_undegraded":"2024-02-19T21:09:54.268765+0000","last_fullsized":"2024-02-19T21:09:54.268765+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'35","last_scrub_stamp":"2024-02-19T21:08:01.494484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:01.494484+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":50,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.9","version":"403'71","reported_seq":419,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447341+0000","last_change":"2024-02-19T21:09:54.447341+0000","last_active":"2024-02-19T21:09:54.447341+0000","last_peered":"2024-02-19T21:09:54.447341+0000","last_clean":"2024-02-19T21:09:54.447341+0000","last_became_active":"2024-02-19T21:04:54.671646+0000","last_became_peered":"2024-02-19T21:04:54.671646+0000","last_unstale":"2024-02-19T21:09:54.447341+0000","last_undegraded":"2024-02-19T21:09:54.447341+0000","last_fullsized":"2024-02-19T21:09:54.447341+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'68","last_scrub_stamp":"2024-02-19T21:07:31.724135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:07:31.724135+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.e","version":"393'40","reported_seq":427,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.448037+0000","last_change":"2024-02-19T21:09:54.448037+0000","last_active":"2024-02-19T21:09:54.448037+0000","last_peered":"2024-02-19T21:09:54.448037+0000","last_clean":"2024-02-19T21:09:54.448037+0000","last_became_active":"2024-02-19T21:08:41.280291+0000","last_became_peered":"2024-02-19T21:08:41.280291+0000","last_unstale":"2024-02-19T21:09:54.448037+0000","last_undegraded":"2024-02-19T21:09:54.448037+0000","last_fullsized":"2024-02-19T21:09:54.448037+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'34","last_scrub_stamp":"2024-02-19T21:06:59.674445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:06:59.674445+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.f","version":"399'37","reported_seq":420,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.276366+0000","last_change":"2024-02-19T21:09:54.276366+0000","last_active":"2024-02-19T21:09:54.276366+0000","last_peered":"2024-02-19T21:09:54.276366+0000","last_clean":"2024-02-19T21:09:54.276366+0000","last_became_active":"2024-02-19T21:08:41.303778+0000","last_became_peered":"2024-02-19T21:08:41.303778+0000","last_unstale":"2024-02-19T21:09:54.276366+0000","last_undegraded":"2024-02-19T21:09:54.276366+0000","last_fullsized":"2024-02-19T21:09:54.276366+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'34","last_scrub_stamp":"2024-02-19T21:06:58.720103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:06:58.720103+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.c","version":"249'73","reported_seq":578,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270914+0000","last_change":"2024-02-19T21:09:54.270914+0000","last_active":"2024-02-19T21:09:54.270914+0000","last_peered":"2024-02-19T21:09:54.270914+0000","last_clean":"2024-02-19T21:09:54.270914+0000","last_became_active":"2024-02-19T21:08:41.280328+0000","last_became_peered":"2024-02-19T21:08:41.280328+0000","last_unstale":"2024-02-19T21:09:54.270914+0000","last_undegraded":"2024-02-19T21:09:54.270914+0000","last_fullsized":"2024-02-19T21:09:54.270914+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'73","last_scrub_stamp":"2024-02-19T21:08:49.613462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.613462+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.d","version":"393'39","reported_seq":390,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270298+0000","last_change":"2024-02-19T21:09:54.270298+0000","last_active":"2024-02-19T21:09:54.270298+0000","last_peered":"2024-02-19T21:09:54.270298+0000","last_clean":"2024-02-19T21:09:54.270298+0000","last_became_active":"2024-02-19T21:04:54.644749+0000","last_became_peered":"2024-02-19T21:04:54.644749+0000","last_unstale":"2024-02-19T21:09:54.270298+0000","last_undegraded":"2024-02-19T21:09:54.270298+0000","last_fullsized":"2024-02-19T21:09:54.270298+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'34","last_scrub_stamp":"2024-02-19T21:08:12.445646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.445646+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.12","version":"390'47","reported_seq":474,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.280518+0000","last_change":"2024-02-19T21:09:54.280518+0000","last_active":"2024-02-19T21:09:54.280518+0000","last_peered":"2024-02-19T21:09:54.280518+0000","last_clean":"2024-02-19T21:09:54.280518+0000","last_became_active":"2024-02-19T21:08:04.263085+0000","last_became_peered":"2024-02-19T21:08:04.263085+0000","last_unstale":"2024-02-19T21:09:54.280518+0000","last_undegraded":"2024-02-19T21:09:54.280518+0000","last_fullsized":"2024-02-19T21:09:54.280518+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'45","last_scrub_stamp":"2024-02-19T21:08:49.965991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.965991+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.13","version":"370'71","reported_seq":475,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.274552+0000","last_change":"2024-02-19T21:09:54.274552+0000","last_active":"2024-02-19T21:09:54.274552+0000","last_peered":"2024-02-19T21:09:54.274552+0000","last_clean":"2024-02-19T21:09:54.274552+0000","last_became_active":"2024-02-19T21:08:41.264164+0000","last_became_peered":"2024-02-19T21:08:41.264164+0000","last_unstale":"2024-02-19T21:09:54.274552+0000","last_undegraded":"2024-02-19T21:09:54.274552+0000","last_fullsized":"2024-02-19T21:09:54.274552+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'68","last_scrub_stamp":"2024-02-19T21:08:12.764542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.764542+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":60,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.10","version":"230'45","reported_seq":397,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.274139+0000","last_change":"2024-02-19T21:09:54.274139+0000","last_active":"2024-02-19T21:09:54.274139+0000","last_peered":"2024-02-19T21:09:54.274139+0000","last_clean":"2024-02-19T21:09:54.274139+0000","last_became_active":"2024-02-19T21:04:54.626393+0000","last_became_peered":"2024-02-19T21:04:54.626393+0000","last_unstale":"2024-02-19T21:09:54.274139+0000","last_undegraded":"2024-02-19T21:09:54.274139+0000","last_fullsized":"2024-02-19T21:09:54.274139+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'45","last_scrub_stamp":"2024-02-19T21:08:39.942302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.942302+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.11","version":"338'35","reported_seq":387,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.279500+0000","last_change":"2024-02-19T21:09:54.279500+0000","last_active":"2024-02-19T21:09:54.279500+0000","last_peered":"2024-02-19T21:09:54.279500+0000","last_clean":"2024-02-19T21:09:54.279500+0000","last_became_active":"2024-02-19T21:04:54.659096+0000","last_became_peered":"2024-02-19T21:04:54.659096+0000","last_unstale":"2024-02-19T21:09:54.279500+0000","last_undegraded":"2024-02-19T21:09:54.279500+0000","last_fullsized":"2024-02-19T21:09:54.279500+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'35","last_scrub_stamp":"2024-02-19T21:08:41.145092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:41.145092+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.16","version":"411'43","reported_seq":302,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:56.272992+0000","last_change":"2024-02-19T21:09:56.272992+0000","last_active":"2024-02-19T21:09:56.272992+0000","last_peered":"2024-02-19T21:09:56.272992+0000","last_clean":"2024-02-19T21:09:56.272992+0000","last_became_active":"2024-02-19T21:08:41.269638+0000","last_became_peered":"2024-02-19T21:08:41.269638+0000","last_unstale":"2024-02-19T21:09:56.272992+0000","last_undegraded":"2024-02-19T21:09:56.272992+0000","last_fullsized":"2024-02-19T21:09:56.272992+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-02-19T21:06:57.889852+0000","last_deep_scrub":"237'24","last_deep_scrub_stamp":"2024-02-19T21:06:57.889852+0000","last_clean_scrub_stamp":"2024-02-19T21:06:57.889852+0000","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.17","version":"286'24","reported_seq":411,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.284847+0000","last_change":"2024-02-19T21:09:54.284847+0000","last_active":"2024-02-19T21:09:54.284847+0000","last_peered":"2024-02-19T21:09:54.284847+0000","last_clean":"2024-02-19T21:09:54.284847+0000","last_became_active":"2024-02-19T21:08:41.285673+0000","last_became_peered":"2024-02-19T21:08:41.285673+0000","last_unstale":"2024-02-19T21:09:54.284847+0000","last_undegraded":"2024-02-19T21:09:54.284847+0000","last_fullsized":"2024-02-19T21:09:54.284847+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'24","last_scrub_stamp":"2024-02-19T21:08:48.023079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:48.023079+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":27,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.14","version":"338'52","reported_seq":381,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.284416+0000","last_change":"2024-02-19T21:09:54.284416+0000","last_active":"2024-02-19T21:09:54.284416+0000","last_peered":"2024-02-19T21:09:54.284416+0000","last_clean":"2024-02-19T21:09:54.284416+0000","last_became_active":"2024-02-19T21:06:40.227840+0000","last_became_peered":"2024-02-19T21:06:40.227840+0000","last_unstale":"2024-02-19T21:09:54.284416+0000","last_undegraded":"2024-02-19T21:09:54.284416+0000","last_fullsized":"2024-02-19T21:09:54.284416+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'52","last_scrub_stamp":"2024-02-19T21:09:28.997593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.997593+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.15","version":"370'52","reported_seq":525,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.276777+0000","last_change":"2024-02-19T21:09:54.276777+0000","last_active":"2024-02-19T21:09:54.276777+0000","last_peered":"2024-02-19T21:09:54.276777+0000","last_clean":"2024-02-19T21:09:54.276777+0000","last_became_active":"2024-02-19T21:08:41.277044+0000","last_became_peered":"2024-02-19T21:08:41.277044+0000","last_unstale":"2024-02-19T21:09:54.276777+0000","last_undegraded":"2024-02-19T21:09:54.276777+0000","last_fullsized":"2024-02-19T21:09:54.276777+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'51","last_scrub_stamp":"2024-02-19T21:07:28.859888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:07:28.859888+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":99,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.1a","version":"338'91","reported_seq":566,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.274279+0000","last_change":"2024-02-19T21:09:54.274279+0000","last_active":"2024-02-19T21:09:54.274279+0000","last_peered":"2024-02-19T21:09:54.274279+0000","last_clean":"2024-02-19T21:09:54.274279+0000","last_became_active":"2024-02-19T21:08:04.265588+0000","last_became_peered":"2024-02-19T21:08:04.265588+0000","last_unstale":"2024-02-19T21:09:54.274279+0000","last_undegraded":"2024-02-19T21:09:54.274279+0000","last_fullsized":"2024-02-19T21:09:54.274279+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'90","last_scrub_stamp":"2024-02-19T21:08:02.861467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:02.861467+0000","log_size":91,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.1b","version":"338'52","reported_seq":155,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270086+0000","last_change":"2024-02-19T21:09:54.270086+0000","last_active":"2024-02-19T21:09:54.270086+0000","last_peered":"2024-02-19T21:09:54.270086+0000","last_clean":"2024-02-19T21:09:54.270086+0000","last_became_active":"2024-02-19T21:08:04.262932+0000","last_became_peered":"2024-02-19T21:08:04.262932+0000","last_unstale":"2024-02-19T21:09:54.270086+0000","last_undegraded":"2024-02-19T21:09:54.270086+0000","last_fullsized":"2024-02-19T21:09:54.270086+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'51","last_scrub_stamp":"2024-02-19T21:08:00.485092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:00.485092+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.18","version":"214'102","reported_seq":419,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268817+0000","last_change":"2024-02-19T21:09:54.268817+0000","last_active":"2024-02-19T21:09:54.268817+0000","last_peered":"2024-02-19T21:09:54.268817+0000","last_clean":"2024-02-19T21:09:54.268817+0000","last_became_active":"2024-02-19T21:08:44.236717+0000","last_became_peered":"2024-02-19T21:08:44.236717+0000","last_unstale":"2024-02-19T21:09:54.268817+0000","last_undegraded":"2024-02-19T21:09:54.268817+0000","last_fullsized":"2024-02-19T21:09:54.268817+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-19T21:08:50.838363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:50.838363+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.19","version":"338'69","reported_seq":470,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269653+0000","last_change":"2024-02-19T21:09:54.269653+0000","last_active":"2024-02-19T21:09:54.269653+0000","last_peered":"2024-02-19T21:09:54.269653+0000","last_clean":"2024-02-19T21:09:54.269653+0000","last_became_active":"2024-02-19T21:08:41.292658+0000","last_became_peered":"2024-02-19T21:08:41.292658+0000","last_unstale":"2024-02-19T21:09:54.269653+0000","last_undegraded":"2024-02-19T21:09:54.269653+0000","last_fullsized":"2024-02-19T21:09:54.269653+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-19T21:08:39.466543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.466543+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.1e","version":"338'50","reported_seq":405,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.276257+0000","last_change":"2024-02-19T21:09:54.276257+0000","last_active":"2024-02-19T21:09:54.276257+0000","last_peered":"2024-02-19T21:09:54.276257+0000","last_clean":"2024-02-19T21:09:54.276257+0000","last_became_active":"2024-02-19T21:04:54.629433+0000","last_became_peered":"2024-02-19T21:04:54.629433+0000","last_unstale":"2024-02-19T21:09:54.276257+0000","last_undegraded":"2024-02-19T21:09:54.276257+0000","last_fullsized":"2024-02-19T21:09:54.276257+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'49","last_scrub_stamp":"2024-02-19T21:07:01.787521+0000","last_deep_scrub":"241'49","last_deep_scrub_stamp":"2024-02-19T21:07:01.787521+0000","last_clean_scrub_stamp":"2024-02-19T21:07:01.787521+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":48,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.1f","version":"249'44","reported_seq":396,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.285026+0000","last_change":"2024-02-19T21:09:54.285026+0000","last_active":"2024-02-19T21:09:54.285026+0000","last_peered":"2024-02-19T21:09:54.285026+0000","last_clean":"2024-02-19T21:09:54.285026+0000","last_became_active":"2024-02-19T21:04:54.671969+0000","last_became_peered":"2024-02-19T21:04:54.671969+0000","last_unstale":"2024-02-19T21:09:54.285026+0000","last_undegraded":"2024-02-19T21:09:54.285026+0000","last_fullsized":"2024-02-19T21:09:54.285026+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'44","last_scrub_stamp":"2024-02-19T21:09:25.988978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:09:25.988978+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.1c","version":"338'137","reported_seq":574,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270752+0000","last_change":"2024-02-19T21:09:54.270752+0000","last_active":"2024-02-19T21:09:54.270752+0000","last_peered":"2024-02-19T21:09:54.270752+0000","last_clean":"2024-02-19T21:09:54.270752+0000","last_became_active":"2024-02-19T21:08:48.248009+0000","last_became_peered":"2024-02-19T21:08:48.248009+0000","last_unstale":"2024-02-19T21:09:54.270752+0000","last_undegraded":"2024-02-19T21:09:54.270752+0000","last_fullsized":"2024-02-19T21:09:54.270752+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'136","last_scrub_stamp":"2024-02-19T21:08:37.523303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:37.523303+0000","log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":96,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":41,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":96,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"61.1d","version":"370'36","reported_seq":389,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.448779+0000","last_change":"2024-02-19T21:09:54.448779+0000","last_active":"2024-02-19T21:09:54.448779+0000","last_peered":"2024-02-19T21:09:54.448779+0000","last_clean":"2024-02-19T21:09:54.448779+0000","last_became_active":"2024-02-19T21:04:54.655665+0000","last_became_peered":"2024-02-19T21:04:54.655665+0000","last_unstale":"2024-02-19T21:09:54.448779+0000","last_undegraded":"2024-02-19T21:09:54.448779+0000","last_fullsized":"2024-02-19T21:09:54.448779+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'35","last_scrub_stamp":"2024-02-19T21:08:56.661296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:56.661296+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16","length":"1"}]},{"pgid":"5.9","version":"344'10","reported_seq":503,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-02-19T21:09:59.292597+0000","last_change":"2024-02-19T21:09:42.967840+0000","last_active":"2024-02-19T21:09:59.292597+0000","last_peered":"2024-02-19T21:09:59.292597+0000","last_clean":"2024-02-19T21:09:59.292597+0000","last_became_active":"2024-02-19T21:08:41.307616+0000","last_became_peered":"2024-02-19T21:08:41.307616+0000","last_unstale":"2024-02-19T21:09:59.292597+0000","last_undegraded":"2024-02-19T21:09:59.292597+0000","last_fullsized":"2024-02-19T21:09:59.292597+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'10","last_scrub_stamp":"2024-02-19T21:09:42.967771+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:09:42.967771+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":402,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268137+0000","last_change":"2024-02-19T21:08:51.616197+0000","last_active":"2024-02-19T21:09:54.268137+0000","last_peered":"2024-02-19T21:09:54.268137+0000","last_clean":"2024-02-19T21:09:54.268137+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:09:54.268137+0000","last_undegraded":"2024-02-19T21:09:54.268137+0000","last_fullsized":"2024-02-19T21:09:54.268137+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:08:51.616123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:51.616123+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":398,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.282387+0000","last_change":"2024-02-19T21:09:44.922372+0000","last_active":"2024-02-19T21:09:54.282387+0000","last_peered":"2024-02-19T21:09:54.282387+0000","last_clean":"2024-02-19T21:09:54.282387+0000","last_became_active":"2024-02-19T21:08:04.239474+0000","last_became_peered":"2024-02-19T21:08:04.239474+0000","last_unstale":"2024-02-19T21:09:54.282387+0000","last_undegraded":"2024-02-19T21:09:54.282387+0000","last_fullsized":"2024-02-19T21:09:54.282387+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:44.922303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_clean_scrub_stamp":"2024-02-19T21:09:44.922303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":370,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268821+0000","last_change":"2024-02-19T21:09:27.878844+0000","last_active":"2024-02-19T21:09:54.268821+0000","last_peered":"2024-02-19T21:09:54.268821+0000","last_clean":"2024-02-19T21:09:54.268821+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:09:54.268821+0000","last_undegraded":"2024-02-19T21:09:54.268821+0000","last_fullsized":"2024-02-19T21:09:54.268821+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.878760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.878760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":323,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.284104+0000","last_change":"2024-02-19T21:06:46.312127+0000","last_active":"2024-02-19T21:09:54.284104+0000","last_peered":"2024-02-19T21:09:54.284104+0000","last_clean":"2024-02-19T21:09:54.284104+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:09:54.284104+0000","last_undegraded":"2024-02-19T21:09:54.284104+0000","last_fullsized":"2024-02-19T21:09:54.284104+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:46.312025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.312025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":253,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.450090+0000","last_change":"2024-02-19T21:08:53.783361+0000","last_active":"2024-02-19T21:09:54.450090+0000","last_peered":"2024-02-19T21:09:54.450090+0000","last_clean":"2024-02-19T21:09:54.450090+0000","last_became_active":"2024-02-19T21:08:41.314093+0000","last_became_peered":"2024-02-19T21:08:41.314093+0000","last_unstale":"2024-02-19T21:09:54.450090+0000","last_undegraded":"2024-02-19T21:09:54.450090+0000","last_fullsized":"2024-02-19T21:09:54.450090+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:08:53.783292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:53.783292+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":407,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447868+0000","last_change":"2024-02-19T21:08:41.309125+0000","last_active":"2024-02-19T21:09:54.447868+0000","last_peered":"2024-02-19T21:09:54.447868+0000","last_clean":"2024-02-19T21:09:54.447868+0000","last_became_active":"2024-02-19T21:08:41.302799+0000","last_became_peered":"2024-02-19T21:08:41.302799+0000","last_unstale":"2024-02-19T21:09:54.447868+0000","last_undegraded":"2024-02-19T21:09:54.447868+0000","last_fullsized":"2024-02-19T21:09:54.447868+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:39.700315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.700315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":398,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.280000+0000","last_change":"2024-02-19T21:08:41.277873+0000","last_active":"2024-02-19T21:09:54.280000+0000","last_peered":"2024-02-19T21:09:54.280000+0000","last_clean":"2024-02-19T21:09:54.280000+0000","last_became_active":"2024-02-19T21:08:41.276911+0000","last_became_peered":"2024-02-19T21:08:41.276911+0000","last_unstale":"2024-02-19T21:09:54.280000+0000","last_undegraded":"2024-02-19T21:09:54.280000+0000","last_fullsized":"2024-02-19T21:09:54.280000+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:27.303307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:27.303307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":207,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.275546+0000","last_change":"2024-02-19T21:09:34.791174+0000","last_active":"2024-02-19T21:09:54.275546+0000","last_peered":"2024-02-19T21:09:54.275546+0000","last_clean":"2024-02-19T21:09:54.275546+0000","last_became_active":"2024-02-19T21:06:40.227822+0000","last_became_peered":"2024-02-19T21:06:40.227822+0000","last_unstale":"2024-02-19T21:09:54.275546+0000","last_undegraded":"2024-02-19T21:09:54.275546+0000","last_fullsized":"2024-02-19T21:09:54.275546+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:34.791063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:34.791063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":324,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269334+0000","last_change":"2024-02-19T21:07:00.449622+0000","last_active":"2024-02-19T21:09:54.269334+0000","last_peered":"2024-02-19T21:09:54.269334+0000","last_clean":"2024-02-19T21:09:54.269334+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:09:54.269334+0000","last_undegraded":"2024-02-19T21:09:54.269334+0000","last_fullsized":"2024-02-19T21:09:54.269334+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:00.449534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:00.449534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":407,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446808+0000","last_change":"2024-02-19T21:09:34.573994+0000","last_active":"2024-02-19T21:09:54.446808+0000","last_peered":"2024-02-19T21:09:54.446808+0000","last_clean":"2024-02-19T21:09:54.446808+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:09:54.446808+0000","last_undegraded":"2024-02-19T21:09:54.446808+0000","last_fullsized":"2024-02-19T21:09:54.446808+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:09:34.573889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:34.573889+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":380,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.283988+0000","last_change":"2024-02-19T21:09:30.950696+0000","last_active":"2024-02-19T21:09:54.283988+0000","last_peered":"2024-02-19T21:09:54.283988+0000","last_clean":"2024-02-19T21:09:54.283988+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:09:54.283988+0000","last_undegraded":"2024-02-19T21:09:54.283988+0000","last_fullsized":"2024-02-19T21:09:54.283988+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:30.950626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:09:30.950626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":371,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.281523+0000","last_change":"2024-02-19T21:09:28.031934+0000","last_active":"2024-02-19T21:09:54.281523+0000","last_peered":"2024-02-19T21:09:54.281523+0000","last_clean":"2024-02-19T21:09:54.281523+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:09:54.281523+0000","last_undegraded":"2024-02-19T21:09:54.281523+0000","last_fullsized":"2024-02-19T21:09:54.281523+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:28.031872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.031872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":387,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.272919+0000","last_change":"2024-02-19T21:08:53.964809+0000","last_active":"2024-02-19T21:09:54.272919+0000","last_peered":"2024-02-19T21:09:54.272919+0000","last_clean":"2024-02-19T21:09:54.272919+0000","last_became_active":"2024-02-19T21:08:04.265081+0000","last_became_peered":"2024-02-19T21:08:04.265081+0000","last_unstale":"2024-02-19T21:09:54.272919+0000","last_undegraded":"2024-02-19T21:09:54.272919+0000","last_fullsized":"2024-02-19T21:09:54.272919+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_clean_scrub_stamp":"2024-02-19T21:08:53.964703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":338,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.274450+0000","last_change":"2024-02-19T21:09:43.984159+0000","last_active":"2024-02-19T21:09:54.274450+0000","last_peered":"2024-02-19T21:09:54.274450+0000","last_clean":"2024-02-19T21:09:54.274450+0000","last_became_active":"2024-02-19T21:08:41.304395+0000","last_became_peered":"2024-02-19T21:08:41.304395+0000","last_unstale":"2024-02-19T21:09:54.274450+0000","last_undegraded":"2024-02-19T21:09:54.274450+0000","last_fullsized":"2024-02-19T21:09:54.274450+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:43.984079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:09:43.984079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":144,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.281225+0000","last_change":"2024-02-19T21:09:41.955664+0000","last_active":"2024-02-19T21:09:54.281225+0000","last_peered":"2024-02-19T21:09:54.281225+0000","last_clean":"2024-02-19T21:09:54.281225+0000","last_became_active":"2024-02-19T21:08:41.286244+0000","last_became_peered":"2024-02-19T21:08:41.286244+0000","last_unstale":"2024-02-19T21:09:54.281225+0000","last_undegraded":"2024-02-19T21:09:54.281225+0000","last_fullsized":"2024-02-19T21:09:54.281225+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:09:41.955563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:41.955563+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":394,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.273960+0000","last_change":"2024-02-19T21:08:41.367028+0000","last_active":"2024-02-19T21:09:54.273960+0000","last_peered":"2024-02-19T21:09:54.273960+0000","last_clean":"2024-02-19T21:09:54.273960+0000","last_became_active":"2024-02-19T21:08:41.273961+0000","last_became_peered":"2024-02-19T21:08:41.273961+0000","last_unstale":"2024-02-19T21:09:54.273960+0000","last_undegraded":"2024-02-19T21:09:54.273960+0000","last_fullsized":"2024-02-19T21:09:54.273960+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:06:50.694248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:50.694248+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":385,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446995+0000","last_change":"2024-02-19T21:09:36.685164+0000","last_active":"2024-02-19T21:09:54.446995+0000","last_peered":"2024-02-19T21:09:54.446995+0000","last_clean":"2024-02-19T21:09:54.446995+0000","last_became_active":"2024-02-19T21:08:41.302537+0000","last_became_peered":"2024-02-19T21:08:41.302537+0000","last_unstale":"2024-02-19T21:09:54.446995+0000","last_undegraded":"2024-02-19T21:09:54.446995+0000","last_fullsized":"2024-02-19T21:09:54.446995+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:36.685063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:36.685063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":396,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.282615+0000","last_change":"2024-02-19T21:08:41.278598+0000","last_active":"2024-02-19T21:09:54.282615+0000","last_peered":"2024-02-19T21:09:54.282615+0000","last_clean":"2024-02-19T21:09:54.282615+0000","last_became_active":"2024-02-19T21:08:41.278298+0000","last_became_peered":"2024-02-19T21:08:41.278298+0000","last_unstale":"2024-02-19T21:09:54.282615+0000","last_undegraded":"2024-02-19T21:09:54.282615+0000","last_fullsized":"2024-02-19T21:09:54.282615+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:13.136532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:06:13.136532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":309,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270608+0000","last_change":"2024-02-19T21:08:41.296517+0000","last_active":"2024-02-19T21:09:54.270608+0000","last_peered":"2024-02-19T21:09:54.270608+0000","last_clean":"2024-02-19T21:09:54.270608+0000","last_became_active":"2024-02-19T21:08:41.296219+0000","last_became_peered":"2024-02-19T21:08:41.296219+0000","last_unstale":"2024-02-19T21:09:54.270608+0000","last_undegraded":"2024-02-19T21:09:54.270608+0000","last_fullsized":"2024-02-19T21:09:54.270608+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.878060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.878060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"82'15","reported_seq":403,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.271120+0000","last_change":"2024-02-19T21:08:42.367657+0000","last_active":"2024-02-19T21:09:54.271120+0000","last_peered":"2024-02-19T21:09:54.271120+0000","last_clean":"2024-02-19T21:09:54.271120+0000","last_became_active":"2024-02-19T21:08:41.295931+0000","last_became_peered":"2024-02-19T21:08:41.295931+0000","last_unstale":"2024-02-19T21:09:54.271120+0000","last_undegraded":"2024-02-19T21:09:54.271120+0000","last_fullsized":"2024-02-19T21:09:54.271120+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:06:28.746972+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:06:28.746972+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":393,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446453+0000","last_change":"2024-02-19T21:08:04.269175+0000","last_active":"2024-02-19T21:09:54.446453+0000","last_peered":"2024-02-19T21:09:54.446453+0000","last_clean":"2024-02-19T21:09:54.446453+0000","last_became_active":"2024-02-19T21:08:04.268406+0000","last_became_peered":"2024-02-19T21:08:04.268406+0000","last_unstale":"2024-02-19T21:09:54.446453+0000","last_undegraded":"2024-02-19T21:09:54.446453+0000","last_fullsized":"2024-02-19T21:09:54.446453+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:14.687938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.687938+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":484,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-02-19T21:09:59.291459+0000","last_change":"2024-02-19T21:06:12.177597+0000","last_active":"2024-02-19T21:09:59.291459+0000","last_peered":"2024-02-19T21:09:59.291459+0000","last_clean":"2024-02-19T21:09:59.291459+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:09:59.291459+0000","last_undegraded":"2024-02-19T21:09:59.291459+0000","last_fullsized":"2024-02-19T21:09:59.291459+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:12.177524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:12.177524+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":82,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447126+0000","last_change":"2024-02-19T21:08:41.300913+0000","last_active":"2024-02-19T21:09:54.447126+0000","last_peered":"2024-02-19T21:09:54.447126+0000","last_clean":"2024-02-19T21:09:54.447126+0000","last_became_active":"2024-02-19T21:08:41.300707+0000","last_became_peered":"2024-02-19T21:08:41.300707+0000","last_unstale":"2024-02-19T21:09:54.447126+0000","last_undegraded":"2024-02-19T21:09:54.447126+0000","last_fullsized":"2024-02-19T21:09:54.447126+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:18.704447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:06:18.704447+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":123,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268239+0000","last_change":"2024-02-19T21:08:36.847613+0000","last_active":"2024-02-19T21:09:54.268239+0000","last_peered":"2024-02-19T21:09:54.268239+0000","last_clean":"2024-02-19T21:09:54.268239+0000","last_became_active":"2024-02-19T21:08:04.242244+0000","last_became_peered":"2024-02-19T21:08:04.242244+0000","last_unstale":"2024-02-19T21:09:54.268239+0000","last_undegraded":"2024-02-19T21:09:54.268239+0000","last_fullsized":"2024-02-19T21:09:54.268239+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:36.847548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.847548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":473,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446874+0000","last_change":"2024-02-19T21:08:43.178829+0000","last_active":"2024-02-19T21:09:54.446874+0000","last_peered":"2024-02-19T21:09:54.446874+0000","last_clean":"2024-02-19T21:09:54.446874+0000","last_became_active":"2024-02-19T21:08:41.311383+0000","last_became_peered":"2024-02-19T21:08:41.311383+0000","last_unstale":"2024-02-19T21:09:54.446874+0000","last_undegraded":"2024-02-19T21:09:54.446874+0000","last_fullsized":"2024-02-19T21:09:54.446874+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:07:59.834282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:07:59.834282+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":420,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446408+0000","last_change":"2024-02-19T21:08:04.271314+0000","last_active":"2024-02-19T21:09:54.446408+0000","last_peered":"2024-02-19T21:09:54.446408+0000","last_clean":"2024-02-19T21:09:54.446408+0000","last_became_active":"2024-02-19T21:08:04.270601+0000","last_became_peered":"2024-02-19T21:08:04.270601+0000","last_unstale":"2024-02-19T21:09:54.446408+0000","last_undegraded":"2024-02-19T21:09:54.446408+0000","last_fullsized":"2024-02-19T21:09:54.446408+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:07.874362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:07.874362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"344'9","reported_seq":121,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-02-19T21:09:59.258039+0000","last_change":"2024-02-19T21:08:42.577019+0000","last_active":"2024-02-19T21:09:59.258039+0000","last_peered":"2024-02-19T21:09:59.258039+0000","last_clean":"2024-02-19T21:09:59.258039+0000","last_became_active":"2024-02-19T21:08:41.293491+0000","last_became_peered":"2024-02-19T21:08:41.293491+0000","last_unstale":"2024-02-19T21:09:59.258039+0000","last_undegraded":"2024-02-19T21:09:59.258039+0000","last_fullsized":"2024-02-19T21:09:59.258039+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:20.710731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:20.710731+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":375,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269940+0000","last_change":"2024-02-19T21:09:31.856861+0000","last_active":"2024-02-19T21:09:54.269940+0000","last_peered":"2024-02-19T21:09:54.269940+0000","last_clean":"2024-02-19T21:09:54.269940+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:09:54.269940+0000","last_undegraded":"2024-02-19T21:09:54.269940+0000","last_fullsized":"2024-02-19T21:09:54.269940+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:31.856784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:31.856784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":324,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269367+0000","last_change":"2024-02-19T21:06:51.423833+0000","last_active":"2024-02-19T21:09:54.269367+0000","last_peered":"2024-02-19T21:09:54.269367+0000","last_clean":"2024-02-19T21:09:54.269367+0000","last_became_active":"2024-02-19T21:04:39.657052+0000","last_became_peered":"2024-02-19T21:04:39.657052+0000","last_unstale":"2024-02-19T21:09:54.269367+0000","last_undegraded":"2024-02-19T21:09:54.269367+0000","last_fullsized":"2024-02-19T21:09:54.269367+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.423769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.423769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":129,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447563+0000","last_change":"2024-02-19T21:08:49.181165+0000","last_active":"2024-02-19T21:09:54.447563+0000","last_peered":"2024-02-19T21:09:54.447563+0000","last_clean":"2024-02-19T21:09:54.447563+0000","last_became_active":"2024-02-19T21:08:41.278810+0000","last_became_peered":"2024-02-19T21:08:41.278810+0000","last_unstale":"2024-02-19T21:09:54.447563+0000","last_undegraded":"2024-02-19T21:09:54.447563+0000","last_fullsized":"2024-02-19T21:09:54.447563+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:49.181100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.181100+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":407,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.273046+0000","last_change":"2024-02-19T21:09:37.870271+0000","last_active":"2024-02-19T21:09:54.273046+0000","last_peered":"2024-02-19T21:09:54.273046+0000","last_clean":"2024-02-19T21:09:54.273046+0000","last_became_active":"2024-02-19T21:07:32.227557+0000","last_became_peered":"2024-02-19T21:07:32.227557+0000","last_unstale":"2024-02-19T21:09:54.273046+0000","last_undegraded":"2024-02-19T21:09:54.273046+0000","last_fullsized":"2024-02-19T21:09:54.273046+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.870218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.870218+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":381,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269630+0000","last_change":"2024-02-19T21:08:41.293711+0000","last_active":"2024-02-19T21:09:54.269630+0000","last_peered":"2024-02-19T21:09:54.269630+0000","last_clean":"2024-02-19T21:09:54.269630+0000","last_became_active":"2024-02-19T21:08:41.292701+0000","last_became_peered":"2024-02-19T21:08:41.292701+0000","last_unstale":"2024-02-19T21:09:54.269630+0000","last_undegraded":"2024-02-19T21:09:54.269630+0000","last_fullsized":"2024-02-19T21:09:54.269630+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:53.428093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:53.428093+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":118,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.271306+0000","last_change":"2024-02-19T21:08:04.279238+0000","last_active":"2024-02-19T21:09:54.271306+0000","last_peered":"2024-02-19T21:09:54.271306+0000","last_clean":"2024-02-19T21:09:54.271306+0000","last_became_active":"2024-02-19T21:08:04.279008+0000","last_became_peered":"2024-02-19T21:08:04.279008+0000","last_unstale":"2024-02-19T21:09:54.271306+0000","last_undegraded":"2024-02-19T21:09:54.271306+0000","last_fullsized":"2024-02-19T21:09:54.271306+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:05.721175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:05.721175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":118,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270864+0000","last_change":"2024-02-19T21:08:04.280833+0000","last_active":"2024-02-19T21:09:54.270864+0000","last_peered":"2024-02-19T21:09:54.270864+0000","last_clean":"2024-02-19T21:09:54.270864+0000","last_became_active":"2024-02-19T21:08:04.280514+0000","last_became_peered":"2024-02-19T21:08:04.280514+0000","last_unstale":"2024-02-19T21:09:54.270864+0000","last_undegraded":"2024-02-19T21:09:54.270864+0000","last_fullsized":"2024-02-19T21:09:54.270864+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:58.499841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:58.499841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"344'11","reported_seq":166,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-02-19T21:09:59.260693+0000","last_change":"2024-02-19T21:09:39.597074+0000","last_active":"2024-02-19T21:09:59.260693+0000","last_peered":"2024-02-19T21:09:59.260693+0000","last_clean":"2024-02-19T21:09:59.260693+0000","last_became_active":"2024-02-19T21:08:41.296797+0000","last_became_peered":"2024-02-19T21:08:41.296797+0000","last_unstale":"2024-02-19T21:09:59.260693+0000","last_undegraded":"2024-02-19T21:09:59.260693+0000","last_fullsized":"2024-02-19T21:09:59.260693+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'11","last_scrub_stamp":"2024-02-19T21:09:39.597001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:39.597001+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":131,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268701+0000","last_change":"2024-02-19T21:08:41.921037+0000","last_active":"2024-02-19T21:09:54.268701+0000","last_peered":"2024-02-19T21:09:54.268701+0000","last_clean":"2024-02-19T21:09:54.268701+0000","last_became_active":"2024-02-19T21:08:41.268914+0000","last_became_peered":"2024-02-19T21:08:41.268914+0000","last_unstale":"2024-02-19T21:09:54.268701+0000","last_undegraded":"2024-02-19T21:09:54.268701+0000","last_fullsized":"2024-02-19T21:09:54.268701+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:06:46.813814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.813814+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":405,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447396+0000","last_change":"2024-02-19T21:09:38.541487+0000","last_active":"2024-02-19T21:09:54.447396+0000","last_peered":"2024-02-19T21:09:54.447396+0000","last_clean":"2024-02-19T21:09:54.447396+0000","last_became_active":"2024-02-19T21:07:36.263299+0000","last_became_peered":"2024-02-19T21:07:36.263299+0000","last_unstale":"2024-02-19T21:09:54.447396+0000","last_undegraded":"2024-02-19T21:09:54.447396+0000","last_fullsized":"2024-02-19T21:09:54.447396+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:38.541390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:38.541390+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":371,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.274864+0000","last_change":"2024-02-19T21:07:25.812691+0000","last_active":"2024-02-19T21:09:54.274864+0000","last_peered":"2024-02-19T21:09:54.274864+0000","last_clean":"2024-02-19T21:09:54.274864+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:09:54.274864+0000","last_undegraded":"2024-02-19T21:09:54.274864+0000","last_fullsized":"2024-02-19T21:09:54.274864+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:25.812627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:07:25.812627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":201,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.448619+0000","last_change":"2024-02-19T21:06:40.245744+0000","last_active":"2024-02-19T21:09:54.448619+0000","last_peered":"2024-02-19T21:09:54.448619+0000","last_clean":"2024-02-19T21:09:54.448619+0000","last_became_active":"2024-02-19T21:06:40.245385+0000","last_became_peered":"2024-02-19T21:06:40.245385+0000","last_unstale":"2024-02-19T21:09:54.448619+0000","last_undegraded":"2024-02-19T21:09:54.448619+0000","last_fullsized":"2024-02-19T21:09:54.448619+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:16.780601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.780601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":352,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270665+0000","last_change":"2024-02-19T21:08:41.270754+0000","last_active":"2024-02-19T21:09:54.270665+0000","last_peered":"2024-02-19T21:09:54.270665+0000","last_clean":"2024-02-19T21:09:54.270665+0000","last_became_active":"2024-02-19T21:08:41.270483+0000","last_became_peered":"2024-02-19T21:08:41.270483+0000","last_unstale":"2024-02-19T21:09:54.270665+0000","last_undegraded":"2024-02-19T21:09:54.270665+0000","last_fullsized":"2024-02-19T21:09:54.270665+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:12.606585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.606585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":136,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268403+0000","last_change":"2024-02-19T21:08:04.343458+0000","last_active":"2024-02-19T21:09:54.268403+0000","last_peered":"2024-02-19T21:09:54.268403+0000","last_clean":"2024-02-19T21:09:54.268403+0000","last_became_active":"2024-02-19T21:08:04.242466+0000","last_became_peered":"2024-02-19T21:08:04.242466+0000","last_unstale":"2024-02-19T21:09:54.268403+0000","last_undegraded":"2024-02-19T21:09:54.268403+0000","last_fullsized":"2024-02-19T21:09:54.268403+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:06:46.816121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:46.816121+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":411,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446866+0000","last_change":"2024-02-19T21:08:41.294617+0000","last_active":"2024-02-19T21:09:54.446866+0000","last_peered":"2024-02-19T21:09:54.446866+0000","last_clean":"2024-02-19T21:09:54.446866+0000","last_became_active":"2024-02-19T21:08:41.294281+0000","last_became_peered":"2024-02-19T21:08:41.294281+0000","last_unstale":"2024-02-19T21:09:54.446866+0000","last_undegraded":"2024-02-19T21:09:54.446866+0000","last_fullsized":"2024-02-19T21:09:54.446866+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":355,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270620+0000","last_change":"2024-02-19T21:09:40.549754+0000","last_active":"2024-02-19T21:09:54.270620+0000","last_peered":"2024-02-19T21:09:54.270620+0000","last_clean":"2024-02-19T21:09:54.270620+0000","last_became_active":"2024-02-19T21:08:41.266934+0000","last_became_peered":"2024-02-19T21:08:41.266934+0000","last_unstale":"2024-02-19T21:09:54.270620+0000","last_undegraded":"2024-02-19T21:09:54.270620+0000","last_fullsized":"2024-02-19T21:09:54.270620+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:40.549690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.549690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":118,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.271393+0000","last_change":"2024-02-19T21:08:04.267005+0000","last_active":"2024-02-19T21:09:54.271393+0000","last_peered":"2024-02-19T21:09:54.271393+0000","last_clean":"2024-02-19T21:09:54.271393+0000","last_became_active":"2024-02-19T21:08:04.266538+0000","last_became_peered":"2024-02-19T21:08:04.266538+0000","last_unstale":"2024-02-19T21:09:54.271393+0000","last_undegraded":"2024-02-19T21:09:54.271393+0000","last_fullsized":"2024-02-19T21:09:54.271393+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:58.847915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:58.847915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":374,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446871+0000","last_change":"2024-02-19T21:07:00.676713+0000","last_active":"2024-02-19T21:09:54.446871+0000","last_peered":"2024-02-19T21:09:54.446871+0000","last_clean":"2024-02-19T21:09:54.446871+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:09:54.446871+0000","last_undegraded":"2024-02-19T21:09:54.446871+0000","last_fullsized":"2024-02-19T21:09:54.446871+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:00.676634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:07:00.676634+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":250,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269728+0000","last_change":"2024-02-19T21:06:40.224213+0000","last_active":"2024-02-19T21:09:54.269728+0000","last_peered":"2024-02-19T21:09:54.269728+0000","last_clean":"2024-02-19T21:09:54.269728+0000","last_became_active":"2024-02-19T21:06:40.223768+0000","last_became_peered":"2024-02-19T21:06:40.223768+0000","last_unstale":"2024-02-19T21:09:54.269728+0000","last_undegraded":"2024-02-19T21:09:54.269728+0000","last_fullsized":"2024-02-19T21:09:54.269728+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:06:10.826889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:10.826889+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":418,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.276052+0000","last_change":"2024-02-19T21:09:35.824601+0000","last_active":"2024-02-19T21:09:54.276052+0000","last_peered":"2024-02-19T21:09:54.276052+0000","last_clean":"2024-02-19T21:09:54.276052+0000","last_became_active":"2024-02-19T21:08:41.303606+0000","last_became_peered":"2024-02-19T21:08:41.303606+0000","last_unstale":"2024-02-19T21:09:54.276052+0000","last_undegraded":"2024-02-19T21:09:54.276052+0000","last_fullsized":"2024-02-19T21:09:54.276052+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:35.824520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:35.824520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":369,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446573+0000","last_change":"2024-02-19T21:08:57.653280+0000","last_active":"2024-02-19T21:09:54.446573+0000","last_peered":"2024-02-19T21:09:54.446573+0000","last_clean":"2024-02-19T21:09:54.446573+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:09:54.446573+0000","last_undegraded":"2024-02-19T21:09:54.446573+0000","last_fullsized":"2024-02-19T21:09:54.446573+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:57.653204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:08:57.653204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":323,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.282763+0000","last_change":"2024-02-19T21:08:12.309008+0000","last_active":"2024-02-19T21:09:54.282763+0000","last_peered":"2024-02-19T21:09:54.282763+0000","last_clean":"2024-02-19T21:09:54.282763+0000","last_became_active":"2024-02-19T21:04:39.718926+0000","last_became_peered":"2024-02-19T21:04:39.718926+0000","last_unstale":"2024-02-19T21:09:54.282763+0000","last_undegraded":"2024-02-19T21:09:54.282763+0000","last_fullsized":"2024-02-19T21:09:54.282763+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:12.308951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.308951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":372,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269761+0000","last_change":"2024-02-19T21:08:11.431726+0000","last_active":"2024-02-19T21:09:54.269761+0000","last_peered":"2024-02-19T21:09:54.269761+0000","last_clean":"2024-02-19T21:09:54.269761+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:09:54.269761+0000","last_undegraded":"2024-02-19T21:09:54.269761+0000","last_fullsized":"2024-02-19T21:09:54.269761+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:11.431637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:11.431637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":236,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447015+0000","last_change":"2024-02-19T21:09:30.704618+0000","last_active":"2024-02-19T21:09:54.447015+0000","last_peered":"2024-02-19T21:09:54.447015+0000","last_clean":"2024-02-19T21:09:54.447015+0000","last_became_active":"2024-02-19T21:06:40.240321+0000","last_became_peered":"2024-02-19T21:06:40.240321+0000","last_unstale":"2024-02-19T21:09:54.447015+0000","last_undegraded":"2024-02-19T21:09:54.447015+0000","last_fullsized":"2024-02-19T21:09:54.447015+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:09:30.704543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:30.704543+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"397'2","reported_seq":415,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.272966+0000","last_change":"2024-02-19T21:09:40.934534+0000","last_active":"2024-02-19T21:09:54.272966+0000","last_peered":"2024-02-19T21:09:54.272966+0000","last_clean":"2024-02-19T21:09:54.272966+0000","last_became_active":"2024-02-19T21:08:10.225480+0000","last_became_peered":"2024-02-19T21:08:10.225480+0000","last_unstale":"2024-02-19T21:09:54.272966+0000","last_undegraded":"2024-02-19T21:09:54.272966+0000","last_fullsized":"2024-02-19T21:09:54.272966+0000","mapping_epoch":57,"log_start":"58'1","ondisk_log_start":"58'1","created":57,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.934474+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":187,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270534+0000","last_change":"2024-02-19T21:08:41.277088+0000","last_active":"2024-02-19T21:09:54.270534+0000","last_peered":"2024-02-19T21:09:54.270534+0000","last_clean":"2024-02-19T21:09:54.270534+0000","last_became_active":"2024-02-19T21:08:41.276642+0000","last_became_peered":"2024-02-19T21:08:41.276642+0000","last_unstale":"2024-02-19T21:09:54.270534+0000","last_undegraded":"2024-02-19T21:09:54.270534+0000","last_fullsized":"2024-02-19T21:09:54.270534+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:35.542752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:08:35.542752+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":387,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447028+0000","last_change":"2024-02-19T21:08:04.282673+0000","last_active":"2024-02-19T21:09:54.447028+0000","last_peered":"2024-02-19T21:09:54.447028+0000","last_clean":"2024-02-19T21:09:54.447028+0000","last_became_active":"2024-02-19T21:08:04.282243+0000","last_became_peered":"2024-02-19T21:08:04.282243+0000","last_unstale":"2024-02-19T21:09:54.447028+0000","last_undegraded":"2024-02-19T21:09:54.447028+0000","last_fullsized":"2024-02-19T21:09:54.447028+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.627708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.627708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":323,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446456+0000","last_change":"2024-02-19T21:07:34.550279+0000","last_active":"2024-02-19T21:09:54.446456+0000","last_peered":"2024-02-19T21:09:54.446456+0000","last_clean":"2024-02-19T21:09:54.446456+0000","last_became_active":"2024-02-19T21:04:39.662125+0000","last_became_peered":"2024-02-19T21:04:39.662125+0000","last_unstale":"2024-02-19T21:09:54.446456+0000","last_undegraded":"2024-02-19T21:09:54.446456+0000","last_fullsized":"2024-02-19T21:09:54.446456+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:34.550217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:34.550217+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":371,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.274858+0000","last_change":"2024-02-19T21:06:51.666781+0000","last_active":"2024-02-19T21:09:54.274858+0000","last_peered":"2024-02-19T21:09:54.274858+0000","last_clean":"2024-02-19T21:09:54.274858+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:09:54.274858+0000","last_undegraded":"2024-02-19T21:09:54.274858+0000","last_fullsized":"2024-02-19T21:09:54.274858+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:51.666708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:51.666708+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":418,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447259+0000","last_change":"2024-02-19T21:08:41.312655+0000","last_active":"2024-02-19T21:09:54.447259+0000","last_peered":"2024-02-19T21:09:54.447259+0000","last_clean":"2024-02-19T21:09:54.447259+0000","last_became_active":"2024-02-19T21:08:41.263010+0000","last_became_peered":"2024-02-19T21:08:41.263010+0000","last_unstale":"2024-02-19T21:09:54.447259+0000","last_undegraded":"2024-02-19T21:09:54.447259+0000","last_fullsized":"2024-02-19T21:09:54.447259+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:06:26.776963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:26.776963+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":384,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.273007+0000","last_change":"2024-02-19T21:08:37.935245+0000","last_active":"2024-02-19T21:09:54.273007+0000","last_peered":"2024-02-19T21:09:54.273007+0000","last_clean":"2024-02-19T21:09:54.273007+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:09:54.273007+0000","last_undegraded":"2024-02-19T21:09:54.273007+0000","last_fullsized":"2024-02-19T21:09:54.273007+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:37.935175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:08:37.935175+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":403,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269575+0000","last_change":"2024-02-19T21:07:58.239278+0000","last_active":"2024-02-19T21:09:54.269575+0000","last_peered":"2024-02-19T21:09:54.269575+0000","last_clean":"2024-02-19T21:09:54.269575+0000","last_became_active":"2024-02-19T21:07:58.238944+0000","last_became_peered":"2024-02-19T21:07:58.238944+0000","last_unstale":"2024-02-19T21:09:54.269575+0000","last_undegraded":"2024-02-19T21:09:54.269575+0000","last_fullsized":"2024-02-19T21:09:54.269575+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:17.907318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:17.907318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":123,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270909+0000","last_change":"2024-02-19T21:08:53.820016+0000","last_active":"2024-02-19T21:09:54.270909+0000","last_peered":"2024-02-19T21:09:54.270909+0000","last_clean":"2024-02-19T21:09:54.270909+0000","last_became_active":"2024-02-19T21:08:04.278265+0000","last_became_peered":"2024-02-19T21:08:04.278265+0000","last_unstale":"2024-02-19T21:09:54.270909+0000","last_undegraded":"2024-02-19T21:09:54.270909+0000","last_fullsized":"2024-02-19T21:09:54.270909+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:53.819949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:53.819949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":397,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269852+0000","last_change":"2024-02-19T21:08:41.306530+0000","last_active":"2024-02-19T21:09:54.269852+0000","last_peered":"2024-02-19T21:09:54.269852+0000","last_clean":"2024-02-19T21:09:54.269852+0000","last_became_active":"2024-02-19T21:08:41.305847+0000","last_became_peered":"2024-02-19T21:08:41.305847+0000","last_unstale":"2024-02-19T21:09:54.269852+0000","last_undegraded":"2024-02-19T21:09:54.269852+0000","last_fullsized":"2024-02-19T21:09:54.269852+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:16.863398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:16.863398+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":324,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268574+0000","last_change":"2024-02-19T21:06:52.403019+0000","last_active":"2024-02-19T21:09:54.268574+0000","last_peered":"2024-02-19T21:09:54.268574+0000","last_clean":"2024-02-19T21:09:54.268574+0000","last_became_active":"2024-02-19T21:04:39.666739+0000","last_became_peered":"2024-02-19T21:04:39.666739+0000","last_unstale":"2024-02-19T21:09:54.268574+0000","last_undegraded":"2024-02-19T21:09:54.268574+0000","last_fullsized":"2024-02-19T21:09:54.268574+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:52.402941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:52.402941+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":371,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.282472+0000","last_change":"2024-02-19T21:06:23.273349+0000","last_active":"2024-02-19T21:09:54.282472+0000","last_peered":"2024-02-19T21:09:54.282472+0000","last_clean":"2024-02-19T21:09:54.282472+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:09:54.282472+0000","last_undegraded":"2024-02-19T21:09:54.282472+0000","last_fullsized":"2024-02-19T21:09:54.282472+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:23.273274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:23.273274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":389,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.282693+0000","last_change":"2024-02-19T21:07:01.264194+0000","last_active":"2024-02-19T21:09:54.282693+0000","last_peered":"2024-02-19T21:09:54.282693+0000","last_clean":"2024-02-19T21:09:54.282693+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:09:54.282693+0000","last_undegraded":"2024-02-19T21:09:54.282693+0000","last_fullsized":"2024-02-19T21:09:54.282693+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:07:01.264093+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:07:01.264093+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":421,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.279137+0000","last_change":"2024-02-19T21:09:37.955966+0000","last_active":"2024-02-19T21:09:54.279137+0000","last_peered":"2024-02-19T21:09:54.279137+0000","last_clean":"2024-02-19T21:09:54.279137+0000","last_became_active":"2024-02-19T21:08:06.473207+0000","last_became_peered":"2024-02-19T21:08:06.473207+0000","last_unstale":"2024-02-19T21:09:54.279137+0000","last_undegraded":"2024-02-19T21:09:54.279137+0000","last_fullsized":"2024-02-19T21:09:54.279137+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.955872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.955872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":894,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-02-19T21:09:59.291322+0000","last_change":"2024-02-19T21:06:10.791107+0000","last_active":"2024-02-19T21:09:59.291322+0000","last_peered":"2024-02-19T21:09:59.291322+0000","last_clean":"2024-02-19T21:09:59.291322+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:09:59.291322+0000","last_undegraded":"2024-02-19T21:09:59.291322+0000","last_fullsized":"2024-02-19T21:09:59.291322+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:06:10.791036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:10.791036+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":516,"num_read_kb":511,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":387,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.276011+0000","last_change":"2024-02-19T21:08:04.265242+0000","last_active":"2024-02-19T21:09:54.276011+0000","last_peered":"2024-02-19T21:09:54.276011+0000","last_clean":"2024-02-19T21:09:54.276011+0000","last_became_active":"2024-02-19T21:08:04.264896+0000","last_became_peered":"2024-02-19T21:08:04.264896+0000","last_unstale":"2024-02-19T21:09:54.276011+0000","last_undegraded":"2024-02-19T21:09:54.276011+0000","last_fullsized":"2024-02-19T21:09:54.276011+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:49.690596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.690596+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":323,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447448+0000","last_change":"2024-02-19T21:08:12.584947+0000","last_active":"2024-02-19T21:09:54.447448+0000","last_peered":"2024-02-19T21:09:54.447448+0000","last_clean":"2024-02-19T21:09:54.447448+0000","last_became_active":"2024-02-19T21:04:39.691434+0000","last_became_peered":"2024-02-19T21:04:39.691434+0000","last_unstale":"2024-02-19T21:09:54.447448+0000","last_undegraded":"2024-02-19T21:09:54.447448+0000","last_fullsized":"2024-02-19T21:09:54.447448+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:12.584874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.584874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":215,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447620+0000","last_change":"2024-02-19T21:08:41.309414+0000","last_active":"2024-02-19T21:09:54.447620+0000","last_peered":"2024-02-19T21:09:54.447620+0000","last_clean":"2024-02-19T21:09:54.447620+0000","last_became_active":"2024-02-19T21:08:41.303115+0000","last_became_peered":"2024-02-19T21:08:41.303115+0000","last_unstale":"2024-02-19T21:09:54.447620+0000","last_undegraded":"2024-02-19T21:09:54.447620+0000","last_fullsized":"2024-02-19T21:09:54.447620+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:36.716464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.716464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"304'30","reported_seq":588,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-02-19T21:09:59.292238+0000","last_change":"2024-02-19T21:08:05.292869+0000","last_active":"2024-02-19T21:09:59.292238+0000","last_peered":"2024-02-19T21:09:59.292238+0000","last_clean":"2024-02-19T21:09:59.292238+0000","last_became_active":"2024-02-19T21:08:04.266322+0000","last_became_peered":"2024-02-19T21:08:04.266322+0000","last_unstale":"2024-02-19T21:09:59.292238+0000","last_undegraded":"2024-02-19T21:09:59.292238+0000","last_fullsized":"2024-02-19T21:09:59.292238+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'28","last_scrub_stamp":"2024-02-19T21:05:42.927640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:05:42.927640+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":403,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268996+0000","last_change":"2024-02-19T21:08:02.435986+0000","last_active":"2024-02-19T21:09:54.268996+0000","last_peered":"2024-02-19T21:09:54.268996+0000","last_clean":"2024-02-19T21:09:54.268996+0000","last_became_active":"2024-02-19T21:08:02.435663+0000","last_became_peered":"2024-02-19T21:08:02.435663+0000","last_unstale":"2024-02-19T21:09:54.268996+0000","last_undegraded":"2024-02-19T21:09:54.268996+0000","last_fullsized":"2024-02-19T21:09:54.268996+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:06:19.845394+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":127,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269833+0000","last_change":"2024-02-19T21:08:41.272198+0000","last_active":"2024-02-19T21:09:54.269833+0000","last_peered":"2024-02-19T21:09:54.269833+0000","last_clean":"2024-02-19T21:09:54.269833+0000","last_became_active":"2024-02-19T21:08:41.271603+0000","last_became_peered":"2024-02-19T21:08:41.271603+0000","last_unstale":"2024-02-19T21:09:54.269833+0000","last_undegraded":"2024-02-19T21:09:54.269833+0000","last_fullsized":"2024-02-19T21:09:54.269833+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:26.886817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:07:26.886817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":323,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.276652+0000","last_change":"2024-02-19T21:06:54.355870+0000","last_active":"2024-02-19T21:09:54.276652+0000","last_peered":"2024-02-19T21:09:54.276652+0000","last_clean":"2024-02-19T21:09:54.276652+0000","last_became_active":"2024-02-19T21:04:39.721416+0000","last_became_peered":"2024-02-19T21:04:39.721416+0000","last_unstale":"2024-02-19T21:09:54.276652+0000","last_undegraded":"2024-02-19T21:09:54.276652+0000","last_fullsized":"2024-02-19T21:09:54.276652+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.355809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.355809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"220'9","reported_seq":262,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-02-19T21:09:59.291941+0000","last_change":"2024-02-19T21:06:40.228012+0000","last_active":"2024-02-19T21:09:59.291941+0000","last_peered":"2024-02-19T21:09:59.291941+0000","last_clean":"2024-02-19T21:09:59.291941+0000","last_became_active":"2024-02-19T21:06:40.227604+0000","last_became_peered":"2024-02-19T21:06:40.227604+0000","last_unstale":"2024-02-19T21:09:59.291941+0000","last_undegraded":"2024-02-19T21:09:59.291941+0000","last_fullsized":"2024-02-19T21:09:59.291941+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:06:21.819105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:21.819105+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":409,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446512+0000","last_change":"2024-02-19T21:09:28.659562+0000","last_active":"2024-02-19T21:09:54.446512+0000","last_peered":"2024-02-19T21:09:54.446512+0000","last_clean":"2024-02-19T21:09:54.446512+0000","last_became_active":"2024-02-19T21:08:41.306070+0000","last_became_peered":"2024-02-19T21:08:41.306070+0000","last_unstale":"2024-02-19T21:09:54.446512+0000","last_undegraded":"2024-02-19T21:09:54.446512+0000","last_fullsized":"2024-02-19T21:09:54.446512+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:09:28.659495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.659495+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":420,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446847+0000","last_change":"2024-02-19T21:07:54.235439+0000","last_active":"2024-02-19T21:09:54.446847+0000","last_peered":"2024-02-19T21:09:54.446847+0000","last_clean":"2024-02-19T21:09:54.446847+0000","last_became_active":"2024-02-19T21:07:54.235080+0000","last_became_peered":"2024-02-19T21:07:54.235080+0000","last_unstale":"2024-02-19T21:09:54.446847+0000","last_undegraded":"2024-02-19T21:09:54.446847+0000","last_fullsized":"2024-02-19T21:09:54.446847+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:47.885024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:05:47.885024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":123,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268378+0000","last_change":"2024-02-19T21:09:27.747880+0000","last_active":"2024-02-19T21:09:54.268378+0000","last_peered":"2024-02-19T21:09:54.268378+0000","last_clean":"2024-02-19T21:09:54.268378+0000","last_became_active":"2024-02-19T21:08:04.244317+0000","last_became_peered":"2024-02-19T21:08:04.244317+0000","last_unstale":"2024-02-19T21:09:54.268378+0000","last_undegraded":"2024-02-19T21:09:54.268378+0000","last_fullsized":"2024-02-19T21:09:54.268378+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.747789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.747789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":118,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269910+0000","last_change":"2024-02-19T21:08:04.261805+0000","last_active":"2024-02-19T21:09:54.269910+0000","last_peered":"2024-02-19T21:09:54.269910+0000","last_clean":"2024-02-19T21:09:54.269910+0000","last_became_active":"2024-02-19T21:08:04.261408+0000","last_became_peered":"2024-02-19T21:08:04.261408+0000","last_unstale":"2024-02-19T21:09:54.269910+0000","last_undegraded":"2024-02-19T21:09:54.269910+0000","last_fullsized":"2024-02-19T21:09:54.269910+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:58.726858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:58.726858+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":371,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.276554+0000","last_change":"2024-02-19T21:06:20.194149+0000","last_active":"2024-02-19T21:09:54.276554+0000","last_peered":"2024-02-19T21:09:54.276554+0000","last_clean":"2024-02-19T21:09:54.276554+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:09:54.276554+0000","last_undegraded":"2024-02-19T21:09:54.276554+0000","last_fullsized":"2024-02-19T21:09:54.276554+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:20.194018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:06:20.194018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"65'12","reported_seq":129,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446369+0000","last_change":"2024-02-19T21:08:42.355747+0000","last_active":"2024-02-19T21:09:54.446369+0000","last_peered":"2024-02-19T21:09:54.446369+0000","last_clean":"2024-02-19T21:09:54.446369+0000","last_became_active":"2024-02-19T21:08:41.294283+0000","last_became_peered":"2024-02-19T21:08:41.294283+0000","last_unstale":"2024-02-19T21:09:54.446369+0000","last_undegraded":"2024-02-19T21:09:54.446369+0000","last_fullsized":"2024-02-19T21:09:54.446369+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:06:48.844301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.844301+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":141,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.275202+0000","last_change":"2024-02-19T21:08:41.279224+0000","last_active":"2024-02-19T21:09:54.275202+0000","last_peered":"2024-02-19T21:09:54.275202+0000","last_clean":"2024-02-19T21:09:54.275202+0000","last_became_active":"2024-02-19T21:08:41.269102+0000","last_became_peered":"2024-02-19T21:08:41.269102+0000","last_unstale":"2024-02-19T21:09:54.275202+0000","last_undegraded":"2024-02-19T21:09:54.275202+0000","last_fullsized":"2024-02-19T21:09:54.275202+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:07:33.765035+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:07:33.765035+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":123,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269616+0000","last_change":"2024-02-19T21:08:57.881002+0000","last_active":"2024-02-19T21:09:54.269616+0000","last_peered":"2024-02-19T21:09:54.269616+0000","last_clean":"2024-02-19T21:09:54.269616+0000","last_became_active":"2024-02-19T21:08:04.264593+0000","last_became_peered":"2024-02-19T21:08:04.264593+0000","last_unstale":"2024-02-19T21:09:54.269616+0000","last_undegraded":"2024-02-19T21:09:54.269616+0000","last_fullsized":"2024-02-19T21:09:54.269616+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:57.880935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:08:57.880935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":323,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.281302+0000","last_change":"2024-02-19T21:08:03.177937+0000","last_active":"2024-02-19T21:09:54.281302+0000","last_peered":"2024-02-19T21:09:54.281302+0000","last_clean":"2024-02-19T21:09:54.281302+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:09:54.281302+0000","last_undegraded":"2024-02-19T21:09:54.281302+0000","last_fullsized":"2024-02-19T21:09:54.281302+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:03.177867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:03.177867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":128,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269747+0000","last_change":"2024-02-19T21:08:06.478479+0000","last_active":"2024-02-19T21:09:54.269747+0000","last_peered":"2024-02-19T21:09:54.269747+0000","last_clean":"2024-02-19T21:09:54.269747+0000","last_became_active":"2024-02-19T21:08:04.258740+0000","last_became_peered":"2024-02-19T21:08:04.258740+0000","last_unstale":"2024-02-19T21:09:54.269747+0000","last_undegraded":"2024-02-19T21:09:54.269747+0000","last_fullsized":"2024-02-19T21:09:54.269747+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:06:09.793208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.793208+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":375,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.276501+0000","last_change":"2024-02-19T21:08:11.307651+0000","last_active":"2024-02-19T21:09:54.276501+0000","last_peered":"2024-02-19T21:09:54.276501+0000","last_clean":"2024-02-19T21:09:54.276501+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:09:54.276501+0000","last_undegraded":"2024-02-19T21:09:54.276501+0000","last_fullsized":"2024-02-19T21:09:54.276501+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:11.307561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:11.307561+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":118,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270514+0000","last_change":"2024-02-19T21:08:04.265732+0000","last_active":"2024-02-19T21:09:54.270514+0000","last_peered":"2024-02-19T21:09:54.270514+0000","last_clean":"2024-02-19T21:09:54.270514+0000","last_became_active":"2024-02-19T21:08:04.264972+0000","last_became_peered":"2024-02-19T21:08:04.264972+0000","last_unstale":"2024-02-19T21:09:54.270514+0000","last_undegraded":"2024-02-19T21:09:54.270514+0000","last_fullsized":"2024-02-19T21:09:54.270514+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:56.709750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:06:56.709750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":383,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.283958+0000","last_change":"2024-02-19T21:08:48.995892+0000","last_active":"2024-02-19T21:09:54.283958+0000","last_peered":"2024-02-19T21:09:54.283958+0000","last_clean":"2024-02-19T21:09:54.283958+0000","last_became_active":"2024-02-19T21:08:41.286874+0000","last_became_peered":"2024-02-19T21:08:41.286874+0000","last_unstale":"2024-02-19T21:09:54.283958+0000","last_undegraded":"2024-02-19T21:09:54.283958+0000","last_fullsized":"2024-02-19T21:09:54.283958+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:48.995828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:08:48.995828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":82,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446922+0000","last_change":"2024-02-19T21:08:41.311611+0000","last_active":"2024-02-19T21:09:54.446922+0000","last_peered":"2024-02-19T21:09:54.446922+0000","last_clean":"2024-02-19T21:09:54.446922+0000","last_became_active":"2024-02-19T21:08:41.311298+0000","last_became_peered":"2024-02-19T21:08:41.311298+0000","last_unstale":"2024-02-19T21:09:54.446922+0000","last_undegraded":"2024-02-19T21:09:54.446922+0000","last_fullsized":"2024-02-19T21:09:54.446922+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:21.704719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:21.704719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":126,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270436+0000","last_change":"2024-02-19T21:08:04.356074+0000","last_active":"2024-02-19T21:09:54.270436+0000","last_peered":"2024-02-19T21:09:54.270436+0000","last_clean":"2024-02-19T21:09:54.270436+0000","last_became_active":"2024-02-19T21:08:04.264472+0000","last_became_peered":"2024-02-19T21:08:04.264472+0000","last_unstale":"2024-02-19T21:09:54.270436+0000","last_undegraded":"2024-02-19T21:09:54.270436+0000","last_fullsized":"2024-02-19T21:09:54.270436+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:06:48.855268+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.855268+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":387,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446450+0000","last_change":"2024-02-19T21:09:41.519382+0000","last_active":"2024-02-19T21:09:54.446450+0000","last_peered":"2024-02-19T21:09:54.446450+0000","last_clean":"2024-02-19T21:09:54.446450+0000","last_became_active":"2024-02-19T21:08:41.295743+0000","last_became_peered":"2024-02-19T21:08:41.295743+0000","last_unstale":"2024-02-19T21:09:54.446450+0000","last_undegraded":"2024-02-19T21:09:54.446450+0000","last_fullsized":"2024-02-19T21:09:54.446450+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:41.519294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:41.519294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":392,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270620+0000","last_change":"2024-02-19T21:08:56.601041+0000","last_active":"2024-02-19T21:09:54.270620+0000","last_peered":"2024-02-19T21:09:54.270620+0000","last_clean":"2024-02-19T21:09:54.270620+0000","last_became_active":"2024-02-19T21:08:04.265573+0000","last_became_peered":"2024-02-19T21:08:04.265573+0000","last_unstale":"2024-02-19T21:09:54.270620+0000","last_undegraded":"2024-02-19T21:09:54.270620+0000","last_fullsized":"2024-02-19T21:09:54.270620+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:56.600951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:56.600951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":383,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.275864+0000","last_change":"2024-02-19T21:09:27.770579+0000","last_active":"2024-02-19T21:09:54.275864+0000","last_peered":"2024-02-19T21:09:54.275864+0000","last_clean":"2024-02-19T21:09:54.275864+0000","last_became_active":"2024-02-19T21:08:41.268962+0000","last_became_peered":"2024-02-19T21:08:41.268962+0000","last_unstale":"2024-02-19T21:09:54.275864+0000","last_undegraded":"2024-02-19T21:09:54.275864+0000","last_fullsized":"2024-02-19T21:09:54.275864+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.770501+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":333,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269297+0000","last_change":"2024-02-19T21:08:50.600747+0000","last_active":"2024-02-19T21:09:54.269297+0000","last_peered":"2024-02-19T21:09:54.269297+0000","last_clean":"2024-02-19T21:09:54.269297+0000","last_became_active":"2024-02-19T21:08:41.292646+0000","last_became_peered":"2024-02-19T21:08:41.292646+0000","last_unstale":"2024-02-19T21:09:54.269297+0000","last_undegraded":"2024-02-19T21:09:54.269297+0000","last_fullsized":"2024-02-19T21:09:54.269297+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:50.600659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:50.600659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":130,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.271315+0000","last_change":"2024-02-19T21:08:04.361409+0000","last_active":"2024-02-19T21:09:54.271315+0000","last_peered":"2024-02-19T21:09:54.271315+0000","last_clean":"2024-02-19T21:09:54.271315+0000","last_became_active":"2024-02-19T21:08:04.267216+0000","last_became_peered":"2024-02-19T21:08:04.267216+0000","last_unstale":"2024-02-19T21:09:54.271315+0000","last_undegraded":"2024-02-19T21:09:54.271315+0000","last_fullsized":"2024-02-19T21:09:54.271315+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:07:01.731068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:07:01.731068+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":399,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268914+0000","last_change":"2024-02-19T21:09:40.986717+0000","last_active":"2024-02-19T21:09:54.268914+0000","last_peered":"2024-02-19T21:09:54.268914+0000","last_clean":"2024-02-19T21:09:54.268914+0000","last_became_active":"2024-02-19T21:08:04.266728+0000","last_became_peered":"2024-02-19T21:08:04.266728+0000","last_unstale":"2024-02-19T21:09:54.268914+0000","last_undegraded":"2024-02-19T21:09:54.268914+0000","last_fullsized":"2024-02-19T21:09:54.268914+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.986635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"65'8","reported_seq":485,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-02-19T21:09:59.257702+0000","last_change":"2024-02-19T21:09:26.916067+0000","last_active":"2024-02-19T21:09:59.257702+0000","last_peered":"2024-02-19T21:09:59.257702+0000","last_clean":"2024-02-19T21:09:59.257702+0000","last_became_active":"2024-02-19T21:01:20.288799+0000","last_became_peered":"2024-02-19T21:01:20.288799+0000","last_unstale":"2024-02-19T21:09:59.257702+0000","last_undegraded":"2024-02-19T21:09:59.257702+0000","last_fullsized":"2024-02-19T21:09:59.257702+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:09:26.916002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:26.916002+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":242,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.275229+0000","last_change":"2024-02-19T21:08:57.007906+0000","last_active":"2024-02-19T21:09:54.275229+0000","last_peered":"2024-02-19T21:09:54.275229+0000","last_clean":"2024-02-19T21:09:54.275229+0000","last_became_active":"2024-02-19T21:08:04.266747+0000","last_became_peered":"2024-02-19T21:08:04.266747+0000","last_unstale":"2024-02-19T21:09:54.275229+0000","last_undegraded":"2024-02-19T21:09:54.275229+0000","last_fullsized":"2024-02-19T21:09:54.275229+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:08:57.007815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:08:57.007815+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":204,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446408+0000","last_change":"2024-02-19T21:08:55.202543+0000","last_active":"2024-02-19T21:09:54.446408+0000","last_peered":"2024-02-19T21:09:54.446408+0000","last_clean":"2024-02-19T21:09:54.446408+0000","last_became_active":"2024-02-19T21:06:40.219590+0000","last_became_peered":"2024-02-19T21:06:40.219590+0000","last_unstale":"2024-02-19T21:09:54.446408+0000","last_undegraded":"2024-02-19T21:09:54.446408+0000","last_fullsized":"2024-02-19T21:09:54.446408+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:55.202463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:55.202463+0000","last_clean_scrub_stamp":"2024-02-19T21:08:55.202463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":428,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269929+0000","last_change":"2024-02-19T21:08:04.681714+0000","last_active":"2024-02-19T21:09:54.269929+0000","last_peered":"2024-02-19T21:09:54.269929+0000","last_clean":"2024-02-19T21:09:54.269929+0000","last_became_active":"2024-02-19T21:08:04.266103+0000","last_became_peered":"2024-02-19T21:08:04.266103+0000","last_unstale":"2024-02-19T21:09:54.269929+0000","last_undegraded":"2024-02-19T21:09:54.269929+0000","last_fullsized":"2024-02-19T21:09:54.269929+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:06:47.541012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:47.541012+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":192,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269275+0000","last_change":"2024-02-19T21:08:41.280409+0000","last_active":"2024-02-19T21:09:54.269275+0000","last_peered":"2024-02-19T21:09:54.269275+0000","last_clean":"2024-02-19T21:09:54.269275+0000","last_became_active":"2024-02-19T21:08:41.279782+0000","last_became_peered":"2024-02-19T21:08:41.279782+0000","last_unstale":"2024-02-19T21:09:54.269275+0000","last_undegraded":"2024-02-19T21:09:54.269275+0000","last_fullsized":"2024-02-19T21:09:54.269275+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:30.769388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:07:30.769388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":151,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-02-19T21:09:59.261486+0000","last_change":"2024-02-19T21:08:37.867477+0000","last_active":"2024-02-19T21:09:59.261486+0000","last_peered":"2024-02-19T21:09:59.261486+0000","last_clean":"2024-02-19T21:09:59.261486+0000","last_became_active":"2024-02-19T21:08:04.256045+0000","last_became_peered":"2024-02-19T21:08:04.256045+0000","last_unstale":"2024-02-19T21:09:59.261486+0000","last_undegraded":"2024-02-19T21:09:59.261486+0000","last_fullsized":"2024-02-19T21:09:59.261486+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'10","last_scrub_stamp":"2024-02-19T21:08:37.867405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:37.867405+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":86,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.282684+0000","last_change":"2024-02-19T21:09:43.936218+0000","last_active":"2024-02-19T21:09:54.282684+0000","last_peered":"2024-02-19T21:09:54.282684+0000","last_clean":"2024-02-19T21:09:54.282684+0000","last_became_active":"2024-02-19T21:08:41.286185+0000","last_became_peered":"2024-02-19T21:08:41.286185+0000","last_unstale":"2024-02-19T21:09:54.282684+0000","last_undegraded":"2024-02-19T21:09:54.282684+0000","last_fullsized":"2024-02-19T21:09:54.282684+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_clean_scrub_stamp":"2024-02-19T21:09:43.936153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":336,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.278759+0000","last_change":"2024-02-19T21:06:49.305197+0000","last_active":"2024-02-19T21:09:54.278759+0000","last_peered":"2024-02-19T21:09:54.278759+0000","last_clean":"2024-02-19T21:09:54.278759+0000","last_became_active":"2024-02-19T21:04:39.760167+0000","last_became_peered":"2024-02-19T21:04:39.760167+0000","last_unstale":"2024-02-19T21:09:54.278759+0000","last_undegraded":"2024-02-19T21:09:54.278759+0000","last_fullsized":"2024-02-19T21:09:54.278759+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-02-19T21:06:49.305118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:49.305118+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":110,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446143+0000","last_change":"2024-02-19T21:09:40.676522+0000","last_active":"2024-02-19T21:09:54.446143+0000","last_peered":"2024-02-19T21:09:54.446143+0000","last_clean":"2024-02-19T21:09:54.446143+0000","last_became_active":"2024-02-19T21:08:26.235018+0000","last_became_peered":"2024-02-19T21:08:26.235018+0000","last_unstale":"2024-02-19T21:09:54.446143+0000","last_undegraded":"2024-02-19T21:09:54.446143+0000","last_fullsized":"2024-02-19T21:09:54.446143+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:09:40.676439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.676439+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":396,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447155+0000","last_change":"2024-02-19T21:08:09.588799+0000","last_active":"2024-02-19T21:09:54.447155+0000","last_peered":"2024-02-19T21:09:54.447155+0000","last_clean":"2024-02-19T21:09:54.447155+0000","last_became_active":"2024-02-19T21:08:04.258699+0000","last_became_peered":"2024-02-19T21:08:04.258699+0000","last_unstale":"2024-02-19T21:09:54.447155+0000","last_undegraded":"2024-02-19T21:09:54.447155+0000","last_fullsized":"2024-02-19T21:09:54.447155+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:09.588687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:08:09.588687+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":123,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269743+0000","last_change":"2024-02-19T21:08:12.856127+0000","last_active":"2024-02-19T21:09:54.269743+0000","last_peered":"2024-02-19T21:09:54.269743+0000","last_clean":"2024-02-19T21:09:54.269743+0000","last_became_active":"2024-02-19T21:08:04.264983+0000","last_became_peered":"2024-02-19T21:08:04.264983+0000","last_unstale":"2024-02-19T21:09:54.269743+0000","last_undegraded":"2024-02-19T21:09:54.269743+0000","last_fullsized":"2024-02-19T21:09:54.269743+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.856015+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":207,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268272+0000","last_change":"2024-02-19T21:09:38.941012+0000","last_active":"2024-02-19T21:09:54.268272+0000","last_peered":"2024-02-19T21:09:54.268272+0000","last_clean":"2024-02-19T21:09:54.268272+0000","last_became_active":"2024-02-19T21:06:40.239199+0000","last_became_peered":"2024-02-19T21:06:40.239199+0000","last_unstale":"2024-02-19T21:09:54.268272+0000","last_undegraded":"2024-02-19T21:09:54.268272+0000","last_fullsized":"2024-02-19T21:09:54.268272+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:38.940935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:38.940935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":321,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447754+0000","last_change":"2024-02-19T21:07:35.505665+0000","last_active":"2024-02-19T21:09:54.447754+0000","last_peered":"2024-02-19T21:09:54.447754+0000","last_clean":"2024-02-19T21:09:54.447754+0000","last_became_active":"2024-02-19T21:04:39.672228+0000","last_became_peered":"2024-02-19T21:04:39.672228+0000","last_unstale":"2024-02-19T21:09:54.447754+0000","last_undegraded":"2024-02-19T21:09:54.447754+0000","last_fullsized":"2024-02-19T21:09:54.447754+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:35.505607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:35.505607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":129,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446875+0000","last_change":"2024-02-19T21:08:52.202056+0000","last_active":"2024-02-19T21:09:54.446875+0000","last_peered":"2024-02-19T21:09:54.446875+0000","last_clean":"2024-02-19T21:09:54.446875+0000","last_became_active":"2024-02-19T21:08:41.295736+0000","last_became_peered":"2024-02-19T21:08:41.295736+0000","last_unstale":"2024-02-19T21:09:54.446875+0000","last_undegraded":"2024-02-19T21:09:54.446875+0000","last_fullsized":"2024-02-19T21:09:54.446875+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:52.201972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:52.201972+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":380,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447792+0000","last_change":"2024-02-19T21:09:32.623020+0000","last_active":"2024-02-19T21:09:54.447792+0000","last_peered":"2024-02-19T21:09:54.447792+0000","last_clean":"2024-02-19T21:09:54.447792+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:09:54.447792+0000","last_undegraded":"2024-02-19T21:09:54.447792+0000","last_fullsized":"2024-02-19T21:09:54.447792+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:32.622939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:09:32.622939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":394,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446614+0000","last_change":"2024-02-19T21:09:34.642491+0000","last_active":"2024-02-19T21:09:54.446614+0000","last_peered":"2024-02-19T21:09:54.446614+0000","last_clean":"2024-02-19T21:09:54.446614+0000","last_became_active":"2024-02-19T21:08:04.268847+0000","last_became_peered":"2024-02-19T21:08:04.268847+0000","last_unstale":"2024-02-19T21:09:54.446614+0000","last_undegraded":"2024-02-19T21:09:54.446614+0000","last_fullsized":"2024-02-19T21:09:54.446614+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:34.642432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:34.642432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":118,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270451+0000","last_change":"2024-02-19T21:08:04.280460+0000","last_active":"2024-02-19T21:09:54.270451+0000","last_peered":"2024-02-19T21:09:54.270451+0000","last_clean":"2024-02-19T21:09:54.270451+0000","last_became_active":"2024-02-19T21:08:04.280233+0000","last_became_peered":"2024-02-19T21:08:04.280233+0000","last_unstale":"2024-02-19T21:09:54.270451+0000","last_undegraded":"2024-02-19T21:09:54.270451+0000","last_fullsized":"2024-02-19T21:09:54.270451+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:59.446267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:07:59.446267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":118,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270574+0000","last_change":"2024-02-19T21:08:04.258981+0000","last_active":"2024-02-19T21:09:54.270574+0000","last_peered":"2024-02-19T21:09:54.270574+0000","last_clean":"2024-02-19T21:09:54.270574+0000","last_became_active":"2024-02-19T21:08:04.258568+0000","last_became_peered":"2024-02-19T21:08:04.258568+0000","last_unstale":"2024-02-19T21:09:54.270574+0000","last_undegraded":"2024-02-19T21:09:54.270574+0000","last_fullsized":"2024-02-19T21:09:54.270574+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:29.946015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:29.946015+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":129,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268328+0000","last_change":"2024-02-19T21:09:28.712106+0000","last_active":"2024-02-19T21:09:54.268328+0000","last_peered":"2024-02-19T21:09:54.268328+0000","last_clean":"2024-02-19T21:09:54.268328+0000","last_became_active":"2024-02-19T21:08:04.274921+0000","last_became_peered":"2024-02-19T21:08:04.274921+0000","last_unstale":"2024-02-19T21:09:54.268328+0000","last_undegraded":"2024-02-19T21:09:54.268328+0000","last_fullsized":"2024-02-19T21:09:54.268328+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:09:28.712037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.712037+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":381,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270691+0000","last_change":"2024-02-19T21:09:42.000906+0000","last_active":"2024-02-19T21:09:54.270691+0000","last_peered":"2024-02-19T21:09:54.270691+0000","last_clean":"2024-02-19T21:09:54.270691+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:09:54.270691+0000","last_undegraded":"2024-02-19T21:09:54.270691+0000","last_fullsized":"2024-02-19T21:09:54.270691+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:42.000850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:42.000850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":383,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446492+0000","last_change":"2024-02-19T21:08:41.261809+0000","last_active":"2024-02-19T21:09:54.446492+0000","last_peered":"2024-02-19T21:09:54.446492+0000","last_clean":"2024-02-19T21:09:54.446492+0000","last_became_active":"2024-02-19T21:08:41.261268+0000","last_became_peered":"2024-02-19T21:08:41.261268+0000","last_unstale":"2024-02-19T21:09:54.446492+0000","last_undegraded":"2024-02-19T21:09:54.446492+0000","last_fullsized":"2024-02-19T21:09:54.446492+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:11.593368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:11.593368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":383,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.273476+0000","last_change":"2024-02-19T21:09:26.731948+0000","last_active":"2024-02-19T21:09:54.273476+0000","last_peered":"2024-02-19T21:09:54.273476+0000","last_clean":"2024-02-19T21:09:54.273476+0000","last_became_active":"2024-02-19T21:08:41.274144+0000","last_became_peered":"2024-02-19T21:08:41.274144+0000","last_unstale":"2024-02-19T21:09:54.273476+0000","last_undegraded":"2024-02-19T21:09:54.273476+0000","last_fullsized":"2024-02-19T21:09:54.273476+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:26.731860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:26.731860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":123,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.271386+0000","last_change":"2024-02-19T21:08:10.824761+0000","last_active":"2024-02-19T21:09:54.271386+0000","last_peered":"2024-02-19T21:09:54.271386+0000","last_clean":"2024-02-19T21:09:54.271386+0000","last_became_active":"2024-02-19T21:08:04.266161+0000","last_became_peered":"2024-02-19T21:08:04.266161+0000","last_unstale":"2024-02-19T21:09:54.271386+0000","last_undegraded":"2024-02-19T21:09:54.271386+0000","last_fullsized":"2024-02-19T21:09:54.271386+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:10.824694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:10.824694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":135,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270766+0000","last_change":"2024-02-19T21:08:11.828624+0000","last_active":"2024-02-19T21:09:54.270766+0000","last_peered":"2024-02-19T21:09:54.270766+0000","last_clean":"2024-02-19T21:09:54.270766+0000","last_became_active":"2024-02-19T21:08:04.250742+0000","last_became_peered":"2024-02-19T21:08:04.250742+0000","last_unstale":"2024-02-19T21:09:54.270766+0000","last_undegraded":"2024-02-19T21:09:54.270766+0000","last_fullsized":"2024-02-19T21:09:54.270766+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:11.828537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:11.828537+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":378,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447407+0000","last_change":"2024-02-19T21:08:40.270859+0000","last_active":"2024-02-19T21:09:54.447407+0000","last_peered":"2024-02-19T21:09:54.447407+0000","last_clean":"2024-02-19T21:09:54.447407+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:09:54.447407+0000","last_undegraded":"2024-02-19T21:09:54.447407+0000","last_fullsized":"2024-02-19T21:09:54.447407+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:40.270758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:40.270758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":380,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.273173+0000","last_change":"2024-02-19T21:09:29.783688+0000","last_active":"2024-02-19T21:09:54.273173+0000","last_peered":"2024-02-19T21:09:54.273173+0000","last_clean":"2024-02-19T21:09:54.273173+0000","last_became_active":"2024-02-19T21:08:41.267236+0000","last_became_peered":"2024-02-19T21:08:41.267236+0000","last_unstale":"2024-02-19T21:09:54.273173+0000","last_undegraded":"2024-02-19T21:09:54.273173+0000","last_fullsized":"2024-02-19T21:09:54.273173+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:29.783609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:29.783609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":118,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269820+0000","last_change":"2024-02-19T21:08:04.254279+0000","last_active":"2024-02-19T21:09:54.269820+0000","last_peered":"2024-02-19T21:09:54.269820+0000","last_clean":"2024-02-19T21:09:54.269820+0000","last_became_active":"2024-02-19T21:08:04.251987+0000","last_became_peered":"2024-02-19T21:08:04.251987+0000","last_unstale":"2024-02-19T21:09:54.269820+0000","last_undegraded":"2024-02-19T21:09:54.269820+0000","last_fullsized":"2024-02-19T21:09:54.269820+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:08.757198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:08.757198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":204,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447354+0000","last_change":"2024-02-19T21:06:54.664879+0000","last_active":"2024-02-19T21:09:54.447354+0000","last_peered":"2024-02-19T21:09:54.447354+0000","last_clean":"2024-02-19T21:09:54.447354+0000","last_became_active":"2024-02-19T21:06:40.226196+0000","last_became_peered":"2024-02-19T21:06:40.226196+0000","last_unstale":"2024-02-19T21:09:54.447354+0000","last_undegraded":"2024-02-19T21:09:54.447354+0000","last_fullsized":"2024-02-19T21:09:54.447354+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.664814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":276,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.275743+0000","last_change":"2024-02-19T21:09:30.782123+0000","last_active":"2024-02-19T21:09:54.275743+0000","last_peered":"2024-02-19T21:09:54.275743+0000","last_clean":"2024-02-19T21:09:54.275743+0000","last_became_active":"2024-02-19T21:08:04.267222+0000","last_became_peered":"2024-02-19T21:08:04.267222+0000","last_unstale":"2024-02-19T21:09:54.275743+0000","last_undegraded":"2024-02-19T21:09:54.275743+0000","last_fullsized":"2024-02-19T21:09:54.275743+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:09:30.782002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:30.782002+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":385,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268212+0000","last_change":"2024-02-19T21:08:52.579253+0000","last_active":"2024-02-19T21:09:54.268212+0000","last_peered":"2024-02-19T21:09:54.268212+0000","last_clean":"2024-02-19T21:09:54.268212+0000","last_became_active":"2024-02-19T21:08:41.264843+0000","last_became_peered":"2024-02-19T21:08:41.264843+0000","last_unstale":"2024-02-19T21:09:54.268212+0000","last_undegraded":"2024-02-19T21:09:54.268212+0000","last_fullsized":"2024-02-19T21:09:54.268212+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:52.579178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:52.579178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"344'9","reported_seq":503,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-02-19T21:09:59.292481+0000","last_change":"2024-02-19T21:08:41.319305+0000","last_active":"2024-02-19T21:09:59.292481+0000","last_peered":"2024-02-19T21:09:59.292481+0000","last_clean":"2024-02-19T21:09:59.292481+0000","last_became_active":"2024-02-19T21:08:41.281854+0000","last_became_peered":"2024-02-19T21:08:41.281854+0000","last_unstale":"2024-02-19T21:09:59.292481+0000","last_undegraded":"2024-02-19T21:09:59.292481+0000","last_fullsized":"2024-02-19T21:09:59.292481+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:08:36.251958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.251958+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":376,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.275088+0000","last_change":"2024-02-19T21:09:41.951642+0000","last_active":"2024-02-19T21:09:54.275088+0000","last_peered":"2024-02-19T21:09:54.275088+0000","last_clean":"2024-02-19T21:09:54.275088+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:09:54.275088+0000","last_undegraded":"2024-02-19T21:09:54.275088+0000","last_fullsized":"2024-02-19T21:09:54.275088+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:09:41.951576+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:09:41.951576+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":326,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.280208+0000","last_change":"2024-02-19T21:08:39.107325+0000","last_active":"2024-02-19T21:09:54.280208+0000","last_peered":"2024-02-19T21:09:54.280208+0000","last_clean":"2024-02-19T21:09:54.280208+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:09:54.280208+0000","last_undegraded":"2024-02-19T21:09:54.280208+0000","last_fullsized":"2024-02-19T21:09:54.280208+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:39.107240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.107240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":395,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447500+0000","last_change":"2024-02-19T21:08:03.561292+0000","last_active":"2024-02-19T21:09:54.447500+0000","last_peered":"2024-02-19T21:09:54.447500+0000","last_clean":"2024-02-19T21:09:54.447500+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:09:54.447500+0000","last_undegraded":"2024-02-19T21:09:54.447500+0000","last_fullsized":"2024-02-19T21:09:54.447500+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:03.561223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:03.561223+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":118,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270747+0000","last_change":"2024-02-19T21:08:04.254214+0000","last_active":"2024-02-19T21:09:54.270747+0000","last_peered":"2024-02-19T21:09:54.270747+0000","last_clean":"2024-02-19T21:09:54.270747+0000","last_became_active":"2024-02-19T21:08:04.251241+0000","last_became_peered":"2024-02-19T21:08:04.251241+0000","last_unstale":"2024-02-19T21:09:54.270747+0000","last_undegraded":"2024-02-19T21:09:54.270747+0000","last_fullsized":"2024-02-19T21:09:54.270747+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:01.530618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:01.530618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":207,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.275867+0000","last_change":"2024-02-19T21:08:36.903762+0000","last_active":"2024-02-19T21:09:54.275867+0000","last_peered":"2024-02-19T21:09:54.275867+0000","last_clean":"2024-02-19T21:09:54.275867+0000","last_became_active":"2024-02-19T21:06:40.234026+0000","last_became_peered":"2024-02-19T21:06:40.234026+0000","last_unstale":"2024-02-19T21:09:54.275867+0000","last_undegraded":"2024-02-19T21:09:54.275867+0000","last_fullsized":"2024-02-19T21:09:54.275867+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:36.903674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.903674+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":378,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447535+0000","last_change":"2024-02-19T21:08:48.760695+0000","last_active":"2024-02-19T21:09:54.447535+0000","last_peered":"2024-02-19T21:09:54.447535+0000","last_clean":"2024-02-19T21:09:54.447535+0000","last_became_active":"2024-02-19T21:08:41.309795+0000","last_became_peered":"2024-02-19T21:08:41.309795+0000","last_unstale":"2024-02-19T21:09:54.447535+0000","last_undegraded":"2024-02-19T21:09:54.447535+0000","last_fullsized":"2024-02-19T21:09:54.447535+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:48.760625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:08:48.760625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":332,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447353+0000","last_change":"2024-02-19T21:08:41.310765+0000","last_active":"2024-02-19T21:09:54.447353+0000","last_peered":"2024-02-19T21:09:54.447353+0000","last_clean":"2024-02-19T21:09:54.447353+0000","last_became_active":"2024-02-19T21:08:41.310546+0000","last_became_peered":"2024-02-19T21:08:41.310546+0000","last_unstale":"2024-02-19T21:09:54.447353+0000","last_undegraded":"2024-02-19T21:09:54.447353+0000","last_fullsized":"2024-02-19T21:09:54.447353+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:33.686917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:33.686917+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":448,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.284057+0000","last_change":"2024-02-19T21:08:04.337335+0000","last_active":"2024-02-19T21:09:54.284057+0000","last_peered":"2024-02-19T21:09:54.284057+0000","last_clean":"2024-02-19T21:09:54.284057+0000","last_became_active":"2024-02-19T21:08:04.248378+0000","last_became_peered":"2024-02-19T21:08:04.248378+0000","last_unstale":"2024-02-19T21:09:54.284057+0000","last_undegraded":"2024-02-19T21:09:54.284057+0000","last_fullsized":"2024-02-19T21:09:54.284057+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:06:48.310459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.310459+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":387,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446830+0000","last_change":"2024-02-19T21:09:26.417332+0000","last_active":"2024-02-19T21:09:54.446830+0000","last_peered":"2024-02-19T21:09:54.446830+0000","last_clean":"2024-02-19T21:09:54.446830+0000","last_became_active":"2024-02-19T21:08:41.278751+0000","last_became_peered":"2024-02-19T21:08:41.278751+0000","last_unstale":"2024-02-19T21:09:54.446830+0000","last_undegraded":"2024-02-19T21:09:54.446830+0000","last_fullsized":"2024-02-19T21:09:54.446830+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:26.417247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:26.417247+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":380,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.448613+0000","last_change":"2024-02-19T21:08:41.308944+0000","last_active":"2024-02-19T21:09:54.448613+0000","last_peered":"2024-02-19T21:09:54.448613+0000","last_clean":"2024-02-19T21:09:54.448613+0000","last_became_active":"2024-02-19T21:08:41.302364+0000","last_became_peered":"2024-02-19T21:08:41.302364+0000","last_unstale":"2024-02-19T21:09:54.448613+0000","last_undegraded":"2024-02-19T21:09:54.448613+0000","last_fullsized":"2024-02-19T21:09:54.448613+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:53.629475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:06:53.629475+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":81,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447470+0000","last_change":"2024-02-19T21:08:41.279946+0000","last_active":"2024-02-19T21:09:54.447470+0000","last_peered":"2024-02-19T21:09:54.447470+0000","last_clean":"2024-02-19T21:09:54.447470+0000","last_became_active":"2024-02-19T21:08:41.278358+0000","last_became_peered":"2024-02-19T21:08:41.278358+0000","last_unstale":"2024-02-19T21:09:54.447470+0000","last_undegraded":"2024-02-19T21:09:54.447470+0000","last_fullsized":"2024-02-19T21:09:54.447470+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:36.723492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:07:36.723492+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":204,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446326+0000","last_change":"2024-02-19T21:09:27.464040+0000","last_active":"2024-02-19T21:09:54.446326+0000","last_peered":"2024-02-19T21:09:54.446326+0000","last_clean":"2024-02-19T21:09:54.446326+0000","last_became_active":"2024-02-19T21:06:40.240859+0000","last_became_peered":"2024-02-19T21:06:40.240859+0000","last_unstale":"2024-02-19T21:09:54.446326+0000","last_undegraded":"2024-02-19T21:09:54.446326+0000","last_fullsized":"2024-02-19T21:09:54.446326+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.463966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.463966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":156,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269169+0000","last_change":"2024-02-19T21:08:49.869657+0000","last_active":"2024-02-19T21:09:54.269169+0000","last_peered":"2024-02-19T21:09:54.269169+0000","last_clean":"2024-02-19T21:09:54.269169+0000","last_became_active":"2024-02-19T21:08:41.295566+0000","last_became_peered":"2024-02-19T21:08:41.295566+0000","last_unstale":"2024-02-19T21:09:54.269169+0000","last_undegraded":"2024-02-19T21:09:54.269169+0000","last_fullsized":"2024-02-19T21:09:54.269169+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:08:49.869588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.869588+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":118,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269878+0000","last_change":"2024-02-19T21:08:04.254443+0000","last_active":"2024-02-19T21:09:54.269878+0000","last_peered":"2024-02-19T21:09:54.269878+0000","last_clean":"2024-02-19T21:09:54.269878+0000","last_became_active":"2024-02-19T21:08:04.253058+0000","last_became_peered":"2024-02-19T21:08:04.253058+0000","last_unstale":"2024-02-19T21:09:54.269878+0000","last_undegraded":"2024-02-19T21:09:54.269878+0000","last_fullsized":"2024-02-19T21:09:54.269878+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:02.693025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:02.693025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":125,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.271203+0000","last_change":"2024-02-19T21:09:35.626645+0000","last_active":"2024-02-19T21:09:54.271203+0000","last_peered":"2024-02-19T21:09:54.271203+0000","last_clean":"2024-02-19T21:09:54.271203+0000","last_became_active":"2024-02-19T21:08:41.288781+0000","last_became_peered":"2024-02-19T21:08:41.288781+0000","last_unstale":"2024-02-19T21:09:54.271203+0000","last_undegraded":"2024-02-19T21:09:54.271203+0000","last_fullsized":"2024-02-19T21:09:54.271203+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:35.626548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:35.626548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":118,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270596+0000","last_change":"2024-02-19T21:08:04.278426+0000","last_active":"2024-02-19T21:09:54.270596+0000","last_peered":"2024-02-19T21:09:54.270596+0000","last_clean":"2024-02-19T21:09:54.270596+0000","last_became_active":"2024-02-19T21:08:04.278045+0000","last_became_peered":"2024-02-19T21:08:04.278045+0000","last_unstale":"2024-02-19T21:09:54.270596+0000","last_undegraded":"2024-02-19T21:09:54.270596+0000","last_fullsized":"2024-02-19T21:09:54.270596+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:06:54.723156+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":323,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447088+0000","last_change":"2024-02-19T21:07:25.735056+0000","last_active":"2024-02-19T21:09:54.447088+0000","last_peered":"2024-02-19T21:09:54.447088+0000","last_clean":"2024-02-19T21:09:54.447088+0000","last_became_active":"2024-02-19T21:04:39.777442+0000","last_became_peered":"2024-02-19T21:04:39.777442+0000","last_unstale":"2024-02-19T21:09:54.447088+0000","last_undegraded":"2024-02-19T21:09:54.447088+0000","last_fullsized":"2024-02-19T21:09:54.447088+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:25.734988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:25.734988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"194'13","reported_seq":399,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270681+0000","last_change":"2024-02-19T21:08:41.890452+0000","last_active":"2024-02-19T21:09:54.270681+0000","last_peered":"2024-02-19T21:09:54.270681+0000","last_clean":"2024-02-19T21:09:54.270681+0000","last_became_active":"2024-02-19T21:08:41.283514+0000","last_became_peered":"2024-02-19T21:08:41.283514+0000","last_unstale":"2024-02-19T21:09:54.270681+0000","last_undegraded":"2024-02-19T21:09:54.270681+0000","last_fullsized":"2024-02-19T21:09:54.270681+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'13","last_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:06:14.772417+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":375,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.274274+0000","last_change":"2024-02-19T21:08:51.955830+0000","last_active":"2024-02-19T21:09:54.274274+0000","last_peered":"2024-02-19T21:09:54.274274+0000","last_clean":"2024-02-19T21:09:54.274274+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:09:54.274274+0000","last_undegraded":"2024-02-19T21:09:54.274274+0000","last_fullsized":"2024-02-19T21:09:54.274274+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:51.955738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:51.955738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":372,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269499+0000","last_change":"2024-02-19T21:09:29.852892+0000","last_active":"2024-02-19T21:09:54.269499+0000","last_peered":"2024-02-19T21:09:54.269499+0000","last_clean":"2024-02-19T21:09:54.269499+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:09:54.269499+0000","last_undegraded":"2024-02-19T21:09:54.269499+0000","last_fullsized":"2024-02-19T21:09:54.269499+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:29.852792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:29.852792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":201,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446199+0000","last_change":"2024-02-19T21:06:40.239902+0000","last_active":"2024-02-19T21:09:54.446199+0000","last_peered":"2024-02-19T21:09:54.446199+0000","last_clean":"2024-02-19T21:09:54.446199+0000","last_became_active":"2024-02-19T21:06:40.239603+0000","last_became_peered":"2024-02-19T21:06:40.239603+0000","last_unstale":"2024-02-19T21:09:54.446199+0000","last_undegraded":"2024-02-19T21:09:54.446199+0000","last_fullsized":"2024-02-19T21:09:54.446199+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:06:25.785366+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":332,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.282592+0000","last_change":"2024-02-19T21:08:41.276256+0000","last_active":"2024-02-19T21:09:54.282592+0000","last_peered":"2024-02-19T21:09:54.282592+0000","last_clean":"2024-02-19T21:09:54.282592+0000","last_became_active":"2024-02-19T21:08:41.268425+0000","last_became_peered":"2024-02-19T21:08:41.268425+0000","last_unstale":"2024-02-19T21:09:54.282592+0000","last_undegraded":"2024-02-19T21:09:54.282592+0000","last_fullsized":"2024-02-19T21:09:54.282592+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:58.310306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:58.310306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":386,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269373+0000","last_change":"2024-02-19T21:09:44.596395+0000","last_active":"2024-02-19T21:09:54.269373+0000","last_peered":"2024-02-19T21:09:54.269373+0000","last_clean":"2024-02-19T21:09:54.269373+0000","last_became_active":"2024-02-19T21:08:41.298233+0000","last_became_peered":"2024-02-19T21:08:41.298233+0000","last_unstale":"2024-02-19T21:09:54.269373+0000","last_undegraded":"2024-02-19T21:09:54.269373+0000","last_fullsized":"2024-02-19T21:09:54.269373+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:09:44.596326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:44.596326+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":380,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.275910+0000","last_change":"2024-02-19T21:09:25.733055+0000","last_active":"2024-02-19T21:09:54.275910+0000","last_peered":"2024-02-19T21:09:54.275910+0000","last_clean":"2024-02-19T21:09:54.275910+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:09:54.275910+0000","last_undegraded":"2024-02-19T21:09:54.275910+0000","last_fullsized":"2024-02-19T21:09:54.275910+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:25.732955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:25.732955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":385,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.273602+0000","last_change":"2024-02-19T21:09:36.862378+0000","last_active":"2024-02-19T21:09:54.273602+0000","last_peered":"2024-02-19T21:09:54.273602+0000","last_clean":"2024-02-19T21:09:54.273602+0000","last_became_active":"2024-02-19T21:08:41.304581+0000","last_became_peered":"2024-02-19T21:08:41.304581+0000","last_unstale":"2024-02-19T21:09:54.273602+0000","last_undegraded":"2024-02-19T21:09:54.273602+0000","last_fullsized":"2024-02-19T21:09:54.273602+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:36.862297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:36.862297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":388,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268775+0000","last_change":"2024-02-19T21:08:04.266429+0000","last_active":"2024-02-19T21:09:54.268775+0000","last_peered":"2024-02-19T21:09:54.268775+0000","last_clean":"2024-02-19T21:09:54.268775+0000","last_became_active":"2024-02-19T21:08:04.266100+0000","last_became_peered":"2024-02-19T21:08:04.266100+0000","last_unstale":"2024-02-19T21:09:54.268775+0000","last_undegraded":"2024-02-19T21:09:54.268775+0000","last_fullsized":"2024-02-19T21:09:54.268775+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:48.506964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:48.506964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":207,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269880+0000","last_change":"2024-02-19T21:06:50.467486+0000","last_active":"2024-02-19T21:09:54.269880+0000","last_peered":"2024-02-19T21:09:54.269880+0000","last_clean":"2024-02-19T21:09:54.269880+0000","last_became_active":"2024-02-19T21:06:40.227713+0000","last_became_peered":"2024-02-19T21:06:40.227713+0000","last_unstale":"2024-02-19T21:09:54.269880+0000","last_undegraded":"2024-02-19T21:09:54.269880+0000","last_fullsized":"2024-02-19T21:09:54.269880+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:50.467411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:50.467411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":119,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446745+0000","last_change":"2024-02-19T21:08:42.700047+0000","last_active":"2024-02-19T21:09:54.446745+0000","last_peered":"2024-02-19T21:09:54.446745+0000","last_clean":"2024-02-19T21:09:54.446745+0000","last_became_active":"2024-02-19T21:08:41.266486+0000","last_became_peered":"2024-02-19T21:08:41.266486+0000","last_unstale":"2024-02-19T21:09:54.446745+0000","last_undegraded":"2024-02-19T21:09:54.446745+0000","last_fullsized":"2024-02-19T21:09:54.446745+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:08:36.593172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.593172+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":118,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269966+0000","last_change":"2024-02-19T21:08:04.278604+0000","last_active":"2024-02-19T21:09:54.269966+0000","last_peered":"2024-02-19T21:09:54.269966+0000","last_clean":"2024-02-19T21:09:54.269966+0000","last_became_active":"2024-02-19T21:08:04.277576+0000","last_became_peered":"2024-02-19T21:08:04.277576+0000","last_unstale":"2024-02-19T21:09:54.269966+0000","last_undegraded":"2024-02-19T21:09:54.269966+0000","last_fullsized":"2024-02-19T21:09:54.269966+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:53.752543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:06:53.752543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":371,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446481+0000","last_change":"2024-02-19T21:06:24.702277+0000","last_active":"2024-02-19T21:09:54.446481+0000","last_peered":"2024-02-19T21:09:54.446481+0000","last_clean":"2024-02-19T21:09:54.446481+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:09:54.446481+0000","last_undegraded":"2024-02-19T21:09:54.446481+0000","last_fullsized":"2024-02-19T21:09:54.446481+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:06:24.702197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":82,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.268745+0000","last_change":"2024-02-19T21:08:41.307487+0000","last_active":"2024-02-19T21:09:54.268745+0000","last_peered":"2024-02-19T21:09:54.268745+0000","last_clean":"2024-02-19T21:09:54.268745+0000","last_became_active":"2024-02-19T21:08:41.306708+0000","last_became_peered":"2024-02-19T21:08:41.306708+0000","last_unstale":"2024-02-19T21:09:54.268745+0000","last_undegraded":"2024-02-19T21:09:54.268745+0000","last_fullsized":"2024-02-19T21:09:54.268745+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:04.600973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:06:04.600973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":333,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270467+0000","last_change":"2024-02-19T21:08:41.293045+0000","last_active":"2024-02-19T21:09:54.270467+0000","last_peered":"2024-02-19T21:09:54.270467+0000","last_clean":"2024-02-19T21:09:54.270467+0000","last_became_active":"2024-02-19T21:08:41.292651+0000","last_became_peered":"2024-02-19T21:08:41.292651+0000","last_unstale":"2024-02-19T21:09:54.270467+0000","last_undegraded":"2024-02-19T21:09:54.270467+0000","last_fullsized":"2024-02-19T21:09:54.270467+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:58.468840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:58.468840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":134,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.270368+0000","last_change":"2024-02-19T21:08:04.341196+0000","last_active":"2024-02-19T21:09:54.270368+0000","last_peered":"2024-02-19T21:09:54.270368+0000","last_clean":"2024-02-19T21:09:54.270368+0000","last_became_active":"2024-02-19T21:08:04.249443+0000","last_became_peered":"2024-02-19T21:08:04.249443+0000","last_unstale":"2024-02-19T21:09:54.270368+0000","last_undegraded":"2024-02-19T21:09:54.270368+0000","last_fullsized":"2024-02-19T21:09:54.270368+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:06:09.763482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:06:09.763482+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":218,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.275966+0000","last_change":"2024-02-19T21:08:47.903239+0000","last_active":"2024-02-19T21:09:54.275966+0000","last_peered":"2024-02-19T21:09:54.275966+0000","last_clean":"2024-02-19T21:09:54.275966+0000","last_became_active":"2024-02-19T21:08:41.263206+0000","last_became_peered":"2024-02-19T21:08:41.263206+0000","last_unstale":"2024-02-19T21:09:54.275966+0000","last_undegraded":"2024-02-19T21:09:54.275966+0000","last_fullsized":"2024-02-19T21:09:54.275966+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:47.903127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:47.903127+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":81,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.446671+0000","last_change":"2024-02-19T21:08:41.263525+0000","last_active":"2024-02-19T21:09:54.446671+0000","last_peered":"2024-02-19T21:09:54.446671+0000","last_clean":"2024-02-19T21:09:54.446671+0000","last_became_active":"2024-02-19T21:08:41.263160+0000","last_became_peered":"2024-02-19T21:08:41.263160+0000","last_unstale":"2024-02-19T21:09:54.446671+0000","last_undegraded":"2024-02-19T21:09:54.446671+0000","last_fullsized":"2024-02-19T21:09:54.446671+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:39.574835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.574835+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":123,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.269673+0000","last_change":"2024-02-19T21:09:33.559448+0000","last_active":"2024-02-19T21:09:54.269673+0000","last_peered":"2024-02-19T21:09:54.269673+0000","last_clean":"2024-02-19T21:09:54.269673+0000","last_became_active":"2024-02-19T21:08:04.252640+0000","last_became_peered":"2024-02-19T21:08:04.252640+0000","last_unstale":"2024-02-19T21:09:54.269673+0000","last_undegraded":"2024-02-19T21:09:54.269673+0000","last_fullsized":"2024-02-19T21:09:54.269673+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:33.559363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:33.559363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":339,"reported_epoch":411,"state":"active+clean","last_fresh":"2024-02-19T21:09:54.447190+0000","last_change":"2024-02-19T21:08:04.251168+0000","last_active":"2024-02-19T21:09:54.447190+0000","last_peered":"2024-02-19T21:09:54.447190+0000","last_clean":"2024-02-19T21:09:54.447190+0000","last_became_active":"2024-02-19T21:08:04.250937+0000","last_became_peered":"2024-02-19T21:08:04.250937+0000","last_unstale":"2024-02-19T21:09:54.447190+0000","last_undegraded":"2024-02-19T21:09:54.447190+0000","last_fullsized":"2024-02-19T21:09:54.447190+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:06:27.744835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:06:27.744835+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":212,"num_pg":32,"stat_sum":{"num_bytes":14,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":10,"num_read_kb":7,"num_write":10,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":42,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":7},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":4001489,"num_objects":160,"num_object_clones":4,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":107,"num_read_kb":9,"num_write":596,"num_write_kb":4495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":491,"num_bytes_recovered":1160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13959168,"data_stored":12004581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1871,"ondisk_log_size":1871,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":677,"num_read_kb":434,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":4500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":410,"ondisk_log_size":410,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":516,"num_read_kb":511,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":4419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266795,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311820,"kb_used_data":10496,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93433332,"statfs":{"total":95995035648,"available":95675731968,"internally_reserved":0,"allocated":10747904,"data_stored":7819984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":1.603,"15min":0.86899999999999999},"min":{"1min":0.26000000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.65200000000000002,"5min":89.456999999999994,"15min":89.456999999999994},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":1.377,"15min":0.78600000000000003},"min":{"1min":0.28499999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.67900000000000005,"5min":70.451999999999998,"15min":70.451999999999998},"last":0.49099999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.0740000000000001,"15min":0.70199999999999996},"min":{"1min":0.36299999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.85099999999999998,"5min":34.273000000000003,"15min":34.273000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.46899999999999997,"5min":1.7889999999999999,"15min":0.94099999999999995},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.254},"max":{"1min":0.79200000000000004,"5min":89.594999999999999,"15min":89.594999999999999},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":1.837,"15min":0.94399999999999995},"min":{"1min":0.35099999999999998,"5min":0.27600000000000002,"15min":0.24299999999999999},"max":{"1min":0.67100000000000004,"5min":104.413,"15min":104.413},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":1.8009999999999999,"15min":0.94999999999999996},"min":{"1min":0.35599999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70599999999999996,"5min":104.268,"15min":104.268},"last":0.44800000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":1.7969999999999999,"15min":0.95199999999999996},"min":{"1min":0.38800000000000001,"5min":0.29099999999999998,"15min":0.254},"max":{"1min":1.2949999999999999,"5min":105.3,"15min":105.3},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.79700000000000004,"15min":0.59799999999999998},"min":{"1min":0.34300000000000003,"5min":0.315,"15min":0.315},"max":{"1min":0.879,"5min":13.243,"15min":13.243},"last":0.53800000000000003}]},{"osd":4,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.42799999999999999,"15min":0.47599999999999998},"min":{"1min":0.315,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63200000000000001,"5min":0.69299999999999995,"15min":1.373},"last":0.623},{"interface":"front","average":{"1min":0.505,"5min":0.45700000000000002,"15min":0.47999999999999998},"min":{"1min":0.32700000000000001,"5min":0.26600000000000001,"15min":0.25900000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":1.0740000000000001},"last":0.60899999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.44700000000000001,"15min":0.46000000000000002},"min":{"1min":0.28599999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.65900000000000003,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.48999999999999999,"15min":0.50900000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69399999999999995,"5min":0.77400000000000002,"15min":0.85899999999999999},"last":0.52500000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:09:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.49199999999999999,"15min":0.52400000000000002},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.70199999999999996,"5min":0.76500000000000001,"15min":1.0049999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52300000000000002,"15min":0.56100000000000005},"min":{"1min":0.34100000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72499999999999998,"5min":0.75900000000000001,"15min":1.262},"last":0.56699999999999995}]}]},{"osd":6,"up_from":45,"seq":193273528433,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307932,"kb_used_data":6608,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437220,"statfs":{"total":95995035648,"available":95679713280,"internally_reserved":0,"allocated":6766592,"data_stored":3827061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.86299999999999999,"15min":0.61699999999999999},"min":{"1min":0.22700000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.73099999999999998,"5min":17.486999999999998,"15min":17.486999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.98299999999999998,"15min":0.65400000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64900000000000002,"5min":17.43,"15min":17.43},"last":0.30299999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.873,"15min":0.61799999999999999},"min":{"1min":0.28699999999999998,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.71599999999999997,"5min":14.692,"15min":14.692},"last":0.312},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.99299999999999999,"15min":0.66400000000000003},"min":{"1min":0.29099999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.74099999999999999,"5min":14.795,"15min":14.795},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.79100000000000004,"15min":0.60799999999999998},"min":{"1min":0.309,"5min":0.255,"15min":0.255},"max":{"1min":0.61899999999999999,"5min":6.931,"15min":6.931},"last":0.434},{"interface":"front","average":{"1min":0.47699999999999998,"5min":1.0249999999999999,"15min":0.66900000000000004},"min":{"1min":0.34100000000000003,"5min":0.318,"15min":0.318},"max":{"1min":0.71899999999999997,"5min":14.391,"15min":14.391},"last":0.32900000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":1.077,"15min":0.69999999999999996},"min":{"1min":0.378,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.73599999999999999,"5min":14.510999999999999,"15min":14.510999999999999},"last":0.371},{"interface":"front","average":{"1min":0.499,"5min":0.95299999999999996,"15min":0.65300000000000002},"min":{"1min":0.35299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.64500000000000002,"5min":17.585000000000001,"15min":17.585000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40100000000000002,"15min":0.39200000000000002},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.63800000000000001,"15min":1.363},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42699999999999999,"15min":0.41599999999999998},"min":{"1min":0.29799999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.63500000000000001,"5min":0.69899999999999995,"15min":0.70999999999999996},"last":0.33700000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46100000000000002,"15min":0.438},"min":{"1min":0.30299999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.59699999999999998,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.504,"15min":0.47299999999999998},"min":{"1min":0.35099999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.627,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.42599999999999999}]},{"osd":7,"last update":"Mon Feb 19 21:09:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.47699999999999998,"15min":0.503},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":1.1160000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.46999999999999997,"15min":0.51100000000000001},"min":{"1min":0.32700000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":1.534},"last":0.40699999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311896,"kb_used_data":10572,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93433256,"statfs":{"total":95995035648,"available":95675654144,"internally_reserved":0,"allocated":10825728,"data_stored":7828690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:07 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.92900000000000005,"15min":0.54600000000000004},"min":{"1min":0.16700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.51400000000000001,"5min":27.777000000000001,"15min":27.777000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.98799999999999999,"15min":0.57399999999999995},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.217},"max":{"1min":0.70899999999999996,"5min":27.631,"15min":27.631},"last":0.45400000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":1.3109999999999999,"15min":0.71099999999999997},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.66300000000000003,"5min":27.798999999999999,"15min":27.798999999999999},"last":0.433},{"interface":"front","average":{"1min":0.372,"5min":1.3420000000000001,"15min":0.72599999999999998},"min":{"1min":0.24299999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.73299999999999998,"5min":27.606999999999999,"15min":27.606999999999999},"last":0.42799999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:09:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.88800000000000001,"15min":0.57599999999999996},"min":{"1min":0.23699999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.65000000000000002,"5min":15.574999999999999,"15min":15.574999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":1.0860000000000001,"15min":0.67200000000000004},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.74399999999999999,"5min":27.82,"15min":27.82},"last":0.46600000000000003}]},{"osd":4,"last update":"Mon Feb 19 21:09:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.86099999999999999,"15min":0.66200000000000003},"min":{"1min":0.36099999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.80400000000000005,"5min":9.6430000000000007,"15min":9.6430000000000007},"last":0.627},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.127,"15min":0.755},"min":{"1min":0.40200000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.93000000000000005,"5min":27.707000000000001,"15min":27.707000000000001},"last":0.52400000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":1.075,"15min":0.75},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":0.94199999999999995,"5min":27.529,"15min":27.529},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.86799999999999999,"15min":0.69599999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.84499999999999997,"5min":9.6799999999999997,"15min":9.6799999999999997},"last":0.59099999999999997}]},{"osd":6,"last update":"Mon Feb 19 21:09:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.83799999999999997,"15min":0.67300000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.82799999999999996,"5min":9.6989999999999998,"15min":9.6989999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":1.103,"15min":0.79600000000000004},"min":{"1min":0.41899999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.0470000000000002,"5min":27.733000000000001,"15min":27.733000000000001},"last":0.55500000000000005}]},{"osd":7,"last update":"Mon Feb 19 21:09:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.139,"15min":0.77600000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.92000000000000004,"5min":27.576000000000001,"15min":27.576000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":1.2170000000000001,"15min":0.81200000000000006},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":2.0169999999999999,"5min":27.756,"15min":27.756},"last":0.57599999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738510,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307900,"kb_used_data":6568,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93437252,"statfs":{"total":95995035648,"available":95679746048,"internally_reserved":0,"allocated":6725632,"data_stored":3819615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":1.119,"15min":0.59899999999999998},"min":{"1min":0.20899999999999999,"5min":0.188,"15min":0.16200000000000001},"max":{"1min":0.53800000000000003,"5min":20.966999999999999,"15min":20.966999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":1.179,"15min":0.60699999999999998},"min":{"1min":0.216,"5min":0.20399999999999999,"15min":0.188},"max":{"1min":0.68200000000000005,"5min":20.366,"15min":20.366},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:09:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":1.369,"15min":0.71599999999999997},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51000000000000001,"5min":27.916,"15min":27.916},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.39300000000000002,"5min":1.081,"15min":0.63200000000000001},"min":{"1min":0.23999999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.622,"5min":22.004000000000001,"15min":22.004000000000001},"last":0.38}]},{"osd":3,"last update":"Mon Feb 19 21:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":1.347,"15min":0.754},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73199999999999998,"5min":25.669,"15min":25.669},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":1.161,"15min":0.69899999999999995},"min":{"1min":0.28899999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.59999999999999998,"5min":25.006,"15min":25.006},"last":0.54500000000000004}]},{"osd":4,"last update":"Mon Feb 19 21:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":1.0960000000000001,"15min":0.71499999999999997},"min":{"1min":0.34399999999999997,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.86499999999999999,"5min":20.411999999999999,"15min":20.411999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":1.153,"15min":0.748},"min":{"1min":0.376,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.1160000000000001,"5min":19.986999999999998,"15min":19.986999999999998},"last":0.53100000000000003}]},{"osd":5,"last update":"Mon Feb 19 21:09:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":1.2430000000000001,"15min":0.83299999999999996},"min":{"1min":0.40100000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.1890000000000001,"5min":20.021999999999998,"15min":20.021999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.86199999999999999,"5min":1.173,"15min":0.91000000000000003},"min":{"1min":0.317,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":6.2859999999999996,"5min":20.908999999999999,"15min":20.908999999999999},"last":0.66700000000000004}]},{"osd":6,"last update":"Mon Feb 19 21:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":1.0960000000000001,"15min":0.76600000000000001},"min":{"1min":0.39200000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.8320000000000001,"5min":20.445,"15min":20.445},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":1.1830000000000001,"15min":0.81999999999999995},"min":{"1min":0.41399999999999998,"5min":0.374,"15min":0.33400000000000002},"max":{"1min":1.748,"5min":20.047999999999998,"15min":20.047999999999998},"last":0.58499999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:09:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.135,"15min":0.89300000000000002},"min":{"1min":0.41099999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.95499999999999996,"5min":20.940999999999999,"15min":20.940999999999999},"last":0.629},{"interface":"front","average":{"1min":0.60299999999999998,"5min":1.2170000000000001,"15min":0.82499999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.3420000000000001,"5min":20.471,"15min":20.471},"last":0.56200000000000006}]}]},{"osd":2,"up_from":18,"seq":77309411459,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307728,"kb_used_data":10492,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93437424,"statfs":{"total":95995035648,"available":95679922176,"internally_reserved":0,"allocated":10743808,"data_stored":7819325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:27 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.85499999999999998,"15min":0.53900000000000003},"min":{"1min":0.222,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.54100000000000004,"5min":26.690999999999999,"15min":26.690999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.32200000000000001,"5min":1.0069999999999999,"15min":0.59099999999999997},"min":{"1min":0.247,"5min":0.23200000000000001,"15min":0.214},"max":{"1min":0.48999999999999999,"5min":24.315000000000001,"15min":24.315000000000001},"last":0.372}]},{"osd":1,"last update":"Mon Feb 19 21:09:27 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":1.3480000000000001,"15min":0.70699999999999996},"min":{"1min":0.252,"5min":0.22,"15min":0.20100000000000001},"max":{"1min":0.52300000000000002,"5min":28.882000000000001,"15min":28.882000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":1.2,"15min":0.68400000000000005},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":0.50600000000000001,"5min":28.382999999999999,"15min":28.382999999999999},"last":0.31}]},{"osd":3,"last update":"Mon Feb 19 21:09:54 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":1.276,"15min":0.72599999999999998},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56200000000000006,"5min":28.742000000000001,"15min":28.742000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.96899999999999997,"15min":0.64700000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.25700000000000001},"max":{"1min":0.57299999999999995,"5min":26.719999999999999,"15min":26.719999999999999},"last":0.38600000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:09:16 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.89300000000000002,"15min":0.70999999999999996},"min":{"1min":0.34799999999999998,"5min":0.309,"15min":0.30599999999999999},"max":{"1min":3.4039999999999999,"5min":24.356000000000002,"15min":24.356000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.96999999999999997,"15min":0.74299999999999999},"min":{"1min":0.39000000000000001,"5min":0.32800000000000001,"15min":0.32400000000000001},"max":{"1min":3.3919999999999999,"5min":20.707000000000001,"15min":20.707000000000001},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:09:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.83999999999999997,"15min":0.70399999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.748,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":1.081,"15min":0.80700000000000005},"min":{"1min":0.41299999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85099999999999998,"5min":26.606000000000002,"15min":26.606000000000002},"last":0.45700000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":1.1870000000000001,"15min":0.81499999999999995},"min":{"1min":0.46200000000000002,"5min":0.376,"15min":0.376},"max":{"1min":1.4510000000000001,"5min":24.381,"15min":24.381},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.996,"15min":0.75},"min":{"1min":0.42199999999999999,"5min":0.42099999999999999,"15min":0.39100000000000001},"max":{"1min":0.93600000000000005,"5min":28.706,"15min":28.706},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Feb 19 21:09:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.099,"15min":0.77800000000000002},"min":{"1min":0.46000000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76200000000000001,"5min":26.658000000000001,"15min":26.658000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.93400000000000005,"15min":0.73699999999999999},"min":{"1min":0.439,"5min":0.39100000000000001,"15min":0.379},"max":{"1min":0.76700000000000002,"5min":24.402000000000001,"15min":24.402000000000001},"last":0.53900000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182527,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320156,"kb_used_data":6544,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93424996,"statfs":{"total":95995035648,"available":95667195904,"internally_reserved":0,"allocated":6701056,"data_stored":3827775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.74199999999999999,"15min":0.71499999999999997},"min":{"1min":0.22800000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.439,"5min":28.422000000000001,"15min":77.662000000000006},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.57899999999999996,"15min":0.505},"min":{"1min":0.25900000000000001,"5min":0.22600000000000001,"15min":0.19700000000000001},"max":{"1min":0.52200000000000002,"5min":8.5790000000000006,"15min":17.456},"last":0.36499999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.68300000000000005,"15min":0.53400000000000003},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":0.5,"5min":9.2780000000000005,"15min":17.327999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.745,"15min":0.51400000000000001},"min":{"1min":0.26500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.50900000000000001,"5min":15.004,"15min":15.004},"last":0.52400000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:09:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.67000000000000004,"15min":0.67100000000000004},"min":{"1min":0.27400000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.49199999999999999,"5min":12.925000000000001,"15min":56.890999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.74299999999999999,"15min":0.65500000000000003},"min":{"1min":0.27400000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.58299999999999996,"5min":17.177,"15min":44.600999999999999},"last":0.32100000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:09:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.82099999999999995,"15min":0.63800000000000001},"min":{"1min":0.33100000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.74399999999999999,"5min":12.847,"15min":12.847},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.026,"15min":0.71599999999999997},"min":{"1min":0.34000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.83899999999999997,"5min":26.393999999999998,"15min":26.393999999999998},"last":0.47799999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:09:34 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.92600000000000005,"15min":0.69299999999999995},"min":{"1min":0.34699999999999998,"5min":0.313,"15min":0.313},"max":{"1min":9.2360000000000007,"5min":17.401,"15min":17.401},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.85599999999999998,"15min":0.66100000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.9690000000000001,"5min":12.882,"15min":12.882},"last":0.47299999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.84999999999999998,"15min":0.65900000000000003},"min":{"1min":0.39000000000000001,"5min":0.35299999999999998,"15min":0.34300000000000003},"max":{"1min":1.9790000000000001,"5min":17.367000000000001,"15min":17.367000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.84599999999999997,"15min":0.67500000000000004},"min":{"1min":0.35699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.88,"5min":12.904,"15min":12.904},"last":0.5}]},{"osd":7,"last update":"Mon Feb 19 21:09:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.83799999999999997,"15min":0.65600000000000003},"min":{"1min":0.38600000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":17.428999999999998,"15min":17.428999999999998},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.626,"5min":1.2509999999999999,"15min":0.80100000000000005},"min":{"1min":0.41899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85899999999999999,"5min":44.378,"15min":44.378},"last":0.53000000000000003}]}]},{"osd":4,"up_from":0,"seq":133143986299,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":37,"seq":158913790070,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312072,"kb_used_data":6652,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93433080,"statfs":{"total":95995035648,"available":95675473920,"internally_reserved":0,"allocated":6811648,"data_stored":3818228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":4.6790000000000003,"15min":1.8919999999999999},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.69999999999999996,"5min":276.47899999999998,"15min":276.47899999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":1.6759999999999999,"15min":0.91700000000000004},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.95299999999999996,"5min":52.887,"15min":52.887},"last":0.374}]},{"osd":1,"last update":"Mon Feb 19 21:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":2.8250000000000002,"15min":1.282},"min":{"1min":0.254,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.599,"5min":120.011,"15min":120.011},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":2.024,"15min":1.0309999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.97699999999999998,"5min":118.105,"15min":118.105},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":1.9690000000000001,"15min":1.006},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66600000000000004,"5min":82.242000000000004,"15min":82.242000000000004},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":3.4910000000000001,"15min":1.514},"min":{"1min":0.309,"5min":0.309,"15min":0.30199999999999999},"max":{"1min":4.5019999999999998,"5min":232.84999999999999,"15min":232.84999999999999},"last":0.40899999999999997}]},{"osd":3,"last update":"Mon Feb 19 21:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":2.1680000000000001,"15min":1.075},"min":{"1min":0.30199999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.4640000000000004,"5min":109.70099999999999,"15min":109.70099999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.6599999999999999,"15min":0.91600000000000004},"min":{"1min":0.34799999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72199999999999998,"5min":65.519000000000005,"15min":65.519000000000005},"last":0.32000000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.443,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.0129999999999999,"5min":1.861,"15min":1.861},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45300000000000001,"15min":0.42199999999999999},"min":{"1min":0.308,"5min":0.248,"15min":0.248},"max":{"1min":0.72499999999999998,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":0.51400000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:09:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.0289999999999999,"5min":1.823,"15min":1.823},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.46899999999999997,"15min":0.45600000000000002},"min":{"1min":0.37,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.64200000000000002,"5min":0.71399999999999997,"15min":0.73399999999999999},"last":0.45400000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:09:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.36199999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.052,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.501,"15min":0.50800000000000001},"min":{"1min":0.38400000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.871,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.41899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4321280,"data_stored":4000784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4263936,"data_stored":4000607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":96,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4268032,"data_stored":4000619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":2,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":6,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":4,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":212,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:10:05.400 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool set .rgw.root pgp_num 32 2024-02-19T21:10:05.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:05.502 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:10:05.527 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:03.995653+0000 osd.7 (osd.7) 181 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:10:05.527 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:03.998555+0000 osd.7 (osd.7) 182 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:10:05.527 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:04.015155+0000 osd.2 (osd.2) 154 : cluster [DBG] 5.d scrub starts 2024-02-19T21:10:05.527 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:04.016615+0000 osd.2 (osd.2) 155 : cluster [DBG] 5.d scrub ok 2024-02-19T21:10:05.527 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:03.995653+0000 osd.7 (osd.7) 181 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:10:05.527 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:03.998555+0000 osd.7 (osd.7) 182 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:10:05.528 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:04.015155+0000 osd.2 (osd.2) 154 : cluster [DBG] 5.d scrub starts 2024-02-19T21:10:05.528 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:04.016615+0000 osd.2 (osd.2) 155 : cluster [DBG] 5.d scrub ok 2024-02-19T21:10:05.529 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:05.548 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:10:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:05.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:10:05.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:03.995653+0000 osd.7 (osd.7) 181 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:10:05.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:03.998555+0000 osd.7 (osd.7) 182 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:10:05.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:04.015155+0000 osd.2 (osd.2) 154 : cluster [DBG] 5.d scrub starts 2024-02-19T21:10:05.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:04.016615+0000 osd.2 (osd.2) 155 : cluster [DBG] 5.d scrub ok 2024-02-19T21:10:05.652 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:05.680 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:05.680 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:05.704 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:05.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:05.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:05.808 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:05.838 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:06.020 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:06.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:06.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:06.031 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-19T21:10:06.125 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:06.152 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:06.162 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:10:06.205 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:06.208 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:06.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:06.309 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:06.338 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:06.357 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:06.360 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:06.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:06.461 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:06.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:04.545219+0000 osd.3 (osd.3) 177 : cluster [DBG] 2.1b scrub starts 2024-02-19T21:10:06.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:04.545984+0000 osd.3 (osd.3) 178 : cluster [DBG] 2.1b scrub ok 2024-02-19T21:10:06.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:05.179527+0000 osd.5 (osd.5) 164 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:10:06.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:05.182667+0000 osd.5 (osd.5) 165 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:10:06.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:05.210295+0000 mgr.y (mgr.14550) 374 : cluster [DBG] pgmap v598: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-19T21:10:06.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:05.273625+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e415: 8 total, 8 up, 7 in 2024-02-19T21:10:06.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:04.545219+0000 osd.3 (osd.3) 177 : cluster [DBG] 2.1b scrub starts 2024-02-19T21:10:06.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:04.545984+0000 osd.3 (osd.3) 178 : cluster [DBG] 2.1b scrub ok 2024-02-19T21:10:06.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:05.179527+0000 osd.5 (osd.5) 164 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:10:06.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:05.182667+0000 osd.5 (osd.5) 165 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:10:06.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:05.210295+0000 mgr.y (mgr.14550) 374 : cluster [DBG] pgmap v598: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-19T21:10:06.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:05.273625+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e415: 8 total, 8 up, 7 in 2024-02-19T21:10:06.489 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:06.526 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:06.528 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:06.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:06.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:04.545219+0000 osd.3 (osd.3) 177 : cluster [DBG] 2.1b scrub starts 2024-02-19T21:10:06.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:04.545984+0000 osd.3 (osd.3) 178 : cluster [DBG] 2.1b scrub ok 2024-02-19T21:10:06.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:05.179527+0000 osd.5 (osd.5) 164 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:10:06.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:05.182667+0000 osd.5 (osd.5) 165 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:10:06.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:05.210295+0000 mgr.y (mgr.14550) 374 : cluster [DBG] pgmap v598: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-19T21:10:06.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:05.273625+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e415: 8 total, 8 up, 7 in 2024-02-19T21:10:06.630 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:06.657 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:06.676 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:06.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:06.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:06.780 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:06.806 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:06.825 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:06.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:06.929 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:06.956 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:06.974 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:06.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:06.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:07.031 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:10:07.077 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:07.105 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:07.178 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:07.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:07.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:07.282 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:07.310 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:07.329 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:07.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:07.433 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:07.460 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:07.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.469+0000 7f19d5bd3700 1 -- 172.21.15.163:0/2147213636 >> v1:172.21.15.163:6789/0 conn(0x7f19d00fd9f0 legacy=0x7f19d00fddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:07.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.469+0000 7f19d5bd3700 1 -- 172.21.15.163:0/2147213636 shutdown_connections 2024-02-19T21:10:07.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.469+0000 7f19d5bd3700 1 -- 172.21.15.163:0/2147213636 >> 172.21.15.163:0/2147213636 conn(0x7f19d006fdc0 msgr2=0x7f19d00f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:07.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.469+0000 7f19d5bd3700 1 -- 172.21.15.163:0/2147213636 shutdown_connections 2024-02-19T21:10:07.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.469+0000 7f19d5bd3700 1 -- 172.21.15.163:0/2147213636 wait complete. 2024-02-19T21:10:07.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.470+0000 7f19d5bd3700 1 librados: starting msgr at 2024-02-19T21:10:07.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.470+0000 7f19d5bd3700 1 librados: starting objecter 2024-02-19T21:10:07.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.470+0000 7f19d5bd3700 1 Processor -- start 2024-02-19T21:10:07.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.470+0000 7f19d5bd3700 1 -- start start 2024-02-19T21:10:07.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.470+0000 7f19d5bd3700 1 librados: setting wanted keys 2024-02-19T21:10:07.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.470+0000 7f19d5bd3700 1 librados: calling monclient init 2024-02-19T21:10:07.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19d5bd3700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19d00fce00 con 0x7f19d01042f0 2024-02-19T21:10:07.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19d5bd3700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19d00fcfd0 con 0x7f19d00fd9f0 2024-02-19T21:10:07.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19d5bd3700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19d00fd1a0 con 0x7f19d0100720 2024-02-19T21:10:07.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ceffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f19d0100720 0x7f19d0114d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:50214/0 (socket says 172.21.15.163:50214) 2024-02-19T21:10:07.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ceffd700 1 -- 172.21.15.163:0/3688768891 learned_addr learned my addr 172.21.15.163:0/3688768891 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:07.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 10 client.?.objecter ms_handle_connect 0x7f19d00fd9f0 2024-02-19T21:10:07.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:07.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 10 client.?.objecter ms_handle_connect 0x7f19d0100720 2024-02-19T21:10:07.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:07.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 10 client.?.objecter ms_handle_connect 0x7f19d01042f0 2024-02-19T21:10:07.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:07.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712490705 0 0) 0x7f19d00fcfd0 con 0x7f19d00fd9f0 2024-02-19T21:10:07.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b80038e0 con 0x7f19d00fd9f0 2024-02-19T21:10:07.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855437545 0 0) 0x7f19d00fd1a0 con 0x7f19d0100720 2024-02-19T21:10:07.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d00fcfd0 con 0x7f19d0100720 2024-02-19T21:10:07.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444028670 0 0) 0x7f19d00fce00 con 0x7f19d01042f0 2024-02-19T21:10:07.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d00fd1a0 con 0x7f19d01042f0 2024-02-19T21:10:07.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2842649911 0 0) 0x7f19b80038e0 con 0x7f19d00fd9f0 2024-02-19T21:10:07.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19d00fce00 con 0x7f19d00fd9f0 2024-02-19T21:10:07.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f19c0001510 con 0x7f19d00fd9f0 2024-02-19T21:10:07.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2092096500 0 0) 0x7f19d00fcfd0 con 0x7f19d0100720 2024-02-19T21:10:07.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19b80038e0 con 0x7f19d0100720 2024-02-19T21:10:07.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.471+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f19c40019f0 con 0x7f19d0100720 2024-02-19T21:10:07.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.472+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 715230207 0 0) 0x7f19d00fce00 con 0x7f19d00fd9f0 2024-02-19T21:10:07.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.472+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 >> v1:172.21.15.163:6790/0 conn(0x7f19d0100720 legacy=0x7f19d0114d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:07.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.472+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 >> v1:172.21.15.163:6789/0 conn(0x7f19d01042f0 legacy=0x7f19d0118470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:07.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.472+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d011cbc0 con 0x7f19d00fd9f0 2024-02-19T21:10:07.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.472+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f19c0001ae0 con 0x7f19d00fd9f0 2024-02-19T21:10:07.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.472+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f19c00037d0 con 0x7f19d00fd9f0 2024-02-19T21:10:07.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.473+0000 7f19d5bd3700 1 -- 172.21.15.163:0/3688768891 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d0119b90 con 0x7f19d00fd9f0 2024-02-19T21:10:07.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.473+0000 7f19d5bd3700 10 client.48646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:07.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.473+0000 7f19d5bd3700 1 -- 172.21.15.163:0/3688768891 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d011a000 con 0x7f19d00fd9f0 2024-02-19T21:10:07.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.473+0000 7f19d5bd3700 1 librados: init done 2024-02-19T21:10:07.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.474+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f19c0002b60 con 0x7f19d00fd9f0 2024-02-19T21:10:07.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.475+0000 7f19d5bd3700 1 -- 172.21.15.163:0/3688768891 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d005f700 con 0x7f19d00fd9f0 2024-02-19T21:10:07.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.475+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(415..415 src has 43..415) v4 ==== 7974+0+0 (unknown 2343044777 0 0) 0x7f19c0076840 con 0x7f19d00fd9f0 2024-02-19T21:10:07.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.475+0000 7f19ccff9700 10 client.48646.objecter ms_dispatch 0x7f19d0001550 osd_map(415..415 src has 43..415) v4 2024-02-19T21:10:07.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.475+0000 7f19ccff9700 3 client.48646.objecter handle_osd_map got epochs [415,415] > 0 2024-02-19T21:10:07.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.475+0000 7f19ccff9700 3 client.48646.objecter handle_osd_map decoding full epoch 415 2024-02-19T21:10:07.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.475+0000 7f19ccff9700 20 client.48646.objecter dump_active .. 0 homeless 2024-02-19T21:10:07.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.475+0000 7f19ccff9700 10 client.48646.objecter ms_handle_connect 0x7f19b8063d70 2024-02-19T21:10:07.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.479+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f19c0049c90 con 0x7f19d00fd9f0 2024-02-19T21:10:07.483 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:07.486 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:07.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:07.520 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:07.568 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:07.571 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:07.588 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:10:07.615 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:07.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:07.680+0000 7f19d5bd3700 1 -- 172.21.15.163:0/3688768891 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"} v 0) v1 -- 0x7f19d0047540 con 0x7f19d00fd9f0 2024-02-19T21:10:07.813 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:10:07.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:07.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:10:07.917 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:07.944 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:08.068 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:10:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:10:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:10:08.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:08.290+0000 7f19ccff9700 1 -- 172.21.15.163:0/3688768891 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]=0 set pool 2 pgp_num to 32 v416) v1 ==== 136+0+0 (unknown 1726246433 0 0) 0x7f19c0048100 con 0x7f19d00fd9f0 2024-02-19T21:10:08.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:08.293+0000 7f19d5bd3700 10 librados: watch_flush enter 2024-02-19T21:10:08.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:08.298+0000 7f19d5bd3700 10 librados: watch_flush exit 2024-02-19T21:10:08.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:08.298+0000 7f19d5bd3700 20 client.48646.objecter shutdown clearing up homeless session... 2024-02-19T21:10:08.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:08.298+0000 7f19d5bd3700 10 client.48646.objecter successfully canceled tick 2024-02-19T21:10:08.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:08.298+0000 7f19d5bd3700 1 -- 172.21.15.163:0/3688768891 >> v1:172.21.15.163:6800/1552514681 conn(0x7f19b8063d70 legacy=0x7f19b8066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:08.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:08.298+0000 7f19d5bd3700 1 -- 172.21.15.163:0/3688768891 >> v1:172.21.15.193:6789/0 conn(0x7f19d00fd9f0 legacy=0x7f19d00fc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:08.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:08.298+0000 7f19d5bd3700 1 -- 172.21.15.163:0/3688768891 shutdown_connections 2024-02-19T21:10:08.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:08.298+0000 7f19d5bd3700 1 -- 172.21.15.163:0/3688768891 >> 172.21.15.163:0/3688768891 conn(0x7f19d006fdc0 msgr2=0x7f19d00f4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:08.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:08.298+0000 7f19d5bd3700 1 -- 172.21.15.163:0/3688768891 shutdown_connections 2024-02-19T21:10:08.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:08.298+0000 7f19d5bd3700 1 -- 172.21.15.163:0/3688768891 wait complete. 2024-02-19T21:10:08.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:08.298+0000 7f19d5bd3700 1 librados: shutdown 2024-02-19T21:10:08.297 INFO:teuthology.orchestra.run.smithi163.stderr:set pool 2 pgp_num to 32 2024-02-19T21:10:08.326 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:06.546441+0000 osd.3 (osd.3) 179 : cluster [DBG] 6.c scrub starts 2024-02-19T21:10:08.326 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:06.547491+0000 osd.3 (osd.3) 180 : cluster [DBG] 6.c scrub ok 2024-02-19T21:10:08.327 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:06.600270+0000 osd.1 (osd.1) 189 : cluster [DBG] 16.15 scrub starts 2024-02-19T21:10:08.327 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:06.601221+0000 osd.1 (osd.1) 190 : cluster [DBG] 16.15 scrub ok 2024-02-19T21:10:08.327 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:07.040044+0000 osd.2 (osd.2) 156 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:10:08.327 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:07.040889+0000 osd.2 (osd.2) 157 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:10:08.327 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:07.210769+0000 mgr.y (mgr.14550) 375 : cluster [DBG] pgmap v600: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 888 B/s rd, 0 op/s 2024-02-19T21:10:08.327 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:07.682442+0000 mon.b (mon.1) 577 : audit [INF] from='client.? v1:172.21.15.163:0/3688768891' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-19T21:10:08.327 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:07.682740+0000 mon.a (mon.0) 2517 : audit [INF] from='client.48646 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-19T21:10:08.327 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:06.546441+0000 osd.3 (osd.3) 179 : cluster [DBG] 6.c scrub starts 2024-02-19T21:10:08.327 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:06.547491+0000 osd.3 (osd.3) 180 : cluster [DBG] 6.c scrub ok 2024-02-19T21:10:08.327 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:06.600270+0000 osd.1 (osd.1) 189 : cluster [DBG] 16.15 scrub starts 2024-02-19T21:10:08.328 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:06.601221+0000 osd.1 (osd.1) 190 : cluster [DBG] 16.15 scrub ok 2024-02-19T21:10:08.328 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:07.040044+0000 osd.2 (osd.2) 156 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:10:08.328 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:07.040889+0000 osd.2 (osd.2) 157 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:10:08.328 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:07.210769+0000 mgr.y (mgr.14550) 375 : cluster [DBG] pgmap v600: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 888 B/s rd, 0 op/s 2024-02-19T21:10:08.328 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:07.682442+0000 mon.b (mon.1) 577 : audit [INF] from='client.? v1:172.21.15.163:0/3688768891' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-19T21:10:08.328 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:07.682740+0000 mon.a (mon.0) 2517 : audit [INF] from='client.48646 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-19T21:10:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:06.546441+0000 osd.3 (osd.3) 179 : cluster [DBG] 6.c scrub starts 2024-02-19T21:10:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:06.547491+0000 osd.3 (osd.3) 180 : cluster [DBG] 6.c scrub ok 2024-02-19T21:10:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:06.600270+0000 osd.1 (osd.1) 189 : cluster [DBG] 16.15 scrub starts 2024-02-19T21:10:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:06.601221+0000 osd.1 (osd.1) 190 : cluster [DBG] 16.15 scrub ok 2024-02-19T21:10:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:07.040044+0000 osd.2 (osd.2) 156 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:10:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:07.040889+0000 osd.2 (osd.2) 157 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:10:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:07.210769+0000 mgr.y (mgr.14550) 375 : cluster [DBG] pgmap v600: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 888 B/s rd, 0 op/s 2024-02-19T21:10:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:07.682442+0000 mon.b (mon.1) 577 : audit [INF] from='client.? v1:172.21.15.163:0/3688768891' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-19T21:10:08.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:07.682740+0000 mon.a (mon.0) 2517 : audit [INF] from='client.48646 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-19T21:10:08.591 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:08.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:08.696 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:10:08.728 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:08.750 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:10:08.753 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:08.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:10:08.855 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:10:08.886 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:09.637 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:07.882006+0000 osd.7 (osd.7) 183 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:10:09.637 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:07.884900+0000 osd.7 (osd.7) 184 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:10:09.637 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:08.289676+0000 mon.a (mon.0) 2518 : audit [INF] from='client.48646 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]': finished 2024-02-19T21:10:09.637 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:08.289782+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e416: 8 total, 8 up, 7 in 2024-02-19T21:10:09.637 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:07.882006+0000 osd.7 (osd.7) 183 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:10:09.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:07.884900+0000 osd.7 (osd.7) 184 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:10:09.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:08.289676+0000 mon.a (mon.0) 2518 : audit [INF] from='client.48646 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]': finished 2024-02-19T21:10:09.638 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:08.289782+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e416: 8 total, 8 up, 7 in 2024-02-19T21:10:09.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:07.882006+0000 osd.7 (osd.7) 183 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:10:09.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:07.884900+0000 osd.7 (osd.7) 184 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:10:09.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:08.289676+0000 mon.a (mon.0) 2518 : audit [INF] from='client.48646 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "32"}]': finished 2024-02-19T21:10:09.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:08.289782+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e416: 8 total, 8 up, 7 in 2024-02-19T21:10:10.563 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:10:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:10.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:10:10.667 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:10.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:08.603381+0000 osd.3 (osd.3) 181 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:10:10.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:08.605361+0000 osd.3 (osd.3) 182 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:10:10.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:08.874038+0000 osd.7 (osd.7) 185 : cluster [DBG] 2.b scrub starts 2024-02-19T21:10:10.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:08.878158+0000 osd.7 (osd.7) 186 : cluster [DBG] 2.b scrub ok 2024-02-19T21:10:10.695 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:09.211318+0000 mgr.y (mgr.14550) 376 : cluster [DBG] pgmap v602: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:10:10.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:08.603381+0000 osd.3 (osd.3) 181 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:10:10.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:08.605361+0000 osd.3 (osd.3) 182 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:10:10.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:08.874038+0000 osd.7 (osd.7) 185 : cluster [DBG] 2.b scrub starts 2024-02-19T21:10:10.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:08.878158+0000 osd.7 (osd.7) 186 : cluster [DBG] 2.b scrub ok 2024-02-19T21:10:10.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:09.211318+0000 mgr.y (mgr.14550) 376 : cluster [DBG] pgmap v602: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:10:10.701 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:10.774 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:10.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:10.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:10.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:10:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:10:10.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:08.603381+0000 osd.3 (osd.3) 181 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:10:10.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:08.605361+0000 osd.3 (osd.3) 182 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:10:10.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:08.874038+0000 osd.7 (osd.7) 185 : cluster [DBG] 2.b scrub starts 2024-02-19T21:10:10.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:08.878158+0000 osd.7 (osd.7) 186 : cluster [DBG] 2.b scrub ok 2024-02-19T21:10:10.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:09.211318+0000 mgr.y (mgr.14550) 376 : cluster [DBG] pgmap v602: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:10:10.878 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:10.913 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:11.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.669+0000 7f8e368ab700 1 -- 172.21.15.163:0/3552112075 >> v1:172.21.15.163:6789/0 conn(0x7f8e300f6790 legacy=0x7f8e300f6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:11.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.669+0000 7f8e368ab700 1 -- 172.21.15.163:0/3552112075 shutdown_connections 2024-02-19T21:10:11.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.670+0000 7f8e368ab700 1 -- 172.21.15.163:0/3552112075 >> 172.21.15.163:0/3552112075 conn(0x7f8e300f2470 msgr2=0x7f8e300f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:11.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.670+0000 7f8e368ab700 1 -- 172.21.15.163:0/3552112075 shutdown_connections 2024-02-19T21:10:11.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.670+0000 7f8e368ab700 1 -- 172.21.15.163:0/3552112075 wait complete. 2024-02-19T21:10:11.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.670+0000 7f8e368ab700 1 librados: starting msgr at 2024-02-19T21:10:11.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.670+0000 7f8e368ab700 1 librados: starting objecter 2024-02-19T21:10:11.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.670+0000 7f8e368ab700 1 Processor -- start 2024-02-19T21:10:11.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.670+0000 7f8e368ab700 1 -- start start 2024-02-19T21:10:11.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.670+0000 7f8e368ab700 1 librados: setting wanted keys 2024-02-19T21:10:11.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.670+0000 7f8e368ab700 1 librados: calling monclient init 2024-02-19T21:10:11.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e368ab700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8e30119340 con 0x7f8e300f6790 2024-02-19T21:10:11.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e368ab700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8e3011a520 con 0x7f8e300f7280 2024-02-19T21:10:11.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e368ab700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8e3011b700 con 0x7f8e30106530 2024-02-19T21:10:11.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2ffff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8e300f6790 0x7f8e30102710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:49836/0 (socket says 172.21.15.163:49836) 2024-02-19T21:10:11.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2ffff700 1 -- 172.21.15.163:0/3369752311 learned_addr learned my addr 172.21.15.163:0/3369752311 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:11.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e34e48700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f8e30106530 0x7f8e30117b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:50246/0 (socket says 172.21.15.163:50246) 2024-02-19T21:10:11.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 10 client.?.objecter ms_handle_connect 0x7f8e30106530 2024-02-19T21:10:11.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:11.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 10 client.?.objecter ms_handle_connect 0x7f8e300f6790 2024-02-19T21:10:11.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:11.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 10 client.?.objecter ms_handle_connect 0x7f8e300f7280 2024-02-19T21:10:11.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:11.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826794376 0 0) 0x7f8e30119340 con 0x7f8e300f6790 2024-02-19T21:10:11.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e18003030 con 0x7f8e300f6790 2024-02-19T21:10:11.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386872238 0 0) 0x7f8e3011b700 con 0x7f8e30106530 2024-02-19T21:10:11.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e30119340 con 0x7f8e30106530 2024-02-19T21:10:11.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101424696 0 0) 0x7f8e3011a520 con 0x7f8e300f7280 2024-02-19T21:10:11.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e3011b700 con 0x7f8e300f7280 2024-02-19T21:10:11.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1172674826 0 0) 0x7f8e18003030 con 0x7f8e300f6790 2024-02-19T21:10:11.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e3011a520 con 0x7f8e300f6790 2024-02-19T21:10:11.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1466294918 0 0) 0x7f8e30119340 con 0x7f8e30106530 2024-02-19T21:10:11.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e18003030 con 0x7f8e30106530 2024-02-19T21:10:11.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8e20001430 con 0x7f8e300f6790 2024-02-19T21:10:11.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8e28001ba0 con 0x7f8e30106530 2024-02-19T21:10:11.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3303205301 0 0) 0x7f8e3011a520 con 0x7f8e300f6790 2024-02-19T21:10:11.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 >> v1:172.21.15.163:6790/0 conn(0x7f8e30106530 legacy=0x7f8e30117b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:11.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 >> v1:172.21.15.193:6789/0 conn(0x7f8e300f7280 legacy=0x7f8e30102e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:11.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.671+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3011c900 con 0x7f8e300f6790 2024-02-19T21:10:11.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.672+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8e20002400 con 0x7f8e300f6790 2024-02-19T21:10:11.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.672+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8e20003850 con 0x7f8e300f6790 2024-02-19T21:10:11.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.672+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e30119530 con 0x7f8e300f6790 2024-02-19T21:10:11.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.672+0000 7f8e368ab700 10 client.23115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:11.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.672+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e301199a0 con 0x7f8e300f6790 2024-02-19T21:10:11.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.672+0000 7f8e368ab700 1 librados: init done 2024-02-19T21:10:11.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.673+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f8e30060350 con 0x7f8e300f6790 2024-02-19T21:10:11.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.673+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f8e20001840 con 0x7f8e300f6790 2024-02-19T21:10:11.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(416..416 src has 43..416) v4 ==== 7946+0+0 (unknown 3284864456 0 0) 0x7f8e20078fe0 con 0x7f8e300f6790 2024-02-19T21:10:11.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 osd_map(416..416 src has 43..416) v4 2024-02-19T21:10:11.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e2d7fa700 3 client.23115.objecter handle_osd_map got epochs [416,416] > 0 2024-02-19T21:10:11.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e2d7fa700 3 client.23115.objecter handle_osd_map decoding full epoch 416 2024-02-19T21:10:11.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e2d7fa700 20 client.23115.objecter dump_active .. 0 homeless 2024-02-19T21:10:11.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v416) v1 ==== 54+0+16 (unknown 2351755728 0 2420000192) 0x7f8e2004c4c0 con 0x7f8e300f6790 2024-02-19T21:10:11.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e2d7fa700 10 client.23115.objecter ms_handle_connect 0x7f8e18063af0 2024-02-19T21:10:11.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e368ab700 10 client.23115.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:11.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 1 2024-02-19T21:10:11.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e300614c0 osd=0 3 2024-02-19T21:10:11.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e300614c0 osd=0 3 2024-02-19T21:10:11.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e300614c0 osd=0 3 2024-02-19T21:10:11.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 1 2024-02-19T21:10:11.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e300614c0 osd=0 3 2024-02-19T21:10:11.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 0 1 2024-02-19T21:10:11.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e300614c0 osd=0 4 2024-02-19T21:10:11.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e368ab700 10 client.23115.objecter _send_command 1 2024-02-19T21:10:11.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.674+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e30047540 con 0x7f8e3011b890 2024-02-19T21:10:11.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.675+0000 7f8e2d7fa700 10 client.23115.objecter ms_handle_connect 0x7f8e3011b890 2024-02-19T21:10:11.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.675+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8e30047540 con 0x7f8e3011b890 2024-02-19T21:10:11.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.675+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:10:11.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.675+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 1 = system:0 2024-02-19T21:10:11.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.675+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e300614c0 osd=0 4 2024-02-19T21:10:11.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.675+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 0 1 2024-02-19T21:10:11.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.691+0000 7f8e368ab700 10 client.23115.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:11.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.691+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 2 2024-02-19T21:10:11.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.691+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e300614c0 osd=0 3 2024-02-19T21:10:11.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.691+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e300614c0 osd=0 3 2024-02-19T21:10:11.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.691+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e300614c0 osd=0 3 2024-02-19T21:10:11.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.691+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 2 2024-02-19T21:10:11.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.691+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e300614c0 osd=0 3 2024-02-19T21:10:11.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.691+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 0 2 2024-02-19T21:10:11.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.691+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e300614c0 osd=0 4 2024-02-19T21:10:11.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.691+0000 7f8e368ab700 10 client.23115.objecter _send_command 2 2024-02-19T21:10:11.105 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.691+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8e3010c130 con 0x7f8e3011b890 2024-02-19T21:10:11.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.692+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8e3010c130 con 0x7f8e3011b890 2024-02-19T21:10:11.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.692+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:11.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.692+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.692+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e300614c0 osd=0 4 2024-02-19T21:10:11.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.692+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 0 2 2024-02-19T21:10:11.106 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:10:11.107 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:10:11.107 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:10:11.107 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:10:11.107 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:10:11.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.692+0000 7f8e368ab700 10 client.23115.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:11.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.692+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 3 2024-02-19T21:10:11.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.692+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e300f3860 osd=1 3 2024-02-19T21:10:11.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.692+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e300f3860 osd=1 3 2024-02-19T21:10:11.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.692+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e300f3860 osd=1 3 2024-02-19T21:10:11.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.693+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 3 2024-02-19T21:10:11.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.693+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e300f3860 osd=1 3 2024-02-19T21:10:11.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.693+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 1 3 2024-02-19T21:10:11.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.693+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e300f3860 osd=1 4 2024-02-19T21:10:11.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.693+0000 7f8e368ab700 10 client.23115.objecter _send_command 3 2024-02-19T21:10:11.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.693+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e300283d0 con 0x7f8e300f3ee0 2024-02-19T21:10:11.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.693+0000 7f8e2d7fa700 10 client.23115.objecter ms_handle_connect 0x7f8e300f3ee0 2024-02-19T21:10:11.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.693+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8e300283d0 con 0x7f8e300f3ee0 2024-02-19T21:10:11.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.693+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:10:11.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.693+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 3 = system:0 2024-02-19T21:10:11.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.693+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e300f3860 osd=1 4 2024-02-19T21:10:11.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.693+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 1 3 2024-02-19T21:10:11.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.712+0000 7f8e368ab700 10 client.23115.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:11.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.712+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 4 2024-02-19T21:10:11.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.712+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e300f3860 osd=1 3 2024-02-19T21:10:11.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.712+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e300f3860 osd=1 3 2024-02-19T21:10:11.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.712+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e300f3860 osd=1 3 2024-02-19T21:10:11.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.712+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 4 2024-02-19T21:10:11.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.712+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e300f3860 osd=1 3 2024-02-19T21:10:11.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.712+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 1 4 2024-02-19T21:10:11.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.712+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e300f3860 osd=1 4 2024-02-19T21:10:11.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.712+0000 7f8e368ab700 10 client.23115.objecter _send_command 4 2024-02-19T21:10:11.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.712+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8e30028780 con 0x7f8e300f3ee0 2024-02-19T21:10:11.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8e30028780 con 0x7f8e300f3ee0 2024-02-19T21:10:11.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:11.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e300f3860 osd=1 4 2024-02-19T21:10:11.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 1 4 2024-02-19T21:10:11.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e368ab700 10 client.23115.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:11.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 5 2024-02-19T21:10:11.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30119ed0 osd=2 3 2024-02-19T21:10:11.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30119ed0 osd=2 3 2024-02-19T21:10:11.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30119ed0 osd=2 3 2024-02-19T21:10:11.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 5 2024-02-19T21:10:11.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e30119ed0 osd=2 3 2024-02-19T21:10:11.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 2 5 2024-02-19T21:10:11.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30119ed0 osd=2 4 2024-02-19T21:10:11.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e368ab700 10 client.23115.objecter _send_command 5 2024-02-19T21:10:11.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e30059e90 con 0x7f8e3011a040 2024-02-19T21:10:11.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.714+0000 7f8e2d7fa700 10 client.23115.objecter ms_handle_connect 0x7f8e3011a040 2024-02-19T21:10:11.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.715+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8e30059e90 con 0x7f8e3011a040 2024-02-19T21:10:11.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.715+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:10:11.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.715+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 5 = system:0 2024-02-19T21:10:11.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.715+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e30119ed0 osd=2 4 2024-02-19T21:10:11.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.715+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 2 5 2024-02-19T21:10:11.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.730+0000 7f8e368ab700 10 client.23115.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:11.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.730+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 6 2024-02-19T21:10:11.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.730+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30119ed0 osd=2 3 2024-02-19T21:10:11.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.730+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30119ed0 osd=2 3 2024-02-19T21:10:11.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.730+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30119ed0 osd=2 3 2024-02-19T21:10:11.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.730+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 6 2024-02-19T21:10:11.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.730+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e30119ed0 osd=2 3 2024-02-19T21:10:11.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.730+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 2 6 2024-02-19T21:10:11.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.730+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30119ed0 osd=2 4 2024-02-19T21:10:11.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.730+0000 7f8e368ab700 10 client.23115.objecter _send_command 6 2024-02-19T21:10:11.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.730+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8e3005a0b0 con 0x7f8e3011a040 2024-02-19T21:10:11.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8e3005a0b0 con 0x7f8e3011a040 2024-02-19T21:10:11.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:11.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e30119ed0 osd=2 4 2024-02-19T21:10:11.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 2 6 2024-02-19T21:10:11.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e368ab700 10 client.23115.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:11.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 7 2024-02-19T21:10:11.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e300595f0 osd=3 3 2024-02-19T21:10:11.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e300595f0 osd=3 3 2024-02-19T21:10:11.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e300595f0 osd=3 3 2024-02-19T21:10:11.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 7 2024-02-19T21:10:11.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e300595f0 osd=3 3 2024-02-19T21:10:11.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 3 7 2024-02-19T21:10:11.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e300595f0 osd=3 4 2024-02-19T21:10:11.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e368ab700 10 client.23115.objecter _send_command 7 2024-02-19T21:10:11.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e30129cd0 con 0x7f8e30121850 2024-02-19T21:10:11.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.732+0000 7f8e2d7fa700 10 client.23115.objecter ms_handle_connect 0x7f8e30121850 2024-02-19T21:10:11.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.733+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8e30129cd0 con 0x7f8e30121850 2024-02-19T21:10:11.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.733+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:10:11.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.733+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 7 = system:0 2024-02-19T21:10:11.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.733+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e300595f0 osd=3 4 2024-02-19T21:10:11.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.733+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 3 7 2024-02-19T21:10:11.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.747+0000 7f8e368ab700 10 client.23115.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:11.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.747+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 8 2024-02-19T21:10:11.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.747+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e300595f0 osd=3 3 2024-02-19T21:10:11.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.747+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e300595f0 osd=3 3 2024-02-19T21:10:11.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.747+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e300595f0 osd=3 3 2024-02-19T21:10:11.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.747+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 8 2024-02-19T21:10:11.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.747+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e300595f0 osd=3 3 2024-02-19T21:10:11.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.747+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 3 8 2024-02-19T21:10:11.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.747+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e300595f0 osd=3 4 2024-02-19T21:10:11.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.747+0000 7f8e368ab700 10 client.23115.objecter _send_command 8 2024-02-19T21:10:11.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.747+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8e30120db0 con 0x7f8e30121850 2024-02-19T21:10:11.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.748+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8e30120db0 con 0x7f8e30121850 2024-02-19T21:10:11.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.748+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:11.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.748+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.748+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e300595f0 osd=3 4 2024-02-19T21:10:11.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.748+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 3 8 2024-02-19T21:10:11.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.749+0000 7f8e368ab700 10 client.23115.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:11.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.749+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 9 2024-02-19T21:10:11.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.749+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30120f50 osd=4 3 2024-02-19T21:10:11.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.749+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30120f50 osd=4 3 2024-02-19T21:10:11.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.749+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30120f50 osd=4 3 2024-02-19T21:10:11.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.749+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 9 2024-02-19T21:10:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.749+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e30120f50 osd=4 3 2024-02-19T21:10:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.749+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 4 9 2024-02-19T21:10:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.749+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30120f50 osd=4 4 2024-02-19T21:10:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.749+0000 7f8e368ab700 10 client.23115.objecter _send_command 9 2024-02-19T21:10:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.749+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e30121100 con 0x7f8e30126100 2024-02-19T21:10:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.749+0000 7f8e2d7fa700 10 client.23115.objecter ms_handle_connect 0x7f8e30126100 2024-02-19T21:10:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.750+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8e30121100 con 0x7f8e30126100 2024-02-19T21:10:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.750+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:10:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.750+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 9 = system:0 2024-02-19T21:10:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.750+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e30120f50 osd=4 4 2024-02-19T21:10:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.750+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 4 9 2024-02-19T21:10:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.765+0000 7f8e368ab700 10 client.23115.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.765+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 10 2024-02-19T21:10:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.765+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30120f50 osd=4 3 2024-02-19T21:10:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.765+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30120f50 osd=4 3 2024-02-19T21:10:11.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.765+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30120f50 osd=4 3 2024-02-19T21:10:11.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.765+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 10 2024-02-19T21:10:11.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.765+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e30120f50 osd=4 3 2024-02-19T21:10:11.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.765+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 4 10 2024-02-19T21:10:11.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.765+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30120f50 osd=4 4 2024-02-19T21:10:11.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.765+0000 7f8e368ab700 10 client.23115.objecter _send_command 10 2024-02-19T21:10:11.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.765+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8e30121100 con 0x7f8e30126100 2024-02-19T21:10:11.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8e30121100 con 0x7f8e30126100 2024-02-19T21:10:11.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:11.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e30120f50 osd=4 4 2024-02-19T21:10:11.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 4 10 2024-02-19T21:10:11.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e368ab700 10 client.23115.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:11.120 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:10:11.120 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:10:11.120 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:10:11.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 11 2024-02-19T21:10:11.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30121100 osd=5 3 2024-02-19T21:10:11.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30121100 osd=5 3 2024-02-19T21:10:11.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30121100 osd=5 3 2024-02-19T21:10:11.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 11 2024-02-19T21:10:11.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e30121100 osd=5 3 2024-02-19T21:10:11.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 5 11 2024-02-19T21:10:11.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30121100 osd=5 4 2024-02-19T21:10:11.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e368ab700 10 client.23115.objecter _send_command 11 2024-02-19T21:10:11.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e3012c180 con 0x7f8e3012bbe0 2024-02-19T21:10:11.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.767+0000 7f8e2d7fa700 10 client.23115.objecter ms_handle_connect 0x7f8e3012bbe0 2024-02-19T21:10:11.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.768+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8e3012c180 con 0x7f8e3012bbe0 2024-02-19T21:10:11.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.768+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:10:11.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.768+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 11 = system:0 2024-02-19T21:10:11.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.768+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e30121100 osd=5 4 2024-02-19T21:10:11.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.768+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 5 11 2024-02-19T21:10:11.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.786+0000 7f8e368ab700 10 client.23115.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:11.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.786+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 12 2024-02-19T21:10:11.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.786+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30121100 osd=5 3 2024-02-19T21:10:11.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.786+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30121100 osd=5 3 2024-02-19T21:10:11.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.786+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30121100 osd=5 3 2024-02-19T21:10:11.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.786+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 12 2024-02-19T21:10:11.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.786+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e30121100 osd=5 3 2024-02-19T21:10:11.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.786+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 5 12 2024-02-19T21:10:11.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.786+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30121100 osd=5 4 2024-02-19T21:10:11.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.786+0000 7f8e368ab700 10 client.23115.objecter _send_command 12 2024-02-19T21:10:11.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.786+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8e3012c180 con 0x7f8e3012bbe0 2024-02-19T21:10:11.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8e3012c180 con 0x7f8e3012bbe0 2024-02-19T21:10:11.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:11.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e30121100 osd=5 4 2024-02-19T21:10:11.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 5 12 2024-02-19T21:10:11.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e368ab700 10 client.23115.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:11.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 13 2024-02-19T21:10:11.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e3012c180 osd=6 3 2024-02-19T21:10:11.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e3012c180 osd=6 3 2024-02-19T21:10:11.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e3012c180 osd=6 3 2024-02-19T21:10:11.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 13 2024-02-19T21:10:11.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e3012c180 osd=6 3 2024-02-19T21:10:11.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 6 13 2024-02-19T21:10:11.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e3012c180 osd=6 4 2024-02-19T21:10:11.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e368ab700 10 client.23115.objecter _send_command 13 2024-02-19T21:10:11.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.788+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e30136dd0 con 0x7f8e30133250 2024-02-19T21:10:11.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.789+0000 7f8e2d7fa700 10 client.23115.objecter ms_handle_connect 0x7f8e30133250 2024-02-19T21:10:11.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.790+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8e30136dd0 con 0x7f8e30133250 2024-02-19T21:10:11.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.790+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:10:11.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.790+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 13 = system:0 2024-02-19T21:10:11.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.790+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e3012c180 osd=6 4 2024-02-19T21:10:11.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.790+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 6 13 2024-02-19T21:10:11.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.805+0000 7f8e368ab700 10 client.23115.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:11.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.805+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 14 2024-02-19T21:10:11.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.805+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e3012c180 osd=6 3 2024-02-19T21:10:11.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.805+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e3012c180 osd=6 3 2024-02-19T21:10:11.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.805+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e3012c180 osd=6 3 2024-02-19T21:10:11.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.805+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 14 2024-02-19T21:10:11.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.805+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e3012c180 osd=6 3 2024-02-19T21:10:11.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.805+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 6 14 2024-02-19T21:10:11.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.805+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e3012c180 osd=6 4 2024-02-19T21:10:11.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.805+0000 7f8e368ab700 10 client.23115.objecter _send_command 14 2024-02-19T21:10:11.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.805+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8e30136dd0 con 0x7f8e30133250 2024-02-19T21:10:11.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8e30136dd0 con 0x7f8e30133250 2024-02-19T21:10:11.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:11.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e3012c180 osd=6 4 2024-02-19T21:10:11.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 6 14 2024-02-19T21:10:11.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e368ab700 10 client.23115.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:11.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 15 2024-02-19T21:10:11.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30136dd0 osd=7 3 2024-02-19T21:10:11.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30136dd0 osd=7 3 2024-02-19T21:10:11.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30136dd0 osd=7 3 2024-02-19T21:10:11.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 15 2024-02-19T21:10:11.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e30136dd0 osd=7 3 2024-02-19T21:10:11.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 7 15 2024-02-19T21:10:11.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30136dd0 osd=7 4 2024-02-19T21:10:11.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e368ab700 10 client.23115.objecter _send_command 15 2024-02-19T21:10:11.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.807+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e30138020 con 0x7f8e30138ad0 2024-02-19T21:10:11.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.808+0000 7f8e2d7fa700 10 client.23115.objecter ms_handle_connect 0x7f8e30138ad0 2024-02-19T21:10:11.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.808+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8e30138020 con 0x7f8e30138ad0 2024-02-19T21:10:11.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.808+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:10:11.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.808+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 15 = system:0 2024-02-19T21:10:11.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.808+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e30136dd0 osd=7 4 2024-02-19T21:10:11.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.808+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 7 15 2024-02-19T21:10:11.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.823+0000 7f8e368ab700 10 client.23115.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:11.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.823+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign -1 16 2024-02-19T21:10:11.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.823+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30136dd0 osd=7 3 2024-02-19T21:10:11.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.823+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30136dd0 osd=7 3 2024-02-19T21:10:11.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.823+0000 7f8e368ab700 20 client.23115.objecter _get_session s=0x7f8e30136dd0 osd=7 3 2024-02-19T21:10:11.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.823+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_remove -1 16 2024-02-19T21:10:11.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.823+0000 7f8e368ab700 20 client.23115.objecter get_session s=0x7f8e30136dd0 osd=7 3 2024-02-19T21:10:11.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.823+0000 7f8e368ab700 15 client.23115.objecter _session_command_op_assign 7 16 2024-02-19T21:10:11.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.823+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30136dd0 osd=7 4 2024-02-19T21:10:11.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.823+0000 7f8e368ab700 10 client.23115.objecter _send_command 16 2024-02-19T21:10:11.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.823+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8e30138020 con 0x7f8e30138ad0 2024-02-19T21:10:11.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e2d7fa700 1 -- 172.21.15.163:0/3369752311 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8e30138020 con 0x7f8e30138ad0 2024-02-19T21:10:11.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e2d7fa700 10 client.23115.objecter ms_dispatch 0x7f8e30001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:11.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e2d7fa700 10 client.23115.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e2d7fa700 20 client.23115.objecter put_session s=0x7f8e30136dd0 osd=7 4 2024-02-19T21:10:11.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e2d7fa700 15 client.23115.objecter _session_command_op_remove 7 16 2024-02-19T21:10:11.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 10 librados: watch_flush enter 2024-02-19T21:10:11.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 10 librados: watch_flush exit 2024-02-19T21:10:11.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 10 client.23115.objecter close_session for osd.0 2024-02-19T21:10:11.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 >> v1:172.21.15.163:6801/1559418115 conn(0x7f8e3011b890 legacy=0x7f8e3011bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:11.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e300614c0 osd=0 1 2024-02-19T21:10:11.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 10 client.23115.objecter close_session for osd.1 2024-02-19T21:10:11.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 >> v1:172.21.15.163:6805/3125047432 conn(0x7f8e300f3ee0 legacy=0x7f8e300f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:11.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e300f3860 osd=1 1 2024-02-19T21:10:11.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 10 client.23115.objecter close_session for osd.2 2024-02-19T21:10:11.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 >> v1:172.21.15.163:6809/1563158487 conn(0x7f8e3011a040 legacy=0x7f8e30123d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:11.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30119ed0 osd=2 1 2024-02-19T21:10:11.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 10 client.23115.objecter close_session for osd.3 2024-02-19T21:10:11.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 >> v1:172.21.15.163:6813/2112279769 conn(0x7f8e30121850 legacy=0x7f8e301285b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:11.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e300595f0 osd=3 1 2024-02-19T21:10:11.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 10 client.23115.objecter close_session for osd.4 2024-02-19T21:10:11.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 >> v1:172.21.15.193:6800/3129716965 conn(0x7f8e30126100 legacy=0x7f8e3012afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:11.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e34e48700 1 -- 172.21.15.163:0/3369752311 reap_dead start 2024-02-19T21:10:11.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30120f50 osd=4 1 2024-02-19T21:10:11.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 10 client.23115.objecter close_session for osd.5 2024-02-19T21:10:11.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 >> v1:172.21.15.193:6804/805161991 conn(0x7f8e3012bbe0 legacy=0x7f8e30131620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:11.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30121100 osd=5 1 2024-02-19T21:10:11.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 10 client.23115.objecter close_session for osd.6 2024-02-19T21:10:11.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 >> v1:172.21.15.193:6808/787604615 conn(0x7f8e30133250 legacy=0x7f8e30135640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:11.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e34e48700 1 -- 172.21.15.163:0/3369752311 reap_dead start 2024-02-19T21:10:11.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e34e48700 1 -- 172.21.15.163:0/3369752311 reap_dead start 2024-02-19T21:10:11.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e3012c180 osd=6 1 2024-02-19T21:10:11.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 10 client.23115.objecter close_session for osd.7 2024-02-19T21:10:11.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 >> v1:172.21.15.193:6812/157927432 conn(0x7f8e30138ad0 legacy=0x7f8e30138f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:11.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 20 client.23115.objecter put_session s=0x7f8e30136dd0 osd=7 1 2024-02-19T21:10:11.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 20 client.23115.objecter shutdown clearing up homeless session... 2024-02-19T21:10:11.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 10 client.23115.objecter successfully canceled tick 2024-02-19T21:10:11.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8e18063af0 legacy=0x7f8e18065fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:11.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 >> v1:172.21.15.163:6789/0 conn(0x7f8e300f6790 legacy=0x7f8e30102710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:11.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 shutdown_connections 2024-02-19T21:10:11.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 >> 172.21.15.163:0/3369752311 conn(0x7f8e300f2470 msgr2=0x7f8e30108940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:11.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 shutdown_connections 2024-02-19T21:10:11.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 1 -- 172.21.15.163:0/3369752311 wait complete. 2024-02-19T21:10:11.130 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:10.825+0000 7f8e368ab700 1 librados: shutdown 2024-02-19T21:10:11.130 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.130 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.130 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.130 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.130 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.130 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.130 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.131 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:10:11.371 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:10.134459+0000 osd.2 (osd.2) 158 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:10:11.372 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:10.137352+0000 osd.2 (osd.2) 159 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:10:11.372 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:10.145749+0000 osd.5 (osd.5) 166 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:10:11.372 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:10.148729+0000 osd.5 (osd.5) 167 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:10:11.372 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:10.391605+0000 mgr.y (mgr.14550) 377 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:11.372 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:10.649379+0000 osd.3 (osd.3) 183 : cluster [DBG] 4.c scrub starts 2024-02-19T21:10:11.372 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:10.650658+0000 osd.3 (osd.3) 184 : cluster [DBG] 4.c scrub ok 2024-02-19T21:10:11.372 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:10.675383+0000 mon.a (mon.0) 2520 : audit [DBG] from='client.? v1:172.21.15.163:0/3369752311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:11.372 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:11.209724+0000 mon.a (mon.0) 2521 : audit [INF] from='client.? v1:172.21.15.163:0/54770121' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:10:11.372 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:10.134459+0000 osd.2 (osd.2) 158 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:10:11.372 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:10.137352+0000 osd.2 (osd.2) 159 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:10:11.373 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:10.145749+0000 osd.5 (osd.5) 166 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:10:11.373 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:10.148729+0000 osd.5 (osd.5) 167 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:10:11.373 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:10.391605+0000 mgr.y (mgr.14550) 377 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:11.373 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:10.649379+0000 osd.3 (osd.3) 183 : cluster [DBG] 4.c scrub starts 2024-02-19T21:10:11.373 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:10.650658+0000 osd.3 (osd.3) 184 : cluster [DBG] 4.c scrub ok 2024-02-19T21:10:11.373 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:10.675383+0000 mon.a (mon.0) 2520 : audit [DBG] from='client.? v1:172.21.15.163:0/3369752311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:11.373 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:11.209724+0000 mon.a (mon.0) 2521 : audit [INF] from='client.? v1:172.21.15.163:0/54770121' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:10:11.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:10.134459+0000 osd.2 (osd.2) 158 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:10:11.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:10.137352+0000 osd.2 (osd.2) 159 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:10:11.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:10.145749+0000 osd.5 (osd.5) 166 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:10:11.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:10.148729+0000 osd.5 (osd.5) 167 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:10:11.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:10.391605+0000 mgr.y (mgr.14550) 377 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:11.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:10.649379+0000 osd.3 (osd.3) 183 : cluster [DBG] 4.c scrub starts 2024-02-19T21:10:11.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:10.650658+0000 osd.3 (osd.3) 184 : cluster [DBG] 4.c scrub ok 2024-02-19T21:10:11.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:10.675383+0000 mon.a (mon.0) 2520 : audit [DBG] from='client.? v1:172.21.15.163:0/3369752311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:11.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:11.209724+0000 mon.a (mon.0) 2521 : audit [INF] from='client.? v1:172.21.15.163:0/54770121' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:10:12.741 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:12.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:10.639583+0000 osd.1 (osd.1) 191 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:10:12.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:10.642595+0000 osd.1 (osd.1) 192 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:10:12.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:11.162369+0000 osd.5 (osd.5) 168 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:10:12.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:11.165966+0000 osd.5 (osd.5) 169 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:10:12.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:11.212079+0000 mgr.y (mgr.14550) 378 : cluster [DBG] pgmap v603: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-19T21:10:12.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:11.363617+0000 mon.a (mon.0) 2522 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:10:12.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:11.367387+0000 mon.a (mon.0) 2523 : audit [INF] from='client.? v1:172.21.15.163:0/54770121' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:10:12.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:11.367439+0000 mon.a (mon.0) 2524 : cluster [DBG] osdmap e417: 8 total, 8 up, 7 in 2024-02-19T21:10:12.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:11.410957+0000 osd.6 (osd.6) 155 : cluster [DBG] 16.d scrub starts 2024-02-19T21:10:12.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:11.412265+0000 osd.6 (osd.6) 156 : cluster [DBG] 16.d scrub ok 2024-02-19T21:10:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:10.639583+0000 osd.1 (osd.1) 191 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:10:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:10.642595+0000 osd.1 (osd.1) 192 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:10:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:11.162369+0000 osd.5 (osd.5) 168 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:10:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:11.165966+0000 osd.5 (osd.5) 169 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:10:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:11.212079+0000 mgr.y (mgr.14550) 378 : cluster [DBG] pgmap v603: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-19T21:10:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:11.363617+0000 mon.a (mon.0) 2522 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:10:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:11.367387+0000 mon.a (mon.0) 2523 : audit [INF] from='client.? v1:172.21.15.163:0/54770121' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:10:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:11.367439+0000 mon.a (mon.0) 2524 : cluster [DBG] osdmap e417: 8 total, 8 up, 7 in 2024-02-19T21:10:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:11.410957+0000 osd.6 (osd.6) 155 : cluster [DBG] 16.d scrub starts 2024-02-19T21:10:12.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:11.412265+0000 osd.6 (osd.6) 156 : cluster [DBG] 16.d scrub ok 2024-02-19T21:10:12.746 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:12.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:12.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:10.639583+0000 osd.1 (osd.1) 191 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:10:12.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:10.642595+0000 osd.1 (osd.1) 192 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:10:12.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:11.162369+0000 osd.5 (osd.5) 168 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:10:12.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:11.165966+0000 osd.5 (osd.5) 169 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:10:12.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:11.212079+0000 mgr.y (mgr.14550) 378 : cluster [DBG] pgmap v603: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-19T21:10:12.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:11.363617+0000 mon.a (mon.0) 2522 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:10:12.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:11.367387+0000 mon.a (mon.0) 2523 : audit [INF] from='client.? v1:172.21.15.163:0/54770121' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:10:12.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:11.367439+0000 mon.a (mon.0) 2524 : cluster [DBG] osdmap e417: 8 total, 8 up, 7 in 2024-02-19T21:10:12.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:11.410957+0000 osd.6 (osd.6) 155 : cluster [DBG] 16.d scrub starts 2024-02-19T21:10:12.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:11.412265+0000 osd.6 (osd.6) 156 : cluster [DBG] 16.d scrub ok 2024-02-19T21:10:12.847 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:10:12.876 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:12.949 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:10:12.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:12.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:10:13.053 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:13.066 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:10:13.066 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:10:13.080 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:13.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.022+0000 7f8e0b369700 1 -- 172.21.15.163:0/3540825411 >> v1:172.21.15.163:6789/0 conn(0x7f8e041042c0 legacy=0x7f8e04106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:13.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.022+0000 7f8e0b369700 1 -- 172.21.15.163:0/3540825411 shutdown_connections 2024-02-19T21:10:13.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.022+0000 7f8e0b369700 1 -- 172.21.15.163:0/3540825411 >> 172.21.15.163:0/3540825411 conn(0x7f8e040f35a0 msgr2=0x7f8e040f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:13.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.022+0000 7f8e0b369700 1 -- 172.21.15.163:0/3540825411 shutdown_connections 2024-02-19T21:10:13.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.022+0000 7f8e0b369700 1 -- 172.21.15.163:0/3540825411 wait complete. 2024-02-19T21:10:13.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8e0b369700 1 librados: starting msgr at 2024-02-19T21:10:13.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8e0b369700 1 librados: starting objecter 2024-02-19T21:10:13.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8e0b369700 1 Processor -- start 2024-02-19T21:10:13.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8e0b369700 1 -- start start 2024-02-19T21:10:13.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8e0b369700 1 librados: setting wanted keys 2024-02-19T21:10:13.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8e0b369700 1 librados: calling monclient init 2024-02-19T21:10:13.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8e0b369700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8e041171c0 con 0x7f8e04100780 2024-02-19T21:10:13.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8e0b369700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8e041183a0 con 0x7f8e041042c0 2024-02-19T21:10:13.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8e0b369700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8e041195a0 con 0x7f8e040fd980 2024-02-19T21:10:13.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8e08904700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8e04100780 0x7f8e040f9e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:49854/0 (socket says 172.21.15.163:49854) 2024-02-19T21:10:13.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8e08904700 1 -- 172.21.15.163:0/54770121 learned_addr learned my addr 172.21.15.163:0/54770121 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:13.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8e09105700 1 --1- 172.21.15.163:0/54770121 >> v1:172.21.15.163:6790/0 conn(0x7f8e040fd980 0x7f8e040f9780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:50266/0 (socket says 172.21.15.163:50266) 2024-02-19T21:10:13.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8dfa7fc700 10 client.?.objecter ms_handle_connect 0x7f8e04100780 2024-02-19T21:10:13.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8dfa7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:13.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8dfa7fc700 10 client.?.objecter ms_handle_connect 0x7f8e040fd980 2024-02-19T21:10:13.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8dfa7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:13.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399497945 0 0) 0x7f8e041171c0 con 0x7f8e04100780 2024-02-19T21:10:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df00038e0 con 0x7f8e04100780 2024-02-19T21:10:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8dfa7fc700 10 client.?.objecter ms_handle_connect 0x7f8e041042c0 2024-02-19T21:10:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8dfa7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367413416 0 0) 0x7f8e041195a0 con 0x7f8e040fd980 2024-02-19T21:10:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.023+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e041171c0 con 0x7f8e040fd980 2024-02-19T21:10:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283848754 0 0) 0x7f8e041183a0 con 0x7f8e041042c0 2024-02-19T21:10:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e041195a0 con 0x7f8e041042c0 2024-02-19T21:10:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4030290778 0 0) 0x7f8df00038e0 con 0x7f8e04100780 2024-02-19T21:10:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e041183a0 con 0x7f8e04100780 2024-02-19T21:10:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8e00001920 con 0x7f8e04100780 2024-02-19T21:10:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3972307385 0 0) 0x7f8e041171c0 con 0x7f8e040fd980 2024-02-19T21:10:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8df00038e0 con 0x7f8e040fd980 2024-02-19T21:10:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8df4001880 con 0x7f8e040fd980 2024-02-19T21:10:13.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 771153815 0 0) 0x7f8e041195a0 con 0x7f8e041042c0 2024-02-19T21:10:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e041171c0 con 0x7f8e041042c0 2024-02-19T21:10:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8dfc001950 con 0x7f8e041042c0 2024-02-19T21:10:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 272634785 0 0) 0x7f8e041183a0 con 0x7f8e04100780 2024-02-19T21:10:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 >> v1:172.21.15.163:6790/0 conn(0x7f8e040fd980 legacy=0x7f8e040f9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 >> v1:172.21.15.193:6789/0 conn(0x7f8e041042c0 legacy=0x7f8e04115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0411a7a0 con 0x7f8e04100780 2024-02-19T21:10:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8e0b369700 1 -- 172.21.15.163:0/54770121 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e04119790 con 0x7f8e04100780 2024-02-19T21:10:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8e0b369700 10 client.23121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8e0b369700 1 -- 172.21.15.163:0/54770121 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e04119c00 con 0x7f8e04100780 2024-02-19T21:10:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8e0b369700 1 librados: init done 2024-02-19T21:10:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8e000023a0 con 0x7f8e04100780 2024-02-19T21:10:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.024+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8e00005a50 con 0x7f8e04100780 2024-02-19T21:10:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.025+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f8e000188e0 con 0x7f8e04100780 2024-02-19T21:10:13.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.026+0000 7f8e0b369700 1 -- 172.21.15.163:0/54770121 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de8004f20 con 0x7f8e04100780 2024-02-19T21:10:13.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.026+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(416..416 src has 43..416) v4 ==== 7946+0+0 (unknown 3284864456 0 0) 0x7f8e00078be0 con 0x7f8e04100780 2024-02-19T21:10:13.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.026+0000 7f8dfa7fc700 10 client.23121.objecter ms_dispatch 0x7f8e04001550 osd_map(416..416 src has 43..416) v4 2024-02-19T21:10:13.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.026+0000 7f8dfa7fc700 3 client.23121.objecter handle_osd_map got epochs [416,416] > 0 2024-02-19T21:10:13.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.026+0000 7f8dfa7fc700 3 client.23121.objecter handle_osd_map decoding full epoch 416 2024-02-19T21:10:13.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.026+0000 7f8dfa7fc700 20 client.23121.objecter dump_active .. 0 homeless 2024-02-19T21:10:13.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.026+0000 7f8dfa7fc700 10 client.23121.objecter ms_handle_connect 0x7f8df0063bd0 2024-02-19T21:10:13.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.030+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8e0004c040 con 0x7f8e04100780 2024-02-19T21:10:13.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.208+0000 7f8e0b369700 1 -- 172.21.15.163:0/54770121 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f8de80051d0 con 0x7f8e04100780 2024-02-19T21:10:13.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.368+0000 7f8dfa7fc700 1 -- 172.21.15.163:0/54770121 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v417) v1 ==== 101+0+0 (unknown 516737330 0 0) 0x7f8e0004a450 con 0x7f8e04100780 2024-02-19T21:10:13.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.371+0000 7f8e0b369700 10 librados: watch_flush enter 2024-02-19T21:10:13.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.371+0000 7f8e0b369700 10 librados: watch_flush exit 2024-02-19T21:10:13.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.371+0000 7f8e0b369700 20 client.23121.objecter shutdown clearing up homeless session... 2024-02-19T21:10:13.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.371+0000 7f8e0b369700 10 client.23121.objecter successfully canceled tick 2024-02-19T21:10:13.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.371+0000 7f8e0b369700 1 -- 172.21.15.163:0/54770121 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8df0063bd0 legacy=0x7f8df0066090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:13.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.371+0000 7f8e0b369700 1 -- 172.21.15.163:0/54770121 >> v1:172.21.15.163:6789/0 conn(0x7f8e04100780 legacy=0x7f8e040f9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:13.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.371+0000 7f8e0b369700 1 -- 172.21.15.163:0/54770121 shutdown_connections 2024-02-19T21:10:13.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.371+0000 7f8e0b369700 1 -- 172.21.15.163:0/54770121 >> 172.21.15.163:0/54770121 conn(0x7f8e040f35a0 msgr2=0x7f8e04107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:13.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.373+0000 7f8e0b369700 1 -- 172.21.15.163:0/54770121 shutdown_connections 2024-02-19T21:10:13.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.375+0000 7f8e0b369700 1 -- 172.21.15.163:0/54770121 wait complete. 2024-02-19T21:10:13.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:11.375+0000 7f8e0b369700 1 librados: shutdown 2024-02-19T21:10:13.564 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:10:13.745 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:10:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:10:13] "GET /metrics HTTP/1.1" 200 189597 "" "Prometheus/2.33.4" 2024-02-19T21:10:13.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:12.182721+0000 osd.5 (osd.5) 170 : cluster [DBG] 5.e scrub starts 2024-02-19T21:10:13.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:12.186168+0000 osd.5 (osd.5) 171 : cluster [DBG] 5.e scrub ok 2024-02-19T21:10:13.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:12.453460+0000 osd.6 (osd.6) 157 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:10:13.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:12.455048+0000 osd.6 (osd.6) 158 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:10:13.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:12.182721+0000 osd.5 (osd.5) 170 : cluster [DBG] 5.e scrub starts 2024-02-19T21:10:13.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:12.186168+0000 osd.5 (osd.5) 171 : cluster [DBG] 5.e scrub ok 2024-02-19T21:10:13.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:12.453460+0000 osd.6 (osd.6) 157 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:10:13.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:12.455048+0000 osd.6 (osd.6) 158 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:10:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:12.182721+0000 osd.5 (osd.5) 170 : cluster [DBG] 5.e scrub starts 2024-02-19T21:10:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:12.186168+0000 osd.5 (osd.5) 171 : cluster [DBG] 5.e scrub ok 2024-02-19T21:10:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:12.453460+0000 osd.6 (osd.6) 157 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:10:13.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:12.455048+0000 osd.6 (osd.6) 158 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:10:14.129 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:14.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:14.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:14.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:14.267 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:14.288 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:14.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:14.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:14.377 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:12.625855+0000 osd.1 (osd.1) 193 : cluster [DBG] 2.a scrub starts 2024-02-19T21:10:14.377 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:12.628797+0000 osd.1 (osd.1) 194 : cluster [DBG] 2.a scrub ok 2024-02-19T21:10:14.391 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:14.421 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:14.441 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:14.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:14.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:14.545 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:14.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:12.625855+0000 osd.1 (osd.1) 193 : cluster [DBG] 2.a scrub starts 2024-02-19T21:10:14.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:12.628797+0000 osd.1 (osd.1) 194 : cluster [DBG] 2.a scrub ok 2024-02-19T21:10:14.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:13.035157+0000 osd.2 (osd.2) 160 : cluster [DBG] 5.a scrub starts 2024-02-19T21:10:14.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:13.035935+0000 osd.2 (osd.2) 161 : cluster [DBG] 5.a scrub ok 2024-02-19T21:10:14.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:13.212599+0000 mgr.y (mgr.14550) 379 : cluster [DBG] pgmap v605: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 902 B/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:10:14.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:14.262382+0000 mon.a (mon.0) 2525 : cluster [DBG] osdmap e418: 8 total, 8 up, 7 in 2024-02-19T21:10:14.572 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:14.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:13.035157+0000 osd.2 (osd.2) 160 : cluster [DBG] 5.a scrub starts 2024-02-19T21:10:14.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:13.035935+0000 osd.2 (osd.2) 161 : cluster [DBG] 5.a scrub ok 2024-02-19T21:10:14.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:13.212599+0000 mgr.y (mgr.14550) 379 : cluster [DBG] pgmap v605: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 902 B/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:10:14.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:14.262382+0000 mon.a (mon.0) 2525 : cluster [DBG] osdmap e418: 8 total, 8 up, 7 in 2024-02-19T21:10:14.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:12.625855+0000 osd.1 (osd.1) 193 : cluster [DBG] 2.a scrub starts 2024-02-19T21:10:14.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:12.628797+0000 osd.1 (osd.1) 194 : cluster [DBG] 2.a scrub ok 2024-02-19T21:10:14.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:13.035157+0000 osd.2 (osd.2) 160 : cluster [DBG] 5.a scrub starts 2024-02-19T21:10:14.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:13.035935+0000 osd.2 (osd.2) 161 : cluster [DBG] 5.a scrub ok 2024-02-19T21:10:14.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:13.212599+0000 mgr.y (mgr.14550) 379 : cluster [DBG] pgmap v605: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 902 B/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:10:14.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:14.262382+0000 mon.a (mon.0) 2525 : cluster [DBG] osdmap e418: 8 total, 8 up, 7 in 2024-02-19T21:10:15.103 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-19T21:10:15.120 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:15.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:15.224 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:10:15.256 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:15.277 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:10:15.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:15.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:10:15.382 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:15.383 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:13.755738+0000 osd.7 (osd.7) 187 : cluster [DBG] 61.f scrub starts 2024-02-19T21:10:15.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:13.757793+0000 osd.7 (osd.7) 188 : cluster [DBG] 61.f scrub ok 2024-02-19T21:10:15.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:14.011887+0000 osd.2 (osd.2) 162 : cluster [DBG] 16.1 scrub starts 2024-02-19T21:10:15.384 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:14.015408+0000 osd.2 (osd.2) 163 : cluster [DBG] 16.1 scrub ok 2024-02-19T21:10:15.391 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:15.411 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:15.431 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:15.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:15.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:15.474 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:10:15.535 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:10:15.562 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:15.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:13.755738+0000 osd.7 (osd.7) 187 : cluster [DBG] 61.f scrub starts 2024-02-19T21:10:15.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:13.757793+0000 osd.7 (osd.7) 188 : cluster [DBG] 61.f scrub ok 2024-02-19T21:10:15.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:14.011887+0000 osd.2 (osd.2) 162 : cluster [DBG] 16.1 scrub starts 2024-02-19T21:10:15.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:14.015408+0000 osd.2 (osd.2) 163 : cluster [DBG] 16.1 scrub ok 2024-02-19T21:10:15.634 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:10:15.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:15.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:10:15.639 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:13.755738+0000 osd.7 (osd.7) 187 : cluster [DBG] 61.f scrub starts 2024-02-19T21:10:15.639 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:13.757793+0000 osd.7 (osd.7) 188 : cluster [DBG] 61.f scrub ok 2024-02-19T21:10:15.639 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:14.011887+0000 osd.2 (osd.2) 162 : cluster [DBG] 16.1 scrub starts 2024-02-19T21:10:15.639 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:14.015408+0000 osd.2 (osd.2) 163 : cluster [DBG] 16.1 scrub ok 2024-02-19T21:10:15.740 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:15.767 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:15.890 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:15.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:15.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:15.994 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:16.026 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:16.048 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:16.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:16.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:16.149 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:10:16.152 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:16.223 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:16.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:14.627873+0000 osd.1 (osd.1) 195 : cluster [DBG] 16.2 scrub starts 2024-02-19T21:10:16.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:14.631325+0000 osd.1 (osd.1) 196 : cluster [DBG] 16.2 scrub ok 2024-02-19T21:10:16.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:15.158397+0000 osd.5 (osd.5) 172 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:10:16.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:15.161393+0000 osd.5 (osd.5) 173 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:10:16.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:15.213485+0000 mgr.y (mgr.14550) 380 : cluster [DBG] pgmap v607: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-19T21:10:16.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:15.389327+0000 mon.a (mon.0) 2526 : cluster [DBG] osdmap e419: 8 total, 8 up, 7 in 2024-02-19T21:10:16.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:15.515936+0000 osd.6 (osd.6) 159 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:10:16.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:15.518550+0000 osd.6 (osd.6) 160 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:10:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:14.627873+0000 osd.1 (osd.1) 195 : cluster [DBG] 16.2 scrub starts 2024-02-19T21:10:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:14.631325+0000 osd.1 (osd.1) 196 : cluster [DBG] 16.2 scrub ok 2024-02-19T21:10:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:15.158397+0000 osd.5 (osd.5) 172 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:10:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:15.161393+0000 osd.5 (osd.5) 173 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:10:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:15.213485+0000 mgr.y (mgr.14550) 380 : cluster [DBG] pgmap v607: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-19T21:10:16.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:15.389327+0000 mon.a (mon.0) 2526 : cluster [DBG] osdmap e419: 8 total, 8 up, 7 in 2024-02-19T21:10:16.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:15.515936+0000 osd.6 (osd.6) 159 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:10:16.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:15.518550+0000 osd.6 (osd.6) 160 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:10:16.651 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:16.676 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:16.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:16.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:16.702 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 3] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:10:16.702 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:10:16.702 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-19T21:10:16.702 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.750587 2024-02-19T21:10:16.703 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd primary-affinity 6 0.750587277064386 2024-02-19T21:10:16.780 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:10:16.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:14.627873+0000 osd.1 (osd.1) 195 : cluster [DBG] 16.2 scrub starts 2024-02-19T21:10:16.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:14.631325+0000 osd.1 (osd.1) 196 : cluster [DBG] 16.2 scrub ok 2024-02-19T21:10:16.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:15.158397+0000 osd.5 (osd.5) 172 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:10:16.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:15.161393+0000 osd.5 (osd.5) 173 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:10:16.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:15.213485+0000 mgr.y (mgr.14550) 380 : cluster [DBG] pgmap v607: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-02-19T21:10:16.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:15.389327+0000 mon.a (mon.0) 2526 : cluster [DBG] osdmap e419: 8 total, 8 up, 7 in 2024-02-19T21:10:16.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:15.515936+0000 osd.6 (osd.6) 159 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:10:16.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:15.518550+0000 osd.6 (osd.6) 160 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:10:16.812 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:16.834 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:10:16.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:16.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:10:16.938 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:10:16.968 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:17.172 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:10:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:17.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:10:17.277 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:17.308 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:17.329 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:17.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:17.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:17.433 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:10:17.463 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:17.483 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:10:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:17.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:10:17.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:16.528544+0000 osd.6 (osd.6) 161 : cluster [DBG] 16.5 scrub starts 2024-02-19T21:10:17.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:16.529619+0000 osd.6 (osd.6) 162 : cluster [DBG] 16.5 scrub ok 2024-02-19T21:10:17.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:16.843459+0000 osd.7 (osd.7) 189 : cluster [DBG] 61.1e scrub starts 2024-02-19T21:10:17.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:16.846923+0000 osd.7 (osd.7) 190 : cluster [DBG] 61.1e scrub ok 2024-02-19T21:10:17.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:16.528544+0000 osd.6 (osd.6) 161 : cluster [DBG] 16.5 scrub starts 2024-02-19T21:10:17.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:16.529619+0000 osd.6 (osd.6) 162 : cluster [DBG] 16.5 scrub ok 2024-02-19T21:10:17.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:16.843459+0000 osd.7 (osd.7) 189 : cluster [DBG] 61.1e scrub starts 2024-02-19T21:10:17.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:16.846923+0000 osd.7 (osd.7) 190 : cluster [DBG] 61.1e scrub ok 2024-02-19T21:10:17.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:16.528544+0000 osd.6 (osd.6) 161 : cluster [DBG] 16.5 scrub starts 2024-02-19T21:10:17.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:16.529619+0000 osd.6 (osd.6) 162 : cluster [DBG] 16.5 scrub ok 2024-02-19T21:10:17.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:16.843459+0000 osd.7 (osd.7) 189 : cluster [DBG] 61.1e scrub starts 2024-02-19T21:10:17.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:16.846923+0000 osd.7 (osd.7) 190 : cluster [DBG] 61.1e scrub ok 2024-02-19T21:10:17.586 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:17.617 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:17.635 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:17.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:17.739 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:17.767 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:17.787 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:17.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:17.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:17.891 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:17.920 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:17.939 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:17.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:17.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:17.951 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:10:17.951 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:10:18.044 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:18.068 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:10:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:10:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:10:18.071 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:18.202 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:18.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:18.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:18.306 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:18.332 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:18.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:17.081167+0000 osd.2 (osd.2) 164 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:10:18.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:17.082099+0000 osd.2 (osd.2) 165 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:10:18.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:17.213843+0000 mgr.y (mgr.14550) 381 : cluster [DBG] pgmap v609: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:10:18.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:17.081167+0000 osd.2 (osd.2) 164 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:10:18.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:17.082099+0000 osd.2 (osd.2) 165 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:10:18.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:17.213843+0000 mgr.y (mgr.14550) 381 : cluster [DBG] pgmap v609: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:10:18.758 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:18.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:17.081167+0000 osd.2 (osd.2) 164 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:10:18.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:17.082099+0000 osd.2 (osd.2) 165 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:10:18.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:17.213843+0000 mgr.y (mgr.14550) 381 : cluster [DBG] pgmap v609: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:10:19.126 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:19.149 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-19T21:10:19.381 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:19.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:19.486 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:19.515 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:19.642 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:18.510676+0000 osd.6 (osd.6) 163 : cluster [DBG] 61.e scrub starts 2024-02-19T21:10:19.643 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:18.512559+0000 osd.6 (osd.6) 164 : cluster [DBG] 61.e scrub ok 2024-02-19T21:10:19.643 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:19.265921+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e420: 8 total, 8 up, 7 in 2024-02-19T21:10:19.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:18.510676+0000 osd.6 (osd.6) 163 : cluster [DBG] 61.e scrub starts 2024-02-19T21:10:19.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:18.512559+0000 osd.6 (osd.6) 164 : cluster [DBG] 61.e scrub ok 2024-02-19T21:10:19.643 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:19.265921+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e420: 8 total, 8 up, 7 in 2024-02-19T21:10:19.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:18.510676+0000 osd.6 (osd.6) 163 : cluster [DBG] 61.e scrub starts 2024-02-19T21:10:19.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:18.512559+0000 osd.6 (osd.6) 164 : cluster [DBG] 61.e scrub ok 2024-02-19T21:10:19.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:19.265921+0000 mon.a (mon.0) 2527 : cluster [DBG] osdmap e420: 8 total, 8 up, 7 in 2024-02-19T21:10:20.665 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:20.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:18.734117+0000 osd.1 (osd.1) 197 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:10:20.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:18.737172+0000 osd.1 (osd.1) 198 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:10:20.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:19.104848+0000 osd.2 (osd.2) 166 : cluster [DBG] 4.1c deep-scrub starts 2024-02-19T21:10:20.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:19.109234+0000 osd.2 (osd.2) 167 : cluster [DBG] 4.1c deep-scrub ok 2024-02-19T21:10:20.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:19.214212+0000 mgr.y (mgr.14550) 382 : cluster [DBG] pgmap v610: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 652 B/s rd, 0 op/s 2024-02-19T21:10:20.667 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:18.734117+0000 osd.1 (osd.1) 197 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:10:20.667 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:18.737172+0000 osd.1 (osd.1) 198 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:10:20.667 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:19.104848+0000 osd.2 (osd.2) 166 : cluster [DBG] 4.1c deep-scrub starts 2024-02-19T21:10:20.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:19.109234+0000 osd.2 (osd.2) 167 : cluster [DBG] 4.1c deep-scrub ok 2024-02-19T21:10:20.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:19.214212+0000 mgr.y (mgr.14550) 382 : cluster [DBG] pgmap v610: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 652 B/s rd, 0 op/s 2024-02-19T21:10:20.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:20.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:20.770 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:20.800 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:20.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:10:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:10:20.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:18.734117+0000 osd.1 (osd.1) 197 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:10:20.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:18.737172+0000 osd.1 (osd.1) 198 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:10:20.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:19.104848+0000 osd.2 (osd.2) 166 : cluster [DBG] 4.1c deep-scrub starts 2024-02-19T21:10:20.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:19.109234+0000 osd.2 (osd.2) 167 : cluster [DBG] 4.1c deep-scrub ok 2024-02-19T21:10:20.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:19.214212+0000 mgr.y (mgr.14550) 382 : cluster [DBG] pgmap v610: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 652 B/s rd, 0 op/s 2024-02-19T21:10:20.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.888+0000 7f3150905700 1 -- 172.21.15.163:0/1902279576 >> v1:172.21.15.163:6790/0 conn(0x7f314c1007c0 legacy=0x7f314c102bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:20.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.888+0000 7f3150905700 1 -- 172.21.15.163:0/1902279576 shutdown_connections 2024-02-19T21:10:20.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.888+0000 7f3150905700 1 -- 172.21.15.163:0/1902279576 >> 172.21.15.163:0/1902279576 conn(0x7f314c06cc00 msgr2=0x7f314c0f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:20.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.888+0000 7f3150905700 1 -- 172.21.15.163:0/1902279576 shutdown_connections 2024-02-19T21:10:20.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.888+0000 7f3150905700 1 -- 172.21.15.163:0/1902279576 wait complete. 2024-02-19T21:10:20.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.888+0000 7f3150905700 1 librados: starting msgr at 2024-02-19T21:10:20.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.888+0000 7f3150905700 1 librados: starting objecter 2024-02-19T21:10:20.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f3150905700 1 Processor -- start 2024-02-19T21:10:20.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f3150905700 1 -- start start 2024-02-19T21:10:20.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f3150905700 1 librados: setting wanted keys 2024-02-19T21:10:20.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f3150905700 1 librados: calling monclient init 2024-02-19T21:10:20.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f3150905700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f314c117350 con 0x7f314c0fd9d0 2024-02-19T21:10:20.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f3150905700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f314c118530 con 0x7f314c104330 2024-02-19T21:10:20.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f3150905700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f314c119730 con 0x7f314c1007c0 2024-02-19T21:10:20.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f314a59c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f314c0fd9d0 0x7f314c0f9a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43908/0 (socket says 172.21.15.163:43908) 2024-02-19T21:10:20.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f314a59c700 1 -- 172.21.15.163:0/2470646849 learned_addr learned my addr 172.21.15.163:0/2470646849 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:20.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f313b7fe700 10 client.?.objecter ms_handle_connect 0x7f314c0fd9d0 2024-02-19T21:10:20.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f313b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:20.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f313b7fe700 10 client.?.objecter ms_handle_connect 0x7f314c104330 2024-02-19T21:10:20.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f313b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:20.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f313b7fe700 10 client.?.objecter ms_handle_connect 0x7f314c1007c0 2024-02-19T21:10:20.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f313b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:20.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848111375 0 0) 0x7f314c117350 con 0x7f314c0fd9d0 2024-02-19T21:10:20.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31340038e0 con 0x7f314c0fd9d0 2024-02-19T21:10:20.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970926713 0 0) 0x7f314c118530 con 0x7f314c104330 2024-02-19T21:10:20.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f314c117350 con 0x7f314c104330 2024-02-19T21:10:20.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129969119 0 0) 0x7f314c119730 con 0x7f314c1007c0 2024-02-19T21:10:20.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f314c118530 con 0x7f314c1007c0 2024-02-19T21:10:20.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1132389453 0 0) 0x7f31340038e0 con 0x7f314c0fd9d0 2024-02-19T21:10:20.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f314c119730 con 0x7f314c0fd9d0 2024-02-19T21:10:20.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.889+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f313c0019f0 con 0x7f314c0fd9d0 2024-02-19T21:10:20.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4107479213 0 0) 0x7f314c118530 con 0x7f314c1007c0 2024-02-19T21:10:20.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31340038e0 con 0x7f314c1007c0 2024-02-19T21:10:20.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 164991216 0 0) 0x7f314c117350 con 0x7f314c104330 2024-02-19T21:10:20.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f314c118530 con 0x7f314c104330 2024-02-19T21:10:20.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3140002d90 con 0x7f314c1007c0 2024-02-19T21:10:20.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3144001b00 con 0x7f314c104330 2024-02-19T21:10:20.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1350303402 0 0) 0x7f314c119730 con 0x7f314c0fd9d0 2024-02-19T21:10:20.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 >> v1:172.21.15.163:6790/0 conn(0x7f314c1007c0 legacy=0x7f314c0fa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:20.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 >> v1:172.21.15.193:6789/0 conn(0x7f314c104330 legacy=0x7f314c115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:20.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314c11a930 con 0x7f314c0fd9d0 2024-02-19T21:10:20.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f314c118720 con 0x7f314c0fd9d0 2024-02-19T21:10:20.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f3150905700 10 client.23127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:20.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f314c118b90 con 0x7f314c0fd9d0 2024-02-19T21:10:20.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f3150905700 1 librados: init done 2024-02-19T21:10:20.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f313c002410 con 0x7f314c0fd9d0 2024-02-19T21:10:20.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f313c005ab0 con 0x7f314c0fd9d0 2024-02-19T21:10:20.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.890+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f312c004f00 con 0x7f314c0fd9d0 2024-02-19T21:10:20.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.891+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f313c001f70 con 0x7f314c0fd9d0 2024-02-19T21:10:20.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(420..420 src has 43..420) v4 ==== 7946+0+0 (unknown 358967885 0 0) 0x7f313c078b60 con 0x7f314c0fd9d0 2024-02-19T21:10:20.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 osd_map(420..420 src has 43..420) v4 2024-02-19T21:10:20.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f313b7fe700 3 client.23127.objecter handle_osd_map got epochs [420,420] > 0 2024-02-19T21:10:20.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f313b7fe700 3 client.23127.objecter handle_osd_map decoding full epoch 420 2024-02-19T21:10:20.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f313b7fe700 20 client.23127.objecter dump_active .. 0 homeless 2024-02-19T21:10:20.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v420) v1 ==== 54+0+16 (unknown 2811898740 0 2420000192) 0x7f313c079060 con 0x7f314c0fd9d0 2024-02-19T21:10:20.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f313b7fe700 10 client.23127.objecter ms_handle_connect 0x7f3134063ce0 2024-02-19T21:10:20.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f3150905700 10 client.23127.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:20.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 1 2024-02-19T21:10:20.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c0051a0 osd=0 3 2024-02-19T21:10:20.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c0051a0 osd=0 3 2024-02-19T21:10:20.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c0051a0 osd=0 3 2024-02-19T21:10:20.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 1 2024-02-19T21:10:20.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c0051a0 osd=0 3 2024-02-19T21:10:20.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 0 1 2024-02-19T21:10:20.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c0051a0 osd=0 4 2024-02-19T21:10:20.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f3150905700 10 client.23127.objecter _send_command 1 2024-02-19T21:10:20.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f312c004950 con 0x7f312c004530 2024-02-19T21:10:20.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.892+0000 7f313b7fe700 10 client.23127.objecter ms_handle_connect 0x7f312c004530 2024-02-19T21:10:20.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.893+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f312c004950 con 0x7f312c004530 2024-02-19T21:10:20.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.893+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:10:20.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.893+0000 7f313b7fe700 10 client.23127.objecter _finish_command 1 = system:0 2024-02-19T21:10:20.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.893+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c0051a0 osd=0 4 2024-02-19T21:10:20.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.893+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 0 1 2024-02-19T21:10:20.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.908+0000 7f3150905700 10 client.23127.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:20.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.908+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 2 2024-02-19T21:10:20.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.908+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c0051a0 osd=0 3 2024-02-19T21:10:20.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.908+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c0051a0 osd=0 3 2024-02-19T21:10:20.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.908+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c0051a0 osd=0 3 2024-02-19T21:10:20.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.908+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 2 2024-02-19T21:10:20.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.908+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c0051a0 osd=0 3 2024-02-19T21:10:20.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.908+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 0 2 2024-02-19T21:10:20.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.908+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c0051a0 osd=0 4 2024-02-19T21:10:20.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.908+0000 7f3150905700 10 client.23127.objecter _send_command 2 2024-02-19T21:10:20.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.908+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f312c004d30 con 0x7f312c004530 2024-02-19T21:10:20.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f312c004d30 con 0x7f312c004530 2024-02-19T21:10:20.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:20.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f313b7fe700 10 client.23127.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:20.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c0051a0 osd=0 4 2024-02-19T21:10:20.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 0 2 2024-02-19T21:10:20.910 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:10:20.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f3150905700 10 client.23127.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:20.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 3 2024-02-19T21:10:20.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c004270 osd=1 3 2024-02-19T21:10:20.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c004270 osd=1 3 2024-02-19T21:10:20.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c004270 osd=1 3 2024-02-19T21:10:20.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 3 2024-02-19T21:10:20.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c004270 osd=1 3 2024-02-19T21:10:20.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 1 3 2024-02-19T21:10:20.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c004270 osd=1 4 2024-02-19T21:10:20.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f3150905700 10 client.23127.objecter _send_command 3 2024-02-19T21:10:20.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.909+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f312c00bf60 con 0x7f312c003b40 2024-02-19T21:10:20.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.910+0000 7f313b7fe700 10 client.23127.objecter ms_handle_connect 0x7f312c003b40 2024-02-19T21:10:20.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.910+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f312c00bf60 con 0x7f312c003b40 2024-02-19T21:10:20.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.910+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:10:20.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.910+0000 7f313b7fe700 10 client.23127.objecter _finish_command 3 = system:0 2024-02-19T21:10:20.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.910+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c004270 osd=1 4 2024-02-19T21:10:20.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.910+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 1 3 2024-02-19T21:10:20.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.925+0000 7f3150905700 10 client.23127.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:20.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.925+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 4 2024-02-19T21:10:20.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.925+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c004270 osd=1 3 2024-02-19T21:10:20.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.925+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c004270 osd=1 3 2024-02-19T21:10:20.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.925+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c004270 osd=1 3 2024-02-19T21:10:20.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.925+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 4 2024-02-19T21:10:20.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.925+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c004270 osd=1 3 2024-02-19T21:10:20.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.925+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 1 4 2024-02-19T21:10:20.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.925+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c004270 osd=1 4 2024-02-19T21:10:20.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.925+0000 7f3150905700 10 client.23127.objecter _send_command 4 2024-02-19T21:10:20.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.925+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f312c00d250 con 0x7f312c003b40 2024-02-19T21:10:20.924 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:20.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f312c00d250 con 0x7f312c003b40 2024-02-19T21:10:20.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:20.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f313b7fe700 10 client.23127.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:20.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c004270 osd=1 4 2024-02-19T21:10:20.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 1 4 2024-02-19T21:10:20.926 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:10:20.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f3150905700 10 client.23127.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:20.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 5 2024-02-19T21:10:20.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c0033a0 osd=2 3 2024-02-19T21:10:20.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c0033a0 osd=2 3 2024-02-19T21:10:20.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c0033a0 osd=2 3 2024-02-19T21:10:20.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 5 2024-02-19T21:10:20.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c0033a0 osd=2 3 2024-02-19T21:10:20.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 2 5 2024-02-19T21:10:20.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c0033a0 osd=2 4 2024-02-19T21:10:20.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f3150905700 10 client.23127.objecter _send_command 5 2024-02-19T21:10:20.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f312c010cd0 con 0x7f312c002070 2024-02-19T21:10:20.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.927+0000 7f313b7fe700 10 client.23127.objecter ms_handle_connect 0x7f312c002070 2024-02-19T21:10:20.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.928+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f312c010cd0 con 0x7f312c002070 2024-02-19T21:10:20.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.928+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:10:20.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.928+0000 7f313b7fe700 10 client.23127.objecter _finish_command 5 = system:0 2024-02-19T21:10:20.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.928+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c0033a0 osd=2 4 2024-02-19T21:10:20.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.928+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 2 5 2024-02-19T21:10:20.928 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:20.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:20.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.942+0000 7f3150905700 10 client.23127.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:20.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.943+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 6 2024-02-19T21:10:20.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.943+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c0033a0 osd=2 3 2024-02-19T21:10:20.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.943+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c0033a0 osd=2 3 2024-02-19T21:10:20.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.943+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c0033a0 osd=2 3 2024-02-19T21:10:20.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.943+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 6 2024-02-19T21:10:20.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.943+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c0033a0 osd=2 3 2024-02-19T21:10:20.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.943+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 2 6 2024-02-19T21:10:20.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.943+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c0033a0 osd=2 4 2024-02-19T21:10:20.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.943+0000 7f3150905700 10 client.23127.objecter _send_command 6 2024-02-19T21:10:20.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.943+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f312c001630 con 0x7f312c002070 2024-02-19T21:10:20.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f312c001630 con 0x7f312c002070 2024-02-19T21:10:20.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:20.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f313b7fe700 10 client.23127.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:20.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c0033a0 osd=2 4 2024-02-19T21:10:20.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 2 6 2024-02-19T21:10:20.943 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:10:20.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f3150905700 10 client.23127.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:20.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 7 2024-02-19T21:10:20.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c00c8a0 osd=3 3 2024-02-19T21:10:20.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c00c8a0 osd=3 3 2024-02-19T21:10:20.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c00c8a0 osd=3 3 2024-02-19T21:10:20.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 7 2024-02-19T21:10:20.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c00c8a0 osd=3 3 2024-02-19T21:10:20.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 3 7 2024-02-19T21:10:20.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c00c8a0 osd=3 4 2024-02-19T21:10:20.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f3150905700 10 client.23127.objecter _send_command 7 2024-02-19T21:10:20.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.944+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f312c00d050 con 0x7f312c012480 2024-02-19T21:10:20.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.945+0000 7f313b7fe700 10 client.23127.objecter ms_handle_connect 0x7f312c012480 2024-02-19T21:10:20.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.945+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f312c00d050 con 0x7f312c012480 2024-02-19T21:10:20.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.945+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:10:20.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.945+0000 7f313b7fe700 10 client.23127.objecter _finish_command 7 = system:0 2024-02-19T21:10:20.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.945+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c00c8a0 osd=3 4 2024-02-19T21:10:20.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.945+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 3 7 2024-02-19T21:10:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.960+0000 7f3150905700 10 client.23127.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.960+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 8 2024-02-19T21:10:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.960+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c00c8a0 osd=3 3 2024-02-19T21:10:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.960+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c00c8a0 osd=3 3 2024-02-19T21:10:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.960+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c00c8a0 osd=3 3 2024-02-19T21:10:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.960+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 8 2024-02-19T21:10:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.960+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c00c8a0 osd=3 3 2024-02-19T21:10:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.960+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 3 8 2024-02-19T21:10:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.960+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c00c8a0 osd=3 4 2024-02-19T21:10:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.960+0000 7f3150905700 10 client.23127.objecter _send_command 8 2024-02-19T21:10:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.960+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f312c012110 con 0x7f312c012480 2024-02-19T21:10:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f312c012110 con 0x7f312c012480 2024-02-19T21:10:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:20.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f313b7fe700 10 client.23127.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:20.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c00c8a0 osd=3 4 2024-02-19T21:10:20.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 3 8 2024-02-19T21:10:20.961 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:10:20.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f3150905700 10 client.23127.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:20.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 9 2024-02-19T21:10:20.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c012130 osd=4 3 2024-02-19T21:10:20.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c012130 osd=4 3 2024-02-19T21:10:20.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c012130 osd=4 3 2024-02-19T21:10:20.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 9 2024-02-19T21:10:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c012130 osd=4 3 2024-02-19T21:10:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 4 9 2024-02-19T21:10:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c012130 osd=4 4 2024-02-19T21:10:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f3150905700 10 client.23127.objecter _send_command 9 2024-02-19T21:10:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.962+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f312c01e720 con 0x7f312c01ac10 2024-02-19T21:10:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.963+0000 7f313b7fe700 10 client.23127.objecter ms_handle_connect 0x7f312c01ac10 2024-02-19T21:10:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.964+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f312c01e720 con 0x7f312c01ac10 2024-02-19T21:10:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.964+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:10:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.964+0000 7f313b7fe700 10 client.23127.objecter _finish_command 9 = system:0 2024-02-19T21:10:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.964+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c012130 osd=4 4 2024-02-19T21:10:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.964+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 4 9 2024-02-19T21:10:20.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.979+0000 7f3150905700 10 client.23127.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:20.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.979+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 10 2024-02-19T21:10:20.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.979+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c012130 osd=4 3 2024-02-19T21:10:20.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.979+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c012130 osd=4 3 2024-02-19T21:10:20.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.979+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c012130 osd=4 3 2024-02-19T21:10:20.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.979+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 10 2024-02-19T21:10:20.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.979+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c012130 osd=4 3 2024-02-19T21:10:20.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.979+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 4 10 2024-02-19T21:10:20.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.979+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c012130 osd=4 4 2024-02-19T21:10:20.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.979+0000 7f3150905700 10 client.23127.objecter _send_command 10 2024-02-19T21:10:20.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.979+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f312c01e720 con 0x7f312c01ac10 2024-02-19T21:10:20.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.981+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f312c01e720 con 0x7f312c01ac10 2024-02-19T21:10:20.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.981+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:20.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.981+0000 7f313b7fe700 10 client.23127.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:20.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.981+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c012130 osd=4 4 2024-02-19T21:10:20.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.981+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 4 10 2024-02-19T21:10:20.980 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:10:20.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.982+0000 7f3150905700 10 client.23127.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:20.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.982+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 11 2024-02-19T21:10:20.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.982+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c01e720 osd=5 3 2024-02-19T21:10:20.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.982+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c01e720 osd=5 3 2024-02-19T21:10:20.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.982+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c01e720 osd=5 3 2024-02-19T21:10:20.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.982+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 11 2024-02-19T21:10:20.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.982+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c01e720 osd=5 3 2024-02-19T21:10:20.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.982+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 5 11 2024-02-19T21:10:20.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.982+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c01e720 osd=5 4 2024-02-19T21:10:20.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.982+0000 7f3150905700 10 client.23127.objecter _send_command 11 2024-02-19T21:10:20.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.982+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f312c022a30 con 0x7f312c01eda0 2024-02-19T21:10:20.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.983+0000 7f313b7fe700 10 client.23127.objecter ms_handle_connect 0x7f312c01eda0 2024-02-19T21:10:20.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.983+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f312c022a30 con 0x7f312c01eda0 2024-02-19T21:10:20.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.983+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:10:20.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.983+0000 7f313b7fe700 10 client.23127.objecter _finish_command 11 = system:0 2024-02-19T21:10:20.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.983+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c01e720 osd=5 4 2024-02-19T21:10:20.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.983+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 5 11 2024-02-19T21:10:20.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.998+0000 7f3150905700 10 client.23127.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:20.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.998+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 12 2024-02-19T21:10:20.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.998+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c01e720 osd=5 3 2024-02-19T21:10:20.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.998+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c01e720 osd=5 3 2024-02-19T21:10:20.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.998+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c01e720 osd=5 3 2024-02-19T21:10:20.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.998+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 12 2024-02-19T21:10:20.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.998+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c01e720 osd=5 3 2024-02-19T21:10:20.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.998+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 5 12 2024-02-19T21:10:20.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.998+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c01e720 osd=5 4 2024-02-19T21:10:20.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.998+0000 7f3150905700 10 client.23127.objecter _send_command 12 2024-02-19T21:10:20.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:20.998+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f312c022a30 con 0x7f312c01eda0 2024-02-19T21:10:20.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f312c022a30 con 0x7f312c01eda0 2024-02-19T21:10:20.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:20.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f313b7fe700 10 client.23127.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:20.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c01e720 osd=5 4 2024-02-19T21:10:20.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 5 12 2024-02-19T21:10:20.998 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:10:21.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f3150905700 10 client.23127.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:21.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 13 2024-02-19T21:10:21.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c013640 osd=6 3 2024-02-19T21:10:21.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c013640 osd=6 3 2024-02-19T21:10:21.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c013640 osd=6 3 2024-02-19T21:10:21.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 13 2024-02-19T21:10:21.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c013640 osd=6 3 2024-02-19T21:10:21.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 6 13 2024-02-19T21:10:21.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c013640 osd=6 4 2024-02-19T21:10:21.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f3150905700 10 client.23127.objecter _send_command 13 2024-02-19T21:10:21.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f312c014140 con 0x7f312c013cc0 2024-02-19T21:10:21.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.000+0000 7f313b7fe700 10 client.23127.objecter ms_handle_connect 0x7f312c013cc0 2024-02-19T21:10:21.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.001+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f312c014140 con 0x7f312c013cc0 2024-02-19T21:10:21.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.001+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:10:21.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.001+0000 7f313b7fe700 10 client.23127.objecter _finish_command 13 = system:0 2024-02-19T21:10:21.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.001+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c013640 osd=6 4 2024-02-19T21:10:21.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.001+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 6 13 2024-02-19T21:10:21.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.015+0000 7f3150905700 10 client.23127.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:21.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.015+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 14 2024-02-19T21:10:21.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.015+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c013640 osd=6 3 2024-02-19T21:10:21.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.015+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c013640 osd=6 3 2024-02-19T21:10:21.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.015+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c013640 osd=6 3 2024-02-19T21:10:21.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.015+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 14 2024-02-19T21:10:21.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.015+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c013640 osd=6 3 2024-02-19T21:10:21.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.015+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 6 14 2024-02-19T21:10:21.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.015+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c013640 osd=6 4 2024-02-19T21:10:21.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.015+0000 7f3150905700 10 client.23127.objecter _send_command 14 2024-02-19T21:10:21.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.015+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f312c014140 con 0x7f312c013cc0 2024-02-19T21:10:21.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.017+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f312c014140 con 0x7f312c013cc0 2024-02-19T21:10:21.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.017+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:21.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.017+0000 7f313b7fe700 10 client.23127.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:21.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.017+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c013640 osd=6 4 2024-02-19T21:10:21.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.017+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 6 14 2024-02-19T21:10:21.017 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:10:21.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.018+0000 7f3150905700 10 client.23127.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:21.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.018+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 15 2024-02-19T21:10:21.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.018+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c014140 osd=7 3 2024-02-19T21:10:21.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.018+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c014140 osd=7 3 2024-02-19T21:10:21.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.018+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c014140 osd=7 3 2024-02-19T21:10:21.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.018+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 15 2024-02-19T21:10:21.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.018+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c014140 osd=7 3 2024-02-19T21:10:21.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.018+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 7 15 2024-02-19T21:10:21.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.018+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c014140 osd=7 4 2024-02-19T21:10:21.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.018+0000 7f3150905700 10 client.23127.objecter _send_command 15 2024-02-19T21:10:21.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.018+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f312c02b460 con 0x7f312c014380 2024-02-19T21:10:21.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.018+0000 7f313b7fe700 10 client.23127.objecter ms_handle_connect 0x7f312c014380 2024-02-19T21:10:21.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.019+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f312c02b460 con 0x7f312c014380 2024-02-19T21:10:21.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.019+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:10:21.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.019+0000 7f313b7fe700 10 client.23127.objecter _finish_command 15 = system:0 2024-02-19T21:10:21.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.019+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c014140 osd=7 4 2024-02-19T21:10:21.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.019+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 7 15 2024-02-19T21:10:21.030 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:21.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.033+0000 7f3150905700 10 client.23127.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:21.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.033+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign -1 16 2024-02-19T21:10:21.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.033+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c014140 osd=7 3 2024-02-19T21:10:21.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.033+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c014140 osd=7 3 2024-02-19T21:10:21.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.033+0000 7f3150905700 20 client.23127.objecter _get_session s=0x7f312c014140 osd=7 3 2024-02-19T21:10:21.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.033+0000 7f3150905700 15 client.23127.objecter _session_command_op_remove -1 16 2024-02-19T21:10:21.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.033+0000 7f3150905700 20 client.23127.objecter get_session s=0x7f312c014140 osd=7 3 2024-02-19T21:10:21.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.033+0000 7f3150905700 15 client.23127.objecter _session_command_op_assign 7 16 2024-02-19T21:10:21.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.033+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c014140 osd=7 4 2024-02-19T21:10:21.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.033+0000 7f3150905700 10 client.23127.objecter _send_command 16 2024-02-19T21:10:21.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.033+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f312c02b460 con 0x7f312c014380 2024-02-19T21:10:21.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f313b7fe700 1 -- 172.21.15.163:0/2470646849 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f312c02b460 con 0x7f312c014380 2024-02-19T21:10:21.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f313b7fe700 10 client.23127.objecter ms_dispatch 0x7f314c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:21.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f313b7fe700 10 client.23127.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:21.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f313b7fe700 20 client.23127.objecter put_session s=0x7f312c014140 osd=7 4 2024-02-19T21:10:21.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f313b7fe700 15 client.23127.objecter _session_command_op_remove 7 16 2024-02-19T21:10:21.036 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:10:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 10 librados: watch_flush enter 2024-02-19T21:10:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 10 librados: watch_flush exit 2024-02-19T21:10:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 10 client.23127.objecter close_session for osd.0 2024-02-19T21:10:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 >> v1:172.21.15.163:6801/1559418115 conn(0x7f312c004530 legacy=0x7f312c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c0051a0 osd=0 1 2024-02-19T21:10:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 10 client.23127.objecter close_session for osd.1 2024-02-19T21:10:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 >> v1:172.21.15.163:6805/3125047432 conn(0x7f312c003b40 legacy=0x7f312c00a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c004270 osd=1 1 2024-02-19T21:10:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 10 client.23127.objecter close_session for osd.2 2024-02-19T21:10:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 >> v1:172.21.15.163:6809/1563158487 conn(0x7f312c002070 legacy=0x7f312c00f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c0033a0 osd=2 1 2024-02-19T21:10:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 10 client.23127.objecter close_session for osd.3 2024-02-19T21:10:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 >> v1:172.21.15.163:6813/2112279769 conn(0x7f312c012480 legacy=0x7f312c0128e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c00c8a0 osd=3 1 2024-02-19T21:10:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 10 client.23127.objecter close_session for osd.4 2024-02-19T21:10:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 >> v1:172.21.15.193:6800/3129716965 conn(0x7f312c01ac10 legacy=0x7f312c01d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c012130 osd=4 1 2024-02-19T21:10:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 10 client.23127.objecter close_session for osd.5 2024-02-19T21:10:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 >> v1:172.21.15.193:6804/805161991 conn(0x7f312c01eda0 legacy=0x7f312c0211b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c01e720 osd=5 1 2024-02-19T21:10:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 10 client.23127.objecter close_session for osd.6 2024-02-19T21:10:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 >> v1:172.21.15.193:6808/787604615 conn(0x7f312c013cc0 legacy=0x7f312c022bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c013640 osd=6 1 2024-02-19T21:10:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 10 client.23127.objecter close_session for osd.7 2024-02-19T21:10:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 >> v1:172.21.15.193:6812/157927432 conn(0x7f312c014380 legacy=0x7f312c024970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 20 client.23127.objecter put_session s=0x7f312c014140 osd=7 1 2024-02-19T21:10:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 20 client.23127.objecter shutdown clearing up homeless session... 2024-02-19T21:10:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 10 client.23127.objecter successfully canceled tick 2024-02-19T21:10:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f314ad9d700 1 -- 172.21.15.163:0/2470646849 reap_dead start 2024-02-19T21:10:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f314ad9d700 1 -- 172.21.15.163:0/2470646849 reap_dead start 2024-02-19T21:10:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f314ad9d700 1 -- 172.21.15.163:0/2470646849 reap_dead start 2024-02-19T21:10:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f314ad9d700 1 -- 172.21.15.163:0/2470646849 reap_dead start 2024-02-19T21:10:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f314ad9d700 1 -- 172.21.15.163:0/2470646849 reap_dead start 2024-02-19T21:10:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 >> v1:172.21.15.163:6800/1552514681 conn(0x7f3134063ce0 legacy=0x7f31340661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 >> v1:172.21.15.163:6789/0 conn(0x7f314c0fd9d0 legacy=0x7f314c0f9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 shutdown_connections 2024-02-19T21:10:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 >> 172.21.15.163:0/2470646849 conn(0x7f314c06cc00 msgr2=0x7f314c1077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 shutdown_connections 2024-02-19T21:10:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 1 -- 172.21.15.163:0/2470646849 wait complete. 2024-02-19T21:10:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:21.036+0000 7f3150905700 1 librados: shutdown 2024-02-19T21:10:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:10:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:10:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:10:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:10:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:10:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:10:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:10:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:10:21.066 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:21.088 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:21.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:21.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:21.192 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:21.220 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:21.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:19.716669+0000 osd.1 (osd.1) 199 : cluster [DBG] 4.9 deep-scrub starts 2024-02-19T21:10:21.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:19.722299+0000 osd.1 (osd.1) 200 : cluster [DBG] 4.9 deep-scrub ok 2024-02-19T21:10:21.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:20.201931+0000 osd.5 (osd.5) 174 : cluster [DBG] 16.11 deep-scrub starts 2024-02-19T21:10:21.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:20.204876+0000 osd.5 (osd.5) 175 : cluster [DBG] 16.11 deep-scrub ok 2024-02-19T21:10:21.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:20.402617+0000 mgr.y (mgr.14550) 383 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:21.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:20.473835+0000 osd.6 (osd.6) 165 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:10:21.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:20.474840+0000 osd.6 (osd.6) 166 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:10:21.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:20.892535+0000 mon.a (mon.0) 2528 : audit [DBG] from='client.? v1:172.21.15.163:0/2470646849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:21.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:19.716669+0000 osd.1 (osd.1) 199 : cluster [DBG] 4.9 deep-scrub starts 2024-02-19T21:10:21.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:19.722299+0000 osd.1 (osd.1) 200 : cluster [DBG] 4.9 deep-scrub ok 2024-02-19T21:10:21.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:20.201931+0000 osd.5 (osd.5) 174 : cluster [DBG] 16.11 deep-scrub starts 2024-02-19T21:10:21.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:20.204876+0000 osd.5 (osd.5) 175 : cluster [DBG] 16.11 deep-scrub ok 2024-02-19T21:10:21.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:20.402617+0000 mgr.y (mgr.14550) 383 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:21.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:20.473835+0000 osd.6 (osd.6) 165 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:10:21.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:20.474840+0000 osd.6 (osd.6) 166 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:10:21.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:20.892535+0000 mon.a (mon.0) 2528 : audit [DBG] from='client.? v1:172.21.15.163:0/2470646849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:21.669 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:21.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:21.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:21.773 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:21.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:19.716669+0000 osd.1 (osd.1) 199 : cluster [DBG] 4.9 deep-scrub starts 2024-02-19T21:10:21.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:19.722299+0000 osd.1 (osd.1) 200 : cluster [DBG] 4.9 deep-scrub ok 2024-02-19T21:10:21.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:20.201931+0000 osd.5 (osd.5) 174 : cluster [DBG] 16.11 deep-scrub starts 2024-02-19T21:10:21.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:20.204876+0000 osd.5 (osd.5) 175 : cluster [DBG] 16.11 deep-scrub ok 2024-02-19T21:10:21.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:20.402617+0000 mgr.y (mgr.14550) 383 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:21.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:20.473835+0000 osd.6 (osd.6) 165 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:10:21.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:20.474840+0000 osd.6 (osd.6) 166 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:10:21.804 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:20.892535+0000 mon.a (mon.0) 2528 : audit [DBG] from='client.? v1:172.21.15.163:0/2470646849' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:21.805 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:21.827 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:21.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:21.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:21.931 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:21.962 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:21.983 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:21.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:21.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:22.087 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:22.116 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:22.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.707+0000 7f6cc9686700 1 -- 172.21.15.163:0/1495074375 >> v1:172.21.15.163:6789/0 conn(0x7f6cc40fd910 legacy=0x7f6cc40fdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:22.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.707+0000 7f6cc9686700 1 -- 172.21.15.163:0/1495074375 shutdown_connections 2024-02-19T21:10:22.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.707+0000 7f6cc9686700 1 -- 172.21.15.163:0/1495074375 >> 172.21.15.163:0/1495074375 conn(0x7f6cc40f3530 msgr2=0x7f6cc40f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:22.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.707+0000 7f6cc9686700 1 -- 172.21.15.163:0/1495074375 shutdown_connections 2024-02-19T21:10:22.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.707+0000 7f6cc9686700 1 -- 172.21.15.163:0/1495074375 wait complete. 2024-02-19T21:10:22.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.708+0000 7f6cc9686700 1 librados: starting msgr at 2024-02-19T21:10:22.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.708+0000 7f6cc9686700 1 librados: starting objecter 2024-02-19T21:10:22.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.708+0000 7f6cc9686700 1 Processor -- start 2024-02-19T21:10:22.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.708+0000 7f6cc9686700 1 -- start start 2024-02-19T21:10:22.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.708+0000 7f6cc9686700 1 librados: setting wanted keys 2024-02-19T21:10:22.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.708+0000 7f6cc9686700 1 librados: calling monclient init 2024-02-19T21:10:22.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.708+0000 7f6cc9686700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6cc4117150 con 0x7f6cc4104250 2024-02-19T21:10:22.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.708+0000 7f6cc9686700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6cc41182f0 con 0x7f6cc40fd910 2024-02-19T21:10:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.708+0000 7f6cc9686700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6cc41194f0 con 0x7f6cc4100710 2024-02-19T21:10:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.708+0000 7f6cc37fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f6cc4104250 0x7f6cc41158d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43938/0 (socket says 172.21.15.163:43938) 2024-02-19T21:10:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.708+0000 7f6cc27fc700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f6cc4100710 0x7f6cc40f9df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45974/0 (socket says 172.21.15.163:45974) 2024-02-19T21:10:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.708+0000 7f6cc37fe700 1 -- 172.21.15.163:0/3303207114 learned_addr learned my addr 172.21.15.163:0/3303207114 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 10 client.?.objecter ms_handle_connect 0x7f6cc4104250 2024-02-19T21:10:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 10 client.?.objecter ms_handle_connect 0x7f6cc40fd910 2024-02-19T21:10:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 10 client.?.objecter ms_handle_connect 0x7f6cc4100710 2024-02-19T21:10:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699537950 0 0) 0x7f6cc4117150 con 0x7f6cc4104250 2024-02-19T21:10:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cac003030 con 0x7f6cc4104250 2024-02-19T21:10:22.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255765774 0 0) 0x7f6cc41182f0 con 0x7f6cc40fd910 2024-02-19T21:10:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc4117150 con 0x7f6cc40fd910 2024-02-19T21:10:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904353760 0 0) 0x7f6cc41194f0 con 0x7f6cc4100710 2024-02-19T21:10:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc41182f0 con 0x7f6cc4100710 2024-02-19T21:10:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 161868050 0 0) 0x7f6cac003030 con 0x7f6cc4104250 2024-02-19T21:10:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cc41194f0 con 0x7f6cc4104250 2024-02-19T21:10:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6cbc001cf0 con 0x7f6cc4104250 2024-02-19T21:10:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2257927656 0 0) 0x7f6cc4117150 con 0x7f6cc40fd910 2024-02-19T21:10:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cac003030 con 0x7f6cc40fd910 2024-02-19T21:10:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6cb4001f50 con 0x7f6cc40fd910 2024-02-19T21:10:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 842470190 0 0) 0x7f6cc41182f0 con 0x7f6cc4100710 2024-02-19T21:10:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cc4117150 con 0x7f6cc4100710 2024-02-19T21:10:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6cb8001900 con 0x7f6cc4100710 2024-02-19T21:10:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3155336437 0 0) 0x7f6cc41194f0 con 0x7f6cc4104250 2024-02-19T21:10:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 >> v1:172.21.15.163:6790/0 conn(0x7f6cc4100710 legacy=0x7f6cc40f9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:22.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 >> v1:172.21.15.193:6789/0 conn(0x7f6cc40fd910 legacy=0x7f6cc40f96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:22.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc411a6f0 con 0x7f6cc4104250 2024-02-19T21:10:22.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cc9686700 1 -- 172.21.15.163:0/3303207114 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc41184e0 con 0x7f6cc4104250 2024-02-19T21:10:22.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cc9686700 10 client.23133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:22.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cc9686700 1 -- 172.21.15.163:0/3303207114 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc41189d0 con 0x7f6cc4104250 2024-02-19T21:10:22.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.709+0000 7f6cc9686700 1 librados: init done 2024-02-19T21:10:22.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.710+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6cbc002790 con 0x7f6cc4104250 2024-02-19T21:10:22.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.710+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6cbc005df0 con 0x7f6cc4104250 2024-02-19T21:10:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.711+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6cbc018c80 con 0x7f6cc4104250 2024-02-19T21:10:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.711+0000 7f6cc9686700 1 -- 172.21.15.163:0/3303207114 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc405f5f0 con 0x7f6cc4104250 2024-02-19T21:10:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.711+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(420..420 src has 43..420) v4 ==== 7946+0+0 (unknown 358967885 0 0) 0x7f6cbc079e60 con 0x7f6cc4104250 2024-02-19T21:10:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.711+0000 7f6cabfff700 10 client.23133.objecter ms_dispatch 0x7f6cc4001550 osd_map(420..420 src has 43..420) v4 2024-02-19T21:10:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.711+0000 7f6cabfff700 3 client.23133.objecter handle_osd_map got epochs [420,420] > 0 2024-02-19T21:10:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.711+0000 7f6cabfff700 3 client.23133.objecter handle_osd_map decoding full epoch 420 2024-02-19T21:10:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.711+0000 7f6cabfff700 20 client.23133.objecter dump_active .. 0 homeless 2024-02-19T21:10:22.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.712+0000 7f6cabfff700 10 client.23133.objecter ms_handle_connect 0x7f6cac063c10 2024-02-19T21:10:22.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.716+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6cbc04d320 con 0x7f6cc4104250 2024-02-19T21:10:22.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:21.193922+0000 osd.2 (osd.2) 168 : cluster [DBG] 16.e scrub starts 2024-02-19T21:10:22.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:21.194706+0000 osd.2 (osd.2) 169 : cluster [DBG] 16.e scrub ok 2024-02-19T21:10:22.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:21.215024+0000 mgr.y (mgr.14550) 384 : cluster [DBG] pgmap v612: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-19T21:10:22.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:21.483785+0000 osd.6 (osd.6) 167 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:10:22.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:21.484696+0000 osd.6 (osd.6) 168 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:10:22.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:21.193922+0000 osd.2 (osd.2) 168 : cluster [DBG] 16.e scrub starts 2024-02-19T21:10:22.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:21.194706+0000 osd.2 (osd.2) 169 : cluster [DBG] 16.e scrub ok 2024-02-19T21:10:22.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:21.215024+0000 mgr.y (mgr.14550) 384 : cluster [DBG] pgmap v612: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-19T21:10:22.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:21.483785+0000 osd.6 (osd.6) 167 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:10:22.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:21.484696+0000 osd.6 (osd.6) 168 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:10:22.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:21.193922+0000 osd.2 (osd.2) 168 : cluster [DBG] 16.e scrub starts 2024-02-19T21:10:22.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:21.194706+0000 osd.2 (osd.2) 169 : cluster [DBG] 16.e scrub ok 2024-02-19T21:10:22.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:21.215024+0000 mgr.y (mgr.14550) 384 : cluster [DBG] pgmap v612: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-19T21:10:22.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:21.483785+0000 osd.6 (osd.6) 167 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:10:22.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:21.484696+0000 osd.6 (osd.6) 168 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:10:22.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:22.903+0000 7f6cc9686700 1 -- 172.21.15.163:0/3303207114 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 0.750587277064386} v 0) v1 -- 0x7f6cc4047540 con 0x7f6cc4104250 2024-02-19T21:10:23.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.135+0000 7f8b819ac700 1 -- 172.21.15.163:0/2317138106 >> v1:172.21.15.163:6789/0 conn(0x7f8b7c0f7280 legacy=0x7f8b7c104e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:23.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.135+0000 7f8b819ac700 1 -- 172.21.15.163:0/2317138106 shutdown_connections 2024-02-19T21:10:23.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.135+0000 7f8b819ac700 1 -- 172.21.15.163:0/2317138106 >> 172.21.15.163:0/2317138106 conn(0x7f8b7c0f2470 msgr2=0x7f8b7c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:23.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.135+0000 7f8b819ac700 1 -- 172.21.15.163:0/2317138106 shutdown_connections 2024-02-19T21:10:23.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.135+0000 7f8b819ac700 1 -- 172.21.15.163:0/2317138106 wait complete. 2024-02-19T21:10:23.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b819ac700 1 librados: starting msgr at 2024-02-19T21:10:23.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b819ac700 1 librados: starting objecter 2024-02-19T21:10:23.134 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b819ac700 1 Processor -- start 2024-02-19T21:10:23.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b819ac700 1 -- start start 2024-02-19T21:10:23.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b819ac700 1 librados: setting wanted keys 2024-02-19T21:10:23.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b819ac700 1 librados: calling monclient init 2024-02-19T21:10:23.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b819ac700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8b7c119570 con 0x7f8b7c0f6790 2024-02-19T21:10:23.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b819ac700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8b7c11a750 con 0x7f8b7c0f7280 2024-02-19T21:10:23.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b819ac700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8b7c11b950 con 0x7f8b7c1065c0 2024-02-19T21:10:23.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b7affd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8b7c0f6790 0x7f8b7c102ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43946/0 (socket says 172.21.15.163:43946) 2024-02-19T21:10:23.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b7affd700 1 -- 172.21.15.163:0/3957512817 learned_addr learned my addr 172.21.15.163:0/3957512817 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:23.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b63fff700 10 client.?.objecter ms_handle_connect 0x7f8b7c1065c0 2024-02-19T21:10:23.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b63fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:23.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b63fff700 10 client.?.objecter ms_handle_connect 0x7f8b7c0f6790 2024-02-19T21:10:23.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b63fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:23.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106822770 0 0) 0x7f8b7c11b950 con 0x7f8b7c1065c0 2024-02-19T21:10:23.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.136+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b64003030 con 0x7f8b7c1065c0 2024-02-19T21:10:23.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 10 client.?.objecter ms_handle_connect 0x7f8b7c0f7280 2024-02-19T21:10:23.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:23.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1731753751 0 0) 0x7f8b7c119570 con 0x7f8b7c0f6790 2024-02-19T21:10:23.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b7c11b950 con 0x7f8b7c0f6790 2024-02-19T21:10:23.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074011170 0 0) 0x7f8b7c11a750 con 0x7f8b7c0f7280 2024-02-19T21:10:23.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b7c119570 con 0x7f8b7c0f7280 2024-02-19T21:10:23.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1947870505 0 0) 0x7f8b64003030 con 0x7f8b7c1065c0 2024-02-19T21:10:23.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b7c11a750 con 0x7f8b7c1065c0 2024-02-19T21:10:23.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8b74001ca0 con 0x7f8b7c1065c0 2024-02-19T21:10:23.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 119104787 0 0) 0x7f8b7c11b950 con 0x7f8b7c0f6790 2024-02-19T21:10:23.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b64003030 con 0x7f8b7c0f6790 2024-02-19T21:10:23.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8b6c001880 con 0x7f8b7c0f6790 2024-02-19T21:10:23.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 827224851 0 0) 0x7f8b7c119570 con 0x7f8b7c0f7280 2024-02-19T21:10:23.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b7c11b950 con 0x7f8b7c0f7280 2024-02-19T21:10:23.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8b70002c20 con 0x7f8b7c0f7280 2024-02-19T21:10:23.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3489270338 0 0) 0x7f8b7c11a750 con 0x7f8b7c1065c0 2024-02-19T21:10:23.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 >> v1:172.21.15.193:6789/0 conn(0x7f8b7c0f7280 legacy=0x7f8b7c1032b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:23.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 >> v1:172.21.15.163:6789/0 conn(0x7f8b7c0f6790 legacy=0x7f8b7c102ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:23.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7c11cb50 con 0x7f8b7c1065c0 2024-02-19T21:10:23.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b819ac700 1 -- 172.21.15.163:0/3957512817 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b7c119760 con 0x7f8b7c1065c0 2024-02-19T21:10:23.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b819ac700 10 client.49859.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:23.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b819ac700 1 -- 172.21.15.163:0/3957512817 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b7c119bd0 con 0x7f8b7c1065c0 2024-02-19T21:10:23.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b819ac700 1 librados: init done 2024-02-19T21:10:23.139 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8b74002610 con 0x7f8b7c1065c0 2024-02-19T21:10:23.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.137+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8b74003670 con 0x7f8b7c1065c0 2024-02-19T21:10:23.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.138+0000 7f8b819ac700 1 -- 172.21.15.163:0/3957512817 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b68004f20 con 0x7f8b7c1065c0 2024-02-19T21:10:23.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.138+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f8b74002e00 con 0x7f8b7c1065c0 2024-02-19T21:10:23.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.139+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(420..420 src has 43..420) v4 ==== 7946+0+0 (unknown 358967885 0 0) 0x7f8b74076710 con 0x7f8b7c1065c0 2024-02-19T21:10:23.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.139+0000 7f8b63fff700 10 client.49859.objecter ms_dispatch 0x7f8b7c001550 osd_map(420..420 src has 43..420) v4 2024-02-19T21:10:23.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.139+0000 7f8b63fff700 3 client.49859.objecter handle_osd_map got epochs [420,420] > 0 2024-02-19T21:10:23.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.139+0000 7f8b63fff700 3 client.49859.objecter handle_osd_map decoding full epoch 420 2024-02-19T21:10:23.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.139+0000 7f8b63fff700 20 client.49859.objecter dump_active .. 0 homeless 2024-02-19T21:10:23.140 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.139+0000 7f8b63fff700 10 client.49859.objecter ms_handle_connect 0x7f8b64063d10 2024-02-19T21:10:23.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.143+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8b74049b70 con 0x7f8b7c1065c0 2024-02-19T21:10:23.164 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:23.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:23.197 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:23.270 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:10:23.298 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:23.319 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:10:23.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.329+0000 7f8b819ac700 1 -- 172.21.15.163:0/3957512817 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f8b680051d0 con 0x7f8b7c1065c0 2024-02-19T21:10:23.329 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:23.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:10:23.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.416+0000 7f8b63fff700 1 -- 172.21.15.163:0/3957512817 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v421) v1 ==== 87+0+0 (unknown 1671751658 0 0) 0x7f8b74047f80 con 0x7f8b7c1065c0 2024-02-19T21:10:23.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.416+0000 7f6cabfff700 1 -- 172.21.15.163:0/3303207114 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 0.750587277064386}]=0 set osd.6 primary-affinity to 0.750587 (8491902) v421) v1 ==== 154+0+0 (unknown 2054790827 0 0) 0x7f6cbc04b730 con 0x7f6cc4104250 2024-02-19T21:10:23.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.419+0000 7f6cc9686700 10 librados: watch_flush enter 2024-02-19T21:10:23.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.419+0000 7f6cc9686700 10 librados: watch_flush exit 2024-02-19T21:10:23.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.419+0000 7f6cc9686700 20 client.23133.objecter shutdown clearing up homeless session... 2024-02-19T21:10:23.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.419+0000 7f6cc9686700 10 client.23133.objecter successfully canceled tick 2024-02-19T21:10:23.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.419+0000 7f6cc9686700 1 -- 172.21.15.163:0/3303207114 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6cac063c10 legacy=0x7f6cac0660d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:23.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.419+0000 7f6cc9686700 1 -- 172.21.15.163:0/3303207114 >> v1:172.21.15.163:6789/0 conn(0x7f6cc4104250 legacy=0x7f6cc41158d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:23.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.419+0000 7f6cc9686700 1 -- 172.21.15.163:0/3303207114 shutdown_connections 2024-02-19T21:10:23.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.419+0000 7f6cc9686700 1 -- 172.21.15.163:0/3303207114 >> 172.21.15.163:0/3303207114 conn(0x7f6cc40f3530 msgr2=0x7f6cc41076c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:23.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.419+0000 7f6cc9686700 1 -- 172.21.15.163:0/3303207114 shutdown_connections 2024-02-19T21:10:23.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.419+0000 7f6cc9686700 1 -- 172.21.15.163:0/3303207114 wait complete. 2024-02-19T21:10:23.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.419+0000 7f6cc9686700 1 librados: shutdown 2024-02-19T21:10:23.420 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.6 primary-affinity to 0.750587 (8491902) 2024-02-19T21:10:23.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.421+0000 7f8b819ac700 10 librados: watch_flush enter 2024-02-19T21:10:23.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.421+0000 7f8b819ac700 10 librados: watch_flush exit 2024-02-19T21:10:23.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.421+0000 7f8b819ac700 20 client.49859.objecter shutdown clearing up homeless session... 2024-02-19T21:10:23.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.421+0000 7f8b819ac700 10 client.49859.objecter successfully canceled tick 2024-02-19T21:10:23.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.421+0000 7f8b819ac700 1 -- 172.21.15.163:0/3957512817 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8b64063d10 legacy=0x7f8b640661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:23.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.421+0000 7f8b819ac700 1 -- 172.21.15.163:0/3957512817 >> v1:172.21.15.163:6790/0 conn(0x7f8b7c1065c0 legacy=0x7f8b7c117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:23.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.425+0000 7f8b819ac700 1 -- 172.21.15.163:0/3957512817 shutdown_connections 2024-02-19T21:10:23.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.425+0000 7f8b819ac700 1 -- 172.21.15.163:0/3957512817 >> 172.21.15.163:0/3957512817 conn(0x7f8b7c0f2470 msgr2=0x7f8b7c109980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:23.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.427+0000 7f8b819ac700 1 -- 172.21.15.163:0/3957512817 shutdown_connections 2024-02-19T21:10:23.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.430+0000 7f8b819ac700 1 -- 172.21.15.163:0/3957512817 wait complete. 2024-02-19T21:10:23.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:23.430+0000 7f8b819ac700 1 librados: shutdown 2024-02-19T21:10:23.429 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:10:23.431 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:23.462 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:23.484 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:23.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:23.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:23.588 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:23.614 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:10:23] "GET /metrics HTTP/1.1" 200 189597 "" "Prometheus/2.33.4" 2024-02-19T21:10:23.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:21.731866+0000 osd.1 (osd.1) 201 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:10:23.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:21.732773+0000 osd.1 (osd.1) 202 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:10:23.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:22.457977+0000 osd.3 (osd.3) 185 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:10:23.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:22.459803+0000 osd.3 (osd.3) 186 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:10:23.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:22.843927+0000 osd.7 (osd.7) 191 : cluster [DBG] 61.15 scrub starts 2024-02-19T21:10:23.614 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:22.847434+0000 osd.7 (osd.7) 192 : cluster [DBG] 61.15 scrub ok 2024-02-19T21:10:23.615 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:22.905262+0000 mon.a (mon.0) 2529 : audit [INF] from='client.? v1:172.21.15.163:0/3303207114' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.750587277064386}]: dispatch 2024-02-19T21:10:23.615 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:23.331461+0000 mon.c (mon.2) 390 : audit [INF] from='client.? v1:172.21.15.163:0/3957512817' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:10:23.615 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:23.331823+0000 mon.a (mon.0) 2530 : audit [INF] from='client.49859 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:10:23.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:21.731866+0000 osd.1 (osd.1) 201 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:10:23.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:21.732773+0000 osd.1 (osd.1) 202 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:10:23.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:22.457977+0000 osd.3 (osd.3) 185 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:10:23.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:22.459803+0000 osd.3 (osd.3) 186 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:10:23.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:22.843927+0000 osd.7 (osd.7) 191 : cluster [DBG] 61.15 scrub starts 2024-02-19T21:10:23.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:22.847434+0000 osd.7 (osd.7) 192 : cluster [DBG] 61.15 scrub ok 2024-02-19T21:10:23.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:22.905262+0000 mon.a (mon.0) 2529 : audit [INF] from='client.? v1:172.21.15.163:0/3303207114' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.750587277064386}]: dispatch 2024-02-19T21:10:23.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:23.331461+0000 mon.c (mon.2) 390 : audit [INF] from='client.? v1:172.21.15.163:0/3957512817' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:10:23.615 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:23.331823+0000 mon.a (mon.0) 2530 : audit [INF] from='client.49859 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:10:23.616 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:23.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:21.731866+0000 osd.1 (osd.1) 201 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:10:23.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:21.732773+0000 osd.1 (osd.1) 202 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:10:23.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:22.457977+0000 osd.3 (osd.3) 185 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:10:23.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:22.459803+0000 osd.3 (osd.3) 186 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:10:23.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:22.843927+0000 osd.7 (osd.7) 191 : cluster [DBG] 61.15 scrub starts 2024-02-19T21:10:23.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:22.847434+0000 osd.7 (osd.7) 192 : cluster [DBG] 61.15 scrub ok 2024-02-19T21:10:23.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:22.905262+0000 mon.a (mon.0) 2529 : audit [INF] from='client.? v1:172.21.15.163:0/3303207114' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.750587277064386}]: dispatch 2024-02-19T21:10:23.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:23.331461+0000 mon.c (mon.2) 390 : audit [INF] from='client.? v1:172.21.15.163:0/3957512817' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:10:23.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:23.331823+0000 mon.a (mon.0) 2530 : audit [INF] from='client.49859 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:10:24.466 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:24.477 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:24.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:24.578 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:24.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:23.215394+0000 mgr.y (mgr.14550) 385 : cluster [DBG] pgmap v613: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-02-19T21:10:24.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:23.412198+0000 mon.a (mon.0) 2531 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:10:24.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:23.416620+0000 mon.a (mon.0) 2532 : audit [INF] from='client.? v1:172.21.15.163:0/3303207114' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.750587277064386}]': finished 2024-02-19T21:10:24.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:23.416685+0000 mon.a (mon.0) 2533 : audit [INF] from='client.49859 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:10:24.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:23.416734+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e421: 8 total, 8 up, 7 in 2024-02-19T21:10:24.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:24.268741+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e422: 8 total, 8 up, 7 in 2024-02-19T21:10:24.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:23.215394+0000 mgr.y (mgr.14550) 385 : cluster [DBG] pgmap v613: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-02-19T21:10:24.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:23.412198+0000 mon.a (mon.0) 2531 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:10:24.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:23.416620+0000 mon.a (mon.0) 2532 : audit [INF] from='client.? v1:172.21.15.163:0/3303207114' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.750587277064386}]': finished 2024-02-19T21:10:24.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:23.416685+0000 mon.a (mon.0) 2533 : audit [INF] from='client.49859 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:10:24.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:23.416734+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e421: 8 total, 8 up, 7 in 2024-02-19T21:10:24.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:24.268741+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e422: 8 total, 8 up, 7 in 2024-02-19T21:10:24.606 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:24.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:23.215394+0000 mgr.y (mgr.14550) 385 : cluster [DBG] pgmap v613: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-02-19T21:10:24.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:23.412198+0000 mon.a (mon.0) 2531 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:10:24.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:23.416620+0000 mon.a (mon.0) 2532 : audit [INF] from='client.? v1:172.21.15.163:0/3303207114' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.750587277064386}]': finished 2024-02-19T21:10:24.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:23.416685+0000 mon.a (mon.0) 2533 : audit [INF] from='client.49859 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:10:24.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:23.416734+0000 mon.a (mon.0) 2534 : cluster [DBG] osdmap e421: 8 total, 8 up, 7 in 2024-02-19T21:10:24.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:24.268741+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e422: 8 total, 8 up, 7 in 2024-02-19T21:10:25.689 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:10:25.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:25.406741+0000 mon.b (mon.1) 578 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:25.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:25.407170+0000 mon.a (mon.0) 2536 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:25.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:25.407736+0000 mon.b (mon.1) 579 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-61"}]: dispatch 2024-02-19T21:10:25.714 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:25.407992+0000 mon.a (mon.0) 2537 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-61"}]: dispatch 2024-02-19T21:10:25.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:25.406741+0000 mon.b (mon.1) 578 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:25.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:25.407170+0000 mon.a (mon.0) 2536 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:25.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:25.407736+0000 mon.b (mon.1) 579 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-61"}]: dispatch 2024-02-19T21:10:25.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:25.407992+0000 mon.a (mon.0) 2537 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-61"}]: dispatch 2024-02-19T21:10:25.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:25.406741+0000 mon.b (mon.1) 578 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:25.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:25.407170+0000 mon.a (mon.0) 2536 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:25.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:25.407736+0000 mon.b (mon.1) 579 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-61"}]: dispatch 2024-02-19T21:10:25.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:25.407992+0000 mon.a (mon.0) 2537 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-61"}]: dispatch 2024-02-19T21:10:26.084 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:26.093 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:26.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:26.145 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:26.195 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:26.227 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:26.249 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:26.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:26.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:26.353 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:26.382 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:26.403 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:26.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:26.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:26.507 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:26.537 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:26.661 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:26.662 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:25.216499+0000 mgr.y (mgr.14550) 386 : cluster [DBG] pgmap v616: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-19T21:10:26.662 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:25.489181+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e423: 8 total, 8 up, 7 in 2024-02-19T21:10:26.663 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:25.216499+0000 mgr.y (mgr.14550) 386 : cluster [DBG] pgmap v616: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-19T21:10:26.663 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:25.489181+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e423: 8 total, 8 up, 7 in 2024-02-19T21:10:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:26.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:26.765 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:26.791 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:26.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:25.216499+0000 mgr.y (mgr.14550) 386 : cluster [DBG] pgmap v616: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-19T21:10:26.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:25.489181+0000 mon.a (mon.0) 2538 : cluster [DBG] osdmap e423: 8 total, 8 up, 7 in 2024-02-19T21:10:26.915 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:10:27.683 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:26.493067+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e424: 8 total, 8 up, 7 in 2024-02-19T21:10:27.683 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:26.496744+0000 mon.b (mon.1) 580 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:27.683 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:26.497131+0000 mon.a (mon.0) 2540 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:27.683 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:27.077595+0000 mon.b (mon.1) 581 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:10:27.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:26.493067+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e424: 8 total, 8 up, 7 in 2024-02-19T21:10:27.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:26.496744+0000 mon.b (mon.1) 580 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:27.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:26.497131+0000 mon.a (mon.0) 2540 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:27.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:27.077595+0000 mon.b (mon.1) 581 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:10:27.770 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:26.493067+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e424: 8 total, 8 up, 7 in 2024-02-19T21:10:27.771 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:26.496744+0000 mon.b (mon.1) 580 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:27.771 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:26.497131+0000 mon.a (mon.0) 2540 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:27.771 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:27.077595+0000 mon.b (mon.1) 581 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:10:28.040 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:10:28.040 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-02-19T21:10:28.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-02-19T21:10:28.040 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-02-19T21:10:28.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.041 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.24919.0:2074 28.18 e102/99 28:19125b2f:test-rados-api-smithi163-83920-17::big3:head v 102'2, mlcod=0'0)", 2024-02-19T21:10:28.041 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:42.235636+0000", 2024-02-19T21:10:28.041 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 344.68697033199999, 2024-02-19T21:10:28.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.058551367, 2024-02-19T21:10:28.041 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.041 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:28.041 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235636+0000", 2024-02-19T21:10:28.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235636+0000", 2024-02-19T21:10:28.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999962 2024-02-19T21:10:28.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235632+0000", 2024-02-19T21:10:28.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013158625 2024-02-19T21:10:28.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.248791+0000", 2024-02-19T21:10:28.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2994999999999998e-05 2024-02-19T21:10:28.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.248824+0000", 2024-02-19T21:10:28.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0746e-05 2024-02-19T21:10:28.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.248834+0000", 2024-02-19T21:10:28.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0368e-05 2024-02-19T21:10:28.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.248915+0000", 2024-02-19T21:10:28.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2949e-05 2024-02-19T21:10:28.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.248938+0000", 2024-02-19T21:10:28.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.045192251000000003 2024-02-19T21:10:28.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:10:28.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.294130+0000", 2024-02-19T21:10:28.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7206000000000001e-05 2024-02-19T21:10:28.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.294187+0000", 2024-02-19T21:10:28.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.058551367 2024-02-19T21:10:28.045 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.045 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.045 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.045 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.18006.0:2 77.1f e127/126 77:fd8eaea7:nspace::oid_SimpleWritePP_:head v 127'1, mlcod=0'0)", 2024-02-19T21:10:28.045 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:07.144012+0000", 2024-02-19T21:10:28.045 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 319.778593774, 2024-02-19T21:10:28.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040142901000000002, 2024-02-19T21:10:28.046 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.046 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:28.046 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.046 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.144012+0000", 2024-02-19T21:10:28.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.144012+0000", 2024-02-19T21:10:28.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:10:28.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.144012+0000", 2024-02-19T21:10:28.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0854e-05 2024-02-19T21:10:28.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.144023+0000", 2024-02-19T21:10:28.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6949999999999997e-06 2024-02-19T21:10:28.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.144028+0000", 2024-02-19T21:10:28.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2780000000000001e-06 2024-02-19T21:10:28.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.144034+0000", 2024-02-19T21:10:28.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00078022499999999995 2024-02-19T21:10:28.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.144814+0000", 2024-02-19T21:10:28.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3957e-05 2024-02-19T21:10:28.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.144828+0000", 2024-02-19T21:10:28.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039272906000000003 2024-02-19T21:10:28.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:10:28.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.184101+0000", 2024-02-19T21:10:28.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4342000000000002e-05 2024-02-19T21:10:28.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.184155+0000", 2024-02-19T21:10:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040142901000000002 2024-02-19T21:10:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.35590.0:2 78.6 78:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected e128)", 2024-02-19T21:10:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:08.125592+0000", 2024-02-19T21:10:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 318.79701362200001, 2024-02-19T21:10:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.21091431599999999, 2024-02-19T21:10:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:10:28.051 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.35590", 2024-02-19T21:10:28.051 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/4170288922", 2024-02-19T21:10:28.051 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:10:28.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.051 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.125592+0000", 2024-02-19T21:10:28.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.125592+0000", 2024-02-19T21:10:28.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:10:28.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.125590+0000", 2024-02-19T21:10:28.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2050000000000002e-06 2024-02-19T21:10:28.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.125598+0000", 2024-02-19T21:10:28.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1060000000000001e-05 2024-02-19T21:10:28.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.125609+0000", 2024-02-19T21:10:28.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9550000000000003e-06 2024-02-19T21:10:28.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.125616+0000", 2024-02-19T21:10:28.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2948999999999998e-05 2024-02-19T21:10:28.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.125659+0000", 2024-02-19T21:10:28.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8620000000000002e-06 2024-02-19T21:10:28.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:10:28.055 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.125664+0000", 2024-02-19T21:10:28.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0025039559999999999 2024-02-19T21:10:28.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.055 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.055 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.128168+0000", 2024-02-19T21:10:28.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2759999999999997e-06 2024-02-19T21:10:28.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.055 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:10:28.055 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.128174+0000", 2024-02-19T21:10:28.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.13131832600000001 2024-02-19T21:10:28.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.055 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.056 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.259493+0000", 2024-02-19T21:10:28.056 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013339499999999999 2024-02-19T21:10:28.056 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.259626+0000", 2024-02-19T21:10:28.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015394599999999999 2024-02-19T21:10:28.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,5", 2024-02-19T21:10:28.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.259780+0000", 2024-02-19T21:10:28.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.005432114 2024-02-19T21:10:28.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:28.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.265212+0000", 2024-02-19T21:10:28.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00035388999999999999 2024-02-19T21:10:28.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:10:28.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.265566+0000", 2024-02-19T21:10:28.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.070841705000000005 2024-02-19T21:10:28.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:28.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.336408+0000", 2024-02-19T21:10:28.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6057999999999998e-05 2024-02-19T21:10:28.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:10:28.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.336464+0000", 2024-02-19T21:10:28.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2828999999999999e-05 2024-02-19T21:10:28.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.336507+0000", 2024-02-19T21:10:28.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.21091431599999999 2024-02-19T21:10:28.059 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.059 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.059 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.059 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.26962.0:28 64.1e 64:7e799bab:test-rados-api-smithi163-83952-10::foo:head [write 0~128] snapc a=[a] ondisk+write+known_if_redirected e128)", 2024-02-19T21:10:28.059 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:08.262935+0000", 2024-02-19T21:10:28.059 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 318.65967075100002, 2024-02-19T21:10:28.060 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073680541000000002, 2024-02-19T21:10:28.060 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.060 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:28.060 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:10:28.060 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.26962", 2024-02-19T21:10:28.060 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2924448120", 2024-02-19T21:10:28.060 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 28 2024-02-19T21:10:28.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.060 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.060 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.060 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.061 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.262935+0000", 2024-02-19T21:10:28.061 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.061 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.061 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.061 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.061 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.262935+0000", 2024-02-19T21:10:28.061 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:10:28.061 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.062 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.062 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.262933+0000", 2024-02-19T21:10:28.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6049999999999998e-06 2024-02-19T21:10:28.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.062 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.063 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.262941+0000", 2024-02-19T21:10:28.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0489999999999993e-06 2024-02-19T21:10:28.063 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.063 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.063 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.063 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.262950+0000", 2024-02-19T21:10:28.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2439999999999996e-06 2024-02-19T21:10:28.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.064 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.262958+0000", 2024-02-19T21:10:28.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8603999999999997e-05 2024-02-19T21:10:28.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.263006+0000", 2024-02-19T21:10:28.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000116204 2024-02-19T21:10:28.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.263122+0000", 2024-02-19T21:10:28.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015602699999999999 2024-02-19T21:10:28.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,4", 2024-02-19T21:10:28.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.263279+0000", 2024-02-19T21:10:28.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019019219999999999 2024-02-19T21:10:28.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:28.068 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.265180+0000", 2024-02-19T21:10:28.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00038971900000000001 2024-02-19T21:10:28.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:10:28.068 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.265570+0000", 2024-02-19T21:10:28.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.070975108999999995 2024-02-19T21:10:28.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:28.068 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.336545+0000", 2024-02-19T21:10:28.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6949999999999997e-05 2024-02-19T21:10:28.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.069 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:10:28.069 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.336582+0000", 2024-02-19T21:10:28.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3593999999999997e-05 2024-02-19T21:10:28.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.069 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.069 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.336616+0000", 2024-02-19T21:10:28.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.073680541000000002 2024-02-19T21:10:28.069 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.069 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.069 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.070 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.26962.0:35 64.1e 64:7e799bab:test-rados-api-smithi163-83952-10::foo:head [write 128~128] snapc b=[b,a] ondisk+write+known_if_redirected e130)", 2024-02-19T21:10:28.070 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:10.031101+0000", 2024-02-19T21:10:28.070 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 316.89150475999998, 2024-02-19T21:10:28.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040883140999999998, 2024-02-19T21:10:28.070 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.070 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:28.071 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:10:28.071 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.26962", 2024-02-19T21:10:28.071 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2924448120", 2024-02-19T21:10:28.071 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 35 2024-02-19T21:10:28.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.071 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.031101+0000", 2024-02-19T21:10:28.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.031101+0000", 2024-02-19T21:10:28.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:10:28.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.031099+0000", 2024-02-19T21:10:28.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7650000000000005e-06 2024-02-19T21:10:28.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.031109+0000", 2024-02-19T21:10:28.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5739999999999996e-06 2024-02-19T21:10:28.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.072 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.072 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.031117+0000", 2024-02-19T21:10:28.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1810000000000004e-06 2024-02-19T21:10:28.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.031124+0000", 2024-02-19T21:10:28.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002524344 2024-02-19T21:10:28.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.033649+0000", 2024-02-19T21:10:28.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0978999999999997e-05 2024-02-19T21:10:28.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.033740+0000", 2024-02-19T21:10:28.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018644399999999999 2024-02-19T21:10:28.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.074 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,4", 2024-02-19T21:10:28.074 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.033926+0000", 2024-02-19T21:10:28.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.035522170999999998 2024-02-19T21:10:28.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.074 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:10:28.074 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.069448+0000", 2024-02-19T21:10:28.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000941117 2024-02-19T21:10:28.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.074 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:28.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.070389+0000", 2024-02-19T21:10:28.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014978350000000001 2024-02-19T21:10:28.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:28.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.071887+0000", 2024-02-19T21:10:28.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1100000000000002e-05 2024-02-19T21:10:28.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:10:28.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.071938+0000", 2024-02-19T21:10:28.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6072000000000002e-05 2024-02-19T21:10:28.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.076 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.071984+0000", 2024-02-19T21:10:28.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040883140999999998 2024-02-19T21:10:28.076 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.076 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.076 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.076 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.076 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.26962.0:36 64.1e 64:7e799bab:test-rados-api-smithi163-83952-10::foo:head [write 384~128] snapc b=[b,a] ondisk+write+known_if_redirected e130)", 2024-02-19T21:10:28.076 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:10.076311+0000", 2024-02-19T21:10:28.076 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 316.84629535699997, 2024-02-19T21:10:28.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.069196832999999999, 2024-02-19T21:10:28.076 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.076 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:28.076 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:10:28.077 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.26962", 2024-02-19T21:10:28.077 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2924448120", 2024-02-19T21:10:28.077 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 36 2024-02-19T21:10:28.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.077 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.076311+0000", 2024-02-19T21:10:28.077 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.076311+0000", 2024-02-19T21:10:28.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999967 2024-02-19T21:10:28.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.078 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.076308+0000", 2024-02-19T21:10:28.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0696999999999999e-05 2024-02-19T21:10:28.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.078 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.076318+0000", 2024-02-19T21:10:28.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00096680900000000001 2024-02-19T21:10:28.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.077285+0000", 2024-02-19T21:10:28.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2339e-05 2024-02-19T21:10:28.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.077297+0000", 2024-02-19T21:10:28.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0095283250000000007 2024-02-19T21:10:28.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.080 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.086826+0000", 2024-02-19T21:10:28.080 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4335999999999999e-05 2024-02-19T21:10:28.080 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.080 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.080 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.086920+0000", 2024-02-19T21:10:28.080 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016475699999999999 2024-02-19T21:10:28.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,4", 2024-02-19T21:10:28.081 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.087085+0000", 2024-02-19T21:10:28.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057520519999999997 2024-02-19T21:10:28.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:28.081 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.092837+0000", 2024-02-19T21:10:28.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00031790600000000002 2024-02-19T21:10:28.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:28.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.093155+0000", 2024-02-19T21:10:28.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052249808000000002 2024-02-19T21:10:28.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:10:28.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.145405+0000", 2024-02-19T21:10:28.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9243999999999997e-05 2024-02-19T21:10:28.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:10:28.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.145464+0000", 2024-02-19T21:10:28.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3702999999999998e-05 2024-02-19T21:10:28.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.145507+0000", 2024-02-19T21:10:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.069196832999999999 2024-02-19T21:10:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.37339.0:2 81.7 e130/129 81:e732be06:::oid_WaitForSafePP_:head v 130'1, mlcod=0'0)", 2024-02-19T21:10:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:10.094671+0000", 2024-02-19T21:10:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 316.827935422, 2024-02-19T21:10:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051593172, 2024-02-19T21:10:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.094671+0000", 2024-02-19T21:10:28.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.094671+0000", 2024-02-19T21:10:28.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:10:28.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.094669+0000", 2024-02-19T21:10:28.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4699999999999996e-06 2024-02-19T21:10:28.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.094677+0000", 2024-02-19T21:10:28.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1141000000000001e-05 2024-02-19T21:10:28.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.094688+0000", 2024-02-19T21:10:28.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9869999999999999e-06 2024-02-19T21:10:28.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.094693+0000", 2024-02-19T21:10:28.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9091999999999998e-05 2024-02-19T21:10:28.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.094732+0000", 2024-02-19T21:10:28.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1522000000000001e-05 2024-02-19T21:10:28.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.094744+0000", 2024-02-19T21:10:28.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051476114000000003 2024-02-19T21:10:28.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:10:28.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.146220+0000", 2024-02-19T21:10:28.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4097000000000001e-05 2024-02-19T21:10:28.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.146264+0000", 2024-02-19T21:10:28.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051593172 2024-02-19T21:10:28.087 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.087 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.087 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.087 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.26962.0:42 64.1e 64:7e799bab:test-rados-api-smithi163-83952-10::foo:head [write 128~128] snapc c=[c,b,a] ondisk+write+known_if_redirected e131)", 2024-02-19T21:10:28.087 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:11.022187+0000", 2024-02-19T21:10:28.087 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 315.90041932600002, 2024-02-19T21:10:28.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071742103000000002, 2024-02-19T21:10:28.088 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.088 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:28.088 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:10:28.088 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.26962", 2024-02-19T21:10:28.088 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2924448120", 2024-02-19T21:10:28.088 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 42 2024-02-19T21:10:28.088 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.089 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.022187+0000", 2024-02-19T21:10:28.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.022187+0000", 2024-02-19T21:10:28.089 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999967 2024-02-19T21:10:28.089 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.089 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.089 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.089 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.022184+0000", 2024-02-19T21:10:28.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3356000000000001e-05 2024-02-19T21:10:28.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.022197+0000", 2024-02-19T21:10:28.090 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9962e-05 2024-02-19T21:10:28.090 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.090 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.090 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.090 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.022217+0000", 2024-02-19T21:10:28.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1600000000000004e-06 2024-02-19T21:10:28.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.091 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.091 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.091 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.022226+0000", 2024-02-19T21:10:28.091 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000518798 2024-02-19T21:10:28.091 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.022745+0000", 2024-02-19T21:10:28.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0090000000000007e-06 2024-02-19T21:10:28.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:10:28.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.022753+0000", 2024-02-19T21:10:28.092 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047432034999999997 2024-02-19T21:10:28.092 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.092 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.092 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.092 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.070185+0000", 2024-02-19T21:10:28.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078765090000000003 2024-02-19T21:10:28.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.078061+0000", 2024-02-19T21:10:28.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00022438499999999999 2024-02-19T21:10:28.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,4", 2024-02-19T21:10:28.093 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.078286+0000", 2024-02-19T21:10:28.093 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015273909 2024-02-19T21:10:28.093 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.093 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.093 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:10:28.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.093560+0000", 2024-02-19T21:10:28.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000235543 2024-02-19T21:10:28.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:28.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.093795+0000", 2024-02-19T21:10:28.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3720999999999998e-05 2024-02-19T21:10:28.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:28.094 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.093829+0000", 2024-02-19T21:10:28.094 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2135000000000001e-05 2024-02-19T21:10:28.094 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.094 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.094 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:10:28.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.093881+0000", 2024-02-19T21:10:28.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7689999999999999e-05 2024-02-19T21:10:28.095 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.095 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.095 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.095 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.093929+0000", 2024-02-19T21:10:28.095 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.071742103000000002 2024-02-19T21:10:28.095 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.095 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.095 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.095 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.095 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.095 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(76.3s2 131/126 ECSubWrite(tid=273, reqid=client.32629.0:7, at_version=0'0, trim_to=0'0, roll_forward_to=131'1))", 2024-02-19T21:10:28.095 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:11.026203+0000", 2024-02-19T21:10:28.096 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 315.89640256000001, 2024-02-19T21:10:28.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034294303999999998, 2024-02-19T21:10:28.096 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.096 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:10:28.096 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.096 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.096 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.096 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.026203+0000", 2024-02-19T21:10:28.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.096 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.096 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.096 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.096 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.026203+0000", 2024-02-19T21:10:28.096 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999948 2024-02-19T21:10:28.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.026198+0000", 2024-02-19T21:10:28.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2126000000000001e-05 2024-02-19T21:10:28.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.026221+0000", 2024-02-19T21:10:28.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0046410770000000004 2024-02-19T21:10:28.097 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.097 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.097 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.097 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.030862+0000", 2024-02-19T21:10:28.097 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0959e-05 2024-02-19T21:10:28.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.030873+0000", 2024-02-19T21:10:28.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012572500000000001 2024-02-19T21:10:28.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.030998+0000", 2024-02-19T21:10:28.098 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2343e-05 2024-02-19T21:10:28.098 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.098 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.098 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:10:28.098 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.031011+0000", 2024-02-19T21:10:28.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.029470832999999998 2024-02-19T21:10:28.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:10:28.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.060481+0000", 2024-02-19T21:10:28.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6317e-05 2024-02-19T21:10:28.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.099 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.099 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.099 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:11.060498+0000", 2024-02-19T21:10:28.099 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034294303999999998 2024-02-19T21:10:28.099 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.099 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.099 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.099 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.100 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.100 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(76.3s2 132/126 ECSubWrite(tid=274, reqid=client.32629.0:9, at_version=132'3, trim_to=0'0, roll_forward_to=131'1))", 2024-02-19T21:10:28.100 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:12.095839+0000", 2024-02-19T21:10:28.100 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 314.82676697699998, 2024-02-19T21:10:28.100 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.080104199000000001, 2024-02-19T21:10:28.100 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.100 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:10:28.100 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.100 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.100 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.100 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.095839+0000", 2024-02-19T21:10:28.100 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.100 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.100 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.101 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.101 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.095839+0000", 2024-02-19T21:10:28.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999967 2024-02-19T21:10:28.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.101 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.101 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.095836+0000", 2024-02-19T21:10:28.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0513999999999998e-05 2024-02-19T21:10:28.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.101 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.101 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.095886+0000", 2024-02-19T21:10:28.101 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1696999999999999e-05 2024-02-19T21:10:28.101 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.101 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.102 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.102 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.095908+0000", 2024-02-19T21:10:28.102 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0474e-05 2024-02-19T21:10:28.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.102 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.102 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.102 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.095918+0000", 2024-02-19T21:10:28.102 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028923695999999999 2024-02-19T21:10:28.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.102 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.102 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.102 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.124842+0000", 2024-02-19T21:10:28.102 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3617e-05 2024-02-19T21:10:28.102 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:10:28.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.124856+0000", 2024-02-19T21:10:28.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.051069145000000003 2024-02-19T21:10:28.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:10:28.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.175925+0000", 2024-02-19T21:10:28.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8382000000000001e-05 2024-02-19T21:10:28.103 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.103 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.103 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.103 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.175943+0000", 2024-02-19T21:10:28.103 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.080104199000000001 2024-02-19T21:10:28.103 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.104 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.104 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.104 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.104 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(76.3s2 132/126 ECSubWrite(tid=277, reqid=client.32629.0:10, at_version=0'0, trim_to=0'0, roll_forward_to=132'4))", 2024-02-19T21:10:28.104 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:12.194142+0000", 2024-02-19T21:10:28.104 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 314.72846398500002, 2024-02-19T21:10:28.104 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.075776198000000003, 2024-02-19T21:10:28.104 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.104 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:10:28.104 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.104 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.104 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.104 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.194142+0000", 2024-02-19T21:10:28.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.105 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.194142+0000", 2024-02-19T21:10:28.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999967 2024-02-19T21:10:28.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.105 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.194139+0000", 2024-02-19T21:10:28.105 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3570999999999999e-05 2024-02-19T21:10:28.105 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.105 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.105 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.194152+0000", 2024-02-19T21:10:28.106 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5959999999999993e-06 2024-02-19T21:10:28.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.106 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.194162+0000", 2024-02-19T21:10:28.106 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4579999999999998e-06 2024-02-19T21:10:28.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.106 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.106 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.194168+0000", 2024-02-19T21:10:28.106 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7952000000000003e-05 2024-02-19T21:10:28.106 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.106 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.106 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.194236+0000", 2024-02-19T21:10:28.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0896999999999999e-05 2024-02-19T21:10:28.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:10:28.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.194247+0000", 2024-02-19T21:10:28.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.075647968999999995 2024-02-19T21:10:28.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:10:28.107 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.269895+0000", 2024-02-19T21:10:28.107 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3059999999999999e-05 2024-02-19T21:10:28.107 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.107 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.107 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.108 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.269918+0000", 2024-02-19T21:10:28.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.075776198000000003 2024-02-19T21:10:28.108 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.108 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.108 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.108 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.108 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.108 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(osd.0.0:261 64.1e e132/116 ondisk, result = 0)", 2024-02-19T21:10:28.108 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:12.195760+0000", 2024-02-19T21:10:28.108 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 314.72684567599998, 2024-02-19T21:10:28.108 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.032371255000000002, 2024-02-19T21:10:28.108 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.108 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:10:28.108 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.109 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.109 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.109 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.195760+0000", 2024-02-19T21:10:28.109 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.109 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.109 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.109 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.109 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.195760+0000", 2024-02-19T21:10:28.109 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:10:28.109 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.109 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.109 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.109 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.195759+0000", 2024-02-19T21:10:28.109 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5490000000000003e-06 2024-02-19T21:10:28.109 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.195766+0000", 2024-02-19T21:10:28.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1839999999999999e-06 2024-02-19T21:10:28.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.195774+0000", 2024-02-19T21:10:28.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.694e-06 2024-02-19T21:10:28.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.110 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.110 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.110 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.195780+0000", 2024-02-19T21:10:28.110 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.032320793 2024-02-19T21:10:28.110 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.111 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.111 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.111 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.228100+0000", 2024-02-19T21:10:28.111 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2460000000000003e-06 2024-02-19T21:10:28.111 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.111 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.111 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.111 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.228109+0000", 2024-02-19T21:10:28.111 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3e-05 2024-02-19T21:10:28.111 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.111 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.111 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.111 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.228132+0000", 2024-02-19T21:10:28.111 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.032371255000000002 2024-02-19T21:10:28.111 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.112 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.112 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.112 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.112 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.112 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.1.0:241 28.15 e132/99 -30:ad087405:::temp_28.15_0_14259_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:10:28.112 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:12.394974+0000", 2024-02-19T21:10:28.112 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 314.52763167199998, 2024-02-19T21:10:28.112 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024743474000000001, 2024-02-19T21:10:28.112 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.112 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:28.112 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.112 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.112 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.112 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.394974+0000", 2024-02-19T21:10:28.112 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.113 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.113 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.113 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.394974+0000", 2024-02-19T21:10:28.113 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999957 2024-02-19T21:10:28.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.113 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.113 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.113 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.394970+0000", 2024-02-19T21:10:28.113 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015431281999999999 2024-02-19T21:10:28.113 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.113 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.113 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.113 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.410401+0000", 2024-02-19T21:10:28.113 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7678000000000002e-05 2024-02-19T21:10:28.114 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.114 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.114 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.114 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.410429+0000", 2024-02-19T21:10:28.114 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2159e-05 2024-02-19T21:10:28.114 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.114 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.114 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.114 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.410441+0000", 2024-02-19T21:10:28.114 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7043999999999999e-05 2024-02-19T21:10:28.114 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.114 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.114 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.114 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.410488+0000", 2024-02-19T21:10:28.114 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8236999999999998e-05 2024-02-19T21:10:28.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.115 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.115 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.115 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.410507+0000", 2024-02-19T21:10:28.115 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0091551280000000002 2024-02-19T21:10:28.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.115 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.115 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:10:28.115 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.419662+0000", 2024-02-19T21:10:28.115 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6150000000000003e-05 2024-02-19T21:10:28.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.115 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.115 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.115 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.419718+0000", 2024-02-19T21:10:28.115 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024743474000000001 2024-02-19T21:10:28.116 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.116 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.116 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.116 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.116 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.116 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.1.0:242 28.15 e132/99 -30:ad087405:::temp_28.15_0_14259_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:10:28.116 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:12.457486+0000", 2024-02-19T21:10:28.116 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 314.46511985799998, 2024-02-19T21:10:28.116 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028618069999999999, 2024-02-19T21:10:28.116 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.116 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:28.116 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.116 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.116 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.117 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.457486+0000", 2024-02-19T21:10:28.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.117 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.117 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.457486+0000", 2024-02-19T21:10:28.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999967 2024-02-19T21:10:28.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.117 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.117 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.457483+0000", 2024-02-19T21:10:28.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0030976419999999998 2024-02-19T21:10:28.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.460581+0000", 2024-02-19T21:10:28.118 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8076999999999999e-05 2024-02-19T21:10:28.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.118 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.460609+0000", 2024-02-19T21:10:28.118 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0805000000000001e-05 2024-02-19T21:10:28.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.118 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.118 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.118 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.460620+0000", 2024-02-19T21:10:28.118 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3998999999999997e-05 2024-02-19T21:10:28.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.118 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.119 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.119 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.460664+0000", 2024-02-19T21:10:28.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7855000000000001e-05 2024-02-19T21:10:28.119 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.119 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.119 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.119 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.460681+0000", 2024-02-19T21:10:28.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025371474000000001 2024-02-19T21:10:28.119 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.119 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.119 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:10:28.119 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.486053+0000", 2024-02-19T21:10:28.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1366000000000003e-05 2024-02-19T21:10:28.119 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.120 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.120 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.120 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.486104+0000", 2024-02-19T21:10:28.120 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028618069999999999 2024-02-19T21:10:28.120 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.120 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.120 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.120 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.120 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24919.0:2080 28.2 28:4327d3d8:test-rados-api-smithi163-83920-17::big3.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-02-19T21:10:28.120 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:12.735204+0000", 2024-02-19T21:10:28.120 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 314.18740229000002, 2024-02-19T21:10:28.120 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.11288258, 2024-02-19T21:10:28.120 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.120 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:28.121 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:10:28.121 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24919", 2024-02-19T21:10:28.121 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/4089460683", 2024-02-19T21:10:28.121 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2080 2024-02-19T21:10:28.121 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.121 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.121 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.121 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.121 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.735204+0000", 2024-02-19T21:10:28.121 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.121 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.121 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.121 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.121 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.735204+0000", 2024-02-19T21:10:28.122 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999957 2024-02-19T21:10:28.122 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.122 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.122 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.122 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.735199+0000", 2024-02-19T21:10:28.122 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8963000000000001e-05 2024-02-19T21:10:28.122 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.122 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.122 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.122 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.735218+0000", 2024-02-19T21:10:28.122 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2756999999999999e-05 2024-02-19T21:10:28.122 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.122 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.123 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.123 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.735241+0000", 2024-02-19T21:10:28.123 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2425e-05 2024-02-19T21:10:28.123 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.123 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.123 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.123 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.735263+0000", 2024-02-19T21:10:28.123 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5859999999999996e-05 2024-02-19T21:10:28.123 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.123 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.123 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.123 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.735329+0000", 2024-02-19T21:10:28.123 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014922899999999999 2024-02-19T21:10:28.123 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.123 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.124 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.124 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.735478+0000", 2024-02-19T21:10:28.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085817493999999994 2024-02-19T21:10:28.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.124 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.124 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,6", 2024-02-19T21:10:28.124 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.821296+0000", 2024-02-19T21:10:28.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.01048439 2024-02-19T21:10:28.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.124 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.124 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:10:28.124 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.831780+0000", 2024-02-19T21:10:28.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010506199000000001 2024-02-19T21:10:28.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:28.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.842287+0000", 2024-02-19T21:10:28.125 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0057363889999999997 2024-02-19T21:10:28.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:28.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.848023+0000", 2024-02-19T21:10:28.125 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5840999999999998e-05 2024-02-19T21:10:28.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:10:28.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.848059+0000", 2024-02-19T21:10:28.125 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7531000000000001e-05 2024-02-19T21:10:28.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.126 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.126 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.126 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.848086+0000", 2024-02-19T21:10:28.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.11288258 2024-02-19T21:10:28.126 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.126 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.126 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.126 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.126 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.126 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.1.0:248 28.1 e135/99 28:81ba0521:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa::a:head v 135'4, mlcod=101'2)", 2024-02-19T21:10:28.126 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:15.879399+0000", 2024-02-19T21:10:28.126 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 311.04320750199997, 2024-02-19T21:10:28.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.030664420000000001, 2024-02-19T21:10:28.126 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.126 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:28.127 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.127 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.127 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.127 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.879399+0000", 2024-02-19T21:10:28.127 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.127 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.127 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.127 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.879399+0000", 2024-02-19T21:10:28.127 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:10:28.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.127 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.127 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.127 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.879396+0000", 2024-02-19T21:10:28.128 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2748000000000001e-05 2024-02-19T21:10:28.128 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.128 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.128 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.128 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.879409+0000", 2024-02-19T21:10:28.128 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9813000000000002e-05 2024-02-19T21:10:28.128 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.128 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.128 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.128 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.879429+0000", 2024-02-19T21:10:28.128 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0879999999999997e-06 2024-02-19T21:10:28.128 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.128 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.128 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.128 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.879436+0000", 2024-02-19T21:10:28.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.011592504 2024-02-19T21:10:28.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.129 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.129 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.891028+0000", 2024-02-19T21:10:28.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2391000000000002e-05 2024-02-19T21:10:28.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.129 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.129 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.891061+0000", 2024-02-19T21:10:28.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018928383999999999 2024-02-19T21:10:28.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.129 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:10:28.129 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.909989+0000", 2024-02-19T21:10:28.130 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3757000000000006e-05 2024-02-19T21:10:28.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.130 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.910063+0000", 2024-02-19T21:10:28.130 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.030664420000000001 2024-02-19T21:10:28.130 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.130 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.130 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.130 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.26962.0:57 64.3 e139/116 64:c6c3d28d:test-rados-api-smithi163-83952-12::foo:head v 139'1, mlcod=0'0)", 2024-02-19T21:10:28.130 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:19.236525+0000", 2024-02-19T21:10:28.130 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 307.68608131500002, 2024-02-19T21:10:28.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022859982000000001, 2024-02-19T21:10:28.131 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.131 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:28.131 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.131 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.236525+0000", 2024-02-19T21:10:28.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.131 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.236525+0000", 2024-02-19T21:10:28.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:10:28.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.132 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.236522+0000", 2024-02-19T21:10:28.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6226999999999999e-05 2024-02-19T21:10:28.132 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.132 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.132 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.236538+0000", 2024-02-19T21:10:28.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018783389999999999 2024-02-19T21:10:28.132 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.132 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.132 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.238416+0000", 2024-02-19T21:10:28.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0345e-05 2024-02-19T21:10:28.132 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.238427+0000", 2024-02-19T21:10:28.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00082106099999999999 2024-02-19T21:10:28.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.239248+0000", 2024-02-19T21:10:28.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9076000000000003e-05 2024-02-19T21:10:28.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.239287+0000", 2024-02-19T21:10:28.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020049533000000001 2024-02-19T21:10:28.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.134 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:10:28.134 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.259336+0000", 2024-02-19T21:10:28.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8401e-05 2024-02-19T21:10:28.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.134 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.134 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.259385+0000", 2024-02-19T21:10:28.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022859982000000001 2024-02-19T21:10:28.134 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.134 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.134 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.34832.0:1058 8.e 8:70000000::::head [pgnls start_epoch 167] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e167)", 2024-02-19T21:10:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:47.972694+0000", 2024-02-19T21:10:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 278.94991238099999, 2024-02-19T21:10:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.25516716699999997, 2024-02-19T21:10:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:10:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:10:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.34832", 2024-02-19T21:10:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/21182929", 2024-02-19T21:10:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1058 2024-02-19T21:10:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:47.972694+0000", 2024-02-19T21:10:28.136 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.136 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.136 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.136 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.136 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:47.972694+0000", 2024-02-19T21:10:28.136 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:10:28.136 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.136 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.136 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.136 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:47.972692+0000", 2024-02-19T21:10:28.136 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9970000000000004e-06 2024-02-19T21:10:28.136 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.136 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.136 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:47.972698+0000", 2024-02-19T21:10:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0899999999999994e-06 2024-02-19T21:10:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:47.972707+0000", 2024-02-19T21:10:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9420000000000003e-06 2024-02-19T21:10:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:47.972712+0000", 2024-02-19T21:10:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2963999999999997e-05 2024-02-19T21:10:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:47.972755+0000", 2024-02-19T21:10:28.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2589999999999997e-06 2024-02-19T21:10:28.138 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.138 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:10:28.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:47.972762+0000", 2024-02-19T21:10:28.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.25496217900000001 2024-02-19T21:10:28.138 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.138 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:48.227724+0000", 2024-02-19T21:10:28.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4984000000000003e-05 2024-02-19T21:10:28.138 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.138 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:48.227759+0000", 2024-02-19T21:10:28.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010208 2024-02-19T21:10:28.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:48.227861+0000", 2024-02-19T21:10:28.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.25516716699999997 2024-02-19T21:10:28.139 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.139 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.139 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.139 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.49013.0:2 121.6 121:602f83fe:::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e177)", 2024-02-19T21:10:28.139 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:57.188301+0000", 2024-02-19T21:10:28.139 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 269.73430495399998, 2024-02-19T21:10:28.140 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023548389999999999, 2024-02-19T21:10:28.140 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.140 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:28.140 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:10:28.140 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.49013", 2024-02-19T21:10:28.140 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3219076896", 2024-02-19T21:10:28.140 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:10:28.140 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.140 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.140 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.140 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.140 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.188301+0000", 2024-02-19T21:10:28.140 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.140 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.141 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.141 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.141 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.188301+0000", 2024-02-19T21:10:28.141 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:10:28.141 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.141 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.141 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.141 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.188299+0000", 2024-02-19T21:10:28.141 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0073e-05 2024-02-19T21:10:28.141 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.141 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.141 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.141 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.188309+0000", 2024-02-19T21:10:28.141 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0526e-05 2024-02-19T21:10:28.141 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.142 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.142 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.142 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.188329+0000", 2024-02-19T21:10:28.142 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9959999999999999e-06 2024-02-19T21:10:28.142 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.142 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.142 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.142 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.188338+0000", 2024-02-19T21:10:28.142 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2525999999999998e-05 2024-02-19T21:10:28.142 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.142 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.143 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.143 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.188391+0000", 2024-02-19T21:10:28.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9600000000000003e-06 2024-02-19T21:10:28.143 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.143 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.143 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:10:28.143 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.188398+0000", 2024-02-19T21:10:28.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078013830000000003 2024-02-19T21:10:28.143 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.143 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.143 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.143 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.196199+0000", 2024-02-19T21:10:28.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6174000000000001e-05 2024-02-19T21:10:28.143 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.144 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.144 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:10:28.144 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.196215+0000", 2024-02-19T21:10:28.144 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012055725 2024-02-19T21:10:28.144 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.144 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.144 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.144 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.208271+0000", 2024-02-19T21:10:28.144 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011015599999999999 2024-02-19T21:10:28.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.147 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.147 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.147 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.208381+0000", 2024-02-19T21:10:28.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000151442 2024-02-19T21:10:28.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.147 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.147 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,7", 2024-02-19T21:10:28.147 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.208533+0000", 2024-02-19T21:10:28.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000265445 2024-02-19T21:10:28.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.148 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.148 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:10:28.148 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.208798+0000", 2024-02-19T21:10:28.148 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028852719999999999 2024-02-19T21:10:28.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.148 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.148 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:28.148 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.211683+0000", 2024-02-19T21:10:28.148 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000108552 2024-02-19T21:10:28.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.148 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.148 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:10:28.148 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.211792+0000", 2024-02-19T21:10:28.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2673999999999999e-05 2024-02-19T21:10:28.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.149 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:10:28.149 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.211825+0000", 2024-02-19T21:10:28.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4946000000000001e-05 2024-02-19T21:10:28.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.150 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.150 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:57.211849+0000", 2024-02-19T21:10:28.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023548389999999999 2024-02-19T21:10:28.150 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.150 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.150 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.150 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.47941.0:1 136.6s0 136:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e195)", 2024-02-19T21:10:28.150 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:06:15.197589+0000", 2024-02-19T21:10:28.150 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 251.72501679600001, 2024-02-19T21:10:28.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.009813734, 2024-02-19T21:10:28.151 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:10:28.151 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:10:28.151 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:10:28.151 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.47941", 2024-02-19T21:10:28.151 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/1153528047", 2024-02-19T21:10:28.151 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:10:28.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.151 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:10:28.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:10:28.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:15.197589+0000", 2024-02-19T21:10:28.151 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:10:28.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:10:28.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:15.197589+0000", 2024-02-19T21:10:28.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:10:28.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:10:28.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:15.197587+0000", 2024-02-19T21:10:28.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1840000000000004e-06 2024-02-19T21:10:28.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:10:28.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:15.197594+0000", 2024-02-19T21:10:28.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.114e-06 2024-02-19T21:10:28.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:10:28.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:15.197601+0000", 2024-02-19T21:10:28.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2460000000000003e-06 2024-02-19T21:10:28.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:10:28.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:15.197606+0000", 2024-02-19T21:10:28.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2356e-05 2024-02-19T21:10:28.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:15.197638+0000", 2024-02-19T21:10:28.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2850000000000001e-06 2024-02-19T21:10:28.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:10:28.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:15.197641+0000", 2024-02-19T21:10:28.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.99697799600000003 2024-02-19T21:10:28.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:10:28.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:16.194619+0000", 2024-02-19T21:10:28.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011509400000000001 2024-02-19T21:10:28.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:10:28.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:16.194734+0000", 2024-02-19T21:10:28.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00047517499999999999 2024-02-19T21:10:28.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.155 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:10:28.155 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:16.195209+0000", 2024-02-19T21:10:28.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0078615660000000004 2024-02-19T21:10:28.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.155 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:10:28.155 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:16.203070+0000", 2024-02-19T21:10:28.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0042641789999999999 2024-02-19T21:10:28.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.155 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:10:28.155 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:16.207335+0000", 2024-02-19T21:10:28.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8275999999999995e-05 2024-02-19T21:10:28.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:28.156 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:28.156 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:10:28.156 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:16.207403+0000", 2024-02-19T21:10:28.156 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.009813734 2024-02-19T21:10:28.156 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.156 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.156 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.156 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:28.156 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:28.156 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:10:28.157 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:10:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:10:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:10:28.299 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:28.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:28.404 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:28.434 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:28.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.410+0000 7febd2e06700 1 -- 172.21.15.163:0/1733474566 >> v1:172.21.15.163:6789/0 conn(0x7febcc0fc870 legacy=0x7febcc0fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:28.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.411+0000 7febd2e06700 1 -- 172.21.15.163:0/1733474566 shutdown_connections 2024-02-19T21:10:28.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.411+0000 7febd2e06700 1 -- 172.21.15.163:0/1733474566 >> 172.21.15.163:0/1733474566 conn(0x7febcc0f2470 msgr2=0x7febcc0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:28.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.411+0000 7febd2e06700 1 -- 172.21.15.163:0/1733474566 shutdown_connections 2024-02-19T21:10:28.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.411+0000 7febd2e06700 1 -- 172.21.15.163:0/1733474566 wait complete. 2024-02-19T21:10:28.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febd2e06700 1 librados: starting msgr at 2024-02-19T21:10:28.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febd2e06700 1 librados: starting objecter 2024-02-19T21:10:28.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febd2e06700 1 Processor -- start 2024-02-19T21:10:28.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febd2e06700 1 -- start start 2024-02-19T21:10:28.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febd2e06700 1 librados: setting wanted keys 2024-02-19T21:10:28.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febd2e06700 1 librados: calling monclient init 2024-02-19T21:10:28.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febd2e06700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7febcc1054c0 con 0x7febcc0ff670 2024-02-19T21:10:28.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febd2e06700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7febcc0fbe10 con 0x7febcc1031b0 2024-02-19T21:10:28.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febd2e06700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7febcc0fbfa0 con 0x7febcc0fc870 2024-02-19T21:10:28.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febd0ba2700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7febcc0fc870 0x7febcc0fb700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:34180/0 (socket says 172.21.15.163:34180) 2024-02-19T21:10:28.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febd0ba2700 1 -- 172.21.15.163:0/293037476 learned_addr learned my addr 172.21.15.163:0/293037476 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:28.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febc9ffb700 10 client.?.objecter ms_handle_connect 0x7febcc0fc870 2024-02-19T21:10:28.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febc9ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:28.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febc9ffb700 10 client.?.objecter ms_handle_connect 0x7febcc0ff670 2024-02-19T21:10:28.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febc9ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:28.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febc9ffb700 10 client.?.objecter ms_handle_connect 0x7febcc1031b0 2024-02-19T21:10:28.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febc9ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:28.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.412+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660090814 0 0) 0x7febcc1054c0 con 0x7febcc0ff670 2024-02-19T21:10:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febbc002220 con 0x7febcc0ff670 2024-02-19T21:10:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407458881 0 0) 0x7febcc0fbfa0 con 0x7febcc0fc870 2024-02-19T21:10:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febcc1054c0 con 0x7febcc0fc870 2024-02-19T21:10:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152249028 0 0) 0x7febcc0fbe10 con 0x7febcc1031b0 2024-02-19T21:10:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febcc0fbfa0 con 0x7febcc1031b0 2024-02-19T21:10:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 354475506 0 0) 0x7febbc002220 con 0x7febcc0ff670 2024-02-19T21:10:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febcc0fbe10 con 0x7febcc0ff670 2024-02-19T21:10:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7febb8002730 con 0x7febcc0ff670 2024-02-19T21:10:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3668020263 0 0) 0x7febcc1054c0 con 0x7febcc0fc870 2024-02-19T21:10:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febbc002220 con 0x7febcc0fc870 2024-02-19T21:10:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7febc0000fe0 con 0x7febcc0fc870 2024-02-19T21:10:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4115761549 0 0) 0x7febcc0fbfa0 con 0x7febcc1031b0 2024-02-19T21:10:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febcc1054c0 con 0x7febcc1031b0 2024-02-19T21:10:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7febc4001b80 con 0x7febcc1031b0 2024-02-19T21:10:28.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4201023522 0 0) 0x7febcc0fbe10 con 0x7febcc0ff670 2024-02-19T21:10:28.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 >> v1:172.21.15.163:6790/0 conn(0x7febcc0fc870 legacy=0x7febcc0fb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:28.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 >> v1:172.21.15.193:6789/0 conn(0x7febcc1031b0 legacy=0x7febcc11cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:28.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febcc0fc130 con 0x7febcc0ff670 2024-02-19T21:10:28.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febcc11e240 con 0x7febcc0ff670 2024-02-19T21:10:28.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7febb8001370 con 0x7febcc0ff670 2024-02-19T21:10:28.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7febb8003200 con 0x7febcc0ff670 2024-02-19T21:10:28.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febd2e06700 10 client.23145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:28.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febcc11e590 con 0x7febcc0ff670 2024-02-19T21:10:28.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.413+0000 7febd2e06700 1 librados: init done 2024-02-19T21:10:28.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.414+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7febcc05f700 con 0x7febcc0ff670 2024-02-19T21:10:28.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7febb8000ed0 con 0x7febcc0ff670 2024-02-19T21:10:28.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(425..425 src has 43..425) v4 ==== 7918+0+0 (unknown 2315237338 0 0) 0x7febb8076470 con 0x7febcc0ff670 2024-02-19T21:10:28.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 osd_map(425..425 src has 43..425) v4 2024-02-19T21:10:28.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febc9ffb700 3 client.23145.objecter handle_osd_map got epochs [425,425] > 0 2024-02-19T21:10:28.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febc9ffb700 3 client.23145.objecter handle_osd_map decoding full epoch 425 2024-02-19T21:10:28.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febc9ffb700 20 client.23145.objecter dump_active .. 0 homeless 2024-02-19T21:10:28.521 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:10:28.521 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:10:28.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v425) v1 ==== 54+0+16 (unknown 3517772337 0 2420000192) 0x7febb8078860 con 0x7febcc0ff670 2024-02-19T21:10:28.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febc9ffb700 10 client.23145.objecter ms_handle_connect 0x7febbc063d20 2024-02-19T21:10:28.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febd2e06700 10 client.23145.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:28.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 1 2024-02-19T21:10:28.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc047540 osd=0 3 2024-02-19T21:10:28.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc047540 osd=0 3 2024-02-19T21:10:28.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc047540 osd=0 3 2024-02-19T21:10:28.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 1 2024-02-19T21:10:28.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc047540 osd=0 3 2024-02-19T21:10:28.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 0 1 2024-02-19T21:10:28.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc047540 osd=0 4 2024-02-19T21:10:28.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febd2e06700 10 client.23145.objecter _send_command 1 2024-02-19T21:10:28.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.415+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7febcc10a0b0 con 0x7febcc028540 2024-02-19T21:10:28.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.416+0000 7febc9ffb700 10 client.23145.objecter ms_handle_connect 0x7febcc028540 2024-02-19T21:10:28.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.416+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febcc10a0b0 con 0x7febcc028540 2024-02-19T21:10:28.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.416+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:10:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.416+0000 7febc9ffb700 10 client.23145.objecter _finish_command 1 = system:0 2024-02-19T21:10:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.416+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc047540 osd=0 4 2024-02-19T21:10:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.416+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 0 1 2024-02-19T21:10:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.432+0000 7febd2e06700 10 client.23145.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.432+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 2 2024-02-19T21:10:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.432+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc047540 osd=0 3 2024-02-19T21:10:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.432+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc047540 osd=0 3 2024-02-19T21:10:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.432+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc047540 osd=0 3 2024-02-19T21:10:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.432+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 2 2024-02-19T21:10:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.432+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc047540 osd=0 3 2024-02-19T21:10:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.432+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 0 2 2024-02-19T21:10:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.432+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc047540 osd=0 4 2024-02-19T21:10:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.432+0000 7febd2e06700 10 client.23145.objecter _send_command 2 2024-02-19T21:10:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.432+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febcc0f3f70 con 0x7febcc028540 2024-02-19T21:10:28.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.433+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febcc0f3f70 con 0x7febcc028540 2024-02-19T21:10:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.433+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.433+0000 7febc9ffb700 10 client.23145.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.433+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc047540 osd=0 4 2024-02-19T21:10:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.433+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 0 2 2024-02-19T21:10:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.434+0000 7febd2e06700 10 client.23145.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.434+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 3 2024-02-19T21:10:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.434+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc0595a0 osd=1 3 2024-02-19T21:10:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.434+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc0595a0 osd=1 3 2024-02-19T21:10:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.434+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc0595a0 osd=1 3 2024-02-19T21:10:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.434+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 3 2024-02-19T21:10:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.434+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc0595a0 osd=1 3 2024-02-19T21:10:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.434+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 1 3 2024-02-19T21:10:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.434+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc0595a0 osd=1 4 2024-02-19T21:10:28.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.434+0000 7febd2e06700 10 client.23145.objecter _send_command 3 2024-02-19T21:10:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.434+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7febcc0280a0 con 0x7febcc059710 2024-02-19T21:10:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.434+0000 7febc9ffb700 10 client.23145.objecter ms_handle_connect 0x7febcc059710 2024-02-19T21:10:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.434+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febcc0280a0 con 0x7febcc059710 2024-02-19T21:10:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.435+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:10:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.435+0000 7febc9ffb700 10 client.23145.objecter _finish_command 3 = system:0 2024-02-19T21:10:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.435+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc0595a0 osd=1 4 2024-02-19T21:10:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.435+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 1 3 2024-02-19T21:10:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.450+0000 7febd2e06700 10 client.23145.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.450+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 4 2024-02-19T21:10:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.450+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc0595a0 osd=1 3 2024-02-19T21:10:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.450+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc0595a0 osd=1 3 2024-02-19T21:10:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.450+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc0595a0 osd=1 3 2024-02-19T21:10:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.450+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 4 2024-02-19T21:10:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.450+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc0595a0 osd=1 3 2024-02-19T21:10:28.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.450+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 1 4 2024-02-19T21:10:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.450+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc0595a0 osd=1 4 2024-02-19T21:10:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.450+0000 7febd2e06700 10 client.23145.objecter _send_command 4 2024-02-19T21:10:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.450+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febcc125100 con 0x7febcc059710 2024-02-19T21:10:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febcc125100 con 0x7febcc059710 2024-02-19T21:10:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febc9ffb700 10 client.23145.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc0595a0 osd=1 4 2024-02-19T21:10:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 1 4 2024-02-19T21:10:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febd2e06700 10 client.23145.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 5 2024-02-19T21:10:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc027600 osd=2 3 2024-02-19T21:10:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc027600 osd=2 3 2024-02-19T21:10:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc027600 osd=2 3 2024-02-19T21:10:28.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 5 2024-02-19T21:10:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc027600 osd=2 3 2024-02-19T21:10:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 2 5 2024-02-19T21:10:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc027600 osd=2 4 2024-02-19T21:10:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febd2e06700 10 client.23145.objecter _send_command 5 2024-02-19T21:10:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.452+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7febcc121530 con 0x7febcc027770 2024-02-19T21:10:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.453+0000 7febc9ffb700 10 client.23145.objecter ms_handle_connect 0x7febcc027770 2024-02-19T21:10:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.453+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febcc121530 con 0x7febcc027770 2024-02-19T21:10:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.453+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:10:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.453+0000 7febc9ffb700 10 client.23145.objecter _finish_command 5 = system:0 2024-02-19T21:10:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.453+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc027600 osd=2 4 2024-02-19T21:10:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.453+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 2 5 2024-02-19T21:10:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.468+0000 7febd2e06700 10 client.23145.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.468+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 6 2024-02-19T21:10:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.468+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc027600 osd=2 3 2024-02-19T21:10:28.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.468+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc027600 osd=2 3 2024-02-19T21:10:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.468+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc027600 osd=2 3 2024-02-19T21:10:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.468+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 6 2024-02-19T21:10:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.468+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc027600 osd=2 3 2024-02-19T21:10:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.468+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 2 6 2024-02-19T21:10:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.468+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc027600 osd=2 4 2024-02-19T21:10:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.468+0000 7febd2e06700 10 client.23145.objecter _send_command 6 2024-02-19T21:10:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.468+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febcc129b80 con 0x7febcc027770 2024-02-19T21:10:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.469+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febcc129b80 con 0x7febcc027770 2024-02-19T21:10:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.469+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.469+0000 7febc9ffb700 10 client.23145.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.469+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc027600 osd=2 4 2024-02-19T21:10:28.529 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:10:28.529 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:10:28.529 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:10:28.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.469+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 2 6 2024-02-19T21:10:28.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.469+0000 7febd2e06700 10 client.23145.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:28.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 7 2024-02-19T21:10:28.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc120a90 osd=3 3 2024-02-19T21:10:28.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc120a90 osd=3 3 2024-02-19T21:10:28.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc120a90 osd=3 3 2024-02-19T21:10:28.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 7 2024-02-19T21:10:28.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc120a90 osd=3 3 2024-02-19T21:10:28.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 3 7 2024-02-19T21:10:28.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc120a90 osd=3 4 2024-02-19T21:10:28.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febd2e06700 10 client.23145.objecter _send_command 7 2024-02-19T21:10:28.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7febcc126010 con 0x7febcc120c00 2024-02-19T21:10:28.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febc9ffb700 10 client.23145.objecter ms_handle_connect 0x7febcc120c00 2024-02-19T21:10:28.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febcc126010 con 0x7febcc120c00 2024-02-19T21:10:28.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:10:28.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febc9ffb700 10 client.23145.objecter _finish_command 7 = system:0 2024-02-19T21:10:28.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc120a90 osd=3 4 2024-02-19T21:10:28.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.470+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 3 7 2024-02-19T21:10:28.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.484+0000 7febd2e06700 10 client.23145.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:28.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.485+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 8 2024-02-19T21:10:28.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.485+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc120a90 osd=3 3 2024-02-19T21:10:28.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.485+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc120a90 osd=3 3 2024-02-19T21:10:28.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.485+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc120a90 osd=3 3 2024-02-19T21:10:28.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.485+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 8 2024-02-19T21:10:28.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.485+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc120a90 osd=3 3 2024-02-19T21:10:28.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.485+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 3 8 2024-02-19T21:10:28.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.485+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc120a90 osd=3 4 2024-02-19T21:10:28.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.485+0000 7febd2e06700 10 client.23145.objecter _send_command 8 2024-02-19T21:10:28.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.485+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febcc126010 con 0x7febcc120c00 2024-02-19T21:10:28.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febcc126010 con 0x7febcc120c00 2024-02-19T21:10:28.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:28.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febc9ffb700 10 client.23145.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc120a90 osd=3 4 2024-02-19T21:10:28.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 3 8 2024-02-19T21:10:28.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febd2e06700 10 client.23145.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:28.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 9 2024-02-19T21:10:28.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc126010 osd=4 3 2024-02-19T21:10:28.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc126010 osd=4 3 2024-02-19T21:10:28.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc126010 osd=4 3 2024-02-19T21:10:28.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 9 2024-02-19T21:10:28.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc126010 osd=4 3 2024-02-19T21:10:28.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 4 9 2024-02-19T21:10:28.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc126010 osd=4 4 2024-02-19T21:10:28.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febd2e06700 10 client.23145.objecter _send_command 9 2024-02-19T21:10:28.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.486+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7febcc12a900 con 0x7febcc12a4c0 2024-02-19T21:10:28.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.487+0000 7febc9ffb700 10 client.23145.objecter ms_handle_connect 0x7febcc12a4c0 2024-02-19T21:10:28.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.488+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febcc12a900 con 0x7febcc12a4c0 2024-02-19T21:10:28.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.488+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:10:28.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.488+0000 7febc9ffb700 10 client.23145.objecter _finish_command 9 = system:0 2024-02-19T21:10:28.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.488+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc126010 osd=4 4 2024-02-19T21:10:28.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.488+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 4 9 2024-02-19T21:10:28.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.506+0000 7febd2e06700 10 client.23145.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:28.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.506+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 10 2024-02-19T21:10:28.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.506+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc126010 osd=4 3 2024-02-19T21:10:28.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.506+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc126010 osd=4 3 2024-02-19T21:10:28.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.506+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc126010 osd=4 3 2024-02-19T21:10:28.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.506+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 10 2024-02-19T21:10:28.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.506+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc126010 osd=4 3 2024-02-19T21:10:28.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.506+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 4 10 2024-02-19T21:10:28.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.506+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc126010 osd=4 4 2024-02-19T21:10:28.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.506+0000 7febd2e06700 10 client.23145.objecter _send_command 10 2024-02-19T21:10:28.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.506+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febcc12a900 con 0x7febcc12a4c0 2024-02-19T21:10:28.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febcc12a900 con 0x7febcc12a4c0 2024-02-19T21:10:28.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:28.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febc9ffb700 10 client.23145.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc126010 osd=4 4 2024-02-19T21:10:28.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 4 10 2024-02-19T21:10:28.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febd2e06700 10 client.23145.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:28.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 11 2024-02-19T21:10:28.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc12a900 osd=5 3 2024-02-19T21:10:28.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc12a900 osd=5 3 2024-02-19T21:10:28.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc12a900 osd=5 3 2024-02-19T21:10:28.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 11 2024-02-19T21:10:28.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc12a900 osd=5 3 2024-02-19T21:10:28.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 5 11 2024-02-19T21:10:28.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc12a900 osd=5 4 2024-02-19T21:10:28.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febd2e06700 10 client.23145.objecter _send_command 11 2024-02-19T21:10:28.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.508+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7febcc136e70 con 0x7febcc133360 2024-02-19T21:10:28.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.509+0000 7febc9ffb700 10 client.23145.objecter ms_handle_connect 0x7febcc133360 2024-02-19T21:10:28.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.512+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febcc136e70 con 0x7febcc133360 2024-02-19T21:10:28.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.512+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:10:28.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.512+0000 7febc9ffb700 10 client.23145.objecter _finish_command 11 = system:0 2024-02-19T21:10:28.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.512+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc12a900 osd=5 4 2024-02-19T21:10:28.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.512+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 5 11 2024-02-19T21:10:28.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.531+0000 7febd2e06700 10 client.23145.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:28.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.531+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 12 2024-02-19T21:10:28.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.531+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc12a900 osd=5 3 2024-02-19T21:10:28.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.531+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc12a900 osd=5 3 2024-02-19T21:10:28.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.531+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc12a900 osd=5 3 2024-02-19T21:10:28.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.531+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 12 2024-02-19T21:10:28.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.531+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc12a900 osd=5 3 2024-02-19T21:10:28.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.531+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 5 12 2024-02-19T21:10:28.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.531+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc12a900 osd=5 4 2024-02-19T21:10:28.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.531+0000 7febd2e06700 10 client.23145.objecter _send_command 12 2024-02-19T21:10:28.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.531+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febcc136e70 con 0x7febcc133360 2024-02-19T21:10:28.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febcc136e70 con 0x7febcc133360 2024-02-19T21:10:28.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:28.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febc9ffb700 10 client.23145.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc12a900 osd=5 4 2024-02-19T21:10:28.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 5 12 2024-02-19T21:10:28.537 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:10:28.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febd2e06700 10 client.23145.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:28.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 13 2024-02-19T21:10:28.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc136e70 osd=6 3 2024-02-19T21:10:28.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc136e70 osd=6 3 2024-02-19T21:10:28.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc136e70 osd=6 3 2024-02-19T21:10:28.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 13 2024-02-19T21:10:28.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc136e70 osd=6 3 2024-02-19T21:10:28.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 6 13 2024-02-19T21:10:28.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc136e70 osd=6 4 2024-02-19T21:10:28.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febd2e06700 10 client.23145.objecter _send_command 13 2024-02-19T21:10:28.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7febcc13b1b0 con 0x7febcc1374f0 2024-02-19T21:10:28.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.533+0000 7febc9ffb700 10 client.23145.objecter ms_handle_connect 0x7febcc1374f0 2024-02-19T21:10:28.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.534+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febcc13b1b0 con 0x7febcc1374f0 2024-02-19T21:10:28.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.534+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:10:28.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.534+0000 7febc9ffb700 10 client.23145.objecter _finish_command 13 = system:0 2024-02-19T21:10:28.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.534+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc136e70 osd=6 4 2024-02-19T21:10:28.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.534+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 6 13 2024-02-19T21:10:28.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:28.540 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:28.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:28.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.549+0000 7febd2e06700 10 client.23145.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:28.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.549+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 14 2024-02-19T21:10:28.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.549+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc136e70 osd=6 3 2024-02-19T21:10:28.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.549+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc136e70 osd=6 3 2024-02-19T21:10:28.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.549+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc136e70 osd=6 3 2024-02-19T21:10:28.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.549+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 14 2024-02-19T21:10:28.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.549+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc136e70 osd=6 3 2024-02-19T21:10:28.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.549+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 6 14 2024-02-19T21:10:28.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.549+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc136e70 osd=6 4 2024-02-19T21:10:28.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.549+0000 7febd2e06700 10 client.23145.objecter _send_command 14 2024-02-19T21:10:28.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.549+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febcc13b1b0 con 0x7febcc1374f0 2024-02-19T21:10:28.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.551+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febcc13b1b0 con 0x7febcc1374f0 2024-02-19T21:10:28.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.551+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:28.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.551+0000 7febc9ffb700 10 client.23145.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.551+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc136e70 osd=6 4 2024-02-19T21:10:28.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.551+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 6 14 2024-02-19T21:10:28.550 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:10:28.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.552+0000 7febd2e06700 10 client.23145.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:28.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.552+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 15 2024-02-19T21:10:28.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.552+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc13b1b0 osd=7 3 2024-02-19T21:10:28.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.552+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc13b1b0 osd=7 3 2024-02-19T21:10:28.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.552+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc13b1b0 osd=7 3 2024-02-19T21:10:28.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.552+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 15 2024-02-19T21:10:28.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.552+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc13b1b0 osd=7 3 2024-02-19T21:10:28.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.552+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 7 15 2024-02-19T21:10:28.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.552+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc13b1b0 osd=7 4 2024-02-19T21:10:28.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.552+0000 7febd2e06700 10 client.23145.objecter _send_command 15 2024-02-19T21:10:28.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.552+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7febcc13c400 con 0x7febcc13ceb0 2024-02-19T21:10:28.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.552+0000 7febc9ffb700 10 client.23145.objecter ms_handle_connect 0x7febcc13ceb0 2024-02-19T21:10:28.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.553+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7febcc13c400 con 0x7febcc13ceb0 2024-02-19T21:10:28.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.553+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:10:28.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.553+0000 7febc9ffb700 10 client.23145.objecter _finish_command 15 = system:0 2024-02-19T21:10:28.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.553+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc13b1b0 osd=7 4 2024-02-19T21:10:28.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.553+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 7 15 2024-02-19T21:10:28.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.567+0000 7febd2e06700 10 client.23145.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:28.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.567+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign -1 16 2024-02-19T21:10:28.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.567+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc13b1b0 osd=7 3 2024-02-19T21:10:28.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.567+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc13b1b0 osd=7 3 2024-02-19T21:10:28.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.567+0000 7febd2e06700 20 client.23145.objecter _get_session s=0x7febcc13b1b0 osd=7 3 2024-02-19T21:10:28.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.567+0000 7febd2e06700 15 client.23145.objecter _session_command_op_remove -1 16 2024-02-19T21:10:28.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.567+0000 7febd2e06700 20 client.23145.objecter get_session s=0x7febcc13b1b0 osd=7 3 2024-02-19T21:10:28.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.567+0000 7febd2e06700 15 client.23145.objecter _session_command_op_assign 7 16 2024-02-19T21:10:28.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.567+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc13b1b0 osd=7 4 2024-02-19T21:10:28.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.567+0000 7febd2e06700 10 client.23145.objecter _send_command 16 2024-02-19T21:10:28.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.567+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7febcc13c400 con 0x7febcc13ceb0 2024-02-19T21:10:28.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febc9ffb700 1 -- 172.21.15.163:0/293037476 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7febcc13c400 con 0x7febcc13ceb0 2024-02-19T21:10:28.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febc9ffb700 10 client.23145.objecter ms_dispatch 0x7febcc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:28.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febc9ffb700 10 client.23145.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febc9ffb700 20 client.23145.objecter put_session s=0x7febcc13b1b0 osd=7 4 2024-02-19T21:10:28.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febc9ffb700 15 client.23145.objecter _session_command_op_remove 7 16 2024-02-19T21:10:28.568 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:10:28.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 10 librados: watch_flush enter 2024-02-19T21:10:28.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 10 librados: watch_flush exit 2024-02-19T21:10:28.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 10 client.23145.objecter close_session for osd.0 2024-02-19T21:10:28.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 >> v1:172.21.15.163:6801/1559418115 conn(0x7febcc028540 legacy=0x7febcc0f2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:28.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc047540 osd=0 1 2024-02-19T21:10:28.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 10 client.23145.objecter close_session for osd.1 2024-02-19T21:10:28.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 >> v1:172.21.15.163:6805/3125047432 conn(0x7febcc059710 legacy=0x7febcc1239e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:28.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc0595a0 osd=1 1 2024-02-19T21:10:28.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 10 client.23145.objecter close_session for osd.2 2024-02-19T21:10:28.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 >> v1:172.21.15.163:6809/1563158487 conn(0x7febcc027770 legacy=0x7febcc128460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:28.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc027600 osd=2 1 2024-02-19T21:10:28.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 10 client.23145.objecter close_session for osd.3 2024-02-19T21:10:28.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd13a3700 1 -- 172.21.15.163:0/293037476 reap_dead start 2024-02-19T21:10:28.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 >> v1:172.21.15.163:6813/2112279769 conn(0x7febcc120c00 legacy=0x7febcc12ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:28.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc120a90 osd=3 1 2024-02-19T21:10:28.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 10 client.23145.objecter close_session for osd.4 2024-02-19T21:10:28.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 >> v1:172.21.15.193:6800/3129716965 conn(0x7febcc12a4c0 legacy=0x7febcc131730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:28.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc126010 osd=4 1 2024-02-19T21:10:28.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 10 client.23145.objecter close_session for osd.5 2024-02-19T21:10:28.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 >> v1:172.21.15.193:6804/805161991 conn(0x7febcc133360 legacy=0x7febcc135750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:28.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc12a900 osd=5 1 2024-02-19T21:10:28.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 10 client.23145.objecter close_session for osd.6 2024-02-19T21:10:28.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 >> v1:172.21.15.193:6808/787604615 conn(0x7febcc1374f0 legacy=0x7febcc139900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:28.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc136e70 osd=6 1 2024-02-19T21:10:28.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 10 client.23145.objecter close_session for osd.7 2024-02-19T21:10:28.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 >> v1:172.21.15.193:6812/157927432 conn(0x7febcc13ceb0 legacy=0x7febcc13d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:28.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 20 client.23145.objecter put_session s=0x7febcc13b1b0 osd=7 1 2024-02-19T21:10:28.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 20 client.23145.objecter shutdown clearing up homeless session... 2024-02-19T21:10:28.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 10 client.23145.objecter successfully canceled tick 2024-02-19T21:10:28.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd13a3700 1 -- 172.21.15.163:0/293037476 reap_dead start 2024-02-19T21:10:28.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 >> v1:172.21.15.163:6800/1552514681 conn(0x7febbc063d20 legacy=0x7febbc0661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:28.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 >> v1:172.21.15.163:6789/0 conn(0x7febcc0ff670 legacy=0x7febcc1193f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:28.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 shutdown_connections 2024-02-19T21:10:28.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 >> 172.21.15.163:0/293037476 conn(0x7febcc0f2470 msgr2=0x7febcc061200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:28.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 shutdown_connections 2024-02-19T21:10:28.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 1 -- 172.21.15.163:0/293037476 wait complete. 2024-02-19T21:10:28.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:28.570+0000 7febd2e06700 1 librados: shutdown 2024-02-19T21:10:28.572 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.572 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.572 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.572 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.573 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.573 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.573 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.573 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:10:28.592 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:27.217134+0000 mgr.y (mgr.14550) 387 : cluster [DBG] pgmap v619: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:10:28.592 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:27.500648+0000 mon.a (mon.0) 2541 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:10:28.592 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:27.500823+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e425: 8 total, 8 up, 7 in 2024-02-19T21:10:28.592 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:27.503814+0000 mon.b (mon.1) 582 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:10:28.592 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:27.506069+0000 mon.b (mon.1) 583 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:10:28.593 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:27.509049+0000 mon.a (mon.0) 2543 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:10:28.593 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:27.793412+0000 mon.b (mon.1) 584 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:10:28.593 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:27.794119+0000 mon.b (mon.1) 585 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:10:28.593 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:27.799034+0000 mon.a (mon.0) 2544 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:10:28.593 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:28.416550+0000 mon.a (mon.0) 2545 : audit [DBG] from='client.? v1:172.21.15.163:0/293037476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:28.593 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:27.217134+0000 mgr.y (mgr.14550) 387 : cluster [DBG] pgmap v619: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:10:28.593 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:27.500648+0000 mon.a (mon.0) 2541 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:10:28.593 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:27.500823+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e425: 8 total, 8 up, 7 in 2024-02-19T21:10:28.593 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:27.503814+0000 mon.b (mon.1) 582 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:10:28.593 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:27.506069+0000 mon.b (mon.1) 583 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:10:28.593 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:27.509049+0000 mon.a (mon.0) 2543 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:10:28.593 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:27.793412+0000 mon.b (mon.1) 584 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:10:28.594 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:27.794119+0000 mon.b (mon.1) 585 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:10:28.594 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:27.799034+0000 mon.a (mon.0) 2544 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:10:28.594 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:28.416550+0000 mon.a (mon.0) 2545 : audit [DBG] from='client.? v1:172.21.15.163:0/293037476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:28.641 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:28.670 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:28.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:27.217134+0000 mgr.y (mgr.14550) 387 : cluster [DBG] pgmap v619: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:10:28.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:27.500648+0000 mon.a (mon.0) 2541 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:10:28.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:27.500823+0000 mon.a (mon.0) 2542 : cluster [DBG] osdmap e425: 8 total, 8 up, 7 in 2024-02-19T21:10:28.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:27.503814+0000 mon.b (mon.1) 582 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:10:28.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:27.506069+0000 mon.b (mon.1) 583 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:10:28.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:27.509049+0000 mon.a (mon.0) 2543 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:10:28.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:27.793412+0000 mon.b (mon.1) 584 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:10:28.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:27.794119+0000 mon.b (mon.1) 585 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:10:28.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:27.799034+0000 mon.a (mon.0) 2544 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:10:28.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:28.416550+0000 mon.a (mon.0) 2545 : audit [DBG] from='client.? v1:172.21.15.163:0/293037476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:29.520 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:29.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:29.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:29.526 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-19T21:10:29.624 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:29.653 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:29.672 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:29.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:29.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:29.776 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:29.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:28.501158+0000 mon.a (mon.0) 2546 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:10:29.803 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:28.501232+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e426: 8 total, 8 up, 7 in 2024-02-19T21:10:29.804 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:29.823 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:29.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:29.882 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:28.501158+0000 mon.a (mon.0) 2546 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:10:29.882 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:28.501232+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e426: 8 total, 8 up, 7 in 2024-02-19T21:10:29.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:28.501158+0000 mon.a (mon.0) 2546 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:10:29.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:28.501232+0000 mon.a (mon.0) 2547 : cluster [DBG] osdmap e426: 8 total, 8 up, 7 in 2024-02-19T21:10:29.927 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:29.955 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:29.973 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:29.976 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:29.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:30.077 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:10:30.104 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:30.122 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:10:30.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:30.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:10:30.226 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:30.253 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:30.273 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:30.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:30.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:30.342 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:30.376 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:30.381 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:30.403 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:30.421 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:30.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:30.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:30.525 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:10:30.551 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:30.570 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:10:30.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:30.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:10:30.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:30.701 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:30.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:10:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:10:30.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:29.217634+0000 mgr.y (mgr.14550) 388 : cluster [DBG] pgmap v622: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:10:30.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:29.512445+0000 mon.a (mon.0) 2548 : cluster [DBG] osdmap e427: 8 total, 8 up, 7 in 2024-02-19T21:10:30.861 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:29.217634+0000 mgr.y (mgr.14550) 388 : cluster [DBG] pgmap v622: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:10:30.861 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:29.512445+0000 mon.a (mon.0) 2548 : cluster [DBG] osdmap e427: 8 total, 8 up, 7 in 2024-02-19T21:10:30.861 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:29.217634+0000 mgr.y (mgr.14550) 388 : cluster [DBG] pgmap v622: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:10:30.862 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:29.512445+0000 mon.a (mon.0) 2548 : cluster [DBG] osdmap e427: 8 total, 8 up, 7 in 2024-02-19T21:10:30.899 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:30.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:31.002 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:31.030 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:31.107 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 3] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:10:31.108 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:10:31.108 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-02-19T21:10:31.108 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd in 4 2024-02-19T21:10:31.378 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:31.380 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:31.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:31.481 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:31.514 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:31.540 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:31.543 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:31.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:31.644 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:31.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:30.413429+0000 mgr.y (mgr.14550) 389 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:31.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:30.516121+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e428: 8 total, 8 up, 7 in 2024-02-19T21:10:31.670 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:30.413429+0000 mgr.y (mgr.14550) 389 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:31.670 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:30.516121+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e428: 8 total, 8 up, 7 in 2024-02-19T21:10:31.671 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:31.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:30.413429+0000 mgr.y (mgr.14550) 389 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:31.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:30.516121+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e428: 8 total, 8 up, 7 in 2024-02-19T21:10:31.819 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:31.822 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:31.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:31.924 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:31.951 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:32.149 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:32.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:32.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:32.255 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:10:32.283 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:32.292 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:10:32.633 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:10:32.636 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:32.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:10:32.737 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:32.765 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:32.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:31.218043+0000 mgr.y (mgr.14550) 390 : cluster [DBG] pgmap v625: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-02-19T21:10:32.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:31.522848+0000 mon.a (mon.0) 2550 : cluster [DBG] osdmap e429: 8 total, 8 up, 7 in 2024-02-19T21:10:32.903 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:31.218043+0000 mgr.y (mgr.14550) 390 : cluster [DBG] pgmap v625: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-02-19T21:10:32.904 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:31.522848+0000 mon.a (mon.0) 2550 : cluster [DBG] osdmap e429: 8 total, 8 up, 7 in 2024-02-19T21:10:32.904 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:31.218043+0000 mgr.y (mgr.14550) 390 : cluster [DBG] pgmap v625: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-02-19T21:10:32.904 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:31.522848+0000 mon.a (mon.0) 2550 : cluster [DBG] osdmap e429: 8 total, 8 up, 7 in 2024-02-19T21:10:32.909 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:32.913 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:32.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:33.022 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:10:33.055 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:33.076 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:10:33.079 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:33.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:10:33.180 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:33.210 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:33.231 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:33.234 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:33.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:33.336 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:10:33.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.344+0000 7fe6b22ac700 1 -- 172.21.15.163:0/3007126225 >> v1:172.21.15.163:6789/0 conn(0x7fe6ac1042c0 legacy=0x7fe6ac106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:33.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.345+0000 7fe6b22ac700 1 -- 172.21.15.163:0/3007126225 shutdown_connections 2024-02-19T21:10:33.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.345+0000 7fe6b22ac700 1 -- 172.21.15.163:0/3007126225 >> 172.21.15.163:0/3007126225 conn(0x7fe6ac0f35a0 msgr2=0x7fe6ac0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:33.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.345+0000 7fe6b22ac700 1 -- 172.21.15.163:0/3007126225 shutdown_connections 2024-02-19T21:10:33.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.345+0000 7fe6b22ac700 1 -- 172.21.15.163:0/3007126225 wait complete. 2024-02-19T21:10:33.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.345+0000 7fe6b22ac700 1 librados: starting msgr at 2024-02-19T21:10:33.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.345+0000 7fe6b22ac700 1 librados: starting objecter 2024-02-19T21:10:33.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.345+0000 7fe6b22ac700 1 Processor -- start 2024-02-19T21:10:33.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.345+0000 7fe6b22ac700 1 -- start start 2024-02-19T21:10:33.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.345+0000 7fe6b22ac700 1 librados: setting wanted keys 2024-02-19T21:10:33.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.345+0000 7fe6b22ac700 1 librados: calling monclient init 2024-02-19T21:10:33.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.345+0000 7fe6b22ac700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6ac1171c0 con 0x7fe6ac0fd980 2024-02-19T21:10:33.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.345+0000 7fe6b22ac700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6ac1183a0 con 0x7fe6ac1042c0 2024-02-19T21:10:33.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.345+0000 7fe6b22ac700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe6ac119580 con 0x7fe6ac100780 2024-02-19T21:10:33.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a3fff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fe6ac100780 0x7fe6ac0f9e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:34208/0 (socket says 172.21.15.163:34208) 2024-02-19T21:10:33.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a3fff700 1 -- 172.21.15.163:0/3315251923 learned_addr learned my addr 172.21.15.163:0/3315251923 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:33.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 10 client.?.objecter ms_handle_connect 0x7fe6ac100780 2024-02-19T21:10:33.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:33.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 10 client.?.objecter ms_handle_connect 0x7fe6ac1042c0 2024-02-19T21:10:33.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:33.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516659229 0 0) 0x7fe6ac119580 con 0x7fe6ac100780 2024-02-19T21:10:33.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe698003030 con 0x7fe6ac100780 2024-02-19T21:10:33.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892686325 0 0) 0x7fe6ac1183a0 con 0x7fe6ac1042c0 2024-02-19T21:10:33.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6ac119580 con 0x7fe6ac1042c0 2024-02-19T21:10:33.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 86474488 0 0) 0x7fe698003030 con 0x7fe6ac100780 2024-02-19T21:10:33.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6ac1183a0 con 0x7fe6ac100780 2024-02-19T21:10:33.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 10 client.?.objecter ms_handle_connect 0x7fe6ac0fd980 2024-02-19T21:10:33.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:33.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe694001940 con 0x7fe6ac100780 2024-02-19T21:10:33.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3075262899 0 0) 0x7fe6ac119580 con 0x7fe6ac1042c0 2024-02-19T21:10:33.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe698003030 con 0x7fe6ac1042c0 2024-02-19T21:10:33.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe6a40017b0 con 0x7fe6ac1042c0 2024-02-19T21:10:33.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454922141 0 0) 0x7fe6ac1171c0 con 0x7fe6ac0fd980 2024-02-19T21:10:33.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6ac119580 con 0x7fe6ac0fd980 2024-02-19T21:10:33.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 456114314 0 0) 0x7fe6ac1183a0 con 0x7fe6ac100780 2024-02-19T21:10:33.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 >> v1:172.21.15.193:6789/0 conn(0x7fe6ac1042c0 legacy=0x7fe6ac115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:33.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 >> v1:172.21.15.163:6789/0 conn(0x7fe6ac0fd980 legacy=0x7fe6ac0f9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:33.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ac11a780 con 0x7fe6ac100780 2024-02-19T21:10:33.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6b22ac700 1 -- 172.21.15.163:0/3315251923 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6ac118590 con 0x7fe6ac100780 2024-02-19T21:10:33.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6b22ac700 10 client.49868.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:33.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6b22ac700 1 -- 172.21.15.163:0/3315251923 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6ac118a80 con 0x7fe6ac100780 2024-02-19T21:10:33.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6b22ac700 1 librados: init done 2024-02-19T21:10:33.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fe6940024a0 con 0x7fe6ac100780 2024-02-19T21:10:33.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.346+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe694005b50 con 0x7fe6ac100780 2024-02-19T21:10:33.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.347+0000 7fe6b22ac700 1 -- 172.21.15.163:0/3315251923 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe68c004f20 con 0x7fe6ac100780 2024-02-19T21:10:33.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.348+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fe694006e20 con 0x7fe6ac100780 2024-02-19T21:10:33.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.348+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(429..429 src has 43..429) v4 ==== 7974+0+0 (unknown 916420470 0 0) 0x7fe694078c90 con 0x7fe6ac100780 2024-02-19T21:10:33.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.348+0000 7fe6a9f76700 10 client.49868.objecter ms_dispatch 0x7fe6ac001550 osd_map(429..429 src has 43..429) v4 2024-02-19T21:10:33.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.348+0000 7fe6a9f76700 3 client.49868.objecter handle_osd_map got epochs [429,429] > 0 2024-02-19T21:10:33.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.348+0000 7fe6a9f76700 3 client.49868.objecter handle_osd_map decoding full epoch 429 2024-02-19T21:10:33.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.348+0000 7fe6a9f76700 20 client.49868.objecter dump_active .. 0 homeless 2024-02-19T21:10:33.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.349+0000 7fe6a9f76700 10 client.49868.objecter ms_handle_connect 0x7fe698063c00 2024-02-19T21:10:33.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.354+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe69404c290 con 0x7fe6ac100780 2024-02-19T21:10:33.361 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:33.367 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:33.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:33.552+0000 7fe6b22ac700 1 -- 172.21.15.163:0/3315251923 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fe68c0051d0 con 0x7fe6ac100780 2024-02-19T21:10:33.746 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:10:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:10:33] "GET /metrics HTTP/1.1" 200 189600 "" "Prometheus/2.33.4" 2024-02-19T21:10:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:34.275+0000 7fe6a9f76700 1 -- 172.21.15.163:0/3315251923 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v430) v1 ==== 97+0+0 (unknown 4069813474 0 0) 0x7fe69404a6a0 con 0x7fe6ac100780 2024-02-19T21:10:34.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:34.277+0000 7fe6b22ac700 10 librados: watch_flush enter 2024-02-19T21:10:34.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:34.277+0000 7fe6b22ac700 10 librados: watch_flush exit 2024-02-19T21:10:34.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:34.277+0000 7fe6b22ac700 20 client.49868.objecter shutdown clearing up homeless session... 2024-02-19T21:10:34.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:34.277+0000 7fe6b22ac700 10 client.49868.objecter successfully canceled tick 2024-02-19T21:10:34.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:34.277+0000 7fe6b22ac700 1 -- 172.21.15.163:0/3315251923 >> v1:172.21.15.163:6800/1552514681 conn(0x7fe698063c00 legacy=0x7fe6980660c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:34.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:34.277+0000 7fe6b22ac700 1 -- 172.21.15.163:0/3315251923 >> v1:172.21.15.163:6790/0 conn(0x7fe6ac100780 legacy=0x7fe6ac0f9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:34.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:34.277+0000 7fe6b22ac700 1 -- 172.21.15.163:0/3315251923 shutdown_connections 2024-02-19T21:10:34.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:34.277+0000 7fe6b22ac700 1 -- 172.21.15.163:0/3315251923 >> 172.21.15.163:0/3315251923 conn(0x7fe6ac0f35a0 msgr2=0x7fe6ac107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:34.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:34.277+0000 7fe6b22ac700 1 -- 172.21.15.163:0/3315251923 shutdown_connections 2024-02-19T21:10:34.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:34.277+0000 7fe6b22ac700 1 -- 172.21.15.163:0/3315251923 wait complete. 2024-02-19T21:10:34.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:34.278+0000 7fe6b22ac700 1 librados: shutdown 2024-02-19T21:10:34.276 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:10:34.573 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:33.218624+0000 mgr.y (mgr.14550) 391 : cluster [DBG] pgmap v627: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-02-19T21:10:34.573 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:33.554269+0000 mon.c (mon.2) 391 : audit [INF] from='client.? v1:172.21.15.163:0/3315251923' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:10:34.573 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:33.554564+0000 mon.a (mon.0) 2551 : audit [INF] from='client.49868 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:10:34.573 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:34.265448+0000 mon.a (mon.0) 2552 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:10:34.573 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:34.273185+0000 mon.a (mon.0) 2553 : audit [INF] from='client.49868 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:10:34.573 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:34.273272+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e430: 8 total, 8 up, 7 in 2024-02-19T21:10:34.573 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:33.218624+0000 mgr.y (mgr.14550) 391 : cluster [DBG] pgmap v627: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-02-19T21:10:34.573 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:33.554269+0000 mon.c (mon.2) 391 : audit [INF] from='client.? v1:172.21.15.163:0/3315251923' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:10:34.573 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:33.554564+0000 mon.a (mon.0) 2551 : audit [INF] from='client.49868 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:10:34.573 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:34.265448+0000 mon.a (mon.0) 2552 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:10:34.574 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:34.273185+0000 mon.a (mon.0) 2553 : audit [INF] from='client.49868 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:10:34.574 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:34.273272+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e430: 8 total, 8 up, 7 in 2024-02-19T21:10:34.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:33.218624+0000 mgr.y (mgr.14550) 391 : cluster [DBG] pgmap v627: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-02-19T21:10:34.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:33.554269+0000 mon.c (mon.2) 391 : audit [INF] from='client.? v1:172.21.15.163:0/3315251923' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:10:34.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:33.554564+0000 mon.a (mon.0) 2551 : audit [INF] from='client.49868 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:10:34.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:34.265448+0000 mon.a (mon.0) 2552 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:10:34.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:34.273185+0000 mon.a (mon.0) 2553 : audit [INF] from='client.49868 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:10:34.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:34.273272+0000 mon.a (mon.0) 2554 : cluster [DBG] osdmap e430: 8 total, 8 up, 7 in 2024-02-19T21:10:34.988 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:10:34.988 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:10:35.318 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:10:35.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:35.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:10:35.422 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:35.452 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:35.505 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:35.508 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:35.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:35.609 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:35.638 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:35.761 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:35.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:35.865 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:35.901 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:35.923 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:35.925 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:35.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:36.027 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:36.057 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:36.256 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:36.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:36.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:36.360 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:10:36.391 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:36.411 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:10:36.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:36.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:10:36.515 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:36.544 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:36.565 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:36.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:36.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:36.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.607+0000 7f98a2907700 1 -- 172.21.15.163:0/3114317588 >> v1:172.21.15.163:6789/0 conn(0x7f989c1042c0 legacy=0x7f989c106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:36.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.607+0000 7f98a2907700 1 -- 172.21.15.163:0/3114317588 shutdown_connections 2024-02-19T21:10:36.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.607+0000 7f98a2907700 1 -- 172.21.15.163:0/3114317588 >> 172.21.15.163:0/3114317588 conn(0x7f989c0f35a0 msgr2=0x7f989c0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:36.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.607+0000 7f98a2907700 1 -- 172.21.15.163:0/3114317588 shutdown_connections 2024-02-19T21:10:36.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.607+0000 7f98a2907700 1 -- 172.21.15.163:0/3114317588 wait complete. 2024-02-19T21:10:36.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f98a2907700 1 librados: starting msgr at 2024-02-19T21:10:36.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f98a2907700 1 librados: starting objecter 2024-02-19T21:10:36.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f98a2907700 1 Processor -- start 2024-02-19T21:10:36.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f98a2907700 1 -- start start 2024-02-19T21:10:36.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f98a2907700 1 librados: setting wanted keys 2024-02-19T21:10:36.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f98a2907700 1 librados: calling monclient init 2024-02-19T21:10:36.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f98a2907700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f989c117330 con 0x7f989c0fd980 2024-02-19T21:10:36.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f98a2907700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f989c118510 con 0x7f989c100780 2024-02-19T21:10:36.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f98a2907700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f989c119710 con 0x7f989c1042c0 2024-02-19T21:10:36.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f989bfff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f989c0fd980 0x7f989c0f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:52866/0 (socket says 172.21.15.163:52866) 2024-02-19T21:10:36.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f989bfff700 1 -- 172.21.15.163:0/1980947580 learned_addr learned my addr 172.21.15.163:0/1980947580 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:36.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f98a0ea4700 1 --1- 172.21.15.163:0/1980947580 >> v1:172.21.15.163:6790/0 conn(0x7f989c1042c0 0x7f989c1159f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:42992/0 (socket says 172.21.15.163:42992) 2024-02-19T21:10:36.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f98997fa700 10 client.?.objecter ms_handle_connect 0x7f989c0fd980 2024-02-19T21:10:36.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f98997fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:36.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f98997fa700 10 client.?.objecter ms_handle_connect 0x7f989c1042c0 2024-02-19T21:10:36.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.608+0000 7f98997fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:36.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 10 client.?.objecter ms_handle_connect 0x7f989c100780 2024-02-19T21:10:36.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:36.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867282056 0 0) 0x7f989c119710 con 0x7f989c1042c0 2024-02-19T21:10:36.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98880038e0 con 0x7f989c1042c0 2024-02-19T21:10:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291755984 0 0) 0x7f989c117330 con 0x7f989c0fd980 2024-02-19T21:10:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f989c119710 con 0x7f989c0fd980 2024-02-19T21:10:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086606187 0 0) 0x7f989c118510 con 0x7f989c100780 2024-02-19T21:10:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f989c117330 con 0x7f989c100780 2024-02-19T21:10:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3302880477 0 0) 0x7f98880038e0 con 0x7f989c1042c0 2024-02-19T21:10:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f989c118510 con 0x7f989c1042c0 2024-02-19T21:10:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3834521710 0 0) 0x7f989c119710 con 0x7f989c0fd980 2024-02-19T21:10:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98880038e0 con 0x7f989c0fd980 2024-02-19T21:10:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9894001360 con 0x7f989c1042c0 2024-02-19T21:10:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9884001940 con 0x7f989c0fd980 2024-02-19T21:10:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3985118427 0 0) 0x7f989c117330 con 0x7f989c100780 2024-02-19T21:10:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f989c119710 con 0x7f989c100780 2024-02-19T21:10:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f988c0019f0 con 0x7f989c100780 2024-02-19T21:10:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2992319784 0 0) 0x7f989c118510 con 0x7f989c1042c0 2024-02-19T21:10:36.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 >> v1:172.21.15.193:6789/0 conn(0x7f989c100780 legacy=0x7f989c0fa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:36.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 >> v1:172.21.15.163:6789/0 conn(0x7f989c0fd980 legacy=0x7f989c0f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:36.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f989c11a910 con 0x7f989c1042c0 2024-02-19T21:10:36.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f989c119900 con 0x7f989c1042c0 2024-02-19T21:10:36.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98a2907700 10 client.49874.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:36.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f989c119df0 con 0x7f989c1042c0 2024-02-19T21:10:36.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98a2907700 1 librados: init done 2024-02-19T21:10:36.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9894002ab0 con 0x7f989c1042c0 2024-02-19T21:10:36.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.609+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9894003ab0 con 0x7f989c1042c0 2024-02-19T21:10:36.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.610+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f987c004f00 con 0x7f989c1042c0 2024-02-19T21:10:36.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.610+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9894001c40 con 0x7f989c1042c0 2024-02-19T21:10:36.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(431..431 src has 43..431) v4 ==== 7946+0+0 (unknown 547643217 0 0) 0x7f9894079330 con 0x7f989c1042c0 2024-02-19T21:10:36.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 osd_map(431..431 src has 43..431) v4 2024-02-19T21:10:36.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98997fa700 3 client.49874.objecter handle_osd_map got epochs [431,431] > 0 2024-02-19T21:10:36.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98997fa700 3 client.49874.objecter handle_osd_map decoding full epoch 431 2024-02-19T21:10:36.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98997fa700 20 client.49874.objecter dump_active .. 0 homeless 2024-02-19T21:10:36.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v431) v1 ==== 54+0+16 (unknown 4022420935 0 2420000192) 0x7f989407b730 con 0x7f989c1042c0 2024-02-19T21:10:36.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98997fa700 10 client.49874.objecter ms_handle_connect 0x7f9888063c60 2024-02-19T21:10:36.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98a2907700 10 client.49874.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:36.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 1 2024-02-19T21:10:36.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c0051a0 osd=0 3 2024-02-19T21:10:36.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c0051a0 osd=0 3 2024-02-19T21:10:36.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c0051a0 osd=0 3 2024-02-19T21:10:36.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 1 2024-02-19T21:10:36.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c0051a0 osd=0 3 2024-02-19T21:10:36.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 0 1 2024-02-19T21:10:36.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c0051a0 osd=0 4 2024-02-19T21:10:36.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98a2907700 10 client.49874.objecter _send_command 1 2024-02-19T21:10:36.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.611+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f987c004950 con 0x7f987c004530 2024-02-19T21:10:36.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.612+0000 7f98997fa700 10 client.49874.objecter ms_handle_connect 0x7f987c004530 2024-02-19T21:10:36.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.612+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f987c004950 con 0x7f987c004530 2024-02-19T21:10:36.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.612+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:10:36.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.612+0000 7f98997fa700 10 client.49874.objecter _finish_command 1 = system:0 2024-02-19T21:10:36.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.612+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c0051a0 osd=0 4 2024-02-19T21:10:36.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.612+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 0 1 2024-02-19T21:10:36.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.627+0000 7f98a2907700 10 client.49874.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:36.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.627+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 2 2024-02-19T21:10:36.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.627+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c0051a0 osd=0 3 2024-02-19T21:10:36.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.627+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c0051a0 osd=0 3 2024-02-19T21:10:36.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.627+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c0051a0 osd=0 3 2024-02-19T21:10:36.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.627+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 2 2024-02-19T21:10:36.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.627+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c0051a0 osd=0 3 2024-02-19T21:10:36.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.627+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 0 2 2024-02-19T21:10:36.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.627+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c0051a0 osd=0 4 2024-02-19T21:10:36.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.627+0000 7f98a2907700 10 client.49874.objecter _send_command 2 2024-02-19T21:10:36.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.627+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f987c004d30 con 0x7f987c004530 2024-02-19T21:10:36.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.628+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f987c004d30 con 0x7f987c004530 2024-02-19T21:10:36.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.628+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:36.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.628+0000 7f98997fa700 10 client.49874.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.628+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c0051a0 osd=0 4 2024-02-19T21:10:36.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.628+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 0 2 2024-02-19T21:10:36.626 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:10:36.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.628+0000 7f98a2907700 10 client.49874.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:36.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.628+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 3 2024-02-19T21:10:36.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.629+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c003fe0 osd=1 3 2024-02-19T21:10:36.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.629+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c003fe0 osd=1 3 2024-02-19T21:10:36.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.629+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c003fe0 osd=1 3 2024-02-19T21:10:36.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.629+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 3 2024-02-19T21:10:36.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.629+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c003fe0 osd=1 3 2024-02-19T21:10:36.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.629+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 1 3 2024-02-19T21:10:36.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.629+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c003fe0 osd=1 4 2024-02-19T21:10:36.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.629+0000 7f98a2907700 10 client.49874.objecter _send_command 3 2024-02-19T21:10:36.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.629+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f987c00c000 con 0x7f987c004150 2024-02-19T21:10:36.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.629+0000 7f98997fa700 10 client.49874.objecter ms_handle_connect 0x7f987c004150 2024-02-19T21:10:36.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.629+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f987c00c000 con 0x7f987c004150 2024-02-19T21:10:36.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.629+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:10:36.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.629+0000 7f98997fa700 10 client.49874.objecter _finish_command 3 = system:0 2024-02-19T21:10:36.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.629+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c003fe0 osd=1 4 2024-02-19T21:10:36.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.629+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 1 3 2024-02-19T21:10:36.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.644+0000 7f98a2907700 10 client.49874.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:36.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.644+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 4 2024-02-19T21:10:36.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.644+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c003fe0 osd=1 3 2024-02-19T21:10:36.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.644+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c003fe0 osd=1 3 2024-02-19T21:10:36.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.644+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c003fe0 osd=1 3 2024-02-19T21:10:36.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.644+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 4 2024-02-19T21:10:36.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.644+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c003fe0 osd=1 3 2024-02-19T21:10:36.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.644+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 1 4 2024-02-19T21:10:36.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.644+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c003fe0 osd=1 4 2024-02-19T21:10:36.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.644+0000 7f98a2907700 10 client.49874.objecter _send_command 4 2024-02-19T21:10:36.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.644+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f987c0018b0 con 0x7f987c004150 2024-02-19T21:10:36.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f987c0018b0 con 0x7f987c004150 2024-02-19T21:10:36.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:36.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98997fa700 10 client.49874.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c003fe0 osd=1 4 2024-02-19T21:10:36.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 1 4 2024-02-19T21:10:36.643 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:10:36.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98a2907700 10 client.49874.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:36.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 5 2024-02-19T21:10:36.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c003630 osd=2 3 2024-02-19T21:10:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c003630 osd=2 3 2024-02-19T21:10:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c003630 osd=2 3 2024-02-19T21:10:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 5 2024-02-19T21:10:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c003630 osd=2 3 2024-02-19T21:10:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 2 5 2024-02-19T21:10:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c003630 osd=2 4 2024-02-19T21:10:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98a2907700 10 client.49874.objecter _send_command 5 2024-02-19T21:10:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.645+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f987c00de80 con 0x7f987c00d2f0 2024-02-19T21:10:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.646+0000 7f98997fa700 10 client.49874.objecter ms_handle_connect 0x7f987c00d2f0 2024-02-19T21:10:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.646+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f987c00de80 con 0x7f987c00d2f0 2024-02-19T21:10:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.646+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:10:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.646+0000 7f98997fa700 10 client.49874.objecter _finish_command 5 = system:0 2024-02-19T21:10:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.646+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c003630 osd=2 4 2024-02-19T21:10:36.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.646+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 2 5 2024-02-19T21:10:36.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.660+0000 7f98a2907700 10 client.49874.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:36.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.660+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 6 2024-02-19T21:10:36.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.660+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c003630 osd=2 3 2024-02-19T21:10:36.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.660+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c003630 osd=2 3 2024-02-19T21:10:36.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.660+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c003630 osd=2 3 2024-02-19T21:10:36.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.660+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 6 2024-02-19T21:10:36.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.660+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c003630 osd=2 3 2024-02-19T21:10:36.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.660+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 2 6 2024-02-19T21:10:36.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.660+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c003630 osd=2 4 2024-02-19T21:10:36.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.660+0000 7f98a2907700 10 client.49874.objecter _send_command 6 2024-02-19T21:10:36.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.660+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f987c0030a0 con 0x7f987c00d2f0 2024-02-19T21:10:36.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f987c0030a0 con 0x7f987c00d2f0 2024-02-19T21:10:36.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:36.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98997fa700 10 client.49874.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c003630 osd=2 4 2024-02-19T21:10:36.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 2 6 2024-02-19T21:10:36.660 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:10:36.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98a2907700 10 client.49874.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:36.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 7 2024-02-19T21:10:36.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c0034b0 osd=3 3 2024-02-19T21:10:36.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c0034b0 osd=3 3 2024-02-19T21:10:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c0034b0 osd=3 3 2024-02-19T21:10:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 7 2024-02-19T21:10:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c0034b0 osd=3 3 2024-02-19T21:10:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 3 7 2024-02-19T21:10:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c0034b0 osd=3 4 2024-02-19T21:10:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98a2907700 10 client.49874.objecter _send_command 7 2024-02-19T21:10:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f987c015aa0 con 0x7f987c011e50 2024-02-19T21:10:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.662+0000 7f98997fa700 10 client.49874.objecter ms_handle_connect 0x7f987c011e50 2024-02-19T21:10:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.663+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f987c015aa0 con 0x7f987c011e50 2024-02-19T21:10:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.663+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:10:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.663+0000 7f98997fa700 10 client.49874.objecter _finish_command 7 = system:0 2024-02-19T21:10:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.663+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c0034b0 osd=3 4 2024-02-19T21:10:36.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.663+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 3 7 2024-02-19T21:10:36.669 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:36.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.677+0000 7f98a2907700 10 client.49874.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:36.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.677+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 8 2024-02-19T21:10:36.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.677+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c0034b0 osd=3 3 2024-02-19T21:10:36.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.677+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c0034b0 osd=3 3 2024-02-19T21:10:36.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.677+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c0034b0 osd=3 3 2024-02-19T21:10:36.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.677+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 8 2024-02-19T21:10:36.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.677+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c0034b0 osd=3 3 2024-02-19T21:10:36.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.677+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 3 8 2024-02-19T21:10:36.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.677+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c0034b0 osd=3 4 2024-02-19T21:10:36.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.677+0000 7f98a2907700 10 client.49874.objecter _send_command 8 2024-02-19T21:10:36.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.677+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f987c00e230 con 0x7f987c011e50 2024-02-19T21:10:36.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f987c00e230 con 0x7f987c011e50 2024-02-19T21:10:36.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:36.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98997fa700 10 client.49874.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c0034b0 osd=3 4 2024-02-19T21:10:36.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 3 8 2024-02-19T21:10:36.677 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:10:36.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98a2907700 10 client.49874.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:36.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 9 2024-02-19T21:10:36.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c00e230 osd=4 3 2024-02-19T21:10:36.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c00e230 osd=4 3 2024-02-19T21:10:36.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c00e230 osd=4 3 2024-02-19T21:10:36.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 9 2024-02-19T21:10:36.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c00e230 osd=4 3 2024-02-19T21:10:36.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 4 9 2024-02-19T21:10:36.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c00e230 osd=4 4 2024-02-19T21:10:36.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98a2907700 10 client.49874.objecter _send_command 9 2024-02-19T21:10:36.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.679+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f987c0161b0 con 0x7f987c00ce50 2024-02-19T21:10:36.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.680+0000 7f98997fa700 10 client.49874.objecter ms_handle_connect 0x7f987c00ce50 2024-02-19T21:10:36.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.680+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f987c0161b0 con 0x7f987c00ce50 2024-02-19T21:10:36.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.680+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:10:36.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.680+0000 7f98997fa700 10 client.49874.objecter _finish_command 9 = system:0 2024-02-19T21:10:36.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.680+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c00e230 osd=4 4 2024-02-19T21:10:36.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.680+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 4 9 2024-02-19T21:10:36.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.695+0000 7f98a2907700 10 client.49874.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:36.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.695+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 10 2024-02-19T21:10:36.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.695+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c00e230 osd=4 3 2024-02-19T21:10:36.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.695+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c00e230 osd=4 3 2024-02-19T21:10:36.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.695+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c00e230 osd=4 3 2024-02-19T21:10:36.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.695+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 10 2024-02-19T21:10:36.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.695+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c00e230 osd=4 3 2024-02-19T21:10:36.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.695+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 4 10 2024-02-19T21:10:36.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.695+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c00e230 osd=4 4 2024-02-19T21:10:36.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.695+0000 7f98a2907700 10 client.49874.objecter _send_command 10 2024-02-19T21:10:36.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.695+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f987c0161b0 con 0x7f987c00ce50 2024-02-19T21:10:36.695 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:35.219494+0000 mgr.y (mgr.14550) 392 : cluster [DBG] pgmap v629: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.5 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-02-19T21:10:36.695 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:35.832740+0000 osd.3 (osd.3) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:10:36.695 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:35.832942+0000 osd.3 (osd.3) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:10:36.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f987c0161b0 con 0x7f987c00ce50 2024-02-19T21:10:36.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:36.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98997fa700 10 client.49874.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c00e230 osd=4 4 2024-02-19T21:10:36.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 4 10 2024-02-19T21:10:36.697 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:10:36.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98a2907700 10 client.49874.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 11 2024-02-19T21:10:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c0161b0 osd=5 3 2024-02-19T21:10:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c0161b0 osd=5 3 2024-02-19T21:10:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c0161b0 osd=5 3 2024-02-19T21:10:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 11 2024-02-19T21:10:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c0161b0 osd=5 3 2024-02-19T21:10:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 5 11 2024-02-19T21:10:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c0161b0 osd=5 4 2024-02-19T21:10:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98a2907700 10 client.49874.objecter _send_command 11 2024-02-19T21:10:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f987c016e00 con 0x7f987c016860 2024-02-19T21:10:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.697+0000 7f98997fa700 10 client.49874.objecter ms_handle_connect 0x7f987c016860 2024-02-19T21:10:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.698+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f987c016e00 con 0x7f987c016860 2024-02-19T21:10:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.698+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:10:36.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.698+0000 7f98997fa700 10 client.49874.objecter _finish_command 11 = system:0 2024-02-19T21:10:36.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.698+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c0161b0 osd=5 4 2024-02-19T21:10:36.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.698+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 5 11 2024-02-19T21:10:36.699 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:36.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.712+0000 7f98a2907700 10 client.49874.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:36.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.712+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 12 2024-02-19T21:10:36.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.712+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c0161b0 osd=5 3 2024-02-19T21:10:36.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.712+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c0161b0 osd=5 3 2024-02-19T21:10:36.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.712+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c0161b0 osd=5 3 2024-02-19T21:10:36.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.712+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 12 2024-02-19T21:10:36.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.712+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c0161b0 osd=5 3 2024-02-19T21:10:36.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.712+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 5 12 2024-02-19T21:10:36.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.712+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c0161b0 osd=5 4 2024-02-19T21:10:36.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.712+0000 7f98a2907700 10 client.49874.objecter _send_command 12 2024-02-19T21:10:36.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.712+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f987c016e00 con 0x7f987c016860 2024-02-19T21:10:36.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f987c016e00 con 0x7f987c016860 2024-02-19T21:10:36.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:36.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98997fa700 10 client.49874.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.712 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:10:36.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c0161b0 osd=5 4 2024-02-19T21:10:36.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 5 12 2024-02-19T21:10:36.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98a2907700 10 client.49874.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:36.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 13 2024-02-19T21:10:36.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c016e00 osd=6 3 2024-02-19T21:10:36.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c016e00 osd=6 3 2024-02-19T21:10:36.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c016e00 osd=6 3 2024-02-19T21:10:36.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 13 2024-02-19T21:10:36.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c016e00 osd=6 3 2024-02-19T21:10:36.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 6 13 2024-02-19T21:10:36.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c016e00 osd=6 4 2024-02-19T21:10:36.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98a2907700 10 client.49874.objecter _send_command 13 2024-02-19T21:10:36.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f987c022d90 con 0x7f987c01f0a0 2024-02-19T21:10:36.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.715+0000 7f98997fa700 10 client.49874.objecter ms_handle_connect 0x7f987c01f0a0 2024-02-19T21:10:36.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.716+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f987c022d90 con 0x7f987c01f0a0 2024-02-19T21:10:36.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.716+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:10:36.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.716+0000 7f98997fa700 10 client.49874.objecter _finish_command 13 = system:0 2024-02-19T21:10:36.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.716+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c016e00 osd=6 4 2024-02-19T21:10:36.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.716+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 6 13 2024-02-19T21:10:36.716 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:36.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:36.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:36.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.731+0000 7f98a2907700 10 client.49874.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:36.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.731+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 14 2024-02-19T21:10:36.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.731+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c016e00 osd=6 3 2024-02-19T21:10:36.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.731+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c016e00 osd=6 3 2024-02-19T21:10:36.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.731+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c016e00 osd=6 3 2024-02-19T21:10:36.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.731+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 14 2024-02-19T21:10:36.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.731+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c016e00 osd=6 3 2024-02-19T21:10:36.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.731+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 6 14 2024-02-19T21:10:36.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.731+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c016e00 osd=6 4 2024-02-19T21:10:36.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.731+0000 7f98a2907700 10 client.49874.objecter _send_command 14 2024-02-19T21:10:36.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.731+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f987c024540 con 0x7f987c01f0a0 2024-02-19T21:10:36.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.732+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f987c024540 con 0x7f987c01f0a0 2024-02-19T21:10:36.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.732+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:36.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.732+0000 7f98997fa700 10 client.49874.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.732+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c016e00 osd=6 4 2024-02-19T21:10:36.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.732+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 6 14 2024-02-19T21:10:36.730 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:10:36.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.733+0000 7f98a2907700 10 client.49874.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:36.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.733+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 15 2024-02-19T21:10:36.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.733+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c024540 osd=7 3 2024-02-19T21:10:36.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.733+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c024540 osd=7 3 2024-02-19T21:10:36.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.733+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c024540 osd=7 3 2024-02-19T21:10:36.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.733+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 15 2024-02-19T21:10:36.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.733+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c024540 osd=7 3 2024-02-19T21:10:36.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.733+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 7 15 2024-02-19T21:10:36.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.733+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c024540 osd=7 4 2024-02-19T21:10:36.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.733+0000 7f98a2907700 10 client.49874.objecter _send_command 15 2024-02-19T21:10:36.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.733+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f987c025100 con 0x7f987c024bc0 2024-02-19T21:10:36.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.733+0000 7f98997fa700 10 client.49874.objecter ms_handle_connect 0x7f987c024bc0 2024-02-19T21:10:36.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.734+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f987c025100 con 0x7f987c024bc0 2024-02-19T21:10:36.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.734+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:10:36.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.734+0000 7f98997fa700 10 client.49874.objecter _finish_command 15 = system:0 2024-02-19T21:10:36.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.734+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c024540 osd=7 4 2024-02-19T21:10:36.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.734+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 7 15 2024-02-19T21:10:36.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.748+0000 7f98a2907700 10 client.49874.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:36.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.748+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign -1 16 2024-02-19T21:10:36.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.748+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c024540 osd=7 3 2024-02-19T21:10:36.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.748+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c024540 osd=7 3 2024-02-19T21:10:36.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.748+0000 7f98a2907700 20 client.49874.objecter _get_session s=0x7f987c024540 osd=7 3 2024-02-19T21:10:36.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.748+0000 7f98a2907700 15 client.49874.objecter _session_command_op_remove -1 16 2024-02-19T21:10:36.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.748+0000 7f98a2907700 20 client.49874.objecter get_session s=0x7f987c024540 osd=7 3 2024-02-19T21:10:36.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.748+0000 7f98a2907700 15 client.49874.objecter _session_command_op_assign 7 16 2024-02-19T21:10:36.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.748+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c024540 osd=7 4 2024-02-19T21:10:36.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.748+0000 7f98a2907700 10 client.49874.objecter _send_command 16 2024-02-19T21:10:36.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.748+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f987c025100 con 0x7f987c024bc0 2024-02-19T21:10:36.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98997fa700 1 -- 172.21.15.163:0/1980947580 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f987c025100 con 0x7f987c024bc0 2024-02-19T21:10:36.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98997fa700 10 client.49874.objecter ms_dispatch 0x7f989c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:36.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98997fa700 10 client.49874.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98997fa700 20 client.49874.objecter put_session s=0x7f987c024540 osd=7 4 2024-02-19T21:10:36.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98997fa700 15 client.49874.objecter _session_command_op_remove 7 16 2024-02-19T21:10:36.748 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:10:36.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 10 librados: watch_flush enter 2024-02-19T21:10:36.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 10 librados: watch_flush exit 2024-02-19T21:10:36.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 10 client.49874.objecter close_session for osd.0 2024-02-19T21:10:36.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 >> v1:172.21.15.163:6801/1559418115 conn(0x7f987c004530 legacy=0x7f987c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:36.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c0051a0 osd=0 1 2024-02-19T21:10:36.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 10 client.49874.objecter close_session for osd.1 2024-02-19T21:10:36.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 >> v1:172.21.15.163:6805/3125047432 conn(0x7f987c004150 legacy=0x7f987c00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:36.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c003fe0 osd=1 1 2024-02-19T21:10:36.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 10 client.49874.objecter close_session for osd.2 2024-02-19T21:10:36.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 >> v1:172.21.15.163:6809/1563158487 conn(0x7f987c00d2f0 legacy=0x7f987c00d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:36.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c003630 osd=2 1 2024-02-19T21:10:36.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 10 client.49874.objecter close_session for osd.3 2024-02-19T21:10:36.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 >> v1:172.21.15.163:6813/2112279769 conn(0x7f987c011e50 legacy=0x7f987c014240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:36.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c0034b0 osd=3 1 2024-02-19T21:10:36.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 10 client.49874.objecter close_session for osd.4 2024-02-19T21:10:36.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 >> v1:172.21.15.193:6800/3129716965 conn(0x7f987c00ce50 legacy=0x7f987c015aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:36.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a0ea4700 1 -- 172.21.15.163:0/1980947580 reap_dead start 2024-02-19T21:10:36.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a0ea4700 1 -- 172.21.15.163:0/1980947580 reap_dead start 2024-02-19T21:10:36.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c00e230 osd=4 1 2024-02-19T21:10:36.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 10 client.49874.objecter close_session for osd.5 2024-02-19T21:10:36.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 >> v1:172.21.15.193:6804/805161991 conn(0x7f987c016860 legacy=0x7f987c01d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:36.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c0161b0 osd=5 1 2024-02-19T21:10:36.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 10 client.49874.objecter close_session for osd.6 2024-02-19T21:10:36.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 >> v1:172.21.15.193:6808/787604615 conn(0x7f987c01f0a0 legacy=0x7f987c021510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:36.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c016e00 osd=6 1 2024-02-19T21:10:36.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 10 client.49874.objecter close_session for osd.7 2024-02-19T21:10:36.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 >> v1:172.21.15.193:6812/157927432 conn(0x7f987c024bc0 legacy=0x7f987c022d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:36.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 20 client.49874.objecter put_session s=0x7f987c024540 osd=7 1 2024-02-19T21:10:36.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 20 client.49874.objecter shutdown clearing up homeless session... 2024-02-19T21:10:36.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 10 client.49874.objecter successfully canceled tick 2024-02-19T21:10:36.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 >> v1:172.21.15.163:6800/1552514681 conn(0x7f9888063c60 legacy=0x7f9888066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:36.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 >> v1:172.21.15.163:6790/0 conn(0x7f989c1042c0 legacy=0x7f989c1159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:36.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.750+0000 7f98a0ea4700 1 -- 172.21.15.163:0/1980947580 reap_dead start 2024-02-19T21:10:36.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.751+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 shutdown_connections 2024-02-19T21:10:36.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.751+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 >> 172.21.15.163:0/1980947580 conn(0x7f989c0f35a0 msgr2=0x7f989c107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:36.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.751+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 shutdown_connections 2024-02-19T21:10:36.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.751+0000 7f98a2907700 1 -- 172.21.15.163:0/1980947580 wait complete. 2024-02-19T21:10:36.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:36.751+0000 7f98a2907700 1 librados: shutdown 2024-02-19T21:10:36.752 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.752 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.752 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.752 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.752 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.753 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.753 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.753 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:10:36.777 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:35.219494+0000 mgr.y (mgr.14550) 392 : cluster [DBG] pgmap v629: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.5 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-02-19T21:10:36.777 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:35.832740+0000 osd.3 (osd.3) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:10:36.777 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:35.832942+0000 osd.3 (osd.3) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:10:36.777 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:35.219494+0000 mgr.y (mgr.14550) 392 : cluster [DBG] pgmap v629: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 3.5 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2024-02-19T21:10:36.777 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:35.832740+0000 osd.3 (osd.3) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:10:36.777 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:35.832942+0000 osd.3 (osd.3) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:10:36.819 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:36.849 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:37.049 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:37.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:37.154 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:37.182 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:37.201 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:37.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:37.305 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:37.331 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:37.726 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:36.534557+0000 mon.a (mon.0) 2555 : cluster [DBG] osdmap e431: 8 total, 8 up, 7 in 2024-02-19T21:10:37.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:36.611345+0000 mon.c (mon.2) 392 : audit [DBG] from='client.? v1:172.21.15.163:0/1980947580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:37.727 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:36.534557+0000 mon.a (mon.0) 2555 : cluster [DBG] osdmap e431: 8 total, 8 up, 7 in 2024-02-19T21:10:37.727 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:36.611345+0000 mon.c (mon.2) 392 : audit [DBG] from='client.? v1:172.21.15.163:0/1980947580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:37.819 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:10:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:10:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:10:37.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:36.534557+0000 mon.a (mon.0) 2555 : cluster [DBG] osdmap e431: 8 total, 8 up, 7 in 2024-02-19T21:10:37.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:36.611345+0000 mon.c (mon.2) 392 : audit [DBG] from='client.? v1:172.21.15.163:0/1980947580' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:38.581 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:38.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:38.595 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-19T21:10:38.686 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:38.718 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:38.739 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:38.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:38.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:38.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:37.219968+0000 mgr.y (mgr.14550) 393 : cluster [DBG] pgmap v631: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.3 KiB/s rd, 341 B/s wr, 3 op/s 2024-02-19T21:10:38.841 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:37.219968+0000 mgr.y (mgr.14550) 393 : cluster [DBG] pgmap v631: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.3 KiB/s rd, 341 B/s wr, 3 op/s 2024-02-19T21:10:38.841 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:37.219968+0000 mgr.y (mgr.14550) 393 : cluster [DBG] pgmap v631: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.3 KiB/s rd, 341 B/s wr, 3 op/s 2024-02-19T21:10:38.844 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:38.874 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:38.894 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:38.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:38.998 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:39.028 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:39.046 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:39.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:39.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:39.058 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:10:39.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.148+0000 7f3bb84c0700 1 -- 172.21.15.163:0/914964887 >> v1:172.21.15.163:6789/0 conn(0x7f3bb006bbe0 legacy=0x7f3bb0104d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:39.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.148+0000 7f3bb84c0700 1 -- 172.21.15.163:0/914964887 shutdown_connections 2024-02-19T21:10:39.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.148+0000 7f3bb84c0700 1 -- 172.21.15.163:0/914964887 >> 172.21.15.163:0/914964887 conn(0x7f3bb00f35a0 msgr2=0x7f3bb00f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:39.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.148+0000 7f3bb84c0700 1 -- 172.21.15.163:0/914964887 shutdown_connections 2024-02-19T21:10:39.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.148+0000 7f3bb84c0700 1 -- 172.21.15.163:0/914964887 wait complete. 2024-02-19T21:10:39.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.148+0000 7f3bb84c0700 1 librados: starting msgr at 2024-02-19T21:10:39.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.148+0000 7f3bb84c0700 1 librados: starting objecter 2024-02-19T21:10:39.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.148+0000 7f3bb84c0700 1 Processor -- start 2024-02-19T21:10:39.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3bb84c0700 1 -- start start 2024-02-19T21:10:39.148 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3bb84c0700 1 librados: setting wanted keys 2024-02-19T21:10:39.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3bb84c0700 1 librados: calling monclient init 2024-02-19T21:10:39.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3bb84c0700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3bb01193b0 con 0x7f3bb006bbe0 2024-02-19T21:10:39.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3bb84c0700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3bb011a590 con 0x7f3bb0106510 2024-02-19T21:10:39.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3bb84c0700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3bb011b790 con 0x7f3bb006b0f0 2024-02-19T21:10:39.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3bb625c700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f3bb006b0f0 0x7f3bb01026c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:43008/0 (socket says 172.21.15.163:43008) 2024-02-19T21:10:39.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3bb625c700 1 -- 172.21.15.163:0/1037603649 learned_addr learned my addr 172.21.15.163:0/1037603649 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:39.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3ba77fe700 10 client.?.objecter ms_handle_connect 0x7f3bb006b0f0 2024-02-19T21:10:39.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3ba77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:39.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3ba77fe700 10 client.?.objecter ms_handle_connect 0x7f3bb006bbe0 2024-02-19T21:10:39.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3ba77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:39.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351102744 0 0) 0x7f3bb011b790 con 0x7f3bb006b0f0 2024-02-19T21:10:39.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b9c0038e0 con 0x7f3bb006b0f0 2024-02-19T21:10:39.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3ba77fe700 10 client.?.objecter ms_handle_connect 0x7f3bb0106510 2024-02-19T21:10:39.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3ba77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:39.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476172090 0 0) 0x7f3bb011a590 con 0x7f3bb0106510 2024-02-19T21:10:39.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.149+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bb011b790 con 0x7f3bb0106510 2024-02-19T21:10:39.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 439789332 0 0) 0x7f3b9c0038e0 con 0x7f3bb006b0f0 2024-02-19T21:10:39.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bb011a590 con 0x7f3bb006b0f0 2024-02-19T21:10:39.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3bac001990 con 0x7f3bb006b0f0 2024-02-19T21:10:39.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1731989157 0 0) 0x7f3bb011b790 con 0x7f3bb0106510 2024-02-19T21:10:39.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b9c0038e0 con 0x7f3bb0106510 2024-02-19T21:10:39.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3ba8001d70 con 0x7f3bb0106510 2024-02-19T21:10:39.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2163769185 0 0) 0x7f3bb011a590 con 0x7f3bb006b0f0 2024-02-19T21:10:39.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 >> v1:172.21.15.193:6789/0 conn(0x7f3bb0106510 legacy=0x7f3bb0117b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:39.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 >> v1:172.21.15.163:6789/0 conn(0x7f3bb006bbe0 legacy=0x7f3bb0102dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:39.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb011c990 con 0x7f3bb006b0f0 2024-02-19T21:10:39.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3bb84c0700 1 -- 172.21.15.163:0/1037603649 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb01195a0 con 0x7f3bb006b0f0 2024-02-19T21:10:39.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3bb84c0700 10 client.49880.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:39.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3bb84c0700 1 -- 172.21.15.163:0/1037603649 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb0119a10 con 0x7f3bb006b0f0 2024-02-19T21:10:39.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3bb84c0700 1 librados: init done 2024-02-19T21:10:39.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3bac001d20 con 0x7f3bb006b0f0 2024-02-19T21:10:39.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.150+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3bac005b50 con 0x7f3bb006b0f0 2024-02-19T21:10:39.151 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:39.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.151+0000 7f3bb84c0700 1 -- 172.21.15.163:0/1037603649 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b94004f20 con 0x7f3bb006b0f0 2024-02-19T21:10:39.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.151+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f3bac002b30 con 0x7f3bb006b0f0 2024-02-19T21:10:39.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.152+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(431..431 src has 43..431) v4 ==== 7946+0+0 (unknown 547643217 0 0) 0x7f3bac078d30 con 0x7f3bb006b0f0 2024-02-19T21:10:39.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.152+0000 7f3ba77fe700 10 client.49880.objecter ms_dispatch 0x7f3bb0001550 osd_map(431..431 src has 43..431) v4 2024-02-19T21:10:39.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.152+0000 7f3ba77fe700 3 client.49880.objecter handle_osd_map got epochs [431,431] > 0 2024-02-19T21:10:39.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.152+0000 7f3ba77fe700 3 client.49880.objecter handle_osd_map decoding full epoch 431 2024-02-19T21:10:39.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.152+0000 7f3ba77fe700 20 client.49880.objecter dump_active .. 0 homeless 2024-02-19T21:10:39.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.152+0000 7f3ba77fe700 10 client.49880.objecter ms_handle_connect 0x7f3b9c063c10 2024-02-19T21:10:39.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.157+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3bac04c1f0 con 0x7f3bb006b0f0 2024-02-19T21:10:39.185 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:39.194 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:39.206 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:39.209 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:39.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:39.311 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:39.340 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:39.361+0000 7f3bb84c0700 1 -- 172.21.15.163:0/1037603649 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd in", "ids": ["4"]} v 0) v1 -- 0x7f3b940051d0 con 0x7f3bb006b0f0 2024-02-19T21:10:39.359 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:39.362 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:39.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:39.458 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:10:39.362+0000 7fb606608700 -1 mon.a@0(leader).osd e432 definitely_dead 0 2024-02-19T21:10:39.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:39.494 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:39.515 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:39.518 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:39.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:39.619 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:39.650 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:39.670 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:39.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:39.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:39.746 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:10:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:38.943836+0000 mon.b (mon.1) 586 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 1]}]: dispatch 2024-02-19T21:10:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:38.944060+0000 mon.b (mon.1) 587 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-19T21:10:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:38.944268+0000 mon.b (mon.1) 588 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]: dispatch 2024-02-19T21:10:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:38.944273+0000 mon.a (mon.0) 2556 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 1]}]: dispatch 2024-02-19T21:10:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:38.944442+0000 mon.b (mon.1) 589 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 2]}]: dispatch 2024-02-19T21:10:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:38.944614+0000 mon.b (mon.1) 590 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]: dispatch 2024-02-19T21:10:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:38.944712+0000 mon.a (mon.0) 2557 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-19T21:10:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:38.944784+0000 mon.b (mon.1) 591 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1", "id": [1, 3]}]: dispatch 2024-02-19T21:10:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:38.944932+0000 mon.a (mon.0) 2558 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]: dispatch 2024-02-19T21:10:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:38.944952+0000 mon.b (mon.1) 592 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "61.4", "id": [5, 3]}]: dispatch 2024-02-19T21:10:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:38.945205+0000 mon.a (mon.0) 2559 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 2]}]: dispatch 2024-02-19T21:10:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:38.945386+0000 mon.a (mon.0) 2560 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]: dispatch 2024-02-19T21:10:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:38.945553+0000 mon.a (mon.0) 2561 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1", "id": [1, 3]}]: dispatch 2024-02-19T21:10:39.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:38.943836+0000 mon.b (mon.1) 586 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 1]}]: dispatch 2024-02-19T21:10:39.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:38.944060+0000 mon.b (mon.1) 587 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-19T21:10:39.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:38.944268+0000 mon.b (mon.1) 588 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]: dispatch 2024-02-19T21:10:39.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:38.944273+0000 mon.a (mon.0) 2556 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 1]}]: dispatch 2024-02-19T21:10:39.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:38.944442+0000 mon.b (mon.1) 589 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 2]}]: dispatch 2024-02-19T21:10:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:38.944614+0000 mon.b (mon.1) 590 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]: dispatch 2024-02-19T21:10:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:38.944712+0000 mon.a (mon.0) 2557 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-19T21:10:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:38.944784+0000 mon.b (mon.1) 591 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1", "id": [1, 3]}]: dispatch 2024-02-19T21:10:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:38.944932+0000 mon.a (mon.0) 2558 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]: dispatch 2024-02-19T21:10:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:38.944952+0000 mon.b (mon.1) 592 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "61.4", "id": [5, 3]}]: dispatch 2024-02-19T21:10:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:38.945205+0000 mon.a (mon.0) 2559 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 2]}]: dispatch 2024-02-19T21:10:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:38.945386+0000 mon.a (mon.0) 2560 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]: dispatch 2024-02-19T21:10:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:38.945553+0000 mon.a (mon.0) 2561 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1", "id": [1, 3]}]: dispatch 2024-02-19T21:10:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:38.945712+0000 mon.a (mon.0) 2562 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "61.4", "id": [5, 3]}]: dispatch 2024-02-19T21:10:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:39.271521+0000 mon.a (mon.0) 2563 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 1]}]': finished 2024-02-19T21:10:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:39.271607+0000 mon.a (mon.0) 2564 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-02-19T21:10:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:39.271669+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]': finished 2024-02-19T21:10:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:39.271729+0000 mon.a (mon.0) 2566 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 2]}]': finished 2024-02-19T21:10:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:39.271788+0000 mon.a (mon.0) 2567 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]': finished 2024-02-19T21:10:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:39.271847+0000 mon.a (mon.0) 2568 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1", "id": [1, 3]}]': finished 2024-02-19T21:10:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:39.271910+0000 mon.a (mon.0) 2569 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "61.4", "id": [5, 3]}]': finished 2024-02-19T21:10:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:39.271965+0000 mon.a (mon.0) 2570 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2024-02-19T21:10:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:39.361852+0000 mon.c (mon.2) 393 : audit [DBG] from='client.? v1:172.21.15.163:0/1037603649' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-19T21:10:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:39.362269+0000 mon.a (mon.0) 2571 : audit [DBG] from='client.49880 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-19T21:10:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:38.945712+0000 mon.a (mon.0) 2562 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "61.4", "id": [5, 3]}]: dispatch 2024-02-19T21:10:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:39.271521+0000 mon.a (mon.0) 2563 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 1]}]': finished 2024-02-19T21:10:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:39.271607+0000 mon.a (mon.0) 2564 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-02-19T21:10:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:39.271669+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]': finished 2024-02-19T21:10:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:39.271729+0000 mon.a (mon.0) 2566 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 2]}]': finished 2024-02-19T21:10:39.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:39.271788+0000 mon.a (mon.0) 2567 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]': finished 2024-02-19T21:10:39.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:39.271847+0000 mon.a (mon.0) 2568 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1", "id": [1, 3]}]': finished 2024-02-19T21:10:39.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:39.271910+0000 mon.a (mon.0) 2569 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "61.4", "id": [5, 3]}]': finished 2024-02-19T21:10:39.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:39.271965+0000 mon.a (mon.0) 2570 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2024-02-19T21:10:39.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:39.361852+0000 mon.c (mon.2) 393 : audit [DBG] from='client.? v1:172.21.15.163:0/1037603649' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-19T21:10:39.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:39.362269+0000 mon.a (mon.0) 2571 : audit [DBG] from='client.49880 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-19T21:10:39.773 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:10:39.799 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:38.943836+0000 mon.b (mon.1) 586 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 1]}]: dispatch 2024-02-19T21:10:39.799 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:38.944060+0000 mon.b (mon.1) 587 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-19T21:10:39.799 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:38.944268+0000 mon.b (mon.1) 588 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]: dispatch 2024-02-19T21:10:39.800 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:38.944273+0000 mon.a (mon.0) 2556 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 1]}]: dispatch 2024-02-19T21:10:39.800 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:38.944442+0000 mon.b (mon.1) 589 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 2]}]: dispatch 2024-02-19T21:10:39.800 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:38.944614+0000 mon.b (mon.1) 590 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]: dispatch 2024-02-19T21:10:39.800 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:38.944712+0000 mon.a (mon.0) 2557 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-02-19T21:10:39.800 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:38.944784+0000 mon.b (mon.1) 591 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1", "id": [1, 3]}]: dispatch 2024-02-19T21:10:39.800 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:38.944932+0000 mon.a (mon.0) 2558 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]: dispatch 2024-02-19T21:10:39.800 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:38.944952+0000 mon.b (mon.1) 592 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "61.4", "id": [5, 3]}]: dispatch 2024-02-19T21:10:39.800 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:38.945205+0000 mon.a (mon.0) 2559 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 2]}]: dispatch 2024-02-19T21:10:39.800 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:38.945386+0000 mon.a (mon.0) 2560 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]: dispatch 2024-02-19T21:10:39.800 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:38.945553+0000 mon.a (mon.0) 2561 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1", "id": [1, 3]}]: dispatch 2024-02-19T21:10:39.800 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:38.945712+0000 mon.a (mon.0) 2562 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "61.4", "id": [5, 3]}]: dispatch 2024-02-19T21:10:39.800 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:39.271521+0000 mon.a (mon.0) 2563 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 1]}]': finished 2024-02-19T21:10:39.800 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:39.271607+0000 mon.a (mon.0) 2564 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-02-19T21:10:39.801 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:39.271669+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]': finished 2024-02-19T21:10:39.801 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:39.271729+0000 mon.a (mon.0) 2566 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [3, 2]}]': finished 2024-02-19T21:10:39.801 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:39.271788+0000 mon.a (mon.0) 2567 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [0, 3]}]': finished 2024-02-19T21:10:39.801 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:39.271847+0000 mon.a (mon.0) 2568 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.1", "id": [1, 3]}]': finished 2024-02-19T21:10:39.801 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:39.271910+0000 mon.a (mon.0) 2569 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "61.4", "id": [5, 3]}]': finished 2024-02-19T21:10:39.801 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:39.271965+0000 mon.a (mon.0) 2570 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2024-02-19T21:10:39.801 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:39.361852+0000 mon.c (mon.2) 393 : audit [DBG] from='client.? v1:172.21.15.163:0/1037603649' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-19T21:10:39.801 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:39.362269+0000 mon.a (mon.0) 2571 : audit [DBG] from='client.49880 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-19T21:10:39.801 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:39.821 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:10:39.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:39.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:10:39.925 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:39.952 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:39.975 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:39.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:39.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:40.079 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:10:40.107 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:40.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:40.276+0000 7f3ba77fe700 1 -- 172.21.15.163:0/1037603649 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd in", "ids": ["4"]}]=0 marked in osd.4. v433) v1 ==== 85+0+0 (unknown 2670164216 0 0) 0x7f3bac04a600 con 0x7f3bb006b0f0 2024-02-19T21:10:40.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:40.281+0000 7f3bb84c0700 10 librados: watch_flush enter 2024-02-19T21:10:40.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:40.281+0000 7f3bb84c0700 10 librados: watch_flush exit 2024-02-19T21:10:40.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:40.281+0000 7f3bb84c0700 20 client.49880.objecter shutdown clearing up homeless session... 2024-02-19T21:10:40.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:40.281+0000 7f3bb84c0700 10 client.49880.objecter successfully canceled tick 2024-02-19T21:10:40.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:40.282+0000 7f3bb84c0700 1 -- 172.21.15.163:0/1037603649 >> v1:172.21.15.163:6800/1552514681 conn(0x7f3b9c063c10 legacy=0x7f3b9c0660d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:40.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:40.282+0000 7f3bb84c0700 1 -- 172.21.15.163:0/1037603649 >> v1:172.21.15.163:6790/0 conn(0x7f3bb006b0f0 legacy=0x7f3bb01026c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:40.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:40.293+0000 7f3bb84c0700 1 -- 172.21.15.163:0/1037603649 shutdown_connections 2024-02-19T21:10:40.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:40.293+0000 7f3bb84c0700 1 -- 172.21.15.163:0/1037603649 >> 172.21.15.163:0/1037603649 conn(0x7f3bb00f35a0 msgr2=0x7f3bb0108920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:40.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:40.293+0000 7f3bb84c0700 1 -- 172.21.15.163:0/1037603649 shutdown_connections 2024-02-19T21:10:40.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:40.293+0000 7f3bb84c0700 1 -- 172.21.15.163:0/1037603649 wait complete. 2024-02-19T21:10:40.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:40.294+0000 7f3bb84c0700 1 librados: shutdown 2024-02-19T21:10:40.293 INFO:teuthology.orchestra.run.smithi163.stderr:marked in osd.4. 2024-02-19T21:10:40.726 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:40.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:39.220385+0000 mgr.y (mgr.14550) 394 : cluster [DBG] pgmap v632: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 266 B/s wr, 2 op/s 2024-02-19T21:10:40.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:40.275858+0000 mon.a (mon.0) 2572 : audit [INF] from='client.49880 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-19T21:10:40.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:40.275907+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e433: 8 total, 8 up, 8 in 2024-02-19T21:10:40.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:39.220385+0000 mgr.y (mgr.14550) 394 : cluster [DBG] pgmap v632: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 266 B/s wr, 2 op/s 2024-02-19T21:10:40.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:40.275858+0000 mon.a (mon.0) 2572 : audit [INF] from='client.49880 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-19T21:10:40.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:40.275907+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e433: 8 total, 8 up, 8 in 2024-02-19T21:10:40.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:10:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:10:40.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:39.220385+0000 mgr.y (mgr.14550) 394 : cluster [DBG] pgmap v632: 228 pgs: 228 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 266 B/s wr, 2 op/s 2024-02-19T21:10:40.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:40.275858+0000 mon.a (mon.0) 2572 : audit [INF] from='client.49880 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-19T21:10:40.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:40.275907+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e433: 8 total, 8 up, 8 in 2024-02-19T21:10:41.191 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:41.201 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-02-19T21:10:41.456 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:10:41.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:41.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:10:41.563 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:41.589 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:41.607 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:41.610 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:41.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:41.691 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:40.418094+0000 mgr.y (mgr.14550) 395 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:41.691 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:41.277979+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e434: 8 total, 8 up, 8 in 2024-02-19T21:10:41.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:40.418094+0000 mgr.y (mgr.14550) 395 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:41.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:41.277979+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e434: 8 total, 8 up, 8 in 2024-02-19T21:10:41.712 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:41.739 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:41.759 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:41.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:41.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:40.418094+0000 mgr.y (mgr.14550) 395 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:41.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:41.277979+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e434: 8 total, 8 up, 8 in 2024-02-19T21:10:41.862 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:41.888 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:41.908 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:41.911 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:41.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:42.012 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:10:42.044 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:42.086 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:10:42.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:42.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:10:42.193 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:42.216 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:10:42.216 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:10:42.216 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-02-19T21:10:42.216 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-02-19T21:10:42.216 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:10:42.220 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:41.220849+0000 mgr.y (mgr.14550) 396 : cluster [DBG] pgmap v635: 228 pgs: 7 peering, 221 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:10:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:42.276278+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e435: 8 total, 8 up, 8 in 2024-02-19T21:10:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:42.304093+0000 mon.b (mon.1) 593 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:42.304626+0000 mon.a (mon.0) 2576 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:42.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:42.305167+0000 mon.b (mon.1) 594 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-63"}]: dispatch 2024-02-19T21:10:42.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:42.305390+0000 mon.a (mon.0) 2577 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-63"}]: dispatch 2024-02-19T21:10:42.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:41.220849+0000 mgr.y (mgr.14550) 396 : cluster [DBG] pgmap v635: 228 pgs: 7 peering, 221 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:10:42.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:42.276278+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e435: 8 total, 8 up, 8 in 2024-02-19T21:10:42.915 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:42.304093+0000 mon.b (mon.1) 593 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:42.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:42.304626+0000 mon.a (mon.0) 2576 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:42.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:42.305167+0000 mon.b (mon.1) 594 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-63"}]: dispatch 2024-02-19T21:10:42.916 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:42.305390+0000 mon.a (mon.0) 2577 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-63"}]: dispatch 2024-02-19T21:10:42.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:41.220849+0000 mgr.y (mgr.14550) 396 : cluster [DBG] pgmap v635: 228 pgs: 7 peering, 221 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:10:42.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:42.276278+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e435: 8 total, 8 up, 8 in 2024-02-19T21:10:42.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:42.304093+0000 mon.b (mon.1) 593 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:42.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:42.304626+0000 mon.a (mon.0) 2576 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:42.916 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:42.305167+0000 mon.b (mon.1) 594 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-63"}]: dispatch 2024-02-19T21:10:42.917 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:42.305390+0000 mon.a (mon.0) 2577 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-63"}]: dispatch 2024-02-19T21:10:43.686 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:10:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:10:43] "GET /metrics HTTP/1.1" 200 189714 "" "Prometheus/2.33.4" 2024-02-19T21:10:43.974 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:43.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:43.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:44.078 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:44.103 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:44.121 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:44.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:44.226 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:44.252 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:44.285 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:44.288 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:44.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:44.390 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:10:44.405 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-19T21:10:44.418 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:43.221306+0000 mgr.y (mgr.14550) 397 : cluster [DBG] pgmap v638: 228 pgs: 7 peering, 221 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:10:44.418 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:43.277524+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e436: 8 total, 8 up, 8 in 2024-02-19T21:10:44.418 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:44.273588+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e437: 8 total, 8 up, 8 in 2024-02-19T21:10:44.419 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:44.439 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:10:44.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:44.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:10:44.543 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:44.568 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:43.221306+0000 mgr.y (mgr.14550) 397 : cluster [DBG] pgmap v638: 228 pgs: 7 peering, 221 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:10:44.568 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:43.277524+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e436: 8 total, 8 up, 8 in 2024-02-19T21:10:44.568 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:44.273588+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e437: 8 total, 8 up, 8 in 2024-02-19T21:10:44.568 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:43.221306+0000 mgr.y (mgr.14550) 397 : cluster [DBG] pgmap v638: 228 pgs: 7 peering, 221 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:10:44.568 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:43.277524+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e436: 8 total, 8 up, 8 in 2024-02-19T21:10:44.569 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:44.273588+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e437: 8 total, 8 up, 8 in 2024-02-19T21:10:44.570 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:45.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.101+0000 7feaf02d6700 1 -- 172.21.15.163:0/2052346800 >> v1:172.21.15.163:6789/0 conn(0x7feae81042e0 legacy=0x7feae8106780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.101+0000 7feaf02d6700 1 -- 172.21.15.163:0/2052346800 shutdown_connections 2024-02-19T21:10:45.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.101+0000 7feaf02d6700 1 -- 172.21.15.163:0/2052346800 >> 172.21.15.163:0/2052346800 conn(0x7feae80f2470 msgr2=0x7feae80f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:45.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.101+0000 7feaf02d6700 1 -- 172.21.15.163:0/2052346800 shutdown_connections 2024-02-19T21:10:45.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.101+0000 7feaf02d6700 1 -- 172.21.15.163:0/2052346800 wait complete. 2024-02-19T21:10:45.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.102+0000 7feaf02d6700 1 librados: starting msgr at 2024-02-19T21:10:45.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.102+0000 7feaf02d6700 1 librados: starting objecter 2024-02-19T21:10:45.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.102+0000 7feaf02d6700 1 Processor -- start 2024-02-19T21:10:45.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.102+0000 7feaf02d6700 1 -- start start 2024-02-19T21:10:45.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.102+0000 7feaf02d6700 1 librados: setting wanted keys 2024-02-19T21:10:45.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.102+0000 7feaf02d6700 1 librados: calling monclient init 2024-02-19T21:10:45.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feaf02d6700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feae8102b80 con 0x7feae80f89e0 2024-02-19T21:10:45.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feaf02d6700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feae8102d50 con 0x7feae81042e0 2024-02-19T21:10:45.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feaf02d6700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feae8102f40 con 0x7feae80f94d0 2024-02-19T21:10:45.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feaed871700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7feae80f94d0 0x7feae8114ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:33382/0 (socket says 172.21.15.163:33382) 2024-02-19T21:10:45.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feaed871700 1 -- 172.21.15.163:0/1151724448 learned_addr learned my addr 172.21.15.163:0/1151724448 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:45.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 10 client.?.objecter ms_handle_connect 0x7feae80f94d0 2024-02-19T21:10:45.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:45.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 10 client.?.objecter ms_handle_connect 0x7feae81042e0 2024-02-19T21:10:45.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:45.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 10 client.?.objecter ms_handle_connect 0x7feae80f89e0 2024-02-19T21:10:45.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:45.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694432516 0 0) 0x7feae8102f40 con 0x7feae80f94d0 2024-02-19T21:10:45.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead40033f0 con 0x7feae80f94d0 2024-02-19T21:10:45.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099713970 0 0) 0x7feae8102d50 con 0x7feae81042e0 2024-02-19T21:10:45.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feae8102f40 con 0x7feae81042e0 2024-02-19T21:10:45.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249169188 0 0) 0x7feae8102b80 con 0x7feae80f89e0 2024-02-19T21:10:45.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feae8102d50 con 0x7feae80f89e0 2024-02-19T21:10:45.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 717066549 0 0) 0x7feae8102f40 con 0x7feae81042e0 2024-02-19T21:10:45.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feae8102b80 con 0x7feae81042e0 2024-02-19T21:10:45.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7feae00023f0 con 0x7feae81042e0 2024-02-19T21:10:45.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 45644615 0 0) 0x7fead40033f0 con 0x7feae80f94d0 2024-02-19T21:10:45.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feae8102f40 con 0x7feae80f94d0 2024-02-19T21:10:45.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2345056351 0 0) 0x7feae8102d50 con 0x7feae80f89e0 2024-02-19T21:10:45.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fead40033f0 con 0x7feae80f89e0 2024-02-19T21:10:45.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fead8001950 con 0x7feae80f94d0 2024-02-19T21:10:45.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7feae4001840 con 0x7feae80f89e0 2024-02-19T21:10:45.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3751772299 0 0) 0x7feae8102b80 con 0x7feae81042e0 2024-02-19T21:10:45.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 >> v1:172.21.15.163:6790/0 conn(0x7feae80f94d0 legacy=0x7feae8114ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 >> v1:172.21.15.163:6789/0 conn(0x7feae80f89e0 legacy=0x7feae81023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae811cb60 con 0x7feae81042e0 2024-02-19T21:10:45.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feaf02d6700 1 -- 172.21.15.163:0/1151724448 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae8119b90 con 0x7feae81042e0 2024-02-19T21:10:45.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feaf02d6700 10 client.48706.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:45.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feaf02d6700 1 -- 172.21.15.163:0/1151724448 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feae811a000 con 0x7feae81042e0 2024-02-19T21:10:45.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.103+0000 7feaf02d6700 1 librados: init done 2024-02-19T21:10:45.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.104+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7feae0002590 con 0x7feae81042e0 2024-02-19T21:10:45.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.104+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7feae0003a60 con 0x7feae81042e0 2024-02-19T21:10:45.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.105+0000 7feaf02d6700 1 -- 172.21.15.163:0/1151724448 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feacc004f20 con 0x7feae81042e0 2024-02-19T21:10:45.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.105+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7feae00168f0 con 0x7feae81042e0 2024-02-19T21:10:45.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.105+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(437..437 src has 43..437) v4 ==== 8108+0+0 (unknown 1413329446 0 0) 0x7feae0076af0 con 0x7feae81042e0 2024-02-19T21:10:45.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.105+0000 7feadf779700 10 client.48706.objecter ms_dispatch 0x7feae8001550 osd_map(437..437 src has 43..437) v4 2024-02-19T21:10:45.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.105+0000 7feadf779700 3 client.48706.objecter handle_osd_map got epochs [437,437] > 0 2024-02-19T21:10:45.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.105+0000 7feadf779700 3 client.48706.objecter handle_osd_map decoding full epoch 437 2024-02-19T21:10:45.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.106+0000 7feadf779700 20 client.48706.objecter dump_active .. 0 homeless 2024-02-19T21:10:45.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.106+0000 7feadf779700 10 client.48706.objecter ms_handle_connect 0x7fead4063d80 2024-02-19T21:10:45.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.109+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7feae0049f10 con 0x7feae81042e0 2024-02-19T21:10:45.280 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:10:45.280 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:10:45.280 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:10:45.280 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:10:45.280 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:10:45.280 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:10:45.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.155+0000 7efefd399700 1 -- 172.21.15.163:0/1179486900 >> v1:172.21.15.193:6789/0 conn(0x7efef80f8920 legacy=0x7efef80f8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.155+0000 7efefd399700 1 -- 172.21.15.163:0/1179486900 shutdown_connections 2024-02-19T21:10:45.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.155+0000 7efefd399700 1 -- 172.21.15.163:0/1179486900 >> 172.21.15.163:0/1179486900 conn(0x7efef80f2470 msgr2=0x7efef80f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:45.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.155+0000 7efefd399700 1 -- 172.21.15.163:0/1179486900 shutdown_connections 2024-02-19T21:10:45.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.155+0000 7efefd399700 1 -- 172.21.15.163:0/1179486900 wait complete. 2024-02-19T21:10:45.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.155+0000 7efefd399700 1 librados: starting msgr at 2024-02-19T21:10:45.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.155+0000 7efefd399700 1 librados: starting objecter 2024-02-19T21:10:45.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efefd399700 1 Processor -- start 2024-02-19T21:10:45.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efefd399700 1 -- start start 2024-02-19T21:10:45.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efefd399700 1 librados: setting wanted keys 2024-02-19T21:10:45.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efefd399700 1 librados: calling monclient init 2024-02-19T21:10:45.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efefd399700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efef8102a40 con 0x7efef81042b0 2024-02-19T21:10:45.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efefd399700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efef8102c10 con 0x7efef80f8920 2024-02-19T21:10:45.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efefd399700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efef8102e00 con 0x7efef80f94d0 2024-02-19T21:10:45.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efef77fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7efef81042b0 0x7efef8118430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:35628/0 (socket says 172.21.15.163:35628) 2024-02-19T21:10:45.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efef67fc700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7efef80f94d0 0x7efef8114d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:33396/0 (socket says 172.21.15.163:33396) 2024-02-19T21:10:45.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efef77fe700 1 -- 172.21.15.163:0/1316423178 learned_addr learned my addr 172.21.15.163:0/1316423178 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:45.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efedffff700 10 client.?.objecter ms_handle_connect 0x7efef80f94d0 2024-02-19T21:10:45.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efedffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:45.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efedffff700 10 client.?.objecter ms_handle_connect 0x7efef81042b0 2024-02-19T21:10:45.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efedffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:45.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efedffff700 10 client.?.objecter ms_handle_connect 0x7efef80f8920 2024-02-19T21:10:45.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efedffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:45.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299018443 0 0) 0x7efef8102e00 con 0x7efef80f94d0 2024-02-19T21:10:45.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee00033f0 con 0x7efef80f94d0 2024-02-19T21:10:45.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829848107 0 0) 0x7efef8102a40 con 0x7efef81042b0 2024-02-19T21:10:45.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efef8102e00 con 0x7efef81042b0 2024-02-19T21:10:45.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405041521 0 0) 0x7efef8102c10 con 0x7efef80f8920 2024-02-19T21:10:45.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.156+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efef8102a40 con 0x7efef80f8920 2024-02-19T21:10:45.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 407351508 0 0) 0x7efee00033f0 con 0x7efef80f94d0 2024-02-19T21:10:45.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efef8102c10 con 0x7efef80f94d0 2024-02-19T21:10:45.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7efeec001910 con 0x7efef80f94d0 2024-02-19T21:10:45.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1713211660 0 0) 0x7efef8102e00 con 0x7efef81042b0 2024-02-19T21:10:45.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efee00033f0 con 0x7efef81042b0 2024-02-19T21:10:45.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3026094426 0 0) 0x7efef8102a40 con 0x7efef80f8920 2024-02-19T21:10:45.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efef8102e00 con 0x7efef80f8920 2024-02-19T21:10:45.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7efef0001d50 con 0x7efef81042b0 2024-02-19T21:10:45.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7efee8002070 con 0x7efef80f8920 2024-02-19T21:10:45.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 573449283 0 0) 0x7efef8102c10 con 0x7efef80f94d0 2024-02-19T21:10:45.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 >> v1:172.21.15.193:6789/0 conn(0x7efef80f8920 legacy=0x7efef81022a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 >> v1:172.21.15.163:6789/0 conn(0x7efef81042b0 legacy=0x7efef8118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef811cb80 con 0x7efef80f94d0 2024-02-19T21:10:45.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef8119bb0 con 0x7efef80f94d0 2024-02-19T21:10:45.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efefd399700 10 client.49892.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:45.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efef811a020 con 0x7efef80f94d0 2024-02-19T21:10:45.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efefd399700 1 librados: init done 2024-02-19T21:10:45.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7efeec0023a0 con 0x7efef80f94d0 2024-02-19T21:10:45.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.157+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7efeec0059e0 con 0x7efef80f94d0 2024-02-19T21:10:45.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.158+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7efee4004f00 con 0x7efef80f94d0 2024-02-19T21:10:45.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.158+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7efeec001f00 con 0x7efef80f94d0 2024-02-19T21:10:45.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(437..437 src has 43..437) v4 ==== 8108+0+0 (unknown 1413329446 0 0) 0x7efeec078cc0 con 0x7efef80f94d0 2024-02-19T21:10:45.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 osd_map(437..437 src has 43..437) v4 2024-02-19T21:10:45.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efedffff700 3 client.49892.objecter handle_osd_map got epochs [437,437] > 0 2024-02-19T21:10:45.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efedffff700 3 client.49892.objecter handle_osd_map decoding full epoch 437 2024-02-19T21:10:45.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efedffff700 20 client.49892.objecter dump_active .. 0 homeless 2024-02-19T21:10:45.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v437) v1 ==== 54+0+16 (unknown 61667149 0 2420000192) 0x7efeec07b110 con 0x7efef80f94d0 2024-02-19T21:10:45.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efedffff700 10 client.49892.objecter ms_handle_connect 0x7efee0063d00 2024-02-19T21:10:45.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efefd399700 10 client.49892.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:45.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 1 2024-02-19T21:10:45.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee40051a0 osd=0 3 2024-02-19T21:10:45.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee40051a0 osd=0 3 2024-02-19T21:10:45.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee40051a0 osd=0 3 2024-02-19T21:10:45.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 1 2024-02-19T21:10:45.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee40051a0 osd=0 3 2024-02-19T21:10:45.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 0 1 2024-02-19T21:10:45.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee40051a0 osd=0 4 2024-02-19T21:10:45.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efefd399700 10 client.49892.objecter _send_command 1 2024-02-19T21:10:45.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efee4004950 con 0x7efee4004530 2024-02-19T21:10:45.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.159+0000 7efedffff700 10 client.49892.objecter ms_handle_connect 0x7efee4004530 2024-02-19T21:10:45.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.160+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efee4004950 con 0x7efee4004530 2024-02-19T21:10:45.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.160+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:10:45.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.160+0000 7efedffff700 10 client.49892.objecter _finish_command 1 = system:0 2024-02-19T21:10:45.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.160+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee40051a0 osd=0 4 2024-02-19T21:10:45.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.160+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 0 1 2024-02-19T21:10:45.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.175+0000 7efefd399700 10 client.49892.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:45.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.175+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 2 2024-02-19T21:10:45.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.175+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee40051a0 osd=0 3 2024-02-19T21:10:45.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.175+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee40051a0 osd=0 3 2024-02-19T21:10:45.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.175+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee40051a0 osd=0 3 2024-02-19T21:10:45.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.175+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 2 2024-02-19T21:10:45.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.175+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee40051a0 osd=0 3 2024-02-19T21:10:45.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.175+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 0 2 2024-02-19T21:10:45.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.175+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee40051a0 osd=0 4 2024-02-19T21:10:45.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.175+0000 7efefd399700 10 client.49892.objecter _send_command 2 2024-02-19T21:10:45.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.175+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efee4004d30 con 0x7efee4004530 2024-02-19T21:10:45.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.176+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efee4004d30 con 0x7efee4004530 2024-02-19T21:10:45.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.176+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:45.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.176+0000 7efedffff700 10 client.49892.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.176+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee40051a0 osd=0 4 2024-02-19T21:10:45.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.176+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 0 2 2024-02-19T21:10:45.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.177+0000 7efefd399700 10 client.49892.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:45.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.177+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 3 2024-02-19T21:10:45.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.177+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4003fe0 osd=1 3 2024-02-19T21:10:45.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.177+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4003fe0 osd=1 3 2024-02-19T21:10:45.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.177+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4003fe0 osd=1 3 2024-02-19T21:10:45.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.177+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 3 2024-02-19T21:10:45.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.177+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee4003fe0 osd=1 3 2024-02-19T21:10:45.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.177+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 1 3 2024-02-19T21:10:45.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.177+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4003fe0 osd=1 4 2024-02-19T21:10:45.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.177+0000 7efefd399700 10 client.49892.objecter _send_command 3 2024-02-19T21:10:45.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.177+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7efee400c000 con 0x7efee4004150 2024-02-19T21:10:45.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.177+0000 7efedffff700 10 client.49892.objecter ms_handle_connect 0x7efee4004150 2024-02-19T21:10:45.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.177+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efee400c000 con 0x7efee4004150 2024-02-19T21:10:45.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.178+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:10:45.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.178+0000 7efedffff700 10 client.49892.objecter _finish_command 3 = system:0 2024-02-19T21:10:45.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.178+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee4003fe0 osd=1 4 2024-02-19T21:10:45.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.178+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 1 3 2024-02-19T21:10:45.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.193+0000 7efefd399700 10 client.49892.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:45.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.193+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 4 2024-02-19T21:10:45.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.193+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4003fe0 osd=1 3 2024-02-19T21:10:45.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.193+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4003fe0 osd=1 3 2024-02-19T21:10:45.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.193+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4003fe0 osd=1 3 2024-02-19T21:10:45.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.193+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 4 2024-02-19T21:10:45.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.193+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee4003fe0 osd=1 3 2024-02-19T21:10:45.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.193+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 1 4 2024-02-19T21:10:45.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.193+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4003fe0 osd=1 4 2024-02-19T21:10:45.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.193+0000 7efefd399700 10 client.49892.objecter _send_command 4 2024-02-19T21:10:45.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.193+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efee40018b0 con 0x7efee4004150 2024-02-19T21:10:45.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efee40018b0 con 0x7efee4004150 2024-02-19T21:10:45.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:45.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efedffff700 10 client.49892.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee4003fe0 osd=1 4 2024-02-19T21:10:45.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 1 4 2024-02-19T21:10:45.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efefd399700 10 client.49892.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:45.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 5 2024-02-19T21:10:45.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4003630 osd=2 3 2024-02-19T21:10:45.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4003630 osd=2 3 2024-02-19T21:10:45.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4003630 osd=2 3 2024-02-19T21:10:45.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 5 2024-02-19T21:10:45.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee4003630 osd=2 3 2024-02-19T21:10:45.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 2 5 2024-02-19T21:10:45.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4003630 osd=2 4 2024-02-19T21:10:45.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efefd399700 10 client.49892.objecter _send_command 5 2024-02-19T21:10:45.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7efee400de90 con 0x7efee400d2f0 2024-02-19T21:10:45.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.195+0000 7efedffff700 10 client.49892.objecter ms_handle_connect 0x7efee400d2f0 2024-02-19T21:10:45.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.196+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efee400de90 con 0x7efee400d2f0 2024-02-19T21:10:45.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.196+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:10:45.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.196+0000 7efedffff700 10 client.49892.objecter _finish_command 5 = system:0 2024-02-19T21:10:45.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.196+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee4003630 osd=2 4 2024-02-19T21:10:45.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.196+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 2 5 2024-02-19T21:10:45.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.210+0000 7efefd399700 10 client.49892.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:45.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.210+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 6 2024-02-19T21:10:45.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.210+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4003630 osd=2 3 2024-02-19T21:10:45.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.210+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4003630 osd=2 3 2024-02-19T21:10:45.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.210+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4003630 osd=2 3 2024-02-19T21:10:45.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.210+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 6 2024-02-19T21:10:45.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.210+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee4003630 osd=2 3 2024-02-19T21:10:45.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.210+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 2 6 2024-02-19T21:10:45.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.210+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4003630 osd=2 4 2024-02-19T21:10:45.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.210+0000 7efefd399700 10 client.49892.objecter _send_command 6 2024-02-19T21:10:45.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.210+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efee4003160 con 0x7efee400d2f0 2024-02-19T21:10:45.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efee4003160 con 0x7efee400d2f0 2024-02-19T21:10:45.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:45.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efedffff700 10 client.49892.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee4003630 osd=2 4 2024-02-19T21:10:45.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 2 6 2024-02-19T21:10:45.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efefd399700 10 client.49892.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:45.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 7 2024-02-19T21:10:45.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4011970 osd=3 3 2024-02-19T21:10:45.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4011970 osd=3 3 2024-02-19T21:10:45.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4011970 osd=3 3 2024-02-19T21:10:45.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 7 2024-02-19T21:10:45.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee4011970 osd=3 3 2024-02-19T21:10:45.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 3 7 2024-02-19T21:10:45.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4011970 osd=3 4 2024-02-19T21:10:45.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efefd399700 10 client.49892.objecter _send_command 7 2024-02-19T21:10:45.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7efee4015ba0 con 0x7efee4011ff0 2024-02-19T21:10:45.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.212+0000 7efedffff700 10 client.49892.objecter ms_handle_connect 0x7efee4011ff0 2024-02-19T21:10:45.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.213+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efee4015ba0 con 0x7efee4011ff0 2024-02-19T21:10:45.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.213+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:10:45.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.213+0000 7efedffff700 10 client.49892.objecter _finish_command 7 = system:0 2024-02-19T21:10:45.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.213+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee4011970 osd=3 4 2024-02-19T21:10:45.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.213+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 3 7 2024-02-19T21:10:45.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.227+0000 7efefd399700 10 client.49892.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:45.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.227+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 8 2024-02-19T21:10:45.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.227+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4011970 osd=3 3 2024-02-19T21:10:45.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.227+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4011970 osd=3 3 2024-02-19T21:10:45.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.227+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4011970 osd=3 3 2024-02-19T21:10:45.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.227+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 8 2024-02-19T21:10:45.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.227+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee4011970 osd=3 3 2024-02-19T21:10:45.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.227+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 3 8 2024-02-19T21:10:45.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.227+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4011970 osd=3 4 2024-02-19T21:10:45.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.227+0000 7efefd399700 10 client.49892.objecter _send_command 8 2024-02-19T21:10:45.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.227+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efee4015ba0 con 0x7efee4011ff0 2024-02-19T21:10:45.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efee4015ba0 con 0x7efee4011ff0 2024-02-19T21:10:45.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:45.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efedffff700 10 client.49892.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee4011970 osd=3 4 2024-02-19T21:10:45.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 3 8 2024-02-19T21:10:45.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efefd399700 10 client.49892.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:45.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 9 2024-02-19T21:10:45.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4015ba0 osd=4 3 2024-02-19T21:10:45.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4015ba0 osd=4 3 2024-02-19T21:10:45.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4015ba0 osd=4 3 2024-02-19T21:10:45.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 9 2024-02-19T21:10:45.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee4015ba0 osd=4 3 2024-02-19T21:10:45.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 4 9 2024-02-19T21:10:45.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4015ba0 osd=4 4 2024-02-19T21:10:45.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efefd399700 10 client.49892.objecter _send_command 9 2024-02-19T21:10:45.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7efee400cee0 con 0x7efee400c940 2024-02-19T21:10:45.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.229+0000 7efedffff700 10 client.49892.objecter ms_handle_connect 0x7efee400c940 2024-02-19T21:10:45.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.230+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efee400cee0 con 0x7efee400c940 2024-02-19T21:10:45.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.230+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:10:45.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.230+0000 7efedffff700 10 client.49892.objecter _finish_command 9 = system:0 2024-02-19T21:10:45.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.230+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee4015ba0 osd=4 4 2024-02-19T21:10:45.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.230+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 4 9 2024-02-19T21:10:45.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.245+0000 7efefd399700 10 client.49892.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:45.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.245+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 10 2024-02-19T21:10:45.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.245+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4015ba0 osd=4 3 2024-02-19T21:10:45.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.245+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4015ba0 osd=4 3 2024-02-19T21:10:45.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.245+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4015ba0 osd=4 3 2024-02-19T21:10:45.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.245+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 10 2024-02-19T21:10:45.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.245+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee4015ba0 osd=4 3 2024-02-19T21:10:45.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.245+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 4 10 2024-02-19T21:10:45.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.245+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4015ba0 osd=4 4 2024-02-19T21:10:45.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.245+0000 7efefd399700 10 client.49892.objecter _send_command 10 2024-02-19T21:10:45.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.245+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efee400cee0 con 0x7efee400c940 2024-02-19T21:10:45.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efee400cee0 con 0x7efee400c940 2024-02-19T21:10:45.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:45.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efedffff700 10 client.49892.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee4015ba0 osd=4 4 2024-02-19T21:10:45.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 4 10 2024-02-19T21:10:45.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efefd399700 10 client.49892.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:45.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 11 2024-02-19T21:10:45.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee400cee0 osd=5 3 2024-02-19T21:10:45.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee400cee0 osd=5 3 2024-02-19T21:10:45.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee400cee0 osd=5 3 2024-02-19T21:10:45.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 11 2024-02-19T21:10:45.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee400cee0 osd=5 3 2024-02-19T21:10:45.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 5 11 2024-02-19T21:10:45.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee400cee0 osd=5 4 2024-02-19T21:10:45.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efefd399700 10 client.49892.objecter _send_command 11 2024-02-19T21:10:45.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.247+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7efee4016da0 con 0x7efee4016960 2024-02-19T21:10:45.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.248+0000 7efedffff700 10 client.49892.objecter ms_handle_connect 0x7efee4016960 2024-02-19T21:10:45.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.248+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efee4016da0 con 0x7efee4016960 2024-02-19T21:10:45.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.248+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:10:45.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.248+0000 7efedffff700 10 client.49892.objecter _finish_command 11 = system:0 2024-02-19T21:10:45.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.248+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee400cee0 osd=5 4 2024-02-19T21:10:45.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.248+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 5 11 2024-02-19T21:10:45.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.263+0000 7efefd399700 10 client.49892.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:45.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.263+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 12 2024-02-19T21:10:45.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.263+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee400cee0 osd=5 3 2024-02-19T21:10:45.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.263+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee400cee0 osd=5 3 2024-02-19T21:10:45.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.263+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee400cee0 osd=5 3 2024-02-19T21:10:45.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.263+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 12 2024-02-19T21:10:45.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.263+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee400cee0 osd=5 3 2024-02-19T21:10:45.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.263+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 5 12 2024-02-19T21:10:45.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.263+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee400cee0 osd=5 4 2024-02-19T21:10:45.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.263+0000 7efefd399700 10 client.49892.objecter _send_command 12 2024-02-19T21:10:45.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.263+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efee4016da0 con 0x7efee4016960 2024-02-19T21:10:45.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.264+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efee4016da0 con 0x7efee4016960 2024-02-19T21:10:45.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.264+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:45.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.264+0000 7efedffff700 10 client.49892.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.264+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee400cee0 osd=5 4 2024-02-19T21:10:45.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.264+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 5 12 2024-02-19T21:10:45.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.265+0000 7efefd399700 10 client.49892.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:45.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.265+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 13 2024-02-19T21:10:45.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.265+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4016da0 osd=6 3 2024-02-19T21:10:45.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.265+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4016da0 osd=6 3 2024-02-19T21:10:45.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.265+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4016da0 osd=6 3 2024-02-19T21:10:45.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.265+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 13 2024-02-19T21:10:45.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.265+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee4016da0 osd=6 3 2024-02-19T21:10:45.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.265+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 6 13 2024-02-19T21:10:45.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.265+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4016da0 osd=6 4 2024-02-19T21:10:45.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.265+0000 7efefd399700 10 client.49892.objecter _send_command 13 2024-02-19T21:10:45.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.265+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7efee4017190 con 0x7efee401f1a0 2024-02-19T21:10:45.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.265+0000 7efedffff700 10 client.49892.objecter ms_handle_connect 0x7efee401f1a0 2024-02-19T21:10:45.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.265+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efee4017190 con 0x7efee401f1a0 2024-02-19T21:10:45.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.266+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:10:45.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.266+0000 7efedffff700 10 client.49892.objecter _finish_command 13 = system:0 2024-02-19T21:10:45.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.266+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee4016da0 osd=6 4 2024-02-19T21:10:45.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.266+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 6 13 2024-02-19T21:10:45.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.286+0000 7efefd399700 10 client.49892.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:45.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.286+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 14 2024-02-19T21:10:45.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.286+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4016da0 osd=6 3 2024-02-19T21:10:45.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.286+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4016da0 osd=6 3 2024-02-19T21:10:45.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.286+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4016da0 osd=6 3 2024-02-19T21:10:45.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.286+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 14 2024-02-19T21:10:45.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.286+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee4016da0 osd=6 3 2024-02-19T21:10:45.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.286+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 6 14 2024-02-19T21:10:45.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.286+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4016da0 osd=6 4 2024-02-19T21:10:45.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.286+0000 7efefd399700 10 client.49892.objecter _send_command 14 2024-02-19T21:10:45.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.286+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efee4017190 con 0x7efee401f1a0 2024-02-19T21:10:45.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.296+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efee4017190 con 0x7efee401f1a0 2024-02-19T21:10:45.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.296+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:45.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.296+0000 7efedffff700 10 client.49892.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.296+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee4016da0 osd=6 4 2024-02-19T21:10:45.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.296+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 6 14 2024-02-19T21:10:45.303 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:10:45.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.298+0000 7efefd399700 10 client.49892.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:45.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.298+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 15 2024-02-19T21:10:45.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.298+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4017190 osd=7 3 2024-02-19T21:10:45.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.298+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4017190 osd=7 3 2024-02-19T21:10:45.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.298+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4017190 osd=7 3 2024-02-19T21:10:45.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.298+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 15 2024-02-19T21:10:45.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.298+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee4017190 osd=7 3 2024-02-19T21:10:45.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.298+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 7 15 2024-02-19T21:10:45.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.298+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4017190 osd=7 4 2024-02-19T21:10:45.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.298+0000 7efefd399700 10 client.49892.objecter _send_command 15 2024-02-19T21:10:45.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.298+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7efee4023d60 con 0x7efee4024840 2024-02-19T21:10:45.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.299+0000 7efedffff700 10 client.49892.objecter ms_handle_connect 0x7efee4024840 2024-02-19T21:10:45.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.300+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efee4023d60 con 0x7efee4024840 2024-02-19T21:10:45.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.300+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:10:45.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.300+0000 7efedffff700 10 client.49892.objecter _finish_command 15 = system:0 2024-02-19T21:10:45.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.300+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee4017190 osd=7 4 2024-02-19T21:10:45.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.300+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 7 15 2024-02-19T21:10:45.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.314+0000 7feaf02d6700 1 -- 172.21.15.163:0/1151724448 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7feacc0051d0 con 0x7feae81042e0 2024-02-19T21:10:45.319 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:45.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:45.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:45.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.327+0000 7efefd399700 10 client.49892.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:10:45.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.327+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign -1 16 2024-02-19T21:10:45.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.327+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4017190 osd=7 3 2024-02-19T21:10:45.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.327+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4017190 osd=7 3 2024-02-19T21:10:45.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.327+0000 7efefd399700 20 client.49892.objecter _get_session s=0x7efee4017190 osd=7 3 2024-02-19T21:10:45.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.327+0000 7efefd399700 15 client.49892.objecter _session_command_op_remove -1 16 2024-02-19T21:10:45.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.327+0000 7efefd399700 20 client.49892.objecter get_session s=0x7efee4017190 osd=7 3 2024-02-19T21:10:45.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.327+0000 7efefd399700 15 client.49892.objecter _session_command_op_assign 7 16 2024-02-19T21:10:45.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.327+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4017190 osd=7 4 2024-02-19T21:10:45.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.327+0000 7efefd399700 10 client.49892.objecter _send_command 16 2024-02-19T21:10:45.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.327+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7efee4023d60 con 0x7efee4024840 2024-02-19T21:10:45.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efedffff700 1 -- 172.21.15.163:0/1316423178 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7efee4023d60 con 0x7efee4024840 2024-02-19T21:10:45.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efedffff700 10 client.49892.objecter ms_dispatch 0x7efef8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:10:45.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efedffff700 10 client.49892.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efedffff700 20 client.49892.objecter put_session s=0x7efee4017190 osd=7 4 2024-02-19T21:10:45.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efedffff700 15 client.49892.objecter _session_command_op_remove 7 16 2024-02-19T21:10:45.328 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:10:45.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 10 librados: watch_flush enter 2024-02-19T21:10:45.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 10 librados: watch_flush exit 2024-02-19T21:10:45.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 10 client.49892.objecter close_session for osd.0 2024-02-19T21:10:45.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 >> v1:172.21.15.163:6801/1559418115 conn(0x7efee4004530 legacy=0x7efee4002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee40051a0 osd=0 1 2024-02-19T21:10:45.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 10 client.49892.objecter close_session for osd.1 2024-02-19T21:10:45.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 >> v1:172.21.15.163:6805/3125047432 conn(0x7efee4004150 legacy=0x7efee400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4003fe0 osd=1 1 2024-02-19T21:10:45.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 10 client.49892.objecter close_session for osd.2 2024-02-19T21:10:45.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 >> v1:172.21.15.163:6809/1563158487 conn(0x7efee400d2f0 legacy=0x7efee400d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4003630 osd=2 1 2024-02-19T21:10:45.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 10 client.49892.objecter close_session for osd.3 2024-02-19T21:10:45.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 >> v1:172.21.15.163:6813/2112279769 conn(0x7efee4011ff0 legacy=0x7efee40143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4011970 osd=3 1 2024-02-19T21:10:45.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 10 client.49892.objecter close_session for osd.4 2024-02-19T21:10:45.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 >> v1:172.21.15.193:6800/3129716965 conn(0x7efee400c940 legacy=0x7efee4015d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4015ba0 osd=4 1 2024-02-19T21:10:45.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 10 client.49892.objecter close_session for osd.5 2024-02-19T21:10:45.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 >> v1:172.21.15.193:6804/805161991 conn(0x7efee4016960 legacy=0x7efee401d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efef77fe700 1 -- 172.21.15.163:0/1316423178 reap_dead start 2024-02-19T21:10:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efef77fe700 1 -- 172.21.15.163:0/1316423178 reap_dead start 2024-02-19T21:10:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee400cee0 osd=5 1 2024-02-19T21:10:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 10 client.49892.objecter close_session for osd.6 2024-02-19T21:10:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 >> v1:172.21.15.193:6808/787604615 conn(0x7efee401f1a0 legacy=0x7efee4021590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4016da0 osd=6 1 2024-02-19T21:10:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 10 client.49892.objecter close_session for osd.7 2024-02-19T21:10:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 >> v1:172.21.15.193:6812/157927432 conn(0x7efee4024840 legacy=0x7efee4024c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 20 client.49892.objecter put_session s=0x7efee4017190 osd=7 1 2024-02-19T21:10:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 20 client.49892.objecter shutdown clearing up homeless session... 2024-02-19T21:10:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efef77fe700 1 -- 172.21.15.163:0/1316423178 reap_dead start 2024-02-19T21:10:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 10 client.49892.objecter successfully canceled tick 2024-02-19T21:10:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 >> v1:172.21.15.163:6800/1552514681 conn(0x7efee0063d00 legacy=0x7efee00661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 >> v1:172.21.15.163:6790/0 conn(0x7efef80f94d0 legacy=0x7efef8114d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 shutdown_connections 2024-02-19T21:10:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.329+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 >> 172.21.15.163:0/1316423178 conn(0x7efef80f2470 msgr2=0x7efef8103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.330+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 shutdown_connections 2024-02-19T21:10:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.330+0000 7efefd399700 1 -- 172.21.15.163:0/1316423178 wait complete. 2024-02-19T21:10:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:45.330+0000 7efefd399700 1 librados: shutdown 2024-02-19T21:10:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:10:45.343 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:45.425 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:45.452 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:45.471 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:45.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:45.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:44.288782+0000 mon.b (mon.1) 595 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:45.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:44.299673+0000 mon.a (mon.0) 2580 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:45.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:45.159151+0000 mon.c (mon.2) 394 : audit [DBG] from='client.? v1:172.21.15.163:0/1316423178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:45.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:45.278525+0000 mon.a (mon.0) 2581 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:10:45.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:45.278600+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e438: 8 total, 8 up, 8 in 2024-02-19T21:10:45.576 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:45.603 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:44.288782+0000 mon.b (mon.1) 595 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:45.603 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:44.299673+0000 mon.a (mon.0) 2580 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:45.603 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:45.159151+0000 mon.c (mon.2) 394 : audit [DBG] from='client.? v1:172.21.15.163:0/1316423178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:45.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:45.278525+0000 mon.a (mon.0) 2581 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:10:45.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:45.278600+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e438: 8 total, 8 up, 8 in 2024-02-19T21:10:45.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:44.288782+0000 mon.b (mon.1) 595 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:45.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:44.299673+0000 mon.a (mon.0) 2580 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:45.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:45.159151+0000 mon.c (mon.2) 394 : audit [DBG] from='client.? v1:172.21.15.163:0/1316423178' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:45.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:45.278525+0000 mon.a (mon.0) 2581 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:10:45.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:45.278600+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e438: 8 total, 8 up, 8 in 2024-02-19T21:10:45.606 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:45.807 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:45.810 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:45.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:45.912 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:10:45.939 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:46.208 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 3, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:10:46.208 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:10:46.208 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-19T21:10:46.209 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-19T21:10:46.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:46.276+0000 7feadf779700 1 -- 172.21.15.163:0/1151724448 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v439) v1 ==== 91+0+0 (unknown 2278372611 0 0) 0x7feae0048380 con 0x7feae81042e0 2024-02-19T21:10:46.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:46.279+0000 7feaf02d6700 10 librados: watch_flush enter 2024-02-19T21:10:46.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:46.279+0000 7feaf02d6700 10 librados: watch_flush exit 2024-02-19T21:10:46.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:46.279+0000 7feaf02d6700 20 client.48706.objecter shutdown clearing up homeless session... 2024-02-19T21:10:46.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:46.279+0000 7feaf02d6700 10 client.48706.objecter successfully canceled tick 2024-02-19T21:10:46.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:46.280+0000 7feaf02d6700 1 -- 172.21.15.163:0/1151724448 >> v1:172.21.15.163:6800/1552514681 conn(0x7fead4063d80 legacy=0x7fead4066240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:46.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:46.280+0000 7feaf02d6700 1 -- 172.21.15.163:0/1151724448 >> v1:172.21.15.193:6789/0 conn(0x7feae81042e0 legacy=0x7feae8118410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:46.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:46.280+0000 7feaf02d6700 1 -- 172.21.15.163:0/1151724448 shutdown_connections 2024-02-19T21:10:46.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:46.280+0000 7feaf02d6700 1 -- 172.21.15.163:0/1151724448 >> 172.21.15.163:0/1151724448 conn(0x7feae80f2470 msgr2=0x7feae8103a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:46.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:46.281+0000 7feaf02d6700 1 -- 172.21.15.163:0/1151724448 shutdown_connections 2024-02-19T21:10:46.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:46.281+0000 7feaf02d6700 1 -- 172.21.15.163:0/1151724448 wait complete. 2024-02-19T21:10:46.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:46.281+0000 7feaf02d6700 1 librados: shutdown 2024-02-19T21:10:46.280 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:10:46.287 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:10:46.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:46.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:10:46.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:46.422 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:46.442 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:46.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:46.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:46.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:44.663341+0000 osd.4 (osd.4) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:10:46.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:44.663495+0000 osd.4 (osd.4) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:10:46.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:45.222147+0000 mgr.y (mgr.14550) 398 : cluster [DBG] pgmap v641: 228 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering, 32 creating+peering, 192 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s; 19/1059 objects misplaced (1.794%); 520 B/s, 69 objects/s recovering 2024-02-19T21:10:46.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:45.296416+0000 mon.b (mon.1) 596 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:10:46.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:45.314578+0000 mon.b (mon.1) 597 : audit [INF] from='client.? v1:172.21.15.163:0/1151724448' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:10:46.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:45.315176+0000 mon.a (mon.0) 2583 : audit [INF] from='client.48706 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:10:46.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:46.272938+0000 mon.a (mon.0) 2584 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:10:46.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:46.276726+0000 mon.a (mon.0) 2585 : audit [INF] from='client.48706 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:10:46.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:46.276819+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e439: 8 total, 8 up, 8 in 2024-02-19T21:10:46.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:44.663341+0000 osd.4 (osd.4) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:10:46.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:44.663495+0000 osd.4 (osd.4) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:10:46.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:45.222147+0000 mgr.y (mgr.14550) 398 : cluster [DBG] pgmap v641: 228 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering, 32 creating+peering, 192 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s; 19/1059 objects misplaced (1.794%); 520 B/s, 69 objects/s recovering 2024-02-19T21:10:46.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:45.296416+0000 mon.b (mon.1) 596 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:10:46.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:45.314578+0000 mon.b (mon.1) 597 : audit [INF] from='client.? v1:172.21.15.163:0/1151724448' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:10:46.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:45.315176+0000 mon.a (mon.0) 2583 : audit [INF] from='client.48706 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:10:46.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:46.272938+0000 mon.a (mon.0) 2584 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:10:46.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:46.276726+0000 mon.a (mon.0) 2585 : audit [INF] from='client.48706 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:10:46.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:46.276819+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e439: 8 total, 8 up, 8 in 2024-02-19T21:10:46.548 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:46.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:44.663341+0000 osd.4 (osd.4) 147 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:10:46.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:44.663495+0000 osd.4 (osd.4) 148 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:10:46.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:45.222147+0000 mgr.y (mgr.14550) 398 : cluster [DBG] pgmap v641: 228 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering, 32 creating+peering, 192 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s; 19/1059 objects misplaced (1.794%); 520 B/s, 69 objects/s recovering 2024-02-19T21:10:46.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:45.296416+0000 mon.b (mon.1) 596 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:10:46.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:45.314578+0000 mon.b (mon.1) 597 : audit [INF] from='client.? v1:172.21.15.163:0/1151724448' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:10:46.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:45.315176+0000 mon.a (mon.0) 2583 : audit [INF] from='client.48706 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:10:46.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:46.272938+0000 mon.a (mon.0) 2584 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:10:46.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:46.276726+0000 mon.a (mon.0) 2585 : audit [INF] from='client.48706 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:10:46.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:46.276819+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e439: 8 total, 8 up, 8 in 2024-02-19T21:10:46.573 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:46.592 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:46.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:46.696 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:46.724 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:46.742 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:46.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:46.846 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:46.873 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:47.071 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:47.074 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:47.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:47.175 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:47.201 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:47.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:46.314057+0000 mon.b (mon.1) 598 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:47.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:46.314360+0000 mon.a (mon.0) 2587 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:47.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:46.314872+0000 mon.b (mon.1) 599 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-65"}]: dispatch 2024-02-19T21:10:47.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:46.315083+0000 mon.a (mon.0) 2588 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-65"}]: dispatch 2024-02-19T21:10:47.667 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:46.314057+0000 mon.b (mon.1) 598 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:47.667 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:46.314360+0000 mon.a (mon.0) 2587 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:47.667 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:46.314872+0000 mon.b (mon.1) 599 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-65"}]: dispatch 2024-02-19T21:10:47.667 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:46.315083+0000 mon.a (mon.0) 2588 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-65"}]: dispatch 2024-02-19T21:10:47.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:46.314057+0000 mon.b (mon.1) 598 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:47.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:46.314360+0000 mon.a (mon.0) 2587 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:47.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:46.314872+0000 mon.b (mon.1) 599 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-65"}]: dispatch 2024-02-19T21:10:47.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:46.315083+0000 mon.a (mon.0) 2588 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-65"}]: dispatch 2024-02-19T21:10:47.950 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:47.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:48.054 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:48.069 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:10:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:10:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:10:48.083 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:48.185 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:48.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:48.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:48.206 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:48.208 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:10:48.208 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:10:48.290 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:10:48.316 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:10:48.321 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:48.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:47.222580+0000 mgr.y (mgr.14550) 399 : cluster [DBG] pgmap v644: 228 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering, 32 creating+peering, 192 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s; 19/1059 objects misplaced (1.794%); 520 B/s, 69 objects/s recovering 2024-02-19T21:10:48.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:47.316677+0000 mon.a (mon.0) 2589 : cluster [DBG] osdmap e440: 8 total, 8 up, 8 in 2024-02-19T21:10:48.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:47.222580+0000 mgr.y (mgr.14550) 399 : cluster [DBG] pgmap v644: 228 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering, 32 creating+peering, 192 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s; 19/1059 objects misplaced (1.794%); 520 B/s, 69 objects/s recovering 2024-02-19T21:10:48.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:47.316677+0000 mon.a (mon.0) 2589 : cluster [DBG] osdmap e440: 8 total, 8 up, 8 in 2024-02-19T21:10:48.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:47.222580+0000 mgr.y (mgr.14550) 399 : cluster [DBG] pgmap v644: 228 pgs: 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering, 32 creating+peering, 192 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s; 19/1059 objects misplaced (1.794%); 520 B/s, 69 objects/s recovering 2024-02-19T21:10:48.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:47.316677+0000 mon.a (mon.0) 2589 : cluster [DBG] osdmap e440: 8 total, 8 up, 8 in 2024-02-19T21:10:48.673 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:10:48.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:48.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:10:48.777 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:48.802 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:48.821 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:48.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:48.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:48.925 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:48.952 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:49.065 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:10:49.100 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:49.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:49.205 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:49.232 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:49.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:48.323025+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e441: 8 total, 8 up, 8 in 2024-02-19T21:10:49.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:48.326134+0000 mon.b (mon.1) 600 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:49.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:48.326745+0000 mon.a (mon.0) 2591 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:49.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:49.281793+0000 mon.a (mon.0) 2592 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:10:49.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:49.281952+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e442: 8 total, 8 up, 8 in 2024-02-19T21:10:49.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:49.304164+0000 mon.b (mon.1) 601 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:10:49.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:48.323025+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e441: 8 total, 8 up, 8 in 2024-02-19T21:10:49.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:48.326134+0000 mon.b (mon.1) 600 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:49.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:48.326745+0000 mon.a (mon.0) 2591 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:49.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:49.281793+0000 mon.a (mon.0) 2592 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:10:49.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:49.281952+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e442: 8 total, 8 up, 8 in 2024-02-19T21:10:49.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:49.304164+0000 mon.b (mon.1) 601 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:10:49.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:48.323025+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e441: 8 total, 8 up, 8 in 2024-02-19T21:10:49.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:48.326134+0000 mon.b (mon.1) 600 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:49.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:48.326745+0000 mon.a (mon.0) 2591 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:49.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:49.281793+0000 mon.a (mon.0) 2592 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:10:49.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:49.281952+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e442: 8 total, 8 up, 8 in 2024-02-19T21:10:49.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:49.304164+0000 mon.b (mon.1) 601 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:10:50.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.148+0000 7f82f3b5a700 1 -- 172.21.15.163:0/567367344 >> v1:172.21.15.163:6790/0 conn(0x7f82ec0fc8a0 legacy=0x7f82ec0fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:50.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.148+0000 7f82f3b5a700 1 -- 172.21.15.163:0/567367344 shutdown_connections 2024-02-19T21:10:50.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.148+0000 7f82f3b5a700 1 -- 172.21.15.163:0/567367344 >> 172.21.15.163:0/567367344 conn(0x7f82ec06fdc0 msgr2=0x7f82ec0f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:50.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.148+0000 7f82f3b5a700 1 -- 172.21.15.163:0/567367344 shutdown_connections 2024-02-19T21:10:50.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.148+0000 7f82f3b5a700 1 -- 172.21.15.163:0/567367344 wait complete. 2024-02-19T21:10:50.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.148+0000 7f82f3b5a700 1 librados: starting msgr at 2024-02-19T21:10:50.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.148+0000 7f82f3b5a700 1 librados: starting objecter 2024-02-19T21:10:50.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.148+0000 7f82f3b5a700 1 Processor -- start 2024-02-19T21:10:50.149 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.148+0000 7f82f3b5a700 1 -- start start 2024-02-19T21:10:50.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.148+0000 7f82f3b5a700 1 librados: setting wanted keys 2024-02-19T21:10:50.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.148+0000 7f82f3b5a700 1 librados: calling monclient init 2024-02-19T21:10:50.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82f3b5a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f82ec117300 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82f3b5a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f82ec1184e0 con 0x7f82ec0ff680 2024-02-19T21:10:50.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82f3b5a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f82ec1196e0 con 0x7f82ec1031c0 2024-02-19T21:10:50.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82f18f6700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f82ec0fc8a0 0x7f82ec0f9bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:35654/0 (socket says 172.21.15.163:35654) 2024-02-19T21:10:50.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82f18f6700 1 -- 172.21.15.163:0/3123303898 learned_addr learned my addr 172.21.15.163:0/3123303898 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:50.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 10 client.?.objecter ms_handle_connect 0x7f82ec0fc8a0 2024-02-19T21:10:50.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:50.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598901454 0 0) 0x7f82ec117300 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d8003030 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 10 client.?.objecter ms_handle_connect 0x7f82ec0ff680 2024-02-19T21:10:50.150 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:50.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 10 client.?.objecter ms_handle_connect 0x7f82ec1031c0 2024-02-19T21:10:50.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:50.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083586452 0 0) 0x7f82ec1184e0 con 0x7f82ec0ff680 2024-02-19T21:10:50.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82ec117300 con 0x7f82ec0ff680 2024-02-19T21:10:50.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056036603 0 0) 0x7f82ec1196e0 con 0x7f82ec1031c0 2024-02-19T21:10:50.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82ec1184e0 con 0x7f82ec1031c0 2024-02-19T21:10:50.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2515721888 0 0) 0x7f82d8003030 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82ec1196e0 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f82e8003030 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1521523891 0 0) 0x7f82ec117300 con 0x7f82ec0ff680 2024-02-19T21:10:50.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82d8003030 con 0x7f82ec0ff680 2024-02-19T21:10:50.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f82dc001960 con 0x7f82ec0ff680 2024-02-19T21:10:50.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4087297608 0 0) 0x7f82ec1184e0 con 0x7f82ec1031c0 2024-02-19T21:10:50.151 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82ec117300 con 0x7f82ec1031c0 2024-02-19T21:10:50.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3661523192 0 0) 0x7f82ec1196e0 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 >> v1:172.21.15.163:6790/0 conn(0x7f82ec1031c0 legacy=0x7f82ec1159c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:50.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 >> v1:172.21.15.193:6789/0 conn(0x7f82ec0ff680 legacy=0x7f82ec0fa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:50.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec11a8e0 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.149+0000 7f82f3b5a700 1 -- 172.21.15.163:0/3123303898 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec1186d0 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.150+0000 7f82f3b5a700 10 client.23178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:50.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.150+0000 7f82f3b5a700 1 -- 172.21.15.163:0/3123303898 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ec118b40 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.150+0000 7f82f3b5a700 1 librados: init done 2024-02-19T21:10:50.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.150+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f82e8001bd0 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.150+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f82e80039b0 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.152 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.150+0000 7f82f3b5a700 1 -- 172.21.15.163:0/3123303898 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d0004f20 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.151+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f82e8003bb0 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.151+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(442..442 src has 43..442) v4 ==== 8092+0+0 (unknown 1200077923 0 0) 0x7f82e80790c0 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.151+0000 7f82e2ffd700 10 client.23178.objecter ms_dispatch 0x7f82ec001550 osd_map(442..442 src has 43..442) v4 2024-02-19T21:10:50.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.151+0000 7f82e2ffd700 3 client.23178.objecter handle_osd_map got epochs [442,442] > 0 2024-02-19T21:10:50.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.151+0000 7f82e2ffd700 3 client.23178.objecter handle_osd_map decoding full epoch 442 2024-02-19T21:10:50.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.151+0000 7f82e2ffd700 20 client.23178.objecter dump_active .. 0 homeless 2024-02-19T21:10:50.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.152+0000 7f82e2ffd700 10 client.23178.objecter ms_handle_connect 0x7f82d8063e50 2024-02-19T21:10:50.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.156+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f82e804c4f0 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.184 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:50.190 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:50.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:50.291 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:50.320 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:50.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.350+0000 7f82f3b5a700 1 -- 172.21.15.163:0/3123303898 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f82d00051d0 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.528+0000 7f82e2ffd700 1 -- 172.21.15.163:0/3123303898 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v443) v1 ==== 81+0+45866 (unknown 626668285 0 2105295158) 0x7f82e804a960 con 0x7f82ec0fc8a0 2024-02-19T21:10:50.528 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:10:50.528 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:10:50.528 INFO:teuthology.orchestra.run.smithi163.stdout: "epoch": 443, 2024-02-19T21:10:50.529 INFO:teuthology.orchestra.run.smithi163.stdout: "fsid": "2591809a-cf69-11ee-95bb-87774f69a715", 2024-02-19T21:10:50.529 INFO:teuthology.orchestra.run.smithi163.stdout: "created": "2024-02-19T20:55:37.794885+0000", 2024-02-19T21:10:50.529 INFO:teuthology.orchestra.run.smithi163.stdout: "modified": "2024-02-19T21:10:50.273875+0000", 2024-02-19T21:10:50.529 INFO:teuthology.orchestra.run.smithi163.stdout: "last_up_change": "2024-02-19T21:01:09.214134+0000", 2024-02-19T21:10:50.529 INFO:teuthology.orchestra.run.smithi163.stdout: "last_in_change": "2024-02-19T21:10:40.268547+0000", 2024-02-19T21:10:50.529 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-19T21:10:50.529 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_num": 5804032, 2024-02-19T21:10:50.529 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_set": [ 2024-02-19T21:10:50.529 INFO:teuthology.orchestra.run.smithi163.stdout: "nodeep-scrub", 2024-02-19T21:10:50.529 INFO:teuthology.orchestra.run.smithi163.stdout: "pglog_hardlimit", 2024-02-19T21:10:50.529 INFO:teuthology.orchestra.run.smithi163.stdout: "purged_snapdirs", 2024-02-19T21:10:50.529 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_deletes", 2024-02-19T21:10:50.530 INFO:teuthology.orchestra.run.smithi163.stdout: "sortbitwise" 2024-02-19T21:10:50.530 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:10:50.530 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_version": 103, 2024-02-19T21:10:50.530 INFO:teuthology.orchestra.run.smithi163.stdout: "full_ratio": 0.94999998807907104, 2024-02-19T21:10:50.530 INFO:teuthology.orchestra.run.smithi163.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-19T21:10:50.530 INFO:teuthology.orchestra.run.smithi163.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-19T21:10:50.530 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_snapshot": "", 2024-02-19T21:10:50.530 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_max": 215, 2024-02-19T21:10:50.530 INFO:teuthology.orchestra.run.smithi163.stdout: "max_osd": 8, 2024-02-19T21:10:50.530 INFO:teuthology.orchestra.run.smithi163.stdout: "require_min_compat_client": "luminous", 2024-02-19T21:10:50.530 INFO:teuthology.orchestra.run.smithi163.stdout: "min_compat_client": "luminous", 2024-02-19T21:10:50.530 INFO:teuthology.orchestra.run.smithi163.stdout: "require_osd_release": "pacific", 2024-02-19T21:10:50.530 INFO:teuthology.orchestra.run.smithi163.stdout: "pools": [ 2024-02-19T21:10:50.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.531 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 1, 2024-02-19T21:10:50.531 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "device_health_metrics", 2024-02-19T21:10:50.531 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T20:59:09.425169+0000", 2024-02-19T21:10:50.531 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:10:50.531 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:10:50.531 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:10:50.531 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:10:50.531 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:10:50.531 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:10:50.531 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:10:50.531 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:10:50.531 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:10:50.531 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:10:50.531 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:10:50.532 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:10:50.532 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 1, 2024-02-19T21:10:50.532 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 1, 2024-02-19T21:10:50.532 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 1, 2024-02-19T21:10:50.532 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 1, 2024-02-19T21:10:50.532 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 1, 2024-02-19T21:10:50.532 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:10:50.532 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:10:50.532 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:10:50.532 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:10:50.532 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:10:50.532 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:10:50.532 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:10:50.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.533 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "74", 2024-02-19T21:10:50.533 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:10:50.533 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:10:50.533 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:10:50.533 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:10:50.533 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:10:50.533 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:10:50.533 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:10:50.533 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:10:50.533 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:10:50.533 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:10:50.533 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:10:50.533 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:10:50.534 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:10:50.534 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:10:50.534 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:10:50.534 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:10:50.534 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:10:50.534 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:10:50.534 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:10:50.534 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:10:50.534 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:10:50.534 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:10:50.534 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:10:50.534 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:10:50.534 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:10:50.534 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:10:50.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.535 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:10:50.535 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:10:50.535 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:10:50.535 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:10:50.535 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:10:50.535 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:10:50.535 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:10:50.535 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:10:50.535 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:10:50.535 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:10:50.535 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:10:50.535 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:10:50.536 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_max": 32, 2024-02-19T21:10:50.536 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_min": 1 2024-02-19T21:10:50.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.536 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:10:50.536 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_devicehealth": {} 2024-02-19T21:10:50.536 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.536 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 2, 2024-02-19T21:10:50.536 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".rgw.root", 2024-02-19T21:10:50.536 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:14.658945+0000", 2024-02-19T21:10:50.536 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:10:50.536 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:10:50.536 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:10:50.537 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:10:50.537 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:10:50.537 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:10:50.537 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:10:50.537 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:10:50.537 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:10:50.537 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:10:50.537 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:10:50.537 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:10:50.537 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:10:50.537 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:10:50.537 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:10:50.537 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:10:50.537 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:10:50.537 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:10:50.538 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "2.20", 2024-02-19T21:10:50.538 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 309, 2024-02-19T21:10:50.538 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 308, 2024-02-19T21:10:50.538 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 308, 2024-02-19T21:10:50.538 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "58'1", 2024-02-19T21:10:50.538 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "58'1" 2024-02-19T21:10:50.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.538 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "416", 2024-02-19T21:10:50.538 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:10:50.538 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "309", 2024-02-19T21:10:50.538 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "307", 2024-02-19T21:10:50.538 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:10:50.538 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:10:50.538 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:10:50.539 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:10:50.539 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:10:50.539 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:10:50.539 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:10:50.539 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:10:50.539 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:10:50.539 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:10:50.539 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:10:50.539 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:10:50.539 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:10:50.539 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:10:50.539 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:10:50.539 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:10:50.539 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:10:50.539 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:10:50.540 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:10:50.540 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:10:50.540 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:10:50.540 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:10:50.540 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:10:50.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.540 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:10:50.540 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:10:50.540 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:10:50.540 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:10:50.540 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:10:50.540 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:10:50.540 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:10:50.541 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:10:50.541 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:10:50.541 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:10:50.541 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:10:50.541 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:10:50.541 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:10:50.541 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:10:50.541 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.541 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 3, 2024-02-19T21:10:50.541 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "iscsi", 2024-02-19T21:10:50.541 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:14.958800+0000", 2024-02-19T21:10:50.542 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:10:50.542 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:10:50.542 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:10:50.542 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:10:50.542 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:10:50.542 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:10:50.542 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:10:50.542 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:10:50.542 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:10:50.542 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:10:50.542 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:10:50.542 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:10:50.542 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 3, 2024-02-19T21:10:50.542 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 3, 2024-02-19T21:10:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 3, 2024-02-19T21:10:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 3, 2024-02-19T21:10:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 3, 2024-02-19T21:10:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:10:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:10:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:10:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:10:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:10:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:10:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:10:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "100", 2024-02-19T21:10:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:10:50.543 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:10:50.544 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:10:50.544 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:10:50.544 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:10:50.544 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:10:50.544 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:10:50.544 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:10:50.544 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:10:50.544 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:10:50.544 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:10:50.544 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:10:50.544 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:10:50.544 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:10:50.544 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:10:50.544 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:10:50.545 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:10:50.545 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:10:50.545 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:10:50.545 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:10:50.545 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:10:50.545 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:10:50.545 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:10:50.545 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:10:50.545 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:10:50.545 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:10:50.545 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.545 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:10:50.545 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:10:50.545 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:10:50.546 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:10:50.546 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:10:50.546 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:10:50.546 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:10:50.546 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:10:50.546 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:10:50.546 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:10:50.546 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:10:50.546 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:10:50.546 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:10:50.546 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd": {} 2024-02-19T21:10:50.546 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.546 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.546 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.546 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 4, 2024-02-19T21:10:50.547 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.log", 2024-02-19T21:10:50.547 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:16.543816+0000", 2024-02-19T21:10:50.547 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:10:50.547 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:10:50.547 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:10:50.547 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:10:50.547 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:10:50.547 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:10:50.547 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:10:50.547 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:10:50.547 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:10:50.547 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:10:50.547 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:10:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:10:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:10:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:10:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:10:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:10:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:10:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:10:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:10:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:10:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:10:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:10:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:10:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:10:50.548 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.549 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "61", 2024-02-19T21:10:50.549 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:10:50.549 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:10:50.549 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:10:50.549 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:10:50.549 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:10:50.549 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:10:50.549 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:10:50.549 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:10:50.549 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:10:50.549 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:10:50.549 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:10:50.549 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:10:50.549 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:10:50.550 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:10:50.550 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:10:50.550 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:10:50.550 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:10:50.550 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:10:50.550 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:10:50.550 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:10:50.550 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:10:50.550 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:10:50.550 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:10:50.550 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:10:50.550 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:10:50.550 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:10:50.550 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.551 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:10:50.551 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:10:50.551 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:10:50.551 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:10:50.551 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:10:50.551 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:10:50.551 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:10:50.551 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:10:50.551 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:10:50.551 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:10:50.551 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:10:50.551 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:10:50.551 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:10:50.551 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:10:50.552 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.552 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.552 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.552 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 5, 2024-02-19T21:10:50.552 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.control", 2024-02-19T21:10:50.552 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:18.521482+0000", 2024-02-19T21:10:50.552 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:10:50.552 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:10:50.552 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:10:50.552 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:10:50.552 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:10:50.552 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:10:50.552 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:10:50.552 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:10:50.553 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:10:50.553 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:10:50.553 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:10:50.553 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:10:50.553 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:10:50.553 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:10:50.553 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:10:50.553 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:10:50.553 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:10:50.553 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:10:50.553 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:10:50.553 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:10:50.553 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:10:50.553 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:10:50.554 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:10:50.554 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:10:50.554 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.554 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "63", 2024-02-19T21:10:50.554 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:10:50.554 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:10:50.554 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:10:50.554 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:10:50.554 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:10:50.554 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:10:50.554 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:10:50.554 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:10:50.554 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:10:50.554 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:10:50.555 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:10:50.555 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:10:50.555 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:10:50.555 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:10:50.555 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:10:50.555 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:10:50.555 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:10:50.555 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:10:50.555 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:10:50.555 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:10:50.555 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:10:50.555 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:10:50.555 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:10:50.555 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:10:50.556 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:10:50.556 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:10:50.556 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.556 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:10:50.556 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:10:50.556 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:10:50.556 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:10:50.556 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:10:50.556 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:10:50.556 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:10:50.556 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:10:50.556 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:10:50.556 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:10:50.556 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:10:50.557 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:10:50.557 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:10:50.557 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:10:50.557 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.557 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.557 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.557 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 6, 2024-02-19T21:10:50.557 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.meta", 2024-02-19T21:10:50.557 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:20.586564+0000", 2024-02-19T21:10:50.557 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:10:50.557 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:10:50.557 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:10:50.557 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:10:50.557 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:10:50.558 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:10:50.558 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:10:50.558 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:10:50.558 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:10:50.558 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:10:50.558 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:10:50.558 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:10:50.558 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:10:50.558 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:10:50.558 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:10:50.558 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:10:50.558 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:10:50.558 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:10:50.558 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:10:50.559 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:10:50.559 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:10:50.559 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:10:50.559 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:10:50.559 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:10:50.559 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.559 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "65", 2024-02-19T21:10:50.559 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:10:50.559 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:10:50.559 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:10:50.559 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:10:50.559 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:10:50.559 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:10:50.560 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:10:50.560 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:10:50.560 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:10:50.560 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:10:50.560 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:10:50.560 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:10:50.560 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:10:50.560 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:10:50.560 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:10:50.560 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:10:50.560 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:10:50.560 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:10:50.560 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:10:50.560 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:10:50.561 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:10:50.561 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:10:50.561 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:10:50.561 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:10:50.561 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:10:50.561 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:10:50.561 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.561 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:10:50.561 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:10:50.561 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:10:50.561 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:10:50.561 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:10:50.561 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:10:50.562 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:10:50.562 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:10:50.562 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:10:50.562 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:10:50.562 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:10:50.562 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:10:50.562 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_bias": 4 2024-02-19T21:10:50.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.562 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:10:50.562 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:10:50.562 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.562 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.562 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.562 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 16, 2024-02-19T21:10:50.563 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-19T21:10:50.563 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:35.333192+0000", 2024-02-19T21:10:50.563 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:10:50.563 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:10:50.563 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:10:50.563 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:10:50.563 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:10:50.563 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:10:50.563 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:10:50.563 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:10:50.563 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:10:50.563 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:10:50.563 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:10:50.564 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:10:50.564 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:10:50.564 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:10:50.564 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:10:50.564 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:10:50.564 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:10:50.564 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:10:50.564 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:10:50.564 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:10:50.564 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:10:50.564 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:10:50.564 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:10:50.564 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:10:50.565 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.565 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "101", 2024-02-19T21:10:50.565 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:10:50.565 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:10:50.565 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:10:50.565 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:10:50.565 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:10:50.565 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:10:50.565 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:10:50.565 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:10:50.565 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:10:50.565 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:10:50.565 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:10:50.566 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:10:50.566 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:10:50.566 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:10:50.566 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:10:50.566 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:10:50.566 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:10:50.566 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:10:50.566 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:10:50.566 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:10:50.566 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:10:50.566 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:10:50.566 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:10:50.566 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:10:50.566 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:10:50.567 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:10:50.567 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.567 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:10:50.567 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:10:50.567 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:10:50.567 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:10:50.567 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:10:50.567 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:10:50.567 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:10:50.567 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:10:50.567 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:10:50.567 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:10:50.567 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:10:50.567 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:10:50.567 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": {} 2024-02-19T21:10:50.568 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.568 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.568 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 61, 2024-02-19T21:10:50.568 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "test-rados-api-smithi163-83928-6", 2024-02-19T21:10:50.568 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:52.991355+0000", 2024-02-19T21:10:50.568 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8193, 2024-02-19T21:10:50.568 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-19T21:10:50.568 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:10:50.568 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:10:50.568 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:10:50.568 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:10:50.568 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:10:50.568 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:10:50.569 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:10:50.569 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:10:50.569 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:10:50.569 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:10:50.569 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:10:50.569 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:10:50.569 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:10:50.569 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:10:50.569 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:10:50.569 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:10:50.569 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:10:50.569 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:10:50.570 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:10:50.570 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:10:50.570 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:10:50.570 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:10:50.570 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.570 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "439", 2024-02-19T21:10:50.570 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "386", 2024-02-19T21:10:50.570 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "386", 2024-02-19T21:10:50.570 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "386", 2024-02-19T21:10:50.570 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:10:50.570 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:10:50.571 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 31, 2024-02-19T21:10:50.571 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 439, 2024-02-19T21:10:50.571 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:10:50.571 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:10:50.571 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:10:50.571 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:10:50.571 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:10:50.571 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:10:50.571 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:10:50.571 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:10:50.571 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:10:50.571 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:10:50.571 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:10:50.571 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:10:50.572 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:10:50.572 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:10:50.572 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:10:50.572 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:10:50.572 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:10:50.572 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:10:50.572 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:10:50.572 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.573 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:10:50.573 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:10:50.573 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:10:50.573 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:10:50.573 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:10:50.573 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:10:50.573 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:10:50.573 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:10:50.573 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:10:50.573 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:10:50.573 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:10:50.573 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:10:50.573 INFO:teuthology.orchestra.run.smithi163.stdout: "fingerprint_algorithm": "sha1" 2024-02-19T21:10:50.573 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.574 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:10:50.574 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-02-19T21:10:50.574 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.574 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.574 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:10:50.574 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-02-19T21:10:50.574 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.574 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-02-19T21:10:50.574 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088", 2024-02-19T21:10:50.574 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:10:50.574 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:10:50.574 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:10:50.574 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0, 2024-02-19T21:10:50.574 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:10:50.575 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:10:50.575 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 8, 2024-02-19T21:10:50.575 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 218, 2024-02-19T21:10:50.575 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:10:50.575 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:10:50.575 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:10:50.575 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.575 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.575 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.575 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6801", 2024-02-19T21:10:50.575 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:10:50.575 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.575 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.575 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.576 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:10:50.576 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.576 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.576 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.576 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6802", 2024-02-19T21:10:50.576 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:10:50.576 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.576 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.576 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.576 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:10:50.576 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.576 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.576 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.577 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6804", 2024-02-19T21:10:50.577 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:10:50.577 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.577 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.577 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:10:50.577 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.577 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.577 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.577 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6803", 2024-02-19T21:10:50.577 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:10:50.577 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.577 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.577 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.577 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6801/1559418115", 2024-02-19T21:10:50.578 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6802/1559418115", 2024-02-19T21:10:50.578 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6804/1559418115", 2024-02-19T21:10:50.578 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6803/1559418115", 2024-02-19T21:10:50.578 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:10:50.578 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:10:50.578 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:10:50.578 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.578 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.578 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.578 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-02-19T21:10:50.578 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "e74a65fa-cb9c-42c3-9293-602cf7e9dc67", 2024-02-19T21:10:50.578 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:10:50.579 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:10:50.579 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:10:50.579 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:10:50.579 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:10:50.579 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:10:50.579 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 13, 2024-02-19T21:10:50.579 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 441, 2024-02-19T21:10:50.579 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:10:50.579 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:10:50.579 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:10:50.579 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.579 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.579 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.579 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6805", 2024-02-19T21:10:50.580 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:10:50.580 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.580 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.580 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:10:50.580 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.580 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.580 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.580 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6806", 2024-02-19T21:10:50.580 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:10:50.580 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.580 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.580 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.580 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:10:50.581 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.581 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.581 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6808", 2024-02-19T21:10:50.581 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:10:50.581 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.581 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.581 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.581 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:10:50.581 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.581 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.581 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.581 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6807", 2024-02-19T21:10:50.582 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:10:50.582 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.582 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.582 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6805/3125047432", 2024-02-19T21:10:50.582 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6806/3125047432", 2024-02-19T21:10:50.582 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6808/3125047432", 2024-02-19T21:10:50.582 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6807/3125047432", 2024-02-19T21:10:50.582 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:10:50.582 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:10:50.582 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:10:50.582 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.582 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.582 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.582 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-02-19T21:10:50.583 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "6980a625-cdc9-487e-8f8c-7997ba716a97", 2024-02-19T21:10:50.583 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:10:50.583 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:10:50.583 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:10:50.583 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:10:50.583 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:10:50.583 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:10:50.583 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 18, 2024-02-19T21:10:50.583 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 441, 2024-02-19T21:10:50.584 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:10:50.584 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:10:50.584 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:10:50.584 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.584 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.584 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.584 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6809", 2024-02-19T21:10:50.584 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:10:50.584 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.584 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.584 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.584 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:10:50.584 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.585 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.585 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6810", 2024-02-19T21:10:50.585 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:10:50.585 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.585 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.585 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.585 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:10:50.585 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.585 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.585 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.585 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6812", 2024-02-19T21:10:50.585 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:10:50.585 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.586 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.586 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:10:50.586 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.586 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.586 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.586 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6811", 2024-02-19T21:10:50.586 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:10:50.586 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.586 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.586 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.586 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6809/1563158487", 2024-02-19T21:10:50.586 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6810/1563158487", 2024-02-19T21:10:50.586 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6812/1563158487", 2024-02-19T21:10:50.586 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6811/1563158487", 2024-02-19T21:10:50.587 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:10:50.587 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:10:50.587 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:10:50.587 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.587 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.587 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.587 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-02-19T21:10:50.587 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5", 2024-02-19T21:10:50.587 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:10:50.587 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:10:50.587 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:10:50.587 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:10:50.587 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:10:50.587 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:10:50.588 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 25, 2024-02-19T21:10:50.588 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 441, 2024-02-19T21:10:50.588 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:10:50.588 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:10:50.588 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:10:50.588 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.588 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.588 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.588 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6813", 2024-02-19T21:10:50.588 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:10:50.588 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.588 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.588 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.589 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:10:50.589 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.589 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.589 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.589 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6814", 2024-02-19T21:10:50.589 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:10:50.589 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.589 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.589 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.589 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:10:50.589 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.589 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.589 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.589 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6816", 2024-02-19T21:10:50.590 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:10:50.590 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.590 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.590 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:10:50.590 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.590 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.590 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.590 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6815", 2024-02-19T21:10:50.590 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:10:50.590 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.590 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.590 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.590 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6813/2112279769", 2024-02-19T21:10:50.591 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6814/2112279769", 2024-02-19T21:10:50.591 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6816/2112279769", 2024-02-19T21:10:50.591 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6815/2112279769", 2024-02-19T21:10:50.591 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:10:50.591 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:10:50.591 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:10:50.591 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.591 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.591 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.591 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-02-19T21:10:50.591 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0", 2024-02-19T21:10:50.591 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:10:50.591 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:10:50.592 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:10:50.592 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:10:50.592 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:10:50.592 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:10:50.592 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 31, 2024-02-19T21:10:50.592 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 441, 2024-02-19T21:10:50.592 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:10:50.592 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:10:50.592 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:10:50.592 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.592 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.592 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.592 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6800", 2024-02-19T21:10:50.592 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:10:50.593 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.593 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.593 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.593 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:10:50.593 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.593 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.593 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.593 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6801", 2024-02-19T21:10:50.593 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:10:50.593 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.593 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.593 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.593 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:10:50.594 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.594 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.594 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6803", 2024-02-19T21:10:50.594 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:10:50.594 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.594 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.594 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.594 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:10:50.594 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.594 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.594 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.594 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6802", 2024-02-19T21:10:50.594 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:10:50.594 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.595 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.595 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6800/3129716965", 2024-02-19T21:10:50.595 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6801/3129716965", 2024-02-19T21:10:50.595 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6803/3129716965", 2024-02-19T21:10:50.595 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6802/3129716965", 2024-02-19T21:10:50.595 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:10:50.595 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:10:50.595 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:10:50.595 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.595 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.595 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.595 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-02-19T21:10:50.595 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f", 2024-02-19T21:10:50.595 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:10:50.596 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:10:50.596 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:10:50.596 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:10:50.596 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:10:50.596 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:10:50.596 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 37, 2024-02-19T21:10:50.596 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 442, 2024-02-19T21:10:50.596 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:10:50.596 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:10:50.596 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:10:50.596 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.596 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.597 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.597 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6804", 2024-02-19T21:10:50.597 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:10:50.597 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.597 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.597 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.597 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:10:50.597 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.597 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.597 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.597 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6805", 2024-02-19T21:10:50.597 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:10:50.597 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.597 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.598 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:10:50.598 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.598 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.598 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6807", 2024-02-19T21:10:50.598 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:10:50.598 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.598 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.598 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.598 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:10:50.598 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.598 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.598 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.598 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6806", 2024-02-19T21:10:50.599 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:10:50.599 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.599 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.599 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6804/805161991", 2024-02-19T21:10:50.599 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6805/805161991", 2024-02-19T21:10:50.599 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6807/805161991", 2024-02-19T21:10:50.599 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6806/805161991", 2024-02-19T21:10:50.599 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:10:50.599 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:10:50.599 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:10:50.599 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.599 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.599 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.599 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-02-19T21:10:50.600 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "302c95df-3842-4818-8afd-d79d8184684f", 2024-02-19T21:10:50.600 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:10:50.600 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:10:50.600 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:10:50.600 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0.750579833984375, 2024-02-19T21:10:50.600 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:10:50.600 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:10:50.600 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 45, 2024-02-19T21:10:50.600 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 441, 2024-02-19T21:10:50.601 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:10:50.601 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:10:50.601 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:10:50.601 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.601 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.601 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.601 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6808", 2024-02-19T21:10:50.601 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:10:50.601 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.601 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.601 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.602 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:10:50.602 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.602 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.602 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6809", 2024-02-19T21:10:50.602 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:10:50.602 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.602 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.602 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.602 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:10:50.602 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.602 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.602 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.603 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6811", 2024-02-19T21:10:50.603 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:10:50.603 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.603 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.603 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.603 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:10:50.603 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.603 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.603 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.603 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6810", 2024-02-19T21:10:50.603 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:10:50.603 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.603 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.604 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6808/787604615", 2024-02-19T21:10:50.604 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6809/787604615", 2024-02-19T21:10:50.604 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6811/787604615", 2024-02-19T21:10:50.604 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6810/787604615", 2024-02-19T21:10:50.604 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:10:50.604 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:10:50.604 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:10:50.604 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.604 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.604 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.604 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-02-19T21:10:50.604 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f", 2024-02-19T21:10:50.604 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:10:50.605 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:10:50.605 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:10:50.605 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:10:50.605 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:10:50.605 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:10:50.605 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 53, 2024-02-19T21:10:50.605 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 441, 2024-02-19T21:10:50.605 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:10:50.605 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:10:50.605 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:10:50.605 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.605 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.605 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.605 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6812", 2024-02-19T21:10:50.606 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:10:50.606 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.606 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.606 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.606 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:10:50.606 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.606 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.606 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.606 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6813", 2024-02-19T21:10:50.606 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:10:50.606 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.606 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.606 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.607 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:10:50.607 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.607 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.607 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6815", 2024-02-19T21:10:50.607 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:10:50.607 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.607 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.607 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.607 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:10:50.607 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:10:50.607 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.607 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:10:50.607 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6814", 2024-02-19T21:10:50.608 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:10:50.608 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.608 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.608 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.608 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6812/157927432", 2024-02-19T21:10:50.608 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6813/157927432", 2024-02-19T21:10:50.608 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6815/157927432", 2024-02-19T21:10:50.608 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6814/157927432", 2024-02-19T21:10:50.608 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:10:50.608 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:10:50.608 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:10:50.608 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.609 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.609 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:10:50.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_xinfo": [ 2024-02-19T21:10:50.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-02-19T21:10:50.609 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:10:50.609 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:10:50.609 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:10:50.609 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:10:50.609 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:10:50.609 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:10:48.847238+0000", 2024-02-19T21:10:50.609 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:10:50.609 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.609 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.609 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-02-19T21:10:50.610 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:10:50.610 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:10:50.610 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:10:50.610 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:10:50.610 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:10:50.610 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:09:35.283896+0000", 2024-02-19T21:10:50.610 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:10:50.610 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.610 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.610 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-02-19T21:10:50.610 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:10:50.610 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:10:50.610 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:10:50.610 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:10:50.611 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:10:50.611 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:09:59.922975+0000", 2024-02-19T21:10:50.611 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:10:50.611 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.611 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.611 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-02-19T21:10:50.611 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:10:50.611 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:10:50.611 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:10:50.611 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:10:50.611 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:10:50.611 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:10:35.832944+0000", 2024-02-19T21:10:50.612 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:10:50.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.612 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-02-19T21:10:50.612 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:10:50.612 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:10:50.612 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:10:50.612 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:10:50.612 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:10:50.612 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:10:44.663496+0000", 2024-02-19T21:10:50.612 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:10:50.612 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.612 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.612 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-02-19T21:10:50.613 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:10:50.613 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:10:50.613 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:10:50.613 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:10:50.613 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:10:50.613 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:09:37.894663+0000", 2024-02-19T21:10:50.613 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:10:50.613 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.613 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.613 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-02-19T21:10:50.613 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:10:50.613 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:10:50.613 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:10:50.613 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:10:50.614 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:10:50.614 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:09:30.003714+0000", 2024-02-19T21:10:50.614 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:10:50.614 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.614 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.614 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-02-19T21:10:50.614 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:10:50.614 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:10:50.614 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:10:50.614 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:10:50.614 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:10:50.614 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:09:54.085752+0000", 2024-02-19T21:10:50.614 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:10:50.614 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.614 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:10:50.615 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap": [], 2024-02-19T21:10:50.615 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_items": [ 2024-02-19T21:10:50.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.615 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "2.e", 2024-02-19T21:10:50.615 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:10:50.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.615 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 7, 2024-02-19T21:10:50.615 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 1 2024-02-19T21:10:50.615 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.615 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.615 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.615 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.7", 2024-02-19T21:10:50.615 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:10:50.615 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.616 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:10:50.616 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-02-19T21:10:50.616 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.616 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.616 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.d", 2024-02-19T21:10:50.616 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:10:50.616 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.616 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:10:50.616 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 5 2024-02-19T21:10:50.616 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.616 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.616 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.617 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.f", 2024-02-19T21:10:50.617 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:10:50.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.617 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:10:50.617 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-02-19T21:10:50.617 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.617 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.617 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.617 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.10", 2024-02-19T21:10:50.617 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:10:50.617 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.617 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 3, 2024-02-19T21:10:50.618 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 5 2024-02-19T21:10:50.618 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.618 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.618 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.17", 2024-02-19T21:10:50.618 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:10:50.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.618 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 3, 2024-02-19T21:10:50.618 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-02-19T21:10:50.618 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.618 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.618 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.618 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.618 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "5.15", 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 0, 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 3 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "16.1", 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 3 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.619 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.620 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:10:50.620 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_temp": [], 2024-02-19T21:10:50.620 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_temp": [], 2024-02-19T21:10:50.620 INFO:teuthology.orchestra.run.smithi163.stdout: "blocklist": { 2024-02-19T21:10:50.620 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1427774891": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:10:50.620 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3282102947": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:10:50.620 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2735092549": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:10:50.620 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1235588846": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:10:50.620 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2731225015": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:10:50.620 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/506491651": "2024-02-20T20:56:02.185409+0000", 2024-02-19T21:10:50.620 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/168117826": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:10:50.620 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1458009465": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:10:50.620 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/541631472": "2024-02-20T20:56:02.185409+0000", 2024-02-19T21:10:50.620 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2248999541": "2024-02-20T20:56:02.185409+0000" 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: "range_blocklist": {}, 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profiles": { 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: "default": { 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: "ruleset-failure-domain": "osd", 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van" 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps_queue": [ 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 11, 2024-02-19T21:10:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:10:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:10:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:10:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 9, 2024-02-19T21:10:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:10:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.623 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 25, 2024-02-19T21:10:50.623 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:10:50.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.623 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:10:50.623 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.623 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:10:50.623 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.623 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.623 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.623 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:10:50.623 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.623 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.623 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.624 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 57, 2024-02-19T21:10:50.624 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:10:50.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.624 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:10:50.624 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:10:50.624 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.624 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.624 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:10:50.625 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:10:50.625 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.625 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.625 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.625 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 76, 2024-02-19T21:10:50.625 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:10:50.625 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.625 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:10:50.626 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.626 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:10:50.626 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.626 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:10:50.626 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.626 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.626 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.626 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.626 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.626 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 85, 2024-02-19T21:10:50.627 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:10:50.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.627 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:10:50.627 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.627 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.627 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.627 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.627 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 92, 2024-02-19T21:10:50.627 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:10:50.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.627 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:10:50.627 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:10:50.627 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.627 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.628 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:10:50.628 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:10:50.628 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.628 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.628 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.628 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 94, 2024-02-19T21:10:50.628 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:10:50.628 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.628 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 4, 2024-02-19T21:10:50.628 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 4 2024-02-19T21:10:50.628 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.628 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.629 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 107, 2024-02-19T21:10:50.629 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:10:50.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.629 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:10:50.629 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.629 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.629 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.629 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.629 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.630 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 112, 2024-02-19T21:10:50.630 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:10:50.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.630 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 1, 2024-02-19T21:10:50.630 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.630 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:10:50.630 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.630 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.630 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:10:50.630 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.630 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.631 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:10:50.631 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.631 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.631 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.631 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 126, 2024-02-19T21:10:50.631 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:10:50.631 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.631 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:10:50.631 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:10:50.631 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.632 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:10:50.632 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:10:50.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.632 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 10, 2024-02-19T21:10:50.632 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:10:50.632 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.632 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.632 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.632 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.632 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 132, 2024-02-19T21:10:50.632 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:10:50.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.633 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 12, 2024-02-19T21:10:50.633 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.633 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.633 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.633 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.633 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 154, 2024-02-19T21:10:50.633 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:10:50.633 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.633 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 14, 2024-02-19T21:10:50.633 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.633 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.633 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.634 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 164, 2024-02-19T21:10:50.634 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:10:50.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.634 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 17, 2024-02-19T21:10:50.634 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.634 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.634 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.634 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:10:50.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.634 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 178, 2024-02-19T21:10:50.634 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:10:50.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:10:50.635 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 20, 2024-02-19T21:10:50.635 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:10:50.635 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.635 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:10:50.635 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.635 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:10:50.635 INFO:teuthology.orchestra.run.smithi163.stdout: "new_removed_snaps": [], 2024-02-19T21:10:50.635 INFO:teuthology.orchestra.run.smithi163.stdout: "new_purged_snaps": [], 2024-02-19T21:10:50.635 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_node_flags": {}, 2024-02-19T21:10:50.635 INFO:teuthology.orchestra.run.smithi163.stdout: "device_class_flags": {}, 2024-02-19T21:10:50.635 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode": { 2024-02-19T21:10:50.635 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_enabled": false, 2024-02-19T21:10:50.635 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_bucket_count": 0, 2024-02-19T21:10:50.636 INFO:teuthology.orchestra.run.smithi163.stdout: "degraded_stretch_mode": 0, 2024-02-19T21:10:50.636 INFO:teuthology.orchestra.run.smithi163.stdout: "recovering_stretch_mode": 0, 2024-02-19T21:10:50.636 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_bucket": 0 2024-02-19T21:10:50.636 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:10:50.636 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:10:50.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.531+0000 7f82f3b5a700 10 librados: watch_flush enter 2024-02-19T21:10:50.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.531+0000 7f82f3b5a700 10 librados: watch_flush exit 2024-02-19T21:10:50.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.531+0000 7f82f3b5a700 20 client.23178.objecter shutdown clearing up homeless session... 2024-02-19T21:10:50.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.531+0000 7f82f3b5a700 10 client.23178.objecter successfully canceled tick 2024-02-19T21:10:50.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.531+0000 7f82f3b5a700 1 -- 172.21.15.163:0/3123303898 >> v1:172.21.15.163:6800/1552514681 conn(0x7f82d8063e50 legacy=0x7f82d8066310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:50.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.531+0000 7f82f3b5a700 1 -- 172.21.15.163:0/3123303898 >> v1:172.21.15.163:6789/0 conn(0x7f82ec0fc8a0 legacy=0x7f82ec0f9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:50.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.531+0000 7f82f3b5a700 1 -- 172.21.15.163:0/3123303898 shutdown_connections 2024-02-19T21:10:50.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.531+0000 7f82f3b5a700 1 -- 172.21.15.163:0/3123303898 >> 172.21.15.163:0/3123303898 conn(0x7f82ec06fdc0 msgr2=0x7f82ec0f6780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:50.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.532+0000 7f82f3b5a700 1 -- 172.21.15.163:0/3123303898 shutdown_connections 2024-02-19T21:10:50.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.532+0000 7f82f3b5a700 1 -- 172.21.15.163:0/3123303898 wait complete. 2024-02-19T21:10:50.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:50.532+0000 7f82f3b5a700 1 librados: shutdown 2024-02-19T21:10:50.647 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-19T21:10:50.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:10:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:10:50.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:48.846964+0000 osd.0 (osd.0) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:10:50.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:48.847221+0000 osd.0 (osd.0) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:10:50.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:49.222958+0000 mgr.y (mgr.14550) 400 : cluster [DBG] pgmap v647: 228 pgs: 32 unknown, 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering, 192 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 19/1059 objects misplaced (1.794%) 2024-02-19T21:10:50.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:49.388767+0000 mon.b (mon.1) 602 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:50.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:49.389163+0000 mon.a (mon.0) 2594 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:50.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:49.389758+0000 mon.b (mon.1) 603 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-67"}]: dispatch 2024-02-19T21:10:50.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:49.390072+0000 mon.a (mon.0) 2595 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-67"}]: dispatch 2024-02-19T21:10:50.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:50.279505+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e443: 8 total, 8 up, 8 in 2024-02-19T21:10:50.861 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:50.874 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:50.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:50.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:50.879 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:48.846964+0000 osd.0 (osd.0) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:10:50.879 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:48.847221+0000 osd.0 (osd.0) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:10:50.879 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:49.222958+0000 mgr.y (mgr.14550) 400 : cluster [DBG] pgmap v647: 228 pgs: 32 unknown, 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering, 192 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 19/1059 objects misplaced (1.794%) 2024-02-19T21:10:50.879 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:49.388767+0000 mon.b (mon.1) 602 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:50.879 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:49.389163+0000 mon.a (mon.0) 2594 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:50.879 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:49.389758+0000 mon.b (mon.1) 603 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-67"}]: dispatch 2024-02-19T21:10:50.879 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:49.390072+0000 mon.a (mon.0) 2595 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-67"}]: dispatch 2024-02-19T21:10:50.879 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:50.279505+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e443: 8 total, 8 up, 8 in 2024-02-19T21:10:50.879 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:48.846964+0000 osd.0 (osd.0) 125 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:10:50.880 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:48.847221+0000 osd.0 (osd.0) 126 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:10:50.880 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:49.222958+0000 mgr.y (mgr.14550) 400 : cluster [DBG] pgmap v647: 228 pgs: 32 unknown, 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering, 192 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 19/1059 objects misplaced (1.794%) 2024-02-19T21:10:50.880 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:49.388767+0000 mon.b (mon.1) 602 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:50.880 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:49.389163+0000 mon.a (mon.0) 2594 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:10:50.880 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:49.389758+0000 mon.b (mon.1) 603 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-67"}]: dispatch 2024-02-19T21:10:50.880 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:49.390072+0000 mon.a (mon.0) 2595 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-67"}]: dispatch 2024-02-19T21:10:50.880 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:50.279505+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e443: 8 total, 8 up, 8 in 2024-02-19T21:10:50.980 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:51.008 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:51.444 INFO:tasks.thrashosds.thrasher:j is {'epoch': 443, 'fsid': '2591809a-cf69-11ee-95bb-87774f69a715', 'created': '2024-02-19T20:55:37.794885+0000', 'modified': '2024-02-19T21:10:50.273875+0000', 'last_up_change': '2024-02-19T21:01:09.214134+0000', 'last_in_change': '2024-02-19T21:10:40.268547+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 103, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 215, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-19T20:59:09.425169+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-19T21:01:14.658945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '2.20', 'ready_epoch': 309, 'last_epoch_started': 308, 'last_epoch_clean': 308, 'source_version': "58'1", 'target_version': "58'1"}, 'last_change': '416', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '309', 'last_force_op_resend_preluminous': '307', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-19T21:01:14.958800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-19T21:01:16.543816+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-19T21:01:18.521482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-19T21:01:20.586564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 16, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-19T21:04:35.333192+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 61, 'pool_name': 'test-rados-api-smithi163-83928-6', 'create_time': '2024-02-19T21:04:52.991355+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '439', 'last_force_op_resend': '386', 'last_force_op_resend_prenautilus': '386', 'last_force_op_resend_preluminous': '386', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 31, 'snap_epoch': 439, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '4a14b406-7c21-40f3-8b12-8cf9df811088', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 218, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6801', 'nonce': 1559418115}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6802', 'nonce': 1559418115}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6804', 'nonce': 1559418115}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6803', 'nonce': 1559418115}]}, 'public_addr': '172.21.15.163:6801/1559418115', 'cluster_addr': '172.21.15.163:6802/1559418115', 'heartbeat_back_addr': '172.21.15.163:6804/1559418115', 'heartbeat_front_addr': '172.21.15.163:6803/1559418115', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e74a65fa-cb9c-42c3-9293-602cf7e9dc67', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 441, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6805', 'nonce': 3125047432}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6806', 'nonce': 3125047432}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6808', 'nonce': 3125047432}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6807', 'nonce': 3125047432}]}, 'public_addr': '172.21.15.163:6805/3125047432', 'cluster_addr': '172.21.15.163:6806/3125047432', 'heartbeat_back_addr': '172.21.15.163:6808/3125047432', 'heartbeat_front_addr': '172.21.15.163:6807/3125047432', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6980a625-cdc9-487e-8f8c-7997ba716a97', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 441, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6809', 'nonce': 1563158487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6810', 'nonce': 1563158487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6812', 'nonce': 1563158487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6811', 'nonce': 1563158487}]}, 'public_addr': '172.21.15.163:6809/1563158487', 'cluster_addr': '172.21.15.163:6810/1563158487', 'heartbeat_back_addr': '172.21.15.163:6812/1563158487', 'heartbeat_front_addr': '172.21.15.163:6811/1563158487', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '359e2ee0-9d1d-4074-8119-2d69bc2ca3a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 441, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6813', 'nonce': 2112279769}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6814', 'nonce': 2112279769}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6816', 'nonce': 2112279769}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6815', 'nonce': 2112279769}]}, 'public_addr': '172.21.15.163:6813/2112279769', 'cluster_addr': '172.21.15.163:6814/2112279769', 'heartbeat_back_addr': '172.21.15.163:6816/2112279769', 'heartbeat_front_addr': '172.21.15.163:6815/2112279769', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2bf5ef05-292b-4693-85a9-97991f2091e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 441, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6800', 'nonce': 3129716965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6801', 'nonce': 3129716965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6803', 'nonce': 3129716965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6802', 'nonce': 3129716965}]}, 'public_addr': '172.21.15.193:6800/3129716965', 'cluster_addr': '172.21.15.193:6801/3129716965', 'heartbeat_back_addr': '172.21.15.193:6803/3129716965', 'heartbeat_front_addr': '172.21.15.193:6802/3129716965', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '59065f9b-9240-414b-9c91-e91cda4aed6f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 442, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6804', 'nonce': 805161991}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6805', 'nonce': 805161991}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6807', 'nonce': 805161991}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6806', 'nonce': 805161991}]}, 'public_addr': '172.21.15.193:6804/805161991', 'cluster_addr': '172.21.15.193:6805/805161991', 'heartbeat_back_addr': '172.21.15.193:6807/805161991', 'heartbeat_front_addr': '172.21.15.193:6806/805161991', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '302c95df-3842-4818-8afd-d79d8184684f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.750579833984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 441, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6808', 'nonce': 787604615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6809', 'nonce': 787604615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6811', 'nonce': 787604615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6810', 'nonce': 787604615}]}, 'public_addr': '172.21.15.193:6808/787604615', 'cluster_addr': '172.21.15.193:6809/787604615', 'heartbeat_back_addr': '172.21.15.193:6811/787604615', 'heartbeat_front_addr': '172.21.15.193:6810/787604615', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '9ebeff27-c2f8-4a66-923f-a0d7573ddd3f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 441, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6812', 'nonce': 157927432}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6813', 'nonce': 157927432}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6815', 'nonce': 157927432}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6814', 'nonce': 157927432}]}, 'public_addr': '172.21.15.193:6812/157927432', 'cluster_addr': '172.21.15.193:6813/157927432', 'heartbeat_back_addr': '172.21.15.193:6815/157927432', 'heartbeat_front_addr': '172.21.15.193:6814/157927432', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:10:48.847238+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:09:35.283896+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:09:59.922975+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:10:35.832944+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:10:44.663496+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:09:37.894663+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:09:30.003714+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:09:54.085752+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.e', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.10', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.17', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '5.15', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '16.1', 'mappings': [{'from': 1, 'to': 3}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.163:0/1427774891': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/3282102947': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:6800/2735092549': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/1235588846': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:6800/2731225015': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:0/506491651': '2024-02-20T20:56:02.185409+0000', '172.21.15.163:0/168117826': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/1458009465': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:0/541631472': '2024-02-20T20:56:02.185409+0000', '172.21.15.163:6800/2248999541': '2024-02-20T20:56:02.185409+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 11, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 25, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 57, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 76, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 85, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 92, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 94, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 126, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 132, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 178, 'snaps': [{'begin': 20, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-19T21:10:51.445 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 16.1 2024-02-19T21:10:51.445 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd rm-pg-upmap-items 16.1 2024-02-19T21:10:51.456 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:51.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:51.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:51.479 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:51.560 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:51.587 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:51.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:49.464438+0000 osd.4 (osd.4) 149 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:10:51.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:49.466221+0000 osd.4 (osd.4) 150 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:10:51.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:50.049746+0000 osd.5 (osd.5) 176 : cluster [DBG] 4.15 scrub starts 2024-02-19T21:10:51.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:50.053223+0000 osd.5 (osd.5) 177 : cluster [DBG] 4.15 scrub ok 2024-02-19T21:10:51.729 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:50.423099+0000 osd.4 (osd.4) 151 : cluster [DBG] 16.19 scrub starts 2024-02-19T21:10:51.729 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:50.424375+0000 osd.4 (osd.4) 152 : cluster [DBG] 16.19 scrub ok 2024-02-19T21:10:51.729 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:50.428510+0000 mgr.y (mgr.14550) 401 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:51.729 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:50.528968+0000 mon.a (mon.0) 2597 : audit [DBG] from='client.? v1:172.21.15.163:0/3123303898' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:10:51.729 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:50.768485+0000 osd.3 (osd.3) 189 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:10:51.729 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:50.769612+0000 osd.3 (osd.3) 190 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:10:51.729 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:51.280631+0000 mon.a (mon.0) 2598 : cluster [DBG] osdmap e444: 8 total, 8 up, 8 in 2024-02-19T21:10:51.729 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:51.287651+0000 mon.b (mon.1) 604 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:51.729 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:51.289575+0000 mon.a (mon.0) 2599 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:49.464438+0000 osd.4 (osd.4) 149 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:10:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:49.466221+0000 osd.4 (osd.4) 150 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:10:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:50.049746+0000 osd.5 (osd.5) 176 : cluster [DBG] 4.15 scrub starts 2024-02-19T21:10:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:50.053223+0000 osd.5 (osd.5) 177 : cluster [DBG] 4.15 scrub ok 2024-02-19T21:10:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:50.423099+0000 osd.4 (osd.4) 151 : cluster [DBG] 16.19 scrub starts 2024-02-19T21:10:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:50.424375+0000 osd.4 (osd.4) 152 : cluster [DBG] 16.19 scrub ok 2024-02-19T21:10:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:50.428510+0000 mgr.y (mgr.14550) 401 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:50.528968+0000 mon.a (mon.0) 2597 : audit [DBG] from='client.? v1:172.21.15.163:0/3123303898' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:10:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:50.768485+0000 osd.3 (osd.3) 189 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:10:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:50.769612+0000 osd.3 (osd.3) 190 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:10:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:51.280631+0000 mon.a (mon.0) 2598 : cluster [DBG] osdmap e444: 8 total, 8 up, 8 in 2024-02-19T21:10:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:51.287651+0000 mon.b (mon.1) 604 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:51.730 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:51.289575+0000 mon.a (mon.0) 2599 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:51.736 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:51.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:51.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:49.464438+0000 osd.4 (osd.4) 149 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:10:51.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:49.466221+0000 osd.4 (osd.4) 150 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:10:51.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:50.049746+0000 osd.5 (osd.5) 176 : cluster [DBG] 4.15 scrub starts 2024-02-19T21:10:51.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:50.053223+0000 osd.5 (osd.5) 177 : cluster [DBG] 4.15 scrub ok 2024-02-19T21:10:51.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:50.423099+0000 osd.4 (osd.4) 151 : cluster [DBG] 16.19 scrub starts 2024-02-19T21:10:51.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:50.424375+0000 osd.4 (osd.4) 152 : cluster [DBG] 16.19 scrub ok 2024-02-19T21:10:51.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:50.428510+0000 mgr.y (mgr.14550) 401 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:10:51.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:50.528968+0000 mon.a (mon.0) 2597 : audit [DBG] from='client.? v1:172.21.15.163:0/3123303898' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:10:51.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:50.768485+0000 osd.3 (osd.3) 189 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:10:51.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:50.769612+0000 osd.3 (osd.3) 190 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:10:51.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:51.280631+0000 mon.a (mon.0) 2598 : cluster [DBG] osdmap e444: 8 total, 8 up, 8 in 2024-02-19T21:10:51.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:51.287651+0000 mon.b (mon.1) 604 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:51.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:51.289575+0000 mon.a (mon.0) 2599 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:10:51.841 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:51.868 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:52.066 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:52.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:52.170 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:52.203 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:52.223 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:52.227 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:52.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:52.329 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:52.356 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:52.376 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:52.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:52.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:52.479 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:52.510 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:52.684 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:52.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:51.078883+0000 osd.5 (osd.5) 178 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:10:52.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:51.081984+0000 osd.5 (osd.5) 179 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:10:52.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:51.223402+0000 mgr.y (mgr.14550) 402 : cluster [DBG] pgmap v650: 196 pgs: 1 peering, 195 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 46 B/s, 11 objects/s recovering 2024-02-19T21:10:52.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:51.492253+0000 osd.6 (osd.6) 169 : cluster [DBG] 16.6 scrub starts 2024-02-19T21:10:52.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:51.493201+0000 osd.6 (osd.6) 170 : cluster [DBG] 16.6 scrub ok 2024-02-19T21:10:52.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:51.848164+0000 osd.7 (osd.7) 193 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:10:52.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:51.851114+0000 osd.7 (osd.7) 194 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:10:52.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:52.287038+0000 mon.a (mon.0) 2600 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:10:52.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:52.287117+0000 mon.a (mon.0) 2601 : cluster [DBG] osdmap e445: 8 total, 8 up, 8 in 2024-02-19T21:10:52.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:52.298236+0000 mon.b (mon.1) 605 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:10:52.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:52.300739+0000 mon.b (mon.1) 606 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:10:52.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:52.301029+0000 mon.a (mon.0) 2602 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:10:52.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:51.078883+0000 osd.5 (osd.5) 178 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:10:52.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:51.081984+0000 osd.5 (osd.5) 179 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:10:52.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:51.223402+0000 mgr.y (mgr.14550) 402 : cluster [DBG] pgmap v650: 196 pgs: 1 peering, 195 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 46 B/s, 11 objects/s recovering 2024-02-19T21:10:52.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:51.492253+0000 osd.6 (osd.6) 169 : cluster [DBG] 16.6 scrub starts 2024-02-19T21:10:52.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:51.493201+0000 osd.6 (osd.6) 170 : cluster [DBG] 16.6 scrub ok 2024-02-19T21:10:52.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:51.848164+0000 osd.7 (osd.7) 193 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:10:52.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:51.851114+0000 osd.7 (osd.7) 194 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:10:52.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:52.287038+0000 mon.a (mon.0) 2600 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:10:52.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:52.287117+0000 mon.a (mon.0) 2601 : cluster [DBG] osdmap e445: 8 total, 8 up, 8 in 2024-02-19T21:10:52.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:52.298236+0000 mon.b (mon.1) 605 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:10:52.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:52.300739+0000 mon.b (mon.1) 606 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:10:52.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:52.301029+0000 mon.a (mon.0) 2602 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:10:52.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:51.078883+0000 osd.5 (osd.5) 178 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:10:52.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:51.081984+0000 osd.5 (osd.5) 179 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:10:52.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:51.223402+0000 mgr.y (mgr.14550) 402 : cluster [DBG] pgmap v650: 196 pgs: 1 peering, 195 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 46 B/s, 11 objects/s recovering 2024-02-19T21:10:52.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:51.492253+0000 osd.6 (osd.6) 169 : cluster [DBG] 16.6 scrub starts 2024-02-19T21:10:52.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:51.493201+0000 osd.6 (osd.6) 170 : cluster [DBG] 16.6 scrub ok 2024-02-19T21:10:52.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:51.848164+0000 osd.7 (osd.7) 193 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:10:52.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:51.851114+0000 osd.7 (osd.7) 194 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:10:52.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:52.287038+0000 mon.a (mon.0) 2600 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:10:52.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:52.287117+0000 mon.a (mon.0) 2601 : cluster [DBG] osdmap e445: 8 total, 8 up, 8 in 2024-02-19T21:10:52.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:52.298236+0000 mon.b (mon.1) 605 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:10:52.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:52.300739+0000 mon.b (mon.1) 606 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:10:52.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:52.301029+0000 mon.a (mon.0) 2602 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:10:53.746 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:10:53] "GET /metrics HTTP/1.1" 200 189714 "" "Prometheus/2.33.4" 2024-02-19T21:10:53.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:51.690071+0000 osd.1 (osd.1) 203 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:10:53.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:51.691280+0000 osd.1 (osd.1) 204 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:10:53.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:52.086791+0000 osd.5 (osd.5) 180 : cluster [DBG] 2.2 scrub starts 2024-02-19T21:10:53.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:52.089711+0000 osd.5 (osd.5) 181 : cluster [DBG] 2.2 scrub ok 2024-02-19T21:10:53.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:52.464582+0000 osd.6 (osd.6) 171 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:10:53.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:52.467472+0000 osd.6 (osd.6) 172 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:10:53.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:53.283441+0000 mon.a (mon.0) 2603 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:10:53.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:53.283510+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e446: 8 total, 8 up, 8 in 2024-02-19T21:10:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:51.690071+0000 osd.1 (osd.1) 203 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:10:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:51.691280+0000 osd.1 (osd.1) 204 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:10:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:52.086791+0000 osd.5 (osd.5) 180 : cluster [DBG] 2.2 scrub starts 2024-02-19T21:10:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:52.089711+0000 osd.5 (osd.5) 181 : cluster [DBG] 2.2 scrub ok 2024-02-19T21:10:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:52.464582+0000 osd.6 (osd.6) 171 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:10:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:52.467472+0000 osd.6 (osd.6) 172 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:10:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:53.283441+0000 mon.a (mon.0) 2603 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:10:53.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:53.283510+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e446: 8 total, 8 up, 8 in 2024-02-19T21:10:53.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:51.690071+0000 osd.1 (osd.1) 203 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:10:53.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:51.691280+0000 osd.1 (osd.1) 204 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:10:53.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:52.086791+0000 osd.5 (osd.5) 180 : cluster [DBG] 2.2 scrub starts 2024-02-19T21:10:53.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:52.089711+0000 osd.5 (osd.5) 181 : cluster [DBG] 2.2 scrub ok 2024-02-19T21:10:53.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:52.464582+0000 osd.6 (osd.6) 171 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:10:53.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:52.467472+0000 osd.6 (osd.6) 172 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:10:53.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:53.283441+0000 mon.a (mon.0) 2603 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:10:53.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:53.283510+0000 mon.a (mon.0) 2604 : cluster [DBG] osdmap e446: 8 total, 8 up, 8 in 2024-02-19T21:10:54.661 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:54.665 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:54.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:54.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:52.735642+0000 osd.1 (osd.1) 205 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:10:54.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:52.738596+0000 osd.1 (osd.1) 206 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:10:54.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:53.099912+0000 osd.2 (osd.2) 170 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:10:54.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:53.101416+0000 osd.2 (osd.2) 171 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:10:54.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:53.223926+0000 mgr.y (mgr.14550) 403 : cluster [DBG] pgmap v653: 228 pgs: 32 unknown, 1 peering, 195 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 46 B/s, 11 objects/s recovering 2024-02-19T21:10:54.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:53.702894+0000 osd.3 (osd.3) 191 : cluster [DBG] 5.17 scrub starts 2024-02-19T21:10:54.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:53.703722+0000 osd.3 (osd.3) 192 : cluster [DBG] 5.17 scrub ok 2024-02-19T21:10:54.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:54.285578+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e447: 8 total, 8 up, 8 in 2024-02-19T21:10:54.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:52.735642+0000 osd.1 (osd.1) 205 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:10:54.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:52.738596+0000 osd.1 (osd.1) 206 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:10:54.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:53.099912+0000 osd.2 (osd.2) 170 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:10:54.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:53.101416+0000 osd.2 (osd.2) 171 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:10:54.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:53.223926+0000 mgr.y (mgr.14550) 403 : cluster [DBG] pgmap v653: 228 pgs: 32 unknown, 1 peering, 195 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 46 B/s, 11 objects/s recovering 2024-02-19T21:10:54.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:53.702894+0000 osd.3 (osd.3) 191 : cluster [DBG] 5.17 scrub starts 2024-02-19T21:10:54.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:53.703722+0000 osd.3 (osd.3) 192 : cluster [DBG] 5.17 scrub ok 2024-02-19T21:10:54.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:54.285578+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e447: 8 total, 8 up, 8 in 2024-02-19T21:10:54.766 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:54.790 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:10:54.791 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:10:54.795 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:54.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:52.735642+0000 osd.1 (osd.1) 205 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:10:54.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:52.738596+0000 osd.1 (osd.1) 206 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:10:54.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:53.099912+0000 osd.2 (osd.2) 170 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:10:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:53.101416+0000 osd.2 (osd.2) 171 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:10:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:53.223926+0000 mgr.y (mgr.14550) 403 : cluster [DBG] pgmap v653: 228 pgs: 32 unknown, 1 peering, 195 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 46 B/s, 11 objects/s recovering 2024-02-19T21:10:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:53.702894+0000 osd.3 (osd.3) 191 : cluster [DBG] 5.17 scrub starts 2024-02-19T21:10:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:53.703722+0000 osd.3 (osd.3) 192 : cluster [DBG] 5.17 scrub ok 2024-02-19T21:10:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:54.285578+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e447: 8 total, 8 up, 8 in 2024-02-19T21:10:55.046 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:55.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:55.198 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:55.230 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:55.252 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:55.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:55.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:55.356 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:55.385 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:55.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.765+0000 7f3766702700 1 -- 172.21.15.163:0/2055325815 >> v1:172.21.15.163:6789/0 conn(0x7f37601031b0 legacy=0x7f3760105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:55.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.765+0000 7f3766702700 1 -- 172.21.15.163:0/2055325815 shutdown_connections 2024-02-19T21:10:55.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.765+0000 7f3766702700 1 -- 172.21.15.163:0/2055325815 >> 172.21.15.163:0/2055325815 conn(0x7f37600f2470 msgr2=0x7f37600f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:55.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.765+0000 7f3766702700 1 -- 172.21.15.163:0/2055325815 shutdown_connections 2024-02-19T21:10:55.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.765+0000 7f3766702700 1 -- 172.21.15.163:0/2055325815 wait complete. 2024-02-19T21:10:55.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f3766702700 1 librados: starting msgr at 2024-02-19T21:10:55.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f3766702700 1 librados: starting objecter 2024-02-19T21:10:55.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f3766702700 1 Processor -- start 2024-02-19T21:10:55.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f3766702700 1 -- start start 2024-02-19T21:10:55.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f3766702700 1 librados: setting wanted keys 2024-02-19T21:10:55.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f3766702700 1 librados: calling monclient init 2024-02-19T21:10:55.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f3766702700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f37601172e0 con 0x7f37600ff670 2024-02-19T21:10:55.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f3766702700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f37601184c0 con 0x7f37600fc870 2024-02-19T21:10:55.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f3766702700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f37601196c0 con 0x7f37601031b0 2024-02-19T21:10:55.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f375f7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f37600ff670 0x7f37600612c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:40968/0 (socket says 172.21.15.163:40968) 2024-02-19T21:10:55.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f375f7fe700 1 -- 172.21.15.163:0/745575642 learned_addr learned my addr 172.21.15.163:0/745575642 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:55.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f375d7fa700 10 client.?.objecter ms_handle_connect 0x7f37601031b0 2024-02-19T21:10:55.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f375d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:55.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f375d7fa700 10 client.?.objecter ms_handle_connect 0x7f37600ff670 2024-02-19T21:10:55.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f375d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:55.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.766+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914366418 0 0) 0x7f37601196c0 con 0x7f37601031b0 2024-02-19T21:10:55.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3748003030 con 0x7f37601031b0 2024-02-19T21:10:55.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 10 client.?.objecter ms_handle_connect 0x7f37600fc870 2024-02-19T21:10:55.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:55.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501952525 0 0) 0x7f37601172e0 con 0x7f37600ff670 2024-02-19T21:10:55.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37601196c0 con 0x7f37600ff670 2024-02-19T21:10:55.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2141246966 0 0) 0x7f37601184c0 con 0x7f37600fc870 2024-02-19T21:10:55.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37601172e0 con 0x7f37600fc870 2024-02-19T21:10:55.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4183537558 0 0) 0x7f3748003030 con 0x7f37601031b0 2024-02-19T21:10:55.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37601184c0 con 0x7f37601031b0 2024-02-19T21:10:55.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 804157817 0 0) 0x7f37601196c0 con 0x7f37600ff670 2024-02-19T21:10:55.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3748003030 con 0x7f37600ff670 2024-02-19T21:10:55.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f37580023d0 con 0x7f37601031b0 2024-02-19T21:10:55.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f37540019f0 con 0x7f37600ff670 2024-02-19T21:10:55.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4164593285 0 0) 0x7f37601172e0 con 0x7f37600fc870 2024-02-19T21:10:55.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37601196c0 con 0x7f37600fc870 2024-02-19T21:10:55.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f37500018c0 con 0x7f37600fc870 2024-02-19T21:10:55.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4266465120 0 0) 0x7f37601184c0 con 0x7f37601031b0 2024-02-19T21:10:55.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 >> v1:172.21.15.193:6789/0 conn(0x7f37600fc870 legacy=0x7f3760060b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:55.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 >> v1:172.21.15.163:6789/0 conn(0x7f37600ff670 legacy=0x7f37600612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:55.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376011a8c0 con 0x7f37601031b0 2024-02-19T21:10:55.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f3766702700 1 -- 172.21.15.163:0/745575642 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37601174d0 con 0x7f37601031b0 2024-02-19T21:10:55.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f3766702700 10 client.49904.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:55.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f3766702700 1 -- 172.21.15.163:0/745575642 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37601179c0 con 0x7f37601031b0 2024-02-19T21:10:55.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f3766702700 1 librados: init done 2024-02-19T21:10:55.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3758003330 con 0x7f37601031b0 2024-02-19T21:10:55.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.767+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3758003a10 con 0x7f37601031b0 2024-02-19T21:10:55.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.768+0000 7f3766702700 1 -- 172.21.15.163:0/745575642 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f374c004f20 con 0x7f37601031b0 2024-02-19T21:10:55.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.768+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f3758016920 con 0x7f37601031b0 2024-02-19T21:10:55.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.769+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(447..447 src has 43..447) v4 ==== 8092+0+0 (unknown 792516295 0 0) 0x7f3758076c20 con 0x7f37601031b0 2024-02-19T21:10:55.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.769+0000 7f375d7fa700 10 client.49904.objecter ms_dispatch 0x7f3760001550 osd_map(447..447 src has 43..447) v4 2024-02-19T21:10:55.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.769+0000 7f375d7fa700 3 client.49904.objecter handle_osd_map got epochs [447,447] > 0 2024-02-19T21:10:55.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.769+0000 7f375d7fa700 3 client.49904.objecter handle_osd_map decoding full epoch 447 2024-02-19T21:10:55.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.769+0000 7f375d7fa700 20 client.49904.objecter dump_active .. 0 homeless 2024-02-19T21:10:55.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.769+0000 7f375d7fa700 10 client.49904.objecter ms_handle_connect 0x7f3748063c40 2024-02-19T21:10:55.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.773+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f375804a050 con 0x7f37601031b0 2024-02-19T21:10:55.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:54.955+0000 7f3766702700 1 -- 172.21.15.163:0/745575642 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f374c0051d0 con 0x7f37601031b0 2024-02-19T21:10:55.425 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:55.425 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:55.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:55.526 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:10:55.530 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:55.558 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:55.656 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:10:55.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:53.708673+0000 osd.1 (osd.1) 207 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:10:55.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:53.709452+0000 osd.1 (osd.1) 208 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:10:55.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:54.139841+0000 osd.2 (osd.2) 172 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:10:55.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:54.142892+0000 osd.2 (osd.2) 173 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:10:55.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:54.362659+0000 osd.4 (osd.4) 153 : cluster [DBG] 61.16 scrub starts 2024-02-19T21:10:55.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:54.364816+0000 osd.4 (osd.4) 154 : cluster [DBG] 61.16 scrub ok 2024-02-19T21:10:55.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:54.433993+0000 osd.6 (osd.6) 173 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:10:55.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:54.435834+0000 osd.6 (osd.6) 174 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:10:55.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:54.654267+0000 mon.a (mon.0) 2606 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:10:55.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:54.956523+0000 mon.c (mon.2) 395 : audit [INF] from='client.? v1:172.21.15.163:0/745575642' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:10:55.657 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:54.956845+0000 mon.a (mon.0) 2607 : audit [INF] from='client.49904 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:10:55.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:53.708673+0000 osd.1 (osd.1) 207 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:10:55.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:53.709452+0000 osd.1 (osd.1) 208 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:10:55.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:54.139841+0000 osd.2 (osd.2) 172 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:10:55.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:54.142892+0000 osd.2 (osd.2) 173 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:10:55.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:54.362659+0000 osd.4 (osd.4) 153 : cluster [DBG] 61.16 scrub starts 2024-02-19T21:10:55.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:54.364816+0000 osd.4 (osd.4) 154 : cluster [DBG] 61.16 scrub ok 2024-02-19T21:10:55.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:54.433993+0000 osd.6 (osd.6) 173 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:10:55.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:54.435834+0000 osd.6 (osd.6) 174 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:10:55.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:54.654267+0000 mon.a (mon.0) 2606 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:10:55.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:54.956523+0000 mon.c (mon.2) 395 : audit [INF] from='client.? v1:172.21.15.163:0/745575642' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:10:55.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:54.956845+0000 mon.a (mon.0) 2607 : audit [INF] from='client.49904 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:10:55.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:55.665+0000 7f375d7fa700 1 -- 172.21.15.163:0/745575642 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v448) v1 ==== 101+0+0 (unknown 3951373384 0 0) 0x7f37580484c0 con 0x7f37601031b0 2024-02-19T21:10:55.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:55.668+0000 7f3766702700 10 librados: watch_flush enter 2024-02-19T21:10:55.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:55.668+0000 7f3766702700 10 librados: watch_flush exit 2024-02-19T21:10:55.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:55.668+0000 7f3766702700 20 client.49904.objecter shutdown clearing up homeless session... 2024-02-19T21:10:55.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:55.668+0000 7f3766702700 10 client.49904.objecter successfully canceled tick 2024-02-19T21:10:55.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:55.668+0000 7f3766702700 1 -- 172.21.15.163:0/745575642 >> v1:172.21.15.163:6800/1552514681 conn(0x7f3748063c40 legacy=0x7f3748066100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:55.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:55.668+0000 7f3766702700 1 -- 172.21.15.163:0/745575642 >> v1:172.21.15.163:6790/0 conn(0x7f37601031b0 legacy=0x7f37601159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:55.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:55.668+0000 7f3766702700 1 -- 172.21.15.163:0/745575642 shutdown_connections 2024-02-19T21:10:55.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:55.668+0000 7f3766702700 1 -- 172.21.15.163:0/745575642 >> 172.21.15.163:0/745575642 conn(0x7f37600f2470 msgr2=0x7f3760106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:55.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:55.669+0000 7f3766702700 1 -- 172.21.15.163:0/745575642 shutdown_connections 2024-02-19T21:10:55.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:55.669+0000 7f3766702700 1 -- 172.21.15.163:0/745575642 wait complete. 2024-02-19T21:10:55.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:55.669+0000 7f3766702700 1 librados: shutdown 2024-02-19T21:10:55.668 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:10:55.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:55.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:10:55.779 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:55.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:53.708673+0000 osd.1 (osd.1) 207 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:10:55.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:53.709452+0000 osd.1 (osd.1) 208 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:10:55.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:54.139841+0000 osd.2 (osd.2) 172 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:10:55.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:54.142892+0000 osd.2 (osd.2) 173 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:10:55.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:54.362659+0000 osd.4 (osd.4) 153 : cluster [DBG] 61.16 scrub starts 2024-02-19T21:10:55.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:54.364816+0000 osd.4 (osd.4) 154 : cluster [DBG] 61.16 scrub ok 2024-02-19T21:10:55.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:54.433993+0000 osd.6 (osd.6) 173 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:10:55.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:54.435834+0000 osd.6 (osd.6) 174 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:10:55.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:54.654267+0000 mon.a (mon.0) 2606 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:10:55.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:54.956523+0000 mon.c (mon.2) 395 : audit [INF] from='client.? v1:172.21.15.163:0/745575642' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:10:55.811 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:54.956845+0000 mon.a (mon.0) 2607 : audit [INF] from='client.49904 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:10:55.812 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:55.834 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:55.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:55.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:55.937 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:55.966 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:55.987 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:55.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:55.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:56.090 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:56.119 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:56.171 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:10:56.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:56.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:10:56.275 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:10:56.305 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:56.326 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:10:56.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:56.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:10:56.430 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:56.458 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:56.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:54.669225+0000 osd.1 (osd.1) 209 : cluster [DBG] 61.1 scrub starts 2024-02-19T21:10:56.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:54.672701+0000 osd.1 (osd.1) 210 : cluster [DBG] 61.1 scrub ok 2024-02-19T21:10:56.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:55.062006+0000 osd.6 (osd.6) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:10:56.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:55.062272+0000 osd.6 (osd.6) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:10:56.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:55.224993+0000 mgr.y (mgr.14550) 404 : cluster [DBG] pgmap v656: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-02-19T21:10:56.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:55.652773+0000 mon.a (mon.0) 2608 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:10:56.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:55.666114+0000 mon.a (mon.0) 2609 : audit [INF] from='client.49904 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:10:56.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:55.666178+0000 mon.a (mon.0) 2610 : cluster [DBG] osdmap e448: 8 total, 8 up, 8 in 2024-02-19T21:10:56.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:55.790235+0000 osd.7 (osd.7) 195 : cluster [DBG] 61.1a deep-scrub starts 2024-02-19T21:10:56.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:55.797789+0000 osd.7 (osd.7) 196 : cluster [DBG] 61.1a deep-scrub ok 2024-02-19T21:10:56.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:54.669225+0000 osd.1 (osd.1) 209 : cluster [DBG] 61.1 scrub starts 2024-02-19T21:10:56.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:54.672701+0000 osd.1 (osd.1) 210 : cluster [DBG] 61.1 scrub ok 2024-02-19T21:10:56.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:55.062006+0000 osd.6 (osd.6) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:10:56.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:55.062272+0000 osd.6 (osd.6) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:10:56.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:55.224993+0000 mgr.y (mgr.14550) 404 : cluster [DBG] pgmap v656: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-02-19T21:10:56.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:55.652773+0000 mon.a (mon.0) 2608 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:10:56.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:55.666114+0000 mon.a (mon.0) 2609 : audit [INF] from='client.49904 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:10:56.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:55.666178+0000 mon.a (mon.0) 2610 : cluster [DBG] osdmap e448: 8 total, 8 up, 8 in 2024-02-19T21:10:56.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:55.790235+0000 osd.7 (osd.7) 195 : cluster [DBG] 61.1a deep-scrub starts 2024-02-19T21:10:56.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:55.797789+0000 osd.7 (osd.7) 196 : cluster [DBG] 61.1a deep-scrub ok 2024-02-19T21:10:56.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:54.669225+0000 osd.1 (osd.1) 209 : cluster [DBG] 61.1 scrub starts 2024-02-19T21:10:56.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:54.672701+0000 osd.1 (osd.1) 210 : cluster [DBG] 61.1 scrub ok 2024-02-19T21:10:56.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:55.062006+0000 osd.6 (osd.6) 175 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:10:56.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:55.062272+0000 osd.6 (osd.6) 176 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:10:56.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:55.224993+0000 mgr.y (mgr.14550) 404 : cluster [DBG] pgmap v656: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-02-19T21:10:56.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:55.652773+0000 mon.a (mon.0) 2608 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:10:56.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:55.666114+0000 mon.a (mon.0) 2609 : audit [INF] from='client.49904 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:10:56.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:55.666178+0000 mon.a (mon.0) 2610 : cluster [DBG] osdmap e448: 8 total, 8 up, 8 in 2024-02-19T21:10:56.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:55.790235+0000 osd.7 (osd.7) 195 : cluster [DBG] 61.1a deep-scrub starts 2024-02-19T21:10:56.743 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:55.797789+0000 osd.7 (osd.7) 196 : cluster [DBG] 61.1a deep-scrub ok 2024-02-19T21:10:57.104 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-19T21:10:57.107 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:57.111 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:57.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:57.212 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:57.239 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:57.489 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:57.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:57.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:57.596 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:10:57.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:55.674367+0000 osd.1 (osd.1) 211 : cluster [DBG] 61.9 scrub starts 2024-02-19T21:10:57.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:55.678441+0000 osd.1 (osd.1) 212 : cluster [DBG] 61.9 scrub ok 2024-02-19T21:10:57.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:56.079713+0000 osd.5 (osd.5) 182 : cluster [DBG] 16.f scrub starts 2024-02-19T21:10:57.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:56.083398+0000 osd.5 (osd.5) 183 : cluster [DBG] 16.f scrub ok 2024-02-19T21:10:57.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:56.326377+0000 osd.4 (osd.4) 155 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:10:57.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:56.327312+0000 osd.4 (osd.4) 156 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:10:57.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:55.674367+0000 osd.1 (osd.1) 211 : cluster [DBG] 61.9 scrub starts 2024-02-19T21:10:57.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:55.678441+0000 osd.1 (osd.1) 212 : cluster [DBG] 61.9 scrub ok 2024-02-19T21:10:57.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:56.079713+0000 osd.5 (osd.5) 182 : cluster [DBG] 16.f scrub starts 2024-02-19T21:10:57.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:56.083398+0000 osd.5 (osd.5) 183 : cluster [DBG] 16.f scrub ok 2024-02-19T21:10:57.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:56.326377+0000 osd.4 (osd.4) 155 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:10:57.605 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:56.327312+0000 osd.4 (osd.4) 156 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:10:57.628 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:57.649 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:10:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:57.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:10:57.753 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:10:57.782 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:57.819 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:10:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:10:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:10:57.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:55.674367+0000 osd.1 (osd.1) 211 : cluster [DBG] 61.9 scrub starts 2024-02-19T21:10:57.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:55.678441+0000 osd.1 (osd.1) 212 : cluster [DBG] 61.9 scrub ok 2024-02-19T21:10:57.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:56.079713+0000 osd.5 (osd.5) 182 : cluster [DBG] 16.f scrub starts 2024-02-19T21:10:57.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:56.083398+0000 osd.5 (osd.5) 183 : cluster [DBG] 16.f scrub ok 2024-02-19T21:10:57.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:56.326377+0000 osd.4 (osd.4) 155 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:10:57.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:56.327312+0000 osd.4 (osd.4) 156 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:10:57.880 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:10:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:57.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:10:57.984 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:58.016 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:58.038 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:58.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:58.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:58.141 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:10:58.171 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:58.190 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:10:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:58.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:10:58.294 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:58.322 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:58.341 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:58.343 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:58.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:58.444 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:58.471 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:58.490 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:58.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:58.594 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:58.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:57.127061+0000 osd.5 (osd.5) 184 : cluster [DBG] 16.1f scrub starts 2024-02-19T21:10:58.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:57.130163+0000 osd.5 (osd.5) 185 : cluster [DBG] 16.1f scrub ok 2024-02-19T21:10:58.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:57.225424+0000 mgr.y (mgr.14550) 405 : cluster [DBG] pgmap v658: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-02-19T21:10:58.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:57.334188+0000 osd.4 (osd.4) 157 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:10:58.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:57.335148+0000 osd.4 (osd.4) 158 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:10:58.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:57.421385+0000 osd.6 (osd.6) 177 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:10:58.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:57.422319+0000 osd.6 (osd.6) 178 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:10:58.619 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:57.785541+0000 osd.7 (osd.7) 197 : cluster [DBG] 61.13 scrub starts 2024-02-19T21:10:58.620 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:57.789097+0000 osd.7 (osd.7) 198 : cluster [DBG] 61.13 scrub ok 2024-02-19T21:10:58.620 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:58.640 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:58.641 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:58.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:58.743 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:58.760 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:10:58.769 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:58.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:57.127061+0000 osd.5 (osd.5) 184 : cluster [DBG] 16.1f scrub starts 2024-02-19T21:10:58.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:57.130163+0000 osd.5 (osd.5) 185 : cluster [DBG] 16.1f scrub ok 2024-02-19T21:10:58.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:57.225424+0000 mgr.y (mgr.14550) 405 : cluster [DBG] pgmap v658: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-02-19T21:10:58.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:57.334188+0000 osd.4 (osd.4) 157 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:10:58.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:57.335148+0000 osd.4 (osd.4) 158 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:10:58.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:57.421385+0000 osd.6 (osd.6) 177 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:10:58.787 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:57.422319+0000 osd.6 (osd.6) 178 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:10:58.788 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:57.785541+0000 osd.7 (osd.7) 197 : cluster [DBG] 61.13 scrub starts 2024-02-19T21:10:58.788 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:57.789097+0000 osd.7 (osd.7) 198 : cluster [DBG] 61.13 scrub ok 2024-02-19T21:10:58.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:57.127061+0000 osd.5 (osd.5) 184 : cluster [DBG] 16.1f scrub starts 2024-02-19T21:10:58.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:57.130163+0000 osd.5 (osd.5) 185 : cluster [DBG] 16.1f scrub ok 2024-02-19T21:10:58.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:57.225424+0000 mgr.y (mgr.14550) 405 : cluster [DBG] pgmap v658: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.6 KiB/s wr, 6 op/s 2024-02-19T21:10:58.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:57.334188+0000 osd.4 (osd.4) 157 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:10:58.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:57.335148+0000 osd.4 (osd.4) 158 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:10:58.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:57.421385+0000 osd.6 (osd.6) 177 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:10:58.788 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:57.422319+0000 osd.6 (osd.6) 178 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:10:58.789 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:57.785541+0000 osd.7 (osd.7) 197 : cluster [DBG] 61.13 scrub starts 2024-02-19T21:10:58.789 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:57.789097+0000 osd.7 (osd.7) 198 : cluster [DBG] 61.13 scrub ok 2024-02-19T21:10:58.790 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:58.791 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:58.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:58.892 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:58.918 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:58.936 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:58.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:58.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:59.040 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:10:59.066 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:59.085 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:10:59.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:59.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:10:59.188 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:10:59.217 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:59.316 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:10:59.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:10:59.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:10:59.321 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:10:59.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.327+0000 7f5428740700 1 -- 172.21.15.163:0/2788508955 >> v1:172.21.15.163:6790/0 conn(0x7f54201042c0 legacy=0x7f5420106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.327+0000 7f5428740700 1 -- 172.21.15.163:0/2788508955 shutdown_connections 2024-02-19T21:10:59.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.327+0000 7f5428740700 1 -- 172.21.15.163:0/2788508955 >> 172.21.15.163:0/2788508955 conn(0x7f54200f35a0 msgr2=0x7f54200f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:59.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.327+0000 7f5428740700 1 -- 172.21.15.163:0/2788508955 shutdown_connections 2024-02-19T21:10:59.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.327+0000 7f5428740700 1 -- 172.21.15.163:0/2788508955 wait complete. 2024-02-19T21:10:59.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f5428740700 1 librados: starting msgr at 2024-02-19T21:10:59.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f5428740700 1 librados: starting objecter 2024-02-19T21:10:59.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f5428740700 1 Processor -- start 2024-02-19T21:10:59.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f5428740700 1 -- start start 2024-02-19T21:10:59.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f5428740700 1 librados: setting wanted keys 2024-02-19T21:10:59.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f5428740700 1 librados: calling monclient init 2024-02-19T21:10:59.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f5428740700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f542006cc10 con 0x7f5420100780 2024-02-19T21:10:59.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f5428740700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f542006cde0 con 0x7f54201042c0 2024-02-19T21:10:59.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f5428740700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f542011bb80 con 0x7f54200fd980 2024-02-19T21:10:59.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f54264dc700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f54200fd980 0x7f54200fc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:49812/0 (socket says 172.21.15.163:49812) 2024-02-19T21:10:59.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f5425cdb700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f5420100780 0x7f5420114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:40984/0 (socket says 172.21.15.163:40984) 2024-02-19T21:10:59.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f54264dc700 1 -- 172.21.15.163:0/1735243891 learned_addr learned my addr 172.21.15.163:0/1735243891 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:59.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f54177fe700 10 client.?.objecter ms_handle_connect 0x7f54200fd980 2024-02-19T21:10:59.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f54177fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:59.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f54177fe700 10 client.?.objecter ms_handle_connect 0x7f5420100780 2024-02-19T21:10:59.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.328+0000 7f54177fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:59.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592719231 0 0) 0x7f542011bb80 con 0x7f54200fd980 2024-02-19T21:10:59.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f540c0033f0 con 0x7f54200fd980 2024-02-19T21:10:59.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 10 client.?.objecter ms_handle_connect 0x7f54201042c0 2024-02-19T21:10:59.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:59.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485264772 0 0) 0x7f542006cc10 con 0x7f5420100780 2024-02-19T21:10:59.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f542011bb80 con 0x7f5420100780 2024-02-19T21:10:59.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5342331 0 0) 0x7f542006cde0 con 0x7f54201042c0 2024-02-19T21:10:59.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f542006cc10 con 0x7f54201042c0 2024-02-19T21:10:59.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 518609482 0 0) 0x7f540c0033f0 con 0x7f54200fd980 2024-02-19T21:10:59.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f542006cde0 con 0x7f54200fd980 2024-02-19T21:10:59.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4125561693 0 0) 0x7f542011bb80 con 0x7f5420100780 2024-02-19T21:10:59.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f540c0033f0 con 0x7f5420100780 2024-02-19T21:10:59.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f541c0019d0 con 0x7f54200fd980 2024-02-19T21:10:59.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5410001880 con 0x7f5420100780 2024-02-19T21:10:59.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3102891261 0 0) 0x7f542006cc10 con 0x7f54201042c0 2024-02-19T21:10:59.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f542011bb80 con 0x7f54201042c0 2024-02-19T21:10:59.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5418003280 con 0x7f54201042c0 2024-02-19T21:10:59.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1789476304 0 0) 0x7f542006cde0 con 0x7f54200fd980 2024-02-19T21:10:59.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 >> v1:172.21.15.193:6789/0 conn(0x7f54201042c0 legacy=0x7f5420118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 >> v1:172.21.15.163:6789/0 conn(0x7f5420100780 legacy=0x7f5420114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542011cd20 con 0x7f54200fd980 2024-02-19T21:10:59.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f5428740700 1 -- 172.21.15.163:0/1735243891 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f542011bd70 con 0x7f54200fd980 2024-02-19T21:10:59.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f5428740700 10 client.49910.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:59.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f5428740700 1 -- 172.21.15.163:0/1735243891 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f542011c260 con 0x7f54200fd980 2024-02-19T21:10:59.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f5428740700 1 librados: init done 2024-02-19T21:10:59.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f541c002d40 con 0x7f54200fd980 2024-02-19T21:10:59.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.329+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f541c005ae0 con 0x7f54200fd980 2024-02-19T21:10:59.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.330+0000 7f5428740700 1 -- 172.21.15.163:0/1735243891 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5404004f20 con 0x7f54200fd980 2024-02-19T21:10:59.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.331+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f541c001ea0 con 0x7f54200fd980 2024-02-19T21:10:59.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.331+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(448..448 src has 43..448) v4 ==== 8092+0+0 (unknown 4113603515 0 0) 0x7f541c078c40 con 0x7f54200fd980 2024-02-19T21:10:59.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.331+0000 7f54177fe700 10 client.49910.objecter ms_dispatch 0x7f5420001550 osd_map(448..448 src has 43..448) v4 2024-02-19T21:10:59.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.331+0000 7f54177fe700 3 client.49910.objecter handle_osd_map got epochs [448,448] > 0 2024-02-19T21:10:59.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.331+0000 7f54177fe700 3 client.49910.objecter handle_osd_map decoding full epoch 448 2024-02-19T21:10:59.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.331+0000 7f54177fe700 20 client.49910.objecter dump_active .. 0 homeless 2024-02-19T21:10:59.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.331+0000 7f54177fe700 10 client.49910.objecter ms_handle_connect 0x7f540c063d10 2024-02-19T21:10:59.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.336+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f541c04c070 con 0x7f54200fd980 2024-02-19T21:10:59.421 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:10:59.449 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:10:59.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.526+0000 7f5428740700 1 -- 172.21.15.163:0/1735243891 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd rm-pg-upmap-items", "pgid": "16.1"} v 0) v1 -- 0x7f54040051d0 con 0x7f54200fd980 2024-02-19T21:10:59.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.548+0000 7fa826370700 1 -- 172.21.15.163:0/1263318907 >> v1:172.21.15.163:6789/0 conn(0x7fa8201042b0 legacy=0x7fa820106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.548+0000 7fa826370700 1 -- 172.21.15.163:0/1263318907 shutdown_connections 2024-02-19T21:10:59.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.548+0000 7fa826370700 1 -- 172.21.15.163:0/1263318907 >> 172.21.15.163:0/1263318907 conn(0x7fa8200f2470 msgr2=0x7fa8200f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:59.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.548+0000 7fa826370700 1 -- 172.21.15.163:0/1263318907 shutdown_connections 2024-02-19T21:10:59.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.548+0000 7fa826370700 1 -- 172.21.15.163:0/1263318907 wait complete. 2024-02-19T21:10:59.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.548+0000 7fa826370700 1 librados: starting msgr at 2024-02-19T21:10:59.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.548+0000 7fa826370700 1 librados: starting objecter 2024-02-19T21:10:59.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.548+0000 7fa826370700 1 Processor -- start 2024-02-19T21:10:59.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.548+0000 7fa826370700 1 -- start start 2024-02-19T21:10:59.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.548+0000 7fa826370700 1 librados: setting wanted keys 2024-02-19T21:10:59.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.548+0000 7fa826370700 1 librados: calling monclient init 2024-02-19T21:10:59.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa826370700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa820117270 con 0x7fa8201042b0 2024-02-19T21:10:59.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa826370700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa820118450 con 0x7fa8200f94d0 2024-02-19T21:10:59.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa826370700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa820119650 con 0x7fa8200f8920 2024-02-19T21:10:59.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa82490d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fa8201042b0 0x7fa820115930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:41010/0 (socket says 172.21.15.163:41010) 2024-02-19T21:10:59.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa82490d700 1 -- 172.21.15.163:0/2631054632 learned_addr learned my addr 172.21.15.163:0/2631054632 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:10:59.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81ffff700 1 --1- 172.21.15.163:0/2631054632 >> v1:172.21.15.163:6790/0 conn(0x7fa8200f8920 0x7fa8200ff3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:49836/0 (socket says 172.21.15.163:49836) 2024-02-19T21:10:59.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 10 client.?.objecter ms_handle_connect 0x7fa8200f8920 2024-02-19T21:10:59.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:59.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 10 client.?.objecter ms_handle_connect 0x7fa8201042b0 2024-02-19T21:10:59.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:59.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 10 client.?.objecter ms_handle_connect 0x7fa8200f94d0 2024-02-19T21:10:59.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:10:59.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409212237 0 0) 0x7fa820118450 con 0x7fa8200f94d0 2024-02-19T21:10:59.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8080033f0 con 0x7fa8200f94d0 2024-02-19T21:10:59.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662371942 0 0) 0x7fa820119650 con 0x7fa8200f8920 2024-02-19T21:10:59.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa820118450 con 0x7fa8200f8920 2024-02-19T21:10:59.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044910941 0 0) 0x7fa820117270 con 0x7fa8201042b0 2024-02-19T21:10:59.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa820119650 con 0x7fa8201042b0 2024-02-19T21:10:59.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3142973347 0 0) 0x7fa820118450 con 0x7fa8200f8920 2024-02-19T21:10:59.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa820117270 con 0x7fa8200f8920 2024-02-19T21:10:59.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa810001990 con 0x7fa8200f8920 2024-02-19T21:10:59.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2667805004 0 0) 0x7fa8080033f0 con 0x7fa8200f94d0 2024-02-19T21:10:59.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa820118450 con 0x7fa8200f94d0 2024-02-19T21:10:59.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.549+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa814001880 con 0x7fa8200f94d0 2024-02-19T21:10:59.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.550+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 161188301 0 0) 0x7fa820117270 con 0x7fa8200f8920 2024-02-19T21:10:59.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.550+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 >> v1:172.21.15.193:6789/0 conn(0x7fa8200f94d0 legacy=0x7fa8200ffab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.550+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 >> v1:172.21.15.163:6789/0 conn(0x7fa8201042b0 legacy=0x7fa820115930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.550+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82011a850 con 0x7fa8200f8920 2024-02-19T21:10:59.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.550+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa820117460 con 0x7fa8200f8920 2024-02-19T21:10:59.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.550+0000 7fa826370700 10 client.49916.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:10:59.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.550+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8201178d0 con 0x7fa8200f8920 2024-02-19T21:10:59.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.550+0000 7fa826370700 1 librados: init done 2024-02-19T21:10:59.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.550+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fa810002470 con 0x7fa8200f8920 2024-02-19T21:10:59.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.550+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa810005b00 con 0x7fa8200f8920 2024-02-19T21:10:59.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.551+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa80c004f00 con 0x7fa8200f8920 2024-02-19T21:10:59.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.551+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fa810018a70 con 0x7fa8200f8920 2024-02-19T21:10:59.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.551+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(448..448 src has 43..448) v4 ==== 8092+0+0 (unknown 4113603515 0 0) 0x7fa810078d80 con 0x7fa8200f8920 2024-02-19T21:10:59.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.551+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 osd_map(448..448 src has 43..448) v4 2024-02-19T21:10:59.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.551+0000 7fa81d7fa700 3 client.49916.objecter handle_osd_map got epochs [448,448] > 0 2024-02-19T21:10:59.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.551+0000 7fa81d7fa700 3 client.49916.objecter handle_osd_map decoding full epoch 448 2024-02-19T21:10:59.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.552+0000 7fa81d7fa700 20 client.49916.objecter dump_active .. 0 homeless 2024-02-19T21:10:59.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.552+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v448) v1 ==== 54+0+16 (unknown 1690506659 0 2420000192) 0x7fa81007b180 con 0x7fa8200f8920 2024-02-19T21:10:59.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.552+0000 7fa826370700 10 client.49916.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:59.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.552+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 1 2024-02-19T21:10:59.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.552+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c0051a0 osd=0 3 2024-02-19T21:10:59.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.552+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c0051a0 osd=0 3 2024-02-19T21:10:59.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.552+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c0051a0 osd=0 3 2024-02-19T21:10:59.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.552+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 1 2024-02-19T21:10:59.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.552+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c0051a0 osd=0 3 2024-02-19T21:10:59.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.552+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 0 1 2024-02-19T21:10:59.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.552+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c0051a0 osd=0 4 2024-02-19T21:10:59.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.552+0000 7fa826370700 10 client.49916.objecter _send_command 1 2024-02-19T21:10:59.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.552+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa80c004950 con 0x7fa80c004530 2024-02-19T21:10:59.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.552+0000 7fa81d7fa700 10 client.49916.objecter ms_handle_connect 0x7fa808067da0 2024-02-19T21:10:59.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.552+0000 7fa81d7fa700 10 client.49916.objecter ms_handle_connect 0x7fa80c004530 2024-02-19T21:10:59.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.553+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa80c004950 con 0x7fa80c004530 2024-02-19T21:10:59.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.553+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:10:59.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.553+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 1 = system:0 2024-02-19T21:10:59.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.553+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c0051a0 osd=0 4 2024-02-19T21:10:59.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.553+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 0 1 2024-02-19T21:10:59.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.569+0000 7f54177fe700 1 -- 172.21.15.163:0/1735243891 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap-items", "pgid": "16.1"}]=0 clear 16.1 pg_upmap_items mapping v449) v1 ==== 118+0+0 (unknown 3154265301 0 0) 0x7f541c04a480 con 0x7f54200fd980 2024-02-19T21:10:59.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.571+0000 7f5428740700 10 librados: watch_flush enter 2024-02-19T21:10:59.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.571+0000 7f5428740700 10 librados: watch_flush exit 2024-02-19T21:10:59.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.571+0000 7f5428740700 20 client.49910.objecter shutdown clearing up homeless session... 2024-02-19T21:10:59.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.571+0000 7f5428740700 10 client.49910.objecter successfully canceled tick 2024-02-19T21:10:59.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7fa826370700 10 client.49916.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:59.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 2 2024-02-19T21:10:59.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c0051a0 osd=0 3 2024-02-19T21:10:59.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c0051a0 osd=0 3 2024-02-19T21:10:59.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c0051a0 osd=0 3 2024-02-19T21:10:59.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 2 2024-02-19T21:10:59.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c0051a0 osd=0 3 2024-02-19T21:10:59.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 0 2 2024-02-19T21:10:59.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c0051a0 osd=0 4 2024-02-19T21:10:59.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7fa826370700 10 client.49916.objecter _send_command 2 2024-02-19T21:10:59.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa80c004d30 con 0x7fa80c004530 2024-02-19T21:10:59.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7f5428740700 1 -- 172.21.15.163:0/1735243891 >> v1:172.21.15.163:6800/1552514681 conn(0x7f540c063d10 legacy=0x7f540c0661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7f5428740700 1 -- 172.21.15.163:0/1735243891 >> v1:172.21.15.163:6790/0 conn(0x7f54200fd980 legacy=0x7f54200fc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7f5428740700 1 -- 172.21.15.163:0/1735243891 shutdown_connections 2024-02-19T21:10:59.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7f5428740700 1 -- 172.21.15.163:0/1735243891 >> 172.21.15.163:0/1735243891 conn(0x7f54200f35a0 msgr2=0x7f5420107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:59.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7f5428740700 1 -- 172.21.15.163:0/1735243891 shutdown_connections 2024-02-19T21:10:59.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.573+0000 7f5428740700 1 -- 172.21.15.163:0/1735243891 wait complete. 2024-02-19T21:10:59.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.574+0000 7f5428740700 1 librados: shutdown 2024-02-19T21:10:59.574 INFO:teuthology.orchestra.run.smithi163.stderr:clear 16.1 pg_upmap_items mapping 2024-02-19T21:10:59.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa80c004d30 con 0x7fa80c004530 2024-02-19T21:10:59.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:59.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c0051a0 osd=0 4 2024-02-19T21:10:59.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 0 2 2024-02-19T21:10:59.575 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:10:59.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa826370700 10 client.49916.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:59.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 3 2024-02-19T21:10:59.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c003fe0 osd=1 3 2024-02-19T21:10:59.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c003fe0 osd=1 3 2024-02-19T21:10:59.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c003fe0 osd=1 3 2024-02-19T21:10:59.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 3 2024-02-19T21:10:59.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c003fe0 osd=1 3 2024-02-19T21:10:59.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 1 3 2024-02-19T21:10:59.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c003fe0 osd=1 4 2024-02-19T21:10:59.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa826370700 10 client.49916.objecter _send_command 3 2024-02-19T21:10:59.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa80c00c000 con 0x7fa80c004150 2024-02-19T21:10:59.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.575+0000 7fa81d7fa700 10 client.49916.objecter ms_handle_connect 0x7fa80c004150 2024-02-19T21:10:59.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.576+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa80c00c000 con 0x7fa80c004150 2024-02-19T21:10:59.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.576+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:10:59.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.576+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 3 = system:0 2024-02-19T21:10:59.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.576+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c003fe0 osd=1 4 2024-02-19T21:10:59.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.576+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 1 3 2024-02-19T21:10:59.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.591+0000 7fa826370700 10 client.49916.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:59.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.591+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 4 2024-02-19T21:10:59.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.591+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c003fe0 osd=1 3 2024-02-19T21:10:59.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.591+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c003fe0 osd=1 3 2024-02-19T21:10:59.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.591+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c003fe0 osd=1 3 2024-02-19T21:10:59.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.591+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 4 2024-02-19T21:10:59.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.591+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c003fe0 osd=1 3 2024-02-19T21:10:59.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.591+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 1 4 2024-02-19T21:10:59.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.591+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c003fe0 osd=1 4 2024-02-19T21:10:59.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.591+0000 7fa826370700 10 client.49916.objecter _send_command 4 2024-02-19T21:10:59.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.591+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa80c001850 con 0x7fa80c004150 2024-02-19T21:10:59.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa80c001850 con 0x7fa80c004150 2024-02-19T21:10:59.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:59.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c003fe0 osd=1 4 2024-02-19T21:10:59.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 1 4 2024-02-19T21:10:59.592 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:10:59.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa826370700 10 client.49916.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:59.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 5 2024-02-19T21:10:59.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c003630 osd=2 3 2024-02-19T21:10:59.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c003630 osd=2 3 2024-02-19T21:10:59.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c003630 osd=2 3 2024-02-19T21:10:59.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 5 2024-02-19T21:10:59.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c003630 osd=2 3 2024-02-19T21:10:59.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 2 5 2024-02-19T21:10:59.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c003630 osd=2 4 2024-02-19T21:10:59.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa826370700 10 client.49916.objecter _send_command 5 2024-02-19T21:10:59.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa80c003e70 con 0x7fa80c00d2f0 2024-02-19T21:10:59.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.593+0000 7fa81d7fa700 10 client.49916.objecter ms_handle_connect 0x7fa80c00d2f0 2024-02-19T21:10:59.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.594+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa80c003e70 con 0x7fa80c00d2f0 2024-02-19T21:10:59.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.594+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:10:59.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.594+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 5 = system:0 2024-02-19T21:10:59.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.594+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c003630 osd=2 4 2024-02-19T21:10:59.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.594+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 2 5 2024-02-19T21:10:59.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.609+0000 7fa826370700 10 client.49916.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:59.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.609+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 6 2024-02-19T21:10:59.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.609+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c003630 osd=2 3 2024-02-19T21:10:59.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.609+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c003630 osd=2 3 2024-02-19T21:10:59.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.609+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c003630 osd=2 3 2024-02-19T21:10:59.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.609+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 6 2024-02-19T21:10:59.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.609+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c003630 osd=2 3 2024-02-19T21:10:59.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.609+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 2 6 2024-02-19T21:10:59.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.609+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c003630 osd=2 4 2024-02-19T21:10:59.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.609+0000 7fa826370700 10 client.49916.objecter _send_command 6 2024-02-19T21:10:59.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.609+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa80c00e6c0 con 0x7fa80c00d2f0 2024-02-19T21:10:59.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa80c00e6c0 con 0x7fa80c00d2f0 2024-02-19T21:10:59.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:59.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c003630 osd=2 4 2024-02-19T21:10:59.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 2 6 2024-02-19T21:10:59.609 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:10:59.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa826370700 10 client.49916.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:59.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 7 2024-02-19T21:10:59.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c003430 osd=3 3 2024-02-19T21:10:59.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c003430 osd=3 3 2024-02-19T21:10:59.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c003430 osd=3 3 2024-02-19T21:10:59.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 7 2024-02-19T21:10:59.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c003430 osd=3 3 2024-02-19T21:10:59.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 3 7 2024-02-19T21:10:59.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c003430 osd=3 4 2024-02-19T21:10:59.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa826370700 10 client.49916.objecter _send_command 7 2024-02-19T21:10:59.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.610+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa80c015b00 con 0x7fa80c011e80 2024-02-19T21:10:59.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.611+0000 7fa81d7fa700 10 client.49916.objecter ms_handle_connect 0x7fa80c011e80 2024-02-19T21:10:59.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.611+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa80c015b00 con 0x7fa80c011e80 2024-02-19T21:10:59.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.611+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:10:59.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.611+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 7 = system:0 2024-02-19T21:10:59.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.611+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c003430 osd=3 4 2024-02-19T21:10:59.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.611+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 3 7 2024-02-19T21:10:59.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.626+0000 7fa826370700 10 client.49916.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:59.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.626+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 8 2024-02-19T21:10:59.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.626+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c003430 osd=3 3 2024-02-19T21:10:59.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.626+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c003430 osd=3 3 2024-02-19T21:10:59.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.626+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c003430 osd=3 3 2024-02-19T21:10:59.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.626+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 8 2024-02-19T21:10:59.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.626+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c003430 osd=3 3 2024-02-19T21:10:59.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.626+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 3 8 2024-02-19T21:10:59.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.626+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c003430 osd=3 4 2024-02-19T21:10:59.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.626+0000 7fa826370700 10 client.49916.objecter _send_command 8 2024-02-19T21:10:59.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.626+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa80c015b00 con 0x7fa80c011e80 2024-02-19T21:10:59.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa80c015b00 con 0x7fa80c011e80 2024-02-19T21:10:59.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:59.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c003430 osd=3 4 2024-02-19T21:10:59.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 3 8 2024-02-19T21:10:59.626 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:10:59.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa826370700 10 client.49916.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:59.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 9 2024-02-19T21:10:59.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c00e180 osd=4 3 2024-02-19T21:10:59.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c00e180 osd=4 3 2024-02-19T21:10:59.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c00e180 osd=4 3 2024-02-19T21:10:59.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 9 2024-02-19T21:10:59.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c00e180 osd=4 3 2024-02-19T21:10:59.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 4 9 2024-02-19T21:10:59.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c00e180 osd=4 4 2024-02-19T21:10:59.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa826370700 10 client.49916.objecter _send_command 9 2024-02-19T21:10:59.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.627+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa80c00e360 con 0x7fa80c00ce50 2024-02-19T21:10:59.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.628+0000 7fa81d7fa700 10 client.49916.objecter ms_handle_connect 0x7fa80c00ce50 2024-02-19T21:10:59.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.629+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa80c00e360 con 0x7fa80c00ce50 2024-02-19T21:10:59.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.629+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:10:59.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.629+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 9 = system:0 2024-02-19T21:10:59.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.629+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c00e180 osd=4 4 2024-02-19T21:10:59.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.629+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 4 9 2024-02-19T21:10:59.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.643+0000 7fa826370700 10 client.49916.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:59.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.643+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 10 2024-02-19T21:10:59.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.643+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c00e180 osd=4 3 2024-02-19T21:10:59.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.643+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c00e180 osd=4 3 2024-02-19T21:10:59.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.643+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c00e180 osd=4 3 2024-02-19T21:10:59.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.643+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 10 2024-02-19T21:10:59.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.643+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c00e180 osd=4 3 2024-02-19T21:10:59.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.643+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 4 10 2024-02-19T21:10:59.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.643+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c00e180 osd=4 4 2024-02-19T21:10:59.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.643+0000 7fa826370700 10 client.49916.objecter _send_command 10 2024-02-19T21:10:59.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.643+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa80c00e360 con 0x7fa80c00ce50 2024-02-19T21:10:59.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa80c00e360 con 0x7fa80c00ce50 2024-02-19T21:10:59.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:59.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c00e180 osd=4 4 2024-02-19T21:10:59.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 4 10 2024-02-19T21:10:59.644 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:10:59.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa826370700 10 client.49916.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:59.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 11 2024-02-19T21:10:59.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c00e360 osd=5 3 2024-02-19T21:10:59.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c00e360 osd=5 3 2024-02-19T21:10:59.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c00e360 osd=5 3 2024-02-19T21:10:59.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 11 2024-02-19T21:10:59.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c00e360 osd=5 3 2024-02-19T21:10:59.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 5 11 2024-02-19T21:10:59.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c00e360 osd=5 4 2024-02-19T21:10:59.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa826370700 10 client.49916.objecter _send_command 11 2024-02-19T21:10:59.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.645+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa80c017f70 con 0x7fa80c017a40 2024-02-19T21:10:59.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.646+0000 7fa81d7fa700 10 client.49916.objecter ms_handle_connect 0x7fa80c017a40 2024-02-19T21:10:59.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.646+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa80c017f70 con 0x7fa80c017a40 2024-02-19T21:10:59.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.646+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:10:59.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.646+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 11 = system:0 2024-02-19T21:10:59.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.646+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c00e360 osd=5 4 2024-02-19T21:10:59.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.646+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 5 11 2024-02-19T21:10:59.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.671+0000 7fa826370700 10 client.49916.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:59.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.671+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 12 2024-02-19T21:10:59.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.671+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c00e360 osd=5 3 2024-02-19T21:10:59.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.671+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c00e360 osd=5 3 2024-02-19T21:10:59.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.671+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c00e360 osd=5 3 2024-02-19T21:10:59.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.671+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 12 2024-02-19T21:10:59.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.671+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c00e360 osd=5 3 2024-02-19T21:10:59.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.671+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 5 12 2024-02-19T21:10:59.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.671+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c00e360 osd=5 4 2024-02-19T21:10:59.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.671+0000 7fa826370700 10 client.49916.objecter _send_command 12 2024-02-19T21:10:59.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.671+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa80c017f70 con 0x7fa80c017a40 2024-02-19T21:10:59.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa80c017f70 con 0x7fa80c017a40 2024-02-19T21:10:59.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:59.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c00e360 osd=5 4 2024-02-19T21:10:59.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 5 12 2024-02-19T21:10:59.672 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:10:59.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa826370700 10 client.49916.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:59.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 13 2024-02-19T21:10:59.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c017f70 osd=6 3 2024-02-19T21:10:59.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c017f70 osd=6 3 2024-02-19T21:10:59.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c017f70 osd=6 3 2024-02-19T21:10:59.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 13 2024-02-19T21:10:59.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c017f70 osd=6 3 2024-02-19T21:10:59.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 6 13 2024-02-19T21:10:59.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c017f70 osd=6 4 2024-02-19T21:10:59.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa826370700 10 client.49916.objecter _send_command 13 2024-02-19T21:10:59.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.673+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa80c022c10 con 0x7fa80c01f100 2024-02-19T21:10:59.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.674+0000 7fa81d7fa700 10 client.49916.objecter ms_handle_connect 0x7fa80c01f100 2024-02-19T21:10:59.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.674+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa80c022c10 con 0x7fa80c01f100 2024-02-19T21:10:59.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.674+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:10:59.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.674+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 13 = system:0 2024-02-19T21:10:59.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.674+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c017f70 osd=6 4 2024-02-19T21:10:59.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.674+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 6 13 2024-02-19T21:10:59.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.689+0000 7fa826370700 10 client.49916.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:59.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.689+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 14 2024-02-19T21:10:59.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.689+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c017f70 osd=6 3 2024-02-19T21:10:59.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.689+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c017f70 osd=6 3 2024-02-19T21:10:59.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.689+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c017f70 osd=6 3 2024-02-19T21:10:59.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.689+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 14 2024-02-19T21:10:59.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.689+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c017f70 osd=6 3 2024-02-19T21:10:59.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.689+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 6 14 2024-02-19T21:10:59.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.689+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c017f70 osd=6 4 2024-02-19T21:10:59.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.689+0000 7fa826370700 10 client.49916.objecter _send_command 14 2024-02-19T21:10:59.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.689+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa80c024390 con 0x7fa80c01f100 2024-02-19T21:10:59.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa80c024390 con 0x7fa80c01f100 2024-02-19T21:10:59.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:59.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c017f70 osd=6 4 2024-02-19T21:10:59.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 6 14 2024-02-19T21:10:59.691 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:10:59.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa826370700 10 client.49916.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:10:59.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 15 2024-02-19T21:10:59.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c024390 osd=7 3 2024-02-19T21:10:59.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c024390 osd=7 3 2024-02-19T21:10:59.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c024390 osd=7 3 2024-02-19T21:10:59.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 15 2024-02-19T21:10:59.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c024390 osd=7 3 2024-02-19T21:10:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 7 15 2024-02-19T21:10:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c024390 osd=7 4 2024-02-19T21:10:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa826370700 10 client.49916.objecter _send_command 15 2024-02-19T21:10:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa80c024f50 con 0x7fa80c024a10 2024-02-19T21:10:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.691+0000 7fa81d7fa700 10 client.49916.objecter ms_handle_connect 0x7fa80c024a10 2024-02-19T21:10:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.692+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa80c024f50 con 0x7fa80c024a10 2024-02-19T21:10:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.692+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:10:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.692+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 15 = system:0 2024-02-19T21:10:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.692+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c024390 osd=7 4 2024-02-19T21:10:59.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.692+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 7 15 2024-02-19T21:10:59.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.709+0000 7fa826370700 10 client.49916.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:10:59.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.709+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign -1 16 2024-02-19T21:10:59.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.709+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c024390 osd=7 3 2024-02-19T21:10:59.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.709+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c024390 osd=7 3 2024-02-19T21:10:59.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.709+0000 7fa826370700 20 client.49916.objecter _get_session s=0x7fa80c024390 osd=7 3 2024-02-19T21:10:59.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.709+0000 7fa826370700 15 client.49916.objecter _session_command_op_remove -1 16 2024-02-19T21:10:59.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.709+0000 7fa826370700 20 client.49916.objecter get_session s=0x7fa80c024390 osd=7 3 2024-02-19T21:10:59.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.709+0000 7fa826370700 15 client.49916.objecter _session_command_op_assign 7 16 2024-02-19T21:10:59.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.709+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c024390 osd=7 4 2024-02-19T21:10:59.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.709+0000 7fa826370700 10 client.49916.objecter _send_command 16 2024-02-19T21:10:59.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.709+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fa80c024f10 con 0x7fa80c024a10 2024-02-19T21:10:59.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa81d7fa700 1 -- 172.21.15.163:0/2631054632 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fa80c024f10 con 0x7fa80c024a10 2024-02-19T21:10:59.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa81d7fa700 10 client.49916.objecter ms_dispatch 0x7fa820001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:10:59.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa81d7fa700 10 client.49916.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa81d7fa700 20 client.49916.objecter put_session s=0x7fa80c024390 osd=7 4 2024-02-19T21:10:59.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa81d7fa700 15 client.49916.objecter _session_command_op_remove 7 16 2024-02-19T21:10:59.709 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:10:59.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 10 librados: watch_flush enter 2024-02-19T21:10:59.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 10 librados: watch_flush exit 2024-02-19T21:10:59.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 10 client.49916.objecter close_session for osd.0 2024-02-19T21:10:59.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 >> v1:172.21.15.163:6801/1559418115 conn(0x7fa80c004530 legacy=0x7fa80c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c0051a0 osd=0 1 2024-02-19T21:10:59.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 10 client.49916.objecter close_session for osd.1 2024-02-19T21:10:59.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 >> v1:172.21.15.163:6805/3125047432 conn(0x7fa80c004150 legacy=0x7fa80c00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c003fe0 osd=1 1 2024-02-19T21:10:59.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 10 client.49916.objecter close_session for osd.2 2024-02-19T21:10:59.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 >> v1:172.21.15.163:6809/1563158487 conn(0x7fa80c00d2f0 legacy=0x7fa80c00d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c003630 osd=2 1 2024-02-19T21:10:59.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 10 client.49916.objecter close_session for osd.3 2024-02-19T21:10:59.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 >> v1:172.21.15.163:6813/2112279769 conn(0x7fa80c011e80 legacy=0x7fa80c014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c003430 osd=3 1 2024-02-19T21:10:59.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 10 client.49916.objecter close_session for osd.4 2024-02-19T21:10:59.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 >> v1:172.21.15.193:6800/3129716965 conn(0x7fa80c00ce50 legacy=0x7fa80c015ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa82490d700 1 -- 172.21.15.163:0/2631054632 reap_dead start 2024-02-19T21:10:59.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c00e180 osd=4 1 2024-02-19T21:10:59.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 10 client.49916.objecter close_session for osd.5 2024-02-19T21:10:59.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 >> v1:172.21.15.193:6804/805161991 conn(0x7fa80c017a40 legacy=0x7fa80c01d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa82490d700 1 -- 172.21.15.163:0/2631054632 reap_dead start 2024-02-19T21:10:59.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c00e360 osd=5 1 2024-02-19T21:10:59.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 10 client.49916.objecter close_session for osd.6 2024-02-19T21:10:59.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 >> v1:172.21.15.193:6808/787604615 conn(0x7fa80c01f100 legacy=0x7fa80c0214f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c017f70 osd=6 1 2024-02-19T21:10:59.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 10 client.49916.objecter close_session for osd.7 2024-02-19T21:10:59.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 >> v1:172.21.15.193:6812/157927432 conn(0x7fa80c024a10 legacy=0x7fa80c022c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 20 client.49916.objecter put_session s=0x7fa80c024390 osd=7 1 2024-02-19T21:10:59.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 20 client.49916.objecter shutdown clearing up homeless session... 2024-02-19T21:10:59.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 10 client.49916.objecter successfully canceled tick 2024-02-19T21:10:59.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa82490d700 1 -- 172.21.15.163:0/2631054632 reap_dead start 2024-02-19T21:10:59.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 >> v1:172.21.15.163:6800/1552514681 conn(0x7fa808067da0 legacy=0x7fa80806a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 >> v1:172.21.15.163:6790/0 conn(0x7fa8200f8920 legacy=0x7fa8200ff3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:10:59.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 shutdown_connections 2024-02-19T21:10:59.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 >> 172.21.15.163:0/2631054632 conn(0x7fa8200f2470 msgr2=0x7fa820103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:10:59.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 shutdown_connections 2024-02-19T21:10:59.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 1 -- 172.21.15.163:0/2631054632 wait complete. 2024-02-19T21:10:59.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:10:59.711+0000 7fa826370700 1 librados: shutdown 2024-02-19T21:10:59.714 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.714 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.714 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.714 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.714 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.714 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.714 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.714 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:10:59.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:58.093134+0000 osd.5 (osd.5) 186 : cluster [DBG] 5.f scrub starts 2024-02-19T21:10:59.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:58.096020+0000 osd.5 (osd.5) 187 : cluster [DBG] 5.f scrub ok 2024-02-19T21:10:59.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:58.331744+0000 osd.4 (osd.4) 159 : cluster [DBG] 2.f scrub starts 2024-02-19T21:10:59.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:58.333159+0000 osd.4 (osd.4) 160 : cluster [DBG] 2.f scrub ok 2024-02-19T21:10:59.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:58.687018+0000 osd.3 (osd.3) 193 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:10:59.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:58.688161+0000 osd.3 (osd.3) 194 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:10:59.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:59.527983+0000 mon.c (mon.2) 396 : audit [INF] from='client.? v1:172.21.15.163:0/1735243891' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "16.1"}]: dispatch 2024-02-19T21:10:59.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:59.528300+0000 mon.a (mon.0) 2611 : audit [INF] from='client.49910 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "16.1"}]: dispatch 2024-02-19T21:10:59.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:59.552910+0000 mon.c (mon.2) 397 : audit [DBG] from='client.? v1:172.21.15.163:0/2631054632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:59.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:58.093134+0000 osd.5 (osd.5) 186 : cluster [DBG] 5.f scrub starts 2024-02-19T21:10:59.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:58.096020+0000 osd.5 (osd.5) 187 : cluster [DBG] 5.f scrub ok 2024-02-19T21:10:59.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:58.331744+0000 osd.4 (osd.4) 159 : cluster [DBG] 2.f scrub starts 2024-02-19T21:10:59.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:58.333159+0000 osd.4 (osd.4) 160 : cluster [DBG] 2.f scrub ok 2024-02-19T21:10:59.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:58.687018+0000 osd.3 (osd.3) 193 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:10:59.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:58.688161+0000 osd.3 (osd.3) 194 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:10:59.737 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:59.527983+0000 mon.c (mon.2) 396 : audit [INF] from='client.? v1:172.21.15.163:0/1735243891' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "16.1"}]: dispatch 2024-02-19T21:10:59.738 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:59.528300+0000 mon.a (mon.0) 2611 : audit [INF] from='client.49910 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "16.1"}]: dispatch 2024-02-19T21:10:59.738 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:10:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:59.552910+0000 mon.c (mon.2) 397 : audit [DBG] from='client.? v1:172.21.15.163:0/2631054632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:10:59.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:58.093134+0000 osd.5 (osd.5) 186 : cluster [DBG] 5.f scrub starts 2024-02-19T21:10:59.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:58.096020+0000 osd.5 (osd.5) 187 : cluster [DBG] 5.f scrub ok 2024-02-19T21:10:59.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:58.331744+0000 osd.4 (osd.4) 159 : cluster [DBG] 2.f scrub starts 2024-02-19T21:10:59.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:58.333159+0000 osd.4 (osd.4) 160 : cluster [DBG] 2.f scrub ok 2024-02-19T21:10:59.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:58.687018+0000 osd.3 (osd.3) 193 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:10:59.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:58.688161+0000 osd.3 (osd.3) 194 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:10:59.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:59.527983+0000 mon.c (mon.2) 396 : audit [INF] from='client.? v1:172.21.15.163:0/1735243891' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "16.1"}]: dispatch 2024-02-19T21:10:59.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:59.528300+0000 mon.a (mon.0) 2611 : audit [INF] from='client.49910 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "16.1"}]: dispatch 2024-02-19T21:10:59.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:10:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:59.552910+0000 mon.c (mon.2) 397 : audit [DBG] from='client.? v1:172.21.15.163:0/2631054632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:00.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:59.225827+0000 mgr.y (mgr.14550) 406 : cluster [DBG] pgmap v659: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-02-19T21:11:00.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:59.312062+0000 osd.4 (osd.4) 161 : cluster [DBG] 5.1 scrub starts 2024-02-19T21:11:00.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:59.312857+0000 osd.4 (osd.4) 162 : cluster [DBG] 5.1 scrub ok 2024-02-19T21:11:00.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:59.404401+0000 osd.6 (osd.6) 179 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:11:00.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:59.405413+0000 osd.6 (osd.6) 180 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:11:00.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:59.566566+0000 mon.a (mon.0) 2612 : audit [INF] from='client.49910 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "16.1"}]': finished 2024-02-19T21:11:00.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:59.566626+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e449: 8 total, 8 up, 8 in 2024-02-19T21:11:00.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:10:59.665137+0000 mon.a (mon.0) 2614 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:11:00.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:59.683385+0000 osd.3 (osd.3) 195 : cluster [DBG] 4.14 scrub starts 2024-02-19T21:11:00.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:59.685146+0000 osd.3 (osd.3) 196 : cluster [DBG] 4.14 scrub ok 2024-02-19T21:11:00.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:59.225827+0000 mgr.y (mgr.14550) 406 : cluster [DBG] pgmap v659: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-02-19T21:11:00.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:59.312062+0000 osd.4 (osd.4) 161 : cluster [DBG] 5.1 scrub starts 2024-02-19T21:11:00.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:59.312857+0000 osd.4 (osd.4) 162 : cluster [DBG] 5.1 scrub ok 2024-02-19T21:11:00.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:59.404401+0000 osd.6 (osd.6) 179 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:11:00.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:59.405413+0000 osd.6 (osd.6) 180 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:11:00.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:59.566566+0000 mon.a (mon.0) 2612 : audit [INF] from='client.49910 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "16.1"}]': finished 2024-02-19T21:11:00.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:59.566626+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e449: 8 total, 8 up, 8 in 2024-02-19T21:11:00.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:10:59.665137+0000 mon.a (mon.0) 2614 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:11:00.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:59.683385+0000 osd.3 (osd.3) 195 : cluster [DBG] 4.14 scrub starts 2024-02-19T21:11:00.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:59.685146+0000 osd.3 (osd.3) 196 : cluster [DBG] 4.14 scrub ok 2024-02-19T21:11:00.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:11:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:11:00.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:59.225827+0000 mgr.y (mgr.14550) 406 : cluster [DBG] pgmap v659: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-02-19T21:11:00.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:59.312062+0000 osd.4 (osd.4) 161 : cluster [DBG] 5.1 scrub starts 2024-02-19T21:11:00.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:59.312857+0000 osd.4 (osd.4) 162 : cluster [DBG] 5.1 scrub ok 2024-02-19T21:11:00.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:59.404401+0000 osd.6 (osd.6) 179 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:11:00.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:59.405413+0000 osd.6 (osd.6) 180 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:11:00.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:59.566566+0000 mon.a (mon.0) 2612 : audit [INF] from='client.49910 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "16.1"}]': finished 2024-02-19T21:11:00.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:59.566626+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e449: 8 total, 8 up, 8 in 2024-02-19T21:11:00.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:10:59.665137+0000 mon.a (mon.0) 2614 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:11:00.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:59.683385+0000 osd.3 (osd.3) 195 : cluster [DBG] 4.14 scrub starts 2024-02-19T21:11:00.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:59.685146+0000 osd.3 (osd.3) 196 : cluster [DBG] 4.14 scrub ok 2024-02-19T21:11:01.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:59.779475+0000 osd.1 (osd.1) 213 : cluster [DBG] 16.b scrub starts 2024-02-19T21:11:01.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:10:59.782362+0000 osd.1 (osd.1) 214 : cluster [DBG] 16.b scrub ok 2024-02-19T21:11:01.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:00.189212+0000 osd.2 (osd.2) 174 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:11:01.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:00.191200+0000 osd.2 (osd.2) 175 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:11:01.828 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:00.381985+0000 osd.6 (osd.6) 181 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:11:01.828 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:00.382942+0000 osd.6 (osd.6) 182 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:11:01.828 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:00.439418+0000 mgr.y (mgr.14550) 407 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:01.828 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:00.572921+0000 mon.a (mon.0) 2615 : cluster [DBG] osdmap e450: 8 total, 8 up, 8 in 2024-02-19T21:11:01.828 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:59.779475+0000 osd.1 (osd.1) 213 : cluster [DBG] 16.b scrub starts 2024-02-19T21:11:01.828 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:10:59.782362+0000 osd.1 (osd.1) 214 : cluster [DBG] 16.b scrub ok 2024-02-19T21:11:01.828 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:00.189212+0000 osd.2 (osd.2) 174 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:11:01.828 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:00.191200+0000 osd.2 (osd.2) 175 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:11:01.828 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:00.381985+0000 osd.6 (osd.6) 181 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:11:01.828 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:00.382942+0000 osd.6 (osd.6) 182 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:11:01.829 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:00.439418+0000 mgr.y (mgr.14550) 407 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:01.829 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:00.572921+0000 mon.a (mon.0) 2615 : cluster [DBG] osdmap e450: 8 total, 8 up, 8 in 2024-02-19T21:11:02.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:59.779475+0000 osd.1 (osd.1) 213 : cluster [DBG] 16.b scrub starts 2024-02-19T21:11:02.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:10:59.782362+0000 osd.1 (osd.1) 214 : cluster [DBG] 16.b scrub ok 2024-02-19T21:11:02.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:00.189212+0000 osd.2 (osd.2) 174 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:11:02.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:00.191200+0000 osd.2 (osd.2) 175 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:11:02.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:00.381985+0000 osd.6 (osd.6) 181 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:11:02.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:00.382942+0000 osd.6 (osd.6) 182 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:11:02.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:00.439418+0000 mgr.y (mgr.14550) 407 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:02.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:00.572921+0000 mon.a (mon.0) 2615 : cluster [DBG] osdmap e450: 8 total, 8 up, 8 in 2024-02-19T21:11:02.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:00.750314+0000 osd.1 (osd.1) 215 : cluster [DBG] 16.9 scrub starts 2024-02-19T21:11:02.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:00.754321+0000 osd.1 (osd.1) 216 : cluster [DBG] 16.9 scrub ok 2024-02-19T21:11:02.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:01.076089+0000 osd.5 (osd.5) 188 : cluster [DBG] 16.1b scrub starts 2024-02-19T21:11:02.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:01.079347+0000 osd.5 (osd.5) 189 : cluster [DBG] 16.1b scrub ok 2024-02-19T21:11:02.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:01.123282+0000 osd.1 (osd.1) 217 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:11:02.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:01.123442+0000 osd.1 (osd.1) 218 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:11:02.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:01.160972+0000 osd.2 (osd.2) 176 : cluster [DBG] 61.4 scrub starts 2024-02-19T21:11:02.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:01.162733+0000 osd.2 (osd.2) 177 : cluster [DBG] 61.4 scrub ok 2024-02-19T21:11:02.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:01.226508+0000 mgr.y (mgr.14550) 408 : cluster [DBG] pgmap v662: 228 pgs: 1 peering, 227 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:11:02.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:01.376035+0000 osd.6 (osd.6) 183 : cluster [DBG] 16.0 scrub starts 2024-02-19T21:11:02.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:01.378921+0000 osd.6 (osd.6) 184 : cluster [DBG] 16.0 scrub ok 2024-02-19T21:11:02.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:01.581955+0000 mon.a (mon.0) 2616 : cluster [DBG] osdmap e451: 8 total, 8 up, 8 in 2024-02-19T21:11:02.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:01.688889+0000 osd.7 (osd.7) 199 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:11:02.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:01.692256+0000 osd.7 (osd.7) 200 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:11:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:00.750314+0000 osd.1 (osd.1) 215 : cluster [DBG] 16.9 scrub starts 2024-02-19T21:11:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:00.754321+0000 osd.1 (osd.1) 216 : cluster [DBG] 16.9 scrub ok 2024-02-19T21:11:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:01.076089+0000 osd.5 (osd.5) 188 : cluster [DBG] 16.1b scrub starts 2024-02-19T21:11:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:01.079347+0000 osd.5 (osd.5) 189 : cluster [DBG] 16.1b scrub ok 2024-02-19T21:11:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:01.123282+0000 osd.1 (osd.1) 217 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:11:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:01.123442+0000 osd.1 (osd.1) 218 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:11:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:01.160972+0000 osd.2 (osd.2) 176 : cluster [DBG] 61.4 scrub starts 2024-02-19T21:11:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:01.162733+0000 osd.2 (osd.2) 177 : cluster [DBG] 61.4 scrub ok 2024-02-19T21:11:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:01.226508+0000 mgr.y (mgr.14550) 408 : cluster [DBG] pgmap v662: 228 pgs: 1 peering, 227 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:11:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:01.376035+0000 osd.6 (osd.6) 183 : cluster [DBG] 16.0 scrub starts 2024-02-19T21:11:02.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:01.378921+0000 osd.6 (osd.6) 184 : cluster [DBG] 16.0 scrub ok 2024-02-19T21:11:02.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:01.581955+0000 mon.a (mon.0) 2616 : cluster [DBG] osdmap e451: 8 total, 8 up, 8 in 2024-02-19T21:11:02.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:01.688889+0000 osd.7 (osd.7) 199 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:11:02.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:01.692256+0000 osd.7 (osd.7) 200 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:11:02.792 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:11:02.792 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:11:02.792 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-02-19T21:11:02.792 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-02-19T21:11:02.792 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:11:02.810 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:02.813 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:02.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:02.914 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:02.944 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:00.750314+0000 osd.1 (osd.1) 215 : cluster [DBG] 16.9 scrub starts 2024-02-19T21:11:02.944 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:00.754321+0000 osd.1 (osd.1) 216 : cluster [DBG] 16.9 scrub ok 2024-02-19T21:11:02.944 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:01.076089+0000 osd.5 (osd.5) 188 : cluster [DBG] 16.1b scrub starts 2024-02-19T21:11:02.944 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:01.079347+0000 osd.5 (osd.5) 189 : cluster [DBG] 16.1b scrub ok 2024-02-19T21:11:02.944 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:01.123282+0000 osd.1 (osd.1) 217 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:11:02.944 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:01.123442+0000 osd.1 (osd.1) 218 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:11:02.944 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:01.160972+0000 osd.2 (osd.2) 176 : cluster [DBG] 61.4 scrub starts 2024-02-19T21:11:02.944 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:01.162733+0000 osd.2 (osd.2) 177 : cluster [DBG] 61.4 scrub ok 2024-02-19T21:11:02.944 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:01.226508+0000 mgr.y (mgr.14550) 408 : cluster [DBG] pgmap v662: 228 pgs: 1 peering, 227 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:11:02.944 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:01.376035+0000 osd.6 (osd.6) 183 : cluster [DBG] 16.0 scrub starts 2024-02-19T21:11:02.944 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:01.378921+0000 osd.6 (osd.6) 184 : cluster [DBG] 16.0 scrub ok 2024-02-19T21:11:02.945 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:01.581955+0000 mon.a (mon.0) 2616 : cluster [DBG] osdmap e451: 8 total, 8 up, 8 in 2024-02-19T21:11:02.945 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:01.688889+0000 osd.7 (osd.7) 199 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:11:02.945 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:01.692256+0000 osd.7 (osd.7) 200 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:11:02.946 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:02.967 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:02.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:03.071 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:03.100 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:03.120 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:03.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:03.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:03.224 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:03.252 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:03.272 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:03.274 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:03.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:03.375 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:11:03.403 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:03.421 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:11:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:03.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:11:03.525 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:11:03.552 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:03.570 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:11:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:03.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:11:03.674 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:03.700 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:03.718 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:03.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:03.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:03.746 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:11:03] "GET /metrics HTTP/1.1" 200 189820 "" "Prometheus/2.33.4" 2024-02-19T21:11:03.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:02.188183+0000 osd.2 (osd.2) 178 : cluster [DBG] 16.17 deep-scrub starts 2024-02-19T21:11:03.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:02.189504+0000 osd.2 (osd.2) 179 : cluster [DBG] 16.17 deep-scrub ok 2024-02-19T21:11:03.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:02.338825+0000 osd.6 (osd.6) 185 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:11:03.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:02.342167+0000 osd.6 (osd.6) 186 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:11:03.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:02.406619+0000 osd.4 (osd.4) 163 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:11:03.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:02.407539+0000 osd.4 (osd.4) 164 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:11:03.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:02.669352+0000 osd.3 (osd.3) 197 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:11:03.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:02.671830+0000 osd.3 (osd.3) 198 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:11:03.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:02.719725+0000 osd.7 (osd.7) 201 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:11:03.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:02.720579+0000 osd.7 (osd.7) 202 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:11:03.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:02.923724+0000 osd.7 (osd.7) 203 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:11:03.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:02.923907+0000 osd.7 (osd.7) 204 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:11:03.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:02.188183+0000 osd.2 (osd.2) 178 : cluster [DBG] 16.17 deep-scrub starts 2024-02-19T21:11:03.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:02.189504+0000 osd.2 (osd.2) 179 : cluster [DBG] 16.17 deep-scrub ok 2024-02-19T21:11:03.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:02.338825+0000 osd.6 (osd.6) 185 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:11:03.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:02.342167+0000 osd.6 (osd.6) 186 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:11:03.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:02.406619+0000 osd.4 (osd.4) 163 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:11:03.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:02.407539+0000 osd.4 (osd.4) 164 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:11:03.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:02.669352+0000 osd.3 (osd.3) 197 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:11:03.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:02.671830+0000 osd.3 (osd.3) 198 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:11:03.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:02.719725+0000 osd.7 (osd.7) 201 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:11:03.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:02.720579+0000 osd.7 (osd.7) 202 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:11:03.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:02.923724+0000 osd.7 (osd.7) 203 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:11:03.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:02.923907+0000 osd.7 (osd.7) 204 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:11:03.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:02.188183+0000 osd.2 (osd.2) 178 : cluster [DBG] 16.17 deep-scrub starts 2024-02-19T21:11:03.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:02.189504+0000 osd.2 (osd.2) 179 : cluster [DBG] 16.17 deep-scrub ok 2024-02-19T21:11:03.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:02.338825+0000 osd.6 (osd.6) 185 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:11:03.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:02.342167+0000 osd.6 (osd.6) 186 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:11:03.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:02.406619+0000 osd.4 (osd.4) 163 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:11:03.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:02.407539+0000 osd.4 (osd.4) 164 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:11:03.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:02.669352+0000 osd.3 (osd.3) 197 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:11:03.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:02.671830+0000 osd.3 (osd.3) 198 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:11:03.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:02.719725+0000 osd.7 (osd.7) 201 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:11:03.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:02.720579+0000 osd.7 (osd.7) 202 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:11:03.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:02.923724+0000 osd.7 (osd.7) 203 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:11:03.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:02.923907+0000 osd.7 (osd.7) 204 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:11:03.822 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:11:03.848 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:03.865 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:11:03.867 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:03.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:11:03.905 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-19T21:11:03.941 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:03.969 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:03.995 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:04.014 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:04.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:04.120 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:04.147 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:04.173 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:04.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:04.279 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:04.318 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:04.442 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:04.444 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:04.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:04.546 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:04.573 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:04.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:03.194271+0000 osd.2 (osd.2) 180 : cluster [DBG] 16.1d scrub starts 2024-02-19T21:11:04.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:03.195612+0000 osd.2 (osd.2) 181 : cluster [DBG] 16.1d scrub ok 2024-02-19T21:11:04.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:03.226993+0000 mgr.y (mgr.14550) 409 : cluster [DBG] pgmap v664: 228 pgs: 1 peering, 227 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:11:04.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:03.378406+0000 osd.6 (osd.6) 187 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:11:04.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:03.379327+0000 osd.6 (osd.6) 188 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:11:04.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:03.422074+0000 osd.4 (osd.4) 165 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:11:04.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:03.423762+0000 osd.4 (osd.4) 166 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:11:04.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:03.587271+0000 mon.a (mon.0) 2617 : cluster [DBG] osdmap e452: 8 total, 8 up, 8 in 2024-02-19T21:11:04.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:03.743698+0000 osd.7 (osd.7) 205 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:11:04.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:03.747101+0000 osd.7 (osd.7) 206 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:11:04.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:03.194271+0000 osd.2 (osd.2) 180 : cluster [DBG] 16.1d scrub starts 2024-02-19T21:11:04.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:03.195612+0000 osd.2 (osd.2) 181 : cluster [DBG] 16.1d scrub ok 2024-02-19T21:11:04.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:03.226993+0000 mgr.y (mgr.14550) 409 : cluster [DBG] pgmap v664: 228 pgs: 1 peering, 227 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:11:04.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:03.378406+0000 osd.6 (osd.6) 187 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:11:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:03.379327+0000 osd.6 (osd.6) 188 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:11:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:03.422074+0000 osd.4 (osd.4) 165 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:11:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:03.423762+0000 osd.4 (osd.4) 166 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:11:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:03.587271+0000 mon.a (mon.0) 2617 : cluster [DBG] osdmap e452: 8 total, 8 up, 8 in 2024-02-19T21:11:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:03.743698+0000 osd.7 (osd.7) 205 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:11:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:03.747101+0000 osd.7 (osd.7) 206 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:11:04.771 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:04.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:04.789 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:04.875 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:11:04.903 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:05.051 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:11:05.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:05.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:11:05.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:03.194271+0000 osd.2 (osd.2) 180 : cluster [DBG] 16.1d scrub starts 2024-02-19T21:11:05.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:03.195612+0000 osd.2 (osd.2) 181 : cluster [DBG] 16.1d scrub ok 2024-02-19T21:11:05.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:03.226993+0000 mgr.y (mgr.14550) 409 : cluster [DBG] pgmap v664: 228 pgs: 1 peering, 227 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:11:05.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:03.378406+0000 osd.6 (osd.6) 187 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:11:05.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:03.379327+0000 osd.6 (osd.6) 188 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:11:05.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:03.422074+0000 osd.4 (osd.4) 165 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:11:05.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:03.423762+0000 osd.4 (osd.4) 166 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:11:05.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:03.587271+0000 mon.a (mon.0) 2617 : cluster [DBG] osdmap e452: 8 total, 8 up, 8 in 2024-02-19T21:11:05.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:03.743698+0000 osd.7 (osd.7) 205 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:11:05.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:03.747101+0000 osd.7 (osd.7) 206 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:11:05.155 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:11:05.182 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:05.201 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:11:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:05.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:11:05.304 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:11:05.331 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:05.349 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:11:05.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:05.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:11:05.445 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:11:05.453 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:05.479 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:05.498 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:05.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:05.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:05.601 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:11:05.628 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:05.647 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:11:05.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:05.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:11:05.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:03.817742+0000 osd.1 (osd.1) 219 : cluster [DBG] 61.b scrub starts 2024-02-19T21:11:05.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:03.821862+0000 osd.1 (osd.1) 220 : cluster [DBG] 61.b scrub ok 2024-02-19T21:11:05.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:04.343882+0000 osd.6 (osd.6) 189 : cluster [DBG] 4.a scrub starts 2024-02-19T21:11:05.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:04.345831+0000 osd.6 (osd.6) 190 : cluster [DBG] 4.a scrub ok 2024-02-19T21:11:05.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:04.678426+0000 mon.a (mon.0) 2618 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:11:05.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:04.744780+0000 osd.7 (osd.7) 207 : cluster [DBG] 61.10 deep-scrub starts 2024-02-19T21:11:05.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:04.749928+0000 osd.7 (osd.7) 208 : cluster [DBG] 61.10 deep-scrub ok 2024-02-19T21:11:05.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:03.817742+0000 osd.1 (osd.1) 219 : cluster [DBG] 61.b scrub starts 2024-02-19T21:11:05.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:03.821862+0000 osd.1 (osd.1) 220 : cluster [DBG] 61.b scrub ok 2024-02-19T21:11:05.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:04.343882+0000 osd.6 (osd.6) 189 : cluster [DBG] 4.a scrub starts 2024-02-19T21:11:05.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:04.345831+0000 osd.6 (osd.6) 190 : cluster [DBG] 4.a scrub ok 2024-02-19T21:11:05.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:04.678426+0000 mon.a (mon.0) 2618 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:11:05.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:04.744780+0000 osd.7 (osd.7) 207 : cluster [DBG] 61.10 deep-scrub starts 2024-02-19T21:11:05.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:04.749928+0000 osd.7 (osd.7) 208 : cluster [DBG] 61.10 deep-scrub ok 2024-02-19T21:11:05.755 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:11:05.780 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:03.817742+0000 osd.1 (osd.1) 219 : cluster [DBG] 61.b scrub starts 2024-02-19T21:11:05.780 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:03.821862+0000 osd.1 (osd.1) 220 : cluster [DBG] 61.b scrub ok 2024-02-19T21:11:05.780 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:04.343882+0000 osd.6 (osd.6) 189 : cluster [DBG] 4.a scrub starts 2024-02-19T21:11:05.780 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:04.345831+0000 osd.6 (osd.6) 190 : cluster [DBG] 4.a scrub ok 2024-02-19T21:11:05.780 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:04.678426+0000 mon.a (mon.0) 2618 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:11:05.780 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:04.744780+0000 osd.7 (osd.7) 207 : cluster [DBG] 61.10 deep-scrub starts 2024-02-19T21:11:05.780 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:04.749928+0000 osd.7 (osd.7) 208 : cluster [DBG] 61.10 deep-scrub ok 2024-02-19T21:11:05.782 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:05.800 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:11:05.803 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:05.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:11:05.904 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:11:05.930 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:05.948 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:11:05.950 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:05.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:11:06.052 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:06.078 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:06.096 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:06.098 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:06.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:06.199 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:06.228 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:06.753 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:05.026881+0000 osd.5 (osd.5) 190 : cluster [DBG] 61.7 scrub starts 2024-02-19T21:11:06.753 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:05.030350+0000 osd.5 (osd.5) 191 : cluster [DBG] 61.7 scrub ok 2024-02-19T21:11:06.753 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:05.227889+0000 mgr.y (mgr.14550) 410 : cluster [DBG] pgmap v666: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 542 B/s wr, 4 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:11:06.753 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:05.363922+0000 osd.6 (osd.6) 191 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:11:06.753 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:05.364728+0000 osd.6 (osd.6) 192 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:11:06.753 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:05.603090+0000 osd.3 (osd.3) 199 : cluster [DBG] 16.16 scrub starts 2024-02-19T21:11:06.753 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:05.604021+0000 osd.3 (osd.3) 200 : cluster [DBG] 16.16 scrub ok 2024-02-19T21:11:06.753 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:05.026881+0000 osd.5 (osd.5) 190 : cluster [DBG] 61.7 scrub starts 2024-02-19T21:11:06.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:05.030350+0000 osd.5 (osd.5) 191 : cluster [DBG] 61.7 scrub ok 2024-02-19T21:11:06.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:05.227889+0000 mgr.y (mgr.14550) 410 : cluster [DBG] pgmap v666: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 542 B/s wr, 4 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:11:06.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:05.363922+0000 osd.6 (osd.6) 191 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:11:06.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:05.364728+0000 osd.6 (osd.6) 192 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:11:06.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:05.603090+0000 osd.3 (osd.3) 199 : cluster [DBG] 16.16 scrub starts 2024-02-19T21:11:06.754 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:05.604021+0000 osd.3 (osd.3) 200 : cluster [DBG] 16.16 scrub ok 2024-02-19T21:11:06.785 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:06.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:06.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:06.796 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:06.889 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:11:06.918 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:05.026881+0000 osd.5 (osd.5) 190 : cluster [DBG] 61.7 scrub starts 2024-02-19T21:11:06.919 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:05.030350+0000 osd.5 (osd.5) 191 : cluster [DBG] 61.7 scrub ok 2024-02-19T21:11:06.919 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:05.227889+0000 mgr.y (mgr.14550) 410 : cluster [DBG] pgmap v666: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 542 B/s wr, 4 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:11:06.919 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:05.363922+0000 osd.6 (osd.6) 191 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:11:06.919 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:05.364728+0000 osd.6 (osd.6) 192 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:11:06.919 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:05.603090+0000 osd.3 (osd.3) 199 : cluster [DBG] 16.16 scrub starts 2024-02-19T21:11:06.919 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:05.604021+0000 osd.3 (osd.3) 200 : cluster [DBG] 16.16 scrub ok 2024-02-19T21:11:06.921 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:06.941 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:11:06.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:06.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:11:07.044 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:07.074 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:07.147 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 3, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:11:07.147 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:11:07.147 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-19T21:11:07.147 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-19T21:11:07.273 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:07.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:07.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:07.377 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:11:07.404 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:07.640 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:11:07.640 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-02-19T21:11:07.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-02-19T21:11:07.640 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-02-19T21:11:07.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.640 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.35189.0:5 36.12 36:4a6d8757:::3.obj:head [write 0~115] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-19T21:11:07.640 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:43.796288+0000", 2024-02-19T21:11:07.640 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 383.73048277100003, 2024-02-19T21:11:07.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023332470000000001, 2024-02-19T21:11:07.640 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.640 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.641 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:07.641 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.35189", 2024-02-19T21:11:07.641 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3441459623", 2024-02-19T21:11:07.641 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 5 2024-02-19T21:11:07.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.641 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796288+0000", 2024-02-19T21:11:07.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.641 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.641 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796288+0000", 2024-02-19T21:11:07.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:07.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796286+0000", 2024-02-19T21:11:07.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9190000000000005e-06 2024-02-19T21:11:07.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796295+0000", 2024-02-19T21:11:07.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0984e-05 2024-02-19T21:11:07.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796306+0000", 2024-02-19T21:11:07.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9120000000000001e-06 2024-02-19T21:11:07.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796313+0000", 2024-02-19T21:11:07.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2296999999999999e-05 2024-02-19T21:11:07.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796345+0000", 2024-02-19T21:11:07.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9303999999999998e-05 2024-02-19T21:11:07.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796424+0000", 2024-02-19T21:11:07.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1744999999999997e-05 2024-02-19T21:11:07.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 3,6", 2024-02-19T21:11:07.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.796516+0000", 2024-02-19T21:11:07.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00058284700000000005 2024-02-19T21:11:07.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:11:07.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.797099+0000", 2024-02-19T21:11:07.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000270282 2024-02-19T21:11:07.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:07.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.797369+0000", 2024-02-19T21:11:07.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022161777000000001 2024-02-19T21:11:07.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:07.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.819531+0000", 2024-02-19T21:11:07.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.621e-05 2024-02-19T21:11:07.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.819577+0000", 2024-02-19T21:11:07.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3671000000000003e-05 2024-02-19T21:11:07.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:43.819621+0000", 2024-02-19T21:11:07.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023332470000000001 2024-02-19T21:11:07.646 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.646 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.646 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.646 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24865.0:14 11.18 11:1d8f7fe2:test-rados-api-smithi163-83952-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:11:07.646 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:47.966832+0000", 2024-02-19T21:11:07.646 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 379.55993880099999, 2024-02-19T21:11:07.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040096484000000002, 2024-02-19T21:11:07.646 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.647 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.647 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:07.647 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24865", 2024-02-19T21:11:07.647 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3187889786", 2024-02-19T21:11:07.647 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 14 2024-02-19T21:11:07.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.647 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.966832+0000", 2024-02-19T21:11:07.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.647 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.966832+0000", 2024-02-19T21:11:07.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:07.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.966830+0000", 2024-02-19T21:11:07.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7490000000000004e-06 2024-02-19T21:11:07.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.966839+0000", 2024-02-19T21:11:07.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0332e-05 2024-02-19T21:11:07.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.966849+0000", 2024-02-19T21:11:07.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1670000000000002e-06 2024-02-19T21:11:07.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.966857+0000", 2024-02-19T21:11:07.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0093742600000000006 2024-02-19T21:11:07.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.976231+0000", 2024-02-19T21:11:07.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013006500000000001 2024-02-19T21:11:07.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.976361+0000", 2024-02-19T21:11:07.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000153556 2024-02-19T21:11:07.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,6", 2024-02-19T21:11:07.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.976514+0000", 2024-02-19T21:11:07.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00053995200000000001 2024-02-19T21:11:07.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:11:07.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.977054+0000", 2024-02-19T21:11:07.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0085838589999999992 2024-02-19T21:11:07.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:07.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.985638+0000", 2024-02-19T21:11:07.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.021197450999999999 2024-02-19T21:11:07.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:07.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.006836+0000", 2024-02-19T21:11:07.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0127000000000003e-05 2024-02-19T21:11:07.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.006886+0000", 2024-02-19T21:11:07.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3053999999999998e-05 2024-02-19T21:11:07.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.652 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.006929+0000", 2024-02-19T21:11:07.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040096484000000002 2024-02-19T21:11:07.652 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.652 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.652 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.652 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.652 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.652 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25351.0:2 56.15 56:ad551702:::test:head [create,write 0~4,writefull 0~2] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-19T21:11:07.652 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:50.578524+0000", 2024-02-19T21:11:07.652 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 376.94824722300001, 2024-02-19T21:11:07.652 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040752544000000002, 2024-02-19T21:11:07.652 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.652 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.653 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:07.653 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25351", 2024-02-19T21:11:07.653 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/581384461", 2024-02-19T21:11:07.653 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:11:07.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.653 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.578524+0000", 2024-02-19T21:11:07.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.578524+0000", 2024-02-19T21:11:07.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:07.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.578522+0000", 2024-02-19T21:11:07.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1309999999999995e-06 2024-02-19T21:11:07.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.578531+0000", 2024-02-19T21:11:07.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0318e-05 2024-02-19T21:11:07.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.578541+0000", 2024-02-19T21:11:07.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1940000000000001e-06 2024-02-19T21:11:07.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.578548+0000", 2024-02-19T21:11:07.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001524381 2024-02-19T21:11:07.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.655 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.580073+0000", 2024-02-19T21:11:07.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6190000000000002e-06 2024-02-19T21:11:07.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.655 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:11:07.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.580080+0000", 2024-02-19T21:11:07.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020204755000000001 2024-02-19T21:11:07.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.600285+0000", 2024-02-19T21:11:07.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000110561 2024-02-19T21:11:07.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.600396+0000", 2024-02-19T21:11:07.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000158414 2024-02-19T21:11:07.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 4,7", 2024-02-19T21:11:07.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.600554+0000", 2024-02-19T21:11:07.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0043549599999999997 2024-02-19T21:11:07.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:11:07.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.604909+0000", 2024-02-19T21:11:07.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014248733 2024-02-19T21:11:07.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:07.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.619158+0000", 2024-02-19T21:11:07.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7781999999999999e-05 2024-02-19T21:11:07.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:07.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.619186+0000", 2024-02-19T21:11:07.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7846999999999998e-05 2024-02-19T21:11:07.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.619234+0000", 2024-02-19T21:11:07.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3025999999999998e-05 2024-02-19T21:11:07.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.619277+0000", 2024-02-19T21:11:07.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040752544000000002 2024-02-19T21:11:07.659 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.659 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.659 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.659 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.32629.0:1 76.2s0 76:46da943f:test-rados-api-smithi163-83944-10::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e126)", 2024-02-19T21:11:07.659 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:06.064459+0000", 2024-02-19T21:11:07.659 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 361.46231266799998, 2024-02-19T21:11:07.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.077442319, 2024-02-19T21:11:07.659 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.659 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.659 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:07.659 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.32629", 2024-02-19T21:11:07.659 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/1643190421", 2024-02-19T21:11:07.660 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:11:07.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.660 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:06.064459+0000", 2024-02-19T21:11:07.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:06.064459+0000", 2024-02-19T21:11:07.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:07.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:06.064456+0000", 2024-02-19T21:11:07.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1240000000000005e-06 2024-02-19T21:11:07.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:06.064464+0000", 2024-02-19T21:11:07.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0166999999999999e-05 2024-02-19T21:11:07.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.661 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.661 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:06.064474+0000", 2024-02-19T21:11:07.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4464000000000001e-05 2024-02-19T21:11:07.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:06.064489+0000", 2024-02-19T21:11:07.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.022010106000000002 2024-02-19T21:11:07.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:06.086499+0000", 2024-02-19T21:11:07.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8210000000000003e-06 2024-02-19T21:11:07.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:11:07.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:06.086503+0000", 2024-02-19T21:11:07.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.049210397 2024-02-19T21:11:07.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.135713+0000", 2024-02-19T21:11:07.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000119616 2024-02-19T21:11:07.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.135833+0000", 2024-02-19T21:11:07.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039238199999999998 2024-02-19T21:11:07.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:11:07.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.136225+0000", 2024-02-19T21:11:07.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00051139800000000004 2024-02-19T21:11:07.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:11:07.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.136736+0000", 2024-02-19T21:11:07.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050892560000000003 2024-02-19T21:11:07.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.141826+0000", 2024-02-19T21:11:07.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5197000000000005e-05 2024-02-19T21:11:07.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:07.141901+0000", 2024-02-19T21:11:07.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.077442319 2024-02-19T21:11:07.665 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.665 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.665 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.665 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.32629.0:4 76.2s0 76:5382e900:test-rados-api-smithi163-83944-11::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e129)", 2024-02-19T21:11:07.665 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:09.008457+0000", 2024-02-19T21:11:07.665 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 358.518314003, 2024-02-19T21:11:07.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.075471435000000003, 2024-02-19T21:11:07.665 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.665 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.665 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:07.665 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.32629", 2024-02-19T21:11:07.665 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/1643190421", 2024-02-19T21:11:07.666 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 4 2024-02-19T21:11:07.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.666 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.008457+0000", 2024-02-19T21:11:07.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.008457+0000", 2024-02-19T21:11:07.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999962 2024-02-19T21:11:07.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.008454+0000", 2024-02-19T21:11:07.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3213999999999999e-05 2024-02-19T21:11:07.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.667 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.008467+0000", 2024-02-19T21:11:07.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5468000000000002e-05 2024-02-19T21:11:07.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.667 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.008482+0000", 2024-02-19T21:11:07.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4210000000000001e-06 2024-02-19T21:11:07.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.667 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.008492+0000", 2024-02-19T21:11:07.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2969000000000001e-05 2024-02-19T21:11:07.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.668 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.008545+0000", 2024-02-19T21:11:07.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7880000000000007e-06 2024-02-19T21:11:07.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.668 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:11:07.668 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.008552+0000", 2024-02-19T21:11:07.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.062900254000000003 2024-02-19T21:11:07.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.071453+0000", 2024-02-19T21:11:07.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013771700000000001 2024-02-19T21:11:07.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.071590+0000", 2024-02-19T21:11:07.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00045800099999999998 2024-02-19T21:11:07.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:11:07.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.072048+0000", 2024-02-19T21:11:07.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084048679999999994 2024-02-19T21:11:07.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:11:07.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.080453+0000", 2024-02-19T21:11:07.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033983360000000001 2024-02-19T21:11:07.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.083852+0000", 2024-02-19T21:11:07.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6990000000000007e-05 2024-02-19T21:11:07.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.083929+0000", 2024-02-19T21:11:07.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.075471435000000003 2024-02-19T21:11:07.670 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.671 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.671 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.671 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.671 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.671 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.37339.0:2 81.7 81:e732be06:::oid_WaitForSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e130)", 2024-02-19T21:11:07.671 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:10.050036+0000", 2024-02-19T21:11:07.671 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 357.47673541500001, 2024-02-19T21:11:07.671 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.096519626999999997, 2024-02-19T21:11:07.671 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.671 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.671 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:07.671 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.37339", 2024-02-19T21:11:07.671 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2227392415", 2024-02-19T21:11:07.671 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:11:07.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.672 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.050036+0000", 2024-02-19T21:11:07.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.050036+0000", 2024-02-19T21:11:07.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:07.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.050034+0000", 2024-02-19T21:11:07.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1225e-05 2024-02-19T21:11:07.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.050045+0000", 2024-02-19T21:11:07.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0119999999999994e-06 2024-02-19T21:11:07.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.050054+0000", 2024-02-19T21:11:07.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2949999999999996e-06 2024-02-19T21:11:07.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.050061+0000", 2024-02-19T21:11:07.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0050346799999999997 2024-02-19T21:11:07.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.055096+0000", 2024-02-19T21:11:07.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0490000000000004e-06 2024-02-19T21:11:07.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:11:07.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.055104+0000", 2024-02-19T21:11:07.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039004045000000001 2024-02-19T21:11:07.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.094108+0000", 2024-02-19T21:11:07.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011377100000000001 2024-02-19T21:11:07.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.094222+0000", 2024-02-19T21:11:07.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012717199999999999 2024-02-19T21:11:07.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,5", 2024-02-19T21:11:07.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.094349+0000", 2024-02-19T21:11:07.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00093972200000000004 2024-02-19T21:11:07.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:07.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.095289+0000", 2024-02-19T21:11:07.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044955054000000001 2024-02-19T21:11:07.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.676 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:11:07.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.140244+0000", 2024-02-19T21:11:07.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0062051199999999997 2024-02-19T21:11:07.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.676 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:07.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.146449+0000", 2024-02-19T21:11:07.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1298999999999997e-05 2024-02-19T21:11:07.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.146510+0000", 2024-02-19T21:11:07.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5163000000000001e-05 2024-02-19T21:11:07.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:10.146555+0000", 2024-02-19T21:11:07.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.096519626999999997 2024-02-19T21:11:07.677 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.677 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.677 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.677 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.38953.0:6 82.c e132/131 82:3197012c:::foo_rmxattr:head v 132'1, mlcod=0'0)", 2024-02-19T21:11:07.678 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:12.231914+0000", 2024-02-19T21:11:07.678 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 355.29485727899998, 2024-02-19T21:11:07.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042112255000000001, 2024-02-19T21:11:07.678 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.678 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.678 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.231914+0000", 2024-02-19T21:11:07.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.231914+0000", 2024-02-19T21:11:07.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:07.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.231911+0000", 2024-02-19T21:11:07.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4381e-05 2024-02-19T21:11:07.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.231926+0000", 2024-02-19T21:11:07.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4878000000000001e-05 2024-02-19T21:11:07.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.231941+0000", 2024-02-19T21:11:07.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4670000000000004e-06 2024-02-19T21:11:07.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.231949+0000", 2024-02-19T21:11:07.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001357816 2024-02-19T21:11:07.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.233307+0000", 2024-02-19T21:11:07.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1797000000000002e-05 2024-02-19T21:11:07.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.233329+0000", 2024-02-19T21:11:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.040667275000000003 2024-02-19T21:11:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.273996+0000", 2024-02-19T21:11:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0195000000000001e-05 2024-02-19T21:11:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:12.274026+0000", 2024-02-19T21:11:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.042112255000000001 2024-02-19T21:11:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.681 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.25753.0:248 61.1e e134/114 61:780b1894:test-rados-api-smithi163-83928-12::bar:head v 134'2, mlcod=0'0)", 2024-02-19T21:11:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:14.247758+0000", 2024-02-19T21:11:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 353.27901343799999, 2024-02-19T21:11:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024500101, 2024-02-19T21:11:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.247758+0000", 2024-02-19T21:11:07.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.247758+0000", 2024-02-19T21:11:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:11:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.247757+0000", 2024-02-19T21:11:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2125999999999997e-05 2024-02-19T21:11:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.247789+0000", 2024-02-19T21:11:07.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1099e-05 2024-02-19T21:11:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.247800+0000", 2024-02-19T21:11:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5089999999999996e-06 2024-02-19T21:11:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.247808+0000", 2024-02-19T21:11:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015558193 2024-02-19T21:11:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.263366+0000", 2024-02-19T21:11:07.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0985e-05 2024-02-19T21:11:07.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.263387+0000", 2024-02-19T21:11:07.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0088353919999999992 2024-02-19T21:11:07.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.272222+0000", 2024-02-19T21:11:07.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5672e-05 2024-02-19T21:11:07.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.272258+0000", 2024-02-19T21:11:07.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.024500101 2024-02-19T21:11:07.686 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.686 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.686 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.686 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.25753.0:253 61.1e e135/114 61:780b1894:test-rados-api-smithi163-83928-12::bar:head v 135'4, mlcod=126'1)", 2024-02-19T21:11:07.686 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:15.196772+0000", 2024-02-19T21:11:07.686 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 352.32999884399999, 2024-02-19T21:11:07.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.086393664999999994, 2024-02-19T21:11:07.686 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.686 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.686 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.196772+0000", 2024-02-19T21:11:07.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.196772+0000", 2024-02-19T21:11:07.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:07.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.196770+0000", 2024-02-19T21:11:07.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.483e-05 2024-02-19T21:11:07.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.688 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.196785+0000", 2024-02-19T21:11:07.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0017793780000000001 2024-02-19T21:11:07.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.688 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.198565+0000", 2024-02-19T21:11:07.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5960000000000004e-06 2024-02-19T21:11:07.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.688 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.198573+0000", 2024-02-19T21:11:07.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026450318 2024-02-19T21:11:07.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.225024+0000", 2024-02-19T21:11:07.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0607000000000001e-05 2024-02-19T21:11:07.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.225044+0000", 2024-02-19T21:11:07.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.058074551000000002 2024-02-19T21:11:07.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.283119+0000", 2024-02-19T21:11:07.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7290000000000003e-05 2024-02-19T21:11:07.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.283166+0000", 2024-02-19T21:11:07.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.086393664999999994 2024-02-19T21:11:07.690 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.690 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.690 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.690 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.41221.0:2 87.6 87:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected e135)", 2024-02-19T21:11:07.690 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:15.207430+0000", 2024-02-19T21:11:07.690 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 352.31934086500002, 2024-02-19T21:11:07.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.093118128999999994, 2024-02-19T21:11:07.690 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.691 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.691 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:07.691 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.41221", 2024-02-19T21:11:07.691 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2273804006", 2024-02-19T21:11:07.691 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:11:07.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.691 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.207430+0000", 2024-02-19T21:11:07.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.207430+0000", 2024-02-19T21:11:07.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:07.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.207428+0000", 2024-02-19T21:11:07.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8279999999999992e-06 2024-02-19T21:11:07.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.207437+0000", 2024-02-19T21:11:07.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0592e-05 2024-02-19T21:11:07.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.207447+0000", 2024-02-19T21:11:07.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7330000000000003e-06 2024-02-19T21:11:07.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.207455+0000", 2024-02-19T21:11:07.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.018026001999999999 2024-02-19T21:11:07.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.225481+0000", 2024-02-19T21:11:07.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4030000000000001e-06 2024-02-19T21:11:07.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:11:07.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.225488+0000", 2024-02-19T21:11:07.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.058216288999999997 2024-02-19T21:11:07.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.283705+0000", 2024-02-19T21:11:07.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000119333 2024-02-19T21:11:07.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.283824+0000", 2024-02-19T21:11:07.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014911899999999999 2024-02-19T21:11:07.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,7", 2024-02-19T21:11:07.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.283973+0000", 2024-02-19T21:11:07.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041614590000000002 2024-02-19T21:11:07.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:11:07.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.288135+0000", 2024-02-19T21:11:07.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.008069105 2024-02-19T21:11:07.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:07.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.296204+0000", 2024-02-19T21:11:07.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0041469330000000002 2024-02-19T21:11:07.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:07.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.300351+0000", 2024-02-19T21:11:07.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5433000000000003e-05 2024-02-19T21:11:07.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.300406+0000", 2024-02-19T21:11:07.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014231199999999999 2024-02-19T21:11:07.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:15.300548+0000", 2024-02-19T21:11:07.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.093118128999999994 2024-02-19T21:11:07.697 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.697 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.697 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.697 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:256 85.16 85:68726f76:test-rados-api-smithi163-83928-12::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e138)", 2024-02-19T21:11:07.697 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:18.406353+0000", 2024-02-19T21:11:07.697 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 349.12041822100002, 2024-02-19T21:11:07.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.16278585500000001, 2024-02-19T21:11:07.697 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.697 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:11:07.697 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:07.697 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:11:07.697 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:11:07.698 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 256 2024-02-19T21:11:07.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.698 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.406353+0000", 2024-02-19T21:11:07.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.406353+0000", 2024-02-19T21:11:07.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:07.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.406351+0000", 2024-02-19T21:11:07.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2789999999999998e-06 2024-02-19T21:11:07.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.406359+0000", 2024-02-19T21:11:07.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0567999999999999e-05 2024-02-19T21:11:07.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.406370+0000", 2024-02-19T21:11:07.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6610000000000006e-06 2024-02-19T21:11:07.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.406378+0000", 2024-02-19T21:11:07.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010087200000000001 2024-02-19T21:11:07.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.406479+0000", 2024-02-19T21:11:07.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.117e-06 2024-02-19T21:11:07.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:11:07.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.406485+0000", 2024-02-19T21:11:07.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014150813 2024-02-19T21:11:07.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.420636+0000", 2024-02-19T21:11:07.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000229533 2024-02-19T21:11:07.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", 2024-02-19T21:11:07.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.420866+0000", 2024-02-19T21:11:07.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.095515298999999998 2024-02-19T21:11:07.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.516381+0000", 2024-02-19T21:11:07.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9480000000000004e-05 2024-02-19T21:11:07.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-02-19T21:11:07.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.516471+0000", 2024-02-19T21:11:07.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052278073000000001 2024-02-19T21:11:07.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.568749+0000", 2024-02-19T21:11:07.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5988000000000003e-05 2024-02-19T21:11:07.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.568845+0000", 2024-02-19T21:11:07.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00029426300000000001 2024-02-19T21:11:07.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.569139+0000", 2024-02-19T21:11:07.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.16278585500000001 2024-02-19T21:11:07.703 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.703 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.703 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.703 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:258 85.16 85:68726f76:test-rados-api-smithi163-83928-12::foo:2 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e138)", 2024-02-19T21:11:07.703 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:18.663361+0000", 2024-02-19T21:11:07.703 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 348.86341025500002, 2024-02-19T21:11:07.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.032544218999999999, 2024-02-19T21:11:07.704 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.704 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:11:07.704 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:07.704 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:11:07.704 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:11:07.704 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 258 2024-02-19T21:11:07.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.704 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.663361+0000", 2024-02-19T21:11:07.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.663361+0000", 2024-02-19T21:11:07.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:11:07.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.663358+0000", 2024-02-19T21:11:07.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0773e-05 2024-02-19T21:11:07.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.663369+0000", 2024-02-19T21:11:07.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3528e-05 2024-02-19T21:11:07.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.663382+0000", 2024-02-19T21:11:07.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3540000000000006e-06 2024-02-19T21:11:07.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.663392+0000", 2024-02-19T21:11:07.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039003300000000002 2024-02-19T21:11:07.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.663782+0000", 2024-02-19T21:11:07.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000232334 2024-02-19T21:11:07.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", 2024-02-19T21:11:07.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.664014+0000", 2024-02-19T21:11:07.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0081638519999999992 2024-02-19T21:11:07.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.672178+0000", 2024-02-19T21:11:07.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010873 2024-02-19T21:11:07.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-02-19T21:11:07.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.672287+0000", 2024-02-19T21:11:07.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.020611722999999998 2024-02-19T21:11:07.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.692898+0000", 2024-02-19T21:11:07.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120211 2024-02-19T21:11:07.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.693019+0000", 2024-02-19T21:11:07.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028865430000000001 2024-02-19T21:11:07.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.695905+0000", 2024-02-19T21:11:07.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.032544218999999999 2024-02-19T21:11:07.709 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.709 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.709 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.709 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.45484.0:2 92.7s0 92:ed965bc3:test-rados-api-smithi163-83944-15::foo:head [write 0~8192] snapc 2=[2] ondisk+write+known_if_redirected e142)", 2024-02-19T21:11:07.709 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:22.184478+0000", 2024-02-19T21:11:07.709 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 345.34229340500002, 2024-02-19T21:11:07.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046650480000000001, 2024-02-19T21:11:07.709 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.709 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.709 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:07.710 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.45484", 2024-02-19T21:11:07.710 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/4096098867", 2024-02-19T21:11:07.710 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:11:07.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.710 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.710 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184478+0000", 2024-02-19T21:11:07.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.710 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184478+0000", 2024-02-19T21:11:07.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:07.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184476+0000", 2024-02-19T21:11:07.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6813999999999999e-05 2024-02-19T21:11:07.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184492+0000", 2024-02-19T21:11:07.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1202e-05 2024-02-19T21:11:07.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184504+0000", 2024-02-19T21:11:07.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8180000000000004e-06 2024-02-19T21:11:07.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184511+0000", 2024-02-19T21:11:07.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7104000000000002e-05 2024-02-19T21:11:07.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184558+0000", 2024-02-19T21:11:07.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3579999999999999e-06 2024-02-19T21:11:07.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:11:07.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184562+0000", 2024-02-19T21:11:07.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.033812456999999997 2024-02-19T21:11:07.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.218374+0000", 2024-02-19T21:11:07.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012377300000000001 2024-02-19T21:11:07.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.218498+0000", 2024-02-19T21:11:07.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00042174699999999999 2024-02-19T21:11:07.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:11:07.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.218920+0000", 2024-02-19T21:11:07.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0096631270000000005 2024-02-19T21:11:07.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:11:07.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.228583+0000", 2024-02-19T21:11:07.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024784189999999999 2024-02-19T21:11:07.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.231061+0000", 2024-02-19T21:11:07.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6945000000000004e-05 2024-02-19T21:11:07.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.715 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.231128+0000", 2024-02-19T21:11:07.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046650480000000001 2024-02-19T21:11:07.715 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.715 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.715 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.715 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.47779.0:2 114.13 e169/168 114:cbb9903e:::test_obj:head v 169'1, mlcod=0'0)", 2024-02-19T21:11:07.715 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:49.216541+0000", 2024-02-19T21:11:07.715 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 318.31023000099998, 2024-02-19T21:11:07.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.033635538999999999, 2024-02-19T21:11:07.715 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.716 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.716 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.216541+0000", 2024-02-19T21:11:07.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.216541+0000", 2024-02-19T21:11:07.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:11:07.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.216541+0000", 2024-02-19T21:11:07.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8630000000000002e-06 2024-02-19T21:11:07.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.216548+0000", 2024-02-19T21:11:07.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5470000000000001e-06 2024-02-19T21:11:07.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.216551+0000", 2024-02-19T21:11:07.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5249999999999999e-06 2024-02-19T21:11:07.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.216556+0000", 2024-02-19T21:11:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027499585 2024-02-19T21:11:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.244055+0000", 2024-02-19T21:11:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3051999999999999e-05 2024-02-19T21:11:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.244079+0000", 2024-02-19T21:11:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060570659999999998 2024-02-19T21:11:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.250136+0000", 2024-02-19T21:11:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1170000000000001e-05 2024-02-19T21:11:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.250177+0000", 2024-02-19T21:11:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.033635538999999999 2024-02-19T21:11:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:1429 61.12 61:48d9abf6:test-rados-api-smithi163-83928-16::foo:head [writefull 0~5] snapc c=[c] ondisk+write+known_if_redirected e191)", 2024-02-19T21:11:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:06:11.199452+0000", 2024-02-19T21:11:07.719 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 296.327319574, 2024-02-19T21:11:07.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052298378, 2024-02-19T21:11:07.720 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.720 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.720 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:07.720 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:11:07.720 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:11:07.720 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1429 2024-02-19T21:11:07.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.720 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.720 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.720 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.199452+0000", 2024-02-19T21:11:07.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.199452+0000", 2024-02-19T21:11:07.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999957 2024-02-19T21:11:07.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.199447+0000", 2024-02-19T21:11:07.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6351999999999999e-05 2024-02-19T21:11:07.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.199464+0000", 2024-02-19T21:11:07.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5387999999999999e-05 2024-02-19T21:11:07.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.199479+0000", 2024-02-19T21:11:07.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3209999999999994e-06 2024-02-19T21:11:07.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.199489+0000", 2024-02-19T21:11:07.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.01039052 2024-02-19T21:11:07.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.209879+0000", 2024-02-19T21:11:07.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010921699999999999 2024-02-19T21:11:07.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.209988+0000", 2024-02-19T21:11:07.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00021893100000000001 2024-02-19T21:11:07.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3", 2024-02-19T21:11:07.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.210207+0000", 2024-02-19T21:11:07.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028331134000000001 2024-02-19T21:11:07.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:11:07.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.238538+0000", 2024-02-19T21:11:07.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00095535699999999999 2024-02-19T21:11:07.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:07.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.239494+0000", 2024-02-19T21:11:07.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012141102000000001 2024-02-19T21:11:07.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:07.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.251635+0000", 2024-02-19T21:11:07.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0659000000000003e-05 2024-02-19T21:11:07.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.251695+0000", 2024-02-19T21:11:07.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4573000000000001e-05 2024-02-19T21:11:07.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.251750+0000", 2024-02-19T21:11:07.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052298378 2024-02-19T21:11:07.725 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.725 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.725 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.725 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.22407.0:26 126.6s0 126:7ff15785:test-rados-api-smithi163-83952-23::foo:head [write 0~8192] snapc a=[a] ondisk+write+known_if_redirected e191)", 2024-02-19T21:11:07.725 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:06:11.207404+0000", 2024-02-19T21:11:07.725 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 296.319367344, 2024-02-19T21:11:07.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.033762901999999997, 2024-02-19T21:11:07.726 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.726 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.726 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:07.726 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.22407", 2024-02-19T21:11:07.726 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/4166546335", 2024-02-19T21:11:07.726 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 26 2024-02-19T21:11:07.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.726 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.726 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.726 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.207404+0000", 2024-02-19T21:11:07.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.726 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.207404+0000", 2024-02-19T21:11:07.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:07.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.207401+0000", 2024-02-19T21:11:07.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0035999999999999e-05 2024-02-19T21:11:07.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.207421+0000", 2024-02-19T21:11:07.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014837427 2024-02-19T21:11:07.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.222259+0000", 2024-02-19T21:11:07.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5291999999999999e-05 2024-02-19T21:11:07.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.222274+0000", 2024-02-19T21:11:07.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00045052100000000001 2024-02-19T21:11:07.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.222725+0000", 2024-02-19T21:11:07.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014116500000000001 2024-02-19T21:11:07.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.222866+0000", 2024-02-19T21:11:07.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00053662000000000002 2024-02-19T21:11:07.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:11:07.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.223402+0000", 2024-02-19T21:11:07.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015991278000000001 2024-02-19T21:11:07.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:11:07.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.239394+0000", 2024-02-19T21:11:07.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016954959999999999 2024-02-19T21:11:07.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.241089+0000", 2024-02-19T21:11:07.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7653999999999999e-05 2024-02-19T21:11:07.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.241167+0000", 2024-02-19T21:11:07.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.033762901999999997 2024-02-19T21:11:07.730 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.730 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.730 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.731 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.49400.0:1 175.6s0 175:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e265)", 2024-02-19T21:11:07.731 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:07:25.213923+0000", 2024-02-19T21:11:07.731 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 222.31284862300001, 2024-02-19T21:11:07.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0203528529999999, 2024-02-19T21:11:07.731 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.731 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.731 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:07.731 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.49400", 2024-02-19T21:11:07.731 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/896457325", 2024-02-19T21:11:07.731 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:11:07.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.731 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213923+0000", 2024-02-19T21:11:07.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213923+0000", 2024-02-19T21:11:07.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:07.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213921+0000", 2024-02-19T21:11:07.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5069999999999997e-06 2024-02-19T21:11:07.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213927+0000", 2024-02-19T21:11:07.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2819999999999996e-06 2024-02-19T21:11:07.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213936+0000", 2024-02-19T21:11:07.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2279999999999998e-06 2024-02-19T21:11:07.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213941+0000", 2024-02-19T21:11:07.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3912999999999997e-05 2024-02-19T21:11:07.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213995+0000", 2024-02-19T21:11:07.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4039999999999999e-06 2024-02-19T21:11:07.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:11:07.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213998+0000", 2024-02-19T21:11:07.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0148429459999999 2024-02-19T21:11:07.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:26.228841+0000", 2024-02-19T21:11:07.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112502 2024-02-19T21:11:07.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:26.228954+0000", 2024-02-19T21:11:07.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00035581099999999999 2024-02-19T21:11:07.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:11:07.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:26.229309+0000", 2024-02-19T21:11:07.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00088503599999999996 2024-02-19T21:11:07.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:11:07.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:26.230194+0000", 2024-02-19T21:11:07.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039628809999999997 2024-02-19T21:11:07.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.736 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.736 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:26.234157+0000", 2024-02-19T21:11:07.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118199 2024-02-19T21:11:07.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.736 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.736 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:26.234275+0000", 2024-02-19T21:11:07.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0203528529999999 2024-02-19T21:11:07.736 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.736 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.736 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.737 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:3227 186.3 186:c0520479:test-rados-api-smithi163-83928-38::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected e294)", 2024-02-19T21:11:07.737 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:07:54.246514+0000", 2024-02-19T21:11:07.737 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 193.28025690999999, 2024-02-19T21:11:07.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026188934000000001, 2024-02-19T21:11:07.737 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.737 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:11:07.737 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:07.737 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:11:07.737 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:11:07.737 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3227 2024-02-19T21:11:07.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.737 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.246514+0000", 2024-02-19T21:11:07.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.246514+0000", 2024-02-19T21:11:07.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:11:07.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.246513+0000", 2024-02-19T21:11:07.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9750000000000001e-06 2024-02-19T21:11:07.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.246517+0000", 2024-02-19T21:11:07.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7299999999999999e-06 2024-02-19T21:11:07.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.246524+0000", 2024-02-19T21:11:07.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4610000000000001e-06 2024-02-19T21:11:07.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.246529+0000", 2024-02-19T21:11:07.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010273137999999999 2024-02-19T21:11:07.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.256802+0000", 2024-02-19T21:11:07.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7600000000000003e-05 2024-02-19T21:11:07.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.256869+0000", 2024-02-19T21:11:07.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015678469 2024-02-19T21:11:07.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.272548+0000", 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9383999999999997e-05 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.272637+0000", 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6144999999999998e-05 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.272703+0000", 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026188934000000001 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.742 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.3.25:34 216.16 e452/444 216:6dfc172a:test-rados-api-smithi163-83928-70::aab9b7bd17003da21256cca5c7d2d7ff4ea384b6:head v 452'3, mlcod=446'1)", 2024-02-19T21:11:07.742 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:11:04.382842+0000", 2024-02-19T21:11:07.742 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.1439290849999999, 2024-02-19T21:11:07.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025448117999999999, 2024-02-19T21:11:07.742 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.742 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.742 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382842+0000", 2024-02-19T21:11:07.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382842+0000", 2024-02-19T21:11:07.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:11:07.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382839+0000", 2024-02-19T21:11:07.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4117000000000001e-05 2024-02-19T21:11:07.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382854+0000", 2024-02-19T21:11:07.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1486000000000001e-05 2024-02-19T21:11:07.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382875+0000", 2024-02-19T21:11:07.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6940000000000007e-06 2024-02-19T21:11:07.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382885+0000", 2024-02-19T21:11:07.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4758000000000005e-05 2024-02-19T21:11:07.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382959+0000", 2024-02-19T21:11:07.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7180000000000001e-05 2024-02-19T21:11:07.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382987+0000", 2024-02-19T21:11:07.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025254843999999999 2024-02-19T21:11:07.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408242+0000", 2024-02-19T21:11:07.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8742999999999997e-05 2024-02-19T21:11:07.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408290+0000", 2024-02-19T21:11:07.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025448117999999999 2024-02-19T21:11:07.745 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.745 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.746 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.746 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.3.25:35 216.0 e452/444 216:06397337:test-rados-api-smithi163-83928-70::b1a341c59dc4c60bbe00244ac8bba87528936bc7:head v 452'2, mlcod=0'0)", 2024-02-19T21:11:07.746 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:11:04.408969+0000", 2024-02-19T21:11:07.746 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.1178017389999999, 2024-02-19T21:11:07.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023900965999999999, 2024-02-19T21:11:07.746 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:07.746 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:07.746 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:07.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:07.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408969+0000", 2024-02-19T21:11:07.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:07.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:07.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408969+0000", 2024-02-19T21:11:07.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:07.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:07.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408968+0000", 2024-02-19T21:11:07.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0178e-05 2024-02-19T21:11:07.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:07.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408978+0000", 2024-02-19T21:11:07.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3319000000000001e-05 2024-02-19T21:11:07.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:07.748 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408991+0000", 2024-02-19T21:11:07.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1670000000000002e-06 2024-02-19T21:11:07.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:07.748 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408998+0000", 2024-02-19T21:11:07.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0093999999999998e-05 2024-02-19T21:11:07.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:07.749 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.409048+0000", 2024-02-19T21:11:07.749 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3366999999999998e-05 2024-02-19T21:11:07.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.749 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:07.749 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.409072+0000", 2024-02-19T21:11:07.749 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023722753999999999 2024-02-19T21:11:07.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.749 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:07.749 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.432795+0000", 2024-02-19T21:11:07.749 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5873999999999999e-05 2024-02-19T21:11:07.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:07.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:07.749 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:07.750 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.432870+0000", 2024-02-19T21:11:07.750 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023900965999999999 2024-02-19T21:11:07.750 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.750 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.750 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.750 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:07.750 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:07.750 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:11:07.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:06.162349+0000 osd.5 (osd.5) 192 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:11:07.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:06.162572+0000 osd.5 (osd.5) 193 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:11:07.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:06.681479+0000 mon.a (mon.0) 2619 : cluster [DBG] osdmap e453: 8 total, 8 up, 8 in 2024-02-19T21:11:07.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:06.162349+0000 osd.5 (osd.5) 192 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:11:07.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:06.162572+0000 osd.5 (osd.5) 193 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:11:07.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:06.681479+0000 mon.a (mon.0) 2619 : cluster [DBG] osdmap e453: 8 total, 8 up, 8 in 2024-02-19T21:11:07.853 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:11:07.855 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:07.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:11:07.956 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:11:07.984 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:08.069 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:11:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:11:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:11:08.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:06.162349+0000 osd.5 (osd.5) 192 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:11:08.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:06.162572+0000 osd.5 (osd.5) 193 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:11:08.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:06.681479+0000 mon.a (mon.0) 2619 : cluster [DBG] osdmap e453: 8 total, 8 up, 8 in 2024-02-19T21:11:08.083 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:11:08.085 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:08.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:11:08.186 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:08.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.592+0000 7f68686c5700 1 -- 172.21.15.163:0/2882774818 >> v1:172.21.15.163:6790/0 conn(0x7f68600f6710 legacy=0x7f68600f6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:08.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.592+0000 7f68686c5700 1 -- 172.21.15.163:0/2882774818 shutdown_connections 2024-02-19T21:11:08.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.592+0000 7f68686c5700 1 -- 172.21.15.163:0/2882774818 >> 172.21.15.163:0/2882774818 conn(0x7f68600f2470 msgr2=0x7f68600f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:08.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.592+0000 7f68686c5700 1 -- 172.21.15.163:0/2882774818 shutdown_connections 2024-02-19T21:11:08.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.592+0000 7f68686c5700 1 -- 172.21.15.163:0/2882774818 wait complete. 2024-02-19T21:11:08.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.593+0000 7f68686c5700 1 librados: starting msgr at 2024-02-19T21:11:08.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.593+0000 7f68686c5700 1 librados: starting objecter 2024-02-19T21:11:08.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.593+0000 7f68686c5700 1 Processor -- start 2024-02-19T21:11:08.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.593+0000 7f68686c5700 1 -- start start 2024-02-19T21:11:08.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.593+0000 7f68686c5700 1 librados: setting wanted keys 2024-02-19T21:11:08.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.593+0000 7f68686c5700 1 librados: calling monclient init 2024-02-19T21:11:08.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.593+0000 7f68686c5700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6860119530 con 0x7f68600f6710 2024-02-19T21:11:08.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.593+0000 7f68686c5700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f686011a710 con 0x7f68601065e0 2024-02-19T21:11:08.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.593+0000 7f68686c5700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f686011b910 con 0x7f68600f7200 2024-02-19T21:11:08.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.593+0000 7f6866461700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f68600f6710 0x7f6860101860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:39504/0 (socket says 172.21.15.163:39504) 2024-02-19T21:11:08.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.593+0000 7f6866461700 1 -- 172.21.15.163:0/2162063573 learned_addr learned my addr 172.21.15.163:0/2162063573 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:08.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.593+0000 7f68577fe700 10 client.?.objecter ms_handle_connect 0x7f68600f6710 2024-02-19T21:11:08.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:08.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 10 client.?.objecter ms_handle_connect 0x7f68600f7200 2024-02-19T21:11:08.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:08.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 10 client.?.objecter ms_handle_connect 0x7f68601065e0 2024-02-19T21:11:08.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:08.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557725170 0 0) 0x7f6860119530 con 0x7f68600f6710 2024-02-19T21:11:08.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f684c003030 con 0x7f68600f6710 2024-02-19T21:11:08.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141221789 0 0) 0x7f686011b910 con 0x7f68600f7200 2024-02-19T21:11:08.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6860119530 con 0x7f68600f7200 2024-02-19T21:11:08.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037305226 0 0) 0x7f686011a710 con 0x7f68601065e0 2024-02-19T21:11:08.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f686011b910 con 0x7f68601065e0 2024-02-19T21:11:08.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1593160424 0 0) 0x7f684c003030 con 0x7f68600f6710 2024-02-19T21:11:08.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f686011a710 con 0x7f68600f6710 2024-02-19T21:11:08.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f685c002070 con 0x7f68600f6710 2024-02-19T21:11:08.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3221891956 0 0) 0x7f6860119530 con 0x7f68600f7200 2024-02-19T21:11:08.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f684c003030 con 0x7f68600f7200 2024-02-19T21:11:08.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f68500017c0 con 0x7f68600f7200 2024-02-19T21:11:08.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4122752537 0 0) 0x7f686011b910 con 0x7f68601065e0 2024-02-19T21:11:08.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6860119530 con 0x7f68601065e0 2024-02-19T21:11:08.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f68580018d0 con 0x7f68601065e0 2024-02-19T21:11:08.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1311319208 0 0) 0x7f686011a710 con 0x7f68600f6710 2024-02-19T21:11:08.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 >> v1:172.21.15.163:6790/0 conn(0x7f68600f7200 legacy=0x7f6860101fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:08.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 >> v1:172.21.15.193:6789/0 conn(0x7f68601065e0 legacy=0x7f6860117bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:08.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686011cb10 con 0x7f68600f6710 2024-02-19T21:11:08.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68686c5700 1 -- 172.21.15.163:0/2162063573 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f686011bb00 con 0x7f68600f6710 2024-02-19T21:11:08.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68686c5700 10 client.23202.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:08.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68686c5700 1 -- 172.21.15.163:0/2162063573 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f686011bff0 con 0x7f68600f6710 2024-02-19T21:11:08.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68686c5700 1 librados: init done 2024-02-19T21:11:08.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f685c001510 con 0x7f68600f6710 2024-02-19T21:11:08.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.594+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f685c003990 con 0x7f68600f6710 2024-02-19T21:11:08.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.595+0000 7f68686c5700 1 -- 172.21.15.163:0/2162063573 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6844004f20 con 0x7f68600f6710 2024-02-19T21:11:08.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.596+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f685c003140 con 0x7f68600f6710 2024-02-19T21:11:08.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.596+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(453..453 src has 43..453) v4 ==== 8063+0+0 (unknown 2568459083 0 0) 0x7f685c079050 con 0x7f68600f6710 2024-02-19T21:11:08.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.596+0000 7f68577fe700 10 client.23202.objecter ms_dispatch 0x7f6860001550 osd_map(453..453 src has 43..453) v4 2024-02-19T21:11:08.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.596+0000 7f68577fe700 3 client.23202.objecter handle_osd_map got epochs [453,453] > 0 2024-02-19T21:11:08.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.596+0000 7f68577fe700 3 client.23202.objecter handle_osd_map decoding full epoch 453 2024-02-19T21:11:08.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.596+0000 7f68577fe700 20 client.23202.objecter dump_active .. 0 homeless 2024-02-19T21:11:08.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.597+0000 7f68577fe700 10 client.23202.objecter ms_handle_connect 0x7f684c063d30 2024-02-19T21:11:08.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.601+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f685c04c560 con 0x7f68600f6710 2024-02-19T21:11:08.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:07.781+0000 7f68686c5700 1 -- 172.21.15.163:0/2162063573 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f68440051d0 con 0x7f68600f6710 2024-02-19T21:11:08.219 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:08.240 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:08.242 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:08.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:08.344 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:11:08.373 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:08.571 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:11:08.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:08.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:11:08.675 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:08.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:08.683+0000 7f68577fe700 1 -- 172.21.15.163:0/2162063573 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v454) v1 ==== 87+0+0 (unknown 3876041041 0 0) 0x7f685c04a9d0 con 0x7f68600f6710 2024-02-19T21:11:08.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:08.685+0000 7f68686c5700 10 librados: watch_flush enter 2024-02-19T21:11:08.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:08.686+0000 7f68686c5700 10 librados: watch_flush exit 2024-02-19T21:11:08.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:08.686+0000 7f68686c5700 20 client.23202.objecter shutdown clearing up homeless session... 2024-02-19T21:11:08.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:08.686+0000 7f68686c5700 10 client.23202.objecter successfully canceled tick 2024-02-19T21:11:08.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:08.686+0000 7f68686c5700 1 -- 172.21.15.163:0/2162063573 >> v1:172.21.15.163:6800/1552514681 conn(0x7f684c063d30 legacy=0x7f684c0661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:08.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:08.686+0000 7f68686c5700 1 -- 172.21.15.163:0/2162063573 >> v1:172.21.15.163:6789/0 conn(0x7f68600f6710 legacy=0x7f6860101860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:08.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:08.686+0000 7f68686c5700 1 -- 172.21.15.163:0/2162063573 shutdown_connections 2024-02-19T21:11:08.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:08.686+0000 7f68686c5700 1 -- 172.21.15.163:0/2162063573 >> 172.21.15.163:0/2162063573 conn(0x7f68600f2470 msgr2=0x7f6860109cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:08.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:08.686+0000 7f68686c5700 1 -- 172.21.15.163:0/2162063573 shutdown_connections 2024-02-19T21:11:08.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:08.686+0000 7f68686c5700 1 -- 172.21.15.163:0/2162063573 wait complete. 2024-02-19T21:11:08.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:08.686+0000 7f68686c5700 1 librados: shutdown 2024-02-19T21:11:08.686 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:11:08.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:06.896890+0000 osd.1 (osd.1) 221 : cluster [DBG] 61.3 scrub starts 2024-02-19T21:11:08.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:06.901562+0000 osd.1 (osd.1) 222 : cluster [DBG] 61.3 scrub ok 2024-02-19T21:11:08.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:07.228334+0000 mgr.y (mgr.14550) 411 : cluster [DBG] pgmap v668: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 511 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:11:08.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:07.241364+0000 osd.2 (osd.2) 182 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:11:08.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:07.244280+0000 osd.2 (osd.2) 183 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:11:08.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:07.454098+0000 osd.4 (osd.4) 167 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:11:08.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:07.455985+0000 osd.4 (osd.4) 168 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:11:08.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:07.598353+0000 osd.3 (osd.3) 201 : cluster [DBG] 16.7 scrub starts 2024-02-19T21:11:08.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:07.599467+0000 osd.3 (osd.3) 202 : cluster [DBG] 16.7 scrub ok 2024-02-19T21:11:08.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:07.782897+0000 mon.a (mon.0) 2620 : audit [INF] from='client.? v1:172.21.15.163:0/2162063573' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:11:08.707 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:06.896890+0000 osd.1 (osd.1) 221 : cluster [DBG] 61.3 scrub starts 2024-02-19T21:11:08.707 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:06.901562+0000 osd.1 (osd.1) 222 : cluster [DBG] 61.3 scrub ok 2024-02-19T21:11:08.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:07.228334+0000 mgr.y (mgr.14550) 411 : cluster [DBG] pgmap v668: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 511 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:11:08.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:07.241364+0000 osd.2 (osd.2) 182 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:11:08.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:07.244280+0000 osd.2 (osd.2) 183 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:11:08.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:07.454098+0000 osd.4 (osd.4) 167 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:11:08.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:07.455985+0000 osd.4 (osd.4) 168 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:11:08.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:07.598353+0000 osd.3 (osd.3) 201 : cluster [DBG] 16.7 scrub starts 2024-02-19T21:11:08.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:07.599467+0000 osd.3 (osd.3) 202 : cluster [DBG] 16.7 scrub ok 2024-02-19T21:11:08.708 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:07.782897+0000 mon.a (mon.0) 2620 : audit [INF] from='client.? v1:172.21.15.163:0/2162063573' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:11:08.709 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:09.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:06.896890+0000 osd.1 (osd.1) 221 : cluster [DBG] 61.3 scrub starts 2024-02-19T21:11:09.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:06.901562+0000 osd.1 (osd.1) 222 : cluster [DBG] 61.3 scrub ok 2024-02-19T21:11:09.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:07.228334+0000 mgr.y (mgr.14550) 411 : cluster [DBG] pgmap v668: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 511 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:11:09.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:07.241364+0000 osd.2 (osd.2) 182 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:11:09.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:07.244280+0000 osd.2 (osd.2) 183 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:11:09.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:07.454098+0000 osd.4 (osd.4) 167 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:11:09.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:07.455985+0000 osd.4 (osd.4) 168 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:11:09.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:07.598353+0000 osd.3 (osd.3) 201 : cluster [DBG] 16.7 scrub starts 2024-02-19T21:11:09.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:07.599467+0000 osd.3 (osd.3) 202 : cluster [DBG] 16.7 scrub ok 2024-02-19T21:11:09.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:07.782897+0000 mon.a (mon.0) 2620 : audit [INF] from='client.? v1:172.21.15.163:0/2162063573' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:11:09.156 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:09.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:09.260 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:09.293 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:09.442 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:09.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:09.550 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:09.577 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:09.740 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:07.943755+0000 osd.1 (osd.1) 223 : cluster [DBG] 16.13 deep-scrub starts 2024-02-19T21:11:09.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:07.947085+0000 osd.1 (osd.1) 224 : cluster [DBG] 16.13 deep-scrub ok 2024-02-19T21:11:09.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:08.039828+0000 osd.5 (osd.5) 194 : cluster [DBG] 16.c scrub starts 2024-02-19T21:11:09.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:08.043016+0000 osd.5 (osd.5) 195 : cluster [DBG] 16.c scrub ok 2024-02-19T21:11:09.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:08.200845+0000 osd.2 (osd.2) 184 : cluster [DBG] 4.f scrub starts 2024-02-19T21:11:09.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:08.202203+0000 osd.2 (osd.2) 185 : cluster [DBG] 4.f scrub ok 2024-02-19T21:11:09.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:08.243440+0000 osd.2 (osd.2) 186 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:11:09.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:08.243706+0000 osd.2 (osd.2) 187 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:11:09.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:08.680159+0000 mon.a (mon.0) 2621 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:11:09.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:08.683876+0000 mon.a (mon.0) 2622 : audit [INF] from='client.? v1:172.21.15.163:0/2162063573' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:11:09.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:08.683937+0000 mon.a (mon.0) 2623 : cluster [DBG] osdmap e454: 8 total, 8 up, 8 in 2024-02-19T21:11:09.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:07.943755+0000 osd.1 (osd.1) 223 : cluster [DBG] 16.13 deep-scrub starts 2024-02-19T21:11:09.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:07.947085+0000 osd.1 (osd.1) 224 : cluster [DBG] 16.13 deep-scrub ok 2024-02-19T21:11:09.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:08.039828+0000 osd.5 (osd.5) 194 : cluster [DBG] 16.c scrub starts 2024-02-19T21:11:09.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:08.043016+0000 osd.5 (osd.5) 195 : cluster [DBG] 16.c scrub ok 2024-02-19T21:11:09.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:08.200845+0000 osd.2 (osd.2) 184 : cluster [DBG] 4.f scrub starts 2024-02-19T21:11:09.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:08.202203+0000 osd.2 (osd.2) 185 : cluster [DBG] 4.f scrub ok 2024-02-19T21:11:09.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:08.243440+0000 osd.2 (osd.2) 186 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:11:09.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:08.243706+0000 osd.2 (osd.2) 187 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:11:09.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:08.680159+0000 mon.a (mon.0) 2621 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:11:09.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:08.683876+0000 mon.a (mon.0) 2622 : audit [INF] from='client.? v1:172.21.15.163:0/2162063573' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:11:09.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:08.683937+0000 mon.a (mon.0) 2623 : cluster [DBG] osdmap e454: 8 total, 8 up, 8 in 2024-02-19T21:11:09.758 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-19T21:11:09.775 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:09.778 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:09.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:09.780 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:09.879 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:09.905 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:10.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:07.943755+0000 osd.1 (osd.1) 223 : cluster [DBG] 16.13 deep-scrub starts 2024-02-19T21:11:10.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:07.947085+0000 osd.1 (osd.1) 224 : cluster [DBG] 16.13 deep-scrub ok 2024-02-19T21:11:10.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:08.039828+0000 osd.5 (osd.5) 194 : cluster [DBG] 16.c scrub starts 2024-02-19T21:11:10.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:08.043016+0000 osd.5 (osd.5) 195 : cluster [DBG] 16.c scrub ok 2024-02-19T21:11:10.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:08.200845+0000 osd.2 (osd.2) 184 : cluster [DBG] 4.f scrub starts 2024-02-19T21:11:10.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:08.202203+0000 osd.2 (osd.2) 185 : cluster [DBG] 4.f scrub ok 2024-02-19T21:11:10.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:08.243440+0000 osd.2 (osd.2) 186 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:11:10.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:08.243706+0000 osd.2 (osd.2) 187 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:11:10.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:08.680159+0000 mon.a (mon.0) 2621 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:11:10.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:08.683876+0000 mon.a (mon.0) 2622 : audit [INF] from='client.? v1:172.21.15.163:0/2162063573' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:11:10.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:08.683937+0000 mon.a (mon.0) 2623 : cluster [DBG] osdmap e454: 8 total, 8 up, 8 in 2024-02-19T21:11:10.160 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:10.169 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:10.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:10.271 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:10.298 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:10.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:11:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:11:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:09.228949+0000 mgr.y (mgr.14550) 412 : cluster [DBG] pgmap v670: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 511 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:11:10.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:09.228949+0000 mgr.y (mgr.14550) 412 : cluster [DBG] pgmap v670: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 511 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:11:10.913 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:09.228949+0000 mgr.y (mgr.14550) 412 : cluster [DBG] pgmap v670: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 511 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:11:11.748 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:11.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:11.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:11.852 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:11.885 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:11.907 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:11.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:11.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:11.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.972+0000 7f96012a3700 1 -- 172.21.15.163:0/1528291512 >> v1:172.21.15.163:6789/0 conn(0x7f95fc1031b0 legacy=0x7f95fc105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:11.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.972+0000 7f96012a3700 1 -- 172.21.15.163:0/1528291512 shutdown_connections 2024-02-19T21:11:11.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.972+0000 7f96012a3700 1 -- 172.21.15.163:0/1528291512 >> 172.21.15.163:0/1528291512 conn(0x7f95fc0f2470 msgr2=0x7f95fc0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:11.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.972+0000 7f96012a3700 1 -- 172.21.15.163:0/1528291512 shutdown_connections 2024-02-19T21:11:11.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.972+0000 7f96012a3700 1 -- 172.21.15.163:0/1528291512 wait complete. 2024-02-19T21:11:11.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.972+0000 7f96012a3700 1 librados: starting msgr at 2024-02-19T21:11:11.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.972+0000 7f96012a3700 1 librados: starting objecter 2024-02-19T21:11:11.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.972+0000 7f96012a3700 1 Processor -- start 2024-02-19T21:11:11.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f96012a3700 1 -- start start 2024-02-19T21:11:11.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f96012a3700 1 librados: setting wanted keys 2024-02-19T21:11:11.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f96012a3700 1 librados: calling monclient init 2024-02-19T21:11:11.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f96012a3700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f95fc116260 con 0x7f95fc1031b0 2024-02-19T21:11:11.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f96012a3700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f95fc117440 con 0x7f95fc0fc870 2024-02-19T21:11:11.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f96012a3700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f95fc118640 con 0x7f95fc0ff670 2024-02-19T21:11:11.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95fb7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f95fc1031b0 0x7f95fc114920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:39518/0 (socket says 172.21.15.163:39518) 2024-02-19T21:11:11.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95fb7fe700 1 -- 172.21.15.163:0/45174354 learned_addr learned my addr 172.21.15.163:0/45174354 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:11.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 10 client.?.objecter ms_handle_connect 0x7f95fc1031b0 2024-02-19T21:11:11.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:11.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 10 client.?.objecter ms_handle_connect 0x7f95fc0fc870 2024-02-19T21:11:11.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:11.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 10 client.?.objecter ms_handle_connect 0x7f95fc0ff670 2024-02-19T21:11:11.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:11.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211328486 0 0) 0x7f95fc116260 con 0x7f95fc1031b0 2024-02-19T21:11:11.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e4003030 con 0x7f95fc1031b0 2024-02-19T21:11:11.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270245252 0 0) 0x7f95fc117440 con 0x7f95fc0fc870 2024-02-19T21:11:11.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95fc116260 con 0x7f95fc0fc870 2024-02-19T21:11:11.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790003079 0 0) 0x7f95fc118640 con 0x7f95fc0ff670 2024-02-19T21:11:11.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95fc117440 con 0x7f95fc0ff670 2024-02-19T21:11:11.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 576336468 0 0) 0x7f95e4003030 con 0x7f95fc1031b0 2024-02-19T21:11:11.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95fc118640 con 0x7f95fc1031b0 2024-02-19T21:11:11.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f95f4001950 con 0x7f95fc1031b0 2024-02-19T21:11:11.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.973+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3781323642 0 0) 0x7f95fc117440 con 0x7f95fc0ff670 2024-02-19T21:11:11.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.974+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95e4003030 con 0x7f95fc0ff670 2024-02-19T21:11:11.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.974+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f95ec0018d0 con 0x7f95fc0ff670 2024-02-19T21:11:11.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.974+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3291372615 0 0) 0x7f95fc118640 con 0x7f95fc1031b0 2024-02-19T21:11:11.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.974+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 >> v1:172.21.15.163:6790/0 conn(0x7f95fc0ff670 legacy=0x7f95fc0f9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:11.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.974+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 >> v1:172.21.15.193:6789/0 conn(0x7f95fc0fc870 legacy=0x7f95fc0f8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:11.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.974+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95fc119840 con 0x7f95fc1031b0 2024-02-19T21:11:11.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.974+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95fc117630 con 0x7f95fc1031b0 2024-02-19T21:11:11.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.974+0000 7f96012a3700 10 client.23208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.974+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95fc117aa0 con 0x7f95fc1031b0 2024-02-19T21:11:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.974+0000 7f96012a3700 1 librados: init done 2024-02-19T21:11:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.974+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f95f4001500 con 0x7f95fc1031b0 2024-02-19T21:11:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.974+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f95f4003b60 con 0x7f95fc1031b0 2024-02-19T21:11:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.975+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f95e8004f00 con 0x7f95fc1031b0 2024-02-19T21:11:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.975+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f95f4003ce0 con 0x7f95fc1031b0 2024-02-19T21:11:11.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(454..454 src has 43..454) v4 ==== 8063+0+0 (unknown 1903578085 0 0) 0x7f95f4079350 con 0x7f95fc1031b0 2024-02-19T21:11:11.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 osd_map(454..454 src has 43..454) v4 2024-02-19T21:11:11.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f95e3fff700 3 client.23208.objecter handle_osd_map got epochs [454,454] > 0 2024-02-19T21:11:11.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f95e3fff700 3 client.23208.objecter handle_osd_map decoding full epoch 454 2024-02-19T21:11:11.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f95e3fff700 20 client.23208.objecter dump_active .. 0 homeless 2024-02-19T21:11:11.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v454) v1 ==== 54+0+16 (unknown 1521394261 0 2420000192) 0x7f95f407b740 con 0x7f95fc1031b0 2024-02-19T21:11:11.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f95e3fff700 10 client.23208.objecter ms_handle_connect 0x7f95e4063c60 2024-02-19T21:11:11.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f96012a3700 10 client.23208.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:11.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 1 2024-02-19T21:11:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e80051a0 osd=0 3 2024-02-19T21:11:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e80051a0 osd=0 3 2024-02-19T21:11:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e80051a0 osd=0 3 2024-02-19T21:11:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 1 2024-02-19T21:11:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e80051a0 osd=0 3 2024-02-19T21:11:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 0 1 2024-02-19T21:11:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e80051a0 osd=0 4 2024-02-19T21:11:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f96012a3700 10 client.23208.objecter _send_command 1 2024-02-19T21:11:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95e8004950 con 0x7f95e8004530 2024-02-19T21:11:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.976+0000 7f95e3fff700 10 client.23208.objecter ms_handle_connect 0x7f95e8004530 2024-02-19T21:11:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.977+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95e8004950 con 0x7f95e8004530 2024-02-19T21:11:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.977+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:11:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.977+0000 7f95e3fff700 10 client.23208.objecter _finish_command 1 = system:0 2024-02-19T21:11:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.977+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e80051a0 osd=0 4 2024-02-19T21:11:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.977+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 0 1 2024-02-19T21:11:11.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.991+0000 7f96012a3700 10 client.23208.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:11.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.991+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 2 2024-02-19T21:11:11.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.991+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e80051a0 osd=0 3 2024-02-19T21:11:11.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.992+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e80051a0 osd=0 3 2024-02-19T21:11:11.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.992+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e80051a0 osd=0 3 2024-02-19T21:11:11.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.992+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 2 2024-02-19T21:11:11.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.992+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e80051a0 osd=0 3 2024-02-19T21:11:11.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.992+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 0 2 2024-02-19T21:11:11.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.992+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e80051a0 osd=0 4 2024-02-19T21:11:11.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.992+0000 7f96012a3700 10 client.23208.objecter _send_command 2 2024-02-19T21:11:11.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.992+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95e8004d30 con 0x7f95e8004530 2024-02-19T21:11:11.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95e8004d30 con 0x7f95e8004530 2024-02-19T21:11:11.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:11.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f95e3fff700 10 client.23208.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:11.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e80051a0 osd=0 4 2024-02-19T21:11:11.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 0 2 2024-02-19T21:11:11.992 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:11:11.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f96012a3700 10 client.23208.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:11.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 3 2024-02-19T21:11:11.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8003fe0 osd=1 3 2024-02-19T21:11:11.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8003fe0 osd=1 3 2024-02-19T21:11:11.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8003fe0 osd=1 3 2024-02-19T21:11:11.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 3 2024-02-19T21:11:11.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e8003fe0 osd=1 3 2024-02-19T21:11:11.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 1 3 2024-02-19T21:11:11.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8003fe0 osd=1 4 2024-02-19T21:11:11.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f96012a3700 10 client.23208.objecter _send_command 3 2024-02-19T21:11:11.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95e800c000 con 0x7f95e8004150 2024-02-19T21:11:11.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.993+0000 7f95e3fff700 10 client.23208.objecter ms_handle_connect 0x7f95e8004150 2024-02-19T21:11:11.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.994+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95e800c000 con 0x7f95e8004150 2024-02-19T21:11:11.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.994+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:11:11.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.994+0000 7f95e3fff700 10 client.23208.objecter _finish_command 3 = system:0 2024-02-19T21:11:11.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.994+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e8003fe0 osd=1 4 2024-02-19T21:11:11.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:11.994+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 1 3 2024-02-19T21:11:11.995 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:10.450471+0000 mgr.y (mgr.14550) 413 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:11.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:10.450471+0000 mgr.y (mgr.14550) 413 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:12.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.009+0000 7f96012a3700 10 client.23208.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:12.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.009+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 4 2024-02-19T21:11:12.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.009+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8003fe0 osd=1 3 2024-02-19T21:11:12.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.009+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8003fe0 osd=1 3 2024-02-19T21:11:12.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.009+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8003fe0 osd=1 3 2024-02-19T21:11:12.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.009+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 4 2024-02-19T21:11:12.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.009+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e8003fe0 osd=1 3 2024-02-19T21:11:12.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.009+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 1 4 2024-02-19T21:11:12.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.009+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8003fe0 osd=1 4 2024-02-19T21:11:12.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.009+0000 7f96012a3700 10 client.23208.objecter _send_command 4 2024-02-19T21:11:12.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.009+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95e80018b0 con 0x7f95e8004150 2024-02-19T21:11:12.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95e80018b0 con 0x7f95e8004150 2024-02-19T21:11:12.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:12.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f95e3fff700 10 client.23208.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:12.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e8003fe0 osd=1 4 2024-02-19T21:11:12.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 1 4 2024-02-19T21:11:12.009 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:11:12.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f96012a3700 10 client.23208.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:12.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 5 2024-02-19T21:11:12.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8003630 osd=2 3 2024-02-19T21:11:12.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8003630 osd=2 3 2024-02-19T21:11:12.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8003630 osd=2 3 2024-02-19T21:11:12.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 5 2024-02-19T21:11:12.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e8003630 osd=2 3 2024-02-19T21:11:12.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 2 5 2024-02-19T21:11:12.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8003630 osd=2 4 2024-02-19T21:11:12.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f96012a3700 10 client.23208.objecter _send_command 5 2024-02-19T21:11:12.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.010+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95e800de90 con 0x7f95e800d2f0 2024-02-19T21:11:12.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.011+0000 7f95e3fff700 10 client.23208.objecter ms_handle_connect 0x7f95e800d2f0 2024-02-19T21:11:12.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.011+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95e800de90 con 0x7f95e800d2f0 2024-02-19T21:11:12.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.011+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:11:12.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.011+0000 7f95e3fff700 10 client.23208.objecter _finish_command 5 = system:0 2024-02-19T21:11:12.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.011+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e8003630 osd=2 4 2024-02-19T21:11:12.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.011+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 2 5 2024-02-19T21:11:12.011 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:12.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.026+0000 7f96012a3700 10 client.23208.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:12.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.026+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 6 2024-02-19T21:11:12.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.026+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8003630 osd=2 3 2024-02-19T21:11:12.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.026+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8003630 osd=2 3 2024-02-19T21:11:12.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.026+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8003630 osd=2 3 2024-02-19T21:11:12.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.026+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 6 2024-02-19T21:11:12.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.026+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e8003630 osd=2 3 2024-02-19T21:11:12.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.026+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 2 6 2024-02-19T21:11:12.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.026+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8003630 osd=2 4 2024-02-19T21:11:12.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.026+0000 7f96012a3700 10 client.23208.objecter _send_command 6 2024-02-19T21:11:12.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.026+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95e8003100 con 0x7f95e800d2f0 2024-02-19T21:11:12.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.027+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95e8003100 con 0x7f95e800d2f0 2024-02-19T21:11:12.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.027+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:12.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.027+0000 7f95e3fff700 10 client.23208.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:12.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.027+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e8003630 osd=2 4 2024-02-19T21:11:12.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.027+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 2 6 2024-02-19T21:11:12.027 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:11:12.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.027+0000 7f96012a3700 10 client.23208.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:12.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.028+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 7 2024-02-19T21:11:12.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.028+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8011970 osd=3 3 2024-02-19T21:11:12.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.028+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8011970 osd=3 3 2024-02-19T21:11:12.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.028+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8011970 osd=3 3 2024-02-19T21:11:12.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.028+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 7 2024-02-19T21:11:12.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.028+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e8011970 osd=3 3 2024-02-19T21:11:12.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.028+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 3 7 2024-02-19T21:11:12.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.028+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8011970 osd=3 4 2024-02-19T21:11:12.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.028+0000 7f96012a3700 10 client.23208.objecter _send_command 7 2024-02-19T21:11:12.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.028+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95e8015b00 con 0x7f95e8011ff0 2024-02-19T21:11:12.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.028+0000 7f95e3fff700 10 client.23208.objecter ms_handle_connect 0x7f95e8011ff0 2024-02-19T21:11:12.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.028+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95e8015b00 con 0x7f95e8011ff0 2024-02-19T21:11:12.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.029+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:11:12.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.029+0000 7f95e3fff700 10 client.23208.objecter _finish_command 7 = system:0 2024-02-19T21:11:12.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.029+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e8011970 osd=3 4 2024-02-19T21:11:12.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.029+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 3 7 2024-02-19T21:11:12.041 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:12.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.044+0000 7f96012a3700 10 client.23208.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:12.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.044+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 8 2024-02-19T21:11:12.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.044+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8011970 osd=3 3 2024-02-19T21:11:12.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.044+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8011970 osd=3 3 2024-02-19T21:11:12.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.044+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8011970 osd=3 3 2024-02-19T21:11:12.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.044+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 8 2024-02-19T21:11:12.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.044+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e8011970 osd=3 3 2024-02-19T21:11:12.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.044+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 3 8 2024-02-19T21:11:12.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.044+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8011970 osd=3 4 2024-02-19T21:11:12.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.044+0000 7f96012a3700 10 client.23208.objecter _send_command 8 2024-02-19T21:11:12.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.044+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95e800e270 con 0x7f95e8011ff0 2024-02-19T21:11:12.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.045+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95e800e270 con 0x7f95e8011ff0 2024-02-19T21:11:12.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.045+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:12.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.045+0000 7f95e3fff700 10 client.23208.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:12.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.045+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e8011970 osd=3 4 2024-02-19T21:11:12.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.045+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 3 8 2024-02-19T21:11:12.045 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:11:12.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.046+0000 7f96012a3700 10 client.23208.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:12.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.046+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 9 2024-02-19T21:11:12.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.046+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e800e270 osd=4 3 2024-02-19T21:11:12.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.046+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e800e270 osd=4 3 2024-02-19T21:11:12.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.046+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e800e270 osd=4 3 2024-02-19T21:11:12.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.046+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 9 2024-02-19T21:11:12.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.046+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e800e270 osd=4 3 2024-02-19T21:11:12.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.046+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 4 9 2024-02-19T21:11:12.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.046+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e800e270 osd=4 4 2024-02-19T21:11:12.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.046+0000 7f96012a3700 10 client.23208.objecter _send_command 9 2024-02-19T21:11:12.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.046+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95e8018160 con 0x7f95e800e3e0 2024-02-19T21:11:12.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.046+0000 7f95e3fff700 10 client.23208.objecter ms_handle_connect 0x7f95e800e3e0 2024-02-19T21:11:12.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.047+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95e8018160 con 0x7f95e800e3e0 2024-02-19T21:11:12.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.047+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:11:12.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.047+0000 7f95e3fff700 10 client.23208.objecter _finish_command 9 = system:0 2024-02-19T21:11:12.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.047+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e800e270 osd=4 4 2024-02-19T21:11:12.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.047+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 4 9 2024-02-19T21:11:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.065+0000 7f96012a3700 10 client.23208.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.065+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 10 2024-02-19T21:11:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.065+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e800e270 osd=4 3 2024-02-19T21:11:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.065+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e800e270 osd=4 3 2024-02-19T21:11:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.065+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e800e270 osd=4 3 2024-02-19T21:11:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.065+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 10 2024-02-19T21:11:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.065+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e800e270 osd=4 3 2024-02-19T21:11:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.065+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 4 10 2024-02-19T21:11:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.065+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e800e270 osd=4 4 2024-02-19T21:11:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.065+0000 7f96012a3700 10 client.23208.objecter _send_command 10 2024-02-19T21:11:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.065+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95e8018160 con 0x7f95e800e3e0 2024-02-19T21:11:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95e8018160 con 0x7f95e800e3e0 2024-02-19T21:11:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f95e3fff700 10 client.23208.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e800e270 osd=4 4 2024-02-19T21:11:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 4 10 2024-02-19T21:11:12.066 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:11:12.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f96012a3700 10 client.23208.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:12.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 11 2024-02-19T21:11:12.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8018160 osd=5 3 2024-02-19T21:11:12.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8018160 osd=5 3 2024-02-19T21:11:12.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8018160 osd=5 3 2024-02-19T21:11:12.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 11 2024-02-19T21:11:12.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e8018160 osd=5 3 2024-02-19T21:11:12.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 5 11 2024-02-19T21:11:12.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8018160 osd=5 4 2024-02-19T21:11:12.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f96012a3700 10 client.23208.objecter _send_command 11 2024-02-19T21:11:12.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95e8019050 con 0x7f95e8018b90 2024-02-19T21:11:12.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.067+0000 7f95e3fff700 10 client.23208.objecter ms_handle_connect 0x7f95e8018b90 2024-02-19T21:11:12.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.068+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95e8019050 con 0x7f95e8018b90 2024-02-19T21:11:12.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.068+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:11:12.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.068+0000 7f95e3fff700 10 client.23208.objecter _finish_command 11 = system:0 2024-02-19T21:11:12.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.068+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e8018160 osd=5 4 2024-02-19T21:11:12.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.068+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 5 11 2024-02-19T21:11:12.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:10.450471+0000 mgr.y (mgr.14550) 413 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:12.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.082+0000 7f96012a3700 10 client.23208.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:12.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.082+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 12 2024-02-19T21:11:12.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.082+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8018160 osd=5 3 2024-02-19T21:11:12.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.082+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8018160 osd=5 3 2024-02-19T21:11:12.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.082+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8018160 osd=5 3 2024-02-19T21:11:12.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.082+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 12 2024-02-19T21:11:12.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.082+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e8018160 osd=5 3 2024-02-19T21:11:12.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.082+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 5 12 2024-02-19T21:11:12.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.082+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8018160 osd=5 4 2024-02-19T21:11:12.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.082+0000 7f96012a3700 10 client.23208.objecter _send_command 12 2024-02-19T21:11:12.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.082+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95e8019050 con 0x7f95e8018b90 2024-02-19T21:11:12.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95e8019050 con 0x7f95e8018b90 2024-02-19T21:11:12.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:12.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f95e3fff700 10 client.23208.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:12.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e8018160 osd=5 4 2024-02-19T21:11:12.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 5 12 2024-02-19T21:11:12.083 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:11:12.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f96012a3700 10 client.23208.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:12.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 13 2024-02-19T21:11:12.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8019050 osd=6 3 2024-02-19T21:11:12.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8019050 osd=6 3 2024-02-19T21:11:12.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8019050 osd=6 3 2024-02-19T21:11:12.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 13 2024-02-19T21:11:12.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e8019050 osd=6 3 2024-02-19T21:11:12.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 6 13 2024-02-19T21:11:12.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8019050 osd=6 4 2024-02-19T21:11:12.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f96012a3700 10 client.23208.objecter _send_command 13 2024-02-19T21:11:12.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.084+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95e8019470 con 0x7f95e8023660 2024-02-19T21:11:12.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.085+0000 7f95e3fff700 10 client.23208.objecter ms_handle_connect 0x7f95e8023660 2024-02-19T21:11:12.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.086+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95e8019470 con 0x7f95e8023660 2024-02-19T21:11:12.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.086+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:11:12.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.086+0000 7f95e3fff700 10 client.23208.objecter _finish_command 13 = system:0 2024-02-19T21:11:12.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.086+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e8019050 osd=6 4 2024-02-19T21:11:12.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.086+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 6 13 2024-02-19T21:11:12.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.099+0000 7f96012a3700 10 client.23208.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:12.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.099+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 14 2024-02-19T21:11:12.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.099+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8019050 osd=6 3 2024-02-19T21:11:12.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.099+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8019050 osd=6 3 2024-02-19T21:11:12.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.099+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8019050 osd=6 3 2024-02-19T21:11:12.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.099+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 14 2024-02-19T21:11:12.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.099+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e8019050 osd=6 3 2024-02-19T21:11:12.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.099+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 6 14 2024-02-19T21:11:12.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.099+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8019050 osd=6 4 2024-02-19T21:11:12.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.099+0000 7f96012a3700 10 client.23208.objecter _send_command 14 2024-02-19T21:11:12.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.099+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95e8019470 con 0x7f95e8023660 2024-02-19T21:11:12.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.101+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95e8019470 con 0x7f95e8023660 2024-02-19T21:11:12.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.101+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:12.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.101+0000 7f95e3fff700 10 client.23208.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:12.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.101+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e8019050 osd=6 4 2024-02-19T21:11:12.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.101+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 6 14 2024-02-19T21:11:12.101 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:11:12.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.101+0000 7f96012a3700 10 client.23208.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:12.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.102+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 15 2024-02-19T21:11:12.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.102+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8019470 osd=7 3 2024-02-19T21:11:12.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.102+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8019470 osd=7 3 2024-02-19T21:11:12.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.102+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8019470 osd=7 3 2024-02-19T21:11:12.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.102+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 15 2024-02-19T21:11:12.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.102+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e8019470 osd=7 3 2024-02-19T21:11:12.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.102+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 7 15 2024-02-19T21:11:12.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.102+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8019470 osd=7 4 2024-02-19T21:11:12.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.102+0000 7f96012a3700 10 client.23208.objecter _send_command 15 2024-02-19T21:11:12.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.102+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95e80281f0 con 0x7f95e8028cc0 2024-02-19T21:11:12.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.102+0000 7f95e3fff700 10 client.23208.objecter ms_handle_connect 0x7f95e8028cc0 2024-02-19T21:11:12.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.103+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95e80281f0 con 0x7f95e8028cc0 2024-02-19T21:11:12.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.103+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:11:12.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.103+0000 7f95e3fff700 10 client.23208.objecter _finish_command 15 = system:0 2024-02-19T21:11:12.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.103+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e8019470 osd=7 4 2024-02-19T21:11:12.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.103+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 7 15 2024-02-19T21:11:12.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.117+0000 7f96012a3700 10 client.23208.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:12.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.117+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign -1 16 2024-02-19T21:11:12.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.117+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8019470 osd=7 3 2024-02-19T21:11:12.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.117+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8019470 osd=7 3 2024-02-19T21:11:12.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.117+0000 7f96012a3700 20 client.23208.objecter _get_session s=0x7f95e8019470 osd=7 3 2024-02-19T21:11:12.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.117+0000 7f96012a3700 15 client.23208.objecter _session_command_op_remove -1 16 2024-02-19T21:11:12.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.117+0000 7f96012a3700 20 client.23208.objecter get_session s=0x7f95e8019470 osd=7 3 2024-02-19T21:11:12.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.117+0000 7f96012a3700 15 client.23208.objecter _session_command_op_assign 7 16 2024-02-19T21:11:12.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.117+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8019470 osd=7 4 2024-02-19T21:11:12.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.117+0000 7f96012a3700 10 client.23208.objecter _send_command 16 2024-02-19T21:11:12.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.117+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95e80281f0 con 0x7f95e8028cc0 2024-02-19T21:11:12.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f95e3fff700 1 -- 172.21.15.163:0/45174354 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95e80281f0 con 0x7f95e8028cc0 2024-02-19T21:11:12.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f95e3fff700 10 client.23208.objecter ms_dispatch 0x7f95fc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:12.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f95e3fff700 10 client.23208.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:12.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f95e3fff700 20 client.23208.objecter put_session s=0x7f95e8019470 osd=7 4 2024-02-19T21:11:12.119 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f95e3fff700 15 client.23208.objecter _session_command_op_remove 7 16 2024-02-19T21:11:12.120 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:11:12.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 10 librados: watch_flush enter 2024-02-19T21:11:12.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 10 librados: watch_flush exit 2024-02-19T21:11:12.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 10 client.23208.objecter close_session for osd.0 2024-02-19T21:11:12.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 >> v1:172.21.15.163:6801/1559418115 conn(0x7f95e8004530 legacy=0x7f95e8002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:12.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e80051a0 osd=0 1 2024-02-19T21:11:12.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 10 client.23208.objecter close_session for osd.1 2024-02-19T21:11:12.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 >> v1:172.21.15.163:6805/3125047432 conn(0x7f95e8004150 legacy=0x7f95e800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:12.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8003fe0 osd=1 1 2024-02-19T21:11:12.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 10 client.23208.objecter close_session for osd.2 2024-02-19T21:11:12.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 >> v1:172.21.15.163:6809/1563158487 conn(0x7f95e800d2f0 legacy=0x7f95e800d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:12.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8003630 osd=2 1 2024-02-19T21:11:12.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 10 client.23208.objecter close_session for osd.3 2024-02-19T21:11:12.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 >> v1:172.21.15.163:6813/2112279769 conn(0x7f95e8011ff0 legacy=0x7f95e80143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:12.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f95fb7fe700 1 -- 172.21.15.163:0/45174354 reap_dead start 2024-02-19T21:11:12.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8011970 osd=3 1 2024-02-19T21:11:12.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 10 client.23208.objecter close_session for osd.4 2024-02-19T21:11:12.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 >> v1:172.21.15.193:6800/3129716965 conn(0x7f95e800e3e0 legacy=0x7f95e8018480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:12.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e800e270 osd=4 1 2024-02-19T21:11:12.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 10 client.23208.objecter close_session for osd.5 2024-02-19T21:11:12.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 >> v1:172.21.15.193:6804/805161991 conn(0x7f95e8018b90 legacy=0x7f95e8021a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:12.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8018160 osd=5 1 2024-02-19T21:11:12.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 10 client.23208.objecter close_session for osd.6 2024-02-19T21:11:12.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 >> v1:172.21.15.193:6808/787604615 conn(0x7f95e8023660 legacy=0x7f95e8025a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:12.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8019050 osd=6 1 2024-02-19T21:11:12.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 10 client.23208.objecter close_session for osd.7 2024-02-19T21:11:12.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 >> v1:172.21.15.193:6812/157927432 conn(0x7f95e8028cc0 legacy=0x7f95e8029120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:12.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 20 client.23208.objecter put_session s=0x7f95e8019470 osd=7 1 2024-02-19T21:11:12.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 20 client.23208.objecter shutdown clearing up homeless session... 2024-02-19T21:11:12.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 10 client.23208.objecter successfully canceled tick 2024-02-19T21:11:12.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f95fb7fe700 1 -- 172.21.15.163:0/45174354 reap_dead start 2024-02-19T21:11:12.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 >> v1:172.21.15.163:6800/1552514681 conn(0x7f95e4063c60 legacy=0x7f95e4066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:12.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.119+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 >> v1:172.21.15.163:6789/0 conn(0x7f95fc1031b0 legacy=0x7f95fc114920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:12.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.120+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 shutdown_connections 2024-02-19T21:11:12.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.120+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 >> 172.21.15.163:0/45174354 conn(0x7f95fc0f2470 msgr2=0x7f95fc106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:12.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.120+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 shutdown_connections 2024-02-19T21:11:12.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.120+0000 7f96012a3700 1 -- 172.21.15.163:0/45174354 wait complete. 2024-02-19T21:11:12.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.120+0000 7f96012a3700 1 librados: shutdown 2024-02-19T21:11:12.124 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:11:12.124 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:11:12.124 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:11:12.125 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:11:12.125 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:11:12.125 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:11:12.125 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:11:12.125 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:11:12.164 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:12.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:12.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:12.268 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:12.298 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:12.319 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:12.322 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:12.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:12.423 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:11:12.451 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:12.471 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:11:12.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:12.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:11:12.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.558+0000 7f251de6a700 1 -- 172.21.15.163:0/3235280406 >> v1:172.21.15.163:6790/0 conn(0x7f25180f9530 legacy=0x7f2518102a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:12.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.558+0000 7f251de6a700 1 -- 172.21.15.163:0/3235280406 shutdown_connections 2024-02-19T21:11:12.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.558+0000 7f251de6a700 1 -- 172.21.15.163:0/3235280406 >> 172.21.15.163:0/3235280406 conn(0x7f25180f2470 msgr2=0x7f25180f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:12.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.558+0000 7f251de6a700 1 -- 172.21.15.163:0/3235280406 shutdown_connections 2024-02-19T21:11:12.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.558+0000 7f251de6a700 1 -- 172.21.15.163:0/3235280406 wait complete. 2024-02-19T21:11:12.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.558+0000 7f251de6a700 1 librados: starting msgr at 2024-02-19T21:11:12.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.558+0000 7f251de6a700 1 librados: starting objecter 2024-02-19T21:11:12.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.558+0000 7f251de6a700 1 Processor -- start 2024-02-19T21:11:12.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.558+0000 7f251de6a700 1 -- start start 2024-02-19T21:11:12.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f251de6a700 1 librados: setting wanted keys 2024-02-19T21:11:12.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f251de6a700 1 librados: calling monclient init 2024-02-19T21:11:12.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f251de6a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f25181173a0 con 0x7f25181042b0 2024-02-19T21:11:12.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f251de6a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2518118580 con 0x7f25180f9530 2024-02-19T21:11:12.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f251de6a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2518119780 con 0x7f25180f8980 2024-02-19T21:11:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f25177fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f25180f8980 0x7f25180ff6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45582/0 (socket says 172.21.15.163:45582) 2024-02-19T21:11:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f25177fe700 1 -- 172.21.15.163:0/2170793919 learned_addr learned my addr 172.21.15.163:0/2170793919 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 10 client.?.objecter ms_handle_connect 0x7f25180f8980 2024-02-19T21:11:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 10 client.?.objecter ms_handle_connect 0x7f25181042b0 2024-02-19T21:11:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 10 client.?.objecter ms_handle_connect 0x7f25180f9530 2024-02-19T21:11:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362033321 0 0) 0x7f2518119780 con 0x7f25180f8980 2024-02-19T21:11:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2504003630 con 0x7f25180f8980 2024-02-19T21:11:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249311999 0 0) 0x7f25181173a0 con 0x7f25181042b0 2024-02-19T21:11:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2518119780 con 0x7f25181042b0 2024-02-19T21:11:12.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559502849 0 0) 0x7f2518118580 con 0x7f25180f9530 2024-02-19T21:11:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25181173a0 con 0x7f25180f9530 2024-02-19T21:11:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 978414821 0 0) 0x7f2518119780 con 0x7f25181042b0 2024-02-19T21:11:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2518118580 con 0x7f25181042b0 2024-02-19T21:11:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1306379164 0 0) 0x7f2504003630 con 0x7f25180f8980 2024-02-19T21:11:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2518119780 con 0x7f25180f8980 2024-02-19T21:11:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2510001b40 con 0x7f25181042b0 2024-02-19T21:11:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f25080028d0 con 0x7f25180f8980 2024-02-19T21:11:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 682705418 0 0) 0x7f25181173a0 con 0x7f25180f9530 2024-02-19T21:11:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.559+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2504003630 con 0x7f25180f9530 2024-02-19T21:11:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.560+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2500002050 con 0x7f25180f9530 2024-02-19T21:11:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.560+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4190306504 0 0) 0x7f2518118580 con 0x7f25181042b0 2024-02-19T21:11:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.560+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 >> v1:172.21.15.163:6790/0 conn(0x7f25180f8980 legacy=0x7f25180ff6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.560+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 >> v1:172.21.15.193:6789/0 conn(0x7f25180f9530 legacy=0x7f25180ffe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.560+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251811a980 con 0x7f25181042b0 2024-02-19T21:11:12.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.560+0000 7f251de6a700 1 -- 172.21.15.163:0/2170793919 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2518117590 con 0x7f25181042b0 2024-02-19T21:11:12.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.560+0000 7f251de6a700 10 client.23214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:12.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.560+0000 7f251de6a700 1 -- 172.21.15.163:0/2170793919 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2518117a00 con 0x7f25181042b0 2024-02-19T21:11:12.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.560+0000 7f251de6a700 1 librados: init done 2024-02-19T21:11:12.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.560+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f2510002200 con 0x7f25181042b0 2024-02-19T21:11:12.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.560+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2510003510 con 0x7f25181042b0 2024-02-19T21:11:12.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.561+0000 7f251de6a700 1 -- 172.21.15.163:0/2170793919 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f8004f20 con 0x7f25181042b0 2024-02-19T21:11:12.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.561+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f2510001d60 con 0x7f25181042b0 2024-02-19T21:11:12.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.562+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(454..454 src has 43..454) v4 ==== 8063+0+0 (unknown 1903578085 0 0) 0x7f2510076740 con 0x7f25181042b0 2024-02-19T21:11:12.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.562+0000 7f2514ff9700 10 client.23214.objecter ms_dispatch 0x7f2518001550 osd_map(454..454 src has 43..454) v4 2024-02-19T21:11:12.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.562+0000 7f2514ff9700 3 client.23214.objecter handle_osd_map got epochs [454,454] > 0 2024-02-19T21:11:12.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.562+0000 7f2514ff9700 3 client.23214.objecter handle_osd_map decoding full epoch 454 2024-02-19T21:11:12.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.562+0000 7f2514ff9700 20 client.23214.objecter dump_active .. 0 homeless 2024-02-19T21:11:12.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.562+0000 7f2514ff9700 10 client.23214.objecter ms_handle_connect 0x7f2504063ca0 2024-02-19T21:11:12.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.566+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f2510049b90 con 0x7f25181042b0 2024-02-19T21:11:12.575 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:12.592 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:11:12.605 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:12.640 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:12.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:12.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:12.747 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:12.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.753+0000 7f251de6a700 1 -- 172.21.15.163:0/2170793919 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f24f80051d0 con 0x7f25181042b0 2024-02-19T21:11:12.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.755+0000 7f2514ff9700 1 -- 172.21.15.163:0/2170793919 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v454) v1 ==== 81+0+48269 (unknown 442685706 0 3827065048) 0x7f2510048000 con 0x7f25181042b0 2024-02-19T21:11:12.755 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:11:12.755 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:11:12.755 INFO:teuthology.orchestra.run.smithi163.stdout: "epoch": 454, 2024-02-19T21:11:12.755 INFO:teuthology.orchestra.run.smithi163.stdout: "fsid": "2591809a-cf69-11ee-95bb-87774f69a715", 2024-02-19T21:11:12.755 INFO:teuthology.orchestra.run.smithi163.stdout: "created": "2024-02-19T20:55:37.794885+0000", 2024-02-19T21:11:12.755 INFO:teuthology.orchestra.run.smithi163.stdout: "modified": "2024-02-19T21:11:08.679950+0000", 2024-02-19T21:11:12.755 INFO:teuthology.orchestra.run.smithi163.stdout: "last_up_change": "2024-02-19T21:01:09.214134+0000", 2024-02-19T21:11:12.755 INFO:teuthology.orchestra.run.smithi163.stdout: "last_in_change": "2024-02-19T21:10:40.268547+0000", 2024-02-19T21:11:12.755 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-19T21:11:12.755 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_num": 5801984, 2024-02-19T21:11:12.755 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_set": [ 2024-02-19T21:11:12.755 INFO:teuthology.orchestra.run.smithi163.stdout: "noscrub", 2024-02-19T21:11:12.756 INFO:teuthology.orchestra.run.smithi163.stdout: "pglog_hardlimit", 2024-02-19T21:11:12.756 INFO:teuthology.orchestra.run.smithi163.stdout: "purged_snapdirs", 2024-02-19T21:11:12.756 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_deletes", 2024-02-19T21:11:12.756 INFO:teuthology.orchestra.run.smithi163.stdout: "sortbitwise" 2024-02-19T21:11:12.756 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:11:12.756 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_version": 103, 2024-02-19T21:11:12.756 INFO:teuthology.orchestra.run.smithi163.stdout: "full_ratio": 0.94999998807907104, 2024-02-19T21:11:12.756 INFO:teuthology.orchestra.run.smithi163.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-19T21:11:12.756 INFO:teuthology.orchestra.run.smithi163.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-19T21:11:12.757 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_snapshot": "", 2024-02-19T21:11:12.757 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_max": 216, 2024-02-19T21:11:12.757 INFO:teuthology.orchestra.run.smithi163.stdout: "max_osd": 8, 2024-02-19T21:11:12.757 INFO:teuthology.orchestra.run.smithi163.stdout: "require_min_compat_client": "luminous", 2024-02-19T21:11:12.757 INFO:teuthology.orchestra.run.smithi163.stdout: "min_compat_client": "luminous", 2024-02-19T21:11:12.757 INFO:teuthology.orchestra.run.smithi163.stdout: "require_osd_release": "pacific", 2024-02-19T21:11:12.757 INFO:teuthology.orchestra.run.smithi163.stdout: "pools": [ 2024-02-19T21:11:12.757 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.757 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 1, 2024-02-19T21:11:12.757 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "device_health_metrics", 2024-02-19T21:11:12.757 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T20:59:09.425169+0000", 2024-02-19T21:11:12.757 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:11:12.757 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:11:12.758 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:11:12.758 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:11:12.758 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:11:12.758 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:11:12.758 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:11:12.758 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:11:12.758 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:11:12.758 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:11:12.758 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:11:12.758 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:11:12.758 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 1, 2024-02-19T21:11:12.758 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 1, 2024-02-19T21:11:12.758 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 1, 2024-02-19T21:11:12.758 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 1, 2024-02-19T21:11:12.758 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 1, 2024-02-19T21:11:12.759 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:11:12.759 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:11:12.759 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:11:12.759 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:11:12.759 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:11:12.759 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:11:12.759 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:11:12.759 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.759 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "74", 2024-02-19T21:11:12.759 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:11:12.759 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:11:12.759 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:11:12.759 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:11:12.759 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:11:12.760 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:11:12.760 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:11:12.760 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:11:12.760 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:11:12.760 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:11:12.760 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:11:12.760 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:11:12.760 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:11:12.760 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:11:12.760 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:11:12.760 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:11:12.760 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:11:12.760 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:11:12.760 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:11:12.761 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:11:12.761 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:11:12.761 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:11:12.761 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:11:12.761 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:11:12.761 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:11:12.761 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:11:12.761 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.761 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:11:12.761 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:11:12.761 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:11:12.761 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:11:12.762 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:11:12.762 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:11:12.762 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:11:12.762 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:11:12.762 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:11:12.762 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:11:12.762 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:11:12.762 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:11:12.762 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_max": 32, 2024-02-19T21:11:12.762 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_min": 1 2024-02-19T21:11:12.762 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.762 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:11:12.762 INFO:teuthology.orchestra.run.smithi163.stdout: "mgr_devicehealth": {} 2024-02-19T21:11:12.762 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.763 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.763 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.763 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 2, 2024-02-19T21:11:12.763 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": ".rgw.root", 2024-02-19T21:11:12.763 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:14.658945+0000", 2024-02-19T21:11:12.763 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:11:12.763 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:11:12.763 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:11:12.763 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:11:12.763 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:11:12.763 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:11:12.763 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:11:12.763 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:11:12.763 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:11:12.763 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:11:12.764 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:11:12.764 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:11:12.764 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:11:12.764 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:11:12.764 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:11:12.764 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:11:12.764 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:11:12.764 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:11:12.764 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "2.20", 2024-02-19T21:11:12.764 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 309, 2024-02-19T21:11:12.764 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 308, 2024-02-19T21:11:12.764 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 308, 2024-02-19T21:11:12.764 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "58'1", 2024-02-19T21:11:12.764 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "58'1" 2024-02-19T21:11:12.765 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.765 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "416", 2024-02-19T21:11:12.765 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:11:12.765 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "309", 2024-02-19T21:11:12.765 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "307", 2024-02-19T21:11:12.765 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:11:12.765 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:11:12.765 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:11:12.765 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:11:12.765 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:11:12.765 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:11:12.765 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:11:12.765 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:11:12.765 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:11:12.766 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:11:12.766 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:11:12.766 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:11:12.766 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:11:12.766 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:11:12.766 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:11:12.766 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:11:12.766 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:11:12.766 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:11:12.767 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:11:12.767 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:11:12.767 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:11:12.767 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:11:12.767 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:11:12.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.767 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:11:12.767 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:11:12.767 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:11:12.767 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:11:12.767 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:11:12.767 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:11:12.767 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:11:12.768 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:11:12.768 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:11:12.768 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:11:12.768 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:11:12.768 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:11:12.768 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:11:12.768 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:11:12.768 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.768 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 3, 2024-02-19T21:11:12.768 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "iscsi", 2024-02-19T21:11:12.768 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:14.958800+0000", 2024-02-19T21:11:12.768 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:11:12.769 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:11:12.769 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:11:12.769 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:11:12.769 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:11:12.769 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:11:12.769 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:11:12.769 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:11:12.769 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:11:12.769 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:11:12.769 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:11:12.769 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:11:12.769 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 3, 2024-02-19T21:11:12.769 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 3, 2024-02-19T21:11:12.769 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 3, 2024-02-19T21:11:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 3, 2024-02-19T21:11:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 3, 2024-02-19T21:11:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:11:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:11:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:11:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:11:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:11:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:11:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:11:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "100", 2024-02-19T21:11:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:11:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:11:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:11:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:11:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:11:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:11:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:11:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:11:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:11:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:11:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:11:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:11:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:11:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:11:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:11:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:11:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:11:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:11:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:11:12.772 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:11:12.772 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:11:12.772 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:11:12.772 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:11:12.772 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:11:12.772 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:11:12.772 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:11:12.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.772 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:11:12.772 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:11:12.772 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:11:12.772 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:11:12.772 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:11:12.772 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:11:12.773 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:11:12.773 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:11:12.773 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:11:12.773 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:11:12.773 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:11:12.773 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:11:12.773 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:11:12.773 INFO:teuthology.orchestra.run.smithi163.stdout: "rbd": {} 2024-02-19T21:11:12.773 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.773 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 4, 2024-02-19T21:11:12.773 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.log", 2024-02-19T21:11:12.773 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:16.543816+0000", 2024-02-19T21:11:12.774 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:11:12.774 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:11:12.774 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:11:12.774 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:11:12.774 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:11:12.774 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:11:12.774 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:11:12.774 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:11:12.774 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:11:12.774 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:11:12.774 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:11:12.774 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:11:12.774 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:11:12.774 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:11:12.774 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:11:12.775 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:11:12.775 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:11:12.775 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:11:12.775 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:11:12.775 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:11:12.775 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:11:12.775 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:11:12.775 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:11:12.775 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:11:12.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.775 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "61", 2024-02-19T21:11:12.775 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:11:12.775 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:11:12.775 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:11:12.776 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:11:12.776 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:11:12.776 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:11:12.776 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:11:12.776 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:11:12.776 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:11:12.776 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:11:12.776 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:11:12.776 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:11:12.776 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:11:12.776 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:11:12.776 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:11:12.776 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:11:12.776 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:11:12.777 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:11:12.777 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:11:12.777 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:11:12.777 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:11:12.777 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:11:12.777 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:11:12.777 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:11:12.777 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:11:12.778 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:11:12.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.778 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:11:12.778 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:11:12.778 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:11:12.778 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:11:12.778 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:11:12.778 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:11:12.778 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:11:12.778 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:11:12.778 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:11:12.778 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:11:12.778 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:11:12.778 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:11:12.779 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:11:12.779 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:11:12.779 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.779 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 5, 2024-02-19T21:11:12.779 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.control", 2024-02-19T21:11:12.779 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:18.521482+0000", 2024-02-19T21:11:12.779 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:11:12.779 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:11:12.779 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:11:12.779 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:11:12.779 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:11:12.779 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:11:12.779 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:11:12.780 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:11:12.780 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:11:12.780 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:11:12.780 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:11:12.780 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:11:12.780 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:11:12.780 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:11:12.780 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:11:12.780 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:11:12.780 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:11:12.780 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:11:12.780 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:11:12.780 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:11:12.780 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:11:12.781 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:11:12.781 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:11:12.781 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:11:12.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.781 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "63", 2024-02-19T21:11:12.781 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:11:12.781 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:11:12.781 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:11:12.781 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:11:12.781 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:11:12.781 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:11:12.781 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:11:12.781 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:11:12.781 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:11:12.782 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:11:12.782 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:11:12.782 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:11:12.782 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:11:12.782 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:11:12.782 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:11:12.782 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:11:12.782 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:11:12.782 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:11:12.782 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:11:12.782 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:11:12.782 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:11:12.782 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:11:12.782 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:11:12.783 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:11:12.783 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:11:12.783 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:11:12.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.783 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:11:12.783 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:11:12.783 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:11:12.783 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:11:12.783 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:11:12.783 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:11:12.783 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:11:12.783 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:11:12.783 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:11:12.783 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:11:12.784 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:11:12.784 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:11:12.784 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:11:12.784 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:11:12.784 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.784 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 6, 2024-02-19T21:11:12.784 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "default.rgw.meta", 2024-02-19T21:11:12.784 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:01:20.586564+0000", 2024-02-19T21:11:12.784 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:11:12.784 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:11:12.784 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:11:12.784 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:11:12.785 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:11:12.785 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:11:12.785 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:11:12.785 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:11:12.785 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:11:12.785 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:11:12.785 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:11:12.785 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:11:12.785 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:11:12.785 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:11:12.785 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:11:12.785 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:11:12.785 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:11:12.785 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:11:12.785 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:11:12.786 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:11:12.786 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:11:12.786 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:11:12.786 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:11:12.786 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:11:12.786 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.786 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "65", 2024-02-19T21:11:12.786 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:11:12.786 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:11:12.786 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:11:12.786 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:11:12.786 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:11:12.786 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:11:12.786 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:11:12.787 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:11:12.787 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:11:12.787 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:11:12.787 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:11:12.787 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:11:12.787 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:11:12.787 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:11:12.787 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:11:12.787 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:11:12.788 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:11:12.788 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:11:12.788 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:11:12.788 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:11:12.788 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:11:12.788 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:11:12.788 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:11:12.788 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:11:12.788 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:11:12.788 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:11:12.788 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.788 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:11:12.789 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:11:12.789 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:11:12.789 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:11:12.789 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:11:12.789 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:11:12.789 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:11:12.789 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:11:12.789 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:11:12.789 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:11:12.789 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:11:12.789 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:11:12.789 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_bias": 4 2024-02-19T21:11:12.789 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.789 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:11:12.790 INFO:teuthology.orchestra.run.smithi163.stdout: "rgw": {} 2024-02-19T21:11:12.790 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.790 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.790 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.790 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 16, 2024-02-19T21:11:12.790 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-02-19T21:11:12.790 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:35.333192+0000", 2024-02-19T21:11:12.790 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:11:12.790 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:11:12.790 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:11:12.790 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:11:12.790 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:11:12.790 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:11:12.790 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:11:12.791 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:11:12.791 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:11:12.791 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:11:12.791 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:11:12.791 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:11:12.791 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:11:12.791 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:11:12.791 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:11:12.791 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:11:12.791 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:11:12.791 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:11:12.791 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:11:12.791 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:11:12.791 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:11:12.792 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:11:12.792 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:11:12.792 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:11:12.792 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.792 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "101", 2024-02-19T21:11:12.792 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:11:12.792 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:11:12.792 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:11:12.792 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:11:12.792 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:11:12.792 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:11:12.792 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:11:12.792 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:11:12.793 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:11:12.793 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:11:12.793 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:11:12.793 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:11:12.793 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:11:12.793 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:11:12.793 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:11:12.793 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:11:12.793 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:11:12.793 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:11:12.793 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:11:12.793 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:11:12.793 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:11:12.793 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:11:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:11:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:11:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:11:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:11:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:11:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:11:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:11:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:11:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:11:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:11:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:11:12.794 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:11:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:11:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:11:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:11:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:11:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": {} 2024-02-19T21:11:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 61, 2024-02-19T21:11:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "test-rados-api-smithi163-83928-6", 2024-02-19T21:11:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:04:52.991355+0000", 2024-02-19T21:11:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 8193, 2024-02-19T21:11:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-19T21:11:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:11:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:11:12.795 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:11:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:11:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:11:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:11:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:11:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:11:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:11:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:11:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:11:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:11:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:11:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:11:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:11:12.796 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:11:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:11:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:11:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:11:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:11:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:11:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:11:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "447", 2024-02-19T21:11:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "386", 2024-02-19T21:11:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "386", 2024-02-19T21:11:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "386", 2024-02-19T21:11:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:11:12.797 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:11:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 32, 2024-02-19T21:11:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 447, 2024-02-19T21:11:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:11:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:11:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:11:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:11:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:11:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:11:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:11:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:11:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:11:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:11:12.798 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:11:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:11:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:11:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:11:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:11:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:11:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:11:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:11:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:11:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:11:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:11:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:11:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:11:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:11:12.799 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:11:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:11:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:11:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:11:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:11:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:11:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: "options": { 2024-02-19T21:11:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: "fingerprint_algorithm": "sha1" 2024-02-19T21:11:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:11:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-02-19T21:11:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.800 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 216, 2024-02-19T21:11:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_name": "test-rados-api-smithi163-83928-69", 2024-02-19T21:11:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "create_time": "2024-02-19T21:10:50.528170+0000", 2024-02-19T21:11:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "flags": 1, 2024-02-19T21:11:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "flags_names": "hashpspool", 2024-02-19T21:11:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "type": 1, 2024-02-19T21:11:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 3, 2024-02-19T21:11:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "min_size": 2, 2024-02-19T21:11:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_rule": 0, 2024-02-19T21:11:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_count": 0, 2024-02-19T21:11:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_target": 0, 2024-02-19T21:11:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_barrier": 0, 2024-02-19T21:11:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-19T21:11:12.801 INFO:teuthology.orchestra.run.smithi163.stdout: "object_hash": 2, 2024-02-19T21:11:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_autoscale_mode": "off", 2024-02-19T21:11:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num": 32, 2024-02-19T21:11:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num": 32, 2024-02-19T21:11:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_placement_num_target": 32, 2024-02-19T21:11:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_target": 32, 2024-02-19T21:11:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_num_pending": 32, 2024-02-19T21:11:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: "last_pg_merge_meta": { 2024-02-19T21:11:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: "source_pgid": "0.0", 2024-02-19T21:11:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: "ready_epoch": 0, 2024-02-19T21:11:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_started": 0, 2024-02-19T21:11:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: "last_epoch_clean": 0, 2024-02-19T21:11:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: "source_version": "0'0", 2024-02-19T21:11:12.802 INFO:teuthology.orchestra.run.smithi163.stdout: "target_version": "0'0" 2024-02-19T21:11:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "last_change": "446", 2024-02-19T21:11:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend": "0", 2024-02-19T21:11:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-19T21:11:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "last_force_op_resend_preluminous": "0", 2024-02-19T21:11:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "auid": 0, 2024-02-19T21:11:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_mode": "selfmanaged", 2024-02-19T21:11:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_seq": 0, 2024-02-19T21:11:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "snap_epoch": 0, 2024-02-19T21:11:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "pool_snaps": [], 2024-02-19T21:11:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps": "[]", 2024-02-19T21:11:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_bytes": 0, 2024-02-19T21:11:12.803 INFO:teuthology.orchestra.run.smithi163.stdout: "quota_max_objects": 0, 2024-02-19T21:11:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "tiers": [], 2024-02-19T21:11:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "tier_of": -1, 2024-02-19T21:11:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "read_tier": -1, 2024-02-19T21:11:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "write_tier": -1, 2024-02-19T21:11:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_mode": "none", 2024-02-19T21:11:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_bytes": 0, 2024-02-19T21:11:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "target_max_objects": 0, 2024-02-19T21:11:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-19T21:11:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-19T21:11:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-19T21:11:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_flush_age": 0, 2024-02-19T21:11:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "cache_min_evict_age": 0, 2024-02-19T21:11:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profile": "", 2024-02-19T21:11:12.804 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_params": { 2024-02-19T21:11:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "none" 2024-02-19T21:11:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_period": 0, 2024-02-19T21:11:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_count": 0, 2024-02-19T21:11:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "use_gmt_hitset": true, 2024-02-19T21:11:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "min_read_recency_for_promote": 0, 2024-02-19T21:11:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "min_write_recency_for_promote": 0, 2024-02-19T21:11:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_grade_decay_rate": 0, 2024-02-19T21:11:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "hit_set_search_last_n": 0, 2024-02-19T21:11:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "grade_table": [], 2024-02-19T21:11:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "stripe_width": 0, 2024-02-19T21:11:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "expected_num_objects": 0, 2024-02-19T21:11:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "fast_read": false, 2024-02-19T21:11:12.805 INFO:teuthology.orchestra.run.smithi163.stdout: "options": {}, 2024-02-19T21:11:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: "application_metadata": { 2024-02-19T21:11:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: "rados": {} 2024-02-19T21:11:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:11:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: "osds": [ 2024-02-19T21:11:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-02-19T21:11:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "4a14b406-7c21-40f3-8b12-8cf9df811088", 2024-02-19T21:11:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:11:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:11:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:11:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0, 2024-02-19T21:11:12.806 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:11:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:11:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 8, 2024-02-19T21:11:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 218, 2024-02-19T21:11:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:11:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:11:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:11:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6801", 2024-02-19T21:11:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:11:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.807 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:11:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6802", 2024-02-19T21:11:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:11:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:11:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.808 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6804", 2024-02-19T21:11:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:11:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:11:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6803", 2024-02-19T21:11:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1559418115 2024-02-19T21:11:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.809 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6801/1559418115", 2024-02-19T21:11:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6802/1559418115", 2024-02-19T21:11:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6804/1559418115", 2024-02-19T21:11:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6803/1559418115", 2024-02-19T21:11:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:11:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:11:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:11:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.810 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-02-19T21:11:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "e74a65fa-cb9c-42c3-9293-602cf7e9dc67", 2024-02-19T21:11:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:11:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:11:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:11:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:11:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:11:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:11:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 13, 2024-02-19T21:11:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 444, 2024-02-19T21:11:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:11:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:11:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:11:12.811 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6805", 2024-02-19T21:11:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:11:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:11:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6806", 2024-02-19T21:11:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:11:12.812 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:11:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6808", 2024-02-19T21:11:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:11:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:11:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.813 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6807", 2024-02-19T21:11:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3125047432 2024-02-19T21:11:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6805/3125047432", 2024-02-19T21:11:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6806/3125047432", 2024-02-19T21:11:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6808/3125047432", 2024-02-19T21:11:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6807/3125047432", 2024-02-19T21:11:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:11:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:11:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:11:12.814 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-02-19T21:11:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "6980a625-cdc9-487e-8f8c-7997ba716a97", 2024-02-19T21:11:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:11:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:11:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:11:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:11:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:11:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:11:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 18, 2024-02-19T21:11:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 449, 2024-02-19T21:11:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:11:12.815 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:11:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:11:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6809", 2024-02-19T21:11:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:11:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:11:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.816 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6810", 2024-02-19T21:11:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:11:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:11:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6812", 2024-02-19T21:11:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:11:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.817 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:11:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6811", 2024-02-19T21:11:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1563158487 2024-02-19T21:11:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6809/1563158487", 2024-02-19T21:11:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6810/1563158487", 2024-02-19T21:11:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6812/1563158487", 2024-02-19T21:11:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6811/1563158487", 2024-02-19T21:11:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:11:12.818 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:11:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:11:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-02-19T21:11:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "359e2ee0-9d1d-4074-8119-2d69bc2ca3a5", 2024-02-19T21:11:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:11:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:11:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:11:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:11:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:11:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:11:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 25, 2024-02-19T21:11:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 444, 2024-02-19T21:11:12.819 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:11:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:11:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:11:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6813", 2024-02-19T21:11:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:11:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:11:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.820 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6814", 2024-02-19T21:11:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:11:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:11:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6816", 2024-02-19T21:11:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:11:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.821 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:11:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:6815", 2024-02-19T21:11:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2112279769 2024-02-19T21:11:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.823 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.163:6813/2112279769", 2024-02-19T21:11:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.163:6814/2112279769", 2024-02-19T21:11:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.163:6816/2112279769", 2024-02-19T21:11:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.163:6815/2112279769", 2024-02-19T21:11:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:11:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:11:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:11:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-02-19T21:11:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "2bf5ef05-292b-4693-85a9-97991f2091e0", 2024-02-19T21:11:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:11:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:11:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:11:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:11:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:11:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:11:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 31, 2024-02-19T21:11:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 444, 2024-02-19T21:11:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:11:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:11:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:11:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6800", 2024-02-19T21:11:12.825 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:11:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:11:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6801", 2024-02-19T21:11:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:11:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:11:12.826 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6803", 2024-02-19T21:11:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:11:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:11:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6802", 2024-02-19T21:11:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3129716965 2024-02-19T21:11:12.827 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6800/3129716965", 2024-02-19T21:11:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6801/3129716965", 2024-02-19T21:11:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6803/3129716965", 2024-02-19T21:11:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6802/3129716965", 2024-02-19T21:11:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:11:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:11:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:11:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.828 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-02-19T21:11:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "59065f9b-9240-414b-9c91-e91cda4aed6f", 2024-02-19T21:11:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:11:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:11:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:11:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:11:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:11:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:11:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 37, 2024-02-19T21:11:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 444, 2024-02-19T21:11:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:11:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:11:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:11:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.829 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6804", 2024-02-19T21:11:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:11:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:11:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6805", 2024-02-19T21:11:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:11:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.830 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:11:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6807", 2024-02-19T21:11:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:11:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:11:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.831 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6806", 2024-02-19T21:11:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 805161991 2024-02-19T21:11:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6804/805161991", 2024-02-19T21:11:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6805/805161991", 2024-02-19T21:11:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6807/805161991", 2024-02-19T21:11:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6806/805161991", 2024-02-19T21:11:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:11:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:11:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:11:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-02-19T21:11:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "302c95df-3842-4818-8afd-d79d8184684f", 2024-02-19T21:11:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:11:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:11:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:11:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 0.750579833984375, 2024-02-19T21:11:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:11:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:11:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 45, 2024-02-19T21:11:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 444, 2024-02-19T21:11:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:11:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:11:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:11:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6808", 2024-02-19T21:11:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:11:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:11:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6809", 2024-02-19T21:11:12.834 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:11:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:11:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6811", 2024-02-19T21:11:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:11:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.835 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:11:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6810", 2024-02-19T21:11:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 787604615 2024-02-19T21:11:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6808/787604615", 2024-02-19T21:11:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6809/787604615", 2024-02-19T21:11:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6811/787604615", 2024-02-19T21:11:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6810/787604615", 2024-02-19T21:11:12.836 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:11:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:11:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:11:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-02-19T21:11:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "uuid": "9ebeff27-c2f8-4a66-923f-a0d7573ddd3f", 2024-02-19T21:11:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "up": 1, 2024-02-19T21:11:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "in": 1, 2024-02-19T21:11:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "weight": 1, 2024-02-19T21:11:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_affinity": 1, 2024-02-19T21:11:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_begin": 0, 2024-02-19T21:11:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "last_clean_end": 0, 2024-02-19T21:11:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "up_from": 53, 2024-02-19T21:11:12.837 INFO:teuthology.orchestra.run.smithi163.stdout: "up_thru": 444, 2024-02-19T21:11:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: "down_at": 0, 2024-02-19T21:11:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: "lost_at": 0, 2024-02-19T21:11:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addrs": { 2024-02-19T21:11:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6812", 2024-02-19T21:11:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:11:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addrs": { 2024-02-19T21:11:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.838 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6813", 2024-02-19T21:11:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:11:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addrs": { 2024-02-19T21:11:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6815", 2024-02-19T21:11:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:11:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.839 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addrs": { 2024-02-19T21:11:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: "addrvec": [ 2024-02-19T21:11:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2024-02-19T21:11:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.193:6814", 2024-02-19T21:11:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 157927432 2024-02-19T21:11:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: "public_addr": "172.21.15.193:6812/157927432", 2024-02-19T21:11:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: "cluster_addr": "172.21.15.193:6813/157927432", 2024-02-19T21:11:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_back_addr": "172.21.15.193:6815/157927432", 2024-02-19T21:11:12.840 INFO:teuthology.orchestra.run.smithi163.stdout: "heartbeat_front_addr": "172.21.15.193:6814/157927432", 2024-02-19T21:11:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "state": [ 2024-02-19T21:11:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "exists", 2024-02-19T21:11:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "up" 2024-02-19T21:11:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:11:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_xinfo": [ 2024-02-19T21:11:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 0, 2024-02-19T21:11:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:11:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:11:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:11:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:11:12.841 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:11:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:10:48.847238+0000", 2024-02-19T21:11:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:11:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 1, 2024-02-19T21:11:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:11:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:11:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:11:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:11:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:11:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:11:01.123445+0000", 2024-02-19T21:11:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:11:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.842 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 2, 2024-02-19T21:11:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:11:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:11:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:11:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:11:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:11:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:11:08.243708+0000", 2024-02-19T21:11:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:11:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 3, 2024-02-19T21:11:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:11:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:11:12.843 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:11:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:11:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:11:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:10:35.832944+0000", 2024-02-19T21:11:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:11:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 4, 2024-02-19T21:11:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:11:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:11:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:11:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:11:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:11:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:10:44.663496+0000", 2024-02-19T21:11:12.844 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:11:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 5, 2024-02-19T21:11:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:11:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:11:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:11:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:11:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:11:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:11:06.162575+0000", 2024-02-19T21:11:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:11:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 6, 2024-02-19T21:11:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:11:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:11:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:11:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:11:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:11:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:10:55.062276+0000", 2024-02-19T21:11:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:11:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "osd": 7, 2024-02-19T21:11:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "down_stamp": "0.000000", 2024-02-19T21:11:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_probability": 0, 2024-02-19T21:11:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "laggy_interval": 0, 2024-02-19T21:11:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "features": 4540138314316775423, 2024-02-19T21:11:12.846 INFO:teuthology.orchestra.run.smithi163.stdout: "old_weight": 0, 2024-02-19T21:11:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: "last_purged_snaps_scrub": "2024-02-19T21:11:02.923910+0000", 2024-02-19T21:11:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: "dead_epoch": 0 2024-02-19T21:11:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:11:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap": [], 2024-02-19T21:11:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_upmap_items": [ 2024-02-19T21:11:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "2.e", 2024-02-19T21:11:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:11:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 7, 2024-02-19T21:11:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 1 2024-02-19T21:11:12.847 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.7", 2024-02-19T21:11:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:11:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:11:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-02-19T21:11:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.d", 2024-02-19T21:11:12.848 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:11:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:11:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 5 2024-02-19T21:11:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.f", 2024-02-19T21:11:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:11:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 1, 2024-02-19T21:11:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-02-19T21:11:12.849 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.10", 2024-02-19T21:11:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:11:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 3, 2024-02-19T21:11:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 5 2024-02-19T21:11:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "4.17", 2024-02-19T21:11:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:11:12.850 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 3, 2024-02-19T21:11:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 2 2024-02-19T21:11:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: "pgid": "5.15", 2024-02-19T21:11:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: "mappings": [ 2024-02-19T21:11:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: "from": 0, 2024-02-19T21:11:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: "to": 3 2024-02-19T21:11:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.851 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:11:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "pg_temp": [], 2024-02-19T21:11:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "primary_temp": [], 2024-02-19T21:11:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "blocklist": { 2024-02-19T21:11:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1427774891": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:11:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/3282102947": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:11:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2735092549": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:11:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1235588846": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:11:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2731225015": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:11:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/506491651": "2024-02-20T20:56:02.185409+0000", 2024-02-19T21:11:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/168117826": "2024-02-20T21:01:38.896227+0000", 2024-02-19T21:11:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/1458009465": "2024-02-20T20:56:29.603030+0000", 2024-02-19T21:11:12.852 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:0/541631472": "2024-02-20T20:56:02.185409+0000", 2024-02-19T21:11:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "172.21.15.163:6800/2248999541": "2024-02-20T20:56:02.185409+0000" 2024-02-19T21:11:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "range_blocklist": {}, 2024-02-19T21:11:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "erasure_code_profiles": { 2024-02-19T21:11:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "default": { 2024-02-19T21:11:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "crush-failure-domain": "osd", 2024-02-19T21:11:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "k": "2", 2024-02-19T21:11:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "m": "1", 2024-02-19T21:11:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "plugin": "jerasure", 2024-02-19T21:11:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "ruleset-failure-domain": "osd", 2024-02-19T21:11:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "technique": "reed_sol_van" 2024-02-19T21:11:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: "removed_snaps_queue": [ 2024-02-19T21:11:12.853 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 11, 2024-02-19T21:11:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:11:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:11:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:11:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 9, 2024-02-19T21:11:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:11:12.854 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 25, 2024-02-19T21:11:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:11:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:11:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:11:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.855 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:11:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 57, 2024-02-19T21:11:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:11:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:11:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:11:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.856 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:11:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:11:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 76, 2024-02-19T21:11:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:11:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:11:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:11:12.857 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:11:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 85, 2024-02-19T21:11:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:11:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:11:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.858 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 92, 2024-02-19T21:11:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:11:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:11:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:11:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:11:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:11:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.859 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 94, 2024-02-19T21:11:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:11:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 4, 2024-02-19T21:11:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 4 2024-02-19T21:11:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 107, 2024-02-19T21:11:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:11:12.860 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:11:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 112, 2024-02-19T21:11:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:11:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 1, 2024-02-19T21:11:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.861 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 3, 2024-02-19T21:11:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 5, 2024-02-19T21:11:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 7, 2024-02-19T21:11:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.862 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 126, 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 2, 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 6, 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 10, 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 2 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.863 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 132, 2024-02-19T21:11:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:11:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 12, 2024-02-19T21:11:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 154, 2024-02-19T21:11:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:11:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.864 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 14, 2024-02-19T21:11:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 164, 2024-02-19T21:11:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:11:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 17, 2024-02-19T21:11:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:12.865 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: "pool": 178, 2024-02-19T21:11:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: "snaps": [ 2024-02-19T21:11:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: "begin": 20, 2024-02-19T21:11:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: "length": 1 2024-02-19T21:11:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: ], 2024-02-19T21:11:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: "new_removed_snaps": [], 2024-02-19T21:11:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: "new_purged_snaps": [], 2024-02-19T21:11:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: "crush_node_flags": {}, 2024-02-19T21:11:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: "device_class_flags": {}, 2024-02-19T21:11:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode": { 2024-02-19T21:11:12.866 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_enabled": false, 2024-02-19T21:11:12.867 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_bucket_count": 0, 2024-02-19T21:11:12.867 INFO:teuthology.orchestra.run.smithi163.stdout: "degraded_stretch_mode": 0, 2024-02-19T21:11:12.867 INFO:teuthology.orchestra.run.smithi163.stdout: "recovering_stretch_mode": 0, 2024-02-19T21:11:12.867 INFO:teuthology.orchestra.run.smithi163.stdout: "stretch_mode_bucket": 0 2024-02-19T21:11:12.867 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:12.867 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:11:12.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:12.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.757+0000 7f251de6a700 10 librados: watch_flush enter 2024-02-19T21:11:12.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.757+0000 7f251de6a700 10 librados: watch_flush exit 2024-02-19T21:11:12.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.757+0000 7f251de6a700 20 client.23214.objecter shutdown clearing up homeless session... 2024-02-19T21:11:12.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.757+0000 7f251de6a700 10 client.23214.objecter successfully canceled tick 2024-02-19T21:11:12.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.757+0000 7f251de6a700 1 -- 172.21.15.163:0/2170793919 >> v1:172.21.15.163:6800/1552514681 conn(0x7f2504063ca0 legacy=0x7f2504066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:12.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.757+0000 7f251de6a700 1 -- 172.21.15.163:0/2170793919 >> v1:172.21.15.163:6789/0 conn(0x7f25181042b0 legacy=0x7f2518115a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:12.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.757+0000 7f251de6a700 1 -- 172.21.15.163:0/2170793919 shutdown_connections 2024-02-19T21:11:12.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.757+0000 7f251de6a700 1 -- 172.21.15.163:0/2170793919 >> 172.21.15.163:0/2170793919 conn(0x7f25180f2470 msgr2=0x7f2518107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:12.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.757+0000 7f251de6a700 1 -- 172.21.15.163:0/2170793919 shutdown_connections 2024-02-19T21:11:12.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.757+0000 7f251de6a700 1 -- 172.21.15.163:0/2170793919 wait complete. 2024-02-19T21:11:12.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:12.757+0000 7f251de6a700 1 librados: shutdown 2024-02-19T21:11:12.870 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:12.870 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:12.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:12.971 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:12.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:11.229806+0000 mgr.y (mgr.14550) 414 : cluster [DBG] pgmap v671: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 402 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:11:12.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:11.976942+0000 mon.a (mon.0) 2624 : audit [DBG] from='client.? v1:172.21.15.163:0/45174354' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:12.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:11.229806+0000 mgr.y (mgr.14550) 414 : cluster [DBG] pgmap v671: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 402 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:11:12.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:11.976942+0000 mon.a (mon.0) 2624 : audit [DBG] from='client.? v1:172.21.15.163:0/45174354' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:12.998 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:13.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:11.229806+0000 mgr.y (mgr.14550) 414 : cluster [DBG] pgmap v671: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 402 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:11:13.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:11.976942+0000 mon.a (mon.0) 2624 : audit [DBG] from='client.? v1:172.21.15.163:0/45174354' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:13.197 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:13.199 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:13.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:13.301 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:11:13.328 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:13.693 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:11:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:11:13] "GET /metrics HTTP/1.1" 200 189845 "" "Prometheus/2.33.4" 2024-02-19T21:11:13.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:12.755319+0000 mon.a (mon.0) 2625 : audit [DBG] from='client.? v1:172.21.15.163:0/2170793919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:11:13.996 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:12.755319+0000 mon.a (mon.0) 2625 : audit [DBG] from='client.? v1:172.21.15.163:0/2170793919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:11:14.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:12.755319+0000 mon.a (mon.0) 2625 : audit [DBG] from='client.? v1:172.21.15.163:0/2170793919' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-19T21:11:14.469 INFO:tasks.thrashosds.thrasher:j is {'epoch': 454, 'fsid': '2591809a-cf69-11ee-95bb-87774f69a715', 'created': '2024-02-19T20:55:37.794885+0000', 'modified': '2024-02-19T21:11:08.679950+0000', 'last_up_change': '2024-02-19T21:01:09.214134+0000', 'last_in_change': '2024-02-19T21:10:40.268547+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 103, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 216, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'pacific', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2024-02-19T20:59:09.425169+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-02-19T21:01:14.658945+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '2.20', 'ready_epoch': 309, 'last_epoch_started': 308, 'last_epoch_clean': 308, 'source_version': "58'1", 'target_version': "58'1"}, 'last_change': '416', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '309', 'last_force_op_resend_preluminous': '307', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'iscsi', 'create_time': '2024-02-19T21:01:14.958800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-19T21:01:16.543816+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-19T21:01:18.521482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-19T21:01:20.586564+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 16, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-02-19T21:04:35.333192+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}}, {'pool': 61, 'pool_name': 'test-rados-api-smithi163-83928-6', 'create_time': '2024-02-19T21:04:52.991355+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '447', 'last_force_op_resend': '386', 'last_force_op_resend_prenautilus': '386', 'last_force_op_resend_preluminous': '386', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 32, 'snap_epoch': 447, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'fingerprint_algorithm': 'sha1'}, 'application_metadata': {'rados': {}}}, {'pool': 216, 'pool_name': 'test-rados-api-smithi163-83928-69', 'create_time': '2024-02-19T21:10:50.528170+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '446', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '4a14b406-7c21-40f3-8b12-8cf9df811088', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 218, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6801', 'nonce': 1559418115}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6802', 'nonce': 1559418115}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6804', 'nonce': 1559418115}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6803', 'nonce': 1559418115}]}, 'public_addr': '172.21.15.163:6801/1559418115', 'cluster_addr': '172.21.15.163:6802/1559418115', 'heartbeat_back_addr': '172.21.15.163:6804/1559418115', 'heartbeat_front_addr': '172.21.15.163:6803/1559418115', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'e74a65fa-cb9c-42c3-9293-602cf7e9dc67', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 444, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6805', 'nonce': 3125047432}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6806', 'nonce': 3125047432}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6808', 'nonce': 3125047432}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6807', 'nonce': 3125047432}]}, 'public_addr': '172.21.15.163:6805/3125047432', 'cluster_addr': '172.21.15.163:6806/3125047432', 'heartbeat_back_addr': '172.21.15.163:6808/3125047432', 'heartbeat_front_addr': '172.21.15.163:6807/3125047432', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6980a625-cdc9-487e-8f8c-7997ba716a97', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 449, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6809', 'nonce': 1563158487}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6810', 'nonce': 1563158487}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6812', 'nonce': 1563158487}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6811', 'nonce': 1563158487}]}, 'public_addr': '172.21.15.163:6809/1563158487', 'cluster_addr': '172.21.15.163:6810/1563158487', 'heartbeat_back_addr': '172.21.15.163:6812/1563158487', 'heartbeat_front_addr': '172.21.15.163:6811/1563158487', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '359e2ee0-9d1d-4074-8119-2d69bc2ca3a5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 444, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6813', 'nonce': 2112279769}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6814', 'nonce': 2112279769}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6816', 'nonce': 2112279769}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.163:6815', 'nonce': 2112279769}]}, 'public_addr': '172.21.15.163:6813/2112279769', 'cluster_addr': '172.21.15.163:6814/2112279769', 'heartbeat_back_addr': '172.21.15.163:6816/2112279769', 'heartbeat_front_addr': '172.21.15.163:6815/2112279769', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2bf5ef05-292b-4693-85a9-97991f2091e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 444, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6800', 'nonce': 3129716965}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6801', 'nonce': 3129716965}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6803', 'nonce': 3129716965}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6802', 'nonce': 3129716965}]}, 'public_addr': '172.21.15.193:6800/3129716965', 'cluster_addr': '172.21.15.193:6801/3129716965', 'heartbeat_back_addr': '172.21.15.193:6803/3129716965', 'heartbeat_front_addr': '172.21.15.193:6802/3129716965', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '59065f9b-9240-414b-9c91-e91cda4aed6f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 37, 'up_thru': 444, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6804', 'nonce': 805161991}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6805', 'nonce': 805161991}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6807', 'nonce': 805161991}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6806', 'nonce': 805161991}]}, 'public_addr': '172.21.15.193:6804/805161991', 'cluster_addr': '172.21.15.193:6805/805161991', 'heartbeat_back_addr': '172.21.15.193:6807/805161991', 'heartbeat_front_addr': '172.21.15.193:6806/805161991', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '302c95df-3842-4818-8afd-d79d8184684f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.750579833984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 444, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6808', 'nonce': 787604615}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6809', 'nonce': 787604615}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6811', 'nonce': 787604615}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6810', 'nonce': 787604615}]}, 'public_addr': '172.21.15.193:6808/787604615', 'cluster_addr': '172.21.15.193:6809/787604615', 'heartbeat_back_addr': '172.21.15.193:6811/787604615', 'heartbeat_front_addr': '172.21.15.193:6810/787604615', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '9ebeff27-c2f8-4a66-923f-a0d7573ddd3f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 53, 'up_thru': 444, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6812', 'nonce': 157927432}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6813', 'nonce': 157927432}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6815', 'nonce': 157927432}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.193:6814', 'nonce': 157927432}]}, 'public_addr': '172.21.15.193:6812/157927432', 'cluster_addr': '172.21.15.193:6813/157927432', 'heartbeat_back_addr': '172.21.15.193:6815/157927432', 'heartbeat_front_addr': '172.21.15.193:6814/157927432', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:10:48.847238+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:11:01.123445+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:11:08.243708+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:10:35.832944+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:10:44.663496+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:11:06.162575+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:10:55.062276+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138314316775423, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-19T21:11:02.923910+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.e', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.d', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.10', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.17', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '5.15', 'mappings': [{'from': 0, 'to': 3}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.163:0/1427774891': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/3282102947': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:6800/2735092549': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/1235588846': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:6800/2731225015': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:0/506491651': '2024-02-20T20:56:02.185409+0000', '172.21.15.163:0/168117826': '2024-02-20T21:01:38.896227+0000', '172.21.15.163:0/1458009465': '2024-02-20T20:56:29.603030+0000', '172.21.15.163:0/541631472': '2024-02-20T20:56:02.185409+0000', '172.21.15.163:6800/2248999541': '2024-02-20T20:56:02.185409+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 11, 'snaps': [{'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}, {'begin': 9, 'length': 2}]}, {'pool': 25, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 57, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 76, 'snaps': [{'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 85, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 92, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}]}, {'pool': 94, 'snaps': [{'begin': 4, 'length': 4}]}, {'pool': 107, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 112, 'snaps': [{'begin': 1, 'length': 1}, {'begin': 3, 'length': 1}, {'begin': 5, 'length': 1}, {'begin': 7, 'length': 1}]}, {'pool': 126, 'snaps': [{'begin': 2, 'length': 2}, {'begin': 6, 'length': 2}, {'begin': 10, 'length': 2}]}, {'pool': 132, 'snaps': [{'begin': 12, 'length': 1}]}, {'pool': 154, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 164, 'snaps': [{'begin': 17, 'length': 1}]}, {'pool': 178, 'snaps': [{'begin': 20, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-19T21:11:14.470 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:11:14.477 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:11:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:14.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:11:14.581 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:11:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:14.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:13.230166+0000 mgr.y (mgr.14550) 415 : cluster [DBG] pgmap v672: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:11:14.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:14.454964+0000 mon.b (mon.1) 607 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:14.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:14.455339+0000 mon.a (mon.0) 2626 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:14.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:14.455815+0000 mon.b (mon.1) 608 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-69"}]: dispatch 2024-02-19T21:11:14.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:14.455986+0000 mon.a (mon.0) 2627 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-69"}]: dispatch 2024-02-19T21:11:14.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:13.230166+0000 mgr.y (mgr.14550) 415 : cluster [DBG] pgmap v672: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:11:14.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:14.454964+0000 mon.b (mon.1) 607 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:14.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:14.455339+0000 mon.a (mon.0) 2626 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:14.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:14.455815+0000 mon.b (mon.1) 608 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-69"}]: dispatch 2024-02-19T21:11:14.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:14.455986+0000 mon.a (mon.0) 2627 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-69"}]: dispatch 2024-02-19T21:11:15.062 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:11:15.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:15.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:11:15.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:13.230166+0000 mgr.y (mgr.14550) 415 : cluster [DBG] pgmap v672: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:11:15.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:14.454964+0000 mon.b (mon.1) 607 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:15.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:14.455339+0000 mon.a (mon.0) 2626 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:15.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:14.455815+0000 mon.b (mon.1) 608 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-69"}]: dispatch 2024-02-19T21:11:15.069 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:14.455986+0000 mon.a (mon.0) 2627 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-69"}]: dispatch 2024-02-19T21:11:15.166 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:15.190 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:11:15.198 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:15.219 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:15.223 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:15.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:15.251 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:15.324 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:15.352 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:15.927 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:11:15.927 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:11:16.105 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:16.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:16.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:16.209 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:11:16.241 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:16.263 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:11:16.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:16.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:11:16.367 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:16.394 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:15.231001+0000 mgr.y (mgr.14550) 416 : cluster [DBG] pgmap v673: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:11:16.394 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:15.285981+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e455: 8 total, 8 up, 8 in 2024-02-19T21:11:16.394 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:15.231001+0000 mgr.y (mgr.14550) 416 : cluster [DBG] pgmap v673: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:11:16.394 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:15.285981+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e455: 8 total, 8 up, 8 in 2024-02-19T21:11:16.396 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:16.544 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:16.547 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:16.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:16.553 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:16.556 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-19T21:11:16.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:15.231001+0000 mgr.y (mgr.14550) 416 : cluster [DBG] pgmap v673: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:11:16.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:15.285981+0000 mon.a (mon.0) 2628 : cluster [DBG] osdmap e455: 8 total, 8 up, 8 in 2024-02-19T21:11:16.649 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:16.681 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:16.703 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:16.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:16.806 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:16.832 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:16.860 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:16.863 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:16.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:16.964 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:11:16.996 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:17.017 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:11:17.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:17.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:11:17.121 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:11:17.151 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:17.171 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:11:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:17.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:11:17.274 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:17.293 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:16.292576+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e456: 8 total, 8 up, 8 in 2024-02-19T21:11:17.293 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:16.295046+0000 mon.b (mon.1) 609 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:17.294 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:16.295795+0000 mon.a (mon.0) 2630 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:17.310 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:17.337 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:17.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:17.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:17.441 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:17.470 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:17.490 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:17.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:17.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:17.565 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:16.292576+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e456: 8 total, 8 up, 8 in 2024-02-19T21:11:17.565 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:16.295046+0000 mon.b (mon.1) 609 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:17.565 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:16.295795+0000 mon.a (mon.0) 2630 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:17.565 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:16.292576+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e456: 8 total, 8 up, 8 in 2024-02-19T21:11:17.565 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:16.295046+0000 mon.b (mon.1) 609 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:17.565 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:16.295795+0000 mon.a (mon.0) 2630 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:17.594 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:17.622 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:17.641 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:17.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:17.745 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:11:17.772 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:17.791 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:11:17.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:17.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:11:17.895 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:11:17.923 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:18.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.841+0000 7f3d56028700 1 -- 172.21.15.163:0/1506965972 >> v1:172.21.15.163:6790/0 conn(0x7f3d50100780 legacy=0x7f3d50102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:18.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.842+0000 7f3d56028700 1 -- 172.21.15.163:0/1506965972 shutdown_connections 2024-02-19T21:11:18.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.842+0000 7f3d56028700 1 -- 172.21.15.163:0/1506965972 >> 172.21.15.163:0/1506965972 conn(0x7f3d500f35a0 msgr2=0x7f3d500f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:18.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.842+0000 7f3d56028700 1 -- 172.21.15.163:0/1506965972 shutdown_connections 2024-02-19T21:11:18.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.842+0000 7f3d56028700 1 -- 172.21.15.163:0/1506965972 wait complete. 2024-02-19T21:11:18.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.842+0000 7f3d56028700 1 librados: starting msgr at 2024-02-19T21:11:18.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.842+0000 7f3d56028700 1 librados: starting objecter 2024-02-19T21:11:18.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.842+0000 7f3d56028700 1 Processor -- start 2024-02-19T21:11:18.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.842+0000 7f3d56028700 1 -- start start 2024-02-19T21:11:18.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.842+0000 7f3d56028700 1 librados: setting wanted keys 2024-02-19T21:11:18.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.842+0000 7f3d56028700 1 librados: calling monclient init 2024-02-19T21:11:18.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d56028700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3d501172c0 con 0x7f3d50100780 2024-02-19T21:11:18.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d56028700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3d501184a0 con 0x7f3d501042c0 2024-02-19T21:11:18.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d56028700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3d501196a0 con 0x7f3d500fd980 2024-02-19T21:11:18.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4f7fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f3d500fd980 0x7f3d500f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:41434/0 (socket says 172.21.15.163:41434) 2024-02-19T21:11:18.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4f7fe700 1 -- 172.21.15.163:0/3261517312 learned_addr learned my addr 172.21.15.163:0/3261517312 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:18.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4effd700 1 --1- 172.21.15.163:0/3261517312 >> v1:172.21.15.163:6789/0 conn(0x7f3d50100780 0x7f3d500fa070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:48630/0 (socket says 172.21.15.163:48630) 2024-02-19T21:11:18.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 10 client.?.objecter ms_handle_connect 0x7f3d500fd980 2024-02-19T21:11:18.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:18.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 10 client.?.objecter ms_handle_connect 0x7f3d50100780 2024-02-19T21:11:18.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:18.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172075273 0 0) 0x7f3d501172c0 con 0x7f3d50100780 2024-02-19T21:11:18.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d38003800 con 0x7f3d50100780 2024-02-19T21:11:18.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563701509 0 0) 0x7f3d501196a0 con 0x7f3d500fd980 2024-02-19T21:11:18.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d501172c0 con 0x7f3d500fd980 2024-02-19T21:11:18.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 10 client.?.objecter ms_handle_connect 0x7f3d501042c0 2024-02-19T21:11:18.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:18.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 659760249 0 0) 0x7f3d38003800 con 0x7f3d50100780 2024-02-19T21:11:18.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d501196a0 con 0x7f3d50100780 2024-02-19T21:11:18.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3d44002f10 con 0x7f3d50100780 2024-02-19T21:11:18.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1776938205 0 0) 0x7f3d501172c0 con 0x7f3d500fd980 2024-02-19T21:11:18.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d38003800 con 0x7f3d500fd980 2024-02-19T21:11:18.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077648756 0 0) 0x7f3d501184a0 con 0x7f3d501042c0 2024-02-19T21:11:18.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d501172c0 con 0x7f3d501042c0 2024-02-19T21:11:18.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3d400019f0 con 0x7f3d500fd980 2024-02-19T21:11:18.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1678411602 0 0) 0x7f3d501196a0 con 0x7f3d50100780 2024-02-19T21:11:18.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 >> v1:172.21.15.163:6790/0 conn(0x7f3d500fd980 legacy=0x7f3d500f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:18.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 >> v1:172.21.15.193:6789/0 conn(0x7f3d501042c0 legacy=0x7f3d50115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:18.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5011a8a0 con 0x7f3d50100780 2024-02-19T21:11:18.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d56028700 1 -- 172.21.15.163:0/3261517312 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d501174b0 con 0x7f3d50100780 2024-02-19T21:11:18.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d56028700 10 client.23220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:18.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d56028700 1 -- 172.21.15.163:0/3261517312 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d50117920 con 0x7f3d50100780 2024-02-19T21:11:18.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.843+0000 7f3d56028700 1 librados: init done 2024-02-19T21:11:18.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.844+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3d44001b40 con 0x7f3d50100780 2024-02-19T21:11:18.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.844+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3d44003850 con 0x7f3d50100780 2024-02-19T21:11:18.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.844+0000 7f3d56028700 1 -- 172.21.15.163:0/3261517312 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3c004f20 con 0x7f3d50100780 2024-02-19T21:11:18.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.845+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f3d440039d0 con 0x7f3d50100780 2024-02-19T21:11:18.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.845+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(457..457 src has 43..457) v4 ==== 8063+0+0 (unknown 916744928 0 0) 0x7f3d44078e90 con 0x7f3d50100780 2024-02-19T21:11:18.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.845+0000 7f3d4cff9700 10 client.23220.objecter ms_dispatch 0x7f3d50001550 osd_map(457..457 src has 43..457) v4 2024-02-19T21:11:18.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.845+0000 7f3d4cff9700 3 client.23220.objecter handle_osd_map got epochs [457,457] > 0 2024-02-19T21:11:18.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.845+0000 7f3d4cff9700 3 client.23220.objecter handle_osd_map decoding full epoch 457 2024-02-19T21:11:18.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.845+0000 7f3d4cff9700 20 client.23220.objecter dump_active .. 0 homeless 2024-02-19T21:11:18.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.845+0000 7f3d4cff9700 10 client.23220.objecter ms_handle_connect 0x7f3d38063d00 2024-02-19T21:11:18.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:17.849+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3d4404c3a0 con 0x7f3d50100780 2024-02-19T21:11:18.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:18.037+0000 7f3d56028700 1 -- 172.21.15.163:0/3261517312 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f3d3c0051d0 con 0x7f3d50100780 2024-02-19T21:11:18.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:11:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:11:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:11:18.079 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:11:18.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:18.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:11:18.183 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:18.212 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:18.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:18.303+0000 7f3d4cff9700 1 -- 172.21.15.163:0/3261517312 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v458) v1 ==== 97+0+0 (unknown 660903667 0 0) 0x7f3d4404a810 con 0x7f3d50100780 2024-02-19T21:11:18.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:18.306+0000 7f3d56028700 10 librados: watch_flush enter 2024-02-19T21:11:18.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:18.306+0000 7f3d56028700 10 librados: watch_flush exit 2024-02-19T21:11:18.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:18.306+0000 7f3d56028700 20 client.23220.objecter shutdown clearing up homeless session... 2024-02-19T21:11:18.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:18.306+0000 7f3d56028700 10 client.23220.objecter successfully canceled tick 2024-02-19T21:11:18.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:18.306+0000 7f3d56028700 1 -- 172.21.15.163:0/3261517312 >> v1:172.21.15.163:6800/1552514681 conn(0x7f3d38063d00 legacy=0x7f3d380661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:18.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:18.306+0000 7f3d56028700 1 -- 172.21.15.163:0/3261517312 >> v1:172.21.15.163:6789/0 conn(0x7f3d50100780 legacy=0x7f3d500fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:18.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:18.306+0000 7f3d56028700 1 -- 172.21.15.163:0/3261517312 shutdown_connections 2024-02-19T21:11:18.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:18.306+0000 7f3d56028700 1 -- 172.21.15.163:0/3261517312 >> 172.21.15.163:0/3261517312 conn(0x7f3d500f35a0 msgr2=0x7f3d50107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:18.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:18.306+0000 7f3d56028700 1 -- 172.21.15.163:0/3261517312 shutdown_connections 2024-02-19T21:11:18.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:18.306+0000 7f3d56028700 1 -- 172.21.15.163:0/3261517312 wait complete. 2024-02-19T21:11:18.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:18.306+0000 7f3d56028700 1 librados: shutdown 2024-02-19T21:11:18.306 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:11:18.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:17.231496+0000 mgr.y (mgr.14550) 417 : cluster [DBG] pgmap v676: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:11:18.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:17.297389+0000 mon.a (mon.0) 2631 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:11:18.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:17.297517+0000 mon.a (mon.0) 2632 : cluster [DBG] osdmap e457: 8 total, 8 up, 8 in 2024-02-19T21:11:18.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:17.300551+0000 mon.b (mon.1) 610 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:11:18.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:18.039323+0000 mon.a (mon.0) 2633 : audit [INF] from='client.? v1:172.21.15.163:0/3261517312' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:11:18.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:17.231496+0000 mgr.y (mgr.14550) 417 : cluster [DBG] pgmap v676: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:11:18.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:17.297389+0000 mon.a (mon.0) 2631 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:11:18.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:17.297517+0000 mon.a (mon.0) 2632 : cluster [DBG] osdmap e457: 8 total, 8 up, 8 in 2024-02-19T21:11:18.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:17.300551+0000 mon.b (mon.1) 610 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:11:18.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:18.039323+0000 mon.a (mon.0) 2633 : audit [INF] from='client.? v1:172.21.15.163:0/3261517312' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:11:18.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:17.231496+0000 mgr.y (mgr.14550) 417 : cluster [DBG] pgmap v676: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:11:18.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:17.297389+0000 mon.a (mon.0) 2631 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:11:18.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:17.297517+0000 mon.a (mon.0) 2632 : cluster [DBG] osdmap e457: 8 total, 8 up, 8 in 2024-02-19T21:11:18.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:17.300551+0000 mon.b (mon.1) 610 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:11:18.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:18.039323+0000 mon.a (mon.0) 2633 : audit [INF] from='client.? v1:172.21.15.163:0/3261517312' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:11:19.460 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:18.294452+0000 mon.a (mon.0) 2634 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:11:19.461 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:18.304795+0000 mon.a (mon.0) 2635 : audit [INF] from='client.? v1:172.21.15.163:0/3261517312' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:11:19.461 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:18.304857+0000 mon.a (mon.0) 2636 : cluster [DBG] osdmap e458: 8 total, 8 up, 8 in 2024-02-19T21:11:19.461 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:18.294452+0000 mon.a (mon.0) 2634 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:11:19.461 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:18.304795+0000 mon.a (mon.0) 2635 : audit [INF] from='client.? v1:172.21.15.163:0/3261517312' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:11:19.461 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:18.304857+0000 mon.a (mon.0) 2636 : cluster [DBG] osdmap e458: 8 total, 8 up, 8 in 2024-02-19T21:11:19.461 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:19.465 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:19.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:19.567 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:11:19.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:18.294452+0000 mon.a (mon.0) 2634 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:11:19.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:18.304795+0000 mon.a (mon.0) 2635 : audit [INF] from='client.? v1:172.21.15.163:0/3261517312' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:11:19.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:18.304857+0000 mon.a (mon.0) 2636 : cluster [DBG] osdmap e458: 8 total, 8 up, 8 in 2024-02-19T21:11:19.597 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:19.719 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:11:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:19.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:11:19.823 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:19.850 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:20.050 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:20.052 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:20.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:20.153 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:11:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.817+0000 7f765ac3a700 1 -- 172.21.15.163:0/3218957835 >> v1:172.21.15.163:6789/0 conn(0x7f76541042c0 legacy=0x7f7654106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.817+0000 7f765ac3a700 1 -- 172.21.15.163:0/3218957835 shutdown_connections 2024-02-19T21:11:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.817+0000 7f765ac3a700 1 -- 172.21.15.163:0/3218957835 >> 172.21.15.163:0/3218957835 conn(0x7f76540f35a0 msgr2=0x7f76540f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.817+0000 7f765ac3a700 1 -- 172.21.15.163:0/3218957835 shutdown_connections 2024-02-19T21:11:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.817+0000 7f765ac3a700 1 -- 172.21.15.163:0/3218957835 wait complete. 2024-02-19T21:11:20.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.817+0000 7f765ac3a700 1 librados: starting msgr at 2024-02-19T21:11:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.817+0000 7f765ac3a700 1 librados: starting objecter 2024-02-19T21:11:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.817+0000 7f765ac3a700 1 Processor -- start 2024-02-19T21:11:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.817+0000 7f765ac3a700 1 -- start start 2024-02-19T21:11:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.817+0000 7f765ac3a700 1 librados: setting wanted keys 2024-02-19T21:11:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.817+0000 7f765ac3a700 1 librados: calling monclient init 2024-02-19T21:11:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f765ac3a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76541172e0 con 0x7f76540fd980 2024-02-19T21:11:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f765ac3a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76541184c0 con 0x7f7654100780 2024-02-19T21:11:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f765ac3a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f76541196c0 con 0x7f76541042c0 2024-02-19T21:11:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f76589d6700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f76540fd980 0x7f76540f98c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:48656/0 (socket says 172.21.15.163:48656) 2024-02-19T21:11:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f76589d6700 1 -- 172.21.15.163:0/53644826 learned_addr learned my addr 172.21.15.163:0/53644826 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 10 client.?.objecter ms_handle_connect 0x7f76541042c0 2024-02-19T21:11:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 10 client.?.objecter ms_handle_connect 0x7f76540fd980 2024-02-19T21:11:20.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 10 client.?.objecter ms_handle_connect 0x7f7654100780 2024-02-19T21:11:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690718378 0 0) 0x7f76541196c0 con 0x7f76541042c0 2024-02-19T21:11:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7644003030 con 0x7f76541042c0 2024-02-19T21:11:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772666954 0 0) 0x7f76541172e0 con 0x7f76540fd980 2024-02-19T21:11:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76541196c0 con 0x7f76540fd980 2024-02-19T21:11:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568333054 0 0) 0x7f76541184c0 con 0x7f7654100780 2024-02-19T21:11:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76541172e0 con 0x7f7654100780 2024-02-19T21:11:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2425091078 0 0) 0x7f7644003030 con 0x7f76541042c0 2024-02-19T21:11:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76541184c0 con 0x7f76541042c0 2024-02-19T21:11:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f764c0017b0 con 0x7f76541042c0 2024-02-19T21:11:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1486047076 0 0) 0x7f76541196c0 con 0x7f76540fd980 2024-02-19T21:11:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7644003030 con 0x7f76540fd980 2024-02-19T21:11:20.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f76480019f0 con 0x7f76540fd980 2024-02-19T21:11:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3028560553 0 0) 0x7f76541172e0 con 0x7f7654100780 2024-02-19T21:11:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76541196c0 con 0x7f7654100780 2024-02-19T21:11:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f76400018c0 con 0x7f7654100780 2024-02-19T21:11:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1151993714 0 0) 0x7f76541184c0 con 0x7f76541042c0 2024-02-19T21:11:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 >> v1:172.21.15.193:6789/0 conn(0x7f7654100780 legacy=0x7f76540f9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 >> v1:172.21.15.163:6789/0 conn(0x7f76540fd980 legacy=0x7f76540f98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765411a8c0 con 0x7f76541042c0 2024-02-19T21:11:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.818+0000 7f765ac3a700 1 -- 172.21.15.163:0/53644826 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76541174d0 con 0x7f76541042c0 2024-02-19T21:11:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.819+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f764c0019d0 con 0x7f76541042c0 2024-02-19T21:11:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.819+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f764c003a70 con 0x7f76541042c0 2024-02-19T21:11:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.819+0000 7f765ac3a700 10 client.49946.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.819+0000 7f765ac3a700 1 -- 172.21.15.163:0/53644826 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76541179c0 con 0x7f76541042c0 2024-02-19T21:11:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.819+0000 7f765ac3a700 1 librados: init done 2024-02-19T21:11:20.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.820+0000 7f765ac3a700 1 -- 172.21.15.163:0/53644826 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f765405f660 con 0x7f76541042c0 2024-02-19T21:11:20.176 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:11:20.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.820+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f764c001bc0 con 0x7f76541042c0 2024-02-19T21:11:20.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.820+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(459..459 src has 43..459) v4 ==== 8063+0+0 (unknown 2263156180 0 0) 0x7f764c076b10 con 0x7f76541042c0 2024-02-19T21:11:20.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.820+0000 7f7651ffb700 10 client.49946.objecter ms_dispatch 0x7f7654001550 osd_map(459..459 src has 43..459) v4 2024-02-19T21:11:20.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.820+0000 7f7651ffb700 3 client.49946.objecter handle_osd_map got epochs [459,459] > 0 2024-02-19T21:11:20.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.820+0000 7f7651ffb700 3 client.49946.objecter handle_osd_map decoding full epoch 459 2024-02-19T21:11:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.820+0000 7f7651ffb700 20 client.49946.objecter dump_active .. 0 homeless 2024-02-19T21:11:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.820+0000 7f7651ffb700 10 client.49946.objecter ms_handle_connect 0x7f7644063cf0 2024-02-19T21:11:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.825+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f764c049f60 con 0x7f76541042c0 2024-02-19T21:11:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:19.992+0000 7f765ac3a700 1 -- 172.21.15.163:0/53644826 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7654109ff0 con 0x7f7644063cf0 2024-02-19T21:11:20.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:20.008+0000 7f7651ffb700 1 -- 172.21.15.163:0/53644826 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+576970 (unknown 2965378022 0 3238406163) 0x7f7654109ff0 con 0x7f7644063cf0 2024-02-19T21:11:20.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:20.177+0000 7f765ac3a700 10 librados: watch_flush enter 2024-02-19T21:11:20.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:20.177+0000 7f765ac3a700 10 librados: watch_flush exit 2024-02-19T21:11:20.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:20.177+0000 7f765ac3a700 20 client.49946.objecter shutdown clearing up homeless session... 2024-02-19T21:11:20.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:20.177+0000 7f765ac3a700 10 client.49946.objecter successfully canceled tick 2024-02-19T21:11:20.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:20.177+0000 7f765ac3a700 1 -- 172.21.15.163:0/53644826 >> v1:172.21.15.163:6800/1552514681 conn(0x7f7644063cf0 legacy=0x7f76440661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:20.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:20.177+0000 7f765ac3a700 1 -- 172.21.15.163:0/53644826 >> v1:172.21.15.163:6790/0 conn(0x7f76541042c0 legacy=0x7f76541159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:20.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:20.177+0000 7f765ac3a700 1 -- 172.21.15.163:0/53644826 shutdown_connections 2024-02-19T21:11:20.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:20.177+0000 7f765ac3a700 1 -- 172.21.15.163:0/53644826 >> 172.21.15.163:0/53644826 conn(0x7f76540f35a0 msgr2=0x7f7654107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:20.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:20.177+0000 7f765ac3a700 1 -- 172.21.15.163:0/53644826 shutdown_connections 2024-02-19T21:11:20.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:20.177+0000 7f765ac3a700 1 -- 172.21.15.163:0/53644826 wait complete. 2024-02-19T21:11:20.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:20.177+0000 7f765ac3a700 1 librados: shutdown 2024-02-19T21:11:20.194 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:11:20.195 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:20.208 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:11:20.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:20.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:11:20.309 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:20.311 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:11:20.341 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:20.459 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:19.231962+0000 mgr.y (mgr.14550) 418 : cluster [DBG] pgmap v679: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:11:20.460 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:19.315732+0000 mon.a (mon.0) 2637 : cluster [DBG] osdmap e459: 8 total, 8 up, 8 in 2024-02-19T21:11:20.460 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:19.992760+0000 mgr.y (mgr.14550) 419 : audit [DBG] from='client.49946 v1:172.21.15.163:0/53644826' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:11:20.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:19.231962+0000 mgr.y (mgr.14550) 418 : cluster [DBG] pgmap v679: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:11:20.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:19.315732+0000 mon.a (mon.0) 2637 : cluster [DBG] osdmap e459: 8 total, 8 up, 8 in 2024-02-19T21:11:20.576 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:19.992760+0000 mgr.y (mgr.14550) 419 : audit [DBG] from='client.49946 v1:172.21.15.163:0/53644826' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:11:20.576 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:19.231962+0000 mgr.y (mgr.14550) 418 : cluster [DBG] pgmap v679: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:11:20.576 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:19.315732+0000 mon.a (mon.0) 2637 : cluster [DBG] osdmap e459: 8 total, 8 up, 8 in 2024-02-19T21:11:20.576 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:19.992760+0000 mgr.y (mgr.14550) 419 : audit [DBG] from='client.49946 v1:172.21.15.163:0/53644826' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:11:20.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:11:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:11:20.945 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":680,"stamp":"2024-02-19T21:11:19.309700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007339,"num_objects":358,"num_object_clones":5,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":1397,"num_read_kb":1050,"num_write":1048,"num_write_kb":4560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1004,"num_bytes_recovered":8733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":12,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2452,"ondisk_log_size":2452,"up":588,"acting":588,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":684,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2479028,"kb_used_data":68428,"kb_used_omap":0,"kb_used_meta":2410496,"kb_avail":747482188,"statfs":{"total":767960285184,"available":765421760512,"internally_reserved":0,"allocated":70070272,"data_stored":45464681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2468347904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.023244"},"pg_stats":[{"pgid":"217.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"217.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:16.283520+0000","last_change":"2024-02-19T21:11:16.283520+0000","last_active":"2024-02-19T21:11:16.283520+0000","last_peered":"2024-02-19T21:11:16.283520+0000","last_clean":"2024-02-19T21:11:16.283520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:16.283520+0000","last_undegraded":"2024-02-19T21:11:16.283520+0000","last_fullsized":"2024-02-19T21:11:16.283520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:16.283520+0000","last_clean_scrub_stamp":"2024-02-19T21:11:16.283520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.2","version":"452'102","reported_seq":259,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683110+0000","last_change":"2024-02-19T21:10:31.528468+0000","last_active":"2024-02-19T21:11:06.683110+0000","last_peered":"2024-02-19T21:11:06.683110+0000","last_clean":"2024-02-19T21:11:06.683110+0000","last_became_active":"2024-02-19T21:08:04.247600+0000","last_became_peered":"2024-02-19T21:08:04.247600+0000","last_unstale":"2024-02-19T21:11:06.683110+0000","last_undegraded":"2024-02-19T21:11:06.683110+0000","last_fullsized":"2024-02-19T21:11:06.683110+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'85","last_scrub_stamp":"2024-02-19T21:08:55.882988+0000","last_deep_scrub":"355'85","last_deep_scrub_stamp":"2024-02-19T21:08:55.882988+0000","last_clean_scrub_stamp":"2024-02-19T21:08:55.882988+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.3","version":"338'71","reported_seq":477,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.901678+0000","last_change":"2024-02-19T21:11:06.901648+0000","last_active":"2024-02-19T21:11:06.901678+0000","last_peered":"2024-02-19T21:11:06.901678+0000","last_clean":"2024-02-19T21:11:06.901678+0000","last_became_active":"2024-02-19T21:04:54.641884+0000","last_became_peered":"2024-02-19T21:04:54.641884+0000","last_unstale":"2024-02-19T21:11:06.901678+0000","last_undegraded":"2024-02-19T21:11:06.901678+0000","last_fullsized":"2024-02-19T21:11:06.901678+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'71","last_scrub_stamp":"2024-02-19T21:11:06.901567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:06.901567+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000068,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":23,"num_write_kb":3929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.0","version":"257'35","reported_seq":419,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.897786+0000","last_change":"2024-02-19T21:10:31.531190+0000","last_active":"2024-02-19T21:11:06.897786+0000","last_peered":"2024-02-19T21:11:06.897786+0000","last_clean":"2024-02-19T21:11:06.897786+0000","last_became_active":"2024-02-19T21:06:40.245157+0000","last_became_peered":"2024-02-19T21:06:40.245157+0000","last_unstale":"2024-02-19T21:11:06.897786+0000","last_undegraded":"2024-02-19T21:11:06.897786+0000","last_fullsized":"2024-02-19T21:11:06.897786+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'35","last_scrub_stamp":"2024-02-19T21:08:55.692313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:55.692313+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1","version":"338'69","reported_seq":582,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898081+0000","last_change":"2024-02-19T21:10:54.672773+0000","last_active":"2024-02-19T21:11:06.898081+0000","last_peered":"2024-02-19T21:11:06.898081+0000","last_clean":"2024-02-19T21:11:06.898081+0000","last_became_active":"2024-02-19T21:10:41.311298+0000","last_became_peered":"2024-02-19T21:10:41.311298+0000","last_unstale":"2024-02-19T21:11:06.898081+0000","last_undegraded":"2024-02-19T21:11:06.898081+0000","last_fullsized":"2024-02-19T21:11:06.898081+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-19T21:10:54.672705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.672705+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.6","version":"202'34","reported_seq":439,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681997+0000","last_change":"2024-02-19T21:10:31.536500+0000","last_active":"2024-02-19T21:11:06.681997+0000","last_peered":"2024-02-19T21:11:06.681997+0000","last_clean":"2024-02-19T21:11:06.681997+0000","last_became_active":"2024-02-19T21:04:54.658542+0000","last_became_peered":"2024-02-19T21:04:54.658542+0000","last_unstale":"2024-02-19T21:11:06.681997+0000","last_undegraded":"2024-02-19T21:11:06.681997+0000","last_fullsized":"2024-02-19T21:11:06.681997+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'34","last_scrub_stamp":"2024-02-19T21:08:58.036955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:58.036955+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.7","version":"396'20","reported_seq":439,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.059210+0000","last_change":"2024-02-19T21:11:05.030418+0000","last_active":"2024-02-19T21:11:07.059210+0000","last_peered":"2024-02-19T21:11:07.059210+0000","last_clean":"2024-02-19T21:11:07.059210+0000","last_became_active":"2024-02-19T21:04:54.598061+0000","last_became_peered":"2024-02-19T21:04:54.598061+0000","last_unstale":"2024-02-19T21:11:07.059210+0000","last_undegraded":"2024-02-19T21:11:07.059210+0000","last_fullsized":"2024-02-19T21:11:07.059210+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'20","last_scrub_stamp":"2024-02-19T21:11:05.030354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.030354+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.4","version":"403'70","reported_seq":581,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681858+0000","last_change":"2024-02-19T21:11:01.162809+0000","last_active":"2024-02-19T21:11:06.681858+0000","last_peered":"2024-02-19T21:11:06.681858+0000","last_clean":"2024-02-19T21:11:06.681858+0000","last_became_active":"2024-02-19T21:10:41.314275+0000","last_became_peered":"2024-02-19T21:10:41.314275+0000","last_unstale":"2024-02-19T21:11:06.681858+0000","last_undegraded":"2024-02-19T21:11:06.681858+0000","last_fullsized":"2024-02-19T21:11:06.681858+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'70","last_scrub_stamp":"2024-02-19T21:11:01.162737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.162737+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":96,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.5","version":"338'121","reported_seq":510,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898966+0000","last_change":"2024-02-19T21:10:31.535793+0000","last_active":"2024-02-19T21:11:06.898966+0000","last_peered":"2024-02-19T21:11:06.898966+0000","last_clean":"2024-02-19T21:11:06.898966+0000","last_became_active":"2024-02-19T21:04:54.646842+0000","last_became_peered":"2024-02-19T21:04:54.646842+0000","last_unstale":"2024-02-19T21:11:06.898966+0000","last_undegraded":"2024-02-19T21:11:06.898966+0000","last_fullsized":"2024-02-19T21:11:06.898966+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'121","last_scrub_stamp":"2024-02-19T21:08:49.752418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.752418+0000","log_size":121,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.a","version":"236'22","reported_seq":252,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683387+0000","last_change":"2024-02-19T21:10:41.434077+0000","last_active":"2024-02-19T21:11:06.683387+0000","last_peered":"2024-02-19T21:11:06.683387+0000","last_clean":"2024-02-19T21:11:06.683387+0000","last_became_active":"2024-02-19T21:10:41.305485+0000","last_became_peered":"2024-02-19T21:10:41.305485+0000","last_unstale":"2024-02-19T21:11:06.683387+0000","last_undegraded":"2024-02-19T21:11:06.683387+0000","last_fullsized":"2024-02-19T21:11:06.683387+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-02-19T21:08:01.607121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:01.607121+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":96,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.b","version":"278'95","reported_seq":496,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898562+0000","last_change":"2024-02-19T21:11:03.821979+0000","last_active":"2024-02-19T21:11:06.898562+0000","last_peered":"2024-02-19T21:11:06.898562+0000","last_clean":"2024-02-19T21:11:06.898562+0000","last_became_active":"2024-02-19T21:04:54.640024+0000","last_became_peered":"2024-02-19T21:04:54.640024+0000","last_unstale":"2024-02-19T21:11:06.898562+0000","last_undegraded":"2024-02-19T21:11:06.898562+0000","last_fullsized":"2024-02-19T21:11:06.898562+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'95","last_scrub_stamp":"2024-02-19T21:11:03.821869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.821869+0000","log_size":95,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.8","version":"399'39","reported_seq":507,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058173+0000","last_change":"2024-02-19T21:10:31.526940+0000","last_active":"2024-02-19T21:11:07.058173+0000","last_peered":"2024-02-19T21:11:07.058173+0000","last_clean":"2024-02-19T21:11:07.058173+0000","last_became_active":"2024-02-19T21:08:04.266021+0000","last_became_peered":"2024-02-19T21:08:04.266021+0000","last_unstale":"2024-02-19T21:11:07.058173+0000","last_undegraded":"2024-02-19T21:11:07.058173+0000","last_fullsized":"2024-02-19T21:11:07.058173+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'35","last_scrub_stamp":"2024-02-19T21:08:01.494484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:01.494484+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":50,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.9","version":"439'77","reported_seq":486,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898057+0000","last_change":"2024-02-19T21:10:55.678589+0000","last_active":"2024-02-19T21:11:06.898057+0000","last_peered":"2024-02-19T21:11:06.898057+0000","last_clean":"2024-02-19T21:11:06.898057+0000","last_became_active":"2024-02-19T21:04:54.671646+0000","last_became_peered":"2024-02-19T21:04:54.671646+0000","last_unstale":"2024-02-19T21:11:06.898057+0000","last_undegraded":"2024-02-19T21:11:06.898057+0000","last_fullsized":"2024-02-19T21:11:06.898057+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'77","last_scrub_stamp":"2024-02-19T21:10:55.678451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:55.678451+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":71,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.e","version":"393'40","reported_seq":526,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681509+0000","last_change":"2024-02-19T21:10:41.425839+0000","last_active":"2024-02-19T21:11:06.681509+0000","last_peered":"2024-02-19T21:11:06.681509+0000","last_clean":"2024-02-19T21:11:06.681509+0000","last_became_active":"2024-02-19T21:10:41.298057+0000","last_became_peered":"2024-02-19T21:10:41.298057+0000","last_unstale":"2024-02-19T21:11:06.681509+0000","last_undegraded":"2024-02-19T21:11:06.681509+0000","last_fullsized":"2024-02-19T21:11:06.681509+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'40","last_scrub_stamp":"2024-02-19T21:10:18.512566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:18.512566+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":81,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.f","version":"399'37","reported_seq":515,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.901478+0000","last_change":"2024-02-19T21:10:41.464188+0000","last_active":"2024-02-19T21:11:06.901478+0000","last_peered":"2024-02-19T21:11:06.901478+0000","last_clean":"2024-02-19T21:11:06.901478+0000","last_became_active":"2024-02-19T21:10:41.315097+0000","last_became_peered":"2024-02-19T21:10:41.315097+0000","last_unstale":"2024-02-19T21:11:06.901478+0000","last_undegraded":"2024-02-19T21:11:06.901478+0000","last_fullsized":"2024-02-19T21:11:06.901478+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'37","last_scrub_stamp":"2024-02-19T21:10:13.757798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:13.757798+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.c","version":"249'73","reported_seq":691,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.059030+0000","last_change":"2024-02-19T21:10:46.664562+0000","last_active":"2024-02-19T21:11:07.059030+0000","last_peered":"2024-02-19T21:11:07.059030+0000","last_clean":"2024-02-19T21:11:07.059030+0000","last_became_active":"2024-02-19T21:10:41.321453+0000","last_became_peered":"2024-02-19T21:10:41.321453+0000","last_unstale":"2024-02-19T21:11:07.059030+0000","last_undegraded":"2024-02-19T21:11:07.059030+0000","last_fullsized":"2024-02-19T21:11:07.059030+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'73","last_scrub_stamp":"2024-02-19T21:08:49.613462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.613462+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.d","version":"442'44","reported_seq":454,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058825+0000","last_change":"2024-02-19T21:10:31.528317+0000","last_active":"2024-02-19T21:11:07.058825+0000","last_peered":"2024-02-19T21:11:07.058825+0000","last_clean":"2024-02-19T21:11:07.058825+0000","last_became_active":"2024-02-19T21:04:54.644749+0000","last_became_peered":"2024-02-19T21:04:54.644749+0000","last_unstale":"2024-02-19T21:11:07.058825+0000","last_undegraded":"2024-02-19T21:11:07.058825+0000","last_fullsized":"2024-02-19T21:11:07.058825+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'34","last_scrub_stamp":"2024-02-19T21:08:12.445646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.445646+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.12","version":"390'47","reported_seq":532,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682040+0000","last_change":"2024-02-19T21:10:31.534697+0000","last_active":"2024-02-19T21:11:06.682040+0000","last_peered":"2024-02-19T21:11:06.682040+0000","last_clean":"2024-02-19T21:11:06.682040+0000","last_became_active":"2024-02-19T21:08:04.263085+0000","last_became_peered":"2024-02-19T21:08:04.263085+0000","last_unstale":"2024-02-19T21:11:06.682040+0000","last_undegraded":"2024-02-19T21:11:06.682040+0000","last_fullsized":"2024-02-19T21:11:06.682040+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'45","last_scrub_stamp":"2024-02-19T21:08:49.965991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.965991+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.13","version":"370'71","reported_seq":594,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.900476+0000","last_change":"2024-02-19T21:10:57.789184+0000","last_active":"2024-02-19T21:11:06.900476+0000","last_peered":"2024-02-19T21:11:06.900476+0000","last_clean":"2024-02-19T21:11:06.900476+0000","last_became_active":"2024-02-19T21:10:41.317677+0000","last_became_peered":"2024-02-19T21:10:41.317677+0000","last_unstale":"2024-02-19T21:11:06.900476+0000","last_undegraded":"2024-02-19T21:11:06.900476+0000","last_fullsized":"2024-02-19T21:11:06.900476+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'71","last_scrub_stamp":"2024-02-19T21:10:57.789103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.789103+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.10","version":"452'49","reported_seq":463,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.899785+0000","last_change":"2024-02-19T21:11:04.750002+0000","last_active":"2024-02-19T21:11:06.899785+0000","last_peered":"2024-02-19T21:11:06.899785+0000","last_clean":"2024-02-19T21:11:06.899785+0000","last_became_active":"2024-02-19T21:04:54.626393+0000","last_became_peered":"2024-02-19T21:04:54.626393+0000","last_unstale":"2024-02-19T21:11:06.899785+0000","last_undegraded":"2024-02-19T21:11:06.899785+0000","last_fullsized":"2024-02-19T21:11:06.899785+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'49","last_scrub_stamp":"2024-02-19T21:11:04.749932+0000","last_deep_scrub":"452'49","last_deep_scrub_stamp":"2024-02-19T21:11:04.749932+0000","last_clean_scrub_stamp":"2024-02-19T21:11:04.749932+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.11","version":"338'35","reported_seq":445,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.680924+0000","last_change":"2024-02-19T21:10:31.533725+0000","last_active":"2024-02-19T21:11:06.680924+0000","last_peered":"2024-02-19T21:11:06.680924+0000","last_clean":"2024-02-19T21:11:06.680924+0000","last_became_active":"2024-02-19T21:04:54.659096+0000","last_became_peered":"2024-02-19T21:04:54.659096+0000","last_unstale":"2024-02-19T21:11:06.680924+0000","last_undegraded":"2024-02-19T21:11:06.680924+0000","last_fullsized":"2024-02-19T21:11:06.680924+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'35","last_scrub_stamp":"2024-02-19T21:08:41.145092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:41.145092+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.16","version":"419'47","reported_seq":61,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.456655+0000","last_change":"2024-02-19T21:10:54.364931+0000","last_active":"2024-02-19T21:11:07.456655+0000","last_peered":"2024-02-19T21:11:07.456655+0000","last_clean":"2024-02-19T21:11:07.456655+0000","last_became_active":"2024-02-19T21:10:41.313393+0000","last_became_peered":"2024-02-19T21:10:41.313393+0000","last_unstale":"2024-02-19T21:11:07.456655+0000","last_undegraded":"2024-02-19T21:11:07.456655+0000","last_fullsized":"2024-02-19T21:11:07.456655+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'47","last_scrub_stamp":"2024-02-19T21:10:54.364824+0000","last_deep_scrub":"237'24","last_deep_scrub_stamp":"2024-02-19T21:06:57.889852+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.364824+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":40,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"61.17","version":"286'24","reported_seq":494,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682104+0000","last_change":"2024-02-19T21:10:41.753888+0000","last_active":"2024-02-19T21:11:06.682104+0000","last_peered":"2024-02-19T21:11:06.682104+0000","last_clean":"2024-02-19T21:11:06.682104+0000","last_became_active":"2024-02-19T21:10:41.314188+0000","last_became_peered":"2024-02-19T21:10:41.314188+0000","last_unstale":"2024-02-19T21:11:06.682104+0000","last_undegraded":"2024-02-19T21:11:06.682104+0000","last_fullsized":"2024-02-19T21:11:06.682104+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'24","last_scrub_stamp":"2024-02-19T21:08:48.023079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:48.023079+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.14","version":"338'52","reported_seq":439,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681762+0000","last_change":"2024-02-19T21:10:31.536240+0000","last_active":"2024-02-19T21:11:06.681762+0000","last_peered":"2024-02-19T21:11:06.681762+0000","last_clean":"2024-02-19T21:11:06.681762+0000","last_became_active":"2024-02-19T21:06:40.227840+0000","last_became_peered":"2024-02-19T21:06:40.227840+0000","last_unstale":"2024-02-19T21:11:06.681762+0000","last_undegraded":"2024-02-19T21:11:06.681762+0000","last_fullsized":"2024-02-19T21:11:06.681762+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'52","last_scrub_stamp":"2024-02-19T21:09:28.997593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.997593+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.15","version":"370'52","reported_seq":630,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.900896+0000","last_change":"2024-02-19T21:10:43.140651+0000","last_active":"2024-02-19T21:11:06.900896+0000","last_peered":"2024-02-19T21:11:06.900896+0000","last_clean":"2024-02-19T21:11:06.900896+0000","last_became_active":"2024-02-19T21:10:41.316332+0000","last_became_peered":"2024-02-19T21:10:41.316332+0000","last_unstale":"2024-02-19T21:11:06.900896+0000","last_undegraded":"2024-02-19T21:11:06.900896+0000","last_fullsized":"2024-02-19T21:11:06.900896+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'52","last_scrub_stamp":"2024-02-19T21:10:22.847438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:22.847438+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1a","version":"448'98","reported_seq":635,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.900202+0000","last_change":"2024-02-19T21:10:55.797877+0000","last_active":"2024-02-19T21:11:06.900202+0000","last_peered":"2024-02-19T21:11:06.900202+0000","last_clean":"2024-02-19T21:11:06.900202+0000","last_became_active":"2024-02-19T21:08:04.265588+0000","last_became_peered":"2024-02-19T21:08:04.265588+0000","last_unstale":"2024-02-19T21:11:06.900202+0000","last_undegraded":"2024-02-19T21:11:06.900202+0000","last_fullsized":"2024-02-19T21:11:06.900202+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'98","last_scrub_stamp":"2024-02-19T21:10:55.797794+0000","last_deep_scrub":"448'98","last_deep_scrub_stamp":"2024-02-19T21:10:55.797794+0000","last_clean_scrub_stamp":"2024-02-19T21:10:55.797794+0000","log_size":98,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1b","version":"338'52","reported_seq":213,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681417+0000","last_change":"2024-02-19T21:10:31.528748+0000","last_active":"2024-02-19T21:11:06.681417+0000","last_peered":"2024-02-19T21:11:06.681417+0000","last_clean":"2024-02-19T21:11:06.681417+0000","last_became_active":"2024-02-19T21:08:04.262932+0000","last_became_peered":"2024-02-19T21:08:04.262932+0000","last_unstale":"2024-02-19T21:11:06.681417+0000","last_undegraded":"2024-02-19T21:11:06.681417+0000","last_fullsized":"2024-02-19T21:11:06.681417+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'51","last_scrub_stamp":"2024-02-19T21:08:00.485092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:00.485092+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.18","version":"214'102","reported_seq":556,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683343+0000","last_change":"2024-02-19T21:10:45.292818+0000","last_active":"2024-02-19T21:11:06.683343+0000","last_peered":"2024-02-19T21:11:06.683343+0000","last_clean":"2024-02-19T21:11:06.683343+0000","last_became_active":"2024-02-19T21:10:45.292216+0000","last_became_peered":"2024-02-19T21:10:45.292216+0000","last_unstale":"2024-02-19T21:11:06.683343+0000","last_undegraded":"2024-02-19T21:11:06.683343+0000","last_fullsized":"2024-02-19T21:11:06.683343+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-19T21:08:50.838363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:50.838363+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":198,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.19","version":"338'69","reported_seq":581,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058137+0000","last_change":"2024-02-19T21:10:43.469699+0000","last_active":"2024-02-19T21:11:07.058137+0000","last_peered":"2024-02-19T21:11:07.058137+0000","last_clean":"2024-02-19T21:11:07.058137+0000","last_became_active":"2024-02-19T21:10:41.296786+0000","last_became_peered":"2024-02-19T21:10:41.296786+0000","last_unstale":"2024-02-19T21:11:07.058137+0000","last_undegraded":"2024-02-19T21:11:07.058137+0000","last_fullsized":"2024-02-19T21:11:07.058137+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-19T21:08:39.466543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.466543+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.1e","version":"338'50","reported_seq":467,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.901290+0000","last_change":"2024-02-19T21:10:31.710212+0000","last_active":"2024-02-19T21:11:06.901290+0000","last_peered":"2024-02-19T21:11:06.901290+0000","last_clean":"2024-02-19T21:11:06.901290+0000","last_became_active":"2024-02-19T21:04:54.629433+0000","last_became_peered":"2024-02-19T21:04:54.629433+0000","last_unstale":"2024-02-19T21:11:06.901290+0000","last_undegraded":"2024-02-19T21:11:06.901290+0000","last_fullsized":"2024-02-19T21:11:06.901290+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'50","last_scrub_stamp":"2024-02-19T21:10:16.846927+0000","last_deep_scrub":"241'49","last_deep_scrub_stamp":"2024-02-19T21:07:01.787521+0000","last_clean_scrub_stamp":"2024-02-19T21:10:16.846927+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":48,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1f","version":"249'44","reported_seq":454,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682199+0000","last_change":"2024-02-19T21:10:31.536663+0000","last_active":"2024-02-19T21:11:06.682199+0000","last_peered":"2024-02-19T21:11:06.682199+0000","last_clean":"2024-02-19T21:11:06.682199+0000","last_became_active":"2024-02-19T21:04:54.671969+0000","last_became_peered":"2024-02-19T21:04:54.671969+0000","last_unstale":"2024-02-19T21:11:06.682199+0000","last_undegraded":"2024-02-19T21:11:06.682199+0000","last_fullsized":"2024-02-19T21:11:06.682199+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'44","last_scrub_stamp":"2024-02-19T21:09:25.988978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:09:25.988978+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.1c","version":"429'153","reported_seq":753,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058712+0000","last_change":"2024-02-19T21:10:50.292413+0000","last_active":"2024-02-19T21:11:07.058712+0000","last_peered":"2024-02-19T21:11:07.058712+0000","last_clean":"2024-02-19T21:11:07.058712+0000","last_became_active":"2024-02-19T21:10:50.291607+0000","last_became_peered":"2024-02-19T21:10:50.291607+0000","last_unstale":"2024-02-19T21:11:07.058712+0000","last_undegraded":"2024-02-19T21:11:07.058712+0000","last_fullsized":"2024-02-19T21:11:07.058712+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'136","last_scrub_stamp":"2024-02-19T21:08:37.523303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:37.523303+0000","log_size":153,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":46,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.1d","version":"370'36","reported_seq":445,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898879+0000","last_change":"2024-02-19T21:10:31.535725+0000","last_active":"2024-02-19T21:11:06.898879+0000","last_peered":"2024-02-19T21:11:06.898879+0000","last_clean":"2024-02-19T21:11:06.898879+0000","last_became_active":"2024-02-19T21:04:54.655665+0000","last_became_peered":"2024-02-19T21:04:54.655665+0000","last_unstale":"2024-02-19T21:11:06.898879+0000","last_undegraded":"2024-02-19T21:11:06.898879+0000","last_fullsized":"2024-02-19T21:11:06.898879+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'35","last_scrub_stamp":"2024-02-19T21:08:56.661296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:56.661296+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"434'11","reported_seq":573,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-19T21:11:09.707943+0000","last_change":"2024-02-19T21:10:43.270608+0000","last_active":"2024-02-19T21:11:09.707943+0000","last_peered":"2024-02-19T21:11:09.707943+0000","last_clean":"2024-02-19T21:11:09.707943+0000","last_became_active":"2024-02-19T21:10:41.316587+0000","last_became_peered":"2024-02-19T21:10:41.316587+0000","last_unstale":"2024-02-19T21:11:09.707943+0000","last_undegraded":"2024-02-19T21:11:09.707943+0000","last_fullsized":"2024-02-19T21:11:09.707943+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'10","last_scrub_stamp":"2024-02-19T21:09:42.967771+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:09:42.967771+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'15","reported_seq":444,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058189+0000","last_change":"2024-02-19T21:08:51.616197+0000","last_active":"2024-02-19T21:11:07.058189+0000","last_peered":"2024-02-19T21:11:07.058189+0000","last_clean":"2024-02-19T21:11:07.058189+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:11:07.058189+0000","last_undegraded":"2024-02-19T21:11:07.058189+0000","last_fullsized":"2024-02-19T21:11:07.058189+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:08:51.616123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:51.616123+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":449,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682140+0000","last_change":"2024-02-19T21:10:40.300037+0000","last_active":"2024-02-19T21:11:06.682140+0000","last_peered":"2024-02-19T21:11:06.682140+0000","last_clean":"2024-02-19T21:11:06.682140+0000","last_became_active":"2024-02-19T21:10:40.299425+0000","last_became_peered":"2024-02-19T21:10:40.299425+0000","last_unstale":"2024-02-19T21:11:06.682140+0000","last_undegraded":"2024-02-19T21:11:06.682140+0000","last_fullsized":"2024-02-19T21:11:06.682140+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:44.922303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_clean_scrub_stamp":"2024-02-19T21:09:44.922303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":412,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058508+0000","last_change":"2024-02-19T21:09:27.878844+0000","last_active":"2024-02-19T21:11:07.058508+0000","last_peered":"2024-02-19T21:11:07.058508+0000","last_clean":"2024-02-19T21:11:07.058508+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:11:07.058508+0000","last_undegraded":"2024-02-19T21:11:07.058508+0000","last_fullsized":"2024-02-19T21:11:07.058508+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.878760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.878760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":370,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682471+0000","last_change":"2024-02-19T21:10:17.082171+0000","last_active":"2024-02-19T21:11:06.682471+0000","last_peered":"2024-02-19T21:11:06.682471+0000","last_clean":"2024-02-19T21:11:06.682471+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:11:06.682471+0000","last_undegraded":"2024-02-19T21:11:06.682471+0000","last_fullsized":"2024-02-19T21:11:06.682471+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:17.082103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:17.082103+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"65'9","reported_seq":312,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898726+0000","last_change":"2024-02-19T21:10:41.659407+0000","last_active":"2024-02-19T21:11:06.898726+0000","last_peered":"2024-02-19T21:11:06.898726+0000","last_clean":"2024-02-19T21:11:06.898726+0000","last_became_active":"2024-02-19T21:10:41.295175+0000","last_became_peered":"2024-02-19T21:10:41.295175+0000","last_unstale":"2024-02-19T21:11:06.898726+0000","last_undegraded":"2024-02-19T21:11:06.898726+0000","last_fullsized":"2024-02-19T21:11:06.898726+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:08:53.783292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:53.783292+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":456,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.899337+0000","last_change":"2024-02-19T21:10:41.312903+0000","last_active":"2024-02-19T21:11:06.899337+0000","last_peered":"2024-02-19T21:11:06.899337+0000","last_clean":"2024-02-19T21:11:06.899337+0000","last_became_active":"2024-02-19T21:10:41.312116+0000","last_became_peered":"2024-02-19T21:10:41.312116+0000","last_unstale":"2024-02-19T21:11:06.899337+0000","last_undegraded":"2024-02-19T21:11:06.899337+0000","last_fullsized":"2024-02-19T21:11:06.899337+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:39.700315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.700315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":454,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682121+0000","last_change":"2024-02-19T21:10:41.306360+0000","last_active":"2024-02-19T21:11:06.682121+0000","last_peered":"2024-02-19T21:11:06.682121+0000","last_clean":"2024-02-19T21:11:06.682121+0000","last_became_active":"2024-02-19T21:10:41.305574+0000","last_became_peered":"2024-02-19T21:10:41.305574+0000","last_unstale":"2024-02-19T21:11:06.682121+0000","last_undegraded":"2024-02-19T21:11:06.682121+0000","last_fullsized":"2024-02-19T21:11:06.682121+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:13.035939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:13.035939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":248,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.900022+0000","last_change":"2024-02-19T21:09:34.791174+0000","last_active":"2024-02-19T21:11:06.900022+0000","last_peered":"2024-02-19T21:11:06.900022+0000","last_clean":"2024-02-19T21:11:06.900022+0000","last_became_active":"2024-02-19T21:06:40.227822+0000","last_became_peered":"2024-02-19T21:06:40.227822+0000","last_unstale":"2024-02-19T21:11:06.900022+0000","last_undegraded":"2024-02-19T21:11:06.900022+0000","last_fullsized":"2024-02-19T21:11:06.900022+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:34.791063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:34.791063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":371,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.059301+0000","last_change":"2024-02-19T21:10:57.130256+0000","last_active":"2024-02-19T21:11:07.059301+0000","last_peered":"2024-02-19T21:11:07.059301+0000","last_clean":"2024-02-19T21:11:07.059301+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:11:07.059301+0000","last_undegraded":"2024-02-19T21:11:07.059301+0000","last_fullsized":"2024-02-19T21:11:07.059301+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.130171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.130171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"65'19","reported_seq":453,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.680908+0000","last_change":"2024-02-19T21:11:04.345915+0000","last_active":"2024-02-19T21:11:06.680908+0000","last_peered":"2024-02-19T21:11:06.680908+0000","last_clean":"2024-02-19T21:11:06.680908+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:11:06.680908+0000","last_undegraded":"2024-02-19T21:11:06.680908+0000","last_fullsized":"2024-02-19T21:11:06.680908+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:11:04.345836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:04.345836+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":422,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682183+0000","last_change":"2024-02-19T21:09:30.950696+0000","last_active":"2024-02-19T21:11:06.682183+0000","last_peered":"2024-02-19T21:11:06.682183+0000","last_clean":"2024-02-19T21:11:06.682183+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:11:06.682183+0000","last_undegraded":"2024-02-19T21:11:06.682183+0000","last_fullsized":"2024-02-19T21:11:06.682183+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:30.950626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:09:30.950626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":413,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681286+0000","last_change":"2024-02-19T21:09:28.031934+0000","last_active":"2024-02-19T21:11:06.681286+0000","last_peered":"2024-02-19T21:11:06.681286+0000","last_clean":"2024-02-19T21:11:06.681286+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:11:06.681286+0000","last_undegraded":"2024-02-19T21:11:06.681286+0000","last_fullsized":"2024-02-19T21:11:06.681286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:28.031872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.031872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":428,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.899897+0000","last_change":"2024-02-19T21:08:53.964809+0000","last_active":"2024-02-19T21:11:06.899897+0000","last_peered":"2024-02-19T21:11:06.899897+0000","last_clean":"2024-02-19T21:11:06.899897+0000","last_became_active":"2024-02-19T21:08:04.265081+0000","last_became_peered":"2024-02-19T21:08:04.265081+0000","last_unstale":"2024-02-19T21:11:06.899897+0000","last_undegraded":"2024-02-19T21:11:06.899897+0000","last_fullsized":"2024-02-19T21:11:06.899897+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_clean_scrub_stamp":"2024-02-19T21:08:53.964703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":388,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.901324+0000","last_change":"2024-02-19T21:10:41.317716+0000","last_active":"2024-02-19T21:11:06.901324+0000","last_peered":"2024-02-19T21:11:06.901324+0000","last_clean":"2024-02-19T21:11:06.901324+0000","last_became_active":"2024-02-19T21:10:41.317284+0000","last_became_peered":"2024-02-19T21:10:41.317284+0000","last_unstale":"2024-02-19T21:11:06.901324+0000","last_undegraded":"2024-02-19T21:11:06.901324+0000","last_fullsized":"2024-02-19T21:11:06.901324+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:43.984079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:09:43.984079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"65'17","reported_seq":48,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.456566+0000","last_change":"2024-02-19T21:10:41.676984+0000","last_active":"2024-02-19T21:11:07.456566+0000","last_peered":"2024-02-19T21:11:07.456566+0000","last_clean":"2024-02-19T21:11:07.456566+0000","last_became_active":"2024-02-19T21:10:41.313853+0000","last_became_peered":"2024-02-19T21:10:41.313853+0000","last_unstale":"2024-02-19T21:11:07.456566+0000","last_undegraded":"2024-02-19T21:11:07.456566+0000","last_fullsized":"2024-02-19T21:11:07.456566+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:09:41.955563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:41.955563+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":453,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.900675+0000","last_change":"2024-02-19T21:10:41.408960+0000","last_active":"2024-02-19T21:11:06.900675+0000","last_peered":"2024-02-19T21:11:06.900675+0000","last_clean":"2024-02-19T21:11:06.900675+0000","last_became_active":"2024-02-19T21:10:41.307517+0000","last_became_peered":"2024-02-19T21:10:41.307517+0000","last_unstale":"2024-02-19T21:11:06.900675+0000","last_undegraded":"2024-02-19T21:11:06.900675+0000","last_fullsized":"2024-02-19T21:11:06.900675+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:10:08.878163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:08.878163+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":434,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898887+0000","last_change":"2024-02-19T21:10:41.312601+0000","last_active":"2024-02-19T21:11:06.898887+0000","last_peered":"2024-02-19T21:11:06.898887+0000","last_clean":"2024-02-19T21:11:06.898887+0000","last_became_active":"2024-02-19T21:10:41.312259+0000","last_became_peered":"2024-02-19T21:10:41.312259+0000","last_unstale":"2024-02-19T21:11:06.898887+0000","last_undegraded":"2024-02-19T21:11:06.898887+0000","last_fullsized":"2024-02-19T21:11:06.898887+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:36.685063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:36.685063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":452,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683167+0000","last_change":"2024-02-19T21:10:41.307901+0000","last_active":"2024-02-19T21:11:06.683167+0000","last_peered":"2024-02-19T21:11:06.683167+0000","last_clean":"2024-02-19T21:11:06.683167+0000","last_became_active":"2024-02-19T21:10:41.307577+0000","last_became_peered":"2024-02-19T21:10:41.307577+0000","last_unstale":"2024-02-19T21:11:06.683167+0000","last_undegraded":"2024-02-19T21:11:06.683167+0000","last_fullsized":"2024-02-19T21:11:06.683167+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:03.053807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:10:03.053807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":35,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.455668+0000","last_change":"2024-02-19T21:10:50.424462+0000","last_active":"2024-02-19T21:11:07.455668+0000","last_peered":"2024-02-19T21:11:07.455668+0000","last_clean":"2024-02-19T21:11:07.455668+0000","last_became_active":"2024-02-19T21:10:41.302670+0000","last_became_peered":"2024-02-19T21:10:41.302670+0000","last_unstale":"2024-02-19T21:11:07.455668+0000","last_undegraded":"2024-02-19T21:11:07.455668+0000","last_fullsized":"2024-02-19T21:11:07.455668+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:50.424381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.424381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"82'15","reported_seq":42,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.456374+0000","last_change":"2024-02-19T21:10:41.329217+0000","last_active":"2024-02-19T21:11:07.456374+0000","last_peered":"2024-02-19T21:11:07.456374+0000","last_clean":"2024-02-19T21:11:07.456374+0000","last_became_active":"2024-02-19T21:10:41.291470+0000","last_became_peered":"2024-02-19T21:10:41.291470+0000","last_unstale":"2024-02-19T21:11:07.456374+0000","last_undegraded":"2024-02-19T21:11:07.456374+0000","last_fullsized":"2024-02-19T21:11:07.456374+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:10:10.650661+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:10:10.650661+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":438,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898646+0000","last_change":"2024-02-19T21:10:12.628865+0000","last_active":"2024-02-19T21:11:06.898646+0000","last_peered":"2024-02-19T21:11:06.898646+0000","last_clean":"2024-02-19T21:11:06.898646+0000","last_became_active":"2024-02-19T21:08:04.268406+0000","last_became_peered":"2024-02-19T21:08:04.268406+0000","last_unstale":"2024-02-19T21:11:06.898646+0000","last_undegraded":"2024-02-19T21:11:06.898646+0000","last_fullsized":"2024-02-19T21:11:06.898646+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:12.628801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:12.628801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":546,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-19T21:11:09.707690+0000","last_change":"2024-02-19T21:10:04.016684+0000","last_active":"2024-02-19T21:11:09.707690+0000","last_peered":"2024-02-19T21:11:09.707690+0000","last_clean":"2024-02-19T21:11:09.707690+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:11:09.707690+0000","last_undegraded":"2024-02-19T21:11:09.707690+0000","last_fullsized":"2024-02-19T21:11:09.707690+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:10:04.016619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:04.016619+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":30,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.456732+0000","last_change":"2024-02-19T21:10:41.299284+0000","last_active":"2024-02-19T21:11:07.456732+0000","last_peered":"2024-02-19T21:11:07.456732+0000","last_clean":"2024-02-19T21:11:07.456732+0000","last_became_active":"2024-02-19T21:10:41.298893+0000","last_became_peered":"2024-02-19T21:10:41.298893+0000","last_unstale":"2024-02-19T21:11:07.456732+0000","last_undegraded":"2024-02-19T21:11:07.456732+0000","last_fullsized":"2024-02-19T21:11:07.456732+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:02.519642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:10:02.519642+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683178+0000","last_change":"2024-02-19T21:08:36.847613+0000","last_active":"2024-02-19T21:11:06.683178+0000","last_peered":"2024-02-19T21:11:06.683178+0000","last_clean":"2024-02-19T21:11:06.683178+0000","last_became_active":"2024-02-19T21:08:04.242244+0000","last_became_peered":"2024-02-19T21:08:04.242244+0000","last_unstale":"2024-02-19T21:11:06.683178+0000","last_undegraded":"2024-02-19T21:11:06.683178+0000","last_fullsized":"2024-02-19T21:11:06.683178+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:36.847548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.847548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'15","reported_seq":61,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:08.202306+0000","last_change":"2024-02-19T21:11:08.202280+0000","last_active":"2024-02-19T21:11:08.202306+0000","last_peered":"2024-02-19T21:11:08.202306+0000","last_clean":"2024-02-19T21:11:08.202306+0000","last_became_active":"2024-02-19T21:10:41.307576+0000","last_became_peered":"2024-02-19T21:10:41.307576+0000","last_unstale":"2024-02-19T21:11:08.202306+0000","last_undegraded":"2024-02-19T21:11:08.202306+0000","last_fullsized":"2024-02-19T21:11:08.202306+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:11:08.202208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:08.202208+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":465,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898401+0000","last_change":"2024-02-19T21:10:52.738673+0000","last_active":"2024-02-19T21:11:06.898401+0000","last_peered":"2024-02-19T21:11:06.898401+0000","last_clean":"2024-02-19T21:11:06.898401+0000","last_became_active":"2024-02-19T21:08:04.270601+0000","last_became_peered":"2024-02-19T21:08:04.270601+0000","last_unstale":"2024-02-19T21:11:06.898401+0000","last_undegraded":"2024-02-19T21:11:06.898401+0000","last_fullsized":"2024-02-19T21:11:06.898401+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.738600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.738600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"434'10","reported_seq":42,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-19T21:11:09.707934+0000","last_change":"2024-02-19T21:10:41.665978+0000","last_active":"2024-02-19T21:11:09.707934+0000","last_peered":"2024-02-19T21:11:09.707934+0000","last_clean":"2024-02-19T21:11:09.707934+0000","last_became_active":"2024-02-19T21:10:41.312459+0000","last_became_peered":"2024-02-19T21:10:41.312459+0000","last_unstale":"2024-02-19T21:11:09.707934+0000","last_undegraded":"2024-02-19T21:11:09.707934+0000","last_fullsized":"2024-02-19T21:11:09.707934+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'9","last_scrub_stamp":"2024-02-19T21:10:12.186173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:12.186173+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":417,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.059261+0000","last_change":"2024-02-19T21:09:31.856861+0000","last_active":"2024-02-19T21:11:07.059261+0000","last_peered":"2024-02-19T21:11:07.059261+0000","last_clean":"2024-02-19T21:11:07.059261+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:11:07.059261+0000","last_undegraded":"2024-02-19T21:11:07.059261+0000","last_fullsized":"2024-02-19T21:11:07.059261+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:31.856784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:31.856784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":371,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058317+0000","last_change":"2024-02-19T21:11:01.079431+0000","last_active":"2024-02-19T21:11:07.058317+0000","last_peered":"2024-02-19T21:11:07.058317+0000","last_clean":"2024-02-19T21:11:07.058317+0000","last_became_active":"2024-02-19T21:04:39.657052+0000","last_became_peered":"2024-02-19T21:04:39.657052+0000","last_unstale":"2024-02-19T21:11:07.058317+0000","last_undegraded":"2024-02-19T21:11:07.058317+0000","last_fullsized":"2024-02-19T21:11:07.058317+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.079353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.079353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"65'11","reported_seq":42,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.456716+0000","last_change":"2024-02-19T21:10:41.335249+0000","last_active":"2024-02-19T21:11:07.456716+0000","last_peered":"2024-02-19T21:11:07.456716+0000","last_clean":"2024-02-19T21:11:07.456716+0000","last_became_active":"2024-02-19T21:10:41.296921+0000","last_became_peered":"2024-02-19T21:10:41.296921+0000","last_unstale":"2024-02-19T21:11:07.456716+0000","last_undegraded":"2024-02-19T21:11:07.456716+0000","last_fullsized":"2024-02-19T21:11:07.456716+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:49.181100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.181100+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":448,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.900063+0000","last_change":"2024-02-19T21:09:37.870271+0000","last_active":"2024-02-19T21:11:06.900063+0000","last_peered":"2024-02-19T21:11:06.900063+0000","last_clean":"2024-02-19T21:11:06.900063+0000","last_became_active":"2024-02-19T21:07:32.227557+0000","last_became_peered":"2024-02-19T21:07:32.227557+0000","last_unstale":"2024-02-19T21:11:06.900063+0000","last_undegraded":"2024-02-19T21:11:06.900063+0000","last_fullsized":"2024-02-19T21:11:06.900063+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.870218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.870218+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":437,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.059411+0000","last_change":"2024-02-19T21:10:58.096107+0000","last_active":"2024-02-19T21:11:07.059411+0000","last_peered":"2024-02-19T21:11:07.059411+0000","last_clean":"2024-02-19T21:11:07.059411+0000","last_became_active":"2024-02-19T21:10:41.314907+0000","last_became_peered":"2024-02-19T21:10:41.314907+0000","last_unstale":"2024-02-19T21:11:07.059411+0000","last_undegraded":"2024-02-19T21:11:07.059411+0000","last_fullsized":"2024-02-19T21:11:07.059411+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:58.096025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.096025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683451+0000","last_change":"2024-02-19T21:10:06.547564+0000","last_active":"2024-02-19T21:11:06.683451+0000","last_peered":"2024-02-19T21:11:06.683451+0000","last_clean":"2024-02-19T21:11:06.683451+0000","last_became_active":"2024-02-19T21:08:04.279008+0000","last_became_peered":"2024-02-19T21:08:04.279008+0000","last_unstale":"2024-02-19T21:11:06.683451+0000","last_undegraded":"2024-02-19T21:11:06.683451+0000","last_fullsized":"2024-02-19T21:11:06.683451+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:06.547494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:06.547494+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":160,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683208+0000","last_change":"2024-02-19T21:08:04.280833+0000","last_active":"2024-02-19T21:11:06.683208+0000","last_peered":"2024-02-19T21:11:06.683208+0000","last_clean":"2024-02-19T21:11:06.683208+0000","last_became_active":"2024-02-19T21:08:04.280514+0000","last_became_peered":"2024-02-19T21:08:04.280514+0000","last_unstale":"2024-02-19T21:11:06.683208+0000","last_undegraded":"2024-02-19T21:11:06.683208+0000","last_fullsized":"2024-02-19T21:11:06.683208+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:58.499841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:58.499841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"434'12","reported_seq":237,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-19T21:11:09.707681+0000","last_change":"2024-02-19T21:10:41.321881+0000","last_active":"2024-02-19T21:11:09.707681+0000","last_peered":"2024-02-19T21:11:09.707681+0000","last_clean":"2024-02-19T21:11:09.707681+0000","last_became_active":"2024-02-19T21:10:41.297897+0000","last_became_peered":"2024-02-19T21:10:41.297897+0000","last_unstale":"2024-02-19T21:11:09.707681+0000","last_undegraded":"2024-02-19T21:11:09.707681+0000","last_fullsized":"2024-02-19T21:11:09.707681+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'11","last_scrub_stamp":"2024-02-19T21:09:39.597001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:39.597001+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"65'14","reported_seq":51,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.455787+0000","last_change":"2024-02-19T21:10:49.466291+0000","last_active":"2024-02-19T21:11:07.455787+0000","last_peered":"2024-02-19T21:11:07.455787+0000","last_clean":"2024-02-19T21:11:07.455787+0000","last_became_active":"2024-02-19T21:10:41.308428+0000","last_became_peered":"2024-02-19T21:10:41.308428+0000","last_unstale":"2024-02-19T21:11:07.455787+0000","last_undegraded":"2024-02-19T21:11:07.455787+0000","last_fullsized":"2024-02-19T21:11:07.455787+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:10:49.466227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:49.466227+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":451,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681783+0000","last_change":"2024-02-19T21:11:05.364793+0000","last_active":"2024-02-19T21:11:06.681783+0000","last_peered":"2024-02-19T21:11:06.681783+0000","last_clean":"2024-02-19T21:11:06.681783+0000","last_became_active":"2024-02-19T21:07:36.263299+0000","last_became_peered":"2024-02-19T21:07:36.263299+0000","last_unstale":"2024-02-19T21:11:06.681783+0000","last_undegraded":"2024-02-19T21:11:06.681783+0000","last_fullsized":"2024-02-19T21:11:06.681783+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.364732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.364732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":419,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.900900+0000","last_change":"2024-02-19T21:10:51.851228+0000","last_active":"2024-02-19T21:11:06.900900+0000","last_peered":"2024-02-19T21:11:06.900900+0000","last_clean":"2024-02-19T21:11:06.900900+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:11:06.900900+0000","last_undegraded":"2024-02-19T21:11:06.900900+0000","last_fullsized":"2024-02-19T21:11:06.900900+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.851123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.851123+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":246,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.899457+0000","last_change":"2024-02-19T21:10:06.601328+0000","last_active":"2024-02-19T21:11:06.899457+0000","last_peered":"2024-02-19T21:11:06.899457+0000","last_clean":"2024-02-19T21:11:06.899457+0000","last_became_active":"2024-02-19T21:06:40.245385+0000","last_became_peered":"2024-02-19T21:06:40.245385+0000","last_unstale":"2024-02-19T21:11:06.899457+0000","last_undegraded":"2024-02-19T21:11:06.899457+0000","last_fullsized":"2024-02-19T21:11:06.899457+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:06.601228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:06.601228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":35,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.455546+0000","last_change":"2024-02-19T21:10:59.312931+0000","last_active":"2024-02-19T21:11:07.455546+0000","last_peered":"2024-02-19T21:11:07.455546+0000","last_clean":"2024-02-19T21:11:07.455546+0000","last_became_active":"2024-02-19T21:10:41.304695+0000","last_became_peered":"2024-02-19T21:10:41.304695+0000","last_unstale":"2024-02-19T21:11:07.455546+0000","last_undegraded":"2024-02-19T21:11:07.455546+0000","last_fullsized":"2024-02-19T21:11:07.455546+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:59.312863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.312863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"65'18","reported_seq":183,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683320+0000","last_change":"2024-02-19T21:11:02.671928+0000","last_active":"2024-02-19T21:11:06.683320+0000","last_peered":"2024-02-19T21:11:06.683320+0000","last_clean":"2024-02-19T21:11:06.683320+0000","last_became_active":"2024-02-19T21:08:04.242466+0000","last_became_peered":"2024-02-19T21:08:04.242466+0000","last_unstale":"2024-02-19T21:11:06.683320+0000","last_undegraded":"2024-02-19T21:11:06.683320+0000","last_fullsized":"2024-02-19T21:11:06.683320+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:11:02.671837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.671837+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":460,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.899392+0000","last_change":"2024-02-19T21:10:41.296353+0000","last_active":"2024-02-19T21:11:06.899392+0000","last_peered":"2024-02-19T21:11:06.899392+0000","last_clean":"2024-02-19T21:11:06.899392+0000","last_became_active":"2024-02-19T21:10:41.296010+0000","last_became_peered":"2024-02-19T21:10:41.296010+0000","last_unstale":"2024-02-19T21:11:06.899392+0000","last_undegraded":"2024-02-19T21:11:06.899392+0000","last_fullsized":"2024-02-19T21:11:06.899392+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":30,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.455755+0000","last_change":"2024-02-19T21:10:41.295616+0000","last_active":"2024-02-19T21:11:07.455755+0000","last_peered":"2024-02-19T21:11:07.455755+0000","last_clean":"2024-02-19T21:11:07.455755+0000","last_became_active":"2024-02-19T21:10:41.295301+0000","last_became_peered":"2024-02-19T21:10:41.295301+0000","last_unstale":"2024-02-19T21:11:07.455755+0000","last_undegraded":"2024-02-19T21:11:07.455755+0000","last_fullsized":"2024-02-19T21:11:07.455755+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:40.549690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.549690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":160,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683519+0000","last_change":"2024-02-19T21:08:04.267005+0000","last_active":"2024-02-19T21:11:06.683519+0000","last_peered":"2024-02-19T21:11:06.683519+0000","last_clean":"2024-02-19T21:11:06.683519+0000","last_became_active":"2024-02-19T21:08:04.266538+0000","last_became_peered":"2024-02-19T21:08:04.266538+0000","last_unstale":"2024-02-19T21:11:06.683519+0000","last_undegraded":"2024-02-19T21:11:06.683519+0000","last_fullsized":"2024-02-19T21:11:06.683519+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:58.847915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:58.847915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":420,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681039+0000","last_change":"2024-02-19T21:10:20.474926+0000","last_active":"2024-02-19T21:11:06.681039+0000","last_peered":"2024-02-19T21:11:06.681039+0000","last_clean":"2024-02-19T21:11:06.681039+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:11:06.681039+0000","last_undegraded":"2024-02-19T21:11:06.681039+0000","last_fullsized":"2024-02-19T21:11:06.681039+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:20.474844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:20.474844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"65'19","reported_seq":297,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058569+0000","last_change":"2024-02-19T21:10:11.166057+0000","last_active":"2024-02-19T21:11:07.058569+0000","last_peered":"2024-02-19T21:11:07.058569+0000","last_clean":"2024-02-19T21:11:07.058569+0000","last_became_active":"2024-02-19T21:06:40.223768+0000","last_became_peered":"2024-02-19T21:06:40.223768+0000","last_unstale":"2024-02-19T21:11:07.058569+0000","last_undegraded":"2024-02-19T21:11:07.058569+0000","last_fullsized":"2024-02-19T21:11:07.058569+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:10:11.165973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:11.165973+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":468,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.901167+0000","last_change":"2024-02-19T21:10:41.317015+0000","last_active":"2024-02-19T21:11:06.901167+0000","last_peered":"2024-02-19T21:11:06.901167+0000","last_clean":"2024-02-19T21:11:06.901167+0000","last_became_active":"2024-02-19T21:10:41.315864+0000","last_became_peered":"2024-02-19T21:10:41.315864+0000","last_unstale":"2024-02-19T21:11:06.901167+0000","last_undegraded":"2024-02-19T21:11:06.901167+0000","last_fullsized":"2024-02-19T21:11:06.901167+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:35.824520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:35.824520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":409,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898502+0000","last_change":"2024-02-19T21:08:57.653280+0000","last_active":"2024-02-19T21:11:06.898502+0000","last_peered":"2024-02-19T21:11:06.898502+0000","last_clean":"2024-02-19T21:11:06.898502+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:11:06.898502+0000","last_undegraded":"2024-02-19T21:11:06.898502+0000","last_fullsized":"2024-02-19T21:11:06.898502+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:57.653204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:08:57.653204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":370,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683283+0000","last_change":"2024-02-19T21:11:02.189586+0000","last_active":"2024-02-19T21:11:06.683283+0000","last_peered":"2024-02-19T21:11:06.683283+0000","last_clean":"2024-02-19T21:11:06.683283+0000","last_became_active":"2024-02-19T21:04:39.718926+0000","last_became_peered":"2024-02-19T21:04:39.718926+0000","last_unstale":"2024-02-19T21:11:06.683283+0000","last_undegraded":"2024-02-19T21:11:06.683283+0000","last_fullsized":"2024-02-19T21:11:06.683283+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.189509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":414,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.059490+0000","last_change":"2024-02-19T21:08:11.431726+0000","last_active":"2024-02-19T21:11:07.059490+0000","last_peered":"2024-02-19T21:11:07.059490+0000","last_clean":"2024-02-19T21:11:07.059490+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:11:07.059490+0000","last_undegraded":"2024-02-19T21:11:07.059490+0000","last_fullsized":"2024-02-19T21:11:07.059490+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:11.431637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:11.431637+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"65'12","reported_seq":276,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.899485+0000","last_change":"2024-02-19T21:09:30.704618+0000","last_active":"2024-02-19T21:11:06.899485+0000","last_peered":"2024-02-19T21:11:06.899485+0000","last_clean":"2024-02-19T21:11:06.899485+0000","last_became_active":"2024-02-19T21:06:40.240321+0000","last_became_peered":"2024-02-19T21:06:40.240321+0000","last_unstale":"2024-02-19T21:11:06.899485+0000","last_undegraded":"2024-02-19T21:11:06.899485+0000","last_fullsized":"2024-02-19T21:11:06.899485+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:09:30.704543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:30.704543+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"397'2","reported_seq":456,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.899943+0000","last_change":"2024-02-19T21:09:40.934534+0000","last_active":"2024-02-19T21:11:06.899943+0000","last_peered":"2024-02-19T21:11:06.899943+0000","last_clean":"2024-02-19T21:11:06.899943+0000","last_became_active":"2024-02-19T21:08:10.225480+0000","last_became_peered":"2024-02-19T21:08:10.225480+0000","last_unstale":"2024-02-19T21:11:06.899943+0000","last_undegraded":"2024-02-19T21:11:06.899943+0000","last_fullsized":"2024-02-19T21:11:06.899943+0000","mapping_epoch":57,"log_start":"58'1","ondisk_log_start":"58'1","created":57,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.934474+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":35,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.456227+0000","last_change":"2024-02-19T21:11:07.456130+0000","last_active":"2024-02-19T21:11:07.456227+0000","last_peered":"2024-02-19T21:11:07.456227+0000","last_clean":"2024-02-19T21:11:07.456227+0000","last_became_active":"2024-02-19T21:10:41.294427+0000","last_became_peered":"2024-02-19T21:10:41.294427+0000","last_unstale":"2024-02-19T21:11:07.456227+0000","last_undegraded":"2024-02-19T21:11:07.456227+0000","last_fullsized":"2024-02-19T21:11:07.456227+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.455993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.455993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":432,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898934+0000","last_change":"2024-02-19T21:10:51.691357+0000","last_active":"2024-02-19T21:11:06.898934+0000","last_peered":"2024-02-19T21:11:06.898934+0000","last_clean":"2024-02-19T21:11:06.898934+0000","last_became_active":"2024-02-19T21:08:04.282243+0000","last_became_peered":"2024-02-19T21:08:04.282243+0000","last_unstale":"2024-02-19T21:11:06.898934+0000","last_undegraded":"2024-02-19T21:11:06.898934+0000","last_fullsized":"2024-02-19T21:11:06.898934+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.691284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.691284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":369,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.680781+0000","last_change":"2024-02-19T21:10:21.484781+0000","last_active":"2024-02-19T21:11:06.680781+0000","last_peered":"2024-02-19T21:11:06.680781+0000","last_clean":"2024-02-19T21:11:06.680781+0000","last_became_active":"2024-02-19T21:04:39.662125+0000","last_became_peered":"2024-02-19T21:04:39.662125+0000","last_unstale":"2024-02-19T21:11:06.680781+0000","last_undegraded":"2024-02-19T21:11:06.680781+0000","last_fullsized":"2024-02-19T21:11:06.680781+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:21.484700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:21.484700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":417,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.900910+0000","last_change":"2024-02-19T21:10:03.998625+0000","last_active":"2024-02-19T21:11:06.900910+0000","last_peered":"2024-02-19T21:11:06.900910+0000","last_clean":"2024-02-19T21:11:06.900910+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:11:06.900910+0000","last_undegraded":"2024-02-19T21:11:06.900910+0000","last_fullsized":"2024-02-19T21:11:06.900910+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:03.998559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:03.998559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"65'14","reported_seq":489,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681475+0000","last_change":"2024-02-19T21:10:41.414468+0000","last_active":"2024-02-19T21:11:06.681475+0000","last_peered":"2024-02-19T21:11:06.681475+0000","last_clean":"2024-02-19T21:11:06.681475+0000","last_became_active":"2024-02-19T21:10:41.298131+0000","last_became_peered":"2024-02-19T21:10:41.298131+0000","last_unstale":"2024-02-19T21:11:06.681475+0000","last_undegraded":"2024-02-19T21:11:06.681475+0000","last_fullsized":"2024-02-19T21:11:06.681475+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:10:15.518555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:15.518555+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":430,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.899996+0000","last_change":"2024-02-19T21:11:03.747202+0000","last_active":"2024-02-19T21:11:06.899996+0000","last_peered":"2024-02-19T21:11:06.899996+0000","last_clean":"2024-02-19T21:11:06.899996+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:11:06.899996+0000","last_undegraded":"2024-02-19T21:11:06.899996+0000","last_fullsized":"2024-02-19T21:11:06.899996+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.747108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.747108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":450,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058505+0000","last_change":"2024-02-19T21:10:51.082102+0000","last_active":"2024-02-19T21:11:07.058505+0000","last_peered":"2024-02-19T21:11:07.058505+0000","last_clean":"2024-02-19T21:11:07.058505+0000","last_became_active":"2024-02-19T21:07:58.238944+0000","last_became_peered":"2024-02-19T21:07:58.238944+0000","last_unstale":"2024-02-19T21:11:07.058505+0000","last_undegraded":"2024-02-19T21:11:07.058505+0000","last_fullsized":"2024-02-19T21:11:07.058505+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.081993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.081993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683301+0000","last_change":"2024-02-19T21:08:53.820016+0000","last_active":"2024-02-19T21:11:06.683301+0000","last_peered":"2024-02-19T21:11:06.683301+0000","last_clean":"2024-02-19T21:11:06.683301+0000","last_became_active":"2024-02-19T21:08:04.278265+0000","last_became_peered":"2024-02-19T21:08:04.278265+0000","last_unstale":"2024-02-19T21:11:06.683301+0000","last_undegraded":"2024-02-19T21:11:06.683301+0000","last_fullsized":"2024-02-19T21:11:06.683301+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:53.819949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:53.819949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":453,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.059496+0000","last_change":"2024-02-19T21:10:41.315872+0000","last_active":"2024-02-19T21:11:07.059496+0000","last_peered":"2024-02-19T21:11:07.059496+0000","last_clean":"2024-02-19T21:11:07.059496+0000","last_became_active":"2024-02-19T21:10:41.315397+0000","last_became_peered":"2024-02-19T21:10:41.315397+0000","last_unstale":"2024-02-19T21:11:07.059496+0000","last_undegraded":"2024-02-19T21:11:07.059496+0000","last_fullsized":"2024-02-19T21:11:07.059496+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:10.148733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:10.148733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":371,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058237+0000","last_change":"2024-02-19T21:10:20.204982+0000","last_active":"2024-02-19T21:11:07.058237+0000","last_peered":"2024-02-19T21:11:07.058237+0000","last_clean":"2024-02-19T21:11:07.058237+0000","last_became_active":"2024-02-19T21:04:39.666739+0000","last_became_peered":"2024-02-19T21:04:39.666739+0000","last_unstale":"2024-02-19T21:11:07.058237+0000","last_undegraded":"2024-02-19T21:11:07.058237+0000","last_fullsized":"2024-02-19T21:11:07.058237+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_clean_scrub_stamp":"2024-02-19T21:10:20.204880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":418,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682300+0000","last_change":"2024-02-19T21:10:07.040966+0000","last_active":"2024-02-19T21:11:06.682300+0000","last_peered":"2024-02-19T21:11:06.682300+0000","last_clean":"2024-02-19T21:11:06.682300+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:11:06.682300+0000","last_undegraded":"2024-02-19T21:11:06.682300+0000","last_fullsized":"2024-02-19T21:11:06.682300+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:07.040893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:07.040893+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"86'18","reported_seq":436,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683281+0000","last_change":"2024-02-19T21:11:00.191283+0000","last_active":"2024-02-19T21:11:06.683281+0000","last_peered":"2024-02-19T21:11:06.683281+0000","last_clean":"2024-02-19T21:11:06.683281+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:11:06.683281+0000","last_undegraded":"2024-02-19T21:11:06.683281+0000","last_fullsized":"2024-02-19T21:11:06.683281+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:11:00.191206+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.191206+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681287+0000","last_change":"2024-02-19T21:09:37.955966+0000","last_active":"2024-02-19T21:11:06.681287+0000","last_peered":"2024-02-19T21:11:06.681287+0000","last_clean":"2024-02-19T21:11:06.681287+0000","last_became_active":"2024-02-19T21:08:06.473207+0000","last_became_peered":"2024-02-19T21:08:06.473207+0000","last_unstale":"2024-02-19T21:11:06.681287+0000","last_undegraded":"2024-02-19T21:11:06.681287+0000","last_fullsized":"2024-02-19T21:11:06.681287+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.955872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.955872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":1013,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-19T21:11:12.991738+0000","last_change":"2024-02-19T21:10:03.552765+0000","last_active":"2024-02-19T21:11:12.991738+0000","last_peered":"2024-02-19T21:11:12.991738+0000","last_clean":"2024-02-19T21:11:12.991738+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:11:12.991738+0000","last_undegraded":"2024-02-19T21:11:12.991738+0000","last_fullsized":"2024-02-19T21:11:12.991738+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:10:03.552690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:03.552690+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":590,"num_read_kb":585,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":433,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.901158+0000","last_change":"2024-02-19T21:10:07.884980+0000","last_active":"2024-02-19T21:11:06.901158+0000","last_peered":"2024-02-19T21:11:06.901158+0000","last_clean":"2024-02-19T21:11:06.901158+0000","last_became_active":"2024-02-19T21:08:04.264896+0000","last_became_peered":"2024-02-19T21:08:04.264896+0000","last_unstale":"2024-02-19T21:11:06.901158+0000","last_undegraded":"2024-02-19T21:11:06.901158+0000","last_fullsized":"2024-02-19T21:11:06.901158+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:07.884905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:07.884905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":368,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.947199+0000","last_change":"2024-02-19T21:11:07.947170+0000","last_active":"2024-02-19T21:11:07.947199+0000","last_peered":"2024-02-19T21:11:07.947199+0000","last_clean":"2024-02-19T21:11:07.947199+0000","last_became_active":"2024-02-19T21:04:39.691434+0000","last_became_peered":"2024-02-19T21:04:39.691434+0000","last_unstale":"2024-02-19T21:11:07.947199+0000","last_undegraded":"2024-02-19T21:11:07.947199+0000","last_fullsized":"2024-02-19T21:11:07.947199+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.947089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":264,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.899212+0000","last_change":"2024-02-19T21:10:41.296313+0000","last_active":"2024-02-19T21:11:06.899212+0000","last_peered":"2024-02-19T21:11:06.899212+0000","last_clean":"2024-02-19T21:11:06.899212+0000","last_became_active":"2024-02-19T21:10:41.295968+0000","last_became_peered":"2024-02-19T21:10:41.295968+0000","last_unstale":"2024-02-19T21:11:06.899212+0000","last_undegraded":"2024-02-19T21:11:06.899212+0000","last_fullsized":"2024-02-19T21:11:06.899212+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:36.716464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.716464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"304'30","reported_seq":648,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-19T21:11:09.707677+0000","last_change":"2024-02-19T21:10:01.545972+0000","last_active":"2024-02-19T21:11:09.707677+0000","last_peered":"2024-02-19T21:11:09.707677+0000","last_clean":"2024-02-19T21:11:09.707677+0000","last_became_active":"2024-02-19T21:08:04.266322+0000","last_became_peered":"2024-02-19T21:08:04.266322+0000","last_unstale":"2024-02-19T21:11:09.707677+0000","last_undegraded":"2024-02-19T21:11:09.707677+0000","last_fullsized":"2024-02-19T21:11:09.707677+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'30","last_scrub_stamp":"2024-02-19T21:10:01.545897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:01.545897+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":450,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058577+0000","last_change":"2024-02-19T21:10:52.089790+0000","last_active":"2024-02-19T21:11:07.058577+0000","last_peered":"2024-02-19T21:11:07.058577+0000","last_clean":"2024-02-19T21:11:07.058577+0000","last_became_active":"2024-02-19T21:08:02.435663+0000","last_became_peered":"2024-02-19T21:08:02.435663+0000","last_unstale":"2024-02-19T21:11:07.058577+0000","last_undegraded":"2024-02-19T21:11:07.058577+0000","last_fullsized":"2024-02-19T21:11:07.058577+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.089715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.089715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":178,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683084+0000","last_change":"2024-02-19T21:10:41.306137+0000","last_active":"2024-02-19T21:11:06.683084+0000","last_peered":"2024-02-19T21:11:06.683084+0000","last_clean":"2024-02-19T21:11:06.683084+0000","last_became_active":"2024-02-19T21:10:41.305829+0000","last_became_peered":"2024-02-19T21:10:41.305829+0000","last_unstale":"2024-02-19T21:11:06.683084+0000","last_undegraded":"2024-02-19T21:11:06.683084+0000","last_fullsized":"2024-02-19T21:11:06.683084+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:26.886817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:07:26.886817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":370,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681186+0000","last_change":"2024-02-19T21:10:54.142968+0000","last_active":"2024-02-19T21:11:06.681186+0000","last_peered":"2024-02-19T21:11:06.681186+0000","last_clean":"2024-02-19T21:11:06.681186+0000","last_became_active":"2024-02-19T21:04:39.721416+0000","last_became_peered":"2024-02-19T21:04:39.721416+0000","last_unstale":"2024-02-19T21:11:06.681186+0000","last_undegraded":"2024-02-19T21:11:06.681186+0000","last_fullsized":"2024-02-19T21:11:06.681186+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:54.142898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.142898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"220'9","reported_seq":323,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-19T21:11:09.708153+0000","last_change":"2024-02-19T21:10:12.455153+0000","last_active":"2024-02-19T21:11:09.708153+0000","last_peered":"2024-02-19T21:11:09.708153+0000","last_clean":"2024-02-19T21:11:09.708153+0000","last_became_active":"2024-02-19T21:06:40.227604+0000","last_became_peered":"2024-02-19T21:06:40.227604+0000","last_unstale":"2024-02-19T21:11:09.708153+0000","last_undegraded":"2024-02-19T21:11:09.708153+0000","last_fullsized":"2024-02-19T21:11:09.708153+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'9","last_scrub_stamp":"2024-02-19T21:10:12.455053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:12.455053+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"65'10","reported_seq":470,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898662+0000","last_change":"2024-02-19T21:10:41.747145+0000","last_active":"2024-02-19T21:11:06.898662+0000","last_peered":"2024-02-19T21:11:06.898662+0000","last_clean":"2024-02-19T21:11:06.898662+0000","last_became_active":"2024-02-19T21:10:41.296275+0000","last_became_peered":"2024-02-19T21:10:41.296275+0000","last_unstale":"2024-02-19T21:11:06.898662+0000","last_undegraded":"2024-02-19T21:11:06.898662+0000","last_fullsized":"2024-02-19T21:11:06.898662+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:09:28.659495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.659495+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":465,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898585+0000","last_change":"2024-02-19T21:10:21.732847+0000","last_active":"2024-02-19T21:11:06.898585+0000","last_peered":"2024-02-19T21:11:06.898585+0000","last_clean":"2024-02-19T21:11:06.898585+0000","last_became_active":"2024-02-19T21:07:54.235080+0000","last_became_peered":"2024-02-19T21:07:54.235080+0000","last_unstale":"2024-02-19T21:11:06.898585+0000","last_undegraded":"2024-02-19T21:11:06.898585+0000","last_fullsized":"2024-02-19T21:11:06.898585+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:21.732778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:10:21.732778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683272+0000","last_change":"2024-02-19T21:09:27.747880+0000","last_active":"2024-02-19T21:11:06.683272+0000","last_peered":"2024-02-19T21:11:06.683272+0000","last_clean":"2024-02-19T21:11:06.683272+0000","last_became_active":"2024-02-19T21:08:04.244317+0000","last_became_peered":"2024-02-19T21:08:04.244317+0000","last_unstale":"2024-02-19T21:11:06.683272+0000","last_undegraded":"2024-02-19T21:11:06.683272+0000","last_fullsized":"2024-02-19T21:11:06.683272+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.747789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.747789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683077+0000","last_change":"2024-02-19T21:11:05.604091+0000","last_active":"2024-02-19T21:11:06.683077+0000","last_peered":"2024-02-19T21:11:06.683077+0000","last_clean":"2024-02-19T21:11:06.683077+0000","last_became_active":"2024-02-19T21:08:04.261408+0000","last_became_peered":"2024-02-19T21:08:04.261408+0000","last_unstale":"2024-02-19T21:11:06.683077+0000","last_undegraded":"2024-02-19T21:11:06.683077+0000","last_fullsized":"2024-02-19T21:11:06.683077+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.604025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.604025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":418,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681018+0000","last_change":"2024-02-19T21:10:10.137424+0000","last_active":"2024-02-19T21:11:06.681018+0000","last_peered":"2024-02-19T21:11:06.681018+0000","last_clean":"2024-02-19T21:11:06.681018+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:11:06.681018+0000","last_undegraded":"2024-02-19T21:11:06.681018+0000","last_fullsized":"2024-02-19T21:11:06.681018+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:10.137356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:10:10.137356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"420'17","reported_seq":42,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.455661+0000","last_change":"2024-02-19T21:10:41.328739+0000","last_active":"2024-02-19T21:11:07.455661+0000","last_peered":"2024-02-19T21:11:07.455661+0000","last_clean":"2024-02-19T21:11:07.455661+0000","last_became_active":"2024-02-19T21:10:41.287134+0000","last_became_peered":"2024-02-19T21:10:41.287134+0000","last_unstale":"2024-02-19T21:11:07.455661+0000","last_undegraded":"2024-02-19T21:11:07.455661+0000","last_fullsized":"2024-02-19T21:11:07.455661+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'17","last_scrub_stamp":"2024-02-19T21:10:19.722303+0000","last_deep_scrub":"420'17","last_deep_scrub_stamp":"2024-02-19T21:10:19.722303+0000","last_clean_scrub_stamp":"2024-02-19T21:10:19.722303+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":41,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.456658+0000","last_change":"2024-02-19T21:10:58.333279+0000","last_active":"2024-02-19T21:11:07.456658+0000","last_peered":"2024-02-19T21:11:07.456658+0000","last_clean":"2024-02-19T21:11:07.456658+0000","last_became_active":"2024-02-19T21:10:41.302730+0000","last_became_peered":"2024-02-19T21:10:41.302730+0000","last_unstale":"2024-02-19T21:11:07.456658+0000","last_undegraded":"2024-02-19T21:11:07.456658+0000","last_fullsized":"2024-02-19T21:11:07.456658+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:10:58.333168+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.333168+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681442+0000","last_change":"2024-02-19T21:08:57.881002+0000","last_active":"2024-02-19T21:11:06.681442+0000","last_peered":"2024-02-19T21:11:06.681442+0000","last_clean":"2024-02-19T21:11:06.681442+0000","last_became_active":"2024-02-19T21:08:04.264593+0000","last_became_peered":"2024-02-19T21:08:04.264593+0000","last_unstale":"2024-02-19T21:11:06.681442+0000","last_undegraded":"2024-02-19T21:11:06.681442+0000","last_fullsized":"2024-02-19T21:11:06.681442+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:57.880935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:08:57.880935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":370,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682464+0000","last_change":"2024-02-19T21:11:03.195680+0000","last_active":"2024-02-19T21:11:06.682464+0000","last_peered":"2024-02-19T21:11:06.682464+0000","last_clean":"2024-02-19T21:11:06.682464+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:11:06.682464+0000","last_undegraded":"2024-02-19T21:11:06.682464+0000","last_fullsized":"2024-02-19T21:11:06.682464+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.195616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.195616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"65'9","reported_seq":177,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682936+0000","last_change":"2024-02-19T21:10:22.459882+0000","last_active":"2024-02-19T21:11:06.682936+0000","last_peered":"2024-02-19T21:11:06.682936+0000","last_clean":"2024-02-19T21:11:06.682936+0000","last_became_active":"2024-02-19T21:08:04.258740+0000","last_became_peered":"2024-02-19T21:08:04.258740+0000","last_unstale":"2024-02-19T21:11:06.682936+0000","last_undegraded":"2024-02-19T21:11:06.682936+0000","last_fullsized":"2024-02-19T21:11:06.682936+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:10:22.459807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:22.459807+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":422,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.244381+0000","last_change":"2024-02-19T21:11:07.244356+0000","last_active":"2024-02-19T21:11:07.244381+0000","last_peered":"2024-02-19T21:11:07.244381+0000","last_clean":"2024-02-19T21:11:07.244381+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:11:07.244381+0000","last_undegraded":"2024-02-19T21:11:07.244381+0000","last_fullsized":"2024-02-19T21:11:07.244381+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.244285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.244285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682521+0000","last_change":"2024-02-19T21:10:53.703797+0000","last_active":"2024-02-19T21:11:06.682521+0000","last_peered":"2024-02-19T21:11:06.682521+0000","last_clean":"2024-02-19T21:11:06.682521+0000","last_became_active":"2024-02-19T21:08:04.264972+0000","last_became_peered":"2024-02-19T21:08:04.264972+0000","last_unstale":"2024-02-19T21:11:06.682521+0000","last_undegraded":"2024-02-19T21:11:06.682521+0000","last_fullsized":"2024-02-19T21:11:06.682521+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:53.703727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.703727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":434,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682054+0000","last_change":"2024-02-19T21:10:41.314453+0000","last_active":"2024-02-19T21:11:06.682054+0000","last_peered":"2024-02-19T21:11:06.682054+0000","last_clean":"2024-02-19T21:11:06.682054+0000","last_became_active":"2024-02-19T21:10:41.314114+0000","last_became_peered":"2024-02-19T21:10:41.314114+0000","last_unstale":"2024-02-19T21:11:06.682054+0000","last_undegraded":"2024-02-19T21:11:06.682054+0000","last_fullsized":"2024-02-19T21:11:06.682054+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:48.995828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:08:48.995828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":30,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.455972+0000","last_change":"2024-02-19T21:10:41.303286+0000","last_active":"2024-02-19T21:11:07.455972+0000","last_peered":"2024-02-19T21:11:07.455972+0000","last_clean":"2024-02-19T21:11:07.455972+0000","last_became_active":"2024-02-19T21:10:41.300574+0000","last_became_peered":"2024-02-19T21:10:41.300574+0000","last_unstale":"2024-02-19T21:11:07.455972+0000","last_undegraded":"2024-02-19T21:11:07.455972+0000","last_fullsized":"2024-02-19T21:11:07.455972+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:14.631330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:14.631330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"78'9","reported_seq":44,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681906+0000","last_change":"2024-02-19T21:10:53.101489+0000","last_active":"2024-02-19T21:11:06.681906+0000","last_peered":"2024-02-19T21:11:06.681906+0000","last_clean":"2024-02-19T21:11:06.681906+0000","last_became_active":"2024-02-19T21:10:40.297867+0000","last_became_peered":"2024-02-19T21:10:40.297867+0000","last_unstale":"2024-02-19T21:11:06.681906+0000","last_undegraded":"2024-02-19T21:11:06.681906+0000","last_fullsized":"2024-02-19T21:11:06.681906+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:10:53.101422+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.101422+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":437,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681189+0000","last_change":"2024-02-19T21:10:41.311736+0000","last_active":"2024-02-19T21:11:06.681189+0000","last_peered":"2024-02-19T21:11:06.681189+0000","last_clean":"2024-02-19T21:11:06.681189+0000","last_became_active":"2024-02-19T21:10:41.311362+0000","last_became_peered":"2024-02-19T21:10:41.311362+0000","last_unstale":"2024-02-19T21:11:06.681189+0000","last_undegraded":"2024-02-19T21:11:06.681189+0000","last_fullsized":"2024-02-19T21:11:06.681189+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:41.519294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:41.519294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":434,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.059177+0000","last_change":"2024-02-19T21:08:56.601041+0000","last_active":"2024-02-19T21:11:07.059177+0000","last_peered":"2024-02-19T21:11:07.059177+0000","last_clean":"2024-02-19T21:11:07.059177+0000","last_became_active":"2024-02-19T21:08:04.265573+0000","last_became_peered":"2024-02-19T21:08:04.265573+0000","last_unstale":"2024-02-19T21:11:07.059177+0000","last_undegraded":"2024-02-19T21:11:07.059177+0000","last_fullsized":"2024-02-19T21:11:07.059177+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:56.600951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:56.600951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":433,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.901007+0000","last_change":"2024-02-19T21:10:41.316846+0000","last_active":"2024-02-19T21:11:06.901007+0000","last_peered":"2024-02-19T21:11:06.901007+0000","last_clean":"2024-02-19T21:11:06.901007+0000","last_became_active":"2024-02-19T21:10:41.315426+0000","last_became_peered":"2024-02-19T21:10:41.315426+0000","last_unstale":"2024-02-19T21:11:06.901007+0000","last_undegraded":"2024-02-19T21:11:06.901007+0000","last_fullsized":"2024-02-19T21:11:06.901007+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.770501+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":384,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058329+0000","last_change":"2024-02-19T21:10:41.315331+0000","last_active":"2024-02-19T21:11:07.058329+0000","last_peered":"2024-02-19T21:11:07.058329+0000","last_clean":"2024-02-19T21:11:07.058329+0000","last_became_active":"2024-02-19T21:10:41.314967+0000","last_became_peered":"2024-02-19T21:10:41.314967+0000","last_unstale":"2024-02-19T21:11:07.058329+0000","last_undegraded":"2024-02-19T21:11:07.058329+0000","last_fullsized":"2024-02-19T21:11:07.058329+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:50.600659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:50.600659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"65'10","reported_seq":177,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683475+0000","last_change":"2024-02-19T21:10:59.685223+0000","last_active":"2024-02-19T21:11:06.683475+0000","last_peered":"2024-02-19T21:11:06.683475+0000","last_clean":"2024-02-19T21:11:06.683475+0000","last_became_active":"2024-02-19T21:08:04.267216+0000","last_became_peered":"2024-02-19T21:08:04.267216+0000","last_unstale":"2024-02-19T21:11:06.683475+0000","last_undegraded":"2024-02-19T21:11:06.683475+0000","last_fullsized":"2024-02-19T21:11:06.683475+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:10:59.685150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.685150+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":441,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058489+0000","last_change":"2024-02-19T21:09:40.986717+0000","last_active":"2024-02-19T21:11:07.058489+0000","last_peered":"2024-02-19T21:11:07.058489+0000","last_clean":"2024-02-19T21:11:07.058489+0000","last_became_active":"2024-02-19T21:08:04.266728+0000","last_became_peered":"2024-02-19T21:08:04.266728+0000","last_unstale":"2024-02-19T21:11:07.058489+0000","last_undegraded":"2024-02-19T21:11:07.058489+0000","last_fullsized":"2024-02-19T21:11:07.058489+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.986635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"433'9","reported_seq":555,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-19T21:11:09.707942+0000","last_change":"2024-02-19T21:10:40.399539+0000","last_active":"2024-02-19T21:11:09.707942+0000","last_peered":"2024-02-19T21:11:09.707942+0000","last_clean":"2024-02-19T21:11:09.707942+0000","last_became_active":"2024-02-19T21:10:40.310303+0000","last_became_peered":"2024-02-19T21:10:40.310303+0000","last_unstale":"2024-02-19T21:11:09.707942+0000","last_undegraded":"2024-02-19T21:11:09.707942+0000","last_fullsized":"2024-02-19T21:11:09.707942+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:09:26.916002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:26.916002+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":283,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.901427+0000","last_change":"2024-02-19T21:08:57.007906+0000","last_active":"2024-02-19T21:11:06.901427+0000","last_peered":"2024-02-19T21:11:06.901427+0000","last_clean":"2024-02-19T21:11:06.901427+0000","last_became_active":"2024-02-19T21:08:04.266747+0000","last_became_peered":"2024-02-19T21:08:04.266747+0000","last_unstale":"2024-02-19T21:11:06.901427+0000","last_undegraded":"2024-02-19T21:11:06.901427+0000","last_fullsized":"2024-02-19T21:11:06.901427+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:08:57.007815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:08:57.007815+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":250,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.680559+0000","last_change":"2024-02-19T21:11:01.379002+0000","last_active":"2024-02-19T21:11:06.680559+0000","last_peered":"2024-02-19T21:11:06.680559+0000","last_clean":"2024-02-19T21:11:06.680559+0000","last_became_active":"2024-02-19T21:06:40.219590+0000","last_became_peered":"2024-02-19T21:06:40.219590+0000","last_unstale":"2024-02-19T21:11:06.680559+0000","last_undegraded":"2024-02-19T21:11:06.680559+0000","last_fullsized":"2024-02-19T21:11:06.680559+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.378925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:55.202463+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.378925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"65'9","reported_seq":475,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058933+0000","last_change":"2024-02-19T21:10:50.053312+0000","last_active":"2024-02-19T21:11:07.058933+0000","last_peered":"2024-02-19T21:11:07.058933+0000","last_clean":"2024-02-19T21:11:07.058933+0000","last_became_active":"2024-02-19T21:08:04.266103+0000","last_became_peered":"2024-02-19T21:08:04.266103+0000","last_unstale":"2024-02-19T21:11:07.058933+0000","last_undegraded":"2024-02-19T21:11:07.058933+0000","last_fullsized":"2024-02-19T21:11:07.058933+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:10:50.053229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.053229+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":35,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.456044+0000","last_change":"2024-02-19T21:10:56.327430+0000","last_active":"2024-02-19T21:11:07.456044+0000","last_peered":"2024-02-19T21:11:07.456044+0000","last_clean":"2024-02-19T21:11:07.456044+0000","last_became_active":"2024-02-19T21:10:41.295618+0000","last_became_peered":"2024-02-19T21:10:41.295618+0000","last_unstale":"2024-02-19T21:11:07.456044+0000","last_undegraded":"2024-02-19T21:11:07.456044+0000","last_fullsized":"2024-02-19T21:11:07.456044+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:56.327321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:56.327321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":208,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-19T21:11:09.707686+0000","last_change":"2024-02-19T21:08:37.867477+0000","last_active":"2024-02-19T21:11:09.707686+0000","last_peered":"2024-02-19T21:11:09.707686+0000","last_clean":"2024-02-19T21:11:09.707686+0000","last_became_active":"2024-02-19T21:08:04.256045+0000","last_became_peered":"2024-02-19T21:08:04.256045+0000","last_unstale":"2024-02-19T21:11:09.707686+0000","last_undegraded":"2024-02-19T21:11:09.707686+0000","last_fullsized":"2024-02-19T21:11:09.707686+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'10","last_scrub_stamp":"2024-02-19T21:08:37.867405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:37.867405+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":30,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.456653+0000","last_change":"2024-02-19T21:10:41.312156+0000","last_active":"2024-02-19T21:11:07.456653+0000","last_peered":"2024-02-19T21:11:07.456653+0000","last_clean":"2024-02-19T21:11:07.456653+0000","last_became_active":"2024-02-19T21:10:41.310279+0000","last_became_peered":"2024-02-19T21:10:41.310279+0000","last_unstale":"2024-02-19T21:11:07.456653+0000","last_undegraded":"2024-02-19T21:11:07.456653+0000","last_fullsized":"2024-02-19T21:11:07.456653+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_clean_scrub_stamp":"2024-02-19T21:09:43.936153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":409,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681069+0000","last_change":"2024-02-19T21:11:00.839892+0000","last_active":"2024-02-19T21:11:06.681069+0000","last_peered":"2024-02-19T21:11:06.681069+0000","last_clean":"2024-02-19T21:11:06.681069+0000","last_became_active":"2024-02-19T21:11:00.752591+0000","last_became_peered":"2024-02-19T21:11:00.752591+0000","last_unstale":"2024-02-19T21:11:06.681069+0000","last_undegraded":"2024-02-19T21:11:06.681069+0000","last_fullsized":"2024-02-19T21:11:06.681069+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-02-19T21:10:14.015412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:14.015412+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"65'9","reported_seq":150,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898320+0000","last_change":"2024-02-19T21:09:40.676522+0000","last_active":"2024-02-19T21:11:06.898320+0000","last_peered":"2024-02-19T21:11:06.898320+0000","last_clean":"2024-02-19T21:11:06.898320+0000","last_became_active":"2024-02-19T21:08:26.235018+0000","last_became_peered":"2024-02-19T21:08:26.235018+0000","last_unstale":"2024-02-19T21:11:06.898320+0000","last_undegraded":"2024-02-19T21:11:06.898320+0000","last_fullsized":"2024-02-19T21:11:06.898320+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:09:40.676439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.676439+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":442,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681283+0000","last_change":"2024-02-19T21:10:57.422515+0000","last_active":"2024-02-19T21:11:06.681283+0000","last_peered":"2024-02-19T21:11:06.681283+0000","last_clean":"2024-02-19T21:11:06.681283+0000","last_became_active":"2024-02-19T21:08:04.258699+0000","last_became_peered":"2024-02-19T21:08:04.258699+0000","last_unstale":"2024-02-19T21:11:06.681283+0000","last_undegraded":"2024-02-19T21:11:06.681283+0000","last_fullsized":"2024-02-19T21:11:06.681283+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.422323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.422323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681518+0000","last_change":"2024-02-19T21:08:12.856127+0000","last_active":"2024-02-19T21:11:06.681518+0000","last_peered":"2024-02-19T21:11:06.681518+0000","last_clean":"2024-02-19T21:11:06.681518+0000","last_became_active":"2024-02-19T21:08:04.264983+0000","last_became_peered":"2024-02-19T21:08:04.264983+0000","last_unstale":"2024-02-19T21:11:06.681518+0000","last_undegraded":"2024-02-19T21:11:06.681518+0000","last_fullsized":"2024-02-19T21:11:06.681518+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.856015+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":249,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058318+0000","last_change":"2024-02-19T21:09:38.941012+0000","last_active":"2024-02-19T21:11:07.058318+0000","last_peered":"2024-02-19T21:11:07.058318+0000","last_clean":"2024-02-19T21:11:07.058318+0000","last_became_active":"2024-02-19T21:06:40.239199+0000","last_became_peered":"2024-02-19T21:06:40.239199+0000","last_unstale":"2024-02-19T21:11:07.058318+0000","last_undegraded":"2024-02-19T21:11:07.058318+0000","last_fullsized":"2024-02-19T21:11:07.058318+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:38.940935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:38.940935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":369,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681694+0000","last_change":"2024-02-19T21:10:51.493283+0000","last_active":"2024-02-19T21:11:06.681694+0000","last_peered":"2024-02-19T21:11:06.681694+0000","last_clean":"2024-02-19T21:11:06.681694+0000","last_became_active":"2024-02-19T21:04:39.672228+0000","last_became_peered":"2024-02-19T21:04:39.672228+0000","last_unstale":"2024-02-19T21:11:06.681694+0000","last_undegraded":"2024-02-19T21:11:06.681694+0000","last_fullsized":"2024-02-19T21:11:06.681694+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.493206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.493206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"65'11","reported_seq":42,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.456229+0000","last_change":"2024-02-19T21:10:41.335974+0000","last_active":"2024-02-19T21:11:07.456229+0000","last_peered":"2024-02-19T21:11:07.456229+0000","last_clean":"2024-02-19T21:11:07.456229+0000","last_became_active":"2024-02-19T21:10:41.297317+0000","last_became_peered":"2024-02-19T21:10:41.297317+0000","last_unstale":"2024-02-19T21:11:07.456229+0000","last_undegraded":"2024-02-19T21:11:07.456229+0000","last_fullsized":"2024-02-19T21:11:07.456229+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:52.201972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:52.201972+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":420,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.899302+0000","last_change":"2024-02-19T21:09:32.623020+0000","last_active":"2024-02-19T21:11:06.899302+0000","last_peered":"2024-02-19T21:11:06.899302+0000","last_clean":"2024-02-19T21:11:06.899302+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:11:06.899302+0000","last_undegraded":"2024-02-19T21:11:06.899302+0000","last_fullsized":"2024-02-19T21:11:06.899302+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:32.622939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:09:32.622939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":434,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898745+0000","last_change":"2024-02-19T21:09:34.642491+0000","last_active":"2024-02-19T21:11:06.898745+0000","last_peered":"2024-02-19T21:11:06.898745+0000","last_clean":"2024-02-19T21:11:06.898745+0000","last_became_active":"2024-02-19T21:08:04.268847+0000","last_became_peered":"2024-02-19T21:08:04.268847+0000","last_unstale":"2024-02-19T21:11:06.898745+0000","last_undegraded":"2024-02-19T21:11:06.898745+0000","last_fullsized":"2024-02-19T21:11:06.898745+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:34.642432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:34.642432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":160,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682637+0000","last_change":"2024-02-19T21:08:04.280460+0000","last_active":"2024-02-19T21:11:06.682637+0000","last_peered":"2024-02-19T21:11:06.682637+0000","last_clean":"2024-02-19T21:11:06.682637+0000","last_became_active":"2024-02-19T21:08:04.280233+0000","last_became_peered":"2024-02-19T21:08:04.280233+0000","last_unstale":"2024-02-19T21:11:06.682637+0000","last_undegraded":"2024-02-19T21:11:06.682637+0000","last_fullsized":"2024-02-19T21:11:06.682637+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:59.446267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:07:59.446267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.599616+0000","last_change":"2024-02-19T21:11:07.599578+0000","last_active":"2024-02-19T21:11:07.599616+0000","last_peered":"2024-02-19T21:11:07.599616+0000","last_clean":"2024-02-19T21:11:07.599616+0000","last_became_active":"2024-02-19T21:08:04.258568+0000","last_became_peered":"2024-02-19T21:08:04.258568+0000","last_unstale":"2024-02-19T21:11:07.599616+0000","last_undegraded":"2024-02-19T21:11:07.599616+0000","last_fullsized":"2024-02-19T21:11:07.599616+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.599474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.599474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"65'4","reported_seq":37,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.057993+0000","last_change":"2024-02-19T21:10:40.651979+0000","last_active":"2024-02-19T21:11:07.057993+0000","last_peered":"2024-02-19T21:11:07.057993+0000","last_clean":"2024-02-19T21:11:07.057993+0000","last_became_active":"2024-02-19T21:10:40.298769+0000","last_became_peered":"2024-02-19T21:10:40.298769+0000","last_unstale":"2024-02-19T21:11:07.057993+0000","last_undegraded":"2024-02-19T21:11:07.057993+0000","last_fullsized":"2024-02-19T21:11:07.057993+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:09:28.712037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.712037+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":423,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.059239+0000","last_change":"2024-02-19T21:09:42.000906+0000","last_active":"2024-02-19T21:11:07.059239+0000","last_peered":"2024-02-19T21:11:07.059239+0000","last_clean":"2024-02-19T21:11:07.059239+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:11:07.059239+0000","last_undegraded":"2024-02-19T21:11:07.059239+0000","last_fullsized":"2024-02-19T21:11:07.059239+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:42.000850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:42.000850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":438,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681216+0000","last_change":"2024-02-19T21:10:52.467547+0000","last_active":"2024-02-19T21:11:06.681216+0000","last_peered":"2024-02-19T21:11:06.681216+0000","last_clean":"2024-02-19T21:11:06.681216+0000","last_became_active":"2024-02-19T21:10:41.311989+0000","last_became_peered":"2024-02-19T21:10:41.311989+0000","last_unstale":"2024-02-19T21:11:06.681216+0000","last_undegraded":"2024-02-19T21:11:06.681216+0000","last_fullsized":"2024-02-19T21:11:06.681216+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.467476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.467476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":433,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.900345+0000","last_change":"2024-02-19T21:10:41.314970+0000","last_active":"2024-02-19T21:11:06.900345+0000","last_peered":"2024-02-19T21:11:06.900345+0000","last_clean":"2024-02-19T21:11:06.900345+0000","last_became_active":"2024-02-19T21:10:41.314589+0000","last_became_peered":"2024-02-19T21:10:41.314589+0000","last_unstale":"2024-02-19T21:11:06.900345+0000","last_undegraded":"2024-02-19T21:11:06.900345+0000","last_fullsized":"2024-02-19T21:11:06.900345+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:26.731860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:26.731860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683498+0000","last_change":"2024-02-19T21:08:10.824761+0000","last_active":"2024-02-19T21:11:06.683498+0000","last_peered":"2024-02-19T21:11:06.683498+0000","last_clean":"2024-02-19T21:11:06.683498+0000","last_became_active":"2024-02-19T21:08:04.266161+0000","last_became_peered":"2024-02-19T21:08:04.266161+0000","last_unstale":"2024-02-19T21:11:06.683498+0000","last_undegraded":"2024-02-19T21:11:06.683498+0000","last_fullsized":"2024-02-19T21:11:06.683498+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:10.824694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:10.824694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"65'11","reported_seq":177,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682847+0000","last_change":"2024-02-19T21:08:11.828624+0000","last_active":"2024-02-19T21:11:06.682847+0000","last_peered":"2024-02-19T21:11:06.682847+0000","last_clean":"2024-02-19T21:11:06.682847+0000","last_became_active":"2024-02-19T21:08:04.250742+0000","last_became_peered":"2024-02-19T21:08:04.250742+0000","last_unstale":"2024-02-19T21:11:06.682847+0000","last_undegraded":"2024-02-19T21:11:06.682847+0000","last_fullsized":"2024-02-19T21:11:06.682847+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:11.828537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:11.828537+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":424,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681768+0000","last_change":"2024-02-19T21:11:00.383023+0000","last_active":"2024-02-19T21:11:06.681768+0000","last_peered":"2024-02-19T21:11:06.681768+0000","last_clean":"2024-02-19T21:11:06.681768+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:11:06.681768+0000","last_undegraded":"2024-02-19T21:11:06.681768+0000","last_fullsized":"2024-02-19T21:11:06.681768+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:00.382946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.382946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":430,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.900221+0000","last_change":"2024-02-19T21:10:41.307832+0000","last_active":"2024-02-19T21:11:06.900221+0000","last_peered":"2024-02-19T21:11:06.900221+0000","last_clean":"2024-02-19T21:11:06.900221+0000","last_became_active":"2024-02-19T21:10:41.307598+0000","last_became_peered":"2024-02-19T21:10:41.307598+0000","last_unstale":"2024-02-19T21:11:06.900221+0000","last_undegraded":"2024-02-19T21:11:06.900221+0000","last_fullsized":"2024-02-19T21:11:06.900221+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:29.783609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:29.783609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682560+0000","last_change":"2024-02-19T21:10:00.547305+0000","last_active":"2024-02-19T21:11:06.682560+0000","last_peered":"2024-02-19T21:11:06.682560+0000","last_clean":"2024-02-19T21:11:06.682560+0000","last_became_active":"2024-02-19T21:08:04.251987+0000","last_became_peered":"2024-02-19T21:08:04.251987+0000","last_unstale":"2024-02-19T21:11:06.682560+0000","last_undegraded":"2024-02-19T21:11:06.682560+0000","last_fullsized":"2024-02-19T21:11:06.682560+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:00.547228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:00.547228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":250,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681611+0000","last_change":"2024-02-19T21:10:16.529711+0000","last_active":"2024-02-19T21:11:06.681611+0000","last_peered":"2024-02-19T21:11:06.681611+0000","last_clean":"2024-02-19T21:11:06.681611+0000","last_became_active":"2024-02-19T21:06:40.226196+0000","last_became_peered":"2024-02-19T21:06:40.226196+0000","last_unstale":"2024-02-19T21:11:06.681611+0000","last_undegraded":"2024-02-19T21:11:06.681611+0000","last_fullsized":"2024-02-19T21:11:06.681611+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:16.529623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:10:16.529623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"65'10","reported_seq":317,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.901522+0000","last_change":"2024-02-19T21:09:30.782123+0000","last_active":"2024-02-19T21:11:06.901522+0000","last_peered":"2024-02-19T21:11:06.901522+0000","last_clean":"2024-02-19T21:11:06.901522+0000","last_became_active":"2024-02-19T21:08:04.267222+0000","last_became_peered":"2024-02-19T21:08:04.267222+0000","last_unstale":"2024-02-19T21:11:06.901522+0000","last_undegraded":"2024-02-19T21:11:06.901522+0000","last_fullsized":"2024-02-19T21:11:06.901522+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:09:30.782002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:30.782002+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":436,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058293+0000","last_change":"2024-02-19T21:10:41.294958+0000","last_active":"2024-02-19T21:11:07.058293+0000","last_peered":"2024-02-19T21:11:07.058293+0000","last_clean":"2024-02-19T21:11:07.058293+0000","last_became_active":"2024-02-19T21:10:41.294592+0000","last_became_peered":"2024-02-19T21:10:41.294592+0000","last_unstale":"2024-02-19T21:11:07.058293+0000","last_undegraded":"2024-02-19T21:11:07.058293+0000","last_fullsized":"2024-02-19T21:11:07.058293+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:52.579178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:52.579178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"434'10","reported_seq":578,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-02-19T21:11:09.708127+0000","last_change":"2024-02-19T21:10:59.405489+0000","last_active":"2024-02-19T21:11:09.708127+0000","last_peered":"2024-02-19T21:11:09.708127+0000","last_clean":"2024-02-19T21:11:09.708127+0000","last_became_active":"2024-02-19T21:10:41.302307+0000","last_became_peered":"2024-02-19T21:10:41.302307+0000","last_unstale":"2024-02-19T21:11:09.708127+0000","last_undegraded":"2024-02-19T21:11:09.708127+0000","last_fullsized":"2024-02-19T21:11:09.708127+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'10","last_scrub_stamp":"2024-02-19T21:10:59.405417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.405417+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":417,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.901028+0000","last_change":"2024-02-19T21:09:41.951642+0000","last_active":"2024-02-19T21:11:06.901028+0000","last_peered":"2024-02-19T21:11:06.901028+0000","last_clean":"2024-02-19T21:11:06.901028+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:11:06.901028+0000","last_undegraded":"2024-02-19T21:11:06.901028+0000","last_fullsized":"2024-02-19T21:11:06.901028+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:09:41.951576+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:09:41.951576+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":368,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682507+0000","last_change":"2024-02-19T21:08:39.107325+0000","last_active":"2024-02-19T21:11:06.682507+0000","last_peered":"2024-02-19T21:11:06.682507+0000","last_clean":"2024-02-19T21:11:06.682507+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:11:06.682507+0000","last_undegraded":"2024-02-19T21:11:06.682507+0000","last_fullsized":"2024-02-19T21:11:06.682507+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:39.107240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.107240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"65'11","reported_seq":441,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681837+0000","last_change":"2024-02-19T21:10:54.435913+0000","last_active":"2024-02-19T21:11:06.681837+0000","last_peered":"2024-02-19T21:11:06.681837+0000","last_clean":"2024-02-19T21:11:06.681837+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:11:06.681837+0000","last_undegraded":"2024-02-19T21:11:06.681837+0000","last_fullsized":"2024-02-19T21:11:06.681837+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:10:54.435839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.435839+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":160,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682963+0000","last_change":"2024-02-19T21:08:04.254214+0000","last_active":"2024-02-19T21:11:06.682963+0000","last_peered":"2024-02-19T21:11:06.682963+0000","last_clean":"2024-02-19T21:11:06.682963+0000","last_became_active":"2024-02-19T21:08:04.251241+0000","last_became_peered":"2024-02-19T21:08:04.251241+0000","last_unstale":"2024-02-19T21:11:06.682963+0000","last_undegraded":"2024-02-19T21:11:06.682963+0000","last_fullsized":"2024-02-19T21:11:06.682963+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:01.530618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:01.530618+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":253,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.901610+0000","last_change":"2024-02-19T21:11:01.692337+0000","last_active":"2024-02-19T21:11:06.901610+0000","last_peered":"2024-02-19T21:11:06.901610+0000","last_clean":"2024-02-19T21:11:06.901610+0000","last_became_active":"2024-02-19T21:06:40.234026+0000","last_became_peered":"2024-02-19T21:06:40.234026+0000","last_unstale":"2024-02-19T21:11:06.901610+0000","last_undegraded":"2024-02-19T21:11:06.901610+0000","last_fullsized":"2024-02-19T21:11:06.901610+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.692260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.692260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":427,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.899180+0000","last_change":"2024-02-19T21:10:41.313086+0000","last_active":"2024-02-19T21:11:06.899180+0000","last_peered":"2024-02-19T21:11:06.899180+0000","last_clean":"2024-02-19T21:11:06.899180+0000","last_became_active":"2024-02-19T21:10:41.312577+0000","last_became_peered":"2024-02-19T21:10:41.312577+0000","last_unstale":"2024-02-19T21:11:06.899180+0000","last_undegraded":"2024-02-19T21:11:06.899180+0000","last_fullsized":"2024-02-19T21:11:06.899180+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:48.760625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:08:48.760625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":386,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898843+0000","last_change":"2024-02-19T21:10:59.782435+0000","last_active":"2024-02-19T21:11:06.898843+0000","last_peered":"2024-02-19T21:11:06.898843+0000","last_clean":"2024-02-19T21:11:06.898843+0000","last_became_active":"2024-02-19T21:10:41.311050+0000","last_became_peered":"2024-02-19T21:10:41.311050+0000","last_unstale":"2024-02-19T21:11:06.898843+0000","last_undegraded":"2024-02-19T21:11:06.898843+0000","last_fullsized":"2024-02-19T21:11:06.898843+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:59.782367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.782367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"419'22","reported_seq":502,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682279+0000","last_change":"2024-02-19T21:10:19.109353+0000","last_active":"2024-02-19T21:11:06.682279+0000","last_peered":"2024-02-19T21:11:06.682279+0000","last_clean":"2024-02-19T21:11:06.682279+0000","last_became_active":"2024-02-19T21:08:04.248378+0000","last_became_peered":"2024-02-19T21:08:04.248378+0000","last_unstale":"2024-02-19T21:11:06.682279+0000","last_undegraded":"2024-02-19T21:11:06.682279+0000","last_fullsized":"2024-02-19T21:11:06.682279+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'22","last_scrub_stamp":"2024-02-19T21:10:19.109244+0000","last_deep_scrub":"419'22","last_deep_scrub_stamp":"2024-02-19T21:10:19.109244+0000","last_clean_scrub_stamp":"2024-02-19T21:10:19.109244+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":442,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681014+0000","last_change":"2024-02-19T21:11:03.379435+0000","last_active":"2024-02-19T21:11:06.681014+0000","last_peered":"2024-02-19T21:11:06.681014+0000","last_clean":"2024-02-19T21:11:06.681014+0000","last_became_active":"2024-02-19T21:10:41.311203+0000","last_became_peered":"2024-02-19T21:10:41.311203+0000","last_unstale":"2024-02-19T21:11:06.681014+0000","last_undegraded":"2024-02-19T21:11:06.681014+0000","last_fullsized":"2024-02-19T21:11:06.681014+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.379335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.379335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":434,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.899432+0000","last_change":"2024-02-19T21:10:53.709519+0000","last_active":"2024-02-19T21:11:06.899432+0000","last_peered":"2024-02-19T21:11:06.899432+0000","last_clean":"2024-02-19T21:11:06.899432+0000","last_became_active":"2024-02-19T21:10:41.312789+0000","last_became_peered":"2024-02-19T21:10:41.312789+0000","last_unstale":"2024-02-19T21:11:06.899432+0000","last_undegraded":"2024-02-19T21:11:06.899432+0000","last_fullsized":"2024-02-19T21:11:06.899432+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:53.709456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.709456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":35,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.456633+0000","last_change":"2024-02-19T21:10:57.335233+0000","last_active":"2024-02-19T21:11:07.456633+0000","last_peered":"2024-02-19T21:11:07.456633+0000","last_clean":"2024-02-19T21:11:07.456633+0000","last_became_active":"2024-02-19T21:10:41.313042+0000","last_became_peered":"2024-02-19T21:10:41.313042+0000","last_unstale":"2024-02-19T21:11:07.456633+0000","last_undegraded":"2024-02-19T21:11:07.456633+0000","last_fullsized":"2024-02-19T21:11:07.456633+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.335152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.335152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":250,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.680501+0000","last_change":"2024-02-19T21:11:02.342277+0000","last_active":"2024-02-19T21:11:06.680501+0000","last_peered":"2024-02-19T21:11:06.680501+0000","last_clean":"2024-02-19T21:11:06.680501+0000","last_became_active":"2024-02-19T21:06:40.240859+0000","last_became_peered":"2024-02-19T21:06:40.240859+0000","last_unstale":"2024-02-19T21:11:06.680501+0000","last_undegraded":"2024-02-19T21:11:06.680501+0000","last_fullsized":"2024-02-19T21:11:06.680501+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.342173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.342173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"65'12","reported_seq":219,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682128+0000","last_change":"2024-02-19T21:10:41.664579+0000","last_active":"2024-02-19T21:11:06.682128+0000","last_peered":"2024-02-19T21:11:06.682128+0000","last_clean":"2024-02-19T21:11:06.682128+0000","last_became_active":"2024-02-19T21:10:41.307690+0000","last_became_peered":"2024-02-19T21:10:41.307690+0000","last_unstale":"2024-02-19T21:11:06.682128+0000","last_undegraded":"2024-02-19T21:11:06.682128+0000","last_fullsized":"2024-02-19T21:11:06.682128+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:08:49.869588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.869588+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681602+0000","last_change":"2024-02-19T21:10:04.546052+0000","last_active":"2024-02-19T21:11:06.681602+0000","last_peered":"2024-02-19T21:11:06.681602+0000","last_clean":"2024-02-19T21:11:06.681602+0000","last_became_active":"2024-02-19T21:08:04.253058+0000","last_became_peered":"2024-02-19T21:08:04.253058+0000","last_unstale":"2024-02-19T21:11:06.681602+0000","last_undegraded":"2024-02-19T21:11:06.681602+0000","last_fullsized":"2024-02-19T21:11:06.681602+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:04.545988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:04.545988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":30,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.456120+0000","last_change":"2024-02-19T21:10:41.303228+0000","last_active":"2024-02-19T21:11:07.456120+0000","last_peered":"2024-02-19T21:11:07.456120+0000","last_clean":"2024-02-19T21:11:07.456120+0000","last_became_active":"2024-02-19T21:10:41.296849+0000","last_became_peered":"2024-02-19T21:10:41.296849+0000","last_unstale":"2024-02-19T21:11:07.456120+0000","last_undegraded":"2024-02-19T21:11:07.456120+0000","last_fullsized":"2024-02-19T21:11:07.456120+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:35.626548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:35.626548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682675+0000","last_change":"2024-02-19T21:10:58.688232+0000","last_active":"2024-02-19T21:11:06.682675+0000","last_peered":"2024-02-19T21:11:06.682675+0000","last_clean":"2024-02-19T21:11:06.682675+0000","last_became_active":"2024-02-19T21:08:04.278045+0000","last_became_peered":"2024-02-19T21:08:04.278045+0000","last_unstale":"2024-02-19T21:11:06.682675+0000","last_undegraded":"2024-02-19T21:11:06.682675+0000","last_fullsized":"2024-02-19T21:11:06.682675+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:58.688166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.688166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":368,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.899099+0000","last_change":"2024-02-19T21:11:00.754397+0000","last_active":"2024-02-19T21:11:06.899099+0000","last_peered":"2024-02-19T21:11:06.899099+0000","last_clean":"2024-02-19T21:11:06.899099+0000","last_became_active":"2024-02-19T21:04:39.777442+0000","last_became_peered":"2024-02-19T21:04:39.777442+0000","last_unstale":"2024-02-19T21:11:06.899099+0000","last_undegraded":"2024-02-19T21:11:06.899099+0000","last_fullsized":"2024-02-19T21:11:06.899099+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:00.754327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.754327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"194'13","reported_seq":40,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.455837+0000","last_change":"2024-02-19T21:10:41.325454+0000","last_active":"2024-02-19T21:11:07.455837+0000","last_peered":"2024-02-19T21:11:07.455837+0000","last_clean":"2024-02-19T21:11:07.455837+0000","last_became_active":"2024-02-19T21:10:41.295673+0000","last_became_peered":"2024-02-19T21:10:41.295673+0000","last_unstale":"2024-02-19T21:11:07.455837+0000","last_undegraded":"2024-02-19T21:11:07.455837+0000","last_fullsized":"2024-02-19T21:11:07.455837+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'13","last_scrub_stamp":"2024-02-19T21:10:08.605366+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:10:08.605366+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":416,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.900810+0000","last_change":"2024-02-19T21:08:51.955830+0000","last_active":"2024-02-19T21:11:06.900810+0000","last_peered":"2024-02-19T21:11:06.900810+0000","last_clean":"2024-02-19T21:11:06.900810+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:11:06.900810+0000","last_undegraded":"2024-02-19T21:11:06.900810+0000","last_fullsized":"2024-02-19T21:11:06.900810+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:51.955738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:51.955738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":414,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058468+0000","last_change":"2024-02-19T21:09:29.852892+0000","last_active":"2024-02-19T21:11:07.058468+0000","last_peered":"2024-02-19T21:11:07.058468+0000","last_clean":"2024-02-19T21:11:07.058468+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:11:07.058468+0000","last_undegraded":"2024-02-19T21:11:07.058468+0000","last_fullsized":"2024-02-19T21:11:07.058468+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:29.852792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:29.852792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":246,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.897916+0000","last_change":"2024-02-19T21:10:10.642674+0000","last_active":"2024-02-19T21:11:06.897916+0000","last_peered":"2024-02-19T21:11:06.897916+0000","last_clean":"2024-02-19T21:11:06.897916+0000","last_became_active":"2024-02-19T21:06:40.239603+0000","last_became_peered":"2024-02-19T21:06:40.239603+0000","last_unstale":"2024-02-19T21:11:06.897916+0000","last_undegraded":"2024-02-19T21:11:06.897916+0000","last_fullsized":"2024-02-19T21:11:06.897916+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:10.642603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:10:10.642603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":388,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682551+0000","last_change":"2024-02-19T21:10:41.307666+0000","last_active":"2024-02-19T21:11:06.682551+0000","last_peered":"2024-02-19T21:11:06.682551+0000","last_clean":"2024-02-19T21:11:06.682551+0000","last_became_active":"2024-02-19T21:10:41.307352+0000","last_became_peered":"2024-02-19T21:10:41.307352+0000","last_unstale":"2024-02-19T21:11:06.682551+0000","last_undegraded":"2024-02-19T21:11:06.682551+0000","last_fullsized":"2024-02-19T21:11:06.682551+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:21.194709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:21.194709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"65'5","reported_seq":36,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.456240+0000","last_change":"2024-02-19T21:10:41.767925+0000","last_active":"2024-02-19T21:11:07.456240+0000","last_peered":"2024-02-19T21:11:07.456240+0000","last_clean":"2024-02-19T21:11:07.456240+0000","last_became_active":"2024-02-19T21:10:41.297295+0000","last_became_peered":"2024-02-19T21:10:41.297295+0000","last_unstale":"2024-02-19T21:11:07.456240+0000","last_undegraded":"2024-02-19T21:11:07.456240+0000","last_fullsized":"2024-02-19T21:11:07.456240+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:09:44.596326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:44.596326+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":421,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.901004+0000","last_change":"2024-02-19T21:09:25.733055+0000","last_active":"2024-02-19T21:11:06.901004+0000","last_peered":"2024-02-19T21:11:06.901004+0000","last_clean":"2024-02-19T21:11:06.901004+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:11:06.901004+0000","last_undegraded":"2024-02-19T21:11:06.901004+0000","last_fullsized":"2024-02-19T21:11:06.901004+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:25.732955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:25.732955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":435,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.900499+0000","last_change":"2024-02-19T21:10:41.307635+0000","last_active":"2024-02-19T21:11:06.900499+0000","last_peered":"2024-02-19T21:11:06.900499+0000","last_clean":"2024-02-19T21:11:06.900499+0000","last_became_active":"2024-02-19T21:10:41.307400+0000","last_became_peered":"2024-02-19T21:10:41.307400+0000","last_unstale":"2024-02-19T21:11:06.900499+0000","last_undegraded":"2024-02-19T21:11:06.900499+0000","last_fullsized":"2024-02-19T21:11:06.900499+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:36.862297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:36.862297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":435,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.058425+0000","last_change":"2024-02-19T21:10:15.161470+0000","last_active":"2024-02-19T21:11:07.058425+0000","last_peered":"2024-02-19T21:11:07.058425+0000","last_clean":"2024-02-19T21:11:07.058425+0000","last_became_active":"2024-02-19T21:08:04.266100+0000","last_became_peered":"2024-02-19T21:08:04.266100+0000","last_unstale":"2024-02-19T21:11:07.058425+0000","last_undegraded":"2024-02-19T21:11:07.058425+0000","last_fullsized":"2024-02-19T21:11:07.058425+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:15.161397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:15.161397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":254,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.059257+0000","last_change":"2024-02-19T21:10:56.083471+0000","last_active":"2024-02-19T21:11:07.059257+0000","last_peered":"2024-02-19T21:11:07.059257+0000","last_clean":"2024-02-19T21:11:07.059257+0000","last_became_active":"2024-02-19T21:06:40.227713+0000","last_became_peered":"2024-02-19T21:06:40.227713+0000","last_unstale":"2024-02-19T21:11:07.059257+0000","last_undegraded":"2024-02-19T21:11:07.059257+0000","last_fullsized":"2024-02-19T21:11:07.059257+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:56.083403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:56.083403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'9","reported_seq":45,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.455425+0000","last_change":"2024-02-19T21:11:03.423846+0000","last_active":"2024-02-19T21:11:07.455425+0000","last_peered":"2024-02-19T21:11:07.455425+0000","last_clean":"2024-02-19T21:11:07.455425+0000","last_became_active":"2024-02-19T21:10:41.298840+0000","last_became_peered":"2024-02-19T21:10:41.298840+0000","last_unstale":"2024-02-19T21:11:07.455425+0000","last_undegraded":"2024-02-19T21:11:07.455425+0000","last_fullsized":"2024-02-19T21:11:07.455425+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:11:03.423767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.423767+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.683161+0000","last_change":"2024-02-19T21:10:50.769725+0000","last_active":"2024-02-19T21:11:06.683161+0000","last_peered":"2024-02-19T21:11:06.683161+0000","last_clean":"2024-02-19T21:11:06.683161+0000","last_became_active":"2024-02-19T21:08:04.277576+0000","last_became_peered":"2024-02-19T21:08:04.277576+0000","last_unstale":"2024-02-19T21:11:06.683161+0000","last_undegraded":"2024-02-19T21:11:06.683161+0000","last_fullsized":"2024-02-19T21:11:06.683161+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:50.769620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.769620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":416,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.898383+0000","last_change":"2024-02-19T21:10:18.737245+0000","last_active":"2024-02-19T21:11:06.898383+0000","last_peered":"2024-02-19T21:11:06.898383+0000","last_clean":"2024-02-19T21:11:06.898383+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:11:06.898383+0000","last_undegraded":"2024-02-19T21:11:06.898383+0000","last_fullsized":"2024-02-19T21:11:06.898383+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:18.737176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:10:18.737176+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":30,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.455867+0000","last_change":"2024-02-19T21:10:41.309872+0000","last_active":"2024-02-19T21:11:07.455867+0000","last_peered":"2024-02-19T21:11:07.455867+0000","last_clean":"2024-02-19T21:11:07.455867+0000","last_became_active":"2024-02-19T21:10:41.309550+0000","last_became_peered":"2024-02-19T21:10:41.309550+0000","last_unstale":"2024-02-19T21:11:07.455867+0000","last_undegraded":"2024-02-19T21:11:07.455867+0000","last_fullsized":"2024-02-19T21:11:07.455867+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:05.182671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:05.182671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":389,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:08.043175+0000","last_change":"2024-02-19T21:11:08.043134+0000","last_active":"2024-02-19T21:11:08.043175+0000","last_peered":"2024-02-19T21:11:08.043175+0000","last_clean":"2024-02-19T21:11:08.043175+0000","last_became_active":"2024-02-19T21:10:41.315817+0000","last_became_peered":"2024-02-19T21:10:41.315817+0000","last_unstale":"2024-02-19T21:11:08.043175+0000","last_undegraded":"2024-02-19T21:11:08.043175+0000","last_fullsized":"2024-02-19T21:11:08.043175+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:08.043028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:08.043028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"65'15","reported_seq":181,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.682561+0000","last_change":"2024-02-19T21:10:02.532862+0000","last_active":"2024-02-19T21:11:06.682561+0000","last_peered":"2024-02-19T21:11:06.682561+0000","last_clean":"2024-02-19T21:11:06.682561+0000","last_became_active":"2024-02-19T21:08:04.249443+0000","last_became_peered":"2024-02-19T21:08:04.249443+0000","last_unstale":"2024-02-19T21:11:06.682561+0000","last_undegraded":"2024-02-19T21:11:06.682561+0000","last_fullsized":"2024-02-19T21:11:06.682561+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:10:02.532799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:02.532799+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":273,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.901702+0000","last_change":"2024-02-19T21:11:02.720658+0000","last_active":"2024-02-19T21:11:06.901702+0000","last_peered":"2024-02-19T21:11:06.901702+0000","last_clean":"2024-02-19T21:11:06.901702+0000","last_became_active":"2024-02-19T21:10:41.315636+0000","last_became_peered":"2024-02-19T21:10:41.315636+0000","last_unstale":"2024-02-19T21:11:06.901702+0000","last_undegraded":"2024-02-19T21:11:06.901702+0000","last_fullsized":"2024-02-19T21:11:06.901702+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.720583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.720583+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":35,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:07.455484+0000","last_change":"2024-02-19T21:11:02.407624+0000","last_active":"2024-02-19T21:11:07.455484+0000","last_peered":"2024-02-19T21:11:07.455484+0000","last_clean":"2024-02-19T21:11:07.455484+0000","last_became_active":"2024-02-19T21:10:41.297843+0000","last_became_peered":"2024-02-19T21:10:41.297843+0000","last_unstale":"2024-02-19T21:11:07.455484+0000","last_undegraded":"2024-02-19T21:11:07.455484+0000","last_fullsized":"2024-02-19T21:11:07.455484+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.407543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.407543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681502+0000","last_change":"2024-02-19T21:09:33.559448+0000","last_active":"2024-02-19T21:11:06.681502+0000","last_peered":"2024-02-19T21:11:06.681502+0000","last_clean":"2024-02-19T21:11:06.681502+0000","last_became_active":"2024-02-19T21:08:04.252640+0000","last_became_peered":"2024-02-19T21:08:04.252640+0000","last_unstale":"2024-02-19T21:11:06.681502+0000","last_undegraded":"2024-02-19T21:11:06.681502+0000","last_fullsized":"2024-02-19T21:11:06.681502+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:33.559363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:33.559363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":385,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-02-19T21:11:06.681371+0000","last_change":"2024-02-19T21:10:11.412403+0000","last_active":"2024-02-19T21:11:06.681371+0000","last_peered":"2024-02-19T21:11:06.681371+0000","last_clean":"2024-02-19T21:11:06.681371+0000","last_became_active":"2024-02-19T21:08:04.250937+0000","last_became_peered":"2024-02-19T21:08:04.250937+0000","last_unstale":"2024-02-19T21:11:06.681371+0000","last_undegraded":"2024-02-19T21:11:06.681371+0000","last_fullsized":"2024-02-19T21:11:06.681371+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:11.412276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:11.412276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":217,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":4001538,"num_objects":165,"num_object_clones":5,"num_object_copies":495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":109,"num_read_kb":10,"num_write":614,"num_write_kb":4513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":740,"num_bytes_recovered":1812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":12,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13979648,"data_stored":12004647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1930,"ondisk_log_size":1930,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":676,"num_read_kb":433,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":5204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":422,"ondisk_log_size":422,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":590,"num_read_kb":585,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266810,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316440,"kb_used_data":11016,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93428712,"statfs":{"total":95995035648,"available":95671001088,"internally_reserved":0,"allocated":11280384,"data_stored":8183829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:10:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":1.45,"15min":0.86599999999999999},"min":{"1min":0.28299999999999997,"5min":0.215,"15min":0.215},"max":{"1min":0.70099999999999996,"5min":89.456999999999994,"15min":89.456999999999994},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":1.2390000000000001,"15min":0.78400000000000003},"min":{"1min":0.308,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.65300000000000002,"5min":70.451999999999998,"15min":70.451999999999998},"last":0.39600000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:10:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.85599999999999998,"15min":0.69999999999999996},"min":{"1min":0.33300000000000002,"5min":0.28299999999999997,"15min":0.28199999999999997},"max":{"1min":0.63600000000000001,"5min":34.273000000000003,"15min":34.273000000000003},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":1.4510000000000001,"15min":0.93700000000000006},"min":{"1min":0.33100000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.67100000000000004,"5min":89.594999999999999,"15min":89.594999999999999},"last":0.41799999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:10:23 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":1.621,"15min":0.93999999999999995},"min":{"1min":0.314,"5min":0.27600000000000002,"15min":0.24299999999999999},"max":{"1min":0.68899999999999995,"5min":104.413,"15min":104.413},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.505,"5min":1.647,"15min":0.94899999999999995},"min":{"1min":0.36499999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.66200000000000003,"5min":104.268,"15min":104.268},"last":0.48699999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:10:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":1.679,"15min":0.95599999999999996},"min":{"1min":0.34899999999999998,"5min":0.29099999999999998,"15min":0.254},"max":{"1min":1.8939999999999999,"5min":105.3,"15min":105.3},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.64400000000000002,"15min":0.59699999999999998},"min":{"1min":0.371,"5min":0.32700000000000001,"15min":0.315},"max":{"1min":0.63100000000000001,"5min":13.243,"15min":13.243},"last":0.432}]},{"osd":4,"last update":"Mon Feb 19 21:10:23 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.434,"15min":0.47099999999999997},"min":{"1min":0.28499999999999998,"5min":0.27400000000000002,"15min":0.24299999999999999},"max":{"1min":0.64500000000000002,"5min":0.69299999999999995,"15min":1.373},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.46000000000000002,"15min":0.47799999999999998},"min":{"1min":0.29599999999999999,"5min":0.28000000000000003,"15min":0.25900000000000001},"max":{"1min":0.751,"5min":0.751,"15min":1.0740000000000001},"last":0.56200000000000006}]},{"osd":5,"last update":"Mon Feb 19 21:10:23 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.45000000000000001,"15min":0.45800000000000002},"min":{"1min":0.32400000000000001,"5min":0.28599999999999998,"15min":0.252},"max":{"1min":0.63700000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.48899999999999999,"15min":0.50600000000000001},"min":{"1min":0.308,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70599999999999996,"5min":0.77400000000000002,"15min":0.85899999999999999},"last":0.54300000000000004}]},{"osd":6,"last update":"Mon Feb 19 21:10:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.495,"15min":0.52000000000000002},"min":{"1min":0.33000000000000002,"5min":0.32500000000000001,"15min":0.28799999999999998},"max":{"1min":0.72999999999999998,"5min":0.76500000000000001,"15min":1.0049999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52100000000000002,"15min":0.55500000000000005},"min":{"1min":0.35699999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66900000000000004,"5min":0.75900000000000001,"15min":1.262},"last":0.61199999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528448,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308420,"kb_used_data":7096,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436732,"statfs":{"total":95995035648,"available":95679213568,"internally_reserved":0,"allocated":7266304,"data_stored":4182321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.49099999999999999,"15min":0.61399999999999999},"min":{"1min":0.26900000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.67700000000000005,"5min":2.9129999999999998,"15min":17.486999999999998},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52900000000000003,"15min":0.65400000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.64500000000000002,"5min":2.8999999999999999,"15min":17.43},"last":0.60399999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.502,"15min":0.61799999999999999},"min":{"1min":0.311,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.66700000000000004,"5min":2.0270000000000001,"15min":14.692},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.497,"15min":0.66200000000000003},"min":{"1min":0.318,"5min":0.29099999999999998,"15min":0.23400000000000001},"max":{"1min":0.68400000000000005,"5min":1.069,"15min":14.795},"last":0.51500000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.60399999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.73599999999999999,"5min":2.0979999999999999,"15min":6.931},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.66800000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67400000000000004,"5min":0.71899999999999997,"15min":14.391},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.54100000000000004,"15min":0.69899999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.29199999999999998},"max":{"1min":1.1499999999999999,"5min":3.0499999999999998,"15min":14.510999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53700000000000003,"15min":0.65700000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.316},"max":{"1min":0.91600000000000004,"5min":1.927,"15min":17.585000000000001},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Feb 19 21:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.437,"15min":0.40500000000000003},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23200000000000001},"max":{"1min":2.9289999999999998,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.46200000000000002,"15min":0.42999999999999999},"min":{"1min":0.27000000000000002,"5min":0.25,"15min":0.22700000000000001},"max":{"1min":3.431,"5min":3.431,"15min":3.431},"last":0.498}]},{"osd":5,"last update":"Mon Feb 19 21:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.495,"15min":0.45000000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.52800000000000002,"15min":0.48399999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.7010000000000001,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Feb 19 21:10:29 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.48799999999999999,"15min":0.502},"min":{"1min":0.32400000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.69299999999999995,"5min":0.80300000000000005,"15min":1.1160000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.48499999999999999,"15min":0.50800000000000001},"min":{"1min":0.33900000000000002,"5min":0.30099999999999999,"15min":0.28399999999999997},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":1.534},"last":0.58599999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574999,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312336,"kb_used_data":11008,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93432816,"statfs":{"total":95995035648,"available":95675203584,"internally_reserved":0,"allocated":11272192,"data_stored":8184205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32100000000000001,"15min":0.54400000000000004},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.46999999999999997,"5min":0.58499999999999996,"15min":27.777000000000001},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.33600000000000002,"15min":0.57099999999999995},"min":{"1min":0.216,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.55500000000000005,"5min":0.70899999999999996,"15min":27.631},"last":0.20200000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35299999999999998,"15min":0.70799999999999996},"min":{"1min":0.247,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.502,"5min":0.66300000000000003,"15min":27.798999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36499999999999999,"15min":0.72199999999999998},"min":{"1min":0.254,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.68500000000000005,"5min":0.73299999999999998,"15min":27.606999999999999},"last":0.26600000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.36299999999999999,"15min":0.57299999999999995},"min":{"1min":0.251,"5min":0.217,"15min":0.217},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":15.574999999999999},"last":0.254},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39300000000000002,"15min":0.66600000000000004},"min":{"1min":0.29199999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.64700000000000002,"5min":0.74399999999999999,"15min":27.82},"last":0.32600000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:10:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.64400000000000002,"15min":0.65800000000000003},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.73899999999999999,"5min":9.6430000000000007,"15min":9.6430000000000007},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.68300000000000005,"15min":0.752},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.77100000000000002,"5min":11.119,"15min":27.707000000000001},"last":0.41299999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52700000000000002,"15min":0.746},"min":{"1min":0.318,"5min":0.314,"15min":0.314},"max":{"1min":0.75700000000000001,"5min":0.94199999999999995,"15min":27.529},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53000000000000003,"15min":0.69099999999999995},"min":{"1min":0.38700000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":9.6799999999999997},"last":0.44500000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53800000000000003,"15min":0.67000000000000004},"min":{"1min":0.35899999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":9.6989999999999998},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57599999999999996,"15min":0.78800000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81699999999999995,"5min":2.0470000000000002,"15min":27.733000000000001},"last":0.48099999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.55500000000000005,"15min":0.77300000000000002},"min":{"1min":0.371,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77400000000000002,"5min":0.92000000000000004,"15min":27.576000000000001},"last":0.371},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57999999999999996,"15min":0.80800000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76100000000000001,"5min":2.0169999999999999,"15min":27.756},"last":0.503}]}]},{"osd":0,"up_from":8,"seq":34359738525,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308360,"kb_used_data":7032,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436792,"statfs":{"total":95995035648,"available":95679275008,"internally_reserved":0,"allocated":7200768,"data_stored":4183292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:11:12 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35699999999999998,"15min":0.60499999999999998},"min":{"1min":0.21299999999999999,"5min":0.20499999999999999,"15min":0.16200000000000001},"max":{"1min":0.49199999999999999,"5min":2.2189999999999999,"15min":20.966999999999999},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.40100000000000002,"15min":0.61399999999999999},"min":{"1min":0.23999999999999999,"5min":0.193,"15min":0.188},"max":{"1min":0.60799999999999998,"5min":4.8570000000000002,"15min":20.366},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.47499999999999998,"15min":0.71399999999999997},"min":{"1min":0.23400000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.47699999999999998,"5min":4.4660000000000002,"15min":27.916},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.373,"5min":0.42899999999999999,"15min":0.63200000000000001},"min":{"1min":0.252,"5min":0.218,"15min":0.218},"max":{"1min":0.58199999999999996,"5min":4.2949999999999999,"15min":22.004000000000001},"last":0.24399999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:10:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.44,"15min":0.748},"min":{"1min":0.26800000000000002,"5min":0.248,"15min":0.23200000000000001},"max":{"1min":0.5,"5min":5.9569999999999999,"15min":25.669},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.46200000000000002,"15min":0.69499999999999995},"min":{"1min":0.26800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.55000000000000004,"5min":4.1029999999999998,"15min":25.006},"last":0.28100000000000003}]},{"osd":4,"last update":"Mon Feb 19 21:11:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.71999999999999997},"min":{"1min":0.32500000000000001,"5min":0.32100000000000001,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.3899999999999999,"15min":20.411999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.59499999999999997,"15min":0.75},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.83299999999999996,"5min":4.3730000000000002,"15min":19.986999999999998},"last":0.499}]},{"osd":5,"last update":"Mon Feb 19 21:10:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.61299999999999999,"15min":0.82899999999999996},"min":{"1min":0.46400000000000002,"5min":0.35499999999999998,"15min":0.27200000000000002},"max":{"1min":0.79900000000000004,"5min":4.4349999999999996,"15min":20.021999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.67100000000000004,"15min":0.89500000000000002},"min":{"1min":0.46700000000000003,"5min":0.317,"15min":0.30399999999999999},"max":{"1min":0.79100000000000004,"5min":6.2859999999999996,"15min":20.908999999999999},"last":0.42299999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60699999999999998,"15min":0.76200000000000001},"min":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.33900000000000002},"max":{"1min":0.78900000000000003,"5min":2.3279999999999998,"15min":20.445},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.65300000000000002,"15min":0.81200000000000006},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.33400000000000002},"max":{"1min":0.72599999999999998,"5min":4.4080000000000004,"15min":20.047999999999998},"last":0.48199999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:10:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.65700000000000003,"15min":0.88},"min":{"1min":0.433,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81299999999999994,"5min":4.1479999999999997,"15min":20.940999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.63100000000000001,"15min":0.82399999999999995},"min":{"1min":0.39400000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0840000000000001,"5min":3.5030000000000001,"15min":20.471},"last":0.373}]}]},{"osd":2,"up_from":18,"seq":77309411474,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308208,"kb_used_data":10980,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93436944,"statfs":{"total":95995035648,"available":95679430656,"internally_reserved":0,"allocated":11243520,"data_stored":8183089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:10:29 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.41399999999999998,"15min":0.53900000000000003},"min":{"1min":0.22900000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.54600000000000004,"5min":4.665,"15min":26.690999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.48699999999999999,"15min":0.59099999999999997},"min":{"1min":0.24299999999999999,"5min":0.23200000000000001,"15min":0.214},"max":{"1min":0.47599999999999998,"5min":7.3849999999999998,"15min":24.315000000000001},"last":0.373}]},{"osd":1,"last update":"Mon Feb 19 21:10:29 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.67500000000000004,"15min":0.70599999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20100000000000001},"max":{"1min":0.45100000000000001,"5min":28.882000000000001,"15min":28.882000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.68799999999999994,"15min":0.68400000000000005},"min":{"1min":0.23999999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":0.59099999999999997,"5min":28.382999999999999,"15min":28.382999999999999},"last":0.34799999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.88100000000000001,"15min":0.72099999999999997},"min":{"1min":0.27100000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.66800000000000004,"5min":28.742000000000001,"15min":28.742000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.45300000000000001,"15min":0.64200000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.25700000000000001},"max":{"1min":1.0009999999999999,"5min":3.6499999999999999,"15min":26.719999999999999},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:10:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.84499999999999997,"15min":0.70499999999999996},"min":{"1min":0.37,"5min":0.34000000000000002,"15min":0.30599999999999999},"max":{"1min":0.755,"5min":24.356000000000002,"15min":24.356000000000002},"last":0.497},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.88600000000000001,"15min":0.73499999999999999},"min":{"1min":0.38100000000000001,"5min":0.32800000000000001,"15min":0.32400000000000001},"max":{"1min":0.88300000000000001,"5min":20.707000000000001,"15min":20.707000000000001},"last":0.44800000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:10:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.73299999999999998,"15min":0.69499999999999995},"min":{"1min":0.36399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.71599999999999997,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.67300000000000004,"15min":0.79900000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.38400000000000001},"max":{"1min":0.75800000000000001,"5min":4.9359999999999999,"15min":26.606000000000002},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:11:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.83599999999999997,"15min":0.80600000000000005},"min":{"1min":0.35299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81200000000000006,"5min":20.617000000000001,"15min":24.381},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.88600000000000001,"15min":0.74099999999999999},"min":{"1min":0.38800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.79800000000000004,"5min":28.706,"15min":28.706},"last":0.47999999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:10:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.68200000000000005,"15min":0.77700000000000002},"min":{"1min":0.42199999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89200000000000002,"5min":4.9139999999999997,"15min":26.658000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.625,"15min":0.73499999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.80500000000000005,"5min":4.258,"15min":24.402000000000001},"last":0.47199999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182542,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304272,"kb_used_data":7048,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93440880,"statfs":{"total":95995035648,"available":95683461120,"internally_reserved":0,"allocated":7217152,"data_stored":4182920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.33200000000000002,"15min":0.71699999999999997},"min":{"1min":0.219,"5min":0.19400000000000001,"15min":0.156},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":77.662000000000006},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.34399999999999997,"15min":0.502},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.19700000000000001},"max":{"1min":0.60999999999999999,"5min":0.72999999999999998,"15min":17.456},"last":0.52800000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40400000000000003,"15min":0.53300000000000003},"min":{"1min":0.27500000000000002,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":0.58299999999999996,"5min":6.9299999999999997,"15min":17.327999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.46100000000000002,"15min":0.51100000000000001},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.62,"5min":7.3620000000000001,"15min":15.004},"last":0.51500000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40699999999999997,"15min":0.66900000000000004},"min":{"1min":0.26400000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.65100000000000002,"5min":3.9319999999999999,"15min":56.890999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36599999999999999,"15min":0.65200000000000002},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.20899999999999999},"max":{"1min":0.59499999999999997,"5min":0.91900000000000004,"15min":44.600999999999999},"last":0.57499999999999996}]},{"osd":4,"last update":"Mon Feb 19 21:10:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.69799999999999995,"15min":0.63800000000000001},"min":{"1min":0.39800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71399999999999997,"5min":12.847,"15min":12.847},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55900000000000005,"15min":0.71499999999999997},"min":{"1min":0.35199999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73999999999999999,"5min":3.294,"15min":26.393999999999998},"last":0.625}]},{"osd":5,"last update":"Mon Feb 19 21:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.62,"15min":0.69199999999999995},"min":{"1min":0.373,"5min":0.313,"15min":0.313},"max":{"1min":0.93500000000000005,"5min":9.2360000000000007,"15min":17.401},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.58399999999999996,"15min":0.65900000000000003},"min":{"1min":0.35499999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.83199999999999996,"5min":4.0890000000000004,"15min":12.882},"last":0.70299999999999996}]},{"osd":6,"last update":"Mon Feb 19 21:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56499999999999995,"15min":0.65800000000000003},"min":{"1min":0.41599999999999998,"5min":0.34799999999999998,"15min":0.34300000000000003},"max":{"1min":0.82599999999999996,"5min":1.9790000000000001,"15min":17.367000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.59499999999999997,"15min":0.66900000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":0.92600000000000005,"5min":4.024,"15min":12.904},"last":0.63200000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:10:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56100000000000005,"15min":0.65800000000000003},"min":{"1min":0.42899999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":17.428999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57599999999999996,"15min":0.80200000000000005},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.91300000000000003,"5min":0.997,"15min":44.378},"last":0.65500000000000003}]}]},{"osd":4,"up_from":31,"seq":133143986314,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308444,"kb_used_data":7120,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436708,"statfs":{"total":95995035648,"available":95679188992,"internally_reserved":0,"allocated":7290880,"data_stored":4183144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.67600000000000005,"15min":0.624},"min":{"1min":0.30299999999999999,"5min":0.23599999999999999,"15min":0.22500000000000001},"max":{"1min":0.72599999999999998,"5min":19.878,"15min":19.878},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.88900000000000001,"15min":0.85999999999999999},"min":{"1min":0.25900000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68999999999999995,"5min":17.675000000000001,"15min":70.063000000000002},"last":0.58599999999999997}]},{"osd":1,"last update":"Mon Feb 19 21:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.59499999999999997,"15min":0.64600000000000002},"min":{"1min":0.29899999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.76200000000000001,"5min":9.1920000000000002,"15min":26.510999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.505,"5min":0.91200000000000003,"15min":0.68700000000000006},"min":{"1min":0.316,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.75800000000000001,"5min":25.949000000000002,"15min":25.949000000000002},"last":0.67400000000000004}]},{"osd":2,"last update":"Mon Feb 19 21:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.63300000000000001,"15min":0.70199999999999996},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.625,"5min":7.516,"15min":37.320999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.498,"5min":1.3720000000000001,"15min":1.0620000000000001},"min":{"1min":0.29899999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.77400000000000002,"5min":69.936999999999998,"15min":69.936999999999998},"last":0.71499999999999997}]},{"osd":3,"last update":"Mon Feb 19 21:10:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":1.393,"15min":0.90500000000000003},"min":{"1min":0.38900000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70099999999999996,"5min":69.793000000000006,"15min":69.793000000000006},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.80600000000000005,"15min":0.76800000000000002},"min":{"1min":0.33400000000000002,"5min":0.29499999999999998,"15min":0.29099999999999998},"max":{"1min":0.76700000000000002,"5min":17.986999999999998,"15min":17.986999999999998},"last":0.60799999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39700000000000002,"15min":0.42199999999999999},"min":{"1min":0.24299999999999999,"5min":0.23999999999999999,"15min":0.218},"max":{"1min":0.71199999999999997,"5min":0.75,"15min":3.2799999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.45300000000000001,"15min":0.45600000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.23899999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":4.4340000000000002},"last":0.53600000000000003}]},{"osd":6,"last update":"Mon Feb 19 21:11:05 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.54700000000000004,"15min":0.45800000000000002},"min":{"1min":0.27800000000000002,"5min":0.27600000000000002,"15min":0.24399999999999999},"max":{"1min":8.0969999999999995,"5min":8.0969999999999995,"15min":8.0969999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.54000000000000004,"15min":0.495},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":3.0270000000000001,"5min":3.0270000000000001,"15min":4.5609999999999999},"last":0.64300000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.314,"15min":0.29999999999999999},"max":{"1min":0.84999999999999998,"5min":1.004,"15min":6.0730000000000004},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.52800000000000002,"15min":0.5},"min":{"1min":0.36399999999999999,"5min":0.31,"15min":0.26400000000000001},"max":{"1min":0.81999999999999995,"5min":1.0209999999999999,"15min":3.202},"last":0.624}]}]},{"osd":5,"up_from":37,"seq":158913790085,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312548,"kb_used_data":7128,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93432604,"statfs":{"total":95995035648,"available":95674986496,"internally_reserved":0,"allocated":7299072,"data_stored":4181881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:10:41 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.45200000000000001,"15min":1.8859999999999999},"min":{"1min":0.24099999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":276.47899999999998},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.49099999999999999,"15min":0.91200000000000003},"min":{"1min":0.23799999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59799999999999998,"5min":0.95399999999999996,"15min":52.887},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:10:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.50700000000000001,"15min":1.278},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.23499999999999999},"max":{"1min":0.69699999999999995,"5min":3.3159999999999998,"15min":120.011},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.51500000000000001,"15min":1.0269999999999999},"min":{"1min":0.32700000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64400000000000002,"5min":1.7470000000000001,"15min":118.105},"last":0.55500000000000005}]},{"osd":2,"last update":"Mon Feb 19 21:10:41 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.52100000000000002,"15min":1.0009999999999999},"min":{"1min":0.33800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.629,"5min":3.2130000000000001,"15min":82.242000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.56000000000000005,"15min":1.5089999999999999},"min":{"1min":0.32300000000000001,"5min":0.309,"15min":0.30199999999999999},"max":{"1min":0.63400000000000001,"5min":4.5019999999999998,"15min":232.84999999999999},"last":0.61799999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:10:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.54900000000000004,"15min":1.071},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":0.68200000000000005,"5min":4.4640000000000004,"15min":109.70099999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.52500000000000002,"15min":0.91100000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67200000000000004,"5min":2.6019999999999999,"15min":65.519000000000005},"last":0.59299999999999997}]},{"osd":4,"last update":"Mon Feb 19 21:10:41 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45300000000000001,"15min":0.41699999999999998},"min":{"1min":0.32000000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61099999999999999,"5min":1.861,"15min":1.861},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.46500000000000002,"15min":0.42699999999999999},"min":{"1min":0.31,"5min":0.28199999999999997,"15min":0.248},"max":{"1min":0.64700000000000002,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":0.44500000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48399999999999999,"15min":0.44900000000000001},"min":{"1min":0.28999999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.871,"5min":1.823,"15min":1.823},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48299999999999998,"15min":0.45800000000000002},"min":{"1min":0.30399999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69899999999999995,"5min":0.71399999999999997,"15min":0.73399999999999999},"last":0.40100000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:10:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51500000000000001,"15min":0.504},"min":{"1min":0.35499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65700000000000003,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.432},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51100000000000001,"15min":0.50700000000000001},"min":{"1min":0.34699999999999998,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.66000000000000003,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.48699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4243456,"data_stored":4000588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4227072,"data_stored":4000522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":311296,"data_stored":735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4268032,"data_stored":4000623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:11:20.955 INFO:tasks.thrashosds.thrasher:Setting 61.1e to [7, 6, 3, 1, 0, 2] 2024-02-19T21:11:20.955 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '61.1e', '7', '6', '3', '1', '0', '2'] 2024-02-19T21:11:20.955 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pg-upmap-items 61.1e 7 6 3 1 0 2 2024-02-19T21:11:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:21.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:20.461593+0000 mgr.y (mgr.14550) 420 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:21.435 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:20.461593+0000 mgr.y (mgr.14550) 420 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:21.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:20.461593+0000 mgr.y (mgr.14550) 420 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:22.331 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:11:22.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:21.232850+0000 mgr.y (mgr.14550) 421 : cluster [DBG] pgmap v681: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-02-19T21:11:22.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:21.232850+0000 mgr.y (mgr.14550) 421 : cluster [DBG] pgmap v681: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-02-19T21:11:22.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:21.232850+0000 mgr.y (mgr.14550) 421 : cluster [DBG] pgmap v681: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-02-19T21:11:22.792 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:11:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:22.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:11:22.899 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:11:22.927 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:23.563 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:11:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:11:23] "GET /metrics HTTP/1.1" 200 189845 "" "Prometheus/2.33.4" 2024-02-19T21:11:23.576 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:11:23.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:23.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:11:23.680 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:23.713 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:23.735 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:23.737 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:23.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:23.837 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:11:23.838 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:11:23.839 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:11:23.840 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:23.874 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:24.125 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:11:24.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:24.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:11:24.229 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:24.258 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:24.435 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:23.233337+0000 mgr.y (mgr.14550) 422 : cluster [DBG] pgmap v682: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-19T21:11:24.435 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:24.288125+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e460: 8 total, 8 up, 8 in 2024-02-19T21:11:24.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:23.233337+0000 mgr.y (mgr.14550) 422 : cluster [DBG] pgmap v682: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-19T21:11:24.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:24.288125+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e460: 8 total, 8 up, 8 in 2024-02-19T21:11:24.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:23.233337+0000 mgr.y (mgr.14550) 422 : cluster [DBG] pgmap v682: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-19T21:11:24.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:24.288125+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e460: 8 total, 8 up, 8 in 2024-02-19T21:11:24.735 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-19T21:11:24.794 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:25.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.035+0000 7feeefbb4700 1 -- 172.21.15.163:0/3494663393 >> v1:172.21.15.163:6789/0 conn(0x7feee80ff680 legacy=0x7feee8101a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:25.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.035+0000 7feeefbb4700 1 -- 172.21.15.163:0/3494663393 shutdown_connections 2024-02-19T21:11:25.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.035+0000 7feeefbb4700 1 -- 172.21.15.163:0/3494663393 >> 172.21.15.163:0/3494663393 conn(0x7feee806bff0 msgr2=0x7feee806c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:25.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.035+0000 7feeefbb4700 1 -- 172.21.15.163:0/3494663393 shutdown_connections 2024-02-19T21:11:25.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.035+0000 7feeefbb4700 1 -- 172.21.15.163:0/3494663393 wait complete. 2024-02-19T21:11:25.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.035+0000 7feeefbb4700 1 librados: starting msgr at 2024-02-19T21:11:25.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.035+0000 7feeefbb4700 1 librados: starting objecter 2024-02-19T21:11:25.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.036+0000 7feeefbb4700 1 Processor -- start 2024-02-19T21:11:25.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.036+0000 7feeefbb4700 1 -- start start 2024-02-19T21:11:25.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.036+0000 7feeefbb4700 1 librados: setting wanted keys 2024-02-19T21:11:25.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.036+0000 7feeefbb4700 1 librados: calling monclient init 2024-02-19T21:11:25.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.036+0000 7feeefbb4700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feee8117300 con 0x7feee80fc8c0 2024-02-19T21:11:25.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.036+0000 7feeefbb4700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feee81184e0 con 0x7feee80ff680 2024-02-19T21:11:25.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.036+0000 7feeefbb4700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7feee81196e0 con 0x7feee80f60f0 2024-02-19T21:11:25.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.036+0000 7feeed14f700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7feee80fc8c0 0x7feee80f9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:39618/0 (socket says 172.21.15.163:39618) 2024-02-19T21:11:25.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.036+0000 7feeed14f700 1 -- 172.21.15.163:0/3679945625 learned_addr learned my addr 172.21.15.163:0/3679945625 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:25.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.036+0000 7feedeffd700 10 client.?.objecter ms_handle_connect 0x7feee80fc8c0 2024-02-19T21:11:25.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.036+0000 7feedeffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:25.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.037+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991270942 0 0) 0x7feee8117300 con 0x7feee80fc8c0 2024-02-19T21:11:25.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.037+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed40038e0 con 0x7feee80fc8c0 2024-02-19T21:11:25.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.038+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3340361379 0 0) 0x7feed40038e0 con 0x7feee80fc8c0 2024-02-19T21:11:25.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.038+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feee8117300 con 0x7feee80fc8c0 2024-02-19T21:11:25.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.038+0000 7feedeffd700 10 client.?.objecter ms_handle_connect 0x7feee80f60f0 2024-02-19T21:11:25.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.038+0000 7feedeffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:25.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.038+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7feed8001ff0 con 0x7feee80fc8c0 2024-02-19T21:11:25.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.038+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 136195600 0 0) 0x7feee8117300 con 0x7feee80fc8c0 2024-02-19T21:11:25.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.038+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 >> v1:172.21.15.163:6790/0 conn(0x7feee80f60f0 legacy=0x7feee80f88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:25.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.038+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 >> v1:172.21.15.193:6789/0 conn(0x7feee80ff680 legacy=0x7feee8115a60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T21:11:25.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.038+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee811a8e0 con 0x7feee80fc8c0 2024-02-19T21:11:25.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.038+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee81198d0 con 0x7feee80fc8c0 2024-02-19T21:11:25.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.038+0000 7feeefbb4700 10 client.23232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:25.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.038+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feee8119d40 con 0x7feee80fc8c0 2024-02-19T21:11:25.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.038+0000 7feeefbb4700 1 librados: init done 2024-02-19T21:11:25.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.038+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7feed8001150 con 0x7feee80fc8c0 2024-02-19T21:11:25.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.038+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7feed8003810 con 0x7feee80fc8c0 2024-02-19T21:11:25.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.039+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7feecc004f30 con 0x7feee80fc8c0 2024-02-19T21:11:25.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.040+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7feed80030c0 con 0x7feee80fc8c0 2024-02-19T21:11:25.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(460..460 src has 43..460) v4 ==== 8063+0+0 (unknown 199914979 0 0) 0x7feed8078ed0 con 0x7feee80fc8c0 2024-02-19T21:11:25.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 osd_map(460..460 src has 43..460) v4 2024-02-19T21:11:25.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feedeffd700 3 client.23232.objecter handle_osd_map got epochs [460,460] > 0 2024-02-19T21:11:25.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feedeffd700 3 client.23232.objecter handle_osd_map decoding full epoch 460 2024-02-19T21:11:25.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feedeffd700 20 client.23232.objecter dump_active .. 0 homeless 2024-02-19T21:11:25.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v460) v1 ==== 54+0+16 (unknown 404257359 0 2420000192) 0x7feed80792f0 con 0x7feee80fc8c0 2024-02-19T21:11:25.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feeefbb4700 10 client.23232.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:25.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 1 2024-02-19T21:11:25.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc0051d0 osd=0 3 2024-02-19T21:11:25.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc0051d0 osd=0 3 2024-02-19T21:11:25.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc0051d0 osd=0 3 2024-02-19T21:11:25.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 1 2024-02-19T21:11:25.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc0051d0 osd=0 3 2024-02-19T21:11:25.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 0 1 2024-02-19T21:11:25.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc0051d0 osd=0 4 2024-02-19T21:11:25.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feeefbb4700 10 client.23232.objecter _send_command 1 2024-02-19T21:11:25.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feecc004980 con 0x7feecc004560 2024-02-19T21:11:25.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.041+0000 7feedeffd700 10 client.23232.objecter ms_handle_connect 0x7feed4063b40 2024-02-19T21:11:25.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.042+0000 7feedeffd700 10 client.23232.objecter ms_handle_connect 0x7feecc004560 2024-02-19T21:11:25.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.042+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7feecc004980 con 0x7feecc004560 2024-02-19T21:11:25.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.042+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:11:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.042+0000 7feedeffd700 10 client.23232.objecter _finish_command 1 = system:0 2024-02-19T21:11:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.042+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc0051d0 osd=0 4 2024-02-19T21:11:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.042+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 0 1 2024-02-19T21:11:25.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.058+0000 7feeefbb4700 10 client.23232.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:25.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.058+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 2 2024-02-19T21:11:25.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.058+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc0051d0 osd=0 3 2024-02-19T21:11:25.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.058+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc0051d0 osd=0 3 2024-02-19T21:11:25.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.058+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc0051d0 osd=0 3 2024-02-19T21:11:25.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.058+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 2 2024-02-19T21:11:25.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.058+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc0051d0 osd=0 3 2024-02-19T21:11:25.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.058+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 0 2 2024-02-19T21:11:25.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.058+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc0051d0 osd=0 4 2024-02-19T21:11:25.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.058+0000 7feeefbb4700 10 client.23232.objecter _send_command 2 2024-02-19T21:11:25.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.058+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feecc004d60 con 0x7feecc004560 2024-02-19T21:11:25.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7feecc004d60 con 0x7feecc004560 2024-02-19T21:11:25.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:25.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feedeffd700 10 client.23232.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc0051d0 osd=0 4 2024-02-19T21:11:25.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 0 2 2024-02-19T21:11:25.059 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:11:25.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feeefbb4700 10 client.23232.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:25.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 3 2024-02-19T21:11:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc0042a0 osd=1 3 2024-02-19T21:11:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc0042a0 osd=1 3 2024-02-19T21:11:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc0042a0 osd=1 3 2024-02-19T21:11:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 3 2024-02-19T21:11:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc0042a0 osd=1 3 2024-02-19T21:11:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 1 3 2024-02-19T21:11:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc0042a0 osd=1 4 2024-02-19T21:11:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feeefbb4700 10 client.23232.objecter _send_command 3 2024-02-19T21:11:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7feecc0103b0 con 0x7feecc003b70 2024-02-19T21:11:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.060+0000 7feedeffd700 10 client.23232.objecter ms_handle_connect 0x7feecc003b70 2024-02-19T21:11:25.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.061+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7feecc0103b0 con 0x7feecc003b70 2024-02-19T21:11:25.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.061+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:11:25.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.061+0000 7feedeffd700 10 client.23232.objecter _finish_command 3 = system:0 2024-02-19T21:11:25.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.061+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc0042a0 osd=1 4 2024-02-19T21:11:25.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.061+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 1 3 2024-02-19T21:11:25.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.075+0000 7feeefbb4700 10 client.23232.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:25.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.075+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 4 2024-02-19T21:11:25.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.075+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc0042a0 osd=1 3 2024-02-19T21:11:25.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.075+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc0042a0 osd=1 3 2024-02-19T21:11:25.073 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.075+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc0042a0 osd=1 3 2024-02-19T21:11:25.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.075+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 4 2024-02-19T21:11:25.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.075+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc0042a0 osd=1 3 2024-02-19T21:11:25.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.075+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 1 4 2024-02-19T21:11:25.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.075+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc0042a0 osd=1 4 2024-02-19T21:11:25.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.075+0000 7feeefbb4700 10 client.23232.objecter _send_command 4 2024-02-19T21:11:25.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.075+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feecc0116d0 con 0x7feecc003b70 2024-02-19T21:11:25.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7feecc0116d0 con 0x7feecc003b70 2024-02-19T21:11:25.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:25.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feedeffd700 10 client.23232.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc0042a0 osd=1 4 2024-02-19T21:11:25.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 1 4 2024-02-19T21:11:25.075 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:11:25.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feeefbb4700 10 client.23232.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:25.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 5 2024-02-19T21:11:25.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc011920 osd=2 3 2024-02-19T21:11:25.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc011920 osd=2 3 2024-02-19T21:11:25.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc011920 osd=2 3 2024-02-19T21:11:25.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 5 2024-02-19T21:11:25.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc011920 osd=2 3 2024-02-19T21:11:25.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 2 5 2024-02-19T21:11:25.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc011920 osd=2 4 2024-02-19T21:11:25.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feeefbb4700 10 client.23232.objecter _send_command 5 2024-02-19T21:11:25.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7feecc015290 con 0x7feecc0020a0 2024-02-19T21:11:25.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.077+0000 7feedeffd700 10 client.23232.objecter ms_handle_connect 0x7feecc0020a0 2024-02-19T21:11:25.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.078+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7feecc015290 con 0x7feecc0020a0 2024-02-19T21:11:25.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.078+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:11:25.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.078+0000 7feedeffd700 10 client.23232.objecter _finish_command 5 = system:0 2024-02-19T21:11:25.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.078+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc011920 osd=2 4 2024-02-19T21:11:25.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.078+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 2 5 2024-02-19T21:11:25.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.092+0000 7feeefbb4700 10 client.23232.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:25.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.092+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 6 2024-02-19T21:11:25.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.092+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc011920 osd=2 3 2024-02-19T21:11:25.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.092+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc011920 osd=2 3 2024-02-19T21:11:25.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.092+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc011920 osd=2 3 2024-02-19T21:11:25.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.092+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 6 2024-02-19T21:11:25.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.092+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc011920 osd=2 3 2024-02-19T21:11:25.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.092+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 2 6 2024-02-19T21:11:25.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.092+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc011920 osd=2 4 2024-02-19T21:11:25.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.092+0000 7feeefbb4700 10 client.23232.objecter _send_command 6 2024-02-19T21:11:25.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.092+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feecc0016c0 con 0x7feecc0020a0 2024-02-19T21:11:25.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.093+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7feecc0016c0 con 0x7feecc0020a0 2024-02-19T21:11:25.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.093+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:25.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.093+0000 7feedeffd700 10 client.23232.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.093+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc011920 osd=2 4 2024-02-19T21:11:25.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.093+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 2 6 2024-02-19T21:11:25.092 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:11:25.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feeefbb4700 10 client.23232.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:25.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 7 2024-02-19T21:11:25.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc010d20 osd=3 3 2024-02-19T21:11:25.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc010d20 osd=3 3 2024-02-19T21:11:25.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc010d20 osd=3 3 2024-02-19T21:11:25.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 7 2024-02-19T21:11:25.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc010d20 osd=3 3 2024-02-19T21:11:25.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 3 7 2024-02-19T21:11:25.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc010d20 osd=3 4 2024-02-19T21:11:25.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feeefbb4700 10 client.23232.objecter _send_command 7 2024-02-19T21:11:25.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7feecc011520 con 0x7feecc016580 2024-02-19T21:11:25.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feedeffd700 10 client.23232.objecter ms_handle_connect 0x7feecc016580 2024-02-19T21:11:25.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7feecc011520 con 0x7feecc016580 2024-02-19T21:11:25.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:11:25.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feedeffd700 10 client.23232.objecter _finish_command 7 = system:0 2024-02-19T21:11:25.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc010d20 osd=3 4 2024-02-19T21:11:25.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.094+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 3 7 2024-02-19T21:11:25.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.109+0000 7feeefbb4700 10 client.23232.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:25.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.109+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 8 2024-02-19T21:11:25.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.109+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc010d20 osd=3 3 2024-02-19T21:11:25.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.109+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc010d20 osd=3 3 2024-02-19T21:11:25.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.109+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc010d20 osd=3 3 2024-02-19T21:11:25.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.109+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 8 2024-02-19T21:11:25.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.109+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc010d20 osd=3 3 2024-02-19T21:11:25.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.109+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 3 8 2024-02-19T21:11:25.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.109+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc010d20 osd=3 4 2024-02-19T21:11:25.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.109+0000 7feeefbb4700 10 client.23232.objecter _send_command 8 2024-02-19T21:11:25.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.109+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feecc011520 con 0x7feecc016580 2024-02-19T21:11:25.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7feecc011520 con 0x7feecc016580 2024-02-19T21:11:25.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:25.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feedeffd700 10 client.23232.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc010d20 osd=3 4 2024-02-19T21:11:25.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 3 8 2024-02-19T21:11:25.111 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:11:25.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feeefbb4700 10 client.23232.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:25.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 9 2024-02-19T21:11:25.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc011520 osd=4 3 2024-02-19T21:11:25.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc011520 osd=4 3 2024-02-19T21:11:25.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc011520 osd=4 3 2024-02-19T21:11:25.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 9 2024-02-19T21:11:25.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc011520 osd=4 3 2024-02-19T21:11:25.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 4 9 2024-02-19T21:11:25.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc011520 osd=4 4 2024-02-19T21:11:25.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feeefbb4700 10 client.23232.objecter _send_command 9 2024-02-19T21:11:25.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.110+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7feecc016170 con 0x7feecc015bd0 2024-02-19T21:11:25.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.111+0000 7feedeffd700 10 client.23232.objecter ms_handle_connect 0x7feecc015bd0 2024-02-19T21:11:25.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.111+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7feecc016170 con 0x7feecc015bd0 2024-02-19T21:11:25.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.111+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:11:25.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.111+0000 7feedeffd700 10 client.23232.objecter _finish_command 9 = system:0 2024-02-19T21:11:25.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.111+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc011520 osd=4 4 2024-02-19T21:11:25.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.111+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 4 9 2024-02-19T21:11:25.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.126+0000 7feeefbb4700 10 client.23232.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:25.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.126+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 10 2024-02-19T21:11:25.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.126+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc011520 osd=4 3 2024-02-19T21:11:25.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.126+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc011520 osd=4 3 2024-02-19T21:11:25.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.126+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc011520 osd=4 3 2024-02-19T21:11:25.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.126+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 10 2024-02-19T21:11:25.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.126+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc011520 osd=4 3 2024-02-19T21:11:25.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.126+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 4 10 2024-02-19T21:11:25.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.126+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc011520 osd=4 4 2024-02-19T21:11:25.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.126+0000 7feeefbb4700 10 client.23232.objecter _send_command 10 2024-02-19T21:11:25.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.126+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feecc016170 con 0x7feecc015bd0 2024-02-19T21:11:25.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7feee400e020 con 0x7feecc015bd0 2024-02-19T21:11:25.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:25.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feedeffd700 10 client.23232.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc011520 osd=4 4 2024-02-19T21:11:25.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 4 10 2024-02-19T21:11:25.126 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:11:25.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feeefbb4700 10 client.23232.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:25.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 11 2024-02-19T21:11:25.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc016390 osd=5 3 2024-02-19T21:11:25.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc016390 osd=5 3 2024-02-19T21:11:25.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc016390 osd=5 3 2024-02-19T21:11:25.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 11 2024-02-19T21:11:25.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc016390 osd=5 3 2024-02-19T21:11:25.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 5 11 2024-02-19T21:11:25.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc016390 osd=5 4 2024-02-19T21:11:25.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feeefbb4700 10 client.23232.objecter _send_command 11 2024-02-19T21:11:25.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7feecc01c3b0 con 0x7feecc01be30 2024-02-19T21:11:25.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.128+0000 7feedeffd700 10 client.23232.objecter ms_handle_connect 0x7feecc01be30 2024-02-19T21:11:25.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.129+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7feecc01c3b0 con 0x7feecc01be30 2024-02-19T21:11:25.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.129+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:11:25.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.129+0000 7feedeffd700 10 client.23232.objecter _finish_command 11 = system:0 2024-02-19T21:11:25.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.129+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc016390 osd=5 4 2024-02-19T21:11:25.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.129+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 5 11 2024-02-19T21:11:25.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.143+0000 7feeefbb4700 10 client.23232.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:25.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.143+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 12 2024-02-19T21:11:25.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.143+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc016390 osd=5 3 2024-02-19T21:11:25.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.143+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc016390 osd=5 3 2024-02-19T21:11:25.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.143+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc016390 osd=5 3 2024-02-19T21:11:25.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.143+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 12 2024-02-19T21:11:25.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.143+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc016390 osd=5 3 2024-02-19T21:11:25.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.143+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 5 12 2024-02-19T21:11:25.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.143+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc016390 osd=5 4 2024-02-19T21:11:25.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.143+0000 7feeefbb4700 10 client.23232.objecter _send_command 12 2024-02-19T21:11:25.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.143+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feecc01c3b0 con 0x7feecc01be30 2024-02-19T21:11:25.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7feecc01c3b0 con 0x7feecc01be30 2024-02-19T21:11:25.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:25.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feedeffd700 10 client.23232.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc016390 osd=5 4 2024-02-19T21:11:25.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 5 12 2024-02-19T21:11:25.143 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:11:25.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feeefbb4700 10 client.23232.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:25.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 13 2024-02-19T21:11:25.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc01c3b0 osd=6 3 2024-02-19T21:11:25.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc01c3b0 osd=6 3 2024-02-19T21:11:25.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc01c3b0 osd=6 3 2024-02-19T21:11:25.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 13 2024-02-19T21:11:25.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc01c3b0 osd=6 3 2024-02-19T21:11:25.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 6 13 2024-02-19T21:11:25.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc01c3b0 osd=6 4 2024-02-19T21:11:25.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feeefbb4700 10 client.23232.objecter _send_command 13 2024-02-19T21:11:25.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.145+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7feecc027000 con 0x7feecc0234f0 2024-02-19T21:11:25.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.146+0000 7feedeffd700 10 client.23232.objecter ms_handle_connect 0x7feecc0234f0 2024-02-19T21:11:25.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.146+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7feecc027000 con 0x7feecc0234f0 2024-02-19T21:11:25.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.146+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:11:25.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.146+0000 7feedeffd700 10 client.23232.objecter _finish_command 13 = system:0 2024-02-19T21:11:25.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.146+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc01c3b0 osd=6 4 2024-02-19T21:11:25.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.146+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 6 13 2024-02-19T21:11:25.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.161+0000 7feeefbb4700 10 client.23232.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:25.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.161+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 14 2024-02-19T21:11:25.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.161+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc01c3b0 osd=6 3 2024-02-19T21:11:25.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.161+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc01c3b0 osd=6 3 2024-02-19T21:11:25.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.161+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc01c3b0 osd=6 3 2024-02-19T21:11:25.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.161+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 14 2024-02-19T21:11:25.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.161+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc01c3b0 osd=6 3 2024-02-19T21:11:25.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.161+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 6 14 2024-02-19T21:11:25.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.161+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc01c3b0 osd=6 4 2024-02-19T21:11:25.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.161+0000 7feeefbb4700 10 client.23232.objecter _send_command 14 2024-02-19T21:11:25.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.161+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feecc027000 con 0x7feecc0234f0 2024-02-19T21:11:25.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.162+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7feecc027000 con 0x7feecc0234f0 2024-02-19T21:11:25.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.162+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:25.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.162+0000 7feedeffd700 10 client.23232.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.162+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc01c3b0 osd=6 4 2024-02-19T21:11:25.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.162+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 6 14 2024-02-19T21:11:25.161 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:11:25.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.163+0000 7feeefbb4700 10 client.23232.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:25.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.163+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 15 2024-02-19T21:11:25.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.163+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc027000 osd=7 3 2024-02-19T21:11:25.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.163+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc027000 osd=7 3 2024-02-19T21:11:25.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.163+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc027000 osd=7 3 2024-02-19T21:11:25.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.163+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 15 2024-02-19T21:11:25.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.163+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc027000 osd=7 3 2024-02-19T21:11:25.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.163+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 7 15 2024-02-19T21:11:25.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.163+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc027000 osd=7 4 2024-02-19T21:11:25.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.163+0000 7feeefbb4700 10 client.23232.objecter _send_command 15 2024-02-19T21:11:25.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.163+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7feecc0293c0 con 0x7feecc027680 2024-02-19T21:11:25.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.163+0000 7feedeffd700 10 client.23232.objecter ms_handle_connect 0x7feecc027680 2024-02-19T21:11:25.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.164+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7feecc0293c0 con 0x7feecc027680 2024-02-19T21:11:25.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.164+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:11:25.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.164+0000 7feedeffd700 10 client.23232.objecter _finish_command 15 = system:0 2024-02-19T21:11:25.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.164+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc027000 osd=7 4 2024-02-19T21:11:25.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.164+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 7 15 2024-02-19T21:11:25.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.178+0000 7feeefbb4700 10 client.23232.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:25.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.178+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign -1 16 2024-02-19T21:11:25.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.178+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc027000 osd=7 3 2024-02-19T21:11:25.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.178+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc027000 osd=7 3 2024-02-19T21:11:25.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.178+0000 7feeefbb4700 20 client.23232.objecter _get_session s=0x7feecc027000 osd=7 3 2024-02-19T21:11:25.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.178+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_remove -1 16 2024-02-19T21:11:25.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.178+0000 7feeefbb4700 20 client.23232.objecter get_session s=0x7feecc027000 osd=7 3 2024-02-19T21:11:25.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.178+0000 7feeefbb4700 15 client.23232.objecter _session_command_op_assign 7 16 2024-02-19T21:11:25.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.178+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc027000 osd=7 4 2024-02-19T21:11:25.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.178+0000 7feeefbb4700 10 client.23232.objecter _send_command 16 2024-02-19T21:11:25.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.178+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7feecc0293c0 con 0x7feecc027680 2024-02-19T21:11:25.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feedeffd700 1 -- 172.21.15.163:0/3679945625 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7feecc0293c0 con 0x7feecc027680 2024-02-19T21:11:25.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feedeffd700 10 client.23232.objecter ms_dispatch 0x7feee8001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:25.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feedeffd700 10 client.23232.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feedeffd700 20 client.23232.objecter put_session s=0x7feecc027000 osd=7 4 2024-02-19T21:11:25.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feedeffd700 15 client.23232.objecter _session_command_op_remove 7 16 2024-02-19T21:11:25.179 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:11:25.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 10 librados: watch_flush enter 2024-02-19T21:11:25.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 10 librados: watch_flush exit 2024-02-19T21:11:25.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 10 client.23232.objecter close_session for osd.0 2024-02-19T21:11:25.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 >> v1:172.21.15.163:6801/1559418115 conn(0x7feecc004560 legacy=0x7feecc002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:25.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc0051d0 osd=0 1 2024-02-19T21:11:25.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 10 client.23232.objecter close_session for osd.1 2024-02-19T21:11:25.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 >> v1:172.21.15.163:6805/3125047432 conn(0x7feecc003b70 legacy=0x7feecc00a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:25.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc0042a0 osd=1 1 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 10 client.23232.objecter close_session for osd.2 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 >> v1:172.21.15.163:6809/1563158487 conn(0x7feecc0020a0 legacy=0x7feecc013b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc011920 osd=2 1 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 10 client.23232.objecter close_session for osd.3 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 >> v1:172.21.15.163:6813/2112279769 conn(0x7feecc016580 legacy=0x7feecc018970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeee151700 1 -- 172.21.15.163:0/3679945625 reap_dead start 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc010d20 osd=3 1 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 10 client.23232.objecter close_session for osd.4 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 >> v1:172.21.15.193:6800/3129716965 conn(0x7feecc015bd0 legacy=0x7feecc01a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeee151700 1 -- 172.21.15.163:0/3679945625 reap_dead start 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc011520 osd=4 1 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 10 client.23232.objecter close_session for osd.5 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.180+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 >> v1:172.21.15.193:6804/805161991 conn(0x7feecc01be30 legacy=0x7feecc0218c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc016390 osd=5 1 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 10 client.23232.objecter close_session for osd.6 2024-02-19T21:11:25.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 >> v1:172.21.15.193:6808/787604615 conn(0x7feecc0234f0 legacy=0x7feecc0258e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:25.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc01c3b0 osd=6 1 2024-02-19T21:11:25.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 10 client.23232.objecter close_session for osd.7 2024-02-19T21:11:25.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 >> v1:172.21.15.193:6812/157927432 conn(0x7feecc027680 legacy=0x7feecc027b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:25.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 20 client.23232.objecter put_session s=0x7feecc027000 osd=7 1 2024-02-19T21:11:25.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 20 client.23232.objecter shutdown clearing up homeless session... 2024-02-19T21:11:25.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 10 client.23232.objecter successfully canceled tick 2024-02-19T21:11:25.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 >> v1:172.21.15.163:6800/1552514681 conn(0x7feed4063b40 legacy=0x7feed4066000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:25.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 >> v1:172.21.15.163:6789/0 conn(0x7feee80fc8c0 legacy=0x7feee80f9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:25.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 shutdown_connections 2024-02-19T21:11:25.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 >> 172.21.15.163:0/3679945625 conn(0x7feee806bff0 msgr2=0x7feee806c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:25.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 shutdown_connections 2024-02-19T21:11:25.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 1 -- 172.21.15.163:0/3679945625 wait complete. 2024-02-19T21:11:25.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:25.181+0000 7feeefbb4700 1 librados: shutdown 2024-02-19T21:11:25.182 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.183 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.183 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.183 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.183 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.183 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.183 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.183 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:11:25.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:25.040222+0000 mon.a (mon.0) 2639 : audit [DBG] from='client.? v1:172.21.15.163:0/3679945625' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:25.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:25.040222+0000 mon.a (mon.0) 2639 : audit [DBG] from='client.? v1:172.21.15.163:0/3679945625' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:25.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:25.040222+0000 mon.a (mon.0) 2639 : audit [DBG] from='client.? v1:172.21.15.163:0/3679945625' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:26.603 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:25.234358+0000 mgr.y (mgr.14550) 423 : cluster [DBG] pgmap v684: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 64 KiB/s rd, 1.0 KiB/s wr, 107 op/s 2024-02-19T21:11:26.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:25.234358+0000 mgr.y (mgr.14550) 423 : cluster [DBG] pgmap v684: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 64 KiB/s rd, 1.0 KiB/s wr, 107 op/s 2024-02-19T21:11:26.611 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:26.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:26.715 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:26.744 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:26.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:25.234358+0000 mgr.y (mgr.14550) 423 : cluster [DBG] pgmap v684: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 64 KiB/s rd, 1.0 KiB/s wr, 107 op/s 2024-02-19T21:11:27.896 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:27.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:28.000 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:28.028 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:28.069 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:11:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:11:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:11:28.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.656+0000 7fa3f8bf4700 1 -- 172.21.15.163:0/2559518084 >> v1:172.21.15.163:6789/0 conn(0x7fa3f4102930 legacy=0x7fa3f4104dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:28.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.656+0000 7fa3f8bf4700 1 -- 172.21.15.163:0/2559518084 shutdown_connections 2024-02-19T21:11:28.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.656+0000 7fa3f8bf4700 1 -- 172.21.15.163:0/2559518084 >> 172.21.15.163:0/2559518084 conn(0x7fa3f406cc00 msgr2=0x7fa3f40f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:28.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.657+0000 7fa3f8bf4700 1 -- 172.21.15.163:0/2559518084 shutdown_connections 2024-02-19T21:11:28.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.657+0000 7fa3f8bf4700 1 -- 172.21.15.163:0/2559518084 wait complete. 2024-02-19T21:11:28.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.657+0000 7fa3f8bf4700 1 librados: starting msgr at 2024-02-19T21:11:28.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.657+0000 7fa3f8bf4700 1 librados: starting objecter 2024-02-19T21:11:28.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.657+0000 7fa3f8bf4700 1 Processor -- start 2024-02-19T21:11:28.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3f8bf4700 1 -- start start 2024-02-19T21:11:28.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3f8bf4700 1 librados: setting wanted keys 2024-02-19T21:11:28.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3f8bf4700 1 librados: calling monclient init 2024-02-19T21:11:28.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3f8bf4700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa3f41172a0 con 0x7fa3f40fee20 2024-02-19T21:11:28.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3f8bf4700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa3f4118480 con 0x7fa3f40fbfd0 2024-02-19T21:11:28.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3f8bf4700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa3f4119680 con 0x7fa3f4102930 2024-02-19T21:11:28.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3f1d9b700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fa3f40fee20 0x7fa3f4111fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:39644/0 (socket says 172.21.15.163:39644) 2024-02-19T21:11:28.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3f1d9b700 1 -- 172.21.15.163:0/2176739845 learned_addr learned my addr 172.21.15.163:0/2176739845 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:28.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3f2d9d700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fa3f4102930 0x7fa3f4115960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:34860/0 (socket says 172.21.15.163:34860) 2024-02-19T21:11:28.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3e37fe700 10 client.?.objecter ms_handle_connect 0x7fa3f40fee20 2024-02-19T21:11:28.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3e37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:28.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3e37fe700 10 client.?.objecter ms_handle_connect 0x7fa3f4102930 2024-02-19T21:11:28.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3e37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:28.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3e37fe700 10 client.?.objecter ms_handle_connect 0x7fa3f40fbfd0 2024-02-19T21:11:28.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3e37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:28.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491466355 0 0) 0x7fa3f41172a0 con 0x7fa3f40fee20 2024-02-19T21:11:28.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3dc004980 con 0x7fa3f40fee20 2024-02-19T21:11:28.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100479703 0 0) 0x7fa3f4119680 con 0x7fa3f4102930 2024-02-19T21:11:28.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f41172a0 con 0x7fa3f4102930 2024-02-19T21:11:28.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563195967 0 0) 0x7fa3f4118480 con 0x7fa3f40fbfd0 2024-02-19T21:11:28.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.658+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f4119680 con 0x7fa3f40fbfd0 2024-02-19T21:11:28.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 42295631 0 0) 0x7fa3dc004980 con 0x7fa3f40fee20 2024-02-19T21:11:28.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3f4118480 con 0x7fa3f40fee20 2024-02-19T21:11:28.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3e40019f0 con 0x7fa3f40fee20 2024-02-19T21:11:28.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 218930950 0 0) 0x7fa3f4119680 con 0x7fa3f40fbfd0 2024-02-19T21:11:28.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3dc004980 con 0x7fa3f40fbfd0 2024-02-19T21:11:28.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3e80018c0 con 0x7fa3f40fbfd0 2024-02-19T21:11:28.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3757957720 0 0) 0x7fa3f41172a0 con 0x7fa3f4102930 2024-02-19T21:11:28.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3f4119680 con 0x7fa3f4102930 2024-02-19T21:11:28.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3ec003100 con 0x7fa3f4102930 2024-02-19T21:11:28.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2368855638 0 0) 0x7fa3f4118480 con 0x7fa3f40fee20 2024-02-19T21:11:28.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 >> v1:172.21.15.163:6790/0 conn(0x7fa3f4102930 legacy=0x7fa3f4115960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:28.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 >> v1:172.21.15.193:6789/0 conn(0x7fa3f40fbfd0 legacy=0x7fa3f40fb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:28.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f411a880 con 0x7fa3f40fee20 2024-02-19T21:11:28.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3f8bf4700 1 -- 172.21.15.163:0/2176739845 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f4119870 con 0x7fa3f40fee20 2024-02-19T21:11:28.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3f8bf4700 10 client.23238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:28.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3f8bf4700 1 -- 172.21.15.163:0/2176739845 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f4119ce0 con 0x7fa3f40fee20 2024-02-19T21:11:28.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3f8bf4700 1 librados: init done 2024-02-19T21:11:28.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fa3e4002550 con 0x7fa3f40fee20 2024-02-19T21:11:28.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.659+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3e4005c20 con 0x7fa3f40fee20 2024-02-19T21:11:28.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.660+0000 7fa3f8bf4700 1 -- 172.21.15.163:0/2176739845 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d4004f20 con 0x7fa3f40fee20 2024-02-19T21:11:28.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.660+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fa3e4018ab0 con 0x7fa3f40fee20 2024-02-19T21:11:28.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.661+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(460..460 src has 43..460) v4 ==== 8063+0+0 (unknown 199914979 0 0) 0x7fa3e4078d60 con 0x7fa3f40fee20 2024-02-19T21:11:28.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.661+0000 7fa3e37fe700 10 client.23238.objecter ms_dispatch 0x7fa3f4001550 osd_map(460..460 src has 43..460) v4 2024-02-19T21:11:28.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.661+0000 7fa3e37fe700 3 client.23238.objecter handle_osd_map got epochs [460,460] > 0 2024-02-19T21:11:28.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.661+0000 7fa3e37fe700 3 client.23238.objecter handle_osd_map decoding full epoch 460 2024-02-19T21:11:28.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.661+0000 7fa3e37fe700 20 client.23238.objecter dump_active .. 0 homeless 2024-02-19T21:11:28.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.661+0000 7fa3e37fe700 10 client.23238.objecter ms_handle_connect 0x7fa3dc063cb0 2024-02-19T21:11:28.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.665+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa3e404c2a0 con 0x7fa3f40fee20 2024-02-19T21:11:28.677 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:28.681 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:27.234793+0000 mgr.y (mgr.14550) 424 : cluster [DBG] pgmap v685: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 895 B/s wr, 93 op/s 2024-02-19T21:11:28.681 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:27.806868+0000 mon.b (mon.1) 611 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:11:28.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:28.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:28.682 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:27.234793+0000 mgr.y (mgr.14550) 424 : cluster [DBG] pgmap v685: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 895 B/s wr, 93 op/s 2024-02-19T21:11:28.682 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:27.806868+0000 mon.b (mon.1) 611 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:11:28.783 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:11:28.817 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:28.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:27.234793+0000 mgr.y (mgr.14550) 424 : cluster [DBG] pgmap v685: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 55 KiB/s rd, 895 B/s wr, 93 op/s 2024-02-19T21:11:28.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:27.806868+0000 mon.b (mon.1) 611 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:11:28.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:28.855+0000 7fa3f8bf4700 1 -- 172.21.15.163:0/2176739845 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fa3d40051d0 con 0x7fa3f40fee20 2024-02-19T21:11:28.916 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:11:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:28.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:11:28.996 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:29.020 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:29.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.046+0000 7f6d28ea2700 1 -- 172.21.15.163:0/1051804552 >> v1:172.21.15.163:6790/0 conn(0x7f6d240fc890 legacy=0x7f6d240fcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:29.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.046+0000 7f6d28ea2700 1 -- 172.21.15.163:0/1051804552 shutdown_connections 2024-02-19T21:11:29.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.046+0000 7f6d28ea2700 1 -- 172.21.15.163:0/1051804552 >> 172.21.15.163:0/1051804552 conn(0x7f6d240f2470 msgr2=0x7f6d240f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:29.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.046+0000 7f6d28ea2700 1 -- 172.21.15.163:0/1051804552 shutdown_connections 2024-02-19T21:11:29.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.047+0000 7f6d28ea2700 1 -- 172.21.15.163:0/1051804552 wait complete. 2024-02-19T21:11:29.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.047+0000 7f6d28ea2700 1 librados: starting msgr at 2024-02-19T21:11:29.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.047+0000 7f6d28ea2700 1 librados: starting objecter 2024-02-19T21:11:29.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.047+0000 7f6d28ea2700 1 Processor -- start 2024-02-19T21:11:29.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.047+0000 7f6d28ea2700 1 -- start start 2024-02-19T21:11:29.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.047+0000 7f6d28ea2700 1 librados: setting wanted keys 2024-02-19T21:11:29.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.047+0000 7f6d28ea2700 1 librados: calling monclient init 2024-02-19T21:11:29.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d28ea2700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6d241172e0 con 0x7f6d240fc890 2024-02-19T21:11:29.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d28ea2700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6d241184c0 con 0x7f6d241031d0 2024-02-19T21:11:29.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d28ea2700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6d241196c0 con 0x7f6d240ff690 2024-02-19T21:11:29.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d2259c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f6d240fc890 0x7f6d24060cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:39650/0 (socket says 172.21.15.163:39650) 2024-02-19T21:11:29.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d21d9b700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f6d240ff690 0x7f6d24061420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:34872/0 (socket says 172.21.15.163:34872) 2024-02-19T21:11:29.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d2259c700 1 -- 172.21.15.163:0/1864622639 learned_addr learned my addr 172.21.15.163:0/1864622639 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:29.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 10 client.?.objecter ms_handle_connect 0x7f6d240ff690 2024-02-19T21:11:29.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:29.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 10 client.?.objecter ms_handle_connect 0x7f6d240fc890 2024-02-19T21:11:29.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:29.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 10 client.?.objecter ms_handle_connect 0x7f6d241031d0 2024-02-19T21:11:29.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:29.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36017158 0 0) 0x7f6d241196c0 con 0x7f6d240ff690 2024-02-19T21:11:29.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d0c0033f0 con 0x7f6d240ff690 2024-02-19T21:11:29.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195034436 0 0) 0x7f6d241172e0 con 0x7f6d240fc890 2024-02-19T21:11:29.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d241196c0 con 0x7f6d240fc890 2024-02-19T21:11:29.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165958543 0 0) 0x7f6d241184c0 con 0x7f6d241031d0 2024-02-19T21:11:29.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d241172e0 con 0x7f6d241031d0 2024-02-19T21:11:29.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1030933031 0 0) 0x7f6d0c0033f0 con 0x7f6d240ff690 2024-02-19T21:11:29.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d241184c0 con 0x7f6d240ff690 2024-02-19T21:11:29.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 806794616 0 0) 0x7f6d241172e0 con 0x7f6d241031d0 2024-02-19T21:11:29.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d0c0033f0 con 0x7f6d241031d0 2024-02-19T21:11:29.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 445292372 0 0) 0x7f6d241196c0 con 0x7f6d240fc890 2024-02-19T21:11:29.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d241172e0 con 0x7f6d240fc890 2024-02-19T21:11:29.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6d180019d0 con 0x7f6d240ff690 2024-02-19T21:11:29.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6d1c001bc0 con 0x7f6d241031d0 2024-02-19T21:11:29.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.048+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6d14001f20 con 0x7f6d240fc890 2024-02-19T21:11:29.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.049+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1825940884 0 0) 0x7f6d241184c0 con 0x7f6d240ff690 2024-02-19T21:11:29.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.049+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 >> v1:172.21.15.193:6789/0 conn(0x7f6d241031d0 legacy=0x7f6d241159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:29.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.049+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 >> v1:172.21.15.163:6789/0 conn(0x7f6d240fc890 legacy=0x7f6d24060cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:29.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.049+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2411a8c0 con 0x7f6d240ff690 2024-02-19T21:11:29.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.049+0000 7f6d28ea2700 1 -- 172.21.15.163:0/1864622639 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d241198b0 con 0x7f6d240ff690 2024-02-19T21:11:29.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.049+0000 7f6d28ea2700 10 client.49964.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:29.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.049+0000 7f6d28ea2700 1 -- 172.21.15.163:0/1864622639 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d24119d20 con 0x7f6d240ff690 2024-02-19T21:11:29.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.049+0000 7f6d28ea2700 1 librados: init done 2024-02-19T21:11:29.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.049+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6d18002420 con 0x7f6d240ff690 2024-02-19T21:11:29.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.049+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6d18005a10 con 0x7f6d240ff690 2024-02-19T21:11:29.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.050+0000 7f6d28ea2700 1 -- 172.21.15.163:0/1864622639 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d04004f20 con 0x7f6d240ff690 2024-02-19T21:11:29.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.050+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6d18018980 con 0x7f6d240ff690 2024-02-19T21:11:29.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.051+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(460..460 src has 43..460) v4 ==== 8063+0+0 (unknown 199914979 0 0) 0x7f6d18078d80 con 0x7f6d240ff690 2024-02-19T21:11:29.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.051+0000 7f6d137fe700 10 client.49964.objecter ms_dispatch 0x7f6d24001550 osd_map(460..460 src has 43..460) v4 2024-02-19T21:11:29.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.051+0000 7f6d137fe700 3 client.49964.objecter handle_osd_map got epochs [460,460] > 0 2024-02-19T21:11:29.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.051+0000 7f6d137fe700 3 client.49964.objecter handle_osd_map decoding full epoch 460 2024-02-19T21:11:29.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.051+0000 7f6d137fe700 20 client.49964.objecter dump_active .. 0 homeless 2024-02-19T21:11:29.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.051+0000 7f6d137fe700 10 client.49964.objecter ms_handle_connect 0x7f6d0c063e50 2024-02-19T21:11:29.052 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:29.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.055+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6d1804c320 con 0x7f6d240ff690 2024-02-19T21:11:29.073 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:29.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:29.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:29.177 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:29.206 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:29.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.239+0000 7f6d28ea2700 1 -- 172.21.15.163:0/1864622639 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd pg-upmap-items", "pgid": "61.1e", "id": [7, 6, 3, 1, 0, 2]} v 0) v1 -- 0x7f6d040051d0 con 0x7f6d240ff690 2024-02-19T21:11:29.279 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:29.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:29.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:29.382 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:29.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.385+0000 7fa3e37fe700 1 -- 172.21.15.163:0/2176739845 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v461) v1 ==== 91+0+0 (unknown 2595916371 0 0) 0x7fa3e404a6b0 con 0x7fa3f40fee20 2024-02-19T21:11:29.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.385+0000 7f6d137fe700 1 -- 172.21.15.163:0/1864622639 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd pg-upmap-items", "pgid": "61.1e", "id": [7, 6, 3, 1, 0, 2]}]=0 set 61.1e pg_upmap_items mapping to [7->6,3->1,0->2] v461) v1 ==== 161+0+0 (unknown 1281038702 0 0) 0x7f6d1804a730 con 0x7f6d240ff690 2024-02-19T21:11:29.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.390+0000 7fa3f8bf4700 10 librados: watch_flush enter 2024-02-19T21:11:29.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.390+0000 7fa3f8bf4700 10 librados: watch_flush exit 2024-02-19T21:11:29.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.390+0000 7fa3f8bf4700 20 client.23238.objecter shutdown clearing up homeless session... 2024-02-19T21:11:29.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.390+0000 7fa3f8bf4700 10 client.23238.objecter successfully canceled tick 2024-02-19T21:11:29.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.393+0000 7fa3f8bf4700 1 -- 172.21.15.163:0/2176739845 >> v1:172.21.15.163:6800/1552514681 conn(0x7fa3dc063cb0 legacy=0x7fa3dc066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:29.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.393+0000 7fa3f8bf4700 1 -- 172.21.15.163:0/2176739845 >> v1:172.21.15.163:6789/0 conn(0x7fa3f40fee20 legacy=0x7fa3f4111fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:29.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.394+0000 7f6d28ea2700 10 librados: watch_flush enter 2024-02-19T21:11:29.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.394+0000 7f6d28ea2700 10 librados: watch_flush exit 2024-02-19T21:11:29.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.394+0000 7f6d28ea2700 20 client.49964.objecter shutdown clearing up homeless session... 2024-02-19T21:11:29.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.394+0000 7f6d28ea2700 10 client.49964.objecter successfully canceled tick 2024-02-19T21:11:29.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.394+0000 7f6d28ea2700 1 -- 172.21.15.163:0/1864622639 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6d0c063e50 legacy=0x7f6d0c066310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:29.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.394+0000 7f6d28ea2700 1 -- 172.21.15.163:0/1864622639 >> v1:172.21.15.163:6790/0 conn(0x7f6d240ff690 legacy=0x7f6d24061420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:29.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.394+0000 7f6d28ea2700 1 -- 172.21.15.163:0/1864622639 shutdown_connections 2024-02-19T21:11:29.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.394+0000 7f6d28ea2700 1 -- 172.21.15.163:0/1864622639 >> 172.21.15.163:0/1864622639 conn(0x7f6d240f2470 msgr2=0x7f6d24106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:29.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.394+0000 7f6d28ea2700 1 -- 172.21.15.163:0/1864622639 shutdown_connections 2024-02-19T21:11:29.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.394+0000 7f6d28ea2700 1 -- 172.21.15.163:0/1864622639 wait complete. 2024-02-19T21:11:29.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.394+0000 7f6d28ea2700 1 librados: shutdown 2024-02-19T21:11:29.393 INFO:teuthology.orchestra.run.smithi163.stderr:set 61.1e pg_upmap_items mapping to [7->6,3->1,0->2] 2024-02-19T21:11:29.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.397+0000 7fa3f8bf4700 1 -- 172.21.15.163:0/2176739845 shutdown_connections 2024-02-19T21:11:29.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.397+0000 7fa3f8bf4700 1 -- 172.21.15.163:0/2176739845 >> 172.21.15.163:0/2176739845 conn(0x7fa3f406cc00 msgr2=0x7fa3f4105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:29.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.400+0000 7fa3f8bf4700 1 -- 172.21.15.163:0/2176739845 shutdown_connections 2024-02-19T21:11:29.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.400+0000 7fa3f8bf4700 1 -- 172.21.15.163:0/2176739845 wait complete. 2024-02-19T21:11:29.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:29.400+0000 7fa3f8bf4700 1 librados: shutdown 2024-02-19T21:11:29.399 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:11:29.417 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:29.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:28.855511+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? v1:172.21.15.163:0/2176739845' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:11:29.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:29.048385+0000 mon.b (mon.1) 612 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:11:29.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:29.049209+0000 mon.b (mon.1) 613 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:11:29.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:29.058042+0000 mon.a (mon.0) 2641 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:11:29.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:29.240028+0000 mon.c (mon.2) 398 : audit [INF] from='client.? v1:172.21.15.163:0/1864622639' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "61.1e", "id": [7, 6, 3, 1, 0, 2]}]: dispatch 2024-02-19T21:11:29.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:29.240317+0000 mon.a (mon.0) 2642 : audit [INF] from='client.49964 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "61.1e", "id": [7, 6, 3, 1, 0, 2]}]: dispatch 2024-02-19T21:11:29.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:28.855511+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? v1:172.21.15.163:0/2176739845' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:11:29.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:29.048385+0000 mon.b (mon.1) 612 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:11:29.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:29.049209+0000 mon.b (mon.1) 613 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:11:29.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:29.058042+0000 mon.a (mon.0) 2641 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:11:29.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:29.240028+0000 mon.c (mon.2) 398 : audit [INF] from='client.? v1:172.21.15.163:0/1864622639' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "61.1e", "id": [7, 6, 3, 1, 0, 2]}]: dispatch 2024-02-19T21:11:29.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:29.240317+0000 mon.a (mon.0) 2642 : audit [INF] from='client.49964 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "61.1e", "id": [7, 6, 3, 1, 0, 2]}]: dispatch 2024-02-19T21:11:29.540 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:29.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:29.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:29.645 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:11:29.672 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:29.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:28.855511+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? v1:172.21.15.163:0/2176739845' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:11:29.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:29.048385+0000 mon.b (mon.1) 612 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:11:29.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:29.049209+0000 mon.b (mon.1) 613 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:11:29.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:29.058042+0000 mon.a (mon.0) 2641 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:11:29.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:29.240028+0000 mon.c (mon.2) 398 : audit [INF] from='client.? v1:172.21.15.163:0/1864622639' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "61.1e", "id": [7, 6, 3, 1, 0, 2]}]: dispatch 2024-02-19T21:11:29.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:29.240317+0000 mon.a (mon.0) 2642 : audit [INF] from='client.49964 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "61.1e", "id": [7, 6, 3, 1, 0, 2]}]: dispatch 2024-02-19T21:11:29.871 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:11:29.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:29.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:11:29.975 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:30.003 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:30.367 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:11:30.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:29.235174+0000 mgr.y (mgr.14550) 425 : cluster [DBG] pgmap v686: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 45 KiB/s rd, 722 B/s wr, 75 op/s 2024-02-19T21:11:30.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:29.354005+0000 mon.b (mon.1) 614 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:30.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:29.370381+0000 mon.a (mon.0) 2643 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:11:30.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:29.370513+0000 mon.a (mon.0) 2644 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:30.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:29.381871+0000 mon.a (mon.0) 2645 : audit [INF] from='client.? v1:172.21.15.163:0/2176739845' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:11:30.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:29.381930+0000 mon.a (mon.0) 2646 : audit [INF] from='client.49964 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "61.1e", "id": [7, 6, 3, 1, 0, 2]}]': finished 2024-02-19T21:11:30.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:29.382135+0000 mon.a (mon.0) 2647 : cluster [DBG] osdmap e461: 8 total, 8 up, 8 in 2024-02-19T21:11:30.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:29.383632+0000 mon.b (mon.1) 615 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-71"}]: dispatch 2024-02-19T21:11:30.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:29.386928+0000 mon.a (mon.0) 2648 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-71"}]: dispatch 2024-02-19T21:11:30.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:29.235174+0000 mgr.y (mgr.14550) 425 : cluster [DBG] pgmap v686: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 45 KiB/s rd, 722 B/s wr, 75 op/s 2024-02-19T21:11:30.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:29.354005+0000 mon.b (mon.1) 614 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:30.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:29.370381+0000 mon.a (mon.0) 2643 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:11:30.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:29.370513+0000 mon.a (mon.0) 2644 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:30.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:29.381871+0000 mon.a (mon.0) 2645 : audit [INF] from='client.? v1:172.21.15.163:0/2176739845' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:11:30.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:29.381930+0000 mon.a (mon.0) 2646 : audit [INF] from='client.49964 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "61.1e", "id": [7, 6, 3, 1, 0, 2]}]': finished 2024-02-19T21:11:30.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:29.382135+0000 mon.a (mon.0) 2647 : cluster [DBG] osdmap e461: 8 total, 8 up, 8 in 2024-02-19T21:11:30.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:29.383632+0000 mon.b (mon.1) 615 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-71"}]: dispatch 2024-02-19T21:11:30.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:29.386928+0000 mon.a (mon.0) 2648 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-71"}]: dispatch 2024-02-19T21:11:30.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:11:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:11:30.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:29.235174+0000 mgr.y (mgr.14550) 425 : cluster [DBG] pgmap v686: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 45 KiB/s rd, 722 B/s wr, 75 op/s 2024-02-19T21:11:30.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:29.354005+0000 mon.b (mon.1) 614 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:30.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:29.370381+0000 mon.a (mon.0) 2643 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:11:30.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:29.370513+0000 mon.a (mon.0) 2644 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:30.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:29.381871+0000 mon.a (mon.0) 2645 : audit [INF] from='client.? v1:172.21.15.163:0/2176739845' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:11:30.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:29.381930+0000 mon.a (mon.0) 2646 : audit [INF] from='client.49964 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "61.1e", "id": [7, 6, 3, 1, 0, 2]}]': finished 2024-02-19T21:11:30.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:29.382135+0000 mon.a (mon.0) 2647 : cluster [DBG] osdmap e461: 8 total, 8 up, 8 in 2024-02-19T21:11:30.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:29.383632+0000 mon.b (mon.1) 615 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-71"}]: dispatch 2024-02-19T21:11:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:29.386928+0000 mon.a (mon.0) 2648 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-71"}]: dispatch 2024-02-19T21:11:30.851 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:30.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:30.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:30.955 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:30.982 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:31.431 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:31.433 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:31.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:31.535 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:11:31.567 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:31.589 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:11:31.592 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:31.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:11:31.693 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:31.719 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:30.385629+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e462: 8 total, 8 up, 8 in 2024-02-19T21:11:31.719 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:30.472366+0000 mgr.y (mgr.14550) 426 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:31.721 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:31.740 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:31.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:31.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:31.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:30.385629+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e462: 8 total, 8 up, 8 in 2024-02-19T21:11:31.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:30.472366+0000 mgr.y (mgr.14550) 426 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:31.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:30.385629+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e462: 8 total, 8 up, 8 in 2024-02-19T21:11:31.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:30.472366+0000 mgr.y (mgr.14550) 426 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:31.805 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:31.844 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:31.872 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:31.891 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:31.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:31.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:31.995 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:32.022 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:32.043 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:32.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:32.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:32.147 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:32.175 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:32.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:30.469686+0000 osd.7 (osd.7) 209 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:11:32.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:30.472587+0000 osd.7 (osd.7) 210 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:11:32.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:30.888395+0000 osd.5 (osd.5) 196 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:11:32.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:30.893477+0000 osd.5 (osd.5) 197 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:11:32.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:31.235697+0000 mgr.y (mgr.14550) 427 : cluster [DBG] pgmap v689: 196 pgs: 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 101 op/s 2024-02-19T21:11:32.695 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:31.396049+0000 mon.a (mon.0) 2650 : cluster [DBG] osdmap e463: 8 total, 8 up, 8 in 2024-02-19T21:11:32.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:31.402566+0000 mon.b (mon.1) 616 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:32.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:31.403216+0000 mon.a (mon.0) 2651 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:32.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:31.497115+0000 osd.7 (osd.7) 211 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:11:32.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:31.500506+0000 osd.7 (osd.7) 212 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:11:32.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:31.628114+0000 osd.3 (osd.3) 203 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:11:32.696 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:31.629054+0000 osd.3 (osd.3) 204 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:11:32.700 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:31.667121+0000 osd.6 (osd.6) 193 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:11:32.700 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:31.670022+0000 osd.6 (osd.6) 194 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:11:32.700 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:30.469686+0000 osd.7 (osd.7) 209 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:11:32.700 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:30.472587+0000 osd.7 (osd.7) 210 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:11:32.700 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:30.888395+0000 osd.5 (osd.5) 196 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:11:32.700 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:30.893477+0000 osd.5 (osd.5) 197 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:11:32.700 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:31.235697+0000 mgr.y (mgr.14550) 427 : cluster [DBG] pgmap v689: 196 pgs: 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 101 op/s 2024-02-19T21:11:32.700 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:31.396049+0000 mon.a (mon.0) 2650 : cluster [DBG] osdmap e463: 8 total, 8 up, 8 in 2024-02-19T21:11:32.700 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:31.402566+0000 mon.b (mon.1) 616 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:32.701 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:31.403216+0000 mon.a (mon.0) 2651 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:32.701 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:31.497115+0000 osd.7 (osd.7) 211 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:11:32.701 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:31.500506+0000 osd.7 (osd.7) 212 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:11:32.701 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:31.628114+0000 osd.3 (osd.3) 203 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:11:32.701 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:31.629054+0000 osd.3 (osd.3) 204 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:11:32.701 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:31.667121+0000 osd.6 (osd.6) 193 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:11:32.701 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:31.670022+0000 osd.6 (osd.6) 194 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:11:32.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:30.469686+0000 osd.7 (osd.7) 209 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:11:32.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:30.472587+0000 osd.7 (osd.7) 210 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:11:32.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:30.888395+0000 osd.5 (osd.5) 196 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:11:32.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:30.893477+0000 osd.5 (osd.5) 197 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:11:32.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:31.235697+0000 mgr.y (mgr.14550) 427 : cluster [DBG] pgmap v689: 196 pgs: 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 0 B/s wr, 101 op/s 2024-02-19T21:11:32.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:31.396049+0000 mon.a (mon.0) 2650 : cluster [DBG] osdmap e463: 8 total, 8 up, 8 in 2024-02-19T21:11:32.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:31.402566+0000 mon.b (mon.1) 616 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:32.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:31.403216+0000 mon.a (mon.0) 2651 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:32.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:31.497115+0000 osd.7 (osd.7) 211 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:11:32.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:31.500506+0000 osd.7 (osd.7) 212 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:11:32.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:31.628114+0000 osd.3 (osd.3) 203 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:11:32.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:31.629054+0000 osd.3 (osd.3) 204 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:11:32.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:31.667121+0000 osd.6 (osd.6) 193 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:11:32.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:31.670022+0000 osd.6 (osd.6) 194 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:11:33.124 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:33.139 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:33.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:33.240 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:33.269 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:33.406 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:31.927591+0000 osd.1 (osd.1) 225 : cluster [DBG] 61.0 scrub starts 2024-02-19T21:11:33.407 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:31.931141+0000 osd.1 (osd.1) 226 : cluster [DBG] 61.0 scrub ok 2024-02-19T21:11:33.407 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:32.350129+0000 osd.4 (osd.4) 169 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:11:33.407 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:32.351875+0000 osd.4 (osd.4) 170 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:11:33.407 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:32.406591+0000 mon.a (mon.0) 2652 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:11:33.407 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:32.406680+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e464: 8 total, 8 up, 8 in 2024-02-19T21:11:33.407 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:32.410389+0000 mon.b (mon.1) 617 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:11:33.407 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:32.599418+0000 osd.3 (osd.3) 205 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:11:33.407 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:32.600208+0000 osd.3 (osd.3) 206 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:11:33.444 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:11:33.634 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:11:33.634 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-02-19T21:11:33.634 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-02-19T21:11:33.634 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-02-19T21:11:33.634 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.635 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24919.0:2074 28.18 28:19125b2f:test-rados-api-smithi163-83920-17::big3:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected e102)", 2024-02-19T21:11:33.635 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:42.228484+0000", 2024-02-19T21:11:33.635 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 410.49906861400001, 2024-02-19T21:11:33.635 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.066405645999999999, 2024-02-19T21:11:33.635 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.635 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:33.635 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:33.635 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24919", 2024-02-19T21:11:33.635 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/4089460683", 2024-02-19T21:11:33.635 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2074 2024-02-19T21:11:33.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.635 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.635 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.228484+0000", 2024-02-19T21:11:33.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.228484+0000", 2024-02-19T21:11:33.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999967 2024-02-19T21:11:33.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.636 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.636 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.228481+0000", 2024-02-19T21:11:33.636 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0067282870000000003 2024-02-19T21:11:33.636 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.636 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235209+0000", 2024-02-19T21:11:33.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8715999999999998e-05 2024-02-19T21:11:33.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.637 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.637 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235228+0000", 2024-02-19T21:11:33.637 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7010000000000002e-06 2024-02-19T21:11:33.637 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.637 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235237+0000", 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5971000000000003e-05 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235292+0000", 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000144649 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235437+0000", 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000111014 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.638 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,5", 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235548+0000", 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.050371437999999998 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.285920+0000", 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084572139999999994 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.294377+0000", 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040791500000000001 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:11:33.639 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.294785+0000", 2024-02-19T21:11:33.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6798000000000001e-05 2024-02-19T21:11:33.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.640 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:33.640 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.294842+0000", 2024-02-19T21:11:33.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8254000000000003e-05 2024-02-19T21:11:33.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.640 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.640 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.640 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.294890+0000", 2024-02-19T21:11:33.640 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.066405645999999999 2024-02-19T21:11:33.640 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.640 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.640 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.641 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25258.0:1 45.6s0 45:6dd63e69:test-rados-api-smithi163-83864-16::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e106)", 2024-02-19T21:11:33.641 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:45.971389+0000", 2024-02-19T21:11:33.641 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 406.75616406500001, 2024-02-19T21:11:33.641 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0304329539999999, 2024-02-19T21:11:33.641 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.641 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:33.641 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:33.641 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25258", 2024-02-19T21:11:33.641 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3498182538", 2024-02-19T21:11:33.641 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:11:33.641 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.641 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.641 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971389+0000", 2024-02-19T21:11:33.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971389+0000", 2024-02-19T21:11:33.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999967 2024-02-19T21:11:33.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.642 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.642 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971386+0000", 2024-02-19T21:11:33.642 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4117000000000001e-05 2024-02-19T21:11:33.642 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.642 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971400+0000", 2024-02-19T21:11:33.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.344e-05 2024-02-19T21:11:33.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971413+0000", 2024-02-19T21:11:33.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8219999999999996e-06 2024-02-19T21:11:33.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.643 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.643 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.643 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971422+0000", 2024-02-19T21:11:33.643 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2419999999999997e-05 2024-02-19T21:11:33.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971464+0000", 2024-02-19T21:11:33.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3889999999999998e-06 2024-02-19T21:11:33.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:11:33.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971469+0000", 2024-02-19T21:11:33.644 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0243061849999999 2024-02-19T21:11:33.644 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.644 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.644 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.644 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.995775+0000", 2024-02-19T21:11:33.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012886000000000001 2024-02-19T21:11:33.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.995904+0000", 2024-02-19T21:11:33.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023266189999999998 2024-02-19T21:11:33.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:11:33.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.998230+0000", 2024-02-19T21:11:33.645 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033170790000000001 2024-02-19T21:11:33.645 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.645 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.645 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:11:33.645 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.001547+0000", 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9880000000000002e-05 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.001617+0000", 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000204291 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.001822+0000", 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0304329539999999 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.646 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.647 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25318.0:2 48.6 48:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:11:33.647 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:47.974844+0000", 2024-02-19T21:11:33.647 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 404.7527091, 2024-02-19T21:11:33.647 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.048850169999999998, 2024-02-19T21:11:33.647 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.647 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:33.647 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:33.647 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25318", 2024-02-19T21:11:33.647 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/4284884290", 2024-02-19T21:11:33.647 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:11:33.647 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.647 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.647 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.647 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974844+0000", 2024-02-19T21:11:33.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974844+0000", 2024-02-19T21:11:33.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:11:33.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.648 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.648 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.648 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974841+0000", 2024-02-19T21:11:33.648 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2949999999999999e-06 2024-02-19T21:11:33.648 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974850+0000", 2024-02-19T21:11:33.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1317e-05 2024-02-19T21:11:33.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974862+0000", 2024-02-19T21:11:33.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6129999999999998e-06 2024-02-19T21:11:33.649 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.649 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.649 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.649 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974869+0000", 2024-02-19T21:11:33.649 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018860630000000001 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.976755+0000", 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2200000000000003e-06 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.976762+0000", 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043147748 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.019910+0000", 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9196000000000006e-05 2024-02-19T21:11:33.650 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.019979+0000", 2024-02-19T21:11:33.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141478 2024-02-19T21:11:33.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,5", 2024-02-19T21:11:33.651 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.020121+0000", 2024-02-19T21:11:33.651 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00053344600000000003 2024-02-19T21:11:33.651 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.651 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.651 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:11:33.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.020654+0000", 2024-02-19T21:11:33.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001117311 2024-02-19T21:11:33.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:33.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.021772+0000", 2024-02-19T21:11:33.653 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001839909 2024-02-19T21:11:33.653 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.653 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.653 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:33.653 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.023612+0000", 2024-02-19T21:11:33.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2648000000000002e-05 2024-02-19T21:11:33.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:33.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.023654+0000", 2024-02-19T21:11:33.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9617999999999999e-05 2024-02-19T21:11:33.654 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.654 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.654 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.654 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.023694+0000", 2024-02-19T21:11:33.654 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.048850169999999998 2024-02-19T21:11:33.654 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.654 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.654 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25294.0:1 51.3s0 51:d77cf283:test-rados-api-smithi163-83896-11::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:11:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:47.988394+0000", 2024-02-19T21:11:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 404.73915847400002, 2024-02-19T21:11:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.593213569, 2024-02-19T21:11:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25294", 2024-02-19T21:11:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2922168330", 2024-02-19T21:11:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:11:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.655 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988394+0000", 2024-02-19T21:11:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988394+0000", 2024-02-19T21:11:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988392+0000", 2024-02-19T21:11:33.656 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9700000000000002e-06 2024-02-19T21:11:33.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988399+0000", 2024-02-19T21:11:33.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3440000000000001e-06 2024-02-19T21:11:33.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988408+0000", 2024-02-19T21:11:33.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2559999999999996e-06 2024-02-19T21:11:33.657 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.657 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.657 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.657 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988415+0000", 2024-02-19T21:11:33.657 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013816699999999999 2024-02-19T21:11:33.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988553+0000", 2024-02-19T21:11:33.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.056e-06 2024-02-19T21:11:33.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:11:33.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988556+0000", 2024-02-19T21:11:33.658 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.57455238200000003 2024-02-19T21:11:33.658 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.658 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.658 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.658 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.563108+0000", 2024-02-19T21:11:33.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000100029 2024-02-19T21:11:33.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.659 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.659 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.563208+0000", 2024-02-19T21:11:33.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00041083999999999999 2024-02-19T21:11:33.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.659 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:11:33.659 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.563619+0000", 2024-02-19T21:11:33.659 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0070195659999999997 2024-02-19T21:11:33.659 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.659 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.659 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:11:33.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.570639+0000", 2024-02-19T21:11:33.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.01090327 2024-02-19T21:11:33.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:33.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.581542+0000", 2024-02-19T21:11:33.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5691999999999994e-05 2024-02-19T21:11:33.660 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.660 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.660 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.660 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.581608+0000", 2024-02-19T21:11:33.660 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.593213569 2024-02-19T21:11:33.660 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.660 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.661 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.661 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.661 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14952.0:2 53.6 53:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-19T21:11:33.661 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:50.605965+0000", 2024-02-19T21:11:33.661 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 402.12158785299999, 2024-02-19T21:11:33.661 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036561746999999999, 2024-02-19T21:11:33.661 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.661 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:33.661 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:33.661 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14952", 2024-02-19T21:11:33.661 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2459019407", 2024-02-19T21:11:33.661 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:11:33.661 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.662 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605965+0000", 2024-02-19T21:11:33.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.662 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605965+0000", 2024-02-19T21:11:33.662 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:11:33.662 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.662 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.662 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605963+0000", 2024-02-19T21:11:33.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1330000000000004e-06 2024-02-19T21:11:33.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605969+0000", 2024-02-19T21:11:33.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5689999999999999e-06 2024-02-19T21:11:33.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.663 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.663 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605977+0000", 2024-02-19T21:11:33.663 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4559999999999999e-06 2024-02-19T21:11:33.663 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.663 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605982+0000", 2024-02-19T21:11:33.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003383039 2024-02-19T21:11:33.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.609365+0000", 2024-02-19T21:11:33.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3409999999999997e-06 2024-02-19T21:11:33.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.664 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.664 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:11:33.664 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.609373+0000", 2024-02-19T21:11:33.664 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026694262 2024-02-19T21:11:33.664 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.665 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.665 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.636067+0000", 2024-02-19T21:11:33.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000103708 2024-02-19T21:11:33.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.665 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.665 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.636171+0000", 2024-02-19T21:11:33.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118288 2024-02-19T21:11:33.665 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.665 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.665 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4", 2024-02-19T21:11:33.665 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.636289+0000", 2024-02-19T21:11:33.665 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0017480130000000001 2024-02-19T21:11:33.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:33.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.638037+0000", 2024-02-19T21:11:33.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00049063000000000004 2024-02-19T21:11:33.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:11:33.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.638528+0000", 2024-02-19T21:11:33.666 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039461189999999997 2024-02-19T21:11:33.666 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.666 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.666 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:33.666 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.642474+0000", 2024-02-19T21:11:33.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1164000000000002e-05 2024-02-19T21:11:33.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.667 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:33.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.642505+0000", 2024-02-19T21:11:33.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1668999999999999e-05 2024-02-19T21:11:33.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.667 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.667 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.667 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.642527+0000", 2024-02-19T21:11:33.667 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036561746999999999 2024-02-19T21:11:33.667 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.667 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.667 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.667 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.668 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.668 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24865.0:21 11.8 11:1381d4a2:test-rados-api-smithi163-83952-6::foo:head [create,delete] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-19T21:11:33.668 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:51.573042+0000", 2024-02-19T21:11:33.668 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 401.15451078699999, 2024-02-19T21:11:33.668 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.070965300999999995, 2024-02-19T21:11:33.668 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.668 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:33.668 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:33.668 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24865", 2024-02-19T21:11:33.668 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3187889786", 2024-02-19T21:11:33.668 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 21 2024-02-19T21:11:33.668 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.668 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573042+0000", 2024-02-19T21:11:33.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573042+0000", 2024-02-19T21:11:33.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:33.669 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.669 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.669 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.669 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573040+0000", 2024-02-19T21:11:33.669 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2414e-05 2024-02-19T21:11:33.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573052+0000", 2024-02-19T21:11:33.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00026839099999999999 2024-02-19T21:11:33.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573321+0000", 2024-02-19T21:11:33.670 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0720000000000001e-05 2024-02-19T21:11:33.670 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.670 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.670 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.670 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573331+0000", 2024-02-19T21:11:33.671 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012845001 2024-02-19T21:11:33.671 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.671 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.671 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.671 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.586176+0000", 2024-02-19T21:11:33.671 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5872999999999995e-05 2024-02-19T21:11:33.671 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.671 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.671 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.671 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.586272+0000", 2024-02-19T21:11:33.671 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018172699999999999 2024-02-19T21:11:33.671 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.671 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-02-19T21:11:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.586454+0000", 2024-02-19T21:11:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034122739999999999 2024-02-19T21:11:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:11:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.589866+0000", 2024-02-19T21:11:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047893630999999999 2024-02-19T21:11:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.637760+0000", 2024-02-19T21:11:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061464759999999997 2024-02-19T21:11:33.672 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.643906+0000", 2024-02-19T21:11:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4803999999999999e-05 2024-02-19T21:11:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.643961+0000", 2024-02-19T21:11:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6133e-05 2024-02-19T21:11:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.644007+0000", 2024-02-19T21:11:33.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.070965300999999995 2024-02-19T21:11:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.35590.0:2 78.6 e128/127 78:602f83fe:::foo:head v 128'1, mlcod=0'0)", 2024-02-19T21:11:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:08.266769+0000", 2024-02-19T21:11:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 384.46078411500002, 2024-02-19T21:11:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057433619999999998, 2024-02-19T21:11:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266769+0000", 2024-02-19T21:11:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266769+0000", 2024-02-19T21:11:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:11:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266768+0000", 2024-02-19T21:11:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2753e-05 2024-02-19T21:11:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266781+0000", 2024-02-19T21:11:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6710000000000004e-06 2024-02-19T21:11:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266787+0000", 2024-02-19T21:11:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8820000000000001e-06 2024-02-19T21:11:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266795+0000", 2024-02-19T21:11:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001516304 2024-02-19T21:11:33.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.268311+0000", 2024-02-19T21:11:33.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7995e-05 2024-02-19T21:11:33.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.268329+0000", 2024-02-19T21:11:33.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055833582999999999 2024-02-19T21:11:33.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:33.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.324162+0000", 2024-02-19T21:11:33.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9811000000000002e-05 2024-02-19T21:11:33.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.324202+0000", 2024-02-19T21:11:33.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057433619999999998 2024-02-19T21:11:33.678 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.678 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.678 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.678 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.26962.0:28 64.1e e128/116 64:7e799bab:test-rados-api-smithi163-83952-10::foo:head v 128'1, mlcod=0'0)", 2024-02-19T21:11:33.678 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:08.266799+0000", 2024-02-19T21:11:33.678 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 384.46075358799999, 2024-02-19T21:11:33.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057427275999999999, 2024-02-19T21:11:33.678 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.679 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:33.679 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266799+0000", 2024-02-19T21:11:33.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266799+0000", 2024-02-19T21:11:33.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:11:33.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266799+0000", 2024-02-19T21:11:33.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4129999999999999e-06 2024-02-19T21:11:33.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266803+0000", 2024-02-19T21:11:33.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6290000000000001e-06 2024-02-19T21:11:33.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266808+0000", 2024-02-19T21:11:33.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4929999999999998e-06 2024-02-19T21:11:33.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266811+0000", 2024-02-19T21:11:33.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001716396 2024-02-19T21:11:33.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.268527+0000", 2024-02-19T21:11:33.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.647e-05 2024-02-19T21:11:33.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.268544+0000", 2024-02-19T21:11:33.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055662087999999998 2024-02-19T21:11:33.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.682 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:33.682 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.324206+0000", 2024-02-19T21:11:33.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0389000000000001e-05 2024-02-19T21:11:33.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.682 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.682 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.324226+0000", 2024-02-19T21:11:33.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057427275999999999 2024-02-19T21:11:33.682 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.682 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.682 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.682 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:110 75.1e 75:7b0234b0:test-rados-api-smithi163-83928-10::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e129)", 2024-02-19T21:11:33.682 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:09.003796+0000", 2024-02-19T21:11:33.683 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 383.72375717, 2024-02-19T21:11:33.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.093054726000000004, 2024-02-19T21:11:33.683 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.683 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:11:33.683 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:33.683 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:11:33.683 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:11:33.683 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 110 2024-02-19T21:11:33.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.683 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003796+0000", 2024-02-19T21:11:33.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003796+0000", 2024-02-19T21:11:33.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:33.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003793+0000", 2024-02-19T21:11:33.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6710000000000005e-06 2024-02-19T21:11:33.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003802+0000", 2024-02-19T21:11:33.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2583e-05 2024-02-19T21:11:33.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003814+0000", 2024-02-19T21:11:33.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8140000000000007e-06 2024-02-19T21:11:33.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003823+0000", 2024-02-19T21:11:33.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.905e-05 2024-02-19T21:11:33.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003872+0000", 2024-02-19T21:11:33.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1829999999999997e-06 2024-02-19T21:11:33.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:11:33.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003880+0000", 2024-02-19T21:11:33.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010142976999999999 2024-02-19T21:11:33.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.014023+0000", 2024-02-19T21:11:33.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000209595 2024-02-19T21:11:33.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", 2024-02-19T21:11:33.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.014232+0000", 2024-02-19T21:11:33.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017728766999999999 2024-02-19T21:11:33.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.031961+0000", 2024-02-19T21:11:33.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0221000000000003e-05 2024-02-19T21:11:33.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-02-19T21:11:33.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032041+0000", 2024-02-19T21:11:33.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.059847310000000001 2024-02-19T21:11:33.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.091888+0000", 2024-02-19T21:11:33.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118545 2024-02-19T21:11:33.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.688 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.092007+0000", 2024-02-19T21:11:33.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0048433310000000002 2024-02-19T21:11:33.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.688 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.096850+0000", 2024-02-19T21:11:33.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.093054726000000004 2024-02-19T21:11:33.688 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.688 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.688 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.689 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(osd.3.0:301 75.1e e129/125 ondisk, result = 0)", 2024-02-19T21:11:33.689 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:09.032674+0000", 2024-02-19T21:11:33.689 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 383.69487832900001, 2024-02-19T21:11:33.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036648739999999999, 2024-02-19T21:11:33.689 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.689 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:11:33.689 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032674+0000", 2024-02-19T21:11:33.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032674+0000", 2024-02-19T21:11:33.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:11:33.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032673+0000", 2024-02-19T21:11:33.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2400000000000001e-06 2024-02-19T21:11:33.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032680+0000", 2024-02-19T21:11:33.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5481e-05 2024-02-19T21:11:33.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032696+0000", 2024-02-19T21:11:33.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5610000000000001e-06 2024-02-19T21:11:33.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.691 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.691 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032703+0000", 2024-02-19T21:11:33.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036602245999999998 2024-02-19T21:11:33.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.069305+0000", 2024-02-19T21:11:33.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4330000000000004e-06 2024-02-19T21:11:33.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.069314+0000", 2024-02-19T21:11:33.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3780000000000007e-06 2024-02-19T21:11:33.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.069323+0000", 2024-02-19T21:11:33.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036648739999999999 2024-02-19T21:11:33.692 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.693 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.693 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.693 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:111 75.5 75:a23b9a9e:test-rados-api-smithi163-83928-10::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e129)", 2024-02-19T21:11:33.693 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:09.097539+0000", 2024-02-19T21:11:33.693 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 383.63001338599997, 2024-02-19T21:11:33.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.13010182100000001, 2024-02-19T21:11:33.693 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.693 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:11:33.694 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:33.694 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:11:33.694 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:11:33.694 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 111 2024-02-19T21:11:33.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.694 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097539+0000", 2024-02-19T21:11:33.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097539+0000", 2024-02-19T21:11:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999962 2024-02-19T21:11:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097536+0000", 2024-02-19T21:11:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4768e-05 2024-02-19T21:11:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097550+0000", 2024-02-19T21:11:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8895000000000001e-05 2024-02-19T21:11:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097569+0000", 2024-02-19T21:11:33.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6809999999999993e-06 2024-02-19T21:11:33.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097579+0000", 2024-02-19T21:11:33.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001749121 2024-02-19T21:11:33.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.099328+0000", 2024-02-19T21:11:33.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00027112000000000002 2024-02-19T21:11:33.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", 2024-02-19T21:11:33.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.099599+0000", 2024-02-19T21:11:33.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002897892 2024-02-19T21:11:33.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.102497+0000", 2024-02-19T21:11:33.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6007000000000001e-05 2024-02-19T21:11:33.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-02-19T21:11:33.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.102553+0000", 2024-02-19T21:11:33.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.12498295099999999 2024-02-19T21:11:33.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.227536+0000", 2024-02-19T21:11:33.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105049 2024-02-19T21:11:33.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.227641+0000", 2024-02-19T21:11:33.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.13010182100000001 2024-02-19T21:11:33.698 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.698 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.698 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.699 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(76.3s1 133/126 ECSubWrite(tid=279, reqid=osd.4.0:279, at_version=133'6, trim_to=0'0, roll_forward_to=132'4))", 2024-02-19T21:11:33.699 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:13.106749+0000", 2024-02-19T21:11:33.699 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 379.62080418599999, 2024-02-19T21:11:33.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039746143999999997, 2024-02-19T21:11:33.699 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.699 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:11:33.699 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.106749+0000", 2024-02-19T21:11:33.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.699 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.699 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.106749+0000", 2024-02-19T21:11:33.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:11:33.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.106747+0000", 2024-02-19T21:11:33.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3730999999999999e-05 2024-02-19T21:11:33.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.106781+0000", 2024-02-19T21:11:33.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015052899999999999 2024-02-19T21:11:33.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.121834+0000", 2024-02-19T21:11:33.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7899999999999994e-06 2024-02-19T21:11:33.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.121844+0000", 2024-02-19T21:11:33.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001291719 2024-02-19T21:11:33.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.123135+0000", 2024-02-19T21:11:33.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0396e-05 2024-02-19T21:11:33.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:11:33.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.123146+0000", 2024-02-19T21:11:33.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023338983000000001 2024-02-19T21:11:33.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:11:33.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.146485+0000", 2024-02-19T21:11:33.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0078000000000001e-05 2024-02-19T21:11:33.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.146495+0000", 2024-02-19T21:11:33.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039746143999999997 2024-02-19T21:11:33.702 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.703 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.703 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.703 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.39709.0:2 84.3 84:c656db43:::oid_RoundTripPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-02-19T21:11:33.703 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:13.135107+0000", 2024-02-19T21:11:33.703 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 379.59244564099998, 2024-02-19T21:11:33.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037126581999999998, 2024-02-19T21:11:33.703 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.703 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:33.703 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:33.703 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.39709", 2024-02-19T21:11:33.703 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3505054706", 2024-02-19T21:11:33.703 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:11:33.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.704 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135107+0000", 2024-02-19T21:11:33.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135107+0000", 2024-02-19T21:11:33.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:11:33.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135104+0000", 2024-02-19T21:11:33.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8099999999999992e-06 2024-02-19T21:11:33.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135114+0000", 2024-02-19T21:11:33.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1422e-05 2024-02-19T21:11:33.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135125+0000", 2024-02-19T21:11:33.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5689999999999999e-06 2024-02-19T21:11:33.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135133+0000", 2024-02-19T21:11:33.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8407e-05 2024-02-19T21:11:33.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135171+0000", 2024-02-19T21:11:33.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2189999999999998e-06 2024-02-19T21:11:33.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:11:33.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135179+0000", 2024-02-19T21:11:33.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034870916000000002 2024-02-19T21:11:33.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.170050+0000", 2024-02-19T21:11:33.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011553900000000001 2024-02-19T21:11:33.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.170165+0000", 2024-02-19T21:11:33.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019117400000000001 2024-02-19T21:11:33.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,7", 2024-02-19T21:11:33.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.170356+0000", 2024-02-19T21:11:33.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015803569999999999 2024-02-19T21:11:33.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:11:33.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.171937+0000", 2024-02-19T21:11:33.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000119746 2024-02-19T21:11:33.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:33.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.172056+0000", 2024-02-19T21:11:33.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104515 2024-02-19T21:11:33.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:33.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.172161+0000", 2024-02-19T21:11:33.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2716000000000003e-05 2024-02-19T21:11:33.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:33.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.172194+0000", 2024-02-19T21:11:33.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-02-19T21:11:33.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.172234+0000", 2024-02-19T21:11:33.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037126581999999998 2024-02-19T21:11:33.709 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.709 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.710 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.710 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.0.0:267 64.1e e134/116 64:7e799bab:test-rados-api-smithi163-83952-10::foo:b v 134'20, mlcod=131'16)", 2024-02-19T21:11:33.710 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:14.266716+0000", 2024-02-19T21:11:33.710 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 378.46083687700002, 2024-02-19T21:11:33.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.083584948000000006, 2024-02-19T21:11:33.710 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.710 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:33.710 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.710 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266716+0000", 2024-02-19T21:11:33.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266716+0000", 2024-02-19T21:11:33.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:11:33.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266715+0000", 2024-02-19T21:11:33.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3909000000000001e-05 2024-02-19T21:11:33.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266739+0000", 2024-02-19T21:11:33.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.178e-06 2024-02-19T21:11:33.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266745+0000", 2024-02-19T21:11:33.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2620000000000004e-06 2024-02-19T21:11:33.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266752+0000", 2024-02-19T21:11:33.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015674777000000001 2024-02-19T21:11:33.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.282426+0000", 2024-02-19T21:11:33.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4134999999999998e-05 2024-02-19T21:11:33.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.282450+0000", 2024-02-19T21:11:33.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067824522999999998 2024-02-19T21:11:33.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:33.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.350275+0000", 2024-02-19T21:11:33.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5811000000000001e-05 2024-02-19T21:11:33.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.350301+0000", 2024-02-19T21:11:33.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.083584948000000006 2024-02-19T21:11:33.714 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.714 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.714 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.714 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:257 85.2 85:43403dbe:test-rados-api-smithi163-83928-12::bam:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e138)", 2024-02-19T21:11:33.714 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:18.587348+0000", 2024-02-19T21:11:33.714 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 374.14020453400002, 2024-02-19T21:11:33.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.069587300000000005, 2024-02-19T21:11:33.714 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.715 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:11:33.715 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:33.715 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:11:33.715 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:11:33.715 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 257 2024-02-19T21:11:33.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.715 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.715 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587348+0000", 2024-02-19T21:11:33.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.715 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587348+0000", 2024-02-19T21:11:33.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999957 2024-02-19T21:11:33.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587344+0000", 2024-02-19T21:11:33.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6024000000000001e-05 2024-02-19T21:11:33.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587360+0000", 2024-02-19T21:11:33.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5661999999999999e-05 2024-02-19T21:11:33.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587386+0000", 2024-02-19T21:11:33.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0662999999999999e-05 2024-02-19T21:11:33.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587396+0000", 2024-02-19T21:11:33.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4304e-05 2024-02-19T21:11:33.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587450+0000", 2024-02-19T21:11:33.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023577900000000001 2024-02-19T21:11:33.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", 2024-02-19T21:11:33.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587686+0000", 2024-02-19T21:11:33.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0062748650000000001 2024-02-19T21:11:33.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.593961+0000", 2024-02-19T21:11:33.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8112000000000001e-05 2024-02-19T21:11:33.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-02-19T21:11:33.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.594029+0000", 2024-02-19T21:11:33.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.059753043999999998 2024-02-19T21:11:33.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.653782+0000", 2024-02-19T21:11:33.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7813999999999996e-05 2024-02-19T21:11:33.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.653880+0000", 2024-02-19T21:11:33.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003055412 2024-02-19T21:11:33.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.656935+0000", 2024-02-19T21:11:33.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.069587300000000005 2024-02-19T21:11:33.720 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.720 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.720 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.720 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.26962.0:57 64.3 64:c6c3d28d:test-rados-api-smithi163-83952-12::foo:head [write 0~128] snapc 14=[14] ondisk+write+ordersnap+known_if_redirected e139)", 2024-02-19T21:11:33.720 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:19.170619+0000", 2024-02-19T21:11:33.720 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 373.55693324399999, 2024-02-19T21:11:33.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.10751445900000001, 2024-02-19T21:11:33.720 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.720 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:33.720 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:33.720 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.26962", 2024-02-19T21:11:33.720 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2924448120", 2024-02-19T21:11:33.721 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 57 2024-02-19T21:11:33.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.721 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170619+0000", 2024-02-19T21:11:33.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170619+0000", 2024-02-19T21:11:33.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:33.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170617+0000", 2024-02-19T21:11:33.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0165e-05 2024-02-19T21:11:33.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170627+0000", 2024-02-19T21:11:33.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2262999999999999e-05 2024-02-19T21:11:33.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170640+0000", 2024-02-19T21:11:33.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2870000000000004e-06 2024-02-19T21:11:33.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170648+0000", 2024-02-19T21:11:33.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0032999999999999e-05 2024-02-19T21:11:33.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170708+0000", 2024-02-19T21:11:33.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7159999999999997e-06 2024-02-19T21:11:33.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:11:33.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170715+0000", 2024-02-19T21:11:33.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047117838000000002 2024-02-19T21:11:33.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.217833+0000", 2024-02-19T21:11:33.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012449 2024-02-19T21:11:33.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.217957+0000", 2024-02-19T21:11:33.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015379599999999999 2024-02-19T21:11:33.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,6", 2024-02-19T21:11:33.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.218111+0000", 2024-02-19T21:11:33.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014134015999999999 2024-02-19T21:11:33.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:33.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.232245+0000", 2024-02-19T21:11:33.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039615919999999999 2024-02-19T21:11:33.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:11:33.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.271861+0000", 2024-02-19T21:11:33.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061818860000000002 2024-02-19T21:11:33.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:33.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.278043+0000", 2024-02-19T21:11:33.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8167999999999997e-05 2024-02-19T21:11:33.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.726 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:33.726 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.278091+0000", 2024-02-19T21:11:33.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3086e-05 2024-02-19T21:11:33.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.726 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.726 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.278134+0000", 2024-02-19T21:11:33.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.10751445900000001 2024-02-19T21:11:33.726 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.726 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.726 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.727 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.49370.0:1 171.6s0 171:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-02-19T21:11:33.727 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:07:18.236944+0000", 2024-02-19T21:11:33.727 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 254.490608522, 2024-02-19T21:11:33.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2320055759999999, 2024-02-19T21:11:33.727 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.727 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:33.727 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:33.727 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.49370", 2024-02-19T21:11:33.727 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2010916473", 2024-02-19T21:11:33.727 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:11:33.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.727 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.236944+0000", 2024-02-19T21:11:33.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.236944+0000", 2024-02-19T21:11:33.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:33.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.236942+0000", 2024-02-19T21:11:33.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8730000000000007e-06 2024-02-19T21:11:33.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.236950+0000", 2024-02-19T21:11:33.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7600000000000008e-06 2024-02-19T21:11:33.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.236959+0000", 2024-02-19T21:11:33.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1589999999999998e-06 2024-02-19T21:11:33.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.236965+0000", 2024-02-19T21:11:33.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00044211899999999999 2024-02-19T21:11:33.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.237407+0000", 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.399e-06 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.237410+0000", 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2273940489999999 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:19.464804+0000", 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013275200000000001 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.731 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:19.464937+0000", 2024-02-19T21:11:33.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000395337 2024-02-19T21:11:33.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.731 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:11:33.731 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:19.465333+0000", 2024-02-19T21:11:33.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0011654949999999999 2024-02-19T21:11:33.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.731 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:11:33.731 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:19.466498+0000", 2024-02-19T21:11:33.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023873789999999998 2024-02-19T21:11:33.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:33.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:19.468885+0000", 2024-02-19T21:11:33.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4368000000000004e-05 2024-02-19T21:11:33.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:19.468950+0000", 2024-02-19T21:11:33.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2320055759999999 2024-02-19T21:11:33.732 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.732 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.732 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.732 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.49394.0:1 174.7s0 174:e732be06:::oid_WaitForSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e262)", 2024-02-19T21:11:33.733 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:07:22.216139+0000", 2024-02-19T21:11:33.733 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 250.51141408199999, 2024-02-19T21:11:33.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0674298339999999, 2024-02-19T21:11:33.733 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.733 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:33.733 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:33.733 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.49394", 2024-02-19T21:11:33.733 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3062946080", 2024-02-19T21:11:33.733 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:11:33.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.733 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216139+0000", 2024-02-19T21:11:33.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216139+0000", 2024-02-19T21:11:33.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:33.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216136+0000", 2024-02-19T21:11:33.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7820000000000005e-06 2024-02-19T21:11:33.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216146+0000", 2024-02-19T21:11:33.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1165000000000001e-05 2024-02-19T21:11:33.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216157+0000", 2024-02-19T21:11:33.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9519999999999994e-06 2024-02-19T21:11:33.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216165+0000", 2024-02-19T21:11:33.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6961e-05 2024-02-19T21:11:33.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.736 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.736 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216212+0000", 2024-02-19T21:11:33.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4010000000000001e-06 2024-02-19T21:11:33.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.736 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:11:33.736 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216216+0000", 2024-02-19T21:11:33.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.064811247 2024-02-19T21:11:33.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.736 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.736 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:23.281027+0000", 2024-02-19T21:11:33.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013814900000000001 2024-02-19T21:11:33.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:23.281165+0000", 2024-02-19T21:11:33.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00046475999999999999 2024-02-19T21:11:33.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:11:33.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:23.281630+0000", 2024-02-19T21:11:33.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00044463700000000001 2024-02-19T21:11:33.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:11:33.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:23.282074+0000", 2024-02-19T21:11:33.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001428971 2024-02-19T21:11:33.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:33.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:23.283503+0000", 2024-02-19T21:11:33.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5203000000000006e-05 2024-02-19T21:11:33.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:23.283568+0000", 2024-02-19T21:11:33.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0674298339999999 2024-02-19T21:11:33.738 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.738 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.738 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.739 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.25753.0:4348 191.8 e314/308 191:10fb1d02:::199:head v 314'8, mlcod=313'6)", 2024-02-19T21:11:33.739 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:08:14.258279+0000", 2024-02-19T21:11:33.739 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 198.46927392500001, 2024-02-19T21:11:33.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039096335000000003, 2024-02-19T21:11:33.739 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.739 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:33.739 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258279+0000", 2024-02-19T21:11:33.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258279+0000", 2024-02-19T21:11:33.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:11:33.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258277+0000", 2024-02-19T21:11:33.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1771e-05 2024-02-19T21:11:33.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.740 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258289+0000", 2024-02-19T21:11:33.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4579999999999993e-06 2024-02-19T21:11:33.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.740 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258297+0000", 2024-02-19T21:11:33.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.113e-06 2024-02-19T21:11:33.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258303+0000", 2024-02-19T21:11:33.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4646e-05 2024-02-19T21:11:33.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258337+0000", 2024-02-19T21:11:33.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1143999999999999e-05 2024-02-19T21:11:33.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258358+0000", 2024-02-19T21:11:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038976552999999997 2024-02-19T21:11:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:11:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.297335+0000", 2024-02-19T21:11:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0234000000000001e-05 2024-02-19T21:11:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.297375+0000", 2024-02-19T21:11:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039096335000000003 2024-02-19T21:11:33.742 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.743 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.743 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.743 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.34352.0:1422 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94681540387968 gen 4] snapc 0=[] ondisk+write+known_if_redirected e433)", 2024-02-19T21:11:33.743 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:10:40.311647+0000", 2024-02-19T21:11:33.743 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 52.415905352000003, 2024-02-19T21:11:33.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0091623380000001, 2024-02-19T21:11:33.743 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:11:33.743 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:11:33.743 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:11:33.743 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.34352", 2024-02-19T21:11:33.743 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:11:33.743 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1422 2024-02-19T21:11:33.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.744 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:11:33.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:11:33.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311647+0000", 2024-02-19T21:11:33.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:11:33.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:11:33.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311647+0000", 2024-02-19T21:11:33.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:11:33.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:11:33.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311645+0000", 2024-02-19T21:11:33.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4200000000000007e-06 2024-02-19T21:11:33.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:11:33.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311653+0000", 2024-02-19T21:11:33.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1905e-05 2024-02-19T21:11:33.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:11:33.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311665+0000", 2024-02-19T21:11:33.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1360000000000002e-06 2024-02-19T21:11:33.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:11:33.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311671+0000", 2024-02-19T21:11:33.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8489999999999998e-05 2024-02-19T21:11:33.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311720+0000", 2024-02-19T21:11:33.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9099999999999994e-06 2024-02-19T21:11:33.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:11:33.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311728+0000", 2024-02-19T21:11:33.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98640665500000002 2024-02-19T21:11:33.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:41.298135+0000", 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030824800000000001 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for degraded object", 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:41.298443+0000", 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016746660999999999 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:41.315190+0000", 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7898000000000001e-05 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:11:33.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:41.315238+0000", 2024-02-19T21:11:33.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.005571767 2024-02-19T21:11:33.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:11:33.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:11:33.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:11:33.748 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:41.320810+0000", 2024-02-19T21:11:33.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0091623380000001 2024-02-19T21:11:33.748 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.748 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.748 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.748 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:11:33.748 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:11:33.748 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:11:33.749 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:11:33] "GET /metrics HTTP/1.1" 200 189963 "" "Prometheus/2.33.4" 2024-02-19T21:11:33.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:31.927591+0000 osd.1 (osd.1) 225 : cluster [DBG] 61.0 scrub starts 2024-02-19T21:11:33.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:31.931141+0000 osd.1 (osd.1) 226 : cluster [DBG] 61.0 scrub ok 2024-02-19T21:11:33.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:32.350129+0000 osd.4 (osd.4) 169 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:11:33.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:32.351875+0000 osd.4 (osd.4) 170 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:11:33.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:32.406591+0000 mon.a (mon.0) 2652 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:11:33.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:32.406680+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e464: 8 total, 8 up, 8 in 2024-02-19T21:11:33.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:32.410389+0000 mon.b (mon.1) 617 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:11:33.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:32.599418+0000 osd.3 (osd.3) 205 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:11:33.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:32.600208+0000 osd.3 (osd.3) 206 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:11:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:31.927591+0000 osd.1 (osd.1) 225 : cluster [DBG] 61.0 scrub starts 2024-02-19T21:11:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:31.931141+0000 osd.1 (osd.1) 226 : cluster [DBG] 61.0 scrub ok 2024-02-19T21:11:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:32.350129+0000 osd.4 (osd.4) 169 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:11:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:32.351875+0000 osd.4 (osd.4) 170 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:11:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:32.406591+0000 mon.a (mon.0) 2652 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:11:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:32.406680+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e464: 8 total, 8 up, 8 in 2024-02-19T21:11:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:32.410389+0000 mon.b (mon.1) 617 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:11:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:32.599418+0000 osd.3 (osd.3) 205 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:11:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:32.600208+0000 osd.3 (osd.3) 206 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:11:34.018 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:34.020 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:34.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:34.122 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:34.154 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:34.175 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:34.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:34.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:34.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.266+0000 7fb66c1d4700 1 -- 172.21.15.163:0/3201208725 >> v1:172.21.15.163:6790/0 conn(0x7fb6641031b0 legacy=0x7fb664105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:34.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.266+0000 7fb66c1d4700 1 -- 172.21.15.163:0/3201208725 shutdown_connections 2024-02-19T21:11:34.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.266+0000 7fb66c1d4700 1 -- 172.21.15.163:0/3201208725 >> 172.21.15.163:0/3201208725 conn(0x7fb6640f2470 msgr2=0x7fb6640f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:34.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.266+0000 7fb66c1d4700 1 -- 172.21.15.163:0/3201208725 shutdown_connections 2024-02-19T21:11:34.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.266+0000 7fb66c1d4700 1 -- 172.21.15.163:0/3201208725 wait complete. 2024-02-19T21:11:34.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb66c1d4700 1 librados: starting msgr at 2024-02-19T21:11:34.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb66c1d4700 1 librados: starting objecter 2024-02-19T21:11:34.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb66c1d4700 1 Processor -- start 2024-02-19T21:11:34.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb66c1d4700 1 -- start start 2024-02-19T21:11:34.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb66c1d4700 1 librados: setting wanted keys 2024-02-19T21:11:34.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb66c1d4700 1 librados: calling monclient init 2024-02-19T21:11:34.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb66c1d4700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb664061830 con 0x7fb6641031b0 2024-02-19T21:11:34.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb66c1d4700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb664118680 con 0x7fb6640fc870 2024-02-19T21:11:34.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb66c1d4700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb664119820 con 0x7fb6640ff670 2024-02-19T21:11:34.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb66976f700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fb6640ff670 0x7fb664112820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:34900/0 (socket says 172.21.15.163:34900) 2024-02-19T21:11:34.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb66976f700 1 -- 172.21.15.163:0/2195445431 learned_addr learned my addr 172.21.15.163:0/2195445431 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:34.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb65affd700 10 client.?.objecter ms_handle_connect 0x7fb6640ff670 2024-02-19T21:11:34.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb65affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:34.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb65affd700 10 client.?.objecter ms_handle_connect 0x7fb6641031b0 2024-02-19T21:11:34.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb65affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:34.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb65affd700 10 client.?.objecter ms_handle_connect 0x7fb6640fc870 2024-02-19T21:11:34.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb65affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:34.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387837792 0 0) 0x7fb664119820 con 0x7fb6640ff670 2024-02-19T21:11:34.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6500038e0 con 0x7fb6640ff670 2024-02-19T21:11:34.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291974811 0 0) 0x7fb664061830 con 0x7fb6641031b0 2024-02-19T21:11:34.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb664119820 con 0x7fb6641031b0 2024-02-19T21:11:34.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673932743 0 0) 0x7fb664118680 con 0x7fb6640fc870 2024-02-19T21:11:34.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.267+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb664061830 con 0x7fb6640fc870 2024-02-19T21:11:34.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3424309892 0 0) 0x7fb6500038e0 con 0x7fb6640ff670 2024-02-19T21:11:34.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb664118680 con 0x7fb6640ff670 2024-02-19T21:11:34.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb6540018e0 con 0x7fb6640ff670 2024-02-19T21:11:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 154079199 0 0) 0x7fb664119820 con 0x7fb6641031b0 2024-02-19T21:11:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6500038e0 con 0x7fb6641031b0 2024-02-19T21:11:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb65c0018d0 con 0x7fb6641031b0 2024-02-19T21:11:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2995902272 0 0) 0x7fb664061830 con 0x7fb6640fc870 2024-02-19T21:11:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb664119820 con 0x7fb6640fc870 2024-02-19T21:11:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb6600018d0 con 0x7fb6640fc870 2024-02-19T21:11:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 670683820 0 0) 0x7fb664118680 con 0x7fb6640ff670 2024-02-19T21:11:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 >> v1:172.21.15.193:6789/0 conn(0x7fb6640fc870 legacy=0x7fb664060c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 >> v1:172.21.15.163:6789/0 conn(0x7fb6641031b0 legacy=0x7fb664115f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66411aa20 con 0x7fb6640ff670 2024-02-19T21:11:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6641176d0 con 0x7fb6640ff670 2024-02-19T21:11:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb66c1d4700 10 client.49970.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:34.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb664117b40 con 0x7fb6640ff670 2024-02-19T21:11:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb66c1d4700 1 librados: init done 2024-02-19T21:11:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb654001c90 con 0x7fb6640ff670 2024-02-19T21:11:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.268+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb654005b70 con 0x7fb6640ff670 2024-02-19T21:11:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.269+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb648004f00 con 0x7fb6640ff670 2024-02-19T21:11:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.269+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fb654002270 con 0x7fb6640ff670 2024-02-19T21:11:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(465..465 src has 43..465) v4 ==== 8092+0+0 (unknown 982134142 0 0) 0x7fb654078e10 con 0x7fb6640ff670 2024-02-19T21:11:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 osd_map(465..465 src has 43..465) v4 2024-02-19T21:11:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb65affd700 3 client.49970.objecter handle_osd_map got epochs [465,465] > 0 2024-02-19T21:11:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb65affd700 3 client.49970.objecter handle_osd_map decoding full epoch 465 2024-02-19T21:11:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb65affd700 20 client.49970.objecter dump_active .. 0 homeless 2024-02-19T21:11:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v465) v1 ==== 54+0+16 (unknown 3237333402 0 2420000192) 0x7fb65407b240 con 0x7fb6640ff670 2024-02-19T21:11:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb65affd700 10 client.49970.objecter ms_handle_connect 0x7fb650063c70 2024-02-19T21:11:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 1 2024-02-19T21:11:34.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb6480051a0 osd=0 3 2024-02-19T21:11:34.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb6480051a0 osd=0 3 2024-02-19T21:11:34.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb6480051a0 osd=0 3 2024-02-19T21:11:34.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 1 2024-02-19T21:11:34.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb6480051a0 osd=0 3 2024-02-19T21:11:34.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 0 1 2024-02-19T21:11:34.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb6480051a0 osd=0 4 2024-02-19T21:11:34.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb66c1d4700 10 client.49970.objecter _send_command 1 2024-02-19T21:11:34.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb648004950 con 0x7fb648004530 2024-02-19T21:11:34.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.270+0000 7fb65affd700 10 client.49970.objecter ms_handle_connect 0x7fb648004530 2024-02-19T21:11:34.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.271+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb648004950 con 0x7fb648004530 2024-02-19T21:11:34.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.271+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:11:34.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.271+0000 7fb65affd700 10 client.49970.objecter _finish_command 1 = system:0 2024-02-19T21:11:34.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.271+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb6480051a0 osd=0 4 2024-02-19T21:11:34.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.271+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 0 1 2024-02-19T21:11:34.279 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:34.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.286+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:34.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.286+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 2 2024-02-19T21:11:34.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.286+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb6480051a0 osd=0 3 2024-02-19T21:11:34.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.286+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb6480051a0 osd=0 3 2024-02-19T21:11:34.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.286+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb6480051a0 osd=0 3 2024-02-19T21:11:34.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.286+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 2 2024-02-19T21:11:34.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.286+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb6480051a0 osd=0 3 2024-02-19T21:11:34.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.286+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 0 2 2024-02-19T21:11:34.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.286+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb6480051a0 osd=0 4 2024-02-19T21:11:34.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.286+0000 7fb66c1d4700 10 client.49970.objecter _send_command 2 2024-02-19T21:11:34.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.286+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb648004d30 con 0x7fb648004530 2024-02-19T21:11:34.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.288+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb648004d30 con 0x7fb648004530 2024-02-19T21:11:34.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.288+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:34.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.288+0000 7fb65affd700 10 client.49970.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.288+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb6480051a0 osd=0 4 2024-02-19T21:11:34.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.288+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 0 2 2024-02-19T21:11:34.291 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:11:34.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.290+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:34.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.290+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 3 2024-02-19T21:11:34.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.290+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648003fe0 osd=1 3 2024-02-19T21:11:34.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.290+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648003fe0 osd=1 3 2024-02-19T21:11:34.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.290+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648003fe0 osd=1 3 2024-02-19T21:11:34.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.290+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 3 2024-02-19T21:11:34.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.290+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb648003fe0 osd=1 3 2024-02-19T21:11:34.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.290+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 1 3 2024-02-19T21:11:34.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.290+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648003fe0 osd=1 4 2024-02-19T21:11:34.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.290+0000 7fb66c1d4700 10 client.49970.objecter _send_command 3 2024-02-19T21:11:34.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.290+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb64800c000 con 0x7fb648004150 2024-02-19T21:11:34.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.292+0000 7fb65affd700 10 client.49970.objecter ms_handle_connect 0x7fb648004150 2024-02-19T21:11:34.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.293+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb64800c000 con 0x7fb648004150 2024-02-19T21:11:34.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.293+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:11:34.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.293+0000 7fb65affd700 10 client.49970.objecter _finish_command 3 = system:0 2024-02-19T21:11:34.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.293+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb648003fe0 osd=1 4 2024-02-19T21:11:34.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.293+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 1 3 2024-02-19T21:11:34.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.309+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:34.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.309+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 4 2024-02-19T21:11:34.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.309+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648003fe0 osd=1 3 2024-02-19T21:11:34.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.309+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648003fe0 osd=1 3 2024-02-19T21:11:34.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.309+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648003fe0 osd=1 3 2024-02-19T21:11:34.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.309+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 4 2024-02-19T21:11:34.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.309+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb648003fe0 osd=1 3 2024-02-19T21:11:34.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.309+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 1 4 2024-02-19T21:11:34.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.309+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648003fe0 osd=1 4 2024-02-19T21:11:34.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.309+0000 7fb66c1d4700 10 client.49970.objecter _send_command 4 2024-02-19T21:11:34.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.309+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb6480018b0 con 0x7fb648004150 2024-02-19T21:11:34.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.311+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb6480018b0 con 0x7fb648004150 2024-02-19T21:11:34.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.311+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:34.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.311+0000 7fb65affd700 10 client.49970.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.311+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb648003fe0 osd=1 4 2024-02-19T21:11:34.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.311+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 1 4 2024-02-19T21:11:34.311 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:11:34.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.312+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:34.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.312+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 5 2024-02-19T21:11:34.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.312+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648003630 osd=2 3 2024-02-19T21:11:34.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.312+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648003630 osd=2 3 2024-02-19T21:11:34.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.312+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648003630 osd=2 3 2024-02-19T21:11:34.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.312+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 5 2024-02-19T21:11:34.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.312+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb648003630 osd=2 3 2024-02-19T21:11:34.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.312+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 2 5 2024-02-19T21:11:34.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.312+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648003630 osd=2 4 2024-02-19T21:11:34.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.312+0000 7fb66c1d4700 10 client.49970.objecter _send_command 5 2024-02-19T21:11:34.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.312+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb64800de90 con 0x7fb64800d2f0 2024-02-19T21:11:34.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.312+0000 7fb65affd700 10 client.49970.objecter ms_handle_connect 0x7fb64800d2f0 2024-02-19T21:11:34.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.313+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb64800de90 con 0x7fb64800d2f0 2024-02-19T21:11:34.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.313+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:11:34.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.313+0000 7fb65affd700 10 client.49970.objecter _finish_command 5 = system:0 2024-02-19T21:11:34.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.313+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb648003630 osd=2 4 2024-02-19T21:11:34.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.313+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 2 5 2024-02-19T21:11:34.314 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:34.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.333+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:34.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.333+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 6 2024-02-19T21:11:34.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.333+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648003630 osd=2 3 2024-02-19T21:11:34.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.333+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648003630 osd=2 3 2024-02-19T21:11:34.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.333+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648003630 osd=2 3 2024-02-19T21:11:34.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.333+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 6 2024-02-19T21:11:34.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.333+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb648003630 osd=2 3 2024-02-19T21:11:34.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.333+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 2 6 2024-02-19T21:11:34.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.333+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648003630 osd=2 4 2024-02-19T21:11:34.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.333+0000 7fb66c1d4700 10 client.49970.objecter _send_command 6 2024-02-19T21:11:34.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.333+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb648003160 con 0x7fb64800d2f0 2024-02-19T21:11:34.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.334+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb648003160 con 0x7fb64800d2f0 2024-02-19T21:11:34.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.334+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:34.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.334+0000 7fb65affd700 10 client.49970.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.334+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb648003630 osd=2 4 2024-02-19T21:11:34.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.334+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 2 6 2024-02-19T21:11:34.334 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:11:34.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.335+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:34.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.335+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 7 2024-02-19T21:11:34.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.335+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648011970 osd=3 3 2024-02-19T21:11:34.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.335+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648011970 osd=3 3 2024-02-19T21:11:34.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.335+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648011970 osd=3 3 2024-02-19T21:11:34.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.335+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 7 2024-02-19T21:11:34.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.335+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb648011970 osd=3 3 2024-02-19T21:11:34.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.335+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 3 7 2024-02-19T21:11:34.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.335+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648011970 osd=3 4 2024-02-19T21:11:34.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.335+0000 7fb66c1d4700 10 client.49970.objecter _send_command 7 2024-02-19T21:11:34.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.335+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb648013c30 con 0x7fb648011ff0 2024-02-19T21:11:34.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.335+0000 7fb65affd700 10 client.49970.objecter ms_handle_connect 0x7fb648011ff0 2024-02-19T21:11:34.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.336+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb648013c30 con 0x7fb648011ff0 2024-02-19T21:11:34.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.336+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:11:34.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.336+0000 7fb65affd700 10 client.49970.objecter _finish_command 7 = system:0 2024-02-19T21:11:34.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.336+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb648011970 osd=3 4 2024-02-19T21:11:34.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.336+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 3 7 2024-02-19T21:11:34.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.351+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:34.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.351+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 8 2024-02-19T21:11:34.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.351+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648011970 osd=3 3 2024-02-19T21:11:34.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.351+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648011970 osd=3 3 2024-02-19T21:11:34.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.351+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648011970 osd=3 3 2024-02-19T21:11:34.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.351+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 8 2024-02-19T21:11:34.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.351+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb648011970 osd=3 3 2024-02-19T21:11:34.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.351+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 3 8 2024-02-19T21:11:34.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.351+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648011970 osd=3 4 2024-02-19T21:11:34.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.351+0000 7fb66c1d4700 10 client.49970.objecter _send_command 8 2024-02-19T21:11:34.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.351+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb64801a4f0 con 0x7fb648011ff0 2024-02-19T21:11:34.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb64801a4f0 con 0x7fb648011ff0 2024-02-19T21:11:34.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:34.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb65affd700 10 client.49970.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb648011970 osd=3 4 2024-02-19T21:11:34.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 3 8 2024-02-19T21:11:34.352 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:11:34.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:34.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 9 2024-02-19T21:11:34.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb64801a4f0 osd=4 3 2024-02-19T21:11:34.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb64801a4f0 osd=4 3 2024-02-19T21:11:34.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb64801a4f0 osd=4 3 2024-02-19T21:11:34.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 9 2024-02-19T21:11:34.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb64801a4f0 osd=4 3 2024-02-19T21:11:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 4 9 2024-02-19T21:11:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb64801a4f0 osd=4 4 2024-02-19T21:11:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb66c1d4700 10 client.49970.objecter _send_command 9 2024-02-19T21:11:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.353+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb64801e830 con 0x7fb64801ab70 2024-02-19T21:11:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.354+0000 7fb65affd700 10 client.49970.objecter ms_handle_connect 0x7fb64801ab70 2024-02-19T21:11:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.354+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb64801e830 con 0x7fb64801ab70 2024-02-19T21:11:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.354+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:11:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.354+0000 7fb65affd700 10 client.49970.objecter _finish_command 9 = system:0 2024-02-19T21:11:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.354+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb64801a4f0 osd=4 4 2024-02-19T21:11:34.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.354+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 4 9 2024-02-19T21:11:34.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.371+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:34.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.371+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 10 2024-02-19T21:11:34.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.371+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb64801a4f0 osd=4 3 2024-02-19T21:11:34.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.371+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb64801a4f0 osd=4 3 2024-02-19T21:11:34.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.371+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb64801a4f0 osd=4 3 2024-02-19T21:11:34.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.371+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 10 2024-02-19T21:11:34.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.371+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb64801a4f0 osd=4 3 2024-02-19T21:11:34.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.371+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 4 10 2024-02-19T21:11:34.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.371+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb64801a4f0 osd=4 4 2024-02-19T21:11:34.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.371+0000 7fb66c1d4700 10 client.49970.objecter _send_command 10 2024-02-19T21:11:34.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.371+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb64801e830 con 0x7fb64801ab70 2024-02-19T21:11:34.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb64801e830 con 0x7fb64801ab70 2024-02-19T21:11:34.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:34.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb65affd700 10 client.49970.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb64801a4f0 osd=4 4 2024-02-19T21:11:34.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 4 10 2024-02-19T21:11:34.372 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:11:34.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:34.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 11 2024-02-19T21:11:34.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb64801e830 osd=5 3 2024-02-19T21:11:34.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb64801e830 osd=5 3 2024-02-19T21:11:34.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb64801e830 osd=5 3 2024-02-19T21:11:34.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 11 2024-02-19T21:11:34.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb64801e830 osd=5 3 2024-02-19T21:11:34.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 5 11 2024-02-19T21:11:34.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb64801e830 osd=5 4 2024-02-19T21:11:34.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb66c1d4700 10 client.49970.objecter _send_command 11 2024-02-19T21:11:34.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.373+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb64801fa50 con 0x7fb648020530 2024-02-19T21:11:34.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.374+0000 7fb65affd700 10 client.49970.objecter ms_handle_connect 0x7fb648020530 2024-02-19T21:11:34.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.374+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb64801fa50 con 0x7fb648020530 2024-02-19T21:11:34.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.374+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:11:34.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.374+0000 7fb65affd700 10 client.49970.objecter _finish_command 11 = system:0 2024-02-19T21:11:34.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.374+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb64801e830 osd=5 4 2024-02-19T21:11:34.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.374+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 5 11 2024-02-19T21:11:34.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.389+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:34.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.389+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 12 2024-02-19T21:11:34.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.389+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb64801e830 osd=5 3 2024-02-19T21:11:34.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.389+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb64801e830 osd=5 3 2024-02-19T21:11:34.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.389+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb64801e830 osd=5 3 2024-02-19T21:11:34.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.389+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 12 2024-02-19T21:11:34.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.389+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb64801e830 osd=5 3 2024-02-19T21:11:34.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.389+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 5 12 2024-02-19T21:11:34.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.389+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb64801e830 osd=5 4 2024-02-19T21:11:34.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.389+0000 7fb66c1d4700 10 client.49970.objecter _send_command 12 2024-02-19T21:11:34.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.389+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb64801fa50 con 0x7fb648020530 2024-02-19T21:11:34.390 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:11:34.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb64801fa50 con 0x7fb648020530 2024-02-19T21:11:34.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:34.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb65affd700 10 client.49970.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb64801e830 osd=5 4 2024-02-19T21:11:34.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 5 12 2024-02-19T21:11:34.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:34.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 13 2024-02-19T21:11:34.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb64801fa50 osd=6 3 2024-02-19T21:11:34.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb64801fa50 osd=6 3 2024-02-19T21:11:34.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb64801fa50 osd=6 3 2024-02-19T21:11:34.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 13 2024-02-19T21:11:34.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb64801fa50 osd=6 3 2024-02-19T21:11:34.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 6 13 2024-02-19T21:11:34.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb64801fa50 osd=6 4 2024-02-19T21:11:34.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb66c1d4700 10 client.49970.objecter _send_command 13 2024-02-19T21:11:34.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.391+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb648027250 con 0x7fb648023740 2024-02-19T21:11:34.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.392+0000 7fb65affd700 10 client.49970.objecter ms_handle_connect 0x7fb648023740 2024-02-19T21:11:34.393 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:34.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.393+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb648027250 con 0x7fb648023740 2024-02-19T21:11:34.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.393+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:11:34.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.393+0000 7fb65affd700 10 client.49970.objecter _finish_command 13 = system:0 2024-02-19T21:11:34.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.393+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb64801fa50 osd=6 4 2024-02-19T21:11:34.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.393+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 6 13 2024-02-19T21:11:34.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.408+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:34.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.408+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 14 2024-02-19T21:11:34.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.408+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb64801fa50 osd=6 3 2024-02-19T21:11:34.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.408+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb64801fa50 osd=6 3 2024-02-19T21:11:34.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.408+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb64801fa50 osd=6 3 2024-02-19T21:11:34.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.408+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 14 2024-02-19T21:11:34.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.408+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb64801fa50 osd=6 3 2024-02-19T21:11:34.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.408+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 6 14 2024-02-19T21:11:34.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.408+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb64801fa50 osd=6 4 2024-02-19T21:11:34.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.408+0000 7fb66c1d4700 10 client.49970.objecter _send_command 14 2024-02-19T21:11:34.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.408+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb648027250 con 0x7fb648023740 2024-02-19T21:11:34.408 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:11:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:11:34.321+0000 7fe7b0176700 -1 snap_mapper.add_oid found existing snaps mapped on 61:0db63738:test-rados-api-smithi163-83928-74::foo:24, removing 2024-02-19T21:11:34.408 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:11:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:11:34.322+0000 7faab64ba700 -1 snap_mapper.add_oid found existing snaps mapped on 61:0db63738:test-rados-api-smithi163-83928-74::foo:24, removing 2024-02-19T21:11:34.409 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:32.939518+0000 osd.1 (osd.1) 227 : cluster [DBG] 61.1d scrub starts 2024-02-19T21:11:34.409 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:32.941412+0000 osd.1 (osd.1) 228 : cluster [DBG] 61.1d scrub ok 2024-02-19T21:11:34.410 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:33.236365+0000 mgr.y (mgr.14550) 428 : cluster [DBG] pgmap v692: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:11:34.410 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:33.410845+0000 mon.a (mon.0) 2654 : cluster [DBG] osdmap e465: 8 total, 8 up, 8 in 2024-02-19T21:11:34.410 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:33.466308+0000 osd.7 (osd.7) 213 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:11:34.410 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:33.469226+0000 osd.7 (osd.7) 214 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:11:34.410 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:34.271061+0000 mon.c (mon.2) 399 : audit [DBG] from='client.? v1:172.21.15.163:0/2195445431' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:34.410 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:34.291117+0000 mon.a (mon.0) 2655 : cluster [DBG] osdmap e466: 8 total, 8 up, 8 in 2024-02-19T21:11:34.410 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:34.345609+0000 mon.b (mon.1) 618 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:34.410 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:34.346035+0000 mon.a (mon.0) 2656 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:34.410 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:34.346563+0000 mon.b (mon.1) 619 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-73"}]: dispatch 2024-02-19T21:11:34.410 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:34.346789+0000 mon.a (mon.0) 2657 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-73"}]: dispatch 2024-02-19T21:11:34.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb648027250 con 0x7fb648023740 2024-02-19T21:11:34.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:34.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb65affd700 10 client.49970.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb64801fa50 osd=6 4 2024-02-19T21:11:34.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 6 14 2024-02-19T21:11:34.411 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:11:34.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:34.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 15 2024-02-19T21:11:34.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648027250 osd=7 3 2024-02-19T21:11:34.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648027250 osd=7 3 2024-02-19T21:11:34.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648027250 osd=7 3 2024-02-19T21:11:34.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 15 2024-02-19T21:11:34.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb648027250 osd=7 3 2024-02-19T21:11:34.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 7 15 2024-02-19T21:11:34.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648027250 osd=7 4 2024-02-19T21:11:34.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb66c1d4700 10 client.49970.objecter _send_command 15 2024-02-19T21:11:34.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.411+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb64802b520 con 0x7fb6480278d0 2024-02-19T21:11:34.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.412+0000 7fb65affd700 10 client.49970.objecter ms_handle_connect 0x7fb6480278d0 2024-02-19T21:11:34.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.413+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb64802b520 con 0x7fb6480278d0 2024-02-19T21:11:34.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.413+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:11:34.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.413+0000 7fb65affd700 10 client.49970.objecter _finish_command 15 = system:0 2024-02-19T21:11:34.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.413+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb648027250 osd=7 4 2024-02-19T21:11:34.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.413+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 7 15 2024-02-19T21:11:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:34.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:34.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.427+0000 7fb66c1d4700 10 client.49970.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:34.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.427+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign -1 16 2024-02-19T21:11:34.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.427+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648027250 osd=7 3 2024-02-19T21:11:34.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.427+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648027250 osd=7 3 2024-02-19T21:11:34.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.427+0000 7fb66c1d4700 20 client.49970.objecter _get_session s=0x7fb648027250 osd=7 3 2024-02-19T21:11:34.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.427+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_remove -1 16 2024-02-19T21:11:34.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.427+0000 7fb66c1d4700 20 client.49970.objecter get_session s=0x7fb648027250 osd=7 3 2024-02-19T21:11:34.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.427+0000 7fb66c1d4700 15 client.49970.objecter _session_command_op_assign 7 16 2024-02-19T21:11:34.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.427+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648027250 osd=7 4 2024-02-19T21:11:34.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.427+0000 7fb66c1d4700 10 client.49970.objecter _send_command 16 2024-02-19T21:11:34.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.427+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb64802ccd0 con 0x7fb6480278d0 2024-02-19T21:11:34.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.429+0000 7fb65affd700 1 -- 172.21.15.163:0/2195445431 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb64802ccd0 con 0x7fb6480278d0 2024-02-19T21:11:34.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.429+0000 7fb65affd700 10 client.49970.objecter ms_dispatch 0x7fb664001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:34.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.429+0000 7fb65affd700 10 client.49970.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.429+0000 7fb65affd700 20 client.49970.objecter put_session s=0x7fb648027250 osd=7 4 2024-02-19T21:11:34.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.429+0000 7fb65affd700 15 client.49970.objecter _session_command_op_remove 7 16 2024-02-19T21:11:34.429 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:11:34.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 10 librados: watch_flush enter 2024-02-19T21:11:34.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 10 librados: watch_flush exit 2024-02-19T21:11:34.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 10 client.49970.objecter close_session for osd.0 2024-02-19T21:11:34.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 >> v1:172.21.15.163:6801/1559418115 conn(0x7fb648004530 legacy=0x7fb648002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:34.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb6480051a0 osd=0 1 2024-02-19T21:11:34.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 10 client.49970.objecter close_session for osd.1 2024-02-19T21:11:34.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 >> v1:172.21.15.163:6805/3125047432 conn(0x7fb648004150 legacy=0x7fb64800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:34.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648003fe0 osd=1 1 2024-02-19T21:11:34.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 10 client.49970.objecter close_session for osd.2 2024-02-19T21:11:34.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 >> v1:172.21.15.163:6809/1563158487 conn(0x7fb64800d2f0 legacy=0x7fb64800d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:34.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648003630 osd=2 1 2024-02-19T21:11:34.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 10 client.49970.objecter close_session for osd.3 2024-02-19T21:11:34.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 >> v1:172.21.15.163:6813/2112279769 conn(0x7fb648011ff0 legacy=0x7fb648012450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:34.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648011970 osd=3 1 2024-02-19T21:11:34.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 10 client.49970.objecter close_session for osd.4 2024-02-19T21:11:34.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 >> v1:172.21.15.193:6800/3129716965 conn(0x7fb64801ab70 legacy=0x7fb64801cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:34.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66a771700 1 -- 172.21.15.163:0/2195445431 reap_dead start 2024-02-19T21:11:34.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb64801a4f0 osd=4 1 2024-02-19T21:11:34.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 10 client.49970.objecter close_session for osd.5 2024-02-19T21:11:34.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 >> v1:172.21.15.193:6804/805161991 conn(0x7fb648020530 legacy=0x7fb648020970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:34.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb64801e830 osd=5 1 2024-02-19T21:11:34.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 10 client.49970.objecter close_session for osd.6 2024-02-19T21:11:34.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 >> v1:172.21.15.193:6808/787604615 conn(0x7fb648023740 legacy=0x7fb648025b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:34.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb64801fa50 osd=6 1 2024-02-19T21:11:34.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 10 client.49970.objecter close_session for osd.7 2024-02-19T21:11:34.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 >> v1:172.21.15.193:6812/157927432 conn(0x7fb6480278d0 legacy=0x7fb648029cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:34.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 20 client.49970.objecter put_session s=0x7fb648027250 osd=7 1 2024-02-19T21:11:34.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 20 client.49970.objecter shutdown clearing up homeless session... 2024-02-19T21:11:34.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 10 client.49970.objecter successfully canceled tick 2024-02-19T21:11:34.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 >> v1:172.21.15.163:6800/1552514681 conn(0x7fb650063c70 legacy=0x7fb650066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:34.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 >> v1:172.21.15.163:6790/0 conn(0x7fb6640ff670 legacy=0x7fb664112820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:34.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66a771700 1 -- 172.21.15.163:0/2195445431 reap_dead start 2024-02-19T21:11:34.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66a771700 1 -- 172.21.15.163:0/2195445431 reap_dead start 2024-02-19T21:11:34.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66a771700 1 -- 172.21.15.163:0/2195445431 reap_dead start 2024-02-19T21:11:34.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66a771700 1 -- 172.21.15.163:0/2195445431 reap_dead start 2024-02-19T21:11:34.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 shutdown_connections 2024-02-19T21:11:34.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.430+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 >> 172.21.15.163:0/2195445431 conn(0x7fb6640f2470 msgr2=0x7fb664106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:34.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.431+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 shutdown_connections 2024-02-19T21:11:34.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.431+0000 7fb66c1d4700 1 -- 172.21.15.163:0/2195445431 wait complete. 2024-02-19T21:11:34.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:34.431+0000 7fb66c1d4700 1 librados: shutdown 2024-02-19T21:11:34.433 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.433 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.433 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.433 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.433 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.433 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.433 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.433 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:11:34.515 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:11:34.543 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:34.721 INFO:journalctl@ceph.osd.2.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-2[57708]: debug 2024-02-19T21:11:34.321+0000 7f5b31524700 -1 snap_mapper.add_oid found existing snaps mapped on 61:0db63738:test-rados-api-smithi163-83928-74::foo:24, removing 2024-02-19T21:11:34.721 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:32.939518+0000 osd.1 (osd.1) 227 : cluster [DBG] 61.1d scrub starts 2024-02-19T21:11:34.721 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:32.941412+0000 osd.1 (osd.1) 228 : cluster [DBG] 61.1d scrub ok 2024-02-19T21:11:34.721 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:33.236365+0000 mgr.y (mgr.14550) 428 : cluster [DBG] pgmap v692: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:11:34.721 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:33.410845+0000 mon.a (mon.0) 2654 : cluster [DBG] osdmap e465: 8 total, 8 up, 8 in 2024-02-19T21:11:34.721 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:33.466308+0000 osd.7 (osd.7) 213 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:11:34.721 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:33.469226+0000 osd.7 (osd.7) 214 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:11:34.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:34.271061+0000 mon.c (mon.2) 399 : audit [DBG] from='client.? v1:172.21.15.163:0/2195445431' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:34.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:34.291117+0000 mon.a (mon.0) 2655 : cluster [DBG] osdmap e466: 8 total, 8 up, 8 in 2024-02-19T21:11:34.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:34.345609+0000 mon.b (mon.1) 618 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:34.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:34.346035+0000 mon.a (mon.0) 2656 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:34.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:34.346563+0000 mon.b (mon.1) 619 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-73"}]: dispatch 2024-02-19T21:11:34.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:34.346789+0000 mon.a (mon.0) 2657 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-73"}]: dispatch 2024-02-19T21:11:34.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:32.939518+0000 osd.1 (osd.1) 227 : cluster [DBG] 61.1d scrub starts 2024-02-19T21:11:34.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:32.941412+0000 osd.1 (osd.1) 228 : cluster [DBG] 61.1d scrub ok 2024-02-19T21:11:34.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:33.236365+0000 mgr.y (mgr.14550) 428 : cluster [DBG] pgmap v692: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:11:34.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:33.410845+0000 mon.a (mon.0) 2654 : cluster [DBG] osdmap e465: 8 total, 8 up, 8 in 2024-02-19T21:11:34.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:33.466308+0000 osd.7 (osd.7) 213 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:11:34.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:33.469226+0000 osd.7 (osd.7) 214 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:11:34.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:34.271061+0000 mon.c (mon.2) 399 : audit [DBG] from='client.? v1:172.21.15.163:0/2195445431' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:34.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:34.291117+0000 mon.a (mon.0) 2655 : cluster [DBG] osdmap e466: 8 total, 8 up, 8 in 2024-02-19T21:11:34.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:34.345609+0000 mon.b (mon.1) 618 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:34.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:34.346035+0000 mon.a (mon.0) 2656 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:34.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:34.346563+0000 mon.b (mon.1) 619 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-73"}]: dispatch 2024-02-19T21:11:34.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:34.346789+0000 mon.a (mon.0) 2657 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-73"}]: dispatch 2024-02-19T21:11:34.738 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-19T21:11:35.000 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:11:35.393 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:11:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:35.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:11:35.452 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:35.499 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:11:35.529 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:35.728 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:11:35.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:35.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:11:35.827 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 3, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:11:35.827 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:11:35.827 INFO:tasks.thrashosds.thrasher:Growing pool .rgw.root 2024-02-19T21:11:35.827 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:11:35.831 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:11:35.899 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:36.508 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:34.501521+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:11:36.508 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:34.504473+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:11:36.508 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:34.521947+0000 osd.3 (osd.3) 207 : cluster [DBG] 61.a scrub starts 2024-02-19T21:11:36.508 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:34.523662+0000 osd.3 (osd.3) 208 : cluster [DBG] 61.a scrub ok 2024-02-19T21:11:36.508 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:35.237278+0000 mgr.y (mgr.14550) 429 : cluster [DBG] pgmap v695: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.0 KiB/s wr, 10 op/s 2024-02-19T21:11:36.509 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:35.292524+0000 mon.a (mon.0) 2658 : cluster [DBG] osdmap e467: 8 total, 8 up, 8 in 2024-02-19T21:11:36.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:34.501521+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:11:36.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:34.504473+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:11:36.741 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:34.521947+0000 osd.3 (osd.3) 207 : cluster [DBG] 61.a scrub starts 2024-02-19T21:11:36.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:34.523662+0000 osd.3 (osd.3) 208 : cluster [DBG] 61.a scrub ok 2024-02-19T21:11:36.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:35.237278+0000 mgr.y (mgr.14550) 429 : cluster [DBG] pgmap v695: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.0 KiB/s wr, 10 op/s 2024-02-19T21:11:36.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:35.292524+0000 mon.a (mon.0) 2658 : cluster [DBG] osdmap e467: 8 total, 8 up, 8 in 2024-02-19T21:11:36.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:34.501521+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:11:36.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:34.504473+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:11:36.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:34.521947+0000 osd.3 (osd.3) 207 : cluster [DBG] 61.a scrub starts 2024-02-19T21:11:36.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:34.523662+0000 osd.3 (osd.3) 208 : cluster [DBG] 61.a scrub ok 2024-02-19T21:11:36.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:35.237278+0000 mgr.y (mgr.14550) 429 : cluster [DBG] pgmap v695: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.0 KiB/s wr, 10 op/s 2024-02-19T21:11:36.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:35.292524+0000 mon.a (mon.0) 2658 : cluster [DBG] osdmap e467: 8 total, 8 up, 8 in 2024-02-19T21:11:36.749 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:11:36.752 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:36.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:11:36.771 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:11:36.771 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:11:36.776 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:36.853 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:11:36.881 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:37.029 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:11:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:37.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:11:37.133 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:11:37.163 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:37.301 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:36.301466+0000 mon.a (mon.0) 2659 : cluster [DBG] osdmap e468: 8 total, 8 up, 8 in 2024-02-19T21:11:37.301 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:36.305557+0000 mon.b (mon.1) 620 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:37.301 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:36.306112+0000 mon.a (mon.0) 2660 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:37.302 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:36.408459+0000 osd.2 (osd.2) 188 : cluster [DBG] 61.17 scrub starts 2024-02-19T21:11:37.302 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:36.409884+0000 osd.2 (osd.2) 189 : cluster [DBG] 61.17 scrub ok 2024-02-19T21:11:37.360 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:11:37.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:36.301466+0000 mon.a (mon.0) 2659 : cluster [DBG] osdmap e468: 8 total, 8 up, 8 in 2024-02-19T21:11:37.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:36.305557+0000 mon.b (mon.1) 620 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:37.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:36.306112+0000 mon.a (mon.0) 2660 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:37.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:36.408459+0000 osd.2 (osd.2) 188 : cluster [DBG] 61.17 scrub starts 2024-02-19T21:11:37.490 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:36.409884+0000 osd.2 (osd.2) 189 : cluster [DBG] 61.17 scrub ok 2024-02-19T21:11:37.507 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-19T21:11:37.512 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:11:37.514 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:37.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:11:37.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.563+0000 7f9d8b093700 1 -- 172.21.15.163:0/3177233647 >> v1:172.21.15.163:6790/0 conn(0x7f9d841007a0 legacy=0x7f9d84102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:37.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.563+0000 7f9d8b093700 1 -- 172.21.15.163:0/3177233647 shutdown_connections 2024-02-19T21:11:37.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.563+0000 7f9d8b093700 1 -- 172.21.15.163:0/3177233647 >> 172.21.15.163:0/3177233647 conn(0x7f9d8406cc00 msgr2=0x7f9d840f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:37.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.564+0000 7f9d8b093700 1 -- 172.21.15.163:0/3177233647 shutdown_connections 2024-02-19T21:11:37.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.564+0000 7f9d8b093700 1 -- 172.21.15.163:0/3177233647 wait complete. 2024-02-19T21:11:37.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.564+0000 7f9d8b093700 1 librados: starting msgr at 2024-02-19T21:11:37.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.564+0000 7f9d8b093700 1 librados: starting objecter 2024-02-19T21:11:37.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.564+0000 7f9d8b093700 1 Processor -- start 2024-02-19T21:11:37.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.564+0000 7f9d8b093700 1 -- start start 2024-02-19T21:11:37.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.564+0000 7f9d8b093700 1 librados: setting wanted keys 2024-02-19T21:11:37.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.564+0000 7f9d8b093700 1 librados: calling monclient init 2024-02-19T21:11:37.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.564+0000 7f9d8b093700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9d840fd020 con 0x7f9d840fd9c0 2024-02-19T21:11:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.564+0000 7f9d8b093700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9d840fd1b0 con 0x7f9d841007a0 2024-02-19T21:11:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.564+0000 7f9d8b093700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9d840fd3a0 con 0x7f9d841042e0 2024-02-19T21:11:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d88e2f700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f9d840fd9c0 0x7f9d840fc880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:39676/0 (socket says 172.21.15.163:39676) 2024-02-19T21:11:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d89630700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f9d841042e0 0x7f9d84118480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:60728/0 (socket says 172.21.15.163:60728) 2024-02-19T21:11:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d83fff700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f9d841007a0 0x7f9d84114d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:59188/0 (socket says 172.21.15.163:59188) 2024-02-19T21:11:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d88e2f700 1 -- 172.21.15.163:0/3001127475 learned_addr learned my addr 172.21.15.163:0/3001127475 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 10 client.?.objecter ms_handle_connect 0x7f9d840fd9c0 2024-02-19T21:11:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 10 client.?.objecter ms_handle_connect 0x7f9d841042e0 2024-02-19T21:11:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 10 client.?.objecter ms_handle_connect 0x7f9d841007a0 2024-02-19T21:11:37.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453912792 0 0) 0x7f9d840fd3a0 con 0x7f9d841042e0 2024-02-19T21:11:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d740033f0 con 0x7f9d841042e0 2024-02-19T21:11:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163986188 0 0) 0x7f9d840fd020 con 0x7f9d840fd9c0 2024-02-19T21:11:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d840fd3a0 con 0x7f9d840fd9c0 2024-02-19T21:11:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385730023 0 0) 0x7f9d840fd1b0 con 0x7f9d841007a0 2024-02-19T21:11:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d840fd020 con 0x7f9d841007a0 2024-02-19T21:11:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 675275869 0 0) 0x7f9d840fd020 con 0x7f9d841007a0 2024-02-19T21:11:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d840fd1b0 con 0x7f9d841007a0 2024-02-19T21:11:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9d70002050 con 0x7f9d841007a0 2024-02-19T21:11:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 870983454 0 0) 0x7f9d740033f0 con 0x7f9d841042e0 2024-02-19T21:11:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d840fd020 con 0x7f9d841042e0 2024-02-19T21:11:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2095802032 0 0) 0x7f9d840fd3a0 con 0x7f9d840fd9c0 2024-02-19T21:11:37.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d740033f0 con 0x7f9d840fd9c0 2024-02-19T21:11:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9d7c001bf0 con 0x7f9d841042e0 2024-02-19T21:11:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9d78002930 con 0x7f9d840fd9c0 2024-02-19T21:11:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3317930221 0 0) 0x7f9d840fd1b0 con 0x7f9d841007a0 2024-02-19T21:11:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 >> v1:172.21.15.163:6790/0 conn(0x7f9d841042e0 legacy=0x7f9d84118480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 >> v1:172.21.15.163:6789/0 conn(0x7f9d840fd9c0 legacy=0x7f9d840fc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.565+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8411cbd0 con 0x7f9d841007a0 2024-02-19T21:11:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.566+0000 7f9d8b093700 1 -- 172.21.15.163:0/3001127475 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d84119c00 con 0x7f9d841007a0 2024-02-19T21:11:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.566+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9d70002390 con 0x7f9d841007a0 2024-02-19T21:11:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.566+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9d70003780 con 0x7f9d841007a0 2024-02-19T21:11:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.566+0000 7f9d8b093700 10 client.48793.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.566+0000 7f9d8b093700 1 -- 172.21.15.163:0/3001127475 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8411a070 con 0x7f9d841007a0 2024-02-19T21:11:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.566+0000 7f9d8b093700 1 librados: init done 2024-02-19T21:11:37.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.567+0000 7f9d8b093700 1 -- 172.21.15.163:0/3001127475 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8405f4e0 con 0x7f9d841007a0 2024-02-19T21:11:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.567+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9d70001880 con 0x7f9d841007a0 2024-02-19T21:11:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.567+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(469..469 src has 43..469) v4 ==== 8092+0+0 (unknown 3884238436 0 0) 0x7f9d70078f70 con 0x7f9d841007a0 2024-02-19T21:11:37.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.567+0000 7f9d81ffb700 10 client.48793.objecter ms_dispatch 0x7f9d84001550 osd_map(469..469 src has 43..469) v4 2024-02-19T21:11:37.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.567+0000 7f9d81ffb700 3 client.48793.objecter handle_osd_map got epochs [469,469] > 0 2024-02-19T21:11:37.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.567+0000 7f9d81ffb700 3 client.48793.objecter handle_osd_map decoding full epoch 469 2024-02-19T21:11:37.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.567+0000 7f9d81ffb700 20 client.48793.objecter dump_active .. 0 homeless 2024-02-19T21:11:37.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.568+0000 7f9d81ffb700 10 client.48793.objecter ms_handle_connect 0x7f9d74063f00 2024-02-19T21:11:37.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.572+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9d7004c3a0 con 0x7f9d841007a0 2024-02-19T21:11:37.615 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:37.640 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:36.301466+0000 mon.a (mon.0) 2659 : cluster [DBG] osdmap e468: 8 total, 8 up, 8 in 2024-02-19T21:11:37.640 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:36.305557+0000 mon.b (mon.1) 620 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:37.640 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:36.306112+0000 mon.a (mon.0) 2660 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:11:37.640 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:36.408459+0000 osd.2 (osd.2) 188 : cluster [DBG] 61.17 scrub starts 2024-02-19T21:11:37.641 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:36.409884+0000 osd.2 (osd.2) 189 : cluster [DBG] 61.17 scrub ok 2024-02-19T21:11:37.642 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:37.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:37.755+0000 7f9d8b093700 1 -- 172.21.15.163:0/3001127475 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f9d840f9200 con 0x7f9d841007a0 2024-02-19T21:11:37.774 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:11:37.841 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:37.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:37.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:37.945 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:11:37.972 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:37.992 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:11:37.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:37.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:11:38.041 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:11:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:11:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:11:38.095 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:11:38.125 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:38.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:38.311+0000 7f9d81ffb700 1 -- 172.21.15.163:0/3001127475 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v470) v1 ==== 101+0+0 (unknown 3132613847 0 0) 0x7f9d7004a810 con 0x7f9d841007a0 2024-02-19T21:11:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:38.314+0000 7f9d8b093700 10 librados: watch_flush enter 2024-02-19T21:11:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:38.314+0000 7f9d8b093700 10 librados: watch_flush exit 2024-02-19T21:11:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:38.314+0000 7f9d8b093700 20 client.48793.objecter shutdown clearing up homeless session... 2024-02-19T21:11:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:38.314+0000 7f9d8b093700 10 client.48793.objecter successfully canceled tick 2024-02-19T21:11:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:38.314+0000 7f9d8b093700 1 -- 172.21.15.163:0/3001127475 >> v1:172.21.15.163:6800/1552514681 conn(0x7f9d74063f00 legacy=0x7f9d740663c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:38.314+0000 7f9d8b093700 1 -- 172.21.15.163:0/3001127475 >> v1:172.21.15.193:6789/0 conn(0x7f9d841007a0 legacy=0x7f9d84114d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:38.314+0000 7f9d8b093700 1 -- 172.21.15.163:0/3001127475 shutdown_connections 2024-02-19T21:11:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:38.314+0000 7f9d8b093700 1 -- 172.21.15.163:0/3001127475 >> 172.21.15.163:0/3001127475 conn(0x7f9d8406cc00 msgr2=0x7f9d840f65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:38.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:38.314+0000 7f9d8b093700 1 -- 172.21.15.163:0/3001127475 shutdown_connections 2024-02-19T21:11:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:38.314+0000 7f9d8b093700 1 -- 172.21.15.163:0/3001127475 wait complete. 2024-02-19T21:11:38.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:38.314+0000 7f9d8b093700 1 librados: shutdown 2024-02-19T21:11:38.316 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:11:38.323 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:11:38.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:38.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:11:38.428 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:11:38.456 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:38.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:36.478689+0000 osd.3 (osd.3) 209 : cluster [DBG] 61.1b scrub starts 2024-02-19T21:11:38.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:36.480573+0000 osd.3 (osd.3) 210 : cluster [DBG] 61.1b scrub ok 2024-02-19T21:11:38.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:37.237695+0000 mgr.y (mgr.14550) 430 : cluster [DBG] pgmap v698: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-19T21:11:38.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:37.304598+0000 mon.a (mon.0) 2661 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:11:38.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:37.304744+0000 mon.a (mon.0) 2662 : cluster [DBG] osdmap e469: 8 total, 8 up, 8 in 2024-02-19T21:11:38.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:37.319538+0000 mon.b (mon.1) 621 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:11:38.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:37.414453+0000 osd.2 (osd.2) 190 : cluster [DBG] 61.11 scrub starts 2024-02-19T21:11:38.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:37.417916+0000 osd.2 (osd.2) 191 : cluster [DBG] 61.11 scrub ok 2024-02-19T21:11:38.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:37.757417+0000 mon.b (mon.1) 622 : audit [INF] from='client.? v1:172.21.15.163:0/3001127475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:11:38.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:37.757707+0000 mon.a (mon.0) 2663 : audit [INF] from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:11:38.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:36.478689+0000 osd.3 (osd.3) 209 : cluster [DBG] 61.1b scrub starts 2024-02-19T21:11:38.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:36.480573+0000 osd.3 (osd.3) 210 : cluster [DBG] 61.1b scrub ok 2024-02-19T21:11:38.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:37.237695+0000 mgr.y (mgr.14550) 430 : cluster [DBG] pgmap v698: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-19T21:11:38.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:37.304598+0000 mon.a (mon.0) 2661 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:11:38.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:37.304744+0000 mon.a (mon.0) 2662 : cluster [DBG] osdmap e469: 8 total, 8 up, 8 in 2024-02-19T21:11:38.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:37.319538+0000 mon.b (mon.1) 621 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:11:38.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:37.414453+0000 osd.2 (osd.2) 190 : cluster [DBG] 61.11 scrub starts 2024-02-19T21:11:38.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:37.417916+0000 osd.2 (osd.2) 191 : cluster [DBG] 61.11 scrub ok 2024-02-19T21:11:38.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:37.757417+0000 mon.b (mon.1) 622 : audit [INF] from='client.? v1:172.21.15.163:0/3001127475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:11:38.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:37.757707+0000 mon.a (mon.0) 2663 : audit [INF] from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:11:38.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:36.478689+0000 osd.3 (osd.3) 209 : cluster [DBG] 61.1b scrub starts 2024-02-19T21:11:38.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:36.480573+0000 osd.3 (osd.3) 210 : cluster [DBG] 61.1b scrub ok 2024-02-19T21:11:38.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:37.237695+0000 mgr.y (mgr.14550) 430 : cluster [DBG] pgmap v698: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-02-19T21:11:38.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:37.304598+0000 mon.a (mon.0) 2661 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:11:38.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:37.304744+0000 mon.a (mon.0) 2662 : cluster [DBG] osdmap e469: 8 total, 8 up, 8 in 2024-02-19T21:11:38.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:37.319538+0000 mon.b (mon.1) 621 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:11:38.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:37.414453+0000 osd.2 (osd.2) 190 : cluster [DBG] 61.11 scrub starts 2024-02-19T21:11:38.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:37.417916+0000 osd.2 (osd.2) 191 : cluster [DBG] 61.11 scrub ok 2024-02-19T21:11:38.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:37.757417+0000 mon.b (mon.1) 622 : audit [INF] from='client.? v1:172.21.15.163:0/3001127475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:11:38.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:37.757707+0000 mon.a (mon.0) 2663 : audit [INF] from='client.48793 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:11:38.969 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:39.234 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:11:39.240 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:39.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:11:39.342 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:11:39.368 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:39.388 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:37.476931+0000 osd.3 (osd.3) 211 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:11:39.388 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:37.477986+0000 osd.3 (osd.3) 212 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:11:39.388 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:37.717684+0000 osd.6 (osd.6) 195 : cluster [DBG] 16.5 scrub starts 2024-02-19T21:11:39.388 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:37.718650+0000 osd.6 (osd.6) 196 : cluster [DBG] 16.5 scrub ok 2024-02-19T21:11:39.388 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:37.964273+0000 osd.5 (osd.5) 198 : cluster [DBG] 61.8 scrub starts 2024-02-19T21:11:39.388 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:37.967758+0000 osd.5 (osd.5) 199 : cluster [DBG] 61.8 scrub ok 2024-02-19T21:11:39.388 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:38.301565+0000 mon.a (mon.0) 2664 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:11:39.389 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:38.312634+0000 mon.a (mon.0) 2665 : audit [INF] from='client.48793 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:11:39.389 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:38.312701+0000 mon.a (mon.0) 2666 : cluster [DBG] osdmap e470: 8 total, 8 up, 8 in 2024-02-19T21:11:39.389 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:38.368619+0000 osd.2 (osd.2) 192 : cluster [DBG] 16.a scrub starts 2024-02-19T21:11:39.389 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:38.369708+0000 osd.2 (osd.2) 193 : cluster [DBG] 16.a scrub ok 2024-02-19T21:11:39.389 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:39.103510+0000 mon.a (mon.0) 2667 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:11:39.399 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:11:39.399 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:11:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:37.476931+0000 osd.3 (osd.3) 211 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:11:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:37.477986+0000 osd.3 (osd.3) 212 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:11:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:37.717684+0000 osd.6 (osd.6) 195 : cluster [DBG] 16.5 scrub starts 2024-02-19T21:11:39.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:37.718650+0000 osd.6 (osd.6) 196 : cluster [DBG] 16.5 scrub ok 2024-02-19T21:11:39.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:37.964273+0000 osd.5 (osd.5) 198 : cluster [DBG] 61.8 scrub starts 2024-02-19T21:11:39.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:37.967758+0000 osd.5 (osd.5) 199 : cluster [DBG] 61.8 scrub ok 2024-02-19T21:11:39.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:38.301565+0000 mon.a (mon.0) 2664 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:11:39.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:38.312634+0000 mon.a (mon.0) 2665 : audit [INF] from='client.48793 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:11:39.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:38.312701+0000 mon.a (mon.0) 2666 : cluster [DBG] osdmap e470: 8 total, 8 up, 8 in 2024-02-19T21:11:39.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:38.368619+0000 osd.2 (osd.2) 192 : cluster [DBG] 16.a scrub starts 2024-02-19T21:11:39.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:38.369708+0000 osd.2 (osd.2) 193 : cluster [DBG] 16.a scrub ok 2024-02-19T21:11:39.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:39.103510+0000 mon.a (mon.0) 2667 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:11:39.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:37.476931+0000 osd.3 (osd.3) 211 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:11:39.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:37.477986+0000 osd.3 (osd.3) 212 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:11:39.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:37.717684+0000 osd.6 (osd.6) 195 : cluster [DBG] 16.5 scrub starts 2024-02-19T21:11:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:37.718650+0000 osd.6 (osd.6) 196 : cluster [DBG] 16.5 scrub ok 2024-02-19T21:11:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:37.964273+0000 osd.5 (osd.5) 198 : cluster [DBG] 61.8 scrub starts 2024-02-19T21:11:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:37.967758+0000 osd.5 (osd.5) 199 : cluster [DBG] 61.8 scrub ok 2024-02-19T21:11:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:38.301565+0000 mon.a (mon.0) 2664 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:11:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:38.312634+0000 mon.a (mon.0) 2665 : audit [INF] from='client.48793 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:11:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:38.312701+0000 mon.a (mon.0) 2666 : cluster [DBG] osdmap e470: 8 total, 8 up, 8 in 2024-02-19T21:11:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:38.368619+0000 osd.2 (osd.2) 192 : cluster [DBG] 16.a scrub starts 2024-02-19T21:11:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:38.369708+0000 osd.2 (osd.2) 193 : cluster [DBG] 16.a scrub ok 2024-02-19T21:11:39.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:39.103510+0000 mon.a (mon.0) 2667 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:11:39.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-19T21:11:40.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.022+0000 7f2b9a3e4700 1 -- 172.21.15.163:0/3368014424 >> v1:172.21.15.163:6789/0 conn(0x7f2b941042b0 legacy=0x7f2b94106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:40.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.022+0000 7f2b9a3e4700 1 -- 172.21.15.163:0/3368014424 shutdown_connections 2024-02-19T21:11:40.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.022+0000 7f2b9a3e4700 1 -- 172.21.15.163:0/3368014424 >> 172.21.15.163:0/3368014424 conn(0x7f2b940f2470 msgr2=0x7f2b940f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:40.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.022+0000 7f2b9a3e4700 1 -- 172.21.15.163:0/3368014424 shutdown_connections 2024-02-19T21:11:40.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.022+0000 7f2b9a3e4700 1 -- 172.21.15.163:0/3368014424 wait complete. 2024-02-19T21:11:40.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.022+0000 7f2b9a3e4700 1 librados: starting msgr at 2024-02-19T21:11:40.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.022+0000 7f2b9a3e4700 1 librados: starting objecter 2024-02-19T21:11:40.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.022+0000 7f2b9a3e4700 1 Processor -- start 2024-02-19T21:11:40.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.022+0000 7f2b9a3e4700 1 -- start start 2024-02-19T21:11:40.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.022+0000 7f2b9a3e4700 1 librados: setting wanted keys 2024-02-19T21:11:40.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.022+0000 7f2b9a3e4700 1 librados: calling monclient init 2024-02-19T21:11:40.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b9a3e4700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2b941173a0 con 0x7f2b940f8920 2024-02-19T21:11:40.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b9a3e4700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2b94118540 con 0x7f2b941042b0 2024-02-19T21:11:40.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b9a3e4700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2b94119740 con 0x7f2b940f94d0 2024-02-19T21:11:40.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b937fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f2b940f94d0 0x7f2b940ffd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:60742/0 (socket says 172.21.15.163:60742) 2024-02-19T21:11:40.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b937fe700 1 -- 172.21.15.163:0/3682509244 learned_addr learned my addr 172.21.15.163:0/3682509244 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:40.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 10 client.?.objecter ms_handle_connect 0x7f2b940f8920 2024-02-19T21:11:40.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:40.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 10 client.?.objecter ms_handle_connect 0x7f2b940f94d0 2024-02-19T21:11:40.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:40.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 10 client.?.objecter ms_handle_connect 0x7f2b941042b0 2024-02-19T21:11:40.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:40.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965287164 0 0) 0x7f2b941173a0 con 0x7f2b940f8920 2024-02-19T21:11:40.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b7c0033f0 con 0x7f2b940f8920 2024-02-19T21:11:40.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264803228 0 0) 0x7f2b94119740 con 0x7f2b940f94d0 2024-02-19T21:11:40.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b941173a0 con 0x7f2b940f94d0 2024-02-19T21:11:40.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400359389 0 0) 0x7f2b94118540 con 0x7f2b941042b0 2024-02-19T21:11:40.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b94119740 con 0x7f2b941042b0 2024-02-19T21:11:40.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1291507985 0 0) 0x7f2b7c0033f0 con 0x7f2b940f8920 2024-02-19T21:11:40.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b94118540 con 0x7f2b940f8920 2024-02-19T21:11:40.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2b840018c0 con 0x7f2b940f8920 2024-02-19T21:11:40.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2340310637 0 0) 0x7f2b941173a0 con 0x7f2b940f94d0 2024-02-19T21:11:40.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b7c0033f0 con 0x7f2b940f94d0 2024-02-19T21:11:40.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2b88001900 con 0x7f2b940f94d0 2024-02-19T21:11:40.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2578644035 0 0) 0x7f2b94119740 con 0x7f2b941042b0 2024-02-19T21:11:40.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b941173a0 con 0x7f2b941042b0 2024-02-19T21:11:40.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2b8c0023d0 con 0x7f2b941042b0 2024-02-19T21:11:40.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.023+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 571941244 0 0) 0x7f2b94118540 con 0x7f2b940f8920 2024-02-19T21:11:40.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.024+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 >> v1:172.21.15.163:6790/0 conn(0x7f2b940f94d0 legacy=0x7f2b940ffd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:40.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.024+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 >> v1:172.21.15.193:6789/0 conn(0x7f2b941042b0 legacy=0x7f2b94115a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:40.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.024+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9411a940 con 0x7f2b940f8920 2024-02-19T21:11:40.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.024+0000 7f2b9a3e4700 1 -- 172.21.15.163:0/3682509244 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b94119930 con 0x7f2b940f8920 2024-02-19T21:11:40.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.024+0000 7f2b9a3e4700 10 client.23262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:40.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.024+0000 7f2b9a3e4700 1 -- 172.21.15.163:0/3682509244 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b94119da0 con 0x7f2b940f8920 2024-02-19T21:11:40.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.024+0000 7f2b9a3e4700 1 librados: init done 2024-02-19T21:11:40.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.024+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f2b84001f80 con 0x7f2b940f8920 2024-02-19T21:11:40.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.024+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2b84003280 con 0x7f2b940f8920 2024-02-19T21:11:40.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.025+0000 7f2b9a3e4700 1 -- 172.21.15.163:0/3682509244 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b80004f20 con 0x7f2b940f8920 2024-02-19T21:11:40.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.025+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f2b840026c0 con 0x7f2b940f8920 2024-02-19T21:11:40.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.026+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(471..471 src has 43..471) v4 ==== 8092+0+0 (unknown 2298721257 0 0) 0x7f2b840764e0 con 0x7f2b940f8920 2024-02-19T21:11:40.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.026+0000 7f2b917fa700 10 client.23262.objecter ms_dispatch 0x7f2b94001550 osd_map(471..471 src has 43..471) v4 2024-02-19T21:11:40.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.026+0000 7f2b917fa700 3 client.23262.objecter handle_osd_map got epochs [471,471] > 0 2024-02-19T21:11:40.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.026+0000 7f2b917fa700 3 client.23262.objecter handle_osd_map decoding full epoch 471 2024-02-19T21:11:40.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.026+0000 7f2b917fa700 20 client.23262.objecter dump_active .. 0 homeless 2024-02-19T21:11:40.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.026+0000 7f2b917fa700 10 client.23262.objecter ms_handle_connect 0x7f2b7c063cb0 2024-02-19T21:11:40.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.030+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f2b84049910 con 0x7f2b940f8920 2024-02-19T21:11:40.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.196+0000 7f2b9a3e4700 1 -- 172.21.15.163:0/3682509244 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b80000b90 con 0x7f2b7c063cb0 2024-02-19T21:11:40.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.212+0000 7f2b917fa700 1 -- 172.21.15.163:0/3682509244 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+577122 (unknown 2965378022 0 2056972863) 0x7f2b80000b90 con 0x7f2b7c063cb0 2024-02-19T21:11:40.218 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:11:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.215+0000 7f2b9a3e4700 10 librados: watch_flush enter 2024-02-19T21:11:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.215+0000 7f2b9a3e4700 10 librados: watch_flush exit 2024-02-19T21:11:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.215+0000 7f2b9a3e4700 20 client.23262.objecter shutdown clearing up homeless session... 2024-02-19T21:11:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.215+0000 7f2b9a3e4700 10 client.23262.objecter successfully canceled tick 2024-02-19T21:11:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.215+0000 7f2b9a3e4700 1 -- 172.21.15.163:0/3682509244 >> v1:172.21.15.163:6800/1552514681 conn(0x7f2b7c063cb0 legacy=0x7f2b7c066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.215+0000 7f2b9a3e4700 1 -- 172.21.15.163:0/3682509244 >> v1:172.21.15.163:6789/0 conn(0x7f2b940f8920 legacy=0x7f2b940ff660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.216+0000 7f2b9a3e4700 1 -- 172.21.15.163:0/3682509244 shutdown_connections 2024-02-19T21:11:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.216+0000 7f2b9a3e4700 1 -- 172.21.15.163:0/3682509244 >> 172.21.15.163:0/3682509244 conn(0x7f2b940f2470 msgr2=0x7f2b94103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.216+0000 7f2b9a3e4700 1 -- 172.21.15.163:0/3682509244 shutdown_connections 2024-02-19T21:11:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.216+0000 7f2b9a3e4700 1 -- 172.21.15.163:0/3682509244 wait complete. 2024-02-19T21:11:40.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:40.216+0000 7f2b9a3e4700 1 librados: shutdown 2024-02-19T21:11:40.225 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:11:40.242 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:11:40.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:38.563316+0000 osd.7 (osd.7) 217 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:11:40.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:38.566205+0000 osd.7 (osd.7) 218 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:11:40.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:38.763603+0000 osd.6 (osd.6) 197 : cluster [DBG] 16.d scrub starts 2024-02-19T21:11:40.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:38.764654+0000 osd.6 (osd.6) 198 : cluster [DBG] 16.d scrub ok 2024-02-19T21:11:40.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:38.944806+0000 osd.1 (osd.1) 229 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:11:40.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:38.945634+0000 osd.1 (osd.1) 230 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:11:40.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:39.238048+0000 mgr.y (mgr.14550) 431 : cluster [DBG] pgmap v701: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:11:40.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:39.311173+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e471: 8 total, 8 up, 8 in 2024-02-19T21:11:40.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:39.323184+0000 mon.b (mon.1) 623 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1b"}]: dispatch 2024-02-19T21:11:40.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:39.323348+0000 mgr.y (mgr.14550) 432 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1b"}]: dispatch 2024-02-19T21:11:40.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:38.563316+0000 osd.7 (osd.7) 217 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:11:40.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:38.566205+0000 osd.7 (osd.7) 218 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:11:40.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:38.763603+0000 osd.6 (osd.6) 197 : cluster [DBG] 16.d scrub starts 2024-02-19T21:11:40.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:38.764654+0000 osd.6 (osd.6) 198 : cluster [DBG] 16.d scrub ok 2024-02-19T21:11:40.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:38.944806+0000 osd.1 (osd.1) 229 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:11:40.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:38.945634+0000 osd.1 (osd.1) 230 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:11:40.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:39.238048+0000 mgr.y (mgr.14550) 431 : cluster [DBG] pgmap v701: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:11:40.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:39.311173+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e471: 8 total, 8 up, 8 in 2024-02-19T21:11:40.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:39.323184+0000 mon.b (mon.1) 623 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1b"}]: dispatch 2024-02-19T21:11:40.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:39.323348+0000 mgr.y (mgr.14550) 432 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1b"}]: dispatch 2024-02-19T21:11:40.570 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:11:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:11:40.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:38.563316+0000 osd.7 (osd.7) 217 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:11:40.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:38.566205+0000 osd.7 (osd.7) 218 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:11:40.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:38.763603+0000 osd.6 (osd.6) 197 : cluster [DBG] 16.d scrub starts 2024-02-19T21:11:40.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:38.764654+0000 osd.6 (osd.6) 198 : cluster [DBG] 16.d scrub ok 2024-02-19T21:11:40.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:38.944806+0000 osd.1 (osd.1) 229 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:11:40.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:38.945634+0000 osd.1 (osd.1) 230 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:11:40.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:39.238048+0000 mgr.y (mgr.14550) 431 : cluster [DBG] pgmap v701: 228 pgs: 32 unknown, 196 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:11:40.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:39.311173+0000 mon.a (mon.0) 2668 : cluster [DBG] osdmap e471: 8 total, 8 up, 8 in 2024-02-19T21:11:40.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:39.323184+0000 mon.b (mon.1) 623 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1b"}]: dispatch 2024-02-19T21:11:40.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:39.323348+0000 mgr.y (mgr.14550) 432 : audit [DBG] from='mon.? v1:172.21.15.193:0/2021529307' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "219.1b"}]: dispatch 2024-02-19T21:11:41.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:40.198148+0000 mgr.y (mgr.14550) 433 : audit [DBG] from='client.23262 v1:172.21.15.163:0/3682509244' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:11:41.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:40.480985+0000 mgr.y (mgr.14550) 434 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:41.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:40.198148+0000 mgr.y (mgr.14550) 433 : audit [DBG] from='client.23262 v1:172.21.15.163:0/3682509244' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:11:41.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:40.480985+0000 mgr.y (mgr.14550) 434 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:41.508 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":702,"stamp":"2024-02-19T21:11:39.309196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4007399,"num_objects":364,"num_object_clones":9,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":364,"num_whiteouts":0,"num_read":1848,"num_read_kb":1499,"num_write":1346,"num_write_kb":4571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1004,"num_bytes_recovered":8733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2765,"ondisk_log_size":2765,"up":588,"acting":588,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":654,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2492592,"kb_used_data":69696,"kb_used_omap":0,"kb_used_meta":2422784,"kb_avail":747468624,"statfs":{"total":767960285184,"available":765407870976,"internally_reserved":0,"allocated":71368704,"data_stored":46279697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2480930816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018346"},"pg_stats":[{"pgid":"219.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"219.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:11:36.289933+0000","last_change":"2024-02-19T21:11:36.289933+0000","last_active":"2024-02-19T21:11:36.289933+0000","last_peered":"2024-02-19T21:11:36.289933+0000","last_clean":"2024-02-19T21:11:36.289933+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:11:36.289933+0000","last_undegraded":"2024-02-19T21:11:36.289933+0000","last_fullsized":"2024-02-19T21:11:36.289933+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:36.289933+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.289933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"61.2","version":"452'102","reported_seq":271,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411712+0000","last_change":"2024-02-19T21:10:31.528468+0000","last_active":"2024-02-19T21:11:33.411712+0000","last_peered":"2024-02-19T21:11:33.411712+0000","last_clean":"2024-02-19T21:11:33.411712+0000","last_became_active":"2024-02-19T21:08:04.247600+0000","last_became_peered":"2024-02-19T21:08:04.247600+0000","last_unstale":"2024-02-19T21:11:33.411712+0000","last_undegraded":"2024-02-19T21:11:33.411712+0000","last_fullsized":"2024-02-19T21:11:33.411712+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'85","last_scrub_stamp":"2024-02-19T21:08:55.882988+0000","last_deep_scrub":"355'85","last_deep_scrub_stamp":"2024-02-19T21:08:55.882988+0000","last_clean_scrub_stamp":"2024-02-19T21:08:55.882988+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.3","version":"338'71","reported_seq":488,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.411635+0000","last_change":"2024-02-19T21:11:06.901648+0000","last_active":"2024-02-19T21:11:32.411635+0000","last_peered":"2024-02-19T21:11:32.411635+0000","last_clean":"2024-02-19T21:11:32.411635+0000","last_became_active":"2024-02-19T21:04:54.641884+0000","last_became_peered":"2024-02-19T21:04:54.641884+0000","last_unstale":"2024-02-19T21:11:32.411635+0000","last_undegraded":"2024-02-19T21:11:32.411635+0000","last_fullsized":"2024-02-19T21:11:32.411635+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'71","last_scrub_stamp":"2024-02-19T21:11:06.901567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:06.901567+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000068,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":23,"num_write_kb":3929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.0","version":"257'35","reported_seq":435,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.410557+0000","last_change":"2024-02-19T21:11:31.931216+0000","last_active":"2024-02-19T21:11:32.410557+0000","last_peered":"2024-02-19T21:11:32.410557+0000","last_clean":"2024-02-19T21:11:32.410557+0000","last_became_active":"2024-02-19T21:06:40.245157+0000","last_became_peered":"2024-02-19T21:06:40.245157+0000","last_unstale":"2024-02-19T21:11:32.410557+0000","last_undegraded":"2024-02-19T21:11:32.410557+0000","last_fullsized":"2024-02-19T21:11:32.410557+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'35","last_scrub_stamp":"2024-02-19T21:11:31.931145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.931145+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1","version":"338'69","reported_seq":593,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.411012+0000","last_change":"2024-02-19T21:10:54.672773+0000","last_active":"2024-02-19T21:11:32.411012+0000","last_peered":"2024-02-19T21:11:32.411012+0000","last_clean":"2024-02-19T21:11:32.411012+0000","last_became_active":"2024-02-19T21:10:41.311298+0000","last_became_peered":"2024-02-19T21:10:41.311298+0000","last_unstale":"2024-02-19T21:11:32.411012+0000","last_undegraded":"2024-02-19T21:11:32.411012+0000","last_fullsized":"2024-02-19T21:11:32.411012+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-19T21:10:54.672705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.672705+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.6","version":"202'34","reported_seq":451,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411085+0000","last_change":"2024-02-19T21:10:31.536500+0000","last_active":"2024-02-19T21:11:33.411085+0000","last_peered":"2024-02-19T21:11:33.411085+0000","last_clean":"2024-02-19T21:11:33.411085+0000","last_became_active":"2024-02-19T21:04:54.658542+0000","last_became_peered":"2024-02-19T21:04:54.658542+0000","last_unstale":"2024-02-19T21:11:33.411085+0000","last_undegraded":"2024-02-19T21:11:33.411085+0000","last_fullsized":"2024-02-19T21:11:33.411085+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'34","last_scrub_stamp":"2024-02-19T21:08:58.036955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:58.036955+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.7","version":"396'20","reported_seq":451,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.419343+0000","last_change":"2024-02-19T21:11:05.030418+0000","last_active":"2024-02-19T21:11:33.419343+0000","last_peered":"2024-02-19T21:11:33.419343+0000","last_clean":"2024-02-19T21:11:33.419343+0000","last_became_active":"2024-02-19T21:04:54.598061+0000","last_became_peered":"2024-02-19T21:04:54.598061+0000","last_unstale":"2024-02-19T21:11:33.419343+0000","last_undegraded":"2024-02-19T21:11:33.419343+0000","last_fullsized":"2024-02-19T21:11:33.419343+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'20","last_scrub_stamp":"2024-02-19T21:11:05.030354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.030354+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.4","version":"403'70","reported_seq":593,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411746+0000","last_change":"2024-02-19T21:11:01.162809+0000","last_active":"2024-02-19T21:11:33.411746+0000","last_peered":"2024-02-19T21:11:33.411746+0000","last_clean":"2024-02-19T21:11:33.411746+0000","last_became_active":"2024-02-19T21:10:41.314275+0000","last_became_peered":"2024-02-19T21:10:41.314275+0000","last_unstale":"2024-02-19T21:11:33.411746+0000","last_undegraded":"2024-02-19T21:11:33.411746+0000","last_fullsized":"2024-02-19T21:11:33.411746+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'70","last_scrub_stamp":"2024-02-19T21:11:01.162737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.162737+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":96,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.5","version":"338'121","reported_seq":521,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.412300+0000","last_change":"2024-02-19T21:10:31.535793+0000","last_active":"2024-02-19T21:11:32.412300+0000","last_peered":"2024-02-19T21:11:32.412300+0000","last_clean":"2024-02-19T21:11:32.412300+0000","last_became_active":"2024-02-19T21:04:54.646842+0000","last_became_peered":"2024-02-19T21:04:54.646842+0000","last_unstale":"2024-02-19T21:11:32.412300+0000","last_undegraded":"2024-02-19T21:11:32.412300+0000","last_fullsized":"2024-02-19T21:11:32.412300+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'121","last_scrub_stamp":"2024-02-19T21:08:49.752418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.752418+0000","log_size":121,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":37,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.a","version":"236'22","reported_seq":269,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-19T21:11:34.523734+0000","last_change":"2024-02-19T21:11:34.523734+0000","last_active":"2024-02-19T21:11:34.523734+0000","last_peered":"2024-02-19T21:11:34.523734+0000","last_clean":"2024-02-19T21:11:34.523734+0000","last_became_active":"2024-02-19T21:10:41.305485+0000","last_became_peered":"2024-02-19T21:10:41.305485+0000","last_unstale":"2024-02-19T21:11:34.523734+0000","last_undegraded":"2024-02-19T21:11:34.523734+0000","last_fullsized":"2024-02-19T21:11:34.523734+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-02-19T21:11:34.523666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:34.523666+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":96,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.b","version":"278'95","reported_seq":507,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.411721+0000","last_change":"2024-02-19T21:11:03.821979+0000","last_active":"2024-02-19T21:11:32.411721+0000","last_peered":"2024-02-19T21:11:32.411721+0000","last_clean":"2024-02-19T21:11:32.411721+0000","last_became_active":"2024-02-19T21:04:54.640024+0000","last_became_peered":"2024-02-19T21:04:54.640024+0000","last_unstale":"2024-02-19T21:11:32.411721+0000","last_undegraded":"2024-02-19T21:11:32.411721+0000","last_fullsized":"2024-02-19T21:11:32.411721+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'95","last_scrub_stamp":"2024-02-19T21:11:03.821869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.821869+0000","log_size":95,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.8","version":"399'39","reported_seq":519,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418339+0000","last_change":"2024-02-19T21:10:31.526940+0000","last_active":"2024-02-19T21:11:33.418339+0000","last_peered":"2024-02-19T21:11:33.418339+0000","last_clean":"2024-02-19T21:11:33.418339+0000","last_became_active":"2024-02-19T21:08:04.266021+0000","last_became_peered":"2024-02-19T21:08:04.266021+0000","last_unstale":"2024-02-19T21:11:33.418339+0000","last_undegraded":"2024-02-19T21:11:33.418339+0000","last_fullsized":"2024-02-19T21:11:33.418339+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'35","last_scrub_stamp":"2024-02-19T21:08:01.494484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:01.494484+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":50,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.9","version":"439'77","reported_seq":497,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.410946+0000","last_change":"2024-02-19T21:10:55.678589+0000","last_active":"2024-02-19T21:11:32.410946+0000","last_peered":"2024-02-19T21:11:32.410946+0000","last_clean":"2024-02-19T21:11:32.410946+0000","last_became_active":"2024-02-19T21:04:54.671646+0000","last_became_peered":"2024-02-19T21:04:54.671646+0000","last_unstale":"2024-02-19T21:11:32.410946+0000","last_undegraded":"2024-02-19T21:11:32.410946+0000","last_fullsized":"2024-02-19T21:11:32.410946+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'77","last_scrub_stamp":"2024-02-19T21:10:55.678451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:55.678451+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":71,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":24,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.e","version":"393'40","reported_seq":537,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.416807+0000","last_change":"2024-02-19T21:10:41.425839+0000","last_active":"2024-02-19T21:11:32.416807+0000","last_peered":"2024-02-19T21:11:32.416807+0000","last_clean":"2024-02-19T21:11:32.416807+0000","last_became_active":"2024-02-19T21:10:41.298057+0000","last_became_peered":"2024-02-19T21:10:41.298057+0000","last_unstale":"2024-02-19T21:11:32.416807+0000","last_undegraded":"2024-02-19T21:11:32.416807+0000","last_fullsized":"2024-02-19T21:11:32.416807+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'40","last_scrub_stamp":"2024-02-19T21:10:18.512566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:18.512566+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":81,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"61.f","version":"399'37","reported_seq":527,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.417205+0000","last_change":"2024-02-19T21:10:41.464188+0000","last_active":"2024-02-19T21:11:33.417205+0000","last_peered":"2024-02-19T21:11:33.417205+0000","last_clean":"2024-02-19T21:11:33.417205+0000","last_became_active":"2024-02-19T21:10:41.315097+0000","last_became_peered":"2024-02-19T21:10:41.315097+0000","last_unstale":"2024-02-19T21:11:33.417205+0000","last_undegraded":"2024-02-19T21:11:33.417205+0000","last_fullsized":"2024-02-19T21:11:33.417205+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'37","last_scrub_stamp":"2024-02-19T21:10:13.757798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:13.757798+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.c","version":"249'73","reported_seq":703,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.419315+0000","last_change":"2024-02-19T21:10:46.664562+0000","last_active":"2024-02-19T21:11:33.419315+0000","last_peered":"2024-02-19T21:11:33.419315+0000","last_clean":"2024-02-19T21:11:33.419315+0000","last_became_active":"2024-02-19T21:10:41.321453+0000","last_became_peered":"2024-02-19T21:10:41.321453+0000","last_unstale":"2024-02-19T21:11:33.419315+0000","last_undegraded":"2024-02-19T21:11:33.419315+0000","last_fullsized":"2024-02-19T21:11:33.419315+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'73","last_scrub_stamp":"2024-02-19T21:08:49.613462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.613462+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.d","version":"442'44","reported_seq":466,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.419101+0000","last_change":"2024-02-19T21:10:31.528317+0000","last_active":"2024-02-19T21:11:33.419101+0000","last_peered":"2024-02-19T21:11:33.419101+0000","last_clean":"2024-02-19T21:11:33.419101+0000","last_became_active":"2024-02-19T21:04:54.644749+0000","last_became_peered":"2024-02-19T21:04:54.644749+0000","last_unstale":"2024-02-19T21:11:33.419101+0000","last_undegraded":"2024-02-19T21:11:33.419101+0000","last_fullsized":"2024-02-19T21:11:33.419101+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'34","last_scrub_stamp":"2024-02-19T21:08:12.445646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.445646+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.12","version":"390'47","reported_seq":544,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.409683+0000","last_change":"2024-02-19T21:10:31.534697+0000","last_active":"2024-02-19T21:11:33.409683+0000","last_peered":"2024-02-19T21:11:33.409683+0000","last_clean":"2024-02-19T21:11:33.409683+0000","last_became_active":"2024-02-19T21:08:04.263085+0000","last_became_peered":"2024-02-19T21:08:04.263085+0000","last_unstale":"2024-02-19T21:11:33.409683+0000","last_undegraded":"2024-02-19T21:11:33.409683+0000","last_fullsized":"2024-02-19T21:11:33.409683+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'45","last_scrub_stamp":"2024-02-19T21:08:49.965991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.965991+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.13","version":"370'71","reported_seq":606,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.415854+0000","last_change":"2024-02-19T21:10:57.789184+0000","last_active":"2024-02-19T21:11:33.415854+0000","last_peered":"2024-02-19T21:11:33.415854+0000","last_clean":"2024-02-19T21:11:33.415854+0000","last_became_active":"2024-02-19T21:10:41.317677+0000","last_became_peered":"2024-02-19T21:10:41.317677+0000","last_unstale":"2024-02-19T21:11:33.415854+0000","last_undegraded":"2024-02-19T21:11:33.415854+0000","last_fullsized":"2024-02-19T21:11:33.415854+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'71","last_scrub_stamp":"2024-02-19T21:10:57.789103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.789103+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":23,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.10","version":"466'64","reported_seq":491,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-19T21:11:34.324205+0000","last_change":"2024-02-19T21:11:04.750002+0000","last_active":"2024-02-19T21:11:34.324205+0000","last_peered":"2024-02-19T21:11:34.324205+0000","last_clean":"2024-02-19T21:11:34.324205+0000","last_became_active":"2024-02-19T21:04:54.626393+0000","last_became_peered":"2024-02-19T21:04:54.626393+0000","last_unstale":"2024-02-19T21:11:34.324205+0000","last_undegraded":"2024-02-19T21:11:34.324205+0000","last_fullsized":"2024-02-19T21:11:34.324205+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'49","last_scrub_stamp":"2024-02-19T21:11:04.749932+0000","last_deep_scrub":"452'49","last_deep_scrub_stamp":"2024-02-19T21:11:04.749932+0000","last_clean_scrub_stamp":"2024-02-19T21:11:04.749932+0000","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":24,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.11","version":"338'35","reported_seq":457,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.409209+0000","last_change":"2024-02-19T21:10:31.533725+0000","last_active":"2024-02-19T21:11:33.409209+0000","last_peered":"2024-02-19T21:11:33.409209+0000","last_clean":"2024-02-19T21:11:33.409209+0000","last_became_active":"2024-02-19T21:04:54.659096+0000","last_became_peered":"2024-02-19T21:04:54.659096+0000","last_unstale":"2024-02-19T21:11:33.409209+0000","last_undegraded":"2024-02-19T21:11:33.409209+0000","last_fullsized":"2024-02-19T21:11:33.409209+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'35","last_scrub_stamp":"2024-02-19T21:08:41.145092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:41.145092+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.16","version":"419'47","reported_seq":73,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412701+0000","last_change":"2024-02-19T21:10:54.364931+0000","last_active":"2024-02-19T21:11:33.412701+0000","last_peered":"2024-02-19T21:11:33.412701+0000","last_clean":"2024-02-19T21:11:33.412701+0000","last_became_active":"2024-02-19T21:10:41.313393+0000","last_became_peered":"2024-02-19T21:10:41.313393+0000","last_unstale":"2024-02-19T21:11:33.412701+0000","last_undegraded":"2024-02-19T21:11:33.412701+0000","last_fullsized":"2024-02-19T21:11:33.412701+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'47","last_scrub_stamp":"2024-02-19T21:10:54.364824+0000","last_deep_scrub":"237'24","last_deep_scrub_stamp":"2024-02-19T21:06:57.889852+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.364824+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":40,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"61.17","version":"286'24","reported_seq":506,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412753+0000","last_change":"2024-02-19T21:10:41.753888+0000","last_active":"2024-02-19T21:11:33.412753+0000","last_peered":"2024-02-19T21:11:33.412753+0000","last_clean":"2024-02-19T21:11:33.412753+0000","last_became_active":"2024-02-19T21:10:41.314188+0000","last_became_peered":"2024-02-19T21:10:41.314188+0000","last_unstale":"2024-02-19T21:11:33.412753+0000","last_undegraded":"2024-02-19T21:11:33.412753+0000","last_fullsized":"2024-02-19T21:11:33.412753+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'24","last_scrub_stamp":"2024-02-19T21:08:48.023079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:48.023079+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.14","version":"338'52","reported_seq":451,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411644+0000","last_change":"2024-02-19T21:10:31.536240+0000","last_active":"2024-02-19T21:11:33.411644+0000","last_peered":"2024-02-19T21:11:33.411644+0000","last_clean":"2024-02-19T21:11:33.411644+0000","last_became_active":"2024-02-19T21:06:40.227840+0000","last_became_peered":"2024-02-19T21:06:40.227840+0000","last_unstale":"2024-02-19T21:11:33.411644+0000","last_undegraded":"2024-02-19T21:11:33.411644+0000","last_fullsized":"2024-02-19T21:11:33.411644+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'52","last_scrub_stamp":"2024-02-19T21:09:28.997593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.997593+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.15","version":"370'52","reported_seq":642,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.416698+0000","last_change":"2024-02-19T21:10:43.140651+0000","last_active":"2024-02-19T21:11:33.416698+0000","last_peered":"2024-02-19T21:11:33.416698+0000","last_clean":"2024-02-19T21:11:33.416698+0000","last_became_active":"2024-02-19T21:10:41.316332+0000","last_became_peered":"2024-02-19T21:10:41.316332+0000","last_unstale":"2024-02-19T21:11:33.416698+0000","last_undegraded":"2024-02-19T21:11:33.416698+0000","last_fullsized":"2024-02-19T21:11:33.416698+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'52","last_scrub_stamp":"2024-02-19T21:10:22.847438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:22.847438+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1a","version":"448'98","reported_seq":647,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.415638+0000","last_change":"2024-02-19T21:10:55.797877+0000","last_active":"2024-02-19T21:11:33.415638+0000","last_peered":"2024-02-19T21:11:33.415638+0000","last_clean":"2024-02-19T21:11:33.415638+0000","last_became_active":"2024-02-19T21:08:04.265588+0000","last_became_peered":"2024-02-19T21:08:04.265588+0000","last_unstale":"2024-02-19T21:11:33.415638+0000","last_undegraded":"2024-02-19T21:11:33.415638+0000","last_fullsized":"2024-02-19T21:11:33.415638+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'98","last_scrub_stamp":"2024-02-19T21:10:55.797794+0000","last_deep_scrub":"448'98","last_deep_scrub_stamp":"2024-02-19T21:10:55.797794+0000","last_clean_scrub_stamp":"2024-02-19T21:10:55.797794+0000","log_size":98,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1b","version":"338'52","reported_seq":225,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411875+0000","last_change":"2024-02-19T21:10:31.528748+0000","last_active":"2024-02-19T21:11:33.411875+0000","last_peered":"2024-02-19T21:11:33.411875+0000","last_clean":"2024-02-19T21:11:33.411875+0000","last_became_active":"2024-02-19T21:08:04.262932+0000","last_became_peered":"2024-02-19T21:08:04.262932+0000","last_unstale":"2024-02-19T21:11:33.411875+0000","last_undegraded":"2024-02-19T21:11:33.411875+0000","last_fullsized":"2024-02-19T21:11:33.411875+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'51","last_scrub_stamp":"2024-02-19T21:08:00.485092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:00.485092+0000","log_size":52,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.18","version":"214'102","reported_seq":568,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.410459+0000","last_change":"2024-02-19T21:10:45.292818+0000","last_active":"2024-02-19T21:11:33.410459+0000","last_peered":"2024-02-19T21:11:33.410459+0000","last_clean":"2024-02-19T21:11:33.410459+0000","last_became_active":"2024-02-19T21:10:45.292216+0000","last_became_peered":"2024-02-19T21:10:45.292216+0000","last_unstale":"2024-02-19T21:11:33.410459+0000","last_undegraded":"2024-02-19T21:11:33.410459+0000","last_fullsized":"2024-02-19T21:11:33.410459+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-19T21:08:50.838363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:50.838363+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":198,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.19","version":"338'69","reported_seq":593,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418097+0000","last_change":"2024-02-19T21:10:43.469699+0000","last_active":"2024-02-19T21:11:33.418097+0000","last_peered":"2024-02-19T21:11:33.418097+0000","last_clean":"2024-02-19T21:11:33.418097+0000","last_became_active":"2024-02-19T21:10:41.296786+0000","last_became_peered":"2024-02-19T21:10:41.296786+0000","last_unstale":"2024-02-19T21:11:33.418097+0000","last_undegraded":"2024-02-19T21:11:33.418097+0000","last_fullsized":"2024-02-19T21:11:33.418097+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-19T21:08:39.466543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.466543+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.1e","version":"459'62","reported_seq":490,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.416964+0000","last_change":"2024-02-19T21:10:31.710212+0000","last_active":"2024-02-19T21:11:33.416964+0000","last_peered":"2024-02-19T21:11:33.416964+0000","last_clean":"2024-02-19T21:11:33.416964+0000","last_became_active":"2024-02-19T21:04:54.629433+0000","last_became_peered":"2024-02-19T21:04:54.629433+0000","last_unstale":"2024-02-19T21:11:33.416964+0000","last_undegraded":"2024-02-19T21:11:33.416964+0000","last_fullsized":"2024-02-19T21:11:33.416964+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'50","last_scrub_stamp":"2024-02-19T21:10:16.846927+0000","last_deep_scrub":"241'49","last_deep_scrub_stamp":"2024-02-19T21:07:01.787521+0000","last_clean_scrub_stamp":"2024-02-19T21:10:16.846927+0000","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78,"num_objects":9,"num_object_clones":2,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1f","version":"249'44","reported_seq":466,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412756+0000","last_change":"2024-02-19T21:10:31.536663+0000","last_active":"2024-02-19T21:11:33.412756+0000","last_peered":"2024-02-19T21:11:33.412756+0000","last_clean":"2024-02-19T21:11:33.412756+0000","last_became_active":"2024-02-19T21:04:54.671969+0000","last_became_peered":"2024-02-19T21:04:54.671969+0000","last_unstale":"2024-02-19T21:11:33.412756+0000","last_undegraded":"2024-02-19T21:11:33.412756+0000","last_fullsized":"2024-02-19T21:11:33.412756+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'44","last_scrub_stamp":"2024-02-19T21:09:25.988978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:09:25.988978+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":14,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.1c","version":"429'153","reported_seq":765,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418789+0000","last_change":"2024-02-19T21:10:50.292413+0000","last_active":"2024-02-19T21:11:33.418789+0000","last_peered":"2024-02-19T21:11:33.418789+0000","last_clean":"2024-02-19T21:11:33.418789+0000","last_became_active":"2024-02-19T21:10:50.291607+0000","last_became_peered":"2024-02-19T21:10:50.291607+0000","last_unstale":"2024-02-19T21:11:33.418789+0000","last_undegraded":"2024-02-19T21:11:33.418789+0000","last_fullsized":"2024-02-19T21:11:33.418789+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'136","last_scrub_stamp":"2024-02-19T21:08:37.523303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:08:37.523303+0000","log_size":153,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":46,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.1d","version":"370'36","reported_seq":461,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.941532+0000","last_change":"2024-02-19T21:11:32.941492+0000","last_active":"2024-02-19T21:11:32.941532+0000","last_peered":"2024-02-19T21:11:32.941532+0000","last_clean":"2024-02-19T21:11:32.941532+0000","last_became_active":"2024-02-19T21:04:54.655665+0000","last_became_peered":"2024-02-19T21:04:54.655665+0000","last_unstale":"2024-02-19T21:11:32.941532+0000","last_undegraded":"2024-02-19T21:11:32.941532+0000","last_fullsized":"2024-02-19T21:11:32.941532+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'36","last_scrub_stamp":"2024-02-19T21:11:32.941416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:32.941416+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"434'11","reported_seq":589,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-19T21:11:34.294789+0000","last_change":"2024-02-19T21:10:43.270608+0000","last_active":"2024-02-19T21:11:34.294789+0000","last_peered":"2024-02-19T21:11:34.294789+0000","last_clean":"2024-02-19T21:11:34.294789+0000","last_became_active":"2024-02-19T21:10:41.316587+0000","last_became_peered":"2024-02-19T21:10:41.316587+0000","last_unstale":"2024-02-19T21:11:34.294789+0000","last_undegraded":"2024-02-19T21:11:34.294789+0000","last_fullsized":"2024-02-19T21:11:34.294789+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'10","last_scrub_stamp":"2024-02-19T21:09:42.967771+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:09:42.967771+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"459'27","reported_seq":474,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418449+0000","last_change":"2024-02-19T21:08:51.616197+0000","last_active":"2024-02-19T21:11:33.418449+0000","last_peered":"2024-02-19T21:11:33.418449+0000","last_clean":"2024-02-19T21:11:33.418449+0000","last_became_active":"2024-02-19T21:01:18.293825+0000","last_became_peered":"2024-02-19T21:01:18.293825+0000","last_unstale":"2024-02-19T21:11:33.418449+0000","last_undegraded":"2024-02-19T21:11:33.418449+0000","last_fullsized":"2024-02-19T21:11:33.418449+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:08:51.616123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:51.616123+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":461,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411212+0000","last_change":"2024-02-19T21:10:40.300037+0000","last_active":"2024-02-19T21:11:33.411212+0000","last_peered":"2024-02-19T21:11:33.411212+0000","last_clean":"2024-02-19T21:11:33.411212+0000","last_became_active":"2024-02-19T21:10:40.299425+0000","last_became_peered":"2024-02-19T21:10:40.299425+0000","last_unstale":"2024-02-19T21:11:33.411212+0000","last_undegraded":"2024-02-19T21:11:33.411212+0000","last_fullsized":"2024-02-19T21:11:33.411212+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:44.922303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_clean_scrub_stamp":"2024-02-19T21:09:44.922303+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":424,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418521+0000","last_change":"2024-02-19T21:09:27.878844+0000","last_active":"2024-02-19T21:11:33.418521+0000","last_peered":"2024-02-19T21:11:33.418521+0000","last_clean":"2024-02-19T21:11:33.418521+0000","last_became_active":"2024-02-19T21:01:22.607122+0000","last_became_peered":"2024-02-19T21:01:22.607122+0000","last_unstale":"2024-02-19T21:11:33.418521+0000","last_undegraded":"2024-02-19T21:11:33.418521+0000","last_fullsized":"2024-02-19T21:11:33.418521+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.878760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.878760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":382,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412353+0000","last_change":"2024-02-19T21:10:17.082171+0000","last_active":"2024-02-19T21:11:33.412353+0000","last_peered":"2024-02-19T21:11:33.412353+0000","last_clean":"2024-02-19T21:11:33.412353+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:11:33.412353+0000","last_undegraded":"2024-02-19T21:11:33.412353+0000","last_fullsized":"2024-02-19T21:11:33.412353+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:17.082103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:17.082103+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"459'15","reported_seq":332,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.412703+0000","last_change":"2024-02-19T21:10:41.659407+0000","last_active":"2024-02-19T21:11:32.412703+0000","last_peered":"2024-02-19T21:11:32.412703+0000","last_clean":"2024-02-19T21:11:32.412703+0000","last_became_active":"2024-02-19T21:10:41.295175+0000","last_became_peered":"2024-02-19T21:10:41.295175+0000","last_unstale":"2024-02-19T21:11:32.412703+0000","last_undegraded":"2024-02-19T21:11:32.412703+0000","last_fullsized":"2024-02-19T21:11:32.412703+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:08:53.783292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:53.783292+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":467,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.412502+0000","last_change":"2024-02-19T21:10:41.312903+0000","last_active":"2024-02-19T21:11:32.412502+0000","last_peered":"2024-02-19T21:11:32.412502+0000","last_clean":"2024-02-19T21:11:32.412502+0000","last_became_active":"2024-02-19T21:10:41.312116+0000","last_became_peered":"2024-02-19T21:10:41.312116+0000","last_unstale":"2024-02-19T21:11:32.412502+0000","last_undegraded":"2024-02-19T21:11:32.412502+0000","last_fullsized":"2024-02-19T21:11:32.412502+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:39.700315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.700315+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":466,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.409730+0000","last_change":"2024-02-19T21:10:41.306360+0000","last_active":"2024-02-19T21:11:33.409730+0000","last_peered":"2024-02-19T21:11:33.409730+0000","last_clean":"2024-02-19T21:11:33.409730+0000","last_became_active":"2024-02-19T21:10:41.305574+0000","last_became_peered":"2024-02-19T21:10:41.305574+0000","last_unstale":"2024-02-19T21:11:33.409730+0000","last_undegraded":"2024-02-19T21:11:33.409730+0000","last_fullsized":"2024-02-19T21:11:33.409730+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:13.035939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:13.035939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":265,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-19T21:11:34.504544+0000","last_change":"2024-02-19T21:11:34.504544+0000","last_active":"2024-02-19T21:11:34.504544+0000","last_peered":"2024-02-19T21:11:34.504544+0000","last_clean":"2024-02-19T21:11:34.504544+0000","last_became_active":"2024-02-19T21:06:40.227822+0000","last_became_peered":"2024-02-19T21:06:40.227822+0000","last_unstale":"2024-02-19T21:11:34.504544+0000","last_undegraded":"2024-02-19T21:11:34.504544+0000","last_fullsized":"2024-02-19T21:11:34.504544+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:34.504478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:34.504478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":383,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.419476+0000","last_change":"2024-02-19T21:10:57.130256+0000","last_active":"2024-02-19T21:11:33.419476+0000","last_peered":"2024-02-19T21:11:33.419476+0000","last_clean":"2024-02-19T21:11:33.419476+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:11:33.419476+0000","last_undegraded":"2024-02-19T21:11:33.419476+0000","last_fullsized":"2024-02-19T21:11:33.419476+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.130171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.130171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"459'35","reported_seq":488,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.414916+0000","last_change":"2024-02-19T21:11:04.345915+0000","last_active":"2024-02-19T21:11:32.414916+0000","last_peered":"2024-02-19T21:11:32.414916+0000","last_clean":"2024-02-19T21:11:32.414916+0000","last_became_active":"2024-02-19T21:01:18.293762+0000","last_became_peered":"2024-02-19T21:01:18.293762+0000","last_unstale":"2024-02-19T21:11:32.414916+0000","last_undegraded":"2024-02-19T21:11:32.414916+0000","last_fullsized":"2024-02-19T21:11:32.414916+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:11:04.345836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:04.345836+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":434,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412070+0000","last_change":"2024-02-19T21:09:30.950696+0000","last_active":"2024-02-19T21:11:33.412070+0000","last_peered":"2024-02-19T21:11:33.412070+0000","last_clean":"2024-02-19T21:11:33.412070+0000","last_became_active":"2024-02-19T21:01:16.275830+0000","last_became_peered":"2024-02-19T21:01:16.275830+0000","last_unstale":"2024-02-19T21:11:33.412070+0000","last_undegraded":"2024-02-19T21:11:33.412070+0000","last_fullsized":"2024-02-19T21:11:33.412070+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:30.950626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:09:30.950626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":425,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.410660+0000","last_change":"2024-02-19T21:09:28.031934+0000","last_active":"2024-02-19T21:11:33.410660+0000","last_peered":"2024-02-19T21:11:33.410660+0000","last_clean":"2024-02-19T21:11:33.410660+0000","last_became_active":"2024-02-19T21:01:20.287301+0000","last_became_peered":"2024-02-19T21:01:20.287301+0000","last_unstale":"2024-02-19T21:11:33.410660+0000","last_undegraded":"2024-02-19T21:11:33.410660+0000","last_fullsized":"2024-02-19T21:11:33.410660+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:28.031872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.031872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":445,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.469345+0000","last_change":"2024-02-19T21:11:33.469310+0000","last_active":"2024-02-19T21:11:33.469345+0000","last_peered":"2024-02-19T21:11:33.469345+0000","last_clean":"2024-02-19T21:11:33.469345+0000","last_became_active":"2024-02-19T21:08:04.265081+0000","last_became_peered":"2024-02-19T21:08:04.265081+0000","last_unstale":"2024-02-19T21:11:33.469345+0000","last_undegraded":"2024-02-19T21:11:33.469345+0000","last_fullsized":"2024-02-19T21:11:33.469345+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:33.469232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_clean_scrub_stamp":"2024-02-19T21:11:33.469232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":400,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.416948+0000","last_change":"2024-02-19T21:10:41.317716+0000","last_active":"2024-02-19T21:11:33.416948+0000","last_peered":"2024-02-19T21:11:33.416948+0000","last_clean":"2024-02-19T21:11:33.416948+0000","last_became_active":"2024-02-19T21:10:41.317284+0000","last_became_peered":"2024-02-19T21:10:41.317284+0000","last_unstale":"2024-02-19T21:11:33.416948+0000","last_undegraded":"2024-02-19T21:11:33.416948+0000","last_fullsized":"2024-02-19T21:11:33.416948+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:43.984079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:09:43.984079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"459'31","reported_seq":81,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412677+0000","last_change":"2024-02-19T21:10:41.676984+0000","last_active":"2024-02-19T21:11:33.412677+0000","last_peered":"2024-02-19T21:11:33.412677+0000","last_clean":"2024-02-19T21:11:33.412677+0000","last_became_active":"2024-02-19T21:10:41.313853+0000","last_became_peered":"2024-02-19T21:10:41.313853+0000","last_unstale":"2024-02-19T21:11:33.412677+0000","last_undegraded":"2024-02-19T21:11:33.412677+0000","last_fullsized":"2024-02-19T21:11:33.412677+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2024-02-19T21:09:41.955563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:41.955563+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":465,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.416074+0000","last_change":"2024-02-19T21:10:41.408960+0000","last_active":"2024-02-19T21:11:33.416074+0000","last_peered":"2024-02-19T21:11:33.416074+0000","last_clean":"2024-02-19T21:11:33.416074+0000","last_became_active":"2024-02-19T21:10:41.307517+0000","last_became_peered":"2024-02-19T21:10:41.307517+0000","last_unstale":"2024-02-19T21:11:33.416074+0000","last_undegraded":"2024-02-19T21:11:33.416074+0000","last_fullsized":"2024-02-19T21:11:33.416074+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:10:08.878163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:08.878163+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":445,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.412706+0000","last_change":"2024-02-19T21:10:41.312601+0000","last_active":"2024-02-19T21:11:32.412706+0000","last_peered":"2024-02-19T21:11:32.412706+0000","last_clean":"2024-02-19T21:11:32.412706+0000","last_became_active":"2024-02-19T21:10:41.312259+0000","last_became_peered":"2024-02-19T21:10:41.312259+0000","last_unstale":"2024-02-19T21:11:32.412706+0000","last_undegraded":"2024-02-19T21:11:32.412706+0000","last_fullsized":"2024-02-19T21:11:32.412706+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:36.685063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:36.685063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":464,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412886+0000","last_change":"2024-02-19T21:10:41.307901+0000","last_active":"2024-02-19T21:11:33.412886+0000","last_peered":"2024-02-19T21:11:33.412886+0000","last_clean":"2024-02-19T21:11:33.412886+0000","last_became_active":"2024-02-19T21:10:41.307577+0000","last_became_peered":"2024-02-19T21:10:41.307577+0000","last_unstale":"2024-02-19T21:11:33.412886+0000","last_undegraded":"2024-02-19T21:11:33.412886+0000","last_fullsized":"2024-02-19T21:11:33.412886+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:03.053807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:10:03.053807+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":47,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411861+0000","last_change":"2024-02-19T21:10:50.424462+0000","last_active":"2024-02-19T21:11:33.411861+0000","last_peered":"2024-02-19T21:11:33.411861+0000","last_clean":"2024-02-19T21:11:33.411861+0000","last_became_active":"2024-02-19T21:10:41.302670+0000","last_became_peered":"2024-02-19T21:10:41.302670+0000","last_unstale":"2024-02-19T21:11:33.411861+0000","last_undegraded":"2024-02-19T21:11:33.411861+0000","last_fullsized":"2024-02-19T21:11:33.411861+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:50.424381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.424381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"459'25","reported_seq":69,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412461+0000","last_change":"2024-02-19T21:10:41.329217+0000","last_active":"2024-02-19T21:11:33.412461+0000","last_peered":"2024-02-19T21:11:33.412461+0000","last_clean":"2024-02-19T21:11:33.412461+0000","last_became_active":"2024-02-19T21:10:41.291470+0000","last_became_peered":"2024-02-19T21:10:41.291470+0000","last_unstale":"2024-02-19T21:11:33.412461+0000","last_undegraded":"2024-02-19T21:11:33.412461+0000","last_fullsized":"2024-02-19T21:11:33.412461+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'15","last_scrub_stamp":"2024-02-19T21:10:10.650661+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:10:10.650661+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":449,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.411752+0000","last_change":"2024-02-19T21:10:12.628865+0000","last_active":"2024-02-19T21:11:32.411752+0000","last_peered":"2024-02-19T21:11:32.411752+0000","last_clean":"2024-02-19T21:11:32.411752+0000","last_became_active":"2024-02-19T21:08:04.268406+0000","last_became_peered":"2024-02-19T21:08:04.268406+0000","last_unstale":"2024-02-19T21:11:32.411752+0000","last_undegraded":"2024-02-19T21:11:32.411752+0000","last_fullsized":"2024-02-19T21:11:32.411752+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:12.628801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:12.628801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":562,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-19T21:11:34.298661+0000","last_change":"2024-02-19T21:10:04.016684+0000","last_active":"2024-02-19T21:11:34.298661+0000","last_peered":"2024-02-19T21:11:34.298661+0000","last_clean":"2024-02-19T21:11:34.298661+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:11:34.298661+0000","last_undegraded":"2024-02-19T21:11:34.298661+0000","last_fullsized":"2024-02-19T21:11:34.298661+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:10:04.016619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:04.016619+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":42,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412795+0000","last_change":"2024-02-19T21:10:41.299284+0000","last_active":"2024-02-19T21:11:33.412795+0000","last_peered":"2024-02-19T21:11:33.412795+0000","last_clean":"2024-02-19T21:11:33.412795+0000","last_became_active":"2024-02-19T21:10:41.298893+0000","last_became_peered":"2024-02-19T21:10:41.298893+0000","last_unstale":"2024-02-19T21:11:33.412795+0000","last_undegraded":"2024-02-19T21:11:33.412795+0000","last_fullsized":"2024-02-19T21:11:33.412795+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:02.519642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:10:02.519642+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.410275+0000","last_change":"2024-02-19T21:08:36.847613+0000","last_active":"2024-02-19T21:11:33.410275+0000","last_peered":"2024-02-19T21:11:33.410275+0000","last_clean":"2024-02-19T21:11:33.410275+0000","last_became_active":"2024-02-19T21:08:04.242244+0000","last_became_peered":"2024-02-19T21:08:04.242244+0000","last_unstale":"2024-02-19T21:11:33.410275+0000","last_undegraded":"2024-02-19T21:11:33.410275+0000","last_fullsized":"2024-02-19T21:11:33.410275+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:36.847548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.847548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"459'27","reported_seq":91,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.409294+0000","last_change":"2024-02-19T21:11:08.202280+0000","last_active":"2024-02-19T21:11:33.409294+0000","last_peered":"2024-02-19T21:11:33.409294+0000","last_clean":"2024-02-19T21:11:33.409294+0000","last_became_active":"2024-02-19T21:10:41.307576+0000","last_became_peered":"2024-02-19T21:10:41.307576+0000","last_unstale":"2024-02-19T21:11:33.409294+0000","last_undegraded":"2024-02-19T21:11:33.409294+0000","last_fullsized":"2024-02-19T21:11:33.409294+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:11:08.202208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:08.202208+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":476,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.411318+0000","last_change":"2024-02-19T21:10:52.738673+0000","last_active":"2024-02-19T21:11:32.411318+0000","last_peered":"2024-02-19T21:11:32.411318+0000","last_clean":"2024-02-19T21:11:32.411318+0000","last_became_active":"2024-02-19T21:08:04.270601+0000","last_became_peered":"2024-02-19T21:08:04.270601+0000","last_unstale":"2024-02-19T21:11:32.411318+0000","last_undegraded":"2024-02-19T21:11:32.411318+0000","last_fullsized":"2024-02-19T21:11:32.411318+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.738600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.738600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"434'10","reported_seq":58,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-19T21:11:34.304496+0000","last_change":"2024-02-19T21:10:41.665978+0000","last_active":"2024-02-19T21:11:34.304496+0000","last_peered":"2024-02-19T21:11:34.304496+0000","last_clean":"2024-02-19T21:11:34.304496+0000","last_became_active":"2024-02-19T21:10:41.312459+0000","last_became_peered":"2024-02-19T21:10:41.312459+0000","last_unstale":"2024-02-19T21:11:34.304496+0000","last_undegraded":"2024-02-19T21:11:34.304496+0000","last_fullsized":"2024-02-19T21:11:34.304496+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'9","last_scrub_stamp":"2024-02-19T21:10:12.186173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:12.186173+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":429,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.419680+0000","last_change":"2024-02-19T21:09:31.856861+0000","last_active":"2024-02-19T21:11:33.419680+0000","last_peered":"2024-02-19T21:11:33.419680+0000","last_clean":"2024-02-19T21:11:33.419680+0000","last_became_active":"2024-02-19T21:01:22.293254+0000","last_became_peered":"2024-02-19T21:01:22.293254+0000","last_unstale":"2024-02-19T21:11:33.419680+0000","last_undegraded":"2024-02-19T21:11:33.419680+0000","last_fullsized":"2024-02-19T21:11:33.419680+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:31.856784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:31.856784+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":383,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418648+0000","last_change":"2024-02-19T21:11:01.079431+0000","last_active":"2024-02-19T21:11:33.418648+0000","last_peered":"2024-02-19T21:11:33.418648+0000","last_clean":"2024-02-19T21:11:33.418648+0000","last_became_active":"2024-02-19T21:04:39.657052+0000","last_became_peered":"2024-02-19T21:04:39.657052+0000","last_unstale":"2024-02-19T21:11:33.418648+0000","last_undegraded":"2024-02-19T21:11:33.418648+0000","last_fullsized":"2024-02-19T21:11:33.418648+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.079353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.079353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"459'19","reported_seq":66,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412742+0000","last_change":"2024-02-19T21:10:41.335249+0000","last_active":"2024-02-19T21:11:33.412742+0000","last_peered":"2024-02-19T21:11:33.412742+0000","last_clean":"2024-02-19T21:11:33.412742+0000","last_became_active":"2024-02-19T21:10:41.296921+0000","last_became_peered":"2024-02-19T21:10:41.296921+0000","last_unstale":"2024-02-19T21:11:33.412742+0000","last_undegraded":"2024-02-19T21:11:33.412742+0000","last_fullsized":"2024-02-19T21:11:33.412742+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:49.181100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.181100+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":460,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.416131+0000","last_change":"2024-02-19T21:09:37.870271+0000","last_active":"2024-02-19T21:11:33.416131+0000","last_peered":"2024-02-19T21:11:33.416131+0000","last_clean":"2024-02-19T21:11:33.416131+0000","last_became_active":"2024-02-19T21:07:32.227557+0000","last_became_peered":"2024-02-19T21:07:32.227557+0000","last_unstale":"2024-02-19T21:11:33.416131+0000","last_undegraded":"2024-02-19T21:11:33.416131+0000","last_fullsized":"2024-02-19T21:11:33.416131+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.870218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.870218+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":449,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.419612+0000","last_change":"2024-02-19T21:10:58.096107+0000","last_active":"2024-02-19T21:11:33.419612+0000","last_peered":"2024-02-19T21:11:33.419612+0000","last_clean":"2024-02-19T21:11:33.419612+0000","last_became_active":"2024-02-19T21:10:41.314907+0000","last_became_peered":"2024-02-19T21:10:41.314907+0000","last_unstale":"2024-02-19T21:11:33.419612+0000","last_undegraded":"2024-02-19T21:11:33.419612+0000","last_fullsized":"2024-02-19T21:11:33.419612+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:58.096025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.096025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412440+0000","last_change":"2024-02-19T21:10:06.547564+0000","last_active":"2024-02-19T21:11:33.412440+0000","last_peered":"2024-02-19T21:11:33.412440+0000","last_clean":"2024-02-19T21:11:33.412440+0000","last_became_active":"2024-02-19T21:08:04.279008+0000","last_became_peered":"2024-02-19T21:08:04.279008+0000","last_unstale":"2024-02-19T21:11:33.412440+0000","last_undegraded":"2024-02-19T21:11:33.412440+0000","last_fullsized":"2024-02-19T21:11:33.412440+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:06.547494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:06.547494+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":172,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411696+0000","last_change":"2024-02-19T21:08:04.280833+0000","last_active":"2024-02-19T21:11:33.411696+0000","last_peered":"2024-02-19T21:11:33.411696+0000","last_clean":"2024-02-19T21:11:33.411696+0000","last_became_active":"2024-02-19T21:08:04.280514+0000","last_became_peered":"2024-02-19T21:08:04.280514+0000","last_unstale":"2024-02-19T21:11:33.411696+0000","last_undegraded":"2024-02-19T21:11:33.411696+0000","last_fullsized":"2024-02-19T21:11:33.411696+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:58.499841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:58.499841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"434'12","reported_seq":253,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-19T21:11:34.293321+0000","last_change":"2024-02-19T21:10:41.321881+0000","last_active":"2024-02-19T21:11:34.293321+0000","last_peered":"2024-02-19T21:11:34.293321+0000","last_clean":"2024-02-19T21:11:34.293321+0000","last_became_active":"2024-02-19T21:10:41.297897+0000","last_became_peered":"2024-02-19T21:10:41.297897+0000","last_unstale":"2024-02-19T21:11:34.293321+0000","last_undegraded":"2024-02-19T21:11:34.293321+0000","last_fullsized":"2024-02-19T21:11:34.293321+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'11","last_scrub_stamp":"2024-02-19T21:09:39.597001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:39.597001+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"459'28","reported_seq":84,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411918+0000","last_change":"2024-02-19T21:10:49.466291+0000","last_active":"2024-02-19T21:11:33.411918+0000","last_peered":"2024-02-19T21:11:33.411918+0000","last_clean":"2024-02-19T21:11:33.411918+0000","last_became_active":"2024-02-19T21:10:41.308428+0000","last_became_peered":"2024-02-19T21:10:41.308428+0000","last_unstale":"2024-02-19T21:11:33.411918+0000","last_undegraded":"2024-02-19T21:11:33.411918+0000","last_fullsized":"2024-02-19T21:11:33.411918+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:10:49.466227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:49.466227+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":462,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.416905+0000","last_change":"2024-02-19T21:11:05.364793+0000","last_active":"2024-02-19T21:11:32.416905+0000","last_peered":"2024-02-19T21:11:32.416905+0000","last_clean":"2024-02-19T21:11:32.416905+0000","last_became_active":"2024-02-19T21:07:36.263299+0000","last_became_peered":"2024-02-19T21:07:36.263299+0000","last_unstale":"2024-02-19T21:11:32.416905+0000","last_undegraded":"2024-02-19T21:11:32.416905+0000","last_fullsized":"2024-02-19T21:11:32.416905+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.364732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.364732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":431,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.416347+0000","last_change":"2024-02-19T21:10:51.851228+0000","last_active":"2024-02-19T21:11:33.416347+0000","last_peered":"2024-02-19T21:11:33.416347+0000","last_clean":"2024-02-19T21:11:33.416347+0000","last_became_active":"2024-02-19T21:01:22.606779+0000","last_became_peered":"2024-02-19T21:01:22.606779+0000","last_unstale":"2024-02-19T21:11:33.416347+0000","last_undegraded":"2024-02-19T21:11:33.416347+0000","last_fullsized":"2024-02-19T21:11:33.416347+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.851123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.851123+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":257,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.412606+0000","last_change":"2024-02-19T21:10:06.601328+0000","last_active":"2024-02-19T21:11:32.412606+0000","last_peered":"2024-02-19T21:11:32.412606+0000","last_clean":"2024-02-19T21:11:32.412606+0000","last_became_active":"2024-02-19T21:06:40.245385+0000","last_became_peered":"2024-02-19T21:06:40.245385+0000","last_unstale":"2024-02-19T21:11:32.412606+0000","last_undegraded":"2024-02-19T21:11:32.412606+0000","last_fullsized":"2024-02-19T21:11:32.412606+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:06.601228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:06.601228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":47,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411626+0000","last_change":"2024-02-19T21:10:59.312931+0000","last_active":"2024-02-19T21:11:33.411626+0000","last_peered":"2024-02-19T21:11:33.411626+0000","last_clean":"2024-02-19T21:11:33.411626+0000","last_became_active":"2024-02-19T21:10:41.304695+0000","last_became_peered":"2024-02-19T21:10:41.304695+0000","last_unstale":"2024-02-19T21:11:33.411626+0000","last_undegraded":"2024-02-19T21:11:33.411626+0000","last_fullsized":"2024-02-19T21:11:33.411626+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:59.312863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.312863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"459'30","reported_seq":213,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.410370+0000","last_change":"2024-02-19T21:11:02.671928+0000","last_active":"2024-02-19T21:11:33.410370+0000","last_peered":"2024-02-19T21:11:33.410370+0000","last_clean":"2024-02-19T21:11:33.410370+0000","last_became_active":"2024-02-19T21:08:04.242466+0000","last_became_peered":"2024-02-19T21:08:04.242466+0000","last_unstale":"2024-02-19T21:11:33.410370+0000","last_undegraded":"2024-02-19T21:11:33.410370+0000","last_fullsized":"2024-02-19T21:11:33.410370+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:11:02.671837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.671837+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":471,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.412027+0000","last_change":"2024-02-19T21:10:41.296353+0000","last_active":"2024-02-19T21:11:32.412027+0000","last_peered":"2024-02-19T21:11:32.412027+0000","last_clean":"2024-02-19T21:11:32.412027+0000","last_became_active":"2024-02-19T21:10:41.296010+0000","last_became_peered":"2024-02-19T21:10:41.296010+0000","last_unstale":"2024-02-19T21:11:32.412027+0000","last_undegraded":"2024-02-19T21:11:32.412027+0000","last_fullsized":"2024-02-19T21:11:32.412027+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":42,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411908+0000","last_change":"2024-02-19T21:10:41.295616+0000","last_active":"2024-02-19T21:11:33.411908+0000","last_peered":"2024-02-19T21:11:33.411908+0000","last_clean":"2024-02-19T21:11:33.411908+0000","last_became_active":"2024-02-19T21:10:41.295301+0000","last_became_peered":"2024-02-19T21:10:41.295301+0000","last_unstale":"2024-02-19T21:11:33.411908+0000","last_undegraded":"2024-02-19T21:11:33.411908+0000","last_fullsized":"2024-02-19T21:11:33.411908+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:40.549690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.549690+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":172,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412598+0000","last_change":"2024-02-19T21:08:04.267005+0000","last_active":"2024-02-19T21:11:33.412598+0000","last_peered":"2024-02-19T21:11:33.412598+0000","last_clean":"2024-02-19T21:11:33.412598+0000","last_became_active":"2024-02-19T21:08:04.266538+0000","last_became_peered":"2024-02-19T21:08:04.266538+0000","last_unstale":"2024-02-19T21:11:33.412598+0000","last_undegraded":"2024-02-19T21:11:33.412598+0000","last_fullsized":"2024-02-19T21:11:33.412598+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:58.847915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:07:58.847915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":431,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.415011+0000","last_change":"2024-02-19T21:10:20.474926+0000","last_active":"2024-02-19T21:11:32.415011+0000","last_peered":"2024-02-19T21:11:32.415011+0000","last_clean":"2024-02-19T21:11:32.415011+0000","last_became_active":"2024-02-19T21:01:20.287390+0000","last_became_peered":"2024-02-19T21:01:20.287390+0000","last_unstale":"2024-02-19T21:11:32.415011+0000","last_undegraded":"2024-02-19T21:11:32.415011+0000","last_fullsized":"2024-02-19T21:11:32.415011+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:20.474844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:20.474844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"459'29","reported_seq":324,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418943+0000","last_change":"2024-02-19T21:10:11.166057+0000","last_active":"2024-02-19T21:11:33.418943+0000","last_peered":"2024-02-19T21:11:33.418943+0000","last_clean":"2024-02-19T21:11:33.418943+0000","last_became_active":"2024-02-19T21:06:40.223768+0000","last_became_peered":"2024-02-19T21:06:40.223768+0000","last_unstale":"2024-02-19T21:11:33.418943+0000","last_undegraded":"2024-02-19T21:11:33.418943+0000","last_fullsized":"2024-02-19T21:11:33.418943+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:10:11.165973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:11.165973+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":480,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.417317+0000","last_change":"2024-02-19T21:10:41.317015+0000","last_active":"2024-02-19T21:11:33.417317+0000","last_peered":"2024-02-19T21:11:33.417317+0000","last_clean":"2024-02-19T21:11:33.417317+0000","last_became_active":"2024-02-19T21:10:41.315864+0000","last_became_peered":"2024-02-19T21:10:41.315864+0000","last_unstale":"2024-02-19T21:11:33.417317+0000","last_undegraded":"2024-02-19T21:11:33.417317+0000","last_fullsized":"2024-02-19T21:11:33.417317+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:35.824520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:35.824520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":420,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.411435+0000","last_change":"2024-02-19T21:08:57.653280+0000","last_active":"2024-02-19T21:11:32.411435+0000","last_peered":"2024-02-19T21:11:32.411435+0000","last_clean":"2024-02-19T21:11:32.411435+0000","last_became_active":"2024-02-19T21:01:22.606981+0000","last_became_peered":"2024-02-19T21:01:22.606981+0000","last_unstale":"2024-02-19T21:11:32.411435+0000","last_undegraded":"2024-02-19T21:11:32.411435+0000","last_fullsized":"2024-02-19T21:11:32.411435+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:57.653204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:08:57.653204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":382,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412935+0000","last_change":"2024-02-19T21:11:02.189586+0000","last_active":"2024-02-19T21:11:33.412935+0000","last_peered":"2024-02-19T21:11:33.412935+0000","last_clean":"2024-02-19T21:11:33.412935+0000","last_became_active":"2024-02-19T21:04:39.718926+0000","last_became_peered":"2024-02-19T21:04:39.718926+0000","last_unstale":"2024-02-19T21:11:33.412935+0000","last_undegraded":"2024-02-19T21:11:33.412935+0000","last_fullsized":"2024-02-19T21:11:33.412935+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.189509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":431,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.419663+0000","last_change":"2024-02-19T21:11:30.893566+0000","last_active":"2024-02-19T21:11:33.419663+0000","last_peered":"2024-02-19T21:11:33.419663+0000","last_clean":"2024-02-19T21:11:33.419663+0000","last_became_active":"2024-02-19T21:01:20.288714+0000","last_became_peered":"2024-02-19T21:01:20.288714+0000","last_unstale":"2024-02-19T21:11:33.419663+0000","last_undegraded":"2024-02-19T21:11:33.419663+0000","last_fullsized":"2024-02-19T21:11:33.419663+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:30.893485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:30.893485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"459'24","reported_seq":305,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.412157+0000","last_change":"2024-02-19T21:09:30.704618+0000","last_active":"2024-02-19T21:11:32.412157+0000","last_peered":"2024-02-19T21:11:32.412157+0000","last_clean":"2024-02-19T21:11:32.412157+0000","last_became_active":"2024-02-19T21:06:40.240321+0000","last_became_peered":"2024-02-19T21:06:40.240321+0000","last_unstale":"2024-02-19T21:11:32.412157+0000","last_undegraded":"2024-02-19T21:11:32.412157+0000","last_fullsized":"2024-02-19T21:11:32.412157+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:09:30.704543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:30.704543+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"397'2","reported_seq":468,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.415775+0000","last_change":"2024-02-19T21:09:40.934534+0000","last_active":"2024-02-19T21:11:33.415775+0000","last_peered":"2024-02-19T21:11:33.415775+0000","last_clean":"2024-02-19T21:11:33.415775+0000","last_became_active":"2024-02-19T21:08:10.225480+0000","last_became_peered":"2024-02-19T21:08:10.225480+0000","last_unstale":"2024-02-19T21:11:33.415775+0000","last_undegraded":"2024-02-19T21:11:33.415775+0000","last_fullsized":"2024-02-19T21:11:33.415775+0000","mapping_epoch":57,"log_start":"58'1","ondisk_log_start":"58'1","created":57,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.934474+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":47,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411968+0000","last_change":"2024-02-19T21:11:07.456130+0000","last_active":"2024-02-19T21:11:33.411968+0000","last_peered":"2024-02-19T21:11:33.411968+0000","last_clean":"2024-02-19T21:11:33.411968+0000","last_became_active":"2024-02-19T21:10:41.294427+0000","last_became_peered":"2024-02-19T21:10:41.294427+0000","last_unstale":"2024-02-19T21:11:33.411968+0000","last_undegraded":"2024-02-19T21:11:33.411968+0000","last_fullsized":"2024-02-19T21:11:33.411968+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.455993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.455993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":443,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.412795+0000","last_change":"2024-02-19T21:10:51.691357+0000","last_active":"2024-02-19T21:11:32.412795+0000","last_peered":"2024-02-19T21:11:32.412795+0000","last_clean":"2024-02-19T21:11:32.412795+0000","last_became_active":"2024-02-19T21:08:04.282243+0000","last_became_peered":"2024-02-19T21:08:04.282243+0000","last_unstale":"2024-02-19T21:11:32.412795+0000","last_undegraded":"2024-02-19T21:11:32.412795+0000","last_fullsized":"2024-02-19T21:11:32.412795+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.691284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.691284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":380,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.414764+0000","last_change":"2024-02-19T21:10:21.484781+0000","last_active":"2024-02-19T21:11:32.414764+0000","last_peered":"2024-02-19T21:11:32.414764+0000","last_clean":"2024-02-19T21:11:32.414764+0000","last_became_active":"2024-02-19T21:04:39.662125+0000","last_became_peered":"2024-02-19T21:04:39.662125+0000","last_unstale":"2024-02-19T21:11:32.414764+0000","last_undegraded":"2024-02-19T21:11:32.414764+0000","last_fullsized":"2024-02-19T21:11:32.414764+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:21.484700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:21.484700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":429,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.416552+0000","last_change":"2024-02-19T21:10:03.998625+0000","last_active":"2024-02-19T21:11:33.416552+0000","last_peered":"2024-02-19T21:11:33.416552+0000","last_clean":"2024-02-19T21:11:33.416552+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:11:33.416552+0000","last_undegraded":"2024-02-19T21:11:33.416552+0000","last_fullsized":"2024-02-19T21:11:33.416552+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:03.998559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:03.998559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"459'26","reported_seq":518,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.417102+0000","last_change":"2024-02-19T21:10:41.414468+0000","last_active":"2024-02-19T21:11:32.417102+0000","last_peered":"2024-02-19T21:11:32.417102+0000","last_clean":"2024-02-19T21:11:32.417102+0000","last_became_active":"2024-02-19T21:10:41.298131+0000","last_became_peered":"2024-02-19T21:10:41.298131+0000","last_unstale":"2024-02-19T21:11:32.417102+0000","last_undegraded":"2024-02-19T21:11:32.417102+0000","last_fullsized":"2024-02-19T21:11:32.417102+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:10:15.518555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:15.518555+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":442,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.416002+0000","last_change":"2024-02-19T21:11:03.747202+0000","last_active":"2024-02-19T21:11:33.416002+0000","last_peered":"2024-02-19T21:11:33.416002+0000","last_clean":"2024-02-19T21:11:33.416002+0000","last_became_active":"2024-02-19T21:01:11.714212+0000","last_became_peered":"2024-02-19T21:01:11.714212+0000","last_unstale":"2024-02-19T21:11:33.416002+0000","last_undegraded":"2024-02-19T21:11:33.416002+0000","last_fullsized":"2024-02-19T21:11:33.416002+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.747108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.747108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":462,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418827+0000","last_change":"2024-02-19T21:10:51.082102+0000","last_active":"2024-02-19T21:11:33.418827+0000","last_peered":"2024-02-19T21:11:33.418827+0000","last_clean":"2024-02-19T21:11:33.418827+0000","last_became_active":"2024-02-19T21:07:58.238944+0000","last_became_peered":"2024-02-19T21:07:58.238944+0000","last_unstale":"2024-02-19T21:11:33.418827+0000","last_undegraded":"2024-02-19T21:11:33.418827+0000","last_fullsized":"2024-02-19T21:11:33.418827+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.081993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.081993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411778+0000","last_change":"2024-02-19T21:08:53.820016+0000","last_active":"2024-02-19T21:11:33.411778+0000","last_peered":"2024-02-19T21:11:33.411778+0000","last_clean":"2024-02-19T21:11:33.411778+0000","last_became_active":"2024-02-19T21:08:04.278265+0000","last_became_peered":"2024-02-19T21:08:04.278265+0000","last_unstale":"2024-02-19T21:11:33.411778+0000","last_undegraded":"2024-02-19T21:11:33.411778+0000","last_fullsized":"2024-02-19T21:11:33.411778+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:53.819949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:53.819949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":465,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.419728+0000","last_change":"2024-02-19T21:10:41.315872+0000","last_active":"2024-02-19T21:11:33.419728+0000","last_peered":"2024-02-19T21:11:33.419728+0000","last_clean":"2024-02-19T21:11:33.419728+0000","last_became_active":"2024-02-19T21:10:41.315397+0000","last_became_peered":"2024-02-19T21:10:41.315397+0000","last_unstale":"2024-02-19T21:11:33.419728+0000","last_undegraded":"2024-02-19T21:11:33.419728+0000","last_fullsized":"2024-02-19T21:11:33.419728+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:10.148733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:10.148733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":383,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418174+0000","last_change":"2024-02-19T21:10:20.204982+0000","last_active":"2024-02-19T21:11:33.418174+0000","last_peered":"2024-02-19T21:11:33.418174+0000","last_clean":"2024-02-19T21:11:33.418174+0000","last_became_active":"2024-02-19T21:04:39.666739+0000","last_became_peered":"2024-02-19T21:04:39.666739+0000","last_unstale":"2024-02-19T21:11:33.418174+0000","last_undegraded":"2024-02-19T21:11:33.418174+0000","last_fullsized":"2024-02-19T21:11:33.418174+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_clean_scrub_stamp":"2024-02-19T21:10:20.204880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":430,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411324+0000","last_change":"2024-02-19T21:10:07.040966+0000","last_active":"2024-02-19T21:11:33.411324+0000","last_peered":"2024-02-19T21:11:33.411324+0000","last_clean":"2024-02-19T21:11:33.411324+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:11:33.411324+0000","last_undegraded":"2024-02-19T21:11:33.411324+0000","last_fullsized":"2024-02-19T21:11:33.411324+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:07.040893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:07.040893+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"459'22","reported_seq":454,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412926+0000","last_change":"2024-02-19T21:11:00.191283+0000","last_active":"2024-02-19T21:11:33.412926+0000","last_peered":"2024-02-19T21:11:33.412926+0000","last_clean":"2024-02-19T21:11:33.412926+0000","last_became_active":"2024-02-19T21:01:30.929797+0000","last_became_peered":"2024-02-19T21:01:30.929797+0000","last_unstale":"2024-02-19T21:11:33.412926+0000","last_undegraded":"2024-02-19T21:11:33.412926+0000","last_fullsized":"2024-02-19T21:11:33.412926+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:11:00.191206+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.191206+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":475,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.409484+0000","last_change":"2024-02-19T21:09:37.955966+0000","last_active":"2024-02-19T21:11:33.409484+0000","last_peered":"2024-02-19T21:11:33.409484+0000","last_clean":"2024-02-19T21:11:33.409484+0000","last_became_active":"2024-02-19T21:08:06.473207+0000","last_became_peered":"2024-02-19T21:08:06.473207+0000","last_unstale":"2024-02-19T21:11:33.409484+0000","last_undegraded":"2024-02-19T21:11:33.409484+0000","last_fullsized":"2024-02-19T21:11:33.409484+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.955872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.955872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":1044,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:34.029687+0000","last_change":"2024-02-19T21:10:03.552765+0000","last_active":"2024-02-19T21:11:34.029687+0000","last_peered":"2024-02-19T21:11:34.029687+0000","last_clean":"2024-02-19T21:11:34.029687+0000","last_became_active":"2024-02-19T21:01:16.277931+0000","last_became_peered":"2024-02-19T21:01:16.277931+0000","last_unstale":"2024-02-19T21:11:34.029687+0000","last_undegraded":"2024-02-19T21:11:34.029687+0000","last_fullsized":"2024-02-19T21:11:34.029687+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:10:03.552690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:03.552690+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":610,"num_read_kb":605,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":445,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.417204+0000","last_change":"2024-02-19T21:10:07.884980+0000","last_active":"2024-02-19T21:11:33.417204+0000","last_peered":"2024-02-19T21:11:33.417204+0000","last_clean":"2024-02-19T21:11:33.417204+0000","last_became_active":"2024-02-19T21:08:04.264896+0000","last_became_peered":"2024-02-19T21:08:04.264896+0000","last_unstale":"2024-02-19T21:11:33.417204+0000","last_undegraded":"2024-02-19T21:11:33.417204+0000","last_fullsized":"2024-02-19T21:11:33.417204+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:07.884905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:07.884905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":379,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.412846+0000","last_change":"2024-02-19T21:11:07.947170+0000","last_active":"2024-02-19T21:11:32.412846+0000","last_peered":"2024-02-19T21:11:32.412846+0000","last_clean":"2024-02-19T21:11:32.412846+0000","last_became_active":"2024-02-19T21:04:39.691434+0000","last_became_peered":"2024-02-19T21:04:39.691434+0000","last_unstale":"2024-02-19T21:11:32.412846+0000","last_undegraded":"2024-02-19T21:11:32.412846+0000","last_fullsized":"2024-02-19T21:11:32.412846+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.947089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":275,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.412412+0000","last_change":"2024-02-19T21:10:41.296313+0000","last_active":"2024-02-19T21:11:32.412412+0000","last_peered":"2024-02-19T21:11:32.412412+0000","last_clean":"2024-02-19T21:11:32.412412+0000","last_became_active":"2024-02-19T21:10:41.295968+0000","last_became_peered":"2024-02-19T21:10:41.295968+0000","last_unstale":"2024-02-19T21:11:32.412412+0000","last_undegraded":"2024-02-19T21:11:32.412412+0000","last_fullsized":"2024-02-19T21:11:32.412412+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:36.716464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:08:36.716464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"459'42","reported_seq":681,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-19T21:11:34.330339+0000","last_change":"2024-02-19T21:10:01.545972+0000","last_active":"2024-02-19T21:11:34.330339+0000","last_peered":"2024-02-19T21:11:34.330339+0000","last_clean":"2024-02-19T21:11:34.330339+0000","last_became_active":"2024-02-19T21:08:04.266322+0000","last_became_peered":"2024-02-19T21:08:04.266322+0000","last_unstale":"2024-02-19T21:11:34.330339+0000","last_undegraded":"2024-02-19T21:11:34.330339+0000","last_fullsized":"2024-02-19T21:11:34.330339+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'30","last_scrub_stamp":"2024-02-19T21:10:01.545897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:01.545897+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":462,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418627+0000","last_change":"2024-02-19T21:10:52.089790+0000","last_active":"2024-02-19T21:11:33.418627+0000","last_peered":"2024-02-19T21:11:33.418627+0000","last_clean":"2024-02-19T21:11:33.418627+0000","last_became_active":"2024-02-19T21:08:02.435663+0000","last_became_peered":"2024-02-19T21:08:02.435663+0000","last_unstale":"2024-02-19T21:11:33.418627+0000","last_undegraded":"2024-02-19T21:11:33.418627+0000","last_fullsized":"2024-02-19T21:11:33.418627+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.089715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.089715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":195,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412861+0000","last_change":"2024-02-19T21:11:31.629134+0000","last_active":"2024-02-19T21:11:33.412861+0000","last_peered":"2024-02-19T21:11:33.412861+0000","last_clean":"2024-02-19T21:11:33.412861+0000","last_became_active":"2024-02-19T21:10:41.305829+0000","last_became_peered":"2024-02-19T21:10:41.305829+0000","last_unstale":"2024-02-19T21:11:33.412861+0000","last_undegraded":"2024-02-19T21:11:33.412861+0000","last_fullsized":"2024-02-19T21:11:33.412861+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:31.629057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.629057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":382,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.409165+0000","last_change":"2024-02-19T21:10:54.142968+0000","last_active":"2024-02-19T21:11:33.409165+0000","last_peered":"2024-02-19T21:11:33.409165+0000","last_clean":"2024-02-19T21:11:33.409165+0000","last_became_active":"2024-02-19T21:04:39.721416+0000","last_became_peered":"2024-02-19T21:04:39.721416+0000","last_unstale":"2024-02-19T21:11:33.409165+0000","last_undegraded":"2024-02-19T21:11:33.409165+0000","last_fullsized":"2024-02-19T21:11:33.409165+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:54.142898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.142898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"220'9","reported_seq":338,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-19T21:11:34.307297+0000","last_change":"2024-02-19T21:10:12.455153+0000","last_active":"2024-02-19T21:11:34.307297+0000","last_peered":"2024-02-19T21:11:34.307297+0000","last_clean":"2024-02-19T21:11:34.307297+0000","last_became_active":"2024-02-19T21:06:40.227604+0000","last_became_peered":"2024-02-19T21:06:40.227604+0000","last_unstale":"2024-02-19T21:11:34.307297+0000","last_undegraded":"2024-02-19T21:11:34.307297+0000","last_fullsized":"2024-02-19T21:11:34.307297+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'9","last_scrub_stamp":"2024-02-19T21:10:12.455053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:12.455053+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"459'20","reported_seq":496,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.411846+0000","last_change":"2024-02-19T21:10:41.747145+0000","last_active":"2024-02-19T21:11:32.411846+0000","last_peered":"2024-02-19T21:11:32.411846+0000","last_clean":"2024-02-19T21:11:32.411846+0000","last_became_active":"2024-02-19T21:10:41.296275+0000","last_became_peered":"2024-02-19T21:10:41.296275+0000","last_unstale":"2024-02-19T21:11:32.411846+0000","last_undegraded":"2024-02-19T21:11:32.411846+0000","last_fullsized":"2024-02-19T21:11:32.411846+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:09:28.659495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.659495+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":476,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.411897+0000","last_change":"2024-02-19T21:10:21.732847+0000","last_active":"2024-02-19T21:11:32.411897+0000","last_peered":"2024-02-19T21:11:32.411897+0000","last_clean":"2024-02-19T21:11:32.411897+0000","last_became_active":"2024-02-19T21:07:54.235080+0000","last_became_peered":"2024-02-19T21:07:54.235080+0000","last_unstale":"2024-02-19T21:11:32.411897+0000","last_undegraded":"2024-02-19T21:11:32.411897+0000","last_fullsized":"2024-02-19T21:11:32.411897+0000","mapping_epoch":285,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:21.732778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:10:21.732778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.410363+0000","last_change":"2024-02-19T21:09:27.747880+0000","last_active":"2024-02-19T21:11:33.410363+0000","last_peered":"2024-02-19T21:11:33.410363+0000","last_clean":"2024-02-19T21:11:33.410363+0000","last_became_active":"2024-02-19T21:08:04.244317+0000","last_became_peered":"2024-02-19T21:08:04.244317+0000","last_unstale":"2024-02-19T21:11:33.410363+0000","last_undegraded":"2024-02-19T21:11:33.410363+0000","last_fullsized":"2024-02-19T21:11:33.410363+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.747789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.747789+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412933+0000","last_change":"2024-02-19T21:11:05.604091+0000","last_active":"2024-02-19T21:11:33.412933+0000","last_peered":"2024-02-19T21:11:33.412933+0000","last_clean":"2024-02-19T21:11:33.412933+0000","last_became_active":"2024-02-19T21:08:04.261408+0000","last_became_peered":"2024-02-19T21:08:04.261408+0000","last_unstale":"2024-02-19T21:11:33.412933+0000","last_undegraded":"2024-02-19T21:11:33.412933+0000","last_fullsized":"2024-02-19T21:11:33.412933+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.604025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.604025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":430,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.409060+0000","last_change":"2024-02-19T21:10:10.137424+0000","last_active":"2024-02-19T21:11:33.409060+0000","last_peered":"2024-02-19T21:11:33.409060+0000","last_clean":"2024-02-19T21:11:33.409060+0000","last_became_active":"2024-02-19T21:01:20.285899+0000","last_became_peered":"2024-02-19T21:01:20.285899+0000","last_unstale":"2024-02-19T21:11:33.409060+0000","last_undegraded":"2024-02-19T21:11:33.409060+0000","last_fullsized":"2024-02-19T21:11:33.409060+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:10.137356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:10:10.137356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"459'23","reported_seq":63,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411727+0000","last_change":"2024-02-19T21:10:41.328739+0000","last_active":"2024-02-19T21:11:33.411727+0000","last_peered":"2024-02-19T21:11:33.411727+0000","last_clean":"2024-02-19T21:11:33.411727+0000","last_became_active":"2024-02-19T21:10:41.287134+0000","last_became_peered":"2024-02-19T21:10:41.287134+0000","last_unstale":"2024-02-19T21:11:33.411727+0000","last_undegraded":"2024-02-19T21:11:33.411727+0000","last_fullsized":"2024-02-19T21:11:33.411727+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'17","last_scrub_stamp":"2024-02-19T21:10:19.722303+0000","last_deep_scrub":"420'17","last_deep_scrub_stamp":"2024-02-19T21:10:19.722303+0000","last_clean_scrub_stamp":"2024-02-19T21:10:19.722303+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":53,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.413067+0000","last_change":"2024-02-19T21:10:58.333279+0000","last_active":"2024-02-19T21:11:33.413067+0000","last_peered":"2024-02-19T21:11:33.413067+0000","last_clean":"2024-02-19T21:11:33.413067+0000","last_became_active":"2024-02-19T21:10:41.302730+0000","last_became_peered":"2024-02-19T21:10:41.302730+0000","last_unstale":"2024-02-19T21:11:33.413067+0000","last_undegraded":"2024-02-19T21:11:33.413067+0000","last_fullsized":"2024-02-19T21:11:33.413067+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:10:58.333168+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.333168+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411880+0000","last_change":"2024-02-19T21:08:57.881002+0000","last_active":"2024-02-19T21:11:33.411880+0000","last_peered":"2024-02-19T21:11:33.411880+0000","last_clean":"2024-02-19T21:11:33.411880+0000","last_became_active":"2024-02-19T21:08:04.264593+0000","last_became_peered":"2024-02-19T21:08:04.264593+0000","last_unstale":"2024-02-19T21:11:33.411880+0000","last_undegraded":"2024-02-19T21:11:33.411880+0000","last_fullsized":"2024-02-19T21:11:33.411880+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:57.880935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:08:57.880935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":382,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.410333+0000","last_change":"2024-02-19T21:11:03.195680+0000","last_active":"2024-02-19T21:11:33.410333+0000","last_peered":"2024-02-19T21:11:33.410333+0000","last_clean":"2024-02-19T21:11:33.410333+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:11:33.410333+0000","last_undegraded":"2024-02-19T21:11:33.410333+0000","last_fullsized":"2024-02-19T21:11:33.410333+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.195616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.195616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"459'15","reported_seq":198,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412781+0000","last_change":"2024-02-19T21:10:22.459882+0000","last_active":"2024-02-19T21:11:33.412781+0000","last_peered":"2024-02-19T21:11:33.412781+0000","last_clean":"2024-02-19T21:11:33.412781+0000","last_became_active":"2024-02-19T21:08:04.258740+0000","last_became_peered":"2024-02-19T21:08:04.258740+0000","last_unstale":"2024-02-19T21:11:33.412781+0000","last_undegraded":"2024-02-19T21:11:33.412781+0000","last_fullsized":"2024-02-19T21:11:33.412781+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:10:22.459807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:22.459807+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":434,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.409012+0000","last_change":"2024-02-19T21:11:07.244356+0000","last_active":"2024-02-19T21:11:33.409012+0000","last_peered":"2024-02-19T21:11:33.409012+0000","last_clean":"2024-02-19T21:11:33.409012+0000","last_became_active":"2024-02-19T21:01:16.276563+0000","last_became_peered":"2024-02-19T21:01:16.276563+0000","last_unstale":"2024-02-19T21:11:33.409012+0000","last_undegraded":"2024-02-19T21:11:33.409012+0000","last_fullsized":"2024-02-19T21:11:33.409012+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.244285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.244285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412548+0000","last_change":"2024-02-19T21:10:53.703797+0000","last_active":"2024-02-19T21:11:33.412548+0000","last_peered":"2024-02-19T21:11:33.412548+0000","last_clean":"2024-02-19T21:11:33.412548+0000","last_became_active":"2024-02-19T21:08:04.264972+0000","last_became_peered":"2024-02-19T21:08:04.264972+0000","last_unstale":"2024-02-19T21:11:33.412548+0000","last_undegraded":"2024-02-19T21:11:33.412548+0000","last_fullsized":"2024-02-19T21:11:33.412548+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:53.703727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.703727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":446,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411944+0000","last_change":"2024-02-19T21:10:41.314453+0000","last_active":"2024-02-19T21:11:33.411944+0000","last_peered":"2024-02-19T21:11:33.411944+0000","last_clean":"2024-02-19T21:11:33.411944+0000","last_became_active":"2024-02-19T21:10:41.314114+0000","last_became_peered":"2024-02-19T21:10:41.314114+0000","last_unstale":"2024-02-19T21:11:33.411944+0000","last_undegraded":"2024-02-19T21:11:33.411944+0000","last_fullsized":"2024-02-19T21:11:33.411944+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:48.995828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:08:48.995828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":42,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412115+0000","last_change":"2024-02-19T21:10:41.303286+0000","last_active":"2024-02-19T21:11:33.412115+0000","last_peered":"2024-02-19T21:11:33.412115+0000","last_clean":"2024-02-19T21:11:33.412115+0000","last_became_active":"2024-02-19T21:10:41.300574+0000","last_became_peered":"2024-02-19T21:10:41.300574+0000","last_unstale":"2024-02-19T21:11:33.412115+0000","last_undegraded":"2024-02-19T21:11:33.412115+0000","last_fullsized":"2024-02-19T21:11:33.412115+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:14.631330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:14.631330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"459'15","reported_seq":65,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.409186+0000","last_change":"2024-02-19T21:10:53.101489+0000","last_active":"2024-02-19T21:11:33.409186+0000","last_peered":"2024-02-19T21:11:33.409186+0000","last_clean":"2024-02-19T21:11:33.409186+0000","last_became_active":"2024-02-19T21:10:40.297867+0000","last_became_peered":"2024-02-19T21:10:40.297867+0000","last_unstale":"2024-02-19T21:11:33.409186+0000","last_undegraded":"2024-02-19T21:11:33.409186+0000","last_fullsized":"2024-02-19T21:11:33.409186+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:10:53.101422+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.101422+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":453,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.415260+0000","last_change":"2024-02-19T21:11:31.670103+0000","last_active":"2024-02-19T21:11:32.415260+0000","last_peered":"2024-02-19T21:11:32.415260+0000","last_clean":"2024-02-19T21:11:32.415260+0000","last_became_active":"2024-02-19T21:10:41.311362+0000","last_became_peered":"2024-02-19T21:10:41.311362+0000","last_unstale":"2024-02-19T21:11:32.415260+0000","last_undegraded":"2024-02-19T21:11:32.415260+0000","last_fullsized":"2024-02-19T21:11:32.415260+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:31.670026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.670026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":446,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.419388+0000","last_change":"2024-02-19T21:08:56.601041+0000","last_active":"2024-02-19T21:11:33.419388+0000","last_peered":"2024-02-19T21:11:33.419388+0000","last_clean":"2024-02-19T21:11:33.419388+0000","last_became_active":"2024-02-19T21:08:04.265573+0000","last_became_peered":"2024-02-19T21:08:04.265573+0000","last_unstale":"2024-02-19T21:11:33.419388+0000","last_undegraded":"2024-02-19T21:11:33.419388+0000","last_fullsized":"2024-02-19T21:11:33.419388+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:56.600951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:56.600951+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":445,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.416872+0000","last_change":"2024-02-19T21:10:41.316846+0000","last_active":"2024-02-19T21:11:33.416872+0000","last_peered":"2024-02-19T21:11:33.416872+0000","last_clean":"2024-02-19T21:11:33.416872+0000","last_became_active":"2024-02-19T21:10:41.315426+0000","last_became_peered":"2024-02-19T21:10:41.315426+0000","last_unstale":"2024-02-19T21:11:33.416872+0000","last_undegraded":"2024-02-19T21:11:33.416872+0000","last_fullsized":"2024-02-19T21:11:33.416872+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:27.770501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:27.770501+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":396,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418564+0000","last_change":"2024-02-19T21:10:41.315331+0000","last_active":"2024-02-19T21:11:33.418564+0000","last_peered":"2024-02-19T21:11:33.418564+0000","last_clean":"2024-02-19T21:11:33.418564+0000","last_became_active":"2024-02-19T21:10:41.314967+0000","last_became_peered":"2024-02-19T21:10:41.314967+0000","last_unstale":"2024-02-19T21:11:33.418564+0000","last_undegraded":"2024-02-19T21:11:33.418564+0000","last_fullsized":"2024-02-19T21:11:33.418564+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:50.600659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:50.600659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"459'20","reported_seq":204,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412509+0000","last_change":"2024-02-19T21:10:59.685223+0000","last_active":"2024-02-19T21:11:33.412509+0000","last_peered":"2024-02-19T21:11:33.412509+0000","last_clean":"2024-02-19T21:11:33.412509+0000","last_became_active":"2024-02-19T21:08:04.267216+0000","last_became_peered":"2024-02-19T21:08:04.267216+0000","last_unstale":"2024-02-19T21:11:33.412509+0000","last_undegraded":"2024-02-19T21:11:33.412509+0000","last_fullsized":"2024-02-19T21:11:33.412509+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:10:59.685150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.685150+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":453,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418873+0000","last_change":"2024-02-19T21:09:40.986717+0000","last_active":"2024-02-19T21:11:33.418873+0000","last_peered":"2024-02-19T21:11:33.418873+0000","last_clean":"2024-02-19T21:11:33.418873+0000","last_became_active":"2024-02-19T21:08:04.266728+0000","last_became_peered":"2024-02-19T21:08:04.266728+0000","last_unstale":"2024-02-19T21:11:33.418873+0000","last_undegraded":"2024-02-19T21:11:33.418873+0000","last_fullsized":"2024-02-19T21:11:33.418873+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.986635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"433'9","reported_seq":571,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-19T21:11:34.291405+0000","last_change":"2024-02-19T21:10:40.399539+0000","last_active":"2024-02-19T21:11:34.291405+0000","last_peered":"2024-02-19T21:11:34.291405+0000","last_clean":"2024-02-19T21:11:34.291405+0000","last_became_active":"2024-02-19T21:10:40.310303+0000","last_became_peered":"2024-02-19T21:10:40.310303+0000","last_unstale":"2024-02-19T21:11:34.291405+0000","last_undegraded":"2024-02-19T21:11:34.291405+0000","last_fullsized":"2024-02-19T21:11:34.291405+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:09:26.916002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:26.916002+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":300,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.417128+0000","last_change":"2024-02-19T21:11:31.500613+0000","last_active":"2024-02-19T21:11:33.417128+0000","last_peered":"2024-02-19T21:11:33.417128+0000","last_clean":"2024-02-19T21:11:33.417128+0000","last_became_active":"2024-02-19T21:08:04.266747+0000","last_became_peered":"2024-02-19T21:08:04.266747+0000","last_unstale":"2024-02-19T21:11:33.417128+0000","last_undegraded":"2024-02-19T21:11:33.417128+0000","last_fullsized":"2024-02-19T21:11:33.417128+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:11:31.500511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.500511+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":261,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.413866+0000","last_change":"2024-02-19T21:11:01.379002+0000","last_active":"2024-02-19T21:11:32.413866+0000","last_peered":"2024-02-19T21:11:32.413866+0000","last_clean":"2024-02-19T21:11:32.413866+0000","last_became_active":"2024-02-19T21:06:40.219590+0000","last_became_peered":"2024-02-19T21:06:40.219590+0000","last_unstale":"2024-02-19T21:11:32.413866+0000","last_undegraded":"2024-02-19T21:11:32.413866+0000","last_fullsized":"2024-02-19T21:11:32.413866+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.378925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:55.202463+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.378925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"459'15","reported_seq":496,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.419403+0000","last_change":"2024-02-19T21:10:50.053312+0000","last_active":"2024-02-19T21:11:33.419403+0000","last_peered":"2024-02-19T21:11:33.419403+0000","last_clean":"2024-02-19T21:11:33.419403+0000","last_became_active":"2024-02-19T21:08:04.266103+0000","last_became_peered":"2024-02-19T21:08:04.266103+0000","last_unstale":"2024-02-19T21:11:33.419403+0000","last_undegraded":"2024-02-19T21:11:33.419403+0000","last_fullsized":"2024-02-19T21:11:33.419403+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:10:50.053229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.053229+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":47,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411850+0000","last_change":"2024-02-19T21:10:56.327430+0000","last_active":"2024-02-19T21:11:33.411850+0000","last_peered":"2024-02-19T21:11:33.411850+0000","last_clean":"2024-02-19T21:11:33.411850+0000","last_became_active":"2024-02-19T21:10:41.295618+0000","last_became_peered":"2024-02-19T21:10:41.295618+0000","last_unstale":"2024-02-19T21:11:33.411850+0000","last_undegraded":"2024-02-19T21:11:33.411850+0000","last_fullsized":"2024-02-19T21:11:33.411850+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:56.327321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:56.327321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":224,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-19T21:11:34.293735+0000","last_change":"2024-02-19T21:08:37.867477+0000","last_active":"2024-02-19T21:11:34.293735+0000","last_peered":"2024-02-19T21:11:34.293735+0000","last_clean":"2024-02-19T21:11:34.293735+0000","last_became_active":"2024-02-19T21:08:04.256045+0000","last_became_peered":"2024-02-19T21:08:04.256045+0000","last_unstale":"2024-02-19T21:11:34.293735+0000","last_undegraded":"2024-02-19T21:11:34.293735+0000","last_fullsized":"2024-02-19T21:11:34.293735+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'10","last_scrub_stamp":"2024-02-19T21:08:37.867405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:08:37.867405+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":42,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.413129+0000","last_change":"2024-02-19T21:10:41.312156+0000","last_active":"2024-02-19T21:11:33.413129+0000","last_peered":"2024-02-19T21:11:33.413129+0000","last_clean":"2024-02-19T21:11:33.413129+0000","last_became_active":"2024-02-19T21:10:41.310279+0000","last_became_peered":"2024-02-19T21:10:41.310279+0000","last_unstale":"2024-02-19T21:11:33.413129+0000","last_undegraded":"2024-02-19T21:11:33.413129+0000","last_fullsized":"2024-02-19T21:11:33.413129+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_clean_scrub_stamp":"2024-02-19T21:09:43.936153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":421,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.409368+0000","last_change":"2024-02-19T21:11:00.839892+0000","last_active":"2024-02-19T21:11:33.409368+0000","last_peered":"2024-02-19T21:11:33.409368+0000","last_clean":"2024-02-19T21:11:33.409368+0000","last_became_active":"2024-02-19T21:11:00.752591+0000","last_became_peered":"2024-02-19T21:11:00.752591+0000","last_unstale":"2024-02-19T21:11:33.409368+0000","last_undegraded":"2024-02-19T21:11:33.409368+0000","last_fullsized":"2024-02-19T21:11:33.409368+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-02-19T21:10:14.015412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:14.015412+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"459'15","reported_seq":170,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.411512+0000","last_change":"2024-02-19T21:09:40.676522+0000","last_active":"2024-02-19T21:11:32.411512+0000","last_peered":"2024-02-19T21:11:32.411512+0000","last_clean":"2024-02-19T21:11:32.411512+0000","last_became_active":"2024-02-19T21:08:26.235018+0000","last_became_peered":"2024-02-19T21:08:26.235018+0000","last_unstale":"2024-02-19T21:11:32.411512+0000","last_undegraded":"2024-02-19T21:11:32.411512+0000","last_fullsized":"2024-02-19T21:11:32.411512+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:09:40.676439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.676439+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":453,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.416805+0000","last_change":"2024-02-19T21:10:57.422515+0000","last_active":"2024-02-19T21:11:32.416805+0000","last_peered":"2024-02-19T21:11:32.416805+0000","last_clean":"2024-02-19T21:11:32.416805+0000","last_became_active":"2024-02-19T21:08:04.258699+0000","last_became_peered":"2024-02-19T21:08:04.258699+0000","last_unstale":"2024-02-19T21:11:32.416805+0000","last_undegraded":"2024-02-19T21:11:32.416805+0000","last_fullsized":"2024-02-19T21:11:32.416805+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.422323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.422323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412045+0000","last_change":"2024-02-19T21:08:12.856127+0000","last_active":"2024-02-19T21:11:33.412045+0000","last_peered":"2024-02-19T21:11:33.412045+0000","last_clean":"2024-02-19T21:11:33.412045+0000","last_became_active":"2024-02-19T21:08:04.264983+0000","last_became_peered":"2024-02-19T21:08:04.264983+0000","last_unstale":"2024-02-19T21:11:33.412045+0000","last_undegraded":"2024-02-19T21:11:33.412045+0000","last_fullsized":"2024-02-19T21:11:33.412045+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_clean_scrub_stamp":"2024-02-19T21:08:12.856015+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":261,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418561+0000","last_change":"2024-02-19T21:09:38.941012+0000","last_active":"2024-02-19T21:11:33.418561+0000","last_peered":"2024-02-19T21:11:33.418561+0000","last_clean":"2024-02-19T21:11:33.418561+0000","last_became_active":"2024-02-19T21:06:40.239199+0000","last_became_peered":"2024-02-19T21:06:40.239199+0000","last_unstale":"2024-02-19T21:11:33.418561+0000","last_undegraded":"2024-02-19T21:11:33.418561+0000","last_fullsized":"2024-02-19T21:11:33.418561+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:38.940935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:38.940935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":380,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.417151+0000","last_change":"2024-02-19T21:10:51.493283+0000","last_active":"2024-02-19T21:11:32.417151+0000","last_peered":"2024-02-19T21:11:32.417151+0000","last_clean":"2024-02-19T21:11:32.417151+0000","last_became_active":"2024-02-19T21:04:39.672228+0000","last_became_peered":"2024-02-19T21:04:39.672228+0000","last_unstale":"2024-02-19T21:11:32.417151+0000","last_undegraded":"2024-02-19T21:11:32.417151+0000","last_fullsized":"2024-02-19T21:11:32.417151+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.493206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.493206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"459'19","reported_seq":71,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411980+0000","last_change":"2024-02-19T21:11:32.351959+0000","last_active":"2024-02-19T21:11:33.411980+0000","last_peered":"2024-02-19T21:11:33.411980+0000","last_clean":"2024-02-19T21:11:33.411980+0000","last_became_active":"2024-02-19T21:10:41.297317+0000","last_became_peered":"2024-02-19T21:10:41.297317+0000","last_unstale":"2024-02-19T21:11:33.411980+0000","last_undegraded":"2024-02-19T21:11:33.411980+0000","last_fullsized":"2024-02-19T21:11:33.411980+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:11:32.351880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:32.351880+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":431,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.412483+0000","last_change":"2024-02-19T21:09:32.623020+0000","last_active":"2024-02-19T21:11:32.412483+0000","last_peered":"2024-02-19T21:11:32.412483+0000","last_clean":"2024-02-19T21:11:32.412483+0000","last_became_active":"2024-02-19T21:01:16.270549+0000","last_became_peered":"2024-02-19T21:01:16.270549+0000","last_unstale":"2024-02-19T21:11:32.412483+0000","last_undegraded":"2024-02-19T21:11:32.412483+0000","last_fullsized":"2024-02-19T21:11:32.412483+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:32.622939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:09:32.622939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":445,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.411926+0000","last_change":"2024-02-19T21:09:34.642491+0000","last_active":"2024-02-19T21:11:32.411926+0000","last_peered":"2024-02-19T21:11:32.411926+0000","last_clean":"2024-02-19T21:11:32.411926+0000","last_became_active":"2024-02-19T21:08:04.268847+0000","last_became_peered":"2024-02-19T21:08:04.268847+0000","last_unstale":"2024-02-19T21:11:32.411926+0000","last_undegraded":"2024-02-19T21:11:32.411926+0000","last_fullsized":"2024-02-19T21:11:32.411926+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:34.642432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:34.642432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":172,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.410774+0000","last_change":"2024-02-19T21:08:04.280460+0000","last_active":"2024-02-19T21:11:33.410774+0000","last_peered":"2024-02-19T21:11:33.410774+0000","last_clean":"2024-02-19T21:11:33.410774+0000","last_became_active":"2024-02-19T21:08:04.280233+0000","last_became_peered":"2024-02-19T21:08:04.280233+0000","last_unstale":"2024-02-19T21:11:33.410774+0000","last_undegraded":"2024-02-19T21:11:33.410774+0000","last_fullsized":"2024-02-19T21:11:33.410774+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:07:59.446267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:07:59.446267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412587+0000","last_change":"2024-02-19T21:11:07.599578+0000","last_active":"2024-02-19T21:11:33.412587+0000","last_peered":"2024-02-19T21:11:33.412587+0000","last_clean":"2024-02-19T21:11:33.412587+0000","last_became_active":"2024-02-19T21:08:04.258568+0000","last_became_peered":"2024-02-19T21:08:04.258568+0000","last_unstale":"2024-02-19T21:11:33.412587+0000","last_undegraded":"2024-02-19T21:11:33.412587+0000","last_fullsized":"2024-02-19T21:11:33.412587+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.599474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.599474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"459'8","reported_seq":55,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418048+0000","last_change":"2024-02-19T21:10:40.651979+0000","last_active":"2024-02-19T21:11:33.418048+0000","last_peered":"2024-02-19T21:11:33.418048+0000","last_clean":"2024-02-19T21:11:33.418048+0000","last_became_active":"2024-02-19T21:10:40.298769+0000","last_became_peered":"2024-02-19T21:10:40.298769+0000","last_unstale":"2024-02-19T21:11:33.418048+0000","last_undegraded":"2024-02-19T21:11:33.418048+0000","last_fullsized":"2024-02-19T21:11:33.418048+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:09:28.712037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.712037+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":435,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.419493+0000","last_change":"2024-02-19T21:09:42.000906+0000","last_active":"2024-02-19T21:11:33.419493+0000","last_peered":"2024-02-19T21:11:33.419493+0000","last_clean":"2024-02-19T21:11:33.419493+0000","last_became_active":"2024-02-19T21:01:16.276381+0000","last_became_peered":"2024-02-19T21:01:16.276381+0000","last_unstale":"2024-02-19T21:11:33.419493+0000","last_undegraded":"2024-02-19T21:11:33.419493+0000","last_fullsized":"2024-02-19T21:11:33.419493+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:42.000850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:42.000850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":449,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.415176+0000","last_change":"2024-02-19T21:10:52.467547+0000","last_active":"2024-02-19T21:11:32.415176+0000","last_peered":"2024-02-19T21:11:32.415176+0000","last_clean":"2024-02-19T21:11:32.415176+0000","last_became_active":"2024-02-19T21:10:41.311989+0000","last_became_peered":"2024-02-19T21:10:41.311989+0000","last_unstale":"2024-02-19T21:11:32.415176+0000","last_undegraded":"2024-02-19T21:11:32.415176+0000","last_fullsized":"2024-02-19T21:11:32.415176+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.467476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.467476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":445,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.415833+0000","last_change":"2024-02-19T21:10:41.314970+0000","last_active":"2024-02-19T21:11:33.415833+0000","last_peered":"2024-02-19T21:11:33.415833+0000","last_clean":"2024-02-19T21:11:33.415833+0000","last_became_active":"2024-02-19T21:10:41.314589+0000","last_became_peered":"2024-02-19T21:10:41.314589+0000","last_unstale":"2024-02-19T21:11:33.415833+0000","last_undegraded":"2024-02-19T21:11:33.415833+0000","last_fullsized":"2024-02-19T21:11:33.415833+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:26.731860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:26.731860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412528+0000","last_change":"2024-02-19T21:08:10.824761+0000","last_active":"2024-02-19T21:11:33.412528+0000","last_peered":"2024-02-19T21:11:33.412528+0000","last_clean":"2024-02-19T21:11:33.412528+0000","last_became_active":"2024-02-19T21:08:04.266161+0000","last_became_peered":"2024-02-19T21:08:04.266161+0000","last_unstale":"2024-02-19T21:11:33.412528+0000","last_undegraded":"2024-02-19T21:11:33.412528+0000","last_fullsized":"2024-02-19T21:11:33.412528+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:10.824694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:10.824694+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"459'19","reported_seq":201,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.410970+0000","last_change":"2024-02-19T21:08:11.828624+0000","last_active":"2024-02-19T21:11:33.410970+0000","last_peered":"2024-02-19T21:11:33.410970+0000","last_clean":"2024-02-19T21:11:33.410970+0000","last_became_active":"2024-02-19T21:08:04.250742+0000","last_became_peered":"2024-02-19T21:08:04.250742+0000","last_unstale":"2024-02-19T21:11:33.410970+0000","last_undegraded":"2024-02-19T21:11:33.410970+0000","last_fullsized":"2024-02-19T21:11:33.410970+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:08:11.828537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:11.828537+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":435,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.416821+0000","last_change":"2024-02-19T21:11:00.383023+0000","last_active":"2024-02-19T21:11:32.416821+0000","last_peered":"2024-02-19T21:11:32.416821+0000","last_clean":"2024-02-19T21:11:32.416821+0000","last_became_active":"2024-02-19T21:01:16.270262+0000","last_became_peered":"2024-02-19T21:01:16.270262+0000","last_unstale":"2024-02-19T21:11:32.416821+0000","last_undegraded":"2024-02-19T21:11:32.416821+0000","last_fullsized":"2024-02-19T21:11:32.416821+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:00.382946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.382946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":442,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.416386+0000","last_change":"2024-02-19T21:10:41.307832+0000","last_active":"2024-02-19T21:11:33.416386+0000","last_peered":"2024-02-19T21:11:33.416386+0000","last_clean":"2024-02-19T21:11:33.416386+0000","last_became_active":"2024-02-19T21:10:41.307598+0000","last_became_peered":"2024-02-19T21:10:41.307598+0000","last_unstale":"2024-02-19T21:11:33.416386+0000","last_undegraded":"2024-02-19T21:11:33.416386+0000","last_fullsized":"2024-02-19T21:11:33.416386+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:29.783609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:29.783609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412134+0000","last_change":"2024-02-19T21:10:00.547305+0000","last_active":"2024-02-19T21:11:33.412134+0000","last_peered":"2024-02-19T21:11:33.412134+0000","last_clean":"2024-02-19T21:11:33.412134+0000","last_became_active":"2024-02-19T21:08:04.251987+0000","last_became_peered":"2024-02-19T21:08:04.251987+0000","last_unstale":"2024-02-19T21:11:33.412134+0000","last_undegraded":"2024-02-19T21:11:33.412134+0000","last_fullsized":"2024-02-19T21:11:33.412134+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:00.547228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:00.547228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":261,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.417399+0000","last_change":"2024-02-19T21:10:16.529711+0000","last_active":"2024-02-19T21:11:32.417399+0000","last_peered":"2024-02-19T21:11:32.417399+0000","last_clean":"2024-02-19T21:11:32.417399+0000","last_became_active":"2024-02-19T21:06:40.226196+0000","last_became_peered":"2024-02-19T21:06:40.226196+0000","last_unstale":"2024-02-19T21:11:32.417399+0000","last_undegraded":"2024-02-19T21:11:32.417399+0000","last_fullsized":"2024-02-19T21:11:32.417399+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:16.529623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:10:16.529623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"459'20","reported_seq":344,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.417252+0000","last_change":"2024-02-19T21:09:30.782123+0000","last_active":"2024-02-19T21:11:33.417252+0000","last_peered":"2024-02-19T21:11:33.417252+0000","last_clean":"2024-02-19T21:11:33.417252+0000","last_became_active":"2024-02-19T21:08:04.267222+0000","last_became_peered":"2024-02-19T21:08:04.267222+0000","last_unstale":"2024-02-19T21:11:33.417252+0000","last_undegraded":"2024-02-19T21:11:33.417252+0000","last_fullsized":"2024-02-19T21:11:33.417252+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:09:30.782002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:30.782002+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":448,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418448+0000","last_change":"2024-02-19T21:10:41.294958+0000","last_active":"2024-02-19T21:11:33.418448+0000","last_peered":"2024-02-19T21:11:33.418448+0000","last_clean":"2024-02-19T21:11:33.418448+0000","last_became_active":"2024-02-19T21:10:41.294592+0000","last_became_peered":"2024-02-19T21:10:41.294592+0000","last_unstale":"2024-02-19T21:11:33.418448+0000","last_undegraded":"2024-02-19T21:11:33.418448+0000","last_fullsized":"2024-02-19T21:11:33.418448+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:52.579178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:08:52.579178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"434'10","reported_seq":593,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-02-19T21:11:34.307409+0000","last_change":"2024-02-19T21:10:59.405489+0000","last_active":"2024-02-19T21:11:34.307409+0000","last_peered":"2024-02-19T21:11:34.307409+0000","last_clean":"2024-02-19T21:11:34.307409+0000","last_became_active":"2024-02-19T21:10:41.302307+0000","last_became_peered":"2024-02-19T21:10:41.302307+0000","last_unstale":"2024-02-19T21:11:34.307409+0000","last_undegraded":"2024-02-19T21:11:34.307409+0000","last_fullsized":"2024-02-19T21:11:34.307409+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'10","last_scrub_stamp":"2024-02-19T21:10:59.405417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.405417+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":429,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.416616+0000","last_change":"2024-02-19T21:09:41.951642+0000","last_active":"2024-02-19T21:11:33.416616+0000","last_peered":"2024-02-19T21:11:33.416616+0000","last_clean":"2024-02-19T21:11:33.416616+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:11:33.416616+0000","last_undegraded":"2024-02-19T21:11:33.416616+0000","last_fullsized":"2024-02-19T21:11:33.416616+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:09:41.951576+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:09:41.951576+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":380,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.409914+0000","last_change":"2024-02-19T21:08:39.107325+0000","last_active":"2024-02-19T21:11:33.409914+0000","last_peered":"2024-02-19T21:11:33.409914+0000","last_clean":"2024-02-19T21:11:33.409914+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:11:33.409914+0000","last_undegraded":"2024-02-19T21:11:33.409914+0000","last_fullsized":"2024-02-19T21:11:33.409914+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:39.107240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:08:39.107240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"459'19","reported_seq":464,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.417106+0000","last_change":"2024-02-19T21:10:54.435913+0000","last_active":"2024-02-19T21:11:32.417106+0000","last_peered":"2024-02-19T21:11:32.417106+0000","last_clean":"2024-02-19T21:11:32.417106+0000","last_became_active":"2024-02-19T21:01:18.280034+0000","last_became_peered":"2024-02-19T21:01:18.280034+0000","last_unstale":"2024-02-19T21:11:32.417106+0000","last_undegraded":"2024-02-19T21:11:32.417106+0000","last_fullsized":"2024-02-19T21:11:32.417106+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:10:54.435839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.435839+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.410920+0000","last_change":"2024-02-19T21:11:32.600294+0000","last_active":"2024-02-19T21:11:33.410920+0000","last_peered":"2024-02-19T21:11:33.410920+0000","last_clean":"2024-02-19T21:11:33.410920+0000","last_became_active":"2024-02-19T21:08:04.251241+0000","last_became_peered":"2024-02-19T21:08:04.251241+0000","last_unstale":"2024-02-19T21:11:33.410920+0000","last_undegraded":"2024-02-19T21:11:33.410920+0000","last_fullsized":"2024-02-19T21:11:33.410920+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:32.600213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:32.600213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":265,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.417317+0000","last_change":"2024-02-19T21:11:01.692337+0000","last_active":"2024-02-19T21:11:33.417317+0000","last_peered":"2024-02-19T21:11:33.417317+0000","last_clean":"2024-02-19T21:11:33.417317+0000","last_became_active":"2024-02-19T21:06:40.234026+0000","last_became_peered":"2024-02-19T21:06:40.234026+0000","last_unstale":"2024-02-19T21:11:33.417317+0000","last_undegraded":"2024-02-19T21:11:33.417317+0000","last_fullsized":"2024-02-19T21:11:33.417317+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.692260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.692260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":438,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.412376+0000","last_change":"2024-02-19T21:10:41.313086+0000","last_active":"2024-02-19T21:11:32.412376+0000","last_peered":"2024-02-19T21:11:32.412376+0000","last_clean":"2024-02-19T21:11:32.412376+0000","last_became_active":"2024-02-19T21:10:41.312577+0000","last_became_peered":"2024-02-19T21:10:41.312577+0000","last_unstale":"2024-02-19T21:11:32.412376+0000","last_undegraded":"2024-02-19T21:11:32.412376+0000","last_fullsized":"2024-02-19T21:11:32.412376+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:08:48.760625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:08:48.760625+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":397,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.412782+0000","last_change":"2024-02-19T21:10:59.782435+0000","last_active":"2024-02-19T21:11:32.412782+0000","last_peered":"2024-02-19T21:11:32.412782+0000","last_clean":"2024-02-19T21:11:32.412782+0000","last_became_active":"2024-02-19T21:10:41.311050+0000","last_became_peered":"2024-02-19T21:10:41.311050+0000","last_unstale":"2024-02-19T21:11:32.412782+0000","last_undegraded":"2024-02-19T21:11:32.412782+0000","last_fullsized":"2024-02-19T21:11:32.412782+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:59.782367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.782367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"459'34","reported_seq":532,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412151+0000","last_change":"2024-02-19T21:10:19.109353+0000","last_active":"2024-02-19T21:11:33.412151+0000","last_peered":"2024-02-19T21:11:33.412151+0000","last_clean":"2024-02-19T21:11:33.412151+0000","last_became_active":"2024-02-19T21:08:04.248378+0000","last_became_peered":"2024-02-19T21:08:04.248378+0000","last_unstale":"2024-02-19T21:11:33.412151+0000","last_undegraded":"2024-02-19T21:11:33.412151+0000","last_fullsized":"2024-02-19T21:11:33.412151+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'22","last_scrub_stamp":"2024-02-19T21:10:19.109244+0000","last_deep_scrub":"419'22","last_deep_scrub_stamp":"2024-02-19T21:10:19.109244+0000","last_clean_scrub_stamp":"2024-02-19T21:10:19.109244+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":453,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.414982+0000","last_change":"2024-02-19T21:11:03.379435+0000","last_active":"2024-02-19T21:11:32.414982+0000","last_peered":"2024-02-19T21:11:32.414982+0000","last_clean":"2024-02-19T21:11:32.414982+0000","last_became_active":"2024-02-19T21:10:41.311203+0000","last_became_peered":"2024-02-19T21:10:41.311203+0000","last_unstale":"2024-02-19T21:11:32.414982+0000","last_undegraded":"2024-02-19T21:11:32.414982+0000","last_fullsized":"2024-02-19T21:11:32.414982+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.379335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.379335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":445,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.412570+0000","last_change":"2024-02-19T21:10:53.709519+0000","last_active":"2024-02-19T21:11:32.412570+0000","last_peered":"2024-02-19T21:11:32.412570+0000","last_clean":"2024-02-19T21:11:32.412570+0000","last_became_active":"2024-02-19T21:10:41.312789+0000","last_became_peered":"2024-02-19T21:10:41.312789+0000","last_unstale":"2024-02-19T21:11:32.412570+0000","last_undegraded":"2024-02-19T21:11:32.412570+0000","last_fullsized":"2024-02-19T21:11:32.412570+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:53.709456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.709456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":47,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412623+0000","last_change":"2024-02-19T21:10:57.335233+0000","last_active":"2024-02-19T21:11:33.412623+0000","last_peered":"2024-02-19T21:11:33.412623+0000","last_clean":"2024-02-19T21:11:33.412623+0000","last_became_active":"2024-02-19T21:10:41.313042+0000","last_became_peered":"2024-02-19T21:10:41.313042+0000","last_unstale":"2024-02-19T21:11:33.412623+0000","last_undegraded":"2024-02-19T21:11:33.412623+0000","last_fullsized":"2024-02-19T21:11:33.412623+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.335152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.335152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":261,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.413779+0000","last_change":"2024-02-19T21:11:02.342277+0000","last_active":"2024-02-19T21:11:32.413779+0000","last_peered":"2024-02-19T21:11:32.413779+0000","last_clean":"2024-02-19T21:11:32.413779+0000","last_became_active":"2024-02-19T21:06:40.240859+0000","last_became_peered":"2024-02-19T21:06:40.240859+0000","last_unstale":"2024-02-19T21:11:32.413779+0000","last_undegraded":"2024-02-19T21:11:32.413779+0000","last_fullsized":"2024-02-19T21:11:32.413779+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.342173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.342173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"459'18","reported_seq":240,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412747+0000","last_change":"2024-02-19T21:10:41.664579+0000","last_active":"2024-02-19T21:11:33.412747+0000","last_peered":"2024-02-19T21:11:33.412747+0000","last_clean":"2024-02-19T21:11:33.412747+0000","last_became_active":"2024-02-19T21:10:41.307690+0000","last_became_peered":"2024-02-19T21:10:41.307690+0000","last_unstale":"2024-02-19T21:11:33.412747+0000","last_undegraded":"2024-02-19T21:11:33.412747+0000","last_fullsized":"2024-02-19T21:11:33.412747+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2024-02-19T21:08:49.869588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:08:49.869588+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412183+0000","last_change":"2024-02-19T21:10:04.546052+0000","last_active":"2024-02-19T21:11:33.412183+0000","last_peered":"2024-02-19T21:11:33.412183+0000","last_clean":"2024-02-19T21:11:33.412183+0000","last_became_active":"2024-02-19T21:08:04.253058+0000","last_became_peered":"2024-02-19T21:08:04.253058+0000","last_unstale":"2024-02-19T21:11:33.412183+0000","last_undegraded":"2024-02-19T21:11:33.412183+0000","last_fullsized":"2024-02-19T21:11:33.412183+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:04.545988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:04.545988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":42,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412410+0000","last_change":"2024-02-19T21:10:41.303228+0000","last_active":"2024-02-19T21:11:33.412410+0000","last_peered":"2024-02-19T21:11:33.412410+0000","last_clean":"2024-02-19T21:11:33.412410+0000","last_became_active":"2024-02-19T21:10:41.296849+0000","last_became_peered":"2024-02-19T21:10:41.296849+0000","last_unstale":"2024-02-19T21:11:33.412410+0000","last_undegraded":"2024-02-19T21:11:33.412410+0000","last_fullsized":"2024-02-19T21:11:33.412410+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:35.626548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:35.626548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412621+0000","last_change":"2024-02-19T21:10:58.688232+0000","last_active":"2024-02-19T21:11:33.412621+0000","last_peered":"2024-02-19T21:11:33.412621+0000","last_clean":"2024-02-19T21:11:33.412621+0000","last_became_active":"2024-02-19T21:08:04.278045+0000","last_became_peered":"2024-02-19T21:08:04.278045+0000","last_unstale":"2024-02-19T21:11:33.412621+0000","last_undegraded":"2024-02-19T21:11:33.412621+0000","last_fullsized":"2024-02-19T21:11:33.412621+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:58.688166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.688166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":379,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.411518+0000","last_change":"2024-02-19T21:11:00.754397+0000","last_active":"2024-02-19T21:11:32.411518+0000","last_peered":"2024-02-19T21:11:32.411518+0000","last_clean":"2024-02-19T21:11:32.411518+0000","last_became_active":"2024-02-19T21:04:39.777442+0000","last_became_peered":"2024-02-19T21:04:39.777442+0000","last_unstale":"2024-02-19T21:11:32.411518+0000","last_undegraded":"2024-02-19T21:11:32.411518+0000","last_fullsized":"2024-02-19T21:11:32.411518+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:00.754327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.754327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"459'19","reported_seq":61,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412022+0000","last_change":"2024-02-19T21:10:41.325454+0000","last_active":"2024-02-19T21:11:33.412022+0000","last_peered":"2024-02-19T21:11:33.412022+0000","last_clean":"2024-02-19T21:11:33.412022+0000","last_became_active":"2024-02-19T21:10:41.295673+0000","last_became_peered":"2024-02-19T21:10:41.295673+0000","last_unstale":"2024-02-19T21:11:33.412022+0000","last_undegraded":"2024-02-19T21:11:33.412022+0000","last_fullsized":"2024-02-19T21:11:33.412022+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'13","last_scrub_stamp":"2024-02-19T21:10:08.605366+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:10:08.605366+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":433,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.416434+0000","last_change":"2024-02-19T21:11:30.472665+0000","last_active":"2024-02-19T21:11:33.416434+0000","last_peered":"2024-02-19T21:11:33.416434+0000","last_clean":"2024-02-19T21:11:33.416434+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:11:33.416434+0000","last_undegraded":"2024-02-19T21:11:33.416434+0000","last_fullsized":"2024-02-19T21:11:33.416434+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:30.472591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:30.472591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":426,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418817+0000","last_change":"2024-02-19T21:09:29.852892+0000","last_active":"2024-02-19T21:11:33.418817+0000","last_peered":"2024-02-19T21:11:33.418817+0000","last_clean":"2024-02-19T21:11:33.418817+0000","last_became_active":"2024-02-19T21:01:20.293097+0000","last_became_peered":"2024-02-19T21:01:20.293097+0000","last_unstale":"2024-02-19T21:11:33.418817+0000","last_undegraded":"2024-02-19T21:11:33.418817+0000","last_fullsized":"2024-02-19T21:11:33.418817+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:29.852792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:29.852792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":257,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.410644+0000","last_change":"2024-02-19T21:10:10.642674+0000","last_active":"2024-02-19T21:11:32.410644+0000","last_peered":"2024-02-19T21:11:32.410644+0000","last_clean":"2024-02-19T21:11:32.410644+0000","last_became_active":"2024-02-19T21:06:40.239603+0000","last_became_peered":"2024-02-19T21:06:40.239603+0000","last_unstale":"2024-02-19T21:11:32.410644+0000","last_undegraded":"2024-02-19T21:11:32.410644+0000","last_fullsized":"2024-02-19T21:11:32.410644+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:10.642603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:10:10.642603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":400,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411453+0000","last_change":"2024-02-19T21:10:41.307666+0000","last_active":"2024-02-19T21:11:33.411453+0000","last_peered":"2024-02-19T21:11:33.411453+0000","last_clean":"2024-02-19T21:11:33.411453+0000","last_became_active":"2024-02-19T21:10:41.307352+0000","last_became_peered":"2024-02-19T21:10:41.307352+0000","last_unstale":"2024-02-19T21:11:33.411453+0000","last_undegraded":"2024-02-19T21:11:33.411453+0000","last_fullsized":"2024-02-19T21:11:33.411453+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:21.194709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:21.194709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"459'7","reported_seq":51,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411970+0000","last_change":"2024-02-19T21:10:41.767925+0000","last_active":"2024-02-19T21:11:33.411970+0000","last_peered":"2024-02-19T21:11:33.411970+0000","last_clean":"2024-02-19T21:11:33.411970+0000","last_became_active":"2024-02-19T21:10:41.297295+0000","last_became_peered":"2024-02-19T21:10:41.297295+0000","last_unstale":"2024-02-19T21:11:33.411970+0000","last_undegraded":"2024-02-19T21:11:33.411970+0000","last_fullsized":"2024-02-19T21:11:33.411970+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:09:44.596326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:44.596326+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":433,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.416975+0000","last_change":"2024-02-19T21:09:25.733055+0000","last_active":"2024-02-19T21:11:33.416975+0000","last_peered":"2024-02-19T21:11:33.416975+0000","last_clean":"2024-02-19T21:11:33.416975+0000","last_became_active":"2024-02-19T21:01:16.273756+0000","last_became_peered":"2024-02-19T21:01:16.273756+0000","last_unstale":"2024-02-19T21:11:33.416975+0000","last_undegraded":"2024-02-19T21:11:33.416975+0000","last_fullsized":"2024-02-19T21:11:33.416975+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:25.732955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:25.732955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":447,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.415927+0000","last_change":"2024-02-19T21:10:41.307635+0000","last_active":"2024-02-19T21:11:33.415927+0000","last_peered":"2024-02-19T21:11:33.415927+0000","last_clean":"2024-02-19T21:11:33.415927+0000","last_became_active":"2024-02-19T21:10:41.307400+0000","last_became_peered":"2024-02-19T21:10:41.307400+0000","last_unstale":"2024-02-19T21:11:33.415927+0000","last_undegraded":"2024-02-19T21:11:33.415927+0000","last_fullsized":"2024-02-19T21:11:33.415927+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:36.862297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:36.862297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":447,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.418306+0000","last_change":"2024-02-19T21:10:15.161470+0000","last_active":"2024-02-19T21:11:33.418306+0000","last_peered":"2024-02-19T21:11:33.418306+0000","last_clean":"2024-02-19T21:11:33.418306+0000","last_became_active":"2024-02-19T21:08:04.266100+0000","last_became_peered":"2024-02-19T21:08:04.266100+0000","last_unstale":"2024-02-19T21:11:33.418306+0000","last_undegraded":"2024-02-19T21:11:33.418306+0000","last_fullsized":"2024-02-19T21:11:33.418306+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:15.161397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:15.161397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":266,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.419323+0000","last_change":"2024-02-19T21:10:56.083471+0000","last_active":"2024-02-19T21:11:33.419323+0000","last_peered":"2024-02-19T21:11:33.419323+0000","last_clean":"2024-02-19T21:11:33.419323+0000","last_became_active":"2024-02-19T21:06:40.227713+0000","last_became_peered":"2024-02-19T21:06:40.227713+0000","last_unstale":"2024-02-19T21:11:33.419323+0000","last_undegraded":"2024-02-19T21:11:33.419323+0000","last_fullsized":"2024-02-19T21:11:33.419323+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:56.083403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:56.083403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"459'15","reported_seq":66,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411523+0000","last_change":"2024-02-19T21:11:03.423846+0000","last_active":"2024-02-19T21:11:33.411523+0000","last_peered":"2024-02-19T21:11:33.411523+0000","last_clean":"2024-02-19T21:11:33.411523+0000","last_became_active":"2024-02-19T21:10:41.298840+0000","last_became_peered":"2024-02-19T21:10:41.298840+0000","last_unstale":"2024-02-19T21:11:33.411523+0000","last_undegraded":"2024-02-19T21:11:33.411523+0000","last_fullsized":"2024-02-19T21:11:33.411523+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:11:03.423767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.423767+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.413008+0000","last_change":"2024-02-19T21:10:50.769725+0000","last_active":"2024-02-19T21:11:33.413008+0000","last_peered":"2024-02-19T21:11:33.413008+0000","last_clean":"2024-02-19T21:11:33.413008+0000","last_became_active":"2024-02-19T21:08:04.277576+0000","last_became_peered":"2024-02-19T21:08:04.277576+0000","last_unstale":"2024-02-19T21:11:33.413008+0000","last_undegraded":"2024-02-19T21:11:33.413008+0000","last_fullsized":"2024-02-19T21:11:33.413008+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:50.769620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.769620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":427,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.411387+0000","last_change":"2024-02-19T21:10:18.737245+0000","last_active":"2024-02-19T21:11:32.411387+0000","last_peered":"2024-02-19T21:11:32.411387+0000","last_clean":"2024-02-19T21:11:32.411387+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:11:32.411387+0000","last_undegraded":"2024-02-19T21:11:32.411387+0000","last_fullsized":"2024-02-19T21:11:32.411387+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:18.737176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:10:18.737176+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":42,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.412048+0000","last_change":"2024-02-19T21:10:41.309872+0000","last_active":"2024-02-19T21:11:33.412048+0000","last_peered":"2024-02-19T21:11:33.412048+0000","last_clean":"2024-02-19T21:11:33.412048+0000","last_became_active":"2024-02-19T21:10:41.309550+0000","last_became_peered":"2024-02-19T21:10:41.309550+0000","last_unstale":"2024-02-19T21:11:33.412048+0000","last_undegraded":"2024-02-19T21:11:33.412048+0000","last_fullsized":"2024-02-19T21:11:33.412048+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:05.182671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:05.182671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":401,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.419440+0000","last_change":"2024-02-19T21:11:08.043134+0000","last_active":"2024-02-19T21:11:33.419440+0000","last_peered":"2024-02-19T21:11:33.419440+0000","last_clean":"2024-02-19T21:11:33.419440+0000","last_became_active":"2024-02-19T21:10:41.315817+0000","last_became_peered":"2024-02-19T21:10:41.315817+0000","last_unstale":"2024-02-19T21:11:33.419440+0000","last_undegraded":"2024-02-19T21:11:33.419440+0000","last_fullsized":"2024-02-19T21:11:33.419440+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:08.043028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:08.043028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"459'27","reported_seq":211,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.410719+0000","last_change":"2024-02-19T21:10:02.532862+0000","last_active":"2024-02-19T21:11:33.410719+0000","last_peered":"2024-02-19T21:11:33.410719+0000","last_clean":"2024-02-19T21:11:33.410719+0000","last_became_active":"2024-02-19T21:08:04.249443+0000","last_became_peered":"2024-02-19T21:08:04.249443+0000","last_unstale":"2024-02-19T21:11:33.410719+0000","last_undegraded":"2024-02-19T21:11:33.410719+0000","last_fullsized":"2024-02-19T21:11:33.410719+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:10:02.532799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:02.532799+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":285,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.417409+0000","last_change":"2024-02-19T21:11:02.720658+0000","last_active":"2024-02-19T21:11:33.417409+0000","last_peered":"2024-02-19T21:11:33.417409+0000","last_clean":"2024-02-19T21:11:33.417409+0000","last_became_active":"2024-02-19T21:10:41.315636+0000","last_became_peered":"2024-02-19T21:10:41.315636+0000","last_unstale":"2024-02-19T21:11:33.417409+0000","last_undegraded":"2024-02-19T21:11:33.417409+0000","last_fullsized":"2024-02-19T21:11:33.417409+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.720583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.720583+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":47,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411667+0000","last_change":"2024-02-19T21:11:02.407624+0000","last_active":"2024-02-19T21:11:33.411667+0000","last_peered":"2024-02-19T21:11:33.411667+0000","last_clean":"2024-02-19T21:11:33.411667+0000","last_became_active":"2024-02-19T21:10:41.297843+0000","last_became_peered":"2024-02-19T21:10:41.297843+0000","last_unstale":"2024-02-19T21:11:33.411667+0000","last_undegraded":"2024-02-19T21:11:33.411667+0000","last_fullsized":"2024-02-19T21:11:33.411667+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.407543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.407543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":177,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-02-19T21:11:33.411967+0000","last_change":"2024-02-19T21:09:33.559448+0000","last_active":"2024-02-19T21:11:33.411967+0000","last_peered":"2024-02-19T21:11:33.411967+0000","last_clean":"2024-02-19T21:11:33.411967+0000","last_became_active":"2024-02-19T21:08:04.252640+0000","last_became_peered":"2024-02-19T21:08:04.252640+0000","last_unstale":"2024-02-19T21:11:33.411967+0000","last_undegraded":"2024-02-19T21:11:33.411967+0000","last_fullsized":"2024-02-19T21:11:33.411967+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:33.559363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:33.559363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":396,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-02-19T21:11:32.416996+0000","last_change":"2024-02-19T21:10:11.412403+0000","last_active":"2024-02-19T21:11:32.416996+0000","last_peered":"2024-02-19T21:11:32.416996+0000","last_clean":"2024-02-19T21:11:32.416996+0000","last_became_active":"2024-02-19T21:08:04.250937+0000","last_became_peered":"2024-02-19T21:08:04.250937+0000","last_unstale":"2024-02-19T21:11:32.416996+0000","last_undegraded":"2024-02-19T21:11:32.416996+0000","last_fullsized":"2024-02-19T21:11:32.416996+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:11.412276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:11.412276+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":219,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":4001598,"num_objects":171,"num_object_clones":9,"num_object_copies":513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":111,"num_read_kb":10,"num_write":626,"num_write_kb":4524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":740,"num_bytes_recovered":1812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14053376,"data_stored":12004827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1957,"ondisk_log_size":1957,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1105,"num_read_kb":862,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":245,"num_bytes_recovered":5204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":708,"ondisk_log_size":708,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":610,"num_read_kb":605,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":10,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266814,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316616,"kb_used_data":11192,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93428536,"statfs":{"total":95995035648,"available":95670820864,"internally_reserved":0,"allocated":11460608,"data_stored":8286822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:11:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.47799999999999998,"15min":0.86599999999999999},"min":{"1min":0.35599999999999998,"5min":0.223,"15min":0.215},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":89.456999999999994},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47199999999999998,"15min":0.78300000000000003},"min":{"1min":0.32300000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.81299999999999994,"5min":0.96999999999999997,"15min":70.451999999999998},"last":0.46400000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:11:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.48799999999999999,"15min":0.69899999999999995},"min":{"1min":0.29999999999999999,"5min":0.28299999999999997,"15min":0.28199999999999997},"max":{"1min":0.71699999999999997,"5min":0.85099999999999998,"15min":34.273000000000003},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46600000000000003,"15min":0.93300000000000005},"min":{"1min":0.30099999999999999,"5min":0.28100000000000003,"15min":0.254},"max":{"1min":0.73999999999999999,"5min":0.79200000000000004,"15min":89.594999999999999},"last":0.57599999999999996}]},{"osd":2,"last update":"Mon Feb 19 21:11:26 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47199999999999998,"15min":0.93899999999999995},"min":{"1min":0.312,"5min":0.28100000000000003,"15min":0.24299999999999999},"max":{"1min":0.753,"5min":0.753,"15min":104.413},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.499,"15min":0.94899999999999995},"min":{"1min":0.39800000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":104.268},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:11:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52700000000000002,"15min":0.95599999999999996},"min":{"1min":0.38300000000000001,"5min":0.29099999999999998,"15min":0.254},"max":{"1min":0.77000000000000002,"5min":1.8939999999999999,"15min":105.3},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51300000000000001,"15min":0.59999999999999998},"min":{"1min":0.35899999999999999,"5min":0.34300000000000003,"15min":0.315},"max":{"1min":0.78200000000000003,"5min":0.879,"15min":13.243},"last":0.63800000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:11:26 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.443,"15min":0.46700000000000003},"min":{"1min":0.29299999999999998,"5min":0.27400000000000002,"15min":0.24299999999999999},"max":{"1min":0.58099999999999996,"5min":0.69299999999999995,"15min":1.373},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.495,"5min":0.47499999999999998,"15min":0.47799999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":0.66300000000000003,"5min":0.751,"15min":1.0740000000000001},"last":0.65300000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:11:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46200000000000002,"15min":0.45800000000000002},"min":{"1min":0.29099999999999998,"5min":0.28599999999999998,"15min":0.252},"max":{"1min":0.68400000000000005,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.50600000000000001,"15min":0.50700000000000001},"min":{"1min":0.32600000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.85899999999999999},"last":0.625}]},{"osd":6,"last update":"Mon Feb 19 21:11:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.50800000000000001,"15min":0.51900000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.28799999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":1.0049999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.53800000000000003,"15min":0.55300000000000005},"min":{"1min":0.34999999999999998,"5min":0.33800000000000002,"15min":0.33700000000000002},"max":{"1min":0.749,"5min":0.75900000000000001,"15min":1.262},"last":0.66500000000000004}]}]},{"osd":6,"up_from":45,"seq":193273528452,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312680,"kb_used_data":7260,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93432472,"statfs":{"total":95995035648,"available":95674851328,"internally_reserved":0,"allocated":7434240,"data_stored":4285284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.49099999999999999,"15min":0.61399999999999999},"min":{"1min":0.26900000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.67700000000000005,"5min":2.9129999999999998,"15min":17.486999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52900000000000003,"15min":0.65400000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.64500000000000002,"5min":2.8999999999999999,"15min":17.43},"last":0.499}]},{"osd":1,"last update":"Mon Feb 19 21:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.502,"15min":0.61799999999999999},"min":{"1min":0.311,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.66700000000000004,"5min":2.0270000000000001,"15min":14.692},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.497,"15min":0.66200000000000003},"min":{"1min":0.318,"5min":0.29099999999999998,"15min":0.23400000000000001},"max":{"1min":0.68400000000000005,"5min":1.069,"15min":14.795},"last":0.46100000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.60399999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.73599999999999999,"5min":2.0979999999999999,"15min":6.931},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.66800000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67400000000000004,"5min":0.71899999999999997,"15min":14.391},"last":0.442}]},{"osd":3,"last update":"Mon Feb 19 21:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.54100000000000004,"15min":0.69899999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.29199999999999998},"max":{"1min":1.1499999999999999,"5min":3.0499999999999998,"15min":14.510999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.53700000000000003,"15min":0.65700000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.316},"max":{"1min":0.91600000000000004,"5min":1.927,"15min":17.585000000000001},"last":0.55100000000000005}]},{"osd":4,"last update":"Mon Feb 19 21:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.437,"15min":0.40500000000000003},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23200000000000001},"max":{"1min":2.9289999999999998,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.46200000000000002,"15min":0.42999999999999999},"min":{"1min":0.27000000000000002,"5min":0.25,"15min":0.22700000000000001},"max":{"1min":3.431,"5min":3.431,"15min":3.431},"last":0.28699999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.495,"15min":0.45000000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.9620000000000002,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.318},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.52800000000000002,"15min":0.48399999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.7010000000000001,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.33500000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:11:30 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.496,"15min":0.5},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.615,"5min":1.615,"15min":1.615},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48899999999999999,"15min":0.505},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.28399999999999997},"max":{"1min":0.60399999999999998,"5min":0.72399999999999998,"15min":1.534},"last":0.35599999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575003,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312484,"kb_used_data":11160,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93432668,"statfs":{"total":95995035648,"available":95675052032,"internally_reserved":0,"allocated":11427840,"data_stored":8287138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.32100000000000001,"15min":0.54400000000000004},"min":{"1min":0.20200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.46999999999999997,"5min":0.58499999999999996,"15min":27.777000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.33600000000000002,"15min":0.57099999999999995},"min":{"1min":0.216,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.55500000000000005,"5min":0.70899999999999996,"15min":27.631},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:11:34 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.36099999999999999,"15min":0.70699999999999996},"min":{"1min":0.26100000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.157,"5min":1.157,"15min":27.798999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.372,"15min":0.72099999999999997},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":27.606999999999999},"last":0.38800000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.36299999999999999,"15min":0.57299999999999995},"min":{"1min":0.251,"5min":0.217,"15min":0.217},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":15.574999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39300000000000002,"15min":0.66600000000000004},"min":{"1min":0.29199999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.64700000000000002,"5min":0.74399999999999999,"15min":27.82},"last":0.374}]},{"osd":4,"last update":"Mon Feb 19 21:11:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.51800000000000002,"15min":0.65800000000000003},"min":{"1min":0.36499999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.149,"5min":1.149,"15min":9.6430000000000007},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.52900000000000003,"15min":0.752},"min":{"1min":0.38400000000000001,"5min":0.308,"15min":0.308},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":27.707000000000001},"last":0.40500000000000003}]},{"osd":5,"last update":"Mon Feb 19 21:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52700000000000002,"15min":0.746},"min":{"1min":0.318,"5min":0.314,"15min":0.314},"max":{"1min":0.75700000000000001,"5min":0.94199999999999995,"15min":27.529},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53000000000000003,"15min":0.69099999999999995},"min":{"1min":0.38700000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":9.6799999999999997},"last":0.46700000000000003}]},{"osd":6,"last update":"Mon Feb 19 21:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53800000000000003,"15min":0.67000000000000004},"min":{"1min":0.35899999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":9.6989999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57599999999999996,"15min":0.78800000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81699999999999995,"5min":2.0470000000000002,"15min":27.733000000000001},"last":0.53100000000000003}]},{"osd":7,"last update":"Mon Feb 19 21:11:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54300000000000004,"15min":0.77000000000000002},"min":{"1min":0.371,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":27.576000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58499999999999996,"15min":0.80800000000000005},"min":{"1min":0.45900000000000002,"5min":0.375,"15min":0.375},"max":{"1min":1.1299999999999999,"5min":2.0169999999999999,"15min":27.756},"last":0.499}]}]},{"osd":0,"up_from":8,"seq":34359738529,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308508,"kb_used_data":7184,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436644,"statfs":{"total":95995035648,"available":95679123456,"internally_reserved":0,"allocated":7356416,"data_stored":4286225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:11:12 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.35699999999999998,"15min":0.60499999999999998},"min":{"1min":0.21299999999999999,"5min":0.20499999999999999,"15min":0.16200000000000001},"max":{"1min":0.49199999999999999,"5min":2.2189999999999999,"15min":20.966999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.40100000000000002,"15min":0.61399999999999999},"min":{"1min":0.23999999999999999,"5min":0.193,"15min":0.188},"max":{"1min":0.60799999999999998,"5min":4.8570000000000002,"15min":20.366},"last":0.91000000000000003}]},{"osd":2,"last update":"Mon Feb 19 21:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.47499999999999998,"15min":0.71399999999999997},"min":{"1min":0.23400000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.47699999999999998,"5min":4.4660000000000002,"15min":27.916},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.373,"5min":0.42899999999999999,"15min":0.63200000000000001},"min":{"1min":0.252,"5min":0.218,"15min":0.218},"max":{"1min":0.58199999999999996,"5min":4.2949999999999999,"15min":22.004000000000001},"last":0.51900000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:10:53 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.44,"15min":0.748},"min":{"1min":0.26800000000000002,"5min":0.248,"15min":0.23200000000000001},"max":{"1min":0.5,"5min":5.9569999999999999,"15min":25.669},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.46200000000000002,"15min":0.69499999999999995},"min":{"1min":0.26800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.55000000000000004,"5min":4.1029999999999998,"15min":25.006},"last":0.29199999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:11:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54700000000000004,"15min":0.71999999999999997},"min":{"1min":0.32500000000000001,"5min":0.32100000000000001,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.3899999999999999,"15min":20.411999999999999},"last":0.878},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.59499999999999997,"15min":0.75},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.83299999999999996,"5min":4.3730000000000002,"15min":19.986999999999998},"last":2.093}]},{"osd":5,"last update":"Mon Feb 19 21:10:37 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.61299999999999999,"15min":0.82899999999999996},"min":{"1min":0.46400000000000002,"5min":0.35499999999999998,"15min":0.27200000000000002},"max":{"1min":0.79900000000000004,"5min":4.4349999999999996,"15min":20.021999999999998},"last":1.3560000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.67100000000000004,"15min":0.89500000000000002},"min":{"1min":0.46700000000000003,"5min":0.317,"15min":0.30399999999999999},"max":{"1min":0.79100000000000004,"5min":6.2859999999999996,"15min":20.908999999999999},"last":0.59199999999999997}]},{"osd":6,"last update":"Mon Feb 19 21:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.60699999999999998,"15min":0.76200000000000001},"min":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.33900000000000002},"max":{"1min":0.78900000000000003,"5min":2.3279999999999998,"15min":20.445},"last":2.1459999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.65300000000000002,"15min":0.81200000000000006},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.33400000000000002},"max":{"1min":0.72599999999999998,"5min":4.4080000000000004,"15min":20.047999999999998},"last":1.026}]},{"osd":7,"last update":"Mon Feb 19 21:11:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.62,"15min":0.86199999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.82399999999999995,"5min":3.9700000000000002,"15min":20.940999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.59199999999999997,"15min":0.81699999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.71899999999999997,"5min":1.4510000000000001,"15min":20.471},"last":1.034}]}]},{"osd":2,"up_from":18,"seq":77309411478,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312472,"kb_used_data":11148,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93432680,"statfs":{"total":95995035648,"available":95675064320,"internally_reserved":0,"allocated":11415552,"data_stored":8286078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:11:29 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.374,"15min":0.53700000000000003},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.45100000000000001,"5min":3.593,"15min":26.690999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.45100000000000001,"15min":0.58999999999999997},"min":{"1min":0.23400000000000001,"5min":0.23200000000000001,"15min":0.214},"max":{"1min":0.44600000000000001,"5min":7.3849999999999998,"15min":24.315000000000001},"last":0.30099999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:11:29 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.42799999999999999,"15min":0.70599999999999996},"min":{"1min":0.20699999999999999,"5min":0.20599999999999999,"15min":0.20100000000000001},"max":{"1min":0.59399999999999997,"5min":7.0010000000000003,"15min":28.882000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.41699999999999998,"15min":0.68300000000000005},"min":{"1min":0.252,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":0.77600000000000002,"5min":3.3010000000000002,"15min":28.382999999999999},"last":0.33800000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.88100000000000001,"15min":0.72099999999999997},"min":{"1min":0.27100000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.66800000000000004,"5min":28.742000000000001,"15min":28.742000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.45300000000000001,"15min":0.64200000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.25700000000000001},"max":{"1min":1.0009999999999999,"5min":3.6499999999999999,"15min":26.719999999999999},"last":0.38900000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:11:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.60599999999999998,"15min":0.70199999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.30599999999999999},"max":{"1min":0.83999999999999997,"5min":3.4039999999999999,"15min":24.356000000000002},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.67300000000000004,"15min":0.72799999999999998},"min":{"1min":0.38900000000000001,"5min":0.32800000000000001,"15min":0.32400000000000001},"max":{"1min":0.79600000000000004,"5min":7.1340000000000003,"15min":20.707000000000001},"last":0.56999999999999995}]},{"osd":5,"last update":"Mon Feb 19 21:10:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.73299999999999998,"15min":0.69499999999999995},"min":{"1min":0.36399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.71599999999999997,"5min":9.9499999999999993,"15min":9.9499999999999993},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.67300000000000004,"15min":0.79900000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.38400000000000001},"max":{"1min":0.75800000000000001,"5min":4.9359999999999999,"15min":26.606000000000002},"last":0.69499999999999995}]},{"osd":6,"last update":"Mon Feb 19 21:11:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.83599999999999997,"15min":0.80600000000000005},"min":{"1min":0.35299999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81200000000000006,"5min":20.617000000000001,"15min":24.381},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.88600000000000001,"15min":0.74099999999999999},"min":{"1min":0.38800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.79800000000000004,"5min":28.706,"15min":28.706},"last":0.67300000000000004}]},{"osd":7,"last update":"Mon Feb 19 21:11:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.629,"15min":0.77700000000000002},"min":{"1min":0.435,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81200000000000006,"5min":3.3359999999999999,"15min":26.658000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60599999999999998,"15min":0.73399999999999999},"min":{"1min":0.39800000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86599999999999999,"5min":1.5109999999999999,"15min":24.402000000000001},"last":0.70999999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182546,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304436,"kb_used_data":7200,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93440716,"statfs":{"total":95995035648,"available":95683293184,"internally_reserved":0,"allocated":7372800,"data_stored":4285853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.33200000000000002,"15min":0.71699999999999997},"min":{"1min":0.219,"5min":0.19400000000000001,"15min":0.156},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":77.662000000000006},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.34399999999999997,"15min":0.502},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.19700000000000001},"max":{"1min":0.60999999999999999,"5min":0.72999999999999998,"15min":17.456},"last":0.33900000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40400000000000003,"15min":0.53300000000000003},"min":{"1min":0.27500000000000002,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":0.58299999999999996,"5min":6.9299999999999997,"15min":17.327999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.46100000000000002,"15min":0.51100000000000001},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.62,"5min":7.3620000000000001,"15min":15.004},"last":0.307}]},{"osd":2,"last update":"Mon Feb 19 21:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40699999999999997,"15min":0.66900000000000004},"min":{"1min":0.26400000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.65100000000000002,"5min":3.9319999999999999,"15min":56.890999999999998},"last":0.378},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.36599999999999999,"15min":0.65200000000000002},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.20899999999999999},"max":{"1min":0.59499999999999997,"5min":0.91900000000000004,"15min":44.600999999999999},"last":0.38900000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.56899999999999995,"15min":0.63600000000000001},"min":{"1min":0.32100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81499999999999995,"5min":3.9729999999999999,"15min":12.847},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.71299999999999997},"min":{"1min":0.373,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.94299999999999995,"5min":1.046,"15min":26.393999999999998},"last":0.433}]},{"osd":5,"last update":"Mon Feb 19 21:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.62,"15min":0.69199999999999995},"min":{"1min":0.373,"5min":0.313,"15min":0.313},"max":{"1min":0.93500000000000005,"5min":9.2360000000000007,"15min":17.401},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.58399999999999996,"15min":0.65900000000000003},"min":{"1min":0.35499999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.83199999999999996,"5min":4.0890000000000004,"15min":12.882},"last":0.497}]},{"osd":6,"last update":"Mon Feb 19 21:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56499999999999995,"15min":0.65800000000000003},"min":{"1min":0.41599999999999998,"5min":0.34799999999999998,"15min":0.34300000000000003},"max":{"1min":0.82599999999999996,"5min":1.9790000000000001,"15min":17.367000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.59499999999999997,"15min":0.66900000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":0.92600000000000005,"5min":4.024,"15min":12.904},"last":0.45000000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:11:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.56799999999999995,"15min":0.65700000000000003},"min":{"1min":0.40600000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72599999999999998,"5min":0.84199999999999997,"15min":17.428999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58099999999999996,"15min":0.80200000000000005},"min":{"1min":0.45200000000000001,"5min":0.38400000000000001,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":0.997,"15min":44.378},"last":0.47099999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986318,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308600,"kb_used_data":7272,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93436552,"statfs":{"total":95995035648,"available":95679029248,"internally_reserved":0,"allocated":7446528,"data_stored":4286093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.53300000000000003,"15min":0.64300000000000002},"min":{"1min":0.32100000000000001,"5min":0.23599999999999999,"15min":0.22500000000000001},"max":{"1min":8.0530000000000008,"5min":8.0530000000000008,"15min":19.878},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.48699999999999999,"15min":0.86399999999999999},"min":{"1min":0.26700000000000002,"5min":0.215,"15min":0.20499999999999999},"max":{"1min":3.3530000000000002,"5min":3.3530000000000002,"15min":70.063000000000002},"last":0.59799999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.46500000000000002,"15min":0.64200000000000002},"min":{"1min":0.29399999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.72599999999999998,"5min":0.76200000000000001,"15min":26.510999999999999},"last":0.499},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.48099999999999998,"15min":0.68700000000000006},"min":{"1min":0.32900000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.68700000000000006,"5min":0.82999999999999996,"15min":25.949000000000002},"last":0.56599999999999995}]},{"osd":2,"last update":"Mon Feb 19 21:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.57699999999999996,"15min":0.72299999999999998},"min":{"1min":0.29399999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":8.1240000000000006,"5min":8.1240000000000006,"15min":37.320999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.50800000000000001,"15min":1.0660000000000001},"min":{"1min":0.30599999999999999,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":2.9969999999999999,"5min":2.9969999999999999,"15min":69.936999999999998},"last":0.50700000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:11:19 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.53000000000000003,"15min":0.91300000000000003},"min":{"1min":0.28999999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.323,"5min":3.323,"15min":69.793000000000006},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51400000000000001,"15min":0.76700000000000002},"min":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.29099999999999998},"max":{"1min":0.69599999999999995,"5min":0.81799999999999995,"15min":17.986999999999998},"last":0.53600000000000003}]},{"osd":5,"last update":"Mon Feb 19 21:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39700000000000002,"15min":0.42199999999999999},"min":{"1min":0.24299999999999999,"5min":0.23999999999999999,"15min":0.218},"max":{"1min":0.71199999999999997,"5min":0.75,"15min":3.2799999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.45300000000000001,"15min":0.45600000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.23899999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":4.4340000000000002},"last":0.40400000000000003}]},{"osd":6,"last update":"Mon Feb 19 21:11:05 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.54700000000000004,"15min":0.45800000000000002},"min":{"1min":0.27800000000000002,"5min":0.27600000000000002,"15min":0.24399999999999999},"max":{"1min":8.0969999999999995,"5min":8.0969999999999995,"15min":8.0969999999999995},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.54000000000000004,"15min":0.495},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":3.0270000000000001,"5min":3.0270000000000001,"15min":4.5609999999999999},"last":0.57799999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:10:36 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51900000000000002,"15min":0.48899999999999999},"min":{"1min":0.33600000000000002,"5min":0.314,"15min":0.29999999999999999},"max":{"1min":0.84999999999999998,"5min":1.004,"15min":6.0730000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.52800000000000002,"15min":0.5},"min":{"1min":0.36399999999999999,"5min":0.31,"15min":0.26400000000000001},"max":{"1min":0.81999999999999995,"5min":1.0209999999999999,"15min":3.202},"last":0.60299999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790089,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316796,"kb_used_data":7280,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93428356,"statfs":{"total":95995035648,"available":95670636544,"internally_reserved":0,"allocated":7454720,"data_stored":4276204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:10:41 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.45200000000000001,"15min":1.8859999999999999},"min":{"1min":0.24099999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":276.47899999999998},"last":0.433},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.49099999999999999,"15min":0.91200000000000003},"min":{"1min":0.23799999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.59799999999999998,"5min":0.95399999999999996,"15min":52.887},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:10:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.50700000000000001,"15min":1.278},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.23499999999999999},"max":{"1min":0.69699999999999995,"5min":3.3159999999999998,"15min":120.011},"last":0.307},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.51500000000000001,"15min":1.0269999999999999},"min":{"1min":0.32700000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.64400000000000002,"5min":1.7470000000000001,"15min":118.105},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:10:41 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.52100000000000002,"15min":1.0009999999999999},"min":{"1min":0.33800000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.629,"5min":3.2130000000000001,"15min":82.242000000000004},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.56000000000000005,"15min":1.5089999999999999},"min":{"1min":0.32300000000000001,"5min":0.309,"15min":0.30199999999999999},"max":{"1min":0.63400000000000001,"5min":4.5019999999999998,"15min":232.84999999999999},"last":0.47399999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:10:41 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.54900000000000004,"15min":1.071},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":0.68200000000000005,"5min":4.4640000000000004,"15min":109.70099999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.52500000000000002,"15min":0.91100000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67200000000000004,"5min":2.6019999999999999,"15min":65.519000000000005},"last":0.51000000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:10:41 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.45300000000000001,"15min":0.41699999999999998},"min":{"1min":0.32000000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61099999999999999,"5min":1.861,"15min":1.861},"last":0.499},{"interface":"front","average":{"1min":0.442,"5min":0.46500000000000002,"15min":0.42699999999999999},"min":{"1min":0.31,"5min":0.28199999999999997,"15min":0.248},"max":{"1min":0.64700000000000002,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":0.54100000000000004}]},{"osd":6,"last update":"Mon Feb 19 21:10:57 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.48399999999999999,"15min":0.44900000000000001},"min":{"1min":0.28999999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.871,"5min":1.823,"15min":1.823},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48299999999999998,"15min":0.45800000000000002},"min":{"1min":0.30399999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69899999999999995,"5min":0.71399999999999997,"15min":0.73399999999999999},"last":0.51700000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:10:37 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51500000000000001,"15min":0.504},"min":{"1min":0.35499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65700000000000003,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.51100000000000001,"15min":0.50700000000000001},"min":{"1min":0.34699999999999998,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.66000000000000003,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.44700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4243456,"data_stored":4000588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4251648,"data_stored":4000582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":290816,"data_stored":676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4000683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:11:41.517 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-19T21:11:41.518 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool set .rgw.root pg_num 42 2024-02-19T21:11:41.519 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:11:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:41.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:11:41.569 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:40.198148+0000 mgr.y (mgr.14550) 433 : audit [DBG] from='client.23262 v1:172.21.15.163:0/3682509244' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:11:41.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:40.480985+0000 mgr.y (mgr.14550) 434 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:41.624 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:41.655 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:41.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.758+0000 7f19b8f0c700 1 -- 172.21.15.163:0/1563820429 >> v1:172.21.15.163:6790/0 conn(0x7f19b4100780 legacy=0x7f19b4102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:41.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.758+0000 7f19b8f0c700 1 -- 172.21.15.163:0/1563820429 shutdown_connections 2024-02-19T21:11:41.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.758+0000 7f19b8f0c700 1 -- 172.21.15.163:0/1563820429 >> 172.21.15.163:0/1563820429 conn(0x7f19b40f35a0 msgr2=0x7f19b40f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:41.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.758+0000 7f19b8f0c700 1 -- 172.21.15.163:0/1563820429 shutdown_connections 2024-02-19T21:11:41.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.758+0000 7f19b8f0c700 1 -- 172.21.15.163:0/1563820429 wait complete. 2024-02-19T21:11:41.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.759+0000 7f19b8f0c700 1 librados: starting msgr at 2024-02-19T21:11:41.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.759+0000 7f19b8f0c700 1 librados: starting objecter 2024-02-19T21:11:41.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.759+0000 7f19b8f0c700 1 Processor -- start 2024-02-19T21:11:41.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.759+0000 7f19b8f0c700 1 -- start start 2024-02-19T21:11:41.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.759+0000 7f19b8f0c700 1 librados: setting wanted keys 2024-02-19T21:11:41.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.759+0000 7f19b8f0c700 1 librados: calling monclient init 2024-02-19T21:11:41.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.759+0000 7f19b8f0c700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19b4117280 con 0x7f19b4100780 2024-02-19T21:11:41.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.759+0000 7f19b8f0c700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19b4118460 con 0x7f19b40fd980 2024-02-19T21:11:41.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.759+0000 7f19b8f0c700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f19b4119660 con 0x7f19b41042c0 2024-02-19T21:11:41.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.759+0000 7f19b1d9b700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f19b4100780 0x7f19b40fa070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:39706/0 (socket says 172.21.15.163:39706) 2024-02-19T21:11:41.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.759+0000 7f19b1d9b700 1 -- 172.21.15.163:0/3432772882 learned_addr learned my addr 172.21.15.163:0/3432772882 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:41.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 10 client.?.objecter ms_handle_connect 0x7f19b4100780 2024-02-19T21:11:41.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:41.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 10 client.?.objecter ms_handle_connect 0x7f19b40fd980 2024-02-19T21:11:41.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:41.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 10 client.?.objecter ms_handle_connect 0x7f19b41042c0 2024-02-19T21:11:41.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:41.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615162148 0 0) 0x7f19b4118460 con 0x7f19b40fd980 2024-02-19T21:11:41.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f199c003800 con 0x7f19b40fd980 2024-02-19T21:11:41.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570687378 0 0) 0x7f19b4117280 con 0x7f19b4100780 2024-02-19T21:11:41.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b4118460 con 0x7f19b4100780 2024-02-19T21:11:41.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018496890 0 0) 0x7f19b4119660 con 0x7f19b41042c0 2024-02-19T21:11:41.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b4117280 con 0x7f19b41042c0 2024-02-19T21:11:41.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4105695652 0 0) 0x7f199c003800 con 0x7f19b40fd980 2024-02-19T21:11:41.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19b4119660 con 0x7f19b40fd980 2024-02-19T21:11:41.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f19a80018c0 con 0x7f19b40fd980 2024-02-19T21:11:41.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2798752907 0 0) 0x7f19b4118460 con 0x7f19b4100780 2024-02-19T21:11:41.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f199c003800 con 0x7f19b4100780 2024-02-19T21:11:41.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f19a4002ea0 con 0x7f19b4100780 2024-02-19T21:11:41.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1988656463 0 0) 0x7f19b4117280 con 0x7f19b41042c0 2024-02-19T21:11:41.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19b4118460 con 0x7f19b41042c0 2024-02-19T21:11:41.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f19ac001c50 con 0x7f19b41042c0 2024-02-19T21:11:41.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1753479329 0 0) 0x7f19b4119660 con 0x7f19b40fd980 2024-02-19T21:11:41.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 >> v1:172.21.15.163:6790/0 conn(0x7f19b41042c0 legacy=0x7f19b4115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:41.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 >> v1:172.21.15.163:6789/0 conn(0x7f19b4100780 legacy=0x7f19b40fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:41.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b411a860 con 0x7f19b40fd980 2024-02-19T21:11:41.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b4118650 con 0x7f19b40fd980 2024-02-19T21:11:41.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19b8f0c700 10 client.48805.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:41.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b4118b40 con 0x7f19b40fd980 2024-02-19T21:11:41.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19b8f0c700 1 librados: init done 2024-02-19T21:11:41.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f19a8001f80 con 0x7f19b40fd980 2024-02-19T21:11:41.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.760+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f19a8003240 con 0x7f19b40fd980 2024-02-19T21:11:41.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.761+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1994004f00 con 0x7f19b40fd980 2024-02-19T21:11:41.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f19a80033a0 con 0x7f19b40fd980 2024-02-19T21:11:41.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(471..471 src has 43..471) v4 ==== 8092+0+0 (unknown 2298721257 0 0) 0x7f19a8076310 con 0x7f19b40fd980 2024-02-19T21:11:41.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 osd_map(471..471 src has 43..471) v4 2024-02-19T21:11:41.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19a37fe700 3 client.48805.objecter handle_osd_map got epochs [471,471] > 0 2024-02-19T21:11:41.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19a37fe700 3 client.48805.objecter handle_osd_map decoding full epoch 471 2024-02-19T21:11:41.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19a37fe700 20 client.48805.objecter dump_active .. 0 homeless 2024-02-19T21:11:41.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v471) v1 ==== 54+0+16 (unknown 4271406700 0 2420000192) 0x7f19a8049890 con 0x7f19b40fd980 2024-02-19T21:11:41.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19a37fe700 10 client.48805.objecter ms_handle_connect 0x7f199c063cc0 2024-02-19T21:11:41.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:41.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 1 2024-02-19T21:11:41.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f19940051a0 osd=0 3 2024-02-19T21:11:41.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f19940051a0 osd=0 3 2024-02-19T21:11:41.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f19940051a0 osd=0 3 2024-02-19T21:11:41.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 1 2024-02-19T21:11:41.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f19940051a0 osd=0 3 2024-02-19T21:11:41.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 0 1 2024-02-19T21:11:41.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f19940051a0 osd=0 4 2024-02-19T21:11:41.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19b8f0c700 10 client.48805.objecter _send_command 1 2024-02-19T21:11:41.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.762+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1994004950 con 0x7f1994004530 2024-02-19T21:11:41.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.763+0000 7f19a37fe700 10 client.48805.objecter ms_handle_connect 0x7f1994004530 2024-02-19T21:11:41.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.763+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1994004950 con 0x7f1994004530 2024-02-19T21:11:41.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.763+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:11:41.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.763+0000 7f19a37fe700 10 client.48805.objecter _finish_command 1 = system:0 2024-02-19T21:11:41.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.763+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f19940051a0 osd=0 4 2024-02-19T21:11:41.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.763+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 0 1 2024-02-19T21:11:41.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.778+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:41.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.778+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 2 2024-02-19T21:11:41.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.778+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f19940051a0 osd=0 3 2024-02-19T21:11:41.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.778+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f19940051a0 osd=0 3 2024-02-19T21:11:41.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.778+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f19940051a0 osd=0 3 2024-02-19T21:11:41.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.778+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 2 2024-02-19T21:11:41.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.778+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f19940051a0 osd=0 3 2024-02-19T21:11:41.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.778+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 0 2 2024-02-19T21:11:41.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.778+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f19940051a0 osd=0 4 2024-02-19T21:11:41.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.778+0000 7f19b8f0c700 10 client.48805.objecter _send_command 2 2024-02-19T21:11:41.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.778+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1994004d30 con 0x7f1994004530 2024-02-19T21:11:41.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1994004d30 con 0x7f1994004530 2024-02-19T21:11:41.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:41.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19a37fe700 10 client.48805.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:41.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f19940051a0 osd=0 4 2024-02-19T21:11:41.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 0 2 2024-02-19T21:11:41.780 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:11:41.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:41.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 3 2024-02-19T21:11:41.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994003fe0 osd=1 3 2024-02-19T21:11:41.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994003fe0 osd=1 3 2024-02-19T21:11:41.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994003fe0 osd=1 3 2024-02-19T21:11:41.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 3 2024-02-19T21:11:41.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f1994003fe0 osd=1 3 2024-02-19T21:11:41.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 1 3 2024-02-19T21:11:41.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994003fe0 osd=1 4 2024-02-19T21:11:41.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19b8f0c700 10 client.48805.objecter _send_command 3 2024-02-19T21:11:41.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f199400c000 con 0x7f1994004150 2024-02-19T21:11:41.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.780+0000 7f19a37fe700 10 client.48805.objecter ms_handle_connect 0x7f1994004150 2024-02-19T21:11:41.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.781+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f199400c000 con 0x7f1994004150 2024-02-19T21:11:41.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.781+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:11:41.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.781+0000 7f19a37fe700 10 client.48805.objecter _finish_command 3 = system:0 2024-02-19T21:11:41.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.781+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f1994003fe0 osd=1 4 2024-02-19T21:11:41.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.781+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 1 3 2024-02-19T21:11:41.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.795+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:41.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.795+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 4 2024-02-19T21:11:41.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.795+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994003fe0 osd=1 3 2024-02-19T21:11:41.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.795+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994003fe0 osd=1 3 2024-02-19T21:11:41.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.795+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994003fe0 osd=1 3 2024-02-19T21:11:41.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.795+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 4 2024-02-19T21:11:41.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.795+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f1994003fe0 osd=1 3 2024-02-19T21:11:41.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.795+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 1 4 2024-02-19T21:11:41.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.795+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994003fe0 osd=1 4 2024-02-19T21:11:41.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.795+0000 7f19b8f0c700 10 client.48805.objecter _send_command 4 2024-02-19T21:11:41.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.795+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1994001850 con 0x7f1994004150 2024-02-19T21:11:41.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1994001850 con 0x7f1994004150 2024-02-19T21:11:41.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:41.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19a37fe700 10 client.48805.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:41.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f1994003fe0 osd=1 4 2024-02-19T21:11:41.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 1 4 2024-02-19T21:11:41.796 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:11:41.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:41.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 5 2024-02-19T21:11:41.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994003630 osd=2 3 2024-02-19T21:11:41.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994003630 osd=2 3 2024-02-19T21:11:41.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994003630 osd=2 3 2024-02-19T21:11:41.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 5 2024-02-19T21:11:41.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f1994003630 osd=2 3 2024-02-19T21:11:41.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 2 5 2024-02-19T21:11:41.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994003630 osd=2 4 2024-02-19T21:11:41.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19b8f0c700 10 client.48805.objecter _send_command 5 2024-02-19T21:11:41.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1994003e70 con 0x7f199400c170 2024-02-19T21:11:41.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.797+0000 7f19a37fe700 10 client.48805.objecter ms_handle_connect 0x7f199400c170 2024-02-19T21:11:41.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.798+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1994003e70 con 0x7f199400c170 2024-02-19T21:11:41.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.798+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:11:41.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.798+0000 7f19a37fe700 10 client.48805.objecter _finish_command 5 = system:0 2024-02-19T21:11:41.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.798+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f1994003630 osd=2 4 2024-02-19T21:11:41.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.798+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 2 5 2024-02-19T21:11:41.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.812+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:41.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.812+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 6 2024-02-19T21:11:41.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.812+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994003630 osd=2 3 2024-02-19T21:11:41.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.812+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994003630 osd=2 3 2024-02-19T21:11:41.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.812+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994003630 osd=2 3 2024-02-19T21:11:41.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.812+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 6 2024-02-19T21:11:41.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.812+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f1994003630 osd=2 3 2024-02-19T21:11:41.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.812+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 2 6 2024-02-19T21:11:41.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.812+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994003630 osd=2 4 2024-02-19T21:11:41.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.812+0000 7f19b8f0c700 10 client.48805.objecter _send_command 6 2024-02-19T21:11:41.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.812+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f199400d5b0 con 0x7f199400c170 2024-02-19T21:11:41.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f199400d5b0 con 0x7f199400c170 2024-02-19T21:11:41.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:41.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19a37fe700 10 client.48805.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:41.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f1994003630 osd=2 4 2024-02-19T21:11:41.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 2 6 2024-02-19T21:11:41.814 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:11:41.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:41.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 7 2024-02-19T21:11:41.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994003470 osd=3 3 2024-02-19T21:11:41.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994003470 osd=3 3 2024-02-19T21:11:41.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994003470 osd=3 3 2024-02-19T21:11:41.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 7 2024-02-19T21:11:41.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f1994003470 osd=3 3 2024-02-19T21:11:41.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 3 7 2024-02-19T21:11:41.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994003470 osd=3 4 2024-02-19T21:11:41.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19b8f0c700 10 client.48805.objecter _send_command 7 2024-02-19T21:11:41.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1994019f90 con 0x7f1994016290 2024-02-19T21:11:41.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.814+0000 7f19a37fe700 10 client.48805.objecter ms_handle_connect 0x7f1994016290 2024-02-19T21:11:41.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.815+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1994019f90 con 0x7f1994016290 2024-02-19T21:11:41.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.815+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:11:41.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.815+0000 7f19a37fe700 10 client.48805.objecter _finish_command 7 = system:0 2024-02-19T21:11:41.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.815+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f1994003470 osd=3 4 2024-02-19T21:11:41.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.815+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 3 7 2024-02-19T21:11:41.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.830+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:41.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.830+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 8 2024-02-19T21:11:41.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.830+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994003470 osd=3 3 2024-02-19T21:11:41.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.830+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994003470 osd=3 3 2024-02-19T21:11:41.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.830+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994003470 osd=3 3 2024-02-19T21:11:41.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.830+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 8 2024-02-19T21:11:41.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.830+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f1994003470 osd=3 3 2024-02-19T21:11:41.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.830+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 3 8 2024-02-19T21:11:41.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.830+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994003470 osd=3 4 2024-02-19T21:11:41.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.830+0000 7f19b8f0c700 10 client.48805.objecter _send_command 8 2024-02-19T21:11:41.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.830+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f199400d020 con 0x7f1994016290 2024-02-19T21:11:41.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f199400d020 con 0x7f1994016290 2024-02-19T21:11:41.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:41.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19a37fe700 10 client.48805.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:41.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f1994003470 osd=3 4 2024-02-19T21:11:41.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 3 8 2024-02-19T21:11:41.831 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:11:41.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:41.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 9 2024-02-19T21:11:41.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f199400d020 osd=4 3 2024-02-19T21:11:41.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f199400d020 osd=4 3 2024-02-19T21:11:41.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f199400d020 osd=4 3 2024-02-19T21:11:41.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 9 2024-02-19T21:11:41.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f199400d020 osd=4 3 2024-02-19T21:11:41.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 4 9 2024-02-19T21:11:41.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f199400d020 osd=4 4 2024-02-19T21:11:41.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19b8f0c700 10 client.48805.objecter _send_command 9 2024-02-19T21:11:41.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f199401a840 con 0x7f199400e460 2024-02-19T21:11:41.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.832+0000 7f19a37fe700 10 client.48805.objecter ms_handle_connect 0x7f199400e460 2024-02-19T21:11:41.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.833+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f199401a840 con 0x7f199400e460 2024-02-19T21:11:41.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.833+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:11:41.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.833+0000 7f19a37fe700 10 client.48805.objecter _finish_command 9 = system:0 2024-02-19T21:11:41.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.833+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f199400d020 osd=4 4 2024-02-19T21:11:41.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.833+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 4 9 2024-02-19T21:11:41.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.848+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:41.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.848+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 10 2024-02-19T21:11:41.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.848+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f199400d020 osd=4 3 2024-02-19T21:11:41.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.848+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f199400d020 osd=4 3 2024-02-19T21:11:41.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.848+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f199400d020 osd=4 3 2024-02-19T21:11:41.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.848+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 10 2024-02-19T21:11:41.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.848+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f199400d020 osd=4 3 2024-02-19T21:11:41.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.848+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 4 10 2024-02-19T21:11:41.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.848+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f199400d020 osd=4 4 2024-02-19T21:11:41.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.848+0000 7f19b8f0c700 10 client.48805.objecter _send_command 10 2024-02-19T21:11:41.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.848+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f199401a840 con 0x7f199400e460 2024-02-19T21:11:41.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f199401a840 con 0x7f199400e460 2024-02-19T21:11:41.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:41.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19a37fe700 10 client.48805.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:41.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f199400d020 osd=4 4 2024-02-19T21:11:41.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 4 10 2024-02-19T21:11:41.852 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:11:41.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:41.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 11 2024-02-19T21:11:41.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f199401a840 osd=5 3 2024-02-19T21:11:41.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f199401a840 osd=5 3 2024-02-19T21:11:41.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f199401a840 osd=5 3 2024-02-19T21:11:41.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 11 2024-02-19T21:11:41.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f199401a840 osd=5 3 2024-02-19T21:11:41.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 5 11 2024-02-19T21:11:41.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f199401a840 osd=5 4 2024-02-19T21:11:41.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19b8f0c700 10 client.48805.objecter _send_command 11 2024-02-19T21:11:41.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f199401c6f0 con 0x7f199401c100 2024-02-19T21:11:41.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.850+0000 7f19a37fe700 10 client.48805.objecter ms_handle_connect 0x7f199401c100 2024-02-19T21:11:41.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.851+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f199401c6f0 con 0x7f199401c100 2024-02-19T21:11:41.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.851+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:11:41.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.851+0000 7f19a37fe700 10 client.48805.objecter _finish_command 11 = system:0 2024-02-19T21:11:41.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.851+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f199401a840 osd=5 4 2024-02-19T21:11:41.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.851+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 5 11 2024-02-19T21:11:41.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.866+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:41.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.866+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 12 2024-02-19T21:11:41.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.866+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f199401a840 osd=5 3 2024-02-19T21:11:41.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.866+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f199401a840 osd=5 3 2024-02-19T21:11:41.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.866+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f199401a840 osd=5 3 2024-02-19T21:11:41.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.866+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 12 2024-02-19T21:11:41.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.866+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f199401a840 osd=5 3 2024-02-19T21:11:41.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.866+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 5 12 2024-02-19T21:11:41.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.866+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f199401a840 osd=5 4 2024-02-19T21:11:41.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.866+0000 7f19b8f0c700 10 client.48805.objecter _send_command 12 2024-02-19T21:11:41.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.866+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f199401c6f0 con 0x7f199401c100 2024-02-19T21:11:41.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.869+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f199401c6f0 con 0x7f199401c100 2024-02-19T21:11:41.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.869+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:41.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.869+0000 7f19a37fe700 10 client.48805.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:41.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.869+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f199401a840 osd=5 4 2024-02-19T21:11:41.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.869+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 5 12 2024-02-19T21:11:41.869 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:11:41.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.870+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:41.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.870+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 13 2024-02-19T21:11:41.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.870+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f199401c6f0 osd=6 3 2024-02-19T21:11:41.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.870+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f199401c6f0 osd=6 3 2024-02-19T21:11:41.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.870+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f199401c6f0 osd=6 3 2024-02-19T21:11:41.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.870+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 13 2024-02-19T21:11:41.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.870+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f199401c6f0 osd=6 3 2024-02-19T21:11:41.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.870+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 6 13 2024-02-19T21:11:41.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.870+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f199401c6f0 osd=6 4 2024-02-19T21:11:41.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.870+0000 7f19b8f0c700 10 client.48805.objecter _send_command 13 2024-02-19T21:11:41.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.870+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1994025290 con 0x7f1994023590 2024-02-19T21:11:41.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.870+0000 7f19a37fe700 10 client.48805.objecter ms_handle_connect 0x7f1994023590 2024-02-19T21:11:41.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.871+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1994025290 con 0x7f1994023590 2024-02-19T21:11:41.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.871+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:11:41.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.871+0000 7f19a37fe700 10 client.48805.objecter _finish_command 13 = system:0 2024-02-19T21:11:41.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.871+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f199401c6f0 osd=6 4 2024-02-19T21:11:41.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.871+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 6 13 2024-02-19T21:11:41.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.886+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:41.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.886+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 14 2024-02-19T21:11:41.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.886+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f199401c6f0 osd=6 3 2024-02-19T21:11:41.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.886+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f199401c6f0 osd=6 3 2024-02-19T21:11:41.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.886+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f199401c6f0 osd=6 3 2024-02-19T21:11:41.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.886+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 14 2024-02-19T21:11:41.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.886+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f199401c6f0 osd=6 3 2024-02-19T21:11:41.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.886+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 6 14 2024-02-19T21:11:41.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.886+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f199401c6f0 osd=6 4 2024-02-19T21:11:41.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.886+0000 7f19b8f0c700 10 client.48805.objecter _send_command 14 2024-02-19T21:11:41.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.886+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1994025290 con 0x7f1994023590 2024-02-19T21:11:41.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1994025290 con 0x7f1994023590 2024-02-19T21:11:41.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:41.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19a37fe700 10 client.48805.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:41.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f199401c6f0 osd=6 4 2024-02-19T21:11:41.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 6 14 2024-02-19T21:11:41.887 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:11:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 15 2024-02-19T21:11:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994025290 osd=7 3 2024-02-19T21:11:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994025290 osd=7 3 2024-02-19T21:11:41.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994025290 osd=7 3 2024-02-19T21:11:41.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 15 2024-02-19T21:11:41.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f1994025290 osd=7 3 2024-02-19T21:11:41.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 7 15 2024-02-19T21:11:41.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994025290 osd=7 4 2024-02-19T21:11:41.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19b8f0c700 10 client.48805.objecter _send_command 15 2024-02-19T21:11:41.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f199402b880 con 0x7f1994027d70 2024-02-19T21:11:41.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.888+0000 7f19a37fe700 10 client.48805.objecter ms_handle_connect 0x7f1994027d70 2024-02-19T21:11:41.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.889+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f199402b880 con 0x7f1994027d70 2024-02-19T21:11:41.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.889+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:11:41.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.889+0000 7f19a37fe700 10 client.48805.objecter _finish_command 15 = system:0 2024-02-19T21:11:41.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.889+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f1994025290 osd=7 4 2024-02-19T21:11:41.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.889+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 7 15 2024-02-19T21:11:41.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.903+0000 7f19b8f0c700 10 client.48805.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:41.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.903+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign -1 16 2024-02-19T21:11:41.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.903+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994025290 osd=7 3 2024-02-19T21:11:41.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.903+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994025290 osd=7 3 2024-02-19T21:11:41.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.903+0000 7f19b8f0c700 20 client.48805.objecter _get_session s=0x7f1994025290 osd=7 3 2024-02-19T21:11:41.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.903+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_remove -1 16 2024-02-19T21:11:41.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.903+0000 7f19b8f0c700 20 client.48805.objecter get_session s=0x7f1994025290 osd=7 3 2024-02-19T21:11:41.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.903+0000 7f19b8f0c700 15 client.48805.objecter _session_command_op_assign 7 16 2024-02-19T21:11:41.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.903+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994025290 osd=7 4 2024-02-19T21:11:41.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.903+0000 7f19b8f0c700 10 client.48805.objecter _send_command 16 2024-02-19T21:11:41.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.903+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f199402b880 con 0x7f1994027d70 2024-02-19T21:11:41.905 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:11:41.905 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-02-19T21:11:41.905 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-02-19T21:11:41.906 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-02-19T21:11:41.906 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.906 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35189.0:4 36.16 36:681f27cd:::2.obj:head [write 0~105] snapc 0=[] ondisk+write+known_if_redirected e103)", 2024-02-19T21:11:41.906 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:42.759848+0000", 2024-02-19T21:11:41.906 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 419.14429875799999, 2024-02-19T21:11:41.906 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.034936716, 2024-02-19T21:11:41.906 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:41.906 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:41.906 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:41.906 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35189", 2024-02-19T21:11:41.906 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3441459623", 2024-02-19T21:11:41.906 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 4 2024-02-19T21:11:41.907 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.907 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:41.907 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.907 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:41.907 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759848+0000", 2024-02-19T21:11:41.907 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:41.907 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.907 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.907 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:41.907 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759848+0000", 2024-02-19T21:11:41.907 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:41.907 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.907 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.907 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:41.907 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759846+0000", 2024-02-19T21:11:41.908 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8869999999999998e-06 2024-02-19T21:11:41.908 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.908 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.908 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:41.908 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759855+0000", 2024-02-19T21:11:41.908 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2349999999999999e-06 2024-02-19T21:11:41.908 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.908 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.908 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:41.908 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759863+0000", 2024-02-19T21:11:41.908 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2473999999999999e-05 2024-02-19T21:11:41.908 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.909 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.909 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:41.909 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759876+0000", 2024-02-19T21:11:41.909 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6888000000000003e-05 2024-02-19T21:11:41.909 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.909 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.909 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.909 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759913+0000", 2024-02-19T21:11:41.909 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5630000000000003e-06 2024-02-19T21:11:41.909 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.909 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.909 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:11:41.909 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759920+0000", 2024-02-19T21:11:41.909 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.99619900800000005 2024-02-19T21:11:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.756119+0000", 2024-02-19T21:11:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000116972 2024-02-19T21:11:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:41.910 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.756236+0000", 2024-02-19T21:11:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015304199999999999 2024-02-19T21:11:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,3", 2024-02-19T21:11:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.756389+0000", 2024-02-19T21:11:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011827809999999999 2024-02-19T21:11:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:11:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.768217+0000", 2024-02-19T21:11:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011764907 2024-02-19T21:11:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:41.911 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.779982+0000", 2024-02-19T21:11:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014710997999999999 2024-02-19T21:11:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.794693+0000", 2024-02-19T21:11:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1137999999999997e-05 2024-02-19T21:11:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.912 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.913 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:41.913 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.794744+0000", 2024-02-19T21:11:41.913 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0725000000000001e-05 2024-02-19T21:11:41.913 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.913 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.913 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:41.913 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.794785+0000", 2024-02-19T21:11:41.913 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.034936716 2024-02-19T21:11:41.913 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.913 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:41.913 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.913 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.913 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.913 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25213.0:3 34.9 34:9294e480:nspace::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e103)", 2024-02-19T21:11:41.914 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:42.759875+0000", 2024-02-19T21:11:41.914 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 419.14427171, 2024-02-19T21:11:41.914 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.053943584, 2024-02-19T21:11:41.914 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:41.914 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:41.914 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:41.914 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25213", 2024-02-19T21:11:41.914 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2094902030", 2024-02-19T21:11:41.914 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 3 2024-02-19T21:11:41.914 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.915 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:41.915 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.915 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:41.915 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759875+0000", 2024-02-19T21:11:41.915 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:41.915 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.915 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.915 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:41.915 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759875+0000", 2024-02-19T21:11:41.915 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:11:41.915 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.915 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.915 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:41.915 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759874+0000", 2024-02-19T21:11:41.916 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4439999999999999e-06 2024-02-19T21:11:41.916 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.916 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.916 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:41.916 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759879+0000", 2024-02-19T21:11:41.916 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8599999999999999e-06 2024-02-19T21:11:41.916 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.916 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.916 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:41.916 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759888+0000", 2024-02-19T21:11:41.916 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7299999999999996e-06 2024-02-19T21:11:41.916 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.916 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.917 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:41.917 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759893+0000", 2024-02-19T21:11:41.917 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1069000000000003e-05 2024-02-19T21:11:41.917 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.917 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.917 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.917 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759934+0000", 2024-02-19T21:11:41.917 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.422e-06 2024-02-19T21:11:41.917 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.917 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.917 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:11:41.917 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759937+0000", 2024-02-19T21:11:41.917 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0133646970000001 2024-02-19T21:11:41.917 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.918 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.918 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.918 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.773302+0000", 2024-02-19T21:11:41.918 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012570699999999999 2024-02-19T21:11:41.918 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.918 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.918 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:41.918 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.773427+0000", 2024-02-19T21:11:41.918 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000139443 2024-02-19T21:11:41.918 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.918 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.919 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,3", 2024-02-19T21:11:41.919 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.773567+0000", 2024-02-19T21:11:41.919 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0052079140000000001 2024-02-19T21:11:41.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.919 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.919 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:11:41.919 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.778775+0000", 2024-02-19T21:11:41.919 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.030842405999999999 2024-02-19T21:11:41.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.919 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.919 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:41.919 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.809617+0000", 2024-02-19T21:11:41.919 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0041308889999999996 2024-02-19T21:11:41.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.919 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.920 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:41.920 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.813748+0000", 2024-02-19T21:11:41.920 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3467999999999999e-05 2024-02-19T21:11:41.920 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.920 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.920 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:41.920 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.813782+0000", 2024-02-19T21:11:41.920 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6962999999999998e-05 2024-02-19T21:11:41.920 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.920 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.920 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:41.920 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.813819+0000", 2024-02-19T21:11:41.920 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.053943584 2024-02-19T21:11:41.921 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.921 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:41.921 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.921 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.921 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.921 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25105.0:3 40.6 40:602f83fe:::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:11:41.921 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:47.968492+0000", 2024-02-19T21:11:41.921 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 413.93565504200001, 2024-02-19T21:11:41.921 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.042153208999999997, 2024-02-19T21:11:41.922 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:41.922 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:41.922 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:41.922 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25105", 2024-02-19T21:11:41.922 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3624732563", 2024-02-19T21:11:41.922 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 3 2024-02-19T21:11:41.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.922 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:41.922 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.922 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:41.922 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968492+0000", 2024-02-19T21:11:41.922 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:41.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.922 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.923 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:41.923 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968492+0000", 2024-02-19T21:11:41.923 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:41.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.923 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.923 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:41.923 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968490+0000", 2024-02-19T21:11:41.923 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4610000000000008e-06 2024-02-19T21:11:41.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.923 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.923 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:41.923 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968498+0000", 2024-02-19T21:11:41.923 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4069999999999993e-06 2024-02-19T21:11:41.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.924 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.924 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:41.924 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968506+0000", 2024-02-19T21:11:41.924 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0049999999999998e-06 2024-02-19T21:11:41.924 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.924 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.924 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:41.924 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.968513+0000", 2024-02-19T21:11:41.924 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001534941 2024-02-19T21:11:41.924 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.924 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.924 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.925 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.970048+0000", 2024-02-19T21:11:41.925 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0030718680000000002 2024-02-19T21:11:41.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.925 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.925 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:11:41.925 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.973120+0000", 2024-02-19T21:11:41.925 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.023296391 2024-02-19T21:11:41.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.925 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.925 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.925 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.996417+0000", 2024-02-19T21:11:41.926 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5544000000000006e-05 2024-02-19T21:11:41.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.926 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:41.926 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.996502+0000", 2024-02-19T21:11:41.926 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000144523 2024-02-19T21:11:41.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.926 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-02-19T21:11:41.926 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.996647+0000", 2024-02-19T21:11:41.926 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00039452900000000002 2024-02-19T21:11:41.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.927 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:11:41.927 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.997041+0000", 2024-02-19T21:11:41.927 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0016365290000000001 2024-02-19T21:11:41.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.927 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:41.927 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.998678+0000", 2024-02-19T21:11:41.927 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011897256 2024-02-19T21:11:41.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.928 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:41.928 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010575+0000", 2024-02-19T21:11:41.928 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0837000000000003e-05 2024-02-19T21:11:41.928 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.928 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.928 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:41.928 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010606+0000", 2024-02-19T21:11:41.928 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9029000000000002e-05 2024-02-19T21:11:41.928 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.928 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.928 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:41.928 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010645+0000", 2024-02-19T21:11:41.928 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.042153208999999997 2024-02-19T21:11:41.928 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.929 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:41.929 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.929 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.929 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25321.0:2 49.15 49:ad551702:::test:head [create,setxattr key (5) in=3b] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:11:41.929 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:47.976371+0000", 2024-02-19T21:11:41.929 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 413.92777598100002, 2024-02-19T21:11:41.929 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037843445000000003, 2024-02-19T21:11:41.929 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:41.929 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:41.929 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:41.929 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25321", 2024-02-19T21:11:41.929 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/826688492", 2024-02-19T21:11:41.930 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:11:41.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.930 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:41.930 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.930 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:41.930 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976371+0000", 2024-02-19T21:11:41.930 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:41.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.930 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.930 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:41.930 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976371+0000", 2024-02-19T21:11:41.930 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:41.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.930 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:41.931 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976369+0000", 2024-02-19T21:11:41.931 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4760000000000004e-06 2024-02-19T21:11:41.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:41.931 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976378+0000", 2024-02-19T21:11:41.931 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1931000000000001e-05 2024-02-19T21:11:41.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:41.932 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976400+0000", 2024-02-19T21:11:41.932 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.422e-06 2024-02-19T21:11:41.932 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.932 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.932 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:41.932 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976408+0000", 2024-02-19T21:11:41.932 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7206000000000001e-05 2024-02-19T21:11:41.932 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.932 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.932 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.932 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976466+0000", 2024-02-19T21:11:41.932 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3210000000000005e-06 2024-02-19T21:11:41.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.933 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.933 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:11:41.933 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976474+0000", 2024-02-19T21:11:41.933 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.028525920999999999 2024-02-19T21:11:41.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.933 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.933 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.933 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.005000+0000", 2024-02-19T21:11:41.933 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011637399999999999 2024-02-19T21:11:41.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.933 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.933 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:41.933 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.005116+0000", 2024-02-19T21:11:41.933 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013954899999999999 2024-02-19T21:11:41.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.934 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.934 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-02-19T21:11:41.934 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.005256+0000", 2024-02-19T21:11:41.934 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00050348099999999996 2024-02-19T21:11:41.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.934 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.934 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:11:41.934 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.005759+0000", 2024-02-19T21:11:41.934 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0074804399999999997 2024-02-19T21:11:41.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.934 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.935 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:41.935 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.013240+0000", 2024-02-19T21:11:41.935 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00090275499999999999 2024-02-19T21:11:41.935 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.935 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.935 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:41.935 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014143+0000", 2024-02-19T21:11:41.935 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3145999999999997e-05 2024-02-19T21:11:41.935 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.935 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.935 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:41.935 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014176+0000", 2024-02-19T21:11:41.936 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8489999999999999e-05 2024-02-19T21:11:41.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.936 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.936 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:41.936 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.014214+0000", 2024-02-19T21:11:41.936 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037843445000000003 2024-02-19T21:11:41.936 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.936 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:41.936 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.936 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.936 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25105.0:4 40.6 40:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e108)", 2024-02-19T21:11:41.936 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:48.010891+0000", 2024-02-19T21:11:41.936 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 413.89325600400002, 2024-02-19T21:11:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010449923999999999, 2024-02-19T21:11:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:11:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25105", 2024-02-19T21:11:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3624732563", 2024-02-19T21:11:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 4 2024-02-19T21:11:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010891+0000", 2024-02-19T21:11:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.937 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.938 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:41.938 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010891+0000", 2024-02-19T21:11:41.938 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:11:41.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.938 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:41.938 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010889+0000", 2024-02-19T21:11:41.938 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3759999999999999e-06 2024-02-19T21:11:41.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.938 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:41.939 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010894+0000", 2024-02-19T21:11:41.939 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7290000000000006e-06 2024-02-19T21:11:41.939 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.939 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.939 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:41.939 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010903+0000", 2024-02-19T21:11:41.939 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9740000000000001e-06 2024-02-19T21:11:41.939 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.939 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.939 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:41.939 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010908+0000", 2024-02-19T21:11:41.939 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6906000000000002e-05 2024-02-19T21:11:41.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.940 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.940 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010945+0000", 2024-02-19T21:11:41.940 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5568000000000006e-05 2024-02-19T21:11:41.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.940 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:41.940 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.011011+0000", 2024-02-19T21:11:41.940 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010224851 2024-02-19T21:11:41.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.940 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021235+0000", 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8914e-05 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021284+0000", 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6223e-05 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021341+0000", 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010449923999999999 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.942 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.942 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25105.0:5 40.6 40:602f83fe:::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:11:41.942 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:48.010924+0000", 2024-02-19T21:11:41.942 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 413.89322283500002, 2024-02-19T21:11:41.942 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011507269000000001, 2024-02-19T21:11:41.942 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:41.942 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:41.942 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:41.943 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25105", 2024-02-19T21:11:41.943 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3624732563", 2024-02-19T21:11:41.943 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 5 2024-02-19T21:11:41.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.943 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:41.943 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.943 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:41.943 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010924+0000", 2024-02-19T21:11:41.943 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:41.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.943 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.943 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:41.943 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010924+0000", 2024-02-19T21:11:41.944 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:11:41.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.944 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:41.944 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010923+0000", 2024-02-19T21:11:41.944 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.438e-06 2024-02-19T21:11:41.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.944 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:41.944 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010929+0000", 2024-02-19T21:11:41.944 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0610000000000001e-06 2024-02-19T21:11:41.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.945 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:41.945 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010935+0000", 2024-02-19T21:11:41.945 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1400000000000002e-06 2024-02-19T21:11:41.945 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.945 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.945 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:41.945 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.010940+0000", 2024-02-19T21:11:41.945 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00023434000000000001 2024-02-19T21:11:41.945 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.945 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.945 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.945 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.011174+0000", 2024-02-19T21:11:41.946 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3954e-05 2024-02-19T21:11:41.946 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.946 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.946 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:11:41.946 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.011218+0000", 2024-02-19T21:11:41.946 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010153373 2024-02-19T21:11:41.946 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.946 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.946 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.946 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021371+0000", 2024-02-19T21:11:41.946 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9161999999999998e-05 2024-02-19T21:11:41.946 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.946 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.946 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:41.947 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021460+0000", 2024-02-19T21:11:41.947 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013853699999999999 2024-02-19T21:11:41.947 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.947 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.947 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-02-19T21:11:41.947 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021599+0000", 2024-02-19T21:11:41.947 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000369594 2024-02-19T21:11:41.947 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.947 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.947 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:11:41.947 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.021969+0000", 2024-02-19T21:11:41.948 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000122967 2024-02-19T21:11:41.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.948 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:41.948 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.022092+0000", 2024-02-19T21:11:41.948 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000293484 2024-02-19T21:11:41.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.948 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:41.948 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.022385+0000", 2024-02-19T21:11:41.948 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.586e-05 2024-02-19T21:11:41.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.949 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:41.949 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.022411+0000", 2024-02-19T21:11:41.949 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0248e-05 2024-02-19T21:11:41.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.949 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.949 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:41.949 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.022431+0000", 2024-02-19T21:11:41.949 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011507269000000001 2024-02-19T21:11:41.949 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.949 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:41.949 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.949 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.949 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.32629.0:7 76.3s0 76:dfb9de0f:test-rados-api-smithi163-83944-12::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e131)", 2024-02-19T21:11:41.950 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:11.017799+0000", 2024-02-19T21:11:41.950 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 390.88634771699998, 2024-02-19T21:11:41.950 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0068870470000000003, 2024-02-19T21:11:41.950 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:41.950 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:41.950 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:41.950 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.32629", 2024-02-19T21:11:41.950 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1643190421", 2024-02-19T21:11:41.950 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 7 2024-02-19T21:11:41.950 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.950 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:41.950 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.950 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:41.950 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017799+0000", 2024-02-19T21:11:41.951 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:41.951 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.951 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.951 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:41.951 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017799+0000", 2024-02-19T21:11:41.951 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:41.951 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.951 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.951 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:41.951 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017797+0000", 2024-02-19T21:11:41.951 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3829999999999992e-06 2024-02-19T21:11:41.951 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.951 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.951 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:41.952 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017805+0000", 2024-02-19T21:11:41.952 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3729999999999999e-06 2024-02-19T21:11:41.952 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.952 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.952 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:41.952 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017815+0000", 2024-02-19T21:11:41.952 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0319999999999998e-06 2024-02-19T21:11:41.952 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.952 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.952 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:41.952 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017822+0000", 2024-02-19T21:11:41.952 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3600999999999999e-05 2024-02-19T21:11:41.952 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.953 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.953 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.953 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017855+0000", 2024-02-19T21:11:41.953 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000114827 2024-02-19T21:11:41.953 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.953 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.953 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:41.953 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.017970+0000", 2024-02-19T21:11:41.953 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00040942799999999998 2024-02-19T21:11:41.953 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.954 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.954 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:11:41.954 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.018379+0000", 2024-02-19T21:11:41.954 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00058837500000000003 2024-02-19T21:11:41.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.954 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.954 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:11:41.954 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.018968+0000", 2024-02-19T21:11:41.954 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0056430810000000003 2024-02-19T21:11:41.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.955 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.955 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:41.955 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.024611+0000", 2024-02-19T21:11:41.955 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5131999999999998e-05 2024-02-19T21:11:41.955 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.955 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.955 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:41.955 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:11.024686+0000", 2024-02-19T21:11:41.955 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0068870470000000003 2024-02-19T21:11:41.955 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.956 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:41.956 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.956 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.956 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.956 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.32629.0:9 76.3s0 76:dfb9de0f:test-rados-api-smithi163-83944-12::foo:head [writefull 0~128] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-02-19T21:11:41.956 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.093636+0000", 2024-02-19T21:11:41.956 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 389.81051104199997, 2024-02-19T21:11:41.956 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.084125466999999995, 2024-02-19T21:11:41.956 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:41.956 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:41.956 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:41.956 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.32629", 2024-02-19T21:11:41.956 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1643190421", 2024-02-19T21:11:41.957 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 9 2024-02-19T21:11:41.957 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.957 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:41.957 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.957 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:41.957 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093636+0000", 2024-02-19T21:11:41.957 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:41.957 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.957 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.957 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:41.957 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093636+0000", 2024-02-19T21:11:41.958 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:11:41.958 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.958 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.958 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:41.958 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093633+0000", 2024-02-19T21:11:41.958 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4586000000000001e-05 2024-02-19T21:11:41.958 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.958 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.958 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:41.958 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093648+0000", 2024-02-19T21:11:41.958 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4055e-05 2024-02-19T21:11:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093662+0000", 2024-02-19T21:11:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7350000000000002e-06 2024-02-19T21:11:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093670+0000", 2024-02-19T21:11:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1807000000000001e-05 2024-02-19T21:11:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093722+0000", 2024-02-19T21:11:41.959 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0189000000000001e-05 2024-02-19T21:11:41.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.960 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.960 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:41.960 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.093802+0000", 2024-02-19T21:11:41.960 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00059241199999999995 2024-02-19T21:11:41.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.960 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.960 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:11:41.960 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.094395+0000", 2024-02-19T21:11:41.960 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0013219200000000001 2024-02-19T21:11:41.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.960 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.960 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:11:41.960 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.095717+0000", 2024-02-19T21:11:41.961 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.081990402000000004 2024-02-19T21:11:41.961 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.961 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.961 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:41.961 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.177707+0000", 2024-02-19T21:11:41.961 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4097000000000001e-05 2024-02-19T21:11:41.961 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.961 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.961 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:41.961 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.177761+0000", 2024-02-19T21:11:41.961 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.084125466999999995 2024-02-19T21:11:41.961 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.961 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:41.961 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.961 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.962 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.962 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.32629.0:10 76.3s0 76:dfb9de0f:test-rados-api-smithi163-83944-12::foo:head [rollback 5] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-02-19T21:11:41.962 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.178134+0000", 2024-02-19T21:11:41.962 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 389.72601225400001, 2024-02-19T21:11:41.962 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015877571, 2024-02-19T21:11:41.962 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:41.962 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:41.962 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:41.962 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.32629", 2024-02-19T21:11:41.962 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1643190421", 2024-02-19T21:11:41.962 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 10 2024-02-19T21:11:41.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.962 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:41.962 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.962 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:41.963 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178134+0000", 2024-02-19T21:11:41.963 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:41.963 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.963 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.963 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:41.963 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178134+0000", 2024-02-19T21:11:41.963 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:41.963 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.963 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.963 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:41.964 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178133+0000", 2024-02-19T21:11:41.964 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6609999999999997e-06 2024-02-19T21:11:41.964 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.964 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.964 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:41.964 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178138+0000", 2024-02-19T21:11:41.964 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7979999999999995e-06 2024-02-19T21:11:41.964 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.965 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.965 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:41.965 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178147+0000", 2024-02-19T21:11:41.965 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7389999999999999e-06 2024-02-19T21:11:41.965 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.965 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.966 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:41.966 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178152+0000", 2024-02-19T21:11:41.966 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7440000000000002e-05 2024-02-19T21:11:41.966 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.966 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.966 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.966 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178179+0000", 2024-02-19T21:11:41.966 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4633000000000003e-05 2024-02-19T21:11:41.967 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.967 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.967 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:41.967 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178244+0000", 2024-02-19T21:11:41.967 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00033892500000000001 2024-02-19T21:11:41.967 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.967 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.967 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:11:41.967 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178583+0000", 2024-02-19T21:11:41.967 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025176799999999998 2024-02-19T21:11:41.967 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.967 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.967 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:11:41.968 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.178835+0000", 2024-02-19T21:11:41.968 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015138621999999999 2024-02-19T21:11:41.968 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.968 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.968 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:41.968 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.193973+0000", 2024-02-19T21:11:41.968 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8704000000000001e-05 2024-02-19T21:11:41.968 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.968 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.968 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:41.968 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.194012+0000", 2024-02-19T21:11:41.968 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015877571 2024-02-19T21:11:41.968 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.968 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:41.968 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.32629.0:11 76.3s0 76:dfb9de0f:test-rados-api-smithi163-83944-12::foo:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e132)", 2024-02-19T21:11:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.201158+0000", 2024-02-19T21:11:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 389.70298846999998, 2024-02-19T21:11:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011048186, 2024-02-19T21:11:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:11:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.32629", 2024-02-19T21:11:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1643190421", 2024-02-19T21:11:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 11 2024-02-19T21:11:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.970 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:41.970 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201158+0000", 2024-02-19T21:11:41.970 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:41.970 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.970 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.970 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:41.970 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201158+0000", 2024-02-19T21:11:41.970 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:11:41.970 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.970 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.970 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:41.970 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201157+0000", 2024-02-19T21:11:41.970 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0796e-05 2024-02-19T21:11:41.970 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201168+0000", 2024-02-19T21:11:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9330000000000007e-06 2024-02-19T21:11:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201176+0000", 2024-02-19T21:11:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.229e-06 2024-02-19T21:11:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:41.971 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201183+0000", 2024-02-19T21:11:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9822999999999999e-05 2024-02-19T21:11:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201213+0000", 2024-02-19T21:11:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0374000000000001e-05 2024-02-19T21:11:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.201283+0000", 2024-02-19T21:11:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010923510000000001 2024-02-19T21:11:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.972 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.973 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:41.973 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.212206+0000", 2024-02-19T21:11:41.973 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011048186 2024-02-19T21:11:41.973 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.973 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:41.973 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.973 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.973 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.973 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:252 61.16 61:68726f76:test-rados-api-smithi163-83928-12::foo:head [writefull 0~5] snapc 2=[2] ondisk+write+known_if_redirected e135)", 2024-02-19T21:11:41.973 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:15.184366+0000", 2024-02-19T21:11:41.973 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 386.71978086299998, 2024-02-19T21:11:41.973 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0086795729999999995, 2024-02-19T21:11:41.973 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:41.974 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:41.974 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:41.974 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:11:41.974 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:11:41.974 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 252 2024-02-19T21:11:41.974 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.974 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:41.975 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.975 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:41.975 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184366+0000", 2024-02-19T21:11:41.975 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:41.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.975 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.975 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:41.975 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184366+0000", 2024-02-19T21:11:41.975 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:11:41.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.976 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:41.976 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184363+0000", 2024-02-19T21:11:41.976 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1565e-05 2024-02-19T21:11:41.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.976 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:41.976 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184375+0000", 2024-02-19T21:11:41.976 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.38e-05 2024-02-19T21:11:41.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.976 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:41.976 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184388+0000", 2024-02-19T21:11:41.977 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.366e-05 2024-02-19T21:11:41.977 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.977 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.977 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:41.977 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.184402+0000", 2024-02-19T21:11:41.977 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0022266690000000001 2024-02-19T21:11:41.977 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.977 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.977 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.977 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.186629+0000", 2024-02-19T21:11:41.977 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010571800000000001 2024-02-19T21:11:41.977 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.977 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.977 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:41.977 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.186734+0000", 2024-02-19T21:11:41.978 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00043829699999999998 2024-02-19T21:11:41.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.978 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.978 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 5,7", 2024-02-19T21:11:41.978 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.187173+0000", 2024-02-19T21:11:41.978 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0040489430000000002 2024-02-19T21:11:41.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.978 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.978 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:41.978 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.191222+0000", 2024-02-19T21:11:41.978 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0153000000000002e-05 2024-02-19T21:11:41.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.978 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.978 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:11:41.978 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.191312+0000", 2024-02-19T21:11:41.979 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001633362 2024-02-19T21:11:41.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.979 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.979 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:41.979 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.192945+0000", 2024-02-19T21:11:41.979 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9512000000000003e-05 2024-02-19T21:11:41.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.979 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.979 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:41.979 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.192995+0000", 2024-02-19T21:11:41.979 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0753e-05 2024-02-19T21:11:41.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.979 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.979 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:41.979 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193045+0000", 2024-02-19T21:11:41.980 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0086795729999999995 2024-02-19T21:11:41.980 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.980 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:41.980 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.980 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.980 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.980 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.47503.0:2 93.18 93:1e0071ee:::oid_RoundTripSparseReadPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e142)", 2024-02-19T21:11:41.980 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:22.198653+0000", 2024-02-19T21:11:41.980 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 379.70549326499997, 2024-02-19T21:11:41.980 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025958305000000001, 2024-02-19T21:11:41.980 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:41.980 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:41.980 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:41.980 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.47503", 2024-02-19T21:11:41.980 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1485143480", 2024-02-19T21:11:41.981 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:11:41.981 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.981 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:41.981 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.981 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:41.981 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198653+0000", 2024-02-19T21:11:41.981 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:41.981 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.981 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.981 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:41.981 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198653+0000", 2024-02-19T21:11:41.981 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:11:41.981 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.981 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.981 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:41.982 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198651+0000", 2024-02-19T21:11:41.982 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2534000000000001e-05 2024-02-19T21:11:41.982 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.982 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.982 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:41.982 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198663+0000", 2024-02-19T21:11:41.982 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8508999999999998e-05 2024-02-19T21:11:41.982 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.982 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.982 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:41.982 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198682+0000", 2024-02-19T21:11:41.982 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2040000000000003e-06 2024-02-19T21:11:41.982 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.982 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.983 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:41.983 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198691+0000", 2024-02-19T21:11:41.983 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6539999999999997e-05 2024-02-19T21:11:41.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.983 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.983 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.983 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198747+0000", 2024-02-19T21:11:41.983 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0387000000000001e-05 2024-02-19T21:11:41.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.983 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.983 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:11:41.983 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.198758+0000", 2024-02-19T21:11:41.983 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015289329000000001 2024-02-19T21:11:41.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.984 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.984 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.984 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.214047+0000", 2024-02-19T21:11:41.984 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000131958 2024-02-19T21:11:41.984 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.984 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.984 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:41.984 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.214179+0000", 2024-02-19T21:11:41.984 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000165496 2024-02-19T21:11:41.984 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.984 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.984 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,5", 2024-02-19T21:11:41.984 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.214345+0000", 2024-02-19T21:11:41.984 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00067452899999999999 2024-02-19T21:11:41.985 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.985 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.985 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:11:41.985 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.215019+0000", 2024-02-19T21:11:41.985 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0069561409999999999 2024-02-19T21:11:41.985 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.985 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.985 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:41.985 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.221975+0000", 2024-02-19T21:11:41.985 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002534653 2024-02-19T21:11:41.985 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.985 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.985 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:41.985 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.224510+0000", 2024-02-19T21:11:41.985 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7840000000000002e-05 2024-02-19T21:11:41.986 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.986 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.986 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:41.986 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.224568+0000", 2024-02-19T21:11:41.986 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4063999999999997e-05 2024-02-19T21:11:41.986 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.986 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.986 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:41.986 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:22.224612+0000", 2024-02-19T21:11:41.986 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025958305000000001 2024-02-19T21:11:41.986 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.986 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:41.986 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.986 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.986 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.987 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:873 94.e 94:778c9cf1:test-rados-api-smithi163-83928-14::foo0:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-19T21:11:41.987 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:31.178555+0000", 2024-02-19T21:11:41.987 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 370.72559197099997, 2024-02-19T21:11:41.987 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022321049999999999, 2024-02-19T21:11:41.987 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:41.987 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:11:41.987 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:41.987 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:11:41.987 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:11:41.987 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 873 2024-02-19T21:11:41.987 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.987 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:41.987 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.987 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:41.987 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178555+0000", 2024-02-19T21:11:41.988 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:41.988 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.988 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.988 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:41.988 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178555+0000", 2024-02-19T21:11:41.988 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:11:41.988 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.988 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.988 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:41.988 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178552+0000", 2024-02-19T21:11:41.988 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4859999999999997e-06 2024-02-19T21:11:41.988 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.988 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.988 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:41.989 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178560+0000", 2024-02-19T21:11:41.989 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0013999999999999e-05 2024-02-19T21:11:41.989 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.989 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.989 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:41.989 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178570+0000", 2024-02-19T21:11:41.989 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1750000000000005e-06 2024-02-19T21:11:41.989 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.989 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.989 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:41.989 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178579+0000", 2024-02-19T21:11:41.989 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5300999999999999e-05 2024-02-19T21:11:41.989 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.989 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.989 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.990 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178624+0000", 2024-02-19T21:11:41.990 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8860000000000001e-06 2024-02-19T21:11:41.990 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.990 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.990 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:11:41.990 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.178630+0000", 2024-02-19T21:11:41.990 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0094360829999999996 2024-02-19T21:11:41.990 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.990 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.990 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.990 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.188066+0000", 2024-02-19T21:11:41.990 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00022991600000000001 2024-02-19T21:11:41.990 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.990 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.990 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:11:41.991 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.188296+0000", 2024-02-19T21:11:41.991 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0070897570000000004 2024-02-19T21:11:41.991 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.991 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.991 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.991 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.195386+0000", 2024-02-19T21:11:41.991 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7982000000000001e-05 2024-02-19T21:11:41.991 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.991 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.991 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:11:41.991 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.195454+0000", 2024-02-19T21:11:41.991 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0051389959999999998 2024-02-19T21:11:41.991 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.991 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.991 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.992 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.200593+0000", 2024-02-19T21:11:41.992 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4277999999999996e-05 2024-02-19T21:11:41.992 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.992 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.992 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:41.992 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.200667+0000", 2024-02-19T21:11:41.992 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00020899300000000001 2024-02-19T21:11:41.992 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.992 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.992 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:41.992 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.200876+0000", 2024-02-19T21:11:41.992 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022321049999999999 2024-02-19T21:11:41.992 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.992 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:41.992 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.993 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.993 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:967 94.1d 94:bb38b02c:test-rados-api-smithi163-83928-14::foo54:6 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e151)", 2024-02-19T21:11:41.993 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:31.370348+0000", 2024-02-19T21:11:41.993 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 370.53379886200003, 2024-02-19T21:11:41.993 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0052976730000000001, 2024-02-19T21:11:41.993 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:41.993 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:11:41.993 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:41.993 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:11:41.993 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:11:41.993 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 967 2024-02-19T21:11:41.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.993 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:41.993 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.994 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:41.994 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370348+0000", 2024-02-19T21:11:41.994 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:41.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.994 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.994 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:41.994 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370348+0000", 2024-02-19T21:11:41.994 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:11:41.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.994 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.994 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:41.994 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370347+0000", 2024-02-19T21:11:41.994 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7369999999999997e-06 2024-02-19T21:11:41.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.995 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.995 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:41.995 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370351+0000", 2024-02-19T21:11:41.995 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2239999999999995e-06 2024-02-19T21:11:41.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.995 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.995 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:41.995 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370360+0000", 2024-02-19T21:11:41.995 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6620000000000002e-06 2024-02-19T21:11:41.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.995 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.995 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:41.995 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370365+0000", 2024-02-19T21:11:41.995 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3609999999999998e-05 2024-02-19T21:11:41.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.996 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.996 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.996 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370399+0000", 2024-02-19T21:11:41.996 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013085899999999999 2024-02-19T21:11:41.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.996 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.996 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:11:41.996 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.370530+0000", 2024-02-19T21:11:41.996 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001259874 2024-02-19T21:11:41.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.996 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.996 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.996 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.371790+0000", 2024-02-19T21:11:41.996 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00017583099999999999 2024-02-19T21:11:41.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.997 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.997 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:11:41.997 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.371966+0000", 2024-02-19T21:11:41.997 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00083226100000000005 2024-02-19T21:11:41.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.997 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.997 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.997 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.372798+0000", 2024-02-19T21:11:41.997 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8339999999999998e-05 2024-02-19T21:11:41.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.997 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.997 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:11:41.997 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.372886+0000", 2024-02-19T21:11:41.997 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00027153799999999999 2024-02-19T21:11:41.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.998 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.998 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:41.998 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.373158+0000", 2024-02-19T21:11:41.998 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3312000000000006e-05 2024-02-19T21:11:41.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.998 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.998 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:41.998 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.373221+0000", 2024-02-19T21:11:41.998 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002424537 2024-02-19T21:11:41.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.998 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.998 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:41.998 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:31.375646+0000", 2024-02-19T21:11:41.998 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0052976730000000001 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.22476.0:2 137.a e196/195 137:53c7e431:::oid_AioUnlockPP_:head v 196'1, mlcod=0'0)", 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:06:16.194139+0000", 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 325.710007717, 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0057252520000000001, 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194139+0000", 2024-02-19T21:11:41.999 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:42.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.000 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:42.000 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194139+0000", 2024-02-19T21:11:42.000 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:11:42.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.000 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:42.000 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194138+0000", 2024-02-19T21:11:42.000 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1670000000000002e-06 2024-02-19T21:11:42.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.000 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:42.000 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194145+0000", 2024-02-19T21:11:42.001 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8575000000000001e-05 2024-02-19T21:11:42.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.001 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.001 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:42.001 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194164+0000", 2024-02-19T21:11:42.001 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3140000000000004e-06 2024-02-19T21:11:42.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.001 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.001 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:42.001 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194172+0000", 2024-02-19T21:11:42.001 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014666399999999999 2024-02-19T21:11:42.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.001 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.001 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:42.001 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194319+0000", 2024-02-19T21:11:42.002 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9656e-05 2024-02-19T21:11:42.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.002 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.002 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:42.002 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.194339+0000", 2024-02-19T21:11:42.002 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.005498111 2024-02-19T21:11:42.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.002 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.002 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:42.002 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.199837+0000", 2024-02-19T21:11:42.002 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7557999999999999e-05 2024-02-19T21:11:42.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.002 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.002 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:42.002 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.199864+0000", 2024-02-19T21:11:42.003 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0057252520000000001 2024-02-19T21:11:42.003 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:42.003 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:42.003 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:42.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.003 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.003 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "MOSDECSubOpWrite(136.6s2 196/195 ECSubWrite(tid=402, reqid=client.47941.0:1, at_version=196'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-19T21:11:42.003 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:06:16.196362+0000", 2024-02-19T21:11:42.003 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 325.70778490700002, 2024-02-19T21:11:42.003 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010287370000000001, 2024-02-19T21:11:42.003 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:42.003 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-02-19T21:11:42.003 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:42.003 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.003 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:42.004 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196362+0000", 2024-02-19T21:11:42.004 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:42.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.004 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.004 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:42.004 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196362+0000", 2024-02-19T21:11:42.004 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:11:42.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.004 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.004 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:42.004 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196361+0000", 2024-02-19T21:11:42.004 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3603999999999999e-05 2024-02-19T21:11:42.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.004 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.004 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196395+0000", 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.551e-06 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196404+0000", 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8769999999999993e-06 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196411+0000", 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00026643000000000001 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:42.005 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196678+0000", 2024-02-19T21:11:42.006 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0366e-05 2024-02-19T21:11:42.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.006 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:11:42.006 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196688+0000", 2024-02-19T21:11:42.006 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0099375339999999996 2024-02-19T21:11:42.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.006 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:11:42.006 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.206626+0000", 2024-02-19T21:11:42.006 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3348000000000001e-05 2024-02-19T21:11:42.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.006 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:42.006 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.206649+0000", 2024-02-19T21:11:42.007 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010287370000000001 2024-02-19T21:11:42.007 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:42.007 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:42.007 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:42.007 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.007 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.007 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.25753.0:3126 186.3 e294/290 186:c0520479:test-rados-api-smithi163-83928-38::foo:head v 294'2, mlcod=0'0)", 2024-02-19T21:11:42.007 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:54.225630+0000", 2024-02-19T21:11:42.007 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 227.678516715, 2024-02-19T21:11:42.007 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017481528, 2024-02-19T21:11:42.007 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:42.007 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:42.007 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:42.007 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.007 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:42.008 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.225630+0000", 2024-02-19T21:11:42.008 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:42.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.008 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.008 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:42.008 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.225630+0000", 2024-02-19T21:11:42.008 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:11:42.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.008 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.008 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:42.008 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.225628+0000", 2024-02-19T21:11:42.008 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015168282 2024-02-19T21:11:42.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.008 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.009 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:42.009 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.240797+0000", 2024-02-19T21:11:42.009 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3131e-05 2024-02-19T21:11:42.009 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.009 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.009 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:42.009 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.240810+0000", 2024-02-19T21:11:42.009 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2637e-05 2024-02-19T21:11:42.009 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.009 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.009 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:42.009 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.240832+0000", 2024-02-19T21:11:42.009 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4966999999999998e-05 2024-02-19T21:11:42.009 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.010 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.010 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:42.010 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.240877+0000", 2024-02-19T21:11:42.010 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5106e-05 2024-02-19T21:11:42.010 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.010 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.010 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:42.010 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.240903+0000", 2024-02-19T21:11:42.010 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0021667320000000002 2024-02-19T21:11:42.010 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.010 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.010 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:42.010 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.243069+0000", 2024-02-19T21:11:42.010 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2271999999999998e-05 2024-02-19T21:11:42.011 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.011 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.011 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:42.011 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.243112+0000", 2024-02-19T21:11:42.011 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017481528 2024-02-19T21:11:42.011 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:42.011 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:42.011 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:42.011 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.011 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.011 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:1423 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94681540390272 gen 2] snapc 0=[] ondisk+write+known_if_redirected e433)", 2024-02-19T21:11:42.011 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:10:40.311888+0000", 2024-02-19T21:11:42.011 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 61.592258327000003, 2024-02-19T21:11:42.011 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.01168904, 2024-02-19T21:11:42.011 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:42.012 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:11:42.012 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:42.012 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:11:42.012 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:11:42.012 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1423 2024-02-19T21:11:42.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.012 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:42.012 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.012 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:42.012 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311888+0000", 2024-02-19T21:11:42.012 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:42.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.012 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.012 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:42.013 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311888+0000", 2024-02-19T21:11:42.013 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:11:42.013 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.013 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.013 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:42.013 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311886+0000", 2024-02-19T21:11:42.013 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0350000000000007e-06 2024-02-19T21:11:42.013 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.013 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.013 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:42.013 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311895+0000", 2024-02-19T21:11:42.013 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0015e-05 2024-02-19T21:11:42.013 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.013 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.013 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:42.014 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311905+0000", 2024-02-19T21:11:42.014 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0219999999999999e-06 2024-02-19T21:11:42.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.014 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.014 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:42.014 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311912+0000", 2024-02-19T21:11:42.014 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00048395300000000002 2024-02-19T21:11:42.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.014 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.014 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:42.014 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.312396+0000", 2024-02-19T21:11:42.014 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7010000000000002e-06 2024-02-19T21:11:42.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.014 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.014 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:11:42.015 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.312404+0000", 2024-02-19T21:11:42.015 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.001316297 2024-02-19T21:11:42.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.015 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.015 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:42.015 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.313721+0000", 2024-02-19T21:11:42.015 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000109879 2024-02-19T21:11:42.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.015 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.015 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for missing object", 2024-02-19T21:11:42.015 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.313831+0000", 2024-02-19T21:11:42.015 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0062348129999999996 2024-02-19T21:11:42.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.015 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.015 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:42.016 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.320065+0000", 2024-02-19T21:11:42.016 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000102769 2024-02-19T21:11:42.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.016 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.016 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:11:42.016 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.320168+0000", 2024-02-19T21:11:42.016 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00084819799999999999 2024-02-19T21:11:42.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.016 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.016 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:42.016 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.321016+0000", 2024-02-19T21:11:42.016 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5853999999999997e-05 2024-02-19T21:11:42.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.016 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.321082+0000", 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00249519 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.323577+0000", 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.01168904 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(osd.3.25:34 216.16 216:6dfc172a:test-rados-api-smithi163-83928-70::aab9b7bd17003da21256cca5c7d2d7ff4ea384b6:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e452)", 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:11:04.382329+0000", 2024-02-19T21:11:42.017 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 37.521817853999998, 2024-02-19T21:11:42.018 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026243446, 2024-02-19T21:11:42.018 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:42.018 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:42.018 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:42.018 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.018 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:42.018 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382329+0000", 2024-02-19T21:11:42.018 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:42.018 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.018 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.018 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:42.018 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382329+0000", 2024-02-19T21:11:42.018 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:42.018 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.018 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382326+0000", 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0463e-05 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382337+0000", 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.613e-05 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382353+0000", 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6540000000000002e-06 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.019 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:42.020 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382360+0000", 2024-02-19T21:11:42.020 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7012000000000002e-05 2024-02-19T21:11:42.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.020 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:42.020 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382417+0000", 2024-02-19T21:11:42.020 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000108528 2024-02-19T21:11:42.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.020 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:42.020 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382525+0000", 2024-02-19T21:11:42.020 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00020552499999999999 2024-02-19T21:11:42.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.021 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-02-19T21:11:42.021 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382731+0000", 2024-02-19T21:11:42.021 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00039365899999999998 2024-02-19T21:11:42.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.021 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.021 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:11:42.021 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.383124+0000", 2024-02-19T21:11:42.021 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0028295439999999998 2024-02-19T21:11:42.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.021 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.021 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:42.021 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.385954+0000", 2024-02-19T21:11:42.021 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022528835000000001 2024-02-19T21:11:42.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.022 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.022 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:42.022 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.408483+0000", 2024-02-19T21:11:42.022 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3378000000000003e-05 2024-02-19T21:11:42.022 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.022 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.022 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:42.022 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.408536+0000", 2024-02-19T21:11:42.022 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6229999999999997e-05 2024-02-19T21:11:42.022 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.022 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.022 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:42.022 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.408572+0000", 2024-02-19T21:11:42.022 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026243446 2024-02-19T21:11:42.022 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:42.023 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:42.023 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:42.023 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.023 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.023 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(osd.7.53:52 218.19 218:98e59dac:test-rados-api-smithi163-83928-74::chunk4:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e466)", 2024-02-19T21:11:42.023 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:11:34.301468+0000", 2024-02-19T21:11:42.023 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 7.6026789279999996, 2024-02-19T21:11:42.023 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0059556599999999998, 2024-02-19T21:11:42.023 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:42.023 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:42.023 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:42.023 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.023 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:42.023 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.301468+0000", 2024-02-19T21:11:42.023 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:42.024 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.024 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.024 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:42.024 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.301468+0000", 2024-02-19T21:11:42.024 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:42.024 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.024 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.024 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:42.024 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.301465+0000", 2024-02-19T21:11:42.024 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6389999999999998e-06 2024-02-19T21:11:42.024 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.024 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.024 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:42.024 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.301474+0000", 2024-02-19T21:11:42.025 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0288999999999999e-05 2024-02-19T21:11:42.025 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.025 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.025 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:42.025 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.301484+0000", 2024-02-19T21:11:42.025 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6699999999999994e-06 2024-02-19T21:11:42.025 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.025 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.025 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:42.025 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.301492+0000", 2024-02-19T21:11:42.025 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6393e-05 2024-02-19T21:11:42.025 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.025 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.025 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:42.025 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.301548+0000", 2024-02-19T21:11:42.026 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2979999999999999e-06 2024-02-19T21:11:42.026 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.026 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.026 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:11:42.026 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.301554+0000", 2024-02-19T21:11:42.026 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020830480000000001 2024-02-19T21:11:42.026 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.026 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.026 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:42.026 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.303637+0000", 2024-02-19T21:11:42.026 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000104967 2024-02-19T21:11:42.026 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.026 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.026 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:42.026 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.303742+0000", 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000221504 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 6,7", 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.303963+0000", 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00071045899999999998 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.304673+0000", 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00053209600000000002 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.305206+0000", 2024-02-19T21:11:42.027 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0021259959999999998 2024-02-19T21:11:42.028 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.028 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.028 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:11:42.028 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.307332+0000", 2024-02-19T21:11:42.028 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3253e-05 2024-02-19T21:11:42.028 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.028 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.028 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:42.028 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.307385+0000", 2024-02-19T21:11:42.028 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8596000000000001e-05 2024-02-19T21:11:42.028 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:42.028 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:42.028 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:42.028 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.307423+0000", 2024-02-19T21:11:42.029 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0059556599999999998 2024-02-19T21:11:42.029 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:42.029 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:42.029 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:42.029 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:42.029 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:42.029 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:11:42.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:42.030 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:42.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:42.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.905+0000 7f19a37fe700 1 -- 172.21.15.163:0/3432772882 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f199402b880 con 0x7f1994027d70 2024-02-19T21:11:42.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.905+0000 7f19a37fe700 10 client.48805.objecter ms_dispatch 0x7f19b4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:42.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.905+0000 7f19a37fe700 10 client.48805.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:42.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.905+0000 7f19a37fe700 20 client.48805.objecter put_session s=0x7f1994025290 osd=7 4 2024-02-19T21:11:42.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.905+0000 7f19a37fe700 15 client.48805.objecter _session_command_op_remove 7 16 2024-02-19T21:11:42.031 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:11:42.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 10 librados: watch_flush enter 2024-02-19T21:11:42.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 10 librados: watch_flush exit 2024-02-19T21:11:42.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 10 client.48805.objecter close_session for osd.0 2024-02-19T21:11:42.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 >> v1:172.21.15.163:6801/1559418115 conn(0x7f1994004530 legacy=0x7f1994002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:42.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f19940051a0 osd=0 1 2024-02-19T21:11:42.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 10 client.48805.objecter close_session for osd.1 2024-02-19T21:11:42.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 >> v1:172.21.15.163:6805/3125047432 conn(0x7f1994004150 legacy=0x7f199400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:42.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994003fe0 osd=1 1 2024-02-19T21:11:42.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 10 client.48805.objecter close_session for osd.2 2024-02-19T21:11:42.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 >> v1:172.21.15.163:6809/1563158487 conn(0x7f199400c170 legacy=0x7f199400c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:42.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994003630 osd=2 1 2024-02-19T21:11:42.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 10 client.48805.objecter close_session for osd.3 2024-02-19T21:11:42.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 >> v1:172.21.15.163:6813/2112279769 conn(0x7f1994016290 legacy=0x7f19940186e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:42.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b2d9d700 1 -- 172.21.15.163:0/3432772882 reap_dead start 2024-02-19T21:11:42.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994003470 osd=3 1 2024-02-19T21:11:42.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 10 client.48805.objecter close_session for osd.4 2024-02-19T21:11:42.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 >> v1:172.21.15.193:6800/3129716965 conn(0x7f199400e460 legacy=0x7f199401a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:42.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f199400d020 osd=4 1 2024-02-19T21:11:42.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 10 client.48805.objecter close_session for osd.5 2024-02-19T21:11:42.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 >> v1:172.21.15.193:6804/805161991 conn(0x7f199401c100 legacy=0x7f1994021960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:42.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f199401a840 osd=5 1 2024-02-19T21:11:42.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 10 client.48805.objecter close_session for osd.6 2024-02-19T21:11:42.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 >> v1:172.21.15.193:6808/787604615 conn(0x7f1994023590 legacy=0x7f1994023a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:42.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f199401c6f0 osd=6 1 2024-02-19T21:11:42.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 10 client.48805.objecter close_session for osd.7 2024-02-19T21:11:42.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 >> v1:172.21.15.193:6812/157927432 conn(0x7f1994027d70 legacy=0x7f199402a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:42.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 20 client.48805.objecter put_session s=0x7f1994025290 osd=7 1 2024-02-19T21:11:42.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 20 client.48805.objecter shutdown clearing up homeless session... 2024-02-19T21:11:42.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 10 client.48805.objecter successfully canceled tick 2024-02-19T21:11:42.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b2d9d700 1 -- 172.21.15.163:0/3432772882 reap_dead start 2024-02-19T21:11:42.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b2d9d700 1 -- 172.21.15.163:0/3432772882 reap_dead start 2024-02-19T21:11:42.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b2d9d700 1 -- 172.21.15.163:0/3432772882 reap_dead start 2024-02-19T21:11:42.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b2d9d700 1 -- 172.21.15.163:0/3432772882 reap_dead start 2024-02-19T21:11:42.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 >> v1:172.21.15.163:6800/1552514681 conn(0x7f199c063cc0 legacy=0x7f199c066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:42.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 >> v1:172.21.15.193:6789/0 conn(0x7f19b40fd980 legacy=0x7f19b40f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:42.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 shutdown_connections 2024-02-19T21:11:42.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 >> 172.21.15.163:0/3432772882 conn(0x7f19b40f35a0 msgr2=0x7f19b4107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:42.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 shutdown_connections 2024-02-19T21:11:42.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 1 -- 172.21.15.163:0/3432772882 wait complete. 2024-02-19T21:11:42.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:41.906+0000 7f19b8f0c700 1 librados: shutdown 2024-02-19T21:11:42.035 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:11:42.035 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:11:42.035 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:11:42.035 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:11:42.036 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:11:42.036 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:11:42.036 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:11:42.036 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:11:42.131 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:42.158 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:42.357 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:42.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:42.439 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:40.577111+0000 osd.7 (osd.7) 219 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:11:42.439 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:40.580340+0000 osd.7 (osd.7) 220 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:11:42.439 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:40.736921+0000 osd.6 (osd.6) 199 : cluster [DBG] 61.e scrub starts 2024-02-19T21:11:42.439 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:40.738797+0000 osd.6 (osd.6) 200 : cluster [DBG] 61.e scrub ok 2024-02-19T21:11:42.439 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:41.239025+0000 mgr.y (mgr.14550) 435 : cluster [DBG] pgmap v703: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-02-19T21:11:42.440 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:41.763579+0000 mon.b (mon.1) 624 : audit [DBG] from='client.? v1:172.21.15.163:0/3432772882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:42.440 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:40.577111+0000 osd.7 (osd.7) 219 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:11:42.440 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:40.580340+0000 osd.7 (osd.7) 220 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:11:42.440 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:40.736921+0000 osd.6 (osd.6) 199 : cluster [DBG] 61.e scrub starts 2024-02-19T21:11:42.440 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:40.738797+0000 osd.6 (osd.6) 200 : cluster [DBG] 61.e scrub ok 2024-02-19T21:11:42.440 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:41.239025+0000 mgr.y (mgr.14550) 435 : cluster [DBG] pgmap v703: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-02-19T21:11:42.440 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:41.763579+0000 mon.b (mon.1) 624 : audit [DBG] from='client.? v1:172.21.15.163:0/3432772882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:42.462 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:42.490 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:42.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:40.577111+0000 osd.7 (osd.7) 219 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:11:42.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:40.580340+0000 osd.7 (osd.7) 220 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:11:42.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:40.736921+0000 osd.6 (osd.6) 199 : cluster [DBG] 61.e scrub starts 2024-02-19T21:11:42.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:40.738797+0000 osd.6 (osd.6) 200 : cluster [DBG] 61.e scrub ok 2024-02-19T21:11:42.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:41.239025+0000 mgr.y (mgr.14550) 435 : cluster [DBG] pgmap v703: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-02-19T21:11:42.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:41.763579+0000 mon.b (mon.1) 624 : audit [DBG] from='client.? v1:172.21.15.163:0/3432772882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:42.577 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-19T21:11:42.843 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:11:42.937 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:42.940 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:42.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:42.941 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:43.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:11:43.070 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:43.168 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:11:43.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:43.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:11:43.272 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:43.299 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:43.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:41.773451+0000 osd.6 (osd.6) 201 : cluster [DBG] 5.3 deep-scrub starts 2024-02-19T21:11:43.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:41.775413+0000 osd.6 (osd.6) 202 : cluster [DBG] 5.3 deep-scrub ok 2024-02-19T21:11:43.468 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:41.979850+0000 osd.1 (osd.1) 231 : cluster [DBG] 219.1b deep-scrub starts 2024-02-19T21:11:43.469 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:41.983339+0000 osd.1 (osd.1) 232 : cluster [DBG] 219.1b deep-scrub ok 2024-02-19T21:11:43.469 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:42.332308+0000 osd.4 (osd.4) 171 : cluster [DBG] 4.e scrub starts 2024-02-19T21:11:43.469 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:42.333776+0000 osd.4 (osd.4) 172 : cluster [DBG] 4.e scrub ok 2024-02-19T21:11:43.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:41.773451+0000 osd.6 (osd.6) 201 : cluster [DBG] 5.3 deep-scrub starts 2024-02-19T21:11:43.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:41.775413+0000 osd.6 (osd.6) 202 : cluster [DBG] 5.3 deep-scrub ok 2024-02-19T21:11:43.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:41.979850+0000 osd.1 (osd.1) 231 : cluster [DBG] 219.1b deep-scrub starts 2024-02-19T21:11:43.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:41.983339+0000 osd.1 (osd.1) 232 : cluster [DBG] 219.1b deep-scrub ok 2024-02-19T21:11:43.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:42.332308+0000 osd.4 (osd.4) 171 : cluster [DBG] 4.e scrub starts 2024-02-19T21:11:43.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:42.333776+0000 osd.4 (osd.4) 172 : cluster [DBG] 4.e scrub ok 2024-02-19T21:11:43.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:41.773451+0000 osd.6 (osd.6) 201 : cluster [DBG] 5.3 deep-scrub starts 2024-02-19T21:11:43.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:41.775413+0000 osd.6 (osd.6) 202 : cluster [DBG] 5.3 deep-scrub ok 2024-02-19T21:11:43.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:41.979850+0000 osd.1 (osd.1) 231 : cluster [DBG] 219.1b deep-scrub starts 2024-02-19T21:11:43.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:41.983339+0000 osd.1 (osd.1) 232 : cluster [DBG] 219.1b deep-scrub ok 2024-02-19T21:11:43.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:42.332308+0000 osd.4 (osd.4) 171 : cluster [DBG] 4.e scrub starts 2024-02-19T21:11:43.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:42.333776+0000 osd.4 (osd.4) 172 : cluster [DBG] 4.e scrub ok 2024-02-19T21:11:43.746 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:11:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:11:43] "GET /metrics HTTP/1.1" 200 190066 "" "Prometheus/2.33.4" 2024-02-19T21:11:43.748 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:43.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:43.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:43.853 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:11:43.881 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:44.155 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:11:44.330 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:11:44.334 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:44.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:11:44.435 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:44.465 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:44.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:42.526603+0000 osd.7 (osd.7) 221 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:11:44.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:42.529486+0000 osd.7 (osd.7) 222 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:11:44.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:43.239521+0000 mgr.y (mgr.14550) 436 : cluster [DBG] pgmap v704: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-02-19T21:11:44.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:43.368709+0000 osd.4 (osd.4) 173 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:11:44.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:43.369602+0000 osd.4 (osd.4) 174 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:11:44.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:43.406549+0000 osd.3 (osd.3) 213 : cluster [DBG] 16.1a scrub starts 2024-02-19T21:11:44.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:43.407509+0000 osd.3 (osd.3) 214 : cluster [DBG] 16.1a scrub ok 2024-02-19T21:11:44.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:44.294722+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e472: 8 total, 8 up, 8 in 2024-02-19T21:11:44.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:42.526603+0000 osd.7 (osd.7) 221 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:11:44.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:42.529486+0000 osd.7 (osd.7) 222 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:11:44.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:43.239521+0000 mgr.y (mgr.14550) 436 : cluster [DBG] pgmap v704: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-02-19T21:11:44.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:43.368709+0000 osd.4 (osd.4) 173 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:11:44.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:43.369602+0000 osd.4 (osd.4) 174 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:11:44.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:43.406549+0000 osd.3 (osd.3) 213 : cluster [DBG] 16.1a scrub starts 2024-02-19T21:11:44.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:43.407509+0000 osd.3 (osd.3) 214 : cluster [DBG] 16.1a scrub ok 2024-02-19T21:11:44.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:44.294722+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e472: 8 total, 8 up, 8 in 2024-02-19T21:11:44.659 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:11:44.659 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:11:44.659 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-02-19T21:11:44.659 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:11:44.663 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:44.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:44.667 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:42.526603+0000 osd.7 (osd.7) 221 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:11:44.667 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:42.529486+0000 osd.7 (osd.7) 222 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:11:44.667 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:43.239521+0000 mgr.y (mgr.14550) 436 : cluster [DBG] pgmap v704: 228 pgs: 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-02-19T21:11:44.667 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:43.368709+0000 osd.4 (osd.4) 173 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:11:44.668 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:43.369602+0000 osd.4 (osd.4) 174 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:11:44.668 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:43.406549+0000 osd.3 (osd.3) 213 : cluster [DBG] 16.1a scrub starts 2024-02-19T21:11:44.668 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:43.407509+0000 osd.3 (osd.3) 214 : cluster [DBG] 16.1a scrub ok 2024-02-19T21:11:44.668 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:44.294722+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e472: 8 total, 8 up, 8 in 2024-02-19T21:11:44.768 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:44.796 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:44.890 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:11:44.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.910+0000 7facb1905700 1 -- 172.21.15.163:0/3730925478 >> v1:172.21.15.163:6790/0 conn(0x7facac1031b0 legacy=0x7facac105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:44.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.910+0000 7facb1905700 1 -- 172.21.15.163:0/3730925478 shutdown_connections 2024-02-19T21:11:44.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.910+0000 7facb1905700 1 -- 172.21.15.163:0/3730925478 >> 172.21.15.163:0/3730925478 conn(0x7facac0f2470 msgr2=0x7facac0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:44.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.910+0000 7facb1905700 1 -- 172.21.15.163:0/3730925478 shutdown_connections 2024-02-19T21:11:44.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.910+0000 7facb1905700 1 -- 172.21.15.163:0/3730925478 wait complete. 2024-02-19T21:11:44.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.910+0000 7facb1905700 1 librados: starting msgr at 2024-02-19T21:11:44.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.910+0000 7facb1905700 1 librados: starting objecter 2024-02-19T21:11:44.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7facb1905700 1 Processor -- start 2024-02-19T21:11:44.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7facb1905700 1 -- start start 2024-02-19T21:11:44.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7facb1905700 1 librados: setting wanted keys 2024-02-19T21:11:44.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7facb1905700 1 librados: calling monclient init 2024-02-19T21:11:44.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7facb1905700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7facac117180 con 0x7facac0fc870 2024-02-19T21:11:44.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7facb1905700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7facac118360 con 0x7facac1031b0 2024-02-19T21:11:44.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7facb1905700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7facac119560 con 0x7facac0ff670 2024-02-19T21:11:44.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7facaaffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7facac0fc870 0x7facac060bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43428/0 (socket says 172.21.15.163:43428) 2024-02-19T21:11:44.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7facaaffd700 1 -- 172.21.15.163:0/1766486745 learned_addr learned my addr 172.21.15.163:0/1766486745 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:44.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7facaa7fc700 1 --1- 172.21.15.163:0/1766486745 >> v1:172.21.15.163:6790/0 conn(0x7facac0ff670 0x7facac0612d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:43138/0 (socket says 172.21.15.163:43138) 2024-02-19T21:11:44.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7fac93fff700 10 client.?.objecter ms_handle_connect 0x7facac0ff670 2024-02-19T21:11:44.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7fac93fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:44.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7fac93fff700 10 client.?.objecter ms_handle_connect 0x7facac0fc870 2024-02-19T21:11:44.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7fac93fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:44.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7fac93fff700 10 client.?.objecter ms_handle_connect 0x7facac1031b0 2024-02-19T21:11:44.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7fac93fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:44.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.911+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843530437 0 0) 0x7facac119560 con 0x7facac0ff670 2024-02-19T21:11:44.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac940033f0 con 0x7facac0ff670 2024-02-19T21:11:44.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511731711 0 0) 0x7facac118360 con 0x7facac1031b0 2024-02-19T21:11:44.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facac119560 con 0x7facac1031b0 2024-02-19T21:11:44.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701429297 0 0) 0x7facac117180 con 0x7facac0fc870 2024-02-19T21:11:44.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facac118360 con 0x7facac0fc870 2024-02-19T21:11:44.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1946435579 0 0) 0x7facac119560 con 0x7facac1031b0 2024-02-19T21:11:44.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facac117180 con 0x7facac1031b0 2024-02-19T21:11:44.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7faca4002fe0 con 0x7facac1031b0 2024-02-19T21:11:44.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3326818667 0 0) 0x7fac940033f0 con 0x7facac0ff670 2024-02-19T21:11:44.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facac119560 con 0x7facac0ff670 2024-02-19T21:11:44.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7faca00019d0 con 0x7facac0ff670 2024-02-19T21:11:44.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3498073966 0 0) 0x7facac118360 con 0x7facac0fc870 2024-02-19T21:11:44.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac940033f0 con 0x7facac0fc870 2024-02-19T21:11:44.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fac9c001910 con 0x7facac0fc870 2024-02-19T21:11:44.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2357332154 0 0) 0x7facac119560 con 0x7facac0ff670 2024-02-19T21:11:44.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 >> v1:172.21.15.193:6789/0 conn(0x7facac1031b0 legacy=0x7facac115840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:44.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 >> v1:172.21.15.163:6789/0 conn(0x7facac0fc870 legacy=0x7facac060bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:44.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac11a760 con 0x7facac0ff670 2024-02-19T21:11:44.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7facb1905700 1 -- 172.21.15.163:0/1766486745 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facac117370 con 0x7facac0ff670 2024-02-19T21:11:44.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7facb1905700 10 client.49994.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:44.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7facb1905700 1 -- 172.21.15.163:0/1766486745 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7facac1177e0 con 0x7facac0ff670 2024-02-19T21:11:44.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7facb1905700 1 librados: init done 2024-02-19T21:11:44.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7faca0002340 con 0x7facac0ff670 2024-02-19T21:11:44.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.912+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7faca0005950 con 0x7facac0ff670 2024-02-19T21:11:44.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.913+0000 7facb1905700 1 -- 172.21.15.163:0/1766486745 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac98004f20 con 0x7facac0ff670 2024-02-19T21:11:44.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.914+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7faca00187e0 con 0x7facac0ff670 2024-02-19T21:11:44.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.915+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(472..472 src has 43..472) v4 ==== 8092+0+0 (unknown 818809861 0 0) 0x7faca0078bf0 con 0x7facac0ff670 2024-02-19T21:11:44.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.915+0000 7fac93fff700 10 client.49994.objecter ms_dispatch 0x7facac001550 osd_map(472..472 src has 43..472) v4 2024-02-19T21:11:44.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.915+0000 7fac93fff700 3 client.49994.objecter handle_osd_map got epochs [472,472] > 0 2024-02-19T21:11:44.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.915+0000 7fac93fff700 3 client.49994.objecter handle_osd_map decoding full epoch 472 2024-02-19T21:11:44.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.915+0000 7fac93fff700 20 client.49994.objecter dump_active .. 0 homeless 2024-02-19T21:11:44.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.915+0000 7fac93fff700 10 client.49994.objecter ms_handle_connect 0x7fac94063d20 2024-02-19T21:11:44.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:44.919+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7faca004c020 con 0x7facac0ff670 2024-02-19T21:11:44.944 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:44.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:44.968 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:45.047 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:11:45.076 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:45.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:45.114+0000 7facb1905700 1 -- 172.21.15.163:0/1766486745 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"} v 0) v1 -- 0x7fac980051d0 con 0x7facac0ff670 2024-02-19T21:11:45.224 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:11:45.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:45.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:11:45.327 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:11:45.337 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-19T21:11:45.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:45.336+0000 7fac93fff700 1 -- 172.21.15.163:0/1766486745 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]=0 set pool 2 pg_num to 42 v473) v1 ==== 134+0+0 (unknown 3773956312 0 0) 0x7faca004a430 con 0x7facac0ff670 2024-02-19T21:11:45.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:45.339+0000 7facb1905700 10 librados: watch_flush enter 2024-02-19T21:11:45.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:45.339+0000 7facb1905700 10 librados: watch_flush exit 2024-02-19T21:11:45.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:45.339+0000 7facb1905700 20 client.49994.objecter shutdown clearing up homeless session... 2024-02-19T21:11:45.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:45.339+0000 7facb1905700 10 client.49994.objecter successfully canceled tick 2024-02-19T21:11:45.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:45.339+0000 7facb1905700 1 -- 172.21.15.163:0/1766486745 >> v1:172.21.15.163:6800/1552514681 conn(0x7fac94063d20 legacy=0x7fac940661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:45.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:45.339+0000 7facb1905700 1 -- 172.21.15.163:0/1766486745 >> v1:172.21.15.163:6790/0 conn(0x7facac0ff670 legacy=0x7facac0612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:45.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:45.339+0000 7facb1905700 1 -- 172.21.15.163:0/1766486745 shutdown_connections 2024-02-19T21:11:45.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:45.339+0000 7facb1905700 1 -- 172.21.15.163:0/1766486745 >> 172.21.15.163:0/1766486745 conn(0x7facac0f2470 msgr2=0x7facac106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:45.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:45.339+0000 7facb1905700 1 -- 172.21.15.163:0/1766486745 shutdown_connections 2024-02-19T21:11:45.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:45.340+0000 7facb1905700 1 -- 172.21.15.163:0/1766486745 wait complete. 2024-02-19T21:11:45.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:45.340+0000 7facb1905700 1 librados: shutdown 2024-02-19T21:11:45.340 INFO:teuthology.orchestra.run.smithi163.stderr:set pool 2 pg_num to 42 2024-02-19T21:11:45.359 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:45.472 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:43.542904+0000 osd.7 (osd.7) 223 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:11:45.472 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:43.546379+0000 osd.7 (osd.7) 224 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:11:45.472 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:43.924619+0000 osd.5 (osd.5) 200 : cluster [DBG] 61.d scrub starts 2024-02-19T21:11:45.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:43.928111+0000 osd.5 (osd.5) 201 : cluster [DBG] 61.d scrub ok 2024-02-19T21:11:45.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:44.418609+0000 osd.4 (osd.4) 175 : cluster [DBG] 4.d scrub starts 2024-02-19T21:11:45.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:44.421028+0000 osd.4 (osd.4) 176 : cluster [DBG] 4.d scrub ok 2024-02-19T21:11:45.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:45.116148+0000 mon.c (mon.2) 400 : audit [INF] from='client.? v1:172.21.15.163:0/1766486745' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2024-02-19T21:11:45.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:45.116514+0000 mon.a (mon.0) 2670 : audit [INF] from='client.49994 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2024-02-19T21:11:45.473 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:43.542904+0000 osd.7 (osd.7) 223 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:11:45.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:43.546379+0000 osd.7 (osd.7) 224 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:11:45.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:43.924619+0000 osd.5 (osd.5) 200 : cluster [DBG] 61.d scrub starts 2024-02-19T21:11:45.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:43.928111+0000 osd.5 (osd.5) 201 : cluster [DBG] 61.d scrub ok 2024-02-19T21:11:45.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:44.418609+0000 osd.4 (osd.4) 175 : cluster [DBG] 4.d scrub starts 2024-02-19T21:11:45.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:44.421028+0000 osd.4 (osd.4) 176 : cluster [DBG] 4.d scrub ok 2024-02-19T21:11:45.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:45.116148+0000 mon.c (mon.2) 400 : audit [INF] from='client.? v1:172.21.15.163:0/1766486745' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2024-02-19T21:11:45.474 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:45.116514+0000 mon.a (mon.0) 2670 : audit [INF] from='client.49994 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2024-02-19T21:11:45.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:43.542904+0000 osd.7 (osd.7) 223 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:11:45.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:43.546379+0000 osd.7 (osd.7) 224 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:11:45.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:43.924619+0000 osd.5 (osd.5) 200 : cluster [DBG] 61.d scrub starts 2024-02-19T21:11:45.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:43.928111+0000 osd.5 (osd.5) 201 : cluster [DBG] 61.d scrub ok 2024-02-19T21:11:45.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:44.418609+0000 osd.4 (osd.4) 175 : cluster [DBG] 4.d scrub starts 2024-02-19T21:11:45.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:44.421028+0000 osd.4 (osd.4) 176 : cluster [DBG] 4.d scrub ok 2024-02-19T21:11:45.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:45.116148+0000 mon.c (mon.2) 400 : audit [INF] from='client.? v1:172.21.15.163:0/1766486745' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2024-02-19T21:11:45.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:45.116514+0000 mon.a (mon.0) 2670 : audit [INF] from='client.49994 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]: dispatch 2024-02-19T21:11:45.597 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:11:46.332 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:44.771977+0000 osd.6 (osd.6) 203 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:11:46.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:44.771977+0000 osd.6 (osd.6) 203 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:11:46.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:44.774852+0000 osd.6 (osd.6) 204 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:11:46.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:44.962124+0000 osd.5 (osd.5) 202 : cluster [DBG] 61.1c scrub starts 2024-02-19T21:11:46.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:44.965802+0000 osd.5 (osd.5) 203 : cluster [DBG] 61.1c scrub ok 2024-02-19T21:11:46.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:45.240408+0000 mgr.y (mgr.14550) 437 : cluster [DBG] pgmap v706: 228 pgs: 1 active+clean+scrubbing, 227 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 738 B/s wr, 2 op/s 2024-02-19T21:11:46.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:45.337563+0000 mon.a (mon.0) 2671 : audit [INF] from='client.49994 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]': finished 2024-02-19T21:11:46.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:45.337628+0000 mon.a (mon.0) 2672 : cluster [DBG] osdmap e473: 8 total, 8 up, 8 in 2024-02-19T21:11:46.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:45.403310+0000 osd.3 (osd.3) 215 : cluster [DBG] 5.13 scrub starts 2024-02-19T21:11:46.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:45.404238+0000 osd.3 (osd.3) 216 : cluster [DBG] 5.13 scrub ok 2024-02-19T21:11:46.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:44.774852+0000 osd.6 (osd.6) 204 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:11:46.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:44.962124+0000 osd.5 (osd.5) 202 : cluster [DBG] 61.1c scrub starts 2024-02-19T21:11:46.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:44.965802+0000 osd.5 (osd.5) 203 : cluster [DBG] 61.1c scrub ok 2024-02-19T21:11:46.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:45.240408+0000 mgr.y (mgr.14550) 437 : cluster [DBG] pgmap v706: 228 pgs: 1 active+clean+scrubbing, 227 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 738 B/s wr, 2 op/s 2024-02-19T21:11:46.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:45.337563+0000 mon.a (mon.0) 2671 : audit [INF] from='client.49994 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]': finished 2024-02-19T21:11:46.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:45.337628+0000 mon.a (mon.0) 2672 : cluster [DBG] osdmap e473: 8 total, 8 up, 8 in 2024-02-19T21:11:46.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:45.403310+0000 osd.3 (osd.3) 215 : cluster [DBG] 5.13 scrub starts 2024-02-19T21:11:46.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:45.404238+0000 osd.3 (osd.3) 216 : cluster [DBG] 5.13 scrub ok 2024-02-19T21:11:46.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:44.771977+0000 osd.6 (osd.6) 203 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:11:46.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:44.774852+0000 osd.6 (osd.6) 204 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:11:46.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:44.962124+0000 osd.5 (osd.5) 202 : cluster [DBG] 61.1c scrub starts 2024-02-19T21:11:46.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:44.965802+0000 osd.5 (osd.5) 203 : cluster [DBG] 61.1c scrub ok 2024-02-19T21:11:46.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:45.240408+0000 mgr.y (mgr.14550) 437 : cluster [DBG] pgmap v706: 228 pgs: 1 active+clean+scrubbing, 227 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 738 B/s wr, 2 op/s 2024-02-19T21:11:46.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:45.337563+0000 mon.a (mon.0) 2671 : audit [INF] from='client.49994 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "42"}]': finished 2024-02-19T21:11:46.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:45.337628+0000 mon.a (mon.0) 2672 : cluster [DBG] osdmap e473: 8 total, 8 up, 8 in 2024-02-19T21:11:46.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:45.403310+0000 osd.3 (osd.3) 215 : cluster [DBG] 5.13 scrub starts 2024-02-19T21:11:46.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:45.404238+0000 osd.3 (osd.3) 216 : cluster [DBG] 5.13 scrub ok 2024-02-19T21:11:46.916 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:11:46.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:46.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:11:46.927 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:47.021 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:47.051 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:47.078 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:47.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:47.182 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:11:47.210 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:47.335 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:11:47.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:47.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:11:47.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:46.007095+0000 osd.5 (osd.5) 204 : cluster [DBG] 61.19 scrub starts 2024-02-19T21:11:47.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:46.010755+0000 osd.5 (osd.5) 205 : cluster [DBG] 61.19 scrub ok 2024-02-19T21:11:47.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:46.405303+0000 osd.3 (osd.3) 217 : cluster [DBG] 16.14 scrub starts 2024-02-19T21:11:47.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:46.406397+0000 osd.3 (osd.3) 218 : cluster [DBG] 16.14 scrub ok 2024-02-19T21:11:47.418 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:47.241352+0000 mon.b (mon.1) 625 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-19T21:11:47.418 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:47.242238+0000 mon.a (mon.0) 2673 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-19T21:11:47.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:46.007095+0000 osd.5 (osd.5) 204 : cluster [DBG] 61.19 scrub starts 2024-02-19T21:11:47.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:46.010755+0000 osd.5 (osd.5) 205 : cluster [DBG] 61.19 scrub ok 2024-02-19T21:11:47.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:46.405303+0000 osd.3 (osd.3) 217 : cluster [DBG] 16.14 scrub starts 2024-02-19T21:11:47.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:46.406397+0000 osd.3 (osd.3) 218 : cluster [DBG] 16.14 scrub ok 2024-02-19T21:11:47.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:47.241352+0000 mon.b (mon.1) 625 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-19T21:11:47.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:47.242238+0000 mon.a (mon.0) 2673 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-19T21:11:47.439 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:47.454 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:11:47.454 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:11:47.454 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-02-19T21:11:47.454 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-02-19T21:11:47.454 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:11:47.464 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:46.007095+0000 osd.5 (osd.5) 204 : cluster [DBG] 61.19 scrub starts 2024-02-19T21:11:47.464 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:46.010755+0000 osd.5 (osd.5) 205 : cluster [DBG] 61.19 scrub ok 2024-02-19T21:11:47.464 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:46.405303+0000 osd.3 (osd.3) 217 : cluster [DBG] 16.14 scrub starts 2024-02-19T21:11:47.464 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:46.406397+0000 osd.3 (osd.3) 218 : cluster [DBG] 16.14 scrub ok 2024-02-19T21:11:47.464 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:47.241352+0000 mon.b (mon.1) 625 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-19T21:11:47.464 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:47.242238+0000 mon.a (mon.0) 2673 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]: dispatch 2024-02-19T21:11:47.466 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:47.486 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:47.488 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:47.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:47.590 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:47.616 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:47.740 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:47.742 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:47.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:47.843 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:11:47.871 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:48.070 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:11:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:11:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:11:48.163 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-19T21:11:48.219 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:11:48.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:48.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:11:48.323 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:48.354 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:48.360 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:46.952370+0000 osd.1 (osd.1) 233 : cluster [DBG] 4.b deep-scrub starts 2024-02-19T21:11:48.360 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:46.956418+0000 osd.1 (osd.1) 234 : cluster [DBG] 4.b deep-scrub ok 2024-02-19T21:11:48.360 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:47.047255+0000 osd.5 (osd.5) 206 : cluster [DBG] 4.8 deep-scrub starts 2024-02-19T21:11:48.360 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:47.051818+0000 osd.5 (osd.5) 207 : cluster [DBG] 4.8 deep-scrub ok 2024-02-19T21:11:48.378 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:48.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:48.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:48.482 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:48.486 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:11:48.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.659+0000 7fa8f3859700 1 -- 172.21.15.163:0/2376717766 >> v1:172.21.15.163:6790/0 conn(0x7fa8ec1042c0 legacy=0x7fa8ec106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:48.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.659+0000 7fa8f3859700 1 -- 172.21.15.163:0/2376717766 shutdown_connections 2024-02-19T21:11:48.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.659+0000 7fa8f3859700 1 -- 172.21.15.163:0/2376717766 >> 172.21.15.163:0/2376717766 conn(0x7fa8ec0f35a0 msgr2=0x7fa8ec0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:48.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.659+0000 7fa8f3859700 1 -- 172.21.15.163:0/2376717766 shutdown_connections 2024-02-19T21:11:48.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.659+0000 7fa8f3859700 1 -- 172.21.15.163:0/2376717766 wait complete. 2024-02-19T21:11:48.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.659+0000 7fa8f3859700 1 librados: starting msgr at 2024-02-19T21:11:48.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.659+0000 7fa8f3859700 1 librados: starting objecter 2024-02-19T21:11:48.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8f3859700 1 Processor -- start 2024-02-19T21:11:48.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8f3859700 1 -- start start 2024-02-19T21:11:48.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8f3859700 1 librados: setting wanted keys 2024-02-19T21:11:48.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8f3859700 1 librados: calling monclient init 2024-02-19T21:11:48.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8f3859700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa8ec117390 con 0x7fa8ec0fd980 2024-02-19T21:11:48.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8f3859700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa8ec118570 con 0x7fa8ec100780 2024-02-19T21:11:48.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8f3859700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa8ec119770 con 0x7fa8ec1042c0 2024-02-19T21:11:48.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8f15f5700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fa8ec0fd980 0x7fa8ec0f9ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43452/0 (socket says 172.21.15.163:43452) 2024-02-19T21:11:48.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8f15f5700 1 -- 172.21.15.163:0/1646293344 learned_addr learned my addr 172.21.15.163:0/1646293344 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:48.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8f1df6700 1 --1- 172.21.15.163:0/1646293344 >> v1:172.21.15.163:6790/0 conn(0x7fa8ec1042c0 0x7fa8ec115a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:43156/0 (socket says 172.21.15.163:43156) 2024-02-19T21:11:48.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 10 client.?.objecter ms_handle_connect 0x7fa8ec0fd980 2024-02-19T21:11:48.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:48.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 10 client.?.objecter ms_handle_connect 0x7fa8ec1042c0 2024-02-19T21:11:48.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:48.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 10 client.?.objecter ms_handle_connect 0x7fa8ec100780 2024-02-19T21:11:48.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:48.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282531321 0 0) 0x7fa8ec119770 con 0x7fa8ec1042c0 2024-02-19T21:11:48.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d8003030 con 0x7fa8ec1042c0 2024-02-19T21:11:48.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500185447 0 0) 0x7fa8ec117390 con 0x7fa8ec0fd980 2024-02-19T21:11:48.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8ec119770 con 0x7fa8ec0fd980 2024-02-19T21:11:48.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791099188 0 0) 0x7fa8ec118570 con 0x7fa8ec100780 2024-02-19T21:11:48.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8ec117390 con 0x7fa8ec100780 2024-02-19T21:11:48.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 194946839 0 0) 0x7fa8ec119770 con 0x7fa8ec0fd980 2024-02-19T21:11:48.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8ec118570 con 0x7fa8ec0fd980 2024-02-19T21:11:48.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3976451447 0 0) 0x7fa8d8003030 con 0x7fa8ec1042c0 2024-02-19T21:11:48.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8ec119770 con 0x7fa8ec1042c0 2024-02-19T21:11:48.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa8dc0019f0 con 0x7fa8ec0fd980 2024-02-19T21:11:48.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa8e4002470 con 0x7fa8ec1042c0 2024-02-19T21:11:48.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 962609653 0 0) 0x7fa8ec117390 con 0x7fa8ec100780 2024-02-19T21:11:48.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8d8003030 con 0x7fa8ec100780 2024-02-19T21:11:48.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.660+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa8e80017c0 con 0x7fa8ec100780 2024-02-19T21:11:48.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.661+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 696257662 0 0) 0x7fa8ec118570 con 0x7fa8ec0fd980 2024-02-19T21:11:48.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.661+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 >> v1:172.21.15.163:6790/0 conn(0x7fa8ec1042c0 legacy=0x7fa8ec115a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:48.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.661+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 >> v1:172.21.15.193:6789/0 conn(0x7fa8ec100780 legacy=0x7fa8ec0fa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:48.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.661+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec11a970 con 0x7fa8ec0fd980 2024-02-19T21:11:48.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.661+0000 7fa8f3859700 1 -- 172.21.15.163:0/1646293344 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8ec119960 con 0x7fa8ec0fd980 2024-02-19T21:11:48.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.661+0000 7fa8f3859700 10 client.23280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:48.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.661+0000 7fa8f3859700 1 -- 172.21.15.163:0/1646293344 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8ec119dd0 con 0x7fa8ec0fd980 2024-02-19T21:11:48.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.661+0000 7fa8f3859700 1 librados: init done 2024-02-19T21:11:48.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.661+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fa8dc002520 con 0x7fa8ec0fd980 2024-02-19T21:11:48.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.661+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa8dc005b70 con 0x7fa8ec0fd980 2024-02-19T21:11:48.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.662+0000 7fa8f3859700 1 -- 172.21.15.163:0/1646293344 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d0004f20 con 0x7fa8ec0fd980 2024-02-19T21:11:48.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.662+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fa8dc002080 con 0x7fa8ec0fd980 2024-02-19T21:11:48.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.662+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(474..474 src has 43..474) v4 ==== 8092+0+0 (unknown 1581873544 0 0) 0x7fa8dc078da0 con 0x7fa8ec0fd980 2024-02-19T21:11:48.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.663+0000 7fa8e27fc700 10 client.23280.objecter ms_dispatch 0x7fa8ec001550 osd_map(474..474 src has 43..474) v4 2024-02-19T21:11:48.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.663+0000 7fa8e27fc700 3 client.23280.objecter handle_osd_map got epochs [474,474] > 0 2024-02-19T21:11:48.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.663+0000 7fa8e27fc700 3 client.23280.objecter handle_osd_map decoding full epoch 474 2024-02-19T21:11:48.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.663+0000 7fa8e27fc700 20 client.23280.objecter dump_active .. 0 homeless 2024-02-19T21:11:48.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.663+0000 7fa8e27fc700 10 client.23280.objecter ms_handle_connect 0x7fa8d8063c90 2024-02-19T21:11:48.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.667+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa8dc04c1d0 con 0x7fa8ec0fd980 2024-02-19T21:11:48.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:47.843+0000 7fa8f3859700 1 -- 172.21.15.163:0/1646293344 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fa8d00051d0 con 0x7fa8ec0fd980 2024-02-19T21:11:48.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:48.350+0000 7fa8e27fc700 1 -- 172.21.15.163:0/1646293344 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v475) v1 ==== 87+0+0 (unknown 2089373202 0 0) 0x7fa8dc04a5e0 con 0x7fa8ec0fd980 2024-02-19T21:11:48.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:48.356+0000 7fa8f3859700 10 librados: watch_flush enter 2024-02-19T21:11:48.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:48.357+0000 7fa8f3859700 10 librados: watch_flush exit 2024-02-19T21:11:48.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:48.357+0000 7fa8f3859700 20 client.23280.objecter shutdown clearing up homeless session... 2024-02-19T21:11:48.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:48.357+0000 7fa8f3859700 10 client.23280.objecter successfully canceled tick 2024-02-19T21:11:48.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:48.360+0000 7fa8f3859700 1 -- 172.21.15.163:0/1646293344 >> v1:172.21.15.163:6800/1552514681 conn(0x7fa8d8063c90 legacy=0x7fa8d8066150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:48.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:48.360+0000 7fa8f3859700 1 -- 172.21.15.163:0/1646293344 >> v1:172.21.15.163:6789/0 conn(0x7fa8ec0fd980 legacy=0x7fa8ec0f9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:48.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:48.360+0000 7fa8f3859700 1 -- 172.21.15.163:0/1646293344 shutdown_connections 2024-02-19T21:11:48.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:48.360+0000 7fa8f3859700 1 -- 172.21.15.163:0/1646293344 >> 172.21.15.163:0/1646293344 conn(0x7fa8ec0f35a0 msgr2=0x7fa8ec107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:48.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:48.362+0000 7fa8f3859700 1 -- 172.21.15.163:0/1646293344 shutdown_connections 2024-02-19T21:11:48.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:48.362+0000 7fa8f3859700 1 -- 172.21.15.163:0/1646293344 wait complete. 2024-02-19T21:11:48.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:48.362+0000 7fa8f3859700 1 librados: shutdown 2024-02-19T21:11:48.514 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:11:48.514 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:48.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:46.952370+0000 osd.1 (osd.1) 233 : cluster [DBG] 4.b deep-scrub starts 2024-02-19T21:11:48.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:46.956418+0000 osd.1 (osd.1) 234 : cluster [DBG] 4.b deep-scrub ok 2024-02-19T21:11:48.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:47.047255+0000 osd.5 (osd.5) 206 : cluster [DBG] 4.8 deep-scrub starts 2024-02-19T21:11:48.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:47.051818+0000 osd.5 (osd.5) 207 : cluster [DBG] 4.8 deep-scrub ok 2024-02-19T21:11:48.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:47.240813+0000 mgr.y (mgr.14550) 438 : cluster [DBG] pgmap v708: 228 pgs: 1 active+clean+scrubbing, 227 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 645 B/s wr, 2 op/s 2024-02-19T21:11:48.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:47.347363+0000 mon.a (mon.0) 2674 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]': finished 2024-02-19T21:11:48.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:47.347446+0000 mon.a (mon.0) 2675 : cluster [DBG] osdmap e474: 8 total, 8 up, 8 in 2024-02-19T21:11:48.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:47.844750+0000 mon.a (mon.0) 2676 : audit [INF] from='client.? v1:172.21.15.163:0/1646293344' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:11:48.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:47.240813+0000 mgr.y (mgr.14550) 438 : cluster [DBG] pgmap v708: 228 pgs: 1 active+clean+scrubbing, 227 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 645 B/s wr, 2 op/s 2024-02-19T21:11:48.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:47.347363+0000 mon.a (mon.0) 2674 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]': finished 2024-02-19T21:11:48.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:47.347446+0000 mon.a (mon.0) 2675 : cluster [DBG] osdmap e474: 8 total, 8 up, 8 in 2024-02-19T21:11:48.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:47.844750+0000 mon.a (mon.0) 2676 : audit [INF] from='client.? v1:172.21.15.163:0/1646293344' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:11:48.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:46.952370+0000 osd.1 (osd.1) 233 : cluster [DBG] 4.b deep-scrub starts 2024-02-19T21:11:48.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:46.956418+0000 osd.1 (osd.1) 234 : cluster [DBG] 4.b deep-scrub ok 2024-02-19T21:11:48.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:47.047255+0000 osd.5 (osd.5) 206 : cluster [DBG] 4.8 deep-scrub starts 2024-02-19T21:11:48.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:47.051818+0000 osd.5 (osd.5) 207 : cluster [DBG] 4.8 deep-scrub ok 2024-02-19T21:11:48.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:47.240813+0000 mgr.y (mgr.14550) 438 : cluster [DBG] pgmap v708: 228 pgs: 1 active+clean+scrubbing, 227 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 645 B/s wr, 2 op/s 2024-02-19T21:11:48.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:47.347363+0000 mon.a (mon.0) 2674 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "42"}]': finished 2024-02-19T21:11:48.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:47.347446+0000 mon.a (mon.0) 2675 : cluster [DBG] osdmap e474: 8 total, 8 up, 8 in 2024-02-19T21:11:48.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:47.844750+0000 mon.a (mon.0) 2676 : audit [INF] from='client.? v1:172.21.15.163:0/1646293344' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:11:48.760 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:48.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:48.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:48.864 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:48.891 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:49.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:48.341877+0000 mon.a (mon.0) 2677 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:11:49.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:48.342888+0000 osd.2 (osd.2) 194 : cluster [DBG] 61.12 deep-scrub starts 2024-02-19T21:11:49.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:48.351097+0000 mon.a (mon.0) 2678 : audit [INF] from='client.? v1:172.21.15.163:0/1646293344' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:11:49.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:48.351155+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e475: 8 total, 8 up, 8 in 2024-02-19T21:11:49.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:48.360116+0000 osd.2 (osd.2) 195 : cluster [DBG] 61.12 deep-scrub ok 2024-02-19T21:11:49.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:49.241852+0000 mon.b (mon.1) 626 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-19T21:11:49.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:49.242821+0000 mon.a (mon.0) 2680 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-19T21:11:49.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:49.295839+0000 mon.a (mon.0) 2681 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]': finished 2024-02-19T21:11:49.419 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:49.295914+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e476: 8 total, 8 up, 8 in 2024-02-19T21:11:49.419 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:48.341877+0000 mon.a (mon.0) 2677 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:11:49.419 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:48.342888+0000 osd.2 (osd.2) 194 : cluster [DBG] 61.12 deep-scrub starts 2024-02-19T21:11:49.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:48.351097+0000 mon.a (mon.0) 2678 : audit [INF] from='client.? v1:172.21.15.163:0/1646293344' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:11:49.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:48.351155+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e475: 8 total, 8 up, 8 in 2024-02-19T21:11:49.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:48.360116+0000 osd.2 (osd.2) 195 : cluster [DBG] 61.12 deep-scrub ok 2024-02-19T21:11:49.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:49.241852+0000 mon.b (mon.1) 626 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-19T21:11:49.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:49.242821+0000 mon.a (mon.0) 2680 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-19T21:11:49.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:49.295839+0000 mon.a (mon.0) 2681 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]': finished 2024-02-19T21:11:49.420 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:49.295914+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e476: 8 total, 8 up, 8 in 2024-02-19T21:11:49.440 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:49.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:49.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:49.544 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:49.574 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:49.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.654+0000 7f5237508700 1 -- 172.21.15.163:0/2597821309 >> v1:172.21.15.163:6790/0 conn(0x7f52300ff680 legacy=0x7f5230101a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:49.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.655+0000 7f5237508700 1 -- 172.21.15.163:0/2597821309 shutdown_connections 2024-02-19T21:11:49.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.655+0000 7f5237508700 1 -- 172.21.15.163:0/2597821309 >> 172.21.15.163:0/2597821309 conn(0x7f523006fdc0 msgr2=0x7f52300f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:49.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.655+0000 7f5237508700 1 -- 172.21.15.163:0/2597821309 shutdown_connections 2024-02-19T21:11:49.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.655+0000 7f5237508700 1 -- 172.21.15.163:0/2597821309 wait complete. 2024-02-19T21:11:49.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.655+0000 7f5237508700 1 librados: starting msgr at 2024-02-19T21:11:49.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.655+0000 7f5237508700 1 librados: starting objecter 2024-02-19T21:11:49.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.655+0000 7f5237508700 1 Processor -- start 2024-02-19T21:11:49.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.655+0000 7f5237508700 1 -- start start 2024-02-19T21:11:49.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.655+0000 7f5237508700 1 librados: setting wanted keys 2024-02-19T21:11:49.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.655+0000 7f5237508700 1 librados: calling monclient init 2024-02-19T21:11:49.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.655+0000 7f5237508700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5230119500 con 0x7f52300ff680 2024-02-19T21:11:49.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.655+0000 7f5237508700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f523011a6e0 con 0x7f52300fc8a0 2024-02-19T21:11:49.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.655+0000 7f5237508700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f523011b8e0 con 0x7f52301031c0 2024-02-19T21:11:49.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.655+0000 7f5234aa3700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f52300ff680 0x7f52300fbe10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43472/0 (socket says 172.21.15.163:43472) 2024-02-19T21:11:49.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f5234aa3700 1 -- 172.21.15.163:0/1365831279 learned_addr learned my addr 172.21.15.163:0/1365831279 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:49.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 10 client.?.objecter ms_handle_connect 0x7f52300ff680 2024-02-19T21:11:49.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:49.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 10 client.?.objecter ms_handle_connect 0x7f52301031c0 2024-02-19T21:11:49.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:49.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 10 client.?.objecter ms_handle_connect 0x7f52300fc8a0 2024-02-19T21:11:49.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:49.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617202852 0 0) 0x7f5230119500 con 0x7f52300ff680 2024-02-19T21:11:49.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f521c0038e0 con 0x7f52300ff680 2024-02-19T21:11:49.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892592141 0 0) 0x7f523011b8e0 con 0x7f52301031c0 2024-02-19T21:11:49.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5230119500 con 0x7f52301031c0 2024-02-19T21:11:49.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875845145 0 0) 0x7f523011a6e0 con 0x7f52300fc8a0 2024-02-19T21:11:49.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f523011b8e0 con 0x7f52300fc8a0 2024-02-19T21:11:49.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1423440408 0 0) 0x7f521c0038e0 con 0x7f52300ff680 2024-02-19T21:11:49.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f523011a6e0 con 0x7f52300ff680 2024-02-19T21:11:49.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f522c002d10 con 0x7f52300ff680 2024-02-19T21:11:49.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 488925145 0 0) 0x7f523011b8e0 con 0x7f52300fc8a0 2024-02-19T21:11:49.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f521c0038e0 con 0x7f52300fc8a0 2024-02-19T21:11:49.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3540587233 0 0) 0x7f5230119500 con 0x7f52301031c0 2024-02-19T21:11:49.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f523011b8e0 con 0x7f52301031c0 2024-02-19T21:11:49.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f52200017c0 con 0x7f52300fc8a0 2024-02-19T21:11:49.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f52280019a0 con 0x7f52301031c0 2024-02-19T21:11:49.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 538975409 0 0) 0x7f523011a6e0 con 0x7f52300ff680 2024-02-19T21:11:49.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 >> v1:172.21.15.163:6790/0 conn(0x7f52301031c0 legacy=0x7f5230117c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:49.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 >> v1:172.21.15.193:6789/0 conn(0x7f52300fc8a0 legacy=0x7f52300fb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:49.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523011cae0 con 0x7f52300ff680 2024-02-19T21:11:49.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523011bad0 con 0x7f52300ff680 2024-02-19T21:11:49.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f5237508700 10 client.23286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:49.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f523011bfc0 con 0x7f52300ff680 2024-02-19T21:11:49.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f5237508700 1 librados: init done 2024-02-19T21:11:49.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f522c002370 con 0x7f52300ff680 2024-02-19T21:11:49.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.656+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f522c003870 con 0x7f52300ff680 2024-02-19T21:11:49.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.657+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f5214004f00 con 0x7f52300ff680 2024-02-19T21:11:49.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f522c0030d0 con 0x7f52300ff680 2024-02-19T21:11:49.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(476..476 src has 43..476) v4 ==== 8092+0+0 (unknown 3988842271 0 0) 0x7f522c078f20 con 0x7f52300ff680 2024-02-19T21:11:49.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 osd_map(476..476 src has 43..476) v4 2024-02-19T21:11:49.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f52267fc700 3 client.23286.objecter handle_osd_map got epochs [476,476] > 0 2024-02-19T21:11:49.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f52267fc700 3 client.23286.objecter handle_osd_map decoding full epoch 476 2024-02-19T21:11:49.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f52267fc700 20 client.23286.objecter dump_active .. 0 homeless 2024-02-19T21:11:49.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v476) v1 ==== 54+0+16 (unknown 3066237151 0 2420000192) 0x7f522c07b350 con 0x7f52300ff680 2024-02-19T21:11:49.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f52267fc700 10 client.23286.objecter ms_handle_connect 0x7f521c063e20 2024-02-19T21:11:49.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f5237508700 10 client.23286.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:49.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 1 2024-02-19T21:11:49.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f52140051a0 osd=0 3 2024-02-19T21:11:49.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f52140051a0 osd=0 3 2024-02-19T21:11:49.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f52140051a0 osd=0 3 2024-02-19T21:11:49.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 1 2024-02-19T21:11:49.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f52140051a0 osd=0 3 2024-02-19T21:11:49.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 0 1 2024-02-19T21:11:49.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f52140051a0 osd=0 4 2024-02-19T21:11:49.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f5237508700 10 client.23286.objecter _send_command 1 2024-02-19T21:11:49.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.658+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5214004950 con 0x7f5214004530 2024-02-19T21:11:49.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.659+0000 7f52267fc700 10 client.23286.objecter ms_handle_connect 0x7f5214004530 2024-02-19T21:11:49.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.659+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5214004950 con 0x7f5214004530 2024-02-19T21:11:49.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.659+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:11:49.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.659+0000 7f52267fc700 10 client.23286.objecter _finish_command 1 = system:0 2024-02-19T21:11:49.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.659+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f52140051a0 osd=0 4 2024-02-19T21:11:49.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.659+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 0 1 2024-02-19T21:11:49.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.674+0000 7f5237508700 10 client.23286.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:49.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.674+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 2 2024-02-19T21:11:49.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.674+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f52140051a0 osd=0 3 2024-02-19T21:11:49.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.674+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f52140051a0 osd=0 3 2024-02-19T21:11:49.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.674+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f52140051a0 osd=0 3 2024-02-19T21:11:49.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.674+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 2 2024-02-19T21:11:49.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.674+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f52140051a0 osd=0 3 2024-02-19T21:11:49.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.674+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 0 2 2024-02-19T21:11:49.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.674+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f52140051a0 osd=0 4 2024-02-19T21:11:49.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.674+0000 7f5237508700 10 client.23286.objecter _send_command 2 2024-02-19T21:11:49.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.674+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5214004d30 con 0x7f5214004530 2024-02-19T21:11:49.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.675+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5214004d30 con 0x7f5214004530 2024-02-19T21:11:49.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.675+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:49.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.675+0000 7f52267fc700 10 client.23286.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.675+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f52140051a0 osd=0 4 2024-02-19T21:11:49.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.675+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 0 2 2024-02-19T21:11:49.675 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:11:49.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.676+0000 7f5237508700 10 client.23286.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:49.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.676+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 3 2024-02-19T21:11:49.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.676+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214003fe0 osd=1 3 2024-02-19T21:11:49.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.676+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214003fe0 osd=1 3 2024-02-19T21:11:49.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.676+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214003fe0 osd=1 3 2024-02-19T21:11:49.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.676+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 3 2024-02-19T21:11:49.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.676+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f5214003fe0 osd=1 3 2024-02-19T21:11:49.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.676+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 1 3 2024-02-19T21:11:49.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.676+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214003fe0 osd=1 4 2024-02-19T21:11:49.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.676+0000 7f5237508700 10 client.23286.objecter _send_command 3 2024-02-19T21:11:49.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.676+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f521400c000 con 0x7f5214004150 2024-02-19T21:11:49.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.676+0000 7f52267fc700 10 client.23286.objecter ms_handle_connect 0x7f5214004150 2024-02-19T21:11:49.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.677+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f521400c000 con 0x7f5214004150 2024-02-19T21:11:49.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.677+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:11:49.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.677+0000 7f52267fc700 10 client.23286.objecter _finish_command 3 = system:0 2024-02-19T21:11:49.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.677+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f5214003fe0 osd=1 4 2024-02-19T21:11:49.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.677+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 1 3 2024-02-19T21:11:49.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.691+0000 7f5237508700 10 client.23286.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:49.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.691+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 4 2024-02-19T21:11:49.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.691+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214003fe0 osd=1 3 2024-02-19T21:11:49.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.691+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214003fe0 osd=1 3 2024-02-19T21:11:49.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.691+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214003fe0 osd=1 3 2024-02-19T21:11:49.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.691+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 4 2024-02-19T21:11:49.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.691+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f5214003fe0 osd=1 3 2024-02-19T21:11:49.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.691+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 1 4 2024-02-19T21:11:49.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.691+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214003fe0 osd=1 4 2024-02-19T21:11:49.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.691+0000 7f5237508700 10 client.23286.objecter _send_command 4 2024-02-19T21:11:49.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.691+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f52140018b0 con 0x7f5214004150 2024-02-19T21:11:49.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.692+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f52140018b0 con 0x7f5214004150 2024-02-19T21:11:49.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.692+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:49.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.692+0000 7f52267fc700 10 client.23286.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.692+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f5214003fe0 osd=1 4 2024-02-19T21:11:49.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.692+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 1 4 2024-02-19T21:11:49.692 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:11:49.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.692+0000 7f5237508700 10 client.23286.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:49.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 5 2024-02-19T21:11:49.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214003630 osd=2 3 2024-02-19T21:11:49.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214003630 osd=2 3 2024-02-19T21:11:49.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214003630 osd=2 3 2024-02-19T21:11:49.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 5 2024-02-19T21:11:49.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f5214003630 osd=2 3 2024-02-19T21:11:49.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 2 5 2024-02-19T21:11:49.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214003630 osd=2 4 2024-02-19T21:11:49.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f5237508700 10 client.23286.objecter _send_command 5 2024-02-19T21:11:49.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f521400de90 con 0x7f521400d2f0 2024-02-19T21:11:49.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f52267fc700 10 client.23286.objecter ms_handle_connect 0x7f521400d2f0 2024-02-19T21:11:49.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f521400de90 con 0x7f521400d2f0 2024-02-19T21:11:49.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:11:49.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f52267fc700 10 client.23286.objecter _finish_command 5 = system:0 2024-02-19T21:11:49.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f5214003630 osd=2 4 2024-02-19T21:11:49.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.693+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 2 5 2024-02-19T21:11:49.696 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:48.341877+0000 mon.a (mon.0) 2677 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:11:49.696 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:48.342888+0000 osd.2 (osd.2) 194 : cluster [DBG] 61.12 deep-scrub starts 2024-02-19T21:11:49.696 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:48.351097+0000 mon.a (mon.0) 2678 : audit [INF] from='client.? v1:172.21.15.163:0/1646293344' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:11:49.696 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:48.351155+0000 mon.a (mon.0) 2679 : cluster [DBG] osdmap e475: 8 total, 8 up, 8 in 2024-02-19T21:11:49.696 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:48.360116+0000 osd.2 (osd.2) 195 : cluster [DBG] 61.12 deep-scrub ok 2024-02-19T21:11:49.696 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:49.241852+0000 mon.b (mon.1) 626 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-19T21:11:49.696 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:49.242821+0000 mon.a (mon.0) 2680 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]: dispatch 2024-02-19T21:11:49.696 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:49.295839+0000 mon.a (mon.0) 2681 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "42"}]': finished 2024-02-19T21:11:49.696 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:49.295914+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e476: 8 total, 8 up, 8 in 2024-02-19T21:11:49.697 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:49.700 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:49.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:49.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.707+0000 7f5237508700 10 client.23286.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:49.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.707+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 6 2024-02-19T21:11:49.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.707+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214003630 osd=2 3 2024-02-19T21:11:49.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.707+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214003630 osd=2 3 2024-02-19T21:11:49.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.707+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214003630 osd=2 3 2024-02-19T21:11:49.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.707+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 6 2024-02-19T21:11:49.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.707+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f5214003630 osd=2 3 2024-02-19T21:11:49.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.707+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 2 6 2024-02-19T21:11:49.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.707+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214003630 osd=2 4 2024-02-19T21:11:49.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.707+0000 7f5237508700 10 client.23286.objecter _send_command 6 2024-02-19T21:11:49.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.707+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5214003160 con 0x7f521400d2f0 2024-02-19T21:11:49.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5214003160 con 0x7f521400d2f0 2024-02-19T21:11:49.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:49.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f52267fc700 10 client.23286.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f5214003630 osd=2 4 2024-02-19T21:11:49.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 2 6 2024-02-19T21:11:49.708 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:11:49.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f5237508700 10 client.23286.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:49.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 7 2024-02-19T21:11:49.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214011970 osd=3 3 2024-02-19T21:11:49.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214011970 osd=3 3 2024-02-19T21:11:49.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214011970 osd=3 3 2024-02-19T21:11:49.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 7 2024-02-19T21:11:49.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f5214011970 osd=3 3 2024-02-19T21:11:49.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 3 7 2024-02-19T21:11:49.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214011970 osd=3 4 2024-02-19T21:11:49.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f5237508700 10 client.23286.objecter _send_command 7 2024-02-19T21:11:49.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5214015bb0 con 0x7f5214011ff0 2024-02-19T21:11:49.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.709+0000 7f52267fc700 10 client.23286.objecter ms_handle_connect 0x7f5214011ff0 2024-02-19T21:11:49.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.710+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5214015bb0 con 0x7f5214011ff0 2024-02-19T21:11:49.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.710+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:11:49.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.710+0000 7f52267fc700 10 client.23286.objecter _finish_command 7 = system:0 2024-02-19T21:11:49.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.710+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f5214011970 osd=3 4 2024-02-19T21:11:49.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.710+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 3 7 2024-02-19T21:11:49.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.724+0000 7f5237508700 10 client.23286.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:49.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.724+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 8 2024-02-19T21:11:49.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.724+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214011970 osd=3 3 2024-02-19T21:11:49.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.724+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214011970 osd=3 3 2024-02-19T21:11:49.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.724+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214011970 osd=3 3 2024-02-19T21:11:49.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.724+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 8 2024-02-19T21:11:49.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.724+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f5214011970 osd=3 3 2024-02-19T21:11:49.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.724+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 3 8 2024-02-19T21:11:49.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.724+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214011970 osd=3 4 2024-02-19T21:11:49.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.724+0000 7f5237508700 10 client.23286.objecter _send_command 8 2024-02-19T21:11:49.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.724+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f521400e240 con 0x7f5214011ff0 2024-02-19T21:11:49.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f521400e240 con 0x7f5214011ff0 2024-02-19T21:11:49.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:49.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f52267fc700 10 client.23286.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f5214011970 osd=3 4 2024-02-19T21:11:49.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 3 8 2024-02-19T21:11:49.725 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:11:49.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f5237508700 10 client.23286.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:49.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 9 2024-02-19T21:11:49.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f521400e240 osd=4 3 2024-02-19T21:11:49.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f521400e240 osd=4 3 2024-02-19T21:11:49.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f521400e240 osd=4 3 2024-02-19T21:11:49.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 9 2024-02-19T21:11:49.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f521400e240 osd=4 3 2024-02-19T21:11:49.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 4 9 2024-02-19T21:11:49.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f521400e240 osd=4 4 2024-02-19T21:11:49.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f5237508700 10 client.23286.objecter _send_command 9 2024-02-19T21:11:49.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f521400e530 con 0x7f521400ce50 2024-02-19T21:11:49.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.726+0000 7f52267fc700 10 client.23286.objecter ms_handle_connect 0x7f521400ce50 2024-02-19T21:11:49.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.727+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f521400e530 con 0x7f521400ce50 2024-02-19T21:11:49.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.727+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:11:49.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.727+0000 7f52267fc700 10 client.23286.objecter _finish_command 9 = system:0 2024-02-19T21:11:49.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.727+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f521400e240 osd=4 4 2024-02-19T21:11:49.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.727+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 4 9 2024-02-19T21:11:49.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.741+0000 7f5237508700 10 client.23286.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:49.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.741+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 10 2024-02-19T21:11:49.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.741+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f521400e240 osd=4 3 2024-02-19T21:11:49.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.741+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f521400e240 osd=4 3 2024-02-19T21:11:49.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.741+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f521400e240 osd=4 3 2024-02-19T21:11:49.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.741+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 10 2024-02-19T21:11:49.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.741+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f521400e240 osd=4 3 2024-02-19T21:11:49.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.741+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 4 10 2024-02-19T21:11:49.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.741+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f521400e240 osd=4 4 2024-02-19T21:11:49.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.741+0000 7f5237508700 10 client.23286.objecter _send_command 10 2024-02-19T21:11:49.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.741+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f521400e530 con 0x7f521400ce50 2024-02-19T21:11:49.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f521400e530 con 0x7f521400ce50 2024-02-19T21:11:49.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:49.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f52267fc700 10 client.23286.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f521400e240 osd=4 4 2024-02-19T21:11:49.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 4 10 2024-02-19T21:11:49.743 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:11:49.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f5237508700 10 client.23286.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:49.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 11 2024-02-19T21:11:49.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f521400e530 osd=5 3 2024-02-19T21:11:49.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f521400e530 osd=5 3 2024-02-19T21:11:49.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f521400e530 osd=5 3 2024-02-19T21:11:49.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 11 2024-02-19T21:11:49.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f521400e530 osd=5 3 2024-02-19T21:11:49.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 5 11 2024-02-19T21:11:49.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f521400e530 osd=5 4 2024-02-19T21:11:49.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f5237508700 10 client.23286.objecter _send_command 11 2024-02-19T21:11:49.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.744+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5214016f10 con 0x7f5214016970 2024-02-19T21:11:49.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.745+0000 7f52267fc700 10 client.23286.objecter ms_handle_connect 0x7f5214016970 2024-02-19T21:11:49.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.745+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5214016f10 con 0x7f5214016970 2024-02-19T21:11:49.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.745+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:11:49.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.745+0000 7f52267fc700 10 client.23286.objecter _finish_command 11 = system:0 2024-02-19T21:11:49.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.745+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f521400e530 osd=5 4 2024-02-19T21:11:49.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.745+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 5 11 2024-02-19T21:11:49.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.759+0000 7f5237508700 10 client.23286.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:49.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.759+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 12 2024-02-19T21:11:49.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.759+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f521400e530 osd=5 3 2024-02-19T21:11:49.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.759+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f521400e530 osd=5 3 2024-02-19T21:11:49.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.759+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f521400e530 osd=5 3 2024-02-19T21:11:49.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.759+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 12 2024-02-19T21:11:49.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.759+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f521400e530 osd=5 3 2024-02-19T21:11:49.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.759+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 5 12 2024-02-19T21:11:49.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.759+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f521400e530 osd=5 4 2024-02-19T21:11:49.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.759+0000 7f5237508700 10 client.23286.objecter _send_command 12 2024-02-19T21:11:49.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.759+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5214016f10 con 0x7f5214016970 2024-02-19T21:11:49.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5214016f10 con 0x7f5214016970 2024-02-19T21:11:49.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:49.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f52267fc700 10 client.23286.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f521400e530 osd=5 4 2024-02-19T21:11:49.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 5 12 2024-02-19T21:11:49.761 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:11:49.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f5237508700 10 client.23286.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:49.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 13 2024-02-19T21:11:49.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214016f10 osd=6 3 2024-02-19T21:11:49.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214016f10 osd=6 3 2024-02-19T21:11:49.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214016f10 osd=6 3 2024-02-19T21:11:49.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 13 2024-02-19T21:11:49.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f5214016f10 osd=6 3 2024-02-19T21:11:49.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 6 13 2024-02-19T21:11:49.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214016f10 osd=6 4 2024-02-19T21:11:49.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f5237508700 10 client.23286.objecter _send_command 13 2024-02-19T21:11:49.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.761+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5214022cc0 con 0x7f521401f1b0 2024-02-19T21:11:49.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.762+0000 7f52267fc700 10 client.23286.objecter ms_handle_connect 0x7f521401f1b0 2024-02-19T21:11:49.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.762+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5214022cc0 con 0x7f521401f1b0 2024-02-19T21:11:49.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.762+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:11:49.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.762+0000 7f52267fc700 10 client.23286.objecter _finish_command 13 = system:0 2024-02-19T21:11:49.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.762+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f5214016f10 osd=6 4 2024-02-19T21:11:49.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.762+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 6 13 2024-02-19T21:11:49.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.776+0000 7f5237508700 10 client.23286.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:49.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.777+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 14 2024-02-19T21:11:49.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.777+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214016f10 osd=6 3 2024-02-19T21:11:49.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.777+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214016f10 osd=6 3 2024-02-19T21:11:49.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.777+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214016f10 osd=6 3 2024-02-19T21:11:49.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.777+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 14 2024-02-19T21:11:49.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.777+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f5214016f10 osd=6 3 2024-02-19T21:11:49.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.777+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 6 14 2024-02-19T21:11:49.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.777+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214016f10 osd=6 4 2024-02-19T21:11:49.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.777+0000 7f5237508700 10 client.23286.objecter _send_command 14 2024-02-19T21:11:49.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.777+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5214024470 con 0x7f521401f1b0 2024-02-19T21:11:49.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.778+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5214024470 con 0x7f521401f1b0 2024-02-19T21:11:49.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.778+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:49.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.778+0000 7f52267fc700 10 client.23286.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.778+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f5214016f10 osd=6 4 2024-02-19T21:11:49.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.778+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 6 14 2024-02-19T21:11:49.778 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:11:49.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.779+0000 7f5237508700 10 client.23286.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:49.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.779+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 15 2024-02-19T21:11:49.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.779+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214024470 osd=7 3 2024-02-19T21:11:49.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.779+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214024470 osd=7 3 2024-02-19T21:11:49.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.779+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214024470 osd=7 3 2024-02-19T21:11:49.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.779+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 15 2024-02-19T21:11:49.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.779+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f5214024470 osd=7 3 2024-02-19T21:11:49.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.779+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 7 15 2024-02-19T21:11:49.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.779+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214024470 osd=7 4 2024-02-19T21:11:49.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.779+0000 7f5237508700 10 client.23286.objecter _send_command 15 2024-02-19T21:11:49.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.779+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5214025070 con 0x7f5214024b30 2024-02-19T21:11:49.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.779+0000 7f52267fc700 10 client.23286.objecter ms_handle_connect 0x7f5214024b30 2024-02-19T21:11:49.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.780+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5214025070 con 0x7f5214024b30 2024-02-19T21:11:49.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.780+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:11:49.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.780+0000 7f52267fc700 10 client.23286.objecter _finish_command 15 = system:0 2024-02-19T21:11:49.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.780+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f5214024470 osd=7 4 2024-02-19T21:11:49.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.780+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 7 15 2024-02-19T21:11:49.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.794+0000 7f5237508700 10 client.23286.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:11:49.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.794+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign -1 16 2024-02-19T21:11:49.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.794+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214024470 osd=7 3 2024-02-19T21:11:49.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.794+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214024470 osd=7 3 2024-02-19T21:11:49.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.794+0000 7f5237508700 20 client.23286.objecter _get_session s=0x7f5214024470 osd=7 3 2024-02-19T21:11:49.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.794+0000 7f5237508700 15 client.23286.objecter _session_command_op_remove -1 16 2024-02-19T21:11:49.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.794+0000 7f5237508700 20 client.23286.objecter get_session s=0x7f5214024470 osd=7 3 2024-02-19T21:11:49.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.794+0000 7f5237508700 15 client.23286.objecter _session_command_op_assign 7 16 2024-02-19T21:11:49.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.794+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214024470 osd=7 4 2024-02-19T21:11:49.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.794+0000 7f5237508700 10 client.23286.objecter _send_command 16 2024-02-19T21:11:49.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.794+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5214025070 con 0x7f5214024b30 2024-02-19T21:11:49.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f52267fc700 1 -- 172.21.15.163:0/1365831279 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5214025070 con 0x7f5214024b30 2024-02-19T21:11:49.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f52267fc700 10 client.23286.objecter ms_dispatch 0x7f5230001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:11:49.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f52267fc700 10 client.23286.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f52267fc700 20 client.23286.objecter put_session s=0x7f5214024470 osd=7 4 2024-02-19T21:11:49.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f52267fc700 15 client.23286.objecter _session_command_op_remove 7 16 2024-02-19T21:11:49.796 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:11:49.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 10 librados: watch_flush enter 2024-02-19T21:11:49.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 10 librados: watch_flush exit 2024-02-19T21:11:49.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 10 client.23286.objecter close_session for osd.0 2024-02-19T21:11:49.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 >> v1:172.21.15.163:6801/1559418115 conn(0x7f5214004530 legacy=0x7f5214002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:49.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f52140051a0 osd=0 1 2024-02-19T21:11:49.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 10 client.23286.objecter close_session for osd.1 2024-02-19T21:11:49.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 >> v1:172.21.15.163:6805/3125047432 conn(0x7f5214004150 legacy=0x7f521400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:49.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214003fe0 osd=1 1 2024-02-19T21:11:49.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 10 client.23286.objecter close_session for osd.2 2024-02-19T21:11:49.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 >> v1:172.21.15.163:6809/1563158487 conn(0x7f521400d2f0 legacy=0x7f521400d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:49.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214003630 osd=2 1 2024-02-19T21:11:49.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 10 client.23286.objecter close_session for osd.3 2024-02-19T21:11:49.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 >> v1:172.21.15.163:6813/2112279769 conn(0x7f5214011ff0 legacy=0x7f52140143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:49.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214011970 osd=3 1 2024-02-19T21:11:49.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 10 client.23286.objecter close_session for osd.4 2024-02-19T21:11:49.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 >> v1:172.21.15.193:6800/3129716965 conn(0x7f521400ce50 legacy=0x7f5214015d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:49.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5235aa5700 1 -- 172.21.15.163:0/1365831279 reap_dead start 2024-02-19T21:11:49.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5235aa5700 1 -- 172.21.15.163:0/1365831279 reap_dead start 2024-02-19T21:11:49.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f521400e240 osd=4 1 2024-02-19T21:11:49.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 10 client.23286.objecter close_session for osd.5 2024-02-19T21:11:49.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 >> v1:172.21.15.193:6804/805161991 conn(0x7f5214016970 legacy=0x7f521401d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:49.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f521400e530 osd=5 1 2024-02-19T21:11:49.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 10 client.23286.objecter close_session for osd.6 2024-02-19T21:11:49.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 >> v1:172.21.15.193:6808/787604615 conn(0x7f521401f1b0 legacy=0x7f52140215a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:49.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214016f10 osd=6 1 2024-02-19T21:11:49.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 10 client.23286.objecter close_session for osd.7 2024-02-19T21:11:49.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 >> v1:172.21.15.193:6812/157927432 conn(0x7f5214024b30 legacy=0x7f5214022cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:49.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 20 client.23286.objecter put_session s=0x7f5214024470 osd=7 1 2024-02-19T21:11:49.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 20 client.23286.objecter shutdown clearing up homeless session... 2024-02-19T21:11:49.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 10 client.23286.objecter successfully canceled tick 2024-02-19T21:11:49.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 >> v1:172.21.15.163:6800/1552514681 conn(0x7f521c063e20 legacy=0x7f521c0662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:49.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 >> v1:172.21.15.163:6789/0 conn(0x7f52300ff680 legacy=0x7f52300fbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:49.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5235aa5700 1 -- 172.21.15.163:0/1365831279 reap_dead start 2024-02-19T21:11:49.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 shutdown_connections 2024-02-19T21:11:49.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 >> 172.21.15.163:0/1365831279 conn(0x7f523006fdc0 msgr2=0x7f52300f4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:49.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 shutdown_connections 2024-02-19T21:11:49.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 1 -- 172.21.15.163:0/1365831279 wait complete. 2024-02-19T21:11:49.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:49.796+0000 7f5237508700 1 librados: shutdown 2024-02-19T21:11:49.801 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.801 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.801 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.801 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.801 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.801 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.801 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.801 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:11:49.801 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:11:49.830 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:49.850 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:11:49.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:49.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:11:49.953 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:49.982 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:50.104 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:50.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:50.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:50.208 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:11:50.235 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:50.254 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:11:50.254 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:11:50.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:49.241381+0000 mgr.y (mgr.14550) 439 : cluster [DBG] pgmap v711: 238 pgs: 10 unknown, 1 active+clean+scrubbing, 227 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:11:50.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:49.659432+0000 mon.a (mon.0) 2683 : audit [DBG] from='client.? v1:172.21.15.163:0/1365831279' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:50.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:50.297783+0000 mon.a (mon.0) 2684 : cluster [DBG] osdmap e477: 8 total, 8 up, 8 in 2024-02-19T21:11:50.570 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:11:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:11:50.583 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:11:50.586 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:50.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:11:50.687 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:50.716 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:50.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:49.241381+0000 mgr.y (mgr.14550) 439 : cluster [DBG] pgmap v711: 238 pgs: 10 unknown, 1 active+clean+scrubbing, 227 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:11:50.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:49.659432+0000 mon.a (mon.0) 2683 : audit [DBG] from='client.? v1:172.21.15.163:0/1365831279' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:50.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:50.297783+0000 mon.a (mon.0) 2684 : cluster [DBG] osdmap e477: 8 total, 8 up, 8 in 2024-02-19T21:11:50.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:49.241381+0000 mgr.y (mgr.14550) 439 : cluster [DBG] pgmap v711: 238 pgs: 10 unknown, 1 active+clean+scrubbing, 227 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:11:50.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:49.659432+0000 mon.a (mon.0) 2683 : audit [DBG] from='client.? v1:172.21.15.163:0/1365831279' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:50.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:50.297783+0000 mon.a (mon.0) 2684 : cluster [DBG] osdmap e477: 8 total, 8 up, 8 in 2024-02-19T21:11:50.964 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:50.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:50.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:50.971 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-19T21:11:51.069 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:11:51.095 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:51.115 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:11:51.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:51.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:11:51.186 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:11:51.219 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:51.248 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:51.267 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:51.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:51.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:51.285 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:11:51.371 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:51.398 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:51.419 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:51.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:51.467 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:51.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:50.488344+0000 mgr.y (mgr.14550) 440 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:51.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:51.301537+0000 mon.a (mon.0) 2685 : cluster [DBG] osdmap e478: 8 total, 8 up, 8 in 2024-02-19T21:11:51.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:51.303240+0000 osd.7 (osd.7) 225 : cluster [DBG] 2.20 starting backfill to osd.5 from (0'0,0'0] MAX to 397'2 2024-02-19T21:11:51.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:51.303317+0000 osd.7 (osd.7) 226 : cluster [DBG] 2.20 starting backfill to osd.6 from (0'0,0'0] MAX to 397'2 2024-02-19T21:11:51.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:50.488344+0000 mgr.y (mgr.14550) 440 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:51.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:51.301537+0000 mon.a (mon.0) 2685 : cluster [DBG] osdmap e478: 8 total, 8 up, 8 in 2024-02-19T21:11:51.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:51.303240+0000 osd.7 (osd.7) 225 : cluster [DBG] 2.20 starting backfill to osd.5 from (0'0,0'0] MAX to 397'2 2024-02-19T21:11:51.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:51.303317+0000 osd.7 (osd.7) 226 : cluster [DBG] 2.20 starting backfill to osd.6 from (0'0,0'0] MAX to 397'2 2024-02-19T21:11:51.500 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:50.488344+0000 mgr.y (mgr.14550) 440 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:11:51.500 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:51.301537+0000 mon.a (mon.0) 2685 : cluster [DBG] osdmap e478: 8 total, 8 up, 8 in 2024-02-19T21:11:51.500 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:51.303240+0000 osd.7 (osd.7) 225 : cluster [DBG] 2.20 starting backfill to osd.5 from (0'0,0'0] MAX to 397'2 2024-02-19T21:11:51.500 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:51.303317+0000 osd.7 (osd.7) 226 : cluster [DBG] 2.20 starting backfill to osd.6 from (0'0,0'0] MAX to 397'2 2024-02-19T21:11:51.524 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:51.553 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:51.571 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:51.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:51.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:51.675 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:11:51.702 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:51.724 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:11:51.727 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:51.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:11:51.828 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:11:51.856 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:51.911 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7, 3, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:11:51.912 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:11:51.912 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 5, 6, 7, 3, 4] 2024-02-19T21:11:51.912 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd out 0 2024-02-19T21:11:52.559 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:11:52.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:51.241842+0000 mgr.y (mgr.14550) 441 : cluster [DBG] pgmap v714: 238 pgs: 10 peering, 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:11:52.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:51.241842+0000 mgr.y (mgr.14550) 441 : cluster [DBG] pgmap v714: 238 pgs: 10 peering, 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:11:52.750 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:51.241842+0000 mgr.y (mgr.14550) 441 : cluster [DBG] pgmap v714: 238 pgs: 10 peering, 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:11:52.832 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:53.005 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:11:53.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:53.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:11:53.009 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:11:53.010 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:11:53.108 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:11:53.138 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:53.420 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:52.417757+0000 mon.a (mon.0) 2686 : cluster [DBG] osdmap e479: 8 total, 8 up, 8 in 2024-02-19T21:11:53.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.297+0000 7f4141d7f700 1 -- 172.21.15.163:0/3005255862 >> v1:172.21.15.163:6789/0 conn(0x7f413c1042c0 legacy=0x7f413c106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:53.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.297+0000 7f4141d7f700 1 -- 172.21.15.163:0/3005255862 shutdown_connections 2024-02-19T21:11:53.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.297+0000 7f4141d7f700 1 -- 172.21.15.163:0/3005255862 >> 172.21.15.163:0/3005255862 conn(0x7f413c0f35a0 msgr2=0x7f413c0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:53.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.297+0000 7f4141d7f700 1 -- 172.21.15.163:0/3005255862 shutdown_connections 2024-02-19T21:11:53.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.297+0000 7f4141d7f700 1 -- 172.21.15.163:0/3005255862 wait complete. 2024-02-19T21:11:53.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.298+0000 7f4141d7f700 1 librados: starting msgr at 2024-02-19T21:11:53.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.298+0000 7f4141d7f700 1 librados: starting objecter 2024-02-19T21:11:53.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.298+0000 7f4141d7f700 1 Processor -- start 2024-02-19T21:11:53.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.298+0000 7f4141d7f700 1 -- start start 2024-02-19T21:11:53.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.298+0000 7f4141d7f700 1 librados: setting wanted keys 2024-02-19T21:11:53.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.298+0000 7f4141d7f700 1 librados: calling monclient init 2024-02-19T21:11:53.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.298+0000 7f4141d7f700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f413c1171c0 con 0x7f413c100780 2024-02-19T21:11:53.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.298+0000 7f4141d7f700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f413c1183a0 con 0x7f413c0fd980 2024-02-19T21:11:53.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.298+0000 7f4141d7f700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f413c1195a0 con 0x7f413c1042c0 2024-02-19T21:11:53.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f413affd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f413c100780 0x7f413c0f9e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43486/0 (socket says 172.21.15.163:43486) 2024-02-19T21:11:53.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f413affd700 1 -- 172.21.15.163:0/706455757 learned_addr learned my addr 172.21.15.163:0/706455757 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:53.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f413b7fe700 1 --1- 172.21.15.163:0/706455757 >> v1:172.21.15.193:6789/0 conn(0x7f413c0fd980 0x7f413c0f9780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:40982/0 (socket says 172.21.15.163:40982) 2024-02-19T21:11:53.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 10 client.?.objecter ms_handle_connect 0x7f413c0fd980 2024-02-19T21:11:53.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:53.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 10 client.?.objecter ms_handle_connect 0x7f413c100780 2024-02-19T21:11:53.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:53.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 10 client.?.objecter ms_handle_connect 0x7f413c1042c0 2024-02-19T21:11:53.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:53.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637712743 0 0) 0x7f413c1183a0 con 0x7f413c0fd980 2024-02-19T21:11:53.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4124003800 con 0x7f413c0fd980 2024-02-19T21:11:53.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074245215 0 0) 0x7f413c1171c0 con 0x7f413c100780 2024-02-19T21:11:53.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f413c1183a0 con 0x7f413c100780 2024-02-19T21:11:53.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143326763 0 0) 0x7f413c1195a0 con 0x7f413c1042c0 2024-02-19T21:11:53.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f413c1171c0 con 0x7f413c1042c0 2024-02-19T21:11:53.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1018052398 0 0) 0x7f4124003800 con 0x7f413c0fd980 2024-02-19T21:11:53.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f413c1195a0 con 0x7f413c0fd980 2024-02-19T21:11:53.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f412c001990 con 0x7f413c0fd980 2024-02-19T21:11:53.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1655490272 0 0) 0x7f413c1183a0 con 0x7f413c100780 2024-02-19T21:11:53.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4124003800 con 0x7f413c100780 2024-02-19T21:11:53.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f41300019f0 con 0x7f413c100780 2024-02-19T21:11:53.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 410396233 0 0) 0x7f413c1171c0 con 0x7f413c1042c0 2024-02-19T21:11:53.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f413c1183a0 con 0x7f413c1042c0 2024-02-19T21:11:53.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4134002ea0 con 0x7f413c1042c0 2024-02-19T21:11:53.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 655941179 0 0) 0x7f413c1195a0 con 0x7f413c0fd980 2024-02-19T21:11:53.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 >> v1:172.21.15.163:6790/0 conn(0x7f413c1042c0 legacy=0x7f413c115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:53.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 >> v1:172.21.15.163:6789/0 conn(0x7f413c100780 legacy=0x7f413c0f9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413c11a7a0 con 0x7f413c0fd980 2024-02-19T21:11:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4141d7f700 1 -- 172.21.15.163:0/706455757 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413c1173b0 con 0x7f413c0fd980 2024-02-19T21:11:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4141d7f700 10 client.48826.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4141d7f700 1 -- 172.21.15.163:0/706455757 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f413c117820 con 0x7f413c0fd980 2024-02-19T21:11:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.299+0000 7f4141d7f700 1 librados: init done 2024-02-19T21:11:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.300+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f412c0023f0 con 0x7f413c0fd980 2024-02-19T21:11:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.300+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f412c005aa0 con 0x7f413c0fd980 2024-02-19T21:11:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.300+0000 7f4141d7f700 1 -- 172.21.15.163:0/706455757 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4128004f20 con 0x7f413c0fd980 2024-02-19T21:11:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.301+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f412c001f50 con 0x7f413c0fd980 2024-02-19T21:11:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.301+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(479..479 src has 43..479) v4 ==== 8092+0+0 (unknown 921834284 0 0) 0x7f412c078c80 con 0x7f413c0fd980 2024-02-19T21:11:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.301+0000 7f4138ff9700 10 client.48826.objecter ms_dispatch 0x7f413c001550 osd_map(479..479 src has 43..479) v4 2024-02-19T21:11:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.301+0000 7f4138ff9700 3 client.48826.objecter handle_osd_map got epochs [479,479] > 0 2024-02-19T21:11:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.301+0000 7f4138ff9700 3 client.48826.objecter handle_osd_map decoding full epoch 479 2024-02-19T21:11:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.301+0000 7f4138ff9700 20 client.48826.objecter dump_active .. 0 homeless 2024-02-19T21:11:53.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.301+0000 7f4138ff9700 10 client.48826.objecter ms_handle_connect 0x7f4124063d60 2024-02-19T21:11:53.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.306+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f412c04c0b0 con 0x7f413c0fd980 2024-02-19T21:11:53.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:53.495+0000 7f4141d7f700 1 -- 172.21.15.163:0/706455757 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f41280051d0 con 0x7f413c0fd980 2024-02-19T21:11:53.531 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:53.684 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:52.417757+0000 mon.a (mon.0) 2686 : cluster [DBG] osdmap e479: 8 total, 8 up, 8 in 2024-02-19T21:11:53.702 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-19T21:11:53.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:52.417757+0000 mon.a (mon.0) 2686 : cluster [DBG] osdmap e479: 8 total, 8 up, 8 in 2024-02-19T21:11:53.746 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:11:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:11:53] "GET /metrics HTTP/1.1" 200 190066 "" "Prometheus/2.33.4" 2024-02-19T21:11:53.987 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:11:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:53.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:11:54.008 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:11:54.090 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:54.117 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:54.265 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:54.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:54.369 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:11:54.397 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:54.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:54.428+0000 7f4138ff9700 1 -- 172.21.15.163:0/706455757 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v481) v1 ==== 97+0+0 (unknown 3144545211 0 0) 0x7f412c04a4c0 con 0x7f413c0fd980 2024-02-19T21:11:54.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:54.430+0000 7f4141d7f700 10 librados: watch_flush enter 2024-02-19T21:11:54.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:54.430+0000 7f4141d7f700 10 librados: watch_flush exit 2024-02-19T21:11:54.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:54.430+0000 7f4141d7f700 20 client.48826.objecter shutdown clearing up homeless session... 2024-02-19T21:11:54.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:54.430+0000 7f4141d7f700 10 client.48826.objecter successfully canceled tick 2024-02-19T21:11:54.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:54.430+0000 7f4141d7f700 1 -- 172.21.15.163:0/706455757 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4124063d60 legacy=0x7f4124066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:54.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:54.430+0000 7f4141d7f700 1 -- 172.21.15.163:0/706455757 >> v1:172.21.15.193:6789/0 conn(0x7f413c0fd980 legacy=0x7f413c0f9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:54.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:54.430+0000 7f4141d7f700 1 -- 172.21.15.163:0/706455757 shutdown_connections 2024-02-19T21:11:54.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:54.430+0000 7f4141d7f700 1 -- 172.21.15.163:0/706455757 >> 172.21.15.163:0/706455757 conn(0x7f413c0f35a0 msgr2=0x7f413c0f6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:54.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:54.430+0000 7f4141d7f700 1 -- 172.21.15.163:0/706455757 shutdown_connections 2024-02-19T21:11:54.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:54.430+0000 7f4141d7f700 1 -- 172.21.15.163:0/706455757 wait complete. 2024-02-19T21:11:54.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:54.430+0000 7f4141d7f700 1 librados: shutdown 2024-02-19T21:11:54.431 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:11:54.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:53.242300+0000 mgr.y (mgr.14550) 442 : cluster [DBG] pgmap v717: 238 pgs: 10 peering, 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:11:54.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:53.422730+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e480: 8 total, 8 up, 8 in 2024-02-19T21:11:54.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:53.497210+0000 mon.b (mon.1) 627 : audit [INF] from='client.? v1:172.21.15.163:0/706455757' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:11:54.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:53.497581+0000 mon.a (mon.0) 2688 : audit [INF] from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:11:54.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:54.103543+0000 mon.a (mon.0) 2689 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:11:54.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:53.242300+0000 mgr.y (mgr.14550) 442 : cluster [DBG] pgmap v717: 238 pgs: 10 peering, 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:11:54.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:53.422730+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e480: 8 total, 8 up, 8 in 2024-02-19T21:11:54.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:53.497210+0000 mon.b (mon.1) 627 : audit [INF] from='client.? v1:172.21.15.163:0/706455757' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:11:54.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:53.497581+0000 mon.a (mon.0) 2688 : audit [INF] from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:11:54.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:54.103543+0000 mon.a (mon.0) 2689 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:11:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:53.242300+0000 mgr.y (mgr.14550) 442 : cluster [DBG] pgmap v717: 238 pgs: 10 peering, 228 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:11:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:53.422730+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e480: 8 total, 8 up, 8 in 2024-02-19T21:11:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:53.497210+0000 mon.b (mon.1) 627 : audit [INF] from='client.? v1:172.21.15.163:0/706455757' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:11:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:53.497581+0000 mon.a (mon.0) 2688 : audit [INF] from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:11:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:54.103543+0000 mon.a (mon.0) 2689 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:11:55.048 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:11:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:55.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:11:55.153 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:55.181 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:55.715 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:54.420788+0000 mon.a (mon.0) 2690 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:11:55.715 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:54.427637+0000 mon.a (mon.0) 2691 : audit [INF] from='client.48826 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:11:55.715 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:54.427706+0000 mon.a (mon.0) 2692 : cluster [DBG] osdmap e481: 8 total, 8 up, 8 in 2024-02-19T21:11:55.731 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:55.732 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:54.420788+0000 mon.a (mon.0) 2690 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:11:55.732 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:54.427637+0000 mon.a (mon.0) 2691 : audit [INF] from='client.48826 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:11:55.732 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:54.427706+0000 mon.a (mon.0) 2692 : cluster [DBG] osdmap e481: 8 total, 8 up, 8 in 2024-02-19T21:11:55.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:54.420788+0000 mon.a (mon.0) 2690 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:11:55.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:54.427637+0000 mon.a (mon.0) 2691 : audit [INF] from='client.48826 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:11:55.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:54.427706+0000 mon.a (mon.0) 2692 : cluster [DBG] osdmap e481: 8 total, 8 up, 8 in 2024-02-19T21:11:55.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:55.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:55.834 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:55.864 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:55.962 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:55.964 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:55.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:55.986 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:11:55.986 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:11:56.065 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:56.094 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:56.241 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:56.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:56.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:56.345 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:56.373 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:56.591 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:55.243169+0000 mgr.y (mgr.14550) 443 : cluster [DBG] pgmap v720: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:11:56.592 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:55.243169+0000 mgr.y (mgr.14550) 443 : cluster [DBG] pgmap v720: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:11:56.621 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:56.623 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:55.243169+0000 mgr.y (mgr.14550) 443 : cluster [DBG] pgmap v720: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:11:56.623 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:56.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:56.634 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-19T21:11:56.725 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:56.755 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:56.853 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:11:56.857 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:56.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:11:56.937 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:11:56.958 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:11:56.987 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:57.085 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:11:57.087 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:57.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:11:57.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.089+0000 7fe3d5638700 1 -- 172.21.15.163:0/3525332337 >> v1:172.21.15.163:6789/0 conn(0x7fe3d00fd980 legacy=0x7fe3d00fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.093 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.089+0000 7fe3d5638700 1 -- 172.21.15.163:0/3525332337 shutdown_connections 2024-02-19T21:11:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.089+0000 7fe3d5638700 1 -- 172.21.15.163:0/3525332337 >> 172.21.15.163:0/3525332337 conn(0x7fe3d00f35a0 msgr2=0x7fe3d00f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.089+0000 7fe3d5638700 1 -- 172.21.15.163:0/3525332337 shutdown_connections 2024-02-19T21:11:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.089+0000 7fe3d5638700 1 -- 172.21.15.163:0/3525332337 wait complete. 2024-02-19T21:11:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.089+0000 7fe3d5638700 1 librados: starting msgr at 2024-02-19T21:11:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.089+0000 7fe3d5638700 1 librados: starting objecter 2024-02-19T21:11:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.089+0000 7fe3d5638700 1 Processor -- start 2024-02-19T21:11:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.089+0000 7fe3d5638700 1 -- start start 2024-02-19T21:11:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.089+0000 7fe3d5638700 1 librados: setting wanted keys 2024-02-19T21:11:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.089+0000 7fe3d5638700 1 librados: calling monclient init 2024-02-19T21:11:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3d5638700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe3d01172a0 con 0x7fe3d0100780 2024-02-19T21:11:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3d5638700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe3d0118480 con 0x7fe3d01042c0 2024-02-19T21:11:57.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3d5638700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe3d0119680 con 0x7fe3d00fd980 2024-02-19T21:11:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3ceffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fe3d00fd980 0x7fe3d00f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45524/0 (socket says 172.21.15.163:45524) 2024-02-19T21:11:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3ceffd700 1 -- 172.21.15.163:0/164579917 learned_addr learned my addr 172.21.15.163:0/164579917 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 10 client.?.objecter ms_handle_connect 0x7fe3d0100780 2024-02-19T21:11:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 10 client.?.objecter ms_handle_connect 0x7fe3d00fd980 2024-02-19T21:11:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 10 client.?.objecter ms_handle_connect 0x7fe3d01042c0 2024-02-19T21:11:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113962441 0 0) 0x7fe3d0119680 con 0x7fe3d00fd980 2024-02-19T21:11:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3b80033f0 con 0x7fe3d00fd980 2024-02-19T21:11:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689170744 0 0) 0x7fe3d01172a0 con 0x7fe3d0100780 2024-02-19T21:11:57.095 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3d0119680 con 0x7fe3d0100780 2024-02-19T21:11:57.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175479206 0 0) 0x7fe3d0118480 con 0x7fe3d01042c0 2024-02-19T21:11:57.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3d01172a0 con 0x7fe3d01042c0 2024-02-19T21:11:57.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3119890233 0 0) 0x7fe3b80033f0 con 0x7fe3d00fd980 2024-02-19T21:11:57.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3d0118480 con 0x7fe3d00fd980 2024-02-19T21:11:57.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1118501401 0 0) 0x7fe3d01172a0 con 0x7fe3d01042c0 2024-02-19T21:11:57.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3b80033f0 con 0x7fe3d01042c0 2024-02-19T21:11:57.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe3c00010c0 con 0x7fe3d00fd980 2024-02-19T21:11:57.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1132949940 0 0) 0x7fe3d0119680 con 0x7fe3d0100780 2024-02-19T21:11:57.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3d01172a0 con 0x7fe3d0100780 2024-02-19T21:11:57.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe3c8001bb0 con 0x7fe3d01042c0 2024-02-19T21:11:57.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.090+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe3c4001840 con 0x7fe3d0100780 2024-02-19T21:11:57.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.091+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2291620754 0 0) 0x7fe3d0118480 con 0x7fe3d00fd980 2024-02-19T21:11:57.096 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.091+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 >> v1:172.21.15.193:6789/0 conn(0x7fe3d01042c0 legacy=0x7fe3d0115960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.091+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 >> v1:172.21.15.163:6789/0 conn(0x7fe3d0100780 legacy=0x7fe3d00fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.091+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d011a880 con 0x7fe3d00fd980 2024-02-19T21:11:57.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.091+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d0119870 con 0x7fe3d00fd980 2024-02-19T21:11:57.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.091+0000 7fe3d5638700 10 client.50018.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:57.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.091+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d0119d60 con 0x7fe3d00fd980 2024-02-19T21:11:57.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.091+0000 7fe3d5638700 1 librados: init done 2024-02-19T21:11:57.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.091+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fe3c0002390 con 0x7fe3d00fd980 2024-02-19T21:11:57.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.091+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe3c0003840 con 0x7fe3d00fd980 2024-02-19T21:11:57.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.092+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe3bc004f00 con 0x7fe3d00fd980 2024-02-19T21:11:57.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.092+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fe3c0001880 con 0x7fe3d00fd980 2024-02-19T21:11:57.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(481..481 src has 43..481) v4 ==== 8092+0+0 (unknown 3030086899 0 0) 0x7fe3c0078f90 con 0x7fe3d00fd980 2024-02-19T21:11:57.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 osd_map(481..481 src has 43..481) v4 2024-02-19T21:11:57.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3b7fff700 3 client.50018.objecter handle_osd_map got epochs [481,481] > 0 2024-02-19T21:11:57.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3b7fff700 3 client.50018.objecter handle_osd_map decoding full epoch 481 2024-02-19T21:11:57.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3b7fff700 20 client.50018.objecter dump_active .. 0 homeless 2024-02-19T21:11:57.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v481) v1 ==== 54+0+16 (unknown 913572059 0 2420000192) 0x7fe3c0079360 con 0x7fe3d00fd980 2024-02-19T21:11:57.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3b7fff700 10 client.50018.objecter ms_handle_connect 0x7fe3b8063cb0 2024-02-19T21:11:57.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3d5638700 10 client.50018.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:57.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 1 2024-02-19T21:11:57.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc0051a0 osd=0 3 2024-02-19T21:11:57.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc0051a0 osd=0 3 2024-02-19T21:11:57.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc0051a0 osd=0 3 2024-02-19T21:11:57.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 1 2024-02-19T21:11:57.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc0051a0 osd=0 3 2024-02-19T21:11:57.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 0 1 2024-02-19T21:11:57.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc0051a0 osd=0 4 2024-02-19T21:11:57.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3d5638700 10 client.50018.objecter _send_command 1 2024-02-19T21:11:57.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3bc004950 con 0x7fe3bc004530 2024-02-19T21:11:57.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.093+0000 7fe3b7fff700 10 client.50018.objecter ms_handle_connect 0x7fe3bc004530 2024-02-19T21:11:57.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.094+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe3bc004950 con 0x7fe3bc004530 2024-02-19T21:11:57.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.094+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:11:57.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.094+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 1 = system:0 2024-02-19T21:11:57.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.094+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc0051a0 osd=0 4 2024-02-19T21:11:57.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.094+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 0 1 2024-02-19T21:11:57.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.109+0000 7fe3d5638700 10 client.50018.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:57.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.109+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 2 2024-02-19T21:11:57.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.109+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc0051a0 osd=0 3 2024-02-19T21:11:57.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.109+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc0051a0 osd=0 3 2024-02-19T21:11:57.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.109+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc0051a0 osd=0 3 2024-02-19T21:11:57.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.109+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 2 2024-02-19T21:11:57.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.109+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc0051a0 osd=0 3 2024-02-19T21:11:57.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.109+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 0 2 2024-02-19T21:11:57.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.109+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc0051a0 osd=0 4 2024-02-19T21:11:57.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.109+0000 7fe3d5638700 10 client.50018.objecter _send_command 2 2024-02-19T21:11:57.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.109+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe3bc004d30 con 0x7fe3bc004530 2024-02-19T21:11:57.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe3bc004d30 con 0x7fe3bc004530 2024-02-19T21:11:57.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:57.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc0051a0 osd=0 4 2024-02-19T21:11:57.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 0 2 2024-02-19T21:11:57.112 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:11:57.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3d5638700 10 client.50018.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:57.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 3 2024-02-19T21:11:57.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc004270 osd=1 3 2024-02-19T21:11:57.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc004270 osd=1 3 2024-02-19T21:11:57.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc004270 osd=1 3 2024-02-19T21:11:57.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 3 2024-02-19T21:11:57.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc004270 osd=1 3 2024-02-19T21:11:57.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 1 3 2024-02-19T21:11:57.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc004270 osd=1 4 2024-02-19T21:11:57.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3d5638700 10 client.50018.objecter _send_command 3 2024-02-19T21:11:57.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.110+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3bc00bf60 con 0x7fe3bc003b40 2024-02-19T21:11:57.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.111+0000 7fe3b7fff700 10 client.50018.objecter ms_handle_connect 0x7fe3bc003b40 2024-02-19T21:11:57.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.111+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe3bc00bf60 con 0x7fe3bc003b40 2024-02-19T21:11:57.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.111+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:11:57.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.111+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 3 = system:0 2024-02-19T21:11:57.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.111+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc004270 osd=1 4 2024-02-19T21:11:57.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.111+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 1 3 2024-02-19T21:11:57.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.126+0000 7fe3d5638700 10 client.50018.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:57.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.126+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 4 2024-02-19T21:11:57.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.126+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc004270 osd=1 3 2024-02-19T21:11:57.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.126+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc004270 osd=1 3 2024-02-19T21:11:57.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.126+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc004270 osd=1 3 2024-02-19T21:11:57.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.126+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 4 2024-02-19T21:11:57.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.126+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc004270 osd=1 3 2024-02-19T21:11:57.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.126+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 1 4 2024-02-19T21:11:57.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.126+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc004270 osd=1 4 2024-02-19T21:11:57.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.126+0000 7fe3d5638700 10 client.50018.objecter _send_command 4 2024-02-19T21:11:57.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.126+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe3bc00d250 con 0x7fe3bc003b40 2024-02-19T21:11:57.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe3bc00d250 con 0x7fe3bc003b40 2024-02-19T21:11:57.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:57.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc004270 osd=1 4 2024-02-19T21:11:57.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 1 4 2024-02-19T21:11:57.127 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:11:57.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3d5638700 10 client.50018.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:57.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 5 2024-02-19T21:11:57.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc003160 osd=2 3 2024-02-19T21:11:57.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc003160 osd=2 3 2024-02-19T21:11:57.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc003160 osd=2 3 2024-02-19T21:11:57.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 5 2024-02-19T21:11:57.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc003160 osd=2 3 2024-02-19T21:11:57.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 2 5 2024-02-19T21:11:57.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc003160 osd=2 4 2024-02-19T21:11:57.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3d5638700 10 client.50018.objecter _send_command 5 2024-02-19T21:11:57.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.127+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3bc003450 con 0x7fe3bc002070 2024-02-19T21:11:57.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.128+0000 7fe3b7fff700 10 client.50018.objecter ms_handle_connect 0x7fe3bc002070 2024-02-19T21:11:57.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.128+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe3bc003450 con 0x7fe3bc002070 2024-02-19T21:11:57.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.128+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:11:57.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.128+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 5 = system:0 2024-02-19T21:11:57.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.128+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc003160 osd=2 4 2024-02-19T21:11:57.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.128+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 2 5 2024-02-19T21:11:57.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.142+0000 7fe3d5638700 10 client.50018.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:57.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.142+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 6 2024-02-19T21:11:57.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.142+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc003160 osd=2 3 2024-02-19T21:11:57.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.142+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc003160 osd=2 3 2024-02-19T21:11:57.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.142+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc003160 osd=2 3 2024-02-19T21:11:57.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.142+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 6 2024-02-19T21:11:57.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.142+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc003160 osd=2 3 2024-02-19T21:11:57.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.142+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 2 6 2024-02-19T21:11:57.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.142+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc003160 osd=2 4 2024-02-19T21:11:57.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.142+0000 7fe3d5638700 10 client.50018.objecter _send_command 6 2024-02-19T21:11:57.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.142+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe3bc001690 con 0x7fe3bc002070 2024-02-19T21:11:57.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe3bc001690 con 0x7fe3bc002070 2024-02-19T21:11:57.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:57.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc003160 osd=2 4 2024-02-19T21:11:57.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 2 6 2024-02-19T21:11:57.143 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:11:57.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3d5638700 10 client.50018.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:57.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 7 2024-02-19T21:11:57.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc00c8a0 osd=3 3 2024-02-19T21:11:57.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc00c8a0 osd=3 3 2024-02-19T21:11:57.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc00c8a0 osd=3 3 2024-02-19T21:11:57.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 7 2024-02-19T21:11:57.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc00c8a0 osd=3 3 2024-02-19T21:11:57.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 3 7 2024-02-19T21:11:57.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc00c8a0 osd=3 4 2024-02-19T21:11:57.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3d5638700 10 client.50018.objecter _send_command 7 2024-02-19T21:11:57.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3bc00d080 con 0x7fe3bc016460 2024-02-19T21:11:57.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.144+0000 7fe3b7fff700 10 client.50018.objecter ms_handle_connect 0x7fe3bc016460 2024-02-19T21:11:57.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.145+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe3bc00d080 con 0x7fe3bc016460 2024-02-19T21:11:57.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.145+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:11:57.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.145+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 7 = system:0 2024-02-19T21:11:57.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.145+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc00c8a0 osd=3 4 2024-02-19T21:11:57.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.145+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 3 7 2024-02-19T21:11:57.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.159+0000 7fe3d5638700 10 client.50018.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:57.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.159+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 8 2024-02-19T21:11:57.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.159+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc00c8a0 osd=3 3 2024-02-19T21:11:57.158 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.159+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc00c8a0 osd=3 3 2024-02-19T21:11:57.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.159+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc00c8a0 osd=3 3 2024-02-19T21:11:57.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.159+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 8 2024-02-19T21:11:57.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.159+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc00c8a0 osd=3 3 2024-02-19T21:11:57.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.159+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 3 8 2024-02-19T21:11:57.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.159+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc00c8a0 osd=3 4 2024-02-19T21:11:57.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.159+0000 7fe3d5638700 10 client.50018.objecter _send_command 8 2024-02-19T21:11:57.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.159+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe3bc00d080 con 0x7fe3bc016460 2024-02-19T21:11:57.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe3bc00d080 con 0x7fe3bc016460 2024-02-19T21:11:57.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:57.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc00c8a0 osd=3 4 2024-02-19T21:11:57.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 3 8 2024-02-19T21:11:57.160 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:11:57.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3d5638700 10 client.50018.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:57.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 9 2024-02-19T21:11:57.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc00d080 osd=4 3 2024-02-19T21:11:57.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc00d080 osd=4 3 2024-02-19T21:11:57.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc00d080 osd=4 3 2024-02-19T21:11:57.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 9 2024-02-19T21:11:57.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc00d080 osd=4 3 2024-02-19T21:11:57.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 4 9 2024-02-19T21:11:57.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc00d080 osd=4 4 2024-02-19T21:11:57.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3d5638700 10 client.50018.objecter _send_command 9 2024-02-19T21:11:57.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3bc01e5e0 con 0x7fe3bc01aad0 2024-02-19T21:11:57.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.161+0000 7fe3b7fff700 10 client.50018.objecter ms_handle_connect 0x7fe3bc01aad0 2024-02-19T21:11:57.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.162+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe3bc01e5e0 con 0x7fe3bc01aad0 2024-02-19T21:11:57.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.162+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:11:57.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.162+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 9 = system:0 2024-02-19T21:11:57.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.162+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc00d080 osd=4 4 2024-02-19T21:11:57.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.162+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 4 9 2024-02-19T21:11:57.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.176+0000 7fe3d5638700 10 client.50018.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:57.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.176+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 10 2024-02-19T21:11:57.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.176+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc00d080 osd=4 3 2024-02-19T21:11:57.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.176+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc00d080 osd=4 3 2024-02-19T21:11:57.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.176+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc00d080 osd=4 3 2024-02-19T21:11:57.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.176+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 10 2024-02-19T21:11:57.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.176+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc00d080 osd=4 3 2024-02-19T21:11:57.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.176+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 4 10 2024-02-19T21:11:57.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.176+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc00d080 osd=4 4 2024-02-19T21:11:57.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.176+0000 7fe3d5638700 10 client.50018.objecter _send_command 10 2024-02-19T21:11:57.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.176+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe3bc01e5e0 con 0x7fe3bc01aad0 2024-02-19T21:11:57.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe3bc01e5e0 con 0x7fe3bc01aad0 2024-02-19T21:11:57.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:57.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc00d080 osd=4 4 2024-02-19T21:11:57.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 4 10 2024-02-19T21:11:57.178 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:11:57.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3d5638700 10 client.50018.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 11 2024-02-19T21:11:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc01e5e0 osd=5 3 2024-02-19T21:11:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc01e5e0 osd=5 3 2024-02-19T21:11:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc01e5e0 osd=5 3 2024-02-19T21:11:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 11 2024-02-19T21:11:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc01e5e0 osd=5 3 2024-02-19T21:11:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 5 11 2024-02-19T21:11:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc01e5e0 osd=5 4 2024-02-19T21:11:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3d5638700 10 client.50018.objecter _send_command 11 2024-02-19T21:11:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.178+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3bc022910 con 0x7fe3bc01ecc0 2024-02-19T21:11:57.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.179+0000 7fe3b7fff700 10 client.50018.objecter ms_handle_connect 0x7fe3bc01ecc0 2024-02-19T21:11:57.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.179+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe3bc022910 con 0x7fe3bc01ecc0 2024-02-19T21:11:57.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.179+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:11:57.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.179+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 11 = system:0 2024-02-19T21:11:57.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.179+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc01e5e0 osd=5 4 2024-02-19T21:11:57.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.179+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 5 11 2024-02-19T21:11:57.189 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:11:57.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.193+0000 7fe3d5638700 10 client.50018.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:57.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.193+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 12 2024-02-19T21:11:57.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.193+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc01e5e0 osd=5 3 2024-02-19T21:11:57.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.193+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc01e5e0 osd=5 3 2024-02-19T21:11:57.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.193+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc01e5e0 osd=5 3 2024-02-19T21:11:57.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.193+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 12 2024-02-19T21:11:57.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.193+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc01e5e0 osd=5 3 2024-02-19T21:11:57.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.193+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 5 12 2024-02-19T21:11:57.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.193+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc01e5e0 osd=5 4 2024-02-19T21:11:57.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.193+0000 7fe3d5638700 10 client.50018.objecter _send_command 12 2024-02-19T21:11:57.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.193+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe3bc022910 con 0x7fe3bc01ecc0 2024-02-19T21:11:57.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe3bc022910 con 0x7fe3bc01ecc0 2024-02-19T21:11:57.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:57.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc01e5e0 osd=5 4 2024-02-19T21:11:57.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 5 12 2024-02-19T21:11:57.195 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:11:57.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3d5638700 10 client.50018.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:57.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 13 2024-02-19T21:11:57.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc0240c0 osd=6 3 2024-02-19T21:11:57.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc0240c0 osd=6 3 2024-02-19T21:11:57.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc0240c0 osd=6 3 2024-02-19T21:11:57.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 13 2024-02-19T21:11:57.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc0240c0 osd=6 3 2024-02-19T21:11:57.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 6 13 2024-02-19T21:11:57.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc0240c0 osd=6 4 2024-02-19T21:11:57.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3d5638700 10 client.50018.objecter _send_command 13 2024-02-19T21:11:57.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.195+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3bc024c10 con 0x7fe3bc024740 2024-02-19T21:11:57.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.196+0000 7fe3b7fff700 10 client.50018.objecter ms_handle_connect 0x7fe3bc024740 2024-02-19T21:11:57.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.196+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe3bc024c10 con 0x7fe3bc024740 2024-02-19T21:11:57.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.196+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:11:57.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.196+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 13 = system:0 2024-02-19T21:11:57.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.196+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc0240c0 osd=6 4 2024-02-19T21:11:57.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.196+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 6 13 2024-02-19T21:11:57.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.211+0000 7fe3d5638700 10 client.50018.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:57.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.211+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 14 2024-02-19T21:11:57.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.211+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc0240c0 osd=6 3 2024-02-19T21:11:57.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.211+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc0240c0 osd=6 3 2024-02-19T21:11:57.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.211+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc0240c0 osd=6 3 2024-02-19T21:11:57.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.211+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 14 2024-02-19T21:11:57.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.211+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc0240c0 osd=6 3 2024-02-19T21:11:57.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.211+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 6 14 2024-02-19T21:11:57.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.211+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc0240c0 osd=6 4 2024-02-19T21:11:57.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.211+0000 7fe3d5638700 10 client.50018.objecter _send_command 14 2024-02-19T21:11:57.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.211+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe3bc024c10 con 0x7fe3bc024740 2024-02-19T21:11:57.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.212+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe3bc024c10 con 0x7fe3bc024740 2024-02-19T21:11:57.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.212+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:57.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.212+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.212+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc0240c0 osd=6 4 2024-02-19T21:11:57.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.212+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 6 14 2024-02-19T21:11:57.212 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:11:57.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.213+0000 7fe3d5638700 10 client.50018.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:11:57.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.213+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 15 2024-02-19T21:11:57.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.213+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc024c10 osd=7 3 2024-02-19T21:11:57.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.213+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc024c10 osd=7 3 2024-02-19T21:11:57.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.213+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc024c10 osd=7 3 2024-02-19T21:11:57.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.213+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 15 2024-02-19T21:11:57.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.213+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc024c10 osd=7 3 2024-02-19T21:11:57.213 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.213+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 7 15 2024-02-19T21:11:57.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.213+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc024c10 osd=7 4 2024-02-19T21:11:57.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.213+0000 7fe3d5638700 10 client.50018.objecter _send_command 15 2024-02-19T21:11:57.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.213+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3bc02b2f0 con 0x7fe3bc024e50 2024-02-19T21:11:57.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.213+0000 7fe3b7fff700 10 client.50018.objecter ms_handle_connect 0x7fe3bc024e50 2024-02-19T21:11:57.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.214+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe3bc02b2f0 con 0x7fe3bc024e50 2024-02-19T21:11:57.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.214+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:11:57.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.214+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 15 = system:0 2024-02-19T21:11:57.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.214+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc024c10 osd=7 4 2024-02-19T21:11:57.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.214+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 7 15 2024-02-19T21:11:57.218 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:57.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.229+0000 7fe3d5638700 10 client.50018.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:11:57.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.229+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign -1 16 2024-02-19T21:11:57.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.229+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc024c10 osd=7 3 2024-02-19T21:11:57.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.229+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc024c10 osd=7 3 2024-02-19T21:11:57.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.229+0000 7fe3d5638700 20 client.50018.objecter _get_session s=0x7fe3bc024c10 osd=7 3 2024-02-19T21:11:57.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.229+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_remove -1 16 2024-02-19T21:11:57.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.229+0000 7fe3d5638700 20 client.50018.objecter get_session s=0x7fe3bc024c10 osd=7 3 2024-02-19T21:11:57.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.229+0000 7fe3d5638700 15 client.50018.objecter _session_command_op_assign 7 16 2024-02-19T21:11:57.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.229+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc024c10 osd=7 4 2024-02-19T21:11:57.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.229+0000 7fe3d5638700 10 client.50018.objecter _send_command 16 2024-02-19T21:11:57.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.229+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe3bc02b2f0 con 0x7fe3bc024e50 2024-02-19T21:11:57.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.230+0000 7fe3b7fff700 1 -- 172.21.15.163:0/164579917 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe3bc02b2f0 con 0x7fe3bc024e50 2024-02-19T21:11:57.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.230+0000 7fe3b7fff700 10 client.50018.objecter ms_dispatch 0x7fe3d0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:11:57.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.230+0000 7fe3b7fff700 10 client.50018.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.230+0000 7fe3b7fff700 20 client.50018.objecter put_session s=0x7fe3bc024c10 osd=7 4 2024-02-19T21:11:57.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.230+0000 7fe3b7fff700 15 client.50018.objecter _session_command_op_remove 7 16 2024-02-19T21:11:57.230 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:11:57.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 10 librados: watch_flush enter 2024-02-19T21:11:57.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 10 librados: watch_flush exit 2024-02-19T21:11:57.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 10 client.50018.objecter close_session for osd.0 2024-02-19T21:11:57.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 >> v1:172.21.15.163:6801/1559418115 conn(0x7fe3bc004530 legacy=0x7fe3bc002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc0051a0 osd=0 1 2024-02-19T21:11:57.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 10 client.50018.objecter close_session for osd.1 2024-02-19T21:11:57.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 >> v1:172.21.15.163:6805/3125047432 conn(0x7fe3bc003b40 legacy=0x7fe3bc00a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc004270 osd=1 1 2024-02-19T21:11:57.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 10 client.50018.objecter close_session for osd.2 2024-02-19T21:11:57.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 >> v1:172.21.15.163:6809/1563158487 conn(0x7fe3bc002070 legacy=0x7fe3bc013b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc003160 osd=2 1 2024-02-19T21:11:57.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 10 client.50018.objecter close_session for osd.3 2024-02-19T21:11:57.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 >> v1:172.21.15.163:6813/2112279769 conn(0x7fe3bc016460 legacy=0x7fe3bc0168a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc00c8a0 osd=3 1 2024-02-19T21:11:57.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 10 client.50018.objecter close_session for osd.4 2024-02-19T21:11:57.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3cf7fe700 1 -- 172.21.15.163:0/164579917 reap_dead start 2024-02-19T21:11:57.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 >> v1:172.21.15.193:6800/3129716965 conn(0x7fe3bc01aad0 legacy=0x7fe3bc01cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc00d080 osd=4 1 2024-02-19T21:11:57.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 10 client.50018.objecter close_session for osd.5 2024-02-19T21:11:57.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 >> v1:172.21.15.193:6804/805161991 conn(0x7fe3bc01ecc0 legacy=0x7fe3bc0210b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc01e5e0 osd=5 1 2024-02-19T21:11:57.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 10 client.50018.objecter close_session for osd.6 2024-02-19T21:11:57.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 >> v1:172.21.15.193:6808/787604615 conn(0x7fe3bc024740 legacy=0x7fe3bc022ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc0240c0 osd=6 1 2024-02-19T21:11:57.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 10 client.50018.objecter close_session for osd.7 2024-02-19T21:11:57.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 >> v1:172.21.15.193:6812/157927432 conn(0x7fe3bc024e50 legacy=0x7fe3bc0236d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 20 client.50018.objecter put_session s=0x7fe3bc024c10 osd=7 1 2024-02-19T21:11:57.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 20 client.50018.objecter shutdown clearing up homeless session... 2024-02-19T21:11:57.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 10 client.50018.objecter successfully canceled tick 2024-02-19T21:11:57.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3cf7fe700 1 -- 172.21.15.163:0/164579917 reap_dead start 2024-02-19T21:11:57.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 >> v1:172.21.15.163:6800/1552514681 conn(0x7fe3b8063cb0 legacy=0x7fe3b8066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 >> v1:172.21.15.163:6790/0 conn(0x7fe3d00fd980 legacy=0x7fe3d00f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 shutdown_connections 2024-02-19T21:11:57.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 >> 172.21.15.163:0/164579917 conn(0x7fe3d00f35a0 msgr2=0x7fe3d0107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:57.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 shutdown_connections 2024-02-19T21:11:57.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 1 -- 172.21.15.163:0/164579917 wait complete. 2024-02-19T21:11:57.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.231+0000 7fe3d5638700 1 librados: shutdown 2024-02-19T21:11:57.235 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.235 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.235 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.235 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.235 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.235 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.236 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.236 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:11:57.237 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:11:57.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:57.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:11:57.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.318+0000 7f134bb2a700 1 -- 172.21.15.163:0/766258681 >> v1:172.21.15.163:6790/0 conn(0x7f13440ff670 legacy=0x7f1344101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.318+0000 7f134bb2a700 1 -- 172.21.15.163:0/766258681 shutdown_connections 2024-02-19T21:11:57.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.318+0000 7f134bb2a700 1 -- 172.21.15.163:0/766258681 >> 172.21.15.163:0/766258681 conn(0x7f13440f2470 msgr2=0x7f13440f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:57.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.318+0000 7f134bb2a700 1 -- 172.21.15.163:0/766258681 shutdown_connections 2024-02-19T21:11:57.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.318+0000 7f134bb2a700 1 -- 172.21.15.163:0/766258681 wait complete. 2024-02-19T21:11:57.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f134bb2a700 1 librados: starting msgr at 2024-02-19T21:11:57.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f134bb2a700 1 librados: starting objecter 2024-02-19T21:11:57.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f134bb2a700 1 Processor -- start 2024-02-19T21:11:57.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f134bb2a700 1 -- start start 2024-02-19T21:11:57.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f134bb2a700 1 librados: setting wanted keys 2024-02-19T21:11:57.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f134bb2a700 1 librados: calling monclient init 2024-02-19T21:11:57.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f134bb2a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f13441194f0 con 0x7f13440ff670 2024-02-19T21:11:57.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f134bb2a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f134411a6d0 con 0x7f13440fc870 2024-02-19T21:11:57.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f134bb2a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f134411b8d0 con 0x7f13441031b0 2024-02-19T21:11:57.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f13490c5700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f13440ff670 0x7f13440f8b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:38970/0 (socket says 172.21.15.163:38970) 2024-02-19T21:11:57.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f13490c5700 1 -- 172.21.15.163:0/3787558960 learned_addr learned my addr 172.21.15.163:0/3787558960 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:11:57.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f134a0c7700 1 --1- 172.21.15.163:0/3787558960 >> v1:172.21.15.163:6790/0 conn(0x7f13441031b0 0x7f1344117c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45552/0 (socket says 172.21.15.163:45552) 2024-02-19T21:11:57.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f133affd700 10 client.?.objecter ms_handle_connect 0x7f13440ff670 2024-02-19T21:11:57.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f133affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:57.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f133affd700 10 client.?.objecter ms_handle_connect 0x7f13441031b0 2024-02-19T21:11:57.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f133affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:57.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f133affd700 10 client.?.objecter ms_handle_connect 0x7f13440fc870 2024-02-19T21:11:57.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.319+0000 7f133affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:11:57.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956997546 0 0) 0x7f134411a6d0 con 0x7f13440fc870 2024-02-19T21:11:57.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13300033f0 con 0x7f13440fc870 2024-02-19T21:11:57.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952930003 0 0) 0x7f13441194f0 con 0x7f13440ff670 2024-02-19T21:11:57.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f134411a6d0 con 0x7f13440ff670 2024-02-19T21:11:57.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392835067 0 0) 0x7f134411b8d0 con 0x7f13441031b0 2024-02-19T21:11:57.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13441194f0 con 0x7f13441031b0 2024-02-19T21:11:57.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 136512709 0 0) 0x7f13300033f0 con 0x7f13440fc870 2024-02-19T21:11:57.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f134411b8d0 con 0x7f13440fc870 2024-02-19T21:11:57.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f13400018c0 con 0x7f13440fc870 2024-02-19T21:11:57.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1104802193 0 0) 0x7f13441194f0 con 0x7f13441031b0 2024-02-19T21:11:57.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13300033f0 con 0x7f13441031b0 2024-02-19T21:11:57.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f133c001c40 con 0x7f13441031b0 2024-02-19T21:11:57.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1323618420 0 0) 0x7f134411a6d0 con 0x7f13440ff670 2024-02-19T21:11:57.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13441194f0 con 0x7f13440ff670 2024-02-19T21:11:57.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1334001f90 con 0x7f13440ff670 2024-02-19T21:11:57.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2433342465 0 0) 0x7f134411b8d0 con 0x7f13440fc870 2024-02-19T21:11:57.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 >> v1:172.21.15.163:6790/0 conn(0x7f13441031b0 legacy=0x7f1344117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 >> v1:172.21.15.163:6789/0 conn(0x7f13440ff670 legacy=0x7f13440f8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:57.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134411cad0 con 0x7f13440fc870 2024-02-19T21:11:57.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f134bb2a700 1 -- 172.21.15.163:0/3787558960 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134411a8c0 con 0x7f13440fc870 2024-02-19T21:11:57.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f134bb2a700 10 client.48838.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:11:57.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f134bb2a700 1 -- 172.21.15.163:0/3787558960 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f134411ad30 con 0x7f13440fc870 2024-02-19T21:11:57.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f134bb2a700 1 librados: init done 2024-02-19T21:11:57.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f1340001f80 con 0x7f13440fc870 2024-02-19T21:11:57.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.320+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f13400031f0 con 0x7f13440fc870 2024-02-19T21:11:57.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.321+0000 7f134bb2a700 1 -- 172.21.15.163:0/3787558960 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1328004f20 con 0x7f13440fc870 2024-02-19T21:11:57.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.321+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f1340001ae0 con 0x7f13440fc870 2024-02-19T21:11:57.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.322+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(481..481 src has 43..481) v4 ==== 8092+0+0 (unknown 3030086899 0 0) 0x7f1340076520 con 0x7f13440fc870 2024-02-19T21:11:57.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.322+0000 7f133affd700 10 client.48838.objecter ms_dispatch 0x7f1344001550 osd_map(481..481 src has 43..481) v4 2024-02-19T21:11:57.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.322+0000 7f133affd700 3 client.48838.objecter handle_osd_map got epochs [481,481] > 0 2024-02-19T21:11:57.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.322+0000 7f133affd700 3 client.48838.objecter handle_osd_map decoding full epoch 481 2024-02-19T21:11:57.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.322+0000 7f133affd700 20 client.48838.objecter dump_active .. 0 homeless 2024-02-19T21:11:57.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.322+0000 7f133affd700 10 client.48838.objecter ms_handle_connect 0x7f1330063d10 2024-02-19T21:11:57.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.326+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f13400498f0 con 0x7f13440fc870 2024-02-19T21:11:57.340 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:11:57.366 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:57.393 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:11:57.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:57.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:11:57.497 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:57.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:57.505+0000 7f134bb2a700 1 -- 172.21.15.163:0/3787558960 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd out", "ids": ["0"]} v 0) v1 -- 0x7f13280051d0 con 0x7f13440fc870 2024-02-19T21:11:57.523 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:57.721 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:57.724 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:57.093499+0000 mon.c (mon.2) 401 : audit [DBG] from='client.? v1:172.21.15.163:0/164579917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:57.724 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:57.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:57.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:57.093499+0000 mon.c (mon.2) 401 : audit [DBG] from='client.? v1:172.21.15.163:0/164579917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:57.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:57.093499+0000 mon.c (mon.2) 401 : audit [DBG] from='client.? v1:172.21.15.163:0/164579917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:11:57.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:11:57.506+0000 7fb606608700 -1 mon.a@0(leader).osd e481 definitely_dead 0 2024-02-19T21:11:57.825 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:11:57.852 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:58.069 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:11:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:11:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:11:58.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:58.456+0000 7f133affd700 1 -- 172.21.15.163:0/3787558960 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd out", "ids": ["0"]}]=0 marked out osd.0. v482) v1 ==== 87+0+0 (unknown 339202742 0 0) 0x7f1340047d00 con 0x7f13440fc870 2024-02-19T21:11:58.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:58.459+0000 7f134bb2a700 10 librados: watch_flush enter 2024-02-19T21:11:58.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:58.460+0000 7f134bb2a700 10 librados: watch_flush exit 2024-02-19T21:11:58.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:58.460+0000 7f134bb2a700 20 client.48838.objecter shutdown clearing up homeless session... 2024-02-19T21:11:58.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:58.460+0000 7f134bb2a700 10 client.48838.objecter successfully canceled tick 2024-02-19T21:11:58.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:58.460+0000 7f134bb2a700 1 -- 172.21.15.163:0/3787558960 >> v1:172.21.15.163:6800/1552514681 conn(0x7f1330063d10 legacy=0x7f13300661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:58.459 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:58.460+0000 7f134bb2a700 1 -- 172.21.15.163:0/3787558960 >> v1:172.21.15.193:6789/0 conn(0x7f13440fc870 legacy=0x7f13440f8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:11:58.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:58.461+0000 7f134bb2a700 1 -- 172.21.15.163:0/3787558960 shutdown_connections 2024-02-19T21:11:58.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:58.461+0000 7f134bb2a700 1 -- 172.21.15.163:0/3787558960 >> 172.21.15.163:0/3787558960 conn(0x7f13440f2470 msgr2=0x7f1344106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:11:58.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:58.461+0000 7f134bb2a700 1 -- 172.21.15.163:0/3787558960 shutdown_connections 2024-02-19T21:11:58.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:58.461+0000 7f134bb2a700 1 -- 172.21.15.163:0/3787558960 wait complete. 2024-02-19T21:11:58.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:11:58.461+0000 7f134bb2a700 1 librados: shutdown 2024-02-19T21:11:58.470 INFO:teuthology.orchestra.run.smithi163.stderr:marked out osd.0. 2024-02-19T21:11:58.601 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:11:58.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:58.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:11:58.611 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:11:58.611 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-02-19T21:11:58.611 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-02-19T21:11:58.611 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-02-19T21:11:58.611 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.611 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.3.0:299 11.8 e113/99 11:1381d4a2:test-rados-api-smithi163-83952-6::foo:9 v 113'6, mlcod=111'3)", 2024-02-19T21:11:58.612 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:52.583933+0000", 2024-02-19T21:11:58.612 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 426.027063607, 2024-02-19T21:11:58.612 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0114605, 2024-02-19T21:11:58.612 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.612 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.612 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.612 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.612 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.612 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583933+0000", 2024-02-19T21:11:58.612 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.612 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.612 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.612 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.613 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583933+0000", 2024-02-19T21:11:58.613 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:11:58.613 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.613 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.613 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.613 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583933+0000", 2024-02-19T21:11:58.613 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9749999999999998e-06 2024-02-19T21:11:58.613 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.613 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.613 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.613 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583938+0000", 2024-02-19T21:11:58.613 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2570000000000004e-06 2024-02-19T21:11:58.613 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.613 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.613 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.614 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583944+0000", 2024-02-19T21:11:58.614 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.657e-06 2024-02-19T21:11:58.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.614 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.614 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.614 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.583951+0000", 2024-02-19T21:11:58.614 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00042205900000000002 2024-02-19T21:11:58.614 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.614 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.614 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.615 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.584373+0000", 2024-02-19T21:11:58.615 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3680999999999999e-05 2024-02-19T21:11:58.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.615 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.615 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.615 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.584396+0000", 2024-02-19T21:11:58.615 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010962929 2024-02-19T21:11:58.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.615 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.615 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.615 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.595359+0000", 2024-02-19T21:11:58.615 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4283999999999999e-05 2024-02-19T21:11:58.615 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.615 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.616 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.616 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:52.595394+0000", 2024-02-19T21:11:58.616 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0114605 2024-02-19T21:11:58.616 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.616 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.616 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.616 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.616 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.616 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "replica_scrub(pg: 4.10,from:0'0,to:65'4,epoch:127/59,start:4:08000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-02-19T21:11:58.616 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:08.142968+0000", 2024-02-19T21:11:58.616 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 410.46802929699999, 2024-02-19T21:11:58.616 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010775658, 2024-02-19T21:11:58.616 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.616 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.142968+0000", 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.142968+0000", 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.142966+0000", 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2630000000000003e-06 2024-02-19T21:11:58.617 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.618 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.618 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.618 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.142974+0000", 2024-02-19T21:11:58.618 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0028049339999999998 2024-02-19T21:11:58.618 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.618 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.618 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.618 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.145779+0000", 2024-02-19T21:11:58.618 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3135999999999999e-05 2024-02-19T21:11:58.618 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.618 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.619 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.619 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.145792+0000", 2024-02-19T21:11:58.619 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0078870490000000001 2024-02-19T21:11:58.619 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.619 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.619 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.619 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.153679+0000", 2024-02-19T21:11:58.619 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5184000000000001e-05 2024-02-19T21:11:58.619 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.619 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.619 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.620 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.153695+0000", 2024-02-19T21:11:58.620 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8640999999999997e-05 2024-02-19T21:11:58.620 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.620 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.620 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.620 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:08.153743+0000", 2024-02-19T21:11:58.620 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010775658 2024-02-19T21:11:58.620 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.620 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.620 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.620 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.620 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.620 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.1.0:241 28.15 e132/99 -30:ad087405:::temp_28.15_0_14259_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:11:58.621 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.401838+0000", 2024-02-19T21:11:58.621 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 406.20915842699998, 2024-02-19T21:11:58.621 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025390421, 2024-02-19T21:11:58.621 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.621 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.621 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.621 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.621 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.621 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.401838+0000", 2024-02-19T21:11:58.621 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.621 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.621 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.621 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.622 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.401838+0000", 2024-02-19T21:11:58.622 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:58.622 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.622 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.622 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.622 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.401837+0000", 2024-02-19T21:11:58.622 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.016854948000000002 2024-02-19T21:11:58.622 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.622 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.622 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.622 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.418692+0000", 2024-02-19T21:11:58.622 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8876e-05 2024-02-19T21:11:58.622 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.622 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.623 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.623 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.418710+0000", 2024-02-19T21:11:58.623 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4498000000000002e-05 2024-02-19T21:11:58.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.623 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.623 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.623 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.418735+0000", 2024-02-19T21:11:58.623 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7998000000000003e-05 2024-02-19T21:11:58.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.623 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.623 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.623 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.418793+0000", 2024-02-19T21:11:58.623 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6917e-05 2024-02-19T21:11:58.623 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.624 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.624 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.624 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.418820+0000", 2024-02-19T21:11:58.624 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0083708069999999992 2024-02-19T21:11:58.624 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.624 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.624 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.624 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.427191+0000", 2024-02-19T21:11:58.624 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8194000000000001e-05 2024-02-19T21:11:58.624 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.624 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.624 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.624 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.427229+0000", 2024-02-19T21:11:58.625 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025390421 2024-02-19T21:11:58.625 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.625 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.625 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.625 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.625 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.625 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.1.0:242 28.15 e132/99 -30:ad087405:::temp_28.15_0_14259_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:11:58.625 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.454353+0000", 2024-02-19T21:11:58.625 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 406.15664427500002, 2024-02-19T21:11:58.625 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.029471536999999999, 2024-02-19T21:11:58.625 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.625 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.625 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.625 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.626 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.626 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.454353+0000", 2024-02-19T21:11:58.626 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.626 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.626 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.626 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.626 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.454353+0000", 2024-02-19T21:11:58.626 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:58.626 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.626 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.626 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.626 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.454350+0000", 2024-02-19T21:11:58.627 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.021044232999999999 2024-02-19T21:11:58.627 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.627 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.627 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.627 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.475394+0000", 2024-02-19T21:11:58.627 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7527999999999999e-05 2024-02-19T21:11:58.627 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.627 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.627 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.627 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.475412+0000", 2024-02-19T21:11:58.627 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1040000000000007e-06 2024-02-19T21:11:58.627 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.627 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.627 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.628 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.475420+0000", 2024-02-19T21:11:58.628 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3225000000000005e-05 2024-02-19T21:11:58.628 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.628 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.628 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.628 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.475483+0000", 2024-02-19T21:11:58.628 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4318e-05 2024-02-19T21:11:58.628 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.628 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.628 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.628 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.475508+0000", 2024-02-19T21:11:58.628 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0082776299999999994 2024-02-19T21:11:58.628 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.629 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.629 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.629 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.483785+0000", 2024-02-19T21:11:58.629 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8816999999999998e-05 2024-02-19T21:11:58.629 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.629 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.629 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.629 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.483824+0000", 2024-02-19T21:11:58.629 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.029471536999999999 2024-02-19T21:11:58.629 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.630 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.630 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.630 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.630 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.630 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.1.0:243 28.15 e132/99 -30:ad087405:::temp_28.15_0_14259_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:11:58.630 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.477725+0000", 2024-02-19T21:11:58.630 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 406.13327187099998, 2024-02-19T21:11:58.630 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.020820749, 2024-02-19T21:11:58.630 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.630 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.630 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.630 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.630 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.631 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.477725+0000", 2024-02-19T21:11:58.631 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.631 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.631 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.631 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.631 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.477725+0000", 2024-02-19T21:11:58.631 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:58.631 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.631 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.631 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.631 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.477723+0000", 2024-02-19T21:11:58.631 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.012259177 2024-02-19T21:11:58.631 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.631 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.632 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.632 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.489982+0000", 2024-02-19T21:11:58.632 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6534000000000001e-05 2024-02-19T21:11:58.632 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.632 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.632 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.632 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.489999+0000", 2024-02-19T21:11:58.632 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8809999999999996e-06 2024-02-19T21:11:58.632 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.632 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.632 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.632 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.490006+0000", 2024-02-19T21:11:58.633 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2170999999999997e-05 2024-02-19T21:11:58.633 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.633 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.633 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.633 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.490079+0000", 2024-02-19T21:11:58.633 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3287e-05 2024-02-19T21:11:58.633 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.633 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.633 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.633 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.490132+0000", 2024-02-19T21:11:58.633 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0083842740000000006 2024-02-19T21:11:58.633 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.633 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.633 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.634 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.498516+0000", 2024-02-19T21:11:58.634 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9481000000000001e-05 2024-02-19T21:11:58.634 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.634 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.634 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.634 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.498546+0000", 2024-02-19T21:11:58.634 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.020820749 2024-02-19T21:11:58.634 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.634 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.634 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.634 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.634 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.634 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.5.0:1270 28.9 e132/99 -30:90852524:::temp_28.9_0_24280_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:11:58.634 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.578695+0000", 2024-02-19T21:11:58.634 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 406.032302276, 2024-02-19T21:11:58.635 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017938424000000001, 2024-02-19T21:11:58.635 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.635 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.635 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.635 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.635 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.635 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.578695+0000", 2024-02-19T21:11:58.635 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.635 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.635 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.635 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.578695+0000", 2024-02-19T21:11:58.635 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:11:58.635 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.636 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.636 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.636 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.578691+0000", 2024-02-19T21:11:58.636 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0030402350000000001 2024-02-19T21:11:58.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.636 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.636 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.636 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.581731+0000", 2024-02-19T21:11:58.636 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2955e-05 2024-02-19T21:11:58.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.636 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.636 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.636 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.581754+0000", 2024-02-19T21:11:58.636 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0519999999999999e-05 2024-02-19T21:11:58.636 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.637 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.637 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.637 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.581765+0000", 2024-02-19T21:11:58.637 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7444e-05 2024-02-19T21:11:58.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.637 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.637 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.637 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.581832+0000", 2024-02-19T21:11:58.637 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9349000000000001e-05 2024-02-19T21:11:58.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.637 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.637 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.637 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.581851+0000", 2024-02-19T21:11:58.637 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014735747 2024-02-19T21:11:58.637 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.638 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.638 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.638 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.596587+0000", 2024-02-19T21:11:58.638 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5918000000000003e-05 2024-02-19T21:11:58.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.638 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.638 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.638 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.596633+0000", 2024-02-19T21:11:58.638 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017938424000000001 2024-02-19T21:11:58.638 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.638 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.638 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.638 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.638 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.639 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.5.0:1271 28.9 e132/99 -30:90852524:::temp_28.9_0_24280_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:11:58.639 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.608168+0000", 2024-02-19T21:11:58.639 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 406.00282903999999, 2024-02-19T21:11:58.639 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017383068000000002, 2024-02-19T21:11:58.639 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.639 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.639 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.639 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.639 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.639 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.608168+0000", 2024-02-19T21:11:58.639 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.640 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.640 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.640 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.608168+0000", 2024-02-19T21:11:58.640 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:11:58.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.640 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.640 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.640 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.608164+0000", 2024-02-19T21:11:58.640 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003639705 2024-02-19T21:11:58.640 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.640 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.641 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.641 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.611804+0000", 2024-02-19T21:11:58.641 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6741e-05 2024-02-19T21:11:58.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.641 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.641 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.641 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.611830+0000", 2024-02-19T21:11:58.641 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0243e-05 2024-02-19T21:11:58.641 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.641 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.641 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.641 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.611841+0000", 2024-02-19T21:11:58.641 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8799999999999999e-05 2024-02-19T21:11:58.642 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.642 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.642 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.642 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.611899+0000", 2024-02-19T21:11:58.642 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7115e-05 2024-02-19T21:11:58.642 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.642 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.642 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.642 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.611927+0000", 2024-02-19T21:11:58.642 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013572864 2024-02-19T21:11:58.642 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.642 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.642 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.642 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.625499+0000", 2024-02-19T21:11:58.643 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1449999999999997e-05 2024-02-19T21:11:58.643 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.643 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.643 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.643 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.625551+0000", 2024-02-19T21:11:58.643 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017383068000000002 2024-02-19T21:11:58.643 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.643 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.643 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.643 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.643 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.643 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.5.0:1272 28.9 e132/99 -30:90852524:::temp_28.9_0_24280_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:11:58.643 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.668531+0000", 2024-02-19T21:11:58.643 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 405.94246544499998, 2024-02-19T21:11:58.643 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017253208999999999, 2024-02-19T21:11:58.644 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.644 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.644 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.644 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.644 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.644 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.668531+0000", 2024-02-19T21:11:58.644 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.644 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.644 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.644 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.644 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.668531+0000", 2024-02-19T21:11:58.644 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999957 2024-02-19T21:11:58.645 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.645 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.645 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.645 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.668527+0000", 2024-02-19T21:11:58.645 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0031829409999999999 2024-02-19T21:11:58.645 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.645 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.645 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.645 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.671710+0000", 2024-02-19T21:11:58.645 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5936999999999999e-05 2024-02-19T21:11:58.645 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.645 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.645 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.646 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.671736+0000", 2024-02-19T21:11:58.646 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1466e-05 2024-02-19T21:11:58.646 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.646 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.646 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.646 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.671747+0000", 2024-02-19T21:11:58.646 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2371e-05 2024-02-19T21:11:58.646 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.646 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.646 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.646 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.671840+0000", 2024-02-19T21:11:58.646 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0371000000000001e-05 2024-02-19T21:11:58.646 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.646 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.646 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.647 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.671870+0000", 2024-02-19T21:11:58.647 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013870054 2024-02-19T21:11:58.647 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.647 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.647 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.647 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.685740+0000", 2024-02-19T21:11:58.647 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4508000000000001e-05 2024-02-19T21:11:58.647 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.647 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.647 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.647 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.685785+0000", 2024-02-19T21:11:58.647 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017253208999999999 2024-02-19T21:11:58.647 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.647 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.648 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.648 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.648 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.648 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.0.0:262 28.2 e132/99 -30:4327d3d8:::temp_28.2_0_14232_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:11:58.648 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.757694+0000", 2024-02-19T21:11:58.648 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 405.85330238500001, 2024-02-19T21:11:58.648 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037835989, 2024-02-19T21:11:58.648 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.648 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.648 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.648 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.648 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.648 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.757694+0000", 2024-02-19T21:11:58.648 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.649 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.649 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.649 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.649 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.757694+0000", 2024-02-19T21:11:58.649 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:58.649 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.649 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.649 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.649 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.757692+0000", 2024-02-19T21:11:58.649 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.029339015 2024-02-19T21:11:58.649 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.649 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.650 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.650 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787031+0000", 2024-02-19T21:11:58.650 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0139000000000001e-05 2024-02-19T21:11:58.650 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.650 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.650 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.650 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787052+0000", 2024-02-19T21:11:58.650 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0044000000000001e-05 2024-02-19T21:11:58.650 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.650 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.650 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.651 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787062+0000", 2024-02-19T21:11:58.651 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8369000000000003e-05 2024-02-19T21:11:58.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.651 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.651 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.651 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787120+0000", 2024-02-19T21:11:58.651 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7753000000000001e-05 2024-02-19T21:11:58.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.651 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.651 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.651 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787138+0000", 2024-02-19T21:11:58.651 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0083446010000000001 2024-02-19T21:11:58.651 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.652 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.652 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.652 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.795482+0000", 2024-02-19T21:11:58.652 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8127000000000002e-05 2024-02-19T21:11:58.652 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.652 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.652 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.652 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.795530+0000", 2024-02-19T21:11:58.652 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037835989 2024-02-19T21:11:58.652 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.652 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.652 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.652 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.653 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.653 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.0.0:263 28.2 e132/99 -30:4327d3d8:::temp_28.2_0_14232_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:11:58.653 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.787076+0000", 2024-02-19T21:11:58.653 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 405.82392083899998, 2024-02-19T21:11:58.653 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.019684811, 2024-02-19T21:11:58.653 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.653 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.653 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.653 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.653 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.653 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787076+0000", 2024-02-19T21:11:58.653 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.653 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.653 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.653 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.654 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787076+0000", 2024-02-19T21:11:58.654 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:58.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.654 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.654 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.654 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.787074+0000", 2024-02-19T21:11:58.654 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011113661 2024-02-19T21:11:58.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.654 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.654 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.654 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.798187+0000", 2024-02-19T21:11:58.654 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2937e-05 2024-02-19T21:11:58.654 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.654 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.655 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.655 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.798200+0000", 2024-02-19T21:11:58.655 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1439999999999997e-06 2024-02-19T21:11:58.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.655 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.655 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.655 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.798207+0000", 2024-02-19T21:11:58.655 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7419000000000002e-05 2024-02-19T21:11:58.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.655 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.655 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.655 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.798245+0000", 2024-02-19T21:11:58.655 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7419999999999999e-05 2024-02-19T21:11:58.655 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.656 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.656 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.656 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.798262+0000", 2024-02-19T21:11:58.656 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0084319070000000006 2024-02-19T21:11:58.656 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.656 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.656 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.656 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.806694+0000", 2024-02-19T21:11:58.656 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6605000000000004e-05 2024-02-19T21:11:58.656 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.656 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.656 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.656 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.806761+0000", 2024-02-19T21:11:58.656 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.019684811 2024-02-19T21:11:58.657 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.657 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.657 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.657 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.657 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.657 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.0.0:264 28.2 e132/99 -30:4327d3d8:::temp_28.2_0_14232_1:head v 0'0, mlcod=0'0)", 2024-02-19T21:11:58.657 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.801755+0000", 2024-02-19T21:11:58.657 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 405.809241633, 2024-02-19T21:11:58.657 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022050645000000001, 2024-02-19T21:11:58.657 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.657 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.657 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.657 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.657 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.658 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.801755+0000", 2024-02-19T21:11:58.658 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.658 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.658 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.658 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.658 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.801755+0000", 2024-02-19T21:11:58.658 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:11:58.658 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.658 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.658 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.658 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.801753+0000", 2024-02-19T21:11:58.658 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013699945999999999 2024-02-19T21:11:58.658 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.658 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.659 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.659 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.815453+0000", 2024-02-19T21:11:58.659 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7495000000000001e-05 2024-02-19T21:11:58.659 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.659 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.659 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.659 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.815471+0000", 2024-02-19T21:11:58.659 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2979999999999992e-06 2024-02-19T21:11:58.659 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.659 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.659 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.659 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.815479+0000", 2024-02-19T21:11:58.659 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8300999999999997e-05 2024-02-19T21:11:58.659 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.659 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.660 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.660 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.815527+0000", 2024-02-19T21:11:58.660 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7969999999999999e-05 2024-02-19T21:11:58.660 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.660 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.660 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.660 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.815545+0000", 2024-02-19T21:11:58.660 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0082266249999999996 2024-02-19T21:11:58.660 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.660 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.660 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.660 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.823772+0000", 2024-02-19T21:11:58.660 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4051000000000003e-05 2024-02-19T21:11:58.660 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.661 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.661 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.661 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.823806+0000", 2024-02-19T21:11:58.661 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022050645000000001 2024-02-19T21:11:58.661 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.661 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.661 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.661 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.661 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.661 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.48947.0:1 112.0s0 112:0f35bed6:test-rados-api-smithi163-83952-16::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e167)", 2024-02-19T21:11:58.661 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:47.190422+0000", 2024-02-19T21:11:58.661 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 371.42057457999999, 2024-02-19T21:11:58.661 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0380564720000001, 2024-02-19T21:11:58.661 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.662 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.662 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:58.662 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.48947", 2024-02-19T21:11:58.662 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3754083491", 2024-02-19T21:11:58.662 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:11:58.662 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.662 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.662 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.662 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.662 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190422+0000", 2024-02-19T21:11:58.662 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.662 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.662 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.662 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.662 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190422+0000", 2024-02-19T21:11:58.663 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:58.663 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.663 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.663 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.663 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190420+0000", 2024-02-19T21:11:58.663 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4949999999999997e-06 2024-02-19T21:11:58.663 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.663 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.663 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.663 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190430+0000", 2024-02-19T21:11:58.663 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0844e-05 2024-02-19T21:11:58.663 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.663 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.663 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.664 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190440+0000", 2024-02-19T21:11:58.664 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3660000000000003e-06 2024-02-19T21:11:58.664 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.664 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.664 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.664 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190448+0000", 2024-02-19T21:11:58.664 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.083e-05 2024-02-19T21:11:58.664 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.664 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.664 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.664 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190499+0000", 2024-02-19T21:11:58.664 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0529999999999998e-06 2024-02-19T21:11:58.664 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.664 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.664 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:11:58.665 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:47.190502+0000", 2024-02-19T21:11:58.665 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.030012302 2024-02-19T21:11:58.665 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.665 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.665 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.665 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:48.220514+0000", 2024-02-19T21:11:58.665 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000132175 2024-02-19T21:11:58.665 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.665 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.665 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.665 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:48.220646+0000", 2024-02-19T21:11:58.665 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00040324399999999998 2024-02-19T21:11:58.665 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.665 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.666 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:11:58.666 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:48.221049+0000", 2024-02-19T21:11:58.666 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00034994699999999998 2024-02-19T21:11:58.666 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.666 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.666 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:11:58.666 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:48.221399+0000", 2024-02-19T21:11:58.666 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.007007498 2024-02-19T21:11:58.666 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.666 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.666 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.666 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:48.228407+0000", 2024-02-19T21:11:58.666 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1938000000000001e-05 2024-02-19T21:11:58.666 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:48.228479+0000", 2024-02-19T21:11:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0380564720000001 2024-02-19T21:11:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.49199.0:1 149.6s0 149:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e216)", 2024-02-19T21:11:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:06:36.212055+0000", 2024-02-19T21:11:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 322.39894200999998, 2024-02-19T21:11:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.008338194, 2024-02-19T21:11:58.667 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.49199", 2024-02-19T21:11:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2827260261", 2024-02-19T21:11:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:11:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212055+0000", 2024-02-19T21:11:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.668 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212055+0000", 2024-02-19T21:11:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:11:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212052+0000", 2024-02-19T21:11:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0199000000000001e-05 2024-02-19T21:11:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212073+0000", 2024-02-19T21:11:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1531000000000001e-05 2024-02-19T21:11:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.669 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212094+0000", 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6829999999999997e-06 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212103+0000", 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00054734599999999999 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212650+0000", 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8489999999999999e-06 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.670 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:11:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212654+0000", 2024-02-19T21:11:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0043875600000001 2024-02-19T21:11:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.217042+0000", 2024-02-19T21:11:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012985000000000001 2024-02-19T21:11:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.217171+0000", 2024-02-19T21:11:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00037497799999999999 2024-02-19T21:11:58.671 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.672 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.672 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:11:58.672 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.217546+0000", 2024-02-19T21:11:58.672 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020797239999999998 2024-02-19T21:11:58.672 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.672 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.672 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:11:58.672 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.219626+0000", 2024-02-19T21:11:58.672 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00069722600000000001 2024-02-19T21:11:58.672 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.672 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.672 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.672 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.220323+0000", 2024-02-19T21:11:58.673 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9684000000000006e-05 2024-02-19T21:11:58.673 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.673 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.673 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.673 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.220393+0000", 2024-02-19T21:11:58.673 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.008338194 2024-02-19T21:11:58.673 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.673 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.673 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.673 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.673 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.673 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22689.0:1 172.6s0 172:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-02-19T21:11:58.673 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:20.216387+0000", 2024-02-19T21:11:58.673 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 278.394610081, 2024-02-19T21:11:58.674 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0565913250000001, 2024-02-19T21:11:58.674 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.674 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.674 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:58.674 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22689", 2024-02-19T21:11:58.674 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/585802156", 2024-02-19T21:11:58.674 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:11:58.674 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.674 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.674 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.674 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.674 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216387+0000", 2024-02-19T21:11:58.675 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.675 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.675 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.675 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.675 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216387+0000", 2024-02-19T21:11:58.675 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:11:58.675 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.675 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.675 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.675 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216384+0000", 2024-02-19T21:11:58.675 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2157000000000001e-05 2024-02-19T21:11:58.675 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.675 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.675 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.675 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216396+0000", 2024-02-19T21:11:58.676 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.003e-05 2024-02-19T21:11:58.676 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.676 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.676 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.676 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216416+0000", 2024-02-19T21:11:58.676 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7499999999999998e-06 2024-02-19T21:11:58.676 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.676 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.676 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.676 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216426+0000", 2024-02-19T21:11:58.676 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00066776200000000002 2024-02-19T21:11:58.676 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.676 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.676 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.676 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.217093+0000", 2024-02-19T21:11:58.677 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6430000000000003e-06 2024-02-19T21:11:58.677 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.677 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.677 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:11:58.677 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.217098+0000", 2024-02-19T21:11:58.677 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.051702409 2024-02-19T21:11:58.677 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.677 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.677 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.677 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.268800+0000", 2024-02-19T21:11:58.677 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018998999999999999 2024-02-19T21:11:58.677 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.677 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.677 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.677 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.268990+0000", 2024-02-19T21:11:58.678 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00037534100000000001 2024-02-19T21:11:58.678 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.678 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.678 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:11:58.678 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.269366+0000", 2024-02-19T21:11:58.678 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001076088 2024-02-19T21:11:58.678 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.678 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.678 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:11:58.678 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.270442+0000", 2024-02-19T21:11:58.678 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024701089999999998 2024-02-19T21:11:58.678 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.678 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.678 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.679 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.272912+0000", 2024-02-19T21:11:58.679 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6147000000000002e-05 2024-02-19T21:11:58.679 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.679 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.679 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.679 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.272978+0000", 2024-02-19T21:11:58.679 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0565913250000001 2024-02-19T21:11:58.679 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.679 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.679 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.679 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.679 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.679 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.1.0:634 61.b e278/114 61:d461b0a5:test-rados-api-smithi163-83928-34::foo:14 v 278'95, mlcod=275'91)", 2024-02-19T21:11:58.679 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:38.232659+0000", 2024-02-19T21:11:58.679 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 260.37833732500002, 2024-02-19T21:11:58.680 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0151769, 2024-02-19T21:11:58.680 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.680 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.680 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.680 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.680 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.680 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.232659+0000", 2024-02-19T21:11:58.680 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.680 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.680 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.680 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.680 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.232659+0000", 2024-02-19T21:11:58.680 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:11:58.680 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.680 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.681 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.681 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.232659+0000", 2024-02-19T21:11:58.681 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3270000000000001e-06 2024-02-19T21:11:58.681 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.681 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.681 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.681 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.232667+0000", 2024-02-19T21:11:58.681 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00080230000000000004 2024-02-19T21:11:58.681 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.681 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.681 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.681 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.233470+0000", 2024-02-19T21:11:58.681 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7861999999999999e-05 2024-02-19T21:11:58.682 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.682 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.682 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.682 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.233488+0000", 2024-02-19T21:11:58.682 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3637000000000003e-05 2024-02-19T21:11:58.682 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.682 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.682 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.682 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.233531+0000", 2024-02-19T21:11:58.682 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1175999999999999e-05 2024-02-19T21:11:58.682 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.682 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.682 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.682 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.233552+0000", 2024-02-19T21:11:58.682 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014260111000000001 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.247812+0000", 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3896999999999998e-05 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.247836+0000", 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0151769 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.683 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.49649.0:1 201.2s0 201:5a7a12bd:::oid_SimpleStatPPNS_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e352)", 2024-02-19T21:11:58.684 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:08:52.333687+0000", 2024-02-19T21:11:58.684 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 186.27730947699999, 2024-02-19T21:11:58.684 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0054582620000001, 2024-02-19T21:11:58.684 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.684 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.684 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:58.684 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.49649", 2024-02-19T21:11:58.684 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2569144910", 2024-02-19T21:11:58.684 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:11:58.684 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.684 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.684 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.684 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.684 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333687+0000", 2024-02-19T21:11:58.685 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.685 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.685 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.685 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.685 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333687+0000", 2024-02-19T21:11:58.685 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:11:58.685 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.685 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.685 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.685 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333685+0000", 2024-02-19T21:11:58.685 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0747000000000001e-05 2024-02-19T21:11:58.685 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.685 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.685 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.686 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333695+0000", 2024-02-19T21:11:58.686 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3277e-05 2024-02-19T21:11:58.686 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.686 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.686 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.686 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333709+0000", 2024-02-19T21:11:58.686 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0219999999999993e-06 2024-02-19T21:11:58.686 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.686 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.686 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.686 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333718+0000", 2024-02-19T21:11:58.686 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8285999999999998e-05 2024-02-19T21:11:58.686 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.686 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.687 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.687 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333766+0000", 2024-02-19T21:11:58.687 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9040000000000002e-06 2024-02-19T21:11:58.687 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.687 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.687 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:11:58.687 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333770+0000", 2024-02-19T21:11:58.687 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.003658465 2024-02-19T21:11:58.687 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.687 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.687 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.687 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:53.337428+0000", 2024-02-19T21:11:58.687 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00016740599999999999 2024-02-19T21:11:58.687 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.688 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.688 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.688 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:53.337596+0000", 2024-02-19T21:11:58.688 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00055418199999999996 2024-02-19T21:11:58.688 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.688 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.688 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:11:58.688 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:53.338150+0000", 2024-02-19T21:11:58.688 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000445174 2024-02-19T21:11:58.688 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.688 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.688 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:11:58.688 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:53.338595+0000", 2024-02-19T21:11:58.688 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00050577499999999997 2024-02-19T21:11:58.688 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.689 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.689 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.689 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:53.339101+0000", 2024-02-19T21:11:58.689 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4749000000000001e-05 2024-02-19T21:11:58.689 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.689 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.689 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.689 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:53.339146+0000", 2024-02-19T21:11:58.689 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0054582620000001 2024-02-19T21:11:58.689 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.689 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.689 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.689 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.689 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.690 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:1424 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94681540386816 gen 2] snapc 0=[] ondisk+write+known_if_redirected e433)", 2024-02-19T21:11:58.690 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:10:40.311743+0000", 2024-02-19T21:11:58.690 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 78.299254159, 2024-02-19T21:11:58.690 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.012662921, 2024-02-19T21:11:58.690 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.690 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:11:58.690 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:58.690 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:11:58.690 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:11:58.690 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1424 2024-02-19T21:11:58.690 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.690 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.690 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.690 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.690 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311743+0000", 2024-02-19T21:11:58.691 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.691 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.691 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.691 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.691 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311743+0000", 2024-02-19T21:11:58.691 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:11:58.691 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.691 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.691 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.691 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311740+0000", 2024-02-19T21:11:58.691 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0318000000000001e-05 2024-02-19T21:11:58.691 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.691 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.691 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.692 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311760+0000", 2024-02-19T21:11:58.692 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3355e-05 2024-02-19T21:11:58.692 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.692 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.692 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.692 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311773+0000", 2024-02-19T21:11:58.692 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3659999999999992e-06 2024-02-19T21:11:58.692 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.692 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.692 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.692 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311782+0000", 2024-02-19T21:11:58.692 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4440000000000001e-05 2024-02-19T21:11:58.692 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.692 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.692 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.693 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311836+0000", 2024-02-19T21:11:58.693 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1285e-05 2024-02-19T21:11:58.693 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.693 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.693 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:11:58.693 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311848+0000", 2024-02-19T21:11:58.693 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.99090941099999996 2024-02-19T21:11:58.693 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.693 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.693 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.693 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.302757+0000", 2024-02-19T21:11:58.693 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00026452600000000001 2024-02-19T21:11:58.693 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.693 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.694 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for degraded object", 2024-02-19T21:11:58.694 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.303021+0000", 2024-02-19T21:11:58.694 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.020182997000000001 2024-02-19T21:11:58.694 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.694 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.694 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.694 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.323204+0000", 2024-02-19T21:11:58.694 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4702999999999994e-05 2024-02-19T21:11:58.694 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.694 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.694 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.694 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.323269+0000", 2024-02-19T21:11:58.694 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001136399 2024-02-19T21:11:58.694 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.694 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.695 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.695 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.324406+0000", 2024-02-19T21:11:58.695 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.012662921 2024-02-19T21:11:58.695 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.695 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.695 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.695 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.695 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.695 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.25753.0:6559 61.2 e452/303 61:4782fe63:test-rados-api-smithi163-83928-70::foo:head v 452'97, mlcod=447'95)", 2024-02-19T21:11:58.695 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:11:04.369908+0000", 2024-02-19T21:11:58.695 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 54.241088996999999, 2024-02-19T21:11:58.695 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.01199457, 2024-02-19T21:11:58.695 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.695 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:11:58.696 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.696 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.696 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.696 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.369908+0000", 2024-02-19T21:11:58.696 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.696 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.696 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.696 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.696 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.369908+0000", 2024-02-19T21:11:58.696 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:11:58.696 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.696 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.696 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.696 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.369905+0000", 2024-02-19T21:11:58.697 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3260000000000001e-05 2024-02-19T21:11:58.697 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.697 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.697 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.697 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.369928+0000", 2024-02-19T21:11:58.697 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9531e-05 2024-02-19T21:11:58.697 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.697 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.697 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.697 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.369947+0000", 2024-02-19T21:11:58.697 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2879999999999998e-06 2024-02-19T21:11:58.697 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.697 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.697 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.698 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.369957+0000", 2024-02-19T21:11:58.698 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4946999999999998e-05 2024-02-19T21:11:58.698 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.698 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.698 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.698 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.370042+0000", 2024-02-19T21:11:58.698 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7177000000000001e-05 2024-02-19T21:11:58.698 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.698 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.698 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.698 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.370069+0000", 2024-02-19T21:11:58.698 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011802573 2024-02-19T21:11:58.698 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.698 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.698 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:11:58.699 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.381871+0000", 2024-02-19T21:11:58.699 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0961999999999999e-05 2024-02-19T21:11:58.699 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.699 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.699 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.699 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.381902+0000", 2024-02-19T21:11:58.699 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.01199457 2024-02-19T21:11:58.699 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.699 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.699 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.699 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.699 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.699 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:6758 217.1d 217:b8000000::::head [pgnls start_epoch 461] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e461)", 2024-02-19T21:11:58.699 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:11:29.468678+0000", 2024-02-19T21:11:58.700 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 29.142318801999998, 2024-02-19T21:11:58.700 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025771940000000001, 2024-02-19T21:11:58.700 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.700 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:11:58.700 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:11:58.700 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:11:58.700 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:11:58.700 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 6758 2024-02-19T21:11:58.700 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.700 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.700 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.700 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.700 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.468678+0000", 2024-02-19T21:11:58.700 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.701 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.701 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.701 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.701 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.468678+0000", 2024-02-19T21:11:58.701 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:11:58.701 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.701 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.701 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.701 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.468675+0000", 2024-02-19T21:11:58.701 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0536e-05 2024-02-19T21:11:58.701 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.701 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.701 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.701 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.468686+0000", 2024-02-19T21:11:58.702 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5574e-05 2024-02-19T21:11:58.702 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.702 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.702 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.702 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.468701+0000", 2024-02-19T21:11:58.702 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3204999999999999e-05 2024-02-19T21:11:58.702 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.702 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.702 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.702 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.468724+0000", 2024-02-19T21:11:58.702 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0540000000000001e-05 2024-02-19T21:11:58.702 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.702 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.702 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.703 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.468775+0000", 2024-02-19T21:11:58.703 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8099999999999997e-06 2024-02-19T21:11:58.703 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.703 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.703 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:11:58.703 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.468780+0000", 2024-02-19T21:11:58.703 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002325569 2024-02-19T21:11:58.703 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.703 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.703 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.703 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.471105+0000", 2024-02-19T21:11:58.703 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4730999999999997e-05 2024-02-19T21:11:58.703 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.703 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.703 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.704 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.471140+0000", 2024-02-19T21:11:58.704 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000115102 2024-02-19T21:11:58.704 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.704 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.704 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.704 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.471255+0000", 2024-02-19T21:11:58.704 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025771940000000001 2024-02-19T21:11:58.704 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.704 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.705 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.705 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.705 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.705 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "replica_scrub(pg: 61.1c,from:0'0,to:429'153,epoch:472/442,start:61:38000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-02-19T21:11:58.705 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:11:44.963360+0000", 2024-02-19T21:11:58.705 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 13.647636614, 2024-02-19T21:11:58.705 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00074667199999999998, 2024-02-19T21:11:58.705 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:11:58.705 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:11:58.705 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:11:58.705 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.705 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:11:58.705 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:44.963360+0000", 2024-02-19T21:11:58.705 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:11:58.705 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.706 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.706 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:11:58.706 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:44.963360+0000", 2024-02-19T21:11:58.706 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:11:58.706 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.706 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.706 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:11:58.706 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:44.963359+0000", 2024-02-19T21:11:58.706 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5930000000000002e-06 2024-02-19T21:11:58.706 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.706 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.706 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:11:58.706 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:44.963366+0000", 2024-02-19T21:11:58.706 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2038000000000001e-05 2024-02-19T21:11:58.707 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.707 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.707 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:11:58.707 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:44.963378+0000", 2024-02-19T21:11:58.707 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4290000000000004e-06 2024-02-19T21:11:58.707 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.707 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.707 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:11:58.707 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:44.963385+0000", 2024-02-19T21:11:58.707 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00067298 2024-02-19T21:11:58.707 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.707 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.707 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:11:58.707 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:44.964058+0000", 2024-02-19T21:11:58.707 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0095000000000001e-05 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:44.964068+0000", 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8769000000000001e-05 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:44.964107+0000", 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00074667199999999998 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:11:58.708 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:11:58.709 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:11:58.738 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:57.243589+0000 mgr.y (mgr.14550) 444 : cluster [DBG] pgmap v721: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 860 B/s rd, 0 op/s; 146 B/s, 0 objects/s recovering 2024-02-19T21:11:58.738 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:57.507230+0000 mon.b (mon.1) 628 : audit [INF] from='client.? v1:172.21.15.163:0/3787558960' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-19T21:11:58.738 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:57.507505+0000 mon.a (mon.0) 2693 : audit [INF] from='client.48838 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-19T21:11:58.738 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:57.507581+0000 mon.a (mon.0) 2694 : cluster [INF] Client client.admin marked osd.0 out, while it was still marked up 2024-02-19T21:11:58.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:57.243589+0000 mgr.y (mgr.14550) 444 : cluster [DBG] pgmap v721: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 860 B/s rd, 0 op/s; 146 B/s, 0 objects/s recovering 2024-02-19T21:11:58.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:57.507230+0000 mon.b (mon.1) 628 : audit [INF] from='client.? v1:172.21.15.163:0/3787558960' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-19T21:11:58.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:57.507505+0000 mon.a (mon.0) 2693 : audit [INF] from='client.48838 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-19T21:11:58.738 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:57.507581+0000 mon.a (mon.0) 2694 : cluster [INF] Client client.admin marked osd.0 out, while it was still marked up 2024-02-19T21:11:58.740 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:58.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:57.243589+0000 mgr.y (mgr.14550) 444 : cluster [DBG] pgmap v721: 238 pgs: 238 active+clean; 3.8 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 860 B/s rd, 0 op/s; 146 B/s, 0 objects/s recovering 2024-02-19T21:11:58.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:57.507230+0000 mon.b (mon.1) 628 : audit [INF] from='client.? v1:172.21.15.163:0/3787558960' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-19T21:11:58.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:57.507505+0000 mon.a (mon.0) 2693 : audit [INF] from='client.48838 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-19T21:11:58.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:57.507581+0000 mon.a (mon.0) 2694 : cluster [INF] Client client.admin marked osd.0 out, while it was still marked up 2024-02-19T21:11:58.877 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:11:59.192 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:11:59.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:59.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:11:59.255 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:11:59.267 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-19T21:11:59.297 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:11:59.350 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:59.445 INFO:journalctl@ceph.osd.1.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-1[53263]: debug 2024-02-19T21:11:59.380+0000 7f78e1789700 -1 snap_mapper.add_oid found existing snaps mapped on 219:dfd5f9d5:test-rados-api-smithi163-83928-76::foo:2, removing 2024-02-19T21:11:59.446 INFO:journalctl@ceph.osd.3.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-3[62371]: debug 2024-02-19T21:11:59.381+0000 7f39548d3700 -1 snap_mapper.add_oid found existing snaps mapped on 219:dfd5f9d5:test-rados-api-smithi163-83928-76::foo:2, removing 2024-02-19T21:11:59.446 INFO:journalctl@ceph.osd.2.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-2[57708]: debug 2024-02-19T21:11:59.381+0000 7f5b2fd21700 -1 snap_mapper.add_oid found existing snaps mapped on 219:dfd5f9d5:test-rados-api-smithi163-83928-76::foo:2, removing 2024-02-19T21:11:59.552 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:11:59.555 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:59.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:11:59.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:58.447727+0000 mon.a (mon.0) 2695 : audit [INF] from='client.48838 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-02-19T21:11:59.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:58.447814+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e482: 8 total, 8 up, 7 in 2024-02-19T21:11:59.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:59.299384+0000 mon.a (mon.0) 2697 : cluster [DBG] osdmap e483: 8 total, 8 up, 7 in 2024-02-19T21:11:59.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:59.383535+0000 mon.b (mon.1) 629 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:59.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:59.383950+0000 mon.a (mon.0) 2698 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:59.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:59.384509+0000 mon.b (mon.1) 630 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-75"}]: dispatch 2024-02-19T21:11:59.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:11:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:11:59.384758+0000 mon.a (mon.0) 2699 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-75"}]: dispatch 2024-02-19T21:11:59.625 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:11:59.656 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:11:59.683 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:11:59.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:58.447727+0000 mon.a (mon.0) 2695 : audit [INF] from='client.48838 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-02-19T21:11:59.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:58.447814+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e482: 8 total, 8 up, 7 in 2024-02-19T21:11:59.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:59.299384+0000 mon.a (mon.0) 2697 : cluster [DBG] osdmap e483: 8 total, 8 up, 7 in 2024-02-19T21:11:59.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:59.383535+0000 mon.b (mon.1) 629 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:59.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:59.383950+0000 mon.a (mon.0) 2698 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:59.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:59.384509+0000 mon.b (mon.1) 630 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-75"}]: dispatch 2024-02-19T21:11:59.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:11:59.384758+0000 mon.a (mon.0) 2699 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-75"}]: dispatch 2024-02-19T21:11:59.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:58.447727+0000 mon.a (mon.0) 2695 : audit [INF] from='client.48838 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-02-19T21:11:59.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:58.447814+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e482: 8 total, 8 up, 7 in 2024-02-19T21:11:59.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:59.299384+0000 mon.a (mon.0) 2697 : cluster [DBG] osdmap e483: 8 total, 8 up, 7 in 2024-02-19T21:11:59.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:59.383535+0000 mon.b (mon.1) 629 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:59.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:59.383950+0000 mon.a (mon.0) 2698 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:11:59.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:59.384509+0000 mon.b (mon.1) 630 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-75"}]: dispatch 2024-02-19T21:11:59.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:11:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:11:59.384758+0000 mon.a (mon.0) 2699 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-75"}]: dispatch 2024-02-19T21:11:59.756 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:11:59.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:11:59.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:11:59.860 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:11:59.887 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:00.009 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:12:00.085 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:00.088 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:00.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:00.189 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:00.218 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:00.666 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:00.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:00.771 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:00.797 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:00.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:12:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:12:00.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:11:59.244083+0000 mgr.y (mgr.14550) 445 : cluster [DBG] pgmap v723: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 145 B/s, 0 objects/s recovering 2024-02-19T21:12:00.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:00.300245+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e484: 8 total, 8 up, 7 in 2024-02-19T21:12:00.959 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:11:59.244083+0000 mgr.y (mgr.14550) 445 : cluster [DBG] pgmap v723: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 145 B/s, 0 objects/s recovering 2024-02-19T21:12:00.959 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:00.300245+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e484: 8 total, 8 up, 7 in 2024-02-19T21:12:00.959 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:11:59.244083+0000 mgr.y (mgr.14550) 445 : cluster [DBG] pgmap v723: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 145 B/s, 0 objects/s recovering 2024-02-19T21:12:00.959 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:00.300245+0000 mon.a (mon.0) 2700 : cluster [DBG] osdmap e484: 8 total, 8 up, 7 in 2024-02-19T21:12:00.995 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:00.998 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:00.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:01.099 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:01.129 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:01.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.213+0000 7f1311971700 1 -- 172.21.15.163:0/516788331 >> v1:172.21.15.163:6790/0 conn(0x7f130c0f94d0 legacy=0x7f130c1029a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:01.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.213+0000 7f1311971700 1 -- 172.21.15.163:0/516788331 shutdown_connections 2024-02-19T21:12:01.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.213+0000 7f1311971700 1 -- 172.21.15.163:0/516788331 >> 172.21.15.163:0/516788331 conn(0x7f130c0f2470 msgr2=0x7f130c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:01.217 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.213+0000 7f1311971700 1 -- 172.21.15.163:0/516788331 shutdown_connections 2024-02-19T21:12:01.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.213+0000 7f1311971700 1 -- 172.21.15.163:0/516788331 wait complete. 2024-02-19T21:12:01.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.213+0000 7f1311971700 1 librados: starting msgr at 2024-02-19T21:12:01.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.213+0000 7f1311971700 1 librados: starting objecter 2024-02-19T21:12:01.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f1311971700 1 Processor -- start 2024-02-19T21:12:01.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f1311971700 1 -- start start 2024-02-19T21:12:01.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f1311971700 1 librados: setting wanted keys 2024-02-19T21:12:01.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f1311971700 1 librados: calling monclient init 2024-02-19T21:12:01.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f1311971700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f130c117290 con 0x7f130c1042b0 2024-02-19T21:12:01.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f1311971700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f130c118470 con 0x7f130c0f8920 2024-02-19T21:12:01.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f1311971700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f130c119670 con 0x7f130c0f94d0 2024-02-19T21:12:01.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f130affd700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f130c0f8920 0x7f130c0ff3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:56282/0 (socket says 172.21.15.163:56282) 2024-02-19T21:12:01.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f130affd700 1 -- 172.21.15.163:0/3808377504 learned_addr learned my addr 172.21.15.163:0/3808377504 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:01.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f12f3fff700 10 client.?.objecter ms_handle_connect 0x7f130c0f8920 2024-02-19T21:12:01.219 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f12f3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:01.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f12f3fff700 10 client.?.objecter ms_handle_connect 0x7f130c1042b0 2024-02-19T21:12:01.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f12f3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:01.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f12f3fff700 10 client.?.objecter ms_handle_connect 0x7f130c0f94d0 2024-02-19T21:12:01.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f12f3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:01.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083263623 0 0) 0x7f130c118470 con 0x7f130c0f8920 2024-02-19T21:12:01.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.214+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12f40033f0 con 0x7f130c0f8920 2024-02-19T21:12:01.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275382621 0 0) 0x7f130c119670 con 0x7f130c0f94d0 2024-02-19T21:12:01.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f130c118470 con 0x7f130c0f94d0 2024-02-19T21:12:01.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729919354 0 0) 0x7f130c117290 con 0x7f130c1042b0 2024-02-19T21:12:01.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f130c119670 con 0x7f130c1042b0 2024-02-19T21:12:01.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 131251642 0 0) 0x7f12f40033f0 con 0x7f130c0f8920 2024-02-19T21:12:01.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f130c117290 con 0x7f130c0f8920 2024-02-19T21:12:01.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f12fc001950 con 0x7f130c0f8920 2024-02-19T21:12:01.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3775833191 0 0) 0x7f130c118470 con 0x7f130c0f94d0 2024-02-19T21:12:01.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12f40033f0 con 0x7f130c0f94d0 2024-02-19T21:12:01.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1300000fe0 con 0x7f130c0f94d0 2024-02-19T21:12:01.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2986896792 0 0) 0x7f130c119670 con 0x7f130c1042b0 2024-02-19T21:12:01.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f130c118470 con 0x7f130c1042b0 2024-02-19T21:12:01.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1304001af0 con 0x7f130c1042b0 2024-02-19T21:12:01.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2313176569 0 0) 0x7f130c117290 con 0x7f130c0f8920 2024-02-19T21:12:01.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 >> v1:172.21.15.163:6790/0 conn(0x7f130c0f94d0 legacy=0x7f130c0ffad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:01.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 >> v1:172.21.15.163:6789/0 conn(0x7f130c1042b0 legacy=0x7f130c115950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:01.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c11a870 con 0x7f130c0f8920 2024-02-19T21:12:01.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f1311971700 1 -- 172.21.15.163:0/3808377504 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130c117480 con 0x7f130c0f8920 2024-02-19T21:12:01.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f1311971700 10 client.48844.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:01.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f1311971700 1 -- 172.21.15.163:0/3808377504 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f130c1178f0 con 0x7f130c0f8920 2024-02-19T21:12:01.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f1311971700 1 librados: init done 2024-02-19T21:12:01.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f12fc002340 con 0x7f130c0f8920 2024-02-19T21:12:01.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.215+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f12fc005a80 con 0x7f130c0f8920 2024-02-19T21:12:01.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.216+0000 7f1311971700 1 -- 172.21.15.163:0/3808377504 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f8004f20 con 0x7f130c0f8920 2024-02-19T21:12:01.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.217+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f12fc001ea0 con 0x7f130c0f8920 2024-02-19T21:12:01.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.217+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(484..484 src has 43..484) v4 ==== 7740+0+0 (unknown 3827605294 0 0) 0x7f12fc078b10 con 0x7f130c0f8920 2024-02-19T21:12:01.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.217+0000 7f12f3fff700 10 client.48844.objecter ms_dispatch 0x7f130c001550 osd_map(484..484 src has 43..484) v4 2024-02-19T21:12:01.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.217+0000 7f12f3fff700 3 client.48844.objecter handle_osd_map got epochs [484,484] > 0 2024-02-19T21:12:01.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.217+0000 7f12f3fff700 3 client.48844.objecter handle_osd_map decoding full epoch 484 2024-02-19T21:12:01.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.217+0000 7f12f3fff700 20 client.48844.objecter dump_active .. 0 homeless 2024-02-19T21:12:01.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.217+0000 7f12f3fff700 10 client.48844.objecter ms_handle_connect 0x7f12f4063ed0 2024-02-19T21:12:01.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.221+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f12fc04c040 con 0x7f130c0f8920 2024-02-19T21:12:01.227 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:01.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:01.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:01.255 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:01.332 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:01.360 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:01.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:01.432+0000 7f1311971700 1 -- 172.21.15.163:0/3808377504 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f12f80051d0 con 0x7f130c0f8920 2024-02-19T21:12:01.561 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:01.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:01.665 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:01.712 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:00.300056+0000 osd.7 (osd.7) 227 : cluster [DBG] 2.0 starting backfill to osd.3 from (0'0,0'0] MAX to 397'2 2024-02-19T21:12:01.712 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:00.499014+0000 mgr.y (mgr.14550) 446 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:01.712 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:01.302486+0000 mon.a (mon.0) 2701 : cluster [DBG] osdmap e485: 8 total, 8 up, 7 in 2024-02-19T21:12:01.712 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:01.312064+0000 mon.b (mon.1) 631 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:01.712 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:01.313531+0000 mon.a (mon.0) 2702 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:01.712 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:01.434446+0000 mon.b (mon.1) 632 : audit [INF] from='client.? v1:172.21.15.163:0/3808377504' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:12:01.712 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:01.434788+0000 mon.a (mon.0) 2703 : audit [INF] from='client.48844 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:12:01.718 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:01.737 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:01.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:01.842 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:01.870 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:01.898 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:01.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:01.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:01.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:00.300056+0000 osd.7 (osd.7) 227 : cluster [DBG] 2.0 starting backfill to osd.3 from (0'0,0'0] MAX to 397'2 2024-02-19T21:12:01.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:00.499014+0000 mgr.y (mgr.14550) 446 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:01.302486+0000 mon.a (mon.0) 2701 : cluster [DBG] osdmap e485: 8 total, 8 up, 7 in 2024-02-19T21:12:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:01.312064+0000 mon.b (mon.1) 631 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:01.313531+0000 mon.a (mon.0) 2702 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:01.434446+0000 mon.b (mon.1) 632 : audit [INF] from='client.? v1:172.21.15.163:0/3808377504' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:12:01.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:01.434788+0000 mon.a (mon.0) 2703 : audit [INF] from='client.48844 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:12:01.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:00.300056+0000 osd.7 (osd.7) 227 : cluster [DBG] 2.0 starting backfill to osd.3 from (0'0,0'0] MAX to 397'2 2024-02-19T21:12:01.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:00.499014+0000 mgr.y (mgr.14550) 446 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:01.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:01.302486+0000 mon.a (mon.0) 2701 : cluster [DBG] osdmap e485: 8 total, 8 up, 7 in 2024-02-19T21:12:01.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:01.312064+0000 mon.b (mon.1) 631 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:01.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:01.313531+0000 mon.a (mon.0) 2702 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:01.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:01.434446+0000 mon.b (mon.1) 632 : audit [INF] from='client.? v1:172.21.15.163:0/3808377504' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:12:01.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:01.434788+0000 mon.a (mon.0) 2703 : audit [INF] from='client.48844 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:12:02.001 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:02.022 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:12:02.022 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:12:02.022 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-02-19T21:12:02.022 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:12:02.028 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:02.047 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:02.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:02.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:02.151 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:02.178 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:02.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:02.311+0000 7f12f3fff700 1 -- 172.21.15.163:0/3808377504 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v486) v1 ==== 91+0+0 (unknown 7732122 0 0) 0x7f12fc04a450 con 0x7f130c0f8920 2024-02-19T21:12:02.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:02.313+0000 7f1311971700 10 librados: watch_flush enter 2024-02-19T21:12:02.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:02.313+0000 7f1311971700 10 librados: watch_flush exit 2024-02-19T21:12:02.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:02.313+0000 7f1311971700 20 client.48844.objecter shutdown clearing up homeless session... 2024-02-19T21:12:02.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:02.313+0000 7f1311971700 10 client.48844.objecter successfully canceled tick 2024-02-19T21:12:02.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:02.314+0000 7f1311971700 1 -- 172.21.15.163:0/3808377504 >> v1:172.21.15.163:6800/1552514681 conn(0x7f12f4063ed0 legacy=0x7f12f4066390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:02.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:02.314+0000 7f1311971700 1 -- 172.21.15.163:0/3808377504 >> v1:172.21.15.193:6789/0 conn(0x7f130c0f8920 legacy=0x7f130c0ff3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:02.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:02.315+0000 7f1311971700 1 -- 172.21.15.163:0/3808377504 shutdown_connections 2024-02-19T21:12:02.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:02.315+0000 7f1311971700 1 -- 172.21.15.163:0/3808377504 >> 172.21.15.163:0/3808377504 conn(0x7f130c0f2470 msgr2=0x7f130c0f4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:02.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:02.315+0000 7f1311971700 1 -- 172.21.15.163:0/3808377504 shutdown_connections 2024-02-19T21:12:02.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:02.315+0000 7f1311971700 1 -- 172.21.15.163:0/3808377504 wait complete. 2024-02-19T21:12:02.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:02.315+0000 7f1311971700 1 librados: shutdown 2024-02-19T21:12:02.322 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:12:02.677 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:01.244618+0000 mgr.y (mgr.14550) 447 : cluster [DBG] pgmap v726: 206 pgs: 8 active+recovery_wait+degraded, 4 active+recovering, 2 remapped+peering, 192 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 95/1095 objects degraded (8.676%); 7/1095 objects misplaced (0.639%); 183 B/s, 13 objects/s recovering 2024-02-19T21:12:02.678 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:01.559079+0000 mon.a (mon.0) 2704 : cluster [WRN] Health check failed: Degraded data redundancy: 95/1095 objects degraded (8.676%), 8 pgs degraded (PG_DEGRADED) 2024-02-19T21:12:02.678 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:02.299765+0000 mon.a (mon.0) 2705 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:12:02.678 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:02.308793+0000 mon.a (mon.0) 2706 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:02.678 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:02.308879+0000 mon.a (mon.0) 2707 : audit [INF] from='client.48844 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:12:02.678 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:02.308931+0000 mon.a (mon.0) 2708 : cluster [DBG] osdmap e486: 8 total, 8 up, 7 in 2024-02-19T21:12:02.678 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:02.318719+0000 mon.b (mon.1) 633 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:12:02.678 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:02.324486+0000 mon.b (mon.1) 634 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:02.678 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:02.324959+0000 mon.a (mon.0) 2709 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:02.695 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-19T21:12:02.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:01.244618+0000 mgr.y (mgr.14550) 447 : cluster [DBG] pgmap v726: 206 pgs: 8 active+recovery_wait+degraded, 4 active+recovering, 2 remapped+peering, 192 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 95/1095 objects degraded (8.676%); 7/1095 objects misplaced (0.639%); 183 B/s, 13 objects/s recovering 2024-02-19T21:12:02.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:01.559079+0000 mon.a (mon.0) 2704 : cluster [WRN] Health check failed: Degraded data redundancy: 95/1095 objects degraded (8.676%), 8 pgs degraded (PG_DEGRADED) 2024-02-19T21:12:02.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:02.299765+0000 mon.a (mon.0) 2705 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:12:02.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:02.308793+0000 mon.a (mon.0) 2706 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:02.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:02.308879+0000 mon.a (mon.0) 2707 : audit [INF] from='client.48844 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:12:02.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:02.308931+0000 mon.a (mon.0) 2708 : cluster [DBG] osdmap e486: 8 total, 8 up, 7 in 2024-02-19T21:12:02.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:02.318719+0000 mon.b (mon.1) 633 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:12:02.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:02.324486+0000 mon.b (mon.1) 634 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:02.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:02.324959+0000 mon.a (mon.0) 2709 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:02.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:01.244618+0000 mgr.y (mgr.14550) 447 : cluster [DBG] pgmap v726: 206 pgs: 8 active+recovery_wait+degraded, 4 active+recovering, 2 remapped+peering, 192 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s; 95/1095 objects degraded (8.676%); 7/1095 objects misplaced (0.639%); 183 B/s, 13 objects/s recovering 2024-02-19T21:12:02.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:01.559079+0000 mon.a (mon.0) 2704 : cluster [WRN] Health check failed: Degraded data redundancy: 95/1095 objects degraded (8.676%), 8 pgs degraded (PG_DEGRADED) 2024-02-19T21:12:02.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:02.299765+0000 mon.a (mon.0) 2705 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:12:02.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:02.308793+0000 mon.a (mon.0) 2706 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:02.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:02.308879+0000 mon.a (mon.0) 2707 : audit [INF] from='client.48844 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:12:02.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:02.308931+0000 mon.a (mon.0) 2708 : cluster [DBG] osdmap e486: 8 total, 8 up, 7 in 2024-02-19T21:12:02.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:02.318719+0000 mon.b (mon.1) 633 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:12:02.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:02.324486+0000 mon.b (mon.1) 634 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:02.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:02.324959+0000 mon.a (mon.0) 2709 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:02.961 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:12:03.746 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:12:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:12:03] "GET /metrics HTTP/1.1" 200 190107 "" "Prometheus/2.33.4" 2024-02-19T21:12:04.206 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 6, 7, 3, 4] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:12:04.207 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:12:04.207 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [1, 2, 5, 6, 7, 3, 4] 2024-02-19T21:12:04.207 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd out 6 2024-02-19T21:12:04.428 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:04.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:04.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:02.520104+0000 osd.4 (osd.4) 177 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:12:04.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:02.521518+0000 osd.4 (osd.4) 178 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:12:04.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:03.245246+0000 mgr.y (mgr.14550) 448 : cluster [DBG] pgmap v729: 238 pgs: 32 unknown, 8 active+recovery_wait+degraded, 4 active+recovering, 2 remapped+peering, 192 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s; 95/1095 objects degraded (8.676%); 7/1095 objects misplaced (0.639%); 275 B/s, 20 objects/s recovering 2024-02-19T21:12:04.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:03.253392+0000 osd.3 (osd.3) 219 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:12:04.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:03.255389+0000 osd.3 (osd.3) 220 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:12:04.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:03.306163+0000 mon.a (mon.0) 2710 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:12:04.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:03.306246+0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e487: 8 total, 8 up, 7 in 2024-02-19T21:12:04.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:03.307878+0000 mon.b (mon.1) 635 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:04.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:03.308388+0000 mon.a (mon.0) 2712 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:04.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:02.520104+0000 osd.4 (osd.4) 177 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:12:04.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:02.521518+0000 osd.4 (osd.4) 178 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:12:04.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:03.245246+0000 mgr.y (mgr.14550) 448 : cluster [DBG] pgmap v729: 238 pgs: 32 unknown, 8 active+recovery_wait+degraded, 4 active+recovering, 2 remapped+peering, 192 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s; 95/1095 objects degraded (8.676%); 7/1095 objects misplaced (0.639%); 275 B/s, 20 objects/s recovering 2024-02-19T21:12:04.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:03.253392+0000 osd.3 (osd.3) 219 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:12:04.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:03.255389+0000 osd.3 (osd.3) 220 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:12:04.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:03.306163+0000 mon.a (mon.0) 2710 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:12:04.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:03.306246+0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e487: 8 total, 8 up, 7 in 2024-02-19T21:12:04.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:03.307878+0000 mon.b (mon.1) 635 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:04.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:03.308388+0000 mon.a (mon.0) 2712 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:04.507 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:12:04.507 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:12:04.507 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-02-19T21:12:04.507 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-02-19T21:12:04.507 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:12:04.531 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:04.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:02.520104+0000 osd.4 (osd.4) 177 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:12:04.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:02.521518+0000 osd.4 (osd.4) 178 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:12:04.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:03.245246+0000 mgr.y (mgr.14550) 448 : cluster [DBG] pgmap v729: 238 pgs: 32 unknown, 8 active+recovery_wait+degraded, 4 active+recovering, 2 remapped+peering, 192 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s; 95/1095 objects degraded (8.676%); 7/1095 objects misplaced (0.639%); 275 B/s, 20 objects/s recovering 2024-02-19T21:12:04.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:03.253392+0000 osd.3 (osd.3) 219 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:12:04.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:03.255389+0000 osd.3 (osd.3) 220 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:12:04.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:03.306163+0000 mon.a (mon.0) 2710 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:12:04.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:03.306246+0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e487: 8 total, 8 up, 7 in 2024-02-19T21:12:04.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:03.307878+0000 mon.b (mon.1) 635 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:04.535 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:03.308388+0000 mon.a (mon.0) 2712 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:04.567 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:04.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.917+0000 7f007f1b3700 1 -- 172.21.15.163:0/1903878345 >> v1:172.21.15.163:6789/0 conn(0x7f00780f94a0 legacy=0x7f0078102a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:04.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.917+0000 7f007f1b3700 1 -- 172.21.15.163:0/1903878345 shutdown_connections 2024-02-19T21:12:04.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.917+0000 7f007f1b3700 1 -- 172.21.15.163:0/1903878345 >> 172.21.15.163:0/1903878345 conn(0x7f00780f24a0 msgr2=0x7f00780f48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:04.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.917+0000 7f007f1b3700 1 -- 172.21.15.163:0/1903878345 shutdown_connections 2024-02-19T21:12:04.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.917+0000 7f007f1b3700 1 -- 172.21.15.163:0/1903878345 wait complete. 2024-02-19T21:12:04.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f007f1b3700 1 librados: starting msgr at 2024-02-19T21:12:04.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f007f1b3700 1 librados: starting objecter 2024-02-19T21:12:04.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f007f1b3700 1 Processor -- start 2024-02-19T21:12:04.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f007f1b3700 1 -- start start 2024-02-19T21:12:04.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f007f1b3700 1 librados: setting wanted keys 2024-02-19T21:12:04.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f007f1b3700 1 librados: calling monclient init 2024-02-19T21:12:04.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f007f1b3700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0078102ad0 con 0x7f00781042b0 2024-02-19T21:12:04.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f007f1b3700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0078102ca0 con 0x7f00780f89b0 2024-02-19T21:12:04.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f007f1b3700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0078102e90 con 0x7f00780f94a0 2024-02-19T21:12:04.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f007d750700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f00781042b0 0x7f0078118420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:58336/0 (socket says 172.21.15.163:58336) 2024-02-19T21:12:04.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f0077fff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f00780f94a0 0x7f0078114cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:56690/0 (socket says 172.21.15.163:56690) 2024-02-19T21:12:04.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f007d750700 1 -- 172.21.15.163:0/2157435889 learned_addr learned my addr 172.21.15.163:0/2157435889 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:04.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f0075ffb700 10 client.?.objecter ms_handle_connect 0x7f00781042b0 2024-02-19T21:12:04.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f0075ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:04.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f0075ffb700 10 client.?.objecter ms_handle_connect 0x7f00780f94a0 2024-02-19T21:12:04.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f0075ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:04.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.918+0000 7f0075ffb700 10 client.?.objecter ms_handle_connect 0x7f00780f89b0 2024-02-19T21:12:04.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:04.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234138144 0 0) 0x7f0078102e90 con 0x7f00780f94a0 2024-02-19T21:12:04.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00680033f0 con 0x7f00780f94a0 2024-02-19T21:12:04.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061967532 0 0) 0x7f0078102ad0 con 0x7f00781042b0 2024-02-19T21:12:04.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0078102e90 con 0x7f00781042b0 2024-02-19T21:12:04.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237874333 0 0) 0x7f0078102ca0 con 0x7f00780f89b0 2024-02-19T21:12:04.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0078102ad0 con 0x7f00780f89b0 2024-02-19T21:12:04.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1434805182 0 0) 0x7f00680033f0 con 0x7f00780f94a0 2024-02-19T21:12:04.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0078102ca0 con 0x7f00780f94a0 2024-02-19T21:12:04.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0064002c60 con 0x7f00780f94a0 2024-02-19T21:12:04.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4119536596 0 0) 0x7f0078102e90 con 0x7f00781042b0 2024-02-19T21:12:04.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00680033f0 con 0x7f00781042b0 2024-02-19T21:12:04.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1858149166 0 0) 0x7f0078102ad0 con 0x7f00780f89b0 2024-02-19T21:12:04.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0078102e90 con 0x7f00780f89b0 2024-02-19T21:12:04.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0070001d50 con 0x7f00781042b0 2024-02-19T21:12:04.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f006c001800 con 0x7f00780f89b0 2024-02-19T21:12:04.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2254641927 0 0) 0x7f0078102ca0 con 0x7f00780f94a0 2024-02-19T21:12:04.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 >> v1:172.21.15.193:6789/0 conn(0x7f00780f89b0 legacy=0x7f0078102330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:04.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 >> v1:172.21.15.163:6789/0 conn(0x7f00781042b0 legacy=0x7f0078118420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:04.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007811cb70 con 0x7f00780f94a0 2024-02-19T21:12:04.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0078119ba0 con 0x7f00780f94a0 2024-02-19T21:12:04.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f0064001b60 con 0x7f00780f94a0 2024-02-19T21:12:04.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0064003790 con 0x7f00780f94a0 2024-02-19T21:12:04.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f007f1b3700 10 client.50036.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:04.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f007811a010 con 0x7f00780f94a0 2024-02-19T21:12:04.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.919+0000 7f007f1b3700 1 librados: init done 2024-02-19T21:12:04.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.920+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0078060350 con 0x7f00780f94a0 2024-02-19T21:12:04.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.920+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f00640039d0 con 0x7f00780f94a0 2024-02-19T21:12:04.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(488..488 src has 43..488) v4 ==== 8095+0+0 (unknown 683823731 0 0) 0x7f0064078fe0 con 0x7f00780f94a0 2024-02-19T21:12:04.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 osd_map(488..488 src has 43..488) v4 2024-02-19T21:12:04.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f0075ffb700 3 client.50036.objecter handle_osd_map got epochs [488,488] > 0 2024-02-19T21:12:04.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f0075ffb700 3 client.50036.objecter handle_osd_map decoding full epoch 488 2024-02-19T21:12:04.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f0075ffb700 20 client.50036.objecter dump_active .. 0 homeless 2024-02-19T21:12:04.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v488) v1 ==== 54+0+16 (unknown 1811153722 0 2420000192) 0x7f006407b3d0 con 0x7f00780f94a0 2024-02-19T21:12:04.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f0075ffb700 10 client.50036.objecter ms_handle_connect 0x7f0068063e00 2024-02-19T21:12:04.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f007f1b3700 10 client.50036.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:04.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 1 2024-02-19T21:12:04.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078047540 osd=0 3 2024-02-19T21:12:04.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078047540 osd=0 3 2024-02-19T21:12:04.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078047540 osd=0 3 2024-02-19T21:12:04.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 1 2024-02-19T21:12:04.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f0078047540 osd=0 3 2024-02-19T21:12:04.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 0 1 2024-02-19T21:12:04.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078047540 osd=0 4 2024-02-19T21:12:04.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f007f1b3700 10 client.50036.objecter _send_command 1 2024-02-19T21:12:04.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.921+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f007811c9e0 con 0x7f00780598f0 2024-02-19T21:12:04.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.922+0000 7f0075ffb700 10 client.50036.objecter ms_handle_connect 0x7f00780598f0 2024-02-19T21:12:04.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.922+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f007811c9e0 con 0x7f00780598f0 2024-02-19T21:12:04.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.922+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:12:04.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.922+0000 7f0075ffb700 10 client.50036.objecter _finish_command 1 = system:0 2024-02-19T21:12:04.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.922+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f0078047540 osd=0 4 2024-02-19T21:12:04.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.922+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 0 1 2024-02-19T21:12:04.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.938+0000 7f007f1b3700 10 client.50036.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:04.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.938+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 2 2024-02-19T21:12:04.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.938+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078047540 osd=0 3 2024-02-19T21:12:04.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.938+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078047540 osd=0 3 2024-02-19T21:12:04.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.938+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078047540 osd=0 3 2024-02-19T21:12:04.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.938+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 2 2024-02-19T21:12:04.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.938+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f0078047540 osd=0 3 2024-02-19T21:12:04.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.938+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 0 2 2024-02-19T21:12:04.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.938+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078047540 osd=0 4 2024-02-19T21:12:04.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.938+0000 7f007f1b3700 10 client.50036.objecter _send_command 2 2024-02-19T21:12:04.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.938+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0078109eb0 con 0x7f00780598f0 2024-02-19T21:12:04.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.939+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0078109eb0 con 0x7f00780598f0 2024-02-19T21:12:04.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.939+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:04.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.939+0000 7f0075ffb700 10 client.50036.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:04.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.939+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f0078047540 osd=0 4 2024-02-19T21:12:04.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.939+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 0 2 2024-02-19T21:12:04.942 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:12:04.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.940+0000 7f007f1b3700 10 client.50036.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:04.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.940+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 3 2024-02-19T21:12:04.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.940+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078059cd0 osd=1 3 2024-02-19T21:12:04.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.940+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078059cd0 osd=1 3 2024-02-19T21:12:04.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.940+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078059cd0 osd=1 3 2024-02-19T21:12:04.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.940+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 3 2024-02-19T21:12:04.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.940+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f0078059cd0 osd=1 3 2024-02-19T21:12:04.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.940+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 1 3 2024-02-19T21:12:04.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.940+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078059cd0 osd=1 4 2024-02-19T21:12:04.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.940+0000 7f007f1b3700 10 client.50036.objecter _send_command 3 2024-02-19T21:12:04.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.940+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0078120940 con 0x7f0078059e40 2024-02-19T21:12:04.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.940+0000 7f0075ffb700 10 client.50036.objecter ms_handle_connect 0x7f0078059e40 2024-02-19T21:12:04.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.941+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0078120940 con 0x7f0078059e40 2024-02-19T21:12:04.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.941+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:12:04.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.941+0000 7f0075ffb700 10 client.50036.objecter _finish_command 3 = system:0 2024-02-19T21:12:04.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.941+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f0078059cd0 osd=1 4 2024-02-19T21:12:04.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.941+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 1 3 2024-02-19T21:12:04.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.966+0000 7f007f1b3700 10 client.50036.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:04.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.966+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 4 2024-02-19T21:12:04.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.966+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078059cd0 osd=1 3 2024-02-19T21:12:04.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.966+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078059cd0 osd=1 3 2024-02-19T21:12:04.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.966+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078059cd0 osd=1 3 2024-02-19T21:12:04.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.966+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 4 2024-02-19T21:12:04.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.966+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f0078059cd0 osd=1 3 2024-02-19T21:12:04.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.966+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 1 4 2024-02-19T21:12:04.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.966+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078059cd0 osd=1 4 2024-02-19T21:12:04.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.966+0000 7f007f1b3700 10 client.50036.objecter _send_command 4 2024-02-19T21:12:04.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.966+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0078059360 con 0x7f0078059e40 2024-02-19T21:12:04.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.967+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0078059360 con 0x7f0078059e40 2024-02-19T21:12:04.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.967+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:04.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.967+0000 7f0075ffb700 10 client.50036.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:04.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.967+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f0078059cd0 osd=1 4 2024-02-19T21:12:04.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.967+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 1 4 2024-02-19T21:12:04.966 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:12:04.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.968+0000 7f007f1b3700 10 client.50036.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:04.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.968+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 5 2024-02-19T21:12:04.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.968+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078059770 osd=2 3 2024-02-19T21:12:04.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.968+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078059770 osd=2 3 2024-02-19T21:12:04.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.968+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078059770 osd=2 3 2024-02-19T21:12:04.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.968+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 5 2024-02-19T21:12:04.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.968+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f0078059770 osd=2 3 2024-02-19T21:12:04.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.968+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 2 5 2024-02-19T21:12:04.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.968+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078059770 osd=2 4 2024-02-19T21:12:04.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.968+0000 7f007f1b3700 10 client.50036.objecter _send_command 5 2024-02-19T21:12:04.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.968+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0078125360 con 0x7f00780fa610 2024-02-19T21:12:04.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.968+0000 7f0075ffb700 10 client.50036.objecter ms_handle_connect 0x7f00780fa610 2024-02-19T21:12:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.969+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0078125360 con 0x7f00780fa610 2024-02-19T21:12:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.969+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:12:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.969+0000 7f0075ffb700 10 client.50036.objecter _finish_command 5 = system:0 2024-02-19T21:12:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.969+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f0078059770 osd=2 4 2024-02-19T21:12:04.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.969+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 2 5 2024-02-19T21:12:04.980 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:04.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.985+0000 7f007f1b3700 10 client.50036.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:04.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.985+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 6 2024-02-19T21:12:04.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.985+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078059770 osd=2 3 2024-02-19T21:12:04.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.985+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078059770 osd=2 3 2024-02-19T21:12:04.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.985+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078059770 osd=2 3 2024-02-19T21:12:04.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.985+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 6 2024-02-19T21:12:04.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.985+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f0078059770 osd=2 3 2024-02-19T21:12:04.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.985+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 2 6 2024-02-19T21:12:04.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.985+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078059770 osd=2 4 2024-02-19T21:12:04.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.985+0000 7f007f1b3700 10 client.50036.objecter _send_command 6 2024-02-19T21:12:04.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.985+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f00780f9b70 con 0x7f00780fa610 2024-02-19T21:12:04.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.986+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f00780f9b70 con 0x7f00780fa610 2024-02-19T21:12:04.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:04.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f0075ffb700 10 client.50036.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:04.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f0078059770 osd=2 4 2024-02-19T21:12:04.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 2 6 2024-02-19T21:12:04.986 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:12:04.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f007f1b3700 10 client.50036.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:04.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 7 2024-02-19T21:12:04.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f00780f9f80 osd=3 3 2024-02-19T21:12:04.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f00780f9f80 osd=3 3 2024-02-19T21:12:04.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f00780f9f80 osd=3 3 2024-02-19T21:12:04.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 7 2024-02-19T21:12:04.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f00780f9f80 osd=3 3 2024-02-19T21:12:04.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 3 7 2024-02-19T21:12:04.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f00780f9f80 osd=3 4 2024-02-19T21:12:04.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f007f1b3700 10 client.50036.objecter _send_command 7 2024-02-19T21:12:04.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0078129e10 con 0x7f0078121790 2024-02-19T21:12:04.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.987+0000 7f0075ffb700 10 client.50036.objecter ms_handle_connect 0x7f0078121790 2024-02-19T21:12:04.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.988+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0078129e10 con 0x7f0078121790 2024-02-19T21:12:04.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.988+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:12:04.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.988+0000 7f0075ffb700 10 client.50036.objecter _finish_command 7 = system:0 2024-02-19T21:12:04.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.988+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f00780f9f80 osd=3 4 2024-02-19T21:12:04.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:04.988+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 3 7 2024-02-19T21:12:05.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.004+0000 7f007f1b3700 10 client.50036.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:05.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.004+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 8 2024-02-19T21:12:05.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.004+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f00780f9f80 osd=3 3 2024-02-19T21:12:05.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.004+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f00780f9f80 osd=3 3 2024-02-19T21:12:05.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.004+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f00780f9f80 osd=3 3 2024-02-19T21:12:05.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.004+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 8 2024-02-19T21:12:05.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.004+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f00780f9f80 osd=3 3 2024-02-19T21:12:05.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.004+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 3 8 2024-02-19T21:12:05.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.004+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f00780f9f80 osd=3 4 2024-02-19T21:12:05.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.004+0000 7f007f1b3700 10 client.50036.objecter _send_command 8 2024-02-19T21:12:05.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.004+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0078120cf0 con 0x7f0078121790 2024-02-19T21:12:05.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0078120cf0 con 0x7f0078121790 2024-02-19T21:12:05.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:05.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f0075ffb700 10 client.50036.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:05.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f00780f9f80 osd=3 4 2024-02-19T21:12:05.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 3 8 2024-02-19T21:12:05.005 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:12:05.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f007f1b3700 10 client.50036.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 9 2024-02-19T21:12:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078120ab0 osd=4 3 2024-02-19T21:12:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078120ab0 osd=4 3 2024-02-19T21:12:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078120ab0 osd=4 3 2024-02-19T21:12:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 9 2024-02-19T21:12:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f0078120ab0 osd=4 3 2024-02-19T21:12:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 4 9 2024-02-19T21:12:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078120ab0 osd=4 4 2024-02-19T21:12:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f007f1b3700 10 client.50036.objecter _send_command 9 2024-02-19T21:12:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.006+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0078120c60 con 0x7f00781261b0 2024-02-19T21:12:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.007+0000 7f0075ffb700 10 client.50036.objecter ms_handle_connect 0x7f00781261b0 2024-02-19T21:12:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.007+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0078120c60 con 0x7f00781261b0 2024-02-19T21:12:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.007+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:12:05.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.007+0000 7f0075ffb700 10 client.50036.objecter _finish_command 9 = system:0 2024-02-19T21:12:05.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.007+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f0078120ab0 osd=4 4 2024-02-19T21:12:05.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.007+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 4 9 2024-02-19T21:12:05.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.023+0000 7f007f1b3700 10 client.50036.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:05.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.023+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 10 2024-02-19T21:12:05.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.023+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078120ab0 osd=4 3 2024-02-19T21:12:05.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.023+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078120ab0 osd=4 3 2024-02-19T21:12:05.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.023+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078120ab0 osd=4 3 2024-02-19T21:12:05.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.023+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 10 2024-02-19T21:12:05.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.023+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f0078120ab0 osd=4 3 2024-02-19T21:12:05.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.023+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 4 10 2024-02-19T21:12:05.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.023+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078120ab0 osd=4 4 2024-02-19T21:12:05.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.023+0000 7f007f1b3700 10 client.50036.objecter _send_command 10 2024-02-19T21:12:05.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.023+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0078120c60 con 0x7f00781261b0 2024-02-19T21:12:05.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0078120c60 con 0x7f00781261b0 2024-02-19T21:12:05.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:05.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f0075ffb700 10 client.50036.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:05.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f0078120ab0 osd=4 4 2024-02-19T21:12:05.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 4 10 2024-02-19T21:12:05.025 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:12:05.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f007f1b3700 10 client.50036.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:05.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 11 2024-02-19T21:12:05.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078120c60 osd=5 3 2024-02-19T21:12:05.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078120c60 osd=5 3 2024-02-19T21:12:05.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078120c60 osd=5 3 2024-02-19T21:12:05.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 11 2024-02-19T21:12:05.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f0078120c60 osd=5 3 2024-02-19T21:12:05.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 5 11 2024-02-19T21:12:05.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078120c60 osd=5 4 2024-02-19T21:12:05.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f007f1b3700 10 client.50036.objecter _send_command 11 2024-02-19T21:12:05.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.025+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f007812c390 con 0x7f007812bda0 2024-02-19T21:12:05.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.026+0000 7f0075ffb700 10 client.50036.objecter ms_handle_connect 0x7f007812bda0 2024-02-19T21:12:05.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.027+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f007812c390 con 0x7f007812bda0 2024-02-19T21:12:05.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.027+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:12:05.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.027+0000 7f0075ffb700 10 client.50036.objecter _finish_command 11 = system:0 2024-02-19T21:12:05.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.027+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f0078120c60 osd=5 4 2024-02-19T21:12:05.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.027+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 5 11 2024-02-19T21:12:05.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.041+0000 7f007f1b3700 10 client.50036.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:05.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.041+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 12 2024-02-19T21:12:05.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.041+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078120c60 osd=5 3 2024-02-19T21:12:05.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.041+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078120c60 osd=5 3 2024-02-19T21:12:05.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.041+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078120c60 osd=5 3 2024-02-19T21:12:05.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.041+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 12 2024-02-19T21:12:05.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.041+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f0078120c60 osd=5 3 2024-02-19T21:12:05.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.041+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 5 12 2024-02-19T21:12:05.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.041+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078120c60 osd=5 4 2024-02-19T21:12:05.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.041+0000 7f007f1b3700 10 client.50036.objecter _send_command 12 2024-02-19T21:12:05.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.041+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f007812c5b0 con 0x7f007812bda0 2024-02-19T21:12:05.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f007812c5b0 con 0x7f007812bda0 2024-02-19T21:12:05.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:05.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f0075ffb700 10 client.50036.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:05.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f0078120c60 osd=5 4 2024-02-19T21:12:05.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 5 12 2024-02-19T21:12:05.043 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:12:05.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f007f1b3700 10 client.50036.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:05.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 13 2024-02-19T21:12:05.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f007812c5b0 osd=6 3 2024-02-19T21:12:05.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f007812c5b0 osd=6 3 2024-02-19T21:12:05.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f007812c5b0 osd=6 3 2024-02-19T21:12:05.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 13 2024-02-19T21:12:05.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f007812c5b0 osd=6 3 2024-02-19T21:12:05.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 6 13 2024-02-19T21:12:05.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f007812c5b0 osd=6 4 2024-02-19T21:12:05.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f007f1b3700 10 client.50036.objecter _send_command 13 2024-02-19T21:12:05.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.043+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0078136fc0 con 0x7f0078133410 2024-02-19T21:12:05.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.044+0000 7f0075ffb700 10 client.50036.objecter ms_handle_connect 0x7f0078133410 2024-02-19T21:12:05.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.045+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0078136fc0 con 0x7f0078133410 2024-02-19T21:12:05.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.045+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:12:05.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.045+0000 7f0075ffb700 10 client.50036.objecter _finish_command 13 = system:0 2024-02-19T21:12:05.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.045+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f007812c5b0 osd=6 4 2024-02-19T21:12:05.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.045+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 6 13 2024-02-19T21:12:05.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.059+0000 7f007f1b3700 10 client.50036.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:05.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.059+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 14 2024-02-19T21:12:05.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.059+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f007812c5b0 osd=6 3 2024-02-19T21:12:05.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.059+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f007812c5b0 osd=6 3 2024-02-19T21:12:05.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.059+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f007812c5b0 osd=6 3 2024-02-19T21:12:05.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.059+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 14 2024-02-19T21:12:05.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.059+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f007812c5b0 osd=6 3 2024-02-19T21:12:05.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.059+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 6 14 2024-02-19T21:12:05.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.059+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f007812c5b0 osd=6 4 2024-02-19T21:12:05.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.059+0000 7f007f1b3700 10 client.50036.objecter _send_command 14 2024-02-19T21:12:05.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.059+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f0078136fc0 con 0x7f0078133410 2024-02-19T21:12:05.063 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:12:05.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f0078136fc0 con 0x7f0078133410 2024-02-19T21:12:05.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:05.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f0075ffb700 10 client.50036.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:05.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f007812c5b0 osd=6 4 2024-02-19T21:12:05.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 6 14 2024-02-19T21:12:05.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f007f1b3700 10 client.50036.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:05.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 15 2024-02-19T21:12:05.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078136fc0 osd=7 3 2024-02-19T21:12:05.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078136fc0 osd=7 3 2024-02-19T21:12:05.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078136fc0 osd=7 3 2024-02-19T21:12:05.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 15 2024-02-19T21:12:05.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f0078136fc0 osd=7 3 2024-02-19T21:12:05.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 7 15 2024-02-19T21:12:05.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078136fc0 osd=7 4 2024-02-19T21:12:05.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f007f1b3700 10 client.50036.objecter _send_command 15 2024-02-19T21:12:05.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.061+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00781381e0 con 0x7f0078138c80 2024-02-19T21:12:05.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.062+0000 7f0075ffb700 10 client.50036.objecter ms_handle_connect 0x7f0078138c80 2024-02-19T21:12:05.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.062+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f00781381e0 con 0x7f0078138c80 2024-02-19T21:12:05.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.062+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:12:05.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.062+0000 7f0075ffb700 10 client.50036.objecter _finish_command 15 = system:0 2024-02-19T21:12:05.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.062+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f0078136fc0 osd=7 4 2024-02-19T21:12:05.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.062+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 7 15 2024-02-19T21:12:05.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.077+0000 7f007f1b3700 10 client.50036.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:05.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.077+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign -1 16 2024-02-19T21:12:05.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.077+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078136fc0 osd=7 3 2024-02-19T21:12:05.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.077+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078136fc0 osd=7 3 2024-02-19T21:12:05.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.077+0000 7f007f1b3700 20 client.50036.objecter _get_session s=0x7f0078136fc0 osd=7 3 2024-02-19T21:12:05.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.077+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_remove -1 16 2024-02-19T21:12:05.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.077+0000 7f007f1b3700 20 client.50036.objecter get_session s=0x7f0078136fc0 osd=7 3 2024-02-19T21:12:05.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.077+0000 7f007f1b3700 15 client.50036.objecter _session_command_op_assign 7 16 2024-02-19T21:12:05.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.077+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078136fc0 osd=7 4 2024-02-19T21:12:05.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.077+0000 7f007f1b3700 10 client.50036.objecter _send_command 16 2024-02-19T21:12:05.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.077+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f00781381e0 con 0x7f0078138c80 2024-02-19T21:12:05.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f0075ffb700 1 -- 172.21.15.163:0/2157435889 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f00781381e0 con 0x7f0078138c80 2024-02-19T21:12:05.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f0075ffb700 10 client.50036.objecter ms_dispatch 0x7f0078001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:05.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f0075ffb700 10 client.50036.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:05.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f0075ffb700 20 client.50036.objecter put_session s=0x7f0078136fc0 osd=7 4 2024-02-19T21:12:05.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f0075ffb700 15 client.50036.objecter _session_command_op_remove 7 16 2024-02-19T21:12:05.082 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:12:05.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 10 librados: watch_flush enter 2024-02-19T21:12:05.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 10 librados: watch_flush exit 2024-02-19T21:12:05.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 10 client.50036.objecter close_session for osd.0 2024-02-19T21:12:05.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 >> v1:172.21.15.163:6801/1559418115 conn(0x7f00780598f0 legacy=0x7f007811c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:05.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078047540 osd=0 1 2024-02-19T21:12:05.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 10 client.50036.objecter close_session for osd.1 2024-02-19T21:12:05.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 >> v1:172.21.15.163:6805/3125047432 conn(0x7f0078059e40 legacy=0x7f007811f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:05.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078059cd0 osd=1 1 2024-02-19T21:12:05.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 10 client.50036.objecter close_session for osd.2 2024-02-19T21:12:05.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 >> v1:172.21.15.163:6809/1563158487 conn(0x7f00780fa610 legacy=0x7f0078123c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:05.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078059770 osd=2 1 2024-02-19T21:12:05.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 10 client.50036.objecter close_session for osd.3 2024-02-19T21:12:05.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 >> v1:172.21.15.163:6813/2112279769 conn(0x7f0078121790 legacy=0x7f0078128660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:05.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f00780f9f80 osd=3 1 2024-02-19T21:12:05.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 10 client.50036.objecter close_session for osd.4 2024-02-19T21:12:05.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 >> v1:172.21.15.193:6800/3129716965 conn(0x7f00781261b0 legacy=0x7f0078129fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:05.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007d750700 1 -- 172.21.15.163:0/2157435889 reap_dead start 2024-02-19T21:12:05.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007d750700 1 -- 172.21.15.163:0/2157435889 reap_dead start 2024-02-19T21:12:05.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078120ab0 osd=4 1 2024-02-19T21:12:05.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 10 client.50036.objecter close_session for osd.5 2024-02-19T21:12:05.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 >> v1:172.21.15.193:6804/805161991 conn(0x7f007812bda0 legacy=0x7f00781317e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:05.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078120c60 osd=5 1 2024-02-19T21:12:05.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 10 client.50036.objecter close_session for osd.6 2024-02-19T21:12:05.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 >> v1:172.21.15.193:6808/787604615 conn(0x7f0078133410 legacy=0x7f0078135800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:05.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f007812c5b0 osd=6 1 2024-02-19T21:12:05.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 10 client.50036.objecter close_session for osd.7 2024-02-19T21:12:05.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 >> v1:172.21.15.193:6812/157927432 conn(0x7f0078138c80 legacy=0x7f00781390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:05.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 20 client.50036.objecter put_session s=0x7f0078136fc0 osd=7 1 2024-02-19T21:12:05.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 20 client.50036.objecter shutdown clearing up homeless session... 2024-02-19T21:12:05.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.079+0000 7f007f1b3700 10 client.50036.objecter successfully canceled tick 2024-02-19T21:12:05.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.080+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 >> v1:172.21.15.163:6800/1552514681 conn(0x7f0068063e00 legacy=0x7f00680662c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:05.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.080+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 >> v1:172.21.15.163:6790/0 conn(0x7f00780f94a0 legacy=0x7f0078114cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:05.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.080+0000 7f007d750700 1 -- 172.21.15.163:0/2157435889 reap_dead start 2024-02-19T21:12:05.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.080+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 shutdown_connections 2024-02-19T21:12:05.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.080+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 >> 172.21.15.163:0/2157435889 conn(0x7f00780f24a0 msgr2=0x7f0078107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:05.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.080+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 shutdown_connections 2024-02-19T21:12:05.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.080+0000 7f007f1b3700 1 -- 172.21.15.163:0/2157435889 wait complete. 2024-02-19T21:12:05.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:05.080+0000 7f007f1b3700 1 librados: shutdown 2024-02-19T21:12:05.090 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:12:05.090 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:12:05.090 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:12:05.090 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:12:05.090 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:12:05.091 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:12:05.091 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:12:05.091 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:12:05.202 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-19T21:12:05.219 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:05.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:05.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:05.322 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:12:05.348 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:05.374 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:12:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:05.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:12:05.435 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:03.887923+0000 osd.3 (osd.3) 221 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:05.436 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:03.888117+0000 osd.3 (osd.3) 222 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:05.436 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:04.301149+0000 mon.a (mon.0) 2713 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:12:05.436 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:04.301232+0000 mon.a (mon.0) 2714 : cluster [DBG] osdmap e488: 8 total, 8 up, 7 in 2024-02-19T21:12:05.436 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:04.302430+0000 mon.b (mon.1) 636 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:05.436 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:04.302860+0000 mon.a (mon.0) 2715 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:05.436 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:04.921342+0000 mon.c (mon.2) 402 : audit [DBG] from='client.? v1:172.21.15.163:0/2157435889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:05.436 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:03.887923+0000 osd.3 (osd.3) 221 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:05.437 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:03.888117+0000 osd.3 (osd.3) 222 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:05.437 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:04.301149+0000 mon.a (mon.0) 2713 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:12:05.437 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:04.301232+0000 mon.a (mon.0) 2714 : cluster [DBG] osdmap e488: 8 total, 8 up, 7 in 2024-02-19T21:12:05.437 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:04.302430+0000 mon.b (mon.1) 636 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:05.437 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:04.302860+0000 mon.a (mon.0) 2715 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:05.437 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:04.921342+0000 mon.c (mon.2) 402 : audit [DBG] from='client.? v1:172.21.15.163:0/2157435889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:05.477 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:05.499 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:12:05.506 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:05.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:03.887923+0000 osd.3 (osd.3) 221 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:05.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:03.888117+0000 osd.3 (osd.3) 222 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:05.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:04.301149+0000 mon.a (mon.0) 2713 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:12:05.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:04.301232+0000 mon.a (mon.0) 2714 : cluster [DBG] osdmap e488: 8 total, 8 up, 7 in 2024-02-19T21:12:05.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:04.302430+0000 mon.b (mon.1) 636 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:05.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:04.302860+0000 mon.a (mon.0) 2715 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:05.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:04.921342+0000 mon.c (mon.2) 402 : audit [DBG] from='client.? v1:172.21.15.163:0/2157435889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:06.404 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:12:06.666 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:04.544941+0000 osd.7 (osd.7) 228 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:12:06.666 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:04.547951+0000 osd.7 (osd.7) 229 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:12:06.666 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:05.246226+0000 mgr.y (mgr.14550) 449 : cluster [DBG] pgmap v732: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 191 B/s, 32 objects/s recovering 2024-02-19T21:12:06.666 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:05.303657+0000 mon.a (mon.0) 2716 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-19T21:12:06.666 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:05.303720+0000 mon.a (mon.0) 2717 : cluster [DBG] osdmap e489: 8 total, 8 up, 7 in 2024-02-19T21:12:06.666 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:05.303831+0000 mon.a (mon.0) 2718 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 95/1095 objects degraded (8.676%), 8 pgs degraded) 2024-02-19T21:12:06.666 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:05.307701+0000 mon.b (mon.1) 637 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:06.666 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:05.312555+0000 mon.a (mon.0) 2719 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:06.667 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:05.444389+0000 osd.4 (osd.4) 179 : cluster [DBG] 6.17 scrub starts 2024-02-19T21:12:06.667 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:05.445566+0000 osd.4 (osd.4) 180 : cluster [DBG] 6.17 scrub ok 2024-02-19T21:12:06.667 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:05.662847+0000 osd.6 (osd.6) 205 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:12:06.667 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:05.664418+0000 osd.6 (osd.6) 206 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:12:06.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:04.544941+0000 osd.7 (osd.7) 228 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:12:06.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:04.547951+0000 osd.7 (osd.7) 229 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:12:06.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:05.246226+0000 mgr.y (mgr.14550) 449 : cluster [DBG] pgmap v732: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 191 B/s, 32 objects/s recovering 2024-02-19T21:12:06.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:05.303657+0000 mon.a (mon.0) 2716 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-19T21:12:06.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:05.303720+0000 mon.a (mon.0) 2717 : cluster [DBG] osdmap e489: 8 total, 8 up, 7 in 2024-02-19T21:12:06.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:05.303831+0000 mon.a (mon.0) 2718 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 95/1095 objects degraded (8.676%), 8 pgs degraded) 2024-02-19T21:12:06.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:05.307701+0000 mon.b (mon.1) 637 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:06.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:05.312555+0000 mon.a (mon.0) 2719 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:06.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:05.444389+0000 osd.4 (osd.4) 179 : cluster [DBG] 6.17 scrub starts 2024-02-19T21:12:06.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:05.445566+0000 osd.4 (osd.4) 180 : cluster [DBG] 6.17 scrub ok 2024-02-19T21:12:06.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:05.662847+0000 osd.6 (osd.6) 205 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:12:06.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:05.664418+0000 osd.6 (osd.6) 206 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:12:06.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:04.544941+0000 osd.7 (osd.7) 228 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:12:06.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:04.547951+0000 osd.7 (osd.7) 229 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:12:06.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:05.246226+0000 mgr.y (mgr.14550) 449 : cluster [DBG] pgmap v732: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 191 B/s, 32 objects/s recovering 2024-02-19T21:12:06.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:05.303657+0000 mon.a (mon.0) 2716 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-19T21:12:06.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:05.303720+0000 mon.a (mon.0) 2717 : cluster [DBG] osdmap e489: 8 total, 8 up, 7 in 2024-02-19T21:12:06.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:05.303831+0000 mon.a (mon.0) 2718 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 95/1095 objects degraded (8.676%), 8 pgs degraded) 2024-02-19T21:12:06.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:05.307701+0000 mon.b (mon.1) 637 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:06.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:05.312555+0000 mon.a (mon.0) 2719 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:06.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:05.444389+0000 osd.4 (osd.4) 179 : cluster [DBG] 6.17 scrub starts 2024-02-19T21:12:06.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:05.445566+0000 osd.4 (osd.4) 180 : cluster [DBG] 6.17 scrub ok 2024-02-19T21:12:06.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:05.662847+0000 osd.6 (osd.6) 205 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:12:06.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:05.664418+0000 osd.6 (osd.6) 206 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:12:06.756 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:06.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:06.860 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:06.891 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:06.912 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:06.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:07.016 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:07.047 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:07.068 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:07.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:07.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:07.158 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:12:07.158 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:12:07.172 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:07.200 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:07.333 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:05.525899+0000 osd.7 (osd.7) 230 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:12:07.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:05.530034+0000 osd.7 (osd.7) 231 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:12:07.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:06.015932+0000 osd.1 (osd.1) 235 : cluster [DBG] 2.d scrub starts 2024-02-19T21:12:07.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:06.016813+0000 osd.1 (osd.1) 236 : cluster [DBG] 2.d scrub ok 2024-02-19T21:12:07.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:06.095812+0000 osd.5 (osd.5) 208 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:12:07.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:06.098699+0000 osd.5 (osd.5) 209 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:12:07.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:06.260878+0000 osd.2 (osd.2) 196 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:12:07.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:06.261666+0000 osd.2 (osd.2) 197 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:12:07.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:06.311973+0000 mon.a (mon.0) 2720 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:12:07.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:06.312046+0000 mon.a (mon.0) 2721 : cluster [DBG] osdmap e490: 8 total, 8 up, 7 in 2024-02-19T21:12:07.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:06.338358+0000 mon.b (mon.1) 638 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-19T21:12:07.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:06.338852+0000 mon.a (mon.0) 2722 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-19T21:12:07.334 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:06.410063+0000 osd.4 (osd.4) 181 : cluster [DBG] 6.2 scrub starts 2024-02-19T21:12:07.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:06.410843+0000 osd.4 (osd.4) 182 : cluster [DBG] 6.2 scrub ok 2024-02-19T21:12:07.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:06.668506+0000 osd.6 (osd.6) 207 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:12:07.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:06.669332+0000 osd.6 (osd.6) 208 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:12:07.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:07.258419+0000 osd.3 (osd.3) 223 : cluster [DBG] 16.4 scrub starts 2024-02-19T21:12:07.335 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:07.259287+0000 osd.3 (osd.3) 224 : cluster [DBG] 16.4 scrub ok 2024-02-19T21:12:07.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.354+0000 7f7fe9f03700 1 -- 172.21.15.163:0/1586197032 >> v1:172.21.15.163:6790/0 conn(0x7f7fe41031b0 legacy=0x7f7fe4105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:07.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.354+0000 7f7fe9f03700 1 -- 172.21.15.163:0/1586197032 shutdown_connections 2024-02-19T21:12:07.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.354+0000 7f7fe9f03700 1 -- 172.21.15.163:0/1586197032 >> 172.21.15.163:0/1586197032 conn(0x7f7fe40f2470 msgr2=0x7f7fe40f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:07.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.354+0000 7f7fe9f03700 1 -- 172.21.15.163:0/1586197032 shutdown_connections 2024-02-19T21:12:07.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.354+0000 7f7fe9f03700 1 -- 172.21.15.163:0/1586197032 wait complete. 2024-02-19T21:12:07.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.354+0000 7f7fe9f03700 1 librados: starting msgr at 2024-02-19T21:12:07.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.354+0000 7f7fe9f03700 1 librados: starting objecter 2024-02-19T21:12:07.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe9f03700 1 Processor -- start 2024-02-19T21:12:07.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe9f03700 1 -- start start 2024-02-19T21:12:07.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe9f03700 1 librados: setting wanted keys 2024-02-19T21:12:07.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe9f03700 1 librados: calling monclient init 2024-02-19T21:12:07.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe9f03700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7fe4108e00 con 0x7f7fe41031b0 2024-02-19T21:12:07.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe9f03700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7fe4108ff0 con 0x7f7fe40ff670 2024-02-19T21:12:07.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe9f03700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7fe411b760 con 0x7f7fe40fc870 2024-02-19T21:12:07.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe37fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f7fe40fc870 0x7f7fe4107130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:56710/0 (socket says 172.21.15.163:56710) 2024-02-19T21:12:07.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe3fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f7fe41031b0 0x7f7fe4117fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:58346/0 (socket says 172.21.15.163:58346) 2024-02-19T21:12:07.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe37fe700 1 -- 172.21.15.163:0/2348286668 learned_addr learned my addr 172.21.15.163:0/2348286668 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:07.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe0ff9700 10 client.?.objecter ms_handle_connect 0x7f7fe40fc870 2024-02-19T21:12:07.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:07.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe0ff9700 10 client.?.objecter ms_handle_connect 0x7f7fe41031b0 2024-02-19T21:12:07.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:07.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe0ff9700 10 client.?.objecter ms_handle_connect 0x7f7fe40ff670 2024-02-19T21:12:07.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe0ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:07.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505047569 0 0) 0x7f7fe4108e00 con 0x7f7fe41031b0 2024-02-19T21:12:07.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fcc002220 con 0x7f7fe41031b0 2024-02-19T21:12:07.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867014090 0 0) 0x7f7fe411b760 con 0x7f7fe40fc870 2024-02-19T21:12:07.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe4108e00 con 0x7f7fe40fc870 2024-02-19T21:12:07.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.355+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576282556 0 0) 0x7f7fe4108ff0 con 0x7f7fe40ff670 2024-02-19T21:12:07.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe411b760 con 0x7f7fe40ff670 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2577623764 0 0) 0x7f7fe411b760 con 0x7f7fe40ff670 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fe4108ff0 con 0x7f7fe40ff670 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7fd8001650 con 0x7f7fe40ff670 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 573005025 0 0) 0x7f7fcc002220 con 0x7f7fe41031b0 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fe411b760 con 0x7f7fe41031b0 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2036456426 0 0) 0x7f7fe4108e00 con 0x7f7fe40fc870 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fcc002220 con 0x7f7fe40fc870 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7fdc001c50 con 0x7f7fe41031b0 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7fd4001780 con 0x7f7fe40fc870 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2661765775 0 0) 0x7f7fe4108ff0 con 0x7f7fe40ff670 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 >> v1:172.21.15.163:6790/0 conn(0x7f7fe40fc870 legacy=0x7f7fe4107130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 >> v1:172.21.15.163:6789/0 conn(0x7f7fe41031b0 legacy=0x7f7fe4117fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe411c960 con 0x7f7fe40ff670 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe9f03700 1 -- 172.21.15.163:0/2348286668 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe411b950 con 0x7f7fe40ff670 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe9f03700 10 client.48853.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:07.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe9f03700 1 -- 172.21.15.163:0/2348286668 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe411bdc0 con 0x7f7fe40ff670 2024-02-19T21:12:07.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe9f03700 1 librados: init done 2024-02-19T21:12:07.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f7fd8002080 con 0x7f7fe40ff670 2024-02-19T21:12:07.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.356+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7fd8003100 con 0x7f7fe40ff670 2024-02-19T21:12:07.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.357+0000 7f7fe9f03700 1 -- 172.21.15.163:0/2348286668 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe405f700 con 0x7f7fe40ff670 2024-02-19T21:12:07.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.357+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f7fd8001be0 con 0x7f7fe40ff670 2024-02-19T21:12:07.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.358+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(491..491 src has 43..491) v4 ==== 8130+0+0 (unknown 629500436 0 0) 0x7f7fd8076300 con 0x7f7fe40ff670 2024-02-19T21:12:07.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.358+0000 7f7fe0ff9700 10 client.48853.objecter ms_dispatch 0x7f7fe4001550 osd_map(491..491 src has 43..491) v4 2024-02-19T21:12:07.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.358+0000 7f7fe0ff9700 3 client.48853.objecter handle_osd_map got epochs [491,491] > 0 2024-02-19T21:12:07.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.358+0000 7f7fe0ff9700 3 client.48853.objecter handle_osd_map decoding full epoch 491 2024-02-19T21:12:07.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.358+0000 7f7fe0ff9700 20 client.48853.objecter dump_active .. 0 homeless 2024-02-19T21:12:07.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.358+0000 7f7fe0ff9700 10 client.48853.objecter ms_handle_connect 0x7f7fcc063f90 2024-02-19T21:12:07.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.362+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f7fd80496b0 con 0x7f7fe40ff670 2024-02-19T21:12:07.396 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:07.400 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:07.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:07.505 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:07.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:05.525899+0000 osd.7 (osd.7) 230 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:12:07.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:05.530034+0000 osd.7 (osd.7) 231 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:12:07.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:06.015932+0000 osd.1 (osd.1) 235 : cluster [DBG] 2.d scrub starts 2024-02-19T21:12:07.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:06.016813+0000 osd.1 (osd.1) 236 : cluster [DBG] 2.d scrub ok 2024-02-19T21:12:07.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:06.095812+0000 osd.5 (osd.5) 208 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:12:07.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:06.098699+0000 osd.5 (osd.5) 209 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:12:07.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:06.260878+0000 osd.2 (osd.2) 196 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:12:07.531 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:06.261666+0000 osd.2 (osd.2) 197 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:12:07.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:06.311973+0000 mon.a (mon.0) 2720 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:12:07.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:06.312046+0000 mon.a (mon.0) 2721 : cluster [DBG] osdmap e490: 8 total, 8 up, 7 in 2024-02-19T21:12:07.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:06.338358+0000 mon.b (mon.1) 638 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-19T21:12:07.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:06.338852+0000 mon.a (mon.0) 2722 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-19T21:12:07.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:06.410063+0000 osd.4 (osd.4) 181 : cluster [DBG] 6.2 scrub starts 2024-02-19T21:12:07.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:06.410843+0000 osd.4 (osd.4) 182 : cluster [DBG] 6.2 scrub ok 2024-02-19T21:12:07.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:06.668506+0000 osd.6 (osd.6) 207 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:12:07.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:06.669332+0000 osd.6 (osd.6) 208 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:12:07.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:07.258419+0000 osd.3 (osd.3) 223 : cluster [DBG] 16.4 scrub starts 2024-02-19T21:12:07.532 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:07.259287+0000 osd.3 (osd.3) 224 : cluster [DBG] 16.4 scrub ok 2024-02-19T21:12:07.534 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:07.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:07.550+0000 7f7fe9f03700 1 -- 172.21.15.163:0/2348286668 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd out", "ids": ["6"]} v 0) v1 -- 0x7f7fe4047540 con 0x7f7fe40ff670 2024-02-19T21:12:07.561 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:07.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:07.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:07.625 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:05.525899+0000 osd.7 (osd.7) 230 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:12:07.625 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:05.530034+0000 osd.7 (osd.7) 231 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:12:07.625 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:06.015932+0000 osd.1 (osd.1) 235 : cluster [DBG] 2.d scrub starts 2024-02-19T21:12:07.625 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:06.016813+0000 osd.1 (osd.1) 236 : cluster [DBG] 2.d scrub ok 2024-02-19T21:12:07.626 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:06.095812+0000 osd.5 (osd.5) 208 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:12:07.626 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:06.098699+0000 osd.5 (osd.5) 209 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:12:07.626 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:06.260878+0000 osd.2 (osd.2) 196 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:12:07.626 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:06.261666+0000 osd.2 (osd.2) 197 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:12:07.626 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:06.311973+0000 mon.a (mon.0) 2720 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:12:07.626 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:06.312046+0000 mon.a (mon.0) 2721 : cluster [DBG] osdmap e490: 8 total, 8 up, 7 in 2024-02-19T21:12:07.626 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:06.338358+0000 mon.b (mon.1) 638 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-19T21:12:07.626 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:06.338852+0000 mon.a (mon.0) 2722 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-02-19T21:12:07.626 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:06.410063+0000 osd.4 (osd.4) 181 : cluster [DBG] 6.2 scrub starts 2024-02-19T21:12:07.626 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:06.410843+0000 osd.4 (osd.4) 182 : cluster [DBG] 6.2 scrub ok 2024-02-19T21:12:07.626 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:06.668506+0000 osd.6 (osd.6) 207 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:12:07.626 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:06.669332+0000 osd.6 (osd.6) 208 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:12:07.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:07.258419+0000 osd.3 (osd.3) 223 : cluster [DBG] 16.4 scrub starts 2024-02-19T21:12:07.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:07.259287+0000 osd.3 (osd.3) 224 : cluster [DBG] 16.4 scrub ok 2024-02-19T21:12:07.627 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:12:07.551+0000 7fb606608700 -1 mon.a@0(leader).osd e491 definitely_dead 0 2024-02-19T21:12:07.673 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:07.700 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:07.819 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:12:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:12:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:12:07.898 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:07.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:07.907 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-19T21:12:08.002 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:08.028 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:08.047 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:08.050 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:08.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:08.151 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:08.177 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:08.305 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:12:08.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:08.357+0000 7f7fe0ff9700 1 -- 172.21.15.163:0/2348286668 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd out", "ids": ["6"]}]=0 marked out osd.6. v492) v1 ==== 87+0+0 (unknown 505218075 0 0) 0x7f7fd8047b20 con 0x7f7fe40ff670 2024-02-19T21:12:08.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:08.366+0000 7f7fe9f03700 10 librados: watch_flush enter 2024-02-19T21:12:08.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:08.366+0000 7f7fe9f03700 10 librados: watch_flush exit 2024-02-19T21:12:08.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:08.366+0000 7f7fe9f03700 20 client.48853.objecter shutdown clearing up homeless session... 2024-02-19T21:12:08.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:08.366+0000 7f7fe9f03700 10 client.48853.objecter successfully canceled tick 2024-02-19T21:12:08.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:08.366+0000 7f7fe9f03700 1 -- 172.21.15.163:0/2348286668 >> v1:172.21.15.163:6800/1552514681 conn(0x7f7fcc063f90 legacy=0x7f7fcc066450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:08.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:08.366+0000 7f7fe9f03700 1 -- 172.21.15.163:0/2348286668 >> v1:172.21.15.193:6789/0 conn(0x7f7fe40ff670 legacy=0x7f7fe4107840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:08.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:08.368+0000 7f7fe9f03700 1 -- 172.21.15.163:0/2348286668 shutdown_connections 2024-02-19T21:12:08.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:08.368+0000 7f7fe9f03700 1 -- 172.21.15.163:0/2348286668 >> 172.21.15.163:0/2348286668 conn(0x7f7fe40f2470 msgr2=0x7f7fe40f4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:08.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:08.368+0000 7f7fe9f03700 1 -- 172.21.15.163:0/2348286668 shutdown_connections 2024-02-19T21:12:08.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:08.368+0000 7f7fe9f03700 1 -- 172.21.15.163:0/2348286668 wait complete. 2024-02-19T21:12:08.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:08.368+0000 7f7fe9f03700 1 librados: shutdown 2024-02-19T21:12:08.367 INFO:teuthology.orchestra.run.smithi163.stderr:marked out osd.6. 2024-02-19T21:12:08.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:06.990037+0000 osd.1 (osd.1) 237 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:12:08.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:06.991032+0000 osd.1 (osd.1) 238 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:12:08.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:07.246681+0000 mgr.y (mgr.14550) 450 : cluster [DBG] pgmap v735: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 191 B/s, 32 objects/s recovering 2024-02-19T21:12:08.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:07.283755+0000 osd.2 (osd.2) 198 : cluster [DBG] 61.6 scrub starts 2024-02-19T21:12:08.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:07.285227+0000 osd.2 (osd.2) 199 : cluster [DBG] 61.6 scrub ok 2024-02-19T21:12:08.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:07.333918+0000 mon.a (mon.0) 2723 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-02-19T21:12:08.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:07.333982+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e491: 8 total, 8 up, 7 in 2024-02-19T21:12:08.620 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:07.352799+0000 mon.b (mon.1) 639 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-19T21:12:08.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:07.354622+0000 mon.a (mon.0) 2725 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-19T21:12:08.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:07.415351+0000 osd.4 (osd.4) 183 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:12:08.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:07.416146+0000 osd.4 (osd.4) 184 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:12:08.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:07.551382+0000 mon.b (mon.1) 640 : audit [INF] from='client.? v1:172.21.15.163:0/2348286668' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-19T21:12:08.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:07.551732+0000 mon.a (mon.0) 2726 : audit [INF] from='client.48853 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-19T21:12:08.621 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:07.551809+0000 mon.a (mon.0) 2727 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2024-02-19T21:12:08.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:06.990037+0000 osd.1 (osd.1) 237 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:12:08.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:06.991032+0000 osd.1 (osd.1) 238 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:12:08.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:07.246681+0000 mgr.y (mgr.14550) 450 : cluster [DBG] pgmap v735: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 191 B/s, 32 objects/s recovering 2024-02-19T21:12:08.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:07.283755+0000 osd.2 (osd.2) 198 : cluster [DBG] 61.6 scrub starts 2024-02-19T21:12:08.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:07.285227+0000 osd.2 (osd.2) 199 : cluster [DBG] 61.6 scrub ok 2024-02-19T21:12:08.621 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:07.333918+0000 mon.a (mon.0) 2723 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-02-19T21:12:08.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:07.333982+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e491: 8 total, 8 up, 7 in 2024-02-19T21:12:08.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:07.352799+0000 mon.b (mon.1) 639 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-19T21:12:08.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:07.354622+0000 mon.a (mon.0) 2725 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-19T21:12:08.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:07.415351+0000 osd.4 (osd.4) 183 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:12:08.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:07.416146+0000 osd.4 (osd.4) 184 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:12:08.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:07.551382+0000 mon.b (mon.1) 640 : audit [INF] from='client.? v1:172.21.15.163:0/2348286668' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-19T21:12:08.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:07.551732+0000 mon.a (mon.0) 2726 : audit [INF] from='client.48853 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-19T21:12:08.622 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:07.551809+0000 mon.a (mon.0) 2727 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2024-02-19T21:12:08.625 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:08.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:08.728 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:08.754 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:06.990037+0000 osd.1 (osd.1) 237 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:12:08.755 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:06.991032+0000 osd.1 (osd.1) 238 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:12:08.755 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:07.246681+0000 mgr.y (mgr.14550) 450 : cluster [DBG] pgmap v735: 238 pgs: 238 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 191 B/s, 32 objects/s recovering 2024-02-19T21:12:08.755 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:07.283755+0000 osd.2 (osd.2) 198 : cluster [DBG] 61.6 scrub starts 2024-02-19T21:12:08.755 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:07.285227+0000 osd.2 (osd.2) 199 : cluster [DBG] 61.6 scrub ok 2024-02-19T21:12:08.755 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:07.333918+0000 mon.a (mon.0) 2723 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-02-19T21:12:08.755 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:07.333982+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e491: 8 total, 8 up, 7 in 2024-02-19T21:12:08.755 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:07.352799+0000 mon.b (mon.1) 639 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-19T21:12:08.755 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:07.354622+0000 mon.a (mon.0) 2725 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-02-19T21:12:08.755 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:07.415351+0000 osd.4 (osd.4) 183 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:12:08.755 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:07.416146+0000 osd.4 (osd.4) 184 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:12:08.755 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:07.551382+0000 mon.b (mon.1) 640 : audit [INF] from='client.? v1:172.21.15.163:0/2348286668' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-19T21:12:08.755 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:07.551732+0000 mon.a (mon.0) 2726 : audit [INF] from='client.48853 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-19T21:12:08.756 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:07.551809+0000 mon.a (mon.0) 2727 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2024-02-19T21:12:08.756 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:08.774 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:08.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:08.878 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:08.905 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:09.561 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:09.564 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:09.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:09.569 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:09.665 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:12:09.691 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:07.996664+0000 osd.1 (osd.1) 239 : cluster [DBG] 61.5 scrub starts 2024-02-19T21:12:09.691 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:07.998846+0000 osd.1 (osd.1) 240 : cluster [DBG] 61.5 scrub ok 2024-02-19T21:12:09.691 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:08.315512+0000 osd.2 (osd.2) 200 : cluster [DBG] 61.14 scrub starts 2024-02-19T21:12:09.691 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:08.317270+0000 osd.2 (osd.2) 201 : cluster [DBG] 61.14 scrub ok 2024-02-19T21:12:09.691 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:08.347015+0000 mon.a (mon.0) 2728 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-02-19T21:12:09.692 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:08.347093+0000 mon.a (mon.0) 2729 : audit [INF] from='client.48853 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-02-19T21:12:09.692 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:08.347147+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e492: 8 total, 8 up, 6 in 2024-02-19T21:12:09.692 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:08.430070+0000 osd.4 (osd.4) 185 : cluster [DBG] 16.2 scrub starts 2024-02-19T21:12:09.692 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:08.430871+0000 osd.4 (osd.4) 186 : cluster [DBG] 16.2 scrub ok 2024-02-19T21:12:09.692 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:09.302904+0000 mon.a (mon.0) 2731 : cluster [DBG] osdmap e493: 8 total, 8 up, 6 in 2024-02-19T21:12:09.693 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:09.711 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:12:09.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:09.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:12:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:07.996664+0000 osd.1 (osd.1) 239 : cluster [DBG] 61.5 scrub starts 2024-02-19T21:12:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:07.998846+0000 osd.1 (osd.1) 240 : cluster [DBG] 61.5 scrub ok 2024-02-19T21:12:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:08.315512+0000 osd.2 (osd.2) 200 : cluster [DBG] 61.14 scrub starts 2024-02-19T21:12:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:08.317270+0000 osd.2 (osd.2) 201 : cluster [DBG] 61.14 scrub ok 2024-02-19T21:12:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:08.347015+0000 mon.a (mon.0) 2728 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-02-19T21:12:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:08.347093+0000 mon.a (mon.0) 2729 : audit [INF] from='client.48853 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-02-19T21:12:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:08.347147+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e492: 8 total, 8 up, 6 in 2024-02-19T21:12:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:08.430070+0000 osd.4 (osd.4) 185 : cluster [DBG] 16.2 scrub starts 2024-02-19T21:12:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:08.430871+0000 osd.4 (osd.4) 186 : cluster [DBG] 16.2 scrub ok 2024-02-19T21:12:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:09.302904+0000 mon.a (mon.0) 2731 : cluster [DBG] osdmap e493: 8 total, 8 up, 6 in 2024-02-19T21:12:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:07.996664+0000 osd.1 (osd.1) 239 : cluster [DBG] 61.5 scrub starts 2024-02-19T21:12:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:07.998846+0000 osd.1 (osd.1) 240 : cluster [DBG] 61.5 scrub ok 2024-02-19T21:12:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:08.315512+0000 osd.2 (osd.2) 200 : cluster [DBG] 61.14 scrub starts 2024-02-19T21:12:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:08.317270+0000 osd.2 (osd.2) 201 : cluster [DBG] 61.14 scrub ok 2024-02-19T21:12:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:08.347015+0000 mon.a (mon.0) 2728 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-02-19T21:12:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:08.347093+0000 mon.a (mon.0) 2729 : audit [INF] from='client.48853 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-02-19T21:12:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:08.347147+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e492: 8 total, 8 up, 6 in 2024-02-19T21:12:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:08.430070+0000 osd.4 (osd.4) 185 : cluster [DBG] 16.2 scrub starts 2024-02-19T21:12:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:08.430871+0000 osd.4 (osd.4) 186 : cluster [DBG] 16.2 scrub ok 2024-02-19T21:12:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:09.302904+0000 mon.a (mon.0) 2731 : cluster [DBG] osdmap e493: 8 total, 8 up, 6 in 2024-02-19T21:12:09.816 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:09.842 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:09.967 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:09.969 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:09.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:10.070 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:10.097 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:10.119 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:10.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:10.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:10.223 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:10.263 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:10.412 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:09.026417+0000 osd.1 (osd.1) 241 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:12:10.412 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:09.030210+0000 osd.1 (osd.1) 242 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:12:10.412 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:09.051447+0000 osd.5 (osd.5) 210 : cluster [DBG] 61.c scrub starts 2024-02-19T21:12:10.415 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:09.026417+0000 osd.1 (osd.1) 241 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:12:10.415 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:09.030210+0000 osd.1 (osd.1) 242 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:12:10.415 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:09.051447+0000 osd.5 (osd.5) 210 : cluster [DBG] 61.c scrub starts 2024-02-19T21:12:10.415 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:09.054913+0000 osd.5 (osd.5) 211 : cluster [DBG] 61.c scrub ok 2024-02-19T21:12:10.415 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:09.247070+0000 mgr.y (mgr.14550) 451 : cluster [DBG] pgmap v738: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:12:10.415 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:09.325766+0000 osd.2 (osd.2) 202 : cluster [DBG] 2.c scrub starts 2024-02-19T21:12:10.415 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:09.339140+0000 osd.2 (osd.2) 203 : cluster [DBG] 2.c scrub ok 2024-02-19T21:12:10.415 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:09.428261+0000 mon.b (mon.1) 641 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:10.415 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:09.428659+0000 mon.a (mon.0) 2732 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:10.416 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:10.304414+0000 mon.a (mon.0) 2733 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:12:10.416 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:10.304509+0000 mon.a (mon.0) 2734 : cluster [DBG] osdmap e494: 8 total, 8 up, 6 in 2024-02-19T21:12:10.416 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:10.306461+0000 osd.7 (osd.7) 232 : cluster [DBG] 2.20 starting backfill to osd.1 from (0'0,0'0] MAX to 397'2 2024-02-19T21:12:10.416 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:10.368721+0000 mon.b (mon.1) 642 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:10.416 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:10.369521+0000 mon.a (mon.0) 2735 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:10.416 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:10.370124+0000 mon.b (mon.1) 643 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-77"}]: dispatch 2024-02-19T21:12:10.416 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:10.370326+0000 mon.a (mon.0) 2736 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-77"}]: dispatch 2024-02-19T21:12:10.416 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:09.026417+0000 osd.1 (osd.1) 241 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:12:10.416 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:09.030210+0000 osd.1 (osd.1) 242 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:12:10.416 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:09.051447+0000 osd.5 (osd.5) 210 : cluster [DBG] 61.c scrub starts 2024-02-19T21:12:10.416 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:09.054913+0000 osd.5 (osd.5) 211 : cluster [DBG] 61.c scrub ok 2024-02-19T21:12:10.416 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:09.247070+0000 mgr.y (mgr.14550) 451 : cluster [DBG] pgmap v738: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:12:10.417 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:09.325766+0000 osd.2 (osd.2) 202 : cluster [DBG] 2.c scrub starts 2024-02-19T21:12:10.417 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:09.339140+0000 osd.2 (osd.2) 203 : cluster [DBG] 2.c scrub ok 2024-02-19T21:12:10.417 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:09.428261+0000 mon.b (mon.1) 641 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:10.417 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:09.428659+0000 mon.a (mon.0) 2732 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:10.417 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:10.304414+0000 mon.a (mon.0) 2733 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:12:10.417 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:10.304509+0000 mon.a (mon.0) 2734 : cluster [DBG] osdmap e494: 8 total, 8 up, 6 in 2024-02-19T21:12:10.417 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:10.306461+0000 osd.7 (osd.7) 232 : cluster [DBG] 2.20 starting backfill to osd.1 from (0'0,0'0] MAX to 397'2 2024-02-19T21:12:10.417 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:10.368721+0000 mon.b (mon.1) 642 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:10.417 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:10.369521+0000 mon.a (mon.0) 2735 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:10.417 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:10.370124+0000 mon.b (mon.1) 643 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-77"}]: dispatch 2024-02-19T21:12:10.417 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:10.370326+0000 mon.a (mon.0) 2736 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-77"}]: dispatch 2024-02-19T21:12:10.623 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:10.627 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:10.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:10.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.323+0000 7f61a7e22700 1 -- 172.21.15.163:0/3429697612 >> v1:172.21.15.163:6790/0 conn(0x7f61a010cbd0 legacy=0x7f61a010efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:10.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.323+0000 7f61a7e22700 1 -- 172.21.15.163:0/3429697612 shutdown_connections 2024-02-19T21:12:10.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.323+0000 7f61a7e22700 1 -- 172.21.15.163:0/3429697612 >> 172.21.15.163:0/3429697612 conn(0x7f61a00643b0 msgr2=0x7f61a00647c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:10.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.323+0000 7f61a7e22700 1 -- 172.21.15.163:0/3429697612 shutdown_connections 2024-02-19T21:12:10.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.323+0000 7f61a7e22700 1 -- 172.21.15.163:0/3429697612 wait complete. 2024-02-19T21:12:10.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.324+0000 7f61a7e22700 1 librados: starting msgr at 2024-02-19T21:12:10.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.324+0000 7f61a7e22700 1 librados: starting objecter 2024-02-19T21:12:10.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.324+0000 7f61a7e22700 1 Processor -- start 2024-02-19T21:12:10.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.324+0000 7f61a7e22700 1 -- start start 2024-02-19T21:12:10.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.324+0000 7f61a7e22700 1 librados: setting wanted keys 2024-02-19T21:12:10.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.324+0000 7f61a7e22700 1 librados: calling monclient init 2024-02-19T21:12:10.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.324+0000 7f61a7e22700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f61a011fc80 con 0x7f61a00ffea0 2024-02-19T21:12:10.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.324+0000 7f61a7e22700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f61a0120e80 con 0x7f61a01090c0 2024-02-19T21:12:10.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.324+0000 7f61a7e22700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f61a0122080 con 0x7f61a010cbd0 2024-02-19T21:12:10.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f61a5bbe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f61a00ffea0 0x7f61a0105770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:58368/0 (socket says 172.21.15.163:58368) 2024-02-19T21:12:10.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f61a5bbe700 1 -- 172.21.15.163:0/3808100302 learned_addr learned my addr 172.21.15.163:0/3808100302 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:10.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f6196ffd700 10 client.?.objecter ms_handle_connect 0x7f61a00ffea0 2024-02-19T21:12:10.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f6196ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:10.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579139823 0 0) 0x7f61a011fc80 con 0x7f61a00ffea0 2024-02-19T21:12:10.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f618c003630 con 0x7f61a00ffea0 2024-02-19T21:12:10.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f6196ffd700 10 client.?.objecter ms_handle_connect 0x7f61a010cbd0 2024-02-19T21:12:10.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f6196ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468993987 0 0) 0x7f61a0122080 con 0x7f61a010cbd0 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a011fc80 con 0x7f61a010cbd0 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1120412927 0 0) 0x7f618c003630 con 0x7f61a00ffea0 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61a0122080 con 0x7f61a00ffea0 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6198002d10 con 0x7f61a00ffea0 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3520836074 0 0) 0x7f61a0122080 con 0x7f61a00ffea0 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 >> v1:172.21.15.163:6790/0 conn(0x7f61a010cbd0 legacy=0x7f61a011e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 >> v1:172.21.15.193:6789/0 conn(0x7f61a01090c0 legacy=0x7f61a0105e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.325+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a0123280 con 0x7f61a00ffea0 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.326+0000 7f61a7e22700 1 -- 172.21.15.163:0/3808100302 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a0122270 con 0x7f61a00ffea0 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.326+0000 7f61a7e22700 10 client.23325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.326+0000 7f61a7e22700 1 -- 172.21.15.163:0/3808100302 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a0122760 con 0x7f61a00ffea0 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.326+0000 7f61a7e22700 1 librados: init done 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.326+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6198001c10 con 0x7f61a00ffea0 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.326+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6198005ec0 con 0x7f61a00ffea0 2024-02-19T21:12:10.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.326+0000 7f61a7e22700 1 -- 172.21.15.163:0/3808100302 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6184004f20 con 0x7f61a00ffea0 2024-02-19T21:12:10.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.327+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6198001770 con 0x7f61a00ffea0 2024-02-19T21:12:10.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.328+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(494..494 src has 43..494) v4 ==== 8221+0+0 (unknown 4062292378 0 0) 0x7f61980792e0 con 0x7f61a00ffea0 2024-02-19T21:12:10.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.328+0000 7f6196ffd700 10 client.23325.objecter ms_dispatch 0x7f61a0001550 osd_map(494..494 src has 43..494) v4 2024-02-19T21:12:10.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.328+0000 7f6196ffd700 3 client.23325.objecter handle_osd_map got epochs [494,494] > 0 2024-02-19T21:12:10.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.328+0000 7f6196ffd700 3 client.23325.objecter handle_osd_map decoding full epoch 494 2024-02-19T21:12:10.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.328+0000 7f6196ffd700 20 client.23325.objecter dump_active .. 0 homeless 2024-02-19T21:12:10.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.329+0000 7f6196ffd700 10 client.23325.objecter ms_handle_connect 0x7f618c063810 2024-02-19T21:12:10.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.336+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f619804c630 con 0x7f61a00ffea0 2024-02-19T21:12:10.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:10.532+0000 7f61a7e22700 1 -- 172.21.15.163:0/3808100302 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f61840051d0 con 0x7f61a00ffea0 2024-02-19T21:12:10.727 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:10.728 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:10.755 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:10.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:12:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:09.054913+0000 osd.5 (osd.5) 211 : cluster [DBG] 61.c scrub ok 2024-02-19T21:12:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:09.247070+0000 mgr.y (mgr.14550) 451 : cluster [DBG] pgmap v738: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:12:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:09.325766+0000 osd.2 (osd.2) 202 : cluster [DBG] 2.c scrub starts 2024-02-19T21:12:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:09.339140+0000 osd.2 (osd.2) 203 : cluster [DBG] 2.c scrub ok 2024-02-19T21:12:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:09.428261+0000 mon.b (mon.1) 641 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:09.428659+0000 mon.a (mon.0) 2732 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:10.304414+0000 mon.a (mon.0) 2733 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-77","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:12:10.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:10.304509+0000 mon.a (mon.0) 2734 : cluster [DBG] osdmap e494: 8 total, 8 up, 6 in 2024-02-19T21:12:10.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:10.306461+0000 osd.7 (osd.7) 232 : cluster [DBG] 2.20 starting backfill to osd.1 from (0'0,0'0] MAX to 397'2 2024-02-19T21:12:10.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:10.368721+0000 mon.b (mon.1) 642 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:10.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:10.369521+0000 mon.a (mon.0) 2735 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:10.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:10.370124+0000 mon.b (mon.1) 643 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-77"}]: dispatch 2024-02-19T21:12:10.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:10.370326+0000 mon.a (mon.0) 2736 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-77"}]: dispatch 2024-02-19T21:12:11.011 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:11.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:11.114 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:11.141 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:11.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:11.305+0000 7f6196ffd700 1 -- 172.21.15.163:0/3808100302 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v495) v1 ==== 101+0+0 (unknown 249124128 0 0) 0x7f619804aa40 con 0x7f61a00ffea0 2024-02-19T21:12:11.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:11.311+0000 7f61a7e22700 10 librados: watch_flush enter 2024-02-19T21:12:11.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:11.311+0000 7f61a7e22700 10 librados: watch_flush exit 2024-02-19T21:12:11.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:11.311+0000 7f61a7e22700 20 client.23325.objecter shutdown clearing up homeless session... 2024-02-19T21:12:11.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:11.311+0000 7f61a7e22700 10 client.23325.objecter successfully canceled tick 2024-02-19T21:12:11.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:11.312+0000 7f61a7e22700 1 -- 172.21.15.163:0/3808100302 >> v1:172.21.15.163:6800/1552514681 conn(0x7f618c063810 legacy=0x7f618c065cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:11.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:11.312+0000 7f61a7e22700 1 -- 172.21.15.163:0/3808100302 >> v1:172.21.15.163:6789/0 conn(0x7f61a00ffea0 legacy=0x7f61a0105770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:11.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:11.313+0000 7f61a7e22700 1 -- 172.21.15.163:0/3808100302 shutdown_connections 2024-02-19T21:12:11.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:11.313+0000 7f61a7e22700 1 -- 172.21.15.163:0/3808100302 >> 172.21.15.163:0/3808100302 conn(0x7f61a00643b0 msgr2=0x7f61a0064fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:11.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:11.313+0000 7f61a7e22700 1 -- 172.21.15.163:0/3808100302 shutdown_connections 2024-02-19T21:12:11.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:11.314+0000 7f61a7e22700 1 -- 172.21.15.163:0/3808100302 wait complete. 2024-02-19T21:12:11.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:11.314+0000 7f61a7e22700 1 librados: shutdown 2024-02-19T21:12:11.313 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:12:11.638 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:10.509573+0000 mgr.y (mgr.14550) 452 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:11.638 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:10.533037+0000 mon.a (mon.0) 2737 : audit [INF] from='client.? v1:172.21.15.163:0/3808100302' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:12:11.638 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:11.301558+0000 mon.a (mon.0) 2738 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:12:11.638 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:11.305945+0000 mon.a (mon.0) 2739 : audit [INF] from='client.? v1:172.21.15.163:0/3808100302' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:12:11.638 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:11.306028+0000 mon.a (mon.0) 2740 : cluster [DBG] osdmap e495: 8 total, 8 up, 6 in 2024-02-19T21:12:11.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:10.509573+0000 mgr.y (mgr.14550) 452 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:11.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:10.533037+0000 mon.a (mon.0) 2737 : audit [INF] from='client.? v1:172.21.15.163:0/3808100302' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:12:11.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:11.301558+0000 mon.a (mon.0) 2738 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:12:11.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:11.305945+0000 mon.a (mon.0) 2739 : audit [INF] from='client.? v1:172.21.15.163:0/3808100302' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:12:11.639 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:11.306028+0000 mon.a (mon.0) 2740 : cluster [DBG] osdmap e495: 8 total, 8 up, 6 in 2024-02-19T21:12:11.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:10.509573+0000 mgr.y (mgr.14550) 452 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:11.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:10.533037+0000 mon.a (mon.0) 2737 : audit [INF] from='client.? v1:172.21.15.163:0/3808100302' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:12:11.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:11.301558+0000 mon.a (mon.0) 2738 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:12:11.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:11.305945+0000 mon.a (mon.0) 2739 : audit [INF] from='client.? v1:172.21.15.163:0/3808100302' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:12:11.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:11.306028+0000 mon.a (mon.0) 2740 : cluster [DBG] osdmap e495: 8 total, 8 up, 6 in 2024-02-19T21:12:12.090 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:12.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:12.195 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:12:12.222 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:12.245 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:12:12.248 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:12.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:12:12.349 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:12.375 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:12.393 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:12.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:12.496 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:12.521 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:12.540 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:12.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:12.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:12.644 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:12.669 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:12.688 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:12.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:12.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:11.247595+0000 mgr.y (mgr.14550) 453 : cluster [DBG] pgmap v741: 238 pgs: 1 active+recovery_wait, 7 active+recovering, 9 active+recovery_wait+degraded, 4 remapped+peering, 217 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 6.2 KiB/s rd, 11 KiB/s wr, 28 op/s; 85/1101 objects degraded (7.720%); 28/1101 objects misplaced (2.543%); 2.9 KiB/s, 14 objects/s recovering 2024-02-19T21:12:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:11.558446+0000 mon.a (mon.0) 2741 : cluster [WRN] Health check failed: Degraded data redundancy: 85/1101 objects degraded (7.720%), 9 pgs degraded (PG_DEGRADED) 2024-02-19T21:12:12.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:12.309654+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e496: 8 total, 8 up, 6 in 2024-02-19T21:12:12.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:12.311707+0000 mon.b (mon.1) 644 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:12.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:12.312676+0000 mon.a (mon.0) 2743 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:12.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:11.247595+0000 mgr.y (mgr.14550) 453 : cluster [DBG] pgmap v741: 238 pgs: 1 active+recovery_wait, 7 active+recovering, 9 active+recovery_wait+degraded, 4 remapped+peering, 217 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 6.2 KiB/s rd, 11 KiB/s wr, 28 op/s; 85/1101 objects degraded (7.720%); 28/1101 objects misplaced (2.543%); 2.9 KiB/s, 14 objects/s recovering 2024-02-19T21:12:12.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:11.558446+0000 mon.a (mon.0) 2741 : cluster [WRN] Health check failed: Degraded data redundancy: 85/1101 objects degraded (7.720%), 9 pgs degraded (PG_DEGRADED) 2024-02-19T21:12:12.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:12.309654+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e496: 8 total, 8 up, 6 in 2024-02-19T21:12:12.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:12.311707+0000 mon.b (mon.1) 644 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:12.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:12.312676+0000 mon.a (mon.0) 2743 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:12.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:12.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:11.247595+0000 mgr.y (mgr.14550) 453 : cluster [DBG] pgmap v741: 238 pgs: 1 active+recovery_wait, 7 active+recovering, 9 active+recovery_wait+degraded, 4 remapped+peering, 217 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 6.2 KiB/s rd, 11 KiB/s wr, 28 op/s; 85/1101 objects degraded (7.720%); 28/1101 objects misplaced (2.543%); 2.9 KiB/s, 14 objects/s recovering 2024-02-19T21:12:12.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:11.558446+0000 mon.a (mon.0) 2741 : cluster [WRN] Health check failed: Degraded data redundancy: 85/1101 objects degraded (7.720%), 9 pgs degraded (PG_DEGRADED) 2024-02-19T21:12:12.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:12.309654+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e496: 8 total, 8 up, 6 in 2024-02-19T21:12:12.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:12.311707+0000 mon.b (mon.1) 644 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:12.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:12.312676+0000 mon.a (mon.0) 2743 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:12.821 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:12.900 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:12.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:13.004 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:13.030 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:13.049 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:13.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:13.118 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:12:13.118 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:12:13.153 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:13.181 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:13.746 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:12:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:12:13] "GET /metrics HTTP/1.1" 200 190214 "" "Prometheus/2.33.4" 2024-02-19T21:12:13.936 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:13.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:13.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:14.041 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:14.068 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:14.446 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:14.448 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:14.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:14.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-19T21:12:14.550 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:14.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:13.248016+0000 mgr.y (mgr.14550) 454 : cluster [DBG] pgmap v744: 238 pgs: 32 unknown, 1 active+recovery_wait, 7 active+recovering, 9 active+recovery_wait+degraded, 4 remapped+peering, 185 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 6.2 KiB/s rd, 9.0 KiB/s wr, 27 op/s; 85/1098 objects degraded (7.741%); 28/1098 objects misplaced (2.550%); 877 B/s, 13 objects/s recovering 2024-02-19T21:12:14.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:13.312600+0000 mon.a (mon.0) 2744 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:14.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:13.312748+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e497: 8 total, 8 up, 6 in 2024-02-19T21:12:14.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:13.324705+0000 mon.b (mon.1) 645 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:12:14.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:13.328094+0000 mon.b (mon.1) 646 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:14.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:13.329857+0000 mon.a (mon.0) 2746 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:14.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:14.304384+0000 mon.a (mon.0) 2747 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:12:14.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:14.304470+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e498: 8 total, 8 up, 6 in 2024-02-19T21:12:14.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:14.306342+0000 mon.b (mon.1) 647 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:14.572 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 7, 3, 4] out_osds: [0, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:12:14.572 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 1 1 2024-02-19T21:12:14.574 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:13.248016+0000 mgr.y (mgr.14550) 454 : cluster [DBG] pgmap v744: 238 pgs: 32 unknown, 1 active+recovery_wait, 7 active+recovering, 9 active+recovery_wait+degraded, 4 remapped+peering, 185 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 6.2 KiB/s rd, 9.0 KiB/s wr, 27 op/s; 85/1098 objects degraded (7.741%); 28/1098 objects misplaced (2.550%); 877 B/s, 13 objects/s recovering 2024-02-19T21:12:14.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:13.312600+0000 mon.a (mon.0) 2744 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:14.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:13.312748+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e497: 8 total, 8 up, 6 in 2024-02-19T21:12:14.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:13.324705+0000 mon.b (mon.1) 645 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:12:14.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:13.328094+0000 mon.b (mon.1) 646 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:14.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:13.329857+0000 mon.a (mon.0) 2746 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:14.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:14.304384+0000 mon.a (mon.0) 2747 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:12:14.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:14.304470+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e498: 8 total, 8 up, 6 in 2024-02-19T21:12:14.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:14.306342+0000 mon.b (mon.1) 647 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:14.575 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:13.248016+0000 mgr.y (mgr.14550) 454 : cluster [DBG] pgmap v744: 238 pgs: 32 unknown, 1 active+recovery_wait, 7 active+recovering, 9 active+recovery_wait+degraded, 4 remapped+peering, 185 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 6.2 KiB/s rd, 9.0 KiB/s wr, 27 op/s; 85/1098 objects degraded (7.741%); 28/1098 objects misplaced (2.550%); 877 B/s, 13 objects/s recovering 2024-02-19T21:12:14.575 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:13.312600+0000 mon.a (mon.0) 2744 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:14.576 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:13.312748+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e497: 8 total, 8 up, 6 in 2024-02-19T21:12:14.576 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:13.324705+0000 mon.b (mon.1) 645 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:12:14.576 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:13.328094+0000 mon.b (mon.1) 646 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:14.576 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:13.329857+0000 mon.a (mon.0) 2746 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:14.576 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:14.304384+0000 mon.a (mon.0) 2747 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:12:14.576 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:14.304470+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e498: 8 total, 8 up, 6 in 2024-02-19T21:12:14.576 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:14.306342+0000 mon.b (mon.1) 647 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:14.577 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:14.725 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:14.732 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:14.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:14.833 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:14.861 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:14.899 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:14.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:14.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:14.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:12:15.002 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:15.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:14.999+0000 7f4e9450d700 1 -- 172.21.15.163:0/2430408398 >> v1:172.21.15.163:6790/0 conn(0x7f4e8c1007b0 legacy=0x7f4e8c102ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:15.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:14.999+0000 7f4e9450d700 1 -- 172.21.15.163:0/2430408398 shutdown_connections 2024-02-19T21:12:15.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:14.999+0000 7f4e9450d700 1 -- 172.21.15.163:0/2430408398 >> 172.21.15.163:0/2430408398 conn(0x7f4e8c06cc00 msgr2=0x7f4e8c0f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:15.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:14.999+0000 7f4e9450d700 1 -- 172.21.15.163:0/2430408398 shutdown_connections 2024-02-19T21:12:15.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:14.999+0000 7f4e9450d700 1 -- 172.21.15.163:0/2430408398 wait complete. 2024-02-19T21:12:15.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:14.999+0000 7f4e9450d700 1 librados: starting msgr at 2024-02-19T21:12:15.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:14.999+0000 7f4e9450d700 1 librados: starting objecter 2024-02-19T21:12:15.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:14.999+0000 7f4e9450d700 1 Processor -- start 2024-02-19T21:12:15.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:14.999+0000 7f4e9450d700 1 -- start start 2024-02-19T21:12:15.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:14.999+0000 7f4e9450d700 1 librados: setting wanted keys 2024-02-19T21:12:15.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:14.999+0000 7f4e9450d700 1 librados: calling monclient init 2024-02-19T21:12:15.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e9450d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4e8c0f8b60 con 0x7f4e8c1007b0 2024-02-19T21:12:15.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e9450d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4e8c0f8cf0 con 0x7f4e8c0fd960 2024-02-19T21:12:15.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e9450d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4e8c0f8e80 con 0x7f4e8c1042c0 2024-02-19T21:12:15.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e91aa8700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f4e8c1007b0 0x7f4e8c114d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:56506/0 (socket says 172.21.15.163:56506) 2024-02-19T21:12:15.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e92aaa700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f4e8c1042c0 0x7f4e8c118450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45916/0 (socket says 172.21.15.163:45916) 2024-02-19T21:12:15.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e91aa8700 1 -- 172.21.15.163:0/2844655645 learned_addr learned my addr 172.21.15.163:0/2844655645 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:15.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 10 client.?.objecter ms_handle_connect 0x7f4e8c1042c0 2024-02-19T21:12:15.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:15.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 10 client.?.objecter ms_handle_connect 0x7f4e8c1007b0 2024-02-19T21:12:15.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:15.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 10 client.?.objecter ms_handle_connect 0x7f4e8c0fd960 2024-02-19T21:12:15.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:15.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103008748 0 0) 0x7f4e8c0f8e80 con 0x7f4e8c1042c0 2024-02-19T21:12:15.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e780033f0 con 0x7f4e8c1042c0 2024-02-19T21:12:15.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321638091 0 0) 0x7f4e8c0f8cf0 con 0x7f4e8c0fd960 2024-02-19T21:12:15.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e8c0f8e80 con 0x7f4e8c0fd960 2024-02-19T21:12:15.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788273824 0 0) 0x7f4e8c0f8b60 con 0x7f4e8c1007b0 2024-02-19T21:12:15.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e8c0f8cf0 con 0x7f4e8c1007b0 2024-02-19T21:12:15.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2964154776 0 0) 0x7f4e8c0f8e80 con 0x7f4e8c0fd960 2024-02-19T21:12:15.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e8c0f8b60 con 0x7f4e8c0fd960 2024-02-19T21:12:15.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4e880018c0 con 0x7f4e8c0fd960 2024-02-19T21:12:15.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2721525190 0 0) 0x7f4e8c0f8cf0 con 0x7f4e8c1007b0 2024-02-19T21:12:15.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e8c0f8e80 con 0x7f4e8c1007b0 2024-02-19T21:12:15.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2464319260 0 0) 0x7f4e780033f0 con 0x7f4e8c1042c0 2024-02-19T21:12:15.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e8c0f8cf0 con 0x7f4e8c1042c0 2024-02-19T21:12:15.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4e7c001f10 con 0x7f4e8c1007b0 2024-02-19T21:12:15.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.000+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4e84001d00 con 0x7f4e8c1042c0 2024-02-19T21:12:15.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.001+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2366740039 0 0) 0x7f4e8c0f8b60 con 0x7f4e8c0fd960 2024-02-19T21:12:15.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.001+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 >> v1:172.21.15.163:6790/0 conn(0x7f4e8c1042c0 legacy=0x7f4e8c118450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:15.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.001+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 >> v1:172.21.15.163:6789/0 conn(0x7f4e8c1007b0 legacy=0x7f4e8c114d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:15.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.001+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c11cba0 con 0x7f4e8c0fd960 2024-02-19T21:12:15.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.001+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8c119bd0 con 0x7f4e8c0fd960 2024-02-19T21:12:15.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.001+0000 7f4e9450d700 10 client.48862.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:15.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.001+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8c11a0c0 con 0x7f4e8c0fd960 2024-02-19T21:12:15.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.001+0000 7f4e9450d700 1 librados: init done 2024-02-19T21:12:15.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.001+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f4e88001f80 con 0x7f4e8c0fd960 2024-02-19T21:12:15.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.001+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4e88003270 con 0x7f4e8c0fd960 2024-02-19T21:12:15.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.002+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4e70004f00 con 0x7f4e8c0fd960 2024-02-19T21:12:15.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.002+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f4e88001ae0 con 0x7f4e8c0fd960 2024-02-19T21:12:15.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.002+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(498..498 src has 43..498) v4 ==== 8112+0+0 (unknown 3552407552 0 0) 0x7f4e88076540 con 0x7f4e8c0fd960 2024-02-19T21:12:15.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.002+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 osd_map(498..498 src has 43..498) v4 2024-02-19T21:12:15.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.002+0000 7f4e837fe700 3 client.48862.objecter handle_osd_map got epochs [498,498] > 0 2024-02-19T21:12:15.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.002+0000 7f4e837fe700 3 client.48862.objecter handle_osd_map decoding full epoch 498 2024-02-19T21:12:15.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.003+0000 7f4e837fe700 20 client.48862.objecter dump_active .. 0 homeless 2024-02-19T21:12:15.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.003+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v498) v1 ==== 54+0+16 (unknown 2274996656 0 2420000192) 0x7f4e88078930 con 0x7f4e8c0fd960 2024-02-19T21:12:15.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.003+0000 7f4e837fe700 10 client.48862.objecter ms_handle_connect 0x7f4e78063eb0 2024-02-19T21:12:15.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.003+0000 7f4e9450d700 10 client.48862.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:15.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.003+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 1 2024-02-19T21:12:15.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.003+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e700051a0 osd=0 3 2024-02-19T21:12:15.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.003+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e700051a0 osd=0 3 2024-02-19T21:12:15.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.003+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e700051a0 osd=0 3 2024-02-19T21:12:15.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.003+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 1 2024-02-19T21:12:15.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.003+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e700051a0 osd=0 3 2024-02-19T21:12:15.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.003+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 0 1 2024-02-19T21:12:15.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.003+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e700051a0 osd=0 4 2024-02-19T21:12:15.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.003+0000 7f4e9450d700 10 client.48862.objecter _send_command 1 2024-02-19T21:12:15.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.003+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e70004950 con 0x7f4e70004530 2024-02-19T21:12:15.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.003+0000 7f4e837fe700 10 client.48862.objecter ms_handle_connect 0x7f4e70004530 2024-02-19T21:12:15.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.004+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e70004950 con 0x7f4e70004530 2024-02-19T21:12:15.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.004+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:12:15.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.004+0000 7f4e837fe700 10 client.48862.objecter _finish_command 1 = system:0 2024-02-19T21:12:15.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.004+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e700051a0 osd=0 4 2024-02-19T21:12:15.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.004+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 0 1 2024-02-19T21:12:15.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.034+0000 7f4e9450d700 10 client.48862.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:15.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.034+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 2 2024-02-19T21:12:15.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.034+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e700051a0 osd=0 3 2024-02-19T21:12:15.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.034+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e700051a0 osd=0 3 2024-02-19T21:12:15.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.034+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e700051a0 osd=0 3 2024-02-19T21:12:15.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.034+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 2 2024-02-19T21:12:15.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.034+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e700051a0 osd=0 3 2024-02-19T21:12:15.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.034+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 0 2 2024-02-19T21:12:15.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.034+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e700051a0 osd=0 4 2024-02-19T21:12:15.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.034+0000 7f4e9450d700 10 client.48862.objecter _send_command 2 2024-02-19T21:12:15.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.034+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e70004d30 con 0x7f4e70004530 2024-02-19T21:12:15.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.036+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4e70004d30 con 0x7f4e70004530 2024-02-19T21:12:15.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.036+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:15.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.036+0000 7f4e837fe700 10 client.48862.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.036+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e700051a0 osd=0 4 2024-02-19T21:12:15.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.036+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 0 2 2024-02-19T21:12:15.037 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:12:15.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.037+0000 7f4e9450d700 10 client.48862.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:15.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.037+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 3 2024-02-19T21:12:15.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.037+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70003fe0 osd=1 3 2024-02-19T21:12:15.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.037+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70003fe0 osd=1 3 2024-02-19T21:12:15.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.037+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70003fe0 osd=1 3 2024-02-19T21:12:15.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.037+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 3 2024-02-19T21:12:15.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.037+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e70003fe0 osd=1 3 2024-02-19T21:12:15.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.037+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 1 3 2024-02-19T21:12:15.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.037+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70003fe0 osd=1 4 2024-02-19T21:12:15.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.037+0000 7f4e9450d700 10 client.48862.objecter _send_command 3 2024-02-19T21:12:15.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.037+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e7000c000 con 0x7f4e70004150 2024-02-19T21:12:15.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.037+0000 7f4e837fe700 10 client.48862.objecter ms_handle_connect 0x7f4e70004150 2024-02-19T21:12:15.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.038+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e7000c000 con 0x7f4e70004150 2024-02-19T21:12:15.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.038+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:12:15.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.038+0000 7f4e837fe700 10 client.48862.objecter _finish_command 3 = system:0 2024-02-19T21:12:15.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.038+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e70003fe0 osd=1 4 2024-02-19T21:12:15.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.038+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 1 3 2024-02-19T21:12:15.050 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:15.053 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:15.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.061+0000 7f4e9450d700 10 client.48862.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:15.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.061+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 4 2024-02-19T21:12:15.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.061+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70003fe0 osd=1 3 2024-02-19T21:12:15.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.061+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70003fe0 osd=1 3 2024-02-19T21:12:15.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.061+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70003fe0 osd=1 3 2024-02-19T21:12:15.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.061+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 4 2024-02-19T21:12:15.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.061+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e70003fe0 osd=1 3 2024-02-19T21:12:15.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.061+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 1 4 2024-02-19T21:12:15.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.061+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70003fe0 osd=1 4 2024-02-19T21:12:15.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.061+0000 7f4e9450d700 10 client.48862.objecter _send_command 4 2024-02-19T21:12:15.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.061+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e70001850 con 0x7f4e70004150 2024-02-19T21:12:15.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.063+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4e70001850 con 0x7f4e70004150 2024-02-19T21:12:15.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.063+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:15.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.064+0000 7f4e837fe700 10 client.48862.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.064+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e70003fe0 osd=1 4 2024-02-19T21:12:15.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.064+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 1 4 2024-02-19T21:12:15.063 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:12:15.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.065+0000 7f4e9450d700 10 client.48862.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:15.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.065+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 5 2024-02-19T21:12:15.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.065+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70003630 osd=2 3 2024-02-19T21:12:15.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.065+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70003630 osd=2 3 2024-02-19T21:12:15.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.065+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70003630 osd=2 3 2024-02-19T21:12:15.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.065+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 5 2024-02-19T21:12:15.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.065+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e70003630 osd=2 3 2024-02-19T21:12:15.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.065+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 2 5 2024-02-19T21:12:15.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.065+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70003630 osd=2 4 2024-02-19T21:12:15.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.065+0000 7f4e9450d700 10 client.48862.objecter _send_command 5 2024-02-19T21:12:15.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.065+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e70003e70 con 0x7f4e7000d2f0 2024-02-19T21:12:15.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.065+0000 7f4e837fe700 10 client.48862.objecter ms_handle_connect 0x7f4e7000d2f0 2024-02-19T21:12:15.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.066+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e70003e70 con 0x7f4e7000d2f0 2024-02-19T21:12:15.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.066+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:12:15.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.066+0000 7f4e837fe700 10 client.48862.objecter _finish_command 5 = system:0 2024-02-19T21:12:15.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.066+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e70003630 osd=2 4 2024-02-19T21:12:15.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.066+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 2 5 2024-02-19T21:12:15.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.083+0000 7f4e9450d700 10 client.48862.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:15.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.083+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 6 2024-02-19T21:12:15.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.083+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70003630 osd=2 3 2024-02-19T21:12:15.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.083+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70003630 osd=2 3 2024-02-19T21:12:15.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.083+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70003630 osd=2 3 2024-02-19T21:12:15.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.083+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 6 2024-02-19T21:12:15.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.083+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e70003630 osd=2 3 2024-02-19T21:12:15.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.083+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 2 6 2024-02-19T21:12:15.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.083+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70003630 osd=2 4 2024-02-19T21:12:15.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.083+0000 7f4e9450d700 10 client.48862.objecter _send_command 6 2024-02-19T21:12:15.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.083+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e7000e6e0 con 0x7f4e7000d2f0 2024-02-19T21:12:15.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.086+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4e7000e6e0 con 0x7f4e7000d2f0 2024-02-19T21:12:15.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.086+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:15.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.086+0000 7f4e837fe700 10 client.48862.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.086+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e70003630 osd=2 4 2024-02-19T21:12:15.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.086+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 2 6 2024-02-19T21:12:15.086 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:15.087 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:12:15.087 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.087+0000 7f4e9450d700 10 client.48862.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:15.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.087+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 7 2024-02-19T21:12:15.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.087+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70003440 osd=3 3 2024-02-19T21:12:15.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.087+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70003440 osd=3 3 2024-02-19T21:12:15.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.087+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70003440 osd=3 3 2024-02-19T21:12:15.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.087+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 7 2024-02-19T21:12:15.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.087+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e70003440 osd=3 3 2024-02-19T21:12:15.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.087+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 3 7 2024-02-19T21:12:15.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.087+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70003440 osd=3 4 2024-02-19T21:12:15.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.087+0000 7f4e9450d700 10 client.48862.objecter _send_command 7 2024-02-19T21:12:15.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.087+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e70015b10 con 0x7f4e70011e80 2024-02-19T21:12:15.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.089+0000 7f4e837fe700 10 client.48862.objecter ms_handle_connect 0x7f4e70011e80 2024-02-19T21:12:15.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:15.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:15.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.090+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e70015b10 con 0x7f4e70011e80 2024-02-19T21:12:15.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.090+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:12:15.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.090+0000 7f4e837fe700 10 client.48862.objecter _finish_command 7 = system:0 2024-02-19T21:12:15.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.090+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e70003440 osd=3 4 2024-02-19T21:12:15.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.090+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 3 7 2024-02-19T21:12:15.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.106+0000 7f4e9450d700 10 client.48862.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:15.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.106+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 8 2024-02-19T21:12:15.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.106+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70003440 osd=3 3 2024-02-19T21:12:15.106 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.106+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70003440 osd=3 3 2024-02-19T21:12:15.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.106+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70003440 osd=3 3 2024-02-19T21:12:15.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.106+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 8 2024-02-19T21:12:15.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.106+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e70003440 osd=3 3 2024-02-19T21:12:15.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.106+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 3 8 2024-02-19T21:12:15.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.106+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70003440 osd=3 4 2024-02-19T21:12:15.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.106+0000 7f4e9450d700 10 client.48862.objecter _send_command 8 2024-02-19T21:12:15.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.106+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e70016e30 con 0x7f4e70011e80 2024-02-19T21:12:15.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4e70016e30 con 0x7f4e70011e80 2024-02-19T21:12:15.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:15.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e837fe700 10 client.48862.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e70003440 osd=3 4 2024-02-19T21:12:15.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 3 8 2024-02-19T21:12:15.110 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:12:15.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e9450d700 10 client.48862.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:15.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 9 2024-02-19T21:12:15.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70015cb0 osd=4 3 2024-02-19T21:12:15.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70015cb0 osd=4 3 2024-02-19T21:12:15.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70015cb0 osd=4 3 2024-02-19T21:12:15.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 9 2024-02-19T21:12:15.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e70015cb0 osd=4 3 2024-02-19T21:12:15.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 4 9 2024-02-19T21:12:15.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70015cb0 osd=4 4 2024-02-19T21:12:15.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e9450d700 10 client.48862.objecter _send_command 9 2024-02-19T21:12:15.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.108+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e7001a320 con 0x7f4e70016330 2024-02-19T21:12:15.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.110+0000 7f4e837fe700 10 client.48862.objecter ms_handle_connect 0x7f4e70016330 2024-02-19T21:12:15.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.110+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e7001a320 con 0x7f4e70016330 2024-02-19T21:12:15.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.110+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:12:15.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.110+0000 7f4e837fe700 10 client.48862.objecter _finish_command 9 = system:0 2024-02-19T21:12:15.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.110+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e70015cb0 osd=4 4 2024-02-19T21:12:15.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.110+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 4 9 2024-02-19T21:12:15.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.125+0000 7f4e9450d700 10 client.48862.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:15.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.125+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 10 2024-02-19T21:12:15.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.125+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70015cb0 osd=4 3 2024-02-19T21:12:15.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.125+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70015cb0 osd=4 3 2024-02-19T21:12:15.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.125+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70015cb0 osd=4 3 2024-02-19T21:12:15.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.125+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 10 2024-02-19T21:12:15.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.125+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e70015cb0 osd=4 3 2024-02-19T21:12:15.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.125+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 4 10 2024-02-19T21:12:15.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.125+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70015cb0 osd=4 4 2024-02-19T21:12:15.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.125+0000 7f4e9450d700 10 client.48862.objecter _send_command 10 2024-02-19T21:12:15.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.125+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e7001bad0 con 0x7f4e70016330 2024-02-19T21:12:15.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4e7001bad0 con 0x7f4e70016330 2024-02-19T21:12:15.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:15.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e837fe700 10 client.48862.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e70015cb0 osd=4 4 2024-02-19T21:12:15.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 4 10 2024-02-19T21:12:15.127 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:12:15.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e9450d700 10 client.48862.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:15.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 11 2024-02-19T21:12:15.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e7001bad0 osd=5 3 2024-02-19T21:12:15.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e7001bad0 osd=5 3 2024-02-19T21:12:15.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e7001bad0 osd=5 3 2024-02-19T21:12:15.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 11 2024-02-19T21:12:15.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e7001bad0 osd=5 3 2024-02-19T21:12:15.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 5 11 2024-02-19T21:12:15.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e7001bad0 osd=5 4 2024-02-19T21:12:15.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e9450d700 10 client.48862.objecter _send_command 11 2024-02-19T21:12:15.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.127+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e7001c700 con 0x7f4e7001c150 2024-02-19T21:12:15.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.128+0000 7f4e837fe700 10 client.48862.objecter ms_handle_connect 0x7f4e7001c150 2024-02-19T21:12:15.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.128+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e7001c700 con 0x7f4e7001c150 2024-02-19T21:12:15.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.128+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:12:15.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.128+0000 7f4e837fe700 10 client.48862.objecter _finish_command 11 = system:0 2024-02-19T21:12:15.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.128+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e7001bad0 osd=5 4 2024-02-19T21:12:15.129 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.128+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 5 11 2024-02-19T21:12:15.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.143+0000 7f4e9450d700 10 client.48862.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:15.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.143+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 12 2024-02-19T21:12:15.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.143+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e7001bad0 osd=5 3 2024-02-19T21:12:15.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.143+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e7001bad0 osd=5 3 2024-02-19T21:12:15.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.143+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e7001bad0 osd=5 3 2024-02-19T21:12:15.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.143+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 12 2024-02-19T21:12:15.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.143+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e7001bad0 osd=5 3 2024-02-19T21:12:15.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.143+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 5 12 2024-02-19T21:12:15.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.143+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e7001bad0 osd=5 4 2024-02-19T21:12:15.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.143+0000 7f4e9450d700 10 client.48862.objecter _send_command 12 2024-02-19T21:12:15.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.143+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e7001c700 con 0x7f4e7001c150 2024-02-19T21:12:15.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.144+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4e7001c700 con 0x7f4e7001c150 2024-02-19T21:12:15.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.144+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:15.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.144+0000 7f4e837fe700 10 client.48862.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.144+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e7001bad0 osd=5 4 2024-02-19T21:12:15.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.144+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 5 12 2024-02-19T21:12:15.146 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:12:15.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.145+0000 7f4e9450d700 10 client.48862.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:15.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.145+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 13 2024-02-19T21:12:15.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.145+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e7001c700 osd=6 3 2024-02-19T21:12:15.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.145+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e7001c700 osd=6 3 2024-02-19T21:12:15.146 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.145+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e7001c700 osd=6 3 2024-02-19T21:12:15.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.145+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 13 2024-02-19T21:12:15.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.145+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e7001c700 osd=6 3 2024-02-19T21:12:15.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.145+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 6 13 2024-02-19T21:12:15.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.145+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e7001c700 osd=6 4 2024-02-19T21:12:15.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.145+0000 7f4e9450d700 10 client.48862.objecter _send_command 13 2024-02-19T21:12:15.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.145+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e7001cb00 con 0x7f4e7001f1e0 2024-02-19T21:12:15.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.145+0000 7f4e837fe700 10 client.48862.objecter ms_handle_connect 0x7f4e7001f1e0 2024-02-19T21:12:15.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.146+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e7001cb00 con 0x7f4e7001f1e0 2024-02-19T21:12:15.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.146+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:12:15.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.146+0000 7f4e837fe700 10 client.48862.objecter _finish_command 13 = system:0 2024-02-19T21:12:15.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.146+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e7001c700 osd=6 4 2024-02-19T21:12:15.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.146+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 6 13 2024-02-19T21:12:15.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.160+0000 7f4e9450d700 10 client.48862.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:15.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.160+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 14 2024-02-19T21:12:15.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.160+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e7001c700 osd=6 3 2024-02-19T21:12:15.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.160+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e7001c700 osd=6 3 2024-02-19T21:12:15.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.160+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e7001c700 osd=6 3 2024-02-19T21:12:15.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.160+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 14 2024-02-19T21:12:15.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.160+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e7001c700 osd=6 3 2024-02-19T21:12:15.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.160+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 6 14 2024-02-19T21:12:15.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.160+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e7001c700 osd=6 4 2024-02-19T21:12:15.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.160+0000 7f4e9450d700 10 client.48862.objecter _send_command 14 2024-02-19T21:12:15.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.160+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e7001cb00 con 0x7f4e7001f1e0 2024-02-19T21:12:15.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4e7001cb00 con 0x7f4e7001f1e0 2024-02-19T21:12:15.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:15.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e837fe700 10 client.48862.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e7001c700 osd=6 4 2024-02-19T21:12:15.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 6 14 2024-02-19T21:12:15.162 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:12:15.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e9450d700 10 client.48862.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:15.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 15 2024-02-19T21:12:15.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70024360 osd=7 3 2024-02-19T21:12:15.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70024360 osd=7 3 2024-02-19T21:12:15.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70024360 osd=7 3 2024-02-19T21:12:15.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 15 2024-02-19T21:12:15.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e70024360 osd=7 3 2024-02-19T21:12:15.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 7 15 2024-02-19T21:12:15.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70024360 osd=7 4 2024-02-19T21:12:15.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e9450d700 10 client.48862.objecter _send_command 15 2024-02-19T21:12:15.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.162+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e70024ea0 con 0x7f4e700249e0 2024-02-19T21:12:15.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.163+0000 7f4e837fe700 10 client.48862.objecter ms_handle_connect 0x7f4e700249e0 2024-02-19T21:12:15.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.164+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e70024ea0 con 0x7f4e700249e0 2024-02-19T21:12:15.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.164+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:12:15.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.164+0000 7f4e837fe700 10 client.48862.objecter _finish_command 15 = system:0 2024-02-19T21:12:15.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.164+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e70024360 osd=7 4 2024-02-19T21:12:15.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.164+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 7 15 2024-02-19T21:12:15.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.179+0000 7f4e9450d700 10 client.48862.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:15.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.179+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign -1 16 2024-02-19T21:12:15.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.179+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70024360 osd=7 3 2024-02-19T21:12:15.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.179+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70024360 osd=7 3 2024-02-19T21:12:15.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.179+0000 7f4e9450d700 20 client.48862.objecter _get_session s=0x7f4e70024360 osd=7 3 2024-02-19T21:12:15.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.179+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_remove -1 16 2024-02-19T21:12:15.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.179+0000 7f4e9450d700 20 client.48862.objecter get_session s=0x7f4e70024360 osd=7 3 2024-02-19T21:12:15.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.179+0000 7f4e9450d700 15 client.48862.objecter _session_command_op_assign 7 16 2024-02-19T21:12:15.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.179+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70024360 osd=7 4 2024-02-19T21:12:15.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.179+0000 7f4e9450d700 10 client.48862.objecter _send_command 16 2024-02-19T21:12:15.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.179+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4e70024ea0 con 0x7f4e700249e0 2024-02-19T21:12:15.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e837fe700 1 -- 172.21.15.163:0/2844655645 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4e70024ea0 con 0x7f4e700249e0 2024-02-19T21:12:15.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e837fe700 10 client.48862.objecter ms_dispatch 0x7f4e8c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:15.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e837fe700 10 client.48862.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e837fe700 20 client.48862.objecter put_session s=0x7f4e70024360 osd=7 4 2024-02-19T21:12:15.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e837fe700 15 client.48862.objecter _session_command_op_remove 7 16 2024-02-19T21:12:15.180 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:12:15.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 10 librados: watch_flush enter 2024-02-19T21:12:15.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 10 librados: watch_flush exit 2024-02-19T21:12:15.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 10 client.48862.objecter close_session for osd.0 2024-02-19T21:12:15.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 >> v1:172.21.15.163:6801/1559418115 conn(0x7f4e70004530 legacy=0x7f4e70002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:15.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e700051a0 osd=0 1 2024-02-19T21:12:15.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 10 client.48862.objecter close_session for osd.1 2024-02-19T21:12:15.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 >> v1:172.21.15.163:6805/3125047432 conn(0x7f4e70004150 legacy=0x7f4e7000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:15.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70003fe0 osd=1 1 2024-02-19T21:12:15.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 10 client.48862.objecter close_session for osd.2 2024-02-19T21:12:15.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 >> v1:172.21.15.163:6809/1563158487 conn(0x7f4e7000d2f0 legacy=0x7f4e7000d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70003630 osd=2 1 2024-02-19T21:12:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 10 client.48862.objecter close_session for osd.3 2024-02-19T21:12:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 >> v1:172.21.15.163:6813/2112279769 conn(0x7f4e70011e80 legacy=0x7f4e70014290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70003440 osd=3 1 2024-02-19T21:12:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 10 client.48862.objecter close_session for osd.4 2024-02-19T21:12:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 >> v1:172.21.15.193:6800/3129716965 conn(0x7f4e70016330 legacy=0x7f4e70016770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70015cb0 osd=4 1 2024-02-19T21:12:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 10 client.48862.objecter close_session for osd.5 2024-02-19T21:12:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 >> v1:172.21.15.193:6804/805161991 conn(0x7f4e7001c150 legacy=0x7f4e7001a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e92aaa700 1 -- 172.21.15.163:0/2844655645 reap_dead start 2024-02-19T21:12:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e7001bad0 osd=5 1 2024-02-19T21:12:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 10 client.48862.objecter close_session for osd.6 2024-02-19T21:12:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 >> v1:172.21.15.193:6808/787604615 conn(0x7f4e7001f1e0 legacy=0x7f4e700215d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e92aaa700 1 -- 172.21.15.163:0/2844655645 reap_dead start 2024-02-19T21:12:15.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e92aaa700 1 -- 172.21.15.163:0/2844655645 reap_dead start 2024-02-19T21:12:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e92aaa700 1 -- 172.21.15.163:0/2844655645 reap_dead start 2024-02-19T21:12:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e7001c700 osd=6 1 2024-02-19T21:12:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 10 client.48862.objecter close_session for osd.7 2024-02-19T21:12:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 >> v1:172.21.15.193:6812/157927432 conn(0x7f4e700249e0 legacy=0x7f4e70022d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 20 client.48862.objecter put_session s=0x7f4e70024360 osd=7 1 2024-02-19T21:12:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 20 client.48862.objecter shutdown clearing up homeless session... 2024-02-19T21:12:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 10 client.48862.objecter successfully canceled tick 2024-02-19T21:12:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4e78063eb0 legacy=0x7f4e78066370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.181+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 >> v1:172.21.15.193:6789/0 conn(0x7f4e8c0fd960 legacy=0x7f4e8c0fca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.182+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 shutdown_connections 2024-02-19T21:12:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.182+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 >> 172.21.15.163:0/2844655645 conn(0x7f4e8c06cc00 msgr2=0x7f4e8c107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.182+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 shutdown_connections 2024-02-19T21:12:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.182+0000 7f4e9450d700 1 -- 172.21.15.163:0/2844655645 wait complete. 2024-02-19T21:12:15.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:15.182+0000 7f4e9450d700 1 librados: shutdown 2024-02-19T21:12:15.185 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.185 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.185 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.185 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.185 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.185 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.185 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.185 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:12:15.190 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:15.217 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:15.341 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:15.345 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:15.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:15.446 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:12:15.474 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:15.492 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:12:15.495 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:15.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:12:15.496 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:14.307369+0000 mon.a (mon.0) 2749 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:15.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:14.743025+0000 mon.a (mon.0) 2750 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:15.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:15.003962+0000 mon.b (mon.1) 648 : audit [DBG] from='client.? v1:172.21.15.163:0/2844655645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:15.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:15.306211+0000 mon.a (mon.0) 2751 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:12:15.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:15.306304+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e499: 8 total, 8 up, 6 in 2024-02-19T21:12:15.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:15.307541+0000 mon.b (mon.1) 649 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:15.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:15.307812+0000 mon.a (mon.0) 2753 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:15.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:14.307369+0000 mon.a (mon.0) 2749 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:15.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:14.743025+0000 mon.a (mon.0) 2750 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:15.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:15.003962+0000 mon.b (mon.1) 648 : audit [DBG] from='client.? v1:172.21.15.163:0/2844655645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:15.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:15.306211+0000 mon.a (mon.0) 2751 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:12:15.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:15.306304+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e499: 8 total, 8 up, 6 in 2024-02-19T21:12:15.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:15.307541+0000 mon.b (mon.1) 649 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:15.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:15.307812+0000 mon.a (mon.0) 2753 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:15.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:14.307369+0000 mon.a (mon.0) 2749 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:15.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:14.743025+0000 mon.a (mon.0) 2750 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:15.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:15.003962+0000 mon.b (mon.1) 648 : audit [DBG] from='client.? v1:172.21.15.163:0/2844655645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:15.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:15.306211+0000 mon.a (mon.0) 2751 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:12:15.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:15.306304+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e499: 8 total, 8 up, 6 in 2024-02-19T21:12:15.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:15.307541+0000 mon.b (mon.1) 649 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:15.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:15.307812+0000 mon.a (mon.0) 2753 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:15.596 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:15.625 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:15.723 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:15.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:15.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:15.831 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:15.858 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:15.877 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:15.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:15.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:15.981 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:16.001 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:16.007 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:16.025 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:16.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:16.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:16.129 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:16.157 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:16.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:14.639246+0000 osd.4 (osd.4) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:16.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:14.639423+0000 osd.4 (osd.4) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:16.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:15.249410+0000 mgr.y (mgr.14550) 455 : cluster [DBG] pgmap v747: 238 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 236 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 767 B/s wr, 11 op/s; 2/1095 objects misplaced (0.183%); 0 B/s, 61 objects/s recovering 2024-02-19T21:12:16.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:15.336822+0000 osd.2 (osd.2) 204 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:16.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:15.337017+0000 osd.2 (osd.2) 205 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:16.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:15.661003+0000 osd.6 (osd.6) 209 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:16.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:15.661201+0000 osd.6 (osd.6) 210 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:16.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:15.740006+0000 mon.a (mon.0) 2754 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-19T21:12:16.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:15.740062+0000 mon.a (mon.0) 2755 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 85/1098 objects degraded (7.741%), 9 pgs degraded) 2024-02-19T21:12:16.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:16.307876+0000 mon.a (mon.0) 2756 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-19T21:12:16.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:16.307967+0000 mon.a (mon.0) 2757 : cluster [DBG] osdmap e500: 8 total, 8 up, 6 in 2024-02-19T21:12:16.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:16.309594+0000 mon.b (mon.1) 650 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:16.454 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:16.309936+0000 mon.a (mon.0) 2758 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:16.454 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:14.639246+0000 osd.4 (osd.4) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:16.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:14.639423+0000 osd.4 (osd.4) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:16.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:15.249410+0000 mgr.y (mgr.14550) 455 : cluster [DBG] pgmap v747: 238 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 236 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 767 B/s wr, 11 op/s; 2/1095 objects misplaced (0.183%); 0 B/s, 61 objects/s recovering 2024-02-19T21:12:16.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:15.336822+0000 osd.2 (osd.2) 204 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:16.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:15.337017+0000 osd.2 (osd.2) 205 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:16.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:15.661003+0000 osd.6 (osd.6) 209 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:16.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:15.661201+0000 osd.6 (osd.6) 210 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:16.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:15.740006+0000 mon.a (mon.0) 2754 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-19T21:12:16.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:15.740062+0000 mon.a (mon.0) 2755 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 85/1098 objects degraded (7.741%), 9 pgs degraded) 2024-02-19T21:12:16.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:16.307876+0000 mon.a (mon.0) 2756 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-19T21:12:16.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:16.307967+0000 mon.a (mon.0) 2757 : cluster [DBG] osdmap e500: 8 total, 8 up, 6 in 2024-02-19T21:12:16.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:16.309594+0000 mon.b (mon.1) 650 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:16.455 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:16.309936+0000 mon.a (mon.0) 2758 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:16.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:14.639246+0000 osd.4 (osd.4) 187 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:16.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:14.639423+0000 osd.4 (osd.4) 188 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:16.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:15.249410+0000 mgr.y (mgr.14550) 455 : cluster [DBG] pgmap v747: 238 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 236 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 767 B/s wr, 11 op/s; 2/1095 objects misplaced (0.183%); 0 B/s, 61 objects/s recovering 2024-02-19T21:12:16.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:15.336822+0000 osd.2 (osd.2) 204 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:16.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:15.337017+0000 osd.2 (osd.2) 205 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:16.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:15.661003+0000 osd.6 (osd.6) 209 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:16.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:15.661201+0000 osd.6 (osd.6) 210 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:16.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:15.740006+0000 mon.a (mon.0) 2754 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-19T21:12:16.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:15.740062+0000 mon.a (mon.0) 2755 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 85/1098 objects degraded (7.741%), 9 pgs degraded) 2024-02-19T21:12:16.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:16.307876+0000 mon.a (mon.0) 2756 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-19T21:12:16.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:16.307967+0000 mon.a (mon.0) 2757 : cluster [DBG] osdmap e500: 8 total, 8 up, 6 in 2024-02-19T21:12:16.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:16.309594+0000 mon.b (mon.1) 650 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:16.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:16.309936+0000 mon.a (mon.0) 2758 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:17.106 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:17.108 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:17.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:17.210 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:17.242 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:17.262 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:17.265 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:17.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:17.322 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:15.815185+0000 osd.1 (osd.1) 243 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:17.322 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:15.815357+0000 osd.1 (osd.1) 244 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:17.322 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:16.243016+0000 osd.2 (osd.2) 206 : cluster [DBG] 5.b scrub starts 2024-02-19T21:12:17.323 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:16.244099+0000 osd.2 (osd.2) 207 : cluster [DBG] 5.b scrub ok 2024-02-19T21:12:17.323 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:17.309378+0000 mon.a (mon.0) 2759 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:12:17.323 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:17.309442+0000 mon.a (mon.0) 2760 : cluster [DBG] osdmap e501: 8 total, 8 up, 6 in 2024-02-19T21:12:17.366 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:17.395 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:17.413 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:17.415 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:17.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:17.516 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:17.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:15.815185+0000 osd.1 (osd.1) 243 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:17.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:15.815357+0000 osd.1 (osd.1) 244 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:17.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:16.243016+0000 osd.2 (osd.2) 206 : cluster [DBG] 5.b scrub starts 2024-02-19T21:12:17.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:16.244099+0000 osd.2 (osd.2) 207 : cluster [DBG] 5.b scrub ok 2024-02-19T21:12:17.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:17.309378+0000 mon.a (mon.0) 2759 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:12:17.542 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:17.309442+0000 mon.a (mon.0) 2760 : cluster [DBG] osdmap e501: 8 total, 8 up, 6 in 2024-02-19T21:12:17.544 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:17.562 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:17.565 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:17.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:17.666 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:17.692 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:17.711 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:17.713 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:17.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:17.718 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:15.815185+0000 osd.1 (osd.1) 243 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:17.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:15.815357+0000 osd.1 (osd.1) 244 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:17.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:16.243016+0000 osd.2 (osd.2) 206 : cluster [DBG] 5.b scrub starts 2024-02-19T21:12:17.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:16.244099+0000 osd.2 (osd.2) 207 : cluster [DBG] 5.b scrub ok 2024-02-19T21:12:17.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:17.309378+0000 mon.a (mon.0) 2759 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-79","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:12:17.719 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:17.309442+0000 mon.a (mon.0) 2760 : cluster [DBG] osdmap e501: 8 total, 8 up, 6 in 2024-02-19T21:12:17.814 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:17.820 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:12:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:12:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:12:17.846 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:17.865 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:17.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:17.969 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:17.996 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:18.015 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:18.017 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:18.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:18.118 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:18.146 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:18.394 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:18.397 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:18.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:18.498 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:18.512 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:12:18.512 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:12:18.512 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-02-19T21:12:18.512 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-02-19T21:12:18.512 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:12:18.526 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:18.546 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:18.548 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:18.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:18.650 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:18.675 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:16.337274+0000 osd.3 (osd.3) 225 : cluster [DBG] 16.18 scrub starts 2024-02-19T21:12:18.675 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:16.339573+0000 osd.3 (osd.3) 226 : cluster [DBG] 16.18 scrub ok 2024-02-19T21:12:18.675 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:17.250022+0000 mgr.y (mgr.14550) 456 : cluster [DBG] pgmap v750: 238 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 236 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 2/1095 objects misplaced (0.183%); 337 B/s, 61 objects/s recovering 2024-02-19T21:12:18.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:17.343326+0000 osd.3 (osd.3) 227 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:12:18.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:17.344908+0000 osd.3 (osd.3) 228 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:12:18.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:16.337274+0000 osd.3 (osd.3) 225 : cluster [DBG] 16.18 scrub starts 2024-02-19T21:12:18.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:16.339573+0000 osd.3 (osd.3) 226 : cluster [DBG] 16.18 scrub ok 2024-02-19T21:12:18.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:17.250022+0000 mgr.y (mgr.14550) 456 : cluster [DBG] pgmap v750: 238 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 236 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 2/1095 objects misplaced (0.183%); 337 B/s, 61 objects/s recovering 2024-02-19T21:12:18.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:17.343326+0000 osd.3 (osd.3) 227 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:12:18.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:17.344908+0000 osd.3 (osd.3) 228 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:12:18.677 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:18.776 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:18.780 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:18.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:18.781 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:12:18.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:16.337274+0000 osd.3 (osd.3) 225 : cluster [DBG] 16.18 scrub starts 2024-02-19T21:12:18.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:16.339573+0000 osd.3 (osd.3) 226 : cluster [DBG] 16.18 scrub ok 2024-02-19T21:12:18.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:17.250022+0000 mgr.y (mgr.14550) 456 : cluster [DBG] pgmap v750: 238 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 236 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 11 op/s; 2/1095 objects misplaced (0.183%); 337 B/s, 61 objects/s recovering 2024-02-19T21:12:18.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:17.343326+0000 osd.3 (osd.3) 227 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:12:18.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:17.344908+0000 osd.3 (osd.3) 228 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:12:18.882 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:18.908 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:19.112 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:19.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:19.120 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-19T21:12:19.216 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:19.248 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:19.269 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:19.271 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:19.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:19.373 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:19.401 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:19.421 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:19.423 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:19.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:19.524 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:19.552 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:19.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:17.449070+0000 osd.7 (osd.7) 233 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:12:19.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:17.451959+0000 osd.7 (osd.7) 234 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:12:19.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:18.329755+0000 mon.a (mon.0) 2761 : cluster [DBG] osdmap e502: 8 total, 8 up, 6 in 2024-02-19T21:12:19.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:18.338251+0000 osd.4 (osd.4) 189 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:12:19.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:18.339924+0000 osd.4 (osd.4) 190 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:12:19.572 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:19.574 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:19.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:19.612 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:17.449070+0000 osd.7 (osd.7) 233 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:12:19.612 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:17.451959+0000 osd.7 (osd.7) 234 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:12:19.612 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:18.329755+0000 mon.a (mon.0) 2761 : cluster [DBG] osdmap e502: 8 total, 8 up, 6 in 2024-02-19T21:12:19.612 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:18.338251+0000 osd.4 (osd.4) 189 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:12:19.613 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:18.339924+0000 osd.4 (osd.4) 190 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:12:19.613 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:17.449070+0000 osd.7 (osd.7) 233 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:12:19.613 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:17.451959+0000 osd.7 (osd.7) 234 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:12:19.613 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:18.329755+0000 mon.a (mon.0) 2761 : cluster [DBG] osdmap e502: 8 total, 8 up, 6 in 2024-02-19T21:12:19.613 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:18.338251+0000 osd.4 (osd.4) 189 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:12:19.613 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:18.339924+0000 osd.4 (osd.4) 190 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:12:19.676 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:19.705 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:19.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.890+0000 7f1241150700 1 -- 172.21.15.163:0/3805721921 >> v1:172.21.15.163:6790/0 conn(0x7f123c0fd980 legacy=0x7f123c0fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:19.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.891+0000 7f1241150700 1 -- 172.21.15.163:0/3805721921 shutdown_connections 2024-02-19T21:12:19.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.891+0000 7f1241150700 1 -- 172.21.15.163:0/3805721921 >> 172.21.15.163:0/3805721921 conn(0x7f123c0f35a0 msgr2=0x7f123c0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:19.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.891+0000 7f1241150700 1 -- 172.21.15.163:0/3805721921 shutdown_connections 2024-02-19T21:12:19.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.891+0000 7f1241150700 1 -- 172.21.15.163:0/3805721921 wait complete. 2024-02-19T21:12:19.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.891+0000 7f1241150700 1 librados: starting msgr at 2024-02-19T21:12:19.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.891+0000 7f1241150700 1 librados: starting objecter 2024-02-19T21:12:19.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.891+0000 7f1241150700 1 Processor -- start 2024-02-19T21:12:19.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1241150700 1 -- start start 2024-02-19T21:12:19.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1241150700 1 librados: setting wanted keys 2024-02-19T21:12:19.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1241150700 1 librados: calling monclient init 2024-02-19T21:12:19.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1241150700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f123c1172c0 con 0x7f123c0fd980 2024-02-19T21:12:19.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1241150700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f123c1184a0 con 0x7f123c1042c0 2024-02-19T21:12:19.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1241150700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f123c1196a0 con 0x7f123c100780 2024-02-19T21:12:19.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f123ad9d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f123c0fd980 0x7f123c0f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:56522/0 (socket says 172.21.15.163:56522) 2024-02-19T21:12:19.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f123ad9d700 1 -- 172.21.15.163:0/2699522698 learned_addr learned my addr 172.21.15.163:0/2699522698 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f123a59c700 1 --1- 172.21.15.163:0/2699522698 >> v1:172.21.15.163:6790/0 conn(0x7f123c100780 0x7f123c0fa070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45928/0 (socket says 172.21.15.163:45928) 2024-02-19T21:12:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1223fff700 10 client.?.objecter ms_handle_connect 0x7f123c0fd980 2024-02-19T21:12:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1223fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1223fff700 10 client.?.objecter ms_handle_connect 0x7f123c100780 2024-02-19T21:12:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1223fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361956966 0 0) 0x7f123c1172c0 con 0x7f123c0fd980 2024-02-19T21:12:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1224003800 con 0x7f123c0fd980 2024-02-19T21:12:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1223fff700 10 client.?.objecter ms_handle_connect 0x7f123c1042c0 2024-02-19T21:12:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1223fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287322772 0 0) 0x7f123c1196a0 con 0x7f123c100780 2024-02-19T21:12:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f123c1172c0 con 0x7f123c100780 2024-02-19T21:12:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927658872 0 0) 0x7f123c1184a0 con 0x7f123c1042c0 2024-02-19T21:12:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f123c1196a0 con 0x7f123c1042c0 2024-02-19T21:12:19.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3426082541 0 0) 0x7f1224003800 con 0x7f123c0fd980 2024-02-19T21:12:19.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.892+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f123c1184a0 con 0x7f123c0fd980 2024-02-19T21:12:19.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f122c002e20 con 0x7f123c0fd980 2024-02-19T21:12:19.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1546770528 0 0) 0x7f123c1172c0 con 0x7f123c100780 2024-02-19T21:12:19.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1224003800 con 0x7f123c100780 2024-02-19T21:12:19.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1230001840 con 0x7f123c100780 2024-02-19T21:12:19.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 631705059 0 0) 0x7f123c1196a0 con 0x7f123c1042c0 2024-02-19T21:12:19.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f123c1172c0 con 0x7f123c1042c0 2024-02-19T21:12:19.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1234001950 con 0x7f123c1042c0 2024-02-19T21:12:19.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4002736706 0 0) 0x7f123c1184a0 con 0x7f123c0fd980 2024-02-19T21:12:19.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 >> v1:172.21.15.163:6790/0 conn(0x7f123c100780 legacy=0x7f123c0fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:19.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 >> v1:172.21.15.193:6789/0 conn(0x7f123c1042c0 legacy=0x7f123c115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:19.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123c11a8a0 con 0x7f123c0fd980 2024-02-19T21:12:19.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1241150700 1 -- 172.21.15.163:0/2699522698 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123c119890 con 0x7f123c0fd980 2024-02-19T21:12:19.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1241150700 10 client.23337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:19.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1241150700 1 -- 172.21.15.163:0/2699522698 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f123c119d80 con 0x7f123c0fd980 2024-02-19T21:12:19.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1241150700 1 librados: init done 2024-02-19T21:12:19.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f122c002400 con 0x7f123c0fd980 2024-02-19T21:12:19.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.893+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f122c003950 con 0x7f123c0fd980 2024-02-19T21:12:19.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.894+0000 7f1241150700 1 -- 172.21.15.163:0/2699522698 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1228004f20 con 0x7f123c0fd980 2024-02-19T21:12:19.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.894+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f122c001f60 con 0x7f123c0fd980 2024-02-19T21:12:19.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.895+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(503..503 src has 43..503) v4 ==== 8134+0+0 (unknown 2805338405 0 0) 0x7f122c079050 con 0x7f123c0fd980 2024-02-19T21:12:19.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.895+0000 7f1223fff700 10 client.23337.objecter ms_dispatch 0x7f123c001550 osd_map(503..503 src has 43..503) v4 2024-02-19T21:12:19.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.895+0000 7f1223fff700 3 client.23337.objecter handle_osd_map got epochs [503,503] > 0 2024-02-19T21:12:19.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.895+0000 7f1223fff700 3 client.23337.objecter handle_osd_map decoding full epoch 503 2024-02-19T21:12:19.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.895+0000 7f1223fff700 20 client.23337.objecter dump_active .. 0 homeless 2024-02-19T21:12:19.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.895+0000 7f1223fff700 10 client.23337.objecter ms_handle_connect 0x7f1224063d00 2024-02-19T21:12:19.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:19.899+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f122c04c520 con 0x7f123c0fd980 2024-02-19T21:12:19.904 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:19.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:19.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:20.007 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:20.036 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:20.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:20.084+0000 7f1241150700 1 -- 172.21.15.163:0/2699522698 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7f12280051d0 con 0x7f123c0fd980 2024-02-19T21:12:20.217 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:20.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:20.349+0000 7f1223fff700 1 -- 172.21.15.163:0/2699522698 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v504) v1 ==== 113+0+0 (unknown 1119887780 0 0) 0x7f122c04a990 con 0x7f123c0fd980 2024-02-19T21:12:20.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:20.353+0000 7f1241150700 10 librados: watch_flush enter 2024-02-19T21:12:20.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:20.353+0000 7f1241150700 10 librados: watch_flush exit 2024-02-19T21:12:20.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:20.353+0000 7f1241150700 20 client.23337.objecter shutdown clearing up homeless session... 2024-02-19T21:12:20.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:20.353+0000 7f1241150700 10 client.23337.objecter successfully canceled tick 2024-02-19T21:12:20.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:20.354+0000 7f1241150700 1 -- 172.21.15.163:0/2699522698 >> v1:172.21.15.163:6800/1552514681 conn(0x7f1224063d00 legacy=0x7f12240661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:20.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:20.354+0000 7f1241150700 1 -- 172.21.15.163:0/2699522698 >> v1:172.21.15.163:6789/0 conn(0x7f123c0fd980 legacy=0x7f123c0f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:20.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:20.354+0000 7f1241150700 1 -- 172.21.15.163:0/2699522698 shutdown_connections 2024-02-19T21:12:20.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:20.354+0000 7f1241150700 1 -- 172.21.15.163:0/2699522698 >> 172.21.15.163:0/2699522698 conn(0x7f123c0f35a0 msgr2=0x7f123c107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:20.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:20.354+0000 7f1241150700 1 -- 172.21.15.163:0/2699522698 shutdown_connections 2024-02-19T21:12:20.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:20.354+0000 7f1241150700 1 -- 172.21.15.163:0/2699522698 wait complete. 2024-02-19T21:12:20.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:20.354+0000 7f1241150700 1 librados: shutdown 2024-02-19T21:12:20.354 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.1 to 1 (10000) 2024-02-19T21:12:20.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:18.448525+0000 osd.7 (osd.7) 235 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:12:20.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:18.452016+0000 osd.7 (osd.7) 236 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:12:20.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:18.909289+0000 osd.0 (osd.0) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:20.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:18.909468+0000 osd.0 (osd.0) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:20.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:19.250429+0000 mgr.y (mgr.14550) 457 : cluster [DBG] pgmap v753: 238 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 236 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2/1095 objects misplaced (0.183%) 2024-02-19T21:12:20.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:19.344950+0000 mon.a (mon.0) 2762 : cluster [DBG] osdmap e503: 8 total, 8 up, 6 in 2024-02-19T21:12:20.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:19.382358+0000 mon.b (mon.1) 651 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:20.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:19.382713+0000 mon.a (mon.0) 2763 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:20.719 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:19.383206+0000 mon.b (mon.1) 652 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-79"}]: dispatch 2024-02-19T21:12:20.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:19.383429+0000 mon.a (mon.0) 2764 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-79"}]: dispatch 2024-02-19T21:12:20.720 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:20.085697+0000 mon.a (mon.0) 2765 : audit [INF] from='client.? v1:172.21.15.163:0/2699522698' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:12:20.720 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:18.448525+0000 osd.7 (osd.7) 235 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:12:20.720 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:18.452016+0000 osd.7 (osd.7) 236 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:12:20.720 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:18.909289+0000 osd.0 (osd.0) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:20.720 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:18.909468+0000 osd.0 (osd.0) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:20.720 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:19.250429+0000 mgr.y (mgr.14550) 457 : cluster [DBG] pgmap v753: 238 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 236 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2/1095 objects misplaced (0.183%) 2024-02-19T21:12:20.720 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:19.344950+0000 mon.a (mon.0) 2762 : cluster [DBG] osdmap e503: 8 total, 8 up, 6 in 2024-02-19T21:12:20.720 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:19.382358+0000 mon.b (mon.1) 651 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:20.720 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:19.382713+0000 mon.a (mon.0) 2763 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:20.720 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:19.383206+0000 mon.b (mon.1) 652 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-79"}]: dispatch 2024-02-19T21:12:20.721 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:19.383429+0000 mon.a (mon.0) 2764 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-79"}]: dispatch 2024-02-19T21:12:20.721 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:20.085697+0000 mon.a (mon.0) 2765 : audit [INF] from='client.? v1:172.21.15.163:0/2699522698' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:12:20.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:12:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:12:20.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:18.448525+0000 osd.7 (osd.7) 235 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:12:20.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:18.452016+0000 osd.7 (osd.7) 236 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:12:20.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:18.909289+0000 osd.0 (osd.0) 127 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:20.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:18.909468+0000 osd.0 (osd.0) 128 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:20.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:19.250429+0000 mgr.y (mgr.14550) 457 : cluster [DBG] pgmap v753: 238 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 236 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2/1095 objects misplaced (0.183%) 2024-02-19T21:12:20.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:19.344950+0000 mon.a (mon.0) 2762 : cluster [DBG] osdmap e503: 8 total, 8 up, 6 in 2024-02-19T21:12:20.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:19.382358+0000 mon.b (mon.1) 651 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:20.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:19.382713+0000 mon.a (mon.0) 2763 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:20.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:19.383206+0000 mon.b (mon.1) 652 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-79"}]: dispatch 2024-02-19T21:12:20.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:19.383429+0000 mon.a (mon.0) 2764 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-79"}]: dispatch 2024-02-19T21:12:20.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:20.085697+0000 mon.a (mon.0) 2765 : audit [INF] from='client.? v1:172.21.15.163:0/2699522698' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:12:21.185 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:21.189 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:21.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:21.291 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:21.323 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:21.345 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:21.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:21.448 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:21.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:20.178157+0000 osd.2 (osd.2) 208 : cluster [DBG] 61.1f scrub starts 2024-02-19T21:12:21.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:20.180047+0000 osd.2 (osd.2) 209 : cluster [DBG] 61.1f scrub ok 2024-02-19T21:12:21.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:20.350731+0000 mon.a (mon.0) 2766 : audit [INF] from='client.? v1:172.21.15.163:0/2699522698' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-19T21:12:21.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:20.350767+0000 mon.a (mon.0) 2767 : cluster [DBG] osdmap e504: 8 total, 8 up, 6 in 2024-02-19T21:12:21.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:20.510566+0000 mgr.y (mgr.14550) 458 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:21.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:20.178157+0000 osd.2 (osd.2) 208 : cluster [DBG] 61.1f scrub starts 2024-02-19T21:12:21.476 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:20.180047+0000 osd.2 (osd.2) 209 : cluster [DBG] 61.1f scrub ok 2024-02-19T21:12:21.476 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:20.350731+0000 mon.a (mon.0) 2766 : audit [INF] from='client.? v1:172.21.15.163:0/2699522698' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-19T21:12:21.476 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:20.350767+0000 mon.a (mon.0) 2767 : cluster [DBG] osdmap e504: 8 total, 8 up, 6 in 2024-02-19T21:12:21.476 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:20.510566+0000 mgr.y (mgr.14550) 458 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:21.477 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:21.601 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:21.603 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:21.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:21.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.697+0000 7fbf10dd2700 1 -- 172.21.15.163:0/1516559073 >> v1:172.21.15.163:6789/0 conn(0x7fbf0c1031d0 legacy=0x7fbf0c105650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:21.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.697+0000 7fbf10dd2700 1 -- 172.21.15.163:0/1516559073 shutdown_connections 2024-02-19T21:12:21.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.697+0000 7fbf10dd2700 1 -- 172.21.15.163:0/1516559073 >> 172.21.15.163:0/1516559073 conn(0x7fbf0c0f2470 msgr2=0x7fbf0c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:21.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.697+0000 7fbf10dd2700 1 -- 172.21.15.163:0/1516559073 shutdown_connections 2024-02-19T21:12:21.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.697+0000 7fbf10dd2700 1 -- 172.21.15.163:0/1516559073 wait complete. 2024-02-19T21:12:21.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbf10dd2700 1 librados: starting msgr at 2024-02-19T21:12:21.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbf10dd2700 1 librados: starting objecter 2024-02-19T21:12:21.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbf10dd2700 1 Processor -- start 2024-02-19T21:12:21.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbf10dd2700 1 -- start start 2024-02-19T21:12:21.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbf10dd2700 1 librados: setting wanted keys 2024-02-19T21:12:21.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbf10dd2700 1 librados: calling monclient init 2024-02-19T21:12:21.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbf10dd2700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbf0c1162c0 con 0x7fbf0c0fc890 2024-02-19T21:12:21.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbf10dd2700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbf0c1174c0 con 0x7fbf0c0ff690 2024-02-19T21:12:21.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbf10dd2700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbf0c1186c0 con 0x7fbf0c1031d0 2024-02-19T21:12:21.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbf0ad9d700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fbf0c1031d0 0x7fbf0c114980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45952/0 (socket says 172.21.15.163:45952) 2024-02-19T21:12:21.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbf0a59c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fbf0c0fc890 0x7fbf0c0f8870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:56538/0 (socket says 172.21.15.163:56538) 2024-02-19T21:12:21.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbf0ad9d700 1 -- 172.21.15.163:0/3938765615 learned_addr learned my addr 172.21.15.163:0/3938765615 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:21.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbefb7fe700 10 client.?.objecter ms_handle_connect 0x7fbf0c1031d0 2024-02-19T21:12:21.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbefb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:21.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbefb7fe700 10 client.?.objecter ms_handle_connect 0x7fbf0c0fc890 2024-02-19T21:12:21.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbefb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:21.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbefb7fe700 10 client.?.objecter ms_handle_connect 0x7fbf0c0ff690 2024-02-19T21:12:21.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbefb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:21.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222846800 0 0) 0x7fbf0c1186c0 con 0x7fbf0c1031d0 2024-02-19T21:12:21.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbef4003030 con 0x7fbf0c1031d0 2024-02-19T21:12:21.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979044582 0 0) 0x7fbf0c1162c0 con 0x7fbf0c0fc890 2024-02-19T21:12:21.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.698+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf0c1186c0 con 0x7fbf0c0fc890 2024-02-19T21:12:21.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618824195 0 0) 0x7fbf0c1174c0 con 0x7fbf0c0ff690 2024-02-19T21:12:21.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf0c1162c0 con 0x7fbf0c0ff690 2024-02-19T21:12:21.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1561167261 0 0) 0x7fbf0c1186c0 con 0x7fbf0c0fc890 2024-02-19T21:12:21.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf0c1174c0 con 0x7fbf0c0fc890 2024-02-19T21:12:21.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbf000018c0 con 0x7fbf0c0fc890 2024-02-19T21:12:21.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1830583809 0 0) 0x7fbef4003030 con 0x7fbf0c1031d0 2024-02-19T21:12:21.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf0c1186c0 con 0x7fbf0c1031d0 2024-02-19T21:12:21.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbf040023a0 con 0x7fbf0c1031d0 2024-02-19T21:12:21.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1711946901 0 0) 0x7fbf0c1162c0 con 0x7fbf0c0ff690 2024-02-19T21:12:21.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbef4003030 con 0x7fbf0c0ff690 2024-02-19T21:12:21.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbefc001970 con 0x7fbf0c0ff690 2024-02-19T21:12:21.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2931946786 0 0) 0x7fbf0c1174c0 con 0x7fbf0c0fc890 2024-02-19T21:12:21.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 >> v1:172.21.15.163:6790/0 conn(0x7fbf0c1031d0 legacy=0x7fbf0c114980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:21.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 >> v1:172.21.15.193:6789/0 conn(0x7fbf0c0ff690 legacy=0x7fbf0c0f9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:21.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c1198c0 con 0x7fbf0c0fc890 2024-02-19T21:12:21.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbf10dd2700 1 -- 172.21.15.163:0/3938765615 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf0c1164b0 con 0x7fbf0c0fc890 2024-02-19T21:12:21.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbf10dd2700 10 client.23343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:21.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbf10dd2700 1 -- 172.21.15.163:0/3938765615 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf0c116920 con 0x7fbf0c0fc890 2024-02-19T21:12:21.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbf10dd2700 1 librados: init done 2024-02-19T21:12:21.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fbf00002460 con 0x7fbf0c0fc890 2024-02-19T21:12:21.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.699+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbf00005ae0 con 0x7fbf0c0fc890 2024-02-19T21:12:21.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.700+0000 7fbf10dd2700 1 -- 172.21.15.163:0/3938765615 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeec004f20 con 0x7fbf0c0fc890 2024-02-19T21:12:21.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.701+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fbf00001fc0 con 0x7fbf0c0fc890 2024-02-19T21:12:21.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.701+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(505..505 src has 43..505) v4 ==== 8054+0+0 (unknown 2006407815 0 0) 0x7fbf00078b90 con 0x7fbf0c0fc890 2024-02-19T21:12:21.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.701+0000 7fbefb7fe700 10 client.23343.objecter ms_dispatch 0x7fbf0c001550 osd_map(505..505 src has 43..505) v4 2024-02-19T21:12:21.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.701+0000 7fbefb7fe700 3 client.23343.objecter handle_osd_map got epochs [505,505] > 0 2024-02-19T21:12:21.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.701+0000 7fbefb7fe700 3 client.23343.objecter handle_osd_map decoding full epoch 505 2024-02-19T21:12:21.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.701+0000 7fbefb7fe700 20 client.23343.objecter dump_active .. 0 homeless 2024-02-19T21:12:21.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.701+0000 7fbefb7fe700 10 client.23343.objecter ms_handle_connect 0x7fbef4065e20 2024-02-19T21:12:21.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.705+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fbf0004bff0 con 0x7fbf0c0fc890 2024-02-19T21:12:21.704 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:21.732 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:21.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:20.178157+0000 osd.2 (osd.2) 208 : cluster [DBG] 61.1f scrub starts 2024-02-19T21:12:21.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:20.180047+0000 osd.2 (osd.2) 209 : cluster [DBG] 61.1f scrub ok 2024-02-19T21:12:21.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:20.350731+0000 mon.a (mon.0) 2766 : audit [INF] from='client.? v1:172.21.15.163:0/2699522698' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-19T21:12:21.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:20.350767+0000 mon.a (mon.0) 2767 : cluster [DBG] osdmap e504: 8 total, 8 up, 6 in 2024-02-19T21:12:21.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:20.510566+0000 mgr.y (mgr.14550) 458 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:21.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:21.884+0000 7fbf10dd2700 1 -- 172.21.15.163:0/3938765615 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fbeec005750 con 0x7fbf0c0fc890 2024-02-19T21:12:21.980 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:21.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:21.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:22.084 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:22.116 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:22.137 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:22.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:22.241 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:22.268 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:22.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:22.359+0000 7fbefb7fe700 1 -- 172.21.15.163:0/3938765615 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v506) v1 ==== 87+0+0 (unknown 4232860116 0 0) 0x7fbf0004a400 con 0x7fbf0c0fc890 2024-02-19T21:12:22.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:22.363+0000 7fbf10dd2700 10 librados: watch_flush enter 2024-02-19T21:12:22.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:22.363+0000 7fbf10dd2700 10 librados: watch_flush exit 2024-02-19T21:12:22.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:22.363+0000 7fbf10dd2700 20 client.23343.objecter shutdown clearing up homeless session... 2024-02-19T21:12:22.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:22.363+0000 7fbf10dd2700 10 client.23343.objecter successfully canceled tick 2024-02-19T21:12:22.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:22.363+0000 7fbf10dd2700 1 -- 172.21.15.163:0/3938765615 >> v1:172.21.15.163:6800/1552514681 conn(0x7fbef4065e20 legacy=0x7fbef40682e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:22.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:22.363+0000 7fbf10dd2700 1 -- 172.21.15.163:0/3938765615 >> v1:172.21.15.163:6789/0 conn(0x7fbf0c0fc890 legacy=0x7fbf0c0f8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:22.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:22.369+0000 7fbf10dd2700 1 -- 172.21.15.163:0/3938765615 shutdown_connections 2024-02-19T21:12:22.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:22.369+0000 7fbf10dd2700 1 -- 172.21.15.163:0/3938765615 >> 172.21.15.163:0/3938765615 conn(0x7fbf0c0f2470 msgr2=0x7fbf0c106640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:22.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:22.369+0000 7fbf10dd2700 1 -- 172.21.15.163:0/3938765615 shutdown_connections 2024-02-19T21:12:22.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:22.369+0000 7fbf10dd2700 1 -- 172.21.15.163:0/3938765615 wait complete. 2024-02-19T21:12:22.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:22.376+0000 7fbf10dd2700 1 librados: shutdown 2024-02-19T21:12:22.376 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:12:22.378 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:20.898847+0000 osd.1 (osd.1) 245 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:12:22.391 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:22.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:22.495 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:22.526 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:22.547 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:22.549 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:22.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:22.650 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:22.677 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:20.902290+0000 osd.1 (osd.1) 246 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:12:22.677 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:21.200615+0000 osd.2 (osd.2) 210 : cluster [DBG] 5.d scrub starts 2024-02-19T21:12:22.677 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:21.202367+0000 osd.2 (osd.2) 211 : cluster [DBG] 5.d scrub ok 2024-02-19T21:12:22.677 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:21.250826+0000 mgr.y (mgr.14550) 459 : cluster [DBG] pgmap v756: 206 pgs: 1 active+remapped+backfill_toofull, 205 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 1/1119 objects misplaced (0.089%); 1 B/s, 0 objects/s recovering 2024-02-19T21:12:22.677 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:21.363074+0000 mon.a (mon.0) 2768 : cluster [DBG] osdmap e505: 8 total, 8 up, 6 in 2024-02-19T21:12:22.677 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:21.365378+0000 mon.b (mon.1) 653 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:22.677 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:21.367083+0000 mon.a (mon.0) 2769 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:22.677 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:21.379777+0000 osd.4 (osd.4) 191 : cluster [DBG] 6.e scrub starts 2024-02-19T21:12:22.677 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:21.380730+0000 osd.4 (osd.4) 192 : cluster [DBG] 6.e scrub ok 2024-02-19T21:12:22.677 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:21.885566+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? v1:172.21.15.163:0/3938765615' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:12:22.678 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:22.698 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:22.701 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:22.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:22.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:20.898847+0000 osd.1 (osd.1) 245 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:12:22.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:20.902290+0000 osd.1 (osd.1) 246 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:12:22.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:21.200615+0000 osd.2 (osd.2) 210 : cluster [DBG] 5.d scrub starts 2024-02-19T21:12:22.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:21.202367+0000 osd.2 (osd.2) 211 : cluster [DBG] 5.d scrub ok 2024-02-19T21:12:22.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:21.250826+0000 mgr.y (mgr.14550) 459 : cluster [DBG] pgmap v756: 206 pgs: 1 active+remapped+backfill_toofull, 205 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 1/1119 objects misplaced (0.089%); 1 B/s, 0 objects/s recovering 2024-02-19T21:12:22.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:21.363074+0000 mon.a (mon.0) 2768 : cluster [DBG] osdmap e505: 8 total, 8 up, 6 in 2024-02-19T21:12:22.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:21.365378+0000 mon.b (mon.1) 653 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:22.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:21.367083+0000 mon.a (mon.0) 2769 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:22.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:21.379777+0000 osd.4 (osd.4) 191 : cluster [DBG] 6.e scrub starts 2024-02-19T21:12:22.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:21.380730+0000 osd.4 (osd.4) 192 : cluster [DBG] 6.e scrub ok 2024-02-19T21:12:22.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:21.885566+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? v1:172.21.15.163:0/3938765615' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:12:22.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:20.898847+0000 osd.1 (osd.1) 245 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:12:22.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:20.902290+0000 osd.1 (osd.1) 246 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:12:22.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:21.200615+0000 osd.2 (osd.2) 210 : cluster [DBG] 5.d scrub starts 2024-02-19T21:12:22.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:21.202367+0000 osd.2 (osd.2) 211 : cluster [DBG] 5.d scrub ok 2024-02-19T21:12:22.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:21.250826+0000 mgr.y (mgr.14550) 459 : cluster [DBG] pgmap v756: 206 pgs: 1 active+remapped+backfill_toofull, 205 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 1/1119 objects misplaced (0.089%); 1 B/s, 0 objects/s recovering 2024-02-19T21:12:22.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:21.363074+0000 mon.a (mon.0) 2768 : cluster [DBG] osdmap e505: 8 total, 8 up, 6 in 2024-02-19T21:12:22.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:21.365378+0000 mon.b (mon.1) 653 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:22.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:21.367083+0000 mon.a (mon.0) 2769 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:22.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:21.379777+0000 osd.4 (osd.4) 191 : cluster [DBG] 6.e scrub starts 2024-02-19T21:12:22.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:21.380730+0000 osd.4 (osd.4) 192 : cluster [DBG] 6.e scrub ok 2024-02-19T21:12:22.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:21.885566+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? v1:172.21.15.163:0/3938765615' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:12:22.802 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:22.830 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:23.344 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:23.371 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 2 1 2024-02-19T21:12:23.380 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:23.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:23.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:23.485 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:23.518 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:23.539 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:23.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:23.642 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:23.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:21.799952+0000 osd.7 (osd.7) 237 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:23.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:21.800142+0000 osd.7 (osd.7) 238 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:23.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:22.214675+0000 osd.2 (osd.2) 212 : cluster [DBG] 2.e scrub starts 2024-02-19T21:12:23.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:22.215541+0000 osd.2 (osd.2) 213 : cluster [DBG] 2.e scrub ok 2024-02-19T21:12:23.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:22.356876+0000 mon.a (mon.0) 2771 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:12:23.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:22.360349+0000 mon.a (mon.0) 2772 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:23.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:22.360497+0000 mon.a (mon.0) 2773 : audit [INF] from='client.? v1:172.21.15.163:0/3938765615' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:12:23.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:22.360547+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e506: 8 total, 8 up, 6 in 2024-02-19T21:12:23.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:22.385197+0000 mon.b (mon.1) 654 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:12:23.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:22.387015+0000 mon.b (mon.1) 655 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:23.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:22.387325+0000 mon.a (mon.0) 2775 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:23.644 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:23.365298+0000 mon.a (mon.0) 2776 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:12:23.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:23.365357+0000 mon.a (mon.0) 2777 : cluster [DBG] osdmap e507: 8 total, 8 up, 6 in 2024-02-19T21:12:23.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:23.370766+0000 mon.b (mon.1) 656 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:23.645 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:23.371085+0000 mon.a (mon.0) 2778 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:23.645 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:12:23] "GET /metrics HTTP/1.1" 200 190214 "" "Prometheus/2.33.4" 2024-02-19T21:12:23.645 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:21.799952+0000 osd.7 (osd.7) 237 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:23.645 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:21.800142+0000 osd.7 (osd.7) 238 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:23.645 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:22.214675+0000 osd.2 (osd.2) 212 : cluster [DBG] 2.e scrub starts 2024-02-19T21:12:23.645 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:22.215541+0000 osd.2 (osd.2) 213 : cluster [DBG] 2.e scrub ok 2024-02-19T21:12:23.645 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:22.356876+0000 mon.a (mon.0) 2771 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:12:23.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:22.360349+0000 mon.a (mon.0) 2772 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:23.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:22.360497+0000 mon.a (mon.0) 2773 : audit [INF] from='client.? v1:172.21.15.163:0/3938765615' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:12:23.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:22.360547+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e506: 8 total, 8 up, 6 in 2024-02-19T21:12:23.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:22.385197+0000 mon.b (mon.1) 654 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:12:23.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:22.387015+0000 mon.b (mon.1) 655 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:23.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:22.387325+0000 mon.a (mon.0) 2775 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:23.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:23.365298+0000 mon.a (mon.0) 2776 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:12:23.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:23.365357+0000 mon.a (mon.0) 2777 : cluster [DBG] osdmap e507: 8 total, 8 up, 6 in 2024-02-19T21:12:23.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:23.370766+0000 mon.b (mon.1) 656 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:23.646 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:23.371085+0000 mon.a (mon.0) 2778 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:23.669 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:21.799952+0000 osd.7 (osd.7) 237 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:23.669 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:21.800142+0000 osd.7 (osd.7) 238 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:23.670 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:22.214675+0000 osd.2 (osd.2) 212 : cluster [DBG] 2.e scrub starts 2024-02-19T21:12:23.670 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:22.215541+0000 osd.2 (osd.2) 213 : cluster [DBG] 2.e scrub ok 2024-02-19T21:12:23.670 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:22.356876+0000 mon.a (mon.0) 2771 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:12:23.670 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:22.360349+0000 mon.a (mon.0) 2772 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:23.670 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:22.360497+0000 mon.a (mon.0) 2773 : audit [INF] from='client.? v1:172.21.15.163:0/3938765615' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:12:23.670 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:22.360547+0000 mon.a (mon.0) 2774 : cluster [DBG] osdmap e506: 8 total, 8 up, 6 in 2024-02-19T21:12:23.670 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:22.385197+0000 mon.b (mon.1) 654 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:12:23.670 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:22.387015+0000 mon.b (mon.1) 655 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:23.670 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:22.387325+0000 mon.a (mon.0) 2775 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:23.670 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:23.365298+0000 mon.a (mon.0) 2776 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:12:23.670 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:23.365357+0000 mon.a (mon.0) 2777 : cluster [DBG] osdmap e507: 8 total, 8 up, 6 in 2024-02-19T21:12:23.670 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:23.370766+0000 mon.b (mon.1) 656 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:23.670 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:23.371085+0000 mon.a (mon.0) 2778 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:23.671 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:23.692 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:23.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:23.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:23.795 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:23.824 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:23.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.892+0000 7fe579e52700 1 -- 172.21.15.163:0/2341340515 >> v1:172.21.15.163:6789/0 conn(0x7fe574100780 legacy=0x7fe574102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:23.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.892+0000 7fe579e52700 1 -- 172.21.15.163:0/2341340515 shutdown_connections 2024-02-19T21:12:23.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.892+0000 7fe579e52700 1 -- 172.21.15.163:0/2341340515 >> 172.21.15.163:0/2341340515 conn(0x7fe5740f35a0 msgr2=0x7fe5740f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:23.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.892+0000 7fe579e52700 1 -- 172.21.15.163:0/2341340515 shutdown_connections 2024-02-19T21:12:23.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.892+0000 7fe579e52700 1 -- 172.21.15.163:0/2341340515 wait complete. 2024-02-19T21:12:23.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.893+0000 7fe579e52700 1 librados: starting msgr at 2024-02-19T21:12:23.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.893+0000 7fe579e52700 1 librados: starting objecter 2024-02-19T21:12:23.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.893+0000 7fe579e52700 1 Processor -- start 2024-02-19T21:12:23.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.893+0000 7fe579e52700 1 -- start start 2024-02-19T21:12:23.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.893+0000 7fe579e52700 1 librados: setting wanted keys 2024-02-19T21:12:23.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.893+0000 7fe579e52700 1 librados: calling monclient init 2024-02-19T21:12:23.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.893+0000 7fe579e52700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe5741174e0 con 0x7fe5741042c0 2024-02-19T21:12:23.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.893+0000 7fe579e52700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe5741186e0 con 0x7fe5740fd980 2024-02-19T21:12:23.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.893+0000 7fe579e52700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe5741198e0 con 0x7fe574100780 2024-02-19T21:12:23.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.893+0000 7fe572ffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fe574100780 0x7fe5740fa380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45974/0 (socket says 172.21.15.163:45974) 2024-02-19T21:12:23.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.893+0000 7fe572ffd700 1 -- 172.21.15.163:0/1345152662 learned_addr learned my addr 172.21.15.163:0/1345152662 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:23.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.893+0000 7fe5737fe700 1 --1- 172.21.15.163:0/1345152662 >> v1:172.21.15.193:6789/0 conn(0x7fe5740fd980 0x7fe5740f9c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:39186/0 (socket says 172.21.15.163:39186) 2024-02-19T21:12:23.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 10 client.?.objecter ms_handle_connect 0x7fe5740fd980 2024-02-19T21:12:23.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:23.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 10 client.?.objecter ms_handle_connect 0x7fe574100780 2024-02-19T21:12:23.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:23.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 10 client.?.objecter ms_handle_connect 0x7fe5741042c0 2024-02-19T21:12:23.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:23.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617695405 0 0) 0x7fe5741186e0 con 0x7fe5740fd980 2024-02-19T21:12:23.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe55c003030 con 0x7fe5740fd980 2024-02-19T21:12:23.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263523497 0 0) 0x7fe5741198e0 con 0x7fe574100780 2024-02-19T21:12:23.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5741186e0 con 0x7fe574100780 2024-02-19T21:12:23.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965096831 0 0) 0x7fe5741174e0 con 0x7fe5741042c0 2024-02-19T21:12:23.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5741198e0 con 0x7fe5741042c0 2024-02-19T21:12:23.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 653894950 0 0) 0x7fe55c003030 con 0x7fe5740fd980 2024-02-19T21:12:23.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5741174e0 con 0x7fe5740fd980 2024-02-19T21:12:23.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3278991606 0 0) 0x7fe5741186e0 con 0x7fe574100780 2024-02-19T21:12:23.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe55c003030 con 0x7fe574100780 2024-02-19T21:12:23.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe5640018c0 con 0x7fe5740fd980 2024-02-19T21:12:23.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe568002050 con 0x7fe574100780 2024-02-19T21:12:23.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3504324185 0 0) 0x7fe5741198e0 con 0x7fe5741042c0 2024-02-19T21:12:23.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5741186e0 con 0x7fe5741042c0 2024-02-19T21:12:23.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe56c001cb0 con 0x7fe5741042c0 2024-02-19T21:12:23.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2568073430 0 0) 0x7fe55c003030 con 0x7fe574100780 2024-02-19T21:12:23.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 >> v1:172.21.15.193:6789/0 conn(0x7fe5740fd980 legacy=0x7fe5740f9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:23.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 >> v1:172.21.15.163:6789/0 conn(0x7fe5741042c0 legacy=0x7fe574115ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:23.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57411aae0 con 0x7fe574100780 2024-02-19T21:12:23.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe574119ad0 con 0x7fe574100780 2024-02-19T21:12:23.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe579e52700 10 client.50072.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:23.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe574119f40 con 0x7fe574100780 2024-02-19T21:12:23.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.894+0000 7fe579e52700 1 librados: init done 2024-02-19T21:12:23.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.895+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fe568002390 con 0x7fe574100780 2024-02-19T21:12:23.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.895+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe5680037b0 con 0x7fe574100780 2024-02-19T21:12:23.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.896+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe57405f660 con 0x7fe574100780 2024-02-19T21:12:23.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.896+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fe568003a30 con 0x7fe574100780 2024-02-19T21:12:23.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(507..507 src has 43..507) v4 ==== 8083+0+0 (unknown 4133792065 0 0) 0x7fe568078ee0 con 0x7fe574100780 2024-02-19T21:12:23.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 osd_map(507..507 src has 43..507) v4 2024-02-19T21:12:23.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe570ff9700 3 client.50072.objecter handle_osd_map got epochs [507,507] > 0 2024-02-19T21:12:23.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe570ff9700 3 client.50072.objecter handle_osd_map decoding full epoch 507 2024-02-19T21:12:23.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe570ff9700 20 client.50072.objecter dump_active .. 0 homeless 2024-02-19T21:12:23.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v507) v1 ==== 54+0+16 (unknown 3659117137 0 2420000192) 0x7fe56807b2e0 con 0x7fe574100780 2024-02-19T21:12:23.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe570ff9700 10 client.50072.objecter ms_handle_connect 0x7fe55c063d70 2024-02-19T21:12:23.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe579e52700 10 client.50072.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:23.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 1 2024-02-19T21:12:23.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574047540 osd=0 3 2024-02-19T21:12:23.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574047540 osd=0 3 2024-02-19T21:12:23.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574047540 osd=0 3 2024-02-19T21:12:23.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 1 2024-02-19T21:12:23.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe574047540 osd=0 3 2024-02-19T21:12:23.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 0 1 2024-02-19T21:12:23.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574047540 osd=0 4 2024-02-19T21:12:23.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe579e52700 10 client.50072.objecter _send_command 1 2024-02-19T21:12:23.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe574109fc0 con 0x7fe574118870 2024-02-19T21:12:23.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.897+0000 7fe570ff9700 10 client.50072.objecter ms_handle_connect 0x7fe574118870 2024-02-19T21:12:23.900 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:23.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.898+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe574109fc0 con 0x7fe574118870 2024-02-19T21:12:23.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.898+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:12:23.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.898+0000 7fe570ff9700 10 client.50072.objecter _finish_command 1 = system:0 2024-02-19T21:12:23.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.898+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe574047540 osd=0 4 2024-02-19T21:12:23.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.898+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 0 1 2024-02-19T21:12:23.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:23.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:23.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.914+0000 7fe579e52700 10 client.50072.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:23.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.914+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 2 2024-02-19T21:12:23.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.914+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574047540 osd=0 3 2024-02-19T21:12:23.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.914+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574047540 osd=0 3 2024-02-19T21:12:23.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.914+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574047540 osd=0 3 2024-02-19T21:12:23.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.914+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 2 2024-02-19T21:12:23.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.914+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe574047540 osd=0 3 2024-02-19T21:12:23.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.914+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 0 2 2024-02-19T21:12:23.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.914+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574047540 osd=0 4 2024-02-19T21:12:23.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.914+0000 7fe579e52700 10 client.50072.objecter _send_command 2 2024-02-19T21:12:23.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.914+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe5741195a0 con 0x7fe574118870 2024-02-19T21:12:23.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fe5741195a0 con 0x7fe574118870 2024-02-19T21:12:23.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:23.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe570ff9700 10 client.50072.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:23.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe574047540 osd=0 4 2024-02-19T21:12:23.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 0 2 2024-02-19T21:12:23.916 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:12:23.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe579e52700 10 client.50072.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:23.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 3 2024-02-19T21:12:23.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574119710 osd=1 3 2024-02-19T21:12:23.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574119710 osd=1 3 2024-02-19T21:12:23.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574119710 osd=1 3 2024-02-19T21:12:23.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 3 2024-02-19T21:12:23.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe574119710 osd=1 3 2024-02-19T21:12:23.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 1 3 2024-02-19T21:12:23.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574119710 osd=1 4 2024-02-19T21:12:23.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe579e52700 10 client.50072.objecter _send_command 3 2024-02-19T21:12:23.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe574059d10 con 0x7fe5740598f0 2024-02-19T21:12:23.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.916+0000 7fe570ff9700 10 client.50072.objecter ms_handle_connect 0x7fe5740598f0 2024-02-19T21:12:23.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.917+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe574059d10 con 0x7fe5740598f0 2024-02-19T21:12:23.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.917+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:12:23.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.917+0000 7fe570ff9700 10 client.50072.objecter _finish_command 3 = system:0 2024-02-19T21:12:23.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.917+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe574119710 osd=1 4 2024-02-19T21:12:23.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.917+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 1 3 2024-02-19T21:12:23.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.933+0000 7fe579e52700 10 client.50072.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:23.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.933+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 4 2024-02-19T21:12:23.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.933+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574119710 osd=1 3 2024-02-19T21:12:23.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.933+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574119710 osd=1 3 2024-02-19T21:12:23.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.933+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574119710 osd=1 3 2024-02-19T21:12:23.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.933+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 4 2024-02-19T21:12:23.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.933+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe574119710 osd=1 3 2024-02-19T21:12:23.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.933+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 1 4 2024-02-19T21:12:23.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.933+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574119710 osd=1 4 2024-02-19T21:12:23.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.933+0000 7fe579e52700 10 client.50072.objecter _send_command 4 2024-02-19T21:12:23.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.933+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe57405a0f0 con 0x7fe5740598f0 2024-02-19T21:12:23.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.934+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fe57405a0f0 con 0x7fe5740598f0 2024-02-19T21:12:23.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.934+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:23.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.934+0000 7fe570ff9700 10 client.50072.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:23.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.935+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe574119710 osd=1 4 2024-02-19T21:12:23.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.935+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 1 4 2024-02-19T21:12:23.934 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:12:23.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.935+0000 7fe579e52700 10 client.50072.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:23.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.935+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 5 2024-02-19T21:12:23.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.935+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574059360 osd=2 3 2024-02-19T21:12:23.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.935+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574059360 osd=2 3 2024-02-19T21:12:23.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.935+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574059360 osd=2 3 2024-02-19T21:12:23.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.935+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 5 2024-02-19T21:12:23.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.935+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe574059360 osd=2 3 2024-02-19T21:12:23.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.935+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 2 5 2024-02-19T21:12:23.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.935+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574059360 osd=2 4 2024-02-19T21:12:23.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.935+0000 7fe579e52700 10 client.50072.objecter _send_command 5 2024-02-19T21:12:23.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.935+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe574123490 con 0x7fe5740594d0 2024-02-19T21:12:23.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.935+0000 7fe570ff9700 10 client.50072.objecter ms_handle_connect 0x7fe5740594d0 2024-02-19T21:12:23.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.936+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe574123490 con 0x7fe5740594d0 2024-02-19T21:12:23.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.936+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:12:23.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.936+0000 7fe570ff9700 10 client.50072.objecter _finish_command 5 = system:0 2024-02-19T21:12:23.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.936+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe574059360 osd=2 4 2024-02-19T21:12:23.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.936+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 2 5 2024-02-19T21:12:23.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.950+0000 7fe579e52700 10 client.50072.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:23.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.950+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 6 2024-02-19T21:12:23.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.950+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574059360 osd=2 3 2024-02-19T21:12:23.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.950+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574059360 osd=2 3 2024-02-19T21:12:23.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.950+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574059360 osd=2 3 2024-02-19T21:12:23.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.950+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 6 2024-02-19T21:12:23.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.950+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe574059360 osd=2 3 2024-02-19T21:12:23.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.950+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 2 6 2024-02-19T21:12:23.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.950+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574059360 osd=2 4 2024-02-19T21:12:23.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.950+0000 7fe579e52700 10 client.50072.objecter _send_command 6 2024-02-19T21:12:23.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.950+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe574027d10 con 0x7fe5740594d0 2024-02-19T21:12:23.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.951+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fe574027d10 con 0x7fe5740594d0 2024-02-19T21:12:23.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.951+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:23.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.951+0000 7fe570ff9700 10 client.50072.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:23.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.951+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe574059360 osd=2 4 2024-02-19T21:12:23.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.951+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 2 6 2024-02-19T21:12:23.951 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:12:23.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe579e52700 10 client.50072.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:23.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 7 2024-02-19T21:12:23.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe5740f4150 osd=3 3 2024-02-19T21:12:23.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe5740f4150 osd=3 3 2024-02-19T21:12:23.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe5740f4150 osd=3 3 2024-02-19T21:12:23.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 7 2024-02-19T21:12:23.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe5740f4150 osd=3 3 2024-02-19T21:12:23.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 3 7 2024-02-19T21:12:23.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe5740f4150 osd=3 4 2024-02-19T21:12:23.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe579e52700 10 client.50072.objecter _send_command 7 2024-02-19T21:12:23.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5740f4840 con 0x7fe574124780 2024-02-19T21:12:23.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe570ff9700 10 client.50072.objecter ms_handle_connect 0x7fe574124780 2024-02-19T21:12:23.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe5740f4840 con 0x7fe574124780 2024-02-19T21:12:23.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:12:23.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe570ff9700 10 client.50072.objecter _finish_command 7 = system:0 2024-02-19T21:12:23.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe5740f4150 osd=3 4 2024-02-19T21:12:23.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.952+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 3 7 2024-02-19T21:12:23.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.967+0000 7fe579e52700 10 client.50072.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:23.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.967+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 8 2024-02-19T21:12:23.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.967+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe5740f4150 osd=3 3 2024-02-19T21:12:23.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.967+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe5740f4150 osd=3 3 2024-02-19T21:12:23.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.967+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe5740f4150 osd=3 3 2024-02-19T21:12:23.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.967+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 8 2024-02-19T21:12:23.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.967+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe5740f4150 osd=3 3 2024-02-19T21:12:23.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.967+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 3 8 2024-02-19T21:12:23.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.967+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe5740f4150 osd=3 4 2024-02-19T21:12:23.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.967+0000 7fe579e52700 10 client.50072.objecter _send_command 8 2024-02-19T21:12:23.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.967+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe5740f3bc0 con 0x7fe574124780 2024-02-19T21:12:23.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fe5740f3bc0 con 0x7fe574124780 2024-02-19T21:12:23.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:23.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe570ff9700 10 client.50072.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:23.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe5740f4150 osd=3 4 2024-02-19T21:12:23.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 3 8 2024-02-19T21:12:23.968 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:12:23.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe579e52700 10 client.50072.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:23.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 9 2024-02-19T21:12:23.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe5740f3bc0 osd=4 3 2024-02-19T21:12:23.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe5740f3bc0 osd=4 3 2024-02-19T21:12:23.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe5740f3bc0 osd=4 3 2024-02-19T21:12:23.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 9 2024-02-19T21:12:23.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe5740f3bc0 osd=4 3 2024-02-19T21:12:23.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 4 9 2024-02-19T21:12:23.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe5740f3bc0 osd=4 4 2024-02-19T21:12:23.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe579e52700 10 client.50072.objecter _send_command 9 2024-02-19T21:12:23.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.968+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe574124440 con 0x7fe5740f3d30 2024-02-19T21:12:23.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.969+0000 7fe570ff9700 10 client.50072.objecter ms_handle_connect 0x7fe5740f3d30 2024-02-19T21:12:23.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.970+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe574124440 con 0x7fe5740f3d30 2024-02-19T21:12:23.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.970+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:12:23.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.970+0000 7fe570ff9700 10 client.50072.objecter _finish_command 9 = system:0 2024-02-19T21:12:23.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.970+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe5740f3bc0 osd=4 4 2024-02-19T21:12:23.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.970+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 4 9 2024-02-19T21:12:23.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.984+0000 7fe579e52700 10 client.50072.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:23.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.984+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 10 2024-02-19T21:12:23.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.984+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe5740f3bc0 osd=4 3 2024-02-19T21:12:23.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.984+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe5740f3bc0 osd=4 3 2024-02-19T21:12:23.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.984+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe5740f3bc0 osd=4 3 2024-02-19T21:12:23.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.984+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 10 2024-02-19T21:12:23.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.984+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe5740f3bc0 osd=4 3 2024-02-19T21:12:23.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.984+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 4 10 2024-02-19T21:12:23.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.984+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe5740f3bc0 osd=4 4 2024-02-19T21:12:23.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.984+0000 7fe579e52700 10 client.50072.objecter _send_command 10 2024-02-19T21:12:23.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.984+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe574124440 con 0x7fe5740f3d30 2024-02-19T21:12:23.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fe574124440 con 0x7fe5740f3d30 2024-02-19T21:12:23.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:23.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe570ff9700 10 client.50072.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:23.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe5740f3bc0 osd=4 4 2024-02-19T21:12:23.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 4 10 2024-02-19T21:12:23.986 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:12:23.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe579e52700 10 client.50072.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:23.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 11 2024-02-19T21:12:23.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574124440 osd=5 3 2024-02-19T21:12:23.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574124440 osd=5 3 2024-02-19T21:12:23.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574124440 osd=5 3 2024-02-19T21:12:23.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 11 2024-02-19T21:12:23.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe574124440 osd=5 3 2024-02-19T21:12:23.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 5 11 2024-02-19T21:12:23.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574124440 osd=5 4 2024-02-19T21:12:23.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe579e52700 10 client.50072.objecter _send_command 11 2024-02-19T21:12:23.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.986+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe574130c30 con 0x7fe57412d050 2024-02-19T21:12:23.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.987+0000 7fe570ff9700 10 client.50072.objecter ms_handle_connect 0x7fe57412d050 2024-02-19T21:12:23.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.988+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe574130c30 con 0x7fe57412d050 2024-02-19T21:12:23.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.988+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:12:23.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.988+0000 7fe570ff9700 10 client.50072.objecter _finish_command 11 = system:0 2024-02-19T21:12:23.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.988+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe574124440 osd=5 4 2024-02-19T21:12:23.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:23.988+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 5 11 2024-02-19T21:12:24.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.002+0000 7fe579e52700 10 client.50072.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:24.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.002+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 12 2024-02-19T21:12:24.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.002+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574124440 osd=5 3 2024-02-19T21:12:24.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.002+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574124440 osd=5 3 2024-02-19T21:12:24.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.002+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574124440 osd=5 3 2024-02-19T21:12:24.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.002+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 12 2024-02-19T21:12:24.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.002+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe574124440 osd=5 3 2024-02-19T21:12:24.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.002+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 5 12 2024-02-19T21:12:24.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.002+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574124440 osd=5 4 2024-02-19T21:12:24.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.002+0000 7fe579e52700 10 client.50072.objecter _send_command 12 2024-02-19T21:12:24.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.002+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe5741323e0 con 0x7fe57412d050 2024-02-19T21:12:24.003 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:24.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fe5741323e0 con 0x7fe57412d050 2024-02-19T21:12:24.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:24.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe570ff9700 10 client.50072.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:24.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe574124440 osd=5 4 2024-02-19T21:12:24.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 5 12 2024-02-19T21:12:24.004 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:12:24.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe579e52700 10 client.50072.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:24.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 13 2024-02-19T21:12:24.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe5741323e0 osd=6 3 2024-02-19T21:12:24.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe5741323e0 osd=6 3 2024-02-19T21:12:24.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe5741323e0 osd=6 3 2024-02-19T21:12:24.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 13 2024-02-19T21:12:24.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe5741323e0 osd=6 3 2024-02-19T21:12:24.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 6 13 2024-02-19T21:12:24.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe5741323e0 osd=6 4 2024-02-19T21:12:24.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe579e52700 10 client.50072.objecter _send_command 13 2024-02-19T21:12:24.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.004+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe574133010 con 0x7fe574132a60 2024-02-19T21:12:24.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.005+0000 7fe570ff9700 10 client.50072.objecter ms_handle_connect 0x7fe574132a60 2024-02-19T21:12:24.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.006+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe574133010 con 0x7fe574132a60 2024-02-19T21:12:24.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.006+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:12:24.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.006+0000 7fe570ff9700 10 client.50072.objecter _finish_command 13 = system:0 2024-02-19T21:12:24.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.006+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe5741323e0 osd=6 4 2024-02-19T21:12:24.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.006+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 6 13 2024-02-19T21:12:24.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.020+0000 7fe579e52700 10 client.50072.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:24.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.020+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 14 2024-02-19T21:12:24.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.020+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe5741323e0 osd=6 3 2024-02-19T21:12:24.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.020+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe5741323e0 osd=6 3 2024-02-19T21:12:24.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.020+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe5741323e0 osd=6 3 2024-02-19T21:12:24.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.020+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 14 2024-02-19T21:12:24.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.020+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe5741323e0 osd=6 3 2024-02-19T21:12:24.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.020+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 6 14 2024-02-19T21:12:24.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.020+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe5741323e0 osd=6 4 2024-02-19T21:12:24.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.020+0000 7fe579e52700 10 client.50072.objecter _send_command 14 2024-02-19T21:12:24.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.020+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe574133010 con 0x7fe574132a60 2024-02-19T21:12:24.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.021+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fe574133010 con 0x7fe574132a60 2024-02-19T21:12:24.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:24.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe570ff9700 10 client.50072.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:24.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe5741323e0 osd=6 4 2024-02-19T21:12:24.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 6 14 2024-02-19T21:12:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:12:24.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe579e52700 10 client.50072.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:24.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 15 2024-02-19T21:12:24.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574133010 osd=7 3 2024-02-19T21:12:24.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574133010 osd=7 3 2024-02-19T21:12:24.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574133010 osd=7 3 2024-02-19T21:12:24.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 15 2024-02-19T21:12:24.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe574133010 osd=7 3 2024-02-19T21:12:24.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 7 15 2024-02-19T21:12:24.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574133010 osd=7 4 2024-02-19T21:12:24.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe579e52700 10 client.50072.objecter _send_command 15 2024-02-19T21:12:24.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe574133350 con 0x7fe57406b420 2024-02-19T21:12:24.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.022+0000 7fe570ff9700 10 client.50072.objecter ms_handle_connect 0x7fe57406b420 2024-02-19T21:12:24.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.023+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe574133350 con 0x7fe57406b420 2024-02-19T21:12:24.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.023+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:12:24.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.023+0000 7fe570ff9700 10 client.50072.objecter _finish_command 15 = system:0 2024-02-19T21:12:24.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.023+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe574133010 osd=7 4 2024-02-19T21:12:24.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.023+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 7 15 2024-02-19T21:12:24.038 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:24.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.038+0000 7fe579e52700 10 client.50072.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:24.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.038+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign -1 16 2024-02-19T21:12:24.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.038+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574133010 osd=7 3 2024-02-19T21:12:24.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.038+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574133010 osd=7 3 2024-02-19T21:12:24.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.038+0000 7fe579e52700 20 client.50072.objecter _get_session s=0x7fe574133010 osd=7 3 2024-02-19T21:12:24.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.038+0000 7fe579e52700 15 client.50072.objecter _session_command_op_remove -1 16 2024-02-19T21:12:24.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.038+0000 7fe579e52700 20 client.50072.objecter get_session s=0x7fe574133010 osd=7 3 2024-02-19T21:12:24.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.038+0000 7fe579e52700 15 client.50072.objecter _session_command_op_assign 7 16 2024-02-19T21:12:24.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.038+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574133010 osd=7 4 2024-02-19T21:12:24.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.038+0000 7fe579e52700 10 client.50072.objecter _send_command 16 2024-02-19T21:12:24.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.038+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fe574133350 con 0x7fe57406b420 2024-02-19T21:12:24.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.040+0000 7fe570ff9700 1 -- 172.21.15.163:0/1345152662 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fe574133350 con 0x7fe57406b420 2024-02-19T21:12:24.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.040+0000 7fe570ff9700 10 client.50072.objecter ms_dispatch 0x7fe574001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:24.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.040+0000 7fe570ff9700 10 client.50072.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:24.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.040+0000 7fe570ff9700 20 client.50072.objecter put_session s=0x7fe574133010 osd=7 4 2024-02-19T21:12:24.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.040+0000 7fe570ff9700 15 client.50072.objecter _session_command_op_remove 7 16 2024-02-19T21:12:24.040 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:12:24.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 10 librados: watch_flush enter 2024-02-19T21:12:24.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 10 librados: watch_flush exit 2024-02-19T21:12:24.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 10 client.50072.objecter close_session for osd.0 2024-02-19T21:12:24.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 >> v1:172.21.15.163:6801/1559418115 conn(0x7fe574118870 legacy=0x7fe574118c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:24.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574047540 osd=0 1 2024-02-19T21:12:24.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 10 client.50072.objecter close_session for osd.1 2024-02-19T21:12:24.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 >> v1:172.21.15.163:6805/3125047432 conn(0x7fe5740598f0 legacy=0x7fe5740273c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:24.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574119710 osd=1 1 2024-02-19T21:12:24.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 10 client.50072.objecter close_session for osd.2 2024-02-19T21:12:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 >> v1:172.21.15.163:6809/1563158487 conn(0x7fe5740594d0 legacy=0x7fe574121d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574059360 osd=2 1 2024-02-19T21:12:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 10 client.50072.objecter close_session for osd.3 2024-02-19T21:12:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 >> v1:172.21.15.163:6813/2112279769 conn(0x7fe574124780 legacy=0x7fe574126b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe5740f4150 osd=3 1 2024-02-19T21:12:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 10 client.50072.objecter close_session for osd.4 2024-02-19T21:12:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 >> v1:172.21.15.193:6800/3129716965 conn(0x7fe5740f3d30 legacy=0x7fe57412b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe5740f3bc0 osd=4 1 2024-02-19T21:12:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 10 client.50072.objecter close_session for osd.5 2024-02-19T21:12:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 >> v1:172.21.15.193:6804/805161991 conn(0x7fe57412d050 legacy=0x7fe57412f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe573fff700 1 -- 172.21.15.163:0/1345152662 reap_dead start 2024-02-19T21:12:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe573fff700 1 -- 172.21.15.163:0/1345152662 reap_dead start 2024-02-19T21:12:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe573fff700 1 -- 172.21.15.163:0/1345152662 reap_dead start 2024-02-19T21:12:24.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574124440 osd=5 1 2024-02-19T21:12:24.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 10 client.50072.objecter close_session for osd.6 2024-02-19T21:12:24.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 >> v1:172.21.15.193:6808/787604615 conn(0x7fe574132a60 legacy=0x7fe574130c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:24.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe5741323e0 osd=6 1 2024-02-19T21:12:24.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 10 client.50072.objecter close_session for osd.7 2024-02-19T21:12:24.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 >> v1:172.21.15.193:6812/157927432 conn(0x7fe57406b420 legacy=0x7fe57406b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:24.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 20 client.50072.objecter put_session s=0x7fe574133010 osd=7 1 2024-02-19T21:12:24.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 20 client.50072.objecter shutdown clearing up homeless session... 2024-02-19T21:12:24.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 10 client.50072.objecter successfully canceled tick 2024-02-19T21:12:24.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 >> v1:172.21.15.163:6800/1552514681 conn(0x7fe55c063d70 legacy=0x7fe55c066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:24.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 >> v1:172.21.15.163:6790/0 conn(0x7fe574100780 legacy=0x7fe5740fa380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:24.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 shutdown_connections 2024-02-19T21:12:24.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 >> 172.21.15.163:0/1345152662 conn(0x7fe5740f35a0 msgr2=0x7fe574107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:24.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 shutdown_connections 2024-02-19T21:12:24.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 1 -- 172.21.15.163:0/1345152662 wait complete. 2024-02-19T21:12:24.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:24.041+0000 7fe579e52700 1 librados: shutdown 2024-02-19T21:12:24.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:12:24.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:12:24.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:12:24.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:12:24.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:12:24.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:12:24.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:12:24.046 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:12:24.059 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:24.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:24.162 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:24.193 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:24.363 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:24.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:23.251278+0000 mgr.y (mgr.14550) 460 : cluster [DBG] pgmap v759: 238 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 205 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 1/1119 objects misplaced (0.089%); 1 B/s, 0 objects/s recovering 2024-02-19T21:12:24.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:23.898005+0000 mon.c (mon.2) 403 : audit [DBG] from='client.? v1:172.21.15.163:0/1345152662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:24.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:24.104847+0000 mon.a (mon.0) 2779 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:24.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:24.312356+0000 mon.a (mon.0) 2780 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:12:24.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:24.312430+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e508: 8 total, 8 up, 6 in 2024-02-19T21:12:24.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:24.315139+0000 mon.b (mon.1) 657 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:24.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:24.320821+0000 mon.a (mon.0) 2782 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:24.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:23.251278+0000 mgr.y (mgr.14550) 460 : cluster [DBG] pgmap v759: 238 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 205 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 1/1119 objects misplaced (0.089%); 1 B/s, 0 objects/s recovering 2024-02-19T21:12:24.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:23.898005+0000 mon.c (mon.2) 403 : audit [DBG] from='client.? v1:172.21.15.163:0/1345152662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:24.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:24.104847+0000 mon.a (mon.0) 2779 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:24.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:24.312356+0000 mon.a (mon.0) 2780 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:12:24.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:24.312430+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e508: 8 total, 8 up, 6 in 2024-02-19T21:12:24.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:24.315139+0000 mon.b (mon.1) 657 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:24.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:24.320821+0000 mon.a (mon.0) 2782 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:24.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:23.251278+0000 mgr.y (mgr.14550) 460 : cluster [DBG] pgmap v759: 238 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 205 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 1/1119 objects misplaced (0.089%); 1 B/s, 0 objects/s recovering 2024-02-19T21:12:24.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:23.898005+0000 mon.c (mon.2) 403 : audit [DBG] from='client.? v1:172.21.15.163:0/1345152662' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:24.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:24.104847+0000 mon.a (mon.0) 2779 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:24.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:24.312356+0000 mon.a (mon.0) 2780 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_tier","val": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:12:24.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:24.312430+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e508: 8 total, 8 up, 6 in 2024-02-19T21:12:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:24.315139+0000 mon.b (mon.1) 657 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:24.320821+0000 mon.a (mon.0) 2782 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:24.841 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:24.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:24.945 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:24.974 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:25.027 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:25.029 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:25.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:25.056 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:12:25.131 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:25.164 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:25.186 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:25.188 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:25.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:25.289 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:25.321 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:25.343 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:25.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:25.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:25.447 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:25.473 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:25.996 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:24.765451+0000 mon.a (mon.0) 2783 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:25.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:25.322136+0000 mon.a (mon.0) 2784 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-19T21:12:25.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:25.322195+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e509: 8 total, 8 up, 6 in 2024-02-19T21:12:25.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:25.323253+0000 mon.b (mon.1) 658 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:25.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:25.323570+0000 mon.a (mon.0) 2786 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:25.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:24.765451+0000 mon.a (mon.0) 2783 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:25.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:25.322136+0000 mon.a (mon.0) 2784 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-19T21:12:25.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:25.322195+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e509: 8 total, 8 up, 6 in 2024-02-19T21:12:25.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:25.323253+0000 mon.b (mon.1) 658 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:25.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:25.323570+0000 mon.a (mon.0) 2786 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:26.022 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:26.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:26.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:24.765451+0000 mon.a (mon.0) 2783 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:26.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:25.322136+0000 mon.a (mon.0) 2784 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-19T21:12:26.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:25.322195+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e509: 8 total, 8 up, 6 in 2024-02-19T21:12:26.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:25.323253+0000 mon.b (mon.1) 658 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:26.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:25.323570+0000 mon.a (mon.0) 2786 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:26.126 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:26.154 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:26.253 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:26.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:26.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:26.357 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:26.388 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:26.409 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:26.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:26.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:26.513 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:26.543 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:26.579 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:12:26.579 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:12:26.741 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:26.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:26.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:26.845 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:26.870 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:25.252186+0000 mgr.y (mgr.14550) 461 : cluster [DBG] pgmap v762: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1119 objects misplaced (0.089%) 2024-02-19T21:12:26.870 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:26.322509+0000 mon.a (mon.0) 2787 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:12:26.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:26.322588+0000 mon.a (mon.0) 2788 : cluster [DBG] osdmap e510: 8 total, 8 up, 6 in 2024-02-19T21:12:26.871 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:25.252186+0000 mgr.y (mgr.14550) 461 : cluster [DBG] pgmap v762: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1119 objects misplaced (0.089%) 2024-02-19T21:12:26.871 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:26.322509+0000 mon.a (mon.0) 2787 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:12:26.871 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:26.322588+0000 mon.a (mon.0) 2788 : cluster [DBG] osdmap e510: 8 total, 8 up, 6 in 2024-02-19T21:12:26.873 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:27.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:25.252186+0000 mgr.y (mgr.14550) 461 : cluster [DBG] pgmap v762: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1119 objects misplaced (0.089%) 2024-02-19T21:12:27.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:26.322509+0000 mon.a (mon.0) 2787 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:12:27.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:26.322588+0000 mon.a (mon.0) 2788 : cluster [DBG] osdmap e510: 8 total, 8 up, 6 in 2024-02-19T21:12:27.121 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:27.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:27.130 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-19T21:12:27.225 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:27.257 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:27.279 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:27.281 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:27.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:27.383 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:27.412 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:27.432 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:27.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:27.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:27.536 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:27.563 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:27.582 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:27.589 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:27.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:27.691 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:27.717 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:27.764 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:12:28.070 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:12:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:12:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:12:28.074 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:28.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:28.178 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:28.204 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:28.277 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:28.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:28.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:28.305 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:28.381 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:28.408 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:28.445 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:28.447 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:28.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:28.549 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:28.549 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:28.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.558+0000 7f8ceff13700 1 -- 172.21.15.163:0/4106887630 >> v1:172.21.15.163:6790/0 conn(0x7f8ce8100780 legacy=0x7f8ce8102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:28.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.558+0000 7f8ceff13700 1 -- 172.21.15.163:0/4106887630 shutdown_connections 2024-02-19T21:12:28.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.558+0000 7f8ceff13700 1 -- 172.21.15.163:0/4106887630 >> 172.21.15.163:0/4106887630 conn(0x7f8ce80f35a0 msgr2=0x7f8ce80f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:28.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.558+0000 7f8ceff13700 1 -- 172.21.15.163:0/4106887630 shutdown_connections 2024-02-19T21:12:28.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.558+0000 7f8ceff13700 1 -- 172.21.15.163:0/4106887630 wait complete. 2024-02-19T21:12:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.558+0000 7f8ceff13700 1 librados: starting msgr at 2024-02-19T21:12:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.558+0000 7f8ceff13700 1 librados: starting objecter 2024-02-19T21:12:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.558+0000 7f8ceff13700 1 Processor -- start 2024-02-19T21:12:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8ceff13700 1 -- start start 2024-02-19T21:12:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8ceff13700 1 librados: setting wanted keys 2024-02-19T21:12:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8ceff13700 1 librados: calling monclient init 2024-02-19T21:12:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8ceff13700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8ce81172c0 con 0x7f8ce80fd980 2024-02-19T21:12:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8ceff13700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8ce81184a0 con 0x7f8ce81042c0 2024-02-19T21:12:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8ceff13700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8ce81196a0 con 0x7f8ce8100780 2024-02-19T21:12:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8ced4ae700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f8ce8100780 0x7f8ce80fa070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:49708/0 (socket says 172.21.15.163:49708) 2024-02-19T21:12:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8ced4ae700 1 -- 172.21.15.163:0/1182976971 learned_addr learned my addr 172.21.15.163:0/1182976971 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:28.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8cdeffd700 10 client.?.objecter ms_handle_connect 0x7f8ce8100780 2024-02-19T21:12:28.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8cdeffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:28.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8cdeffd700 10 client.?.objecter ms_handle_connect 0x7f8ce80fd980 2024-02-19T21:12:28.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8cdeffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:28.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8cdeffd700 10 client.?.objecter ms_handle_connect 0x7f8ce81042c0 2024-02-19T21:12:28.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8cdeffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:28.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008071431 0 0) 0x7f8ce81172c0 con 0x7f8ce80fd980 2024-02-19T21:12:28.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd40033f0 con 0x7f8ce80fd980 2024-02-19T21:12:28.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792839533 0 0) 0x7f8ce81196a0 con 0x7f8ce8100780 2024-02-19T21:12:28.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ce81172c0 con 0x7f8ce8100780 2024-02-19T21:12:28.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257771425 0 0) 0x7f8ce81184a0 con 0x7f8ce81042c0 2024-02-19T21:12:28.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.559+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ce81196a0 con 0x7f8ce81042c0 2024-02-19T21:12:28.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 907341549 0 0) 0x7f8cd40033f0 con 0x7f8ce80fd980 2024-02-19T21:12:28.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ce81184a0 con 0x7f8ce80fd980 2024-02-19T21:12:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8ce4001900 con 0x7f8ce80fd980 2024-02-19T21:12:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2547807331 0 0) 0x7f8ce81172c0 con 0x7f8ce8100780 2024-02-19T21:12:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cd40033f0 con 0x7f8ce8100780 2024-02-19T21:12:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3294749662 0 0) 0x7f8ce81196a0 con 0x7f8ce81042c0 2024-02-19T21:12:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ce81172c0 con 0x7f8ce81042c0 2024-02-19T21:12:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8cd8001f90 con 0x7f8ce8100780 2024-02-19T21:12:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8ce0001bc0 con 0x7f8ce81042c0 2024-02-19T21:12:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2827581886 0 0) 0x7f8ce81184a0 con 0x7f8ce80fd980 2024-02-19T21:12:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 >> v1:172.21.15.163:6790/0 conn(0x7f8ce8100780 legacy=0x7f8ce80fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 >> v1:172.21.15.193:6789/0 conn(0x7f8ce81042c0 legacy=0x7f8ce8115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce811a8a0 con 0x7f8ce80fd980 2024-02-19T21:12:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8ceff13700 1 -- 172.21.15.163:0/1182976971 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce81174b0 con 0x7f8ce80fd980 2024-02-19T21:12:28.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8ceff13700 10 client.23355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8ceff13700 1 -- 172.21.15.163:0/1182976971 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce81179a0 con 0x7f8ce80fd980 2024-02-19T21:12:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8ceff13700 1 librados: init done 2024-02-19T21:12:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8ce4002340 con 0x7f8ce80fd980 2024-02-19T21:12:28.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.560+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8ce4003420 con 0x7f8ce80fd980 2024-02-19T21:12:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.561+0000 7f8ceff13700 1 -- 172.21.15.163:0/1182976971 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ccc004f20 con 0x7f8ce80fd980 2024-02-19T21:12:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.561+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f8ce4003580 con 0x7f8ce80fd980 2024-02-19T21:12:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.562+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(512..512 src has 43..512) v4 ==== 8134+0+0 (unknown 2153900502 0 0) 0x7f8ce4076570 con 0x7f8ce80fd980 2024-02-19T21:12:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.562+0000 7f8cdeffd700 10 client.23355.objecter ms_dispatch 0x7f8ce8001550 osd_map(512..512 src has 43..512) v4 2024-02-19T21:12:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.562+0000 7f8cdeffd700 3 client.23355.objecter handle_osd_map got epochs [512,512] > 0 2024-02-19T21:12:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.562+0000 7f8cdeffd700 3 client.23355.objecter handle_osd_map decoding full epoch 512 2024-02-19T21:12:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.562+0000 7f8cdeffd700 20 client.23355.objecter dump_active .. 0 homeless 2024-02-19T21:12:28.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.562+0000 7f8cdeffd700 10 client.23355.objecter ms_handle_connect 0x7f8cd4063e00 2024-02-19T21:12:28.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.567+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8ce4049920 con 0x7f8ce80fd980 2024-02-19T21:12:28.577 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:27.252567+0000 mgr.y (mgr.14550) 462 : cluster [DBG] pgmap v765: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1119 objects misplaced (0.089%) 2024-02-19T21:12:28.577 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:27.326207+0000 mon.a (mon.0) 2789 : cluster [DBG] osdmap e511: 8 total, 8 up, 6 in 2024-02-19T21:12:28.580 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:28.599 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:28.602 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:28.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:28.704 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:28.732 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:28.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:27.252567+0000 mgr.y (mgr.14550) 462 : cluster [DBG] pgmap v765: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1119 objects misplaced (0.089%) 2024-02-19T21:12:28.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:27.326207+0000 mon.a (mon.0) 2789 : cluster [DBG] osdmap e511: 8 total, 8 up, 6 in 2024-02-19T21:12:28.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:27.252567+0000 mgr.y (mgr.14550) 462 : cluster [DBG] pgmap v765: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1119 objects misplaced (0.089%) 2024-02-19T21:12:28.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:27.326207+0000 mon.a (mon.0) 2789 : cluster [DBG] osdmap e511: 8 total, 8 up, 6 in 2024-02-19T21:12:28.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:28.751+0000 7f8ceff13700 1 -- 172.21.15.163:0/1182976971 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7f8ccc0051d0 con 0x7f8ce80fd980 2024-02-19T21:12:28.751 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:28.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:28.855 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:28.882 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:29.005 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:29.007 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:29.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:29.108 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:29.136 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:29.158 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:29.161 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:29.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:29.262 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:29.290 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:29.310 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:29.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:29.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:29.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:29.396+0000 7f8cdeffd700 1 -- 172.21.15.163:0/1182976971 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v513) v1 ==== 113+0+0 (unknown 1018670794 0 0) 0x7f8ce4047d30 con 0x7f8ce80fd980 2024-02-19T21:12:29.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:29.402+0000 7f8ceff13700 10 librados: watch_flush enter 2024-02-19T21:12:29.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:29.403+0000 7f8ceff13700 10 librados: watch_flush exit 2024-02-19T21:12:29.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:29.403+0000 7f8ceff13700 20 client.23355.objecter shutdown clearing up homeless session... 2024-02-19T21:12:29.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:29.403+0000 7f8ceff13700 10 client.23355.objecter successfully canceled tick 2024-02-19T21:12:29.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:29.404+0000 7f8ceff13700 1 -- 172.21.15.163:0/1182976971 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8cd4063e00 legacy=0x7f8cd40662c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:29.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:29.404+0000 7f8ceff13700 1 -- 172.21.15.163:0/1182976971 >> v1:172.21.15.163:6789/0 conn(0x7f8ce80fd980 legacy=0x7f8ce80f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:29.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:29.407+0000 7f8ceff13700 1 -- 172.21.15.163:0/1182976971 shutdown_connections 2024-02-19T21:12:29.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:29.407+0000 7f8ceff13700 1 -- 172.21.15.163:0/1182976971 >> 172.21.15.163:0/1182976971 conn(0x7f8ce80f35a0 msgr2=0x7f8ce8107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:29.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:29.407+0000 7f8ceff13700 1 -- 172.21.15.163:0/1182976971 shutdown_connections 2024-02-19T21:12:29.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:29.407+0000 7f8ceff13700 1 -- 172.21.15.163:0/1182976971 wait complete. 2024-02-19T21:12:29.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:29.407+0000 7f8ceff13700 1 librados: shutdown 2024-02-19T21:12:29.407 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.2 to 1 (10000) 2024-02-19T21:12:29.415 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:29.443 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:29.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:28.168144+0000 osd.5 (osd.5) 212 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:29.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:28.168406+0000 osd.5 (osd.5) 213 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:29.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:28.333938+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e512: 8 total, 8 up, 6 in 2024-02-19T21:12:29.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:28.368459+0000 mon.b (mon.1) 659 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:29.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:28.368857+0000 mon.a (mon.0) 2791 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:29.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:28.369312+0000 mon.b (mon.1) 660 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-81"}]: dispatch 2024-02-19T21:12:29.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:28.369517+0000 mon.a (mon.0) 2792 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-81"}]: dispatch 2024-02-19T21:12:29.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:28.753231+0000 mon.a (mon.0) 2793 : audit [INF] from='client.? v1:172.21.15.163:0/1182976971' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:12:29.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.063071+0000 mon.b (mon.1) 661 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:12:29.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:28.168144+0000 osd.5 (osd.5) 212 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:29.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:28.168406+0000 osd.5 (osd.5) 213 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:29.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:28.333938+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e512: 8 total, 8 up, 6 in 2024-02-19T21:12:29.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:28.368459+0000 mon.b (mon.1) 659 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:29.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:28.368857+0000 mon.a (mon.0) 2791 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:29.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:28.369312+0000 mon.b (mon.1) 660 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-81"}]: dispatch 2024-02-19T21:12:29.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:28.369517+0000 mon.a (mon.0) 2792 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-81"}]: dispatch 2024-02-19T21:12:29.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:28.753231+0000 mon.a (mon.0) 2793 : audit [INF] from='client.? v1:172.21.15.163:0/1182976971' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:12:29.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.063071+0000 mon.b (mon.1) 661 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:12:29.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:28.168144+0000 osd.5 (osd.5) 212 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:12:29.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:28.168406+0000 osd.5 (osd.5) 213 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:12:29.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:28.333938+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e512: 8 total, 8 up, 6 in 2024-02-19T21:12:29.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:28.368459+0000 mon.b (mon.1) 659 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:29.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:28.368857+0000 mon.a (mon.0) 2791 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:29.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:28.369312+0000 mon.b (mon.1) 660 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-81"}]: dispatch 2024-02-19T21:12:29.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:28.369517+0000 mon.a (mon.0) 2792 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-81"}]: dispatch 2024-02-19T21:12:29.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:28.753231+0000 mon.a (mon.0) 2793 : audit [INF] from='client.? v1:172.21.15.163:0/1182976971' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:12:29.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.063071+0000 mon.b (mon.1) 661 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:12:29.896 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:29.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:30.000 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:12:30.028 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:30.047 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:12:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:30.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:12:30.151 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:30.179 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:30.277 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:30.279 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:30.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:30.381 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:30.414 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:30.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:29.253121+0000 mgr.y (mgr.14550) 463 : cluster [DBG] pgmap v768: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1/1119 objects misplaced (0.089%) 2024-02-19T21:12:30.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.366006+0000 mon.b (mon.1) 662 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.387279+0000 mon.a (mon.0) 2794 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.394502+0000 mon.a (mon.0) 2795 : audit [INF] from='client.? v1:172.21.15.163:0/1182976971' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-19T21:12:30.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:29.394552+0000 mon.a (mon.0) 2796 : cluster [DBG] osdmap e513: 8 total, 8 up, 6 in 2024-02-19T21:12:30.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.399221+0000 mon.b (mon.1) 663 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.452811+0000 mon.b (mon.1) 664 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.646787+0000 mon.a (mon.0) 2797 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.647000+0000 mon.a (mon.0) 2798 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.647833+0000 mon.b (mon.1) 665 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.647974+0000 mon.b (mon.1) 666 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.648062+0000 mon.a (mon.0) 2799 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.648356+0000 mon.a (mon.0) 2800 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.648705+0000 mon.b (mon.1) 667 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:29.253121+0000 mgr.y (mgr.14550) 463 : cluster [DBG] pgmap v768: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1/1119 objects misplaced (0.089%) 2024-02-19T21:12:30.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.366006+0000 mon.b (mon.1) 662 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.387279+0000 mon.a (mon.0) 2794 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.394502+0000 mon.a (mon.0) 2795 : audit [INF] from='client.? v1:172.21.15.163:0/1182976971' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-19T21:12:30.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:29.394552+0000 mon.a (mon.0) 2796 : cluster [DBG] osdmap e513: 8 total, 8 up, 6 in 2024-02-19T21:12:30.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.399221+0000 mon.b (mon.1) 663 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.452811+0000 mon.b (mon.1) 664 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.646787+0000 mon.a (mon.0) 2797 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.647000+0000 mon.a (mon.0) 2798 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.647833+0000 mon.b (mon.1) 665 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.647974+0000 mon.b (mon.1) 666 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.648062+0000 mon.a (mon.0) 2799 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.648356+0000 mon.a (mon.0) 2800 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.648705+0000 mon.b (mon.1) 667 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.648891+0000 mon.a (mon.0) 2801 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.649118+0000 mon.b (mon.1) 668 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.649315+0000 mon.a (mon.0) 2802 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:12:29.649465+0000 mgr.y (mgr.14550) 464 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 3256M 2024-02-19T21:12:30.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.650022+0000 mon.b (mon.1) 669 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.650212+0000 mon.a (mon.0) 2803 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.654309+0000 mon.a (mon.0) 2804 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:30.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cephadm 2024-02-19T21:12:29.655087+0000 mgr.y (mgr.14550) 465 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 3256M 2024-02-19T21:12:30.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.658459+0000 mon.a (mon.0) 2805 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:30.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.662245+0000 mon.b (mon.1) 670 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:12:30.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.662784+0000 mon.b (mon.1) 671 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:12:30.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.667002+0000 mon.a (mon.0) 2806 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:30.501 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:29.776165+0000 mon.a (mon.0) 2807 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:30.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.648891+0000 mon.a (mon.0) 2801 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.649118+0000 mon.b (mon.1) 668 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.649315+0000 mon.a (mon.0) 2802 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:12:29.649465+0000 mgr.y (mgr.14550) 464 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 3256M 2024-02-19T21:12:30.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.650022+0000 mon.b (mon.1) 669 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.650212+0000 mon.a (mon.0) 2803 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.654309+0000 mon.a (mon.0) 2804 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:30.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cephadm 2024-02-19T21:12:29.655087+0000 mgr.y (mgr.14550) 465 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 3256M 2024-02-19T21:12:30.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.658459+0000 mon.a (mon.0) 2805 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:30.501 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.662245+0000 mon.b (mon.1) 670 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:12:30.502 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.662784+0000 mon.b (mon.1) 671 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:12:30.502 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.667002+0000 mon.a (mon.0) 2806 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:30.502 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:29.776165+0000 mon.a (mon.0) 2807 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:30.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:12:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:29.253121+0000 mgr.y (mgr.14550) 463 : cluster [DBG] pgmap v768: 238 pgs: 1 active+remapped+backfill_toofull, 237 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1/1119 objects misplaced (0.089%) 2024-02-19T21:12:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.366006+0000 mon.b (mon.1) 662 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.387279+0000 mon.a (mon.0) 2794 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.394502+0000 mon.a (mon.0) 2795 : audit [INF] from='client.? v1:172.21.15.163:0/1182976971' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-19T21:12:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:29.394552+0000 mon.a (mon.0) 2796 : cluster [DBG] osdmap e513: 8 total, 8 up, 6 in 2024-02-19T21:12:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.399221+0000 mon.b (mon.1) 663 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.452811+0000 mon.b (mon.1) 664 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.646787+0000 mon.a (mon.0) 2797 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.647000+0000 mon.a (mon.0) 2798 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.647833+0000 mon.b (mon.1) 665 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.647974+0000 mon.b (mon.1) 666 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.648062+0000 mon.a (mon.0) 2799 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.648356+0000 mon.a (mon.0) 2800 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.648705+0000 mon.b (mon.1) 667 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.648891+0000 mon.a (mon.0) 2801 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.649118+0000 mon.b (mon.1) 668 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.649315+0000 mon.a (mon.0) 2802 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:12:29.649465+0000 mgr.y (mgr.14550) 464 : cephadm [INF] Adjusting osd_memory_target on smithi193 to 3256M 2024-02-19T21:12:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.650022+0000 mon.b (mon.1) 669 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.650212+0000 mon.a (mon.0) 2803 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T21:12:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.654309+0000 mon.a (mon.0) 2804 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cephadm 2024-02-19T21:12:29.655087+0000 mgr.y (mgr.14550) 465 : cephadm [INF] Adjusting osd_memory_target on smithi163 to 3256M 2024-02-19T21:12:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.658459+0000 mon.a (mon.0) 2805 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.662245+0000 mon.b (mon.1) 670 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:12:30.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.662784+0000 mon.b (mon.1) 671 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:12:30.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.667002+0000 mon.a (mon.0) 2806 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:30.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:29.776165+0000 mon.a (mon.0) 2807 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:30.856 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:30.862 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 5 1 2024-02-19T21:12:30.862 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:30.865 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:30.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:30.966 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:30.996 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:31.016 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:31.019 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:31.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:31.120 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:31.147 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:31.219 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:31.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:31.323 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:31.355 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:31.376 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:31.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:31.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:31.480 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:31.508 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:31.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.651+0000 7f394ec16700 1 -- 172.21.15.163:0/897751319 >> v1:172.21.15.163:6790/0 conn(0x7f39481042b0 legacy=0x7f3948106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:31.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.651+0000 7f394ec16700 1 -- 172.21.15.163:0/897751319 shutdown_connections 2024-02-19T21:12:31.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.651+0000 7f394ec16700 1 -- 172.21.15.163:0/897751319 >> 172.21.15.163:0/897751319 conn(0x7f39480f2470 msgr2=0x7f39480f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:31.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.651+0000 7f394ec16700 1 -- 172.21.15.163:0/897751319 shutdown_connections 2024-02-19T21:12:31.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.651+0000 7f394ec16700 1 -- 172.21.15.163:0/897751319 wait complete. 2024-02-19T21:12:31.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.651+0000 7f394ec16700 1 librados: starting msgr at 2024-02-19T21:12:31.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.651+0000 7f394ec16700 1 librados: starting objecter 2024-02-19T21:12:31.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.651+0000 7f394ec16700 1 Processor -- start 2024-02-19T21:12:31.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.651+0000 7f394ec16700 1 -- start start 2024-02-19T21:12:31.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.651+0000 7f394ec16700 1 librados: setting wanted keys 2024-02-19T21:12:31.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.651+0000 7f394ec16700 1 librados: calling monclient init 2024-02-19T21:12:31.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f394ec16700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3948102ca0 con 0x7f39481042b0 2024-02-19T21:12:31.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f394ec16700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3948102e70 con 0x7f39480f8920 2024-02-19T21:12:31.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f394ec16700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f394811bb70 con 0x7f39480f94d0 2024-02-19T21:12:31.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3947fff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f39480f94d0 0x7f3948114d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:49726/0 (socket says 172.21.15.163:49726) 2024-02-19T21:12:31.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3947fff700 1 -- 172.21.15.163:0/4055320434 learned_addr learned my addr 172.21.15.163:0/4055320434 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:31.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 10 client.?.objecter ms_handle_connect 0x7f39480f94d0 2024-02-19T21:12:31.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:31.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 10 client.?.objecter ms_handle_connect 0x7f39481042b0 2024-02-19T21:12:31.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:31.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 10 client.?.objecter ms_handle_connect 0x7f39480f8920 2024-02-19T21:12:31.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:31.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711650838 0 0) 0x7f394811bb70 con 0x7f39480f94d0 2024-02-19T21:12:31.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3938002220 con 0x7f39480f94d0 2024-02-19T21:12:31.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227593729 0 0) 0x7f3948102e70 con 0x7f39480f8920 2024-02-19T21:12:31.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f394811bb70 con 0x7f39480f8920 2024-02-19T21:12:31.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1791332282 0 0) 0x7f3938002220 con 0x7f39480f94d0 2024-02-19T21:12:31.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3948102e70 con 0x7f39480f94d0 2024-02-19T21:12:31.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3934001850 con 0x7f39480f94d0 2024-02-19T21:12:31.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2903437740 0 0) 0x7f394811bb70 con 0x7f39480f8920 2024-02-19T21:12:31.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3938002220 con 0x7f39480f8920 2024-02-19T21:12:31.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.652+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f393c001840 con 0x7f39480f8920 2024-02-19T21:12:31.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.653+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2177579713 0 0) 0x7f3948102e70 con 0x7f39480f94d0 2024-02-19T21:12:31.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.653+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 >> v1:172.21.15.193:6789/0 conn(0x7f39480f8920 legacy=0x7f3948102500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:31.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.653+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 >> v1:172.21.15.163:6789/0 conn(0x7f39481042b0 legacy=0x7f3948118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:31.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.653+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394811cd50 con 0x7f39480f94d0 2024-02-19T21:12:31.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.653+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394811bd60 con 0x7f39480f94d0 2024-02-19T21:12:31.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.653+0000 7f394ec16700 10 client.50084.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:31.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.653+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f394811c1d0 con 0x7f39480f94d0 2024-02-19T21:12:31.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.653+0000 7f394ec16700 1 librados: init done 2024-02-19T21:12:31.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.654+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f394805f700 con 0x7f39480f94d0 2024-02-19T21:12:31.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.654+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3934002340 con 0x7f39480f94d0 2024-02-19T21:12:31.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.654+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3934005a00 con 0x7f39480f94d0 2024-02-19T21:12:31.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.655+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f3934001ea0 con 0x7f39480f94d0 2024-02-19T21:12:31.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(515..515 src has 43..515) v4 ==== 8067+0+0 (unknown 3890369222 0 0) 0x7f3934078d00 con 0x7f39480f94d0 2024-02-19T21:12:31.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 osd_map(515..515 src has 43..515) v4 2024-02-19T21:12:31.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f3945ffb700 3 client.50084.objecter handle_osd_map got epochs [515,515] > 0 2024-02-19T21:12:31.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f3945ffb700 3 client.50084.objecter handle_osd_map decoding full epoch 515 2024-02-19T21:12:31.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f3945ffb700 20 client.50084.objecter dump_active .. 0 homeless 2024-02-19T21:12:31.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v515) v1 ==== 54+0+16 (unknown 4047246080 0 2420000192) 0x7f3934079120 con 0x7f39480f94d0 2024-02-19T21:12:31.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f3945ffb700 10 client.50084.objecter ms_handle_connect 0x7f3938063c00 2024-02-19T21:12:31.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f394ec16700 10 client.50084.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:31.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 1 2024-02-19T21:12:31.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948047540 osd=0 3 2024-02-19T21:12:31.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948047540 osd=0 3 2024-02-19T21:12:31.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948047540 osd=0 3 2024-02-19T21:12:31.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 1 2024-02-19T21:12:31.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f3948047540 osd=0 3 2024-02-19T21:12:31.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 0 1 2024-02-19T21:12:31.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948047540 osd=0 4 2024-02-19T21:12:31.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f394ec16700 10 client.50084.objecter _send_command 1 2024-02-19T21:12:31.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3948061360 con 0x7f39480f9930 2024-02-19T21:12:31.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.656+0000 7f3945ffb700 10 client.50084.objecter ms_handle_connect 0x7f39480f9930 2024-02-19T21:12:31.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.657+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3948061360 con 0x7f39480f9930 2024-02-19T21:12:31.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.657+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:12:31.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.657+0000 7f3945ffb700 10 client.50084.objecter _finish_command 1 = system:0 2024-02-19T21:12:31.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.657+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f3948047540 osd=0 4 2024-02-19T21:12:31.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.657+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 0 1 2024-02-19T21:12:31.660 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:31.662 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:30.399588+0000 mon.a (mon.0) 2808 : cluster [DBG] osdmap e514: 8 total, 8 up, 6 in 2024-02-19T21:12:31.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:30.408378+0000 mon.b (mon.1) 672 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:31.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:30.409841+0000 mon.a (mon.0) 2809 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:31.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:30.521596+0000 mgr.y (mgr.14550) 466 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:31.663 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:30.399588+0000 mon.a (mon.0) 2808 : cluster [DBG] osdmap e514: 8 total, 8 up, 6 in 2024-02-19T21:12:31.663 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:30.408378+0000 mon.b (mon.1) 672 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:31.663 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:30.409841+0000 mon.a (mon.0) 2809 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:31.663 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:30.521596+0000 mgr.y (mgr.14550) 466 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:31.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:31.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.674+0000 7f394ec16700 10 client.50084.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:31.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.674+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 2 2024-02-19T21:12:31.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.674+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948047540 osd=0 3 2024-02-19T21:12:31.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.674+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948047540 osd=0 3 2024-02-19T21:12:31.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.674+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948047540 osd=0 3 2024-02-19T21:12:31.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.674+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 2 2024-02-19T21:12:31.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.674+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f3948047540 osd=0 3 2024-02-19T21:12:31.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.674+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 0 2 2024-02-19T21:12:31.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.674+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948047540 osd=0 4 2024-02-19T21:12:31.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.674+0000 7f394ec16700 10 client.50084.objecter _send_command 2 2024-02-19T21:12:31.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.674+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3948109fb0 con 0x7f39480f9930 2024-02-19T21:12:31.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3948109fb0 con 0x7f39480f9930 2024-02-19T21:12:31.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:31.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f3945ffb700 10 client.50084.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f3948047540 osd=0 4 2024-02-19T21:12:31.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 0 2 2024-02-19T21:12:31.676 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:12:31.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f394ec16700 10 client.50084.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:31.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 3 2024-02-19T21:12:31.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f39480fa420 osd=1 3 2024-02-19T21:12:31.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f39480fa420 osd=1 3 2024-02-19T21:12:31.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f39480fa420 osd=1 3 2024-02-19T21:12:31.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 3 2024-02-19T21:12:31.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f39480fa420 osd=1 3 2024-02-19T21:12:31.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 1 3 2024-02-19T21:12:31.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f39480fa420 osd=1 4 2024-02-19T21:12:31.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f394ec16700 10 client.50084.objecter _send_command 3 2024-02-19T21:12:31.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.676+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3948120ab0 con 0x7f394811cfa0 2024-02-19T21:12:31.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.677+0000 7f3945ffb700 10 client.50084.objecter ms_handle_connect 0x7f394811cfa0 2024-02-19T21:12:31.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.677+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3948120ab0 con 0x7f394811cfa0 2024-02-19T21:12:31.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.677+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:12:31.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.677+0000 7f3945ffb700 10 client.50084.objecter _finish_command 3 = system:0 2024-02-19T21:12:31.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.677+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f39480fa420 osd=1 4 2024-02-19T21:12:31.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.677+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 1 3 2024-02-19T21:12:31.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.692+0000 7f394ec16700 10 client.50084.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:31.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.692+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 4 2024-02-19T21:12:31.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.692+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f39480fa420 osd=1 3 2024-02-19T21:12:31.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.692+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f39480fa420 osd=1 3 2024-02-19T21:12:31.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.692+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f39480fa420 osd=1 3 2024-02-19T21:12:31.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.692+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 4 2024-02-19T21:12:31.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.692+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f39480fa420 osd=1 3 2024-02-19T21:12:31.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.692+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 1 4 2024-02-19T21:12:31.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.692+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f39480fa420 osd=1 4 2024-02-19T21:12:31.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.692+0000 7f394ec16700 10 client.50084.objecter _send_command 4 2024-02-19T21:12:31.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.692+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f394811c700 con 0x7f394811cfa0 2024-02-19T21:12:31.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f394811c700 con 0x7f394811cfa0 2024-02-19T21:12:31.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:31.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f3945ffb700 10 client.50084.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f39480fa420 osd=1 4 2024-02-19T21:12:31.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 1 4 2024-02-19T21:12:31.697 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:12:31.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f394ec16700 10 client.50084.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:31.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 5 2024-02-19T21:12:31.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f394811cb10 osd=2 3 2024-02-19T21:12:31.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f394811cb10 osd=2 3 2024-02-19T21:12:31.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f394811cb10 osd=2 3 2024-02-19T21:12:31.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 5 2024-02-19T21:12:31.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f394811cb10 osd=2 3 2024-02-19T21:12:31.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 2 5 2024-02-19T21:12:31.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f394811cb10 osd=2 4 2024-02-19T21:12:31.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f394ec16700 10 client.50084.objecter _send_command 5 2024-02-19T21:12:31.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.693+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f39481254d0 con 0x7f3948059e00 2024-02-19T21:12:31.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.694+0000 7f3945ffb700 10 client.50084.objecter ms_handle_connect 0x7f3948059e00 2024-02-19T21:12:31.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.694+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f39481254d0 con 0x7f3948059e00 2024-02-19T21:12:31.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.694+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:12:31.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.694+0000 7f3945ffb700 10 client.50084.objecter _finish_command 5 = system:0 2024-02-19T21:12:31.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.694+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f394811cb10 osd=2 4 2024-02-19T21:12:31.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.694+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 2 5 2024-02-19T21:12:31.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.709+0000 7f394ec16700 10 client.50084.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:31.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.709+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 6 2024-02-19T21:12:31.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.709+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f394811cb10 osd=2 3 2024-02-19T21:12:31.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.709+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f394811cb10 osd=2 3 2024-02-19T21:12:31.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.709+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f394811cb10 osd=2 3 2024-02-19T21:12:31.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.709+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 6 2024-02-19T21:12:31.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.709+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f394811cb10 osd=2 3 2024-02-19T21:12:31.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.709+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 2 6 2024-02-19T21:12:31.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.709+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f394811cb10 osd=2 4 2024-02-19T21:12:31.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.709+0000 7f394ec16700 10 client.50084.objecter _send_command 6 2024-02-19T21:12:31.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.709+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3948059360 con 0x7f3948059e00 2024-02-19T21:12:31.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3948059360 con 0x7f3948059e00 2024-02-19T21:12:31.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:31.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f3945ffb700 10 client.50084.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f394811cb10 osd=2 4 2024-02-19T21:12:31.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 2 6 2024-02-19T21:12:31.710 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:12:31.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f394ec16700 10 client.50084.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:31.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 7 2024-02-19T21:12:31.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948059770 osd=3 3 2024-02-19T21:12:31.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948059770 osd=3 3 2024-02-19T21:12:31.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948059770 osd=3 3 2024-02-19T21:12:31.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 7 2024-02-19T21:12:31.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f3948059770 osd=3 3 2024-02-19T21:12:31.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 3 7 2024-02-19T21:12:31.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948059770 osd=3 4 2024-02-19T21:12:31.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f394ec16700 10 client.50084.objecter _send_command 7 2024-02-19T21:12:31.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.710+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3948129ef0 con 0x7f3948121900 2024-02-19T21:12:31.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.711+0000 7f3945ffb700 10 client.50084.objecter ms_handle_connect 0x7f3948121900 2024-02-19T21:12:31.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.711+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3948129ef0 con 0x7f3948121900 2024-02-19T21:12:31.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.711+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:12:31.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.711+0000 7f3945ffb700 10 client.50084.objecter _finish_command 7 = system:0 2024-02-19T21:12:31.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.711+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f3948059770 osd=3 4 2024-02-19T21:12:31.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.711+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 3 7 2024-02-19T21:12:31.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.725+0000 7f394ec16700 10 client.50084.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:31.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.725+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 8 2024-02-19T21:12:31.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.725+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948059770 osd=3 3 2024-02-19T21:12:31.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.725+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948059770 osd=3 3 2024-02-19T21:12:31.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.725+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948059770 osd=3 3 2024-02-19T21:12:31.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.725+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 8 2024-02-19T21:12:31.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.725+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f3948059770 osd=3 3 2024-02-19T21:12:31.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.725+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 3 8 2024-02-19T21:12:31.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.725+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948059770 osd=3 4 2024-02-19T21:12:31.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.725+0000 7f394ec16700 10 client.50084.objecter _send_command 8 2024-02-19T21:12:31.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.725+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3948120e60 con 0x7f3948121900 2024-02-19T21:12:31.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3948120e60 con 0x7f3948121900 2024-02-19T21:12:31.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:31.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f3945ffb700 10 client.50084.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f3948059770 osd=3 4 2024-02-19T21:12:31.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 3 8 2024-02-19T21:12:31.726 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:12:31.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f394ec16700 10 client.50084.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:31.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 9 2024-02-19T21:12:31.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948120e60 osd=4 3 2024-02-19T21:12:31.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948120e60 osd=4 3 2024-02-19T21:12:31.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948120e60 osd=4 3 2024-02-19T21:12:31.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 9 2024-02-19T21:12:31.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f3948120e60 osd=4 3 2024-02-19T21:12:31.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 4 9 2024-02-19T21:12:31.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948120e60 osd=4 4 2024-02-19T21:12:31.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f394ec16700 10 client.50084.objecter _send_command 9 2024-02-19T21:12:31.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.727+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3948121120 con 0x7f3948126320 2024-02-19T21:12:31.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.728+0000 7f3945ffb700 10 client.50084.objecter ms_handle_connect 0x7f3948126320 2024-02-19T21:12:31.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.728+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3948121120 con 0x7f3948126320 2024-02-19T21:12:31.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.728+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:12:31.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.728+0000 7f3945ffb700 10 client.50084.objecter _finish_command 9 = system:0 2024-02-19T21:12:31.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.728+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f3948120e60 osd=4 4 2024-02-19T21:12:31.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.728+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 4 9 2024-02-19T21:12:31.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.743+0000 7f394ec16700 10 client.50084.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:31.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.743+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 10 2024-02-19T21:12:31.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.743+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948120e60 osd=4 3 2024-02-19T21:12:31.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.743+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948120e60 osd=4 3 2024-02-19T21:12:31.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.743+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948120e60 osd=4 3 2024-02-19T21:12:31.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.743+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 10 2024-02-19T21:12:31.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.743+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f3948120e60 osd=4 3 2024-02-19T21:12:31.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.743+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 4 10 2024-02-19T21:12:31.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.743+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948120e60 osd=4 4 2024-02-19T21:12:31.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.743+0000 7f394ec16700 10 client.50084.objecter _send_command 10 2024-02-19T21:12:31.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.743+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3948121120 con 0x7f3948126320 2024-02-19T21:12:31.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3948121120 con 0x7f3948126320 2024-02-19T21:12:31.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:31.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f3945ffb700 10 client.50084.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f3948120e60 osd=4 4 2024-02-19T21:12:31.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 4 10 2024-02-19T21:12:31.745 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:12:31.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f394ec16700 10 client.50084.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:31.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 11 2024-02-19T21:12:31.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948121120 osd=5 3 2024-02-19T21:12:31.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948121120 osd=5 3 2024-02-19T21:12:31.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948121120 osd=5 3 2024-02-19T21:12:31.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 11 2024-02-19T21:12:31.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f3948121120 osd=5 3 2024-02-19T21:12:31.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 5 11 2024-02-19T21:12:31.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948121120 osd=5 4 2024-02-19T21:12:31.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f394ec16700 10 client.50084.objecter _send_command 11 2024-02-19T21:12:31.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.745+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f394812c3e0 con 0x7f394812be30 2024-02-19T21:12:31.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.746+0000 7f3945ffb700 10 client.50084.objecter ms_handle_connect 0x7f394812be30 2024-02-19T21:12:31.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.746+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f394812c3e0 con 0x7f394812be30 2024-02-19T21:12:31.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.746+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:12:31.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.746+0000 7f3945ffb700 10 client.50084.objecter _finish_command 11 = system:0 2024-02-19T21:12:31.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.746+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f3948121120 osd=5 4 2024-02-19T21:12:31.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.746+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 5 11 2024-02-19T21:12:31.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.761+0000 7f394ec16700 10 client.50084.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:31.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.761+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 12 2024-02-19T21:12:31.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.761+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948121120 osd=5 3 2024-02-19T21:12:31.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.761+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948121120 osd=5 3 2024-02-19T21:12:31.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.761+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948121120 osd=5 3 2024-02-19T21:12:31.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.761+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 12 2024-02-19T21:12:31.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.761+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f3948121120 osd=5 3 2024-02-19T21:12:31.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.761+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 5 12 2024-02-19T21:12:31.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.761+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948121120 osd=5 4 2024-02-19T21:12:31.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.761+0000 7f394ec16700 10 client.50084.objecter _send_command 12 2024-02-19T21:12:31.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.761+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f394812c3e0 con 0x7f394812be30 2024-02-19T21:12:31.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f394812c3e0 con 0x7f394812be30 2024-02-19T21:12:31.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:31.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f3945ffb700 10 client.50084.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f3948121120 osd=5 4 2024-02-19T21:12:31.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 5 12 2024-02-19T21:12:31.763 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:12:31.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f394ec16700 10 client.50084.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:31.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 13 2024-02-19T21:12:31.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f394812c3e0 osd=6 3 2024-02-19T21:12:31.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f394812c3e0 osd=6 3 2024-02-19T21:12:31.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f394812c3e0 osd=6 3 2024-02-19T21:12:31.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 13 2024-02-19T21:12:31.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f394812c3e0 osd=6 3 2024-02-19T21:12:31.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 6 13 2024-02-19T21:12:31.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f394812c3e0 osd=6 4 2024-02-19T21:12:31.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f394ec16700 10 client.50084.objecter _send_command 13 2024-02-19T21:12:31.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.763+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3948136fe0 con 0x7f39481334d0 2024-02-19T21:12:31.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.764+0000 7f3945ffb700 10 client.50084.objecter ms_handle_connect 0x7f39481334d0 2024-02-19T21:12:31.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.764+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3948136fe0 con 0x7f39481334d0 2024-02-19T21:12:31.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.764+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:12:31.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.764+0000 7f3945ffb700 10 client.50084.objecter _finish_command 13 = system:0 2024-02-19T21:12:31.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.764+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f394812c3e0 osd=6 4 2024-02-19T21:12:31.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.764+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 6 13 2024-02-19T21:12:31.766 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:12:31.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.778+0000 7f394ec16700 10 client.50084.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:31.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.778+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 14 2024-02-19T21:12:31.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.778+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f394812c3e0 osd=6 3 2024-02-19T21:12:31.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.778+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f394812c3e0 osd=6 3 2024-02-19T21:12:31.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.778+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f394812c3e0 osd=6 3 2024-02-19T21:12:31.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.778+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 14 2024-02-19T21:12:31.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.778+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f394812c3e0 osd=6 3 2024-02-19T21:12:31.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.778+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 6 14 2024-02-19T21:12:31.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.778+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f394812c3e0 osd=6 4 2024-02-19T21:12:31.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.778+0000 7f394ec16700 10 client.50084.objecter _send_command 14 2024-02-19T21:12:31.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.779+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3948138760 con 0x7f39481334d0 2024-02-19T21:12:31.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.780+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3948138760 con 0x7f39481334d0 2024-02-19T21:12:31.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.780+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:31.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.780+0000 7f3945ffb700 10 client.50084.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.780+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f394812c3e0 osd=6 4 2024-02-19T21:12:31.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.780+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 6 14 2024-02-19T21:12:31.780 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:12:31.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.781+0000 7f394ec16700 10 client.50084.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:31.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.781+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 15 2024-02-19T21:12:31.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.781+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948138760 osd=7 3 2024-02-19T21:12:31.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.781+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948138760 osd=7 3 2024-02-19T21:12:31.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.781+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948138760 osd=7 3 2024-02-19T21:12:31.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.781+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 15 2024-02-19T21:12:31.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.781+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f3948138760 osd=7 3 2024-02-19T21:12:31.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.781+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 7 15 2024-02-19T21:12:31.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.781+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948138760 osd=7 4 2024-02-19T21:12:31.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.781+0000 7f394ec16700 10 client.50084.objecter _send_command 15 2024-02-19T21:12:31.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.781+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3948139390 con 0x7f3948138de0 2024-02-19T21:12:31.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.781+0000 7f3945ffb700 10 client.50084.objecter ms_handle_connect 0x7f3948138de0 2024-02-19T21:12:31.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.782+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3948139390 con 0x7f3948138de0 2024-02-19T21:12:31.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.782+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:12:31.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.782+0000 7f3945ffb700 10 client.50084.objecter _finish_command 15 = system:0 2024-02-19T21:12:31.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.782+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f3948138760 osd=7 4 2024-02-19T21:12:31.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.782+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 7 15 2024-02-19T21:12:31.795 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:30.399588+0000 mon.a (mon.0) 2808 : cluster [DBG] osdmap e514: 8 total, 8 up, 6 in 2024-02-19T21:12:31.795 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:30.408378+0000 mon.b (mon.1) 672 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:31.795 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:30.409841+0000 mon.a (mon.0) 2809 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:31.795 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:30.521596+0000 mgr.y (mgr.14550) 466 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:31.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.797+0000 7f394ec16700 10 client.50084.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:31.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.797+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign -1 16 2024-02-19T21:12:31.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.797+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948138760 osd=7 3 2024-02-19T21:12:31.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.797+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948138760 osd=7 3 2024-02-19T21:12:31.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.797+0000 7f394ec16700 20 client.50084.objecter _get_session s=0x7f3948138760 osd=7 3 2024-02-19T21:12:31.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.797+0000 7f394ec16700 15 client.50084.objecter _session_command_op_remove -1 16 2024-02-19T21:12:31.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.797+0000 7f394ec16700 20 client.50084.objecter get_session s=0x7f3948138760 osd=7 3 2024-02-19T21:12:31.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.797+0000 7f394ec16700 15 client.50084.objecter _session_command_op_assign 7 16 2024-02-19T21:12:31.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.797+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948138760 osd=7 4 2024-02-19T21:12:31.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.797+0000 7f394ec16700 10 client.50084.objecter _send_command 16 2024-02-19T21:12:31.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.797+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f3948139390 con 0x7f3948138de0 2024-02-19T21:12:31.798 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:31.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.799+0000 7f3945ffb700 1 -- 172.21.15.163:0/4055320434 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f3948139390 con 0x7f3948138de0 2024-02-19T21:12:31.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.799+0000 7f3945ffb700 10 client.50084.objecter ms_dispatch 0x7f3948001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:31.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.799+0000 7f3945ffb700 10 client.50084.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.799+0000 7f3945ffb700 20 client.50084.objecter put_session s=0x7f3948138760 osd=7 4 2024-02-19T21:12:31.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.799+0000 7f3945ffb700 15 client.50084.objecter _session_command_op_remove 7 16 2024-02-19T21:12:31.799 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:12:31.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 10 librados: watch_flush enter 2024-02-19T21:12:31.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 10 librados: watch_flush exit 2024-02-19T21:12:31.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 10 client.50084.objecter close_session for osd.0 2024-02-19T21:12:31.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 >> v1:172.21.15.163:6801/1559418115 conn(0x7f39480f9930 legacy=0x7f39480f9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:31.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948047540 osd=0 1 2024-02-19T21:12:31.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 10 client.50084.objecter close_session for osd.1 2024-02-19T21:12:31.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 >> v1:172.21.15.163:6805/3125047432 conn(0x7f394811cfa0 legacy=0x7f394811f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:31.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f39480fa420 osd=1 1 2024-02-19T21:12:31.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 10 client.50084.objecter close_session for osd.2 2024-02-19T21:12:31.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 >> v1:172.21.15.163:6809/1563158487 conn(0x7f3948059e00 legacy=0x7f3948123db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:31.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f394811cb10 osd=2 1 2024-02-19T21:12:31.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 10 client.50084.objecter close_session for osd.3 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 >> v1:172.21.15.163:6813/2112279769 conn(0x7f3948121900 legacy=0x7f39481287d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948059770 osd=3 1 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 10 client.50084.objecter close_session for osd.4 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 >> v1:172.21.15.193:6800/3129716965 conn(0x7f3948126320 legacy=0x7f394812a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948120e60 osd=4 1 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 10 client.50084.objecter close_session for osd.5 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 >> v1:172.21.15.193:6804/805161991 conn(0x7f394812be30 legacy=0x7f39481318a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948121120 osd=5 1 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 10 client.50084.objecter close_session for osd.6 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394d1b3700 1 -- 172.21.15.163:0/4055320434 reap_dead start 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 >> v1:172.21.15.193:6808/787604615 conn(0x7f39481334d0 legacy=0x7f39481358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394d1b3700 1 -- 172.21.15.163:0/4055320434 reap_dead start 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f394812c3e0 osd=6 1 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 10 client.50084.objecter close_session for osd.7 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 >> v1:172.21.15.193:6812/157927432 conn(0x7f3948138de0 legacy=0x7f3948136fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:31.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 20 client.50084.objecter put_session s=0x7f3948138760 osd=7 1 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 20 client.50084.objecter shutdown clearing up homeless session... 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 10 client.50084.objecter successfully canceled tick 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394d1b3700 1 -- 172.21.15.163:0/4055320434 reap_dead start 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 >> v1:172.21.15.163:6800/1552514681 conn(0x7f3938063c00 legacy=0x7f39380660c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 >> v1:172.21.15.163:6790/0 conn(0x7f39480f94d0 legacy=0x7f3948114d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 shutdown_connections 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 >> 172.21.15.163:0/4055320434 conn(0x7f39480f2470 msgr2=0x7f3948107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 shutdown_connections 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 1 -- 172.21.15.163:0/4055320434 wait complete. 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:31.800+0000 7f394ec16700 1 librados: shutdown 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.803 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.804 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.804 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:12:31.817 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:12:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:31.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:12:31.921 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:31.950 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:31.971 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:31.973 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:31.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:32.075 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:32.103 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:32.122 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:32.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:32.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:32.226 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:32.255 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:32.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:31.253623+0000 mgr.y (mgr.14550) 467 : cluster [DBG] pgmap v771: 238 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 205 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:32.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:31.405766+0000 mon.a (mon.0) 2810 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:32.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:31.405831+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e515: 8 total, 8 up, 6 in 2024-02-19T21:12:32.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:31.408787+0000 mon.b (mon.1) 673 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:12:32.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:31.656145+0000 mon.c (mon.2) 404 : audit [DBG] from='client.? v1:172.21.15.163:0/4055320434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:32.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:31.253623+0000 mgr.y (mgr.14550) 467 : cluster [DBG] pgmap v771: 238 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 205 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:32.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:31.405766+0000 mon.a (mon.0) 2810 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:32.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:31.405831+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e515: 8 total, 8 up, 6 in 2024-02-19T21:12:32.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:31.408787+0000 mon.b (mon.1) 673 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:12:32.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:31.656145+0000 mon.c (mon.2) 404 : audit [DBG] from='client.? v1:172.21.15.163:0/4055320434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:32.605 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:32.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:32.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:32.709 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:32.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:31.253623+0000 mgr.y (mgr.14550) 467 : cluster [DBG] pgmap v771: 238 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 205 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:32.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:31.405766+0000 mon.a (mon.0) 2810 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:32.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:31.405831+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e515: 8 total, 8 up, 6 in 2024-02-19T21:12:32.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:31.408787+0000 mon.b (mon.1) 673 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:12:32.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:31.656145+0000 mon.c (mon.2) 404 : audit [DBG] from='client.? v1:172.21.15.163:0/4055320434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:32.740 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:32.761 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:32.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:32.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:32.864 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:12:32.894 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:32.914 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:12:32.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:32.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:12:33.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:12:33.047 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:33.067 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:12:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:33.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:12:33.171 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:33.199 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:33.399 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:33.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:33.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:33.505 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:33.532 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:33.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:32.411365+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e516: 8 total, 8 up, 6 in 2024-02-19T21:12:33.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:32.416229+0000 mon.b (mon.1) 674 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:33.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:32.416708+0000 mon.a (mon.0) 2813 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:33.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:32.411365+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e516: 8 total, 8 up, 6 in 2024-02-19T21:12:33.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:32.416229+0000 mon.b (mon.1) 674 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:33.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:32.416708+0000 mon.a (mon.0) 2813 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:33.747 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:12:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:12:33] "GET /metrics HTTP/1.1" 200 190415 "" "Prometheus/2.33.4" 2024-02-19T21:12:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:32.411365+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e516: 8 total, 8 up, 6 in 2024-02-19T21:12:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:32.416229+0000 mon.b (mon.1) 674 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:33.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:32.416708+0000 mon.a (mon.0) 2813 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:34.381 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:34.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:34.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:34.485 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:34.516 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:34.615 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:34.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:34.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:34.675 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:33.254040+0000 mgr.y (mgr.14550) 468 : cluster [DBG] pgmap v774: 270 pgs: 64 unknown, 1 active+remapped+backfill_toofull, 205 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:34.675 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:33.415740+0000 mon.a (mon.0) 2814 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:12:34.675 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:33.415859+0000 mon.a (mon.0) 2815 : cluster [DBG] osdmap e517: 8 total, 8 up, 6 in 2024-02-19T21:12:34.675 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:33.426294+0000 mon.b (mon.1) 675 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_tier","val": "test-rados-api-smithi163-83928-85-test-flush"}]: dispatch 2024-02-19T21:12:34.675 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:33.432099+0000 mon.a (mon.0) 2816 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_tier","val": "test-rados-api-smithi163-83928-85-test-flush"}]: dispatch 2024-02-19T21:12:34.675 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:34.312203+0000 mon.a (mon.0) 2817 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_tier","val": "test-rados-api-smithi163-83928-85-test-flush"}]': finished 2024-02-19T21:12:34.675 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:34.312289+0000 mon.a (mon.0) 2818 : cluster [DBG] osdmap e518: 8 total, 8 up, 6 in 2024-02-19T21:12:34.675 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:34.318843+0000 mon.b (mon.1) 676 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:34.675 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:34.319153+0000 mon.a (mon.0) 2819 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:34.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:33.254040+0000 mgr.y (mgr.14550) 468 : cluster [DBG] pgmap v774: 270 pgs: 64 unknown, 1 active+remapped+backfill_toofull, 205 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:34.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:33.415740+0000 mon.a (mon.0) 2814 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:12:34.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:33.415859+0000 mon.a (mon.0) 2815 : cluster [DBG] osdmap e517: 8 total, 8 up, 6 in 2024-02-19T21:12:34.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:33.426294+0000 mon.b (mon.1) 675 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_tier","val": "test-rados-api-smithi163-83928-85-test-flush"}]: dispatch 2024-02-19T21:12:34.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:33.432099+0000 mon.a (mon.0) 2816 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_tier","val": "test-rados-api-smithi163-83928-85-test-flush"}]: dispatch 2024-02-19T21:12:34.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:34.312203+0000 mon.a (mon.0) 2817 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_tier","val": "test-rados-api-smithi163-83928-85-test-flush"}]': finished 2024-02-19T21:12:34.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:34.312289+0000 mon.a (mon.0) 2818 : cluster [DBG] osdmap e518: 8 total, 8 up, 6 in 2024-02-19T21:12:34.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:34.318843+0000 mon.b (mon.1) 676 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:34.676 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:34.319153+0000 mon.a (mon.0) 2819 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:34.720 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:34.747 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:34.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:33.254040+0000 mgr.y (mgr.14550) 468 : cluster [DBG] pgmap v774: 270 pgs: 64 unknown, 1 active+remapped+backfill_toofull, 205 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:34.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:33.415740+0000 mon.a (mon.0) 2814 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:12:34.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:33.415859+0000 mon.a (mon.0) 2815 : cluster [DBG] osdmap e517: 8 total, 8 up, 6 in 2024-02-19T21:12:34.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:33.426294+0000 mon.b (mon.1) 675 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_tier","val": "test-rados-api-smithi163-83928-85-test-flush"}]: dispatch 2024-02-19T21:12:34.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:33.432099+0000 mon.a (mon.0) 2816 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_tier","val": "test-rados-api-smithi163-83928-85-test-flush"}]: dispatch 2024-02-19T21:12:34.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:34.312203+0000 mon.a (mon.0) 2817 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_tier","val": "test-rados-api-smithi163-83928-85-test-flush"}]': finished 2024-02-19T21:12:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:34.312289+0000 mon.a (mon.0) 2818 : cluster [DBG] osdmap e518: 8 total, 8 up, 6 in 2024-02-19T21:12:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:34.318843+0000 mon.b (mon.1) 676 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:34.319153+0000 mon.a (mon.0) 2819 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:12:34.977 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:12:34.977 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:12:34.977 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-02-19T21:12:34.977 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:12:34.996 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:34.999 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:34.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:35.100 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:35.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.865+0000 7f8415bdd700 1 -- 172.21.15.163:0/3310220691 >> v1:172.21.15.163:6790/0 conn(0x7f84100f8920 legacy=0x7f84100f8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:35.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.865+0000 7f8415bdd700 1 -- 172.21.15.163:0/3310220691 shutdown_connections 2024-02-19T21:12:35.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.865+0000 7f8415bdd700 1 -- 172.21.15.163:0/3310220691 >> 172.21.15.163:0/3310220691 conn(0x7f84100f2470 msgr2=0x7f84100f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:35.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.865+0000 7f8415bdd700 1 -- 172.21.15.163:0/3310220691 shutdown_connections 2024-02-19T21:12:35.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.865+0000 7f8415bdd700 1 -- 172.21.15.163:0/3310220691 wait complete. 2024-02-19T21:12:35.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.865+0000 7f8415bdd700 1 librados: starting msgr at 2024-02-19T21:12:35.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.865+0000 7f8415bdd700 1 librados: starting objecter 2024-02-19T21:12:35.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.866+0000 7f8415bdd700 1 Processor -- start 2024-02-19T21:12:35.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.866+0000 7f8415bdd700 1 -- start start 2024-02-19T21:12:35.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.866+0000 7f8415bdd700 1 librados: setting wanted keys 2024-02-19T21:12:35.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.866+0000 7f8415bdd700 1 librados: calling monclient init 2024-02-19T21:12:35.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.866+0000 7f8415bdd700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f84101172d0 con 0x7f84100f94d0 2024-02-19T21:12:35.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.866+0000 7f8415bdd700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f84101184b0 con 0x7f84101042b0 2024-02-19T21:12:35.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.866+0000 7f8415bdd700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f84101196b0 con 0x7f84100f8920 2024-02-19T21:12:35.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.866+0000 7f840effd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f84100f94d0 0x7f84100ffb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:52004/0 (socket says 172.21.15.163:52004) 2024-02-19T21:12:35.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.866+0000 7f840effd700 1 -- 172.21.15.163:0/3109689437 learned_addr learned my addr 172.21.15.163:0/3109689437 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:35.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.866+0000 7f840cff9700 10 client.?.objecter ms_handle_connect 0x7f84100f94d0 2024-02-19T21:12:35.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.866+0000 7f840cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:35.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.866+0000 7f840cff9700 10 client.?.objecter ms_handle_connect 0x7f84100f8920 2024-02-19T21:12:35.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.866+0000 7f840cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:35.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.866+0000 7f840cff9700 10 client.?.objecter ms_handle_connect 0x7f84101042b0 2024-02-19T21:12:35.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.866+0000 7f840cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:35.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044687049 0 0) 0x7f84101172d0 con 0x7f84100f94d0 2024-02-19T21:12:35.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f8003800 con 0x7f84100f94d0 2024-02-19T21:12:35.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599270606 0 0) 0x7f84101196b0 con 0x7f84100f8920 2024-02-19T21:12:35.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84101172d0 con 0x7f84100f8920 2024-02-19T21:12:35.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416423635 0 0) 0x7f84101184b0 con 0x7f84101042b0 2024-02-19T21:12:35.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84101196b0 con 0x7f84101042b0 2024-02-19T21:12:35.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2406850095 0 0) 0x7f84101196b0 con 0x7f84101042b0 2024-02-19T21:12:35.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84101184b0 con 0x7f84101042b0 2024-02-19T21:12:35.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 14888341 0 0) 0x7f83f8003800 con 0x7f84100f94d0 2024-02-19T21:12:35.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84101196b0 con 0x7f84100f94d0 2024-02-19T21:12:35.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8408001bc0 con 0x7f84101042b0 2024-02-19T21:12:35.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3352557327 0 0) 0x7f84101172d0 con 0x7f84100f8920 2024-02-19T21:12:35.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83f8003800 con 0x7f84100f8920 2024-02-19T21:12:35.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f84040019f0 con 0x7f84100f94d0 2024-02-19T21:12:35.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8400002e50 con 0x7f84100f8920 2024-02-19T21:12:35.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2940428860 0 0) 0x7f84101184b0 con 0x7f84101042b0 2024-02-19T21:12:35.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 >> v1:172.21.15.163:6790/0 conn(0x7f84100f8920 legacy=0x7f84100ff470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:35.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 >> v1:172.21.15.163:6789/0 conn(0x7f84100f94d0 legacy=0x7f84100ffb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:35.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841011a8b0 con 0x7f84101042b0 2024-02-19T21:12:35.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f8415bdd700 1 -- 172.21.15.163:0/3109689437 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84101198a0 con 0x7f84101042b0 2024-02-19T21:12:35.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f8415bdd700 10 client.48898.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:35.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f8415bdd700 1 -- 172.21.15.163:0/3109689437 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8410119d10 con 0x7f84101042b0 2024-02-19T21:12:35.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f8415bdd700 1 librados: init done 2024-02-19T21:12:35.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8408002280 con 0x7f84101042b0 2024-02-19T21:12:35.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.867+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f84080034d0 con 0x7f84101042b0 2024-02-19T21:12:35.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.868+0000 7f8415bdd700 1 -- 172.21.15.163:0/3109689437 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83fc004f20 con 0x7f84101042b0 2024-02-19T21:12:35.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.869+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f8408001de0 con 0x7f84101042b0 2024-02-19T21:12:35.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.870+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(518..518 src has 43..518) v4 ==== 8482+0+0 (unknown 3354832603 0 0) 0x7f84080768c0 con 0x7f84101042b0 2024-02-19T21:12:35.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.870+0000 7f840cff9700 10 client.48898.objecter ms_dispatch 0x7f8410001550 osd_map(518..518 src has 43..518) v4 2024-02-19T21:12:35.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.870+0000 7f840cff9700 3 client.48898.objecter handle_osd_map got epochs [518,518] > 0 2024-02-19T21:12:35.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.870+0000 7f840cff9700 3 client.48898.objecter handle_osd_map decoding full epoch 518 2024-02-19T21:12:35.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.870+0000 7f840cff9700 20 client.48898.objecter dump_active .. 0 homeless 2024-02-19T21:12:35.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.870+0000 7f840cff9700 10 client.48898.objecter ms_handle_connect 0x7f83f8063cc0 2024-02-19T21:12:35.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:34.873+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8408049b10 con 0x7f84101042b0 2024-02-19T21:12:35.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:35.055+0000 7f8415bdd700 1 -- 172.21.15.163:0/3109689437 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f83fc0051d0 con 0x7f84101042b0 2024-02-19T21:12:35.116 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:35.130 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:35.281 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:35.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:35.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:35.320+0000 7f840cff9700 1 -- 172.21.15.163:0/3109689437 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v519) v1 ==== 97+0+0 (unknown 2819986756 0 0) 0x7f8408047f20 con 0x7f84101042b0 2024-02-19T21:12:35.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:35.323+0000 7f8415bdd700 10 librados: watch_flush enter 2024-02-19T21:12:35.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:35.323+0000 7f8415bdd700 10 librados: watch_flush exit 2024-02-19T21:12:35.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:35.323+0000 7f8415bdd700 20 client.48898.objecter shutdown clearing up homeless session... 2024-02-19T21:12:35.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:35.323+0000 7f8415bdd700 10 client.48898.objecter successfully canceled tick 2024-02-19T21:12:35.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:35.323+0000 7f8415bdd700 1 -- 172.21.15.163:0/3109689437 >> v1:172.21.15.163:6800/1552514681 conn(0x7f83f8063cc0 legacy=0x7f83f8066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:35.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:35.323+0000 7f8415bdd700 1 -- 172.21.15.163:0/3109689437 >> v1:172.21.15.193:6789/0 conn(0x7f84101042b0 legacy=0x7f8410115990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:35.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:35.323+0000 7f8415bdd700 1 -- 172.21.15.163:0/3109689437 shutdown_connections 2024-02-19T21:12:35.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:35.323+0000 7f8415bdd700 1 -- 172.21.15.163:0/3109689437 >> 172.21.15.163:0/3109689437 conn(0x7f84100f2470 msgr2=0x7f8410107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:35.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:35.324+0000 7f8415bdd700 1 -- 172.21.15.163:0/3109689437 shutdown_connections 2024-02-19T21:12:35.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:35.324+0000 7f8415bdd700 1 -- 172.21.15.163:0/3109689437 wait complete. 2024-02-19T21:12:35.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:35.324+0000 7f8415bdd700 1 librados: shutdown 2024-02-19T21:12:35.326 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:12:35.386 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:35.414 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:35.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:35.057872+0000 mon.b (mon.1) 677 : audit [INF] from='client.? v1:172.21.15.163:0/3109689437' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:12:35.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:35.058487+0000 mon.a (mon.0) 2820 : audit [INF] from='client.48898 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:12:35.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:35.315824+0000 mon.a (mon.0) 2821 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:12:35.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:35.321450+0000 mon.a (mon.0) 2822 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-19T21:12:35.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:35.321513+0000 mon.a (mon.0) 2823 : audit [INF] from='client.48898 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:12:35.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:35.321566+0000 mon.a (mon.0) 2824 : cluster [DBG] osdmap e519: 8 total, 8 up, 6 in 2024-02-19T21:12:35.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:35.323176+0000 mon.b (mon.1) 678 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:35.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:35.323526+0000 mon.a (mon.0) 2825 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:35.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:35.057872+0000 mon.b (mon.1) 677 : audit [INF] from='client.? v1:172.21.15.163:0/3109689437' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:12:35.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:35.058487+0000 mon.a (mon.0) 2820 : audit [INF] from='client.48898 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:12:35.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:35.315824+0000 mon.a (mon.0) 2821 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:12:35.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:35.321450+0000 mon.a (mon.0) 2822 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-19T21:12:35.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:35.321513+0000 mon.a (mon.0) 2823 : audit [INF] from='client.48898 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:12:35.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:35.321566+0000 mon.a (mon.0) 2824 : cluster [DBG] osdmap e519: 8 total, 8 up, 6 in 2024-02-19T21:12:35.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:35.323176+0000 mon.b (mon.1) 678 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:35.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:35.323526+0000 mon.a (mon.0) 2825 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:35.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:35.057872+0000 mon.b (mon.1) 677 : audit [INF] from='client.? v1:172.21.15.163:0/3109689437' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:12:35.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:35.058487+0000 mon.a (mon.0) 2820 : audit [INF] from='client.48898 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:12:35.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:35.315824+0000 mon.a (mon.0) 2821 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:12:35.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:35.321450+0000 mon.a (mon.0) 2822 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-19T21:12:35.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:35.321513+0000 mon.a (mon.0) 2823 : audit [INF] from='client.48898 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:12:35.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:35.321566+0000 mon.a (mon.0) 2824 : cluster [DBG] osdmap e519: 8 total, 8 up, 6 in 2024-02-19T21:12:35.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:35.323176+0000 mon.b (mon.1) 678 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:35.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:35.323526+0000 mon.a (mon.0) 2825 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:12:35.822 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:12:36.667 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:36.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:36.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:35.255005+0000 mgr.y (mgr.14550) 469 : cluster [DBG] pgmap v777: 270 pgs: 1 active+remapped+backfill_toofull, 269 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:36.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:36.321700+0000 mon.a (mon.0) 2826 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:12:36.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:36.321779+0000 mon.a (mon.0) 2827 : cluster [DBG] osdmap e520: 8 total, 8 up, 6 in 2024-02-19T21:12:36.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:35.255005+0000 mgr.y (mgr.14550) 469 : cluster [DBG] pgmap v777: 270 pgs: 1 active+remapped+backfill_toofull, 269 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:36.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:36.321700+0000 mon.a (mon.0) 2826 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:12:36.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:36.321779+0000 mon.a (mon.0) 2827 : cluster [DBG] osdmap e520: 8 total, 8 up, 6 in 2024-02-19T21:12:36.776 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:36.803 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:36.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:35.255005+0000 mgr.y (mgr.14550) 469 : cluster [DBG] pgmap v777: 270 pgs: 1 active+remapped+backfill_toofull, 269 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:36.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:36.321700+0000 mon.a (mon.0) 2826 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:12:36.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:36.321779+0000 mon.a (mon.0) 2827 : cluster [DBG] osdmap e520: 8 total, 8 up, 6 in 2024-02-19T21:12:37.082 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-19T21:12:37.252 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:37.255 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:37.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:37.357 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:12:37.389 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:37.409 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:12:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:37.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:12:37.513 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:37.542 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:38.070 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:12:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:12:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:12:38.291 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:38.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:38.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:38.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:38.422 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:38.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:37.255579+0000 mgr.y (mgr.14550) 470 : cluster [DBG] pgmap v780: 270 pgs: 1 active+remapped+backfill_toofull, 269 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:38.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:37.326021+0000 mon.a (mon.0) 2828 : cluster [DBG] osdmap e521: 8 total, 8 up, 6 in 2024-02-19T21:12:38.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:37.368963+0000 mon.b (mon.1) 679 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:38.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:37.369317+0000 mon.a (mon.0) 2829 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:38.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:37.369819+0000 mon.b (mon.1) 680 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-83"}]: dispatch 2024-02-19T21:12:38.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:37.370005+0000 mon.a (mon.0) 2830 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-83"}]: dispatch 2024-02-19T21:12:38.683 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:37.255579+0000 mgr.y (mgr.14550) 470 : cluster [DBG] pgmap v780: 270 pgs: 1 active+remapped+backfill_toofull, 269 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:38.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:37.326021+0000 mon.a (mon.0) 2828 : cluster [DBG] osdmap e521: 8 total, 8 up, 6 in 2024-02-19T21:12:38.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:37.368963+0000 mon.b (mon.1) 679 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:38.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:37.369317+0000 mon.a (mon.0) 2829 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:38.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:37.369819+0000 mon.b (mon.1) 680 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-83"}]: dispatch 2024-02-19T21:12:38.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:37.370005+0000 mon.a (mon.0) 2830 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-83"}]: dispatch 2024-02-19T21:12:38.772 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:38.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:38.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:37.255579+0000 mgr.y (mgr.14550) 470 : cluster [DBG] pgmap v780: 270 pgs: 1 active+remapped+backfill_toofull, 269 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:38.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:37.326021+0000 mon.a (mon.0) 2828 : cluster [DBG] osdmap e521: 8 total, 8 up, 6 in 2024-02-19T21:12:38.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:37.368963+0000 mon.b (mon.1) 679 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:38.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:37.369317+0000 mon.a (mon.0) 2829 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:38.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:37.369819+0000 mon.b (mon.1) 680 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-83"}]: dispatch 2024-02-19T21:12:38.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:37.370005+0000 mon.a (mon.0) 2830 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-83"}]: dispatch 2024-02-19T21:12:38.877 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:12:38.910 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:38.931 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:12:38.934 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:38.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:12:39.035 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:39.064 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:39.116 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:39.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:39.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:39.220 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.222+0000 7faca823a700 1 -- 172.21.15.163:0/2312956573 >> v1:172.21.15.163:6790/0 conn(0x7faca0100780 legacy=0x7faca0102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.222+0000 7faca823a700 1 -- 172.21.15.163:0/2312956573 shutdown_connections 2024-02-19T21:12:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.222+0000 7faca823a700 1 -- 172.21.15.163:0/2312956573 >> 172.21.15.163:0/2312956573 conn(0x7faca00f35a0 msgr2=0x7faca00f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.222+0000 7faca823a700 1 -- 172.21.15.163:0/2312956573 shutdown_connections 2024-02-19T21:12:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.222+0000 7faca823a700 1 -- 172.21.15.163:0/2312956573 wait complete. 2024-02-19T21:12:39.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.222+0000 7faca823a700 1 librados: starting msgr at 2024-02-19T21:12:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.222+0000 7faca823a700 1 librados: starting objecter 2024-02-19T21:12:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7faca823a700 1 Processor -- start 2024-02-19T21:12:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7faca823a700 1 -- start start 2024-02-19T21:12:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7faca823a700 1 librados: setting wanted keys 2024-02-19T21:12:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7faca823a700 1 librados: calling monclient init 2024-02-19T21:12:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7faca823a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faca01174b0 con 0x7faca00fd980 2024-02-19T21:12:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7faca823a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faca0118690 con 0x7faca01042c0 2024-02-19T21:12:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7faca823a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faca0119890 con 0x7faca0100780 2024-02-19T21:12:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7faca57d5700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7faca0100780 0x7faca00fa300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:35408/0 (socket says 172.21.15.163:35408) 2024-02-19T21:12:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7faca57d5700 1 -- 172.21.15.163:0/2642919002 learned_addr learned my addr 172.21.15.163:0/2642919002 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7fac96ffd700 10 client.?.objecter ms_handle_connect 0x7faca0100780 2024-02-19T21:12:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7fac96ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7fac96ffd700 10 client.?.objecter ms_handle_connect 0x7faca01042c0 2024-02-19T21:12:39.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7fac96ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555238278 0 0) 0x7faca0119890 con 0x7faca0100780 2024-02-19T21:12:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac8c0033f0 con 0x7faca0100780 2024-02-19T21:12:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7fac96ffd700 10 client.?.objecter ms_handle_connect 0x7faca00fd980 2024-02-19T21:12:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7fac96ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489949634 0 0) 0x7faca0118690 con 0x7faca01042c0 2024-02-19T21:12:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faca0119890 con 0x7faca01042c0 2024-02-19T21:12:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314956109 0 0) 0x7faca01174b0 con 0x7faca00fd980 2024-02-19T21:12:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.223+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faca0118690 con 0x7faca00fd980 2024-02-19T21:12:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 962521982 0 0) 0x7faca0119890 con 0x7faca01042c0 2024-02-19T21:12:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faca01174b0 con 0x7faca01042c0 2024-02-19T21:12:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fac98001c40 con 0x7faca01042c0 2024-02-19T21:12:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3904732529 0 0) 0x7fac8c0033f0 con 0x7faca0100780 2024-02-19T21:12:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faca0119890 con 0x7faca0100780 2024-02-19T21:12:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fac90002d60 con 0x7faca0100780 2024-02-19T21:12:39.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2634618053 0 0) 0x7faca0118690 con 0x7faca00fd980 2024-02-19T21:12:39.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac8c0033f0 con 0x7faca00fd980 2024-02-19T21:12:39.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fac9c0018c0 con 0x7faca00fd980 2024-02-19T21:12:39.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1221345313 0 0) 0x7faca0119890 con 0x7faca0100780 2024-02-19T21:12:39.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 >> v1:172.21.15.193:6789/0 conn(0x7faca01042c0 legacy=0x7faca0115b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:39.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 >> v1:172.21.15.163:6789/0 conn(0x7faca00fd980 legacy=0x7faca00f9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:39.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca011aa90 con 0x7faca0100780 2024-02-19T21:12:39.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7faca823a700 1 -- 172.21.15.163:0/2642919002 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca0118880 con 0x7faca0100780 2024-02-19T21:12:39.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7faca823a700 10 client.50096.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:39.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7faca823a700 1 -- 172.21.15.163:0/2642919002 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faca0118cf0 con 0x7faca0100780 2024-02-19T21:12:39.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7faca823a700 1 librados: init done 2024-02-19T21:12:39.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fac90003030 con 0x7faca0100780 2024-02-19T21:12:39.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.224+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fac90003820 con 0x7faca0100780 2024-02-19T21:12:39.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.225+0000 7faca823a700 1 -- 172.21.15.163:0/2642919002 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac84004f20 con 0x7faca0100780 2024-02-19T21:12:39.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.226+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fac90003980 con 0x7faca0100780 2024-02-19T21:12:39.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.227+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(522..522 src has 43..522) v4 ==== 7682+0+0 (unknown 3137564848 0 0) 0x7fac90078e20 con 0x7faca0100780 2024-02-19T21:12:39.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.227+0000 7fac96ffd700 10 client.50096.objecter ms_dispatch 0x7faca0001550 osd_map(522..522 src has 43..522) v4 2024-02-19T21:12:39.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.227+0000 7fac96ffd700 3 client.50096.objecter handle_osd_map got epochs [522,522] > 0 2024-02-19T21:12:39.228 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.227+0000 7fac96ffd700 3 client.50096.objecter handle_osd_map decoding full epoch 522 2024-02-19T21:12:39.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.227+0000 7fac96ffd700 20 client.50096.objecter dump_active .. 0 homeless 2024-02-19T21:12:39.229 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.227+0000 7fac96ffd700 10 client.50096.objecter ms_handle_connect 0x7fac8c068090 2024-02-19T21:12:39.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.233+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fac9004c5a0 con 0x7faca0100780 2024-02-19T21:12:39.233 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:39.233 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:39.255 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:39.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:39.463+0000 7faca823a700 1 -- 172.21.15.163:0/2642919002 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7fac840051d0 con 0x7faca0100780 2024-02-19T21:12:39.507 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:39.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:38.334301+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e522: 8 total, 8 up, 6 in 2024-02-19T21:12:39.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:39.278568+0000 mon.b (mon.1) 681 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-19T21:12:39.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:39.278793+0000 mon.b (mon.1) 682 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 7]}]: dispatch 2024-02-19T21:12:39.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:39.279034+0000 mon.a (mon.0) 2832 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-19T21:12:39.509 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:39.279534+0000 mon.a (mon.0) 2833 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 7]}]: dispatch 2024-02-19T21:12:39.509 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:38.334301+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e522: 8 total, 8 up, 6 in 2024-02-19T21:12:39.509 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:39.278568+0000 mon.b (mon.1) 681 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-19T21:12:39.509 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:39.278793+0000 mon.b (mon.1) 682 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 7]}]: dispatch 2024-02-19T21:12:39.509 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:39.279034+0000 mon.a (mon.0) 2832 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-19T21:12:39.509 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:39.279534+0000 mon.a (mon.0) 2833 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 7]}]: dispatch 2024-02-19T21:12:39.510 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:39.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:39.611 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:39.641 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:39.663 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:39.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:39.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:39.767 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:39.796 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:38.334301+0000 mon.a (mon.0) 2831 : cluster [DBG] osdmap e522: 8 total, 8 up, 6 in 2024-02-19T21:12:39.797 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:39.278568+0000 mon.b (mon.1) 681 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-19T21:12:39.797 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:39.278793+0000 mon.b (mon.1) 682 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 7]}]: dispatch 2024-02-19T21:12:39.797 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:39.279034+0000 mon.a (mon.0) 2832 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-19T21:12:39.797 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:39.279534+0000 mon.a (mon.0) 2833 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 7]}]: dispatch 2024-02-19T21:12:39.799 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:39.821 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:39.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:39.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:39.925 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:39.954 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:40.203 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:40.205 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:40.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:40.306 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:40.334 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:40.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:40.362+0000 7fac96ffd700 1 -- 172.21.15.163:0/2642919002 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v524) v1 ==== 113+0+0 (unknown 2073096644 0 0) 0x7fac9004aa10 con 0x7faca0100780 2024-02-19T21:12:40.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:40.366+0000 7faca823a700 10 librados: watch_flush enter 2024-02-19T21:12:40.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:40.370+0000 7faca823a700 10 librados: watch_flush exit 2024-02-19T21:12:40.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:40.370+0000 7faca823a700 20 client.50096.objecter shutdown clearing up homeless session... 2024-02-19T21:12:40.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:40.370+0000 7faca823a700 10 client.50096.objecter successfully canceled tick 2024-02-19T21:12:40.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:40.370+0000 7faca823a700 1 -- 172.21.15.163:0/2642919002 >> v1:172.21.15.163:6800/1552514681 conn(0x7fac8c068090 legacy=0x7fac8c06a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:40.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:40.370+0000 7faca823a700 1 -- 172.21.15.163:0/2642919002 >> v1:172.21.15.163:6790/0 conn(0x7faca0100780 legacy=0x7faca00fa300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:40.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:40.371+0000 7faca823a700 1 -- 172.21.15.163:0/2642919002 shutdown_connections 2024-02-19T21:12:40.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:40.371+0000 7faca823a700 1 -- 172.21.15.163:0/2642919002 >> 172.21.15.163:0/2642919002 conn(0x7faca00f35a0 msgr2=0x7faca0107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:40.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:40.371+0000 7faca823a700 1 -- 172.21.15.163:0/2642919002 shutdown_connections 2024-02-19T21:12:40.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:40.371+0000 7faca823a700 1 -- 172.21.15.163:0/2642919002 wait complete. 2024-02-19T21:12:40.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:40.374+0000 7faca823a700 1 librados: shutdown 2024-02-19T21:12:40.375 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.5 to 1 (10000) 2024-02-19T21:12:40.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:39.255966+0000 mgr.y (mgr.14550) 471 : cluster [DBG] pgmap v783: 206 pgs: 1 active+remapped+backfill_toofull, 205 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:40.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:39.377814+0000 mon.a (mon.0) 2834 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-02-19T21:12:40.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:39.377864+0000 mon.a (mon.0) 2835 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 7]}]': finished 2024-02-19T21:12:40.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:39.377900+0000 mon.a (mon.0) 2836 : cluster [DBG] osdmap e523: 8 total, 8 up, 6 in 2024-02-19T21:12:40.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:39.380877+0000 mon.b (mon.1) 683 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:40.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:39.381257+0000 mon.a (mon.0) 2837 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:40.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:39.464808+0000 mon.c (mon.2) 405 : audit [INF] from='client.? v1:172.21.15.163:0/2642919002' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:12:40.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:39.465125+0000 mon.a (mon.0) 2838 : audit [INF] from='client.50096 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:12:40.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:39.255966+0000 mgr.y (mgr.14550) 471 : cluster [DBG] pgmap v783: 206 pgs: 1 active+remapped+backfill_toofull, 205 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:40.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:39.377814+0000 mon.a (mon.0) 2834 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-02-19T21:12:40.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:39.377864+0000 mon.a (mon.0) 2835 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 7]}]': finished 2024-02-19T21:12:40.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:39.377900+0000 mon.a (mon.0) 2836 : cluster [DBG] osdmap e523: 8 total, 8 up, 6 in 2024-02-19T21:12:40.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:39.380877+0000 mon.b (mon.1) 683 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:40.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:39.381257+0000 mon.a (mon.0) 2837 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:40.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:39.464808+0000 mon.c (mon.2) 405 : audit [INF] from='client.? v1:172.21.15.163:0/2642919002' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:12:40.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:39.465125+0000 mon.a (mon.0) 2838 : audit [INF] from='client.50096 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:12:40.700 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:12:40.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:12:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:12:40.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:39.255966+0000 mgr.y (mgr.14550) 471 : cluster [DBG] pgmap v783: 206 pgs: 1 active+remapped+backfill_toofull, 205 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:40.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:39.377814+0000 mon.a (mon.0) 2834 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-02-19T21:12:40.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:39.377864+0000 mon.a (mon.0) 2835 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "16.f", "id": [1, 7]}]': finished 2024-02-19T21:12:40.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:39.377900+0000 mon.a (mon.0) 2836 : cluster [DBG] osdmap e523: 8 total, 8 up, 6 in 2024-02-19T21:12:40.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:39.380877+0000 mon.b (mon.1) 683 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:40.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:39.381257+0000 mon.a (mon.0) 2837 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-86","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:40.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:39.464808+0000 mon.c (mon.2) 405 : audit [INF] from='client.? v1:172.21.15.163:0/2642919002' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:12:40.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:39.465125+0000 mon.a (mon.0) 2838 : audit [INF] from='client.50096 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:12:41.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:40.354318+0000 mon.a (mon.0) 2839 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-86","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:41.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:40.354490+0000 mon.a (mon.0) 2840 : audit [INF] from='client.50096 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-19T21:12:41.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:40.354578+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e524: 8 total, 8 up, 6 in 2024-02-19T21:12:41.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:40.367375+0000 mon.b (mon.1) 684 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:12:41.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:40.374219+0000 mon.b (mon.1) 685 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:41.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:40.375165+0000 mon.a (mon.0) 2842 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:41.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:40.532480+0000 mgr.y (mgr.14550) 472 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:41.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:40.354318+0000 mon.a (mon.0) 2839 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-86","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:41.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:40.354490+0000 mon.a (mon.0) 2840 : audit [INF] from='client.50096 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-19T21:12:41.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:40.354578+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e524: 8 total, 8 up, 6 in 2024-02-19T21:12:41.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:40.367375+0000 mon.b (mon.1) 684 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:12:41.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:40.374219+0000 mon.b (mon.1) 685 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:41.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:40.375165+0000 mon.a (mon.0) 2842 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:41.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:40.532480+0000 mgr.y (mgr.14550) 472 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:41.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:40.354318+0000 mon.a (mon.0) 2839 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-86","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:41.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:40.354490+0000 mon.a (mon.0) 2840 : audit [INF] from='client.50096 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-19T21:12:41.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:40.354578+0000 mon.a (mon.0) 2841 : cluster [DBG] osdmap e524: 8 total, 8 up, 6 in 2024-02-19T21:12:41.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:40.367375+0000 mon.b (mon.1) 684 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:12:41.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:40.374219+0000 mon.b (mon.1) 685 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:41.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:40.375165+0000 mon.a (mon.0) 2842 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:12:41.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:40.532480+0000 mgr.y (mgr.14550) 472 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:41.994 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:41.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:41.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:42.098 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:42.125 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:42.145 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:42.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:42.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:42.249 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:42.277 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:42.526 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:42.527 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:41.256428+0000 mgr.y (mgr.14550) 473 : cluster [DBG] pgmap v786: 238 pgs: 1 active+remapped+backfill_toofull, 2 peering, 32 creating+peering, 203 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:42.528 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:41.354945+0000 mon.a (mon.0) 2843 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:12:42.528 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:41.355028+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e525: 8 total, 8 up, 6 in 2024-02-19T21:12:42.528 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:41.256428+0000 mgr.y (mgr.14550) 473 : cluster [DBG] pgmap v786: 238 pgs: 1 active+remapped+backfill_toofull, 2 peering, 32 creating+peering, 203 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:42.528 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:41.354945+0000 mon.a (mon.0) 2843 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:12:42.528 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:41.355028+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e525: 8 total, 8 up, 6 in 2024-02-19T21:12:42.529 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:42.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:42.559 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:12:42.559 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:12:42.559 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-02-19T21:12:42.559 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-02-19T21:12:42.559 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:12:42.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:41.256428+0000 mgr.y (mgr.14550) 473 : cluster [DBG] pgmap v786: 238 pgs: 1 active+remapped+backfill_toofull, 2 peering, 32 creating+peering, 203 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:42.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:41.354945+0000 mon.a (mon.0) 2843 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:12:42.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:41.355028+0000 mon.a (mon.0) 2844 : cluster [DBG] osdmap e525: 8 total, 8 up, 6 in 2024-02-19T21:12:42.630 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:42.657 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:42.675 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:42.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:42.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:42.779 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:42.806 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:42.824 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:42.826 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 7 1 2024-02-19T21:12:42.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:42.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:42.928 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:42.955 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:42.981 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:42.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:42.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:43.085 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:43.111 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:43.130 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:43.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:43.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:43.260 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:43.278 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:43.280 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:43.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:43.382 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:43.407 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-19T21:12:43.408 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:43.428 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:43.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:43.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:43.532 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:43.561 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:43.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.618+0000 7f8ca424f700 1 -- 172.21.15.163:0/4139837695 >> v1:172.21.15.193:6789/0 conn(0x7f8c9c1042c0 legacy=0x7f8c9c106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:43.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.618+0000 7f8ca424f700 1 -- 172.21.15.163:0/4139837695 shutdown_connections 2024-02-19T21:12:43.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.618+0000 7f8ca424f700 1 -- 172.21.15.163:0/4139837695 >> 172.21.15.163:0/4139837695 conn(0x7f8c9c0f35a0 msgr2=0x7f8c9c0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:43.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.618+0000 7f8ca424f700 1 -- 172.21.15.163:0/4139837695 shutdown_connections 2024-02-19T21:12:43.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.618+0000 7f8ca424f700 1 -- 172.21.15.163:0/4139837695 wait complete. 2024-02-19T21:12:43.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.619+0000 7f8ca424f700 1 librados: starting msgr at 2024-02-19T21:12:43.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.619+0000 7f8ca424f700 1 librados: starting objecter 2024-02-19T21:12:43.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.619+0000 7f8ca424f700 1 Processor -- start 2024-02-19T21:12:43.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.619+0000 7f8ca424f700 1 -- start start 2024-02-19T21:12:43.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.619+0000 7f8ca424f700 1 librados: setting wanted keys 2024-02-19T21:12:43.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.619+0000 7f8ca424f700 1 librados: calling monclient init 2024-02-19T21:12:43.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.619+0000 7f8ca424f700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8c9c1262a0 con 0x7f8c9c100780 2024-02-19T21:12:43.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.619+0000 7f8ca424f700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8c9c127480 con 0x7f8c9c0fd980 2024-02-19T21:12:43.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.619+0000 7f8ca424f700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8c9c128680 con 0x7f8c9c1042c0 2024-02-19T21:12:43.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8ca17ea700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8c9c100780 0x7f8c9c0fa360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:52062/0 (socket says 172.21.15.163:52062) 2024-02-19T21:12:43.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8ca17ea700 1 -- 172.21.15.163:0/2215212845 learned_addr learned my addr 172.21.15.163:0/2215212845 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:43.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 10 client.?.objecter ms_handle_connect 0x7f8c9c0fd980 2024-02-19T21:12:43.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:43.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 10 client.?.objecter ms_handle_connect 0x7f8c9c100780 2024-02-19T21:12:43.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:43.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 10 client.?.objecter ms_handle_connect 0x7f8c9c1042c0 2024-02-19T21:12:43.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:43.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813439624 0 0) 0x7f8c9c127480 con 0x7f8c9c0fd980 2024-02-19T21:12:43.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c880033f0 con 0x7f8c9c0fd980 2024-02-19T21:12:43.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022760455 0 0) 0x7f8c9c128680 con 0x7f8c9c1042c0 2024-02-19T21:12:43.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c9c127480 con 0x7f8c9c1042c0 2024-02-19T21:12:43.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63075863 0 0) 0x7f8c9c1262a0 con 0x7f8c9c100780 2024-02-19T21:12:43.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c9c128680 con 0x7f8c9c100780 2024-02-19T21:12:43.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 573135415 0 0) 0x7f8c9c127480 con 0x7f8c9c1042c0 2024-02-19T21:12:43.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c9c1262a0 con 0x7f8c9c1042c0 2024-02-19T21:12:43.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 254456504 0 0) 0x7f8c880033f0 con 0x7f8c9c0fd980 2024-02-19T21:12:43.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c9c127480 con 0x7f8c9c0fd980 2024-02-19T21:12:43.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8c940023e0 con 0x7f8c9c1042c0 2024-02-19T21:12:43.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.620+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8c980017c0 con 0x7f8c9c0fd980 2024-02-19T21:12:43.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.621+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 719897462 0 0) 0x7f8c9c128680 con 0x7f8c9c100780 2024-02-19T21:12:43.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.621+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c880033f0 con 0x7f8c9c100780 2024-02-19T21:12:43.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.621+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8c8c0018c0 con 0x7f8c9c100780 2024-02-19T21:12:43.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.621+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 348824054 0 0) 0x7f8c9c127480 con 0x7f8c9c0fd980 2024-02-19T21:12:43.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.621+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 >> v1:172.21.15.163:6790/0 conn(0x7f8c9c1042c0 legacy=0x7f8c9c06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:43.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.621+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 >> v1:172.21.15.163:6789/0 conn(0x7f8c9c100780 legacy=0x7f8c9c0fa360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:43.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.621+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c9c129880 con 0x7f8c9c0fd980 2024-02-19T21:12:43.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.621+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c9c126490 con 0x7f8c9c0fd980 2024-02-19T21:12:43.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.621+0000 7f8ca424f700 10 client.48910.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:43.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.621+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c9c126980 con 0x7f8c9c0fd980 2024-02-19T21:12:43.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.621+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8c98001dc0 con 0x7f8c9c0fd980 2024-02-19T21:12:43.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.621+0000 7f8ca424f700 1 librados: init done 2024-02-19T21:12:43.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.621+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8c980031c0 con 0x7f8c9c0fd980 2024-02-19T21:12:43.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.622+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f8c80004f00 con 0x7f8c9c0fd980 2024-02-19T21:12:43.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.622+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f8c98001920 con 0x7f8c9c0fd980 2024-02-19T21:12:43.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(527..527 src has 43..527) v4 ==== 8092+0+0 (unknown 2971042009 0 0) 0x7f8c98076350 con 0x7f8c9c0fd980 2024-02-19T21:12:43.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 osd_map(527..527 src has 43..527) v4 2024-02-19T21:12:43.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8c92ffd700 3 client.48910.objecter handle_osd_map got epochs [527,527] > 0 2024-02-19T21:12:43.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8c92ffd700 3 client.48910.objecter handle_osd_map decoding full epoch 527 2024-02-19T21:12:43.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8c92ffd700 20 client.48910.objecter dump_active .. 0 homeless 2024-02-19T21:12:43.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v527) v1 ==== 54+0+16 (unknown 2380758252 0 2420000192) 0x7f8c98049720 con 0x7f8c9c0fd980 2024-02-19T21:12:43.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8ca424f700 10 client.48910.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:43.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 1 2024-02-19T21:12:43.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c800051a0 osd=0 3 2024-02-19T21:12:43.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c800051a0 osd=0 3 2024-02-19T21:12:43.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c800051a0 osd=0 3 2024-02-19T21:12:43.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 1 2024-02-19T21:12:43.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c800051a0 osd=0 3 2024-02-19T21:12:43.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 0 1 2024-02-19T21:12:43.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c800051a0 osd=0 4 2024-02-19T21:12:43.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8ca424f700 10 client.48910.objecter _send_command 1 2024-02-19T21:12:43.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.623+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c80004950 con 0x7f8c80004530 2024-02-19T21:12:43.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.624+0000 7f8c92ffd700 10 client.48910.objecter ms_handle_connect 0x7f8c88063c80 2024-02-19T21:12:43.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.625+0000 7f8c92ffd700 10 client.48910.objecter ms_handle_connect 0x7f8c80004530 2024-02-19T21:12:43.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.625+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8c80004950 con 0x7f8c80004530 2024-02-19T21:12:43.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.625+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:12:43.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.625+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 1 = system:0 2024-02-19T21:12:43.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.625+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c800051a0 osd=0 4 2024-02-19T21:12:43.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.625+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 0 1 2024-02-19T21:12:43.634 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:43.637 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:12:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:12:43] "GET /metrics HTTP/1.1" 200 187969 "" "Prometheus/2.33.4" 2024-02-19T21:12:43.637 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:42.355959+0000 mon.a (mon.0) 2845 : cluster [DBG] osdmap e526: 8 total, 8 up, 6 in 2024-02-19T21:12:43.637 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:42.355959+0000 mon.a (mon.0) 2845 : cluster [DBG] osdmap e526: 8 total, 8 up, 6 in 2024-02-19T21:12:43.638 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:43.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:43.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.646+0000 7f8ca424f700 10 client.48910.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:43.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.646+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 2 2024-02-19T21:12:43.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.646+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c800051a0 osd=0 3 2024-02-19T21:12:43.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.646+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c800051a0 osd=0 3 2024-02-19T21:12:43.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.646+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c800051a0 osd=0 3 2024-02-19T21:12:43.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.646+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 2 2024-02-19T21:12:43.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.646+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c800051a0 osd=0 3 2024-02-19T21:12:43.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.646+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 0 2 2024-02-19T21:12:43.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.646+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c800051a0 osd=0 4 2024-02-19T21:12:43.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.646+0000 7f8ca424f700 10 client.48910.objecter _send_command 2 2024-02-19T21:12:43.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.646+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c80004d30 con 0x7f8c80004530 2024-02-19T21:12:43.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.648+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8c80004d30 con 0x7f8c80004530 2024-02-19T21:12:43.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.648+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:43.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.648+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.648+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c800051a0 osd=0 4 2024-02-19T21:12:43.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.648+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 0 2 2024-02-19T21:12:43.649 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:12:43.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.649+0000 7f8ca424f700 10 client.48910.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:43.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.649+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 3 2024-02-19T21:12:43.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.649+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c80003fe0 osd=1 3 2024-02-19T21:12:43.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.649+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c80003fe0 osd=1 3 2024-02-19T21:12:43.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.649+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c80003fe0 osd=1 3 2024-02-19T21:12:43.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.649+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 3 2024-02-19T21:12:43.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.649+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c80003fe0 osd=1 3 2024-02-19T21:12:43.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.649+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 1 3 2024-02-19T21:12:43.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.649+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c80003fe0 osd=1 4 2024-02-19T21:12:43.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.649+0000 7f8ca424f700 10 client.48910.objecter _send_command 3 2024-02-19T21:12:43.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.649+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c8000c000 con 0x7f8c80004150 2024-02-19T21:12:43.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.649+0000 7f8c92ffd700 10 client.48910.objecter ms_handle_connect 0x7f8c80004150 2024-02-19T21:12:43.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.650+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8c8000c000 con 0x7f8c80004150 2024-02-19T21:12:43.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.650+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:12:43.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.650+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 3 = system:0 2024-02-19T21:12:43.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.650+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c80003fe0 osd=1 4 2024-02-19T21:12:43.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.650+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 1 3 2024-02-19T21:12:43.668 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:43.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.674+0000 7f8ca424f700 10 client.48910.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:43.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.674+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 4 2024-02-19T21:12:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.674+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c80003fe0 osd=1 3 2024-02-19T21:12:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.674+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c80003fe0 osd=1 3 2024-02-19T21:12:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.674+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c80003fe0 osd=1 3 2024-02-19T21:12:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.674+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 4 2024-02-19T21:12:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.674+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c80003fe0 osd=1 3 2024-02-19T21:12:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.674+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 1 4 2024-02-19T21:12:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.674+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c80003fe0 osd=1 4 2024-02-19T21:12:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.674+0000 7f8ca424f700 10 client.48910.objecter _send_command 4 2024-02-19T21:12:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.674+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c80001850 con 0x7f8c80004150 2024-02-19T21:12:43.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.676+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8c80001850 con 0x7f8c80004150 2024-02-19T21:12:43.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.676+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:43.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.676+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.676+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c80003fe0 osd=1 4 2024-02-19T21:12:43.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.676+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 1 4 2024-02-19T21:12:43.676 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:12:43.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.677+0000 7f8ca424f700 10 client.48910.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:43.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.677+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 5 2024-02-19T21:12:43.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.677+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c80003630 osd=2 3 2024-02-19T21:12:43.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.677+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c80003630 osd=2 3 2024-02-19T21:12:43.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.677+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c80003630 osd=2 3 2024-02-19T21:12:43.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.677+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 5 2024-02-19T21:12:43.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.677+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c80003630 osd=2 3 2024-02-19T21:12:43.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.677+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 2 5 2024-02-19T21:12:43.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.677+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c80003630 osd=2 4 2024-02-19T21:12:43.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.677+0000 7f8ca424f700 10 client.48910.objecter _send_command 5 2024-02-19T21:12:43.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.677+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c80003e70 con 0x7f8c8000d2f0 2024-02-19T21:12:43.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.681+0000 7f8c92ffd700 10 client.48910.objecter ms_handle_connect 0x7f8c8000d2f0 2024-02-19T21:12:43.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.681+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8c80003e70 con 0x7f8c8000d2f0 2024-02-19T21:12:43.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.681+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:12:43.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.681+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 5 = system:0 2024-02-19T21:12:43.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.681+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c80003630 osd=2 4 2024-02-19T21:12:43.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.681+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 2 5 2024-02-19T21:12:43.686 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:43.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.699+0000 7f8ca424f700 10 client.48910.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:43.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.699+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 6 2024-02-19T21:12:43.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.699+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c80003630 osd=2 3 2024-02-19T21:12:43.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.699+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c80003630 osd=2 3 2024-02-19T21:12:43.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.699+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c80003630 osd=2 3 2024-02-19T21:12:43.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.699+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 6 2024-02-19T21:12:43.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.699+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c80003630 osd=2 3 2024-02-19T21:12:43.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.699+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 2 6 2024-02-19T21:12:43.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.699+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c80003630 osd=2 4 2024-02-19T21:12:43.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.699+0000 7f8ca424f700 10 client.48910.objecter _send_command 6 2024-02-19T21:12:43.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.699+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c8000e750 con 0x7f8c8000d2f0 2024-02-19T21:12:43.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.701+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8c8000e750 con 0x7f8c8000d2f0 2024-02-19T21:12:43.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.701+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:43.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.701+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.701+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c80003630 osd=2 4 2024-02-19T21:12:43.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.701+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 2 6 2024-02-19T21:12:43.701 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:12:43.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.702+0000 7f8ca424f700 10 client.48910.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:43.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.702+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 7 2024-02-19T21:12:43.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.702+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c80003470 osd=3 3 2024-02-19T21:12:43.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.702+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c80003470 osd=3 3 2024-02-19T21:12:43.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.702+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c80003470 osd=3 3 2024-02-19T21:12:43.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.702+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 7 2024-02-19T21:12:43.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.702+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c80003470 osd=3 3 2024-02-19T21:12:43.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.702+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 3 7 2024-02-19T21:12:43.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.702+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c80003470 osd=3 4 2024-02-19T21:12:43.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.702+0000 7f8ca424f700 10 client.48910.objecter _send_command 7 2024-02-19T21:12:43.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.702+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c80015b60 con 0x7f8c80011e80 2024-02-19T21:12:43.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.702+0000 7f8c92ffd700 10 client.48910.objecter ms_handle_connect 0x7f8c80011e80 2024-02-19T21:12:43.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.703+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8c80015b60 con 0x7f8c80011e80 2024-02-19T21:12:43.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.703+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:12:43.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.703+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 7 = system:0 2024-02-19T21:12:43.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.703+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c80003470 osd=3 4 2024-02-19T21:12:43.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.703+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 3 7 2024-02-19T21:12:43.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.722+0000 7f8ca424f700 10 client.48910.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:43.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.722+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 8 2024-02-19T21:12:43.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.722+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c80003470 osd=3 3 2024-02-19T21:12:43.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.722+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c80003470 osd=3 3 2024-02-19T21:12:43.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.722+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c80003470 osd=3 3 2024-02-19T21:12:43.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.722+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 8 2024-02-19T21:12:43.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.722+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c80003470 osd=3 3 2024-02-19T21:12:43.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.722+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 3 8 2024-02-19T21:12:43.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.722+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c80003470 osd=3 4 2024-02-19T21:12:43.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.722+0000 7f8ca424f700 10 client.48910.objecter _send_command 8 2024-02-19T21:12:43.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.722+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c8000e1c0 con 0x7f8c80011e80 2024-02-19T21:12:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.723+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8c8000e1c0 con 0x7f8c80011e80 2024-02-19T21:12:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.723+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.723+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.723+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c80003470 osd=3 4 2024-02-19T21:12:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.723+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 3 8 2024-02-19T21:12:43.723 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:12:43.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.723+0000 7f8ca424f700 10 client.48910.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:43.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.723+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 9 2024-02-19T21:12:43.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.724+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c8000df80 osd=4 3 2024-02-19T21:12:43.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.724+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c8000df80 osd=4 3 2024-02-19T21:12:43.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.724+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c8000df80 osd=4 3 2024-02-19T21:12:43.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.724+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 9 2024-02-19T21:12:43.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.724+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c8000df80 osd=4 3 2024-02-19T21:12:43.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.724+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 4 9 2024-02-19T21:12:43.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.724+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c8000df80 osd=4 4 2024-02-19T21:12:43.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.724+0000 7f8ca424f700 10 client.48910.objecter _send_command 9 2024-02-19T21:12:43.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.724+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c8000e120 con 0x7f8c8000ce50 2024-02-19T21:12:43.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.724+0000 7f8c92ffd700 10 client.48910.objecter ms_handle_connect 0x7f8c8000ce50 2024-02-19T21:12:43.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.725+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8c8000e120 con 0x7f8c8000ce50 2024-02-19T21:12:43.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.725+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:12:43.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.725+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 9 = system:0 2024-02-19T21:12:43.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.725+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c8000df80 osd=4 4 2024-02-19T21:12:43.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.725+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 4 9 2024-02-19T21:12:43.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.739+0000 7f8ca424f700 10 client.48910.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:43.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.739+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 10 2024-02-19T21:12:43.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.739+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c8000df80 osd=4 3 2024-02-19T21:12:43.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.739+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c8000df80 osd=4 3 2024-02-19T21:12:43.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.739+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c8000df80 osd=4 3 2024-02-19T21:12:43.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.739+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 10 2024-02-19T21:12:43.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.739+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c8000df80 osd=4 3 2024-02-19T21:12:43.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.739+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 4 10 2024-02-19T21:12:43.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.739+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c8000df80 osd=4 4 2024-02-19T21:12:43.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.739+0000 7f8ca424f700 10 client.48910.objecter _send_command 10 2024-02-19T21:12:43.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.739+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c8000e120 con 0x7f8c8000ce50 2024-02-19T21:12:43.740 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:43.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8c8000e120 con 0x7f8c8000ce50 2024-02-19T21:12:43.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:43.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c8000df80 osd=4 4 2024-02-19T21:12:43.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 4 10 2024-02-19T21:12:43.741 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:12:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8ca424f700 10 client.48910.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 11 2024-02-19T21:12:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c8000e120 osd=5 3 2024-02-19T21:12:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c8000e120 osd=5 3 2024-02-19T21:12:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c8000e120 osd=5 3 2024-02-19T21:12:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 11 2024-02-19T21:12:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c8000e120 osd=5 3 2024-02-19T21:12:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 5 11 2024-02-19T21:12:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c8000e120 osd=5 4 2024-02-19T21:12:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8ca424f700 10 client.48910.objecter _send_command 11 2024-02-19T21:12:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.741+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c800180c0 con 0x7f8c80017af0 2024-02-19T21:12:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.742+0000 7f8c92ffd700 10 client.48910.objecter ms_handle_connect 0x7f8c80017af0 2024-02-19T21:12:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.742+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8c800180c0 con 0x7f8c80017af0 2024-02-19T21:12:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.742+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:12:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.742+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 11 = system:0 2024-02-19T21:12:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.742+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c8000e120 osd=5 4 2024-02-19T21:12:43.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.742+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 5 11 2024-02-19T21:12:43.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.757+0000 7f8ca424f700 10 client.48910.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:43.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.757+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 12 2024-02-19T21:12:43.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.757+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c8000e120 osd=5 3 2024-02-19T21:12:43.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.757+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c8000e120 osd=5 3 2024-02-19T21:12:43.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.757+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c8000e120 osd=5 3 2024-02-19T21:12:43.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.757+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 12 2024-02-19T21:12:43.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.757+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c8000e120 osd=5 3 2024-02-19T21:12:43.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.757+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 5 12 2024-02-19T21:12:43.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.757+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c8000e120 osd=5 4 2024-02-19T21:12:43.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.757+0000 7f8ca424f700 10 client.48910.objecter _send_command 12 2024-02-19T21:12:43.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.757+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c800180c0 con 0x7f8c80017af0 2024-02-19T21:12:43.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8c800180c0 con 0x7f8c80017af0 2024-02-19T21:12:43.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:43.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c8000e120 osd=5 4 2024-02-19T21:12:43.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 5 12 2024-02-19T21:12:43.759 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:12:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8ca424f700 10 client.48910.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 13 2024-02-19T21:12:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c800180c0 osd=6 3 2024-02-19T21:12:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c800180c0 osd=6 3 2024-02-19T21:12:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c800180c0 osd=6 3 2024-02-19T21:12:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 13 2024-02-19T21:12:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c800180c0 osd=6 3 2024-02-19T21:12:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 6 13 2024-02-19T21:12:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c800180c0 osd=6 4 2024-02-19T21:12:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8ca424f700 10 client.48910.objecter _send_command 13 2024-02-19T21:12:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.759+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c80022d30 con 0x7f8c8001f160 2024-02-19T21:12:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.760+0000 7f8c92ffd700 10 client.48910.objecter ms_handle_connect 0x7f8c8001f160 2024-02-19T21:12:43.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.760+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8c80022d30 con 0x7f8c8001f160 2024-02-19T21:12:43.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.760+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:12:43.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.760+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 13 = system:0 2024-02-19T21:12:43.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.760+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c800180c0 osd=6 4 2024-02-19T21:12:43.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.760+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 6 13 2024-02-19T21:12:43.767 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:43.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.776+0000 7f8ca424f700 10 client.48910.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:43.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.776+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 14 2024-02-19T21:12:43.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.776+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c800180c0 osd=6 3 2024-02-19T21:12:43.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.776+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c800180c0 osd=6 3 2024-02-19T21:12:43.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.776+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c800180c0 osd=6 3 2024-02-19T21:12:43.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.776+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 14 2024-02-19T21:12:43.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.776+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c800180c0 osd=6 3 2024-02-19T21:12:43.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.776+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 6 14 2024-02-19T21:12:43.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.776+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c800180c0 osd=6 4 2024-02-19T21:12:43.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.776+0000 7f8ca424f700 10 client.48910.objecter _send_command 14 2024-02-19T21:12:43.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.776+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c800244e0 con 0x7f8c8001f160 2024-02-19T21:12:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.777+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8c800244e0 con 0x7f8c8001f160 2024-02-19T21:12:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.777+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.777+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.777+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c800180c0 osd=6 4 2024-02-19T21:12:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.777+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 6 14 2024-02-19T21:12:43.778 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:12:43.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.778+0000 7f8ca424f700 10 client.48910.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:43.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.778+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 15 2024-02-19T21:12:43.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.778+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c800244e0 osd=7 3 2024-02-19T21:12:43.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.778+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c800244e0 osd=7 3 2024-02-19T21:12:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.778+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c800244e0 osd=7 3 2024-02-19T21:12:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.778+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 15 2024-02-19T21:12:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.778+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c800244e0 osd=7 3 2024-02-19T21:12:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.778+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 7 15 2024-02-19T21:12:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.778+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c800244e0 osd=7 4 2024-02-19T21:12:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.778+0000 7f8ca424f700 10 client.48910.objecter _send_command 15 2024-02-19T21:12:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.778+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c80025110 con 0x7f8c80024b60 2024-02-19T21:12:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.778+0000 7f8c92ffd700 10 client.48910.objecter ms_handle_connect 0x7f8c80024b60 2024-02-19T21:12:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.779+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f8c80025110 con 0x7f8c80024b60 2024-02-19T21:12:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.779+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:12:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.779+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 15 = system:0 2024-02-19T21:12:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.779+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c800244e0 osd=7 4 2024-02-19T21:12:43.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.779+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 7 15 2024-02-19T21:12:43.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.794+0000 7f8ca424f700 10 client.48910.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:12:43.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.794+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign -1 16 2024-02-19T21:12:43.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.794+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c800244e0 osd=7 3 2024-02-19T21:12:43.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.794+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c800244e0 osd=7 3 2024-02-19T21:12:43.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.794+0000 7f8ca424f700 20 client.48910.objecter _get_session s=0x7f8c800244e0 osd=7 3 2024-02-19T21:12:43.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.794+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_remove -1 16 2024-02-19T21:12:43.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.794+0000 7f8ca424f700 20 client.48910.objecter get_session s=0x7f8c800244e0 osd=7 3 2024-02-19T21:12:43.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.794+0000 7f8ca424f700 15 client.48910.objecter _session_command_op_assign 7 16 2024-02-19T21:12:43.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.794+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c800244e0 osd=7 4 2024-02-19T21:12:43.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.794+0000 7f8ca424f700 10 client.48910.objecter _send_command 16 2024-02-19T21:12:43.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.794+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f8c80025110 con 0x7f8c80024b60 2024-02-19T21:12:43.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8c92ffd700 1 -- 172.21.15.163:0/2215212845 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f8c80025110 con 0x7f8c80024b60 2024-02-19T21:12:43.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8c92ffd700 10 client.48910.objecter ms_dispatch 0x7f8c9c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:12:43.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8c92ffd700 10 client.48910.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8c92ffd700 20 client.48910.objecter put_session s=0x7f8c800244e0 osd=7 4 2024-02-19T21:12:43.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8c92ffd700 15 client.48910.objecter _session_command_op_remove 7 16 2024-02-19T21:12:43.796 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:12:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 10 librados: watch_flush enter 2024-02-19T21:12:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 10 librados: watch_flush exit 2024-02-19T21:12:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 10 client.48910.objecter close_session for osd.0 2024-02-19T21:12:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 >> v1:172.21.15.163:6801/1559418115 conn(0x7f8c80004530 legacy=0x7f8c80002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c800051a0 osd=0 1 2024-02-19T21:12:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 10 client.48910.objecter close_session for osd.1 2024-02-19T21:12:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 >> v1:172.21.15.163:6805/3125047432 conn(0x7f8c80004150 legacy=0x7f8c8000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c80003fe0 osd=1 1 2024-02-19T21:12:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 10 client.48910.objecter close_session for osd.2 2024-02-19T21:12:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 >> v1:172.21.15.163:6809/1563158487 conn(0x7f8c8000d2f0 legacy=0x7f8c8000d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c80003630 osd=2 1 2024-02-19T21:12:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 10 client.48910.objecter close_session for osd.3 2024-02-19T21:12:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 >> v1:172.21.15.163:6813/2112279769 conn(0x7f8c80011e80 legacy=0x7f8c800142b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c80003470 osd=3 1 2024-02-19T21:12:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 10 client.48910.objecter close_session for osd.4 2024-02-19T21:12:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 >> v1:172.21.15.193:6800/3129716965 conn(0x7f8c8000ce50 legacy=0x7f8c80015d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c8000df80 osd=4 1 2024-02-19T21:12:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 10 client.48910.objecter close_session for osd.5 2024-02-19T21:12:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca27ec700 1 -- 172.21.15.163:0/2215212845 reap_dead start 2024-02-19T21:12:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 >> v1:172.21.15.193:6804/805161991 conn(0x7f8c80017af0 legacy=0x7f8c8001d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca27ec700 1 -- 172.21.15.163:0/2215212845 reap_dead start 2024-02-19T21:12:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca27ec700 1 -- 172.21.15.163:0/2215212845 reap_dead start 2024-02-19T21:12:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c8000e120 osd=5 1 2024-02-19T21:12:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 10 client.48910.objecter close_session for osd.6 2024-02-19T21:12:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 >> v1:172.21.15.193:6808/787604615 conn(0x7f8c8001f160 legacy=0x7f8c80021550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c800180c0 osd=6 1 2024-02-19T21:12:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 10 client.48910.objecter close_session for osd.7 2024-02-19T21:12:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 >> v1:172.21.15.193:6812/157927432 conn(0x7f8c80024b60 legacy=0x7f8c80022d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 20 client.48910.objecter put_session s=0x7f8c800244e0 osd=7 1 2024-02-19T21:12:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 20 client.48910.objecter shutdown clearing up homeless session... 2024-02-19T21:12:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 10 client.48910.objecter successfully canceled tick 2024-02-19T21:12:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8c88063c80 legacy=0x7f8c88066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 >> v1:172.21.15.193:6789/0 conn(0x7f8c9c0fd980 legacy=0x7f8c9c0f9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca27ec700 1 -- 172.21.15.163:0/2215212845 reap_dead start 2024-02-19T21:12:43.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 shutdown_connections 2024-02-19T21:12:43.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.796+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 >> 172.21.15.163:0/2215212845 conn(0x7f8c9c0f35a0 msgr2=0x7f8c9c107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:43.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.797+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 shutdown_connections 2024-02-19T21:12:43.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.797+0000 7f8ca424f700 1 -- 172.21.15.163:0/2215212845 wait complete. 2024-02-19T21:12:43.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:43.797+0000 7f8ca424f700 1 librados: shutdown 2024-02-19T21:12:43.800 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.800 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.800 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.800 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.800 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.800 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.800 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.800 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:12:43.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:42.355959+0000 mon.a (mon.0) 2845 : cluster [DBG] osdmap e526: 8 total, 8 up, 6 in 2024-02-19T21:12:44.319 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (23023 ms) 2024-02-19T21:12:44.319 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-02-19T21:12:44.319 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3019 ms) 2024-02-19T21:12:44.320 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-02-19T21:12:44.320 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.320 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3026 ms) 2024-02-19T21:12:44.320 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-02-19T21:12:44.320 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (2924 ms) 2024-02-19T21:12:44.320 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-02-19T21:12:44.320 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.320 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3003 ms) 2024-02-19T21:12:44.320 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-02-19T21:12:44.320 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.320 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4007 ms) 2024-02-19T21:12:44.320 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-02-19T21:12:44.320 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.320 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38226 ms) 2024-02-19T21:12:44.321 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-02-19T21:12:44.321 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.321 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17785 ms) 2024-02-19T21:12:44.321 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-02-19T21:12:44.321 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.321 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4039 ms) 2024-02-19T21:12:44.321 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-02-19T21:12:44.321 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.321 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (2962 ms) 2024-02-19T21:12:44.321 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-02-19T21:12:44.321 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.321 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25009 ms) 2024-02-19T21:12:44.321 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-02-19T21:12:44.321 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.321 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15096 ms) 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (4912 ms) 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: waiting for scrubs... 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: done waiting 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25008 ms) 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (11002 ms) 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9049 ms) 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-02-19T21:12:44.322 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.323 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9039 ms) 2024-02-19T21:12:44.323 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-02-19T21:12:44.323 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.323 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (8941 ms) 2024-02-19T21:12:44.323 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-02-19T21:12:44.323 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:12:44.323 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (5984 ms) 2024-02-19T21:12:44.567 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:43.257024+0000 mgr.y (mgr.14550) 474 : cluster [DBG] pgmap v789: 238 pgs: 1 active+remapped+backfill_toofull, 2 peering, 32 creating+peering, 203 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:44.567 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:43.373568+0000 mon.a (mon.0) 2846 : cluster [DBG] osdmap e527: 8 total, 8 up, 6 in 2024-02-19T21:12:44.567 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:43.400006+0000 mon.b (mon.1) 686 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:44.567 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:43.400389+0000 mon.a (mon.0) 2847 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:44.567 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:43.400872+0000 mon.b (mon.1) 687 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-86"}]: dispatch 2024-02-19T21:12:44.567 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:43.401075+0000 mon.a (mon.0) 2848 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-86"}]: dispatch 2024-02-19T21:12:44.567 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:43.624051+0000 mon.b (mon.1) 688 : audit [DBG] from='client.? v1:172.21.15.163:0/2215212845' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:44.567 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:44.314889+0000 mon.a (mon.0) 2849 : cluster [DBG] osdmap e528: 8 total, 8 up, 6 in 2024-02-19T21:12:44.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:43.257024+0000 mgr.y (mgr.14550) 474 : cluster [DBG] pgmap v789: 238 pgs: 1 active+remapped+backfill_toofull, 2 peering, 32 creating+peering, 203 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:44.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:43.373568+0000 mon.a (mon.0) 2846 : cluster [DBG] osdmap e527: 8 total, 8 up, 6 in 2024-02-19T21:12:44.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:43.400006+0000 mon.b (mon.1) 686 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:43.400389+0000 mon.a (mon.0) 2847 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:43.400872+0000 mon.b (mon.1) 687 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-86"}]: dispatch 2024-02-19T21:12:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:43.401075+0000 mon.a (mon.0) 2848 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-86"}]: dispatch 2024-02-19T21:12:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:43.624051+0000 mon.b (mon.1) 688 : audit [DBG] from='client.? v1:172.21.15.163:0/2215212845' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:44.314889+0000 mon.a (mon.0) 2849 : cluster [DBG] osdmap e528: 8 total, 8 up, 6 in 2024-02-19T21:12:44.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:43.257024+0000 mgr.y (mgr.14550) 474 : cluster [DBG] pgmap v789: 238 pgs: 1 active+remapped+backfill_toofull, 2 peering, 32 creating+peering, 203 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 1/1143 objects misplaced (0.087%) 2024-02-19T21:12:44.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:43.373568+0000 mon.a (mon.0) 2846 : cluster [DBG] osdmap e527: 8 total, 8 up, 6 in 2024-02-19T21:12:44.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:43.400006+0000 mon.b (mon.1) 686 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:44.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:43.400389+0000 mon.a (mon.0) 2847 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:44.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:43.400872+0000 mon.b (mon.1) 687 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-86"}]: dispatch 2024-02-19T21:12:44.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:43.401075+0000 mon.a (mon.0) 2848 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-86"}]: dispatch 2024-02-19T21:12:44.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:43.624051+0000 mon.b (mon.1) 688 : audit [DBG] from='client.? v1:172.21.15.163:0/2215212845' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:44.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:44.314889+0000 mon.a (mon.0) 2849 : cluster [DBG] osdmap e528: 8 total, 8 up, 6 in 2024-02-19T21:12:45.916 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:45.920 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:45.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:45.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:46.027 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:46.057 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:46.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:45.257821+0000 mgr.y (mgr.14550) 475 : cluster [DBG] pgmap v792: 206 pgs: 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:12:46.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:45.317946+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e529: 8 total, 8 up, 6 in 2024-02-19T21:12:46.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:45.324498+0000 mon.b (mon.1) 689 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:46.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:45.325911+0000 mon.a (mon.0) 2851 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:46.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:45.564790+0000 mon.a (mon.0) 2852 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-19T21:12:46.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.585+0000 7fcf7f2b6700 1 -- 172.21.15.163:0/1955119975 >> v1:172.21.15.163:6790/0 conn(0x7fcf780fd980 legacy=0x7fcf780fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:46.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.585+0000 7fcf7f2b6700 1 -- 172.21.15.163:0/1955119975 shutdown_connections 2024-02-19T21:12:46.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.585+0000 7fcf7f2b6700 1 -- 172.21.15.163:0/1955119975 >> 172.21.15.163:0/1955119975 conn(0x7fcf780f35a0 msgr2=0x7fcf780f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:46.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.585+0000 7fcf7f2b6700 1 -- 172.21.15.163:0/1955119975 shutdown_connections 2024-02-19T21:12:46.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.585+0000 7fcf7f2b6700 1 -- 172.21.15.163:0/1955119975 wait complete. 2024-02-19T21:12:46.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.585+0000 7fcf7f2b6700 1 librados: starting msgr at 2024-02-19T21:12:46.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.585+0000 7fcf7f2b6700 1 librados: starting objecter 2024-02-19T21:12:46.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.585+0000 7fcf7f2b6700 1 Processor -- start 2024-02-19T21:12:46.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.585+0000 7fcf7f2b6700 1 -- start start 2024-02-19T21:12:46.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.585+0000 7fcf7f2b6700 1 librados: setting wanted keys 2024-02-19T21:12:46.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.585+0000 7fcf7f2b6700 1 librados: calling monclient init 2024-02-19T21:12:46.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf7f2b6700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcf780fcee0 con 0x7fcf780fd980 2024-02-19T21:12:46.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf7f2b6700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcf780fd070 con 0x7fcf781042c0 2024-02-19T21:12:46.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf7f2b6700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcf780fd260 con 0x7fcf78100780 2024-02-19T21:12:46.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf7c851700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fcf78100780 0x7fcf78114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:47020/0 (socket says 172.21.15.163:47020) 2024-02-19T21:12:46.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf7c851700 1 -- 172.21.15.163:0/2423670799 learned_addr learned my addr 172.21.15.163:0/2423670799 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:46.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 10 client.?.objecter ms_handle_connect 0x7fcf780fd980 2024-02-19T21:12:46.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:46.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 10 client.?.objecter ms_handle_connect 0x7fcf78100780 2024-02-19T21:12:46.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:46.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 10 client.?.objecter ms_handle_connect 0x7fcf781042c0 2024-02-19T21:12:46.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:46.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566377641 0 0) 0x7fcf780fd260 con 0x7fcf78100780 2024-02-19T21:12:46.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf64003800 con 0x7fcf78100780 2024-02-19T21:12:46.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124521321 0 0) 0x7fcf780fcee0 con 0x7fcf780fd980 2024-02-19T21:12:46.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf780fd260 con 0x7fcf780fd980 2024-02-19T21:12:46.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190541741 0 0) 0x7fcf780fd070 con 0x7fcf781042c0 2024-02-19T21:12:46.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf780fcee0 con 0x7fcf781042c0 2024-02-19T21:12:46.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 605778067 0 0) 0x7fcf64003800 con 0x7fcf78100780 2024-02-19T21:12:46.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf780fd070 con 0x7fcf78100780 2024-02-19T21:12:46.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcf68001880 con 0x7fcf78100780 2024-02-19T21:12:46.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 334705429 0 0) 0x7fcf780fcee0 con 0x7fcf781042c0 2024-02-19T21:12:46.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf64003800 con 0x7fcf781042c0 2024-02-19T21:12:46.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2284144615 0 0) 0x7fcf780fd260 con 0x7fcf780fd980 2024-02-19T21:12:46.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf780fcee0 con 0x7fcf780fd980 2024-02-19T21:12:46.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcf700019a0 con 0x7fcf781042c0 2024-02-19T21:12:46.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcf74002f20 con 0x7fcf780fd980 2024-02-19T21:12:46.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4217554781 0 0) 0x7fcf780fd070 con 0x7fcf78100780 2024-02-19T21:12:46.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.586+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 >> v1:172.21.15.193:6789/0 conn(0x7fcf781042c0 legacy=0x7fcf78118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:46.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.587+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 >> v1:172.21.15.163:6789/0 conn(0x7fcf780fd980 legacy=0x7fcf780fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:46.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.587+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7811cb90 con 0x7fcf78100780 2024-02-19T21:12:46.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.587+0000 7fcf7f2b6700 1 -- 172.21.15.163:0/2423670799 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf78119bc0 con 0x7fcf78100780 2024-02-19T21:12:46.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.587+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fcf68001f40 con 0x7fcf78100780 2024-02-19T21:12:46.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.587+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcf68005aa0 con 0x7fcf78100780 2024-02-19T21:12:46.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.587+0000 7fcf7f2b6700 10 client.50108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:46.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.587+0000 7fcf7f2b6700 1 -- 172.21.15.163:0/2423670799 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7811a0b0 con 0x7fcf78100780 2024-02-19T21:12:46.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.587+0000 7fcf7f2b6700 1 librados: init done 2024-02-19T21:12:46.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.588+0000 7fcf7f2b6700 1 -- 172.21.15.163:0/2423670799 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf7805f660 con 0x7fcf78100780 2024-02-19T21:12:46.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.588+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fcf68001aa0 con 0x7fcf78100780 2024-02-19T21:12:46.592 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.588+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(530..530 src has 43..530) v4 ==== 8092+0+0 (unknown 475233731 0 0) 0x7fcf68078d80 con 0x7fcf78100780 2024-02-19T21:12:46.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.588+0000 7fcf6e777700 10 client.50108.objecter ms_dispatch 0x7fcf78001550 osd_map(530..530 src has 43..530) v4 2024-02-19T21:12:46.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.588+0000 7fcf6e777700 3 client.50108.objecter handle_osd_map got epochs [530,530] > 0 2024-02-19T21:12:46.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.588+0000 7fcf6e777700 3 client.50108.objecter handle_osd_map decoding full epoch 530 2024-02-19T21:12:46.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.589+0000 7fcf6e777700 20 client.50108.objecter dump_active .. 0 homeless 2024-02-19T21:12:46.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.589+0000 7fcf6e777700 10 client.50108.objecter ms_handle_connect 0x7fcf64063e20 2024-02-19T21:12:46.593 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.593+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fcf6804c1b0 con 0x7fcf78100780 2024-02-19T21:12:46.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:45.257821+0000 mgr.y (mgr.14550) 475 : cluster [DBG] pgmap v792: 206 pgs: 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:12:46.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:45.317946+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e529: 8 total, 8 up, 6 in 2024-02-19T21:12:46.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:45.324498+0000 mon.b (mon.1) 689 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:46.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:45.325911+0000 mon.a (mon.0) 2851 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:46.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:45.564790+0000 mon.a (mon.0) 2852 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-19T21:12:46.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:45.257821+0000 mgr.y (mgr.14550) 475 : cluster [DBG] pgmap v792: 206 pgs: 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:12:46.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:45.317946+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e529: 8 total, 8 up, 6 in 2024-02-19T21:12:46.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:45.324498+0000 mon.b (mon.1) 689 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:46.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:45.325911+0000 mon.a (mon.0) 2851 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-88","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:46.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:45.564790+0000 mon.a (mon.0) 2852 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-19T21:12:46.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:46.771+0000 7fcf7f2b6700 1 -- 172.21.15.163:0/2423670799 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fcf780f91e0 con 0x7fcf78100780 2024-02-19T21:12:47.330 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:46.322671+0000 mon.a (mon.0) 2853 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-88","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:47.330 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:46.322778+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e530: 8 total, 8 up, 6 in 2024-02-19T21:12:47.330 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:46.337299+0000 mon.b (mon.1) 690 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:12:47.330 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:46.338216+0000 mon.a (mon.0) 2855 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:12:47.330 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:46.772987+0000 mon.c (mon.2) 406 : audit [INF] from='client.? v1:172.21.15.163:0/2423670799' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:12:47.331 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:46.773245+0000 mon.a (mon.0) 2856 : audit [INF] from='client.50108 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:12:47.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:47.333+0000 7fcf6e777700 1 -- 172.21.15.163:0/2423670799 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v531) v1 ==== 91+0+0 (unknown 461353430 0 0) 0x7fcf6804a5c0 con 0x7fcf78100780 2024-02-19T21:12:47.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:47.336+0000 7fcf7f2b6700 10 librados: watch_flush enter 2024-02-19T21:12:47.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:47.336+0000 7fcf7f2b6700 10 librados: watch_flush exit 2024-02-19T21:12:47.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:47.336+0000 7fcf7f2b6700 20 client.50108.objecter shutdown clearing up homeless session... 2024-02-19T21:12:47.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:47.336+0000 7fcf7f2b6700 10 client.50108.objecter successfully canceled tick 2024-02-19T21:12:47.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:47.336+0000 7fcf7f2b6700 1 -- 172.21.15.163:0/2423670799 >> v1:172.21.15.163:6800/1552514681 conn(0x7fcf64063e20 legacy=0x7fcf640662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:47.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:47.336+0000 7fcf7f2b6700 1 -- 172.21.15.163:0/2423670799 >> v1:172.21.15.163:6790/0 conn(0x7fcf78100780 legacy=0x7fcf78114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:47.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:47.336+0000 7fcf7f2b6700 1 -- 172.21.15.163:0/2423670799 shutdown_connections 2024-02-19T21:12:47.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:47.336+0000 7fcf7f2b6700 1 -- 172.21.15.163:0/2423670799 >> 172.21.15.163:0/2423670799 conn(0x7fcf780f35a0 msgr2=0x7fcf78107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:47.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:47.336+0000 7fcf7f2b6700 1 -- 172.21.15.163:0/2423670799 shutdown_connections 2024-02-19T21:12:47.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:47.337+0000 7fcf7f2b6700 1 -- 172.21.15.163:0/2423670799 wait complete. 2024-02-19T21:12:47.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:47.337+0000 7fcf7f2b6700 1 librados: shutdown 2024-02-19T21:12:47.337 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:12:47.693 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:46.322671+0000 mon.a (mon.0) 2853 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-88","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:47.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:46.322778+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e530: 8 total, 8 up, 6 in 2024-02-19T21:12:47.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:46.337299+0000 mon.b (mon.1) 690 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:12:47.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:46.338216+0000 mon.a (mon.0) 2855 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:12:47.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:46.772987+0000 mon.c (mon.2) 406 : audit [INF] from='client.? v1:172.21.15.163:0/2423670799' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:12:47.694 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:46.773245+0000 mon.a (mon.0) 2856 : audit [INF] from='client.50108 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:12:47.771 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:46.322671+0000 mon.a (mon.0) 2853 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-88","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:47.771 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:46.322778+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e530: 8 total, 8 up, 6 in 2024-02-19T21:12:47.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:46.337299+0000 mon.b (mon.1) 690 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:12:47.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:46.338216+0000 mon.a (mon.0) 2855 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:12:47.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:46.772987+0000 mon.c (mon.2) 406 : audit [INF] from='client.? v1:172.21.15.163:0/2423670799' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:12:47.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:46.773245+0000 mon.a (mon.0) 2856 : audit [INF] from='client.50108 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:12:48.070 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:12:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:12:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:12:48.669 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:47.258352+0000 mgr.y (mgr.14550) 476 : cluster [DBG] pgmap v795: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:12:48.669 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:47.320136+0000 mon.a (mon.0) 2857 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:12:48.669 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:47.332652+0000 mon.a (mon.0) 2858 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:12:48.669 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:47.332733+0000 mon.a (mon.0) 2859 : audit [INF] from='client.50108 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:12:48.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:47.332791+0000 mon.a (mon.0) 2860 : cluster [DBG] osdmap e531: 8 total, 8 up, 6 in 2024-02-19T21:12:48.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:47.336822+0000 mon.b (mon.1) 691 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-88", "mode": "writeback"}]: dispatch 2024-02-19T21:12:48.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:47.337226+0000 mon.a (mon.0) 2861 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-88", "mode": "writeback"}]: dispatch 2024-02-19T21:12:48.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:47.425225+0000 osd.5 (osd.5) 214 : cluster [DBG] 16.3 scrub starts 2024-02-19T21:12:48.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:47.428051+0000 osd.5 (osd.5) 215 : cluster [DBG] 16.3 scrub ok 2024-02-19T21:12:48.670 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:47.258352+0000 mgr.y (mgr.14550) 476 : cluster [DBG] pgmap v795: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:12:48.670 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:47.320136+0000 mon.a (mon.0) 2857 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:12:48.670 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:47.332652+0000 mon.a (mon.0) 2858 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:12:48.670 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:47.332733+0000 mon.a (mon.0) 2859 : audit [INF] from='client.50108 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:12:48.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:47.332791+0000 mon.a (mon.0) 2860 : cluster [DBG] osdmap e531: 8 total, 8 up, 6 in 2024-02-19T21:12:48.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:47.336822+0000 mon.b (mon.1) 691 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-88", "mode": "writeback"}]: dispatch 2024-02-19T21:12:48.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:47.337226+0000 mon.a (mon.0) 2861 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-88", "mode": "writeback"}]: dispatch 2024-02-19T21:12:48.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:47.425225+0000 osd.5 (osd.5) 214 : cluster [DBG] 16.3 scrub starts 2024-02-19T21:12:48.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:47.428051+0000 osd.5 (osd.5) 215 : cluster [DBG] 16.3 scrub ok 2024-02-19T21:12:48.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:47.258352+0000 mgr.y (mgr.14550) 476 : cluster [DBG] pgmap v795: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s; 218 B/s, 0 objects/s recovering 2024-02-19T21:12:48.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:47.320136+0000 mon.a (mon.0) 2857 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:12:48.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:47.332652+0000 mon.a (mon.0) 2858 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:12:48.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:47.332733+0000 mon.a (mon.0) 2859 : audit [INF] from='client.50108 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:12:48.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:47.332791+0000 mon.a (mon.0) 2860 : cluster [DBG] osdmap e531: 8 total, 8 up, 6 in 2024-02-19T21:12:48.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:47.336822+0000 mon.b (mon.1) 691 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-88", "mode": "writeback"}]: dispatch 2024-02-19T21:12:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:47.337226+0000 mon.a (mon.0) 2861 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-88", "mode": "writeback"}]: dispatch 2024-02-19T21:12:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:47.425225+0000 osd.5 (osd.5) 214 : cluster [DBG] 16.3 scrub starts 2024-02-19T21:12:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:47.428051+0000 osd.5 (osd.5) 215 : cluster [DBG] 16.3 scrub ok 2024-02-19T21:12:49.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:48.326864+0000 mon.a (mon.0) 2862 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:12:49.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:48.333971+0000 mon.a (mon.0) 2863 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-88", "mode": "writeback"}]': finished 2024-02-19T21:12:49.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:48.334050+0000 mon.a (mon.0) 2864 : cluster [DBG] osdmap e532: 8 total, 8 up, 6 in 2024-02-19T21:12:49.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:48.339465+0000 mon.b (mon.1) 692 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-88"}]: dispatch 2024-02-19T21:12:49.579 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:48.339890+0000 mon.a (mon.0) 2865 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-88"}]: dispatch 2024-02-19T21:12:49.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:48.425511+0000 osd.5 (osd.5) 216 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:12:49.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:48.430215+0000 osd.5 (osd.5) 217 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:12:49.580 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:48.861970+0000 mon.a (mon.0) 2866 : audit [INF] from='client.? v1:172.21.15.163:0/105338450' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:12:49.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:48.326864+0000 mon.a (mon.0) 2862 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:12:49.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:48.333971+0000 mon.a (mon.0) 2863 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-88", "mode": "writeback"}]': finished 2024-02-19T21:12:49.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:48.334050+0000 mon.a (mon.0) 2864 : cluster [DBG] osdmap e532: 8 total, 8 up, 6 in 2024-02-19T21:12:49.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:48.339465+0000 mon.b (mon.1) 692 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-88"}]: dispatch 2024-02-19T21:12:49.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:48.339890+0000 mon.a (mon.0) 2865 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-88"}]: dispatch 2024-02-19T21:12:49.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:48.425511+0000 osd.5 (osd.5) 216 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:12:49.580 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:48.430215+0000 osd.5 (osd.5) 217 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:12:49.581 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:48.861970+0000 mon.a (mon.0) 2866 : audit [INF] from='client.? v1:172.21.15.163:0/105338450' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:12:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.672+0000 7fd10bb40700 1 -- 172.21.15.163:0/2690813358 >> v1:172.21.15.163:6790/0 conn(0x7fd1041031b0 legacy=0x7fd104105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.672+0000 7fd10bb40700 1 -- 172.21.15.163:0/2690813358 shutdown_connections 2024-02-19T21:12:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.672+0000 7fd10bb40700 1 -- 172.21.15.163:0/2690813358 >> 172.21.15.163:0/2690813358 conn(0x7fd1040f2470 msgr2=0x7fd1040f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.672+0000 7fd10bb40700 1 -- 172.21.15.163:0/2690813358 shutdown_connections 2024-02-19T21:12:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.672+0000 7fd10bb40700 1 -- 172.21.15.163:0/2690813358 wait complete. 2024-02-19T21:12:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd10bb40700 1 librados: starting msgr at 2024-02-19T21:12:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd10bb40700 1 librados: starting objecter 2024-02-19T21:12:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd10bb40700 1 Processor -- start 2024-02-19T21:12:49.683 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd10bb40700 1 -- start start 2024-02-19T21:12:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd10bb40700 1 librados: setting wanted keys 2024-02-19T21:12:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd10bb40700 1 librados: calling monclient init 2024-02-19T21:12:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd10bb40700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd1041162a0 con 0x7fd1040ff670 2024-02-19T21:12:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd10bb40700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd1041174a0 con 0x7fd1040fc870 2024-02-19T21:12:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd10bb40700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd1041186a0 con 0x7fd1041031b0 2024-02-19T21:12:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd1090db700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fd1040ff670 0x7fd1040f9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:51356/0 (socket says 172.21.15.163:51356) 2024-02-19T21:12:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd1090db700 1 -- 172.21.15.163:0/105338450 learned_addr learned my addr 172.21.15.163:0/105338450 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd10a0dd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fd1041031b0 0x7fd104114960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:47044/0 (socket says 172.21.15.163:47044) 2024-02-19T21:12:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd0faffd700 10 client.?.objecter ms_handle_connect 0x7fd1040ff670 2024-02-19T21:12:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd0faffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd0faffd700 10 client.?.objecter ms_handle_connect 0x7fd1041031b0 2024-02-19T21:12:49.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd0faffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd0faffd700 10 client.?.objecter ms_handle_connect 0x7fd1040fc870 2024-02-19T21:12:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd0faffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268786571 0 0) 0x7fd1041162a0 con 0x7fd1040ff670 2024-02-19T21:12:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.673+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0f0003030 con 0x7fd1040ff670 2024-02-19T21:12:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403156254 0 0) 0x7fd1041186a0 con 0x7fd1041031b0 2024-02-19T21:12:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1041162a0 con 0x7fd1041031b0 2024-02-19T21:12:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898642674 0 0) 0x7fd1041174a0 con 0x7fd1040fc870 2024-02-19T21:12:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1041186a0 con 0x7fd1040fc870 2024-02-19T21:12:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4262809250 0 0) 0x7fd0f0003030 con 0x7fd1040ff670 2024-02-19T21:12:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1041174a0 con 0x7fd1040ff670 2024-02-19T21:12:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3951136953 0 0) 0x7fd1041162a0 con 0x7fd1041031b0 2024-02-19T21:12:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0f0003030 con 0x7fd1041031b0 2024-02-19T21:12:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd0f4001970 con 0x7fd1040ff670 2024-02-19T21:12:49.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd0fc001360 con 0x7fd1041031b0 2024-02-19T21:12:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3166943024 0 0) 0x7fd1041186a0 con 0x7fd1040fc870 2024-02-19T21:12:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1041162a0 con 0x7fd1040fc870 2024-02-19T21:12:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd1000018c0 con 0x7fd1040fc870 2024-02-19T21:12:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 966825827 0 0) 0x7fd1041174a0 con 0x7fd1040ff670 2024-02-19T21:12:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 >> v1:172.21.15.163:6790/0 conn(0x7fd1041031b0 legacy=0x7fd104114960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 >> v1:172.21.15.193:6789/0 conn(0x7fd1040fc870 legacy=0x7fd1040f8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1041198a0 con 0x7fd1040ff670 2024-02-19T21:12:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd10bb40700 1 -- 172.21.15.163:0/105338450 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd104118890 con 0x7fd1040ff670 2024-02-19T21:12:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd10bb40700 10 client.23385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd10bb40700 1 -- 172.21.15.163:0/105338450 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd104118d00 con 0x7fd1040ff670 2024-02-19T21:12:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd10bb40700 1 librados: init done 2024-02-19T21:12:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fd0f4002380 con 0x7fd1040ff670 2024-02-19T21:12:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.674+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd0f4005a40 con 0x7fd1040ff670 2024-02-19T21:12:49.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.675+0000 7fd10bb40700 1 -- 172.21.15.163:0/105338450 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e8004f20 con 0x7fd1040ff670 2024-02-19T21:12:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.675+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fd0f40024f0 con 0x7fd1040ff670 2024-02-19T21:12:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.676+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(532..532 src has 43..532) v4 ==== 8100+0+0 (unknown 1408415120 0 0) 0x7fd0f4078cc0 con 0x7fd1040ff670 2024-02-19T21:12:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.676+0000 7fd0faffd700 10 client.23385.objecter ms_dispatch 0x7fd104001550 osd_map(532..532 src has 43..532) v4 2024-02-19T21:12:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.676+0000 7fd0faffd700 3 client.23385.objecter handle_osd_map got epochs [532,532] > 0 2024-02-19T21:12:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.676+0000 7fd0faffd700 3 client.23385.objecter handle_osd_map decoding full epoch 532 2024-02-19T21:12:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.676+0000 7fd0faffd700 20 client.23385.objecter dump_active .. 0 homeless 2024-02-19T21:12:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.676+0000 7fd0faffd700 10 client.23385.objecter ms_handle_connect 0x7fd0f0063c40 2024-02-19T21:12:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.680+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd0f404c0f0 con 0x7fd1040ff670 2024-02-19T21:12:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:48.861+0000 7fd10bb40700 1 -- 172.21.15.163:0/105338450 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7fd0e80051d0 con 0x7fd1040ff670 2024-02-19T21:12:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:49.347+0000 7fd0faffd700 1 -- 172.21.15.163:0/105338450 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v533) v1 ==== 113+0+0 (unknown 104521377 0 0) 0x7fd0f404a500 con 0x7fd1040ff670 2024-02-19T21:12:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:49.349+0000 7fd10bb40700 10 librados: watch_flush enter 2024-02-19T21:12:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:49.349+0000 7fd10bb40700 10 librados: watch_flush exit 2024-02-19T21:12:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:49.349+0000 7fd10bb40700 20 client.23385.objecter shutdown clearing up homeless session... 2024-02-19T21:12:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:49.349+0000 7fd10bb40700 10 client.23385.objecter successfully canceled tick 2024-02-19T21:12:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:49.349+0000 7fd10bb40700 1 -- 172.21.15.163:0/105338450 >> v1:172.21.15.163:6800/1552514681 conn(0x7fd0f0063c40 legacy=0x7fd0f0066100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:49.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:49.349+0000 7fd10bb40700 1 -- 172.21.15.163:0/105338450 >> v1:172.21.15.163:6789/0 conn(0x7fd1040ff670 legacy=0x7fd1040f9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:49.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:49.349+0000 7fd10bb40700 1 -- 172.21.15.163:0/105338450 shutdown_connections 2024-02-19T21:12:49.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:49.349+0000 7fd10bb40700 1 -- 172.21.15.163:0/105338450 >> 172.21.15.163:0/105338450 conn(0x7fd1040f2470 msgr2=0x7fd104106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:49.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:49.349+0000 7fd10bb40700 1 -- 172.21.15.163:0/105338450 shutdown_connections 2024-02-19T21:12:49.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:49.349+0000 7fd10bb40700 1 -- 172.21.15.163:0/105338450 wait complete. 2024-02-19T21:12:49.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:49.349+0000 7fd10bb40700 1 librados: shutdown 2024-02-19T21:12:49.688 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.7 to 1 (10000) 2024-02-19T21:12:49.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:48.326864+0000 mon.a (mon.0) 2862 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:12:49.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:48.333971+0000 mon.a (mon.0) 2863 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-88", "mode": "writeback"}]': finished 2024-02-19T21:12:49.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:48.334050+0000 mon.a (mon.0) 2864 : cluster [DBG] osdmap e532: 8 total, 8 up, 6 in 2024-02-19T21:12:49.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:48.339465+0000 mon.b (mon.1) 692 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-88"}]: dispatch 2024-02-19T21:12:49.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:48.339890+0000 mon.a (mon.0) 2865 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-88"}]: dispatch 2024-02-19T21:12:49.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:48.425511+0000 osd.5 (osd.5) 216 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:12:49.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:48.430215+0000 osd.5 (osd.5) 217 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:12:49.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:48.861970+0000 mon.a (mon.0) 2866 : audit [INF] from='client.? v1:172.21.15.163:0/105338450' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:12:50.236 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 3 1 2024-02-19T21:12:50.337 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:12:50.337 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:12:50.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:48.419320+0000 osd.7 (osd.7) 239 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:12:50.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:48.422757+0000 osd.7 (osd.7) 240 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:12:50.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:49.137824+0000 osd.2 (osd.2) 214 : cluster [DBG] 16.1 scrub starts 2024-02-19T21:12:50.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:49.142153+0000 osd.2 (osd.2) 215 : cluster [DBG] 16.1 scrub ok 2024-02-19T21:12:50.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:49.258748+0000 mgr.y (mgr.14550) 477 : cluster [DBG] pgmap v798: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:12:50.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:49.343870+0000 mon.a (mon.0) 2867 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-88"}]': finished 2024-02-19T21:12:50.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:49.343969+0000 mon.a (mon.0) 2868 : audit [INF] from='client.? v1:172.21.15.163:0/105338450' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-19T21:12:50.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:49.344018+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e533: 8 total, 8 up, 6 in 2024-02-19T21:12:50.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:49.346673+0000 mon.b (mon.1) 693 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:12:50.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:49.347080+0000 mon.a (mon.0) 2870 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:12:50.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:50.241918+0000 osd.3 (osd.3) 229 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:12:50.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:50.243493+0000 osd.3 (osd.3) 230 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:12:50.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:48.419320+0000 osd.7 (osd.7) 239 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:12:50.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:48.422757+0000 osd.7 (osd.7) 240 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:12:50.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:49.137824+0000 osd.2 (osd.2) 214 : cluster [DBG] 16.1 scrub starts 2024-02-19T21:12:50.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:49.142153+0000 osd.2 (osd.2) 215 : cluster [DBG] 16.1 scrub ok 2024-02-19T21:12:50.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:49.258748+0000 mgr.y (mgr.14550) 477 : cluster [DBG] pgmap v798: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:12:50.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:49.343870+0000 mon.a (mon.0) 2867 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-88"}]': finished 2024-02-19T21:12:50.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:49.343969+0000 mon.a (mon.0) 2868 : audit [INF] from='client.? v1:172.21.15.163:0/105338450' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-19T21:12:50.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:49.344018+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e533: 8 total, 8 up, 6 in 2024-02-19T21:12:50.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:49.346673+0000 mon.b (mon.1) 693 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:12:50.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:49.347080+0000 mon.a (mon.0) 2870 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:12:50.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:50.241918+0000 osd.3 (osd.3) 229 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:12:50.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:50.243493+0000 osd.3 (osd.3) 230 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:12:50.639 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:50.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:50.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:50.743 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:50.772 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:50.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:12:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:12:50.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:48.419320+0000 osd.7 (osd.7) 239 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:12:50.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:48.422757+0000 osd.7 (osd.7) 240 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:12:50.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:49.137824+0000 osd.2 (osd.2) 214 : cluster [DBG] 16.1 scrub starts 2024-02-19T21:12:50.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:49.142153+0000 osd.2 (osd.2) 215 : cluster [DBG] 16.1 scrub ok 2024-02-19T21:12:50.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:49.258748+0000 mgr.y (mgr.14550) 477 : cluster [DBG] pgmap v798: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:12:50.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:49.343870+0000 mon.a (mon.0) 2867 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-88"}]': finished 2024-02-19T21:12:50.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:49.343969+0000 mon.a (mon.0) 2868 : audit [INF] from='client.? v1:172.21.15.163:0/105338450' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-19T21:12:50.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:49.344018+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e533: 8 total, 8 up, 6 in 2024-02-19T21:12:50.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:49.346673+0000 mon.b (mon.1) 693 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:12:50.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:49.347080+0000 mon.a (mon.0) 2870 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:12:50.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:50.241918+0000 osd.3 (osd.3) 229 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:12:50.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:50.243493+0000 osd.3 (osd.3) 230 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:12:50.949 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-19T21:12:50.970 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:50.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:50.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:51.025 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:51.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:51.102 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:51.179 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:12:51.300 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:12:51.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:51.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:12:51.404 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:51.436 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:51.456 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:51.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:51.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:51.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:50.027571+0000 osd.1 (osd.1) 247 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:12:51.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:50.030852+0000 osd.1 (osd.1) 248 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:12:51.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:50.340399+0000 mon.a (mon.0) 2871 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:12:51.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:50.347670+0000 mon.a (mon.0) 2872 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:12:51.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:50.347750+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e534: 8 total, 8 up, 6 in 2024-02-19T21:12:51.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:50.349072+0000 mon.b (mon.1) 694 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-19T21:12:51.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:50.349709+0000 mon.a (mon.0) 2874 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-19T21:12:51.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:50.543341+0000 mgr.y (mgr.14550) 478 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:51.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:51.346722+0000 mon.a (mon.0) 2875 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_count","val": "1"}]': finished 2024-02-19T21:12:51.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:51.346796+0000 mon.a (mon.0) 2876 : cluster [DBG] osdmap e535: 8 total, 8 up, 6 in 2024-02-19T21:12:51.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:51.348114+0000 mon.b (mon.1) 695 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:12:51.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:51.348419+0000 mon.a (mon.0) 2877 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:12:51.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:50.027571+0000 osd.1 (osd.1) 247 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:12:51.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:50.030852+0000 osd.1 (osd.1) 248 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:12:51.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:50.340399+0000 mon.a (mon.0) 2871 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:12:51.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:50.347670+0000 mon.a (mon.0) 2872 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:12:51.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:50.347750+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e534: 8 total, 8 up, 6 in 2024-02-19T21:12:51.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:50.349072+0000 mon.b (mon.1) 694 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-19T21:12:51.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:50.349709+0000 mon.a (mon.0) 2874 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-19T21:12:51.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:50.543341+0000 mgr.y (mgr.14550) 478 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:51.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:51.346722+0000 mon.a (mon.0) 2875 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_count","val": "1"}]': finished 2024-02-19T21:12:51.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:51.346796+0000 mon.a (mon.0) 2876 : cluster [DBG] osdmap e535: 8 total, 8 up, 6 in 2024-02-19T21:12:51.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:51.348114+0000 mon.b (mon.1) 695 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:12:51.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:51.348419+0000 mon.a (mon.0) 2877 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:12:51.560 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:51.589 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:51.597 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:12:51.609 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:51.610 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:50.027571+0000 osd.1 (osd.1) 247 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:12:51.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:50.030852+0000 osd.1 (osd.1) 248 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:12:51.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:50.340399+0000 mon.a (mon.0) 2871 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:12:51.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:50.347670+0000 mon.a (mon.0) 2872 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:12:51.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:50.347750+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e534: 8 total, 8 up, 6 in 2024-02-19T21:12:51.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:50.349072+0000 mon.b (mon.1) 694 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-19T21:12:51.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:50.349709+0000 mon.a (mon.0) 2874 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_count","val": "1"}]: dispatch 2024-02-19T21:12:51.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:50.543341+0000 mgr.y (mgr.14550) 478 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:12:51.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:51.346722+0000 mon.a (mon.0) 2875 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_count","val": "1"}]': finished 2024-02-19T21:12:51.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:51.346796+0000 mon.a (mon.0) 2876 : cluster [DBG] osdmap e535: 8 total, 8 up, 6 in 2024-02-19T21:12:51.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:51.348114+0000 mon.b (mon.1) 695 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:12:51.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:51.348419+0000 mon.a (mon.0) 2877 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:12:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:51.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:51.713 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:51.740 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:51.868 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:51.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:51.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:51.908 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:51.909 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:51.972 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:51.999 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:52.500 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:52.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:51.259186+0000 mgr.y (mgr.14550) 479 : cluster [DBG] pgmap v801: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:12:52.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:51.418960+0000 osd.4 (osd.4) 193 : cluster [DBG] 4.c scrub starts 2024-02-19T21:12:52.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:51.420393+0000 osd.4 (osd.4) 194 : cluster [DBG] 4.c scrub ok 2024-02-19T21:12:52.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:51.466100+0000 osd.5 (osd.5) 218 : cluster [DBG] 6.a scrub starts 2024-02-19T21:12:52.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:51.468968+0000 osd.5 (osd.5) 219 : cluster [DBG] 6.a scrub ok 2024-02-19T21:12:52.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:52.347334+0000 mon.a (mon.0) 2878 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:12:52.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:52.347407+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e536: 8 total, 8 up, 6 in 2024-02-19T21:12:52.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:52.350692+0000 mon.b (mon.1) 696 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-19T21:12:52.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:52.351578+0000 mon.a (mon.0) 2880 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-19T21:12:52.698 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:51.259186+0000 mgr.y (mgr.14550) 479 : cluster [DBG] pgmap v801: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:12:52.698 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:51.418960+0000 osd.4 (osd.4) 193 : cluster [DBG] 4.c scrub starts 2024-02-19T21:12:52.698 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:51.420393+0000 osd.4 (osd.4) 194 : cluster [DBG] 4.c scrub ok 2024-02-19T21:12:52.698 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:51.466100+0000 osd.5 (osd.5) 218 : cluster [DBG] 6.a scrub starts 2024-02-19T21:12:52.698 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:51.468968+0000 osd.5 (osd.5) 219 : cluster [DBG] 6.a scrub ok 2024-02-19T21:12:52.698 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:52.347334+0000 mon.a (mon.0) 2878 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:12:52.698 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:52.347407+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e536: 8 total, 8 up, 6 in 2024-02-19T21:12:52.698 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:52.350692+0000 mon.b (mon.1) 696 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-19T21:12:52.698 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:52.351578+0000 mon.a (mon.0) 2880 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-19T21:12:52.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:51.259186+0000 mgr.y (mgr.14550) 479 : cluster [DBG] pgmap v801: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:12:52.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:51.418960+0000 osd.4 (osd.4) 193 : cluster [DBG] 4.c scrub starts 2024-02-19T21:12:52.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:51.420393+0000 osd.4 (osd.4) 194 : cluster [DBG] 4.c scrub ok 2024-02-19T21:12:52.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:51.466100+0000 osd.5 (osd.5) 218 : cluster [DBG] 6.a scrub starts 2024-02-19T21:12:52.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:51.468968+0000 osd.5 (osd.5) 219 : cluster [DBG] 6.a scrub ok 2024-02-19T21:12:52.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:52.347334+0000 mon.a (mon.0) 2878 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:12:52.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:52.347407+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e536: 8 total, 8 up, 6 in 2024-02-19T21:12:52.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:52.350692+0000 mon.b (mon.1) 696 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-19T21:12:52.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:52.351578+0000 mon.a (mon.0) 2880 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "target_max_objects","val": "250"}]: dispatch 2024-02-19T21:12:53.667 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:12:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:12:53] "GET /metrics HTTP/1.1" 200 187969 "" "Prometheus/2.33.4" 2024-02-19T21:12:53.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:51.406963+0000 osd.7 (osd.7) 241 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:12:53.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:51.411566+0000 osd.7 (osd.7) 242 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:12:53.667 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:53.349199+0000 mon.a (mon.0) 2881 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "target_max_objects","val": "250"}]': finished 2024-02-19T21:12:53.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:53.349282+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e537: 8 total, 8 up, 6 in 2024-02-19T21:12:53.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:51.406963+0000 osd.7 (osd.7) 241 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:12:53.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:51.411566+0000 osd.7 (osd.7) 242 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:12:53.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:53.349199+0000 mon.a (mon.0) 2881 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "target_max_objects","val": "250"}]': finished 2024-02-19T21:12:53.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:53.349282+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e537: 8 total, 8 up, 6 in 2024-02-19T21:12:53.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:51.406963+0000 osd.7 (osd.7) 241 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:12:53.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:51.411566+0000 osd.7 (osd.7) 242 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:12:53.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:53.349199+0000 mon.a (mon.0) 2881 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-88","var": "target_max_objects","val": "250"}]': finished 2024-02-19T21:12:53.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:53.349282+0000 mon.a (mon.0) 2882 : cluster [DBG] osdmap e537: 8 total, 8 up, 6 in 2024-02-19T21:12:54.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.199+0000 7f887921b700 1 -- 172.21.15.163:0/4135962736 >> v1:172.21.15.163:6789/0 conn(0x7f88741042c0 legacy=0x7f8874106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.199+0000 7f887921b700 1 -- 172.21.15.163:0/4135962736 shutdown_connections 2024-02-19T21:12:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.199+0000 7f887921b700 1 -- 172.21.15.163:0/4135962736 >> 172.21.15.163:0/4135962736 conn(0x7f88740f35a0 msgr2=0x7f88740f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.199+0000 7f887921b700 1 -- 172.21.15.163:0/4135962736 shutdown_connections 2024-02-19T21:12:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.199+0000 7f887921b700 1 -- 172.21.15.163:0/4135962736 wait complete. 2024-02-19T21:12:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.200+0000 7f887921b700 1 librados: starting msgr at 2024-02-19T21:12:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.200+0000 7f887921b700 1 librados: starting objecter 2024-02-19T21:12:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.200+0000 7f887921b700 1 Processor -- start 2024-02-19T21:12:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.200+0000 7f887921b700 1 -- start start 2024-02-19T21:12:54.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.200+0000 7f887921b700 1 librados: setting wanted keys 2024-02-19T21:12:54.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.200+0000 7f887921b700 1 librados: calling monclient init 2024-02-19T21:12:54.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.200+0000 7f887921b700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f887406cb80 con 0x7f8874100780 2024-02-19T21:12:54.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.200+0000 7f887921b700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f887406cd50 con 0x7f88741042c0 2024-02-19T21:12:54.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.200+0000 7f887921b700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f887406cf40 con 0x7f88740fd980 2024-02-19T21:12:54.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f8872d9d700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f88740fd980 0x7f88740fc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:47066/0 (socket says 172.21.15.163:47066) 2024-02-19T21:12:54.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f8872d9d700 1 -- 172.21.15.163:0/3028537366 learned_addr learned my addr 172.21.15.163:0/3028537366 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:54.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f887259c700 1 --1- 172.21.15.163:0/3028537366 >> v1:172.21.15.163:6789/0 conn(0x7f8874100780 0x7f8874114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:51372/0 (socket says 172.21.15.163:51372) 2024-02-19T21:12:54.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 10 client.?.objecter ms_handle_connect 0x7f88740fd980 2024-02-19T21:12:54.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:54.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 10 client.?.objecter ms_handle_connect 0x7f8874100780 2024-02-19T21:12:54.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:54.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 10 client.?.objecter ms_handle_connect 0x7f88741042c0 2024-02-19T21:12:54.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:54.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058604102 0 0) 0x7f887406cf40 con 0x7f88740fd980 2024-02-19T21:12:54.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8860003030 con 0x7f88740fd980 2024-02-19T21:12:54.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897987967 0 0) 0x7f887406cb80 con 0x7f8874100780 2024-02-19T21:12:54.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f887406cf40 con 0x7f8874100780 2024-02-19T21:12:54.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278645413 0 0) 0x7f887406cd50 con 0x7f88741042c0 2024-02-19T21:12:54.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f887406cb80 con 0x7f88741042c0 2024-02-19T21:12:54.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 180439070 0 0) 0x7f8860003030 con 0x7f88740fd980 2024-02-19T21:12:54.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f887406cd50 con 0x7f88740fd980 2024-02-19T21:12:54.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8864001d70 con 0x7f88740fd980 2024-02-19T21:12:54.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3446228818 0 0) 0x7f887406cf40 con 0x7f8874100780 2024-02-19T21:12:54.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8860003030 con 0x7f8874100780 2024-02-19T21:12:54.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f885c001840 con 0x7f8874100780 2024-02-19T21:12:54.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1683720984 0 0) 0x7f887406cb80 con 0x7f88741042c0 2024-02-19T21:12:54.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f887406cf40 con 0x7f88741042c0 2024-02-19T21:12:54.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f886c0015d0 con 0x7f88741042c0 2024-02-19T21:12:54.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 347280168 0 0) 0x7f887406cd50 con 0x7f88740fd980 2024-02-19T21:12:54.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 >> v1:172.21.15.193:6789/0 conn(0x7f88741042c0 legacy=0x7f8874118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:54.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 >> v1:172.21.15.163:6789/0 conn(0x7f8874100780 legacy=0x7f8874114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:54.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887411cbd0 con 0x7f88740fd980 2024-02-19T21:12:54.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f887921b700 1 -- 172.21.15.163:0/3028537366 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8874119bc0 con 0x7f88740fd980 2024-02-19T21:12:54.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f887921b700 10 client.50120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:54.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f887921b700 1 -- 172.21.15.163:0/3028537366 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f887411a030 con 0x7f88740fd980 2024-02-19T21:12:54.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f887921b700 1 librados: init done 2024-02-19T21:12:54.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.201+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8864002830 con 0x7f88740fd980 2024-02-19T21:12:54.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.202+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8864005e70 con 0x7f88740fd980 2024-02-19T21:12:54.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.202+0000 7f887921b700 1 -- 172.21.15.163:0/3028537366 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8854004f20 con 0x7f88740fd980 2024-02-19T21:12:54.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.203+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f8864018de0 con 0x7f88740fd980 2024-02-19T21:12:54.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.203+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(537..537 src has 43..537) v4 ==== 8126+0+0 (unknown 2364564488 0 0) 0x7f8864079100 con 0x7f88740fd980 2024-02-19T21:12:54.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.203+0000 7f886bfff700 10 client.50120.objecter ms_dispatch 0x7f8874001550 osd_map(537..537 src has 43..537) v4 2024-02-19T21:12:54.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.203+0000 7f886bfff700 3 client.50120.objecter handle_osd_map got epochs [537,537] > 0 2024-02-19T21:12:54.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.203+0000 7f886bfff700 3 client.50120.objecter handle_osd_map decoding full epoch 537 2024-02-19T21:12:54.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.203+0000 7f886bfff700 20 client.50120.objecter dump_active .. 0 homeless 2024-02-19T21:12:54.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.203+0000 7f886bfff700 10 client.50120.objecter ms_handle_connect 0x7f8860063c30 2024-02-19T21:12:54.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.207+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f886404c660 con 0x7f88740fd980 2024-02-19T21:12:54.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:54.400+0000 7f887921b700 1 -- 172.21.15.163:0/3028537366 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f88540051d0 con 0x7f88740fd980 2024-02-19T21:12:54.671 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:53.259685+0000 mgr.y (mgr.14550) 480 : cluster [DBG] pgmap v804: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:12:54.671 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:53.374682+0000 mon.b (mon.1) 697 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:54.671 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:53.374976+0000 mon.a (mon.0) 2883 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:54.671 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:54.105724+0000 mon.a (mon.0) 2884 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:54.672 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:54.319326+0000 mon.a (mon.0) 2885 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:12:54.672 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:54.319403+0000 mon.a (mon.0) 2886 : cluster [DBG] osdmap e538: 8 total, 8 up, 6 in 2024-02-19T21:12:54.672 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:54.320985+0000 mon.b (mon.1) 698 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88"}]: dispatch 2024-02-19T21:12:54.672 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:54.321355+0000 mon.a (mon.0) 2887 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88"}]: dispatch 2024-02-19T21:12:54.672 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:53.259685+0000 mgr.y (mgr.14550) 480 : cluster [DBG] pgmap v804: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:12:54.672 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:53.374682+0000 mon.b (mon.1) 697 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:54.672 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:53.374976+0000 mon.a (mon.0) 2883 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:54.672 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:54.105724+0000 mon.a (mon.0) 2884 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:54.672 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:54.319326+0000 mon.a (mon.0) 2885 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:12:54.673 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:54.319403+0000 mon.a (mon.0) 2886 : cluster [DBG] osdmap e538: 8 total, 8 up, 6 in 2024-02-19T21:12:54.673 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:54.320985+0000 mon.b (mon.1) 698 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88"}]: dispatch 2024-02-19T21:12:54.673 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:54.321355+0000 mon.a (mon.0) 2887 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88"}]: dispatch 2024-02-19T21:12:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:53.259685+0000 mgr.y (mgr.14550) 480 : cluster [DBG] pgmap v804: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:12:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:53.374682+0000 mon.b (mon.1) 697 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:54.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:53.374976+0000 mon.a (mon.0) 2883 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:12:54.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:54.105724+0000 mon.a (mon.0) 2884 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:12:54.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:54.319326+0000 mon.a (mon.0) 2885 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:12:54.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:54.319403+0000 mon.a (mon.0) 2886 : cluster [DBG] osdmap e538: 8 total, 8 up, 6 in 2024-02-19T21:12:54.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:54.320985+0000 mon.b (mon.1) 698 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88"}]: dispatch 2024-02-19T21:12:54.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:54.321355+0000 mon.a (mon.0) 2887 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88"}]: dispatch 2024-02-19T21:12:55.254 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:55.258 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:55.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:55.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:55.327+0000 7f886bfff700 1 -- 172.21.15.163:0/3028537366 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v539) v1 ==== 113+0+0 (unknown 3122561795 0 0) 0x7f886404aa70 con 0x7f88740fd980 2024-02-19T21:12:55.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:55.329+0000 7f887921b700 10 librados: watch_flush enter 2024-02-19T21:12:55.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:55.329+0000 7f887921b700 10 librados: watch_flush exit 2024-02-19T21:12:55.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:55.329+0000 7f887921b700 20 client.50120.objecter shutdown clearing up homeless session... 2024-02-19T21:12:55.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:55.329+0000 7f887921b700 10 client.50120.objecter successfully canceled tick 2024-02-19T21:12:55.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:55.329+0000 7f887921b700 1 -- 172.21.15.163:0/3028537366 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8860063c30 legacy=0x7f88600660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:55.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:55.329+0000 7f887921b700 1 -- 172.21.15.163:0/3028537366 >> v1:172.21.15.163:6790/0 conn(0x7f88740fd980 legacy=0x7f88740fc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:55.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:55.329+0000 7f887921b700 1 -- 172.21.15.163:0/3028537366 shutdown_connections 2024-02-19T21:12:55.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:55.329+0000 7f887921b700 1 -- 172.21.15.163:0/3028537366 >> 172.21.15.163:0/3028537366 conn(0x7f88740f35a0 msgr2=0x7f8874107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:55.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:55.329+0000 7f887921b700 1 -- 172.21.15.163:0/3028537366 shutdown_connections 2024-02-19T21:12:55.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:55.329+0000 7f887921b700 1 -- 172.21.15.163:0/3028537366 wait complete. 2024-02-19T21:12:55.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:55.329+0000 7f887921b700 1 librados: shutdown 2024-02-19T21:12:55.329 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.3 to 1 (10000) 2024-02-19T21:12:55.360 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:55.388 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:55.408 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:55.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:55.512 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:12:55.543 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:55.715 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:54.400526+0000 mon.c (mon.2) 407 : audit [INF] from='client.? v1:172.21.15.163:0/3028537366' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:12:55.715 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:54.404701+0000 mon.a (mon.0) 2888 : audit [INF] from='client.50120 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:12:55.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:54.433866+0000 osd.5 (osd.5) 220 : cluster [DBG] 6.d scrub starts 2024-02-19T21:12:55.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:54.436969+0000 osd.5 (osd.5) 221 : cluster [DBG] 6.d scrub ok 2024-02-19T21:12:55.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:55.247319+0000 osd.3 (osd.3) 231 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:12:55.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:55.248254+0000 osd.3 (osd.3) 232 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:12:55.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:55.325089+0000 mon.a (mon.0) 2889 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88"}]': finished 2024-02-19T21:12:55.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:55.325174+0000 mon.a (mon.0) 2890 : audit [INF] from='client.50120 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-19T21:12:55.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:55.325225+0000 mon.a (mon.0) 2891 : cluster [DBG] osdmap e539: 8 total, 8 up, 6 in 2024-02-19T21:12:55.716 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:54.400526+0000 mon.c (mon.2) 407 : audit [INF] from='client.? v1:172.21.15.163:0/3028537366' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:12:55.716 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:54.404701+0000 mon.a (mon.0) 2888 : audit [INF] from='client.50120 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:12:55.716 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:54.433866+0000 osd.5 (osd.5) 220 : cluster [DBG] 6.d scrub starts 2024-02-19T21:12:55.716 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:54.436969+0000 osd.5 (osd.5) 221 : cluster [DBG] 6.d scrub ok 2024-02-19T21:12:55.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:55.247319+0000 osd.3 (osd.3) 231 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:12:55.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:55.248254+0000 osd.3 (osd.3) 232 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:12:55.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:55.325089+0000 mon.a (mon.0) 2889 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88"}]': finished 2024-02-19T21:12:55.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:55.325174+0000 mon.a (mon.0) 2890 : audit [INF] from='client.50120 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-19T21:12:55.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:55.325225+0000 mon.a (mon.0) 2891 : cluster [DBG] osdmap e539: 8 total, 8 up, 6 in 2024-02-19T21:12:55.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:54.400526+0000 mon.c (mon.2) 407 : audit [INF] from='client.? v1:172.21.15.163:0/3028537366' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:12:55.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:54.404701+0000 mon.a (mon.0) 2888 : audit [INF] from='client.50120 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:12:55.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:54.433866+0000 osd.5 (osd.5) 220 : cluster [DBG] 6.d scrub starts 2024-02-19T21:12:55.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:54.436969+0000 osd.5 (osd.5) 221 : cluster [DBG] 6.d scrub ok 2024-02-19T21:12:55.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:55.247319+0000 osd.3 (osd.3) 231 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:12:55.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:55.248254+0000 osd.3 (osd.3) 232 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:12:55.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:55.325089+0000 mon.a (mon.0) 2889 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-88"}]': finished 2024-02-19T21:12:55.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:55.325174+0000 mon.a (mon.0) 2890 : audit [INF] from='client.50120 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-19T21:12:55.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:55.325225+0000 mon.a (mon.0) 2891 : cluster [DBG] osdmap e539: 8 total, 8 up, 6 in 2024-02-19T21:12:56.439 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 4 1 2024-02-19T21:12:56.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.652+0000 7f5c5a013700 1 -- 172.21.15.163:0/364297602 >> v1:172.21.15.163:6789/0 conn(0x7f5c5406bc60 legacy=0x7f5c54104ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.653+0000 7f5c5a013700 1 -- 172.21.15.163:0/364297602 shutdown_connections 2024-02-19T21:12:56.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.653+0000 7f5c5a013700 1 -- 172.21.15.163:0/364297602 >> 172.21.15.163:0/364297602 conn(0x7f5c540f35a0 msgr2=0x7f5c540f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:56.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.653+0000 7f5c5a013700 1 -- 172.21.15.163:0/364297602 shutdown_connections 2024-02-19T21:12:56.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.653+0000 7f5c5a013700 1 -- 172.21.15.163:0/364297602 wait complete. 2024-02-19T21:12:56.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.653+0000 7f5c5a013700 1 librados: starting msgr at 2024-02-19T21:12:56.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.653+0000 7f5c5a013700 1 librados: starting objecter 2024-02-19T21:12:56.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.653+0000 7f5c5a013700 1 Processor -- start 2024-02-19T21:12:56.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c5a013700 1 -- start start 2024-02-19T21:12:56.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c5a013700 1 librados: setting wanted keys 2024-02-19T21:12:56.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c5a013700 1 librados: calling monclient init 2024-02-19T21:12:56.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c5a013700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c541194d0 con 0x7f5c54106530 2024-02-19T21:12:56.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c5a013700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c5411a6b0 con 0x7f5c5406bc60 2024-02-19T21:12:56.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c5a013700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c5411b8b0 con 0x7f5c5406b170 2024-02-19T21:12:56.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c537fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f5c5406b170 0x7f5c54101770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:60510/0 (socket says 172.21.15.163:60510) 2024-02-19T21:12:56.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c537fe700 1 -- 172.21.15.163:0/383288601 learned_addr learned my addr 172.21.15.163:0/383288601 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:56.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c50ff9700 10 client.?.objecter ms_handle_connect 0x7f5c54106530 2024-02-19T21:12:56.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c50ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:56.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c50ff9700 10 client.?.objecter ms_handle_connect 0x7f5c5406b170 2024-02-19T21:12:56.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c50ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:56.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c50ff9700 10 client.?.objecter ms_handle_connect 0x7f5c5406bc60 2024-02-19T21:12:56.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c50ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:56.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012662808 0 0) 0x7f5c541194d0 con 0x7f5c54106530 2024-02-19T21:12:56.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c3c0038e0 con 0x7f5c54106530 2024-02-19T21:12:56.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309346085 0 0) 0x7f5c5411b8b0 con 0x7f5c5406b170 2024-02-19T21:12:56.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c541194d0 con 0x7f5c5406b170 2024-02-19T21:12:56.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533235872 0 0) 0x7f5c5411a6b0 con 0x7f5c5406bc60 2024-02-19T21:12:56.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.654+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c5411b8b0 con 0x7f5c5406bc60 2024-02-19T21:12:56.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 110998762 0 0) 0x7f5c3c0038e0 con 0x7f5c54106530 2024-02-19T21:12:56.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c5411a6b0 con 0x7f5c54106530 2024-02-19T21:12:56.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5c4c001b80 con 0x7f5c54106530 2024-02-19T21:12:56.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2407532714 0 0) 0x7f5c5411b8b0 con 0x7f5c5406bc60 2024-02-19T21:12:56.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c3c0038e0 con 0x7f5c5406bc60 2024-02-19T21:12:56.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5c48002070 con 0x7f5c5406bc60 2024-02-19T21:12:56.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 884173086 0 0) 0x7f5c541194d0 con 0x7f5c5406b170 2024-02-19T21:12:56.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c5411b8b0 con 0x7f5c5406b170 2024-02-19T21:12:56.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5c440019f0 con 0x7f5c5406b170 2024-02-19T21:12:56.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2800656020 0 0) 0x7f5c5411a6b0 con 0x7f5c54106530 2024-02-19T21:12:56.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 >> v1:172.21.15.163:6790/0 conn(0x7f5c5406b170 legacy=0x7f5c54101770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 >> v1:172.21.15.193:6789/0 conn(0x7f5c5406bc60 legacy=0x7f5c54101e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5411cab0 con 0x7f5c54106530 2024-02-19T21:12:56.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5411baa0 con 0x7f5c54106530 2024-02-19T21:12:56.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c5a013700 10 client.23397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:56.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5411bf10 con 0x7f5c54106530 2024-02-19T21:12:56.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c5a013700 1 librados: init done 2024-02-19T21:12:56.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f5c4c002290 con 0x7f5c54106530 2024-02-19T21:12:56.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.655+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5c4c003690 con 0x7f5c54106530 2024-02-19T21:12:56.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.656+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f5c40004f00 con 0x7f5c54106530 2024-02-19T21:12:56.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f5c4c001e40 con 0x7f5c54106530 2024-02-19T21:12:56.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(540..540 src has 43..540) v4 ==== 7707+0+0 (unknown 2830566104 0 0) 0x7f5c4c076650 con 0x7f5c54106530 2024-02-19T21:12:56.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 osd_map(540..540 src has 43..540) v4 2024-02-19T21:12:56.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c50ff9700 3 client.23397.objecter handle_osd_map got epochs [540,540] > 0 2024-02-19T21:12:56.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c50ff9700 3 client.23397.objecter handle_osd_map decoding full epoch 540 2024-02-19T21:12:56.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c50ff9700 20 client.23397.objecter dump_active .. 0 homeless 2024-02-19T21:12:56.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v540) v1 ==== 54+0+16 (unknown 1007340935 0 2420000192) 0x7f5c4c076a50 con 0x7f5c54106530 2024-02-19T21:12:56.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c50ff9700 10 client.23397.objecter ms_handle_connect 0x7f5c3c063c70 2024-02-19T21:12:56.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c5a013700 10 client.23397.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:56.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 1 2024-02-19T21:12:56.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c400051a0 osd=0 3 2024-02-19T21:12:56.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c400051a0 osd=0 3 2024-02-19T21:12:56.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c400051a0 osd=0 3 2024-02-19T21:12:56.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 1 2024-02-19T21:12:56.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c400051a0 osd=0 3 2024-02-19T21:12:56.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 0 1 2024-02-19T21:12:56.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c400051a0 osd=0 4 2024-02-19T21:12:56.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c5a013700 10 client.23397.objecter _send_command 1 2024-02-19T21:12:56.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.657+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c40004950 con 0x7f5c40004530 2024-02-19T21:12:56.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.658+0000 7f5c50ff9700 10 client.23397.objecter ms_handle_connect 0x7f5c40004530 2024-02-19T21:12:56.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.658+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5c40004950 con 0x7f5c40004530 2024-02-19T21:12:56.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.658+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:12:56.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.658+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 1 = system:0 2024-02-19T21:12:56.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.658+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c400051a0 osd=0 4 2024-02-19T21:12:56.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.658+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 0 1 2024-02-19T21:12:56.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.673+0000 7f5c5a013700 10 client.23397.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:56.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.673+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 2 2024-02-19T21:12:56.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.673+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c400051a0 osd=0 3 2024-02-19T21:12:56.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.673+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c400051a0 osd=0 3 2024-02-19T21:12:56.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.673+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c400051a0 osd=0 3 2024-02-19T21:12:56.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.673+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 2 2024-02-19T21:12:56.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.673+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c400051a0 osd=0 3 2024-02-19T21:12:56.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.673+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 0 2 2024-02-19T21:12:56.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.673+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c400051a0 osd=0 4 2024-02-19T21:12:56.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.673+0000 7f5c5a013700 10 client.23397.objecter _send_command 2 2024-02-19T21:12:56.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.673+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c40004d30 con 0x7f5c40004530 2024-02-19T21:12:56.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5c40004d30 con 0x7f5c40004530 2024-02-19T21:12:56.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:56.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c400051a0 osd=0 4 2024-02-19T21:12:56.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 0 2 2024-02-19T21:12:56.674 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:12:56.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c5a013700 10 client.23397.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:56.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 3 2024-02-19T21:12:56.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c40004270 osd=1 3 2024-02-19T21:12:56.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c40004270 osd=1 3 2024-02-19T21:12:56.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c40004270 osd=1 3 2024-02-19T21:12:56.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 3 2024-02-19T21:12:56.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c40004270 osd=1 3 2024-02-19T21:12:56.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 1 3 2024-02-19T21:12:56.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c40004270 osd=1 4 2024-02-19T21:12:56.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c5a013700 10 client.23397.objecter _send_command 3 2024-02-19T21:12:56.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c4000bf30 con 0x7f5c40003b40 2024-02-19T21:12:56.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.675+0000 7f5c50ff9700 10 client.23397.objecter ms_handle_connect 0x7f5c40003b40 2024-02-19T21:12:56.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.676+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5c4000bf30 con 0x7f5c40003b40 2024-02-19T21:12:56.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.676+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:12:56.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.676+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 3 = system:0 2024-02-19T21:12:56.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.676+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c40004270 osd=1 4 2024-02-19T21:12:56.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.676+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 1 3 2024-02-19T21:12:56.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.701+0000 7f5c5a013700 10 client.23397.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:56.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.701+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 4 2024-02-19T21:12:56.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.701+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c40004270 osd=1 3 2024-02-19T21:12:56.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.701+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c40004270 osd=1 3 2024-02-19T21:12:56.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.701+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c40004270 osd=1 3 2024-02-19T21:12:56.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.701+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 4 2024-02-19T21:12:56.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.701+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c40004270 osd=1 3 2024-02-19T21:12:56.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.701+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 1 4 2024-02-19T21:12:56.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.701+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c40004270 osd=1 4 2024-02-19T21:12:56.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.701+0000 7f5c5a013700 10 client.23397.objecter _send_command 4 2024-02-19T21:12:56.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.701+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c4000d250 con 0x7f5c40003b40 2024-02-19T21:12:56.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.702+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5c4000d250 con 0x7f5c40003b40 2024-02-19T21:12:56.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.702+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:56.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.702+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.703+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c40004270 osd=1 4 2024-02-19T21:12:56.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.703+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 1 4 2024-02-19T21:12:56.701 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:12:56.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.703+0000 7f5c5a013700 10 client.23397.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:56.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.703+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 5 2024-02-19T21:12:56.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.703+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c400033a0 osd=2 3 2024-02-19T21:12:56.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.703+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c400033a0 osd=2 3 2024-02-19T21:12:56.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.703+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c400033a0 osd=2 3 2024-02-19T21:12:56.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.703+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 5 2024-02-19T21:12:56.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.703+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c400033a0 osd=2 3 2024-02-19T21:12:56.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.703+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 2 5 2024-02-19T21:12:56.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.703+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c400033a0 osd=2 4 2024-02-19T21:12:56.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.703+0000 7f5c5a013700 10 client.23397.objecter _send_command 5 2024-02-19T21:12:56.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.703+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c40010d00 con 0x7f5c40002070 2024-02-19T21:12:56.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.703+0000 7f5c50ff9700 10 client.23397.objecter ms_handle_connect 0x7f5c40002070 2024-02-19T21:12:56.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.704+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5c40010d00 con 0x7f5c40002070 2024-02-19T21:12:56.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.704+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:12:56.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.704+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 5 = system:0 2024-02-19T21:12:56.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.704+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c400033a0 osd=2 4 2024-02-19T21:12:56.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.704+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 2 5 2024-02-19T21:12:56.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.718+0000 7f5c5a013700 10 client.23397.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:56.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.718+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 6 2024-02-19T21:12:56.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.718+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c400033a0 osd=2 3 2024-02-19T21:12:56.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.718+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c400033a0 osd=2 3 2024-02-19T21:12:56.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.718+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c400033a0 osd=2 3 2024-02-19T21:12:56.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.718+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 6 2024-02-19T21:12:56.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.718+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c400033a0 osd=2 3 2024-02-19T21:12:56.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.718+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 2 6 2024-02-19T21:12:56.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.718+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c400033a0 osd=2 4 2024-02-19T21:12:56.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.718+0000 7f5c5a013700 10 client.23397.objecter _send_command 6 2024-02-19T21:12:56.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.718+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c40001690 con 0x7f5c40002070 2024-02-19T21:12:56.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5c40001690 con 0x7f5c40002070 2024-02-19T21:12:56.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:56.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c400033a0 osd=2 4 2024-02-19T21:12:56.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 2 6 2024-02-19T21:12:56.718 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:12:56.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c5a013700 10 client.23397.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:56.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 7 2024-02-19T21:12:56.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4000c8a0 osd=3 3 2024-02-19T21:12:56.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4000c8a0 osd=3 3 2024-02-19T21:12:56.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4000c8a0 osd=3 3 2024-02-19T21:12:56.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 7 2024-02-19T21:12:56.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c4000c8a0 osd=3 3 2024-02-19T21:12:56.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 3 7 2024-02-19T21:12:56.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4000c8a0 osd=3 4 2024-02-19T21:12:56.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c5a013700 10 client.23397.objecter _send_command 7 2024-02-19T21:12:56.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.719+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c4000cfe0 con 0x7f5c40010ea0 2024-02-19T21:12:56.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.720+0000 7f5c50ff9700 10 client.23397.objecter ms_handle_connect 0x7f5c40010ea0 2024-02-19T21:12:56.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.720+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5c4000cfe0 con 0x7f5c40010ea0 2024-02-19T21:12:56.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.720+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:12:56.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.720+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 7 = system:0 2024-02-19T21:12:56.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.720+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c4000c8a0 osd=3 4 2024-02-19T21:12:56.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.720+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 3 7 2024-02-19T21:12:56.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.735+0000 7f5c5a013700 10 client.23397.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:56.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.735+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 8 2024-02-19T21:12:56.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.735+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4000c8a0 osd=3 3 2024-02-19T21:12:56.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.735+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4000c8a0 osd=3 3 2024-02-19T21:12:56.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.735+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4000c8a0 osd=3 3 2024-02-19T21:12:56.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.735+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 8 2024-02-19T21:12:56.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.735+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c4000c8a0 osd=3 3 2024-02-19T21:12:56.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.735+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 3 8 2024-02-19T21:12:56.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.735+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4000c8a0 osd=3 4 2024-02-19T21:12:56.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.735+0000 7f5c5a013700 10 client.23397.objecter _send_command 8 2024-02-19T21:12:56.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.735+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c4000cfe0 con 0x7f5c40010ea0 2024-02-19T21:12:56.735 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5c4000cfe0 con 0x7f5c40010ea0 2024-02-19T21:12:56.735 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:56.735 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.735 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c4000c8a0 osd=3 4 2024-02-19T21:12:56.735 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 3 8 2024-02-19T21:12:56.735 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:12:56.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c5a013700 10 client.23397.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:56.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 9 2024-02-19T21:12:56.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4000cfe0 osd=4 3 2024-02-19T21:12:56.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4000cfe0 osd=4 3 2024-02-19T21:12:56.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4000cfe0 osd=4 3 2024-02-19T21:12:56.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 9 2024-02-19T21:12:56.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c4000cfe0 osd=4 3 2024-02-19T21:12:56.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 4 9 2024-02-19T21:12:56.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4000cfe0 osd=4 4 2024-02-19T21:12:56.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c5a013700 10 client.23397.objecter _send_command 9 2024-02-19T21:12:56.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.736+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c4001a1d0 con 0x7f5c400166c0 2024-02-19T21:12:56.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.737+0000 7f5c50ff9700 10 client.23397.objecter ms_handle_connect 0x7f5c400166c0 2024-02-19T21:12:56.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.738+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5c4001a1d0 con 0x7f5c400166c0 2024-02-19T21:12:56.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.738+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:12:56.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.738+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 9 = system:0 2024-02-19T21:12:56.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.738+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c4000cfe0 osd=4 4 2024-02-19T21:12:56.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.738+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 4 9 2024-02-19T21:12:56.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:54.954607+0000 osd.1 (osd.1) 249 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:12:56.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:54.957580+0000 osd.1 (osd.1) 250 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:12:56.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:55.260836+0000 mgr.y (mgr.14550) 481 : cluster [DBG] pgmap v807: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:12:56.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:55.428347+0000 osd.4 (osd.4) 195 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:12:56.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:55.429796+0000 osd.4 (osd.4) 196 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:12:56.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:56.324173+0000 mon.a (mon.0) 2892 : cluster [DBG] osdmap e540: 8 total, 8 up, 6 in 2024-02-19T21:12:56.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:54.954607+0000 osd.1 (osd.1) 249 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:12:56.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:54.957580+0000 osd.1 (osd.1) 250 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:12:56.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:55.260836+0000 mgr.y (mgr.14550) 481 : cluster [DBG] pgmap v807: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:12:56.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:55.428347+0000 osd.4 (osd.4) 195 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:12:56.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:55.429796+0000 osd.4 (osd.4) 196 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:12:56.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:56.324173+0000 mon.a (mon.0) 2892 : cluster [DBG] osdmap e540: 8 total, 8 up, 6 in 2024-02-19T21:12:56.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.752+0000 7f5c5a013700 10 client.23397.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:56.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.752+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 10 2024-02-19T21:12:56.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.752+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4000cfe0 osd=4 3 2024-02-19T21:12:56.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.752+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4000cfe0 osd=4 3 2024-02-19T21:12:56.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.752+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4000cfe0 osd=4 3 2024-02-19T21:12:56.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.752+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 10 2024-02-19T21:12:56.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.752+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c4000cfe0 osd=4 3 2024-02-19T21:12:56.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.752+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 4 10 2024-02-19T21:12:56.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.752+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4000cfe0 osd=4 4 2024-02-19T21:12:56.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.752+0000 7f5c5a013700 10 client.23397.objecter _send_command 10 2024-02-19T21:12:56.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.752+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c4001a1d0 con 0x7f5c400166c0 2024-02-19T21:12:56.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5c4001a1d0 con 0x7f5c400166c0 2024-02-19T21:12:56.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:56.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c4000cfe0 osd=4 4 2024-02-19T21:12:56.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 4 10 2024-02-19T21:12:56.753 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:12:56.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c5a013700 10 client.23397.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:56.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 11 2024-02-19T21:12:56.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4001a1d0 osd=5 3 2024-02-19T21:12:56.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4001a1d0 osd=5 3 2024-02-19T21:12:56.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4001a1d0 osd=5 3 2024-02-19T21:12:56.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 11 2024-02-19T21:12:56.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c4001a1d0 osd=5 3 2024-02-19T21:12:56.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 5 11 2024-02-19T21:12:56.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4001a1d0 osd=5 4 2024-02-19T21:12:56.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c5a013700 10 client.23397.objecter _send_command 11 2024-02-19T21:12:56.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.754+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c4001c580 con 0x7f5c4001a890 2024-02-19T21:12:56.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.755+0000 7f5c50ff9700 10 client.23397.objecter ms_handle_connect 0x7f5c4001a890 2024-02-19T21:12:56.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.756+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5c4001c580 con 0x7f5c4001a890 2024-02-19T21:12:56.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.756+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:12:56.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.756+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 11 = system:0 2024-02-19T21:12:56.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.756+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c4001a1d0 osd=5 4 2024-02-19T21:12:56.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.756+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 5 11 2024-02-19T21:12:56.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.770+0000 7f5c5a013700 10 client.23397.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:56.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.770+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 12 2024-02-19T21:12:56.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.770+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4001a1d0 osd=5 3 2024-02-19T21:12:56.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.770+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4001a1d0 osd=5 3 2024-02-19T21:12:56.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.770+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4001a1d0 osd=5 3 2024-02-19T21:12:56.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.770+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 12 2024-02-19T21:12:56.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.770+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c4001a1d0 osd=5 3 2024-02-19T21:12:56.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.770+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 5 12 2024-02-19T21:12:56.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.770+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4001a1d0 osd=5 4 2024-02-19T21:12:56.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.770+0000 7f5c5a013700 10 client.23397.objecter _send_command 12 2024-02-19T21:12:56.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.770+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c4001c580 con 0x7f5c4001a890 2024-02-19T21:12:56.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5c4001c580 con 0x7f5c4001a890 2024-02-19T21:12:56.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:56.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c4001a1d0 osd=5 4 2024-02-19T21:12:56.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 5 12 2024-02-19T21:12:56.771 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:12:56.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c5a013700 10 client.23397.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:56.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 13 2024-02-19T21:12:56.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4001c580 osd=6 3 2024-02-19T21:12:56.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4001c580 osd=6 3 2024-02-19T21:12:56.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4001c580 osd=6 3 2024-02-19T21:12:56.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 13 2024-02-19T21:12:56.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c4001c580 osd=6 3 2024-02-19T21:12:56.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 6 13 2024-02-19T21:12:56.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4001c580 osd=6 4 2024-02-19T21:12:56.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c5a013700 10 client.23397.objecter _send_command 13 2024-02-19T21:12:56.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.772+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c40022b20 con 0x7f5c4001f010 2024-02-19T21:12:56.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.773+0000 7f5c50ff9700 10 client.23397.objecter ms_handle_connect 0x7f5c4001f010 2024-02-19T21:12:56.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.773+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5c40022b20 con 0x7f5c4001f010 2024-02-19T21:12:56.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.773+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:12:56.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.773+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 13 = system:0 2024-02-19T21:12:56.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.773+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c4001c580 osd=6 4 2024-02-19T21:12:56.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.773+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 6 13 2024-02-19T21:12:56.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.788+0000 7f5c5a013700 10 client.23397.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:56.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.788+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 14 2024-02-19T21:12:56.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.788+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4001c580 osd=6 3 2024-02-19T21:12:56.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.788+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4001c580 osd=6 3 2024-02-19T21:12:56.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.788+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c4001c580 osd=6 3 2024-02-19T21:12:56.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.788+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 14 2024-02-19T21:12:56.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.788+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c4001c580 osd=6 3 2024-02-19T21:12:56.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.788+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 6 14 2024-02-19T21:12:56.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.788+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4001c580 osd=6 4 2024-02-19T21:12:56.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.788+0000 7f5c5a013700 10 client.23397.objecter _send_command 14 2024-02-19T21:12:56.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.788+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c40022b20 con 0x7f5c4001f010 2024-02-19T21:12:56.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5c40022b20 con 0x7f5c4001f010 2024-02-19T21:12:56.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:56.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c4001c580 osd=6 4 2024-02-19T21:12:56.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 6 14 2024-02-19T21:12:56.790 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:12:56.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c5a013700 10 client.23397.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:12:56.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 15 2024-02-19T21:12:56.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c40022b20 osd=7 3 2024-02-19T21:12:56.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c40022b20 osd=7 3 2024-02-19T21:12:56.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c40022b20 osd=7 3 2024-02-19T21:12:56.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 15 2024-02-19T21:12:56.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c40022b20 osd=7 3 2024-02-19T21:12:56.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 7 15 2024-02-19T21:12:56.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c40022b20 osd=7 4 2024-02-19T21:12:56.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c5a013700 10 client.23397.objecter _send_command 15 2024-02-19T21:12:56.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.790+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c40024ea0 con 0x7f5c400231a0 2024-02-19T21:12:56.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.791+0000 7f5c50ff9700 10 client.23397.objecter ms_handle_connect 0x7f5c400231a0 2024-02-19T21:12:56.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.792+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5c40024ea0 con 0x7f5c400231a0 2024-02-19T21:12:56.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.792+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:12:56.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.792+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 15 = system:0 2024-02-19T21:12:56.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.792+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c40022b20 osd=7 4 2024-02-19T21:12:56.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.792+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 7 15 2024-02-19T21:12:56.794 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:12:56.796 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:56.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:12:56.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.809+0000 7f5c5a013700 10 client.23397.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:12:56.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.809+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign -1 16 2024-02-19T21:12:56.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.809+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c40022b20 osd=7 3 2024-02-19T21:12:56.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.809+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c40022b20 osd=7 3 2024-02-19T21:12:56.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.809+0000 7f5c5a013700 20 client.23397.objecter _get_session s=0x7f5c40022b20 osd=7 3 2024-02-19T21:12:56.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.809+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_remove -1 16 2024-02-19T21:12:56.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.809+0000 7f5c5a013700 20 client.23397.objecter get_session s=0x7f5c40022b20 osd=7 3 2024-02-19T21:12:56.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.809+0000 7f5c5a013700 15 client.23397.objecter _session_command_op_assign 7 16 2024-02-19T21:12:56.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.809+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c40022b20 osd=7 4 2024-02-19T21:12:56.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.809+0000 7f5c5a013700 10 client.23397.objecter _send_command 16 2024-02-19T21:12:56.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.809+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f5c40024ea0 con 0x7f5c400231a0 2024-02-19T21:12:56.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.811+0000 7f5c50ff9700 1 -- 172.21.15.163:0/383288601 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f5c40024ea0 con 0x7f5c400231a0 2024-02-19T21:12:56.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.811+0000 7f5c50ff9700 10 client.23397.objecter ms_dispatch 0x7f5c54001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:12:56.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.811+0000 7f5c50ff9700 10 client.23397.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.811+0000 7f5c50ff9700 20 client.23397.objecter put_session s=0x7f5c40022b20 osd=7 4 2024-02-19T21:12:56.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.811+0000 7f5c50ff9700 15 client.23397.objecter _session_command_op_remove 7 16 2024-02-19T21:12:56.810 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:12:56.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 10 librados: watch_flush enter 2024-02-19T21:12:56.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 10 librados: watch_flush exit 2024-02-19T21:12:56.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 10 client.23397.objecter close_session for osd.0 2024-02-19T21:12:56.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 >> v1:172.21.15.163:6801/1559418115 conn(0x7f5c40004530 legacy=0x7f5c40002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c400051a0 osd=0 1 2024-02-19T21:12:56.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 10 client.23397.objecter close_session for osd.1 2024-02-19T21:12:56.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 >> v1:172.21.15.163:6805/3125047432 conn(0x7f5c40003b40 legacy=0x7f5c4000a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c40004270 osd=1 1 2024-02-19T21:12:56.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 10 client.23397.objecter close_session for osd.2 2024-02-19T21:12:56.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 >> v1:172.21.15.163:6809/1563158487 conn(0x7f5c40002070 legacy=0x7f5c4000f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c400033a0 osd=2 1 2024-02-19T21:12:56.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 10 client.23397.objecter close_session for osd.3 2024-02-19T21:12:56.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 >> v1:172.21.15.163:6813/2112279769 conn(0x7f5c40010ea0 legacy=0x7f5c400112e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4000c8a0 osd=3 1 2024-02-19T21:12:56.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 10 client.23397.objecter close_session for osd.4 2024-02-19T21:12:56.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 >> v1:172.21.15.193:6800/3129716965 conn(0x7f5c400166c0 legacy=0x7f5c40018ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4000cfe0 osd=4 1 2024-02-19T21:12:56.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 10 client.23397.objecter close_session for osd.5 2024-02-19T21:12:56.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 >> v1:172.21.15.193:6804/805161991 conn(0x7f5c4001a890 legacy=0x7f5c4001acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4001a1d0 osd=5 1 2024-02-19T21:12:56.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 10 client.23397.objecter close_session for osd.6 2024-02-19T21:12:56.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 >> v1:172.21.15.193:6808/787604615 conn(0x7f5c4001f010 legacy=0x7f5c40021400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c4001c580 osd=6 1 2024-02-19T21:12:56.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 10 client.23397.objecter close_session for osd.7 2024-02-19T21:12:56.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 >> v1:172.21.15.193:6812/157927432 conn(0x7f5c400231a0 legacy=0x7f5c40023620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 20 client.23397.objecter put_session s=0x7f5c40022b20 osd=7 1 2024-02-19T21:12:56.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 20 client.23397.objecter shutdown clearing up homeless session... 2024-02-19T21:12:56.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 10 client.23397.objecter successfully canceled tick 2024-02-19T21:12:56.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c53fff700 1 -- 172.21.15.163:0/383288601 reap_dead start 2024-02-19T21:12:56.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c53fff700 1 -- 172.21.15.163:0/383288601 reap_dead start 2024-02-19T21:12:56.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c53fff700 1 -- 172.21.15.163:0/383288601 reap_dead start 2024-02-19T21:12:56.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c53fff700 1 -- 172.21.15.163:0/383288601 reap_dead start 2024-02-19T21:12:56.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c53fff700 1 -- 172.21.15.163:0/383288601 reap_dead start 2024-02-19T21:12:56.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5c3c063c70 legacy=0x7f5c3c066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 >> v1:172.21.15.163:6789/0 conn(0x7f5c54106530 legacy=0x7f5c54117b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 shutdown_connections 2024-02-19T21:12:56.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 >> 172.21.15.163:0/383288601 conn(0x7f5c540f35a0 msgr2=0x7f5c54109930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:56.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 shutdown_connections 2024-02-19T21:12:56.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 1 -- 172.21.15.163:0/383288601 wait complete. 2024-02-19T21:12:56.815 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.812+0000 7f5c5a013700 1 librados: shutdown 2024-02-19T21:12:56.815 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.815 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.815 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.815 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.816 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.816 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.816 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.816 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:12:56.898 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:56.928 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:56.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.906+0000 7f64c21de700 1 -- 172.21.15.163:0/2200922331 >> v1:172.21.15.163:6789/0 conn(0x7f64bc0fc870 legacy=0x7f64bc0fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.906+0000 7f64c21de700 1 -- 172.21.15.163:0/2200922331 shutdown_connections 2024-02-19T21:12:56.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.906+0000 7f64c21de700 1 -- 172.21.15.163:0/2200922331 >> 172.21.15.163:0/2200922331 conn(0x7f64bc0f2470 msgr2=0x7f64bc0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:56.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.906+0000 7f64c21de700 1 -- 172.21.15.163:0/2200922331 shutdown_connections 2024-02-19T21:12:56.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.906+0000 7f64c21de700 1 -- 172.21.15.163:0/2200922331 wait complete. 2024-02-19T21:12:56.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.907+0000 7f64c21de700 1 librados: starting msgr at 2024-02-19T21:12:56.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.907+0000 7f64c21de700 1 librados: starting objecter 2024-02-19T21:12:56.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.907+0000 7f64c21de700 1 Processor -- start 2024-02-19T21:12:56.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.907+0000 7f64c21de700 1 -- start start 2024-02-19T21:12:56.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.907+0000 7f64c21de700 1 librados: setting wanted keys 2024-02-19T21:12:56.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.907+0000 7f64c21de700 1 librados: calling monclient init 2024-02-19T21:12:56.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64c21de700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f64bc116120 con 0x7f64bc1031b0 2024-02-19T21:12:56.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64c21de700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f64bc117300 con 0x7f64bc0ff670 2024-02-19T21:12:56.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64c21de700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f64bc118500 con 0x7f64bc0fc870 2024-02-19T21:12:56.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64bbfff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f64bc1031b0 0x7f64bc114830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:53044/0 (socket says 172.21.15.163:53044) 2024-02-19T21:12:56.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64bbfff700 1 -- 172.21.15.163:0/1567319517 learned_addr learned my addr 172.21.15.163:0/1567319517 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:12:56.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 10 client.?.objecter ms_handle_connect 0x7f64bc1031b0 2024-02-19T21:12:56.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:56.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 10 client.?.objecter ms_handle_connect 0x7f64bc0ff670 2024-02-19T21:12:56.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:56.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 10 client.?.objecter ms_handle_connect 0x7f64bc0fc870 2024-02-19T21:12:56.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:12:56.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691880993 0 0) 0x7f64bc116120 con 0x7f64bc1031b0 2024-02-19T21:12:56.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a4003030 con 0x7f64bc1031b0 2024-02-19T21:12:56.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010411794 0 0) 0x7f64bc117300 con 0x7f64bc0ff670 2024-02-19T21:12:56.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64bc116120 con 0x7f64bc0ff670 2024-02-19T21:12:56.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703594369 0 0) 0x7f64bc118500 con 0x7f64bc0fc870 2024-02-19T21:12:56.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64bc117300 con 0x7f64bc0fc870 2024-02-19T21:12:56.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1155822005 0 0) 0x7f64a4003030 con 0x7f64bc1031b0 2024-02-19T21:12:56.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64bc118500 con 0x7f64bc1031b0 2024-02-19T21:12:56.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f64b4001c90 con 0x7f64bc1031b0 2024-02-19T21:12:56.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3024131033 0 0) 0x7f64bc117300 con 0x7f64bc0fc870 2024-02-19T21:12:56.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64a4003030 con 0x7f64bc0fc870 2024-02-19T21:12:56.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3426328856 0 0) 0x7f64bc116120 con 0x7f64bc0ff670 2024-02-19T21:12:56.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64bc117300 con 0x7f64bc0ff670 2024-02-19T21:12:56.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f64ac000f10 con 0x7f64bc0fc870 2024-02-19T21:12:56.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.908+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f64b00019f0 con 0x7f64bc0ff670 2024-02-19T21:12:56.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.909+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2124247650 0 0) 0x7f64bc118500 con 0x7f64bc1031b0 2024-02-19T21:12:56.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.909+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 >> v1:172.21.15.163:6790/0 conn(0x7f64bc0fc870 legacy=0x7f64bc0f8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.909+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 >> v1:172.21.15.193:6789/0 conn(0x7f64bc0ff670 legacy=0x7f64bc0f8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:56.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.909+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64bc119700 con 0x7f64bc1031b0 2024-02-19T21:12:56.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.909+0000 7f64c21de700 1 -- 172.21.15.163:0/1567319517 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64bc1174f0 con 0x7f64bc1031b0 2024-02-19T21:12:56.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.909+0000 7f64c21de700 10 client.23403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:12:56.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.909+0000 7f64c21de700 1 -- 172.21.15.163:0/1567319517 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64bc117960 con 0x7f64bc1031b0 2024-02-19T21:12:56.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.909+0000 7f64c21de700 1 librados: init done 2024-02-19T21:12:56.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.909+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f64b4002020 con 0x7f64bc1031b0 2024-02-19T21:12:56.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.909+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f64b4005f10 con 0x7f64bc1031b0 2024-02-19T21:12:56.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.911+0000 7f64c21de700 1 -- 172.21.15.163:0/1567319517 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64bc05f700 con 0x7f64bc1031b0 2024-02-19T21:12:56.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.911+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f64b4018da0 con 0x7f64bc1031b0 2024-02-19T21:12:56.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.911+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(540..540 src has 43..540) v4 ==== 7707+0+0 (unknown 2830566104 0 0) 0x7f64b4079070 con 0x7f64bc1031b0 2024-02-19T21:12:56.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.911+0000 7f64b8ff9700 10 client.23403.objecter ms_dispatch 0x7f64bc001550 osd_map(540..540 src has 43..540) v4 2024-02-19T21:12:56.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.911+0000 7f64b8ff9700 3 client.23403.objecter handle_osd_map got epochs [540,540] > 0 2024-02-19T21:12:56.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.911+0000 7f64b8ff9700 3 client.23403.objecter handle_osd_map decoding full epoch 540 2024-02-19T21:12:56.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.911+0000 7f64b8ff9700 20 client.23403.objecter dump_active .. 0 homeless 2024-02-19T21:12:56.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.912+0000 7f64b8ff9700 10 client.23403.objecter ms_handle_connect 0x7f64a4063d80 2024-02-19T21:12:56.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:56.916+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f64b404d3f0 con 0x7f64bc1031b0 2024-02-19T21:12:56.956 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:56.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:57.060 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:57.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:54.954607+0000 osd.1 (osd.1) 249 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:12:57.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:54.957580+0000 osd.1 (osd.1) 250 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:12:57.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:55.260836+0000 mgr.y (mgr.14550) 481 : cluster [DBG] pgmap v807: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:12:57.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:55.428347+0000 osd.4 (osd.4) 195 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:12:57.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:55.429796+0000 osd.4 (osd.4) 196 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:12:57.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:56.324173+0000 mon.a (mon.0) 2892 : cluster [DBG] osdmap e540: 8 total, 8 up, 6 in 2024-02-19T21:12:57.092 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:57.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:57.100+0000 7f64c21de700 1 -- 172.21.15.163:0/1567319517 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f64bc108e10 con 0x7f64bc1031b0 2024-02-19T21:12:57.113 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:57.116 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:57.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:57.217 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:57.244 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:57.263 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:57.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:57.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:57.326+0000 7f64b8ff9700 1 -- 172.21.15.163:0/1567319517 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v541) v1 ==== 101+0+0 (unknown 3675614183 0 0) 0x7f64b404b800 con 0x7f64bc1031b0 2024-02-19T21:12:57.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:57.330+0000 7f64c21de700 10 librados: watch_flush enter 2024-02-19T21:12:57.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:57.330+0000 7f64c21de700 10 librados: watch_flush exit 2024-02-19T21:12:57.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:57.330+0000 7f64c21de700 20 client.23403.objecter shutdown clearing up homeless session... 2024-02-19T21:12:57.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:57.330+0000 7f64c21de700 10 client.23403.objecter successfully canceled tick 2024-02-19T21:12:57.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:57.330+0000 7f64c21de700 1 -- 172.21.15.163:0/1567319517 >> v1:172.21.15.163:6800/1552514681 conn(0x7f64a4063d80 legacy=0x7f64a4066240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:57.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:57.330+0000 7f64c21de700 1 -- 172.21.15.163:0/1567319517 >> v1:172.21.15.163:6789/0 conn(0x7f64bc1031b0 legacy=0x7f64bc114830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:12:57.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:57.331+0000 7f64c21de700 1 -- 172.21.15.163:0/1567319517 shutdown_connections 2024-02-19T21:12:57.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:57.331+0000 7f64c21de700 1 -- 172.21.15.163:0/1567319517 >> 172.21.15.163:0/1567319517 conn(0x7f64bc0f2470 msgr2=0x7f64bc106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:12:57.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:57.333+0000 7f64c21de700 1 -- 172.21.15.163:0/1567319517 shutdown_connections 2024-02-19T21:12:57.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:57.333+0000 7f64c21de700 1 -- 172.21.15.163:0/1567319517 wait complete. 2024-02-19T21:12:57.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:12:57.333+0000 7f64c21de700 1 librados: shutdown 2024-02-19T21:12:57.333 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:12:57.376 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:57.405 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:57.425 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:57.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:57.529 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:57.557 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:57.576 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:57.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:57.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:57.681 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:12:57.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:55.488746+0000 osd.7 (osd.7) 243 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:12:57.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:55.491659+0000 osd.7 (osd.7) 244 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:12:57.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:56.448505+0000 osd.4 (osd.4) 197 : cluster [DBG] 5.e scrub starts 2024-02-19T21:12:57.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:56.450039+0000 osd.4 (osd.4) 198 : cluster [DBG] 5.e scrub ok 2024-02-19T21:12:57.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:56.656962+0000 mon.a (mon.0) 2893 : audit [DBG] from='client.? v1:172.21.15.163:0/383288601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:57.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:57.101181+0000 mon.a (mon.0) 2894 : audit [INF] from='client.? v1:172.21.15.163:0/1567319517' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:12:57.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:57.214229+0000 osd.3 (osd.3) 233 : cluster [DBG] 61.2 scrub starts 2024-02-19T21:12:57.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:57.216335+0000 osd.3 (osd.3) 234 : cluster [DBG] 61.2 scrub ok 2024-02-19T21:12:57.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:57.318891+0000 mon.a (mon.0) 2895 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:12:57.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:57.322917+0000 mon.a (mon.0) 2896 : audit [INF] from='client.? v1:172.21.15.163:0/1567319517' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:12:57.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:57.322966+0000 mon.a (mon.0) 2897 : cluster [DBG] osdmap e541: 8 total, 8 up, 6 in 2024-02-19T21:12:57.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:57.329335+0000 mon.b (mon.1) 699 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:57.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:57.330035+0000 mon.a (mon.0) 2898 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:57.708 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:57.727 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:12:57.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:57.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:12:57.831 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:57.858 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:57.876 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:57.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:57.980 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:57.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:55.488746+0000 osd.7 (osd.7) 243 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:12:57.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:55.491659+0000 osd.7 (osd.7) 244 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:12:57.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:56.448505+0000 osd.4 (osd.4) 197 : cluster [DBG] 5.e scrub starts 2024-02-19T21:12:57.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:56.450039+0000 osd.4 (osd.4) 198 : cluster [DBG] 5.e scrub ok 2024-02-19T21:12:57.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:56.656962+0000 mon.a (mon.0) 2893 : audit [DBG] from='client.? v1:172.21.15.163:0/383288601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:57.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:57.101181+0000 mon.a (mon.0) 2894 : audit [INF] from='client.? v1:172.21.15.163:0/1567319517' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:12:57.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:57.214229+0000 osd.3 (osd.3) 233 : cluster [DBG] 61.2 scrub starts 2024-02-19T21:12:57.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:57.216335+0000 osd.3 (osd.3) 234 : cluster [DBG] 61.2 scrub ok 2024-02-19T21:12:57.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:57.318891+0000 mon.a (mon.0) 2895 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:12:57.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:57.322917+0000 mon.a (mon.0) 2896 : audit [INF] from='client.? v1:172.21.15.163:0/1567319517' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:12:57.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:57.322966+0000 mon.a (mon.0) 2897 : cluster [DBG] osdmap e541: 8 total, 8 up, 6 in 2024-02-19T21:12:57.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:57.329335+0000 mon.b (mon.1) 699 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:57.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:57.330035+0000 mon.a (mon.0) 2898 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:57.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:55.488746+0000 osd.7 (osd.7) 243 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:12:57.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:55.491659+0000 osd.7 (osd.7) 244 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:12:57.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:56.448505+0000 osd.4 (osd.4) 197 : cluster [DBG] 5.e scrub starts 2024-02-19T21:12:57.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:56.450039+0000 osd.4 (osd.4) 198 : cluster [DBG] 5.e scrub ok 2024-02-19T21:12:57.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:56.656962+0000 mon.a (mon.0) 2893 : audit [DBG] from='client.? v1:172.21.15.163:0/383288601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:12:57.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:57.101181+0000 mon.a (mon.0) 2894 : audit [INF] from='client.? v1:172.21.15.163:0/1567319517' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:12:57.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:57.214229+0000 osd.3 (osd.3) 233 : cluster [DBG] 61.2 scrub starts 2024-02-19T21:12:57.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:57.216335+0000 osd.3 (osd.3) 234 : cluster [DBG] 61.2 scrub ok 2024-02-19T21:12:57.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:57.318891+0000 mon.a (mon.0) 2895 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:12:57.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:57.322917+0000 mon.a (mon.0) 2896 : audit [INF] from='client.? v1:172.21.15.163:0/1567319517' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:12:57.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:57.322966+0000 mon.a (mon.0) 2897 : cluster [DBG] osdmap e541: 8 total, 8 up, 6 in 2024-02-19T21:12:57.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:57.329335+0000 mon.b (mon.1) 699 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:57.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:57.330035+0000 mon.a (mon.0) 2898 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:12:58.004 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:12:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:12:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:12:58.006 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:58.024 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:58.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:58.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:58.128 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:12:58.155 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:58.228 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:12:58.233 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:58.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:12:58.334 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:12:58.363 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:58.382 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:12:58.384 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:58.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:12:58.486 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:58.512 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:58.530 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:58.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:58.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:58.634 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:12:58.658 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:57.261260+0000 mgr.y (mgr.14550) 482 : cluster [DBG] pgmap v810: 206 pgs: 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:12:58.659 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:58.327730+0000 mon.a (mon.0) 2899 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:58.659 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:58.327852+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e542: 8 total, 8 up, 6 in 2024-02-19T21:12:58.659 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:58.329552+0000 mon.b (mon.1) 700 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:12:58.659 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:58.332317+0000 mon.a (mon.0) 2901 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:12:58.660 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:58.679 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:12:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:58.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:12:58.783 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:12:58.811 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:58.832 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:12:58.834 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:12:58.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:57.261260+0000 mgr.y (mgr.14550) 482 : cluster [DBG] pgmap v810: 206 pgs: 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:12:58.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:58.327730+0000 mon.a (mon.0) 2899 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:58.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:58.327852+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e542: 8 total, 8 up, 6 in 2024-02-19T21:12:58.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:58.329552+0000 mon.b (mon.1) 700 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:12:58.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:58.332317+0000 mon.a (mon.0) 2901 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:12:58.836 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:58.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:12:58.837 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:57.261260+0000 mgr.y (mgr.14550) 482 : cluster [DBG] pgmap v810: 206 pgs: 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:12:58.837 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:58.327730+0000 mon.a (mon.0) 2899 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:12:58.837 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:58.327852+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e542: 8 total, 8 up, 6 in 2024-02-19T21:12:58.837 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:58.329552+0000 mon.b (mon.1) 700 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:12:58.837 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:58.332317+0000 mon.a (mon.0) 2901 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:12:58.938 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:12:58.965 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:58.984 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:12:58.986 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:58.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:12:59.087 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:12:59.114 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:59.133 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:12:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:59.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:12:59.236 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:12:59.263 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:59.281 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:12:59.283 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:12:59.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:12:59.385 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:12:59.413 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:12:59.717 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:12:59.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:58.080241+0000 osd.2 (osd.2) 216 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:12:59.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:58.081099+0000 osd.2 (osd.2) 217 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:12:59.744 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:59.326331+0000 mon.a (mon.0) 2902 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:12:59.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:59.326402+0000 mon.a (mon.0) 2903 : cluster [DBG] osdmap e543: 8 total, 8 up, 6 in 2024-02-19T21:12:59.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:59.329711+0000 mon.b (mon.1) 701 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-90"}]: dispatch 2024-02-19T21:12:59.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:12:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:59.330412+0000 mon.a (mon.0) 2904 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-90"}]: dispatch 2024-02-19T21:12:59.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:58.080241+0000 osd.2 (osd.2) 216 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:12:59.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:58.081099+0000 osd.2 (osd.2) 217 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:12:59.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:59.326331+0000 mon.a (mon.0) 2902 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:12:59.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:59.326402+0000 mon.a (mon.0) 2903 : cluster [DBG] osdmap e543: 8 total, 8 up, 6 in 2024-02-19T21:12:59.745 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:59.329711+0000 mon.b (mon.1) 701 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-90"}]: dispatch 2024-02-19T21:12:59.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:12:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:59.330412+0000 mon.a (mon.0) 2904 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-90"}]: dispatch 2024-02-19T21:13:00.043 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:13:00.043 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:13:00.043 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-02-19T21:13:00.043 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:13:00.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:58.080241+0000 osd.2 (osd.2) 216 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:13:00.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:58.081099+0000 osd.2 (osd.2) 217 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:13:00.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:59.326331+0000 mon.a (mon.0) 2902 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:13:00.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:59.326402+0000 mon.a (mon.0) 2903 : cluster [DBG] osdmap e543: 8 total, 8 up, 6 in 2024-02-19T21:13:00.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:59.329711+0000 mon.b (mon.1) 701 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-90"}]: dispatch 2024-02-19T21:13:00.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:12:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:59.330412+0000 mon.a (mon.0) 2904 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-90"}]: dispatch 2024-02-19T21:13:00.734 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:58.836894+0000 osd.1 (osd.1) 251 : cluster [DBG] 5.c scrub starts 2024-02-19T21:13:00.734 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:58.837965+0000 osd.1 (osd.1) 252 : cluster [DBG] 5.c scrub ok 2024-02-19T21:13:00.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:59.262021+0000 mgr.y (mgr.14550) 483 : cluster [DBG] pgmap v813: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:00.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:59.464983+0000 osd.7 (osd.7) 245 : cluster [DBG] 61.f deep-scrub starts 2024-02-19T21:13:00.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:12:59.471877+0000 osd.7 (osd.7) 246 : cluster [DBG] 61.f deep-scrub ok 2024-02-19T21:13:00.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:12:59.797048+0000 mon.a (mon.0) 2905 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:00.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:00.165409+0000 osd.3 (osd.3) 235 : cluster [DBG] 6.b scrub starts 2024-02-19T21:13:00.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:00.166474+0000 osd.3 (osd.3) 236 : cluster [DBG] 6.b scrub ok 2024-02-19T21:13:00.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:00.328840+0000 mon.a (mon.0) 2906 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-90"}]': finished 2024-02-19T21:13:00.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:00.328912+0000 mon.a (mon.0) 2907 : cluster [DBG] osdmap e544: 8 total, 8 up, 6 in 2024-02-19T21:13:00.735 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:00.331692+0000 mon.b (mon.1) 702 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-90", "mode": "writeback"}]: dispatch 2024-02-19T21:13:00.736 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:00.332000+0000 mon.a (mon.0) 2908 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-90", "mode": "writeback"}]: dispatch 2024-02-19T21:13:00.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:58.836894+0000 osd.1 (osd.1) 251 : cluster [DBG] 5.c scrub starts 2024-02-19T21:13:00.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:58.837965+0000 osd.1 (osd.1) 252 : cluster [DBG] 5.c scrub ok 2024-02-19T21:13:00.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:59.262021+0000 mgr.y (mgr.14550) 483 : cluster [DBG] pgmap v813: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:00.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:59.464983+0000 osd.7 (osd.7) 245 : cluster [DBG] 61.f deep-scrub starts 2024-02-19T21:13:00.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:12:59.471877+0000 osd.7 (osd.7) 246 : cluster [DBG] 61.f deep-scrub ok 2024-02-19T21:13:00.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:12:59.797048+0000 mon.a (mon.0) 2905 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:00.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:00.165409+0000 osd.3 (osd.3) 235 : cluster [DBG] 6.b scrub starts 2024-02-19T21:13:00.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:00.166474+0000 osd.3 (osd.3) 236 : cluster [DBG] 6.b scrub ok 2024-02-19T21:13:00.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:00.328840+0000 mon.a (mon.0) 2906 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-90"}]': finished 2024-02-19T21:13:00.736 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:00.328912+0000 mon.a (mon.0) 2907 : cluster [DBG] osdmap e544: 8 total, 8 up, 6 in 2024-02-19T21:13:00.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:00.331692+0000 mon.b (mon.1) 702 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-90", "mode": "writeback"}]: dispatch 2024-02-19T21:13:00.737 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:00.332000+0000 mon.a (mon.0) 2908 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-90", "mode": "writeback"}]: dispatch 2024-02-19T21:13:00.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:13:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:13:00.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:58.836894+0000 osd.1 (osd.1) 251 : cluster [DBG] 5.c scrub starts 2024-02-19T21:13:00.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:58.837965+0000 osd.1 (osd.1) 252 : cluster [DBG] 5.c scrub ok 2024-02-19T21:13:00.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:59.262021+0000 mgr.y (mgr.14550) 483 : cluster [DBG] pgmap v813: 238 pgs: 32 unknown, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:00.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:59.464983+0000 osd.7 (osd.7) 245 : cluster [DBG] 61.f deep-scrub starts 2024-02-19T21:13:00.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:12:59.471877+0000 osd.7 (osd.7) 246 : cluster [DBG] 61.f deep-scrub ok 2024-02-19T21:13:00.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:12:59.797048+0000 mon.a (mon.0) 2905 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:00.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:00.165409+0000 osd.3 (osd.3) 235 : cluster [DBG] 6.b scrub starts 2024-02-19T21:13:00.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:00.166474+0000 osd.3 (osd.3) 236 : cluster [DBG] 6.b scrub ok 2024-02-19T21:13:00.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:00.328840+0000 mon.a (mon.0) 2906 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-6", "overlaypool": "test-rados-api-smithi163-83928-90"}]': finished 2024-02-19T21:13:00.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:00.328912+0000 mon.a (mon.0) 2907 : cluster [DBG] osdmap e544: 8 total, 8 up, 6 in 2024-02-19T21:13:00.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:00.331692+0000 mon.b (mon.1) 702 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-90", "mode": "writeback"}]: dispatch 2024-02-19T21:13:00.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:00.332000+0000 mon.a (mon.0) 2908 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-90", "mode": "writeback"}]: dispatch 2024-02-19T21:13:01.025 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-19T21:13:01.701 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:00.110018+0000 osd.2 (osd.2) 218 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:13:01.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:00.111167+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:13:01.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:00.425072+0000 osd.5 (osd.5) 222 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:13:01.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:00.428572+0000 osd.5 (osd.5) 223 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:13:01.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:00.554205+0000 mgr.y (mgr.14550) 484 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:01.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:01.325077+0000 mon.a (mon.0) 2909 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:13:01.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:01.329248+0000 mon.a (mon.0) 2910 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-90", "mode": "writeback"}]': finished 2024-02-19T21:13:01.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:01.329318+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e545: 8 total, 8 up, 6 in 2024-02-19T21:13:01.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:01.333200+0000 mon.b (mon.1) 703 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:13:01.702 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:01.333553+0000 mon.a (mon.0) 2912 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:13:01.703 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:00.110018+0000 osd.2 (osd.2) 218 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:13:01.703 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:00.111167+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:13:01.703 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:00.425072+0000 osd.5 (osd.5) 222 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:13:01.703 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:00.428572+0000 osd.5 (osd.5) 223 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:13:01.703 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:00.554205+0000 mgr.y (mgr.14550) 484 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:01.703 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:01.325077+0000 mon.a (mon.0) 2909 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:13:01.703 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:01.329248+0000 mon.a (mon.0) 2910 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-90", "mode": "writeback"}]': finished 2024-02-19T21:13:01.703 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:01.329318+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e545: 8 total, 8 up, 6 in 2024-02-19T21:13:01.703 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:01.333200+0000 mon.b (mon.1) 703 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:13:01.703 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:01.333553+0000 mon.a (mon.0) 2912 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:13:01.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.951+0000 7fa2242bb700 1 -- 172.21.15.163:0/1458029623 >> v1:172.21.15.163:6789/0 conn(0x7fa21c0f94d0 legacy=0x7fa21c1029a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:01.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.951+0000 7fa2242bb700 1 -- 172.21.15.163:0/1458029623 shutdown_connections 2024-02-19T21:13:01.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.951+0000 7fa2242bb700 1 -- 172.21.15.163:0/1458029623 >> 172.21.15.163:0/1458029623 conn(0x7fa21c0f2470 msgr2=0x7fa21c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:01.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.951+0000 7fa2242bb700 1 -- 172.21.15.163:0/1458029623 shutdown_connections 2024-02-19T21:13:01.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.951+0000 7fa2242bb700 1 -- 172.21.15.163:0/1458029623 wait complete. 2024-02-19T21:13:01.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa2242bb700 1 librados: starting msgr at 2024-02-19T21:13:01.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa2242bb700 1 librados: starting objecter 2024-02-19T21:13:01.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa2242bb700 1 Processor -- start 2024-02-19T21:13:01.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa2242bb700 1 -- start start 2024-02-19T21:13:01.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa2242bb700 1 librados: setting wanted keys 2024-02-19T21:13:01.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa2242bb700 1 librados: calling monclient init 2024-02-19T21:13:01.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa2242bb700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa21c117350 con 0x7fa21c1042b0 2024-02-19T21:13:01.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa2242bb700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa21c118530 con 0x7fa21c0f8920 2024-02-19T21:13:01.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa2242bb700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa21c119730 con 0x7fa21c0f94d0 2024-02-19T21:13:01.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa221856700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fa21c0f94d0 0x7fa21c0ffd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:60550/0 (socket says 172.21.15.163:60550) 2024-02-19T21:13:01.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa221856700 1 -- 172.21.15.163:0/3818331903 learned_addr learned my addr 172.21.15.163:0/3818331903 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:01.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa213779700 10 client.?.objecter ms_handle_connect 0x7fa21c0f94d0 2024-02-19T21:13:01.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa213779700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:01.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa213779700 10 client.?.objecter ms_handle_connect 0x7fa21c1042b0 2024-02-19T21:13:01.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa213779700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:01.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa213779700 10 client.?.objecter ms_handle_connect 0x7fa21c0f8920 2024-02-19T21:13:01.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa213779700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:01.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495968810 0 0) 0x7fa21c119730 con 0x7fa21c0f94d0 2024-02-19T21:13:01.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa208003030 con 0x7fa21c0f94d0 2024-02-19T21:13:01.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215756070 0 0) 0x7fa21c117350 con 0x7fa21c1042b0 2024-02-19T21:13:01.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.952+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa21c119730 con 0x7fa21c1042b0 2024-02-19T21:13:01.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075078950 0 0) 0x7fa21c118530 con 0x7fa21c0f8920 2024-02-19T21:13:01.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa21c117350 con 0x7fa21c0f8920 2024-02-19T21:13:01.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 485972021 0 0) 0x7fa208003030 con 0x7fa21c0f94d0 2024-02-19T21:13:01.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa21c118530 con 0x7fa21c0f94d0 2024-02-19T21:13:01.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa2180015d0 con 0x7fa21c0f94d0 2024-02-19T21:13:01.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 462830254 0 0) 0x7fa21c119730 con 0x7fa21c1042b0 2024-02-19T21:13:01.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa208003030 con 0x7fa21c1042b0 2024-02-19T21:13:01.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa214001b00 con 0x7fa21c1042b0 2024-02-19T21:13:01.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3101103112 0 0) 0x7fa21c117350 con 0x7fa21c0f8920 2024-02-19T21:13:01.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa21c119730 con 0x7fa21c0f8920 2024-02-19T21:13:01.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa20c0019f0 con 0x7fa21c0f8920 2024-02-19T21:13:01.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3405323101 0 0) 0x7fa208003030 con 0x7fa21c1042b0 2024-02-19T21:13:01.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 >> v1:172.21.15.163:6790/0 conn(0x7fa21c0f94d0 legacy=0x7fa21c0ffd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:01.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 >> v1:172.21.15.193:6789/0 conn(0x7fa21c0f8920 legacy=0x7fa21c0ff650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:01.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c11a930 con 0x7fa21c1042b0 2024-02-19T21:13:01.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa2242bb700 1 -- 172.21.15.163:0/3818331903 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa21c117540 con 0x7fa21c1042b0 2024-02-19T21:13:01.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa2242bb700 10 client.23409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:01.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa2242bb700 1 -- 172.21.15.163:0/3818331903 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa21c1179b0 con 0x7fa21c1042b0 2024-02-19T21:13:01.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.953+0000 7fa2242bb700 1 librados: init done 2024-02-19T21:13:01.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.954+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fa2140021c0 con 0x7fa21c1042b0 2024-02-19T21:13:01.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.954+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa214003520 con 0x7fa21c1042b0 2024-02-19T21:13:01.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.954+0000 7fa2242bb700 1 -- 172.21.15.163:0/3818331903 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa200004f20 con 0x7fa21c1042b0 2024-02-19T21:13:01.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.954+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fa2140163b0 con 0x7fa21c1042b0 2024-02-19T21:13:01.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.955+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(545..545 src has 43..545) v4 ==== 8100+0+0 (unknown 1401920718 0 0) 0x7fa2140767a0 con 0x7fa21c1042b0 2024-02-19T21:13:01.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.955+0000 7fa213779700 10 client.23409.objecter ms_dispatch 0x7fa21c001550 osd_map(545..545 src has 43..545) v4 2024-02-19T21:13:01.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.955+0000 7fa213779700 3 client.23409.objecter handle_osd_map got epochs [545,545] > 0 2024-02-19T21:13:01.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.955+0000 7fa213779700 3 client.23409.objecter handle_osd_map decoding full epoch 545 2024-02-19T21:13:01.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.955+0000 7fa213779700 20 client.23409.objecter dump_active .. 0 homeless 2024-02-19T21:13:01.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.955+0000 7fa213779700 10 client.23409.objecter ms_handle_connect 0x7fa208063cc0 2024-02-19T21:13:01.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:01.959+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa214049bd0 con 0x7fa21c1042b0 2024-02-19T21:13:01.963 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:13:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:01.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:13:02.017 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:02.017 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:02.068 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:02.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:00.110018+0000 osd.2 (osd.2) 218 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:13:02.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:00.111167+0000 osd.2 (osd.2) 219 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:13:02.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:00.425072+0000 osd.5 (osd.5) 222 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:13:02.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:00.428572+0000 osd.5 (osd.5) 223 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:13:02.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:00.554205+0000 mgr.y (mgr.14550) 484 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:02.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:01.325077+0000 mon.a (mon.0) 2909 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:13:02.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:01.329248+0000 mon.a (mon.0) 2910 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-90", "mode": "writeback"}]': finished 2024-02-19T21:13:02.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:01.329318+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e545: 8 total, 8 up, 6 in 2024-02-19T21:13:02.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:01.333200+0000 mon.b (mon.1) 703 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:13:02.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:01.333553+0000 mon.a (mon.0) 2912 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:13:02.100 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:02.122 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:02.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:02.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:02.159+0000 7fa2242bb700 1 -- 172.21.15.163:0/3818331903 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7fa2000051d0 con 0x7fa21c1042b0 2024-02-19T21:13:02.226 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:02.255 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:02.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:02.330+0000 7fa213779700 1 -- 172.21.15.163:0/3818331903 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v546) v1 ==== 113+0+0 (unknown 971582716 0 0) 0x7fa214047fe0 con 0x7fa21c1042b0 2024-02-19T21:13:02.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:02.334+0000 7fa2242bb700 10 librados: watch_flush enter 2024-02-19T21:13:02.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:02.334+0000 7fa2242bb700 10 librados: watch_flush exit 2024-02-19T21:13:02.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:02.334+0000 7fa2242bb700 20 client.23409.objecter shutdown clearing up homeless session... 2024-02-19T21:13:02.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:02.334+0000 7fa2242bb700 10 client.23409.objecter successfully canceled tick 2024-02-19T21:13:02.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:02.334+0000 7fa2242bb700 1 -- 172.21.15.163:0/3818331903 >> v1:172.21.15.163:6800/1552514681 conn(0x7fa208063cc0 legacy=0x7fa208066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:02.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:02.334+0000 7fa2242bb700 1 -- 172.21.15.163:0/3818331903 >> v1:172.21.15.163:6789/0 conn(0x7fa21c1042b0 legacy=0x7fa21c115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:02.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:02.335+0000 7fa2242bb700 1 -- 172.21.15.163:0/3818331903 shutdown_connections 2024-02-19T21:13:02.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:02.335+0000 7fa2242bb700 1 -- 172.21.15.163:0/3818331903 >> 172.21.15.163:0/3818331903 conn(0x7fa21c0f2470 msgr2=0x7fa21c107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:02.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:02.335+0000 7fa2242bb700 1 -- 172.21.15.163:0/3818331903 shutdown_connections 2024-02-19T21:13:02.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:02.335+0000 7fa2242bb700 1 -- 172.21.15.163:0/3818331903 wait complete. 2024-02-19T21:13:02.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:02.335+0000 7fa2242bb700 1 librados: shutdown 2024-02-19T21:13:02.336 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.4 to 1 (10000) 2024-02-19T21:13:02.378 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:02.381 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:02.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:02.483 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:02.517 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:02.539 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:02.542 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:02.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:02.597 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:00.873313+0000 osd.1 (osd.1) 253 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:13:02.597 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:00.874314+0000 osd.1 (osd.1) 254 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:13:02.597 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:01.262409+0000 mgr.y (mgr.14550) 485 : cluster [DBG] pgmap v816: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:02.598 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:01.467322+0000 osd.4 (osd.4) 199 : cluster [DBG] 61.16 scrub starts 2024-02-19T21:13:02.598 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:01.469177+0000 osd.4 (osd.4) 200 : cluster [DBG] 61.16 scrub ok 2024-02-19T21:13:02.598 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:02.161450+0000 mon.a (mon.0) 2913 : audit [INF] from='client.? v1:172.21.15.163:0/3818331903' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:13:02.598 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:02.331103+0000 mon.a (mon.0) 2914 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:13:02.598 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:02.331181+0000 mon.a (mon.0) 2915 : audit [INF] from='client.? v1:172.21.15.163:0/3818331903' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-19T21:13:02.598 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:02.331229+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e546: 8 total, 8 up, 6 in 2024-02-19T21:13:02.598 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:02.333481+0000 mon.b (mon.1) 704 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:13:02.598 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:02.333820+0000 mon.a (mon.0) 2917 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:13:02.642 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:13:02.672 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:02.691 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:13:02.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:02.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:13:02.796 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:13:02.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:00.873313+0000 osd.1 (osd.1) 253 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:13:02.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:00.874314+0000 osd.1 (osd.1) 254 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:13:02.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:01.262409+0000 mgr.y (mgr.14550) 485 : cluster [DBG] pgmap v816: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:02.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:01.467322+0000 osd.4 (osd.4) 199 : cluster [DBG] 61.16 scrub starts 2024-02-19T21:13:02.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:01.469177+0000 osd.4 (osd.4) 200 : cluster [DBG] 61.16 scrub ok 2024-02-19T21:13:02.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:02.161450+0000 mon.a (mon.0) 2913 : audit [INF] from='client.? v1:172.21.15.163:0/3818331903' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:13:02.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:02.331103+0000 mon.a (mon.0) 2914 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:13:02.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:02.331181+0000 mon.a (mon.0) 2915 : audit [INF] from='client.? v1:172.21.15.163:0/3818331903' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-19T21:13:02.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:02.331229+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e546: 8 total, 8 up, 6 in 2024-02-19T21:13:02.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:02.333481+0000 mon.b (mon.1) 704 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:13:02.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:02.333820+0000 mon.a (mon.0) 2917 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:13:02.823 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:02.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:00.873313+0000 osd.1 (osd.1) 253 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:13:02.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:00.874314+0000 osd.1 (osd.1) 254 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:13:02.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:01.262409+0000 mgr.y (mgr.14550) 485 : cluster [DBG] pgmap v816: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:02.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:01.467322+0000 osd.4 (osd.4) 199 : cluster [DBG] 61.16 scrub starts 2024-02-19T21:13:02.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:01.469177+0000 osd.4 (osd.4) 200 : cluster [DBG] 61.16 scrub ok 2024-02-19T21:13:02.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:02.161450+0000 mon.a (mon.0) 2913 : audit [INF] from='client.? v1:172.21.15.163:0/3818331903' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:13:02.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:02.331103+0000 mon.a (mon.0) 2914 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:13:02.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:02.331181+0000 mon.a (mon.0) 2915 : audit [INF] from='client.? v1:172.21.15.163:0/3818331903' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-19T21:13:02.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:02.331229+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e546: 8 total, 8 up, 6 in 2024-02-19T21:13:02.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:02.333481+0000 mon.b (mon.1) 704 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:13:02.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:02.333820+0000 mon.a (mon.0) 2917 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:13:03.470 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:13:03.544 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-19T21:13:03.544 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:13:03.747 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:13:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:13:03] "GET /metrics HTTP/1.1" 200 190516 "" "Prometheus/2.33.4" 2024-02-19T21:13:04.073 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:13:04.076 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:04.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:13:04.177 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:04.205 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:04.353 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:04.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:04.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:04.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:13:04.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:03.125478+0000 osd.2 (osd.2) 220 : cluster [DBG] 6.f scrub starts 2024-02-19T21:13:04.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:03.126410+0000 osd.2 (osd.2) 221 : cluster [DBG] 6.f scrub ok 2024-02-19T21:13:04.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:03.262746+0000 mgr.y (mgr.14550) 486 : cluster [DBG] pgmap v819: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:04.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:03.332480+0000 mon.a (mon.0) 2918 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:13:04.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:03.332556+0000 mon.a (mon.0) 2919 : cluster [DBG] osdmap e547: 8 total, 8 up, 6 in 2024-02-19T21:13:04.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:03.334172+0000 mon.b (mon.1) 705 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:13:04.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:03.336741+0000 mon.a (mon.0) 2920 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:13:04.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:04.316893+0000 mon.a (mon.0) 2921 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:13:04.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:04.321368+0000 mon.a (mon.0) 2922 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-19T21:13:04.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:04.321449+0000 mon.a (mon.0) 2923 : cluster [DBG] osdmap e548: 8 total, 8 up, 6 in 2024-02-19T21:13:04.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:04.324991+0000 mon.b (mon.1) 706 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-19T21:13:04.484 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:04.325240+0000 mon.a (mon.0) 2924 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-19T21:13:04.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:03.125478+0000 osd.2 (osd.2) 220 : cluster [DBG] 6.f scrub starts 2024-02-19T21:13:04.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:03.126410+0000 osd.2 (osd.2) 221 : cluster [DBG] 6.f scrub ok 2024-02-19T21:13:04.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:03.262746+0000 mgr.y (mgr.14550) 486 : cluster [DBG] pgmap v819: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:04.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:03.332480+0000 mon.a (mon.0) 2918 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:13:04.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:03.332556+0000 mon.a (mon.0) 2919 : cluster [DBG] osdmap e547: 8 total, 8 up, 6 in 2024-02-19T21:13:04.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:03.334172+0000 mon.b (mon.1) 705 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:13:04.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:03.336741+0000 mon.a (mon.0) 2920 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:13:04.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:04.316893+0000 mon.a (mon.0) 2921 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:13:04.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:04.321368+0000 mon.a (mon.0) 2922 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-19T21:13:04.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:04.321449+0000 mon.a (mon.0) 2923 : cluster [DBG] osdmap e548: 8 total, 8 up, 6 in 2024-02-19T21:13:04.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:04.324991+0000 mon.b (mon.1) 706 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-19T21:13:04.485 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:04.325240+0000 mon.a (mon.0) 2924 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-19T21:13:04.486 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:04.538 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:13:04.540 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:04.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:13:04.642 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:04.670 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:04.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:03.125478+0000 osd.2 (osd.2) 220 : cluster [DBG] 6.f scrub starts 2024-02-19T21:13:04.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:03.126410+0000 osd.2 (osd.2) 221 : cluster [DBG] 6.f scrub ok 2024-02-19T21:13:04.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:03.262746+0000 mgr.y (mgr.14550) 486 : cluster [DBG] pgmap v819: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:04.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:03.332480+0000 mon.a (mon.0) 2918 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:13:04.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:03.332556+0000 mon.a (mon.0) 2919 : cluster [DBG] osdmap e547: 8 total, 8 up, 6 in 2024-02-19T21:13:04.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:03.334172+0000 mon.b (mon.1) 705 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:13:04.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:03.336741+0000 mon.a (mon.0) 2920 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:13:04.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:04.316893+0000 mon.a (mon.0) 2921 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:13:04.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:04.321368+0000 mon.a (mon.0) 2922 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-19T21:13:04.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:04.321449+0000 mon.a (mon.0) 2923 : cluster [DBG] osdmap e548: 8 total, 8 up, 6 in 2024-02-19T21:13:04.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:04.324991+0000 mon.b (mon.1) 706 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-19T21:13:04.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:04.325240+0000 mon.a (mon.0) 2924 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-02-19T21:13:04.867 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:04.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:04.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:04.971 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:05.003 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:05.024 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:05.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:05.128 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:05.157 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:05.406 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:05.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:05.510 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:05.539 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.614+0000 7faea4fc0700 1 -- 172.21.15.163:0/4260326606 >> v1:172.21.15.163:6790/0 conn(0x7faea00fd970 legacy=0x7faea00fdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.614+0000 7faea4fc0700 1 -- 172.21.15.163:0/4260326606 shutdown_connections 2024-02-19T21:13:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.614+0000 7faea4fc0700 1 -- 172.21.15.163:0/4260326606 >> 172.21.15.163:0/4260326606 conn(0x7faea00f3570 msgr2=0x7faea00f5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.614+0000 7faea4fc0700 1 -- 172.21.15.163:0/4260326606 shutdown_connections 2024-02-19T21:13:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.614+0000 7faea4fc0700 1 -- 172.21.15.163:0/4260326606 wait complete. 2024-02-19T21:13:05.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.614+0000 7faea4fc0700 1 librados: starting msgr at 2024-02-19T21:13:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.614+0000 7faea4fc0700 1 librados: starting objecter 2024-02-19T21:13:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7faea4fc0700 1 Processor -- start 2024-02-19T21:13:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7faea4fc0700 1 -- start start 2024-02-19T21:13:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7faea4fc0700 1 librados: setting wanted keys 2024-02-19T21:13:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7faea4fc0700 1 librados: calling monclient init 2024-02-19T21:13:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7faea4fc0700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faea0117290 con 0x7faea00fd970 2024-02-19T21:13:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7faea4fc0700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faea0118470 con 0x7faea0100770 2024-02-19T21:13:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7faea4fc0700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faea0119670 con 0x7faea01042b0 2024-02-19T21:13:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7fae9e59c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7faea00fd970 0x7faea00f9950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33468/0 (socket says 172.21.15.163:33468) 2024-02-19T21:13:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7fae9e59c700 1 -- 172.21.15.163:0/3780174867 learned_addr learned my addr 172.21.15.163:0/3780174867 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7fae8f7fe700 10 client.?.objecter ms_handle_connect 0x7faea01042b0 2024-02-19T21:13:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7fae8f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:05.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7fae8f7fe700 10 client.?.objecter ms_handle_connect 0x7faea00fd970 2024-02-19T21:13:05.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7fae8f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:05.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7fae8f7fe700 10 client.?.objecter ms_handle_connect 0x7faea0100770 2024-02-19T21:13:05.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7fae8f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:05.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.615+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860290498 0 0) 0x7faea0119670 con 0x7faea01042b0 2024-02-19T21:13:05.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae880033f0 con 0x7faea01042b0 2024-02-19T21:13:05.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859378907 0 0) 0x7faea0117290 con 0x7faea00fd970 2024-02-19T21:13:05.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faea0119670 con 0x7faea00fd970 2024-02-19T21:13:05.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075254493 0 0) 0x7faea0118470 con 0x7faea0100770 2024-02-19T21:13:05.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faea0117290 con 0x7faea0100770 2024-02-19T21:13:05.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2236548386 0 0) 0x7fae880033f0 con 0x7faea01042b0 2024-02-19T21:13:05.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faea0118470 con 0x7faea01042b0 2024-02-19T21:13:05.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fae98001c40 con 0x7faea01042b0 2024-02-19T21:13:05.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1664888833 0 0) 0x7faea0119670 con 0x7faea00fd970 2024-02-19T21:13:05.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae880033f0 con 0x7faea00fd970 2024-02-19T21:13:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 198666271 0 0) 0x7faea0117290 con 0x7faea0100770 2024-02-19T21:13:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faea0119670 con 0x7faea0100770 2024-02-19T21:13:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fae900020b0 con 0x7faea00fd970 2024-02-19T21:13:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fae94001830 con 0x7faea0100770 2024-02-19T21:13:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 729074262 0 0) 0x7faea0118470 con 0x7faea01042b0 2024-02-19T21:13:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 >> v1:172.21.15.193:6789/0 conn(0x7faea0100770 legacy=0x7faea00fa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 >> v1:172.21.15.163:6789/0 conn(0x7faea00fd970 legacy=0x7faea00f9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea011a870 con 0x7faea01042b0 2024-02-19T21:13:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea0117480 con 0x7faea01042b0 2024-02-19T21:13:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7faea4fc0700 10 client.50144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faea0117970 con 0x7faea01042b0 2024-02-19T21:13:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.616+0000 7faea4fc0700 1 librados: init done 2024-02-19T21:13:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.617+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fae980023d0 con 0x7faea01042b0 2024-02-19T21:13:05.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.617+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fae98003690 con 0x7faea01042b0 2024-02-19T21:13:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.617+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fae98016620 con 0x7faea01042b0 2024-02-19T21:13:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.617+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7faea0060370 con 0x7faea01042b0 2024-02-19T21:13:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(549..549 src has 43..549) v4 ==== 8100+0+0 (unknown 1721027671 0 0) 0x7fae98077790 con 0x7faea01042b0 2024-02-19T21:13:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 osd_map(549..549 src has 43..549) v4 2024-02-19T21:13:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7fae8f7fe700 3 client.50144.objecter handle_osd_map got epochs [549,549] > 0 2024-02-19T21:13:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7fae8f7fe700 3 client.50144.objecter handle_osd_map decoding full epoch 549 2024-02-19T21:13:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7fae8f7fe700 20 client.50144.objecter dump_active .. 0 homeless 2024-02-19T21:13:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v549) v1 ==== 54+0+16 (unknown 2534082343 0 2420000192) 0x7fae98046970 con 0x7faea01042b0 2024-02-19T21:13:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7fae8f7fe700 10 client.50144.objecter ms_handle_connect 0x7fae88068300 2024-02-19T21:13:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7faea4fc0700 10 client.50144.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 1 2024-02-19T21:13:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea0047540 osd=0 3 2024-02-19T21:13:05.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea0047540 osd=0 3 2024-02-19T21:13:05.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea0047540 osd=0 3 2024-02-19T21:13:05.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 1 2024-02-19T21:13:05.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea0047540 osd=0 3 2024-02-19T21:13:05.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 0 1 2024-02-19T21:13:05.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea0047540 osd=0 4 2024-02-19T21:13:05.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7faea4fc0700 10 client.50144.objecter _send_command 1 2024-02-19T21:13:05.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faea006cf20 con 0x7faea0118600 2024-02-19T21:13:05.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.618+0000 7fae8f7fe700 10 client.50144.objecter ms_handle_connect 0x7faea0118600 2024-02-19T21:13:05.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.619+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7faea006cf20 con 0x7faea0118600 2024-02-19T21:13:05.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.619+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:13:05.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.619+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 1 = system:0 2024-02-19T21:13:05.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.619+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea0047540 osd=0 4 2024-02-19T21:13:05.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.619+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 0 1 2024-02-19T21:13:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:05.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.635+0000 7faea4fc0700 10 client.50144.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:05.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.635+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 2 2024-02-19T21:13:05.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.635+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea0047540 osd=0 3 2024-02-19T21:13:05.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.635+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea0047540 osd=0 3 2024-02-19T21:13:05.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.635+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea0047540 osd=0 3 2024-02-19T21:13:05.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.635+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 2 2024-02-19T21:13:05.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.635+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea0047540 osd=0 3 2024-02-19T21:13:05.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.635+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 0 2 2024-02-19T21:13:05.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.635+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea0047540 osd=0 4 2024-02-19T21:13:05.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.635+0000 7faea4fc0700 10 client.50144.objecter _send_command 2 2024-02-19T21:13:05.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.635+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faea0109e90 con 0x7faea0118600 2024-02-19T21:13:05.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.638+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7faea0109e90 con 0x7faea0118600 2024-02-19T21:13:05.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.638+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:05.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.638+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.638+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea0047540 osd=0 4 2024-02-19T21:13:05.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.638+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 0 2 2024-02-19T21:13:05.639 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:05.639 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:13:05.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.639+0000 7faea4fc0700 10 client.50144.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:05.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.639+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 3 2024-02-19T21:13:05.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.639+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea00f3950 osd=1 3 2024-02-19T21:13:05.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.639+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea00f3950 osd=1 3 2024-02-19T21:13:05.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.639+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea00f3950 osd=1 3 2024-02-19T21:13:05.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.639+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 3 2024-02-19T21:13:05.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.639+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea00f3950 osd=1 3 2024-02-19T21:13:05.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.639+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 1 3 2024-02-19T21:13:05.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.639+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea00f3950 osd=1 4 2024-02-19T21:13:05.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.639+0000 7faea4fc0700 10 client.50144.objecter _send_command 3 2024-02-19T21:13:05.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.639+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7faea00283d0 con 0x7faea00f3fd0 2024-02-19T21:13:05.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.640+0000 7fae8f7fe700 10 client.50144.objecter ms_handle_connect 0x7faea00f3fd0 2024-02-19T21:13:05.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.641+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7faea00283d0 con 0x7faea00f3fd0 2024-02-19T21:13:05.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.641+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:13:05.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.641+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 3 = system:0 2024-02-19T21:13:05.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.641+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea00f3950 osd=1 4 2024-02-19T21:13:05.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.641+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 1 3 2024-02-19T21:13:05.643 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:05.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:05.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.657+0000 7faea4fc0700 10 client.50144.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:05.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.657+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 4 2024-02-19T21:13:05.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.657+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea00f3950 osd=1 3 2024-02-19T21:13:05.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.657+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea00f3950 osd=1 3 2024-02-19T21:13:05.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.657+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea00f3950 osd=1 3 2024-02-19T21:13:05.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.658+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 4 2024-02-19T21:13:05.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.658+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea00f3950 osd=1 3 2024-02-19T21:13:05.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.658+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 1 4 2024-02-19T21:13:05.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.658+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea00f3950 osd=1 4 2024-02-19T21:13:05.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.658+0000 7faea4fc0700 10 client.50144.objecter _send_command 4 2024-02-19T21:13:05.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.658+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faea00287b0 con 0x7faea00f3fd0 2024-02-19T21:13:05.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7faea00287b0 con 0x7faea00f3fd0 2024-02-19T21:13:05.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:05.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea00f3950 osd=1 4 2024-02-19T21:13:05.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 1 4 2024-02-19T21:13:05.659 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:13:05.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7faea4fc0700 10 client.50144.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:05.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 5 2024-02-19T21:13:05.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea0117ea0 osd=2 3 2024-02-19T21:13:05.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea0117ea0 osd=2 3 2024-02-19T21:13:05.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea0117ea0 osd=2 3 2024-02-19T21:13:05.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 5 2024-02-19T21:13:05.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea0117ea0 osd=2 3 2024-02-19T21:13:05.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 2 5 2024-02-19T21:13:05.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea0117ea0 osd=2 4 2024-02-19T21:13:05.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7faea4fc0700 10 client.50144.objecter _send_command 5 2024-02-19T21:13:05.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.659+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7faea0059e00 con 0x7faea0118010 2024-02-19T21:13:05.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.660+0000 7fae8f7fe700 10 client.50144.objecter ms_handle_connect 0x7faea0118010 2024-02-19T21:13:05.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.660+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7faea0059e00 con 0x7faea0118010 2024-02-19T21:13:05.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.660+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:13:05.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.660+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 5 = system:0 2024-02-19T21:13:05.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.660+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea0117ea0 osd=2 4 2024-02-19T21:13:05.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.660+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 2 5 2024-02-19T21:13:05.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.675+0000 7faea4fc0700 10 client.50144.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:05.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.675+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 6 2024-02-19T21:13:05.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.675+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea0117ea0 osd=2 3 2024-02-19T21:13:05.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.675+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea0117ea0 osd=2 3 2024-02-19T21:13:05.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.675+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea0117ea0 osd=2 3 2024-02-19T21:13:05.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.675+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 6 2024-02-19T21:13:05.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.675+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea0117ea0 osd=2 3 2024-02-19T21:13:05.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.675+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 2 6 2024-02-19T21:13:05.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.675+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea0117ea0 osd=2 4 2024-02-19T21:13:05.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.675+0000 7faea4fc0700 10 client.50144.objecter _send_command 6 2024-02-19T21:13:05.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.675+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faea0121380 con 0x7faea0118010 2024-02-19T21:13:05.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7faea0121380 con 0x7faea0118010 2024-02-19T21:13:05.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:05.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea0117ea0 osd=2 4 2024-02-19T21:13:05.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 2 6 2024-02-19T21:13:05.676 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:13:05.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7faea4fc0700 10 client.50144.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:05.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 7 2024-02-19T21:13:05.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea0059360 osd=3 3 2024-02-19T21:13:05.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea0059360 osd=3 3 2024-02-19T21:13:05.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea0059360 osd=3 3 2024-02-19T21:13:05.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 7 2024-02-19T21:13:05.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea0059360 osd=3 3 2024-02-19T21:13:05.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 3 7 2024-02-19T21:13:05.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea0059360 osd=3 4 2024-02-19T21:13:05.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7faea4fc0700 10 client.50144.objecter _send_command 7 2024-02-19T21:13:05.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.676+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7faea011d8a0 con 0x7faea00594d0 2024-02-19T21:13:05.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.677+0000 7fae8f7fe700 10 client.50144.objecter ms_handle_connect 0x7faea00594d0 2024-02-19T21:13:05.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.677+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7faea011d8a0 con 0x7faea00594d0 2024-02-19T21:13:05.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.677+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:13:05.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.677+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 7 = system:0 2024-02-19T21:13:05.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.677+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea0059360 osd=3 4 2024-02-19T21:13:05.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.677+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 3 7 2024-02-19T21:13:05.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.691+0000 7faea4fc0700 10 client.50144.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:05.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.691+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 8 2024-02-19T21:13:05.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.691+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea0059360 osd=3 3 2024-02-19T21:13:05.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.691+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea0059360 osd=3 3 2024-02-19T21:13:05.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.691+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea0059360 osd=3 3 2024-02-19T21:13:05.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.691+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 8 2024-02-19T21:13:05.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.691+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea0059360 osd=3 3 2024-02-19T21:13:05.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.691+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 3 8 2024-02-19T21:13:05.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.691+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea0059360 osd=3 4 2024-02-19T21:13:05.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.691+0000 7faea4fc0700 10 client.50144.objecter _send_command 8 2024-02-19T21:13:05.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.691+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faea011d8a0 con 0x7faea00594d0 2024-02-19T21:13:05.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7faea011d8a0 con 0x7faea00594d0 2024-02-19T21:13:05.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:05.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea0059360 osd=3 4 2024-02-19T21:13:05.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 3 8 2024-02-19T21:13:05.693 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:13:05.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7faea4fc0700 10 client.50144.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:05.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 9 2024-02-19T21:13:05.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea011d8a0 osd=4 3 2024-02-19T21:13:05.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea011d8a0 osd=4 3 2024-02-19T21:13:05.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea011d8a0 osd=4 3 2024-02-19T21:13:05.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 9 2024-02-19T21:13:05.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea011d8a0 osd=4 3 2024-02-19T21:13:05.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 4 9 2024-02-19T21:13:05.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea011d8a0 osd=4 4 2024-02-19T21:13:05.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7faea4fc0700 10 client.50144.objecter _send_command 9 2024-02-19T21:13:05.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7faea011dae0 con 0x7faea0123590 2024-02-19T21:13:05.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.693+0000 7fae8f7fe700 10 client.50144.objecter ms_handle_connect 0x7faea0123590 2024-02-19T21:13:05.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.694+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7faea011dae0 con 0x7faea0123590 2024-02-19T21:13:05.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.694+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:13:05.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.694+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 9 = system:0 2024-02-19T21:13:05.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.694+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea011d8a0 osd=4 4 2024-02-19T21:13:05.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.694+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 4 9 2024-02-19T21:13:05.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.709+0000 7faea4fc0700 10 client.50144.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:05.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.709+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 10 2024-02-19T21:13:05.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.709+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea011d8a0 osd=4 3 2024-02-19T21:13:05.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.709+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea011d8a0 osd=4 3 2024-02-19T21:13:05.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.709+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea011d8a0 osd=4 3 2024-02-19T21:13:05.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.709+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 10 2024-02-19T21:13:05.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.709+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea011d8a0 osd=4 3 2024-02-19T21:13:05.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.709+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 4 10 2024-02-19T21:13:05.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.709+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea011d8a0 osd=4 4 2024-02-19T21:13:05.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.709+0000 7faea4fc0700 10 client.50144.objecter _send_command 10 2024-02-19T21:13:05.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.709+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faea011dae0 con 0x7faea0123590 2024-02-19T21:13:05.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7faea011dae0 con 0x7faea0123590 2024-02-19T21:13:05.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:05.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea011d8a0 osd=4 4 2024-02-19T21:13:05.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 4 10 2024-02-19T21:13:05.711 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:13:05.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7faea4fc0700 10 client.50144.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:05.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 11 2024-02-19T21:13:05.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea011dae0 osd=5 3 2024-02-19T21:13:05.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea011dae0 osd=5 3 2024-02-19T21:13:05.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea011dae0 osd=5 3 2024-02-19T21:13:05.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 11 2024-02-19T21:13:05.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea011dae0 osd=5 3 2024-02-19T21:13:05.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 5 11 2024-02-19T21:13:05.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea011dae0 osd=5 4 2024-02-19T21:13:05.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7faea4fc0700 10 client.50144.objecter _send_command 11 2024-02-19T21:13:05.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7faea012e5a0 con 0x7faea012a900 2024-02-19T21:13:05.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.711+0000 7fae8f7fe700 10 client.50144.objecter ms_handle_connect 0x7faea012a900 2024-02-19T21:13:05.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.712+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7faea012e5a0 con 0x7faea012a900 2024-02-19T21:13:05.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.712+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:13:05.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.712+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 11 = system:0 2024-02-19T21:13:05.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.712+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea011dae0 osd=5 4 2024-02-19T21:13:05.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.712+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 5 11 2024-02-19T21:13:05.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.726+0000 7faea4fc0700 10 client.50144.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:05.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.726+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 12 2024-02-19T21:13:05.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.726+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea011dae0 osd=5 3 2024-02-19T21:13:05.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.726+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea011dae0 osd=5 3 2024-02-19T21:13:05.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.726+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea011dae0 osd=5 3 2024-02-19T21:13:05.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.726+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 12 2024-02-19T21:13:05.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.726+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea011dae0 osd=5 3 2024-02-19T21:13:05.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.726+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 5 12 2024-02-19T21:13:05.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.726+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea011dae0 osd=5 4 2024-02-19T21:13:05.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.726+0000 7faea4fc0700 10 client.50144.objecter _send_command 12 2024-02-19T21:13:05.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.726+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faea012e5a0 con 0x7faea012a900 2024-02-19T21:13:05.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.728+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7faea012e5a0 con 0x7faea012a900 2024-02-19T21:13:05.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.728+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:05.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.728+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.728+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea011dae0 osd=5 4 2024-02-19T21:13:05.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.728+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 5 12 2024-02-19T21:13:05.729 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:13:05.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.729+0000 7faea4fc0700 10 client.50144.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:05.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.729+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 13 2024-02-19T21:13:05.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.729+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea012e5a0 osd=6 3 2024-02-19T21:13:05.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.729+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea012e5a0 osd=6 3 2024-02-19T21:13:05.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.729+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea012e5a0 osd=6 3 2024-02-19T21:13:05.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.729+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 13 2024-02-19T21:13:05.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.729+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea012e5a0 osd=6 3 2024-02-19T21:13:05.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.729+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 6 13 2024-02-19T21:13:05.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.729+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea012e5a0 osd=6 4 2024-02-19T21:13:05.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.729+0000 7faea4fc0700 10 client.50144.objecter _send_command 13 2024-02-19T21:13:05.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.729+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7faea012f780 con 0x7faea0130260 2024-02-19T21:13:05.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.729+0000 7fae8f7fe700 10 client.50144.objecter ms_handle_connect 0x7faea0130260 2024-02-19T21:13:05.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.730+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7faea012f780 con 0x7faea0130260 2024-02-19T21:13:05.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.730+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:13:05.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.730+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 13 = system:0 2024-02-19T21:13:05.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.730+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea012e5a0 osd=6 4 2024-02-19T21:13:05.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.730+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 6 13 2024-02-19T21:13:05.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.744+0000 7faea4fc0700 10 client.50144.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:05.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.744+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 14 2024-02-19T21:13:05.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.744+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea012e5a0 osd=6 3 2024-02-19T21:13:05.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.744+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea012e5a0 osd=6 3 2024-02-19T21:13:05.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.744+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea012e5a0 osd=6 3 2024-02-19T21:13:05.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.744+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 14 2024-02-19T21:13:05.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.744+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea012e5a0 osd=6 3 2024-02-19T21:13:05.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.744+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 6 14 2024-02-19T21:13:05.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.744+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea012e5a0 osd=6 4 2024-02-19T21:13:05.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.744+0000 7faea4fc0700 10 client.50144.objecter _send_command 14 2024-02-19T21:13:05.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.744+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faea012f780 con 0x7faea0130260 2024-02-19T21:13:05.745 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:05.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7faea012f780 con 0x7faea0130260 2024-02-19T21:13:05.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:05.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea012e5a0 osd=6 4 2024-02-19T21:13:05.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 6 14 2024-02-19T21:13:05.746 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:13:05.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7faea4fc0700 10 client.50144.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:05.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 15 2024-02-19T21:13:05.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea012f780 osd=7 3 2024-02-19T21:13:05.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea012f780 osd=7 3 2024-02-19T21:13:05.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea012f780 osd=7 3 2024-02-19T21:13:05.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 15 2024-02-19T21:13:05.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea012f780 osd=7 3 2024-02-19T21:13:05.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 7 15 2024-02-19T21:13:05.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea012f780 osd=7 4 2024-02-19T21:13:05.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7faea4fc0700 10 client.50144.objecter _send_command 15 2024-02-19T21:13:05.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.746+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7faea012fba0 con 0x7faea01334b0 2024-02-19T21:13:05.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.747+0000 7fae8f7fe700 10 client.50144.objecter ms_handle_connect 0x7faea01334b0 2024-02-19T21:13:05.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.747+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7faea012fba0 con 0x7faea01334b0 2024-02-19T21:13:05.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.747+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:13:05.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.747+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 15 = system:0 2024-02-19T21:13:05.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.747+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea012f780 osd=7 4 2024-02-19T21:13:05.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.747+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 7 15 2024-02-19T21:13:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.762+0000 7faea4fc0700 10 client.50144.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.762+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign -1 16 2024-02-19T21:13:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.762+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea012f780 osd=7 3 2024-02-19T21:13:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.762+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea012f780 osd=7 3 2024-02-19T21:13:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.762+0000 7faea4fc0700 20 client.50144.objecter _get_session s=0x7faea012f780 osd=7 3 2024-02-19T21:13:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.762+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_remove -1 16 2024-02-19T21:13:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.762+0000 7faea4fc0700 20 client.50144.objecter get_session s=0x7faea012f780 osd=7 3 2024-02-19T21:13:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.762+0000 7faea4fc0700 15 client.50144.objecter _session_command_op_assign 7 16 2024-02-19T21:13:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.762+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea012f780 osd=7 4 2024-02-19T21:13:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.762+0000 7faea4fc0700 10 client.50144.objecter _send_command 16 2024-02-19T21:13:05.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.762+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7faea012fba0 con 0x7faea01334b0 2024-02-19T21:13:05.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.764+0000 7fae8f7fe700 1 -- 172.21.15.163:0/3780174867 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7faea012fba0 con 0x7faea01334b0 2024-02-19T21:13:05.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.764+0000 7fae8f7fe700 10 client.50144.objecter ms_dispatch 0x7faea0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:05.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.764+0000 7fae8f7fe700 10 client.50144.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.764+0000 7fae8f7fe700 20 client.50144.objecter put_session s=0x7faea012f780 osd=7 4 2024-02-19T21:13:05.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.764+0000 7fae8f7fe700 15 client.50144.objecter _session_command_op_remove 7 16 2024-02-19T21:13:05.767 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:13:05.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.764+0000 7faea4fc0700 10 librados: watch_flush enter 2024-02-19T21:13:05.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.764+0000 7faea4fc0700 10 librados: watch_flush exit 2024-02-19T21:13:05.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 10 client.50144.objecter close_session for osd.0 2024-02-19T21:13:05.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 >> v1:172.21.15.163:6801/1559418115 conn(0x7faea0118600 legacy=0x7faea01189e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:05.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea0047540 osd=0 1 2024-02-19T21:13:05.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 10 client.50144.objecter close_session for osd.1 2024-02-19T21:13:05.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 >> v1:172.21.15.163:6805/3125047432 conn(0x7faea00f3fd0 legacy=0x7faea00f43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:05.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea00f3950 osd=1 1 2024-02-19T21:13:05.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 10 client.50144.objecter close_session for osd.2 2024-02-19T21:13:05.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 >> v1:172.21.15.163:6809/1563158487 conn(0x7faea0118010 legacy=0x7faea011fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:05.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea0117ea0 osd=2 1 2024-02-19T21:13:05.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 10 client.50144.objecter close_session for osd.3 2024-02-19T21:13:05.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 >> v1:172.21.15.163:6813/2112279769 conn(0x7faea00594d0 legacy=0x7faea0122670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:05.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea0059360 osd=3 1 2024-02-19T21:13:05.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 10 client.50144.objecter close_session for osd.4 2024-02-19T21:13:05.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 >> v1:172.21.15.193:6800/3129716965 conn(0x7faea0123590 legacy=0x7faea0128cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:05.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea011d8a0 osd=4 1 2024-02-19T21:13:05.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 10 client.50144.objecter close_session for osd.5 2024-02-19T21:13:05.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 >> v1:172.21.15.193:6804/805161991 conn(0x7faea012a900 legacy=0x7faea012ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:05.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea011dae0 osd=5 1 2024-02-19T21:13:05.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 10 client.50144.objecter close_session for osd.6 2024-02-19T21:13:05.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 >> v1:172.21.15.193:6808/787604615 conn(0x7faea0130260 legacy=0x7faea01306c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:05.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea012e5a0 osd=6 1 2024-02-19T21:13:05.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 10 client.50144.objecter close_session for osd.7 2024-02-19T21:13:05.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 >> v1:172.21.15.193:6812/157927432 conn(0x7faea01334b0 legacy=0x7faea01358a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:05.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 20 client.50144.objecter put_session s=0x7faea012f780 osd=7 1 2024-02-19T21:13:05.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 20 client.50144.objecter shutdown clearing up homeless session... 2024-02-19T21:13:05.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 10 client.50144.objecter successfully canceled tick 2024-02-19T21:13:05.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7fae9ed9d700 1 -- 172.21.15.163:0/3780174867 reap_dead start 2024-02-19T21:13:05.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7fae9ed9d700 1 -- 172.21.15.163:0/3780174867 reap_dead start 2024-02-19T21:13:05.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7fae9ed9d700 1 -- 172.21.15.163:0/3780174867 reap_dead start 2024-02-19T21:13:05.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7fae9ed9d700 1 -- 172.21.15.163:0/3780174867 reap_dead start 2024-02-19T21:13:05.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7fae9ed9d700 1 -- 172.21.15.163:0/3780174867 reap_dead start 2024-02-19T21:13:05.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 >> v1:172.21.15.163:6800/1552514681 conn(0x7fae88068300 legacy=0x7fae8806a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:05.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 >> v1:172.21.15.163:6790/0 conn(0x7faea01042b0 legacy=0x7faea0115950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:05.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 shutdown_connections 2024-02-19T21:13:05.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 >> 172.21.15.163:0/3780174867 conn(0x7faea00f3570 msgr2=0x7faea0107700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:05.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 shutdown_connections 2024-02-19T21:13:05.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 1 -- 172.21.15.163:0/3780174867 wait complete. 2024-02-19T21:13:05.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:05.765+0000 7faea4fc0700 1 librados: shutdown 2024-02-19T21:13:05.771 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.771 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.771 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.771 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.771 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.771 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.771 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.771 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:13:05.772 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:06.221 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:06.224 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:06.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:06.249 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:13:06.249 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:13:06.326 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:13:06.361 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:06.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:04.424983+0000 osd.7 (osd.7) 247 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:13:06.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:04.428229+0000 osd.7 (osd.7) 248 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:13:06.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:05.170359+0000 osd.3 (osd.3) 237 : cluster [DBG] 61.18 scrub starts 2024-02-19T21:13:06.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:05.172167+0000 osd.3 (osd.3) 238 : cluster [DBG] 61.18 scrub ok 2024-02-19T21:13:06.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:05.192237+0000 osd.2 (osd.2) 222 : cluster [DBG] 5.a scrub starts 2024-02-19T21:13:06.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:05.193218+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.a scrub ok 2024-02-19T21:13:06.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:05.263635+0000 mgr.y (mgr.14550) 487 : cluster [DBG] pgmap v822: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:06.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:05.322043+0000 mon.a (mon.0) 2925 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-02-19T21:13:06.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:05.322092+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e549: 8 total, 8 up, 6 in 2024-02-19T21:13:06.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:05.345965+0000 mon.b (mon.1) 707 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:13:06.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:05.346294+0000 mon.a (mon.0) 2927 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:13:06.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:05.619194+0000 mon.c (mon.2) 408 : audit [DBG] from='client.? v1:172.21.15.163:0/3780174867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:06.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:04.424983+0000 osd.7 (osd.7) 247 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:13:06.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:04.428229+0000 osd.7 (osd.7) 248 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:13:06.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:05.170359+0000 osd.3 (osd.3) 237 : cluster [DBG] 61.18 scrub starts 2024-02-19T21:13:06.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:05.172167+0000 osd.3 (osd.3) 238 : cluster [DBG] 61.18 scrub ok 2024-02-19T21:13:06.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:05.192237+0000 osd.2 (osd.2) 222 : cluster [DBG] 5.a scrub starts 2024-02-19T21:13:06.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:05.193218+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.a scrub ok 2024-02-19T21:13:06.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:05.263635+0000 mgr.y (mgr.14550) 487 : cluster [DBG] pgmap v822: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:06.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:05.322043+0000 mon.a (mon.0) 2925 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-02-19T21:13:06.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:05.322092+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e549: 8 total, 8 up, 6 in 2024-02-19T21:13:06.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:05.345965+0000 mon.b (mon.1) 707 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:13:06.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:05.346294+0000 mon.a (mon.0) 2927 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:13:06.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:05.619194+0000 mon.c (mon.2) 408 : audit [DBG] from='client.? v1:172.21.15.163:0/3780174867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:06.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:04.424983+0000 osd.7 (osd.7) 247 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:13:06.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:04.428229+0000 osd.7 (osd.7) 248 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:13:06.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:05.170359+0000 osd.3 (osd.3) 237 : cluster [DBG] 61.18 scrub starts 2024-02-19T21:13:06.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:05.172167+0000 osd.3 (osd.3) 238 : cluster [DBG] 61.18 scrub ok 2024-02-19T21:13:06.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:05.192237+0000 osd.2 (osd.2) 222 : cluster [DBG] 5.a scrub starts 2024-02-19T21:13:06.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:05.193218+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.a scrub ok 2024-02-19T21:13:06.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:05.263635+0000 mgr.y (mgr.14550) 487 : cluster [DBG] pgmap v822: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:06.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:05.322043+0000 mon.a (mon.0) 2925 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-90","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-02-19T21:13:06.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:05.322092+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e549: 8 total, 8 up, 6 in 2024-02-19T21:13:06.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:05.345965+0000 mon.b (mon.1) 707 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:13:06.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:05.346294+0000 mon.a (mon.0) 2927 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:13:06.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:05.619194+0000 mon.c (mon.2) 408 : audit [DBG] from='client.? v1:172.21.15.163:0/3780174867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:06.745 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:07.011 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:13:07.013 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:07.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:13:07.115 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:13:07.143 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:07.344 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:06.340865+0000 mon.a (mon.0) 2928 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:13:07.345 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:06.340930+0000 mon.a (mon.0) 2929 : cluster [DBG] osdmap e550: 8 total, 8 up, 6 in 2024-02-19T21:13:07.345 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:06.345073+0000 mon.b (mon.1) 708 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90"}]: dispatch 2024-02-19T21:13:07.345 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:06.345968+0000 mon.a (mon.0) 2930 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90"}]: dispatch 2024-02-19T21:13:07.345 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:06.391686+0000 osd.5 (osd.5) 224 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:13:07.345 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:06.395644+0000 osd.5 (osd.5) 225 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:13:07.345 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:07.220678+0000 osd.3 (osd.3) 239 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:13:07.345 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:07.221591+0000 osd.3 (osd.3) 240 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:13:07.346 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:06.340865+0000 mon.a (mon.0) 2928 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:13:07.346 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:06.340930+0000 mon.a (mon.0) 2929 : cluster [DBG] osdmap e550: 8 total, 8 up, 6 in 2024-02-19T21:13:07.346 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:06.345073+0000 mon.b (mon.1) 708 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90"}]: dispatch 2024-02-19T21:13:07.346 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:06.345968+0000 mon.a (mon.0) 2930 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90"}]: dispatch 2024-02-19T21:13:07.346 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:06.391686+0000 osd.5 (osd.5) 224 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:13:07.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:06.395644+0000 osd.5 (osd.5) 225 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:13:07.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:07.220678+0000 osd.3 (osd.3) 239 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:13:07.347 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:07.221591+0000 osd.3 (osd.3) 240 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:13:07.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:06.340865+0000 mon.a (mon.0) 2928 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]': finished 2024-02-19T21:13:07.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:06.340930+0000 mon.a (mon.0) 2929 : cluster [DBG] osdmap e550: 8 total, 8 up, 6 in 2024-02-19T21:13:07.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:06.345073+0000 mon.b (mon.1) 708 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90"}]: dispatch 2024-02-19T21:13:07.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:06.345968+0000 mon.a (mon.0) 2930 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90"}]: dispatch 2024-02-19T21:13:07.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:06.391686+0000 osd.5 (osd.5) 224 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:13:07.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:06.395644+0000 osd.5 (osd.5) 225 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:13:07.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:07.220678+0000 osd.3 (osd.3) 239 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:13:07.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:07.221591+0000 osd.3 (osd.3) 240 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:13:08.070 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:13:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:13:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:13:08.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:06.443541+0000 osd.7 (osd.7) 249 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:13:08.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:06.446519+0000 osd.7 (osd.7) 250 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:13:08.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:06.954770+0000 osd.1 (osd.1) 255 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:13:08.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:06.958510+0000 osd.1 (osd.1) 256 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:13:08.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:07.264011+0000 mgr.y (mgr.14550) 488 : cluster [DBG] pgmap v825: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:08.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:07.347080+0000 mon.a (mon.0) 2931 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90"}]': finished 2024-02-19T21:13:08.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:07.347147+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e551: 8 total, 8 up, 6 in 2024-02-19T21:13:08.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:07.506107+0000 osd.4 (osd.4) 201 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:13:08.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:07.506940+0000 osd.4 (osd.4) 202 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:13:08.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:07.743418+0000 osd.3 (osd.3) 241 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:08.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:07.743576+0000 osd.3 (osd.3) 242 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:08.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:06.443541+0000 osd.7 (osd.7) 249 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:13:08.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:06.446519+0000 osd.7 (osd.7) 250 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:13:08.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:06.954770+0000 osd.1 (osd.1) 255 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:13:08.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:06.958510+0000 osd.1 (osd.1) 256 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:13:08.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:07.264011+0000 mgr.y (mgr.14550) 488 : cluster [DBG] pgmap v825: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:08.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:07.347080+0000 mon.a (mon.0) 2931 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90"}]': finished 2024-02-19T21:13:08.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:07.347147+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e551: 8 total, 8 up, 6 in 2024-02-19T21:13:08.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:07.506107+0000 osd.4 (osd.4) 201 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:13:08.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:07.506940+0000 osd.4 (osd.4) 202 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:13:08.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:07.743418+0000 osd.3 (osd.3) 241 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:08.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:07.743576+0000 osd.3 (osd.3) 242 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:08.793 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:13:08.795 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:08.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:13:08.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:06.443541+0000 osd.7 (osd.7) 249 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:13:08.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:06.446519+0000 osd.7 (osd.7) 250 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:13:08.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:06.954770+0000 osd.1 (osd.1) 255 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:13:08.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:06.958510+0000 osd.1 (osd.1) 256 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:13:08.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:07.264011+0000 mgr.y (mgr.14550) 488 : cluster [DBG] pgmap v825: 238 pgs: 238 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:08.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:07.347080+0000 mon.a (mon.0) 2931 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-90"}]': finished 2024-02-19T21:13:08.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:07.347147+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e551: 8 total, 8 up, 6 in 2024-02-19T21:13:08.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:07.506107+0000 osd.4 (osd.4) 201 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:13:08.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:07.506940+0000 osd.4 (osd.4) 202 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:13:08.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:07.743418+0000 osd.3 (osd.3) 241 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:08.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:07.743576+0000 osd.3 (osd.3) 242 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:08.897 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:13:08.928 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:08.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-19T21:13:09.027 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:13:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:09.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:13:09.131 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:09.164 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:09.184 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:09.187 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:09.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:09.288 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:09.317 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:09.339 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:09.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:09.442 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:13:09.471 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:09.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:07.446631+0000 osd.7 (osd.7) 251 : cluster [DBG] 2.b scrub starts 2024-02-19T21:13:09.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:07.450119+0000 osd.7 (osd.7) 252 : cluster [DBG] 2.b scrub ok 2024-02-19T21:13:09.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:08.349119+0000 mon.a (mon.0) 2933 : cluster [DBG] osdmap e552: 8 total, 8 up, 6 in 2024-02-19T21:13:09.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:09.106327+0000 mon.a (mon.0) 2934 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:07.446631+0000 osd.7 (osd.7) 251 : cluster [DBG] 2.b scrub starts 2024-02-19T21:13:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:07.450119+0000 osd.7 (osd.7) 252 : cluster [DBG] 2.b scrub ok 2024-02-19T21:13:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:08.349119+0000 mon.a (mon.0) 2933 : cluster [DBG] osdmap e552: 8 total, 8 up, 6 in 2024-02-19T21:13:09.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:09.106327+0000 mon.a (mon.0) 2934 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:09.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:07.446631+0000 osd.7 (osd.7) 251 : cluster [DBG] 2.b scrub starts 2024-02-19T21:13:09.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:07.450119+0000 osd.7 (osd.7) 252 : cluster [DBG] 2.b scrub ok 2024-02-19T21:13:09.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:08.349119+0000 mon.a (mon.0) 2933 : cluster [DBG] osdmap e552: 8 total, 8 up, 6 in 2024-02-19T21:13:09.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:09.106327+0000 mon.a (mon.0) 2934 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:09.827 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:13:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:09.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:13:09.930 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:09.960 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:09.981 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:09.983 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:09.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:10.085 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:10.114 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:10.134 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:10.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:10.238 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:10.267 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:10.287 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:10.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:10.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:10.391 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:10.419 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:10.439 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:10.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:10.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:10.542 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:10.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:08.486899+0000 osd.7 (osd.7) 253 : cluster [DBG] 61.1e scrub starts 2024-02-19T21:13:10.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:08.494748+0000 osd.7 (osd.7) 254 : cluster [DBG] 61.1e scrub ok 2024-02-19T21:13:10.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:08.980011+0000 osd.1 (osd.1) 257 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:13:10.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:08.986354+0000 osd.1 (osd.1) 258 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:13:10.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:09.181076+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:13:10.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:09.183958+0000 osd.2 (osd.2) 225 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:13:10.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:09.264618+0000 mgr.y (mgr.14550) 489 : cluster [DBG] pgmap v828: 206 pgs: 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:10.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:09.482605+0000 mon.a (mon.0) 2935 : cluster [DBG] osdmap e553: 8 total, 8 up, 6 in 2024-02-19T21:13:10.559 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:09.491253+0000 mon.b (mon.1) 709 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:10.560 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:09.492105+0000 mon.a (mon.0) 2936 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:10.570 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:10.586 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:08.486899+0000 osd.7 (osd.7) 253 : cluster [DBG] 61.1e scrub starts 2024-02-19T21:13:10.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:08.494748+0000 osd.7 (osd.7) 254 : cluster [DBG] 61.1e scrub ok 2024-02-19T21:13:10.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:08.980011+0000 osd.1 (osd.1) 257 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:13:10.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:08.986354+0000 osd.1 (osd.1) 258 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:13:10.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:09.181076+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:13:10.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:09.183958+0000 osd.2 (osd.2) 225 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:13:10.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:09.264618+0000 mgr.y (mgr.14550) 489 : cluster [DBG] pgmap v828: 206 pgs: 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:10.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:09.482605+0000 mon.a (mon.0) 2935 : cluster [DBG] osdmap e553: 8 total, 8 up, 6 in 2024-02-19T21:13:10.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:09.491253+0000 mon.b (mon.1) 709 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:10.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:09.492105+0000 mon.a (mon.0) 2936 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:10.588 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:08.486899+0000 osd.7 (osd.7) 253 : cluster [DBG] 61.1e scrub starts 2024-02-19T21:13:10.588 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:08.494748+0000 osd.7 (osd.7) 254 : cluster [DBG] 61.1e scrub ok 2024-02-19T21:13:10.588 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:08.980011+0000 osd.1 (osd.1) 257 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:13:10.588 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:08.986354+0000 osd.1 (osd.1) 258 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:13:10.588 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:09.181076+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:13:10.588 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:09.183958+0000 osd.2 (osd.2) 225 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:13:10.588 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:09.264618+0000 mgr.y (mgr.14550) 489 : cluster [DBG] pgmap v828: 206 pgs: 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:10.588 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:09.482605+0000 mon.a (mon.0) 2935 : cluster [DBG] osdmap e553: 8 total, 8 up, 6 in 2024-02-19T21:13:10.588 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:09.491253+0000 mon.b (mon.1) 709 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:10.588 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:09.492105+0000 mon.a (mon.0) 2936 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:10.592 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:10.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:10.697 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:10.724 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:10.743 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:10.745 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:10.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:10.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:13:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:13:10.847 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:13:10.876 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:10.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.959+0000 7f247b413700 1 -- 172.21.15.163:0/3205129041 >> v1:172.21.15.163:6789/0 conn(0x7f24740fd980 legacy=0x7f24740fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.959+0000 7f247b413700 1 -- 172.21.15.163:0/3205129041 shutdown_connections 2024-02-19T21:13:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.959+0000 7f247b413700 1 -- 172.21.15.163:0/3205129041 >> 172.21.15.163:0/3205129041 conn(0x7f24740f35a0 msgr2=0x7f24740f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.959+0000 7f247b413700 1 -- 172.21.15.163:0/3205129041 shutdown_connections 2024-02-19T21:13:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.959+0000 7f247b413700 1 -- 172.21.15.163:0/3205129041 wait complete. 2024-02-19T21:13:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f247b413700 1 librados: starting msgr at 2024-02-19T21:13:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f247b413700 1 librados: starting objecter 2024-02-19T21:13:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f247b413700 1 Processor -- start 2024-02-19T21:13:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f247b413700 1 -- start start 2024-02-19T21:13:10.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f247b413700 1 librados: setting wanted keys 2024-02-19T21:13:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f247b413700 1 librados: calling monclient init 2024-02-19T21:13:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f247b413700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f247406cc90 con 0x7f2474100780 2024-02-19T21:13:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f247b413700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f247406ce20 con 0x7f24740fd980 2024-02-19T21:13:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f247b413700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f247411bb80 con 0x7f24741042c0 2024-02-19T21:13:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f24789ae700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f2474100780 0x7f2474114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33474/0 (socket says 172.21.15.163:33474) 2024-02-19T21:13:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f24799b0700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f24741042c0 0x7f2474118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:48492/0 (socket says 172.21.15.163:48492) 2024-02-19T21:13:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f24789ae700 1 -- 172.21.15.163:0/2435800927 learned_addr learned my addr 172.21.15.163:0/2435800927 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f246a7fc700 10 client.?.objecter ms_handle_connect 0x7f24741042c0 2024-02-19T21:13:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f246a7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f246a7fc700 10 client.?.objecter ms_handle_connect 0x7f2474100780 2024-02-19T21:13:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f246a7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:10.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f246a7fc700 10 client.?.objecter ms_handle_connect 0x7f24740fd980 2024-02-19T21:13:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f246a7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247148430 0 0) 0x7f247411bb80 con 0x7f24741042c0 2024-02-19T21:13:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2460003030 con 0x7f24741042c0 2024-02-19T21:13:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401623713 0 0) 0x7f247406cc90 con 0x7f2474100780 2024-02-19T21:13:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f247411bb80 con 0x7f2474100780 2024-02-19T21:13:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993726642 0 0) 0x7f247406ce20 con 0x7f24740fd980 2024-02-19T21:13:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.960+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f247406cc90 con 0x7f24740fd980 2024-02-19T21:13:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2857381337 0 0) 0x7f2460003030 con 0x7f24741042c0 2024-02-19T21:13:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f247406ce20 con 0x7f24741042c0 2024-02-19T21:13:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f246c001e00 con 0x7f24741042c0 2024-02-19T21:13:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 109920815 0 0) 0x7f247406cc90 con 0x7f24740fd980 2024-02-19T21:13:10.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2460003030 con 0x7f24740fd980 2024-02-19T21:13:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 603869775 0 0) 0x7f247411bb80 con 0x7f2474100780 2024-02-19T21:13:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f247406cc90 con 0x7f2474100780 2024-02-19T21:13:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2464001430 con 0x7f24740fd980 2024-02-19T21:13:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f24700017c0 con 0x7f2474100780 2024-02-19T21:13:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2138583094 0 0) 0x7f247406ce20 con 0x7f24741042c0 2024-02-19T21:13:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 >> v1:172.21.15.193:6789/0 conn(0x7f24740fd980 legacy=0x7f24740fc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 >> v1:172.21.15.163:6789/0 conn(0x7f2474100780 legacy=0x7f2474114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247411cd20 con 0x7f24741042c0 2024-02-19T21:13:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f247b413700 1 -- 172.21.15.163:0/2435800927 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247411bd70 con 0x7f24741042c0 2024-02-19T21:13:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f247b413700 10 client.50150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f247b413700 1 -- 172.21.15.163:0/2435800927 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f247411c1e0 con 0x7f24741042c0 2024-02-19T21:13:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f247b413700 1 librados: init done 2024-02-19T21:13:10.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f246c002840 con 0x7f24741042c0 2024-02-19T21:13:10.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.961+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f246c005e80 con 0x7f24741042c0 2024-02-19T21:13:10.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.962+0000 7f247b413700 1 -- 172.21.15.163:0/2435800927 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2458004f20 con 0x7f24741042c0 2024-02-19T21:13:10.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.962+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f246c018d10 con 0x7f24741042c0 2024-02-19T21:13:10.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.963+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(554..554 src has 434..554) v4 ==== 8092+0+0 (unknown 3585501312 0 0) 0x7f246c078f00 con 0x7f24741042c0 2024-02-19T21:13:10.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.963+0000 7f246a7fc700 10 client.50150.objecter ms_dispatch 0x7f2474001550 osd_map(554..554 src has 434..554) v4 2024-02-19T21:13:10.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.963+0000 7f246a7fc700 3 client.50150.objecter handle_osd_map got epochs [554,554] > 0 2024-02-19T21:13:10.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.963+0000 7f246a7fc700 3 client.50150.objecter handle_osd_map decoding full epoch 554 2024-02-19T21:13:10.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.963+0000 7f246a7fc700 20 client.50150.objecter dump_active .. 0 homeless 2024-02-19T21:13:10.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.963+0000 7f246a7fc700 10 client.50150.objecter ms_handle_connect 0x7f2460063e60 2024-02-19T21:13:10.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:10.967+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f246c04c420 con 0x7f24741042c0 2024-02-19T21:13:10.974 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:13:10.977 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:10.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:13:11.079 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:11.106 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:11.125 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:11.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:11.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:11.131 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:11.132+0000 7f247b413700 1 -- 172.21.15.163:0/2435800927 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2458000b90 con 0x7f2460063e60 2024-02-19T21:13:11.147 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:11.148+0000 7f246a7fc700 1 -- 172.21.15.163:0/2435800927 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+592674 (unknown 2965378022 0 156075173) 0x7f2458000b90 con 0x7f2460063e60 2024-02-19T21:13:11.156 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:13:11.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:11.151+0000 7f247b413700 10 librados: watch_flush enter 2024-02-19T21:13:11.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:11.151+0000 7f247b413700 10 librados: watch_flush exit 2024-02-19T21:13:11.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:11.151+0000 7f247b413700 20 client.50150.objecter shutdown clearing up homeless session... 2024-02-19T21:13:11.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:11.151+0000 7f247b413700 10 client.50150.objecter successfully canceled tick 2024-02-19T21:13:11.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:11.152+0000 7f247b413700 1 -- 172.21.15.163:0/2435800927 >> v1:172.21.15.163:6800/1552514681 conn(0x7f2460063e60 legacy=0x7f2460066320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:11.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:11.152+0000 7f247b413700 1 -- 172.21.15.163:0/2435800927 >> v1:172.21.15.163:6790/0 conn(0x7f24741042c0 legacy=0x7f2474118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:11.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:11.152+0000 7f247b413700 1 -- 172.21.15.163:0/2435800927 shutdown_connections 2024-02-19T21:13:11.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:11.152+0000 7f247b413700 1 -- 172.21.15.163:0/2435800927 >> 172.21.15.163:0/2435800927 conn(0x7f24740f35a0 msgr2=0x7f2474107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:11.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:11.152+0000 7f247b413700 1 -- 172.21.15.163:0/2435800927 shutdown_connections 2024-02-19T21:13:11.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:11.152+0000 7f247b413700 1 -- 172.21.15.163:0/2435800927 wait complete. 2024-02-19T21:13:11.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:11.152+0000 7f247b413700 1 librados: shutdown 2024-02-19T21:13:11.163 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:13:11.229 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:11.256 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:11.275 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:11.277 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:11.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:11.378 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:11.404 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:11.422 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:11.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:11.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:11.526 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:11.552 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:11.570 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:11.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:11.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:11.674 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:11.698 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:10.145343+0000 osd.2 (osd.2) 226 : cluster [DBG] 16.e scrub starts 2024-02-19T21:13:11.698 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:10.146099+0000 osd.2 (osd.2) 227 : cluster [DBG] 16.e scrub ok 2024-02-19T21:13:11.699 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:10.480943+0000 mon.a (mon.0) 2937 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:13:11.699 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:10.481035+0000 mon.a (mon.0) 2938 : cluster [DBG] osdmap e554: 8 total, 8 up, 6 in 2024-02-19T21:13:11.699 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:10.492979+0000 mon.b (mon.1) 710 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:13:11.699 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:10.496266+0000 mon.b (mon.1) 711 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:13:11.699 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:10.497822+0000 mon.a (mon.0) 2939 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:13:11.699 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:10.560236+0000 mgr.y (mgr.14550) 490 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:11.700 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:11.719 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:11.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:11.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:10.145343+0000 osd.2 (osd.2) 226 : cluster [DBG] 16.e scrub starts 2024-02-19T21:13:11.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:10.146099+0000 osd.2 (osd.2) 227 : cluster [DBG] 16.e scrub ok 2024-02-19T21:13:11.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:10.480943+0000 mon.a (mon.0) 2937 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:13:11.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:10.481035+0000 mon.a (mon.0) 2938 : cluster [DBG] osdmap e554: 8 total, 8 up, 6 in 2024-02-19T21:13:11.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:10.492979+0000 mon.b (mon.1) 710 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:13:11.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:10.496266+0000 mon.b (mon.1) 711 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:13:11.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:10.497822+0000 mon.a (mon.0) 2939 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:13:11.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:10.560236+0000 mgr.y (mgr.14550) 490 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:11.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:10.145343+0000 osd.2 (osd.2) 226 : cluster [DBG] 16.e scrub starts 2024-02-19T21:13:11.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:10.146099+0000 osd.2 (osd.2) 227 : cluster [DBG] 16.e scrub ok 2024-02-19T21:13:11.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:10.480943+0000 mon.a (mon.0) 2937 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:13:11.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:10.481035+0000 mon.a (mon.0) 2938 : cluster [DBG] osdmap e554: 8 total, 8 up, 6 in 2024-02-19T21:13:11.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:10.492979+0000 mon.b (mon.1) 710 : audit [DBG] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-02-19T21:13:11.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:10.496266+0000 mon.b (mon.1) 711 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:13:11.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:10.497822+0000 mon.a (mon.0) 2939 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-02-19T21:13:11.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:10.560236+0000 mgr.y (mgr.14550) 490 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:11.823 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:11.828 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:13:11.849 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:11.868 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:11.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:11.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:11.971 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:11.998 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:12.016 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:12.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:12.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:12.120 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:12.149 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:12.590 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":830,"stamp":"2024-02-19T21:13:10.477532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4026236,"num_objects":386,"num_object_clones":11,"num_object_copies":1158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":386,"num_whiteouts":0,"num_read":2040,"num_read_kb":1636,"num_write":1499,"num_write_kb":4666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1525,"num_bytes_recovered":17215,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":21,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2898,"ondisk_log_size":2898,"up":714,"acting":714,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":680,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":1911580,"kb_used_data":61752,"kb_used_omap":0,"kb_used_meta":1848832,"kb_avail":560559332,"statfs":{"total":575970213888,"available":574012755968,"internally_reserved":0,"allocated":63234048,"data_stored":41820575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1893203968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":5,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.138352"},"pg_stats":[{"pgid":"228.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.488798+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"228.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.477336+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"228.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.491555+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"228.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.487043+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"228.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.487630+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"228.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.481968+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"228.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.490902+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"228.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.481820+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"228.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.479680+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"228.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.482525+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"228.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.481996+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"228.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.487304+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"228.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.484988+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"228.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.486669+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"228.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.482256+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"228.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.477292+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"228.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.477339+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"228.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.488080+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"228.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.488272+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"228.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.477257+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"228.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.491395+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"228.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.485765+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"228.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.479780+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"228.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.479530+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"228.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.484990+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"228.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.482278+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"228.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.485162+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"228.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.481930+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"228.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.479834+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"228.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.485468+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,7],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"228.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.490548+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"228.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"creating+peering","last_fresh":"2024-02-19T21:13:09.390406+0000","last_change":"2024-02-19T21:13:09.486222+0000","last_active":"2024-02-19T21:13:09.390406+0000","last_peered":"2024-02-19T21:13:09.390406+0000","last_clean":"2024-02-19T21:13:09.390406+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:09.390406+0000","last_undegraded":"2024-02-19T21:13:09.390406+0000","last_fullsized":"2024-02-19T21:13:09.390406+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":553,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:09.390406+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.390406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.2","version":"452'102","reported_seq":509,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350001+0000","last_change":"2024-02-19T21:12:57.216407+0000","last_active":"2024-02-19T21:13:08.350001+0000","last_peered":"2024-02-19T21:13:08.350001+0000","last_clean":"2024-02-19T21:13:08.350001+0000","last_became_active":"2024-02-19T21:12:13.330097+0000","last_became_peered":"2024-02-19T21:12:13.330097+0000","last_unstale":"2024-02-19T21:13:08.350001+0000","last_undegraded":"2024-02-19T21:13:08.350001+0000","last_fullsized":"2024-02-19T21:13:08.350001+0000","mapping_epoch":496,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'102","last_scrub_stamp":"2024-02-19T21:12:57.216339+0000","last_deep_scrub":"355'85","last_deep_scrub_stamp":"2024-02-19T21:08:55.882988+0000","last_clean_scrub_stamp":"2024-02-19T21:12:57.216339+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":33,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.3","version":"338'71","reported_seq":574,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.982346+0000","last_change":"2024-02-19T21:11:06.901648+0000","last_active":"2024-02-19T21:13:08.982346+0000","last_peered":"2024-02-19T21:13:08.982346+0000","last_clean":"2024-02-19T21:13:08.982346+0000","last_became_active":"2024-02-19T21:04:54.641884+0000","last_became_peered":"2024-02-19T21:04:54.641884+0000","last_unstale":"2024-02-19T21:13:08.982346+0000","last_undegraded":"2024-02-19T21:13:08.982346+0000","last_fullsized":"2024-02-19T21:13:08.982346+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'71","last_scrub_stamp":"2024-02-19T21:11:06.901567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:06.901567+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000068,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":23,"num_write_kb":3929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.0","version":"257'35","reported_seq":554,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.981639+0000","last_change":"2024-02-19T21:12:00.304083+0000","last_active":"2024-02-19T21:13:08.981639+0000","last_peered":"2024-02-19T21:13:08.981639+0000","last_clean":"2024-02-19T21:13:08.981639+0000","last_became_active":"2024-02-19T21:11:59.319551+0000","last_became_peered":"2024-02-19T21:11:59.319551+0000","last_unstale":"2024-02-19T21:13:08.981639+0000","last_undegraded":"2024-02-19T21:13:08.981639+0000","last_fullsized":"2024-02-19T21:13:08.981639+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'35","last_scrub_stamp":"2024-02-19T21:11:31.931145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.931145+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":30,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1","version":"338'69","reported_seq":679,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.981499+0000","last_change":"2024-02-19T21:10:54.672773+0000","last_active":"2024-02-19T21:13:08.981499+0000","last_peered":"2024-02-19T21:13:08.981499+0000","last_clean":"2024-02-19T21:13:08.981499+0000","last_became_active":"2024-02-19T21:10:41.311298+0000","last_became_peered":"2024-02-19T21:10:41.311298+0000","last_unstale":"2024-02-19T21:13:08.981499+0000","last_undegraded":"2024-02-19T21:13:08.981499+0000","last_fullsized":"2024-02-19T21:13:08.981499+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-19T21:10:54.672705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.672705+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.6","version":"202'34","reported_seq":542,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.492218+0000","last_change":"2024-02-19T21:12:07.285304+0000","last_active":"2024-02-19T21:13:08.492218+0000","last_peered":"2024-02-19T21:13:08.492218+0000","last_clean":"2024-02-19T21:13:08.492218+0000","last_became_active":"2024-02-19T21:04:54.658542+0000","last_became_peered":"2024-02-19T21:04:54.658542+0000","last_unstale":"2024-02-19T21:13:08.492218+0000","last_undegraded":"2024-02-19T21:13:08.492218+0000","last_fullsized":"2024-02-19T21:13:08.492218+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'34","last_scrub_stamp":"2024-02-19T21:12:07.285231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:12:07.285231+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.7","version":"512'21","reported_seq":561,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.988568+0000","last_change":"2024-02-19T21:11:59.982829+0000","last_active":"2024-02-19T21:13:08.988568+0000","last_peered":"2024-02-19T21:13:08.988568+0000","last_clean":"2024-02-19T21:13:08.988568+0000","last_became_active":"2024-02-19T21:11:59.333176+0000","last_became_peered":"2024-02-19T21:11:59.333176+0000","last_unstale":"2024-02-19T21:13:08.988568+0000","last_undegraded":"2024-02-19T21:13:08.988568+0000","last_fullsized":"2024-02-19T21:13:08.988568+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"396'20","last_scrub_stamp":"2024-02-19T21:11:05.030354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.030354+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1240,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":25,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.4","version":"483'73","reported_seq":682,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.490630+0000","last_change":"2024-02-19T21:11:01.162809+0000","last_active":"2024-02-19T21:13:08.490630+0000","last_peered":"2024-02-19T21:13:08.490630+0000","last_clean":"2024-02-19T21:13:08.490630+0000","last_became_active":"2024-02-19T21:10:41.314275+0000","last_became_peered":"2024-02-19T21:10:41.314275+0000","last_unstale":"2024-02-19T21:13:08.490630+0000","last_undegraded":"2024-02-19T21:13:08.490630+0000","last_fullsized":"2024-02-19T21:13:08.490630+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'70","last_scrub_stamp":"2024-02-19T21:11:01.162737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.162737+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":24,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":96,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.5","version":"503'122","reported_seq":706,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.982736+0000","last_change":"2024-02-19T21:12:12.321378+0000","last_active":"2024-02-19T21:13:08.982736+0000","last_peered":"2024-02-19T21:13:08.982736+0000","last_clean":"2024-02-19T21:13:08.982736+0000","last_became_active":"2024-02-19T21:12:12.320664+0000","last_became_peered":"2024-02-19T21:12:12.320664+0000","last_unstale":"2024-02-19T21:13:08.982736+0000","last_undegraded":"2024-02-19T21:13:08.982736+0000","last_fullsized":"2024-02-19T21:13:08.982736+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'121","last_scrub_stamp":"2024-02-19T21:12:07.998855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:12:07.998855+0000","log_size":122,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1308,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":39,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":93,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.a","version":"236'22","reported_seq":356,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350546+0000","last_change":"2024-02-19T21:11:34.523734+0000","last_active":"2024-02-19T21:13:08.350546+0000","last_peered":"2024-02-19T21:13:08.350546+0000","last_clean":"2024-02-19T21:13:08.350546+0000","last_became_active":"2024-02-19T21:10:41.305485+0000","last_became_peered":"2024-02-19T21:10:41.305485+0000","last_unstale":"2024-02-19T21:13:08.350546+0000","last_undegraded":"2024-02-19T21:13:08.350546+0000","last_fullsized":"2024-02-19T21:13:08.350546+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-02-19T21:11:34.523666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:34.523666+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":96,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.b","version":"278'95","reported_seq":662,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.982232+0000","last_change":"2024-02-19T21:12:13.598705+0000","last_active":"2024-02-19T21:13:08.982232+0000","last_peered":"2024-02-19T21:13:08.982232+0000","last_clean":"2024-02-19T21:13:08.982232+0000","last_became_active":"2024-02-19T21:12:09.370116+0000","last_became_peered":"2024-02-19T21:12:09.370116+0000","last_unstale":"2024-02-19T21:13:08.982232+0000","last_undegraded":"2024-02-19T21:13:08.982232+0000","last_fullsized":"2024-02-19T21:13:08.982232+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":497,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'95","last_scrub_stamp":"2024-02-19T21:11:03.821869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.821869+0000","log_size":95,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":30,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.8","version":"494'41","reported_seq":647,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.984634+0000","last_change":"2024-02-19T21:12:00.440463+0000","last_active":"2024-02-19T21:13:08.984634+0000","last_peered":"2024-02-19T21:13:08.984634+0000","last_clean":"2024-02-19T21:13:08.984634+0000","last_became_active":"2024-02-19T21:11:59.320588+0000","last_became_peered":"2024-02-19T21:11:59.320588+0000","last_unstale":"2024-02-19T21:13:08.984634+0000","last_undegraded":"2024-02-19T21:13:08.984634+0000","last_fullsized":"2024-02-19T21:13:08.984634+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'39","last_scrub_stamp":"2024-02-19T21:11:37.967767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:37.967767+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":18,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":83,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.9","version":"512'91","reported_seq":661,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.981402+0000","last_change":"2024-02-19T21:12:02.072794+0000","last_active":"2024-02-19T21:13:08.981402+0000","last_peered":"2024-02-19T21:13:08.981402+0000","last_clean":"2024-02-19T21:13:08.981402+0000","last_became_active":"2024-02-19T21:11:59.331693+0000","last_became_peered":"2024-02-19T21:11:59.331693+0000","last_unstale":"2024-02-19T21:13:08.981402+0000","last_undegraded":"2024-02-19T21:13:08.981402+0000","last_fullsized":"2024-02-19T21:13:08.981402+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'77","last_scrub_stamp":"2024-02-19T21:10:55.678451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:55.678451+0000","log_size":91,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1348,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":22,"num_read_kb":8,"num_write":45,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.e","version":"393'40","reported_seq":568,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351284+0000","last_change":"2024-02-19T21:12:10.565046+0000","last_active":"2024-02-19T21:13:08.351284+0000","last_peered":"2024-02-19T21:13:08.351284+0000","last_clean":"2024-02-19T21:13:08.351284+0000","last_became_active":"2024-02-19T21:12:09.369670+0000","last_became_peered":"2024-02-19T21:12:09.369670+0000","last_unstale":"2024-02-19T21:13:08.351284+0000","last_undegraded":"2024-02-19T21:13:08.351284+0000","last_fullsized":"2024-02-19T21:13:08.351284+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'40","last_scrub_stamp":"2024-02-19T21:11:40.738806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:40.738806+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":81,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"61.f","version":"543'44","reported_seq":623,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.492547+0000","last_change":"2024-02-19T21:12:59.471947+0000","last_active":"2024-02-19T21:13:08.492547+0000","last_peered":"2024-02-19T21:13:08.492547+0000","last_clean":"2024-02-19T21:13:08.492547+0000","last_became_active":"2024-02-19T21:10:41.315097+0000","last_became_peered":"2024-02-19T21:10:41.315097+0000","last_unstale":"2024-02-19T21:13:08.492547+0000","last_undegraded":"2024-02-19T21:13:08.492547+0000","last_fullsized":"2024-02-19T21:13:08.492547+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"543'44","last_scrub_stamp":"2024-02-19T21:12:59.471882+0000","last_deep_scrub":"543'44","last_deep_scrub_stamp":"2024-02-19T21:12:59.471882+0000","last_clean_scrub_stamp":"2024-02-19T21:12:59.471882+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":18,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":54,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.c","version":"249'73","reported_seq":792,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.986573+0000","last_change":"2024-02-19T21:12:09.054982+0000","last_active":"2024-02-19T21:13:08.986573+0000","last_peered":"2024-02-19T21:13:08.986573+0000","last_clean":"2024-02-19T21:13:08.986573+0000","last_became_active":"2024-02-19T21:10:41.321453+0000","last_became_peered":"2024-02-19T21:10:41.321453+0000","last_unstale":"2024-02-19T21:13:08.986573+0000","last_undegraded":"2024-02-19T21:13:08.986573+0000","last_fullsized":"2024-02-19T21:13:08.986573+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":439,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'73","last_scrub_stamp":"2024-02-19T21:12:09.054917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:12:09.054917+0000","log_size":73,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.d","version":"512'49","reported_seq":562,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.986439+0000","last_change":"2024-02-19T21:11:43.928189+0000","last_active":"2024-02-19T21:13:08.986439+0000","last_peered":"2024-02-19T21:13:08.986439+0000","last_clean":"2024-02-19T21:13:08.986439+0000","last_became_active":"2024-02-19T21:04:54.644749+0000","last_became_peered":"2024-02-19T21:04:54.644749+0000","last_unstale":"2024-02-19T21:13:08.986439+0000","last_undegraded":"2024-02-19T21:13:08.986439+0000","last_fullsized":"2024-02-19T21:13:08.986439+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'44","last_scrub_stamp":"2024-02-19T21:11:43.928120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.928120+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1387,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":24,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"61.12","version":"503'63","reported_seq":696,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.491919+0000","last_change":"2024-02-19T21:12:00.009617+0000","last_active":"2024-02-19T21:13:08.491919+0000","last_peered":"2024-02-19T21:13:08.491919+0000","last_clean":"2024-02-19T21:13:08.491919+0000","last_became_active":"2024-02-19T21:11:59.341496+0000","last_became_peered":"2024-02-19T21:11:59.341496+0000","last_unstale":"2024-02-19T21:13:08.491919+0000","last_undegraded":"2024-02-19T21:13:08.491919+0000","last_fullsized":"2024-02-19T21:13:08.491919+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'52","last_scrub_stamp":"2024-02-19T21:11:48.360126+0000","last_deep_scrub":"474'52","last_deep_scrub_stamp":"2024-02-19T21:11:48.360126+0000","last_clean_scrub_stamp":"2024-02-19T21:11:48.360126+0000","log_size":63,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":556,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":15,"num_read_kb":8,"num_write":33,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":89,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.13","version":"512'75","reported_seq":752,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.489329+0000","last_change":"2024-02-19T21:12:11.371454+0000","last_active":"2024-02-19T21:13:08.489329+0000","last_peered":"2024-02-19T21:13:08.489329+0000","last_clean":"2024-02-19T21:13:08.489329+0000","last_became_active":"2024-02-19T21:12:09.334497+0000","last_became_peered":"2024-02-19T21:12:09.334497+0000","last_unstale":"2024-02-19T21:13:08.489329+0000","last_undegraded":"2024-02-19T21:13:08.489329+0000","last_fullsized":"2024-02-19T21:13:08.489329+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":495,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'71","last_scrub_stamp":"2024-02-19T21:10:57.789103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.789103+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":68,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":9,"num_read_kb":3,"num_write":29,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.10","version":"549'67","reported_seq":580,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.488727+0000","last_change":"2024-02-19T21:11:04.750002+0000","last_active":"2024-02-19T21:13:08.488727+0000","last_peered":"2024-02-19T21:13:08.488727+0000","last_clean":"2024-02-19T21:13:08.488727+0000","last_became_active":"2024-02-19T21:04:54.626393+0000","last_became_peered":"2024-02-19T21:04:54.626393+0000","last_unstale":"2024-02-19T21:13:08.488727+0000","last_undegraded":"2024-02-19T21:13:08.488727+0000","last_fullsized":"2024-02-19T21:13:08.488727+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'49","last_scrub_stamp":"2024-02-19T21:11:04.749932+0000","last_deep_scrub":"452'49","last_deep_scrub_stamp":"2024-02-19T21:11:04.749932+0000","last_clean_scrub_stamp":"2024-02-19T21:11:04.749932+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2367,"num_objects":9,"num_object_clones":2,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":28,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.11","version":"503'40","reported_seq":588,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.491240+0000","last_change":"2024-02-19T21:12:11.071765+0000","last_active":"2024-02-19T21:13:08.491240+0000","last_peered":"2024-02-19T21:13:08.491240+0000","last_clean":"2024-02-19T21:13:08.491240+0000","last_became_active":"2024-02-19T21:12:09.343017+0000","last_became_peered":"2024-02-19T21:12:09.343017+0000","last_unstale":"2024-02-19T21:13:08.491240+0000","last_undegraded":"2024-02-19T21:13:08.491240+0000","last_fullsized":"2024-02-19T21:13:08.491240+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'35","last_scrub_stamp":"2024-02-19T21:11:37.417921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:37.417921+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":17,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":556,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351192+0000","last_change":"2024-02-19T21:12:09.321398+0000","last_active":"2024-02-19T21:13:08.351192+0000","last_peered":"2024-02-19T21:13:08.351192+0000","last_clean":"2024-02-19T21:13:08.351192+0000","last_became_active":"2024-02-19T21:12:09.320924+0000","last_became_peered":"2024-02-19T21:12:09.320924+0000","last_unstale":"2024-02-19T21:13:08.351192+0000","last_undegraded":"2024-02-19T21:13:08.351192+0000","last_fullsized":"2024-02-19T21:13:08.351192+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.738600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.738600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.16","version":"494'55","reported_seq":172,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351982+0000","last_change":"2024-02-19T21:13:01.469267+0000","last_active":"2024-02-19T21:13:08.351982+0000","last_peered":"2024-02-19T21:13:08.351982+0000","last_clean":"2024-02-19T21:13:08.351982+0000","last_became_active":"2024-02-19T21:10:41.313393+0000","last_became_peered":"2024-02-19T21:10:41.313393+0000","last_unstale":"2024-02-19T21:13:08.351982+0000","last_undegraded":"2024-02-19T21:13:08.351982+0000","last_fullsized":"2024-02-19T21:13:08.351982+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'55","last_scrub_stamp":"2024-02-19T21:13:01.469183+0000","last_deep_scrub":"237'24","last_deep_scrub_stamp":"2024-02-19T21:06:57.889852+0000","last_clean_scrub_stamp":"2024-02-19T21:13:01.469183+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":28,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":40,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":86,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.348696+0000","last_change":"2024-02-19T21:11:50.304127+0000","last_active":"2024-02-19T21:13:08.348696+0000","last_peered":"2024-02-19T21:13:08.348696+0000","last_clean":"2024-02-19T21:13:08.348696+0000","last_became_active":"2024-02-19T21:11:50.303790+0000","last_became_peered":"2024-02-19T21:11:50.303790+0000","last_unstale":"2024-02-19T21:13:08.348696+0000","last_undegraded":"2024-02-19T21:13:08.348696+0000","last_fullsized":"2024-02-19T21:13:08.348696+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:42.529489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:42.529489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.17","version":"503'27","reported_seq":625,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.495635+0000","last_change":"2024-02-19T21:11:59.407600+0000","last_active":"2024-02-19T21:13:08.495635+0000","last_peered":"2024-02-19T21:13:08.495635+0000","last_clean":"2024-02-19T21:13:08.495635+0000","last_became_active":"2024-02-19T21:11:59.343086+0000","last_became_peered":"2024-02-19T21:11:59.343086+0000","last_unstale":"2024-02-19T21:13:08.495635+0000","last_undegraded":"2024-02-19T21:13:08.495635+0000","last_fullsized":"2024-02-19T21:13:08.495635+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'24","last_scrub_stamp":"2024-02-19T21:11:36.409891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.409891+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1569,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":17,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":81,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.14","version":"537'66","reported_seq":601,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.490579+0000","last_change":"2024-02-19T21:12:08.317353+0000","last_active":"2024-02-19T21:13:08.490579+0000","last_peered":"2024-02-19T21:13:08.490579+0000","last_clean":"2024-02-19T21:13:08.490579+0000","last_became_active":"2024-02-19T21:11:59.336871+0000","last_became_peered":"2024-02-19T21:11:59.336871+0000","last_unstale":"2024-02-19T21:13:08.490579+0000","last_undegraded":"2024-02-19T21:13:08.490579+0000","last_fullsized":"2024-02-19T21:13:08.490579+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'57","last_scrub_stamp":"2024-02-19T21:12:08.317274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:12:08.317274+0000","log_size":66,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1899,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":13,"num_read_kb":4,"num_write":33,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.15","version":"527'66","reported_seq":742,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.490906+0000","last_change":"2024-02-19T21:10:43.140651+0000","last_active":"2024-02-19T21:13:08.490906+0000","last_peered":"2024-02-19T21:13:08.490906+0000","last_clean":"2024-02-19T21:13:08.490906+0000","last_became_active":"2024-02-19T21:10:41.316332+0000","last_became_peered":"2024-02-19T21:10:41.316332+0000","last_unstale":"2024-02-19T21:13:08.490906+0000","last_undegraded":"2024-02-19T21:13:08.490906+0000","last_fullsized":"2024-02-19T21:13:08.490906+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'52","last_scrub_stamp":"2024-02-19T21:10:22.847438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:10:22.847438+0000","log_size":66,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":54,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.25","version":"0'0","reported_seq":589,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349253+0000","last_change":"2024-02-19T21:12:09.349611+0000","last_active":"2024-02-19T21:13:08.349253+0000","last_peered":"2024-02-19T21:13:08.349253+0000","last_clean":"2024-02-19T21:13:08.349253+0000","last_became_active":"2024-02-19T21:12:09.349271+0000","last_became_peered":"2024-02-19T21:12:09.349271+0000","last_unstale":"2024-02-19T21:13:08.349253+0000","last_undegraded":"2024-02-19T21:13:08.349253+0000","last_fullsized":"2024-02-19T21:13:08.349253+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:40.580344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:40.580344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"61.1a","version":"503'99","reported_seq":799,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.489255+0000","last_change":"2024-02-19T21:12:11.754530+0000","last_active":"2024-02-19T21:13:08.489255+0000","last_peered":"2024-02-19T21:13:08.489255+0000","last_clean":"2024-02-19T21:13:08.489255+0000","last_became_active":"2024-02-19T21:12:09.322877+0000","last_became_peered":"2024-02-19T21:12:09.322877+0000","last_unstale":"2024-02-19T21:13:08.489255+0000","last_undegraded":"2024-02-19T21:13:08.489255+0000","last_fullsized":"2024-02-19T21:13:08.489255+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":495,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'98","last_scrub_stamp":"2024-02-19T21:10:55.797794+0000","last_deep_scrub":"448'98","last_deep_scrub_stamp":"2024-02-19T21:10:55.797794+0000","last_clean_scrub_stamp":"2024-02-19T21:10:55.797794+0000","log_size":99,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1283,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":188,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":563,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.983934+0000","last_change":"2024-02-19T21:12:09.339659+0000","last_active":"2024-02-19T21:13:08.983934+0000","last_peered":"2024-02-19T21:13:08.983934+0000","last_clean":"2024-02-19T21:13:08.983934+0000","last_became_active":"2024-02-19T21:12:09.339310+0000","last_became_peered":"2024-02-19T21:12:09.339310+0000","last_unstale":"2024-02-19T21:13:08.983934+0000","last_undegraded":"2024-02-19T21:13:08.983934+0000","last_fullsized":"2024-02-19T21:13:08.983934+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:21.732778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:10:21.732778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"61.1b","version":"512'59","reported_seq":372,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350205+0000","last_change":"2024-02-19T21:12:09.394440+0000","last_active":"2024-02-19T21:13:08.350205+0000","last_peered":"2024-02-19T21:13:08.350205+0000","last_clean":"2024-02-19T21:13:08.350205+0000","last_became_active":"2024-02-19T21:12:09.334066+0000","last_became_peered":"2024-02-19T21:12:09.334066+0000","last_unstale":"2024-02-19T21:13:08.350205+0000","last_undegraded":"2024-02-19T21:13:08.350205+0000","last_fullsized":"2024-02-19T21:13:08.350205+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'52","last_scrub_stamp":"2024-02-19T21:11:36.480577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:36.480577+0000","log_size":59,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":25,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1530,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":555,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.491388+0000","last_change":"2024-02-19T21:11:50.313926+0000","last_active":"2024-02-19T21:13:08.491388+0000","last_peered":"2024-02-19T21:13:08.491388+0000","last_clean":"2024-02-19T21:13:08.491388+0000","last_became_active":"2024-02-19T21:11:50.313560+0000","last_became_peered":"2024-02-19T21:11:50.313560+0000","last_unstale":"2024-02-19T21:13:08.491388+0000","last_undegraded":"2024-02-19T21:13:08.491388+0000","last_fullsized":"2024-02-19T21:13:08.491388+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.364732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.364732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.18","version":"214'102","reported_seq":660,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349355+0000","last_change":"2024-02-19T21:13:05.172245+0000","last_active":"2024-02-19T21:13:08.349355+0000","last_peered":"2024-02-19T21:13:08.349355+0000","last_clean":"2024-02-19T21:13:08.349355+0000","last_became_active":"2024-02-19T21:10:45.292216+0000","last_became_peered":"2024-02-19T21:10:45.292216+0000","last_unstale":"2024-02-19T21:13:08.349355+0000","last_undegraded":"2024-02-19T21:13:08.349355+0000","last_fullsized":"2024-02-19T21:13:08.349355+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'102","last_scrub_stamp":"2024-02-19T21:13:05.172172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:13:05.172172+0000","log_size":102,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":198,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.26","version":"0'0","reported_seq":92,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.492918+0000","last_change":"2024-02-19T21:12:09.339184+0000","last_active":"2024-02-19T21:13:08.492918+0000","last_peered":"2024-02-19T21:13:08.492918+0000","last_clean":"2024-02-19T21:13:08.492918+0000","last_became_active":"2024-02-19T21:12:09.338737+0000","last_became_peered":"2024-02-19T21:12:09.338737+0000","last_unstale":"2024-02-19T21:13:08.492918+0000","last_undegraded":"2024-02-19T21:13:08.492918+0000","last_fullsized":"2024-02-19T21:13:08.492918+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.19","version":"521'76","reported_seq":742,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.984990+0000","last_change":"2024-02-19T21:12:01.788396+0000","last_active":"2024-02-19T21:13:08.984990+0000","last_peered":"2024-02-19T21:13:08.984990+0000","last_clean":"2024-02-19T21:13:08.984990+0000","last_became_active":"2024-02-19T21:11:59.344724+0000","last_became_peered":"2024-02-19T21:11:59.344724+0000","last_unstale":"2024-02-19T21:13:08.984990+0000","last_undegraded":"2024-02-19T21:13:08.984990+0000","last_fullsized":"2024-02-19T21:13:08.984990+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'69","last_scrub_stamp":"2024-02-19T21:11:46.010760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:46.010760+0000","log_size":76,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":3,"num_write":30,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":596,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.490679+0000","last_change":"2024-02-19T21:12:09.343445+0000","last_active":"2024-02-19T21:13:08.490679+0000","last_peered":"2024-02-19T21:13:08.490679+0000","last_clean":"2024-02-19T21:13:08.490679+0000","last_became_active":"2024-02-19T21:12:09.342583+0000","last_became_peered":"2024-02-19T21:12:09.342583+0000","last_unstale":"2024-02-19T21:13:08.490679+0000","last_undegraded":"2024-02-19T21:13:08.490679+0000","last_fullsized":"2024-02-19T21:13:08.490679+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.955872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.955872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"61.1e","version":"512'67","reported_seq":635,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.494922+0000","last_change":"2024-02-19T21:13:08.494875+0000","last_active":"2024-02-19T21:13:08.494922+0000","last_peered":"2024-02-19T21:13:08.494922+0000","last_clean":"2024-02-19T21:13:08.494922+0000","last_became_active":"2024-02-19T21:12:09.348550+0000","last_became_peered":"2024-02-19T21:12:09.348550+0000","last_unstale":"2024-02-19T21:13:08.494922+0000","last_undegraded":"2024-02-19T21:13:08.494922+0000","last_fullsized":"2024-02-19T21:13:08.494922+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":495,"parent":"0.0","parent_split_bits":0,"last_scrub":"512'67","last_scrub_stamp":"2024-02-19T21:13:08.494760+0000","last_deep_scrub":"241'49","last_deep_scrub_stamp":"2024-02-19T21:07:01.787521+0000","last_clean_scrub_stamp":"2024-02-19T21:13:08.494760+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4497,"num_objects":12,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":9,"num_read_kb":1,"num_write":28,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.20","version":"397'2","reported_seq":617,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.488615+0000","last_change":"2024-02-19T21:12:43.380032+0000","last_active":"2024-02-19T21:13:08.488615+0000","last_peered":"2024-02-19T21:13:08.488615+0000","last_clean":"2024-02-19T21:13:08.488615+0000","last_became_active":"2024-02-19T21:12:43.379693+0000","last_became_peered":"2024-02-19T21:12:43.379693+0000","last_unstale":"2024-02-19T21:13:08.488615+0000","last_undegraded":"2024-02-19T21:13:08.488615+0000","last_fullsized":"2024-02-19T21:13:08.488615+0000","mapping_epoch":526,"log_start":"58'1","ondisk_log_start":"58'1","created":474,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":6,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.934474+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":437,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"61.1f","version":"512'46","reported_seq":600,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.495728+0000","last_change":"2024-02-19T21:12:20.180123+0000","last_active":"2024-02-19T21:13:08.495728+0000","last_peered":"2024-02-19T21:13:08.495728+0000","last_clean":"2024-02-19T21:13:08.495728+0000","last_became_active":"2024-02-19T21:11:59.358460+0000","last_became_peered":"2024-02-19T21:11:59.358460+0000","last_unstale":"2024-02-19T21:13:08.495728+0000","last_undegraded":"2024-02-19T21:13:08.495728+0000","last_fullsized":"2024-02-19T21:13:08.495728+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'44","last_scrub_stamp":"2024-02-19T21:12:20.180051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:12:20.180051+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1942,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":18,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":27,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":95,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349615+0000","last_change":"2024-02-19T21:12:09.337725+0000","last_active":"2024-02-19T21:13:08.349615+0000","last_peered":"2024-02-19T21:13:08.349615+0000","last_clean":"2024-02-19T21:13:08.349615+0000","last_became_active":"2024-02-19T21:12:09.337337+0000","last_became_peered":"2024-02-19T21:12:09.337337+0000","last_unstale":"2024-02-19T21:13:08.349615+0000","last_undegraded":"2024-02-19T21:13:08.349615+0000","last_fullsized":"2024-02-19T21:13:08.349615+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.081993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.081993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"61.1c","version":"503'155","reported_seq":963,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.985885+0000","last_change":"2024-02-19T21:12:16.312907+0000","last_active":"2024-02-19T21:13:08.985885+0000","last_peered":"2024-02-19T21:13:08.985885+0000","last_clean":"2024-02-19T21:13:08.985885+0000","last_became_active":"2024-02-19T21:12:16.312435+0000","last_became_peered":"2024-02-19T21:12:16.312435+0000","last_unstale":"2024-02-19T21:13:08.985885+0000","last_undegraded":"2024-02-19T21:13:08.985885+0000","last_fullsized":"2024-02-19T21:13:08.985885+0000","mapping_epoch":499,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":500,"parent":"0.0","parent_split_bits":0,"last_scrub":"429'153","last_scrub_stamp":"2024-02-19T21:11:44.965810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:44.965810+0000","log_size":155,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114,"num_objects":11,"num_object_clones":1,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":10,"num_read_kb":1,"num_write":49,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":544,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.347984+0000","last_change":"2024-02-19T21:12:09.353297+0000","last_active":"2024-02-19T21:13:08.347984+0000","last_peered":"2024-02-19T21:13:08.347984+0000","last_clean":"2024-02-19T21:13:08.347984+0000","last_became_active":"2024-02-19T21:12:09.352904+0000","last_became_peered":"2024-02-19T21:12:09.352904+0000","last_unstale":"2024-02-19T21:13:08.347984+0000","last_undegraded":"2024-02-19T21:13:08.347984+0000","last_fullsized":"2024-02-19T21:13:08.347984+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.089715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.089715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"61.1d","version":"370'36","reported_seq":547,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.982631+0000","last_change":"2024-02-19T21:11:32.941492+0000","last_active":"2024-02-19T21:13:08.982631+0000","last_peered":"2024-02-19T21:13:08.982631+0000","last_clean":"2024-02-19T21:13:08.982631+0000","last_became_active":"2024-02-19T21:04:54.655665+0000","last_became_peered":"2024-02-19T21:04:54.655665+0000","last_unstale":"2024-02-19T21:13:08.982631+0000","last_undegraded":"2024-02-19T21:13:08.982631+0000","last_fullsized":"2024-02-19T21:13:08.982631+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":114,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'36","last_scrub_stamp":"2024-02-19T21:11:32.941416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:53.511966+0000","last_clean_scrub_stamp":"2024-02-19T21:11:32.941416+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"493'12","reported_seq":710,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-02-19T21:13:09.480910+0000","last_change":"2024-02-19T21:12:18.452113+0000","last_active":"2024-02-19T21:13:09.480910+0000","last_peered":"2024-02-19T21:13:09.480910+0000","last_clean":"2024-02-19T21:13:09.480910+0000","last_became_active":"2024-02-19T21:12:09.332507+0000","last_became_peered":"2024-02-19T21:12:09.332507+0000","last_unstale":"2024-02-19T21:13:09.480910+0000","last_undegraded":"2024-02-19T21:13:09.480910+0000","last_fullsized":"2024-02-19T21:13:09.480910+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'12","last_scrub_stamp":"2024-02-19T21:12:18.452020+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:12:18.452020+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'28","reported_seq":589,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.984675+0000","last_change":"2024-02-19T21:12:10.211768+0000","last_active":"2024-02-19T21:13:08.984675+0000","last_peered":"2024-02-19T21:13:08.984675+0000","last_clean":"2024-02-19T21:13:08.984675+0000","last_became_active":"2024-02-19T21:12:09.341122+0000","last_became_peered":"2024-02-19T21:12:09.341122+0000","last_unstale":"2024-02-19T21:13:08.984675+0000","last_undegraded":"2024-02-19T21:13:08.984675+0000","last_fullsized":"2024-02-19T21:13:08.984675+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-19T21:11:47.051823+0000","last_deep_scrub":"473'28","last_deep_scrub_stamp":"2024-02-19T21:11:47.051823+0000","last_clean_scrub_stamp":"2024-02-19T21:11:47.051823+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":552,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.492316+0000","last_change":"2024-02-19T21:12:22.215650+0000","last_active":"2024-02-19T21:13:08.492316+0000","last_peered":"2024-02-19T21:13:08.492316+0000","last_clean":"2024-02-19T21:13:08.492316+0000","last_became_active":"2024-02-19T21:10:40.299425+0000","last_became_peered":"2024-02-19T21:10:40.299425+0000","last_unstale":"2024-02-19T21:13:08.492316+0000","last_undegraded":"2024-02-19T21:13:08.492316+0000","last_fullsized":"2024-02-19T21:13:08.492316+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:22.215548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_clean_scrub_stamp":"2024-02-19T21:12:22.215548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":531,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.986551+0000","last_change":"2024-02-19T21:12:51.469049+0000","last_active":"2024-02-19T21:13:08.986551+0000","last_peered":"2024-02-19T21:13:08.986551+0000","last_clean":"2024-02-19T21:13:08.986551+0000","last_became_active":"2024-02-19T21:12:09.326173+0000","last_became_peered":"2024-02-19T21:12:09.326173+0000","last_unstale":"2024-02-19T21:13:08.986551+0000","last_undegraded":"2024-02-19T21:13:08.986551+0000","last_fullsized":"2024-02-19T21:13:08.986551+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:51.468971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.468971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":473,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.492069+0000","last_change":"2024-02-19T21:12:58.081198+0000","last_active":"2024-02-19T21:13:08.492069+0000","last_peered":"2024-02-19T21:13:08.492069+0000","last_clean":"2024-02-19T21:13:08.492069+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:13:08.492069+0000","last_undegraded":"2024-02-19T21:13:08.492069+0000","last_fullsized":"2024-02-19T21:13:08.492069+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:58.081106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:58.081106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'16","reported_seq":443,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.982675+0000","last_change":"2024-02-19T21:11:59.756726+0000","last_active":"2024-02-19T21:13:08.982675+0000","last_peered":"2024-02-19T21:13:08.982675+0000","last_clean":"2024-02-19T21:13:08.982675+0000","last_became_active":"2024-02-19T21:11:59.319397+0000","last_became_peered":"2024-02-19T21:11:59.319397+0000","last_unstale":"2024-02-19T21:13:08.982675+0000","last_undegraded":"2024-02-19T21:13:08.982675+0000","last_fullsized":"2024-02-19T21:13:08.982675+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'16","last_scrub_stamp":"2024-02-19T21:11:46.956424+0000","last_deep_scrub":"473'16","last_deep_scrub_stamp":"2024-02-19T21:11:46.956424+0000","last_clean_scrub_stamp":"2024-02-19T21:11:46.956424+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":558,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.983927+0000","last_change":"2024-02-19T21:12:06.016884+0000","last_active":"2024-02-19T21:13:08.983927+0000","last_peered":"2024-02-19T21:13:08.983927+0000","last_clean":"2024-02-19T21:13:08.983927+0000","last_became_active":"2024-02-19T21:10:41.312116+0000","last_became_peered":"2024-02-19T21:10:41.312116+0000","last_unstale":"2024-02-19T21:13:08.983927+0000","last_undegraded":"2024-02-19T21:13:08.983927+0000","last_fullsized":"2024-02-19T21:13:08.983927+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.016817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.016817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":557,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.492019+0000","last_change":"2024-02-19T21:13:05.193297+0000","last_active":"2024-02-19T21:13:08.492019+0000","last_peered":"2024-02-19T21:13:08.492019+0000","last_clean":"2024-02-19T21:13:08.492019+0000","last_became_active":"2024-02-19T21:10:41.305574+0000","last_became_peered":"2024-02-19T21:10:41.305574+0000","last_unstale":"2024-02-19T21:13:08.492019+0000","last_undegraded":"2024-02-19T21:13:08.492019+0000","last_fullsized":"2024-02-19T21:13:08.492019+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:05.193223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:05.193223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":358,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.488912+0000","last_change":"2024-02-19T21:11:59.323172+0000","last_active":"2024-02-19T21:13:08.488912+0000","last_peered":"2024-02-19T21:13:08.488912+0000","last_clean":"2024-02-19T21:13:08.488912+0000","last_became_active":"2024-02-19T21:11:59.322814+0000","last_became_peered":"2024-02-19T21:11:59.322814+0000","last_unstale":"2024-02-19T21:13:08.488912+0000","last_undegraded":"2024-02-19T21:13:08.488912+0000","last_fullsized":"2024-02-19T21:13:08.488912+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:34.504478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:34.504478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":467,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.988811+0000","last_change":"2024-02-19T21:10:57.130256+0000","last_active":"2024-02-19T21:13:08.988811+0000","last_peered":"2024-02-19T21:13:08.988811+0000","last_clean":"2024-02-19T21:13:08.988811+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:13:08.988811+0000","last_undegraded":"2024-02-19T21:13:08.988811+0000","last_fullsized":"2024-02-19T21:13:08.988811+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.130171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.130171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"459'35","reported_seq":571,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.981912+0000","last_change":"2024-02-19T21:12:10.552781+0000","last_active":"2024-02-19T21:13:08.981912+0000","last_peered":"2024-02-19T21:13:08.981912+0000","last_clean":"2024-02-19T21:13:08.981912+0000","last_became_active":"2024-02-19T21:12:09.333541+0000","last_became_peered":"2024-02-19T21:12:09.333541+0000","last_unstale":"2024-02-19T21:13:08.981912+0000","last_undegraded":"2024-02-19T21:13:08.981912+0000","last_fullsized":"2024-02-19T21:13:08.981912+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:11:04.345836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:04.345836+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":534,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.491857+0000","last_change":"2024-02-19T21:12:09.339263+0000","last_active":"2024-02-19T21:13:08.491857+0000","last_peered":"2024-02-19T21:13:08.491857+0000","last_clean":"2024-02-19T21:13:08.491857+0000","last_became_active":"2024-02-19T21:11:59.336426+0000","last_became_peered":"2024-02-19T21:11:59.336426+0000","last_unstale":"2024-02-19T21:13:08.491857+0000","last_undegraded":"2024-02-19T21:13:08.491857+0000","last_fullsized":"2024-02-19T21:13:08.491857+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:09.339147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:12:09.339147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":527,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.491338+0000","last_change":"2024-02-19T21:12:16.244173+0000","last_active":"2024-02-19T21:13:08.491338+0000","last_peered":"2024-02-19T21:13:08.491338+0000","last_clean":"2024-02-19T21:13:08.491338+0000","last_became_active":"2024-02-19T21:11:59.340544+0000","last_became_peered":"2024-02-19T21:11:59.340544+0000","last_unstale":"2024-02-19T21:13:08.491338+0000","last_undegraded":"2024-02-19T21:13:08.491338+0000","last_fullsized":"2024-02-19T21:13:08.491338+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:16.244104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:16.244104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":529,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.489456+0000","last_change":"2024-02-19T21:11:33.469310+0000","last_active":"2024-02-19T21:13:08.489456+0000","last_peered":"2024-02-19T21:13:08.489456+0000","last_clean":"2024-02-19T21:13:08.489456+0000","last_became_active":"2024-02-19T21:08:04.265081+0000","last_became_peered":"2024-02-19T21:08:04.265081+0000","last_unstale":"2024-02-19T21:13:08.489456+0000","last_undegraded":"2024-02-19T21:13:08.489456+0000","last_fullsized":"2024-02-19T21:13:08.489456+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:33.469232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_clean_scrub_stamp":"2024-02-19T21:11:33.469232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":489,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.493226+0000","last_change":"2024-02-19T21:12:55.491781+0000","last_active":"2024-02-19T21:13:08.493226+0000","last_peered":"2024-02-19T21:13:08.493226+0000","last_clean":"2024-02-19T21:13:08.493226+0000","last_became_active":"2024-02-19T21:10:41.317284+0000","last_became_peered":"2024-02-19T21:10:41.317284+0000","last_unstale":"2024-02-19T21:13:08.493226+0000","last_undegraded":"2024-02-19T21:13:08.493226+0000","last_fullsized":"2024-02-19T21:13:08.493226+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:55.491665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.491665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"459'31","reported_seq":172,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350494+0000","last_change":"2024-02-19T21:11:44.421114+0000","last_active":"2024-02-19T21:13:08.350494+0000","last_peered":"2024-02-19T21:13:08.350494+0000","last_clean":"2024-02-19T21:13:08.350494+0000","last_became_active":"2024-02-19T21:10:41.313853+0000","last_became_peered":"2024-02-19T21:10:41.313853+0000","last_unstale":"2024-02-19T21:13:08.350494+0000","last_undegraded":"2024-02-19T21:13:08.350494+0000","last_fullsized":"2024-02-19T21:13:08.350494+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'31","last_scrub_stamp":"2024-02-19T21:11:44.421033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:44.421033+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":554,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.490076+0000","last_change":"2024-02-19T21:13:07.450202+0000","last_active":"2024-02-19T21:13:08.490076+0000","last_peered":"2024-02-19T21:13:08.490076+0000","last_clean":"2024-02-19T21:13:08.490076+0000","last_became_active":"2024-02-19T21:10:41.307517+0000","last_became_peered":"2024-02-19T21:10:41.307517+0000","last_unstale":"2024-02-19T21:13:08.490076+0000","last_undegraded":"2024-02-19T21:13:08.490076+0000","last_fullsized":"2024-02-19T21:13:08.490076+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:13:07.450123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.450123+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":554,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.984985+0000","last_change":"2024-02-19T21:12:58.838081+0000","last_active":"2024-02-19T21:13:08.984985+0000","last_peered":"2024-02-19T21:13:08.984985+0000","last_clean":"2024-02-19T21:13:08.984985+0000","last_became_active":"2024-02-19T21:12:09.369409+0000","last_became_peered":"2024-02-19T21:12:09.369409+0000","last_unstale":"2024-02-19T21:13:08.984985+0000","last_undegraded":"2024-02-19T21:13:08.984985+0000","last_fullsized":"2024-02-19T21:13:08.984985+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:58.837973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:58.837973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":555,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.495944+0000","last_change":"2024-02-19T21:13:03.126485+0000","last_active":"2024-02-19T21:13:08.495944+0000","last_peered":"2024-02-19T21:13:08.495944+0000","last_clean":"2024-02-19T21:13:08.495944+0000","last_became_active":"2024-02-19T21:10:41.307577+0000","last_became_peered":"2024-02-19T21:10:41.307577+0000","last_unstale":"2024-02-19T21:13:08.495944+0000","last_undegraded":"2024-02-19T21:13:08.495944+0000","last_fullsized":"2024-02-19T21:13:08.495944+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:03.126414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:13:03.126414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":133,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349047+0000","last_change":"2024-02-19T21:10:50.424462+0000","last_active":"2024-02-19T21:13:08.349047+0000","last_peered":"2024-02-19T21:13:08.349047+0000","last_clean":"2024-02-19T21:13:08.349047+0000","last_became_active":"2024-02-19T21:10:41.302670+0000","last_became_peered":"2024-02-19T21:10:41.302670+0000","last_unstale":"2024-02-19T21:13:08.349047+0000","last_undegraded":"2024-02-19T21:13:08.349047+0000","last_fullsized":"2024-02-19T21:13:08.349047+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:50.424381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.424381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"459'25","reported_seq":181,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350388+0000","last_change":"2024-02-19T21:12:51.420519+0000","last_active":"2024-02-19T21:13:08.350388+0000","last_peered":"2024-02-19T21:13:08.350388+0000","last_clean":"2024-02-19T21:13:08.350388+0000","last_became_active":"2024-02-19T21:12:09.408620+0000","last_became_peered":"2024-02-19T21:12:09.408620+0000","last_unstale":"2024-02-19T21:13:08.350388+0000","last_undegraded":"2024-02-19T21:13:08.350388+0000","last_fullsized":"2024-02-19T21:13:08.350388+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'25","last_scrub_stamp":"2024-02-19T21:12:51.420403+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.420403+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":535,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.981998+0000","last_change":"2024-02-19T21:10:12.628865+0000","last_active":"2024-02-19T21:13:08.981998+0000","last_peered":"2024-02-19T21:13:08.981998+0000","last_clean":"2024-02-19T21:13:08.981998+0000","last_became_active":"2024-02-19T21:08:04.268406+0000","last_became_peered":"2024-02-19T21:08:04.268406+0000","last_unstale":"2024-02-19T21:13:08.981998+0000","last_undegraded":"2024-02-19T21:13:08.981998+0000","last_fullsized":"2024-02-19T21:13:08.981998+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:12.628801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:12.628801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":672,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-02-19T21:13:09.487179+0000","last_change":"2024-02-19T21:12:21.202447+0000","last_active":"2024-02-19T21:13:09.487179+0000","last_peered":"2024-02-19T21:13:09.487179+0000","last_clean":"2024-02-19T21:13:09.487179+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:13:09.487179+0000","last_undegraded":"2024-02-19T21:13:09.487179+0000","last_fullsized":"2024-02-19T21:13:09.487179+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:12:21.202371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:21.202371+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":133,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.352193+0000","last_change":"2024-02-19T21:12:21.380805+0000","last_active":"2024-02-19T21:13:08.352193+0000","last_peered":"2024-02-19T21:13:08.352193+0000","last_clean":"2024-02-19T21:13:08.352193+0000","last_became_active":"2024-02-19T21:10:41.298893+0000","last_became_peered":"2024-02-19T21:10:41.298893+0000","last_unstale":"2024-02-19T21:13:08.352193+0000","last_undegraded":"2024-02-19T21:13:08.352193+0000","last_fullsized":"2024-02-19T21:13:08.352193+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:21.380734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:12:21.380734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":280,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349114+0000","last_change":"2024-02-19T21:12:16.339644+0000","last_active":"2024-02-19T21:13:08.349114+0000","last_peered":"2024-02-19T21:13:08.349114+0000","last_clean":"2024-02-19T21:13:08.349114+0000","last_became_active":"2024-02-19T21:11:59.318845+0000","last_became_peered":"2024-02-19T21:11:59.318845+0000","last_unstale":"2024-02-19T21:13:08.349114+0000","last_undegraded":"2024-02-19T21:13:08.349114+0000","last_fullsized":"2024-02-19T21:13:08.349114+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:16.339577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:16.339577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"459'27","reported_seq":177,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.495539+0000","last_change":"2024-02-19T21:11:08.202280+0000","last_active":"2024-02-19T21:13:08.495539+0000","last_peered":"2024-02-19T21:13:08.495539+0000","last_clean":"2024-02-19T21:13:08.495539+0000","last_became_active":"2024-02-19T21:10:41.307576+0000","last_became_peered":"2024-02-19T21:10:41.307576+0000","last_unstale":"2024-02-19T21:13:08.495539+0000","last_undegraded":"2024-02-19T21:13:08.495539+0000","last_fullsized":"2024-02-19T21:13:08.495539+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:11:08.202208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:08.202208+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":580,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.981743+0000","last_change":"2024-02-19T21:12:40.370031+0000","last_active":"2024-02-19T21:13:08.981743+0000","last_peered":"2024-02-19T21:13:08.981743+0000","last_clean":"2024-02-19T21:13:08.981743+0000","last_became_active":"2024-02-19T21:12:40.369640+0000","last_became_peered":"2024-02-19T21:12:40.369640+0000","last_unstale":"2024-02-19T21:13:08.981743+0000","last_undegraded":"2024-02-19T21:13:08.981743+0000","last_fullsized":"2024-02-19T21:13:08.981743+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.738600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.738600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"483'11","reported_seq":181,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-02-19T21:13:09.479411+0000","last_change":"2024-02-19T21:12:56.450133+0000","last_active":"2024-02-19T21:13:09.479411+0000","last_peered":"2024-02-19T21:13:09.479411+0000","last_clean":"2024-02-19T21:13:09.479411+0000","last_became_active":"2024-02-19T21:11:59.336645+0000","last_became_peered":"2024-02-19T21:11:59.336645+0000","last_unstale":"2024-02-19T21:13:09.479411+0000","last_undegraded":"2024-02-19T21:13:09.479411+0000","last_fullsized":"2024-02-19T21:13:09.479411+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"483'11","last_scrub_stamp":"2024-02-19T21:12:56.450044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:56.450044+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":527,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.987247+0000","last_change":"2024-02-19T21:12:54.437089+0000","last_active":"2024-02-19T21:13:08.987247+0000","last_peered":"2024-02-19T21:13:08.987247+0000","last_clean":"2024-02-19T21:13:08.987247+0000","last_became_active":"2024-02-19T21:11:59.344440+0000","last_became_peered":"2024-02-19T21:11:59.344440+0000","last_unstale":"2024-02-19T21:13:08.987247+0000","last_undegraded":"2024-02-19T21:13:08.987247+0000","last_fullsized":"2024-02-19T21:13:08.987247+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:54.436977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:54.436977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":476,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.985371+0000","last_change":"2024-02-19T21:12:09.327574+0000","last_active":"2024-02-19T21:13:08.985371+0000","last_peered":"2024-02-19T21:13:08.985371+0000","last_clean":"2024-02-19T21:13:08.985371+0000","last_became_active":"2024-02-19T21:12:09.326909+0000","last_became_peered":"2024-02-19T21:12:09.326909+0000","last_unstale":"2024-02-19T21:13:08.985371+0000","last_undegraded":"2024-02-19T21:13:08.985371+0000","last_fullsized":"2024-02-19T21:13:08.985371+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.079353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.079353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"459'19","reported_seq":180,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.352091+0000","last_change":"2024-02-19T21:12:10.647686+0000","last_active":"2024-02-19T21:13:08.352091+0000","last_peered":"2024-02-19T21:13:08.352091+0000","last_clean":"2024-02-19T21:13:08.352091+0000","last_became_active":"2024-02-19T21:12:09.409065+0000","last_became_peered":"2024-02-19T21:12:09.409065+0000","last_unstale":"2024-02-19T21:13:08.352091+0000","last_undegraded":"2024-02-19T21:13:08.352091+0000","last_fullsized":"2024-02-19T21:13:08.352091+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:11:42.333781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:42.333781+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":567,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.489796+0000","last_change":"2024-02-19T21:11:59.332430+0000","last_active":"2024-02-19T21:13:08.489796+0000","last_peered":"2024-02-19T21:13:08.489796+0000","last_clean":"2024-02-19T21:13:08.489796+0000","last_became_active":"2024-02-19T21:11:59.331675+0000","last_became_peered":"2024-02-19T21:11:59.331675+0000","last_unstale":"2024-02-19T21:13:08.489796+0000","last_undegraded":"2024-02-19T21:13:08.489796+0000","last_fullsized":"2024-02-19T21:13:08.489796+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:42.529489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:42.529489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":541,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.989206+0000","last_change":"2024-02-19T21:12:09.350127+0000","last_active":"2024-02-19T21:13:08.989206+0000","last_peered":"2024-02-19T21:13:08.989206+0000","last_clean":"2024-02-19T21:13:08.989206+0000","last_became_active":"2024-02-19T21:12:09.349667+0000","last_became_peered":"2024-02-19T21:12:09.349667+0000","last_unstale":"2024-02-19T21:13:08.989206+0000","last_undegraded":"2024-02-19T21:13:08.989206+0000","last_fullsized":"2024-02-19T21:13:08.989206+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:58.096025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.096025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":273,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351784+0000","last_change":"2024-02-19T21:12:09.335125+0000","last_active":"2024-02-19T21:13:08.351784+0000","last_peered":"2024-02-19T21:13:08.351784+0000","last_clean":"2024-02-19T21:13:08.351784+0000","last_became_active":"2024-02-19T21:12:09.333625+0000","last_became_peered":"2024-02-19T21:12:09.333625+0000","last_unstale":"2024-02-19T21:13:08.351784+0000","last_undegraded":"2024-02-19T21:13:08.351784+0000","last_fullsized":"2024-02-19T21:13:08.351784+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:06.547494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:06.547494+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":273,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350294+0000","last_change":"2024-02-19T21:12:09.348064+0000","last_active":"2024-02-19T21:13:08.350294+0000","last_peered":"2024-02-19T21:13:08.350294+0000","last_clean":"2024-02-19T21:13:08.350294+0000","last_became_active":"2024-02-19T21:12:09.347240+0000","last_became_peered":"2024-02-19T21:12:09.347240+0000","last_unstale":"2024-02-19T21:13:08.350294+0000","last_undegraded":"2024-02-19T21:13:08.350294+0000","last_fullsized":"2024-02-19T21:13:08.350294+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:43.407517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.407517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"434'12","reported_seq":359,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-02-19T21:13:09.481883+0000","last_change":"2024-02-19T21:10:41.321881+0000","last_active":"2024-02-19T21:13:09.481883+0000","last_peered":"2024-02-19T21:13:09.481883+0000","last_clean":"2024-02-19T21:13:09.481883+0000","last_became_active":"2024-02-19T21:10:41.297897+0000","last_became_peered":"2024-02-19T21:10:41.297897+0000","last_unstale":"2024-02-19T21:13:09.481883+0000","last_undegraded":"2024-02-19T21:13:09.481883+0000","last_fullsized":"2024-02-19T21:13:09.481883+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'11","last_scrub_stamp":"2024-02-19T21:09:39.597001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:39.597001+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"459'28","reported_seq":195,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349164+0000","last_change":"2024-02-19T21:12:10.323234+0000","last_active":"2024-02-19T21:13:08.349164+0000","last_peered":"2024-02-19T21:13:08.349164+0000","last_clean":"2024-02-19T21:13:08.349164+0000","last_became_active":"2024-02-19T21:12:09.369385+0000","last_became_peered":"2024-02-19T21:12:09.369385+0000","last_unstale":"2024-02-19T21:13:08.349164+0000","last_undegraded":"2024-02-19T21:13:08.349164+0000","last_fullsized":"2024-02-19T21:13:08.349164+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:10:49.466227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:49.466227+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":540,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.493151+0000","last_change":"2024-02-19T21:12:09.343164+0000","last_active":"2024-02-19T21:13:08.493151+0000","last_peered":"2024-02-19T21:13:08.493151+0000","last_clean":"2024-02-19T21:13:08.493151+0000","last_became_active":"2024-02-19T21:12:09.342814+0000","last_became_peered":"2024-02-19T21:12:09.342814+0000","last_unstale":"2024-02-19T21:13:08.493151+0000","last_undegraded":"2024-02-19T21:13:08.493151+0000","last_fullsized":"2024-02-19T21:13:08.493151+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.364732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.364732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":524,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.491071+0000","last_change":"2024-02-19T21:12:09.335134+0000","last_active":"2024-02-19T21:13:08.491071+0000","last_peered":"2024-02-19T21:13:08.491071+0000","last_clean":"2024-02-19T21:13:08.491071+0000","last_became_active":"2024-02-19T21:12:09.333970+0000","last_became_peered":"2024-02-19T21:12:09.333970+0000","last_unstale":"2024-02-19T21:13:08.491071+0000","last_undegraded":"2024-02-19T21:13:08.491071+0000","last_fullsized":"2024-02-19T21:13:08.491071+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.851123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.851123+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":352,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.984154+0000","last_change":"2024-02-19T21:11:59.327843+0000","last_active":"2024-02-19T21:13:08.984154+0000","last_peered":"2024-02-19T21:13:08.984154+0000","last_clean":"2024-02-19T21:13:08.984154+0000","last_became_active":"2024-02-19T21:11:59.323698+0000","last_became_peered":"2024-02-19T21:11:59.323698+0000","last_unstale":"2024-02-19T21:13:08.984154+0000","last_undegraded":"2024-02-19T21:13:08.984154+0000","last_fullsized":"2024-02-19T21:13:08.984154+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:06.601228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:06.601228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":133,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.348220+0000","last_change":"2024-02-19T21:10:59.312931+0000","last_active":"2024-02-19T21:13:08.348220+0000","last_peered":"2024-02-19T21:13:08.348220+0000","last_clean":"2024-02-19T21:13:08.348220+0000","last_became_active":"2024-02-19T21:10:41.304695+0000","last_became_peered":"2024-02-19T21:10:41.304695+0000","last_unstale":"2024-02-19T21:13:08.348220+0000","last_undegraded":"2024-02-19T21:13:08.348220+0000","last_fullsized":"2024-02-19T21:13:08.348220+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:59.312863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.312863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"459'30","reported_seq":327,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349326+0000","last_change":"2024-02-19T21:11:59.380681+0000","last_active":"2024-02-19T21:13:08.349326+0000","last_peered":"2024-02-19T21:13:08.349326+0000","last_clean":"2024-02-19T21:13:08.349326+0000","last_became_active":"2024-02-19T21:11:59.327859+0000","last_became_peered":"2024-02-19T21:11:59.327859+0000","last_unstale":"2024-02-19T21:13:08.349326+0000","last_undegraded":"2024-02-19T21:13:08.349326+0000","last_fullsized":"2024-02-19T21:13:08.349326+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:11:02.671837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.671837+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":575,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.982623+0000","last_change":"2024-02-19T21:12:09.369780+0000","last_active":"2024-02-19T21:13:08.982623+0000","last_peered":"2024-02-19T21:13:08.982623+0000","last_clean":"2024-02-19T21:13:08.982623+0000","last_became_active":"2024-02-19T21:12:09.369477+0000","last_became_peered":"2024-02-19T21:12:09.369477+0000","last_unstale":"2024-02-19T21:13:08.982623+0000","last_undegraded":"2024-02-19T21:13:08.982623+0000","last_fullsized":"2024-02-19T21:13:08.982623+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":133,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349453+0000","last_change":"2024-02-19T21:12:06.410909+0000","last_active":"2024-02-19T21:13:08.349453+0000","last_peered":"2024-02-19T21:13:08.349453+0000","last_clean":"2024-02-19T21:13:08.349453+0000","last_became_active":"2024-02-19T21:10:41.295301+0000","last_became_peered":"2024-02-19T21:10:41.295301+0000","last_unstale":"2024-02-19T21:13:08.349453+0000","last_undegraded":"2024-02-19T21:13:08.349453+0000","last_fullsized":"2024-02-19T21:13:08.349453+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.410847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.410847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":282,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.352073+0000","last_change":"2024-02-19T21:12:09.332689+0000","last_active":"2024-02-19T21:13:08.352073+0000","last_peered":"2024-02-19T21:13:08.352073+0000","last_clean":"2024-02-19T21:13:08.352073+0000","last_became_active":"2024-02-19T21:12:09.332349+0000","last_became_peered":"2024-02-19T21:12:09.332349+0000","last_unstale":"2024-02-19T21:13:08.352073+0000","last_undegraded":"2024-02-19T21:13:08.352073+0000","last_fullsized":"2024-02-19T21:13:08.352073+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:46.406402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:46.406402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":518,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.987249+0000","last_change":"2024-02-19T21:12:09.340947+0000","last_active":"2024-02-19T21:13:08.987249+0000","last_peered":"2024-02-19T21:13:08.987249+0000","last_clean":"2024-02-19T21:13:08.987249+0000","last_became_active":"2024-02-19T21:12:09.340614+0000","last_became_peered":"2024-02-19T21:12:09.340614+0000","last_unstale":"2024-02-19T21:13:08.987249+0000","last_undegraded":"2024-02-19T21:13:08.987249+0000","last_fullsized":"2024-02-19T21:13:08.987249+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.669336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.669336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"459'29","reported_seq":435,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.985869+0000","last_change":"2024-02-19T21:12:00.056470+0000","last_active":"2024-02-19T21:13:08.985869+0000","last_peered":"2024-02-19T21:13:08.985869+0000","last_clean":"2024-02-19T21:13:08.985869+0000","last_became_active":"2024-02-19T21:11:59.332542+0000","last_became_peered":"2024-02-19T21:11:59.332542+0000","last_unstale":"2024-02-19T21:13:08.985869+0000","last_undegraded":"2024-02-19T21:13:08.985869+0000","last_fullsized":"2024-02-19T21:13:08.985869+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:10:11.165973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:11.165973+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":587,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.491555+0000","last_change":"2024-02-19T21:11:59.335901+0000","last_active":"2024-02-19T21:13:08.491555+0000","last_peered":"2024-02-19T21:13:08.491555+0000","last_clean":"2024-02-19T21:13:08.491555+0000","last_became_active":"2024-02-19T21:11:59.335475+0000","last_became_peered":"2024-02-19T21:11:59.335475+0000","last_unstale":"2024-02-19T21:13:08.491555+0000","last_undegraded":"2024-02-19T21:13:08.491555+0000","last_fullsized":"2024-02-19T21:13:08.491555+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:40.580344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:40.580344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":520,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.981855+0000","last_change":"2024-02-19T21:12:20.902387+0000","last_active":"2024-02-19T21:13:08.981855+0000","last_peered":"2024-02-19T21:13:08.981855+0000","last_clean":"2024-02-19T21:13:08.981855+0000","last_became_active":"2024-02-19T21:12:09.347197+0000","last_became_peered":"2024-02-19T21:12:09.347197+0000","last_unstale":"2024-02-19T21:13:08.981855+0000","last_undegraded":"2024-02-19T21:13:08.981855+0000","last_fullsized":"2024-02-19T21:13:08.981855+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:20.902297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:12:20.902297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":477,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.496022+0000","last_change":"2024-02-19T21:11:59.359210+0000","last_active":"2024-02-19T21:13:08.496022+0000","last_peered":"2024-02-19T21:13:08.496022+0000","last_clean":"2024-02-19T21:13:08.496022+0000","last_became_active":"2024-02-19T21:11:59.358933+0000","last_became_peered":"2024-02-19T21:11:59.358933+0000","last_unstale":"2024-02-19T21:13:08.496022+0000","last_undegraded":"2024-02-19T21:13:08.496022+0000","last_fullsized":"2024-02-19T21:13:08.496022+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.189509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":523,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.989278+0000","last_change":"2024-02-19T21:11:59.334472+0000","last_active":"2024-02-19T21:13:08.989278+0000","last_peered":"2024-02-19T21:13:08.989278+0000","last_clean":"2024-02-19T21:13:08.989278+0000","last_became_active":"2024-02-19T21:11:59.333491+0000","last_became_peered":"2024-02-19T21:11:59.333491+0000","last_unstale":"2024-02-19T21:13:08.989278+0000","last_undegraded":"2024-02-19T21:13:08.989278+0000","last_fullsized":"2024-02-19T21:13:08.989278+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:30.893485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:30.893485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"459'24","reported_seq":419,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.982733+0000","last_change":"2024-02-19T21:12:50.030922+0000","last_active":"2024-02-19T21:13:08.982733+0000","last_peered":"2024-02-19T21:13:08.982733+0000","last_clean":"2024-02-19T21:13:08.982733+0000","last_became_active":"2024-02-19T21:11:59.348806+0000","last_became_peered":"2024-02-19T21:11:59.348806+0000","last_unstale":"2024-02-19T21:13:08.982733+0000","last_undegraded":"2024-02-19T21:13:08.982733+0000","last_fullsized":"2024-02-19T21:13:08.982733+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'24","last_scrub_stamp":"2024-02-19T21:12:50.030857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:50.030857+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"397'2","reported_seq":584,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.489563+0000","last_change":"2024-02-19T21:12:02.320023+0000","last_active":"2024-02-19T21:13:08.489563+0000","last_peered":"2024-02-19T21:13:08.489563+0000","last_clean":"2024-02-19T21:13:08.489563+0000","last_became_active":"2024-02-19T21:12:02.319577+0000","last_became_peered":"2024-02-19T21:12:02.319577+0000","last_unstale":"2024-02-19T21:13:08.489563+0000","last_undegraded":"2024-02-19T21:13:08.489563+0000","last_fullsized":"2024-02-19T21:13:08.489563+0000","mapping_epoch":485,"log_start":"58'1","ondisk_log_start":"58'1","created":57,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.934474+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":436,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":142,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349270+0000","last_change":"2024-02-19T21:11:59.320654+0000","last_active":"2024-02-19T21:13:08.349270+0000","last_peered":"2024-02-19T21:13:08.349270+0000","last_clean":"2024-02-19T21:13:08.349270+0000","last_became_active":"2024-02-19T21:11:59.320327+0000","last_became_peered":"2024-02-19T21:11:59.320327+0000","last_unstale":"2024-02-19T21:13:08.349270+0000","last_undegraded":"2024-02-19T21:13:08.349270+0000","last_fullsized":"2024-02-19T21:13:08.349270+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.455993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.455993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":529,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.985200+0000","last_change":"2024-02-19T21:10:51.691357+0000","last_active":"2024-02-19T21:13:08.985200+0000","last_peered":"2024-02-19T21:13:08.985200+0000","last_clean":"2024-02-19T21:13:08.985200+0000","last_became_active":"2024-02-19T21:08:04.282243+0000","last_became_peered":"2024-02-19T21:08:04.282243+0000","last_unstale":"2024-02-19T21:13:08.985200+0000","last_undegraded":"2024-02-19T21:13:08.985200+0000","last_fullsized":"2024-02-19T21:13:08.985200+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.691284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.691284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":467,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.985747+0000","last_change":"2024-02-19T21:12:09.328499+0000","last_active":"2024-02-19T21:13:08.985747+0000","last_peered":"2024-02-19T21:13:08.985747+0000","last_clean":"2024-02-19T21:13:08.985747+0000","last_became_active":"2024-02-19T21:12:09.326936+0000","last_became_peered":"2024-02-19T21:12:09.326936+0000","last_unstale":"2024-02-19T21:13:08.985747+0000","last_undegraded":"2024-02-19T21:13:08.985747+0000","last_fullsized":"2024-02-19T21:13:08.985747+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:05.664425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:05.664425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":518,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.490024+0000","last_change":"2024-02-19T21:13:06.446599+0000","last_active":"2024-02-19T21:13:08.490024+0000","last_peered":"2024-02-19T21:13:08.490024+0000","last_clean":"2024-02-19T21:13:08.490024+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:13:08.490024+0000","last_undegraded":"2024-02-19T21:13:08.490024+0000","last_fullsized":"2024-02-19T21:13:08.490024+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:06.446523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.446523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"459'26","reported_seq":597,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350318+0000","last_change":"2024-02-19T21:12:09.420794+0000","last_active":"2024-02-19T21:13:08.350318+0000","last_peered":"2024-02-19T21:13:08.350318+0000","last_clean":"2024-02-19T21:13:08.350318+0000","last_became_active":"2024-02-19T21:12:09.352705+0000","last_became_peered":"2024-02-19T21:12:09.352705+0000","last_unstale":"2024-02-19T21:13:08.350318+0000","last_undegraded":"2024-02-19T21:13:08.350318+0000","last_fullsized":"2024-02-19T21:13:08.350318+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'26","last_scrub_stamp":"2024-02-19T21:11:44.774856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:44.774856+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":544,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.489673+0000","last_change":"2024-02-19T21:12:09.311418+0000","last_active":"2024-02-19T21:13:08.489673+0000","last_peered":"2024-02-19T21:13:08.489673+0000","last_clean":"2024-02-19T21:13:08.489673+0000","last_became_active":"2024-02-19T21:12:09.310949+0000","last_became_peered":"2024-02-19T21:12:09.310949+0000","last_unstale":"2024-02-19T21:13:08.489673+0000","last_undegraded":"2024-02-19T21:13:08.489673+0000","last_fullsized":"2024-02-19T21:13:08.489673+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.747108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.747108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":555,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.985727+0000","last_change":"2024-02-19T21:11:48.359614+0000","last_active":"2024-02-19T21:13:08.985727+0000","last_peered":"2024-02-19T21:13:08.985727+0000","last_clean":"2024-02-19T21:13:08.985727+0000","last_became_active":"2024-02-19T21:11:48.359271+0000","last_became_peered":"2024-02-19T21:11:48.359271+0000","last_unstale":"2024-02-19T21:13:08.985727+0000","last_undegraded":"2024-02-19T21:13:08.985727+0000","last_fullsized":"2024-02-19T21:13:08.985727+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.081993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.081993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":278,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351124+0000","last_change":"2024-02-19T21:12:55.248327+0000","last_active":"2024-02-19T21:13:08.351124+0000","last_peered":"2024-02-19T21:13:08.351124+0000","last_clean":"2024-02-19T21:13:08.351124+0000","last_became_active":"2024-02-19T21:12:09.347643+0000","last_became_peered":"2024-02-19T21:12:09.347643+0000","last_unstale":"2024-02-19T21:13:08.351124+0000","last_undegraded":"2024-02-19T21:13:08.351124+0000","last_fullsized":"2024-02-19T21:13:08.351124+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:55.248259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.248259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":548,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.989315+0000","last_change":"2024-02-19T21:10:41.315872+0000","last_active":"2024-02-19T21:13:08.989315+0000","last_peered":"2024-02-19T21:13:08.989315+0000","last_clean":"2024-02-19T21:13:08.989315+0000","last_became_active":"2024-02-19T21:10:41.315397+0000","last_became_peered":"2024-02-19T21:10:41.315397+0000","last_unstale":"2024-02-19T21:13:08.989315+0000","last_undegraded":"2024-02-19T21:13:08.989315+0000","last_fullsized":"2024-02-19T21:13:08.989315+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:10.148733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:10.148733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":476,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.984993+0000","last_change":"2024-02-19T21:11:59.333800+0000","last_active":"2024-02-19T21:13:08.984993+0000","last_peered":"2024-02-19T21:13:08.984993+0000","last_clean":"2024-02-19T21:13:08.984993+0000","last_became_active":"2024-02-19T21:11:59.333572+0000","last_became_peered":"2024-02-19T21:11:59.333572+0000","last_unstale":"2024-02-19T21:13:08.984993+0000","last_undegraded":"2024-02-19T21:13:08.984993+0000","last_fullsized":"2024-02-19T21:13:08.984993+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_clean_scrub_stamp":"2024-02-19T21:10:20.204880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":521,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.492358+0000","last_change":"2024-02-19T21:13:00.111238+0000","last_active":"2024-02-19T21:13:08.492358+0000","last_peered":"2024-02-19T21:13:08.492358+0000","last_clean":"2024-02-19T21:13:08.492358+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:13:08.492358+0000","last_undegraded":"2024-02-19T21:13:08.492358+0000","last_fullsized":"2024-02-19T21:13:08.492358+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.111171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.111171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"459'22","reported_seq":561,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.496033+0000","last_change":"2024-02-19T21:11:59.988716+0000","last_active":"2024-02-19T21:13:08.496033+0000","last_peered":"2024-02-19T21:13:08.496033+0000","last_clean":"2024-02-19T21:13:08.496033+0000","last_became_active":"2024-02-19T21:11:59.343010+0000","last_became_peered":"2024-02-19T21:11:59.343010+0000","last_unstale":"2024-02-19T21:13:08.496033+0000","last_undegraded":"2024-02-19T21:13:08.496033+0000","last_fullsized":"2024-02-19T21:13:08.496033+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:11:00.191206+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.191206+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":579,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.492560+0000","last_change":"2024-02-19T21:11:59.352625+0000","last_active":"2024-02-19T21:13:08.492560+0000","last_peered":"2024-02-19T21:13:08.492560+0000","last_clean":"2024-02-19T21:13:08.492560+0000","last_became_active":"2024-02-19T21:11:59.352283+0000","last_became_peered":"2024-02-19T21:11:59.352283+0000","last_unstale":"2024-02-19T21:13:08.492560+0000","last_undegraded":"2024-02-19T21:13:08.492560+0000","last_fullsized":"2024-02-19T21:13:08.492560+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.955872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.955872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":1243,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:09.434643+0000","last_change":"2024-02-19T21:13:08.986433+0000","last_active":"2024-02-19T21:13:09.434643+0000","last_peered":"2024-02-19T21:13:09.434643+0000","last_clean":"2024-02-19T21:13:09.434643+0000","last_became_active":"2024-02-19T21:12:09.340570+0000","last_became_peered":"2024-02-19T21:12:09.340570+0000","last_unstale":"2024-02-19T21:13:09.434643+0000","last_undegraded":"2024-02-19T21:13:09.434643+0000","last_fullsized":"2024-02-19T21:13:09.434643+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:13:08.986361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:08.986361+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":704,"num_read_kb":699,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":543,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.491477+0000","last_change":"2024-02-19T21:13:04.428341+0000","last_active":"2024-02-19T21:13:08.491477+0000","last_peered":"2024-02-19T21:13:08.491477+0000","last_clean":"2024-02-19T21:13:08.491477+0000","last_became_active":"2024-02-19T21:12:09.323625+0000","last_became_peered":"2024-02-19T21:12:09.323625+0000","last_unstale":"2024-02-19T21:13:08.491477+0000","last_undegraded":"2024-02-19T21:13:08.491477+0000","last_fullsized":"2024-02-19T21:13:08.491477+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:04.428236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:04.428236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":474,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.983573+0000","last_change":"2024-02-19T21:11:59.352488+0000","last_active":"2024-02-19T21:13:08.983573+0000","last_peered":"2024-02-19T21:13:08.983573+0000","last_clean":"2024-02-19T21:13:08.983573+0000","last_became_active":"2024-02-19T21:11:59.351887+0000","last_became_peered":"2024-02-19T21:11:59.351887+0000","last_unstale":"2024-02-19T21:13:08.983573+0000","last_undegraded":"2024-02-19T21:13:08.983573+0000","last_fullsized":"2024-02-19T21:13:08.983573+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.947089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":384,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.983726+0000","last_change":"2024-02-19T21:12:09.369780+0000","last_active":"2024-02-19T21:13:08.983726+0000","last_peered":"2024-02-19T21:13:08.983726+0000","last_clean":"2024-02-19T21:13:08.983726+0000","last_became_active":"2024-02-19T21:12:09.369431+0000","last_became_peered":"2024-02-19T21:12:09.369431+0000","last_unstale":"2024-02-19T21:13:08.983726+0000","last_undegraded":"2024-02-19T21:13:08.983726+0000","last_fullsized":"2024-02-19T21:13:08.983726+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.945639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.945639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"459'42","reported_seq":786,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-02-19T21:13:09.494606+0000","last_change":"2024-02-19T21:10:01.545972+0000","last_active":"2024-02-19T21:13:09.494606+0000","last_peered":"2024-02-19T21:13:09.494606+0000","last_clean":"2024-02-19T21:13:09.494606+0000","last_became_active":"2024-02-19T21:08:04.266322+0000","last_became_peered":"2024-02-19T21:08:04.266322+0000","last_unstale":"2024-02-19T21:13:09.494606+0000","last_undegraded":"2024-02-19T21:13:09.494606+0000","last_fullsized":"2024-02-19T21:13:09.494606+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'30","last_scrub_stamp":"2024-02-19T21:10:01.545897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:01.545897+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":564,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.986874+0000","last_change":"2024-02-19T21:12:09.328219+0000","last_active":"2024-02-19T21:13:08.986874+0000","last_peered":"2024-02-19T21:13:08.986874+0000","last_clean":"2024-02-19T21:13:08.986874+0000","last_became_active":"2024-02-19T21:12:09.326579+0000","last_became_peered":"2024-02-19T21:12:09.326579+0000","last_unstale":"2024-02-19T21:13:08.986874+0000","last_undegraded":"2024-02-19T21:13:08.986874+0000","last_fullsized":"2024-02-19T21:13:08.986874+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.089715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.089715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":282,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351779+0000","last_change":"2024-02-19T21:11:31.629134+0000","last_active":"2024-02-19T21:13:08.351779+0000","last_peered":"2024-02-19T21:13:08.351779+0000","last_clean":"2024-02-19T21:13:08.351779+0000","last_became_active":"2024-02-19T21:10:41.305829+0000","last_became_peered":"2024-02-19T21:10:41.305829+0000","last_unstale":"2024-02-19T21:13:08.351779+0000","last_undegraded":"2024-02-19T21:13:08.351779+0000","last_fullsized":"2024-02-19T21:13:08.351779+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:31.629057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.629057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":477,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.490265+0000","last_change":"2024-02-19T21:11:59.333878+0000","last_active":"2024-02-19T21:13:08.490265+0000","last_peered":"2024-02-19T21:13:08.490265+0000","last_clean":"2024-02-19T21:13:08.490265+0000","last_became_active":"2024-02-19T21:11:59.331967+0000","last_became_peered":"2024-02-19T21:11:59.331967+0000","last_unstale":"2024-02-19T21:13:08.490265+0000","last_undegraded":"2024-02-19T21:13:08.490265+0000","last_fullsized":"2024-02-19T21:13:08.490265+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:54.142898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.142898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"493'12","reported_seq":432,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-02-19T21:13:09.486014+0000","last_change":"2024-02-19T21:12:09.433235+0000","last_active":"2024-02-19T21:13:09.486014+0000","last_peered":"2024-02-19T21:13:09.486014+0000","last_clean":"2024-02-19T21:13:09.486014+0000","last_became_active":"2024-02-19T21:12:09.326372+0000","last_became_peered":"2024-02-19T21:12:09.326372+0000","last_unstale":"2024-02-19T21:13:09.486014+0000","last_undegraded":"2024-02-19T21:13:09.486014+0000","last_fullsized":"2024-02-19T21:13:09.486014+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'10","last_scrub_stamp":"2024-02-19T21:11:41.775418+0000","last_deep_scrub":"471'10","last_deep_scrub_stamp":"2024-02-19T21:11:41.775418+0000","last_clean_scrub_stamp":"2024-02-19T21:11:41.775418+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"459'20","reported_seq":587,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.982033+0000","last_change":"2024-02-19T21:12:09.030294+0000","last_active":"2024-02-19T21:13:08.982033+0000","last_peered":"2024-02-19T21:13:08.982033+0000","last_clean":"2024-02-19T21:13:08.982033+0000","last_became_active":"2024-02-19T21:10:41.296275+0000","last_became_peered":"2024-02-19T21:10:41.296275+0000","last_unstale":"2024-02-19T21:13:08.982033+0000","last_undegraded":"2024-02-19T21:13:08.982033+0000","last_fullsized":"2024-02-19T21:13:08.982033+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:12:09.030215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:09.030215+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":580,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.984194+0000","last_change":"2024-02-19T21:11:59.351637+0000","last_active":"2024-02-19T21:13:08.984194+0000","last_peered":"2024-02-19T21:13:08.984194+0000","last_clean":"2024-02-19T21:13:08.984194+0000","last_became_active":"2024-02-19T21:11:59.351241+0000","last_became_peered":"2024-02-19T21:11:59.351241+0000","last_unstale":"2024-02-19T21:13:08.984194+0000","last_undegraded":"2024-02-19T21:13:08.984194+0000","last_fullsized":"2024-02-19T21:13:08.984194+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:21.732778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:10:21.732778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":278,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349219+0000","last_change":"2024-02-19T21:13:07.221665+0000","last_active":"2024-02-19T21:13:08.349219+0000","last_peered":"2024-02-19T21:13:08.349219+0000","last_clean":"2024-02-19T21:13:08.349219+0000","last_became_active":"2024-02-19T21:11:59.335798+0000","last_became_peered":"2024-02-19T21:11:59.335798+0000","last_unstale":"2024-02-19T21:13:08.349219+0000","last_undegraded":"2024-02-19T21:13:08.349219+0000","last_fullsized":"2024-02-19T21:13:08.349219+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:07.221594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.221594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":273,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351894+0000","last_change":"2024-02-19T21:12:09.332210+0000","last_active":"2024-02-19T21:13:08.351894+0000","last_peered":"2024-02-19T21:13:08.351894+0000","last_clean":"2024-02-19T21:13:08.351894+0000","last_became_active":"2024-02-19T21:12:09.331195+0000","last_became_peered":"2024-02-19T21:12:09.331195+0000","last_unstale":"2024-02-19T21:13:08.351894+0000","last_undegraded":"2024-02-19T21:13:08.351894+0000","last_fullsized":"2024-02-19T21:13:08.351894+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.604025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.604025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":530,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:09.184050+0000","last_change":"2024-02-19T21:13:09.184029+0000","last_active":"2024-02-19T21:13:09.184050+0000","last_peered":"2024-02-19T21:13:09.184050+0000","last_clean":"2024-02-19T21:13:09.184050+0000","last_became_active":"2024-02-19T21:11:59.333275+0000","last_became_peered":"2024-02-19T21:11:59.333275+0000","last_unstale":"2024-02-19T21:13:09.184050+0000","last_undegraded":"2024-02-19T21:13:09.184050+0000","last_fullsized":"2024-02-19T21:13:09.184050+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.183962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.183962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"459'23","reported_seq":154,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.348940+0000","last_change":"2024-02-19T21:12:18.340005+0000","last_active":"2024-02-19T21:13:08.348940+0000","last_peered":"2024-02-19T21:13:08.348940+0000","last_clean":"2024-02-19T21:13:08.348940+0000","last_became_active":"2024-02-19T21:10:41.287134+0000","last_became_peered":"2024-02-19T21:10:41.287134+0000","last_unstale":"2024-02-19T21:13:08.348940+0000","last_undegraded":"2024-02-19T21:13:08.348940+0000","last_fullsized":"2024-02-19T21:13:08.348940+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'23","last_scrub_stamp":"2024-02-19T21:12:18.339929+0000","last_deep_scrub":"420'17","last_deep_scrub_stamp":"2024-02-19T21:10:19.722303+0000","last_clean_scrub_stamp":"2024-02-19T21:12:18.339929+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":154,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351166+0000","last_change":"2024-02-19T21:11:59.342221+0000","last_active":"2024-02-19T21:13:08.351166+0000","last_peered":"2024-02-19T21:13:08.351166+0000","last_clean":"2024-02-19T21:13:08.351166+0000","last_became_active":"2024-02-19T21:11:59.323514+0000","last_became_peered":"2024-02-19T21:11:59.323514+0000","last_unstale":"2024-02-19T21:13:08.351166+0000","last_undegraded":"2024-02-19T21:13:08.351166+0000","last_fullsized":"2024-02-19T21:13:08.351166+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:10:58.333168+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.333168+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":269,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350403+0000","last_change":"2024-02-19T21:13:00.166548+0000","last_active":"2024-02-19T21:13:08.350403+0000","last_peered":"2024-02-19T21:13:08.350403+0000","last_clean":"2024-02-19T21:13:08.350403+0000","last_became_active":"2024-02-19T21:08:04.264593+0000","last_became_peered":"2024-02-19T21:08:04.264593+0000","last_unstale":"2024-02-19T21:13:08.350403+0000","last_undegraded":"2024-02-19T21:13:08.350403+0000","last_fullsized":"2024-02-19T21:13:08.350403+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.166479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.166479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":468,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.491897+0000","last_change":"2024-02-19T21:11:03.195680+0000","last_active":"2024-02-19T21:13:08.491897+0000","last_peered":"2024-02-19T21:13:08.491897+0000","last_clean":"2024-02-19T21:13:08.491897+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:13:08.491897+0000","last_undegraded":"2024-02-19T21:13:08.491897+0000","last_fullsized":"2024-02-19T21:13:08.491897+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.195616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.195616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"459'15","reported_seq":304,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351461+0000","last_change":"2024-02-19T21:11:59.364955+0000","last_active":"2024-02-19T21:13:08.351461+0000","last_peered":"2024-02-19T21:13:08.351461+0000","last_clean":"2024-02-19T21:13:08.351461+0000","last_became_active":"2024-02-19T21:11:59.336293+0000","last_became_peered":"2024-02-19T21:11:59.336293+0000","last_unstale":"2024-02-19T21:13:08.351461+0000","last_undegraded":"2024-02-19T21:13:08.351461+0000","last_fullsized":"2024-02-19T21:13:08.351461+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:10:22.459807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:22.459807+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":529,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.490000+0000","last_change":"2024-02-19T21:11:59.334071+0000","last_active":"2024-02-19T21:13:08.490000+0000","last_peered":"2024-02-19T21:13:08.490000+0000","last_clean":"2024-02-19T21:13:08.490000+0000","last_became_active":"2024-02-19T21:11:59.333684+0000","last_became_peered":"2024-02-19T21:11:59.333684+0000","last_unstale":"2024-02-19T21:13:08.490000+0000","last_undegraded":"2024-02-19T21:13:08.490000+0000","last_fullsized":"2024-02-19T21:13:08.490000+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.244285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.244285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":264,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350006+0000","last_change":"2024-02-19T21:10:53.703797+0000","last_active":"2024-02-19T21:13:08.350006+0000","last_peered":"2024-02-19T21:13:08.350006+0000","last_clean":"2024-02-19T21:13:08.350006+0000","last_became_active":"2024-02-19T21:08:04.264972+0000","last_became_peered":"2024-02-19T21:08:04.264972+0000","last_unstale":"2024-02-19T21:13:08.350006+0000","last_undegraded":"2024-02-19T21:13:08.350006+0000","last_fullsized":"2024-02-19T21:13:08.350006+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:53.703727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.703727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":537,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.491156+0000","last_change":"2024-02-19T21:12:06.261731+0000","last_active":"2024-02-19T21:13:08.491156+0000","last_peered":"2024-02-19T21:13:08.491156+0000","last_clean":"2024-02-19T21:13:08.491156+0000","last_became_active":"2024-02-19T21:10:41.314114+0000","last_became_peered":"2024-02-19T21:10:41.314114+0000","last_unstale":"2024-02-19T21:13:08.491156+0000","last_undegraded":"2024-02-19T21:13:08.491156+0000","last_fullsized":"2024-02-19T21:13:08.491156+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.261670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.261670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":133,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350392+0000","last_change":"2024-02-19T21:12:08.430936+0000","last_active":"2024-02-19T21:13:08.350392+0000","last_peered":"2024-02-19T21:13:08.350392+0000","last_clean":"2024-02-19T21:13:08.350392+0000","last_became_active":"2024-02-19T21:10:41.300574+0000","last_became_peered":"2024-02-19T21:10:41.300574+0000","last_unstale":"2024-02-19T21:13:08.350392+0000","last_undegraded":"2024-02-19T21:13:08.350392+0000","last_fullsized":"2024-02-19T21:13:08.350392+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:08.430874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:08.430874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"459'15","reported_seq":151,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.495788+0000","last_change":"2024-02-19T21:10:53.101489+0000","last_active":"2024-02-19T21:13:08.495788+0000","last_peered":"2024-02-19T21:13:08.495788+0000","last_clean":"2024-02-19T21:13:08.495788+0000","last_became_active":"2024-02-19T21:10:40.297867+0000","last_became_peered":"2024-02-19T21:10:40.297867+0000","last_unstale":"2024-02-19T21:13:08.495788+0000","last_undegraded":"2024-02-19T21:13:08.495788+0000","last_fullsized":"2024-02-19T21:13:08.495788+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:10:53.101422+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.101422+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":479,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.348001+0000","last_change":"2024-02-19T21:12:09.369726+0000","last_active":"2024-02-19T21:13:08.348001+0000","last_peered":"2024-02-19T21:13:08.348001+0000","last_clean":"2024-02-19T21:13:08.348001+0000","last_became_active":"2024-02-19T21:12:09.369407+0000","last_became_peered":"2024-02-19T21:12:09.369407+0000","last_unstale":"2024-02-19T21:13:08.348001+0000","last_undegraded":"2024-02-19T21:13:08.348001+0000","last_fullsized":"2024-02-19T21:13:08.348001+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:31.670026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.670026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":535,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.986656+0000","last_change":"2024-02-19T21:12:06.098778+0000","last_active":"2024-02-19T21:13:08.986656+0000","last_peered":"2024-02-19T21:13:08.986656+0000","last_clean":"2024-02-19T21:13:08.986656+0000","last_became_active":"2024-02-19T21:08:04.265573+0000","last_became_peered":"2024-02-19T21:08:04.265573+0000","last_unstale":"2024-02-19T21:13:08.986656+0000","last_undegraded":"2024-02-19T21:13:08.986656+0000","last_fullsized":"2024-02-19T21:13:08.986656+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.098703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.098703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":545,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.491135+0000","last_change":"2024-02-19T21:12:17.452030+0000","last_active":"2024-02-19T21:13:08.491135+0000","last_peered":"2024-02-19T21:13:08.491135+0000","last_clean":"2024-02-19T21:13:08.491135+0000","last_became_active":"2024-02-19T21:12:09.341406+0000","last_became_peered":"2024-02-19T21:12:09.341406+0000","last_unstale":"2024-02-19T21:13:08.491135+0000","last_undegraded":"2024-02-19T21:13:08.491135+0000","last_fullsized":"2024-02-19T21:13:08.491135+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:17.451964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:17.451964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":494,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.985189+0000","last_change":"2024-02-19T21:12:47.428132+0000","last_active":"2024-02-19T21:13:08.985189+0000","last_peered":"2024-02-19T21:13:08.985189+0000","last_clean":"2024-02-19T21:13:08.985189+0000","last_became_active":"2024-02-19T21:12:09.349695+0000","last_became_peered":"2024-02-19T21:12:09.349695+0000","last_unstale":"2024-02-19T21:13:08.985189+0000","last_undegraded":"2024-02-19T21:13:08.985189+0000","last_fullsized":"2024-02-19T21:13:08.985189+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:47.428055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:47.428055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"459'20","reported_seq":291,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351884+0000","last_change":"2024-02-19T21:10:59.685223+0000","last_active":"2024-02-19T21:13:08.351884+0000","last_peered":"2024-02-19T21:13:08.351884+0000","last_clean":"2024-02-19T21:13:08.351884+0000","last_became_active":"2024-02-19T21:08:04.267216+0000","last_became_peered":"2024-02-19T21:08:04.267216+0000","last_unstale":"2024-02-19T21:13:08.351884+0000","last_undegraded":"2024-02-19T21:13:08.351884+0000","last_fullsized":"2024-02-19T21:13:08.351884+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:10:59.685150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.685150+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":537,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.986675+0000","last_change":"2024-02-19T21:09:40.986717+0000","last_active":"2024-02-19T21:13:08.986675+0000","last_peered":"2024-02-19T21:13:08.986675+0000","last_clean":"2024-02-19T21:13:08.986675+0000","last_became_active":"2024-02-19T21:08:04.266728+0000","last_became_peered":"2024-02-19T21:08:04.266728+0000","last_unstale":"2024-02-19T21:13:08.986675+0000","last_undegraded":"2024-02-19T21:13:08.986675+0000","last_fullsized":"2024-02-19T21:13:08.986675+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.986635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"433'9","reported_seq":679,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-02-19T21:13:09.484336+0000","last_change":"2024-02-19T21:13:00.428710+0000","last_active":"2024-02-19T21:13:09.484336+0000","last_peered":"2024-02-19T21:13:09.484336+0000","last_clean":"2024-02-19T21:13:09.484336+0000","last_became_active":"2024-02-19T21:10:40.310303+0000","last_became_peered":"2024-02-19T21:10:40.310303+0000","last_unstale":"2024-02-19T21:13:09.484336+0000","last_undegraded":"2024-02-19T21:13:09.484336+0000","last_fullsized":"2024-02-19T21:13:09.484336+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'9","last_scrub_stamp":"2024-02-19T21:13:00.428581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.428581+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":397,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.493851+0000","last_change":"2024-02-19T21:11:59.843545+0000","last_active":"2024-02-19T21:13:08.493851+0000","last_peered":"2024-02-19T21:13:08.493851+0000","last_clean":"2024-02-19T21:13:08.493851+0000","last_became_active":"2024-02-19T21:11:59.323539+0000","last_became_peered":"2024-02-19T21:11:59.323539+0000","last_unstale":"2024-02-19T21:13:08.493851+0000","last_undegraded":"2024-02-19T21:13:08.493851+0000","last_fullsized":"2024-02-19T21:13:08.493851+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:11:31.500511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.500511+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":349,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.489803+0000","last_change":"2024-02-19T21:12:09.345187+0000","last_active":"2024-02-19T21:13:08.489803+0000","last_peered":"2024-02-19T21:13:08.489803+0000","last_clean":"2024-02-19T21:13:08.489803+0000","last_became_active":"2024-02-19T21:12:09.344609+0000","last_became_peered":"2024-02-19T21:12:09.344609+0000","last_unstale":"2024-02-19T21:13:08.489803+0000","last_undegraded":"2024-02-19T21:13:08.489803+0000","last_fullsized":"2024-02-19T21:13:08.489803+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.378925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:55.202463+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.378925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"459'15","reported_seq":580,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.986606+0000","last_change":"2024-02-19T21:10:50.053312+0000","last_active":"2024-02-19T21:13:08.986606+0000","last_peered":"2024-02-19T21:13:08.986606+0000","last_clean":"2024-02-19T21:13:08.986606+0000","last_became_active":"2024-02-19T21:08:04.266103+0000","last_became_peered":"2024-02-19T21:08:04.266103+0000","last_unstale":"2024-02-19T21:13:08.986606+0000","last_undegraded":"2024-02-19T21:13:08.986606+0000","last_fullsized":"2024-02-19T21:13:08.986606+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:10:50.053229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.053229+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":147,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349648+0000","last_change":"2024-02-19T21:13:07.507018+0000","last_active":"2024-02-19T21:13:08.349648+0000","last_peered":"2024-02-19T21:13:08.349648+0000","last_clean":"2024-02-19T21:13:08.349648+0000","last_became_active":"2024-02-19T21:11:59.324861+0000","last_became_peered":"2024-02-19T21:11:59.324861+0000","last_unstale":"2024-02-19T21:13:08.349648+0000","last_undegraded":"2024-02-19T21:13:08.349648+0000","last_fullsized":"2024-02-19T21:13:08.349648+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:07.506946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.506946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":335,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-02-19T21:13:09.486714+0000","last_change":"2024-02-19T21:12:17.345023+0000","last_active":"2024-02-19T21:13:09.486714+0000","last_peered":"2024-02-19T21:13:09.486714+0000","last_clean":"2024-02-19T21:13:09.486714+0000","last_became_active":"2024-02-19T21:08:04.256045+0000","last_became_peered":"2024-02-19T21:08:04.256045+0000","last_unstale":"2024-02-19T21:13:09.486714+0000","last_undegraded":"2024-02-19T21:13:09.486714+0000","last_fullsized":"2024-02-19T21:13:09.486714+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'10","last_scrub_stamp":"2024-02-19T21:12:17.344916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:17.344916+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":133,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351213+0000","last_change":"2024-02-19T21:12:05.445686+0000","last_active":"2024-02-19T21:13:08.351213+0000","last_peered":"2024-02-19T21:13:08.351213+0000","last_clean":"2024-02-19T21:13:08.351213+0000","last_became_active":"2024-02-19T21:10:41.310279+0000","last_became_peered":"2024-02-19T21:10:41.310279+0000","last_unstale":"2024-02-19T21:13:08.351213+0000","last_undegraded":"2024-02-19T21:13:08.351213+0000","last_fullsized":"2024-02-19T21:13:08.351213+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:05.445575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_clean_scrub_stamp":"2024-02-19T21:12:05.445575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":525,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.491424+0000","last_change":"2024-02-19T21:12:49.142231+0000","last_active":"2024-02-19T21:13:08.491424+0000","last_peered":"2024-02-19T21:13:08.491424+0000","last_clean":"2024-02-19T21:13:08.491424+0000","last_became_active":"2024-02-19T21:12:09.339844+0000","last_became_peered":"2024-02-19T21:12:09.339844+0000","last_unstale":"2024-02-19T21:13:08.491424+0000","last_undegraded":"2024-02-19T21:13:08.491424+0000","last_fullsized":"2024-02-19T21:13:08.491424+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-02-19T21:12:49.142158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:49.142158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":15,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"459'15","reported_seq":280,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.981353+0000","last_change":"2024-02-19T21:13:06.958616+0000","last_active":"2024-02-19T21:13:08.981353+0000","last_peered":"2024-02-19T21:13:08.981353+0000","last_clean":"2024-02-19T21:13:08.981353+0000","last_became_active":"2024-02-19T21:12:09.341038+0000","last_became_peered":"2024-02-19T21:12:09.341038+0000","last_unstale":"2024-02-19T21:13:08.981353+0000","last_undegraded":"2024-02-19T21:13:08.981353+0000","last_fullsized":"2024-02-19T21:13:08.981353+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:13:06.958518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.958518+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":348,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349942+0000","last_change":"2024-02-19T21:12:09.333684+0000","last_active":"2024-02-19T21:13:08.349942+0000","last_peered":"2024-02-19T21:13:08.349942+0000","last_clean":"2024-02-19T21:13:08.349942+0000","last_became_active":"2024-02-19T21:12:09.333350+0000","last_became_peered":"2024-02-19T21:12:09.333350+0000","last_unstale":"2024-02-19T21:13:08.349942+0000","last_undegraded":"2024-02-19T21:13:08.349942+0000","last_fullsized":"2024-02-19T21:13:08.349942+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.422323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.422323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":278,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350515+0000","last_change":"2024-02-19T21:11:59.349259+0000","last_active":"2024-02-19T21:13:08.350515+0000","last_peered":"2024-02-19T21:13:08.350515+0000","last_clean":"2024-02-19T21:13:08.350515+0000","last_became_active":"2024-02-19T21:11:59.348485+0000","last_became_peered":"2024-02-19T21:11:59.348485+0000","last_unstale":"2024-02-19T21:13:08.350515+0000","last_undegraded":"2024-02-19T21:13:08.350515+0000","last_fullsized":"2024-02-19T21:13:08.350515+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:45.404242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_clean_scrub_stamp":"2024-02-19T21:11:45.404242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":363,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.985234+0000","last_change":"2024-02-19T21:12:09.327887+0000","last_active":"2024-02-19T21:13:08.985234+0000","last_peered":"2024-02-19T21:13:08.985234+0000","last_clean":"2024-02-19T21:13:08.985234+0000","last_became_active":"2024-02-19T21:12:09.327390+0000","last_became_peered":"2024-02-19T21:12:09.327390+0000","last_unstale":"2024-02-19T21:13:08.985234+0000","last_undegraded":"2024-02-19T21:13:08.985234+0000","last_fullsized":"2024-02-19T21:13:08.985234+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:38.940935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:38.940935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":467,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.986566+0000","last_change":"2024-02-19T21:12:09.330102+0000","last_active":"2024-02-19T21:13:08.986566+0000","last_peered":"2024-02-19T21:13:08.986566+0000","last_clean":"2024-02-19T21:13:08.986566+0000","last_became_active":"2024-02-19T21:12:09.329246+0000","last_became_peered":"2024-02-19T21:12:09.329246+0000","last_unstale":"2024-02-19T21:13:08.986566+0000","last_undegraded":"2024-02-19T21:13:08.986566+0000","last_fullsized":"2024-02-19T21:13:08.986566+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.493206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.493206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"459'19","reported_seq":178,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349814+0000","last_change":"2024-02-19T21:12:10.570353+0000","last_active":"2024-02-19T21:13:08.349814+0000","last_peered":"2024-02-19T21:13:08.349814+0000","last_clean":"2024-02-19T21:13:08.349814+0000","last_became_active":"2024-02-19T21:12:09.399929+0000","last_became_peered":"2024-02-19T21:12:09.399929+0000","last_unstale":"2024-02-19T21:13:08.349814+0000","last_undegraded":"2024-02-19T21:13:08.349814+0000","last_fullsized":"2024-02-19T21:13:08.349814+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:11:32.351880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:32.351880+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":540,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.983854+0000","last_change":"2024-02-19T21:13:00.874389+0000","last_active":"2024-02-19T21:13:08.983854+0000","last_peered":"2024-02-19T21:13:08.983854+0000","last_clean":"2024-02-19T21:13:08.983854+0000","last_became_active":"2024-02-19T21:12:09.342120+0000","last_became_peered":"2024-02-19T21:12:09.342120+0000","last_unstale":"2024-02-19T21:13:08.983854+0000","last_undegraded":"2024-02-19T21:13:08.983854+0000","last_fullsized":"2024-02-19T21:13:08.983854+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.874318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.874318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":536,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.982081+0000","last_change":"2024-02-19T21:12:54.957657+0000","last_active":"2024-02-19T21:13:08.982081+0000","last_peered":"2024-02-19T21:13:08.982081+0000","last_clean":"2024-02-19T21:13:08.982081+0000","last_became_active":"2024-02-19T21:08:04.268847+0000","last_became_peered":"2024-02-19T21:08:04.268847+0000","last_unstale":"2024-02-19T21:13:08.982081+0000","last_undegraded":"2024-02-19T21:13:08.982081+0000","last_fullsized":"2024-02-19T21:13:08.982081+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:54.957585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:54.957585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":273,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351697+0000","last_change":"2024-02-19T21:11:59.341209+0000","last_active":"2024-02-19T21:13:08.351697+0000","last_peered":"2024-02-19T21:13:08.351697+0000","last_clean":"2024-02-19T21:13:08.351697+0000","last_became_active":"2024-02-19T21:11:59.340742+0000","last_became_peered":"2024-02-19T21:11:59.340742+0000","last_unstale":"2024-02-19T21:13:08.351697+0000","last_undegraded":"2024-02-19T21:13:08.351697+0000","last_fullsized":"2024-02-19T21:13:08.351697+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:37.477995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:37.477995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":273,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350131+0000","last_change":"2024-02-19T21:11:59.338599+0000","last_active":"2024-02-19T21:13:08.350131+0000","last_peered":"2024-02-19T21:13:08.350131+0000","last_clean":"2024-02-19T21:13:08.350131+0000","last_became_active":"2024-02-19T21:11:59.337764+0000","last_became_peered":"2024-02-19T21:11:59.337764+0000","last_unstale":"2024-02-19T21:13:08.350131+0000","last_undegraded":"2024-02-19T21:13:08.350131+0000","last_fullsized":"2024-02-19T21:13:08.350131+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.599474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.599474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"459'8","reported_seq":154,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.984421+0000","last_change":"2024-02-19T21:12:09.954000+0000","last_active":"2024-02-19T21:13:08.984421+0000","last_peered":"2024-02-19T21:13:08.984421+0000","last_clean":"2024-02-19T21:13:08.984421+0000","last_became_active":"2024-02-19T21:12:09.340298+0000","last_became_peered":"2024-02-19T21:12:09.340298+0000","last_unstale":"2024-02-19T21:13:08.984421+0000","last_undegraded":"2024-02-19T21:13:08.984421+0000","last_fullsized":"2024-02-19T21:13:08.984421+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'4","last_scrub_stamp":"2024-02-19T21:09:28.712037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:09:28.712037+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":528,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.986885+0000","last_change":"2024-02-19T21:12:09.330300+0000","last_active":"2024-02-19T21:13:08.986885+0000","last_peered":"2024-02-19T21:13:08.986885+0000","last_clean":"2024-02-19T21:13:08.986885+0000","last_became_active":"2024-02-19T21:12:09.329752+0000","last_became_peered":"2024-02-19T21:12:09.329752+0000","last_unstale":"2024-02-19T21:13:08.986885+0000","last_undegraded":"2024-02-19T21:13:08.986885+0000","last_fullsized":"2024-02-19T21:13:08.986885+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:42.000850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:42.000850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":475,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.347883+0000","last_change":"2024-02-19T21:12:09.346328+0000","last_active":"2024-02-19T21:13:08.347883+0000","last_peered":"2024-02-19T21:13:08.347883+0000","last_clean":"2024-02-19T21:13:08.347883+0000","last_became_active":"2024-02-19T21:12:09.345654+0000","last_became_peered":"2024-02-19T21:12:09.345654+0000","last_unstale":"2024-02-19T21:13:08.347883+0000","last_undegraded":"2024-02-19T21:13:08.347883+0000","last_fullsized":"2024-02-19T21:13:08.347883+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.467476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.467476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":534,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.489876+0000","last_change":"2024-02-19T21:12:05.530113+0000","last_active":"2024-02-19T21:13:08.489876+0000","last_peered":"2024-02-19T21:13:08.489876+0000","last_clean":"2024-02-19T21:13:08.489876+0000","last_became_active":"2024-02-19T21:10:41.314589+0000","last_became_peered":"2024-02-19T21:10:41.314589+0000","last_unstale":"2024-02-19T21:13:08.489876+0000","last_undegraded":"2024-02-19T21:13:08.489876+0000","last_fullsized":"2024-02-19T21:13:08.489876+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:05.530039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:05.530039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":269,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351985+0000","last_change":"2024-02-19T21:12:07.259353+0000","last_active":"2024-02-19T21:13:08.351985+0000","last_peered":"2024-02-19T21:13:08.351985+0000","last_clean":"2024-02-19T21:13:08.351985+0000","last_became_active":"2024-02-19T21:08:04.266161+0000","last_became_peered":"2024-02-19T21:08:04.266161+0000","last_unstale":"2024-02-19T21:13:08.351985+0000","last_undegraded":"2024-02-19T21:13:08.351985+0000","last_fullsized":"2024-02-19T21:13:08.351985+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:07.259290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:07.259290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"459'19","reported_seq":314,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.352012+0000","last_change":"2024-02-19T21:12:09.375334+0000","last_active":"2024-02-19T21:13:08.352012+0000","last_peered":"2024-02-19T21:13:08.352012+0000","last_clean":"2024-02-19T21:13:08.352012+0000","last_became_active":"2024-02-19T21:12:09.321408+0000","last_became_peered":"2024-02-19T21:12:09.321408+0000","last_unstale":"2024-02-19T21:13:08.352012+0000","last_undegraded":"2024-02-19T21:13:08.352012+0000","last_fullsized":"2024-02-19T21:13:08.352012+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:12:03.255395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:03.255395+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":69,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.989121+0000","last_change":"2024-02-19T21:12:09.345352+0000","last_active":"2024-02-19T21:13:08.989121+0000","last_peered":"2024-02-19T21:13:08.989121+0000","last_clean":"2024-02-19T21:13:08.989121+0000","last_became_active":"2024-02-19T21:12:09.345000+0000","last_became_peered":"2024-02-19T21:12:09.345000+0000","last_unstale":"2024-02-19T21:13:08.989121+0000","last_undegraded":"2024-02-19T21:13:08.989121+0000","last_fullsized":"2024-02-19T21:13:08.989121+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:00.382946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.382946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":540,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.491003+0000","last_change":"2024-02-19T21:12:09.332889+0000","last_active":"2024-02-19T21:13:08.491003+0000","last_peered":"2024-02-19T21:13:08.491003+0000","last_clean":"2024-02-19T21:13:08.491003+0000","last_became_active":"2024-02-19T21:12:09.332485+0000","last_became_peered":"2024-02-19T21:12:09.332485+0000","last_unstale":"2024-02-19T21:13:08.491003+0000","last_undegraded":"2024-02-19T21:13:08.491003+0000","last_fullsized":"2024-02-19T21:13:08.491003+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.566208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.566208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":282,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350624+0000","last_change":"2024-02-19T21:12:09.344855+0000","last_active":"2024-02-19T21:13:08.350624+0000","last_peered":"2024-02-19T21:13:08.350624+0000","last_clean":"2024-02-19T21:13:08.350624+0000","last_became_active":"2024-02-19T21:12:09.344536+0000","last_became_peered":"2024-02-19T21:12:09.344536+0000","last_unstale":"2024-02-19T21:13:08.350624+0000","last_undegraded":"2024-02-19T21:13:08.350624+0000","last_fullsized":"2024-02-19T21:13:08.350624+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:00.547228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:00.547228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":355,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.987255+0000","last_change":"2024-02-19T21:12:09.343909+0000","last_active":"2024-02-19T21:13:08.987255+0000","last_peered":"2024-02-19T21:13:08.987255+0000","last_clean":"2024-02-19T21:13:08.987255+0000","last_became_active":"2024-02-19T21:12:09.342686+0000","last_became_peered":"2024-02-19T21:12:09.342686+0000","last_unstale":"2024-02-19T21:13:08.987255+0000","last_undegraded":"2024-02-19T21:13:08.987255+0000","last_fullsized":"2024-02-19T21:13:08.987255+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:37.718654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:11:37.718654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"459'20","reported_seq":454,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.493968+0000","last_change":"2024-02-19T21:12:48.422862+0000","last_active":"2024-02-19T21:13:08.493968+0000","last_peered":"2024-02-19T21:13:08.493968+0000","last_clean":"2024-02-19T21:13:08.493968+0000","last_became_active":"2024-02-19T21:11:59.350142+0000","last_became_peered":"2024-02-19T21:11:59.350142+0000","last_unstale":"2024-02-19T21:13:08.493968+0000","last_undegraded":"2024-02-19T21:13:08.493968+0000","last_fullsized":"2024-02-19T21:13:08.493968+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:12:48.422763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:48.422763+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":537,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.984789+0000","last_change":"2024-02-19T21:12:48.430341+0000","last_active":"2024-02-19T21:13:08.984789+0000","last_peered":"2024-02-19T21:13:08.984789+0000","last_clean":"2024-02-19T21:13:08.984789+0000","last_became_active":"2024-02-19T21:10:41.294592+0000","last_became_peered":"2024-02-19T21:10:41.294592+0000","last_unstale":"2024-02-19T21:13:08.984789+0000","last_undegraded":"2024-02-19T21:13:08.984789+0000","last_fullsized":"2024-02-19T21:13:08.984789+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:48.430223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:48.430223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"493'11","reported_seq":620,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-02-19T21:13:09.479470+0000","last_change":"2024-02-19T21:12:10.210682+0000","last_active":"2024-02-19T21:13:09.479470+0000","last_peered":"2024-02-19T21:13:09.479470+0000","last_clean":"2024-02-19T21:13:09.479470+0000","last_became_active":"2024-02-19T21:12:09.352204+0000","last_became_peered":"2024-02-19T21:12:09.352204+0000","last_unstale":"2024-02-19T21:13:09.479470+0000","last_undegraded":"2024-02-19T21:13:09.479470+0000","last_fullsized":"2024-02-19T21:13:09.479470+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'10","last_scrub_stamp":"2024-02-19T21:10:59.405417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.405417+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":518,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.490273+0000","last_change":"2024-02-19T21:11:43.546468+0000","last_active":"2024-02-19T21:13:08.490273+0000","last_peered":"2024-02-19T21:13:08.490273+0000","last_clean":"2024-02-19T21:13:08.490273+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:13:08.490273+0000","last_undegraded":"2024-02-19T21:13:08.490273+0000","last_fullsized":"2024-02-19T21:13:08.490273+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:11:43.546383+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.546383+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":471,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.492413+0000","last_change":"2024-02-19T21:11:38.369815+0000","last_active":"2024-02-19T21:13:08.492413+0000","last_peered":"2024-02-19T21:13:08.492413+0000","last_clean":"2024-02-19T21:13:08.492413+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:13:08.492413+0000","last_undegraded":"2024-02-19T21:13:08.492413+0000","last_fullsized":"2024-02-19T21:13:08.492413+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.369716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.369716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"459'19","reported_seq":537,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.989212+0000","last_change":"2024-02-19T21:12:10.390849+0000","last_active":"2024-02-19T21:13:08.989212+0000","last_peered":"2024-02-19T21:13:08.989212+0000","last_clean":"2024-02-19T21:13:08.989212+0000","last_became_active":"2024-02-19T21:12:09.350490+0000","last_became_peered":"2024-02-19T21:12:09.350490+0000","last_unstale":"2024-02-19T21:13:08.989212+0000","last_undegraded":"2024-02-19T21:13:08.989212+0000","last_fullsized":"2024-02-19T21:13:08.989212+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:10:54.435839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.435839+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":282,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351998+0000","last_change":"2024-02-19T21:12:09.333078+0000","last_active":"2024-02-19T21:13:08.351998+0000","last_peered":"2024-02-19T21:13:08.351998+0000","last_clean":"2024-02-19T21:13:08.351998+0000","last_became_active":"2024-02-19T21:12:09.332587+0000","last_became_peered":"2024-02-19T21:12:09.332587+0000","last_unstale":"2024-02-19T21:13:08.351998+0000","last_undegraded":"2024-02-19T21:13:08.351998+0000","last_fullsized":"2024-02-19T21:13:08.351998+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:32.600213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:32.600213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":358,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.494289+0000","last_change":"2024-02-19T21:11:59.334743+0000","last_active":"2024-02-19T21:13:08.494289+0000","last_peered":"2024-02-19T21:13:08.494289+0000","last_clean":"2024-02-19T21:13:08.494289+0000","last_became_active":"2024-02-19T21:11:59.334454+0000","last_became_peered":"2024-02-19T21:11:59.334454+0000","last_unstale":"2024-02-19T21:13:08.494289+0000","last_undegraded":"2024-02-19T21:13:08.494289+0000","last_fullsized":"2024-02-19T21:13:08.494289+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.692260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.692260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":529,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.983632+0000","last_change":"2024-02-19T21:12:06.991132+0000","last_active":"2024-02-19T21:13:08.983632+0000","last_peered":"2024-02-19T21:13:08.983632+0000","last_clean":"2024-02-19T21:13:08.983632+0000","last_became_active":"2024-02-19T21:10:41.312577+0000","last_became_peered":"2024-02-19T21:10:41.312577+0000","last_unstale":"2024-02-19T21:13:08.983632+0000","last_undegraded":"2024-02-19T21:13:08.983632+0000","last_fullsized":"2024-02-19T21:13:08.983632+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.991039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.991039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":483,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.983506+0000","last_change":"2024-02-19T21:10:59.782435+0000","last_active":"2024-02-19T21:13:08.983506+0000","last_peered":"2024-02-19T21:13:08.983506+0000","last_clean":"2024-02-19T21:13:08.983506+0000","last_became_active":"2024-02-19T21:10:41.311050+0000","last_became_peered":"2024-02-19T21:10:41.311050+0000","last_unstale":"2024-02-19T21:13:08.983506+0000","last_undegraded":"2024-02-19T21:13:08.983506+0000","last_fullsized":"2024-02-19T21:13:08.983506+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:59.782367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.782367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"459'34","reported_seq":618,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.491946+0000","last_change":"2024-02-19T21:10:19.109353+0000","last_active":"2024-02-19T21:13:08.491946+0000","last_peered":"2024-02-19T21:13:08.491946+0000","last_clean":"2024-02-19T21:13:08.491946+0000","last_became_active":"2024-02-19T21:08:04.248378+0000","last_became_peered":"2024-02-19T21:08:04.248378+0000","last_unstale":"2024-02-19T21:13:08.491946+0000","last_undegraded":"2024-02-19T21:13:08.491946+0000","last_fullsized":"2024-02-19T21:13:08.491946+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'22","last_scrub_stamp":"2024-02-19T21:10:19.109244+0000","last_deep_scrub":"419'22","last_deep_scrub_stamp":"2024-02-19T21:10:19.109244+0000","last_clean_scrub_stamp":"2024-02-19T21:10:19.109244+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":479,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.348057+0000","last_change":"2024-02-19T21:12:09.341126+0000","last_active":"2024-02-19T21:13:08.348057+0000","last_peered":"2024-02-19T21:13:08.348057+0000","last_clean":"2024-02-19T21:13:08.348057+0000","last_became_active":"2024-02-19T21:12:09.340803+0000","last_became_peered":"2024-02-19T21:12:09.340803+0000","last_unstale":"2024-02-19T21:13:08.348057+0000","last_undegraded":"2024-02-19T21:13:08.348057+0000","last_fullsized":"2024-02-19T21:13:08.348057+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.379335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.379335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":540,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.984057+0000","last_change":"2024-02-19T21:11:59.323076+0000","last_active":"2024-02-19T21:13:08.984057+0000","last_peered":"2024-02-19T21:13:08.984057+0000","last_clean":"2024-02-19T21:13:08.984057+0000","last_became_active":"2024-02-19T21:11:59.322735+0000","last_became_peered":"2024-02-19T21:11:59.322735+0000","last_unstale":"2024-02-19T21:13:08.984057+0000","last_undegraded":"2024-02-19T21:13:08.984057+0000","last_fullsized":"2024-02-19T21:13:08.984057+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:53.709456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.709456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":142,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351380+0000","last_change":"2024-02-19T21:12:09.370734+0000","last_active":"2024-02-19T21:13:08.351380+0000","last_peered":"2024-02-19T21:13:08.351380+0000","last_clean":"2024-02-19T21:13:08.351380+0000","last_became_active":"2024-02-19T21:12:09.370318+0000","last_became_peered":"2024-02-19T21:12:09.370318+0000","last_unstale":"2024-02-19T21:13:08.351380+0000","last_undegraded":"2024-02-19T21:13:08.351380+0000","last_fullsized":"2024-02-19T21:13:08.351380+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.335152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.335152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":349,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.982055+0000","last_change":"2024-02-19T21:12:09.340710+0000","last_active":"2024-02-19T21:13:08.982055+0000","last_peered":"2024-02-19T21:13:08.982055+0000","last_clean":"2024-02-19T21:13:08.982055+0000","last_became_active":"2024-02-19T21:12:09.340360+0000","last_became_peered":"2024-02-19T21:12:09.340360+0000","last_unstale":"2024-02-19T21:13:08.982055+0000","last_undegraded":"2024-02-19T21:13:08.982055+0000","last_fullsized":"2024-02-19T21:13:08.982055+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.342173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.342173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"459'18","reported_seq":332,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351432+0000","last_change":"2024-02-19T21:12:50.243591+0000","last_active":"2024-02-19T21:13:08.351432+0000","last_peered":"2024-02-19T21:13:08.351432+0000","last_clean":"2024-02-19T21:13:08.351432+0000","last_became_active":"2024-02-19T21:10:41.307690+0000","last_became_peered":"2024-02-19T21:10:41.307690+0000","last_unstale":"2024-02-19T21:13:08.351432+0000","last_undegraded":"2024-02-19T21:13:08.351432+0000","last_fullsized":"2024-02-19T21:13:08.351432+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'18","last_scrub_stamp":"2024-02-19T21:12:50.243498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:50.243498+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":273,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351572+0000","last_change":"2024-02-19T21:12:09.344823+0000","last_active":"2024-02-19T21:13:08.351572+0000","last_peered":"2024-02-19T21:13:08.351572+0000","last_clean":"2024-02-19T21:13:08.351572+0000","last_became_active":"2024-02-19T21:12:09.344422+0000","last_became_peered":"2024-02-19T21:12:09.344422+0000","last_unstale":"2024-02-19T21:13:08.351572+0000","last_undegraded":"2024-02-19T21:13:08.351572+0000","last_fullsized":"2024-02-19T21:13:08.351572+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:04.545988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:04.545988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":133,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350404+0000","last_change":"2024-02-19T21:11:43.369675+0000","last_active":"2024-02-19T21:13:08.350404+0000","last_peered":"2024-02-19T21:13:08.350404+0000","last_clean":"2024-02-19T21:13:08.350404+0000","last_became_active":"2024-02-19T21:10:41.296849+0000","last_became_peered":"2024-02-19T21:10:41.296849+0000","last_unstale":"2024-02-19T21:13:08.350404+0000","last_undegraded":"2024-02-19T21:13:08.350404+0000","last_fullsized":"2024-02-19T21:13:08.350404+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:43.369607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.369607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":273,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350273+0000","last_change":"2024-02-19T21:12:09.332385+0000","last_active":"2024-02-19T21:13:08.350273+0000","last_peered":"2024-02-19T21:13:08.350273+0000","last_clean":"2024-02-19T21:13:08.350273+0000","last_became_active":"2024-02-19T21:12:09.332044+0000","last_became_peered":"2024-02-19T21:12:09.332044+0000","last_unstale":"2024-02-19T21:13:08.350273+0000","last_undegraded":"2024-02-19T21:13:08.350273+0000","last_fullsized":"2024-02-19T21:13:08.350273+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:58.688166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.688166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":474,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.981936+0000","last_change":"2024-02-19T21:12:09.350012+0000","last_active":"2024-02-19T21:13:08.981936+0000","last_peered":"2024-02-19T21:13:08.981936+0000","last_clean":"2024-02-19T21:13:08.981936+0000","last_became_active":"2024-02-19T21:12:09.349709+0000","last_became_peered":"2024-02-19T21:12:09.349709+0000","last_unstale":"2024-02-19T21:13:08.981936+0000","last_undegraded":"2024-02-19T21:13:08.981936+0000","last_fullsized":"2024-02-19T21:13:08.981936+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:00.754327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.754327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"459'19","reported_seq":171,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349890+0000","last_change":"2024-02-19T21:12:55.429884+0000","last_active":"2024-02-19T21:13:08.349890+0000","last_peered":"2024-02-19T21:13:08.349890+0000","last_clean":"2024-02-19T21:13:08.349890+0000","last_became_active":"2024-02-19T21:11:59.336536+0000","last_became_peered":"2024-02-19T21:11:59.336536+0000","last_unstale":"2024-02-19T21:13:08.349890+0000","last_undegraded":"2024-02-19T21:13:08.349890+0000","last_fullsized":"2024-02-19T21:13:08.349890+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:12:55.429801+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.429801+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":517,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.489932+0000","last_change":"2024-02-19T21:11:30.472665+0000","last_active":"2024-02-19T21:13:08.489932+0000","last_peered":"2024-02-19T21:13:08.489932+0000","last_clean":"2024-02-19T21:13:08.489932+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:13:08.489932+0000","last_undegraded":"2024-02-19T21:13:08.489932+0000","last_fullsized":"2024-02-19T21:13:08.489932+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:30.472591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:30.472591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":524,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.985585+0000","last_change":"2024-02-19T21:13:06.395759+0000","last_active":"2024-02-19T21:13:08.985585+0000","last_peered":"2024-02-19T21:13:08.985585+0000","last_clean":"2024-02-19T21:13:08.985585+0000","last_became_active":"2024-02-19T21:11:59.323593+0000","last_became_peered":"2024-02-19T21:11:59.323593+0000","last_unstale":"2024-02-19T21:13:08.985585+0000","last_undegraded":"2024-02-19T21:13:08.985585+0000","last_fullsized":"2024-02-19T21:13:08.985585+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:06.395650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.395650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":352,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.981740+0000","last_change":"2024-02-19T21:11:59.352657+0000","last_active":"2024-02-19T21:13:08.981740+0000","last_peered":"2024-02-19T21:13:08.981740+0000","last_clean":"2024-02-19T21:13:08.981740+0000","last_became_active":"2024-02-19T21:11:59.352235+0000","last_became_peered":"2024-02-19T21:11:59.352235+0000","last_unstale":"2024-02-19T21:13:08.981740+0000","last_undegraded":"2024-02-19T21:13:08.981740+0000","last_fullsized":"2024-02-19T21:13:08.981740+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:10.642603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:10:10.642603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":495,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.492513+0000","last_change":"2024-02-19T21:11:59.357670+0000","last_active":"2024-02-19T21:13:08.492513+0000","last_peered":"2024-02-19T21:13:08.492513+0000","last_clean":"2024-02-19T21:13:08.492513+0000","last_became_active":"2024-02-19T21:11:59.357345+0000","last_became_peered":"2024-02-19T21:11:59.357345+0000","last_unstale":"2024-02-19T21:13:08.492513+0000","last_undegraded":"2024-02-19T21:13:08.492513+0000","last_fullsized":"2024-02-19T21:13:08.492513+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:21.194709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:21.194709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"459'7","reported_seq":142,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349771+0000","last_change":"2024-02-19T21:12:02.521592+0000","last_active":"2024-02-19T21:13:08.349771+0000","last_peered":"2024-02-19T21:13:08.349771+0000","last_clean":"2024-02-19T21:13:08.349771+0000","last_became_active":"2024-02-19T21:10:41.297295+0000","last_became_peered":"2024-02-19T21:10:41.297295+0000","last_unstale":"2024-02-19T21:13:08.349771+0000","last_undegraded":"2024-02-19T21:13:08.349771+0000","last_fullsized":"2024-02-19T21:13:08.349771+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'7","last_scrub_stamp":"2024-02-19T21:12:02.521523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:02.521523+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":540,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.491244+0000","last_change":"2024-02-19T21:12:09.324077+0000","last_active":"2024-02-19T21:13:08.491244+0000","last_peered":"2024-02-19T21:13:08.491244+0000","last_clean":"2024-02-19T21:13:08.491244+0000","last_became_active":"2024-02-19T21:12:09.323141+0000","last_became_peered":"2024-02-19T21:12:09.323141+0000","last_unstale":"2024-02-19T21:13:08.491244+0000","last_undegraded":"2024-02-19T21:13:08.491244+0000","last_fullsized":"2024-02-19T21:13:08.491244+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:04.547957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:04.547957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":536,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.490033+0000","last_change":"2024-02-19T21:12:51.411694+0000","last_active":"2024-02-19T21:13:08.490033+0000","last_peered":"2024-02-19T21:13:08.490033+0000","last_clean":"2024-02-19T21:13:08.490033+0000","last_became_active":"2024-02-19T21:10:41.307400+0000","last_became_peered":"2024-02-19T21:10:41.307400+0000","last_unstale":"2024-02-19T21:13:08.490033+0000","last_undegraded":"2024-02-19T21:13:08.490033+0000","last_fullsized":"2024-02-19T21:13:08.490033+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:51.411574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.411574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":531,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.985393+0000","last_change":"2024-02-19T21:10:15.161470+0000","last_active":"2024-02-19T21:13:08.985393+0000","last_peered":"2024-02-19T21:13:08.985393+0000","last_clean":"2024-02-19T21:13:08.985393+0000","last_became_active":"2024-02-19T21:08:04.266100+0000","last_became_peered":"2024-02-19T21:08:04.266100+0000","last_unstale":"2024-02-19T21:13:08.985393+0000","last_undegraded":"2024-02-19T21:13:08.985393+0000","last_fullsized":"2024-02-19T21:13:08.985393+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:15.161397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:15.161397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":377,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.988653+0000","last_change":"2024-02-19T21:12:40.365196+0000","last_active":"2024-02-19T21:13:08.988653+0000","last_peered":"2024-02-19T21:13:08.988653+0000","last_clean":"2024-02-19T21:13:08.988653+0000","last_became_active":"2024-02-19T21:12:40.364772+0000","last_became_peered":"2024-02-19T21:12:40.364772+0000","last_unstale":"2024-02-19T21:13:08.988653+0000","last_undegraded":"2024-02-19T21:13:08.988653+0000","last_fullsized":"2024-02-19T21:13:08.988653+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:56.083403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:56.083403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"459'15","reported_seq":171,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.349971+0000","last_change":"2024-02-19T21:12:09.959562+0000","last_active":"2024-02-19T21:13:08.349971+0000","last_peered":"2024-02-19T21:13:08.349971+0000","last_clean":"2024-02-19T21:13:08.349971+0000","last_became_active":"2024-02-19T21:12:09.352882+0000","last_became_peered":"2024-02-19T21:12:09.352882+0000","last_unstale":"2024-02-19T21:13:08.349971+0000","last_undegraded":"2024-02-19T21:13:08.349971+0000","last_fullsized":"2024-02-19T21:13:08.349971+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:11:03.423767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.423767+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":273,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351946+0000","last_change":"2024-02-19T21:11:59.337265+0000","last_active":"2024-02-19T21:13:08.351946+0000","last_peered":"2024-02-19T21:13:08.351946+0000","last_clean":"2024-02-19T21:13:08.351946+0000","last_became_active":"2024-02-19T21:11:59.336778+0000","last_became_peered":"2024-02-19T21:11:59.336778+0000","last_unstale":"2024-02-19T21:13:08.351946+0000","last_undegraded":"2024-02-19T21:13:08.351946+0000","last_fullsized":"2024-02-19T21:13:08.351946+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:50.769620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.769620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":513,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.981812+0000","last_change":"2024-02-19T21:10:18.737245+0000","last_active":"2024-02-19T21:13:08.981812+0000","last_peered":"2024-02-19T21:13:08.981812+0000","last_clean":"2024-02-19T21:13:08.981812+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:13:08.981812+0000","last_undegraded":"2024-02-19T21:13:08.981812+0000","last_fullsized":"2024-02-19T21:13:08.981812+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:18.737176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:10:18.737176+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":133,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350336+0000","last_change":"2024-02-19T21:12:07.416212+0000","last_active":"2024-02-19T21:13:08.350336+0000","last_peered":"2024-02-19T21:13:08.350336+0000","last_clean":"2024-02-19T21:13:08.350336+0000","last_became_active":"2024-02-19T21:10:41.309550+0000","last_became_peered":"2024-02-19T21:10:41.309550+0000","last_unstale":"2024-02-19T21:13:08.350336+0000","last_undegraded":"2024-02-19T21:13:08.350336+0000","last_fullsized":"2024-02-19T21:13:08.350336+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:07.416150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:07.416150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":494,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.986832+0000","last_change":"2024-02-19T21:11:59.333342+0000","last_active":"2024-02-19T21:13:08.986832+0000","last_peered":"2024-02-19T21:13:08.986832+0000","last_clean":"2024-02-19T21:13:08.986832+0000","last_became_active":"2024-02-19T21:11:59.332987+0000","last_became_peered":"2024-02-19T21:11:59.332987+0000","last_unstale":"2024-02-19T21:13:08.986832+0000","last_undegraded":"2024-02-19T21:13:08.986832+0000","last_fullsized":"2024-02-19T21:13:08.986832+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:08.043028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:08.043028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"459'27","reported_seq":323,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.351574+0000","last_change":"2024-02-19T21:11:59.360178+0000","last_active":"2024-02-19T21:13:08.351574+0000","last_peered":"2024-02-19T21:13:08.351574+0000","last_clean":"2024-02-19T21:13:08.351574+0000","last_became_active":"2024-02-19T21:11:59.340940+0000","last_became_peered":"2024-02-19T21:11:59.340940+0000","last_unstale":"2024-02-19T21:13:08.351574+0000","last_undegraded":"2024-02-19T21:13:08.351574+0000","last_fullsized":"2024-02-19T21:13:08.351574+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:10:02.532799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:02.532799+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":387,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.493046+0000","last_change":"2024-02-19T21:12:09.339398+0000","last_active":"2024-02-19T21:13:08.493046+0000","last_peered":"2024-02-19T21:13:08.493046+0000","last_clean":"2024-02-19T21:13:08.493046+0000","last_became_active":"2024-02-19T21:12:09.338747+0000","last_became_peered":"2024-02-19T21:12:09.338747+0000","last_unstale":"2024-02-19T21:13:08.493046+0000","last_undegraded":"2024-02-19T21:13:08.493046+0000","last_fullsized":"2024-02-19T21:13:08.493046+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.720583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.720583+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":142,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350324+0000","last_change":"2024-02-19T21:12:09.346777+0000","last_active":"2024-02-19T21:13:08.350324+0000","last_peered":"2024-02-19T21:13:08.350324+0000","last_clean":"2024-02-19T21:13:08.350324+0000","last_became_active":"2024-02-19T21:12:09.346258+0000","last_became_peered":"2024-02-19T21:12:09.346258+0000","last_unstale":"2024-02-19T21:13:08.350324+0000","last_undegraded":"2024-02-19T21:13:08.350324+0000","last_fullsized":"2024-02-19T21:13:08.350324+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.407543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.407543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":273,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350379+0000","last_change":"2024-02-19T21:12:09.331143+0000","last_active":"2024-02-19T21:13:08.350379+0000","last_peered":"2024-02-19T21:13:08.350379+0000","last_clean":"2024-02-19T21:13:08.350379+0000","last_became_active":"2024-02-19T21:12:09.330530+0000","last_became_peered":"2024-02-19T21:12:09.330530+0000","last_unstale":"2024-02-19T21:13:08.350379+0000","last_undegraded":"2024-02-19T21:13:08.350379+0000","last_fullsized":"2024-02-19T21:13:08.350379+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:33.559363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:33.559363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":296,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-02-19T21:13:08.350429+0000","last_change":"2024-02-19T21:12:09.345873+0000","last_active":"2024-02-19T21:13:08.350429+0000","last_peered":"2024-02-19T21:13:08.350429+0000","last_clean":"2024-02-19T21:13:08.350429+0000","last_became_active":"2024-02-19T21:12:09.345500+0000","last_became_peered":"2024-02-19T21:12:09.345500+0000","last_unstale":"2024-02-19T21:13:08.350429+0000","last_undegraded":"2024-02-19T21:13:08.350429+0000","last_fullsized":"2024-02-19T21:13:08.350429+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.764658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.764658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":61,"num_pg":32,"stat_sum":{"num_bytes":4020435,"num_objects":193,"num_object_clones":11,"num_object_copies":579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":212,"num_read_kb":56,"num_write":779,"num_write_kb":4619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1135,"num_bytes_recovered":5695,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":21,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14508032,"data_stored":12061879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2081,"ondisk_log_size":2081,"up":96,"acting":96,"num_store_stats":8},{"poolid":228,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1102,"num_read_kb":859,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":7558,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":710,"ondisk_log_size":710,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":704,"num_read_kb":699,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":126,"acting":126,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":15,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266833,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321944,"kb_used_data":12128,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93423208,"statfs":{"total":95995035648,"available":95665364992,"internally_reserved":0,"allocated":12419072,"data_stored":8896183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.86399999999999999},"min":{"1min":0.30399999999999999,"5min":0.223,"15min":0.215},"max":{"1min":0.66000000000000003,"5min":0.79200000000000004,"15min":89.456999999999994},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.47699999999999998,"15min":0.78300000000000003},"min":{"1min":0.29599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.752,"5min":0.96999999999999997,"15min":70.451999999999998},"last":0.626}]},{"osd":1,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48599999999999999,"15min":0.69599999999999995},"min":{"1min":0.36299999999999999,"5min":0.28299999999999997,"15min":0.28199999999999997},"max":{"1min":0.61399999999999999,"5min":0.85099999999999998,"15min":34.273000000000003},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46899999999999997,"15min":0.93100000000000005},"min":{"1min":0.29499999999999998,"5min":0.28299999999999997,"15min":0.254},"max":{"1min":0.63500000000000001,"5min":0.79200000000000004,"15min":89.594999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.46999999999999997,"15min":0.93700000000000006},"min":{"1min":0.33000000000000002,"5min":0.28100000000000003,"15min":0.24299999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":104.413},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.495,"15min":0.94699999999999995},"min":{"1min":0.375,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.624,"5min":0.76400000000000001,"15min":104.268},"last":0.439}]},{"osd":3,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.95399999999999996},"min":{"1min":0.33400000000000002,"5min":0.29099999999999998,"15min":0.254},"max":{"1min":0.62,"5min":1.8939999999999999,"15min":105.3},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.50700000000000001,"15min":0.59999999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.68000000000000005,"5min":0.879,"15min":13.243},"last":0.60799999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.42499999999999999,"15min":0.46000000000000002},"min":{"1min":0.30599999999999999,"5min":0.27400000000000002,"15min":0.24299999999999999},"max":{"1min":0.54400000000000004,"5min":0.69299999999999995,"15min":1.373},"last":0.31},{"interface":"front","average":{"1min":0.439,"5min":0.46000000000000002,"15min":0.47499999999999998},"min":{"1min":0.29199999999999998,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":0.68300000000000005,"5min":0.751,"15min":1.0740000000000001},"last":0.34100000000000003}]},{"osd":5,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.443,"15min":0.45600000000000002},"min":{"1min":0.30299999999999999,"5min":0.28599999999999998,"15min":0.252},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48599999999999999,"15min":0.504},"min":{"1min":0.35499999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.755,"5min":0.79300000000000004,"15min":0.85899999999999999},"last":0.36199999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.51600000000000001},"min":{"1min":0.33700000000000002,"5min":0.30199999999999999,"15min":0.28799999999999998},"max":{"1min":0.71699999999999997,"5min":0.76700000000000002,"15min":1.0049999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52200000000000002,"15min":0.54900000000000004},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33700000000000002},"max":{"1min":0.72399999999999998,"5min":0.75900000000000001,"15min":1.262},"last":0.39600000000000002}]}]},{"osd":6,"up_from":0,"seq":193273528471,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834575022,"num_pgs":124,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321920,"kb_used_data":12092,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93423232,"statfs":{"total":95995035648,"available":95665389568,"internally_reserved":0,"allocated":12382208,"data_stored":8894266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:12:10 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.33300000000000002,"15min":0.54700000000000004},"min":{"1min":0.17999999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":2.0379999999999998,"5min":2.0379999999999998,"15min":27.777000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.35299999999999998,"15min":0.57599999999999996},"min":{"1min":0.20200000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":27.631},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:12:34 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.371,"15min":0.70599999999999996},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":27.798999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.38900000000000001,"15min":0.72299999999999998},"min":{"1min":0.24299999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.089,"5min":2.089,"15min":27.606999999999999},"last":0.443}]},{"osd":3,"last update":"Mon Feb 19 21:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.374,"15min":0.56599999999999995},"min":{"1min":0.253,"5min":0.217,"15min":0.217},"max":{"1min":0.499,"5min":1.7450000000000001,"15min":15.574999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.41099999999999998,"15min":0.66200000000000003},"min":{"1min":0.31,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.622,"5min":2.0070000000000001,"15min":27.82},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:12:15 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.501,"15min":0.65000000000000002},"min":{"1min":0.27800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.59199999999999997,"5min":1.149,"15min":9.6430000000000007},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53200000000000003,"15min":0.752},"min":{"1min":0.34799999999999998,"5min":0.308,"15min":0.308},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":27.707000000000001},"last":0.57199999999999995}]},{"osd":5,"last update":"Mon Feb 19 21:12:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52700000000000002,"15min":0.73899999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98599999999999999,"5min":1.048,"15min":27.529},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.51800000000000002,"15min":0.67500000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67400000000000004,"5min":1.1819999999999999,"15min":9.6799999999999997},"last":0.73899999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52600000000000002,"15min":0.65800000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70399999999999996,"5min":1.1659999999999999,"15min":9.6989999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.60099999999999998,"15min":0.78600000000000003},"min":{"1min":0.40799999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75900000000000001,"5min":3.4359999999999999,"15min":27.733000000000001},"last":0.71899999999999997}]},{"osd":7,"last update":"Mon Feb 19 21:12:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.56799999999999995,"15min":0.77500000000000002},"min":{"1min":0.40000000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":27.576000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.58299999999999996,"15min":0.80500000000000005},"min":{"1min":0.42099999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.71599999999999997,"5min":2.0169999999999999,"15min":27.756},"last":0.76100000000000001}]}]},{"osd":0,"up_from":0,"seq":34359738548,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411497,"num_pgs":119,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317820,"kb_used_data":12400,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93427332,"statfs":{"total":95995035648,"available":95669587968,"internally_reserved":0,"allocated":12697600,"data_stored":9051433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:12:33 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.377,"15min":0.53600000000000003},"min":{"1min":0.22900000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.54400000000000004,"5min":3.593,"15min":26.690999999999999},"last":0.253},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.35899999999999999,"15min":0.59099999999999997},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.214},"max":{"1min":0.56799999999999995,"5min":1.488,"15min":24.315000000000001},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:12:33 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.42199999999999999,"15min":0.70299999999999996},"min":{"1min":0.249,"5min":0.20599999999999999,"15min":0.20100000000000001},"max":{"1min":0.47199999999999998,"5min":7.0010000000000003,"15min":28.882000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.42299999999999999,"15min":0.68200000000000005},"min":{"1min":0.27000000000000002,"5min":0.23899999999999999,"15min":0.22500000000000001},"max":{"1min":0.60899999999999999,"5min":3.3010000000000002,"15min":28.382999999999999},"last":0.32300000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.53700000000000003,"15min":0.71199999999999997},"min":{"1min":0.27800000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68899999999999995,"5min":12.75,"15min":28.742000000000001},"last":0.309},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.44900000000000001,"15min":0.63300000000000001},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.25700000000000001},"max":{"1min":0.58399999999999996,"5min":3.6499999999999999,"15min":26.719999999999999},"last":0.35699999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.60099999999999998,"15min":0.69699999999999995},"min":{"1min":0.36299999999999999,"5min":0.33400000000000002,"15min":0.30599999999999999},"max":{"1min":0.755,"5min":3.4039999999999999,"15min":24.356000000000002},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.67100000000000004,"15min":0.71999999999999997},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.32400000000000001},"max":{"1min":0.72799999999999998,"5min":7.1340000000000003,"15min":20.707000000000001},"last":0.56599999999999995}]},{"osd":5,"last update":"Mon Feb 19 21:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57799999999999996,"15min":0.68000000000000005},"min":{"1min":0.36399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81599999999999995,"5min":3.3090000000000002,"15min":9.9499999999999993},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60899999999999999,"15min":0.79100000000000004},"min":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.38400000000000001},"max":{"1min":0.77500000000000002,"5min":3.2690000000000001,"15min":26.606000000000002},"last":0.49099999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:12:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63200000000000001,"15min":0.80200000000000005},"min":{"1min":0.40899999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81000000000000005,"5min":4.5099999999999998,"15min":24.381},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58799999999999997,"15min":0.73699999999999999},"min":{"1min":0.41899999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.745,"5min":0.93600000000000005,"15min":28.706},"last":0.59899999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:12:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.63200000000000001,"15min":0.77800000000000002},"min":{"1min":0.441,"5min":0.42199999999999999,"15min":0.34200000000000003},"max":{"1min":0.77600000000000002,"5min":3.3359999999999999,"15min":26.658000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59999999999999998,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82099999999999995,"5min":1.5109999999999999,"15min":24.402000000000001},"last":0.58699999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182565,"num_pgs":114,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313808,"kb_used_data":8388,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93431344,"statfs":{"total":95995035648,"available":95673696256,"internally_reserved":0,"allocated":8589312,"data_stored":5039373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.71599999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.156},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":77.662000000000006},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.35499999999999998,"15min":0.49399999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.19700000000000001},"max":{"1min":0.60199999999999998,"5min":0.60999999999999999,"15min":17.456},"last":0.32800000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.52600000000000002},"min":{"1min":0.255,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":0.66300000000000003,"5min":6.9299999999999997,"15min":17.327999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.42499999999999999,"15min":0.50600000000000001},"min":{"1min":0.20699999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.95599999999999996,"5min":7.3620000000000001,"15min":15.004},"last":0.35099999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.376,"15min":0.66300000000000003},"min":{"1min":0.27300000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":56.890999999999998},"last":0.378},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.36599999999999999,"15min":0.64400000000000002},"min":{"1min":0.246,"5min":0.24399999999999999,"15min":0.20899999999999999},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":44.600999999999999},"last":0.42399999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52500000000000002,"15min":0.63400000000000001},"min":{"1min":0.33400000000000002,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":12.847},"last":0.434},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.52300000000000002,"15min":0.70599999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72299999999999998,"5min":0.94299999999999995,"15min":26.393999999999998},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.61799999999999999,"15min":0.68400000000000005},"min":{"1min":0.39100000000000001,"5min":0.34100000000000003,"15min":0.313},"max":{"1min":0.68300000000000005,"5min":9.2360000000000007,"15min":17.401},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54300000000000004,"15min":0.65400000000000003},"min":{"1min":0.35999999999999999,"5min":0.313,"15min":0.313},"max":{"1min":1.3660000000000001,"5min":1.9690000000000001,"15min":12.882},"last":0.55200000000000005}]},{"osd":6,"last update":"Mon Feb 19 21:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.56200000000000006,"15min":0.65400000000000003},"min":{"1min":0.36899999999999999,"5min":0.34799999999999998,"15min":0.34300000000000003},"max":{"1min":0.72499999999999998,"5min":1.9790000000000001,"15min":17.367000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54600000000000004,"15min":0.66500000000000004},"min":{"1min":0.38900000000000001,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":12.904},"last":0.57499999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:12:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55900000000000005,"15min":0.65500000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.038,"5min":1.038,"15min":17.428999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.80000000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.34499999999999997},"max":{"1min":1.111,"5min":1.111,"15min":44.378},"last":0.52900000000000003}]}]},{"osd":4,"up_from":31,"seq":133143986337,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313960,"kb_used_data":8236,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93431192,"statfs":{"total":95995035648,"available":95673540608,"internally_reserved":0,"allocated":8433664,"data_stored":4893181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.54700000000000004,"15min":0.63900000000000001},"min":{"1min":0.28199999999999997,"5min":0.23599999999999999,"15min":0.22500000000000001},"max":{"1min":0.68300000000000005,"5min":8.0530000000000008,"15min":19.878},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.49199999999999999,"15min":0.85799999999999998},"min":{"1min":0.32300000000000001,"5min":0.22500000000000001,"15min":0.20499999999999999},"max":{"1min":0.61099999999999999,"5min":3.3530000000000002,"15min":70.063000000000002},"last":0.379}]},{"osd":1,"last update":"Mon Feb 19 21:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46700000000000003,"15min":0.63600000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":26.510999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.48599999999999999,"15min":0.68200000000000005},"min":{"1min":0.32100000000000001,"5min":0.312,"15min":0.23200000000000001},"max":{"1min":0.61899999999999999,"5min":0.82999999999999996,"15min":25.949000000000002},"last":0.51800000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.57599999999999996,"15min":0.71899999999999997},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.67400000000000004,"5min":8.1240000000000006,"15min":37.320999999999998},"last":2.2040000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51500000000000001,"15min":1.0600000000000001},"min":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.253},"max":{"1min":0.66500000000000004,"5min":2.9969999999999999,"15min":69.936999999999998},"last":0.50900000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.53400000000000003,"15min":0.91200000000000003},"min":{"1min":0.38100000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69399999999999995,"5min":3.323,"15min":69.793000000000006},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52000000000000002,"15min":0.76200000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76500000000000001,"5min":0.81799999999999995,"15min":17.986999999999998},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.218},"max":{"1min":0.78800000000000003,"5min":1.036,"15min":3.2799999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.48199999999999998,"15min":0.46300000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.23899999999999999},"max":{"1min":0.745,"5min":1.121,"15min":4.4340000000000002},"last":0.53500000000000003}]},{"osd":6,"last update":"Mon Feb 19 21:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.57399999999999995,"15min":0.47599999999999998},"min":{"1min":0.29199999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.78700000000000003,"5min":8.0969999999999995,"15min":8.0969999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.51800000000000002},"min":{"1min":0.35599999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.4550000000000001,"5min":3.0270000000000001,"15min":4.5609999999999999},"last":0.59299999999999997}]},{"osd":7,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56299999999999994,"15min":0.51000000000000001},"min":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.71699999999999997,"5min":3.2909999999999999,"15min":6.0730000000000004},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.55200000000000005,"15min":0.51200000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.26400000000000001},"max":{"1min":0.72699999999999998,"5min":1.0209999999999999,"15min":3.202},"last":0.57099999999999995}]}]},{"osd":5,"up_from":37,"seq":158913790108,"num_pgs":123,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":322128,"kb_used_data":8508,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93423024,"statfs":{"total":95995035648,"available":95665176576,"internally_reserved":0,"allocated":8712192,"data_stored":5046139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.45800000000000002,"15min":1.8779999999999999},"min":{"1min":0.26200000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.64400000000000002,"5min":0.71699999999999997,"15min":276.47899999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.48799999999999999,"15min":0.90100000000000002},"min":{"1min":0.27400000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.58099999999999996,"5min":0.95299999999999996,"15min":52.887},"last":0.35099999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.46999999999999997,"15min":1.266},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.23499999999999999},"max":{"1min":0.59599999999999997,"5min":1.599,"15min":120.011},"last":0.376},{"interface":"front","average":{"1min":0.443,"5min":0.48899999999999999,"15min":1.02},"min":{"1min":0.28599999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63100000000000001,"5min":0.97699999999999998,"15min":118.105},"last":0.503}]},{"osd":2,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47799999999999998,"15min":0.99399999999999999},"min":{"1min":0.32900000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65600000000000003,"5min":0.92300000000000004,"15min":82.242000000000004},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.55700000000000005,"15min":1.5129999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65300000000000002,"5min":4.5019999999999998,"15min":232.84999999999999},"last":0.47699999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52700000000000002,"15min":1.0649999999999999},"min":{"1min":0.35299999999999998,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":0.71199999999999997,"5min":4.4640000000000004,"15min":109.70099999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52400000000000002,"15min":0.91100000000000003},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":65.519000000000005},"last":0.41699999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.432,"15min":0.42099999999999999},"min":{"1min":0.26900000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66200000000000003,"5min":1.0129999999999999,"15min":1.861},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.44,"15min":0.433},"min":{"1min":0.255,"5min":0.255,"15min":0.248},"max":{"1min":0.61599999999999999,"5min":0.72499999999999998,"15min":1.9730000000000001},"last":0.51900000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47299999999999998,"15min":0.45500000000000002},"min":{"1min":0.31,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69599999999999995,"5min":1.0289999999999999,"15min":1.823},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47199999999999998,"15min":0.46300000000000002},"min":{"1min":0.30499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.59899999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:12:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.497,"15min":0.5},"min":{"1min":0.34000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63300000000000001,"5min":1.052,"15min":1.8879999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51200000000000001,"15min":0.50700000000000001},"min":{"1min":0.34399999999999997,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.64100000000000001,"5min":0.871,"15min":0.96799999999999997},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4358144,"data_stored":4009587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4407296,"data_stored":4015944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":3600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":438272,"data_stored":10060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":10378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":61,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4415488,"data_stored":4011820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":228,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":228,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":228,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":228,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":228,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":228,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:13:12.599 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:13:12.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.238+0000 7f48fdbc1700 1 -- 172.21.15.163:0/1565486976 >> v1:172.21.15.163:6790/0 conn(0x7f48f81042d0 legacy=0x7f48f8106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:12.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.238+0000 7f48fdbc1700 1 -- 172.21.15.163:0/1565486976 shutdown_connections 2024-02-19T21:13:12.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.239+0000 7f48fdbc1700 1 -- 172.21.15.163:0/1565486976 >> 172.21.15.163:0/1565486976 conn(0x7f48f80f3550 msgr2=0x7f48f80f5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:12.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.239+0000 7f48fdbc1700 1 -- 172.21.15.163:0/1565486976 shutdown_connections 2024-02-19T21:13:12.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.239+0000 7f48fdbc1700 1 -- 172.21.15.163:0/1565486976 wait complete. 2024-02-19T21:13:12.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.239+0000 7f48fdbc1700 1 librados: starting msgr at 2024-02-19T21:13:12.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.239+0000 7f48fdbc1700 1 librados: starting objecter 2024-02-19T21:13:12.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.239+0000 7f48fdbc1700 1 Processor -- start 2024-02-19T21:13:12.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.239+0000 7f48fdbc1700 1 -- start start 2024-02-19T21:13:12.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.239+0000 7f48fdbc1700 1 librados: setting wanted keys 2024-02-19T21:13:12.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.239+0000 7f48fdbc1700 1 librados: calling monclient init 2024-02-19T21:13:12.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48fdbc1700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f48f80fcef0 con 0x7f48f8100790 2024-02-19T21:13:12.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48fdbc1700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f48f80fd080 con 0x7f48f81042d0 2024-02-19T21:13:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48fdbc1700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f48f80fd270 con 0x7f48f80fd990 2024-02-19T21:13:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f77fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f48f80fd990 0x7f48f80fc7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:48518/0 (socket says 172.21.15.163:48518) 2024-02-19T21:13:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f6ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f48f8100790 0x7f48f8114d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33500/0 (socket says 172.21.15.163:33500) 2024-02-19T21:13:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f77fe700 1 -- 172.21.15.163:0/2870230078 learned_addr learned my addr 172.21.15.163:0/2870230078 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 10 client.?.objecter ms_handle_connect 0x7f48f8100790 2024-02-19T21:13:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 10 client.?.objecter ms_handle_connect 0x7f48f80fd990 2024-02-19T21:13:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 10 client.?.objecter ms_handle_connect 0x7f48f81042d0 2024-02-19T21:13:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122216629 0 0) 0x7f48f80fd270 con 0x7f48f80fd990 2024-02-19T21:13:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48e0003030 con 0x7f48f80fd990 2024-02-19T21:13:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880383779 0 0) 0x7f48f80fcef0 con 0x7f48f8100790 2024-02-19T21:13:12.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f80fd270 con 0x7f48f8100790 2024-02-19T21:13:12.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708698495 0 0) 0x7f48f80fd080 con 0x7f48f81042d0 2024-02-19T21:13:12.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f80fcef0 con 0x7f48f81042d0 2024-02-19T21:13:12.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1440571764 0 0) 0x7f48e0003030 con 0x7f48f80fd990 2024-02-19T21:13:12.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48f80fd080 con 0x7f48f80fd990 2024-02-19T21:13:12.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 6295156 0 0) 0x7f48f80fcef0 con 0x7f48f81042d0 2024-02-19T21:13:12.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48e0003030 con 0x7f48f81042d0 2024-02-19T21:13:12.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f48e80018c0 con 0x7f48f80fd990 2024-02-19T21:13:12.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f48f00017b0 con 0x7f48f81042d0 2024-02-19T21:13:12.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2837540063 0 0) 0x7f48f80fd270 con 0x7f48f8100790 2024-02-19T21:13:12.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48f80fcef0 con 0x7f48f8100790 2024-02-19T21:13:12.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f48ec001910 con 0x7f48f8100790 2024-02-19T21:13:12.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3708825454 0 0) 0x7f48e0003030 con 0x7f48f81042d0 2024-02-19T21:13:12.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 >> v1:172.21.15.163:6790/0 conn(0x7f48f80fd990 legacy=0x7f48f80fc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:12.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 >> v1:172.21.15.163:6789/0 conn(0x7f48f8100790 legacy=0x7f48f8114d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:12.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f811cba0 con 0x7f48f81042d0 2024-02-19T21:13:12.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48fdbc1700 1 -- 172.21.15.163:0/2870230078 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f8119bd0 con 0x7f48f81042d0 2024-02-19T21:13:12.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48fdbc1700 10 client.48964.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:12.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48fdbc1700 1 -- 172.21.15.163:0/2870230078 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f811a0c0 con 0x7f48f81042d0 2024-02-19T21:13:12.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.240+0000 7f48fdbc1700 1 librados: init done 2024-02-19T21:13:12.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.241+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f48f0003360 con 0x7f48f81042d0 2024-02-19T21:13:12.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.241+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f48f00037b0 con 0x7f48f81042d0 2024-02-19T21:13:12.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.241+0000 7f48fdbc1700 1 -- 172.21.15.163:0/2870230078 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e4004f20 con 0x7f48f81042d0 2024-02-19T21:13:12.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.242+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f48f0016640 con 0x7f48f81042d0 2024-02-19T21:13:12.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.242+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(555..555 src has 434..555) v4 ==== 8108+0+0 (unknown 2346165935 0 0) 0x7f48f0076950 con 0x7f48f81042d0 2024-02-19T21:13:12.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.242+0000 7f48f4ff9700 10 client.48964.objecter ms_dispatch 0x7f48f8001550 osd_map(555..555 src has 434..555) v4 2024-02-19T21:13:12.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.242+0000 7f48f4ff9700 3 client.48964.objecter handle_osd_map got epochs [555,555] > 0 2024-02-19T21:13:12.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.242+0000 7f48f4ff9700 3 client.48964.objecter handle_osd_map decoding full epoch 555 2024-02-19T21:13:12.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.242+0000 7f48f4ff9700 20 client.48964.objecter dump_active .. 0 homeless 2024-02-19T21:13:12.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.242+0000 7f48f4ff9700 10 client.48964.objecter ms_handle_connect 0x7f48e0063cb0 2024-02-19T21:13:12.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.247+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f48f0049f20 con 0x7f48f81042d0 2024-02-19T21:13:12.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.423+0000 7f48fdbc1700 1 -- 172.21.15.163:0/2870230078 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f48e40051d0 con 0x7f48f81042d0 2024-02-19T21:13:12.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.486+0000 7f48f4ff9700 1 -- 172.21.15.163:0/2870230078 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v556) v1 ==== 87+0+0 (unknown 2685762295 0 0) 0x7f48f0048390 con 0x7f48f81042d0 2024-02-19T21:13:12.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.488+0000 7f48fdbc1700 10 librados: watch_flush enter 2024-02-19T21:13:12.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.488+0000 7f48fdbc1700 10 librados: watch_flush exit 2024-02-19T21:13:12.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.488+0000 7f48fdbc1700 20 client.48964.objecter shutdown clearing up homeless session... 2024-02-19T21:13:12.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.488+0000 7f48fdbc1700 10 client.48964.objecter successfully canceled tick 2024-02-19T21:13:12.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.488+0000 7f48fdbc1700 1 -- 172.21.15.163:0/2870230078 >> v1:172.21.15.163:6800/1552514681 conn(0x7f48e0063cb0 legacy=0x7f48e0066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:12.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.488+0000 7f48fdbc1700 1 -- 172.21.15.163:0/2870230078 >> v1:172.21.15.193:6789/0 conn(0x7f48f81042d0 legacy=0x7f48f8118450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:12.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.488+0000 7f48fdbc1700 1 -- 172.21.15.163:0/2870230078 shutdown_connections 2024-02-19T21:13:12.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.488+0000 7f48fdbc1700 1 -- 172.21.15.163:0/2870230078 >> 172.21.15.163:0/2870230078 conn(0x7f48f80f3550 msgr2=0x7f48f8107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:12.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.488+0000 7f48fdbc1700 1 -- 172.21.15.163:0/2870230078 shutdown_connections 2024-02-19T21:13:12.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.488+0000 7f48fdbc1700 1 -- 172.21.15.163:0/2870230078 wait complete. 2024-02-19T21:13:12.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:12.488+0000 7f48fdbc1700 1 librados: shutdown 2024-02-19T21:13:12.655 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:13:12.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:11.002954+0000 osd.1 (osd.1) 259 : cluster [DBG] 16.15 deep-scrub starts 2024-02-19T21:13:12.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:11.004001+0000 osd.1 (osd.1) 260 : cluster [DBG] 16.15 deep-scrub ok 2024-02-19T21:13:12.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:11.133618+0000 mgr.y (mgr.14550) 491 : audit [DBG] from='client.50150 v1:172.21.15.163:0/2435800927' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:13:12.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:11.265237+0000 mgr.y (mgr.14550) 492 : cluster [DBG] pgmap v831: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:13:12.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:11.485941+0000 mon.a (mon.0) 2940 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:13:12.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:11.486009+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e555: 8 total, 8 up, 6 in 2024-02-19T21:13:12.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:11.487729+0000 mon.b (mon.1) 712 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:13:12.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:11.488000+0000 mon.a (mon.0) 2942 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:13:12.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:12.425278+0000 mon.b (mon.1) 713 : audit [INF] from='client.? v1:172.21.15.163:0/2870230078' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:13:12.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:12.425604+0000 mon.a (mon.0) 2943 : audit [INF] from='client.48964 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:13:12.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:11.002954+0000 osd.1 (osd.1) 259 : cluster [DBG] 16.15 deep-scrub starts 2024-02-19T21:13:12.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:11.004001+0000 osd.1 (osd.1) 260 : cluster [DBG] 16.15 deep-scrub ok 2024-02-19T21:13:12.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:11.133618+0000 mgr.y (mgr.14550) 491 : audit [DBG] from='client.50150 v1:172.21.15.163:0/2435800927' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:13:12.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:11.265237+0000 mgr.y (mgr.14550) 492 : cluster [DBG] pgmap v831: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:13:12.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:11.485941+0000 mon.a (mon.0) 2940 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:13:12.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:11.486009+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e555: 8 total, 8 up, 6 in 2024-02-19T21:13:12.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:11.487729+0000 mon.b (mon.1) 712 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:13:12.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:11.488000+0000 mon.a (mon.0) 2942 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:13:12.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:12.425278+0000 mon.b (mon.1) 713 : audit [INF] from='client.? v1:172.21.15.163:0/2870230078' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:13:12.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:12.425604+0000 mon.a (mon.0) 2943 : audit [INF] from='client.48964 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:13:12.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:11.002954+0000 osd.1 (osd.1) 259 : cluster [DBG] 16.15 deep-scrub starts 2024-02-19T21:13:12.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:11.004001+0000 osd.1 (osd.1) 260 : cluster [DBG] 16.15 deep-scrub ok 2024-02-19T21:13:12.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:11.133618+0000 mgr.y (mgr.14550) 491 : audit [DBG] from='client.50150 v1:172.21.15.163:0/2435800927' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:13:12.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:11.265237+0000 mgr.y (mgr.14550) 492 : cluster [DBG] pgmap v831: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:13:12.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:11.485941+0000 mon.a (mon.0) 2940 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-02-19T21:13:12.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:11.486009+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e555: 8 total, 8 up, 6 in 2024-02-19T21:13:12.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:11.487729+0000 mon.b (mon.1) 712 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:13:12.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:11.488000+0000 mon.a (mon.0) 2942 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-02-19T21:13:12.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:12.425278+0000 mon.b (mon.1) 713 : audit [INF] from='client.? v1:172.21.15.163:0/2870230078' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:13:12.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:12.425604+0000 mon.a (mon.0) 2943 : audit [INF] from='client.48964 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:13:12.900 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:12.902 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:12.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:13.004 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:13.032 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:13.051 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:13.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:13.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:13.155 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:13.181 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:13.199 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:13.201 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:13.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:13.303 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:13.333 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:13.373 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:13.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:13.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:13.477 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:13:13.510 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:13.513 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:13.514 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:13.522 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:13.539 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:13:13.544 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:13.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:13:13.645 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:13.671 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:13.689 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:13.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:13.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:13.747 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:13:13] "GET /metrics HTTP/1.1" 200 188097 "" "Prometheus/2.33.4" 2024-02-19T21:13:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:12.404964+0000 osd.5 (osd.5) 226 : cluster [DBG] 4.10 deep-scrub starts 2024-02-19T21:13:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:12.409032+0000 osd.5 (osd.5) 227 : cluster [DBG] 4.10 deep-scrub ok 2024-02-19T21:13:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:12.483611+0000 mon.a (mon.0) 2944 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:13:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:12.486843+0000 mon.a (mon.0) 2945 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-19T21:13:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:12.486893+0000 mon.a (mon.0) 2946 : audit [INF] from='client.48964 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:13:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:12.486927+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e556: 8 total, 8 up, 6 in 2024-02-19T21:13:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:12.488508+0000 mon.b (mon.1) 714 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:13:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:12.498094+0000 mon.a (mon.0) 2948 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:13:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:12.404964+0000 osd.5 (osd.5) 226 : cluster [DBG] 4.10 deep-scrub starts 2024-02-19T21:13:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:12.409032+0000 osd.5 (osd.5) 227 : cluster [DBG] 4.10 deep-scrub ok 2024-02-19T21:13:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:12.483611+0000 mon.a (mon.0) 2944 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:13:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:12.486843+0000 mon.a (mon.0) 2945 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-19T21:13:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:12.486893+0000 mon.a (mon.0) 2946 : audit [INF] from='client.48964 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:13:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:12.486927+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e556: 8 total, 8 up, 6 in 2024-02-19T21:13:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:12.488508+0000 mon.b (mon.1) 714 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:13:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:12.498094+0000 mon.a (mon.0) 2948 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:13:13.793 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:12.404964+0000 osd.5 (osd.5) 226 : cluster [DBG] 4.10 deep-scrub starts 2024-02-19T21:13:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:12.409032+0000 osd.5 (osd.5) 227 : cluster [DBG] 4.10 deep-scrub ok 2024-02-19T21:13:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:12.483611+0000 mon.a (mon.0) 2944 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:13:13.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:12.486843+0000 mon.a (mon.0) 2945 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-02-19T21:13:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:12.486893+0000 mon.a (mon.0) 2946 : audit [INF] from='client.48964 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:13:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:12.486927+0000 mon.a (mon.0) 2947 : cluster [DBG] osdmap e556: 8 total, 8 up, 6 in 2024-02-19T21:13:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:12.488508+0000 mon.b (mon.1) 714 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:13:13.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:12.498094+0000 mon.a (mon.0) 2948 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-02-19T21:13:13.820 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:13.838 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:13.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:13.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:13.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:13:13.969 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:13.990 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:13:13.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:13.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:13:14.094 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:14.121 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:14.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:13.041440+0000 osd.1 (osd.1) 261 : cluster [DBG] 2.a scrub starts 2024-02-19T21:13:14.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:13.265680+0000 mgr.y (mgr.14550) 493 : cluster [DBG] pgmap v834: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:13:14.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:13.503510+0000 mon.a (mon.0) 2949 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:13:14.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:13.503634+0000 mon.a (mon.0) 2950 : cluster [DBG] osdmap e557: 8 total, 8 up, 6 in 2024-02-19T21:13:14.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:13.550229+0000 mon.b (mon.1) 715 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:13:14.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:13.550607+0000 mon.a (mon.0) 2951 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:13:14.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:13.551127+0000 mon.b (mon.1) 716 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-92"}]: dispatch 2024-02-19T21:13:14.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:13.551345+0000 mon.a (mon.0) 2952 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-92"}]: dispatch 2024-02-19T21:13:14.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:14.329359+0000 mon.a (mon.0) 2953 : cluster [DBG] osdmap e558: 8 total, 8 up, 6 in 2024-02-19T21:13:14.977 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:14.980 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:13.041440+0000 osd.1 (osd.1) 261 : cluster [DBG] 2.a scrub starts 2024-02-19T21:13:14.980 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:13.265680+0000 mgr.y (mgr.14550) 493 : cluster [DBG] pgmap v834: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:13:14.980 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:13.503510+0000 mon.a (mon.0) 2949 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:13:14.980 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:13.503634+0000 mon.a (mon.0) 2950 : cluster [DBG] osdmap e557: 8 total, 8 up, 6 in 2024-02-19T21:13:14.980 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:13.550229+0000 mon.b (mon.1) 715 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:13:14.980 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:13.550607+0000 mon.a (mon.0) 2951 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:13:14.980 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:13.551127+0000 mon.b (mon.1) 716 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-92"}]: dispatch 2024-02-19T21:13:14.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:13.551345+0000 mon.a (mon.0) 2952 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-92"}]: dispatch 2024-02-19T21:13:14.981 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:14.329359+0000 mon.a (mon.0) 2953 : cluster [DBG] osdmap e558: 8 total, 8 up, 6 in 2024-02-19T21:13:14.981 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:13.041440+0000 osd.1 (osd.1) 261 : cluster [DBG] 2.a scrub starts 2024-02-19T21:13:14.981 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:13.265680+0000 mgr.y (mgr.14550) 493 : cluster [DBG] pgmap v834: 238 pgs: 32 creating+peering, 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:13:14.981 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:13.503510+0000 mon.a (mon.0) 2949 : audit [INF] from='client.25753 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-92","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-02-19T21:13:14.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:13.503634+0000 mon.a (mon.0) 2950 : cluster [DBG] osdmap e557: 8 total, 8 up, 6 in 2024-02-19T21:13:14.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:13.550229+0000 mon.b (mon.1) 715 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:13:14.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:13.550607+0000 mon.a (mon.0) 2951 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-6"}]: dispatch 2024-02-19T21:13:14.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:13.551127+0000 mon.b (mon.1) 716 : audit [INF] from='client.? v1:172.21.15.163:0/2605960922' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-92"}]: dispatch 2024-02-19T21:13:14.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:13.551345+0000 mon.a (mon.0) 2952 : audit [INF] from='client.25753 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-6", "tierpool": "test-rados-api-smithi163-83928-92"}]: dispatch 2024-02-19T21:13:14.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:14.329359+0000 mon.a (mon.0) 2953 : cluster [DBG] osdmap e558: 8 total, 8 up, 6 in 2024-02-19T21:13:14.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:14.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:15.085 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:15.115 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:15.135 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:15.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:15.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:15.238 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:15.267 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:15.286 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:15.289 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:15.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:15.390 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:15.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:13:15.419 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:15.619 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:15.628 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:15.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:15.730 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:13:15.758 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:15.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.914+0000 7fd8ff4fc700 1 -- 172.21.15.163:0/2097833441 >> v1:172.21.15.163:6789/0 conn(0x7fd8f80fc870 legacy=0x7fd8f80fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:15.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.914+0000 7fd8ff4fc700 1 -- 172.21.15.163:0/2097833441 shutdown_connections 2024-02-19T21:13:15.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.914+0000 7fd8ff4fc700 1 -- 172.21.15.163:0/2097833441 >> 172.21.15.163:0/2097833441 conn(0x7fd8f80f2470 msgr2=0x7fd8f80f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:15.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.915+0000 7fd8ff4fc700 1 -- 172.21.15.163:0/2097833441 shutdown_connections 2024-02-19T21:13:15.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.915+0000 7fd8ff4fc700 1 -- 172.21.15.163:0/2097833441 wait complete. 2024-02-19T21:13:15.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.915+0000 7fd8ff4fc700 1 librados: starting msgr at 2024-02-19T21:13:15.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.915+0000 7fd8ff4fc700 1 librados: starting objecter 2024-02-19T21:13:15.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.915+0000 7fd8ff4fc700 1 Processor -- start 2024-02-19T21:13:15.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ff4fc700 1 -- start start 2024-02-19T21:13:15.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ff4fc700 1 librados: setting wanted keys 2024-02-19T21:13:15.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ff4fc700 1 librados: calling monclient init 2024-02-19T21:13:15.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ff4fc700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd8f8117330 con 0x7fd8f80ff670 2024-02-19T21:13:15.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ff4fc700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd8f8118510 con 0x7fd8f80fc870 2024-02-19T21:13:15.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ff4fc700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd8f8119710 con 0x7fd8f81031b0 2024-02-19T21:13:15.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8fda99700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fd8f81031b0 0x7fd8f81159f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:33302/0 (socket says 172.21.15.163:33302) 2024-02-19T21:13:15.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8fca97700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fd8f80ff670 0x7fd8f8061350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:39592/0 (socket says 172.21.15.163:39592) 2024-02-19T21:13:15.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8fda99700 1 -- 172.21.15.163:0/1689343109 learned_addr learned my addr 172.21.15.163:0/1689343109 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:15.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ee7fc700 10 client.?.objecter ms_handle_connect 0x7fd8f80ff670 2024-02-19T21:13:15.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ee7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:15.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ee7fc700 10 client.?.objecter ms_handle_connect 0x7fd8f81031b0 2024-02-19T21:13:15.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ee7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:15.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ee7fc700 10 client.?.objecter ms_handle_connect 0x7fd8f80fc870 2024-02-19T21:13:15.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ee7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:15.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156051170 0 0) 0x7fd8f8117330 con 0x7fd8f80ff670 2024-02-19T21:13:15.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e4003030 con 0x7fd8f80ff670 2024-02-19T21:13:15.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098999221 0 0) 0x7fd8f8119710 con 0x7fd8f81031b0 2024-02-19T21:13:15.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8f8117330 con 0x7fd8f81031b0 2024-02-19T21:13:15.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027508734 0 0) 0x7fd8f8118510 con 0x7fd8f80fc870 2024-02-19T21:13:15.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8f8119710 con 0x7fd8f80fc870 2024-02-19T21:13:15.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2114009795 0 0) 0x7fd8e4003030 con 0x7fd8f80ff670 2024-02-19T21:13:15.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8f8118510 con 0x7fd8f80ff670 2024-02-19T21:13:15.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.916+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd8e80018c0 con 0x7fd8f80ff670 2024-02-19T21:13:15.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.917+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 858653473 0 0) 0x7fd8f8117330 con 0x7fd8f81031b0 2024-02-19T21:13:15.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.917+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8e4003030 con 0x7fd8f81031b0 2024-02-19T21:13:15.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.917+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd8f0001d70 con 0x7fd8f81031b0 2024-02-19T21:13:15.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.917+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4014787055 0 0) 0x7fd8f8119710 con 0x7fd8f80fc870 2024-02-19T21:13:15.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.917+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8f8117330 con 0x7fd8f80fc870 2024-02-19T21:13:15.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.917+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd8f4001e50 con 0x7fd8f80fc870 2024-02-19T21:13:15.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.917+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2145279212 0 0) 0x7fd8f8118510 con 0x7fd8f80ff670 2024-02-19T21:13:15.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.917+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 >> v1:172.21.15.163:6790/0 conn(0x7fd8f81031b0 legacy=0x7fd8f81159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:15.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.917+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 >> v1:172.21.15.193:6789/0 conn(0x7fd8f80fc870 legacy=0x7fd8f8060c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:15.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.917+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f811a910 con 0x7fd8f80ff670 2024-02-19T21:13:15.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.917+0000 7fd8ff4fc700 1 -- 172.21.15.163:0/1689343109 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f8119900 con 0x7fd8f80ff670 2024-02-19T21:13:15.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.917+0000 7fd8ff4fc700 10 client.23439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:15.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.917+0000 7fd8ff4fc700 1 -- 172.21.15.163:0/1689343109 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f8119d70 con 0x7fd8f80ff670 2024-02-19T21:13:15.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.917+0000 7fd8ff4fc700 1 librados: init done 2024-02-19T21:13:15.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.918+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fd8e8002350 con 0x7fd8f80ff670 2024-02-19T21:13:15.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.918+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd8e8005a70 con 0x7fd8f80ff670 2024-02-19T21:13:15.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.918+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fd8e8018920 con 0x7fd8f80ff670 2024-02-19T21:13:15.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.918+0000 7fd8ff4fc700 1 -- 172.21.15.163:0/1689343109 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f805f700 con 0x7fd8f80ff670 2024-02-19T21:13:15.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.919+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(559..559 src has 434..559) v4 ==== 7307+0+0 (unknown 2651099090 0 0) 0x7fd8e80024c0 con 0x7fd8f80ff670 2024-02-19T21:13:15.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.919+0000 7fd8ee7fc700 10 client.23439.objecter ms_dispatch 0x7fd8f8001550 osd_map(559..559 src has 434..559) v4 2024-02-19T21:13:15.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.919+0000 7fd8ee7fc700 3 client.23439.objecter handle_osd_map got epochs [559,559] > 0 2024-02-19T21:13:15.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.919+0000 7fd8ee7fc700 3 client.23439.objecter handle_osd_map decoding full epoch 559 2024-02-19T21:13:15.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.919+0000 7fd8ee7fc700 20 client.23439.objecter dump_active .. 0 homeless 2024-02-19T21:13:15.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.919+0000 7fd8ee7fc700 10 client.23439.objecter ms_handle_connect 0x7fd8e4063d60 2024-02-19T21:13:15.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:15.923+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd8e804cec0 con 0x7fd8f80ff670 2024-02-19T21:13:16.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:16.102+0000 7fd8ff4fc700 1 -- 172.21.15.163:0/1689343109 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd8f80f80a0 con 0x7fd8e4063d60 2024-02-19T21:13:16.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:16.115+0000 7fd8ee7fc700 1 -- 172.21.15.163:0/1689343109 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+445228 (unknown 2965378022 0 3497601855) 0x7fd8f80f80a0 con 0x7fd8e4063d60 2024-02-19T21:13:16.120 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:13:16.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:16.118+0000 7fd8ff4fc700 10 librados: watch_flush enter 2024-02-19T21:13:16.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:16.118+0000 7fd8ff4fc700 10 librados: watch_flush exit 2024-02-19T21:13:16.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:16.118+0000 7fd8ff4fc700 20 client.23439.objecter shutdown clearing up homeless session... 2024-02-19T21:13:16.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:16.118+0000 7fd8ff4fc700 10 client.23439.objecter successfully canceled tick 2024-02-19T21:13:16.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:16.118+0000 7fd8ff4fc700 1 -- 172.21.15.163:0/1689343109 >> v1:172.21.15.163:6800/1552514681 conn(0x7fd8e4063d60 legacy=0x7fd8e4066220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:16.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:16.118+0000 7fd8ff4fc700 1 -- 172.21.15.163:0/1689343109 >> v1:172.21.15.163:6789/0 conn(0x7fd8f80ff670 legacy=0x7fd8f8061350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:16.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:16.118+0000 7fd8ff4fc700 1 -- 172.21.15.163:0/1689343109 shutdown_connections 2024-02-19T21:13:16.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:16.118+0000 7fd8ff4fc700 1 -- 172.21.15.163:0/1689343109 >> 172.21.15.163:0/1689343109 conn(0x7fd8f80f2470 msgr2=0x7fd8f8106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:16.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:16.118+0000 7fd8ff4fc700 1 -- 172.21.15.163:0/1689343109 shutdown_connections 2024-02-19T21:13:16.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:16.118+0000 7fd8ff4fc700 1 -- 172.21.15.163:0/1689343109 wait complete. 2024-02-19T21:13:16.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:16.118+0000 7fd8ff4fc700 1 librados: shutdown 2024-02-19T21:13:16.125 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:13:16.586 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:15.266606+0000 mgr.y (mgr.14550) 494 : cluster [DBG] pgmap v837: 206 pgs: 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-19T21:13:16.586 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:15.331145+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e559: 8 total, 8 up, 6 in 2024-02-19T21:13:16.586 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:15.357303+0000 mon.b (mon.1) 717 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:16.586 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:15.357683+0000 mon.a (mon.0) 2955 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:16.586 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:15.358985+0000 mon.b (mon.1) 718 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:16.586 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:15.359364+0000 mon.a (mon.0) 2956 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:16.586 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:15.359959+0000 mon.b (mon.1) 719 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:13:16.587 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:15.360318+0000 mon.a (mon.0) 2957 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:13:16.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:15.266606+0000 mgr.y (mgr.14550) 494 : cluster [DBG] pgmap v837: 206 pgs: 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-19T21:13:16.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:15.331145+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e559: 8 total, 8 up, 6 in 2024-02-19T21:13:16.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:15.357303+0000 mon.b (mon.1) 717 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:16.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:15.357683+0000 mon.a (mon.0) 2955 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:16.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:15.358985+0000 mon.b (mon.1) 718 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:16.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:15.359364+0000 mon.a (mon.0) 2956 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:16.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:15.359959+0000 mon.b (mon.1) 719 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:13:16.587 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:15.360318+0000 mon.a (mon.0) 2957 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:13:16.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:15.266606+0000 mgr.y (mgr.14550) 494 : cluster [DBG] pgmap v837: 206 pgs: 206 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-19T21:13:16.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:15.331145+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e559: 8 total, 8 up, 6 in 2024-02-19T21:13:16.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:15.357303+0000 mon.b (mon.1) 717 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:16.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:15.357683+0000 mon.a (mon.0) 2955 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:16.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:15.358985+0000 mon.b (mon.1) 718 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:16.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:15.359364+0000 mon.a (mon.0) 2956 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:16.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:15.359959+0000 mon.b (mon.1) 719 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:13:16.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:15.360318+0000 mon.a (mon.0) 2957 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:13:17.586 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:13:17.590 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:17.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:13:17.691 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:17.722 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:16.104322+0000 mgr.y (mgr.14550) 495 : audit [DBG] from='client.23439 v1:172.21.15.163:0/1689343109' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:13:17.722 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:16.336162+0000 mon.a (mon.0) 2958 : audit [INF] from='client.48970 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:13:17.722 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:16.336263+0000 mon.a (mon.0) 2959 : cluster [DBG] osdmap e560: 8 total, 8 up, 6 in 2024-02-19T21:13:17.722 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:16.338076+0000 mon.b (mon.1) 720 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi163-83928-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:17.722 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:16.338470+0000 mon.a (mon.0) 2960 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi163-83928-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:17.724 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:17.745 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:17.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:16.104322+0000 mgr.y (mgr.14550) 495 : audit [DBG] from='client.23439 v1:172.21.15.163:0/1689343109' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:13:17.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:16.336162+0000 mon.a (mon.0) 2958 : audit [INF] from='client.48970 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:13:17.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:16.336263+0000 mon.a (mon.0) 2959 : cluster [DBG] osdmap e560: 8 total, 8 up, 6 in 2024-02-19T21:13:17.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:16.338076+0000 mon.b (mon.1) 720 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi163-83928-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:17.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:16.338470+0000 mon.a (mon.0) 2960 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi163-83928-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:17.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:16.104322+0000 mgr.y (mgr.14550) 495 : audit [DBG] from='client.23439 v1:172.21.15.163:0/1689343109' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:13:17.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:16.336162+0000 mon.a (mon.0) 2958 : audit [INF] from='client.48970 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:13:17.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:16.336263+0000 mon.a (mon.0) 2959 : cluster [DBG] osdmap e560: 8 total, 8 up, 6 in 2024-02-19T21:13:17.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:16.338076+0000 mon.b (mon.1) 720 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi163-83928-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:17.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:16.338470+0000 mon.a (mon.0) 2960 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi163-83928-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:17.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:17.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:17.849 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:17.879 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:17.899 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:17.901 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:17.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:18.002 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:13:18.033 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:18.070 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:13:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:13:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:13:18.130 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:13:18.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:18.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:13:18.145 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:13:18.145 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:13:18.235 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:13:18.263 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:18.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:17.266994+0000 mgr.y (mgr.14550) 496 : cluster [DBG] pgmap v840: 174 pgs: 174 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:18.480 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:17.579450+0000 mon.a (mon.0) 2961 : cluster [DBG] osdmap e561: 8 total, 8 up, 6 in 2024-02-19T21:13:18.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:17.266994+0000 mgr.y (mgr.14550) 496 : cluster [DBG] pgmap v840: 174 pgs: 174 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:18.480 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:17.579450+0000 mon.a (mon.0) 2961 : cluster [DBG] osdmap e561: 8 total, 8 up, 6 in 2024-02-19T21:13:18.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:17.266994+0000 mgr.y (mgr.14550) 496 : cluster [DBG] pgmap v840: 174 pgs: 174 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:18.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:17.579450+0000 mon.a (mon.0) 2961 : cluster [DBG] osdmap e561: 8 total, 8 up, 6 in 2024-02-19T21:13:18.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.971+0000 7ff96ee22700 1 -- 172.21.15.163:0/392543505 >> v1:172.21.15.163:6790/0 conn(0x7ff968100790 legacy=0x7ff968102b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:18.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.971+0000 7ff96ee22700 1 -- 172.21.15.163:0/392543505 shutdown_connections 2024-02-19T21:13:18.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.971+0000 7ff96ee22700 1 -- 172.21.15.163:0/392543505 >> 172.21.15.163:0/392543505 conn(0x7ff9680f3550 msgr2=0x7ff9680f5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:18.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.971+0000 7ff96ee22700 1 -- 172.21.15.163:0/392543505 shutdown_connections 2024-02-19T21:13:18.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.971+0000 7ff96ee22700 1 -- 172.21.15.163:0/392543505 wait complete. 2024-02-19T21:13:18.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.972+0000 7ff96ee22700 1 librados: starting msgr at 2024-02-19T21:13:18.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.972+0000 7ff96ee22700 1 librados: starting objecter 2024-02-19T21:13:18.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.972+0000 7ff96ee22700 1 Processor -- start 2024-02-19T21:13:18.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.972+0000 7ff96ee22700 1 -- start start 2024-02-19T21:13:18.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.972+0000 7ff96ee22700 1 librados: setting wanted keys 2024-02-19T21:13:18.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.972+0000 7ff96ee22700 1 librados: calling monclient init 2024-02-19T21:13:18.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.972+0000 7ff96ee22700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9680fd130 con 0x7ff9681042d0 2024-02-19T21:13:18.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.972+0000 7ff96ee22700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9680fd2c0 con 0x7ff9680fd990 2024-02-19T21:13:18.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.972+0000 7ff96ee22700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff96811bb90 con 0x7ff968100790 2024-02-19T21:13:18.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.972+0000 7ff967fff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7ff968100790 0x7ff968114d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:33326/0 (socket says 172.21.15.163:33326) 2024-02-19T21:13:18.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.972+0000 7ff967fff700 1 -- 172.21.15.163:0/2131631199 learned_addr learned my addr 172.21.15.163:0/2131631199 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:18.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.972+0000 7ff965ffb700 10 client.?.objecter ms_handle_connect 0x7ff9681042d0 2024-02-19T21:13:18.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.972+0000 7ff965ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:18.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 10 client.?.objecter ms_handle_connect 0x7ff968100790 2024-02-19T21:13:18.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:18.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 10 client.?.objecter ms_handle_connect 0x7ff9680fd990 2024-02-19T21:13:18.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:18.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252603814 0 0) 0x7ff9680fd130 con 0x7ff9681042d0 2024-02-19T21:13:18.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff958003030 con 0x7ff9681042d0 2024-02-19T21:13:18.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325943682 0 0) 0x7ff96811bb90 con 0x7ff968100790 2024-02-19T21:13:18.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9680fd130 con 0x7ff968100790 2024-02-19T21:13:18.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505115504 0 0) 0x7ff9680fd2c0 con 0x7ff9680fd990 2024-02-19T21:13:18.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff96811bb90 con 0x7ff9680fd990 2024-02-19T21:13:18.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2625072440 0 0) 0x7ff958003030 con 0x7ff9681042d0 2024-02-19T21:13:18.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9680fd2c0 con 0x7ff9681042d0 2024-02-19T21:13:18.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3145328225 0 0) 0x7ff9680fd130 con 0x7ff968100790 2024-02-19T21:13:18.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff958003030 con 0x7ff968100790 2024-02-19T21:13:18.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff960001b80 con 0x7ff9681042d0 2024-02-19T21:13:18.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff954002050 con 0x7ff968100790 2024-02-19T21:13:18.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 112328488 0 0) 0x7ff96811bb90 con 0x7ff9680fd990 2024-02-19T21:13:18.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9680fd130 con 0x7ff9680fd990 2024-02-19T21:13:18.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff95c0018e0 con 0x7ff9680fd990 2024-02-19T21:13:18.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 818574725 0 0) 0x7ff9680fd2c0 con 0x7ff9681042d0 2024-02-19T21:13:18.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 >> v1:172.21.15.163:6790/0 conn(0x7ff968100790 legacy=0x7ff968114d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:18.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 >> v1:172.21.15.193:6789/0 conn(0x7ff9680fd990 legacy=0x7ff9680fc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:18.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96811cd30 con 0x7ff9681042d0 2024-02-19T21:13:18.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff96811bd80 con 0x7ff9681042d0 2024-02-19T21:13:18.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff960002210 con 0x7ff9681042d0 2024-02-19T21:13:18.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff9600035b0 con 0x7ff9681042d0 2024-02-19T21:13:18.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff96ee22700 10 client.23445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:18.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff96811c1f0 con 0x7ff9681042d0 2024-02-19T21:13:18.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.973+0000 7ff96ee22700 1 librados: init done 2024-02-19T21:13:18.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.974+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ff96805f520 con 0x7ff9681042d0 2024-02-19T21:13:18.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ff960001d70 con 0x7ff9681042d0 2024-02-19T21:13:18.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(562..562 src has 434..562) v4 ==== 8131+0+0 (unknown 2056932227 0 0) 0x7ff960076810 con 0x7ff9681042d0 2024-02-19T21:13:18.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 osd_map(562..562 src has 434..562) v4 2024-02-19T21:13:18.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff965ffb700 3 client.23445.objecter handle_osd_map got epochs [562,562] > 0 2024-02-19T21:13:18.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff965ffb700 3 client.23445.objecter handle_osd_map decoding full epoch 562 2024-02-19T21:13:18.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff965ffb700 20 client.23445.objecter dump_active .. 0 homeless 2024-02-19T21:13:18.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v562) v1 ==== 54+0+16 (unknown 223374568 0 2420000192) 0x7ff960076bd0 con 0x7ff9681042d0 2024-02-19T21:13:18.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff965ffb700 10 client.23445.objecter ms_handle_connect 0x7ff958063d00 2024-02-19T21:13:18.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff96ee22700 10 client.23445.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:18.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 1 2024-02-19T21:13:18.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968047540 osd=0 3 2024-02-19T21:13:18.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968047540 osd=0 3 2024-02-19T21:13:18.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968047540 osd=0 3 2024-02-19T21:13:18.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 1 2024-02-19T21:13:18.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff968047540 osd=0 3 2024-02-19T21:13:18.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 0 1 2024-02-19T21:13:18.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968047540 osd=0 4 2024-02-19T21:13:18.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff96ee22700 10 client.23445.objecter _send_command 1 2024-02-19T21:13:18.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.975+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9680f91c0 con 0x7ff9680f3930 2024-02-19T21:13:18.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.976+0000 7ff965ffb700 10 client.23445.objecter ms_handle_connect 0x7ff9680f3930 2024-02-19T21:13:18.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.976+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff9680f91c0 con 0x7ff9680f3930 2024-02-19T21:13:18.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.976+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:13:18.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.976+0000 7ff965ffb700 10 client.23445.objecter _finish_command 1 = system:0 2024-02-19T21:13:18.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.976+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff968047540 osd=0 4 2024-02-19T21:13:18.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.976+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 0 1 2024-02-19T21:13:18.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.991+0000 7ff96ee22700 10 client.23445.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:18.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.991+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 2 2024-02-19T21:13:18.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.991+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968047540 osd=0 3 2024-02-19T21:13:18.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.991+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968047540 osd=0 3 2024-02-19T21:13:18.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.991+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968047540 osd=0 3 2024-02-19T21:13:18.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.991+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 2 2024-02-19T21:13:18.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.991+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff968047540 osd=0 3 2024-02-19T21:13:18.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.991+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 0 2 2024-02-19T21:13:18.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.991+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968047540 osd=0 4 2024-02-19T21:13:18.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.991+0000 7ff96ee22700 10 client.23445.objecter _send_command 2 2024-02-19T21:13:18.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.991+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff968109fd0 con 0x7ff9680f3930 2024-02-19T21:13:18.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff968109fd0 con 0x7ff9680f3930 2024-02-19T21:13:18.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:18.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff965ffb700 10 client.23445.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:18.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff968047540 osd=0 4 2024-02-19T21:13:18.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 0 2 2024-02-19T21:13:18.993 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:13:18.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff96ee22700 10 client.23445.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:18.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 3 2024-02-19T21:13:18.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff9680f4420 osd=1 3 2024-02-19T21:13:18.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff9680f4420 osd=1 3 2024-02-19T21:13:18.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff9680f4420 osd=1 3 2024-02-19T21:13:18.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 3 2024-02-19T21:13:18.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff9680f4420 osd=1 3 2024-02-19T21:13:18.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 1 3 2024-02-19T21:13:18.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff9680f4420 osd=1 4 2024-02-19T21:13:18.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff96ee22700 10 client.23445.objecter _send_command 3 2024-02-19T21:13:18.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff968120a90 con 0x7ff96811cf80 2024-02-19T21:13:18.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.993+0000 7ff965ffb700 10 client.23445.objecter ms_handle_connect 0x7ff96811cf80 2024-02-19T21:13:18.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.994+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff968120a90 con 0x7ff96811cf80 2024-02-19T21:13:18.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.994+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:13:18.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.994+0000 7ff965ffb700 10 client.23445.objecter _finish_command 3 = system:0 2024-02-19T21:13:18.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.994+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff9680f4420 osd=1 4 2024-02-19T21:13:18.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:18.994+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 1 3 2024-02-19T21:13:19.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.008+0000 7ff96ee22700 10 client.23445.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:19.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.008+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 4 2024-02-19T21:13:19.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.008+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff9680f4420 osd=1 3 2024-02-19T21:13:19.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.008+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff9680f4420 osd=1 3 2024-02-19T21:13:19.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.008+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff9680f4420 osd=1 3 2024-02-19T21:13:19.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.008+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 4 2024-02-19T21:13:19.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.008+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff9680f4420 osd=1 3 2024-02-19T21:13:19.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.008+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 1 4 2024-02-19T21:13:19.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.008+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff9680f4420 osd=1 4 2024-02-19T21:13:19.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.008+0000 7ff96ee22700 10 client.23445.objecter _send_command 4 2024-02-19T21:13:19.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.008+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff96811c6e0 con 0x7ff96811cf80 2024-02-19T21:13:19.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff96811c6e0 con 0x7ff96811cf80 2024-02-19T21:13:19.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:19.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff965ffb700 10 client.23445.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:19.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff9680f4420 osd=1 4 2024-02-19T21:13:19.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 1 4 2024-02-19T21:13:19.010 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:13:19.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff96ee22700 10 client.23445.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:19.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 5 2024-02-19T21:13:19.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff96811caf0 osd=2 3 2024-02-19T21:13:19.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff96811caf0 osd=2 3 2024-02-19T21:13:19.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff96811caf0 osd=2 3 2024-02-19T21:13:19.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 5 2024-02-19T21:13:19.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff96811caf0 osd=2 3 2024-02-19T21:13:19.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 2 5 2024-02-19T21:13:19.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff96811caf0 osd=2 4 2024-02-19T21:13:19.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff96ee22700 10 client.23445.objecter _send_command 5 2024-02-19T21:13:19.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9681254b0 con 0x7ff968059e00 2024-02-19T21:13:19.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.010+0000 7ff965ffb700 10 client.23445.objecter ms_handle_connect 0x7ff968059e00 2024-02-19T21:13:19.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.011+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff9681254b0 con 0x7ff968059e00 2024-02-19T21:13:19.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.011+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:13:19.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.011+0000 7ff965ffb700 10 client.23445.objecter _finish_command 5 = system:0 2024-02-19T21:13:19.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.011+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff96811caf0 osd=2 4 2024-02-19T21:13:19.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.011+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 2 5 2024-02-19T21:13:19.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.025+0000 7ff96ee22700 10 client.23445.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:19.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.025+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 6 2024-02-19T21:13:19.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.025+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff96811caf0 osd=2 3 2024-02-19T21:13:19.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.025+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff96811caf0 osd=2 3 2024-02-19T21:13:19.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.025+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff96811caf0 osd=2 3 2024-02-19T21:13:19.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.025+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 6 2024-02-19T21:13:19.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.025+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff96811caf0 osd=2 3 2024-02-19T21:13:19.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.025+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 2 6 2024-02-19T21:13:19.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.025+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff96811caf0 osd=2 4 2024-02-19T21:13:19.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.025+0000 7ff96ee22700 10 client.23445.objecter _send_command 6 2024-02-19T21:13:19.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.025+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff968059360 con 0x7ff968059e00 2024-02-19T21:13:19.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.026+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff968059360 con 0x7ff968059e00 2024-02-19T21:13:19.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.026+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:19.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.026+0000 7ff965ffb700 10 client.23445.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:19.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.026+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff96811caf0 osd=2 4 2024-02-19T21:13:19.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.026+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 2 6 2024-02-19T21:13:19.027 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:13:19.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff96ee22700 10 client.23445.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:19.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 7 2024-02-19T21:13:19.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968059770 osd=3 3 2024-02-19T21:13:19.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968059770 osd=3 3 2024-02-19T21:13:19.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968059770 osd=3 3 2024-02-19T21:13:19.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 7 2024-02-19T21:13:19.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff968059770 osd=3 3 2024-02-19T21:13:19.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 3 7 2024-02-19T21:13:19.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968059770 osd=3 4 2024-02-19T21:13:19.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff96ee22700 10 client.23445.objecter _send_command 7 2024-02-19T21:13:19.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff968129f30 con 0x7ff9681218e0 2024-02-19T21:13:19.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff965ffb700 10 client.23445.objecter ms_handle_connect 0x7ff9681218e0 2024-02-19T21:13:19.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff968129f30 con 0x7ff9681218e0 2024-02-19T21:13:19.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:13:19.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff965ffb700 10 client.23445.objecter _finish_command 7 = system:0 2024-02-19T21:13:19.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff968059770 osd=3 4 2024-02-19T21:13:19.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.027+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 3 7 2024-02-19T21:13:19.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.042+0000 7ff96ee22700 10 client.23445.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:19.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.042+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 8 2024-02-19T21:13:19.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.042+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968059770 osd=3 3 2024-02-19T21:13:19.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.042+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968059770 osd=3 3 2024-02-19T21:13:19.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.042+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968059770 osd=3 3 2024-02-19T21:13:19.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.042+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 8 2024-02-19T21:13:19.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.042+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff968059770 osd=3 3 2024-02-19T21:13:19.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.042+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 3 8 2024-02-19T21:13:19.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.042+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968059770 osd=3 4 2024-02-19T21:13:19.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.042+0000 7ff96ee22700 10 client.23445.objecter _send_command 8 2024-02-19T21:13:19.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.042+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff968120e40 con 0x7ff9681218e0 2024-02-19T21:13:19.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.043+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff968120e40 con 0x7ff9681218e0 2024-02-19T21:13:19.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.043+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:19.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.043+0000 7ff965ffb700 10 client.23445.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:19.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.043+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff968059770 osd=3 4 2024-02-19T21:13:19.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.043+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 3 8 2024-02-19T21:13:19.044 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:13:19.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.044+0000 7ff96ee22700 10 client.23445.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:19.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.044+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 9 2024-02-19T21:13:19.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.044+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968120e40 osd=4 3 2024-02-19T21:13:19.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.044+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968120e40 osd=4 3 2024-02-19T21:13:19.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.044+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968120e40 osd=4 3 2024-02-19T21:13:19.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.044+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 9 2024-02-19T21:13:19.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.044+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff968120e40 osd=4 3 2024-02-19T21:13:19.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.044+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 4 9 2024-02-19T21:13:19.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.044+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968120e40 osd=4 4 2024-02-19T21:13:19.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.044+0000 7ff96ee22700 10 client.23445.objecter _send_command 9 2024-02-19T21:13:19.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.044+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff968121110 con 0x7ff968126300 2024-02-19T21:13:19.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.044+0000 7ff965ffb700 10 client.23445.objecter ms_handle_connect 0x7ff968126300 2024-02-19T21:13:19.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.045+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff968121110 con 0x7ff968126300 2024-02-19T21:13:19.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.045+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:13:19.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.045+0000 7ff965ffb700 10 client.23445.objecter _finish_command 9 = system:0 2024-02-19T21:13:19.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.045+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff968120e40 osd=4 4 2024-02-19T21:13:19.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.045+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 4 9 2024-02-19T21:13:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.059+0000 7ff96ee22700 10 client.23445.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.059+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 10 2024-02-19T21:13:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.059+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968120e40 osd=4 3 2024-02-19T21:13:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.059+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968120e40 osd=4 3 2024-02-19T21:13:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.059+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968120e40 osd=4 3 2024-02-19T21:13:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.059+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 10 2024-02-19T21:13:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.059+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff968120e40 osd=4 3 2024-02-19T21:13:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.059+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 4 10 2024-02-19T21:13:19.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.059+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968120e40 osd=4 4 2024-02-19T21:13:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.059+0000 7ff96ee22700 10 client.23445.objecter _send_command 10 2024-02-19T21:13:19.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.059+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff968121110 con 0x7ff968126300 2024-02-19T21:13:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff968121110 con 0x7ff968126300 2024-02-19T21:13:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff965ffb700 10 client.23445.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff968120e40 osd=4 4 2024-02-19T21:13:19.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 4 10 2024-02-19T21:13:19.061 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:13:19.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff96ee22700 10 client.23445.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:19.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 11 2024-02-19T21:13:19.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968121110 osd=5 3 2024-02-19T21:13:19.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968121110 osd=5 3 2024-02-19T21:13:19.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968121110 osd=5 3 2024-02-19T21:13:19.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 11 2024-02-19T21:13:19.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff968121110 osd=5 3 2024-02-19T21:13:19.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 5 11 2024-02-19T21:13:19.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968121110 osd=5 4 2024-02-19T21:13:19.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff96ee22700 10 client.23445.objecter _send_command 11 2024-02-19T21:13:19.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.061+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff96812b290 con 0x7ff96812acf0 2024-02-19T21:13:19.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.062+0000 7ff965ffb700 10 client.23445.objecter ms_handle_connect 0x7ff96812acf0 2024-02-19T21:13:19.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.063+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff96812b290 con 0x7ff96812acf0 2024-02-19T21:13:19.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.063+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:13:19.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.063+0000 7ff965ffb700 10 client.23445.objecter _finish_command 11 = system:0 2024-02-19T21:13:19.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.063+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff968121110 osd=5 4 2024-02-19T21:13:19.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.063+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 5 11 2024-02-19T21:13:19.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.079+0000 7ff96ee22700 10 client.23445.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:19.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.079+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 12 2024-02-19T21:13:19.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.079+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968121110 osd=5 3 2024-02-19T21:13:19.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.079+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968121110 osd=5 3 2024-02-19T21:13:19.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.079+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968121110 osd=5 3 2024-02-19T21:13:19.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.079+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 12 2024-02-19T21:13:19.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.079+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff968121110 osd=5 3 2024-02-19T21:13:19.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.079+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 5 12 2024-02-19T21:13:19.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.079+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968121110 osd=5 4 2024-02-19T21:13:19.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.079+0000 7ff96ee22700 10 client.23445.objecter _send_command 12 2024-02-19T21:13:19.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.079+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff96812b290 con 0x7ff96812acf0 2024-02-19T21:13:19.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff96812b290 con 0x7ff96812acf0 2024-02-19T21:13:19.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:19.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff965ffb700 10 client.23445.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:19.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff968121110 osd=5 4 2024-02-19T21:13:19.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 5 12 2024-02-19T21:13:19.081 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:13:19.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff96ee22700 10 client.23445.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:19.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 13 2024-02-19T21:13:19.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff96812b290 osd=6 3 2024-02-19T21:13:19.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff96812b290 osd=6 3 2024-02-19T21:13:19.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff96812b290 osd=6 3 2024-02-19T21:13:19.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 13 2024-02-19T21:13:19.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff96812b290 osd=6 3 2024-02-19T21:13:19.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 6 13 2024-02-19T21:13:19.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff96812b290 osd=6 4 2024-02-19T21:13:19.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff96ee22700 10 client.23445.objecter _send_command 13 2024-02-19T21:13:19.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.081+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff968137080 con 0x7ff968133530 2024-02-19T21:13:19.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.082+0000 7ff965ffb700 10 client.23445.objecter ms_handle_connect 0x7ff968133530 2024-02-19T21:13:19.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.083+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff968137080 con 0x7ff968133530 2024-02-19T21:13:19.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.083+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:13:19.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.083+0000 7ff965ffb700 10 client.23445.objecter _finish_command 13 = system:0 2024-02-19T21:13:19.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.083+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff96812b290 osd=6 4 2024-02-19T21:13:19.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.083+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 6 13 2024-02-19T21:13:19.089 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":838,"stamp":"2024-02-19T21:13:15.331166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":193,"num_object_clones":0,"num_object_copies":579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":1833,"num_read_kb":1585,"num_write":720,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":11520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":819,"ondisk_log_size":819,"up":522,"acting":522,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":698,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":1904044,"kb_used_data":48852,"kb_used_omap":0,"kb_used_meta":1848832,"kb_avail":560566868,"statfs":{"total":575970213888,"available":574020472832,"internally_reserved":0,"allocated":50024448,"data_stored":32506906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1893203968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4020435,"num_objects":-193,"num_object_clones":-11,"num_object_copies":-579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-193,"num_whiteouts":0,"num_read":-207,"num_read_kb":-51,"num_write":-779,"num_write_kb":-4619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1135,"num_bytes_recovered":-5695,"num_keys_recovered":0,"num_objects_omap":-1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":-1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-21,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.846590"},"pg_stats":[{"pgid":"2.29","version":"0'0","reported_seq":561,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.508244+0000","last_change":"2024-02-19T21:12:09.321398+0000","last_active":"2024-02-19T21:13:13.508244+0000","last_peered":"2024-02-19T21:13:13.508244+0000","last_clean":"2024-02-19T21:13:13.508244+0000","last_became_active":"2024-02-19T21:12:09.320924+0000","last_became_peered":"2024-02-19T21:12:09.320924+0000","last_unstale":"2024-02-19T21:13:13.508244+0000","last_undegraded":"2024-02-19T21:13:13.508244+0000","last_fullsized":"2024-02-19T21:13:13.508244+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.738600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.738600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":91,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.508051+0000","last_change":"2024-02-19T21:11:50.304127+0000","last_active":"2024-02-19T21:13:13.508051+0000","last_peered":"2024-02-19T21:13:13.508051+0000","last_clean":"2024-02-19T21:13:13.508051+0000","last_became_active":"2024-02-19T21:11:50.303790+0000","last_became_peered":"2024-02-19T21:11:50.303790+0000","last_unstale":"2024-02-19T21:13:13.508051+0000","last_undegraded":"2024-02-19T21:13:13.508051+0000","last_fullsized":"2024-02-19T21:13:13.508051+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:42.529489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:42.529489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.25","version":"0'0","reported_seq":594,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.503568+0000","last_change":"2024-02-19T21:12:09.349611+0000","last_active":"2024-02-19T21:13:13.503568+0000","last_peered":"2024-02-19T21:13:13.503568+0000","last_clean":"2024-02-19T21:13:13.503568+0000","last_became_active":"2024-02-19T21:12:09.349271+0000","last_became_peered":"2024-02-19T21:12:09.349271+0000","last_unstale":"2024-02-19T21:13:13.503568+0000","last_undegraded":"2024-02-19T21:13:13.503568+0000","last_fullsized":"2024-02-19T21:13:13.503568+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:40.580344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:40.580344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":568,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.522062+0000","last_change":"2024-02-19T21:12:09.339659+0000","last_active":"2024-02-19T21:13:13.522062+0000","last_peered":"2024-02-19T21:13:13.522062+0000","last_clean":"2024-02-19T21:13:13.522062+0000","last_became_active":"2024-02-19T21:12:09.339310+0000","last_became_peered":"2024-02-19T21:12:09.339310+0000","last_unstale":"2024-02-19T21:13:13.522062+0000","last_undegraded":"2024-02-19T21:13:13.522062+0000","last_fullsized":"2024-02-19T21:13:13.522062+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:21.732778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:10:21.732778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":560,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.531425+0000","last_change":"2024-02-19T21:11:50.313926+0000","last_active":"2024-02-19T21:13:13.531425+0000","last_peered":"2024-02-19T21:13:13.531425+0000","last_clean":"2024-02-19T21:13:13.531425+0000","last_became_active":"2024-02-19T21:11:50.313560+0000","last_became_peered":"2024-02-19T21:11:50.313560+0000","last_unstale":"2024-02-19T21:13:13.531425+0000","last_undegraded":"2024-02-19T21:13:13.531425+0000","last_fullsized":"2024-02-19T21:13:13.531425+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.364732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.364732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.26","version":"0'0","reported_seq":97,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.531439+0000","last_change":"2024-02-19T21:12:09.339184+0000","last_active":"2024-02-19T21:13:13.531439+0000","last_peered":"2024-02-19T21:13:13.531439+0000","last_clean":"2024-02-19T21:13:13.531439+0000","last_became_active":"2024-02-19T21:12:09.338737+0000","last_became_peered":"2024-02-19T21:12:09.338737+0000","last_unstale":"2024-02-19T21:13:13.531439+0000","last_undegraded":"2024-02-19T21:13:13.531439+0000","last_fullsized":"2024-02-19T21:13:13.531439+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":601,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.519516+0000","last_change":"2024-02-19T21:12:09.343445+0000","last_active":"2024-02-19T21:13:13.519516+0000","last_peered":"2024-02-19T21:13:13.519516+0000","last_clean":"2024-02-19T21:13:13.519516+0000","last_became_active":"2024-02-19T21:12:09.342583+0000","last_became_peered":"2024-02-19T21:12:09.342583+0000","last_unstale":"2024-02-19T21:13:13.519516+0000","last_undegraded":"2024-02-19T21:13:13.519516+0000","last_fullsized":"2024-02-19T21:13:13.519516+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.955872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.955872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.20","version":"397'2","reported_seq":622,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.529896+0000","last_change":"2024-02-19T21:12:43.380032+0000","last_active":"2024-02-19T21:13:13.529896+0000","last_peered":"2024-02-19T21:13:13.529896+0000","last_clean":"2024-02-19T21:13:13.529896+0000","last_became_active":"2024-02-19T21:12:43.379693+0000","last_became_peered":"2024-02-19T21:12:43.379693+0000","last_unstale":"2024-02-19T21:13:13.529896+0000","last_undegraded":"2024-02-19T21:13:13.529896+0000","last_fullsized":"2024-02-19T21:13:13.529896+0000","mapping_epoch":526,"log_start":"58'1","ondisk_log_start":"58'1","created":474,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":6,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.934474+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":437,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":100,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.510668+0000","last_change":"2024-02-19T21:12:09.337725+0000","last_active":"2024-02-19T21:13:13.510668+0000","last_peered":"2024-02-19T21:13:13.510668+0000","last_clean":"2024-02-19T21:13:13.510668+0000","last_became_active":"2024-02-19T21:12:09.337337+0000","last_became_peered":"2024-02-19T21:12:09.337337+0000","last_unstale":"2024-02-19T21:13:13.510668+0000","last_undegraded":"2024-02-19T21:13:13.510668+0000","last_fullsized":"2024-02-19T21:13:13.510668+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.081993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.081993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":549,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.502507+0000","last_change":"2024-02-19T21:12:09.353297+0000","last_active":"2024-02-19T21:13:13.502507+0000","last_peered":"2024-02-19T21:13:13.502507+0000","last_clean":"2024-02-19T21:13:13.502507+0000","last_became_active":"2024-02-19T21:12:09.352904+0000","last_became_peered":"2024-02-19T21:12:09.352904+0000","last_unstale":"2024-02-19T21:13:13.502507+0000","last_undegraded":"2024-02-19T21:13:13.502507+0000","last_fullsized":"2024-02-19T21:13:13.502507+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.089715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.089715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"493'12","reported_seq":716,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-02-19T21:13:14.514664+0000","last_change":"2024-02-19T21:12:18.452113+0000","last_active":"2024-02-19T21:13:14.514664+0000","last_peered":"2024-02-19T21:13:14.514664+0000","last_clean":"2024-02-19T21:13:14.514664+0000","last_became_active":"2024-02-19T21:12:09.332507+0000","last_became_peered":"2024-02-19T21:12:09.332507+0000","last_unstale":"2024-02-19T21:13:14.514664+0000","last_undegraded":"2024-02-19T21:13:14.514664+0000","last_fullsized":"2024-02-19T21:13:14.514664+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'12","last_scrub_stamp":"2024-02-19T21:12:18.452020+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:12:18.452020+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'28","reported_seq":593,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325043+0000","last_change":"2024-02-19T21:12:10.211768+0000","last_active":"2024-02-19T21:13:13.325043+0000","last_peered":"2024-02-19T21:13:13.325043+0000","last_clean":"2024-02-19T21:13:13.325043+0000","last_became_active":"2024-02-19T21:12:09.341122+0000","last_became_peered":"2024-02-19T21:12:09.341122+0000","last_unstale":"2024-02-19T21:13:13.325043+0000","last_undegraded":"2024-02-19T21:13:13.325043+0000","last_fullsized":"2024-02-19T21:13:13.325043+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-19T21:11:47.051823+0000","last_deep_scrub":"473'28","last_deep_scrub_stamp":"2024-02-19T21:11:47.051823+0000","last_clean_scrub_stamp":"2024-02-19T21:11:47.051823+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":557,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.518308+0000","last_change":"2024-02-19T21:12:22.215650+0000","last_active":"2024-02-19T21:13:13.518308+0000","last_peered":"2024-02-19T21:13:13.518308+0000","last_clean":"2024-02-19T21:13:13.518308+0000","last_became_active":"2024-02-19T21:10:40.299425+0000","last_became_peered":"2024-02-19T21:10:40.299425+0000","last_unstale":"2024-02-19T21:13:13.518308+0000","last_undegraded":"2024-02-19T21:13:13.518308+0000","last_fullsized":"2024-02-19T21:13:13.518308+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:22.215548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_clean_scrub_stamp":"2024-02-19T21:12:22.215548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":535,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325348+0000","last_change":"2024-02-19T21:12:51.469049+0000","last_active":"2024-02-19T21:13:13.325348+0000","last_peered":"2024-02-19T21:13:13.325348+0000","last_clean":"2024-02-19T21:13:13.325348+0000","last_became_active":"2024-02-19T21:12:09.326173+0000","last_became_peered":"2024-02-19T21:12:09.326173+0000","last_unstale":"2024-02-19T21:13:13.325348+0000","last_undegraded":"2024-02-19T21:13:13.325348+0000","last_fullsized":"2024-02-19T21:13:13.325348+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:51.468971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.468971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":478,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.520262+0000","last_change":"2024-02-19T21:12:58.081198+0000","last_active":"2024-02-19T21:13:13.520262+0000","last_peered":"2024-02-19T21:13:13.520262+0000","last_clean":"2024-02-19T21:13:13.520262+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:13:13.520262+0000","last_undegraded":"2024-02-19T21:13:13.520262+0000","last_fullsized":"2024-02-19T21:13:13.520262+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:58.081106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:58.081106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'16","reported_seq":448,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.523014+0000","last_change":"2024-02-19T21:11:59.756726+0000","last_active":"2024-02-19T21:13:13.523014+0000","last_peered":"2024-02-19T21:13:13.523014+0000","last_clean":"2024-02-19T21:13:13.523014+0000","last_became_active":"2024-02-19T21:11:59.319397+0000","last_became_peered":"2024-02-19T21:11:59.319397+0000","last_unstale":"2024-02-19T21:13:13.523014+0000","last_undegraded":"2024-02-19T21:13:13.523014+0000","last_fullsized":"2024-02-19T21:13:13.523014+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'16","last_scrub_stamp":"2024-02-19T21:11:46.956424+0000","last_deep_scrub":"473'16","last_deep_scrub_stamp":"2024-02-19T21:11:46.956424+0000","last_clean_scrub_stamp":"2024-02-19T21:11:46.956424+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":563,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.524064+0000","last_change":"2024-02-19T21:12:06.016884+0000","last_active":"2024-02-19T21:13:13.524064+0000","last_peered":"2024-02-19T21:13:13.524064+0000","last_clean":"2024-02-19T21:13:13.524064+0000","last_became_active":"2024-02-19T21:10:41.312116+0000","last_became_peered":"2024-02-19T21:10:41.312116+0000","last_unstale":"2024-02-19T21:13:13.524064+0000","last_undegraded":"2024-02-19T21:13:13.524064+0000","last_fullsized":"2024-02-19T21:13:13.524064+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.016817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.016817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":562,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.517982+0000","last_change":"2024-02-19T21:13:05.193297+0000","last_active":"2024-02-19T21:13:13.517982+0000","last_peered":"2024-02-19T21:13:13.517982+0000","last_clean":"2024-02-19T21:13:13.517982+0000","last_became_active":"2024-02-19T21:10:41.305574+0000","last_became_peered":"2024-02-19T21:10:41.305574+0000","last_unstale":"2024-02-19T21:13:13.517982+0000","last_undegraded":"2024-02-19T21:13:13.517982+0000","last_fullsized":"2024-02-19T21:13:13.517982+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:05.193223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:05.193223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":363,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.530049+0000","last_change":"2024-02-19T21:11:59.323172+0000","last_active":"2024-02-19T21:13:13.530049+0000","last_peered":"2024-02-19T21:13:13.530049+0000","last_clean":"2024-02-19T21:13:13.530049+0000","last_became_active":"2024-02-19T21:11:59.322814+0000","last_became_peered":"2024-02-19T21:11:59.322814+0000","last_unstale":"2024-02-19T21:13:13.530049+0000","last_undegraded":"2024-02-19T21:13:13.530049+0000","last_fullsized":"2024-02-19T21:13:13.530049+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:34.504478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:34.504478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":471,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.327515+0000","last_change":"2024-02-19T21:10:57.130256+0000","last_active":"2024-02-19T21:13:13.327515+0000","last_peered":"2024-02-19T21:13:13.327515+0000","last_clean":"2024-02-19T21:13:13.327515+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:13:13.327515+0000","last_undegraded":"2024-02-19T21:13:13.327515+0000","last_fullsized":"2024-02-19T21:13:13.327515+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.130171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.130171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"459'35","reported_seq":576,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.521925+0000","last_change":"2024-02-19T21:12:10.552781+0000","last_active":"2024-02-19T21:13:13.521925+0000","last_peered":"2024-02-19T21:13:13.521925+0000","last_clean":"2024-02-19T21:13:13.521925+0000","last_became_active":"2024-02-19T21:12:09.333541+0000","last_became_peered":"2024-02-19T21:12:09.333541+0000","last_unstale":"2024-02-19T21:13:13.521925+0000","last_undegraded":"2024-02-19T21:13:13.521925+0000","last_fullsized":"2024-02-19T21:13:13.521925+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:11:04.345836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:04.345836+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":539,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.519829+0000","last_change":"2024-02-19T21:12:09.339263+0000","last_active":"2024-02-19T21:13:13.519829+0000","last_peered":"2024-02-19T21:13:13.519829+0000","last_clean":"2024-02-19T21:13:13.519829+0000","last_became_active":"2024-02-19T21:11:59.336426+0000","last_became_peered":"2024-02-19T21:11:59.336426+0000","last_unstale":"2024-02-19T21:13:13.519829+0000","last_undegraded":"2024-02-19T21:13:13.519829+0000","last_fullsized":"2024-02-19T21:13:13.519829+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:09.339147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:12:09.339147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":532,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.520534+0000","last_change":"2024-02-19T21:12:16.244173+0000","last_active":"2024-02-19T21:13:13.520534+0000","last_peered":"2024-02-19T21:13:13.520534+0000","last_clean":"2024-02-19T21:13:13.520534+0000","last_became_active":"2024-02-19T21:11:59.340544+0000","last_became_peered":"2024-02-19T21:11:59.340544+0000","last_unstale":"2024-02-19T21:13:13.520534+0000","last_undegraded":"2024-02-19T21:13:13.520534+0000","last_fullsized":"2024-02-19T21:13:13.520534+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:16.244104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:16.244104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":534,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.530177+0000","last_change":"2024-02-19T21:11:33.469310+0000","last_active":"2024-02-19T21:13:13.530177+0000","last_peered":"2024-02-19T21:13:13.530177+0000","last_clean":"2024-02-19T21:13:13.530177+0000","last_became_active":"2024-02-19T21:08:04.265081+0000","last_became_peered":"2024-02-19T21:08:04.265081+0000","last_unstale":"2024-02-19T21:13:13.530177+0000","last_undegraded":"2024-02-19T21:13:13.530177+0000","last_fullsized":"2024-02-19T21:13:13.530177+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:33.469232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_clean_scrub_stamp":"2024-02-19T21:11:33.469232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":494,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.531564+0000","last_change":"2024-02-19T21:12:55.491781+0000","last_active":"2024-02-19T21:13:13.531564+0000","last_peered":"2024-02-19T21:13:13.531564+0000","last_clean":"2024-02-19T21:13:13.531564+0000","last_became_active":"2024-02-19T21:10:41.317284+0000","last_became_peered":"2024-02-19T21:10:41.317284+0000","last_unstale":"2024-02-19T21:13:13.531564+0000","last_undegraded":"2024-02-19T21:13:13.531564+0000","last_fullsized":"2024-02-19T21:13:13.531564+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:55.491665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.491665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"459'31","reported_seq":177,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.504313+0000","last_change":"2024-02-19T21:11:44.421114+0000","last_active":"2024-02-19T21:13:13.504313+0000","last_peered":"2024-02-19T21:13:13.504313+0000","last_clean":"2024-02-19T21:13:13.504313+0000","last_became_active":"2024-02-19T21:10:41.313853+0000","last_became_peered":"2024-02-19T21:10:41.313853+0000","last_unstale":"2024-02-19T21:13:13.504313+0000","last_undegraded":"2024-02-19T21:13:13.504313+0000","last_fullsized":"2024-02-19T21:13:13.504313+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'31","last_scrub_stamp":"2024-02-19T21:11:44.421033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:44.421033+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":559,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.530772+0000","last_change":"2024-02-19T21:13:07.450202+0000","last_active":"2024-02-19T21:13:13.530772+0000","last_peered":"2024-02-19T21:13:13.530772+0000","last_clean":"2024-02-19T21:13:13.530772+0000","last_became_active":"2024-02-19T21:10:41.307517+0000","last_became_peered":"2024-02-19T21:10:41.307517+0000","last_unstale":"2024-02-19T21:13:13.530772+0000","last_undegraded":"2024-02-19T21:13:13.530772+0000","last_fullsized":"2024-02-19T21:13:13.530772+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:13:07.450123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.450123+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":559,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.522544+0000","last_change":"2024-02-19T21:12:58.838081+0000","last_active":"2024-02-19T21:13:13.522544+0000","last_peered":"2024-02-19T21:13:13.522544+0000","last_clean":"2024-02-19T21:13:13.522544+0000","last_became_active":"2024-02-19T21:12:09.369409+0000","last_became_peered":"2024-02-19T21:12:09.369409+0000","last_unstale":"2024-02-19T21:13:13.522544+0000","last_undegraded":"2024-02-19T21:13:13.522544+0000","last_fullsized":"2024-02-19T21:13:13.522544+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:58.837973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:58.837973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":560,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.518396+0000","last_change":"2024-02-19T21:13:03.126485+0000","last_active":"2024-02-19T21:13:13.518396+0000","last_peered":"2024-02-19T21:13:13.518396+0000","last_clean":"2024-02-19T21:13:13.518396+0000","last_became_active":"2024-02-19T21:10:41.307577+0000","last_became_peered":"2024-02-19T21:10:41.307577+0000","last_unstale":"2024-02-19T21:13:13.518396+0000","last_undegraded":"2024-02-19T21:13:13.518396+0000","last_fullsized":"2024-02-19T21:13:13.518396+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:03.126414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:13:03.126414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":138,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.502963+0000","last_change":"2024-02-19T21:10:50.424462+0000","last_active":"2024-02-19T21:13:13.502963+0000","last_peered":"2024-02-19T21:13:13.502963+0000","last_clean":"2024-02-19T21:13:13.502963+0000","last_became_active":"2024-02-19T21:10:41.302670+0000","last_became_peered":"2024-02-19T21:10:41.302670+0000","last_unstale":"2024-02-19T21:13:13.502963+0000","last_undegraded":"2024-02-19T21:13:13.502963+0000","last_fullsized":"2024-02-19T21:13:13.502963+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:50.424381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.424381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"459'25","reported_seq":186,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.503519+0000","last_change":"2024-02-19T21:12:51.420519+0000","last_active":"2024-02-19T21:13:13.503519+0000","last_peered":"2024-02-19T21:13:13.503519+0000","last_clean":"2024-02-19T21:13:13.503519+0000","last_became_active":"2024-02-19T21:12:09.408620+0000","last_became_peered":"2024-02-19T21:12:09.408620+0000","last_unstale":"2024-02-19T21:13:13.503519+0000","last_undegraded":"2024-02-19T21:13:13.503519+0000","last_fullsized":"2024-02-19T21:13:13.503519+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'25","last_scrub_stamp":"2024-02-19T21:12:51.420403+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.420403+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":543,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.522000+0000","last_change":"2024-02-19T21:13:13.044378+0000","last_active":"2024-02-19T21:13:13.522000+0000","last_peered":"2024-02-19T21:13:13.522000+0000","last_clean":"2024-02-19T21:13:13.522000+0000","last_became_active":"2024-02-19T21:08:04.268406+0000","last_became_peered":"2024-02-19T21:08:04.268406+0000","last_unstale":"2024-02-19T21:13:13.522000+0000","last_undegraded":"2024-02-19T21:13:13.522000+0000","last_fullsized":"2024-02-19T21:13:13.522000+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:12.628801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:12.628801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":678,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-02-19T21:13:14.331294+0000","last_change":"2024-02-19T21:12:21.202447+0000","last_active":"2024-02-19T21:13:14.331294+0000","last_peered":"2024-02-19T21:13:14.331294+0000","last_clean":"2024-02-19T21:13:14.331294+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:13:14.331294+0000","last_undegraded":"2024-02-19T21:13:14.331294+0000","last_fullsized":"2024-02-19T21:13:14.331294+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:12:21.202371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:21.202371+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":138,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.503772+0000","last_change":"2024-02-19T21:12:21.380805+0000","last_active":"2024-02-19T21:13:13.503772+0000","last_peered":"2024-02-19T21:13:13.503772+0000","last_clean":"2024-02-19T21:13:13.503772+0000","last_became_active":"2024-02-19T21:10:41.298893+0000","last_became_peered":"2024-02-19T21:10:41.298893+0000","last_unstale":"2024-02-19T21:13:13.503772+0000","last_undegraded":"2024-02-19T21:13:13.503772+0000","last_fullsized":"2024-02-19T21:13:13.503772+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:21.380734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:12:21.380734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":285,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.508217+0000","last_change":"2024-02-19T21:12:16.339644+0000","last_active":"2024-02-19T21:13:13.508217+0000","last_peered":"2024-02-19T21:13:13.508217+0000","last_clean":"2024-02-19T21:13:13.508217+0000","last_became_active":"2024-02-19T21:11:59.318845+0000","last_became_peered":"2024-02-19T21:11:59.318845+0000","last_unstale":"2024-02-19T21:13:13.508217+0000","last_undegraded":"2024-02-19T21:13:13.508217+0000","last_fullsized":"2024-02-19T21:13:13.508217+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:16.339577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:16.339577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"459'27","reported_seq":182,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.517819+0000","last_change":"2024-02-19T21:11:08.202280+0000","last_active":"2024-02-19T21:13:13.517819+0000","last_peered":"2024-02-19T21:13:13.517819+0000","last_clean":"2024-02-19T21:13:13.517819+0000","last_became_active":"2024-02-19T21:10:41.307576+0000","last_became_peered":"2024-02-19T21:10:41.307576+0000","last_unstale":"2024-02-19T21:13:13.517819+0000","last_undegraded":"2024-02-19T21:13:13.517819+0000","last_fullsized":"2024-02-19T21:13:13.517819+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:11:08.202208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:08.202208+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":585,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.522806+0000","last_change":"2024-02-19T21:12:40.370031+0000","last_active":"2024-02-19T21:13:13.522806+0000","last_peered":"2024-02-19T21:13:13.522806+0000","last_clean":"2024-02-19T21:13:13.522806+0000","last_became_active":"2024-02-19T21:12:40.369640+0000","last_became_peered":"2024-02-19T21:12:40.369640+0000","last_unstale":"2024-02-19T21:13:13.522806+0000","last_undegraded":"2024-02-19T21:13:13.522806+0000","last_fullsized":"2024-02-19T21:13:13.522806+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.738600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.738600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"483'11","reported_seq":187,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-02-19T21:13:14.332060+0000","last_change":"2024-02-19T21:12:56.450133+0000","last_active":"2024-02-19T21:13:14.332060+0000","last_peered":"2024-02-19T21:13:14.332060+0000","last_clean":"2024-02-19T21:13:14.332060+0000","last_became_active":"2024-02-19T21:11:59.336645+0000","last_became_peered":"2024-02-19T21:11:59.336645+0000","last_unstale":"2024-02-19T21:13:14.332060+0000","last_undegraded":"2024-02-19T21:13:14.332060+0000","last_fullsized":"2024-02-19T21:13:14.332060+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"483'11","last_scrub_stamp":"2024-02-19T21:12:56.450044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:56.450044+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":531,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.326568+0000","last_change":"2024-02-19T21:12:54.437089+0000","last_active":"2024-02-19T21:13:13.326568+0000","last_peered":"2024-02-19T21:13:13.326568+0000","last_clean":"2024-02-19T21:13:13.326568+0000","last_became_active":"2024-02-19T21:11:59.344440+0000","last_became_peered":"2024-02-19T21:11:59.344440+0000","last_unstale":"2024-02-19T21:13:13.326568+0000","last_undegraded":"2024-02-19T21:13:13.326568+0000","last_fullsized":"2024-02-19T21:13:13.326568+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:54.436977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:54.436977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":480,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325492+0000","last_change":"2024-02-19T21:12:09.327574+0000","last_active":"2024-02-19T21:13:13.325492+0000","last_peered":"2024-02-19T21:13:13.325492+0000","last_clean":"2024-02-19T21:13:13.325492+0000","last_became_active":"2024-02-19T21:12:09.326909+0000","last_became_peered":"2024-02-19T21:12:09.326909+0000","last_unstale":"2024-02-19T21:13:13.325492+0000","last_undegraded":"2024-02-19T21:13:13.325492+0000","last_fullsized":"2024-02-19T21:13:13.325492+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.079353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.079353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"459'19","reported_seq":185,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.503733+0000","last_change":"2024-02-19T21:12:10.647686+0000","last_active":"2024-02-19T21:13:13.503733+0000","last_peered":"2024-02-19T21:13:13.503733+0000","last_clean":"2024-02-19T21:13:13.503733+0000","last_became_active":"2024-02-19T21:12:09.409065+0000","last_became_peered":"2024-02-19T21:12:09.409065+0000","last_unstale":"2024-02-19T21:13:13.503733+0000","last_undegraded":"2024-02-19T21:13:13.503733+0000","last_fullsized":"2024-02-19T21:13:13.503733+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:11:42.333781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:42.333781+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":572,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.530357+0000","last_change":"2024-02-19T21:11:59.332430+0000","last_active":"2024-02-19T21:13:13.530357+0000","last_peered":"2024-02-19T21:13:13.530357+0000","last_clean":"2024-02-19T21:13:13.530357+0000","last_became_active":"2024-02-19T21:11:59.331675+0000","last_became_peered":"2024-02-19T21:11:59.331675+0000","last_unstale":"2024-02-19T21:13:13.530357+0000","last_undegraded":"2024-02-19T21:13:13.530357+0000","last_fullsized":"2024-02-19T21:13:13.530357+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:42.529489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:42.529489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":545,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.327613+0000","last_change":"2024-02-19T21:12:09.350127+0000","last_active":"2024-02-19T21:13:13.327613+0000","last_peered":"2024-02-19T21:13:13.327613+0000","last_clean":"2024-02-19T21:13:13.327613+0000","last_became_active":"2024-02-19T21:12:09.349667+0000","last_became_peered":"2024-02-19T21:12:09.349667+0000","last_unstale":"2024-02-19T21:13:13.327613+0000","last_undegraded":"2024-02-19T21:13:13.327613+0000","last_fullsized":"2024-02-19T21:13:13.327613+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:58.096025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.096025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":278,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.511818+0000","last_change":"2024-02-19T21:12:09.335125+0000","last_active":"2024-02-19T21:13:13.511818+0000","last_peered":"2024-02-19T21:13:13.511818+0000","last_clean":"2024-02-19T21:13:13.511818+0000","last_became_active":"2024-02-19T21:12:09.333625+0000","last_became_peered":"2024-02-19T21:12:09.333625+0000","last_unstale":"2024-02-19T21:13:13.511818+0000","last_undegraded":"2024-02-19T21:13:13.511818+0000","last_fullsized":"2024-02-19T21:13:13.511818+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:06.547494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:06.547494+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":278,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.510163+0000","last_change":"2024-02-19T21:12:09.348064+0000","last_active":"2024-02-19T21:13:13.510163+0000","last_peered":"2024-02-19T21:13:13.510163+0000","last_clean":"2024-02-19T21:13:13.510163+0000","last_became_active":"2024-02-19T21:12:09.347240+0000","last_became_peered":"2024-02-19T21:12:09.347240+0000","last_unstale":"2024-02-19T21:13:13.510163+0000","last_undegraded":"2024-02-19T21:13:13.510163+0000","last_fullsized":"2024-02-19T21:13:13.510163+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:43.407517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.407517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"434'12","reported_seq":365,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-02-19T21:13:14.332595+0000","last_change":"2024-02-19T21:10:41.321881+0000","last_active":"2024-02-19T21:13:14.332595+0000","last_peered":"2024-02-19T21:13:14.332595+0000","last_clean":"2024-02-19T21:13:14.332595+0000","last_became_active":"2024-02-19T21:10:41.297897+0000","last_became_peered":"2024-02-19T21:10:41.297897+0000","last_unstale":"2024-02-19T21:13:14.332595+0000","last_undegraded":"2024-02-19T21:13:14.332595+0000","last_fullsized":"2024-02-19T21:13:14.332595+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'11","last_scrub_stamp":"2024-02-19T21:09:39.597001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:39.597001+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"459'28","reported_seq":200,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.503051+0000","last_change":"2024-02-19T21:12:10.323234+0000","last_active":"2024-02-19T21:13:13.503051+0000","last_peered":"2024-02-19T21:13:13.503051+0000","last_clean":"2024-02-19T21:13:13.503051+0000","last_became_active":"2024-02-19T21:12:09.369385+0000","last_became_peered":"2024-02-19T21:12:09.369385+0000","last_unstale":"2024-02-19T21:13:13.503051+0000","last_undegraded":"2024-02-19T21:13:13.503051+0000","last_fullsized":"2024-02-19T21:13:13.503051+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:10:49.466227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:49.466227+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":545,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.531575+0000","last_change":"2024-02-19T21:12:09.343164+0000","last_active":"2024-02-19T21:13:13.531575+0000","last_peered":"2024-02-19T21:13:13.531575+0000","last_clean":"2024-02-19T21:13:13.531575+0000","last_became_active":"2024-02-19T21:12:09.342814+0000","last_became_peered":"2024-02-19T21:12:09.342814+0000","last_unstale":"2024-02-19T21:13:13.531575+0000","last_undegraded":"2024-02-19T21:13:13.531575+0000","last_fullsized":"2024-02-19T21:13:13.531575+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.364732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.364732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":529,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.530948+0000","last_change":"2024-02-19T21:12:09.335134+0000","last_active":"2024-02-19T21:13:13.530948+0000","last_peered":"2024-02-19T21:13:13.530948+0000","last_clean":"2024-02-19T21:13:13.530948+0000","last_became_active":"2024-02-19T21:12:09.333970+0000","last_became_peered":"2024-02-19T21:12:09.333970+0000","last_unstale":"2024-02-19T21:13:13.530948+0000","last_undegraded":"2024-02-19T21:13:13.530948+0000","last_fullsized":"2024-02-19T21:13:13.530948+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.851123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.851123+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":362,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.524179+0000","last_change":"2024-02-19T21:13:11.004073+0000","last_active":"2024-02-19T21:13:13.524179+0000","last_peered":"2024-02-19T21:13:13.524179+0000","last_clean":"2024-02-19T21:13:13.524179+0000","last_became_active":"2024-02-19T21:11:59.323698+0000","last_became_peered":"2024-02-19T21:11:59.323698+0000","last_unstale":"2024-02-19T21:13:13.524179+0000","last_undegraded":"2024-02-19T21:13:13.524179+0000","last_fullsized":"2024-02-19T21:13:13.524179+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:11.004006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:11.004006+0000","last_clean_scrub_stamp":"2024-02-19T21:13:11.004006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":138,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.502787+0000","last_change":"2024-02-19T21:10:59.312931+0000","last_active":"2024-02-19T21:13:13.502787+0000","last_peered":"2024-02-19T21:13:13.502787+0000","last_clean":"2024-02-19T21:13:13.502787+0000","last_became_active":"2024-02-19T21:10:41.304695+0000","last_became_peered":"2024-02-19T21:10:41.304695+0000","last_unstale":"2024-02-19T21:13:13.502787+0000","last_undegraded":"2024-02-19T21:13:13.502787+0000","last_fullsized":"2024-02-19T21:13:13.502787+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:59.312863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.312863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"459'30","reported_seq":332,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.508353+0000","last_change":"2024-02-19T21:11:59.380681+0000","last_active":"2024-02-19T21:13:13.508353+0000","last_peered":"2024-02-19T21:13:13.508353+0000","last_clean":"2024-02-19T21:13:13.508353+0000","last_became_active":"2024-02-19T21:11:59.327859+0000","last_became_peered":"2024-02-19T21:11:59.327859+0000","last_unstale":"2024-02-19T21:13:13.508353+0000","last_undegraded":"2024-02-19T21:13:13.508353+0000","last_fullsized":"2024-02-19T21:13:13.508353+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:11:02.671837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.671837+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":580,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.520851+0000","last_change":"2024-02-19T21:12:09.369780+0000","last_active":"2024-02-19T21:13:13.520851+0000","last_peered":"2024-02-19T21:13:13.520851+0000","last_clean":"2024-02-19T21:13:13.520851+0000","last_became_active":"2024-02-19T21:12:09.369477+0000","last_became_peered":"2024-02-19T21:12:09.369477+0000","last_unstale":"2024-02-19T21:13:13.520851+0000","last_undegraded":"2024-02-19T21:13:13.520851+0000","last_fullsized":"2024-02-19T21:13:13.520851+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":138,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.502933+0000","last_change":"2024-02-19T21:12:06.410909+0000","last_active":"2024-02-19T21:13:13.502933+0000","last_peered":"2024-02-19T21:13:13.502933+0000","last_clean":"2024-02-19T21:13:13.502933+0000","last_became_active":"2024-02-19T21:10:41.295301+0000","last_became_peered":"2024-02-19T21:10:41.295301+0000","last_unstale":"2024-02-19T21:13:13.502933+0000","last_undegraded":"2024-02-19T21:13:13.502933+0000","last_fullsized":"2024-02-19T21:13:13.502933+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.410847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.410847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":287,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.512047+0000","last_change":"2024-02-19T21:12:09.332689+0000","last_active":"2024-02-19T21:13:13.512047+0000","last_peered":"2024-02-19T21:13:13.512047+0000","last_clean":"2024-02-19T21:13:13.512047+0000","last_became_active":"2024-02-19T21:12:09.332349+0000","last_became_peered":"2024-02-19T21:12:09.332349+0000","last_unstale":"2024-02-19T21:13:13.512047+0000","last_undegraded":"2024-02-19T21:13:13.512047+0000","last_fullsized":"2024-02-19T21:13:13.512047+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:46.406402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:46.406402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":522,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325485+0000","last_change":"2024-02-19T21:12:09.340947+0000","last_active":"2024-02-19T21:13:13.325485+0000","last_peered":"2024-02-19T21:13:13.325485+0000","last_clean":"2024-02-19T21:13:13.325485+0000","last_became_active":"2024-02-19T21:12:09.340614+0000","last_became_peered":"2024-02-19T21:12:09.340614+0000","last_unstale":"2024-02-19T21:13:13.325485+0000","last_undegraded":"2024-02-19T21:13:13.325485+0000","last_fullsized":"2024-02-19T21:13:13.325485+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.669336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.669336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"459'29","reported_seq":439,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325689+0000","last_change":"2024-02-19T21:12:00.056470+0000","last_active":"2024-02-19T21:13:13.325689+0000","last_peered":"2024-02-19T21:13:13.325689+0000","last_clean":"2024-02-19T21:13:13.325689+0000","last_became_active":"2024-02-19T21:11:59.332542+0000","last_became_peered":"2024-02-19T21:11:59.332542+0000","last_unstale":"2024-02-19T21:13:13.325689+0000","last_undegraded":"2024-02-19T21:13:13.325689+0000","last_fullsized":"2024-02-19T21:13:13.325689+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:10:11.165973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:11.165973+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":592,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.531145+0000","last_change":"2024-02-19T21:11:59.335901+0000","last_active":"2024-02-19T21:13:13.531145+0000","last_peered":"2024-02-19T21:13:13.531145+0000","last_clean":"2024-02-19T21:13:13.531145+0000","last_became_active":"2024-02-19T21:11:59.335475+0000","last_became_peered":"2024-02-19T21:11:59.335475+0000","last_unstale":"2024-02-19T21:13:13.531145+0000","last_undegraded":"2024-02-19T21:13:13.531145+0000","last_fullsized":"2024-02-19T21:13:13.531145+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:40.580344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:40.580344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":525,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.522952+0000","last_change":"2024-02-19T21:12:20.902387+0000","last_active":"2024-02-19T21:13:13.522952+0000","last_peered":"2024-02-19T21:13:13.522952+0000","last_clean":"2024-02-19T21:13:13.522952+0000","last_became_active":"2024-02-19T21:12:09.347197+0000","last_became_peered":"2024-02-19T21:12:09.347197+0000","last_unstale":"2024-02-19T21:13:13.522952+0000","last_undegraded":"2024-02-19T21:13:13.522952+0000","last_fullsized":"2024-02-19T21:13:13.522952+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:20.902297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:12:20.902297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":482,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.518472+0000","last_change":"2024-02-19T21:11:59.359210+0000","last_active":"2024-02-19T21:13:13.518472+0000","last_peered":"2024-02-19T21:13:13.518472+0000","last_clean":"2024-02-19T21:13:13.518472+0000","last_became_active":"2024-02-19T21:11:59.358933+0000","last_became_peered":"2024-02-19T21:11:59.358933+0000","last_unstale":"2024-02-19T21:13:13.518472+0000","last_undegraded":"2024-02-19T21:13:13.518472+0000","last_fullsized":"2024-02-19T21:13:13.518472+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.189509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":527,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.327663+0000","last_change":"2024-02-19T21:11:59.334472+0000","last_active":"2024-02-19T21:13:13.327663+0000","last_peered":"2024-02-19T21:13:13.327663+0000","last_clean":"2024-02-19T21:13:13.327663+0000","last_became_active":"2024-02-19T21:11:59.333491+0000","last_became_peered":"2024-02-19T21:11:59.333491+0000","last_unstale":"2024-02-19T21:13:13.327663+0000","last_undegraded":"2024-02-19T21:13:13.327663+0000","last_fullsized":"2024-02-19T21:13:13.327663+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:30.893485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:30.893485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"459'24","reported_seq":424,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.520936+0000","last_change":"2024-02-19T21:12:50.030922+0000","last_active":"2024-02-19T21:13:13.520936+0000","last_peered":"2024-02-19T21:13:13.520936+0000","last_clean":"2024-02-19T21:13:13.520936+0000","last_became_active":"2024-02-19T21:11:59.348806+0000","last_became_peered":"2024-02-19T21:11:59.348806+0000","last_unstale":"2024-02-19T21:13:13.520936+0000","last_undegraded":"2024-02-19T21:13:13.520936+0000","last_fullsized":"2024-02-19T21:13:13.520936+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'24","last_scrub_stamp":"2024-02-19T21:12:50.030857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:50.030857+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"397'2","reported_seq":589,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.530120+0000","last_change":"2024-02-19T21:12:02.320023+0000","last_active":"2024-02-19T21:13:13.530120+0000","last_peered":"2024-02-19T21:13:13.530120+0000","last_clean":"2024-02-19T21:13:13.530120+0000","last_became_active":"2024-02-19T21:12:02.319577+0000","last_became_peered":"2024-02-19T21:12:02.319577+0000","last_unstale":"2024-02-19T21:13:13.530120+0000","last_undegraded":"2024-02-19T21:13:13.530120+0000","last_fullsized":"2024-02-19T21:13:13.530120+0000","mapping_epoch":485,"log_start":"58'1","ondisk_log_start":"58'1","created":57,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.934474+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":436,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":147,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.503183+0000","last_change":"2024-02-19T21:11:59.320654+0000","last_active":"2024-02-19T21:13:13.503183+0000","last_peered":"2024-02-19T21:13:13.503183+0000","last_clean":"2024-02-19T21:13:13.503183+0000","last_became_active":"2024-02-19T21:11:59.320327+0000","last_became_peered":"2024-02-19T21:11:59.320327+0000","last_unstale":"2024-02-19T21:13:13.503183+0000","last_undegraded":"2024-02-19T21:13:13.503183+0000","last_fullsized":"2024-02-19T21:13:13.503183+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.455993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.455993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":534,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.522616+0000","last_change":"2024-02-19T21:10:51.691357+0000","last_active":"2024-02-19T21:13:13.522616+0000","last_peered":"2024-02-19T21:13:13.522616+0000","last_clean":"2024-02-19T21:13:13.522616+0000","last_became_active":"2024-02-19T21:08:04.282243+0000","last_became_peered":"2024-02-19T21:08:04.282243+0000","last_unstale":"2024-02-19T21:13:13.522616+0000","last_undegraded":"2024-02-19T21:13:13.522616+0000","last_fullsized":"2024-02-19T21:13:13.522616+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.691284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.691284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":471,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325274+0000","last_change":"2024-02-19T21:12:09.328499+0000","last_active":"2024-02-19T21:13:13.325274+0000","last_peered":"2024-02-19T21:13:13.325274+0000","last_clean":"2024-02-19T21:13:13.325274+0000","last_became_active":"2024-02-19T21:12:09.326936+0000","last_became_peered":"2024-02-19T21:12:09.326936+0000","last_unstale":"2024-02-19T21:13:13.325274+0000","last_undegraded":"2024-02-19T21:13:13.325274+0000","last_fullsized":"2024-02-19T21:13:13.325274+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:05.664425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:05.664425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":523,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.530717+0000","last_change":"2024-02-19T21:13:06.446599+0000","last_active":"2024-02-19T21:13:13.530717+0000","last_peered":"2024-02-19T21:13:13.530717+0000","last_clean":"2024-02-19T21:13:13.530717+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:13:13.530717+0000","last_undegraded":"2024-02-19T21:13:13.530717+0000","last_fullsized":"2024-02-19T21:13:13.530717+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:06.446523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.446523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"459'26","reported_seq":602,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.504250+0000","last_change":"2024-02-19T21:12:09.420794+0000","last_active":"2024-02-19T21:13:13.504250+0000","last_peered":"2024-02-19T21:13:13.504250+0000","last_clean":"2024-02-19T21:13:13.504250+0000","last_became_active":"2024-02-19T21:12:09.352705+0000","last_became_peered":"2024-02-19T21:12:09.352705+0000","last_unstale":"2024-02-19T21:13:13.504250+0000","last_undegraded":"2024-02-19T21:13:13.504250+0000","last_fullsized":"2024-02-19T21:13:13.504250+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'26","last_scrub_stamp":"2024-02-19T21:11:44.774856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:44.774856+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":549,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.530275+0000","last_change":"2024-02-19T21:12:09.311418+0000","last_active":"2024-02-19T21:13:13.530275+0000","last_peered":"2024-02-19T21:13:13.530275+0000","last_clean":"2024-02-19T21:13:13.530275+0000","last_became_active":"2024-02-19T21:12:09.310949+0000","last_became_peered":"2024-02-19T21:12:09.310949+0000","last_unstale":"2024-02-19T21:13:13.530275+0000","last_undegraded":"2024-02-19T21:13:13.530275+0000","last_fullsized":"2024-02-19T21:13:13.530275+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.747108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.747108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":559,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325598+0000","last_change":"2024-02-19T21:11:48.359614+0000","last_active":"2024-02-19T21:13:13.325598+0000","last_peered":"2024-02-19T21:13:13.325598+0000","last_clean":"2024-02-19T21:13:13.325598+0000","last_became_active":"2024-02-19T21:11:48.359271+0000","last_became_peered":"2024-02-19T21:11:48.359271+0000","last_unstale":"2024-02-19T21:13:13.325598+0000","last_undegraded":"2024-02-19T21:13:13.325598+0000","last_fullsized":"2024-02-19T21:13:13.325598+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.081993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.081993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":283,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.510237+0000","last_change":"2024-02-19T21:12:55.248327+0000","last_active":"2024-02-19T21:13:13.510237+0000","last_peered":"2024-02-19T21:13:13.510237+0000","last_clean":"2024-02-19T21:13:13.510237+0000","last_became_active":"2024-02-19T21:12:09.347643+0000","last_became_peered":"2024-02-19T21:12:09.347643+0000","last_unstale":"2024-02-19T21:13:13.510237+0000","last_undegraded":"2024-02-19T21:13:13.510237+0000","last_fullsized":"2024-02-19T21:13:13.510237+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:55.248259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.248259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":552,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.324392+0000","last_change":"2024-02-19T21:10:41.315872+0000","last_active":"2024-02-19T21:13:13.324392+0000","last_peered":"2024-02-19T21:13:13.324392+0000","last_clean":"2024-02-19T21:13:13.324392+0000","last_became_active":"2024-02-19T21:10:41.315397+0000","last_became_peered":"2024-02-19T21:10:41.315397+0000","last_unstale":"2024-02-19T21:13:13.324392+0000","last_undegraded":"2024-02-19T21:13:13.324392+0000","last_fullsized":"2024-02-19T21:13:13.324392+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:10.148733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:10.148733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":480,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325114+0000","last_change":"2024-02-19T21:11:59.333800+0000","last_active":"2024-02-19T21:13:13.325114+0000","last_peered":"2024-02-19T21:13:13.325114+0000","last_clean":"2024-02-19T21:13:13.325114+0000","last_became_active":"2024-02-19T21:11:59.333572+0000","last_became_peered":"2024-02-19T21:11:59.333572+0000","last_unstale":"2024-02-19T21:13:13.325114+0000","last_undegraded":"2024-02-19T21:13:13.325114+0000","last_fullsized":"2024-02-19T21:13:13.325114+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_clean_scrub_stamp":"2024-02-19T21:10:20.204880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":526,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.518436+0000","last_change":"2024-02-19T21:13:00.111238+0000","last_active":"2024-02-19T21:13:13.518436+0000","last_peered":"2024-02-19T21:13:13.518436+0000","last_clean":"2024-02-19T21:13:13.518436+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:13:13.518436+0000","last_undegraded":"2024-02-19T21:13:13.518436+0000","last_fullsized":"2024-02-19T21:13:13.518436+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.111171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.111171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"459'22","reported_seq":566,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.518434+0000","last_change":"2024-02-19T21:11:59.988716+0000","last_active":"2024-02-19T21:13:13.518434+0000","last_peered":"2024-02-19T21:13:13.518434+0000","last_clean":"2024-02-19T21:13:13.518434+0000","last_became_active":"2024-02-19T21:11:59.343010+0000","last_became_peered":"2024-02-19T21:11:59.343010+0000","last_unstale":"2024-02-19T21:13:13.518434+0000","last_undegraded":"2024-02-19T21:13:13.518434+0000","last_fullsized":"2024-02-19T21:13:13.518434+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:11:00.191206+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.191206+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":584,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.520038+0000","last_change":"2024-02-19T21:11:59.352625+0000","last_active":"2024-02-19T21:13:13.520038+0000","last_peered":"2024-02-19T21:13:13.520038+0000","last_clean":"2024-02-19T21:13:13.520038+0000","last_became_active":"2024-02-19T21:11:59.352283+0000","last_became_peered":"2024-02-19T21:11:59.352283+0000","last_unstale":"2024-02-19T21:13:13.520038+0000","last_undegraded":"2024-02-19T21:13:13.520038+0000","last_fullsized":"2024-02-19T21:13:13.520038+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.955872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.955872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":1253,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:14.441499+0000","last_change":"2024-02-19T21:13:08.986433+0000","last_active":"2024-02-19T21:13:14.441499+0000","last_peered":"2024-02-19T21:13:14.441499+0000","last_clean":"2024-02-19T21:13:14.441499+0000","last_became_active":"2024-02-19T21:12:09.340570+0000","last_became_peered":"2024-02-19T21:12:09.340570+0000","last_unstale":"2024-02-19T21:13:14.441499+0000","last_undegraded":"2024-02-19T21:13:14.441499+0000","last_fullsized":"2024-02-19T21:13:14.441499+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:13:08.986361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:08.986361+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":709,"num_read_kb":704,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":548,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.531104+0000","last_change":"2024-02-19T21:13:04.428341+0000","last_active":"2024-02-19T21:13:13.531104+0000","last_peered":"2024-02-19T21:13:13.531104+0000","last_clean":"2024-02-19T21:13:13.531104+0000","last_became_active":"2024-02-19T21:12:09.323625+0000","last_became_peered":"2024-02-19T21:12:09.323625+0000","last_unstale":"2024-02-19T21:13:13.531104+0000","last_undegraded":"2024-02-19T21:13:13.531104+0000","last_fullsized":"2024-02-19T21:13:13.531104+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:04.428236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:04.428236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":479,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.523248+0000","last_change":"2024-02-19T21:11:59.352488+0000","last_active":"2024-02-19T21:13:13.523248+0000","last_peered":"2024-02-19T21:13:13.523248+0000","last_clean":"2024-02-19T21:13:13.523248+0000","last_became_active":"2024-02-19T21:11:59.351887+0000","last_became_peered":"2024-02-19T21:11:59.351887+0000","last_unstale":"2024-02-19T21:13:13.523248+0000","last_undegraded":"2024-02-19T21:13:13.523248+0000","last_fullsized":"2024-02-19T21:13:13.523248+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.947089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":389,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.523949+0000","last_change":"2024-02-19T21:12:09.369780+0000","last_active":"2024-02-19T21:13:13.523949+0000","last_peered":"2024-02-19T21:13:13.523949+0000","last_clean":"2024-02-19T21:13:13.523949+0000","last_became_active":"2024-02-19T21:12:09.369431+0000","last_became_peered":"2024-02-19T21:12:09.369431+0000","last_unstale":"2024-02-19T21:13:13.523949+0000","last_undegraded":"2024-02-19T21:13:13.523949+0000","last_fullsized":"2024-02-19T21:13:13.523949+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.945639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.945639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"459'42","reported_seq":792,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-02-19T21:13:14.714948+0000","last_change":"2024-02-19T21:10:01.545972+0000","last_active":"2024-02-19T21:13:14.714948+0000","last_peered":"2024-02-19T21:13:14.714948+0000","last_clean":"2024-02-19T21:13:14.714948+0000","last_became_active":"2024-02-19T21:08:04.266322+0000","last_became_peered":"2024-02-19T21:08:04.266322+0000","last_unstale":"2024-02-19T21:13:14.714948+0000","last_undegraded":"2024-02-19T21:13:14.714948+0000","last_fullsized":"2024-02-19T21:13:14.714948+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'30","last_scrub_stamp":"2024-02-19T21:10:01.545897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:01.545897+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":568,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325446+0000","last_change":"2024-02-19T21:12:09.328219+0000","last_active":"2024-02-19T21:13:13.325446+0000","last_peered":"2024-02-19T21:13:13.325446+0000","last_clean":"2024-02-19T21:13:13.325446+0000","last_became_active":"2024-02-19T21:12:09.326579+0000","last_became_peered":"2024-02-19T21:12:09.326579+0000","last_unstale":"2024-02-19T21:13:13.325446+0000","last_undegraded":"2024-02-19T21:13:13.325446+0000","last_fullsized":"2024-02-19T21:13:13.325446+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.089715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.089715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":287,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.510092+0000","last_change":"2024-02-19T21:11:31.629134+0000","last_active":"2024-02-19T21:13:13.510092+0000","last_peered":"2024-02-19T21:13:13.510092+0000","last_clean":"2024-02-19T21:13:13.510092+0000","last_became_active":"2024-02-19T21:10:41.305829+0000","last_became_peered":"2024-02-19T21:10:41.305829+0000","last_unstale":"2024-02-19T21:13:13.510092+0000","last_undegraded":"2024-02-19T21:13:13.510092+0000","last_fullsized":"2024-02-19T21:13:13.510092+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:31.629057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.629057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":482,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.517361+0000","last_change":"2024-02-19T21:11:59.333878+0000","last_active":"2024-02-19T21:13:13.517361+0000","last_peered":"2024-02-19T21:13:13.517361+0000","last_clean":"2024-02-19T21:13:13.517361+0000","last_became_active":"2024-02-19T21:11:59.331967+0000","last_became_peered":"2024-02-19T21:11:59.331967+0000","last_unstale":"2024-02-19T21:13:13.517361+0000","last_undegraded":"2024-02-19T21:13:13.517361+0000","last_fullsized":"2024-02-19T21:13:13.517361+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:54.142898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.142898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"493'12","reported_seq":437,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.522585+0000","last_change":"2024-02-19T21:12:09.433235+0000","last_active":"2024-02-19T21:13:13.522585+0000","last_peered":"2024-02-19T21:13:13.522585+0000","last_clean":"2024-02-19T21:13:13.522585+0000","last_became_active":"2024-02-19T21:12:09.326372+0000","last_became_peered":"2024-02-19T21:12:09.326372+0000","last_unstale":"2024-02-19T21:13:13.522585+0000","last_undegraded":"2024-02-19T21:13:13.522585+0000","last_fullsized":"2024-02-19T21:13:13.522585+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'10","last_scrub_stamp":"2024-02-19T21:11:41.775418+0000","last_deep_scrub":"471'10","last_deep_scrub_stamp":"2024-02-19T21:11:41.775418+0000","last_clean_scrub_stamp":"2024-02-19T21:11:41.775418+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"459'20","reported_seq":592,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.522076+0000","last_change":"2024-02-19T21:12:09.030294+0000","last_active":"2024-02-19T21:13:13.522076+0000","last_peered":"2024-02-19T21:13:13.522076+0000","last_clean":"2024-02-19T21:13:13.522076+0000","last_became_active":"2024-02-19T21:10:41.296275+0000","last_became_peered":"2024-02-19T21:10:41.296275+0000","last_unstale":"2024-02-19T21:13:13.522076+0000","last_undegraded":"2024-02-19T21:13:13.522076+0000","last_fullsized":"2024-02-19T21:13:13.522076+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:12:09.030215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:09.030215+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":585,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.522277+0000","last_change":"2024-02-19T21:11:59.351637+0000","last_active":"2024-02-19T21:13:13.522277+0000","last_peered":"2024-02-19T21:13:13.522277+0000","last_clean":"2024-02-19T21:13:13.522277+0000","last_became_active":"2024-02-19T21:11:59.351241+0000","last_became_peered":"2024-02-19T21:11:59.351241+0000","last_unstale":"2024-02-19T21:13:13.522277+0000","last_undegraded":"2024-02-19T21:13:13.522277+0000","last_fullsized":"2024-02-19T21:13:13.522277+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:21.732778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:10:21.732778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":283,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.508317+0000","last_change":"2024-02-19T21:13:07.221665+0000","last_active":"2024-02-19T21:13:13.508317+0000","last_peered":"2024-02-19T21:13:13.508317+0000","last_clean":"2024-02-19T21:13:13.508317+0000","last_became_active":"2024-02-19T21:11:59.335798+0000","last_became_peered":"2024-02-19T21:11:59.335798+0000","last_unstale":"2024-02-19T21:13:13.508317+0000","last_undegraded":"2024-02-19T21:13:13.508317+0000","last_fullsized":"2024-02-19T21:13:13.508317+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:07.221594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.221594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":278,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.510168+0000","last_change":"2024-02-19T21:12:09.332210+0000","last_active":"2024-02-19T21:13:13.510168+0000","last_peered":"2024-02-19T21:13:13.510168+0000","last_clean":"2024-02-19T21:13:13.510168+0000","last_became_active":"2024-02-19T21:12:09.331195+0000","last_became_peered":"2024-02-19T21:12:09.331195+0000","last_unstale":"2024-02-19T21:13:13.510168+0000","last_undegraded":"2024-02-19T21:13:13.510168+0000","last_fullsized":"2024-02-19T21:13:13.510168+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.604025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.604025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":535,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.517248+0000","last_change":"2024-02-19T21:13:09.184029+0000","last_active":"2024-02-19T21:13:13.517248+0000","last_peered":"2024-02-19T21:13:13.517248+0000","last_clean":"2024-02-19T21:13:13.517248+0000","last_became_active":"2024-02-19T21:11:59.333275+0000","last_became_peered":"2024-02-19T21:11:59.333275+0000","last_unstale":"2024-02-19T21:13:13.517248+0000","last_undegraded":"2024-02-19T21:13:13.517248+0000","last_fullsized":"2024-02-19T21:13:13.517248+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.183962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.183962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"459'23","reported_seq":159,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.502878+0000","last_change":"2024-02-19T21:12:18.340005+0000","last_active":"2024-02-19T21:13:13.502878+0000","last_peered":"2024-02-19T21:13:13.502878+0000","last_clean":"2024-02-19T21:13:13.502878+0000","last_became_active":"2024-02-19T21:10:41.287134+0000","last_became_peered":"2024-02-19T21:10:41.287134+0000","last_unstale":"2024-02-19T21:13:13.502878+0000","last_undegraded":"2024-02-19T21:13:13.502878+0000","last_fullsized":"2024-02-19T21:13:13.502878+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'23","last_scrub_stamp":"2024-02-19T21:12:18.339929+0000","last_deep_scrub":"420'17","last_deep_scrub_stamp":"2024-02-19T21:10:19.722303+0000","last_clean_scrub_stamp":"2024-02-19T21:12:18.339929+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":159,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.504263+0000","last_change":"2024-02-19T21:11:59.342221+0000","last_active":"2024-02-19T21:13:13.504263+0000","last_peered":"2024-02-19T21:13:13.504263+0000","last_clean":"2024-02-19T21:13:13.504263+0000","last_became_active":"2024-02-19T21:11:59.323514+0000","last_became_peered":"2024-02-19T21:11:59.323514+0000","last_unstale":"2024-02-19T21:13:13.504263+0000","last_undegraded":"2024-02-19T21:13:13.504263+0000","last_fullsized":"2024-02-19T21:13:13.504263+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:10:58.333168+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.333168+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":274,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.510907+0000","last_change":"2024-02-19T21:13:00.166548+0000","last_active":"2024-02-19T21:13:13.510907+0000","last_peered":"2024-02-19T21:13:13.510907+0000","last_clean":"2024-02-19T21:13:13.510907+0000","last_became_active":"2024-02-19T21:08:04.264593+0000","last_became_peered":"2024-02-19T21:08:04.264593+0000","last_unstale":"2024-02-19T21:13:13.510907+0000","last_undegraded":"2024-02-19T21:13:13.510907+0000","last_fullsized":"2024-02-19T21:13:13.510907+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.166479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.166479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":473,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.517999+0000","last_change":"2024-02-19T21:11:03.195680+0000","last_active":"2024-02-19T21:13:13.517999+0000","last_peered":"2024-02-19T21:13:13.517999+0000","last_clean":"2024-02-19T21:13:13.517999+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:13:13.517999+0000","last_undegraded":"2024-02-19T21:13:13.517999+0000","last_fullsized":"2024-02-19T21:13:13.517999+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.195616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.195616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"459'15","reported_seq":309,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.510010+0000","last_change":"2024-02-19T21:11:59.364955+0000","last_active":"2024-02-19T21:13:13.510010+0000","last_peered":"2024-02-19T21:13:13.510010+0000","last_clean":"2024-02-19T21:13:13.510010+0000","last_became_active":"2024-02-19T21:11:59.336293+0000","last_became_peered":"2024-02-19T21:11:59.336293+0000","last_unstale":"2024-02-19T21:13:13.510010+0000","last_undegraded":"2024-02-19T21:13:13.510010+0000","last_fullsized":"2024-02-19T21:13:13.510010+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:10:22.459807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:22.459807+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":534,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.517211+0000","last_change":"2024-02-19T21:11:59.334071+0000","last_active":"2024-02-19T21:13:13.517211+0000","last_peered":"2024-02-19T21:13:13.517211+0000","last_clean":"2024-02-19T21:13:13.517211+0000","last_became_active":"2024-02-19T21:11:59.333684+0000","last_became_peered":"2024-02-19T21:11:59.333684+0000","last_unstale":"2024-02-19T21:13:13.517211+0000","last_undegraded":"2024-02-19T21:13:13.517211+0000","last_fullsized":"2024-02-19T21:13:13.517211+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.244285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.244285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":269,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.509281+0000","last_change":"2024-02-19T21:10:53.703797+0000","last_active":"2024-02-19T21:13:13.509281+0000","last_peered":"2024-02-19T21:13:13.509281+0000","last_clean":"2024-02-19T21:13:13.509281+0000","last_became_active":"2024-02-19T21:08:04.264972+0000","last_became_peered":"2024-02-19T21:08:04.264972+0000","last_unstale":"2024-02-19T21:13:13.509281+0000","last_undegraded":"2024-02-19T21:13:13.509281+0000","last_fullsized":"2024-02-19T21:13:13.509281+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:53.703727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.703727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":542,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.519698+0000","last_change":"2024-02-19T21:12:06.261731+0000","last_active":"2024-02-19T21:13:13.519698+0000","last_peered":"2024-02-19T21:13:13.519698+0000","last_clean":"2024-02-19T21:13:13.519698+0000","last_became_active":"2024-02-19T21:10:41.314114+0000","last_became_peered":"2024-02-19T21:10:41.314114+0000","last_unstale":"2024-02-19T21:13:13.519698+0000","last_undegraded":"2024-02-19T21:13:13.519698+0000","last_fullsized":"2024-02-19T21:13:13.519698+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.261670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.261670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":138,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.503003+0000","last_change":"2024-02-19T21:12:08.430936+0000","last_active":"2024-02-19T21:13:13.503003+0000","last_peered":"2024-02-19T21:13:13.503003+0000","last_clean":"2024-02-19T21:13:13.503003+0000","last_became_active":"2024-02-19T21:10:41.300574+0000","last_became_peered":"2024-02-19T21:10:41.300574+0000","last_unstale":"2024-02-19T21:13:13.503003+0000","last_undegraded":"2024-02-19T21:13:13.503003+0000","last_fullsized":"2024-02-19T21:13:13.503003+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:08.430874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:08.430874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"459'15","reported_seq":156,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.517718+0000","last_change":"2024-02-19T21:10:53.101489+0000","last_active":"2024-02-19T21:13:13.517718+0000","last_peered":"2024-02-19T21:13:13.517718+0000","last_clean":"2024-02-19T21:13:13.517718+0000","last_became_active":"2024-02-19T21:10:40.297867+0000","last_became_peered":"2024-02-19T21:10:40.297867+0000","last_unstale":"2024-02-19T21:13:13.517718+0000","last_undegraded":"2024-02-19T21:13:13.517718+0000","last_fullsized":"2024-02-19T21:13:13.517718+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:10:53.101422+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.101422+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":484,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.502633+0000","last_change":"2024-02-19T21:12:09.369726+0000","last_active":"2024-02-19T21:13:13.502633+0000","last_peered":"2024-02-19T21:13:13.502633+0000","last_clean":"2024-02-19T21:13:13.502633+0000","last_became_active":"2024-02-19T21:12:09.369407+0000","last_became_peered":"2024-02-19T21:12:09.369407+0000","last_unstale":"2024-02-19T21:13:13.502633+0000","last_undegraded":"2024-02-19T21:13:13.502633+0000","last_fullsized":"2024-02-19T21:13:13.502633+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:31.670026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.670026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":539,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325953+0000","last_change":"2024-02-19T21:12:06.098778+0000","last_active":"2024-02-19T21:13:13.325953+0000","last_peered":"2024-02-19T21:13:13.325953+0000","last_clean":"2024-02-19T21:13:13.325953+0000","last_became_active":"2024-02-19T21:08:04.265573+0000","last_became_peered":"2024-02-19T21:08:04.265573+0000","last_unstale":"2024-02-19T21:13:13.325953+0000","last_undegraded":"2024-02-19T21:13:13.325953+0000","last_fullsized":"2024-02-19T21:13:13.325953+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.098703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.098703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":550,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.530952+0000","last_change":"2024-02-19T21:12:17.452030+0000","last_active":"2024-02-19T21:13:13.530952+0000","last_peered":"2024-02-19T21:13:13.530952+0000","last_clean":"2024-02-19T21:13:13.530952+0000","last_became_active":"2024-02-19T21:12:09.341406+0000","last_became_peered":"2024-02-19T21:12:09.341406+0000","last_unstale":"2024-02-19T21:13:13.530952+0000","last_undegraded":"2024-02-19T21:13:13.530952+0000","last_fullsized":"2024-02-19T21:13:13.530952+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:17.451964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:17.451964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":498,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325456+0000","last_change":"2024-02-19T21:12:47.428132+0000","last_active":"2024-02-19T21:13:13.325456+0000","last_peered":"2024-02-19T21:13:13.325456+0000","last_clean":"2024-02-19T21:13:13.325456+0000","last_became_active":"2024-02-19T21:12:09.349695+0000","last_became_peered":"2024-02-19T21:12:09.349695+0000","last_unstale":"2024-02-19T21:13:13.325456+0000","last_undegraded":"2024-02-19T21:13:13.325456+0000","last_fullsized":"2024-02-19T21:13:13.325456+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:47.428055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:47.428055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"459'20","reported_seq":296,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.511898+0000","last_change":"2024-02-19T21:10:59.685223+0000","last_active":"2024-02-19T21:13:13.511898+0000","last_peered":"2024-02-19T21:13:13.511898+0000","last_clean":"2024-02-19T21:13:13.511898+0000","last_became_active":"2024-02-19T21:08:04.267216+0000","last_became_peered":"2024-02-19T21:08:04.267216+0000","last_unstale":"2024-02-19T21:13:13.511898+0000","last_undegraded":"2024-02-19T21:13:13.511898+0000","last_fullsized":"2024-02-19T21:13:13.511898+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:10:59.685150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.685150+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":541,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325386+0000","last_change":"2024-02-19T21:09:40.986717+0000","last_active":"2024-02-19T21:13:13.325386+0000","last_peered":"2024-02-19T21:13:13.325386+0000","last_clean":"2024-02-19T21:13:13.325386+0000","last_became_active":"2024-02-19T21:08:04.266728+0000","last_became_peered":"2024-02-19T21:08:04.266728+0000","last_unstale":"2024-02-19T21:13:13.325386+0000","last_undegraded":"2024-02-19T21:13:13.325386+0000","last_fullsized":"2024-02-19T21:13:13.325386+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.986635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"433'9","reported_seq":684,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.523105+0000","last_change":"2024-02-19T21:13:00.428710+0000","last_active":"2024-02-19T21:13:13.523105+0000","last_peered":"2024-02-19T21:13:13.523105+0000","last_clean":"2024-02-19T21:13:13.523105+0000","last_became_active":"2024-02-19T21:10:40.310303+0000","last_became_peered":"2024-02-19T21:10:40.310303+0000","last_unstale":"2024-02-19T21:13:13.523105+0000","last_undegraded":"2024-02-19T21:13:13.523105+0000","last_fullsized":"2024-02-19T21:13:13.523105+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'9","last_scrub_stamp":"2024-02-19T21:13:00.428581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.428581+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":402,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.531641+0000","last_change":"2024-02-19T21:11:59.843545+0000","last_active":"2024-02-19T21:13:13.531641+0000","last_peered":"2024-02-19T21:13:13.531641+0000","last_clean":"2024-02-19T21:13:13.531641+0000","last_became_active":"2024-02-19T21:11:59.323539+0000","last_became_peered":"2024-02-19T21:11:59.323539+0000","last_unstale":"2024-02-19T21:13:13.531641+0000","last_undegraded":"2024-02-19T21:13:13.531641+0000","last_fullsized":"2024-02-19T21:13:13.531641+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:11:31.500511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.500511+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":354,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.517130+0000","last_change":"2024-02-19T21:12:09.345187+0000","last_active":"2024-02-19T21:13:13.517130+0000","last_peered":"2024-02-19T21:13:13.517130+0000","last_clean":"2024-02-19T21:13:13.517130+0000","last_became_active":"2024-02-19T21:12:09.344609+0000","last_became_peered":"2024-02-19T21:12:09.344609+0000","last_unstale":"2024-02-19T21:13:13.517130+0000","last_undegraded":"2024-02-19T21:13:13.517130+0000","last_fullsized":"2024-02-19T21:13:13.517130+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.378925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:55.202463+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.378925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"459'15","reported_seq":584,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.326430+0000","last_change":"2024-02-19T21:10:50.053312+0000","last_active":"2024-02-19T21:13:13.326430+0000","last_peered":"2024-02-19T21:13:13.326430+0000","last_clean":"2024-02-19T21:13:13.326430+0000","last_became_active":"2024-02-19T21:08:04.266103+0000","last_became_peered":"2024-02-19T21:08:04.266103+0000","last_unstale":"2024-02-19T21:13:13.326430+0000","last_undegraded":"2024-02-19T21:13:13.326430+0000","last_fullsized":"2024-02-19T21:13:13.326430+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:10:50.053229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.053229+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":152,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.502857+0000","last_change":"2024-02-19T21:13:07.507018+0000","last_active":"2024-02-19T21:13:13.502857+0000","last_peered":"2024-02-19T21:13:13.502857+0000","last_clean":"2024-02-19T21:13:13.502857+0000","last_became_active":"2024-02-19T21:11:59.324861+0000","last_became_peered":"2024-02-19T21:11:59.324861+0000","last_unstale":"2024-02-19T21:13:13.502857+0000","last_undegraded":"2024-02-19T21:13:13.502857+0000","last_fullsized":"2024-02-19T21:13:13.502857+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:07.506946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.506946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":341,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-02-19T21:13:14.336831+0000","last_change":"2024-02-19T21:12:17.345023+0000","last_active":"2024-02-19T21:13:14.336831+0000","last_peered":"2024-02-19T21:13:14.336831+0000","last_clean":"2024-02-19T21:13:14.336831+0000","last_became_active":"2024-02-19T21:08:04.256045+0000","last_became_peered":"2024-02-19T21:08:04.256045+0000","last_unstale":"2024-02-19T21:13:14.336831+0000","last_undegraded":"2024-02-19T21:13:14.336831+0000","last_fullsized":"2024-02-19T21:13:14.336831+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'10","last_scrub_stamp":"2024-02-19T21:12:17.344916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:17.344916+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":138,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.504307+0000","last_change":"2024-02-19T21:12:05.445686+0000","last_active":"2024-02-19T21:13:13.504307+0000","last_peered":"2024-02-19T21:13:13.504307+0000","last_clean":"2024-02-19T21:13:13.504307+0000","last_became_active":"2024-02-19T21:10:41.310279+0000","last_became_peered":"2024-02-19T21:10:41.310279+0000","last_unstale":"2024-02-19T21:13:13.504307+0000","last_undegraded":"2024-02-19T21:13:13.504307+0000","last_fullsized":"2024-02-19T21:13:13.504307+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:05.445575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_clean_scrub_stamp":"2024-02-19T21:12:05.445575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":530,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.519802+0000","last_change":"2024-02-19T21:12:49.142231+0000","last_active":"2024-02-19T21:13:13.519802+0000","last_peered":"2024-02-19T21:13:13.519802+0000","last_clean":"2024-02-19T21:13:13.519802+0000","last_became_active":"2024-02-19T21:12:09.339844+0000","last_became_peered":"2024-02-19T21:12:09.339844+0000","last_unstale":"2024-02-19T21:13:13.519802+0000","last_undegraded":"2024-02-19T21:13:13.519802+0000","last_fullsized":"2024-02-19T21:13:13.519802+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-02-19T21:12:49.142158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:49.142158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":15,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"459'15","reported_seq":285,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.521587+0000","last_change":"2024-02-19T21:13:06.958616+0000","last_active":"2024-02-19T21:13:13.521587+0000","last_peered":"2024-02-19T21:13:13.521587+0000","last_clean":"2024-02-19T21:13:13.521587+0000","last_became_active":"2024-02-19T21:12:09.341038+0000","last_became_peered":"2024-02-19T21:12:09.341038+0000","last_unstale":"2024-02-19T21:13:13.521587+0000","last_undegraded":"2024-02-19T21:13:13.521587+0000","last_fullsized":"2024-02-19T21:13:13.521587+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:13:06.958518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.958518+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":353,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.511442+0000","last_change":"2024-02-19T21:12:09.333684+0000","last_active":"2024-02-19T21:13:13.511442+0000","last_peered":"2024-02-19T21:13:13.511442+0000","last_clean":"2024-02-19T21:13:13.511442+0000","last_became_active":"2024-02-19T21:12:09.333350+0000","last_became_peered":"2024-02-19T21:12:09.333350+0000","last_unstale":"2024-02-19T21:13:13.511442+0000","last_undegraded":"2024-02-19T21:13:13.511442+0000","last_fullsized":"2024-02-19T21:13:13.511442+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.422323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.422323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":283,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.510990+0000","last_change":"2024-02-19T21:11:59.349259+0000","last_active":"2024-02-19T21:13:13.510990+0000","last_peered":"2024-02-19T21:13:13.510990+0000","last_clean":"2024-02-19T21:13:13.510990+0000","last_became_active":"2024-02-19T21:11:59.348485+0000","last_became_peered":"2024-02-19T21:11:59.348485+0000","last_unstale":"2024-02-19T21:13:13.510990+0000","last_undegraded":"2024-02-19T21:13:13.510990+0000","last_fullsized":"2024-02-19T21:13:13.510990+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:45.404242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_clean_scrub_stamp":"2024-02-19T21:11:45.404242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":367,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325079+0000","last_change":"2024-02-19T21:12:09.327887+0000","last_active":"2024-02-19T21:13:13.325079+0000","last_peered":"2024-02-19T21:13:13.325079+0000","last_clean":"2024-02-19T21:13:13.325079+0000","last_became_active":"2024-02-19T21:12:09.327390+0000","last_became_peered":"2024-02-19T21:12:09.327390+0000","last_unstale":"2024-02-19T21:13:13.325079+0000","last_undegraded":"2024-02-19T21:13:13.325079+0000","last_fullsized":"2024-02-19T21:13:13.325079+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:38.940935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:38.940935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":471,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325916+0000","last_change":"2024-02-19T21:12:09.330102+0000","last_active":"2024-02-19T21:13:13.325916+0000","last_peered":"2024-02-19T21:13:13.325916+0000","last_clean":"2024-02-19T21:13:13.325916+0000","last_became_active":"2024-02-19T21:12:09.329246+0000","last_became_peered":"2024-02-19T21:12:09.329246+0000","last_unstale":"2024-02-19T21:13:13.325916+0000","last_undegraded":"2024-02-19T21:13:13.325916+0000","last_fullsized":"2024-02-19T21:13:13.325916+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.493206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.493206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"459'19","reported_seq":183,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.503103+0000","last_change":"2024-02-19T21:12:10.570353+0000","last_active":"2024-02-19T21:13:13.503103+0000","last_peered":"2024-02-19T21:13:13.503103+0000","last_clean":"2024-02-19T21:13:13.503103+0000","last_became_active":"2024-02-19T21:12:09.399929+0000","last_became_peered":"2024-02-19T21:12:09.399929+0000","last_unstale":"2024-02-19T21:13:13.503103+0000","last_undegraded":"2024-02-19T21:13:13.503103+0000","last_fullsized":"2024-02-19T21:13:13.503103+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:11:32.351880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:32.351880+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":545,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.524023+0000","last_change":"2024-02-19T21:13:00.874389+0000","last_active":"2024-02-19T21:13:13.524023+0000","last_peered":"2024-02-19T21:13:13.524023+0000","last_clean":"2024-02-19T21:13:13.524023+0000","last_became_active":"2024-02-19T21:12:09.342120+0000","last_became_peered":"2024-02-19T21:12:09.342120+0000","last_unstale":"2024-02-19T21:13:13.524023+0000","last_undegraded":"2024-02-19T21:13:13.524023+0000","last_fullsized":"2024-02-19T21:13:13.524023+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.874318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.874318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":541,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.522146+0000","last_change":"2024-02-19T21:12:54.957657+0000","last_active":"2024-02-19T21:13:13.522146+0000","last_peered":"2024-02-19T21:13:13.522146+0000","last_clean":"2024-02-19T21:13:13.522146+0000","last_became_active":"2024-02-19T21:08:04.268847+0000","last_became_peered":"2024-02-19T21:08:04.268847+0000","last_unstale":"2024-02-19T21:13:13.522146+0000","last_undegraded":"2024-02-19T21:13:13.522146+0000","last_fullsized":"2024-02-19T21:13:13.522146+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:54.957585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:54.957585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":278,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.508785+0000","last_change":"2024-02-19T21:11:59.341209+0000","last_active":"2024-02-19T21:13:13.508785+0000","last_peered":"2024-02-19T21:13:13.508785+0000","last_clean":"2024-02-19T21:13:13.508785+0000","last_became_active":"2024-02-19T21:11:59.340742+0000","last_became_peered":"2024-02-19T21:11:59.340742+0000","last_unstale":"2024-02-19T21:13:13.508785+0000","last_undegraded":"2024-02-19T21:13:13.508785+0000","last_fullsized":"2024-02-19T21:13:13.508785+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:37.477995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:37.477995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":278,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.509324+0000","last_change":"2024-02-19T21:11:59.338599+0000","last_active":"2024-02-19T21:13:13.509324+0000","last_peered":"2024-02-19T21:13:13.509324+0000","last_clean":"2024-02-19T21:13:13.509324+0000","last_became_active":"2024-02-19T21:11:59.337764+0000","last_became_peered":"2024-02-19T21:11:59.337764+0000","last_unstale":"2024-02-19T21:13:13.509324+0000","last_undegraded":"2024-02-19T21:13:13.509324+0000","last_fullsized":"2024-02-19T21:13:13.509324+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.599474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.599474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"555'10","reported_seq":165,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.324847+0000","last_change":"2024-02-19T21:13:12.409114+0000","last_active":"2024-02-19T21:13:13.324847+0000","last_peered":"2024-02-19T21:13:13.324847+0000","last_clean":"2024-02-19T21:13:13.324847+0000","last_became_active":"2024-02-19T21:12:09.340298+0000","last_became_peered":"2024-02-19T21:12:09.340298+0000","last_unstale":"2024-02-19T21:13:13.324847+0000","last_undegraded":"2024-02-19T21:13:13.324847+0000","last_fullsized":"2024-02-19T21:13:13.324847+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'10","last_scrub_stamp":"2024-02-19T21:13:12.409036+0000","last_deep_scrub":"555'10","last_deep_scrub_stamp":"2024-02-19T21:13:12.409036+0000","last_clean_scrub_stamp":"2024-02-19T21:13:12.409036+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":532,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.326016+0000","last_change":"2024-02-19T21:12:09.330300+0000","last_active":"2024-02-19T21:13:13.326016+0000","last_peered":"2024-02-19T21:13:13.326016+0000","last_clean":"2024-02-19T21:13:13.326016+0000","last_became_active":"2024-02-19T21:12:09.329752+0000","last_became_peered":"2024-02-19T21:12:09.329752+0000","last_unstale":"2024-02-19T21:13:13.326016+0000","last_undegraded":"2024-02-19T21:13:13.326016+0000","last_fullsized":"2024-02-19T21:13:13.326016+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:42.000850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:42.000850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":480,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.502465+0000","last_change":"2024-02-19T21:12:09.346328+0000","last_active":"2024-02-19T21:13:13.502465+0000","last_peered":"2024-02-19T21:13:13.502465+0000","last_clean":"2024-02-19T21:13:13.502465+0000","last_became_active":"2024-02-19T21:12:09.345654+0000","last_became_peered":"2024-02-19T21:12:09.345654+0000","last_unstale":"2024-02-19T21:13:13.502465+0000","last_undegraded":"2024-02-19T21:13:13.502465+0000","last_fullsized":"2024-02-19T21:13:13.502465+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.467476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.467476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":539,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.530125+0000","last_change":"2024-02-19T21:12:05.530113+0000","last_active":"2024-02-19T21:13:13.530125+0000","last_peered":"2024-02-19T21:13:13.530125+0000","last_clean":"2024-02-19T21:13:13.530125+0000","last_became_active":"2024-02-19T21:10:41.314589+0000","last_became_peered":"2024-02-19T21:10:41.314589+0000","last_unstale":"2024-02-19T21:13:13.530125+0000","last_undegraded":"2024-02-19T21:13:13.530125+0000","last_fullsized":"2024-02-19T21:13:13.530125+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:05.530039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:05.530039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":274,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.511971+0000","last_change":"2024-02-19T21:12:07.259353+0000","last_active":"2024-02-19T21:13:13.511971+0000","last_peered":"2024-02-19T21:13:13.511971+0000","last_clean":"2024-02-19T21:13:13.511971+0000","last_became_active":"2024-02-19T21:08:04.266161+0000","last_became_peered":"2024-02-19T21:08:04.266161+0000","last_unstale":"2024-02-19T21:13:13.511971+0000","last_undegraded":"2024-02-19T21:13:13.511971+0000","last_fullsized":"2024-02-19T21:13:13.511971+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:07.259290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:07.259290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"459'19","reported_seq":319,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.509033+0000","last_change":"2024-02-19T21:12:09.375334+0000","last_active":"2024-02-19T21:13:13.509033+0000","last_peered":"2024-02-19T21:13:13.509033+0000","last_clean":"2024-02-19T21:13:13.509033+0000","last_became_active":"2024-02-19T21:12:09.321408+0000","last_became_peered":"2024-02-19T21:12:09.321408+0000","last_unstale":"2024-02-19T21:13:13.509033+0000","last_undegraded":"2024-02-19T21:13:13.509033+0000","last_fullsized":"2024-02-19T21:13:13.509033+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:12:03.255395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:03.255395+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":73,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.327612+0000","last_change":"2024-02-19T21:12:09.345352+0000","last_active":"2024-02-19T21:13:13.327612+0000","last_peered":"2024-02-19T21:13:13.327612+0000","last_clean":"2024-02-19T21:13:13.327612+0000","last_became_active":"2024-02-19T21:12:09.345000+0000","last_became_peered":"2024-02-19T21:12:09.345000+0000","last_unstale":"2024-02-19T21:13:13.327612+0000","last_undegraded":"2024-02-19T21:13:13.327612+0000","last_fullsized":"2024-02-19T21:13:13.327612+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:00.382946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.382946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":545,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.530555+0000","last_change":"2024-02-19T21:12:09.332889+0000","last_active":"2024-02-19T21:13:13.530555+0000","last_peered":"2024-02-19T21:13:13.530555+0000","last_clean":"2024-02-19T21:13:13.530555+0000","last_became_active":"2024-02-19T21:12:09.332485+0000","last_became_peered":"2024-02-19T21:12:09.332485+0000","last_unstale":"2024-02-19T21:13:13.530555+0000","last_undegraded":"2024-02-19T21:13:13.530555+0000","last_fullsized":"2024-02-19T21:13:13.530555+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.566208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.566208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":287,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.511023+0000","last_change":"2024-02-19T21:12:09.344855+0000","last_active":"2024-02-19T21:13:13.511023+0000","last_peered":"2024-02-19T21:13:13.511023+0000","last_clean":"2024-02-19T21:13:13.511023+0000","last_became_active":"2024-02-19T21:12:09.344536+0000","last_became_peered":"2024-02-19T21:12:09.344536+0000","last_unstale":"2024-02-19T21:13:13.511023+0000","last_undegraded":"2024-02-19T21:13:13.511023+0000","last_fullsized":"2024-02-19T21:13:13.511023+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:00.547228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:00.547228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":359,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.326611+0000","last_change":"2024-02-19T21:12:09.343909+0000","last_active":"2024-02-19T21:13:13.326611+0000","last_peered":"2024-02-19T21:13:13.326611+0000","last_clean":"2024-02-19T21:13:13.326611+0000","last_became_active":"2024-02-19T21:12:09.342686+0000","last_became_peered":"2024-02-19T21:12:09.342686+0000","last_unstale":"2024-02-19T21:13:13.326611+0000","last_undegraded":"2024-02-19T21:13:13.326611+0000","last_fullsized":"2024-02-19T21:13:13.326611+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:37.718654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:11:37.718654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"459'20","reported_seq":459,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.531721+0000","last_change":"2024-02-19T21:12:48.422862+0000","last_active":"2024-02-19T21:13:13.531721+0000","last_peered":"2024-02-19T21:13:13.531721+0000","last_clean":"2024-02-19T21:13:13.531721+0000","last_became_active":"2024-02-19T21:11:59.350142+0000","last_became_peered":"2024-02-19T21:11:59.350142+0000","last_unstale":"2024-02-19T21:13:13.531721+0000","last_undegraded":"2024-02-19T21:13:13.531721+0000","last_fullsized":"2024-02-19T21:13:13.531721+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:12:48.422763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:48.422763+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":541,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.324229+0000","last_change":"2024-02-19T21:12:48.430341+0000","last_active":"2024-02-19T21:13:13.324229+0000","last_peered":"2024-02-19T21:13:13.324229+0000","last_clean":"2024-02-19T21:13:13.324229+0000","last_became_active":"2024-02-19T21:10:41.294592+0000","last_became_peered":"2024-02-19T21:10:41.294592+0000","last_unstale":"2024-02-19T21:13:13.324229+0000","last_undegraded":"2024-02-19T21:13:13.324229+0000","last_fullsized":"2024-02-19T21:13:13.324229+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:48.430223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:48.430223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"493'11","reported_seq":626,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-02-19T21:13:14.333149+0000","last_change":"2024-02-19T21:12:10.210682+0000","last_active":"2024-02-19T21:13:14.333149+0000","last_peered":"2024-02-19T21:13:14.333149+0000","last_clean":"2024-02-19T21:13:14.333149+0000","last_became_active":"2024-02-19T21:12:09.352204+0000","last_became_peered":"2024-02-19T21:12:09.352204+0000","last_unstale":"2024-02-19T21:13:14.333149+0000","last_undegraded":"2024-02-19T21:13:14.333149+0000","last_fullsized":"2024-02-19T21:13:14.333149+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'10","last_scrub_stamp":"2024-02-19T21:10:59.405417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.405417+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":523,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.530806+0000","last_change":"2024-02-19T21:11:43.546468+0000","last_active":"2024-02-19T21:13:13.530806+0000","last_peered":"2024-02-19T21:13:13.530806+0000","last_clean":"2024-02-19T21:13:13.530806+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:13:13.530806+0000","last_undegraded":"2024-02-19T21:13:13.530806+0000","last_fullsized":"2024-02-19T21:13:13.530806+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:11:43.546383+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.546383+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":476,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.518245+0000","last_change":"2024-02-19T21:11:38.369815+0000","last_active":"2024-02-19T21:13:13.518245+0000","last_peered":"2024-02-19T21:13:13.518245+0000","last_clean":"2024-02-19T21:13:13.518245+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:13:13.518245+0000","last_undegraded":"2024-02-19T21:13:13.518245+0000","last_fullsized":"2024-02-19T21:13:13.518245+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.369716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.369716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"459'19","reported_seq":541,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.327662+0000","last_change":"2024-02-19T21:12:10.390849+0000","last_active":"2024-02-19T21:13:13.327662+0000","last_peered":"2024-02-19T21:13:13.327662+0000","last_clean":"2024-02-19T21:13:13.327662+0000","last_became_active":"2024-02-19T21:12:09.350490+0000","last_became_peered":"2024-02-19T21:12:09.350490+0000","last_unstale":"2024-02-19T21:13:13.327662+0000","last_undegraded":"2024-02-19T21:13:13.327662+0000","last_fullsized":"2024-02-19T21:13:13.327662+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:10:54.435839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.435839+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":287,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.509116+0000","last_change":"2024-02-19T21:12:09.333078+0000","last_active":"2024-02-19T21:13:13.509116+0000","last_peered":"2024-02-19T21:13:13.509116+0000","last_clean":"2024-02-19T21:13:13.509116+0000","last_became_active":"2024-02-19T21:12:09.332587+0000","last_became_peered":"2024-02-19T21:12:09.332587+0000","last_unstale":"2024-02-19T21:13:13.509116+0000","last_undegraded":"2024-02-19T21:13:13.509116+0000","last_fullsized":"2024-02-19T21:13:13.509116+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:32.600213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:32.600213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":363,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.531816+0000","last_change":"2024-02-19T21:11:59.334743+0000","last_active":"2024-02-19T21:13:13.531816+0000","last_peered":"2024-02-19T21:13:13.531816+0000","last_clean":"2024-02-19T21:13:13.531816+0000","last_became_active":"2024-02-19T21:11:59.334454+0000","last_became_peered":"2024-02-19T21:11:59.334454+0000","last_unstale":"2024-02-19T21:13:13.531816+0000","last_undegraded":"2024-02-19T21:13:13.531816+0000","last_fullsized":"2024-02-19T21:13:13.531816+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.692260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.692260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":534,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.523905+0000","last_change":"2024-02-19T21:12:06.991132+0000","last_active":"2024-02-19T21:13:13.523905+0000","last_peered":"2024-02-19T21:13:13.523905+0000","last_clean":"2024-02-19T21:13:13.523905+0000","last_became_active":"2024-02-19T21:10:41.312577+0000","last_became_peered":"2024-02-19T21:10:41.312577+0000","last_unstale":"2024-02-19T21:13:13.523905+0000","last_undegraded":"2024-02-19T21:13:13.523905+0000","last_fullsized":"2024-02-19T21:13:13.523905+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.991039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.991039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":488,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.523178+0000","last_change":"2024-02-19T21:10:59.782435+0000","last_active":"2024-02-19T21:13:13.523178+0000","last_peered":"2024-02-19T21:13:13.523178+0000","last_clean":"2024-02-19T21:13:13.523178+0000","last_became_active":"2024-02-19T21:10:41.311050+0000","last_became_peered":"2024-02-19T21:10:41.311050+0000","last_unstale":"2024-02-19T21:13:13.523178+0000","last_undegraded":"2024-02-19T21:13:13.523178+0000","last_fullsized":"2024-02-19T21:13:13.523178+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:59.782367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.782367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"459'34","reported_seq":623,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.519967+0000","last_change":"2024-02-19T21:10:19.109353+0000","last_active":"2024-02-19T21:13:13.519967+0000","last_peered":"2024-02-19T21:13:13.519967+0000","last_clean":"2024-02-19T21:13:13.519967+0000","last_became_active":"2024-02-19T21:08:04.248378+0000","last_became_peered":"2024-02-19T21:08:04.248378+0000","last_unstale":"2024-02-19T21:13:13.519967+0000","last_undegraded":"2024-02-19T21:13:13.519967+0000","last_fullsized":"2024-02-19T21:13:13.519967+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'22","last_scrub_stamp":"2024-02-19T21:10:19.109244+0000","last_deep_scrub":"419'22","last_deep_scrub_stamp":"2024-02-19T21:10:19.109244+0000","last_clean_scrub_stamp":"2024-02-19T21:10:19.109244+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":484,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.502703+0000","last_change":"2024-02-19T21:12:09.341126+0000","last_active":"2024-02-19T21:13:13.502703+0000","last_peered":"2024-02-19T21:13:13.502703+0000","last_clean":"2024-02-19T21:13:13.502703+0000","last_became_active":"2024-02-19T21:12:09.340803+0000","last_became_peered":"2024-02-19T21:12:09.340803+0000","last_unstale":"2024-02-19T21:13:13.502703+0000","last_undegraded":"2024-02-19T21:13:13.502703+0000","last_fullsized":"2024-02-19T21:13:13.502703+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.379335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.379335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":545,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.524138+0000","last_change":"2024-02-19T21:11:59.323076+0000","last_active":"2024-02-19T21:13:13.524138+0000","last_peered":"2024-02-19T21:13:13.524138+0000","last_clean":"2024-02-19T21:13:13.524138+0000","last_became_active":"2024-02-19T21:11:59.322735+0000","last_became_peered":"2024-02-19T21:11:59.322735+0000","last_unstale":"2024-02-19T21:13:13.524138+0000","last_undegraded":"2024-02-19T21:13:13.524138+0000","last_fullsized":"2024-02-19T21:13:13.524138+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:53.709456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.709456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":147,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.503648+0000","last_change":"2024-02-19T21:12:09.370734+0000","last_active":"2024-02-19T21:13:13.503648+0000","last_peered":"2024-02-19T21:13:13.503648+0000","last_clean":"2024-02-19T21:13:13.503648+0000","last_became_active":"2024-02-19T21:12:09.370318+0000","last_became_peered":"2024-02-19T21:12:09.370318+0000","last_unstale":"2024-02-19T21:13:13.503648+0000","last_undegraded":"2024-02-19T21:13:13.503648+0000","last_fullsized":"2024-02-19T21:13:13.503648+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.335152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.335152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":354,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.519763+0000","last_change":"2024-02-19T21:12:09.340710+0000","last_active":"2024-02-19T21:13:13.519763+0000","last_peered":"2024-02-19T21:13:13.519763+0000","last_clean":"2024-02-19T21:13:13.519763+0000","last_became_active":"2024-02-19T21:12:09.340360+0000","last_became_peered":"2024-02-19T21:12:09.340360+0000","last_unstale":"2024-02-19T21:13:13.519763+0000","last_undegraded":"2024-02-19T21:13:13.519763+0000","last_fullsized":"2024-02-19T21:13:13.519763+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.342173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.342173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"459'18","reported_seq":337,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.509240+0000","last_change":"2024-02-19T21:12:50.243591+0000","last_active":"2024-02-19T21:13:13.509240+0000","last_peered":"2024-02-19T21:13:13.509240+0000","last_clean":"2024-02-19T21:13:13.509240+0000","last_became_active":"2024-02-19T21:10:41.307690+0000","last_became_peered":"2024-02-19T21:10:41.307690+0000","last_unstale":"2024-02-19T21:13:13.509240+0000","last_undegraded":"2024-02-19T21:13:13.509240+0000","last_fullsized":"2024-02-19T21:13:13.509240+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'18","last_scrub_stamp":"2024-02-19T21:12:50.243498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:50.243498+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":278,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.511068+0000","last_change":"2024-02-19T21:12:09.344823+0000","last_active":"2024-02-19T21:13:13.511068+0000","last_peered":"2024-02-19T21:13:13.511068+0000","last_clean":"2024-02-19T21:13:13.511068+0000","last_became_active":"2024-02-19T21:12:09.344422+0000","last_became_peered":"2024-02-19T21:12:09.344422+0000","last_unstale":"2024-02-19T21:13:13.511068+0000","last_undegraded":"2024-02-19T21:13:13.511068+0000","last_fullsized":"2024-02-19T21:13:13.511068+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:04.545988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:04.545988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":138,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.503451+0000","last_change":"2024-02-19T21:11:43.369675+0000","last_active":"2024-02-19T21:13:13.503451+0000","last_peered":"2024-02-19T21:13:13.503451+0000","last_clean":"2024-02-19T21:13:13.503451+0000","last_became_active":"2024-02-19T21:10:41.296849+0000","last_became_peered":"2024-02-19T21:10:41.296849+0000","last_unstale":"2024-02-19T21:13:13.503451+0000","last_undegraded":"2024-02-19T21:13:13.503451+0000","last_fullsized":"2024-02-19T21:13:13.503451+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:43.369607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.369607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":278,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.509363+0000","last_change":"2024-02-19T21:12:09.332385+0000","last_active":"2024-02-19T21:13:13.509363+0000","last_peered":"2024-02-19T21:13:13.509363+0000","last_clean":"2024-02-19T21:13:13.509363+0000","last_became_active":"2024-02-19T21:12:09.332044+0000","last_became_peered":"2024-02-19T21:12:09.332044+0000","last_unstale":"2024-02-19T21:13:13.509363+0000","last_undegraded":"2024-02-19T21:13:13.509363+0000","last_fullsized":"2024-02-19T21:13:13.509363+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:58.688166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.688166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":479,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.523073+0000","last_change":"2024-02-19T21:12:09.350012+0000","last_active":"2024-02-19T21:13:13.523073+0000","last_peered":"2024-02-19T21:13:13.523073+0000","last_clean":"2024-02-19T21:13:13.523073+0000","last_became_active":"2024-02-19T21:12:09.349709+0000","last_became_peered":"2024-02-19T21:12:09.349709+0000","last_unstale":"2024-02-19T21:13:13.523073+0000","last_undegraded":"2024-02-19T21:13:13.523073+0000","last_fullsized":"2024-02-19T21:13:13.523073+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:00.754327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.754327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"459'19","reported_seq":176,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.502920+0000","last_change":"2024-02-19T21:12:55.429884+0000","last_active":"2024-02-19T21:13:13.502920+0000","last_peered":"2024-02-19T21:13:13.502920+0000","last_clean":"2024-02-19T21:13:13.502920+0000","last_became_active":"2024-02-19T21:11:59.336536+0000","last_became_peered":"2024-02-19T21:11:59.336536+0000","last_unstale":"2024-02-19T21:13:13.502920+0000","last_undegraded":"2024-02-19T21:13:13.502920+0000","last_fullsized":"2024-02-19T21:13:13.502920+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:12:55.429801+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.429801+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":522,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.530646+0000","last_change":"2024-02-19T21:11:30.472665+0000","last_active":"2024-02-19T21:13:13.530646+0000","last_peered":"2024-02-19T21:13:13.530646+0000","last_clean":"2024-02-19T21:13:13.530646+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:13:13.530646+0000","last_undegraded":"2024-02-19T21:13:13.530646+0000","last_fullsized":"2024-02-19T21:13:13.530646+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:30.472591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:30.472591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":528,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325640+0000","last_change":"2024-02-19T21:13:06.395759+0000","last_active":"2024-02-19T21:13:13.325640+0000","last_peered":"2024-02-19T21:13:13.325640+0000","last_clean":"2024-02-19T21:13:13.325640+0000","last_became_active":"2024-02-19T21:11:59.323593+0000","last_became_peered":"2024-02-19T21:11:59.323593+0000","last_unstale":"2024-02-19T21:13:13.325640+0000","last_undegraded":"2024-02-19T21:13:13.325640+0000","last_fullsized":"2024-02-19T21:13:13.325640+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:06.395650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.395650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":357,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.519436+0000","last_change":"2024-02-19T21:11:59.352657+0000","last_active":"2024-02-19T21:13:13.519436+0000","last_peered":"2024-02-19T21:13:13.519436+0000","last_clean":"2024-02-19T21:13:13.519436+0000","last_became_active":"2024-02-19T21:11:59.352235+0000","last_became_peered":"2024-02-19T21:11:59.352235+0000","last_unstale":"2024-02-19T21:13:13.519436+0000","last_undegraded":"2024-02-19T21:13:13.519436+0000","last_fullsized":"2024-02-19T21:13:13.519436+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:10.642603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:10:10.642603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":505,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.518610+0000","last_change":"2024-02-19T21:13:10.146168+0000","last_active":"2024-02-19T21:13:13.518610+0000","last_peered":"2024-02-19T21:13:13.518610+0000","last_clean":"2024-02-19T21:13:13.518610+0000","last_became_active":"2024-02-19T21:11:59.357345+0000","last_became_peered":"2024-02-19T21:11:59.357345+0000","last_unstale":"2024-02-19T21:13:13.518610+0000","last_undegraded":"2024-02-19T21:13:13.518610+0000","last_fullsized":"2024-02-19T21:13:13.518610+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:10.146102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:13:10.146102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"459'7","reported_seq":147,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.503069+0000","last_change":"2024-02-19T21:12:02.521592+0000","last_active":"2024-02-19T21:13:13.503069+0000","last_peered":"2024-02-19T21:13:13.503069+0000","last_clean":"2024-02-19T21:13:13.503069+0000","last_became_active":"2024-02-19T21:10:41.297295+0000","last_became_peered":"2024-02-19T21:10:41.297295+0000","last_unstale":"2024-02-19T21:13:13.503069+0000","last_undegraded":"2024-02-19T21:13:13.503069+0000","last_fullsized":"2024-02-19T21:13:13.503069+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'7","last_scrub_stamp":"2024-02-19T21:12:02.521523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:02.521523+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":545,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.531006+0000","last_change":"2024-02-19T21:12:09.324077+0000","last_active":"2024-02-19T21:13:13.531006+0000","last_peered":"2024-02-19T21:13:13.531006+0000","last_clean":"2024-02-19T21:13:13.531006+0000","last_became_active":"2024-02-19T21:12:09.323141+0000","last_became_peered":"2024-02-19T21:12:09.323141+0000","last_unstale":"2024-02-19T21:13:13.531006+0000","last_undegraded":"2024-02-19T21:13:13.531006+0000","last_fullsized":"2024-02-19T21:13:13.531006+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:04.547957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:04.547957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":541,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.530333+0000","last_change":"2024-02-19T21:12:51.411694+0000","last_active":"2024-02-19T21:13:13.530333+0000","last_peered":"2024-02-19T21:13:13.530333+0000","last_clean":"2024-02-19T21:13:13.530333+0000","last_became_active":"2024-02-19T21:10:41.307400+0000","last_became_peered":"2024-02-19T21:10:41.307400+0000","last_unstale":"2024-02-19T21:13:13.530333+0000","last_undegraded":"2024-02-19T21:13:13.530333+0000","last_fullsized":"2024-02-19T21:13:13.530333+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:51.411574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.411574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":535,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325266+0000","last_change":"2024-02-19T21:10:15.161470+0000","last_active":"2024-02-19T21:13:13.325266+0000","last_peered":"2024-02-19T21:13:13.325266+0000","last_clean":"2024-02-19T21:13:13.325266+0000","last_became_active":"2024-02-19T21:08:04.266100+0000","last_became_peered":"2024-02-19T21:08:04.266100+0000","last_unstale":"2024-02-19T21:13:13.325266+0000","last_undegraded":"2024-02-19T21:13:13.325266+0000","last_fullsized":"2024-02-19T21:13:13.325266+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:15.161397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:15.161397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":381,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.327472+0000","last_change":"2024-02-19T21:12:40.365196+0000","last_active":"2024-02-19T21:13:13.327472+0000","last_peered":"2024-02-19T21:13:13.327472+0000","last_clean":"2024-02-19T21:13:13.327472+0000","last_became_active":"2024-02-19T21:12:40.364772+0000","last_became_peered":"2024-02-19T21:12:40.364772+0000","last_unstale":"2024-02-19T21:13:13.327472+0000","last_undegraded":"2024-02-19T21:13:13.327472+0000","last_fullsized":"2024-02-19T21:13:13.327472+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:56.083403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:56.083403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"459'15","reported_seq":176,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.502545+0000","last_change":"2024-02-19T21:12:09.959562+0000","last_active":"2024-02-19T21:13:13.502545+0000","last_peered":"2024-02-19T21:13:13.502545+0000","last_clean":"2024-02-19T21:13:13.502545+0000","last_became_active":"2024-02-19T21:12:09.352882+0000","last_became_peered":"2024-02-19T21:12:09.352882+0000","last_unstale":"2024-02-19T21:13:13.502545+0000","last_undegraded":"2024-02-19T21:13:13.502545+0000","last_fullsized":"2024-02-19T21:13:13.502545+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:11:03.423767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.423767+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":278,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.510243+0000","last_change":"2024-02-19T21:11:59.337265+0000","last_active":"2024-02-19T21:13:13.510243+0000","last_peered":"2024-02-19T21:13:13.510243+0000","last_clean":"2024-02-19T21:13:13.510243+0000","last_became_active":"2024-02-19T21:11:59.336778+0000","last_became_peered":"2024-02-19T21:11:59.336778+0000","last_unstale":"2024-02-19T21:13:13.510243+0000","last_undegraded":"2024-02-19T21:13:13.510243+0000","last_fullsized":"2024-02-19T21:13:13.510243+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:50.769620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.769620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":518,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.522876+0000","last_change":"2024-02-19T21:10:18.737245+0000","last_active":"2024-02-19T21:13:13.522876+0000","last_peered":"2024-02-19T21:13:13.522876+0000","last_clean":"2024-02-19T21:13:13.522876+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:13:13.522876+0000","last_undegraded":"2024-02-19T21:13:13.522876+0000","last_fullsized":"2024-02-19T21:13:13.522876+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:18.737176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:10:18.737176+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":138,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.502973+0000","last_change":"2024-02-19T21:12:07.416212+0000","last_active":"2024-02-19T21:13:13.502973+0000","last_peered":"2024-02-19T21:13:13.502973+0000","last_clean":"2024-02-19T21:13:13.502973+0000","last_became_active":"2024-02-19T21:10:41.309550+0000","last_became_peered":"2024-02-19T21:10:41.309550+0000","last_unstale":"2024-02-19T21:13:13.502973+0000","last_undegraded":"2024-02-19T21:13:13.502973+0000","last_fullsized":"2024-02-19T21:13:13.502973+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:07.416150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:07.416150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":498,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.325795+0000","last_change":"2024-02-19T21:11:59.333342+0000","last_active":"2024-02-19T21:13:13.325795+0000","last_peered":"2024-02-19T21:13:13.325795+0000","last_clean":"2024-02-19T21:13:13.325795+0000","last_became_active":"2024-02-19T21:11:59.332987+0000","last_became_peered":"2024-02-19T21:11:59.332987+0000","last_unstale":"2024-02-19T21:13:13.325795+0000","last_undegraded":"2024-02-19T21:13:13.325795+0000","last_fullsized":"2024-02-19T21:13:13.325795+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:08.043028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:08.043028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"459'27","reported_seq":328,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.508697+0000","last_change":"2024-02-19T21:11:59.360178+0000","last_active":"2024-02-19T21:13:13.508697+0000","last_peered":"2024-02-19T21:13:13.508697+0000","last_clean":"2024-02-19T21:13:13.508697+0000","last_became_active":"2024-02-19T21:11:59.340940+0000","last_became_peered":"2024-02-19T21:11:59.340940+0000","last_unstale":"2024-02-19T21:13:13.508697+0000","last_undegraded":"2024-02-19T21:13:13.508697+0000","last_fullsized":"2024-02-19T21:13:13.508697+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:10:02.532799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:02.532799+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":392,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.531636+0000","last_change":"2024-02-19T21:12:09.339398+0000","last_active":"2024-02-19T21:13:13.531636+0000","last_peered":"2024-02-19T21:13:13.531636+0000","last_clean":"2024-02-19T21:13:13.531636+0000","last_became_active":"2024-02-19T21:12:09.338747+0000","last_became_peered":"2024-02-19T21:12:09.338747+0000","last_unstale":"2024-02-19T21:13:13.531636+0000","last_undegraded":"2024-02-19T21:13:13.531636+0000","last_fullsized":"2024-02-19T21:13:13.531636+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.720583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.720583+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":147,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.502566+0000","last_change":"2024-02-19T21:12:09.346777+0000","last_active":"2024-02-19T21:13:13.502566+0000","last_peered":"2024-02-19T21:13:13.502566+0000","last_clean":"2024-02-19T21:13:13.502566+0000","last_became_active":"2024-02-19T21:12:09.346258+0000","last_became_peered":"2024-02-19T21:12:09.346258+0000","last_unstale":"2024-02-19T21:13:13.502566+0000","last_undegraded":"2024-02-19T21:13:13.502566+0000","last_fullsized":"2024-02-19T21:13:13.502566+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.407543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.407543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":278,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.510946+0000","last_change":"2024-02-19T21:12:09.331143+0000","last_active":"2024-02-19T21:13:13.510946+0000","last_peered":"2024-02-19T21:13:13.510946+0000","last_clean":"2024-02-19T21:13:13.510946+0000","last_became_active":"2024-02-19T21:12:09.330530+0000","last_became_peered":"2024-02-19T21:12:09.330530+0000","last_unstale":"2024-02-19T21:13:13.510946+0000","last_undegraded":"2024-02-19T21:13:13.510946+0000","last_fullsized":"2024-02-19T21:13:13.510946+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:33.559363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:33.559363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":301,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-02-19T21:13:13.508992+0000","last_change":"2024-02-19T21:12:09.345873+0000","last_active":"2024-02-19T21:13:13.508992+0000","last_peered":"2024-02-19T21:13:13.508992+0000","last_clean":"2024-02-19T21:13:13.508992+0000","last_became_active":"2024-02-19T21:12:09.345500+0000","last_became_peered":"2024-02-19T21:12:09.345500+0000","last_unstale":"2024-02-19T21:13:13.508992+0000","last_undegraded":"2024-02-19T21:13:13.508992+0000","last_fullsized":"2024-02-19T21:13:13.508992+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.764658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.764658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1102,"num_read_kb":859,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":7558,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":712,"ondisk_log_size":712,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":709,"num_read_kb":704,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":126,"acting":126,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":15,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266834,"num_pgs":121,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320768,"kb_used_data":9816,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93424384,"statfs":{"total":95995035648,"available":95666569216,"internally_reserved":0,"allocated":10051584,"data_stored":7209650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.86399999999999999},"min":{"1min":0.30399999999999999,"5min":0.223,"15min":0.215},"max":{"1min":0.66000000000000003,"5min":0.79200000000000004,"15min":89.456999999999994},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.47699999999999998,"15min":0.78300000000000003},"min":{"1min":0.29599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.752,"5min":0.96999999999999997,"15min":70.451999999999998},"last":0.46700000000000003}]},{"osd":1,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48599999999999999,"15min":0.69599999999999995},"min":{"1min":0.36299999999999999,"5min":0.28299999999999997,"15min":0.28199999999999997},"max":{"1min":0.61399999999999999,"5min":0.85099999999999998,"15min":34.273000000000003},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46899999999999997,"15min":0.93100000000000005},"min":{"1min":0.29499999999999998,"5min":0.28299999999999997,"15min":0.254},"max":{"1min":0.63500000000000001,"5min":0.79200000000000004,"15min":89.594999999999999},"last":0.627}]},{"osd":2,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.46999999999999997,"15min":0.93700000000000006},"min":{"1min":0.33000000000000002,"5min":0.28100000000000003,"15min":0.24299999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":104.413},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.495,"15min":0.94699999999999995},"min":{"1min":0.375,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.624,"5min":0.76400000000000001,"15min":104.268},"last":0.66500000000000004}]},{"osd":3,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.95399999999999996},"min":{"1min":0.33400000000000002,"5min":0.29099999999999998,"15min":0.254},"max":{"1min":0.62,"5min":1.8939999999999999,"15min":105.3},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.50700000000000001,"15min":0.59999999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.68000000000000005,"5min":0.879,"15min":13.243},"last":0.64600000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.42499999999999999,"15min":0.46000000000000002},"min":{"1min":0.30599999999999999,"5min":0.27400000000000002,"15min":0.24299999999999999},"max":{"1min":0.54400000000000004,"5min":0.69299999999999995,"15min":1.373},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.439,"5min":0.46000000000000002,"15min":0.47499999999999998},"min":{"1min":0.29199999999999998,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":0.68300000000000005,"5min":0.751,"15min":1.0740000000000001},"last":0.41599999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.443,"15min":0.45600000000000002},"min":{"1min":0.30299999999999999,"5min":0.28599999999999998,"15min":0.252},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.379},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48599999999999999,"15min":0.504},"min":{"1min":0.35499999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.755,"5min":0.79300000000000004,"15min":0.85899999999999999},"last":0.52100000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.51600000000000001},"min":{"1min":0.33700000000000002,"5min":0.30199999999999999,"15min":0.28799999999999998},"max":{"1min":0.71699999999999997,"5min":0.76700000000000002,"15min":1.0049999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52200000000000002,"15min":0.54900000000000004},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33700000000000002},"max":{"1min":0.72399999999999998,"5min":0.75900000000000001,"15min":1.262},"last":0.70599999999999996}]}]},{"osd":6,"up_from":0,"seq":193273528472,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834575023,"num_pgs":132,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320232,"kb_used_data":10240,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93424920,"statfs":{"total":95995035648,"available":95667118080,"internally_reserved":0,"allocated":10485760,"data_stored":7522950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.33300000000000002,"15min":0.54500000000000004},"min":{"1min":0.19700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.38900000000000001,"5min":2.0379999999999998,"15min":27.777000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34799999999999998,"15min":0.57399999999999995},"min":{"1min":0.215,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.51900000000000002,"5min":1.9870000000000001,"15min":27.631},"last":0.372}]},{"osd":2,"last update":"Mon Feb 19 21:12:34 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.371,"15min":0.70599999999999996},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":27.798999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.38900000000000001,"15min":0.72299999999999998},"min":{"1min":0.24299999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.089,"5min":2.089,"15min":27.606999999999999},"last":0.435}]},{"osd":3,"last update":"Mon Feb 19 21:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.374,"15min":0.56599999999999995},"min":{"1min":0.253,"5min":0.217,"15min":0.217},"max":{"1min":0.499,"5min":1.7450000000000001,"15min":15.574999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.41099999999999998,"15min":0.66200000000000003},"min":{"1min":0.31,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.622,"5min":2.0070000000000001,"15min":27.82},"last":0.45700000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:12:15 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.501,"15min":0.65000000000000002},"min":{"1min":0.27800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.59199999999999997,"5min":1.149,"15min":9.6430000000000007},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53200000000000003,"15min":0.752},"min":{"1min":0.34799999999999998,"5min":0.308,"15min":0.308},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":27.707000000000001},"last":0.61599999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:12:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52700000000000002,"15min":0.73899999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98599999999999999,"5min":1.048,"15min":27.529},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.51800000000000002,"15min":0.67500000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67400000000000004,"5min":1.1819999999999999,"15min":9.6799999999999997},"last":0.623}]},{"osd":6,"last update":"Mon Feb 19 21:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52600000000000002,"15min":0.65800000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70399999999999996,"5min":1.1659999999999999,"15min":9.6989999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.60099999999999998,"15min":0.78600000000000003},"min":{"1min":0.40799999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75900000000000001,"5min":3.4359999999999999,"15min":27.733000000000001},"last":0.63500000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:12:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.56799999999999995,"15min":0.77500000000000002},"min":{"1min":0.40000000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":27.576000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.58299999999999996,"15min":0.80500000000000005},"min":{"1min":0.42099999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.71599999999999997,"5min":2.0169999999999999,"15min":27.756},"last":0.66700000000000004}]}]},{"osd":0,"up_from":0,"seq":34359738549,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411498,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316152,"kb_used_data":10256,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93429000,"statfs":{"total":95995035648,"available":95671296000,"internally_reserved":0,"allocated":10502144,"data_stored":7527923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:12:33 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.377,"15min":0.53600000000000003},"min":{"1min":0.22900000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.54400000000000004,"5min":3.593,"15min":26.690999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.35899999999999999,"15min":0.59099999999999997},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.214},"max":{"1min":0.56799999999999995,"5min":1.488,"15min":24.315000000000001},"last":0.40100000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:12:33 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.42199999999999999,"15min":0.70299999999999996},"min":{"1min":0.249,"5min":0.20599999999999999,"15min":0.20100000000000001},"max":{"1min":0.47199999999999998,"5min":7.0010000000000003,"15min":28.882000000000001},"last":0.372},{"interface":"front","average":{"1min":0.372,"5min":0.42299999999999999,"15min":0.68200000000000005},"min":{"1min":0.27000000000000002,"5min":0.23899999999999999,"15min":0.22500000000000001},"max":{"1min":0.60899999999999999,"5min":3.3010000000000002,"15min":28.382999999999999},"last":0.41199999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.53700000000000003,"15min":0.71199999999999997},"min":{"1min":0.27800000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68899999999999995,"5min":12.75,"15min":28.742000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.44900000000000001,"15min":0.63300000000000001},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.25700000000000001},"max":{"1min":0.58399999999999996,"5min":3.6499999999999999,"15min":26.719999999999999},"last":0.44}]},{"osd":4,"last update":"Mon Feb 19 21:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.60099999999999998,"15min":0.69699999999999995},"min":{"1min":0.36299999999999999,"5min":0.33400000000000002,"15min":0.30599999999999999},"max":{"1min":0.755,"5min":3.4039999999999999,"15min":24.356000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.67100000000000004,"15min":0.71999999999999997},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.32400000000000001},"max":{"1min":0.72799999999999998,"5min":7.1340000000000003,"15min":20.707000000000001},"last":0.53300000000000003}]},{"osd":5,"last update":"Mon Feb 19 21:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57799999999999996,"15min":0.68000000000000005},"min":{"1min":0.36399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81599999999999995,"5min":3.3090000000000002,"15min":9.9499999999999993},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60899999999999999,"15min":0.79100000000000004},"min":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.38400000000000001},"max":{"1min":0.77500000000000002,"5min":3.2690000000000001,"15min":26.606000000000002},"last":0.52600000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:12:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63200000000000001,"15min":0.80200000000000005},"min":{"1min":0.40899999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81000000000000005,"5min":4.5099999999999998,"15min":24.381},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58799999999999997,"15min":0.73699999999999999},"min":{"1min":0.41899999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.745,"5min":0.93600000000000005,"15min":28.706},"last":0.58299999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:12:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.63200000000000001,"15min":0.77800000000000002},"min":{"1min":0.441,"5min":0.42199999999999999,"15min":0.34200000000000003},"max":{"1min":0.77600000000000002,"5min":3.3359999999999999,"15min":26.658000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59999999999999998,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82099999999999995,"5min":1.5109999999999999,"15min":24.402000000000001},"last":0.58899999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182566,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313168,"kb_used_data":6244,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93431984,"statfs":{"total":95995035648,"available":95674351616,"internally_reserved":0,"allocated":6393856,"data_stored":3515863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.71599999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.156},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":77.662000000000006},"last":0.31},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.35499999999999998,"15min":0.49399999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.19700000000000001},"max":{"1min":0.60199999999999998,"5min":0.60999999999999999,"15min":17.456},"last":0.27900000000000003}]},{"osd":1,"last update":"Mon Feb 19 21:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.52600000000000002},"min":{"1min":0.255,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":0.66300000000000003,"5min":6.9299999999999997,"15min":17.327999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.42499999999999999,"15min":0.50600000000000001},"min":{"1min":0.20699999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.95599999999999996,"5min":7.3620000000000001,"15min":15.004},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.376,"15min":0.66300000000000003},"min":{"1min":0.27300000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":56.890999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.36599999999999999,"15min":0.64400000000000002},"min":{"1min":0.246,"5min":0.24399999999999999,"15min":0.20899999999999999},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":44.600999999999999},"last":0.26600000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52500000000000002,"15min":0.63400000000000001},"min":{"1min":0.33400000000000002,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":12.847},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.52300000000000002,"15min":0.70599999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72299999999999998,"5min":0.94299999999999995,"15min":26.393999999999998},"last":0.372}]},{"osd":5,"last update":"Mon Feb 19 21:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.61799999999999999,"15min":0.68400000000000005},"min":{"1min":0.39100000000000001,"5min":0.34100000000000003,"15min":0.313},"max":{"1min":0.68300000000000005,"5min":9.2360000000000007,"15min":17.401},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54300000000000004,"15min":0.65400000000000003},"min":{"1min":0.35999999999999999,"5min":0.313,"15min":0.313},"max":{"1min":1.3660000000000001,"5min":1.9690000000000001,"15min":12.882},"last":0.45200000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56000000000000005,"15min":0.65200000000000002},"min":{"1min":0.38,"5min":0.34799999999999998,"15min":0.34300000000000003},"max":{"1min":0.69999999999999996,"5min":1.9790000000000001,"15min":17.367000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54900000000000004,"15min":0.66200000000000003},"min":{"1min":0.377,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":0.89400000000000002,"5min":0.95099999999999996,"15min":12.904},"last":0.52700000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:12:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55900000000000005,"15min":0.65500000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.038,"5min":1.038,"15min":17.428999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.80000000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.34499999999999997},"max":{"1min":1.111,"5min":1.111,"15min":44.378},"last":0.49099999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986338,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312780,"kb_used_data":5924,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93432372,"statfs":{"total":95995035648,"available":95674748928,"internally_reserved":0,"allocated":6066176,"data_stored":3206648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.54700000000000004,"15min":0.63900000000000001},"min":{"1min":0.28199999999999997,"5min":0.23599999999999999,"15min":0.22500000000000001},"max":{"1min":0.68300000000000005,"5min":8.0530000000000008,"15min":19.878},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.49199999999999999,"15min":0.85799999999999998},"min":{"1min":0.32300000000000001,"5min":0.22500000000000001,"15min":0.20499999999999999},"max":{"1min":0.61099999999999999,"5min":3.3530000000000002,"15min":70.063000000000002},"last":0.379}]},{"osd":1,"last update":"Mon Feb 19 21:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46700000000000003,"15min":0.63600000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":26.510999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.48599999999999999,"15min":0.68200000000000005},"min":{"1min":0.32100000000000001,"5min":0.312,"15min":0.23200000000000001},"max":{"1min":0.61899999999999999,"5min":0.82999999999999996,"15min":25.949000000000002},"last":0.51800000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.57599999999999996,"15min":0.71899999999999997},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.67400000000000004,"5min":8.1240000000000006,"15min":37.320999999999998},"last":2.2040000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51500000000000001,"15min":1.0600000000000001},"min":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.253},"max":{"1min":0.66500000000000004,"5min":2.9969999999999999,"15min":69.936999999999998},"last":0.50900000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.53400000000000003,"15min":0.91200000000000003},"min":{"1min":0.38100000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69399999999999995,"5min":3.323,"15min":69.793000000000006},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52000000000000002,"15min":0.76200000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76500000000000001,"5min":0.81799999999999995,"15min":17.986999999999998},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.218},"max":{"1min":0.78800000000000003,"5min":1.036,"15min":3.2799999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.48199999999999998,"15min":0.46300000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.23899999999999999},"max":{"1min":0.745,"5min":1.121,"15min":4.4340000000000002},"last":0.53500000000000003}]},{"osd":6,"last update":"Mon Feb 19 21:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.57399999999999995,"15min":0.47599999999999998},"min":{"1min":0.29199999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.78700000000000003,"5min":8.0969999999999995,"15min":8.0969999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.51800000000000002},"min":{"1min":0.35599999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.4550000000000001,"5min":3.0270000000000001,"15min":4.5609999999999999},"last":0.59299999999999997}]},{"osd":7,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56299999999999994,"15min":0.51000000000000001},"min":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.71699999999999997,"5min":3.2909999999999999,"15min":6.0730000000000004},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.55200000000000005,"15min":0.51200000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.26400000000000001},"max":{"1min":0.72699999999999998,"5min":1.0209999999999999,"15min":3.202},"last":0.57099999999999995}]}]},{"osd":5,"up_from":37,"seq":158913790109,"num_pgs":121,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320944,"kb_used_data":6372,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93424208,"statfs":{"total":95995035648,"available":95666388992,"internally_reserved":0,"allocated":6524928,"data_stored":3523872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.45800000000000002,"15min":1.8779999999999999},"min":{"1min":0.26200000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.64400000000000002,"5min":0.71699999999999997,"15min":276.47899999999998},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.436,"5min":0.48799999999999999,"15min":0.90100000000000002},"min":{"1min":0.27400000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.58099999999999996,"5min":0.95299999999999996,"15min":52.887},"last":0.47999999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.46999999999999997,"15min":1.266},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.23499999999999999},"max":{"1min":0.59599999999999997,"5min":1.599,"15min":120.011},"last":0.433},{"interface":"front","average":{"1min":0.443,"5min":0.48899999999999999,"15min":1.02},"min":{"1min":0.28599999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63100000000000001,"5min":0.97699999999999998,"15min":118.105},"last":0.45300000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47799999999999998,"15min":0.99399999999999999},"min":{"1min":0.32900000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65600000000000003,"5min":0.92300000000000004,"15min":82.242000000000004},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.55700000000000005,"15min":1.5129999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65300000000000002,"5min":4.5019999999999998,"15min":232.84999999999999},"last":0.60199999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52700000000000002,"15min":1.0649999999999999},"min":{"1min":0.35299999999999998,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":0.71199999999999997,"5min":4.4640000000000004,"15min":109.70099999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52400000000000002,"15min":0.91100000000000003},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":65.519000000000005},"last":0.48799999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.432,"15min":0.42099999999999999},"min":{"1min":0.26900000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66200000000000003,"5min":1.0129999999999999,"15min":1.861},"last":0.503},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.44,"15min":0.433},"min":{"1min":0.255,"5min":0.255,"15min":0.248},"max":{"1min":0.61599999999999999,"5min":0.72499999999999998,"15min":1.9730000000000001},"last":0.47299999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47299999999999998,"15min":0.45500000000000002},"min":{"1min":0.31,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69599999999999995,"5min":1.0289999999999999,"15min":1.823},"last":0.376},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47199999999999998,"15min":0.46300000000000002},"min":{"1min":0.30499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.40200000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:12:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.497,"15min":0.5},"min":{"1min":0.34000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63300000000000001,"5min":1.052,"15min":1.8879999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51200000000000001,"15min":0.50700000000000001},"min":{"1min":0.34399999999999997,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.64100000000000001,"5min":0.871,"15min":0.96799999999999997},"last":0.497}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:13:19.096 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-19T21:13:19.096 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:13:19.096 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.meta 2024-02-19T21:13:19.096 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:13:19.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.098+0000 7ff96ee22700 10 client.23445.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:19.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.098+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 14 2024-02-19T21:13:19.097 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.098+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff96812b290 osd=6 3 2024-02-19T21:13:19.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.098+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff96812b290 osd=6 3 2024-02-19T21:13:19.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.098+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff96812b290 osd=6 3 2024-02-19T21:13:19.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.098+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 14 2024-02-19T21:13:19.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.098+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff96812b290 osd=6 3 2024-02-19T21:13:19.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.098+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 6 14 2024-02-19T21:13:19.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.098+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff96812b290 osd=6 4 2024-02-19T21:13:19.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.098+0000 7ff96ee22700 10 client.23445.objecter _send_command 14 2024-02-19T21:13:19.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.098+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff968137040 con 0x7ff968133530 2024-02-19T21:13:19.100 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:13:19.100 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff968137040 con 0x7ff968133530 2024-02-19T21:13:19.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:19.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff965ffb700 10 client.23445.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:19.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff96812b290 osd=6 4 2024-02-19T21:13:19.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 6 14 2024-02-19T21:13:19.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff96ee22700 10 client.23445.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:19.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 15 2024-02-19T21:13:19.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968137080 osd=7 3 2024-02-19T21:13:19.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968137080 osd=7 3 2024-02-19T21:13:19.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968137080 osd=7 3 2024-02-19T21:13:19.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 15 2024-02-19T21:13:19.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff968137080 osd=7 3 2024-02-19T21:13:19.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 7 15 2024-02-19T21:13:19.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968137080 osd=7 4 2024-02-19T21:13:19.101 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff96ee22700 10 client.23445.objecter _send_command 15 2024-02-19T21:13:19.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.099+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9681383e0 con 0x7ff968137740 2024-02-19T21:13:19.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.100+0000 7ff965ffb700 10 client.23445.objecter ms_handle_connect 0x7ff968137740 2024-02-19T21:13:19.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.101+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff9681383e0 con 0x7ff968137740 2024-02-19T21:13:19.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.101+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:13:19.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.101+0000 7ff965ffb700 10 client.23445.objecter _finish_command 15 = system:0 2024-02-19T21:13:19.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.101+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff968137080 osd=7 4 2024-02-19T21:13:19.102 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.101+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 7 15 2024-02-19T21:13:19.111 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:13:19.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:19.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:13:19.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.116+0000 7ff96ee22700 10 client.23445.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:19.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.116+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign -1 16 2024-02-19T21:13:19.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.116+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968137080 osd=7 3 2024-02-19T21:13:19.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.116+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968137080 osd=7 3 2024-02-19T21:13:19.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.116+0000 7ff96ee22700 20 client.23445.objecter _get_session s=0x7ff968137080 osd=7 3 2024-02-19T21:13:19.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.116+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_remove -1 16 2024-02-19T21:13:19.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.116+0000 7ff96ee22700 20 client.23445.objecter get_session s=0x7ff968137080 osd=7 3 2024-02-19T21:13:19.116 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.116+0000 7ff96ee22700 15 client.23445.objecter _session_command_op_assign 7 16 2024-02-19T21:13:19.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.116+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968137080 osd=7 4 2024-02-19T21:13:19.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.116+0000 7ff96ee22700 10 client.23445.objecter _send_command 16 2024-02-19T21:13:19.117 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.116+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ff9681383e0 con 0x7ff968137740 2024-02-19T21:13:19.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.118+0000 7ff965ffb700 1 -- 172.21.15.163:0/2131631199 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ff9681383e0 con 0x7ff968137740 2024-02-19T21:13:19.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.118+0000 7ff965ffb700 10 client.23445.objecter ms_dispatch 0x7ff968001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:19.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.118+0000 7ff965ffb700 10 client.23445.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:19.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.118+0000 7ff965ffb700 20 client.23445.objecter put_session s=0x7ff968137080 osd=7 4 2024-02-19T21:13:19.118 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.118+0000 7ff965ffb700 15 client.23445.objecter _session_command_op_remove 7 16 2024-02-19T21:13:19.119 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:13:19.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 10 librados: watch_flush enter 2024-02-19T21:13:19.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 10 librados: watch_flush exit 2024-02-19T21:13:19.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 10 client.23445.objecter close_session for osd.0 2024-02-19T21:13:19.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 >> v1:172.21.15.163:6801/1559418115 conn(0x7ff9680f3930 legacy=0x7ff9680f3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:19.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968047540 osd=0 1 2024-02-19T21:13:19.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 10 client.23445.objecter close_session for osd.1 2024-02-19T21:13:19.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 >> v1:172.21.15.163:6805/3125047432 conn(0x7ff96811cf80 legacy=0x7ff96811f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:19.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff9680f4420 osd=1 1 2024-02-19T21:13:19.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 10 client.23445.objecter close_session for osd.2 2024-02-19T21:13:19.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 >> v1:172.21.15.163:6809/1563158487 conn(0x7ff968059e00 legacy=0x7ff968123d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:19.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff96811caf0 osd=2 1 2024-02-19T21:13:19.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 10 client.23445.objecter close_session for osd.3 2024-02-19T21:13:19.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 >> v1:172.21.15.163:6813/2112279769 conn(0x7ff9681218e0 legacy=0x7ff9681287b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:19.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968059770 osd=3 1 2024-02-19T21:13:19.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 10 client.23445.objecter close_session for osd.4 2024-02-19T21:13:19.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 >> v1:172.21.15.193:6800/3129716965 conn(0x7ff968126300 legacy=0x7ff96812a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:19.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968120e40 osd=4 1 2024-02-19T21:13:19.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 10 client.23445.objecter close_session for osd.5 2024-02-19T21:13:19.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 >> v1:172.21.15.193:6804/805161991 conn(0x7ff96812acf0 legacy=0x7ff968131900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:19.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968121110 osd=5 1 2024-02-19T21:13:19.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 10 client.23445.objecter close_session for osd.6 2024-02-19T21:13:19.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 >> v1:172.21.15.193:6808/787604615 conn(0x7ff968133530 legacy=0x7ff968135920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:19.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff96812b290 osd=6 1 2024-02-19T21:13:19.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 10 client.23445.objecter close_session for osd.7 2024-02-19T21:13:19.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 >> v1:172.21.15.193:6812/157927432 conn(0x7ff968137740 legacy=0x7ff968137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:19.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 20 client.23445.objecter put_session s=0x7ff968137080 osd=7 1 2024-02-19T21:13:19.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 20 client.23445.objecter shutdown clearing up homeless session... 2024-02-19T21:13:19.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 10 client.23445.objecter successfully canceled tick 2024-02-19T21:13:19.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96d3bf700 1 -- 172.21.15.163:0/2131631199 reap_dead start 2024-02-19T21:13:19.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96d3bf700 1 -- 172.21.15.163:0/2131631199 reap_dead start 2024-02-19T21:13:19.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96d3bf700 1 -- 172.21.15.163:0/2131631199 reap_dead start 2024-02-19T21:13:19.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96d3bf700 1 -- 172.21.15.163:0/2131631199 reap_dead start 2024-02-19T21:13:19.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96d3bf700 1 -- 172.21.15.163:0/2131631199 reap_dead start 2024-02-19T21:13:19.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 >> v1:172.21.15.163:6800/1552514681 conn(0x7ff958063d00 legacy=0x7ff9580661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:19.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 >> v1:172.21.15.163:6789/0 conn(0x7ff9681042d0 legacy=0x7ff968118450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:19.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 shutdown_connections 2024-02-19T21:13:19.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 >> 172.21.15.163:0/2131631199 conn(0x7ff9680f3550 msgr2=0x7ff968107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:19.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 shutdown_connections 2024-02-19T21:13:19.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 1 -- 172.21.15.163:0/2131631199 wait complete. 2024-02-19T21:13:19.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:19.119+0000 7ff96ee22700 1 librados: shutdown 2024-02-19T21:13:19.122 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:13:19.123 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:13:19.123 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:13:19.123 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:13:19.123 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:13:19.123 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:13:19.123 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:13:19.123 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:13:19.215 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:19.247 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:19.268 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:19.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:19.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:19.351 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:19.358 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:18.350784+0000 mon.a (mon.0) 2962 : audit [INF] from='client.48970 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi163-83928-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi163-83928-94"}]': finished 2024-02-19T21:13:19.358 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:18.350866+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e562: 8 total, 8 up, 6 in 2024-02-19T21:13:19.358 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:18.976396+0000 mon.a (mon.0) 2964 : audit [DBG] from='client.? v1:172.21.15.163:0/2131631199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:19.371 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:13:19.404 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:19.478 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:13:19.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:19.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:13:19.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:18.350784+0000 mon.a (mon.0) 2962 : audit [INF] from='client.48970 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi163-83928-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi163-83928-94"}]': finished 2024-02-19T21:13:19.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:18.350866+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e562: 8 total, 8 up, 6 in 2024-02-19T21:13:19.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:18.976396+0000 mon.a (mon.0) 2964 : audit [DBG] from='client.? v1:172.21.15.163:0/2131631199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:19.583 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:19.612 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:19.632 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:19.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:19.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:19.736 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:13:19.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:18.350784+0000 mon.a (mon.0) 2962 : audit [INF] from='client.48970 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi163-83928-94", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi163-83928-94"}]': finished 2024-02-19T21:13:19.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:18.350866+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e562: 8 total, 8 up, 6 in 2024-02-19T21:13:19.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:18.976396+0000 mon.a (mon.0) 2964 : audit [DBG] from='client.? v1:172.21.15.163:0/2131631199' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:19.763 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:19.886 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:13:19.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:19.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:13:19.904 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-19T21:13:19.990 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:20.021 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:20.041 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:20.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:20.145 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:13:20.171 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:20.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:19.267444+0000 mgr.y (mgr.14550) 497 : cluster [DBG] pgmap v843: 182 pgs: 8 unknown, 174 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:20.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:19.358264+0000 mon.a (mon.0) 2965 : cluster [DBG] osdmap e563: 8 total, 8 up, 6 in 2024-02-19T21:13:20.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:19.267444+0000 mgr.y (mgr.14550) 497 : cluster [DBG] pgmap v843: 182 pgs: 8 unknown, 174 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:20.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:19.358264+0000 mon.a (mon.0) 2965 : cluster [DBG] osdmap e563: 8 total, 8 up, 6 in 2024-02-19T21:13:20.519 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:13:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:20.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:13:20.623 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:20.653 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:19.267444+0000 mgr.y (mgr.14550) 497 : cluster [DBG] pgmap v843: 182 pgs: 8 unknown, 174 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:20.653 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:19.358264+0000 mon.a (mon.0) 2965 : cluster [DBG] osdmap e563: 8 total, 8 up, 6 in 2024-02-19T21:13:20.653 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:13:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:13:20.655 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:20.675 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:20.678 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:20.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:20.779 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:20.809 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:20.828 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:20.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:20.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:20.932 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:13:20.962 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:21.245 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:21.289 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:21.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:20.362508+0000 mon.a (mon.0) 2966 : cluster [DBG] osdmap e564: 8 total, 8 up, 6 in 2024-02-19T21:13:21.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:20.368442+0000 mon.b (mon.1) 721 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:21.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:20.369175+0000 mon.a (mon.0) 2967 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:21.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:20.570106+0000 mgr.y (mgr.14550) 498 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:21.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:20.362508+0000 mon.a (mon.0) 2966 : cluster [DBG] osdmap e564: 8 total, 8 up, 6 in 2024-02-19T21:13:21.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:20.368442+0000 mon.b (mon.1) 721 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:21.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:20.369175+0000 mon.a (mon.0) 2967 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:21.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:20.570106+0000 mgr.y (mgr.14550) 498 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:21.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:20.362508+0000 mon.a (mon.0) 2966 : cluster [DBG] osdmap e564: 8 total, 8 up, 6 in 2024-02-19T21:13:21.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:20.368442+0000 mon.b (mon.1) 721 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:21.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:20.369175+0000 mon.a (mon.0) 2967 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:21.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:20.570106+0000 mgr.y (mgr.14550) 498 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:22.509 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:13:22.652 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:21.267917+0000 mgr.y (mgr.14550) 499 : cluster [DBG] pgmap v846: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:13:22.652 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:21.358377+0000 mon.a (mon.0) 2968 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:13:22.653 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:21.367151+0000 mon.a (mon.0) 2969 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:13:22.653 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:21.367303+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e565: 8 total, 8 up, 6 in 2024-02-19T21:13:22.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:21.267917+0000 mgr.y (mgr.14550) 499 : cluster [DBG] pgmap v846: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:13:22.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:21.358377+0000 mon.a (mon.0) 2968 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:13:22.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:21.367151+0000 mon.a (mon.0) 2969 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:13:22.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:21.367303+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e565: 8 total, 8 up, 6 in 2024-02-19T21:13:22.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:21.267917+0000 mgr.y (mgr.14550) 499 : cluster [DBG] pgmap v846: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:13:22.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:21.358377+0000 mon.a (mon.0) 2968 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:13:22.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:21.367151+0000 mon.a (mon.0) 2969 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:13:22.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:21.367303+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e565: 8 total, 8 up, 6 in 2024-02-19T21:13:23.012 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:13:23.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:23.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:13:23.116 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:23.148 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:23.171 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:23.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:23.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:23.273 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:23.302 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:23.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:22.376051+0000 mon.a (mon.0) 2971 : cluster [DBG] osdmap e566: 8 total, 8 up, 6 in 2024-02-19T21:13:23.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:22.379551+0000 mon.b (mon.1) 722 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:23.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:22.380030+0000 mon.a (mon.0) 2972 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:23.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.699+0000 7f10ef932700 1 -- 172.21.15.163:0/2409545576 >> v1:172.21.15.163:6789/0 conn(0x7f10e80fc870 legacy=0x7f10e80fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:23.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.700+0000 7f10ef932700 1 -- 172.21.15.163:0/2409545576 shutdown_connections 2024-02-19T21:13:23.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.700+0000 7f10ef932700 1 -- 172.21.15.163:0/2409545576 >> 172.21.15.163:0/2409545576 conn(0x7f10e80f2470 msgr2=0x7f10e80f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:23.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.700+0000 7f10ef932700 1 -- 172.21.15.163:0/2409545576 shutdown_connections 2024-02-19T21:13:23.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.700+0000 7f10ef932700 1 -- 172.21.15.163:0/2409545576 wait complete. 2024-02-19T21:13:23.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.700+0000 7f10ef932700 1 librados: starting msgr at 2024-02-19T21:13:23.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.700+0000 7f10ef932700 1 librados: starting objecter 2024-02-19T21:13:23.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.700+0000 7f10ef932700 1 Processor -- start 2024-02-19T21:13:23.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.700+0000 7f10ef932700 1 -- start start 2024-02-19T21:13:23.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.700+0000 7f10ef932700 1 librados: setting wanted keys 2024-02-19T21:13:23.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.700+0000 7f10ef932700 1 librados: calling monclient init 2024-02-19T21:13:23.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.700+0000 7f10ef932700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10e81171c0 con 0x7f10e81031b0 2024-02-19T21:13:23.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.700+0000 7f10ef932700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10e81183a0 con 0x7f10e80fc870 2024-02-19T21:13:23.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.700+0000 7f10ef932700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10e81195a0 con 0x7f10e80ff670 2024-02-19T21:13:23.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10ececd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f10e80ff670 0x7f10e80f9eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:33362/0 (socket says 172.21.15.163:33362) 2024-02-19T21:13:23.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10ececd700 1 -- 172.21.15.163:0/3787012232 learned_addr learned my addr 172.21.15.163:0/3787012232 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:23.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10edecf700 1 --1- 172.21.15.163:0/3787012232 >> v1:172.21.15.163:6789/0 conn(0x7f10e81031b0 0x7f10e8115880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:39646/0 (socket says 172.21.15.163:39646) 2024-02-19T21:13:23.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 10 client.?.objecter ms_handle_connect 0x7f10e80ff670 2024-02-19T21:13:23.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:23.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 10 client.?.objecter ms_handle_connect 0x7f10e81031b0 2024-02-19T21:13:23.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:23.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 10 client.?.objecter ms_handle_connect 0x7f10e80fc870 2024-02-19T21:13:23.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:23.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275394287 0 0) 0x7f10e81195a0 con 0x7f10e80ff670 2024-02-19T21:13:23.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10d40038e0 con 0x7f10e80ff670 2024-02-19T21:13:23.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724135181 0 0) 0x7f10e81171c0 con 0x7f10e81031b0 2024-02-19T21:13:23.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e81195a0 con 0x7f10e81031b0 2024-02-19T21:13:23.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775943080 0 0) 0x7f10e81183a0 con 0x7f10e80fc870 2024-02-19T21:13:23.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e81171c0 con 0x7f10e80fc870 2024-02-19T21:13:23.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1159889287 0 0) 0x7f10d40038e0 con 0x7f10e80ff670 2024-02-19T21:13:23.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10e81183a0 con 0x7f10e80ff670 2024-02-19T21:13:23.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f10e4001940 con 0x7f10e80ff670 2024-02-19T21:13:23.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1779453155 0 0) 0x7f10e81195a0 con 0x7f10e81031b0 2024-02-19T21:13:23.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10d40038e0 con 0x7f10e81031b0 2024-02-19T21:13:23.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f10e0001e00 con 0x7f10e81031b0 2024-02-19T21:13:23.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 863080359 0 0) 0x7f10e81183a0 con 0x7f10e80ff670 2024-02-19T21:13:23.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 >> v1:172.21.15.193:6789/0 conn(0x7f10e80fc870 legacy=0x7f10e80f97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:23.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 >> v1:172.21.15.163:6789/0 conn(0x7f10e81031b0 legacy=0x7f10e8115880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:23.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e811a7a0 con 0x7f10e80ff670 2024-02-19T21:13:23.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10ef932700 1 -- 172.21.15.163:0/3787012232 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e81173b0 con 0x7f10e80ff670 2024-02-19T21:13:23.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10ef932700 10 client.50186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:23.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10ef932700 1 -- 172.21.15.163:0/3787012232 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e8117820 con 0x7f10e80ff670 2024-02-19T21:13:23.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10ef932700 1 librados: init done 2024-02-19T21:13:23.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f10e4001c40 con 0x7f10e80ff670 2024-02-19T21:13:23.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.701+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f10e4005bc0 con 0x7f10e80ff670 2024-02-19T21:13:23.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.702+0000 7f10ef932700 1 -- 172.21.15.163:0/3787012232 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10cc004f20 con 0x7f10e80ff670 2024-02-19T21:13:23.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.703+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f10e4002270 con 0x7f10e80ff670 2024-02-19T21:13:23.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.703+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(567..567 src has 434..567) v4 ==== 8907+0+0 (unknown 98095751 0 0) 0x7f10e4079010 con 0x7f10e80ff670 2024-02-19T21:13:23.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.703+0000 7f10de7fc700 10 client.50186.objecter ms_dispatch 0x7f10e8001550 osd_map(567..567 src has 434..567) v4 2024-02-19T21:13:23.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.703+0000 7f10de7fc700 3 client.50186.objecter handle_osd_map got epochs [567,567] > 0 2024-02-19T21:13:23.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.703+0000 7f10de7fc700 3 client.50186.objecter handle_osd_map decoding full epoch 567 2024-02-19T21:13:23.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.703+0000 7f10de7fc700 20 client.50186.objecter dump_active .. 0 homeless 2024-02-19T21:13:23.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.703+0000 7f10de7fc700 10 client.50186.objecter ms_handle_connect 0x7f10d4063c10 2024-02-19T21:13:23.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.707+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f10e404c0d0 con 0x7f10e80ff670 2024-02-19T21:13:23.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:22.376051+0000 mon.a (mon.0) 2971 : cluster [DBG] osdmap e566: 8 total, 8 up, 6 in 2024-02-19T21:13:23.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:22.379551+0000 mon.b (mon.1) 722 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:23.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:22.380030+0000 mon.a (mon.0) 2972 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:23.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:22.376051+0000 mon.a (mon.0) 2971 : cluster [DBG] osdmap e566: 8 total, 8 up, 6 in 2024-02-19T21:13:23.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:22.379551+0000 mon.b (mon.1) 722 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:23.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:22.380030+0000 mon.a (mon.0) 2972 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:23.749 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:13:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:13:23] "GET /metrics HTTP/1.1" 200 188097 "" "Prometheus/2.33.4" 2024-02-19T21:13:23.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.876+0000 7f10ef932700 1 -- 172.21.15.163:0/3787012232 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f10cc000b90 con 0x7f10d4063c10 2024-02-19T21:13:23.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.892+0000 7f10de7fc700 1 -- 172.21.15.163:0/3787012232 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+606043 (unknown 2965378022 0 3723420350) 0x7f10cc000b90 con 0x7f10d4063c10 2024-02-19T21:13:23.900 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:13:23.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.895+0000 7f10ef932700 10 librados: watch_flush enter 2024-02-19T21:13:23.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.895+0000 7f10ef932700 10 librados: watch_flush exit 2024-02-19T21:13:23.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.895+0000 7f10ef932700 20 client.50186.objecter shutdown clearing up homeless session... 2024-02-19T21:13:23.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.895+0000 7f10ef932700 10 client.50186.objecter successfully canceled tick 2024-02-19T21:13:23.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.896+0000 7f10ef932700 1 -- 172.21.15.163:0/3787012232 >> v1:172.21.15.163:6800/1552514681 conn(0x7f10d4063c10 legacy=0x7f10d40660d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:23.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.896+0000 7f10ef932700 1 -- 172.21.15.163:0/3787012232 >> v1:172.21.15.163:6790/0 conn(0x7f10e80ff670 legacy=0x7f10e80f9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:23.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.896+0000 7f10ef932700 1 -- 172.21.15.163:0/3787012232 shutdown_connections 2024-02-19T21:13:23.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.896+0000 7f10ef932700 1 -- 172.21.15.163:0/3787012232 >> 172.21.15.163:0/3787012232 conn(0x7f10e80f2470 msgr2=0x7f10e8106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:23.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.896+0000 7f10ef932700 1 -- 172.21.15.163:0/3787012232 shutdown_connections 2024-02-19T21:13:23.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.896+0000 7f10ef932700 1 -- 172.21.15.163:0/3787012232 wait complete. 2024-02-19T21:13:23.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:23.896+0000 7f10ef932700 1 librados: shutdown 2024-02-19T21:13:23.907 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:13:24.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:23.268428+0000 mgr.y (mgr.14550) 500 : cluster [DBG] pgmap v849: 246 pgs: 64 unknown, 182 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:13:24.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:23.379931+0000 mon.a (mon.0) 2973 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:13:24.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:23.380030+0000 mon.a (mon.0) 2974 : cluster [DBG] osdmap e567: 8 total, 8 up, 6 in 2024-02-19T21:13:24.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:23.398272+0000 mon.b (mon.1) 723 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:24.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:23.398655+0000 mon.a (mon.0) 2975 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:24.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:23.878290+0000 mgr.y (mgr.14550) 501 : audit [DBG] from='client.50186 v1:172.21.15.163:0/3787012232' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:13:24.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:24.105941+0000 mon.a (mon.0) 2976 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:24.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:24.334622+0000 mon.a (mon.0) 2977 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]': finished 2024-02-19T21:13:24.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:24.334704+0000 mon.a (mon.0) 2978 : cluster [DBG] osdmap e568: 8 total, 8 up, 6 in 2024-02-19T21:13:24.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:24.337727+0000 mon.b (mon.1) 724 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-97", "overlaypool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:24.717 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:24.344157+0000 mon.a (mon.0) 2979 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-97", "overlaypool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:24.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:23.268428+0000 mgr.y (mgr.14550) 500 : cluster [DBG] pgmap v849: 246 pgs: 64 unknown, 182 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:13:24.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:23.379931+0000 mon.a (mon.0) 2973 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:13:24.717 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:23.380030+0000 mon.a (mon.0) 2974 : cluster [DBG] osdmap e567: 8 total, 8 up, 6 in 2024-02-19T21:13:24.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:23.398272+0000 mon.b (mon.1) 723 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:24.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:23.398655+0000 mon.a (mon.0) 2975 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:24.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:23.878290+0000 mgr.y (mgr.14550) 501 : audit [DBG] from='client.50186 v1:172.21.15.163:0/3787012232' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:13:24.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:24.105941+0000 mon.a (mon.0) 2976 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:24.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:24.334622+0000 mon.a (mon.0) 2977 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]': finished 2024-02-19T21:13:24.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:24.334704+0000 mon.a (mon.0) 2978 : cluster [DBG] osdmap e568: 8 total, 8 up, 6 in 2024-02-19T21:13:24.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:24.337727+0000 mon.b (mon.1) 724 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-97", "overlaypool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:24.718 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:24.344157+0000 mon.a (mon.0) 2979 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-97", "overlaypool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:24.735 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":850,"stamp":"2024-02-19T21:13:23.372837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":194,"num_object_clones":0,"num_object_copies":582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":1840,"num_read_kb":1590,"num_write":721,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":11520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":823,"ondisk_log_size":823,"up":546,"acting":546,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":573,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":1858028,"kb_used_data":20332,"kb_used_omap":0,"kb_used_meta":1832448,"kb_avail":560612884,"statfs":{"total":575970213888,"available":574067593216,"internally_reserved":0,"allocated":20819968,"data_stored":10734314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1876426752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019789"},"pg_stats":[{"pgid":"231.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"231.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"229.7","version":"0'0","reported_seq":9,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.370447+0000","last_change":"2024-02-19T21:13:19.370447+0000","last_active":"2024-02-19T21:13:19.370447+0000","last_peered":"2024-02-19T21:13:19.370447+0000","last_clean":"2024-02-19T21:13:19.370447+0000","last_became_active":"2024-02-19T21:13:19.370104+0000","last_became_peered":"2024-02-19T21:13:19.370104+0000","last_unstale":"2024-02-19T21:13:19.370447+0000","last_undegraded":"2024-02-19T21:13:19.370447+0000","last_fullsized":"2024-02-19T21:13:19.370447+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"229.6","version":"0'0","reported_seq":9,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.375127+0000","last_change":"2024-02-19T21:13:19.375127+0000","last_active":"2024-02-19T21:13:19.375127+0000","last_peered":"2024-02-19T21:13:19.375127+0000","last_clean":"2024-02-19T21:13:19.375127+0000","last_became_active":"2024-02-19T21:13:19.374800+0000","last_became_peered":"2024-02-19T21:13:19.374800+0000","last_unstale":"2024-02-19T21:13:19.375127+0000","last_undegraded":"2024-02-19T21:13:19.375127+0000","last_fullsized":"2024-02-19T21:13:19.375127+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"231.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"229.5","version":"0'0","reported_seq":9,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.371925+0000","last_change":"2024-02-19T21:13:19.371925+0000","last_active":"2024-02-19T21:13:19.371925+0000","last_peered":"2024-02-19T21:13:19.371925+0000","last_clean":"2024-02-19T21:13:19.371925+0000","last_became_active":"2024-02-19T21:13:19.371633+0000","last_became_peered":"2024-02-19T21:13:19.371633+0000","last_unstale":"2024-02-19T21:13:19.371925+0000","last_undegraded":"2024-02-19T21:13:19.371925+0000","last_fullsized":"2024-02-19T21:13:19.371925+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"231.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"229.4","version":"0'0","reported_seq":9,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.376108+0000","last_change":"2024-02-19T21:13:19.376108+0000","last_active":"2024-02-19T21:13:19.376108+0000","last_peered":"2024-02-19T21:13:19.376108+0000","last_clean":"2024-02-19T21:13:19.376108+0000","last_became_active":"2024-02-19T21:13:19.375814+0000","last_became_peered":"2024-02-19T21:13:19.375814+0000","last_unstale":"2024-02-19T21:13:19.376108+0000","last_undegraded":"2024-02-19T21:13:19.376108+0000","last_fullsized":"2024-02-19T21:13:19.376108+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"231.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"229.3","version":"0'0","reported_seq":9,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.373500+0000","last_change":"2024-02-19T21:13:19.373500+0000","last_active":"2024-02-19T21:13:19.373500+0000","last_peered":"2024-02-19T21:13:19.373500+0000","last_clean":"2024-02-19T21:13:19.373500+0000","last_became_active":"2024-02-19T21:13:19.373259+0000","last_became_peered":"2024-02-19T21:13:19.373259+0000","last_unstale":"2024-02-19T21:13:19.373500+0000","last_undegraded":"2024-02-19T21:13:19.373500+0000","last_fullsized":"2024-02-19T21:13:19.373500+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"231.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"229.2","version":"0'0","reported_seq":9,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.371648+0000","last_change":"2024-02-19T21:13:19.371648+0000","last_active":"2024-02-19T21:13:19.371648+0000","last_peered":"2024-02-19T21:13:19.371648+0000","last_clean":"2024-02-19T21:13:19.371648+0000","last_became_active":"2024-02-19T21:13:19.371295+0000","last_became_peered":"2024-02-19T21:13:19.371295+0000","last_unstale":"2024-02-19T21:13:19.371648+0000","last_undegraded":"2024-02-19T21:13:19.371648+0000","last_fullsized":"2024-02-19T21:13:19.371648+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"231.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"229.1","version":"563'4","reported_seq":15,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.380594+0000","last_change":"2024-02-19T21:13:19.371167+0000","last_active":"2024-02-19T21:13:19.380594+0000","last_peered":"2024-02-19T21:13:19.380594+0000","last_clean":"2024-02-19T21:13:19.380594+0000","last_became_active":"2024-02-19T21:13:19.370517+0000","last_became_peered":"2024-02-19T21:13:19.370517+0000","last_unstale":"2024-02-19T21:13:19.380594+0000","last_undegraded":"2024-02-19T21:13:19.380594+0000","last_fullsized":"2024-02-19T21:13:19.380594+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"231.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:22.364569+0000","last_change":"2024-02-19T21:13:22.364569+0000","last_active":"2024-02-19T21:13:22.364569+0000","last_peered":"2024-02-19T21:13:22.364569+0000","last_clean":"2024-02-19T21:13:22.364569+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:22.364569+0000","last_undegraded":"2024-02-19T21:13:22.364569+0000","last_fullsized":"2024-02-19T21:13:22.364569+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:22.364569+0000","last_clean_scrub_stamp":"2024-02-19T21:13:22.364569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"230.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:13:20.352762+0000","last_change":"2024-02-19T21:13:20.352762+0000","last_active":"2024-02-19T21:13:20.352762+0000","last_peered":"2024-02-19T21:13:20.352762+0000","last_clean":"2024-02-19T21:13:20.352762+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:13:20.352762+0000","last_undegraded":"2024-02-19T21:13:20.352762+0000","last_fullsized":"2024-02-19T21:13:20.352762+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:20.352762+0000","last_clean_scrub_stamp":"2024-02-19T21:13:20.352762+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"229.0","version":"0'0","reported_seq":9,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.374080+0000","last_change":"2024-02-19T21:13:19.374080+0000","last_active":"2024-02-19T21:13:19.374080+0000","last_peered":"2024-02-19T21:13:19.374080+0000","last_clean":"2024-02-19T21:13:19.374080+0000","last_became_active":"2024-02-19T21:13:19.373744+0000","last_became_peered":"2024-02-19T21:13:19.373744+0000","last_unstale":"2024-02-19T21:13:19.374080+0000","last_undegraded":"2024-02-19T21:13:19.374080+0000","last_fullsized":"2024-02-19T21:13:19.374080+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":566,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351752+0000","last_change":"2024-02-19T21:12:09.321398+0000","last_active":"2024-02-19T21:13:18.351752+0000","last_peered":"2024-02-19T21:13:18.351752+0000","last_clean":"2024-02-19T21:13:18.351752+0000","last_became_active":"2024-02-19T21:12:09.320924+0000","last_became_peered":"2024-02-19T21:12:09.320924+0000","last_unstale":"2024-02-19T21:13:18.351752+0000","last_undegraded":"2024-02-19T21:13:18.351752+0000","last_fullsized":"2024-02-19T21:13:18.351752+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.738600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.738600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":96,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.353144+0000","last_change":"2024-02-19T21:11:50.304127+0000","last_active":"2024-02-19T21:13:18.353144+0000","last_peered":"2024-02-19T21:13:18.353144+0000","last_clean":"2024-02-19T21:13:18.353144+0000","last_became_active":"2024-02-19T21:11:50.303790+0000","last_became_peered":"2024-02-19T21:11:50.303790+0000","last_unstale":"2024-02-19T21:13:18.353144+0000","last_undegraded":"2024-02-19T21:13:18.353144+0000","last_fullsized":"2024-02-19T21:13:18.353144+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:42.529489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:42.529489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.25","version":"0'0","reported_seq":599,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914725+0000","last_change":"2024-02-19T21:12:09.349611+0000","last_active":"2024-02-19T21:13:18.914725+0000","last_peered":"2024-02-19T21:13:18.914725+0000","last_clean":"2024-02-19T21:13:18.914725+0000","last_became_active":"2024-02-19T21:12:09.349271+0000","last_became_peered":"2024-02-19T21:12:09.349271+0000","last_unstale":"2024-02-19T21:13:18.914725+0000","last_undegraded":"2024-02-19T21:13:18.914725+0000","last_fullsized":"2024-02-19T21:13:18.914725+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:40.580344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:40.580344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":573,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351779+0000","last_change":"2024-02-19T21:12:09.339659+0000","last_active":"2024-02-19T21:13:18.351779+0000","last_peered":"2024-02-19T21:13:18.351779+0000","last_clean":"2024-02-19T21:13:18.351779+0000","last_became_active":"2024-02-19T21:12:09.339310+0000","last_became_peered":"2024-02-19T21:12:09.339310+0000","last_unstale":"2024-02-19T21:13:18.351779+0000","last_undegraded":"2024-02-19T21:13:18.351779+0000","last_fullsized":"2024-02-19T21:13:18.351779+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:21.732778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:10:21.732778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":565,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349668+0000","last_change":"2024-02-19T21:11:50.313926+0000","last_active":"2024-02-19T21:13:18.349668+0000","last_peered":"2024-02-19T21:13:18.349668+0000","last_clean":"2024-02-19T21:13:18.349668+0000","last_became_active":"2024-02-19T21:11:50.313560+0000","last_became_peered":"2024-02-19T21:11:50.313560+0000","last_unstale":"2024-02-19T21:13:18.349668+0000","last_undegraded":"2024-02-19T21:13:18.349668+0000","last_fullsized":"2024-02-19T21:13:18.349668+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.364732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.364732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.26","version":"0'0","reported_seq":102,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349483+0000","last_change":"2024-02-19T21:12:09.339184+0000","last_active":"2024-02-19T21:13:18.349483+0000","last_peered":"2024-02-19T21:13:18.349483+0000","last_clean":"2024-02-19T21:13:18.349483+0000","last_became_active":"2024-02-19T21:12:09.338737+0000","last_became_peered":"2024-02-19T21:12:09.338737+0000","last_unstale":"2024-02-19T21:13:18.349483+0000","last_undegraded":"2024-02-19T21:13:18.349483+0000","last_fullsized":"2024-02-19T21:13:18.349483+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":606,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.354236+0000","last_change":"2024-02-19T21:12:09.343445+0000","last_active":"2024-02-19T21:13:18.354236+0000","last_peered":"2024-02-19T21:13:18.354236+0000","last_clean":"2024-02-19T21:13:18.354236+0000","last_became_active":"2024-02-19T21:12:09.342583+0000","last_became_peered":"2024-02-19T21:12:09.342583+0000","last_unstale":"2024-02-19T21:13:18.354236+0000","last_undegraded":"2024-02-19T21:13:18.354236+0000","last_fullsized":"2024-02-19T21:13:18.354236+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.955872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.955872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.20","version":"397'2","reported_seq":627,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.348452+0000","last_change":"2024-02-19T21:12:43.380032+0000","last_active":"2024-02-19T21:13:18.348452+0000","last_peered":"2024-02-19T21:13:18.348452+0000","last_clean":"2024-02-19T21:13:18.348452+0000","last_became_active":"2024-02-19T21:12:43.379693+0000","last_became_peered":"2024-02-19T21:12:43.379693+0000","last_unstale":"2024-02-19T21:13:18.348452+0000","last_undegraded":"2024-02-19T21:13:18.348452+0000","last_fullsized":"2024-02-19T21:13:18.348452+0000","mapping_epoch":526,"log_start":"58'1","ondisk_log_start":"58'1","created":474,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":6,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.934474+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":437,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":105,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.350276+0000","last_change":"2024-02-19T21:12:09.337725+0000","last_active":"2024-02-19T21:13:18.350276+0000","last_peered":"2024-02-19T21:13:18.350276+0000","last_clean":"2024-02-19T21:13:18.350276+0000","last_became_active":"2024-02-19T21:12:09.337337+0000","last_became_peered":"2024-02-19T21:12:09.337337+0000","last_unstale":"2024-02-19T21:13:18.350276+0000","last_undegraded":"2024-02-19T21:13:18.350276+0000","last_fullsized":"2024-02-19T21:13:18.350276+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.081993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.081993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":554,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914056+0000","last_change":"2024-02-19T21:12:09.353297+0000","last_active":"2024-02-19T21:13:18.914056+0000","last_peered":"2024-02-19T21:13:18.914056+0000","last_clean":"2024-02-19T21:13:18.914056+0000","last_became_active":"2024-02-19T21:12:09.352904+0000","last_became_peered":"2024-02-19T21:12:09.352904+0000","last_unstale":"2024-02-19T21:13:18.914056+0000","last_undegraded":"2024-02-19T21:13:18.914056+0000","last_fullsized":"2024-02-19T21:13:18.914056+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.089715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.089715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"493'12","reported_seq":722,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.362015+0000","last_change":"2024-02-19T21:12:18.452113+0000","last_active":"2024-02-19T21:13:19.362015+0000","last_peered":"2024-02-19T21:13:19.362015+0000","last_clean":"2024-02-19T21:13:19.362015+0000","last_became_active":"2024-02-19T21:12:09.332507+0000","last_became_peered":"2024-02-19T21:12:09.332507+0000","last_unstale":"2024-02-19T21:13:19.362015+0000","last_undegraded":"2024-02-19T21:13:19.362015+0000","last_fullsized":"2024-02-19T21:13:19.362015+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'12","last_scrub_stamp":"2024-02-19T21:12:18.452020+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:12:18.452020+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'28","reported_seq":599,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357012+0000","last_change":"2024-02-19T21:12:10.211768+0000","last_active":"2024-02-19T21:13:18.357012+0000","last_peered":"2024-02-19T21:13:18.357012+0000","last_clean":"2024-02-19T21:13:18.357012+0000","last_became_active":"2024-02-19T21:12:09.341122+0000","last_became_peered":"2024-02-19T21:12:09.341122+0000","last_unstale":"2024-02-19T21:13:18.357012+0000","last_undegraded":"2024-02-19T21:13:18.357012+0000","last_fullsized":"2024-02-19T21:13:18.357012+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-19T21:11:47.051823+0000","last_deep_scrub":"473'28","last_deep_scrub_stamp":"2024-02-19T21:11:47.051823+0000","last_clean_scrub_stamp":"2024-02-19T21:11:47.051823+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":562,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351611+0000","last_change":"2024-02-19T21:12:22.215650+0000","last_active":"2024-02-19T21:13:18.351611+0000","last_peered":"2024-02-19T21:13:18.351611+0000","last_clean":"2024-02-19T21:13:18.351611+0000","last_became_active":"2024-02-19T21:10:40.299425+0000","last_became_peered":"2024-02-19T21:10:40.299425+0000","last_unstale":"2024-02-19T21:13:18.351611+0000","last_undegraded":"2024-02-19T21:13:18.351611+0000","last_fullsized":"2024-02-19T21:13:18.351611+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:22.215548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_clean_scrub_stamp":"2024-02-19T21:12:22.215548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":541,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357794+0000","last_change":"2024-02-19T21:12:51.469049+0000","last_active":"2024-02-19T21:13:18.357794+0000","last_peered":"2024-02-19T21:13:18.357794+0000","last_clean":"2024-02-19T21:13:18.357794+0000","last_became_active":"2024-02-19T21:12:09.326173+0000","last_became_peered":"2024-02-19T21:12:09.326173+0000","last_unstale":"2024-02-19T21:13:18.357794+0000","last_undegraded":"2024-02-19T21:13:18.357794+0000","last_fullsized":"2024-02-19T21:13:18.357794+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:51.468971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.468971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":483,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.353802+0000","last_change":"2024-02-19T21:12:58.081198+0000","last_active":"2024-02-19T21:13:18.353802+0000","last_peered":"2024-02-19T21:13:18.353802+0000","last_clean":"2024-02-19T21:13:18.353802+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:13:18.353802+0000","last_undegraded":"2024-02-19T21:13:18.353802+0000","last_fullsized":"2024-02-19T21:13:18.353802+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:58.081106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:58.081106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'16","reported_seq":453,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.354867+0000","last_change":"2024-02-19T21:11:59.756726+0000","last_active":"2024-02-19T21:13:18.354867+0000","last_peered":"2024-02-19T21:13:18.354867+0000","last_clean":"2024-02-19T21:13:18.354867+0000","last_became_active":"2024-02-19T21:11:59.319397+0000","last_became_peered":"2024-02-19T21:11:59.319397+0000","last_unstale":"2024-02-19T21:13:18.354867+0000","last_undegraded":"2024-02-19T21:13:18.354867+0000","last_fullsized":"2024-02-19T21:13:18.354867+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'16","last_scrub_stamp":"2024-02-19T21:11:46.956424+0000","last_deep_scrub":"473'16","last_deep_scrub_stamp":"2024-02-19T21:11:46.956424+0000","last_clean_scrub_stamp":"2024-02-19T21:11:46.956424+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":568,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.353068+0000","last_change":"2024-02-19T21:12:06.016884+0000","last_active":"2024-02-19T21:13:18.353068+0000","last_peered":"2024-02-19T21:13:18.353068+0000","last_clean":"2024-02-19T21:13:18.353068+0000","last_became_active":"2024-02-19T21:10:41.312116+0000","last_became_peered":"2024-02-19T21:10:41.312116+0000","last_unstale":"2024-02-19T21:13:18.353068+0000","last_undegraded":"2024-02-19T21:13:18.353068+0000","last_fullsized":"2024-02-19T21:13:18.353068+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.016817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.016817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":567,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.353406+0000","last_change":"2024-02-19T21:13:05.193297+0000","last_active":"2024-02-19T21:13:18.353406+0000","last_peered":"2024-02-19T21:13:18.353406+0000","last_clean":"2024-02-19T21:13:18.353406+0000","last_became_active":"2024-02-19T21:10:41.305574+0000","last_became_peered":"2024-02-19T21:10:41.305574+0000","last_unstale":"2024-02-19T21:13:18.353406+0000","last_undegraded":"2024-02-19T21:13:18.353406+0000","last_fullsized":"2024-02-19T21:13:18.353406+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:05.193223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:05.193223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":368,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.348957+0000","last_change":"2024-02-19T21:11:59.323172+0000","last_active":"2024-02-19T21:13:18.348957+0000","last_peered":"2024-02-19T21:13:18.348957+0000","last_clean":"2024-02-19T21:13:18.348957+0000","last_became_active":"2024-02-19T21:11:59.322814+0000","last_became_peered":"2024-02-19T21:11:59.322814+0000","last_unstale":"2024-02-19T21:13:18.348957+0000","last_undegraded":"2024-02-19T21:13:18.348957+0000","last_fullsized":"2024-02-19T21:13:18.348957+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:34.504478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:34.504478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":477,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.358351+0000","last_change":"2024-02-19T21:10:57.130256+0000","last_active":"2024-02-19T21:13:18.358351+0000","last_peered":"2024-02-19T21:13:18.358351+0000","last_clean":"2024-02-19T21:13:18.358351+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:13:18.358351+0000","last_undegraded":"2024-02-19T21:13:18.358351+0000","last_fullsized":"2024-02-19T21:13:18.358351+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.130171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.130171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"459'35","reported_seq":581,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351283+0000","last_change":"2024-02-19T21:12:10.552781+0000","last_active":"2024-02-19T21:13:18.351283+0000","last_peered":"2024-02-19T21:13:18.351283+0000","last_clean":"2024-02-19T21:13:18.351283+0000","last_became_active":"2024-02-19T21:12:09.333541+0000","last_became_peered":"2024-02-19T21:12:09.333541+0000","last_unstale":"2024-02-19T21:13:18.351283+0000","last_undegraded":"2024-02-19T21:13:18.351283+0000","last_fullsized":"2024-02-19T21:13:18.351283+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:11:04.345836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:04.345836+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":544,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352931+0000","last_change":"2024-02-19T21:12:09.339263+0000","last_active":"2024-02-19T21:13:18.352931+0000","last_peered":"2024-02-19T21:13:18.352931+0000","last_clean":"2024-02-19T21:13:18.352931+0000","last_became_active":"2024-02-19T21:11:59.336426+0000","last_became_peered":"2024-02-19T21:11:59.336426+0000","last_unstale":"2024-02-19T21:13:18.352931+0000","last_undegraded":"2024-02-19T21:13:18.352931+0000","last_fullsized":"2024-02-19T21:13:18.352931+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:09.339147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:12:09.339147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":537,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.354521+0000","last_change":"2024-02-19T21:12:16.244173+0000","last_active":"2024-02-19T21:13:18.354521+0000","last_peered":"2024-02-19T21:13:18.354521+0000","last_clean":"2024-02-19T21:13:18.354521+0000","last_became_active":"2024-02-19T21:11:59.340544+0000","last_became_peered":"2024-02-19T21:11:59.340544+0000","last_unstale":"2024-02-19T21:13:18.354521+0000","last_undegraded":"2024-02-19T21:13:18.354521+0000","last_fullsized":"2024-02-19T21:13:18.354521+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:16.244104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:16.244104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":539,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.348494+0000","last_change":"2024-02-19T21:11:33.469310+0000","last_active":"2024-02-19T21:13:18.348494+0000","last_peered":"2024-02-19T21:13:18.348494+0000","last_clean":"2024-02-19T21:13:18.348494+0000","last_became_active":"2024-02-19T21:08:04.265081+0000","last_became_peered":"2024-02-19T21:08:04.265081+0000","last_unstale":"2024-02-19T21:13:18.348494+0000","last_undegraded":"2024-02-19T21:13:18.348494+0000","last_fullsized":"2024-02-19T21:13:18.348494+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:33.469232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_clean_scrub_stamp":"2024-02-19T21:11:33.469232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":499,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349521+0000","last_change":"2024-02-19T21:12:55.491781+0000","last_active":"2024-02-19T21:13:18.349521+0000","last_peered":"2024-02-19T21:13:18.349521+0000","last_clean":"2024-02-19T21:13:18.349521+0000","last_became_active":"2024-02-19T21:10:41.317284+0000","last_became_peered":"2024-02-19T21:10:41.317284+0000","last_unstale":"2024-02-19T21:13:18.349521+0000","last_undegraded":"2024-02-19T21:13:18.349521+0000","last_fullsized":"2024-02-19T21:13:18.349521+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:55.491665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.491665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"459'31","reported_seq":182,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.915094+0000","last_change":"2024-02-19T21:11:44.421114+0000","last_active":"2024-02-19T21:13:18.915094+0000","last_peered":"2024-02-19T21:13:18.915094+0000","last_clean":"2024-02-19T21:13:18.915094+0000","last_became_active":"2024-02-19T21:10:41.313853+0000","last_became_peered":"2024-02-19T21:10:41.313853+0000","last_unstale":"2024-02-19T21:13:18.915094+0000","last_undegraded":"2024-02-19T21:13:18.915094+0000","last_fullsized":"2024-02-19T21:13:18.915094+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'31","last_scrub_stamp":"2024-02-19T21:11:44.421033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:44.421033+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":564,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349200+0000","last_change":"2024-02-19T21:13:07.450202+0000","last_active":"2024-02-19T21:13:18.349200+0000","last_peered":"2024-02-19T21:13:18.349200+0000","last_clean":"2024-02-19T21:13:18.349200+0000","last_became_active":"2024-02-19T21:10:41.307517+0000","last_became_peered":"2024-02-19T21:10:41.307517+0000","last_unstale":"2024-02-19T21:13:18.349200+0000","last_undegraded":"2024-02-19T21:13:18.349200+0000","last_fullsized":"2024-02-19T21:13:18.349200+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:13:07.450123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.450123+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":564,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352322+0000","last_change":"2024-02-19T21:12:58.838081+0000","last_active":"2024-02-19T21:13:18.352322+0000","last_peered":"2024-02-19T21:13:18.352322+0000","last_clean":"2024-02-19T21:13:18.352322+0000","last_became_active":"2024-02-19T21:12:09.369409+0000","last_became_peered":"2024-02-19T21:12:09.369409+0000","last_unstale":"2024-02-19T21:13:18.352322+0000","last_undegraded":"2024-02-19T21:13:18.352322+0000","last_fullsized":"2024-02-19T21:13:18.352322+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:58.837973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:58.837973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":565,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352473+0000","last_change":"2024-02-19T21:13:03.126485+0000","last_active":"2024-02-19T21:13:18.352473+0000","last_peered":"2024-02-19T21:13:18.352473+0000","last_clean":"2024-02-19T21:13:18.352473+0000","last_became_active":"2024-02-19T21:10:41.307577+0000","last_became_peered":"2024-02-19T21:10:41.307577+0000","last_unstale":"2024-02-19T21:13:18.352473+0000","last_undegraded":"2024-02-19T21:13:18.352473+0000","last_fullsized":"2024-02-19T21:13:18.352473+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:03.126414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:13:03.126414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":143,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914294+0000","last_change":"2024-02-19T21:10:50.424462+0000","last_active":"2024-02-19T21:13:18.914294+0000","last_peered":"2024-02-19T21:13:18.914294+0000","last_clean":"2024-02-19T21:13:18.914294+0000","last_became_active":"2024-02-19T21:10:41.302670+0000","last_became_peered":"2024-02-19T21:10:41.302670+0000","last_unstale":"2024-02-19T21:13:18.914294+0000","last_undegraded":"2024-02-19T21:13:18.914294+0000","last_fullsized":"2024-02-19T21:13:18.914294+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:50.424381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.424381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"459'25","reported_seq":191,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914720+0000","last_change":"2024-02-19T21:12:51.420519+0000","last_active":"2024-02-19T21:13:18.914720+0000","last_peered":"2024-02-19T21:13:18.914720+0000","last_clean":"2024-02-19T21:13:18.914720+0000","last_became_active":"2024-02-19T21:12:09.408620+0000","last_became_peered":"2024-02-19T21:12:09.408620+0000","last_unstale":"2024-02-19T21:13:18.914720+0000","last_undegraded":"2024-02-19T21:13:18.914720+0000","last_fullsized":"2024-02-19T21:13:18.914720+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'25","last_scrub_stamp":"2024-02-19T21:12:51.420403+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.420403+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":548,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351357+0000","last_change":"2024-02-19T21:13:13.044378+0000","last_active":"2024-02-19T21:13:18.351357+0000","last_peered":"2024-02-19T21:13:18.351357+0000","last_clean":"2024-02-19T21:13:18.351357+0000","last_became_active":"2024-02-19T21:08:04.268406+0000","last_became_peered":"2024-02-19T21:08:04.268406+0000","last_unstale":"2024-02-19T21:13:18.351357+0000","last_undegraded":"2024-02-19T21:13:18.351357+0000","last_fullsized":"2024-02-19T21:13:18.351357+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:12.628801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:12.628801+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":684,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.358560+0000","last_change":"2024-02-19T21:12:21.202447+0000","last_active":"2024-02-19T21:13:19.358560+0000","last_peered":"2024-02-19T21:13:19.358560+0000","last_clean":"2024-02-19T21:13:19.358560+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:13:19.358560+0000","last_undegraded":"2024-02-19T21:13:19.358560+0000","last_fullsized":"2024-02-19T21:13:19.358560+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:12:21.202371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:21.202371+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":143,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.915080+0000","last_change":"2024-02-19T21:12:21.380805+0000","last_active":"2024-02-19T21:13:18.915080+0000","last_peered":"2024-02-19T21:13:18.915080+0000","last_clean":"2024-02-19T21:13:18.915080+0000","last_became_active":"2024-02-19T21:10:41.298893+0000","last_became_peered":"2024-02-19T21:10:41.298893+0000","last_unstale":"2024-02-19T21:13:18.915080+0000","last_undegraded":"2024-02-19T21:13:18.915080+0000","last_fullsized":"2024-02-19T21:13:18.915080+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:21.380734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:12:21.380734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":290,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.353379+0000","last_change":"2024-02-19T21:12:16.339644+0000","last_active":"2024-02-19T21:13:18.353379+0000","last_peered":"2024-02-19T21:13:18.353379+0000","last_clean":"2024-02-19T21:13:18.353379+0000","last_became_active":"2024-02-19T21:11:59.318845+0000","last_became_peered":"2024-02-19T21:11:59.318845+0000","last_unstale":"2024-02-19T21:13:18.353379+0000","last_undegraded":"2024-02-19T21:13:18.353379+0000","last_fullsized":"2024-02-19T21:13:18.353379+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:16.339577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:16.339577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"459'27","reported_seq":187,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352232+0000","last_change":"2024-02-19T21:11:08.202280+0000","last_active":"2024-02-19T21:13:18.352232+0000","last_peered":"2024-02-19T21:13:18.352232+0000","last_clean":"2024-02-19T21:13:18.352232+0000","last_became_active":"2024-02-19T21:10:41.307576+0000","last_became_peered":"2024-02-19T21:10:41.307576+0000","last_unstale":"2024-02-19T21:13:18.352232+0000","last_undegraded":"2024-02-19T21:13:18.352232+0000","last_fullsized":"2024-02-19T21:13:18.352232+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:11:08.202208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:08.202208+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":590,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.353915+0000","last_change":"2024-02-19T21:12:40.370031+0000","last_active":"2024-02-19T21:13:18.353915+0000","last_peered":"2024-02-19T21:13:18.353915+0000","last_clean":"2024-02-19T21:13:18.353915+0000","last_became_active":"2024-02-19T21:12:40.369640+0000","last_became_peered":"2024-02-19T21:12:40.369640+0000","last_unstale":"2024-02-19T21:13:18.353915+0000","last_undegraded":"2024-02-19T21:13:18.353915+0000","last_fullsized":"2024-02-19T21:13:18.353915+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.738600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.738600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"483'11","reported_seq":193,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.355541+0000","last_change":"2024-02-19T21:12:56.450133+0000","last_active":"2024-02-19T21:13:19.355541+0000","last_peered":"2024-02-19T21:13:19.355541+0000","last_clean":"2024-02-19T21:13:19.355541+0000","last_became_active":"2024-02-19T21:11:59.336645+0000","last_became_peered":"2024-02-19T21:11:59.336645+0000","last_unstale":"2024-02-19T21:13:19.355541+0000","last_undegraded":"2024-02-19T21:13:19.355541+0000","last_fullsized":"2024-02-19T21:13:19.355541+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"483'11","last_scrub_stamp":"2024-02-19T21:12:56.450044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:56.450044+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":537,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.358336+0000","last_change":"2024-02-19T21:12:54.437089+0000","last_active":"2024-02-19T21:13:18.358336+0000","last_peered":"2024-02-19T21:13:18.358336+0000","last_clean":"2024-02-19T21:13:18.358336+0000","last_became_active":"2024-02-19T21:11:59.344440+0000","last_became_peered":"2024-02-19T21:11:59.344440+0000","last_unstale":"2024-02-19T21:13:18.358336+0000","last_undegraded":"2024-02-19T21:13:18.358336+0000","last_fullsized":"2024-02-19T21:13:18.358336+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:54.436977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:54.436977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":486,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357538+0000","last_change":"2024-02-19T21:12:09.327574+0000","last_active":"2024-02-19T21:13:18.357538+0000","last_peered":"2024-02-19T21:13:18.357538+0000","last_clean":"2024-02-19T21:13:18.357538+0000","last_became_active":"2024-02-19T21:12:09.326909+0000","last_became_peered":"2024-02-19T21:12:09.326909+0000","last_unstale":"2024-02-19T21:13:18.357538+0000","last_undegraded":"2024-02-19T21:13:18.357538+0000","last_fullsized":"2024-02-19T21:13:18.357538+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.079353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.079353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"459'19","reported_seq":190,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.915062+0000","last_change":"2024-02-19T21:12:10.647686+0000","last_active":"2024-02-19T21:13:18.915062+0000","last_peered":"2024-02-19T21:13:18.915062+0000","last_clean":"2024-02-19T21:13:18.915062+0000","last_became_active":"2024-02-19T21:12:09.409065+0000","last_became_peered":"2024-02-19T21:12:09.409065+0000","last_unstale":"2024-02-19T21:13:18.915062+0000","last_undegraded":"2024-02-19T21:13:18.915062+0000","last_fullsized":"2024-02-19T21:13:18.915062+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:11:42.333781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:42.333781+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":577,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.348886+0000","last_change":"2024-02-19T21:11:59.332430+0000","last_active":"2024-02-19T21:13:18.348886+0000","last_peered":"2024-02-19T21:13:18.348886+0000","last_clean":"2024-02-19T21:13:18.348886+0000","last_became_active":"2024-02-19T21:11:59.331675+0000","last_became_peered":"2024-02-19T21:11:59.331675+0000","last_unstale":"2024-02-19T21:13:18.348886+0000","last_undegraded":"2024-02-19T21:13:18.348886+0000","last_fullsized":"2024-02-19T21:13:18.348886+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:42.529489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:42.529489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":551,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.358513+0000","last_change":"2024-02-19T21:12:09.350127+0000","last_active":"2024-02-19T21:13:18.358513+0000","last_peered":"2024-02-19T21:13:18.358513+0000","last_clean":"2024-02-19T21:13:18.358513+0000","last_became_active":"2024-02-19T21:12:09.349667+0000","last_became_peered":"2024-02-19T21:12:09.349667+0000","last_unstale":"2024-02-19T21:13:18.358513+0000","last_undegraded":"2024-02-19T21:13:18.358513+0000","last_fullsized":"2024-02-19T21:13:18.358513+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:58.096025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.096025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":283,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351364+0000","last_change":"2024-02-19T21:12:09.335125+0000","last_active":"2024-02-19T21:13:18.351364+0000","last_peered":"2024-02-19T21:13:18.351364+0000","last_clean":"2024-02-19T21:13:18.351364+0000","last_became_active":"2024-02-19T21:12:09.333625+0000","last_became_peered":"2024-02-19T21:12:09.333625+0000","last_unstale":"2024-02-19T21:13:18.351364+0000","last_undegraded":"2024-02-19T21:13:18.351364+0000","last_fullsized":"2024-02-19T21:13:18.351364+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:06.547494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:06.547494+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":283,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352530+0000","last_change":"2024-02-19T21:12:09.348064+0000","last_active":"2024-02-19T21:13:18.352530+0000","last_peered":"2024-02-19T21:13:18.352530+0000","last_clean":"2024-02-19T21:13:18.352530+0000","last_became_active":"2024-02-19T21:12:09.347240+0000","last_became_peered":"2024-02-19T21:12:09.347240+0000","last_unstale":"2024-02-19T21:13:18.352530+0000","last_undegraded":"2024-02-19T21:13:18.352530+0000","last_fullsized":"2024-02-19T21:13:18.352530+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:43.407517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.407517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"434'12","reported_seq":371,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.363925+0000","last_change":"2024-02-19T21:10:41.321881+0000","last_active":"2024-02-19T21:13:19.363925+0000","last_peered":"2024-02-19T21:13:19.363925+0000","last_clean":"2024-02-19T21:13:19.363925+0000","last_became_active":"2024-02-19T21:10:41.297897+0000","last_became_peered":"2024-02-19T21:10:41.297897+0000","last_unstale":"2024-02-19T21:13:19.363925+0000","last_undegraded":"2024-02-19T21:13:19.363925+0000","last_fullsized":"2024-02-19T21:13:19.363925+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'11","last_scrub_stamp":"2024-02-19T21:09:39.597001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:39.597001+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"459'28","reported_seq":205,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914277+0000","last_change":"2024-02-19T21:12:10.323234+0000","last_active":"2024-02-19T21:13:18.914277+0000","last_peered":"2024-02-19T21:13:18.914277+0000","last_clean":"2024-02-19T21:13:18.914277+0000","last_became_active":"2024-02-19T21:12:09.369385+0000","last_became_peered":"2024-02-19T21:12:09.369385+0000","last_unstale":"2024-02-19T21:13:18.914277+0000","last_undegraded":"2024-02-19T21:13:18.914277+0000","last_fullsized":"2024-02-19T21:13:18.914277+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:10:49.466227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:49.466227+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":550,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349776+0000","last_change":"2024-02-19T21:12:09.343164+0000","last_active":"2024-02-19T21:13:18.349776+0000","last_peered":"2024-02-19T21:13:18.349776+0000","last_clean":"2024-02-19T21:13:18.349776+0000","last_became_active":"2024-02-19T21:12:09.342814+0000","last_became_peered":"2024-02-19T21:12:09.342814+0000","last_unstale":"2024-02-19T21:13:18.349776+0000","last_undegraded":"2024-02-19T21:13:18.349776+0000","last_fullsized":"2024-02-19T21:13:18.349776+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.364732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.364732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":534,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349434+0000","last_change":"2024-02-19T21:12:09.335134+0000","last_active":"2024-02-19T21:13:18.349434+0000","last_peered":"2024-02-19T21:13:18.349434+0000","last_clean":"2024-02-19T21:13:18.349434+0000","last_became_active":"2024-02-19T21:12:09.333970+0000","last_became_peered":"2024-02-19T21:12:09.333970+0000","last_unstale":"2024-02-19T21:13:18.349434+0000","last_undegraded":"2024-02-19T21:13:18.349434+0000","last_fullsized":"2024-02-19T21:13:18.349434+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.851123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.851123+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":367,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.353288+0000","last_change":"2024-02-19T21:13:11.004073+0000","last_active":"2024-02-19T21:13:18.353288+0000","last_peered":"2024-02-19T21:13:18.353288+0000","last_clean":"2024-02-19T21:13:18.353288+0000","last_became_active":"2024-02-19T21:11:59.323698+0000","last_became_peered":"2024-02-19T21:11:59.323698+0000","last_unstale":"2024-02-19T21:13:18.353288+0000","last_undegraded":"2024-02-19T21:13:18.353288+0000","last_fullsized":"2024-02-19T21:13:18.353288+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:11.004006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:11.004006+0000","last_clean_scrub_stamp":"2024-02-19T21:13:11.004006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":143,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914062+0000","last_change":"2024-02-19T21:10:59.312931+0000","last_active":"2024-02-19T21:13:18.914062+0000","last_peered":"2024-02-19T21:13:18.914062+0000","last_clean":"2024-02-19T21:13:18.914062+0000","last_became_active":"2024-02-19T21:10:41.304695+0000","last_became_peered":"2024-02-19T21:10:41.304695+0000","last_unstale":"2024-02-19T21:13:18.914062+0000","last_undegraded":"2024-02-19T21:13:18.914062+0000","last_fullsized":"2024-02-19T21:13:18.914062+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:59.312863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.312863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"459'30","reported_seq":337,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.353557+0000","last_change":"2024-02-19T21:11:59.380681+0000","last_active":"2024-02-19T21:13:18.353557+0000","last_peered":"2024-02-19T21:13:18.353557+0000","last_clean":"2024-02-19T21:13:18.353557+0000","last_became_active":"2024-02-19T21:11:59.327859+0000","last_became_peered":"2024-02-19T21:11:59.327859+0000","last_unstale":"2024-02-19T21:13:18.353557+0000","last_undegraded":"2024-02-19T21:13:18.353557+0000","last_fullsized":"2024-02-19T21:13:18.353557+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:11:02.671837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.671837+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":585,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.350523+0000","last_change":"2024-02-19T21:12:09.369780+0000","last_active":"2024-02-19T21:13:18.350523+0000","last_peered":"2024-02-19T21:13:18.350523+0000","last_clean":"2024-02-19T21:13:18.350523+0000","last_became_active":"2024-02-19T21:12:09.369477+0000","last_became_peered":"2024-02-19T21:12:09.369477+0000","last_unstale":"2024-02-19T21:13:18.350523+0000","last_undegraded":"2024-02-19T21:13:18.350523+0000","last_fullsized":"2024-02-19T21:13:18.350523+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":143,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914399+0000","last_change":"2024-02-19T21:12:06.410909+0000","last_active":"2024-02-19T21:13:18.914399+0000","last_peered":"2024-02-19T21:13:18.914399+0000","last_clean":"2024-02-19T21:13:18.914399+0000","last_became_active":"2024-02-19T21:10:41.295301+0000","last_became_peered":"2024-02-19T21:10:41.295301+0000","last_unstale":"2024-02-19T21:13:18.914399+0000","last_undegraded":"2024-02-19T21:13:18.914399+0000","last_fullsized":"2024-02-19T21:13:18.914399+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.410847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.410847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":292,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351597+0000","last_change":"2024-02-19T21:12:09.332689+0000","last_active":"2024-02-19T21:13:18.351597+0000","last_peered":"2024-02-19T21:13:18.351597+0000","last_clean":"2024-02-19T21:13:18.351597+0000","last_became_active":"2024-02-19T21:12:09.332349+0000","last_became_peered":"2024-02-19T21:12:09.332349+0000","last_unstale":"2024-02-19T21:13:18.351597+0000","last_undegraded":"2024-02-19T21:13:18.351597+0000","last_fullsized":"2024-02-19T21:13:18.351597+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:46.406402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:46.406402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":528,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357910+0000","last_change":"2024-02-19T21:12:09.340947+0000","last_active":"2024-02-19T21:13:18.357910+0000","last_peered":"2024-02-19T21:13:18.357910+0000","last_clean":"2024-02-19T21:13:18.357910+0000","last_became_active":"2024-02-19T21:12:09.340614+0000","last_became_peered":"2024-02-19T21:12:09.340614+0000","last_unstale":"2024-02-19T21:13:18.357910+0000","last_undegraded":"2024-02-19T21:13:18.357910+0000","last_fullsized":"2024-02-19T21:13:18.357910+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.669336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.669336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"459'29","reported_seq":445,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357790+0000","last_change":"2024-02-19T21:12:00.056470+0000","last_active":"2024-02-19T21:13:18.357790+0000","last_peered":"2024-02-19T21:13:18.357790+0000","last_clean":"2024-02-19T21:13:18.357790+0000","last_became_active":"2024-02-19T21:11:59.332542+0000","last_became_peered":"2024-02-19T21:11:59.332542+0000","last_unstale":"2024-02-19T21:13:18.357790+0000","last_undegraded":"2024-02-19T21:13:18.357790+0000","last_fullsized":"2024-02-19T21:13:18.357790+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:10:11.165973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:11.165973+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":597,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349494+0000","last_change":"2024-02-19T21:11:59.335901+0000","last_active":"2024-02-19T21:13:18.349494+0000","last_peered":"2024-02-19T21:13:18.349494+0000","last_clean":"2024-02-19T21:13:18.349494+0000","last_became_active":"2024-02-19T21:11:59.335475+0000","last_became_peered":"2024-02-19T21:11:59.335475+0000","last_unstale":"2024-02-19T21:13:18.349494+0000","last_undegraded":"2024-02-19T21:13:18.349494+0000","last_fullsized":"2024-02-19T21:13:18.349494+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:40.580344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:40.580344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":530,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.354063+0000","last_change":"2024-02-19T21:12:20.902387+0000","last_active":"2024-02-19T21:13:18.354063+0000","last_peered":"2024-02-19T21:13:18.354063+0000","last_clean":"2024-02-19T21:13:18.354063+0000","last_became_active":"2024-02-19T21:12:09.347197+0000","last_became_peered":"2024-02-19T21:12:09.347197+0000","last_unstale":"2024-02-19T21:13:18.354063+0000","last_undegraded":"2024-02-19T21:13:18.354063+0000","last_fullsized":"2024-02-19T21:13:18.354063+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:20.902297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:12:20.902297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":487,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352594+0000","last_change":"2024-02-19T21:11:59.359210+0000","last_active":"2024-02-19T21:13:18.352594+0000","last_peered":"2024-02-19T21:13:18.352594+0000","last_clean":"2024-02-19T21:13:18.352594+0000","last_became_active":"2024-02-19T21:11:59.358933+0000","last_became_peered":"2024-02-19T21:11:59.358933+0000","last_unstale":"2024-02-19T21:13:18.352594+0000","last_undegraded":"2024-02-19T21:13:18.352594+0000","last_fullsized":"2024-02-19T21:13:18.352594+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.189509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":533,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.358621+0000","last_change":"2024-02-19T21:11:59.334472+0000","last_active":"2024-02-19T21:13:18.358621+0000","last_peered":"2024-02-19T21:13:18.358621+0000","last_clean":"2024-02-19T21:13:18.358621+0000","last_became_active":"2024-02-19T21:11:59.333491+0000","last_became_peered":"2024-02-19T21:11:59.333491+0000","last_unstale":"2024-02-19T21:13:18.358621+0000","last_undegraded":"2024-02-19T21:13:18.358621+0000","last_fullsized":"2024-02-19T21:13:18.358621+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:30.893485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:30.893485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"459'24","reported_seq":429,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.350659+0000","last_change":"2024-02-19T21:12:50.030922+0000","last_active":"2024-02-19T21:13:18.350659+0000","last_peered":"2024-02-19T21:13:18.350659+0000","last_clean":"2024-02-19T21:13:18.350659+0000","last_became_active":"2024-02-19T21:11:59.348806+0000","last_became_peered":"2024-02-19T21:11:59.348806+0000","last_unstale":"2024-02-19T21:13:18.350659+0000","last_undegraded":"2024-02-19T21:13:18.350659+0000","last_fullsized":"2024-02-19T21:13:18.350659+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'24","last_scrub_stamp":"2024-02-19T21:12:50.030857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:50.030857+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"397'2","reported_seq":594,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.348634+0000","last_change":"2024-02-19T21:12:02.320023+0000","last_active":"2024-02-19T21:13:18.348634+0000","last_peered":"2024-02-19T21:13:18.348634+0000","last_clean":"2024-02-19T21:13:18.348634+0000","last_became_active":"2024-02-19T21:12:02.319577+0000","last_became_peered":"2024-02-19T21:12:02.319577+0000","last_unstale":"2024-02-19T21:13:18.348634+0000","last_undegraded":"2024-02-19T21:13:18.348634+0000","last_fullsized":"2024-02-19T21:13:18.348634+0000","mapping_epoch":485,"log_start":"58'1","ondisk_log_start":"58'1","created":57,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.934474+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":436,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":152,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914383+0000","last_change":"2024-02-19T21:11:59.320654+0000","last_active":"2024-02-19T21:13:18.914383+0000","last_peered":"2024-02-19T21:13:18.914383+0000","last_clean":"2024-02-19T21:13:18.914383+0000","last_became_active":"2024-02-19T21:11:59.320327+0000","last_became_peered":"2024-02-19T21:11:59.320327+0000","last_unstale":"2024-02-19T21:13:18.914383+0000","last_undegraded":"2024-02-19T21:13:18.914383+0000","last_fullsized":"2024-02-19T21:13:18.914383+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.455993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.455993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":539,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352402+0000","last_change":"2024-02-19T21:10:51.691357+0000","last_active":"2024-02-19T21:13:18.352402+0000","last_peered":"2024-02-19T21:13:18.352402+0000","last_clean":"2024-02-19T21:13:18.352402+0000","last_became_active":"2024-02-19T21:08:04.282243+0000","last_became_peered":"2024-02-19T21:08:04.282243+0000","last_unstale":"2024-02-19T21:13:18.352402+0000","last_undegraded":"2024-02-19T21:13:18.352402+0000","last_fullsized":"2024-02-19T21:13:18.352402+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.691284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.691284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":477,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357679+0000","last_change":"2024-02-19T21:12:09.328499+0000","last_active":"2024-02-19T21:13:18.357679+0000","last_peered":"2024-02-19T21:13:18.357679+0000","last_clean":"2024-02-19T21:13:18.357679+0000","last_became_active":"2024-02-19T21:12:09.326936+0000","last_became_peered":"2024-02-19T21:12:09.326936+0000","last_unstale":"2024-02-19T21:13:18.357679+0000","last_undegraded":"2024-02-19T21:13:18.357679+0000","last_fullsized":"2024-02-19T21:13:18.357679+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:05.664425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:05.664425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":528,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349385+0000","last_change":"2024-02-19T21:13:06.446599+0000","last_active":"2024-02-19T21:13:18.349385+0000","last_peered":"2024-02-19T21:13:18.349385+0000","last_clean":"2024-02-19T21:13:18.349385+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:13:18.349385+0000","last_undegraded":"2024-02-19T21:13:18.349385+0000","last_fullsized":"2024-02-19T21:13:18.349385+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:06.446523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.446523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"459'26","reported_seq":607,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.915073+0000","last_change":"2024-02-19T21:12:09.420794+0000","last_active":"2024-02-19T21:13:18.915073+0000","last_peered":"2024-02-19T21:13:18.915073+0000","last_clean":"2024-02-19T21:13:18.915073+0000","last_became_active":"2024-02-19T21:12:09.352705+0000","last_became_peered":"2024-02-19T21:12:09.352705+0000","last_unstale":"2024-02-19T21:13:18.915073+0000","last_undegraded":"2024-02-19T21:13:18.915073+0000","last_fullsized":"2024-02-19T21:13:18.915073+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'26","last_scrub_stamp":"2024-02-19T21:11:44.774856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:44.774856+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":554,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.348677+0000","last_change":"2024-02-19T21:12:09.311418+0000","last_active":"2024-02-19T21:13:18.348677+0000","last_peered":"2024-02-19T21:13:18.348677+0000","last_clean":"2024-02-19T21:13:18.348677+0000","last_became_active":"2024-02-19T21:12:09.310949+0000","last_became_peered":"2024-02-19T21:12:09.310949+0000","last_unstale":"2024-02-19T21:13:18.348677+0000","last_undegraded":"2024-02-19T21:13:18.348677+0000","last_fullsized":"2024-02-19T21:13:18.348677+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.747108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.747108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":565,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357659+0000","last_change":"2024-02-19T21:11:48.359614+0000","last_active":"2024-02-19T21:13:18.357659+0000","last_peered":"2024-02-19T21:13:18.357659+0000","last_clean":"2024-02-19T21:13:18.357659+0000","last_became_active":"2024-02-19T21:11:48.359271+0000","last_became_peered":"2024-02-19T21:11:48.359271+0000","last_unstale":"2024-02-19T21:13:18.357659+0000","last_undegraded":"2024-02-19T21:13:18.357659+0000","last_fullsized":"2024-02-19T21:13:18.357659+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.081993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.081993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":288,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352613+0000","last_change":"2024-02-19T21:12:55.248327+0000","last_active":"2024-02-19T21:13:18.352613+0000","last_peered":"2024-02-19T21:13:18.352613+0000","last_clean":"2024-02-19T21:13:18.352613+0000","last_became_active":"2024-02-19T21:12:09.347643+0000","last_became_peered":"2024-02-19T21:12:09.347643+0000","last_unstale":"2024-02-19T21:13:18.352613+0000","last_undegraded":"2024-02-19T21:13:18.352613+0000","last_fullsized":"2024-02-19T21:13:18.352613+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:55.248259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.248259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":558,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.358618+0000","last_change":"2024-02-19T21:10:41.315872+0000","last_active":"2024-02-19T21:13:18.358618+0000","last_peered":"2024-02-19T21:13:18.358618+0000","last_clean":"2024-02-19T21:13:18.358618+0000","last_became_active":"2024-02-19T21:10:41.315397+0000","last_became_peered":"2024-02-19T21:10:41.315397+0000","last_unstale":"2024-02-19T21:13:18.358618+0000","last_undegraded":"2024-02-19T21:13:18.358618+0000","last_fullsized":"2024-02-19T21:13:18.358618+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:10.148733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:10.148733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":486,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357293+0000","last_change":"2024-02-19T21:11:59.333800+0000","last_active":"2024-02-19T21:13:18.357293+0000","last_peered":"2024-02-19T21:13:18.357293+0000","last_clean":"2024-02-19T21:13:18.357293+0000","last_became_active":"2024-02-19T21:11:59.333572+0000","last_became_peered":"2024-02-19T21:11:59.333572+0000","last_unstale":"2024-02-19T21:13:18.357293+0000","last_undegraded":"2024-02-19T21:13:18.357293+0000","last_fullsized":"2024-02-19T21:13:18.357293+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_clean_scrub_stamp":"2024-02-19T21:10:20.204880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":531,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351749+0000","last_change":"2024-02-19T21:13:00.111238+0000","last_active":"2024-02-19T21:13:18.351749+0000","last_peered":"2024-02-19T21:13:18.351749+0000","last_clean":"2024-02-19T21:13:18.351749+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:13:18.351749+0000","last_undegraded":"2024-02-19T21:13:18.351749+0000","last_fullsized":"2024-02-19T21:13:18.351749+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.111171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.111171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"459'22","reported_seq":571,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352553+0000","last_change":"2024-02-19T21:11:59.988716+0000","last_active":"2024-02-19T21:13:18.352553+0000","last_peered":"2024-02-19T21:13:18.352553+0000","last_clean":"2024-02-19T21:13:18.352553+0000","last_became_active":"2024-02-19T21:11:59.343010+0000","last_became_peered":"2024-02-19T21:11:59.343010+0000","last_unstale":"2024-02-19T21:13:18.352553+0000","last_undegraded":"2024-02-19T21:13:18.352553+0000","last_fullsized":"2024-02-19T21:13:18.352553+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:11:00.191206+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.191206+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":589,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.354556+0000","last_change":"2024-02-19T21:11:59.352625+0000","last_active":"2024-02-19T21:13:18.354556+0000","last_peered":"2024-02-19T21:13:18.354556+0000","last_clean":"2024-02-19T21:13:18.354556+0000","last_became_active":"2024-02-19T21:11:59.352283+0000","last_became_peered":"2024-02-19T21:11:59.352283+0000","last_unstale":"2024-02-19T21:13:18.354556+0000","last_undegraded":"2024-02-19T21:13:18.354556+0000","last_fullsized":"2024-02-19T21:13:18.354556+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.955872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.955872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":1263,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.367095+0000","last_change":"2024-02-19T21:13:08.986433+0000","last_active":"2024-02-19T21:13:19.367095+0000","last_peered":"2024-02-19T21:13:19.367095+0000","last_clean":"2024-02-19T21:13:19.367095+0000","last_became_active":"2024-02-19T21:12:09.340570+0000","last_became_peered":"2024-02-19T21:12:09.340570+0000","last_unstale":"2024-02-19T21:13:19.367095+0000","last_undegraded":"2024-02-19T21:13:19.367095+0000","last_fullsized":"2024-02-19T21:13:19.367095+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:13:08.986361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:08.986361+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":714,"num_read_kb":709,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":553,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349340+0000","last_change":"2024-02-19T21:13:04.428341+0000","last_active":"2024-02-19T21:13:18.349340+0000","last_peered":"2024-02-19T21:13:18.349340+0000","last_clean":"2024-02-19T21:13:18.349340+0000","last_became_active":"2024-02-19T21:12:09.323625+0000","last_became_peered":"2024-02-19T21:12:09.323625+0000","last_unstale":"2024-02-19T21:13:18.349340+0000","last_undegraded":"2024-02-19T21:13:18.349340+0000","last_fullsized":"2024-02-19T21:13:18.349340+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:04.428236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:04.428236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":484,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.355080+0000","last_change":"2024-02-19T21:11:59.352488+0000","last_active":"2024-02-19T21:13:18.355080+0000","last_peered":"2024-02-19T21:13:18.355080+0000","last_clean":"2024-02-19T21:13:18.355080+0000","last_became_active":"2024-02-19T21:11:59.351887+0000","last_became_peered":"2024-02-19T21:11:59.351887+0000","last_unstale":"2024-02-19T21:13:18.355080+0000","last_undegraded":"2024-02-19T21:13:18.355080+0000","last_fullsized":"2024-02-19T21:13:18.355080+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.947089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":394,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352835+0000","last_change":"2024-02-19T21:12:09.369780+0000","last_active":"2024-02-19T21:13:18.352835+0000","last_peered":"2024-02-19T21:13:18.352835+0000","last_clean":"2024-02-19T21:13:18.352835+0000","last_became_active":"2024-02-19T21:12:09.369431+0000","last_became_peered":"2024-02-19T21:12:09.369431+0000","last_unstale":"2024-02-19T21:13:18.352835+0000","last_undegraded":"2024-02-19T21:13:18.352835+0000","last_fullsized":"2024-02-19T21:13:18.352835+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.945639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.945639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"459'42","reported_seq":798,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.370889+0000","last_change":"2024-02-19T21:10:01.545972+0000","last_active":"2024-02-19T21:13:19.370889+0000","last_peered":"2024-02-19T21:13:19.370889+0000","last_clean":"2024-02-19T21:13:19.370889+0000","last_became_active":"2024-02-19T21:08:04.266322+0000","last_became_peered":"2024-02-19T21:08:04.266322+0000","last_unstale":"2024-02-19T21:13:19.370889+0000","last_undegraded":"2024-02-19T21:13:19.370889+0000","last_fullsized":"2024-02-19T21:13:19.370889+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'30","last_scrub_stamp":"2024-02-19T21:10:01.545897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:01.545897+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":574,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357903+0000","last_change":"2024-02-19T21:12:09.328219+0000","last_active":"2024-02-19T21:13:18.357903+0000","last_peered":"2024-02-19T21:13:18.357903+0000","last_clean":"2024-02-19T21:13:18.357903+0000","last_became_active":"2024-02-19T21:12:09.326579+0000","last_became_peered":"2024-02-19T21:12:09.326579+0000","last_unstale":"2024-02-19T21:13:18.357903+0000","last_undegraded":"2024-02-19T21:13:18.357903+0000","last_fullsized":"2024-02-19T21:13:18.357903+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.089715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.089715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":292,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352171+0000","last_change":"2024-02-19T21:11:31.629134+0000","last_active":"2024-02-19T21:13:18.352171+0000","last_peered":"2024-02-19T21:13:18.352171+0000","last_clean":"2024-02-19T21:13:18.352171+0000","last_became_active":"2024-02-19T21:10:41.305829+0000","last_became_peered":"2024-02-19T21:10:41.305829+0000","last_unstale":"2024-02-19T21:13:18.352171+0000","last_undegraded":"2024-02-19T21:13:18.352171+0000","last_fullsized":"2024-02-19T21:13:18.352171+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:31.629057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.629057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":487,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.350698+0000","last_change":"2024-02-19T21:11:59.333878+0000","last_active":"2024-02-19T21:13:18.350698+0000","last_peered":"2024-02-19T21:13:18.350698+0000","last_clean":"2024-02-19T21:13:18.350698+0000","last_became_active":"2024-02-19T21:11:59.331967+0000","last_became_peered":"2024-02-19T21:11:59.331967+0000","last_unstale":"2024-02-19T21:13:18.350698+0000","last_undegraded":"2024-02-19T21:13:18.350698+0000","last_fullsized":"2024-02-19T21:13:18.350698+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:54.142898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.142898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"493'12","reported_seq":444,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.362528+0000","last_change":"2024-02-19T21:12:09.433235+0000","last_active":"2024-02-19T21:13:19.362528+0000","last_peered":"2024-02-19T21:13:19.362528+0000","last_clean":"2024-02-19T21:13:19.362528+0000","last_became_active":"2024-02-19T21:12:09.326372+0000","last_became_peered":"2024-02-19T21:12:09.326372+0000","last_unstale":"2024-02-19T21:13:19.362528+0000","last_undegraded":"2024-02-19T21:13:19.362528+0000","last_fullsized":"2024-02-19T21:13:19.362528+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'10","last_scrub_stamp":"2024-02-19T21:11:41.775418+0000","last_deep_scrub":"471'10","last_deep_scrub_stamp":"2024-02-19T21:11:41.775418+0000","last_clean_scrub_stamp":"2024-02-19T21:11:41.775418+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"459'20","reported_seq":597,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351432+0000","last_change":"2024-02-19T21:12:09.030294+0000","last_active":"2024-02-19T21:13:18.351432+0000","last_peered":"2024-02-19T21:13:18.351432+0000","last_clean":"2024-02-19T21:13:18.351432+0000","last_became_active":"2024-02-19T21:10:41.296275+0000","last_became_peered":"2024-02-19T21:10:41.296275+0000","last_unstale":"2024-02-19T21:13:18.351432+0000","last_undegraded":"2024-02-19T21:13:18.351432+0000","last_fullsized":"2024-02-19T21:13:18.351432+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:12:09.030215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:09.030215+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":590,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352039+0000","last_change":"2024-02-19T21:11:59.351637+0000","last_active":"2024-02-19T21:13:18.352039+0000","last_peered":"2024-02-19T21:13:18.352039+0000","last_clean":"2024-02-19T21:13:18.352039+0000","last_became_active":"2024-02-19T21:11:59.351241+0000","last_became_peered":"2024-02-19T21:11:59.351241+0000","last_unstale":"2024-02-19T21:13:18.352039+0000","last_undegraded":"2024-02-19T21:13:18.352039+0000","last_fullsized":"2024-02-19T21:13:18.352039+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:21.732778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:10:21.732778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":288,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.353465+0000","last_change":"2024-02-19T21:13:07.221665+0000","last_active":"2024-02-19T21:13:18.353465+0000","last_peered":"2024-02-19T21:13:18.353465+0000","last_clean":"2024-02-19T21:13:18.353465+0000","last_became_active":"2024-02-19T21:11:59.335798+0000","last_became_peered":"2024-02-19T21:11:59.335798+0000","last_unstale":"2024-02-19T21:13:18.353465+0000","last_undegraded":"2024-02-19T21:13:18.353465+0000","last_fullsized":"2024-02-19T21:13:18.353465+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:07.221594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.221594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":283,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352265+0000","last_change":"2024-02-19T21:12:09.332210+0000","last_active":"2024-02-19T21:13:18.352265+0000","last_peered":"2024-02-19T21:13:18.352265+0000","last_clean":"2024-02-19T21:13:18.352265+0000","last_became_active":"2024-02-19T21:12:09.331195+0000","last_became_peered":"2024-02-19T21:12:09.331195+0000","last_unstale":"2024-02-19T21:13:18.352265+0000","last_undegraded":"2024-02-19T21:13:18.352265+0000","last_fullsized":"2024-02-19T21:13:18.352265+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.604025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.604025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":540,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.350515+0000","last_change":"2024-02-19T21:13:09.184029+0000","last_active":"2024-02-19T21:13:18.350515+0000","last_peered":"2024-02-19T21:13:18.350515+0000","last_clean":"2024-02-19T21:13:18.350515+0000","last_became_active":"2024-02-19T21:11:59.333275+0000","last_became_peered":"2024-02-19T21:11:59.333275+0000","last_unstale":"2024-02-19T21:13:18.350515+0000","last_undegraded":"2024-02-19T21:13:18.350515+0000","last_fullsized":"2024-02-19T21:13:18.350515+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.183962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.183962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"459'23","reported_seq":164,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914135+0000","last_change":"2024-02-19T21:12:18.340005+0000","last_active":"2024-02-19T21:13:18.914135+0000","last_peered":"2024-02-19T21:13:18.914135+0000","last_clean":"2024-02-19T21:13:18.914135+0000","last_became_active":"2024-02-19T21:10:41.287134+0000","last_became_peered":"2024-02-19T21:10:41.287134+0000","last_unstale":"2024-02-19T21:13:18.914135+0000","last_undegraded":"2024-02-19T21:13:18.914135+0000","last_fullsized":"2024-02-19T21:13:18.914135+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'23","last_scrub_stamp":"2024-02-19T21:12:18.339929+0000","last_deep_scrub":"420'17","last_deep_scrub_stamp":"2024-02-19T21:10:19.722303+0000","last_clean_scrub_stamp":"2024-02-19T21:12:18.339929+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":164,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.915272+0000","last_change":"2024-02-19T21:11:59.342221+0000","last_active":"2024-02-19T21:13:18.915272+0000","last_peered":"2024-02-19T21:13:18.915272+0000","last_clean":"2024-02-19T21:13:18.915272+0000","last_became_active":"2024-02-19T21:11:59.323514+0000","last_became_peered":"2024-02-19T21:11:59.323514+0000","last_unstale":"2024-02-19T21:13:18.915272+0000","last_undegraded":"2024-02-19T21:13:18.915272+0000","last_fullsized":"2024-02-19T21:13:18.915272+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:10:58.333168+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.333168+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":279,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.350478+0000","last_change":"2024-02-19T21:13:00.166548+0000","last_active":"2024-02-19T21:13:18.350478+0000","last_peered":"2024-02-19T21:13:18.350478+0000","last_clean":"2024-02-19T21:13:18.350478+0000","last_became_active":"2024-02-19T21:08:04.264593+0000","last_became_peered":"2024-02-19T21:08:04.264593+0000","last_unstale":"2024-02-19T21:13:18.350478+0000","last_undegraded":"2024-02-19T21:13:18.350478+0000","last_fullsized":"2024-02-19T21:13:18.350478+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.166479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.166479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":478,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351149+0000","last_change":"2024-02-19T21:11:03.195680+0000","last_active":"2024-02-19T21:13:18.351149+0000","last_peered":"2024-02-19T21:13:18.351149+0000","last_clean":"2024-02-19T21:13:18.351149+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:13:18.351149+0000","last_undegraded":"2024-02-19T21:13:18.351149+0000","last_fullsized":"2024-02-19T21:13:18.351149+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.195616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.195616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"459'15","reported_seq":314,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352078+0000","last_change":"2024-02-19T21:11:59.364955+0000","last_active":"2024-02-19T21:13:18.352078+0000","last_peered":"2024-02-19T21:13:18.352078+0000","last_clean":"2024-02-19T21:13:18.352078+0000","last_became_active":"2024-02-19T21:11:59.336293+0000","last_became_peered":"2024-02-19T21:11:59.336293+0000","last_unstale":"2024-02-19T21:13:18.352078+0000","last_undegraded":"2024-02-19T21:13:18.352078+0000","last_fullsized":"2024-02-19T21:13:18.352078+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:10:22.459807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:22.459807+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":539,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.350440+0000","last_change":"2024-02-19T21:11:59.334071+0000","last_active":"2024-02-19T21:13:18.350440+0000","last_peered":"2024-02-19T21:13:18.350440+0000","last_clean":"2024-02-19T21:13:18.350440+0000","last_became_active":"2024-02-19T21:11:59.333684+0000","last_became_peered":"2024-02-19T21:11:59.333684+0000","last_unstale":"2024-02-19T21:13:18.350440+0000","last_undegraded":"2024-02-19T21:13:18.350440+0000","last_fullsized":"2024-02-19T21:13:18.350440+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.244285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.244285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":274,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351533+0000","last_change":"2024-02-19T21:10:53.703797+0000","last_active":"2024-02-19T21:13:18.351533+0000","last_peered":"2024-02-19T21:13:18.351533+0000","last_clean":"2024-02-19T21:13:18.351533+0000","last_became_active":"2024-02-19T21:08:04.264972+0000","last_became_peered":"2024-02-19T21:08:04.264972+0000","last_unstale":"2024-02-19T21:13:18.351533+0000","last_undegraded":"2024-02-19T21:13:18.351533+0000","last_fullsized":"2024-02-19T21:13:18.351533+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:53.703727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.703727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":547,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352792+0000","last_change":"2024-02-19T21:12:06.261731+0000","last_active":"2024-02-19T21:13:18.352792+0000","last_peered":"2024-02-19T21:13:18.352792+0000","last_clean":"2024-02-19T21:13:18.352792+0000","last_became_active":"2024-02-19T21:10:41.314114+0000","last_became_peered":"2024-02-19T21:10:41.314114+0000","last_unstale":"2024-02-19T21:13:18.352792+0000","last_undegraded":"2024-02-19T21:13:18.352792+0000","last_fullsized":"2024-02-19T21:13:18.352792+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.261670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.261670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":143,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914531+0000","last_change":"2024-02-19T21:12:08.430936+0000","last_active":"2024-02-19T21:13:18.914531+0000","last_peered":"2024-02-19T21:13:18.914531+0000","last_clean":"2024-02-19T21:13:18.914531+0000","last_became_active":"2024-02-19T21:10:41.300574+0000","last_became_peered":"2024-02-19T21:10:41.300574+0000","last_unstale":"2024-02-19T21:13:18.914531+0000","last_undegraded":"2024-02-19T21:13:18.914531+0000","last_fullsized":"2024-02-19T21:13:18.914531+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:08.430874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:08.430874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"459'15","reported_seq":161,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352147+0000","last_change":"2024-02-19T21:10:53.101489+0000","last_active":"2024-02-19T21:13:18.352147+0000","last_peered":"2024-02-19T21:13:18.352147+0000","last_clean":"2024-02-19T21:13:18.352147+0000","last_became_active":"2024-02-19T21:10:40.297867+0000","last_became_peered":"2024-02-19T21:10:40.297867+0000","last_unstale":"2024-02-19T21:13:18.352147+0000","last_undegraded":"2024-02-19T21:13:18.352147+0000","last_fullsized":"2024-02-19T21:13:18.352147+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'9","last_scrub_stamp":"2024-02-19T21:10:53.101422+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.101422+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":489,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.913980+0000","last_change":"2024-02-19T21:12:09.369726+0000","last_active":"2024-02-19T21:13:18.913980+0000","last_peered":"2024-02-19T21:13:18.913980+0000","last_clean":"2024-02-19T21:13:18.913980+0000","last_became_active":"2024-02-19T21:12:09.369407+0000","last_became_peered":"2024-02-19T21:12:09.369407+0000","last_unstale":"2024-02-19T21:13:18.913980+0000","last_undegraded":"2024-02-19T21:13:18.913980+0000","last_fullsized":"2024-02-19T21:13:18.913980+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:31.670026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.670026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":545,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.358135+0000","last_change":"2024-02-19T21:12:06.098778+0000","last_active":"2024-02-19T21:13:18.358135+0000","last_peered":"2024-02-19T21:13:18.358135+0000","last_clean":"2024-02-19T21:13:18.358135+0000","last_became_active":"2024-02-19T21:08:04.265573+0000","last_became_peered":"2024-02-19T21:08:04.265573+0000","last_unstale":"2024-02-19T21:13:18.358135+0000","last_undegraded":"2024-02-19T21:13:18.358135+0000","last_fullsized":"2024-02-19T21:13:18.358135+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.098703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.098703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":555,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349048+0000","last_change":"2024-02-19T21:12:17.452030+0000","last_active":"2024-02-19T21:13:18.349048+0000","last_peered":"2024-02-19T21:13:18.349048+0000","last_clean":"2024-02-19T21:13:18.349048+0000","last_became_active":"2024-02-19T21:12:09.341406+0000","last_became_peered":"2024-02-19T21:12:09.341406+0000","last_unstale":"2024-02-19T21:13:18.349048+0000","last_undegraded":"2024-02-19T21:13:18.349048+0000","last_fullsized":"2024-02-19T21:13:18.349048+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:17.451964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:17.451964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":504,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357452+0000","last_change":"2024-02-19T21:12:47.428132+0000","last_active":"2024-02-19T21:13:18.357452+0000","last_peered":"2024-02-19T21:13:18.357452+0000","last_clean":"2024-02-19T21:13:18.357452+0000","last_became_active":"2024-02-19T21:12:09.349695+0000","last_became_peered":"2024-02-19T21:12:09.349695+0000","last_unstale":"2024-02-19T21:13:18.357452+0000","last_undegraded":"2024-02-19T21:13:18.357452+0000","last_fullsized":"2024-02-19T21:13:18.357452+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:47.428055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:47.428055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"459'20","reported_seq":301,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351449+0000","last_change":"2024-02-19T21:10:59.685223+0000","last_active":"2024-02-19T21:13:18.351449+0000","last_peered":"2024-02-19T21:13:18.351449+0000","last_clean":"2024-02-19T21:13:18.351449+0000","last_became_active":"2024-02-19T21:08:04.267216+0000","last_became_peered":"2024-02-19T21:08:04.267216+0000","last_unstale":"2024-02-19T21:13:18.351449+0000","last_undegraded":"2024-02-19T21:13:18.351449+0000","last_fullsized":"2024-02-19T21:13:18.351449+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:10:59.685150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.685150+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":547,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357786+0000","last_change":"2024-02-19T21:09:40.986717+0000","last_active":"2024-02-19T21:13:18.357786+0000","last_peered":"2024-02-19T21:13:18.357786+0000","last_clean":"2024-02-19T21:13:18.357786+0000","last_became_active":"2024-02-19T21:08:04.266728+0000","last_became_peered":"2024-02-19T21:08:04.266728+0000","last_unstale":"2024-02-19T21:13:18.357786+0000","last_undegraded":"2024-02-19T21:13:18.357786+0000","last_fullsized":"2024-02-19T21:13:18.357786+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.986635+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"433'9","reported_seq":691,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.362413+0000","last_change":"2024-02-19T21:13:00.428710+0000","last_active":"2024-02-19T21:13:19.362413+0000","last_peered":"2024-02-19T21:13:19.362413+0000","last_clean":"2024-02-19T21:13:19.362413+0000","last_became_active":"2024-02-19T21:10:40.310303+0000","last_became_peered":"2024-02-19T21:10:40.310303+0000","last_unstale":"2024-02-19T21:13:19.362413+0000","last_undegraded":"2024-02-19T21:13:19.362413+0000","last_fullsized":"2024-02-19T21:13:19.362413+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'9","last_scrub_stamp":"2024-02-19T21:13:00.428581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.428581+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":407,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349644+0000","last_change":"2024-02-19T21:11:59.843545+0000","last_active":"2024-02-19T21:13:18.349644+0000","last_peered":"2024-02-19T21:13:18.349644+0000","last_clean":"2024-02-19T21:13:18.349644+0000","last_became_active":"2024-02-19T21:11:59.323539+0000","last_became_peered":"2024-02-19T21:11:59.323539+0000","last_unstale":"2024-02-19T21:13:18.349644+0000","last_undegraded":"2024-02-19T21:13:18.349644+0000","last_fullsized":"2024-02-19T21:13:18.349644+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:11:31.500511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.500511+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":359,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.350301+0000","last_change":"2024-02-19T21:12:09.345187+0000","last_active":"2024-02-19T21:13:18.350301+0000","last_peered":"2024-02-19T21:13:18.350301+0000","last_clean":"2024-02-19T21:13:18.350301+0000","last_became_active":"2024-02-19T21:12:09.344609+0000","last_became_peered":"2024-02-19T21:12:09.344609+0000","last_unstale":"2024-02-19T21:13:18.350301+0000","last_undegraded":"2024-02-19T21:13:18.350301+0000","last_fullsized":"2024-02-19T21:13:18.350301+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.378925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:55.202463+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.378925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"459'15","reported_seq":590,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.358106+0000","last_change":"2024-02-19T21:10:50.053312+0000","last_active":"2024-02-19T21:13:18.358106+0000","last_peered":"2024-02-19T21:13:18.358106+0000","last_clean":"2024-02-19T21:13:18.358106+0000","last_became_active":"2024-02-19T21:08:04.266103+0000","last_became_peered":"2024-02-19T21:08:04.266103+0000","last_unstale":"2024-02-19T21:13:18.358106+0000","last_undegraded":"2024-02-19T21:13:18.358106+0000","last_fullsized":"2024-02-19T21:13:18.358106+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:10:50.053229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.053229+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":157,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914493+0000","last_change":"2024-02-19T21:13:07.507018+0000","last_active":"2024-02-19T21:13:18.914493+0000","last_peered":"2024-02-19T21:13:18.914493+0000","last_clean":"2024-02-19T21:13:18.914493+0000","last_became_active":"2024-02-19T21:11:59.324861+0000","last_became_peered":"2024-02-19T21:11:59.324861+0000","last_unstale":"2024-02-19T21:13:18.914493+0000","last_undegraded":"2024-02-19T21:13:18.914493+0000","last_fullsized":"2024-02-19T21:13:18.914493+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:07.506946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.506946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":347,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.359069+0000","last_change":"2024-02-19T21:12:17.345023+0000","last_active":"2024-02-19T21:13:19.359069+0000","last_peered":"2024-02-19T21:13:19.359069+0000","last_clean":"2024-02-19T21:13:19.359069+0000","last_became_active":"2024-02-19T21:08:04.256045+0000","last_became_peered":"2024-02-19T21:08:04.256045+0000","last_unstale":"2024-02-19T21:13:19.359069+0000","last_undegraded":"2024-02-19T21:13:19.359069+0000","last_fullsized":"2024-02-19T21:13:19.359069+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'10","last_scrub_stamp":"2024-02-19T21:12:17.344916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:17.344916+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":143,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.915252+0000","last_change":"2024-02-19T21:12:05.445686+0000","last_active":"2024-02-19T21:13:18.915252+0000","last_peered":"2024-02-19T21:13:18.915252+0000","last_clean":"2024-02-19T21:13:18.915252+0000","last_became_active":"2024-02-19T21:10:41.310279+0000","last_became_peered":"2024-02-19T21:10:41.310279+0000","last_unstale":"2024-02-19T21:13:18.915252+0000","last_undegraded":"2024-02-19T21:13:18.915252+0000","last_fullsized":"2024-02-19T21:13:18.915252+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:05.445575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_clean_scrub_stamp":"2024-02-19T21:12:05.445575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":535,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.354351+0000","last_change":"2024-02-19T21:12:49.142231+0000","last_active":"2024-02-19T21:13:18.354351+0000","last_peered":"2024-02-19T21:13:18.354351+0000","last_clean":"2024-02-19T21:13:18.354351+0000","last_became_active":"2024-02-19T21:12:09.339844+0000","last_became_peered":"2024-02-19T21:12:09.339844+0000","last_unstale":"2024-02-19T21:13:18.354351+0000","last_undegraded":"2024-02-19T21:13:18.354351+0000","last_fullsized":"2024-02-19T21:13:18.354351+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-02-19T21:12:49.142158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:49.142158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":15,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"459'15","reported_seq":290,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.350961+0000","last_change":"2024-02-19T21:13:06.958616+0000","last_active":"2024-02-19T21:13:18.350961+0000","last_peered":"2024-02-19T21:13:18.350961+0000","last_clean":"2024-02-19T21:13:18.350961+0000","last_became_active":"2024-02-19T21:12:09.341038+0000","last_became_peered":"2024-02-19T21:12:09.341038+0000","last_unstale":"2024-02-19T21:13:18.350961+0000","last_undegraded":"2024-02-19T21:13:18.350961+0000","last_fullsized":"2024-02-19T21:13:18.350961+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:13:06.958518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.958518+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":358,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351019+0000","last_change":"2024-02-19T21:12:09.333684+0000","last_active":"2024-02-19T21:13:18.351019+0000","last_peered":"2024-02-19T21:13:18.351019+0000","last_clean":"2024-02-19T21:13:18.351019+0000","last_became_active":"2024-02-19T21:12:09.333350+0000","last_became_peered":"2024-02-19T21:12:09.333350+0000","last_unstale":"2024-02-19T21:13:18.351019+0000","last_undegraded":"2024-02-19T21:13:18.351019+0000","last_fullsized":"2024-02-19T21:13:18.351019+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.422323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.422323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":288,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.350631+0000","last_change":"2024-02-19T21:11:59.349259+0000","last_active":"2024-02-19T21:13:18.350631+0000","last_peered":"2024-02-19T21:13:18.350631+0000","last_clean":"2024-02-19T21:13:18.350631+0000","last_became_active":"2024-02-19T21:11:59.348485+0000","last_became_peered":"2024-02-19T21:11:59.348485+0000","last_unstale":"2024-02-19T21:13:18.350631+0000","last_undegraded":"2024-02-19T21:13:18.350631+0000","last_fullsized":"2024-02-19T21:13:18.350631+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:45.404242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_clean_scrub_stamp":"2024-02-19T21:11:45.404242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":373,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357169+0000","last_change":"2024-02-19T21:12:09.327887+0000","last_active":"2024-02-19T21:13:18.357169+0000","last_peered":"2024-02-19T21:13:18.357169+0000","last_clean":"2024-02-19T21:13:18.357169+0000","last_became_active":"2024-02-19T21:12:09.327390+0000","last_became_peered":"2024-02-19T21:12:09.327390+0000","last_unstale":"2024-02-19T21:13:18.357169+0000","last_undegraded":"2024-02-19T21:13:18.357169+0000","last_fullsized":"2024-02-19T21:13:18.357169+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:38.940935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:38.940935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":477,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.358114+0000","last_change":"2024-02-19T21:12:09.330102+0000","last_active":"2024-02-19T21:13:18.358114+0000","last_peered":"2024-02-19T21:13:18.358114+0000","last_clean":"2024-02-19T21:13:18.358114+0000","last_became_active":"2024-02-19T21:12:09.329246+0000","last_became_peered":"2024-02-19T21:12:09.329246+0000","last_unstale":"2024-02-19T21:13:18.358114+0000","last_undegraded":"2024-02-19T21:13:18.358114+0000","last_fullsized":"2024-02-19T21:13:18.358114+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.493206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.493206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"459'19","reported_seq":188,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914601+0000","last_change":"2024-02-19T21:12:10.570353+0000","last_active":"2024-02-19T21:13:18.914601+0000","last_peered":"2024-02-19T21:13:18.914601+0000","last_clean":"2024-02-19T21:13:18.914601+0000","last_became_active":"2024-02-19T21:12:09.399929+0000","last_became_peered":"2024-02-19T21:12:09.399929+0000","last_unstale":"2024-02-19T21:13:18.914601+0000","last_undegraded":"2024-02-19T21:13:18.914601+0000","last_fullsized":"2024-02-19T21:13:18.914601+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:11:32.351880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:32.351880+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":550,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352990+0000","last_change":"2024-02-19T21:13:00.874389+0000","last_active":"2024-02-19T21:13:18.352990+0000","last_peered":"2024-02-19T21:13:18.352990+0000","last_clean":"2024-02-19T21:13:18.352990+0000","last_became_active":"2024-02-19T21:12:09.342120+0000","last_became_peered":"2024-02-19T21:12:09.342120+0000","last_unstale":"2024-02-19T21:13:18.352990+0000","last_undegraded":"2024-02-19T21:13:18.352990+0000","last_fullsized":"2024-02-19T21:13:18.352990+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.874318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.874318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":546,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351503+0000","last_change":"2024-02-19T21:12:54.957657+0000","last_active":"2024-02-19T21:13:18.351503+0000","last_peered":"2024-02-19T21:13:18.351503+0000","last_clean":"2024-02-19T21:13:18.351503+0000","last_became_active":"2024-02-19T21:08:04.268847+0000","last_became_peered":"2024-02-19T21:08:04.268847+0000","last_unstale":"2024-02-19T21:13:18.351503+0000","last_undegraded":"2024-02-19T21:13:18.351503+0000","last_fullsized":"2024-02-19T21:13:18.351503+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:54.957585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:54.957585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":283,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352149+0000","last_change":"2024-02-19T21:11:59.341209+0000","last_active":"2024-02-19T21:13:18.352149+0000","last_peered":"2024-02-19T21:13:18.352149+0000","last_clean":"2024-02-19T21:13:18.352149+0000","last_became_active":"2024-02-19T21:11:59.340742+0000","last_became_peered":"2024-02-19T21:11:59.340742+0000","last_unstale":"2024-02-19T21:13:18.352149+0000","last_undegraded":"2024-02-19T21:13:18.352149+0000","last_fullsized":"2024-02-19T21:13:18.352149+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:37.477995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:37.477995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":283,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351614+0000","last_change":"2024-02-19T21:11:59.338599+0000","last_active":"2024-02-19T21:13:18.351614+0000","last_peered":"2024-02-19T21:13:18.351614+0000","last_clean":"2024-02-19T21:13:18.351614+0000","last_became_active":"2024-02-19T21:11:59.337764+0000","last_became_peered":"2024-02-19T21:11:59.337764+0000","last_unstale":"2024-02-19T21:13:18.351614+0000","last_undegraded":"2024-02-19T21:13:18.351614+0000","last_fullsized":"2024-02-19T21:13:18.351614+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.599474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.599474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"555'10","reported_seq":171,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.356938+0000","last_change":"2024-02-19T21:13:12.409114+0000","last_active":"2024-02-19T21:13:18.356938+0000","last_peered":"2024-02-19T21:13:18.356938+0000","last_clean":"2024-02-19T21:13:18.356938+0000","last_became_active":"2024-02-19T21:12:09.340298+0000","last_became_peered":"2024-02-19T21:12:09.340298+0000","last_unstale":"2024-02-19T21:13:18.356938+0000","last_undegraded":"2024-02-19T21:13:18.356938+0000","last_fullsized":"2024-02-19T21:13:18.356938+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'10","last_scrub_stamp":"2024-02-19T21:13:12.409036+0000","last_deep_scrub":"555'10","last_deep_scrub_stamp":"2024-02-19T21:13:12.409036+0000","last_clean_scrub_stamp":"2024-02-19T21:13:12.409036+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":538,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.358266+0000","last_change":"2024-02-19T21:12:09.330300+0000","last_active":"2024-02-19T21:13:18.358266+0000","last_peered":"2024-02-19T21:13:18.358266+0000","last_clean":"2024-02-19T21:13:18.358266+0000","last_became_active":"2024-02-19T21:12:09.329752+0000","last_became_peered":"2024-02-19T21:12:09.329752+0000","last_unstale":"2024-02-19T21:13:18.358266+0000","last_undegraded":"2024-02-19T21:13:18.358266+0000","last_fullsized":"2024-02-19T21:13:18.358266+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:42.000850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:42.000850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":485,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.913944+0000","last_change":"2024-02-19T21:12:09.346328+0000","last_active":"2024-02-19T21:13:18.913944+0000","last_peered":"2024-02-19T21:13:18.913944+0000","last_clean":"2024-02-19T21:13:18.913944+0000","last_became_active":"2024-02-19T21:12:09.345654+0000","last_became_peered":"2024-02-19T21:12:09.345654+0000","last_unstale":"2024-02-19T21:13:18.913944+0000","last_undegraded":"2024-02-19T21:13:18.913944+0000","last_fullsized":"2024-02-19T21:13:18.913944+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.467476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.467476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":544,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.348625+0000","last_change":"2024-02-19T21:12:05.530113+0000","last_active":"2024-02-19T21:13:18.348625+0000","last_peered":"2024-02-19T21:13:18.348625+0000","last_clean":"2024-02-19T21:13:18.348625+0000","last_became_active":"2024-02-19T21:10:41.314589+0000","last_became_peered":"2024-02-19T21:10:41.314589+0000","last_unstale":"2024-02-19T21:13:18.348625+0000","last_undegraded":"2024-02-19T21:13:18.348625+0000","last_fullsized":"2024-02-19T21:13:18.348625+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:05.530039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:05.530039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":279,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351523+0000","last_change":"2024-02-19T21:12:07.259353+0000","last_active":"2024-02-19T21:13:18.351523+0000","last_peered":"2024-02-19T21:13:18.351523+0000","last_clean":"2024-02-19T21:13:18.351523+0000","last_became_active":"2024-02-19T21:08:04.266161+0000","last_became_peered":"2024-02-19T21:08:04.266161+0000","last_unstale":"2024-02-19T21:13:18.351523+0000","last_undegraded":"2024-02-19T21:13:18.351523+0000","last_fullsized":"2024-02-19T21:13:18.351523+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:07.259290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:07.259290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"459'19","reported_seq":324,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352312+0000","last_change":"2024-02-19T21:12:09.375334+0000","last_active":"2024-02-19T21:13:18.352312+0000","last_peered":"2024-02-19T21:13:18.352312+0000","last_clean":"2024-02-19T21:13:18.352312+0000","last_became_active":"2024-02-19T21:12:09.321408+0000","last_became_peered":"2024-02-19T21:12:09.321408+0000","last_unstale":"2024-02-19T21:13:18.352312+0000","last_undegraded":"2024-02-19T21:13:18.352312+0000","last_fullsized":"2024-02-19T21:13:18.352312+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:12:03.255395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:03.255395+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":79,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.358530+0000","last_change":"2024-02-19T21:12:09.345352+0000","last_active":"2024-02-19T21:13:18.358530+0000","last_peered":"2024-02-19T21:13:18.358530+0000","last_clean":"2024-02-19T21:13:18.358530+0000","last_became_active":"2024-02-19T21:12:09.345000+0000","last_became_peered":"2024-02-19T21:12:09.345000+0000","last_unstale":"2024-02-19T21:13:18.358530+0000","last_undegraded":"2024-02-19T21:13:18.358530+0000","last_fullsized":"2024-02-19T21:13:18.358530+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:00.382946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.382946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":550,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349118+0000","last_change":"2024-02-19T21:12:09.332889+0000","last_active":"2024-02-19T21:13:18.349118+0000","last_peered":"2024-02-19T21:13:18.349118+0000","last_clean":"2024-02-19T21:13:18.349118+0000","last_became_active":"2024-02-19T21:12:09.332485+0000","last_became_peered":"2024-02-19T21:12:09.332485+0000","last_unstale":"2024-02-19T21:13:18.349118+0000","last_undegraded":"2024-02-19T21:13:18.349118+0000","last_fullsized":"2024-02-19T21:13:18.349118+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.566208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.566208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":292,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.350708+0000","last_change":"2024-02-19T21:12:09.344855+0000","last_active":"2024-02-19T21:13:18.350708+0000","last_peered":"2024-02-19T21:13:18.350708+0000","last_clean":"2024-02-19T21:13:18.350708+0000","last_became_active":"2024-02-19T21:12:09.344536+0000","last_became_peered":"2024-02-19T21:12:09.344536+0000","last_unstale":"2024-02-19T21:13:18.350708+0000","last_undegraded":"2024-02-19T21:13:18.350708+0000","last_fullsized":"2024-02-19T21:13:18.350708+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:00.547228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:00.547228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":365,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.358418+0000","last_change":"2024-02-19T21:12:09.343909+0000","last_active":"2024-02-19T21:13:18.358418+0000","last_peered":"2024-02-19T21:13:18.358418+0000","last_clean":"2024-02-19T21:13:18.358418+0000","last_became_active":"2024-02-19T21:12:09.342686+0000","last_became_peered":"2024-02-19T21:12:09.342686+0000","last_unstale":"2024-02-19T21:13:18.358418+0000","last_undegraded":"2024-02-19T21:13:18.358418+0000","last_fullsized":"2024-02-19T21:13:18.358418+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:37.718654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:11:37.718654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"459'20","reported_seq":464,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349727+0000","last_change":"2024-02-19T21:12:48.422862+0000","last_active":"2024-02-19T21:13:18.349727+0000","last_peered":"2024-02-19T21:13:18.349727+0000","last_clean":"2024-02-19T21:13:18.349727+0000","last_became_active":"2024-02-19T21:11:59.350142+0000","last_became_peered":"2024-02-19T21:11:59.350142+0000","last_unstale":"2024-02-19T21:13:18.349727+0000","last_undegraded":"2024-02-19T21:13:18.349727+0000","last_fullsized":"2024-02-19T21:13:18.349727+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:12:48.422763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:48.422763+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":547,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357037+0000","last_change":"2024-02-19T21:12:48.430341+0000","last_active":"2024-02-19T21:13:18.357037+0000","last_peered":"2024-02-19T21:13:18.357037+0000","last_clean":"2024-02-19T21:13:18.357037+0000","last_became_active":"2024-02-19T21:10:41.294592+0000","last_became_peered":"2024-02-19T21:10:41.294592+0000","last_unstale":"2024-02-19T21:13:18.357037+0000","last_undegraded":"2024-02-19T21:13:18.357037+0000","last_fullsized":"2024-02-19T21:13:18.357037+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:48.430223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:48.430223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"493'11","reported_seq":632,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-02-19T21:13:19.356063+0000","last_change":"2024-02-19T21:12:10.210682+0000","last_active":"2024-02-19T21:13:19.356063+0000","last_peered":"2024-02-19T21:13:19.356063+0000","last_clean":"2024-02-19T21:13:19.356063+0000","last_became_active":"2024-02-19T21:12:09.352204+0000","last_became_peered":"2024-02-19T21:12:09.352204+0000","last_unstale":"2024-02-19T21:13:19.356063+0000","last_undegraded":"2024-02-19T21:13:19.356063+0000","last_fullsized":"2024-02-19T21:13:19.356063+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'10","last_scrub_stamp":"2024-02-19T21:10:59.405417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.405417+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":528,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349602+0000","last_change":"2024-02-19T21:11:43.546468+0000","last_active":"2024-02-19T21:13:18.349602+0000","last_peered":"2024-02-19T21:13:18.349602+0000","last_clean":"2024-02-19T21:13:18.349602+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:13:18.349602+0000","last_undegraded":"2024-02-19T21:13:18.349602+0000","last_fullsized":"2024-02-19T21:13:18.349602+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:11:43.546383+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.546383+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":481,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.353749+0000","last_change":"2024-02-19T21:11:38.369815+0000","last_active":"2024-02-19T21:13:18.353749+0000","last_peered":"2024-02-19T21:13:18.353749+0000","last_clean":"2024-02-19T21:13:18.353749+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:13:18.353749+0000","last_undegraded":"2024-02-19T21:13:18.353749+0000","last_fullsized":"2024-02-19T21:13:18.353749+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.369716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.369716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"459'19","reported_seq":547,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.358566+0000","last_change":"2024-02-19T21:12:10.390849+0000","last_active":"2024-02-19T21:13:18.358566+0000","last_peered":"2024-02-19T21:13:18.358566+0000","last_clean":"2024-02-19T21:13:18.358566+0000","last_became_active":"2024-02-19T21:12:09.350490+0000","last_became_peered":"2024-02-19T21:12:09.350490+0000","last_unstale":"2024-02-19T21:13:18.358566+0000","last_undegraded":"2024-02-19T21:13:18.358566+0000","last_fullsized":"2024-02-19T21:13:18.358566+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:10:54.435839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.435839+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":292,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352365+0000","last_change":"2024-02-19T21:12:09.333078+0000","last_active":"2024-02-19T21:13:18.352365+0000","last_peered":"2024-02-19T21:13:18.352365+0000","last_clean":"2024-02-19T21:13:18.352365+0000","last_became_active":"2024-02-19T21:12:09.332587+0000","last_became_peered":"2024-02-19T21:12:09.332587+0000","last_unstale":"2024-02-19T21:13:18.352365+0000","last_undegraded":"2024-02-19T21:13:18.352365+0000","last_fullsized":"2024-02-19T21:13:18.352365+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:32.600213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:32.600213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":368,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349785+0000","last_change":"2024-02-19T21:11:59.334743+0000","last_active":"2024-02-19T21:13:18.349785+0000","last_peered":"2024-02-19T21:13:18.349785+0000","last_clean":"2024-02-19T21:13:18.349785+0000","last_became_active":"2024-02-19T21:11:59.334454+0000","last_became_peered":"2024-02-19T21:11:59.334454+0000","last_unstale":"2024-02-19T21:13:18.349785+0000","last_undegraded":"2024-02-19T21:13:18.349785+0000","last_fullsized":"2024-02-19T21:13:18.349785+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.692260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.692260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":539,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352759+0000","last_change":"2024-02-19T21:12:06.991132+0000","last_active":"2024-02-19T21:13:18.352759+0000","last_peered":"2024-02-19T21:13:18.352759+0000","last_clean":"2024-02-19T21:13:18.352759+0000","last_became_active":"2024-02-19T21:10:41.312577+0000","last_became_peered":"2024-02-19T21:10:41.312577+0000","last_unstale":"2024-02-19T21:13:18.352759+0000","last_undegraded":"2024-02-19T21:13:18.352759+0000","last_fullsized":"2024-02-19T21:13:18.352759+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.991039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.991039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":493,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.355016+0000","last_change":"2024-02-19T21:10:59.782435+0000","last_active":"2024-02-19T21:13:18.355016+0000","last_peered":"2024-02-19T21:13:18.355016+0000","last_clean":"2024-02-19T21:13:18.355016+0000","last_became_active":"2024-02-19T21:10:41.311050+0000","last_became_peered":"2024-02-19T21:10:41.311050+0000","last_unstale":"2024-02-19T21:13:18.355016+0000","last_undegraded":"2024-02-19T21:13:18.355016+0000","last_fullsized":"2024-02-19T21:13:18.355016+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:59.782367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.782367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"459'34","reported_seq":628,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352986+0000","last_change":"2024-02-19T21:10:19.109353+0000","last_active":"2024-02-19T21:13:18.352986+0000","last_peered":"2024-02-19T21:13:18.352986+0000","last_clean":"2024-02-19T21:13:18.352986+0000","last_became_active":"2024-02-19T21:08:04.248378+0000","last_became_peered":"2024-02-19T21:08:04.248378+0000","last_unstale":"2024-02-19T21:13:18.352986+0000","last_undegraded":"2024-02-19T21:13:18.352986+0000","last_fullsized":"2024-02-19T21:13:18.352986+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'22","last_scrub_stamp":"2024-02-19T21:10:19.109244+0000","last_deep_scrub":"419'22","last_deep_scrub_stamp":"2024-02-19T21:10:19.109244+0000","last_clean_scrub_stamp":"2024-02-19T21:10:19.109244+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":489,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914300+0000","last_change":"2024-02-19T21:12:09.341126+0000","last_active":"2024-02-19T21:13:18.914300+0000","last_peered":"2024-02-19T21:13:18.914300+0000","last_clean":"2024-02-19T21:13:18.914300+0000","last_became_active":"2024-02-19T21:12:09.340803+0000","last_became_peered":"2024-02-19T21:12:09.340803+0000","last_unstale":"2024-02-19T21:13:18.914300+0000","last_undegraded":"2024-02-19T21:13:18.914300+0000","last_fullsized":"2024-02-19T21:13:18.914300+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.379335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.379335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":550,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.353205+0000","last_change":"2024-02-19T21:11:59.323076+0000","last_active":"2024-02-19T21:13:18.353205+0000","last_peered":"2024-02-19T21:13:18.353205+0000","last_clean":"2024-02-19T21:13:18.353205+0000","last_became_active":"2024-02-19T21:11:59.322735+0000","last_became_peered":"2024-02-19T21:11:59.322735+0000","last_unstale":"2024-02-19T21:13:18.353205+0000","last_undegraded":"2024-02-19T21:13:18.353205+0000","last_fullsized":"2024-02-19T21:13:18.353205+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:53.709456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.709456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":152,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914977+0000","last_change":"2024-02-19T21:12:09.370734+0000","last_active":"2024-02-19T21:13:18.914977+0000","last_peered":"2024-02-19T21:13:18.914977+0000","last_clean":"2024-02-19T21:13:18.914977+0000","last_became_active":"2024-02-19T21:12:09.370318+0000","last_became_peered":"2024-02-19T21:12:09.370318+0000","last_unstale":"2024-02-19T21:13:18.914977+0000","last_undegraded":"2024-02-19T21:13:18.914977+0000","last_fullsized":"2024-02-19T21:13:18.914977+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.335152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.335152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":359,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351584+0000","last_change":"2024-02-19T21:12:09.340710+0000","last_active":"2024-02-19T21:13:18.351584+0000","last_peered":"2024-02-19T21:13:18.351584+0000","last_clean":"2024-02-19T21:13:18.351584+0000","last_became_active":"2024-02-19T21:12:09.340360+0000","last_became_peered":"2024-02-19T21:12:09.340360+0000","last_unstale":"2024-02-19T21:13:18.351584+0000","last_undegraded":"2024-02-19T21:13:18.351584+0000","last_fullsized":"2024-02-19T21:13:18.351584+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.342173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.342173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"459'18","reported_seq":342,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352076+0000","last_change":"2024-02-19T21:12:50.243591+0000","last_active":"2024-02-19T21:13:18.352076+0000","last_peered":"2024-02-19T21:13:18.352076+0000","last_clean":"2024-02-19T21:13:18.352076+0000","last_became_active":"2024-02-19T21:10:41.307690+0000","last_became_peered":"2024-02-19T21:10:41.307690+0000","last_unstale":"2024-02-19T21:13:18.352076+0000","last_undegraded":"2024-02-19T21:13:18.352076+0000","last_fullsized":"2024-02-19T21:13:18.352076+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'18","last_scrub_stamp":"2024-02-19T21:12:50.243498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:50.243498+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":283,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.350787+0000","last_change":"2024-02-19T21:12:09.344823+0000","last_active":"2024-02-19T21:13:18.350787+0000","last_peered":"2024-02-19T21:13:18.350787+0000","last_clean":"2024-02-19T21:13:18.350787+0000","last_became_active":"2024-02-19T21:12:09.344422+0000","last_became_peered":"2024-02-19T21:12:09.344422+0000","last_unstale":"2024-02-19T21:13:18.350787+0000","last_undegraded":"2024-02-19T21:13:18.350787+0000","last_fullsized":"2024-02-19T21:13:18.350787+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:04.545988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:04.545988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":143,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914680+0000","last_change":"2024-02-19T21:11:43.369675+0000","last_active":"2024-02-19T21:13:18.914680+0000","last_peered":"2024-02-19T21:13:18.914680+0000","last_clean":"2024-02-19T21:13:18.914680+0000","last_became_active":"2024-02-19T21:10:41.296849+0000","last_became_peered":"2024-02-19T21:10:41.296849+0000","last_unstale":"2024-02-19T21:13:18.914680+0000","last_undegraded":"2024-02-19T21:13:18.914680+0000","last_fullsized":"2024-02-19T21:13:18.914680+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:43.369607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.369607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":283,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351697+0000","last_change":"2024-02-19T21:12:09.332385+0000","last_active":"2024-02-19T21:13:18.351697+0000","last_peered":"2024-02-19T21:13:18.351697+0000","last_clean":"2024-02-19T21:13:18.351697+0000","last_became_active":"2024-02-19T21:12:09.332044+0000","last_became_peered":"2024-02-19T21:12:09.332044+0000","last_unstale":"2024-02-19T21:13:18.351697+0000","last_undegraded":"2024-02-19T21:13:18.351697+0000","last_fullsized":"2024-02-19T21:13:18.351697+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:58.688166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.688166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":484,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.354189+0000","last_change":"2024-02-19T21:12:09.350012+0000","last_active":"2024-02-19T21:13:18.354189+0000","last_peered":"2024-02-19T21:13:18.354189+0000","last_clean":"2024-02-19T21:13:18.354189+0000","last_became_active":"2024-02-19T21:12:09.349709+0000","last_became_peered":"2024-02-19T21:12:09.349709+0000","last_unstale":"2024-02-19T21:13:18.354189+0000","last_undegraded":"2024-02-19T21:13:18.354189+0000","last_fullsized":"2024-02-19T21:13:18.354189+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:00.754327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.754327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"459'19","reported_seq":181,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914407+0000","last_change":"2024-02-19T21:12:55.429884+0000","last_active":"2024-02-19T21:13:18.914407+0000","last_peered":"2024-02-19T21:13:18.914407+0000","last_clean":"2024-02-19T21:13:18.914407+0000","last_became_active":"2024-02-19T21:11:59.336536+0000","last_became_peered":"2024-02-19T21:11:59.336536+0000","last_unstale":"2024-02-19T21:13:18.914407+0000","last_undegraded":"2024-02-19T21:13:18.914407+0000","last_fullsized":"2024-02-19T21:13:18.914407+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:12:55.429801+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.429801+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":527,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349270+0000","last_change":"2024-02-19T21:11:30.472665+0000","last_active":"2024-02-19T21:13:18.349270+0000","last_peered":"2024-02-19T21:13:18.349270+0000","last_clean":"2024-02-19T21:13:18.349270+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:13:18.349270+0000","last_undegraded":"2024-02-19T21:13:18.349270+0000","last_fullsized":"2024-02-19T21:13:18.349270+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:30.472591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:30.472591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":534,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357623+0000","last_change":"2024-02-19T21:13:06.395759+0000","last_active":"2024-02-19T21:13:18.357623+0000","last_peered":"2024-02-19T21:13:18.357623+0000","last_clean":"2024-02-19T21:13:18.357623+0000","last_became_active":"2024-02-19T21:11:59.323593+0000","last_became_peered":"2024-02-19T21:11:59.323593+0000","last_unstale":"2024-02-19T21:13:18.357623+0000","last_undegraded":"2024-02-19T21:13:18.357623+0000","last_fullsized":"2024-02-19T21:13:18.357623+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:06.395650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.395650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":362,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351084+0000","last_change":"2024-02-19T21:11:59.352657+0000","last_active":"2024-02-19T21:13:18.351084+0000","last_peered":"2024-02-19T21:13:18.351084+0000","last_clean":"2024-02-19T21:13:18.351084+0000","last_became_active":"2024-02-19T21:11:59.352235+0000","last_became_peered":"2024-02-19T21:11:59.352235+0000","last_unstale":"2024-02-19T21:13:18.351084+0000","last_undegraded":"2024-02-19T21:13:18.351084+0000","last_fullsized":"2024-02-19T21:13:18.351084+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:10.642603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:10:10.642603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":510,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351932+0000","last_change":"2024-02-19T21:13:10.146168+0000","last_active":"2024-02-19T21:13:18.351932+0000","last_peered":"2024-02-19T21:13:18.351932+0000","last_clean":"2024-02-19T21:13:18.351932+0000","last_became_active":"2024-02-19T21:11:59.357345+0000","last_became_peered":"2024-02-19T21:11:59.357345+0000","last_unstale":"2024-02-19T21:13:18.351932+0000","last_undegraded":"2024-02-19T21:13:18.351932+0000","last_fullsized":"2024-02-19T21:13:18.351932+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:10.146102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:13:10.146102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"459'7","reported_seq":152,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914574+0000","last_change":"2024-02-19T21:12:02.521592+0000","last_active":"2024-02-19T21:13:18.914574+0000","last_peered":"2024-02-19T21:13:18.914574+0000","last_clean":"2024-02-19T21:13:18.914574+0000","last_became_active":"2024-02-19T21:10:41.297295+0000","last_became_peered":"2024-02-19T21:10:41.297295+0000","last_unstale":"2024-02-19T21:13:18.914574+0000","last_undegraded":"2024-02-19T21:13:18.914574+0000","last_fullsized":"2024-02-19T21:13:18.914574+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'7","last_scrub_stamp":"2024-02-19T21:12:02.521523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:02.521523+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":550,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349212+0000","last_change":"2024-02-19T21:12:09.324077+0000","last_active":"2024-02-19T21:13:18.349212+0000","last_peered":"2024-02-19T21:13:18.349212+0000","last_clean":"2024-02-19T21:13:18.349212+0000","last_became_active":"2024-02-19T21:12:09.323141+0000","last_became_peered":"2024-02-19T21:12:09.323141+0000","last_unstale":"2024-02-19T21:13:18.349212+0000","last_undegraded":"2024-02-19T21:13:18.349212+0000","last_fullsized":"2024-02-19T21:13:18.349212+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:04.547957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:04.547957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":546,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.348724+0000","last_change":"2024-02-19T21:12:51.411694+0000","last_active":"2024-02-19T21:13:18.348724+0000","last_peered":"2024-02-19T21:13:18.348724+0000","last_clean":"2024-02-19T21:13:18.348724+0000","last_became_active":"2024-02-19T21:10:41.307400+0000","last_became_peered":"2024-02-19T21:10:41.307400+0000","last_unstale":"2024-02-19T21:13:18.348724+0000","last_undegraded":"2024-02-19T21:13:18.348724+0000","last_fullsized":"2024-02-19T21:13:18.348724+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:51.411574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.411574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":541,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357442+0000","last_change":"2024-02-19T21:10:15.161470+0000","last_active":"2024-02-19T21:13:18.357442+0000","last_peered":"2024-02-19T21:13:18.357442+0000","last_clean":"2024-02-19T21:13:18.357442+0000","last_became_active":"2024-02-19T21:08:04.266100+0000","last_became_peered":"2024-02-19T21:08:04.266100+0000","last_unstale":"2024-02-19T21:13:18.357442+0000","last_undegraded":"2024-02-19T21:13:18.357442+0000","last_fullsized":"2024-02-19T21:13:18.357442+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:15.161397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:15.161397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":387,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.358307+0000","last_change":"2024-02-19T21:12:40.365196+0000","last_active":"2024-02-19T21:13:18.358307+0000","last_peered":"2024-02-19T21:13:18.358307+0000","last_clean":"2024-02-19T21:13:18.358307+0000","last_became_active":"2024-02-19T21:12:40.364772+0000","last_became_peered":"2024-02-19T21:12:40.364772+0000","last_unstale":"2024-02-19T21:13:18.358307+0000","last_undegraded":"2024-02-19T21:13:18.358307+0000","last_fullsized":"2024-02-19T21:13:18.358307+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:56.083403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:56.083403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"459'15","reported_seq":181,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914172+0000","last_change":"2024-02-19T21:12:09.959562+0000","last_active":"2024-02-19T21:13:18.914172+0000","last_peered":"2024-02-19T21:13:18.914172+0000","last_clean":"2024-02-19T21:13:18.914172+0000","last_became_active":"2024-02-19T21:12:09.352882+0000","last_became_peered":"2024-02-19T21:12:09.352882+0000","last_unstale":"2024-02-19T21:13:18.914172+0000","last_undegraded":"2024-02-19T21:13:18.914172+0000","last_fullsized":"2024-02-19T21:13:18.914172+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:11:03.423767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.423767+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":283,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352345+0000","last_change":"2024-02-19T21:11:59.337265+0000","last_active":"2024-02-19T21:13:18.352345+0000","last_peered":"2024-02-19T21:13:18.352345+0000","last_clean":"2024-02-19T21:13:18.352345+0000","last_became_active":"2024-02-19T21:11:59.336778+0000","last_became_peered":"2024-02-19T21:11:59.336778+0000","last_unstale":"2024-02-19T21:13:18.352345+0000","last_undegraded":"2024-02-19T21:13:18.352345+0000","last_fullsized":"2024-02-19T21:13:18.352345+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:50.769620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.769620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":523,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.353989+0000","last_change":"2024-02-19T21:10:18.737245+0000","last_active":"2024-02-19T21:13:18.353989+0000","last_peered":"2024-02-19T21:13:18.353989+0000","last_clean":"2024-02-19T21:13:18.353989+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:13:18.353989+0000","last_undegraded":"2024-02-19T21:13:18.353989+0000","last_fullsized":"2024-02-19T21:13:18.353989+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:18.737176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:10:18.737176+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":143,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914510+0000","last_change":"2024-02-19T21:12:07.416212+0000","last_active":"2024-02-19T21:13:18.914510+0000","last_peered":"2024-02-19T21:13:18.914510+0000","last_clean":"2024-02-19T21:13:18.914510+0000","last_became_active":"2024-02-19T21:10:41.309550+0000","last_became_peered":"2024-02-19T21:10:41.309550+0000","last_unstale":"2024-02-19T21:13:18.914510+0000","last_undegraded":"2024-02-19T21:13:18.914510+0000","last_fullsized":"2024-02-19T21:13:18.914510+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:07.416150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:07.416150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":504,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.357834+0000","last_change":"2024-02-19T21:11:59.333342+0000","last_active":"2024-02-19T21:13:18.357834+0000","last_peered":"2024-02-19T21:13:18.357834+0000","last_clean":"2024-02-19T21:13:18.357834+0000","last_became_active":"2024-02-19T21:11:59.332987+0000","last_became_peered":"2024-02-19T21:11:59.332987+0000","last_unstale":"2024-02-19T21:13:18.357834+0000","last_undegraded":"2024-02-19T21:13:18.357834+0000","last_fullsized":"2024-02-19T21:13:18.357834+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:08.043028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:08.043028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"459'27","reported_seq":333,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.352065+0000","last_change":"2024-02-19T21:11:59.360178+0000","last_active":"2024-02-19T21:13:18.352065+0000","last_peered":"2024-02-19T21:13:18.352065+0000","last_clean":"2024-02-19T21:13:18.352065+0000","last_became_active":"2024-02-19T21:11:59.340940+0000","last_became_peered":"2024-02-19T21:11:59.340940+0000","last_unstale":"2024-02-19T21:13:18.352065+0000","last_undegraded":"2024-02-19T21:13:18.352065+0000","last_fullsized":"2024-02-19T21:13:18.352065+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:10:02.532799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:02.532799+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":397,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.349882+0000","last_change":"2024-02-19T21:12:09.339398+0000","last_active":"2024-02-19T21:13:18.349882+0000","last_peered":"2024-02-19T21:13:18.349882+0000","last_clean":"2024-02-19T21:13:18.349882+0000","last_became_active":"2024-02-19T21:12:09.338747+0000","last_became_peered":"2024-02-19T21:12:09.338747+0000","last_unstale":"2024-02-19T21:13:18.349882+0000","last_undegraded":"2024-02-19T21:13:18.349882+0000","last_fullsized":"2024-02-19T21:13:18.349882+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.720583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.720583+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":152,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.914210+0000","last_change":"2024-02-19T21:12:09.346777+0000","last_active":"2024-02-19T21:13:18.914210+0000","last_peered":"2024-02-19T21:13:18.914210+0000","last_clean":"2024-02-19T21:13:18.914210+0000","last_became_active":"2024-02-19T21:12:09.346258+0000","last_became_peered":"2024-02-19T21:12:09.346258+0000","last_unstale":"2024-02-19T21:13:18.914210+0000","last_undegraded":"2024-02-19T21:13:18.914210+0000","last_fullsized":"2024-02-19T21:13:18.914210+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.407543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.407543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":283,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.350557+0000","last_change":"2024-02-19T21:12:09.331143+0000","last_active":"2024-02-19T21:13:18.350557+0000","last_peered":"2024-02-19T21:13:18.350557+0000","last_clean":"2024-02-19T21:13:18.350557+0000","last_became_active":"2024-02-19T21:12:09.330530+0000","last_became_peered":"2024-02-19T21:12:09.330530+0000","last_unstale":"2024-02-19T21:13:18.350557+0000","last_undegraded":"2024-02-19T21:13:18.350557+0000","last_fullsized":"2024-02-19T21:13:18.350557+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:33.559363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:33.559363+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":306,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-02-19T21:13:18.351900+0000","last_change":"2024-02-19T21:12:09.345873+0000","last_active":"2024-02-19T21:13:18.351900+0000","last_peered":"2024-02-19T21:13:18.351900+0000","last_clean":"2024-02-19T21:13:18.351900+0000","last_became_active":"2024-02-19T21:12:09.345500+0000","last_became_peered":"2024-02-19T21:12:09.345500+0000","last_unstale":"2024-02-19T21:13:18.351900+0000","last_undegraded":"2024-02-19T21:13:18.351900+0000","last_fullsized":"2024-02-19T21:13:18.351900+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.764658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.764658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":231,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":230,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":229,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":24,"acting":24,"num_store_stats":5},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1102,"num_read_kb":859,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":7558,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":712,"ondisk_log_size":712,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":714,"num_read_kb":709,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":126,"acting":126,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":15,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266835,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313684,"kb_used_data":3316,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431468,"statfs":{"total":95995035648,"available":95673823232,"internally_reserved":0,"allocated":3395584,"data_stored":1744773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.86399999999999999},"min":{"1min":0.30399999999999999,"5min":0.223,"15min":0.215},"max":{"1min":0.66000000000000003,"5min":0.79200000000000004,"15min":89.456999999999994},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.47699999999999998,"15min":0.78300000000000003},"min":{"1min":0.29599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.752,"5min":0.96999999999999997,"15min":70.451999999999998},"last":0.58499999999999996}]},{"osd":1,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48599999999999999,"15min":0.69599999999999995},"min":{"1min":0.36299999999999999,"5min":0.28299999999999997,"15min":0.28199999999999997},"max":{"1min":0.61399999999999999,"5min":0.85099999999999998,"15min":34.273000000000003},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46899999999999997,"15min":0.93100000000000005},"min":{"1min":0.29499999999999998,"5min":0.28299999999999997,"15min":0.254},"max":{"1min":0.63500000000000001,"5min":0.79200000000000004,"15min":89.594999999999999},"last":0.70899999999999996}]},{"osd":2,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.46999999999999997,"15min":0.93700000000000006},"min":{"1min":0.33000000000000002,"5min":0.28100000000000003,"15min":0.24299999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":104.413},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.495,"15min":0.94699999999999995},"min":{"1min":0.375,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.624,"5min":0.76400000000000001,"15min":104.268},"last":0.55100000000000005}]},{"osd":3,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.95399999999999996},"min":{"1min":0.33400000000000002,"5min":0.29099999999999998,"15min":0.254},"max":{"1min":0.62,"5min":1.8939999999999999,"15min":105.3},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.50700000000000001,"15min":0.59999999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.68000000000000005,"5min":0.879,"15min":13.243},"last":0.501}]},{"osd":4,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.42499999999999999,"15min":0.46000000000000002},"min":{"1min":0.30599999999999999,"5min":0.27400000000000002,"15min":0.24299999999999999},"max":{"1min":0.54400000000000004,"5min":0.69299999999999995,"15min":1.373},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.46000000000000002,"15min":0.47499999999999998},"min":{"1min":0.29199999999999998,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":0.68300000000000005,"5min":0.751,"15min":1.0740000000000001},"last":0.52200000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.443,"15min":0.45600000000000002},"min":{"1min":0.30299999999999999,"5min":0.28599999999999998,"15min":0.252},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.433},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48599999999999999,"15min":0.504},"min":{"1min":0.35499999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.755,"5min":0.79300000000000004,"15min":0.85899999999999999},"last":0.48199999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:12:30 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.51600000000000001},"min":{"1min":0.33700000000000002,"5min":0.30199999999999999,"15min":0.28799999999999998},"max":{"1min":0.71699999999999997,"5min":0.76700000000000002,"15min":1.0049999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52200000000000002,"15min":0.54900000000000004},"min":{"1min":0.35899999999999999,"5min":0.33800000000000002,"15min":0.33700000000000002},"max":{"1min":0.72399999999999998,"5min":0.75900000000000001,"15min":1.262},"last":0.747}]}]},{"osd":6,"up_from":0,"seq":193273528473,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":132,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313272,"kb_used_data":3336,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431880,"statfs":{"total":95995035648,"available":95674245120,"internally_reserved":0,"allocated":3416064,"data_stored":1745089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.33300000000000002,"15min":0.54500000000000004},"min":{"1min":0.19700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.38900000000000001,"5min":2.0379999999999998,"15min":27.777000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34799999999999998,"15min":0.57399999999999995},"min":{"1min":0.215,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.51900000000000002,"5min":1.9870000000000001,"15min":27.631},"last":0.372}]},{"osd":2,"last update":"Mon Feb 19 21:12:34 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.371,"15min":0.70599999999999996},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":27.798999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.38900000000000001,"15min":0.72299999999999998},"min":{"1min":0.24299999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.089,"5min":2.089,"15min":27.606999999999999},"last":0.435}]},{"osd":3,"last update":"Mon Feb 19 21:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.374,"15min":0.56599999999999995},"min":{"1min":0.253,"5min":0.217,"15min":0.217},"max":{"1min":0.499,"5min":1.7450000000000001,"15min":15.574999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.41099999999999998,"15min":0.66200000000000003},"min":{"1min":0.31,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.622,"5min":2.0070000000000001,"15min":27.82},"last":0.45700000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:12:15 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.501,"15min":0.65000000000000002},"min":{"1min":0.27800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.59199999999999997,"5min":1.149,"15min":9.6430000000000007},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53200000000000003,"15min":0.752},"min":{"1min":0.34799999999999998,"5min":0.308,"15min":0.308},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":27.707000000000001},"last":0.61599999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:12:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52700000000000002,"15min":0.73899999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98599999999999999,"5min":1.048,"15min":27.529},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.51800000000000002,"15min":0.67500000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67400000000000004,"5min":1.1819999999999999,"15min":9.6799999999999997},"last":0.623}]},{"osd":6,"last update":"Mon Feb 19 21:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52600000000000002,"15min":0.65800000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70399999999999996,"5min":1.1659999999999999,"15min":9.6989999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.60099999999999998,"15min":0.78600000000000003},"min":{"1min":0.40799999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75900000000000001,"5min":3.4359999999999999,"15min":27.733000000000001},"last":0.63500000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:12:34 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.56799999999999995,"15min":0.77500000000000002},"min":{"1min":0.40000000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":3.4670000000000001,"5min":3.4670000000000001,"15min":27.576000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.58299999999999996,"15min":0.80500000000000005},"min":{"1min":0.42099999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.71599999999999997,"5min":2.0169999999999999,"15min":27.756},"last":0.66700000000000004}]}]},{"osd":0,"up_from":0,"seq":34359738550,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411499,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310492,"kb_used_data":3304,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93434660,"statfs":{"total":95995035648,"available":95677091840,"internally_reserved":0,"allocated":3383296,"data_stored":1743705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:12:33 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.377,"15min":0.53600000000000003},"min":{"1min":0.22900000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.54400000000000004,"5min":3.593,"15min":26.690999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.35899999999999999,"15min":0.59099999999999997},"min":{"1min":0.26100000000000001,"5min":0.23300000000000001,"15min":0.214},"max":{"1min":0.56799999999999995,"5min":1.488,"15min":24.315000000000001},"last":0.28299999999999997}]},{"osd":1,"last update":"Mon Feb 19 21:12:33 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.42199999999999999,"15min":0.70299999999999996},"min":{"1min":0.249,"5min":0.20599999999999999,"15min":0.20100000000000001},"max":{"1min":0.47199999999999998,"5min":7.0010000000000003,"15min":28.882000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.42299999999999999,"15min":0.68200000000000005},"min":{"1min":0.27000000000000002,"5min":0.23899999999999999,"15min":0.22500000000000001},"max":{"1min":0.60899999999999999,"5min":3.3010000000000002,"15min":28.382999999999999},"last":0.31900000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.53700000000000003,"15min":0.71199999999999997},"min":{"1min":0.27800000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68899999999999995,"5min":12.75,"15min":28.742000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.44900000000000001,"15min":0.63300000000000001},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.25700000000000001},"max":{"1min":0.58399999999999996,"5min":3.6499999999999999,"15min":26.719999999999999},"last":0.38}]},{"osd":4,"last update":"Mon Feb 19 21:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.60099999999999998,"15min":0.69699999999999995},"min":{"1min":0.36299999999999999,"5min":0.33400000000000002,"15min":0.30599999999999999},"max":{"1min":0.755,"5min":3.4039999999999999,"15min":24.356000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.67100000000000004,"15min":0.71999999999999997},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.32400000000000001},"max":{"1min":0.72799999999999998,"5min":7.1340000000000003,"15min":20.707000000000001},"last":0.51100000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57799999999999996,"15min":0.68000000000000005},"min":{"1min":0.36399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81599999999999995,"5min":3.3090000000000002,"15min":9.9499999999999993},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60899999999999999,"15min":0.79100000000000004},"min":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.38400000000000001},"max":{"1min":0.77500000000000002,"5min":3.2690000000000001,"15min":26.606000000000002},"last":0.437}]},{"osd":6,"last update":"Mon Feb 19 21:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.58299999999999996,"15min":0.79800000000000004},"min":{"1min":0.438,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.876,"5min":1.4510000000000001,"15min":24.381},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.73399999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80200000000000005,"5min":0.93600000000000005,"15min":28.706},"last":0.47299999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:12:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.63200000000000001,"15min":0.77800000000000002},"min":{"1min":0.441,"5min":0.42199999999999999,"15min":0.34200000000000003},"max":{"1min":0.77600000000000002,"5min":3.3359999999999999,"15min":26.658000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59999999999999998,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82099999999999995,"5min":1.5109999999999999,"15min":24.402000000000001},"last":0.504}]}]},{"osd":3,"up_from":25,"seq":107374182567,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310028,"kb_used_data":3744,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435124,"statfs":{"total":95995035648,"available":95677566976,"internally_reserved":0,"allocated":3833856,"data_stored":2011996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.71599999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.156},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":77.662000000000006},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.35499999999999998,"15min":0.49399999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.19700000000000001},"max":{"1min":0.60199999999999998,"5min":0.60999999999999999,"15min":17.456},"last":0.30599999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.52600000000000002},"min":{"1min":0.255,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":0.66300000000000003,"5min":6.9299999999999997,"15min":17.327999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.42499999999999999,"15min":0.50600000000000001},"min":{"1min":0.20699999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.95599999999999996,"5min":7.3620000000000001,"15min":15.004},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.376,"15min":0.66300000000000003},"min":{"1min":0.27300000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":56.890999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.36599999999999999,"15min":0.64400000000000002},"min":{"1min":0.246,"5min":0.24399999999999999,"15min":0.20899999999999999},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":44.600999999999999},"last":1.3200000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:12:18 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52500000000000002,"15min":0.63400000000000001},"min":{"1min":0.33400000000000002,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":12.847},"last":1.3340000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.52300000000000002,"15min":0.70599999999999996},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72299999999999998,"5min":0.94299999999999995,"15min":26.393999999999998},"last":1.3560000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.61799999999999999,"15min":0.68400000000000005},"min":{"1min":0.39100000000000001,"5min":0.34100000000000003,"15min":0.313},"max":{"1min":0.68300000000000005,"5min":9.2360000000000007,"15min":17.401},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54300000000000004,"15min":0.65400000000000003},"min":{"1min":0.35999999999999999,"5min":0.313,"15min":0.313},"max":{"1min":1.3660000000000001,"5min":1.9690000000000001,"15min":12.882},"last":1.397}]},{"osd":6,"last update":"Mon Feb 19 21:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56000000000000005,"15min":0.65200000000000002},"min":{"1min":0.38,"5min":0.34799999999999998,"15min":0.34300000000000003},"max":{"1min":0.69999999999999996,"5min":1.9790000000000001,"15min":17.367000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54900000000000004,"15min":0.66200000000000003},"min":{"1min":0.377,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":0.89400000000000002,"5min":0.95099999999999996,"15min":12.904},"last":1.3660000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:12:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.55900000000000005,"15min":0.65500000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.038,"5min":1.038,"15min":17.428999999999998},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57199999999999995,"15min":0.80000000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.34499999999999997},"max":{"1min":1.111,"5min":1.111,"15min":44.378},"last":1.3879999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986339,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309580,"kb_used_data":3308,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93435572,"statfs":{"total":95995035648,"available":95678025728,"internally_reserved":0,"allocated":3387392,"data_stored":1743531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.54700000000000004,"15min":0.63900000000000001},"min":{"1min":0.28199999999999997,"5min":0.23599999999999999,"15min":0.22500000000000001},"max":{"1min":0.68300000000000005,"5min":8.0530000000000008,"15min":19.878},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.442,"5min":0.49199999999999999,"15min":0.85799999999999998},"min":{"1min":0.32300000000000001,"5min":0.22500000000000001,"15min":0.20499999999999999},"max":{"1min":0.61099999999999999,"5min":3.3530000000000002,"15min":70.063000000000002},"last":0.623}]},{"osd":1,"last update":"Mon Feb 19 21:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46700000000000003,"15min":0.63600000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":26.510999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.48599999999999999,"15min":0.68200000000000005},"min":{"1min":0.32100000000000001,"5min":0.312,"15min":0.23200000000000001},"max":{"1min":0.61899999999999999,"5min":0.82999999999999996,"15min":25.949000000000002},"last":0.54800000000000004}]},{"osd":2,"last update":"Mon Feb 19 21:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.57599999999999996,"15min":0.71899999999999997},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.67400000000000004,"5min":8.1240000000000006,"15min":37.320999999999998},"last":0.497},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51500000000000001,"15min":1.0600000000000001},"min":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.253},"max":{"1min":0.66500000000000004,"5min":2.9969999999999999,"15min":69.936999999999998},"last":0.67100000000000004}]},{"osd":3,"last update":"Mon Feb 19 21:12:21 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.53400000000000003,"15min":0.91200000000000003},"min":{"1min":0.38100000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.69399999999999995,"5min":3.323,"15min":69.793000000000006},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52000000000000002,"15min":0.76200000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.76500000000000001,"5min":0.81799999999999995,"15min":17.986999999999998},"last":0.56100000000000005}]},{"osd":5,"last update":"Mon Feb 19 21:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.218},"max":{"1min":0.78800000000000003,"5min":1.036,"15min":3.2799999999999998},"last":0.378},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.48199999999999998,"15min":0.46300000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.23899999999999999},"max":{"1min":0.745,"5min":1.121,"15min":4.4340000000000002},"last":0.47799999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.57399999999999995,"15min":0.47599999999999998},"min":{"1min":0.29199999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.78700000000000003,"5min":8.0969999999999995,"15min":8.0969999999999995},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.51800000000000002},"min":{"1min":0.35599999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.4550000000000001,"5min":3.0270000000000001,"15min":4.5609999999999999},"last":0.51000000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56299999999999994,"15min":0.51000000000000001},"min":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.71699999999999997,"5min":3.2909999999999999,"15min":6.0730000000000004},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.505,"5min":0.55200000000000005,"15min":0.51200000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.26400000000000001},"max":{"1min":0.72699999999999998,"5min":1.0209999999999999,"15min":3.202},"last":0.61299999999999999}]}]},{"osd":5,"up_from":37,"seq":158913790110,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300972,"kb_used_data":3324,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444180,"statfs":{"total":95995035648,"available":95686840320,"internally_reserved":0,"allocated":3403776,"data_stored":1745220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.45800000000000002,"15min":1.8779999999999999},"min":{"1min":0.26200000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.64400000000000002,"5min":0.71699999999999997,"15min":276.47899999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.48799999999999999,"15min":0.90100000000000002},"min":{"1min":0.27400000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.58099999999999996,"5min":0.95299999999999996,"15min":52.887},"last":0.64700000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.46999999999999997,"15min":1.266},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.23499999999999999},"max":{"1min":0.59599999999999997,"5min":1.599,"15min":120.011},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.48899999999999999,"15min":1.02},"min":{"1min":0.28599999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63100000000000001,"5min":0.97699999999999998,"15min":118.105},"last":0.71299999999999997}]},{"osd":2,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47799999999999998,"15min":0.99399999999999999},"min":{"1min":0.32900000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.65600000000000003,"5min":0.92300000000000004,"15min":82.242000000000004},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.55700000000000005,"15min":1.5129999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65300000000000002,"5min":4.5019999999999998,"15min":232.84999999999999},"last":0.46600000000000003}]},{"osd":3,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52700000000000002,"15min":1.0649999999999999},"min":{"1min":0.35299999999999998,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":0.71199999999999997,"5min":4.4640000000000004,"15min":109.70099999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52400000000000002,"15min":0.91100000000000003},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":65.519000000000005},"last":0.56399999999999995}]},{"osd":4,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.432,"15min":0.42099999999999999},"min":{"1min":0.26900000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66200000000000003,"5min":1.0129999999999999,"15min":1.861},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.44,"15min":0.433},"min":{"1min":0.255,"5min":0.255,"15min":0.248},"max":{"1min":0.61599999999999999,"5min":0.72499999999999998,"15min":1.9730000000000001},"last":0.60199999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47299999999999998,"15min":0.45500000000000002},"min":{"1min":0.31,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69599999999999995,"5min":1.0289999999999999,"15min":1.823},"last":0.755},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47199999999999998,"15min":0.46300000000000002},"min":{"1min":0.30499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.53200000000000003}]},{"osd":7,"last update":"Mon Feb 19 21:12:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.497,"15min":0.5},"min":{"1min":0.34000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63300000000000001,"5min":1.052,"15min":1.8879999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51200000000000001,"15min":0.50700000000000001},"min":{"1min":0.34399999999999997,"5min":0.315,"15min":0.27800000000000002},"max":{"1min":0.64100000000000001,"5min":0.871,"15min":0.96799999999999997},"last":0.66300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:13:24.745 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool set default.rgw.meta pgp_num 32 2024-02-19T21:13:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:23.268428+0000 mgr.y (mgr.14550) 500 : cluster [DBG] pgmap v849: 246 pgs: 64 unknown, 182 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-02-19T21:13:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:23.379931+0000 mon.a (mon.0) 2973 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-97-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:13:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:23.380030+0000 mon.a (mon.0) 2974 : cluster [DBG] osdmap e567: 8 total, 8 up, 6 in 2024-02-19T21:13:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:23.398272+0000 mon.b (mon.1) 723 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:23.398655+0000 mon.a (mon.0) 2975 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:23.878290+0000 mgr.y (mgr.14550) 501 : audit [DBG] from='client.50186 v1:172.21.15.163:0/3787012232' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:13:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:24.105941+0000 mon.a (mon.0) 2976 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:24.334622+0000 mon.a (mon.0) 2977 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]': finished 2024-02-19T21:13:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:24.334704+0000 mon.a (mon.0) 2978 : cluster [DBG] osdmap e568: 8 total, 8 up, 6 in 2024-02-19T21:13:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:24.337727+0000 mon.b (mon.1) 724 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-97", "overlaypool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:24.344157+0000 mon.a (mon.0) 2979 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-97", "overlaypool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:24.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.937+0000 7f5afabfb700 1 -- 172.21.15.163:0/3337922379 >> v1:172.21.15.163:6790/0 conn(0x7f5af40fc870 legacy=0x7f5af40fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:24.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.937+0000 7f5afabfb700 1 -- 172.21.15.163:0/3337922379 shutdown_connections 2024-02-19T21:13:24.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.937+0000 7f5afabfb700 1 -- 172.21.15.163:0/3337922379 >> 172.21.15.163:0/3337922379 conn(0x7f5af40f2470 msgr2=0x7f5af40f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:24.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.937+0000 7f5afabfb700 1 -- 172.21.15.163:0/3337922379 shutdown_connections 2024-02-19T21:13:24.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.937+0000 7f5afabfb700 1 -- 172.21.15.163:0/3337922379 wait complete. 2024-02-19T21:13:24.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.937+0000 7f5afabfb700 1 librados: starting msgr at 2024-02-19T21:13:24.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.937+0000 7f5afabfb700 1 librados: starting objecter 2024-02-19T21:13:24.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.937+0000 7f5afabfb700 1 Processor -- start 2024-02-19T21:13:24.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.937+0000 7f5afabfb700 1 -- start start 2024-02-19T21:13:24.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.937+0000 7f5afabfb700 1 librados: setting wanted keys 2024-02-19T21:13:24.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.937+0000 7f5afabfb700 1 librados: calling monclient init 2024-02-19T21:13:24.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5afabfb700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5af40fc100 con 0x7f5af40fc870 2024-02-19T21:13:24.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5afabfb700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5af4118680 con 0x7f5af41031b0 2024-02-19T21:13:24.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5afabfb700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5af4119860 con 0x7f5af40ff670 2024-02-19T21:13:24.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af3fff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f5af40ff670 0x7f5af4112820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:47980/0 (socket says 172.21.15.163:47980) 2024-02-19T21:13:24.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af8997700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f5af40fc870 0x7f5af40fb510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:53438/0 (socket says 172.21.15.163:53438) 2024-02-19T21:13:24.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af3fff700 1 -- 172.21.15.163:0/718651683 learned_addr learned my addr 172.21.15.163:0/718651683 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:24.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 10 client.?.objecter ms_handle_connect 0x7f5af40ff670 2024-02-19T21:13:24.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:24.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 10 client.?.objecter ms_handle_connect 0x7f5af40fc870 2024-02-19T21:13:24.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:24.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 10 client.?.objecter ms_handle_connect 0x7f5af41031b0 2024-02-19T21:13:24.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:24.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494278868 0 0) 0x7f5af4119860 con 0x7f5af40ff670 2024-02-19T21:13:24.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae40033f0 con 0x7f5af40ff670 2024-02-19T21:13:24.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596612810 0 0) 0x7f5af40fc100 con 0x7f5af40fc870 2024-02-19T21:13:24.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5af4119860 con 0x7f5af40fc870 2024-02-19T21:13:24.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965053154 0 0) 0x7f5af4118680 con 0x7f5af41031b0 2024-02-19T21:13:24.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5af40fc100 con 0x7f5af41031b0 2024-02-19T21:13:24.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1631901187 0 0) 0x7f5ae40033f0 con 0x7f5af40ff670 2024-02-19T21:13:24.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5af4118680 con 0x7f5af40ff670 2024-02-19T21:13:24.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5ae00029b0 con 0x7f5af40ff670 2024-02-19T21:13:24.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 468055402 0 0) 0x7f5af4119860 con 0x7f5af40fc870 2024-02-19T21:13:24.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ae40033f0 con 0x7f5af40fc870 2024-02-19T21:13:24.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5ae8002c60 con 0x7f5af40fc870 2024-02-19T21:13:24.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 683129064 0 0) 0x7f5af40fc100 con 0x7f5af41031b0 2024-02-19T21:13:24.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5af4119860 con 0x7f5af41031b0 2024-02-19T21:13:24.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5aec001bb0 con 0x7f5af41031b0 2024-02-19T21:13:24.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2902521459 0 0) 0x7f5af4118680 con 0x7f5af40ff670 2024-02-19T21:13:24.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 >> v1:172.21.15.193:6789/0 conn(0x7f5af41031b0 legacy=0x7f5af4115f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:24.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 >> v1:172.21.15.163:6789/0 conn(0x7f5af40fc870 legacy=0x7f5af40fb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:24.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af411aa60 con 0x7f5af40ff670 2024-02-19T21:13:24.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.938+0000 7f5afabfb700 1 -- 172.21.15.163:0/718651683 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af41176d0 con 0x7f5af40ff670 2024-02-19T21:13:24.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.939+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f5ae0002d60 con 0x7f5af40ff670 2024-02-19T21:13:24.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.939+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5ae0005c90 con 0x7f5af40ff670 2024-02-19T21:13:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.939+0000 7f5afabfb700 10 client.50192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.939+0000 7f5afabfb700 1 -- 172.21.15.163:0/718651683 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af4117b40 con 0x7f5af40ff670 2024-02-19T21:13:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.939+0000 7f5afabfb700 1 librados: init done 2024-02-19T21:13:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.940+0000 7f5afabfb700 1 -- 172.21.15.163:0/718651683 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af40fb190 con 0x7f5af40ff670 2024-02-19T21:13:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.940+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f5ae0018bc0 con 0x7f5af40ff670 2024-02-19T21:13:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.941+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(568..568 src has 434..568) v4 ==== 8915+0+0 (unknown 3289136409 0 0) 0x7f5ae0079100 con 0x7f5af40ff670 2024-02-19T21:13:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.941+0000 7f5af1ffb700 10 client.50192.objecter ms_dispatch 0x7f5af4001550 osd_map(568..568 src has 434..568) v4 2024-02-19T21:13:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.941+0000 7f5af1ffb700 3 client.50192.objecter handle_osd_map got epochs [568,568] > 0 2024-02-19T21:13:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.941+0000 7f5af1ffb700 3 client.50192.objecter handle_osd_map decoding full epoch 568 2024-02-19T21:13:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.941+0000 7f5af1ffb700 20 client.50192.objecter dump_active .. 0 homeless 2024-02-19T21:13:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.941+0000 7f5af1ffb700 10 client.50192.objecter ms_handle_connect 0x7f5ae4063d30 2024-02-19T21:13:24.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:24.945+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5ae004c310 con 0x7f5af40ff670 2024-02-19T21:13:24.951 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:24.954 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:24.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:25.055 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:25.083 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:25.121 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:25.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:25.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:25.137 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:25.137+0000 7f5afabfb700 1 -- 172.21.15.163:0/718651683 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f5af40f80a0 con 0x7f5af40ff670 2024-02-19T21:13:25.230 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:25.263 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:25.284 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:25.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:25.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:25.346+0000 7f5af1ffb700 1 -- 172.21.15.163:0/718651683 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v569) v1 ==== 97+0+0 (unknown 553416395 0 0) 0x7f5ae004a720 con 0x7f5af40ff670 2024-02-19T21:13:25.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:25.350+0000 7f5afabfb700 10 librados: watch_flush enter 2024-02-19T21:13:25.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:25.350+0000 7f5afabfb700 10 librados: watch_flush exit 2024-02-19T21:13:25.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:25.350+0000 7f5afabfb700 20 client.50192.objecter shutdown clearing up homeless session... 2024-02-19T21:13:25.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:25.350+0000 7f5afabfb700 10 client.50192.objecter successfully canceled tick 2024-02-19T21:13:25.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:25.350+0000 7f5afabfb700 1 -- 172.21.15.163:0/718651683 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5ae4063d30 legacy=0x7f5ae40661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:25.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:25.350+0000 7f5afabfb700 1 -- 172.21.15.163:0/718651683 >> v1:172.21.15.163:6790/0 conn(0x7f5af40ff670 legacy=0x7f5af4112820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:25.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:25.350+0000 7f5afabfb700 1 -- 172.21.15.163:0/718651683 shutdown_connections 2024-02-19T21:13:25.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:25.350+0000 7f5afabfb700 1 -- 172.21.15.163:0/718651683 >> 172.21.15.163:0/718651683 conn(0x7f5af40f2470 msgr2=0x7f5af4106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:25.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:25.350+0000 7f5afabfb700 1 -- 172.21.15.163:0/718651683 shutdown_connections 2024-02-19T21:13:25.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:25.350+0000 7f5afabfb700 1 -- 172.21.15.163:0/718651683 wait complete. 2024-02-19T21:13:25.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:25.350+0000 7f5afabfb700 1 librados: shutdown 2024-02-19T21:13:25.351 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:13:25.388 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:25.418 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:25.439 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:25.442 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:25.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:25.543 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:25.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:25.138660+0000 mon.c (mon.2) 409 : audit [INF] from='client.? v1:172.21.15.163:0/718651683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:13:25.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:25.139000+0000 mon.a (mon.0) 2980 : audit [INF] from='client.50192 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:13:25.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:25.341327+0000 mon.a (mon.0) 2981 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:13:25.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:25.345213+0000 mon.a (mon.0) 2982 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-97", "overlaypool": "test-rados-api-smithi163-83928-97-cache"}]': finished 2024-02-19T21:13:25.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:25.345288+0000 mon.a (mon.0) 2983 : audit [INF] from='client.50192 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:13:25.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:25.345330+0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e569: 8 total, 8 up, 6 in 2024-02-19T21:13:25.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:25.346538+0000 mon.b (mon.1) 725 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-97-cache", "mode": "writeback"}]: dispatch 2024-02-19T21:13:25.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:25.346814+0000 mon.a (mon.0) 2985 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-97-cache", "mode": "writeback"}]: dispatch 2024-02-19T21:13:25.571 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:25.714 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:25.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:25.138660+0000 mon.c (mon.2) 409 : audit [INF] from='client.? v1:172.21.15.163:0/718651683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:13:25.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:25.139000+0000 mon.a (mon.0) 2980 : audit [INF] from='client.50192 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:13:25.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:25.341327+0000 mon.a (mon.0) 2981 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:13:25.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:25.345213+0000 mon.a (mon.0) 2982 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-97", "overlaypool": "test-rados-api-smithi163-83928-97-cache"}]': finished 2024-02-19T21:13:25.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:25.345288+0000 mon.a (mon.0) 2983 : audit [INF] from='client.50192 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:13:25.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:25.345330+0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e569: 8 total, 8 up, 6 in 2024-02-19T21:13:25.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:25.346538+0000 mon.b (mon.1) 725 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-97-cache", "mode": "writeback"}]: dispatch 2024-02-19T21:13:25.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:25.346814+0000 mon.a (mon.0) 2985 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-97-cache", "mode": "writeback"}]: dispatch 2024-02-19T21:13:25.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:25.138660+0000 mon.c (mon.2) 409 : audit [INF] from='client.? v1:172.21.15.163:0/718651683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:13:25.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:25.139000+0000 mon.a (mon.0) 2980 : audit [INF] from='client.50192 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:13:25.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:25.341327+0000 mon.a (mon.0) 2981 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:13:25.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:25.345213+0000 mon.a (mon.0) 2982 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-97", "overlaypool": "test-rados-api-smithi163-83928-97-cache"}]': finished 2024-02-19T21:13:25.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:25.345288+0000 mon.a (mon.0) 2983 : audit [INF] from='client.50192 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:13:25.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:25.345330+0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e569: 8 total, 8 up, 6 in 2024-02-19T21:13:25.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:25.346538+0000 mon.b (mon.1) 725 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-97-cache", "mode": "writeback"}]: dispatch 2024-02-19T21:13:25.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:25.346814+0000 mon.a (mon.0) 2985 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-97-cache", "mode": "writeback"}]: dispatch 2024-02-19T21:13:26.149 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:26.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:26.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:26.155 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:26.254 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:26.287 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:26.308 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:26.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:26.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:26.411 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:26.440 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:26.460 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:26.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:26.564 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:26.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:25.269573+0000 mgr.y (mgr.14550) 502 : cluster [DBG] pgmap v852: 246 pgs: 246 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:26.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:25.606476+0000 osd.4 (osd.4) 203 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:26.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:25.606807+0000 osd.4 (osd.4) 204 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:26.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:26.342825+0000 mon.a (mon.0) 2986 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:13:26.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:26.347323+0000 mon.a (mon.0) 2987 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-97-cache", "mode": "writeback"}]': finished 2024-02-19T21:13:26.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:26.347396+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e570: 8 total, 8 up, 6 in 2024-02-19T21:13:26.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:26.367716+0000 mon.b (mon.1) 726 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-97"}]: dispatch 2024-02-19T21:13:26.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:26.368090+0000 mon.a (mon.0) 2989 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-97"}]: dispatch 2024-02-19T21:13:26.592 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:26.612 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:26.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:26.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:25.269573+0000 mgr.y (mgr.14550) 502 : cluster [DBG] pgmap v852: 246 pgs: 246 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:26.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:25.606476+0000 osd.4 (osd.4) 203 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:26.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:25.606807+0000 osd.4 (osd.4) 204 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:26.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:26.342825+0000 mon.a (mon.0) 2986 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:13:26.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:26.347323+0000 mon.a (mon.0) 2987 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-97-cache", "mode": "writeback"}]': finished 2024-02-19T21:13:26.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:26.347396+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e570: 8 total, 8 up, 6 in 2024-02-19T21:13:26.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:26.367716+0000 mon.b (mon.1) 726 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-97"}]: dispatch 2024-02-19T21:13:26.650 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:26.368090+0000 mon.a (mon.0) 2989 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-97"}]: dispatch 2024-02-19T21:13:26.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:25.269573+0000 mgr.y (mgr.14550) 502 : cluster [DBG] pgmap v852: 246 pgs: 246 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:26.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:25.606476+0000 osd.4 (osd.4) 203 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:26.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:25.606807+0000 osd.4 (osd.4) 204 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:26.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:26.342825+0000 mon.a (mon.0) 2986 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:13:26.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:26.347323+0000 mon.a (mon.0) 2987 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-97-cache", "mode": "writeback"}]': finished 2024-02-19T21:13:26.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:26.347396+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e570: 8 total, 8 up, 6 in 2024-02-19T21:13:26.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:26.367716+0000 mon.b (mon.1) 726 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-97"}]: dispatch 2024-02-19T21:13:26.651 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:26.368090+0000 mon.a (mon.0) 2989 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-97"}]: dispatch 2024-02-19T21:13:26.716 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:13:26.745 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:26.868 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:13:26.871 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:26.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:13:26.972 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:27.000 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:27.149 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:27.152 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:27.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:27.253 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:27.260 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:13:27.260 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:13:27.260 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-02-19T21:13:27.260 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:13:27.285 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:27.306 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:27.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:27.410 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:27.438 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:27.459 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:27.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:27.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:27.562 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:27.592 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:27.790 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:27.792 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:27.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:27.894 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:27.921 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:27.940 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:27.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:28.044 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:13:28.070 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:28.070 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:13:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:13:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:13:28.128 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:13:28.268 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:13:28.272 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:28.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:13:28.373 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:13:28.399 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:28.472 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:13:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:28.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:13:28.585 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:13:28.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:27.270162+0000 mgr.y (mgr.14550) 503 : cluster [DBG] pgmap v855: 246 pgs: 246 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:28.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:27.348094+0000 mon.a (mon.0) 2990 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-97"}]': finished 2024-02-19T21:13:28.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:27.348167+0000 mon.a (mon.0) 2991 : cluster [DBG] osdmap e571: 8 total, 8 up, 6 in 2024-02-19T21:13:28.610 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:27.349920+0000 mon.b (mon.1) 727 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:28.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:27.350285+0000 mon.a (mon.0) 2992 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:28.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:27.270162+0000 mgr.y (mgr.14550) 503 : cluster [DBG] pgmap v855: 246 pgs: 246 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:28.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:27.348094+0000 mon.a (mon.0) 2990 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-97"}]': finished 2024-02-19T21:13:28.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:27.348167+0000 mon.a (mon.0) 2991 : cluster [DBG] osdmap e571: 8 total, 8 up, 6 in 2024-02-19T21:13:28.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:27.349920+0000 mon.b (mon.1) 727 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:28.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:27.350285+0000 mon.a (mon.0) 2992 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:28.612 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:28.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:27.270162+0000 mgr.y (mgr.14550) 503 : cluster [DBG] pgmap v855: 246 pgs: 246 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:28.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:27.348094+0000 mon.a (mon.0) 2990 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-97"}]': finished 2024-02-19T21:13:28.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:27.348167+0000 mon.a (mon.0) 2991 : cluster [DBG] osdmap e571: 8 total, 8 up, 6 in 2024-02-19T21:13:28.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:27.349920+0000 mon.b (mon.1) 727 : audit [INF] from='client.? v1:172.21.15.163:0/472848668' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:28.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:27.350285+0000 mon.a (mon.0) 2992 : audit [INF] from='client.48988 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]: dispatch 2024-02-19T21:13:29.377 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-19T21:13:29.565 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:13:29.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:29.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:13:29.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:27.739763+0000 osd.1 (osd.1) 262 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:29.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:27.740001+0000 osd.1 (osd.1) 263 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:29.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:28.345811+0000 mon.a (mon.0) 2993 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:13:29.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:28.352755+0000 mon.a (mon.0) 2994 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]': finished 2024-02-19T21:13:29.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:28.352827+0000 mon.a (mon.0) 2995 : cluster [DBG] osdmap e572: 8 total, 8 up, 6 in 2024-02-19T21:13:29.618 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:27.739763+0000 osd.1 (osd.1) 262 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:29.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:27.740001+0000 osd.1 (osd.1) 263 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:29.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:28.345811+0000 mon.a (mon.0) 2993 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:13:29.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:28.352755+0000 mon.a (mon.0) 2994 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]': finished 2024-02-19T21:13:29.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:28.352827+0000 mon.a (mon.0) 2995 : cluster [DBG] osdmap e572: 8 total, 8 up, 6 in 2024-02-19T21:13:29.671 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:29.702 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:27.739763+0000 osd.1 (osd.1) 262 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:29.702 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:27.740001+0000 osd.1 (osd.1) 263 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:29.702 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:28.345811+0000 mon.a (mon.0) 2993 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:13:29.702 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:28.352755+0000 mon.a (mon.0) 2994 : audit [INF] from='client.48988 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-97", "tierpool": "test-rados-api-smithi163-83928-97-cache"}]': finished 2024-02-19T21:13:29.702 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:28.352827+0000 mon.a (mon.0) 2995 : cluster [DBG] osdmap e572: 8 total, 8 up, 6 in 2024-02-19T21:13:29.704 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:29.730 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:29.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:29.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:29.834 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:29.860 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:29.884 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:29.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:29.991 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:30.018 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:30.037 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:30.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:30.141 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:30.172 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:30.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.175+0000 7ffa956bd700 1 -- 172.21.15.163:0/3766822102 >> v1:172.21.15.163:6789/0 conn(0x7ffa900fc870 legacy=0x7ffa900fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:30.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.175+0000 7ffa956bd700 1 -- 172.21.15.163:0/3766822102 shutdown_connections 2024-02-19T21:13:30.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.175+0000 7ffa956bd700 1 -- 172.21.15.163:0/3766822102 >> 172.21.15.163:0/3766822102 conn(0x7ffa900f2470 msgr2=0x7ffa900f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:30.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.175+0000 7ffa956bd700 1 -- 172.21.15.163:0/3766822102 shutdown_connections 2024-02-19T21:13:30.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.176+0000 7ffa956bd700 1 -- 172.21.15.163:0/3766822102 wait complete. 2024-02-19T21:13:30.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.176+0000 7ffa956bd700 1 librados: starting msgr at 2024-02-19T21:13:30.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.176+0000 7ffa956bd700 1 librados: starting objecter 2024-02-19T21:13:30.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.176+0000 7ffa956bd700 1 Processor -- start 2024-02-19T21:13:30.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.176+0000 7ffa956bd700 1 -- start start 2024-02-19T21:13:30.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.176+0000 7ffa956bd700 1 librados: setting wanted keys 2024-02-19T21:13:30.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.176+0000 7ffa956bd700 1 librados: calling monclient init 2024-02-19T21:13:30.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa956bd700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa90119350 con 0x7ffa901031b0 2024-02-19T21:13:30.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa956bd700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa9011a530 con 0x7ffa900fc870 2024-02-19T21:13:30.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa956bd700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa9011b710 con 0x7ffa900ff670 2024-02-19T21:13:30.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa8e7fc700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7ffa900ff670 0x7ffa900610f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:48008/0 (socket says 172.21.15.163:48008) 2024-02-19T21:13:30.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa8f7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ffa901031b0 0x7ffa90117c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:53444/0 (socket says 172.21.15.163:53444) 2024-02-19T21:13:30.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa8e7fc700 1 -- 172.21.15.163:0/3881623081 learned_addr learned my addr 172.21.15.163:0/3881623081 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:30.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 10 client.?.objecter ms_handle_connect 0x7ffa900ff670 2024-02-19T21:13:30.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:30.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 10 client.?.objecter ms_handle_connect 0x7ffa901031b0 2024-02-19T21:13:30.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:30.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206813255 0 0) 0x7ffa9011b710 con 0x7ffa900ff670 2024-02-19T21:13:30.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa780033f0 con 0x7ffa900ff670 2024-02-19T21:13:30.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 10 client.?.objecter ms_handle_connect 0x7ffa900fc870 2024-02-19T21:13:30.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:30.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342942790 0 0) 0x7ffa90119350 con 0x7ffa901031b0 2024-02-19T21:13:30.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa9011b710 con 0x7ffa901031b0 2024-02-19T21:13:30.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809622555 0 0) 0x7ffa9011a530 con 0x7ffa900fc870 2024-02-19T21:13:30.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa90119350 con 0x7ffa900fc870 2024-02-19T21:13:30.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 33539816 0 0) 0x7ffa780033f0 con 0x7ffa900ff670 2024-02-19T21:13:30.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa9011a530 con 0x7ffa900ff670 2024-02-19T21:13:30.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa84001990 con 0x7ffa900ff670 2024-02-19T21:13:30.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1550269909 0 0) 0x7ffa9011b710 con 0x7ffa901031b0 2024-02-19T21:13:30.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa780033f0 con 0x7ffa901031b0 2024-02-19T21:13:30.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.177+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa88001bb0 con 0x7ffa901031b0 2024-02-19T21:13:30.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.178+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2982962195 0 0) 0x7ffa90119350 con 0x7ffa900fc870 2024-02-19T21:13:30.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.178+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa9011b710 con 0x7ffa900fc870 2024-02-19T21:13:30.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.178+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa80002050 con 0x7ffa900fc870 2024-02-19T21:13:30.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.178+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1769620347 0 0) 0x7ffa9011a530 con 0x7ffa900ff670 2024-02-19T21:13:30.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.178+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 >> v1:172.21.15.193:6789/0 conn(0x7ffa900fc870 legacy=0x7ffa900609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:30.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.178+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 >> v1:172.21.15.163:6789/0 conn(0x7ffa901031b0 legacy=0x7ffa90117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:30.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.178+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9011c910 con 0x7ffa900ff670 2024-02-19T21:13:30.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.178+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa90119540 con 0x7ffa900ff670 2024-02-19T21:13:30.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.178+0000 7ffa956bd700 10 client.50198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:30.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.178+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa90119a30 con 0x7ffa900ff670 2024-02-19T21:13:30.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.178+0000 7ffa956bd700 1 librados: init done 2024-02-19T21:13:30.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.178+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ffa840024a0 con 0x7ffa900ff670 2024-02-19T21:13:30.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.178+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa84005b20 con 0x7ffa900ff670 2024-02-19T21:13:30.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.179+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ffa7c004f00 con 0x7ffa900ff670 2024-02-19T21:13:30.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.179+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ffa84018a90 con 0x7ffa900ff670 2024-02-19T21:13:30.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(573..573 src has 434..573) v4 ==== 8516+0+0 (unknown 2890771586 0 0) 0x7ffa84078f40 con 0x7ffa900ff670 2024-02-19T21:13:30.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 osd_map(573..573 src has 434..573) v4 2024-02-19T21:13:30.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa77fff700 3 client.50198.objecter handle_osd_map got epochs [573,573] > 0 2024-02-19T21:13:30.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa77fff700 3 client.50198.objecter handle_osd_map decoding full epoch 573 2024-02-19T21:13:30.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa77fff700 20 client.50198.objecter dump_active .. 0 homeless 2024-02-19T21:13:30.418 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:13:30.418 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:13:30.418 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:13:30.419 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:13:30.419 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:13:30.419 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:13:30.419 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:13:30.419 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:13:30.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v573) v1 ==== 54+0+16 (unknown 1857935615 0 2420000192) 0x7ffa8407b350 con 0x7ffa900ff670 2024-02-19T21:13:30.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa77fff700 10 client.50198.objecter ms_handle_connect 0x7ffa78063c60 2024-02-19T21:13:30.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa956bd700 10 client.50198.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:30.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 1 2024-02-19T21:13:30.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c0051a0 osd=0 3 2024-02-19T21:13:30.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c0051a0 osd=0 3 2024-02-19T21:13:30.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c0051a0 osd=0 3 2024-02-19T21:13:30.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 1 2024-02-19T21:13:30.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c0051a0 osd=0 3 2024-02-19T21:13:30.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 0 1 2024-02-19T21:13:30.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c0051a0 osd=0 4 2024-02-19T21:13:30.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa956bd700 10 client.50198.objecter _send_command 1 2024-02-19T21:13:30.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa7c004950 con 0x7ffa7c004530 2024-02-19T21:13:30.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.180+0000 7ffa77fff700 10 client.50198.objecter ms_handle_connect 0x7ffa7c004530 2024-02-19T21:13:30.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.181+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa7c004950 con 0x7ffa7c004530 2024-02-19T21:13:30.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.181+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:13:30.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.181+0000 7ffa77fff700 10 client.50198.objecter _finish_command 1 = system:0 2024-02-19T21:13:30.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.181+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c0051a0 osd=0 4 2024-02-19T21:13:30.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.181+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 0 1 2024-02-19T21:13:30.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.196+0000 7ffa956bd700 10 client.50198.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:30.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.196+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 2 2024-02-19T21:13:30.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.196+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c0051a0 osd=0 3 2024-02-19T21:13:30.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.196+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c0051a0 osd=0 3 2024-02-19T21:13:30.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.196+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c0051a0 osd=0 3 2024-02-19T21:13:30.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.196+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 2 2024-02-19T21:13:30.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.196+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c0051a0 osd=0 3 2024-02-19T21:13:30.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.196+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 0 2 2024-02-19T21:13:30.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.196+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c0051a0 osd=0 4 2024-02-19T21:13:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.196+0000 7ffa956bd700 10 client.50198.objecter _send_command 2 2024-02-19T21:13:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.196+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa7c004d30 con 0x7ffa7c004530 2024-02-19T21:13:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ffa7c004d30 con 0x7ffa7c004530 2024-02-19T21:13:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa77fff700 10 client.50198.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c0051a0 osd=0 4 2024-02-19T21:13:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 0 2 2024-02-19T21:13:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa956bd700 10 client.50198.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 3 2024-02-19T21:13:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c003fe0 osd=1 3 2024-02-19T21:13:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c003fe0 osd=1 3 2024-02-19T21:13:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c003fe0 osd=1 3 2024-02-19T21:13:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 3 2024-02-19T21:13:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c003fe0 osd=1 3 2024-02-19T21:13:30.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 1 3 2024-02-19T21:13:30.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c003fe0 osd=1 4 2024-02-19T21:13:30.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa956bd700 10 client.50198.objecter _send_command 3 2024-02-19T21:13:30.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa7c00c000 con 0x7ffa7c004150 2024-02-19T21:13:30.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.198+0000 7ffa77fff700 10 client.50198.objecter ms_handle_connect 0x7ffa7c004150 2024-02-19T21:13:30.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.199+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa7c00c000 con 0x7ffa7c004150 2024-02-19T21:13:30.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.199+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:13:30.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.199+0000 7ffa77fff700 10 client.50198.objecter _finish_command 3 = system:0 2024-02-19T21:13:30.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.199+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c003fe0 osd=1 4 2024-02-19T21:13:30.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.199+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 1 3 2024-02-19T21:13:30.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.214+0000 7ffa956bd700 10 client.50198.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:30.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.214+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 4 2024-02-19T21:13:30.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.214+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c003fe0 osd=1 3 2024-02-19T21:13:30.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.214+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c003fe0 osd=1 3 2024-02-19T21:13:30.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.214+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c003fe0 osd=1 3 2024-02-19T21:13:30.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.214+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 4 2024-02-19T21:13:30.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.214+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c003fe0 osd=1 3 2024-02-19T21:13:30.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.214+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 1 4 2024-02-19T21:13:30.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.214+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c003fe0 osd=1 4 2024-02-19T21:13:30.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.214+0000 7ffa956bd700 10 client.50198.objecter _send_command 4 2024-02-19T21:13:30.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.214+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa7c0018b0 con 0x7ffa7c004150 2024-02-19T21:13:30.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ffa7c0018b0 con 0x7ffa7c004150 2024-02-19T21:13:30.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:30.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa77fff700 10 client.50198.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c003fe0 osd=1 4 2024-02-19T21:13:30.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 1 4 2024-02-19T21:13:30.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa956bd700 10 client.50198.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:30.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 5 2024-02-19T21:13:30.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c003630 osd=2 3 2024-02-19T21:13:30.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c003630 osd=2 3 2024-02-19T21:13:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c003630 osd=2 3 2024-02-19T21:13:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 5 2024-02-19T21:13:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c003630 osd=2 3 2024-02-19T21:13:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 2 5 2024-02-19T21:13:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c003630 osd=2 4 2024-02-19T21:13:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa956bd700 10 client.50198.objecter _send_command 5 2024-02-19T21:13:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa7c00de90 con 0x7ffa7c00d2f0 2024-02-19T21:13:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.216+0000 7ffa77fff700 10 client.50198.objecter ms_handle_connect 0x7ffa7c00d2f0 2024-02-19T21:13:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.217+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa7c00de90 con 0x7ffa7c00d2f0 2024-02-19T21:13:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.217+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:13:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.217+0000 7ffa77fff700 10 client.50198.objecter _finish_command 5 = system:0 2024-02-19T21:13:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.217+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c003630 osd=2 4 2024-02-19T21:13:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.217+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 2 5 2024-02-19T21:13:30.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.231+0000 7ffa956bd700 10 client.50198.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.231+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 6 2024-02-19T21:13:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.231+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c003630 osd=2 3 2024-02-19T21:13:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.231+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c003630 osd=2 3 2024-02-19T21:13:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.231+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c003630 osd=2 3 2024-02-19T21:13:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.231+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 6 2024-02-19T21:13:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.231+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c003630 osd=2 3 2024-02-19T21:13:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.231+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 2 6 2024-02-19T21:13:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.231+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c003630 osd=2 4 2024-02-19T21:13:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.231+0000 7ffa956bd700 10 client.50198.objecter _send_command 6 2024-02-19T21:13:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.231+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa7c0030a0 con 0x7ffa7c00d2f0 2024-02-19T21:13:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ffa7c0030a0 con 0x7ffa7c00d2f0 2024-02-19T21:13:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa77fff700 10 client.50198.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c003630 osd=2 4 2024-02-19T21:13:30.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 2 6 2024-02-19T21:13:30.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa956bd700 10 client.50198.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:30.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 7 2024-02-19T21:13:30.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c003480 osd=3 3 2024-02-19T21:13:30.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c003480 osd=3 3 2024-02-19T21:13:30.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c003480 osd=3 3 2024-02-19T21:13:30.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 7 2024-02-19T21:13:30.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c003480 osd=3 3 2024-02-19T21:13:30.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 3 7 2024-02-19T21:13:30.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c003480 osd=3 4 2024-02-19T21:13:30.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa956bd700 10 client.50198.objecter _send_command 7 2024-02-19T21:13:30.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa7c015ac0 con 0x7ffa7c011e80 2024-02-19T21:13:30.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.233+0000 7ffa77fff700 10 client.50198.objecter ms_handle_connect 0x7ffa7c011e80 2024-02-19T21:13:30.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.234+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa7c015ac0 con 0x7ffa7c011e80 2024-02-19T21:13:30.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.234+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:13:30.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.234+0000 7ffa77fff700 10 client.50198.objecter _finish_command 7 = system:0 2024-02-19T21:13:30.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.234+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c003480 osd=3 4 2024-02-19T21:13:30.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.234+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 3 7 2024-02-19T21:13:30.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.248+0000 7ffa956bd700 10 client.50198.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:30.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.248+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 8 2024-02-19T21:13:30.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.248+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c003480 osd=3 3 2024-02-19T21:13:30.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.248+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c003480 osd=3 3 2024-02-19T21:13:30.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.248+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c003480 osd=3 3 2024-02-19T21:13:30.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.248+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 8 2024-02-19T21:13:30.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.248+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c003480 osd=3 3 2024-02-19T21:13:30.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.248+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 3 8 2024-02-19T21:13:30.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.248+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c003480 osd=3 4 2024-02-19T21:13:30.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.248+0000 7ffa956bd700 10 client.50198.objecter _send_command 8 2024-02-19T21:13:30.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.248+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa7c00e270 con 0x7ffa7c011e80 2024-02-19T21:13:30.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.249+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ffa7c00e270 con 0x7ffa7c011e80 2024-02-19T21:13:30.428 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.249+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:30.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.249+0000 7ffa77fff700 10 client.50198.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.249+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c003480 osd=3 4 2024-02-19T21:13:30.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.249+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 3 8 2024-02-19T21:13:30.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.250+0000 7ffa956bd700 10 client.50198.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:30.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.250+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 9 2024-02-19T21:13:30.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.250+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c00e270 osd=4 3 2024-02-19T21:13:30.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.250+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c00e270 osd=4 3 2024-02-19T21:13:30.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.250+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c00e270 osd=4 3 2024-02-19T21:13:30.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.250+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 9 2024-02-19T21:13:30.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.250+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c00e270 osd=4 3 2024-02-19T21:13:30.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.250+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 4 9 2024-02-19T21:13:30.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.250+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c00e270 osd=4 4 2024-02-19T21:13:30.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.250+0000 7ffa956bd700 10 client.50198.objecter _send_command 9 2024-02-19T21:13:30.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.250+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa7c00e5d0 con 0x7ffa7c00ce50 2024-02-19T21:13:30.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.250+0000 7ffa77fff700 10 client.50198.objecter ms_handle_connect 0x7ffa7c00ce50 2024-02-19T21:13:30.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.251+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa7c00e5d0 con 0x7ffa7c00ce50 2024-02-19T21:13:30.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.251+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:13:30.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.251+0000 7ffa77fff700 10 client.50198.objecter _finish_command 9 = system:0 2024-02-19T21:13:30.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.251+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c00e270 osd=4 4 2024-02-19T21:13:30.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.251+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 4 9 2024-02-19T21:13:30.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.265+0000 7ffa956bd700 10 client.50198.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:30.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.265+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 10 2024-02-19T21:13:30.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.265+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c00e270 osd=4 3 2024-02-19T21:13:30.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.265+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c00e270 osd=4 3 2024-02-19T21:13:30.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.265+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c00e270 osd=4 3 2024-02-19T21:13:30.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.265+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 10 2024-02-19T21:13:30.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.265+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c00e270 osd=4 3 2024-02-19T21:13:30.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.265+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 4 10 2024-02-19T21:13:30.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.265+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c00e270 osd=4 4 2024-02-19T21:13:30.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.265+0000 7ffa956bd700 10 client.50198.objecter _send_command 10 2024-02-19T21:13:30.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.265+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa7c00e5d0 con 0x7ffa7c00ce50 2024-02-19T21:13:30.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ffa7c00e5d0 con 0x7ffa7c00ce50 2024-02-19T21:13:30.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:30.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa77fff700 10 client.50198.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c00e270 osd=4 4 2024-02-19T21:13:30.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 4 10 2024-02-19T21:13:30.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa956bd700 10 client.50198.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:30.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 11 2024-02-19T21:13:30.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c00e5d0 osd=5 3 2024-02-19T21:13:30.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c00e5d0 osd=5 3 2024-02-19T21:13:30.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c00e5d0 osd=5 3 2024-02-19T21:13:30.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 11 2024-02-19T21:13:30.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c00e5d0 osd=5 3 2024-02-19T21:13:30.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 5 11 2024-02-19T21:13:30.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c00e5d0 osd=5 4 2024-02-19T21:13:30.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa956bd700 10 client.50198.objecter _send_command 11 2024-02-19T21:13:30.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa7c017fc0 con 0x7ffa7c017a40 2024-02-19T21:13:30.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.268+0000 7ffa77fff700 10 client.50198.objecter ms_handle_connect 0x7ffa7c017a40 2024-02-19T21:13:30.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.269+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa7c017fc0 con 0x7ffa7c017a40 2024-02-19T21:13:30.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.269+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:13:30.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.269+0000 7ffa77fff700 10 client.50198.objecter _finish_command 11 = system:0 2024-02-19T21:13:30.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.269+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c00e5d0 osd=5 4 2024-02-19T21:13:30.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.269+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 5 11 2024-02-19T21:13:30.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.283+0000 7ffa956bd700 10 client.50198.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:30.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.283+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 12 2024-02-19T21:13:30.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.283+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c00e5d0 osd=5 3 2024-02-19T21:13:30.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.283+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c00e5d0 osd=5 3 2024-02-19T21:13:30.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.283+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c00e5d0 osd=5 3 2024-02-19T21:13:30.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.283+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 12 2024-02-19T21:13:30.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.283+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c00e5d0 osd=5 3 2024-02-19T21:13:30.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.283+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 5 12 2024-02-19T21:13:30.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.283+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c00e5d0 osd=5 4 2024-02-19T21:13:30.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.283+0000 7ffa956bd700 10 client.50198.objecter _send_command 12 2024-02-19T21:13:30.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.283+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa7c017fc0 con 0x7ffa7c017a40 2024-02-19T21:13:30.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ffa7c017fc0 con 0x7ffa7c017a40 2024-02-19T21:13:30.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:30.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa77fff700 10 client.50198.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c00e5d0 osd=5 4 2024-02-19T21:13:30.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 5 12 2024-02-19T21:13:30.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa956bd700 10 client.50198.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:30.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 13 2024-02-19T21:13:30.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c017fc0 osd=6 3 2024-02-19T21:13:30.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c017fc0 osd=6 3 2024-02-19T21:13:30.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c017fc0 osd=6 3 2024-02-19T21:13:30.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 13 2024-02-19T21:13:30.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c017fc0 osd=6 3 2024-02-19T21:13:30.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 6 13 2024-02-19T21:13:30.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c017fc0 osd=6 4 2024-02-19T21:13:30.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa956bd700 10 client.50198.objecter _send_command 13 2024-02-19T21:13:30.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa7c022bb0 con 0x7ffa7c01f0a0 2024-02-19T21:13:30.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa77fff700 10 client.50198.objecter ms_handle_connect 0x7ffa7c01f0a0 2024-02-19T21:13:30.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa7c022bb0 con 0x7ffa7c01f0a0 2024-02-19T21:13:30.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:13:30.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa77fff700 10 client.50198.objecter _finish_command 13 = system:0 2024-02-19T21:13:30.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c017fc0 osd=6 4 2024-02-19T21:13:30.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.286+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 6 13 2024-02-19T21:13:30.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.301+0000 7ffa956bd700 10 client.50198.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:30.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.301+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 14 2024-02-19T21:13:30.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.301+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c017fc0 osd=6 3 2024-02-19T21:13:30.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.301+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c017fc0 osd=6 3 2024-02-19T21:13:30.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.301+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c017fc0 osd=6 3 2024-02-19T21:13:30.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.301+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 14 2024-02-19T21:13:30.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.301+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c017fc0 osd=6 3 2024-02-19T21:13:30.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.301+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 6 14 2024-02-19T21:13:30.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.301+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c017fc0 osd=6 4 2024-02-19T21:13:30.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.301+0000 7ffa956bd700 10 client.50198.objecter _send_command 14 2024-02-19T21:13:30.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.301+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa7c022bb0 con 0x7ffa7c01f0a0 2024-02-19T21:13:30.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ffa7c022bb0 con 0x7ffa7c01f0a0 2024-02-19T21:13:30.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:30.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa77fff700 10 client.50198.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c017fc0 osd=6 4 2024-02-19T21:13:30.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 6 14 2024-02-19T21:13:30.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa956bd700 10 client.50198.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:30.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 15 2024-02-19T21:13:30.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c022bb0 osd=7 3 2024-02-19T21:13:30.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c022bb0 osd=7 3 2024-02-19T21:13:30.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c022bb0 osd=7 3 2024-02-19T21:13:30.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 15 2024-02-19T21:13:30.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c022bb0 osd=7 3 2024-02-19T21:13:30.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 7 15 2024-02-19T21:13:30.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c022bb0 osd=7 4 2024-02-19T21:13:30.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa956bd700 10 client.50198.objecter _send_command 15 2024-02-19T21:13:30.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.303+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa7c023dd0 con 0x7ffa7c024870 2024-02-19T21:13:30.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.305+0000 7ffa77fff700 10 client.50198.objecter ms_handle_connect 0x7ffa7c024870 2024-02-19T21:13:30.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.305+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa7c023dd0 con 0x7ffa7c024870 2024-02-19T21:13:30.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.305+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:13:30.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.305+0000 7ffa77fff700 10 client.50198.objecter _finish_command 15 = system:0 2024-02-19T21:13:30.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.305+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c022bb0 osd=7 4 2024-02-19T21:13:30.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.305+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 7 15 2024-02-19T21:13:30.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.321+0000 7ffa956bd700 10 client.50198.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:30.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.321+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign -1 16 2024-02-19T21:13:30.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.322+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c022bb0 osd=7 3 2024-02-19T21:13:30.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.322+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c022bb0 osd=7 3 2024-02-19T21:13:30.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.322+0000 7ffa956bd700 20 client.50198.objecter _get_session s=0x7ffa7c022bb0 osd=7 3 2024-02-19T21:13:30.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.322+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_remove -1 16 2024-02-19T21:13:30.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.322+0000 7ffa956bd700 20 client.50198.objecter get_session s=0x7ffa7c022bb0 osd=7 3 2024-02-19T21:13:30.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.322+0000 7ffa956bd700 15 client.50198.objecter _session_command_op_assign 7 16 2024-02-19T21:13:30.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.322+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c022bb0 osd=7 4 2024-02-19T21:13:30.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.322+0000 7ffa956bd700 10 client.50198.objecter _send_command 16 2024-02-19T21:13:30.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.322+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7ffa7c023dd0 con 0x7ffa7c024870 2024-02-19T21:13:30.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.323+0000 7ffa77fff700 1 -- 172.21.15.163:0/3881623081 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7ffa7c023dd0 con 0x7ffa7c024870 2024-02-19T21:13:30.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.323+0000 7ffa77fff700 10 client.50198.objecter ms_dispatch 0x7ffa90001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:30.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.323+0000 7ffa77fff700 10 client.50198.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.323+0000 7ffa77fff700 20 client.50198.objecter put_session s=0x7ffa7c022bb0 osd=7 4 2024-02-19T21:13:30.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.323+0000 7ffa77fff700 15 client.50198.objecter _session_command_op_remove 7 16 2024-02-19T21:13:30.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 10 librados: watch_flush enter 2024-02-19T21:13:30.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 10 librados: watch_flush exit 2024-02-19T21:13:30.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 10 client.50198.objecter close_session for osd.0 2024-02-19T21:13:30.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 >> v1:172.21.15.163:6801/1559418115 conn(0x7ffa7c004530 legacy=0x7ffa7c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:30.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c0051a0 osd=0 1 2024-02-19T21:13:30.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 10 client.50198.objecter close_session for osd.1 2024-02-19T21:13:30.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 >> v1:172.21.15.163:6805/3125047432 conn(0x7ffa7c004150 legacy=0x7ffa7c00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:30.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c003fe0 osd=1 1 2024-02-19T21:13:30.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 10 client.50198.objecter close_session for osd.2 2024-02-19T21:13:30.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 >> v1:172.21.15.163:6809/1563158487 conn(0x7ffa7c00d2f0 legacy=0x7ffa7c00d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:30.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c003630 osd=2 1 2024-02-19T21:13:30.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 10 client.50198.objecter close_session for osd.3 2024-02-19T21:13:30.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 >> v1:172.21.15.163:6813/2112279769 conn(0x7ffa7c011e80 legacy=0x7ffa7c014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:30.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c003480 osd=3 1 2024-02-19T21:13:30.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 10 client.50198.objecter close_session for osd.4 2024-02-19T21:13:30.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 >> v1:172.21.15.193:6800/3129716965 conn(0x7ffa7c00ce50 legacy=0x7ffa7c015c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:30.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa8f7fe700 1 -- 172.21.15.163:0/3881623081 reap_dead start 2024-02-19T21:13:30.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa8f7fe700 1 -- 172.21.15.163:0/3881623081 reap_dead start 2024-02-19T21:13:30.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c00e270 osd=4 1 2024-02-19T21:13:30.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 10 client.50198.objecter close_session for osd.5 2024-02-19T21:13:30.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 >> v1:172.21.15.193:6804/805161991 conn(0x7ffa7c017a40 legacy=0x7ffa7c01d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:30.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c00e5d0 osd=5 1 2024-02-19T21:13:30.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 10 client.50198.objecter close_session for osd.6 2024-02-19T21:13:30.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 >> v1:172.21.15.193:6808/787604615 conn(0x7ffa7c01f0a0 legacy=0x7ffa7c021490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:30.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c017fc0 osd=6 1 2024-02-19T21:13:30.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 10 client.50198.objecter close_session for osd.7 2024-02-19T21:13:30.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 >> v1:172.21.15.193:6812/157927432 conn(0x7ffa7c024870 legacy=0x7ffa7c024cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:30.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 20 client.50198.objecter put_session s=0x7ffa7c022bb0 osd=7 1 2024-02-19T21:13:30.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 20 client.50198.objecter shutdown clearing up homeless session... 2024-02-19T21:13:30.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 10 client.50198.objecter successfully canceled tick 2024-02-19T21:13:30.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 >> v1:172.21.15.163:6800/1552514681 conn(0x7ffa78063c60 legacy=0x7ffa78066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:30.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 >> v1:172.21.15.163:6790/0 conn(0x7ffa900ff670 legacy=0x7ffa900610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:30.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa8f7fe700 1 -- 172.21.15.163:0/3881623081 reap_dead start 2024-02-19T21:13:30.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 shutdown_connections 2024-02-19T21:13:30.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 >> 172.21.15.163:0/3881623081 conn(0x7ffa900f2470 msgr2=0x7ffa900f5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:30.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 shutdown_connections 2024-02-19T21:13:30.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 1 -- 172.21.15.163:0/3881623081 wait complete. 2024-02-19T21:13:30.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:30.324+0000 7ffa956bd700 1 librados: shutdown 2024-02-19T21:13:30.440 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.441 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.441 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.441 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.441 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.441 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.441 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.441 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:13:30.621 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:30.626 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:30.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:30.627 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:29.270571+0000 mgr.y (mgr.14550) 504 : cluster [DBG] pgmap v858: 246 pgs: 246 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:30.628 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:29.362371+0000 mon.a (mon.0) 2996 : cluster [DBG] osdmap e573: 8 total, 8 up, 6 in 2024-02-19T21:13:30.628 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:29.673528+0000 mon.b (mon.1) 728 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:13:30.628 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:29.725633+0000 osd.0 (osd.0) 129 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:30.628 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:29.725801+0000 osd.0 (osd.0) 130 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:30.628 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:30.181006+0000 mon.c (mon.2) 410 : audit [DBG] from='client.? v1:172.21.15.163:0/3881623081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:30.628 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:30.364342+0000 mon.a (mon.0) 2997 : cluster [DBG] osdmap e574: 8 total, 8 up, 6 in 2024-02-19T21:13:30.628 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:30.382060+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:30.628 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:30.383465+0000 mon.a (mon.0) 2999 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:30.628 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:30.383734+0000 mon.a (mon.0) 3000 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83928-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:13:30.629 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:29.270571+0000 mgr.y (mgr.14550) 504 : cluster [DBG] pgmap v858: 246 pgs: 246 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:30.629 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:29.362371+0000 mon.a (mon.0) 2996 : cluster [DBG] osdmap e573: 8 total, 8 up, 6 in 2024-02-19T21:13:30.629 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:29.673528+0000 mon.b (mon.1) 728 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:13:30.629 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:29.725633+0000 osd.0 (osd.0) 129 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:30.629 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:29.725801+0000 osd.0 (osd.0) 130 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:30.629 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:30.181006+0000 mon.c (mon.2) 410 : audit [DBG] from='client.? v1:172.21.15.163:0/3881623081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:30.629 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:30.364342+0000 mon.a (mon.0) 2997 : cluster [DBG] osdmap e574: 8 total, 8 up, 6 in 2024-02-19T21:13:30.629 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:30.382060+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:30.629 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:30.383465+0000 mon.a (mon.0) 2999 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:30.629 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:30.383734+0000 mon.a (mon.0) 3000 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83928-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:13:30.727 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:30.756 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:30.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:13:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:13:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:29.270571+0000 mgr.y (mgr.14550) 504 : cluster [DBG] pgmap v858: 246 pgs: 246 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:29.362371+0000 mon.a (mon.0) 2996 : cluster [DBG] osdmap e573: 8 total, 8 up, 6 in 2024-02-19T21:13:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:29.673528+0000 mon.b (mon.1) 728 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:13:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:29.725633+0000 osd.0 (osd.0) 129 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:29.725801+0000 osd.0 (osd.0) 130 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:30.181006+0000 mon.c (mon.2) 410 : audit [DBG] from='client.? v1:172.21.15.163:0/3881623081' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:30.364342+0000 mon.a (mon.0) 2997 : cluster [DBG] osdmap e574: 8 total, 8 up, 6 in 2024-02-19T21:13:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:30.382060+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:30.383465+0000 mon.a (mon.0) 2999 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:30.383734+0000 mon.a (mon.0) 3000 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83928-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:13:31.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:30.581078+0000 mgr.y (mgr.14550) 505 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:31.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:31.364059+0000 mon.a (mon.0) 3001 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83928-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:13:31.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:31.364125+0000 mon.a (mon.0) 3002 : cluster [DBG] osdmap e575: 8 total, 8 up, 6 in 2024-02-19T21:13:31.417 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:30.581078+0000 mgr.y (mgr.14550) 505 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:31.417 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:31.364059+0000 mon.a (mon.0) 3001 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83928-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:13:31.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:31.364125+0000 mon.a (mon.0) 3002 : cluster [DBG] osdmap e575: 8 total, 8 up, 6 in 2024-02-19T21:13:31.418 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:31.368754+0000 mon.a (mon.0) 3003 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83928-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:31.428 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:31.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.432+0000 7f8a4b6fc700 1 -- 172.21.15.163:0/3889092752 >> v1:172.21.15.163:6790/0 conn(0x7f8a440f9540 legacy=0x7f8a44102a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:31.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.432+0000 7f8a4b6fc700 1 -- 172.21.15.163:0/3889092752 shutdown_connections 2024-02-19T21:13:31.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.432+0000 7f8a4b6fc700 1 -- 172.21.15.163:0/3889092752 >> 172.21.15.163:0/3889092752 conn(0x7f8a440f24a0 msgr2=0x7f8a440f48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:31.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.432+0000 7f8a4b6fc700 1 -- 172.21.15.163:0/3889092752 shutdown_connections 2024-02-19T21:13:31.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.432+0000 7f8a4b6fc700 1 -- 172.21.15.163:0/3889092752 wait complete. 2024-02-19T21:13:31.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.433+0000 7f8a4b6fc700 1 librados: starting msgr at 2024-02-19T21:13:31.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.433+0000 7f8a4b6fc700 1 librados: starting objecter 2024-02-19T21:13:31.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.433+0000 7f8a4b6fc700 1 Processor -- start 2024-02-19T21:13:31.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.433+0000 7f8a4b6fc700 1 -- start start 2024-02-19T21:13:31.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.433+0000 7f8a4b6fc700 1 librados: setting wanted keys 2024-02-19T21:13:31.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.433+0000 7f8a4b6fc700 1 librados: calling monclient init 2024-02-19T21:13:31.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.433+0000 7f8a4b6fc700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8a44117230 con 0x7f8a440f9540 2024-02-19T21:13:31.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.433+0000 7f8a4b6fc700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8a44118410 con 0x7f8a440f8990 2024-02-19T21:13:31.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.433+0000 7f8a4b6fc700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8a44119610 con 0x7f8a441042c0 2024-02-19T21:13:31.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a48c97700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8a440f9540 0x7f8a440ffb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:53476/0 (socket says 172.21.15.163:53476) 2024-02-19T21:13:31.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a48c97700 1 -- 172.21.15.163:0/736273242 learned_addr learned my addr 172.21.15.163:0/736273242 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:31.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 10 client.?.objecter ms_handle_connect 0x7f8a440f8990 2024-02-19T21:13:31.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:31.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 10 client.?.objecter ms_handle_connect 0x7f8a440f9540 2024-02-19T21:13:31.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:31.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449125603 0 0) 0x7f8a44118410 con 0x7f8a440f8990 2024-02-19T21:13:31.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a30002300 con 0x7f8a440f8990 2024-02-19T21:13:31.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 10 client.?.objecter ms_handle_connect 0x7f8a441042c0 2024-02-19T21:13:31.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:31.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146320352 0 0) 0x7f8a44119610 con 0x7f8a441042c0 2024-02-19T21:13:31.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a44118410 con 0x7f8a441042c0 2024-02-19T21:13:31.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516691752 0 0) 0x7f8a44117230 con 0x7f8a440f9540 2024-02-19T21:13:31.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a44119610 con 0x7f8a440f9540 2024-02-19T21:13:31.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3242154618 0 0) 0x7f8a30002300 con 0x7f8a440f8990 2024-02-19T21:13:31.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a44117230 con 0x7f8a440f8990 2024-02-19T21:13:31.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8a40001620 con 0x7f8a440f8990 2024-02-19T21:13:31.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3673941289 0 0) 0x7f8a44117230 con 0x7f8a440f8990 2024-02-19T21:13:31.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 >> v1:172.21.15.163:6790/0 conn(0x7f8a441042c0 legacy=0x7f8a44115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:31.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 >> v1:172.21.15.163:6789/0 conn(0x7f8a440f9540 legacy=0x7f8a440ffb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:31.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4411a810 con 0x7f8a440f8990 2024-02-19T21:13:31.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a4b6fc700 1 -- 172.21.15.163:0/736273242 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a44117420 con 0x7f8a440f8990 2024-02-19T21:13:31.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a4b6fc700 10 client.49018.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:31.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a4b6fc700 1 -- 172.21.15.163:0/736273242 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a44117890 con 0x7f8a440f8990 2024-02-19T21:13:31.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.434+0000 7f8a4b6fc700 1 librados: init done 2024-02-19T21:13:31.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.435+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8a40002250 con 0x7f8a440f8990 2024-02-19T21:13:31.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.435+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8a400032d0 con 0x7f8a440f8990 2024-02-19T21:13:31.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.435+0000 7f8a4b6fc700 1 -- 172.21.15.163:0/736273242 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a28004f50 con 0x7f8a440f8990 2024-02-19T21:13:31.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.436+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f8a40003450 con 0x7f8a440f8990 2024-02-19T21:13:31.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.436+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(575..575 src has 434..575) v4 ==== 8421+0+0 (unknown 34374649 0 0) 0x7f8a400764f0 con 0x7f8a440f8990 2024-02-19T21:13:31.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.436+0000 7f8a3a7fc700 10 client.49018.objecter ms_dispatch 0x7f8a44001550 osd_map(575..575 src has 434..575) v4 2024-02-19T21:13:31.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.436+0000 7f8a3a7fc700 3 client.49018.objecter handle_osd_map got epochs [575,575] > 0 2024-02-19T21:13:31.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.436+0000 7f8a3a7fc700 3 client.49018.objecter handle_osd_map decoding full epoch 575 2024-02-19T21:13:31.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.437+0000 7f8a3a7fc700 20 client.49018.objecter dump_active .. 0 homeless 2024-02-19T21:13:31.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.437+0000 7f8a3a7fc700 10 client.49018.objecter ms_handle_connect 0x7f8a30063b10 2024-02-19T21:13:31.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.441+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8a40049780 con 0x7f8a440f8990 2024-02-19T21:13:31.473 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:31.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:30.581078+0000 mgr.y (mgr.14550) 505 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:31.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:31.364059+0000 mon.a (mon.0) 3001 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83928-99", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:13:31.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:31.364125+0000 mon.a (mon.0) 3002 : cluster [DBG] osdmap e575: 8 total, 8 up, 6 in 2024-02-19T21:13:31.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:31.368754+0000 mon.a (mon.0) 3003 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83928-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:31.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:31.621+0000 7f8a4b6fc700 1 -- 172.21.15.163:0/736273242 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"} v 0) v1 -- 0x7f8a28005200 con 0x7f8a440f8990 2024-02-19T21:13:31.681 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:31.368754+0000 mon.a (mon.0) 3003 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83928-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:32.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:32.611+0000 7f8a3a7fc700 1 -- 172.21.15.163:0/736273242 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]=0 set pool 6 pgp_num to 32 v576) v1 ==== 143+0+0 (unknown 3064186339 0 0) 0x7f8a40047b90 con 0x7f8a440f8990 2024-02-19T21:13:32.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:32.613+0000 7f8a4b6fc700 10 librados: watch_flush enter 2024-02-19T21:13:32.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:32.614+0000 7f8a4b6fc700 10 librados: watch_flush exit 2024-02-19T21:13:32.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:32.614+0000 7f8a4b6fc700 20 client.49018.objecter shutdown clearing up homeless session... 2024-02-19T21:13:32.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:32.614+0000 7f8a4b6fc700 10 client.49018.objecter successfully canceled tick 2024-02-19T21:13:32.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:32.614+0000 7f8a4b6fc700 1 -- 172.21.15.163:0/736273242 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8a30063b10 legacy=0x7f8a30065fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:32.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:32.614+0000 7f8a4b6fc700 1 -- 172.21.15.163:0/736273242 >> v1:172.21.15.193:6789/0 conn(0x7f8a440f8990 legacy=0x7f8a440ff440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:32.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:32.614+0000 7f8a4b6fc700 1 -- 172.21.15.163:0/736273242 shutdown_connections 2024-02-19T21:13:32.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:32.614+0000 7f8a4b6fc700 1 -- 172.21.15.163:0/736273242 >> 172.21.15.163:0/736273242 conn(0x7f8a440f24a0 msgr2=0x7f8a44107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:32.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:32.614+0000 7f8a4b6fc700 1 -- 172.21.15.163:0/736273242 shutdown_connections 2024-02-19T21:13:32.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:32.614+0000 7f8a4b6fc700 1 -- 172.21.15.163:0/736273242 wait complete. 2024-02-19T21:13:32.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:32.614+0000 7f8a4b6fc700 1 librados: shutdown 2024-02-19T21:13:32.614 INFO:teuthology.orchestra.run.smithi163.stderr:set pool 6 pgp_num to 32 2024-02-19T21:13:32.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:31.270942+0000 mgr.y (mgr.14550) 506 : cluster [DBG] pgmap v861: 182 pgs: 182 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:32.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:31.623312+0000 mon.b (mon.1) 729 : audit [INF] from='client.? v1:172.21.15.163:0/736273242' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-19T21:13:32.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:31.623625+0000 mon.a (mon.0) 3004 : audit [INF] from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-19T21:13:32.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:31.270942+0000 mgr.y (mgr.14550) 506 : cluster [DBG] pgmap v861: 182 pgs: 182 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:32.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:31.623312+0000 mon.b (mon.1) 729 : audit [INF] from='client.? v1:172.21.15.163:0/736273242' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-19T21:13:32.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:31.623625+0000 mon.a (mon.0) 3004 : audit [INF] from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-19T21:13:32.915 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:32.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:32.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:33.019 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:13:33.049 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:33.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:31.270942+0000 mgr.y (mgr.14550) 506 : cluster [DBG] pgmap v861: 182 pgs: 182 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:33.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:31.623312+0000 mon.b (mon.1) 729 : audit [INF] from='client.? v1:172.21.15.163:0/736273242' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-19T21:13:33.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:31.623625+0000 mon.a (mon.0) 3004 : audit [INF] from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-19T21:13:33.142 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:13:33.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:32.611685+0000 mon.a (mon.0) 3005 : audit [INF] from='client.49018 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]': finished 2024-02-19T21:13:33.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:32.611757+0000 mon.a (mon.0) 3006 : cluster [DBG] osdmap e576: 8 total, 8 up, 6 in 2024-02-19T21:13:33.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:33.367805+0000 mon.a (mon.0) 3007 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83928-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83928-99"}]': finished 2024-02-19T21:13:33.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:33.367860+0000 mon.a (mon.0) 3008 : cluster [DBG] osdmap e577: 8 total, 8 up, 6 in 2024-02-19T21:13:33.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:32.611685+0000 mon.a (mon.0) 3005 : audit [INF] from='client.49018 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]': finished 2024-02-19T21:13:33.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:32.611757+0000 mon.a (mon.0) 3006 : cluster [DBG] osdmap e576: 8 total, 8 up, 6 in 2024-02-19T21:13:33.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:33.367805+0000 mon.a (mon.0) 3007 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83928-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83928-99"}]': finished 2024-02-19T21:13:33.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:33.367860+0000 mon.a (mon.0) 3008 : cluster [DBG] osdmap e577: 8 total, 8 up, 6 in 2024-02-19T21:13:33.748 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:13:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:13:33] "GET /metrics HTTP/1.1" 200 193310 "" "Prometheus/2.33.4" 2024-02-19T21:13:34.070 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:32.611685+0000 mon.a (mon.0) 3005 : audit [INF] from='client.49018 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]': finished 2024-02-19T21:13:34.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:32.611757+0000 mon.a (mon.0) 3006 : cluster [DBG] osdmap e576: 8 total, 8 up, 6 in 2024-02-19T21:13:34.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:33.367805+0000 mon.a (mon.0) 3007 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83928-99", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83928-99"}]': finished 2024-02-19T21:13:34.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:33.367860+0000 mon.a (mon.0) 3008 : cluster [DBG] osdmap e577: 8 total, 8 up, 6 in 2024-02-19T21:13:35.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:33.271338+0000 mgr.y (mgr.14550) 507 : cluster [DBG] pgmap v864: 182 pgs: 182 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:35.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:33.813871+0000 mon.a (mon.0) 3009 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:35.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:34.336185+0000 mon.a (mon.0) 3010 : cluster [DBG] osdmap e578: 8 total, 8 up, 6 in 2024-02-19T21:13:35.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:34.344880+0000 mon.a (mon.0) 3011 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:35.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.164+0000 7f6723e44700 1 -- 172.21.15.163:0/3118655627 >> v1:172.21.15.163:6789/0 conn(0x7f671c1042b0 legacy=0x7f671c106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:35.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.164+0000 7f6723e44700 1 -- 172.21.15.163:0/3118655627 shutdown_connections 2024-02-19T21:13:35.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.164+0000 7f6723e44700 1 -- 172.21.15.163:0/3118655627 >> 172.21.15.163:0/3118655627 conn(0x7f671c0f2470 msgr2=0x7f671c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:35.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.164+0000 7f6723e44700 1 -- 172.21.15.163:0/3118655627 shutdown_connections 2024-02-19T21:13:35.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.164+0000 7f6723e44700 1 -- 172.21.15.163:0/3118655627 wait complete. 2024-02-19T21:13:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6723e44700 1 librados: starting msgr at 2024-02-19T21:13:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6723e44700 1 librados: starting objecter 2024-02-19T21:13:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6723e44700 1 Processor -- start 2024-02-19T21:13:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6723e44700 1 -- start start 2024-02-19T21:13:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6723e44700 1 librados: setting wanted keys 2024-02-19T21:13:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6723e44700 1 librados: calling monclient init 2024-02-19T21:13:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6723e44700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f671c117340 con 0x7f671c0f8920 2024-02-19T21:13:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6723e44700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f671c118520 con 0x7f671c0f94d0 2024-02-19T21:13:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6723e44700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f671c119720 con 0x7f671c1042b0 2024-02-19T21:13:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f67223e1700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f671c1042b0 0x7f671c115a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:58808/0 (socket says 172.21.15.163:58808) 2024-02-19T21:13:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6721be0700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f671c0f8920 0x7f671c0ff480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43680/0 (socket says 172.21.15.163:43680) 2024-02-19T21:13:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f67223e1700 1 -- 172.21.15.163:0/3522094459 learned_addr learned my addr 172.21.15.163:0/3522094459 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:35.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6712ffd700 10 client.?.objecter ms_handle_connect 0x7f671c0f8920 2024-02-19T21:13:35.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6712ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:35.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6712ffd700 10 client.?.objecter ms_handle_connect 0x7f671c1042b0 2024-02-19T21:13:35.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6712ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:35.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6712ffd700 10 client.?.objecter ms_handle_connect 0x7f671c0f94d0 2024-02-19T21:13:35.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6712ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:35.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127401870 0 0) 0x7f671c117340 con 0x7f671c0f8920 2024-02-19T21:13:35.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6708003800 con 0x7f671c0f8920 2024-02-19T21:13:35.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908461567 0 0) 0x7f671c118520 con 0x7f671c0f94d0 2024-02-19T21:13:35.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f671c117340 con 0x7f671c0f94d0 2024-02-19T21:13:35.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592190636 0 0) 0x7f671c119720 con 0x7f671c1042b0 2024-02-19T21:13:35.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.165+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f671c118520 con 0x7f671c1042b0 2024-02-19T21:13:35.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2189085948 0 0) 0x7f6708003800 con 0x7f671c0f8920 2024-02-19T21:13:35.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f671c119720 con 0x7f671c0f8920 2024-02-19T21:13:35.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 386104747 0 0) 0x7f671c117340 con 0x7f671c0f94d0 2024-02-19T21:13:35.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6708003800 con 0x7f671c0f94d0 2024-02-19T21:13:35.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f670c0019f0 con 0x7f671c0f8920 2024-02-19T21:13:35.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6718002870 con 0x7f671c0f94d0 2024-02-19T21:13:35.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 939882625 0 0) 0x7f671c118520 con 0x7f671c1042b0 2024-02-19T21:13:35.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f671c117340 con 0x7f671c1042b0 2024-02-19T21:13:35.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f67140023d0 con 0x7f671c1042b0 2024-02-19T21:13:35.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 568391579 0 0) 0x7f671c117340 con 0x7f671c1042b0 2024-02-19T21:13:35.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 >> v1:172.21.15.193:6789/0 conn(0x7f671c0f94d0 legacy=0x7f671c0ffbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:35.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 >> v1:172.21.15.163:6789/0 conn(0x7f671c0f8920 legacy=0x7f671c0ff480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:35.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c11a920 con 0x7f671c1042b0 2024-02-19T21:13:35.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6723e44700 1 -- 172.21.15.163:0/3522094459 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c119910 con 0x7f671c1042b0 2024-02-19T21:13:35.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6723e44700 10 client.50216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:35.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6723e44700 1 -- 172.21.15.163:0/3522094459 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c119d80 con 0x7f671c1042b0 2024-02-19T21:13:35.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6723e44700 1 librados: init done 2024-02-19T21:13:35.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f67140025a0 con 0x7f671c1042b0 2024-02-19T21:13:35.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.166+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6714003b80 con 0x7f671c1042b0 2024-02-19T21:13:35.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.167+0000 7f6723e44700 1 -- 172.21.15.163:0/3522094459 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6700004f20 con 0x7f671c1042b0 2024-02-19T21:13:35.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.167+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6714003ce0 con 0x7f671c1042b0 2024-02-19T21:13:35.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.168+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(578..578 src has 434..578) v4 ==== 9325+0+0 (unknown 2714013628 0 0) 0x7f6714079760 con 0x7f671c1042b0 2024-02-19T21:13:35.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.168+0000 7f6712ffd700 10 client.50216.objecter ms_dispatch 0x7f671c001550 osd_map(578..578 src has 434..578) v4 2024-02-19T21:13:35.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.168+0000 7f6712ffd700 3 client.50216.objecter handle_osd_map got epochs [578,578] > 0 2024-02-19T21:13:35.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.168+0000 7f6712ffd700 3 client.50216.objecter handle_osd_map decoding full epoch 578 2024-02-19T21:13:35.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.168+0000 7f6712ffd700 20 client.50216.objecter dump_active .. 0 homeless 2024-02-19T21:13:35.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.168+0000 7f6712ffd700 10 client.50216.objecter ms_handle_connect 0x7f6708063d00 2024-02-19T21:13:35.171 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.172+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f67140028f0 con 0x7f671c1042b0 2024-02-19T21:13:35.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:33.271338+0000 mgr.y (mgr.14550) 507 : cluster [DBG] pgmap v864: 182 pgs: 182 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:35.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:33.813871+0000 mon.a (mon.0) 3009 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:35.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:34.336185+0000 mon.a (mon.0) 3010 : cluster [DBG] osdmap e578: 8 total, 8 up, 6 in 2024-02-19T21:13:35.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:34.344880+0000 mon.a (mon.0) 3011 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:35.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:33.271338+0000 mgr.y (mgr.14550) 507 : cluster [DBG] pgmap v864: 182 pgs: 182 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:35.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:33.813871+0000 mon.a (mon.0) 3009 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:35.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:34.336185+0000 mon.a (mon.0) 3010 : cluster [DBG] osdmap e578: 8 total, 8 up, 6 in 2024-02-19T21:13:35.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:34.344880+0000 mon.a (mon.0) 3011 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-99-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:35.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:35.368+0000 7f6723e44700 1 -- 172.21.15.163:0/3522094459 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f67000051d0 con 0x7f671c1042b0 2024-02-19T21:13:35.728 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:13:35.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:35.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:13:35.833 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:35.847 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:13:35.847 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:13:35.866 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:35.874 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:35.888 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:35.891 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:35.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:35.992 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:36.031 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:36.053 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:36.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:36.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:36.157 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:36.188 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:36.209 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:36.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:36.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:36.313 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:13:36.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:36.345+0000 7f6712ffd700 1 -- 172.21.15.163:0/3522094459 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v580) v1 ==== 91+0+0 (unknown 2739168406 0 0) 0x7f671404ba70 con 0x7f671c1042b0 2024-02-19T21:13:36.345 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:36.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:36.348+0000 7f6723e44700 10 librados: watch_flush enter 2024-02-19T21:13:36.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:36.348+0000 7f6723e44700 10 librados: watch_flush exit 2024-02-19T21:13:36.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:36.348+0000 7f6723e44700 20 client.50216.objecter shutdown clearing up homeless session... 2024-02-19T21:13:36.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:36.348+0000 7f6723e44700 10 client.50216.objecter successfully canceled tick 2024-02-19T21:13:36.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:36.348+0000 7f6723e44700 1 -- 172.21.15.163:0/3522094459 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6708063d00 legacy=0x7f67080661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:36.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:36.348+0000 7f6723e44700 1 -- 172.21.15.163:0/3522094459 >> v1:172.21.15.163:6790/0 conn(0x7f671c1042b0 legacy=0x7f671c115a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:36.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:36.348+0000 7f6723e44700 1 -- 172.21.15.163:0/3522094459 shutdown_connections 2024-02-19T21:13:36.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:36.348+0000 7f6723e44700 1 -- 172.21.15.163:0/3522094459 >> 172.21.15.163:0/3522094459 conn(0x7f671c0f2470 msgr2=0x7f671c103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:36.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:36.348+0000 7f6723e44700 1 -- 172.21.15.163:0/3522094459 shutdown_connections 2024-02-19T21:13:36.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:36.348+0000 7f6723e44700 1 -- 172.21.15.163:0/3522094459 wait complete. 2024-02-19T21:13:36.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:36.348+0000 7f6723e44700 1 librados: shutdown 2024-02-19T21:13:36.348 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:13:36.497 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:35.272461+0000 mgr.y (mgr.14550) 508 : cluster [DBG] pgmap v867: 222 pgs: 32 creating+peering, 190 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:36.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:35.337827+0000 mon.a (mon.0) 3012 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-99-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:13:36.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:35.337907+0000 mon.a (mon.0) 3013 : cluster [DBG] osdmap e579: 8 total, 8 up, 6 in 2024-02-19T21:13:36.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:35.340640+0000 mon.a (mon.0) 3014 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-99", "tierpool": "test-rados-api-smithi163-83928-99-cache"}]: dispatch 2024-02-19T21:13:36.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:35.369190+0000 mon.c (mon.2) 411 : audit [INF] from='client.? v1:172.21.15.163:0/3522094459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:13:36.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:35.369461+0000 mon.a (mon.0) 3015 : audit [INF] from='client.50216 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:13:36.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:35.272461+0000 mgr.y (mgr.14550) 508 : cluster [DBG] pgmap v867: 222 pgs: 32 creating+peering, 190 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:36.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:35.337827+0000 mon.a (mon.0) 3012 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-99-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:13:36.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:35.337907+0000 mon.a (mon.0) 3013 : cluster [DBG] osdmap e579: 8 total, 8 up, 6 in 2024-02-19T21:13:36.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:35.340640+0000 mon.a (mon.0) 3014 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-99", "tierpool": "test-rados-api-smithi163-83928-99-cache"}]: dispatch 2024-02-19T21:13:36.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:35.369190+0000 mon.c (mon.2) 411 : audit [INF] from='client.? v1:172.21.15.163:0/3522094459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:13:36.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:35.369461+0000 mon.a (mon.0) 3015 : audit [INF] from='client.50216 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:13:36.634 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-19T21:13:36.796 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:13:36.798 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:36.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:13:36.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:35.272461+0000 mgr.y (mgr.14550) 508 : cluster [DBG] pgmap v867: 222 pgs: 32 creating+peering, 190 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:36.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:35.337827+0000 mon.a (mon.0) 3012 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-99-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:13:36.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:35.337907+0000 mon.a (mon.0) 3013 : cluster [DBG] osdmap e579: 8 total, 8 up, 6 in 2024-02-19T21:13:36.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:35.340640+0000 mon.a (mon.0) 3014 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-99", "tierpool": "test-rados-api-smithi163-83928-99-cache"}]: dispatch 2024-02-19T21:13:36.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:35.369190+0000 mon.c (mon.2) 411 : audit [INF] from='client.? v1:172.21.15.163:0/3522094459' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:13:36.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:35.369461+0000 mon.a (mon.0) 3015 : audit [INF] from='client.50216 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:13:36.900 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:36.932 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:36.954 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:36.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:37.058 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:37.087 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:37.107 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:37.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:37.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:37.211 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:37.240 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:37.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:36.334695+0000 mon.a (mon.0) 3016 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:13:37.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:36.343212+0000 mon.a (mon.0) 3017 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-99", "tierpool": "test-rados-api-smithi163-83928-99-cache"}]': finished 2024-02-19T21:13:37.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:36.343293+0000 mon.a (mon.0) 3018 : audit [INF] from='client.50216 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:13:37.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:36.343342+0000 mon.a (mon.0) 3019 : cluster [DBG] osdmap e580: 8 total, 8 up, 6 in 2024-02-19T21:13:37.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:36.347121+0000 mon.a (mon.0) 3020 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-99", "overlaypool": "test-rados-api-smithi163-83928-99-cache"}]: dispatch 2024-02-19T21:13:37.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:37.344497+0000 mon.a (mon.0) 3021 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-99", "overlaypool": "test-rados-api-smithi163-83928-99-cache"}]': finished 2024-02-19T21:13:37.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:37.344553+0000 mon.a (mon.0) 3022 : cluster [DBG] osdmap e581: 8 total, 8 up, 6 in 2024-02-19T21:13:37.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:37.345367+0000 mon.a (mon.0) 3023 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-99-cache", "mode": "writeback"}]: dispatch 2024-02-19T21:13:37.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:36.334695+0000 mon.a (mon.0) 3016 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:13:37.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:36.343212+0000 mon.a (mon.0) 3017 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-99", "tierpool": "test-rados-api-smithi163-83928-99-cache"}]': finished 2024-02-19T21:13:37.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:36.343293+0000 mon.a (mon.0) 3018 : audit [INF] from='client.50216 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:13:37.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:36.343342+0000 mon.a (mon.0) 3019 : cluster [DBG] osdmap e580: 8 total, 8 up, 6 in 2024-02-19T21:13:37.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:36.347121+0000 mon.a (mon.0) 3020 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-99", "overlaypool": "test-rados-api-smithi163-83928-99-cache"}]: dispatch 2024-02-19T21:13:37.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:37.344497+0000 mon.a (mon.0) 3021 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-99", "overlaypool": "test-rados-api-smithi163-83928-99-cache"}]': finished 2024-02-19T21:13:37.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:37.344553+0000 mon.a (mon.0) 3022 : cluster [DBG] osdmap e581: 8 total, 8 up, 6 in 2024-02-19T21:13:37.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:37.345367+0000 mon.a (mon.0) 3023 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-99-cache", "mode": "writeback"}]: dispatch 2024-02-19T21:13:37.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:36.334695+0000 mon.a (mon.0) 3016 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:13:37.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:36.343212+0000 mon.a (mon.0) 3017 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-99", "tierpool": "test-rados-api-smithi163-83928-99-cache"}]': finished 2024-02-19T21:13:37.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:36.343293+0000 mon.a (mon.0) 3018 : audit [INF] from='client.50216 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:13:37.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:36.343342+0000 mon.a (mon.0) 3019 : cluster [DBG] osdmap e580: 8 total, 8 up, 6 in 2024-02-19T21:13:37.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:36.347121+0000 mon.a (mon.0) 3020 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-99", "overlaypool": "test-rados-api-smithi163-83928-99-cache"}]: dispatch 2024-02-19T21:13:37.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:37.344497+0000 mon.a (mon.0) 3021 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-99", "overlaypool": "test-rados-api-smithi163-83928-99-cache"}]': finished 2024-02-19T21:13:37.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:37.344553+0000 mon.a (mon.0) 3022 : cluster [DBG] osdmap e581: 8 total, 8 up, 6 in 2024-02-19T21:13:37.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:37.345367+0000 mon.a (mon.0) 3023 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-99-cache", "mode": "writeback"}]: dispatch 2024-02-19T21:13:37.890 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:37.893 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:37.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:37.994 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:13:38.022 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:38.071 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:13:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:13:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:13:38.372 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:13:38.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:38.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:13:38.476 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:38.508 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:38.529 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:38.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:38.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:38.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:36.769026+0000 osd.5 (osd.5) 228 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:13:38.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:36.771895+0000 osd.5 (osd.5) 229 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:13:38.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:36.862666+0000 osd.7 (osd.7) 255 : cluster [DBG] 2.27 scrub starts 2024-02-19T21:13:38.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:36.863725+0000 osd.7 (osd.7) 256 : cluster [DBG] 2.27 scrub ok 2024-02-19T21:13:38.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:37.272865+0000 mgr.y (mgr.14550) 509 : cluster [DBG] pgmap v870: 222 pgs: 32 creating+peering, 190 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:38.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:38.337158+0000 mon.a (mon.0) 3024 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:13:38.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:38.341307+0000 mon.a (mon.0) 3025 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-99-cache", "mode": "writeback"}]': finished 2024-02-19T21:13:38.618 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:38.341368+0000 mon.a (mon.0) 3026 : cluster [DBG] osdmap e582: 8 total, 8 up, 6 in 2024-02-19T21:13:38.619 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:38.344707+0000 mon.a (mon.0) 3027 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:13:38.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:36.769026+0000 osd.5 (osd.5) 228 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:13:38.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:36.771895+0000 osd.5 (osd.5) 229 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:13:38.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:36.862666+0000 osd.7 (osd.7) 255 : cluster [DBG] 2.27 scrub starts 2024-02-19T21:13:38.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:36.863725+0000 osd.7 (osd.7) 256 : cluster [DBG] 2.27 scrub ok 2024-02-19T21:13:38.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:37.272865+0000 mgr.y (mgr.14550) 509 : cluster [DBG] pgmap v870: 222 pgs: 32 creating+peering, 190 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:38.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:38.337158+0000 mon.a (mon.0) 3024 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:13:38.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:38.341307+0000 mon.a (mon.0) 3025 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-99-cache", "mode": "writeback"}]': finished 2024-02-19T21:13:38.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:38.341368+0000 mon.a (mon.0) 3026 : cluster [DBG] osdmap e582: 8 total, 8 up, 6 in 2024-02-19T21:13:38.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:38.344707+0000 mon.a (mon.0) 3027 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:13:38.633 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:38.664 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:38.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:36.769026+0000 osd.5 (osd.5) 228 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:13:38.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:36.771895+0000 osd.5 (osd.5) 229 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:13:38.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:36.862666+0000 osd.7 (osd.7) 255 : cluster [DBG] 2.27 scrub starts 2024-02-19T21:13:38.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:36.863725+0000 osd.7 (osd.7) 256 : cluster [DBG] 2.27 scrub ok 2024-02-19T21:13:38.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:37.272865+0000 mgr.y (mgr.14550) 509 : cluster [DBG] pgmap v870: 222 pgs: 32 creating+peering, 190 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:38.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:38.337158+0000 mon.a (mon.0) 3024 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:13:38.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:38.341307+0000 mon.a (mon.0) 3025 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-99-cache", "mode": "writeback"}]': finished 2024-02-19T21:13:38.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:38.341368+0000 mon.a (mon.0) 3026 : cluster [DBG] osdmap e582: 8 total, 8 up, 6 in 2024-02-19T21:13:38.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:38.344707+0000 mon.a (mon.0) 3027 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:13:38.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.870+0000 7fab219de700 1 -- 172.21.15.163:0/2353513631 >> v1:172.21.15.163:6790/0 conn(0x7fab1c0ff670 legacy=0x7fab1c101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:38.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.870+0000 7fab219de700 1 -- 172.21.15.163:0/2353513631 shutdown_connections 2024-02-19T21:13:38.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.870+0000 7fab219de700 1 -- 172.21.15.163:0/2353513631 >> 172.21.15.163:0/2353513631 conn(0x7fab1c0f2470 msgr2=0x7fab1c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:38.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.870+0000 7fab219de700 1 -- 172.21.15.163:0/2353513631 shutdown_connections 2024-02-19T21:13:38.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.870+0000 7fab219de700 1 -- 172.21.15.163:0/2353513631 wait complete. 2024-02-19T21:13:38.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.871+0000 7fab219de700 1 librados: starting msgr at 2024-02-19T21:13:38.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.871+0000 7fab219de700 1 librados: starting objecter 2024-02-19T21:13:38.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.871+0000 7fab219de700 1 Processor -- start 2024-02-19T21:13:38.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.871+0000 7fab219de700 1 -- start start 2024-02-19T21:13:38.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.871+0000 7fab219de700 1 librados: setting wanted keys 2024-02-19T21:13:38.871 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.871+0000 7fab219de700 1 librados: calling monclient init 2024-02-19T21:13:38.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab219de700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fab1c1162a0 con 0x7fab1c0fc870 2024-02-19T21:13:38.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab219de700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fab1c1174a0 con 0x7fab1c0ff670 2024-02-19T21:13:38.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab219de700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fab1c1186a0 con 0x7fab1c1031b0 2024-02-19T21:13:38.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab1b7fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fab1c1031b0 0x7fab1c114960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:58826/0 (socket says 172.21.15.163:58826) 2024-02-19T21:13:38.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab1b7fe700 1 -- 172.21.15.163:0/436261308 learned_addr learned my addr 172.21.15.163:0/436261308 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:38.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 10 client.?.objecter ms_handle_connect 0x7fab1c1031b0 2024-02-19T21:13:38.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:38.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 10 client.?.objecter ms_handle_connect 0x7fab1c0fc870 2024-02-19T21:13:38.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:38.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 10 client.?.objecter ms_handle_connect 0x7fab1c0ff670 2024-02-19T21:13:38.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:38.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292511811 0 0) 0x7fab1c1186a0 con 0x7fab1c1031b0 2024-02-19T21:13:38.873 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab040033f0 con 0x7fab1c1031b0 2024-02-19T21:13:38.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697458951 0 0) 0x7fab1c1174a0 con 0x7fab1c0ff670 2024-02-19T21:13:38.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab1c1186a0 con 0x7fab1c0ff670 2024-02-19T21:13:38.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489291400 0 0) 0x7fab1c1162a0 con 0x7fab1c0fc870 2024-02-19T21:13:38.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab1c1174a0 con 0x7fab1c0fc870 2024-02-19T21:13:38.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 115695595 0 0) 0x7fab040033f0 con 0x7fab1c1031b0 2024-02-19T21:13:38.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab1c1162a0 con 0x7fab1c1031b0 2024-02-19T21:13:38.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fab14002cd0 con 0x7fab1c1031b0 2024-02-19T21:13:38.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3610761738 0 0) 0x7fab1c1186a0 con 0x7fab1c0ff670 2024-02-19T21:13:38.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab040033f0 con 0x7fab1c0ff670 2024-02-19T21:13:38.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fab10001ff0 con 0x7fab1c0ff670 2024-02-19T21:13:38.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 958069027 0 0) 0x7fab1c1162a0 con 0x7fab1c1031b0 2024-02-19T21:13:38.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 >> v1:172.21.15.193:6789/0 conn(0x7fab1c0ff670 legacy=0x7fab1c0f9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:38.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.872+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 >> v1:172.21.15.163:6789/0 conn(0x7fab1c0fc870 legacy=0x7fab1c0f8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:38.874 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.873+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1c1198a0 con 0x7fab1c1031b0 2024-02-19T21:13:38.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.873+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1c116490 con 0x7fab1c1031b0 2024-02-19T21:13:38.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.873+0000 7fab219de700 10 client.50222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:38.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.873+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1c116900 con 0x7fab1c1031b0 2024-02-19T21:13:38.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.873+0000 7fab219de700 1 librados: init done 2024-02-19T21:13:38.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.873+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fab14003060 con 0x7fab1c1031b0 2024-02-19T21:13:38.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.873+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fab14005fb0 con 0x7fab1c1031b0 2024-02-19T21:13:38.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.874+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fab08004f00 con 0x7fab1c1031b0 2024-02-19T21:13:38.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.874+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fab14018e40 con 0x7fab1c1031b0 2024-02-19T21:13:38.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(582..582 src has 434..582) v4 ==== 9333+0+0 (unknown 2739142033 0 0) 0x7fab14079620 con 0x7fab1c1031b0 2024-02-19T21:13:38.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 osd_map(582..582 src has 434..582) v4 2024-02-19T21:13:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab03fff700 3 client.50222.objecter handle_osd_map got epochs [582,582] > 0 2024-02-19T21:13:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab03fff700 3 client.50222.objecter handle_osd_map decoding full epoch 582 2024-02-19T21:13:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab03fff700 20 client.50222.objecter dump_active .. 0 homeless 2024-02-19T21:13:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v582) v1 ==== 54+0+16 (unknown 1624389807 0 2420000192) 0x7fab1407ba30 con 0x7fab1c1031b0 2024-02-19T21:13:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab03fff700 10 client.50222.objecter ms_handle_connect 0x7fab04065bb0 2024-02-19T21:13:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab219de700 10 client.50222.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 1 2024-02-19T21:13:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab080051a0 osd=0 3 2024-02-19T21:13:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab080051a0 osd=0 3 2024-02-19T21:13:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab080051a0 osd=0 3 2024-02-19T21:13:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 1 2024-02-19T21:13:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab080051a0 osd=0 3 2024-02-19T21:13:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 0 1 2024-02-19T21:13:38.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab080051a0 osd=0 4 2024-02-19T21:13:38.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab219de700 10 client.50222.objecter _send_command 1 2024-02-19T21:13:38.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.875+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab08004950 con 0x7fab08004530 2024-02-19T21:13:38.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.876+0000 7fab03fff700 10 client.50222.objecter ms_handle_connect 0x7fab08004530 2024-02-19T21:13:38.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.876+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fab08004950 con 0x7fab08004530 2024-02-19T21:13:38.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.876+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:13:38.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.876+0000 7fab03fff700 10 client.50222.objecter _finish_command 1 = system:0 2024-02-19T21:13:38.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.876+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab080051a0 osd=0 4 2024-02-19T21:13:38.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.876+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 0 1 2024-02-19T21:13:38.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.902+0000 7fab219de700 10 client.50222.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:38.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.902+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 2 2024-02-19T21:13:38.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.902+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab080051a0 osd=0 3 2024-02-19T21:13:38.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.902+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab080051a0 osd=0 3 2024-02-19T21:13:38.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.902+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab080051a0 osd=0 3 2024-02-19T21:13:38.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.902+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 2 2024-02-19T21:13:38.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.902+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab080051a0 osd=0 3 2024-02-19T21:13:38.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.902+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 0 2 2024-02-19T21:13:38.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.902+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab080051a0 osd=0 4 2024-02-19T21:13:38.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.902+0000 7fab219de700 10 client.50222.objecter _send_command 2 2024-02-19T21:13:38.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.902+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fab08004d30 con 0x7fab08004530 2024-02-19T21:13:38.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.903+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fab08004d30 con 0x7fab08004530 2024-02-19T21:13:38.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.903+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:38.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.903+0000 7fab03fff700 10 client.50222.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:38.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.903+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab080051a0 osd=0 4 2024-02-19T21:13:38.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.903+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 0 2 2024-02-19T21:13:38.903 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:13:38.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.904+0000 7fab219de700 10 client.50222.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:38.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.904+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 3 2024-02-19T21:13:38.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.904+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08003fe0 osd=1 3 2024-02-19T21:13:38.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.904+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08003fe0 osd=1 3 2024-02-19T21:13:38.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.904+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08003fe0 osd=1 3 2024-02-19T21:13:38.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.904+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 3 2024-02-19T21:13:38.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.904+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab08003fe0 osd=1 3 2024-02-19T21:13:38.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.904+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 1 3 2024-02-19T21:13:38.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.904+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08003fe0 osd=1 4 2024-02-19T21:13:38.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.904+0000 7fab219de700 10 client.50222.objecter _send_command 3 2024-02-19T21:13:38.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.904+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab0800c000 con 0x7fab08004150 2024-02-19T21:13:38.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.904+0000 7fab03fff700 10 client.50222.objecter ms_handle_connect 0x7fab08004150 2024-02-19T21:13:38.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.905+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fab0800c000 con 0x7fab08004150 2024-02-19T21:13:38.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.905+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:13:38.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.905+0000 7fab03fff700 10 client.50222.objecter _finish_command 3 = system:0 2024-02-19T21:13:38.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.905+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab08003fe0 osd=1 4 2024-02-19T21:13:38.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.905+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 1 3 2024-02-19T21:13:38.915 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:38.917 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:38.920 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:38.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:38.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.925+0000 7fab219de700 10 client.50222.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:38.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.925+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 4 2024-02-19T21:13:38.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.925+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08003fe0 osd=1 3 2024-02-19T21:13:38.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.925+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08003fe0 osd=1 3 2024-02-19T21:13:38.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.925+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08003fe0 osd=1 3 2024-02-19T21:13:38.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.925+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 4 2024-02-19T21:13:38.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.925+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab08003fe0 osd=1 3 2024-02-19T21:13:38.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.925+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 1 4 2024-02-19T21:13:38.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.925+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08003fe0 osd=1 4 2024-02-19T21:13:38.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.925+0000 7fab219de700 10 client.50222.objecter _send_command 4 2024-02-19T21:13:38.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.925+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fab080018b0 con 0x7fab08004150 2024-02-19T21:13:38.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.926+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fab080018b0 con 0x7fab08004150 2024-02-19T21:13:38.925 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:38.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab03fff700 10 client.50222.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:38.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab08003fe0 osd=1 4 2024-02-19T21:13:38.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 1 4 2024-02-19T21:13:38.926 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:13:38.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab219de700 10 client.50222.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:38.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 5 2024-02-19T21:13:38.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08003630 osd=2 3 2024-02-19T21:13:38.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08003630 osd=2 3 2024-02-19T21:13:38.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08003630 osd=2 3 2024-02-19T21:13:38.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 5 2024-02-19T21:13:38.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab08003630 osd=2 3 2024-02-19T21:13:38.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 2 5 2024-02-19T21:13:38.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08003630 osd=2 4 2024-02-19T21:13:38.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab219de700 10 client.50222.objecter _send_command 5 2024-02-19T21:13:38.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab08003e40 con 0x7fab0800d2f0 2024-02-19T21:13:38.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.927+0000 7fab03fff700 10 client.50222.objecter ms_handle_connect 0x7fab0800d2f0 2024-02-19T21:13:38.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.928+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fab08003e40 con 0x7fab0800d2f0 2024-02-19T21:13:38.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.928+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:13:38.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.928+0000 7fab03fff700 10 client.50222.objecter _finish_command 5 = system:0 2024-02-19T21:13:38.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.928+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab08003630 osd=2 4 2024-02-19T21:13:38.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.928+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 2 5 2024-02-19T21:13:38.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.945+0000 7fab219de700 10 client.50222.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:38.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.945+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 6 2024-02-19T21:13:38.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.945+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08003630 osd=2 3 2024-02-19T21:13:38.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.945+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08003630 osd=2 3 2024-02-19T21:13:38.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.945+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08003630 osd=2 3 2024-02-19T21:13:38.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.945+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 6 2024-02-19T21:13:38.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.945+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab08003630 osd=2 3 2024-02-19T21:13:38.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.945+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 2 6 2024-02-19T21:13:38.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.945+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08003630 osd=2 4 2024-02-19T21:13:38.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.945+0000 7fab219de700 10 client.50222.objecter _send_command 6 2024-02-19T21:13:38.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.946+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fab0800e720 con 0x7fab0800d2f0 2024-02-19T21:13:38.947 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:13:38.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fab0800e720 con 0x7fab0800d2f0 2024-02-19T21:13:38.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:38.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab03fff700 10 client.50222.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:38.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab08003630 osd=2 4 2024-02-19T21:13:38.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 2 6 2024-02-19T21:13:38.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab219de700 10 client.50222.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:38.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 7 2024-02-19T21:13:38.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab080033c0 osd=3 3 2024-02-19T21:13:38.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab080033c0 osd=3 3 2024-02-19T21:13:38.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab080033c0 osd=3 3 2024-02-19T21:13:38.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 7 2024-02-19T21:13:38.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab080033c0 osd=3 3 2024-02-19T21:13:38.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 3 7 2024-02-19T21:13:38.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab080033c0 osd=3 4 2024-02-19T21:13:38.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab219de700 10 client.50222.objecter _send_command 7 2024-02-19T21:13:38.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.948+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab08015a10 con 0x7fab08011e50 2024-02-19T21:13:38.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.949+0000 7fab03fff700 10 client.50222.objecter ms_handle_connect 0x7fab08011e50 2024-02-19T21:13:38.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.949+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fab08015a10 con 0x7fab08011e50 2024-02-19T21:13:38.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.949+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:13:38.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.949+0000 7fab03fff700 10 client.50222.objecter _finish_command 7 = system:0 2024-02-19T21:13:38.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.949+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab080033c0 osd=3 4 2024-02-19T21:13:38.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.949+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 3 7 2024-02-19T21:13:38.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.965+0000 7fab219de700 10 client.50222.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:38.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.965+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 8 2024-02-19T21:13:38.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.965+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab080033c0 osd=3 3 2024-02-19T21:13:38.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.965+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab080033c0 osd=3 3 2024-02-19T21:13:38.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.965+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab080033c0 osd=3 3 2024-02-19T21:13:38.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.965+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 8 2024-02-19T21:13:38.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.965+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab080033c0 osd=3 3 2024-02-19T21:13:38.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.965+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 3 8 2024-02-19T21:13:38.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.965+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab080033c0 osd=3 4 2024-02-19T21:13:38.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.965+0000 7fab219de700 10 client.50222.objecter _send_command 8 2024-02-19T21:13:38.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.965+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fab08015a10 con 0x7fab08011e50 2024-02-19T21:13:38.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fab08015a10 con 0x7fab08011e50 2024-02-19T21:13:38.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:38.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab03fff700 10 client.50222.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:38.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab080033c0 osd=3 4 2024-02-19T21:13:38.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 3 8 2024-02-19T21:13:38.966 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:13:38.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab219de700 10 client.50222.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:38.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 9 2024-02-19T21:13:38.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08015a10 osd=4 3 2024-02-19T21:13:38.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08015a10 osd=4 3 2024-02-19T21:13:38.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08015a10 osd=4 3 2024-02-19T21:13:38.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 9 2024-02-19T21:13:38.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab08015a10 osd=4 3 2024-02-19T21:13:38.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 4 9 2024-02-19T21:13:38.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08015a10 osd=4 4 2024-02-19T21:13:38.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab219de700 10 client.50222.objecter _send_command 9 2024-02-19T21:13:38.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab080162c0 con 0x7fab0800ce50 2024-02-19T21:13:38.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.967+0000 7fab03fff700 10 client.50222.objecter ms_handle_connect 0x7fab0800ce50 2024-02-19T21:13:38.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.968+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fab080162c0 con 0x7fab0800ce50 2024-02-19T21:13:38.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.968+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:13:38.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.968+0000 7fab03fff700 10 client.50222.objecter _finish_command 9 = system:0 2024-02-19T21:13:38.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.968+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab08015a10 osd=4 4 2024-02-19T21:13:38.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.968+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 4 9 2024-02-19T21:13:38.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.982+0000 7fab219de700 10 client.50222.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:38.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.982+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 10 2024-02-19T21:13:38.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.982+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08015a10 osd=4 3 2024-02-19T21:13:38.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.982+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08015a10 osd=4 3 2024-02-19T21:13:38.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.982+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08015a10 osd=4 3 2024-02-19T21:13:38.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.982+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 10 2024-02-19T21:13:38.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.982+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab08015a10 osd=4 3 2024-02-19T21:13:38.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.982+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 4 10 2024-02-19T21:13:38.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.982+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08015a10 osd=4 4 2024-02-19T21:13:38.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.982+0000 7fab219de700 10 client.50222.objecter _send_command 10 2024-02-19T21:13:38.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.982+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fab080162c0 con 0x7fab0800ce50 2024-02-19T21:13:38.983 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:13:38.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.984+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fab080162c0 con 0x7fab0800ce50 2024-02-19T21:13:38.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.984+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:38.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.984+0000 7fab03fff700 10 client.50222.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:38.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.984+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab08015a10 osd=4 4 2024-02-19T21:13:38.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.984+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 4 10 2024-02-19T21:13:38.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.985+0000 7fab219de700 10 client.50222.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:38.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.985+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 11 2024-02-19T21:13:38.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.985+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab080162c0 osd=5 3 2024-02-19T21:13:38.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.985+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab080162c0 osd=5 3 2024-02-19T21:13:38.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.985+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab080162c0 osd=5 3 2024-02-19T21:13:38.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.985+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 11 2024-02-19T21:13:38.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.985+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab080162c0 osd=5 3 2024-02-19T21:13:38.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.985+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 5 11 2024-02-19T21:13:38.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.985+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab080162c0 osd=5 4 2024-02-19T21:13:38.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.985+0000 7fab219de700 10 client.50222.objecter _send_command 11 2024-02-19T21:13:38.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.985+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab08016ee0 con 0x7fab08016970 2024-02-19T21:13:38.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.985+0000 7fab03fff700 10 client.50222.objecter ms_handle_connect 0x7fab08016970 2024-02-19T21:13:38.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.986+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fab08016ee0 con 0x7fab08016970 2024-02-19T21:13:38.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.986+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:13:38.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.986+0000 7fab03fff700 10 client.50222.objecter _finish_command 11 = system:0 2024-02-19T21:13:38.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.986+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab080162c0 osd=5 4 2024-02-19T21:13:38.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:38.986+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 5 11 2024-02-19T21:13:38.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.000+0000 7fab219de700 10 client.50222.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:38.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.000+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 12 2024-02-19T21:13:38.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.000+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab080162c0 osd=5 3 2024-02-19T21:13:38.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.000+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab080162c0 osd=5 3 2024-02-19T21:13:38.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.000+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab080162c0 osd=5 3 2024-02-19T21:13:38.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.000+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 12 2024-02-19T21:13:38.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.000+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab080162c0 osd=5 3 2024-02-19T21:13:38.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.000+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 5 12 2024-02-19T21:13:38.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.000+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab080162c0 osd=5 4 2024-02-19T21:13:39.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.000+0000 7fab219de700 10 client.50222.objecter _send_command 12 2024-02-19T21:13:39.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.000+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fab08016ee0 con 0x7fab08016970 2024-02-19T21:13:39.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fab08016ee0 con 0x7fab08016970 2024-02-19T21:13:39.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:39.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab03fff700 10 client.50222.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:39.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab080162c0 osd=5 4 2024-02-19T21:13:39.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 5 12 2024-02-19T21:13:39.001 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:13:39.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab219de700 10 client.50222.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:39.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 13 2024-02-19T21:13:39.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08016ee0 osd=6 3 2024-02-19T21:13:39.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08016ee0 osd=6 3 2024-02-19T21:13:39.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08016ee0 osd=6 3 2024-02-19T21:13:39.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 13 2024-02-19T21:13:39.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab08016ee0 osd=6 3 2024-02-19T21:13:39.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 6 13 2024-02-19T21:13:39.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08016ee0 osd=6 4 2024-02-19T21:13:39.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab219de700 10 client.50222.objecter _send_command 13 2024-02-19T21:13:39.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.002+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab08022c80 con 0x7fab0801f010 2024-02-19T21:13:39.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.003+0000 7fab03fff700 10 client.50222.objecter ms_handle_connect 0x7fab0801f010 2024-02-19T21:13:39.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.003+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fab08022c80 con 0x7fab0801f010 2024-02-19T21:13:39.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.003+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:13:39.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.003+0000 7fab03fff700 10 client.50222.objecter _finish_command 13 = system:0 2024-02-19T21:13:39.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.003+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab08016ee0 osd=6 4 2024-02-19T21:13:39.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.003+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 6 13 2024-02-19T21:13:39.016 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.017+0000 7fab219de700 10 client.50222.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:39.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.017+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 14 2024-02-19T21:13:39.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.017+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08016ee0 osd=6 3 2024-02-19T21:13:39.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.017+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08016ee0 osd=6 3 2024-02-19T21:13:39.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.017+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08016ee0 osd=6 3 2024-02-19T21:13:39.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.017+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 14 2024-02-19T21:13:39.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.017+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab08016ee0 osd=6 3 2024-02-19T21:13:39.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.017+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 6 14 2024-02-19T21:13:39.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.017+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08016ee0 osd=6 4 2024-02-19T21:13:39.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.017+0000 7fab219de700 10 client.50222.objecter _send_command 14 2024-02-19T21:13:39.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.017+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fab08024430 con 0x7fab0801f010 2024-02-19T21:13:39.022 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:39.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fab08024430 con 0x7fab0801f010 2024-02-19T21:13:39.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:39.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab03fff700 10 client.50222.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:39.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab08016ee0 osd=6 4 2024-02-19T21:13:39.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 6 14 2024-02-19T21:13:39.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab219de700 10 client.50222.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:39.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 15 2024-02-19T21:13:39.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08024430 osd=7 3 2024-02-19T21:13:39.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08024430 osd=7 3 2024-02-19T21:13:39.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08024430 osd=7 3 2024-02-19T21:13:39.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 15 2024-02-19T21:13:39.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab08024430 osd=7 3 2024-02-19T21:13:39.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 7 15 2024-02-19T21:13:39.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08024430 osd=7 4 2024-02-19T21:13:39.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab219de700 10 client.50222.objecter _send_command 15 2024-02-19T21:13:39.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.019+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab08025060 con 0x7fab08024ae0 2024-02-19T21:13:39.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.020+0000 7fab03fff700 10 client.50222.objecter ms_handle_connect 0x7fab08024ae0 2024-02-19T21:13:39.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.020+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fab08025060 con 0x7fab08024ae0 2024-02-19T21:13:39.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.020+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:13:39.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.020+0000 7fab03fff700 10 client.50222.objecter _finish_command 15 = system:0 2024-02-19T21:13:39.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.020+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab08024430 osd=7 4 2024-02-19T21:13:39.025 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.020+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 7 15 2024-02-19T21:13:39.025 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:13:39.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.035+0000 7fab219de700 10 client.50222.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:39.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.035+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign -1 16 2024-02-19T21:13:39.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.035+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08024430 osd=7 3 2024-02-19T21:13:39.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.035+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08024430 osd=7 3 2024-02-19T21:13:39.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.035+0000 7fab219de700 20 client.50222.objecter _get_session s=0x7fab08024430 osd=7 3 2024-02-19T21:13:39.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.035+0000 7fab219de700 15 client.50222.objecter _session_command_op_remove -1 16 2024-02-19T21:13:39.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.035+0000 7fab219de700 20 client.50222.objecter get_session s=0x7fab08024430 osd=7 3 2024-02-19T21:13:39.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.035+0000 7fab219de700 15 client.50222.objecter _session_command_op_assign 7 16 2024-02-19T21:13:39.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.035+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08024430 osd=7 4 2024-02-19T21:13:39.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.035+0000 7fab219de700 10 client.50222.objecter _send_command 16 2024-02-19T21:13:39.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.035+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fab08025060 con 0x7fab08024ae0 2024-02-19T21:13:39.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab03fff700 1 -- 172.21.15.163:0/436261308 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fab08025060 con 0x7fab08024ae0 2024-02-19T21:13:39.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab03fff700 10 client.50222.objecter ms_dispatch 0x7fab1c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:39.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab03fff700 10 client.50222.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:39.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab03fff700 20 client.50222.objecter put_session s=0x7fab08024430 osd=7 4 2024-02-19T21:13:39.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab03fff700 15 client.50222.objecter _session_command_op_remove 7 16 2024-02-19T21:13:39.037 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:13:39.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab219de700 10 librados: watch_flush enter 2024-02-19T21:13:39.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab219de700 10 librados: watch_flush exit 2024-02-19T21:13:39.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab219de700 10 client.50222.objecter close_session for osd.0 2024-02-19T21:13:39.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 >> v1:172.21.15.163:6801/1559418115 conn(0x7fab08004530 legacy=0x7fab08002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:39.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab080051a0 osd=0 1 2024-02-19T21:13:39.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab219de700 10 client.50222.objecter close_session for osd.1 2024-02-19T21:13:39.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 >> v1:172.21.15.163:6805/3125047432 conn(0x7fab08004150 legacy=0x7fab0800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:39.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08003fe0 osd=1 1 2024-02-19T21:13:39.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab219de700 10 client.50222.objecter close_session for osd.2 2024-02-19T21:13:39.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 >> v1:172.21.15.163:6809/1563158487 conn(0x7fab0800d2f0 legacy=0x7fab0800d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:39.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08003630 osd=2 1 2024-02-19T21:13:39.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab219de700 10 client.50222.objecter close_session for osd.3 2024-02-19T21:13:39.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.037+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 >> v1:172.21.15.163:6813/2112279769 conn(0x7fab08011e50 legacy=0x7fab08014240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:39.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab080033c0 osd=3 1 2024-02-19T21:13:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 10 client.50222.objecter close_session for osd.4 2024-02-19T21:13:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 >> v1:172.21.15.193:6800/3129716965 conn(0x7fab0800ce50 legacy=0x7fab08015bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab1b7fe700 1 -- 172.21.15.163:0/436261308 reap_dead start 2024-02-19T21:13:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08015a10 osd=4 1 2024-02-19T21:13:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 10 client.50222.objecter close_session for osd.5 2024-02-19T21:13:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 >> v1:172.21.15.193:6804/805161991 conn(0x7fab08016970 legacy=0x7fab0801d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab080162c0 osd=5 1 2024-02-19T21:13:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab1b7fe700 1 -- 172.21.15.163:0/436261308 reap_dead start 2024-02-19T21:13:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 10 client.50222.objecter close_session for osd.6 2024-02-19T21:13:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 >> v1:172.21.15.193:6808/787604615 conn(0x7fab0801f010 legacy=0x7fab08021400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08016ee0 osd=6 1 2024-02-19T21:13:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 10 client.50222.objecter close_session for osd.7 2024-02-19T21:13:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 >> v1:172.21.15.193:6812/157927432 conn(0x7fab08024ae0 legacy=0x7fab08022c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:39.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 20 client.50222.objecter put_session s=0x7fab08024430 osd=7 1 2024-02-19T21:13:39.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 20 client.50222.objecter shutdown clearing up homeless session... 2024-02-19T21:13:39.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 10 client.50222.objecter successfully canceled tick 2024-02-19T21:13:39.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab1b7fe700 1 -- 172.21.15.163:0/436261308 reap_dead start 2024-02-19T21:13:39.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 >> v1:172.21.15.163:6800/1552514681 conn(0x7fab04065bb0 legacy=0x7fab04068070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:39.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 >> v1:172.21.15.163:6790/0 conn(0x7fab1c1031b0 legacy=0x7fab1c114960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:39.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 shutdown_connections 2024-02-19T21:13:39.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 >> 172.21.15.163:0/436261308 conn(0x7fab1c0f2470 msgr2=0x7fab1c106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:39.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 shutdown_connections 2024-02-19T21:13:39.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 1 -- 172.21.15.163:0/436261308 wait complete. 2024-02-19T21:13:39.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:39.038+0000 7fab219de700 1 librados: shutdown 2024-02-19T21:13:39.041 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:13:39.041 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:13:39.041 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:13:39.041 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:13:39.042 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:13:39.042 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:13:39.042 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:13:39.042 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:13:39.049 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:37.824231+0000 osd.7 (osd.7) 257 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:13:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:37.828903+0000 osd.7 (osd.7) 258 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:13:39.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:38.070715+0000 osd.1 (osd.1) 264 : cluster [DBG] 2.a scrub starts 2024-02-19T21:13:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:38.073663+0000 osd.1 (osd.1) 265 : cluster [DBG] 2.a scrub ok 2024-02-19T21:13:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:38.127319+0000 osd.2 (osd.2) 228 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:38.127477+0000 osd.2 (osd.2) 229 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:38.875360+0000 mon.c (mon.2) 412 : audit [DBG] from='client.? v1:172.21.15.163:0/436261308' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:39.106191+0000 mon.a (mon.0) 3028 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:39.343334+0000 mon.a (mon.0) 3029 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:13:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:39.343400+0000 mon.a (mon.0) 3030 : cluster [DBG] osdmap e583: 8 total, 8 up, 6 in 2024-02-19T21:13:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:39.345113+0000 mon.a (mon.0) 3031 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:13:39.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:37.824231+0000 osd.7 (osd.7) 257 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:13:39.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:37.828903+0000 osd.7 (osd.7) 258 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:13:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:38.070715+0000 osd.1 (osd.1) 264 : cluster [DBG] 2.a scrub starts 2024-02-19T21:13:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:38.073663+0000 osd.1 (osd.1) 265 : cluster [DBG] 2.a scrub ok 2024-02-19T21:13:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:38.127319+0000 osd.2 (osd.2) 228 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:38.127477+0000 osd.2 (osd.2) 229 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:38.875360+0000 mon.c (mon.2) 412 : audit [DBG] from='client.? v1:172.21.15.163:0/436261308' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:39.106191+0000 mon.a (mon.0) 3028 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:39.343334+0000 mon.a (mon.0) 3029 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:13:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:39.343400+0000 mon.a (mon.0) 3030 : cluster [DBG] osdmap e583: 8 total, 8 up, 6 in 2024-02-19T21:13:39.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:39.345113+0000 mon.a (mon.0) 3031 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:13:39.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:37.824231+0000 osd.7 (osd.7) 257 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:13:39.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:37.828903+0000 osd.7 (osd.7) 258 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:13:39.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:38.070715+0000 osd.1 (osd.1) 264 : cluster [DBG] 2.a scrub starts 2024-02-19T21:13:39.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:38.073663+0000 osd.1 (osd.1) 265 : cluster [DBG] 2.a scrub ok 2024-02-19T21:13:39.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:38.127319+0000 osd.2 (osd.2) 228 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:39.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:38.127477+0000 osd.2 (osd.2) 229 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:39.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:38.875360+0000 mon.c (mon.2) 412 : audit [DBG] from='client.? v1:172.21.15.163:0/436261308' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:39.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:39.106191+0000 mon.a (mon.0) 3028 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:39.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:39.343334+0000 mon.a (mon.0) 3029 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:13:39.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:39.343400+0000 mon.a (mon.0) 3030 : cluster [DBG] osdmap e583: 8 total, 8 up, 6 in 2024-02-19T21:13:39.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:39.345113+0000 mon.a (mon.0) 3031 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:13:39.893 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:13:39.935 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 7, 3, 4] out_osds: [0, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:13:39.935 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:13:39.936 INFO:tasks.thrashosds.thrasher:fixing pg num pool device_health_metrics 2024-02-19T21:13:39.936 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:13:40.098 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:40.101 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:40.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:40.202 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:40.234 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:40.255 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:40.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:40.358 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:40.388 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:40.407 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:40.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:40.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:38.809950+0000 osd.7 (osd.7) 259 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:13:40.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:38.813278+0000 osd.7 (osd.7) 260 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:13:40.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:39.273410+0000 mgr.y (mgr.14550) 510 : cluster [DBG] pgmap v873: 222 pgs: 32 creating+peering, 190 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:40.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:39.511083+0000 osd.2 (osd.2) 230 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:13:40.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:39.513600+0000 osd.2 (osd.2) 231 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:13:40.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:39.520588+0000 osd.5 (osd.5) 230 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:40.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:39.521070+0000 osd.5 (osd.5) 231 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:40.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:39.690015+0000 osd.4 (osd.4) 205 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:13:40.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:39.691286+0000 osd.4 (osd.4) 206 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:13:40.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:40.303555+0000 osd.3 (osd.3) 243 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:13:40.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:40.304483+0000 osd.3 (osd.3) 244 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:13:40.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:40.346077+0000 mon.a (mon.0) 3032 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:13:40.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:40.346140+0000 mon.a (mon.0) 3033 : cluster [DBG] osdmap e584: 8 total, 8 up, 6 in 2024-02-19T21:13:40.488 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:40.348677+0000 mon.a (mon.0) 3034 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:13:40.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:38.809950+0000 osd.7 (osd.7) 259 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:13:40.488 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:38.813278+0000 osd.7 (osd.7) 260 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:13:40.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:39.273410+0000 mgr.y (mgr.14550) 510 : cluster [DBG] pgmap v873: 222 pgs: 32 creating+peering, 190 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:40.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:39.511083+0000 osd.2 (osd.2) 230 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:13:40.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:39.513600+0000 osd.2 (osd.2) 231 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:13:40.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:39.520588+0000 osd.5 (osd.5) 230 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:40.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:39.521070+0000 osd.5 (osd.5) 231 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:40.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:39.690015+0000 osd.4 (osd.4) 205 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:13:40.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:39.691286+0000 osd.4 (osd.4) 206 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:13:40.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:40.303555+0000 osd.3 (osd.3) 243 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:13:40.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:40.304483+0000 osd.3 (osd.3) 244 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:13:40.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:40.346077+0000 mon.a (mon.0) 3032 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:13:40.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:40.346140+0000 mon.a (mon.0) 3033 : cluster [DBG] osdmap e584: 8 total, 8 up, 6 in 2024-02-19T21:13:40.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:40.348677+0000 mon.a (mon.0) 3034 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:13:40.511 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:40.541 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:40.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:38.809950+0000 osd.7 (osd.7) 259 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:13:40.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:38.813278+0000 osd.7 (osd.7) 260 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:13:40.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:39.273410+0000 mgr.y (mgr.14550) 510 : cluster [DBG] pgmap v873: 222 pgs: 32 creating+peering, 190 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:40.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:39.511083+0000 osd.2 (osd.2) 230 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:13:40.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:39.513600+0000 osd.2 (osd.2) 231 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:13:40.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:39.520588+0000 osd.5 (osd.5) 230 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:40.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:39.521070+0000 osd.5 (osd.5) 231 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:40.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:39.690015+0000 osd.4 (osd.4) 205 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:13:40.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:39.691286+0000 osd.4 (osd.4) 206 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:13:40.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:40.303555+0000 osd.3 (osd.3) 243 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:13:40.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:40.304483+0000 osd.3 (osd.3) 244 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:13:40.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:40.346077+0000 mon.a (mon.0) 3032 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:13:40.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:40.346140+0000 mon.a (mon.0) 3033 : cluster [DBG] osdmap e584: 8 total, 8 up, 6 in 2024-02-19T21:13:40.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:40.348677+0000 mon.a (mon.0) 3034 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:13:41.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:13:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:13:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:40.589514+0000 mgr.y (mgr.14550) 511 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:41.341427+0000 mon.a (mon.0) 3035 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:13:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:41.345621+0000 mon.a (mon.0) 3036 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-19T21:13:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:41.345684+0000 mon.a (mon.0) 3037 : cluster [DBG] osdmap e585: 8 total, 8 up, 6 in 2024-02-19T21:13:41.493 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:41.346880+0000 mon.a (mon.0) 3038 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-19T21:13:41.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:40.589514+0000 mgr.y (mgr.14550) 511 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:41.493 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:41.341427+0000 mon.a (mon.0) 3035 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:13:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:41.345621+0000 mon.a (mon.0) 3036 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-19T21:13:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:41.345684+0000 mon.a (mon.0) 3037 : cluster [DBG] osdmap e585: 8 total, 8 up, 6 in 2024-02-19T21:13:41.494 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:41.346880+0000 mon.a (mon.0) 3038 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-19T21:13:41.719 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:41.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:41.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:40.589514+0000 mgr.y (mgr.14550) 511 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:41.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:41.341427+0000 mon.a (mon.0) 3035 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:13:41.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:41.345621+0000 mon.a (mon.0) 3036 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-19T21:13:41.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:41.345684+0000 mon.a (mon.0) 3037 : cluster [DBG] osdmap e585: 8 total, 8 up, 6 in 2024-02-19T21:13:41.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:41.346880+0000 mon.a (mon.0) 3038 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-02-19T21:13:41.823 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:41.852 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:41.872 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:41.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:41.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:41.979 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:42.011 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:42.033 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:42.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:42.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:42.136 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:42.166 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:42.185 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:42.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:42.293 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:42.323 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:42.343 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:42.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:42.368 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:40.806781+0000 osd.7 (osd.7) 261 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:13:42.368 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:40.809794+0000 osd.7 (osd.7) 262 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:13:42.368 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:41.274095+0000 mgr.y (mgr.14550) 512 : cluster [DBG] pgmap v876: 222 pgs: 222 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:42.368 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:41.600849+0000 osd.2 (osd.2) 232 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:13:42.368 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:41.604177+0000 osd.2 (osd.2) 233 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:13:42.368 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:41.705575+0000 osd.4 (osd.4) 207 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:13:42.368 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:41.706492+0000 osd.4 (osd.4) 208 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:13:42.368 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:42.346798+0000 mon.a (mon.0) 3039 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-02-19T21:13:42.369 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:42.346854+0000 mon.a (mon.0) 3040 : cluster [DBG] osdmap e586: 8 total, 8 up, 6 in 2024-02-19T21:13:42.450 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:42.479 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:42.656 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:42.660 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:42.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:42.661 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:42.661 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:40.806781+0000 osd.7 (osd.7) 261 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:13:42.661 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:40.809794+0000 osd.7 (osd.7) 262 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:13:42.661 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:41.274095+0000 mgr.y (mgr.14550) 512 : cluster [DBG] pgmap v876: 222 pgs: 222 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:42.661 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:41.600849+0000 osd.2 (osd.2) 232 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:13:42.661 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:41.604177+0000 osd.2 (osd.2) 233 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:13:42.662 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:41.705575+0000 osd.4 (osd.4) 207 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:13:42.662 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:41.706492+0000 osd.4 (osd.4) 208 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:13:42.662 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:42.346798+0000 mon.a (mon.0) 3039 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-02-19T21:13:42.662 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:42.346854+0000 mon.a (mon.0) 3040 : cluster [DBG] osdmap e586: 8 total, 8 up, 6 in 2024-02-19T21:13:42.663 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:42.671 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:13:42.672 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-02-19T21:13:42.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-02-19T21:13:42.672 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-02-19T21:13:42.672 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.672 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24919.0:2074 28.18 28:19125b2f:test-rados-api-smithi163-83920-17::big3:head [writefull 0~25165828] snapc 0=[] ondisk+write+known_if_redirected e102)", 2024-02-19T21:13:42.672 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:42.228484+0000", 2024-02-19T21:13:42.672 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 540.441669714, 2024-02-19T21:13:42.672 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.066405645999999999, 2024-02-19T21:13:42.672 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.672 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:13:42.672 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:13:42.672 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24919", 2024-02-19T21:13:42.672 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/4089460683", 2024-02-19T21:13:42.673 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2074 2024-02-19T21:13:42.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.673 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.228484+0000", 2024-02-19T21:13:42.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.673 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.228484+0000", 2024-02-19T21:13:42.673 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999967 2024-02-19T21:13:42.673 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.673 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.673 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.228481+0000", 2024-02-19T21:13:42.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0067282870000000003 2024-02-19T21:13:42.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235209+0000", 2024-02-19T21:13:42.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8715999999999998e-05 2024-02-19T21:13:42.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.674 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.674 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.674 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235228+0000", 2024-02-19T21:13:42.674 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7010000000000002e-06 2024-02-19T21:13:42.674 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235237+0000", 2024-02-19T21:13:42.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5971000000000003e-05 2024-02-19T21:13:42.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235292+0000", 2024-02-19T21:13:42.675 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000144649 2024-02-19T21:13:42.675 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.675 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.675 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.675 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235437+0000", 2024-02-19T21:13:42.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000111014 2024-02-19T21:13:42.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.676 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,5", 2024-02-19T21:13:42.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.235548+0000", 2024-02-19T21:13:42.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.050371437999999998 2024-02-19T21:13:42.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.676 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:13:42.676 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.285920+0000", 2024-02-19T21:13:42.676 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0084572139999999994 2024-02-19T21:13:42.676 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.676 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:13:42.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.294377+0000", 2024-02-19T21:13:42.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040791500000000001 2024-02-19T21:13:42.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:13:42.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.294785+0000", 2024-02-19T21:13:42.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6798000000000001e-05 2024-02-19T21:13:42.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.677 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.677 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:13:42.677 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.294842+0000", 2024-02-19T21:13:42.677 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8254000000000003e-05 2024-02-19T21:13:42.677 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.678 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.678 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:42.294890+0000", 2024-02-19T21:13:42.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.066405645999999999 2024-02-19T21:13:42.678 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.678 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.678 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.678 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.678 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.678 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25258.0:1 45.6s0 45:6dd63e69:test-rados-api-smithi163-83864-16::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e106)", 2024-02-19T21:13:42.678 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:45.971389+0000", 2024-02-19T21:13:42.678 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 536.69876516500005, 2024-02-19T21:13:42.678 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0304329539999999, 2024-02-19T21:13:42.678 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.678 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:13:42.679 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:13:42.679 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25258", 2024-02-19T21:13:42.679 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3498182538", 2024-02-19T21:13:42.679 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:13:42.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.679 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971389+0000", 2024-02-19T21:13:42.679 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.679 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.679 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.679 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.679 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971389+0000", 2024-02-19T21:13:42.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999967 2024-02-19T21:13:42.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971386+0000", 2024-02-19T21:13:42.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4117000000000001e-05 2024-02-19T21:13:42.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971400+0000", 2024-02-19T21:13:42.680 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.344e-05 2024-02-19T21:13:42.680 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.680 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.680 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.680 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971413+0000", 2024-02-19T21:13:42.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8219999999999996e-06 2024-02-19T21:13:42.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971422+0000", 2024-02-19T21:13:42.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2419999999999997e-05 2024-02-19T21:13:42.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971464+0000", 2024-02-19T21:13:42.681 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3889999999999998e-06 2024-02-19T21:13:42.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.681 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.681 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:13:42.681 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:45.971469+0000", 2024-02-19T21:13:42.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0243061849999999 2024-02-19T21:13:42.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.682 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.682 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.995775+0000", 2024-02-19T21:13:42.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012886000000000001 2024-02-19T21:13:42.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.682 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.682 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.995904+0000", 2024-02-19T21:13:42.682 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023266189999999998 2024-02-19T21:13:42.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.682 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.682 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:13:42.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:46.998230+0000", 2024-02-19T21:13:42.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033170790000000001 2024-02-19T21:13:42.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:13:42.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.001547+0000", 2024-02-19T21:13:42.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9880000000000002e-05 2024-02-19T21:13:42.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.683 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:13:42.683 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.001617+0000", 2024-02-19T21:13:42.683 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000204291 2024-02-19T21:13:42.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.683 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.684 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.684 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.001822+0000", 2024-02-19T21:13:42.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0304329539999999 2024-02-19T21:13:42.684 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.684 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.684 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.684 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.684 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25318.0:2 48.6 48:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:13:42.684 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:47.974844+0000", 2024-02-19T21:13:42.684 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 534.69531019999999, 2024-02-19T21:13:42.684 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.048850169999999998, 2024-02-19T21:13:42.684 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.684 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:13:42.685 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:13:42.685 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25318", 2024-02-19T21:13:42.685 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/4284884290", 2024-02-19T21:13:42.685 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:13:42.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.685 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974844+0000", 2024-02-19T21:13:42.685 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.685 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.685 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.685 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974844+0000", 2024-02-19T21:13:42.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:13:42.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974841+0000", 2024-02-19T21:13:42.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2949999999999999e-06 2024-02-19T21:13:42.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.686 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974850+0000", 2024-02-19T21:13:42.686 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1317e-05 2024-02-19T21:13:42.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.686 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.686 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974862+0000", 2024-02-19T21:13:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6129999999999998e-06 2024-02-19T21:13:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.974869+0000", 2024-02-19T21:13:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018860630000000001 2024-02-19T21:13:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.976755+0000", 2024-02-19T21:13:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2200000000000003e-06 2024-02-19T21:13:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:13:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.976762+0000", 2024-02-19T21:13:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043147748 2024-02-19T21:13:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.019910+0000", 2024-02-19T21:13:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9196000000000006e-05 2024-02-19T21:13:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.019979+0000", 2024-02-19T21:13:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000141478 2024-02-19T21:13:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,5", 2024-02-19T21:13:42.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.020121+0000", 2024-02-19T21:13:42.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00053344600000000003 2024-02-19T21:13:42.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:13:42.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.020654+0000", 2024-02-19T21:13:42.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001117311 2024-02-19T21:13:42.689 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.689 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.689 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:13:42.689 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.021772+0000", 2024-02-19T21:13:42.689 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001839909 2024-02-19T21:13:42.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:13:42.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.023612+0000", 2024-02-19T21:13:42.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2648000000000002e-05 2024-02-19T21:13:42.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:13:42.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.023654+0000", 2024-02-19T21:13:42.690 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9617999999999999e-05 2024-02-19T21:13:42.690 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.690 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.690 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.690 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.023694+0000", 2024-02-19T21:13:42.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.048850169999999998 2024-02-19T21:13:42.691 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.691 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.691 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.691 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.691 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.691 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25294.0:1 51.3s0 51:d77cf283:test-rados-api-smithi163-83896-11::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:13:42.691 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:47.988394+0000", 2024-02-19T21:13:42.691 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 534.68175957400001, 2024-02-19T21:13:42.691 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.593213569, 2024-02-19T21:13:42.691 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.691 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:13:42.691 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:13:42.691 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25294", 2024-02-19T21:13:42.692 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2922168330", 2024-02-19T21:13:42.692 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:13:42.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.692 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988394+0000", 2024-02-19T21:13:42.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.692 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.692 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.692 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988394+0000", 2024-02-19T21:13:42.692 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:13:42.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.693 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.693 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.693 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988392+0000", 2024-02-19T21:13:42.693 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9700000000000002e-06 2024-02-19T21:13:42.693 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.694 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.694 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.694 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988399+0000", 2024-02-19T21:13:42.694 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3440000000000001e-06 2024-02-19T21:13:42.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988408+0000", 2024-02-19T21:13:42.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2559999999999996e-06 2024-02-19T21:13:42.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.695 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.695 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988415+0000", 2024-02-19T21:13:42.695 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013816699999999999 2024-02-19T21:13:42.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.695 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988553+0000", 2024-02-19T21:13:42.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.056e-06 2024-02-19T21:13:42.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:13:42.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:47.988556+0000", 2024-02-19T21:13:42.696 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.57455238200000003 2024-02-19T21:13:42.696 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.696 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.696 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.696 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.563108+0000", 2024-02-19T21:13:42.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000100029 2024-02-19T21:13:42.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.563208+0000", 2024-02-19T21:13:42.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00041083999999999999 2024-02-19T21:13:42.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:13:42.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.563619+0000", 2024-02-19T21:13:42.697 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0070195659999999997 2024-02-19T21:13:42.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.697 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.697 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:13:42.697 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.570639+0000", 2024-02-19T21:13:42.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.01090327 2024-02-19T21:13:42.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:13:42.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.581542+0000", 2024-02-19T21:13:42.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5691999999999994e-05 2024-02-19T21:13:42.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.698 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.698 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.698 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:48.581608+0000", 2024-02-19T21:13:42.698 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.593213569 2024-02-19T21:13:42.698 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.698 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.698 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.699 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14952.0:2 53.6 53:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e111)", 2024-02-19T21:13:42.699 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:50.605965+0000", 2024-02-19T21:13:42.699 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 532.06418895299998, 2024-02-19T21:13:42.699 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036561746999999999, 2024-02-19T21:13:42.699 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.699 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:13:42.699 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:13:42.699 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14952", 2024-02-19T21:13:42.699 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2459019407", 2024-02-19T21:13:42.699 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:13:42.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.699 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.699 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605965+0000", 2024-02-19T21:13:42.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605965+0000", 2024-02-19T21:13:42.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:13:42.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.700 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.700 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.700 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605963+0000", 2024-02-19T21:13:42.700 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1330000000000004e-06 2024-02-19T21:13:42.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605969+0000", 2024-02-19T21:13:42.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5689999999999999e-06 2024-02-19T21:13:42.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605977+0000", 2024-02-19T21:13:42.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4559999999999999e-06 2024-02-19T21:13:42.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.701 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.701 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.701 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.605982+0000", 2024-02-19T21:13:42.701 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003383039 2024-02-19T21:13:42.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.609365+0000", 2024-02-19T21:13:42.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3409999999999997e-06 2024-02-19T21:13:42.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:13:42.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.609373+0000", 2024-02-19T21:13:42.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026694262 2024-02-19T21:13:42.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.702 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.702 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.702 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.636067+0000", 2024-02-19T21:13:42.702 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000103708 2024-02-19T21:13:42.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.636171+0000", 2024-02-19T21:13:42.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118288 2024-02-19T21:13:42.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 2,4", 2024-02-19T21:13:42.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.636289+0000", 2024-02-19T21:13:42.703 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0017480130000000001 2024-02-19T21:13:42.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.703 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.703 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:13:42.703 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.638037+0000", 2024-02-19T21:13:42.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00049063000000000004 2024-02-19T21:13:42.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:13:42.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.638528+0000", 2024-02-19T21:13:42.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039461189999999997 2024-02-19T21:13:42.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:13:42.704 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.642474+0000", 2024-02-19T21:13:42.704 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1164000000000002e-05 2024-02-19T21:13:42.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.704 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.704 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:13:42.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.642505+0000", 2024-02-19T21:13:42.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1668999999999999e-05 2024-02-19T21:13:42.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.705 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.705 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:50.642527+0000", 2024-02-19T21:13:42.705 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036561746999999999 2024-02-19T21:13:42.705 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.705 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.705 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.705 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.705 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.24865.0:21 11.8 11:1381d4a2:test-rados-api-smithi163-83952-6::foo:head [create,delete] snapc 0=[] ondisk+write+known_if_redirected e112)", 2024-02-19T21:13:42.705 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:04:51.573042+0000", 2024-02-19T21:13:42.706 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 531.09711188699998, 2024-02-19T21:13:42.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.070965300999999995, 2024-02-19T21:13:42.706 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.706 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:13:42.706 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:13:42.706 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.24865", 2024-02-19T21:13:42.706 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3187889786", 2024-02-19T21:13:42.706 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 21 2024-02-19T21:13:42.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.706 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.706 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.706 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.706 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573042+0000", 2024-02-19T21:13:42.706 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573042+0000", 2024-02-19T21:13:42.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:13:42.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573040+0000", 2024-02-19T21:13:42.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2414e-05 2024-02-19T21:13:42.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.707 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.707 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.707 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573052+0000", 2024-02-19T21:13:42.707 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00026839099999999999 2024-02-19T21:13:42.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573321+0000", 2024-02-19T21:13:42.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0720000000000001e-05 2024-02-19T21:13:42.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.573331+0000", 2024-02-19T21:13:42.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012845001 2024-02-19T21:13:42.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.708 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.708 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.708 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.586176+0000", 2024-02-19T21:13:42.708 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5872999999999995e-05 2024-02-19T21:13:42.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.586272+0000", 2024-02-19T21:13:42.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00018172699999999999 2024-02-19T21:13:42.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 5,6", 2024-02-19T21:13:42.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.586454+0000", 2024-02-19T21:13:42.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0034122739999999999 2024-02-19T21:13:42.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.709 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.709 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:13:42.709 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.589866+0000", 2024-02-19T21:13:42.709 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047893630999999999 2024-02-19T21:13:42.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:13:42.710 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.637760+0000", 2024-02-19T21:13:42.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061464759999999997 2024-02-19T21:13:42.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:13:42.710 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.643906+0000", 2024-02-19T21:13:42.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4803999999999999e-05 2024-02-19T21:13:42.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.710 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.710 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:13:42.710 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.643961+0000", 2024-02-19T21:13:42.710 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6133e-05 2024-02-19T21:13:42.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.711 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.711 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:04:51.644007+0000", 2024-02-19T21:13:42.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.070965300999999995 2024-02-19T21:13:42.711 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.711 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.711 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.711 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.711 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.711 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.35590.0:2 78.6 e128/127 78:602f83fe:::foo:head v 128'1, mlcod=0'0)", 2024-02-19T21:13:42.711 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:08.266769+0000", 2024-02-19T21:13:42.711 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 514.40338521499996, 2024-02-19T21:13:42.711 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057433619999999998, 2024-02-19T21:13:42.711 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.712 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:13:42.712 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266769+0000", 2024-02-19T21:13:42.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.712 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266769+0000", 2024-02-19T21:13:42.712 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:13:42.712 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.712 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.712 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266768+0000", 2024-02-19T21:13:42.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2753e-05 2024-02-19T21:13:42.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266781+0000", 2024-02-19T21:13:42.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6710000000000004e-06 2024-02-19T21:13:42.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.713 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266787+0000", 2024-02-19T21:13:42.713 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8820000000000001e-06 2024-02-19T21:13:42.713 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.713 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.713 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266795+0000", 2024-02-19T21:13:42.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001516304 2024-02-19T21:13:42.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.268311+0000", 2024-02-19T21:13:42.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7995e-05 2024-02-19T21:13:42.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.714 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.714 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.268329+0000", 2024-02-19T21:13:42.714 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055833582999999999 2024-02-19T21:13:42.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.714 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.715 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:13:42.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.324162+0000", 2024-02-19T21:13:42.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9811000000000002e-05 2024-02-19T21:13:42.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.715 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.715 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.324202+0000", 2024-02-19T21:13:42.715 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057433619999999998 2024-02-19T21:13:42.715 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.715 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.715 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.715 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.716 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.26962.0:28 64.1e e128/116 64:7e799bab:test-rados-api-smithi163-83952-10::foo:head v 128'1, mlcod=0'0)", 2024-02-19T21:13:42.716 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:08.266799+0000", 2024-02-19T21:13:42.716 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 514.40335468800004, 2024-02-19T21:13:42.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057427275999999999, 2024-02-19T21:13:42.716 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.716 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:13:42.716 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.716 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266799+0000", 2024-02-19T21:13:42.716 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.716 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.716 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.716 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266799+0000", 2024-02-19T21:13:42.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:13:42.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266799+0000", 2024-02-19T21:13:42.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4129999999999999e-06 2024-02-19T21:13:42.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.717 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.717 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266803+0000", 2024-02-19T21:13:42.717 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6290000000000001e-06 2024-02-19T21:13:42.717 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.717 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266808+0000", 2024-02-19T21:13:42.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4929999999999998e-06 2024-02-19T21:13:42.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.266811+0000", 2024-02-19T21:13:42.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001716396 2024-02-19T21:13:42.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.718 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.718 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.718 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.268527+0000", 2024-02-19T21:13:42.718 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.647e-05 2024-02-19T21:13:42.718 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.268544+0000", 2024-02-19T21:13:42.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.055662087999999998 2024-02-19T21:13:42.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:13:42.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.324206+0000", 2024-02-19T21:13:42.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0389000000000001e-05 2024-02-19T21:13:42.719 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.719 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.719 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.719 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:08.324226+0000", 2024-02-19T21:13:42.719 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057427275999999999 2024-02-19T21:13:42.720 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.720 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.720 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.720 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.720 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:110 75.1e 75:7b0234b0:test-rados-api-smithi163-83928-10::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e129)", 2024-02-19T21:13:42.720 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:09.003796+0000", 2024-02-19T21:13:42.720 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 513.66635827000005, 2024-02-19T21:13:42.720 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.093054726000000004, 2024-02-19T21:13:42.720 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.720 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:13:42.720 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:13:42.720 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:13:42.721 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:13:42.721 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 110 2024-02-19T21:13:42.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.721 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003796+0000", 2024-02-19T21:13:42.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.721 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.721 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.721 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003796+0000", 2024-02-19T21:13:42.721 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:13:42.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003793+0000", 2024-02-19T21:13:42.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6710000000000005e-06 2024-02-19T21:13:42.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003802+0000", 2024-02-19T21:13:42.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2583e-05 2024-02-19T21:13:42.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.722 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.722 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.722 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003814+0000", 2024-02-19T21:13:42.722 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8140000000000007e-06 2024-02-19T21:13:42.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003823+0000", 2024-02-19T21:13:42.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.905e-05 2024-02-19T21:13:42.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003872+0000", 2024-02-19T21:13:42.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1829999999999997e-06 2024-02-19T21:13:42.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.723 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.723 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:13:42.723 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.003880+0000", 2024-02-19T21:13:42.723 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010142976999999999 2024-02-19T21:13:42.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.014023+0000", 2024-02-19T21:13:42.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000209595 2024-02-19T21:13:42.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", 2024-02-19T21:13:42.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.014232+0000", 2024-02-19T21:13:42.724 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.017728766999999999 2024-02-19T21:13:42.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.724 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.724 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.724 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.031961+0000", 2024-02-19T21:13:42.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0221000000000003e-05 2024-02-19T21:13:42.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-02-19T21:13:42.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032041+0000", 2024-02-19T21:13:42.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.059847310000000001 2024-02-19T21:13:42.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.091888+0000", 2024-02-19T21:13:42.725 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118545 2024-02-19T21:13:42.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.725 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.725 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.725 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.092007+0000", 2024-02-19T21:13:42.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0048433310000000002 2024-02-19T21:13:42.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.726 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.726 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.096850+0000", 2024-02-19T21:13:42.726 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.093054726000000004 2024-02-19T21:13:42.726 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.726 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.726 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.726 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.726 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.726 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop_reply(osd.3.0:301 75.1e e129/125 ondisk, result = 0)", 2024-02-19T21:13:42.726 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:09.032674+0000", 2024-02-19T21:13:42.726 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 513.637479429, 2024-02-19T21:13:42.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036648739999999999, 2024-02-19T21:13:42.727 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.727 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:13:42.727 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032674+0000", 2024-02-19T21:13:42.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.727 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.727 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.727 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032674+0000", 2024-02-19T21:13:42.727 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:13:42.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032673+0000", 2024-02-19T21:13:42.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2400000000000001e-06 2024-02-19T21:13:42.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032680+0000", 2024-02-19T21:13:42.728 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5481e-05 2024-02-19T21:13:42.728 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.728 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.728 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.728 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032696+0000", 2024-02-19T21:13:42.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5610000000000001e-06 2024-02-19T21:13:42.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.032703+0000", 2024-02-19T21:13:42.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036602245999999998 2024-02-19T21:13:42.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.069305+0000", 2024-02-19T21:13:42.729 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4330000000000004e-06 2024-02-19T21:13:42.729 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.729 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.729 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.729 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.069314+0000", 2024-02-19T21:13:42.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3780000000000007e-06 2024-02-19T21:13:42.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.730 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.730 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.069323+0000", 2024-02-19T21:13:42.730 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.036648739999999999 2024-02-19T21:13:42.730 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.730 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.730 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.730 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.730 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.730 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:111 75.5 75:a23b9a9e:test-rados-api-smithi163-83928-10::bar:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e129)", 2024-02-19T21:13:42.730 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:09.097539+0000", 2024-02-19T21:13:42.730 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 513.57261448600002, 2024-02-19T21:13:42.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.13010182100000001, 2024-02-19T21:13:42.731 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.731 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:13:42.731 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:13:42.731 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:13:42.731 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:13:42.731 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 111 2024-02-19T21:13:42.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.731 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.731 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.731 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097539+0000", 2024-02-19T21:13:42.731 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.731 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.731 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097539+0000", 2024-02-19T21:13:42.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999962 2024-02-19T21:13:42.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097536+0000", 2024-02-19T21:13:42.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4768e-05 2024-02-19T21:13:42.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.732 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.732 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.732 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097550+0000", 2024-02-19T21:13:42.732 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8895000000000001e-05 2024-02-19T21:13:42.732 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097569+0000", 2024-02-19T21:13:42.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6809999999999993e-06 2024-02-19T21:13:42.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.097579+0000", 2024-02-19T21:13:42.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001749121 2024-02-19T21:13:42.733 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.733 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.733 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.733 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.099328+0000", 2024-02-19T21:13:42.733 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00027112000000000002 2024-02-19T21:13:42.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", 2024-02-19T21:13:42.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.099599+0000", 2024-02-19T21:13:42.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002897892 2024-02-19T21:13:42.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.102497+0000", 2024-02-19T21:13:42.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6007000000000001e-05 2024-02-19T21:13:42.734 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.734 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.734 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-02-19T21:13:42.734 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.102553+0000", 2024-02-19T21:13:42.734 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.12498295099999999 2024-02-19T21:13:42.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.227536+0000", 2024-02-19T21:13:42.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000105049 2024-02-19T21:13:42.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.735 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.735 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:09.227641+0000", 2024-02-19T21:13:42.735 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.13010182100000001 2024-02-19T21:13:42.735 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.735 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.735 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.735 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.735 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.736 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(76.3s1 133/126 ECSubWrite(tid=279, reqid=osd.4.0:279, at_version=133'6, trim_to=0'0, roll_forward_to=132'4))", 2024-02-19T21:13:42.736 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:13.106749+0000", 2024-02-19T21:13:42.736 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 509.56340528599998, 2024-02-19T21:13:42.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039746143999999997, 2024-02-19T21:13:42.736 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.736 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:13:42.736 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.736 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.736 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.106749+0000", 2024-02-19T21:13:42.736 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.736 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.736 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.106749+0000", 2024-02-19T21:13:42.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:13:42.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.106747+0000", 2024-02-19T21:13:42.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3730999999999999e-05 2024-02-19T21:13:42.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.737 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.737 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.106781+0000", 2024-02-19T21:13:42.737 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015052899999999999 2024-02-19T21:13:42.737 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.737 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.121834+0000", 2024-02-19T21:13:42.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7899999999999994e-06 2024-02-19T21:13:42.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.121844+0000", 2024-02-19T21:13:42.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001291719 2024-02-19T21:13:42.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.738 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.738 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.123135+0000", 2024-02-19T21:13:42.738 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0396e-05 2024-02-19T21:13:42.738 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.738 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:13:42.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.123146+0000", 2024-02-19T21:13:42.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023338983000000001 2024-02-19T21:13:42.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:13:42.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.146485+0000", 2024-02-19T21:13:42.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0078000000000001e-05 2024-02-19T21:13:42.739 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.739 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.739 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.739 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.146495+0000", 2024-02-19T21:13:42.739 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039746143999999997 2024-02-19T21:13:42.739 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.740 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.740 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.740 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.740 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.39709.0:2 84.3 84:c656db43:::oid_RoundTripPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e133)", 2024-02-19T21:13:42.740 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:13.135107+0000", 2024-02-19T21:13:42.740 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 509.53504674099997, 2024-02-19T21:13:42.740 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037126581999999998, 2024-02-19T21:13:42.740 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.740 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:13:42.740 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:13:42.740 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.39709", 2024-02-19T21:13:42.740 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3505054706", 2024-02-19T21:13:42.740 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:13:42.740 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.741 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135107+0000", 2024-02-19T21:13:42.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135107+0000", 2024-02-19T21:13:42.741 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:13:42.741 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.741 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.741 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.741 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135104+0000", 2024-02-19T21:13:42.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8099999999999992e-06 2024-02-19T21:13:42.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135114+0000", 2024-02-19T21:13:42.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1422e-05 2024-02-19T21:13:42.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135125+0000", 2024-02-19T21:13:42.742 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5689999999999999e-06 2024-02-19T21:13:42.742 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.742 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.742 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.742 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135133+0000", 2024-02-19T21:13:42.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8407e-05 2024-02-19T21:13:42.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135171+0000", 2024-02-19T21:13:42.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2189999999999998e-06 2024-02-19T21:13:42.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:13:42.743 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.135179+0000", 2024-02-19T21:13:42.743 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.034870916000000002 2024-02-19T21:13:42.743 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.743 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.743 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.170050+0000", 2024-02-19T21:13:42.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011553900000000001 2024-02-19T21:13:42.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.170165+0000", 2024-02-19T21:13:42.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019117400000000001 2024-02-19T21:13:42.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 1,7", 2024-02-19T21:13:42.744 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.170356+0000", 2024-02-19T21:13:42.744 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0015803569999999999 2024-02-19T21:13:42.744 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.744 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.744 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:13:42.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.171937+0000", 2024-02-19T21:13:42.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000119746 2024-02-19T21:13:42.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:13:42.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.172056+0000", 2024-02-19T21:13:42.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000104515 2024-02-19T21:13:42.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.745 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:13:42.745 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.172161+0000", 2024-02-19T21:13:42.745 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2716000000000003e-05 2024-02-19T21:13:42.745 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.745 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:13:42.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.172194+0000", 2024-02-19T21:13:42.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0095000000000002e-05 2024-02-19T21:13:42.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.746 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.746 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:13.172234+0000", 2024-02-19T21:13:42.746 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.037126581999999998 2024-02-19T21:13:42.746 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.746 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.746 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.746 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.746 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.746 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.0.0:267 64.1e e134/116 64:7e799bab:test-rados-api-smithi163-83952-10::foo:b v 134'20, mlcod=131'16)", 2024-02-19T21:13:42.747 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:14.266716+0000", 2024-02-19T21:13:42.747 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 508.40343797700001, 2024-02-19T21:13:42.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.083584948000000006, 2024-02-19T21:13:42.747 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.747 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:13:42.747 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266716+0000", 2024-02-19T21:13:42.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.747 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.747 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.747 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.747 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266716+0000", 2024-02-19T21:13:42.747 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:13:42.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.748 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266715+0000", 2024-02-19T21:13:42.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3909000000000001e-05 2024-02-19T21:13:42.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.748 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266739+0000", 2024-02-19T21:13:42.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.178e-06 2024-02-19T21:13:42.748 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.748 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.748 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.748 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266745+0000", 2024-02-19T21:13:42.748 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2620000000000004e-06 2024-02-19T21:13:42.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.749 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.749 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.266752+0000", 2024-02-19T21:13:42.749 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015674777000000001 2024-02-19T21:13:42.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.749 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.749 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.282426+0000", 2024-02-19T21:13:42.749 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4134999999999998e-05 2024-02-19T21:13:42.749 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.749 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.749 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.749 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.282450+0000", 2024-02-19T21:13:42.750 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.067824522999999998 2024-02-19T21:13:42.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.750 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.750 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:13:42.750 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.350275+0000", 2024-02-19T21:13:42.750 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5811000000000001e-05 2024-02-19T21:13:42.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.750 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.750 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.750 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:14.350301+0000", 2024-02-19T21:13:42.750 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.083584948000000006 2024-02-19T21:13:42.750 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.750 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.750 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.750 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.751 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.751 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:257 85.2 85:43403dbe:test-rados-api-smithi163-83928-12::bam:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e138)", 2024-02-19T21:13:42.751 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:18.587348+0000", 2024-02-19T21:13:42.751 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 504.08280563400001, 2024-02-19T21:13:42.751 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.069587300000000005, 2024-02-19T21:13:42.751 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.751 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:13:42.751 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:13:42.751 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:13:42.751 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:13:42.751 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 257 2024-02-19T21:13:42.751 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.751 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.752 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.752 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.752 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587348+0000", 2024-02-19T21:13:42.752 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.752 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.752 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.752 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.752 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587348+0000", 2024-02-19T21:13:42.752 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999957 2024-02-19T21:13:42.752 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.752 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.752 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.752 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587344+0000", 2024-02-19T21:13:42.752 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6024000000000001e-05 2024-02-19T21:13:42.753 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.753 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.753 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587360+0000", 2024-02-19T21:13:42.753 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5661999999999999e-05 2024-02-19T21:13:42.753 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.753 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.753 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587386+0000", 2024-02-19T21:13:42.753 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0662999999999999e-05 2024-02-19T21:13:42.753 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.753 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.753 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.753 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587396+0000", 2024-02-19T21:13:42.753 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4304e-05 2024-02-19T21:13:42.754 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.754 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.754 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.754 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587450+0000", 2024-02-19T21:13:42.754 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023577900000000001 2024-02-19T21:13:42.754 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.754 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.754 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", 2024-02-19T21:13:42.754 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.587686+0000", 2024-02-19T21:13:42.754 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0062748650000000001 2024-02-19T21:13:42.754 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.754 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.754 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.755 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.593961+0000", 2024-02-19T21:13:42.755 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8112000000000001e-05 2024-02-19T21:13:42.755 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.755 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.755 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for rw locks", 2024-02-19T21:13:42.755 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.594029+0000", 2024-02-19T21:13:42.755 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.059753043999999998 2024-02-19T21:13:42.755 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.755 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.755 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.755 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.653782+0000", 2024-02-19T21:13:42.755 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7813999999999996e-05 2024-02-19T21:13:42.755 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.755 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.755 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.756 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.653880+0000", 2024-02-19T21:13:42.756 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003055412 2024-02-19T21:13:42.756 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.756 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.756 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.756 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:18.656935+0000", 2024-02-19T21:13:42.756 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.069587300000000005 2024-02-19T21:13:42.756 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.756 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.756 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.756 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.756 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.756 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.26962.0:57 64.3 64:c6c3d28d:test-rados-api-smithi163-83952-12::foo:head [write 0~128] snapc 14=[14] ondisk+write+ordersnap+known_if_redirected e139)", 2024-02-19T21:13:42.757 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:19.170619+0000", 2024-02-19T21:13:42.757 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 503.49953434399998, 2024-02-19T21:13:42.757 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.10751445900000001, 2024-02-19T21:13:42.757 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.757 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:13:42.757 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:13:42.757 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.26962", 2024-02-19T21:13:42.757 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2924448120", 2024-02-19T21:13:42.757 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 57 2024-02-19T21:13:42.757 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.757 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.757 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.757 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.758 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170619+0000", 2024-02-19T21:13:42.758 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.758 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.758 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.758 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.758 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170619+0000", 2024-02-19T21:13:42.758 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:13:42.758 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.758 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.758 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.758 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170617+0000", 2024-02-19T21:13:42.758 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0165e-05 2024-02-19T21:13:42.758 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.759 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.759 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.759 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170627+0000", 2024-02-19T21:13:42.759 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2262999999999999e-05 2024-02-19T21:13:42.759 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.759 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.759 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.759 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170640+0000", 2024-02-19T21:13:42.759 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2870000000000004e-06 2024-02-19T21:13:42.759 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.759 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.759 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.759 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170648+0000", 2024-02-19T21:13:42.759 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0032999999999999e-05 2024-02-19T21:13:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170708+0000", 2024-02-19T21:13:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7159999999999997e-06 2024-02-19T21:13:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:13:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.170715+0000", 2024-02-19T21:13:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.047117838000000002 2024-02-19T21:13:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.217833+0000", 2024-02-19T21:13:42.761 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012449 2024-02-19T21:13:42.761 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.761 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.761 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.761 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.217957+0000", 2024-02-19T21:13:42.761 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015379599999999999 2024-02-19T21:13:42.761 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.761 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.761 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,6", 2024-02-19T21:13:42.761 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.218111+0000", 2024-02-19T21:13:42.761 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014134015999999999 2024-02-19T21:13:42.761 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.761 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.761 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:13:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.232245+0000", 2024-02-19T21:13:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039615919999999999 2024-02-19T21:13:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:13:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.271861+0000", 2024-02-19T21:13:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0061818860000000002 2024-02-19T21:13:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:13:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.278043+0000", 2024-02-19T21:13:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8167999999999997e-05 2024-02-19T21:13:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.762 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:13:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.278091+0000", 2024-02-19T21:13:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3086e-05 2024-02-19T21:13:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:19.278134+0000", 2024-02-19T21:13:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.10751445900000001 2024-02-19T21:13:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.49370.0:1 171.6s0 171:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e258)", 2024-02-19T21:13:42.763 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:07:18.236944+0000", 2024-02-19T21:13:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 384.43320962199999, 2024-02-19T21:13:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2320055759999999, 2024-02-19T21:13:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:13:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:13:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.49370", 2024-02-19T21:13:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2010916473", 2024-02-19T21:13:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:13:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.236944+0000", 2024-02-19T21:13:42.764 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.236944+0000", 2024-02-19T21:13:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:13:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.236942+0000", 2024-02-19T21:13:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8730000000000007e-06 2024-02-19T21:13:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.236950+0000", 2024-02-19T21:13:42.765 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7600000000000008e-06 2024-02-19T21:13:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.236959+0000", 2024-02-19T21:13:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1589999999999998e-06 2024-02-19T21:13:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.236965+0000", 2024-02-19T21:13:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00044211899999999999 2024-02-19T21:13:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.766 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.237407+0000", 2024-02-19T21:13:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.399e-06 2024-02-19T21:13:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:13:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:18.237410+0000", 2024-02-19T21:13:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2273940489999999 2024-02-19T21:13:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:19.464804+0000", 2024-02-19T21:13:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013275200000000001 2024-02-19T21:13:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.767 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:19.464937+0000", 2024-02-19T21:13:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000395337 2024-02-19T21:13:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:13:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:19.465333+0000", 2024-02-19T21:13:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0011654949999999999 2024-02-19T21:13:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:13:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:19.466498+0000", 2024-02-19T21:13:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0023873789999999998 2024-02-19T21:13:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.768 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:13:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:19.468885+0000", 2024-02-19T21:13:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4368000000000004e-05 2024-02-19T21:13:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:19.468950+0000", 2024-02-19T21:13:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2320055759999999 2024-02-19T21:13:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.769 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.49394.0:1 174.7s0 174:e732be06:::oid_WaitForSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e262)", 2024-02-19T21:13:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:07:22.216139+0000", 2024-02-19T21:13:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 380.45401518199998, 2024-02-19T21:13:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0674298339999999, 2024-02-19T21:13:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:13:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:13:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.49394", 2024-02-19T21:13:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/3062946080", 2024-02-19T21:13:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:13:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216139+0000", 2024-02-19T21:13:42.770 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216139+0000", 2024-02-19T21:13:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:13:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216136+0000", 2024-02-19T21:13:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7820000000000005e-06 2024-02-19T21:13:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216146+0000", 2024-02-19T21:13:42.771 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1165000000000001e-05 2024-02-19T21:13:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216157+0000", 2024-02-19T21:13:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9519999999999994e-06 2024-02-19T21:13:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216165+0000", 2024-02-19T21:13:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6961e-05 2024-02-19T21:13:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.772 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216212+0000", 2024-02-19T21:13:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4010000000000001e-06 2024-02-19T21:13:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:13:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:22.216216+0000", 2024-02-19T21:13:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.064811247 2024-02-19T21:13:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:23.281027+0000", 2024-02-19T21:13:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013814900000000001 2024-02-19T21:13:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.773 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:23.281165+0000", 2024-02-19T21:13:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00046475999999999999 2024-02-19T21:13:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:13:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:23.281630+0000", 2024-02-19T21:13:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00044463700000000001 2024-02-19T21:13:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:13:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:23.282074+0000", 2024-02-19T21:13:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001428971 2024-02-19T21:13:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:13:42.774 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:23.283503+0000", 2024-02-19T21:13:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5203000000000006e-05 2024-02-19T21:13:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:23.283568+0000", 2024-02-19T21:13:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0674298339999999 2024-02-19T21:13:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.25753.0:4348 191.8 e314/308 191:10fb1d02:::199:head v 314'8, mlcod=313'6)", 2024-02-19T21:13:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:08:14.258279+0000", 2024-02-19T21:13:42.775 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 328.41187502499997, 2024-02-19T21:13:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039096335000000003, 2024-02-19T21:13:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:13:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258279+0000", 2024-02-19T21:13:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258279+0000", 2024-02-19T21:13:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:13:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.776 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258277+0000", 2024-02-19T21:13:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1771e-05 2024-02-19T21:13:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258289+0000", 2024-02-19T21:13:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4579999999999993e-06 2024-02-19T21:13:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258297+0000", 2024-02-19T21:13:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.113e-06 2024-02-19T21:13:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.777 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258303+0000", 2024-02-19T21:13:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4646e-05 2024-02-19T21:13:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258337+0000", 2024-02-19T21:13:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1143999999999999e-05 2024-02-19T21:13:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258358+0000", 2024-02-19T21:13:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038976552999999997 2024-02-19T21:13:42.778 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:13:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.297335+0000", 2024-02-19T21:13:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0234000000000001e-05 2024-02-19T21:13:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.297375+0000", 2024-02-19T21:13:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039096335000000003 2024-02-19T21:13:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.779 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.34352.0:1422 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94681540387968 gen 4] snapc 0=[] ondisk+write+known_if_redirected e433)", 2024-02-19T21:13:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:10:40.311647+0000", 2024-02-19T21:13:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 182.358506452, 2024-02-19T21:13:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0091623380000001, 2024-02-19T21:13:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:13:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:13:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:13:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.34352", 2024-02-19T21:13:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:13:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1422 2024-02-19T21:13:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:13:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.780 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:13:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311647+0000", 2024-02-19T21:13:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:13:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:13:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311647+0000", 2024-02-19T21:13:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:13:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:13:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311645+0000", 2024-02-19T21:13:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4200000000000007e-06 2024-02-19T21:13:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.781 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311653+0000", 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1905e-05 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311665+0000", 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1360000000000002e-06 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311671+0000", 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8489999999999998e-05 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.782 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311720+0000", 2024-02-19T21:13:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9099999999999994e-06 2024-02-19T21:13:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:13:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311728+0000", 2024-02-19T21:13:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98640665500000002 2024-02-19T21:13:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:41.298135+0000", 2024-02-19T21:13:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030824800000000001 2024-02-19T21:13:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.783 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for degraded object", 2024-02-19T21:13:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:41.298443+0000", 2024-02-19T21:13:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016746660999999999 2024-02-19T21:13:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:13:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:41.315190+0000", 2024-02-19T21:13:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7898000000000001e-05 2024-02-19T21:13:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:13:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:41.315238+0000", 2024-02-19T21:13:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.005571767 2024-02-19T21:13:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:13:42.784 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:13:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:13:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:41.320810+0000", 2024-02-19T21:13:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0091623380000001 2024-02-19T21:13:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:13:42.785 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:13:42.785 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:13:42.785 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:42.813 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:42.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:40.806781+0000 osd.7 (osd.7) 261 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:13:42.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:40.809794+0000 osd.7 (osd.7) 262 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:13:42.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:41.274095+0000 mgr.y (mgr.14550) 512 : cluster [DBG] pgmap v876: 222 pgs: 222 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:42.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:41.600849+0000 osd.2 (osd.2) 232 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:13:42.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:41.604177+0000 osd.2 (osd.2) 233 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:13:42.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:41.705575+0000 osd.4 (osd.4) 207 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:13:42.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:41.706492+0000 osd.4 (osd.4) 208 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:13:42.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:42.346798+0000 mon.a (mon.0) 3039 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-99-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-02-19T21:13:42.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:42.346854+0000 mon.a (mon.0) 3040 : cluster [DBG] osdmap e586: 8 total, 8 up, 6 in 2024-02-19T21:13:42.866 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:42.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:42.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:42.971 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:43.003 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:43.025 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:43.028 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:43.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:43.129 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:43.159 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:43.179 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:43.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:43.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:43.283 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:43.312 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:43.333 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:43.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:43.437 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:43.468 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:43.512 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:13:43.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:42.081333+0000 osd.1 (osd.1) 266 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:13:43.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:42.086001+0000 osd.1 (osd.1) 267 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:13:43.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:42.375438+0000 mon.a (mon.0) 3041 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:43.747 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:13:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:13:43] "GET /metrics HTTP/1.1" 200 193528 "" "Prometheus/2.33.4" 2024-02-19T21:13:43.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:42.081333+0000 osd.1 (osd.1) 266 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:13:43.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:42.086001+0000 osd.1 (osd.1) 267 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:13:43.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:42.375438+0000 mon.a (mon.0) 3041 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:43.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:42.081333+0000 osd.1 (osd.1) 266 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:13:43.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:42.086001+0000 osd.1 (osd.1) 267 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:13:43.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:42.375438+0000 mon.a (mon.0) 3041 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:44.418 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:44.421 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:44.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:44.440 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-19T21:13:44.523 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:44.550 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:44.714 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:13:44.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:42.623123+0000 osd.7 (osd.7) 263 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:44.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:42.623288+0000 osd.7 (osd.7) 264 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:44.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:43.106558+0000 osd.1 (osd.1) 268 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:13:44.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:43.109575+0000 osd.1 (osd.1) 269 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:13:44.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:43.274509+0000 mgr.y (mgr.14550) 513 : cluster [DBG] pgmap v879: 222 pgs: 222 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:44.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:43.375312+0000 mon.a (mon.0) 3042 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-99"}]': finished 2024-02-19T21:13:44.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:43.375367+0000 mon.a (mon.0) 3043 : cluster [DBG] osdmap e587: 8 total, 8 up, 6 in 2024-02-19T21:13:44.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:43.379116+0000 mon.a (mon.0) 3044 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-99", "tierpool": "test-rados-api-smithi163-83928-99-cache"}]: dispatch 2024-02-19T21:13:44.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:43.550504+0000 osd.2 (osd.2) 234 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:13:44.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:43.552022+0000 osd.2 (osd.2) 235 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:13:44.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:43.593433+0000 osd.6 (osd.6) 211 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:44.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:43.593601+0000 osd.6 (osd.6) 212 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:44.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:43.734579+0000 osd.4 (osd.4) 209 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:13:44.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:43.735378+0000 osd.4 (osd.4) 210 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:13:44.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:44.341065+0000 mon.a (mon.0) 3045 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-99", "tierpool": "test-rados-api-smithi163-83928-99-cache"}]': finished 2024-02-19T21:13:44.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:44.341126+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e588: 8 total, 8 up, 6 in 2024-02-19T21:13:44.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:42.623123+0000 osd.7 (osd.7) 263 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:44.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:42.623288+0000 osd.7 (osd.7) 264 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:44.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:43.106558+0000 osd.1 (osd.1) 268 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:13:44.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:43.109575+0000 osd.1 (osd.1) 269 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:13:44.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:43.274509+0000 mgr.y (mgr.14550) 513 : cluster [DBG] pgmap v879: 222 pgs: 222 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:44.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:43.375312+0000 mon.a (mon.0) 3042 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-99"}]': finished 2024-02-19T21:13:44.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:43.375367+0000 mon.a (mon.0) 3043 : cluster [DBG] osdmap e587: 8 total, 8 up, 6 in 2024-02-19T21:13:44.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:43.379116+0000 mon.a (mon.0) 3044 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-99", "tierpool": "test-rados-api-smithi163-83928-99-cache"}]: dispatch 2024-02-19T21:13:44.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:43.550504+0000 osd.2 (osd.2) 234 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:13:44.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:43.552022+0000 osd.2 (osd.2) 235 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:13:44.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:43.593433+0000 osd.6 (osd.6) 211 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:44.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:43.593601+0000 osd.6 (osd.6) 212 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:44.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:43.734579+0000 osd.4 (osd.4) 209 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:13:44.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:43.735378+0000 osd.4 (osd.4) 210 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:13:44.753 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:44.341065+0000 mon.a (mon.0) 3045 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-99", "tierpool": "test-rados-api-smithi163-83928-99-cache"}]': finished 2024-02-19T21:13:44.753 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:44.341126+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e588: 8 total, 8 up, 6 in 2024-02-19T21:13:44.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:42.623123+0000 osd.7 (osd.7) 263 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:42.623288+0000 osd.7 (osd.7) 264 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:43.106558+0000 osd.1 (osd.1) 268 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:13:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:43.109575+0000 osd.1 (osd.1) 269 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:13:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:43.274509+0000 mgr.y (mgr.14550) 513 : cluster [DBG] pgmap v879: 222 pgs: 222 active+clean; 5.7 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:43.375312+0000 mon.a (mon.0) 3042 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-99"}]': finished 2024-02-19T21:13:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:43.375367+0000 mon.a (mon.0) 3043 : cluster [DBG] osdmap e587: 8 total, 8 up, 6 in 2024-02-19T21:13:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:43.379116+0000 mon.a (mon.0) 3044 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-99", "tierpool": "test-rados-api-smithi163-83928-99-cache"}]: dispatch 2024-02-19T21:13:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:43.550504+0000 osd.2 (osd.2) 234 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:13:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:43.552022+0000 osd.2 (osd.2) 235 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:13:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:43.593433+0000 osd.6 (osd.6) 211 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:13:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:43.593601+0000 osd.6 (osd.6) 212 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:13:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:43.734579+0000 osd.4 (osd.4) 209 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:13:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:43.735378+0000 osd.4 (osd.4) 210 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:13:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:44.341065+0000 mon.a (mon.0) 3045 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-99", "tierpool": "test-rados-api-smithi163-83928-99-cache"}]': finished 2024-02-19T21:13:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:44.341126+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e588: 8 total, 8 up, 6 in 2024-02-19T21:13:45.500 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:45.503 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:45.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:45.604 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:45.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:43.776606+0000 osd.5 (osd.5) 232 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:13:45.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:43.779533+0000 osd.5 (osd.5) 233 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:13:45.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:44.565849+0000 osd.2 (osd.2) 236 : cluster [DBG] 16.1d scrub starts 2024-02-19T21:13:45.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:44.567191+0000 osd.2 (osd.2) 237 : cluster [DBG] 16.1d scrub ok 2024-02-19T21:13:45.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:45.340136+0000 mon.a (mon.0) 3047 : cluster [DBG] osdmap e589: 8 total, 8 up, 6 in 2024-02-19T21:13:45.633 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:45.731 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:45.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:45.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:43.776606+0000 osd.5 (osd.5) 232 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:13:45.741 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:43.779533+0000 osd.5 (osd.5) 233 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:13:45.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:44.565849+0000 osd.2 (osd.2) 236 : cluster [DBG] 16.1d scrub starts 2024-02-19T21:13:45.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:44.567191+0000 osd.2 (osd.2) 237 : cluster [DBG] 16.1d scrub ok 2024-02-19T21:13:45.742 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:45.340136+0000 mon.a (mon.0) 3047 : cluster [DBG] osdmap e589: 8 total, 8 up, 6 in 2024-02-19T21:13:45.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:43.776606+0000 osd.5 (osd.5) 232 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:13:45.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:43.779533+0000 osd.5 (osd.5) 233 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:13:45.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:44.565849+0000 osd.2 (osd.2) 236 : cluster [DBG] 16.1d scrub starts 2024-02-19T21:13:45.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:44.567191+0000 osd.2 (osd.2) 237 : cluster [DBG] 16.1d scrub ok 2024-02-19T21:13:45.742 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:45.340136+0000 mon.a (mon.0) 3047 : cluster [DBG] osdmap e589: 8 total, 8 up, 6 in 2024-02-19T21:13:45.837 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:45.864 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:46.113 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:46.115 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:46.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:46.217 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:46.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.219+0000 7f0c03a2a700 1 -- 172.21.15.163:0/838209926 >> v1:172.21.15.163:6790/0 conn(0x7f0bfc100780 legacy=0x7f0bfc102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:46.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.219+0000 7f0c03a2a700 1 -- 172.21.15.163:0/838209926 shutdown_connections 2024-02-19T21:13:46.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.219+0000 7f0c03a2a700 1 -- 172.21.15.163:0/838209926 >> 172.21.15.163:0/838209926 conn(0x7f0bfc0f35a0 msgr2=0x7f0bfc0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:46.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.219+0000 7f0c03a2a700 1 -- 172.21.15.163:0/838209926 shutdown_connections 2024-02-19T21:13:46.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.219+0000 7f0c03a2a700 1 -- 172.21.15.163:0/838209926 wait complete. 2024-02-19T21:13:46.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.219+0000 7f0c03a2a700 1 librados: starting msgr at 2024-02-19T21:13:46.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.219+0000 7f0c03a2a700 1 librados: starting objecter 2024-02-19T21:13:46.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.219+0000 7f0c03a2a700 1 Processor -- start 2024-02-19T21:13:46.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.219+0000 7f0c03a2a700 1 -- start start 2024-02-19T21:13:46.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.219+0000 7f0c03a2a700 1 librados: setting wanted keys 2024-02-19T21:13:46.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.219+0000 7f0c03a2a700 1 librados: calling monclient init 2024-02-19T21:13:46.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0c03a2a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0bfc0fcfa0 con 0x7f0bfc100780 2024-02-19T21:13:46.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0c03a2a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0bfc0fd130 con 0x7f0bfc0fd980 2024-02-19T21:13:46.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0c03a2a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0bfc0fd320 con 0x7f0bfc1042c0 2024-02-19T21:13:46.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0c01fc7700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f0bfc1042c0 0x7f0bfc118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:55762/0 (socket says 172.21.15.163:55762) 2024-02-19T21:13:46.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0c01fc7700 1 -- 172.21.15.163:0/3583448038 learned_addr learned my addr 172.21.15.163:0/3583448038 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:46.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 10 client.?.objecter ms_handle_connect 0x7f0bfc1042c0 2024-02-19T21:13:46.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:46.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 10 client.?.objecter ms_handle_connect 0x7f0bfc0fd980 2024-02-19T21:13:46.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:46.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048075990 0 0) 0x7f0bfc0fd320 con 0x7f0bfc1042c0 2024-02-19T21:13:46.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be8003800 con 0x7f0bfc1042c0 2024-02-19T21:13:46.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767913655 0 0) 0x7f0bfc0fd130 con 0x7f0bfc0fd980 2024-02-19T21:13:46.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bfc0fd320 con 0x7f0bfc0fd980 2024-02-19T21:13:46.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 10 client.?.objecter ms_handle_connect 0x7f0bfc100780 2024-02-19T21:13:46.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:46.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705857527 0 0) 0x7f0bfc0fcfa0 con 0x7f0bfc100780 2024-02-19T21:13:46.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bfc0fd130 con 0x7f0bfc100780 2024-02-19T21:13:46.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4108145637 0 0) 0x7f0be8003800 con 0x7f0bfc1042c0 2024-02-19T21:13:46.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bfc0fcfa0 con 0x7f0bfc1042c0 2024-02-19T21:13:46.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0bf4001d70 con 0x7f0bfc1042c0 2024-02-19T21:13:46.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1478828462 0 0) 0x7f0bfc0fd320 con 0x7f0bfc0fd980 2024-02-19T21:13:46.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0be8003800 con 0x7f0bfc0fd980 2024-02-19T21:13:46.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.220+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0bf8001840 con 0x7f0bfc0fd980 2024-02-19T21:13:46.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.221+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1587765291 0 0) 0x7f0bfc0fd130 con 0x7f0bfc100780 2024-02-19T21:13:46.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.221+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bfc0fd320 con 0x7f0bfc100780 2024-02-19T21:13:46.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.221+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0bec002b90 con 0x7f0bfc100780 2024-02-19T21:13:46.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.221+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 731669331 0 0) 0x7f0bfc0fcfa0 con 0x7f0bfc1042c0 2024-02-19T21:13:46.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.221+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 >> v1:172.21.15.193:6789/0 conn(0x7f0bfc0fd980 legacy=0x7f0bfc0fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:46.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.221+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 >> v1:172.21.15.163:6789/0 conn(0x7f0bfc100780 legacy=0x7f0bfc114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:46.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.221+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bfc11cb90 con 0x7f0bfc1042c0 2024-02-19T21:13:46.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.221+0000 7f0c03a2a700 1 -- 172.21.15.163:0/3583448038 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bfc119bc0 con 0x7f0bfc1042c0 2024-02-19T21:13:46.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.221+0000 7f0c03a2a700 10 client.50228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:46.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.221+0000 7f0c03a2a700 1 -- 172.21.15.163:0/3583448038 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bfc11a030 con 0x7f0bfc1042c0 2024-02-19T21:13:46.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.221+0000 7f0c03a2a700 1 librados: init done 2024-02-19T21:13:46.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.221+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f0bf4002740 con 0x7f0bfc1042c0 2024-02-19T21:13:46.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.221+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0bf4005e70 con 0x7f0bfc1042c0 2024-02-19T21:13:46.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.222+0000 7f0c03a2a700 1 -- 172.21.15.163:0/3583448038 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be0004f20 con 0x7f0bfc1042c0 2024-02-19T21:13:46.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.222+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f0bf40028b0 con 0x7f0bfc1042c0 2024-02-19T21:13:46.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.222+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(589..589 src has 434..589) v4 ==== 8947+0+0 (unknown 3529766104 0 0) 0x7f0bf4079390 con 0x7f0bfc1042c0 2024-02-19T21:13:46.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.222+0000 7f0bf27fc700 10 client.50228.objecter ms_dispatch 0x7f0bfc001550 osd_map(589..589 src has 434..589) v4 2024-02-19T21:13:46.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.222+0000 7f0bf27fc700 3 client.50228.objecter handle_osd_map got epochs [589,589] > 0 2024-02-19T21:13:46.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.222+0000 7f0bf27fc700 3 client.50228.objecter handle_osd_map decoding full epoch 589 2024-02-19T21:13:46.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.223+0000 7f0bf27fc700 20 client.50228.objecter dump_active .. 0 homeless 2024-02-19T21:13:46.226 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.223+0000 7f0bf27fc700 10 client.50228.objecter ms_handle_connect 0x7f0be8063d00 2024-02-19T21:13:46.227 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.227+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0bf404c490 con 0x7f0bfc1042c0 2024-02-19T21:13:46.243 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:46.368 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:46.370 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:46.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:46.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.410+0000 7f0c03a2a700 1 -- 172.21.15.163:0/3583448038 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0be0000b90 con 0x7f0be8063d00 2024-02-19T21:13:46.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.423+0000 7f0bf27fc700 1 -- 172.21.15.163:0/3583448038 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+465495 (unknown 2965378022 0 3455882719) 0x7f0be0000b90 con 0x7f0be8063d00 2024-02-19T21:13:46.429 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:13:46.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.426+0000 7f0c03a2a700 10 librados: watch_flush enter 2024-02-19T21:13:46.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.426+0000 7f0c03a2a700 10 librados: watch_flush exit 2024-02-19T21:13:46.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.426+0000 7f0c03a2a700 20 client.50228.objecter shutdown clearing up homeless session... 2024-02-19T21:13:46.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.426+0000 7f0c03a2a700 10 client.50228.objecter successfully canceled tick 2024-02-19T21:13:46.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.426+0000 7f0c03a2a700 1 -- 172.21.15.163:0/3583448038 >> v1:172.21.15.163:6800/1552514681 conn(0x7f0be8063d00 legacy=0x7f0be80661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:46.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.426+0000 7f0c03a2a700 1 -- 172.21.15.163:0/3583448038 >> v1:172.21.15.163:6790/0 conn(0x7f0bfc1042c0 legacy=0x7f0bfc118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:46.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.426+0000 7f0c03a2a700 1 -- 172.21.15.163:0/3583448038 shutdown_connections 2024-02-19T21:13:46.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.426+0000 7f0c03a2a700 1 -- 172.21.15.163:0/3583448038 >> 172.21.15.163:0/3583448038 conn(0x7f0bfc0f35a0 msgr2=0x7f0bfc107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:46.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.426+0000 7f0c03a2a700 1 -- 172.21.15.163:0/3583448038 shutdown_connections 2024-02-19T21:13:46.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.426+0000 7f0c03a2a700 1 -- 172.21.15.163:0/3583448038 wait complete. 2024-02-19T21:13:46.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:46.426+0000 7f0c03a2a700 1 librados: shutdown 2024-02-19T21:13:46.434 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:13:46.471 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:46.499 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:46.635 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:13:46.635 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:13:46.635 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-02-19T21:13:46.635 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:13:46.652 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:46.654 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:45.275506+0000 mgr.y (mgr.14550) 514 : cluster [DBG] pgmap v882: 222 pgs: 222 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-19T21:13:46.654 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:45.375529+0000 mon.a (mon.0) 3048 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:13:46.654 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:45.575157+0000 osd.2 (osd.2) 238 : cluster [DBG] 2.21 scrub starts 2024-02-19T21:13:46.654 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:45.579933+0000 osd.2 (osd.2) 239 : cluster [DBG] 2.21 scrub ok 2024-02-19T21:13:46.654 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:46.057713+0000 mon.a (mon.0) 3049 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:46.654 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:46.341911+0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e590: 8 total, 8 up, 6 in 2024-02-19T21:13:46.654 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:46.343019+0000 mon.a (mon.0) 3051 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:46.655 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:46.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:46.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:45.275506+0000 mgr.y (mgr.14550) 514 : cluster [DBG] pgmap v882: 222 pgs: 222 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-19T21:13:46.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:45.375529+0000 mon.a (mon.0) 3048 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:13:46.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:45.575157+0000 osd.2 (osd.2) 238 : cluster [DBG] 2.21 scrub starts 2024-02-19T21:13:46.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:45.579933+0000 osd.2 (osd.2) 239 : cluster [DBG] 2.21 scrub ok 2024-02-19T21:13:46.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:46.057713+0000 mon.a (mon.0) 3049 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:46.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:46.341911+0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e590: 8 total, 8 up, 6 in 2024-02-19T21:13:46.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:46.343019+0000 mon.a (mon.0) 3051 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:46.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:45.275506+0000 mgr.y (mgr.14550) 514 : cluster [DBG] pgmap v882: 222 pgs: 222 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-19T21:13:46.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:45.375529+0000 mon.a (mon.0) 3048 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:13:46.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:45.575157+0000 osd.2 (osd.2) 238 : cluster [DBG] 2.21 scrub starts 2024-02-19T21:13:46.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:45.579933+0000 osd.2 (osd.2) 239 : cluster [DBG] 2.21 scrub ok 2024-02-19T21:13:46.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:46.057713+0000 mon.a (mon.0) 3049 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:46.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:46.341911+0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e590: 8 total, 8 up, 6 in 2024-02-19T21:13:46.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:46.343019+0000 mon.a (mon.0) 3051 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:46.756 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:13:46.785 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:46.809 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:13:46.812 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:46.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:13:46.913 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:46.940 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:47.245 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-19T21:13:47.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:45.764005+0000 osd.7 (osd.7) 265 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:13:47.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:45.764775+0000 osd.7 (osd.7) 266 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:13:47.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:46.411660+0000 mgr.y (mgr.14550) 515 : audit [DBG] from='client.50228 v1:172.21.15.163:0/3583448038' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:13:47.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:46.460646+0000 mon.b (mon.1) 730 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:13:47.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:46.461383+0000 mon.b (mon.1) 731 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:13:47.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:46.465860+0000 mon.a (mon.0) 3052 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:47.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:46.756924+0000 osd.4 (osd.4) 211 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:13:47.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:46.758731+0000 osd.4 (osd.4) 212 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:13:47.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:47.188463+0000 osd.3 (osd.3) 245 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:13:47.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:47.189807+0000 osd.3 (osd.3) 246 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:13:47.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:47.342317+0000 mon.a (mon.0) 3053 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-99"}]': finished 2024-02-19T21:13:47.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:47.342379+0000 mon.a (mon.0) 3054 : cluster [DBG] osdmap e591: 8 total, 8 up, 6 in 2024-02-19T21:13:47.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:47.344784+0000 mon.a (mon.0) 3055 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:47.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:45.764005+0000 osd.7 (osd.7) 265 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:13:47.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:45.764775+0000 osd.7 (osd.7) 266 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:13:47.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:46.411660+0000 mgr.y (mgr.14550) 515 : audit [DBG] from='client.50228 v1:172.21.15.163:0/3583448038' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:13:47.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:46.460646+0000 mon.b (mon.1) 730 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:13:47.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:46.461383+0000 mon.b (mon.1) 731 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:13:47.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:46.465860+0000 mon.a (mon.0) 3052 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:47.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:46.756924+0000 osd.4 (osd.4) 211 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:13:47.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:46.758731+0000 osd.4 (osd.4) 212 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:13:47.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:47.188463+0000 osd.3 (osd.3) 245 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:13:47.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:47.189807+0000 osd.3 (osd.3) 246 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:13:47.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:47.342317+0000 mon.a (mon.0) 3053 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-99"}]': finished 2024-02-19T21:13:47.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:47.342379+0000 mon.a (mon.0) 3054 : cluster [DBG] osdmap e591: 8 total, 8 up, 6 in 2024-02-19T21:13:47.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:47.344784+0000 mon.a (mon.0) 3055 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:47.514 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:13:47.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.430+0000 7f54ba819700 1 -- 172.21.15.163:0/3825136946 >> v1:172.21.15.163:6790/0 conn(0x7f54b41042c0 legacy=0x7f54b4106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:47.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.430+0000 7f54ba819700 1 -- 172.21.15.163:0/3825136946 shutdown_connections 2024-02-19T21:13:47.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.430+0000 7f54ba819700 1 -- 172.21.15.163:0/3825136946 >> 172.21.15.163:0/3825136946 conn(0x7f54b40f35a0 msgr2=0x7f54b40f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:47.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.430+0000 7f54ba819700 1 -- 172.21.15.163:0/3825136946 shutdown_connections 2024-02-19T21:13:47.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.430+0000 7f54ba819700 1 -- 172.21.15.163:0/3825136946 wait complete. 2024-02-19T21:13:47.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.430+0000 7f54ba819700 1 librados: starting msgr at 2024-02-19T21:13:47.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.430+0000 7f54ba819700 1 librados: starting objecter 2024-02-19T21:13:47.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54ba819700 1 Processor -- start 2024-02-19T21:13:47.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54ba819700 1 -- start start 2024-02-19T21:13:47.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54ba819700 1 librados: setting wanted keys 2024-02-19T21:13:47.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54ba819700 1 librados: calling monclient init 2024-02-19T21:13:47.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54ba819700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f54b4117350 con 0x7f54b40fd980 2024-02-19T21:13:47.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54ba819700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f54b4118530 con 0x7f54b41042c0 2024-02-19T21:13:47.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54ba819700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f54b4119730 con 0x7f54b4100780 2024-02-19T21:13:47.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54b3fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f54b40fd980 0x7f54b40f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57936/0 (socket says 172.21.15.163:57936) 2024-02-19T21:13:47.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54b37fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f54b4100780 0x7f54b40fa0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:55782/0 (socket says 172.21.15.163:55782) 2024-02-19T21:13:47.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54b3fff700 1 -- 172.21.15.163:0/3221298767 learned_addr learned my addr 172.21.15.163:0/3221298767 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:47.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54b17fa700 10 client.?.objecter ms_handle_connect 0x7f54b40fd980 2024-02-19T21:13:47.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54b17fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:47.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54b17fa700 10 client.?.objecter ms_handle_connect 0x7f54b4100780 2024-02-19T21:13:47.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54b17fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:47.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54b17fa700 10 client.?.objecter ms_handle_connect 0x7f54b41042c0 2024-02-19T21:13:47.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54b17fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:47.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978483473 0 0) 0x7f54b4117350 con 0x7f54b40fd980 2024-02-19T21:13:47.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f549c003030 con 0x7f54b40fd980 2024-02-19T21:13:47.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125189121 0 0) 0x7f54b4119730 con 0x7f54b4100780 2024-02-19T21:13:47.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b4117350 con 0x7f54b4100780 2024-02-19T21:13:47.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733291186 0 0) 0x7f54b4118530 con 0x7f54b41042c0 2024-02-19T21:13:47.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.431+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b4119730 con 0x7f54b41042c0 2024-02-19T21:13:47.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3104404769 0 0) 0x7f549c003030 con 0x7f54b40fd980 2024-02-19T21:13:47.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54b4118530 con 0x7f54b40fd980 2024-02-19T21:13:47.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f54a8001940 con 0x7f54b40fd980 2024-02-19T21:13:47.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 836109935 0 0) 0x7f54b4117350 con 0x7f54b4100780 2024-02-19T21:13:47.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f549c003030 con 0x7f54b4100780 2024-02-19T21:13:47.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f54a4001970 con 0x7f54b4100780 2024-02-19T21:13:47.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4244608885 0 0) 0x7f54b4119730 con 0x7f54b41042c0 2024-02-19T21:13:47.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54b4117350 con 0x7f54b41042c0 2024-02-19T21:13:47.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f54ac0023a0 con 0x7f54b41042c0 2024-02-19T21:13:47.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2585834210 0 0) 0x7f54b4118530 con 0x7f54b40fd980 2024-02-19T21:13:47.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 >> v1:172.21.15.163:6790/0 conn(0x7f54b4100780 legacy=0x7f54b40fa0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:47.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 >> v1:172.21.15.193:6789/0 conn(0x7f54b41042c0 legacy=0x7f54b4115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:47.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b411a930 con 0x7f54b40fd980 2024-02-19T21:13:47.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54ba819700 1 -- 172.21.15.163:0/3221298767 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b4119920 con 0x7f54b40fd980 2024-02-19T21:13:47.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54ba819700 10 client.23505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:47.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54ba819700 1 -- 172.21.15.163:0/3221298767 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b4119d90 con 0x7f54b40fd980 2024-02-19T21:13:47.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54ba819700 1 librados: init done 2024-02-19T21:13:47.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f54a8001cd0 con 0x7f54b40fd980 2024-02-19T21:13:47.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.432+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f54a8005bc0 con 0x7f54b40fd980 2024-02-19T21:13:47.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.433+0000 7f54ba819700 1 -- 172.21.15.163:0/3221298767 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a0004f20 con 0x7f54b40fd980 2024-02-19T21:13:47.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.433+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f54a8002ba0 con 0x7f54b40fd980 2024-02-19T21:13:47.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.434+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(591..591 src has 434..591) v4 ==== 8240+0+0 (unknown 2133362005 0 0) 0x7f54a8078e40 con 0x7f54b40fd980 2024-02-19T21:13:47.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.434+0000 7f54b17fa700 10 client.23505.objecter ms_dispatch 0x7f54b4001550 osd_map(591..591 src has 434..591) v4 2024-02-19T21:13:47.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.434+0000 7f54b17fa700 3 client.23505.objecter handle_osd_map got epochs [591,591] > 0 2024-02-19T21:13:47.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.434+0000 7f54b17fa700 3 client.23505.objecter handle_osd_map decoding full epoch 591 2024-02-19T21:13:47.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.434+0000 7f54b17fa700 20 client.23505.objecter dump_active .. 0 homeless 2024-02-19T21:13:47.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.434+0000 7f54b17fa700 10 client.23505.objecter ms_handle_connect 0x7f549c063c70 2024-02-19T21:13:47.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.439+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f54a804c180 con 0x7f54b40fd980 2024-02-19T21:13:47.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:45.764005+0000 osd.7 (osd.7) 265 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:13:47.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:45.764775+0000 osd.7 (osd.7) 266 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:13:47.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:46.411660+0000 mgr.y (mgr.14550) 515 : audit [DBG] from='client.50228 v1:172.21.15.163:0/3583448038' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:13:47.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:46.460646+0000 mon.b (mon.1) 730 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:13:47.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:46.461383+0000 mon.b (mon.1) 731 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:13:47.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:46.465860+0000 mon.a (mon.0) 3052 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:47.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:46.756924+0000 osd.4 (osd.4) 211 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:13:47.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:46.758731+0000 osd.4 (osd.4) 212 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:13:47.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:47.188463+0000 osd.3 (osd.3) 245 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:13:47.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:47.189807+0000 osd.3 (osd.3) 246 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:13:47.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:47.342317+0000 mon.a (mon.0) 3053 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-99"}]': finished 2024-02-19T21:13:47.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:47.342379+0000 mon.a (mon.0) 3054 : cluster [DBG] osdmap e591: 8 total, 8 up, 6 in 2024-02-19T21:13:47.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:47.344784+0000 mon.a (mon.0) 3055 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-99"}]: dispatch 2024-02-19T21:13:47.618 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:47.618+0000 7f54ba819700 1 -- 172.21.15.163:0/3221298767 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f54a00051d0 con 0x7f54b40fd980 2024-02-19T21:13:47.968 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:47.981 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":884,"stamp":"2024-02-19T21:13:46.339799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5365,"num_objects":194,"num_object_clones":0,"num_object_copies":582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":1865,"num_read_kb":1615,"num_write":721,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":11084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":823,"ondisk_log_size":823,"up":546,"acting":546,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":666,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":1858464,"kb_used_data":17776,"kb_used_omap":0,"kb_used_meta":1840640,"kb_avail":560612448,"statfs":{"total":575970213888,"available":574067146752,"internally_reserved":0,"allocated":18202624,"data_stored":8805405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1884815360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-436,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.994856"},"pg_stats":[{"pgid":"229.7","version":"0'0","reported_seq":34,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.383563+0000","last_change":"2024-02-19T21:13:19.370447+0000","last_active":"2024-02-19T21:13:43.383563+0000","last_peered":"2024-02-19T21:13:43.383563+0000","last_clean":"2024-02-19T21:13:43.383563+0000","last_became_active":"2024-02-19T21:13:19.370104+0000","last_became_peered":"2024-02-19T21:13:19.370104+0000","last_unstale":"2024-02-19T21:13:43.383563+0000","last_undegraded":"2024-02-19T21:13:43.383563+0000","last_fullsized":"2024-02-19T21:13:43.383563+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"229.6","version":"0'0","reported_seq":34,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.380940+0000","last_change":"2024-02-19T21:13:19.375127+0000","last_active":"2024-02-19T21:13:43.380940+0000","last_peered":"2024-02-19T21:13:43.380940+0000","last_clean":"2024-02-19T21:13:43.380940+0000","last_became_active":"2024-02-19T21:13:19.374800+0000","last_became_peered":"2024-02-19T21:13:19.374800+0000","last_unstale":"2024-02-19T21:13:43.380940+0000","last_undegraded":"2024-02-19T21:13:43.380940+0000","last_fullsized":"2024-02-19T21:13:43.380940+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"229.5","version":"0'0","reported_seq":34,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382263+0000","last_change":"2024-02-19T21:13:19.371925+0000","last_active":"2024-02-19T21:13:43.382263+0000","last_peered":"2024-02-19T21:13:43.382263+0000","last_clean":"2024-02-19T21:13:43.382263+0000","last_became_active":"2024-02-19T21:13:19.371633+0000","last_became_peered":"2024-02-19T21:13:19.371633+0000","last_unstale":"2024-02-19T21:13:43.382263+0000","last_undegraded":"2024-02-19T21:13:43.382263+0000","last_fullsized":"2024-02-19T21:13:43.382263+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"229.4","version":"0'0","reported_seq":33,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.377914+0000","last_change":"2024-02-19T21:13:19.376108+0000","last_active":"2024-02-19T21:13:43.377914+0000","last_peered":"2024-02-19T21:13:43.377914+0000","last_clean":"2024-02-19T21:13:43.377914+0000","last_became_active":"2024-02-19T21:13:19.375814+0000","last_became_peered":"2024-02-19T21:13:19.375814+0000","last_unstale":"2024-02-19T21:13:43.377914+0000","last_undegraded":"2024-02-19T21:13:43.377914+0000","last_fullsized":"2024-02-19T21:13:43.377914+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"229.3","version":"0'0","reported_seq":32,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381096+0000","last_change":"2024-02-19T21:13:19.373500+0000","last_active":"2024-02-19T21:13:43.381096+0000","last_peered":"2024-02-19T21:13:43.381096+0000","last_clean":"2024-02-19T21:13:43.381096+0000","last_became_active":"2024-02-19T21:13:19.373259+0000","last_became_peered":"2024-02-19T21:13:19.373259+0000","last_unstale":"2024-02-19T21:13:43.381096+0000","last_undegraded":"2024-02-19T21:13:43.381096+0000","last_fullsized":"2024-02-19T21:13:43.381096+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"229.2","version":"0'0","reported_seq":34,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381236+0000","last_change":"2024-02-19T21:13:19.371648+0000","last_active":"2024-02-19T21:13:43.381236+0000","last_peered":"2024-02-19T21:13:43.381236+0000","last_clean":"2024-02-19T21:13:43.381236+0000","last_became_active":"2024-02-19T21:13:19.371295+0000","last_became_peered":"2024-02-19T21:13:19.371295+0000","last_unstale":"2024-02-19T21:13:43.381236+0000","last_undegraded":"2024-02-19T21:13:43.381236+0000","last_fullsized":"2024-02-19T21:13:43.381236+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"229.1","version":"563'4","reported_seq":39,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.377471+0000","last_change":"2024-02-19T21:13:19.371167+0000","last_active":"2024-02-19T21:13:43.377471+0000","last_peered":"2024-02-19T21:13:43.377471+0000","last_clean":"2024-02-19T21:13:43.377471+0000","last_became_active":"2024-02-19T21:13:19.370517+0000","last_became_peered":"2024-02-19T21:13:19.370517+0000","last_unstale":"2024-02-19T21:13:43.377471+0000","last_undegraded":"2024-02-19T21:13:43.377471+0000","last_fullsized":"2024-02-19T21:13:43.377471+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"229.0","version":"0'0","reported_seq":34,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374561+0000","last_change":"2024-02-19T21:13:19.374080+0000","last_active":"2024-02-19T21:13:43.374561+0000","last_peered":"2024-02-19T21:13:43.374561+0000","last_clean":"2024-02-19T21:13:43.374561+0000","last_became_active":"2024-02-19T21:13:19.373744+0000","last_became_peered":"2024-02-19T21:13:19.373744+0000","last_unstale":"2024-02-19T21:13:43.374561+0000","last_undegraded":"2024-02-19T21:13:43.374561+0000","last_fullsized":"2024-02-19T21:13:43.374561+0000","mapping_epoch":562,"log_start":"0'0","ondisk_log_start":"0'0","created":562,"last_epoch_clean":563,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:18.340200+0000","last_clean_scrub_stamp":"2024-02-19T21:13:18.340200+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":591,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375281+0000","last_change":"2024-02-19T21:12:09.321398+0000","last_active":"2024-02-19T21:13:43.375281+0000","last_peered":"2024-02-19T21:13:43.375281+0000","last_clean":"2024-02-19T21:13:43.375281+0000","last_became_active":"2024-02-19T21:12:09.320924+0000","last_became_peered":"2024-02-19T21:12:09.320924+0000","last_unstale":"2024-02-19T21:13:43.375281+0000","last_undegraded":"2024-02-19T21:13:43.375281+0000","last_fullsized":"2024-02-19T21:13:43.375281+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.738600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.738600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":121,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374827+0000","last_change":"2024-02-19T21:11:50.304127+0000","last_active":"2024-02-19T21:13:43.374827+0000","last_peered":"2024-02-19T21:13:43.374827+0000","last_clean":"2024-02-19T21:13:43.374827+0000","last_became_active":"2024-02-19T21:11:50.303790+0000","last_became_peered":"2024-02-19T21:11:50.303790+0000","last_unstale":"2024-02-19T21:13:43.374827+0000","last_undegraded":"2024-02-19T21:13:43.374827+0000","last_fullsized":"2024-02-19T21:13:43.374827+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:42.529489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:42.529489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.25","version":"0'0","reported_seq":624,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375077+0000","last_change":"2024-02-19T21:12:09.349611+0000","last_active":"2024-02-19T21:13:43.375077+0000","last_peered":"2024-02-19T21:13:43.375077+0000","last_clean":"2024-02-19T21:13:43.375077+0000","last_became_active":"2024-02-19T21:12:09.349271+0000","last_became_peered":"2024-02-19T21:12:09.349271+0000","last_unstale":"2024-02-19T21:13:43.375077+0000","last_undegraded":"2024-02-19T21:13:43.375077+0000","last_fullsized":"2024-02-19T21:13:43.375077+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:40.580344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:40.580344+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":598,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381840+0000","last_change":"2024-02-19T21:12:09.339659+0000","last_active":"2024-02-19T21:13:43.381840+0000","last_peered":"2024-02-19T21:13:43.381840+0000","last_clean":"2024-02-19T21:13:43.381840+0000","last_became_active":"2024-02-19T21:12:09.339310+0000","last_became_peered":"2024-02-19T21:12:09.339310+0000","last_unstale":"2024-02-19T21:13:43.381840+0000","last_undegraded":"2024-02-19T21:13:43.381840+0000","last_fullsized":"2024-02-19T21:13:43.381840+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:21.732778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:10:21.732778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":593,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.384202+0000","last_change":"2024-02-19T21:13:36.863848+0000","last_active":"2024-02-19T21:13:43.384202+0000","last_peered":"2024-02-19T21:13:43.384202+0000","last_clean":"2024-02-19T21:13:43.384202+0000","last_became_active":"2024-02-19T21:11:50.313560+0000","last_became_peered":"2024-02-19T21:11:50.313560+0000","last_unstale":"2024-02-19T21:13:43.384202+0000","last_undegraded":"2024-02-19T21:13:43.384202+0000","last_fullsized":"2024-02-19T21:13:43.384202+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:36.863734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:36.863734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.26","version":"0'0","reported_seq":125,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.384410+0000","last_change":"2024-02-19T21:12:09.339184+0000","last_active":"2024-02-19T21:13:43.384410+0000","last_peered":"2024-02-19T21:13:43.384410+0000","last_clean":"2024-02-19T21:13:43.384410+0000","last_became_active":"2024-02-19T21:12:09.338737+0000","last_became_peered":"2024-02-19T21:12:09.338737+0000","last_unstale":"2024-02-19T21:13:43.384410+0000","last_undegraded":"2024-02-19T21:13:43.384410+0000","last_fullsized":"2024-02-19T21:13:43.384410+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":630,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.377750+0000","last_change":"2024-02-19T21:12:09.343445+0000","last_active":"2024-02-19T21:13:43.377750+0000","last_peered":"2024-02-19T21:13:43.377750+0000","last_clean":"2024-02-19T21:13:43.377750+0000","last_became_active":"2024-02-19T21:12:09.342583+0000","last_became_peered":"2024-02-19T21:12:09.342583+0000","last_unstale":"2024-02-19T21:13:43.377750+0000","last_undegraded":"2024-02-19T21:13:43.377750+0000","last_fullsized":"2024-02-19T21:13:43.377750+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:37.955872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:09:37.955872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.20","version":"397'2","reported_seq":650,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381567+0000","last_change":"2024-02-19T21:12:43.380032+0000","last_active":"2024-02-19T21:13:43.381567+0000","last_peered":"2024-02-19T21:13:43.381567+0000","last_clean":"2024-02-19T21:13:43.381567+0000","last_became_active":"2024-02-19T21:12:43.379693+0000","last_became_peered":"2024-02-19T21:12:43.379693+0000","last_unstale":"2024-02-19T21:13:43.381567+0000","last_undegraded":"2024-02-19T21:13:43.381567+0000","last_fullsized":"2024-02-19T21:13:43.381567+0000","mapping_epoch":526,"log_start":"58'1","ondisk_log_start":"58'1","created":474,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":6,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.934474+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":437,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":130,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375668+0000","last_change":"2024-02-19T21:12:09.337725+0000","last_active":"2024-02-19T21:13:43.375668+0000","last_peered":"2024-02-19T21:13:43.375668+0000","last_clean":"2024-02-19T21:13:43.375668+0000","last_became_active":"2024-02-19T21:12:09.337337+0000","last_became_peered":"2024-02-19T21:12:09.337337+0000","last_unstale":"2024-02-19T21:13:43.375668+0000","last_undegraded":"2024-02-19T21:13:43.375668+0000","last_fullsized":"2024-02-19T21:13:43.375668+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.081993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.081993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":579,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374259+0000","last_change":"2024-02-19T21:12:09.353297+0000","last_active":"2024-02-19T21:13:43.374259+0000","last_peered":"2024-02-19T21:13:43.374259+0000","last_clean":"2024-02-19T21:13:43.374259+0000","last_became_active":"2024-02-19T21:12:09.352904+0000","last_became_peered":"2024-02-19T21:12:09.352904+0000","last_unstale":"2024-02-19T21:13:43.374259+0000","last_undegraded":"2024-02-19T21:13:43.374259+0000","last_fullsized":"2024-02-19T21:13:43.374259+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.089715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.089715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"493'12","reported_seq":750,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-02-19T21:13:44.457319+0000","last_change":"2024-02-19T21:12:18.452113+0000","last_active":"2024-02-19T21:13:44.457319+0000","last_peered":"2024-02-19T21:13:44.457319+0000","last_clean":"2024-02-19T21:13:44.457319+0000","last_became_active":"2024-02-19T21:12:09.332507+0000","last_became_peered":"2024-02-19T21:12:09.332507+0000","last_unstale":"2024-02-19T21:13:44.457319+0000","last_undegraded":"2024-02-19T21:13:44.457319+0000","last_fullsized":"2024-02-19T21:13:44.457319+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'12","last_scrub_stamp":"2024-02-19T21:12:18.452020+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:12:18.452020+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'28","reported_seq":624,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381651+0000","last_change":"2024-02-19T21:12:10.211768+0000","last_active":"2024-02-19T21:13:43.381651+0000","last_peered":"2024-02-19T21:13:43.381651+0000","last_clean":"2024-02-19T21:13:43.381651+0000","last_became_active":"2024-02-19T21:12:09.341122+0000","last_became_peered":"2024-02-19T21:12:09.341122+0000","last_unstale":"2024-02-19T21:13:43.381651+0000","last_undegraded":"2024-02-19T21:13:43.381651+0000","last_fullsized":"2024-02-19T21:13:43.381651+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-19T21:11:47.051823+0000","last_deep_scrub":"473'28","last_deep_scrub_stamp":"2024-02-19T21:11:47.051823+0000","last_clean_scrub_stamp":"2024-02-19T21:11:47.051823+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":586,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375813+0000","last_change":"2024-02-19T21:12:22.215650+0000","last_active":"2024-02-19T21:13:43.375813+0000","last_peered":"2024-02-19T21:13:43.375813+0000","last_clean":"2024-02-19T21:13:43.375813+0000","last_became_active":"2024-02-19T21:10:40.299425+0000","last_became_peered":"2024-02-19T21:10:40.299425+0000","last_unstale":"2024-02-19T21:13:43.375813+0000","last_undegraded":"2024-02-19T21:13:43.375813+0000","last_fullsized":"2024-02-19T21:13:43.375813+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:22.215548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_clean_scrub_stamp":"2024-02-19T21:12:22.215548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":566,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381641+0000","last_change":"2024-02-19T21:12:51.469049+0000","last_active":"2024-02-19T21:13:43.381641+0000","last_peered":"2024-02-19T21:13:43.381641+0000","last_clean":"2024-02-19T21:13:43.381641+0000","last_became_active":"2024-02-19T21:12:09.326173+0000","last_became_peered":"2024-02-19T21:12:09.326173+0000","last_unstale":"2024-02-19T21:13:43.381641+0000","last_undegraded":"2024-02-19T21:13:43.381641+0000","last_fullsized":"2024-02-19T21:13:43.381641+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:51.468971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.468971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":507,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.378608+0000","last_change":"2024-02-19T21:12:58.081198+0000","last_active":"2024-02-19T21:13:43.378608+0000","last_peered":"2024-02-19T21:13:43.378608+0000","last_clean":"2024-02-19T21:13:43.378608+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:13:43.378608+0000","last_undegraded":"2024-02-19T21:13:43.378608+0000","last_fullsized":"2024-02-19T21:13:43.378608+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:58.081106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:58.081106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'16","reported_seq":478,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382107+0000","last_change":"2024-02-19T21:11:59.756726+0000","last_active":"2024-02-19T21:13:43.382107+0000","last_peered":"2024-02-19T21:13:43.382107+0000","last_clean":"2024-02-19T21:13:43.382107+0000","last_became_active":"2024-02-19T21:11:59.319397+0000","last_became_peered":"2024-02-19T21:11:59.319397+0000","last_unstale":"2024-02-19T21:13:43.382107+0000","last_undegraded":"2024-02-19T21:13:43.382107+0000","last_fullsized":"2024-02-19T21:13:43.382107+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'16","last_scrub_stamp":"2024-02-19T21:11:46.956424+0000","last_deep_scrub":"473'16","last_deep_scrub_stamp":"2024-02-19T21:11:46.956424+0000","last_clean_scrub_stamp":"2024-02-19T21:11:46.956424+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":593,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382418+0000","last_change":"2024-02-19T21:12:06.016884+0000","last_active":"2024-02-19T21:13:43.382418+0000","last_peered":"2024-02-19T21:13:43.382418+0000","last_clean":"2024-02-19T21:13:43.382418+0000","last_became_active":"2024-02-19T21:10:41.312116+0000","last_became_peered":"2024-02-19T21:10:41.312116+0000","last_unstale":"2024-02-19T21:13:43.382418+0000","last_undegraded":"2024-02-19T21:13:43.382418+0000","last_fullsized":"2024-02-19T21:13:43.382418+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.016817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.016817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":591,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374733+0000","last_change":"2024-02-19T21:13:05.193297+0000","last_active":"2024-02-19T21:13:43.374733+0000","last_peered":"2024-02-19T21:13:43.374733+0000","last_clean":"2024-02-19T21:13:43.374733+0000","last_became_active":"2024-02-19T21:10:41.305574+0000","last_became_peered":"2024-02-19T21:10:41.305574+0000","last_unstale":"2024-02-19T21:13:43.374733+0000","last_undegraded":"2024-02-19T21:13:43.374733+0000","last_fullsized":"2024-02-19T21:13:43.374733+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:05.193223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:05.193223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":391,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.383726+0000","last_change":"2024-02-19T21:11:59.323172+0000","last_active":"2024-02-19T21:13:43.383726+0000","last_peered":"2024-02-19T21:13:43.383726+0000","last_clean":"2024-02-19T21:13:43.383726+0000","last_became_active":"2024-02-19T21:11:59.322814+0000","last_became_peered":"2024-02-19T21:11:59.322814+0000","last_unstale":"2024-02-19T21:13:43.383726+0000","last_undegraded":"2024-02-19T21:13:43.383726+0000","last_fullsized":"2024-02-19T21:13:43.383726+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:34.504478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:34.504478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":502,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.383171+0000","last_change":"2024-02-19T21:10:57.130256+0000","last_active":"2024-02-19T21:13:43.383171+0000","last_peered":"2024-02-19T21:13:43.383171+0000","last_clean":"2024-02-19T21:13:43.383171+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:13:43.383171+0000","last_undegraded":"2024-02-19T21:13:43.383171+0000","last_fullsized":"2024-02-19T21:13:43.383171+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.130171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.130171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"459'35","reported_seq":606,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381818+0000","last_change":"2024-02-19T21:12:10.552781+0000","last_active":"2024-02-19T21:13:43.381818+0000","last_peered":"2024-02-19T21:13:43.381818+0000","last_clean":"2024-02-19T21:13:43.381818+0000","last_became_active":"2024-02-19T21:12:09.333541+0000","last_became_peered":"2024-02-19T21:12:09.333541+0000","last_unstale":"2024-02-19T21:13:43.381818+0000","last_undegraded":"2024-02-19T21:13:43.381818+0000","last_fullsized":"2024-02-19T21:13:43.381818+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:11:04.345836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:04.345836+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":568,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.378440+0000","last_change":"2024-02-19T21:12:09.339263+0000","last_active":"2024-02-19T21:13:43.378440+0000","last_peered":"2024-02-19T21:13:43.378440+0000","last_clean":"2024-02-19T21:13:43.378440+0000","last_became_active":"2024-02-19T21:11:59.336426+0000","last_became_peered":"2024-02-19T21:11:59.336426+0000","last_unstale":"2024-02-19T21:13:43.378440+0000","last_undegraded":"2024-02-19T21:13:43.378440+0000","last_fullsized":"2024-02-19T21:13:43.378440+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:09.339147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:12:09.339147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":561,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376013+0000","last_change":"2024-02-19T21:12:16.244173+0000","last_active":"2024-02-19T21:13:43.376013+0000","last_peered":"2024-02-19T21:13:43.376013+0000","last_clean":"2024-02-19T21:13:43.376013+0000","last_became_active":"2024-02-19T21:11:59.340544+0000","last_became_peered":"2024-02-19T21:11:59.340544+0000","last_unstale":"2024-02-19T21:13:43.376013+0000","last_undegraded":"2024-02-19T21:13:43.376013+0000","last_fullsized":"2024-02-19T21:13:43.376013+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:16.244104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:16.244104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":562,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381714+0000","last_change":"2024-02-19T21:11:33.469310+0000","last_active":"2024-02-19T21:13:43.381714+0000","last_peered":"2024-02-19T21:13:43.381714+0000","last_clean":"2024-02-19T21:13:43.381714+0000","last_became_active":"2024-02-19T21:08:04.265081+0000","last_became_peered":"2024-02-19T21:08:04.265081+0000","last_unstale":"2024-02-19T21:13:43.381714+0000","last_undegraded":"2024-02-19T21:13:43.381714+0000","last_fullsized":"2024-02-19T21:13:43.381714+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:33.469232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_clean_scrub_stamp":"2024-02-19T21:11:33.469232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":522,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.384409+0000","last_change":"2024-02-19T21:12:55.491781+0000","last_active":"2024-02-19T21:13:43.384409+0000","last_peered":"2024-02-19T21:13:43.384409+0000","last_clean":"2024-02-19T21:13:43.384409+0000","last_became_active":"2024-02-19T21:10:41.317284+0000","last_became_peered":"2024-02-19T21:10:41.317284+0000","last_unstale":"2024-02-19T21:13:43.384409+0000","last_undegraded":"2024-02-19T21:13:43.384409+0000","last_fullsized":"2024-02-19T21:13:43.384409+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:55.491665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.491665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"459'31","reported_seq":207,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375541+0000","last_change":"2024-02-19T21:11:44.421114+0000","last_active":"2024-02-19T21:13:43.375541+0000","last_peered":"2024-02-19T21:13:43.375541+0000","last_clean":"2024-02-19T21:13:43.375541+0000","last_became_active":"2024-02-19T21:10:41.313853+0000","last_became_peered":"2024-02-19T21:10:41.313853+0000","last_unstale":"2024-02-19T21:13:43.375541+0000","last_undegraded":"2024-02-19T21:13:43.375541+0000","last_fullsized":"2024-02-19T21:13:43.375541+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'31","last_scrub_stamp":"2024-02-19T21:11:44.421033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:44.421033+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":587,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.383709+0000","last_change":"2024-02-19T21:13:07.450202+0000","last_active":"2024-02-19T21:13:43.383709+0000","last_peered":"2024-02-19T21:13:43.383709+0000","last_clean":"2024-02-19T21:13:43.383709+0000","last_became_active":"2024-02-19T21:10:41.307517+0000","last_became_peered":"2024-02-19T21:10:41.307517+0000","last_unstale":"2024-02-19T21:13:43.383709+0000","last_undegraded":"2024-02-19T21:13:43.383709+0000","last_fullsized":"2024-02-19T21:13:43.383709+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:13:07.450123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.450123+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":589,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382377+0000","last_change":"2024-02-19T21:12:58.838081+0000","last_active":"2024-02-19T21:13:43.382377+0000","last_peered":"2024-02-19T21:13:43.382377+0000","last_clean":"2024-02-19T21:13:43.382377+0000","last_became_active":"2024-02-19T21:12:09.369409+0000","last_became_peered":"2024-02-19T21:12:09.369409+0000","last_unstale":"2024-02-19T21:13:43.382377+0000","last_undegraded":"2024-02-19T21:13:43.382377+0000","last_fullsized":"2024-02-19T21:13:43.382377+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:58.837973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:58.837973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":589,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.377223+0000","last_change":"2024-02-19T21:13:03.126485+0000","last_active":"2024-02-19T21:13:43.377223+0000","last_peered":"2024-02-19T21:13:43.377223+0000","last_clean":"2024-02-19T21:13:43.377223+0000","last_became_active":"2024-02-19T21:10:41.307577+0000","last_became_peered":"2024-02-19T21:10:41.307577+0000","last_unstale":"2024-02-19T21:13:43.377223+0000","last_undegraded":"2024-02-19T21:13:43.377223+0000","last_fullsized":"2024-02-19T21:13:43.377223+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:03.126414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:13:03.126414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":168,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374164+0000","last_change":"2024-02-19T21:10:50.424462+0000","last_active":"2024-02-19T21:13:43.374164+0000","last_peered":"2024-02-19T21:13:43.374164+0000","last_clean":"2024-02-19T21:13:43.374164+0000","last_became_active":"2024-02-19T21:10:41.302670+0000","last_became_peered":"2024-02-19T21:10:41.302670+0000","last_unstale":"2024-02-19T21:13:43.374164+0000","last_undegraded":"2024-02-19T21:13:43.374164+0000","last_fullsized":"2024-02-19T21:13:43.374164+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:50.424381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.424381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"459'25","reported_seq":216,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374522+0000","last_change":"2024-02-19T21:12:51.420519+0000","last_active":"2024-02-19T21:13:43.374522+0000","last_peered":"2024-02-19T21:13:43.374522+0000","last_clean":"2024-02-19T21:13:43.374522+0000","last_became_active":"2024-02-19T21:12:09.408620+0000","last_became_peered":"2024-02-19T21:12:09.408620+0000","last_unstale":"2024-02-19T21:13:43.374522+0000","last_undegraded":"2024-02-19T21:13:43.374522+0000","last_fullsized":"2024-02-19T21:13:43.374522+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'25","last_scrub_stamp":"2024-02-19T21:12:51.420403+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.420403+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":578,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381825+0000","last_change":"2024-02-19T21:13:38.073731+0000","last_active":"2024-02-19T21:13:43.381825+0000","last_peered":"2024-02-19T21:13:43.381825+0000","last_clean":"2024-02-19T21:13:43.381825+0000","last_became_active":"2024-02-19T21:08:04.268406+0000","last_became_peered":"2024-02-19T21:08:04.268406+0000","last_unstale":"2024-02-19T21:13:43.381825+0000","last_undegraded":"2024-02-19T21:13:43.381825+0000","last_fullsized":"2024-02-19T21:13:43.381825+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:38.073668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:38.073668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":713,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-02-19T21:13:44.456169+0000","last_change":"2024-02-19T21:12:21.202447+0000","last_active":"2024-02-19T21:13:44.456169+0000","last_peered":"2024-02-19T21:13:44.456169+0000","last_clean":"2024-02-19T21:13:44.456169+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:13:44.456169+0000","last_undegraded":"2024-02-19T21:13:44.456169+0000","last_fullsized":"2024-02-19T21:13:44.456169+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:12:21.202371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:21.202371+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":168,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375769+0000","last_change":"2024-02-19T21:12:21.380805+0000","last_active":"2024-02-19T21:13:43.375769+0000","last_peered":"2024-02-19T21:13:43.375769+0000","last_clean":"2024-02-19T21:13:43.375769+0000","last_became_active":"2024-02-19T21:10:41.298893+0000","last_became_peered":"2024-02-19T21:10:41.298893+0000","last_unstale":"2024-02-19T21:13:43.375769+0000","last_undegraded":"2024-02-19T21:13:43.375769+0000","last_fullsized":"2024-02-19T21:13:43.375769+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:21.380734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:12:21.380734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":315,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375046+0000","last_change":"2024-02-19T21:12:16.339644+0000","last_active":"2024-02-19T21:13:43.375046+0000","last_peered":"2024-02-19T21:13:43.375046+0000","last_clean":"2024-02-19T21:13:43.375046+0000","last_became_active":"2024-02-19T21:11:59.318845+0000","last_became_peered":"2024-02-19T21:11:59.318845+0000","last_unstale":"2024-02-19T21:13:43.375046+0000","last_undegraded":"2024-02-19T21:13:43.375046+0000","last_fullsized":"2024-02-19T21:13:43.375046+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:16.339577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:16.339577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"459'27","reported_seq":211,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.377025+0000","last_change":"2024-02-19T21:11:08.202280+0000","last_active":"2024-02-19T21:13:43.377025+0000","last_peered":"2024-02-19T21:13:43.377025+0000","last_clean":"2024-02-19T21:13:43.377025+0000","last_became_active":"2024-02-19T21:10:41.307576+0000","last_became_peered":"2024-02-19T21:10:41.307576+0000","last_unstale":"2024-02-19T21:13:43.377025+0000","last_undegraded":"2024-02-19T21:13:43.377025+0000","last_fullsized":"2024-02-19T21:13:43.377025+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:11:08.202208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:08.202208+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":615,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381546+0000","last_change":"2024-02-19T21:12:40.370031+0000","last_active":"2024-02-19T21:13:43.381546+0000","last_peered":"2024-02-19T21:13:43.381546+0000","last_clean":"2024-02-19T21:13:43.381546+0000","last_became_active":"2024-02-19T21:12:40.369640+0000","last_became_peered":"2024-02-19T21:12:40.369640+0000","last_unstale":"2024-02-19T21:13:43.381546+0000","last_undegraded":"2024-02-19T21:13:43.381546+0000","last_fullsized":"2024-02-19T21:13:43.381546+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.738600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.738600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"483'11","reported_seq":223,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-02-19T21:13:44.340139+0000","last_change":"2024-02-19T21:12:56.450133+0000","last_active":"2024-02-19T21:13:44.340139+0000","last_peered":"2024-02-19T21:13:44.340139+0000","last_clean":"2024-02-19T21:13:44.340139+0000","last_became_active":"2024-02-19T21:11:59.336645+0000","last_became_peered":"2024-02-19T21:11:59.336645+0000","last_unstale":"2024-02-19T21:13:44.340139+0000","last_undegraded":"2024-02-19T21:13:44.340139+0000","last_fullsized":"2024-02-19T21:13:44.340139+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"483'11","last_scrub_stamp":"2024-02-19T21:12:56.450044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:56.450044+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":562,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382784+0000","last_change":"2024-02-19T21:12:54.437089+0000","last_active":"2024-02-19T21:13:43.382784+0000","last_peered":"2024-02-19T21:13:43.382784+0000","last_clean":"2024-02-19T21:13:43.382784+0000","last_became_active":"2024-02-19T21:11:59.344440+0000","last_became_peered":"2024-02-19T21:11:59.344440+0000","last_unstale":"2024-02-19T21:13:43.382784+0000","last_undegraded":"2024-02-19T21:13:43.382784+0000","last_fullsized":"2024-02-19T21:13:43.382784+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:54.436977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:54.436977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":511,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382295+0000","last_change":"2024-02-19T21:12:09.327574+0000","last_active":"2024-02-19T21:13:43.382295+0000","last_peered":"2024-02-19T21:13:43.382295+0000","last_clean":"2024-02-19T21:13:43.382295+0000","last_became_active":"2024-02-19T21:12:09.326909+0000","last_became_peered":"2024-02-19T21:12:09.326909+0000","last_unstale":"2024-02-19T21:13:43.382295+0000","last_undegraded":"2024-02-19T21:13:43.382295+0000","last_fullsized":"2024-02-19T21:13:43.382295+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.079353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.079353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"459'19","reported_seq":215,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375728+0000","last_change":"2024-02-19T21:12:10.647686+0000","last_active":"2024-02-19T21:13:43.375728+0000","last_peered":"2024-02-19T21:13:43.375728+0000","last_clean":"2024-02-19T21:13:43.375728+0000","last_became_active":"2024-02-19T21:12:09.409065+0000","last_became_peered":"2024-02-19T21:12:09.409065+0000","last_unstale":"2024-02-19T21:13:43.375728+0000","last_undegraded":"2024-02-19T21:13:43.375728+0000","last_fullsized":"2024-02-19T21:13:43.375728+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:11:42.333781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:42.333781+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":605,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381891+0000","last_change":"2024-02-19T21:13:37.829009+0000","last_active":"2024-02-19T21:13:43.381891+0000","last_peered":"2024-02-19T21:13:43.381891+0000","last_clean":"2024-02-19T21:13:43.381891+0000","last_became_active":"2024-02-19T21:11:59.331675+0000","last_became_peered":"2024-02-19T21:11:59.331675+0000","last_unstale":"2024-02-19T21:13:43.381891+0000","last_undegraded":"2024-02-19T21:13:43.381891+0000","last_fullsized":"2024-02-19T21:13:43.381891+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:37.828911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:37.828911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":576,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.383256+0000","last_change":"2024-02-19T21:12:09.350127+0000","last_active":"2024-02-19T21:13:43.383256+0000","last_peered":"2024-02-19T21:13:43.383256+0000","last_clean":"2024-02-19T21:13:43.383256+0000","last_became_active":"2024-02-19T21:12:09.349667+0000","last_became_peered":"2024-02-19T21:12:09.349667+0000","last_unstale":"2024-02-19T21:13:43.383256+0000","last_undegraded":"2024-02-19T21:13:43.383256+0000","last_fullsized":"2024-02-19T21:13:43.383256+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:58.096025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.096025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":308,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.377019+0000","last_change":"2024-02-19T21:12:09.335125+0000","last_active":"2024-02-19T21:13:43.377019+0000","last_peered":"2024-02-19T21:13:43.377019+0000","last_clean":"2024-02-19T21:13:43.377019+0000","last_became_active":"2024-02-19T21:12:09.333625+0000","last_became_peered":"2024-02-19T21:12:09.333625+0000","last_unstale":"2024-02-19T21:13:43.377019+0000","last_undegraded":"2024-02-19T21:13:43.377019+0000","last_fullsized":"2024-02-19T21:13:43.377019+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:06.547494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:06.547494+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":308,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376742+0000","last_change":"2024-02-19T21:12:09.348064+0000","last_active":"2024-02-19T21:13:43.376742+0000","last_peered":"2024-02-19T21:13:43.376742+0000","last_clean":"2024-02-19T21:13:43.376742+0000","last_became_active":"2024-02-19T21:12:09.347240+0000","last_became_peered":"2024-02-19T21:12:09.347240+0000","last_unstale":"2024-02-19T21:13:43.376742+0000","last_undegraded":"2024-02-19T21:13:43.376742+0000","last_fullsized":"2024-02-19T21:13:43.376742+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:43.407517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.407517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"434'12","reported_seq":401,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-02-19T21:13:44.342997+0000","last_change":"2024-02-19T21:10:41.321881+0000","last_active":"2024-02-19T21:13:44.342997+0000","last_peered":"2024-02-19T21:13:44.342997+0000","last_clean":"2024-02-19T21:13:44.342997+0000","last_became_active":"2024-02-19T21:10:41.297897+0000","last_became_peered":"2024-02-19T21:10:41.297897+0000","last_unstale":"2024-02-19T21:13:44.342997+0000","last_undegraded":"2024-02-19T21:13:44.342997+0000","last_fullsized":"2024-02-19T21:13:44.342997+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'11","last_scrub_stamp":"2024-02-19T21:09:39.597001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:09:39.597001+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"459'28","reported_seq":230,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374198+0000","last_change":"2024-02-19T21:12:10.323234+0000","last_active":"2024-02-19T21:13:43.374198+0000","last_peered":"2024-02-19T21:13:43.374198+0000","last_clean":"2024-02-19T21:13:43.374198+0000","last_became_active":"2024-02-19T21:12:09.369385+0000","last_became_peered":"2024-02-19T21:12:09.369385+0000","last_unstale":"2024-02-19T21:13:43.374198+0000","last_undegraded":"2024-02-19T21:13:43.374198+0000","last_fullsized":"2024-02-19T21:13:43.374198+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2024-02-19T21:10:49.466227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:49.466227+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":573,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.384304+0000","last_change":"2024-02-19T21:12:09.343164+0000","last_active":"2024-02-19T21:13:43.384304+0000","last_peered":"2024-02-19T21:13:43.384304+0000","last_clean":"2024-02-19T21:13:43.384304+0000","last_became_active":"2024-02-19T21:12:09.342814+0000","last_became_peered":"2024-02-19T21:12:09.342814+0000","last_unstale":"2024-02-19T21:13:43.384304+0000","last_undegraded":"2024-02-19T21:13:43.384304+0000","last_fullsized":"2024-02-19T21:13:43.384304+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.364732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.364732+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":557,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.384040+0000","last_change":"2024-02-19T21:12:09.335134+0000","last_active":"2024-02-19T21:13:43.384040+0000","last_peered":"2024-02-19T21:13:43.384040+0000","last_clean":"2024-02-19T21:13:43.384040+0000","last_became_active":"2024-02-19T21:12:09.333970+0000","last_became_peered":"2024-02-19T21:12:09.333970+0000","last_unstale":"2024-02-19T21:13:43.384040+0000","last_undegraded":"2024-02-19T21:13:43.384040+0000","last_fullsized":"2024-02-19T21:13:43.384040+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.851123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.851123+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":392,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382684+0000","last_change":"2024-02-19T21:13:11.004073+0000","last_active":"2024-02-19T21:13:43.382684+0000","last_peered":"2024-02-19T21:13:43.382684+0000","last_clean":"2024-02-19T21:13:43.382684+0000","last_became_active":"2024-02-19T21:11:59.323698+0000","last_became_peered":"2024-02-19T21:11:59.323698+0000","last_unstale":"2024-02-19T21:13:43.382684+0000","last_undegraded":"2024-02-19T21:13:43.382684+0000","last_fullsized":"2024-02-19T21:13:43.382684+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:11.004006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:11.004006+0000","last_clean_scrub_stamp":"2024-02-19T21:13:11.004006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":168,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374076+0000","last_change":"2024-02-19T21:10:59.312931+0000","last_active":"2024-02-19T21:13:43.374076+0000","last_peered":"2024-02-19T21:13:43.374076+0000","last_clean":"2024-02-19T21:13:43.374076+0000","last_became_active":"2024-02-19T21:10:41.304695+0000","last_became_peered":"2024-02-19T21:10:41.304695+0000","last_unstale":"2024-02-19T21:13:43.374076+0000","last_undegraded":"2024-02-19T21:13:43.374076+0000","last_fullsized":"2024-02-19T21:13:43.374076+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:59.312863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.312863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"459'30","reported_seq":362,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375219+0000","last_change":"2024-02-19T21:11:59.380681+0000","last_active":"2024-02-19T21:13:43.375219+0000","last_peered":"2024-02-19T21:13:43.375219+0000","last_clean":"2024-02-19T21:13:43.375219+0000","last_became_active":"2024-02-19T21:11:59.327859+0000","last_became_peered":"2024-02-19T21:11:59.327859+0000","last_unstale":"2024-02-19T21:13:43.375219+0000","last_undegraded":"2024-02-19T21:13:43.375219+0000","last_fullsized":"2024-02-19T21:13:43.375219+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:11:02.671837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.671837+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":610,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.383574+0000","last_change":"2024-02-19T21:12:09.369780+0000","last_active":"2024-02-19T21:13:43.383574+0000","last_peered":"2024-02-19T21:13:43.383574+0000","last_clean":"2024-02-19T21:13:43.383574+0000","last_became_active":"2024-02-19T21:12:09.369477+0000","last_became_peered":"2024-02-19T21:12:09.369477+0000","last_unstale":"2024-02-19T21:13:43.383574+0000","last_undegraded":"2024-02-19T21:13:43.383574+0000","last_fullsized":"2024-02-19T21:13:43.383574+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":168,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374624+0000","last_change":"2024-02-19T21:12:06.410909+0000","last_active":"2024-02-19T21:13:43.374624+0000","last_peered":"2024-02-19T21:13:43.374624+0000","last_clean":"2024-02-19T21:13:43.374624+0000","last_became_active":"2024-02-19T21:10:41.295301+0000","last_became_peered":"2024-02-19T21:10:41.295301+0000","last_unstale":"2024-02-19T21:13:43.374624+0000","last_undegraded":"2024-02-19T21:13:43.374624+0000","last_fullsized":"2024-02-19T21:13:43.374624+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.410847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.410847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":317,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.377282+0000","last_change":"2024-02-19T21:12:09.332689+0000","last_active":"2024-02-19T21:13:43.377282+0000","last_peered":"2024-02-19T21:13:43.377282+0000","last_clean":"2024-02-19T21:13:43.377282+0000","last_became_active":"2024-02-19T21:12:09.332349+0000","last_became_peered":"2024-02-19T21:12:09.332349+0000","last_unstale":"2024-02-19T21:13:43.377282+0000","last_undegraded":"2024-02-19T21:13:43.377282+0000","last_fullsized":"2024-02-19T21:13:43.377282+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:46.406402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:46.406402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":553,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381947+0000","last_change":"2024-02-19T21:12:09.340947+0000","last_active":"2024-02-19T21:13:43.381947+0000","last_peered":"2024-02-19T21:13:43.381947+0000","last_clean":"2024-02-19T21:13:43.381947+0000","last_became_active":"2024-02-19T21:12:09.340614+0000","last_became_peered":"2024-02-19T21:12:09.340614+0000","last_unstale":"2024-02-19T21:13:43.381947+0000","last_undegraded":"2024-02-19T21:13:43.381947+0000","last_fullsized":"2024-02-19T21:13:43.381947+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.669336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.669336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"459'29","reported_seq":470,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382627+0000","last_change":"2024-02-19T21:12:00.056470+0000","last_active":"2024-02-19T21:13:43.382627+0000","last_peered":"2024-02-19T21:13:43.382627+0000","last_clean":"2024-02-19T21:13:43.382627+0000","last_became_active":"2024-02-19T21:11:59.332542+0000","last_became_peered":"2024-02-19T21:11:59.332542+0000","last_unstale":"2024-02-19T21:13:43.382627+0000","last_undegraded":"2024-02-19T21:13:43.382627+0000","last_fullsized":"2024-02-19T21:13:43.382627+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:10:11.165973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:11.165973+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":625,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.384516+0000","last_change":"2024-02-19T21:13:38.813372+0000","last_active":"2024-02-19T21:13:43.384516+0000","last_peered":"2024-02-19T21:13:43.384516+0000","last_clean":"2024-02-19T21:13:43.384516+0000","last_became_active":"2024-02-19T21:11:59.335475+0000","last_became_peered":"2024-02-19T21:11:59.335475+0000","last_unstale":"2024-02-19T21:13:43.384516+0000","last_undegraded":"2024-02-19T21:13:43.384516+0000","last_fullsized":"2024-02-19T21:13:43.384516+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:38.813285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:38.813285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":555,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381667+0000","last_change":"2024-02-19T21:12:20.902387+0000","last_active":"2024-02-19T21:13:43.381667+0000","last_peered":"2024-02-19T21:13:43.381667+0000","last_clean":"2024-02-19T21:13:43.381667+0000","last_became_active":"2024-02-19T21:12:09.347197+0000","last_became_peered":"2024-02-19T21:12:09.347197+0000","last_unstale":"2024-02-19T21:13:43.381667+0000","last_undegraded":"2024-02-19T21:13:43.381667+0000","last_fullsized":"2024-02-19T21:13:43.381667+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:20.902297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:12:20.902297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":511,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.377519+0000","last_change":"2024-02-19T21:11:59.359210+0000","last_active":"2024-02-19T21:13:43.377519+0000","last_peered":"2024-02-19T21:13:43.377519+0000","last_clean":"2024-02-19T21:13:43.377519+0000","last_became_active":"2024-02-19T21:11:59.358933+0000","last_became_peered":"2024-02-19T21:11:59.358933+0000","last_unstale":"2024-02-19T21:13:43.377519+0000","last_undegraded":"2024-02-19T21:13:43.377519+0000","last_fullsized":"2024-02-19T21:13:43.377519+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.189509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":558,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.383332+0000","last_change":"2024-02-19T21:11:59.334472+0000","last_active":"2024-02-19T21:13:43.383332+0000","last_peered":"2024-02-19T21:13:43.383332+0000","last_clean":"2024-02-19T21:13:43.383332+0000","last_became_active":"2024-02-19T21:11:59.333491+0000","last_became_peered":"2024-02-19T21:11:59.333491+0000","last_unstale":"2024-02-19T21:13:43.383332+0000","last_undegraded":"2024-02-19T21:13:43.383332+0000","last_fullsized":"2024-02-19T21:13:43.383332+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:30.893485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:30.893485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"459'24","reported_seq":454,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.383690+0000","last_change":"2024-02-19T21:12:50.030922+0000","last_active":"2024-02-19T21:13:43.383690+0000","last_peered":"2024-02-19T21:13:43.383690+0000","last_clean":"2024-02-19T21:13:43.383690+0000","last_became_active":"2024-02-19T21:11:59.348806+0000","last_became_peered":"2024-02-19T21:11:59.348806+0000","last_unstale":"2024-02-19T21:13:43.383690+0000","last_undegraded":"2024-02-19T21:13:43.383690+0000","last_fullsized":"2024-02-19T21:13:43.383690+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'24","last_scrub_stamp":"2024-02-19T21:12:50.030857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:50.030857+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"397'2","reported_seq":622,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381795+0000","last_change":"2024-02-19T21:13:40.809868+0000","last_active":"2024-02-19T21:13:43.381795+0000","last_peered":"2024-02-19T21:13:43.381795+0000","last_clean":"2024-02-19T21:13:43.381795+0000","last_became_active":"2024-02-19T21:12:02.319577+0000","last_became_peered":"2024-02-19T21:12:02.319577+0000","last_unstale":"2024-02-19T21:13:43.381795+0000","last_undegraded":"2024-02-19T21:13:43.381795+0000","last_fullsized":"2024-02-19T21:13:43.381795+0000","mapping_epoch":485,"log_start":"58'1","ondisk_log_start":"58'1","created":57,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:13:40.809800+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:13:40.809800+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":177,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374234+0000","last_change":"2024-02-19T21:11:59.320654+0000","last_active":"2024-02-19T21:13:43.374234+0000","last_peered":"2024-02-19T21:13:43.374234+0000","last_clean":"2024-02-19T21:13:43.374234+0000","last_became_active":"2024-02-19T21:11:59.320327+0000","last_became_peered":"2024-02-19T21:11:59.320327+0000","last_unstale":"2024-02-19T21:13:43.374234+0000","last_undegraded":"2024-02-19T21:13:43.374234+0000","last_fullsized":"2024-02-19T21:13:43.374234+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.455993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.455993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":564,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382631+0000","last_change":"2024-02-19T21:10:51.691357+0000","last_active":"2024-02-19T21:13:43.382631+0000","last_peered":"2024-02-19T21:13:43.382631+0000","last_clean":"2024-02-19T21:13:43.382631+0000","last_became_active":"2024-02-19T21:08:04.282243+0000","last_became_peered":"2024-02-19T21:08:04.282243+0000","last_unstale":"2024-02-19T21:13:43.382631+0000","last_undegraded":"2024-02-19T21:13:43.382631+0000","last_fullsized":"2024-02-19T21:13:43.382631+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.691284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.691284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":502,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381797+0000","last_change":"2024-02-19T21:12:09.328499+0000","last_active":"2024-02-19T21:13:43.381797+0000","last_peered":"2024-02-19T21:13:43.381797+0000","last_clean":"2024-02-19T21:13:43.381797+0000","last_became_active":"2024-02-19T21:12:09.326936+0000","last_became_peered":"2024-02-19T21:12:09.326936+0000","last_unstale":"2024-02-19T21:13:43.381797+0000","last_undegraded":"2024-02-19T21:13:43.381797+0000","last_fullsized":"2024-02-19T21:13:43.381797+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:05.664425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:05.664425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":551,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.384101+0000","last_change":"2024-02-19T21:13:06.446599+0000","last_active":"2024-02-19T21:13:43.384101+0000","last_peered":"2024-02-19T21:13:43.384101+0000","last_clean":"2024-02-19T21:13:43.384101+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:13:43.384101+0000","last_undegraded":"2024-02-19T21:13:43.384101+0000","last_fullsized":"2024-02-19T21:13:43.384101+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:06.446523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.446523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"459'26","reported_seq":632,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375510+0000","last_change":"2024-02-19T21:12:09.420794+0000","last_active":"2024-02-19T21:13:43.375510+0000","last_peered":"2024-02-19T21:13:43.375510+0000","last_clean":"2024-02-19T21:13:43.375510+0000","last_became_active":"2024-02-19T21:12:09.352705+0000","last_became_peered":"2024-02-19T21:12:09.352705+0000","last_unstale":"2024-02-19T21:13:43.375510+0000","last_undegraded":"2024-02-19T21:13:43.375510+0000","last_fullsized":"2024-02-19T21:13:43.375510+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'26","last_scrub_stamp":"2024-02-19T21:11:44.774856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:44.774856+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":577,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381796+0000","last_change":"2024-02-19T21:12:09.311418+0000","last_active":"2024-02-19T21:13:43.381796+0000","last_peered":"2024-02-19T21:13:43.381796+0000","last_clean":"2024-02-19T21:13:43.381796+0000","last_became_active":"2024-02-19T21:12:09.310949+0000","last_became_peered":"2024-02-19T21:12:09.310949+0000","last_unstale":"2024-02-19T21:13:43.381796+0000","last_undegraded":"2024-02-19T21:13:43.381796+0000","last_fullsized":"2024-02-19T21:13:43.381796+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.747108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.747108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":590,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382519+0000","last_change":"2024-02-19T21:11:48.359614+0000","last_active":"2024-02-19T21:13:43.382519+0000","last_peered":"2024-02-19T21:13:43.382519+0000","last_clean":"2024-02-19T21:13:43.382519+0000","last_became_active":"2024-02-19T21:11:48.359271+0000","last_became_peered":"2024-02-19T21:11:48.359271+0000","last_unstale":"2024-02-19T21:13:43.382519+0000","last_undegraded":"2024-02-19T21:13:43.382519+0000","last_fullsized":"2024-02-19T21:13:43.382519+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.081993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.081993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":313,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376784+0000","last_change":"2024-02-19T21:12:55.248327+0000","last_active":"2024-02-19T21:13:43.376784+0000","last_peered":"2024-02-19T21:13:43.376784+0000","last_clean":"2024-02-19T21:13:43.376784+0000","last_became_active":"2024-02-19T21:12:09.347643+0000","last_became_peered":"2024-02-19T21:12:09.347643+0000","last_unstale":"2024-02-19T21:13:43.376784+0000","last_undegraded":"2024-02-19T21:13:43.376784+0000","last_fullsized":"2024-02-19T21:13:43.376784+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:55.248259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.248259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":583,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.383359+0000","last_change":"2024-02-19T21:10:41.315872+0000","last_active":"2024-02-19T21:13:43.383359+0000","last_peered":"2024-02-19T21:13:43.383359+0000","last_clean":"2024-02-19T21:13:43.383359+0000","last_became_active":"2024-02-19T21:10:41.315397+0000","last_became_peered":"2024-02-19T21:10:41.315397+0000","last_unstale":"2024-02-19T21:13:43.383359+0000","last_undegraded":"2024-02-19T21:13:43.383359+0000","last_fullsized":"2024-02-19T21:13:43.383359+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:10.148733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:10.148733+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":511,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381481+0000","last_change":"2024-02-19T21:11:59.333800+0000","last_active":"2024-02-19T21:13:43.381481+0000","last_peered":"2024-02-19T21:13:43.381481+0000","last_clean":"2024-02-19T21:13:43.381481+0000","last_became_active":"2024-02-19T21:11:59.333572+0000","last_became_peered":"2024-02-19T21:11:59.333572+0000","last_unstale":"2024-02-19T21:13:43.381481+0000","last_undegraded":"2024-02-19T21:13:43.381481+0000","last_fullsized":"2024-02-19T21:13:43.381481+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_clean_scrub_stamp":"2024-02-19T21:10:20.204880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":555,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376007+0000","last_change":"2024-02-19T21:13:00.111238+0000","last_active":"2024-02-19T21:13:43.376007+0000","last_peered":"2024-02-19T21:13:43.376007+0000","last_clean":"2024-02-19T21:13:43.376007+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:13:43.376007+0000","last_undegraded":"2024-02-19T21:13:43.376007+0000","last_fullsized":"2024-02-19T21:13:43.376007+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.111171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.111171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"459'22","reported_seq":595,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.377445+0000","last_change":"2024-02-19T21:11:59.988716+0000","last_active":"2024-02-19T21:13:43.377445+0000","last_peered":"2024-02-19T21:13:43.377445+0000","last_clean":"2024-02-19T21:13:43.377445+0000","last_became_active":"2024-02-19T21:11:59.343010+0000","last_became_peered":"2024-02-19T21:11:59.343010+0000","last_unstale":"2024-02-19T21:13:43.377445+0000","last_undegraded":"2024-02-19T21:13:43.377445+0000","last_fullsized":"2024-02-19T21:13:43.377445+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:11:00.191206+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.191206+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":618,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.378005+0000","last_change":"2024-02-19T21:13:41.604243+0000","last_active":"2024-02-19T21:13:43.378005+0000","last_peered":"2024-02-19T21:13:43.378005+0000","last_clean":"2024-02-19T21:13:43.378005+0000","last_became_active":"2024-02-19T21:11:59.352283+0000","last_became_peered":"2024-02-19T21:11:59.352283+0000","last_unstale":"2024-02-19T21:13:43.378005+0000","last_undegraded":"2024-02-19T21:13:43.378005+0000","last_fullsized":"2024-02-19T21:13:43.378005+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:41.604181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:41.604181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":1313,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-02-19T21:13:44.457022+0000","last_change":"2024-02-19T21:13:08.986433+0000","last_active":"2024-02-19T21:13:44.457022+0000","last_peered":"2024-02-19T21:13:44.457022+0000","last_clean":"2024-02-19T21:13:44.457022+0000","last_became_active":"2024-02-19T21:12:09.340570+0000","last_became_peered":"2024-02-19T21:12:09.340570+0000","last_unstale":"2024-02-19T21:13:44.457022+0000","last_undegraded":"2024-02-19T21:13:44.457022+0000","last_fullsized":"2024-02-19T21:13:44.457022+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:13:08.986361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:08.986361+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":739,"num_read_kb":734,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":576,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.384328+0000","last_change":"2024-02-19T21:13:04.428341+0000","last_active":"2024-02-19T21:13:43.384328+0000","last_peered":"2024-02-19T21:13:43.384328+0000","last_clean":"2024-02-19T21:13:43.384328+0000","last_became_active":"2024-02-19T21:12:09.323625+0000","last_became_peered":"2024-02-19T21:12:09.323625+0000","last_unstale":"2024-02-19T21:13:43.384328+0000","last_undegraded":"2024-02-19T21:13:43.384328+0000","last_fullsized":"2024-02-19T21:13:43.384328+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:04.428236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:04.428236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":509,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382301+0000","last_change":"2024-02-19T21:11:59.352488+0000","last_active":"2024-02-19T21:13:43.382301+0000","last_peered":"2024-02-19T21:13:43.382301+0000","last_clean":"2024-02-19T21:13:43.382301+0000","last_became_active":"2024-02-19T21:11:59.351887+0000","last_became_peered":"2024-02-19T21:11:59.351887+0000","last_unstale":"2024-02-19T21:13:43.382301+0000","last_undegraded":"2024-02-19T21:13:43.382301+0000","last_fullsized":"2024-02-19T21:13:43.382301+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.947089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":419,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382311+0000","last_change":"2024-02-19T21:12:09.369780+0000","last_active":"2024-02-19T21:13:43.382311+0000","last_peered":"2024-02-19T21:13:43.382311+0000","last_clean":"2024-02-19T21:13:43.382311+0000","last_became_active":"2024-02-19T21:12:09.369431+0000","last_became_peered":"2024-02-19T21:12:09.369431+0000","last_unstale":"2024-02-19T21:13:43.382311+0000","last_undegraded":"2024-02-19T21:13:43.382311+0000","last_fullsized":"2024-02-19T21:13:43.382311+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.945639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.945639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"459'42","reported_seq":828,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-02-19T21:13:44.456305+0000","last_change":"2024-02-19T21:10:01.545972+0000","last_active":"2024-02-19T21:13:44.456305+0000","last_peered":"2024-02-19T21:13:44.456305+0000","last_clean":"2024-02-19T21:13:44.456305+0000","last_became_active":"2024-02-19T21:08:04.266322+0000","last_became_peered":"2024-02-19T21:08:04.266322+0000","last_unstale":"2024-02-19T21:13:44.456305+0000","last_undegraded":"2024-02-19T21:13:44.456305+0000","last_fullsized":"2024-02-19T21:13:44.456305+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'30","last_scrub_stamp":"2024-02-19T21:10:01.545897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:01.545897+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":599,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381873+0000","last_change":"2024-02-19T21:12:09.328219+0000","last_active":"2024-02-19T21:13:43.381873+0000","last_peered":"2024-02-19T21:13:43.381873+0000","last_clean":"2024-02-19T21:13:43.381873+0000","last_became_active":"2024-02-19T21:12:09.326579+0000","last_became_peered":"2024-02-19T21:12:09.326579+0000","last_unstale":"2024-02-19T21:13:43.381873+0000","last_undegraded":"2024-02-19T21:13:43.381873+0000","last_fullsized":"2024-02-19T21:13:43.381873+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.089715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.089715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":317,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376050+0000","last_change":"2024-02-19T21:11:31.629134+0000","last_active":"2024-02-19T21:13:43.376050+0000","last_peered":"2024-02-19T21:13:43.376050+0000","last_clean":"2024-02-19T21:13:43.376050+0000","last_became_active":"2024-02-19T21:10:41.305829+0000","last_became_peered":"2024-02-19T21:10:41.305829+0000","last_unstale":"2024-02-19T21:13:43.376050+0000","last_undegraded":"2024-02-19T21:13:43.376050+0000","last_fullsized":"2024-02-19T21:13:43.376050+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:31.629057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.629057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":511,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374587+0000","last_change":"2024-02-19T21:11:59.333878+0000","last_active":"2024-02-19T21:13:43.374587+0000","last_peered":"2024-02-19T21:13:43.374587+0000","last_clean":"2024-02-19T21:13:43.374587+0000","last_became_active":"2024-02-19T21:11:59.331967+0000","last_became_peered":"2024-02-19T21:11:59.331967+0000","last_unstale":"2024-02-19T21:13:43.374587+0000","last_undegraded":"2024-02-19T21:13:43.374587+0000","last_fullsized":"2024-02-19T21:13:43.374587+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:54.142898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.142898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"493'12","reported_seq":474,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-02-19T21:13:44.459323+0000","last_change":"2024-02-19T21:12:09.433235+0000","last_active":"2024-02-19T21:13:44.459323+0000","last_peered":"2024-02-19T21:13:44.459323+0000","last_clean":"2024-02-19T21:13:44.459323+0000","last_became_active":"2024-02-19T21:12:09.326372+0000","last_became_peered":"2024-02-19T21:12:09.326372+0000","last_unstale":"2024-02-19T21:13:44.459323+0000","last_undegraded":"2024-02-19T21:13:44.459323+0000","last_fullsized":"2024-02-19T21:13:44.459323+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'10","last_scrub_stamp":"2024-02-19T21:11:41.775418+0000","last_deep_scrub":"471'10","last_deep_scrub_stamp":"2024-02-19T21:11:41.775418+0000","last_clean_scrub_stamp":"2024-02-19T21:11:41.775418+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"459'20","reported_seq":622,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381926+0000","last_change":"2024-02-19T21:12:09.030294+0000","last_active":"2024-02-19T21:13:43.381926+0000","last_peered":"2024-02-19T21:13:43.381926+0000","last_clean":"2024-02-19T21:13:43.381926+0000","last_became_active":"2024-02-19T21:10:41.296275+0000","last_became_peered":"2024-02-19T21:10:41.296275+0000","last_unstale":"2024-02-19T21:13:43.381926+0000","last_undegraded":"2024-02-19T21:13:43.381926+0000","last_fullsized":"2024-02-19T21:13:43.381926+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:12:09.030215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:09.030215+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":615,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382000+0000","last_change":"2024-02-19T21:11:59.351637+0000","last_active":"2024-02-19T21:13:43.382000+0000","last_peered":"2024-02-19T21:13:43.382000+0000","last_clean":"2024-02-19T21:13:43.382000+0000","last_became_active":"2024-02-19T21:11:59.351241+0000","last_became_peered":"2024-02-19T21:11:59.351241+0000","last_unstale":"2024-02-19T21:13:43.382000+0000","last_undegraded":"2024-02-19T21:13:43.382000+0000","last_fullsized":"2024-02-19T21:13:43.382000+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:21.732778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:10:21.732778+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":313,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375124+0000","last_change":"2024-02-19T21:13:07.221665+0000","last_active":"2024-02-19T21:13:43.375124+0000","last_peered":"2024-02-19T21:13:43.375124+0000","last_clean":"2024-02-19T21:13:43.375124+0000","last_became_active":"2024-02-19T21:11:59.335798+0000","last_became_peered":"2024-02-19T21:11:59.335798+0000","last_unstale":"2024-02-19T21:13:43.375124+0000","last_undegraded":"2024-02-19T21:13:43.375124+0000","last_fullsized":"2024-02-19T21:13:43.375124+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:07.221594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.221594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":308,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376125+0000","last_change":"2024-02-19T21:12:09.332210+0000","last_active":"2024-02-19T21:13:43.376125+0000","last_peered":"2024-02-19T21:13:43.376125+0000","last_clean":"2024-02-19T21:13:43.376125+0000","last_became_active":"2024-02-19T21:12:09.331195+0000","last_became_peered":"2024-02-19T21:12:09.331195+0000","last_unstale":"2024-02-19T21:13:43.376125+0000","last_undegraded":"2024-02-19T21:13:43.376125+0000","last_fullsized":"2024-02-19T21:13:43.376125+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.604025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.604025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":564,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374399+0000","last_change":"2024-02-19T21:13:09.184029+0000","last_active":"2024-02-19T21:13:43.374399+0000","last_peered":"2024-02-19T21:13:43.374399+0000","last_clean":"2024-02-19T21:13:43.374399+0000","last_became_active":"2024-02-19T21:11:59.333275+0000","last_became_peered":"2024-02-19T21:11:59.333275+0000","last_unstale":"2024-02-19T21:13:43.374399+0000","last_undegraded":"2024-02-19T21:13:43.374399+0000","last_fullsized":"2024-02-19T21:13:43.374399+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.183962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.183962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"459'23","reported_seq":189,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374111+0000","last_change":"2024-02-19T21:12:18.340005+0000","last_active":"2024-02-19T21:13:43.374111+0000","last_peered":"2024-02-19T21:13:43.374111+0000","last_clean":"2024-02-19T21:13:43.374111+0000","last_became_active":"2024-02-19T21:10:41.287134+0000","last_became_peered":"2024-02-19T21:10:41.287134+0000","last_unstale":"2024-02-19T21:13:43.374111+0000","last_undegraded":"2024-02-19T21:13:43.374111+0000","last_fullsized":"2024-02-19T21:13:43.374111+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'23","last_scrub_stamp":"2024-02-19T21:12:18.339929+0000","last_deep_scrub":"420'17","last_deep_scrub_stamp":"2024-02-19T21:10:19.722303+0000","last_clean_scrub_stamp":"2024-02-19T21:12:18.339929+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":189,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375976+0000","last_change":"2024-02-19T21:11:59.342221+0000","last_active":"2024-02-19T21:13:43.375976+0000","last_peered":"2024-02-19T21:13:43.375976+0000","last_clean":"2024-02-19T21:13:43.375976+0000","last_became_active":"2024-02-19T21:11:59.323514+0000","last_became_peered":"2024-02-19T21:11:59.323514+0000","last_unstale":"2024-02-19T21:13:43.375976+0000","last_undegraded":"2024-02-19T21:13:43.375976+0000","last_fullsized":"2024-02-19T21:13:43.375976+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:10:58.333168+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.333168+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":304,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376143+0000","last_change":"2024-02-19T21:13:00.166548+0000","last_active":"2024-02-19T21:13:43.376143+0000","last_peered":"2024-02-19T21:13:43.376143+0000","last_clean":"2024-02-19T21:13:43.376143+0000","last_became_active":"2024-02-19T21:08:04.264593+0000","last_became_peered":"2024-02-19T21:08:04.264593+0000","last_unstale":"2024-02-19T21:13:43.376143+0000","last_undegraded":"2024-02-19T21:13:43.376143+0000","last_fullsized":"2024-02-19T21:13:43.376143+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.166479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.166479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":507,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-02-19T21:13:44.567327+0000","last_change":"2024-02-19T21:13:44.567327+0000","last_active":"2024-02-19T21:13:44.567327+0000","last_peered":"2024-02-19T21:13:44.567327+0000","last_clean":"2024-02-19T21:13:44.567327+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:13:44.567327+0000","last_undegraded":"2024-02-19T21:13:44.567327+0000","last_fullsized":"2024-02-19T21:13:44.567327+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:44.567199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:13:44.567199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"459'15","reported_seq":339,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375975+0000","last_change":"2024-02-19T21:11:59.364955+0000","last_active":"2024-02-19T21:13:43.375975+0000","last_peered":"2024-02-19T21:13:43.375975+0000","last_clean":"2024-02-19T21:13:43.375975+0000","last_became_active":"2024-02-19T21:11:59.336293+0000","last_became_peered":"2024-02-19T21:11:59.336293+0000","last_unstale":"2024-02-19T21:13:43.375975+0000","last_undegraded":"2024-02-19T21:13:43.375975+0000","last_fullsized":"2024-02-19T21:13:43.375975+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:10:22.459807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:22.459807+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":563,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376675+0000","last_change":"2024-02-19T21:11:59.334071+0000","last_active":"2024-02-19T21:13:43.376675+0000","last_peered":"2024-02-19T21:13:43.376675+0000","last_clean":"2024-02-19T21:13:43.376675+0000","last_became_active":"2024-02-19T21:11:59.333684+0000","last_became_peered":"2024-02-19T21:11:59.333684+0000","last_unstale":"2024-02-19T21:13:43.376675+0000","last_undegraded":"2024-02-19T21:13:43.376675+0000","last_fullsized":"2024-02-19T21:13:43.376675+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.244285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.244285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":299,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.378771+0000","last_change":"2024-02-19T21:10:53.703797+0000","last_active":"2024-02-19T21:13:43.378771+0000","last_peered":"2024-02-19T21:13:43.378771+0000","last_clean":"2024-02-19T21:13:43.378771+0000","last_became_active":"2024-02-19T21:08:04.264972+0000","last_became_peered":"2024-02-19T21:08:04.264972+0000","last_unstale":"2024-02-19T21:13:43.378771+0000","last_undegraded":"2024-02-19T21:13:43.378771+0000","last_fullsized":"2024-02-19T21:13:43.378771+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:53.703727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.703727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":571,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.378299+0000","last_change":"2024-02-19T21:12:06.261731+0000","last_active":"2024-02-19T21:13:43.378299+0000","last_peered":"2024-02-19T21:13:43.378299+0000","last_clean":"2024-02-19T21:13:43.378299+0000","last_became_active":"2024-02-19T21:10:41.314114+0000","last_became_peered":"2024-02-19T21:10:41.314114+0000","last_unstale":"2024-02-19T21:13:43.378299+0000","last_undegraded":"2024-02-19T21:13:43.378299+0000","last_fullsized":"2024-02-19T21:13:43.378299+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.261670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.261670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":168,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374848+0000","last_change":"2024-02-19T21:12:08.430936+0000","last_active":"2024-02-19T21:13:43.374848+0000","last_peered":"2024-02-19T21:13:43.374848+0000","last_clean":"2024-02-19T21:13:43.374848+0000","last_became_active":"2024-02-19T21:10:41.300574+0000","last_became_peered":"2024-02-19T21:10:41.300574+0000","last_unstale":"2024-02-19T21:13:43.374848+0000","last_undegraded":"2024-02-19T21:13:43.374848+0000","last_fullsized":"2024-02-19T21:13:43.374848+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:08.430874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:08.430874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"459'15","reported_seq":190,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.552136+0000","last_change":"2024-02-19T21:13:43.552100+0000","last_active":"2024-02-19T21:13:43.552136+0000","last_peered":"2024-02-19T21:13:43.552136+0000","last_clean":"2024-02-19T21:13:43.552136+0000","last_became_active":"2024-02-19T21:10:40.297867+0000","last_became_peered":"2024-02-19T21:10:40.297867+0000","last_unstale":"2024-02-19T21:13:43.552136+0000","last_undegraded":"2024-02-19T21:13:43.552136+0000","last_fullsized":"2024-02-19T21:13:43.552136+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:13:43.552026+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:13:43.552026+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":514,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374009+0000","last_change":"2024-02-19T21:12:09.369726+0000","last_active":"2024-02-19T21:13:43.374009+0000","last_peered":"2024-02-19T21:13:43.374009+0000","last_clean":"2024-02-19T21:13:43.374009+0000","last_became_active":"2024-02-19T21:12:09.369407+0000","last_became_peered":"2024-02-19T21:12:09.369407+0000","last_unstale":"2024-02-19T21:13:43.374009+0000","last_undegraded":"2024-02-19T21:13:43.374009+0000","last_fullsized":"2024-02-19T21:13:43.374009+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:31.670026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.670026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":570,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382678+0000","last_change":"2024-02-19T21:12:06.098778+0000","last_active":"2024-02-19T21:13:43.382678+0000","last_peered":"2024-02-19T21:13:43.382678+0000","last_clean":"2024-02-19T21:13:43.382678+0000","last_became_active":"2024-02-19T21:08:04.265573+0000","last_became_peered":"2024-02-19T21:08:04.265573+0000","last_unstale":"2024-02-19T21:13:43.382678+0000","last_undegraded":"2024-02-19T21:13:43.382678+0000","last_fullsized":"2024-02-19T21:13:43.382678+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.098703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.098703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":578,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.384101+0000","last_change":"2024-02-19T21:12:17.452030+0000","last_active":"2024-02-19T21:13:43.384101+0000","last_peered":"2024-02-19T21:13:43.384101+0000","last_clean":"2024-02-19T21:13:43.384101+0000","last_became_active":"2024-02-19T21:12:09.341406+0000","last_became_peered":"2024-02-19T21:12:09.341406+0000","last_unstale":"2024-02-19T21:13:43.384101+0000","last_undegraded":"2024-02-19T21:13:43.384101+0000","last_fullsized":"2024-02-19T21:13:43.384101+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:17.451964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:17.451964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":529,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382221+0000","last_change":"2024-02-19T21:12:47.428132+0000","last_active":"2024-02-19T21:13:43.382221+0000","last_peered":"2024-02-19T21:13:43.382221+0000","last_clean":"2024-02-19T21:13:43.382221+0000","last_became_active":"2024-02-19T21:12:09.349695+0000","last_became_peered":"2024-02-19T21:12:09.349695+0000","last_unstale":"2024-02-19T21:13:43.382221+0000","last_undegraded":"2024-02-19T21:13:43.382221+0000","last_fullsized":"2024-02-19T21:13:43.382221+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:47.428055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:47.428055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"459'20","reported_seq":326,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.377100+0000","last_change":"2024-02-19T21:10:59.685223+0000","last_active":"2024-02-19T21:13:43.377100+0000","last_peered":"2024-02-19T21:13:43.377100+0000","last_clean":"2024-02-19T21:13:43.377100+0000","last_became_active":"2024-02-19T21:08:04.267216+0000","last_became_peered":"2024-02-19T21:08:04.267216+0000","last_unstale":"2024-02-19T21:13:43.377100+0000","last_undegraded":"2024-02-19T21:13:43.377100+0000","last_fullsized":"2024-02-19T21:13:43.377100+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:10:59.685150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.685150+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":577,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382122+0000","last_change":"2024-02-19T21:13:36.771987+0000","last_active":"2024-02-19T21:13:43.382122+0000","last_peered":"2024-02-19T21:13:43.382122+0000","last_clean":"2024-02-19T21:13:43.382122+0000","last_became_active":"2024-02-19T21:08:04.266728+0000","last_became_peered":"2024-02-19T21:08:04.266728+0000","last_unstale":"2024-02-19T21:13:43.382122+0000","last_undegraded":"2024-02-19T21:13:43.382122+0000","last_fullsized":"2024-02-19T21:13:43.382122+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:36.771900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_clean_scrub_stamp":"2024-02-19T21:13:36.771900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"433'9","reported_seq":721,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-02-19T21:13:44.459652+0000","last_change":"2024-02-19T21:13:00.428710+0000","last_active":"2024-02-19T21:13:44.459652+0000","last_peered":"2024-02-19T21:13:44.459652+0000","last_clean":"2024-02-19T21:13:44.459652+0000","last_became_active":"2024-02-19T21:10:40.310303+0000","last_became_peered":"2024-02-19T21:10:40.310303+0000","last_unstale":"2024-02-19T21:13:44.459652+0000","last_undegraded":"2024-02-19T21:13:44.459652+0000","last_fullsized":"2024-02-19T21:13:44.459652+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'9","last_scrub_stamp":"2024-02-19T21:13:00.428581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.428581+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":430,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.384489+0000","last_change":"2024-02-19T21:11:59.843545+0000","last_active":"2024-02-19T21:13:43.384489+0000","last_peered":"2024-02-19T21:13:43.384489+0000","last_clean":"2024-02-19T21:13:43.384489+0000","last_became_active":"2024-02-19T21:11:59.323539+0000","last_became_peered":"2024-02-19T21:11:59.323539+0000","last_unstale":"2024-02-19T21:13:43.384489+0000","last_undegraded":"2024-02-19T21:13:43.384489+0000","last_fullsized":"2024-02-19T21:13:43.384489+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:11:31.500511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.500511+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":383,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374318+0000","last_change":"2024-02-19T21:12:09.345187+0000","last_active":"2024-02-19T21:13:43.374318+0000","last_peered":"2024-02-19T21:13:43.374318+0000","last_clean":"2024-02-19T21:13:43.374318+0000","last_became_active":"2024-02-19T21:12:09.344609+0000","last_became_peered":"2024-02-19T21:12:09.344609+0000","last_unstale":"2024-02-19T21:13:43.374318+0000","last_undegraded":"2024-02-19T21:13:43.374318+0000","last_fullsized":"2024-02-19T21:13:43.374318+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.378925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:55.202463+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.378925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"459'15","reported_seq":615,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382689+0000","last_change":"2024-02-19T21:10:50.053312+0000","last_active":"2024-02-19T21:13:43.382689+0000","last_peered":"2024-02-19T21:13:43.382689+0000","last_clean":"2024-02-19T21:13:43.382689+0000","last_became_active":"2024-02-19T21:08:04.266103+0000","last_became_peered":"2024-02-19T21:08:04.266103+0000","last_unstale":"2024-02-19T21:13:43.382689+0000","last_undegraded":"2024-02-19T21:13:43.382689+0000","last_fullsized":"2024-02-19T21:13:43.382689+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:10:50.053229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.053229+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":182,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375078+0000","last_change":"2024-02-19T21:13:07.507018+0000","last_active":"2024-02-19T21:13:43.375078+0000","last_peered":"2024-02-19T21:13:43.375078+0000","last_clean":"2024-02-19T21:13:43.375078+0000","last_became_active":"2024-02-19T21:11:59.324861+0000","last_became_peered":"2024-02-19T21:11:59.324861+0000","last_unstale":"2024-02-19T21:13:43.375078+0000","last_undegraded":"2024-02-19T21:13:43.375078+0000","last_fullsized":"2024-02-19T21:13:43.375078+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:07.506946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.506946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":377,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-02-19T21:13:44.343989+0000","last_change":"2024-02-19T21:12:17.345023+0000","last_active":"2024-02-19T21:13:44.343989+0000","last_peered":"2024-02-19T21:13:44.343989+0000","last_clean":"2024-02-19T21:13:44.343989+0000","last_became_active":"2024-02-19T21:08:04.256045+0000","last_became_peered":"2024-02-19T21:08:04.256045+0000","last_unstale":"2024-02-19T21:13:44.343989+0000","last_undegraded":"2024-02-19T21:13:44.343989+0000","last_fullsized":"2024-02-19T21:13:44.343989+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'10","last_scrub_stamp":"2024-02-19T21:12:17.344916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:17.344916+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":168,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376004+0000","last_change":"2024-02-19T21:12:05.445686+0000","last_active":"2024-02-19T21:13:43.376004+0000","last_peered":"2024-02-19T21:13:43.376004+0000","last_clean":"2024-02-19T21:13:43.376004+0000","last_became_active":"2024-02-19T21:10:41.310279+0000","last_became_peered":"2024-02-19T21:10:41.310279+0000","last_unstale":"2024-02-19T21:13:43.376004+0000","last_undegraded":"2024-02-19T21:13:43.376004+0000","last_fullsized":"2024-02-19T21:13:43.376004+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:05.445575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_clean_scrub_stamp":"2024-02-19T21:12:05.445575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":559,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.377886+0000","last_change":"2024-02-19T21:12:49.142231+0000","last_active":"2024-02-19T21:13:43.377886+0000","last_peered":"2024-02-19T21:13:43.377886+0000","last_clean":"2024-02-19T21:13:43.377886+0000","last_became_active":"2024-02-19T21:12:09.339844+0000","last_became_peered":"2024-02-19T21:12:09.339844+0000","last_unstale":"2024-02-19T21:13:43.377886+0000","last_undegraded":"2024-02-19T21:13:43.377886+0000","last_fullsized":"2024-02-19T21:13:43.377886+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-02-19T21:12:49.142158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:49.142158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":15,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"459'15","reported_seq":315,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381604+0000","last_change":"2024-02-19T21:13:06.958616+0000","last_active":"2024-02-19T21:13:43.381604+0000","last_peered":"2024-02-19T21:13:43.381604+0000","last_clean":"2024-02-19T21:13:43.381604+0000","last_became_active":"2024-02-19T21:12:09.341038+0000","last_became_peered":"2024-02-19T21:12:09.341038+0000","last_unstale":"2024-02-19T21:13:43.381604+0000","last_undegraded":"2024-02-19T21:13:43.381604+0000","last_fullsized":"2024-02-19T21:13:43.381604+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:13:06.958518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.958518+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":383,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376564+0000","last_change":"2024-02-19T21:12:09.333684+0000","last_active":"2024-02-19T21:13:43.376564+0000","last_peered":"2024-02-19T21:13:43.376564+0000","last_clean":"2024-02-19T21:13:43.376564+0000","last_became_active":"2024-02-19T21:12:09.333350+0000","last_became_peered":"2024-02-19T21:12:09.333350+0000","last_unstale":"2024-02-19T21:13:43.376564+0000","last_undegraded":"2024-02-19T21:13:43.376564+0000","last_fullsized":"2024-02-19T21:13:43.376564+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.422323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.422323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":313,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376314+0000","last_change":"2024-02-19T21:11:59.349259+0000","last_active":"2024-02-19T21:13:43.376314+0000","last_peered":"2024-02-19T21:13:43.376314+0000","last_clean":"2024-02-19T21:13:43.376314+0000","last_became_active":"2024-02-19T21:11:59.348485+0000","last_became_peered":"2024-02-19T21:11:59.348485+0000","last_unstale":"2024-02-19T21:13:43.376314+0000","last_undegraded":"2024-02-19T21:13:43.376314+0000","last_fullsized":"2024-02-19T21:13:43.376314+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:45.404242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_clean_scrub_stamp":"2024-02-19T21:11:45.404242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":398,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381738+0000","last_change":"2024-02-19T21:12:09.327887+0000","last_active":"2024-02-19T21:13:43.381738+0000","last_peered":"2024-02-19T21:13:43.381738+0000","last_clean":"2024-02-19T21:13:43.381738+0000","last_became_active":"2024-02-19T21:12:09.327390+0000","last_became_peered":"2024-02-19T21:12:09.327390+0000","last_unstale":"2024-02-19T21:13:43.381738+0000","last_undegraded":"2024-02-19T21:13:43.381738+0000","last_fullsized":"2024-02-19T21:13:43.381738+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:09:38.940935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:09:38.940935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":502,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382621+0000","last_change":"2024-02-19T21:12:09.330102+0000","last_active":"2024-02-19T21:13:43.382621+0000","last_peered":"2024-02-19T21:13:43.382621+0000","last_clean":"2024-02-19T21:13:43.382621+0000","last_became_active":"2024-02-19T21:12:09.329246+0000","last_became_peered":"2024-02-19T21:12:09.329246+0000","last_unstale":"2024-02-19T21:13:43.382621+0000","last_undegraded":"2024-02-19T21:13:43.382621+0000","last_fullsized":"2024-02-19T21:13:43.382621+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.493206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.493206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"459'19","reported_seq":213,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375250+0000","last_change":"2024-02-19T21:12:10.570353+0000","last_active":"2024-02-19T21:13:43.375250+0000","last_peered":"2024-02-19T21:13:43.375250+0000","last_clean":"2024-02-19T21:13:43.375250+0000","last_became_active":"2024-02-19T21:12:09.399929+0000","last_became_peered":"2024-02-19T21:12:09.399929+0000","last_unstale":"2024-02-19T21:13:43.375250+0000","last_undegraded":"2024-02-19T21:13:43.375250+0000","last_fullsized":"2024-02-19T21:13:43.375250+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:11:32.351880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:32.351880+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":575,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382394+0000","last_change":"2024-02-19T21:13:00.874389+0000","last_active":"2024-02-19T21:13:43.382394+0000","last_peered":"2024-02-19T21:13:43.382394+0000","last_clean":"2024-02-19T21:13:43.382394+0000","last_became_active":"2024-02-19T21:12:09.342120+0000","last_became_peered":"2024-02-19T21:12:09.342120+0000","last_unstale":"2024-02-19T21:13:43.382394+0000","last_undegraded":"2024-02-19T21:13:43.382394+0000","last_fullsized":"2024-02-19T21:13:43.382394+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.874318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.874318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":571,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381916+0000","last_change":"2024-02-19T21:12:54.957657+0000","last_active":"2024-02-19T21:13:43.381916+0000","last_peered":"2024-02-19T21:13:43.381916+0000","last_clean":"2024-02-19T21:13:43.381916+0000","last_became_active":"2024-02-19T21:08:04.268847+0000","last_became_peered":"2024-02-19T21:08:04.268847+0000","last_unstale":"2024-02-19T21:13:43.381916+0000","last_undegraded":"2024-02-19T21:13:43.381916+0000","last_fullsized":"2024-02-19T21:13:43.381916+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:54.957585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:54.957585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":308,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375713+0000","last_change":"2024-02-19T21:11:59.341209+0000","last_active":"2024-02-19T21:13:43.375713+0000","last_peered":"2024-02-19T21:13:43.375713+0000","last_clean":"2024-02-19T21:13:43.375713+0000","last_became_active":"2024-02-19T21:11:59.340742+0000","last_became_peered":"2024-02-19T21:11:59.340742+0000","last_unstale":"2024-02-19T21:13:43.375713+0000","last_undegraded":"2024-02-19T21:13:43.375713+0000","last_fullsized":"2024-02-19T21:13:43.375713+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:37.477995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:37.477995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":308,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.378831+0000","last_change":"2024-02-19T21:11:59.338599+0000","last_active":"2024-02-19T21:13:43.378831+0000","last_peered":"2024-02-19T21:13:43.378831+0000","last_clean":"2024-02-19T21:13:43.378831+0000","last_became_active":"2024-02-19T21:11:59.337764+0000","last_became_peered":"2024-02-19T21:11:59.337764+0000","last_unstale":"2024-02-19T21:13:43.378831+0000","last_undegraded":"2024-02-19T21:13:43.378831+0000","last_fullsized":"2024-02-19T21:13:43.378831+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.599474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.599474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"555'10","reported_seq":196,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381556+0000","last_change":"2024-02-19T21:13:12.409114+0000","last_active":"2024-02-19T21:13:43.381556+0000","last_peered":"2024-02-19T21:13:43.381556+0000","last_clean":"2024-02-19T21:13:43.381556+0000","last_became_active":"2024-02-19T21:12:09.340298+0000","last_became_peered":"2024-02-19T21:12:09.340298+0000","last_unstale":"2024-02-19T21:13:43.381556+0000","last_undegraded":"2024-02-19T21:13:43.381556+0000","last_fullsized":"2024-02-19T21:13:43.381556+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'10","last_scrub_stamp":"2024-02-19T21:13:12.409036+0000","last_deep_scrub":"555'10","last_deep_scrub_stamp":"2024-02-19T21:13:12.409036+0000","last_clean_scrub_stamp":"2024-02-19T21:13:12.409036+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":568,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.779632+0000","last_change":"2024-02-19T21:13:43.779607+0000","last_active":"2024-02-19T21:13:43.779632+0000","last_peered":"2024-02-19T21:13:43.779632+0000","last_clean":"2024-02-19T21:13:43.779632+0000","last_became_active":"2024-02-19T21:12:09.329752+0000","last_became_peered":"2024-02-19T21:12:09.329752+0000","last_unstale":"2024-02-19T21:13:43.779632+0000","last_undegraded":"2024-02-19T21:13:43.779632+0000","last_fullsized":"2024-02-19T21:13:43.779632+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:43.779536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:43.779536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":515,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.373984+0000","last_change":"2024-02-19T21:13:41.706582+0000","last_active":"2024-02-19T21:13:43.373984+0000","last_peered":"2024-02-19T21:13:43.373984+0000","last_clean":"2024-02-19T21:13:43.373984+0000","last_became_active":"2024-02-19T21:12:09.345654+0000","last_became_peered":"2024-02-19T21:12:09.345654+0000","last_unstale":"2024-02-19T21:13:43.373984+0000","last_undegraded":"2024-02-19T21:13:43.373984+0000","last_fullsized":"2024-02-19T21:13:43.373984+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:41.706497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:41.706497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":567,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.383143+0000","last_change":"2024-02-19T21:12:05.530113+0000","last_active":"2024-02-19T21:13:43.383143+0000","last_peered":"2024-02-19T21:13:43.383143+0000","last_clean":"2024-02-19T21:13:43.383143+0000","last_became_active":"2024-02-19T21:10:41.314589+0000","last_became_peered":"2024-02-19T21:10:41.314589+0000","last_unstale":"2024-02-19T21:13:43.383143+0000","last_undegraded":"2024-02-19T21:13:43.383143+0000","last_fullsized":"2024-02-19T21:13:43.383143+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:05.530039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:05.530039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":304,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.377177+0000","last_change":"2024-02-19T21:12:07.259353+0000","last_active":"2024-02-19T21:13:43.377177+0000","last_peered":"2024-02-19T21:13:43.377177+0000","last_clean":"2024-02-19T21:13:43.377177+0000","last_became_active":"2024-02-19T21:08:04.266161+0000","last_became_peered":"2024-02-19T21:08:04.266161+0000","last_unstale":"2024-02-19T21:13:43.377177+0000","last_undegraded":"2024-02-19T21:13:43.377177+0000","last_fullsized":"2024-02-19T21:13:43.377177+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:07.259290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:07.259290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"459'19","reported_seq":349,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375941+0000","last_change":"2024-02-19T21:12:09.375334+0000","last_active":"2024-02-19T21:13:43.375941+0000","last_peered":"2024-02-19T21:13:43.375941+0000","last_clean":"2024-02-19T21:13:43.375941+0000","last_became_active":"2024-02-19T21:12:09.321408+0000","last_became_peered":"2024-02-19T21:12:09.321408+0000","last_unstale":"2024-02-19T21:13:43.375941+0000","last_undegraded":"2024-02-19T21:13:43.375941+0000","last_fullsized":"2024-02-19T21:13:43.375941+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:12:03.255395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:03.255395+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":104,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.383255+0000","last_change":"2024-02-19T21:12:09.345352+0000","last_active":"2024-02-19T21:13:43.383255+0000","last_peered":"2024-02-19T21:13:43.383255+0000","last_clean":"2024-02-19T21:13:43.383255+0000","last_became_active":"2024-02-19T21:12:09.345000+0000","last_became_peered":"2024-02-19T21:12:09.345000+0000","last_unstale":"2024-02-19T21:13:43.383255+0000","last_undegraded":"2024-02-19T21:13:43.383255+0000","last_fullsized":"2024-02-19T21:13:43.383255+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:00.382946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.382946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":573,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382806+0000","last_change":"2024-02-19T21:12:09.332889+0000","last_active":"2024-02-19T21:13:43.382806+0000","last_peered":"2024-02-19T21:13:43.382806+0000","last_clean":"2024-02-19T21:13:43.382806+0000","last_became_active":"2024-02-19T21:12:09.332485+0000","last_became_peered":"2024-02-19T21:12:09.332485+0000","last_unstale":"2024-02-19T21:13:43.382806+0000","last_undegraded":"2024-02-19T21:13:43.382806+0000","last_fullsized":"2024-02-19T21:13:43.382806+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.566208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.566208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":317,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376395+0000","last_change":"2024-02-19T21:12:09.344855+0000","last_active":"2024-02-19T21:13:43.376395+0000","last_peered":"2024-02-19T21:13:43.376395+0000","last_clean":"2024-02-19T21:13:43.376395+0000","last_became_active":"2024-02-19T21:12:09.344536+0000","last_became_peered":"2024-02-19T21:12:09.344536+0000","last_unstale":"2024-02-19T21:13:43.376395+0000","last_undegraded":"2024-02-19T21:13:43.376395+0000","last_fullsized":"2024-02-19T21:13:43.376395+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:00.547228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:00.547228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":390,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382921+0000","last_change":"2024-02-19T21:12:09.343909+0000","last_active":"2024-02-19T21:13:43.382921+0000","last_peered":"2024-02-19T21:13:43.382921+0000","last_clean":"2024-02-19T21:13:43.382921+0000","last_became_active":"2024-02-19T21:12:09.342686+0000","last_became_peered":"2024-02-19T21:12:09.342686+0000","last_unstale":"2024-02-19T21:13:43.382921+0000","last_undegraded":"2024-02-19T21:13:43.382921+0000","last_fullsized":"2024-02-19T21:13:43.382921+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:37.718654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:11:37.718654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"459'20","reported_seq":487,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.384580+0000","last_change":"2024-02-19T21:12:48.422862+0000","last_active":"2024-02-19T21:13:43.384580+0000","last_peered":"2024-02-19T21:13:43.384580+0000","last_clean":"2024-02-19T21:13:43.384580+0000","last_became_active":"2024-02-19T21:11:59.350142+0000","last_became_peered":"2024-02-19T21:11:59.350142+0000","last_unstale":"2024-02-19T21:13:43.384580+0000","last_undegraded":"2024-02-19T21:13:43.384580+0000","last_fullsized":"2024-02-19T21:13:43.384580+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:12:48.422763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:48.422763+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":572,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381648+0000","last_change":"2024-02-19T21:12:48.430341+0000","last_active":"2024-02-19T21:13:43.381648+0000","last_peered":"2024-02-19T21:13:43.381648+0000","last_clean":"2024-02-19T21:13:43.381648+0000","last_became_active":"2024-02-19T21:10:41.294592+0000","last_became_peered":"2024-02-19T21:10:41.294592+0000","last_unstale":"2024-02-19T21:13:43.381648+0000","last_undegraded":"2024-02-19T21:13:43.381648+0000","last_fullsized":"2024-02-19T21:13:43.381648+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:48.430223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:48.430223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"493'11","reported_seq":667,"reported_epoch":588,"state":"active+clean","last_fresh":"2024-02-19T21:13:44.340120+0000","last_change":"2024-02-19T21:13:39.691375+0000","last_active":"2024-02-19T21:13:44.340120+0000","last_peered":"2024-02-19T21:13:44.340120+0000","last_clean":"2024-02-19T21:13:44.340120+0000","last_became_active":"2024-02-19T21:12:09.352204+0000","last_became_peered":"2024-02-19T21:12:09.352204+0000","last_unstale":"2024-02-19T21:13:44.340120+0000","last_undegraded":"2024-02-19T21:13:44.340120+0000","last_fullsized":"2024-02-19T21:13:44.340120+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'11","last_scrub_stamp":"2024-02-19T21:13:39.691291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:39.691291+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":551,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.384327+0000","last_change":"2024-02-19T21:11:43.546468+0000","last_active":"2024-02-19T21:13:43.384327+0000","last_peered":"2024-02-19T21:13:43.384327+0000","last_clean":"2024-02-19T21:13:43.384327+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:13:43.384327+0000","last_undegraded":"2024-02-19T21:13:43.384327+0000","last_fullsized":"2024-02-19T21:13:43.384327+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:11:43.546383+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.546383+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":505,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374930+0000","last_change":"2024-02-19T21:11:38.369815+0000","last_active":"2024-02-19T21:13:43.374930+0000","last_peered":"2024-02-19T21:13:43.374930+0000","last_clean":"2024-02-19T21:13:43.374930+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:13:43.374930+0000","last_undegraded":"2024-02-19T21:13:43.374930+0000","last_fullsized":"2024-02-19T21:13:43.374930+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.369716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.369716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"459'19","reported_seq":572,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.383314+0000","last_change":"2024-02-19T21:12:10.390849+0000","last_active":"2024-02-19T21:13:43.383314+0000","last_peered":"2024-02-19T21:13:43.383314+0000","last_clean":"2024-02-19T21:13:43.383314+0000","last_became_active":"2024-02-19T21:12:09.350490+0000","last_became_peered":"2024-02-19T21:12:09.350490+0000","last_unstale":"2024-02-19T21:13:43.383314+0000","last_undegraded":"2024-02-19T21:13:43.383314+0000","last_fullsized":"2024-02-19T21:13:43.383314+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:10:54.435839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.435839+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":317,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376017+0000","last_change":"2024-02-19T21:12:09.333078+0000","last_active":"2024-02-19T21:13:43.376017+0000","last_peered":"2024-02-19T21:13:43.376017+0000","last_clean":"2024-02-19T21:13:43.376017+0000","last_became_active":"2024-02-19T21:12:09.332587+0000","last_became_peered":"2024-02-19T21:12:09.332587+0000","last_unstale":"2024-02-19T21:13:43.376017+0000","last_undegraded":"2024-02-19T21:13:43.376017+0000","last_fullsized":"2024-02-19T21:13:43.376017+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:32.600213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:32.600213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":391,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.384627+0000","last_change":"2024-02-19T21:11:59.334743+0000","last_active":"2024-02-19T21:13:43.384627+0000","last_peered":"2024-02-19T21:13:43.384627+0000","last_clean":"2024-02-19T21:13:43.384627+0000","last_became_active":"2024-02-19T21:11:59.334454+0000","last_became_peered":"2024-02-19T21:11:59.334454+0000","last_unstale":"2024-02-19T21:13:43.384627+0000","last_undegraded":"2024-02-19T21:13:43.384627+0000","last_fullsized":"2024-02-19T21:13:43.384627+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.692260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.692260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":564,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382289+0000","last_change":"2024-02-19T21:12:06.991132+0000","last_active":"2024-02-19T21:13:43.382289+0000","last_peered":"2024-02-19T21:13:43.382289+0000","last_clean":"2024-02-19T21:13:43.382289+0000","last_became_active":"2024-02-19T21:10:41.312577+0000","last_became_peered":"2024-02-19T21:10:41.312577+0000","last_unstale":"2024-02-19T21:13:43.382289+0000","last_undegraded":"2024-02-19T21:13:43.382289+0000","last_fullsized":"2024-02-19T21:13:43.382289+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.991039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.991039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":518,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382201+0000","last_change":"2024-02-19T21:10:59.782435+0000","last_active":"2024-02-19T21:13:43.382201+0000","last_peered":"2024-02-19T21:13:43.382201+0000","last_clean":"2024-02-19T21:13:43.382201+0000","last_became_active":"2024-02-19T21:10:41.311050+0000","last_became_peered":"2024-02-19T21:10:41.311050+0000","last_unstale":"2024-02-19T21:13:43.382201+0000","last_undegraded":"2024-02-19T21:13:43.382201+0000","last_fullsized":"2024-02-19T21:13:43.382201+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:59.782367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.782367+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"459'34","reported_seq":657,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.378442+0000","last_change":"2024-02-19T21:13:39.513661+0000","last_active":"2024-02-19T21:13:43.378442+0000","last_peered":"2024-02-19T21:13:43.378442+0000","last_clean":"2024-02-19T21:13:43.378442+0000","last_became_active":"2024-02-19T21:08:04.248378+0000","last_became_peered":"2024-02-19T21:08:04.248378+0000","last_unstale":"2024-02-19T21:13:43.378442+0000","last_undegraded":"2024-02-19T21:13:43.378442+0000","last_fullsized":"2024-02-19T21:13:43.378442+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'34","last_scrub_stamp":"2024-02-19T21:13:39.513604+0000","last_deep_scrub":"419'22","last_deep_scrub_stamp":"2024-02-19T21:10:19.109244+0000","last_clean_scrub_stamp":"2024-02-19T21:13:39.513604+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":514,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374496+0000","last_change":"2024-02-19T21:12:09.341126+0000","last_active":"2024-02-19T21:13:43.374496+0000","last_peered":"2024-02-19T21:13:43.374496+0000","last_clean":"2024-02-19T21:13:43.374496+0000","last_became_active":"2024-02-19T21:12:09.340803+0000","last_became_peered":"2024-02-19T21:12:09.340803+0000","last_unstale":"2024-02-19T21:13:43.374496+0000","last_undegraded":"2024-02-19T21:13:43.374496+0000","last_fullsized":"2024-02-19T21:13:43.374496+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.379335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.379335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":575,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382636+0000","last_change":"2024-02-19T21:11:59.323076+0000","last_active":"2024-02-19T21:13:43.382636+0000","last_peered":"2024-02-19T21:13:43.382636+0000","last_clean":"2024-02-19T21:13:43.382636+0000","last_became_active":"2024-02-19T21:11:59.322735+0000","last_became_peered":"2024-02-19T21:11:59.322735+0000","last_unstale":"2024-02-19T21:13:43.382636+0000","last_undegraded":"2024-02-19T21:13:43.382636+0000","last_fullsized":"2024-02-19T21:13:43.382636+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:53.709456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.709456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":182,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.735477+0000","last_change":"2024-02-19T21:13:43.735451+0000","last_active":"2024-02-19T21:13:43.735477+0000","last_peered":"2024-02-19T21:13:43.735477+0000","last_clean":"2024-02-19T21:13:43.735477+0000","last_became_active":"2024-02-19T21:12:09.370318+0000","last_became_peered":"2024-02-19T21:12:09.370318+0000","last_unstale":"2024-02-19T21:13:43.735477+0000","last_undegraded":"2024-02-19T21:13:43.735477+0000","last_fullsized":"2024-02-19T21:13:43.735477+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:43.735382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:13:43.735382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":384,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381495+0000","last_change":"2024-02-19T21:12:09.340710+0000","last_active":"2024-02-19T21:13:43.381495+0000","last_peered":"2024-02-19T21:13:43.381495+0000","last_clean":"2024-02-19T21:13:43.381495+0000","last_became_active":"2024-02-19T21:12:09.340360+0000","last_became_peered":"2024-02-19T21:12:09.340360+0000","last_unstale":"2024-02-19T21:13:43.381495+0000","last_undegraded":"2024-02-19T21:13:43.381495+0000","last_fullsized":"2024-02-19T21:13:43.381495+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.342173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.342173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"459'18","reported_seq":367,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376013+0000","last_change":"2024-02-19T21:12:50.243591+0000","last_active":"2024-02-19T21:13:43.376013+0000","last_peered":"2024-02-19T21:13:43.376013+0000","last_clean":"2024-02-19T21:13:43.376013+0000","last_became_active":"2024-02-19T21:10:41.307690+0000","last_became_peered":"2024-02-19T21:10:41.307690+0000","last_unstale":"2024-02-19T21:13:43.376013+0000","last_undegraded":"2024-02-19T21:13:43.376013+0000","last_fullsized":"2024-02-19T21:13:43.376013+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'18","last_scrub_stamp":"2024-02-19T21:12:50.243498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:50.243498+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":308,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376473+0000","last_change":"2024-02-19T21:12:09.344823+0000","last_active":"2024-02-19T21:13:43.376473+0000","last_peered":"2024-02-19T21:13:43.376473+0000","last_clean":"2024-02-19T21:13:43.376473+0000","last_became_active":"2024-02-19T21:12:09.344422+0000","last_became_peered":"2024-02-19T21:12:09.344422+0000","last_unstale":"2024-02-19T21:13:43.376473+0000","last_undegraded":"2024-02-19T21:13:43.376473+0000","last_fullsized":"2024-02-19T21:13:43.376473+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:04.545988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:04.545988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":168,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374444+0000","last_change":"2024-02-19T21:11:43.369675+0000","last_active":"2024-02-19T21:13:43.374444+0000","last_peered":"2024-02-19T21:13:43.374444+0000","last_clean":"2024-02-19T21:13:43.374444+0000","last_became_active":"2024-02-19T21:10:41.296849+0000","last_became_peered":"2024-02-19T21:10:41.296849+0000","last_unstale":"2024-02-19T21:13:43.374444+0000","last_undegraded":"2024-02-19T21:13:43.374444+0000","last_fullsized":"2024-02-19T21:13:43.374444+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:43.369607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.369607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":308,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.378902+0000","last_change":"2024-02-19T21:12:09.332385+0000","last_active":"2024-02-19T21:13:43.378902+0000","last_peered":"2024-02-19T21:13:43.378902+0000","last_clean":"2024-02-19T21:13:43.378902+0000","last_became_active":"2024-02-19T21:12:09.332044+0000","last_became_peered":"2024-02-19T21:12:09.332044+0000","last_unstale":"2024-02-19T21:13:43.378902+0000","last_undegraded":"2024-02-19T21:13:43.378902+0000","last_fullsized":"2024-02-19T21:13:43.378902+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:58.688166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.688166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":509,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381778+0000","last_change":"2024-02-19T21:12:09.350012+0000","last_active":"2024-02-19T21:13:43.381778+0000","last_peered":"2024-02-19T21:13:43.381778+0000","last_clean":"2024-02-19T21:13:43.381778+0000","last_became_active":"2024-02-19T21:12:09.349709+0000","last_became_peered":"2024-02-19T21:12:09.349709+0000","last_unstale":"2024-02-19T21:13:43.381778+0000","last_undegraded":"2024-02-19T21:13:43.381778+0000","last_fullsized":"2024-02-19T21:13:43.381778+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:00.754327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.754327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"459'19","reported_seq":206,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374701+0000","last_change":"2024-02-19T21:12:55.429884+0000","last_active":"2024-02-19T21:13:43.374701+0000","last_peered":"2024-02-19T21:13:43.374701+0000","last_clean":"2024-02-19T21:13:43.374701+0000","last_became_active":"2024-02-19T21:11:59.336536+0000","last_became_peered":"2024-02-19T21:11:59.336536+0000","last_unstale":"2024-02-19T21:13:43.374701+0000","last_undegraded":"2024-02-19T21:13:43.374701+0000","last_fullsized":"2024-02-19T21:13:43.374701+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:12:55.429801+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.429801+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":550,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.383941+0000","last_change":"2024-02-19T21:11:30.472665+0000","last_active":"2024-02-19T21:13:43.383941+0000","last_peered":"2024-02-19T21:13:43.383941+0000","last_clean":"2024-02-19T21:13:43.383941+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:13:43.383941+0000","last_undegraded":"2024-02-19T21:13:43.383941+0000","last_fullsized":"2024-02-19T21:13:43.383941+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:30.472591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:30.472591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":559,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382438+0000","last_change":"2024-02-19T21:13:06.395759+0000","last_active":"2024-02-19T21:13:43.382438+0000","last_peered":"2024-02-19T21:13:43.382438+0000","last_clean":"2024-02-19T21:13:43.382438+0000","last_became_active":"2024-02-19T21:11:59.323593+0000","last_became_peered":"2024-02-19T21:11:59.323593+0000","last_unstale":"2024-02-19T21:13:43.382438+0000","last_undegraded":"2024-02-19T21:13:43.382438+0000","last_fullsized":"2024-02-19T21:13:43.382438+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:06.395650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.395650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":392,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381186+0000","last_change":"2024-02-19T21:13:43.109663+0000","last_active":"2024-02-19T21:13:43.381186+0000","last_peered":"2024-02-19T21:13:43.381186+0000","last_clean":"2024-02-19T21:13:43.381186+0000","last_became_active":"2024-02-19T21:11:59.352235+0000","last_became_peered":"2024-02-19T21:11:59.352235+0000","last_unstale":"2024-02-19T21:13:43.381186+0000","last_undegraded":"2024-02-19T21:13:43.381186+0000","last_fullsized":"2024-02-19T21:13:43.381186+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:43.109579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:13:43.109579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":534,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376227+0000","last_change":"2024-02-19T21:13:10.146168+0000","last_active":"2024-02-19T21:13:43.376227+0000","last_peered":"2024-02-19T21:13:43.376227+0000","last_clean":"2024-02-19T21:13:43.376227+0000","last_became_active":"2024-02-19T21:11:59.357345+0000","last_became_peered":"2024-02-19T21:11:59.357345+0000","last_unstale":"2024-02-19T21:13:43.376227+0000","last_undegraded":"2024-02-19T21:13:43.376227+0000","last_fullsized":"2024-02-19T21:13:43.376227+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:10.146102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:13:10.146102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"459'7","reported_seq":177,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375168+0000","last_change":"2024-02-19T21:12:02.521592+0000","last_active":"2024-02-19T21:13:43.375168+0000","last_peered":"2024-02-19T21:13:43.375168+0000","last_clean":"2024-02-19T21:13:43.375168+0000","last_became_active":"2024-02-19T21:10:41.297295+0000","last_became_peered":"2024-02-19T21:10:41.297295+0000","last_unstale":"2024-02-19T21:13:43.375168+0000","last_undegraded":"2024-02-19T21:13:43.375168+0000","last_fullsized":"2024-02-19T21:13:43.375168+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'7","last_scrub_stamp":"2024-02-19T21:12:02.521523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:02.521523+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":573,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.384188+0000","last_change":"2024-02-19T21:12:09.324077+0000","last_active":"2024-02-19T21:13:43.384188+0000","last_peered":"2024-02-19T21:13:43.384188+0000","last_clean":"2024-02-19T21:13:43.384188+0000","last_became_active":"2024-02-19T21:12:09.323141+0000","last_became_peered":"2024-02-19T21:12:09.323141+0000","last_unstale":"2024-02-19T21:13:43.384188+0000","last_undegraded":"2024-02-19T21:13:43.384188+0000","last_fullsized":"2024-02-19T21:13:43.384188+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:04.547957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:04.547957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":569,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.383279+0000","last_change":"2024-02-19T21:12:51.411694+0000","last_active":"2024-02-19T21:13:43.383279+0000","last_peered":"2024-02-19T21:13:43.383279+0000","last_clean":"2024-02-19T21:13:43.383279+0000","last_became_active":"2024-02-19T21:10:41.307400+0000","last_became_peered":"2024-02-19T21:10:41.307400+0000","last_unstale":"2024-02-19T21:13:43.383279+0000","last_undegraded":"2024-02-19T21:13:43.383279+0000","last_fullsized":"2024-02-19T21:13:43.383279+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:51.411574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.411574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":566,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381690+0000","last_change":"2024-02-19T21:10:15.161470+0000","last_active":"2024-02-19T21:13:43.381690+0000","last_peered":"2024-02-19T21:13:43.381690+0000","last_clean":"2024-02-19T21:13:43.381690+0000","last_became_active":"2024-02-19T21:08:04.266100+0000","last_became_peered":"2024-02-19T21:08:04.266100+0000","last_unstale":"2024-02-19T21:13:43.381690+0000","last_undegraded":"2024-02-19T21:13:43.381690+0000","last_fullsized":"2024-02-19T21:13:43.381690+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:15.161397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:15.161397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":412,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.383100+0000","last_change":"2024-02-19T21:12:40.365196+0000","last_active":"2024-02-19T21:13:43.383100+0000","last_peered":"2024-02-19T21:13:43.383100+0000","last_clean":"2024-02-19T21:13:43.383100+0000","last_became_active":"2024-02-19T21:12:40.364772+0000","last_became_peered":"2024-02-19T21:12:40.364772+0000","last_unstale":"2024-02-19T21:13:43.383100+0000","last_undegraded":"2024-02-19T21:13:43.383100+0000","last_fullsized":"2024-02-19T21:13:43.383100+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:56.083403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:56.083403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"459'15","reported_seq":206,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374054+0000","last_change":"2024-02-19T21:12:09.959562+0000","last_active":"2024-02-19T21:13:43.374054+0000","last_peered":"2024-02-19T21:13:43.374054+0000","last_clean":"2024-02-19T21:13:43.374054+0000","last_became_active":"2024-02-19T21:12:09.352882+0000","last_became_peered":"2024-02-19T21:12:09.352882+0000","last_unstale":"2024-02-19T21:13:43.374054+0000","last_undegraded":"2024-02-19T21:13:43.374054+0000","last_fullsized":"2024-02-19T21:13:43.374054+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:11:03.423767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.423767+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":308,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376195+0000","last_change":"2024-02-19T21:11:59.337265+0000","last_active":"2024-02-19T21:13:43.376195+0000","last_peered":"2024-02-19T21:13:43.376195+0000","last_clean":"2024-02-19T21:13:43.376195+0000","last_became_active":"2024-02-19T21:11:59.336778+0000","last_became_peered":"2024-02-19T21:11:59.336778+0000","last_unstale":"2024-02-19T21:13:43.376195+0000","last_undegraded":"2024-02-19T21:13:43.376195+0000","last_fullsized":"2024-02-19T21:13:43.376195+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:50.769620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.769620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":553,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.381645+0000","last_change":"2024-02-19T21:13:42.086086+0000","last_active":"2024-02-19T21:13:43.381645+0000","last_peered":"2024-02-19T21:13:43.381645+0000","last_clean":"2024-02-19T21:13:43.381645+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:13:43.381645+0000","last_undegraded":"2024-02-19T21:13:43.381645+0000","last_fullsized":"2024-02-19T21:13:43.381645+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:42.086006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:13:42.086006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":168,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374778+0000","last_change":"2024-02-19T21:12:07.416212+0000","last_active":"2024-02-19T21:13:43.374778+0000","last_peered":"2024-02-19T21:13:43.374778+0000","last_clean":"2024-02-19T21:13:43.374778+0000","last_became_active":"2024-02-19T21:10:41.309550+0000","last_became_peered":"2024-02-19T21:10:41.309550+0000","last_unstale":"2024-02-19T21:13:43.374778+0000","last_undegraded":"2024-02-19T21:13:43.374778+0000","last_fullsized":"2024-02-19T21:13:43.374778+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:07.416150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:07.416150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":529,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.382609+0000","last_change":"2024-02-19T21:11:59.333342+0000","last_active":"2024-02-19T21:13:43.382609+0000","last_peered":"2024-02-19T21:13:43.382609+0000","last_clean":"2024-02-19T21:13:43.382609+0000","last_became_active":"2024-02-19T21:11:59.332987+0000","last_became_peered":"2024-02-19T21:11:59.332987+0000","last_unstale":"2024-02-19T21:13:43.382609+0000","last_undegraded":"2024-02-19T21:13:43.382609+0000","last_fullsized":"2024-02-19T21:13:43.382609+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:08.043028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:08.043028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"459'27","reported_seq":358,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375633+0000","last_change":"2024-02-19T21:11:59.360178+0000","last_active":"2024-02-19T21:13:43.375633+0000","last_peered":"2024-02-19T21:13:43.375633+0000","last_clean":"2024-02-19T21:13:43.375633+0000","last_became_active":"2024-02-19T21:11:59.340940+0000","last_became_peered":"2024-02-19T21:11:59.340940+0000","last_unstale":"2024-02-19T21:13:43.375633+0000","last_undegraded":"2024-02-19T21:13:43.375633+0000","last_fullsized":"2024-02-19T21:13:43.375633+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:10:02.532799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:02.532799+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":420,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.384430+0000","last_change":"2024-02-19T21:12:09.339398+0000","last_active":"2024-02-19T21:13:43.384430+0000","last_peered":"2024-02-19T21:13:43.384430+0000","last_clean":"2024-02-19T21:13:43.384430+0000","last_became_active":"2024-02-19T21:12:09.338747+0000","last_became_peered":"2024-02-19T21:12:09.338747+0000","last_unstale":"2024-02-19T21:13:43.384430+0000","last_undegraded":"2024-02-19T21:13:43.384430+0000","last_fullsized":"2024-02-19T21:13:43.384430+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.720583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.720583+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":177,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.374099+0000","last_change":"2024-02-19T21:12:09.346777+0000","last_active":"2024-02-19T21:13:43.374099+0000","last_peered":"2024-02-19T21:13:43.374099+0000","last_clean":"2024-02-19T21:13:43.374099+0000","last_became_active":"2024-02-19T21:12:09.346258+0000","last_became_peered":"2024-02-19T21:12:09.346258+0000","last_unstale":"2024-02-19T21:13:43.374099+0000","last_undegraded":"2024-02-19T21:13:43.374099+0000","last_fullsized":"2024-02-19T21:13:43.374099+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.407543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.407543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":313,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.376221+0000","last_change":"2024-02-19T21:13:40.304585+0000","last_active":"2024-02-19T21:13:43.376221+0000","last_peered":"2024-02-19T21:13:43.376221+0000","last_clean":"2024-02-19T21:13:43.376221+0000","last_became_active":"2024-02-19T21:12:09.330530+0000","last_became_peered":"2024-02-19T21:12:09.330530+0000","last_unstale":"2024-02-19T21:13:43.376221+0000","last_undegraded":"2024-02-19T21:13:43.376221+0000","last_fullsized":"2024-02-19T21:13:43.376221+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:40.304490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:40.304490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":331,"reported_epoch":587,"state":"active+clean","last_fresh":"2024-02-19T21:13:43.375799+0000","last_change":"2024-02-19T21:12:09.345873+0000","last_active":"2024-02-19T21:13:43.375799+0000","last_peered":"2024-02-19T21:13:43.375799+0000","last_clean":"2024-02-19T21:13:43.375799+0000","last_became_active":"2024-02-19T21:12:09.345500+0000","last_became_peered":"2024-02-19T21:12:09.345500+0000","last_unstale":"2024-02-19T21:13:43.375799+0000","last_undegraded":"2024-02-19T21:13:43.375799+0000","last_fullsized":"2024-02-19T21:13:43.375799+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.764658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.764658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":229,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":24,"acting":24,"num_store_stats":5},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1102,"num_read_kb":859,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":7558,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":712,"ondisk_log_size":712,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":739,"num_read_kb":734,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":887,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":126,"acting":126,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":15,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266840,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316572,"kb_used_data":2964,"kb_used_omap":0,"kb_used_meta":313600,"kb_avail":93428580,"statfs":{"total":95995035648,"available":95670865920,"internally_reserved":0,"allocated":3035136,"data_stored":1469334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":321126400},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48999999999999999,"15min":0.86299999999999999},"min":{"1min":0.32500000000000001,"5min":0.26000000000000001,"15min":0.215},"max":{"1min":0.66800000000000004,"5min":0.79200000000000004,"15min":89.456999999999994},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.48799999999999999,"15min":0.78600000000000003},"min":{"1min":0.41199999999999998,"5min":0.28499999999999998,"15min":0.25900000000000001},"max":{"1min":0.64900000000000002,"5min":0.81299999999999994,"15min":70.451999999999998},"last":0.46400000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.502,"15min":0.69499999999999995},"min":{"1min":0.35699999999999998,"5min":0.29999999999999999,"15min":0.28199999999999997},"max":{"1min":0.67500000000000004,"5min":0.85099999999999998,"15min":34.273000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48199999999999998,"15min":0.93100000000000005},"min":{"1min":0.34000000000000002,"5min":0.28299999999999997,"15min":0.254},"max":{"1min":0.70899999999999996,"5min":0.79200000000000004,"15min":89.594999999999999},"last":0.38900000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.52300000000000002,"15min":0.95199999999999996},"min":{"1min":0.34399999999999997,"5min":0.312,"15min":0.24299999999999999},"max":{"1min":4.3769999999999998,"5min":4.3769999999999998,"15min":104.413},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.53000000000000003,"15min":0.95299999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.27600000000000002},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":104.268},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Feb 19 21:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.57499999999999996,"15min":0.96299999999999997},"min":{"1min":0.41199999999999998,"5min":0.33400000000000002,"15min":0.254},"max":{"1min":2.9769999999999999,"5min":2.9769999999999999,"15min":105.3},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.53700000000000003,"15min":0.60899999999999999},"min":{"1min":0.36799999999999999,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":13.243},"last":0.52300000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.434,"15min":0.45700000000000002},"min":{"1min":0.31,"5min":0.28499999999999998,"15min":0.24299999999999999},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":1.373},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.48099999999999998,"15min":0.47699999999999998},"min":{"1min":0.34100000000000003,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":0.73399999999999999,"5min":0.751,"15min":1.0740000000000001},"last":0.59999999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.45300000000000001,"15min":0.45600000000000002},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.252},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.497,"15min":0.504},"min":{"1min":0.36199999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63900000000000001,"5min":0.79300000000000004,"15min":0.85899999999999999},"last":0.5}]},{"osd":6,"last update":"Mon Feb 19 21:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.51300000000000001,"15min":0.51600000000000001},"min":{"1min":0.35999999999999999,"5min":0.30199999999999999,"15min":0.28799999999999998},"max":{"1min":0.70799999999999996,"5min":0.76700000000000002,"15min":1.0049999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.39500000000000002,"5min":0.34100000000000003,"15min":0.33700000000000002},"max":{"1min":0.747,"5min":0.749,"15min":1.262},"last":0.59499999999999997}]}]},{"osd":6,"up_from":0,"seq":193273528478,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":126,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312480,"kb_used_data":2968,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93432672,"statfs":{"total":95995035648,"available":95675056128,"internally_reserved":0,"allocated":3039232,"data_stored":1459384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.33300000000000002,"15min":0.54500000000000004},"min":{"1min":0.19700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.38900000000000001,"5min":2.0379999999999998,"15min":27.777000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34799999999999998,"15min":0.57399999999999995},"min":{"1min":0.215,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.51900000000000002,"5min":1.9870000000000001,"15min":27.631},"last":0.28000000000000003}]},{"osd":2,"last update":"Mon Feb 19 21:13:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.36799999999999999,"15min":0.70099999999999996},"min":{"1min":0.217,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.41999999999999998,"5min":2.0230000000000001,"15min":27.798999999999999},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.38700000000000001,"15min":0.71799999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.443,"5min":2.089,"15min":27.606999999999999},"last":0.378}]},{"osd":3,"last update":"Mon Feb 19 21:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.374,"15min":0.56599999999999995},"min":{"1min":0.253,"5min":0.217,"15min":0.217},"max":{"1min":0.499,"5min":1.7450000000000001,"15min":15.574999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.41099999999999998,"15min":0.66200000000000003},"min":{"1min":0.31,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.622,"5min":2.0070000000000001,"15min":27.82},"last":0.36699999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:13:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51000000000000001,"15min":0.64700000000000002},"min":{"1min":0.313,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71999999999999997,"5min":1.149,"15min":9.6430000000000007},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54800000000000004,"15min":0.752},"min":{"1min":0.41399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.79800000000000004,"5min":1.9770000000000001,"15min":27.707000000000001},"last":0.58599999999999997}]},{"osd":5,"last update":"Mon Feb 19 21:12:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52700000000000002,"15min":0.73899999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98599999999999999,"5min":1.048,"15min":27.529},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.51800000000000002,"15min":0.67500000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67400000000000004,"5min":1.1819999999999999,"15min":9.6799999999999997},"last":0.59699999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52600000000000002,"15min":0.65800000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70399999999999996,"5min":1.1659999999999999,"15min":9.6989999999999998},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.60099999999999998,"15min":0.78600000000000003},"min":{"1min":0.40799999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75900000000000001,"5min":3.4359999999999999,"15min":27.733000000000001},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Feb 19 21:13:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57599999999999996,"15min":0.77400000000000002},"min":{"1min":0.40500000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79800000000000004,"5min":3.4670000000000001,"15min":27.576000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59199999999999997,"15min":0.80400000000000005},"min":{"1min":0.46800000000000003,"5min":0.375,"15min":0.375},"max":{"1min":0.81599999999999995,"5min":2.0169999999999999,"15min":27.756},"last":0.65200000000000002}]}]},{"osd":0,"up_from":0,"seq":34359738555,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411504,"num_pgs":109,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308368,"kb_used_data":2952,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436784,"statfs":{"total":95995035648,"available":95679266816,"internally_reserved":0,"allocated":3022848,"data_stored":1472354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.53500000000000003},"min":{"1min":0.223,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.53600000000000003,"5min":0.54600000000000004,"15min":26.690999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.34100000000000003,"15min":0.59199999999999997},"min":{"1min":0.24299999999999999,"5min":0.23400000000000001,"15min":0.214},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":24.315000000000001},"last":0.24299999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:13:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.34300000000000003,"15min":0.70599999999999996},"min":{"1min":0.22600000000000001,"5min":0.20599999999999999,"15min":0.20100000000000001},"max":{"1min":0.62,"5min":0.62,"15min":28.882000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.376,"15min":0.68200000000000005},"min":{"1min":0.246,"5min":0.23899999999999999,"15min":0.22500000000000001},"max":{"1min":0.61299999999999999,"5min":0.77600000000000002,"15min":28.382999999999999},"last":0.17999999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.53700000000000003,"15min":0.71199999999999997},"min":{"1min":0.27800000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68899999999999995,"5min":12.75,"15min":28.742000000000001},"last":0.38},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.44900000000000001,"15min":0.63300000000000001},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.25700000000000001},"max":{"1min":0.58399999999999996,"5min":3.6499999999999999,"15min":26.719999999999999},"last":0.31900000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:13:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58399999999999996,"15min":0.69399999999999995},"min":{"1min":0.36899999999999999,"5min":0.33400000000000002,"15min":0.30599999999999999},"max":{"1min":0.85599999999999998,"5min":3.4039999999999999,"15min":24.356000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60299999999999998,"15min":0.71699999999999997},"min":{"1min":0.38300000000000001,"5min":0.33700000000000002,"15min":0.32400000000000001},"max":{"1min":1.2010000000000001,"5min":3.3919999999999999,"15min":20.707000000000001},"last":0.50600000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.57799999999999996,"15min":0.68000000000000005},"min":{"1min":0.36399999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81599999999999995,"5min":3.3090000000000002,"15min":9.9499999999999993},"last":0.44},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.60899999999999999,"15min":0.79100000000000004},"min":{"1min":0.41399999999999998,"5min":0.39800000000000002,"15min":0.38400000000000001},"max":{"1min":0.77500000000000002,"5min":3.2690000000000001,"15min":26.606000000000002},"last":0.39200000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.58299999999999996,"15min":0.79800000000000004},"min":{"1min":0.438,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.876,"5min":1.4510000000000001,"15min":24.381},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.73399999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80200000000000005,"5min":0.93600000000000005,"15min":28.706},"last":0.54300000000000004}]},{"osd":7,"last update":"Mon Feb 19 21:13:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59199999999999997,"15min":0.77800000000000002},"min":{"1min":0.434,"5min":0.42199999999999999,"15min":0.34200000000000003},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":26.658000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59299999999999997,"15min":0.73399999999999999},"min":{"1min":0.42099999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":24.402000000000001},"last":0.35399999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182572,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308384,"kb_used_data":2968,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436768,"statfs":{"total":95995035648,"available":95679250432,"internally_reserved":0,"allocated":3039232,"data_stored":1462380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.71599999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.156},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":77.662000000000006},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.35499999999999998,"15min":0.49399999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.19700000000000001},"max":{"1min":0.60199999999999998,"5min":0.60999999999999999,"15min":17.456},"last":0.58399999999999996}]},{"osd":1,"last update":"Mon Feb 19 21:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.52600000000000002},"min":{"1min":0.255,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":0.66300000000000003,"5min":6.9299999999999997,"15min":17.327999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.42499999999999999,"15min":0.50600000000000001},"min":{"1min":0.20699999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.95599999999999996,"5min":7.3620000000000001,"15min":15.004},"last":0.504}]},{"osd":2,"last update":"Mon Feb 19 21:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.376,"15min":0.66300000000000003},"min":{"1min":0.27300000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":56.890999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.36599999999999999,"15min":0.64400000000000002},"min":{"1min":0.246,"5min":0.24399999999999999,"15min":0.20899999999999999},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":44.600999999999999},"last":0.61399999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:13:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54000000000000004,"15min":0.63600000000000001},"min":{"1min":0.36099999999999999,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":12.847},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.53300000000000003,"15min":0.70499999999999996},"min":{"1min":0.317,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":26.393999999999998},"last":0.65200000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:13:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.61699999999999999,"15min":0.68200000000000005},"min":{"1min":0.39000000000000001,"5min":0.34100000000000003,"15min":0.313},"max":{"1min":0.70799999999999996,"5min":9.2360000000000007,"15min":17.401},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55900000000000005,"15min":0.65500000000000003},"min":{"1min":0.36799999999999999,"5min":0.313,"15min":0.313},"max":{"1min":1.397,"5min":1.9690000000000001,"15min":12.882},"last":0.63}]},{"osd":6,"last update":"Mon Feb 19 21:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56000000000000005,"15min":0.65200000000000002},"min":{"1min":0.38,"5min":0.34799999999999998,"15min":0.34300000000000003},"max":{"1min":0.69999999999999996,"5min":1.9790000000000001,"15min":17.367000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54900000000000004,"15min":0.66200000000000003},"min":{"1min":0.377,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":0.89400000000000002,"5min":0.95099999999999996,"15min":12.904},"last":0.73099999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:13:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56599999999999995,"15min":0.65700000000000003},"min":{"1min":0.42799999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.751,"5min":1.038,"15min":17.428999999999998},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.58999999999999997,"15min":0.80300000000000005},"min":{"1min":0.39700000000000002,"5min":0.374,"15min":0.34499999999999997},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":44.378},"last":0.84199999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986344,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312468,"kb_used_data":2956,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93432684,"statfs":{"total":95995035648,"available":95675068416,"internally_reserved":0,"allocated":3026944,"data_stored":1472180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.56100000000000005,"15min":0.63600000000000001},"min":{"1min":0.35199999999999998,"5min":0.26800000000000002,"15min":0.22500000000000001},"max":{"1min":0.86899999999999999,"5min":8.0530000000000008,"15min":19.878},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51000000000000001,"15min":0.85999999999999999},"min":{"1min":0.32300000000000001,"5min":0.25900000000000001,"15min":0.20499999999999999},"max":{"1min":0.82999999999999996,"5min":3.3530000000000002,"15min":70.063000000000002},"last":0.42299999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:13:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.49299999999999999,"15min":0.63800000000000001},"min":{"1min":0.34499999999999997,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":26.510999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.504,"15min":0.68300000000000005},"min":{"1min":0.32100000000000001,"5min":0.312,"15min":0.23200000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":25.949000000000002},"last":0.57199999999999995}]},{"osd":2,"last update":"Mon Feb 19 21:13:24 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.60199999999999998,"15min":0.72499999999999998},"min":{"1min":0.38700000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.2040000000000002,"5min":8.1240000000000006,"15min":37.320999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53500000000000003,"15min":1.0600000000000001},"min":{"1min":0.39900000000000002,"5min":0.29899999999999999,"15min":0.253},"max":{"1min":0.86599999999999999,"5min":2.9969999999999999,"15min":69.936999999999998},"last":0.48899999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:13:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54500000000000004,"15min":0.91200000000000003},"min":{"1min":0.38400000000000001,"5min":0.28999999999999998,"15min":0.27700000000000002},"max":{"1min":0.76700000000000002,"5min":3.323,"15min":69.793000000000006},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.54800000000000004,"15min":0.76500000000000001},"min":{"1min":0.41199999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.621,"5min":1.621,"15min":17.986999999999998},"last":0.54600000000000004}]},{"osd":5,"last update":"Mon Feb 19 21:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.218},"max":{"1min":0.78800000000000003,"5min":1.036,"15min":3.2799999999999998},"last":0.505},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.48199999999999998,"15min":0.46300000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.23899999999999999},"max":{"1min":0.745,"5min":1.121,"15min":4.4340000000000002},"last":0.57799999999999996}]},{"osd":6,"last update":"Mon Feb 19 21:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.57399999999999995,"15min":0.47599999999999998},"min":{"1min":0.29199999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.78700000000000003,"5min":8.0969999999999995,"15min":8.0969999999999995},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.51800000000000002},"min":{"1min":0.35599999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.4550000000000001,"5min":3.0270000000000001,"15min":4.5609999999999999},"last":0.56100000000000005}]},{"osd":7,"last update":"Mon Feb 19 21:12:43 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.56299999999999994,"15min":0.51000000000000001},"min":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":0.71699999999999997,"5min":3.2909999999999999,"15min":6.0730000000000004},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.55200000000000005,"15min":0.51200000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.26400000000000001},"max":{"1min":0.72699999999999998,"5min":1.0209999999999999,"15min":3.202},"last":0.59799999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790115,"num_pgs":117,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300192,"kb_used_data":2968,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444960,"statfs":{"total":95995035648,"available":95687639040,"internally_reserved":0,"allocated":3039232,"data_stored":1469773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45500000000000002,"15min":1.875},"min":{"1min":0.32900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.61599999999999999,"5min":0.71699999999999997,"15min":276.47899999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.49099999999999999,"15min":0.89900000000000002},"min":{"1min":0.35099999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.64700000000000002,"5min":0.95299999999999996,"15min":52.887},"last":0.47799999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46400000000000002,"15min":1.2609999999999999},"min":{"1min":0.30199999999999999,"5min":0.254,"15min":0.23499999999999999},"max":{"1min":0.61399999999999999,"5min":1.599,"15min":120.011},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.503,"15min":1.02},"min":{"1min":0.372,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":118.105},"last":0.40300000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47799999999999998,"15min":0.98899999999999999},"min":{"1min":0.32300000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64600000000000002,"5min":0.92300000000000004,"15min":82.242000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.56100000000000005,"15min":1.5109999999999999},"min":{"1min":0.41599999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65400000000000003,"5min":4.5019999999999998,"15min":232.84999999999999},"last":0.46400000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54700000000000004,"15min":1.0669999999999999},"min":{"1min":0.34999999999999998,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":2.2210000000000001,"5min":4.4640000000000004,"15min":109.70099999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52500000000000002,"15min":0.90800000000000003},"min":{"1min":0.41699999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.64100000000000001,"5min":1.4590000000000001,"15min":65.519000000000005},"last":0.432}]},{"osd":4,"last update":"Mon Feb 19 21:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.44600000000000001,"15min":0.42799999999999999},"min":{"1min":0.33100000000000002,"5min":0.26900000000000002,"15min":0.23300000000000001},"max":{"1min":0.66100000000000003,"5min":1.0129999999999999,"15min":1.861},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.442},"min":{"1min":0.371,"5min":0.255,"15min":0.248},"max":{"1min":0.71299999999999997,"5min":0.72499999999999998,"15min":1.9730000000000001},"last":0.42099999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47299999999999998,"15min":0.45500000000000002},"min":{"1min":0.31,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69599999999999995,"5min":1.0289999999999999,"15min":1.823},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47199999999999998,"15min":0.46300000000000002},"min":{"1min":0.30499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.442}]},{"osd":7,"last update":"Mon Feb 19 21:13:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.502,"15min":0.502},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.28499999999999998},"max":{"1min":0.73699999999999999,"5min":1.052,"15min":1.8879999999999999},"last":0.501},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.52300000000000002,"15min":0.51000000000000001},"min":{"1min":0.372,"5min":0.34399999999999997,"15min":0.27800000000000002},"max":{"1min":0.754,"5min":0.871,"15min":0.96799999999999997},"last":0.45800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":229,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:13:47.989 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool set device_health_metrics pgp_num 1 2024-02-19T21:13:47.990 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:47.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:47.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:48.070 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:13:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:13:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:13:48.093 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:48.122 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:48.320 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:48.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:48.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:48.343+0000 7f54b17fa700 1 -- 172.21.15.163:0/3221298767 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v592) v1 ==== 101+0+0 (unknown 2352553891 0 0) 0x7f54a804a5f0 con 0x7f54b40fd980 2024-02-19T21:13:48.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:48.346+0000 7f54ba819700 10 librados: watch_flush enter 2024-02-19T21:13:48.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:48.346+0000 7f54ba819700 10 librados: watch_flush exit 2024-02-19T21:13:48.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:48.346+0000 7f54ba819700 20 client.23505.objecter shutdown clearing up homeless session... 2024-02-19T21:13:48.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:48.346+0000 7f54ba819700 10 client.23505.objecter successfully canceled tick 2024-02-19T21:13:48.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:48.346+0000 7f54ba819700 1 -- 172.21.15.163:0/3221298767 >> v1:172.21.15.163:6800/1552514681 conn(0x7f549c063c70 legacy=0x7f549c066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:48.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:48.346+0000 7f54ba819700 1 -- 172.21.15.163:0/3221298767 >> v1:172.21.15.163:6789/0 conn(0x7f54b40fd980 legacy=0x7f54b40f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:48.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:48.346+0000 7f54ba819700 1 -- 172.21.15.163:0/3221298767 shutdown_connections 2024-02-19T21:13:48.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:48.346+0000 7f54ba819700 1 -- 172.21.15.163:0/3221298767 >> 172.21.15.163:0/3221298767 conn(0x7f54b40f35a0 msgr2=0x7f54b4107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:48.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:48.346+0000 7f54ba819700 1 -- 172.21.15.163:0/3221298767 shutdown_connections 2024-02-19T21:13:48.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:48.346+0000 7f54ba819700 1 -- 172.21.15.163:0/3221298767 wait complete. 2024-02-19T21:13:48.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:48.346+0000 7f54ba819700 1 librados: shutdown 2024-02-19T21:13:48.349 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:13:48.423 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:48.453 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:48.726 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:47.030933+0000 osd.1 (osd.1) 270 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:13:48.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:47.032599+0000 osd.1 (osd.1) 271 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:13:48.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:47.275909+0000 mgr.y (mgr.14550) 516 : cluster [DBG] pgmap v885: 182 pgs: 182 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:48.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:47.619504+0000 mon.a (mon.0) 3056 : audit [INF] from='client.? v1:172.21.15.163:0/3221298767' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:13:48.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:47.723486+0000 osd.4 (osd.4) 213 : cluster [DBG] 2.25 scrub starts 2024-02-19T21:13:48.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:47.724344+0000 osd.4 (osd.4) 214 : cluster [DBG] 2.25 scrub ok 2024-02-19T21:13:48.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:47.745361+0000 osd.7 (osd.7) 267 : cluster [DBG] 2.26 scrub starts 2024-02-19T21:13:48.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:47.748275+0000 osd.7 (osd.7) 268 : cluster [DBG] 2.26 scrub ok 2024-02-19T21:13:48.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:48.340288+0000 mon.a (mon.0) 3057 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:13:48.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:48.344507+0000 mon.a (mon.0) 3058 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-99"}]': finished 2024-02-19T21:13:48.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:48.344571+0000 mon.a (mon.0) 3059 : audit [INF] from='client.? v1:172.21.15.163:0/3221298767' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:13:48.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:48.344615+0000 mon.a (mon.0) 3060 : cluster [DBG] osdmap e592: 8 total, 8 up, 6 in 2024-02-19T21:13:48.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:47.030933+0000 osd.1 (osd.1) 270 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:13:48.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:47.032599+0000 osd.1 (osd.1) 271 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:13:48.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:47.275909+0000 mgr.y (mgr.14550) 516 : cluster [DBG] pgmap v885: 182 pgs: 182 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:48.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:47.619504+0000 mon.a (mon.0) 3056 : audit [INF] from='client.? v1:172.21.15.163:0/3221298767' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:13:48.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:47.723486+0000 osd.4 (osd.4) 213 : cluster [DBG] 2.25 scrub starts 2024-02-19T21:13:48.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:47.724344+0000 osd.4 (osd.4) 214 : cluster [DBG] 2.25 scrub ok 2024-02-19T21:13:48.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:47.745361+0000 osd.7 (osd.7) 267 : cluster [DBG] 2.26 scrub starts 2024-02-19T21:13:48.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:47.748275+0000 osd.7 (osd.7) 268 : cluster [DBG] 2.26 scrub ok 2024-02-19T21:13:48.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:48.340288+0000 mon.a (mon.0) 3057 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:13:48.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:48.344507+0000 mon.a (mon.0) 3058 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-99"}]': finished 2024-02-19T21:13:48.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:48.344571+0000 mon.a (mon.0) 3059 : audit [INF] from='client.? v1:172.21.15.163:0/3221298767' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:13:48.728 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:48.344615+0000 mon.a (mon.0) 3060 : cluster [DBG] osdmap e592: 8 total, 8 up, 6 in 2024-02-19T21:13:48.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:47.030933+0000 osd.1 (osd.1) 270 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:13:48.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:47.032599+0000 osd.1 (osd.1) 271 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:13:48.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:47.275909+0000 mgr.y (mgr.14550) 516 : cluster [DBG] pgmap v885: 182 pgs: 182 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:48.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:47.619504+0000 mon.a (mon.0) 3056 : audit [INF] from='client.? v1:172.21.15.163:0/3221298767' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:13:48.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:47.723486+0000 osd.4 (osd.4) 213 : cluster [DBG] 2.25 scrub starts 2024-02-19T21:13:48.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:47.724344+0000 osd.4 (osd.4) 214 : cluster [DBG] 2.25 scrub ok 2024-02-19T21:13:48.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:47.745361+0000 osd.7 (osd.7) 267 : cluster [DBG] 2.26 scrub starts 2024-02-19T21:13:48.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:47.748275+0000 osd.7 (osd.7) 268 : cluster [DBG] 2.26 scrub ok 2024-02-19T21:13:48.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:48.340288+0000 mon.a (mon.0) 3057 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:13:48.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:48.344507+0000 mon.a (mon.0) 3058 : audit [INF] from='client.? v1:172.21.15.163:0/2969201413' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-99"}]': finished 2024-02-19T21:13:48.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:48.344571+0000 mon.a (mon.0) 3059 : audit [INF] from='client.? v1:172.21.15.163:0/3221298767' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:13:48.738 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:48.344615+0000 mon.a (mon.0) 3060 : cluster [DBG] osdmap e592: 8 total, 8 up, 6 in 2024-02-19T21:13:49.215 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:13:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:13:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-02-19T21:13:49.215 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-02-19T21:13:49.216 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:13:49.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:47.826340+0000 osd.5 (osd.5) 234 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:13:49.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:47.829254+0000 osd.5 (osd.5) 235 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:13:49.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:48.002242+0000 osd.1 (osd.1) 272 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:13:49.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:48.003436+0000 osd.1 (osd.1) 273 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:13:49.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:49.352517+0000 mon.a (mon.0) 3061 : cluster [DBG] osdmap e593: 8 total, 8 up, 6 in 2024-02-19T21:13:49.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:49.354299+0000 mon.b (mon.1) 732 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:49.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:49.360549+0000 mon.a (mon.0) 3062 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:49.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:47.826340+0000 osd.5 (osd.5) 234 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:13:49.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:47.829254+0000 osd.5 (osd.5) 235 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:13:49.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:48.002242+0000 osd.1 (osd.1) 272 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:13:49.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:48.003436+0000 osd.1 (osd.1) 273 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:13:49.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:49.352517+0000 mon.a (mon.0) 3061 : cluster [DBG] osdmap e593: 8 total, 8 up, 6 in 2024-02-19T21:13:49.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:49.354299+0000 mon.b (mon.1) 732 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:49.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:49.360549+0000 mon.a (mon.0) 3062 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:49.820 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:47.826340+0000 osd.5 (osd.5) 234 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:13:49.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:47.829254+0000 osd.5 (osd.5) 235 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:13:49.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:48.002242+0000 osd.1 (osd.1) 272 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:13:49.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:48.003436+0000 osd.1 (osd.1) 273 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:13:49.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:49.352517+0000 mon.a (mon.0) 3061 : cluster [DBG] osdmap e593: 8 total, 8 up, 6 in 2024-02-19T21:13:49.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:49.354299+0000 mon.b (mon.1) 732 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:49.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:49.360549+0000 mon.a (mon.0) 3062 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:49.902 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:49.905 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:49.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:49.931 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:49.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-19T21:13:50.006 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:50.035 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:50.056 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:50.058 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:50.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:50.159 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:50.187 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:50.208 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:50.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:50.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:13:50.312 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:50.339 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:50.437 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:50.440 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:50.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:50.541 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:13:50.570 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:50.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.623+0000 7f4074a59700 1 -- 172.21.15.163:0/1727587289 >> v1:172.21.15.163:6789/0 conn(0x7f40700ff670 legacy=0x7f4070101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:50.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.623+0000 7f4074a59700 1 -- 172.21.15.163:0/1727587289 shutdown_connections 2024-02-19T21:13:50.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.623+0000 7f4074a59700 1 -- 172.21.15.163:0/1727587289 >> 172.21.15.163:0/1727587289 conn(0x7f40700f2470 msgr2=0x7f40700f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:50.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.623+0000 7f4074a59700 1 -- 172.21.15.163:0/1727587289 shutdown_connections 2024-02-19T21:13:50.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.623+0000 7f4074a59700 1 -- 172.21.15.163:0/1727587289 wait complete. 2024-02-19T21:13:50.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.623+0000 7f4074a59700 1 librados: starting msgr at 2024-02-19T21:13:50.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.623+0000 7f4074a59700 1 librados: starting objecter 2024-02-19T21:13:50.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f4074a59700 1 Processor -- start 2024-02-19T21:13:50.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f4074a59700 1 -- start start 2024-02-19T21:13:50.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f4074a59700 1 librados: setting wanted keys 2024-02-19T21:13:50.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f4074a59700 1 librados: calling monclient init 2024-02-19T21:13:50.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f4074a59700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4070117270 con 0x7f40700ff670 2024-02-19T21:13:50.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f4074a59700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4070118470 con 0x7f40700fc870 2024-02-19T21:13:50.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f4074a59700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4070119670 con 0x7f40701031b0 2024-02-19T21:13:50.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f406dd9b700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f40700ff670 0x7f40700613e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57950/0 (socket says 172.21.15.163:57950) 2024-02-19T21:13:50.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f406dd9b700 1 -- 172.21.15.163:0/1221565508 learned_addr learned my addr 172.21.15.163:0/1221565508 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:50.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 10 client.?.objecter ms_handle_connect 0x7f40701031b0 2024-02-19T21:13:50.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:50.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 10 client.?.objecter ms_handle_connect 0x7f40700ff670 2024-02-19T21:13:50.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:50.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 10 client.?.objecter ms_handle_connect 0x7f40700fc870 2024-02-19T21:13:50.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:50.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106711053 0 0) 0x7f4070119670 con 0x7f40701031b0 2024-02-19T21:13:50.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40580033f0 con 0x7f40701031b0 2024-02-19T21:13:50.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777723104 0 0) 0x7f4070117270 con 0x7f40700ff670 2024-02-19T21:13:50.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4070119670 con 0x7f40700ff670 2024-02-19T21:13:50.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176860305 0 0) 0x7f4070118470 con 0x7f40700fc870 2024-02-19T21:13:50.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4070117270 con 0x7f40700fc870 2024-02-19T21:13:50.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 492499902 0 0) 0x7f40580033f0 con 0x7f40701031b0 2024-02-19T21:13:50.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4070118470 con 0x7f40701031b0 2024-02-19T21:13:50.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f40600019d0 con 0x7f40701031b0 2024-02-19T21:13:50.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.624+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 278982598 0 0) 0x7f4070119670 con 0x7f40700ff670 2024-02-19T21:13:50.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40580033f0 con 0x7f40700ff670 2024-02-19T21:13:50.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 483827870 0 0) 0x7f4070117270 con 0x7f40700fc870 2024-02-19T21:13:50.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4070119670 con 0x7f40700fc870 2024-02-19T21:13:50.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4064002240 con 0x7f40700ff670 2024-02-19T21:13:50.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4068001840 con 0x7f40700fc870 2024-02-19T21:13:50.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 20116341 0 0) 0x7f4070118470 con 0x7f40701031b0 2024-02-19T21:13:50.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 >> v1:172.21.15.193:6789/0 conn(0x7f40700fc870 legacy=0x7f4070060cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:50.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 >> v1:172.21.15.163:6789/0 conn(0x7f40700ff670 legacy=0x7f40700613e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:50.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407011a870 con 0x7f40701031b0 2024-02-19T21:13:50.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4070117460 con 0x7f40701031b0 2024-02-19T21:13:50.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f4074a59700 10 client.50240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:50.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4070117950 con 0x7f40701031b0 2024-02-19T21:13:50.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f4074a59700 1 librados: init done 2024-02-19T21:13:50.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f4060002340 con 0x7f40701031b0 2024-02-19T21:13:50.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f407005f700 con 0x7f40701031b0 2024-02-19T21:13:50.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.625+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4060003370 con 0x7f40701031b0 2024-02-19T21:13:50.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.626+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f4060003530 con 0x7f40701031b0 2024-02-19T21:13:50.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.626+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(594..594 src has 434..594) v4 ==== 7418+0+0 (unknown 1541227202 0 0) 0x7f40600771e0 con 0x7f40701031b0 2024-02-19T21:13:50.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.626+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 osd_map(594..594 src has 434..594) v4 2024-02-19T21:13:50.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.626+0000 7f405f7fe700 3 client.50240.objecter handle_osd_map got epochs [594,594] > 0 2024-02-19T21:13:50.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.626+0000 7f405f7fe700 3 client.50240.objecter handle_osd_map decoding full epoch 594 2024-02-19T21:13:50.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.627+0000 7f405f7fe700 20 client.50240.objecter dump_active .. 0 homeless 2024-02-19T21:13:50.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.627+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v594) v1 ==== 54+0+16 (unknown 3854534811 0 2420000192) 0x7f4060001f00 con 0x7f40701031b0 2024-02-19T21:13:50.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.627+0000 7f4074a59700 10 client.50240.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:50.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.627+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 1 2024-02-19T21:13:50.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.627+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070047540 osd=0 3 2024-02-19T21:13:50.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.627+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070047540 osd=0 3 2024-02-19T21:13:50.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.627+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070047540 osd=0 3 2024-02-19T21:13:50.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.627+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 1 2024-02-19T21:13:50.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.627+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f4070047540 osd=0 3 2024-02-19T21:13:50.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.627+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 0 1 2024-02-19T21:13:50.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.627+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070047540 osd=0 4 2024-02-19T21:13:50.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.627+0000 7f4074a59700 10 client.50240.objecter _send_command 1 2024-02-19T21:13:50.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.627+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4070108d90 con 0x7f4070118600 2024-02-19T21:13:50.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.627+0000 7f405f7fe700 10 client.50240.objecter ms_handle_connect 0x7f4058063da0 2024-02-19T21:13:50.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.627+0000 7f405f7fe700 10 client.50240.objecter ms_handle_connect 0x7f4070118600 2024-02-19T21:13:50.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.628+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4070108d90 con 0x7f4070118600 2024-02-19T21:13:50.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.628+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:13:50.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.628+0000 7f405f7fe700 10 client.50240.objecter _finish_command 1 = system:0 2024-02-19T21:13:50.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.628+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f4070047540 osd=0 4 2024-02-19T21:13:50.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.628+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 0 1 2024-02-19T21:13:50.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.643+0000 7f4074a59700 10 client.50240.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:50.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.643+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 2 2024-02-19T21:13:50.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.643+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070047540 osd=0 3 2024-02-19T21:13:50.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.643+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070047540 osd=0 3 2024-02-19T21:13:50.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.643+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070047540 osd=0 3 2024-02-19T21:13:50.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.643+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 2 2024-02-19T21:13:50.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.643+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f4070047540 osd=0 3 2024-02-19T21:13:50.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.643+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 0 2 2024-02-19T21:13:50.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.643+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070047540 osd=0 4 2024-02-19T21:13:50.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.643+0000 7f4074a59700 10 client.50240.objecter _send_command 2 2024-02-19T21:13:50.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.643+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4070119330 con 0x7f4070118600 2024-02-19T21:13:50.644 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:13:50.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.645+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4070119330 con 0x7f4070118600 2024-02-19T21:13:50.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.645+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:50.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.645+0000 7f405f7fe700 10 client.50240.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.645+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f4070047540 osd=0 4 2024-02-19T21:13:50.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.645+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 0 2 2024-02-19T21:13:50.646 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:13:50.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f4074a59700 10 client.50240.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:50.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 3 2024-02-19T21:13:50.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f40701194a0 osd=1 3 2024-02-19T21:13:50.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f40701194a0 osd=1 3 2024-02-19T21:13:50.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f40701194a0 osd=1 3 2024-02-19T21:13:50.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 3 2024-02-19T21:13:50.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f40701194a0 osd=1 3 2024-02-19T21:13:50.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 1 3 2024-02-19T21:13:50.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f40701194a0 osd=1 4 2024-02-19T21:13:50.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f4074a59700 10 client.50240.objecter _send_command 3 2024-02-19T21:13:50.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4070059cd0 con 0x7f40700598f0 2024-02-19T21:13:50.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f405f7fe700 10 client.50240.objecter ms_handle_connect 0x7f40700598f0 2024-02-19T21:13:50.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4070059cd0 con 0x7f40700598f0 2024-02-19T21:13:50.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:13:50.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f405f7fe700 10 client.50240.objecter _finish_command 3 = system:0 2024-02-19T21:13:50.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f40701194a0 osd=1 4 2024-02-19T21:13:50.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.646+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 1 3 2024-02-19T21:13:50.648 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:50.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:13:50.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.661+0000 7f4074a59700 10 client.50240.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:50.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.661+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 4 2024-02-19T21:13:50.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.661+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f40701194a0 osd=1 3 2024-02-19T21:13:50.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.661+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f40701194a0 osd=1 3 2024-02-19T21:13:50.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.661+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f40701194a0 osd=1 3 2024-02-19T21:13:50.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.661+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 4 2024-02-19T21:13:50.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.661+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f40701194a0 osd=1 3 2024-02-19T21:13:50.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.661+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 1 4 2024-02-19T21:13:50.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.661+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f40701194a0 osd=1 4 2024-02-19T21:13:50.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.661+0000 7f4074a59700 10 client.50240.objecter _send_command 4 2024-02-19T21:13:50.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.661+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f407005a0b0 con 0x7f40700598f0 2024-02-19T21:13:50.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f407005a0b0 con 0x7f40700598f0 2024-02-19T21:13:50.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:50.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f405f7fe700 10 client.50240.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f40701194a0 osd=1 4 2024-02-19T21:13:50.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 1 4 2024-02-19T21:13:50.663 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:13:50.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f4074a59700 10 client.50240.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:50.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 5 2024-02-19T21:13:50.664 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070059360 osd=2 3 2024-02-19T21:13:50.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070059360 osd=2 3 2024-02-19T21:13:50.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070059360 osd=2 3 2024-02-19T21:13:50.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 5 2024-02-19T21:13:50.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f4070059360 osd=2 3 2024-02-19T21:13:50.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 2 5 2024-02-19T21:13:50.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070059360 osd=2 4 2024-02-19T21:13:50.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f4074a59700 10 client.50240.objecter _send_command 5 2024-02-19T21:13:50.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f40700287b0 con 0x7f40700594d0 2024-02-19T21:13:50.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.663+0000 7f405f7fe700 10 client.50240.objecter ms_handle_connect 0x7f40700594d0 2024-02-19T21:13:50.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.664+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f40700287b0 con 0x7f40700594d0 2024-02-19T21:13:50.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.664+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:13:50.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.664+0000 7f405f7fe700 10 client.50240.objecter _finish_command 5 = system:0 2024-02-19T21:13:50.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.664+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f4070059360 osd=2 4 2024-02-19T21:13:50.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.664+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 2 5 2024-02-19T21:13:50.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.678+0000 7f4074a59700 10 client.50240.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:50.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.678+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 6 2024-02-19T21:13:50.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.678+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070059360 osd=2 3 2024-02-19T21:13:50.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.678+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070059360 osd=2 3 2024-02-19T21:13:50.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.678+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070059360 osd=2 3 2024-02-19T21:13:50.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.678+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 6 2024-02-19T21:13:50.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.678+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f4070059360 osd=2 3 2024-02-19T21:13:50.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.678+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 2 6 2024-02-19T21:13:50.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.678+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070059360 osd=2 4 2024-02-19T21:13:50.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.678+0000 7f4074a59700 10 client.50240.objecter _send_command 6 2024-02-19T21:13:50.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.678+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4070027d10 con 0x7f40700594d0 2024-02-19T21:13:50.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4070027d10 con 0x7f40700594d0 2024-02-19T21:13:50.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:50.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f405f7fe700 10 client.50240.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f4070059360 osd=2 4 2024-02-19T21:13:50.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 2 6 2024-02-19T21:13:50.680 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:13:50.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f4074a59700 10 client.50240.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:50.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 7 2024-02-19T21:13:50.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070028120 osd=3 3 2024-02-19T21:13:50.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070028120 osd=3 3 2024-02-19T21:13:50.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070028120 osd=3 3 2024-02-19T21:13:50.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 7 2024-02-19T21:13:50.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f4070028120 osd=3 3 2024-02-19T21:13:50.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 3 7 2024-02-19T21:13:50.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070028120 osd=3 4 2024-02-19T21:13:50.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f4074a59700 10 client.50240.objecter _send_command 7 2024-02-19T21:13:50.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4070127b40 con 0x7f40701042c0 2024-02-19T21:13:50.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.680+0000 7f405f7fe700 10 client.50240.objecter ms_handle_connect 0x7f40701042c0 2024-02-19T21:13:50.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.681+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4070127b40 con 0x7f40701042c0 2024-02-19T21:13:50.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.681+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:13:50.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.681+0000 7f405f7fe700 10 client.50240.objecter _finish_command 7 = system:0 2024-02-19T21:13:50.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.681+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f4070028120 osd=3 4 2024-02-19T21:13:50.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.681+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 3 7 2024-02-19T21:13:50.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:13:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:13:50.690 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:48.833800+0000 osd.5 (osd.5) 236 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:13:50.690 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:48.837363+0000 osd.5 (osd.5) 237 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:13:50.690 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:49.276416+0000 mgr.y (mgr.14550) 517 : cluster [DBG] pgmap v888: 182 pgs: 182 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:50.690 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:49.745807+0000 osd.7 (osd.7) 269 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:13:50.691 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:49.749129+0000 osd.7 (osd.7) 270 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:13:50.691 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:50.354739+0000 mon.a (mon.0) 3063 : audit [INF] from='client.48970 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94"}]': finished 2024-02-19T21:13:50.691 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:50.354827+0000 mon.a (mon.0) 3064 : cluster [DBG] osdmap e594: 8 total, 8 up, 6 in 2024-02-19T21:13:50.691 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:50.356018+0000 mon.b (mon.1) 733 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:50.691 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:50.356260+0000 mon.a (mon.0) 3065 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:50.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.695+0000 7f4074a59700 10 client.50240.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:50.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.695+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 8 2024-02-19T21:13:50.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.695+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070028120 osd=3 3 2024-02-19T21:13:50.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.695+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070028120 osd=3 3 2024-02-19T21:13:50.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.695+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070028120 osd=3 3 2024-02-19T21:13:50.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.695+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 8 2024-02-19T21:13:50.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.695+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f4070028120 osd=3 3 2024-02-19T21:13:50.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.695+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 3 8 2024-02-19T21:13:50.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.695+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070028120 osd=3 4 2024-02-19T21:13:50.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.695+0000 7f4074a59700 10 client.50240.objecter _send_command 8 2024-02-19T21:13:50.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.695+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4070103820 con 0x7f40701042c0 2024-02-19T21:13:50.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4070103820 con 0x7f40701042c0 2024-02-19T21:13:50.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:50.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f405f7fe700 10 client.50240.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f4070028120 osd=3 4 2024-02-19T21:13:50.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 3 8 2024-02-19T21:13:50.697 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:13:50.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f4074a59700 10 client.50240.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:50.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 9 2024-02-19T21:13:50.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f40701035e0 osd=4 3 2024-02-19T21:13:50.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f40701035e0 osd=4 3 2024-02-19T21:13:50.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f40701035e0 osd=4 3 2024-02-19T21:13:50.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 9 2024-02-19T21:13:50.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f40701035e0 osd=4 3 2024-02-19T21:13:50.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 4 9 2024-02-19T21:13:50.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f40701035e0 osd=4 4 2024-02-19T21:13:50.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f4074a59700 10 client.50240.objecter _send_command 9 2024-02-19T21:13:50.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4070103780 con 0x7f4070123f00 2024-02-19T21:13:50.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.697+0000 7f405f7fe700 10 client.50240.objecter ms_handle_connect 0x7f4070123f00 2024-02-19T21:13:50.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.698+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4070103780 con 0x7f4070123f00 2024-02-19T21:13:50.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.698+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:13:50.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.698+0000 7f405f7fe700 10 client.50240.objecter _finish_command 9 = system:0 2024-02-19T21:13:50.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.698+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f40701035e0 osd=4 4 2024-02-19T21:13:50.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.698+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 4 9 2024-02-19T21:13:50.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.713+0000 7f4074a59700 10 client.50240.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:50.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.713+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 10 2024-02-19T21:13:50.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.713+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f40701035e0 osd=4 3 2024-02-19T21:13:50.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.713+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f40701035e0 osd=4 3 2024-02-19T21:13:50.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.713+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f40701035e0 osd=4 3 2024-02-19T21:13:50.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.713+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 10 2024-02-19T21:13:50.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.713+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f40701035e0 osd=4 3 2024-02-19T21:13:50.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.713+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 4 10 2024-02-19T21:13:50.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.713+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f40701035e0 osd=4 4 2024-02-19T21:13:50.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.713+0000 7f4074a59700 10 client.50240.objecter _send_command 10 2024-02-19T21:13:50.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.713+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4070103780 con 0x7f4070123f00 2024-02-19T21:13:50.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4070103780 con 0x7f4070123f00 2024-02-19T21:13:50.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:50.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f405f7fe700 10 client.50240.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f40701035e0 osd=4 4 2024-02-19T21:13:50.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 4 10 2024-02-19T21:13:50.715 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:13:50.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f4074a59700 10 client.50240.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:50.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 11 2024-02-19T21:13:50.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070103780 osd=5 3 2024-02-19T21:13:50.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070103780 osd=5 3 2024-02-19T21:13:50.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070103780 osd=5 3 2024-02-19T21:13:50.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 11 2024-02-19T21:13:50.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f4070103780 osd=5 3 2024-02-19T21:13:50.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 5 11 2024-02-19T21:13:50.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070103780 osd=5 4 2024-02-19T21:13:50.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f4074a59700 10 client.50240.objecter _send_command 11 2024-02-19T21:13:50.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f407012a0c0 con 0x7f4070129ad0 2024-02-19T21:13:50.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.715+0000 7f405f7fe700 10 client.50240.objecter ms_handle_connect 0x7f4070129ad0 2024-02-19T21:13:50.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.716+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f407012a0c0 con 0x7f4070129ad0 2024-02-19T21:13:50.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.716+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:13:50.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.716+0000 7f405f7fe700 10 client.50240.objecter _finish_command 11 = system:0 2024-02-19T21:13:50.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.716+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f4070103780 osd=5 4 2024-02-19T21:13:50.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.716+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 5 11 2024-02-19T21:13:50.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.730+0000 7f4074a59700 10 client.50240.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:50.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.730+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 12 2024-02-19T21:13:50.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.730+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070103780 osd=5 3 2024-02-19T21:13:50.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.730+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070103780 osd=5 3 2024-02-19T21:13:50.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.730+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f4070103780 osd=5 3 2024-02-19T21:13:50.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.730+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 12 2024-02-19T21:13:50.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.730+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f4070103780 osd=5 3 2024-02-19T21:13:50.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.730+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 5 12 2024-02-19T21:13:50.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.730+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070103780 osd=5 4 2024-02-19T21:13:50.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.730+0000 7f4074a59700 10 client.50240.objecter _send_command 12 2024-02-19T21:13:50.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.730+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f407012a0c0 con 0x7f4070129ad0 2024-02-19T21:13:50.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f407012a0c0 con 0x7f4070129ad0 2024-02-19T21:13:50.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:50.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f405f7fe700 10 client.50240.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f4070103780 osd=5 4 2024-02-19T21:13:50.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 5 12 2024-02-19T21:13:50.732 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:13:50.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f4074a59700 10 client.50240.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:50.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 13 2024-02-19T21:13:50.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f407012a300 osd=6 3 2024-02-19T21:13:50.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f407012a300 osd=6 3 2024-02-19T21:13:50.733 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f407012a300 osd=6 3 2024-02-19T21:13:50.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 13 2024-02-19T21:13:50.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f407012a300 osd=6 3 2024-02-19T21:13:50.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 6 13 2024-02-19T21:13:50.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f407012a300 osd=6 4 2024-02-19T21:13:50.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f4074a59700 10 client.50240.objecter _send_command 13 2024-02-19T21:13:50.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.732+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f407012a0c0 con 0x7f4070131140 2024-02-19T21:13:50.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.733+0000 7f405f7fe700 10 client.50240.objecter ms_handle_connect 0x7f4070131140 2024-02-19T21:13:50.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.733+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f407012a0c0 con 0x7f4070131140 2024-02-19T21:13:50.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.733+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:13:50.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.733+0000 7f405f7fe700 10 client.50240.objecter _finish_command 13 = system:0 2024-02-19T21:13:50.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.733+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f407012a300 osd=6 4 2024-02-19T21:13:50.734 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.733+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 6 13 2024-02-19T21:13:50.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:48.833800+0000 osd.5 (osd.5) 236 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:13:50.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:48.837363+0000 osd.5 (osd.5) 237 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:13:50.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:49.276416+0000 mgr.y (mgr.14550) 517 : cluster [DBG] pgmap v888: 182 pgs: 182 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:50.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:49.745807+0000 osd.7 (osd.7) 269 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:13:50.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:49.749129+0000 osd.7 (osd.7) 270 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:13:50.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:50.354739+0000 mon.a (mon.0) 3063 : audit [INF] from='client.48970 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94"}]': finished 2024-02-19T21:13:50.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:50.354827+0000 mon.a (mon.0) 3064 : cluster [DBG] osdmap e594: 8 total, 8 up, 6 in 2024-02-19T21:13:50.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:50.356018+0000 mon.b (mon.1) 733 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:50.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:50.356260+0000 mon.a (mon.0) 3065 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:50.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:48.833800+0000 osd.5 (osd.5) 236 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:13:50.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:48.837363+0000 osd.5 (osd.5) 237 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:13:50.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:49.276416+0000 mgr.y (mgr.14550) 517 : cluster [DBG] pgmap v888: 182 pgs: 182 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:13:50.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:49.745807+0000 osd.7 (osd.7) 269 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:13:50.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:49.749129+0000 osd.7 (osd.7) 270 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:13:50.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:50.354739+0000 mon.a (mon.0) 3063 : audit [INF] from='client.48970 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi163-83928-94"}]': finished 2024-02-19T21:13:50.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:50.354827+0000 mon.a (mon.0) 3064 : cluster [DBG] osdmap e594: 8 total, 8 up, 6 in 2024-02-19T21:13:50.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:50.356018+0000 mon.b (mon.1) 733 : audit [INF] from='client.? v1:172.21.15.163:0/3331189123' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:50.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:50.356260+0000 mon.a (mon.0) 3065 : audit [INF] from='client.48970 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi163-83928-94"}]: dispatch 2024-02-19T21:13:50.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.748+0000 7f4074a59700 10 client.50240.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:50.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.748+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 14 2024-02-19T21:13:50.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.748+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f407012a300 osd=6 3 2024-02-19T21:13:50.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.748+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f407012a300 osd=6 3 2024-02-19T21:13:50.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.748+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f407012a300 osd=6 3 2024-02-19T21:13:50.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.748+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 14 2024-02-19T21:13:50.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.748+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f407012a300 osd=6 3 2024-02-19T21:13:50.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.748+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 6 14 2024-02-19T21:13:50.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.748+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f407012a300 osd=6 4 2024-02-19T21:13:50.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.748+0000 7f4074a59700 10 client.50240.objecter _send_command 14 2024-02-19T21:13:50.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.748+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f407012a0c0 con 0x7f4070131140 2024-02-19T21:13:50.750 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:50.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.749+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f407012a0c0 con 0x7f4070131140 2024-02-19T21:13:50.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.749+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:50.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.749+0000 7f405f7fe700 10 client.50240.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.749+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f407012a300 osd=6 4 2024-02-19T21:13:50.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.749+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 6 14 2024-02-19T21:13:50.751 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:13:50.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.750+0000 7f4074a59700 10 client.50240.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:50.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.750+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 15 2024-02-19T21:13:50.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.750+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f407012a0c0 osd=7 3 2024-02-19T21:13:50.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.750+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f407012a0c0 osd=7 3 2024-02-19T21:13:50.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.750+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f407012a0c0 osd=7 3 2024-02-19T21:13:50.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.750+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 15 2024-02-19T21:13:50.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.750+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f407012a0c0 osd=7 3 2024-02-19T21:13:50.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.750+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 7 15 2024-02-19T21:13:50.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.750+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f407012a0c0 osd=7 4 2024-02-19T21:13:50.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.750+0000 7f4074a59700 10 client.50240.objecter _send_command 15 2024-02-19T21:13:50.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.750+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4070135ee0 con 0x7f40701369c0 2024-02-19T21:13:50.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.750+0000 7f405f7fe700 10 client.50240.objecter ms_handle_connect 0x7f40701369c0 2024-02-19T21:13:50.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.751+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4070135ee0 con 0x7f40701369c0 2024-02-19T21:13:50.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.751+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:13:50.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.751+0000 7f405f7fe700 10 client.50240.objecter _finish_command 15 = system:0 2024-02-19T21:13:50.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.751+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f407012a0c0 osd=7 4 2024-02-19T21:13:50.753 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.751+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 7 15 2024-02-19T21:13:50.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.767+0000 7f4074a59700 10 client.50240.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:13:50.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.767+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign -1 16 2024-02-19T21:13:50.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.767+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f407012a0c0 osd=7 3 2024-02-19T21:13:50.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.767+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f407012a0c0 osd=7 3 2024-02-19T21:13:50.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.767+0000 7f4074a59700 20 client.50240.objecter _get_session s=0x7f407012a0c0 osd=7 3 2024-02-19T21:13:50.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.767+0000 7f4074a59700 15 client.50240.objecter _session_command_op_remove -1 16 2024-02-19T21:13:50.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.767+0000 7f4074a59700 20 client.50240.objecter get_session s=0x7f407012a0c0 osd=7 3 2024-02-19T21:13:50.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.767+0000 7f4074a59700 15 client.50240.objecter _session_command_op_assign 7 16 2024-02-19T21:13:50.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.767+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f407012a0c0 osd=7 4 2024-02-19T21:13:50.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.767+0000 7f4074a59700 10 client.50240.objecter _send_command 16 2024-02-19T21:13:50.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.767+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4070135ee0 con 0x7f40701369c0 2024-02-19T21:13:50.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.768+0000 7f405f7fe700 1 -- 172.21.15.163:0/1221565508 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4070135ee0 con 0x7f40701369c0 2024-02-19T21:13:50.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.768+0000 7f405f7fe700 10 client.50240.objecter ms_dispatch 0x7f4070001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:13:50.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.768+0000 7f405f7fe700 10 client.50240.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.768+0000 7f405f7fe700 20 client.50240.objecter put_session s=0x7f407012a0c0 osd=7 4 2024-02-19T21:13:50.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.768+0000 7f405f7fe700 15 client.50240.objecter _session_command_op_remove 7 16 2024-02-19T21:13:50.769 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:13:50.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 10 librados: watch_flush enter 2024-02-19T21:13:50.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 10 librados: watch_flush exit 2024-02-19T21:13:50.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 10 client.50240.objecter close_session for osd.0 2024-02-19T21:13:50.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 >> v1:172.21.15.163:6801/1559418115 conn(0x7f4070118600 legacy=0x7f40701189e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:50.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070047540 osd=0 1 2024-02-19T21:13:50.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 10 client.50240.objecter close_session for osd.1 2024-02-19T21:13:50.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 >> v1:172.21.15.163:6805/3125047432 conn(0x7f40700598f0 legacy=0x7f40700273c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:50.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f40701194a0 osd=1 1 2024-02-19T21:13:50.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 10 client.50240.objecter close_session for osd.2 2024-02-19T21:13:50.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 >> v1:172.21.15.163:6809/1563158487 conn(0x7f40700594d0 legacy=0x7f4070121b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:50.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070059360 osd=2 1 2024-02-19T21:13:50.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 10 client.50240.objecter close_session for osd.3 2024-02-19T21:13:50.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 >> v1:172.21.15.163:6813/2112279769 conn(0x7f40701042c0 legacy=0x7f40701263b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:50.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070028120 osd=3 1 2024-02-19T21:13:50.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 10 client.50240.objecter close_session for osd.4 2024-02-19T21:13:50.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 >> v1:172.21.15.193:6800/3129716965 conn(0x7f4070123f00 legacy=0x7f4070127ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:50.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f40701035e0 osd=4 1 2024-02-19T21:13:50.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 10 client.50240.objecter close_session for osd.5 2024-02-19T21:13:50.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 >> v1:172.21.15.193:6804/805161991 conn(0x7f4070129ad0 legacy=0x7f407012f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:50.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f4070103780 osd=5 1 2024-02-19T21:13:50.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 10 client.50240.objecter close_session for osd.6 2024-02-19T21:13:50.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 >> v1:172.21.15.193:6808/787604615 conn(0x7f4070131140 legacy=0x7f40701335b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:50.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f407012a300 osd=6 1 2024-02-19T21:13:50.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 10 client.50240.objecter close_session for osd.7 2024-02-19T21:13:50.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 >> v1:172.21.15.193:6812/157927432 conn(0x7f40701369c0 legacy=0x7f4070136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:50.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 20 client.50240.objecter put_session s=0x7f407012a0c0 osd=7 1 2024-02-19T21:13:50.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 20 client.50240.objecter shutdown clearing up homeless session... 2024-02-19T21:13:50.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 10 client.50240.objecter successfully canceled tick 2024-02-19T21:13:50.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f406ed9d700 1 -- 172.21.15.163:0/1221565508 reap_dead start 2024-02-19T21:13:50.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f406ed9d700 1 -- 172.21.15.163:0/1221565508 reap_dead start 2024-02-19T21:13:50.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4058063da0 legacy=0x7f4058066260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:50.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 >> v1:172.21.15.163:6790/0 conn(0x7f40701031b0 legacy=0x7f4070115970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:50.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f406ed9d700 1 -- 172.21.15.163:0/1221565508 reap_dead start 2024-02-19T21:13:50.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f406ed9d700 1 -- 172.21.15.163:0/1221565508 reap_dead start 2024-02-19T21:13:50.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f406ed9d700 1 -- 172.21.15.163:0/1221565508 reap_dead start 2024-02-19T21:13:50.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 shutdown_connections 2024-02-19T21:13:50.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 >> 172.21.15.163:0/1221565508 conn(0x7f40700f2470 msgr2=0x7f4070106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:50.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 shutdown_connections 2024-02-19T21:13:50.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 1 -- 172.21.15.163:0/1221565508 wait complete. 2024-02-19T21:13:50.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:50.769+0000 7f4074a59700 1 librados: shutdown 2024-02-19T21:13:50.773 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.774 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.774 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.774 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.774 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.774 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.774 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.774 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:13:50.782 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:50.980 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:50.982 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:50.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:51.043 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:13:51.084 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:51.112 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:51.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:50.600491+0000 mgr.y (mgr.14550) 518 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:51.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:50.627842+0000 mon.c (mon.2) 413 : audit [DBG] from='client.? v1:172.21.15.163:0/1221565508' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:51.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:50.721443+0000 osd.7 (osd.7) 271 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:13:51.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:50.724356+0000 osd.7 (osd.7) 272 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:13:51.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:51.216467+0000 osd.3 (osd.3) 247 : cluster [DBG] 6.c scrub starts 2024-02-19T21:13:51.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:51.217365+0000 osd.3 (osd.3) 248 : cluster [DBG] 6.c scrub ok 2024-02-19T21:13:51.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:51.354064+0000 mon.a (mon.0) 3066 : audit [INF] from='client.48970 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi163-83928-94"}]': finished 2024-02-19T21:13:51.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:51.354138+0000 mon.a (mon.0) 3067 : cluster [DBG] osdmap e595: 8 total, 8 up, 6 in 2024-02-19T21:13:51.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:51.371176+0000 mon.a (mon.0) 3068 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:13:51.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:51.372448+0000 mon.a (mon.0) 3069 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:13:51.707 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:51.372679+0000 mon.a (mon.0) 3070 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83928-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:13:51.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:50.600491+0000 mgr.y (mgr.14550) 518 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:51.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:50.627842+0000 mon.c (mon.2) 413 : audit [DBG] from='client.? v1:172.21.15.163:0/1221565508' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:51.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:50.721443+0000 osd.7 (osd.7) 271 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:13:51.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:50.724356+0000 osd.7 (osd.7) 272 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:13:51.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:51.216467+0000 osd.3 (osd.3) 247 : cluster [DBG] 6.c scrub starts 2024-02-19T21:13:51.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:51.217365+0000 osd.3 (osd.3) 248 : cluster [DBG] 6.c scrub ok 2024-02-19T21:13:51.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:51.354064+0000 mon.a (mon.0) 3066 : audit [INF] from='client.48970 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi163-83928-94"}]': finished 2024-02-19T21:13:51.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:51.354138+0000 mon.a (mon.0) 3067 : cluster [DBG] osdmap e595: 8 total, 8 up, 6 in 2024-02-19T21:13:51.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:51.371176+0000 mon.a (mon.0) 3068 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:13:51.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:51.372448+0000 mon.a (mon.0) 3069 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:13:51.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:51.372679+0000 mon.a (mon.0) 3070 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83928-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:13:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:50.600491+0000 mgr.y (mgr.14550) 518 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:13:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:50.627842+0000 mon.c (mon.2) 413 : audit [DBG] from='client.? v1:172.21.15.163:0/1221565508' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:50.721443+0000 osd.7 (osd.7) 271 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:13:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:50.724356+0000 osd.7 (osd.7) 272 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:13:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:51.216467+0000 osd.3 (osd.3) 247 : cluster [DBG] 6.c scrub starts 2024-02-19T21:13:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:51.217365+0000 osd.3 (osd.3) 248 : cluster [DBG] 6.c scrub ok 2024-02-19T21:13:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:51.354064+0000 mon.a (mon.0) 3066 : audit [INF] from='client.48970 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi163-83928-94"}]': finished 2024-02-19T21:13:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:51.354138+0000 mon.a (mon.0) 3067 : cluster [DBG] osdmap e595: 8 total, 8 up, 6 in 2024-02-19T21:13:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:51.371176+0000 mon.a (mon.0) 3068 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:13:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:51.372448+0000 mon.a (mon.0) 3069 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:13:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:51.372679+0000 mon.a (mon.0) 3070 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83928-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-02-19T21:13:51.761 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:51.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:51.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:51.866 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:51.893 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:51.965 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:51.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:51.985 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:13:51.985 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:13:52.070 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:52.098 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:52.197 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:52.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:52.302 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:52.329 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:52.454 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:52.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:52.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:52.558 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:52.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:51.276774+0000 mgr.y (mgr.14550) 519 : cluster [DBG] pgmap v891: 174 pgs: 174 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-19T21:13:52.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:51.496299+0000 osd.2 (osd.2) 240 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:13:52.570 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:51.499295+0000 osd.2 (osd.2) 241 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:13:52.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:51.503178+0000 mon.a (mon.0) 3071 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:13:52.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:52.355491+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83928-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:13:52.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:52.355560+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e596: 8 total, 8 up, 6 in 2024-02-19T21:13:52.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:52.357063+0000 mon.a (mon.0) 3074 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83928-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:13:52.586 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:52.606 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:52.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:52.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:52.624 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-19T21:13:52.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.697+0000 7fcee2c59700 1 -- 172.21.15.163:0/554352224 >> v1:172.21.15.163:6790/0 conn(0x7fcedc1031b0 legacy=0x7fcedc105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:52.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.697+0000 7fcee2c59700 1 -- 172.21.15.163:0/554352224 shutdown_connections 2024-02-19T21:13:52.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.697+0000 7fcee2c59700 1 -- 172.21.15.163:0/554352224 >> 172.21.15.163:0/554352224 conn(0x7fcedc0f2470 msgr2=0x7fcedc0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:52.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.697+0000 7fcee2c59700 1 -- 172.21.15.163:0/554352224 shutdown_connections 2024-02-19T21:13:52.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.697+0000 7fcee2c59700 1 -- 172.21.15.163:0/554352224 wait complete. 2024-02-19T21:13:52.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fcee2c59700 1 librados: starting msgr at 2024-02-19T21:13:52.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fcee2c59700 1 librados: starting objecter 2024-02-19T21:13:52.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fcee2c59700 1 Processor -- start 2024-02-19T21:13:52.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fcee2c59700 1 -- start start 2024-02-19T21:13:52.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fcee2c59700 1 librados: setting wanted keys 2024-02-19T21:13:52.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fcee2c59700 1 librados: calling monclient init 2024-02-19T21:13:52.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fcee2c59700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcedc0fbf10 con 0x7fcedc0fc870 2024-02-19T21:13:52.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fcee2c59700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcedc0fc0e0 con 0x7fcedc1031b0 2024-02-19T21:13:52.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fcee2c59700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcedc11ba80 con 0x7fcedc0ff670 2024-02-19T21:13:52.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fcee09f5700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fcedc0fc870 0x7fcedc0fb770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:57984/0 (socket says 172.21.15.163:57984) 2024-02-19T21:13:52.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fcee09f5700 1 -- 172.21.15.163:0/2200323197 learned_addr learned my addr 172.21.15.163:0/2200323197 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:52.700 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fced9ffb700 10 client.?.objecter ms_handle_connect 0x7fcedc0fc870 2024-02-19T21:13:52.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fced9ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:52.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fced9ffb700 10 client.?.objecter ms_handle_connect 0x7fcedc0ff670 2024-02-19T21:13:52.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fced9ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:52.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119530535 0 0) 0x7fcedc11ba80 con 0x7fcedc0ff670 2024-02-19T21:13:52.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcecc003800 con 0x7fcedc0ff670 2024-02-19T21:13:52.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181954023 0 0) 0x7fcedc0fbf10 con 0x7fcedc0fc870 2024-02-19T21:13:52.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.698+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcedc11ba80 con 0x7fcedc0fc870 2024-02-19T21:13:52.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 344055701 0 0) 0x7fcecc003800 con 0x7fcedc0ff670 2024-02-19T21:13:52.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcedc0fbf10 con 0x7fcedc0ff670 2024-02-19T21:13:52.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcec80018c0 con 0x7fcedc0ff670 2024-02-19T21:13:52.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2346657894 0 0) 0x7fcedc11ba80 con 0x7fcedc0fc870 2024-02-19T21:13:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcecc003800 con 0x7fcedc0fc870 2024-02-19T21:13:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fced00019f0 con 0x7fcedc0fc870 2024-02-19T21:13:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1786277163 0 0) 0x7fcedc0fbf10 con 0x7fcedc0ff670 2024-02-19T21:13:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 >> v1:172.21.15.193:6789/0 conn(0x7fcedc1031b0 legacy=0x7fcedc118340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 >> v1:172.21.15.163:6789/0 conn(0x7fcedc0fc870 legacy=0x7fcedc0fb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcedc11cc20 con 0x7fcedc0ff670 2024-02-19T21:13:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fcee2c59700 1 -- 172.21.15.163:0/2200323197 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcedc11bc70 con 0x7fcedc0ff670 2024-02-19T21:13:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fcec8002040 con 0x7fcedc0ff670 2024-02-19T21:13:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcec80032d0 con 0x7fcedc0ff670 2024-02-19T21:13:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fcee2c59700 10 client.50252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fcee2c59700 1 -- 172.21.15.163:0/2200323197 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcedc11c0e0 con 0x7fcedc0ff670 2024-02-19T21:13:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.699+0000 7fcee2c59700 1 librados: init done 2024-02-19T21:13:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.700+0000 7fcee2c59700 1 -- 172.21.15.163:0/2200323197 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcedc05f700 con 0x7fcedc0ff670 2024-02-19T21:13:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.700+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fcec8016240 con 0x7fcedc0ff670 2024-02-19T21:13:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.701+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(596..596 src has 434..596) v4 ==== 7598+0+0 (unknown 2928485040 0 0) 0x7fcec8076380 con 0x7fcedc0ff670 2024-02-19T21:13:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.701+0000 7fced9ffb700 10 client.50252.objecter ms_dispatch 0x7fcedc001550 osd_map(596..596 src has 434..596) v4 2024-02-19T21:13:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.701+0000 7fced9ffb700 3 client.50252.objecter handle_osd_map got epochs [596,596] > 0 2024-02-19T21:13:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.701+0000 7fced9ffb700 3 client.50252.objecter handle_osd_map decoding full epoch 596 2024-02-19T21:13:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.701+0000 7fced9ffb700 20 client.50252.objecter dump_active .. 0 homeless 2024-02-19T21:13:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.701+0000 7fced9ffb700 10 client.50252.objecter ms_handle_connect 0x7fcecc063b20 2024-02-19T21:13:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.705+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fcec8049980 con 0x7fcedc0ff670 2024-02-19T21:13:52.711 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:52.737 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:52.742 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:52.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:51.276774+0000 mgr.y (mgr.14550) 519 : cluster [DBG] pgmap v891: 174 pgs: 174 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-19T21:13:52.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:51.496299+0000 osd.2 (osd.2) 240 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:13:52.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:51.499295+0000 osd.2 (osd.2) 241 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:13:52.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:51.503178+0000 mon.a (mon.0) 3071 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:13:52.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:52.355491+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83928-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:13:52.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:52.355560+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e596: 8 total, 8 up, 6 in 2024-02-19T21:13:52.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:52.357063+0000 mon.a (mon.0) 3074 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83928-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:13:52.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:51.276774+0000 mgr.y (mgr.14550) 519 : cluster [DBG] pgmap v891: 174 pgs: 174 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-19T21:13:52.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:51.496299+0000 osd.2 (osd.2) 240 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:13:52.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:51.499295+0000 osd.2 (osd.2) 241 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:13:52.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:51.503178+0000 mon.a (mon.0) 3071 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:13:52.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:52.355491+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi163-83928-101", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-02-19T21:13:52.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:52.355560+0000 mon.a (mon.0) 3073 : cluster [DBG] osdmap e596: 8 total, 8 up, 6 in 2024-02-19T21:13:52.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:52.357063+0000 mon.a (mon.0) 3074 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83928-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:13:52.762 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:52.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:52.866 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:52.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:52.882+0000 7fcee2c59700 1 -- 172.21.15.163:0/2200323197 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"} v 0) v1 -- 0x7fcedc108de0 con 0x7fcedc0ff670 2024-02-19T21:13:52.893 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:52.913 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:52.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:52.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:52.942 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:13:53.017 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:53.046 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:53.065 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:53.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:53.169 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:13:53.196 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:53.222 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:13:53.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:53.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:13:53.326 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:53.357 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:53.410 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:53.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:53.497 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:13:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:13:53] "GET /metrics HTTP/1.1" 200 193528 "" "Prometheus/2.33.4" 2024-02-19T21:13:53.514 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:53.541 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:53.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:53.595+0000 7fced9ffb700 1 -- 172.21.15.163:0/2200323197 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]=0 set pool 1 pgp_num to 1 v597) v1 ==== 146+0+0 (unknown 3525521734 0 0) 0x7fcec8047d90 con 0x7fcedc0ff670 2024-02-19T21:13:53.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:53.596+0000 7fcee2c59700 10 librados: watch_flush enter 2024-02-19T21:13:53.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:53.597+0000 7fcee2c59700 10 librados: watch_flush exit 2024-02-19T21:13:53.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:53.597+0000 7fcee2c59700 20 client.50252.objecter shutdown clearing up homeless session... 2024-02-19T21:13:53.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:53.597+0000 7fcee2c59700 10 client.50252.objecter successfully canceled tick 2024-02-19T21:13:53.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:53.597+0000 7fcee2c59700 1 -- 172.21.15.163:0/2200323197 >> v1:172.21.15.163:6800/1552514681 conn(0x7fcecc063b20 legacy=0x7fcecc065fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:53.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:53.597+0000 7fcee2c59700 1 -- 172.21.15.163:0/2200323197 >> v1:172.21.15.163:6790/0 conn(0x7fcedc0ff670 legacy=0x7fcedc114c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:53.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:53.597+0000 7fcee2c59700 1 -- 172.21.15.163:0/2200323197 shutdown_connections 2024-02-19T21:13:53.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:53.597+0000 7fcee2c59700 1 -- 172.21.15.163:0/2200323197 >> 172.21.15.163:0/2200323197 conn(0x7fcedc0f2470 msgr2=0x7fcedc106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:53.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:53.597+0000 7fcee2c59700 1 -- 172.21.15.163:0/2200323197 shutdown_connections 2024-02-19T21:13:53.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:53.597+0000 7fcee2c59700 1 -- 172.21.15.163:0/2200323197 wait complete. 2024-02-19T21:13:53.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:53.597+0000 7fcee2c59700 1 librados: shutdown 2024-02-19T21:13:53.598 INFO:teuthology.orchestra.run.smithi163.stderr:set pool 1 pgp_num to 1 2024-02-19T21:13:53.689 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:53.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:53.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:53.792 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:53.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:52.700766+0000 osd.4 (osd.4) 215 : cluster [DBG] 2.22 scrub starts 2024-02-19T21:13:53.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:52.701793+0000 osd.4 (osd.4) 216 : cluster [DBG] 2.22 scrub ok 2024-02-19T21:13:53.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:52.884259+0000 mon.c (mon.2) 414 : audit [INF] from='client.? v1:172.21.15.163:0/2200323197' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-19T21:13:53.819 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:52.884564+0000 mon.a (mon.0) 3075 : audit [INF] from='client.50252 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-19T21:13:53.820 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:53.879 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:13:53.905 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:52.700766+0000 osd.4 (osd.4) 215 : cluster [DBG] 2.22 scrub starts 2024-02-19T21:13:53.905 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:52.701793+0000 osd.4 (osd.4) 216 : cluster [DBG] 2.22 scrub ok 2024-02-19T21:13:53.906 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:52.884259+0000 mon.c (mon.2) 414 : audit [INF] from='client.? v1:172.21.15.163:0/2200323197' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-19T21:13:53.906 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:52.884564+0000 mon.a (mon.0) 3075 : audit [INF] from='client.50252 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-19T21:13:53.906 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:52.700766+0000 osd.4 (osd.4) 215 : cluster [DBG] 2.22 scrub starts 2024-02-19T21:13:53.906 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:52.701793+0000 osd.4 (osd.4) 216 : cluster [DBG] 2.22 scrub ok 2024-02-19T21:13:53.906 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:52.884259+0000 mon.c (mon.2) 414 : audit [INF] from='client.? v1:172.21.15.163:0/2200323197' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-19T21:13:53.906 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:52.884564+0000 mon.a (mon.0) 3075 : audit [INF] from='client.50252 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-19T21:13:54.648 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:13:54.648 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:13:54.668 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:52.779218+0000 osd.7 (osd.7) 273 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:13:54.668 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:52.779957+0000 osd.7 (osd.7) 274 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:13:54.668 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:53.083384+0000 osd.1 (osd.1) 274 : cluster [DBG] 16.b scrub starts 2024-02-19T21:13:54.668 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:53.086969+0000 osd.1 (osd.1) 275 : cluster [DBG] 16.b scrub ok 2024-02-19T21:13:54.668 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:53.277425+0000 mgr.y (mgr.14550) 520 : cluster [DBG] pgmap v894: 174 pgs: 174 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-19T21:13:54.668 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:53.595725+0000 mon.a (mon.0) 3076 : audit [INF] from='client.50252 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]': finished 2024-02-19T21:13:54.668 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:53.595846+0000 mon.a (mon.0) 3077 : cluster [DBG] osdmap e597: 8 total, 8 up, 6 in 2024-02-19T21:13:54.669 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:54.106068+0000 mon.a (mon.0) 3078 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:54.669 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:54.343826+0000 mon.a (mon.0) 3079 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83928-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:13:54.669 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:54.343895+0000 mon.a (mon.0) 3080 : cluster [DBG] osdmap e598: 8 total, 8 up, 6 in 2024-02-19T21:13:54.670 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:54.672 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:54.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:54.774 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:54.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.780+0000 7f7004163700 1 -- 172.21.15.163:0/1117925380 >> v1:172.21.15.163:6790/0 conn(0x7f6ffc0ff690 legacy=0x7f6ffc101a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:54.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.780+0000 7f7004163700 1 -- 172.21.15.163:0/1117925380 shutdown_connections 2024-02-19T21:13:54.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.780+0000 7f7004163700 1 -- 172.21.15.163:0/1117925380 >> 172.21.15.163:0/1117925380 conn(0x7f6ffc06cc00 msgr2=0x7f6ffc0f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:54.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.780+0000 7f7004163700 1 -- 172.21.15.163:0/1117925380 shutdown_connections 2024-02-19T21:13:54.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.780+0000 7f7004163700 1 -- 172.21.15.163:0/1117925380 wait complete. 2024-02-19T21:13:54.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.780+0000 7f7004163700 1 librados: starting msgr at 2024-02-19T21:13:54.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.780+0000 7f7004163700 1 librados: starting objecter 2024-02-19T21:13:54.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.780+0000 7f7004163700 1 Processor -- start 2024-02-19T21:13:54.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f7004163700 1 -- start start 2024-02-19T21:13:54.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f7004163700 1 librados: setting wanted keys 2024-02-19T21:13:54.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f7004163700 1 librados: calling monclient init 2024-02-19T21:13:54.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f7004163700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6ffc119420 con 0x7f6ffc0ff690 2024-02-19T21:13:54.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f7004163700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6ffc11a5c0 con 0x7f6ffc0fc860 2024-02-19T21:13:54.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f7004163700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6ffc11b7c0 con 0x7f6ffc1031d0 2024-02-19T21:13:54.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f70016fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f6ffc0ff690 0x7f6ffc0f6610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:49150/0 (socket says 172.21.15.163:49150) 2024-02-19T21:13:54.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f70016fe700 1 -- 172.21.15.163:0/1456185028 learned_addr learned my addr 172.21.15.163:0/1456185028 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:54.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f6ff2ffd700 10 client.?.objecter ms_handle_connect 0x7f6ffc0fc860 2024-02-19T21:13:54.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f6ff2ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:54.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f6ff2ffd700 10 client.?.objecter ms_handle_connect 0x7f6ffc1031d0 2024-02-19T21:13:54.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f6ff2ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:54.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f6ff2ffd700 10 client.?.objecter ms_handle_connect 0x7f6ffc0ff690 2024-02-19T21:13:54.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f6ff2ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:54.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181444307 0 0) 0x7f6ffc11a5c0 con 0x7f6ffc0fc860 2024-02-19T21:13:54.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe8003030 con 0x7f6ffc0fc860 2024-02-19T21:13:54.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225565616 0 0) 0x7f6ffc11b7c0 con 0x7f6ffc1031d0 2024-02-19T21:13:54.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ffc11a5c0 con 0x7f6ffc1031d0 2024-02-19T21:13:54.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290873687 0 0) 0x7f6ffc119420 con 0x7f6ffc0ff690 2024-02-19T21:13:54.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ffc11b7c0 con 0x7f6ffc0ff690 2024-02-19T21:13:54.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 12408558 0 0) 0x7f6fe8003030 con 0x7f6ffc0fc860 2024-02-19T21:13:54.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ffc119420 con 0x7f6ffc0fc860 2024-02-19T21:13:54.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.781+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6ff80019d0 con 0x7f6ffc0fc860 2024-02-19T21:13:54.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.782+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1312976815 0 0) 0x7f6ffc11a5c0 con 0x7f6ffc1031d0 2024-02-19T21:13:54.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.782+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fe8003030 con 0x7f6ffc1031d0 2024-02-19T21:13:54.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.782+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6ff4001bc0 con 0x7f6ffc1031d0 2024-02-19T21:13:54.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.782+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2923398352 0 0) 0x7f6ffc119420 con 0x7f6ffc0fc860 2024-02-19T21:13:54.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.782+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 >> v1:172.21.15.163:6790/0 conn(0x7f6ffc1031d0 legacy=0x7f6ffc117c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:54.784 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.782+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 >> v1:172.21.15.163:6789/0 conn(0x7f6ffc0ff690 legacy=0x7f6ffc0f6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:54.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.782+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc11c9c0 con 0x7f6ffc0fc860 2024-02-19T21:13:54.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.782+0000 7f7004163700 1 -- 172.21.15.163:0/1456185028 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ffc11b9b0 con 0x7f6ffc0fc860 2024-02-19T21:13:54.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.782+0000 7f7004163700 10 client.49063.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:54.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.782+0000 7f7004163700 1 -- 172.21.15.163:0/1456185028 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ffc11be20 con 0x7f6ffc0fc860 2024-02-19T21:13:54.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.782+0000 7f7004163700 1 librados: init done 2024-02-19T21:13:54.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.782+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6ff8002280 con 0x7f6ffc0fc860 2024-02-19T21:13:54.785 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.782+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6ff8003330 con 0x7f6ffc0fc860 2024-02-19T21:13:54.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.783+0000 7f7004163700 1 -- 172.21.15.163:0/1456185028 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe0004f20 con 0x7f6ffc0fc860 2024-02-19T21:13:54.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.783+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6ff8003490 con 0x7f6ffc0fc860 2024-02-19T21:13:54.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.783+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(598..598 src has 434..598) v4 ==== 8127+0+0 (unknown 109051598 0 0) 0x7f6ff80764f0 con 0x7f6ffc0fc860 2024-02-19T21:13:54.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.783+0000 7f6ff2ffd700 10 client.49063.objecter ms_dispatch 0x7f6ffc001550 osd_map(598..598 src has 434..598) v4 2024-02-19T21:13:54.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.783+0000 7f6ff2ffd700 3 client.49063.objecter handle_osd_map got epochs [598,598] > 0 2024-02-19T21:13:54.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.783+0000 7f6ff2ffd700 3 client.49063.objecter handle_osd_map decoding full epoch 598 2024-02-19T21:13:54.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.784+0000 7f6ff2ffd700 20 client.49063.objecter dump_active .. 0 homeless 2024-02-19T21:13:54.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.784+0000 7f6ff2ffd700 10 client.49063.objecter ms_handle_connect 0x7f6fe80639f0 2024-02-19T21:13:54.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.788+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6ff80498a0 con 0x7f6ffc0fc860 2024-02-19T21:13:54.801 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:52.779218+0000 osd.7 (osd.7) 273 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:13:54.801 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:52.779957+0000 osd.7 (osd.7) 274 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:13:54.801 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:53.083384+0000 osd.1 (osd.1) 274 : cluster [DBG] 16.b scrub starts 2024-02-19T21:13:54.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:53.086969+0000 osd.1 (osd.1) 275 : cluster [DBG] 16.b scrub ok 2024-02-19T21:13:54.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:53.277425+0000 mgr.y (mgr.14550) 520 : cluster [DBG] pgmap v894: 174 pgs: 174 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-19T21:13:54.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:53.595725+0000 mon.a (mon.0) 3076 : audit [INF] from='client.50252 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]': finished 2024-02-19T21:13:54.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:53.595846+0000 mon.a (mon.0) 3077 : cluster [DBG] osdmap e597: 8 total, 8 up, 6 in 2024-02-19T21:13:54.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:54.106068+0000 mon.a (mon.0) 3078 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:54.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:54.343826+0000 mon.a (mon.0) 3079 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83928-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:13:54.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:54.343895+0000 mon.a (mon.0) 3080 : cluster [DBG] osdmap e598: 8 total, 8 up, 6 in 2024-02-19T21:13:54.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:52.779218+0000 osd.7 (osd.7) 273 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:13:54.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:52.779957+0000 osd.7 (osd.7) 274 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:13:54.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:53.083384+0000 osd.1 (osd.1) 274 : cluster [DBG] 16.b scrub starts 2024-02-19T21:13:54.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:53.086969+0000 osd.1 (osd.1) 275 : cluster [DBG] 16.b scrub ok 2024-02-19T21:13:54.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:53.277425+0000 mgr.y (mgr.14550) 520 : cluster [DBG] pgmap v894: 174 pgs: 174 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-02-19T21:13:54.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:53.595725+0000 mon.a (mon.0) 3076 : audit [INF] from='client.50252 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num", "val": "1"}]': finished 2024-02-19T21:13:54.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:53.595846+0000 mon.a (mon.0) 3077 : cluster [DBG] osdmap e597: 8 total, 8 up, 6 in 2024-02-19T21:13:54.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:54.106068+0000 mon.a (mon.0) 3078 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:13:54.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:54.343826+0000 mon.a (mon.0) 3079 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi163-83928-101", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:13:54.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:54.343895+0000 mon.a (mon.0) 3080 : cluster [DBG] osdmap e598: 8 total, 8 up, 6 in 2024-02-19T21:13:54.804 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:54.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:54.977+0000 7f7004163700 1 -- 172.21.15.163:0/1456185028 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f6fe00051d0 con 0x7f6ffc0fc860 2024-02-19T21:13:55.162 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:55.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:55.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:55.184 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:55.246 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-19T21:13:55.266 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:55.316 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:55.340 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:55.346 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:55.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:55.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:55.351+0000 7f6ff2ffd700 1 -- 172.21.15.163:0/1456185028 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v599) v1 ==== 87+0+0 (unknown 2187300740 0 0) 0x7f6ff8047cb0 con 0x7f6ffc0fc860 2024-02-19T21:13:55.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:55.354+0000 7f7004163700 10 librados: watch_flush enter 2024-02-19T21:13:55.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:55.354+0000 7f7004163700 10 librados: watch_flush exit 2024-02-19T21:13:55.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:55.354+0000 7f7004163700 20 client.49063.objecter shutdown clearing up homeless session... 2024-02-19T21:13:55.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:55.354+0000 7f7004163700 10 client.49063.objecter successfully canceled tick 2024-02-19T21:13:55.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:55.354+0000 7f7004163700 1 -- 172.21.15.163:0/1456185028 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6fe80639f0 legacy=0x7f6fe8065eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:55.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:55.354+0000 7f7004163700 1 -- 172.21.15.163:0/1456185028 >> v1:172.21.15.193:6789/0 conn(0x7f6ffc0fc860 legacy=0x7f6ffc0f5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:55.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:55.355+0000 7f7004163700 1 -- 172.21.15.163:0/1456185028 shutdown_connections 2024-02-19T21:13:55.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:55.355+0000 7f7004163700 1 -- 172.21.15.163:0/1456185028 >> 172.21.15.163:0/1456185028 conn(0x7f6ffc06cc00 msgr2=0x7f6ffc106660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:55.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:55.355+0000 7f7004163700 1 -- 172.21.15.163:0/1456185028 shutdown_connections 2024-02-19T21:13:55.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:55.355+0000 7f7004163700 1 -- 172.21.15.163:0/1456185028 wait complete. 2024-02-19T21:13:55.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:55.356+0000 7f7004163700 1 librados: shutdown 2024-02-19T21:13:55.356 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:13:55.448 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:55.475 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:55.495 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:55.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:55.530 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:13:55.599 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:55.626 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:55.745 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:53.916133+0000 osd.5 (osd.5) 238 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:13:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:53.919021+0000 osd.5 (osd.5) 239 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:13:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:54.642562+0000 osd.4 (osd.4) 217 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:13:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:54.643352+0000 osd.4 (osd.4) 218 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:13:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:54.978924+0000 mon.b (mon.1) 734 : audit [INF] from='client.? v1:172.21.15.163:0/1456185028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:13:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:54.979218+0000 mon.a (mon.0) 3081 : audit [INF] from='client.49063 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:13:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:55.340755+0000 mon.a (mon.0) 3082 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:13:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:55.345169+0000 mon.a (mon.0) 3083 : audit [INF] from='client.49063 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:13:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:55.345236+0000 mon.a (mon.0) 3084 : cluster [DBG] osdmap e599: 8 total, 8 up, 6 in 2024-02-19T21:13:55.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:55.348314+0000 mon.a (mon.0) 3085 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:55.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:53.916133+0000 osd.5 (osd.5) 238 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:13:55.746 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:53.919021+0000 osd.5 (osd.5) 239 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:13:55.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:54.642562+0000 osd.4 (osd.4) 217 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:13:55.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:54.643352+0000 osd.4 (osd.4) 218 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:13:55.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:54.978924+0000 mon.b (mon.1) 734 : audit [INF] from='client.? v1:172.21.15.163:0/1456185028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:13:55.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:54.979218+0000 mon.a (mon.0) 3081 : audit [INF] from='client.49063 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:13:55.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:55.340755+0000 mon.a (mon.0) 3082 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:13:55.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:55.345169+0000 mon.a (mon.0) 3083 : audit [INF] from='client.49063 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:13:55.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:55.345236+0000 mon.a (mon.0) 3084 : cluster [DBG] osdmap e599: 8 total, 8 up, 6 in 2024-02-19T21:13:55.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:55.348314+0000 mon.a (mon.0) 3085 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:55.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:53.916133+0000 osd.5 (osd.5) 238 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:13:55.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:53.919021+0000 osd.5 (osd.5) 239 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:13:55.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:54.642562+0000 osd.4 (osd.4) 217 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:13:55.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:54.643352+0000 osd.4 (osd.4) 218 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:13:55.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:54.978924+0000 mon.b (mon.1) 734 : audit [INF] from='client.? v1:172.21.15.163:0/1456185028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:13:55.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:54.979218+0000 mon.a (mon.0) 3081 : audit [INF] from='client.49063 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:13:55.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:55.340755+0000 mon.a (mon.0) 3082 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:13:55.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:55.345169+0000 mon.a (mon.0) 3083 : audit [INF] from='client.49063 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:13:55.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:55.345236+0000 mon.a (mon.0) 3084 : cluster [DBG] osdmap e599: 8 total, 8 up, 6 in 2024-02-19T21:13:55.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:55.348314+0000 mon.a (mon.0) 3085 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:13:55.824 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:55.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:55.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:55.928 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:55.956 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:56.080 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:56.082 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:56.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:56.184 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:56.212 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:56.309 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:56.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:56.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:56.413 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:56.441 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:56.513 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:56.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:56.624 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:56.692 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:56.992 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:54.925427+0000 osd.5 (osd.5) 240 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:13:56.992 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:54.928313+0000 osd.5 (osd.5) 241 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:13:56.992 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:55.033397+0000 osd.1 (osd.1) 276 : cluster [DBG] 2.24 scrub starts 2024-02-19T21:13:56.992 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:55.034619+0000 osd.1 (osd.1) 277 : cluster [DBG] 2.24 scrub ok 2024-02-19T21:13:56.992 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:55.278370+0000 mgr.y (mgr.14550) 521 : cluster [DBG] pgmap v897: 182 pgs: 8 creating+peering, 174 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:56.993 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:56.345572+0000 mon.a (mon.0) 3086 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:13:56.993 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:56.345685+0000 mon.a (mon.0) 3087 : cluster [DBG] osdmap e600: 8 total, 8 up, 6 in 2024-02-19T21:13:56.993 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:56.367255+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-102", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:13:56.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:54.925427+0000 osd.5 (osd.5) 240 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:13:56.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:54.928313+0000 osd.5 (osd.5) 241 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:13:56.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:55.033397+0000 osd.1 (osd.1) 276 : cluster [DBG] 2.24 scrub starts 2024-02-19T21:13:56.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:55.034619+0000 osd.1 (osd.1) 277 : cluster [DBG] 2.24 scrub ok 2024-02-19T21:13:56.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:55.278370+0000 mgr.y (mgr.14550) 521 : cluster [DBG] pgmap v897: 182 pgs: 8 creating+peering, 174 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:56.997 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:56.345572+0000 mon.a (mon.0) 3086 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:13:56.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:56.345685+0000 mon.a (mon.0) 3087 : cluster [DBG] osdmap e600: 8 total, 8 up, 6 in 2024-02-19T21:13:56.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:56.367255+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-102", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:13:56.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:54.925427+0000 osd.5 (osd.5) 240 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:13:56.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:54.928313+0000 osd.5 (osd.5) 241 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:13:56.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:55.033397+0000 osd.1 (osd.1) 276 : cluster [DBG] 2.24 scrub starts 2024-02-19T21:13:56.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:55.034619+0000 osd.1 (osd.1) 277 : cluster [DBG] 2.24 scrub ok 2024-02-19T21:13:56.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:55.278370+0000 mgr.y (mgr.14550) 521 : cluster [DBG] pgmap v897: 182 pgs: 8 creating+peering, 174 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:56.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:56.345572+0000 mon.a (mon.0) 3086 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:13:56.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:56.345685+0000 mon.a (mon.0) 3087 : cluster [DBG] osdmap e600: 8 total, 8 up, 6 in 2024-02-19T21:13:56.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:56.367255+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-102", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:13:57.140 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:57.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:57.245 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:13:57.272 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:57.292 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:13:57.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:57.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:13:57.294 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:13:57.295 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:13:57.395 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:57.423 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:57.548 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:57.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:57.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:57.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.633+0000 7f52d0a14700 1 -- 172.21.15.163:0/4120041404 >> v1:172.21.15.163:6790/0 conn(0x7f52cc100780 legacy=0x7f52cc102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:57.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.633+0000 7f52d0a14700 1 -- 172.21.15.163:0/4120041404 shutdown_connections 2024-02-19T21:13:57.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.633+0000 7f52d0a14700 1 -- 172.21.15.163:0/4120041404 >> 172.21.15.163:0/4120041404 conn(0x7f52cc0f35a0 msgr2=0x7f52cc0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:57.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.633+0000 7f52d0a14700 1 -- 172.21.15.163:0/4120041404 shutdown_connections 2024-02-19T21:13:57.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.633+0000 7f52d0a14700 1 -- 172.21.15.163:0/4120041404 wait complete. 2024-02-19T21:13:57.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.633+0000 7f52d0a14700 1 librados: starting msgr at 2024-02-19T21:13:57.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.633+0000 7f52d0a14700 1 librados: starting objecter 2024-02-19T21:13:57.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.633+0000 7f52d0a14700 1 Processor -- start 2024-02-19T21:13:57.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52d0a14700 1 -- start start 2024-02-19T21:13:57.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52d0a14700 1 librados: setting wanted keys 2024-02-19T21:13:57.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52d0a14700 1 librados: calling monclient init 2024-02-19T21:13:57.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52d0a14700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f52cc117280 con 0x7f52cc0fd980 2024-02-19T21:13:57.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52d0a14700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f52cc118460 con 0x7f52cc100780 2024-02-19T21:13:57.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52d0a14700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f52cc119660 con 0x7f52cc1042c0 2024-02-19T21:13:57.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52ca59c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f52cc0fd980 0x7f52cc0f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:49170/0 (socket says 172.21.15.163:49170) 2024-02-19T21:13:57.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52ca59c700 1 -- 172.21.15.163:0/2656565702 learned_addr learned my addr 172.21.15.163:0/2656565702 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:13:57.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 10 client.?.objecter ms_handle_connect 0x7f52cc1042c0 2024-02-19T21:13:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 10 client.?.objecter ms_handle_connect 0x7f52cc0fd980 2024-02-19T21:13:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196930446 0 0) 0x7f52cc117280 con 0x7f52cc0fd980 2024-02-19T21:13:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b4003800 con 0x7f52cc0fd980 2024-02-19T21:13:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906683875 0 0) 0x7f52cc119660 con 0x7f52cc1042c0 2024-02-19T21:13:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52cc117280 con 0x7f52cc1042c0 2024-02-19T21:13:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 10 client.?.objecter ms_handle_connect 0x7f52cc100780 2024-02-19T21:13:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:13:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813695401 0 0) 0x7f52cc118460 con 0x7f52cc100780 2024-02-19T21:13:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52cc119660 con 0x7f52cc100780 2024-02-19T21:13:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 699434371 0 0) 0x7f52b4003800 con 0x7f52cc0fd980 2024-02-19T21:13:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52cc118460 con 0x7f52cc0fd980 2024-02-19T21:13:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f52bc001980 con 0x7f52cc0fd980 2024-02-19T21:13:57.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2649427198 0 0) 0x7f52cc117280 con 0x7f52cc1042c0 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52b4003800 con 0x7f52cc1042c0 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f52c4001c50 con 0x7f52cc1042c0 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3896360611 0 0) 0x7f52cc119660 con 0x7f52cc100780 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52cc117280 con 0x7f52cc100780 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.634+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f52c0002ea0 con 0x7f52cc100780 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.635+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 630801330 0 0) 0x7f52cc118460 con 0x7f52cc0fd980 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.635+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 >> v1:172.21.15.163:6790/0 conn(0x7f52cc1042c0 legacy=0x7f52cc115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.635+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 >> v1:172.21.15.193:6789/0 conn(0x7f52cc100780 legacy=0x7f52cc0fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.635+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc11a860 con 0x7f52cc0fd980 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.635+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52cc117470 con 0x7f52cc0fd980 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.635+0000 7f52d0a14700 10 client.23535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.635+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52cc117960 con 0x7f52cc0fd980 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.635+0000 7f52d0a14700 1 librados: init done 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.635+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f52bc002040 con 0x7f52cc0fd980 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.635+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f52bc005a40 con 0x7f52cc0fd980 2024-02-19T21:13:57.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.635+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f52ac004f00 con 0x7f52cc0fd980 2024-02-19T21:13:57.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.636+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f52bc001ba0 con 0x7f52cc0fd980 2024-02-19T21:13:57.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.636+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(601..601 src has 434..601) v4 ==== 8508+0+0 (unknown 1575694870 0 0) 0x7f52bc078e70 con 0x7f52cc0fd980 2024-02-19T21:13:57.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.636+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 osd_map(601..601 src has 434..601) v4 2024-02-19T21:13:57.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.636+0000 7f52bb7fe700 3 client.23535.objecter handle_osd_map got epochs [601,601] > 0 2024-02-19T21:13:57.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.636+0000 7f52bb7fe700 3 client.23535.objecter handle_osd_map decoding full epoch 601 2024-02-19T21:13:57.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.637+0000 7f52bb7fe700 20 client.23535.objecter dump_active .. 0 homeless 2024-02-19T21:13:57.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.637+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v601) v1 ==== 54+0+16 (unknown 2917459496 0 2420000192) 0x7f52bc07b260 con 0x7f52cc0fd980 2024-02-19T21:13:57.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.637+0000 7f52bb7fe700 10 client.23535.objecter ms_handle_connect 0x7f52b4063d10 2024-02-19T21:13:57.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.637+0000 7f52d0a14700 10 client.23535.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:57.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.637+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 1 2024-02-19T21:13:57.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.637+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac0051a0 osd=0 3 2024-02-19T21:13:57.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.637+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac0051a0 osd=0 3 2024-02-19T21:13:57.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.637+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac0051a0 osd=0 3 2024-02-19T21:13:57.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.637+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 1 2024-02-19T21:13:57.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.637+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac0051a0 osd=0 3 2024-02-19T21:13:57.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.637+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 0 1 2024-02-19T21:13:57.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.637+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac0051a0 osd=0 4 2024-02-19T21:13:57.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.637+0000 7f52d0a14700 10 client.23535.objecter _send_command 1 2024-02-19T21:13:57.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.637+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52ac004950 con 0x7f52ac004530 2024-02-19T21:13:57.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.637+0000 7f52bb7fe700 10 client.23535.objecter ms_handle_connect 0x7f52ac004530 2024-02-19T21:13:57.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.638+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f52ac004950 con 0x7f52ac004530 2024-02-19T21:13:57.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.638+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:13:57.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.638+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 1 = system:0 2024-02-19T21:13:57.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.638+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac0051a0 osd=0 4 2024-02-19T21:13:57.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.638+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 0 1 2024-02-19T21:13:57.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.652+0000 7f52d0a14700 10 client.23535.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:57.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.652+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 2 2024-02-19T21:13:57.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.652+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac0051a0 osd=0 3 2024-02-19T21:13:57.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.652+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac0051a0 osd=0 3 2024-02-19T21:13:57.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.652+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac0051a0 osd=0 3 2024-02-19T21:13:57.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.652+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 2 2024-02-19T21:13:57.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.652+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac0051a0 osd=0 3 2024-02-19T21:13:57.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.652+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 0 2 2024-02-19T21:13:57.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.652+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac0051a0 osd=0 4 2024-02-19T21:13:57.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.652+0000 7f52d0a14700 10 client.23535.objecter _send_command 2 2024-02-19T21:13:57.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.652+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f52ac004d30 con 0x7f52ac004530 2024-02-19T21:13:57.653 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:57.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f52ac004d30 con 0x7f52ac004530 2024-02-19T21:13:57.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:57.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac0051a0 osd=0 4 2024-02-19T21:13:57.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 0 2 2024-02-19T21:13:57.655 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:13:57.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52d0a14700 10 client.23535.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:57.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 3 2024-02-19T21:13:57.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac003fe0 osd=1 3 2024-02-19T21:13:57.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac003fe0 osd=1 3 2024-02-19T21:13:57.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac003fe0 osd=1 3 2024-02-19T21:13:57.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 3 2024-02-19T21:13:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac003fe0 osd=1 3 2024-02-19T21:13:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 1 3 2024-02-19T21:13:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac003fe0 osd=1 4 2024-02-19T21:13:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52d0a14700 10 client.23535.objecter _send_command 3 2024-02-19T21:13:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52ac00c000 con 0x7f52ac004150 2024-02-19T21:13:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.654+0000 7f52bb7fe700 10 client.23535.objecter ms_handle_connect 0x7f52ac004150 2024-02-19T21:13:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.655+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f52ac00c000 con 0x7f52ac004150 2024-02-19T21:13:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.655+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:13:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.655+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 3 = system:0 2024-02-19T21:13:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.655+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac003fe0 osd=1 4 2024-02-19T21:13:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.655+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 1 3 2024-02-19T21:13:57.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.669+0000 7f52d0a14700 10 client.23535.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:57.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.669+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 4 2024-02-19T21:13:57.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.669+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac003fe0 osd=1 3 2024-02-19T21:13:57.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.669+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac003fe0 osd=1 3 2024-02-19T21:13:57.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.669+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac003fe0 osd=1 3 2024-02-19T21:13:57.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.669+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 4 2024-02-19T21:13:57.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.669+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac003fe0 osd=1 3 2024-02-19T21:13:57.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.669+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 1 4 2024-02-19T21:13:57.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.669+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac003fe0 osd=1 4 2024-02-19T21:13:57.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.669+0000 7f52d0a14700 10 client.23535.objecter _send_command 4 2024-02-19T21:13:57.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.669+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f52ac001850 con 0x7f52ac004150 2024-02-19T21:13:57.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f52ac001850 con 0x7f52ac004150 2024-02-19T21:13:57.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:57.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac003fe0 osd=1 4 2024-02-19T21:13:57.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 1 4 2024-02-19T21:13:57.671 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:13:57.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52d0a14700 10 client.23535.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:57.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 5 2024-02-19T21:13:57.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac003630 osd=2 3 2024-02-19T21:13:57.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac003630 osd=2 3 2024-02-19T21:13:57.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac003630 osd=2 3 2024-02-19T21:13:57.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 5 2024-02-19T21:13:57.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac003630 osd=2 3 2024-02-19T21:13:57.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 2 5 2024-02-19T21:13:57.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac003630 osd=2 4 2024-02-19T21:13:57.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52d0a14700 10 client.23535.objecter _send_command 5 2024-02-19T21:13:57.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52ac003e70 con 0x7f52ac00d2f0 2024-02-19T21:13:57.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.671+0000 7f52bb7fe700 10 client.23535.objecter ms_handle_connect 0x7f52ac00d2f0 2024-02-19T21:13:57.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.672+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f52ac003e70 con 0x7f52ac00d2f0 2024-02-19T21:13:57.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.672+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:13:57.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.672+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 5 = system:0 2024-02-19T21:13:57.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.672+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac003630 osd=2 4 2024-02-19T21:13:57.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.672+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 2 5 2024-02-19T21:13:57.682 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:57.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.686+0000 7f52d0a14700 10 client.23535.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:57.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.686+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 6 2024-02-19T21:13:57.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.686+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac003630 osd=2 3 2024-02-19T21:13:57.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.686+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac003630 osd=2 3 2024-02-19T21:13:57.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.686+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac003630 osd=2 3 2024-02-19T21:13:57.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.686+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 6 2024-02-19T21:13:57.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.686+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac003630 osd=2 3 2024-02-19T21:13:57.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.686+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 2 6 2024-02-19T21:13:57.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.686+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac003630 osd=2 4 2024-02-19T21:13:57.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.686+0000 7f52d0a14700 10 client.23535.objecter _send_command 6 2024-02-19T21:13:57.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.686+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f52ac00e6c0 con 0x7f52ac00d2f0 2024-02-19T21:13:57.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f52ac00e6c0 con 0x7f52ac00d2f0 2024-02-19T21:13:57.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:57.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac003630 osd=2 4 2024-02-19T21:13:57.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 2 6 2024-02-19T21:13:57.688 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:13:57.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52d0a14700 10 client.23535.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:57.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 7 2024-02-19T21:13:57.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac0033d0 osd=3 3 2024-02-19T21:13:57.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac0033d0 osd=3 3 2024-02-19T21:13:57.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac0033d0 osd=3 3 2024-02-19T21:13:57.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 7 2024-02-19T21:13:57.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac0033d0 osd=3 3 2024-02-19T21:13:57.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 3 7 2024-02-19T21:13:57.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac0033d0 osd=3 4 2024-02-19T21:13:57.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52d0a14700 10 client.23535.objecter _send_command 7 2024-02-19T21:13:57.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.687+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52ac015aa0 con 0x7f52ac011e80 2024-02-19T21:13:57.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.688+0000 7f52bb7fe700 10 client.23535.objecter ms_handle_connect 0x7f52ac011e80 2024-02-19T21:13:57.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.688+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f52ac015aa0 con 0x7f52ac011e80 2024-02-19T21:13:57.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.688+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:13:57.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.688+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 7 = system:0 2024-02-19T21:13:57.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.688+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac0033d0 osd=3 4 2024-02-19T21:13:57.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.688+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 3 7 2024-02-19T21:13:57.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.702+0000 7f52d0a14700 10 client.23535.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:57.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.702+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 8 2024-02-19T21:13:57.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.702+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac0033d0 osd=3 3 2024-02-19T21:13:57.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.702+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac0033d0 osd=3 3 2024-02-19T21:13:57.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.702+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac0033d0 osd=3 3 2024-02-19T21:13:57.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.702+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 8 2024-02-19T21:13:57.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.702+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac0033d0 osd=3 3 2024-02-19T21:13:57.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.702+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 3 8 2024-02-19T21:13:57.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.702+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac0033d0 osd=3 4 2024-02-19T21:13:57.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.702+0000 7f52d0a14700 10 client.23535.objecter _send_command 8 2024-02-19T21:13:57.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.702+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f52ac00e100 con 0x7f52ac011e80 2024-02-19T21:13:57.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f52ac00e100 con 0x7f52ac011e80 2024-02-19T21:13:57.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:57.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac0033d0 osd=3 4 2024-02-19T21:13:57.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 3 8 2024-02-19T21:13:57.704 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:13:57.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52d0a14700 10 client.23535.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:57.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 9 2024-02-19T21:13:57.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac00e300 osd=4 3 2024-02-19T21:13:57.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac00e300 osd=4 3 2024-02-19T21:13:57.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac00e300 osd=4 3 2024-02-19T21:13:57.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 9 2024-02-19T21:13:57.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac00e300 osd=4 3 2024-02-19T21:13:57.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 4 9 2024-02-19T21:13:57.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac00e300 osd=4 4 2024-02-19T21:13:57.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52d0a14700 10 client.23535.objecter _send_command 9 2024-02-19T21:13:57.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.704+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52ac00e4a0 con 0x7f52ac00ce50 2024-02-19T21:13:57.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.705+0000 7f52bb7fe700 10 client.23535.objecter ms_handle_connect 0x7f52ac00ce50 2024-02-19T21:13:57.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.705+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f52ac00e4a0 con 0x7f52ac00ce50 2024-02-19T21:13:57.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.705+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:13:57.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.705+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 9 = system:0 2024-02-19T21:13:57.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.705+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac00e300 osd=4 4 2024-02-19T21:13:57.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.705+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 4 9 2024-02-19T21:13:57.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.720+0000 7f52d0a14700 10 client.23535.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:57.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.720+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 10 2024-02-19T21:13:57.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.720+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac00e300 osd=4 3 2024-02-19T21:13:57.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.720+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac00e300 osd=4 3 2024-02-19T21:13:57.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.720+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac00e300 osd=4 3 2024-02-19T21:13:57.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.720+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 10 2024-02-19T21:13:57.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.720+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac00e300 osd=4 3 2024-02-19T21:13:57.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.720+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 4 10 2024-02-19T21:13:57.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.720+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac00e300 osd=4 4 2024-02-19T21:13:57.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.720+0000 7f52d0a14700 10 client.23535.objecter _send_command 10 2024-02-19T21:13:57.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.720+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f52ac00e4a0 con 0x7f52ac00ce50 2024-02-19T21:13:57.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.721+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f52ac00e4a0 con 0x7f52ac00ce50 2024-02-19T21:13:57.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.721+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:57.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.721+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.721+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac00e300 osd=4 4 2024-02-19T21:13:57.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.721+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 4 10 2024-02-19T21:13:57.722 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:13:57.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.722+0000 7f52d0a14700 10 client.23535.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:57.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.722+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 11 2024-02-19T21:13:57.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.722+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac00e4a0 osd=5 3 2024-02-19T21:13:57.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.722+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac00e4a0 osd=5 3 2024-02-19T21:13:57.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.722+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac00e4a0 osd=5 3 2024-02-19T21:13:57.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.722+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 11 2024-02-19T21:13:57.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.722+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac00e4a0 osd=5 3 2024-02-19T21:13:57.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.722+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 5 11 2024-02-19T21:13:57.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.722+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac00e4a0 osd=5 4 2024-02-19T21:13:57.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.722+0000 7f52d0a14700 10 client.23535.objecter _send_command 11 2024-02-19T21:13:57.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.722+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52ac017fd0 con 0x7f52ac0179e0 2024-02-19T21:13:57.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.722+0000 7f52bb7fe700 10 client.23535.objecter ms_handle_connect 0x7f52ac0179e0 2024-02-19T21:13:57.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.723+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f52ac017fd0 con 0x7f52ac0179e0 2024-02-19T21:13:57.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.723+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:13:57.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.723+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 11 = system:0 2024-02-19T21:13:57.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.723+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac00e4a0 osd=5 4 2024-02-19T21:13:57.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.723+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 5 11 2024-02-19T21:13:57.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.737+0000 7f52d0a14700 10 client.23535.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:57.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.737+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 12 2024-02-19T21:13:57.737 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.737+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac00e4a0 osd=5 3 2024-02-19T21:13:57.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.737+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac00e4a0 osd=5 3 2024-02-19T21:13:57.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.737+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac00e4a0 osd=5 3 2024-02-19T21:13:57.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.737+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 12 2024-02-19T21:13:57.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.737+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac00e4a0 osd=5 3 2024-02-19T21:13:57.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.737+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 5 12 2024-02-19T21:13:57.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.737+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac00e4a0 osd=5 4 2024-02-19T21:13:57.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.737+0000 7f52d0a14700 10 client.23535.objecter _send_command 12 2024-02-19T21:13:57.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.737+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f52ac017fd0 con 0x7f52ac0179e0 2024-02-19T21:13:57.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f52ac017fd0 con 0x7f52ac0179e0 2024-02-19T21:13:57.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:57.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac00e4a0 osd=5 4 2024-02-19T21:13:57.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 5 12 2024-02-19T21:13:57.739 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:13:57.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52d0a14700 10 client.23535.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:57.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 13 2024-02-19T21:13:57.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac017fd0 osd=6 3 2024-02-19T21:13:57.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac017fd0 osd=6 3 2024-02-19T21:13:57.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac017fd0 osd=6 3 2024-02-19T21:13:57.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 13 2024-02-19T21:13:57.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac017fd0 osd=6 3 2024-02-19T21:13:57.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 6 13 2024-02-19T21:13:57.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac017fd0 osd=6 4 2024-02-19T21:13:57.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52d0a14700 10 client.23535.objecter _send_command 13 2024-02-19T21:13:57.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.739+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52ac022be0 con 0x7f52ac01f0a0 2024-02-19T21:13:57.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.740+0000 7f52bb7fe700 10 client.23535.objecter ms_handle_connect 0x7f52ac01f0a0 2024-02-19T21:13:57.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.740+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f52ac022be0 con 0x7f52ac01f0a0 2024-02-19T21:13:57.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.740+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:13:57.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.740+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 13 = system:0 2024-02-19T21:13:57.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.740+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac017fd0 osd=6 4 2024-02-19T21:13:57.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.740+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 6 13 2024-02-19T21:13:57.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.754+0000 7f52d0a14700 10 client.23535.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:57.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.754+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 14 2024-02-19T21:13:57.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.754+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac017fd0 osd=6 3 2024-02-19T21:13:57.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.754+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac017fd0 osd=6 3 2024-02-19T21:13:57.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.754+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac017fd0 osd=6 3 2024-02-19T21:13:57.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.754+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 14 2024-02-19T21:13:57.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.754+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac017fd0 osd=6 3 2024-02-19T21:13:57.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.754+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 6 14 2024-02-19T21:13:57.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.754+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac017fd0 osd=6 4 2024-02-19T21:13:57.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.754+0000 7f52d0a14700 10 client.23535.objecter _send_command 14 2024-02-19T21:13:57.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.754+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f52ac024390 con 0x7f52ac01f0a0 2024-02-19T21:13:57.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f52ac024390 con 0x7f52ac01f0a0 2024-02-19T21:13:57.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:57.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac017fd0 osd=6 4 2024-02-19T21:13:57.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 6 14 2024-02-19T21:13:57.757 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:13:57.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52d0a14700 10 client.23535.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:13:57.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 15 2024-02-19T21:13:57.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac024390 osd=7 3 2024-02-19T21:13:57.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac024390 osd=7 3 2024-02-19T21:13:57.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac024390 osd=7 3 2024-02-19T21:13:57.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 15 2024-02-19T21:13:57.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac024390 osd=7 3 2024-02-19T21:13:57.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 7 15 2024-02-19T21:13:57.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac024390 osd=7 4 2024-02-19T21:13:57.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52d0a14700 10 client.23535.objecter _send_command 15 2024-02-19T21:13:57.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.756+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52ac024fc0 con 0x7f52ac024a10 2024-02-19T21:13:57.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.757+0000 7f52bb7fe700 10 client.23535.objecter ms_handle_connect 0x7f52ac024a10 2024-02-19T21:13:57.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.758+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f52ac024fc0 con 0x7f52ac024a10 2024-02-19T21:13:57.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.758+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:13:57.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.758+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 15 = system:0 2024-02-19T21:13:57.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.758+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac024390 osd=7 4 2024-02-19T21:13:57.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.758+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 7 15 2024-02-19T21:13:57.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.771+0000 7f52d0a14700 10 client.23535.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:13:57.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.771+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign -1 16 2024-02-19T21:13:57.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.771+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac024390 osd=7 3 2024-02-19T21:13:57.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.771+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac024390 osd=7 3 2024-02-19T21:13:57.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.771+0000 7f52d0a14700 20 client.23535.objecter _get_session s=0x7f52ac024390 osd=7 3 2024-02-19T21:13:57.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.771+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_remove -1 16 2024-02-19T21:13:57.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.771+0000 7f52d0a14700 20 client.23535.objecter get_session s=0x7f52ac024390 osd=7 3 2024-02-19T21:13:57.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.771+0000 7f52d0a14700 15 client.23535.objecter _session_command_op_assign 7 16 2024-02-19T21:13:57.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.771+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac024390 osd=7 4 2024-02-19T21:13:57.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.771+0000 7f52d0a14700 10 client.23535.objecter _send_command 16 2024-02-19T21:13:57.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.771+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f52ac024fc0 con 0x7f52ac024a10 2024-02-19T21:13:57.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52bb7fe700 1 -- 172.21.15.163:0/2656565702 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f52ac024fc0 con 0x7f52ac024a10 2024-02-19T21:13:57.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52bb7fe700 10 client.23535.objecter ms_dispatch 0x7f52cc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:13:57.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52bb7fe700 10 client.23535.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52bb7fe700 20 client.23535.objecter put_session s=0x7f52ac024390 osd=7 4 2024-02-19T21:13:57.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52bb7fe700 15 client.23535.objecter _session_command_op_remove 7 16 2024-02-19T21:13:57.774 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:13:57.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 10 librados: watch_flush enter 2024-02-19T21:13:57.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 10 librados: watch_flush exit 2024-02-19T21:13:57.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 10 client.23535.objecter close_session for osd.0 2024-02-19T21:13:57.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 >> v1:172.21.15.163:6801/1559418115 conn(0x7f52ac004530 legacy=0x7f52ac002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:57.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac0051a0 osd=0 1 2024-02-19T21:13:57.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 10 client.23535.objecter close_session for osd.1 2024-02-19T21:13:57.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 >> v1:172.21.15.163:6805/3125047432 conn(0x7f52ac004150 legacy=0x7f52ac00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:57.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac003fe0 osd=1 1 2024-02-19T21:13:57.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 10 client.23535.objecter close_session for osd.2 2024-02-19T21:13:57.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 >> v1:172.21.15.163:6809/1563158487 conn(0x7f52ac00d2f0 legacy=0x7f52ac00d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:57.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac003630 osd=2 1 2024-02-19T21:13:57.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 10 client.23535.objecter close_session for osd.3 2024-02-19T21:13:57.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 >> v1:172.21.15.163:6813/2112279769 conn(0x7f52ac011e80 legacy=0x7f52ac014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:57.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52cad9d700 1 -- 172.21.15.163:0/2656565702 reap_dead start 2024-02-19T21:13:57.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac0033d0 osd=3 1 2024-02-19T21:13:57.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 10 client.23535.objecter close_session for osd.4 2024-02-19T21:13:57.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 >> v1:172.21.15.193:6800/3129716965 conn(0x7f52ac00ce50 legacy=0x7f52ac015c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:57.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52cad9d700 1 -- 172.21.15.163:0/2656565702 reap_dead start 2024-02-19T21:13:57.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac00e300 osd=4 1 2024-02-19T21:13:57.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 10 client.23535.objecter close_session for osd.5 2024-02-19T21:13:57.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 >> v1:172.21.15.193:6804/805161991 conn(0x7f52ac0179e0 legacy=0x7f52ac01d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:57.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac00e4a0 osd=5 1 2024-02-19T21:13:57.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 10 client.23535.objecter close_session for osd.6 2024-02-19T21:13:57.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 >> v1:172.21.15.193:6808/787604615 conn(0x7f52ac01f0a0 legacy=0x7f52ac021490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:57.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac017fd0 osd=6 1 2024-02-19T21:13:57.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 10 client.23535.objecter close_session for osd.7 2024-02-19T21:13:57.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 >> v1:172.21.15.193:6812/157927432 conn(0x7f52ac024a10 legacy=0x7f52ac022be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:57.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 20 client.23535.objecter put_session s=0x7f52ac024390 osd=7 1 2024-02-19T21:13:57.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 20 client.23535.objecter shutdown clearing up homeless session... 2024-02-19T21:13:57.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 10 client.23535.objecter successfully canceled tick 2024-02-19T21:13:57.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 >> v1:172.21.15.163:6800/1552514681 conn(0x7f52b4063d10 legacy=0x7f52b40661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:57.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 >> v1:172.21.15.163:6789/0 conn(0x7f52cc0fd980 legacy=0x7f52cc0f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:13:57.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52cad9d700 1 -- 172.21.15.163:0/2656565702 reap_dead start 2024-02-19T21:13:57.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 shutdown_connections 2024-02-19T21:13:57.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 >> 172.21.15.163:0/2656565702 conn(0x7f52cc0f35a0 msgr2=0x7f52cc107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:13:57.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 shutdown_connections 2024-02-19T21:13:57.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 1 -- 172.21.15.163:0/2656565702 wait complete. 2024-02-19T21:13:57.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:13:57.774+0000 7f52d0a14700 1 librados: shutdown 2024-02-19T21:13:57.778 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.778 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.779 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.779 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.779 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.779 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.779 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.779 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:13:57.820 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:13:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:13:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:13:58.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-19T21:13:58.030 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:58.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:58.133 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:13:58.160 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:58.184 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:13:58.186 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:58.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:13:58.288 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:58.307 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:13:58.316 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:58.336 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:58.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:58.440 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:58.466 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:58.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:57.278832+0000 mgr.y (mgr.14550) 522 : cluster [DBG] pgmap v900: 214 pgs: 32 unknown, 8 creating+peering, 174 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:58.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:57.349088+0000 mon.a (mon.0) 3089 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-102", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:13:58.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:57.349151+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e601: 8 total, 8 up, 6 in 2024-02-19T21:13:58.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:57.349900+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-102"}]: dispatch 2024-02-19T21:13:58.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:57.637913+0000 mon.a (mon.0) 3092 : audit [DBG] from='client.? v1:172.21.15.163:0/2656565702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:58.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:57.278832+0000 mgr.y (mgr.14550) 522 : cluster [DBG] pgmap v900: 214 pgs: 32 unknown, 8 creating+peering, 174 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:58.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:57.349088+0000 mon.a (mon.0) 3089 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-102", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:13:58.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:57.349151+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e601: 8 total, 8 up, 6 in 2024-02-19T21:13:58.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:57.349900+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-102"}]: dispatch 2024-02-19T21:13:58.476 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:57.637913+0000 mon.a (mon.0) 3092 : audit [DBG] from='client.? v1:172.21.15.163:0/2656565702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:58.564 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:58.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:57.278832+0000 mgr.y (mgr.14550) 522 : cluster [DBG] pgmap v900: 214 pgs: 32 unknown, 8 creating+peering, 174 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:13:58.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:57.349088+0000 mon.a (mon.0) 3089 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-102", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:13:58.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:57.349151+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e601: 8 total, 8 up, 6 in 2024-02-19T21:13:58.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:57.349900+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-102"}]: dispatch 2024-02-19T21:13:58.565 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:57.637913+0000 mon.a (mon.0) 3092 : audit [DBG] from='client.? v1:172.21.15.163:0/2656565702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:13:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:58.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:58.627 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:13:58.668 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:13:58.700 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:58.720 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:13:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:58.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:13:58.824 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:13:58.853 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:58.906 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:13:59.051 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:13:59.054 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:59.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:13:59.155 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:13:59.182 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:59.200 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:13:59.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:59.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:13:59.304 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:59.331 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:59.403 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:58.349202+0000 mon.a (mon.0) 3093 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-102"}]': finished 2024-02-19T21:13:59.403 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:58.349278+0000 mon.a (mon.0) 3094 : cluster [DBG] osdmap e602: 8 total, 8 up, 6 in 2024-02-19T21:13:59.404 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:13:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:58.375792+0000 mon.a (mon.0) 3095 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:13:59.404 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:58.349202+0000 mon.a (mon.0) 3093 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-102"}]': finished 2024-02-19T21:13:59.404 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:58.349278+0000 mon.a (mon.0) 3094 : cluster [DBG] osdmap e602: 8 total, 8 up, 6 in 2024-02-19T21:13:59.404 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:13:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:58.375792+0000 mon.a (mon.0) 3095 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:13:59.524 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:58.349202+0000 mon.a (mon.0) 3093 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-102"}]': finished 2024-02-19T21:13:59.524 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:58.349278+0000 mon.a (mon.0) 3094 : cluster [DBG] osdmap e602: 8 total, 8 up, 6 in 2024-02-19T21:13:59.524 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:13:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:58.375792+0000 mon.a (mon.0) 3095 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:13:59.530 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:59.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:59.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:59.634 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:59.660 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:59.759 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:59.761 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:59.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:13:59.863 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:13:59.890 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:13:59.928 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:13:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:13:59.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:14:00.032 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:00.061 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:00.099 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:14:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-02-19T21:14:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-02-19T21:14:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-02-19T21:14:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25213.0:2 34.6 34:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e103)", 2024-02-19T21:14:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:42.715309+0000", 2024-02-19T21:14:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 557.33766603100003, 2024-02-19T21:14:00.099 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.043999731, 2024-02-19T21:14:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25213", 2024-02-19T21:14:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2094902030", 2024-02-19T21:14:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:14:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715309+0000", 2024-02-19T21:14:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.100 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715309+0000", 2024-02-19T21:14:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:14:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715307+0000", 2024-02-19T21:14:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0489999999999993e-06 2024-02-19T21:14:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.101 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715316+0000", 2024-02-19T21:14:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2014999999999999e-05 2024-02-19T21:14:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715328+0000", 2024-02-19T21:14:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3040000000000005e-06 2024-02-19T21:14:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715336+0000", 2024-02-19T21:14:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9018999999999995e-05 2024-02-19T21:14:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.102 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715435+0000", 2024-02-19T21:14:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9249999999999995e-06 2024-02-19T21:14:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715443+0000", 2024-02-19T21:14:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.033608503999999997 2024-02-19T21:14:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.749051+0000", 2024-02-19T21:14:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013195700000000001 2024-02-19T21:14:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.103 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.749183+0000", 2024-02-19T21:14:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014490099999999999 2024-02-19T21:14:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,2", 2024-02-19T21:14:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.749328+0000", 2024-02-19T21:14:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000480402 2024-02-19T21:14:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.749809+0000", 2024-02-19T21:14:00.104 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0069833769999999998 2024-02-19T21:14:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.756792+0000", 2024-02-19T21:14:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024635049999999999 2024-02-19T21:14:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759256+0000", 2024-02-19T21:14:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9428999999999999e-05 2024-02-19T21:14:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:00.105 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759285+0000", 2024-02-19T21:14:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3498000000000001e-05 2024-02-19T21:14:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.759309+0000", 2024-02-19T21:14:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.043999731 2024-02-19T21:14:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25255.0:2 39.1a 39:5c6776db:::0.obj:head [write 0~103] snapc 0=[] ondisk+write+known_if_redirected e103)", 2024-02-19T21:14:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:42.715388+0000", 2024-02-19T21:14:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 557.33758665599998, 2024-02-19T21:14:00.106 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0953346740000001, 2024-02-19T21:14:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25255", 2024-02-19T21:14:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1026204543", 2024-02-19T21:14:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:14:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715388+0000", 2024-02-19T21:14:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.107 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715388+0000", 2024-02-19T21:14:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:14:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715387+0000", 2024-02-19T21:14:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5220000000000004e-06 2024-02-19T21:14:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715393+0000", 2024-02-19T21:14:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1680000000000004e-06 2024-02-19T21:14:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715402+0000", 2024-02-19T21:14:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2669999999999997e-06 2024-02-19T21:14:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.715407+0000", 2024-02-19T21:14:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0061390949999999998 2024-02-19T21:14:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.721546+0000", 2024-02-19T21:14:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.464e-06 2024-02-19T21:14:00.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:42.721553+0000", 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.041467176 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.763021+0000", 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000122223 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.763143+0000", 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000135525 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,7", 2024-02-19T21:14:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.763278+0000", 2024-02-19T21:14:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001775127 2024-02-19T21:14:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.765053+0000", 2024-02-19T21:14:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011170955999999999 2024-02-19T21:14:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.776224+0000", 2024-02-19T21:14:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.034282156000000001 2024-02-19T21:14:00.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.810507+0000", 2024-02-19T21:14:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000115518 2024-02-19T21:14:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.810622+0000", 2024-02-19T21:14:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000100735 2024-02-19T21:14:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.810723+0000", 2024-02-19T21:14:00.112 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0953346740000001 2024-02-19T21:14:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25261.0:2 38.15 38:ad551702:::test:head [create] snapc 0=[] ondisk+write+known_if_redirected e104)", 2024-02-19T21:14:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:43.718525+0000", 2024-02-19T21:14:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 556.33444938100001, 2024-02-19T21:14:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.092407107000000002, 2024-02-19T21:14:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25261", 2024-02-19T21:14:00.113 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3188847468", 2024-02-19T21:14:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:14:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718525+0000", 2024-02-19T21:14:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718525+0000", 2024-02-19T21:14:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:14:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718522+0000", 2024-02-19T21:14:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3332999999999999e-05 2024-02-19T21:14:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718535+0000", 2024-02-19T21:14:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2945000000000001e-05 2024-02-19T21:14:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718548+0000", 2024-02-19T21:14:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9999999999999996e-06 2024-02-19T21:14:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718556+0000", 2024-02-19T21:14:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8600000000000001e-05 2024-02-19T21:14:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718614+0000", 2024-02-19T21:14:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4850000000000003e-06 2024-02-19T21:14:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.718622+0000", 2024-02-19T21:14:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.080586017999999995 2024-02-19T21:14:00.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.799208+0000", 2024-02-19T21:14:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013370499999999999 2024-02-19T21:14:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.799342+0000", 2024-02-19T21:14:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015074000000000001 2024-02-19T21:14:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,1", 2024-02-19T21:14:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.799492+0000", 2024-02-19T21:14:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00074646999999999995 2024-02-19T21:14:00.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.800239+0000", 2024-02-19T21:14:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0041852749999999996 2024-02-19T21:14:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.804424+0000", 2024-02-19T21:14:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.006421637 2024-02-19T21:14:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.810846+0000", 2024-02-19T21:14:00.118 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7818000000000002e-05 2024-02-19T21:14:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.810894+0000", 2024-02-19T21:14:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8928999999999999e-05 2024-02-19T21:14:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:43.810933+0000", 2024-02-19T21:14:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.092407107000000002 2024-02-19T21:14:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25243.0:1 44.6s0 44:6ec48444:test-rados-api-smithi163-83968-8::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e106)", 2024-02-19T21:14:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:45.953526+0000", 2024-02-19T21:14:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 554.09944846200005, 2024-02-19T21:14:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0395346679999999, 2024-02-19T21:14:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25243", 2024-02-19T21:14:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3720420201", 2024-02-19T21:14:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:14:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953526+0000", 2024-02-19T21:14:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953526+0000", 2024-02-19T21:14:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:14:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953525+0000", 2024-02-19T21:14:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8869999999999992e-06 2024-02-19T21:14:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.121 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953532+0000", 2024-02-19T21:14:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020723920000000002 2024-02-19T21:14:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.955605+0000", 2024-02-19T21:14:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4469999999999995e-06 2024-02-19T21:14:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.955614+0000", 2024-02-19T21:14:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9202000000000002e-05 2024-02-19T21:14:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.955654+0000", 2024-02-19T21:14:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7e-06 2024-02-19T21:14:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.955655+0000", 2024-02-19T21:14:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0209778119999999 2024-02-19T21:14:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.976633+0000", 2024-02-19T21:14:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013542099999999999 2024-02-19T21:14:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.123 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.976768+0000", 2024-02-19T21:14:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00051974200000000001 2024-02-19T21:14:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:14:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.977288+0000", 2024-02-19T21:14:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0018645790000000001 2024-02-19T21:14:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:14:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.979153+0000", 2024-02-19T21:14:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013854811 2024-02-19T21:14:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.993008+0000", 2024-02-19T21:14:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3474999999999999e-05 2024-02-19T21:14:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.993061+0000", 2024-02-19T21:14:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0395346679999999 2024-02-19T21:14:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25228.0:1 42.5s0 42:abdeb581:test-rados-api-smithi163-83960-7::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e106)", 2024-02-19T21:14:00.125 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:45.953691+0000", 2024-02-19T21:14:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 554.09928368999999, 2024-02-19T21:14:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.03047446, 2024-02-19T21:14:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25228", 2024-02-19T21:14:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/516814289", 2024-02-19T21:14:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:14:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953691+0000", 2024-02-19T21:14:00.126 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953691+0000", 2024-02-19T21:14:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:14:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953690+0000", 2024-02-19T21:14:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2419999999999997e-06 2024-02-19T21:14:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953696+0000", 2024-02-19T21:14:00.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00023778100000000001 2024-02-19T21:14:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953934+0000", 2024-02-19T21:14:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2859999999999999e-06 2024-02-19T21:14:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.953942+0000", 2024-02-19T21:14:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0016913939999999999 2024-02-19T21:14:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.128 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.955633+0000", 2024-02-19T21:14:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.225e-06 2024-02-19T21:14:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:45.955636+0000", 2024-02-19T21:14:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.008495243 2024-02-19T21:14:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.964132+0000", 2024-02-19T21:14:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000152194 2024-02-19T21:14:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.129 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.964284+0000", 2024-02-19T21:14:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00045546399999999999 2024-02-19T21:14:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:14:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.964739+0000", 2024-02-19T21:14:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00086119899999999999 2024-02-19T21:14:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:14:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.965601+0000", 2024-02-19T21:14:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.018476871999999998 2024-02-19T21:14:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.130 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.984077+0000", 2024-02-19T21:14:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8187000000000001e-05 2024-02-19T21:14:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:46.984166+0000", 2024-02-19T21:14:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.03047446 2024-02-19T21:14:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.24895.0:14 25.c 25:354b2f79:test-rados-api-smithi163-83944-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:14:00.131 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:47.957539+0000", 2024-02-19T21:14:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 552.09543614999995, 2024-02-19T21:14:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.063082621000000005, 2024-02-19T21:14:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.24895", 2024-02-19T21:14:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3326199758", 2024-02-19T21:14:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 14 2024-02-19T21:14:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957539+0000", 2024-02-19T21:14:00.132 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957539+0000", 2024-02-19T21:14:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:14:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957536+0000", 2024-02-19T21:14:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6256000000000002e-05 2024-02-19T21:14:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957552+0000", 2024-02-19T21:14:00.133 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2793e-05 2024-02-19T21:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957575+0000", 2024-02-19T21:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0999999999999993e-06 2024-02-19T21:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957584+0000", 2024-02-19T21:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6638999999999998e-05 2024-02-19T21:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957640+0000", 2024-02-19T21:14:00.134 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6559999999999992e-06 2024-02-19T21:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.957648+0000", 2024-02-19T21:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0094927950000000001 2024-02-19T21:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.967141+0000", 2024-02-19T21:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0094054069999999993 2024-02-19T21:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976546+0000", 2024-02-19T21:14:00.135 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013502699999999999 2024-02-19T21:14:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,7", 2024-02-19T21:14:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.976681+0000", 2024-02-19T21:14:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001538427 2024-02-19T21:14:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.978220+0000", 2024-02-19T21:14:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011847066 2024-02-19T21:14:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.136 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.990067+0000", 2024-02-19T21:14:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.030463536999999999 2024-02-19T21:14:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.020530+0000", 2024-02-19T21:14:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6168000000000002e-05 2024-02-19T21:14:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.020577+0000", 2024-02-19T21:14:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4790999999999999e-05 2024-02-19T21:14:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.137 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.020621+0000", 2024-02-19T21:14:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.063082621000000005 2024-02-19T21:14:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.14940.0:1 50.0s0 50:0c5c848e:test-rados-api-smithi163-83904-11::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e108)", 2024-02-19T21:14:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:47.977197+0000", 2024-02-19T21:14:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 552.07577820899996, 2024-02-19T21:14:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.62635811600000002, 2024-02-19T21:14:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.138 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.14940", 2024-02-19T21:14:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/676446879", 2024-02-19T21:14:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:14:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.977197+0000", 2024-02-19T21:14:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.977197+0000", 2024-02-19T21:14:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:14:00.139 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.977194+0000", 2024-02-19T21:14:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5399999999999996e-06 2024-02-19T21:14:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.977203+0000", 2024-02-19T21:14:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6239999999999997e-06 2024-02-19T21:14:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.140 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.977213+0000", 2024-02-19T21:14:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5740000000000001e-06 2024-02-19T21:14:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.977219+0000", 2024-02-19T21:14:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001035041 2024-02-19T21:14:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.978254+0000", 2024-02-19T21:14:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3029999999999998e-06 2024-02-19T21:14:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:00.141 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:47.978256+0000", 2024-02-19T21:14:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.60994813199999998 2024-02-19T21:14:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.588205+0000", 2024-02-19T21:14:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.1259000000000002e-05 2024-02-19T21:14:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.588296+0000", 2024-02-19T21:14:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00027813000000000002 2024-02-19T21:14:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:14:00.142 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.588574+0000", 2024-02-19T21:14:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000344502 2024-02-19T21:14:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:14:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.588918+0000", 2024-02-19T21:14:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014572966 2024-02-19T21:14:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.603491+0000", 2024-02-19T21:14:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3334e-05 2024-02-19T21:14:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.143 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.603555+0000", 2024-02-19T21:14:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.62635811600000002 2024-02-19T21:14:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.35237.0:2 52.b 52:d7043a55:test-rados-api-smithi163-83984-5::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e109)", 2024-02-19T21:14:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:48.521853+0000", 2024-02-19T21:14:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 551.531121912, 2024-02-19T21:14:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.050592887000000003, 2024-02-19T21:14:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:00.144 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.35237", 2024-02-19T21:14:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1081998331", 2024-02-19T21:14:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:14:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.521853+0000", 2024-02-19T21:14:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.521853+0000", 2024-02-19T21:14:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:14:00.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.521851+0000", 2024-02-19T21:14:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9590000000000001e-06 2024-02-19T21:14:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.521860+0000", 2024-02-19T21:14:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1649e-05 2024-02-19T21:14:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.521871+0000", 2024-02-19T21:14:00.146 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2400000000000007e-06 2024-02-19T21:14:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.521880+0000", 2024-02-19T21:14:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001923971 2024-02-19T21:14:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.523804+0000", 2024-02-19T21:14:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6629999999999999e-06 2024-02-19T21:14:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.523812+0000", 2024-02-19T21:14:00.147 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.040019490999999997 2024-02-19T21:14:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.563832+0000", 2024-02-19T21:14:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.5551999999999997e-05 2024-02-19T21:14:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.563917+0000", 2024-02-19T21:14:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012541700000000001 2024-02-19T21:14:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,7", 2024-02-19T21:14:00.148 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.564043+0000", 2024-02-19T21:14:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00052241799999999997 2024-02-19T21:14:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.564565+0000", 2024-02-19T21:14:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001139024 2024-02-19T21:14:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.565704+0000", 2024-02-19T21:14:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0066732509999999998 2024-02-19T21:14:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.149 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.572377+0000", 2024-02-19T21:14:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2475000000000003e-05 2024-02-19T21:14:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.572410+0000", 2024-02-19T21:14:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5967000000000003e-05 2024-02-19T21:14:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:48.572446+0000", 2024-02-19T21:14:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.050592887000000003 2024-02-19T21:14:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.150 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.14940.0:10 50.0s0 50:0928318f:test-rados-api-smithi163-83904-14::foo:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected e110)", 2024-02-19T21:14:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:50.620751+0000", 2024-02-19T21:14:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 549.43222418599998, 2024-02-19T21:14:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.016526605999999999, 2024-02-19T21:14:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.14940", 2024-02-19T21:14:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/676446879", 2024-02-19T21:14:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 10 2024-02-19T21:14:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.151 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620751+0000", 2024-02-19T21:14:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620751+0000", 2024-02-19T21:14:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:14:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620748+0000", 2024-02-19T21:14:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9029999999999999e-06 2024-02-19T21:14:00.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620758+0000", 2024-02-19T21:14:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7120000000000001e-05 2024-02-19T21:14:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620785+0000", 2024-02-19T21:14:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6659999999999997e-06 2024-02-19T21:14:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620794+0000", 2024-02-19T21:14:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0709000000000003e-05 2024-02-19T21:14:00.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620874+0000", 2024-02-19T21:14:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012368899999999999 2024-02-19T21:14:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.620998+0000", 2024-02-19T21:14:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000371963 2024-02-19T21:14:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:14:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.621370+0000", 2024-02-19T21:14:00.154 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00044841800000000002 2024-02-19T21:14:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:14:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.621819+0000", 2024-02-19T21:14:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015420919999999999 2024-02-19T21:14:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.637239+0000", 2024-02-19T21:14:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7781000000000003e-05 2024-02-19T21:14:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:50.637277+0000", 2024-02-19T21:14:00.155 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.016526605999999999 2024-02-19T21:14:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.24865.0:21 11.8 e112/99 11:1381d4a2:test-rados-api-smithi163-83952-6::foo:head v 112'4, mlcod=110'1)", 2024-02-19T21:14:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:04:51.587142+0000", 2024-02-19T21:14:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 548.46583315400005, 2024-02-19T21:14:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026422297000000001, 2024-02-19T21:14:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.156 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.587142+0000", 2024-02-19T21:14:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.587142+0000", 2024-02-19T21:14:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:14:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.587140+0000", 2024-02-19T21:14:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3057999999999999e-05 2024-02-19T21:14:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.157 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.587153+0000", 2024-02-19T21:14:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00027089900000000001 2024-02-19T21:14:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.587424+0000", 2024-02-19T21:14:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2164000000000001e-05 2024-02-19T21:14:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.587436+0000", 2024-02-19T21:14:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025388634 2024-02-19T21:14:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.158 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.612825+0000", 2024-02-19T21:14:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0619e-05 2024-02-19T21:14:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.612845+0000", 2024-02-19T21:14:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00067890799999999999 2024-02-19T21:14:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.613524+0000", 2024-02-19T21:14:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9892999999999999e-05 2024-02-19T21:14:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.159 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:04:51.613564+0000", 2024-02-19T21:14:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026422297000000001 2024-02-19T21:14:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.24919.0:2079 28.9 28:90852524:test-rados-api-smithi163-83920-17::big2.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-02-19T21:14:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.501465+0000", 2024-02-19T21:14:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 527.55150942700004, 2024-02-19T21:14:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.23286506000000001, 2024-02-19T21:14:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.160 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.24919", 2024-02-19T21:14:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/4089460683", 2024-02-19T21:14:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2079 2024-02-19T21:14:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501465+0000", 2024-02-19T21:14:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501465+0000", 2024-02-19T21:14:00.161 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:14:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501462+0000", 2024-02-19T21:14:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2568000000000001e-05 2024-02-19T21:14:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501475+0000", 2024-02-19T21:14:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6639000000000001e-05 2024-02-19T21:14:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501491+0000", 2024-02-19T21:14:00.162 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9549999999999992e-06 2024-02-19T21:14:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501499+0000", 2024-02-19T21:14:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6508999999999997e-05 2024-02-19T21:14:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501556+0000", 2024-02-19T21:14:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010162699999999999 2024-02-19T21:14:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.501658+0000", 2024-02-19T21:14:00.163 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.18445410800000001 2024-02-19T21:14:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,6", 2024-02-19T21:14:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.686112+0000", 2024-02-19T21:14:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010606519 2024-02-19T21:14:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.696718+0000", 2024-02-19T21:14:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00049238600000000004 2024-02-19T21:14:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:00.164 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.697211+0000", 2024-02-19T21:14:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.037011418999999997 2024-02-19T21:14:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.734222+0000", 2024-02-19T21:14:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6672000000000002e-05 2024-02-19T21:14:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.734279+0000", 2024-02-19T21:14:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1786000000000002e-05 2024-02-19T21:14:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.165 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.734330+0000", 2024-02-19T21:14:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.23286506000000001 2024-02-19T21:14:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.47072.0:2 90.13 90:cbb9903e:::test_obj:head [write 0~128 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected e139)", 2024-02-19T21:14:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:19.211019+0000", 2024-02-19T21:14:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 520.84195565599998, 2024-02-19T21:14:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.110542764, 2024-02-19T21:14:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.166 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.47072", 2024-02-19T21:14:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3219046901", 2024-02-19T21:14:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:14:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211019+0000", 2024-02-19T21:14:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211019+0000", 2024-02-19T21:14:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:14:00.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211017+0000", 2024-02-19T21:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6989999999999997e-06 2024-02-19T21:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211024+0000", 2024-02-19T21:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9810000000000003e-06 2024-02-19T21:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211032+0000", 2024-02-19T21:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.78e-06 2024-02-19T21:14:00.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211037+0000", 2024-02-19T21:14:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1912999999999997e-05 2024-02-19T21:14:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211079+0000", 2024-02-19T21:14:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1199000000000001e-05 2024-02-19T21:14:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.211090+0000", 2024-02-19T21:14:00.169 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015814228999999999 2024-02-19T21:14:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.226904+0000", 2024-02-19T21:14:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9059000000000003e-05 2024-02-19T21:14:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.226983+0000", 2024-02-19T21:14:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010925399999999999 2024-02-19T21:14:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,2", 2024-02-19T21:14:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.227092+0000", 2024-02-19T21:14:00.170 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00033114899999999998 2024-02-19T21:14:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.227424+0000", 2024-02-19T21:14:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.078279331999999993 2024-02-19T21:14:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.305703+0000", 2024-02-19T21:14:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015761226999999999 2024-02-19T21:14:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.171 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.321464+0000", 2024-02-19T21:14:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5554999999999999e-05 2024-02-19T21:14:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.321520+0000", 2024-02-19T21:14:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2196000000000001e-05 2024-02-19T21:14:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:19.321562+0000", 2024-02-19T21:14:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.110542764 2024-02-19T21:14:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.172 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22545.0:4 150.1e 150:7cdf4dc8:::oid_XattrsRoundTripPP_:head [setxattr attr1 (12) in=5b] snapc 0=[] ondisk+write+known_if_redirected e217)", 2024-02-19T21:14:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:06:39.209473+0000", 2024-02-19T21:14:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 440.84350174799999, 2024-02-19T21:14:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022521461999999999, 2024-02-19T21:14:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22545", 2024-02-19T21:14:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3549317140", 2024-02-19T21:14:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 4 2024-02-19T21:14:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.173 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209473+0000", 2024-02-19T21:14:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209473+0000", 2024-02-19T21:14:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:14:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209470+0000", 2024-02-19T21:14:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1134e-05 2024-02-19T21:14:00.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209481+0000", 2024-02-19T21:14:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2923e-05 2024-02-19T21:14:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209494+0000", 2024-02-19T21:14:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9760000000000001e-06 2024-02-19T21:14:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209503+0000", 2024-02-19T21:14:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0747000000000005e-05 2024-02-19T21:14:00.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.176 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.176 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.176 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209584+0000", 2024-02-19T21:14:00.176 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9016000000000002e-05 2024-02-19T21:14:00.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.176 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.176 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.176 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209683+0000", 2024-02-19T21:14:00.176 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015398000000000001 2024-02-19T21:14:00.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.176 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.176 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,7", 2024-02-19T21:14:00.176 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.209837+0000", 2024-02-19T21:14:00.176 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0071038949999999998 2024-02-19T21:14:00.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.177 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.177 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:00.177 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.216941+0000", 2024-02-19T21:14:00.177 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001669797 2024-02-19T21:14:00.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.177 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.177 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.177 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.218611+0000", 2024-02-19T21:14:00.177 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013307836999999999 2024-02-19T21:14:00.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.177 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.177 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.177 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.231919+0000", 2024-02-19T21:14:00.177 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9233000000000001e-05 2024-02-19T21:14:00.178 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.178 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.178 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:00.178 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.231968+0000", 2024-02-19T21:14:00.178 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6804999999999998e-05 2024-02-19T21:14:00.178 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.178 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.178 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.178 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:39.231995+0000", 2024-02-19T21:14:00.178 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022521461999999999 2024-02-19T21:14:00.178 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.178 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.178 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.178 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.178 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.179 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.49328.0:2 165.18 165:1c88de7c:::oid_XattrListPP_:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected e244)", 2024-02-19T21:14:00.179 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:04.204109+0000", 2024-02-19T21:14:00.179 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 415.84886567500001, 2024-02-19T21:14:00.179 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025817819999999998, 2024-02-19T21:14:00.179 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.179 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:00.179 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.179 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.49328", 2024-02-19T21:14:00.179 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/4008237253", 2024-02-19T21:14:00.179 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:14:00.179 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.179 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.179 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.179 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204109+0000", 2024-02-19T21:14:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204109+0000", 2024-02-19T21:14:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:14:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204106+0000", 2024-02-19T21:14:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0121e-05 2024-02-19T21:14:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.181 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.181 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204117+0000", 2024-02-19T21:14:00.181 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6637999999999999e-05 2024-02-19T21:14:00.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.181 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.181 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204133+0000", 2024-02-19T21:14:00.181 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4710000000000007e-06 2024-02-19T21:14:00.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.181 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.181 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204142+0000", 2024-02-19T21:14:00.181 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5637999999999997e-05 2024-02-19T21:14:00.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.182 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.182 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204207+0000", 2024-02-19T21:14:00.182 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.973e-06 2024-02-19T21:14:00.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.182 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.182 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:14:00.182 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.204214+0000", 2024-02-19T21:14:00.182 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0045024840000000002 2024-02-19T21:14:00.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.182 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.182 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.182 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.208717+0000", 2024-02-19T21:14:00.182 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7860000000000002e-06 2024-02-19T21:14:00.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.182 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.183 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:00.183 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.208726+0000", 2024-02-19T21:14:00.183 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.016396628 2024-02-19T21:14:00.183 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.183 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.183 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.183 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.225122+0000", 2024-02-19T21:14:00.183 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013968699999999999 2024-02-19T21:14:00.183 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.183 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.183 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.183 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.225262+0000", 2024-02-19T21:14:00.183 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000145799 2024-02-19T21:14:00.183 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.184 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.184 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,4", 2024-02-19T21:14:00.184 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.225408+0000", 2024-02-19T21:14:00.184 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014860590000000001 2024-02-19T21:14:00.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.184 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.184 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.184 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.226894+0000", 2024-02-19T21:14:00.184 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0022827239999999999 2024-02-19T21:14:00.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.184 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.184 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:00.184 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.229176+0000", 2024-02-19T21:14:00.184 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00046320699999999999 2024-02-19T21:14:00.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.185 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.185 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:00.185 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.229640+0000", 2024-02-19T21:14:00.185 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0414000000000002e-05 2024-02-19T21:14:00.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.185 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.185 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:00.185 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.229690+0000", 2024-02-19T21:14:00.185 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000236868 2024-02-19T21:14:00.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.185 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.185 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.185 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:04.229927+0000", 2024-02-19T21:14:00.185 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025817819999999998 2024-02-19T21:14:00.185 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.186 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.186 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.186 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.186 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.186 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22848.0:1 192.5s0 192:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected e314)", 2024-02-19T21:14:00.186 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:08:14.229643+0000", 2024-02-19T21:14:00.186 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 345.82333165599999, 2024-02-19T21:14:00.186 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4029510890000001, 2024-02-19T21:14:00.186 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.186 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-02-19T21:14:00.186 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.186 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22848", 2024-02-19T21:14:00.186 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1659723686", 2024-02-19T21:14:00.186 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:14:00.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.187 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.187 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.187 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229643+0000", 2024-02-19T21:14:00.187 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.187 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.187 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229643+0000", 2024-02-19T21:14:00.187 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:14:00.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.187 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.187 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229641+0000", 2024-02-19T21:14:00.188 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2750000000000001e-06 2024-02-19T21:14:00.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.188 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.188 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.188 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229647+0000", 2024-02-19T21:14:00.188 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4100000000000008e-06 2024-02-19T21:14:00.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.188 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.188 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.188 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229656+0000", 2024-02-19T21:14:00.188 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4850000000000003e-06 2024-02-19T21:14:00.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.188 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.188 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.188 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229663+0000", 2024-02-19T21:14:00.189 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0076000000000001e-05 2024-02-19T21:14:00.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.189 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.189 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.189 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229703+0000", 2024-02-19T21:14:00.189 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0790000000000001e-06 2024-02-19T21:14:00.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.189 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.189 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:00.189 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.229707+0000", 2024-02-19T21:14:00.189 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4027761860000001 2024-02-19T21:14:00.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.189 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.189 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.190 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:15.632483+0000", 2024-02-19T21:14:00.190 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011157 2024-02-19T21:14:00.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.190 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.190 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:15.632594+0000", 2024-02-19T21:14:00.190 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4029510890000001 2024-02-19T21:14:00.190 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.190 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.190 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.190 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.25753.0:4348 191.8 e314/308 191:10fb1d02:::199:head v 314'8, mlcod=313'6)", 2024-02-19T21:14:00.190 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:08:14.258303+0000", 2024-02-19T21:14:00.190 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 345.79467145299998, 2024-02-19T21:14:00.191 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.023938170000000002, 2024-02-19T21:14:00.191 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.191 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:00.191 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.191 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.191 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258303+0000", 2024-02-19T21:14:00.191 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.191 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.191 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258303+0000", 2024-02-19T21:14:00.191 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:14:00.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.192 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.192 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258302+0000", 2024-02-19T21:14:00.192 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8970000000000008e-06 2024-02-19T21:14:00.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.192 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.192 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258310+0000", 2024-02-19T21:14:00.192 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0559e-05 2024-02-19T21:14:00.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.192 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.192 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258321+0000", 2024-02-19T21:14:00.192 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.2500000000000003e-06 2024-02-19T21:14:00.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.193 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.193 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.193 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258327+0000", 2024-02-19T21:14:00.193 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7713999999999998e-05 2024-02-19T21:14:00.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.193 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.193 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.193 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258365+0000", 2024-02-19T21:14:00.193 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0503999999999999e-05 2024-02-19T21:14:00.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.193 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.193 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.193 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258385+0000", 2024-02-19T21:14:00.193 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.023829350999999999 2024-02-19T21:14:00.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.194 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.194 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:00.194 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.282214+0000", 2024-02-19T21:14:00.194 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7092000000000001e-05 2024-02-19T21:14:00.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.194 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.194 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.194 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.282242+0000", 2024-02-19T21:14:00.194 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.023938170000000002 2024-02-19T21:14:00.194 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.194 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.194 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.194 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.195 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:1416 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94681540389120] snapc 0=[] ondisk+write+known_if_redirected e430)", 2024-02-19T21:14:00.195 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:10:39.701549+0000", 2024-02-19T21:14:00.195 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 200.351425506, 2024-02-19T21:14:00.195 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.61218586399999997, 2024-02-19T21:14:00.195 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.195 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:14:00.195 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.195 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:14:00.195 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:14:00.195 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1416 2024-02-19T21:14:00.195 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.195 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.195 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.195 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.196 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.701549+0000", 2024-02-19T21:14:00.196 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.196 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.196 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.196 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.701549+0000", 2024-02-19T21:14:00.196 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:14:00.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.196 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.196 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.196 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.701547+0000", 2024-02-19T21:14:00.196 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4390000000000006e-06 2024-02-19T21:14:00.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.196 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.197 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.197 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.701556+0000", 2024-02-19T21:14:00.197 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3176999999999999e-05 2024-02-19T21:14:00.197 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.197 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.197 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.197 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.701569+0000", 2024-02-19T21:14:00.197 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8070000000000002e-06 2024-02-19T21:14:00.197 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.197 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.197 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.197 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.701576+0000", 2024-02-19T21:14:00.197 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00031892500000000001 2024-02-19T21:14:00.197 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.197 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.198 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.198 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.701895+0000", 2024-02-19T21:14:00.198 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6076e-05 2024-02-19T21:14:00.198 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.198 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.198 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:00.198 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.701911+0000", 2024-02-19T21:14:00.198 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.60879026199999997 2024-02-19T21:14:00.198 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.198 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.198 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.198 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.310701+0000", 2024-02-19T21:14:00.198 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000323293 2024-02-19T21:14:00.198 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.199 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for degraded object", 2024-02-19T21:14:00.199 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311025+0000", 2024-02-19T21:14:00.199 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025281750000000001 2024-02-19T21:14:00.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.199 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.199 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.313553+0000", 2024-02-19T21:14:00.199 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8599999999999999e-05 2024-02-19T21:14:00.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.199 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.199 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.313641+0000", 2024-02-19T21:14:00.199 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3768000000000001e-05 2024-02-19T21:14:00.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.200 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.200 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.200 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.313735+0000", 2024-02-19T21:14:00.200 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.61218586399999997 2024-02-19T21:14:00.200 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.200 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.200 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.200 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.200 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:1421 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94681540389120 gen 1] snapc 0=[] ondisk+write+known_if_redirected e432)", 2024-02-19T21:14:00.200 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:10:39.702022+0000", 2024-02-19T21:14:00.200 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 200.35095237499999, 2024-02-19T21:14:00.200 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.61459377100000001, 2024-02-19T21:14:00.201 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.201 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:14:00.201 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.201 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:14:00.201 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:14:00.201 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1421 2024-02-19T21:14:00.201 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.201 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.201 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.201 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.201 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.702022+0000", 2024-02-19T21:14:00.201 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.201 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.201 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.201 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.202 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.702022+0000", 2024-02-19T21:14:00.202 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:14:00.202 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.202 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.202 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.202 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.702021+0000", 2024-02-19T21:14:00.202 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5289999999999997e-06 2024-02-19T21:14:00.202 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.202 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.202 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.202 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.702027+0000", 2024-02-19T21:14:00.202 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0067e-05 2024-02-19T21:14:00.202 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.202 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.202 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.702037+0000", 2024-02-19T21:14:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1749999999999996e-06 2024-02-19T21:14:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.702043+0000", 2024-02-19T21:14:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9836999999999997e-05 2024-02-19T21:14:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.702092+0000", 2024-02-19T21:14:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1319999999999999e-06 2024-02-19T21:14:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.203 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:00.204 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:39.702099+0000", 2024-02-19T21:14:00.204 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.60895348800000004 2024-02-19T21:14:00.204 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.204 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.204 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.204 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311052+0000", 2024-02-19T21:14:00.204 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7873000000000006e-05 2024-02-19T21:14:00.204 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.204 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.204 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for degraded object", 2024-02-19T21:14:00.204 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311120+0000", 2024-02-19T21:14:00.204 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0026370439999999998 2024-02-19T21:14:00.204 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.204 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.205 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.205 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.313757+0000", 2024-02-19T21:14:00.205 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1081000000000001e-05 2024-02-19T21:14:00.205 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.205 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.205 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.205 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.313808+0000", 2024-02-19T21:14:00.205 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0028082519999999998 2024-02-19T21:14:00.205 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.205 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.205 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.205 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.316616+0000", 2024-02-19T21:14:00.205 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.61459377100000001 2024-02-19T21:14:00.205 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.206 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.206 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.206 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.206 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.206 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:6198 61.1c 61:3da6406c:test-rados-api-smithi163-83928-64::foo:head [call cas.references_chunk in=19b] snapc 0=[] ondisk+read+known_if_redirected e434)", 2024-02-19T21:14:00.206 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:10:41.529757+0000", 2024-02-19T21:14:00.206 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 198.52321815600001, 2024-02-19T21:14:00.206 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.75805697400000005, 2024-02-19T21:14:00.206 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.206 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:14:00.206 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.206 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:14:00.206 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:14:00.206 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 6198 2024-02-19T21:14:00.206 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.207 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.207 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.207 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.207 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.529757+0000", 2024-02-19T21:14:00.207 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.207 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.207 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.207 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.207 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.529757+0000", 2024-02-19T21:14:00.207 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999957 2024-02-19T21:14:00.207 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.207 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.207 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.207 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.529752+0000", 2024-02-19T21:14:00.207 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.747e-05 2024-02-19T21:14:00.208 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.208 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.208 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.208 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.529770+0000", 2024-02-19T21:14:00.208 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6682999999999998e-05 2024-02-19T21:14:00.208 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.208 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.208 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.208 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.529797+0000", 2024-02-19T21:14:00.208 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3020000000000001e-05 2024-02-19T21:14:00.208 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.208 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.208 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.208 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.529810+0000", 2024-02-19T21:14:00.209 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000103735 2024-02-19T21:14:00.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.209 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.209 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.209 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.529913+0000", 2024-02-19T21:14:00.209 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7299999999999994e-06 2024-02-19T21:14:00.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.209 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.209 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:00.209 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.529922+0000", 2024-02-19T21:14:00.209 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.75752385200000005 2024-02-19T21:14:00.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.209 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.209 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.210 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:42.287446+0000", 2024-02-19T21:14:00.210 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018918599999999999 2024-02-19T21:14:00.210 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.210 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.210 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.210 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:42.287635+0000", 2024-02-19T21:14:00.210 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00017857800000000001 2024-02-19T21:14:00.210 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.210 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.210 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.210 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:42.287814+0000", 2024-02-19T21:14:00.210 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.75805697400000005 2024-02-19T21:14:00.210 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.210 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.210 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.211 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.211 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.211 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:6735 217.0 217:00000000::::head [pgnls start_epoch 461] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e461)", 2024-02-19T21:14:00.211 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:11:29.388310+0000", 2024-02-19T21:14:00.211 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 150.66466458100001, 2024-02-19T21:14:00.211 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.018276126, 2024-02-19T21:14:00.211 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:00.211 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:14:00.211 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:00.211 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:14:00.211 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:14:00.211 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 6735 2024-02-19T21:14:00.211 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.211 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:00.211 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.212 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:00.212 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.388310+0000", 2024-02-19T21:14:00.212 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:00.212 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.212 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.212 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:00.212 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.388310+0000", 2024-02-19T21:14:00.212 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:14:00.212 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.212 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.212 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:00.212 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.388308+0000", 2024-02-19T21:14:00.212 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0043e-05 2024-02-19T21:14:00.212 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.213 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.213 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:00.213 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.388318+0000", 2024-02-19T21:14:00.213 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4234e-05 2024-02-19T21:14:00.213 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.213 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.213 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:00.213 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.388332+0000", 2024-02-19T21:14:00.213 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4918000000000001e-05 2024-02-19T21:14:00.213 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.213 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.213 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:00.213 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.388357+0000", 2024-02-19T21:14:00.213 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3549999999999999e-05 2024-02-19T21:14:00.213 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.214 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.214 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.214 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.388431+0000", 2024-02-19T21:14:00.214 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.31e-06 2024-02-19T21:14:00.214 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.214 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.214 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:14:00.214 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.388436+0000", 2024-02-19T21:14:00.214 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017920037 2024-02-19T21:14:00.214 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.214 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.214 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:00.214 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.406356+0000", 2024-02-19T21:14:00.214 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8012000000000001e-05 2024-02-19T21:14:00.214 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.215 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.215 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:00.215 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.406394+0000", 2024-02-19T21:14:00.215 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00019228800000000001 2024-02-19T21:14:00.215 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:00.215 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:00.215 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:00.215 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:29.406586+0000", 2024-02-19T21:14:00.215 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.018276126 2024-02-19T21:14:00.215 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.215 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.215 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.215 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:00.215 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:00.215 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:14:00.216 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 7, 3, 4] out_osds: [0, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:14:00.216 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:14:00.216 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:14:00.309 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:00.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:00.413 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:14:00.442 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:00.497 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:14:00.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:59.279214+0000 mgr.y (mgr.14550) 523 : cluster [DBG] pgmap v903: 214 pgs: 32 unknown, 8 creating+peering, 174 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:14:00.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:59.360094+0000 mon.a (mon.0) 3096 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:00.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:13:59.360147+0000 mon.a (mon.0) 3097 : cluster [DBG] osdmap e603: 8 total, 8 up, 6 in 2024-02-19T21:14:00.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:13:59.360936+0000 mon.a (mon.0) 3098 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-102"}]: dispatch 2024-02-19T21:14:00.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:59.279214+0000 mgr.y (mgr.14550) 523 : cluster [DBG] pgmap v903: 214 pgs: 32 unknown, 8 creating+peering, 174 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:14:00.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:59.360094+0000 mon.a (mon.0) 3096 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:00.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:13:59.360147+0000 mon.a (mon.0) 3097 : cluster [DBG] osdmap e603: 8 total, 8 up, 6 in 2024-02-19T21:14:00.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:13:59.360936+0000 mon.a (mon.0) 3098 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-102"}]: dispatch 2024-02-19T21:14:00.781 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:14:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:14:00.781 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:59.279214+0000 mgr.y (mgr.14550) 523 : cluster [DBG] pgmap v903: 214 pgs: 32 unknown, 8 creating+peering, 174 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:14:00.781 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:59.360094+0000 mon.a (mon.0) 3096 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:00.781 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:13:59.360147+0000 mon.a (mon.0) 3097 : cluster [DBG] osdmap e603: 8 total, 8 up, 6 in 2024-02-19T21:14:00.781 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:13:59.360936+0000 mon.a (mon.0) 3098 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-102"}]: dispatch 2024-02-19T21:14:00.790 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:14:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:00.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:14:00.799 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-19T21:14:00.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.817+0000 7f65163fc700 1 -- 172.21.15.163:0/1101210619 >> v1:172.21.15.163:6790/0 conn(0x7f65100fbfd0 legacy=0x7f65100fc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:00.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.817+0000 7f65163fc700 1 -- 172.21.15.163:0/1101210619 shutdown_connections 2024-02-19T21:14:00.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.817+0000 7f65163fc700 1 -- 172.21.15.163:0/1101210619 >> 172.21.15.163:0/1101210619 conn(0x7f651006cc00 msgr2=0x7f65100f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:00.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.817+0000 7f65163fc700 1 -- 172.21.15.163:0/1101210619 shutdown_connections 2024-02-19T21:14:00.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.817+0000 7f65163fc700 1 -- 172.21.15.163:0/1101210619 wait complete. 2024-02-19T21:14:00.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.818+0000 7f65163fc700 1 librados: starting msgr at 2024-02-19T21:14:00.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.818+0000 7f65163fc700 1 librados: starting objecter 2024-02-19T21:14:00.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.818+0000 7f65163fc700 1 Processor -- start 2024-02-19T21:14:00.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.818+0000 7f65163fc700 1 -- start start 2024-02-19T21:14:00.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.818+0000 7f65163fc700 1 librados: setting wanted keys 2024-02-19T21:14:00.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.818+0000 7f65163fc700 1 librados: calling monclient init 2024-02-19T21:14:00.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.818+0000 7f65163fc700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f651010e0c0 con 0x7f65100fbfd0 2024-02-19T21:14:00.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.818+0000 7f65163fc700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f651011e9c0 con 0x7f65100fee20 2024-02-19T21:14:00.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.818+0000 7f65163fc700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f651011fb60 con 0x7f6510102930 2024-02-19T21:14:00.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.818+0000 7f650ffff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f65100fbfd0 0x7f6510106fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:49196/0 (socket says 172.21.15.163:49196) 2024-02-19T21:14:00.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.818+0000 7f650ffff700 1 -- 172.21.15.163:0/789416589 learned_addr learned my addr 172.21.15.163:0/789416589 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:00.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.818+0000 7f6514999700 1 --1- 172.21.15.163:0/789416589 >> v1:172.21.15.163:6790/0 conn(0x7f6510102930 0x7f651011c290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45620/0 (socket says 172.21.15.163:45620) 2024-02-19T21:14:00.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 10 client.?.objecter ms_handle_connect 0x7f6510102930 2024-02-19T21:14:00.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:00.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 10 client.?.objecter ms_handle_connect 0x7f65100fbfd0 2024-02-19T21:14:00.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:00.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 10 client.?.objecter ms_handle_connect 0x7f65100fee20 2024-02-19T21:14:00.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:00.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251250146 0 0) 0x7f651011fb60 con 0x7f6510102930 2024-02-19T21:14:00.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f8004a60 con 0x7f6510102930 2024-02-19T21:14:00.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578007025 0 0) 0x7f651010e0c0 con 0x7f65100fbfd0 2024-02-19T21:14:00.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f651011fb60 con 0x7f65100fbfd0 2024-02-19T21:14:00.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745520980 0 0) 0x7f651011e9c0 con 0x7f65100fee20 2024-02-19T21:14:00.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f651010e0c0 con 0x7f65100fee20 2024-02-19T21:14:00.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3030533094 0 0) 0x7f64f8004a60 con 0x7f6510102930 2024-02-19T21:14:00.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f651011e9c0 con 0x7f6510102930 2024-02-19T21:14:00.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6508001c00 con 0x7f6510102930 2024-02-19T21:14:00.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 573122514 0 0) 0x7f651010e0c0 con 0x7f65100fee20 2024-02-19T21:14:00.822 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64f8004a60 con 0x7f65100fee20 2024-02-19T21:14:00.822 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6504001620 con 0x7f65100fee20 2024-02-19T21:14:00.822 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1885798204 0 0) 0x7f651011fb60 con 0x7f65100fbfd0 2024-02-19T21:14:00.822 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f651010e0c0 con 0x7f65100fbfd0 2024-02-19T21:14:00.822 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f65000020f0 con 0x7f65100fbfd0 2024-02-19T21:14:00.822 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2565391417 0 0) 0x7f651011e9c0 con 0x7f6510102930 2024-02-19T21:14:00.822 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 >> v1:172.21.15.193:6789/0 conn(0x7f65100fee20 legacy=0x7f6510118b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:00.822 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 >> v1:172.21.15.163:6789/0 conn(0x7f65100fbfd0 legacy=0x7f6510106fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:00.822 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6510120d40 con 0x7f6510102930 2024-02-19T21:14:00.822 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f65163fc700 1 -- 172.21.15.163:0/789416589 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f651011da10 con 0x7f6510102930 2024-02-19T21:14:00.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f65163fc700 10 client.50270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:00.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f65163fc700 1 -- 172.21.15.163:0/789416589 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f651011de80 con 0x7f6510102930 2024-02-19T21:14:00.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.819+0000 7f65163fc700 1 librados: init done 2024-02-19T21:14:00.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.820+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6508002700 con 0x7f6510102930 2024-02-19T21:14:00.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.820+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6508005e00 con 0x7f6510102930 2024-02-19T21:14:00.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.821+0000 7f65163fc700 1 -- 172.21.15.163:0/789416589 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64fc004f20 con 0x7f6510102930 2024-02-19T21:14:00.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.821+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6508005f60 con 0x7f6510102930 2024-02-19T21:14:00.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.822+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(604..604 src has 434..604) v4 ==== 8500+0+0 (unknown 1204434978 0 0) 0x7f65080790a0 con 0x7f6510102930 2024-02-19T21:14:00.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.822+0000 7f650d7fa700 10 client.50270.objecter ms_dispatch 0x7f6510001550 osd_map(604..604 src has 434..604) v4 2024-02-19T21:14:00.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.822+0000 7f650d7fa700 3 client.50270.objecter handle_osd_map got epochs [604,604] > 0 2024-02-19T21:14:00.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.822+0000 7f650d7fa700 3 client.50270.objecter handle_osd_map decoding full epoch 604 2024-02-19T21:14:00.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.822+0000 7f650d7fa700 20 client.50270.objecter dump_active .. 0 homeless 2024-02-19T21:14:00.824 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.822+0000 7f650d7fa700 10 client.50270.objecter ms_handle_connect 0x7f64f8063d50 2024-02-19T21:14:00.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:00.828+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f650804c300 con 0x7f6510102930 2024-02-19T21:14:00.841 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:00.866 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:00.894 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:00.923 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:01.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:01.027+0000 7f65163fc700 1 -- 172.21.15.163:0/789416589 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f64fc0051d0 con 0x7f6510102930 2024-02-19T21:14:01.105 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:14:01.122 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:01.125 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:01.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:01.226 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:01.253 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:01.352 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:01.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:01.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:01.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:01.377+0000 7f650d7fa700 1 -- 172.21.15.163:0/789416589 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v605) v1 ==== 97+0+0 (unknown 4111240410 0 0) 0x7f650804a710 con 0x7f6510102930 2024-02-19T21:14:01.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:01.380+0000 7f65163fc700 10 librados: watch_flush enter 2024-02-19T21:14:01.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:01.380+0000 7f65163fc700 10 librados: watch_flush exit 2024-02-19T21:14:01.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:01.380+0000 7f65163fc700 20 client.50270.objecter shutdown clearing up homeless session... 2024-02-19T21:14:01.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:01.380+0000 7f65163fc700 10 client.50270.objecter successfully canceled tick 2024-02-19T21:14:01.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:01.380+0000 7f65163fc700 1 -- 172.21.15.163:0/789416589 >> v1:172.21.15.163:6800/1552514681 conn(0x7f64f8063d50 legacy=0x7f64f8066210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:01.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:01.380+0000 7f65163fc700 1 -- 172.21.15.163:0/789416589 >> v1:172.21.15.163:6790/0 conn(0x7f6510102930 legacy=0x7f651011c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:01.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:01.381+0000 7f65163fc700 1 -- 172.21.15.163:0/789416589 shutdown_connections 2024-02-19T21:14:01.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:01.381+0000 7f65163fc700 1 -- 172.21.15.163:0/789416589 >> 172.21.15.163:0/789416589 conn(0x7f651006cc00 msgr2=0x7f65100fcdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:01.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:01.381+0000 7f65163fc700 1 -- 172.21.15.163:0/789416589 shutdown_connections 2024-02-19T21:14:01.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:01.381+0000 7f65163fc700 1 -- 172.21.15.163:0/789416589 wait complete. 2024-02-19T21:14:01.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:01.382+0000 7f65163fc700 1 librados: shutdown 2024-02-19T21:14:01.385 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:14:01.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:01.487 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:01.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:00.360559+0000 mon.a (mon.0) 3099 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-102"}]': finished 2024-02-19T21:14:01.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:00.360613+0000 mon.a (mon.0) 3100 : cluster [DBG] osdmap e604: 8 total, 8 up, 6 in 2024-02-19T21:14:01.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:00.611288+0000 mgr.y (mgr.14550) 524 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:01.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:01.028478+0000 mon.c (mon.2) 415 : audit [INF] from='client.? v1:172.21.15.163:0/789416589' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:01.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:01.028764+0000 mon.a (mon.0) 3101 : audit [INF] from='client.50270 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:01.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:00.360559+0000 mon.a (mon.0) 3099 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-102"}]': finished 2024-02-19T21:14:01.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:00.360613+0000 mon.a (mon.0) 3100 : cluster [DBG] osdmap e604: 8 total, 8 up, 6 in 2024-02-19T21:14:01.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:00.611288+0000 mgr.y (mgr.14550) 524 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:01.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:01.028478+0000 mon.c (mon.2) 415 : audit [INF] from='client.? v1:172.21.15.163:0/789416589' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:01.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:01.028764+0000 mon.a (mon.0) 3101 : audit [INF] from='client.50270 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:01.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:00.360559+0000 mon.a (mon.0) 3099 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-102"}]': finished 2024-02-19T21:14:01.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:00.360613+0000 mon.a (mon.0) 3100 : cluster [DBG] osdmap e604: 8 total, 8 up, 6 in 2024-02-19T21:14:01.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:00.611288+0000 mgr.y (mgr.14550) 524 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:01.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:01.028478+0000 mon.c (mon.2) 415 : audit [INF] from='client.? v1:172.21.15.163:0/789416589' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:01.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:01.028764+0000 mon.a (mon.0) 3101 : audit [INF] from='client.50270 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:02.136 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:02.138 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:02.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:02.239 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:14:02.271 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:02.293 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:14:02.296 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:02.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:14:02.398 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:14:02.424 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:02.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:01.279645+0000 mgr.y (mgr.14550) 525 : cluster [DBG] pgmap v906: 214 pgs: 214 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-19T21:14:02.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:01.366079+0000 mon.a (mon.0) 3102 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:14:02.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:01.366883+0000 mon.a (mon.0) 3103 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:14:02.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:01.375010+0000 mon.a (mon.0) 3104 : audit [INF] from='client.50270 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:14:02.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:01.375062+0000 mon.a (mon.0) 3105 : cluster [DBG] osdmap e605: 8 total, 8 up, 6 in 2024-02-19T21:14:02.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:01.279645+0000 mgr.y (mgr.14550) 525 : cluster [DBG] pgmap v906: 214 pgs: 214 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-19T21:14:02.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:01.366079+0000 mon.a (mon.0) 3102 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:14:02.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:01.366883+0000 mon.a (mon.0) 3103 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:14:02.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:01.375010+0000 mon.a (mon.0) 3104 : audit [INF] from='client.50270 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:14:02.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:01.375062+0000 mon.a (mon.0) 3105 : cluster [DBG] osdmap e605: 8 total, 8 up, 6 in 2024-02-19T21:14:02.521 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:01.279645+0000 mgr.y (mgr.14550) 525 : cluster [DBG] pgmap v906: 214 pgs: 214 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-19T21:14:02.521 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:01.366079+0000 mon.a (mon.0) 3102 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:14:02.521 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:01.366883+0000 mon.a (mon.0) 3103 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:14:02.521 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:01.375010+0000 mon.a (mon.0) 3104 : audit [INF] from='client.50270 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:14:02.521 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:01.375062+0000 mon.a (mon.0) 3105 : cluster [DBG] osdmap e605: 8 total, 8 up, 6 in 2024-02-19T21:14:02.522 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:14:02.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:02.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:14:02.627 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:14:02.656 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:02.811 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:14:02.812 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:14:02.812 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-02-19T21:14:02.812 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:14:03.381 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-19T21:14:03.469 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:02.380178+0000 mon.a (mon.0) 3106 : cluster [DBG] osdmap e606: 8 total, 8 up, 6 in 2024-02-19T21:14:03.470 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:02.383758+0000 mon.a (mon.0) 3107 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-104","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:14:03.470 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:02.380178+0000 mon.a (mon.0) 3106 : cluster [DBG] osdmap e606: 8 total, 8 up, 6 in 2024-02-19T21:14:03.470 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:02.383758+0000 mon.a (mon.0) 3107 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-104","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:14:03.662 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:14:03.705 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:14:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:03.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:14:03.747 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:14:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:14:03] "GET /metrics HTTP/1.1" 200 186270 "" "Prometheus/2.33.4" 2024-02-19T21:14:03.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.381+0000 7f8c92a53700 1 -- 172.21.15.163:0/4106803388 >> v1:172.21.15.163:6789/0 conn(0x7f8c8c104300 legacy=0x7f8c8c1067a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:03.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.381+0000 7f8c92a53700 1 -- 172.21.15.163:0/4106803388 shutdown_connections 2024-02-19T21:14:03.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.381+0000 7f8c92a53700 1 -- 172.21.15.163:0/4106803388 >> 172.21.15.163:0/4106803388 conn(0x7f8c8c0f2470 msgr2=0x7f8c8c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:03.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.381+0000 7f8c92a53700 1 -- 172.21.15.163:0/4106803388 shutdown_connections 2024-02-19T21:14:03.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.381+0000 7f8c92a53700 1 -- 172.21.15.163:0/4106803388 wait complete. 2024-02-19T21:14:03.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.382+0000 7f8c92a53700 1 librados: starting msgr at 2024-02-19T21:14:03.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.382+0000 7f8c92a53700 1 librados: starting objecter 2024-02-19T21:14:03.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.382+0000 7f8c92a53700 1 Processor -- start 2024-02-19T21:14:03.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.382+0000 7f8c92a53700 1 -- start start 2024-02-19T21:14:03.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.382+0000 7f8c92a53700 1 librados: setting wanted keys 2024-02-19T21:14:03.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.382+0000 7f8c92a53700 1 librados: calling monclient init 2024-02-19T21:14:03.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.382+0000 7f8c92a53700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8c8c0f86b0 con 0x7f8c8c0f94f0 2024-02-19T21:14:03.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.382+0000 7f8c92a53700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8c8c11ab60 con 0x7f8c8c104300 2024-02-19T21:14:03.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.382+0000 7f8c92a53700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8c8c11bd00 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.382+0000 7f8c8b7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8c8c0f94f0 0x7f8c8c114d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:49228/0 (socket says 172.21.15.163:49228) 2024-02-19T21:14:03.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.382+0000 7f8c8b7fe700 1 -- 172.21.15.163:0/341391428 learned_addr learned my addr 172.21.15.163:0/341391428 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:03.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.383+0000 7f8c897fa700 10 client.?.objecter ms_handle_connect 0x7f8c8c0f94f0 2024-02-19T21:14:03.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.383+0000 7f8c897fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:03.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.383+0000 7f8c897fa700 10 client.?.objecter ms_handle_connect 0x7f8c8c104300 2024-02-19T21:14:03.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.383+0000 7f8c897fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:03.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.383+0000 7f8c897fa700 10 client.?.objecter ms_handle_connect 0x7f8c8c0f8a00 2024-02-19T21:14:03.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.383+0000 7f8c897fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:03.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.392+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008635994 0 0) 0x7f8c8c11ab60 con 0x7f8c8c104300 2024-02-19T21:14:03.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.392+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c74002220 con 0x7f8c8c104300 2024-02-19T21:14:03.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.392+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709439733 0 0) 0x7f8c8c11bd00 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.392+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c8c11ab60 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.392+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713553582 0 0) 0x7f8c8c0f86b0 con 0x7f8c8c0f94f0 2024-02-19T21:14:03.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.392+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c8c11bd00 con 0x7f8c8c0f94f0 2024-02-19T21:14:03.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.392+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4258344373 0 0) 0x7f8c8c11ab60 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.392+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c8c0f86b0 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.392+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8c7c001800 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.393+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1992757892 0 0) 0x7f8c8c0f86b0 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.393+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 >> v1:172.21.15.193:6789/0 conn(0x7f8c8c104300 legacy=0x7f8c8c118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:03.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.393+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 >> v1:172.21.15.163:6789/0 conn(0x7f8c8c0f94f0 legacy=0x7f8c8c114d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:03.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.393+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8c11cea0 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.393+0000 7f8c92a53700 1 -- 172.21.15.163:0/341391428 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c8c11ad50 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.393+0000 7f8c92a53700 10 client.50273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:03.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.393+0000 7f8c92a53700 1 -- 172.21.15.163:0/341391428 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c8c11b1c0 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.393+0000 7f8c92a53700 1 librados: init done 2024-02-19T21:14:03.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.394+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8c7c001ec0 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.394+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8c7c003230 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.395+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f8c7c016140 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.395+0000 7f8c92a53700 1 -- 172.21.15.163:0/341391428 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c78004f50 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.396+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(607..607 src has 434..607) v4 ==== 8513+0+0 (unknown 2379864776 0 0) 0x7f8c7c0765f0 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.396+0000 7f8c897fa700 10 client.50273.objecter ms_dispatch 0x7f8c8c001550 osd_map(607..607 src has 434..607) v4 2024-02-19T21:14:03.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.396+0000 7f8c897fa700 3 client.50273.objecter handle_osd_map got epochs [607,607] > 0 2024-02-19T21:14:03.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.396+0000 7f8c897fa700 3 client.50273.objecter handle_osd_map decoding full epoch 607 2024-02-19T21:14:03.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.396+0000 7f8c897fa700 20 client.50273.objecter dump_active .. 0 homeless 2024-02-19T21:14:03.794 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:14:03.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.402+0000 7f8c897fa700 10 client.50273.objecter ms_handle_connect 0x7f8c74063cf0 2024-02-19T21:14:03.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.404+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8c7c04a6e0 con 0x7f8c8c0f8a00 2024-02-19T21:14:03.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.569+0000 7f8c92a53700 1 -- 172.21.15.163:0/341391428 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c78000bc0 con 0x7f8c74063cf0 2024-02-19T21:14:03.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.583+0000 7f8c897fa700 1 -- 172.21.15.163:0/341391428 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+536090 (unknown 2965378022 0 2280164196) 0x7f8c78000bc0 con 0x7f8c74063cf0 2024-02-19T21:14:03.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.792+0000 7f8c92a53700 10 librados: watch_flush enter 2024-02-19T21:14:03.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.793+0000 7f8c92a53700 10 librados: watch_flush exit 2024-02-19T21:14:03.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.793+0000 7f8c92a53700 20 client.50273.objecter shutdown clearing up homeless session... 2024-02-19T21:14:03.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.793+0000 7f8c92a53700 10 client.50273.objecter successfully canceled tick 2024-02-19T21:14:03.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.793+0000 7f8c92a53700 1 -- 172.21.15.163:0/341391428 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8c74063cf0 legacy=0x7f8c740661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:03.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.793+0000 7f8c92a53700 1 -- 172.21.15.163:0/341391428 >> v1:172.21.15.163:6790/0 conn(0x7f8c8c0f8a00 legacy=0x7f8c8c1026a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:03.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.793+0000 7f8c92a53700 1 -- 172.21.15.163:0/341391428 shutdown_connections 2024-02-19T21:14:03.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.793+0000 7f8c92a53700 1 -- 172.21.15.163:0/341391428 >> 172.21.15.163:0/341391428 conn(0x7f8c8c0f2470 msgr2=0x7f8c8c1038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:03.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.793+0000 7f8c92a53700 1 -- 172.21.15.163:0/341391428 shutdown_connections 2024-02-19T21:14:03.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.793+0000 7f8c92a53700 1 -- 172.21.15.163:0/341391428 wait complete. 2024-02-19T21:14:03.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:03.793+0000 7f8c92a53700 1 librados: shutdown 2024-02-19T21:14:03.806 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:14:03.808 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:03.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:02.380178+0000 mon.a (mon.0) 3106 : cluster [DBG] osdmap e606: 8 total, 8 up, 6 in 2024-02-19T21:14:03.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:02.383758+0000 mon.a (mon.0) 3107 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-104","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:14:03.873 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:03.901 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:03.912 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:03.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:04.014 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:04.041 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:04.078 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:04.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:04.182 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:14:04.211 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:04.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:03.280064+0000 mgr.y (mgr.14550) 526 : cluster [DBG] pgmap v909: 214 pgs: 32 unknown, 182 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:14:04.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:03.388742+0000 mon.a (mon.0) 3108 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-104","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:14:04.436 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:03.388824+0000 mon.a (mon.0) 3109 : cluster [DBG] osdmap e607: 8 total, 8 up, 6 in 2024-02-19T21:14:04.436 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:03.402808+0000 mon.a (mon.0) 3110 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-104", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:14:04.436 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:03.570909+0000 mgr.y (mgr.14550) 527 : audit [DBG] from='client.50273 v1:172.21.15.163:0/341391428' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:14:04.436 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:04.349003+0000 mon.a (mon.0) 3111 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-104", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:14:04.436 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:04.349062+0000 mon.a (mon.0) 3112 : cluster [DBG] osdmap e608: 8 total, 8 up, 6 in 2024-02-19T21:14:04.560 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:14:04.579 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:03.280064+0000 mgr.y (mgr.14550) 526 : cluster [DBG] pgmap v909: 214 pgs: 32 unknown, 182 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:14:04.579 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:03.388742+0000 mon.a (mon.0) 3108 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-104","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:14:04.579 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:03.388824+0000 mon.a (mon.0) 3109 : cluster [DBG] osdmap e607: 8 total, 8 up, 6 in 2024-02-19T21:14:04.579 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:03.402808+0000 mon.a (mon.0) 3110 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-104", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:14:04.579 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:03.570909+0000 mgr.y (mgr.14550) 527 : audit [DBG] from='client.50273 v1:172.21.15.163:0/341391428' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:14:04.579 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:04.349003+0000 mon.a (mon.0) 3111 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-104", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:14:04.579 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:04.349062+0000 mon.a (mon.0) 3112 : cluster [DBG] osdmap e608: 8 total, 8 up, 6 in 2024-02-19T21:14:04.579 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:04.351212+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-104"}]: dispatch 2024-02-19T21:14:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:04.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:14:04.681 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:14:04.709 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:04.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:04.351212+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-104"}]: dispatch 2024-02-19T21:14:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:03.280064+0000 mgr.y (mgr.14550) 526 : cluster [DBG] pgmap v909: 214 pgs: 32 unknown, 182 active+clean; 5.2 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:14:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:03.388742+0000 mon.a (mon.0) 3108 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-104","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:14:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:03.388824+0000 mon.a (mon.0) 3109 : cluster [DBG] osdmap e607: 8 total, 8 up, 6 in 2024-02-19T21:14:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:03.402808+0000 mon.a (mon.0) 3110 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-104", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:14:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:03.570909+0000 mgr.y (mgr.14550) 527 : audit [DBG] from='client.50273 v1:172.21.15.163:0/341391428' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:14:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:04.349003+0000 mon.a (mon.0) 3111 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-104", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:14:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:04.349062+0000 mon.a (mon.0) 3112 : cluster [DBG] osdmap e608: 8 total, 8 up, 6 in 2024-02-19T21:14:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:04.351212+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-104"}]: dispatch 2024-02-19T21:14:04.980 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":910,"stamp":"2024-02-19T21:14:03.385268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5369,"num_objects":194,"num_object_clones":0,"num_object_copies":582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":1879,"num_read_kb":1631,"num_write":721,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":11084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":820,"ondisk_log_size":820,"up":546,"acting":546,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":628,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":1843392,"kb_used_data":19088,"kb_used_omap":0,"kb_used_meta":1824256,"kb_avail":560627520,"statfs":{"total":575970213888,"available":574082580480,"internally_reserved":0,"allocated":19546112,"data_stored":9594750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1868038144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.028456"},"pg_stats":[{"pgid":"236.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"236.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.7","version":"0'0","reported_seq":13,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356388+0000","last_change":"2024-02-19T21:13:55.359014+0000","last_active":"2024-02-19T21:13:58.356388+0000","last_peered":"2024-02-19T21:13:58.356388+0000","last_clean":"2024-02-19T21:13:58.356388+0000","last_became_active":"2024-02-19T21:13:55.358727+0000","last_became_peered":"2024-02-19T21:13:55.358727+0000","last_unstale":"2024-02-19T21:13:58.356388+0000","last_undegraded":"2024-02-19T21:13:58.356388+0000","last_fullsized":"2024-02-19T21:13:58.356388+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"236.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.6","version":"0'0","reported_seq":13,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.368404+0000","last_change":"2024-02-19T21:13:55.353604+0000","last_active":"2024-02-19T21:13:58.368404+0000","last_peered":"2024-02-19T21:13:58.368404+0000","last_clean":"2024-02-19T21:13:58.368404+0000","last_became_active":"2024-02-19T21:13:55.353296+0000","last_became_peered":"2024-02-19T21:13:55.353296+0000","last_unstale":"2024-02-19T21:13:58.368404+0000","last_undegraded":"2024-02-19T21:13:58.368404+0000","last_fullsized":"2024-02-19T21:13:58.368404+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"236.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.5","version":"0'0","reported_seq":13,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.363505+0000","last_change":"2024-02-19T21:13:55.357065+0000","last_active":"2024-02-19T21:13:58.363505+0000","last_peered":"2024-02-19T21:13:58.363505+0000","last_clean":"2024-02-19T21:13:58.363505+0000","last_became_active":"2024-02-19T21:13:55.356750+0000","last_became_peered":"2024-02-19T21:13:55.356750+0000","last_unstale":"2024-02-19T21:13:58.363505+0000","last_undegraded":"2024-02-19T21:13:58.363505+0000","last_fullsized":"2024-02-19T21:13:58.363505+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"236.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.4","version":"0'0","reported_seq":13,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349002+0000","last_change":"2024-02-19T21:13:55.357041+0000","last_active":"2024-02-19T21:13:58.349002+0000","last_peered":"2024-02-19T21:13:58.349002+0000","last_clean":"2024-02-19T21:13:58.349002+0000","last_became_active":"2024-02-19T21:13:55.356730+0000","last_became_peered":"2024-02-19T21:13:55.356730+0000","last_unstale":"2024-02-19T21:13:58.349002+0000","last_undegraded":"2024-02-19T21:13:58.349002+0000","last_fullsized":"2024-02-19T21:13:58.349002+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"236.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.3","version":"0'0","reported_seq":13,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367144+0000","last_change":"2024-02-19T21:13:55.352789+0000","last_active":"2024-02-19T21:13:58.367144+0000","last_peered":"2024-02-19T21:13:58.367144+0000","last_clean":"2024-02-19T21:13:58.367144+0000","last_became_active":"2024-02-19T21:13:55.351863+0000","last_became_peered":"2024-02-19T21:13:55.351863+0000","last_unstale":"2024-02-19T21:13:58.367144+0000","last_undegraded":"2024-02-19T21:13:58.367144+0000","last_fullsized":"2024-02-19T21:13:58.367144+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"236.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.2","version":"0'0","reported_seq":13,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.366950+0000","last_change":"2024-02-19T21:13:55.355694+0000","last_active":"2024-02-19T21:13:58.366950+0000","last_peered":"2024-02-19T21:13:58.366950+0000","last_clean":"2024-02-19T21:13:58.366950+0000","last_became_active":"2024-02-19T21:13:55.354324+0000","last_became_peered":"2024-02-19T21:13:55.354324+0000","last_unstale":"2024-02-19T21:13:58.366950+0000","last_undegraded":"2024-02-19T21:13:58.366950+0000","last_fullsized":"2024-02-19T21:13:58.366950+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"236.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.1","version":"600'1","reported_seq":15,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.357359+0000","last_change":"2024-02-19T21:13:55.355214+0000","last_active":"2024-02-19T21:13:58.357359+0000","last_peered":"2024-02-19T21:13:58.357359+0000","last_clean":"2024-02-19T21:13:58.357359+0000","last_became_active":"2024-02-19T21:13:55.354926+0000","last_became_peered":"2024-02-19T21:13:55.354926+0000","last_unstale":"2024-02-19T21:13:58.357359+0000","last_undegraded":"2024-02-19T21:13:58.357359+0000","last_fullsized":"2024-02-19T21:13:58.357359+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"236.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:14:02.372050+0000","last_change":"2024-02-19T21:14:02.372050+0000","last_active":"2024-02-19T21:14:02.372050+0000","last_peered":"2024-02-19T21:14:02.372050+0000","last_clean":"2024-02-19T21:14:02.372050+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:14:02.372050+0000","last_undegraded":"2024-02-19T21:14:02.372050+0000","last_fullsized":"2024-02-19T21:14:02.372050+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:02.372050+0000","last_clean_scrub_stamp":"2024-02-19T21:14:02.372050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.0","version":"0'0","reported_seq":13,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.350488+0000","last_change":"2024-02-19T21:13:55.355076+0000","last_active":"2024-02-19T21:13:58.350488+0000","last_peered":"2024-02-19T21:13:58.350488+0000","last_clean":"2024-02-19T21:13:58.350488+0000","last_became_active":"2024-02-19T21:13:55.354765+0000","last_became_peered":"2024-02-19T21:13:55.354765+0000","last_unstale":"2024-02-19T21:13:58.350488+0000","last_undegraded":"2024-02-19T21:13:58.350488+0000","last_fullsized":"2024-02-19T21:13:58.350488+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":606,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.348147+0000","last_change":"2024-02-19T21:12:09.321398+0000","last_active":"2024-02-19T21:13:58.348147+0000","last_peered":"2024-02-19T21:13:58.348147+0000","last_clean":"2024-02-19T21:13:58.348147+0000","last_became_active":"2024-02-19T21:12:09.320924+0000","last_became_peered":"2024-02-19T21:12:09.320924+0000","last_unstale":"2024-02-19T21:13:58.348147+0000","last_undegraded":"2024-02-19T21:13:58.348147+0000","last_fullsized":"2024-02-19T21:13:58.348147+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.738600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.738600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":136,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.348197+0000","last_change":"2024-02-19T21:11:50.304127+0000","last_active":"2024-02-19T21:13:58.348197+0000","last_peered":"2024-02-19T21:13:58.348197+0000","last_clean":"2024-02-19T21:13:58.348197+0000","last_became_active":"2024-02-19T21:11:50.303790+0000","last_became_peered":"2024-02-19T21:11:50.303790+0000","last_unstale":"2024-02-19T21:13:58.348197+0000","last_undegraded":"2024-02-19T21:13:58.348197+0000","last_fullsized":"2024-02-19T21:13:58.348197+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:42.529489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:42.529489+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.25","version":"0'0","reported_seq":644,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351445+0000","last_change":"2024-02-19T21:13:47.724414+0000","last_active":"2024-02-19T21:13:58.351445+0000","last_peered":"2024-02-19T21:13:58.351445+0000","last_clean":"2024-02-19T21:13:58.351445+0000","last_became_active":"2024-02-19T21:12:09.349271+0000","last_became_peered":"2024-02-19T21:12:09.349271+0000","last_unstale":"2024-02-19T21:13:58.351445+0000","last_undegraded":"2024-02-19T21:13:58.351445+0000","last_fullsized":"2024-02-19T21:13:58.351445+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:47.724347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:47.724347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":617,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.432315+0000","last_change":"2024-02-19T21:13:55.034685+0000","last_active":"2024-02-19T21:13:57.432315+0000","last_peered":"2024-02-19T21:13:57.432315+0000","last_clean":"2024-02-19T21:13:57.432315+0000","last_became_active":"2024-02-19T21:12:09.339310+0000","last_became_peered":"2024-02-19T21:12:09.339310+0000","last_unstale":"2024-02-19T21:13:57.432315+0000","last_undegraded":"2024-02-19T21:13:57.432315+0000","last_fullsized":"2024-02-19T21:13:57.432315+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:55.034623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:13:55.034623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":608,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356517+0000","last_change":"2024-02-19T21:13:36.863848+0000","last_active":"2024-02-19T21:13:58.356517+0000","last_peered":"2024-02-19T21:13:58.356517+0000","last_clean":"2024-02-19T21:13:58.356517+0000","last_became_active":"2024-02-19T21:11:50.313560+0000","last_became_peered":"2024-02-19T21:11:50.313560+0000","last_unstale":"2024-02-19T21:13:58.356517+0000","last_undegraded":"2024-02-19T21:13:58.356517+0000","last_fullsized":"2024-02-19T21:13:58.356517+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:36.863734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:36.863734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.26","version":"0'0","reported_seq":145,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356235+0000","last_change":"2024-02-19T21:13:47.748347+0000","last_active":"2024-02-19T21:13:58.356235+0000","last_peered":"2024-02-19T21:13:58.356235+0000","last_clean":"2024-02-19T21:13:58.356235+0000","last_became_active":"2024-02-19T21:12:09.338737+0000","last_became_peered":"2024-02-19T21:12:09.338737+0000","last_unstale":"2024-02-19T21:13:58.356235+0000","last_undegraded":"2024-02-19T21:13:58.356235+0000","last_fullsized":"2024-02-19T21:13:58.356235+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:47.748283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:47.748283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":650,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.363048+0000","last_change":"2024-02-19T21:13:45.580033+0000","last_active":"2024-02-19T21:13:58.363048+0000","last_peered":"2024-02-19T21:13:58.363048+0000","last_clean":"2024-02-19T21:13:58.363048+0000","last_became_active":"2024-02-19T21:12:09.342583+0000","last_became_peered":"2024-02-19T21:12:09.342583+0000","last_unstale":"2024-02-19T21:13:58.363048+0000","last_undegraded":"2024-02-19T21:13:58.363048+0000","last_fullsized":"2024-02-19T21:13:58.363048+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:45.579948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:45.579948+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.20","version":"397'2","reported_seq":665,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.355034+0000","last_change":"2024-02-19T21:12:43.380032+0000","last_active":"2024-02-19T21:13:58.355034+0000","last_peered":"2024-02-19T21:13:58.355034+0000","last_clean":"2024-02-19T21:13:58.355034+0000","last_became_active":"2024-02-19T21:12:43.379693+0000","last_became_peered":"2024-02-19T21:12:43.379693+0000","last_unstale":"2024-02-19T21:13:58.355034+0000","last_undegraded":"2024-02-19T21:13:58.355034+0000","last_fullsized":"2024-02-19T21:13:58.355034+0000","mapping_epoch":526,"log_start":"58'1","ondisk_log_start":"58'1","created":474,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":6,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:09:40.934474+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":437,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":145,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349112+0000","last_change":"2024-02-19T21:12:09.337725+0000","last_active":"2024-02-19T21:13:58.349112+0000","last_peered":"2024-02-19T21:13:58.349112+0000","last_clean":"2024-02-19T21:13:58.349112+0000","last_became_active":"2024-02-19T21:12:09.337337+0000","last_became_peered":"2024-02-19T21:12:09.337337+0000","last_unstale":"2024-02-19T21:13:58.349112+0000","last_undegraded":"2024-02-19T21:13:58.349112+0000","last_fullsized":"2024-02-19T21:13:58.349112+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.081993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.081993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":599,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351799+0000","last_change":"2024-02-19T21:13:52.701902+0000","last_active":"2024-02-19T21:13:58.351799+0000","last_peered":"2024-02-19T21:13:58.351799+0000","last_clean":"2024-02-19T21:13:58.351799+0000","last_became_active":"2024-02-19T21:12:09.352904+0000","last_became_peered":"2024-02-19T21:12:09.352904+0000","last_unstale":"2024-02-19T21:13:58.351799+0000","last_undegraded":"2024-02-19T21:13:58.351799+0000","last_fullsized":"2024-02-19T21:13:58.351799+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:52.701800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:13:52.701800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"493'12","reported_seq":768,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-19T21:13:59.358716+0000","last_change":"2024-02-19T21:12:18.452113+0000","last_active":"2024-02-19T21:13:59.358716+0000","last_peered":"2024-02-19T21:13:59.358716+0000","last_clean":"2024-02-19T21:13:59.358716+0000","last_became_active":"2024-02-19T21:12:09.332507+0000","last_became_peered":"2024-02-19T21:12:09.332507+0000","last_unstale":"2024-02-19T21:13:59.358716+0000","last_undegraded":"2024-02-19T21:13:59.358716+0000","last_fullsized":"2024-02-19T21:13:59.358716+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'12","last_scrub_stamp":"2024-02-19T21:12:18.452020+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:12:18.452020+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'28","reported_seq":639,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.366928+0000","last_change":"2024-02-19T21:12:10.211768+0000","last_active":"2024-02-19T21:13:58.366928+0000","last_peered":"2024-02-19T21:13:58.366928+0000","last_clean":"2024-02-19T21:13:58.366928+0000","last_became_active":"2024-02-19T21:12:09.341122+0000","last_became_peered":"2024-02-19T21:12:09.341122+0000","last_unstale":"2024-02-19T21:13:58.366928+0000","last_undegraded":"2024-02-19T21:13:58.366928+0000","last_fullsized":"2024-02-19T21:13:58.366928+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-19T21:11:47.051823+0000","last_deep_scrub":"473'28","last_deep_scrub_stamp":"2024-02-19T21:11:47.051823+0000","last_clean_scrub_stamp":"2024-02-19T21:11:47.051823+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":601,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.364855+0000","last_change":"2024-02-19T21:12:22.215650+0000","last_active":"2024-02-19T21:13:58.364855+0000","last_peered":"2024-02-19T21:13:58.364855+0000","last_clean":"2024-02-19T21:13:58.364855+0000","last_became_active":"2024-02-19T21:10:40.299425+0000","last_became_peered":"2024-02-19T21:10:40.299425+0000","last_unstale":"2024-02-19T21:13:58.364855+0000","last_undegraded":"2024-02-19T21:13:58.364855+0000","last_fullsized":"2024-02-19T21:13:58.364855+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:22.215548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_clean_scrub_stamp":"2024-02-19T21:12:22.215548+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":581,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367832+0000","last_change":"2024-02-19T21:12:51.469049+0000","last_active":"2024-02-19T21:13:58.367832+0000","last_peered":"2024-02-19T21:13:58.367832+0000","last_clean":"2024-02-19T21:13:58.367832+0000","last_became_active":"2024-02-19T21:12:09.326173+0000","last_became_peered":"2024-02-19T21:12:09.326173+0000","last_unstale":"2024-02-19T21:13:58.367832+0000","last_undegraded":"2024-02-19T21:13:58.367832+0000","last_fullsized":"2024-02-19T21:13:58.367832+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:51.468971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.468971+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":522,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.364905+0000","last_change":"2024-02-19T21:12:58.081198+0000","last_active":"2024-02-19T21:13:58.364905+0000","last_peered":"2024-02-19T21:13:58.364905+0000","last_clean":"2024-02-19T21:13:58.364905+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:13:58.364905+0000","last_undegraded":"2024-02-19T21:13:58.364905+0000","last_fullsized":"2024-02-19T21:13:58.364905+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:58.081106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:58.081106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'16","reported_seq":492,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.431501+0000","last_change":"2024-02-19T21:11:59.756726+0000","last_active":"2024-02-19T21:13:57.431501+0000","last_peered":"2024-02-19T21:13:57.431501+0000","last_clean":"2024-02-19T21:13:57.431501+0000","last_became_active":"2024-02-19T21:11:59.319397+0000","last_became_peered":"2024-02-19T21:11:59.319397+0000","last_unstale":"2024-02-19T21:13:57.431501+0000","last_undegraded":"2024-02-19T21:13:57.431501+0000","last_fullsized":"2024-02-19T21:13:57.431501+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'16","last_scrub_stamp":"2024-02-19T21:11:46.956424+0000","last_deep_scrub":"473'16","last_deep_scrub_stamp":"2024-02-19T21:11:46.956424+0000","last_clean_scrub_stamp":"2024-02-19T21:11:46.956424+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":607,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.432537+0000","last_change":"2024-02-19T21:12:06.016884+0000","last_active":"2024-02-19T21:13:57.432537+0000","last_peered":"2024-02-19T21:13:57.432537+0000","last_clean":"2024-02-19T21:13:57.432537+0000","last_became_active":"2024-02-19T21:10:41.312116+0000","last_became_peered":"2024-02-19T21:10:41.312116+0000","last_unstale":"2024-02-19T21:13:57.432537+0000","last_undegraded":"2024-02-19T21:13:57.432537+0000","last_fullsized":"2024-02-19T21:13:57.432537+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.016817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.016817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":606,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.364030+0000","last_change":"2024-02-19T21:13:05.193297+0000","last_active":"2024-02-19T21:13:58.364030+0000","last_peered":"2024-02-19T21:13:58.364030+0000","last_clean":"2024-02-19T21:13:58.364030+0000","last_became_active":"2024-02-19T21:10:41.305574+0000","last_became_peered":"2024-02-19T21:10:41.305574+0000","last_unstale":"2024-02-19T21:13:58.364030+0000","last_undegraded":"2024-02-19T21:13:58.364030+0000","last_fullsized":"2024-02-19T21:13:58.364030+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:05.193223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:05.193223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":406,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.355534+0000","last_change":"2024-02-19T21:11:59.323172+0000","last_active":"2024-02-19T21:13:58.355534+0000","last_peered":"2024-02-19T21:13:58.355534+0000","last_clean":"2024-02-19T21:13:58.355534+0000","last_became_active":"2024-02-19T21:11:59.322814+0000","last_became_peered":"2024-02-19T21:11:59.322814+0000","last_unstale":"2024-02-19T21:13:58.355534+0000","last_undegraded":"2024-02-19T21:13:58.355534+0000","last_fullsized":"2024-02-19T21:13:58.355534+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:34.504478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:34.504478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":517,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.368419+0000","last_change":"2024-02-19T21:10:57.130256+0000","last_active":"2024-02-19T21:13:58.368419+0000","last_peered":"2024-02-19T21:13:58.368419+0000","last_clean":"2024-02-19T21:13:58.368419+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:13:58.368419+0000","last_undegraded":"2024-02-19T21:13:58.368419+0000","last_fullsized":"2024-02-19T21:13:58.368419+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.130171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.130171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"459'35","reported_seq":620,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.431396+0000","last_change":"2024-02-19T21:12:10.552781+0000","last_active":"2024-02-19T21:13:57.431396+0000","last_peered":"2024-02-19T21:13:57.431396+0000","last_clean":"2024-02-19T21:13:57.431396+0000","last_became_active":"2024-02-19T21:12:09.333541+0000","last_became_peered":"2024-02-19T21:12:09.333541+0000","last_unstale":"2024-02-19T21:13:57.431396+0000","last_undegraded":"2024-02-19T21:13:57.431396+0000","last_fullsized":"2024-02-19T21:13:57.431396+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'19","last_scrub_stamp":"2024-02-19T21:11:04.345836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:04.345836+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":583,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.364734+0000","last_change":"2024-02-19T21:12:09.339263+0000","last_active":"2024-02-19T21:13:58.364734+0000","last_peered":"2024-02-19T21:13:58.364734+0000","last_clean":"2024-02-19T21:13:58.364734+0000","last_became_active":"2024-02-19T21:11:59.336426+0000","last_became_peered":"2024-02-19T21:11:59.336426+0000","last_unstale":"2024-02-19T21:13:58.364734+0000","last_undegraded":"2024-02-19T21:13:58.364734+0000","last_fullsized":"2024-02-19T21:13:58.364734+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:09.339147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:12:09.339147+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":576,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.363699+0000","last_change":"2024-02-19T21:12:16.244173+0000","last_active":"2024-02-19T21:13:58.363699+0000","last_peered":"2024-02-19T21:13:58.363699+0000","last_clean":"2024-02-19T21:13:58.363699+0000","last_became_active":"2024-02-19T21:11:59.340544+0000","last_became_peered":"2024-02-19T21:11:59.340544+0000","last_unstale":"2024-02-19T21:13:58.363699+0000","last_undegraded":"2024-02-19T21:13:58.363699+0000","last_fullsized":"2024-02-19T21:13:58.363699+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:16.244104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:16.244104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":577,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.355177+0000","last_change":"2024-02-19T21:11:33.469310+0000","last_active":"2024-02-19T21:13:58.355177+0000","last_peered":"2024-02-19T21:13:58.355177+0000","last_clean":"2024-02-19T21:13:58.355177+0000","last_became_active":"2024-02-19T21:08:04.265081+0000","last_became_peered":"2024-02-19T21:08:04.265081+0000","last_unstale":"2024-02-19T21:13:58.355177+0000","last_undegraded":"2024-02-19T21:13:58.355177+0000","last_fullsized":"2024-02-19T21:13:58.355177+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:33.469232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_clean_scrub_stamp":"2024-02-19T21:11:33.469232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":537,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356320+0000","last_change":"2024-02-19T21:12:55.491781+0000","last_active":"2024-02-19T21:13:58.356320+0000","last_peered":"2024-02-19T21:13:58.356320+0000","last_clean":"2024-02-19T21:13:58.356320+0000","last_became_active":"2024-02-19T21:10:41.317284+0000","last_became_peered":"2024-02-19T21:10:41.317284+0000","last_unstale":"2024-02-19T21:13:58.356320+0000","last_undegraded":"2024-02-19T21:13:58.356320+0000","last_fullsized":"2024-02-19T21:13:58.356320+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:55.491665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.491665+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"459'31","reported_seq":222,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351983+0000","last_change":"2024-02-19T21:11:44.421114+0000","last_active":"2024-02-19T21:13:58.351983+0000","last_peered":"2024-02-19T21:13:58.351983+0000","last_clean":"2024-02-19T21:13:58.351983+0000","last_became_active":"2024-02-19T21:10:41.313853+0000","last_became_peered":"2024-02-19T21:10:41.313853+0000","last_unstale":"2024-02-19T21:13:58.351983+0000","last_undegraded":"2024-02-19T21:13:58.351983+0000","last_fullsized":"2024-02-19T21:13:58.351983+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'31","last_scrub_stamp":"2024-02-19T21:11:44.421033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:44.421033+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"58'1","reported_seq":602,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.355968+0000","last_change":"2024-02-19T21:13:07.450202+0000","last_active":"2024-02-19T21:13:58.355968+0000","last_peered":"2024-02-19T21:13:58.355968+0000","last_clean":"2024-02-19T21:13:58.355968+0000","last_became_active":"2024-02-19T21:10:41.307517+0000","last_became_peered":"2024-02-19T21:10:41.307517+0000","last_unstale":"2024-02-19T21:13:58.355968+0000","last_undegraded":"2024-02-19T21:13:58.355968+0000","last_fullsized":"2024-02-19T21:13:58.355968+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'1","last_scrub_stamp":"2024-02-19T21:13:07.450123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.450123+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":603,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.432593+0000","last_change":"2024-02-19T21:12:58.838081+0000","last_active":"2024-02-19T21:13:57.432593+0000","last_peered":"2024-02-19T21:13:57.432593+0000","last_clean":"2024-02-19T21:13:57.432593+0000","last_became_active":"2024-02-19T21:12:09.369409+0000","last_became_peered":"2024-02-19T21:12:09.369409+0000","last_unstale":"2024-02-19T21:13:57.432593+0000","last_undegraded":"2024-02-19T21:13:57.432593+0000","last_fullsized":"2024-02-19T21:13:57.432593+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:58.837973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:58.837973+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":604,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.365043+0000","last_change":"2024-02-19T21:13:03.126485+0000","last_active":"2024-02-19T21:13:58.365043+0000","last_peered":"2024-02-19T21:13:58.365043+0000","last_clean":"2024-02-19T21:13:58.365043+0000","last_became_active":"2024-02-19T21:10:41.307577+0000","last_became_peered":"2024-02-19T21:10:41.307577+0000","last_unstale":"2024-02-19T21:13:58.365043+0000","last_undegraded":"2024-02-19T21:13:58.365043+0000","last_fullsized":"2024-02-19T21:13:58.365043+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:03.126414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:13:03.126414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":183,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351114+0000","last_change":"2024-02-19T21:10:50.424462+0000","last_active":"2024-02-19T21:13:58.351114+0000","last_peered":"2024-02-19T21:13:58.351114+0000","last_clean":"2024-02-19T21:13:58.351114+0000","last_became_active":"2024-02-19T21:10:41.302670+0000","last_became_peered":"2024-02-19T21:10:41.302670+0000","last_unstale":"2024-02-19T21:13:58.351114+0000","last_undegraded":"2024-02-19T21:13:58.351114+0000","last_fullsized":"2024-02-19T21:13:58.351114+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:50.424381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.424381+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"459'25","reported_seq":231,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351445+0000","last_change":"2024-02-19T21:12:51.420519+0000","last_active":"2024-02-19T21:13:58.351445+0000","last_peered":"2024-02-19T21:13:58.351445+0000","last_clean":"2024-02-19T21:13:58.351445+0000","last_became_active":"2024-02-19T21:12:09.408620+0000","last_became_peered":"2024-02-19T21:12:09.408620+0000","last_unstale":"2024-02-19T21:13:58.351445+0000","last_undegraded":"2024-02-19T21:13:58.351445+0000","last_fullsized":"2024-02-19T21:13:58.351445+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'25","last_scrub_stamp":"2024-02-19T21:12:51.420403+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.420403+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":592,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.431386+0000","last_change":"2024-02-19T21:13:38.073731+0000","last_active":"2024-02-19T21:13:57.431386+0000","last_peered":"2024-02-19T21:13:57.431386+0000","last_clean":"2024-02-19T21:13:57.431386+0000","last_became_active":"2024-02-19T21:08:04.268406+0000","last_became_peered":"2024-02-19T21:08:04.268406+0000","last_unstale":"2024-02-19T21:13:57.431386+0000","last_undegraded":"2024-02-19T21:13:57.431386+0000","last_fullsized":"2024-02-19T21:13:57.431386+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:38.073668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:38.073668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":731,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-19T21:13:59.359215+0000","last_change":"2024-02-19T21:12:21.202447+0000","last_active":"2024-02-19T21:13:59.359215+0000","last_peered":"2024-02-19T21:13:59.359215+0000","last_clean":"2024-02-19T21:13:59.359215+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:13:59.359215+0000","last_undegraded":"2024-02-19T21:13:59.359215+0000","last_fullsized":"2024-02-19T21:13:59.359215+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:12:21.202371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:21.202371+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":183,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351781+0000","last_change":"2024-02-19T21:12:21.380805+0000","last_active":"2024-02-19T21:13:58.351781+0000","last_peered":"2024-02-19T21:13:58.351781+0000","last_clean":"2024-02-19T21:13:58.351781+0000","last_became_active":"2024-02-19T21:10:41.298893+0000","last_became_peered":"2024-02-19T21:10:41.298893+0000","last_unstale":"2024-02-19T21:13:58.351781+0000","last_undegraded":"2024-02-19T21:13:58.351781+0000","last_fullsized":"2024-02-19T21:13:58.351781+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:21.380734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:12:21.380734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":330,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.348295+0000","last_change":"2024-02-19T21:12:16.339644+0000","last_active":"2024-02-19T21:13:58.348295+0000","last_peered":"2024-02-19T21:13:58.348295+0000","last_clean":"2024-02-19T21:13:58.348295+0000","last_became_active":"2024-02-19T21:11:59.318845+0000","last_became_peered":"2024-02-19T21:11:59.318845+0000","last_unstale":"2024-02-19T21:13:58.348295+0000","last_undegraded":"2024-02-19T21:13:58.348295+0000","last_fullsized":"2024-02-19T21:13:58.348295+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:16.339577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:16.339577+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"459'27","reported_seq":226,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.364818+0000","last_change":"2024-02-19T21:11:08.202280+0000","last_active":"2024-02-19T21:13:58.364818+0000","last_peered":"2024-02-19T21:13:58.364818+0000","last_clean":"2024-02-19T21:13:58.364818+0000","last_became_active":"2024-02-19T21:10:41.307576+0000","last_became_peered":"2024-02-19T21:10:41.307576+0000","last_unstale":"2024-02-19T21:13:58.364818+0000","last_undegraded":"2024-02-19T21:13:58.364818+0000","last_fullsized":"2024-02-19T21:13:58.364818+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:11:08.202208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:08.202208+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":629,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.430816+0000","last_change":"2024-02-19T21:12:40.370031+0000","last_active":"2024-02-19T21:13:57.430816+0000","last_peered":"2024-02-19T21:13:57.430816+0000","last_clean":"2024-02-19T21:13:57.430816+0000","last_became_active":"2024-02-19T21:12:40.369640+0000","last_became_peered":"2024-02-19T21:12:40.369640+0000","last_unstale":"2024-02-19T21:13:57.430816+0000","last_undegraded":"2024-02-19T21:13:57.430816+0000","last_fullsized":"2024-02-19T21:13:57.430816+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.738600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.738600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"483'11","reported_seq":241,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-19T21:13:59.616849+0000","last_change":"2024-02-19T21:12:56.450133+0000","last_active":"2024-02-19T21:13:59.616849+0000","last_peered":"2024-02-19T21:13:59.616849+0000","last_clean":"2024-02-19T21:13:59.616849+0000","last_became_active":"2024-02-19T21:11:59.336645+0000","last_became_peered":"2024-02-19T21:11:59.336645+0000","last_unstale":"2024-02-19T21:13:59.616849+0000","last_undegraded":"2024-02-19T21:13:59.616849+0000","last_fullsized":"2024-02-19T21:13:59.616849+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"483'11","last_scrub_stamp":"2024-02-19T21:12:56.450044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:56.450044+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":577,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367776+0000","last_change":"2024-02-19T21:12:54.437089+0000","last_active":"2024-02-19T21:13:58.367776+0000","last_peered":"2024-02-19T21:13:58.367776+0000","last_clean":"2024-02-19T21:13:58.367776+0000","last_became_active":"2024-02-19T21:11:59.344440+0000","last_became_peered":"2024-02-19T21:11:59.344440+0000","last_unstale":"2024-02-19T21:13:58.367776+0000","last_undegraded":"2024-02-19T21:13:58.367776+0000","last_fullsized":"2024-02-19T21:13:58.367776+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:54.436977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:54.436977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":526,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367362+0000","last_change":"2024-02-19T21:12:09.327574+0000","last_active":"2024-02-19T21:13:58.367362+0000","last_peered":"2024-02-19T21:13:58.367362+0000","last_clean":"2024-02-19T21:13:58.367362+0000","last_became_active":"2024-02-19T21:12:09.326909+0000","last_became_peered":"2024-02-19T21:12:09.326909+0000","last_unstale":"2024-02-19T21:13:58.367362+0000","last_undegraded":"2024-02-19T21:13:58.367362+0000","last_fullsized":"2024-02-19T21:13:58.367362+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.079353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.079353+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"459'19","reported_seq":230,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351605+0000","last_change":"2024-02-19T21:12:10.647686+0000","last_active":"2024-02-19T21:13:58.351605+0000","last_peered":"2024-02-19T21:13:58.351605+0000","last_clean":"2024-02-19T21:13:58.351605+0000","last_became_active":"2024-02-19T21:12:09.409065+0000","last_became_peered":"2024-02-19T21:12:09.409065+0000","last_unstale":"2024-02-19T21:13:58.351605+0000","last_undegraded":"2024-02-19T21:13:58.351605+0000","last_fullsized":"2024-02-19T21:13:58.351605+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:11:42.333781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:42.333781+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":620,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.355311+0000","last_change":"2024-02-19T21:13:37.829009+0000","last_active":"2024-02-19T21:13:58.355311+0000","last_peered":"2024-02-19T21:13:58.355311+0000","last_clean":"2024-02-19T21:13:58.355311+0000","last_became_active":"2024-02-19T21:11:59.331675+0000","last_became_peered":"2024-02-19T21:11:59.331675+0000","last_unstale":"2024-02-19T21:13:58.355311+0000","last_undegraded":"2024-02-19T21:13:58.355311+0000","last_fullsized":"2024-02-19T21:13:58.355311+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:37.828911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:37.828911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":591,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.368596+0000","last_change":"2024-02-19T21:12:09.350127+0000","last_active":"2024-02-19T21:13:58.368596+0000","last_peered":"2024-02-19T21:13:58.368596+0000","last_clean":"2024-02-19T21:13:58.368596+0000","last_became_active":"2024-02-19T21:12:09.349667+0000","last_became_peered":"2024-02-19T21:12:09.349667+0000","last_unstale":"2024-02-19T21:13:58.368596+0000","last_undegraded":"2024-02-19T21:13:58.368596+0000","last_fullsized":"2024-02-19T21:13:58.368596+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:58.096025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.096025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":328,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.350071+0000","last_change":"2024-02-19T21:13:51.217439+0000","last_active":"2024-02-19T21:13:58.350071+0000","last_peered":"2024-02-19T21:13:58.350071+0000","last_clean":"2024-02-19T21:13:58.350071+0000","last_became_active":"2024-02-19T21:12:09.333625+0000","last_became_peered":"2024-02-19T21:12:09.333625+0000","last_unstale":"2024-02-19T21:13:58.350071+0000","last_undegraded":"2024-02-19T21:13:58.350071+0000","last_fullsized":"2024-02-19T21:13:58.350071+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:51.217370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:51.217370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":323,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.348717+0000","last_change":"2024-02-19T21:12:09.348064+0000","last_active":"2024-02-19T21:13:58.348717+0000","last_peered":"2024-02-19T21:13:58.348717+0000","last_clean":"2024-02-19T21:13:58.348717+0000","last_became_active":"2024-02-19T21:12:09.347240+0000","last_became_peered":"2024-02-19T21:12:09.347240+0000","last_unstale":"2024-02-19T21:13:58.348717+0000","last_undegraded":"2024-02-19T21:13:58.348717+0000","last_fullsized":"2024-02-19T21:13:58.348717+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:43.407517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.407517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"434'12","reported_seq":424,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-19T21:13:59.361613+0000","last_change":"2024-02-19T21:13:47.189879+0000","last_active":"2024-02-19T21:13:59.361613+0000","last_peered":"2024-02-19T21:13:59.361613+0000","last_clean":"2024-02-19T21:13:59.361613+0000","last_became_active":"2024-02-19T21:10:41.297897+0000","last_became_peered":"2024-02-19T21:10:41.297897+0000","last_unstale":"2024-02-19T21:13:59.361613+0000","last_undegraded":"2024-02-19T21:13:59.361613+0000","last_fullsized":"2024-02-19T21:13:59.361613+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'12","last_scrub_stamp":"2024-02-19T21:13:47.189812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:47.189812+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"459'28","reported_seq":250,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351337+0000","last_change":"2024-02-19T21:13:46.758802+0000","last_active":"2024-02-19T21:13:58.351337+0000","last_peered":"2024-02-19T21:13:58.351337+0000","last_clean":"2024-02-19T21:13:58.351337+0000","last_became_active":"2024-02-19T21:12:09.369385+0000","last_became_peered":"2024-02-19T21:12:09.369385+0000","last_unstale":"2024-02-19T21:13:58.351337+0000","last_undegraded":"2024-02-19T21:13:58.351337+0000","last_fullsized":"2024-02-19T21:13:58.351337+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'28","last_scrub_stamp":"2024-02-19T21:13:46.758735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:13:46.758735+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":593,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356635+0000","last_change":"2024-02-19T21:13:45.764844+0000","last_active":"2024-02-19T21:13:58.356635+0000","last_peered":"2024-02-19T21:13:58.356635+0000","last_clean":"2024-02-19T21:13:58.356635+0000","last_became_active":"2024-02-19T21:12:09.342814+0000","last_became_peered":"2024-02-19T21:12:09.342814+0000","last_unstale":"2024-02-19T21:13:58.356635+0000","last_undegraded":"2024-02-19T21:13:58.356635+0000","last_fullsized":"2024-02-19T21:13:58.356635+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:45.764786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:45.764786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":577,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356093+0000","last_change":"2024-02-19T21:13:49.749231+0000","last_active":"2024-02-19T21:13:58.356093+0000","last_peered":"2024-02-19T21:13:58.356093+0000","last_clean":"2024-02-19T21:13:58.356093+0000","last_became_active":"2024-02-19T21:12:09.333970+0000","last_became_peered":"2024-02-19T21:12:09.333970+0000","last_unstale":"2024-02-19T21:13:58.356093+0000","last_undegraded":"2024-02-19T21:13:58.356093+0000","last_fullsized":"2024-02-19T21:13:58.356093+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:49.749137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:13:49.749137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":406,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.432754+0000","last_change":"2024-02-19T21:13:11.004073+0000","last_active":"2024-02-19T21:13:57.432754+0000","last_peered":"2024-02-19T21:13:57.432754+0000","last_clean":"2024-02-19T21:13:57.432754+0000","last_became_active":"2024-02-19T21:11:59.323698+0000","last_became_peered":"2024-02-19T21:11:59.323698+0000","last_unstale":"2024-02-19T21:13:57.432754+0000","last_undegraded":"2024-02-19T21:13:57.432754+0000","last_fullsized":"2024-02-19T21:13:57.432754+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:11.004006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:11.004006+0000","last_clean_scrub_stamp":"2024-02-19T21:13:11.004006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":183,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.350937+0000","last_change":"2024-02-19T21:10:59.312931+0000","last_active":"2024-02-19T21:13:58.350937+0000","last_peered":"2024-02-19T21:13:58.350937+0000","last_clean":"2024-02-19T21:13:58.350937+0000","last_became_active":"2024-02-19T21:10:41.304695+0000","last_became_peered":"2024-02-19T21:10:41.304695+0000","last_unstale":"2024-02-19T21:13:58.350937+0000","last_undegraded":"2024-02-19T21:13:58.350937+0000","last_fullsized":"2024-02-19T21:13:58.350937+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:59.312863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.312863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"459'30","reported_seq":377,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.348428+0000","last_change":"2024-02-19T21:11:59.380681+0000","last_active":"2024-02-19T21:13:58.348428+0000","last_peered":"2024-02-19T21:13:58.348428+0000","last_clean":"2024-02-19T21:13:58.348428+0000","last_became_active":"2024-02-19T21:11:59.327859+0000","last_became_peered":"2024-02-19T21:11:59.327859+0000","last_unstale":"2024-02-19T21:13:58.348428+0000","last_undegraded":"2024-02-19T21:13:58.348428+0000","last_fullsized":"2024-02-19T21:13:58.348428+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2024-02-19T21:11:02.671837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.671837+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":624,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.432028+0000","last_change":"2024-02-19T21:12:09.369780+0000","last_active":"2024-02-19T21:13:57.432028+0000","last_peered":"2024-02-19T21:13:57.432028+0000","last_clean":"2024-02-19T21:13:57.432028+0000","last_became_active":"2024-02-19T21:12:09.369477+0000","last_became_peered":"2024-02-19T21:12:09.369477+0000","last_unstale":"2024-02-19T21:13:57.432028+0000","last_undegraded":"2024-02-19T21:13:57.432028+0000","last_fullsized":"2024-02-19T21:13:57.432028+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:05:44.876426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:05:44.876426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":183,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351278+0000","last_change":"2024-02-19T21:12:06.410909+0000","last_active":"2024-02-19T21:13:58.351278+0000","last_peered":"2024-02-19T21:13:58.351278+0000","last_clean":"2024-02-19T21:13:58.351278+0000","last_became_active":"2024-02-19T21:10:41.295301+0000","last_became_peered":"2024-02-19T21:10:41.295301+0000","last_unstale":"2024-02-19T21:13:58.351278+0000","last_undegraded":"2024-02-19T21:13:58.351278+0000","last_fullsized":"2024-02-19T21:13:58.351278+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.410847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.410847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":332,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.350197+0000","last_change":"2024-02-19T21:12:09.332689+0000","last_active":"2024-02-19T21:13:58.350197+0000","last_peered":"2024-02-19T21:13:58.350197+0000","last_clean":"2024-02-19T21:13:58.350197+0000","last_became_active":"2024-02-19T21:12:09.332349+0000","last_became_peered":"2024-02-19T21:12:09.332349+0000","last_unstale":"2024-02-19T21:13:58.350197+0000","last_undegraded":"2024-02-19T21:13:58.350197+0000","last_fullsized":"2024-02-19T21:13:58.350197+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:46.406402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:46.406402+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":568,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.368400+0000","last_change":"2024-02-19T21:12:09.340947+0000","last_active":"2024-02-19T21:13:58.368400+0000","last_peered":"2024-02-19T21:13:58.368400+0000","last_clean":"2024-02-19T21:13:58.368400+0000","last_became_active":"2024-02-19T21:12:09.340614+0000","last_became_peered":"2024-02-19T21:12:09.340614+0000","last_unstale":"2024-02-19T21:13:58.368400+0000","last_undegraded":"2024-02-19T21:13:58.368400+0000","last_fullsized":"2024-02-19T21:13:58.368400+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.669336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.669336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"459'29","reported_seq":490,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367527+0000","last_change":"2024-02-19T21:13:48.837436+0000","last_active":"2024-02-19T21:13:58.367527+0000","last_peered":"2024-02-19T21:13:58.367527+0000","last_clean":"2024-02-19T21:13:58.367527+0000","last_became_active":"2024-02-19T21:11:59.332542+0000","last_became_peered":"2024-02-19T21:11:59.332542+0000","last_unstale":"2024-02-19T21:13:58.367527+0000","last_undegraded":"2024-02-19T21:13:58.367527+0000","last_fullsized":"2024-02-19T21:13:58.367527+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'29","last_scrub_stamp":"2024-02-19T21:13:48.837368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:13:48.837368+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":640,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356683+0000","last_change":"2024-02-19T21:13:38.813372+0000","last_active":"2024-02-19T21:13:58.356683+0000","last_peered":"2024-02-19T21:13:58.356683+0000","last_clean":"2024-02-19T21:13:58.356683+0000","last_became_active":"2024-02-19T21:11:59.335475+0000","last_became_peered":"2024-02-19T21:11:59.335475+0000","last_unstale":"2024-02-19T21:13:58.356683+0000","last_undegraded":"2024-02-19T21:13:58.356683+0000","last_fullsized":"2024-02-19T21:13:58.356683+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:38.813285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:38.813285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":569,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.430940+0000","last_change":"2024-02-19T21:12:20.902387+0000","last_active":"2024-02-19T21:13:57.430940+0000","last_peered":"2024-02-19T21:13:57.430940+0000","last_clean":"2024-02-19T21:13:57.430940+0000","last_became_active":"2024-02-19T21:12:09.347197+0000","last_became_peered":"2024-02-19T21:12:09.347197+0000","last_unstale":"2024-02-19T21:13:57.430940+0000","last_undegraded":"2024-02-19T21:13:57.430940+0000","last_fullsized":"2024-02-19T21:13:57.430940+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:20.902297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:12:20.902297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":526,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.365187+0000","last_change":"2024-02-19T21:11:59.359210+0000","last_active":"2024-02-19T21:13:58.365187+0000","last_peered":"2024-02-19T21:13:58.365187+0000","last_clean":"2024-02-19T21:13:58.365187+0000","last_became_active":"2024-02-19T21:11:59.358933+0000","last_became_peered":"2024-02-19T21:11:59.358933+0000","last_unstale":"2024-02-19T21:13:58.365187+0000","last_undegraded":"2024-02-19T21:13:58.365187+0000","last_fullsized":"2024-02-19T21:13:58.365187+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.189509+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":573,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.368643+0000","last_change":"2024-02-19T21:11:59.334472+0000","last_active":"2024-02-19T21:13:58.368643+0000","last_peered":"2024-02-19T21:13:58.368643+0000","last_clean":"2024-02-19T21:13:58.368643+0000","last_became_active":"2024-02-19T21:11:59.333491+0000","last_became_peered":"2024-02-19T21:11:59.333491+0000","last_unstale":"2024-02-19T21:13:58.368643+0000","last_undegraded":"2024-02-19T21:13:58.368643+0000","last_fullsized":"2024-02-19T21:13:58.368643+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:30.893485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:30.893485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"459'24","reported_seq":468,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.432142+0000","last_change":"2024-02-19T21:12:50.030922+0000","last_active":"2024-02-19T21:13:57.432142+0000","last_peered":"2024-02-19T21:13:57.432142+0000","last_clean":"2024-02-19T21:13:57.432142+0000","last_became_active":"2024-02-19T21:11:59.348806+0000","last_became_peered":"2024-02-19T21:11:59.348806+0000","last_unstale":"2024-02-19T21:13:57.432142+0000","last_undegraded":"2024-02-19T21:13:57.432142+0000","last_fullsized":"2024-02-19T21:13:57.432142+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'24","last_scrub_stamp":"2024-02-19T21:12:50.030857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:50.030857+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"397'2","reported_seq":637,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.355220+0000","last_change":"2024-02-19T21:13:40.809868+0000","last_active":"2024-02-19T21:13:58.355220+0000","last_peered":"2024-02-19T21:13:58.355220+0000","last_clean":"2024-02-19T21:13:58.355220+0000","last_became_active":"2024-02-19T21:12:02.319577+0000","last_became_peered":"2024-02-19T21:12:02.319577+0000","last_unstale":"2024-02-19T21:13:58.355220+0000","last_undegraded":"2024-02-19T21:13:58.355220+0000","last_fullsized":"2024-02-19T21:13:58.355220+0000","mapping_epoch":485,"log_start":"58'1","ondisk_log_start":"58'1","created":57,"last_epoch_clean":486,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:13:40.809800+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:13:40.809800+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":197,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351420+0000","last_change":"2024-02-19T21:13:54.643454+0000","last_active":"2024-02-19T21:13:58.351420+0000","last_peered":"2024-02-19T21:13:58.351420+0000","last_clean":"2024-02-19T21:13:58.351420+0000","last_became_active":"2024-02-19T21:11:59.320327+0000","last_became_peered":"2024-02-19T21:11:59.320327+0000","last_unstale":"2024-02-19T21:13:58.351420+0000","last_undegraded":"2024-02-19T21:13:58.351420+0000","last_fullsized":"2024-02-19T21:13:58.351420+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.643360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.643360+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":578,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.432655+0000","last_change":"2024-02-19T21:10:51.691357+0000","last_active":"2024-02-19T21:13:57.432655+0000","last_peered":"2024-02-19T21:13:57.432655+0000","last_clean":"2024-02-19T21:13:57.432655+0000","last_became_active":"2024-02-19T21:08:04.282243+0000","last_became_peered":"2024-02-19T21:08:04.282243+0000","last_unstale":"2024-02-19T21:13:57.432655+0000","last_undegraded":"2024-02-19T21:13:57.432655+0000","last_fullsized":"2024-02-19T21:13:57.432655+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.691284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.691284+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":517,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367433+0000","last_change":"2024-02-19T21:12:09.328499+0000","last_active":"2024-02-19T21:13:58.367433+0000","last_peered":"2024-02-19T21:13:58.367433+0000","last_clean":"2024-02-19T21:13:58.367433+0000","last_became_active":"2024-02-19T21:12:09.326936+0000","last_became_peered":"2024-02-19T21:12:09.326936+0000","last_unstale":"2024-02-19T21:13:58.367433+0000","last_undegraded":"2024-02-19T21:13:58.367433+0000","last_fullsized":"2024-02-19T21:13:58.367433+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:05.664425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:05.664425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":566,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356314+0000","last_change":"2024-02-19T21:13:06.446599+0000","last_active":"2024-02-19T21:13:58.356314+0000","last_peered":"2024-02-19T21:13:58.356314+0000","last_clean":"2024-02-19T21:13:58.356314+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:13:58.356314+0000","last_undegraded":"2024-02-19T21:13:58.356314+0000","last_fullsized":"2024-02-19T21:13:58.356314+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:06.446523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.446523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"459'26","reported_seq":647,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351982+0000","last_change":"2024-02-19T21:12:09.420794+0000","last_active":"2024-02-19T21:13:58.351982+0000","last_peered":"2024-02-19T21:13:58.351982+0000","last_clean":"2024-02-19T21:13:58.351982+0000","last_became_active":"2024-02-19T21:12:09.352705+0000","last_became_peered":"2024-02-19T21:12:09.352705+0000","last_unstale":"2024-02-19T21:13:58.351982+0000","last_undegraded":"2024-02-19T21:13:58.351982+0000","last_fullsized":"2024-02-19T21:13:58.351982+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'26","last_scrub_stamp":"2024-02-19T21:11:44.774856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:44.774856+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":597,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.355965+0000","last_change":"2024-02-19T21:13:50.724435+0000","last_active":"2024-02-19T21:13:58.355965+0000","last_peered":"2024-02-19T21:13:58.355965+0000","last_clean":"2024-02-19T21:13:58.355965+0000","last_became_active":"2024-02-19T21:12:09.310949+0000","last_became_peered":"2024-02-19T21:12:09.310949+0000","last_unstale":"2024-02-19T21:13:58.355965+0000","last_undegraded":"2024-02-19T21:13:58.355965+0000","last_fullsized":"2024-02-19T21:13:58.355965+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:50.724362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:13:50.724362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":605,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367506+0000","last_change":"2024-02-19T21:11:48.359614+0000","last_active":"2024-02-19T21:13:58.367506+0000","last_peered":"2024-02-19T21:13:58.367506+0000","last_clean":"2024-02-19T21:13:58.367506+0000","last_became_active":"2024-02-19T21:11:48.359271+0000","last_became_peered":"2024-02-19T21:11:48.359271+0000","last_unstale":"2024-02-19T21:13:58.367506+0000","last_undegraded":"2024-02-19T21:13:58.367506+0000","last_fullsized":"2024-02-19T21:13:58.367506+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.081993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.081993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":328,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.348813+0000","last_change":"2024-02-19T21:12:55.248327+0000","last_active":"2024-02-19T21:13:58.348813+0000","last_peered":"2024-02-19T21:13:58.348813+0000","last_clean":"2024-02-19T21:13:58.348813+0000","last_became_active":"2024-02-19T21:12:09.347643+0000","last_became_peered":"2024-02-19T21:12:09.347643+0000","last_unstale":"2024-02-19T21:13:58.348813+0000","last_undegraded":"2024-02-19T21:13:58.348813+0000","last_fullsized":"2024-02-19T21:13:58.348813+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:55.248259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.248259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":603,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.368688+0000","last_change":"2024-02-19T21:13:53.919119+0000","last_active":"2024-02-19T21:13:58.368688+0000","last_peered":"2024-02-19T21:13:58.368688+0000","last_clean":"2024-02-19T21:13:58.368688+0000","last_became_active":"2024-02-19T21:10:41.315397+0000","last_became_peered":"2024-02-19T21:10:41.315397+0000","last_unstale":"2024-02-19T21:13:58.368688+0000","last_undegraded":"2024-02-19T21:13:58.368688+0000","last_fullsized":"2024-02-19T21:13:58.368688+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:53.919024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:53.919024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":526,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367052+0000","last_change":"2024-02-19T21:11:59.333800+0000","last_active":"2024-02-19T21:13:58.367052+0000","last_peered":"2024-02-19T21:13:58.367052+0000","last_clean":"2024-02-19T21:13:58.367052+0000","last_became_active":"2024-02-19T21:11:59.333572+0000","last_became_peered":"2024-02-19T21:11:59.333572+0000","last_unstale":"2024-02-19T21:13:58.367052+0000","last_undegraded":"2024-02-19T21:13:58.367052+0000","last_fullsized":"2024-02-19T21:13:58.367052+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_clean_scrub_stamp":"2024-02-19T21:10:20.204880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":570,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.365016+0000","last_change":"2024-02-19T21:13:00.111238+0000","last_active":"2024-02-19T21:13:58.365016+0000","last_peered":"2024-02-19T21:13:58.365016+0000","last_clean":"2024-02-19T21:13:58.365016+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:13:58.365016+0000","last_undegraded":"2024-02-19T21:13:58.365016+0000","last_fullsized":"2024-02-19T21:13:58.365016+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.111171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.111171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"459'22","reported_seq":610,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.365147+0000","last_change":"2024-02-19T21:11:59.988716+0000","last_active":"2024-02-19T21:13:58.365147+0000","last_peered":"2024-02-19T21:13:58.365147+0000","last_clean":"2024-02-19T21:13:58.365147+0000","last_became_active":"2024-02-19T21:11:59.343010+0000","last_became_peered":"2024-02-19T21:11:59.343010+0000","last_unstale":"2024-02-19T21:13:58.365147+0000","last_undegraded":"2024-02-19T21:13:58.365147+0000","last_fullsized":"2024-02-19T21:13:58.365147+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'18","last_scrub_stamp":"2024-02-19T21:11:00.191206+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.191206+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":633,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.363257+0000","last_change":"2024-02-19T21:13:41.604243+0000","last_active":"2024-02-19T21:13:58.363257+0000","last_peered":"2024-02-19T21:13:58.363257+0000","last_clean":"2024-02-19T21:13:58.363257+0000","last_became_active":"2024-02-19T21:11:59.352283+0000","last_became_peered":"2024-02-19T21:11:59.352283+0000","last_unstale":"2024-02-19T21:13:58.363257+0000","last_undegraded":"2024-02-19T21:13:58.363257+0000","last_fullsized":"2024-02-19T21:13:58.363257+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:41.604181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:41.604181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":1342,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.510706+0000","last_change":"2024-02-19T21:13:08.986433+0000","last_active":"2024-02-19T21:13:58.510706+0000","last_peered":"2024-02-19T21:13:58.510706+0000","last_clean":"2024-02-19T21:13:58.510706+0000","last_became_active":"2024-02-19T21:12:09.340570+0000","last_became_peered":"2024-02-19T21:12:09.340570+0000","last_unstale":"2024-02-19T21:13:58.510706+0000","last_undegraded":"2024-02-19T21:13:58.510706+0000","last_fullsized":"2024-02-19T21:13:58.510706+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:13:08.986361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:08.986361+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":754,"num_read_kb":749,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":591,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356607+0000","last_change":"2024-02-19T21:13:04.428341+0000","last_active":"2024-02-19T21:13:58.356607+0000","last_peered":"2024-02-19T21:13:58.356607+0000","last_clean":"2024-02-19T21:13:58.356607+0000","last_became_active":"2024-02-19T21:12:09.323625+0000","last_became_peered":"2024-02-19T21:12:09.323625+0000","last_unstale":"2024-02-19T21:13:58.356607+0000","last_undegraded":"2024-02-19T21:13:58.356607+0000","last_fullsized":"2024-02-19T21:13:58.356607+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:04.428236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:04.428236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":523,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.431607+0000","last_change":"2024-02-19T21:11:59.352488+0000","last_active":"2024-02-19T21:13:57.431607+0000","last_peered":"2024-02-19T21:13:57.431607+0000","last_clean":"2024-02-19T21:13:57.431607+0000","last_became_active":"2024-02-19T21:11:59.351887+0000","last_became_peered":"2024-02-19T21:11:59.351887+0000","last_unstale":"2024-02-19T21:13:57.431607+0000","last_undegraded":"2024-02-19T21:13:57.431607+0000","last_fullsized":"2024-02-19T21:13:57.431607+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.947089+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":433,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.432325+0000","last_change":"2024-02-19T21:12:09.369780+0000","last_active":"2024-02-19T21:13:57.432325+0000","last_peered":"2024-02-19T21:13:57.432325+0000","last_clean":"2024-02-19T21:13:57.432325+0000","last_became_active":"2024-02-19T21:12:09.369431+0000","last_became_peered":"2024-02-19T21:12:09.369431+0000","last_unstale":"2024-02-19T21:13:57.432325+0000","last_undegraded":"2024-02-19T21:13:57.432325+0000","last_fullsized":"2024-02-19T21:13:57.432325+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.945639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.945639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"459'42","reported_seq":850,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.360601+0000","last_change":"2024-02-19T21:13:47.032674+0000","last_active":"2024-02-19T21:13:58.360601+0000","last_peered":"2024-02-19T21:13:58.360601+0000","last_clean":"2024-02-19T21:13:58.360601+0000","last_became_active":"2024-02-19T21:08:04.266322+0000","last_became_peered":"2024-02-19T21:08:04.266322+0000","last_unstale":"2024-02-19T21:13:58.360601+0000","last_undegraded":"2024-02-19T21:13:58.360601+0000","last_fullsized":"2024-02-19T21:13:58.360601+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'42","last_scrub_stamp":"2024-02-19T21:13:47.032604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:13:47.032604+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":614,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367927+0000","last_change":"2024-02-19T21:12:09.328219+0000","last_active":"2024-02-19T21:13:58.367927+0000","last_peered":"2024-02-19T21:13:58.367927+0000","last_clean":"2024-02-19T21:13:58.367927+0000","last_became_active":"2024-02-19T21:12:09.326579+0000","last_became_peered":"2024-02-19T21:12:09.326579+0000","last_unstale":"2024-02-19T21:13:58.367927+0000","last_undegraded":"2024-02-19T21:13:58.367927+0000","last_fullsized":"2024-02-19T21:13:58.367927+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.089715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.089715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":332,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349760+0000","last_change":"2024-02-19T21:11:31.629134+0000","last_active":"2024-02-19T21:13:58.349760+0000","last_peered":"2024-02-19T21:13:58.349760+0000","last_clean":"2024-02-19T21:13:58.349760+0000","last_became_active":"2024-02-19T21:10:41.305829+0000","last_became_peered":"2024-02-19T21:10:41.305829+0000","last_unstale":"2024-02-19T21:13:58.349760+0000","last_undegraded":"2024-02-19T21:13:58.349760+0000","last_fullsized":"2024-02-19T21:13:58.349760+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:31.629057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.629057+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":531,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.363338+0000","last_change":"2024-02-19T21:13:51.499365+0000","last_active":"2024-02-19T21:13:58.363338+0000","last_peered":"2024-02-19T21:13:58.363338+0000","last_clean":"2024-02-19T21:13:58.363338+0000","last_became_active":"2024-02-19T21:11:59.331967+0000","last_became_peered":"2024-02-19T21:11:59.331967+0000","last_unstale":"2024-02-19T21:13:58.363338+0000","last_undegraded":"2024-02-19T21:13:58.363338+0000","last_fullsized":"2024-02-19T21:13:58.363338+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:51.499300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:13:51.499300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"493'12","reported_seq":492,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:59.737443+0000","last_change":"2024-02-19T21:12:09.433235+0000","last_active":"2024-02-19T21:13:59.737443+0000","last_peered":"2024-02-19T21:13:59.737443+0000","last_clean":"2024-02-19T21:13:59.737443+0000","last_became_active":"2024-02-19T21:12:09.326372+0000","last_became_peered":"2024-02-19T21:12:09.326372+0000","last_unstale":"2024-02-19T21:13:59.737443+0000","last_undegraded":"2024-02-19T21:13:59.737443+0000","last_fullsized":"2024-02-19T21:13:59.737443+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'10","last_scrub_stamp":"2024-02-19T21:11:41.775418+0000","last_deep_scrub":"471'10","last_deep_scrub_stamp":"2024-02-19T21:11:41.775418+0000","last_clean_scrub_stamp":"2024-02-19T21:11:41.775418+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"459'20","reported_seq":636,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.431488+0000","last_change":"2024-02-19T21:12:09.030294+0000","last_active":"2024-02-19T21:13:57.431488+0000","last_peered":"2024-02-19T21:13:57.431488+0000","last_clean":"2024-02-19T21:13:57.431488+0000","last_became_active":"2024-02-19T21:10:41.296275+0000","last_became_peered":"2024-02-19T21:10:41.296275+0000","last_unstale":"2024-02-19T21:13:57.431488+0000","last_undegraded":"2024-02-19T21:13:57.431488+0000","last_fullsized":"2024-02-19T21:13:57.431488+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:12:09.030215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:09.030215+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":634,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.432433+0000","last_change":"2024-02-19T21:13:48.003544+0000","last_active":"2024-02-19T21:13:57.432433+0000","last_peered":"2024-02-19T21:13:57.432433+0000","last_clean":"2024-02-19T21:13:57.432433+0000","last_became_active":"2024-02-19T21:11:59.351241+0000","last_became_peered":"2024-02-19T21:11:59.351241+0000","last_unstale":"2024-02-19T21:13:57.432433+0000","last_undegraded":"2024-02-19T21:13:57.432433+0000","last_fullsized":"2024-02-19T21:13:57.432433+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:48.003444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:13:48.003444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":328,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.348390+0000","last_change":"2024-02-19T21:13:07.221665+0000","last_active":"2024-02-19T21:13:58.348390+0000","last_peered":"2024-02-19T21:13:58.348390+0000","last_clean":"2024-02-19T21:13:58.348390+0000","last_became_active":"2024-02-19T21:11:59.335798+0000","last_became_peered":"2024-02-19T21:11:59.335798+0000","last_unstale":"2024-02-19T21:13:58.348390+0000","last_undegraded":"2024-02-19T21:13:58.348390+0000","last_fullsized":"2024-02-19T21:13:58.348390+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:07.221594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.221594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":323,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349580+0000","last_change":"2024-02-19T21:12:09.332210+0000","last_active":"2024-02-19T21:13:58.349580+0000","last_peered":"2024-02-19T21:13:58.349580+0000","last_clean":"2024-02-19T21:13:58.349580+0000","last_became_active":"2024-02-19T21:12:09.331195+0000","last_became_peered":"2024-02-19T21:12:09.331195+0000","last_unstale":"2024-02-19T21:13:58.349580+0000","last_undegraded":"2024-02-19T21:13:58.349580+0000","last_fullsized":"2024-02-19T21:13:58.349580+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:05.604025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:05.604025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":579,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.363121+0000","last_change":"2024-02-19T21:13:09.184029+0000","last_active":"2024-02-19T21:13:58.363121+0000","last_peered":"2024-02-19T21:13:58.363121+0000","last_clean":"2024-02-19T21:13:58.363121+0000","last_became_active":"2024-02-19T21:11:59.333275+0000","last_became_peered":"2024-02-19T21:11:59.333275+0000","last_unstale":"2024-02-19T21:13:58.363121+0000","last_undegraded":"2024-02-19T21:13:58.363121+0000","last_fullsized":"2024-02-19T21:13:58.363121+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:09.183962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:13:09.183962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"459'23","reported_seq":204,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351031+0000","last_change":"2024-02-19T21:12:18.340005+0000","last_active":"2024-02-19T21:13:58.351031+0000","last_peered":"2024-02-19T21:13:58.351031+0000","last_clean":"2024-02-19T21:13:58.351031+0000","last_became_active":"2024-02-19T21:10:41.287134+0000","last_became_peered":"2024-02-19T21:10:41.287134+0000","last_unstale":"2024-02-19T21:13:58.351031+0000","last_undegraded":"2024-02-19T21:13:58.351031+0000","last_fullsized":"2024-02-19T21:13:58.351031+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'23","last_scrub_stamp":"2024-02-19T21:12:18.339929+0000","last_deep_scrub":"420'17","last_deep_scrub_stamp":"2024-02-19T21:10:19.722303+0000","last_clean_scrub_stamp":"2024-02-19T21:12:18.339929+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":204,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351983+0000","last_change":"2024-02-19T21:11:59.342221+0000","last_active":"2024-02-19T21:13:58.351983+0000","last_peered":"2024-02-19T21:13:58.351983+0000","last_clean":"2024-02-19T21:13:58.351983+0000","last_became_active":"2024-02-19T21:11:59.323514+0000","last_became_peered":"2024-02-19T21:11:59.323514+0000","last_unstale":"2024-02-19T21:13:58.351983+0000","last_undegraded":"2024-02-19T21:13:58.351983+0000","last_fullsized":"2024-02-19T21:13:58.351983+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:10:58.333168+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.333168+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":319,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349485+0000","last_change":"2024-02-19T21:13:00.166548+0000","last_active":"2024-02-19T21:13:58.349485+0000","last_peered":"2024-02-19T21:13:58.349485+0000","last_clean":"2024-02-19T21:13:58.349485+0000","last_became_active":"2024-02-19T21:08:04.264593+0000","last_became_peered":"2024-02-19T21:08:04.264593+0000","last_unstale":"2024-02-19T21:13:58.349485+0000","last_undegraded":"2024-02-19T21:13:58.349485+0000","last_fullsized":"2024-02-19T21:13:58.349485+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.166479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.166479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":522,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.364652+0000","last_change":"2024-02-19T21:13:44.567327+0000","last_active":"2024-02-19T21:13:58.364652+0000","last_peered":"2024-02-19T21:13:58.364652+0000","last_clean":"2024-02-19T21:13:58.364652+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:13:58.364652+0000","last_undegraded":"2024-02-19T21:13:58.364652+0000","last_fullsized":"2024-02-19T21:13:58.364652+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:44.567199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:13:44.567199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"459'15","reported_seq":354,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349489+0000","last_change":"2024-02-19T21:11:59.364955+0000","last_active":"2024-02-19T21:13:58.349489+0000","last_peered":"2024-02-19T21:13:58.349489+0000","last_clean":"2024-02-19T21:13:58.349489+0000","last_became_active":"2024-02-19T21:11:59.336293+0000","last_became_peered":"2024-02-19T21:11:59.336293+0000","last_unstale":"2024-02-19T21:13:58.349489+0000","last_undegraded":"2024-02-19T21:13:58.349489+0000","last_fullsized":"2024-02-19T21:13:58.349489+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:10:22.459807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:22.459807+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":578,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.363119+0000","last_change":"2024-02-19T21:11:59.334071+0000","last_active":"2024-02-19T21:13:58.363119+0000","last_peered":"2024-02-19T21:13:58.363119+0000","last_clean":"2024-02-19T21:13:58.363119+0000","last_became_active":"2024-02-19T21:11:59.333684+0000","last_became_peered":"2024-02-19T21:11:59.333684+0000","last_unstale":"2024-02-19T21:13:58.363119+0000","last_undegraded":"2024-02-19T21:13:58.363119+0000","last_fullsized":"2024-02-19T21:13:58.363119+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.244285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.244285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":314,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349920+0000","last_change":"2024-02-19T21:10:53.703797+0000","last_active":"2024-02-19T21:13:58.349920+0000","last_peered":"2024-02-19T21:13:58.349920+0000","last_clean":"2024-02-19T21:13:58.349920+0000","last_became_active":"2024-02-19T21:08:04.264972+0000","last_became_peered":"2024-02-19T21:08:04.264972+0000","last_unstale":"2024-02-19T21:13:58.349920+0000","last_undegraded":"2024-02-19T21:13:58.349920+0000","last_fullsized":"2024-02-19T21:13:58.349920+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:53.703727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.703727+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":586,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.364717+0000","last_change":"2024-02-19T21:12:06.261731+0000","last_active":"2024-02-19T21:13:58.364717+0000","last_peered":"2024-02-19T21:13:58.364717+0000","last_clean":"2024-02-19T21:13:58.364717+0000","last_became_active":"2024-02-19T21:10:41.314114+0000","last_became_peered":"2024-02-19T21:10:41.314114+0000","last_unstale":"2024-02-19T21:13:58.364717+0000","last_undegraded":"2024-02-19T21:13:58.364717+0000","last_fullsized":"2024-02-19T21:13:58.364717+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.261670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.261670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":183,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351870+0000","last_change":"2024-02-19T21:12:08.430936+0000","last_active":"2024-02-19T21:13:58.351870+0000","last_peered":"2024-02-19T21:13:58.351870+0000","last_clean":"2024-02-19T21:13:58.351870+0000","last_became_active":"2024-02-19T21:10:41.300574+0000","last_became_peered":"2024-02-19T21:10:41.300574+0000","last_unstale":"2024-02-19T21:13:58.351870+0000","last_undegraded":"2024-02-19T21:13:58.351870+0000","last_fullsized":"2024-02-19T21:13:58.351870+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:08.430874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:08.430874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"459'15","reported_seq":205,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.364732+0000","last_change":"2024-02-19T21:13:43.552100+0000","last_active":"2024-02-19T21:13:58.364732+0000","last_peered":"2024-02-19T21:13:58.364732+0000","last_clean":"2024-02-19T21:13:58.364732+0000","last_became_active":"2024-02-19T21:10:40.297867+0000","last_became_peered":"2024-02-19T21:10:40.297867+0000","last_unstale":"2024-02-19T21:13:58.364732+0000","last_undegraded":"2024-02-19T21:13:58.364732+0000","last_fullsized":"2024-02-19T21:13:58.364732+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:13:43.552026+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:13:43.552026+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":529,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.350863+0000","last_change":"2024-02-19T21:12:09.369726+0000","last_active":"2024-02-19T21:13:58.350863+0000","last_peered":"2024-02-19T21:13:58.350863+0000","last_clean":"2024-02-19T21:13:58.350863+0000","last_became_active":"2024-02-19T21:12:09.369407+0000","last_became_peered":"2024-02-19T21:12:09.369407+0000","last_unstale":"2024-02-19T21:13:58.350863+0000","last_undegraded":"2024-02-19T21:13:58.350863+0000","last_fullsized":"2024-02-19T21:13:58.350863+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:31.670026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.670026+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":585,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.368760+0000","last_change":"2024-02-19T21:12:06.098778+0000","last_active":"2024-02-19T21:13:58.368760+0000","last_peered":"2024-02-19T21:13:58.368760+0000","last_clean":"2024-02-19T21:13:58.368760+0000","last_became_active":"2024-02-19T21:08:04.265573+0000","last_became_peered":"2024-02-19T21:08:04.265573+0000","last_unstale":"2024-02-19T21:13:58.368760+0000","last_undegraded":"2024-02-19T21:13:58.368760+0000","last_fullsized":"2024-02-19T21:13:58.368760+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.098703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.098703+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":593,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356365+0000","last_change":"2024-02-19T21:12:17.452030+0000","last_active":"2024-02-19T21:13:58.356365+0000","last_peered":"2024-02-19T21:13:58.356365+0000","last_clean":"2024-02-19T21:13:58.356365+0000","last_became_active":"2024-02-19T21:12:09.341406+0000","last_became_peered":"2024-02-19T21:12:09.341406+0000","last_unstale":"2024-02-19T21:13:58.356365+0000","last_undegraded":"2024-02-19T21:13:58.356365+0000","last_fullsized":"2024-02-19T21:13:58.356365+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:17.451964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:17.451964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":544,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367388+0000","last_change":"2024-02-19T21:12:47.428132+0000","last_active":"2024-02-19T21:13:58.367388+0000","last_peered":"2024-02-19T21:13:58.367388+0000","last_clean":"2024-02-19T21:13:58.367388+0000","last_became_active":"2024-02-19T21:12:09.349695+0000","last_became_peered":"2024-02-19T21:12:09.349695+0000","last_unstale":"2024-02-19T21:13:58.367388+0000","last_undegraded":"2024-02-19T21:13:58.367388+0000","last_fullsized":"2024-02-19T21:13:58.367388+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:47.428055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:47.428055+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"459'20","reported_seq":341,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.350116+0000","last_change":"2024-02-19T21:10:59.685223+0000","last_active":"2024-02-19T21:13:58.350116+0000","last_peered":"2024-02-19T21:13:58.350116+0000","last_clean":"2024-02-19T21:13:58.350116+0000","last_became_active":"2024-02-19T21:08:04.267216+0000","last_became_peered":"2024-02-19T21:08:04.267216+0000","last_unstale":"2024-02-19T21:13:58.350116+0000","last_undegraded":"2024-02-19T21:13:58.350116+0000","last_fullsized":"2024-02-19T21:13:58.350116+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'10","last_scrub_stamp":"2024-02-19T21:10:59.685150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:59.685150+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":592,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367831+0000","last_change":"2024-02-19T21:13:36.771987+0000","last_active":"2024-02-19T21:13:58.367831+0000","last_peered":"2024-02-19T21:13:58.367831+0000","last_clean":"2024-02-19T21:13:58.367831+0000","last_became_active":"2024-02-19T21:08:04.266728+0000","last_became_peered":"2024-02-19T21:08:04.266728+0000","last_unstale":"2024-02-19T21:13:58.367831+0000","last_undegraded":"2024-02-19T21:13:58.367831+0000","last_fullsized":"2024-02-19T21:13:58.367831+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:36.771900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_clean_scrub_stamp":"2024-02-19T21:13:36.771900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"433'9","reported_seq":739,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:59.737478+0000","last_change":"2024-02-19T21:13:00.428710+0000","last_active":"2024-02-19T21:13:59.737478+0000","last_peered":"2024-02-19T21:13:59.737478+0000","last_clean":"2024-02-19T21:13:59.737478+0000","last_became_active":"2024-02-19T21:10:40.310303+0000","last_became_peered":"2024-02-19T21:10:40.310303+0000","last_unstale":"2024-02-19T21:13:59.737478+0000","last_undegraded":"2024-02-19T21:13:59.737478+0000","last_fullsized":"2024-02-19T21:13:59.737478+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'9","last_scrub_stamp":"2024-02-19T21:13:00.428581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.428581+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":445,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356482+0000","last_change":"2024-02-19T21:11:59.843545+0000","last_active":"2024-02-19T21:13:58.356482+0000","last_peered":"2024-02-19T21:13:58.356482+0000","last_clean":"2024-02-19T21:13:58.356482+0000","last_became_active":"2024-02-19T21:11:59.323539+0000","last_became_peered":"2024-02-19T21:11:59.323539+0000","last_unstale":"2024-02-19T21:13:58.356482+0000","last_undegraded":"2024-02-19T21:13:58.356482+0000","last_fullsized":"2024-02-19T21:13:58.356482+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:11:31.500511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:31.500511+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":398,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.363020+0000","last_change":"2024-02-19T21:12:09.345187+0000","last_active":"2024-02-19T21:13:58.363020+0000","last_peered":"2024-02-19T21:13:58.363020+0000","last_clean":"2024-02-19T21:13:58.363020+0000","last_became_active":"2024-02-19T21:12:09.344609+0000","last_became_peered":"2024-02-19T21:12:09.344609+0000","last_unstale":"2024-02-19T21:13:58.363020+0000","last_undegraded":"2024-02-19T21:13:58.363020+0000","last_fullsized":"2024-02-19T21:13:58.363020+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.378925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:55.202463+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.378925+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"459'15","reported_seq":630,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367609+0000","last_change":"2024-02-19T21:10:50.053312+0000","last_active":"2024-02-19T21:13:58.367609+0000","last_peered":"2024-02-19T21:13:58.367609+0000","last_clean":"2024-02-19T21:13:58.367609+0000","last_became_active":"2024-02-19T21:08:04.266103+0000","last_became_peered":"2024-02-19T21:08:04.266103+0000","last_unstale":"2024-02-19T21:13:58.367609+0000","last_undegraded":"2024-02-19T21:13:58.367609+0000","last_fullsized":"2024-02-19T21:13:58.367609+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:10:50.053229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.053229+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":197,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351898+0000","last_change":"2024-02-19T21:13:07.507018+0000","last_active":"2024-02-19T21:13:58.351898+0000","last_peered":"2024-02-19T21:13:58.351898+0000","last_clean":"2024-02-19T21:13:58.351898+0000","last_became_active":"2024-02-19T21:11:59.324861+0000","last_became_peered":"2024-02-19T21:11:59.324861+0000","last_unstale":"2024-02-19T21:13:58.351898+0000","last_undegraded":"2024-02-19T21:13:58.351898+0000","last_fullsized":"2024-02-19T21:13:58.351898+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:07.506946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.506946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":395,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-19T21:13:59.362325+0000","last_change":"2024-02-19T21:12:17.345023+0000","last_active":"2024-02-19T21:13:59.362325+0000","last_peered":"2024-02-19T21:13:59.362325+0000","last_clean":"2024-02-19T21:13:59.362325+0000","last_became_active":"2024-02-19T21:08:04.256045+0000","last_became_peered":"2024-02-19T21:08:04.256045+0000","last_unstale":"2024-02-19T21:13:59.362325+0000","last_undegraded":"2024-02-19T21:13:59.362325+0000","last_fullsized":"2024-02-19T21:13:59.362325+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'10","last_scrub_stamp":"2024-02-19T21:12:17.344916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:17.344916+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":183,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.352106+0000","last_change":"2024-02-19T21:12:05.445686+0000","last_active":"2024-02-19T21:13:58.352106+0000","last_peered":"2024-02-19T21:13:58.352106+0000","last_clean":"2024-02-19T21:13:58.352106+0000","last_became_active":"2024-02-19T21:10:41.310279+0000","last_became_peered":"2024-02-19T21:10:41.310279+0000","last_unstale":"2024-02-19T21:13:58.352106+0000","last_undegraded":"2024-02-19T21:13:58.352106+0000","last_fullsized":"2024-02-19T21:13:58.352106+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:05.445575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_clean_scrub_stamp":"2024-02-19T21:12:05.445575+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":574,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.363148+0000","last_change":"2024-02-19T21:12:49.142231+0000","last_active":"2024-02-19T21:13:58.363148+0000","last_peered":"2024-02-19T21:13:58.363148+0000","last_clean":"2024-02-19T21:13:58.363148+0000","last_became_active":"2024-02-19T21:12:09.339844+0000","last_became_peered":"2024-02-19T21:12:09.339844+0000","last_unstale":"2024-02-19T21:13:58.363148+0000","last_undegraded":"2024-02-19T21:13:58.363148+0000","last_fullsized":"2024-02-19T21:13:58.363148+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-02-19T21:12:49.142158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:49.142158+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":15,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"459'15","reported_seq":329,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.431162+0000","last_change":"2024-02-19T21:13:06.958616+0000","last_active":"2024-02-19T21:13:57.431162+0000","last_peered":"2024-02-19T21:13:57.431162+0000","last_clean":"2024-02-19T21:13:57.431162+0000","last_became_active":"2024-02-19T21:12:09.341038+0000","last_became_peered":"2024-02-19T21:12:09.341038+0000","last_unstale":"2024-02-19T21:13:57.431162+0000","last_undegraded":"2024-02-19T21:13:57.431162+0000","last_fullsized":"2024-02-19T21:13:57.431162+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:13:06.958518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.958518+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":398,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349446+0000","last_change":"2024-02-19T21:12:09.333684+0000","last_active":"2024-02-19T21:13:58.349446+0000","last_peered":"2024-02-19T21:13:58.349446+0000","last_clean":"2024-02-19T21:13:58.349446+0000","last_became_active":"2024-02-19T21:12:09.333350+0000","last_became_peered":"2024-02-19T21:12:09.333350+0000","last_unstale":"2024-02-19T21:13:58.349446+0000","last_undegraded":"2024-02-19T21:13:58.349446+0000","last_fullsized":"2024-02-19T21:13:58.349446+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:57.422323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:10:57.422323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":328,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349689+0000","last_change":"2024-02-19T21:11:59.349259+0000","last_active":"2024-02-19T21:13:58.349689+0000","last_peered":"2024-02-19T21:13:58.349689+0000","last_clean":"2024-02-19T21:13:58.349689+0000","last_became_active":"2024-02-19T21:11:59.348485+0000","last_became_peered":"2024-02-19T21:11:59.348485+0000","last_unstale":"2024-02-19T21:13:58.349689+0000","last_undegraded":"2024-02-19T21:13:58.349689+0000","last_fullsized":"2024-02-19T21:13:58.349689+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:45.404242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_clean_scrub_stamp":"2024-02-19T21:11:45.404242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":418,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367030+0000","last_change":"2024-02-19T21:13:47.829354+0000","last_active":"2024-02-19T21:13:58.367030+0000","last_peered":"2024-02-19T21:13:58.367030+0000","last_clean":"2024-02-19T21:13:58.367030+0000","last_became_active":"2024-02-19T21:12:09.327390+0000","last_became_peered":"2024-02-19T21:12:09.327390+0000","last_unstale":"2024-02-19T21:13:58.367030+0000","last_undegraded":"2024-02-19T21:13:58.367030+0000","last_fullsized":"2024-02-19T21:13:58.367030+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:47.829261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:47.829261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":517,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.368537+0000","last_change":"2024-02-19T21:12:09.330102+0000","last_active":"2024-02-19T21:13:58.368537+0000","last_peered":"2024-02-19T21:13:58.368537+0000","last_clean":"2024-02-19T21:13:58.368537+0000","last_became_active":"2024-02-19T21:12:09.329246+0000","last_became_peered":"2024-02-19T21:12:09.329246+0000","last_unstale":"2024-02-19T21:13:58.368537+0000","last_undegraded":"2024-02-19T21:13:58.368537+0000","last_fullsized":"2024-02-19T21:13:58.368537+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.493206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.493206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"459'19","reported_seq":228,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351863+0000","last_change":"2024-02-19T21:12:10.570353+0000","last_active":"2024-02-19T21:13:58.351863+0000","last_peered":"2024-02-19T21:13:58.351863+0000","last_clean":"2024-02-19T21:13:58.351863+0000","last_became_active":"2024-02-19T21:12:09.399929+0000","last_became_peered":"2024-02-19T21:12:09.399929+0000","last_unstale":"2024-02-19T21:13:58.351863+0000","last_undegraded":"2024-02-19T21:13:58.351863+0000","last_fullsized":"2024-02-19T21:13:58.351863+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:11:32.351880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:32.351880+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":589,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.432463+0000","last_change":"2024-02-19T21:13:00.874389+0000","last_active":"2024-02-19T21:13:57.432463+0000","last_peered":"2024-02-19T21:13:57.432463+0000","last_clean":"2024-02-19T21:13:57.432463+0000","last_became_active":"2024-02-19T21:12:09.342120+0000","last_became_peered":"2024-02-19T21:12:09.342120+0000","last_unstale":"2024-02-19T21:13:57.432463+0000","last_undegraded":"2024-02-19T21:13:57.432463+0000","last_fullsized":"2024-02-19T21:13:57.432463+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.874318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.874318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":585,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.431481+0000","last_change":"2024-02-19T21:12:54.957657+0000","last_active":"2024-02-19T21:13:57.431481+0000","last_peered":"2024-02-19T21:13:57.431481+0000","last_clean":"2024-02-19T21:13:57.431481+0000","last_became_active":"2024-02-19T21:08:04.268847+0000","last_became_peered":"2024-02-19T21:08:04.268847+0000","last_unstale":"2024-02-19T21:13:57.431481+0000","last_undegraded":"2024-02-19T21:13:57.431481+0000","last_fullsized":"2024-02-19T21:13:57.431481+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:54.957585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:54.957585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":323,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.348489+0000","last_change":"2024-02-19T21:11:59.341209+0000","last_active":"2024-02-19T21:13:58.348489+0000","last_peered":"2024-02-19T21:13:58.348489+0000","last_clean":"2024-02-19T21:13:58.348489+0000","last_became_active":"2024-02-19T21:11:59.340742+0000","last_became_peered":"2024-02-19T21:11:59.340742+0000","last_unstale":"2024-02-19T21:13:58.348489+0000","last_undegraded":"2024-02-19T21:13:58.348489+0000","last_fullsized":"2024-02-19T21:13:58.348489+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:37.477995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:11:37.477995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":323,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349944+0000","last_change":"2024-02-19T21:11:59.338599+0000","last_active":"2024-02-19T21:13:58.349944+0000","last_peered":"2024-02-19T21:13:58.349944+0000","last_clean":"2024-02-19T21:13:58.349944+0000","last_became_active":"2024-02-19T21:11:59.337764+0000","last_became_peered":"2024-02-19T21:11:59.337764+0000","last_unstale":"2024-02-19T21:13:58.349944+0000","last_undegraded":"2024-02-19T21:13:58.349944+0000","last_fullsized":"2024-02-19T21:13:58.349944+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:07.599474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:07.599474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"555'10","reported_seq":211,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.366902+0000","last_change":"2024-02-19T21:13:12.409114+0000","last_active":"2024-02-19T21:13:58.366902+0000","last_peered":"2024-02-19T21:13:58.366902+0000","last_clean":"2024-02-19T21:13:58.366902+0000","last_became_active":"2024-02-19T21:12:09.340298+0000","last_became_peered":"2024-02-19T21:12:09.340298+0000","last_unstale":"2024-02-19T21:13:58.366902+0000","last_undegraded":"2024-02-19T21:13:58.366902+0000","last_fullsized":"2024-02-19T21:13:58.366902+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'10","last_scrub_stamp":"2024-02-19T21:13:12.409036+0000","last_deep_scrub":"555'10","last_deep_scrub_stamp":"2024-02-19T21:13:12.409036+0000","last_clean_scrub_stamp":"2024-02-19T21:13:12.409036+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":583,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.368626+0000","last_change":"2024-02-19T21:13:43.779607+0000","last_active":"2024-02-19T21:13:58.368626+0000","last_peered":"2024-02-19T21:13:58.368626+0000","last_clean":"2024-02-19T21:13:58.368626+0000","last_became_active":"2024-02-19T21:12:09.329752+0000","last_became_peered":"2024-02-19T21:12:09.329752+0000","last_unstale":"2024-02-19T21:13:58.368626+0000","last_undegraded":"2024-02-19T21:13:58.368626+0000","last_fullsized":"2024-02-19T21:13:58.368626+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:43.779536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:43.779536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":530,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.350776+0000","last_change":"2024-02-19T21:13:41.706582+0000","last_active":"2024-02-19T21:13:58.350776+0000","last_peered":"2024-02-19T21:13:58.350776+0000","last_clean":"2024-02-19T21:13:58.350776+0000","last_became_active":"2024-02-19T21:12:09.345654+0000","last_became_peered":"2024-02-19T21:12:09.345654+0000","last_unstale":"2024-02-19T21:13:58.350776+0000","last_undegraded":"2024-02-19T21:13:58.350776+0000","last_fullsized":"2024-02-19T21:13:58.350776+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:41.706497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:41.706497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":582,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.355608+0000","last_change":"2024-02-19T21:12:05.530113+0000","last_active":"2024-02-19T21:13:58.355608+0000","last_peered":"2024-02-19T21:13:58.355608+0000","last_clean":"2024-02-19T21:13:58.355608+0000","last_became_active":"2024-02-19T21:10:41.314589+0000","last_became_peered":"2024-02-19T21:10:41.314589+0000","last_unstale":"2024-02-19T21:13:58.355608+0000","last_undegraded":"2024-02-19T21:13:58.355608+0000","last_fullsized":"2024-02-19T21:13:58.355608+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:05.530039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:05.530039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":319,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.350157+0000","last_change":"2024-02-19T21:12:07.259353+0000","last_active":"2024-02-19T21:13:58.350157+0000","last_peered":"2024-02-19T21:13:58.350157+0000","last_clean":"2024-02-19T21:13:58.350157+0000","last_became_active":"2024-02-19T21:08:04.266161+0000","last_became_peered":"2024-02-19T21:08:04.266161+0000","last_unstale":"2024-02-19T21:13:58.350157+0000","last_undegraded":"2024-02-19T21:13:58.350157+0000","last_fullsized":"2024-02-19T21:13:58.350157+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:07.259290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:07.259290+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"459'19","reported_seq":364,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.348633+0000","last_change":"2024-02-19T21:12:09.375334+0000","last_active":"2024-02-19T21:13:58.348633+0000","last_peered":"2024-02-19T21:13:58.348633+0000","last_clean":"2024-02-19T21:13:58.348633+0000","last_became_active":"2024-02-19T21:12:09.321408+0000","last_became_peered":"2024-02-19T21:12:09.321408+0000","last_unstale":"2024-02-19T21:13:58.348633+0000","last_undegraded":"2024-02-19T21:13:58.348633+0000","last_fullsized":"2024-02-19T21:13:58.348633+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:12:03.255395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:03.255395+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":119,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.368576+0000","last_change":"2024-02-19T21:12:09.345352+0000","last_active":"2024-02-19T21:13:58.368576+0000","last_peered":"2024-02-19T21:13:58.368576+0000","last_clean":"2024-02-19T21:13:58.368576+0000","last_became_active":"2024-02-19T21:12:09.345000+0000","last_became_peered":"2024-02-19T21:12:09.345000+0000","last_unstale":"2024-02-19T21:13:58.368576+0000","last_undegraded":"2024-02-19T21:13:58.368576+0000","last_fullsized":"2024-02-19T21:13:58.368576+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:00.382946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.382946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":588,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.355511+0000","last_change":"2024-02-19T21:12:09.332889+0000","last_active":"2024-02-19T21:13:58.355511+0000","last_peered":"2024-02-19T21:13:58.355511+0000","last_clean":"2024-02-19T21:13:58.355511+0000","last_became_active":"2024-02-19T21:12:09.332485+0000","last_became_peered":"2024-02-19T21:12:09.332485+0000","last_unstale":"2024-02-19T21:13:58.355511+0000","last_undegraded":"2024-02-19T21:13:58.355511+0000","last_fullsized":"2024-02-19T21:13:58.355511+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.566208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.566208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":332,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349786+0000","last_change":"2024-02-19T21:12:09.344855+0000","last_active":"2024-02-19T21:13:58.349786+0000","last_peered":"2024-02-19T21:13:58.349786+0000","last_clean":"2024-02-19T21:13:58.349786+0000","last_became_active":"2024-02-19T21:12:09.344536+0000","last_became_peered":"2024-02-19T21:12:09.344536+0000","last_unstale":"2024-02-19T21:13:58.349786+0000","last_undegraded":"2024-02-19T21:13:58.349786+0000","last_fullsized":"2024-02-19T21:13:58.349786+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:00.547228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:10:00.547228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":405,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.370078+0000","last_change":"2024-02-19T21:12:09.343909+0000","last_active":"2024-02-19T21:13:58.370078+0000","last_peered":"2024-02-19T21:13:58.370078+0000","last_clean":"2024-02-19T21:13:58.370078+0000","last_became_active":"2024-02-19T21:12:09.342686+0000","last_became_peered":"2024-02-19T21:12:09.342686+0000","last_unstale":"2024-02-19T21:13:58.370078+0000","last_undegraded":"2024-02-19T21:13:58.370078+0000","last_fullsized":"2024-02-19T21:13:58.370078+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:37.718654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:11:37.718654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"459'20","reported_seq":502,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356582+0000","last_change":"2024-02-19T21:12:48.422862+0000","last_active":"2024-02-19T21:13:58.356582+0000","last_peered":"2024-02-19T21:13:58.356582+0000","last_clean":"2024-02-19T21:13:58.356582+0000","last_became_active":"2024-02-19T21:11:59.350142+0000","last_became_peered":"2024-02-19T21:11:59.350142+0000","last_unstale":"2024-02-19T21:13:58.356582+0000","last_undegraded":"2024-02-19T21:13:58.356582+0000","last_fullsized":"2024-02-19T21:13:58.356582+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:12:48.422763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:48.422763+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":587,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.366993+0000","last_change":"2024-02-19T21:12:48.430341+0000","last_active":"2024-02-19T21:13:58.366993+0000","last_peered":"2024-02-19T21:13:58.366993+0000","last_clean":"2024-02-19T21:13:58.366993+0000","last_became_active":"2024-02-19T21:10:41.294592+0000","last_became_peered":"2024-02-19T21:10:41.294592+0000","last_unstale":"2024-02-19T21:13:58.366993+0000","last_undegraded":"2024-02-19T21:13:58.366993+0000","last_fullsized":"2024-02-19T21:13:58.366993+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:48.430223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:48.430223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"493'11","reported_seq":685,"reported_epoch":603,"state":"active+clean","last_fresh":"2024-02-19T21:13:59.616963+0000","last_change":"2024-02-19T21:13:39.691375+0000","last_active":"2024-02-19T21:13:59.616963+0000","last_peered":"2024-02-19T21:13:59.616963+0000","last_clean":"2024-02-19T21:13:59.616963+0000","last_became_active":"2024-02-19T21:12:09.352204+0000","last_became_peered":"2024-02-19T21:12:09.352204+0000","last_unstale":"2024-02-19T21:13:59.616963+0000","last_undegraded":"2024-02-19T21:13:59.616963+0000","last_fullsized":"2024-02-19T21:13:59.616963+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'11","last_scrub_stamp":"2024-02-19T21:13:39.691291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:39.691291+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":566,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356427+0000","last_change":"2024-02-19T21:11:43.546468+0000","last_active":"2024-02-19T21:13:58.356427+0000","last_peered":"2024-02-19T21:13:58.356427+0000","last_clean":"2024-02-19T21:13:58.356427+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:13:58.356427+0000","last_undegraded":"2024-02-19T21:13:58.356427+0000","last_fullsized":"2024-02-19T21:13:58.356427+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:11:43.546383+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.546383+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":520,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.364221+0000","last_change":"2024-02-19T21:11:38.369815+0000","last_active":"2024-02-19T21:13:58.364221+0000","last_peered":"2024-02-19T21:13:58.364221+0000","last_clean":"2024-02-19T21:13:58.364221+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:13:58.364221+0000","last_undegraded":"2024-02-19T21:13:58.364221+0000","last_fullsized":"2024-02-19T21:13:58.364221+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.369716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.369716+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"459'19","reported_seq":587,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.368673+0000","last_change":"2024-02-19T21:12:10.390849+0000","last_active":"2024-02-19T21:13:58.368673+0000","last_peered":"2024-02-19T21:13:58.368673+0000","last_clean":"2024-02-19T21:13:58.368673+0000","last_became_active":"2024-02-19T21:12:09.350490+0000","last_became_peered":"2024-02-19T21:12:09.350490+0000","last_unstale":"2024-02-19T21:13:58.368673+0000","last_undegraded":"2024-02-19T21:13:58.368673+0000","last_fullsized":"2024-02-19T21:13:58.368673+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'11","last_scrub_stamp":"2024-02-19T21:10:54.435839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:54.435839+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":332,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.348680+0000","last_change":"2024-02-19T21:12:09.333078+0000","last_active":"2024-02-19T21:13:58.348680+0000","last_peered":"2024-02-19T21:13:58.348680+0000","last_clean":"2024-02-19T21:13:58.348680+0000","last_became_active":"2024-02-19T21:12:09.332587+0000","last_became_peered":"2024-02-19T21:12:09.332587+0000","last_unstale":"2024-02-19T21:13:58.348680+0000","last_undegraded":"2024-02-19T21:13:58.348680+0000","last_fullsized":"2024-02-19T21:13:58.348680+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:32.600213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:32.600213+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":406,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356637+0000","last_change":"2024-02-19T21:11:59.334743+0000","last_active":"2024-02-19T21:13:58.356637+0000","last_peered":"2024-02-19T21:13:58.356637+0000","last_clean":"2024-02-19T21:13:58.356637+0000","last_became_active":"2024-02-19T21:11:59.334454+0000","last_became_peered":"2024-02-19T21:11:59.334454+0000","last_unstale":"2024-02-19T21:13:58.356637+0000","last_undegraded":"2024-02-19T21:13:58.356637+0000","last_fullsized":"2024-02-19T21:13:58.356637+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:01.692260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:01.692260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":578,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.432241+0000","last_change":"2024-02-19T21:12:06.991132+0000","last_active":"2024-02-19T21:13:57.432241+0000","last_peered":"2024-02-19T21:13:57.432241+0000","last_clean":"2024-02-19T21:13:57.432241+0000","last_became_active":"2024-02-19T21:10:41.312577+0000","last_became_peered":"2024-02-19T21:10:41.312577+0000","last_unstale":"2024-02-19T21:13:57.432241+0000","last_undegraded":"2024-02-19T21:13:57.432241+0000","last_fullsized":"2024-02-19T21:13:57.432241+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:06.991039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:06.991039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":537,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.431555+0000","last_change":"2024-02-19T21:13:53.087054+0000","last_active":"2024-02-19T21:13:57.431555+0000","last_peered":"2024-02-19T21:13:57.431555+0000","last_clean":"2024-02-19T21:13:57.431555+0000","last_became_active":"2024-02-19T21:10:41.311050+0000","last_became_peered":"2024-02-19T21:10:41.311050+0000","last_unstale":"2024-02-19T21:13:57.431555+0000","last_undegraded":"2024-02-19T21:13:57.431555+0000","last_fullsized":"2024-02-19T21:13:57.431555+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:53.086972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:13:53.086972+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"459'34","reported_seq":672,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.364818+0000","last_change":"2024-02-19T21:13:39.513661+0000","last_active":"2024-02-19T21:13:58.364818+0000","last_peered":"2024-02-19T21:13:58.364818+0000","last_clean":"2024-02-19T21:13:58.364818+0000","last_became_active":"2024-02-19T21:08:04.248378+0000","last_became_peered":"2024-02-19T21:08:04.248378+0000","last_unstale":"2024-02-19T21:13:58.364818+0000","last_undegraded":"2024-02-19T21:13:58.364818+0000","last_fullsized":"2024-02-19T21:13:58.364818+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'34","last_scrub_stamp":"2024-02-19T21:13:39.513604+0000","last_deep_scrub":"419'22","last_deep_scrub_stamp":"2024-02-19T21:10:19.109244+0000","last_clean_scrub_stamp":"2024-02-19T21:13:39.513604+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":529,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351028+0000","last_change":"2024-02-19T21:12:09.341126+0000","last_active":"2024-02-19T21:13:58.351028+0000","last_peered":"2024-02-19T21:13:58.351028+0000","last_clean":"2024-02-19T21:13:58.351028+0000","last_became_active":"2024-02-19T21:12:09.340803+0000","last_became_peered":"2024-02-19T21:12:09.340803+0000","last_unstale":"2024-02-19T21:13:58.351028+0000","last_undegraded":"2024-02-19T21:13:58.351028+0000","last_fullsized":"2024-02-19T21:13:58.351028+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:03.379335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.379335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":589,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.432677+0000","last_change":"2024-02-19T21:11:59.323076+0000","last_active":"2024-02-19T21:13:57.432677+0000","last_peered":"2024-02-19T21:13:57.432677+0000","last_clean":"2024-02-19T21:13:57.432677+0000","last_became_active":"2024-02-19T21:11:59.322735+0000","last_became_peered":"2024-02-19T21:11:59.322735+0000","last_unstale":"2024-02-19T21:13:57.432677+0000","last_undegraded":"2024-02-19T21:13:57.432677+0000","last_fullsized":"2024-02-19T21:13:57.432677+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:53.709456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:10:53.709456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":197,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351552+0000","last_change":"2024-02-19T21:13:43.735451+0000","last_active":"2024-02-19T21:13:58.351552+0000","last_peered":"2024-02-19T21:13:58.351552+0000","last_clean":"2024-02-19T21:13:58.351552+0000","last_became_active":"2024-02-19T21:12:09.370318+0000","last_became_peered":"2024-02-19T21:12:09.370318+0000","last_unstale":"2024-02-19T21:13:58.351552+0000","last_undegraded":"2024-02-19T21:13:58.351552+0000","last_fullsized":"2024-02-19T21:13:58.351552+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:43.735382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:13:43.735382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":398,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.430764+0000","last_change":"2024-02-19T21:12:09.340710+0000","last_active":"2024-02-19T21:13:57.430764+0000","last_peered":"2024-02-19T21:13:57.430764+0000","last_clean":"2024-02-19T21:13:57.430764+0000","last_became_active":"2024-02-19T21:12:09.340360+0000","last_became_peered":"2024-02-19T21:12:09.340360+0000","last_unstale":"2024-02-19T21:13:57.430764+0000","last_undegraded":"2024-02-19T21:13:57.430764+0000","last_fullsized":"2024-02-19T21:13:57.430764+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.342173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.342173+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"459'18","reported_seq":382,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349884+0000","last_change":"2024-02-19T21:12:50.243591+0000","last_active":"2024-02-19T21:13:58.349884+0000","last_peered":"2024-02-19T21:13:58.349884+0000","last_clean":"2024-02-19T21:13:58.349884+0000","last_became_active":"2024-02-19T21:10:41.307690+0000","last_became_peered":"2024-02-19T21:10:41.307690+0000","last_unstale":"2024-02-19T21:13:58.349884+0000","last_undegraded":"2024-02-19T21:13:58.349884+0000","last_fullsized":"2024-02-19T21:13:58.349884+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'18","last_scrub_stamp":"2024-02-19T21:12:50.243498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:50.243498+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":323,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349851+0000","last_change":"2024-02-19T21:12:09.344823+0000","last_active":"2024-02-19T21:13:58.349851+0000","last_peered":"2024-02-19T21:13:58.349851+0000","last_clean":"2024-02-19T21:13:58.349851+0000","last_became_active":"2024-02-19T21:12:09.344422+0000","last_became_peered":"2024-02-19T21:12:09.344422+0000","last_unstale":"2024-02-19T21:13:58.349851+0000","last_undegraded":"2024-02-19T21:13:58.349851+0000","last_fullsized":"2024-02-19T21:13:58.349851+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:04.545988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:04.545988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":183,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351351+0000","last_change":"2024-02-19T21:11:43.369675+0000","last_active":"2024-02-19T21:13:58.351351+0000","last_peered":"2024-02-19T21:13:58.351351+0000","last_clean":"2024-02-19T21:13:58.351351+0000","last_became_active":"2024-02-19T21:10:41.296849+0000","last_became_peered":"2024-02-19T21:10:41.296849+0000","last_unstale":"2024-02-19T21:13:58.351351+0000","last_undegraded":"2024-02-19T21:13:58.351351+0000","last_fullsized":"2024-02-19T21:13:58.351351+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:43.369607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:43.369607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":323,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349969+0000","last_change":"2024-02-19T21:12:09.332385+0000","last_active":"2024-02-19T21:13:58.349969+0000","last_peered":"2024-02-19T21:13:58.349969+0000","last_clean":"2024-02-19T21:13:58.349969+0000","last_became_active":"2024-02-19T21:12:09.332044+0000","last_became_peered":"2024-02-19T21:12:09.332044+0000","last_unstale":"2024-02-19T21:13:58.349969+0000","last_undegraded":"2024-02-19T21:13:58.349969+0000","last_fullsized":"2024-02-19T21:13:58.349969+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:58.688166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:10:58.688166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":523,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.431024+0000","last_change":"2024-02-19T21:12:09.350012+0000","last_active":"2024-02-19T21:13:57.431024+0000","last_peered":"2024-02-19T21:13:57.431024+0000","last_clean":"2024-02-19T21:13:57.431024+0000","last_became_active":"2024-02-19T21:12:09.349709+0000","last_became_peered":"2024-02-19T21:12:09.349709+0000","last_unstale":"2024-02-19T21:13:57.431024+0000","last_undegraded":"2024-02-19T21:13:57.431024+0000","last_fullsized":"2024-02-19T21:13:57.431024+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:00.754327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:00.754327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"459'19","reported_seq":221,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351369+0000","last_change":"2024-02-19T21:12:55.429884+0000","last_active":"2024-02-19T21:13:58.351369+0000","last_peered":"2024-02-19T21:13:58.351369+0000","last_clean":"2024-02-19T21:13:58.351369+0000","last_became_active":"2024-02-19T21:11:59.336536+0000","last_became_peered":"2024-02-19T21:11:59.336536+0000","last_unstale":"2024-02-19T21:13:58.351369+0000","last_undegraded":"2024-02-19T21:13:58.351369+0000","last_fullsized":"2024-02-19T21:13:58.351369+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:12:55.429801+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.429801+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":565,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356229+0000","last_change":"2024-02-19T21:11:30.472665+0000","last_active":"2024-02-19T21:13:58.356229+0000","last_peered":"2024-02-19T21:13:58.356229+0000","last_clean":"2024-02-19T21:13:58.356229+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:13:58.356229+0000","last_undegraded":"2024-02-19T21:13:58.356229+0000","last_fullsized":"2024-02-19T21:13:58.356229+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:30.472591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:11:30.472591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":574,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367446+0000","last_change":"2024-02-19T21:13:06.395759+0000","last_active":"2024-02-19T21:13:58.367446+0000","last_peered":"2024-02-19T21:13:58.367446+0000","last_clean":"2024-02-19T21:13:58.367446+0000","last_became_active":"2024-02-19T21:11:59.323593+0000","last_became_peered":"2024-02-19T21:11:59.323593+0000","last_unstale":"2024-02-19T21:13:58.367446+0000","last_undegraded":"2024-02-19T21:13:58.367446+0000","last_fullsized":"2024-02-19T21:13:58.367446+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:06.395650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.395650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":406,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.430473+0000","last_change":"2024-02-19T21:13:43.109663+0000","last_active":"2024-02-19T21:13:57.430473+0000","last_peered":"2024-02-19T21:13:57.430473+0000","last_clean":"2024-02-19T21:13:57.430473+0000","last_became_active":"2024-02-19T21:11:59.352235+0000","last_became_peered":"2024-02-19T21:11:59.352235+0000","last_unstale":"2024-02-19T21:13:57.430473+0000","last_undegraded":"2024-02-19T21:13:57.430473+0000","last_fullsized":"2024-02-19T21:13:57.430473+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:43.109579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:13:43.109579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":549,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.365083+0000","last_change":"2024-02-19T21:13:10.146168+0000","last_active":"2024-02-19T21:13:58.365083+0000","last_peered":"2024-02-19T21:13:58.365083+0000","last_clean":"2024-02-19T21:13:58.365083+0000","last_became_active":"2024-02-19T21:11:59.357345+0000","last_became_peered":"2024-02-19T21:11:59.357345+0000","last_unstale":"2024-02-19T21:13:58.365083+0000","last_undegraded":"2024-02-19T21:13:58.365083+0000","last_fullsized":"2024-02-19T21:13:58.365083+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:10.146102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:13:10.146102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"459'7","reported_seq":192,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.351771+0000","last_change":"2024-02-19T21:12:02.521592+0000","last_active":"2024-02-19T21:13:58.351771+0000","last_peered":"2024-02-19T21:13:58.351771+0000","last_clean":"2024-02-19T21:13:58.351771+0000","last_became_active":"2024-02-19T21:10:41.297295+0000","last_became_peered":"2024-02-19T21:10:41.297295+0000","last_unstale":"2024-02-19T21:13:58.351771+0000","last_undegraded":"2024-02-19T21:13:58.351771+0000","last_fullsized":"2024-02-19T21:13:58.351771+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'7","last_scrub_stamp":"2024-02-19T21:12:02.521523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:02.521523+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":588,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356457+0000","last_change":"2024-02-19T21:12:09.324077+0000","last_active":"2024-02-19T21:13:58.356457+0000","last_peered":"2024-02-19T21:13:58.356457+0000","last_clean":"2024-02-19T21:13:58.356457+0000","last_became_active":"2024-02-19T21:12:09.323141+0000","last_became_peered":"2024-02-19T21:12:09.323141+0000","last_unstale":"2024-02-19T21:13:58.356457+0000","last_undegraded":"2024-02-19T21:13:58.356457+0000","last_fullsized":"2024-02-19T21:13:58.356457+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:04.547957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:04.547957+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":584,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.355780+0000","last_change":"2024-02-19T21:12:51.411694+0000","last_active":"2024-02-19T21:13:58.355780+0000","last_peered":"2024-02-19T21:13:58.355780+0000","last_clean":"2024-02-19T21:13:58.355780+0000","last_became_active":"2024-02-19T21:10:41.307400+0000","last_became_peered":"2024-02-19T21:10:41.307400+0000","last_unstale":"2024-02-19T21:13:58.355780+0000","last_undegraded":"2024-02-19T21:13:58.355780+0000","last_fullsized":"2024-02-19T21:13:58.355780+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:51.411574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.411574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":586,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367299+0000","last_change":"2024-02-19T21:13:54.928378+0000","last_active":"2024-02-19T21:13:58.367299+0000","last_peered":"2024-02-19T21:13:58.367299+0000","last_clean":"2024-02-19T21:13:58.367299+0000","last_became_active":"2024-02-19T21:08:04.266100+0000","last_became_peered":"2024-02-19T21:08:04.266100+0000","last_unstale":"2024-02-19T21:13:58.367299+0000","last_undegraded":"2024-02-19T21:13:58.367299+0000","last_fullsized":"2024-02-19T21:13:58.367299+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.928318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.928318+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":427,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.368401+0000","last_change":"2024-02-19T21:12:40.365196+0000","last_active":"2024-02-19T21:13:58.368401+0000","last_peered":"2024-02-19T21:13:58.368401+0000","last_clean":"2024-02-19T21:13:58.368401+0000","last_became_active":"2024-02-19T21:12:40.364772+0000","last_became_peered":"2024-02-19T21:12:40.364772+0000","last_unstale":"2024-02-19T21:13:58.368401+0000","last_undegraded":"2024-02-19T21:13:58.368401+0000","last_fullsized":"2024-02-19T21:13:58.368401+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:56.083403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:10:56.083403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"459'15","reported_seq":221,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.350801+0000","last_change":"2024-02-19T21:12:09.959562+0000","last_active":"2024-02-19T21:13:58.350801+0000","last_peered":"2024-02-19T21:13:58.350801+0000","last_clean":"2024-02-19T21:13:58.350801+0000","last_became_active":"2024-02-19T21:12:09.352882+0000","last_became_peered":"2024-02-19T21:12:09.352882+0000","last_unstale":"2024-02-19T21:13:58.350801+0000","last_undegraded":"2024-02-19T21:13:58.350801+0000","last_fullsized":"2024-02-19T21:13:58.350801+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'9","last_scrub_stamp":"2024-02-19T21:11:03.423767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:11:03.423767+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":323,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349666+0000","last_change":"2024-02-19T21:11:59.337265+0000","last_active":"2024-02-19T21:13:58.349666+0000","last_peered":"2024-02-19T21:13:58.349666+0000","last_clean":"2024-02-19T21:13:58.349666+0000","last_became_active":"2024-02-19T21:11:59.336778+0000","last_became_peered":"2024-02-19T21:11:59.336778+0000","last_unstale":"2024-02-19T21:13:58.349666+0000","last_undegraded":"2024-02-19T21:13:58.349666+0000","last_fullsized":"2024-02-19T21:13:58.349666+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:50.769620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:50.769620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":567,"reported_epoch":601,"state":"active+clean","last_fresh":"2024-02-19T21:13:57.430874+0000","last_change":"2024-02-19T21:13:42.086086+0000","last_active":"2024-02-19T21:13:57.430874+0000","last_peered":"2024-02-19T21:13:57.430874+0000","last_clean":"2024-02-19T21:13:57.430874+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:13:57.430874+0000","last_undegraded":"2024-02-19T21:13:57.430874+0000","last_fullsized":"2024-02-19T21:13:57.430874+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:42.086006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:13:42.086006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":183,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.352020+0000","last_change":"2024-02-19T21:12:07.416212+0000","last_active":"2024-02-19T21:13:58.352020+0000","last_peered":"2024-02-19T21:13:58.352020+0000","last_clean":"2024-02-19T21:13:58.352020+0000","last_became_active":"2024-02-19T21:10:41.309550+0000","last_became_peered":"2024-02-19T21:10:41.309550+0000","last_unstale":"2024-02-19T21:13:58.352020+0000","last_undegraded":"2024-02-19T21:13:58.352020+0000","last_fullsized":"2024-02-19T21:13:58.352020+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:07.416150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:12:07.416150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":544,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.367513+0000","last_change":"2024-02-19T21:11:59.333342+0000","last_active":"2024-02-19T21:13:58.367513+0000","last_peered":"2024-02-19T21:13:58.367513+0000","last_clean":"2024-02-19T21:13:58.367513+0000","last_became_active":"2024-02-19T21:11:59.332987+0000","last_became_peered":"2024-02-19T21:11:59.332987+0000","last_unstale":"2024-02-19T21:13:58.367513+0000","last_undegraded":"2024-02-19T21:13:58.367513+0000","last_fullsized":"2024-02-19T21:13:58.367513+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:08.043028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:08.043028+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"459'27","reported_seq":373,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.348450+0000","last_change":"2024-02-19T21:11:59.360178+0000","last_active":"2024-02-19T21:13:58.348450+0000","last_peered":"2024-02-19T21:13:58.348450+0000","last_clean":"2024-02-19T21:13:58.348450+0000","last_became_active":"2024-02-19T21:11:59.340940+0000","last_became_peered":"2024-02-19T21:11:59.340940+0000","last_unstale":"2024-02-19T21:13:58.348450+0000","last_undegraded":"2024-02-19T21:13:58.348450+0000","last_fullsized":"2024-02-19T21:13:58.348450+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2024-02-19T21:10:02.532799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:10:02.532799+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":440,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.356732+0000","last_change":"2024-02-19T21:13:52.780029+0000","last_active":"2024-02-19T21:13:58.356732+0000","last_peered":"2024-02-19T21:13:58.356732+0000","last_clean":"2024-02-19T21:13:58.356732+0000","last_became_active":"2024-02-19T21:12:09.338747+0000","last_became_peered":"2024-02-19T21:12:09.338747+0000","last_unstale":"2024-02-19T21:13:58.356732+0000","last_undegraded":"2024-02-19T21:13:58.356732+0000","last_fullsized":"2024-02-19T21:13:58.356732+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:52.779962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:52.779962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":192,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.350829+0000","last_change":"2024-02-19T21:12:09.346777+0000","last_active":"2024-02-19T21:13:58.350829+0000","last_peered":"2024-02-19T21:13:58.350829+0000","last_clean":"2024-02-19T21:13:58.350829+0000","last_became_active":"2024-02-19T21:12:09.346258+0000","last_became_peered":"2024-02-19T21:12:09.346258+0000","last_unstale":"2024-02-19T21:13:58.350829+0000","last_undegraded":"2024-02-19T21:13:58.350829+0000","last_fullsized":"2024-02-19T21:13:58.350829+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:02.407543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:11:02.407543+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":328,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349569+0000","last_change":"2024-02-19T21:13:40.304585+0000","last_active":"2024-02-19T21:13:58.349569+0000","last_peered":"2024-02-19T21:13:58.349569+0000","last_clean":"2024-02-19T21:13:58.349569+0000","last_became_active":"2024-02-19T21:12:09.330530+0000","last_became_peered":"2024-02-19T21:12:09.330530+0000","last_unstale":"2024-02-19T21:13:58.349569+0000","last_undegraded":"2024-02-19T21:13:58.349569+0000","last_fullsized":"2024-02-19T21:13:58.349569+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:40.304490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:40.304490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":346,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-02-19T21:13:58.349731+0000","last_change":"2024-02-19T21:12:09.345873+0000","last_active":"2024-02-19T21:13:58.349731+0000","last_peered":"2024-02-19T21:13:58.349731+0000","last_clean":"2024-02-19T21:13:58.349731+0000","last_became_active":"2024-02-19T21:12:09.345500+0000","last_became_peered":"2024-02-19T21:12:09.345500+0000","last_unstale":"2024-02-19T21:13:58.349731+0000","last_undegraded":"2024-02-19T21:13:58.349731+0000","last_fullsized":"2024-02-19T21:13:58.349731+0000","mapping_epoch":492,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":493,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:11:38.764658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:11:38.764658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":236,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":4,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":24,"acting":24,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1102,"num_read_kb":859,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":361,"num_bytes_recovered":7558,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":712,"ondisk_log_size":712,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":754,"num_read_kb":749,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":887,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":126,"acting":126,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":15,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266843,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300416,"kb_used_data":3192,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444736,"statfs":{"total":95995035648,"available":95687409664,"internally_reserved":0,"allocated":3268608,"data_stored":1607836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.48999999999999999,"15min":0.86299999999999999},"min":{"1min":0.32500000000000001,"5min":0.26000000000000001,"15min":0.215},"max":{"1min":0.66800000000000004,"5min":0.79200000000000004,"15min":89.456999999999994},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.48799999999999999,"15min":0.78600000000000003},"min":{"1min":0.41199999999999998,"5min":0.28499999999999998,"15min":0.25900000000000001},"max":{"1min":0.64900000000000002,"5min":0.81299999999999994,"15min":70.451999999999998},"last":0.80100000000000005}]},{"osd":1,"last update":"Mon Feb 19 21:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.502,"15min":0.69499999999999995},"min":{"1min":0.35699999999999998,"5min":0.29999999999999999,"15min":0.28199999999999997},"max":{"1min":0.67500000000000004,"5min":0.85099999999999998,"15min":34.273000000000003},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.48199999999999998,"15min":0.93100000000000005},"min":{"1min":0.34000000000000002,"5min":0.28299999999999997,"15min":0.254},"max":{"1min":0.70899999999999996,"5min":0.79200000000000004,"15min":89.594999999999999},"last":0.749}]},{"osd":2,"last update":"Mon Feb 19 21:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.52300000000000002,"15min":0.95199999999999996},"min":{"1min":0.34399999999999997,"5min":0.312,"15min":0.24299999999999999},"max":{"1min":4.3769999999999998,"5min":4.3769999999999998,"15min":104.413},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.53000000000000003,"15min":0.95299999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.27600000000000002},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":104.268},"last":0.78400000000000003}]},{"osd":3,"last update":"Mon Feb 19 21:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.57499999999999996,"15min":0.96299999999999997},"min":{"1min":0.41199999999999998,"5min":0.33400000000000002,"15min":0.254},"max":{"1min":2.9769999999999999,"5min":2.9769999999999999,"15min":105.3},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.53700000000000003,"15min":0.60899999999999999},"min":{"1min":0.36799999999999999,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":2.3050000000000002,"5min":2.3050000000000002,"15min":13.243},"last":0.63700000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.434,"15min":0.45700000000000002},"min":{"1min":0.31,"5min":0.28499999999999998,"15min":0.24299999999999999},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":1.373},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.48099999999999998,"15min":0.47699999999999998},"min":{"1min":0.34100000000000003,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":0.73399999999999999,"5min":0.751,"15min":1.0740000000000001},"last":0.57699999999999996}]},{"osd":5,"last update":"Mon Feb 19 21:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.45300000000000001,"15min":0.45600000000000002},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.252},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.497,"15min":0.504},"min":{"1min":0.36199999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63900000000000001,"5min":0.79300000000000004,"15min":0.85899999999999999},"last":0.622}]},{"osd":6,"last update":"Mon Feb 19 21:13:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.51300000000000001,"15min":0.51600000000000001},"min":{"1min":0.35999999999999999,"5min":0.30199999999999999,"15min":0.28799999999999998},"max":{"1min":0.70799999999999996,"5min":0.76700000000000002,"15min":1.0049999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.54300000000000004,"15min":0.55000000000000004},"min":{"1min":0.39500000000000002,"5min":0.34100000000000003,"15min":0.33700000000000002},"max":{"1min":0.747,"5min":0.749,"15min":1.262},"last":0.76500000000000001}]}]},{"osd":6,"up_from":0,"seq":193273528481,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834575032,"num_pgs":120,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312704,"kb_used_data":3192,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93432448,"statfs":{"total":95995035648,"available":95674826752,"internally_reserved":0,"allocated":3268608,"data_stored":1594648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.33300000000000002,"15min":0.54500000000000004},"min":{"1min":0.19700000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.38900000000000001,"5min":2.0379999999999998,"15min":27.777000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34799999999999998,"15min":0.57399999999999995},"min":{"1min":0.215,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.51900000000000002,"5min":1.9870000000000001,"15min":27.631},"last":0.35599999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:13:36 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.36799999999999999,"15min":0.70099999999999996},"min":{"1min":0.217,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.41999999999999998,"5min":2.0230000000000001,"15min":27.798999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.38700000000000001,"15min":0.71799999999999997},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.443,"5min":2.089,"15min":27.606999999999999},"last":0.313}]},{"osd":3,"last update":"Mon Feb 19 21:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.374,"15min":0.56599999999999995},"min":{"1min":0.253,"5min":0.217,"15min":0.217},"max":{"1min":0.499,"5min":1.7450000000000001,"15min":15.574999999999999},"last":0.376},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.41099999999999998,"15min":0.66200000000000003},"min":{"1min":0.31,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.622,"5min":2.0070000000000001,"15min":27.82},"last":0.442}]},{"osd":4,"last update":"Mon Feb 19 21:13:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51000000000000001,"15min":0.64700000000000002},"min":{"1min":0.313,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71999999999999997,"5min":1.149,"15min":9.6430000000000007},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54800000000000004,"15min":0.752},"min":{"1min":0.41399999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.79800000000000004,"5min":1.9770000000000001,"15min":27.707000000000001},"last":0.56399999999999995}]},{"osd":5,"last update":"Mon Feb 19 21:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54800000000000004,"15min":0.73899999999999999},"min":{"1min":0.42899999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.85399999999999998,"5min":1.048,"15min":27.529},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53000000000000003,"15min":0.67200000000000004},"min":{"1min":0.39500000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.98099999999999998,"5min":1.1819999999999999,"15min":9.6799999999999997},"last":0.433}]},{"osd":6,"last update":"Mon Feb 19 21:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52600000000000002,"15min":0.65800000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.70399999999999996,"5min":1.1659999999999999,"15min":9.6989999999999998},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.60099999999999998,"15min":0.78600000000000003},"min":{"1min":0.40799999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.75900000000000001,"5min":3.4359999999999999,"15min":27.733000000000001},"last":0.53500000000000003}]},{"osd":7,"last update":"Mon Feb 19 21:13:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57599999999999996,"15min":0.77400000000000002},"min":{"1min":0.40500000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79800000000000004,"5min":3.4670000000000001,"15min":27.576000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59199999999999997,"15min":0.80400000000000005},"min":{"1min":0.46800000000000003,"5min":0.375,"15min":0.375},"max":{"1min":0.81599999999999995,"5min":2.0169999999999999,"15min":27.756},"last":0.5}]}]},{"osd":0,"up_from":0,"seq":34359738558,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411507,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308592,"kb_used_data":3176,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436560,"statfs":{"total":95995035648,"available":95679037440,"internally_reserved":0,"allocated":3252224,"data_stored":1602672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.53500000000000003},"min":{"1min":0.223,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.53600000000000003,"5min":0.54600000000000004,"15min":26.690999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.34100000000000003,"15min":0.59199999999999997},"min":{"1min":0.24299999999999999,"5min":0.23400000000000001,"15min":0.214},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":24.315000000000001},"last":0.27100000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:13:36 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.34300000000000003,"15min":0.70599999999999996},"min":{"1min":0.22600000000000001,"5min":0.20599999999999999,"15min":0.20100000000000001},"max":{"1min":0.62,"5min":0.62,"15min":28.882000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.376,"15min":0.68200000000000005},"min":{"1min":0.246,"5min":0.23899999999999999,"15min":0.22500000000000001},"max":{"1min":0.61299999999999999,"5min":0.77600000000000002,"15min":28.382999999999999},"last":0.23400000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.53700000000000003,"15min":0.71199999999999997},"min":{"1min":0.27800000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.68899999999999995,"5min":12.75,"15min":28.742000000000001},"last":0.308},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.44900000000000001,"15min":0.63300000000000001},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.25700000000000001},"max":{"1min":0.58399999999999996,"5min":3.6499999999999999,"15min":26.719999999999999},"last":0.32900000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:13:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58399999999999996,"15min":0.69399999999999995},"min":{"1min":0.36899999999999999,"5min":0.33400000000000002,"15min":0.30599999999999999},"max":{"1min":0.85599999999999998,"5min":3.4039999999999999,"15min":24.356000000000002},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60299999999999998,"15min":0.71699999999999997},"min":{"1min":0.38300000000000001,"5min":0.33700000000000002,"15min":0.32400000000000001},"max":{"1min":1.2010000000000001,"5min":3.3919999999999999,"15min":20.707000000000001},"last":0.436}]},{"osd":5,"last update":"Mon Feb 19 21:13:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.54300000000000004,"15min":0.67300000000000004},"min":{"1min":0.39900000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":9.9499999999999993},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57399999999999995,"15min":0.78400000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.135,"5min":1.135,"15min":26.606000000000002},"last":0.45000000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:13:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.58299999999999996,"15min":0.79800000000000004},"min":{"1min":0.438,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.876,"5min":1.4510000000000001,"15min":24.381},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.73399999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.80200000000000005,"5min":0.93600000000000005,"15min":28.706},"last":0.47199999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:13:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59199999999999997,"15min":0.77800000000000002},"min":{"1min":0.434,"5min":0.42199999999999999,"15min":0.34200000000000003},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":26.658000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59299999999999997,"15min":0.73399999999999999},"min":{"1min":0.42099999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":24.402000000000001},"last":0.52300000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182575,"num_pgs":104,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308604,"kb_used_data":3188,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93436548,"statfs":{"total":95995035648,"available":95679025152,"internally_reserved":0,"allocated":3264512,"data_stored":1597639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.71599999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.156},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":77.662000000000006},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.35499999999999998,"15min":0.49399999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.19700000000000001},"max":{"1min":0.60199999999999998,"5min":0.60999999999999999,"15min":17.456},"last":0.35999999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.41499999999999998,"15min":0.52600000000000002},"min":{"1min":0.255,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":0.66300000000000003,"5min":6.9299999999999997,"15min":17.327999999999999},"last":0.307},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.42499999999999999,"15min":0.50600000000000001},"min":{"1min":0.20699999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.95599999999999996,"5min":7.3620000000000001,"15min":15.004},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:13:00 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.376,"15min":0.66300000000000003},"min":{"1min":0.27300000000000002,"5min":0.219,"15min":0.219},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":56.890999999999998},"last":0.375},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.36599999999999999,"15min":0.64400000000000002},"min":{"1min":0.246,"5min":0.24399999999999999,"15min":0.20899999999999999},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":44.600999999999999},"last":0.33500000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:13:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54000000000000004,"15min":0.63600000000000001},"min":{"1min":0.36099999999999999,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":12.847},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.53300000000000003,"15min":0.70499999999999996},"min":{"1min":0.317,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":26.393999999999998},"last":0.61799999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:13:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.61699999999999999,"15min":0.68200000000000005},"min":{"1min":0.39000000000000001,"5min":0.34100000000000003,"15min":0.313},"max":{"1min":0.70799999999999996,"5min":9.2360000000000007,"15min":17.401},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55900000000000005,"15min":0.65500000000000003},"min":{"1min":0.36799999999999999,"5min":0.313,"15min":0.313},"max":{"1min":1.397,"5min":1.9690000000000001,"15min":12.882},"last":0.54700000000000004}]},{"osd":6,"last update":"Mon Feb 19 21:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.56000000000000005,"15min":0.65200000000000002},"min":{"1min":0.38,"5min":0.34799999999999998,"15min":0.34300000000000003},"max":{"1min":0.69999999999999996,"5min":1.9790000000000001,"15min":17.367000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54900000000000004,"15min":0.66200000000000003},"min":{"1min":0.377,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":0.89400000000000002,"5min":0.95099999999999996,"15min":12.904},"last":0.58699999999999997}]},{"osd":7,"last update":"Mon Feb 19 21:13:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56599999999999995,"15min":0.65700000000000003},"min":{"1min":0.42799999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.751,"5min":1.038,"15min":17.428999999999998},"last":0.627},{"interface":"front","average":{"1min":0.626,"5min":0.58999999999999997,"15min":0.80300000000000005},"min":{"1min":0.39700000000000002,"5min":0.374,"15min":0.34499999999999997},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":44.378},"last":0.54000000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986347,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312676,"kb_used_data":3164,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93432476,"statfs":{"total":95995035648,"available":95674855424,"internally_reserved":0,"allocated":3239936,"data_stored":1597181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.56100000000000005,"15min":0.63600000000000001},"min":{"1min":0.35199999999999998,"5min":0.26800000000000002,"15min":0.22500000000000001},"max":{"1min":0.86899999999999999,"5min":8.0530000000000008,"15min":19.878},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51000000000000001,"15min":0.85999999999999999},"min":{"1min":0.32300000000000001,"5min":0.25900000000000001,"15min":0.20499999999999999},"max":{"1min":0.82999999999999996,"5min":3.3530000000000002,"15min":70.063000000000002},"last":0.53500000000000003}]},{"osd":1,"last update":"Mon Feb 19 21:13:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.49299999999999999,"15min":0.63800000000000001},"min":{"1min":0.34499999999999997,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":26.510999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.504,"15min":0.68300000000000005},"min":{"1min":0.32100000000000001,"5min":0.312,"15min":0.23200000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":25.949000000000002},"last":0.495}]},{"osd":2,"last update":"Mon Feb 19 21:13:24 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.60199999999999998,"15min":0.72499999999999998},"min":{"1min":0.38700000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.2040000000000002,"5min":8.1240000000000006,"15min":37.320999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53500000000000003,"15min":1.0600000000000001},"min":{"1min":0.39900000000000002,"5min":0.29899999999999999,"15min":0.253},"max":{"1min":0.86599999999999999,"5min":2.9969999999999999,"15min":69.936999999999998},"last":0.40200000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:13:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54500000000000004,"15min":0.91200000000000003},"min":{"1min":0.38400000000000001,"5min":0.28999999999999998,"15min":0.27700000000000002},"max":{"1min":0.76700000000000002,"5min":3.323,"15min":69.793000000000006},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.54800000000000004,"15min":0.76500000000000001},"min":{"1min":0.41199999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.621,"5min":1.621,"15min":17.986999999999998},"last":0.48499999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.442,"15min":0.42699999999999999},"min":{"1min":0.24099999999999999,"5min":0.22900000000000001,"15min":0.218},"max":{"1min":1.532,"5min":1.532,"15min":3.2799999999999998},"last":0.31},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.499,"15min":0.46999999999999997},"min":{"1min":0.27900000000000003,"5min":0.252,"15min":0.23899999999999999},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":4.4340000000000002},"last":0.38400000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.57399999999999995,"15min":0.47599999999999998},"min":{"1min":0.29199999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.78700000000000003,"5min":8.0969999999999995,"15min":8.0969999999999995},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.51800000000000002},"min":{"1min":0.35599999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.4550000000000001,"5min":3.0270000000000001,"15min":4.5609999999999999},"last":0.504}]},{"osd":7,"last update":"Mon Feb 19 21:13:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57799999999999996,"15min":0.52100000000000002},"min":{"1min":0.30499999999999999,"5min":0.30299999999999999,"15min":0.29999999999999999},"max":{"1min":1.5049999999999999,"5min":3.2909999999999999,"15min":6.0730000000000004},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56200000000000006,"15min":0.52000000000000002},"min":{"1min":0.34599999999999997,"5min":0.309,"15min":0.26400000000000001},"max":{"1min":0.69299999999999995,"5min":1.0209999999999999,"15min":3.202},"last":0.57799999999999996}]}]},{"osd":5,"up_from":37,"seq":158913790118,"num_pgs":110,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300400,"kb_used_data":3176,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444752,"statfs":{"total":95995035648,"available":95687426048,"internally_reserved":0,"allocated":3252224,"data_stored":1594774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45500000000000002,"15min":1.875},"min":{"1min":0.32900000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.61599999999999999,"5min":0.71699999999999997,"15min":276.47899999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.49099999999999999,"15min":0.89900000000000002},"min":{"1min":0.35099999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.64700000000000002,"5min":0.95299999999999996,"15min":52.887},"last":0.504}]},{"osd":1,"last update":"Mon Feb 19 21:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46400000000000002,"15min":1.2609999999999999},"min":{"1min":0.30199999999999999,"5min":0.254,"15min":0.23499999999999999},"max":{"1min":0.61399999999999999,"5min":1.599,"15min":120.011},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.503,"15min":1.02},"min":{"1min":0.372,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":118.105},"last":0.52000000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47799999999999998,"15min":0.98899999999999999},"min":{"1min":0.32300000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64600000000000002,"5min":0.92300000000000004,"15min":82.242000000000004},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.56100000000000005,"15min":1.5109999999999999},"min":{"1min":0.41599999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65400000000000003,"5min":4.5019999999999998,"15min":232.84999999999999},"last":0.44400000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54700000000000004,"15min":1.0669999999999999},"min":{"1min":0.34999999999999998,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":2.2210000000000001,"5min":4.4640000000000004,"15min":109.70099999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52500000000000002,"15min":0.90800000000000003},"min":{"1min":0.41699999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.64100000000000001,"5min":1.4590000000000001,"15min":65.519000000000005},"last":0.48599999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.44600000000000001,"15min":0.42799999999999999},"min":{"1min":0.33100000000000002,"5min":0.26900000000000002,"15min":0.23300000000000001},"max":{"1min":0.66100000000000003,"5min":1.0129999999999999,"15min":1.861},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.45700000000000002,"15min":0.442},"min":{"1min":0.371,"5min":0.255,"15min":0.248},"max":{"1min":0.71299999999999997,"5min":0.72499999999999998,"15min":1.9730000000000001},"last":0.45800000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:12:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.47299999999999998,"15min":0.45500000000000002},"min":{"1min":0.31,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69599999999999995,"5min":1.0289999999999999,"15min":1.823},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47199999999999998,"15min":0.46300000000000002},"min":{"1min":0.30499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.40999999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:13:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.502,"15min":0.502},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.28499999999999998},"max":{"1min":0.73699999999999999,"5min":1.052,"15min":1.8879999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.52300000000000002,"15min":0.51000000000000001},"min":{"1min":0.372,"5min":0.34399999999999997,"15min":0.27800000000000002},"max":{"1min":0.754,"5min":0.871,"15min":0.96799999999999997},"last":0.497}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:14:05.058 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:14:05.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:05.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:14:05.162 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:14:05.188 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:05.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.382+0000 7f00c40ef700 1 -- 172.21.15.163:0/4200565298 >> v1:172.21.15.163:6789/0 conn(0x7f00bc1042c0 legacy=0x7f00bc106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:05.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.382+0000 7f00c40ef700 1 -- 172.21.15.163:0/4200565298 shutdown_connections 2024-02-19T21:14:05.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.382+0000 7f00c40ef700 1 -- 172.21.15.163:0/4200565298 >> 172.21.15.163:0/4200565298 conn(0x7f00bc0f35a0 msgr2=0x7f00bc0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:05.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.382+0000 7f00c40ef700 1 -- 172.21.15.163:0/4200565298 shutdown_connections 2024-02-19T21:14:05.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.382+0000 7f00c40ef700 1 -- 172.21.15.163:0/4200565298 wait complete. 2024-02-19T21:14:05.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00c40ef700 1 librados: starting msgr at 2024-02-19T21:14:05.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00c40ef700 1 librados: starting objecter 2024-02-19T21:14:05.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00c40ef700 1 Processor -- start 2024-02-19T21:14:05.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00c40ef700 1 -- start start 2024-02-19T21:14:05.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00c40ef700 1 librados: setting wanted keys 2024-02-19T21:14:05.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00c40ef700 1 librados: calling monclient init 2024-02-19T21:14:05.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00c40ef700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f00bc1172c0 con 0x7f00bc100780 2024-02-19T21:14:05.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00c40ef700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f00bc1184a0 con 0x7f00bc1042c0 2024-02-19T21:14:05.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00c168a700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f00bc100780 0x7f00bc0fa070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46344/0 (socket says 172.21.15.163:46344) 2024-02-19T21:14:05.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00c168a700 1 -- 172.21.15.163:0/2426119225 learned_addr learned my addr 172.21.15.163:0/2426119225 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:05.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00c1e8b700 1 --1- 172.21.15.163:0/2426119225 >> v1:172.21.15.163:6790/0 conn(0x7f00bc0fd980 0x7f00bc0f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:49820/0 (socket says 172.21.15.163:49820) 2024-02-19T21:14:05.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f00bc1196a0 con 0x7f00bc0fd980 2024-02-19T21:14:05.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00b2ffd700 10 client.?.objecter ms_handle_connect 0x7f00bc100780 2024-02-19T21:14:05.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00b2ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:05.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00b2ffd700 10 client.?.objecter ms_handle_connect 0x7f00bc0fd980 2024-02-19T21:14:05.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00b2ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:05.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00b2ffd700 10 client.?.objecter ms_handle_connect 0x7f00bc1042c0 2024-02-19T21:14:05.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00b2ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:05.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770874472 0 0) 0x7f00bc1172c0 con 0x7f00bc100780 2024-02-19T21:14:05.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00a80033f0 con 0x7f00bc100780 2024-02-19T21:14:05.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282237324 0 0) 0x7f00bc1196a0 con 0x7f00bc0fd980 2024-02-19T21:14:05.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00bc1172c0 con 0x7f00bc0fd980 2024-02-19T21:14:05.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998785460 0 0) 0x7f00bc1184a0 con 0x7f00bc1042c0 2024-02-19T21:14:05.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00bc1196a0 con 0x7f00bc1042c0 2024-02-19T21:14:05.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 297264970 0 0) 0x7f00bc1196a0 con 0x7f00bc1042c0 2024-02-19T21:14:05.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.383+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00bc1184a0 con 0x7f00bc1042c0 2024-02-19T21:14:05.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f00b4001360 con 0x7f00bc1042c0 2024-02-19T21:14:05.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3805780057 0 0) 0x7f00bc1172c0 con 0x7f00bc0fd980 2024-02-19T21:14:05.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00bc1196a0 con 0x7f00bc0fd980 2024-02-19T21:14:05.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1070423174 0 0) 0x7f00a80033f0 con 0x7f00bc100780 2024-02-19T21:14:05.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00bc1172c0 con 0x7f00bc100780 2024-02-19T21:14:05.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f00b8001840 con 0x7f00bc0fd980 2024-02-19T21:14:05.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f00ac0019d0 con 0x7f00bc100780 2024-02-19T21:14:05.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3345895171 0 0) 0x7f00bc1184a0 con 0x7f00bc1042c0 2024-02-19T21:14:05.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 >> v1:172.21.15.163:6790/0 conn(0x7f00bc0fd980 legacy=0x7f00bc0f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:05.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 >> v1:172.21.15.163:6789/0 conn(0x7f00bc100780 legacy=0x7f00bc0fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:05.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00bc11a8a0 con 0x7f00bc1042c0 2024-02-19T21:14:05.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00bc119890 con 0x7f00bc1042c0 2024-02-19T21:14:05.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00c40ef700 10 client.49087.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:05.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00bc119d00 con 0x7f00bc1042c0 2024-02-19T21:14:05.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f00b4001e30 con 0x7f00bc1042c0 2024-02-19T21:14:05.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00c40ef700 1 librados: init done 2024-02-19T21:14:05.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f00b4003ad0 con 0x7f00bc1042c0 2024-02-19T21:14:05.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.384+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f00bc05f660 con 0x7f00bc1042c0 2024-02-19T21:14:05.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.385+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f00b4003030 con 0x7f00bc1042c0 2024-02-19T21:14:05.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.385+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(609..609 src has 434..609) v4 ==== 8508+0+0 (unknown 3011121702 0 0) 0x7f00b4076dd0 con 0x7f00bc1042c0 2024-02-19T21:14:05.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.385+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 osd_map(609..609 src has 434..609) v4 2024-02-19T21:14:05.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.385+0000 7f00b2ffd700 3 client.49087.objecter handle_osd_map got epochs [609,609] > 0 2024-02-19T21:14:05.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.385+0000 7f00b2ffd700 3 client.49087.objecter handle_osd_map decoding full epoch 609 2024-02-19T21:14:05.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.386+0000 7f00b2ffd700 20 client.49087.objecter dump_active .. 0 homeless 2024-02-19T21:14:05.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.386+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v609) v1 ==== 54+0+16 (unknown 201890337 0 2420000192) 0x7f00b40791c0 con 0x7f00bc1042c0 2024-02-19T21:14:05.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.386+0000 7f00b2ffd700 10 client.49087.objecter ms_handle_connect 0x7f00a8063eb0 2024-02-19T21:14:05.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.386+0000 7f00c40ef700 10 client.49087.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:05.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.386+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 1 2024-02-19T21:14:05.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.386+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc0f9200 osd=0 3 2024-02-19T21:14:05.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.386+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc0f9200 osd=0 3 2024-02-19T21:14:05.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.386+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc0f9200 osd=0 3 2024-02-19T21:14:05.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.386+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 1 2024-02-19T21:14:05.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.386+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc0f9200 osd=0 3 2024-02-19T21:14:05.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.386+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 0 1 2024-02-19T21:14:05.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.386+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc0f9200 osd=0 4 2024-02-19T21:14:05.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.386+0000 7f00c40ef700 10 client.49087.objecter _send_command 1 2024-02-19T21:14:05.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.386+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00bc109ea0 con 0x7f00bc118630 2024-02-19T21:14:05.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.386+0000 7f00b2ffd700 10 client.49087.objecter ms_handle_connect 0x7f00bc118630 2024-02-19T21:14:05.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.387+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f00bc109ea0 con 0x7f00bc118630 2024-02-19T21:14:05.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.387+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:14:05.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.387+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 1 = system:0 2024-02-19T21:14:05.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.387+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc0f9200 osd=0 4 2024-02-19T21:14:05.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.387+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 0 1 2024-02-19T21:14:05.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.401+0000 7f00c40ef700 10 client.49087.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:05.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.401+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 2 2024-02-19T21:14:05.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.401+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc0f9200 osd=0 3 2024-02-19T21:14:05.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.401+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc0f9200 osd=0 3 2024-02-19T21:14:05.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.401+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc0f9200 osd=0 3 2024-02-19T21:14:05.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.401+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 2 2024-02-19T21:14:05.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.401+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc0f9200 osd=0 3 2024-02-19T21:14:05.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.401+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 0 2 2024-02-19T21:14:05.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.401+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc0f9200 osd=0 4 2024-02-19T21:14:05.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.401+0000 7f00c40ef700 10 client.49087.objecter _send_command 2 2024-02-19T21:14:05.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.401+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f00bc119360 con 0x7f00bc118630 2024-02-19T21:14:05.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f00bc119360 con 0x7f00bc118630 2024-02-19T21:14:05.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:05.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc0f9200 osd=0 4 2024-02-19T21:14:05.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 0 2 2024-02-19T21:14:05.403 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:14:05.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00c40ef700 10 client.49087.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:05.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 3 2024-02-19T21:14:05.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc1194d0 osd=1 3 2024-02-19T21:14:05.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc1194d0 osd=1 3 2024-02-19T21:14:05.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc1194d0 osd=1 3 2024-02-19T21:14:05.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 3 2024-02-19T21:14:05.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc1194d0 osd=1 3 2024-02-19T21:14:05.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 1 3 2024-02-19T21:14:05.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc1194d0 osd=1 4 2024-02-19T21:14:05.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00c40ef700 10 client.49087.objecter _send_command 3 2024-02-19T21:14:05.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00bc059cd0 con 0x7f00bc0598f0 2024-02-19T21:14:05.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.403+0000 7f00b2ffd700 10 client.49087.objecter ms_handle_connect 0x7f00bc0598f0 2024-02-19T21:14:05.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.404+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f00bc059cd0 con 0x7f00bc0598f0 2024-02-19T21:14:05.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.404+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:14:05.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.404+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 3 = system:0 2024-02-19T21:14:05.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.404+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc1194d0 osd=1 4 2024-02-19T21:14:05.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.404+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 1 3 2024-02-19T21:14:05.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.418+0000 7f00c40ef700 10 client.49087.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:05.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.418+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 4 2024-02-19T21:14:05.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.418+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc1194d0 osd=1 3 2024-02-19T21:14:05.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.418+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc1194d0 osd=1 3 2024-02-19T21:14:05.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.418+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc1194d0 osd=1 3 2024-02-19T21:14:05.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.418+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 4 2024-02-19T21:14:05.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.418+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc1194d0 osd=1 3 2024-02-19T21:14:05.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.418+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 1 4 2024-02-19T21:14:05.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.418+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc1194d0 osd=1 4 2024-02-19T21:14:05.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.418+0000 7f00c40ef700 10 client.49087.objecter _send_command 4 2024-02-19T21:14:05.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.418+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f00bc05a0b0 con 0x7f00bc0598f0 2024-02-19T21:14:05.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f00bc05a0b0 con 0x7f00bc0598f0 2024-02-19T21:14:05.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:05.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc1194d0 osd=1 4 2024-02-19T21:14:05.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 1 4 2024-02-19T21:14:05.420 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:14:05.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00c40ef700 10 client.49087.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:05.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 5 2024-02-19T21:14:05.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc059360 osd=2 3 2024-02-19T21:14:05.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc059360 osd=2 3 2024-02-19T21:14:05.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc059360 osd=2 3 2024-02-19T21:14:05.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 5 2024-02-19T21:14:05.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc059360 osd=2 3 2024-02-19T21:14:05.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 2 5 2024-02-19T21:14:05.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc059360 osd=2 4 2024-02-19T21:14:05.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00c40ef700 10 client.49087.objecter _send_command 5 2024-02-19T21:14:05.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00bc0287b0 con 0x7f00bc0594d0 2024-02-19T21:14:05.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.420+0000 7f00b2ffd700 10 client.49087.objecter ms_handle_connect 0x7f00bc0594d0 2024-02-19T21:14:05.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.421+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f00bc0287b0 con 0x7f00bc0594d0 2024-02-19T21:14:05.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.421+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:14:05.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.421+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 5 = system:0 2024-02-19T21:14:05.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.421+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc059360 osd=2 4 2024-02-19T21:14:05.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.421+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 2 5 2024-02-19T21:14:05.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.435+0000 7f00c40ef700 10 client.49087.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:05.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.435+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 6 2024-02-19T21:14:05.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.435+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc059360 osd=2 3 2024-02-19T21:14:05.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.435+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc059360 osd=2 3 2024-02-19T21:14:05.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.435+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc059360 osd=2 3 2024-02-19T21:14:05.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.435+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 6 2024-02-19T21:14:05.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.435+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc059360 osd=2 3 2024-02-19T21:14:05.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.435+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 2 6 2024-02-19T21:14:05.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.435+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc059360 osd=2 4 2024-02-19T21:14:05.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.435+0000 7f00c40ef700 10 client.49087.objecter _send_command 6 2024-02-19T21:14:05.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.435+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f00bc027d10 con 0x7f00bc0594d0 2024-02-19T21:14:05.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f00bc027d10 con 0x7f00bc0594d0 2024-02-19T21:14:05.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:05.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc059360 osd=2 4 2024-02-19T21:14:05.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 2 6 2024-02-19T21:14:05.437 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:14:05.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00c40ef700 10 client.49087.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:05.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 7 2024-02-19T21:14:05.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc0280c0 osd=3 3 2024-02-19T21:14:05.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc0280c0 osd=3 3 2024-02-19T21:14:05.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc0280c0 osd=3 3 2024-02-19T21:14:05.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 7 2024-02-19T21:14:05.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc0280c0 osd=3 3 2024-02-19T21:14:05.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 3 7 2024-02-19T21:14:05.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc0280c0 osd=3 4 2024-02-19T21:14:05.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00c40ef700 10 client.49087.objecter _send_command 7 2024-02-19T21:14:05.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00bc127b00 con 0x7f00bc1053d0 2024-02-19T21:14:05.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.437+0000 7f00b2ffd700 10 client.49087.objecter ms_handle_connect 0x7f00bc1053d0 2024-02-19T21:14:05.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.438+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f00bc127b00 con 0x7f00bc1053d0 2024-02-19T21:14:05.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.438+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:14:05.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.438+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 7 = system:0 2024-02-19T21:14:05.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.438+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc0280c0 osd=3 4 2024-02-19T21:14:05.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.438+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 3 7 2024-02-19T21:14:05.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.452+0000 7f00c40ef700 10 client.49087.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:05.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.452+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 8 2024-02-19T21:14:05.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.452+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc0280c0 osd=3 3 2024-02-19T21:14:05.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.452+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc0280c0 osd=3 3 2024-02-19T21:14:05.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.452+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc0280c0 osd=3 3 2024-02-19T21:14:05.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.452+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 8 2024-02-19T21:14:05.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.452+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc0280c0 osd=3 3 2024-02-19T21:14:05.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.452+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 3 8 2024-02-19T21:14:05.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.452+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc0280c0 osd=3 4 2024-02-19T21:14:05.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.452+0000 7f00c40ef700 10 client.49087.objecter _send_command 8 2024-02-19T21:14:05.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.452+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f00bc104930 con 0x7f00bc1053d0 2024-02-19T21:14:05.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f00bc104930 con 0x7f00bc1053d0 2024-02-19T21:14:05.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:05.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc0280c0 osd=3 4 2024-02-19T21:14:05.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 3 8 2024-02-19T21:14:05.454 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:14:05.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00c40ef700 10 client.49087.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:05.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 9 2024-02-19T21:14:05.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc104930 osd=4 3 2024-02-19T21:14:05.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc104930 osd=4 3 2024-02-19T21:14:05.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc104930 osd=4 3 2024-02-19T21:14:05.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 9 2024-02-19T21:14:05.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc104930 osd=4 3 2024-02-19T21:14:05.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 4 9 2024-02-19T21:14:05.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc104930 osd=4 4 2024-02-19T21:14:05.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00c40ef700 10 client.49087.objecter _send_command 9 2024-02-19T21:14:05.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00bc104bf0 con 0x7f00bc123f30 2024-02-19T21:14:05.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.454+0000 7f00b2ffd700 10 client.49087.objecter ms_handle_connect 0x7f00bc123f30 2024-02-19T21:14:05.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.455+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f00bc104bf0 con 0x7f00bc123f30 2024-02-19T21:14:05.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.455+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:14:05.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.455+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 9 = system:0 2024-02-19T21:14:05.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.455+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc104930 osd=4 4 2024-02-19T21:14:05.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.455+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 4 9 2024-02-19T21:14:05.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.471+0000 7f00c40ef700 10 client.49087.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:05.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.471+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 10 2024-02-19T21:14:05.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.471+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc104930 osd=4 3 2024-02-19T21:14:05.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.471+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc104930 osd=4 3 2024-02-19T21:14:05.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.471+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc104930 osd=4 3 2024-02-19T21:14:05.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.471+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 10 2024-02-19T21:14:05.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.471+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc104930 osd=4 3 2024-02-19T21:14:05.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.471+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 4 10 2024-02-19T21:14:05.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.471+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc104930 osd=4 4 2024-02-19T21:14:05.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.471+0000 7f00c40ef700 10 client.49087.objecter _send_command 10 2024-02-19T21:14:05.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.471+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f00bc104bf0 con 0x7f00bc123f30 2024-02-19T21:14:05.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f00bc104bf0 con 0x7f00bc123f30 2024-02-19T21:14:05.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:05.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc104930 osd=4 4 2024-02-19T21:14:05.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 4 10 2024-02-19T21:14:05.474 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:14:05.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00c40ef700 10 client.49087.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:05.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 11 2024-02-19T21:14:05.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc104bf0 osd=5 3 2024-02-19T21:14:05.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc104bf0 osd=5 3 2024-02-19T21:14:05.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc104bf0 osd=5 3 2024-02-19T21:14:05.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 11 2024-02-19T21:14:05.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc104bf0 osd=5 3 2024-02-19T21:14:05.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 5 11 2024-02-19T21:14:05.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc104bf0 osd=5 4 2024-02-19T21:14:05.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00c40ef700 10 client.49087.objecter _send_command 11 2024-02-19T21:14:05.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00bc130580 con 0x7f00bc12c8c0 2024-02-19T21:14:05.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.473+0000 7f00b2ffd700 10 client.49087.objecter ms_handle_connect 0x7f00bc12c8c0 2024-02-19T21:14:05.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.474+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f00bc130580 con 0x7f00bc12c8c0 2024-02-19T21:14:05.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.474+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:14:05.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.474+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 11 = system:0 2024-02-19T21:14:05.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.474+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc104bf0 osd=5 4 2024-02-19T21:14:05.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.474+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 5 11 2024-02-19T21:14:05.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.488+0000 7f00c40ef700 10 client.49087.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:05.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.488+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 12 2024-02-19T21:14:05.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.488+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc104bf0 osd=5 3 2024-02-19T21:14:05.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.488+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc104bf0 osd=5 3 2024-02-19T21:14:05.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.488+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc104bf0 osd=5 3 2024-02-19T21:14:05.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.488+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 12 2024-02-19T21:14:05.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.488+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc104bf0 osd=5 3 2024-02-19T21:14:05.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.488+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 5 12 2024-02-19T21:14:05.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.488+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc104bf0 osd=5 4 2024-02-19T21:14:05.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.488+0000 7f00c40ef700 10 client.49087.objecter _send_command 12 2024-02-19T21:14:05.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.488+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f00bc130580 con 0x7f00bc12c8c0 2024-02-19T21:14:05.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f00bc130580 con 0x7f00bc12c8c0 2024-02-19T21:14:05.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:05.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc104bf0 osd=5 4 2024-02-19T21:14:05.490 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 5 12 2024-02-19T21:14:05.491 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:14:05.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00c40ef700 10 client.49087.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:05.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 13 2024-02-19T21:14:05.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc130580 osd=6 3 2024-02-19T21:14:05.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc130580 osd=6 3 2024-02-19T21:14:05.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc130580 osd=6 3 2024-02-19T21:14:05.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 13 2024-02-19T21:14:05.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc130580 osd=6 3 2024-02-19T21:14:05.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 6 13 2024-02-19T21:14:05.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc130580 osd=6 4 2024-02-19T21:14:05.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00c40ef700 10 client.49087.objecter _send_command 13 2024-02-19T21:14:05.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.490+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00bc1317a0 con 0x7f00bc132240 2024-02-19T21:14:05.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.491+0000 7f00b2ffd700 10 client.49087.objecter ms_handle_connect 0x7f00bc132240 2024-02-19T21:14:05.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.491+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f00bc1317a0 con 0x7f00bc132240 2024-02-19T21:14:05.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.491+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:14:05.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.491+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 13 = system:0 2024-02-19T21:14:05.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.491+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc130580 osd=6 4 2024-02-19T21:14:05.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.491+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 6 13 2024-02-19T21:14:05.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.505+0000 7f00c40ef700 10 client.49087.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:05.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.505+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 14 2024-02-19T21:14:05.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.505+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc130580 osd=6 3 2024-02-19T21:14:05.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.505+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc130580 osd=6 3 2024-02-19T21:14:05.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.505+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc130580 osd=6 3 2024-02-19T21:14:05.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.505+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 14 2024-02-19T21:14:05.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.505+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc130580 osd=6 3 2024-02-19T21:14:05.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.505+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 6 14 2024-02-19T21:14:05.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.505+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc130580 osd=6 4 2024-02-19T21:14:05.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.505+0000 7f00c40ef700 10 client.49087.objecter _send_command 14 2024-02-19T21:14:05.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.505+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f00bc1317a0 con 0x7f00bc132240 2024-02-19T21:14:05.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f00bc1317a0 con 0x7f00bc132240 2024-02-19T21:14:05.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:05.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc130580 osd=6 4 2024-02-19T21:14:05.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 6 14 2024-02-19T21:14:05.507 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:14:05.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00c40ef700 10 client.49087.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:05.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 15 2024-02-19T21:14:05.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc1317a0 osd=7 3 2024-02-19T21:14:05.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc1317a0 osd=7 3 2024-02-19T21:14:05.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc1317a0 osd=7 3 2024-02-19T21:14:05.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 15 2024-02-19T21:14:05.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc1317a0 osd=7 3 2024-02-19T21:14:05.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 7 15 2024-02-19T21:14:05.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc1317a0 osd=7 4 2024-02-19T21:14:05.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00c40ef700 10 client.49087.objecter _send_command 15 2024-02-19T21:14:05.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00bc138f50 con 0x7f00bc135440 2024-02-19T21:14:05.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.507+0000 7f00b2ffd700 10 client.49087.objecter ms_handle_connect 0x7f00bc135440 2024-02-19T21:14:05.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.508+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f00bc138f50 con 0x7f00bc135440 2024-02-19T21:14:05.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.508+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:14:05.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.508+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 15 = system:0 2024-02-19T21:14:05.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.508+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc1317a0 osd=7 4 2024-02-19T21:14:05.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.508+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 7 15 2024-02-19T21:14:05.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.522+0000 7f00c40ef700 10 client.49087.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:05.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.522+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign -1 16 2024-02-19T21:14:05.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.522+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc1317a0 osd=7 3 2024-02-19T21:14:05.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.522+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc1317a0 osd=7 3 2024-02-19T21:14:05.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.522+0000 7f00c40ef700 20 client.49087.objecter _get_session s=0x7f00bc1317a0 osd=7 3 2024-02-19T21:14:05.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.522+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_remove -1 16 2024-02-19T21:14:05.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.522+0000 7f00c40ef700 20 client.49087.objecter get_session s=0x7f00bc1317a0 osd=7 3 2024-02-19T21:14:05.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.522+0000 7f00c40ef700 15 client.49087.objecter _session_command_op_assign 7 16 2024-02-19T21:14:05.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.522+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc1317a0 osd=7 4 2024-02-19T21:14:05.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.522+0000 7f00c40ef700 10 client.49087.objecter _send_command 16 2024-02-19T21:14:05.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.522+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f00bc138f50 con 0x7f00bc135440 2024-02-19T21:14:05.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00b2ffd700 1 -- 172.21.15.163:0/2426119225 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f00bc138f50 con 0x7f00bc135440 2024-02-19T21:14:05.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00b2ffd700 10 client.49087.objecter ms_dispatch 0x7f00bc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:05.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00b2ffd700 10 client.49087.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00b2ffd700 20 client.49087.objecter put_session s=0x7f00bc1317a0 osd=7 4 2024-02-19T21:14:05.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00b2ffd700 15 client.49087.objecter _session_command_op_remove 7 16 2024-02-19T21:14:05.525 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:14:05.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 10 librados: watch_flush enter 2024-02-19T21:14:05.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 10 librados: watch_flush exit 2024-02-19T21:14:05.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 10 client.49087.objecter close_session for osd.0 2024-02-19T21:14:05.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 >> v1:172.21.15.163:6801/1559418115 conn(0x7f00bc118630 legacy=0x7f00bc118a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:05.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc0f9200 osd=0 1 2024-02-19T21:14:05.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 10 client.49087.objecter close_session for osd.1 2024-02-19T21:14:05.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 >> v1:172.21.15.163:6805/3125047432 conn(0x7f00bc0598f0 legacy=0x7f00bc0273c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:05.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc1194d0 osd=1 1 2024-02-19T21:14:05.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 10 client.49087.objecter close_session for osd.2 2024-02-19T21:14:05.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 >> v1:172.21.15.163:6809/1563158487 conn(0x7f00bc0594d0 legacy=0x7f00bc121b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:05.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc059360 osd=2 1 2024-02-19T21:14:05.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 10 client.49087.objecter close_session for osd.3 2024-02-19T21:14:05.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 >> v1:172.21.15.163:6813/2112279769 conn(0x7f00bc1053d0 legacy=0x7f00bc1263e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:05.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc0280c0 osd=3 1 2024-02-19T21:14:05.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 10 client.49087.objecter close_session for osd.4 2024-02-19T21:14:05.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 >> v1:172.21.15.193:6800/3129716965 conn(0x7f00bc123f30 legacy=0x7f00bc12ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:05.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c268c700 1 -- 172.21.15.163:0/2426119225 reap_dead start 2024-02-19T21:14:05.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c268c700 1 -- 172.21.15.163:0/2426119225 reap_dead start 2024-02-19T21:14:05.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc104930 osd=4 1 2024-02-19T21:14:05.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 10 client.49087.objecter close_session for osd.5 2024-02-19T21:14:05.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 >> v1:172.21.15.193:6804/805161991 conn(0x7f00bc12c8c0 legacy=0x7f00bc12ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:05.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc104bf0 osd=5 1 2024-02-19T21:14:05.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 10 client.49087.objecter close_session for osd.6 2024-02-19T21:14:05.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 >> v1:172.21.15.193:6808/787604615 conn(0x7f00bc132240 legacy=0x7f00bc1326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:05.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc130580 osd=6 1 2024-02-19T21:14:05.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 10 client.49087.objecter close_session for osd.7 2024-02-19T21:14:05.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 >> v1:172.21.15.193:6812/157927432 conn(0x7f00bc135440 legacy=0x7f00bc137830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:05.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 20 client.49087.objecter put_session s=0x7f00bc1317a0 osd=7 1 2024-02-19T21:14:05.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 20 client.49087.objecter shutdown clearing up homeless session... 2024-02-19T21:14:05.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.524+0000 7f00c40ef700 10 client.49087.objecter successfully canceled tick 2024-02-19T21:14:05.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.525+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 >> v1:172.21.15.163:6800/1552514681 conn(0x7f00a8063eb0 legacy=0x7f00a8066370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:05.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.525+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 >> v1:172.21.15.193:6789/0 conn(0x7f00bc1042c0 legacy=0x7f00bc115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:05.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.525+0000 7f00c268c700 1 -- 172.21.15.163:0/2426119225 reap_dead start 2024-02-19T21:14:05.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.525+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 shutdown_connections 2024-02-19T21:14:05.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.525+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 >> 172.21.15.163:0/2426119225 conn(0x7f00bc0f35a0 msgr2=0x7f00bc107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:05.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.525+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 shutdown_connections 2024-02-19T21:14:05.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.525+0000 7f00c40ef700 1 -- 172.21.15.163:0/2426119225 wait complete. 2024-02-19T21:14:05.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:05.525+0000 7f00c40ef700 1 librados: shutdown 2024-02-19T21:14:05.529 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.529 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.529 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.529 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.529 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.529 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.529 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.530 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:14:05.697 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:14:05.937 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:14:05.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:05.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:14:06.041 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:06.192 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:14:06.192 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:14:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:06.494 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:05.281097+0000 mgr.y (mgr.14550) 528 : cluster [DBG] pgmap v912: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 767 B/s rd, 0 op/s 2024-02-19T21:14:06.494 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:05.347942+0000 mon.a (mon.0) 3114 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-104"}]': finished 2024-02-19T21:14:06.494 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:05.348008+0000 mon.a (mon.0) 3115 : cluster [DBG] osdmap e609: 8 total, 8 up, 6 in 2024-02-19T21:14:06.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:05.349278+0000 mon.a (mon.0) 3116 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-104", "mode": "writeback"}]: dispatch 2024-02-19T21:14:06.495 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:05.386842+0000 mon.b (mon.1) 735 : audit [DBG] from='client.? v1:172.21.15.163:0/2426119225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:06.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:05.281097+0000 mgr.y (mgr.14550) 528 : cluster [DBG] pgmap v912: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 767 B/s rd, 0 op/s 2024-02-19T21:14:06.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:05.347942+0000 mon.a (mon.0) 3114 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-104"}]': finished 2024-02-19T21:14:06.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:05.348008+0000 mon.a (mon.0) 3115 : cluster [DBG] osdmap e609: 8 total, 8 up, 6 in 2024-02-19T21:14:06.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:05.349278+0000 mon.a (mon.0) 3116 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-104", "mode": "writeback"}]: dispatch 2024-02-19T21:14:06.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:05.386842+0000 mon.b (mon.1) 735 : audit [DBG] from='client.? v1:172.21.15.163:0/2426119225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:06.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:05.281097+0000 mgr.y (mgr.14550) 528 : cluster [DBG] pgmap v912: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 767 B/s rd, 0 op/s 2024-02-19T21:14:06.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:05.347942+0000 mon.a (mon.0) 3114 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-104"}]': finished 2024-02-19T21:14:06.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:05.348008+0000 mon.a (mon.0) 3115 : cluster [DBG] osdmap e609: 8 total, 8 up, 6 in 2024-02-19T21:14:06.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:05.349278+0000 mon.a (mon.0) 3116 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-104", "mode": "writeback"}]: dispatch 2024-02-19T21:14:06.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:05.386842+0000 mon.b (mon.1) 735 : audit [DBG] from='client.? v1:172.21.15.163:0/2426119225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:06.810 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-19T21:14:06.817 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:06.820 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:06.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:06.921 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:06.949 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:07.021 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:07.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:07.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:07.075 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:14:07.125 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:07.152 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:07.350 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:07.353 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:07.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:07.454 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:07.481 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:07.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:06.345149+0000 mon.a (mon.0) 3117 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:14:07.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:06.351826+0000 mon.a (mon.0) 3118 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-104", "mode": "writeback"}]': finished 2024-02-19T21:14:07.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:06.351887+0000 mon.a (mon.0) 3119 : cluster [DBG] osdmap e610: 8 total, 8 up, 6 in 2024-02-19T21:14:07.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:06.386731+0000 mon.a (mon.0) 3120 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:14:07.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:06.345149+0000 mon.a (mon.0) 3117 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:14:07.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:06.351826+0000 mon.a (mon.0) 3118 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-104", "mode": "writeback"}]': finished 2024-02-19T21:14:07.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:06.351887+0000 mon.a (mon.0) 3119 : cluster [DBG] osdmap e610: 8 total, 8 up, 6 in 2024-02-19T21:14:07.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:06.386731+0000 mon.a (mon.0) 3120 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:14:07.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:06.345149+0000 mon.a (mon.0) 3117 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:14:07.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:06.351826+0000 mon.a (mon.0) 3118 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-104", "mode": "writeback"}]': finished 2024-02-19T21:14:07.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:06.351887+0000 mon.a (mon.0) 3119 : cluster [DBG] osdmap e610: 8 total, 8 up, 6 in 2024-02-19T21:14:07.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:06.386731+0000 mon.a (mon.0) 3120 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:14:07.829 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:07.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:07.933 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:07.961 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:08.058 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:08.059 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:14:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:14:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:14:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:08.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:08.162 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:14:08.190 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:08.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.334+0000 7fb328e52700 1 -- 172.21.15.163:0/2624315151 >> v1:172.21.15.163:6789/0 conn(0x7fb3241042b0 legacy=0x7fb324106730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:08.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.334+0000 7fb328e52700 1 -- 172.21.15.163:0/2624315151 shutdown_connections 2024-02-19T21:14:08.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.334+0000 7fb328e52700 1 -- 172.21.15.163:0/2624315151 >> 172.21.15.163:0/2624315151 conn(0x7fb3240f3570 msgr2=0x7fb3240f5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:08.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.334+0000 7fb328e52700 1 -- 172.21.15.163:0/2624315151 shutdown_connections 2024-02-19T21:14:08.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.334+0000 7fb328e52700 1 -- 172.21.15.163:0/2624315151 wait complete. 2024-02-19T21:14:08.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb328e52700 1 librados: starting msgr at 2024-02-19T21:14:08.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb328e52700 1 librados: starting objecter 2024-02-19T21:14:08.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb328e52700 1 Processor -- start 2024-02-19T21:14:08.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb328e52700 1 -- start start 2024-02-19T21:14:08.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb328e52700 1 librados: setting wanted keys 2024-02-19T21:14:08.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb328e52700 1 librados: calling monclient init 2024-02-19T21:14:08.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb328e52700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb3241171d0 con 0x7fb3241042b0 2024-02-19T21:14:08.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb328e52700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb3241183b0 con 0x7fb324100770 2024-02-19T21:14:08.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb328e52700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb3241195b0 con 0x7fb3240fd970 2024-02-19T21:14:08.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb32259c700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fb3240fd970 0x7fb3240f9790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:49836/0 (socket says 172.21.15.163:49836) 2024-02-19T21:14:08.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb32259c700 1 -- 172.21.15.163:0/3653839569 learned_addr learned my addr 172.21.15.163:0/3653839569 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:08.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb322d9d700 1 --1- 172.21.15.163:0/3653839569 >> v1:172.21.15.163:6789/0 conn(0x7fb3241042b0 0x7fb324115930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46362/0 (socket says 172.21.15.163:46362) 2024-02-19T21:14:08.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb3137fe700 10 client.?.objecter ms_handle_connect 0x7fb3240fd970 2024-02-19T21:14:08.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb3137fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:08.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb3137fe700 10 client.?.objecter ms_handle_connect 0x7fb3241042b0 2024-02-19T21:14:08.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb3137fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:08.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb3137fe700 10 client.?.objecter ms_handle_connect 0x7fb324100770 2024-02-19T21:14:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.335+0000 7fb3137fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958537741 0 0) 0x7fb3241171d0 con 0x7fb3241042b0 2024-02-19T21:14:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb30c003800 con 0x7fb3241042b0 2024-02-19T21:14:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406556781 0 0) 0x7fb3241183b0 con 0x7fb324100770 2024-02-19T21:14:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3241171d0 con 0x7fb324100770 2024-02-19T21:14:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061712113 0 0) 0x7fb3241195b0 con 0x7fb3240fd970 2024-02-19T21:14:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3241183b0 con 0x7fb3240fd970 2024-02-19T21:14:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3919880290 0 0) 0x7fb30c003800 con 0x7fb3241042b0 2024-02-19T21:14:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3241195b0 con 0x7fb3241042b0 2024-02-19T21:14:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb31c0032e0 con 0x7fb3241042b0 2024-02-19T21:14:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2521079934 0 0) 0x7fb3241171d0 con 0x7fb324100770 2024-02-19T21:14:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb30c003800 con 0x7fb324100770 2024-02-19T21:14:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3293198123 0 0) 0x7fb3241183b0 con 0x7fb3240fd970 2024-02-19T21:14:08.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3241171d0 con 0x7fb3240fd970 2024-02-19T21:14:08.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb3140015e0 con 0x7fb324100770 2024-02-19T21:14:08.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb3180016c0 con 0x7fb3240fd970 2024-02-19T21:14:08.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2232721451 0 0) 0x7fb3241195b0 con 0x7fb3241042b0 2024-02-19T21:14:08.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 >> v1:172.21.15.163:6790/0 conn(0x7fb3240fd970 legacy=0x7fb3240f9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:08.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 >> v1:172.21.15.193:6789/0 conn(0x7fb324100770 legacy=0x7fb3240f9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:08.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32411a7b0 con 0x7fb3241042b0 2024-02-19T21:14:08.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb328e52700 1 -- 172.21.15.163:0/3653839569 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3241173c0 con 0x7fb3241042b0 2024-02-19T21:14:08.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb328e52700 10 client.23559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:08.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb328e52700 1 -- 172.21.15.163:0/3653839569 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3241178b0 con 0x7fb3241042b0 2024-02-19T21:14:08.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb328e52700 1 librados: init done 2024-02-19T21:14:08.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb31c001360 con 0x7fb3241042b0 2024-02-19T21:14:08.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.336+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb31c003b30 con 0x7fb3241042b0 2024-02-19T21:14:08.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.337+0000 7fb328e52700 1 -- 172.21.15.163:0/3653839569 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb304004f20 con 0x7fb3241042b0 2024-02-19T21:14:08.348 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.337+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fb31c003d30 con 0x7fb3241042b0 2024-02-19T21:14:08.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.338+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(611..611 src has 434..611) v4 ==== 8508+0+0 (unknown 3619256025 0 0) 0x7fb31c079460 con 0x7fb3241042b0 2024-02-19T21:14:08.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.338+0000 7fb3137fe700 10 client.23559.objecter ms_dispatch 0x7fb324001550 osd_map(611..611 src has 434..611) v4 2024-02-19T21:14:08.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.338+0000 7fb3137fe700 3 client.23559.objecter handle_osd_map got epochs [611,611] > 0 2024-02-19T21:14:08.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.338+0000 7fb3137fe700 3 client.23559.objecter handle_osd_map decoding full epoch 611 2024-02-19T21:14:08.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.338+0000 7fb3137fe700 20 client.23559.objecter dump_active .. 0 homeless 2024-02-19T21:14:08.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.338+0000 7fb3137fe700 10 client.23559.objecter ms_handle_connect 0x7fb30c063d00 2024-02-19T21:14:08.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.342+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb31c04c690 con 0x7fb3241042b0 2024-02-19T21:14:08.389 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:14:08.392 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:08.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:14:08.470 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:14:08.493 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:14:08.520 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:08.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:08.526+0000 7fb328e52700 1 -- 172.21.15.163:0/3653839569 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fb3040051d0 con 0x7fb3241042b0 2024-02-19T21:14:08.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:07.281485+0000 mgr.y (mgr.14550) 529 : cluster [DBG] pgmap v915: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:14:08.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:07.364514+0000 mon.a (mon.0) 3121 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:08.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:07.364567+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e611: 8 total, 8 up, 6 in 2024-02-19T21:14:08.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:07.367492+0000 mon.a (mon.0) 3123 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-104"}]: dispatch 2024-02-19T21:14:08.644 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:14:08.646 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:08.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:14:08.737 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:08.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:07.281485+0000 mgr.y (mgr.14550) 529 : cluster [DBG] pgmap v915: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:14:08.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:07.364514+0000 mon.a (mon.0) 3121 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:08.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:07.364567+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e611: 8 total, 8 up, 6 in 2024-02-19T21:14:08.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:07.367492+0000 mon.a (mon.0) 3123 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-104"}]: dispatch 2024-02-19T21:14:08.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:07.281485+0000 mgr.y (mgr.14550) 529 : cluster [DBG] pgmap v915: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:14:08.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:07.364514+0000 mon.a (mon.0) 3121 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:08.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:07.364567+0000 mon.a (mon.0) 3122 : cluster [DBG] osdmap e611: 8 total, 8 up, 6 in 2024-02-19T21:14:08.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:07.367492+0000 mon.a (mon.0) 3123 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-104"}]: dispatch 2024-02-19T21:14:08.748 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:14:08.777 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:08.875 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:14:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:08.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:14:08.887 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:14:08.887 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:14:08.979 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:14:09.009 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:09.029 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:14:09.033 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:09.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:14:09.134 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:09.162 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:09.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:09.433+0000 7fb3137fe700 1 -- 172.21.15.163:0/3653839569 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v613) v1 ==== 91+0+0 (unknown 2030975284 0 0) 0x7fb31c04ab00 con 0x7fb3241042b0 2024-02-19T21:14:09.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:09.437+0000 7fb328e52700 10 librados: watch_flush enter 2024-02-19T21:14:09.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:09.437+0000 7fb328e52700 10 librados: watch_flush exit 2024-02-19T21:14:09.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:09.437+0000 7fb328e52700 20 client.23559.objecter shutdown clearing up homeless session... 2024-02-19T21:14:09.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:09.437+0000 7fb328e52700 10 client.23559.objecter successfully canceled tick 2024-02-19T21:14:09.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:09.437+0000 7fb328e52700 1 -- 172.21.15.163:0/3653839569 >> v1:172.21.15.163:6800/1552514681 conn(0x7fb30c063d00 legacy=0x7fb30c0661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:09.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:09.437+0000 7fb328e52700 1 -- 172.21.15.163:0/3653839569 >> v1:172.21.15.163:6789/0 conn(0x7fb3241042b0 legacy=0x7fb324115930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:09.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:09.438+0000 7fb328e52700 1 -- 172.21.15.163:0/3653839569 shutdown_connections 2024-02-19T21:14:09.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:09.438+0000 7fb328e52700 1 -- 172.21.15.163:0/3653839569 >> 172.21.15.163:0/3653839569 conn(0x7fb3240f3570 msgr2=0x7fb3240f6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:09.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:09.438+0000 7fb328e52700 1 -- 172.21.15.163:0/3653839569 shutdown_connections 2024-02-19T21:14:09.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:09.438+0000 7fb328e52700 1 -- 172.21.15.163:0/3653839569 wait complete. 2024-02-19T21:14:09.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:09.438+0000 7fb328e52700 1 librados: shutdown 2024-02-19T21:14:09.439 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:14:09.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:08.361576+0000 mon.a (mon.0) 3124 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:14:09.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:08.371075+0000 mon.a (mon.0) 3125 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-104"}]': finished 2024-02-19T21:14:09.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:08.371151+0000 mon.a (mon.0) 3126 : cluster [DBG] osdmap e612: 8 total, 8 up, 6 in 2024-02-19T21:14:09.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:08.528149+0000 mon.a (mon.0) 3127 : audit [INF] from='client.? v1:172.21.15.163:0/3653839569' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:14:09.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:09.107114+0000 mon.a (mon.0) 3128 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:14:09.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:09.343311+0000 mon.a (mon.0) 3129 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:14:09.632 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-19T21:14:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:08.361576+0000 mon.a (mon.0) 3124 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:14:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:08.371075+0000 mon.a (mon.0) 3125 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-104"}]': finished 2024-02-19T21:14:09.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:08.371151+0000 mon.a (mon.0) 3126 : cluster [DBG] osdmap e612: 8 total, 8 up, 6 in 2024-02-19T21:14:09.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:08.528149+0000 mon.a (mon.0) 3127 : audit [INF] from='client.? v1:172.21.15.163:0/3653839569' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:14:09.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:09.107114+0000 mon.a (mon.0) 3128 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:14:09.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:09.343311+0000 mon.a (mon.0) 3129 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:14:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:08.361576+0000 mon.a (mon.0) 3124 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:14:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:08.371075+0000 mon.a (mon.0) 3125 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-104"}]': finished 2024-02-19T21:14:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:08.371151+0000 mon.a (mon.0) 3126 : cluster [DBG] osdmap e612: 8 total, 8 up, 6 in 2024-02-19T21:14:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:08.528149+0000 mon.a (mon.0) 3127 : audit [INF] from='client.? v1:172.21.15.163:0/3653839569' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:14:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:09.107114+0000 mon.a (mon.0) 3128 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:14:09.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:09.343311+0000 mon.a (mon.0) 3129 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:14:09.941 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:14:09.989 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 7, 3, 4] out_osds: [0, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:14:09.989 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:14:09.989 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-02-19T21:14:09.989 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd in 6 2024-02-19T21:14:10.212 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:10.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:10.316 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:10.343 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:10.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:09.281806+0000 mgr.y (mgr.14550) 530 : cluster [DBG] pgmap v918: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:14:10.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:09.367506+0000 mon.a (mon.0) 3130 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:14:10.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:09.427886+0000 mon.a (mon.0) 3131 : audit [INF] from='client.? v1:172.21.15.163:0/3653839569' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:14:10.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:09.427943+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e613: 8 total, 8 up, 6 in 2024-02-19T21:14:10.636 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:09.281806+0000 mgr.y (mgr.14550) 530 : cluster [DBG] pgmap v918: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:14:10.636 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:09.367506+0000 mon.a (mon.0) 3130 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:14:10.636 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:09.427886+0000 mon.a (mon.0) 3131 : audit [INF] from='client.? v1:172.21.15.163:0/3653839569' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:14:10.636 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:09.427943+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e613: 8 total, 8 up, 6 in 2024-02-19T21:14:10.637 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:09.281806+0000 mgr.y (mgr.14550) 530 : cluster [DBG] pgmap v918: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:14:10.637 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:09.367506+0000 mon.a (mon.0) 3130 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:14:10.637 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:09.427886+0000 mon.a (mon.0) 3131 : audit [INF] from='client.? v1:172.21.15.163:0/3653839569' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:14:10.637 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:09.427943+0000 mon.a (mon.0) 3132 : cluster [DBG] osdmap e613: 8 total, 8 up, 6 in 2024-02-19T21:14:10.668 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:10.691 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:10.694 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:10.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:10.795 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:10.823 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:11.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:14:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:14:11.672 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:11.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:11.676 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:14:11.676 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:14:11.683 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:10.425033+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e614: 8 total, 8 up, 6 in 2024-02-19T21:14:11.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:10.433093+0000 mon.a (mon.0) 3134 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-106","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:14:11.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:10.530607+0000 osd.4 (osd.4) 219 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:14:11.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:10.531463+0000 osd.4 (osd.4) 220 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:14:11.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:10.613911+0000 mgr.y (mgr.14550) 531 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:11.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:10.425033+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e614: 8 total, 8 up, 6 in 2024-02-19T21:14:11.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:10.433093+0000 mon.a (mon.0) 3134 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-106","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:14:11.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:10.530607+0000 osd.4 (osd.4) 219 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:14:11.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:10.531463+0000 osd.4 (osd.4) 220 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:14:11.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:10.613911+0000 mgr.y (mgr.14550) 531 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:11.697 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:10.425033+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e614: 8 total, 8 up, 6 in 2024-02-19T21:14:11.697 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:10.433093+0000 mon.a (mon.0) 3134 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-106","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:14:11.697 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:10.530607+0000 osd.4 (osd.4) 219 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:14:11.698 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:10.531463+0000 osd.4 (osd.4) 220 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:14:11.698 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:10.613911+0000 mgr.y (mgr.14550) 531 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:11.777 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:14:11.807 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:11.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.913+0000 7fe41afb9700 1 -- 172.21.15.163:0/893588297 >> v1:172.21.15.163:6790/0 conn(0x7fe4140fc870 legacy=0x7fe4140fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:11.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.913+0000 7fe41afb9700 1 -- 172.21.15.163:0/893588297 shutdown_connections 2024-02-19T21:14:11.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.913+0000 7fe41afb9700 1 -- 172.21.15.163:0/893588297 >> 172.21.15.163:0/893588297 conn(0x7fe4140f2470 msgr2=0x7fe4140f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:11.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.913+0000 7fe41afb9700 1 -- 172.21.15.163:0/893588297 shutdown_connections 2024-02-19T21:14:11.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.913+0000 7fe41afb9700 1 -- 172.21.15.163:0/893588297 wait complete. 2024-02-19T21:14:11.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.914+0000 7fe41afb9700 1 librados: starting msgr at 2024-02-19T21:14:11.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.914+0000 7fe41afb9700 1 librados: starting objecter 2024-02-19T21:14:11.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.914+0000 7fe41afb9700 1 Processor -- start 2024-02-19T21:14:11.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.914+0000 7fe41afb9700 1 -- start start 2024-02-19T21:14:11.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.914+0000 7fe41afb9700 1 librados: setting wanted keys 2024-02-19T21:14:11.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.914+0000 7fe41afb9700 1 librados: calling monclient init 2024-02-19T21:14:11.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.914+0000 7fe41afb9700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe414119350 con 0x7fe4140fc870 2024-02-19T21:14:11.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.914+0000 7fe41afb9700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe41411a530 con 0x7fe4140ff670 2024-02-19T21:14:11.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.914+0000 7fe41afb9700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe41411b730 con 0x7fe4141031b0 2024-02-19T21:14:11.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.914+0000 7fe418d55700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fe4140fc870 0x7fe4140fb580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46388/0 (socket says 172.21.15.163:46388) 2024-02-19T21:14:11.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.914+0000 7fe418d55700 1 -- 172.21.15.163:0/2666158633 learned_addr learned my addr 172.21.15.163:0/2666158633 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:11.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.914+0000 7fe411ffb700 10 client.?.objecter ms_handle_connect 0x7fe4140fc870 2024-02-19T21:14:11.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:11.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 10 client.?.objecter ms_handle_connect 0x7fe4141031b0 2024-02-19T21:14:11.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:11.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 10 client.?.objecter ms_handle_connect 0x7fe4140ff670 2024-02-19T21:14:11.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:11.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592670740 0 0) 0x7fe414119350 con 0x7fe4140fc870 2024-02-19T21:14:11.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe404003030 con 0x7fe4140fc870 2024-02-19T21:14:11.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249330318 0 0) 0x7fe41411a530 con 0x7fe4140ff670 2024-02-19T21:14:11.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe414119350 con 0x7fe4140ff670 2024-02-19T21:14:11.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958958545 0 0) 0x7fe41411b730 con 0x7fe4141031b0 2024-02-19T21:14:11.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe41411a530 con 0x7fe4141031b0 2024-02-19T21:14:11.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3460508451 0 0) 0x7fe404003030 con 0x7fe4140fc870 2024-02-19T21:14:11.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe41411b730 con 0x7fe4140fc870 2024-02-19T21:14:11.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe408002070 con 0x7fe4140fc870 2024-02-19T21:14:11.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2512963814 0 0) 0x7fe414119350 con 0x7fe4140ff670 2024-02-19T21:14:11.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe404003030 con 0x7fe4140ff670 2024-02-19T21:14:11.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1778500384 0 0) 0x7fe41411a530 con 0x7fe4141031b0 2024-02-19T21:14:11.919 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe414119350 con 0x7fe4141031b0 2024-02-19T21:14:11.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe400001620 con 0x7fe4140ff670 2024-02-19T21:14:11.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe40c001b40 con 0x7fe4141031b0 2024-02-19T21:14:11.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 58698941 0 0) 0x7fe41411b730 con 0x7fe4140fc870 2024-02-19T21:14:11.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 >> v1:172.21.15.163:6790/0 conn(0x7fe4141031b0 legacy=0x7fe414117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:11.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 >> v1:172.21.15.193:6789/0 conn(0x7fe4140ff670 legacy=0x7fe4140fbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:11.920 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41411c930 con 0x7fe4140fc870 2024-02-19T21:14:11.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41411b920 con 0x7fe4140fc870 2024-02-19T21:14:11.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fe408002e30 con 0x7fe4140fc870 2024-02-19T21:14:11.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe408003ac0 con 0x7fe4140fc870 2024-02-19T21:14:11.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe41afb9700 10 client.23565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:11.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41411be10 con 0x7fe4140fc870 2024-02-19T21:14:11.921 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.915+0000 7fe41afb9700 1 librados: init done 2024-02-19T21:14:11.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.916+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fe41405f700 con 0x7fe4140fc870 2024-02-19T21:14:11.922 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fe408003c20 con 0x7fe4140fc870 2024-02-19T21:14:11.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(615..615 src has 434..615) v4 ==== 8513+0+0 (unknown 1379874286 0 0) 0x7fe408079340 con 0x7fe4140fc870 2024-02-19T21:14:11.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 osd_map(615..615 src has 434..615) v4 2024-02-19T21:14:11.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe411ffb700 3 client.23565.objecter handle_osd_map got epochs [615,615] > 0 2024-02-19T21:14:11.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe411ffb700 3 client.23565.objecter handle_osd_map decoding full epoch 615 2024-02-19T21:14:11.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe411ffb700 20 client.23565.objecter dump_active .. 0 homeless 2024-02-19T21:14:11.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v615) v1 ==== 54+0+16 (unknown 845491671 0 2420000192) 0x7fe40807b730 con 0x7fe4140fc870 2024-02-19T21:14:11.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe411ffb700 10 client.23565.objecter ms_handle_connect 0x7fe404063d70 2024-02-19T21:14:11.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe41afb9700 10 client.23565.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:11.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 1 2024-02-19T21:14:11.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414047540 osd=0 3 2024-02-19T21:14:11.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414047540 osd=0 3 2024-02-19T21:14:11.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414047540 osd=0 3 2024-02-19T21:14:11.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 1 2024-02-19T21:14:11.923 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe414047540 osd=0 3 2024-02-19T21:14:11.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 0 1 2024-02-19T21:14:11.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414047540 osd=0 4 2024-02-19T21:14:11.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe41afb9700 10 client.23565.objecter _send_command 1 2024-02-19T21:14:11.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.917+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe414108e10 con 0x7fe41411ae90 2024-02-19T21:14:11.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.918+0000 7fe411ffb700 10 client.23565.objecter ms_handle_connect 0x7fe41411ae90 2024-02-19T21:14:11.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.918+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe414108e10 con 0x7fe41411ae90 2024-02-19T21:14:11.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.918+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:14:11.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.918+0000 7fe411ffb700 10 client.23565.objecter _finish_command 1 = system:0 2024-02-19T21:14:11.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.918+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe414047540 osd=0 4 2024-02-19T21:14:11.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.918+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 0 1 2024-02-19T21:14:11.930 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:14:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:11.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:14:11.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.933+0000 7fe41afb9700 10 client.23565.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:11.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.933+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 2 2024-02-19T21:14:11.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.933+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414047540 osd=0 3 2024-02-19T21:14:11.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.933+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414047540 osd=0 3 2024-02-19T21:14:11.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.933+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414047540 osd=0 3 2024-02-19T21:14:11.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.933+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 2 2024-02-19T21:14:11.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.933+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe414047540 osd=0 3 2024-02-19T21:14:11.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.933+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 0 2 2024-02-19T21:14:11.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.933+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414047540 osd=0 4 2024-02-19T21:14:11.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.933+0000 7fe41afb9700 10 client.23565.objecter _send_command 2 2024-02-19T21:14:11.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.933+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe41411b4b0 con 0x7fe41411ae90 2024-02-19T21:14:11.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe41411b4b0 con 0x7fe41411ae90 2024-02-19T21:14:11.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:11.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe411ffb700 10 client.23565.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:11.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe414047540 osd=0 4 2024-02-19T21:14:11.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 0 2 2024-02-19T21:14:11.935 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:14:11.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe41afb9700 10 client.23565.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:11.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 3 2024-02-19T21:14:11.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414028610 osd=1 3 2024-02-19T21:14:11.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414028610 osd=1 3 2024-02-19T21:14:11.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414028610 osd=1 3 2024-02-19T21:14:11.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 3 2024-02-19T21:14:11.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe414028610 osd=1 3 2024-02-19T21:14:11.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 1 3 2024-02-19T21:14:11.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414028610 osd=1 4 2024-02-19T21:14:11.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe41afb9700 10 client.23565.objecter _send_command 3 2024-02-19T21:14:11.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe414028780 con 0x7fe41411a100 2024-02-19T21:14:11.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.935+0000 7fe411ffb700 10 client.23565.objecter ms_handle_connect 0x7fe41411a100 2024-02-19T21:14:11.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.936+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe414028780 con 0x7fe41411a100 2024-02-19T21:14:11.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.936+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:14:11.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.936+0000 7fe411ffb700 10 client.23565.objecter _finish_command 3 = system:0 2024-02-19T21:14:11.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.936+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe414028610 osd=1 4 2024-02-19T21:14:11.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.936+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 1 3 2024-02-19T21:14:11.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.950+0000 7fe41afb9700 10 client.23565.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:11.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.950+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 4 2024-02-19T21:14:11.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.950+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414028610 osd=1 3 2024-02-19T21:14:11.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.950+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414028610 osd=1 3 2024-02-19T21:14:11.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.950+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414028610 osd=1 3 2024-02-19T21:14:11.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.950+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 4 2024-02-19T21:14:11.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.950+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe414028610 osd=1 3 2024-02-19T21:14:11.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.950+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 1 4 2024-02-19T21:14:11.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.950+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414028610 osd=1 4 2024-02-19T21:14:11.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.950+0000 7fe41afb9700 10 client.23565.objecter _send_command 4 2024-02-19T21:14:11.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.950+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe4140f3f70 con 0x7fe41411a100 2024-02-19T21:14:11.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe4140f3f70 con 0x7fe41411a100 2024-02-19T21:14:11.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:11.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe411ffb700 10 client.23565.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:11.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe414028610 osd=1 4 2024-02-19T21:14:11.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 1 4 2024-02-19T21:14:11.953 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:14:11.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe41afb9700 10 client.23565.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:11.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 5 2024-02-19T21:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe41411ab40 osd=2 3 2024-02-19T21:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe41411ab40 osd=2 3 2024-02-19T21:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe41411ab40 osd=2 3 2024-02-19T21:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 5 2024-02-19T21:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe41411ab40 osd=2 3 2024-02-19T21:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 2 5 2024-02-19T21:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe41411ab40 osd=2 4 2024-02-19T21:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe41afb9700 10 client.23565.objecter _send_command 5 2024-02-19T21:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe41411acb0 con 0x7fe414059e00 2024-02-19T21:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.952+0000 7fe411ffb700 10 client.23565.objecter ms_handle_connect 0x7fe414059e00 2024-02-19T21:14:11.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.953+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe41411acb0 con 0x7fe414059e00 2024-02-19T21:14:11.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.953+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:14:11.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.953+0000 7fe411ffb700 10 client.23565.objecter _finish_command 5 = system:0 2024-02-19T21:14:11.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.953+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe41411ab40 osd=2 4 2024-02-19T21:14:11.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.953+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 2 5 2024-02-19T21:14:11.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.967+0000 7fe41afb9700 10 client.23565.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:11.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.967+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 6 2024-02-19T21:14:11.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.967+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe41411ab40 osd=2 3 2024-02-19T21:14:11.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.967+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe41411ab40 osd=2 3 2024-02-19T21:14:11.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.967+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe41411ab40 osd=2 3 2024-02-19T21:14:11.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.967+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 6 2024-02-19T21:14:11.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.967+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe41411ab40 osd=2 3 2024-02-19T21:14:11.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.967+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 2 6 2024-02-19T21:14:11.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.967+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe41411ab40 osd=2 4 2024-02-19T21:14:11.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.967+0000 7fe41afb9700 10 client.23565.objecter _send_command 6 2024-02-19T21:14:11.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.967+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe414059360 con 0x7fe414059e00 2024-02-19T21:14:11.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe414059360 con 0x7fe414059e00 2024-02-19T21:14:11.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:11.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe411ffb700 10 client.23565.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:11.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe41411ab40 osd=2 4 2024-02-19T21:14:11.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 2 6 2024-02-19T21:14:11.970 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:14:11.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe41afb9700 10 client.23565.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:11.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 7 2024-02-19T21:14:11.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414059770 osd=3 3 2024-02-19T21:14:11.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414059770 osd=3 3 2024-02-19T21:14:11.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414059770 osd=3 3 2024-02-19T21:14:11.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 7 2024-02-19T21:14:11.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe414059770 osd=3 3 2024-02-19T21:14:11.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 3 7 2024-02-19T21:14:11.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414059770 osd=3 4 2024-02-19T21:14:11.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe41afb9700 10 client.23565.objecter _send_command 7 2024-02-19T21:14:11.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe414129d00 con 0x7fe414121880 2024-02-19T21:14:11.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.969+0000 7fe411ffb700 10 client.23565.objecter ms_handle_connect 0x7fe414121880 2024-02-19T21:14:11.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.970+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe414129d00 con 0x7fe414121880 2024-02-19T21:14:11.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.970+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:14:11.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.970+0000 7fe411ffb700 10 client.23565.objecter _finish_command 7 = system:0 2024-02-19T21:14:11.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.970+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe414059770 osd=3 4 2024-02-19T21:14:11.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.970+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 3 7 2024-02-19T21:14:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.984+0000 7fe41afb9700 10 client.23565.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.984+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 8 2024-02-19T21:14:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.984+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414059770 osd=3 3 2024-02-19T21:14:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.984+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414059770 osd=3 3 2024-02-19T21:14:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.984+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414059770 osd=3 3 2024-02-19T21:14:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.984+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 8 2024-02-19T21:14:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.984+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe414059770 osd=3 3 2024-02-19T21:14:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.984+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 3 8 2024-02-19T21:14:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.984+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414059770 osd=3 4 2024-02-19T21:14:11.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.984+0000 7fe41afb9700 10 client.23565.objecter _send_command 8 2024-02-19T21:14:11.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.984+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe414129d00 con 0x7fe414121880 2024-02-19T21:14:11.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe414129d00 con 0x7fe414121880 2024-02-19T21:14:11.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:11.986 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:14:11.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe411ffb700 10 client.23565.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:11.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe414059770 osd=3 4 2024-02-19T21:14:11.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 3 8 2024-02-19T21:14:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe41afb9700 10 client.23565.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 9 2024-02-19T21:14:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414129d00 osd=4 3 2024-02-19T21:14:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414129d00 osd=4 3 2024-02-19T21:14:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414129d00 osd=4 3 2024-02-19T21:14:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 9 2024-02-19T21:14:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe414129d00 osd=4 3 2024-02-19T21:14:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 4 9 2024-02-19T21:14:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414129d00 osd=4 4 2024-02-19T21:14:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe41afb9700 10 client.23565.objecter _send_command 9 2024-02-19T21:14:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.985+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe4141261a0 con 0x7fe414125c20 2024-02-19T21:14:11.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.986+0000 7fe411ffb700 10 client.23565.objecter ms_handle_connect 0x7fe414125c20 2024-02-19T21:14:11.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.987+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe4141261a0 con 0x7fe414125c20 2024-02-19T21:14:11.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.987+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:14:11.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.987+0000 7fe411ffb700 10 client.23565.objecter _finish_command 9 = system:0 2024-02-19T21:14:11.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.987+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe414129d00 osd=4 4 2024-02-19T21:14:11.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:11.987+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 4 9 2024-02-19T21:14:12.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.001+0000 7fe41afb9700 10 client.23565.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:12.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.001+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 10 2024-02-19T21:14:12.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.001+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414129d00 osd=4 3 2024-02-19T21:14:12.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.001+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414129d00 osd=4 3 2024-02-19T21:14:12.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.001+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414129d00 osd=4 3 2024-02-19T21:14:12.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.001+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 10 2024-02-19T21:14:12.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.001+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe414129d00 osd=4 3 2024-02-19T21:14:12.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.001+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 4 10 2024-02-19T21:14:12.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.001+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414129d00 osd=4 4 2024-02-19T21:14:12.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.001+0000 7fe41afb9700 10 client.23565.objecter _send_command 10 2024-02-19T21:14:12.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.001+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe4141261a0 con 0x7fe414125c20 2024-02-19T21:14:12.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe4141261a0 con 0x7fe414125c20 2024-02-19T21:14:12.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:12.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe411ffb700 10 client.23565.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:12.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe414129d00 osd=4 4 2024-02-19T21:14:12.003 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 4 10 2024-02-19T21:14:12.004 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:14:12.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe41afb9700 10 client.23565.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:12.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 11 2024-02-19T21:14:12.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe4141261a0 osd=5 3 2024-02-19T21:14:12.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe4141261a0 osd=5 3 2024-02-19T21:14:12.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe4141261a0 osd=5 3 2024-02-19T21:14:12.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 11 2024-02-19T21:14:12.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe4141261a0 osd=5 3 2024-02-19T21:14:12.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 5 11 2024-02-19T21:14:12.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe4141261a0 osd=5 4 2024-02-19T21:14:12.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe41afb9700 10 client.23565.objecter _send_command 11 2024-02-19T21:14:12.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.003+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe41412c0d0 con 0x7fe41412bc40 2024-02-19T21:14:12.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.004+0000 7fe411ffb700 10 client.23565.objecter ms_handle_connect 0x7fe41412bc40 2024-02-19T21:14:12.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.004+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe41412c0d0 con 0x7fe41412bc40 2024-02-19T21:14:12.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.004+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:14:12.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.004+0000 7fe411ffb700 10 client.23565.objecter _finish_command 11 = system:0 2024-02-19T21:14:12.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.004+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe4141261a0 osd=5 4 2024-02-19T21:14:12.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.004+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 5 11 2024-02-19T21:14:12.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.019+0000 7fe41afb9700 10 client.23565.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:12.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.019+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 12 2024-02-19T21:14:12.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.019+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe4141261a0 osd=5 3 2024-02-19T21:14:12.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.019+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe4141261a0 osd=5 3 2024-02-19T21:14:12.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.019+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe4141261a0 osd=5 3 2024-02-19T21:14:12.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.019+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 12 2024-02-19T21:14:12.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.019+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe4141261a0 osd=5 3 2024-02-19T21:14:12.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.019+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 5 12 2024-02-19T21:14:12.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.019+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe4141261a0 osd=5 4 2024-02-19T21:14:12.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.019+0000 7fe41afb9700 10 client.23565.objecter _send_command 12 2024-02-19T21:14:12.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.019+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe41412c0d0 con 0x7fe41412bc40 2024-02-19T21:14:12.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe41412c0d0 con 0x7fe41412bc40 2024-02-19T21:14:12.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:12.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe411ffb700 10 client.23565.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:12.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe4141261a0 osd=5 4 2024-02-19T21:14:12.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 5 12 2024-02-19T21:14:12.022 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:14:12.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe41afb9700 10 client.23565.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:12.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 13 2024-02-19T21:14:12.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe41412c0d0 osd=6 3 2024-02-19T21:14:12.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe41412c0d0 osd=6 3 2024-02-19T21:14:12.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe41412c0d0 osd=6 3 2024-02-19T21:14:12.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 13 2024-02-19T21:14:12.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe41412c0d0 osd=6 3 2024-02-19T21:14:12.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 6 13 2024-02-19T21:14:12.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe41412c0d0 osd=6 4 2024-02-19T21:14:12.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe41afb9700 10 client.23565.objecter _send_command 13 2024-02-19T21:14:12.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.021+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe414136df0 con 0x7fe4141332e0 2024-02-19T21:14:12.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.022+0000 7fe411ffb700 10 client.23565.objecter ms_handle_connect 0x7fe4141332e0 2024-02-19T21:14:12.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.022+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe414136df0 con 0x7fe4141332e0 2024-02-19T21:14:12.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.022+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:14:12.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.022+0000 7fe411ffb700 10 client.23565.objecter _finish_command 13 = system:0 2024-02-19T21:14:12.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.022+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe41412c0d0 osd=6 4 2024-02-19T21:14:12.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.022+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 6 13 2024-02-19T21:14:12.034 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:12.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.036+0000 7fe41afb9700 10 client.23565.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:12.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.036+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 14 2024-02-19T21:14:12.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.036+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe41412c0d0 osd=6 3 2024-02-19T21:14:12.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.036+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe41412c0d0 osd=6 3 2024-02-19T21:14:12.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.036+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe41412c0d0 osd=6 3 2024-02-19T21:14:12.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.036+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 14 2024-02-19T21:14:12.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.036+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe41412c0d0 osd=6 3 2024-02-19T21:14:12.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.036+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 6 14 2024-02-19T21:14:12.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.036+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe41412c0d0 osd=6 4 2024-02-19T21:14:12.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.036+0000 7fe41afb9700 10 client.23565.objecter _send_command 14 2024-02-19T21:14:12.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.036+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe414136df0 con 0x7fe4141332e0 2024-02-19T21:14:12.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe414136df0 con 0x7fe4141332e0 2024-02-19T21:14:12.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:12.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe411ffb700 10 client.23565.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:12.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe41412c0d0 osd=6 4 2024-02-19T21:14:12.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 6 14 2024-02-19T21:14:12.039 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:14:12.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe41afb9700 10 client.23565.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:12.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 15 2024-02-19T21:14:12.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414136df0 osd=7 3 2024-02-19T21:14:12.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414136df0 osd=7 3 2024-02-19T21:14:12.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414136df0 osd=7 3 2024-02-19T21:14:12.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 15 2024-02-19T21:14:12.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe414136df0 osd=7 3 2024-02-19T21:14:12.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 7 15 2024-02-19T21:14:12.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414136df0 osd=7 4 2024-02-19T21:14:12.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe41afb9700 10 client.23565.objecter _send_command 15 2024-02-19T21:14:12.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.038+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe41413b0c0 con 0x7fe4141374a0 2024-02-19T21:14:12.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.039+0000 7fe411ffb700 10 client.23565.objecter ms_handle_connect 0x7fe4141374a0 2024-02-19T21:14:12.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.039+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fe41413b0c0 con 0x7fe4141374a0 2024-02-19T21:14:12.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.039+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:14:12.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.039+0000 7fe411ffb700 10 client.23565.objecter _finish_command 15 = system:0 2024-02-19T21:14:12.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.039+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe414136df0 osd=7 4 2024-02-19T21:14:12.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.039+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 7 15 2024-02-19T21:14:12.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.059+0000 7fe41afb9700 10 client.23565.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:12.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.059+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign -1 16 2024-02-19T21:14:12.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.059+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414136df0 osd=7 3 2024-02-19T21:14:12.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.059+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414136df0 osd=7 3 2024-02-19T21:14:12.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.059+0000 7fe41afb9700 20 client.23565.objecter _get_session s=0x7fe414136df0 osd=7 3 2024-02-19T21:14:12.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.059+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_remove -1 16 2024-02-19T21:14:12.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.059+0000 7fe41afb9700 20 client.23565.objecter get_session s=0x7fe414136df0 osd=7 3 2024-02-19T21:14:12.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.059+0000 7fe41afb9700 15 client.23565.objecter _session_command_op_assign 7 16 2024-02-19T21:14:12.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.059+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414136df0 osd=7 4 2024-02-19T21:14:12.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.059+0000 7fe41afb9700 10 client.23565.objecter _send_command 16 2024-02-19T21:14:12.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.059+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fe41413c870 con 0x7fe4141374a0 2024-02-19T21:14:12.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.060+0000 7fe411ffb700 1 -- 172.21.15.163:0/2666158633 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fe41413c870 con 0x7fe4141374a0 2024-02-19T21:14:12.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.060+0000 7fe411ffb700 10 client.23565.objecter ms_dispatch 0x7fe414001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:12.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.060+0000 7fe411ffb700 10 client.23565.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:12.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.060+0000 7fe411ffb700 20 client.23565.objecter put_session s=0x7fe414136df0 osd=7 4 2024-02-19T21:14:12.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.060+0000 7fe411ffb700 15 client.23565.objecter _session_command_op_remove 7 16 2024-02-19T21:14:12.061 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:14:12.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 10 librados: watch_flush enter 2024-02-19T21:14:12.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 10 librados: watch_flush exit 2024-02-19T21:14:12.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 10 client.23565.objecter close_session for osd.0 2024-02-19T21:14:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 >> v1:172.21.15.163:6801/1559418115 conn(0x7fe41411ae90 legacy=0x7fe4141194e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414047540 osd=0 1 2024-02-19T21:14:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 10 client.23565.objecter close_session for osd.1 2024-02-19T21:14:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 >> v1:172.21.15.163:6805/3125047432 conn(0x7fe41411a100 legacy=0x7fe4140f2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414028610 osd=1 1 2024-02-19T21:14:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 10 client.23565.objecter close_session for osd.2 2024-02-19T21:14:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 >> v1:172.21.15.163:6809/1563158487 conn(0x7fe414059e00 legacy=0x7fe414123d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe41411ab40 osd=2 1 2024-02-19T21:14:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 10 client.23565.objecter close_session for osd.3 2024-02-19T21:14:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 >> v1:172.21.15.163:6813/2112279769 conn(0x7fe414121880 legacy=0x7fe4141285e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe419556700 1 -- 172.21.15.163:0/2666158633 reap_dead start 2024-02-19T21:14:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414059770 osd=3 1 2024-02-19T21:14:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 10 client.23565.objecter close_session for osd.4 2024-02-19T21:14:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 >> v1:172.21.15.193:6800/3129716965 conn(0x7fe414125c20 legacy=0x7fe414129ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414129d00 osd=4 1 2024-02-19T21:14:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 10 client.23565.objecter close_session for osd.5 2024-02-19T21:14:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe419556700 1 -- 172.21.15.163:0/2666158633 reap_dead start 2024-02-19T21:14:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 >> v1:172.21.15.193:6804/805161991 conn(0x7fe41412bc40 legacy=0x7fe4141316b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe4141261a0 osd=5 1 2024-02-19T21:14:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 10 client.23565.objecter close_session for osd.6 2024-02-19T21:14:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 >> v1:172.21.15.193:6808/787604615 conn(0x7fe4141332e0 legacy=0x7fe4141356d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe41412c0d0 osd=6 1 2024-02-19T21:14:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 10 client.23565.objecter close_session for osd.7 2024-02-19T21:14:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 >> v1:172.21.15.193:6812/157927432 conn(0x7fe4141374a0 legacy=0x7fe414139890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 20 client.23565.objecter put_session s=0x7fe414136df0 osd=7 1 2024-02-19T21:14:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 20 client.23565.objecter shutdown clearing up homeless session... 2024-02-19T21:14:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 10 client.23565.objecter successfully canceled tick 2024-02-19T21:14:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 >> v1:172.21.15.163:6800/1552514681 conn(0x7fe404063d70 legacy=0x7fe404066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:12.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 >> v1:172.21.15.163:6789/0 conn(0x7fe4140fc870 legacy=0x7fe4140fb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe419556700 1 -- 172.21.15.163:0/2666158633 reap_dead start 2024-02-19T21:14:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 shutdown_connections 2024-02-19T21:14:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 >> 172.21.15.163:0/2666158633 conn(0x7fe4140f2470 msgr2=0x7fe414106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 shutdown_connections 2024-02-19T21:14:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 1 -- 172.21.15.163:0/2666158633 wait complete. 2024-02-19T21:14:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:12.061+0000 7fe41afb9700 1 librados: shutdown 2024-02-19T21:14:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:14:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:14:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:14:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:14:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:14:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:14:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:14:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:14:12.067 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:12.171 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:14:12.312 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-19T21:14:12.434 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:11.282246+0000 mgr.y (mgr.14550) 532 : cluster [DBG] pgmap v921: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:14:12.434 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:11.427642+0000 mon.a (mon.0) 3135 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-106","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:14:12.434 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:11.427726+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e615: 8 total, 8 up, 6 in 2024-02-19T21:14:12.434 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:11.469617+0000 osd.2 (osd.2) 242 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:14:12.434 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:11.282246+0000 mgr.y (mgr.14550) 532 : cluster [DBG] pgmap v921: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:14:12.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:11.427642+0000 mon.a (mon.0) 3135 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-106","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:14:12.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:11.427726+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e615: 8 total, 8 up, 6 in 2024-02-19T21:14:12.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:11.469617+0000 osd.2 (osd.2) 242 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:14:12.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:11.471426+0000 osd.2 (osd.2) 243 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:14:12.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:11.577358+0000 osd.7 (osd.7) 275 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:14:12.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:11.580249+0000 osd.7 (osd.7) 276 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:14:12.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:11.918522+0000 mon.a (mon.0) 3137 : audit [DBG] from='client.? v1:172.21.15.163:0/2666158633' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:12.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:12.106129+0000 osd.3 (osd.3) 249 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:14:12.435 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:12.107005+0000 osd.3 (osd.3) 250 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:14:12.582 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:14:12.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:11.471426+0000 osd.2 (osd.2) 243 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:14:12.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:11.577358+0000 osd.7 (osd.7) 275 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:14:12.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:11.580249+0000 osd.7 (osd.7) 276 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:14:12.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:11.918522+0000 mon.a (mon.0) 3137 : audit [DBG] from='client.? v1:172.21.15.163:0/2666158633' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:12.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:12.106129+0000 osd.3 (osd.3) 249 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:14:12.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:12.107005+0000 osd.3 (osd.3) 250 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:14:12.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:11.282246+0000 mgr.y (mgr.14550) 532 : cluster [DBG] pgmap v921: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:14:12.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:11.427642+0000 mon.a (mon.0) 3135 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-106","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:14:12.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:11.427726+0000 mon.a (mon.0) 3136 : cluster [DBG] osdmap e615: 8 total, 8 up, 6 in 2024-02-19T21:14:12.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:11.469617+0000 osd.2 (osd.2) 242 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:14:12.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:11.471426+0000 osd.2 (osd.2) 243 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:14:12.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:11.577358+0000 osd.7 (osd.7) 275 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:14:12.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:11.580249+0000 osd.7 (osd.7) 276 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:14:12.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:11.918522+0000 mon.a (mon.0) 3137 : audit [DBG] from='client.? v1:172.21.15.163:0/2666158633' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:12.808 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:12.106129+0000 osd.3 (osd.3) 249 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:14:12.809 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:12.107005+0000 osd.3 (osd.3) 250 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:14:13.747 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:14:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:14:13] "GET /metrics HTTP/1.1" 200 191389 "" "Prometheus/2.33.4" 2024-02-19T21:14:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:12.440212+0000 mon.a (mon.0) 3138 : cluster [DBG] osdmap e616: 8 total, 8 up, 6 in 2024-02-19T21:14:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:12.452284+0000 mon.a (mon.0) 3139 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-106", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:14:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:12.479354+0000 osd.2 (osd.2) 244 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:14:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:12.482228+0000 osd.2 (osd.2) 245 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:14:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:12.563902+0000 osd.7 (osd.7) 277 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:14:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:12.566799+0000 osd.7 (osd.7) 278 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:14:13.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:12.440212+0000 mon.a (mon.0) 3138 : cluster [DBG] osdmap e616: 8 total, 8 up, 6 in 2024-02-19T21:14:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:12.452284+0000 mon.a (mon.0) 3139 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-106", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:14:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:12.479354+0000 osd.2 (osd.2) 244 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:14:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:12.482228+0000 osd.2 (osd.2) 245 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:14:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:12.563902+0000 osd.7 (osd.7) 277 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:14:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:12.566799+0000 osd.7 (osd.7) 278 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:14:13.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:12.440212+0000 mon.a (mon.0) 3138 : cluster [DBG] osdmap e616: 8 total, 8 up, 6 in 2024-02-19T21:14:13.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:12.452284+0000 mon.a (mon.0) 3139 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-106", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:14:13.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:12.479354+0000 osd.2 (osd.2) 244 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:14:13.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:12.482228+0000 osd.2 (osd.2) 245 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:14:13.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:12.563902+0000 osd.7 (osd.7) 277 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:14:13.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:12.566799+0000 osd.7 (osd.7) 278 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:14:13.788 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:14.300 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:14:14.301 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:14:14.301 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-02-19T21:14:14.301 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-02-19T21:14:14.301 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:14:14.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.290+0000 7f339bf79700 1 -- 172.21.15.163:0/2955490882 >> v1:172.21.15.163:6790/0 conn(0x7f3394100780 legacy=0x7f3394102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:14.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.290+0000 7f339bf79700 1 -- 172.21.15.163:0/2955490882 shutdown_connections 2024-02-19T21:14:14.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.290+0000 7f339bf79700 1 -- 172.21.15.163:0/2955490882 >> 172.21.15.163:0/2955490882 conn(0x7f33940f35a0 msgr2=0x7f33940f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:14.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.290+0000 7f339bf79700 1 -- 172.21.15.163:0/2955490882 shutdown_connections 2024-02-19T21:14:14.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.290+0000 7f339bf79700 1 -- 172.21.15.163:0/2955490882 wait complete. 2024-02-19T21:14:14.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.290+0000 7f339bf79700 1 librados: starting msgr at 2024-02-19T21:14:14.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.290+0000 7f339bf79700 1 librados: starting objecter 2024-02-19T21:14:14.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f339bf79700 1 Processor -- start 2024-02-19T21:14:14.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f339bf79700 1 -- start start 2024-02-19T21:14:14.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f339bf79700 1 librados: setting wanted keys 2024-02-19T21:14:14.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f339bf79700 1 librados: calling monclient init 2024-02-19T21:14:14.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f339bf79700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3394117440 con 0x7f33941042c0 2024-02-19T21:14:14.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f339bf79700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3394118620 con 0x7f33940fd980 2024-02-19T21:14:14.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f339bf79700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3394119820 con 0x7f3394100780 2024-02-19T21:14:14.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f339a516700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f33941042c0 0x7f3394115b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33920/0 (socket says 172.21.15.163:33920) 2024-02-19T21:14:14.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f3399514700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f3394100780 0x7f33940fa3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:44248/0 (socket says 172.21.15.163:44248) 2024-02-19T21:14:14.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f339a516700 1 -- 172.21.15.163:0/1298070114 learned_addr learned my addr 172.21.15.163:0/1298070114 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:14.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 10 client.?.objecter ms_handle_connect 0x7f3394100780 2024-02-19T21:14:14.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:14.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 10 client.?.objecter ms_handle_connect 0x7f33941042c0 2024-02-19T21:14:14.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:14.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 10 client.?.objecter ms_handle_connect 0x7f33940fd980 2024-02-19T21:14:14.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:14.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103047566 0 0) 0x7f3394119820 con 0x7f3394100780 2024-02-19T21:14:14.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33800033f0 con 0x7f3394100780 2024-02-19T21:14:14.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592120815 0 0) 0x7f3394117440 con 0x7f33941042c0 2024-02-19T21:14:14.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3394119820 con 0x7f33941042c0 2024-02-19T21:14:14.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031699837 0 0) 0x7f3394118620 con 0x7f33940fd980 2024-02-19T21:14:14.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3394117440 con 0x7f33940fd980 2024-02-19T21:14:14.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4038946116 0 0) 0x7f33800033f0 con 0x7f3394100780 2024-02-19T21:14:14.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3394118620 con 0x7f3394100780 2024-02-19T21:14:14.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3390002b10 con 0x7f3394100780 2024-02-19T21:14:14.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3148195591 0 0) 0x7f3394119820 con 0x7f33941042c0 2024-02-19T21:14:14.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33800033f0 con 0x7f33941042c0 2024-02-19T21:14:14.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 632880245 0 0) 0x7f3394117440 con 0x7f33940fd980 2024-02-19T21:14:14.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3394119820 con 0x7f33940fd980 2024-02-19T21:14:14.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.291+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f338c002c50 con 0x7f33941042c0 2024-02-19T21:14:14.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.292+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f33840017c0 con 0x7f33940fd980 2024-02-19T21:14:14.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.292+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3929781885 0 0) 0x7f3394118620 con 0x7f3394100780 2024-02-19T21:14:14.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.292+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 >> v1:172.21.15.193:6789/0 conn(0x7f33940fd980 legacy=0x7f33940f9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:14.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.292+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 >> v1:172.21.15.163:6789/0 conn(0x7f33941042c0 legacy=0x7f3394115b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:14.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.292+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339411aa20 con 0x7f3394100780 2024-02-19T21:14:14.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.292+0000 7f339bf79700 1 -- 172.21.15.163:0/1298070114 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3394117630 con 0x7f3394100780 2024-02-19T21:14:14.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.292+0000 7f339bf79700 10 client.50297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:14.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.292+0000 7f339bf79700 1 -- 172.21.15.163:0/1298070114 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3394117aa0 con 0x7f3394100780 2024-02-19T21:14:14.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.292+0000 7f339bf79700 1 librados: init done 2024-02-19T21:14:14.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.292+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3390001a90 con 0x7f3394100780 2024-02-19T21:14:14.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.292+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f33900037a0 con 0x7f3394100780 2024-02-19T21:14:14.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.293+0000 7f339bf79700 1 -- 172.21.15.163:0/1298070114 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3378004f20 con 0x7f3394100780 2024-02-19T21:14:14.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.293+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f33900039c0 con 0x7f3394100780 2024-02-19T21:14:14.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.293+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(617..617 src has 434..617) v4 ==== 8508+0+0 (unknown 630978347 0 0) 0x7f3390079170 con 0x7f3394100780 2024-02-19T21:14:14.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.293+0000 7f338affd700 10 client.50297.objecter ms_dispatch 0x7f3394001550 osd_map(617..617 src has 434..617) v4 2024-02-19T21:14:14.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.293+0000 7f338affd700 3 client.50297.objecter handle_osd_map got epochs [617,617] > 0 2024-02-19T21:14:14.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.293+0000 7f338affd700 3 client.50297.objecter handle_osd_map decoding full epoch 617 2024-02-19T21:14:14.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.294+0000 7f338affd700 20 client.50297.objecter dump_active .. 0 homeless 2024-02-19T21:14:14.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.294+0000 7f338affd700 10 client.50297.objecter ms_handle_connect 0x7f3380068130 2024-02-19T21:14:14.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.298+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f339004c400 con 0x7f3394100780 2024-02-19T21:14:14.448 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:13.282640+0000 mgr.y (mgr.14550) 533 : cluster [DBG] pgmap v924: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:14:14.448 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:13.447976+0000 mon.a (mon.0) 3140 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-106", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:14:14.448 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:13.448031+0000 mon.a (mon.0) 3141 : cluster [DBG] osdmap e617: 8 total, 8 up, 6 in 2024-02-19T21:14:14.448 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:13.451782+0000 mon.a (mon.0) 3142 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-106"}]: dispatch 2024-02-19T21:14:14.448 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:13.529113+0000 osd.2 (osd.2) 246 : cluster [DBG] 16.0 scrub starts 2024-02-19T21:14:14.449 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:13.532564+0000 osd.2 (osd.2) 247 : cluster [DBG] 16.0 scrub ok 2024-02-19T21:14:14.449 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:14.062667+0000 osd.3 (osd.3) 251 : cluster [DBG] 2.1b scrub starts 2024-02-19T21:14:14.449 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:14.063564+0000 osd.3 (osd.3) 252 : cluster [DBG] 2.1b scrub ok 2024-02-19T21:14:14.449 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:14.354636+0000 mon.a (mon.0) 3143 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-106"}]': finished 2024-02-19T21:14:14.449 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:14.354694+0000 mon.a (mon.0) 3144 : cluster [DBG] osdmap e618: 8 total, 8 up, 6 in 2024-02-19T21:14:14.449 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:14.355960+0000 mon.a (mon.0) 3145 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-106", "mode": "writeback"}]: dispatch 2024-02-19T21:14:14.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:14.483+0000 7f339bf79700 1 -- 172.21.15.163:0/1298070114 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd in", "ids": ["6"]} v 0) v1 -- 0x7f33780051d0 con 0x7f3394100780 2024-02-19T21:14:14.618 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:14.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:14.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:14.711 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:13.282640+0000 mgr.y (mgr.14550) 533 : cluster [DBG] pgmap v924: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:14:14.711 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:13.447976+0000 mon.a (mon.0) 3140 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-106", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:14:14.711 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:13.448031+0000 mon.a (mon.0) 3141 : cluster [DBG] osdmap e617: 8 total, 8 up, 6 in 2024-02-19T21:14:14.711 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:13.451782+0000 mon.a (mon.0) 3142 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-106"}]: dispatch 2024-02-19T21:14:14.711 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:13.529113+0000 osd.2 (osd.2) 246 : cluster [DBG] 16.0 scrub starts 2024-02-19T21:14:14.711 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:13.532564+0000 osd.2 (osd.2) 247 : cluster [DBG] 16.0 scrub ok 2024-02-19T21:14:14.711 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:14.062667+0000 osd.3 (osd.3) 251 : cluster [DBG] 2.1b scrub starts 2024-02-19T21:14:14.711 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:14.063564+0000 osd.3 (osd.3) 252 : cluster [DBG] 2.1b scrub ok 2024-02-19T21:14:14.712 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:14.354636+0000 mon.a (mon.0) 3143 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-106"}]': finished 2024-02-19T21:14:14.712 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:14.354694+0000 mon.a (mon.0) 3144 : cluster [DBG] osdmap e618: 8 total, 8 up, 6 in 2024-02-19T21:14:14.712 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:14.355960+0000 mon.a (mon.0) 3145 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-106", "mode": "writeback"}]: dispatch 2024-02-19T21:14:14.722 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:14.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:14:14.484+0000 7fb606608700 -1 mon.a@0(leader).osd e618 definitely_dead 0 2024-02-19T21:14:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:13.282640+0000 mgr.y (mgr.14550) 533 : cluster [DBG] pgmap v924: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:14:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:13.447976+0000 mon.a (mon.0) 3140 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-106", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:14:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:13.448031+0000 mon.a (mon.0) 3141 : cluster [DBG] osdmap e617: 8 total, 8 up, 6 in 2024-02-19T21:14:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:13.451782+0000 mon.a (mon.0) 3142 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-106"}]: dispatch 2024-02-19T21:14:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:13.529113+0000 osd.2 (osd.2) 246 : cluster [DBG] 16.0 scrub starts 2024-02-19T21:14:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:13.532564+0000 osd.2 (osd.2) 247 : cluster [DBG] 16.0 scrub ok 2024-02-19T21:14:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:14.062667+0000 osd.3 (osd.3) 251 : cluster [DBG] 2.1b scrub starts 2024-02-19T21:14:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:14.063564+0000 osd.3 (osd.3) 252 : cluster [DBG] 2.1b scrub ok 2024-02-19T21:14:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:14.354636+0000 mon.a (mon.0) 3143 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-106"}]': finished 2024-02-19T21:14:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:14.354694+0000 mon.a (mon.0) 3144 : cluster [DBG] osdmap e618: 8 total, 8 up, 6 in 2024-02-19T21:14:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:14.355960+0000 mon.a (mon.0) 3145 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-106", "mode": "writeback"}]: dispatch 2024-02-19T21:14:14.753 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:14.826 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:14.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:14.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:14.930 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:14.959 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:15.044 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-19T21:14:15.208 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:15.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:15.311 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:14:15.314 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:15.342 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:15.354 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:15.354+0000 7f338affd700 1 -- 172.21.15.163:0/1298070114 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd in", "ids": ["6"]}]=0 marked in osd.6. v619) v1 ==== 85+0+0 (unknown 2663386768 0 0) 0x7f339004a870 con 0x7f3394100780 2024-02-19T21:14:15.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:15.357+0000 7f339bf79700 10 librados: watch_flush enter 2024-02-19T21:14:15.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:15.358+0000 7f339bf79700 10 librados: watch_flush exit 2024-02-19T21:14:15.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:15.358+0000 7f339bf79700 20 client.50297.objecter shutdown clearing up homeless session... 2024-02-19T21:14:15.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:15.358+0000 7f339bf79700 10 client.50297.objecter successfully canceled tick 2024-02-19T21:14:15.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:15.359+0000 7f339bf79700 1 -- 172.21.15.163:0/1298070114 >> v1:172.21.15.163:6800/1552514681 conn(0x7f3380068130 legacy=0x7f338006a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:15.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:15.359+0000 7f339bf79700 1 -- 172.21.15.163:0/1298070114 >> v1:172.21.15.163:6790/0 conn(0x7f3394100780 legacy=0x7f33940fa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:15.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:15.360+0000 7f339bf79700 1 -- 172.21.15.163:0/1298070114 shutdown_connections 2024-02-19T21:14:15.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:15.360+0000 7f339bf79700 1 -- 172.21.15.163:0/1298070114 >> 172.21.15.163:0/1298070114 conn(0x7f33940f35a0 msgr2=0x7f3394107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:15.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:15.363+0000 7f339bf79700 1 -- 172.21.15.163:0/1298070114 shutdown_connections 2024-02-19T21:14:15.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:15.365+0000 7f339bf79700 1 -- 172.21.15.163:0/1298070114 wait complete. 2024-02-19T21:14:15.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:15.365+0000 7f339bf79700 1 librados: shutdown 2024-02-19T21:14:15.379 INFO:teuthology.orchestra.run.smithi163.stderr:marked in osd.6. 2024-02-19T21:14:15.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:13.928144+0000 osd.5 (osd.5) 242 : cluster [DBG] 16.11 scrub starts 2024-02-19T21:14:15.497 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:13.932017+0000 osd.5 (osd.5) 243 : cluster [DBG] 16.11 scrub ok 2024-02-19T21:14:15.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:14.485126+0000 mon.c (mon.2) 416 : audit [DBG] from='client.? v1:172.21.15.163:0/1298070114' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-19T21:14:15.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:14.485444+0000 mon.a (mon.0) 3146 : audit [DBG] from='client.50297 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-19T21:14:15.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:15.347595+0000 mon.a (mon.0) 3147 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:14:15.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:15.351508+0000 mon.a (mon.0) 3148 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-106", "mode": "writeback"}]': finished 2024-02-19T21:14:15.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:15.351577+0000 mon.a (mon.0) 3149 : audit [INF] from='client.50297 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-02-19T21:14:15.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:15.351641+0000 mon.a (mon.0) 3150 : cluster [DBG] osdmap e619: 8 total, 8 up, 7 in 2024-02-19T21:14:15.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:13.928144+0000 osd.5 (osd.5) 242 : cluster [DBG] 16.11 scrub starts 2024-02-19T21:14:15.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:13.932017+0000 osd.5 (osd.5) 243 : cluster [DBG] 16.11 scrub ok 2024-02-19T21:14:15.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:14.485126+0000 mon.c (mon.2) 416 : audit [DBG] from='client.? v1:172.21.15.163:0/1298070114' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-19T21:14:15.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:14.485444+0000 mon.a (mon.0) 3146 : audit [DBG] from='client.50297 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-19T21:14:15.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:15.347595+0000 mon.a (mon.0) 3147 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:14:15.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:15.351508+0000 mon.a (mon.0) 3148 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-106", "mode": "writeback"}]': finished 2024-02-19T21:14:15.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:15.351577+0000 mon.a (mon.0) 3149 : audit [INF] from='client.50297 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-02-19T21:14:15.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:15.351641+0000 mon.a (mon.0) 3150 : cluster [DBG] osdmap e619: 8 total, 8 up, 7 in 2024-02-19T21:14:15.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:13.928144+0000 osd.5 (osd.5) 242 : cluster [DBG] 16.11 scrub starts 2024-02-19T21:14:15.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:13.932017+0000 osd.5 (osd.5) 243 : cluster [DBG] 16.11 scrub ok 2024-02-19T21:14:15.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:14.485126+0000 mon.c (mon.2) 416 : audit [DBG] from='client.? v1:172.21.15.163:0/1298070114' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-19T21:14:15.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:14.485444+0000 mon.a (mon.0) 3146 : audit [DBG] from='client.50297 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-19T21:14:15.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:15.347595+0000 mon.a (mon.0) 3147 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:14:15.767 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:15.351508+0000 mon.a (mon.0) 3148 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-106", "mode": "writeback"}]': finished 2024-02-19T21:14:15.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:15.351577+0000 mon.a (mon.0) 3149 : audit [INF] from='client.50297 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-02-19T21:14:15.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:15.351641+0000 mon.a (mon.0) 3150 : cluster [DBG] osdmap e619: 8 total, 8 up, 7 in 2024-02-19T21:14:15.779 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:14:16.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.397+0000 7f401e081700 1 -- 172.21.15.163:0/1434180132 >> v1:172.21.15.163:6789/0 conn(0x7f40181031c0 legacy=0x7f4018105660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:16.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.397+0000 7f401e081700 1 -- 172.21.15.163:0/1434180132 shutdown_connections 2024-02-19T21:14:16.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.397+0000 7f401e081700 1 -- 172.21.15.163:0/1434180132 >> 172.21.15.163:0/1434180132 conn(0x7f401806fdc0 msgr2=0x7f40180f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:16.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.398+0000 7f401e081700 1 -- 172.21.15.163:0/1434180132 shutdown_connections 2024-02-19T21:14:16.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.398+0000 7f401e081700 1 -- 172.21.15.163:0/1434180132 wait complete. 2024-02-19T21:14:16.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.398+0000 7f401e081700 1 librados: starting msgr at 2024-02-19T21:14:16.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.398+0000 7f401e081700 1 librados: starting objecter 2024-02-19T21:14:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.398+0000 7f401e081700 1 Processor -- start 2024-02-19T21:14:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.399+0000 7f401e081700 1 -- start start 2024-02-19T21:14:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.399+0000 7f401e081700 1 librados: setting wanted keys 2024-02-19T21:14:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.399+0000 7f401e081700 1 librados: calling monclient init 2024-02-19T21:14:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.399+0000 7f401e081700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f401806ab50 con 0x7f40180ff680 2024-02-19T21:14:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.399+0000 7f401e081700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f401806ace0 con 0x7f40181031c0 2024-02-19T21:14:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.399+0000 7f401e081700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4018122150 con 0x7f40180fc8a0 2024-02-19T21:14:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.399+0000 7f4017fff700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f40181031c0 0x7f401811d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:42474/0 (socket says 172.21.15.163:42474) 2024-02-19T21:14:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.399+0000 7f4017fff700 1 -- 172.21.15.163:0/3084512438 learned_addr learned my addr 172.21.15.163:0/3084512438 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.399+0000 7f4014ff9700 10 client.?.objecter ms_handle_connect 0x7f40180fc8a0 2024-02-19T21:14:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.399+0000 7f4014ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.399+0000 7f4014ff9700 10 client.?.objecter ms_handle_connect 0x7f40180ff680 2024-02-19T21:14:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.399+0000 7f4014ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 10 client.?.objecter ms_handle_connect 0x7f40181031c0 2024-02-19T21:14:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437023832 0 0) 0x7f4018122150 con 0x7f40180fc8a0 2024-02-19T21:14:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4000003630 con 0x7f40180fc8a0 2024-02-19T21:14:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864865800 0 0) 0x7f401806ab50 con 0x7f40180ff680 2024-02-19T21:14:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4018122150 con 0x7f40180ff680 2024-02-19T21:14:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4052121340 0 0) 0x7f4000003630 con 0x7f40180fc8a0 2024-02-19T21:14:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f401806ab50 con 0x7f40180fc8a0 2024-02-19T21:14:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452929074 0 0) 0x7f401806ace0 con 0x7f40181031c0 2024-02-19T21:14:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4000003630 con 0x7f40181031c0 2024-02-19T21:14:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f40080018c0 con 0x7f40180fc8a0 2024-02-19T21:14:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3097270457 0 0) 0x7f4018122150 con 0x7f40180ff680 2024-02-19T21:14:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f401806ace0 con 0x7f40180ff680 2024-02-19T21:14:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f400c001840 con 0x7f40180ff680 2024-02-19T21:14:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1241119975 0 0) 0x7f401806ace0 con 0x7f40180ff680 2024-02-19T21:14:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 >> v1:172.21.15.163:6790/0 conn(0x7f40180fc8a0 legacy=0x7f40180f8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 >> v1:172.21.15.193:6789/0 conn(0x7f40181031c0 legacy=0x7f401811d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401811ef30 con 0x7f40180ff680 2024-02-19T21:14:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.400+0000 7f401e081700 1 -- 172.21.15.163:0/3084512438 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401811f120 con 0x7f40180ff680 2024-02-19T21:14:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.401+0000 7f401e081700 10 client.23577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.401+0000 7f401e081700 1 -- 172.21.15.163:0/3084512438 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f401811f590 con 0x7f40180ff680 2024-02-19T21:14:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.401+0000 7f401e081700 1 librados: init done 2024-02-19T21:14:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.401+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f400c002270 con 0x7f40180ff680 2024-02-19T21:14:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.401+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f400c0031e0 con 0x7f40180ff680 2024-02-19T21:14:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.402+0000 7f401e081700 1 -- 172.21.15.163:0/3084512438 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4004004f20 con 0x7f40180ff680 2024-02-19T21:14:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.402+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f400c016090 con 0x7f40180ff680 2024-02-19T21:14:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.403+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(620..620 src has 434..620) v4 ==== 8541+0+0 (unknown 3102860178 0 0) 0x7f400c0766b0 con 0x7f40180ff680 2024-02-19T21:14:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.403+0000 7f4014ff9700 10 client.23577.objecter ms_dispatch 0x7f4018001550 osd_map(620..620 src has 434..620) v4 2024-02-19T21:14:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.403+0000 7f4014ff9700 3 client.23577.objecter handle_osd_map got epochs [620,620] > 0 2024-02-19T21:14:16.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.403+0000 7f4014ff9700 3 client.23577.objecter handle_osd_map decoding full epoch 620 2024-02-19T21:14:16.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.403+0000 7f4014ff9700 20 client.23577.objecter dump_active .. 0 homeless 2024-02-19T21:14:16.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.403+0000 7f4014ff9700 10 client.23577.objecter ms_handle_connect 0x7f4000063d10 2024-02-19T21:14:16.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.409+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f400c049a30 con 0x7f40180ff680 2024-02-19T21:14:16.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:14.954798+0000 osd.5 (osd.5) 244 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:14:16.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:14.957715+0000 osd.5 (osd.5) 245 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:14:16.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:15.283480+0000 mgr.y (mgr.14550) 534 : cluster [DBG] pgmap v927: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:14:16.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:16.355658+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e620: 8 total, 8 up, 7 in 2024-02-19T21:14:16.562 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:16.421752+0000 mon.a (mon.0) 3152 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "237.3"}]: dispatch 2024-02-19T21:14:16.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:16.595+0000 7f401e081700 1 -- 172.21.15.163:0/3084512438 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f40040051d0 con 0x7f40180ff680 2024-02-19T21:14:16.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:14.954798+0000 osd.5 (osd.5) 244 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:14:16.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:14.957715+0000 osd.5 (osd.5) 245 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:14:16.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:15.283480+0000 mgr.y (mgr.14550) 534 : cluster [DBG] pgmap v927: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:14:16.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:16.355658+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e620: 8 total, 8 up, 7 in 2024-02-19T21:14:16.715 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:16.421752+0000 mon.a (mon.0) 3152 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "237.3"}]: dispatch 2024-02-19T21:14:16.715 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:14.954798+0000 osd.5 (osd.5) 244 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:14:16.715 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:14.957715+0000 osd.5 (osd.5) 245 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:14:16.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:15.283480+0000 mgr.y (mgr.14550) 534 : cluster [DBG] pgmap v927: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:14:16.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:16.355658+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e620: 8 total, 8 up, 7 in 2024-02-19T21:14:16.716 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:16.421752+0000 mon.a (mon.0) 3152 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "237.3"}]: dispatch 2024-02-19T21:14:16.716 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:16.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:16.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:16.738 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-02-19T21:14:16.751 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:16.819 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:14:16.849 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:16.872 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:14:16.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:16.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:14:16.977 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:14:17.009 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:17.110 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:14:17.113 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:17.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:14:17.126 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:14:17.126 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-02-19T21:14:17.126 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-02-19T21:14:17.126 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-02-19T21:14:17.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.126 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:109 61.1e 61:7b0234b0:test-rados-api-smithi163-83928-10::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e126)", 2024-02-19T21:14:17.126 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:06.034266+0000", 2024-02-19T21:14:17.127 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 551.09011245900001, 2024-02-19T21:14:17.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013927448, 2024-02-19T21:14:17.127 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.127 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:17.127 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.127 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:14:17.127 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:14:17.127 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 109 2024-02-19T21:14:17.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.127 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.127 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034266+0000", 2024-02-19T21:14:17.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.128 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.128 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.128 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034266+0000", 2024-02-19T21:14:17.128 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:14:17.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.128 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.128 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.128 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034264+0000", 2024-02-19T21:14:17.128 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4800000000000001e-06 2024-02-19T21:14:17.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.128 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.129 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.129 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034272+0000", 2024-02-19T21:14:17.129 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8009999999999993e-06 2024-02-19T21:14:17.129 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.129 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.129 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.129 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034281+0000", 2024-02-19T21:14:17.129 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9020000000000002e-06 2024-02-19T21:14:17.129 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.129 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.130 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.130 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034288+0000", 2024-02-19T21:14:17.130 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018518399999999999 2024-02-19T21:14:17.130 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.130 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.130 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.130 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034473+0000", 2024-02-19T21:14:17.130 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012265400000000001 2024-02-19T21:14:17.130 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.130 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.130 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.130 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034596+0000", 2024-02-19T21:14:17.130 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000145477 2024-02-19T21:14:17.131 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.131 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.131 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,6", 2024-02-19T21:14:17.131 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.034742+0000", 2024-02-19T21:14:17.131 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001663596 2024-02-19T21:14:17.131 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.131 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.131 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:17.131 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.036405+0000", 2024-02-19T21:14:17.131 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0045168939999999996 2024-02-19T21:14:17.131 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.132 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.132 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.132 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.040922+0000", 2024-02-19T21:14:17.132 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0072134110000000003 2024-02-19T21:14:17.132 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.132 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.132 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.132 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.048135+0000", 2024-02-19T21:14:17.132 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4739000000000001e-05 2024-02-19T21:14:17.132 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.133 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.133 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:17.133 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.048170+0000", 2024-02-19T21:14:17.133 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3264e-05 2024-02-19T21:14:17.133 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.133 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.133 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.133 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:06.048193+0000", 2024-02-19T21:14:17.133 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.013927448 2024-02-19T21:14:17.133 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.133 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.133 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.133 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.134 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.134 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.38953.0:6 82.c 82:3197012c:::foo_rmxattr:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e132)", 2024-02-19T21:14:17.134 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.230127+0000", 2024-02-19T21:14:17.134 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 544.89425180299997, 2024-02-19T21:14:17.134 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.044067013000000002, 2024-02-19T21:14:17.134 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.134 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:17.134 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.134 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.38953", 2024-02-19T21:14:17.135 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3840798168", 2024-02-19T21:14:17.135 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 6 2024-02-19T21:14:17.135 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.135 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.135 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.135 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.135 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230127+0000", 2024-02-19T21:14:17.135 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.135 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.136 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.136 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.136 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230127+0000", 2024-02-19T21:14:17.136 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:14:17.136 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.136 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.136 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.136 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230125+0000", 2024-02-19T21:14:17.136 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5880000000000004e-06 2024-02-19T21:14:17.137 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.137 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.137 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.137 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230134+0000", 2024-02-19T21:14:17.137 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1489999999999994e-06 2024-02-19T21:14:17.137 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.137 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.137 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.137 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230142+0000", 2024-02-19T21:14:17.137 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9089999999999999e-06 2024-02-19T21:14:17.137 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.138 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.138 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.138 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230147+0000", 2024-02-19T21:14:17.138 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5340999999999997e-05 2024-02-19T21:14:17.138 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.138 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.138 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.138 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230213+0000", 2024-02-19T21:14:17.138 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000120505 2024-02-19T21:14:17.138 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.138 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.139 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.139 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230333+0000", 2024-02-19T21:14:17.139 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015051999999999999 2024-02-19T21:14:17.139 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.139 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.139 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,4", 2024-02-19T21:14:17.139 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230484+0000", 2024-02-19T21:14:17.139 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00033057799999999999 2024-02-19T21:14:17.139 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.139 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.140 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:17.140 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.230814+0000", 2024-02-19T21:14:17.140 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000223568 2024-02-19T21:14:17.140 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.140 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.140 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.231038+0000", 2024-02-19T21:14:17.140 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.043097783000000001 2024-02-19T21:14:17.140 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.141 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.141 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.141 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.274136+0000", 2024-02-19T21:14:17.141 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2132999999999999e-05 2024-02-19T21:14:17.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.141 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.141 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:17.141 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.274168+0000", 2024-02-19T21:14:17.141 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5828000000000001e-05 2024-02-19T21:14:17.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.142 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.142 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.142 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.274194+0000", 2024-02-19T21:14:17.142 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.044067013000000002 2024-02-19T21:14:17.142 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.142 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.142 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.142 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.142 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.142 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.38953.0:8 82.c 82:3197012c:::foo_rmxattr:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e132)", 2024-02-19T21:14:17.143 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:12.288784+0000", 2024-02-19T21:14:17.143 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 544.83559434899996, 2024-02-19T21:14:17.143 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.027473164000000001, 2024-02-19T21:14:17.143 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.143 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:17.143 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.143 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.38953", 2024-02-19T21:14:17.143 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/3840798168", 2024-02-19T21:14:17.143 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 8 2024-02-19T21:14:17.144 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.144 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.144 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.144 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.144 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288784+0000", 2024-02-19T21:14:17.144 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.144 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.144 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.144 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.144 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288784+0000", 2024-02-19T21:14:17.144 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:14:17.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.145 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.145 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288781+0000", 2024-02-19T21:14:17.145 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2977e-05 2024-02-19T21:14:17.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.145 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.145 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288794+0000", 2024-02-19T21:14:17.145 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6616000000000001e-05 2024-02-19T21:14:17.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.145 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.145 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288820+0000", 2024-02-19T21:14:17.146 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2593e-05 2024-02-19T21:14:17.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.146 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.146 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288833+0000", 2024-02-19T21:14:17.146 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9002000000000001e-05 2024-02-19T21:14:17.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.146 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.146 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288902+0000", 2024-02-19T21:14:17.146 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4057000000000002e-05 2024-02-19T21:14:17.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.146 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.147 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.288946+0000", 2024-02-19T21:14:17.147 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2805999999999994e-05 2024-02-19T21:14:17.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.147 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.147 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,4", 2024-02-19T21:14:17.147 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.289019+0000", 2024-02-19T21:14:17.147 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025527699999999999 2024-02-19T21:14:17.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.147 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.147 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:17.147 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.289274+0000", 2024-02-19T21:14:17.147 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000221688 2024-02-19T21:14:17.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.148 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.148 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.148 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.289496+0000", 2024-02-19T21:14:17.148 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026693499999999998 2024-02-19T21:14:17.148 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.148 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.148 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.148 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.316189+0000", 2024-02-19T21:14:17.148 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.042e-05 2024-02-19T21:14:17.148 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.148 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.149 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:17.149 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.316220+0000", 2024-02-19T21:14:17.149 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7688000000000002e-05 2024-02-19T21:14:17.149 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.149 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.149 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.149 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:12.316257+0000", 2024-02-19T21:14:17.149 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.027473164000000001 2024-02-19T21:14:17.149 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.149 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.150 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.150 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.150 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.150 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:248 61.1e 61:780b1894:test-rados-api-smithi163-83928-12::bar:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-02-19T21:14:17.150 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:14.241429+0000", 2024-02-19T21:14:17.150 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 542.88294921600004, 2024-02-19T21:14:17.150 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.036279570999999997, 2024-02-19T21:14:17.150 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.150 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:17.150 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.150 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:14:17.150 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:14:17.150 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 248 2024-02-19T21:14:17.150 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.151 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.151 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.151 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.151 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241429+0000", 2024-02-19T21:14:17.151 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.151 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.151 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.151 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.151 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241429+0000", 2024-02-19T21:14:17.151 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:14:17.151 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.151 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.151 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.151 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241427+0000", 2024-02-19T21:14:17.152 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1410000000000005e-06 2024-02-19T21:14:17.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.152 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.152 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241435+0000", 2024-02-19T21:14:17.152 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7259999999999997e-06 2024-02-19T21:14:17.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.152 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.152 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241445+0000", 2024-02-19T21:14:17.152 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3310000000000003e-06 2024-02-19T21:14:17.152 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.152 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.152 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.153 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241450+0000", 2024-02-19T21:14:17.153 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5508999999999999e-05 2024-02-19T21:14:17.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.153 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.153 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.153 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241516+0000", 2024-02-19T21:14:17.153 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012163399999999999 2024-02-19T21:14:17.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.153 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.153 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.153 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241638+0000", 2024-02-19T21:14:17.153 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000150047 2024-02-19T21:14:17.153 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.153 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.154 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,6", 2024-02-19T21:14:17.154 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.241788+0000", 2024-02-19T21:14:17.154 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00038934700000000001 2024-02-19T21:14:17.154 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.154 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.154 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:17.154 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.242177+0000", 2024-02-19T21:14:17.154 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018429899999999999 2024-02-19T21:14:17.154 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.154 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.154 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.154 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.242361+0000", 2024-02-19T21:14:17.154 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.035272228000000003 2024-02-19T21:14:17.154 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.154 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.155 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.155 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.277633+0000", 2024-02-19T21:14:17.155 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8115000000000003e-05 2024-02-19T21:14:17.155 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.155 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.155 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:17.155 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.277682+0000", 2024-02-19T21:14:17.155 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7237e-05 2024-02-19T21:14:17.155 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.155 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.155 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.155 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.277709+0000", 2024-02-19T21:14:17.155 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.036279570999999997 2024-02-19T21:14:17.155 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.156 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.156 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.156 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.156 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.156 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:249 61.10 61:0d6787e5:test-rados-api-smithi163-83928-12::baz:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e134)", 2024-02-19T21:14:17.156 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:14.291021+0000", 2024-02-19T21:14:17.156 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 542.83335737899995, 2024-02-19T21:14:17.156 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.031923083999999997, 2024-02-19T21:14:17.156 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.156 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:17.156 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.156 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:14:17.156 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:14:17.156 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 249 2024-02-19T21:14:17.157 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.157 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.157 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.157 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.157 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291021+0000", 2024-02-19T21:14:17.157 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.157 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.157 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.157 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.157 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291021+0000", 2024-02-19T21:14:17.157 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:14:17.157 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.157 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.157 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.157 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291019+0000", 2024-02-19T21:14:17.158 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1210000000000001e-06 2024-02-19T21:14:17.158 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.158 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.158 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.158 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291026+0000", 2024-02-19T21:14:17.158 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8940000000000005e-06 2024-02-19T21:14:17.158 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.158 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.158 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.158 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291036+0000", 2024-02-19T21:14:17.158 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8369999999999996e-06 2024-02-19T21:14:17.158 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.158 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.158 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.159 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291041+0000", 2024-02-19T21:14:17.159 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8821999999999999e-05 2024-02-19T21:14:17.159 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.159 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.159 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.159 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291109+0000", 2024-02-19T21:14:17.159 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4403999999999999e-05 2024-02-19T21:14:17.159 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.159 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.159 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.159 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291204+0000", 2024-02-19T21:14:17.159 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2696999999999997e-05 2024-02-19T21:14:17.159 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.159 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.160 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-02-19T21:14:17.160 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291297+0000", 2024-02-19T21:14:17.160 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00037753299999999999 2024-02-19T21:14:17.160 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.160 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.160 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:17.160 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291674+0000", 2024-02-19T21:14:17.160 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000206999 2024-02-19T21:14:17.160 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.160 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.160 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.160 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.291881+0000", 2024-02-19T21:14:17.160 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.030980852999999999 2024-02-19T21:14:17.160 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.161 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.161 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.161 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.322862+0000", 2024-02-19T21:14:17.161 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.5098000000000001e-05 2024-02-19T21:14:17.161 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.161 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.161 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:17.161 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.322907+0000", 2024-02-19T21:14:17.161 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7076000000000002e-05 2024-02-19T21:14:17.161 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.161 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.161 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.161 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:14.322944+0000", 2024-02-19T21:14:17.161 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.031923083999999997 2024-02-19T21:14:17.162 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.162 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.162 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.162 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.162 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.162 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:253 61.1e 61:780b1894:test-rados-api-smithi163-83928-12::bar:head [writefull 0~5] snapc 2=[2] ondisk+write+known_if_redirected e135)", 2024-02-19T21:14:17.162 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:15.193696+0000", 2024-02-19T21:14:17.163 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 541.93068247300005, 2024-02-19T21:14:17.163 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.104949372, 2024-02-19T21:14:17.163 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.163 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:17.163 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.163 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:14:17.163 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:14:17.163 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 253 2024-02-19T21:14:17.163 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.163 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.163 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.164 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.164 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193696+0000", 2024-02-19T21:14:17.164 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.164 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.164 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.164 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193696+0000", 2024-02-19T21:14:17.164 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:14:17.164 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.165 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.165 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.165 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193694+0000", 2024-02-19T21:14:17.165 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0650000000000001e-06 2024-02-19T21:14:17.165 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.165 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.165 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.165 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193701+0000", 2024-02-19T21:14:17.165 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.431e-06 2024-02-19T21:14:17.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.166 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.166 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.166 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193711+0000", 2024-02-19T21:14:17.166 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0629999999999996e-06 2024-02-19T21:14:17.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.166 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.166 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.166 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193716+0000", 2024-02-19T21:14:17.166 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0915000000000003e-05 2024-02-19T21:14:17.166 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.166 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.166 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.166 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193767+0000", 2024-02-19T21:14:17.167 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9301999999999995e-05 2024-02-19T21:14:17.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.167 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.167 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.167 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193836+0000", 2024-02-19T21:14:17.167 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012302 2024-02-19T21:14:17.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.167 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.167 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,6", 2024-02-19T21:14:17.167 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.193959+0000", 2024-02-19T21:14:17.167 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00041847100000000001 2024-02-19T21:14:17.167 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.167 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.167 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:17.167 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.194377+0000", 2024-02-19T21:14:17.168 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00021071800000000001 2024-02-19T21:14:17.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.168 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.168 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.168 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.194588+0000", 2024-02-19T21:14:17.168 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.103953399 2024-02-19T21:14:17.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.168 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.168 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.168 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.298541+0000", 2024-02-19T21:14:17.168 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9375e-05 2024-02-19T21:14:17.168 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.168 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.169 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:17.169 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.298601+0000", 2024-02-19T21:14:17.169 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4469999999999999e-05 2024-02-19T21:14:17.169 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.169 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.169 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.169 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:15.298645+0000", 2024-02-19T21:14:17.169 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.104949372 2024-02-19T21:14:17.169 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.169 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.170 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.170 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.170 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.170 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22230.0:2 102.d 102:b5a28c71:::oid_FlushPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e154)", 2024-02-19T21:14:17.170 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:34.185436+0000", 2024-02-19T21:14:17.170 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 522.93894213299995, 2024-02-19T21:14:17.170 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015189054, 2024-02-19T21:14:17.171 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.171 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:17.171 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.171 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22230", 2024-02-19T21:14:17.171 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/1800034971", 2024-02-19T21:14:17.171 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:14:17.171 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.171 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.171 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.171 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.172 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185436+0000", 2024-02-19T21:14:17.172 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.172 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.172 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.172 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.172 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185436+0000", 2024-02-19T21:14:17.172 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:14:17.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.173 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.173 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.173 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185434+0000", 2024-02-19T21:14:17.173 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.024e-06 2024-02-19T21:14:17.173 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.173 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.173 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.174 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185441+0000", 2024-02-19T21:14:17.174 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6830000000000003e-06 2024-02-19T21:14:17.174 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.174 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.174 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.174 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185451+0000", 2024-02-19T21:14:17.174 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6630000000000004e-06 2024-02-19T21:14:17.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.175 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.175 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.175 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.185458+0000", 2024-02-19T21:14:17.175 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00056233499999999996 2024-02-19T21:14:17.175 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.176 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.176 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.176 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.186020+0000", 2024-02-19T21:14:17.176 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1280000000000008e-06 2024-02-19T21:14:17.176 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.176 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.177 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:17.177 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.186028+0000", 2024-02-19T21:14:17.177 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.012894070000000001 2024-02-19T21:14:17.177 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.177 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.177 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.177 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.198922+0000", 2024-02-19T21:14:17.178 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012016800000000001 2024-02-19T21:14:17.178 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.178 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.178 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.178 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.199042+0000", 2024-02-19T21:14:17.178 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000105312 2024-02-19T21:14:17.178 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.179 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.179 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,6", 2024-02-19T21:14:17.179 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.199148+0000", 2024-02-19T21:14:17.179 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00055710799999999995 2024-02-19T21:14:17.179 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.179 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.180 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:17.180 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.199705+0000", 2024-02-19T21:14:17.180 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7489999999999999e-05 2024-02-19T21:14:17.180 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.180 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.180 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.180 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.199732+0000", 2024-02-19T21:14:17.181 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00081238399999999998 2024-02-19T21:14:17.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.181 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.181 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.200545+0000", 2024-02-19T21:14:17.181 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8712999999999999e-05 2024-02-19T21:14:17.181 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.181 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.181 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:17.181 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.200593+0000", 2024-02-19T21:14:17.182 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1940999999999998e-05 2024-02-19T21:14:17.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.182 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.182 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.182 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:34.200625+0000", 2024-02-19T21:14:17.182 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015189054 2024-02-19T21:14:17.182 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.182 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.182 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.182 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.182 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.182 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22254.0:2 104.9 104:90ca3134:::oid_FlushAsyncPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e157)", 2024-02-19T21:14:17.183 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:37.200018+0000", 2024-02-19T21:14:17.183 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 519.92436091100001, 2024-02-19T21:14:17.183 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0068597529999999997, 2024-02-19T21:14:17.183 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.183 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:17.183 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.183 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22254", 2024-02-19T21:14:17.183 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2521895994", 2024-02-19T21:14:17.183 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:14:17.183 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.183 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.183 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.183 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.184 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200018+0000", 2024-02-19T21:14:17.184 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.184 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.184 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.184 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200018+0000", 2024-02-19T21:14:17.184 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:14:17.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.184 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.184 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.184 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200015+0000", 2024-02-19T21:14:17.184 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8359999999999992e-06 2024-02-19T21:14:17.184 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.185 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.185 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.185 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200023+0000", 2024-02-19T21:14:17.185 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9250000000000007e-06 2024-02-19T21:14:17.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.185 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.185 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.185 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200033+0000", 2024-02-19T21:14:17.185 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7489999999999998e-06 2024-02-19T21:14:17.185 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.185 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.185 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.185 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200041+0000", 2024-02-19T21:14:17.185 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015734399999999999 2024-02-19T21:14:17.186 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.186 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.186 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.186 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200198+0000", 2024-02-19T21:14:17.186 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8997000000000002e-05 2024-02-19T21:14:17.186 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.186 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.186 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.186 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200287+0000", 2024-02-19T21:14:17.186 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000121697 2024-02-19T21:14:17.186 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.186 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.186 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 5,6", 2024-02-19T21:14:17.186 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.200409+0000", 2024-02-19T21:14:17.187 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0042214710000000001 2024-02-19T21:14:17.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.187 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:17.187 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.204630+0000", 2024-02-19T21:14:17.187 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000147804 2024-02-19T21:14:17.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.187 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.187 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.204778+0000", 2024-02-19T21:14:17.187 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020029409999999998 2024-02-19T21:14:17.187 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.187 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.187 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.188 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.206781+0000", 2024-02-19T21:14:17.188 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5789999999999999e-05 2024-02-19T21:14:17.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.188 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.188 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:17.188 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.206837+0000", 2024-02-19T21:14:17.188 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0491999999999998e-05 2024-02-19T21:14:17.188 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.188 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.188 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.188 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:37.206877+0000", 2024-02-19T21:14:17.188 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0068597529999999997 2024-02-19T21:14:17.188 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.189 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.189 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.189 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.189 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.189 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.47779.0:2 114.13 114:cbb9903e:::test_obj:head [write-same] snapc 0=[] ondisk+write+known_if_redirected e169)", 2024-02-19T21:14:17.189 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:05:49.194644+0000", 2024-02-19T21:14:17.189 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 507.92973396500003, 2024-02-19T21:14:17.189 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.061315005999999998, 2024-02-19T21:14:17.189 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.189 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:17.189 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.189 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.47779", 2024-02-19T21:14:17.189 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/4153636191", 2024-02-19T21:14:17.189 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2 2024-02-19T21:14:17.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.190 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.190 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.190 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194644+0000", 2024-02-19T21:14:17.190 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.190 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.190 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194644+0000", 2024-02-19T21:14:17.190 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:14:17.190 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.190 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.190 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.191 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194643+0000", 2024-02-19T21:14:17.191 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8570000000000007e-06 2024-02-19T21:14:17.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.191 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.191 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194653+0000", 2024-02-19T21:14:17.191 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6340000000000007e-06 2024-02-19T21:14:17.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.191 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.191 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194661+0000", 2024-02-19T21:14:17.191 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7300000000000002e-06 2024-02-19T21:14:17.191 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.191 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.191 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.192 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194667+0000", 2024-02-19T21:14:17.192 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0865999999999998e-05 2024-02-19T21:14:17.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.192 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.192 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194708+0000", 2024-02-19T21:14:17.192 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3140000000000003e-06 2024-02-19T21:14:17.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.192 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:17.192 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.194713+0000", 2024-02-19T21:14:17.192 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.020878119000000001 2024-02-19T21:14:17.192 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.192 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.193 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.193 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.215591+0000", 2024-02-19T21:14:17.193 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011031699999999999 2024-02-19T21:14:17.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.193 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.193 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.193 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.215702+0000", 2024-02-19T21:14:17.193 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000156618 2024-02-19T21:14:17.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.193 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.193 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 1,2", 2024-02-19T21:14:17.193 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.215858+0000", 2024-02-19T21:14:17.193 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0048208690000000002 2024-02-19T21:14:17.193 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.194 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.194 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:17.194 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.220679+0000", 2024-02-19T21:14:17.194 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.029705588000000002 2024-02-19T21:14:17.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.194 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.194 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.194 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.250385+0000", 2024-02-19T21:14:17.194 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0055153939999999999 2024-02-19T21:14:17.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.194 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.194 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.194 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.255900+0000", 2024-02-19T21:14:17.194 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1894999999999999e-05 2024-02-19T21:14:17.194 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.195 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.195 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:17.195 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.255932+0000", 2024-02-19T21:14:17.195 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7478e-05 2024-02-19T21:14:17.195 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.195 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.195 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.195 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:05:49.255959+0000", 2024-02-19T21:14:17.195 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.061315005999999998 2024-02-19T21:14:17.195 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.195 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.195 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.195 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.195 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.195 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.49298.0:1 162.6s0 162:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e237)", 2024-02-19T21:14:17.196 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:06:57.212999+0000", 2024-02-19T21:14:17.196 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 439.91137908100001, 2024-02-19T21:14:17.196 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.009269615, 2024-02-19T21:14:17.196 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.196 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:17.196 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.196 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.49298", 2024-02-19T21:14:17.196 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2330262446", 2024-02-19T21:14:17.196 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:14:17.196 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.196 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.196 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.197 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.197 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.212999+0000", 2024-02-19T21:14:17.197 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.197 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.197 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.197 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.197 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.212999+0000", 2024-02-19T21:14:17.197 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:14:17.197 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.197 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.197 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.197 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.212996+0000", 2024-02-19T21:14:17.197 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.08e-05 2024-02-19T21:14:17.197 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.198 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.198 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.198 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.213007+0000", 2024-02-19T21:14:17.198 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1018e-05 2024-02-19T21:14:17.198 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.198 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.198 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.198 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.213018+0000", 2024-02-19T21:14:17.198 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7819999999999994e-06 2024-02-19T21:14:17.198 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.198 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.198 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.198 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.213026+0000", 2024-02-19T21:14:17.198 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9827999999999996e-05 2024-02-19T21:14:17.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.199 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.199 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.213096+0000", 2024-02-19T21:14:17.199 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.399e-06 2024-02-19T21:14:17.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.199 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:17.199 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.213099+0000", 2024-02-19T21:14:17.199 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.006276073 2024-02-19T21:14:17.199 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.199 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.199 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.199 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.219375+0000", 2024-02-19T21:14:17.200 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012834399999999999 2024-02-19T21:14:17.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.200 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.200 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.200 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.219504+0000", 2024-02-19T21:14:17.200 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00046405399999999999 2024-02-19T21:14:17.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.200 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.200 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:14:17.200 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.219968+0000", 2024-02-19T21:14:17.200 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00075405700000000003 2024-02-19T21:14:17.200 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.200 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.201 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:14:17.201 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.220722+0000", 2024-02-19T21:14:17.201 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014760260000000001 2024-02-19T21:14:17.201 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.201 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.201 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:17.201 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.222198+0000", 2024-02-19T21:14:17.201 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1095999999999998e-05 2024-02-19T21:14:17.201 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.201 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.201 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.201 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.222269+0000", 2024-02-19T21:14:17.201 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.009269615 2024-02-19T21:14:17.201 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.201 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.202 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.202 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.202 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.202 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:2468 169.4 169:20000000::::head [pgnls start_epoch 259] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e259)", 2024-02-19T21:14:17.202 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:19.458185+0000", 2024-02-19T21:14:17.202 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 417.666193265, 2024-02-19T21:14:17.202 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0067992360000000002, 2024-02-19T21:14:17.202 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.202 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:14:17.202 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.202 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:14:17.202 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:14:17.202 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2468 2024-02-19T21:14:17.203 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.203 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.203 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.203 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.203 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458185+0000", 2024-02-19T21:14:17.203 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.203 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.204 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.204 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.204 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458185+0000", 2024-02-19T21:14:17.204 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999933 2024-02-19T21:14:17.204 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.204 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.204 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.204 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458179+0000", 2024-02-19T21:14:17.204 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1345e-05 2024-02-19T21:14:17.204 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.204 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.204 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.205 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458200+0000", 2024-02-19T21:14:17.205 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4885999999999999e-05 2024-02-19T21:14:17.205 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.205 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.206 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.206 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458225+0000", 2024-02-19T21:14:17.206 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0018e-05 2024-02-19T21:14:17.206 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.206 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.206 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.206 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458235+0000", 2024-02-19T21:14:17.207 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4640000000000003e-05 2024-02-19T21:14:17.207 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.207 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.207 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.207 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458320+0000", 2024-02-19T21:14:17.207 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6839999999999996e-06 2024-02-19T21:14:17.207 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.207 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.207 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:14:17.208 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.458327+0000", 2024-02-19T21:14:17.208 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0065004249999999998 2024-02-19T21:14:17.208 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.208 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.208 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.208 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.464828+0000", 2024-02-19T21:14:17.208 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8716e-05 2024-02-19T21:14:17.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.209 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.209 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.209 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.464866+0000", 2024-02-19T21:14:17.209 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011796899999999999 2024-02-19T21:14:17.209 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.209 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.209 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.209 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:19.464984+0000", 2024-02-19T21:14:17.210 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0067992360000000002 2024-02-19T21:14:17.210 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.210 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.210 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.210 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.210 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.210 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.2.18:69 61.3 e294/114 61:c0520479:test-rados-api-smithi163-83928-38::foo:head v 294'69, mlcod=230'66)", 2024-02-19T21:14:17.211 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:54.263507+0000", 2024-02-19T21:14:17.211 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 382.86087111900002, 2024-02-19T21:14:17.211 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0078694200000000002, 2024-02-19T21:14:17.211 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.211 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:17.211 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.211 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.212 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.212 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.263507+0000", 2024-02-19T21:14:17.212 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.212 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.212 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.212 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.212 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.263507+0000", 2024-02-19T21:14:17.213 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967296 2024-02-19T21:14:17.213 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.213 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.213 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.213 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.263507+0000", 2024-02-19T21:14:17.213 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003686576 2024-02-19T21:14:17.213 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.213 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.214 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.214 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.267194+0000", 2024-02-19T21:14:17.214 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4829e-05 2024-02-19T21:14:17.214 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.214 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.214 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.214 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.267208+0000", 2024-02-19T21:14:17.215 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0240000000000003e-06 2024-02-19T21:14:17.215 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.215 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.215 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.215 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.267217+0000", 2024-02-19T21:14:17.215 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1632000000000003e-05 2024-02-19T21:14:17.215 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.216 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.216 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.216 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.267269+0000", 2024-02-19T21:14:17.216 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0282999999999999e-05 2024-02-19T21:14:17.216 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.216 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.216 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.217 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.267289+0000", 2024-02-19T21:14:17.217 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0040479779999999998 2024-02-19T21:14:17.217 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.217 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.217 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:17.217 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.271337+0000", 2024-02-19T21:14:17.218 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9335999999999997e-05 2024-02-19T21:14:17.218 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.218 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.218 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.218 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.271377+0000", 2024-02-19T21:14:17.218 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0078694200000000002 2024-02-19T21:14:17.218 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.218 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.219 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.219 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.219 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.219 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:4348 191.8 191:10fb1d02:::199:head [delete] snapc 0=[] ondisk+write+ignore_cache+known_if_redirected e314)", 2024-02-19T21:14:17.219 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:08:14.258045+0000", 2024-02-19T21:14:17.219 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 362.86633345899997, 2024-02-19T21:14:17.219 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.039591451999999999, 2024-02-19T21:14:17.219 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.219 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:17.219 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.219 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:14:17.219 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:14:17.219 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 4348 2024-02-19T21:14:17.220 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.220 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.220 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.220 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.220 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258045+0000", 2024-02-19T21:14:17.220 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.220 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.220 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.220 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.220 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258045+0000", 2024-02-19T21:14:17.220 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:14:17.220 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.220 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.221 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.221 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258043+0000", 2024-02-19T21:14:17.221 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5199999999999997e-06 2024-02-19T21:14:17.221 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.221 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.221 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.221 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258049+0000", 2024-02-19T21:14:17.221 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6630000000000005e-06 2024-02-19T21:14:17.221 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.221 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.221 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.221 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258059+0000", 2024-02-19T21:14:17.221 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0889999999999999e-06 2024-02-19T21:14:17.221 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.222 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.222 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.222 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258064+0000", 2024-02-19T21:14:17.222 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5559999999999998e-05 2024-02-19T21:14:17.222 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.222 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.222 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.222 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258099+0000", 2024-02-19T21:14:17.222 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4418999999999994e-05 2024-02-19T21:14:17.222 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.222 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.222 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.222 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258164+0000", 2024-02-19T21:14:17.222 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1835000000000006e-05 2024-02-19T21:14:17.223 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.223 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.223 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,5", 2024-02-19T21:14:17.223 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258236+0000", 2024-02-19T21:14:17.223 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.021542664 2024-02-19T21:14:17.223 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.223 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.223 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:17.223 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.279778+0000", 2024-02-19T21:14:17.223 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025807920000000002 2024-02-19T21:14:17.223 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.223 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.223 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.223 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.282359+0000", 2024-02-19T21:14:17.223 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015223703 2024-02-19T21:14:17.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.224 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.224 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.224 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297583+0000", 2024-02-19T21:14:17.224 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0100999999999999e-05 2024-02-19T21:14:17.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.224 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.224 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:17.224 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297613+0000", 2024-02-19T21:14:17.224 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3626e-05 2024-02-19T21:14:17.224 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.224 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.224 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297636+0000", 2024-02-19T21:14:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.039591451999999999 2024-02-19T21:14:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:4349 191.8 191:10fdda93:::194:head [delete] snapc 0=[] ondisk+write+ignore_cache+known_if_redirected e314)", 2024-02-19T21:14:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:08:14.297868+0000", 2024-02-19T21:14:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 362.82651058599998, 2024-02-19T21:14:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026989016000000001, 2024-02-19T21:14:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.225 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:14:17.226 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:14:17.226 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 4349 2024-02-19T21:14:17.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.226 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.226 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.226 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.226 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297868+0000", 2024-02-19T21:14:17.226 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.226 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.226 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.226 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297868+0000", 2024-02-19T21:14:17.226 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:14:17.226 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.227 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.227 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.227 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297866+0000", 2024-02-19T21:14:17.227 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4129999999999997e-06 2024-02-19T21:14:17.227 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.227 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.227 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.227 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297871+0000", 2024-02-19T21:14:17.227 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0733999999999999e-05 2024-02-19T21:14:17.227 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.227 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.227 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.227 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297882+0000", 2024-02-19T21:14:17.227 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5829999999999997e-06 2024-02-19T21:14:17.228 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.228 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.228 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.228 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297888+0000", 2024-02-19T21:14:17.228 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4179e-05 2024-02-19T21:14:17.228 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.228 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.228 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.228 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297912+0000", 2024-02-19T21:14:17.228 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7067000000000002e-05 2024-02-19T21:14:17.228 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.228 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.228 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.229 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297959+0000", 2024-02-19T21:14:17.229 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9258999999999995e-05 2024-02-19T21:14:17.229 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.229 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.229 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,5", 2024-02-19T21:14:17.229 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.298028+0000", 2024-02-19T21:14:17.229 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00027415799999999999 2024-02-19T21:14:17.229 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.229 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.229 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:17.229 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.298302+0000", 2024-02-19T21:14:17.229 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018730600000000001 2024-02-19T21:14:17.229 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.229 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.230 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.230 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.298490+0000", 2024-02-19T21:14:17.230 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026316253000000001 2024-02-19T21:14:17.230 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.230 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.230 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.230 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.324806+0000", 2024-02-19T21:14:17.230 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9e-05 2024-02-19T21:14:17.230 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.230 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.230 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:17.230 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.324835+0000", 2024-02-19T21:14:17.230 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1923999999999999e-05 2024-02-19T21:14:17.231 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.231 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.231 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.231 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.324857+0000", 2024-02-19T21:14:17.231 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026989016000000001 2024-02-19T21:14:17.231 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.231 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.231 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.231 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.231 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.231 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22980.0:1 205.3s0 205:cbb9903e:::test_obj:head [omap-set-vals,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected e373)", 2024-02-19T21:14:17.231 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:09:13.274270+0000", 2024-02-19T21:14:17.231 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 303.85010796300003, 2024-02-19T21:14:17.232 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.581927844, 2024-02-19T21:14:17.232 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.232 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:14:17.232 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.232 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22980", 2024-02-19T21:14:17.232 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2118002540", 2024-02-19T21:14:17.232 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:14:17.232 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.232 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.232 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.232 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.232 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274270+0000", 2024-02-19T21:14:17.232 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.233 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.233 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.233 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.233 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274270+0000", 2024-02-19T21:14:17.233 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:14:17.233 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.233 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.233 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.233 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274267+0000", 2024-02-19T21:14:17.233 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1556000000000001e-05 2024-02-19T21:14:17.233 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.233 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.233 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.234 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274279+0000", 2024-02-19T21:14:17.234 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1654000000000001e-05 2024-02-19T21:14:17.234 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.234 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.234 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.234 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274291+0000", 2024-02-19T21:14:17.234 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7020000000000007e-06 2024-02-19T21:14:17.234 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.234 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.234 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.234 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274299+0000", 2024-02-19T21:14:17.235 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014805800000000001 2024-02-19T21:14:17.235 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.235 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.235 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.235 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274447+0000", 2024-02-19T21:14:17.235 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0749999999999999e-06 2024-02-19T21:14:17.235 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.235 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.235 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:17.235 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274452+0000", 2024-02-19T21:14:17.235 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5792793839999999 2024-02-19T21:14:17.235 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.235 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.235 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.236 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:14.853731+0000", 2024-02-19T21:14:17.236 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000125561 2024-02-19T21:14:17.236 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.236 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.236 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.236 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:14.853856+0000", 2024-02-19T21:14:17.236 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0023418330000000002 2024-02-19T21:14:17.236 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.236 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.237 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.237 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:14.856198+0000", 2024-02-19T21:14:17.237 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.581927844 2024-02-19T21:14:17.237 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.237 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.237 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.237 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.237 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.237 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:5600 208.10 208:08000000:%01:::head [pgnls start_epoch 393] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e393)", 2024-02-19T21:14:17.237 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:09:36.343158+0000", 2024-02-19T21:14:17.238 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 280.78122016700001, 2024-02-19T21:14:17.238 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.338287899, 2024-02-19T21:14:17.238 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.238 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:14:17.238 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.238 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:14:17.238 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:14:17.238 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 5600 2024-02-19T21:14:17.238 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.238 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.238 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.238 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.239 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343158+0000", 2024-02-19T21:14:17.239 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.239 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.239 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.239 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.239 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343158+0000", 2024-02-19T21:14:17.239 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:14:17.240 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.240 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.240 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.240 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343157+0000", 2024-02-19T21:14:17.240 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9830000000000001e-06 2024-02-19T21:14:17.240 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.240 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.240 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.240 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343163+0000", 2024-02-19T21:14:17.240 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4930000000000004e-06 2024-02-19T21:14:17.241 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.241 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.241 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.241 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343172+0000", 2024-02-19T21:14:17.243 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2700000000000004e-06 2024-02-19T21:14:17.243 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.243 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.243 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.243 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343178+0000", 2024-02-19T21:14:17.243 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3643000000000002e-05 2024-02-19T21:14:17.243 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.244 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.244 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.244 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343221+0000", 2024-02-19T21:14:17.244 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6779999999999997e-06 2024-02-19T21:14:17.244 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.244 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.244 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:17.244 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343227+0000", 2024-02-19T21:14:17.244 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.33795772099999999 2024-02-19T21:14:17.244 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.244 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.244 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.244 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.681185+0000", 2024-02-19T21:14:17.244 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0336000000000001e-05 2024-02-19T21:14:17.245 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.245 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.245 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.245 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.681235+0000", 2024-02-19T21:14:17.245 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00021113999999999999 2024-02-19T21:14:17.245 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.245 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.245 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.245 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.681446+0000", 2024-02-19T21:14:17.245 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.338287899 2024-02-19T21:14:17.245 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.245 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.245 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.245 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.246 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.246 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:1425 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94681540394880 gen 2] snapc 0=[] ondisk+write+known_if_redirected e433)", 2024-02-19T21:14:17.246 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:10:40.311762+0000", 2024-02-19T21:14:17.246 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 216.81261600600001, 2024-02-19T21:14:17.246 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.012232867, 2024-02-19T21:14:17.246 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.246 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:14:17.246 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.246 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:14:17.246 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:14:17.246 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1425 2024-02-19T21:14:17.246 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.246 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.246 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.247 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.247 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311762+0000", 2024-02-19T21:14:17.247 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.247 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.247 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.247 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.247 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311762+0000", 2024-02-19T21:14:17.247 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:14:17.247 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.247 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.247 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.247 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311759+0000", 2024-02-19T21:14:17.247 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0362e-05 2024-02-19T21:14:17.247 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.248 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.248 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.248 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311769+0000", 2024-02-19T21:14:17.248 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7405999999999999e-05 2024-02-19T21:14:17.248 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.248 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.248 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.248 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311787+0000", 2024-02-19T21:14:17.248 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7710000000000006e-06 2024-02-19T21:14:17.248 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.248 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.248 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.248 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311795+0000", 2024-02-19T21:14:17.249 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8292999999999999e-05 2024-02-19T21:14:17.249 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.249 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.249 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.249 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311853+0000", 2024-02-19T21:14:17.249 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0146000000000001e-05 2024-02-19T21:14:17.249 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.249 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.249 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:17.249 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311863+0000", 2024-02-19T21:14:17.249 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.004993928 2024-02-19T21:14:17.250 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.250 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.250 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.250 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.316857+0000", 2024-02-19T21:14:17.250 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00033791000000000002 2024-02-19T21:14:17.250 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.251 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.251 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for degraded object", 2024-02-19T21:14:17.251 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.317195+0000", 2024-02-19T21:14:17.251 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0054299880000000002 2024-02-19T21:14:17.251 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.251 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.251 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.252 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.322625+0000", 2024-02-19T21:14:17.252 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5621000000000002e-05 2024-02-19T21:14:17.252 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.252 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.252 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.252 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.322680+0000", 2024-02-19T21:14:17.252 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0013148350000000001 2024-02-19T21:14:17.253 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.253 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.253 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.253 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.323995+0000", 2024-02-19T21:14:17.253 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.012232867 2024-02-19T21:14:17.253 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.254 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.254 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.254 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.254 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.254 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:6806 218.e 218:769c1872:test-rados-api-smithi163-83928-74::chunk1:head [writefull 0~10] snapc 0=[] ondisk+write+known_if_redirected e464)", 2024-02-19T21:14:17.254 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:11:32.421792+0000", 2024-02-19T21:14:17.254 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 164.70258644899999, 2024-02-19T21:14:17.254 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.17724985600000001, 2024-02-19T21:14:17.254 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.255 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:17.255 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.255 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:14:17.255 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:14:17.255 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 6806 2024-02-19T21:14:17.255 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.255 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.255 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.255 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.255 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421792+0000", 2024-02-19T21:14:17.255 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.256 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.256 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.256 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.256 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421792+0000", 2024-02-19T21:14:17.256 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:14:17.256 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.256 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.256 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.256 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421790+0000", 2024-02-19T21:14:17.256 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7069999999999998e-06 2024-02-19T21:14:17.256 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.257 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.257 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.257 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421798+0000", 2024-02-19T21:14:17.257 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4839e-05 2024-02-19T21:14:17.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.257 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.257 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.257 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421813+0000", 2024-02-19T21:14:17.257 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1140000000000006e-06 2024-02-19T21:14:17.257 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.257 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.257 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.258 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421821+0000", 2024-02-19T21:14:17.258 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013141300000000001 2024-02-19T21:14:17.258 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.258 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.258 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.258 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421953+0000", 2024-02-19T21:14:17.258 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7780000000000004e-06 2024-02-19T21:14:17.258 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.259 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.259 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:14:17.259 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421959+0000", 2024-02-19T21:14:17.259 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.172664812 2024-02-19T21:14:17.259 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.259 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.259 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.260 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.594623+0000", 2024-02-19T21:14:17.260 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000130576 2024-02-19T21:14:17.260 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.260 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.260 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.260 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.594754+0000", 2024-02-19T21:14:17.260 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015927700000000001 2024-02-19T21:14:17.260 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.260 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.260 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,5", 2024-02-19T21:14:17.260 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.594913+0000", 2024-02-19T21:14:17.261 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0017018960000000001 2024-02-19T21:14:17.261 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.261 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.261 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.261 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.596615+0000", 2024-02-19T21:14:17.261 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2569e-05 2024-02-19T21:14:17.261 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.261 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.261 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:17.261 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.596638+0000", 2024-02-19T21:14:17.262 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0023243410000000002 2024-02-19T21:14:17.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.262 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.262 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.262 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.598962+0000", 2024-02-19T21:14:17.262 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6718000000000003e-05 2024-02-19T21:14:17.262 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.262 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.262 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:17.262 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.598999+0000", 2024-02-19T21:14:17.262 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3105999999999997e-05 2024-02-19T21:14:17.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.263 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.263 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.263 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.599042+0000", 2024-02-19T21:14:17.263 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.17724985600000001 2024-02-19T21:14:17.263 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.263 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.263 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.263 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.264 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.264 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:6815 61.10 61:0db63738:test-rados-api-smithi163-83928-74::foo:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e466)", 2024-02-19T21:14:17.264 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:11:34.300858+0000", 2024-02-19T21:14:17.264 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 162.82352029699999, 2024-02-19T21:14:17.264 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0076779409999999998, 2024-02-19T21:14:17.264 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.264 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:14:17.264 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.264 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:14:17.264 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:14:17.265 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 6815 2024-02-19T21:14:17.265 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.265 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.265 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.265 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.265 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300858+0000", 2024-02-19T21:14:17.265 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.265 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.265 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.265 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.266 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300858+0000", 2024-02-19T21:14:17.266 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:14:17.266 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.266 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.266 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.266 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300857+0000", 2024-02-19T21:14:17.266 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1149999999999999e-06 2024-02-19T21:14:17.266 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.266 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.266 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.267 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300863+0000", 2024-02-19T21:14:17.267 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1065e-05 2024-02-19T21:14:17.267 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.267 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.267 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.267 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300874+0000", 2024-02-19T21:14:17.267 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3249999999999998e-06 2024-02-19T21:14:17.267 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.267 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.268 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.268 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300879+0000", 2024-02-19T21:14:17.268 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3939999999999997e-05 2024-02-19T21:14:17.268 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.268 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.268 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.268 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300913+0000", 2024-02-19T21:14:17.268 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9679000000000002e-05 2024-02-19T21:14:17.268 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.269 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.269 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.269 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300973+0000", 2024-02-19T21:14:17.269 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0067811130000000001 2024-02-19T21:14:17.269 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.269 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.269 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-02-19T21:14:17.269 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.307754+0000", 2024-02-19T21:14:17.269 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00038248399999999999 2024-02-19T21:14:17.270 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.270 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.270 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:14:17.270 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.308136+0000", 2024-02-19T21:14:17.270 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00024405100000000001 2024-02-19T21:14:17.270 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.270 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.270 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.270 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.308381+0000", 2024-02-19T21:14:17.271 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4700999999999997e-05 2024-02-19T21:14:17.271 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.271 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.271 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:14:17.271 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.308435+0000", 2024-02-19T21:14:17.271 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4357999999999997e-05 2024-02-19T21:14:17.271 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.271 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.271 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:14:17.271 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.308490+0000", 2024-02-19T21:14:17.272 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6477999999999998e-05 2024-02-19T21:14:17.272 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.272 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.272 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.272 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.308536+0000", 2024-02-19T21:14:17.272 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0076779409999999998 2024-02-19T21:14:17.272 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.272 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.272 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.273 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.273 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.273 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:7174 221.19 221:985d3dde:test-rados-api-smithi163-83928-80::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e503)", 2024-02-19T21:14:17.273 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:12:19.352715+0000", 2024-02-19T21:14:17.273 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 117.771662971, 2024-02-19T21:14:17.273 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010904531, 2024-02-19T21:14:17.273 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:14:17.273 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:14:17.273 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:14:17.274 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:14:17.274 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:14:17.274 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 7174 2024-02-19T21:14:17.274 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.274 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:14:17.274 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.274 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:14:17.274 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352715+0000", 2024-02-19T21:14:17.274 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:14:17.275 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.275 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.275 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:14:17.275 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352715+0000", 2024-02-19T21:14:17.275 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:14:17.275 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.275 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.276 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:14:17.276 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352715+0000", 2024-02-19T21:14:17.276 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0860000000000002e-06 2024-02-19T21:14:17.276 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.276 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.276 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:14:17.276 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352718+0000", 2024-02-19T21:14:17.276 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0129999999999999e-06 2024-02-19T21:14:17.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.277 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.277 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:14:17.277 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352724+0000", 2024-02-19T21:14:17.277 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9620000000000004e-06 2024-02-19T21:14:17.277 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.277 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.277 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:14:17.278 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352728+0000", 2024-02-19T21:14:17.278 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.8980999999999999e-05 2024-02-19T21:14:17.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.278 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.278 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:14:17.278 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352757+0000", 2024-02-19T21:14:17.278 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3026000000000003e-05 2024-02-19T21:14:17.278 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.278 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.279 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:14:17.279 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352820+0000", 2024-02-19T21:14:17.279 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010800346000000001 2024-02-19T21:14:17.279 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:14:17.279 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:14:17.279 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:14:17.279 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.363620+0000", 2024-02-19T21:14:17.279 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010904531 2024-02-19T21:14:17.280 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.280 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.280 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.280 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:14:17.280 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:14:17.280 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:14:17.281 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:17.312 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:17.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:17.355+0000 7f4014ff9700 1 -- 172.21.15.163:0/3084512438 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v621) v1 ==== 101+0+0 (unknown 3423260316 0 0) 0x7f400c047e40 con 0x7f40180ff680 2024-02-19T21:14:17.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:17.358+0000 7f401e081700 10 librados: watch_flush enter 2024-02-19T21:14:17.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:17.358+0000 7f401e081700 10 librados: watch_flush exit 2024-02-19T21:14:17.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:17.358+0000 7f401e081700 20 client.23577.objecter shutdown clearing up homeless session... 2024-02-19T21:14:17.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:17.358+0000 7f401e081700 10 client.23577.objecter successfully canceled tick 2024-02-19T21:14:17.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:17.362+0000 7f401e081700 1 -- 172.21.15.163:0/3084512438 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4000063d10 legacy=0x7f40000661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:17.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:17.362+0000 7f401e081700 1 -- 172.21.15.163:0/3084512438 >> v1:172.21.15.163:6789/0 conn(0x7f40180ff680 legacy=0x7f40180f9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:17.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:17.362+0000 7f401e081700 1 -- 172.21.15.163:0/3084512438 shutdown_connections 2024-02-19T21:14:17.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:17.362+0000 7f401e081700 1 -- 172.21.15.163:0/3084512438 >> 172.21.15.163:0/3084512438 conn(0x7f401806fdc0 msgr2=0x7f40180f4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:17.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:17.362+0000 7f401e081700 1 -- 172.21.15.163:0/3084512438 shutdown_connections 2024-02-19T21:14:17.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:17.362+0000 7f401e081700 1 -- 172.21.15.163:0/3084512438 wait complete. 2024-02-19T21:14:17.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:17.362+0000 7f401e081700 1 librados: shutdown 2024-02-19T21:14:17.363 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:14:17.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:15.933457+0000 osd.5 (osd.5) 246 : cluster [DBG] 4.15 scrub starts 2024-02-19T21:14:17.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:15.936905+0000 osd.5 (osd.5) 247 : cluster [DBG] 4.15 scrub ok 2024-02-19T21:14:17.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:16.421927+0000 mgr.y (mgr.14550) 535 : audit [DBG] from='mon.0 v1:172.21.15.163:0/4181310973' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "237.3"}]: dispatch 2024-02-19T21:14:17.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:16.596526+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? v1:172.21.15.163:0/3084512438' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:17.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:17.350363+0000 mon.a (mon.0) 3154 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:14:17.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:17.356051+0000 mon.a (mon.0) 3155 : audit [INF] from='client.? v1:172.21.15.163:0/3084512438' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:14:17.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:17.356093+0000 mon.a (mon.0) 3156 : cluster [DBG] osdmap e621: 8 total, 8 up, 7 in 2024-02-19T21:14:17.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:17.357711+0000 osd.7 (osd.7) 279 : cluster [DBG] 2.20 starting backfill to osd.6 from (0'0,0'0] MAX to 397'2 2024-02-19T21:14:17.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:15.933457+0000 osd.5 (osd.5) 246 : cluster [DBG] 4.15 scrub starts 2024-02-19T21:14:17.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:15.936905+0000 osd.5 (osd.5) 247 : cluster [DBG] 4.15 scrub ok 2024-02-19T21:14:17.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:16.421927+0000 mgr.y (mgr.14550) 535 : audit [DBG] from='mon.0 v1:172.21.15.163:0/4181310973' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "237.3"}]: dispatch 2024-02-19T21:14:17.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:16.596526+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? v1:172.21.15.163:0/3084512438' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:17.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:17.350363+0000 mon.a (mon.0) 3154 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:14:17.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:17.356051+0000 mon.a (mon.0) 3155 : audit [INF] from='client.? v1:172.21.15.163:0/3084512438' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:14:17.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:17.356093+0000 mon.a (mon.0) 3156 : cluster [DBG] osdmap e621: 8 total, 8 up, 7 in 2024-02-19T21:14:17.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:17.357711+0000 osd.7 (osd.7) 279 : cluster [DBG] 2.20 starting backfill to osd.6 from (0'0,0'0] MAX to 397'2 2024-02-19T21:14:17.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:15.933457+0000 osd.5 (osd.5) 246 : cluster [DBG] 4.15 scrub starts 2024-02-19T21:14:17.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:15.936905+0000 osd.5 (osd.5) 247 : cluster [DBG] 4.15 scrub ok 2024-02-19T21:14:17.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:16.421927+0000 mgr.y (mgr.14550) 535 : audit [DBG] from='mon.0 v1:172.21.15.163:0/4181310973' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "237.3"}]: dispatch 2024-02-19T21:14:17.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:16.596526+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? v1:172.21.15.163:0/3084512438' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:17.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:17.350363+0000 mon.a (mon.0) 3154 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:14:17.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:17.356051+0000 mon.a (mon.0) 3155 : audit [INF] from='client.? v1:172.21.15.163:0/3084512438' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:14:17.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:17.356093+0000 mon.a (mon.0) 3156 : cluster [DBG] osdmap e621: 8 total, 8 up, 7 in 2024-02-19T21:14:17.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:17.357711+0000 osd.7 (osd.7) 279 : cluster [DBG] 2.20 starting backfill to osd.6 from (0'0,0'0] MAX to 397'2 2024-02-19T21:14:17.881 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-19T21:14:18.071 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:14:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:14:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:14:18.683 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:17.284002+0000 mgr.y (mgr.14550) 536 : cluster [DBG] pgmap v930: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:14:18.683 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:17.284002+0000 mgr.y (mgr.14550) 536 : cluster [DBG] pgmap v930: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:14:18.692 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:18.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:18.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:18.796 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:18.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:17.284002+0000 mgr.y (mgr.14550) 536 : cluster [DBG] pgmap v930: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:14:18.822 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:18.946 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:18.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:19.050 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:19.078 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:19.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.204+0000 7fdfec1f6700 1 -- 172.21.15.163:0/452286978 >> v1:172.21.15.163:6790/0 conn(0x7fdfe40ff670 legacy=0x7fdfe4101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:19.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.204+0000 7fdfec1f6700 1 -- 172.21.15.163:0/452286978 shutdown_connections 2024-02-19T21:14:19.220 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.204+0000 7fdfec1f6700 1 -- 172.21.15.163:0/452286978 >> 172.21.15.163:0/452286978 conn(0x7fdfe40f2470 msgr2=0x7fdfe40f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.204+0000 7fdfec1f6700 1 -- 172.21.15.163:0/452286978 shutdown_connections 2024-02-19T21:14:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.204+0000 7fdfec1f6700 1 -- 172.21.15.163:0/452286978 wait complete. 2024-02-19T21:14:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.205+0000 7fdfec1f6700 1 librados: starting msgr at 2024-02-19T21:14:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.205+0000 7fdfec1f6700 1 librados: starting objecter 2024-02-19T21:14:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.205+0000 7fdfec1f6700 1 Processor -- start 2024-02-19T21:14:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.205+0000 7fdfec1f6700 1 -- start start 2024-02-19T21:14:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.205+0000 7fdfec1f6700 1 librados: setting wanted keys 2024-02-19T21:14:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.205+0000 7fdfec1f6700 1 librados: calling monclient init 2024-02-19T21:14:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.205+0000 7fdfec1f6700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdfe4116190 con 0x7fdfe40fc870 2024-02-19T21:14:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.205+0000 7fdfec1f6700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdfe4117370 con 0x7fdfe40ff670 2024-02-19T21:14:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.205+0000 7fdfec1f6700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdfe4118570 con 0x7fdfe41031b0 2024-02-19T21:14:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.205+0000 7fdfe9f92700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fdfe40fc870 0x7fdfe40f8850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33944/0 (socket says 172.21.15.163:33944) 2024-02-19T21:14:19.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.205+0000 7fdfe9f92700 1 -- 172.21.15.163:0/2106042762 learned_addr learned my addr 172.21.15.163:0/2106042762 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:19.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.205+0000 7fdfd6ffd700 10 client.?.objecter ms_handle_connect 0x7fdfe40fc870 2024-02-19T21:14:19.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.205+0000 7fdfd6ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:19.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171811926 0 0) 0x7fdfe4116190 con 0x7fdfe40fc870 2024-02-19T21:14:19.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfd00038e0 con 0x7fdfe40fc870 2024-02-19T21:14:19.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 10 client.?.objecter ms_handle_connect 0x7fdfe40ff670 2024-02-19T21:14:19.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:19.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 10 client.?.objecter ms_handle_connect 0x7fdfe41031b0 2024-02-19T21:14:19.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:19.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810062123 0 0) 0x7fdfe4117370 con 0x7fdfe40ff670 2024-02-19T21:14:19.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe4116190 con 0x7fdfe40ff670 2024-02-19T21:14:19.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275942262 0 0) 0x7fdfe4118570 con 0x7fdfe41031b0 2024-02-19T21:14:19.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe4117370 con 0x7fdfe41031b0 2024-02-19T21:14:19.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2885053386 0 0) 0x7fdfd00038e0 con 0x7fdfe40fc870 2024-02-19T21:14:19.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfe4118570 con 0x7fdfe40fc870 2024-02-19T21:14:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdfd8001940 con 0x7fdfe40fc870 2024-02-19T21:14:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1700527798 0 0) 0x7fdfe4118570 con 0x7fdfe40fc870 2024-02-19T21:14:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 >> v1:172.21.15.163:6790/0 conn(0x7fdfe41031b0 legacy=0x7fdfe4114850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 >> v1:172.21.15.193:6789/0 conn(0x7fdfe40ff670 legacy=0x7fdfe40f8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe4119770 con 0x7fdfe40fc870 2024-02-19T21:14:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe4117560 con 0x7fdfe40fc870 2024-02-19T21:14:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfec1f6700 10 client.23583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe4117a50 con 0x7fdfe40fc870 2024-02-19T21:14:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfec1f6700 1 librados: init done 2024-02-19T21:14:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fdfd8002460 con 0x7fdfe40fc870 2024-02-19T21:14:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.206+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdfd8005b50 con 0x7fdfe40fc870 2024-02-19T21:14:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.207+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fdfc8004f30 con 0x7fdfe40fc870 2024-02-19T21:14:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.207+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fdfd8002c00 con 0x7fdfe40fc870 2024-02-19T21:14:19.223 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(622..622 src has 434..622) v4 ==== 8508+0+0 (unknown 944362285 0 0) 0x7fdfd8079030 con 0x7fdfe40fc870 2024-02-19T21:14:19.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 osd_map(622..622 src has 434..622) v4 2024-02-19T21:14:19.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfd6ffd700 3 client.23583.objecter handle_osd_map got epochs [622,622] > 0 2024-02-19T21:14:19.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfd6ffd700 3 client.23583.objecter handle_osd_map decoding full epoch 622 2024-02-19T21:14:19.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfd6ffd700 20 client.23583.objecter dump_active .. 0 homeless 2024-02-19T21:14:19.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v622) v1 ==== 54+0+16 (unknown 1877290550 0 2420000192) 0x7fdfd807b420 con 0x7fdfe40fc870 2024-02-19T21:14:19.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfd6ffd700 10 client.23583.objecter ms_handle_connect 0x7fdfd0063830 2024-02-19T21:14:19.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:19.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 1 2024-02-19T21:14:19.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc80051d0 osd=0 3 2024-02-19T21:14:19.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc80051d0 osd=0 3 2024-02-19T21:14:19.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc80051d0 osd=0 3 2024-02-19T21:14:19.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 1 2024-02-19T21:14:19.224 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc80051d0 osd=0 3 2024-02-19T21:14:19.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 0 1 2024-02-19T21:14:19.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc80051d0 osd=0 4 2024-02-19T21:14:19.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfec1f6700 10 client.23583.objecter _send_command 1 2024-02-19T21:14:19.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.208+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdfc8004980 con 0x7fdfc8004560 2024-02-19T21:14:19.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.209+0000 7fdfd6ffd700 10 client.23583.objecter ms_handle_connect 0x7fdfc8004560 2024-02-19T21:14:19.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.209+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdfc8004980 con 0x7fdfc8004560 2024-02-19T21:14:19.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.209+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:14:19.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.209+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 1 = system:0 2024-02-19T21:14:19.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.209+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc80051d0 osd=0 4 2024-02-19T21:14:19.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.209+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 0 1 2024-02-19T21:14:19.226 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:19.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:19.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:19.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.230+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:19.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.230+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 2 2024-02-19T21:14:19.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.230+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc80051d0 osd=0 3 2024-02-19T21:14:19.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.230+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc80051d0 osd=0 3 2024-02-19T21:14:19.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.230+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc80051d0 osd=0 3 2024-02-19T21:14:19.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.230+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 2 2024-02-19T21:14:19.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.230+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc80051d0 osd=0 3 2024-02-19T21:14:19.230 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.230+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 0 2 2024-02-19T21:14:19.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.230+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc80051d0 osd=0 4 2024-02-19T21:14:19.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.230+0000 7fdfec1f6700 10 client.23583.objecter _send_command 2 2024-02-19T21:14:19.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.230+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdfc8004d60 con 0x7fdfc8004560 2024-02-19T21:14:19.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fdfc8004d60 con 0x7fdfc8004560 2024-02-19T21:14:19.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:19.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc80051d0 osd=0 4 2024-02-19T21:14:19.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 0 2 2024-02-19T21:14:19.232 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:14:19.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:19.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 3 2024-02-19T21:14:19.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8004010 osd=1 3 2024-02-19T21:14:19.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8004010 osd=1 3 2024-02-19T21:14:19.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8004010 osd=1 3 2024-02-19T21:14:19.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 3 2024-02-19T21:14:19.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc8004010 osd=1 3 2024-02-19T21:14:19.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 1 3 2024-02-19T21:14:19.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8004010 osd=1 4 2024-02-19T21:14:19.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfec1f6700 10 client.23583.objecter _send_command 3 2024-02-19T21:14:19.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdfc800e240 con 0x7fdfc8004180 2024-02-19T21:14:19.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.232+0000 7fdfd6ffd700 10 client.23583.objecter ms_handle_connect 0x7fdfc8004180 2024-02-19T21:14:19.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.233+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdfc800e240 con 0x7fdfc8004180 2024-02-19T21:14:19.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.233+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:14:19.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.233+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 3 = system:0 2024-02-19T21:14:19.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.233+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc8004010 osd=1 4 2024-02-19T21:14:19.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.233+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 1 3 2024-02-19T21:14:19.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.249+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:19.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.249+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 4 2024-02-19T21:14:19.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.249+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8004010 osd=1 3 2024-02-19T21:14:19.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.249+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8004010 osd=1 3 2024-02-19T21:14:19.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.249+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8004010 osd=1 3 2024-02-19T21:14:19.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.249+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 4 2024-02-19T21:14:19.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.249+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc8004010 osd=1 3 2024-02-19T21:14:19.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.249+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 1 4 2024-02-19T21:14:19.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.249+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8004010 osd=1 4 2024-02-19T21:14:19.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.249+0000 7fdfec1f6700 10 client.23583.objecter _send_command 4 2024-02-19T21:14:19.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.249+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdfc8001880 con 0x7fdfc8004180 2024-02-19T21:14:19.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.251+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fdfc8001880 con 0x7fdfc8004180 2024-02-19T21:14:19.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.251+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:19.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.251+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.251+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc8004010 osd=1 4 2024-02-19T21:14:19.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.251+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 1 4 2024-02-19T21:14:19.251 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:14:19.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.252+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:19.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.252+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 5 2024-02-19T21:14:19.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.252+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8003660 osd=2 3 2024-02-19T21:14:19.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.252+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8003660 osd=2 3 2024-02-19T21:14:19.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.252+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8003660 osd=2 3 2024-02-19T21:14:19.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.252+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 5 2024-02-19T21:14:19.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.252+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc8003660 osd=2 3 2024-02-19T21:14:19.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.252+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 2 5 2024-02-19T21:14:19.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.252+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8003660 osd=2 4 2024-02-19T21:14:19.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.252+0000 7fdfec1f6700 10 client.23583.objecter _send_command 5 2024-02-19T21:14:19.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.252+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdfc8003ea0 con 0x7fdfc800f530 2024-02-19T21:14:19.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.252+0000 7fdfd6ffd700 10 client.23583.objecter ms_handle_connect 0x7fdfc800f530 2024-02-19T21:14:19.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.253+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdfc8003ea0 con 0x7fdfc800f530 2024-02-19T21:14:19.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.253+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:14:19.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.253+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 5 = system:0 2024-02-19T21:14:19.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.253+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc8003660 osd=2 4 2024-02-19T21:14:19.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.253+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 2 5 2024-02-19T21:14:19.257 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:19.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.268+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:19.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.268+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 6 2024-02-19T21:14:19.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.268+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8003660 osd=2 3 2024-02-19T21:14:19.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.268+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8003660 osd=2 3 2024-02-19T21:14:19.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.268+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8003660 osd=2 3 2024-02-19T21:14:19.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.268+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 6 2024-02-19T21:14:19.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.268+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc8003660 osd=2 3 2024-02-19T21:14:19.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.268+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 2 6 2024-02-19T21:14:19.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.268+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8003660 osd=2 4 2024-02-19T21:14:19.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.268+0000 7fdfec1f6700 10 client.23583.objecter _send_command 6 2024-02-19T21:14:19.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.268+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdfc8010970 con 0x7fdfc800f530 2024-02-19T21:14:19.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.269+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fdfc8010970 con 0x7fdfc800f530 2024-02-19T21:14:19.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.269+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:19.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.269+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.269+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc8003660 osd=2 4 2024-02-19T21:14:19.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.269+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 2 6 2024-02-19T21:14:19.269 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:14:19.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.270+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:19.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.270+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 7 2024-02-19T21:14:19.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.270+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8003410 osd=3 3 2024-02-19T21:14:19.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.270+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8003410 osd=3 3 2024-02-19T21:14:19.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.270+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8003410 osd=3 3 2024-02-19T21:14:19.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.270+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 7 2024-02-19T21:14:19.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.270+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc8003410 osd=3 3 2024-02-19T21:14:19.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.270+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 3 7 2024-02-19T21:14:19.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.270+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8003410 osd=3 4 2024-02-19T21:14:19.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.270+0000 7fdfec1f6700 10 client.23583.objecter _send_command 7 2024-02-19T21:14:19.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.270+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdfc800a2d0 con 0x7fdfc8008e10 2024-02-19T21:14:19.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.270+0000 7fdfd6ffd700 10 client.23583.objecter ms_handle_connect 0x7fdfc8008e10 2024-02-19T21:14:19.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.271+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdfc800a2d0 con 0x7fdfc8008e10 2024-02-19T21:14:19.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.271+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:14:19.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.271+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 7 = system:0 2024-02-19T21:14:19.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.271+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc8003410 osd=3 4 2024-02-19T21:14:19.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.271+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 3 7 2024-02-19T21:14:19.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.287+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:19.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.287+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 8 2024-02-19T21:14:19.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.287+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8003410 osd=3 3 2024-02-19T21:14:19.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.287+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8003410 osd=3 3 2024-02-19T21:14:19.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.287+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8003410 osd=3 3 2024-02-19T21:14:19.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.287+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 8 2024-02-19T21:14:19.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.287+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc8003410 osd=3 3 2024-02-19T21:14:19.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.287+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 3 8 2024-02-19T21:14:19.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.287+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8003410 osd=3 4 2024-02-19T21:14:19.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.287+0000 7fdfec1f6700 10 client.23583.objecter _send_command 8 2024-02-19T21:14:19.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.287+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdfc800a730 con 0x7fdfc8008e10 2024-02-19T21:14:19.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.289+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fdfc800a730 con 0x7fdfc8008e10 2024-02-19T21:14:19.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:19.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc8003410 osd=3 4 2024-02-19T21:14:19.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 3 8 2024-02-19T21:14:19.290 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:14:19.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:19.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 9 2024-02-19T21:14:19.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc800a730 osd=4 3 2024-02-19T21:14:19.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc800a730 osd=4 3 2024-02-19T21:14:19.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc800a730 osd=4 3 2024-02-19T21:14:19.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 9 2024-02-19T21:14:19.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc800a730 osd=4 3 2024-02-19T21:14:19.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 4 9 2024-02-19T21:14:19.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc800a730 osd=4 4 2024-02-19T21:14:19.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfec1f6700 10 client.23583.objecter _send_command 9 2024-02-19T21:14:19.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdfc801e740 con 0x7fdfc801ac30 2024-02-19T21:14:19.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.290+0000 7fdfd6ffd700 10 client.23583.objecter ms_handle_connect 0x7fdfc801ac30 2024-02-19T21:14:19.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.291+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdfc801e740 con 0x7fdfc801ac30 2024-02-19T21:14:19.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.291+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:14:19.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.291+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 9 = system:0 2024-02-19T21:14:19.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.291+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc800a730 osd=4 4 2024-02-19T21:14:19.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.291+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 4 9 2024-02-19T21:14:19.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.307+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:19.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.307+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 10 2024-02-19T21:14:19.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.307+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc800a730 osd=4 3 2024-02-19T21:14:19.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.307+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc800a730 osd=4 3 2024-02-19T21:14:19.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.307+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc800a730 osd=4 3 2024-02-19T21:14:19.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.307+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 10 2024-02-19T21:14:19.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.307+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc800a730 osd=4 3 2024-02-19T21:14:19.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.307+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 4 10 2024-02-19T21:14:19.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.307+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc800a730 osd=4 4 2024-02-19T21:14:19.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.307+0000 7fdfec1f6700 10 client.23583.objecter _send_command 10 2024-02-19T21:14:19.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.307+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdfc801fef0 con 0x7fdfc801ac30 2024-02-19T21:14:19.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fdfc801fef0 con 0x7fdfc801ac30 2024-02-19T21:14:19.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:19.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc800a730 osd=4 4 2024-02-19T21:14:19.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 4 10 2024-02-19T21:14:19.309 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:14:19.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:19.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 11 2024-02-19T21:14:19.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc801fef0 osd=5 3 2024-02-19T21:14:19.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc801fef0 osd=5 3 2024-02-19T21:14:19.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc801fef0 osd=5 3 2024-02-19T21:14:19.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 11 2024-02-19T21:14:19.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc801fef0 osd=5 3 2024-02-19T21:14:19.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 5 11 2024-02-19T21:14:19.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc801fef0 osd=5 4 2024-02-19T21:14:19.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfec1f6700 10 client.23583.objecter _send_command 11 2024-02-19T21:14:19.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.309+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdfc8020b20 con 0x7fdfc8020600 2024-02-19T21:14:19.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.310+0000 7fdfd6ffd700 10 client.23583.objecter ms_handle_connect 0x7fdfc8020600 2024-02-19T21:14:19.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.311+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdfc8020b20 con 0x7fdfc8020600 2024-02-19T21:14:19.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.311+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:14:19.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.311+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 11 = system:0 2024-02-19T21:14:19.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.311+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc801fef0 osd=5 4 2024-02-19T21:14:19.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.311+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 5 11 2024-02-19T21:14:19.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.325+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:19.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.325+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 12 2024-02-19T21:14:19.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.325+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc801fef0 osd=5 3 2024-02-19T21:14:19.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.325+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc801fef0 osd=5 3 2024-02-19T21:14:19.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.325+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc801fef0 osd=5 3 2024-02-19T21:14:19.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.325+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 12 2024-02-19T21:14:19.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.325+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc801fef0 osd=5 3 2024-02-19T21:14:19.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.325+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 5 12 2024-02-19T21:14:19.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.325+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc801fef0 osd=5 4 2024-02-19T21:14:19.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.325+0000 7fdfec1f6700 10 client.23583.objecter _send_command 12 2024-02-19T21:14:19.325 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.325+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdfc8020b20 con 0x7fdfc8020600 2024-02-19T21:14:19.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fdfc8020b20 con 0x7fdfc8020600 2024-02-19T21:14:19.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:19.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc801fef0 osd=5 4 2024-02-19T21:14:19.326 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 5 12 2024-02-19T21:14:19.327 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:14:19.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:19.327 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 13 2024-02-19T21:14:19.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8020b20 osd=6 3 2024-02-19T21:14:19.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8020b20 osd=6 3 2024-02-19T21:14:19.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8020b20 osd=6 3 2024-02-19T21:14:19.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 13 2024-02-19T21:14:19.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc8020b20 osd=6 3 2024-02-19T21:14:19.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 6 13 2024-02-19T21:14:19.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8020b20 osd=6 4 2024-02-19T21:14:19.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfec1f6700 10 client.23583.objecter _send_command 13 2024-02-19T21:14:19.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.327+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdfc80270f0 con 0x7fdfc80235e0 2024-02-19T21:14:19.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.328+0000 7fdfd6ffd700 10 client.23583.objecter ms_handle_connect 0x7fdfc80235e0 2024-02-19T21:14:19.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.328+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdfc80270f0 con 0x7fdfc80235e0 2024-02-19T21:14:19.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.328+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:14:19.328 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.328+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 13 = system:0 2024-02-19T21:14:19.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.328+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc8020b20 osd=6 4 2024-02-19T21:14:19.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.328+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 6 13 2024-02-19T21:14:19.331 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:19.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.343+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:19.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.343+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 14 2024-02-19T21:14:19.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.343+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8020b20 osd=6 3 2024-02-19T21:14:19.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.343+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8020b20 osd=6 3 2024-02-19T21:14:19.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.343+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc8020b20 osd=6 3 2024-02-19T21:14:19.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.343+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 14 2024-02-19T21:14:19.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.343+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc8020b20 osd=6 3 2024-02-19T21:14:19.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.343+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 6 14 2024-02-19T21:14:19.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.343+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8020b20 osd=6 4 2024-02-19T21:14:19.343 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.343+0000 7fdfec1f6700 10 client.23583.objecter _send_command 14 2024-02-19T21:14:19.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.343+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdfc80270f0 con 0x7fdfc80235e0 2024-02-19T21:14:19.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fdfc80270f0 con 0x7fdfc80235e0 2024-02-19T21:14:19.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:19.344 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc8020b20 osd=6 4 2024-02-19T21:14:19.345 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 6 14 2024-02-19T21:14:19.345 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:14:19.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:19.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 15 2024-02-19T21:14:19.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc80270f0 osd=7 3 2024-02-19T21:14:19.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc80270f0 osd=7 3 2024-02-19T21:14:19.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc80270f0 osd=7 3 2024-02-19T21:14:19.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 15 2024-02-19T21:14:19.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc80270f0 osd=7 3 2024-02-19T21:14:19.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 7 15 2024-02-19T21:14:19.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc80270f0 osd=7 4 2024-02-19T21:14:19.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfec1f6700 10 client.23583.objecter _send_command 15 2024-02-19T21:14:19.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.345+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdfc802b3c0 con 0x7fdfc8027770 2024-02-19T21:14:19.346 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.346+0000 7fdfd6ffd700 10 client.23583.objecter ms_handle_connect 0x7fdfc8027770 2024-02-19T21:14:19.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.347+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fdfc802b3c0 con 0x7fdfc8027770 2024-02-19T21:14:19.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.347+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:14:19.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.347+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 15 = system:0 2024-02-19T21:14:19.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.347+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc80270f0 osd=7 4 2024-02-19T21:14:19.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.347+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 7 15 2024-02-19T21:14:19.360 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:19.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.364+0000 7fdfec1f6700 10 client.23583.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:19.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.364+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign -1 16 2024-02-19T21:14:19.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.364+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc80270f0 osd=7 3 2024-02-19T21:14:19.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.364+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc80270f0 osd=7 3 2024-02-19T21:14:19.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.364+0000 7fdfec1f6700 20 client.23583.objecter _get_session s=0x7fdfc80270f0 osd=7 3 2024-02-19T21:14:19.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.364+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_remove -1 16 2024-02-19T21:14:19.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.364+0000 7fdfec1f6700 20 client.23583.objecter get_session s=0x7fdfc80270f0 osd=7 3 2024-02-19T21:14:19.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.364+0000 7fdfec1f6700 15 client.23583.objecter _session_command_op_assign 7 16 2024-02-19T21:14:19.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.364+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc80270f0 osd=7 4 2024-02-19T21:14:19.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.364+0000 7fdfec1f6700 10 client.23583.objecter _send_command 16 2024-02-19T21:14:19.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.364+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fdfc802cb70 con 0x7fdfc8027770 2024-02-19T21:14:19.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.366+0000 7fdfd6ffd700 1 -- 172.21.15.163:0/2106042762 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fdfc802cb70 con 0x7fdfc8027770 2024-02-19T21:14:19.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.366+0000 7fdfd6ffd700 10 client.23583.objecter ms_dispatch 0x7fdfe4001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:19.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.366+0000 7fdfd6ffd700 10 client.23583.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.366+0000 7fdfd6ffd700 20 client.23583.objecter put_session s=0x7fdfc80270f0 osd=7 4 2024-02-19T21:14:19.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.366+0000 7fdfd6ffd700 15 client.23583.objecter _session_command_op_remove 7 16 2024-02-19T21:14:19.366 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:14:19.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 10 librados: watch_flush enter 2024-02-19T21:14:19.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 10 librados: watch_flush exit 2024-02-19T21:14:19.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 10 client.23583.objecter close_session for osd.0 2024-02-19T21:14:19.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 >> v1:172.21.15.163:6801/1559418115 conn(0x7fdfc8004560 legacy=0x7fdfc8002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:19.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc80051d0 osd=0 1 2024-02-19T21:14:19.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 10 client.23583.objecter close_session for osd.1 2024-02-19T21:14:19.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 >> v1:172.21.15.163:6805/3125047432 conn(0x7fdfc8004180 legacy=0x7fdfc800cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:19.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8004010 osd=1 1 2024-02-19T21:14:19.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 10 client.23583.objecter close_session for osd.2 2024-02-19T21:14:19.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 >> v1:172.21.15.163:6809/1563158487 conn(0x7fdfc800f530 legacy=0x7fdfc800f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:19.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8003660 osd=2 1 2024-02-19T21:14:19.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 10 client.23583.objecter close_session for osd.3 2024-02-19T21:14:19.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 >> v1:172.21.15.163:6813/2112279769 conn(0x7fdfc8008e10 legacy=0x7fdfc8015bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:19.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8003410 osd=3 1 2024-02-19T21:14:19.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 10 client.23583.objecter close_session for osd.4 2024-02-19T21:14:19.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 >> v1:172.21.15.193:6800/3129716965 conn(0x7fdfc801ac30 legacy=0x7fdfc801d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:19.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc800a730 osd=4 1 2024-02-19T21:14:19.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 10 client.23583.objecter close_session for osd.5 2024-02-19T21:14:19.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 >> v1:172.21.15.193:6804/805161991 conn(0x7fdfc8020600 legacy=0x7fdfc801e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:19.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc801fef0 osd=5 1 2024-02-19T21:14:19.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 10 client.23583.objecter close_session for osd.6 2024-02-19T21:14:19.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 >> v1:172.21.15.193:6808/787604615 conn(0x7fdfc80235e0 legacy=0x7fdfc80259d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:19.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfea793700 1 -- 172.21.15.163:0/2106042762 reap_dead start 2024-02-19T21:14:19.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc8020b20 osd=6 1 2024-02-19T21:14:19.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 10 client.23583.objecter close_session for osd.7 2024-02-19T21:14:19.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 >> v1:172.21.15.193:6812/157927432 conn(0x7fdfc8027770 legacy=0x7fdfc8029b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:19.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 20 client.23583.objecter put_session s=0x7fdfc80270f0 osd=7 1 2024-02-19T21:14:19.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 20 client.23583.objecter shutdown clearing up homeless session... 2024-02-19T21:14:19.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 10 client.23583.objecter successfully canceled tick 2024-02-19T21:14:19.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfea793700 1 -- 172.21.15.163:0/2106042762 reap_dead start 2024-02-19T21:14:19.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfea793700 1 -- 172.21.15.163:0/2106042762 reap_dead start 2024-02-19T21:14:19.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfea793700 1 -- 172.21.15.163:0/2106042762 reap_dead start 2024-02-19T21:14:19.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfea793700 1 -- 172.21.15.163:0/2106042762 reap_dead start 2024-02-19T21:14:19.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 >> v1:172.21.15.163:6800/1552514681 conn(0x7fdfd0063830 legacy=0x7fdfd0065cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 >> v1:172.21.15.163:6789/0 conn(0x7fdfe40fc870 legacy=0x7fdfe40f8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 shutdown_connections 2024-02-19T21:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 >> 172.21.15.163:0/2106042762 conn(0x7fdfe40f2470 msgr2=0x7fdfe4106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 shutdown_connections 2024-02-19T21:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 1 -- 172.21.15.163:0/2106042762 wait complete. 2024-02-19T21:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:19.367+0000 7fdfec1f6700 1 librados: shutdown 2024-02-19T21:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.371 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:14:19.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:18.466871+0000 mon.a (mon.0) 3157 : cluster [DBG] osdmap e622: 8 total, 8 up, 7 in 2024-02-19T21:14:19.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:18.603470+0000 osd.7 (osd.7) 280 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:14:19.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:18.606929+0000 osd.7 (osd.7) 281 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:14:19.495 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:19.208296+0000 mon.a (mon.0) 3158 : audit [DBG] from='client.? v1:172.21.15.163:0/2106042762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:19.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:18.466871+0000 mon.a (mon.0) 3157 : cluster [DBG] osdmap e622: 8 total, 8 up, 7 in 2024-02-19T21:14:19.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:18.603470+0000 osd.7 (osd.7) 280 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:14:19.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:18.606929+0000 osd.7 (osd.7) 281 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:14:19.496 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:19.208296+0000 mon.a (mon.0) 3158 : audit [DBG] from='client.? v1:172.21.15.163:0/2106042762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:19.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:18.466871+0000 mon.a (mon.0) 3157 : cluster [DBG] osdmap e622: 8 total, 8 up, 7 in 2024-02-19T21:14:19.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:18.603470+0000 osd.7 (osd.7) 280 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:14:19.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:18.606929+0000 osd.7 (osd.7) 281 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:14:19.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:19.208296+0000 mon.a (mon.0) 3158 : audit [DBG] from='client.? v1:172.21.15.163:0/2106042762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:20.111 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:20.114 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:20.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:20.215 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:20.245 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:20.290 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:14:20.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:18.909355+0000 osd.5 (osd.5) 248 : cluster [DBG] 16.1f scrub starts 2024-02-19T21:14:20.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:18.913431+0000 osd.5 (osd.5) 249 : cluster [DBG] 16.1f scrub ok 2024-02-19T21:14:20.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:18.960121+0000 osd.1 (osd.1) 278 : cluster [DBG] 237.3 scrub starts 2024-02-19T21:14:20.474 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:18.963046+0000 osd.1 (osd.1) 279 : cluster [DBG] 237.3 scrub ok 2024-02-19T21:14:20.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:19.147723+0000 osd.6 (osd.6) 213 : cluster [DBG] 16.6 scrub starts 2024-02-19T21:14:20.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:19.148616+0000 osd.6 (osd.6) 214 : cluster [DBG] 16.6 scrub ok 2024-02-19T21:14:20.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:19.284370+0000 mgr.y (mgr.14550) 537 : cluster [DBG] pgmap v933: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:14:20.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:19.474747+0000 mon.a (mon.0) 3159 : cluster [DBG] osdmap e623: 8 total, 8 up, 7 in 2024-02-19T21:14:20.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:19.585283+0000 osd.7 (osd.7) 282 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:14:20.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:19.588161+0000 osd.7 (osd.7) 283 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:14:20.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:19.966245+0000 osd.3 (osd.3) 253 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:14:20.475 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:19.968184+0000 osd.3 (osd.3) 254 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:14:20.594 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:20.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:20.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:20.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:20.723 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:18.909355+0000 osd.5 (osd.5) 248 : cluster [DBG] 16.1f scrub starts 2024-02-19T21:14:20.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:18.913431+0000 osd.5 (osd.5) 249 : cluster [DBG] 16.1f scrub ok 2024-02-19T21:14:20.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:18.960121+0000 osd.1 (osd.1) 278 : cluster [DBG] 237.3 scrub starts 2024-02-19T21:14:20.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:18.963046+0000 osd.1 (osd.1) 279 : cluster [DBG] 237.3 scrub ok 2024-02-19T21:14:20.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:19.147723+0000 osd.6 (osd.6) 213 : cluster [DBG] 16.6 scrub starts 2024-02-19T21:14:20.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:19.148616+0000 osd.6 (osd.6) 214 : cluster [DBG] 16.6 scrub ok 2024-02-19T21:14:20.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:19.284370+0000 mgr.y (mgr.14550) 537 : cluster [DBG] pgmap v933: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:14:20.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:19.474747+0000 mon.a (mon.0) 3159 : cluster [DBG] osdmap e623: 8 total, 8 up, 7 in 2024-02-19T21:14:20.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:19.585283+0000 osd.7 (osd.7) 282 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:14:20.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:19.588161+0000 osd.7 (osd.7) 283 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:14:20.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:19.966245+0000 osd.3 (osd.3) 253 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:14:20.724 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:19.968184+0000 osd.3 (osd.3) 254 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:14:20.725 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:20.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:14:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:14:20.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:18.909355+0000 osd.5 (osd.5) 248 : cluster [DBG] 16.1f scrub starts 2024-02-19T21:14:20.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:18.913431+0000 osd.5 (osd.5) 249 : cluster [DBG] 16.1f scrub ok 2024-02-19T21:14:20.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:18.960121+0000 osd.1 (osd.1) 278 : cluster [DBG] 237.3 scrub starts 2024-02-19T21:14:20.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:18.963046+0000 osd.1 (osd.1) 279 : cluster [DBG] 237.3 scrub ok 2024-02-19T21:14:20.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:19.147723+0000 osd.6 (osd.6) 213 : cluster [DBG] 16.6 scrub starts 2024-02-19T21:14:20.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:19.148616+0000 osd.6 (osd.6) 214 : cluster [DBG] 16.6 scrub ok 2024-02-19T21:14:20.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:19.284370+0000 mgr.y (mgr.14550) 537 : cluster [DBG] pgmap v933: 214 pgs: 214 active+clean; 5.3 KiB data, 1.8 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:14:20.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:19.474747+0000 mon.a (mon.0) 3159 : cluster [DBG] osdmap e623: 8 total, 8 up, 7 in 2024-02-19T21:14:20.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:19.585283+0000 osd.7 (osd.7) 282 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:14:20.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:19.588161+0000 osd.7 (osd.7) 283 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:14:20.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:19.966245+0000 osd.3 (osd.3) 253 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:14:20.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:19.968184+0000 osd.3 (osd.3) 254 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:14:20.924 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:20.926 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:20.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:21.027 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:21.056 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:21.157 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:21.182 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:21.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:21.279 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:14:21.280 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:14:21.284 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:21.311 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:21.566 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:19.899791+0000 osd.5 (osd.5) 250 : cluster [DBG] 16.f scrub starts 2024-02-19T21:14:21.567 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:19.904113+0000 osd.5 (osd.5) 251 : cluster [DBG] 16.f scrub ok 2024-02-19T21:14:21.567 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:20.624734+0000 mgr.y (mgr.14550) 538 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:21.567 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:19.899791+0000 osd.5 (osd.5) 250 : cluster [DBG] 16.f scrub starts 2024-02-19T21:14:21.567 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:19.904113+0000 osd.5 (osd.5) 251 : cluster [DBG] 16.f scrub ok 2024-02-19T21:14:21.567 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:20.624734+0000 mgr.y (mgr.14550) 538 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:21.738 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 7, 3, 4, 6] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:14:21.738 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:14:21.739 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-02-19T21:14:21.739 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-19T21:14:21.739 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-19T21:14:21.739 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-02-19T21:14:21.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:19.899791+0000 osd.5 (osd.5) 250 : cluster [DBG] 16.f scrub starts 2024-02-19T21:14:21.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:19.904113+0000 osd.5 (osd.5) 251 : cluster [DBG] 16.f scrub ok 2024-02-19T21:14:21.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:20.624734+0000 mgr.y (mgr.14550) 538 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:22.008 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-19T21:14:22.388 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:22.432 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:22.447 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:22.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:20.911456+0000 osd.5 (osd.5) 252 : cluster [DBG] 5.f scrub starts 2024-02-19T21:14:22.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:20.914416+0000 osd.5 (osd.5) 253 : cluster [DBG] 5.f scrub ok 2024-02-19T21:14:22.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:20.998212+0000 osd.1 (osd.1) 280 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:14:22.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:20.999465+0000 osd.1 (osd.1) 281 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:14:22.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:21.285277+0000 mgr.y (mgr.14550) 539 : cluster [DBG] pgmap v935: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 419 B/s, 12 objects/s recovering 2024-02-19T21:14:22.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:21.481834+0000 osd.2 (osd.2) 248 : cluster [DBG] 16.17 scrub starts 2024-02-19T21:14:22.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:21.483824+0000 osd.2 (osd.2) 249 : cluster [DBG] 16.17 scrub ok 2024-02-19T21:14:22.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:21.585227+0000 osd.4 (osd.4) 221 : cluster [DBG] 4.18 deep-scrub starts 2024-02-19T21:14:22.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:21.589133+0000 osd.4 (osd.4) 222 : cluster [DBG] 4.18 deep-scrub ok 2024-02-19T21:14:22.674 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:21.958352+0000 osd.3 (osd.3) 255 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:14:22.674 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:21.960120+0000 osd.3 (osd.3) 256 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:14:22.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:20.911456+0000 osd.5 (osd.5) 252 : cluster [DBG] 5.f scrub starts 2024-02-19T21:14:22.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:20.914416+0000 osd.5 (osd.5) 253 : cluster [DBG] 5.f scrub ok 2024-02-19T21:14:22.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:20.998212+0000 osd.1 (osd.1) 280 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:14:22.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:20.999465+0000 osd.1 (osd.1) 281 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:14:22.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:21.285277+0000 mgr.y (mgr.14550) 539 : cluster [DBG] pgmap v935: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 419 B/s, 12 objects/s recovering 2024-02-19T21:14:22.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:21.481834+0000 osd.2 (osd.2) 248 : cluster [DBG] 16.17 scrub starts 2024-02-19T21:14:22.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:21.483824+0000 osd.2 (osd.2) 249 : cluster [DBG] 16.17 scrub ok 2024-02-19T21:14:22.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:21.585227+0000 osd.4 (osd.4) 221 : cluster [DBG] 4.18 deep-scrub starts 2024-02-19T21:14:22.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:21.589133+0000 osd.4 (osd.4) 222 : cluster [DBG] 4.18 deep-scrub ok 2024-02-19T21:14:22.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:21.958352+0000 osd.3 (osd.3) 255 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:14:22.675 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:21.960120+0000 osd.3 (osd.3) 256 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:14:22.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:20.911456+0000 osd.5 (osd.5) 252 : cluster [DBG] 5.f scrub starts 2024-02-19T21:14:22.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:20.914416+0000 osd.5 (osd.5) 253 : cluster [DBG] 5.f scrub ok 2024-02-19T21:14:22.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:20.998212+0000 osd.1 (osd.1) 280 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:14:22.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:20.999465+0000 osd.1 (osd.1) 281 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:14:22.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:21.285277+0000 mgr.y (mgr.14550) 539 : cluster [DBG] pgmap v935: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 419 B/s, 12 objects/s recovering 2024-02-19T21:14:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:21.481834+0000 osd.2 (osd.2) 248 : cluster [DBG] 16.17 scrub starts 2024-02-19T21:14:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:21.483824+0000 osd.2 (osd.2) 249 : cluster [DBG] 16.17 scrub ok 2024-02-19T21:14:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:21.585227+0000 osd.4 (osd.4) 221 : cluster [DBG] 4.18 deep-scrub starts 2024-02-19T21:14:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:21.589133+0000 osd.4 (osd.4) 222 : cluster [DBG] 4.18 deep-scrub ok 2024-02-19T21:14:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:21.958352+0000 osd.3 (osd.3) 255 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:14:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:21.960120+0000 osd.3 (osd.3) 256 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:14:23.261 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:23.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:23.365 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:14:23.393 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:23.574 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:22.078047+0000 osd.6 (osd.6) 215 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:14:23.574 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:22.079889+0000 osd.6 (osd.6) 216 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:14:23.574 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:22.520184+0000 osd.2 (osd.2) 250 : cluster [DBG] 4.f scrub starts 2024-02-19T21:14:23.574 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:22.521742+0000 osd.2 (osd.2) 251 : cluster [DBG] 4.f scrub ok 2024-02-19T21:14:23.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:22.078047+0000 osd.6 (osd.6) 215 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:14:23.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:22.079889+0000 osd.6 (osd.6) 216 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:14:23.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:22.520184+0000 osd.2 (osd.2) 250 : cluster [DBG] 4.f scrub starts 2024-02-19T21:14:23.575 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:22.521742+0000 osd.2 (osd.2) 251 : cluster [DBG] 4.f scrub ok 2024-02-19T21:14:23.575 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:14:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:14:23] "GET /metrics HTTP/1.1" 200 191389 "" "Prometheus/2.33.4" 2024-02-19T21:14:23.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:22.078047+0000 osd.6 (osd.6) 215 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:14:23.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:22.079889+0000 osd.6 (osd.6) 216 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:14:23.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:22.520184+0000 osd.2 (osd.2) 250 : cluster [DBG] 4.f scrub starts 2024-02-19T21:14:23.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:22.521742+0000 osd.2 (osd.2) 251 : cluster [DBG] 4.f scrub ok 2024-02-19T21:14:24.369 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:14:24.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:23.024980+0000 osd.1 (osd.1) 282 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:14:24.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:23.025931+0000 osd.1 (osd.1) 283 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:14:24.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:23.285739+0000 mgr.y (mgr.14550) 540 : cluster [DBG] pgmap v936: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 1 op/s; 345 B/s, 10 objects/s recovering 2024-02-19T21:14:24.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:24.106388+0000 mon.a (mon.0) 3160 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:14:24.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:24.356310+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e624: 8 total, 8 up, 7 in 2024-02-19T21:14:24.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:23.024980+0000 osd.1 (osd.1) 282 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:14:24.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:23.025931+0000 osd.1 (osd.1) 283 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:14:24.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:23.285739+0000 mgr.y (mgr.14550) 540 : cluster [DBG] pgmap v936: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 1 op/s; 345 B/s, 10 objects/s recovering 2024-02-19T21:14:24.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:24.106388+0000 mon.a (mon.0) 3160 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:14:24.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:24.356310+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e624: 8 total, 8 up, 7 in 2024-02-19T21:14:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:23.024980+0000 osd.1 (osd.1) 282 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:14:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:23.025931+0000 osd.1 (osd.1) 283 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:14:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:23.285739+0000 mgr.y (mgr.14550) 540 : cluster [DBG] pgmap v936: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1023 B/s rd, 1 op/s; 345 B/s, 10 objects/s recovering 2024-02-19T21:14:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:24.106388+0000 mon.a (mon.0) 3160 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:14:24.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:24.356310+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e624: 8 total, 8 up, 7 in 2024-02-19T21:14:25.524 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:24.510405+0000 osd.2 (osd.2) 252 : cluster [DBG] 16.a scrub starts 2024-02-19T21:14:25.524 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:24.511503+0000 osd.2 (osd.2) 253 : cluster [DBG] 16.a scrub ok 2024-02-19T21:14:25.524 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:24.573847+0000 osd.7 (osd.7) 284 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:14:25.524 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:24.577299+0000 osd.7 (osd.7) 285 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:14:25.524 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:24.859283+0000 mon.a (mon.0) 3162 : audit [INF] from='client.? v1:172.21.15.163:0/354974303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:14:25.524 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:25.067798+0000 osd.3 (osd.3) 257 : cluster [DBG] 2.28 scrub starts 2024-02-19T21:14:25.525 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:25.068697+0000 osd.3 (osd.3) 258 : cluster [DBG] 2.28 scrub ok 2024-02-19T21:14:25.525 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:24.510405+0000 osd.2 (osd.2) 252 : cluster [DBG] 16.a scrub starts 2024-02-19T21:14:25.525 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:24.511503+0000 osd.2 (osd.2) 253 : cluster [DBG] 16.a scrub ok 2024-02-19T21:14:25.525 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:24.573847+0000 osd.7 (osd.7) 284 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:14:25.525 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:24.577299+0000 osd.7 (osd.7) 285 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:14:25.525 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:24.859283+0000 mon.a (mon.0) 3162 : audit [INF] from='client.? v1:172.21.15.163:0/354974303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:14:25.525 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:25.067798+0000 osd.3 (osd.3) 257 : cluster [DBG] 2.28 scrub starts 2024-02-19T21:14:25.525 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:25.068697+0000 osd.3 (osd.3) 258 : cluster [DBG] 2.28 scrub ok 2024-02-19T21:14:25.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.674+0000 7f18865db700 1 -- 172.21.15.163:0/1451475427 >> v1:172.21.15.163:6790/0 conn(0x7f18801042b0 legacy=0x7f1880106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:25.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.674+0000 7f18865db700 1 -- 172.21.15.163:0/1451475427 shutdown_connections 2024-02-19T21:14:25.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.674+0000 7f18865db700 1 -- 172.21.15.163:0/1451475427 >> 172.21.15.163:0/1451475427 conn(0x7f18800f2470 msgr2=0x7f18800f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:25.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.674+0000 7f18865db700 1 -- 172.21.15.163:0/1451475427 shutdown_connections 2024-02-19T21:14:25.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.674+0000 7f18865db700 1 -- 172.21.15.163:0/1451475427 wait complete. 2024-02-19T21:14:25.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.674+0000 7f18865db700 1 librados: starting msgr at 2024-02-19T21:14:25.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.674+0000 7f18865db700 1 librados: starting objecter 2024-02-19T21:14:25.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.674+0000 7f18865db700 1 Processor -- start 2024-02-19T21:14:25.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f18865db700 1 -- start start 2024-02-19T21:14:25.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f18865db700 1 librados: setting wanted keys 2024-02-19T21:14:25.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f18865db700 1 librados: calling monclient init 2024-02-19T21:14:25.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f18865db700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1880117220 con 0x7f18801042b0 2024-02-19T21:14:25.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f18865db700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1880118400 con 0x7f18800f94d0 2024-02-19T21:14:25.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f18865db700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1880119600 con 0x7f18800f8920 2024-02-19T21:14:25.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f1884b78700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f18801042b0 0x7f1880115930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:55906/0 (socket says 172.21.15.163:55906) 2024-02-19T21:14:25.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f1884b78700 1 -- 172.21.15.163:0/354974303 learned_addr learned my addr 172.21.15.163:0/354974303 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:25.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f187ffff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f18800f8920 0x7f18800ff350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:47952/0 (socket says 172.21.15.163:47952) 2024-02-19T21:14:25.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f187d7fa700 10 client.?.objecter ms_handle_connect 0x7f18801042b0 2024-02-19T21:14:25.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f187d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:25.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f187d7fa700 10 client.?.objecter ms_handle_connect 0x7f18800f8920 2024-02-19T21:14:25.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f187d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:25.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732285000 0 0) 0x7f1880117220 con 0x7f18801042b0 2024-02-19T21:14:25.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1868003030 con 0x7f18801042b0 2024-02-19T21:14:25.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895859068 0 0) 0x7f1880119600 con 0x7f18800f8920 2024-02-19T21:14:25.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.675+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1880117220 con 0x7f18800f8920 2024-02-19T21:14:25.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 10 client.?.objecter ms_handle_connect 0x7f18800f94d0 2024-02-19T21:14:25.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:25.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1212415769 0 0) 0x7f1868003030 con 0x7f18801042b0 2024-02-19T21:14:25.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1880119600 con 0x7f18801042b0 2024-02-19T21:14:25.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f18780013f0 con 0x7f18801042b0 2024-02-19T21:14:25.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1524124428 0 0) 0x7f1880117220 con 0x7f18800f8920 2024-02-19T21:14:25.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1868003030 con 0x7f18800f8920 2024-02-19T21:14:25.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301549403 0 0) 0x7f1880118400 con 0x7f18800f94d0 2024-02-19T21:14:25.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1880117220 con 0x7f18800f94d0 2024-02-19T21:14:25.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f18740019b0 con 0x7f18800f8920 2024-02-19T21:14:25.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1118853464 0 0) 0x7f1880119600 con 0x7f18801042b0 2024-02-19T21:14:25.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 >> v1:172.21.15.163:6790/0 conn(0x7f18800f8920 legacy=0x7f18800ff350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:25.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 >> v1:172.21.15.193:6789/0 conn(0x7f18800f94d0 legacy=0x7f18800ffa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:25.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188011a800 con 0x7f18801042b0 2024-02-19T21:14:25.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f18865db700 1 -- 172.21.15.163:0/354974303 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18801197f0 con 0x7f18801042b0 2024-02-19T21:14:25.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f18865db700 10 client.23589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:25.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f18865db700 1 -- 172.21.15.163:0/354974303 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1880119c60 con 0x7f18801042b0 2024-02-19T21:14:25.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f18865db700 1 librados: init done 2024-02-19T21:14:25.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f1878001550 con 0x7f18801042b0 2024-02-19T21:14:25.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.676+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1878003a80 con 0x7f18801042b0 2024-02-19T21:14:25.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.677+0000 7f18865db700 1 -- 172.21.15.163:0/354974303 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f186c004f20 con 0x7f18801042b0 2024-02-19T21:14:25.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.677+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f1878001bc0 con 0x7f18801042b0 2024-02-19T21:14:25.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.678+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(624..624 src has 434..624) v4 ==== 8508+0+0 (unknown 734144458 0 0) 0x7f1878079420 con 0x7f18801042b0 2024-02-19T21:14:25.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.678+0000 7f187d7fa700 10 client.23589.objecter ms_dispatch 0x7f1880001550 osd_map(624..624 src has 434..624) v4 2024-02-19T21:14:25.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.678+0000 7f187d7fa700 3 client.23589.objecter handle_osd_map got epochs [624,624] > 0 2024-02-19T21:14:25.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.678+0000 7f187d7fa700 3 client.23589.objecter handle_osd_map decoding full epoch 624 2024-02-19T21:14:25.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.678+0000 7f187d7fa700 20 client.23589.objecter dump_active .. 0 homeless 2024-02-19T21:14:25.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.678+0000 7f187d7fa700 10 client.23589.objecter ms_handle_connect 0x7f1868063c90 2024-02-19T21:14:25.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.682+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f187804c670 con 0x7f18801042b0 2024-02-19T21:14:25.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:24.858+0000 7f18865db700 1 -- 172.21.15.163:0/354974303 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f186c0051d0 con 0x7f18801042b0 2024-02-19T21:14:25.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:25.501+0000 7f187d7fa700 1 -- 172.21.15.163:0/354974303 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v625) v1 ==== 87+0+0 (unknown 661384039 0 0) 0x7f187804aae0 con 0x7f18801042b0 2024-02-19T21:14:25.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:25.503+0000 7f18865db700 10 librados: watch_flush enter 2024-02-19T21:14:25.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:25.503+0000 7f18865db700 10 librados: watch_flush exit 2024-02-19T21:14:25.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:25.503+0000 7f18865db700 20 client.23589.objecter shutdown clearing up homeless session... 2024-02-19T21:14:25.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:25.503+0000 7f18865db700 10 client.23589.objecter successfully canceled tick 2024-02-19T21:14:25.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:25.503+0000 7f18865db700 1 -- 172.21.15.163:0/354974303 >> v1:172.21.15.163:6800/1552514681 conn(0x7f1868063c90 legacy=0x7f1868066150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:25.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:25.503+0000 7f18865db700 1 -- 172.21.15.163:0/354974303 >> v1:172.21.15.163:6789/0 conn(0x7f18801042b0 legacy=0x7f1880115930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:25.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:25.503+0000 7f18865db700 1 -- 172.21.15.163:0/354974303 shutdown_connections 2024-02-19T21:14:25.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:25.503+0000 7f18865db700 1 -- 172.21.15.163:0/354974303 >> 172.21.15.163:0/354974303 conn(0x7f18800f2470 msgr2=0x7f1880107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:25.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:25.503+0000 7f18865db700 1 -- 172.21.15.163:0/354974303 shutdown_connections 2024-02-19T21:14:25.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:25.503+0000 7f18865db700 1 -- 172.21.15.163:0/354974303 wait complete. 2024-02-19T21:14:25.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:25.503+0000 7f18865db700 1 librados: shutdown 2024-02-19T21:14:25.808 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:14:25.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:24.510405+0000 osd.2 (osd.2) 252 : cluster [DBG] 16.a scrub starts 2024-02-19T21:14:25.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:24.511503+0000 osd.2 (osd.2) 253 : cluster [DBG] 16.a scrub ok 2024-02-19T21:14:25.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:24.573847+0000 osd.7 (osd.7) 284 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:14:25.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:24.577299+0000 osd.7 (osd.7) 285 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:14:25.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:24.859283+0000 mon.a (mon.0) 3162 : audit [INF] from='client.? v1:172.21.15.163:0/354974303' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:14:25.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:25.067798+0000 osd.3 (osd.3) 257 : cluster [DBG] 2.28 scrub starts 2024-02-19T21:14:25.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:25.068697+0000 osd.3 (osd.3) 258 : cluster [DBG] 2.28 scrub ok 2024-02-19T21:14:26.414 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:14:26.415 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:14:26.744 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:14:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:25.021153+0000 osd.1 (osd.1) 284 : cluster [DBG] 16.9 scrub starts 2024-02-19T21:14:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:25.024543+0000 osd.1 (osd.1) 285 : cluster [DBG] 16.9 scrub ok 2024-02-19T21:14:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:25.286619+0000 mgr.y (mgr.14550) 541 : cluster [DBG] pgmap v938: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.6 KiB/s rd, 1 op/s; 303 B/s, 9 objects/s recovering 2024-02-19T21:14:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:25.492673+0000 mon.a (mon.0) 3163 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:14:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:25.500662+0000 mon.a (mon.0) 3164 : audit [INF] from='client.? v1:172.21.15.163:0/354974303' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:14:26.746 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:25.500692+0000 osd.2 (osd.2) 254 : cluster [DBG] 2.c scrub starts 2024-02-19T21:14:26.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:25.500728+0000 mon.a (mon.0) 3165 : cluster [DBG] osdmap e625: 8 total, 8 up, 7 in 2024-02-19T21:14:26.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:25.501892+0000 osd.2 (osd.2) 255 : cluster [DBG] 2.c scrub ok 2024-02-19T21:14:26.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:25.613367+0000 osd.7 (osd.7) 286 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:14:26.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:25.616253+0000 osd.7 (osd.7) 287 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:14:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:26.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:14:26.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:25.021153+0000 osd.1 (osd.1) 284 : cluster [DBG] 16.9 scrub starts 2024-02-19T21:14:26.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:25.024543+0000 osd.1 (osd.1) 285 : cluster [DBG] 16.9 scrub ok 2024-02-19T21:14:26.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:25.286619+0000 mgr.y (mgr.14550) 541 : cluster [DBG] pgmap v938: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.6 KiB/s rd, 1 op/s; 303 B/s, 9 objects/s recovering 2024-02-19T21:14:26.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:25.492673+0000 mon.a (mon.0) 3163 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:14:26.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:25.500662+0000 mon.a (mon.0) 3164 : audit [INF] from='client.? v1:172.21.15.163:0/354974303' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:14:26.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:25.500692+0000 osd.2 (osd.2) 254 : cluster [DBG] 2.c scrub starts 2024-02-19T21:14:26.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:25.500728+0000 mon.a (mon.0) 3165 : cluster [DBG] osdmap e625: 8 total, 8 up, 7 in 2024-02-19T21:14:26.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:25.501892+0000 osd.2 (osd.2) 255 : cluster [DBG] 2.c scrub ok 2024-02-19T21:14:26.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:25.613367+0000 osd.7 (osd.7) 286 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:14:26.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:25.616253+0000 osd.7 (osd.7) 287 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:14:26.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:25.021153+0000 osd.1 (osd.1) 284 : cluster [DBG] 16.9 scrub starts 2024-02-19T21:14:26.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:25.024543+0000 osd.1 (osd.1) 285 : cluster [DBG] 16.9 scrub ok 2024-02-19T21:14:26.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:25.286619+0000 mgr.y (mgr.14550) 541 : cluster [DBG] pgmap v938: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.6 KiB/s rd, 1 op/s; 303 B/s, 9 objects/s recovering 2024-02-19T21:14:26.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:25.492673+0000 mon.a (mon.0) 3163 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:14:26.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:25.500662+0000 mon.a (mon.0) 3164 : audit [INF] from='client.? v1:172.21.15.163:0/354974303' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:14:26.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:25.500692+0000 osd.2 (osd.2) 254 : cluster [DBG] 2.c scrub starts 2024-02-19T21:14:26.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:25.500728+0000 mon.a (mon.0) 3165 : cluster [DBG] osdmap e625: 8 total, 8 up, 7 in 2024-02-19T21:14:26.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:25.501892+0000 osd.2 (osd.2) 255 : cluster [DBG] 2.c scrub ok 2024-02-19T21:14:26.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:25.613367+0000 osd.7 (osd.7) 286 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:14:26.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:25.616253+0000 osd.7 (osd.7) 287 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:14:26.848 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:26.878 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:27.728 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:27.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:27.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:27.831 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:14:27.865 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:27.887 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:14:27.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:27.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:14:27.991 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:14:28.020 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:28.071 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:14:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:14:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:14:28.517 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:27.287021+0000 mgr.y (mgr.14550) 542 : cluster [DBG] pgmap v940: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 265 B/s, 8 objects/s recovering 2024-02-19T21:14:28.517 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:27.287021+0000 mgr.y (mgr.14550) 542 : cluster [DBG] pgmap v940: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 265 B/s, 8 objects/s recovering 2024-02-19T21:14:28.518 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-19T21:14:28.704 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:14:28.705 INFO:teuthology.orchestra.run.smithi163.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-19T21:14:28.705 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:14:28.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:27.287021+0000 mgr.y (mgr.14550) 542 : cluster [DBG] pgmap v940: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s; 265 B/s, 8 objects/s recovering 2024-02-19T21:14:28.887 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:29.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:29.092465+0000 osd.3 (osd.3) 259 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:29.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:29.092545+0000 osd.3 (osd.3) 260 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:29.611 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:29.356541+0000 mon.a (mon.0) 3166 : cluster [DBG] osdmap e626: 8 total, 8 up, 7 in 2024-02-19T21:14:29.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:29.092465+0000 osd.3 (osd.3) 259 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:29.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:29.092545+0000 osd.3 (osd.3) 260 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:29.611 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:29.356541+0000 mon.a (mon.0) 3166 : cluster [DBG] osdmap e626: 8 total, 8 up, 7 in 2024-02-19T21:14:29.671 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:14:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:29.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:14:29.739 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:14:29.774 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:14:29.805 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:29.092465+0000 osd.3 (osd.3) 259 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:29.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:29.092545+0000 osd.3 (osd.3) 260 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:29.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:29.356541+0000 mon.a (mon.0) 3166 : cluster [DBG] osdmap e626: 8 total, 8 up, 7 in 2024-02-19T21:14:29.808 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:29.830 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:14:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:29.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:14:29.933 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:29.961 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:30.119 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:30.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:29.287481+0000 mgr.y (mgr.14550) 543 : cluster [DBG] pgmap v941: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:14:30.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:29.287481+0000 mgr.y (mgr.14550) 543 : cluster [DBG] pgmap v941: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:14:30.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:29.287481+0000 mgr.y (mgr.14550) 543 : cluster [DBG] pgmap v941: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:14:30.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:14:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:14:31.410 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:31.419 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:31.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:31.520 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:14:31.557 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:31.577 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:14:31.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:31.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:14:31.681 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:14:31.710 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:31.729 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:30.630813+0000 mgr.y (mgr.14550) 544 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:31.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:30.630813+0000 mgr.y (mgr.14550) 544 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:31.730 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:14:31.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:31.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:14:31.762 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:31.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:30.630813+0000 mgr.y (mgr.14550) 544 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:31.833 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:31.862 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:31.985 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:31.987 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:31.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:32.088 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:32.118 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:32.217 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:32.220 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:32.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:32.321 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:14:32.353 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:32.376 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:14:32.379 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:32.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:14:32.480 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:14:32.509 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:32.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.611+0000 7f4564f1d700 1 -- 172.21.15.163:0/2672218611 >> v1:172.21.15.163:6789/0 conn(0x7f456010abd0 legacy=0x7f456010cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:32.613 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:14:32.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.611+0000 7f4564f1d700 1 -- 172.21.15.163:0/2672218611 shutdown_connections 2024-02-19T21:14:32.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.611+0000 7f4564f1d700 1 -- 172.21.15.163:0/2672218611 >> 172.21.15.163:0/2672218611 conn(0x7f45600f35a0 msgr2=0x7f45600f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:32.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.612+0000 7f4564f1d700 1 -- 172.21.15.163:0/2672218611 shutdown_connections 2024-02-19T21:14:32.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.612+0000 7f4564f1d700 1 -- 172.21.15.163:0/2672218611 wait complete. 2024-02-19T21:14:32.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.612+0000 7f4564f1d700 1 librados: starting msgr at 2024-02-19T21:14:32.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.612+0000 7f4564f1d700 1 librados: starting objecter 2024-02-19T21:14:32.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.612+0000 7f4564f1d700 1 Processor -- start 2024-02-19T21:14:32.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.612+0000 7f4564f1d700 1 -- start start 2024-02-19T21:14:32.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.612+0000 7f4564f1d700 1 librados: setting wanted keys 2024-02-19T21:14:32.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.612+0000 7f4564f1d700 1 librados: calling monclient init 2024-02-19T21:14:32.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f4564f1d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f456006ab10 con 0x7f456006b880 2024-02-19T21:14:32.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f4564f1d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f456006ace0 con 0x7f456010abd0 2024-02-19T21:14:32.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f4564f1d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4560128650 con 0x7f456006cc20 2024-02-19T21:14:32.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f455e59c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f456006b880 0x7f4560107200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:55922/0 (socket says 172.21.15.163:55922) 2024-02-19T21:14:32.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f455e59c700 1 -- 172.21.15.163:0/2638313847 learned_addr learned my addr 172.21.15.163:0/2638313847 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:32.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 10 client.?.objecter ms_handle_connect 0x7f456006b880 2024-02-19T21:14:32.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:32.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 10 client.?.objecter ms_handle_connect 0x7f456010abd0 2024-02-19T21:14:32.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:32.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419352819 0 0) 0x7f456006ab10 con 0x7f456006b880 2024-02-19T21:14:32.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4548003630 con 0x7f456006b880 2024-02-19T21:14:32.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391156381 0 0) 0x7f456006ace0 con 0x7f456010abd0 2024-02-19T21:14:32.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f456006ab10 con 0x7f456010abd0 2024-02-19T21:14:32.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 414594200 0 0) 0x7f4548003630 con 0x7f456006b880 2024-02-19T21:14:32.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f456006ace0 con 0x7f456006b880 2024-02-19T21:14:32.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f45500018c0 con 0x7f456006b880 2024-02-19T21:14:32.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 10 client.?.objecter ms_handle_connect 0x7f456006cc20 2024-02-19T21:14:32.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:32.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3938763687 0 0) 0x7f456006ace0 con 0x7f456006b880 2024-02-19T21:14:32.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 >> v1:172.21.15.163:6790/0 conn(0x7f456006cc20 legacy=0x7f4560107910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:32.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 >> v1:172.21.15.193:6789/0 conn(0x7f456010abd0 legacy=0x7f4560124f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:32.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4560129830 con 0x7f456006b880 2024-02-19T21:14:32.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.613+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4560128840 con 0x7f456006b880 2024-02-19T21:14:32.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.614+0000 7f4564f1d700 10 client.23595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:32.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.614+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4560128cb0 con 0x7f456006b880 2024-02-19T21:14:32.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.614+0000 7f4564f1d700 1 librados: init done 2024-02-19T21:14:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.614+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f456005f660 con 0x7f456006b880 2024-02-19T21:14:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.614+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f4550002000 con 0x7f456006b880 2024-02-19T21:14:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.614+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4550005aa0 con 0x7f456006b880 2024-02-19T21:14:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.616+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f4550001b60 con 0x7f456006b880 2024-02-19T21:14:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.616+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(628..628 src has 434..628) v4 ==== 8564+0+0 (unknown 945170454 0 0) 0x7f4550078e70 con 0x7f456006b880 2024-02-19T21:14:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.616+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 osd_map(628..628 src has 434..628) v4 2024-02-19T21:14:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.616+0000 7f454f7fe700 3 client.23595.objecter handle_osd_map got epochs [628,628] > 0 2024-02-19T21:14:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.616+0000 7f454f7fe700 3 client.23595.objecter handle_osd_map decoding full epoch 628 2024-02-19T21:14:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.616+0000 7f454f7fe700 20 client.23595.objecter dump_active .. 0 homeless 2024-02-19T21:14:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.616+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v628) v1 ==== 54+0+16 (unknown 2206768316 0 2420000192) 0x7f4550079250 con 0x7f456006b880 2024-02-19T21:14:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.616+0000 7f454f7fe700 10 client.23595.objecter ms_handle_connect 0x7f45480639d0 2024-02-19T21:14:32.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f4564f1d700 10 client.23595.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 1 2024-02-19T21:14:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560047540 osd=0 3 2024-02-19T21:14:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560047540 osd=0 3 2024-02-19T21:14:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560047540 osd=0 3 2024-02-19T21:14:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 1 2024-02-19T21:14:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f4560047540 osd=0 3 2024-02-19T21:14:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 0 1 2024-02-19T21:14:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560047540 osd=0 4 2024-02-19T21:14:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f4564f1d700 10 client.23595.objecter _send_command 1 2024-02-19T21:14:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f456006c860 con 0x7f4560028540 2024-02-19T21:14:32.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f454f7fe700 10 client.23595.objecter ms_handle_connect 0x7f4560028540 2024-02-19T21:14:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:32.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:14:32.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f456006c860 con 0x7f4560028540 2024-02-19T21:14:32.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:14:32.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f454f7fe700 10 client.23595.objecter _finish_command 1 = system:0 2024-02-19T21:14:32.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f4560047540 osd=0 4 2024-02-19T21:14:32.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.617+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 0 1 2024-02-19T21:14:32.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.632+0000 7f4564f1d700 10 client.23595.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:32.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.632+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 2 2024-02-19T21:14:32.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.632+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560047540 osd=0 3 2024-02-19T21:14:32.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.632+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560047540 osd=0 3 2024-02-19T21:14:32.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.632+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560047540 osd=0 3 2024-02-19T21:14:32.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.632+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 2 2024-02-19T21:14:32.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.632+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f4560047540 osd=0 3 2024-02-19T21:14:32.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.632+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 0 2 2024-02-19T21:14:32.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.632+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560047540 osd=0 4 2024-02-19T21:14:32.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.632+0000 7f4564f1d700 10 client.23595.objecter _send_command 2 2024-02-19T21:14:32.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.632+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4560110560 con 0x7f4560028540 2024-02-19T21:14:32.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4560110560 con 0x7f4560028540 2024-02-19T21:14:32.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:32.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f454f7fe700 10 client.23595.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f4560047540 osd=0 4 2024-02-19T21:14:32.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 0 2 2024-02-19T21:14:32.635 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:14:32.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f4564f1d700 10 client.23595.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:32.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 3 2024-02-19T21:14:32.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f45600f8f50 osd=1 3 2024-02-19T21:14:32.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f45600f8f50 osd=1 3 2024-02-19T21:14:32.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f45600f8f50 osd=1 3 2024-02-19T21:14:32.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 3 2024-02-19T21:14:32.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f45600f8f50 osd=1 3 2024-02-19T21:14:32.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 1 3 2024-02-19T21:14:32.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f45600f8f50 osd=1 4 2024-02-19T21:14:32.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f4564f1d700 10 client.23595.objecter _send_command 3 2024-02-19T21:14:32.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f456012d590 con 0x7f4560129a80 2024-02-19T21:14:32.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.634+0000 7f454f7fe700 10 client.23595.objecter ms_handle_connect 0x7f4560129a80 2024-02-19T21:14:32.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.635+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f456012d590 con 0x7f4560129a80 2024-02-19T21:14:32.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.635+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:14:32.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.635+0000 7f454f7fe700 10 client.23595.objecter _finish_command 3 = system:0 2024-02-19T21:14:32.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.635+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f45600f8f50 osd=1 4 2024-02-19T21:14:32.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.635+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 1 3 2024-02-19T21:14:32.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.650+0000 7f4564f1d700 10 client.23595.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:32.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.650+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 4 2024-02-19T21:14:32.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.650+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f45600f8f50 osd=1 3 2024-02-19T21:14:32.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.650+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f45600f8f50 osd=1 3 2024-02-19T21:14:32.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.650+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f45600f8f50 osd=1 3 2024-02-19T21:14:32.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.650+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 4 2024-02-19T21:14:32.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.650+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f45600f8f50 osd=1 3 2024-02-19T21:14:32.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.650+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 1 4 2024-02-19T21:14:32.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.650+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f45600f8f50 osd=1 4 2024-02-19T21:14:32.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.650+0000 7f4564f1d700 10 client.23595.objecter _send_command 4 2024-02-19T21:14:32.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.650+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f45600f89c0 con 0x7f4560129a80 2024-02-19T21:14:32.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.651+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f45600f89c0 con 0x7f4560129a80 2024-02-19T21:14:32.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.651+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:32.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.651+0000 7f454f7fe700 10 client.23595.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.651+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f45600f8f50 osd=1 4 2024-02-19T21:14:32.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.651+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 1 4 2024-02-19T21:14:32.652 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:14:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f4564f1d700 10 client.23595.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 5 2024-02-19T21:14:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f45600f8dd0 osd=2 3 2024-02-19T21:14:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f45600f8dd0 osd=2 3 2024-02-19T21:14:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f45600f8dd0 osd=2 3 2024-02-19T21:14:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 5 2024-02-19T21:14:32.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f45600f8dd0 osd=2 3 2024-02-19T21:14:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 2 5 2024-02-19T21:14:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f45600f8dd0 osd=2 4 2024-02-19T21:14:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f4564f1d700 10 client.23595.objecter _send_command 5 2024-02-19T21:14:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4560131fb0 con 0x7f45600280a0 2024-02-19T21:14:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f454f7fe700 10 client.23595.objecter ms_handle_connect 0x7f45600280a0 2024-02-19T21:14:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4560131fb0 con 0x7f45600280a0 2024-02-19T21:14:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:14:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f454f7fe700 10 client.23595.objecter _finish_command 5 = system:0 2024-02-19T21:14:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f45600f8dd0 osd=2 4 2024-02-19T21:14:32.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.652+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 2 5 2024-02-19T21:14:32.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.667+0000 7f4564f1d700 10 client.23595.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:32.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.667+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 6 2024-02-19T21:14:32.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.667+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f45600f8dd0 osd=2 3 2024-02-19T21:14:32.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.667+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f45600f8dd0 osd=2 3 2024-02-19T21:14:32.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.667+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f45600f8dd0 osd=2 3 2024-02-19T21:14:32.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.667+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 6 2024-02-19T21:14:32.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.667+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f45600f8dd0 osd=2 3 2024-02-19T21:14:32.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.667+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 2 6 2024-02-19T21:14:32.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.667+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f45600f8dd0 osd=2 4 2024-02-19T21:14:32.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.667+0000 7f4564f1d700 10 client.23595.objecter _send_command 6 2024-02-19T21:14:32.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.667+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4560027600 con 0x7f45600280a0 2024-02-19T21:14:32.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.668+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4560027600 con 0x7f45600280a0 2024-02-19T21:14:32.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.668+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:32.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.668+0000 7f454f7fe700 10 client.23595.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.668+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f45600f8dd0 osd=2 4 2024-02-19T21:14:32.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.668+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 2 6 2024-02-19T21:14:32.669 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:14:32.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f4564f1d700 10 client.23595.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:32.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 7 2024-02-19T21:14:32.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560027a10 osd=3 3 2024-02-19T21:14:32.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560027a10 osd=3 3 2024-02-19T21:14:32.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560027a10 osd=3 3 2024-02-19T21:14:32.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 7 2024-02-19T21:14:32.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f4560027a10 osd=3 3 2024-02-19T21:14:32.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 3 7 2024-02-19T21:14:32.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560027a10 osd=3 4 2024-02-19T21:14:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f4564f1d700 10 client.23595.objecter _send_command 7 2024-02-19T21:14:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4560136a40 con 0x7f456012e3e0 2024-02-19T21:14:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f454f7fe700 10 client.23595.objecter ms_handle_connect 0x7f456012e3e0 2024-02-19T21:14:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4560136a40 con 0x7f456012e3e0 2024-02-19T21:14:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:14:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f454f7fe700 10 client.23595.objecter _finish_command 7 = system:0 2024-02-19T21:14:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f4560027a10 osd=3 4 2024-02-19T21:14:32.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.669+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 3 7 2024-02-19T21:14:32.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.684+0000 7f4564f1d700 10 client.23595.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:32.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.684+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 8 2024-02-19T21:14:32.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.684+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560027a10 osd=3 3 2024-02-19T21:14:32.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.684+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560027a10 osd=3 3 2024-02-19T21:14:32.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.684+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560027a10 osd=3 3 2024-02-19T21:14:32.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.684+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 8 2024-02-19T21:14:32.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.684+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f4560027a10 osd=3 3 2024-02-19T21:14:32.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.684+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 3 8 2024-02-19T21:14:32.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.684+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560027a10 osd=3 4 2024-02-19T21:14:32.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.684+0000 7f4564f1d700 10 client.23595.objecter _send_command 8 2024-02-19T21:14:32.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.684+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f456012d940 con 0x7f456012e3e0 2024-02-19T21:14:32.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f456012d940 con 0x7f456012e3e0 2024-02-19T21:14:32.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:32.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f454f7fe700 10 client.23595.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f4560027a10 osd=3 4 2024-02-19T21:14:32.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 3 8 2024-02-19T21:14:32.686 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:14:32.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f4564f1d700 10 client.23595.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:32.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 9 2024-02-19T21:14:32.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f456012d940 osd=4 3 2024-02-19T21:14:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f456012d940 osd=4 3 2024-02-19T21:14:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f456012d940 osd=4 3 2024-02-19T21:14:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 9 2024-02-19T21:14:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f456012d940 osd=4 3 2024-02-19T21:14:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 4 9 2024-02-19T21:14:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f456012d940 osd=4 4 2024-02-19T21:14:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f4564f1d700 10 client.23595.objecter _send_command 9 2024-02-19T21:14:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.685+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4560132fc0 con 0x7f456012daf0 2024-02-19T21:14:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.686+0000 7f454f7fe700 10 client.23595.objecter ms_handle_connect 0x7f456012daf0 2024-02-19T21:14:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.687+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4560132fc0 con 0x7f456012daf0 2024-02-19T21:14:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.687+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:14:32.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.687+0000 7f454f7fe700 10 client.23595.objecter _finish_command 9 = system:0 2024-02-19T21:14:32.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.687+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f456012d940 osd=4 4 2024-02-19T21:14:32.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.687+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 4 9 2024-02-19T21:14:32.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.702+0000 7f4564f1d700 10 client.23595.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:32.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.702+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 10 2024-02-19T21:14:32.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.702+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f456012d940 osd=4 3 2024-02-19T21:14:32.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.702+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f456012d940 osd=4 3 2024-02-19T21:14:32.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.702+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f456012d940 osd=4 3 2024-02-19T21:14:32.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.702+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 10 2024-02-19T21:14:32.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.702+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f456012d940 osd=4 3 2024-02-19T21:14:32.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.702+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 4 10 2024-02-19T21:14:32.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.702+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f456012d940 osd=4 4 2024-02-19T21:14:32.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.702+0000 7f4564f1d700 10 client.23595.objecter _send_command 10 2024-02-19T21:14:32.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.702+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4560132fc0 con 0x7f456012daf0 2024-02-19T21:14:32.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.703+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4560132fc0 con 0x7f456012daf0 2024-02-19T21:14:32.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.703+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:32.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.703+0000 7f454f7fe700 10 client.23595.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.703+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f456012d940 osd=4 4 2024-02-19T21:14:32.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.703+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 4 10 2024-02-19T21:14:32.705 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:14:32.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.704+0000 7f4564f1d700 10 client.23595.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:32.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.704+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 11 2024-02-19T21:14:32.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.704+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560132fc0 osd=5 3 2024-02-19T21:14:32.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.704+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560132fc0 osd=5 3 2024-02-19T21:14:32.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.704+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560132fc0 osd=5 3 2024-02-19T21:14:32.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.704+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 11 2024-02-19T21:14:32.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.704+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f4560132fc0 osd=5 3 2024-02-19T21:14:32.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.704+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 5 11 2024-02-19T21:14:32.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.704+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560132fc0 osd=5 4 2024-02-19T21:14:32.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.704+0000 7f4564f1d700 10 client.23595.objecter _send_command 11 2024-02-19T21:14:32.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.704+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4560138ec0 con 0x7f4560138980 2024-02-19T21:14:32.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.704+0000 7f454f7fe700 10 client.23595.objecter ms_handle_connect 0x7f4560138980 2024-02-19T21:14:32.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.705+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4560138ec0 con 0x7f4560138980 2024-02-19T21:14:32.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.705+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:14:32.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.705+0000 7f454f7fe700 10 client.23595.objecter _finish_command 11 = system:0 2024-02-19T21:14:32.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.705+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f4560132fc0 osd=5 4 2024-02-19T21:14:32.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.705+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 5 11 2024-02-19T21:14:32.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.720+0000 7f4564f1d700 10 client.23595.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:32.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.720+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 12 2024-02-19T21:14:32.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.720+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560132fc0 osd=5 3 2024-02-19T21:14:32.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.720+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560132fc0 osd=5 3 2024-02-19T21:14:32.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.720+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560132fc0 osd=5 3 2024-02-19T21:14:32.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.720+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 12 2024-02-19T21:14:32.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.720+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f4560132fc0 osd=5 3 2024-02-19T21:14:32.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.720+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 5 12 2024-02-19T21:14:32.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.720+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560132fc0 osd=5 4 2024-02-19T21:14:32.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.720+0000 7f4564f1d700 10 client.23595.objecter _send_command 12 2024-02-19T21:14:32.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.720+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4560138ec0 con 0x7f4560138980 2024-02-19T21:14:32.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4560138ec0 con 0x7f4560138980 2024-02-19T21:14:32.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:32.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f454f7fe700 10 client.23595.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f4560132fc0 osd=5 4 2024-02-19T21:14:32.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 5 12 2024-02-19T21:14:32.723 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:14:32.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f4564f1d700 10 client.23595.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:32.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 13 2024-02-19T21:14:32.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560138ec0 osd=6 3 2024-02-19T21:14:32.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560138ec0 osd=6 3 2024-02-19T21:14:32.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560138ec0 osd=6 3 2024-02-19T21:14:32.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 13 2024-02-19T21:14:32.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f4560138ec0 osd=6 3 2024-02-19T21:14:32.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 6 13 2024-02-19T21:14:32.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560138ec0 osd=6 4 2024-02-19T21:14:32.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f4564f1d700 10 client.23595.objecter _send_command 13 2024-02-19T21:14:32.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4560143b50 con 0x7f4560140040 2024-02-19T21:14:32.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.722+0000 7f454f7fe700 10 client.23595.objecter ms_handle_connect 0x7f4560140040 2024-02-19T21:14:32.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.723+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4560143b50 con 0x7f4560140040 2024-02-19T21:14:32.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.723+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:14:32.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.723+0000 7f454f7fe700 10 client.23595.objecter _finish_command 13 = system:0 2024-02-19T21:14:32.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.723+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f4560138ec0 osd=6 4 2024-02-19T21:14:32.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.723+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 6 13 2024-02-19T21:14:32.725 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:14:32.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.740+0000 7f4564f1d700 10 client.23595.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:32.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.740+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 14 2024-02-19T21:14:32.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.740+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560138ec0 osd=6 3 2024-02-19T21:14:32.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.740+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560138ec0 osd=6 3 2024-02-19T21:14:32.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.740+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f4560138ec0 osd=6 3 2024-02-19T21:14:32.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.740+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 14 2024-02-19T21:14:32.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.740+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f4560138ec0 osd=6 3 2024-02-19T21:14:32.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.740+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 6 14 2024-02-19T21:14:32.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.740+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560138ec0 osd=6 4 2024-02-19T21:14:32.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.740+0000 7f4564f1d700 10 client.23595.objecter _send_command 14 2024-02-19T21:14:32.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.740+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f45601452d0 con 0x7f4560140040 2024-02-19T21:14:32.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.742+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f45601452d0 con 0x7f4560140040 2024-02-19T21:14:32.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.742+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:32.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.742+0000 7f454f7fe700 10 client.23595.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.742+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f4560138ec0 osd=6 4 2024-02-19T21:14:32.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.742+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 6 14 2024-02-19T21:14:32.743 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:14:32.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.742+0000 7f4564f1d700 10 client.23595.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:32.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.742+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 15 2024-02-19T21:14:32.744 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:14:32.744 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:14:32.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.742+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f45601452d0 osd=7 3 2024-02-19T21:14:32.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.742+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f45601452d0 osd=7 3 2024-02-19T21:14:32.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.742+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f45601452d0 osd=7 3 2024-02-19T21:14:32.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.743+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 15 2024-02-19T21:14:32.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.743+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f45601452d0 osd=7 3 2024-02-19T21:14:32.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.743+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 7 15 2024-02-19T21:14:32.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.743+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f45601452d0 osd=7 4 2024-02-19T21:14:32.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.743+0000 7f4564f1d700 10 client.23595.objecter _send_command 15 2024-02-19T21:14:32.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.743+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4560145e00 con 0x7f4560145950 2024-02-19T21:14:32.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.743+0000 7f454f7fe700 10 client.23595.objecter ms_handle_connect 0x7f4560145950 2024-02-19T21:14:32.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.744+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4560145e00 con 0x7f4560145950 2024-02-19T21:14:32.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.744+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:14:32.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.744+0000 7f454f7fe700 10 client.23595.objecter _finish_command 15 = system:0 2024-02-19T21:14:32.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.744+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f45601452d0 osd=7 4 2024-02-19T21:14:32.746 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.744+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 7 15 2024-02-19T21:14:32.752 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:32.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.759+0000 7f4564f1d700 10 client.23595.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:32.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.759+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign -1 16 2024-02-19T21:14:32.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.759+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f45601452d0 osd=7 3 2024-02-19T21:14:32.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.759+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f45601452d0 osd=7 3 2024-02-19T21:14:32.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.759+0000 7f4564f1d700 20 client.23595.objecter _get_session s=0x7f45601452d0 osd=7 3 2024-02-19T21:14:32.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.759+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_remove -1 16 2024-02-19T21:14:32.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.759+0000 7f4564f1d700 20 client.23595.objecter get_session s=0x7f45601452d0 osd=7 3 2024-02-19T21:14:32.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.759+0000 7f4564f1d700 15 client.23595.objecter _session_command_op_assign 7 16 2024-02-19T21:14:32.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.759+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f45601452d0 osd=7 4 2024-02-19T21:14:32.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.759+0000 7f4564f1d700 10 client.23595.objecter _send_command 16 2024-02-19T21:14:32.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.759+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4560145e00 con 0x7f4560145950 2024-02-19T21:14:32.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f454f7fe700 1 -- 172.21.15.163:0/2638313847 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4560145e00 con 0x7f4560145950 2024-02-19T21:14:32.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f454f7fe700 10 client.23595.objecter ms_dispatch 0x7f4560001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:32.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f454f7fe700 10 client.23595.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f454f7fe700 20 client.23595.objecter put_session s=0x7f45601452d0 osd=7 4 2024-02-19T21:14:32.761 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f454f7fe700 15 client.23595.objecter _session_command_op_remove 7 16 2024-02-19T21:14:32.762 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:14:32.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 10 librados: watch_flush enter 2024-02-19T21:14:32.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 10 librados: watch_flush exit 2024-02-19T21:14:32.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 10 client.23595.objecter close_session for osd.0 2024-02-19T21:14:32.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 >> v1:172.21.15.163:6801/1559418115 conn(0x7f4560028540 legacy=0x7f45600598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:32.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560047540 osd=0 1 2024-02-19T21:14:32.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 10 client.23595.objecter close_session for osd.1 2024-02-19T21:14:32.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 >> v1:172.21.15.163:6805/3125047432 conn(0x7f4560129a80 legacy=0x7f456012be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:32.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f45600f8f50 osd=1 1 2024-02-19T21:14:32.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 10 client.23595.objecter close_session for osd.2 2024-02-19T21:14:32.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 >> v1:172.21.15.163:6809/1563158487 conn(0x7f45600280a0 legacy=0x7f4560130890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:32.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f45600f8dd0 osd=2 1 2024-02-19T21:14:32.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 10 client.23595.objecter close_session for osd.3 2024-02-19T21:14:32.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 >> v1:172.21.15.163:6813/2112279769 conn(0x7f456012e3e0 legacy=0x7f45601352b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:32.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560027a10 osd=3 1 2024-02-19T21:14:32.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 10 client.23595.objecter close_session for osd.4 2024-02-19T21:14:32.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 >> v1:172.21.15.193:6800/3129716965 conn(0x7f456012daf0 legacy=0x7f4560136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:32.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f456012d940 osd=4 1 2024-02-19T21:14:32.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 10 client.23595.objecter close_session for osd.5 2024-02-19T21:14:32.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f455ed9d700 1 -- 172.21.15.163:0/2638313847 reap_dead start 2024-02-19T21:14:32.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 >> v1:172.21.15.193:6804/805161991 conn(0x7f4560138980 legacy=0x7f456013e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:32.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f455ed9d700 1 -- 172.21.15.163:0/2638313847 reap_dead start 2024-02-19T21:14:32.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560132fc0 osd=5 1 2024-02-19T21:14:32.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 10 client.23595.objecter close_session for osd.6 2024-02-19T21:14:32.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 >> v1:172.21.15.193:6808/787604615 conn(0x7f4560140040 legacy=0x7f4560142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:32.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f4560138ec0 osd=6 1 2024-02-19T21:14:32.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 10 client.23595.objecter close_session for osd.7 2024-02-19T21:14:32.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 >> v1:172.21.15.193:6812/157927432 conn(0x7f4560145950 legacy=0x7f4560143b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:32.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 20 client.23595.objecter put_session s=0x7f45601452d0 osd=7 1 2024-02-19T21:14:32.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f455ed9d700 1 -- 172.21.15.163:0/2638313847 reap_dead start 2024-02-19T21:14:32.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 20 client.23595.objecter shutdown clearing up homeless session... 2024-02-19T21:14:32.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.761+0000 7f4564f1d700 10 client.23595.objecter successfully canceled tick 2024-02-19T21:14:32.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.762+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 >> v1:172.21.15.163:6800/1552514681 conn(0x7f45480639d0 legacy=0x7f4548065e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:32.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.762+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 >> v1:172.21.15.163:6789/0 conn(0x7f456006b880 legacy=0x7f4560107200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:32.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.762+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 shutdown_connections 2024-02-19T21:14:32.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.762+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 >> 172.21.15.163:0/2638313847 conn(0x7f45600f35a0 msgr2=0x7f45600f4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:32.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.762+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 shutdown_connections 2024-02-19T21:14:32.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.762+0000 7f4564f1d700 1 -- 172.21.15.163:0/2638313847 wait complete. 2024-02-19T21:14:32.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:32.762+0000 7f4564f1d700 1 librados: shutdown 2024-02-19T21:14:32.766 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.766 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.766 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.766 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.766 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.766 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.766 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.766 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:14:32.782 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:31.288493+0000 mgr.y (mgr.14550) 545 : cluster [DBG] pgmap v943: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-19T21:14:32.783 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:31.517965+0000 mon.a (mon.0) 3167 : cluster [DBG] osdmap e627: 8 total, 8 up, 7 in 2024-02-19T21:14:32.783 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:31.548582+0000 mon.a (mon.0) 3168 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:14:32.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:31.288493+0000 mgr.y (mgr.14550) 545 : cluster [DBG] pgmap v943: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-19T21:14:32.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:31.517965+0000 mon.a (mon.0) 3167 : cluster [DBG] osdmap e627: 8 total, 8 up, 7 in 2024-02-19T21:14:32.783 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:31.548582+0000 mon.a (mon.0) 3168 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:14:32.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:31.288493+0000 mgr.y (mgr.14550) 545 : cluster [DBG] pgmap v943: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-19T21:14:32.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:31.517965+0000 mon.a (mon.0) 3167 : cluster [DBG] osdmap e627: 8 total, 8 up, 7 in 2024-02-19T21:14:32.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:31.548582+0000 mon.a (mon.0) 3168 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:14:33.497 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:14:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:14:33] "GET /metrics HTTP/1.1" 200 191558 "" "Prometheus/2.33.4" 2024-02-19T21:14:33.645 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-19T21:14:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:32.517786+0000 mon.a (mon.0) 3169 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:32.517831+0000 mon.a (mon.0) 3170 : cluster [DBG] osdmap e628: 8 total, 8 up, 7 in 2024-02-19T21:14:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:32.519576+0000 mon.a (mon.0) 3171 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-106"}]: dispatch 2024-02-19T21:14:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:32.616590+0000 mon.a (mon.0) 3172 : audit [DBG] from='client.? v1:172.21.15.163:0/2638313847' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:33.515298+0000 mon.a (mon.0) 3173 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:14:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:33.518932+0000 mon.a (mon.0) 3174 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-106"}]': finished 2024-02-19T21:14:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:33.518989+0000 mon.a (mon.0) 3175 : cluster [DBG] osdmap e629: 8 total, 8 up, 7 in 2024-02-19T21:14:33.872 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:32.517786+0000 mon.a (mon.0) 3169 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:33.872 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:32.517831+0000 mon.a (mon.0) 3170 : cluster [DBG] osdmap e628: 8 total, 8 up, 7 in 2024-02-19T21:14:33.872 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:32.519576+0000 mon.a (mon.0) 3171 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-106"}]: dispatch 2024-02-19T21:14:33.872 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:32.616590+0000 mon.a (mon.0) 3172 : audit [DBG] from='client.? v1:172.21.15.163:0/2638313847' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:33.872 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:33.515298+0000 mon.a (mon.0) 3173 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:14:33.873 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:33.518932+0000 mon.a (mon.0) 3174 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-106"}]': finished 2024-02-19T21:14:33.873 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:33.518989+0000 mon.a (mon.0) 3175 : cluster [DBG] osdmap e629: 8 total, 8 up, 7 in 2024-02-19T21:14:33.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:32.517786+0000 mon.a (mon.0) 3169 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:33.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:32.517831+0000 mon.a (mon.0) 3170 : cluster [DBG] osdmap e628: 8 total, 8 up, 7 in 2024-02-19T21:14:33.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:32.519576+0000 mon.a (mon.0) 3171 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-106"}]: dispatch 2024-02-19T21:14:33.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:32.616590+0000 mon.a (mon.0) 3172 : audit [DBG] from='client.? v1:172.21.15.163:0/2638313847' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:33.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:33.515298+0000 mon.a (mon.0) 3173 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:14:33.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:33.518932+0000 mon.a (mon.0) 3174 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-106"}]': finished 2024-02-19T21:14:33.873 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:33.518989+0000 mon.a (mon.0) 3175 : cluster [DBG] osdmap e629: 8 total, 8 up, 7 in 2024-02-19T21:14:34.386 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 7, 3, 4, 6] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:14:34.386 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:14:34.386 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-02-19T21:14:34.386 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd in 0 2024-02-19T21:14:34.403 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:14:34.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:34.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:14:34.507 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:14:34.536 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:34.557 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:14:34.559 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:34.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:14:34.661 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:34.686 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:33.289122+0000 mgr.y (mgr.14550) 546 : cluster [DBG] pgmap v946: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:14:34.686 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:34.358317+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e630: 8 total, 8 up, 7 in 2024-02-19T21:14:34.687 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:33.289122+0000 mgr.y (mgr.14550) 546 : cluster [DBG] pgmap v946: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:14:34.687 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:34.358317+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e630: 8 total, 8 up, 7 in 2024-02-19T21:14:34.688 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:34.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:33.289122+0000 mgr.y (mgr.14550) 546 : cluster [DBG] pgmap v946: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:14:34.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:34.358317+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e630: 8 total, 8 up, 7 in 2024-02-19T21:14:34.936 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:34.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:34.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:35.041 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:35.069 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:35.168 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:35.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:35.272 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:14:35.307 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:35.329 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:14:35.331 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:35.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:14:35.433 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:14:35.461 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:35.481 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:14:35.483 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:35.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:14:35.584 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:14:35.612 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:35.631 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:14:35.634 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:35.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:14:35.725 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:35.725 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:35.735 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:35.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.737+0000 7f0fbd23c700 1 -- 172.21.15.163:0/1332242200 >> v1:172.21.15.163:6790/0 conn(0x7f0fb80ff670 legacy=0x7f0fb8101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:35.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.737+0000 7f0fbd23c700 1 -- 172.21.15.163:0/1332242200 shutdown_connections 2024-02-19T21:14:35.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.737+0000 7f0fbd23c700 1 -- 172.21.15.163:0/1332242200 >> 172.21.15.163:0/1332242200 conn(0x7f0fb80f24b0 msgr2=0x7f0fb80f48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:35.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.737+0000 7f0fbd23c700 1 -- 172.21.15.163:0/1332242200 shutdown_connections 2024-02-19T21:14:35.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.737+0000 7f0fbd23c700 1 -- 172.21.15.163:0/1332242200 wait complete. 2024-02-19T21:14:35.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.737+0000 7f0fbd23c700 1 librados: starting msgr at 2024-02-19T21:14:35.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.737+0000 7f0fbd23c700 1 librados: starting objecter 2024-02-19T21:14:35.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.738+0000 7f0fbd23c700 1 Processor -- start 2024-02-19T21:14:35.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.738+0000 7f0fbd23c700 1 -- start start 2024-02-19T21:14:35.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.738+0000 7f0fbd23c700 1 librados: setting wanted keys 2024-02-19T21:14:35.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.738+0000 7f0fbd23c700 1 librados: calling monclient init 2024-02-19T21:14:35.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.738+0000 7f0fbd23c700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0fb80f4830 con 0x7f0fb80fc870 2024-02-19T21:14:35.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.738+0000 7f0fbd23c700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0fb80fbde0 con 0x7f0fb81031b0 2024-02-19T21:14:35.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.738+0000 7f0fbd23c700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0fb80fbf70 con 0x7f0fb80ff670 2024-02-19T21:14:35.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.739+0000 7f0fb6d9d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f0fb80fc870 0x7f0fb80fb6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:51078/0 (socket says 172.21.15.163:51078) 2024-02-19T21:14:35.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.739+0000 7f0fb6d9d700 1 -- 172.21.15.163:0/4283785965 learned_addr learned my addr 172.21.15.163:0/4283785965 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:35.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.739+0000 7f0f9ffff700 10 client.?.objecter ms_handle_connect 0x7f0fb80fc870 2024-02-19T21:14:35.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.739+0000 7f0f9ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:35.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.739+0000 7f0f9ffff700 10 client.?.objecter ms_handle_connect 0x7f0fb81031b0 2024-02-19T21:14:35.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.739+0000 7f0f9ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:35.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.739+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385353159 0 0) 0x7f0fb80f4830 con 0x7f0fb80fc870 2024-02-19T21:14:35.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.739+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa0003800 con 0x7f0fb80fc870 2024-02-19T21:14:35.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.739+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127964192 0 0) 0x7f0fb80fbde0 con 0x7f0fb81031b0 2024-02-19T21:14:35.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.739+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fb80f4830 con 0x7f0fb81031b0 2024-02-19T21:14:35.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.739+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3645992640 0 0) 0x7f0fb80f4830 con 0x7f0fb81031b0 2024-02-19T21:14:35.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.739+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fb80fbde0 con 0x7f0fb81031b0 2024-02-19T21:14:35.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.739+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0fb0001c40 con 0x7f0fb81031b0 2024-02-19T21:14:35.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.740+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2087187710 0 0) 0x7f0fa0003800 con 0x7f0fb80fc870 2024-02-19T21:14:35.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.740+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fb80f4830 con 0x7f0fb80fc870 2024-02-19T21:14:35.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.740+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0fa80019f0 con 0x7f0fb80fc870 2024-02-19T21:14:35.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.740+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 669045901 0 0) 0x7f0fb80fbde0 con 0x7f0fb81031b0 2024-02-19T21:14:35.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.740+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 >> v1:172.21.15.163:6790/0 conn(0x7f0fb80ff670 legacy=0x7f0fb81193f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T21:14:35.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.740+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 >> v1:172.21.15.163:6789/0 conn(0x7f0fb80fc870 legacy=0x7f0fb80fb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:35.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.740+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb80fc100 con 0x7f0fb81031b0 2024-02-19T21:14:35.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.740+0000 7f0fbd23c700 1 -- 172.21.15.163:0/4283785965 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb811e240 con 0x7f0fb81031b0 2024-02-19T21:14:35.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.740+0000 7f0fbd23c700 10 client.49135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:35.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.740+0000 7f0fbd23c700 1 -- 172.21.15.163:0/4283785965 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb811e610 con 0x7f0fb81031b0 2024-02-19T21:14:35.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.740+0000 7f0fbd23c700 1 librados: init done 2024-02-19T21:14:35.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.740+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f0fb0002300 con 0x7f0fb81031b0 2024-02-19T21:14:35.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.741+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0fb0003650 con 0x7f0fb81031b0 2024-02-19T21:14:35.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.741+0000 7f0fbd23c700 1 -- 172.21.15.163:0/4283785965 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb80f43c0 con 0x7f0fb81031b0 2024-02-19T21:14:35.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.742+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f0fb0001ee0 con 0x7f0fb81031b0 2024-02-19T21:14:35.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.742+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(631..631 src has 434..631) v4 ==== 8556+0+0 (unknown 3241299544 0 0) 0x7f0fb0076b90 con 0x7f0fb81031b0 2024-02-19T21:14:35.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.742+0000 7f0f9ffff700 10 client.49135.objecter ms_dispatch 0x7f0fb8001550 osd_map(631..631 src has 434..631) v4 2024-02-19T21:14:35.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.742+0000 7f0f9ffff700 3 client.49135.objecter handle_osd_map got epochs [631,631] > 0 2024-02-19T21:14:35.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.742+0000 7f0f9ffff700 3 client.49135.objecter handle_osd_map decoding full epoch 631 2024-02-19T21:14:35.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.742+0000 7f0f9ffff700 20 client.49135.objecter dump_active .. 0 homeless 2024-02-19T21:14:35.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.743+0000 7f0f9ffff700 10 client.49135.objecter ms_handle_connect 0x7f0fa0063b80 2024-02-19T21:14:35.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.747+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0fb0049dc0 con 0x7f0fb81031b0 2024-02-19T21:14:35.773 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:35.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:35.932+0000 7f0fbd23c700 1 -- 172.21.15.163:0/4283785965 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f0fb805f500 con 0x7f0fb81031b0 2024-02-19T21:14:36.021 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:36.023 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:34.843525+0000 mon.a (mon.0) 3177 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:14:36.023 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:35.367596+0000 mon.a (mon.0) 3178 : cluster [DBG] osdmap e631: 8 total, 8 up, 7 in 2024-02-19T21:14:36.023 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:35.371093+0000 mon.a (mon.0) 3179 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-108","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:14:36.024 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:34.843525+0000 mon.a (mon.0) 3177 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:14:36.024 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:35.367596+0000 mon.a (mon.0) 3178 : cluster [DBG] osdmap e631: 8 total, 8 up, 7 in 2024-02-19T21:14:36.024 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:35.371093+0000 mon.a (mon.0) 3179 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-108","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:14:36.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:36.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:36.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:34.843525+0000 mon.a (mon.0) 3177 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:14:36.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:35.367596+0000 mon.a (mon.0) 3178 : cluster [DBG] osdmap e631: 8 total, 8 up, 7 in 2024-02-19T21:14:36.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:35.371093+0000 mon.a (mon.0) 3179 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-108","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:14:36.126 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:36.153 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:36.252 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:36.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:36.358 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:36.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:36.371+0000 7f0f9ffff700 1 -- 172.21.15.163:0/4283785965 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v632) v1 ==== 97+0+0 (unknown 1738468112 0 0) 0x7f0fb00481d0 con 0x7f0fb81031b0 2024-02-19T21:14:36.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:36.377+0000 7f0fbd23c700 10 librados: watch_flush enter 2024-02-19T21:14:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:36.382+0000 7f0fbd23c700 10 librados: watch_flush exit 2024-02-19T21:14:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:36.382+0000 7f0fbd23c700 20 client.49135.objecter shutdown clearing up homeless session... 2024-02-19T21:14:36.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:36.382+0000 7f0fbd23c700 10 client.49135.objecter successfully canceled tick 2024-02-19T21:14:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:36.384+0000 7f0fbd23c700 1 -- 172.21.15.163:0/4283785965 >> v1:172.21.15.163:6800/1552514681 conn(0x7f0fa0063b80 legacy=0x7f0fa0066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:36.384+0000 7f0fbd23c700 1 -- 172.21.15.163:0/4283785965 >> v1:172.21.15.193:6789/0 conn(0x7f0fb81031b0 legacy=0x7f0fb811cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:36.385+0000 7f0fbd23c700 1 -- 172.21.15.163:0/4283785965 shutdown_connections 2024-02-19T21:14:36.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:36.385+0000 7f0fbd23c700 1 -- 172.21.15.163:0/4283785965 >> 172.21.15.163:0/4283785965 conn(0x7f0fb80f24b0 msgr2=0x7f0fb8106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:36.389+0000 7f0fbd23c700 1 -- 172.21.15.163:0/4283785965 shutdown_connections 2024-02-19T21:14:36.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:36.389+0000 7f0fbd23c700 1 -- 172.21.15.163:0/4283785965 wait complete. 2024-02-19T21:14:36.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:36.390+0000 7f0fbd23c700 1 librados: shutdown 2024-02-19T21:14:36.394 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:14:36.397 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:37.248 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:35.289909+0000 mgr.y (mgr.14550) 547 : cluster [DBG] pgmap v949: 182 pgs: 182 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 1 op/s 2024-02-19T21:14:37.248 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:35.933679+0000 mon.b (mon.1) 736 : audit [INF] from='client.? v1:172.21.15.163:0/4283785965' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:37.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:35.934002+0000 mon.a (mon.0) 3180 : audit [INF] from='client.49135 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:37.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:36.365129+0000 mon.a (mon.0) 3181 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:14:37.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:36.371370+0000 mon.a (mon.0) 3182 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-108","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:14:37.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:36.371535+0000 mon.a (mon.0) 3183 : audit [INF] from='client.49135 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:14:37.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:36.371586+0000 mon.a (mon.0) 3184 : cluster [DBG] osdmap e632: 8 total, 8 up, 7 in 2024-02-19T21:14:37.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:35.289909+0000 mgr.y (mgr.14550) 547 : cluster [DBG] pgmap v949: 182 pgs: 182 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 1 op/s 2024-02-19T21:14:37.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:35.933679+0000 mon.b (mon.1) 736 : audit [INF] from='client.? v1:172.21.15.163:0/4283785965' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:37.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:35.934002+0000 mon.a (mon.0) 3180 : audit [INF] from='client.49135 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:37.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:36.365129+0000 mon.a (mon.0) 3181 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:14:37.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:36.371370+0000 mon.a (mon.0) 3182 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-108","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:14:37.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:36.371535+0000 mon.a (mon.0) 3183 : audit [INF] from='client.49135 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:14:37.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:36.371586+0000 mon.a (mon.0) 3184 : cluster [DBG] osdmap e632: 8 total, 8 up, 7 in 2024-02-19T21:14:37.321 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:35.289909+0000 mgr.y (mgr.14550) 547 : cluster [DBG] pgmap v949: 182 pgs: 182 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 1 op/s 2024-02-19T21:14:37.321 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:35.933679+0000 mon.b (mon.1) 736 : audit [INF] from='client.? v1:172.21.15.163:0/4283785965' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:37.321 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:35.934002+0000 mon.a (mon.0) 3180 : audit [INF] from='client.49135 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:37.321 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:36.365129+0000 mon.a (mon.0) 3181 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:14:37.321 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:36.371370+0000 mon.a (mon.0) 3182 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-108","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:14:37.321 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:36.371535+0000 mon.a (mon.0) 3183 : audit [INF] from='client.49135 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:14:37.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:36.371586+0000 mon.a (mon.0) 3184 : cluster [DBG] osdmap e632: 8 total, 8 up, 7 in 2024-02-19T21:14:37.524 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:14:37.576 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:37.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:37.680 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:14:37.711 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:37.732 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:14:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:37.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:14:37.836 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:37.863 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:37.936 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:37.939 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:37.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:38.040 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:38.067 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:38.071 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:14:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:14:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:14:38.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:37.290433+0000 mgr.y (mgr.14550) 548 : cluster [DBG] pgmap v952: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 1 op/s 2024-02-19T21:14:38.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:37.371604+0000 mon.a (mon.0) 3185 : cluster [DBG] osdmap e633: 8 total, 8 up, 7 in 2024-02-19T21:14:38.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:37.379073+0000 mon.a (mon.0) 3186 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-108", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:14:38.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:37.290433+0000 mgr.y (mgr.14550) 548 : cluster [DBG] pgmap v952: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 1 op/s 2024-02-19T21:14:38.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:37.371604+0000 mon.a (mon.0) 3185 : cluster [DBG] osdmap e633: 8 total, 8 up, 7 in 2024-02-19T21:14:38.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:37.379073+0000 mon.a (mon.0) 3186 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-108", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:14:38.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:37.290433+0000 mgr.y (mgr.14550) 548 : cluster [DBG] pgmap v952: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 767 B/s rd, 1 op/s 2024-02-19T21:14:38.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:37.371604+0000 mon.a (mon.0) 3185 : cluster [DBG] osdmap e633: 8 total, 8 up, 7 in 2024-02-19T21:14:38.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:37.379073+0000 mon.a (mon.0) 3186 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-108", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:14:39.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.481+0000 7f15f6e1e700 1 -- 172.21.15.163:0/2982939571 >> v1:172.21.15.163:6790/0 conn(0x7f15f00f9530 legacy=0x7f15f0102a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:39.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.481+0000 7f15f6e1e700 1 -- 172.21.15.163:0/2982939571 shutdown_connections 2024-02-19T21:14:39.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.481+0000 7f15f6e1e700 1 -- 172.21.15.163:0/2982939571 >> 172.21.15.163:0/2982939571 conn(0x7f15f00f2470 msgr2=0x7f15f00f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:39.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.481+0000 7f15f6e1e700 1 -- 172.21.15.163:0/2982939571 shutdown_connections 2024-02-19T21:14:39.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.481+0000 7f15f6e1e700 1 -- 172.21.15.163:0/2982939571 wait complete. 2024-02-19T21:14:39.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15f6e1e700 1 librados: starting msgr at 2024-02-19T21:14:39.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15f6e1e700 1 librados: starting objecter 2024-02-19T21:14:39.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15f6e1e700 1 Processor -- start 2024-02-19T21:14:39.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15f6e1e700 1 -- start start 2024-02-19T21:14:39.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15f6e1e700 1 librados: setting wanted keys 2024-02-19T21:14:39.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15f6e1e700 1 librados: calling monclient init 2024-02-19T21:14:39.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15f6e1e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15f0119440 con 0x7f15f01042b0 2024-02-19T21:14:39.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15f6e1e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15f011a5e0 con 0x7f15f00f8980 2024-02-19T21:14:39.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15f6e1e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15f011b7c0 con 0x7f15f00f9530 2024-02-19T21:14:39.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15effff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f15f00f9530 0x7f15f01028d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:52338/0 (socket says 172.21.15.163:52338) 2024-02-19T21:14:39.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15effff700 1 -- 172.21.15.163:0/2266228087 learned_addr learned my addr 172.21.15.163:0/2266228087 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:39.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15edffb700 10 client.?.objecter ms_handle_connect 0x7f15f01042b0 2024-02-19T21:14:39.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15edffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:39.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15edffb700 10 client.?.objecter ms_handle_connect 0x7f15f00f9530 2024-02-19T21:14:39.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15edffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:39.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15edffb700 10 client.?.objecter ms_handle_connect 0x7f15f00f8980 2024-02-19T21:14:39.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15edffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:39.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022244818 0 0) 0x7f15f011b7c0 con 0x7f15f00f9530 2024-02-19T21:14:39.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e0003030 con 0x7f15f00f9530 2024-02-19T21:14:39.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624499744 0 0) 0x7f15f0119440 con 0x7f15f01042b0 2024-02-19T21:14:39.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.482+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f011b7c0 con 0x7f15f01042b0 2024-02-19T21:14:39.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956594555 0 0) 0x7f15f011a5e0 con 0x7f15f00f8980 2024-02-19T21:14:39.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f0119440 con 0x7f15f00f8980 2024-02-19T21:14:39.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1774633120 0 0) 0x7f15e0003030 con 0x7f15f00f9530 2024-02-19T21:14:39.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15f011a5e0 con 0x7f15f00f9530 2024-02-19T21:14:39.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f15dc002c20 con 0x7f15f00f9530 2024-02-19T21:14:39.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2010017510 0 0) 0x7f15f011b7c0 con 0x7f15f01042b0 2024-02-19T21:14:39.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15e0003030 con 0x7f15f01042b0 2024-02-19T21:14:39.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f15e80019a0 con 0x7f15f01042b0 2024-02-19T21:14:39.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1716872139 0 0) 0x7f15f0119440 con 0x7f15f00f8980 2024-02-19T21:14:39.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15f011b7c0 con 0x7f15f00f8980 2024-02-19T21:14:39.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f15e4001840 con 0x7f15f00f8980 2024-02-19T21:14:39.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1681130133 0 0) 0x7f15f011a5e0 con 0x7f15f00f9530 2024-02-19T21:14:39.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 >> v1:172.21.15.193:6789/0 conn(0x7f15f00f8980 legacy=0x7f15f01021c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:39.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 >> v1:172.21.15.163:6789/0 conn(0x7f15f01042b0 legacy=0x7f15f0117d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:39.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f011c9a0 con 0x7f15f00f9530 2024-02-19T21:14:39.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15f6e1e700 1 -- 172.21.15.163:0/2266228087 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f011b9b0 con 0x7f15f00f9530 2024-02-19T21:14:39.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f15dc001b60 con 0x7f15f00f9530 2024-02-19T21:14:39.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f15dc0038b0 con 0x7f15f00f9530 2024-02-19T21:14:39.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15f6e1e700 10 client.50330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:39.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15f6e1e700 1 -- 172.21.15.163:0/2266228087 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f011be20 con 0x7f15f00f9530 2024-02-19T21:14:39.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.483+0000 7f15f6e1e700 1 librados: init done 2024-02-19T21:14:39.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.484+0000 7f15f6e1e700 1 -- 172.21.15.163:0/2266228087 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f005f700 con 0x7f15f00f9530 2024-02-19T21:14:39.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.484+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f15dc003af0 con 0x7f15f00f9530 2024-02-19T21:14:39.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.485+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(635..635 src has 434..635) v4 ==== 8564+0+0 (unknown 2326307065 0 0) 0x7f15dc079120 con 0x7f15f00f9530 2024-02-19T21:14:39.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.485+0000 7f15edffb700 10 client.50330.objecter ms_dispatch 0x7f15f0001550 osd_map(635..635 src has 434..635) v4 2024-02-19T21:14:39.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.485+0000 7f15edffb700 3 client.50330.objecter handle_osd_map got epochs [635,635] > 0 2024-02-19T21:14:39.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.485+0000 7f15edffb700 3 client.50330.objecter handle_osd_map decoding full epoch 635 2024-02-19T21:14:39.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.485+0000 7f15edffb700 20 client.50330.objecter dump_active .. 0 homeless 2024-02-19T21:14:39.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.485+0000 7f15edffb700 10 client.50330.objecter ms_handle_connect 0x7f15e0063d70 2024-02-19T21:14:39.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.489+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f15dc04c400 con 0x7f15f00f9530 2024-02-19T21:14:39.574 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:14:39.675 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:39.674+0000 7f15f6e1e700 1 -- 172.21.15.163:0/2266228087 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd in", "ids": ["0"]} v 0) v1 -- 0x7f15f0047540 con 0x7f15f00f9530 2024-02-19T21:14:39.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:38.373712+0000 mon.a (mon.0) 3187 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-108", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:14:39.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:38.373776+0000 mon.a (mon.0) 3188 : cluster [DBG] osdmap e634: 8 total, 8 up, 7 in 2024-02-19T21:14:39.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:38.375192+0000 mon.a (mon.0) 3189 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-108"}]: dispatch 2024-02-19T21:14:39.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:39.107153+0000 mon.a (mon.0) 3190 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:14:39.676 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:39.358406+0000 mon.a (mon.0) 3191 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-108"}]': finished 2024-02-19T21:14:39.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:39.358463+0000 mon.a (mon.0) 3192 : cluster [DBG] osdmap e635: 8 total, 8 up, 7 in 2024-02-19T21:14:39.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:39.358979+0000 mon.a (mon.0) 3193 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-108", "mode": "writeback"}]: dispatch 2024-02-19T21:14:39.677 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:38.373712+0000 mon.a (mon.0) 3187 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-108", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:14:39.677 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:38.373776+0000 mon.a (mon.0) 3188 : cluster [DBG] osdmap e634: 8 total, 8 up, 7 in 2024-02-19T21:14:39.677 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:38.375192+0000 mon.a (mon.0) 3189 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-108"}]: dispatch 2024-02-19T21:14:39.677 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:39.107153+0000 mon.a (mon.0) 3190 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:14:39.677 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:39.358406+0000 mon.a (mon.0) 3191 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-108"}]': finished 2024-02-19T21:14:39.677 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:39.358463+0000 mon.a (mon.0) 3192 : cluster [DBG] osdmap e635: 8 total, 8 up, 7 in 2024-02-19T21:14:39.677 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:39.358979+0000 mon.a (mon.0) 3193 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-108", "mode": "writeback"}]: dispatch 2024-02-19T21:14:39.677 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:14:39.675+0000 7fb606608700 -1 mon.a@0(leader).osd e635 definitely_dead 0 2024-02-19T21:14:39.695 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:14:39.695 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:14:39.695 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-02-19T21:14:39.695 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:14:39.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:38.373712+0000 mon.a (mon.0) 3187 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-108", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:14:39.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:38.373776+0000 mon.a (mon.0) 3188 : cluster [DBG] osdmap e634: 8 total, 8 up, 7 in 2024-02-19T21:14:39.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:38.375192+0000 mon.a (mon.0) 3189 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-108"}]: dispatch 2024-02-19T21:14:39.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:39.107153+0000 mon.a (mon.0) 3190 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:14:39.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:39.358406+0000 mon.a (mon.0) 3191 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-108"}]': finished 2024-02-19T21:14:39.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:39.358463+0000 mon.a (mon.0) 3192 : cluster [DBG] osdmap e635: 8 total, 8 up, 7 in 2024-02-19T21:14:39.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:39.358979+0000 mon.a (mon.0) 3193 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-108", "mode": "writeback"}]: dispatch 2024-02-19T21:14:40.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:40.361+0000 7f15edffb700 1 -- 172.21.15.163:0/2266228087 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd in", "ids": ["0"]}]=0 marked in osd.0. v636) v1 ==== 85+0+0 (unknown 2990859439 0 0) 0x7f15dc04a870 con 0x7f15f00f9530 2024-02-19T21:14:40.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:40.366+0000 7f15f6e1e700 10 librados: watch_flush enter 2024-02-19T21:14:40.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:40.366+0000 7f15f6e1e700 10 librados: watch_flush exit 2024-02-19T21:14:40.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:40.366+0000 7f15f6e1e700 20 client.50330.objecter shutdown clearing up homeless session... 2024-02-19T21:14:40.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:40.366+0000 7f15f6e1e700 10 client.50330.objecter successfully canceled tick 2024-02-19T21:14:40.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:40.368+0000 7f15f6e1e700 1 -- 172.21.15.163:0/2266228087 >> v1:172.21.15.163:6800/1552514681 conn(0x7f15e0063d70 legacy=0x7f15e0066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:40.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:40.368+0000 7f15f6e1e700 1 -- 172.21.15.163:0/2266228087 >> v1:172.21.15.163:6790/0 conn(0x7f15f00f9530 legacy=0x7f15f01028d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:40.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:40.369+0000 7f15f6e1e700 1 -- 172.21.15.163:0/2266228087 shutdown_connections 2024-02-19T21:14:40.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:40.369+0000 7f15f6e1e700 1 -- 172.21.15.163:0/2266228087 >> 172.21.15.163:0/2266228087 conn(0x7f15f00f2470 msgr2=0x7f15f00f4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:40.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:40.369+0000 7f15f6e1e700 1 -- 172.21.15.163:0/2266228087 shutdown_connections 2024-02-19T21:14:40.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:40.370+0000 7f15f6e1e700 1 -- 172.21.15.163:0/2266228087 wait complete. 2024-02-19T21:14:40.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:40.370+0000 7f15f6e1e700 1 librados: shutdown 2024-02-19T21:14:40.380 INFO:teuthology.orchestra.run.smithi163.stderr:marked in osd.0. 2024-02-19T21:14:40.393 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:39.290942+0000 mgr.y (mgr.14550) 549 : cluster [DBG] pgmap v955: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:14:40.393 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:39.676525+0000 mon.c (mon.2) 417 : audit [DBG] from='client.? v1:172.21.15.163:0/2266228087' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-19T21:14:40.393 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:39.676913+0000 mon.a (mon.0) 3194 : audit [DBG] from='client.50330 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-19T21:14:40.393 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:40.356159+0000 mon.a (mon.0) 3195 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:14:40.393 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:40.361664+0000 mon.a (mon.0) 3196 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-108", "mode": "writeback"}]': finished 2024-02-19T21:14:40.393 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:40.361729+0000 mon.a (mon.0) 3197 : audit [INF] from='client.50330 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-02-19T21:14:40.394 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:40.361787+0000 mon.a (mon.0) 3198 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2024-02-19T21:14:40.416 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:40.423 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-19T21:14:40.424 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:40.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:40.526 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:14:40.555 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:40.575 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:14:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:40.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:14:40.616 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:40.622 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:40.679 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:14:40.705 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:14:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:14:40.705 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:39.290942+0000 mgr.y (mgr.14550) 549 : cluster [DBG] pgmap v955: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:14:40.705 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:39.676525+0000 mon.c (mon.2) 417 : audit [DBG] from='client.? v1:172.21.15.163:0/2266228087' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-19T21:14:40.705 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:39.676913+0000 mon.a (mon.0) 3194 : audit [DBG] from='client.50330 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-19T21:14:40.705 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:40.356159+0000 mon.a (mon.0) 3195 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:14:40.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:40.361664+0000 mon.a (mon.0) 3196 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-108", "mode": "writeback"}]': finished 2024-02-19T21:14:40.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:40.361729+0000 mon.a (mon.0) 3197 : audit [INF] from='client.50330 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-02-19T21:14:40.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:40.361787+0000 mon.a (mon.0) 3198 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2024-02-19T21:14:40.706 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:40.726 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:14:40.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:40.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:14:40.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:39.290942+0000 mgr.y (mgr.14550) 549 : cluster [DBG] pgmap v955: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:14:40.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:39.676525+0000 mon.c (mon.2) 417 : audit [DBG] from='client.? v1:172.21.15.163:0/2266228087' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-19T21:14:40.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:39.676913+0000 mon.a (mon.0) 3194 : audit [DBG] from='client.50330 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-19T21:14:40.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:40.356159+0000 mon.a (mon.0) 3195 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:14:40.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:40.361664+0000 mon.a (mon.0) 3196 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-108", "mode": "writeback"}]': finished 2024-02-19T21:14:40.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:40.361729+0000 mon.a (mon.0) 3197 : audit [INF] from='client.50330 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-02-19T21:14:40.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:40.361787+0000 mon.a (mon.0) 3198 : cluster [DBG] osdmap e636: 8 total, 8 up, 8 in 2024-02-19T21:14:40.829 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:40.857 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:41.158 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-02-19T21:14:41.176 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:41.606 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:41.609 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:41.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:41.711 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:14:41.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:40.142165+0000 osd.5 (osd.5) 254 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:41.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:40.142500+0000 osd.5 (osd.5) 255 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:41.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:40.640686+0000 mgr.y (mgr.14550) 550 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:41.731 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:41.365205+0000 mon.a (mon.0) 3199 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2024-02-19T21:14:41.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:40.142165+0000 osd.5 (osd.5) 254 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:41.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:40.142500+0000 osd.5 (osd.5) 255 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:41.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:40.640686+0000 mgr.y (mgr.14550) 550 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:41.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:41.365205+0000 mon.a (mon.0) 3199 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2024-02-19T21:14:41.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:40.142165+0000 osd.5 (osd.5) 254 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:41.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:40.142500+0000 osd.5 (osd.5) 255 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:41.734 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:40.640686+0000 mgr.y (mgr.14550) 550 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:41.735 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:41.365205+0000 mon.a (mon.0) 3199 : cluster [DBG] osdmap e637: 8 total, 8 up, 8 in 2024-02-19T21:14:41.736 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:41.761 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:14:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:41.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:14:41.864 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:14:41.891 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:41.915 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:14:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:41.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:14:42.019 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:42.046 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:42.118 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:42.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:42.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:42.222 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:42.250 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:42.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:41.291356+0000 mgr.y (mgr.14550) 551 : cluster [DBG] pgmap v958: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:14:42.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:41.445469+0000 mon.a (mon.0) 3200 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:14:42.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:42.363699+0000 mon.a (mon.0) 3201 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:42.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:42.363755+0000 mon.a (mon.0) 3202 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2024-02-19T21:14:42.411 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:42.365391+0000 mon.a (mon.0) 3203 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-108"}]: dispatch 2024-02-19T21:14:42.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:41.291356+0000 mgr.y (mgr.14550) 551 : cluster [DBG] pgmap v958: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:14:42.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:41.445469+0000 mon.a (mon.0) 3200 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:14:42.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:42.363699+0000 mon.a (mon.0) 3201 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:42.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:42.363755+0000 mon.a (mon.0) 3202 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2024-02-19T21:14:42.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:42.365391+0000 mon.a (mon.0) 3203 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-108"}]: dispatch 2024-02-19T21:14:42.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:41.291356+0000 mgr.y (mgr.14550) 551 : cluster [DBG] pgmap v958: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:14:42.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:41.445469+0000 mon.a (mon.0) 3200 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:14:42.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:42.363699+0000 mon.a (mon.0) 3201 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:42.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:42.363755+0000 mon.a (mon.0) 3202 : cluster [DBG] osdmap e638: 8 total, 8 up, 8 in 2024-02-19T21:14:42.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:42.365391+0000 mon.a (mon.0) 3203 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-108"}]: dispatch 2024-02-19T21:14:43.431 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:43.434 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:43.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:43.535 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:43.565 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:43.655 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:42.363498+0000 osd.7 (osd.7) 288 : cluster [DBG] 2.0 starting backfill to osd.0 from (0'0,0'0] MAX to 397'2 2024-02-19T21:14:43.655 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:43.360853+0000 mon.a (mon.0) 3204 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:14:43.655 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:43.366327+0000 mon.a (mon.0) 3205 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-108"}]': finished 2024-02-19T21:14:43.655 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:43.366413+0000 mon.a (mon.0) 3206 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2024-02-19T21:14:43.656 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:14:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:14:43] "GET /metrics HTTP/1.1" 200 191638 "" "Prometheus/2.33.4" 2024-02-19T21:14:43.656 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:42.363498+0000 osd.7 (osd.7) 288 : cluster [DBG] 2.0 starting backfill to osd.0 from (0'0,0'0] MAX to 397'2 2024-02-19T21:14:43.656 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:43.360853+0000 mon.a (mon.0) 3204 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:14:43.656 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:43.366327+0000 mon.a (mon.0) 3205 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-108"}]': finished 2024-02-19T21:14:43.656 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:43.366413+0000 mon.a (mon.0) 3206 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2024-02-19T21:14:43.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:42.363498+0000 osd.7 (osd.7) 288 : cluster [DBG] 2.0 starting backfill to osd.0 from (0'0,0'0] MAX to 397'2 2024-02-19T21:14:43.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:43.360853+0000 mon.a (mon.0) 3204 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:14:43.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:43.366327+0000 mon.a (mon.0) 3205 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-108"}]': finished 2024-02-19T21:14:43.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:43.366413+0000 mon.a (mon.0) 3206 : cluster [DBG] osdmap e639: 8 total, 8 up, 8 in 2024-02-19T21:14:44.018 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:44.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:44.125 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:14:44.156 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:44.175 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:14:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:44.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:14:44.279 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:44.309 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:44.433 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:44.436 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:44.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:44.537 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:14:44.566 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:44.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:43.291912+0000 mgr.y (mgr.14550) 552 : cluster [DBG] pgmap v961: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:14:44.817 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:44.360109+0000 mon.a (mon.0) 3207 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2024-02-19T21:14:44.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:43.291912+0000 mgr.y (mgr.14550) 552 : cluster [DBG] pgmap v961: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:14:44.818 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:44.360109+0000 mon.a (mon.0) 3207 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2024-02-19T21:14:45.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:43.291912+0000 mgr.y (mgr.14550) 552 : cluster [DBG] pgmap v961: 214 pgs: 214 active+clean; 5.3 KiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:14:45.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:44.360109+0000 mon.a (mon.0) 3207 : cluster [DBG] osdmap e640: 8 total, 8 up, 8 in 2024-02-19T21:14:45.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.346+0000 7fa3b721a700 1 -- 172.21.15.163:0/1799394945 >> v1:172.21.15.163:6790/0 conn(0x7fa3b01042c0 legacy=0x7fa3b0106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:45.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.346+0000 7fa3b721a700 1 -- 172.21.15.163:0/1799394945 shutdown_connections 2024-02-19T21:14:45.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.346+0000 7fa3b721a700 1 -- 172.21.15.163:0/1799394945 >> 172.21.15.163:0/1799394945 conn(0x7fa3b00f35a0 msgr2=0x7fa3b00f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:45.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.346+0000 7fa3b721a700 1 -- 172.21.15.163:0/1799394945 shutdown_connections 2024-02-19T21:14:45.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.346+0000 7fa3b721a700 1 -- 172.21.15.163:0/1799394945 wait complete. 2024-02-19T21:14:45.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3b721a700 1 librados: starting msgr at 2024-02-19T21:14:45.349 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3b721a700 1 librados: starting objecter 2024-02-19T21:14:45.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3b721a700 1 Processor -- start 2024-02-19T21:14:45.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3b721a700 1 -- start start 2024-02-19T21:14:45.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3b721a700 1 librados: setting wanted keys 2024-02-19T21:14:45.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3b721a700 1 librados: calling monclient init 2024-02-19T21:14:45.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3b721a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa3b00fcee0 con 0x7fa3b01042c0 2024-02-19T21:14:45.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3b721a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa3b00fd0b0 con 0x7fa3b00fd980 2024-02-19T21:14:45.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3b721a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa3b00fd2a0 con 0x7fa3b0100780 2024-02-19T21:14:45.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3b57b7700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fa3b01042c0 0x7fa3b0118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:39896/0 (socket says 172.21.15.163:39896) 2024-02-19T21:14:45.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3b57b7700 1 -- 172.21.15.163:0/3472021223 learned_addr learned my addr 172.21.15.163:0/3472021223 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:45.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3affff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fa3b0100780 0x7fa3b0114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:40272/0 (socket says 172.21.15.163:40272) 2024-02-19T21:14:45.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3adffb700 10 client.?.objecter ms_handle_connect 0x7fa3b01042c0 2024-02-19T21:14:45.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3adffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:45.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3adffb700 10 client.?.objecter ms_handle_connect 0x7fa3b0100780 2024-02-19T21:14:45.350 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3adffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:45.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3adffb700 10 client.?.objecter ms_handle_connect 0x7fa3b00fd980 2024-02-19T21:14:45.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3adffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:45.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796818190 0 0) 0x7fa3b00fcee0 con 0x7fa3b01042c0 2024-02-19T21:14:45.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3a0003030 con 0x7fa3b01042c0 2024-02-19T21:14:45.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131340443 0 0) 0x7fa3b00fd2a0 con 0x7fa3b0100780 2024-02-19T21:14:45.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b00fcee0 con 0x7fa3b0100780 2024-02-19T21:14:45.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609329337 0 0) 0x7fa3b00fd0b0 con 0x7fa3b00fd980 2024-02-19T21:14:45.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.347+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b00fd2a0 con 0x7fa3b00fd980 2024-02-19T21:14:45.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2350812616 0 0) 0x7fa3a0003030 con 0x7fa3b01042c0 2024-02-19T21:14:45.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3b00fd0b0 con 0x7fa3b01042c0 2024-02-19T21:14:45.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3a8001950 con 0x7fa3b01042c0 2024-02-19T21:14:45.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3211871610 0 0) 0x7fa3b00fcee0 con 0x7fa3b0100780 2024-02-19T21:14:45.351 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3a0003030 con 0x7fa3b0100780 2024-02-19T21:14:45.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa39c001860 con 0x7fa3b0100780 2024-02-19T21:14:45.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3421705285 0 0) 0x7fa3b00fd2a0 con 0x7fa3b00fd980 2024-02-19T21:14:45.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3b00fcee0 con 0x7fa3b00fd980 2024-02-19T21:14:45.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3a40017c0 con 0x7fa3b00fd980 2024-02-19T21:14:45.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4086441479 0 0) 0x7fa3b00fd0b0 con 0x7fa3b01042c0 2024-02-19T21:14:45.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 >> v1:172.21.15.163:6790/0 conn(0x7fa3b0100780 legacy=0x7fa3b0114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:45.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 >> v1:172.21.15.193:6789/0 conn(0x7fa3b00fd980 legacy=0x7fa3b00fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:45.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b011cb90 con 0x7fa3b01042c0 2024-02-19T21:14:45.352 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3b721a700 1 -- 172.21.15.163:0/3472021223 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b0119bc0 con 0x7fa3b01042c0 2024-02-19T21:14:45.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fa3a8002aa0 con 0x7fa3b01042c0 2024-02-19T21:14:45.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3a8003df0 con 0x7fa3b01042c0 2024-02-19T21:14:45.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3b721a700 10 client.23619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:45.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3b721a700 1 -- 172.21.15.163:0/3472021223 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b011a030 con 0x7fa3b01042c0 2024-02-19T21:14:45.353 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.348+0000 7fa3b721a700 1 librados: init done 2024-02-19T21:14:45.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.349+0000 7fa3b721a700 1 -- 172.21.15.163:0/3472021223 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b005f660 con 0x7fa3b01042c0 2024-02-19T21:14:45.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.349+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fa3a8002600 con 0x7fa3b01042c0 2024-02-19T21:14:45.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.350+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(640..640 src has 434..640) v4 ==== 8182+0+0 (unknown 2125993142 0 0) 0x7fa3a80795a0 con 0x7fa3b01042c0 2024-02-19T21:14:45.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.350+0000 7fa3adffb700 10 client.23619.objecter ms_dispatch 0x7fa3b0001550 osd_map(640..640 src has 434..640) v4 2024-02-19T21:14:45.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.350+0000 7fa3adffb700 3 client.23619.objecter handle_osd_map got epochs [640,640] > 0 2024-02-19T21:14:45.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.350+0000 7fa3adffb700 3 client.23619.objecter handle_osd_map decoding full epoch 640 2024-02-19T21:14:45.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.350+0000 7fa3adffb700 20 client.23619.objecter dump_active .. 0 homeless 2024-02-19T21:14:45.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.350+0000 7fa3adffb700 10 client.23619.objecter ms_handle_connect 0x7fa3a0063d80 2024-02-19T21:14:45.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.354+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa3a804c9e0 con 0x7fa3b01042c0 2024-02-19T21:14:45.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:45.540+0000 7fa3b721a700 1 -- 172.21.15.163:0/3472021223 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fa3b0047540 con 0x7fa3b01042c0 2024-02-19T21:14:45.621 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:14:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:45.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:14:45.726 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:14:45.758 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:45.780 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:14:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:45.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:14:45.884 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:45.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:43.763149+0000 osd.1 (osd.1) 286 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:45.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:43.763338+0000 osd.1 (osd.1) 287 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:45.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:44.654956+0000 mon.a (mon.0) 3208 : audit [DBG] from='client.? v1:172.21.15.163:0/2777576014' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:45.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:45.362418+0000 mon.a (mon.0) 3209 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2024-02-19T21:14:45.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:45.365891+0000 mon.a (mon.0) 3210 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-110","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:14:45.912 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:45.541649+0000 mon.a (mon.0) 3211 : audit [INF] from='client.? v1:172.21.15.163:0/3472021223' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:14:45.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:43.763149+0000 osd.1 (osd.1) 286 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:45.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:43.763338+0000 osd.1 (osd.1) 287 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:45.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:44.654956+0000 mon.a (mon.0) 3208 : audit [DBG] from='client.? v1:172.21.15.163:0/2777576014' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:45.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:45.362418+0000 mon.a (mon.0) 3209 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2024-02-19T21:14:45.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:45.365891+0000 mon.a (mon.0) 3210 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-110","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:14:45.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:45.541649+0000 mon.a (mon.0) 3211 : audit [INF] from='client.? v1:172.21.15.163:0/3472021223' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:14:45.914 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:46.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:43.763149+0000 osd.1 (osd.1) 286 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:46.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:43.763338+0000 osd.1 (osd.1) 287 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:46.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:44.654956+0000 mon.a (mon.0) 3208 : audit [DBG] from='client.? v1:172.21.15.163:0/2777576014' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:46.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:45.362418+0000 mon.a (mon.0) 3209 : cluster [DBG] osdmap e641: 8 total, 8 up, 8 in 2024-02-19T21:14:46.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:45.365891+0000 mon.a (mon.0) 3210 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-110","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:14:46.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:45.541649+0000 mon.a (mon.0) 3211 : audit [INF] from='client.? v1:172.21.15.163:0/3472021223' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:14:46.158 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 7, 3, 4, 6, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:14:46.159 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 1 1 2024-02-19T21:14:46.162 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:46.165 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:46.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:46.266 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:46.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.649+0000 7f319e135700 1 -- 172.21.15.163:0/3986049166 >> v1:172.21.15.163:6789/0 conn(0x7f31980fd980 legacy=0x7f31980fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:46.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.649+0000 7f319e135700 1 -- 172.21.15.163:0/3986049166 shutdown_connections 2024-02-19T21:14:46.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.649+0000 7f319e135700 1 -- 172.21.15.163:0/3986049166 >> 172.21.15.163:0/3986049166 conn(0x7f31980f35a0 msgr2=0x7f31980f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:46.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.649+0000 7f319e135700 1 -- 172.21.15.163:0/3986049166 shutdown_connections 2024-02-19T21:14:46.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.649+0000 7f319e135700 1 -- 172.21.15.163:0/3986049166 wait complete. 2024-02-19T21:14:46.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.650+0000 7f319e135700 1 librados: starting msgr at 2024-02-19T21:14:46.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.650+0000 7f319e135700 1 librados: starting objecter 2024-02-19T21:14:46.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.650+0000 7f319e135700 1 Processor -- start 2024-02-19T21:14:46.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.650+0000 7f319e135700 1 -- start start 2024-02-19T21:14:46.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.650+0000 7f319e135700 1 librados: setting wanted keys 2024-02-19T21:14:46.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.650+0000 7f319e135700 1 librados: calling monclient init 2024-02-19T21:14:46.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f319e135700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f31981172e0 con 0x7f3198100780 2024-02-19T21:14:46.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f319e135700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f31981184c0 con 0x7f31980fd980 2024-02-19T21:14:46.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f319e135700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f31981196c0 con 0x7f31981042c0 2024-02-19T21:14:46.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3196ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f3198100780 0x7f31980fa070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:39872/0 (socket says 172.21.15.163:39872) 2024-02-19T21:14:46.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3196ffd700 1 -- 172.21.15.163:0/2777576014 learned_addr learned my addr 172.21.15.163:0/2777576014 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:46.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 10 client.?.objecter ms_handle_connect 0x7f3198100780 2024-02-19T21:14:46.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:46.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 10 client.?.objecter ms_handle_connect 0x7f31981042c0 2024-02-19T21:14:46.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:46.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 10 client.?.objecter ms_handle_connect 0x7f31980fd980 2024-02-19T21:14:46.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:46.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200960832 0 0) 0x7f31981172e0 con 0x7f3198100780 2024-02-19T21:14:46.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31800033f0 con 0x7f3198100780 2024-02-19T21:14:46.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775556043 0 0) 0x7f31981196c0 con 0x7f31981042c0 2024-02-19T21:14:46.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31981172e0 con 0x7f31981042c0 2024-02-19T21:14:46.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119980062 0 0) 0x7f31981184c0 con 0x7f31980fd980 2024-02-19T21:14:46.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31981196c0 con 0x7f31980fd980 2024-02-19T21:14:46.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3756222660 0 0) 0x7f31800033f0 con 0x7f3198100780 2024-02-19T21:14:46.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31981184c0 con 0x7f3198100780 2024-02-19T21:14:46.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f31880019d0 con 0x7f3198100780 2024-02-19T21:14:46.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 779129053 0 0) 0x7f31981172e0 con 0x7f31981042c0 2024-02-19T21:14:46.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31800033f0 con 0x7f31981042c0 2024-02-19T21:14:46.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3190001b80 con 0x7f31981042c0 2024-02-19T21:14:46.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 189246786 0 0) 0x7f31981196c0 con 0x7f31980fd980 2024-02-19T21:14:46.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31981172e0 con 0x7f31980fd980 2024-02-19T21:14:46.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f318c002c60 con 0x7f31980fd980 2024-02-19T21:14:46.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1628327326 0 0) 0x7f31981184c0 con 0x7f3198100780 2024-02-19T21:14:46.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 >> v1:172.21.15.163:6790/0 conn(0x7f31981042c0 legacy=0x7f31981159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:46.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 >> v1:172.21.15.193:6789/0 conn(0x7f31980fd980 legacy=0x7f31980f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:46.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319811a8c0 con 0x7f3198100780 2024-02-19T21:14:46.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31981198b0 con 0x7f3198100780 2024-02-19T21:14:46.291 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:14:46.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f319e135700 10 client.23613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:46.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3198119d20 con 0x7f3198100780 2024-02-19T21:14:46.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f319e135700 1 librados: init done 2024-02-19T21:14:46.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3188002460 con 0x7f3198100780 2024-02-19T21:14:46.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.651+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3188005ac0 con 0x7f3198100780 2024-02-19T21:14:46.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.653+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f3184004f00 con 0x7f3198100780 2024-02-19T21:14:46.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.653+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f3188001fc0 con 0x7f3198100780 2024-02-19T21:14:46.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.653+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(640..640 src has 434..640) v4 ==== 8182+0+0 (unknown 2125993142 0 0) 0x7f3188078c70 con 0x7f3198100780 2024-02-19T21:14:46.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.653+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 osd_map(640..640 src has 434..640) v4 2024-02-19T21:14:46.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.653+0000 7f3194ff9700 3 client.23613.objecter handle_osd_map got epochs [640,640] > 0 2024-02-19T21:14:46.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.653+0000 7f3194ff9700 3 client.23613.objecter handle_osd_map decoding full epoch 640 2024-02-19T21:14:46.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.654+0000 7f3194ff9700 20 client.23613.objecter dump_active .. 0 homeless 2024-02-19T21:14:46.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.654+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v640) v1 ==== 54+0+16 (unknown 2324443470 0 2420000192) 0x7f318807b060 con 0x7f3198100780 2024-02-19T21:14:46.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.654+0000 7f3194ff9700 10 client.23613.objecter ms_handle_connect 0x7f3180063cb0 2024-02-19T21:14:46.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.654+0000 7f319e135700 10 client.23613.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:46.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.654+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 1 2024-02-19T21:14:46.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.654+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f31840051a0 osd=0 3 2024-02-19T21:14:46.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.654+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f31840051a0 osd=0 3 2024-02-19T21:14:46.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.654+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f31840051a0 osd=0 3 2024-02-19T21:14:46.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.654+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 1 2024-02-19T21:14:46.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.654+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f31840051a0 osd=0 3 2024-02-19T21:14:46.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.654+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 0 1 2024-02-19T21:14:46.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.654+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f31840051a0 osd=0 4 2024-02-19T21:14:46.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.654+0000 7f319e135700 10 client.23613.objecter _send_command 1 2024-02-19T21:14:46.294 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.654+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3184004950 con 0x7f3184004530 2024-02-19T21:14:46.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.654+0000 7f3194ff9700 10 client.23613.objecter ms_handle_connect 0x7f3184004530 2024-02-19T21:14:46.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.655+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3184004950 con 0x7f3184004530 2024-02-19T21:14:46.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.655+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:14:46.295 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.655+0000 7f3194ff9700 10 client.23613.objecter _finish_command 1 = system:0 2024-02-19T21:14:46.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.655+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f31840051a0 osd=0 4 2024-02-19T21:14:46.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.655+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 0 1 2024-02-19T21:14:46.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.670+0000 7f319e135700 10 client.23613.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:46.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.670+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 2 2024-02-19T21:14:46.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.670+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f31840051a0 osd=0 3 2024-02-19T21:14:46.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.670+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f31840051a0 osd=0 3 2024-02-19T21:14:46.296 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.670+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f31840051a0 osd=0 3 2024-02-19T21:14:46.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.670+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 2 2024-02-19T21:14:46.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.670+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f31840051a0 osd=0 3 2024-02-19T21:14:46.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.670+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 0 2 2024-02-19T21:14:46.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.670+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f31840051a0 osd=0 4 2024-02-19T21:14:46.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.670+0000 7f319e135700 10 client.23613.objecter _send_command 2 2024-02-19T21:14:46.297 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.670+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3184004d30 con 0x7f3184004530 2024-02-19T21:14:46.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.671+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3184004d30 con 0x7f3184004530 2024-02-19T21:14:46.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.671+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:46.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.671+0000 7f3194ff9700 10 client.23613.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.671+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f31840051a0 osd=0 4 2024-02-19T21:14:46.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.671+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 0 2 2024-02-19T21:14:46.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f319e135700 10 client.23613.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:46.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 3 2024-02-19T21:14:46.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184003fe0 osd=1 3 2024-02-19T21:14:46.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184003fe0 osd=1 3 2024-02-19T21:14:46.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184003fe0 osd=1 3 2024-02-19T21:14:46.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 3 2024-02-19T21:14:46.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f3184003fe0 osd=1 3 2024-02-19T21:14:46.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 1 3 2024-02-19T21:14:46.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184003fe0 osd=1 4 2024-02-19T21:14:46.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f319e135700 10 client.23613.objecter _send_command 3 2024-02-19T21:14:46.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f318400c000 con 0x7f3184004150 2024-02-19T21:14:46.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f3194ff9700 10 client.23613.objecter ms_handle_connect 0x7f3184004150 2024-02-19T21:14:46.299 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:14:46.299 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:14:46.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f318400c000 con 0x7f3184004150 2024-02-19T21:14:46.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:14:46.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f3194ff9700 10 client.23613.objecter _finish_command 3 = system:0 2024-02-19T21:14:46.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f3184003fe0 osd=1 4 2024-02-19T21:14:46.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.672+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 1 3 2024-02-19T21:14:46.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.687+0000 7f319e135700 10 client.23613.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:46.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.687+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 4 2024-02-19T21:14:46.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.687+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184003fe0 osd=1 3 2024-02-19T21:14:46.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.687+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184003fe0 osd=1 3 2024-02-19T21:14:46.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.687+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184003fe0 osd=1 3 2024-02-19T21:14:46.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.687+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 4 2024-02-19T21:14:46.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.687+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f3184003fe0 osd=1 3 2024-02-19T21:14:46.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.687+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 1 4 2024-02-19T21:14:46.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.687+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184003fe0 osd=1 4 2024-02-19T21:14:46.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.687+0000 7f319e135700 10 client.23613.objecter _send_command 4 2024-02-19T21:14:46.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.687+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f31840018b0 con 0x7f3184004150 2024-02-19T21:14:46.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.688+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f31840018b0 con 0x7f3184004150 2024-02-19T21:14:46.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.688+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:46.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.688+0000 7f3194ff9700 10 client.23613.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.688+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f3184003fe0 osd=1 4 2024-02-19T21:14:46.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.688+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 1 4 2024-02-19T21:14:46.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f319e135700 10 client.23613.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:46.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 5 2024-02-19T21:14:46.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184003630 osd=2 3 2024-02-19T21:14:46.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184003630 osd=2 3 2024-02-19T21:14:46.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184003630 osd=2 3 2024-02-19T21:14:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 5 2024-02-19T21:14:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f3184003630 osd=2 3 2024-02-19T21:14:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 2 5 2024-02-19T21:14:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184003630 osd=2 4 2024-02-19T21:14:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f319e135700 10 client.23613.objecter _send_command 5 2024-02-19T21:14:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f318400de90 con 0x7f318400d2f0 2024-02-19T21:14:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f3194ff9700 10 client.23613.objecter ms_handle_connect 0x7f318400d2f0 2024-02-19T21:14:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f318400de90 con 0x7f318400d2f0 2024-02-19T21:14:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:14:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f3194ff9700 10 client.23613.objecter _finish_command 5 = system:0 2024-02-19T21:14:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f3184003630 osd=2 4 2024-02-19T21:14:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.689+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 2 5 2024-02-19T21:14:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.704+0000 7f319e135700 10 client.23613.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:46.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.704+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 6 2024-02-19T21:14:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.704+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184003630 osd=2 3 2024-02-19T21:14:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.704+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184003630 osd=2 3 2024-02-19T21:14:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.704+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184003630 osd=2 3 2024-02-19T21:14:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.704+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 6 2024-02-19T21:14:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.704+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f3184003630 osd=2 3 2024-02-19T21:14:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.704+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 2 6 2024-02-19T21:14:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.704+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184003630 osd=2 4 2024-02-19T21:14:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.704+0000 7f319e135700 10 client.23613.objecter _send_command 6 2024-02-19T21:14:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.704+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3184003160 con 0x7f318400d2f0 2024-02-19T21:14:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3184003160 con 0x7f318400d2f0 2024-02-19T21:14:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f3194ff9700 10 client.23613.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f3184003630 osd=2 4 2024-02-19T21:14:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 2 6 2024-02-19T21:14:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f319e135700 10 client.23613.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 7 2024-02-19T21:14:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184011970 osd=3 3 2024-02-19T21:14:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184011970 osd=3 3 2024-02-19T21:14:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184011970 osd=3 3 2024-02-19T21:14:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 7 2024-02-19T21:14:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f3184011970 osd=3 3 2024-02-19T21:14:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 3 7 2024-02-19T21:14:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184011970 osd=3 4 2024-02-19T21:14:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f319e135700 10 client.23613.objecter _send_command 7 2024-02-19T21:14:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.705+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3184015bb0 con 0x7f3184011ff0 2024-02-19T21:14:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.706+0000 7f3194ff9700 10 client.23613.objecter ms_handle_connect 0x7f3184011ff0 2024-02-19T21:14:46.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.706+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3184015bb0 con 0x7f3184011ff0 2024-02-19T21:14:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.706+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:14:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.706+0000 7f3194ff9700 10 client.23613.objecter _finish_command 7 = system:0 2024-02-19T21:14:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.706+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f3184011970 osd=3 4 2024-02-19T21:14:46.306 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:14:46.306 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:14:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.706+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 3 7 2024-02-19T21:14:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.720+0000 7f319e135700 10 client.23613.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:46.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.720+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 8 2024-02-19T21:14:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.720+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184011970 osd=3 3 2024-02-19T21:14:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.720+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184011970 osd=3 3 2024-02-19T21:14:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.720+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184011970 osd=3 3 2024-02-19T21:14:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.720+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 8 2024-02-19T21:14:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.720+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f3184011970 osd=3 3 2024-02-19T21:14:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.720+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 3 8 2024-02-19T21:14:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.720+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184011970 osd=3 4 2024-02-19T21:14:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.720+0000 7f319e135700 10 client.23613.objecter _send_command 8 2024-02-19T21:14:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.720+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f318400e240 con 0x7f3184011ff0 2024-02-19T21:14:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f318400e240 con 0x7f3184011ff0 2024-02-19T21:14:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f3194ff9700 10 client.23613.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f3184011970 osd=3 4 2024-02-19T21:14:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 3 8 2024-02-19T21:14:46.307 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f319e135700 10 client.23613.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 9 2024-02-19T21:14:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f318400e240 osd=4 3 2024-02-19T21:14:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f318400e240 osd=4 3 2024-02-19T21:14:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f318400e240 osd=4 3 2024-02-19T21:14:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 9 2024-02-19T21:14:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f318400e240 osd=4 3 2024-02-19T21:14:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 4 9 2024-02-19T21:14:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f318400e240 osd=4 4 2024-02-19T21:14:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f319e135700 10 client.23613.objecter _send_command 9 2024-02-19T21:14:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f318400e560 con 0x7f318400ce50 2024-02-19T21:14:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.722+0000 7f3194ff9700 10 client.23613.objecter ms_handle_connect 0x7f318400ce50 2024-02-19T21:14:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.723+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f318400e560 con 0x7f318400ce50 2024-02-19T21:14:46.308 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.723+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:14:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.723+0000 7f3194ff9700 10 client.23613.objecter _finish_command 9 = system:0 2024-02-19T21:14:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.723+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f318400e240 osd=4 4 2024-02-19T21:14:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.723+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 4 9 2024-02-19T21:14:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.738+0000 7f319e135700 10 client.23613.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.738+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 10 2024-02-19T21:14:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.738+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f318400e240 osd=4 3 2024-02-19T21:14:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.738+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f318400e240 osd=4 3 2024-02-19T21:14:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.738+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f318400e240 osd=4 3 2024-02-19T21:14:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.738+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 10 2024-02-19T21:14:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.738+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f318400e240 osd=4 3 2024-02-19T21:14:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.738+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 4 10 2024-02-19T21:14:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.738+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f318400e240 osd=4 4 2024-02-19T21:14:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.738+0000 7f319e135700 10 client.23613.objecter _send_command 10 2024-02-19T21:14:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.738+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f318400e560 con 0x7f318400ce50 2024-02-19T21:14:46.309 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f318400e560 con 0x7f318400ce50 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f3194ff9700 10 client.23613.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f318400e240 osd=4 4 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 4 10 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f319e135700 10 client.23613.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 11 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f318400e560 osd=5 3 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f318400e560 osd=5 3 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f318400e560 osd=5 3 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 11 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f318400e560 osd=5 3 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 5 11 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f318400e560 osd=5 4 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f319e135700 10 client.23613.objecter _send_command 11 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3184016f50 con 0x7f31840169b0 2024-02-19T21:14:46.310 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.740+0000 7f3194ff9700 10 client.23613.objecter ms_handle_connect 0x7f31840169b0 2024-02-19T21:14:46.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.741+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3184016f50 con 0x7f31840169b0 2024-02-19T21:14:46.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.741+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:14:46.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.741+0000 7f3194ff9700 10 client.23613.objecter _finish_command 11 = system:0 2024-02-19T21:14:46.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.741+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f318400e560 osd=5 4 2024-02-19T21:14:46.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.741+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 5 11 2024-02-19T21:14:46.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.756+0000 7f319e135700 10 client.23613.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:46.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.756+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 12 2024-02-19T21:14:46.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.756+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f318400e560 osd=5 3 2024-02-19T21:14:46.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.756+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f318400e560 osd=5 3 2024-02-19T21:14:46.311 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.756+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f318400e560 osd=5 3 2024-02-19T21:14:46.311 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:14:46.312 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:14:46.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.756+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 12 2024-02-19T21:14:46.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.756+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f318400e560 osd=5 3 2024-02-19T21:14:46.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.756+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 5 12 2024-02-19T21:14:46.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.756+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f318400e560 osd=5 4 2024-02-19T21:14:46.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.756+0000 7f319e135700 10 client.23613.objecter _send_command 12 2024-02-19T21:14:46.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.756+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3184016f50 con 0x7f31840169b0 2024-02-19T21:14:46.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3184016f50 con 0x7f31840169b0 2024-02-19T21:14:46.312 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:46.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f3194ff9700 10 client.23613.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f318400e560 osd=5 4 2024-02-19T21:14:46.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 5 12 2024-02-19T21:14:46.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f319e135700 10 client.23613.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:46.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 13 2024-02-19T21:14:46.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184016f50 osd=6 3 2024-02-19T21:14:46.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184016f50 osd=6 3 2024-02-19T21:14:46.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184016f50 osd=6 3 2024-02-19T21:14:46.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 13 2024-02-19T21:14:46.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f3184016f50 osd=6 3 2024-02-19T21:14:46.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 6 13 2024-02-19T21:14:46.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184016f50 osd=6 4 2024-02-19T21:14:46.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f319e135700 10 client.23613.objecter _send_command 13 2024-02-19T21:14:46.313 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.758+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f31840270d0 con 0x7f31840235c0 2024-02-19T21:14:46.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.759+0000 7f3194ff9700 10 client.23613.objecter ms_handle_connect 0x7f31840235c0 2024-02-19T21:14:46.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.759+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f31840270d0 con 0x7f31840235c0 2024-02-19T21:14:46.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.759+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:14:46.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.759+0000 7f3194ff9700 10 client.23613.objecter _finish_command 13 = system:0 2024-02-19T21:14:46.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.759+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f3184016f50 osd=6 4 2024-02-19T21:14:46.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.759+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 6 13 2024-02-19T21:14:46.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.773+0000 7f319e135700 10 client.23613.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:46.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.773+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 14 2024-02-19T21:14:46.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.773+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184016f50 osd=6 3 2024-02-19T21:14:46.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.773+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184016f50 osd=6 3 2024-02-19T21:14:46.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.773+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f3184016f50 osd=6 3 2024-02-19T21:14:46.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.773+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 14 2024-02-19T21:14:46.314 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.773+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f3184016f50 osd=6 3 2024-02-19T21:14:46.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.773+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 6 14 2024-02-19T21:14:46.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.773+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184016f50 osd=6 4 2024-02-19T21:14:46.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.773+0000 7f319e135700 10 client.23613.objecter _send_command 14 2024-02-19T21:14:46.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.773+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f31840270d0 con 0x7f31840235c0 2024-02-19T21:14:46.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f31840270d0 con 0x7f31840235c0 2024-02-19T21:14:46.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:46.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f3194ff9700 10 client.23613.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f3184016f50 osd=6 4 2024-02-19T21:14:46.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 6 14 2024-02-19T21:14:46.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f319e135700 10 client.23613.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:46.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 15 2024-02-19T21:14:46.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f31840270d0 osd=7 3 2024-02-19T21:14:46.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f31840270d0 osd=7 3 2024-02-19T21:14:46.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f31840270d0 osd=7 3 2024-02-19T21:14:46.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 15 2024-02-19T21:14:46.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f31840270d0 osd=7 3 2024-02-19T21:14:46.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 7 15 2024-02-19T21:14:46.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f31840270d0 osd=7 4 2024-02-19T21:14:46.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f319e135700 10 client.23613.objecter _send_command 15 2024-02-19T21:14:46.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.775+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3184028320 con 0x7f3184028dd0 2024-02-19T21:14:46.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.776+0000 7f3194ff9700 10 client.23613.objecter ms_handle_connect 0x7f3184028dd0 2024-02-19T21:14:46.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.777+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f3184028320 con 0x7f3184028dd0 2024-02-19T21:14:46.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.777+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:14:46.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.777+0000 7f3194ff9700 10 client.23613.objecter _finish_command 15 = system:0 2024-02-19T21:14:46.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.777+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f31840270d0 osd=7 4 2024-02-19T21:14:46.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.777+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 7 15 2024-02-19T21:14:46.317 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:14:46.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.792+0000 7f319e135700 10 client.23613.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:14:46.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.792+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign -1 16 2024-02-19T21:14:46.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.792+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f31840270d0 osd=7 3 2024-02-19T21:14:46.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.792+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f31840270d0 osd=7 3 2024-02-19T21:14:46.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.792+0000 7f319e135700 20 client.23613.objecter _get_session s=0x7f31840270d0 osd=7 3 2024-02-19T21:14:46.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.792+0000 7f319e135700 15 client.23613.objecter _session_command_op_remove -1 16 2024-02-19T21:14:46.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.792+0000 7f319e135700 20 client.23613.objecter get_session s=0x7f31840270d0 osd=7 3 2024-02-19T21:14:46.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.792+0000 7f319e135700 15 client.23613.objecter _session_command_op_assign 7 16 2024-02-19T21:14:46.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.792+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f31840270d0 osd=7 4 2024-02-19T21:14:46.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.792+0000 7f319e135700 10 client.23613.objecter _send_command 16 2024-02-19T21:14:46.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.792+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f3184028320 con 0x7f3184028dd0 2024-02-19T21:14:46.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f3194ff9700 1 -- 172.21.15.163:0/2777576014 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f3184028320 con 0x7f3184028dd0 2024-02-19T21:14:46.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f3194ff9700 10 client.23613.objecter ms_dispatch 0x7f3198001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:14:46.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f3194ff9700 10 client.23613.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f3194ff9700 20 client.23613.objecter put_session s=0x7f31840270d0 osd=7 4 2024-02-19T21:14:46.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f3194ff9700 15 client.23613.objecter _session_command_op_remove 7 16 2024-02-19T21:14:46.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 10 librados: watch_flush enter 2024-02-19T21:14:46.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 10 librados: watch_flush exit 2024-02-19T21:14:46.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 10 client.23613.objecter close_session for osd.0 2024-02-19T21:14:46.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 >> v1:172.21.15.163:6801/1559418115 conn(0x7f3184004530 legacy=0x7f3184002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:46.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f31840051a0 osd=0 1 2024-02-19T21:14:46.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 10 client.23613.objecter close_session for osd.1 2024-02-19T21:14:46.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 >> v1:172.21.15.163:6805/3125047432 conn(0x7f3184004150 legacy=0x7f318400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:46.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184003fe0 osd=1 1 2024-02-19T21:14:46.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 10 client.23613.objecter close_session for osd.2 2024-02-19T21:14:46.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 >> v1:172.21.15.163:6809/1563158487 conn(0x7f318400d2f0 legacy=0x7f318400d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:46.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184003630 osd=2 1 2024-02-19T21:14:46.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 10 client.23613.objecter close_session for osd.3 2024-02-19T21:14:46.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 >> v1:172.21.15.163:6813/2112279769 conn(0x7f3184011ff0 legacy=0x7f31840143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:46.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184011970 osd=3 1 2024-02-19T21:14:46.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 10 client.23613.objecter close_session for osd.4 2024-02-19T21:14:46.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 >> v1:172.21.15.193:6800/3129716965 conn(0x7f318400ce50 legacy=0x7f3184015d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:46.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f318400e240 osd=4 1 2024-02-19T21:14:46.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 10 client.23613.objecter close_session for osd.5 2024-02-19T21:14:46.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 >> v1:172.21.15.193:6804/805161991 conn(0x7f31840169b0 legacy=0x7f3184021990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:46.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f3197fff700 1 -- 172.21.15.163:0/2777576014 reap_dead start 2024-02-19T21:14:46.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f318400e560 osd=5 1 2024-02-19T21:14:46.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 10 client.23613.objecter close_session for osd.6 2024-02-19T21:14:46.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 >> v1:172.21.15.193:6808/787604615 conn(0x7f31840235c0 legacy=0x7f31840259b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:46.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f3184016f50 osd=6 1 2024-02-19T21:14:46.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 10 client.23613.objecter close_session for osd.7 2024-02-19T21:14:46.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 >> v1:172.21.15.193:6812/157927432 conn(0x7f3184028dd0 legacy=0x7f3184029230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:46.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f3197fff700 1 -- 172.21.15.163:0/2777576014 reap_dead start 2024-02-19T21:14:46.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 20 client.23613.objecter put_session s=0x7f31840270d0 osd=7 1 2024-02-19T21:14:46.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 20 client.23613.objecter shutdown clearing up homeless session... 2024-02-19T21:14:46.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 10 client.23613.objecter successfully canceled tick 2024-02-19T21:14:46.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f3197fff700 1 -- 172.21.15.163:0/2777576014 reap_dead start 2024-02-19T21:14:46.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f3197fff700 1 -- 172.21.15.163:0/2777576014 reap_dead start 2024-02-19T21:14:46.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f3197fff700 1 -- 172.21.15.163:0/2777576014 reap_dead start 2024-02-19T21:14:46.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 >> v1:172.21.15.163:6800/1552514681 conn(0x7f3180063cb0 legacy=0x7f3180066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:46.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 >> v1:172.21.15.163:6789/0 conn(0x7f3198100780 legacy=0x7f31980fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:46.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 shutdown_connections 2024-02-19T21:14:46.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 >> 172.21.15.163:0/2777576014 conn(0x7f31980f35a0 msgr2=0x7f3198107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:46.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 shutdown_connections 2024-02-19T21:14:46.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.794+0000 7f319e135700 1 -- 172.21.15.163:0/2777576014 wait complete. 2024-02-19T21:14:46.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:44.795+0000 7f319e135700 1 librados: shutdown 2024-02-19T21:14:46.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.321 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:14:46.322 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:46.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:46.368+0000 7fa3adffb700 1 -- 172.21.15.163:0/3472021223 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v642) v1 ==== 91+0+0 (unknown 1965114552 0 0) 0x7fa3a804ae50 con 0x7fa3b01042c0 2024-02-19T21:14:46.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:46.371+0000 7fa3b721a700 10 librados: watch_flush enter 2024-02-19T21:14:46.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:46.371+0000 7fa3b721a700 10 librados: watch_flush exit 2024-02-19T21:14:46.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:46.371+0000 7fa3b721a700 20 client.23619.objecter shutdown clearing up homeless session... 2024-02-19T21:14:46.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:46.371+0000 7fa3b721a700 10 client.23619.objecter successfully canceled tick 2024-02-19T21:14:46.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:46.371+0000 7fa3b721a700 1 -- 172.21.15.163:0/3472021223 >> v1:172.21.15.163:6800/1552514681 conn(0x7fa3a0063d80 legacy=0x7fa3a0066240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:46.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:46.371+0000 7fa3b721a700 1 -- 172.21.15.163:0/3472021223 >> v1:172.21.15.163:6789/0 conn(0x7fa3b01042c0 legacy=0x7fa3b0118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:46.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:46.371+0000 7fa3b721a700 1 -- 172.21.15.163:0/3472021223 shutdown_connections 2024-02-19T21:14:46.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:46.371+0000 7fa3b721a700 1 -- 172.21.15.163:0/3472021223 >> 172.21.15.163:0/3472021223 conn(0x7fa3b00f35a0 msgr2=0x7fa3b0107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:46.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:46.371+0000 7fa3b721a700 1 -- 172.21.15.163:0/3472021223 shutdown_connections 2024-02-19T21:14:46.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:46.371+0000 7fa3b721a700 1 -- 172.21.15.163:0/3472021223 wait complete. 2024-02-19T21:14:46.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:46.372+0000 7fa3b721a700 1 librados: shutdown 2024-02-19T21:14:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:14:46.491 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:46.494 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:46.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:46.595 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:46.622 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:46.676 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:46.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:46.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:46.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:45.182232+0000 osd.2 (osd.2) 256 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:46.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:45.182476+0000 osd.2 (osd.2) 257 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:46.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:45.292943+0000 mgr.y (mgr.14550) 553 : cluster [DBG] pgmap v964: 182 pgs: 182 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 363 B/s, 15 objects/s recovering 2024-02-19T21:14:46.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:46.359023+0000 mon.a (mon.0) 3212 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:14:46.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:46.363715+0000 mon.a (mon.0) 3213 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-110","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:14:46.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:46.363805+0000 mon.a (mon.0) 3214 : audit [INF] from='client.? v1:172.21.15.163:0/3472021223' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:14:46.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:46.363837+0000 mon.a (mon.0) 3215 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2024-02-19T21:14:46.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:46.382913+0000 mon.a (mon.0) 3216 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-110", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:14:46.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:46.471483+0000 mon.b (mon.1) 737 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:14:46.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:45.182232+0000 osd.2 (osd.2) 256 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:46.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:45.182476+0000 osd.2 (osd.2) 257 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:46.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:45.292943+0000 mgr.y (mgr.14550) 553 : cluster [DBG] pgmap v964: 182 pgs: 182 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 363 B/s, 15 objects/s recovering 2024-02-19T21:14:46.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:46.359023+0000 mon.a (mon.0) 3212 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:14:46.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:46.363715+0000 mon.a (mon.0) 3213 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-110","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:14:46.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:46.363805+0000 mon.a (mon.0) 3214 : audit [INF] from='client.? v1:172.21.15.163:0/3472021223' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:14:46.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:46.363837+0000 mon.a (mon.0) 3215 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2024-02-19T21:14:46.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:46.382913+0000 mon.a (mon.0) 3216 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-110", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:14:46.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:46.471483+0000 mon.b (mon.1) 737 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:14:46.781 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:14:46.808 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:46.827 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:14:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:46.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:14:46.930 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:14:46.955 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:45.182232+0000 osd.2 (osd.2) 256 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:46.955 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:45.182476+0000 osd.2 (osd.2) 257 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:46.955 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:45.292943+0000 mgr.y (mgr.14550) 553 : cluster [DBG] pgmap v964: 182 pgs: 182 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 363 B/s, 15 objects/s recovering 2024-02-19T21:14:46.955 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:46.359023+0000 mon.a (mon.0) 3212 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:14:46.955 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:46.363715+0000 mon.a (mon.0) 3213 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-110","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:14:46.955 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:46.363805+0000 mon.a (mon.0) 3214 : audit [INF] from='client.? v1:172.21.15.163:0/3472021223' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:14:46.955 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:46.363837+0000 mon.a (mon.0) 3215 : cluster [DBG] osdmap e642: 8 total, 8 up, 8 in 2024-02-19T21:14:46.956 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:46.382913+0000 mon.a (mon.0) 3216 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-110", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:14:46.956 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:46.471483+0000 mon.b (mon.1) 737 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:14:46.957 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:46.975 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:14:46.978 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:46.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:14:47.079 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:47.107 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:47.334 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:47.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:47.439 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:14:47.466 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:47.485 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:14:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:47.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:14:47.589 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:47.618 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:47.772 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:46.770402+0000 mon.b (mon.1) 738 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:14:47.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:46.771160+0000 mon.b (mon.1) 739 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:14:47.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:46.775676+0000 mon.a (mon.0) 3217 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:14:47.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:46.795506+0000 osd.7 (osd.7) 289 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:14:47.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:46.798417+0000 osd.7 (osd.7) 290 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:14:47.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:47.086135+0000 osd.3 (osd.3) 261 : cluster [DBG] 4.14 scrub starts 2024-02-19T21:14:47.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:47.087901+0000 osd.3 (osd.3) 262 : cluster [DBG] 4.14 scrub ok 2024-02-19T21:14:47.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:47.366566+0000 mon.a (mon.0) 3218 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-110", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:14:47.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:47.366632+0000 mon.a (mon.0) 3219 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2024-02-19T21:14:47.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:47.368052+0000 mon.a (mon.0) 3220 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-110"}]: dispatch 2024-02-19T21:14:47.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:46.770402+0000 mon.b (mon.1) 738 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:14:47.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:46.771160+0000 mon.b (mon.1) 739 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:14:47.922 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:46.775676+0000 mon.a (mon.0) 3217 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:14:47.922 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:46.795506+0000 osd.7 (osd.7) 289 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:14:47.922 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:46.798417+0000 osd.7 (osd.7) 290 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:14:47.922 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:47.086135+0000 osd.3 (osd.3) 261 : cluster [DBG] 4.14 scrub starts 2024-02-19T21:14:47.922 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:47.087901+0000 osd.3 (osd.3) 262 : cluster [DBG] 4.14 scrub ok 2024-02-19T21:14:47.922 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:47.366566+0000 mon.a (mon.0) 3218 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-110", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:14:47.922 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:47.366632+0000 mon.a (mon.0) 3219 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2024-02-19T21:14:47.922 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:47.368052+0000 mon.a (mon.0) 3220 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-110"}]: dispatch 2024-02-19T21:14:47.922 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:46.770402+0000 mon.b (mon.1) 738 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:14:47.922 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:46.771160+0000 mon.b (mon.1) 739 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:14:47.922 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:46.775676+0000 mon.a (mon.0) 3217 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:14:47.922 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:46.795506+0000 osd.7 (osd.7) 289 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:14:47.923 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:46.798417+0000 osd.7 (osd.7) 290 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:14:47.923 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:47.086135+0000 osd.3 (osd.3) 261 : cluster [DBG] 4.14 scrub starts 2024-02-19T21:14:47.923 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:47.087901+0000 osd.3 (osd.3) 262 : cluster [DBG] 4.14 scrub ok 2024-02-19T21:14:47.923 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:47.366566+0000 mon.a (mon.0) 3218 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-110", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:14:47.923 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:47.366632+0000 mon.a (mon.0) 3219 : cluster [DBG] osdmap e643: 8 total, 8 up, 8 in 2024-02-19T21:14:47.923 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:47.368052+0000 mon.a (mon.0) 3220 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-110"}]: dispatch 2024-02-19T21:14:47.968 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:47.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:48.066 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:14:48.067 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:14:48.071 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:14:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:14:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:14:48.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:48.101 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:48.298 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:48.301 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:48.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:48.403 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:14:48.431 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:48.837 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-19T21:14:48.855 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:48.880 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:14:48.880 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:47.293495+0000 mgr.y (mgr.14550) 554 : cluster [DBG] pgmap v967: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 363 B/s, 15 objects/s recovering 2024-02-19T21:14:48.880 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:47.309408+0000 osd.6 (osd.6) 217 : cluster [DBG] 4.a scrub starts 2024-02-19T21:14:48.880 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:47.311258+0000 osd.6 (osd.6) 218 : cluster [DBG] 4.a scrub ok 2024-02-19T21:14:48.880 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:48.366910+0000 mon.a (mon.0) 3221 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-110"}]': finished 2024-02-19T21:14:48.881 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:48.366980+0000 mon.a (mon.0) 3222 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2024-02-19T21:14:48.881 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:48.367469+0000 mon.a (mon.0) 3223 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-110", "mode": "writeback"}]: dispatch 2024-02-19T21:14:48.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:47.293495+0000 mgr.y (mgr.14550) 554 : cluster [DBG] pgmap v967: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 363 B/s, 15 objects/s recovering 2024-02-19T21:14:48.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:47.309408+0000 osd.6 (osd.6) 217 : cluster [DBG] 4.a scrub starts 2024-02-19T21:14:48.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:47.311258+0000 osd.6 (osd.6) 218 : cluster [DBG] 4.a scrub ok 2024-02-19T21:14:48.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:48.366910+0000 mon.a (mon.0) 3221 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-110"}]': finished 2024-02-19T21:14:48.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:48.366980+0000 mon.a (mon.0) 3222 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2024-02-19T21:14:48.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:48.367469+0000 mon.a (mon.0) 3223 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-110", "mode": "writeback"}]: dispatch 2024-02-19T21:14:48.883 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:48.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:14:48.984 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:14:49.017 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:47.293495+0000 mgr.y (mgr.14550) 554 : cluster [DBG] pgmap v967: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 363 B/s, 15 objects/s recovering 2024-02-19T21:14:49.017 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:47.309408+0000 osd.6 (osd.6) 217 : cluster [DBG] 4.a scrub starts 2024-02-19T21:14:49.017 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:47.311258+0000 osd.6 (osd.6) 218 : cluster [DBG] 4.a scrub ok 2024-02-19T21:14:49.017 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:48.366910+0000 mon.a (mon.0) 3221 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-110"}]': finished 2024-02-19T21:14:49.017 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:48.366980+0000 mon.a (mon.0) 3222 : cluster [DBG] osdmap e644: 8 total, 8 up, 8 in 2024-02-19T21:14:49.017 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:48.367469+0000 mon.a (mon.0) 3223 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-110", "mode": "writeback"}]: dispatch 2024-02-19T21:14:49.019 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:49.040 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:14:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:49.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:14:49.084 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:14:49.144 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:14:49.174 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:49.195 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:14:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:49.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:14:49.299 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:14:49.329 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:49.349 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:14:49.351 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:49.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:14:49.453 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:49.480 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:49.553 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:49.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:49.658 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:49.687 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:49.940 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:14:49.966 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:48.322065+0000 osd.2 (osd.2) 258 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:14:49.966 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:48.323263+0000 osd.2 (osd.2) 259 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:14:49.966 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:48.836987+0000 osd.7 (osd.7) 291 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:14:49.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:48.841218+0000 osd.7 (osd.7) 292 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:14:49.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:49.357551+0000 mon.a (mon.0) 3224 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:14:49.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:49.362942+0000 mon.a (mon.0) 3225 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-110", "mode": "writeback"}]': finished 2024-02-19T21:14:49.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:49.363008+0000 mon.a (mon.0) 3226 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2024-02-19T21:14:49.967 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:49.473579+0000 mon.a (mon.0) 3227 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:14:49.967 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:48.322065+0000 osd.2 (osd.2) 258 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:14:49.967 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:48.323263+0000 osd.2 (osd.2) 259 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:14:49.968 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:48.836987+0000 osd.7 (osd.7) 291 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:14:49.968 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:48.841218+0000 osd.7 (osd.7) 292 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:14:49.968 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:49.357551+0000 mon.a (mon.0) 3224 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:14:49.968 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:49.362942+0000 mon.a (mon.0) 3225 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-110", "mode": "writeback"}]': finished 2024-02-19T21:14:49.968 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:49.363008+0000 mon.a (mon.0) 3226 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2024-02-19T21:14:49.968 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:49.473579+0000 mon.a (mon.0) 3227 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:14:50.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:48.322065+0000 osd.2 (osd.2) 258 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:14:50.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:48.323263+0000 osd.2 (osd.2) 259 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:14:50.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:48.836987+0000 osd.7 (osd.7) 291 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:14:50.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:48.841218+0000 osd.7 (osd.7) 292 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:14:50.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:49.357551+0000 mon.a (mon.0) 3224 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:14:50.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:49.362942+0000 mon.a (mon.0) 3225 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-110", "mode": "writeback"}]': finished 2024-02-19T21:14:50.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:49.363008+0000 mon.a (mon.0) 3226 : cluster [DBG] osdmap e645: 8 total, 8 up, 8 in 2024-02-19T21:14:50.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:49.473579+0000 mon.a (mon.0) 3227 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:14:50.336 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:50.340 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:50.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:50.441 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:50.472 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:50.560 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:50.573 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:50.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.577+0000 7fec33d3b700 1 -- 172.21.15.163:0/1624974044 >> v1:172.21.15.163:6790/0 conn(0x7fec2c0f6710 legacy=0x7fec2c0f6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:50.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.578+0000 7fec33d3b700 1 -- 172.21.15.163:0/1624974044 shutdown_connections 2024-02-19T21:14:50.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.578+0000 7fec33d3b700 1 -- 172.21.15.163:0/1624974044 >> 172.21.15.163:0/1624974044 conn(0x7fec2c0f2470 msgr2=0x7fec2c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:50.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.578+0000 7fec33d3b700 1 -- 172.21.15.163:0/1624974044 shutdown_connections 2024-02-19T21:14:50.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.578+0000 7fec33d3b700 1 -- 172.21.15.163:0/1624974044 wait complete. 2024-02-19T21:14:50.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.579+0000 7fec33d3b700 1 librados: starting msgr at 2024-02-19T21:14:50.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.579+0000 7fec33d3b700 1 librados: starting objecter 2024-02-19T21:14:50.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.579+0000 7fec33d3b700 1 Processor -- start 2024-02-19T21:14:50.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.579+0000 7fec33d3b700 1 -- start start 2024-02-19T21:14:50.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.579+0000 7fec33d3b700 1 librados: setting wanted keys 2024-02-19T21:14:50.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.579+0000 7fec33d3b700 1 librados: calling monclient init 2024-02-19T21:14:50.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.579+0000 7fec33d3b700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fec2c10aa00 con 0x7fec2c0f6710 2024-02-19T21:14:50.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.579+0000 7fec33d3b700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fec2c127660 con 0x7fec2c0f72c0 2024-02-19T21:14:50.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.579+0000 7fec33d3b700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fec2c128880 con 0x7fec2c0fc950 2024-02-19T21:14:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:50.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:50.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.580+0000 7fec312d6700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fec2c0f72c0 0x7fec2c103da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:35578/0 (socket says 172.21.15.163:35578) 2024-02-19T21:14:50.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.580+0000 7fec312d6700 1 -- 172.21.15.163:0/3358611725 learned_addr learned my addr 172.21.15.163:0/3358611725 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:50.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.580+0000 7fec22ffd700 10 client.?.objecter ms_handle_connect 0x7fec2c0f72c0 2024-02-19T21:14:50.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.580+0000 7fec22ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:50.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.580+0000 7fec22ffd700 1 -- 172.21.15.163:0/3358611725 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707734777 0 0) 0x7fec2c127660 con 0x7fec2c0f72c0 2024-02-19T21:14:50.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.580+0000 7fec22ffd700 1 -- 172.21.15.163:0/3358611725 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec180033f0 con 0x7fec2c0f72c0 2024-02-19T21:14:50.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.581+0000 7fec22ffd700 1 -- 172.21.15.163:0/3358611725 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2999471988 0 0) 0x7fec180033f0 con 0x7fec2c0f72c0 2024-02-19T21:14:50.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.581+0000 7fec22ffd700 1 -- 172.21.15.163:0/3358611725 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec2c127660 con 0x7fec2c0f72c0 2024-02-19T21:14:50.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.581+0000 7fec22ffd700 1 -- 172.21.15.163:0/3358611725 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fec28001940 con 0x7fec2c0f72c0 2024-02-19T21:14:50.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.581+0000 7fec22ffd700 10 client.?.objecter ms_handle_connect 0x7fec2c0f6710 2024-02-19T21:14:50.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.581+0000 7fec22ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:50.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.581+0000 7fec22ffd700 1 -- 172.21.15.163:0/3358611725 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2590453329 0 0) 0x7fec2c127660 con 0x7fec2c0f72c0 2024-02-19T21:14:50.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.581+0000 7fec22ffd700 1 -- 172.21.15.163:0/3358611725 >> v1:172.21.15.163:6790/0 conn(0x7fec2c0fc950 legacy=0x7fec2c124ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:50.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.581+0000 7fec22ffd700 1 -- 172.21.15.163:0/3358611725 >> v1:172.21.15.163:6789/0 conn(0x7fec2c0f6710 legacy=0x7fec2c109e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:50.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.581+0000 7fec22ffd700 1 -- 172.21.15.163:0/3358611725 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c129a80 con 0x7fec2c0f72c0 2024-02-19T21:14:50.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.581+0000 7fec33d3b700 1 -- 172.21.15.163:0/3358611725 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2c126670 con 0x7fec2c0f72c0 2024-02-19T21:14:50.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.581+0000 7fec33d3b700 10 client.49159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:50.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.581+0000 7fec33d3b700 1 -- 172.21.15.163:0/3358611725 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2c126ae0 con 0x7fec2c0f72c0 2024-02-19T21:14:50.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.581+0000 7fec33d3b700 1 librados: init done 2024-02-19T21:14:50.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.581+0000 7fec22ffd700 1 -- 172.21.15.163:0/3358611725 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fec28002da0 con 0x7fec2c0f72c0 2024-02-19T21:14:50.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.581+0000 7fec22ffd700 1 -- 172.21.15.163:0/3358611725 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fec28005cc0 con 0x7fec2c0f72c0 2024-02-19T21:14:50.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.582+0000 7fec33d3b700 1 -- 172.21.15.163:0/3358611725 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec10004f50 con 0x7fec2c0f72c0 2024-02-19T21:14:50.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.583+0000 7fec22ffd700 1 -- 172.21.15.163:0/3358611725 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fec28001fa0 con 0x7fec2c0f72c0 2024-02-19T21:14:50.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.584+0000 7fec22ffd700 1 -- 172.21.15.163:0/3358611725 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(646..646 src has 434..646) v4 ==== 8535+0+0 (unknown 2458384176 0 0) 0x7fec28079000 con 0x7fec2c0f72c0 2024-02-19T21:14:50.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.584+0000 7fec22ffd700 10 client.49159.objecter ms_dispatch 0x7fec2c001550 osd_map(646..646 src has 434..646) v4 2024-02-19T21:14:50.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.584+0000 7fec22ffd700 3 client.49159.objecter handle_osd_map got epochs [646,646] > 0 2024-02-19T21:14:50.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.584+0000 7fec22ffd700 3 client.49159.objecter handle_osd_map decoding full epoch 646 2024-02-19T21:14:50.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.584+0000 7fec22ffd700 20 client.49159.objecter dump_active .. 0 homeless 2024-02-19T21:14:50.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.584+0000 7fec22ffd700 10 client.49159.objecter ms_handle_connect 0x7fec18063a70 2024-02-19T21:14:50.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.588+0000 7fec22ffd700 1 -- 172.21.15.163:0/3358611725 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fec2804c220 con 0x7fec2c0f72c0 2024-02-19T21:14:50.614 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:50.616 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:50.685 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:50.715 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:50.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:50.782+0000 7fec33d3b700 1 -- 172.21.15.163:0/3358611725 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7fec10005200 con 0x7fec2c0f72c0 2024-02-19T21:14:50.813 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:50.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:50.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:49.293944+0000 mgr.y (mgr.14550) 555 : cluster [DBG] pgmap v970: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:14:50.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:49.320976+0000 osd.2 (osd.2) 260 : cluster [DBG] 5.b scrub starts 2024-02-19T21:14:50.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:49.321805+0000 osd.2 (osd.2) 261 : cluster [DBG] 5.b scrub ok 2024-02-19T21:14:50.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:49.573128+0000 osd.4 (osd.4) 223 : cluster [DBG] 2.f scrub starts 2024-02-19T21:14:50.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:49.574020+0000 osd.4 (osd.4) 224 : cluster [DBG] 2.f scrub ok 2024-02-19T21:14:50.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:49.847577+0000 osd.7 (osd.7) 293 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:14:50.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:49.850502+0000 osd.7 (osd.7) 294 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:14:50.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:49.965670+0000 osd.1 (osd.1) 288 : cluster [DBG] 16.13 scrub starts 2024-02-19T21:14:50.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:49.969303+0000 osd.1 (osd.1) 289 : cluster [DBG] 16.13 scrub ok 2024-02-19T21:14:50.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:50.369381+0000 mon.a (mon.0) 3228 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:50.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:50.369441+0000 mon.a (mon.0) 3229 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2024-02-19T21:14:50.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:50.370579+0000 mon.a (mon.0) 3230 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-110"}]: dispatch 2024-02-19T21:14:50.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:49.293944+0000 mgr.y (mgr.14550) 555 : cluster [DBG] pgmap v970: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:14:50.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:49.320976+0000 osd.2 (osd.2) 260 : cluster [DBG] 5.b scrub starts 2024-02-19T21:14:50.836 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:49.321805+0000 osd.2 (osd.2) 261 : cluster [DBG] 5.b scrub ok 2024-02-19T21:14:50.836 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:49.573128+0000 osd.4 (osd.4) 223 : cluster [DBG] 2.f scrub starts 2024-02-19T21:14:50.836 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:49.574020+0000 osd.4 (osd.4) 224 : cluster [DBG] 2.f scrub ok 2024-02-19T21:14:50.836 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:49.847577+0000 osd.7 (osd.7) 293 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:14:50.836 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:49.850502+0000 osd.7 (osd.7) 294 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:14:50.836 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:49.965670+0000 osd.1 (osd.1) 288 : cluster [DBG] 16.13 scrub starts 2024-02-19T21:14:50.836 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:49.969303+0000 osd.1 (osd.1) 289 : cluster [DBG] 16.13 scrub ok 2024-02-19T21:14:50.836 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:50.369381+0000 mon.a (mon.0) 3228 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:50.836 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:50.369441+0000 mon.a (mon.0) 3229 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2024-02-19T21:14:50.836 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:50.370579+0000 mon.a (mon.0) 3230 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-110"}]: dispatch 2024-02-19T21:14:50.917 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:14:50.945 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:51.044 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:14:51.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:51.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:14:51.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:14:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:14:51.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:49.293944+0000 mgr.y (mgr.14550) 555 : cluster [DBG] pgmap v970: 214 pgs: 32 unknown, 182 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:14:51.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:49.320976+0000 osd.2 (osd.2) 260 : cluster [DBG] 5.b scrub starts 2024-02-19T21:14:51.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:49.321805+0000 osd.2 (osd.2) 261 : cluster [DBG] 5.b scrub ok 2024-02-19T21:14:51.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:49.573128+0000 osd.4 (osd.4) 223 : cluster [DBG] 2.f scrub starts 2024-02-19T21:14:51.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:49.574020+0000 osd.4 (osd.4) 224 : cluster [DBG] 2.f scrub ok 2024-02-19T21:14:51.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:49.847577+0000 osd.7 (osd.7) 293 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:14:51.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:49.850502+0000 osd.7 (osd.7) 294 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:14:51.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:49.965670+0000 osd.1 (osd.1) 288 : cluster [DBG] 16.13 scrub starts 2024-02-19T21:14:51.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:49.969303+0000 osd.1 (osd.1) 289 : cluster [DBG] 16.13 scrub ok 2024-02-19T21:14:51.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:50.369381+0000 mon.a (mon.0) 3228 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:51.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:50.369441+0000 mon.a (mon.0) 3229 : cluster [DBG] osdmap e646: 8 total, 8 up, 8 in 2024-02-19T21:14:51.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:50.370579+0000 mon.a (mon.0) 3230 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-110"}]: dispatch 2024-02-19T21:14:51.148 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:14:51.180 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:51.201 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:14:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:51.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:14:51.305 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:14:51.335 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:51.355 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:14:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:51.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:14:51.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:51.371+0000 7fec22ffd700 1 -- 172.21.15.163:0/3358611725 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v647) v1 ==== 113+0+0 (unknown 557314151 0 0) 0x7fec2804a630 con 0x7fec2c0f72c0 2024-02-19T21:14:51.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:51.374+0000 7fec33d3b700 10 librados: watch_flush enter 2024-02-19T21:14:51.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:51.374+0000 7fec33d3b700 10 librados: watch_flush exit 2024-02-19T21:14:51.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:51.374+0000 7fec33d3b700 20 client.49159.objecter shutdown clearing up homeless session... 2024-02-19T21:14:51.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:51.374+0000 7fec33d3b700 10 client.49159.objecter successfully canceled tick 2024-02-19T21:14:51.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:51.374+0000 7fec33d3b700 1 -- 172.21.15.163:0/3358611725 >> v1:172.21.15.163:6800/1552514681 conn(0x7fec18063a70 legacy=0x7fec18065f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:51.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:51.374+0000 7fec33d3b700 1 -- 172.21.15.163:0/3358611725 >> v1:172.21.15.193:6789/0 conn(0x7fec2c0f72c0 legacy=0x7fec2c103da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:51.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:51.374+0000 7fec33d3b700 1 -- 172.21.15.163:0/3358611725 shutdown_connections 2024-02-19T21:14:51.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:51.374+0000 7fec33d3b700 1 -- 172.21.15.163:0/3358611725 >> 172.21.15.163:0/3358611725 conn(0x7fec2c0f2470 msgr2=0x7fec2c0f4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:51.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:51.374+0000 7fec33d3b700 1 -- 172.21.15.163:0/3358611725 shutdown_connections 2024-02-19T21:14:51.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:51.374+0000 7fec33d3b700 1 -- 172.21.15.163:0/3358611725 wait complete. 2024-02-19T21:14:51.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:51.374+0000 7fec33d3b700 1 librados: shutdown 2024-02-19T21:14:51.376 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.1 to 1 (10000) 2024-02-19T21:14:51.459 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:51.488 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:51.716 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:51.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:51.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:50.035402+0000 osd.5 (osd.5) 256 : cluster [DBG] 16.1b scrub starts 2024-02-19T21:14:51.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:50.038862+0000 osd.5 (osd.5) 257 : cluster [DBG] 16.1b scrub ok 2024-02-19T21:14:51.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:50.476223+0000 osd.4 (osd.4) 225 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:50.476521+0000 osd.4 (osd.4) 226 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:50.651399+0000 mgr.y (mgr.14550) 556 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:50.783973+0000 mon.b (mon.1) 740 : audit [INF] from='client.? v1:172.21.15.163:0/3358611725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:14:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:50.784479+0000 mon.a (mon.0) 3231 : audit [INF] from='client.49159 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:14:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:50.842955+0000 osd.7 (osd.7) 295 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:14:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:50.846591+0000 osd.7 (osd.7) 296 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:14:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:51.366135+0000 mon.a (mon.0) 3232 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:14:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:51.370708+0000 mon.a (mon.0) 3233 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-110"}]': finished 2024-02-19T21:14:51.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:51.370775+0000 mon.a (mon.0) 3234 : audit [INF] from='client.49159 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-19T21:14:51.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:51.370827+0000 mon.a (mon.0) 3235 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2024-02-19T21:14:51.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:50.035402+0000 osd.5 (osd.5) 256 : cluster [DBG] 16.1b scrub starts 2024-02-19T21:14:51.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:50.038862+0000 osd.5 (osd.5) 257 : cluster [DBG] 16.1b scrub ok 2024-02-19T21:14:51.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:50.476223+0000 osd.4 (osd.4) 225 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:51.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:50.476521+0000 osd.4 (osd.4) 226 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:51.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:50.651399+0000 mgr.y (mgr.14550) 556 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:51.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:50.783973+0000 mon.b (mon.1) 740 : audit [INF] from='client.? v1:172.21.15.163:0/3358611725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:14:51.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:50.784479+0000 mon.a (mon.0) 3231 : audit [INF] from='client.49159 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:14:51.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:50.842955+0000 osd.7 (osd.7) 295 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:14:51.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:50.846591+0000 osd.7 (osd.7) 296 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:14:51.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:51.366135+0000 mon.a (mon.0) 3232 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:14:51.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:51.370708+0000 mon.a (mon.0) 3233 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-110"}]': finished 2024-02-19T21:14:51.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:51.370775+0000 mon.a (mon.0) 3234 : audit [INF] from='client.49159 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-19T21:14:51.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:51.370827+0000 mon.a (mon.0) 3235 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2024-02-19T21:14:51.823 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:14:51.853 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:51.873 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:14:51.875 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:51.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:14:51.977 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:14:52.004 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:52.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:50.035402+0000 osd.5 (osd.5) 256 : cluster [DBG] 16.1b scrub starts 2024-02-19T21:14:52.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:50.038862+0000 osd.5 (osd.5) 257 : cluster [DBG] 16.1b scrub ok 2024-02-19T21:14:52.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:50.476223+0000 osd.4 (osd.4) 225 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:52.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:50.476521+0000 osd.4 (osd.4) 226 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:52.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:50.651399+0000 mgr.y (mgr.14550) 556 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:14:52.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:50.783973+0000 mon.b (mon.1) 740 : audit [INF] from='client.? v1:172.21.15.163:0/3358611725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:14:52.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:50.784479+0000 mon.a (mon.0) 3231 : audit [INF] from='client.49159 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:14:52.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:50.842955+0000 osd.7 (osd.7) 295 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:14:52.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:50.846591+0000 osd.7 (osd.7) 296 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:14:52.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:51.366135+0000 mon.a (mon.0) 3232 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:14:52.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:51.370708+0000 mon.a (mon.0) 3233 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-110"}]': finished 2024-02-19T21:14:52.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:51.370775+0000 mon.a (mon.0) 3234 : audit [INF] from='client.49159 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-19T21:14:52.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:51.370827+0000 mon.a (mon.0) 3235 : cluster [DBG] osdmap e647: 8 total, 8 up, 8 in 2024-02-19T21:14:52.101 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 2 1 2024-02-19T21:14:52.552 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:14:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:52.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:14:52.657 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:52.685 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:52.972 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:51.294680+0000 mgr.y (mgr.14550) 557 : cluster [DBG] pgmap v973: 214 pgs: 214 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-19T21:14:52.972 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:52.066876+0000 osd.3 (osd.3) 263 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:14:52.972 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:52.068772+0000 osd.3 (osd.3) 264 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:14:52.972 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:52.371617+0000 mon.a (mon.0) 3236 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2024-02-19T21:14:52.972 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:51.294680+0000 mgr.y (mgr.14550) 557 : cluster [DBG] pgmap v973: 214 pgs: 214 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-19T21:14:52.972 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:52.066876+0000 osd.3 (osd.3) 263 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:14:52.972 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:52.068772+0000 osd.3 (osd.3) 264 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:14:52.972 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:52.371617+0000 mon.a (mon.0) 3236 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2024-02-19T21:14:53.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:51.294680+0000 mgr.y (mgr.14550) 557 : cluster [DBG] pgmap v973: 214 pgs: 214 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-19T21:14:53.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:52.066876+0000 osd.3 (osd.3) 263 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:14:53.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:52.068772+0000 osd.3 (osd.3) 264 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:14:53.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:52.371617+0000 mon.a (mon.0) 3236 : cluster [DBG] osdmap e648: 8 total, 8 up, 8 in 2024-02-19T21:14:53.748 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:14:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:14:53] "GET /metrics HTTP/1.1" 200 191638 "" "Prometheus/2.33.4" 2024-02-19T21:14:53.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:52.368590+0000 osd.2 (osd.2) 262 : cluster [DBG] 2.e scrub starts 2024-02-19T21:14:53.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:52.369618+0000 osd.2 (osd.2) 263 : cluster [DBG] 2.e scrub ok 2024-02-19T21:14:53.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:53.373663+0000 mon.a (mon.0) 3237 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2024-02-19T21:14:53.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:53.377225+0000 mon.a (mon.0) 3238 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:14:53.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:52.368590+0000 osd.2 (osd.2) 262 : cluster [DBG] 2.e scrub starts 2024-02-19T21:14:53.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:52.369618+0000 osd.2 (osd.2) 263 : cluster [DBG] 2.e scrub ok 2024-02-19T21:14:53.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:53.373663+0000 mon.a (mon.0) 3237 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2024-02-19T21:14:53.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:53.377225+0000 mon.a (mon.0) 3238 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:14:53.834 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:53.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:53.938 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:53.969 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:54.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:52.368590+0000 osd.2 (osd.2) 262 : cluster [DBG] 2.e scrub starts 2024-02-19T21:14:54.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:52.369618+0000 osd.2 (osd.2) 263 : cluster [DBG] 2.e scrub ok 2024-02-19T21:14:54.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:53.373663+0000 mon.a (mon.0) 3237 : cluster [DBG] osdmap e649: 8 total, 8 up, 8 in 2024-02-19T21:14:54.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:53.377225+0000 mon.a (mon.0) 3238 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:14:54.618 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:54.620 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:54.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:54.651 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:14:54.651 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:14:54.722 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:14:54.755 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:54.776 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:14:54.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:54.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:14:54.880 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:14:54.908 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:53.295228+0000 mgr.y (mgr.14550) 558 : cluster [DBG] pgmap v976: 182 pgs: 182 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:14:54.909 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:53.409137+0000 osd.6 (osd.6) 219 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:14:54.909 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:53.410006+0000 osd.6 (osd.6) 220 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:14:54.909 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:53.604375+0000 osd.4 (osd.4) 227 : cluster [DBG] 16.19 scrub starts 2024-02-19T21:14:54.909 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:53.606469+0000 osd.4 (osd.4) 228 : cluster [DBG] 16.19 scrub ok 2024-02-19T21:14:54.909 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:53.915542+0000 osd.7 (osd.7) 297 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:14:54.909 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:53.918517+0000 osd.7 (osd.7) 298 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:14:54.909 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:54.365093+0000 mon.a (mon.0) 3239 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:14:54.909 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:54.365239+0000 mon.a (mon.0) 3240 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2024-02-19T21:14:54.909 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:54.381989+0000 mon.a (mon.0) 3241 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:14:54.910 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:54.931 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:14:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:54.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:14:54.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:53.295228+0000 mgr.y (mgr.14550) 558 : cluster [DBG] pgmap v976: 182 pgs: 182 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:14:54.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:53.409137+0000 osd.6 (osd.6) 219 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:14:54.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:53.410006+0000 osd.6 (osd.6) 220 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:14:54.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:53.604375+0000 osd.4 (osd.4) 227 : cluster [DBG] 16.19 scrub starts 2024-02-19T21:14:54.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:53.606469+0000 osd.4 (osd.4) 228 : cluster [DBG] 16.19 scrub ok 2024-02-19T21:14:54.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:53.915542+0000 osd.7 (osd.7) 297 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:14:54.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:53.918517+0000 osd.7 (osd.7) 298 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:14:54.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:54.365093+0000 mon.a (mon.0) 3239 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:14:54.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:54.365239+0000 mon.a (mon.0) 3240 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2024-02-19T21:14:54.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:54.381989+0000 mon.a (mon.0) 3241 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:14:54.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:53.295228+0000 mgr.y (mgr.14550) 558 : cluster [DBG] pgmap v976: 182 pgs: 182 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-02-19T21:14:54.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:53.409137+0000 osd.6 (osd.6) 219 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:14:54.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:53.410006+0000 osd.6 (osd.6) 220 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:14:54.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:53.604375+0000 osd.4 (osd.4) 227 : cluster [DBG] 16.19 scrub starts 2024-02-19T21:14:54.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:53.606469+0000 osd.4 (osd.4) 228 : cluster [DBG] 16.19 scrub ok 2024-02-19T21:14:54.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:53.915542+0000 osd.7 (osd.7) 297 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:14:54.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:53.918517+0000 osd.7 (osd.7) 298 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:14:54.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:54.365093+0000 mon.a (mon.0) 3239 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:14:54.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:54.365239+0000 mon.a (mon.0) 3240 : cluster [DBG] osdmap e650: 8 total, 8 up, 8 in 2024-02-19T21:14:54.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:54.381989+0000 mon.a (mon.0) 3241 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:14:55.037 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:14:55.064 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:55.089 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:14:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:55.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:14:55.193 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:14:55.226 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:55.246 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:14:55.249 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:55.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:14:55.350 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:14:55.381 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:55.401 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:14:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:55.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:14:55.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.409+0000 7f0f13a53700 1 -- 172.21.15.163:0/1766497811 >> v1:172.21.15.163:6789/0 conn(0x7f0f0c0fd980 legacy=0x7f0f0c0fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:55.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.409+0000 7f0f13a53700 1 -- 172.21.15.163:0/1766497811 shutdown_connections 2024-02-19T21:14:55.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.409+0000 7f0f13a53700 1 -- 172.21.15.163:0/1766497811 >> 172.21.15.163:0/1766497811 conn(0x7f0f0c0f35a0 msgr2=0x7f0f0c0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:55.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.409+0000 7f0f13a53700 1 -- 172.21.15.163:0/1766497811 shutdown_connections 2024-02-19T21:14:55.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.409+0000 7f0f13a53700 1 -- 172.21.15.163:0/1766497811 wait complete. 2024-02-19T21:14:55.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f13a53700 1 librados: starting msgr at 2024-02-19T21:14:55.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f13a53700 1 librados: starting objecter 2024-02-19T21:14:55.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f13a53700 1 Processor -- start 2024-02-19T21:14:55.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f13a53700 1 -- start start 2024-02-19T21:14:55.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f13a53700 1 librados: setting wanted keys 2024-02-19T21:14:55.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f13a53700 1 librados: calling monclient init 2024-02-19T21:14:55.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f13a53700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0f0c1172e0 con 0x7f0f0c0fd980 2024-02-19T21:14:55.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f13a53700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0f0c1184c0 con 0x7f0f0c100780 2024-02-19T21:14:55.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f13a53700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0f0c1196c0 con 0x7f0f0c1042c0 2024-02-19T21:14:55.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f117ef700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f0f0c0fd980 0x7f0f0c0f9a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:51576/0 (socket says 172.21.15.163:51576) 2024-02-19T21:14:55.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f117ef700 1 -- 172.21.15.163:0/2370486898 learned_addr learned my addr 172.21.15.163:0/2370486898 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:55.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f11ff0700 1 --1- 172.21.15.163:0/2370486898 >> v1:172.21.15.163:6790/0 conn(0x7f0f0c1042c0 0x7f0f0c1159a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:49374/0 (socket says 172.21.15.163:49374) 2024-02-19T21:14:55.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f027fc700 10 client.?.objecter ms_handle_connect 0x7f0f0c0fd980 2024-02-19T21:14:55.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f027fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:55.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f027fc700 10 client.?.objecter ms_handle_connect 0x7f0f0c1042c0 2024-02-19T21:14:55.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.410+0000 7f0f027fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:55.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 10 client.?.objecter ms_handle_connect 0x7f0f0c100780 2024-02-19T21:14:55.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:55.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818998692 0 0) 0x7f0f0c1172e0 con 0x7f0f0c0fd980 2024-02-19T21:14:55.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ef8003800 con 0x7f0f0c0fd980 2024-02-19T21:14:55.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572658130 0 0) 0x7f0f0c1196c0 con 0x7f0f0c1042c0 2024-02-19T21:14:55.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f0c1172e0 con 0x7f0f0c1042c0 2024-02-19T21:14:55.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500279944 0 0) 0x7f0f0c1184c0 con 0x7f0f0c100780 2024-02-19T21:14:55.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f0c1196c0 con 0x7f0f0c100780 2024-02-19T21:14:55.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1319519015 0 0) 0x7f0ef8003800 con 0x7f0f0c0fd980 2024-02-19T21:14:55.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f0c1184c0 con 0x7f0f0c0fd980 2024-02-19T21:14:55.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 43200707 0 0) 0x7f0f0c1172e0 con 0x7f0f0c1042c0 2024-02-19T21:14:55.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ef8003800 con 0x7f0f0c1042c0 2024-02-19T21:14:55.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0efc002dd0 con 0x7f0f0c0fd980 2024-02-19T21:14:55.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 949191871 0 0) 0x7f0f0c1196c0 con 0x7f0f0c100780 2024-02-19T21:14:55.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f0c1172e0 con 0x7f0f0c100780 2024-02-19T21:14:55.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0f04001bc0 con 0x7f0f0c1042c0 2024-02-19T21:14:55.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0f080019f0 con 0x7f0f0c100780 2024-02-19T21:14:55.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 536346146 0 0) 0x7f0f0c1184c0 con 0x7f0f0c0fd980 2024-02-19T21:14:55.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 >> v1:172.21.15.163:6790/0 conn(0x7f0f0c1042c0 legacy=0x7f0f0c1159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:55.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 >> v1:172.21.15.193:6789/0 conn(0x7f0f0c100780 legacy=0x7f0f0c0fa130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:55.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0c11a8c0 con 0x7f0f0c0fd980 2024-02-19T21:14:55.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f13a53700 1 -- 172.21.15.163:0/2370486898 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0c1198b0 con 0x7f0f0c0fd980 2024-02-19T21:14:55.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f13a53700 10 client.23631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:55.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f13a53700 1 -- 172.21.15.163:0/2370486898 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0c119d20 con 0x7f0f0c0fd980 2024-02-19T21:14:55.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f13a53700 1 librados: init done 2024-02-19T21:14:55.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f0efc0016d0 con 0x7f0f0c0fd980 2024-02-19T21:14:55.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.411+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0efc003920 con 0x7f0f0c0fd980 2024-02-19T21:14:55.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.412+0000 7f0f13a53700 1 -- 172.21.15.163:0/2370486898 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef0004f20 con 0x7f0f0c0fd980 2024-02-19T21:14:55.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.412+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f0efc003120 con 0x7f0f0c0fd980 2024-02-19T21:14:55.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.413+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(651..651 src has 434..651) v4 ==== 8535+0+0 (unknown 2277600003 0 0) 0x7f0efc0791d0 con 0x7f0f0c0fd980 2024-02-19T21:14:55.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.413+0000 7f0f027fc700 10 client.23631.objecter ms_dispatch 0x7f0f0c001550 osd_map(651..651 src has 434..651) v4 2024-02-19T21:14:55.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.413+0000 7f0f027fc700 3 client.23631.objecter handle_osd_map got epochs [651,651] > 0 2024-02-19T21:14:55.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.413+0000 7f0f027fc700 3 client.23631.objecter handle_osd_map decoding full epoch 651 2024-02-19T21:14:55.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.413+0000 7f0f027fc700 20 client.23631.objecter dump_active .. 0 homeless 2024-02-19T21:14:55.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.413+0000 7f0f027fc700 10 client.23631.objecter ms_handle_connect 0x7f0ef8063ca0 2024-02-19T21:14:55.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.417+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f0efc04c4d0 con 0x7f0f0c0fd980 2024-02-19T21:14:55.505 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:14:55.535 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:55.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:55.603+0000 7f0f13a53700 1 -- 172.21.15.163:0/2370486898 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f0ef00051d0 con 0x7f0f0c0fd980 2024-02-19T21:14:55.850 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:53.811587+0000 osd.0 (osd.0) 131 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:55.850 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:53.811747+0000 osd.0 (osd.0) 132 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:55.850 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:54.653905+0000 osd.4 (osd.4) 229 : cluster [DBG] 5.1 scrub starts 2024-02-19T21:14:55.850 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:54.654911+0000 osd.4 (osd.4) 230 : cluster [DBG] 5.1 scrub ok 2024-02-19T21:14:55.850 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:54.897124+0000 osd.7 (osd.7) 299 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:14:55.851 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:54.900058+0000 osd.7 (osd.7) 300 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:14:55.851 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:55.001013+0000 osd.1 (osd.1) 290 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:14:55.851 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:55.003927+0000 osd.1 (osd.1) 291 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:14:55.851 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:55.368600+0000 mon.a (mon.0) 3242 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-112", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:14:55.851 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:55.368666+0000 mon.a (mon.0) 3243 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2024-02-19T21:14:55.851 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:55.369188+0000 mon.a (mon.0) 3244 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-112"}]: dispatch 2024-02-19T21:14:55.851 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:55.605026+0000 mon.a (mon.0) 3245 : audit [INF] from='client.? v1:172.21.15.163:0/2370486898' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:55.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:53.811587+0000 osd.0 (osd.0) 131 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:55.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:53.811747+0000 osd.0 (osd.0) 132 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:55.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:54.653905+0000 osd.4 (osd.4) 229 : cluster [DBG] 5.1 scrub starts 2024-02-19T21:14:55.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:54.654911+0000 osd.4 (osd.4) 230 : cluster [DBG] 5.1 scrub ok 2024-02-19T21:14:55.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:54.897124+0000 osd.7 (osd.7) 299 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:14:55.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:54.900058+0000 osd.7 (osd.7) 300 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:14:55.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:55.001013+0000 osd.1 (osd.1) 290 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:14:55.852 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:55.003927+0000 osd.1 (osd.1) 291 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:14:55.853 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:55.368600+0000 mon.a (mon.0) 3242 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-112", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:14:55.853 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:55.368666+0000 mon.a (mon.0) 3243 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2024-02-19T21:14:55.853 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:55.369188+0000 mon.a (mon.0) 3244 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-112"}]: dispatch 2024-02-19T21:14:55.853 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:55.605026+0000 mon.a (mon.0) 3245 : audit [INF] from='client.? v1:172.21.15.163:0/2370486898' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:55.853 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:55.873 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-19T21:14:55.987 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:14:55.993 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:55.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:14:56.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:53.811587+0000 osd.0 (osd.0) 131 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:56.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:53.811747+0000 osd.0 (osd.0) 132 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:56.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:54.653905+0000 osd.4 (osd.4) 229 : cluster [DBG] 5.1 scrub starts 2024-02-19T21:14:56.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:54.654911+0000 osd.4 (osd.4) 230 : cluster [DBG] 5.1 scrub ok 2024-02-19T21:14:56.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:54.897124+0000 osd.7 (osd.7) 299 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:14:56.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:54.900058+0000 osd.7 (osd.7) 300 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:14:56.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:55.001013+0000 osd.1 (osd.1) 290 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:14:56.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:55.003927+0000 osd.1 (osd.1) 291 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:14:56.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:55.368600+0000 mon.a (mon.0) 3242 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-112", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:14:56.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:55.368666+0000 mon.a (mon.0) 3243 : cluster [DBG] osdmap e651: 8 total, 8 up, 8 in 2024-02-19T21:14:56.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:55.369188+0000 mon.a (mon.0) 3244 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-112"}]: dispatch 2024-02-19T21:14:56.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:55.605026+0000 mon.a (mon.0) 3245 : audit [INF] from='client.? v1:172.21.15.163:0/2370486898' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:14:56.094 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:14:56.126 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:56.147 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:14:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:56.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:14:56.250 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:56.278 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:56.300 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:56.303 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:56.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:56.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:56.368+0000 7f0f027fc700 1 -- 172.21.15.163:0/2370486898 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v652) v1 ==== 101+0+0 (unknown 3743603800 0 0) 0x7f0efc04a940 con 0x7f0f0c0fd980 2024-02-19T21:14:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:56.370+0000 7f0f13a53700 10 librados: watch_flush enter 2024-02-19T21:14:56.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:56.370+0000 7f0f13a53700 10 librados: watch_flush exit 2024-02-19T21:14:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:56.370+0000 7f0f13a53700 20 client.23631.objecter shutdown clearing up homeless session... 2024-02-19T21:14:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:56.370+0000 7f0f13a53700 10 client.23631.objecter successfully canceled tick 2024-02-19T21:14:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:56.370+0000 7f0f13a53700 1 -- 172.21.15.163:0/2370486898 >> v1:172.21.15.163:6800/1552514681 conn(0x7f0ef8063ca0 legacy=0x7f0ef8066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:56.370+0000 7f0f13a53700 1 -- 172.21.15.163:0/2370486898 >> v1:172.21.15.163:6789/0 conn(0x7f0f0c0fd980 legacy=0x7f0f0c0f9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:56.370+0000 7f0f13a53700 1 -- 172.21.15.163:0/2370486898 shutdown_connections 2024-02-19T21:14:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:56.370+0000 7f0f13a53700 1 -- 172.21.15.163:0/2370486898 >> 172.21.15.163:0/2370486898 conn(0x7f0f0c0f35a0 msgr2=0x7f0f0c107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:56.370+0000 7f0f13a53700 1 -- 172.21.15.163:0/2370486898 shutdown_connections 2024-02-19T21:14:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:56.370+0000 7f0f13a53700 1 -- 172.21.15.163:0/2370486898 wait complete. 2024-02-19T21:14:56.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:56.370+0000 7f0f13a53700 1 librados: shutdown 2024-02-19T21:14:56.372 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:14:56.405 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:14:56.437 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:56.457 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:14:56.459 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:56.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:14:56.561 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:14:56.588 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:56.737 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:14:56.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:56.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:14:56.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:55.010490+0000 osd.5 (osd.5) 258 : cluster [DBG] 16.c scrub starts 2024-02-19T21:14:56.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:55.013480+0000 osd.5 (osd.5) 259 : cluster [DBG] 16.c scrub ok 2024-02-19T21:14:56.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:55.296017+0000 mgr.y (mgr.14550) 559 : cluster [DBG] pgmap v979: 214 pgs: 214 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:14:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:55.608913+0000 osd.4 (osd.4) 231 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:14:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:55.609801+0000 osd.4 (osd.4) 232 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:14:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:55.892289+0000 osd.7 (osd.7) 301 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:14:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:55.895737+0000 osd.7 (osd.7) 302 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:14:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:55.994342+0000 osd.1 (osd.1) 292 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:14:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:55.995343+0000 osd.1 (osd.1) 293 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:14:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:56.047444+0000 osd.3 (osd.3) 265 : cluster [DBG] 5.17 scrub starts 2024-02-19T21:14:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:56.048467+0000 osd.3 (osd.3) 266 : cluster [DBG] 5.17 scrub ok 2024-02-19T21:14:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:56.363578+0000 mon.a (mon.0) 3246 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:14:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:56.367656+0000 mon.a (mon.0) 3247 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-112"}]': finished 2024-02-19T21:14:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:56.367720+0000 mon.a (mon.0) 3248 : audit [INF] from='client.? v1:172.21.15.163:0/2370486898' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:14:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:56.367769+0000 mon.a (mon.0) 3249 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2024-02-19T21:14:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:56.368524+0000 mon.a (mon.0) 3250 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-112", "mode": "writeback"}]: dispatch 2024-02-19T21:14:56.843 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:56.868 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:55.010490+0000 osd.5 (osd.5) 258 : cluster [DBG] 16.c scrub starts 2024-02-19T21:14:56.868 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:55.013480+0000 osd.5 (osd.5) 259 : cluster [DBG] 16.c scrub ok 2024-02-19T21:14:56.868 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:55.296017+0000 mgr.y (mgr.14550) 559 : cluster [DBG] pgmap v979: 214 pgs: 214 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:14:56.868 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:55.608913+0000 osd.4 (osd.4) 231 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:14:56.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:55.609801+0000 osd.4 (osd.4) 232 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:14:56.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:55.892289+0000 osd.7 (osd.7) 301 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:14:56.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:55.895737+0000 osd.7 (osd.7) 302 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:14:56.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:55.994342+0000 osd.1 (osd.1) 292 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:14:56.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:55.995343+0000 osd.1 (osd.1) 293 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:14:56.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:56.047444+0000 osd.3 (osd.3) 265 : cluster [DBG] 5.17 scrub starts 2024-02-19T21:14:56.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:56.048467+0000 osd.3 (osd.3) 266 : cluster [DBG] 5.17 scrub ok 2024-02-19T21:14:56.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:56.363578+0000 mon.a (mon.0) 3246 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:14:56.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:56.367656+0000 mon.a (mon.0) 3247 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-112"}]': finished 2024-02-19T21:14:56.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:56.367720+0000 mon.a (mon.0) 3248 : audit [INF] from='client.? v1:172.21.15.163:0/2370486898' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:14:56.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:56.367769+0000 mon.a (mon.0) 3249 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2024-02-19T21:14:56.869 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:56.368524+0000 mon.a (mon.0) 3250 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-112", "mode": "writeback"}]: dispatch 2024-02-19T21:14:56.870 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:55.010490+0000 osd.5 (osd.5) 258 : cluster [DBG] 16.c scrub starts 2024-02-19T21:14:56.870 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:55.013480+0000 osd.5 (osd.5) 259 : cluster [DBG] 16.c scrub ok 2024-02-19T21:14:56.870 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:55.296017+0000 mgr.y (mgr.14550) 559 : cluster [DBG] pgmap v979: 214 pgs: 214 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:14:56.870 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:55.608913+0000 osd.4 (osd.4) 231 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:14:56.870 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:55.609801+0000 osd.4 (osd.4) 232 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:14:56.870 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:55.892289+0000 osd.7 (osd.7) 301 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:14:56.870 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:55.895737+0000 osd.7 (osd.7) 302 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:14:56.870 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:55.994342+0000 osd.1 (osd.1) 292 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:14:56.870 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:55.995343+0000 osd.1 (osd.1) 293 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:14:56.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:56.047444+0000 osd.3 (osd.3) 265 : cluster [DBG] 5.17 scrub starts 2024-02-19T21:14:56.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:56.048467+0000 osd.3 (osd.3) 266 : cluster [DBG] 5.17 scrub ok 2024-02-19T21:14:56.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:56.363578+0000 mon.a (mon.0) 3246 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:14:56.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:56.367656+0000 mon.a (mon.0) 3247 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-112"}]': finished 2024-02-19T21:14:56.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:56.367720+0000 mon.a (mon.0) 3248 : audit [INF] from='client.? v1:172.21.15.163:0/2370486898' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:14:56.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:56.367769+0000 mon.a (mon.0) 3249 : cluster [DBG] osdmap e652: 8 total, 8 up, 8 in 2024-02-19T21:14:56.871 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:56.368524+0000 mon.a (mon.0) 3250 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-112", "mode": "writeback"}]: dispatch 2024-02-19T21:14:56.871 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:57.019 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:57.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:57.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:57.122 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:14:57.154 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:57.176 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:14:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:57.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:14:57.280 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:57.310 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:57.760 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:57.764 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:57.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:57.865 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:14:57.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.461+0000 7ffb91b61700 1 -- 172.21.15.163:0/3630280125 >> v1:172.21.15.163:6790/0 conn(0x7ffb8c0ff670 legacy=0x7ffb8c101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:57.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.462+0000 7ffb91b61700 1 -- 172.21.15.163:0/3630280125 shutdown_connections 2024-02-19T21:14:57.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.462+0000 7ffb91b61700 1 -- 172.21.15.163:0/3630280125 >> 172.21.15.163:0/3630280125 conn(0x7ffb8c0f2470 msgr2=0x7ffb8c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:57.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.462+0000 7ffb91b61700 1 -- 172.21.15.163:0/3630280125 shutdown_connections 2024-02-19T21:14:57.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.462+0000 7ffb91b61700 1 -- 172.21.15.163:0/3630280125 wait complete. 2024-02-19T21:14:57.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.462+0000 7ffb91b61700 1 librados: starting msgr at 2024-02-19T21:14:57.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.462+0000 7ffb91b61700 1 librados: starting objecter 2024-02-19T21:14:57.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.462+0000 7ffb91b61700 1 Processor -- start 2024-02-19T21:14:57.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.462+0000 7ffb91b61700 1 -- start start 2024-02-19T21:14:57.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.462+0000 7ffb91b61700 1 librados: setting wanted keys 2024-02-19T21:14:57.875 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.462+0000 7ffb91b61700 1 librados: calling monclient init 2024-02-19T21:14:57.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb91b61700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffb8c119480 con 0x7ffb8c0ff670 2024-02-19T21:14:57.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb91b61700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffb8c11a660 con 0x7ffb8c1031b0 2024-02-19T21:14:57.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb91b61700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffb8c11b860 con 0x7ffb8c0fc870 2024-02-19T21:14:57.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb8b7fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7ffb8c0fc870 0x7ffb8c0609e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:49380/0 (socket says 172.21.15.163:49380) 2024-02-19T21:14:57.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb8b7fe700 1 -- 172.21.15.163:0/3047634374 learned_addr learned my addr 172.21.15.163:0/3047634374 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:14:57.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 10 client.?.objecter ms_handle_connect 0x7ffb8c0fc870 2024-02-19T21:14:57.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:57.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 10 client.?.objecter ms_handle_connect 0x7ffb8c0ff670 2024-02-19T21:14:57.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:57.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242554315 0 0) 0x7ffb8c11b860 con 0x7ffb8c0fc870 2024-02-19T21:14:57.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb740033f0 con 0x7ffb8c0fc870 2024-02-19T21:14:57.876 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979823317 0 0) 0x7ffb8c119480 con 0x7ffb8c0ff670 2024-02-19T21:14:57.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb8c11b860 con 0x7ffb8c0ff670 2024-02-19T21:14:57.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 10 client.?.objecter ms_handle_connect 0x7ffb8c1031b0 2024-02-19T21:14:57.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:14:57.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1540061715 0 0) 0x7ffb740033f0 con 0x7ffb8c0fc870 2024-02-19T21:14:57.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb8c119480 con 0x7ffb8c0fc870 2024-02-19T21:14:57.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffb7c002a10 con 0x7ffb8c0fc870 2024-02-19T21:14:57.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2833968534 0 0) 0x7ffb8c11b860 con 0x7ffb8c0ff670 2024-02-19T21:14:57.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb740033f0 con 0x7ffb8c0ff670 2024-02-19T21:14:57.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169060217 0 0) 0x7ffb8c11a660 con 0x7ffb8c1031b0 2024-02-19T21:14:57.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb8c11b860 con 0x7ffb8c1031b0 2024-02-19T21:14:57.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffb80001fe0 con 0x7ffb8c0ff670 2024-02-19T21:14:57.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 112701470 0 0) 0x7ffb8c119480 con 0x7ffb8c0fc870 2024-02-19T21:14:57.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 >> v1:172.21.15.193:6789/0 conn(0x7ffb8c1031b0 legacy=0x7ffb8c117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:57.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 >> v1:172.21.15.163:6789/0 conn(0x7ffb8c0ff670 legacy=0x7ffb8c0610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:57.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8c11ca60 con 0x7ffb8c0fc870 2024-02-19T21:14:57.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8c119670 con 0x7ffb8c0fc870 2024-02-19T21:14:57.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb91b61700 10 client.50357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:14:57.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8c119b60 con 0x7ffb8c0fc870 2024-02-19T21:14:57.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.463+0000 7ffb91b61700 1 librados: init done 2024-02-19T21:14:57.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.464+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ffb7c002d60 con 0x7ffb8c0fc870 2024-02-19T21:14:57.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.464+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffb7c005c70 con 0x7ffb8c0fc870 2024-02-19T21:14:57.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.464+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7ffb78004f00 con 0x7ffb8c0fc870 2024-02-19T21:14:57.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.465+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ffb7c001720 con 0x7ffb8c0fc870 2024-02-19T21:14:57.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.465+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(653..653 src has 434..653) v4 ==== 8535+0+0 (unknown 4189745579 0 0) 0x7ffb7c0790c0 con 0x7ffb8c0fc870 2024-02-19T21:14:57.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.465+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 osd_map(653..653 src has 434..653) v4 2024-02-19T21:14:57.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.465+0000 7ffb88ff9700 3 client.50357.objecter handle_osd_map got epochs [653,653] > 0 2024-02-19T21:14:57.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.465+0000 7ffb88ff9700 3 client.50357.objecter handle_osd_map decoding full epoch 653 2024-02-19T21:14:57.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.465+0000 7ffb88ff9700 20 client.50357.objecter dump_active .. 0 homeless 2024-02-19T21:14:57.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.465+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v653) v1 ==== 54+0+16 (unknown 4240107531 0 2420000192) 0x7ffb7c07b480 con 0x7ffb8c0fc870 2024-02-19T21:14:57.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.465+0000 7ffb88ff9700 10 client.50357.objecter ms_handle_connect 0x7ffb74063cc0 2024-02-19T21:14:57.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb91b61700 10 client.50357.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:57.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 1 2024-02-19T21:14:57.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb780051a0 osd=0 3 2024-02-19T21:14:57.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb780051a0 osd=0 3 2024-02-19T21:14:57.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb780051a0 osd=0 3 2024-02-19T21:14:57.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 1 2024-02-19T21:14:57.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb780051a0 osd=0 3 2024-02-19T21:14:57.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 0 1 2024-02-19T21:14:57.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb780051a0 osd=0 4 2024-02-19T21:14:57.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb91b61700 10 client.50357.objecter _send_command 1 2024-02-19T21:14:57.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb78004950 con 0x7ffb78004530 2024-02-19T21:14:57.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb88ff9700 10 client.50357.objecter ms_handle_connect 0x7ffb78004530 2024-02-19T21:14:57.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffb78004950 con 0x7ffb78004530 2024-02-19T21:14:57.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:14:57.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 1 = system:0 2024-02-19T21:14:57.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb780051a0 osd=0 4 2024-02-19T21:14:57.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.466+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 0 1 2024-02-19T21:14:57.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.481+0000 7ffb91b61700 10 client.50357.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:57.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.481+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 2 2024-02-19T21:14:57.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.481+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb780051a0 osd=0 3 2024-02-19T21:14:57.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.481+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb780051a0 osd=0 3 2024-02-19T21:14:57.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.481+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb780051a0 osd=0 3 2024-02-19T21:14:57.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.481+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 2 2024-02-19T21:14:57.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.481+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb780051a0 osd=0 3 2024-02-19T21:14:57.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.481+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 0 2 2024-02-19T21:14:57.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.481+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb780051a0 osd=0 4 2024-02-19T21:14:57.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.481+0000 7ffb91b61700 10 client.50357.objecter _send_command 2 2024-02-19T21:14:57.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.481+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffb78004d30 con 0x7ffb78004530 2024-02-19T21:14:57.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffb78004d30 con 0x7ffb78004530 2024-02-19T21:14:57.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:57.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb780051a0 osd=0 4 2024-02-19T21:14:57.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 0 2 2024-02-19T21:14:57.881 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:14:57.881 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:14:57.882 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:14:57.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb91b61700 10 client.50357.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:57.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 3 2024-02-19T21:14:57.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78003fe0 osd=1 3 2024-02-19T21:14:57.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78003fe0 osd=1 3 2024-02-19T21:14:57.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78003fe0 osd=1 3 2024-02-19T21:14:57.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 3 2024-02-19T21:14:57.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb78003fe0 osd=1 3 2024-02-19T21:14:57.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 1 3 2024-02-19T21:14:57.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78003fe0 osd=1 4 2024-02-19T21:14:57.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb91b61700 10 client.50357.objecter _send_command 3 2024-02-19T21:14:57.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb7800c000 con 0x7ffb78004150 2024-02-19T21:14:57.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.483+0000 7ffb88ff9700 10 client.50357.objecter ms_handle_connect 0x7ffb78004150 2024-02-19T21:14:57.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.484+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffb7c005e90 con 0x7ffb78004150 2024-02-19T21:14:57.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.484+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:14:57.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.484+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 3 = system:0 2024-02-19T21:14:57.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.484+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb78003fe0 osd=1 4 2024-02-19T21:14:57.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.484+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 1 3 2024-02-19T21:14:57.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.498+0000 7ffb91b61700 10 client.50357.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:57.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.498+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 4 2024-02-19T21:14:57.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.498+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78003fe0 osd=1 3 2024-02-19T21:14:57.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.498+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78003fe0 osd=1 3 2024-02-19T21:14:57.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.498+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78003fe0 osd=1 3 2024-02-19T21:14:57.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.498+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 4 2024-02-19T21:14:57.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.498+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb78003fe0 osd=1 3 2024-02-19T21:14:57.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.498+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 1 4 2024-02-19T21:14:57.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.498+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78003fe0 osd=1 4 2024-02-19T21:14:57.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.498+0000 7ffb91b61700 10 client.50357.objecter _send_command 4 2024-02-19T21:14:57.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.498+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffb780018b0 con 0x7ffb78004150 2024-02-19T21:14:57.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffb780018b0 con 0x7ffb78004150 2024-02-19T21:14:57.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:57.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb78003fe0 osd=1 4 2024-02-19T21:14:57.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 1 4 2024-02-19T21:14:57.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb91b61700 10 client.50357.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:57.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 5 2024-02-19T21:14:57.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78003630 osd=2 3 2024-02-19T21:14:57.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78003630 osd=2 3 2024-02-19T21:14:57.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78003630 osd=2 3 2024-02-19T21:14:57.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 5 2024-02-19T21:14:57.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb78003630 osd=2 3 2024-02-19T21:14:57.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 2 5 2024-02-19T21:14:57.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78003630 osd=2 4 2024-02-19T21:14:57.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb91b61700 10 client.50357.objecter _send_command 5 2024-02-19T21:14:57.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb78003e70 con 0x7ffb7800d2f0 2024-02-19T21:14:57.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.500+0000 7ffb88ff9700 10 client.50357.objecter ms_handle_connect 0x7ffb7800d2f0 2024-02-19T21:14:57.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.501+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffb78003e70 con 0x7ffb7800d2f0 2024-02-19T21:14:57.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.501+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:14:57.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.501+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 5 = system:0 2024-02-19T21:14:57.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.501+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb78003630 osd=2 4 2024-02-19T21:14:57.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.501+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 2 5 2024-02-19T21:14:57.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.515+0000 7ffb91b61700 10 client.50357.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:57.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.515+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 6 2024-02-19T21:14:57.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.515+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78003630 osd=2 3 2024-02-19T21:14:57.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.515+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78003630 osd=2 3 2024-02-19T21:14:57.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.515+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78003630 osd=2 3 2024-02-19T21:14:57.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.515+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 6 2024-02-19T21:14:57.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.515+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb78003630 osd=2 3 2024-02-19T21:14:57.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.515+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 2 6 2024-02-19T21:14:57.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.515+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78003630 osd=2 4 2024-02-19T21:14:57.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.515+0000 7ffb91b61700 10 client.50357.objecter _send_command 6 2024-02-19T21:14:57.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.515+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffb7800e730 con 0x7ffb7800d2f0 2024-02-19T21:14:57.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffb7800e730 con 0x7ffb7800d2f0 2024-02-19T21:14:57.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:57.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb78003630 osd=2 4 2024-02-19T21:14:57.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 2 6 2024-02-19T21:14:57.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb91b61700 10 client.50357.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:57.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 7 2024-02-19T21:14:57.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78003410 osd=3 3 2024-02-19T21:14:57.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78003410 osd=3 3 2024-02-19T21:14:57.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78003410 osd=3 3 2024-02-19T21:14:57.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 7 2024-02-19T21:14:57.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb78003410 osd=3 3 2024-02-19T21:14:57.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 3 7 2024-02-19T21:14:57.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78003410 osd=3 4 2024-02-19T21:14:57.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb91b61700 10 client.50357.objecter _send_command 7 2024-02-19T21:14:57.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.516+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb78015a60 con 0x7ffb78011e80 2024-02-19T21:14:57.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.517+0000 7ffb88ff9700 10 client.50357.objecter ms_handle_connect 0x7ffb78011e80 2024-02-19T21:14:57.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.517+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffb78015a60 con 0x7ffb78011e80 2024-02-19T21:14:57.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.517+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:14:57.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.517+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 7 = system:0 2024-02-19T21:14:57.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.517+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb78003410 osd=3 4 2024-02-19T21:14:57.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.517+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 3 7 2024-02-19T21:14:57.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.532+0000 7ffb91b61700 10 client.50357.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:57.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.532+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 8 2024-02-19T21:14:57.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.532+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78003410 osd=3 3 2024-02-19T21:14:57.888 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.532+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78003410 osd=3 3 2024-02-19T21:14:57.888 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:14:57.889 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:14:57.889 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:14:57.889 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:14:57.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.532+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78003410 osd=3 3 2024-02-19T21:14:57.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.532+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 8 2024-02-19T21:14:57.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.532+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb78003410 osd=3 3 2024-02-19T21:14:57.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.532+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 3 8 2024-02-19T21:14:57.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.532+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78003410 osd=3 4 2024-02-19T21:14:57.889 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.532+0000 7ffb91b61700 10 client.50357.objecter _send_command 8 2024-02-19T21:14:57.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.532+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffb78015a60 con 0x7ffb78011e80 2024-02-19T21:14:57.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.533+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffb78015a60 con 0x7ffb78011e80 2024-02-19T21:14:57.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.533+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:57.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.533+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.533+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb78003410 osd=3 4 2024-02-19T21:14:57.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.533+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 3 8 2024-02-19T21:14:57.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.534+0000 7ffb91b61700 10 client.50357.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:57.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.534+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 9 2024-02-19T21:14:57.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.534+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78015a60 osd=4 3 2024-02-19T21:14:57.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.534+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78015a60 osd=4 3 2024-02-19T21:14:57.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.534+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78015a60 osd=4 3 2024-02-19T21:14:57.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.534+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 9 2024-02-19T21:14:57.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.534+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb78015a60 osd=4 3 2024-02-19T21:14:57.890 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.534+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 4 9 2024-02-19T21:14:57.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.534+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78015a60 osd=4 4 2024-02-19T21:14:57.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.534+0000 7ffb91b61700 10 client.50357.objecter _send_command 9 2024-02-19T21:14:57.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.534+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb7800cf20 con 0x7ffb7800c980 2024-02-19T21:14:57.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.534+0000 7ffb88ff9700 10 client.50357.objecter ms_handle_connect 0x7ffb7800c980 2024-02-19T21:14:57.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.535+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffb7800cf20 con 0x7ffb7800c980 2024-02-19T21:14:57.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.535+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:14:57.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.535+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 9 = system:0 2024-02-19T21:14:57.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.535+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb78015a60 osd=4 4 2024-02-19T21:14:57.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.535+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 4 9 2024-02-19T21:14:57.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.550+0000 7ffb91b61700 10 client.50357.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:57.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.550+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 10 2024-02-19T21:14:57.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.550+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78015a60 osd=4 3 2024-02-19T21:14:57.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.550+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78015a60 osd=4 3 2024-02-19T21:14:57.891 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.550+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78015a60 osd=4 3 2024-02-19T21:14:57.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.550+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 10 2024-02-19T21:14:57.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.550+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb78015a60 osd=4 3 2024-02-19T21:14:57.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.550+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 4 10 2024-02-19T21:14:57.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.550+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78015a60 osd=4 4 2024-02-19T21:14:57.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.550+0000 7ffb91b61700 10 client.50357.objecter _send_command 10 2024-02-19T21:14:57.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.550+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffb7800d160 con 0x7ffb7800c980 2024-02-19T21:14:57.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffb7800d160 con 0x7ffb7800c980 2024-02-19T21:14:57.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:57.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb78015a60 osd=4 4 2024-02-19T21:14:57.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 4 10 2024-02-19T21:14:57.892 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb91b61700 10 client.50357.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:57.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 11 2024-02-19T21:14:57.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb7800d160 osd=5 3 2024-02-19T21:14:57.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb7800d160 osd=5 3 2024-02-19T21:14:57.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb7800d160 osd=5 3 2024-02-19T21:14:57.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 11 2024-02-19T21:14:57.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb7800d160 osd=5 3 2024-02-19T21:14:57.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 5 11 2024-02-19T21:14:57.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb7800d160 osd=5 4 2024-02-19T21:14:57.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb91b61700 10 client.50357.objecter _send_command 11 2024-02-19T21:14:57.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb78017df0 con 0x7ffb780179a0 2024-02-19T21:14:57.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.552+0000 7ffb88ff9700 10 client.50357.objecter ms_handle_connect 0x7ffb780179a0 2024-02-19T21:14:57.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.553+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffb78017df0 con 0x7ffb780179a0 2024-02-19T21:14:57.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.553+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:14:57.893 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.553+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 11 = system:0 2024-02-19T21:14:57.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.553+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb7800d160 osd=5 4 2024-02-19T21:14:57.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.553+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 5 11 2024-02-19T21:14:57.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.567+0000 7ffb91b61700 10 client.50357.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:57.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.567+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 12 2024-02-19T21:14:57.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.567+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb7800d160 osd=5 3 2024-02-19T21:14:57.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.567+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb7800d160 osd=5 3 2024-02-19T21:14:57.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.567+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb7800d160 osd=5 3 2024-02-19T21:14:57.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.567+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 12 2024-02-19T21:14:57.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.567+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb7800d160 osd=5 3 2024-02-19T21:14:57.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.567+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 5 12 2024-02-19T21:14:57.894 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.567+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb7800d160 osd=5 4 2024-02-19T21:14:57.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.567+0000 7ffb91b61700 10 client.50357.objecter _send_command 12 2024-02-19T21:14:57.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.567+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffb78017df0 con 0x7ffb780179a0 2024-02-19T21:14:57.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffb78017df0 con 0x7ffb780179a0 2024-02-19T21:14:57.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:57.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb7800d160 osd=5 4 2024-02-19T21:14:57.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 5 12 2024-02-19T21:14:57.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb91b61700 10 client.50357.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:57.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 13 2024-02-19T21:14:57.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78017df0 osd=6 3 2024-02-19T21:14:57.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78017df0 osd=6 3 2024-02-19T21:14:57.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78017df0 osd=6 3 2024-02-19T21:14:57.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 13 2024-02-19T21:14:57.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb78017df0 osd=6 3 2024-02-19T21:14:57.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 6 13 2024-02-19T21:14:57.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78017df0 osd=6 4 2024-02-19T21:14:57.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb91b61700 10 client.50357.objecter _send_command 13 2024-02-19T21:14:57.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb78022b70 con 0x7ffb7801f060 2024-02-19T21:14:57.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.569+0000 7ffb88ff9700 10 client.50357.objecter ms_handle_connect 0x7ffb7801f060 2024-02-19T21:14:57.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.570+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffb78022b70 con 0x7ffb7801f060 2024-02-19T21:14:57.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.570+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:14:57.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.570+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 13 = system:0 2024-02-19T21:14:57.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.570+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb78017df0 osd=6 4 2024-02-19T21:14:57.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.570+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 6 13 2024-02-19T21:14:57.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.584+0000 7ffb91b61700 10 client.50357.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:57.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.584+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 14 2024-02-19T21:14:57.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.584+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78017df0 osd=6 3 2024-02-19T21:14:57.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.584+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78017df0 osd=6 3 2024-02-19T21:14:57.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.584+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78017df0 osd=6 3 2024-02-19T21:14:57.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.584+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 14 2024-02-19T21:14:57.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.584+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb78017df0 osd=6 3 2024-02-19T21:14:57.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.584+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 6 14 2024-02-19T21:14:57.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.584+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78017df0 osd=6 4 2024-02-19T21:14:57.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.584+0000 7ffb91b61700 10 client.50357.objecter _send_command 14 2024-02-19T21:14:57.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.584+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffb78022b70 con 0x7ffb7801f060 2024-02-19T21:14:57.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffb78022b70 con 0x7ffb7801f060 2024-02-19T21:14:57.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:57.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb78017df0 osd=6 4 2024-02-19T21:14:57.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 6 14 2024-02-19T21:14:57.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb91b61700 10 client.50357.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:14:57.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 15 2024-02-19T21:14:57.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78022b70 osd=7 3 2024-02-19T21:14:57.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78022b70 osd=7 3 2024-02-19T21:14:57.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78022b70 osd=7 3 2024-02-19T21:14:57.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 15 2024-02-19T21:14:57.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb78022b70 osd=7 3 2024-02-19T21:14:57.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 7 15 2024-02-19T21:14:57.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78022b70 osd=7 4 2024-02-19T21:14:57.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb91b61700 10 client.50357.objecter _send_command 15 2024-02-19T21:14:57.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb78026e30 con 0x7ffb780231f0 2024-02-19T21:14:57.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb88ff9700 10 client.50357.objecter ms_handle_connect 0x7ffb780231f0 2024-02-19T21:14:57.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffb78026e30 con 0x7ffb780231f0 2024-02-19T21:14:57.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:14:57.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 15 = system:0 2024-02-19T21:14:57.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb78022b70 osd=7 4 2024-02-19T21:14:57.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.587+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 7 15 2024-02-19T21:14:57.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.602+0000 7ffb91b61700 10 client.50357.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:14:57.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.602+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign -1 16 2024-02-19T21:14:57.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.602+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78022b70 osd=7 3 2024-02-19T21:14:57.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.602+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78022b70 osd=7 3 2024-02-19T21:14:57.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.602+0000 7ffb91b61700 20 client.50357.objecter _get_session s=0x7ffb78022b70 osd=7 3 2024-02-19T21:14:57.899 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:14:57.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.602+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_remove -1 16 2024-02-19T21:14:57.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.602+0000 7ffb91b61700 20 client.50357.objecter get_session s=0x7ffb78022b70 osd=7 3 2024-02-19T21:14:57.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.602+0000 7ffb91b61700 15 client.50357.objecter _session_command_op_assign 7 16 2024-02-19T21:14:57.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.602+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78022b70 osd=7 4 2024-02-19T21:14:57.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.602+0000 7ffb91b61700 10 client.50357.objecter _send_command 16 2024-02-19T21:14:57.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.602+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7ffb780285e0 con 0x7ffb780231f0 2024-02-19T21:14:57.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb88ff9700 1 -- 172.21.15.163:0/3047634374 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7ffb780285e0 con 0x7ffb780231f0 2024-02-19T21:14:57.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb88ff9700 10 client.50357.objecter ms_dispatch 0x7ffb8c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:14:57.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb88ff9700 10 client.50357.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb88ff9700 20 client.50357.objecter put_session s=0x7ffb78022b70 osd=7 4 2024-02-19T21:14:57.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb88ff9700 15 client.50357.objecter _session_command_op_remove 7 16 2024-02-19T21:14:57.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 10 librados: watch_flush enter 2024-02-19T21:14:57.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 10 librados: watch_flush exit 2024-02-19T21:14:57.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 10 client.50357.objecter close_session for osd.0 2024-02-19T21:14:57.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 >> v1:172.21.15.163:6801/1559418115 conn(0x7ffb78004530 legacy=0x7ffb78002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:57.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb780051a0 osd=0 1 2024-02-19T21:14:57.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 10 client.50357.objecter close_session for osd.1 2024-02-19T21:14:57.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 >> v1:172.21.15.163:6805/3125047432 conn(0x7ffb78004150 legacy=0x7ffb7800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:57.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78003fe0 osd=1 1 2024-02-19T21:14:57.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 10 client.50357.objecter close_session for osd.2 2024-02-19T21:14:57.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 >> v1:172.21.15.163:6809/1563158487 conn(0x7ffb7800d2f0 legacy=0x7ffb7800d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:57.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78003630 osd=2 1 2024-02-19T21:14:57.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 10 client.50357.objecter close_session for osd.3 2024-02-19T21:14:57.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 >> v1:172.21.15.163:6813/2112279769 conn(0x7ffb78011e80 legacy=0x7ffb78014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:57.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78003410 osd=3 1 2024-02-19T21:14:57.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 10 client.50357.objecter close_session for osd.4 2024-02-19T21:14:57.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 >> v1:172.21.15.193:6800/3129716965 conn(0x7ffb7800c980 legacy=0x7ffb78015c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:57.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78015a60 osd=4 1 2024-02-19T21:14:57.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 10 client.50357.objecter close_session for osd.5 2024-02-19T21:14:57.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 >> v1:172.21.15.193:6804/805161991 conn(0x7ffb780179a0 legacy=0x7ffb7801d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:57.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb7800d160 osd=5 1 2024-02-19T21:14:57.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 10 client.50357.objecter close_session for osd.6 2024-02-19T21:14:57.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 >> v1:172.21.15.193:6808/787604615 conn(0x7ffb7801f060 legacy=0x7ffb78021450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:57.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb8bfff700 1 -- 172.21.15.163:0/3047634374 reap_dead start 2024-02-19T21:14:57.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78017df0 osd=6 1 2024-02-19T21:14:57.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 10 client.50357.objecter close_session for osd.7 2024-02-19T21:14:57.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 >> v1:172.21.15.193:6812/157927432 conn(0x7ffb780231f0 legacy=0x7ffb780255e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:57.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb8bfff700 1 -- 172.21.15.163:0/3047634374 reap_dead start 2024-02-19T21:14:57.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 20 client.50357.objecter put_session s=0x7ffb78022b70 osd=7 1 2024-02-19T21:14:57.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 20 client.50357.objecter shutdown clearing up homeless session... 2024-02-19T21:14:57.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 10 client.50357.objecter successfully canceled tick 2024-02-19T21:14:57.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb8bfff700 1 -- 172.21.15.163:0/3047634374 reap_dead start 2024-02-19T21:14:57.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb8bfff700 1 -- 172.21.15.163:0/3047634374 reap_dead start 2024-02-19T21:14:57.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 >> v1:172.21.15.163:6800/1552514681 conn(0x7ffb74063cc0 legacy=0x7ffb74066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:57.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 >> v1:172.21.15.163:6790/0 conn(0x7ffb8c0fc870 legacy=0x7ffb8c0609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:14:57.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 shutdown_connections 2024-02-19T21:14:57.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.604+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 >> 172.21.15.163:0/3047634374 conn(0x7ffb8c0f2470 msgr2=0x7ffb8c106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:14:57.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.605+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 shutdown_connections 2024-02-19T21:14:57.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.605+0000 7ffb91b61700 1 -- 172.21.15.163:0/3047634374 wait complete. 2024-02-19T21:14:57.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:14:57.605+0000 7ffb91b61700 1 librados: shutdown 2024-02-19T21:14:57.904 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.904 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.904 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.904 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.904 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.904 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.904 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.904 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:14:57.905 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:56.607668+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:14:57.905 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:56.610229+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:14:57.905 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:56.856073+0000 osd.7 (osd.7) 303 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:14:57.905 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:56.858951+0000 osd.7 (osd.7) 304 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:14:57.905 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:57.365077+0000 mon.a (mon.0) 3251 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:14:57.905 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:57.371759+0000 mon.a (mon.0) 3252 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-112", "mode": "writeback"}]': finished 2024-02-19T21:14:57.905 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:57.371819+0000 mon.a (mon.0) 3253 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2024-02-19T21:14:57.905 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:57.422163+0000 mon.a (mon.0) 3254 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:14:57.905 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:57.466621+0000 mon.c (mon.2) 418 : audit [DBG] from='client.? v1:172.21.15.163:0/3047634374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:57.905 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:56.607668+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:14:57.906 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:56.610229+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:14:57.906 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:56.856073+0000 osd.7 (osd.7) 303 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:14:57.906 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:56.858951+0000 osd.7 (osd.7) 304 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:14:57.906 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:57.365077+0000 mon.a (mon.0) 3251 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:14:57.906 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:57.371759+0000 mon.a (mon.0) 3252 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-112", "mode": "writeback"}]': finished 2024-02-19T21:14:57.906 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:57.371819+0000 mon.a (mon.0) 3253 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2024-02-19T21:14:57.906 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:57.422163+0000 mon.a (mon.0) 3254 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:14:57.906 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:57.466621+0000 mon.c (mon.2) 418 : audit [DBG] from='client.? v1:172.21.15.163:0/3047634374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:57.906 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:58.071 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:14:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:14:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:14:58.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:56.607668+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:14:58.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:56.610229+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:14:58.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:56.856073+0000 osd.7 (osd.7) 303 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:14:58.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:56.858951+0000 osd.7 (osd.7) 304 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:14:58.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:57.365077+0000 mon.a (mon.0) 3251 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:14:58.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:57.371759+0000 mon.a (mon.0) 3252 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-112", "mode": "writeback"}]': finished 2024-02-19T21:14:58.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:57.371819+0000 mon.a (mon.0) 3253 : cluster [DBG] osdmap e653: 8 total, 8 up, 8 in 2024-02-19T21:14:58.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:57.422163+0000 mon.a (mon.0) 3254 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:14:58.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:57.466621+0000 mon.c (mon.2) 418 : audit [DBG] from='client.? v1:172.21.15.163:0/3047634374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:14:58.121 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:14:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:58.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:14:58.236 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:14:58.268 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:58.289 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:14:58.294 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:58.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:14:58.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:58.425 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:58.548 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:58.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:58.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:58.652 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:58.677 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:57.079835+0000 osd.5 (osd.5) 260 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:14:58.678 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:57.082982+0000 osd.5 (osd.5) 261 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:14:58.678 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:57.296485+0000 mgr.y (mgr.14550) 560 : cluster [DBG] pgmap v982: 214 pgs: 214 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:14:58.678 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:57.634376+0000 osd.7 (osd.7) 305 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:58.678 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:57.634592+0000 osd.7 (osd.7) 306 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:58.678 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:58.018588+0000 osd.3 (osd.3) 267 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:14:58.678 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:58.020111+0000 osd.3 (osd.3) 268 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:14:58.678 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:58.371928+0000 mon.a (mon.0) 3255 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:58.678 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:58.371988+0000 mon.a (mon.0) 3256 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2024-02-19T21:14:58.678 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:58.373587+0000 mon.a (mon.0) 3257 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-112"}]: dispatch 2024-02-19T21:14:58.679 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:57.079835+0000 osd.5 (osd.5) 260 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:14:58.679 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:57.082982+0000 osd.5 (osd.5) 261 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:14:58.679 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:57.296485+0000 mgr.y (mgr.14550) 560 : cluster [DBG] pgmap v982: 214 pgs: 214 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:14:58.679 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:57.634376+0000 osd.7 (osd.7) 305 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:58.679 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:57.634592+0000 osd.7 (osd.7) 306 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:58.679 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:58.018588+0000 osd.3 (osd.3) 267 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:14:58.679 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:58.020111+0000 osd.3 (osd.3) 268 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:14:58.679 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:58.371928+0000 mon.a (mon.0) 3255 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:58.679 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:58.371988+0000 mon.a (mon.0) 3256 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2024-02-19T21:14:58.679 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:58.373587+0000 mon.a (mon.0) 3257 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-112"}]: dispatch 2024-02-19T21:14:58.680 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:58.716 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:58.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:58.820 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:14:58.847 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:59.016 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:14:59.044 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:14:59.047 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:59.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:14:59.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:57.079835+0000 osd.5 (osd.5) 260 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:14:59.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:57.082982+0000 osd.5 (osd.5) 261 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:14:59.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:57.296485+0000 mgr.y (mgr.14550) 560 : cluster [DBG] pgmap v982: 214 pgs: 214 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:14:59.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:57.634376+0000 osd.7 (osd.7) 305 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:59.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:57.634592+0000 osd.7 (osd.7) 306 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:59.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:58.018588+0000 osd.3 (osd.3) 267 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:14:59.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:58.020111+0000 osd.3 (osd.3) 268 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:14:59.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:58.371928+0000 mon.a (mon.0) 3255 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:14:59.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:58.371988+0000 mon.a (mon.0) 3256 : cluster [DBG] osdmap e654: 8 total, 8 up, 8 in 2024-02-19T21:14:59.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:58.373587+0000 mon.a (mon.0) 3257 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-112"}]: dispatch 2024-02-19T21:14:59.149 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:59.177 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:59.725 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:14:59.728 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:14:59.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:14:59.781 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:14:59.829 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:14:59.858 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:14:59.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:57.903495+0000 osd.6 (osd.6) 221 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:59.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:57.903846+0000 osd.6 (osd.6) 222 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:59.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:58.115394+0000 osd.5 (osd.5) 262 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:14:59.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:58.119073+0000 osd.5 (osd.5) 263 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:14:59.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:58.301511+0000 osd.2 (osd.2) 264 : cluster [DBG] 5.d scrub starts 2024-02-19T21:14:59.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:58.303094+0000 osd.2 (osd.2) 265 : cluster [DBG] 5.d scrub ok 2024-02-19T21:14:59.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:58.877043+0000 osd.7 (osd.7) 307 : cluster [DBG] 2.20 scrub starts 2024-02-19T21:14:59.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:58.880617+0000 osd.7 (osd.7) 308 : cluster [DBG] 2.20 scrub ok 2024-02-19T21:14:59.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:59.057710+0000 osd.3 (osd.3) 269 : cluster [DBG] 16.7 deep-scrub starts 2024-02-19T21:14:59.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:59.059567+0000 osd.3 (osd.3) 270 : cluster [DBG] 16.7 deep-scrub ok 2024-02-19T21:14:59.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:59.368385+0000 mon.a (mon.0) 3258 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:14:59.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:59.373953+0000 mon.a (mon.0) 3259 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-112"}]': finished 2024-02-19T21:14:59.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:59.374035+0000 mon.a (mon.0) 3260 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2024-02-19T21:14:59.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:57.903495+0000 osd.6 (osd.6) 221 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:14:59.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:57.903846+0000 osd.6 (osd.6) 222 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:14:59.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:58.115394+0000 osd.5 (osd.5) 262 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:14:59.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:58.119073+0000 osd.5 (osd.5) 263 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:14:59.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:58.301511+0000 osd.2 (osd.2) 264 : cluster [DBG] 5.d scrub starts 2024-02-19T21:14:59.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:58.303094+0000 osd.2 (osd.2) 265 : cluster [DBG] 5.d scrub ok 2024-02-19T21:14:59.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:58.877043+0000 osd.7 (osd.7) 307 : cluster [DBG] 2.20 scrub starts 2024-02-19T21:14:59.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:58.880617+0000 osd.7 (osd.7) 308 : cluster [DBG] 2.20 scrub ok 2024-02-19T21:14:59.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:59.057710+0000 osd.3 (osd.3) 269 : cluster [DBG] 16.7 deep-scrub starts 2024-02-19T21:14:59.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:59.059567+0000 osd.3 (osd.3) 270 : cluster [DBG] 16.7 deep-scrub ok 2024-02-19T21:14:59.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:59.368385+0000 mon.a (mon.0) 3258 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:00.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:59.373953+0000 mon.a (mon.0) 3259 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-112"}]': finished 2024-02-19T21:15:00.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:14:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:59.374035+0000 mon.a (mon.0) 3260 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2024-02-19T21:15:00.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:57.903495+0000 osd.6 (osd.6) 221 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:15:00.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:57.903846+0000 osd.6 (osd.6) 222 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:15:00.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:58.115394+0000 osd.5 (osd.5) 262 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:15:00.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:58.119073+0000 osd.5 (osd.5) 263 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:15:00.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:58.301511+0000 osd.2 (osd.2) 264 : cluster [DBG] 5.d scrub starts 2024-02-19T21:15:00.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:58.303094+0000 osd.2 (osd.2) 265 : cluster [DBG] 5.d scrub ok 2024-02-19T21:15:00.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:58.877043+0000 osd.7 (osd.7) 307 : cluster [DBG] 2.20 scrub starts 2024-02-19T21:15:00.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:58.880617+0000 osd.7 (osd.7) 308 : cluster [DBG] 2.20 scrub ok 2024-02-19T21:15:00.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:59.057710+0000 osd.3 (osd.3) 269 : cluster [DBG] 16.7 deep-scrub starts 2024-02-19T21:15:00.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:59.059567+0000 osd.3 (osd.3) 270 : cluster [DBG] 16.7 deep-scrub ok 2024-02-19T21:15:00.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:59.368385+0000 mon.a (mon.0) 3258 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:00.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:59.373953+0000 mon.a (mon.0) 3259 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-112"}]': finished 2024-02-19T21:15:00.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:14:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:59.374035+0000 mon.a (mon.0) 3260 : cluster [DBG] osdmap e655: 8 total, 8 up, 8 in 2024-02-19T21:15:00.569 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:15:00.896 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:59.157456+0000 osd.5 (osd.5) 264 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:15:00.897 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:59.160461+0000 osd.5 (osd.5) 265 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:15:00.897 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:59.296892+0000 mgr.y (mgr.14550) 561 : cluster [DBG] pgmap v985: 214 pgs: 214 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:15:00.897 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:59.654149+0000 osd.4 (osd.4) 235 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:15:00.897 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:59.654944+0000 osd.4 (osd.4) 236 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:15:00.897 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:14:59.876795+0000 mon.a (mon.0) 3261 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:00.897 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:59.923910+0000 osd.7 (osd.7) 309 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:15:00.897 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:14:59.926996+0000 osd.7 (osd.7) 310 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:15:00.897 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:00.100729+0000 osd.3 (osd.3) 271 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:15:00.897 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:00.101583+0000 osd.3 (osd.3) 272 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:15:00.897 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:00.376442+0000 mon.a (mon.0) 3262 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2024-02-19T21:15:00.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:59.157456+0000 osd.5 (osd.5) 264 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:15:00.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:59.160461+0000 osd.5 (osd.5) 265 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:15:00.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:59.296892+0000 mgr.y (mgr.14550) 561 : cluster [DBG] pgmap v985: 214 pgs: 214 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:15:00.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:59.654149+0000 osd.4 (osd.4) 235 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:15:00.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:59.654944+0000 osd.4 (osd.4) 236 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:15:00.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:14:59.876795+0000 mon.a (mon.0) 3261 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:00.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:59.923910+0000 osd.7 (osd.7) 309 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:15:00.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:14:59.926996+0000 osd.7 (osd.7) 310 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:15:00.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:00.100729+0000 osd.3 (osd.3) 271 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:15:00.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:00.101583+0000 osd.3 (osd.3) 272 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:15:00.898 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:00.376442+0000 mon.a (mon.0) 3262 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2024-02-19T21:15:00.911 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:00.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:01.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:01.045 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:15:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:15:01.045 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:59.157456+0000 osd.5 (osd.5) 264 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:15:01.045 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:59.160461+0000 osd.5 (osd.5) 265 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:15:01.045 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:59.296892+0000 mgr.y (mgr.14550) 561 : cluster [DBG] pgmap v985: 214 pgs: 214 active+clean; 5.3 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:15:01.045 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:59.654149+0000 osd.4 (osd.4) 235 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:15:01.045 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:59.654944+0000 osd.4 (osd.4) 236 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:15:01.046 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:14:59.876795+0000 mon.a (mon.0) 3261 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:01.046 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:59.923910+0000 osd.7 (osd.7) 309 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:15:01.046 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:14:59.926996+0000 osd.7 (osd.7) 310 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:15:01.046 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:00.100729+0000 osd.3 (osd.3) 271 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:15:01.046 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:00.101583+0000 osd.3 (osd.3) 272 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:15:01.046 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:00.376442+0000 mon.a (mon.0) 3262 : cluster [DBG] osdmap e656: 8 total, 8 up, 8 in 2024-02-19T21:15:01.047 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:01.069 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:01.072 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:01.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:01.173 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:01.200 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:01.228 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:01.230 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:01.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:01.331 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:01.361 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:01.392 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:01.395 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:01.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:01.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.433+0000 7f1fda524700 1 -- 172.21.15.163:0/1474038309 >> v1:172.21.15.163:6790/0 conn(0x7f1fd41042c0 legacy=0x7f1fd4106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:01.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.433+0000 7f1fda524700 1 -- 172.21.15.163:0/1474038309 shutdown_connections 2024-02-19T21:15:01.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.433+0000 7f1fda524700 1 -- 172.21.15.163:0/1474038309 >> 172.21.15.163:0/1474038309 conn(0x7f1fd40f35a0 msgr2=0x7f1fd40f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:01.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.433+0000 7f1fda524700 1 -- 172.21.15.163:0/1474038309 shutdown_connections 2024-02-19T21:15:01.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.433+0000 7f1fda524700 1 -- 172.21.15.163:0/1474038309 wait complete. 2024-02-19T21:15:01.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fda524700 1 librados: starting msgr at 2024-02-19T21:15:01.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fda524700 1 librados: starting objecter 2024-02-19T21:15:01.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fda524700 1 Processor -- start 2024-02-19T21:15:01.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fda524700 1 -- start start 2024-02-19T21:15:01.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fda524700 1 librados: setting wanted keys 2024-02-19T21:15:01.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fda524700 1 librados: calling monclient init 2024-02-19T21:15:01.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fda524700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1fd41171e0 con 0x7f1fd40fd980 2024-02-19T21:15:01.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fda524700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1fd41183c0 con 0x7f1fd4100780 2024-02-19T21:15:01.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fda524700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1fd41195c0 con 0x7f1fd41042c0 2024-02-19T21:15:01.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fd8ac1700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f1fd41042c0 0x7f1fd4115940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:49402/0 (socket says 172.21.15.163:49402) 2024-02-19T21:15:01.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fd8ac1700 1 -- 172.21.15.163:0/1576525506 learned_addr learned my addr 172.21.15.163:0/1576525506 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:01.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fd17fa700 10 client.?.objecter ms_handle_connect 0x7f1fd41042c0 2024-02-19T21:15:01.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fd17fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:01.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fd17fa700 10 client.?.objecter ms_handle_connect 0x7f1fd4100780 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fd17fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fd17fa700 10 client.?.objecter ms_handle_connect 0x7f1fd40fd980 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fd17fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.434+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529628072 0 0) 0x7f1fd41195c0 con 0x7f1fd41042c0 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc0003030 con 0x7f1fd41042c0 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112461840 0 0) 0x7f1fd41171e0 con 0x7f1fd40fd980 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fd41195c0 con 0x7f1fd40fd980 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748415736 0 0) 0x7f1fd41183c0 con 0x7f1fd4100780 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fd41171e0 con 0x7f1fd4100780 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 710650031 0 0) 0x7f1fc0003030 con 0x7f1fd41042c0 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fd41183c0 con 0x7f1fd41042c0 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1fcc0030c0 con 0x7f1fd41042c0 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2040276723 0 0) 0x7f1fd41195c0 con 0x7f1fd40fd980 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fc0003030 con 0x7f1fd40fd980 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1fc4001620 con 0x7f1fd40fd980 2024-02-19T21:15:01.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2545086936 0 0) 0x7f1fd41183c0 con 0x7f1fd41042c0 2024-02-19T21:15:01.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 >> v1:172.21.15.193:6789/0 conn(0x7f1fd4100780 legacy=0x7f1fd40f9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:01.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 >> v1:172.21.15.163:6789/0 conn(0x7f1fd40fd980 legacy=0x7f1fd40f97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:01.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd411a7c0 con 0x7f1fd41042c0 2024-02-19T21:15:01.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fda524700 1 -- 172.21.15.163:0/1576525506 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd41185b0 con 0x7f1fd41042c0 2024-02-19T21:15:01.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fda524700 10 client.50363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:01.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fda524700 1 -- 172.21.15.163:0/1576525506 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd4118a20 con 0x7f1fd41042c0 2024-02-19T21:15:01.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.435+0000 7f1fda524700 1 librados: init done 2024-02-19T21:15:01.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.436+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f1fcc0026c0 con 0x7f1fd41042c0 2024-02-19T21:15:01.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.436+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1fcc003ab0 con 0x7f1fd41042c0 2024-02-19T21:15:01.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.436+0000 7f1fda524700 1 -- 172.21.15.163:0/1576525506 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb4004f20 con 0x7f1fd41042c0 2024-02-19T21:15:01.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.436+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f1fcc003c10 con 0x7f1fd41042c0 2024-02-19T21:15:01.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.437+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(657..657 src has 434..657) v4 ==== 8527+0+0 (unknown 320631708 0 0) 0x7f1fcc079480 con 0x7f1fd41042c0 2024-02-19T21:15:01.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.437+0000 7f1fd17fa700 10 client.50363.objecter ms_dispatch 0x7f1fd4001550 osd_map(657..657 src has 434..657) v4 2024-02-19T21:15:01.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.437+0000 7f1fd17fa700 3 client.50363.objecter handle_osd_map got epochs [657,657] > 0 2024-02-19T21:15:01.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.437+0000 7f1fd17fa700 3 client.50363.objecter handle_osd_map decoding full epoch 657 2024-02-19T21:15:01.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.437+0000 7f1fd17fa700 20 client.50363.objecter dump_active .. 0 homeless 2024-02-19T21:15:01.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.437+0000 7f1fd17fa700 10 client.50363.objecter ms_handle_connect 0x7f1fc0063b20 2024-02-19T21:15:01.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.441+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1fcc04d5f0 con 0x7f1fd41042c0 2024-02-19T21:15:01.496 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:01.524 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:01.544 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:01.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:01.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:01.599 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:15:01.600 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:15:01.600 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-02-19T21:15:01.600 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:15:01.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:01.619+0000 7f1fda524700 1 -- 172.21.15.163:0/1576525506 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7f1fb40051d0 con 0x7f1fd41042c0 2024-02-19T21:15:01.647 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:15:01.674 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:01.694 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:15:01.696 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:01.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:15:01.797 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:15:01.824 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:01.843 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:15:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:01.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:15:01.947 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:15:01.971 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:00.660106+0000 mgr.y (mgr.14550) 562 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:01.971 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:00.959359+0000 osd.7 (osd.7) 311 : cluster [DBG] 2.b deep-scrub starts 2024-02-19T21:15:01.972 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:00.963589+0000 osd.7 (osd.7) 312 : cluster [DBG] 2.b deep-scrub ok 2024-02-19T21:15:01.972 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:01.377810+0000 mon.a (mon.0) 3263 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2024-02-19T21:15:01.972 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:01.381711+0000 mon.a (mon.0) 3264 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:01.972 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:01.620154+0000 mon.c (mon.2) 419 : audit [INF] from='client.? v1:172.21.15.163:0/1576525506' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:15:01.972 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:01.620444+0000 mon.a (mon.0) 3265 : audit [INF] from='client.50363 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:15:01.973 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:01.991 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:15:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:01.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:15:01.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:00.660106+0000 mgr.y (mgr.14550) 562 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:01.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:00.959359+0000 osd.7 (osd.7) 311 : cluster [DBG] 2.b deep-scrub starts 2024-02-19T21:15:01.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:00.963589+0000 osd.7 (osd.7) 312 : cluster [DBG] 2.b deep-scrub ok 2024-02-19T21:15:01.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:01.377810+0000 mon.a (mon.0) 3263 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2024-02-19T21:15:01.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:01.381711+0000 mon.a (mon.0) 3264 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:01.620154+0000 mon.c (mon.2) 419 : audit [INF] from='client.? v1:172.21.15.163:0/1576525506' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:15:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:01.620444+0000 mon.a (mon.0) 3265 : audit [INF] from='client.50363 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:15:01.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:00.660106+0000 mgr.y (mgr.14550) 562 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:01.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:00.959359+0000 osd.7 (osd.7) 311 : cluster [DBG] 2.b deep-scrub starts 2024-02-19T21:15:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:00.963589+0000 osd.7 (osd.7) 312 : cluster [DBG] 2.b deep-scrub ok 2024-02-19T21:15:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:01.377810+0000 mon.a (mon.0) 3263 : cluster [DBG] osdmap e657: 8 total, 8 up, 8 in 2024-02-19T21:15:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:01.381711+0000 mon.a (mon.0) 3264 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:01.620154+0000 mon.c (mon.2) 419 : audit [INF] from='client.? v1:172.21.15.163:0/1576525506' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:15:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:01.620444+0000 mon.a (mon.0) 3265 : audit [INF] from='client.50363 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:15:02.095 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:02.124 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:02.353 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:02.358 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-19T21:15:02.358 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:02.372 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:02.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:02.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:02.383+0000 7f1fd17fa700 1 -- 172.21.15.163:0/1576525506 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v658) v1 ==== 113+0+0 (unknown 2049704166 0 0) 0x7f1fcc04ba60 con 0x7f1fd41042c0 2024-02-19T21:15:02.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:02.392+0000 7f1fda524700 10 librados: watch_flush enter 2024-02-19T21:15:02.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:02.396+0000 7f1fda524700 10 librados: watch_flush exit 2024-02-19T21:15:02.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:02.396+0000 7f1fda524700 20 client.50363.objecter shutdown clearing up homeless session... 2024-02-19T21:15:02.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:02.396+0000 7f1fda524700 10 client.50363.objecter successfully canceled tick 2024-02-19T21:15:02.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:02.396+0000 7f1fda524700 1 -- 172.21.15.163:0/1576525506 >> v1:172.21.15.163:6800/1552514681 conn(0x7f1fc0063b20 legacy=0x7f1fc0065fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:02.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:02.396+0000 7f1fda524700 1 -- 172.21.15.163:0/1576525506 >> v1:172.21.15.163:6790/0 conn(0x7f1fd41042c0 legacy=0x7f1fd4115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:02.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:02.396+0000 7f1fda524700 1 -- 172.21.15.163:0/1576525506 shutdown_connections 2024-02-19T21:15:02.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:02.396+0000 7f1fda524700 1 -- 172.21.15.163:0/1576525506 >> 172.21.15.163:0/1576525506 conn(0x7f1fd40f35a0 msgr2=0x7f1fd4107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:02.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:02.398+0000 7f1fda524700 1 -- 172.21.15.163:0/1576525506 shutdown_connections 2024-02-19T21:15:02.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:02.398+0000 7f1fda524700 1 -- 172.21.15.163:0/1576525506 wait complete. 2024-02-19T21:15:02.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:02.402+0000 7f1fda524700 1 librados: shutdown 2024-02-19T21:15:02.403 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.2 to 1 (10000) 2024-02-19T21:15:02.479 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:15:02.507 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:02.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:01.297309+0000 mgr.y (mgr.14550) 563 : cluster [DBG] pgmap v988: 182 pgs: 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:02.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:02.092326+0000 osd.5 (osd.5) 266 : cluster [DBG] 16.3 scrub starts 2024-02-19T21:15:02.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:02.096032+0000 osd.5 (osd.5) 267 : cluster [DBG] 16.3 scrub ok 2024-02-19T21:15:02.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:02.381178+0000 mon.a (mon.0) 3266 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:02.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:02.381362+0000 mon.a (mon.0) 3267 : audit [INF] from='client.50363 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-19T21:15:02.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:02.381421+0000 mon.a (mon.0) 3268 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2024-02-19T21:15:02.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:01.297309+0000 mgr.y (mgr.14550) 563 : cluster [DBG] pgmap v988: 182 pgs: 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:02.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:02.092326+0000 osd.5 (osd.5) 266 : cluster [DBG] 16.3 scrub starts 2024-02-19T21:15:02.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:02.096032+0000 osd.5 (osd.5) 267 : cluster [DBG] 16.3 scrub ok 2024-02-19T21:15:02.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:02.381178+0000 mon.a (mon.0) 3266 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:02.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:02.381362+0000 mon.a (mon.0) 3267 : audit [INF] from='client.50363 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-19T21:15:02.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:02.381421+0000 mon.a (mon.0) 3268 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2024-02-19T21:15:03.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:01.297309+0000 mgr.y (mgr.14550) 563 : cluster [DBG] pgmap v988: 182 pgs: 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:03.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:02.092326+0000 osd.5 (osd.5) 266 : cluster [DBG] 16.3 scrub starts 2024-02-19T21:15:03.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:02.096032+0000 osd.5 (osd.5) 267 : cluster [DBG] 16.3 scrub ok 2024-02-19T21:15:03.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:02.381178+0000 mon.a (mon.0) 3266 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:03.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:02.381362+0000 mon.a (mon.0) 3267 : audit [INF] from='client.50363 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-19T21:15:03.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:02.381421+0000 mon.a (mon.0) 3268 : cluster [DBG] osdmap e658: 8 total, 8 up, 8 in 2024-02-19T21:15:03.679 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:15:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:15:03] "GET /metrics HTTP/1.1" 200 191748 "" "Prometheus/2.33.4" 2024-02-19T21:15:03.680 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:02.243808+0000 osd.6 (osd.6) 223 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:15:03.680 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:02.244545+0000 osd.6 (osd.6) 224 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:15:03.680 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:03.218273+0000 osd.3 (osd.3) 273 : cluster [DBG] 16.16 scrub starts 2024-02-19T21:15:03.680 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:03.219465+0000 osd.3 (osd.3) 274 : cluster [DBG] 16.16 scrub ok 2024-02-19T21:15:03.680 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:03.381849+0000 mon.a (mon.0) 3269 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2024-02-19T21:15:03.680 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:03.399592+0000 mon.a (mon.0) 3270 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:03.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:02.243808+0000 osd.6 (osd.6) 223 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:15:03.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:02.244545+0000 osd.6 (osd.6) 224 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:15:03.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:03.218273+0000 osd.3 (osd.3) 273 : cluster [DBG] 16.16 scrub starts 2024-02-19T21:15:03.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:03.219465+0000 osd.3 (osd.3) 274 : cluster [DBG] 16.16 scrub ok 2024-02-19T21:15:03.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:03.381849+0000 mon.a (mon.0) 3269 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2024-02-19T21:15:03.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:03.399592+0000 mon.a (mon.0) 3270 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:04.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:02.243808+0000 osd.6 (osd.6) 223 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:15:04.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:02.244545+0000 osd.6 (osd.6) 224 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:15:04.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:03.218273+0000 osd.3 (osd.3) 273 : cluster [DBG] 16.16 scrub starts 2024-02-19T21:15:04.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:03.219465+0000 osd.3 (osd.3) 274 : cluster [DBG] 16.16 scrub ok 2024-02-19T21:15:04.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:03.381849+0000 mon.a (mon.0) 3269 : cluster [DBG] osdmap e659: 8 total, 8 up, 8 in 2024-02-19T21:15:04.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:03.399592+0000 mon.a (mon.0) 3270 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:04.454 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 5 1 2024-02-19T21:15:04.459 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:15:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:04.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:15:04.562 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:04.589 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:04.610 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:04.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:04.715 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:04.748 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:03.297776+0000 mgr.y (mgr.14550) 564 : cluster [DBG] pgmap v991: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:04.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:03.368733+0000 osd.2 (osd.2) 266 : cluster [DBG] 16.1 scrub starts 2024-02-19T21:15:04.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:03.372245+0000 osd.2 (osd.2) 267 : cluster [DBG] 16.1 scrub ok 2024-02-19T21:15:04.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:03.570825+0000 osd.4 (osd.4) 237 : cluster [DBG] 4.e scrub starts 2024-02-19T21:15:04.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:03.572682+0000 osd.4 (osd.4) 238 : cluster [DBG] 4.e scrub ok 2024-02-19T21:15:04.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:03.895229+0000 osd.1 (osd.1) 294 : cluster [DBG] 4.b scrub starts 2024-02-19T21:15:04.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:03.898690+0000 osd.1 (osd.1) 295 : cluster [DBG] 4.b scrub ok 2024-02-19T21:15:04.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:04.116510+0000 osd.5 (osd.5) 268 : cluster [DBG] 6.a scrub starts 2024-02-19T21:15:04.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:04.119580+0000 osd.5 (osd.5) 269 : cluster [DBG] 6.a scrub ok 2024-02-19T21:15:04.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:04.369273+0000 mon.a (mon.0) 3271 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-114", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:04.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:04.369344+0000 mon.a (mon.0) 3272 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2024-02-19T21:15:04.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:04.370049+0000 mon.a (mon.0) 3273 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-114"}]: dispatch 2024-02-19T21:15:04.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:03.297776+0000 mgr.y (mgr.14550) 564 : cluster [DBG] pgmap v991: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:04.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:03.368733+0000 osd.2 (osd.2) 266 : cluster [DBG] 16.1 scrub starts 2024-02-19T21:15:04.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:03.372245+0000 osd.2 (osd.2) 267 : cluster [DBG] 16.1 scrub ok 2024-02-19T21:15:04.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:03.570825+0000 osd.4 (osd.4) 237 : cluster [DBG] 4.e scrub starts 2024-02-19T21:15:04.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:03.572682+0000 osd.4 (osd.4) 238 : cluster [DBG] 4.e scrub ok 2024-02-19T21:15:04.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:03.895229+0000 osd.1 (osd.1) 294 : cluster [DBG] 4.b scrub starts 2024-02-19T21:15:04.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:03.898690+0000 osd.1 (osd.1) 295 : cluster [DBG] 4.b scrub ok 2024-02-19T21:15:04.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:04.116510+0000 osd.5 (osd.5) 268 : cluster [DBG] 6.a scrub starts 2024-02-19T21:15:04.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:04.119580+0000 osd.5 (osd.5) 269 : cluster [DBG] 6.a scrub ok 2024-02-19T21:15:04.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:04.369273+0000 mon.a (mon.0) 3271 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-114", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:04.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:04.369344+0000 mon.a (mon.0) 3272 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2024-02-19T21:15:04.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:04.370049+0000 mon.a (mon.0) 3273 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-114"}]: dispatch 2024-02-19T21:15:04.770 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:04.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:04.873 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:04.904 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:04.925 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:04.927 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:04.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:05.028 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:15:05.057 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:05.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:03.297776+0000 mgr.y (mgr.14550) 564 : cluster [DBG] pgmap v991: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:05.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:03.368733+0000 osd.2 (osd.2) 266 : cluster [DBG] 16.1 scrub starts 2024-02-19T21:15:05.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:03.372245+0000 osd.2 (osd.2) 267 : cluster [DBG] 16.1 scrub ok 2024-02-19T21:15:05.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:03.570825+0000 osd.4 (osd.4) 237 : cluster [DBG] 4.e scrub starts 2024-02-19T21:15:05.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:03.572682+0000 osd.4 (osd.4) 238 : cluster [DBG] 4.e scrub ok 2024-02-19T21:15:05.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:03.895229+0000 osd.1 (osd.1) 294 : cluster [DBG] 4.b scrub starts 2024-02-19T21:15:05.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:03.898690+0000 osd.1 (osd.1) 295 : cluster [DBG] 4.b scrub ok 2024-02-19T21:15:05.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:04.116510+0000 osd.5 (osd.5) 268 : cluster [DBG] 6.a scrub starts 2024-02-19T21:15:05.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:04.119580+0000 osd.5 (osd.5) 269 : cluster [DBG] 6.a scrub ok 2024-02-19T21:15:05.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:04.369273+0000 mon.a (mon.0) 3271 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-114", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:05.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:04.369344+0000 mon.a (mon.0) 3272 : cluster [DBG] osdmap e660: 8 total, 8 up, 8 in 2024-02-19T21:15:05.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:04.370049+0000 mon.a (mon.0) 3273 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-114"}]: dispatch 2024-02-19T21:15:05.130 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:15:05.133 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:05.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:15:05.234 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:05.263 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:05.283 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:05.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:05.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:05.386 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:05.417 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:05.932 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:04.198862+0000 osd.6 (osd.6) 225 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:15:05.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:04.202037+0000 osd.6 (osd.6) 226 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:15:05.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:04.347614+0000 osd.2 (osd.2) 268 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:15:05.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:04.348624+0000 osd.2 (osd.2) 269 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:15:05.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:04.884409+0000 mon.a (mon.0) 3274 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:05.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:05.369507+0000 mon.a (mon.0) 3275 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-114"}]': finished 2024-02-19T21:15:05.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:05.369566+0000 mon.a (mon.0) 3276 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2024-02-19T21:15:05.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:05.373574+0000 mon.a (mon.0) 3277 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-114", "mode": "writeback"}]: dispatch 2024-02-19T21:15:05.933 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:05.429805+0000 mon.c (mon.2) 420 : audit [DBG] from='client.? v1:172.21.15.163:0/1712981126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:05.933 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:04.198862+0000 osd.6 (osd.6) 225 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:15:05.933 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:04.202037+0000 osd.6 (osd.6) 226 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:15:05.933 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:04.347614+0000 osd.2 (osd.2) 268 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:15:05.933 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:04.348624+0000 osd.2 (osd.2) 269 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:15:05.934 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:04.884409+0000 mon.a (mon.0) 3274 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:05.934 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:05.369507+0000 mon.a (mon.0) 3275 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-114"}]': finished 2024-02-19T21:15:05.934 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:05.369566+0000 mon.a (mon.0) 3276 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2024-02-19T21:15:05.934 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:05.373574+0000 mon.a (mon.0) 3277 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-114", "mode": "writeback"}]: dispatch 2024-02-19T21:15:05.934 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:05.429805+0000 mon.c (mon.2) 420 : audit [DBG] from='client.? v1:172.21.15.163:0/1712981126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:06.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.425+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1289084839 >> v1:172.21.15.163:6790/0 conn(0x7fb1981007b0 legacy=0x7fb198102ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.425+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1289084839 shutdown_connections 2024-02-19T21:15:06.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.425+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1289084839 >> 172.21.15.163:0/1289084839 conn(0x7fb19806cc00 msgr2=0x7fb1980f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:06.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.426+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1289084839 shutdown_connections 2024-02-19T21:15:06.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.426+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1289084839 wait complete. 2024-02-19T21:15:06.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.426+0000 7fb19e9fd700 1 librados: starting msgr at 2024-02-19T21:15:06.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.426+0000 7fb19e9fd700 1 librados: starting objecter 2024-02-19T21:15:06.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.426+0000 7fb19e9fd700 1 Processor -- start 2024-02-19T21:15:06.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.426+0000 7fb19e9fd700 1 -- start start 2024-02-19T21:15:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.426+0000 7fb19e9fd700 1 librados: setting wanted keys 2024-02-19T21:15:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.426+0000 7fb19e9fd700 1 librados: calling monclient init 2024-02-19T21:15:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb19e9fd700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1980fd080 con 0x7fb1981007b0 2024-02-19T21:15:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb19e9fd700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1980fd210 con 0x7fb1981042c0 2024-02-19T21:15:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb19e9fd700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1980fd400 con 0x7fb1980fd960 2024-02-19T21:15:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb197fff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fb1980fd960 0x7fb1980fc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38694/0 (socket says 172.21.15.163:38694) 2024-02-19T21:15:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb197fff700 1 -- 172.21.15.163:0/1712981126 learned_addr learned my addr 172.21.15.163:0/1712981126 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1977fe700 1 --1- 172.21.15.163:0/1712981126 >> v1:172.21.15.163:6789/0 conn(0x7fb1981007b0 0x7fb198114d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:56868/0 (socket says 172.21.15.163:56868) 2024-02-19T21:15:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 10 client.?.objecter ms_handle_connect 0x7fb1980fd960 2024-02-19T21:15:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 10 client.?.objecter ms_handle_connect 0x7fb1981007b0 2024-02-19T21:15:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364649435 0 0) 0x7fb1980fd400 con 0x7fb1980fd960 2024-02-19T21:15:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1800033f0 con 0x7fb1980fd960 2024-02-19T21:15:06.049 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 10 client.?.objecter ms_handle_connect 0x7fb1981042c0 2024-02-19T21:15:06.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:06.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884444425 0 0) 0x7fb1980fd210 con 0x7fb1981042c0 2024-02-19T21:15:06.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1980fd400 con 0x7fb1981042c0 2024-02-19T21:15:06.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762731588 0 0) 0x7fb1980fd080 con 0x7fb1981007b0 2024-02-19T21:15:06.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1980fd210 con 0x7fb1981007b0 2024-02-19T21:15:06.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3035465415 0 0) 0x7fb1800033f0 con 0x7fb1980fd960 2024-02-19T21:15:06.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1980fd080 con 0x7fb1980fd960 2024-02-19T21:15:06.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb188001940 con 0x7fb1980fd960 2024-02-19T21:15:06.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1196026350 0 0) 0x7fb1980fd400 con 0x7fb1981042c0 2024-02-19T21:15:06.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1800033f0 con 0x7fb1981042c0 2024-02-19T21:15:06.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb190001c40 con 0x7fb1981042c0 2024-02-19T21:15:06.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4137524347 0 0) 0x7fb1980fd080 con 0x7fb1980fd960 2024-02-19T21:15:06.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 >> v1:172.21.15.193:6789/0 conn(0x7fb1981042c0 legacy=0x7fb198118450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 >> v1:172.21.15.163:6789/0 conn(0x7fb1981007b0 legacy=0x7fb198114d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.050 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19811cba0 con 0x7fb1980fd960 2024-02-19T21:15:06.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb198119bd0 con 0x7fb1980fd960 2024-02-19T21:15:06.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.427+0000 7fb19e9fd700 10 client.50369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:06.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.428+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19811a040 con 0x7fb1980fd960 2024-02-19T21:15:06.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.428+0000 7fb19e9fd700 1 librados: init done 2024-02-19T21:15:06.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.428+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb188002460 con 0x7fb1980fd960 2024-02-19T21:15:06.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.428+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb188005ad0 con 0x7fb1980fd960 2024-02-19T21:15:06.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.428+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb184004f00 con 0x7fb1980fd960 2024-02-19T21:15:06.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.429+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fb188001fc0 con 0x7fb1980fd960 2024-02-19T21:15:06.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.429+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(661..661 src has 434..661) v4 ==== 8535+0+0 (unknown 96103930 0 0) 0x7fb188079000 con 0x7fb1980fd960 2024-02-19T21:15:06.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 osd_map(661..661 src has 434..661) v4 2024-02-19T21:15:06.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb1957fa700 3 client.50369.objecter handle_osd_map got epochs [661,661] > 0 2024-02-19T21:15:06.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb1957fa700 3 client.50369.objecter handle_osd_map decoding full epoch 661 2024-02-19T21:15:06.051 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb1957fa700 20 client.50369.objecter dump_active .. 0 homeless 2024-02-19T21:15:06.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v661) v1 ==== 54+0+16 (unknown 84686803 0 2420000192) 0x7fb18807b3c0 con 0x7fb1980fd960 2024-02-19T21:15:06.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb1957fa700 10 client.50369.objecter ms_handle_connect 0x7fb180063990 2024-02-19T21:15:06.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:06.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 1 2024-02-19T21:15:06.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb1840051a0 osd=0 3 2024-02-19T21:15:06.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb1840051a0 osd=0 3 2024-02-19T21:15:06.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb1840051a0 osd=0 3 2024-02-19T21:15:06.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 1 2024-02-19T21:15:06.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb1840051a0 osd=0 3 2024-02-19T21:15:06.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 0 1 2024-02-19T21:15:06.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb1840051a0 osd=0 4 2024-02-19T21:15:06.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb19e9fd700 10 client.50369.objecter _send_command 1 2024-02-19T21:15:06.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb184004950 con 0x7fb184004530 2024-02-19T21:15:06.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.430+0000 7fb1957fa700 10 client.50369.objecter ms_handle_connect 0x7fb184004530 2024-02-19T21:15:06.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.431+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb184004950 con 0x7fb184004530 2024-02-19T21:15:06.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.431+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:15:06.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.431+0000 7fb1957fa700 10 client.50369.objecter _finish_command 1 = system:0 2024-02-19T21:15:06.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.431+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb1840051a0 osd=0 4 2024-02-19T21:15:06.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.431+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 0 1 2024-02-19T21:15:06.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.448+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:06.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.448+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 2 2024-02-19T21:15:06.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.448+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb1840051a0 osd=0 3 2024-02-19T21:15:06.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.448+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb1840051a0 osd=0 3 2024-02-19T21:15:06.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.448+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb1840051a0 osd=0 3 2024-02-19T21:15:06.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.448+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 2 2024-02-19T21:15:06.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.448+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb1840051a0 osd=0 3 2024-02-19T21:15:06.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.448+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 0 2 2024-02-19T21:15:06.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.448+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb1840051a0 osd=0 4 2024-02-19T21:15:06.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.448+0000 7fb19e9fd700 10 client.50369.objecter _send_command 2 2024-02-19T21:15:06.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.448+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb184004d30 con 0x7fb184004530 2024-02-19T21:15:06.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb184004d30 con 0x7fb184004530 2024-02-19T21:15:06.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:06.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb1957fa700 10 client.50369.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb1840051a0 osd=0 4 2024-02-19T21:15:06.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 0 2 2024-02-19T21:15:06.055 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:15:06.055 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:15:06.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:06.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 3 2024-02-19T21:15:06.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184003fe0 osd=1 3 2024-02-19T21:15:06.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184003fe0 osd=1 3 2024-02-19T21:15:06.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184003fe0 osd=1 3 2024-02-19T21:15:06.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 3 2024-02-19T21:15:06.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb184003fe0 osd=1 3 2024-02-19T21:15:06.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 1 3 2024-02-19T21:15:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184003fe0 osd=1 4 2024-02-19T21:15:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb19e9fd700 10 client.50369.objecter _send_command 3 2024-02-19T21:15:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb18400c000 con 0x7fb184004150 2024-02-19T21:15:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.450+0000 7fb1957fa700 10 client.50369.objecter ms_handle_connect 0x7fb184004150 2024-02-19T21:15:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.451+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb18400c000 con 0x7fb184004150 2024-02-19T21:15:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.451+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:15:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.451+0000 7fb1957fa700 10 client.50369.objecter _finish_command 3 = system:0 2024-02-19T21:15:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.451+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb184003fe0 osd=1 4 2024-02-19T21:15:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.451+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 1 3 2024-02-19T21:15:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.465+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.465+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 4 2024-02-19T21:15:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.465+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184003fe0 osd=1 3 2024-02-19T21:15:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.465+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184003fe0 osd=1 3 2024-02-19T21:15:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.465+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184003fe0 osd=1 3 2024-02-19T21:15:06.056 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.465+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 4 2024-02-19T21:15:06.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.465+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb184003fe0 osd=1 3 2024-02-19T21:15:06.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.465+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 1 4 2024-02-19T21:15:06.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.465+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184003fe0 osd=1 4 2024-02-19T21:15:06.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.465+0000 7fb19e9fd700 10 client.50369.objecter _send_command 4 2024-02-19T21:15:06.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.465+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb184001850 con 0x7fb184004150 2024-02-19T21:15:06.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb184001850 con 0x7fb184004150 2024-02-19T21:15:06.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:06.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb1957fa700 10 client.50369.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb184003fe0 osd=1 4 2024-02-19T21:15:06.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 1 4 2024-02-19T21:15:06.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:06.057 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 5 2024-02-19T21:15:06.057 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:15:06.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184003630 osd=2 3 2024-02-19T21:15:06.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184003630 osd=2 3 2024-02-19T21:15:06.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184003630 osd=2 3 2024-02-19T21:15:06.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 5 2024-02-19T21:15:06.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb184003630 osd=2 3 2024-02-19T21:15:06.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 2 5 2024-02-19T21:15:06.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184003630 osd=2 4 2024-02-19T21:15:06.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb19e9fd700 10 client.50369.objecter _send_command 5 2024-02-19T21:15:06.058 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb184003e70 con 0x7fb18400d2f0 2024-02-19T21:15:06.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.467+0000 7fb1957fa700 10 client.50369.objecter ms_handle_connect 0x7fb18400d2f0 2024-02-19T21:15:06.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.468+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb184003e70 con 0x7fb18400d2f0 2024-02-19T21:15:06.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.468+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:15:06.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.468+0000 7fb1957fa700 10 client.50369.objecter _finish_command 5 = system:0 2024-02-19T21:15:06.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.468+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb184003630 osd=2 4 2024-02-19T21:15:06.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.468+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 2 5 2024-02-19T21:15:06.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.482+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:06.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.482+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 6 2024-02-19T21:15:06.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.482+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184003630 osd=2 3 2024-02-19T21:15:06.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.482+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184003630 osd=2 3 2024-02-19T21:15:06.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.482+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184003630 osd=2 3 2024-02-19T21:15:06.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.482+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 6 2024-02-19T21:15:06.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.482+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb184003630 osd=2 3 2024-02-19T21:15:06.059 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.482+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 2 6 2024-02-19T21:15:06.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.482+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184003630 osd=2 4 2024-02-19T21:15:06.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.482+0000 7fb19e9fd700 10 client.50369.objecter _send_command 6 2024-02-19T21:15:06.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.482+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb18400e6c0 con 0x7fb18400d2f0 2024-02-19T21:15:06.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb18400e6c0 con 0x7fb18400d2f0 2024-02-19T21:15:06.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:06.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb1957fa700 10 client.50369.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb184003630 osd=2 4 2024-02-19T21:15:06.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 2 6 2024-02-19T21:15:06.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:06.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 7 2024-02-19T21:15:06.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb1840033d0 osd=3 3 2024-02-19T21:15:06.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb1840033d0 osd=3 3 2024-02-19T21:15:06.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb1840033d0 osd=3 3 2024-02-19T21:15:06.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 7 2024-02-19T21:15:06.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb1840033d0 osd=3 3 2024-02-19T21:15:06.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 3 7 2024-02-19T21:15:06.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb1840033d0 osd=3 4 2024-02-19T21:15:06.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb19e9fd700 10 client.50369.objecter _send_command 7 2024-02-19T21:15:06.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb184015ad0 con 0x7fb184011e80 2024-02-19T21:15:06.061 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:15:06.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.484+0000 7fb1957fa700 10 client.50369.objecter ms_handle_connect 0x7fb184011e80 2024-02-19T21:15:06.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.485+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb184015ad0 con 0x7fb184011e80 2024-02-19T21:15:06.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.485+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.485+0000 7fb1957fa700 10 client.50369.objecter _finish_command 7 = system:0 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.485+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb1840033d0 osd=3 4 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.485+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 3 7 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.499+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.499+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 8 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.499+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb1840033d0 osd=3 3 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.499+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb1840033d0 osd=3 3 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.499+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb1840033d0 osd=3 3 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.499+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 8 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.499+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb1840033d0 osd=3 3 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.499+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 3 8 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.499+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb1840033d0 osd=3 4 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.499+0000 7fb19e9fd700 10 client.50369.objecter _send_command 8 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.499+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb184015ad0 con 0x7fb184011e80 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb184015ad0 con 0x7fb184011e80 2024-02-19T21:15:06.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb1957fa700 10 client.50369.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb1840033d0 osd=3 4 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 3 8 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 9 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184015ad0 osd=4 3 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184015ad0 osd=4 3 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184015ad0 osd=4 3 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 9 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb184015ad0 osd=4 3 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 4 9 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184015ad0 osd=4 4 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb19e9fd700 10 client.50369.objecter _send_command 9 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb18400cf20 con 0x7fb18400c940 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.501+0000 7fb1957fa700 10 client.50369.objecter ms_handle_connect 0x7fb18400c940 2024-02-19T21:15:06.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.502+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb18400cf20 con 0x7fb18400c940 2024-02-19T21:15:06.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.502+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:15:06.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.502+0000 7fb1957fa700 10 client.50369.objecter _finish_command 9 = system:0 2024-02-19T21:15:06.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.502+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb184015ad0 osd=4 4 2024-02-19T21:15:06.064 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:15:06.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.502+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 4 9 2024-02-19T21:15:06.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.517+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:06.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.517+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 10 2024-02-19T21:15:06.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.517+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184015ad0 osd=4 3 2024-02-19T21:15:06.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.517+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184015ad0 osd=4 3 2024-02-19T21:15:06.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.517+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184015ad0 osd=4 3 2024-02-19T21:15:06.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.517+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 10 2024-02-19T21:15:06.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.517+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb184015ad0 osd=4 3 2024-02-19T21:15:06.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.517+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 4 10 2024-02-19T21:15:06.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.517+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184015ad0 osd=4 4 2024-02-19T21:15:06.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.517+0000 7fb19e9fd700 10 client.50369.objecter _send_command 10 2024-02-19T21:15:06.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.517+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb18400d160 con 0x7fb18400c940 2024-02-19T21:15:06.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.518+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb18400d160 con 0x7fb18400c940 2024-02-19T21:15:06.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:06.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb1957fa700 10 client.50369.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb184015ad0 osd=4 4 2024-02-19T21:15:06.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 4 10 2024-02-19T21:15:06.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:06.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 11 2024-02-19T21:15:06.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb18400d160 osd=5 3 2024-02-19T21:15:06.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb18400d160 osd=5 3 2024-02-19T21:15:06.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb18400d160 osd=5 3 2024-02-19T21:15:06.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 11 2024-02-19T21:15:06.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb18400d160 osd=5 3 2024-02-19T21:15:06.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 5 11 2024-02-19T21:15:06.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb18400d160 osd=5 4 2024-02-19T21:15:06.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb19e9fd700 10 client.50369.objecter _send_command 11 2024-02-19T21:15:06.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb184017e60 con 0x7fb184017a10 2024-02-19T21:15:06.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.519+0000 7fb1957fa700 10 client.50369.objecter ms_handle_connect 0x7fb184017a10 2024-02-19T21:15:06.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.520+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb184017e60 con 0x7fb184017a10 2024-02-19T21:15:06.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.520+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:15:06.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.520+0000 7fb1957fa700 10 client.50369.objecter _finish_command 11 = system:0 2024-02-19T21:15:06.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.520+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb18400d160 osd=5 4 2024-02-19T21:15:06.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.520+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 5 11 2024-02-19T21:15:06.067 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:15:06.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.534+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:06.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.534+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 12 2024-02-19T21:15:06.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.534+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb18400d160 osd=5 3 2024-02-19T21:15:06.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.534+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb18400d160 osd=5 3 2024-02-19T21:15:06.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.534+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb18400d160 osd=5 3 2024-02-19T21:15:06.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.534+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 12 2024-02-19T21:15:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.534+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb18400d160 osd=5 3 2024-02-19T21:15:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.534+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 5 12 2024-02-19T21:15:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.534+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb18400d160 osd=5 4 2024-02-19T21:15:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.534+0000 7fb19e9fd700 10 client.50369.objecter _send_command 12 2024-02-19T21:15:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.534+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb184017e60 con 0x7fb184017a10 2024-02-19T21:15:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb184017e60 con 0x7fb184017a10 2024-02-19T21:15:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb1957fa700 10 client.50369.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb18400d160 osd=5 4 2024-02-19T21:15:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 5 12 2024-02-19T21:15:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 13 2024-02-19T21:15:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184017e60 osd=6 3 2024-02-19T21:15:06.068 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184017e60 osd=6 3 2024-02-19T21:15:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184017e60 osd=6 3 2024-02-19T21:15:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 13 2024-02-19T21:15:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb184017e60 osd=6 3 2024-02-19T21:15:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 6 13 2024-02-19T21:15:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184017e60 osd=6 4 2024-02-19T21:15:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb19e9fd700 10 client.50369.objecter _send_command 13 2024-02-19T21:15:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.536+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb184022be0 con 0x7fb18401f0d0 2024-02-19T21:15:06.069 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.537+0000 7fb1957fa700 10 client.50369.objecter ms_handle_connect 0x7fb18401f0d0 2024-02-19T21:15:06.069 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:15:06.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.538+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb184022be0 con 0x7fb18401f0d0 2024-02-19T21:15:06.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.538+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:15:06.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.538+0000 7fb1957fa700 10 client.50369.objecter _finish_command 13 = system:0 2024-02-19T21:15:06.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.538+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb184017e60 osd=6 4 2024-02-19T21:15:06.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.538+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 6 13 2024-02-19T21:15:06.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.552+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:06.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.552+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 14 2024-02-19T21:15:06.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.552+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184017e60 osd=6 3 2024-02-19T21:15:06.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.552+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184017e60 osd=6 3 2024-02-19T21:15:06.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.552+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184017e60 osd=6 3 2024-02-19T21:15:06.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.552+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 14 2024-02-19T21:15:06.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.552+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb184017e60 osd=6 3 2024-02-19T21:15:06.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.552+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 6 14 2024-02-19T21:15:06.070 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.552+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184017e60 osd=6 4 2024-02-19T21:15:06.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.552+0000 7fb19e9fd700 10 client.50369.objecter _send_command 14 2024-02-19T21:15:06.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.552+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb184022be0 con 0x7fb18401f0d0 2024-02-19T21:15:06.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.554+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb184022be0 con 0x7fb18401f0d0 2024-02-19T21:15:06.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.554+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:06.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.554+0000 7fb1957fa700 10 client.50369.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.554+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb184017e60 osd=6 4 2024-02-19T21:15:06.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.554+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 6 14 2024-02-19T21:15:06.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.554+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:06.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.555+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 15 2024-02-19T21:15:06.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.555+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184022be0 osd=7 3 2024-02-19T21:15:06.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.555+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184022be0 osd=7 3 2024-02-19T21:15:06.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.555+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184022be0 osd=7 3 2024-02-19T21:15:06.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.555+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 15 2024-02-19T21:15:06.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.555+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb184022be0 osd=7 3 2024-02-19T21:15:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.555+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 7 15 2024-02-19T21:15:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.555+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184022be0 osd=7 4 2024-02-19T21:15:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.555+0000 7fb19e9fd700 10 client.50369.objecter _send_command 15 2024-02-19T21:15:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.555+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb184026f80 con 0x7fb184023290 2024-02-19T21:15:06.072 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.555+0000 7fb1957fa700 10 client.50369.objecter ms_handle_connect 0x7fb184023290 2024-02-19T21:15:06.072 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:15:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:04.198862+0000 osd.6 (osd.6) 225 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:15:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:04.202037+0000 osd.6 (osd.6) 226 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:15:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:04.347614+0000 osd.2 (osd.2) 268 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:15:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:04.348624+0000 osd.2 (osd.2) 269 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:15:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:04.884409+0000 mon.a (mon.0) 3274 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:05.369507+0000 mon.a (mon.0) 3275 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-114"}]': finished 2024-02-19T21:15:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:05.369566+0000 mon.a (mon.0) 3276 : cluster [DBG] osdmap e661: 8 total, 8 up, 8 in 2024-02-19T21:15:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:05.373574+0000 mon.a (mon.0) 3277 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-114", "mode": "writeback"}]: dispatch 2024-02-19T21:15:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:05.429805+0000 mon.c (mon.2) 420 : audit [DBG] from='client.? v1:172.21.15.163:0/1712981126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.556+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb184026f80 con 0x7fb184023290 2024-02-19T21:15:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.556+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:15:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.556+0000 7fb1957fa700 10 client.50369.objecter _finish_command 15 = system:0 2024-02-19T21:15:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.556+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb184022be0 osd=7 4 2024-02-19T21:15:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.556+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 7 15 2024-02-19T21:15:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.570+0000 7fb19e9fd700 10 client.50369.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.570+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign -1 16 2024-02-19T21:15:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.570+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184022be0 osd=7 3 2024-02-19T21:15:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.570+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184022be0 osd=7 3 2024-02-19T21:15:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.570+0000 7fb19e9fd700 20 client.50369.objecter _get_session s=0x7fb184022be0 osd=7 3 2024-02-19T21:15:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.570+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_remove -1 16 2024-02-19T21:15:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.570+0000 7fb19e9fd700 20 client.50369.objecter get_session s=0x7fb184022be0 osd=7 3 2024-02-19T21:15:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.570+0000 7fb19e9fd700 15 client.50369.objecter _session_command_op_assign 7 16 2024-02-19T21:15:06.074 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.570+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184022be0 osd=7 4 2024-02-19T21:15:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.570+0000 7fb19e9fd700 10 client.50369.objecter _send_command 16 2024-02-19T21:15:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.570+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fb184028730 con 0x7fb184023290 2024-02-19T21:15:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb1957fa700 1 -- 172.21.15.163:0/1712981126 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fb184028730 con 0x7fb184023290 2024-02-19T21:15:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb1957fa700 10 client.50369.objecter ms_dispatch 0x7fb198001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb1957fa700 10 client.50369.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb1957fa700 20 client.50369.objecter put_session s=0x7fb184022be0 osd=7 4 2024-02-19T21:15:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb1957fa700 15 client.50369.objecter _session_command_op_remove 7 16 2024-02-19T21:15:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 10 librados: watch_flush enter 2024-02-19T21:15:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 10 librados: watch_flush exit 2024-02-19T21:15:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 10 client.50369.objecter close_session for osd.0 2024-02-19T21:15:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 >> v1:172.21.15.163:6801/1559418115 conn(0x7fb184004530 legacy=0x7fb184002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb1840051a0 osd=0 1 2024-02-19T21:15:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 10 client.50369.objecter close_session for osd.1 2024-02-19T21:15:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 >> v1:172.21.15.163:6805/3125047432 conn(0x7fb184004150 legacy=0x7fb18400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.075 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184003fe0 osd=1 1 2024-02-19T21:15:06.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 10 client.50369.objecter close_session for osd.2 2024-02-19T21:15:06.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 >> v1:172.21.15.163:6809/1563158487 conn(0x7fb18400d2f0 legacy=0x7fb18400d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184003630 osd=2 1 2024-02-19T21:15:06.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 10 client.50369.objecter close_session for osd.3 2024-02-19T21:15:06.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 >> v1:172.21.15.163:6813/2112279769 conn(0x7fb184011e80 legacy=0x7fb184014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb1840033d0 osd=3 1 2024-02-19T21:15:06.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 10 client.50369.objecter close_session for osd.4 2024-02-19T21:15:06.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 >> v1:172.21.15.193:6800/3129716965 conn(0x7fb18400c940 legacy=0x7fb184015c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184015ad0 osd=4 1 2024-02-19T21:15:06.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 10 client.50369.objecter close_session for osd.5 2024-02-19T21:15:06.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 >> v1:172.21.15.193:6804/805161991 conn(0x7fb184017a10 legacy=0x7fb18401d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19cf9a700 1 -- 172.21.15.163:0/1712981126 reap_dead start 2024-02-19T21:15:06.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19cf9a700 1 -- 172.21.15.163:0/1712981126 reap_dead start 2024-02-19T21:15:06.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb18400d160 osd=5 1 2024-02-19T21:15:06.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 10 client.50369.objecter close_session for osd.6 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 >> v1:172.21.15.193:6808/787604615 conn(0x7fb18401f0d0 legacy=0x7fb1840214c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184017e60 osd=6 1 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 10 client.50369.objecter close_session for osd.7 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 >> v1:172.21.15.193:6812/157927432 conn(0x7fb184023290 legacy=0x7fb184025700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 20 client.50369.objecter put_session s=0x7fb184022be0 osd=7 1 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 20 client.50369.objecter shutdown clearing up homeless session... 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 10 client.50369.objecter successfully canceled tick 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19cf9a700 1 -- 172.21.15.163:0/1712981126 reap_dead start 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 >> v1:172.21.15.163:6800/1552514681 conn(0x7fb180063990 legacy=0x7fb180065e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 >> v1:172.21.15.163:6790/0 conn(0x7fb1980fd960 legacy=0x7fb1980fc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 shutdown_connections 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 >> 172.21.15.163:0/1712981126 conn(0x7fb19806cc00 msgr2=0x7fb198107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 shutdown_connections 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 1 -- 172.21.15.163:0/1712981126 wait complete. 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:05.572+0000 7fb19e9fd700 1 librados: shutdown 2024-02-19T21:15:06.077 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.078 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.078 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.078 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.078 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.078 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.078 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.078 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:15:06.206 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:06.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.377+0000 7ff25cf18700 1 -- 172.21.15.163:0/1995662607 >> v1:172.21.15.193:6789/0 conn(0x7ff2580609e0 legacy=0x7ff258060e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.377+0000 7ff25cf18700 1 -- 172.21.15.163:0/1995662607 shutdown_connections 2024-02-19T21:15:06.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.377+0000 7ff25cf18700 1 -- 172.21.15.163:0/1995662607 >> 172.21.15.163:0/1995662607 conn(0x7ff2580f2470 msgr2=0x7ff2580f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:06.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.377+0000 7ff25cf18700 1 -- 172.21.15.163:0/1995662607 shutdown_connections 2024-02-19T21:15:06.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.377+0000 7ff25cf18700 1 -- 172.21.15.163:0/1995662607 wait complete. 2024-02-19T21:15:06.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.378+0000 7ff25cf18700 1 librados: starting msgr at 2024-02-19T21:15:06.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.378+0000 7ff25cf18700 1 librados: starting objecter 2024-02-19T21:15:06.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.378+0000 7ff25cf18700 1 Processor -- start 2024-02-19T21:15:06.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.379+0000 7ff25cf18700 1 -- start start 2024-02-19T21:15:06.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.379+0000 7ff25cf18700 1 librados: setting wanted keys 2024-02-19T21:15:06.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.379+0000 7ff25cf18700 1 librados: calling monclient init 2024-02-19T21:15:06.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.379+0000 7ff25cf18700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff258103250 con 0x7ff2580609e0 2024-02-19T21:15:06.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.379+0000 7ff25cf18700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff258103420 con 0x7ff258061570 2024-02-19T21:15:06.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.379+0000 7ff25cf18700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff25811fc50 con 0x7ff2580fc870 2024-02-19T21:15:06.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.379+0000 7ff25659c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ff2580609e0 0x7ff258102950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:56884/0 (socket says 172.21.15.163:56884) 2024-02-19T21:15:06.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.379+0000 7ff25659c700 1 -- 172.21.15.163:0/1762515294 learned_addr learned my addr 172.21.15.163:0/1762515294 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:06.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.379+0000 7ff2477fe700 10 client.?.objecter ms_handle_connect 0x7ff2580609e0 2024-02-19T21:15:06.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.379+0000 7ff2477fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:06.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.379+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954703105 0 0) 0x7ff258103250 con 0x7ff2580609e0 2024-02-19T21:15:06.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.379+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff240003630 con 0x7ff2580609e0 2024-02-19T21:15:06.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff2477fe700 10 client.?.objecter ms_handle_connect 0x7ff2580fc870 2024-02-19T21:15:06.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff2477fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:06.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3215072094 0 0) 0x7ff240003630 con 0x7ff2580609e0 2024-02-19T21:15:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff258103250 con 0x7ff2580609e0 2024-02-19T21:15:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031581740 0 0) 0x7ff25811fc50 con 0x7ff2580fc870 2024-02-19T21:15:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff240003630 con 0x7ff2580fc870 2024-02-19T21:15:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff24c001800 con 0x7ff2580609e0 2024-02-19T21:15:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3530937881 0 0) 0x7ff240003630 con 0x7ff2580fc870 2024-02-19T21:15:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff25811fc50 con 0x7ff2580fc870 2024-02-19T21:15:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff2500019a0 con 0x7ff2580fc870 2024-02-19T21:15:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3000612477 0 0) 0x7ff258103250 con 0x7ff2580609e0 2024-02-19T21:15:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 >> v1:172.21.15.163:6790/0 conn(0x7ff2580fc870 legacy=0x7ff25811c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 >> v1:172.21.15.193:6789/0 conn(0x7ff258061570 legacy=0x7ff258106770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff258120df0 con 0x7ff2580609e0 2024-02-19T21:15:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff25cf18700 1 -- 172.21.15.163:0/1762515294 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25811fe40 con 0x7ff2580609e0 2024-02-19T21:15:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.380+0000 7ff25cf18700 10 client.23655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.381+0000 7ff25cf18700 1 -- 172.21.15.163:0/1762515294 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2581202b0 con 0x7ff2580609e0 2024-02-19T21:15:06.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.381+0000 7ff25cf18700 1 librados: init done 2024-02-19T21:15:06.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.381+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff24c001a20 con 0x7ff2580609e0 2024-02-19T21:15:06.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.381+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff24c005ad0 con 0x7ff2580609e0 2024-02-19T21:15:06.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.382+0000 7ff25cf18700 1 -- 172.21.15.163:0/1762515294 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff238004f20 con 0x7ff2580609e0 2024-02-19T21:15:06.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.382+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ff24c0047d0 con 0x7ff2580609e0 2024-02-19T21:15:06.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.383+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(662..662 src has 434..662) v4 ==== 8535+0+0 (unknown 981983406 0 0) 0x7ff24c078f40 con 0x7ff2580609e0 2024-02-19T21:15:06.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.383+0000 7ff2477fe700 10 client.23655.objecter ms_dispatch 0x7ff258001550 osd_map(662..662 src has 434..662) v4 2024-02-19T21:15:06.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.383+0000 7ff2477fe700 3 client.23655.objecter handle_osd_map got epochs [662,662] > 0 2024-02-19T21:15:06.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.383+0000 7ff2477fe700 3 client.23655.objecter handle_osd_map decoding full epoch 662 2024-02-19T21:15:06.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.383+0000 7ff2477fe700 20 client.23655.objecter dump_active .. 0 homeless 2024-02-19T21:15:06.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.383+0000 7ff2477fe700 10 client.23655.objecter ms_handle_connect 0x7ff240063c00 2024-02-19T21:15:06.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.387+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff24c04c160 con 0x7ff2580609e0 2024-02-19T21:15:06.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:06.564+0000 7ff25cf18700 1 -- 172.21.15.163:0/1762515294 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7ff2380051d0 con 0x7ff2580609e0 2024-02-19T21:15:06.969 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:06.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:06.976 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:07.075 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:07.107 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:07.129 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:07.131 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:07.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:07.196 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:05.298825+0000 mgr.y (mgr.14550) 565 : cluster [DBG] pgmap v994: 214 pgs: 214 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:15:07.196 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:05.887557+0000 osd.1 (osd.1) 296 : cluster [DBG] 2.d scrub starts 2024-02-19T21:15:07.196 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:05.888406+0000 osd.1 (osd.1) 297 : cluster [DBG] 2.d scrub ok 2024-02-19T21:15:07.196 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:06.366886+0000 mon.a (mon.0) 3278 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:07.196 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:06.370079+0000 mon.a (mon.0) 3279 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-114", "mode": "writeback"}]': finished 2024-02-19T21:15:07.196 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:06.370121+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2024-02-19T21:15:07.197 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:06.565408+0000 mon.a (mon.0) 3281 : audit [INF] from='client.? v1:172.21.15.163:0/1762515294' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:15:07.197 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:05.298825+0000 mgr.y (mgr.14550) 565 : cluster [DBG] pgmap v994: 214 pgs: 214 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:15:07.197 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:05.887557+0000 osd.1 (osd.1) 296 : cluster [DBG] 2.d scrub starts 2024-02-19T21:15:07.197 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:05.888406+0000 osd.1 (osd.1) 297 : cluster [DBG] 2.d scrub ok 2024-02-19T21:15:07.197 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:06.366886+0000 mon.a (mon.0) 3278 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:07.197 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:06.370079+0000 mon.a (mon.0) 3279 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-114", "mode": "writeback"}]': finished 2024-02-19T21:15:07.197 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:06.370121+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2024-02-19T21:15:07.197 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:06.565408+0000 mon.a (mon.0) 3281 : audit [INF] from='client.? v1:172.21.15.163:0/1762515294' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:15:07.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:07.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:05.298825+0000 mgr.y (mgr.14550) 565 : cluster [DBG] pgmap v994: 214 pgs: 214 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:15:07.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:05.887557+0000 osd.1 (osd.1) 296 : cluster [DBG] 2.d scrub starts 2024-02-19T21:15:07.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:05.888406+0000 osd.1 (osd.1) 297 : cluster [DBG] 2.d scrub ok 2024-02-19T21:15:07.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:06.366886+0000 mon.a (mon.0) 3278 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:07.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:06.370079+0000 mon.a (mon.0) 3279 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-114", "mode": "writeback"}]': finished 2024-02-19T21:15:07.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:06.370121+0000 mon.a (mon.0) 3280 : cluster [DBG] osdmap e662: 8 total, 8 up, 8 in 2024-02-19T21:15:07.260 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:06.565408+0000 mon.a (mon.0) 3281 : audit [INF] from='client.? v1:172.21.15.163:0/1762515294' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:15:07.262 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:07.282 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:07.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:07.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:07.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:07.378+0000 7ff2477fe700 1 -- 172.21.15.163:0/1762515294 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v663) v1 ==== 87+0+0 (unknown 2806988017 0 0) 0x7ff24c04a570 con 0x7ff2580609e0 2024-02-19T21:15:07.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:07.382+0000 7ff25cf18700 10 librados: watch_flush enter 2024-02-19T21:15:07.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:07.383+0000 7ff25cf18700 10 librados: watch_flush exit 2024-02-19T21:15:07.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:07.383+0000 7ff25cf18700 20 client.23655.objecter shutdown clearing up homeless session... 2024-02-19T21:15:07.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:07.383+0000 7ff25cf18700 10 client.23655.objecter successfully canceled tick 2024-02-19T21:15:07.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:07.383+0000 7ff25cf18700 1 -- 172.21.15.163:0/1762515294 >> v1:172.21.15.163:6800/1552514681 conn(0x7ff240063c00 legacy=0x7ff2400660c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:07.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:07.383+0000 7ff25cf18700 1 -- 172.21.15.163:0/1762515294 >> v1:172.21.15.163:6789/0 conn(0x7ff2580609e0 legacy=0x7ff258102950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:07.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:07.383+0000 7ff25cf18700 1 -- 172.21.15.163:0/1762515294 shutdown_connections 2024-02-19T21:15:07.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:07.383+0000 7ff25cf18700 1 -- 172.21.15.163:0/1762515294 >> 172.21.15.163:0/1762515294 conn(0x7ff2580f2470 msgr2=0x7ff2580f3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:07.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:07.383+0000 7ff25cf18700 1 -- 172.21.15.163:0/1762515294 shutdown_connections 2024-02-19T21:15:07.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:07.383+0000 7ff25cf18700 1 -- 172.21.15.163:0/1762515294 wait complete. 2024-02-19T21:15:07.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:07.383+0000 7ff25cf18700 1 librados: shutdown 2024-02-19T21:15:07.384 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:15:07.385 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:15:07.417 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:08.066 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:15:08.068 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:06.572578+0000 osd.4 (osd.4) 239 : cluster [DBG] 4.d deep-scrub starts 2024-02-19T21:15:08.068 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:06.578553+0000 osd.4 (osd.4) 240 : cluster [DBG] 4.d deep-scrub ok 2024-02-19T21:15:08.068 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:07.041651+0000 osd.5 (osd.5) 270 : cluster [DBG] 2.2 scrub starts 2024-02-19T21:15:08.068 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:07.044733+0000 osd.5 (osd.5) 271 : cluster [DBG] 2.2 scrub ok 2024-02-19T21:15:08.068 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:07.217138+0000 osd.6 (osd.6) 227 : cluster [DBG] 2.11 deep-scrub starts 2024-02-19T21:15:08.068 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:07.220018+0000 osd.6 (osd.6) 228 : cluster [DBG] 2.11 deep-scrub ok 2024-02-19T21:15:08.068 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:07.368368+0000 mon.a (mon.0) 3282 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:15:08.069 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:07.377084+0000 mon.a (mon.0) 3283 : audit [INF] from='client.? v1:172.21.15.163:0/1762515294' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:15:08.069 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:07.377139+0000 mon.a (mon.0) 3284 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2024-02-19T21:15:08.069 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:07.404943+0000 mon.a (mon.0) 3285 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:08.069 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:06.572578+0000 osd.4 (osd.4) 239 : cluster [DBG] 4.d deep-scrub starts 2024-02-19T21:15:08.069 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:06.578553+0000 osd.4 (osd.4) 240 : cluster [DBG] 4.d deep-scrub ok 2024-02-19T21:15:08.069 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:07.041651+0000 osd.5 (osd.5) 270 : cluster [DBG] 2.2 scrub starts 2024-02-19T21:15:08.069 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:07.044733+0000 osd.5 (osd.5) 271 : cluster [DBG] 2.2 scrub ok 2024-02-19T21:15:08.069 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:07.217138+0000 osd.6 (osd.6) 227 : cluster [DBG] 2.11 deep-scrub starts 2024-02-19T21:15:08.069 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:07.220018+0000 osd.6 (osd.6) 228 : cluster [DBG] 2.11 deep-scrub ok 2024-02-19T21:15:08.069 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:07.368368+0000 mon.a (mon.0) 3282 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:15:08.070 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:07.377084+0000 mon.a (mon.0) 3283 : audit [INF] from='client.? v1:172.21.15.163:0/1762515294' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:15:08.070 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:07.377139+0000 mon.a (mon.0) 3284 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2024-02-19T21:15:08.070 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:07.404943+0000 mon.a (mon.0) 3285 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:08.070 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:08.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:15:08.071 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:15:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:15:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:15:08.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:06.572578+0000 osd.4 (osd.4) 239 : cluster [DBG] 4.d deep-scrub starts 2024-02-19T21:15:08.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:06.578553+0000 osd.4 (osd.4) 240 : cluster [DBG] 4.d deep-scrub ok 2024-02-19T21:15:08.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:07.041651+0000 osd.5 (osd.5) 270 : cluster [DBG] 2.2 scrub starts 2024-02-19T21:15:08.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:07.044733+0000 osd.5 (osd.5) 271 : cluster [DBG] 2.2 scrub ok 2024-02-19T21:15:08.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:07.217138+0000 osd.6 (osd.6) 227 : cluster [DBG] 2.11 deep-scrub starts 2024-02-19T21:15:08.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:07.220018+0000 osd.6 (osd.6) 228 : cluster [DBG] 2.11 deep-scrub ok 2024-02-19T21:15:08.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:07.368368+0000 mon.a (mon.0) 3282 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:15:08.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:07.377084+0000 mon.a (mon.0) 3283 : audit [INF] from='client.? v1:172.21.15.163:0/1762515294' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:15:08.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:07.377139+0000 mon.a (mon.0) 3284 : cluster [DBG] osdmap e663: 8 total, 8 up, 8 in 2024-02-19T21:15:08.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:07.404943+0000 mon.a (mon.0) 3285 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:08.172 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:08.200 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:08.905 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:15:09.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:07.299232+0000 mgr.y (mgr.14550) 566 : cluster [DBG] pgmap v997: 214 pgs: 214 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:15:09.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:08.376275+0000 mon.a (mon.0) 3286 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:09.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:08.376334+0000 mon.a (mon.0) 3287 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2024-02-19T21:15:09.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:08.378698+0000 mon.a (mon.0) 3288 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-114"}]: dispatch 2024-02-19T21:15:09.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:07.299232+0000 mgr.y (mgr.14550) 566 : cluster [DBG] pgmap v997: 214 pgs: 214 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:15:09.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:08.376275+0000 mon.a (mon.0) 3286 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:09.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:08.376334+0000 mon.a (mon.0) 3287 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2024-02-19T21:15:09.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:08.378698+0000 mon.a (mon.0) 3288 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-114"}]: dispatch 2024-02-19T21:15:09.321 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:07.299232+0000 mgr.y (mgr.14550) 566 : cluster [DBG] pgmap v997: 214 pgs: 214 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-02-19T21:15:09.321 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:08.376275+0000 mon.a (mon.0) 3286 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:09.321 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:08.376334+0000 mon.a (mon.0) 3287 : cluster [DBG] osdmap e664: 8 total, 8 up, 8 in 2024-02-19T21:15:09.321 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:08.378698+0000 mon.a (mon.0) 3288 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-114"}]: dispatch 2024-02-19T21:15:09.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.855+0000 7f91c7e9b700 1 -- 172.21.15.163:0/1819413420 >> v1:172.21.15.163:6789/0 conn(0x7f91c0100780 legacy=0x7f91c0102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:09.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.855+0000 7f91c7e9b700 1 -- 172.21.15.163:0/1819413420 shutdown_connections 2024-02-19T21:15:09.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.855+0000 7f91c7e9b700 1 -- 172.21.15.163:0/1819413420 >> 172.21.15.163:0/1819413420 conn(0x7f91c00f35a0 msgr2=0x7f91c00f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:09.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.855+0000 7f91c7e9b700 1 -- 172.21.15.163:0/1819413420 shutdown_connections 2024-02-19T21:15:09.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.855+0000 7f91c7e9b700 1 -- 172.21.15.163:0/1819413420 wait complete. 2024-02-19T21:15:09.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91c7e9b700 1 librados: starting msgr at 2024-02-19T21:15:09.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91c7e9b700 1 librados: starting objecter 2024-02-19T21:15:09.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91c7e9b700 1 Processor -- start 2024-02-19T21:15:09.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91c7e9b700 1 -- start start 2024-02-19T21:15:09.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91c7e9b700 1 librados: setting wanted keys 2024-02-19T21:15:09.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91c7e9b700 1 librados: calling monclient init 2024-02-19T21:15:09.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91c7e9b700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f91c0117160 con 0x7f91c0100780 2024-02-19T21:15:09.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91c7e9b700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f91c0118340 con 0x7f91c01042c0 2024-02-19T21:15:09.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91c7e9b700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f91c0119540 con 0x7f91c00fd980 2024-02-19T21:15:09.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91c5c37700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f91c00fd980 0x7f91c00f9730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38738/0 (socket says 172.21.15.163:38738) 2024-02-19T21:15:09.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91c5c37700 1 -- 172.21.15.163:0/4263391616 learned_addr learned my addr 172.21.15.163:0/4263391616 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:09.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91b6ffd700 10 client.?.objecter ms_handle_connect 0x7f91c00fd980 2024-02-19T21:15:09.859 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91b6ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:09.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91b6ffd700 10 client.?.objecter ms_handle_connect 0x7f91c0100780 2024-02-19T21:15:09.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91b6ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:09.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91b6ffd700 10 client.?.objecter ms_handle_connect 0x7f91c01042c0 2024-02-19T21:15:09.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91b6ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:09.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 196401309 0 0) 0x7f91c0119540 con 0x7f91c00fd980 2024-02-19T21:15:09.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91ac0038e0 con 0x7f91c00fd980 2024-02-19T21:15:09.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565168616 0 0) 0x7f91c0117160 con 0x7f91c0100780 2024-02-19T21:15:09.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c0119540 con 0x7f91c0100780 2024-02-19T21:15:09.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068081505 0 0) 0x7f91c0118340 con 0x7f91c01042c0 2024-02-19T21:15:09.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.856+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c0117160 con 0x7f91c01042c0 2024-02-19T21:15:09.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2732984345 0 0) 0x7f91ac0038e0 con 0x7f91c00fd980 2024-02-19T21:15:09.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91c0118340 con 0x7f91c00fd980 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f91b00019f0 con 0x7f91c00fd980 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3125354907 0 0) 0x7f91c0119540 con 0x7f91c0100780 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91ac0038e0 con 0x7f91c0100780 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f91bc002e10 con 0x7f91c0100780 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 328637115 0 0) 0x7f91c0117160 con 0x7f91c01042c0 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91c0119540 con 0x7f91c01042c0 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f91b8001bc0 con 0x7f91c01042c0 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4016739638 0 0) 0x7f91c0118340 con 0x7f91c00fd980 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 >> v1:172.21.15.193:6789/0 conn(0x7f91c01042c0 legacy=0x7f91c0115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 >> v1:172.21.15.163:6789/0 conn(0x7f91c0100780 legacy=0x7f91c00f9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c011a740 con 0x7f91c00fd980 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91c7e9b700 1 -- 172.21.15.163:0/4263391616 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c0117350 con 0x7f91c00fd980 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91c7e9b700 10 client.50378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91c7e9b700 1 -- 172.21.15.163:0/4263391616 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c0117840 con 0x7f91c00fd980 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91c7e9b700 1 librados: init done 2024-02-19T21:15:09.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f91b00024d0 con 0x7f91c00fd980 2024-02-19T21:15:09.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.857+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f91b0005b80 con 0x7f91c00fd980 2024-02-19T21:15:09.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.858+0000 7f91c7e9b700 1 -- 172.21.15.163:0/4263391616 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a4004f20 con 0x7f91c00fd980 2024-02-19T21:15:09.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.858+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f91b0002030 con 0x7f91c00fd980 2024-02-19T21:15:09.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.859+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(665..665 src has 434..665) v4 ==== 8583+0+0 (unknown 2991262487 0 0) 0x7f91b0078fc0 con 0x7f91c00fd980 2024-02-19T21:15:09.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.859+0000 7f91b6ffd700 10 client.50378.objecter ms_dispatch 0x7f91c0001550 osd_map(665..665 src has 434..665) v4 2024-02-19T21:15:09.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.859+0000 7f91b6ffd700 3 client.50378.objecter handle_osd_map got epochs [665,665] > 0 2024-02-19T21:15:09.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.859+0000 7f91b6ffd700 3 client.50378.objecter handle_osd_map decoding full epoch 665 2024-02-19T21:15:09.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.859+0000 7f91b6ffd700 20 client.50378.objecter dump_active .. 0 homeless 2024-02-19T21:15:09.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.859+0000 7f91b6ffd700 10 client.50378.objecter ms_handle_connect 0x7f91ac063cc0 2024-02-19T21:15:09.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:09.863+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f91b004c1d0 con 0x7f91c00fd980 2024-02-19T21:15:10.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:10.043+0000 7f91c7e9b700 1 -- 172.21.15.163:0/4263391616 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f91a40051d0 con 0x7f91c00fd980 2024-02-19T21:15:10.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:09.108546+0000 mon.a (mon.0) 3289 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:10.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:09.299808+0000 mgr.y (mgr.14550) 567 : cluster [DBG] pgmap v1000: 214 pgs: 214 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:15:10.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:09.375010+0000 mon.a (mon.0) 3290 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:10.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:09.380518+0000 mon.a (mon.0) 3291 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-114"}]': finished 2024-02-19T21:15:10.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:09.380588+0000 mon.a (mon.0) 3292 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2024-02-19T21:15:10.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:10.043681+0000 mon.c (mon.2) 421 : audit [INF] from='client.? v1:172.21.15.163:0/4263391616' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:15:10.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:10.043930+0000 mon.a (mon.0) 3293 : audit [INF] from='client.50378 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:15:10.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:09.108546+0000 mon.a (mon.0) 3289 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:10.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:09.299808+0000 mgr.y (mgr.14550) 567 : cluster [DBG] pgmap v1000: 214 pgs: 214 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:15:10.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:09.375010+0000 mon.a (mon.0) 3290 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:10.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:09.380518+0000 mon.a (mon.0) 3291 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-114"}]': finished 2024-02-19T21:15:10.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:09.380588+0000 mon.a (mon.0) 3292 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2024-02-19T21:15:10.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:10.043681+0000 mon.c (mon.2) 421 : audit [INF] from='client.? v1:172.21.15.163:0/4263391616' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:15:10.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:10.043930+0000 mon.a (mon.0) 3293 : audit [INF] from='client.50378 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:15:10.350 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:10.352 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:10.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:10.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:10.381+0000 7f91b6ffd700 1 -- 172.21.15.163:0/4263391616 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v666) v1 ==== 113+0+0 (unknown 3471085041 0 0) 0x7f91b004a5e0 con 0x7f91c00fd980 2024-02-19T21:15:10.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:10.385+0000 7f91c7e9b700 10 librados: watch_flush enter 2024-02-19T21:15:10.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:10.385+0000 7f91c7e9b700 10 librados: watch_flush exit 2024-02-19T21:15:10.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:10.385+0000 7f91c7e9b700 20 client.50378.objecter shutdown clearing up homeless session... 2024-02-19T21:15:10.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:10.385+0000 7f91c7e9b700 10 client.50378.objecter successfully canceled tick 2024-02-19T21:15:10.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:10.385+0000 7f91c7e9b700 1 -- 172.21.15.163:0/4263391616 >> v1:172.21.15.163:6800/1552514681 conn(0x7f91ac063cc0 legacy=0x7f91ac066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:10.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:10.385+0000 7f91c7e9b700 1 -- 172.21.15.163:0/4263391616 >> v1:172.21.15.163:6790/0 conn(0x7f91c00fd980 legacy=0x7f91c00f9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:10.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:10.385+0000 7f91c7e9b700 1 -- 172.21.15.163:0/4263391616 shutdown_connections 2024-02-19T21:15:10.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:10.385+0000 7f91c7e9b700 1 -- 172.21.15.163:0/4263391616 >> 172.21.15.163:0/4263391616 conn(0x7f91c00f35a0 msgr2=0x7f91c0107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:10.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:10.388+0000 7f91c7e9b700 1 -- 172.21.15.163:0/4263391616 shutdown_connections 2024-02-19T21:15:10.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:10.388+0000 7f91c7e9b700 1 -- 172.21.15.163:0/4263391616 wait complete. 2024-02-19T21:15:10.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:10.388+0000 7f91c7e9b700 1 librados: shutdown 2024-02-19T21:15:10.388 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.5 to 1 (10000) 2024-02-19T21:15:10.453 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:10.484 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:10.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:09.108546+0000 mon.a (mon.0) 3289 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:10.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:09.299808+0000 mgr.y (mgr.14550) 567 : cluster [DBG] pgmap v1000: 214 pgs: 214 active+clean; 5.8 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:15:10.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:09.375010+0000 mon.a (mon.0) 3290 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:10.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:09.380518+0000 mon.a (mon.0) 3291 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-114"}]': finished 2024-02-19T21:15:10.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:09.380588+0000 mon.a (mon.0) 3292 : cluster [DBG] osdmap e665: 8 total, 8 up, 8 in 2024-02-19T21:15:10.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:10.043681+0000 mon.c (mon.2) 421 : audit [INF] from='client.? v1:172.21.15.163:0/4263391616' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:15:10.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:10.043930+0000 mon.a (mon.0) 3293 : audit [INF] from='client.50378 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:15:10.733 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:10.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:10.837 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:10.869 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:10.890 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:10.892 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:10.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:10.994 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:11.021 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:11.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:15:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:15:11.474 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:15:11.474 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:15:11.671 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:11.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:10.381535+0000 mon.a (mon.0) 3294 : audit [INF] from='client.50378 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-19T21:15:11.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:10.381626+0000 mon.a (mon.0) 3295 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2024-02-19T21:15:11.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:10.670949+0000 mgr.y (mgr.14550) 568 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:11.673 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:10.381535+0000 mon.a (mon.0) 3294 : audit [INF] from='client.50378 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-19T21:15:11.673 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:10.381626+0000 mon.a (mon.0) 3295 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2024-02-19T21:15:11.673 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:10.670949+0000 mgr.y (mgr.14550) 568 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:11.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:11.775 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:11.803 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:11.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:10.381535+0000 mon.a (mon.0) 3294 : audit [INF] from='client.50378 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-19T21:15:11.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:10.381626+0000 mon.a (mon.0) 3295 : cluster [DBG] osdmap e666: 8 total, 8 up, 8 in 2024-02-19T21:15:11.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:10.670949+0000 mgr.y (mgr.14550) 568 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:12.194 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-19T21:15:12.332 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:15:12.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:11.300200+0000 mgr.y (mgr.14550) 569 : cluster [DBG] pgmap v1003: 182 pgs: 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:12.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:11.389456+0000 mon.a (mon.0) 3296 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2024-02-19T21:15:12.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:11.393338+0000 mon.a (mon.0) 3297 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:12.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:11.300200+0000 mgr.y (mgr.14550) 569 : cluster [DBG] pgmap v1003: 182 pgs: 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:12.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:11.389456+0000 mon.a (mon.0) 3296 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2024-02-19T21:15:12.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:11.393338+0000 mon.a (mon.0) 3297 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:12.548 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:12.553 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:12.558 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:12.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:12.582 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 7 1 2024-02-19T21:15:12.659 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:12.689 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:11.300200+0000 mgr.y (mgr.14550) 569 : cluster [DBG] pgmap v1003: 182 pgs: 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:12.689 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:11.389456+0000 mon.a (mon.0) 3296 : cluster [DBG] osdmap e667: 8 total, 8 up, 8 in 2024-02-19T21:15:12.689 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:11.393338+0000 mon.a (mon.0) 3297 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:12.691 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:12.712 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:12.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:12.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:12.816 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:12.848 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:12.870 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:12.872 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:12.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:12.974 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:15:13.000 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:13.018 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:15:13.021 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:13.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:15:13.122 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:15:13.148 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:13.346 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:15:13.350 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:13.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:15:13.353 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:13.394 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:13.399 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:13.409 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:12.392278+0000 mon.a (mon.0) 3298 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:13.409 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:12.392409+0000 mon.a (mon.0) 3299 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2024-02-19T21:15:13.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:12.397325+0000 mon.a (mon.0) 3300 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:13.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:13.394198+0000 mon.a (mon.0) 3301 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-116", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:13.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:13.394285+0000 mon.a (mon.0) 3302 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2024-02-19T21:15:13.410 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:13.394656+0000 mon.a (mon.0) 3303 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-116"}]: dispatch 2024-02-19T21:15:13.451 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:13.478 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:13.601 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:13.604 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:13.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:13.705 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:13.730 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:15:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:15:13] "GET /metrics HTTP/1.1" 200 191881 "" "Prometheus/2.33.4" 2024-02-19T21:15:13.730 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:12.392278+0000 mon.a (mon.0) 3298 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:13.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:12.392409+0000 mon.a (mon.0) 3299 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2024-02-19T21:15:13.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:12.397325+0000 mon.a (mon.0) 3300 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:13.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:13.394198+0000 mon.a (mon.0) 3301 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-116", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:13.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:13.394285+0000 mon.a (mon.0) 3302 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2024-02-19T21:15:13.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:13.394656+0000 mon.a (mon.0) 3303 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-116"}]: dispatch 2024-02-19T21:15:13.732 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:13.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:12.392278+0000 mon.a (mon.0) 3298 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:13.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:12.392409+0000 mon.a (mon.0) 3299 : cluster [DBG] osdmap e668: 8 total, 8 up, 8 in 2024-02-19T21:15:13.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:12.397325+0000 mon.a (mon.0) 3300 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:13.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:13.394198+0000 mon.a (mon.0) 3301 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-116", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:13.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:13.394285+0000 mon.a (mon.0) 3302 : cluster [DBG] osdmap e669: 8 total, 8 up, 8 in 2024-02-19T21:15:13.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:13.394656+0000 mon.a (mon.0) 3303 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-116"}]: dispatch 2024-02-19T21:15:13.930 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:13.933 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:13.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:14.034 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:15:14.059 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:14.609 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:15:14.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:14.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:15:14.713 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:13.300682+0000 mgr.y (mgr.14550) 570 : cluster [DBG] pgmap v1006: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:14.713 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:14.371517+0000 mon.a (mon.0) 3304 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-116"}]': finished 2024-02-19T21:15:14.713 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:14.371575+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2024-02-19T21:15:14.713 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:14.373402+0000 mon.a (mon.0) 3306 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-116", "mode": "writeback"}]: dispatch 2024-02-19T21:15:14.714 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:13.300682+0000 mgr.y (mgr.14550) 570 : cluster [DBG] pgmap v1006: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:14.714 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:14.371517+0000 mon.a (mon.0) 3304 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-116"}]': finished 2024-02-19T21:15:14.714 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:14.371575+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2024-02-19T21:15:14.714 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:14.373402+0000 mon.a (mon.0) 3306 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-116", "mode": "writeback"}]: dispatch 2024-02-19T21:15:14.714 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:15:14.742 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:13.300682+0000 mgr.y (mgr.14550) 570 : cluster [DBG] pgmap v1006: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:14.742 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:14.371517+0000 mon.a (mon.0) 3304 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-116"}]': finished 2024-02-19T21:15:14.742 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:14.371575+0000 mon.a (mon.0) 3305 : cluster [DBG] osdmap e670: 8 total, 8 up, 8 in 2024-02-19T21:15:14.742 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:14.373402+0000 mon.a (mon.0) 3306 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-116", "mode": "writeback"}]: dispatch 2024-02-19T21:15:14.745 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:14.765 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:15:14.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:14.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:15:14.869 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:14.896 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:15.745 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:15.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:15.368792+0000 mon.a (mon.0) 3307 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:15.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:15.375560+0000 mon.a (mon.0) 3308 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-116", "mode": "writeback"}]': finished 2024-02-19T21:15:15.747 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:15.375621+0000 mon.a (mon.0) 3309 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2024-02-19T21:15:15.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:15.368792+0000 mon.a (mon.0) 3307 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:15.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:15.375560+0000 mon.a (mon.0) 3308 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-116", "mode": "writeback"}]': finished 2024-02-19T21:15:15.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:15.375621+0000 mon.a (mon.0) 3309 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2024-02-19T21:15:15.748 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:15.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:15.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:15.368792+0000 mon.a (mon.0) 3307 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:15.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:15.375560+0000 mon.a (mon.0) 3308 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-116", "mode": "writeback"}]': finished 2024-02-19T21:15:15.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:15.375621+0000 mon.a (mon.0) 3309 : cluster [DBG] osdmap e671: 8 total, 8 up, 8 in 2024-02-19T21:15:15.850 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:15.882 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:15.903 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:15.906 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:15.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:16.007 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:15:16.037 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:16.056 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:15:16.059 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:16.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:15:16.160 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:15:16.191 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:16.212 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:15:16.214 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:16.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:15:16.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.260+0000 7f4bd43cd700 1 -- 172.21.15.163:0/852784773 >> v1:172.21.15.163:6789/0 conn(0x7f4bcc0ff670 legacy=0x7f4bcc101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:16.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.260+0000 7f4bd43cd700 1 -- 172.21.15.163:0/852784773 shutdown_connections 2024-02-19T21:15:16.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.260+0000 7f4bd43cd700 1 -- 172.21.15.163:0/852784773 >> 172.21.15.163:0/852784773 conn(0x7f4bcc0f2470 msgr2=0x7f4bcc0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:16.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.260+0000 7f4bd43cd700 1 -- 172.21.15.163:0/852784773 shutdown_connections 2024-02-19T21:15:16.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.260+0000 7f4bd43cd700 1 -- 172.21.15.163:0/852784773 wait complete. 2024-02-19T21:15:16.261 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.260+0000 7f4bd43cd700 1 librados: starting msgr at 2024-02-19T21:15:16.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.260+0000 7f4bd43cd700 1 librados: starting objecter 2024-02-19T21:15:16.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.260+0000 7f4bd43cd700 1 Processor -- start 2024-02-19T21:15:16.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.260+0000 7f4bd43cd700 1 -- start start 2024-02-19T21:15:16.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.260+0000 7f4bd43cd700 1 librados: setting wanted keys 2024-02-19T21:15:16.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.260+0000 7f4bd43cd700 1 librados: calling monclient init 2024-02-19T21:15:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bd43cd700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4bcc119510 con 0x7f4bcc1031b0 2024-02-19T21:15:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bd43cd700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4bcc11a710 con 0x7f4bcc0ff670 2024-02-19T21:15:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bd43cd700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4bcc11b910 con 0x7f4bcc0fc870 2024-02-19T21:15:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bd2169700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f4bcc0fc870 0x7f4bcc0609e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:50856/0 (socket says 172.21.15.163:50856) 2024-02-19T21:15:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bd2169700 1 -- 172.21.15.163:0/736187646 learned_addr learned my addr 172.21.15.163:0/736187646 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 10 client.?.objecter ms_handle_connect 0x7f4bcc1031b0 2024-02-19T21:15:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 10 client.?.objecter ms_handle_connect 0x7f4bcc0fc870 2024-02-19T21:15:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 10 client.?.objecter ms_handle_connect 0x7f4bcc0ff670 2024-02-19T21:15:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880919376 0 0) 0x7f4bcc11b910 con 0x7f4bcc0fc870 2024-02-19T21:15:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb8003030 con 0x7f4bcc0fc870 2024-02-19T21:15:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586995898 0 0) 0x7f4bcc119510 con 0x7f4bcc1031b0 2024-02-19T21:15:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bcc11b910 con 0x7f4bcc1031b0 2024-02-19T21:15:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336375501 0 0) 0x7f4bcc11a710 con 0x7f4bcc0ff670 2024-02-19T21:15:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bcc119510 con 0x7f4bcc0ff670 2024-02-19T21:15:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2108479506 0 0) 0x7f4bb8003030 con 0x7f4bcc0fc870 2024-02-19T21:15:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bcc11a710 con 0x7f4bcc0fc870 2024-02-19T21:15:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3060758683 0 0) 0x7f4bcc11b910 con 0x7f4bcc1031b0 2024-02-19T21:15:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bb8003030 con 0x7f4bcc1031b0 2024-02-19T21:15:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4bc80019f0 con 0x7f4bcc0fc870 2024-02-19T21:15:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1350523849 0 0) 0x7f4bcc119510 con 0x7f4bcc0ff670 2024-02-19T21:15:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bcc11b910 con 0x7f4bcc0ff670 2024-02-19T21:15:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4bc4001b00 con 0x7f4bcc1031b0 2024-02-19T21:15:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.261+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4bbc001070 con 0x7f4bcc0ff670 2024-02-19T21:15:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.262+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2720144596 0 0) 0x7f4bcc11a710 con 0x7f4bcc0fc870 2024-02-19T21:15:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.262+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 >> v1:172.21.15.193:6789/0 conn(0x7f4bcc0ff670 legacy=0x7f4bcc0610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.262+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 >> v1:172.21.15.163:6789/0 conn(0x7f4bcc1031b0 legacy=0x7f4bcc117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.262+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bcc11cb10 con 0x7f4bcc0fc870 2024-02-19T21:15:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.262+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bcc11bb00 con 0x7f4bcc0fc870 2024-02-19T21:15:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.262+0000 7f4bd43cd700 10 client.50384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.262+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bcc11bff0 con 0x7f4bcc0fc870 2024-02-19T21:15:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.262+0000 7f4bd43cd700 1 librados: init done 2024-02-19T21:15:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.262+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f4bc8001be0 con 0x7f4bcc0fc870 2024-02-19T21:15:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.262+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4bc8005b70 con 0x7f4bcc0fc870 2024-02-19T21:15:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.263+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4bb0004f00 con 0x7f4bcc0fc870 2024-02-19T21:15:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.263+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f4bc8018a00 con 0x7f4bcc0fc870 2024-02-19T21:15:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.263+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(671..671 src has 434..671) v4 ==== 8563+0+0 (unknown 2690011502 0 0) 0x7f4bc8078ec0 con 0x7f4bcc0fc870 2024-02-19T21:15:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.263+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 osd_map(671..671 src has 434..671) v4 2024-02-19T21:15:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.263+0000 7f4bc37fe700 3 client.50384.objecter handle_osd_map got epochs [671,671] > 0 2024-02-19T21:15:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.263+0000 7f4bc37fe700 3 client.50384.objecter handle_osd_map decoding full epoch 671 2024-02-19T21:15:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.263+0000 7f4bc37fe700 20 client.50384.objecter dump_active .. 0 homeless 2024-02-19T21:15:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.263+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v671) v1 ==== 54+0+16 (unknown 1203433417 0 2420000192) 0x7f4bc807b2d0 con 0x7f4bcc0fc870 2024-02-19T21:15:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.264+0000 7f4bc37fe700 10 client.50384.objecter ms_handle_connect 0x7f4bb8063d20 2024-02-19T21:15:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.264+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.264+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 1 2024-02-19T21:15:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.264+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb00051a0 osd=0 3 2024-02-19T21:15:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.264+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb00051a0 osd=0 3 2024-02-19T21:15:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.264+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb00051a0 osd=0 3 2024-02-19T21:15:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.264+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 1 2024-02-19T21:15:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.264+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb00051a0 osd=0 3 2024-02-19T21:15:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.264+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 0 1 2024-02-19T21:15:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.264+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb00051a0 osd=0 4 2024-02-19T21:15:16.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.264+0000 7f4bd43cd700 10 client.50384.objecter _send_command 1 2024-02-19T21:15:16.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.264+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bb0004950 con 0x7f4bb0004530 2024-02-19T21:15:16.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.264+0000 7f4bc37fe700 10 client.50384.objecter ms_handle_connect 0x7f4bb0004530 2024-02-19T21:15:16.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.265+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4bb0004950 con 0x7f4bb0004530 2024-02-19T21:15:16.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.265+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:15:16.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.265+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 1 = system:0 2024-02-19T21:15:16.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.265+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb00051a0 osd=0 4 2024-02-19T21:15:16.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.265+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 0 1 2024-02-19T21:15:16.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.280+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:16.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.280+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 2 2024-02-19T21:15:16.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.280+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb00051a0 osd=0 3 2024-02-19T21:15:16.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.280+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb00051a0 osd=0 3 2024-02-19T21:15:16.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.280+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb00051a0 osd=0 3 2024-02-19T21:15:16.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.280+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 2 2024-02-19T21:15:16.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.280+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb00051a0 osd=0 3 2024-02-19T21:15:16.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.280+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 0 2 2024-02-19T21:15:16.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.280+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb00051a0 osd=0 4 2024-02-19T21:15:16.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.280+0000 7f4bd43cd700 10 client.50384.objecter _send_command 2 2024-02-19T21:15:16.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.280+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4bb0004d30 con 0x7f4bb0004530 2024-02-19T21:15:16.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4bb0004d30 con 0x7f4bb0004530 2024-02-19T21:15:16.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:16.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb00051a0 osd=0 4 2024-02-19T21:15:16.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 0 2 2024-02-19T21:15:16.283 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:15:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 3 2024-02-19T21:15:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0003fe0 osd=1 3 2024-02-19T21:15:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0003fe0 osd=1 3 2024-02-19T21:15:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0003fe0 osd=1 3 2024-02-19T21:15:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 3 2024-02-19T21:15:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb0003fe0 osd=1 3 2024-02-19T21:15:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 1 3 2024-02-19T21:15:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0003fe0 osd=1 4 2024-02-19T21:15:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bd43cd700 10 client.50384.objecter _send_command 3 2024-02-19T21:15:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bb000c000 con 0x7f4bb0004150 2024-02-19T21:15:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.282+0000 7f4bc37fe700 10 client.50384.objecter ms_handle_connect 0x7f4bb0004150 2024-02-19T21:15:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.283+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4bb000c000 con 0x7f4bb0004150 2024-02-19T21:15:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.283+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:15:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.283+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 3 = system:0 2024-02-19T21:15:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.283+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb0003fe0 osd=1 4 2024-02-19T21:15:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.283+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 1 3 2024-02-19T21:15:16.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.297+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:16.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.297+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 4 2024-02-19T21:15:16.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.297+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0003fe0 osd=1 3 2024-02-19T21:15:16.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.297+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0003fe0 osd=1 3 2024-02-19T21:15:16.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.297+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0003fe0 osd=1 3 2024-02-19T21:15:16.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.297+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 4 2024-02-19T21:15:16.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.297+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb0003fe0 osd=1 3 2024-02-19T21:15:16.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.297+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 1 4 2024-02-19T21:15:16.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.297+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0003fe0 osd=1 4 2024-02-19T21:15:16.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.297+0000 7f4bd43cd700 10 client.50384.objecter _send_command 4 2024-02-19T21:15:16.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.297+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4bb00018b0 con 0x7f4bb0004150 2024-02-19T21:15:16.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4bb00018b0 con 0x7f4bb0004150 2024-02-19T21:15:16.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:16.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb0003fe0 osd=1 4 2024-02-19T21:15:16.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 1 4 2024-02-19T21:15:16.300 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:15:16.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:16.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 5 2024-02-19T21:15:16.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0003630 osd=2 3 2024-02-19T21:15:16.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0003630 osd=2 3 2024-02-19T21:15:16.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0003630 osd=2 3 2024-02-19T21:15:16.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 5 2024-02-19T21:15:16.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb0003630 osd=2 3 2024-02-19T21:15:16.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 2 5 2024-02-19T21:15:16.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0003630 osd=2 4 2024-02-19T21:15:16.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bd43cd700 10 client.50384.objecter _send_command 5 2024-02-19T21:15:16.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bb0003e70 con 0x7f4bb000d2f0 2024-02-19T21:15:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.299+0000 7f4bc37fe700 10 client.50384.objecter ms_handle_connect 0x7f4bb000d2f0 2024-02-19T21:15:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.300+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4bb0003e70 con 0x7f4bb000d2f0 2024-02-19T21:15:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.300+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:15:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.300+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 5 = system:0 2024-02-19T21:15:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.300+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb0003630 osd=2 4 2024-02-19T21:15:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.300+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 2 5 2024-02-19T21:15:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.314+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.314+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 6 2024-02-19T21:15:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.314+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0003630 osd=2 3 2024-02-19T21:15:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.314+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0003630 osd=2 3 2024-02-19T21:15:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.314+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0003630 osd=2 3 2024-02-19T21:15:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.314+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 6 2024-02-19T21:15:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.314+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb0003630 osd=2 3 2024-02-19T21:15:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.314+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 2 6 2024-02-19T21:15:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.314+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0003630 osd=2 4 2024-02-19T21:15:16.315 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.314+0000 7f4bd43cd700 10 client.50384.objecter _send_command 6 2024-02-19T21:15:16.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.314+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4bb000e710 con 0x7f4bb000d2f0 2024-02-19T21:15:16.316 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:16.316 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4bb000e710 con 0x7f4bb000d2f0 2024-02-19T21:15:16.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:16.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.317 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:15:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb0003630 osd=2 4 2024-02-19T21:15:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 2 6 2024-02-19T21:15:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 7 2024-02-19T21:15:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb00033c0 osd=3 3 2024-02-19T21:15:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb00033c0 osd=3 3 2024-02-19T21:15:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb00033c0 osd=3 3 2024-02-19T21:15:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 7 2024-02-19T21:15:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb00033c0 osd=3 3 2024-02-19T21:15:16.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 3 7 2024-02-19T21:15:16.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb00033c0 osd=3 4 2024-02-19T21:15:16.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bd43cd700 10 client.50384.objecter _send_command 7 2024-02-19T21:15:16.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bb0015a40 con 0x7f4bb0011e80 2024-02-19T21:15:16.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.316+0000 7f4bc37fe700 10 client.50384.objecter ms_handle_connect 0x7f4bb0011e80 2024-02-19T21:15:16.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.317+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4bb0015a40 con 0x7f4bb0011e80 2024-02-19T21:15:16.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.317+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:15:16.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.317+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 7 = system:0 2024-02-19T21:15:16.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.317+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb00033c0 osd=3 4 2024-02-19T21:15:16.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.317+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 3 7 2024-02-19T21:15:16.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.332+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:16.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.332+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 8 2024-02-19T21:15:16.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.332+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb00033c0 osd=3 3 2024-02-19T21:15:16.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.332+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb00033c0 osd=3 3 2024-02-19T21:15:16.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.332+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb00033c0 osd=3 3 2024-02-19T21:15:16.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.332+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 8 2024-02-19T21:15:16.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.332+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb00033c0 osd=3 3 2024-02-19T21:15:16.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.332+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 3 8 2024-02-19T21:15:16.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.332+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb00033c0 osd=3 4 2024-02-19T21:15:16.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.332+0000 7f4bd43cd700 10 client.50384.objecter _send_command 8 2024-02-19T21:15:16.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.332+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4bb000e180 con 0x7f4bb0011e80 2024-02-19T21:15:16.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4bb000e180 con 0x7f4bb0011e80 2024-02-19T21:15:16.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:16.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb00033c0 osd=3 4 2024-02-19T21:15:16.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 3 8 2024-02-19T21:15:16.334 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:15:16.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:16.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 9 2024-02-19T21:15:16.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb000e180 osd=4 3 2024-02-19T21:15:16.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb000e180 osd=4 3 2024-02-19T21:15:16.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb000e180 osd=4 3 2024-02-19T21:15:16.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 9 2024-02-19T21:15:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb000e180 osd=4 3 2024-02-19T21:15:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 4 9 2024-02-19T21:15:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb000e180 osd=4 4 2024-02-19T21:15:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bd43cd700 10 client.50384.objecter _send_command 9 2024-02-19T21:15:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.333+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bb000e4c0 con 0x7f4bb000ce50 2024-02-19T21:15:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.334+0000 7f4bc37fe700 10 client.50384.objecter ms_handle_connect 0x7f4bb000ce50 2024-02-19T21:15:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.335+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4bb000e4c0 con 0x7f4bb000ce50 2024-02-19T21:15:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.335+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:15:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.335+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 9 = system:0 2024-02-19T21:15:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.335+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb000e180 osd=4 4 2024-02-19T21:15:16.336 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.335+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 4 9 2024-02-19T21:15:16.351 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:16.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.355+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:16.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.355+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 10 2024-02-19T21:15:16.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.355+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb000e180 osd=4 3 2024-02-19T21:15:16.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.355+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb000e180 osd=4 3 2024-02-19T21:15:16.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.355+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb000e180 osd=4 3 2024-02-19T21:15:16.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.355+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 10 2024-02-19T21:15:16.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.355+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb000e180 osd=4 3 2024-02-19T21:15:16.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.355+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 4 10 2024-02-19T21:15:16.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.355+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb000e180 osd=4 4 2024-02-19T21:15:16.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.355+0000 7f4bd43cd700 10 client.50384.objecter _send_command 10 2024-02-19T21:15:16.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.355+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4bb000e4c0 con 0x7f4bb000ce50 2024-02-19T21:15:16.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.357+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4bb000e4c0 con 0x7f4bb000ce50 2024-02-19T21:15:16.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.357+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:16.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.357+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.357+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb000e180 osd=4 4 2024-02-19T21:15:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.357+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 4 10 2024-02-19T21:15:16.359 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:15:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.358+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.358+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 11 2024-02-19T21:15:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.358+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb000e4c0 osd=5 3 2024-02-19T21:15:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.358+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb000e4c0 osd=5 3 2024-02-19T21:15:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.358+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb000e4c0 osd=5 3 2024-02-19T21:15:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.358+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 11 2024-02-19T21:15:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.358+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb000e4c0 osd=5 3 2024-02-19T21:15:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.358+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 5 11 2024-02-19T21:15:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.358+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb000e4c0 osd=5 4 2024-02-19T21:15:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.358+0000 7f4bd43cd700 10 client.50384.objecter _send_command 11 2024-02-19T21:15:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.358+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bb0016d70 con 0x7f4bb0016800 2024-02-19T21:15:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.358+0000 7f4bc37fe700 10 client.50384.objecter ms_handle_connect 0x7f4bb0016800 2024-02-19T21:15:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.359+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4bb0016d70 con 0x7f4bb0016800 2024-02-19T21:15:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.359+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:15:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.359+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 11 = system:0 2024-02-19T21:15:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.359+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb000e4c0 osd=5 4 2024-02-19T21:15:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.359+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 5 11 2024-02-19T21:15:16.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.374+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:16.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.374+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 12 2024-02-19T21:15:16.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.374+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb000e4c0 osd=5 3 2024-02-19T21:15:16.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.374+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb000e4c0 osd=5 3 2024-02-19T21:15:16.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.374+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb000e4c0 osd=5 3 2024-02-19T21:15:16.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.374+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 12 2024-02-19T21:15:16.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.374+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb000e4c0 osd=5 3 2024-02-19T21:15:16.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.374+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 5 12 2024-02-19T21:15:16.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.374+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb000e4c0 osd=5 4 2024-02-19T21:15:16.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.374+0000 7f4bd43cd700 10 client.50384.objecter _send_command 12 2024-02-19T21:15:16.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.374+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4bb0016d70 con 0x7f4bb0016800 2024-02-19T21:15:16.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4bb0016d70 con 0x7f4bb0016800 2024-02-19T21:15:16.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:16.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb000e4c0 osd=5 4 2024-02-19T21:15:16.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 5 12 2024-02-19T21:15:16.377 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:15:16.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:16.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 13 2024-02-19T21:15:16.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0016d70 osd=6 3 2024-02-19T21:15:16.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0016d70 osd=6 3 2024-02-19T21:15:16.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0016d70 osd=6 3 2024-02-19T21:15:16.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 13 2024-02-19T21:15:16.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb0016d70 osd=6 3 2024-02-19T21:15:16.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 6 13 2024-02-19T21:15:16.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0016d70 osd=6 4 2024-02-19T21:15:16.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bd43cd700 10 client.50384.objecter _send_command 13 2024-02-19T21:15:16.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.376+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bb0022b50 con 0x7f4bb001f040 2024-02-19T21:15:16.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.377+0000 7f4bc37fe700 10 client.50384.objecter ms_handle_connect 0x7f4bb001f040 2024-02-19T21:15:16.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.378+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4bb0022b50 con 0x7f4bb001f040 2024-02-19T21:15:16.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.378+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:15:16.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.378+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 13 = system:0 2024-02-19T21:15:16.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.378+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb0016d70 osd=6 4 2024-02-19T21:15:16.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.378+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 6 13 2024-02-19T21:15:16.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.392+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:16.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.392+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 14 2024-02-19T21:15:16.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.392+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0016d70 osd=6 3 2024-02-19T21:15:16.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.392+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0016d70 osd=6 3 2024-02-19T21:15:16.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.392+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0016d70 osd=6 3 2024-02-19T21:15:16.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.392+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 14 2024-02-19T21:15:16.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.392+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb0016d70 osd=6 3 2024-02-19T21:15:16.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.392+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 6 14 2024-02-19T21:15:16.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.392+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0016d70 osd=6 4 2024-02-19T21:15:16.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.392+0000 7f4bd43cd700 10 client.50384.objecter _send_command 14 2024-02-19T21:15:16.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.392+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4bb0022b50 con 0x7f4bb001f040 2024-02-19T21:15:16.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4bb0022b50 con 0x7f4bb001f040 2024-02-19T21:15:16.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:16.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb0016d70 osd=6 4 2024-02-19T21:15:16.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 6 14 2024-02-19T21:15:16.395 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:15:16.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:16.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 15 2024-02-19T21:15:16.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0022b50 osd=7 3 2024-02-19T21:15:16.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0022b50 osd=7 3 2024-02-19T21:15:16.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0022b50 osd=7 3 2024-02-19T21:15:16.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 15 2024-02-19T21:15:16.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb0022b50 osd=7 3 2024-02-19T21:15:16.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 7 15 2024-02-19T21:15:16.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0022b50 osd=7 4 2024-02-19T21:15:16.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bd43cd700 10 client.50384.objecter _send_command 15 2024-02-19T21:15:16.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.394+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bb0024f10 con 0x7f4bb0023210 2024-02-19T21:15:16.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.395+0000 7f4bc37fe700 10 client.50384.objecter ms_handle_connect 0x7f4bb0023210 2024-02-19T21:15:16.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.395+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4bb0024f10 con 0x7f4bb0023210 2024-02-19T21:15:16.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.395+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:15:16.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.395+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 15 = system:0 2024-02-19T21:15:16.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.395+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb0022b50 osd=7 4 2024-02-19T21:15:16.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.395+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 7 15 2024-02-19T21:15:16.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.410+0000 7f4bd43cd700 10 client.50384.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:16.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.410+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign -1 16 2024-02-19T21:15:16.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.410+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0022b50 osd=7 3 2024-02-19T21:15:16.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.410+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0022b50 osd=7 3 2024-02-19T21:15:16.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.410+0000 7f4bd43cd700 20 client.50384.objecter _get_session s=0x7f4bb0022b50 osd=7 3 2024-02-19T21:15:16.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.410+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_remove -1 16 2024-02-19T21:15:16.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.410+0000 7f4bd43cd700 20 client.50384.objecter get_session s=0x7f4bb0022b50 osd=7 3 2024-02-19T21:15:16.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.410+0000 7f4bd43cd700 15 client.50384.objecter _session_command_op_assign 7 16 2024-02-19T21:15:16.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.410+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0022b50 osd=7 4 2024-02-19T21:15:16.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.410+0000 7f4bd43cd700 10 client.50384.objecter _send_command 16 2024-02-19T21:15:16.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.410+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4bb0024f10 con 0x7f4bb0023210 2024-02-19T21:15:16.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bc37fe700 1 -- 172.21.15.163:0/736187646 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4bb0024f10 con 0x7f4bb0023210 2024-02-19T21:15:16.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bc37fe700 10 client.50384.objecter ms_dispatch 0x7f4bcc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:16.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bc37fe700 10 client.50384.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bc37fe700 20 client.50384.objecter put_session s=0x7f4bb0022b50 osd=7 4 2024-02-19T21:15:16.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bc37fe700 15 client.50384.objecter _session_command_op_remove 7 16 2024-02-19T21:15:16.413 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:15:16.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 10 librados: watch_flush enter 2024-02-19T21:15:16.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 10 librados: watch_flush exit 2024-02-19T21:15:16.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 10 client.50384.objecter close_session for osd.0 2024-02-19T21:15:16.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 >> v1:172.21.15.163:6801/1559418115 conn(0x7f4bb0004530 legacy=0x7f4bb0002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:16.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb00051a0 osd=0 1 2024-02-19T21:15:16.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 10 client.50384.objecter close_session for osd.1 2024-02-19T21:15:16.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 >> v1:172.21.15.163:6805/3125047432 conn(0x7f4bb0004150 legacy=0x7f4bb000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:16.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0003fe0 osd=1 1 2024-02-19T21:15:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 10 client.50384.objecter close_session for osd.2 2024-02-19T21:15:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 >> v1:172.21.15.163:6809/1563158487 conn(0x7f4bb000d2f0 legacy=0x7f4bb000d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0003630 osd=2 1 2024-02-19T21:15:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 10 client.50384.objecter close_session for osd.3 2024-02-19T21:15:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 >> v1:172.21.15.163:6813/2112279769 conn(0x7f4bb0011e80 legacy=0x7f4bb0014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb00033c0 osd=3 1 2024-02-19T21:15:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 10 client.50384.objecter close_session for osd.4 2024-02-19T21:15:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd296a700 1 -- 172.21.15.163:0/736187646 reap_dead start 2024-02-19T21:15:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 >> v1:172.21.15.193:6800/3129716965 conn(0x7f4bb000ce50 legacy=0x7f4bb0015be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd296a700 1 -- 172.21.15.163:0/736187646 reap_dead start 2024-02-19T21:15:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb000e180 osd=4 1 2024-02-19T21:15:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 10 client.50384.objecter close_session for osd.5 2024-02-19T21:15:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 >> v1:172.21.15.193:6804/805161991 conn(0x7f4bb0016800 legacy=0x7f4bb001d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb000e4c0 osd=5 1 2024-02-19T21:15:16.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 10 client.50384.objecter close_session for osd.6 2024-02-19T21:15:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 >> v1:172.21.15.193:6808/787604615 conn(0x7f4bb001f040 legacy=0x7f4bb0021430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0016d70 osd=6 1 2024-02-19T21:15:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 10 client.50384.objecter close_session for osd.7 2024-02-19T21:15:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 >> v1:172.21.15.193:6812/157927432 conn(0x7f4bb0023210 legacy=0x7f4bb0023690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 20 client.50384.objecter put_session s=0x7f4bb0022b50 osd=7 1 2024-02-19T21:15:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 20 client.50384.objecter shutdown clearing up homeless session... 2024-02-19T21:15:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 10 client.50384.objecter successfully canceled tick 2024-02-19T21:15:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4bb8063d20 legacy=0x7f4bb80661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 >> v1:172.21.15.163:6790/0 conn(0x7f4bcc0fc870 legacy=0x7f4bcc0609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd296a700 1 -- 172.21.15.163:0/736187646 reap_dead start 2024-02-19T21:15:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 shutdown_connections 2024-02-19T21:15:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 >> 172.21.15.163:0/736187646 conn(0x7f4bcc0f2470 msgr2=0x7f4bcc106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:16.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 shutdown_connections 2024-02-19T21:15:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 1 -- 172.21.15.163:0/736187646 wait complete. 2024-02-19T21:15:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:16.412+0000 7f4bd43cd700 1 librados: shutdown 2024-02-19T21:15:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.417 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:15:16.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:15.301571+0000 mgr.y (mgr.14550) 571 : cluster [DBG] pgmap v1009: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:16.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:15.436238+0000 mon.a (mon.0) 3310 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:16.571 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:16.264807+0000 mon.c (mon.2) 422 : audit [DBG] from='client.? v1:172.21.15.163:0/736187646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:16.652 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:15.301571+0000 mgr.y (mgr.14550) 571 : cluster [DBG] pgmap v1009: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:16.652 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:15.436238+0000 mon.a (mon.0) 3310 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:16.652 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:16.264807+0000 mon.c (mon.2) 422 : audit [DBG] from='client.? v1:172.21.15.163:0/736187646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:16.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:15.301571+0000 mgr.y (mgr.14550) 571 : cluster [DBG] pgmap v1009: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:16.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:15.436238+0000 mon.a (mon.0) 3310 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:16.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:16.264807+0000 mon.c (mon.2) 422 : audit [DBG] from='client.? v1:172.21.15.163:0/736187646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:17.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:16.424893+0000 mon.a (mon.0) 3311 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:17.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:16.424954+0000 mon.a (mon.0) 3312 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2024-02-19T21:15:17.658 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:16.425807+0000 mon.a (mon.0) 3313 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-116"}]: dispatch 2024-02-19T21:15:17.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:16.424893+0000 mon.a (mon.0) 3311 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:17.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:16.424954+0000 mon.a (mon.0) 3312 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2024-02-19T21:15:17.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:16.425807+0000 mon.a (mon.0) 3313 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-116"}]: dispatch 2024-02-19T21:15:17.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:16.424893+0000 mon.a (mon.0) 3311 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:17.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:16.424954+0000 mon.a (mon.0) 3312 : cluster [DBG] osdmap e672: 8 total, 8 up, 8 in 2024-02-19T21:15:17.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:16.425807+0000 mon.a (mon.0) 3313 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-116"}]: dispatch 2024-02-19T21:15:18.071 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:15:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:15:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:15:18.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:17.301915+0000 mgr.y (mgr.14550) 572 : cluster [DBG] pgmap v1012: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:18.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:17.422762+0000 mon.a (mon.0) 3314 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:18.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:17.429275+0000 mon.a (mon.0) 3315 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-116"}]': finished 2024-02-19T21:15:18.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:17.429323+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2024-02-19T21:15:18.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:17.301915+0000 mgr.y (mgr.14550) 572 : cluster [DBG] pgmap v1012: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:18.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:17.422762+0000 mon.a (mon.0) 3314 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:18.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:17.429275+0000 mon.a (mon.0) 3315 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-116"}]': finished 2024-02-19T21:15:18.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:17.429323+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2024-02-19T21:15:18.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:17.301915+0000 mgr.y (mgr.14550) 572 : cluster [DBG] pgmap v1012: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:18.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:17.422762+0000 mon.a (mon.0) 3314 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:18.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:17.429275+0000 mon.a (mon.0) 3315 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-116"}]': finished 2024-02-19T21:15:18.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:17.429323+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e673: 8 total, 8 up, 8 in 2024-02-19T21:15:19.101 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:19.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:19.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:19.205 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:19.236 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:19.312 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:19.326 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:19.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:19.428 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:19.449 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:15:19.449 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-02-19T21:15:19.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-02-19T21:15:19.449 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-02-19T21:15:19.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.450 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.45484.0:2 92.7s0 92:ed965bc3:test-rados-api-smithi163-83944-15::foo:head [write 0~8192] snapc 2=[2] ondisk+write+known_if_redirected e142)", 2024-02-19T21:15:19.450 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:22.184478+0000", 2024-02-19T21:15:19.450 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 597.20062928599998, 2024-02-19T21:15:19.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046650480000000001, 2024-02-19T21:15:19.450 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.450 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:19.450 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:15:19.450 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.45484", 2024-02-19T21:15:19.450 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/4096098867", 2024-02-19T21:15:19.450 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2 2024-02-19T21:15:19.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.450 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184478+0000", 2024-02-19T21:15:19.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184478+0000", 2024-02-19T21:15:19.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:15:19.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184476+0000", 2024-02-19T21:15:19.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6813999999999999e-05 2024-02-19T21:15:19.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184492+0000", 2024-02-19T21:15:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1202e-05 2024-02-19T21:15:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184504+0000", 2024-02-19T21:15:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8180000000000004e-06 2024-02-19T21:15:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184511+0000", 2024-02-19T21:15:19.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7104000000000002e-05 2024-02-19T21:15:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184558+0000", 2024-02-19T21:15:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3579999999999999e-06 2024-02-19T21:15:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:15:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.184562+0000", 2024-02-19T21:15:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.033812456999999997 2024-02-19T21:15:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.218374+0000", 2024-02-19T21:15:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012377300000000001 2024-02-19T21:15:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.454 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.218498+0000", 2024-02-19T21:15:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00042174699999999999 2024-02-19T21:15:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:15:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.218920+0000", 2024-02-19T21:15:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0096631270000000005 2024-02-19T21:15:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:15:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.228583+0000", 2024-02-19T21:15:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024784189999999999 2024-02-19T21:15:19.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:15:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.231061+0000", 2024-02-19T21:15:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6945000000000004e-05 2024-02-19T21:15:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:22.231128+0000", 2024-02-19T21:15:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046650480000000001 2024-02-19T21:15:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.47779.0:2 114.13 e169/168 114:cbb9903e:::test_obj:head v 169'1, mlcod=0'0)", 2024-02-19T21:15:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:05:49.216541+0000", 2024-02-19T21:15:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 570.16856588200005, 2024-02-19T21:15:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.033635538999999999, 2024-02-19T21:15:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.216541+0000", 2024-02-19T21:15:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.457 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.216541+0000", 2024-02-19T21:15:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:15:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.216541+0000", 2024-02-19T21:15:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8630000000000002e-06 2024-02-19T21:15:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.216548+0000", 2024-02-19T21:15:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5470000000000001e-06 2024-02-19T21:15:19.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.216551+0000", 2024-02-19T21:15:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5249999999999999e-06 2024-02-19T21:15:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.216556+0000", 2024-02-19T21:15:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027499585 2024-02-19T21:15:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.244055+0000", 2024-02-19T21:15:19.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3051999999999999e-05 2024-02-19T21:15:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.244079+0000", 2024-02-19T21:15:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0060570659999999998 2024-02-19T21:15:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:15:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.250136+0000", 2024-02-19T21:15:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1170000000000001e-05 2024-02-19T21:15:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:05:49.250177+0000", 2024-02-19T21:15:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.033635538999999999 2024-02-19T21:15:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:1429 61.12 61:48d9abf6:test-rados-api-smithi163-83928-16::foo:head [writefull 0~5] snapc c=[c] ondisk+write+known_if_redirected e191)", 2024-02-19T21:15:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:06:11.199452+0000", 2024-02-19T21:15:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 548.18565545499996, 2024-02-19T21:15:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052298378, 2024-02-19T21:15:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:19.461 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:15:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:15:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:15:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1429 2024-02-19T21:15:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.199452+0000", 2024-02-19T21:15:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.199452+0000", 2024-02-19T21:15:19.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999957 2024-02-19T21:15:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.199447+0000", 2024-02-19T21:15:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6351999999999999e-05 2024-02-19T21:15:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.199464+0000", 2024-02-19T21:15:19.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5387999999999999e-05 2024-02-19T21:15:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.199479+0000", 2024-02-19T21:15:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3209999999999994e-06 2024-02-19T21:15:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.199489+0000", 2024-02-19T21:15:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.01039052 2024-02-19T21:15:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.209879+0000", 2024-02-19T21:15:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010921699999999999 2024-02-19T21:15:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.209988+0000", 2024-02-19T21:15:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00021893100000000001 2024-02-19T21:15:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,3", 2024-02-19T21:15:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.210207+0000", 2024-02-19T21:15:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.028331134000000001 2024-02-19T21:15:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:15:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.238538+0000", 2024-02-19T21:15:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00095535699999999999 2024-02-19T21:15:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:15:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.239494+0000", 2024-02-19T21:15:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.012141102000000001 2024-02-19T21:15:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:15:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.251635+0000", 2024-02-19T21:15:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0659000000000003e-05 2024-02-19T21:15:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:15:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.251695+0000", 2024-02-19T21:15:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4573000000000001e-05 2024-02-19T21:15:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.251750+0000", 2024-02-19T21:15:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.052298378 2024-02-19T21:15:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.467 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.22407.0:26 126.6s0 126:7ff15785:test-rados-api-smithi163-83952-23::foo:head [write 0~8192] snapc a=[a] ondisk+write+known_if_redirected e191)", 2024-02-19T21:15:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:06:11.207404+0000", 2024-02-19T21:15:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 548.17770322499996, 2024-02-19T21:15:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.033762901999999997, 2024-02-19T21:15:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:15:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.22407", 2024-02-19T21:15:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/4166546335", 2024-02-19T21:15:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 26 2024-02-19T21:15:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.207404+0000", 2024-02-19T21:15:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.207404+0000", 2024-02-19T21:15:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:15:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.207401+0000", 2024-02-19T21:15:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0035999999999999e-05 2024-02-19T21:15:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.207421+0000", 2024-02-19T21:15:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.014837427 2024-02-19T21:15:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.222259+0000", 2024-02-19T21:15:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5291999999999999e-05 2024-02-19T21:15:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.222274+0000", 2024-02-19T21:15:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00045052100000000001 2024-02-19T21:15:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.222725+0000", 2024-02-19T21:15:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014116500000000001 2024-02-19T21:15:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.222866+0000", 2024-02-19T21:15:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00053662000000000002 2024-02-19T21:15:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:15:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.223402+0000", 2024-02-19T21:15:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015991278000000001 2024-02-19T21:15:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:15:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.239394+0000", 2024-02-19T21:15:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0016954959999999999 2024-02-19T21:15:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:15:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.241089+0000", 2024-02-19T21:15:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7653999999999999e-05 2024-02-19T21:15:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:06:11.241167+0000", 2024-02-19T21:15:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.033762901999999997 2024-02-19T21:15:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.472 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.49400.0:1 175.6s0 175:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e265)", 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:07:25.213923+0000", 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 474.171184504, 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0203528529999999, 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.49400", 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/896457325", 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.473 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213923+0000", 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213923+0000", 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213921+0000", 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5069999999999997e-06 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.474 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213927+0000", 2024-02-19T21:15:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2819999999999996e-06 2024-02-19T21:15:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213936+0000", 2024-02-19T21:15:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2279999999999998e-06 2024-02-19T21:15:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213941+0000", 2024-02-19T21:15:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3912999999999997e-05 2024-02-19T21:15:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213995+0000", 2024-02-19T21:15:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4039999999999999e-06 2024-02-19T21:15:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:15:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:25.213998+0000", 2024-02-19T21:15:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0148429459999999 2024-02-19T21:15:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:26.228841+0000", 2024-02-19T21:15:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000112502 2024-02-19T21:15:19.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:26.228954+0000", 2024-02-19T21:15:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00035581099999999999 2024-02-19T21:15:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:15:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:26.229309+0000", 2024-02-19T21:15:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00088503599999999996 2024-02-19T21:15:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:15:19.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:26.230194+0000", 2024-02-19T21:15:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039628809999999997 2024-02-19T21:15:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:15:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:26.234157+0000", 2024-02-19T21:15:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000118199 2024-02-19T21:15:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:26.234275+0000", 2024-02-19T21:15:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0203528529999999 2024-02-19T21:15:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.478 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:3227 186.3 186:c0520479:test-rados-api-smithi163-83928-38::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected e294)", 2024-02-19T21:15:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:07:54.246514+0000", 2024-02-19T21:15:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 445.13859279100001, 2024-02-19T21:15:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026188934000000001, 2024-02-19T21:15:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:15:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:15:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:15:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:15:19.479 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3227 2024-02-19T21:15:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.246514+0000", 2024-02-19T21:15:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.246514+0000", 2024-02-19T21:15:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:15:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.246513+0000", 2024-02-19T21:15:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9750000000000001e-06 2024-02-19T21:15:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.246517+0000", 2024-02-19T21:15:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7299999999999999e-06 2024-02-19T21:15:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.246524+0000", 2024-02-19T21:15:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4610000000000001e-06 2024-02-19T21:15:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.246529+0000", 2024-02-19T21:15:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.010273137999999999 2024-02-19T21:15:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.256802+0000", 2024-02-19T21:15:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7600000000000003e-05 2024-02-19T21:15:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.256869+0000", 2024-02-19T21:15:19.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015678469 2024-02-19T21:15:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.272548+0000", 2024-02-19T21:15:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9383999999999997e-05 2024-02-19T21:15:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.272637+0000", 2024-02-19T21:15:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6144999999999998e-05 2024-02-19T21:15:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.272703+0000", 2024-02-19T21:15:19.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.026188934000000001 2024-02-19T21:15:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.3.25:34 216.16 e452/444 216:6dfc172a:test-rados-api-smithi163-83928-70::aab9b7bd17003da21256cca5c7d2d7ff4ea384b6:head v 452'3, mlcod=446'1)", 2024-02-19T21:15:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:11:04.382842+0000", 2024-02-19T21:15:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 255.00226496600001, 2024-02-19T21:15:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025448117999999999, 2024-02-19T21:15:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382842+0000", 2024-02-19T21:15:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382842+0000", 2024-02-19T21:15:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:15:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382839+0000", 2024-02-19T21:15:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4117000000000001e-05 2024-02-19T21:15:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382854+0000", 2024-02-19T21:15:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1486000000000001e-05 2024-02-19T21:15:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382875+0000", 2024-02-19T21:15:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.6940000000000007e-06 2024-02-19T21:15:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382885+0000", 2024-02-19T21:15:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4758000000000005e-05 2024-02-19T21:15:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382959+0000", 2024-02-19T21:15:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7180000000000001e-05 2024-02-19T21:15:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382987+0000", 2024-02-19T21:15:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025254843999999999 2024-02-19T21:15:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:15:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408242+0000", 2024-02-19T21:15:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8742999999999997e-05 2024-02-19T21:15:19.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408290+0000", 2024-02-19T21:15:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025448117999999999 2024-02-19T21:15:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:6737 217.8 217:10000000::::head [pgnls start_epoch 461] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e461)", 2024-02-19T21:15:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:11:29.410084+0000", 2024-02-19T21:15:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 229.97502325900001, 2024-02-19T21:15:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046636691000000001, 2024-02-19T21:15:19.488 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:15:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:15:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:15:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:15:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 6737 2024-02-19T21:15:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:29.410084+0000", 2024-02-19T21:15:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:29.410084+0000", 2024-02-19T21:15:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999948 2024-02-19T21:15:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:29.410078+0000", 2024-02-19T21:15:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2959000000000002e-05 2024-02-19T21:15:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:29.410101+0000", 2024-02-19T21:15:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4094000000000003e-05 2024-02-19T21:15:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:29.410136+0000", 2024-02-19T21:15:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3301e-05 2024-02-19T21:15:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:29.410149+0000", 2024-02-19T21:15:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3234999999999999e-05 2024-02-19T21:15:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:29.410222+0000", 2024-02-19T21:15:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4120000000000004e-06 2024-02-19T21:15:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:15:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:29.410229+0000", 2024-02-19T21:15:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046319151000000003 2024-02-19T21:15:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:29.456549+0000", 2024-02-19T21:15:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2302999999999998e-05 2024-02-19T21:15:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:29.456591+0000", 2024-02-19T21:15:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000129589 2024-02-19T21:15:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:29.456721+0000", 2024-02-19T21:15:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046636691000000001 2024-02-19T21:15:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(osd.5.37:87 234.5s0 234:a8d35278:test-rados-api-smithi163-83928-109::foo:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e637)", 2024-02-19T21:15:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:14:41.376821+0000", 2024-02-19T21:15:19.493 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 38.008286054000003, 2024-02-19T21:15:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043573393000000002, 2024-02-19T21:15:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:15:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.376821+0000", 2024-02-19T21:15:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.376821+0000", 2024-02-19T21:15:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:15:19.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.376819+0000", 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.674e-06 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.376825+0000", 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1395e-05 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.376837+0000", 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9640000000000003e-06 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.376845+0000", 2024-02-19T21:15:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9064000000000001e-05 2024-02-19T21:15:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.376914+0000", 2024-02-19T21:15:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.315e-06 2024-02-19T21:15:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:15:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.376917+0000", 2024-02-19T21:15:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027298053999999999 2024-02-19T21:15:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.404215+0000", 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00042614600000000002 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for degraded object", 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.404641+0000", 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.015397551000000001 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.420039+0000", 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000120742 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.420160+0000", 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023484999999999999 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.420394+0000", 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.043573393000000002 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(osd.5.37:88 234.5s0 234:a8d35278:test-rados-api-smithi163-83928-109::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e637)", 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:14:41.376856+0000", 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 38.008250726999997, 2024-02-19T21:15:19.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044678411000000001, 2024-02-19T21:15:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:15:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.376856+0000", 2024-02-19T21:15:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.376856+0000", 2024-02-19T21:15:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:15:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.376856+0000", 2024-02-19T21:15:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8849999999999998e-06 2024-02-19T21:15:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.376860+0000", 2024-02-19T21:15:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4819999999999999e-06 2024-02-19T21:15:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.376867+0000", 2024-02-19T21:15:19.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7820000000000001e-06 2024-02-19T21:15:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.376872+0000", 2024-02-19T21:15:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00037350699999999997 2024-02-19T21:15:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.377245+0000", 2024-02-19T21:15:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.595e-06 2024-02-19T21:15:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:15:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.377247+0000", 2024-02-19T21:15:19.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.027581244000000001 2024-02-19T21:15:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.404828+0000", 2024-02-19T21:15:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5658999999999998e-05 2024-02-19T21:15:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for degraded object", 2024-02-19T21:15:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.404874+0000", 2024-02-19T21:15:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.01555452 2024-02-19T21:15:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.420428+0000", 2024-02-19T21:15:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7227999999999998e-05 2024-02-19T21:15:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.420525+0000", 2024-02-19T21:15:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001009334 2024-02-19T21:15:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:41.421535+0000", 2024-02-19T21:15:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.044678411000000001 2024-02-19T21:15:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.503 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDScrubReserve(2.17 REQUEST e656)", 2024-02-19T21:15:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:01.255342+0000", 2024-02-19T21:15:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.129765509999999, 2024-02-19T21:15:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004904149, 2024-02-19T21:15:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:15:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:01.255342+0000", 2024-02-19T21:15:19.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:01.255342+0000", 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:01.255341+0000", 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0289999999999999e-06 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:01.255345+0000", 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6968000000000001e-05 2024-02-19T21:15:19.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:01.255362+0000", 2024-02-19T21:15:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9680000000000001e-06 2024-02-19T21:15:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:01.255372+0000", 2024-02-19T21:15:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0419e-05 2024-02-19T21:15:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:01.255422+0000", 2024-02-19T21:15:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0090000000000001e-06 2024-02-19T21:15:19.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:01.255429+0000", 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0047737680000000003 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:01.260203+0000", 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4008999999999999e-05 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:01.260217+0000", 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8515000000000001e-05 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:01.260246+0000", 2024-02-19T21:15:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.004904149 2024-02-19T21:15:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:796 234.5s0 234:b8d4d292:test-rados-api-smithi163-83928-115::foo:head [writefull 0~5] snapc 6=[6] ondisk+write+known_if_redirected e659)", 2024-02-19T21:15:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:03.383188+0000", 2024-02-19T21:15:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.001918729, 2024-02-19T21:15:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044953279999999998, 2024-02-19T21:15:19.508 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.509 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:19.509 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:15:19.509 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:15:19.509 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:15:19.509 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 796 2024-02-19T21:15:19.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.509 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:03.383188+0000", 2024-02-19T21:15:19.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:03.383188+0000", 2024-02-19T21:15:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999962 2024-02-19T21:15:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:03.383184+0000", 2024-02-19T21:15:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6333000000000001e-05 2024-02-19T21:15:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:03.383201+0000", 2024-02-19T21:15:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8278e-05 2024-02-19T21:15:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:03.383219+0000", 2024-02-19T21:15:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0380000000000001e-05 2024-02-19T21:15:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:03.383229+0000", 2024-02-19T21:15:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2696999999999999e-05 2024-02-19T21:15:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:03.383302+0000", 2024-02-19T21:15:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0309999999999998e-06 2024-02-19T21:15:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:15:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:03.383310+0000", 2024-02-19T21:15:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0025326820000000001 2024-02-19T21:15:19.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:03.385843+0000", 2024-02-19T21:15:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.3863000000000006e-05 2024-02-19T21:15:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:03.385937+0000", 2024-02-19T21:15:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00052814400000000003 2024-02-19T21:15:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.516 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:15:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:03.386465+0000", 2024-02-19T21:15:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048100999999999998 2024-02-19T21:15:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:15:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:03.386946+0000", 2024-02-19T21:15:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00069724800000000003 2024-02-19T21:15:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:15:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:03.387643+0000", 2024-02-19T21:15:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0618999999999999e-05 2024-02-19T21:15:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:03.387684+0000", 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0044953279999999998 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:1038 242.18 242:18000000::::head [pgnls start_epoch 673] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e673)", 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:17.437632+0000", 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.947475133, 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019707200000000001, 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:15:19.518 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:15:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:15:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1038 2024-02-19T21:15:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.437632+0000", 2024-02-19T21:15:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.437632+0000", 2024-02-19T21:15:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:15:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.437629+0000", 2024-02-19T21:15:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0059e-05 2024-02-19T21:15:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.437639+0000", 2024-02-19T21:15:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1185999999999999e-05 2024-02-19T21:15:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.520 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.437651+0000", 2024-02-19T21:15:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9599999999999997e-06 2024-02-19T21:15:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.437657+0000", 2024-02-19T21:15:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6681000000000002e-05 2024-02-19T21:15:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.437713+0000", 2024-02-19T21:15:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6613999999999999e-05 2024-02-19T21:15:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.437750+0000", 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9182000000000002e-05 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.437829+0000", 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019707200000000001 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:1052 242.11 242:88000000::::head [pgnls start_epoch 673] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e673)", 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:17.449308+0000", 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.9357994870000002, 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017408500000000001, 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.522 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:15:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:15:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:15:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:15:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1052 2024-02-19T21:15:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.449308+0000", 2024-02-19T21:15:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.523 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.449308+0000", 2024-02-19T21:15:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:15:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.449306+0000", 2024-02-19T21:15:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.9170000000000003e-06 2024-02-19T21:15:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.449312+0000", 2024-02-19T21:15:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9890000000000004e-06 2024-02-19T21:15:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.449322+0000", 2024-02-19T21:15:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5589999999999999e-06 2024-02-19T21:15:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.449327+0000", 2024-02-19T21:15:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1415999999999997e-05 2024-02-19T21:15:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.449378+0000", 2024-02-19T21:15:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.731e-05 2024-02-19T21:15:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.449415+0000", 2024-02-19T21:15:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6304999999999996e-05 2024-02-19T21:15:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.449482+0000", 2024-02-19T21:15:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017408500000000001 2024-02-19T21:15:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.526 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:1060 242.13 242:c8000000::::head [pgnls start_epoch 673] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e673)", 2024-02-19T21:15:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:17.451835+0000", 2024-02-19T21:15:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.933272342, 2024-02-19T21:15:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017264, 2024-02-19T21:15:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:15:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:15:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:15:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:15:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1060 2024-02-19T21:15:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.451835+0000", 2024-02-19T21:15:19.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.451835+0000", 2024-02-19T21:15:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:15:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.451834+0000", 2024-02-19T21:15:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5700000000000003e-06 2024-02-19T21:15:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.451838+0000", 2024-02-19T21:15:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2819999999999999e-06 2024-02-19T21:15:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.451846+0000", 2024-02-19T21:15:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5020000000000003e-06 2024-02-19T21:15:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.451850+0000", 2024-02-19T21:15:19.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1269000000000001e-05 2024-02-19T21:15:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.451891+0000", 2024-02-19T21:15:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2370999999999999e-05 2024-02-19T21:15:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.451924+0000", 2024-02-19T21:15:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3574999999999999e-05 2024-02-19T21:15:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.530 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.452007+0000", 2024-02-19T21:15:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017264 2024-02-19T21:15:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:1062 242.1b 242:d8000000::::head [pgnls start_epoch 673] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e673)", 2024-02-19T21:15:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:17.452453+0000", 2024-02-19T21:15:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.932653634, 2024-02-19T21:15:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015254999999999999, 2024-02-19T21:15:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:15:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:15:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:15:19.531 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:15:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1062 2024-02-19T21:15:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.452453+0000", 2024-02-19T21:15:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.452453+0000", 2024-02-19T21:15:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:15:19.532 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.452453+0000", 2024-02-19T21:15:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7749999999999999e-06 2024-02-19T21:15:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.452456+0000", 2024-02-19T21:15:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8859999999999998e-06 2024-02-19T21:15:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.452463+0000", 2024-02-19T21:15:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4680000000000001e-06 2024-02-19T21:15:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.452467+0000", 2024-02-19T21:15:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3260999999999999e-05 2024-02-19T21:15:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.452500+0000", 2024-02-19T21:15:19.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1687000000000001e-05 2024-02-19T21:15:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.452522+0000", 2024-02-19T21:15:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4248999999999999e-05 2024-02-19T21:15:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.452606+0000", 2024-02-19T21:15:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015254999999999999 2024-02-19T21:15:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.535 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:1084 242.11 242:88000000:test-rados-api-smithi163-83928-117:::head [pgnls start_epoch 673] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e673)", 2024-02-19T21:15:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:17.459507+0000", 2024-02-19T21:15:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.925599823, 2024-02-19T21:15:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000154678, 2024-02-19T21:15:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:15:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:15:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:15:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:15:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1084 2024-02-19T21:15:19.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.459507+0000", 2024-02-19T21:15:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.459507+0000", 2024-02-19T21:15:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:15:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.537 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.459506+0000", 2024-02-19T21:15:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8259999999999998e-06 2024-02-19T21:15:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.459512+0000", 2024-02-19T21:15:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0271999999999999e-05 2024-02-19T21:15:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.459522+0000", 2024-02-19T21:15:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7339999999999999e-06 2024-02-19T21:15:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.459527+0000", 2024-02-19T21:15:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9369999999999997e-05 2024-02-19T21:15:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.459566+0000", 2024-02-19T21:15:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.5676999999999997e-05 2024-02-19T21:15:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.459602+0000", 2024-02-19T21:15:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0168999999999999e-05 2024-02-19T21:15:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.459662+0000", 2024-02-19T21:15:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000154678 2024-02-19T21:15:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:1094 242.1b 242:d8000000:test-rados-api-smithi163-83928-117:::head [pgnls start_epoch 673] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e673)", 2024-02-19T21:15:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:17.462650+0000", 2024-02-19T21:15:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 1.9224570079999999, 2024-02-19T21:15:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000160056, 2024-02-19T21:15:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:15:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:15:19.540 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1094 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.462650+0000", 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.462650+0000", 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:15:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.462649+0000", 2024-02-19T21:15:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9149999999999998e-06 2024-02-19T21:15:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:15:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.462653+0000", 2024-02-19T21:15:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.285e-06 2024-02-19T21:15:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:15:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.462659+0000", 2024-02-19T21:15:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7280000000000001e-06 2024-02-19T21:15:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.542 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.462663+0000", 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4790999999999999e-05 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.462698+0000", 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1501999999999998e-05 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.462729+0000", 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0659000000000001e-05 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:15:19.543 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:15:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:17.462810+0000", 2024-02-19T21:15:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000160056 2024-02-19T21:15:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:15:19.544 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:15:19.544 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:15:19.545 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:19.545 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:19.545 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:19.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:19.646 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:19.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:18.431022+0000 mon.a (mon.0) 3317 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2024-02-19T21:15:19.673 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:19.146123+0000 mon.c (mon.2) 423 : audit [INF] from='client.? v1:172.21.15.163:0/71631' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:15:19.674 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:19.146478+0000 mon.a (mon.0) 3318 : audit [INF] from='client.50390 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:15:19.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:18.431022+0000 mon.a (mon.0) 3317 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2024-02-19T21:15:19.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:19.146123+0000 mon.c (mon.2) 423 : audit [INF] from='client.? v1:172.21.15.163:0/71631' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:15:19.674 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:19.146478+0000 mon.a (mon.0) 3318 : audit [INF] from='client.50390 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:15:19.675 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:19.676 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:15:19.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:18.431022+0000 mon.a (mon.0) 3317 : cluster [DBG] osdmap e674: 8 total, 8 up, 8 in 2024-02-19T21:15:19.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:19.146123+0000 mon.c (mon.2) 423 : audit [INF] from='client.? v1:172.21.15.163:0/71631' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:15:19.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:19.146478+0000 mon.a (mon.0) 3318 : audit [INF] from='client.50390 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:15:20.128 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:20.129 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:20.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:20.230 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:20.256 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:20.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.956+0000 7f6dde8df700 1 -- 172.21.15.163:0/310959917 >> v1:172.21.15.163:6790/0 conn(0x7f6dd80fc870 legacy=0x7f6dd80fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:20.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.956+0000 7f6dde8df700 1 -- 172.21.15.163:0/310959917 shutdown_connections 2024-02-19T21:15:20.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.956+0000 7f6dde8df700 1 -- 172.21.15.163:0/310959917 >> 172.21.15.163:0/310959917 conn(0x7f6dd80f2470 msgr2=0x7f6dd80f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:20.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.956+0000 7f6dde8df700 1 -- 172.21.15.163:0/310959917 shutdown_connections 2024-02-19T21:15:20.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.956+0000 7f6dde8df700 1 -- 172.21.15.163:0/310959917 wait complete. 2024-02-19T21:15:20.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.956+0000 7f6dde8df700 1 librados: starting msgr at 2024-02-19T21:15:20.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.956+0000 7f6dde8df700 1 librados: starting objecter 2024-02-19T21:15:20.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.956+0000 7f6dde8df700 1 Processor -- start 2024-02-19T21:15:20.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.956+0000 7f6dde8df700 1 -- start start 2024-02-19T21:15:20.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.956+0000 7f6dde8df700 1 librados: setting wanted keys 2024-02-19T21:15:20.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.956+0000 7f6dde8df700 1 librados: calling monclient init 2024-02-19T21:15:20.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dde8df700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6dd8061780 con 0x7f6dd81031b0 2024-02-19T21:15:20.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dde8df700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6dd8061950 con 0x7f6dd80ff670 2024-02-19T21:15:20.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dde8df700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6dd81196f0 con 0x7f6dd80fc870 2024-02-19T21:15:20.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd7fff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f6dd80fc870 0x7f6dd8060d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:50872/0 (socket says 172.21.15.163:50872) 2024-02-19T21:15:20.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd7fff700 1 -- 172.21.15.163:0/71631 learned_addr learned my addr 172.21.15.163:0/71631 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:20.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 10 client.?.objecter ms_handle_connect 0x7f6dd80fc870 2024-02-19T21:15:20.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:20.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 10 client.?.objecter ms_handle_connect 0x7f6dd81031b0 2024-02-19T21:15:20.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:20.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916683615 0 0) 0x7f6dd81196f0 con 0x7f6dd80fc870 2024-02-19T21:15:20.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dc0003030 con 0x7f6dd80fc870 2024-02-19T21:15:20.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639531862 0 0) 0x7f6dd8061780 con 0x7f6dd81031b0 2024-02-19T21:15:20.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dd81196f0 con 0x7f6dd81031b0 2024-02-19T21:15:20.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 10 client.?.objecter ms_handle_connect 0x7f6dd80ff670 2024-02-19T21:15:20.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:20.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 447627509 0 0) 0x7f6dc0003030 con 0x7f6dd80fc870 2024-02-19T21:15:20.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dd8061780 con 0x7f6dd80fc870 2024-02-19T21:15:20.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6dc80015d0 con 0x7f6dd80fc870 2024-02-19T21:15:20.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1014608599 0 0) 0x7f6dd81196f0 con 0x7f6dd81031b0 2024-02-19T21:15:20.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dc0003030 con 0x7f6dd81031b0 2024-02-19T21:15:20.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198335306 0 0) 0x7f6dcc001230 con 0x7f6dd80ff670 2024-02-19T21:15:20.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dd81196f0 con 0x7f6dd80ff670 2024-02-19T21:15:20.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6dd0001b00 con 0x7f6dd81031b0 2024-02-19T21:15:20.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1977789879 0 0) 0x7f6dd8061780 con 0x7f6dd80fc870 2024-02-19T21:15:20.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 >> v1:172.21.15.193:6789/0 conn(0x7f6dd80ff670 legacy=0x7f6dd8112820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:20.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 >> v1:172.21.15.163:6789/0 conn(0x7f6dd81031b0 legacy=0x7f6dd8115f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:20.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.957+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd811a8f0 con 0x7f6dd80fc870 2024-02-19T21:15:20.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.958+0000 7f6dde8df700 1 -- 172.21.15.163:0/71631 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd81198e0 con 0x7f6dd80fc870 2024-02-19T21:15:20.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.958+0000 7f6dde8df700 10 client.50390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:20.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.958+0000 7f6dde8df700 1 -- 172.21.15.163:0/71631 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd8119d50 con 0x7f6dd80fc870 2024-02-19T21:15:20.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.958+0000 7f6dde8df700 1 librados: init done 2024-02-19T21:15:20.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.958+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6dc8001130 con 0x7f6dd80fc870 2024-02-19T21:15:20.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.958+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6dc8003920 con 0x7f6dd80fc870 2024-02-19T21:15:20.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.959+0000 7f6dde8df700 1 -- 172.21.15.163:0/71631 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc4004f20 con 0x7f6dd80fc870 2024-02-19T21:15:20.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.959+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6dc8001840 con 0x7f6dd80fc870 2024-02-19T21:15:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.959+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(674..674 src has 434..674) v4 ==== 8182+0+0 (unknown 3595176174 0 0) 0x7f6dc80790b0 con 0x7f6dd80fc870 2024-02-19T21:15:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.959+0000 7f6dd57fa700 10 client.50390.objecter ms_dispatch 0x7f6dd8001550 osd_map(674..674 src has 434..674) v4 2024-02-19T21:15:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.959+0000 7f6dd57fa700 3 client.50390.objecter handle_osd_map got epochs [674,674] > 0 2024-02-19T21:15:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.959+0000 7f6dd57fa700 3 client.50390.objecter handle_osd_map decoding full epoch 674 2024-02-19T21:15:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.960+0000 7f6dd57fa700 20 client.50390.objecter dump_active .. 0 homeless 2024-02-19T21:15:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.960+0000 7f6dd57fa700 10 client.50390.objecter ms_handle_connect 0x7f6dc0063c40 2024-02-19T21:15:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:18.964+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6dc804c490 con 0x7f6dd80fc870 2024-02-19T21:15:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:19.144+0000 7f6dde8df700 1 -- 172.21.15.163:0/71631 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7f6dc40051d0 con 0x7f6dd80fc870 2024-02-19T21:15:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:19.473+0000 7f6dd57fa700 1 -- 172.21.15.163:0/71631 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v675) v1 ==== 97+0+0 (unknown 1878394499 0 0) 0x7f6dc804a900 con 0x7f6dd80fc870 2024-02-19T21:15:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:19.477+0000 7f6dde8df700 10 librados: watch_flush enter 2024-02-19T21:15:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:19.477+0000 7f6dde8df700 10 librados: watch_flush exit 2024-02-19T21:15:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:19.477+0000 7f6dde8df700 20 client.50390.objecter shutdown clearing up homeless session... 2024-02-19T21:15:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:19.477+0000 7f6dde8df700 10 client.50390.objecter successfully canceled tick 2024-02-19T21:15:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:19.477+0000 7f6dde8df700 1 -- 172.21.15.163:0/71631 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6dc0063c40 legacy=0x7f6dc0066100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:20.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:19.477+0000 7f6dde8df700 1 -- 172.21.15.163:0/71631 >> v1:172.21.15.163:6790/0 conn(0x7f6dd80fc870 legacy=0x7f6dd8060d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:20.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:19.477+0000 7f6dde8df700 1 -- 172.21.15.163:0/71631 shutdown_connections 2024-02-19T21:15:20.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:19.477+0000 7f6dde8df700 1 -- 172.21.15.163:0/71631 >> 172.21.15.163:0/71631 conn(0x7f6dd80f2470 msgr2=0x7f6dd8106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:20.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:19.477+0000 7f6dde8df700 1 -- 172.21.15.163:0/71631 shutdown_connections 2024-02-19T21:15:20.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:19.477+0000 7f6dde8df700 1 -- 172.21.15.163:0/71631 wait complete. 2024-02-19T21:15:20.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:19.477+0000 7f6dde8df700 1 librados: shutdown 2024-02-19T21:15:20.284 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:15:20.454 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:20.457 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:20.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:20.558 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:20.585 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:20.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:19.302374+0000 mgr.y (mgr.14550) 573 : cluster [DBG] pgmap v1015: 182 pgs: 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:15:20.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:19.462690+0000 mon.a (mon.0) 3319 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:15:20.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:19.473505+0000 mon.a (mon.0) 3320 : audit [INF] from='client.50390 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:15:20.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:19.473570+0000 mon.a (mon.0) 3321 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2024-02-19T21:15:20.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:19.475419+0000 mon.a (mon.0) 3322 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:20.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:19.302374+0000 mgr.y (mgr.14550) 573 : cluster [DBG] pgmap v1015: 182 pgs: 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:15:20.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:19.462690+0000 mon.a (mon.0) 3319 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:15:20.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:19.473505+0000 mon.a (mon.0) 3320 : audit [INF] from='client.50390 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:15:20.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:19.473570+0000 mon.a (mon.0) 3321 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2024-02-19T21:15:20.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:19.475419+0000 mon.a (mon.0) 3322 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:20.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:15:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:15:20.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:19.302374+0000 mgr.y (mgr.14550) 573 : cluster [DBG] pgmap v1015: 182 pgs: 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:15:20.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:19.462690+0000 mon.a (mon.0) 3319 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:15:20.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:19.473505+0000 mon.a (mon.0) 3320 : audit [INF] from='client.50390 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:15:20.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:19.473570+0000 mon.a (mon.0) 3321 : cluster [DBG] osdmap e675: 8 total, 8 up, 8 in 2024-02-19T21:15:20.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:19.475419+0000 mon.a (mon.0) 3322 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:20.962 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:20.965 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:20.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:20.982 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-19T21:15:21.067 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:21.099 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:21.120 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:21.122 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:21.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:21.224 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:15:21.253 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:21.274 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:15:21.276 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:21.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:15:21.378 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:15:21.407 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:21.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:20.476570+0000 mon.a (mon.0) 3323 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:21.486 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:20.476726+0000 mon.a (mon.0) 3324 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2024-02-19T21:15:21.605 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:15:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:21.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:15:21.710 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:21.736 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:20.476570+0000 mon.a (mon.0) 3323 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:21.736 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:20.476726+0000 mon.a (mon.0) 3324 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2024-02-19T21:15:21.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:20.479579+0000 mon.a (mon.0) 3325 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:21.737 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:20.681867+0000 mgr.y (mgr.14550) 574 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:21.737 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:21.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:20.476570+0000 mon.a (mon.0) 3323 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:21.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:20.476726+0000 mon.a (mon.0) 3324 : cluster [DBG] osdmap e676: 8 total, 8 up, 8 in 2024-02-19T21:15:21.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:20.479579+0000 mon.a (mon.0) 3325 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:21.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:20.681867+0000 mgr.y (mgr.14550) 574 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:21.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:20.479579+0000 mon.a (mon.0) 3325 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:21.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:20.681867+0000 mgr.y (mgr.14550) 574 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:21.756 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:21.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:21.860 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:21.888 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:21.906 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:21.909 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:21.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:22.010 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:22.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.010+0000 7fa3b7863700 1 -- 172.21.15.163:0/1753267286 >> v1:172.21.15.163:6790/0 conn(0x7fa3b00fee20 legacy=0x7fa3b0101210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:22.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.011+0000 7fa3b7863700 1 -- 172.21.15.163:0/1753267286 shutdown_connections 2024-02-19T21:15:22.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.011+0000 7fa3b7863700 1 -- 172.21.15.163:0/1753267286 >> 172.21.15.163:0/1753267286 conn(0x7fa3b006cc00 msgr2=0x7fa3b00f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:22.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.011+0000 7fa3b7863700 1 -- 172.21.15.163:0/1753267286 shutdown_connections 2024-02-19T21:15:22.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.011+0000 7fa3b7863700 1 -- 172.21.15.163:0/1753267286 wait complete. 2024-02-19T21:15:22.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.011+0000 7fa3b7863700 1 librados: starting msgr at 2024-02-19T21:15:22.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.011+0000 7fa3b7863700 1 librados: starting objecter 2024-02-19T21:15:22.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.011+0000 7fa3b7863700 1 Processor -- start 2024-02-19T21:15:22.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.011+0000 7fa3b7863700 1 -- start start 2024-02-19T21:15:22.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.011+0000 7fa3b7863700 1 librados: setting wanted keys 2024-02-19T21:15:22.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.011+0000 7fa3b7863700 1 librados: calling monclient init 2024-02-19T21:15:22.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3b7863700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa3b0115170 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3b7863700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa3b0116370 con 0x7fa3b0102930 2024-02-19T21:15:22.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3b7863700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa3b0117570 con 0x7fa3b00fee20 2024-02-19T21:15:22.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3b55ff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fa3b00fbfd0 0x7fa3b00fb3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33802/0 (socket says 172.21.15.163:33802) 2024-02-19T21:15:22.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3b55ff700 1 -- 172.21.15.163:0/2600061902 learned_addr learned my addr 172.21.15.163:0/2600061902 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:22.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 10 client.?.objecter ms_handle_connect 0x7fa3b00fbfd0 2024-02-19T21:15:22.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:22.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 10 client.?.objecter ms_handle_connect 0x7fa3b0102930 2024-02-19T21:15:22.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:22.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 10 client.?.objecter ms_handle_connect 0x7fa3b00fee20 2024-02-19T21:15:22.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:22.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185403208 0 0) 0x7fa3b0115170 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa39c003030 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728894981 0 0) 0x7fa3b0116370 con 0x7fa3b0102930 2024-02-19T21:15:22.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b0115170 con 0x7fa3b0102930 2024-02-19T21:15:22.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832327330 0 0) 0x7fa3b0117570 con 0x7fa3b00fee20 2024-02-19T21:15:22.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b0116370 con 0x7fa3b00fee20 2024-02-19T21:15:22.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3958352884 0 0) 0x7fa39c003030 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3b0117570 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3ac001940 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3214667689 0 0) 0x7fa3b0115170 con 0x7fa3b0102930 2024-02-19T21:15:22.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa39c003030 con 0x7fa3b0102930 2024-02-19T21:15:22.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3a8001cf0 con 0x7fa3b0102930 2024-02-19T21:15:22.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 660230412 0 0) 0x7fa3b0116370 con 0x7fa3b00fee20 2024-02-19T21:15:22.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3b0115170 con 0x7fa3b00fee20 2024-02-19T21:15:22.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3a0000fe0 con 0x7fa3b00fee20 2024-02-19T21:15:22.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2952383639 0 0) 0x7fa3b0117570 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 >> v1:172.21.15.163:6790/0 conn(0x7fa3b00fee20 legacy=0x7fa3b010ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:22.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 >> v1:172.21.15.193:6789/0 conn(0x7fa3b0102930 legacy=0x7fa3b0113830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:22.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b0118770 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3b7863700 1 -- 172.21.15.163:0/2600061902 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b0116560 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3b7863700 10 client.23679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:22.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.012+0000 7fa3b7863700 1 -- 172.21.15.163:0/2600061902 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b01169d0 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.013+0000 7fa3b7863700 1 librados: init done 2024-02-19T21:15:22.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.013+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fa3ac001cd0 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.013+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa3ac005ca0 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.014+0000 7fa3b7863700 1 -- 172.21.15.163:0/2600061902 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa394004f20 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.014+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fa3ac018bb0 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.014+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(677..677 src has 623..677) v4 ==== 8563+0+0 (unknown 2073709768 0 0) 0x7fa3ac078f80 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.014+0000 7fa3a67fc700 10 client.23679.objecter ms_dispatch 0x7fa3b0001550 osd_map(677..677 src has 623..677) v4 2024-02-19T21:15:22.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.014+0000 7fa3a67fc700 3 client.23679.objecter handle_osd_map got epochs [677,677] > 0 2024-02-19T21:15:22.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.014+0000 7fa3a67fc700 3 client.23679.objecter handle_osd_map decoding full epoch 677 2024-02-19T21:15:22.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.015+0000 7fa3a67fc700 20 client.23679.objecter dump_active .. 0 homeless 2024-02-19T21:15:22.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.015+0000 7fa3a67fc700 10 client.23679.objecter ms_handle_connect 0x7fa39c063c20 2024-02-19T21:15:22.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.019+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa3ac04c1a0 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.039 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:22.062 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:22.066 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:22.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:22.080 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:22.084 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:22.167 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:22.194 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:22.225 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.224+0000 7fa3b7863700 1 -- 172.21.15.163:0/2600061902 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7fa3940051d0 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.480+0000 7fa3a67fc700 1 -- 172.21.15.163:0/2600061902 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v678) v1 ==== 113+0+0 (unknown 1174033163 0 0) 0x7fa3ac04a610 con 0x7fa3b00fbfd0 2024-02-19T21:15:22.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.483+0000 7fa3b7863700 10 librados: watch_flush enter 2024-02-19T21:15:22.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.483+0000 7fa3b7863700 10 librados: watch_flush exit 2024-02-19T21:15:22.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.483+0000 7fa3b7863700 20 client.23679.objecter shutdown clearing up homeless session... 2024-02-19T21:15:22.484 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.483+0000 7fa3b7863700 10 client.23679.objecter successfully canceled tick 2024-02-19T21:15:22.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.483+0000 7fa3b7863700 1 -- 172.21.15.163:0/2600061902 >> v1:172.21.15.163:6800/1552514681 conn(0x7fa39c063c20 legacy=0x7fa39c0660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:22.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.483+0000 7fa3b7863700 1 -- 172.21.15.163:0/2600061902 >> v1:172.21.15.163:6789/0 conn(0x7fa3b00fbfd0 legacy=0x7fa3b00fb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:22.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.484+0000 7fa3b7863700 1 -- 172.21.15.163:0/2600061902 shutdown_connections 2024-02-19T21:15:22.485 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.484+0000 7fa3b7863700 1 -- 172.21.15.163:0/2600061902 >> 172.21.15.163:0/2600061902 conn(0x7fa3b006cc00 msgr2=0x7fa3b006cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:22.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.484+0000 7fa3b7863700 1 -- 172.21.15.163:0/2600061902 shutdown_connections 2024-02-19T21:15:22.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.484+0000 7fa3b7863700 1 -- 172.21.15.163:0/2600061902 wait complete. 2024-02-19T21:15:22.486 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:22.484+0000 7fa3b7863700 1 librados: shutdown 2024-02-19T21:15:22.486 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.7 to 1 (10000) 2024-02-19T21:15:22.658 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:21.302803+0000 mgr.y (mgr.14550) 575 : cluster [DBG] pgmap v1018: 214 pgs: 32 creating+peering, 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:22.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:21.480155+0000 mon.a (mon.0) 3326 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-118", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:22.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:21.480205+0000 mon.a (mon.0) 3327 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2024-02-19T21:15:22.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:21.480887+0000 mon.a (mon.0) 3328 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-118"}]: dispatch 2024-02-19T21:15:22.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:22.226473+0000 mon.a (mon.0) 3329 : audit [INF] from='client.? v1:172.21.15.163:0/2600061902' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:15:22.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:22.481565+0000 mon.a (mon.0) 3330 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-118"}]': finished 2024-02-19T21:15:22.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:22.481641+0000 mon.a (mon.0) 3331 : audit [INF] from='client.? v1:172.21.15.163:0/2600061902' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-19T21:15:22.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:22.481681+0000 mon.a (mon.0) 3332 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2024-02-19T21:15:22.659 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:22.484416+0000 mon.a (mon.0) 3333 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-118", "mode": "writeback"}]: dispatch 2024-02-19T21:15:22.659 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:21.302803+0000 mgr.y (mgr.14550) 575 : cluster [DBG] pgmap v1018: 214 pgs: 32 creating+peering, 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:22.659 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:21.480155+0000 mon.a (mon.0) 3326 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-118", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:22.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:21.480205+0000 mon.a (mon.0) 3327 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2024-02-19T21:15:22.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:21.480887+0000 mon.a (mon.0) 3328 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-118"}]: dispatch 2024-02-19T21:15:22.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:22.226473+0000 mon.a (mon.0) 3329 : audit [INF] from='client.? v1:172.21.15.163:0/2600061902' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:15:22.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:22.481565+0000 mon.a (mon.0) 3330 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-118"}]': finished 2024-02-19T21:15:22.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:22.481641+0000 mon.a (mon.0) 3331 : audit [INF] from='client.? v1:172.21.15.163:0/2600061902' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-19T21:15:22.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:22.481681+0000 mon.a (mon.0) 3332 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2024-02-19T21:15:22.660 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:22.484416+0000 mon.a (mon.0) 3333 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-118", "mode": "writeback"}]: dispatch 2024-02-19T21:15:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:21.302803+0000 mgr.y (mgr.14550) 575 : cluster [DBG] pgmap v1018: 214 pgs: 32 creating+peering, 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:21.480155+0000 mon.a (mon.0) 3326 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-118", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:21.480205+0000 mon.a (mon.0) 3327 : cluster [DBG] osdmap e677: 8 total, 8 up, 8 in 2024-02-19T21:15:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:21.480887+0000 mon.a (mon.0) 3328 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-118"}]: dispatch 2024-02-19T21:15:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:22.226473+0000 mon.a (mon.0) 3329 : audit [INF] from='client.? v1:172.21.15.163:0/2600061902' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:15:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:22.481565+0000 mon.a (mon.0) 3330 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-118"}]': finished 2024-02-19T21:15:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:22.481641+0000 mon.a (mon.0) 3331 : audit [INF] from='client.? v1:172.21.15.163:0/2600061902' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-19T21:15:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:22.481681+0000 mon.a (mon.0) 3332 : cluster [DBG] osdmap e678: 8 total, 8 up, 8 in 2024-02-19T21:15:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:22.484416+0000 mon.a (mon.0) 3333 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-118", "mode": "writeback"}]: dispatch 2024-02-19T21:15:23.043 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:23.046 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:23.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:23.147 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:23.179 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:23.201 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:23.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:23.305 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:23.334 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:23.353 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:23.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:23.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:15:23.470 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:15:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:15:23] "GET /metrics HTTP/1.1" 200 191881 "" "Prometheus/2.33.4" 2024-02-19T21:15:23.490 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:23.568 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:15:23.571 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:23.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:15:23.628 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:15:23.672 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:23.700 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:23.719 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:23.721 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:23.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:23.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:23.477077+0000 mon.a (mon.0) 3334 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:23.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:23.481761+0000 mon.a (mon.0) 3335 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-118", "mode": "writeback"}]': finished 2024-02-19T21:15:23.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:23.481817+0000 mon.a (mon.0) 3336 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2024-02-19T21:15:23.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:23.477077+0000 mon.a (mon.0) 3334 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:23.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:23.481761+0000 mon.a (mon.0) 3335 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-118", "mode": "writeback"}]': finished 2024-02-19T21:15:23.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:23.481817+0000 mon.a (mon.0) 3336 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2024-02-19T21:15:23.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:23.477077+0000 mon.a (mon.0) 3334 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:23.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:23.481761+0000 mon.a (mon.0) 3335 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-118", "mode": "writeback"}]': finished 2024-02-19T21:15:23.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:23.481817+0000 mon.a (mon.0) 3336 : cluster [DBG] osdmap e679: 8 total, 8 up, 8 in 2024-02-19T21:15:23.822 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:23.848 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:24.101 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:24.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:24.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:24.205 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:15:24.238 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:24.258 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:15:24.261 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:24.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:15:24.362 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:24.417 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:24.617 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:24.619 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:23.303194+0000 mgr.y (mgr.14550) 576 : cluster [DBG] pgmap v1021: 214 pgs: 32 creating+peering, 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:24.619 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:23.303194+0000 mgr.y (mgr.14550) 576 : cluster [DBG] pgmap v1021: 214 pgs: 32 creating+peering, 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:24.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:24.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:24.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.621+0000 7f95c1da1700 1 -- 172.21.15.163:0/3172443171 >> v1:172.21.15.163:6790/0 conn(0x7f95bc0fee00 legacy=0x7f95bc1011f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:24.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.621+0000 7f95c1da1700 1 -- 172.21.15.163:0/3172443171 shutdown_connections 2024-02-19T21:15:24.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.621+0000 7f95c1da1700 1 -- 172.21.15.163:0/3172443171 >> 172.21.15.163:0/3172443171 conn(0x7f95bc06fdc0 msgr2=0x7f95bc0f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:24.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.621+0000 7f95c1da1700 1 -- 172.21.15.163:0/3172443171 shutdown_connections 2024-02-19T21:15:24.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.621+0000 7f95c1da1700 1 -- 172.21.15.163:0/3172443171 wait complete. 2024-02-19T21:15:24.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.622+0000 7f95c1da1700 1 librados: starting msgr at 2024-02-19T21:15:24.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.622+0000 7f95c1da1700 1 librados: starting objecter 2024-02-19T21:15:24.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.622+0000 7f95c1da1700 1 Processor -- start 2024-02-19T21:15:24.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.622+0000 7f95c1da1700 1 -- start start 2024-02-19T21:15:24.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.622+0000 7f95c1da1700 1 librados: setting wanted keys 2024-02-19T21:15:24.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.622+0000 7f95c1da1700 1 librados: calling monclient init 2024-02-19T21:15:24.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.622+0000 7f95c1da1700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f95bc119530 con 0x7f95bc102910 2024-02-19T21:15:24.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.622+0000 7f95c1da1700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f95bc11a730 con 0x7f95bc0fee00 2024-02-19T21:15:24.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.622+0000 7f95c1da1700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f95bc11b930 con 0x7f95bc0fbfb0 2024-02-19T21:15:24.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.622+0000 7f95bb7fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f95bc0fbfb0 0x7f95bc0fb3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:43098/0 (socket says 172.21.15.163:43098) 2024-02-19T21:15:24.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.622+0000 7f95bb7fe700 1 -- 172.21.15.163:0/744031373 learned_addr learned my addr 172.21.15.163:0/744031373 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:24.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.622+0000 7f95b8ff9700 10 client.?.objecter ms_handle_connect 0x7f95bc0fbfb0 2024-02-19T21:15:24.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.622+0000 7f95b8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:24.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.622+0000 7f95b8ff9700 10 client.?.objecter ms_handle_connect 0x7f95bc102910 2024-02-19T21:15:24.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.622+0000 7f95b8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:24.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358033674 0 0) 0x7f95bc119530 con 0x7f95bc102910 2024-02-19T21:15:24.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95a4003030 con 0x7f95bc102910 2024-02-19T21:15:24.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 10 client.?.objecter ms_handle_connect 0x7f95bc0fee00 2024-02-19T21:15:24.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:24.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1997028150 0 0) 0x7f95bc11b930 con 0x7f95bc0fbfb0 2024-02-19T21:15:24.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95bc119530 con 0x7f95bc0fbfb0 2024-02-19T21:15:24.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797634446 0 0) 0x7f95bc11a730 con 0x7f95bc0fee00 2024-02-19T21:15:24.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95bc11b930 con 0x7f95bc0fee00 2024-02-19T21:15:24.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 891952748 0 0) 0x7f95a4003030 con 0x7f95bc102910 2024-02-19T21:15:24.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95bc11a730 con 0x7f95bc102910 2024-02-19T21:15:24.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f95b4001c40 con 0x7f95bc102910 2024-02-19T21:15:24.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3739684316 0 0) 0x7f95bc119530 con 0x7f95bc0fbfb0 2024-02-19T21:15:24.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95a4003030 con 0x7f95bc0fbfb0 2024-02-19T21:15:24.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f95ac0019d0 con 0x7f95bc0fbfb0 2024-02-19T21:15:24.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1147844854 0 0) 0x7f95bc11b930 con 0x7f95bc0fee00 2024-02-19T21:15:24.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95bc119530 con 0x7f95bc0fee00 2024-02-19T21:15:24.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f95b00020b0 con 0x7f95bc0fee00 2024-02-19T21:15:24.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 416375107 0 0) 0x7f95bc11a730 con 0x7f95bc102910 2024-02-19T21:15:24.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 >> v1:172.21.15.163:6790/0 conn(0x7f95bc0fbfb0 legacy=0x7f95bc0fb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:24.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 >> v1:172.21.15.193:6789/0 conn(0x7f95bc0fee00 legacy=0x7f95bc114360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:24.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc11cb30 con 0x7f95bc102910 2024-02-19T21:15:24.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95bc119720 con 0x7f95bc102910 2024-02-19T21:15:24.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95c1da1700 10 client.23685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:24.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95bc119b90 con 0x7f95bc102910 2024-02-19T21:15:24.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95c1da1700 1 librados: init done 2024-02-19T21:15:24.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f95b4002300 con 0x7f95bc102910 2024-02-19T21:15:24.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.623+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f95b4003620 con 0x7f95bc102910 2024-02-19T21:15:24.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.624+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f95a8004f00 con 0x7f95bc102910 2024-02-19T21:15:24.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f95b4001e60 con 0x7f95bc102910 2024-02-19T21:15:24.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(679..679 src has 623..679) v4 ==== 8563+0+0 (unknown 3768373017 0 0) 0x7f95b4076850 con 0x7f95bc102910 2024-02-19T21:15:24.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 osd_map(679..679 src has 623..679) v4 2024-02-19T21:15:24.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95b8ff9700 3 client.23685.objecter handle_osd_map got epochs [679,679] > 0 2024-02-19T21:15:24.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95b8ff9700 3 client.23685.objecter handle_osd_map decoding full epoch 679 2024-02-19T21:15:24.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95b8ff9700 20 client.23685.objecter dump_active .. 0 homeless 2024-02-19T21:15:24.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v679) v1 ==== 54+0+16 (unknown 3864433600 0 2420000192) 0x7f95b4076cc0 con 0x7f95bc102910 2024-02-19T21:15:24.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95b8ff9700 10 client.23685.objecter ms_handle_connect 0x7f95a4063cf0 2024-02-19T21:15:24.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95c1da1700 10 client.23685.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:24.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 1 2024-02-19T21:15:24.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a80051a0 osd=0 3 2024-02-19T21:15:24.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a80051a0 osd=0 3 2024-02-19T21:15:24.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a80051a0 osd=0 3 2024-02-19T21:15:24.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 1 2024-02-19T21:15:24.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a80051a0 osd=0 3 2024-02-19T21:15:24.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 0 1 2024-02-19T21:15:24.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a80051a0 osd=0 4 2024-02-19T21:15:24.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95c1da1700 10 client.23685.objecter _send_command 1 2024-02-19T21:15:24.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.625+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95a8004950 con 0x7f95a8004530 2024-02-19T21:15:24.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.626+0000 7f95b8ff9700 10 client.23685.objecter ms_handle_connect 0x7f95a8004530 2024-02-19T21:15:24.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.626+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95a8004950 con 0x7f95a8004530 2024-02-19T21:15:24.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.626+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:15:24.630 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.626+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 1 = system:0 2024-02-19T21:15:24.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.626+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a80051a0 osd=0 4 2024-02-19T21:15:24.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.626+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 0 1 2024-02-19T21:15:24.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.641+0000 7f95c1da1700 10 client.23685.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:24.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.641+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 2 2024-02-19T21:15:24.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.641+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a80051a0 osd=0 3 2024-02-19T21:15:24.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.641+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a80051a0 osd=0 3 2024-02-19T21:15:24.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.641+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a80051a0 osd=0 3 2024-02-19T21:15:24.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.641+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 2 2024-02-19T21:15:24.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.641+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a80051a0 osd=0 3 2024-02-19T21:15:24.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.641+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 0 2 2024-02-19T21:15:24.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.641+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a80051a0 osd=0 4 2024-02-19T21:15:24.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.641+0000 7f95c1da1700 10 client.23685.objecter _send_command 2 2024-02-19T21:15:24.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.641+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95a8004d30 con 0x7f95a8004530 2024-02-19T21:15:24.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95a8004d30 con 0x7f95a8004530 2024-02-19T21:15:24.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:24.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a80051a0 osd=0 4 2024-02-19T21:15:24.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 0 2 2024-02-19T21:15:24.644 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95c1da1700 10 client.23685.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 3 2024-02-19T21:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8003fe0 osd=1 3 2024-02-19T21:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8003fe0 osd=1 3 2024-02-19T21:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8003fe0 osd=1 3 2024-02-19T21:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 3 2024-02-19T21:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a8003fe0 osd=1 3 2024-02-19T21:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 1 3 2024-02-19T21:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8003fe0 osd=1 4 2024-02-19T21:15:24.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95c1da1700 10 client.23685.objecter _send_command 3 2024-02-19T21:15:24.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95a800c000 con 0x7f95a8004150 2024-02-19T21:15:24.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.643+0000 7f95b8ff9700 10 client.23685.objecter ms_handle_connect 0x7f95a8004150 2024-02-19T21:15:24.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.644+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95a800c000 con 0x7f95a8004150 2024-02-19T21:15:24.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.644+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:15:24.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.644+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 3 = system:0 2024-02-19T21:15:24.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.644+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a8003fe0 osd=1 4 2024-02-19T21:15:24.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.644+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 1 3 2024-02-19T21:15:24.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.658+0000 7f95c1da1700 10 client.23685.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:24.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.658+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 4 2024-02-19T21:15:24.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.658+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8003fe0 osd=1 3 2024-02-19T21:15:24.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.658+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8003fe0 osd=1 3 2024-02-19T21:15:24.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.658+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8003fe0 osd=1 3 2024-02-19T21:15:24.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.658+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 4 2024-02-19T21:15:24.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.658+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a8003fe0 osd=1 3 2024-02-19T21:15:24.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.658+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 1 4 2024-02-19T21:15:24.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.658+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8003fe0 osd=1 4 2024-02-19T21:15:24.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.658+0000 7f95c1da1700 10 client.23685.objecter _send_command 4 2024-02-19T21:15:24.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.658+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95a80018b0 con 0x7f95a8004150 2024-02-19T21:15:24.660 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95a80018b0 con 0x7f95a8004150 2024-02-19T21:15:24.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:24.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a8003fe0 osd=1 4 2024-02-19T21:15:24.661 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 1 4 2024-02-19T21:15:24.661 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:15:24.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95c1da1700 10 client.23685.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:24.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 5 2024-02-19T21:15:24.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8003630 osd=2 3 2024-02-19T21:15:24.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8003630 osd=2 3 2024-02-19T21:15:24.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8003630 osd=2 3 2024-02-19T21:15:24.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 5 2024-02-19T21:15:24.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a8003630 osd=2 3 2024-02-19T21:15:24.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 2 5 2024-02-19T21:15:24.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8003630 osd=2 4 2024-02-19T21:15:24.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95c1da1700 10 client.23685.objecter _send_command 5 2024-02-19T21:15:24.662 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95a800de90 con 0x7f95a800d2f0 2024-02-19T21:15:24.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.660+0000 7f95b8ff9700 10 client.23685.objecter ms_handle_connect 0x7f95a800d2f0 2024-02-19T21:15:24.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.661+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95a800de90 con 0x7f95a800d2f0 2024-02-19T21:15:24.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.661+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:15:24.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.661+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 5 = system:0 2024-02-19T21:15:24.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.661+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a8003630 osd=2 4 2024-02-19T21:15:24.663 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.661+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 2 5 2024-02-19T21:15:24.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.675+0000 7f95c1da1700 10 client.23685.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:24.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.675+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 6 2024-02-19T21:15:24.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.675+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8003630 osd=2 3 2024-02-19T21:15:24.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.675+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8003630 osd=2 3 2024-02-19T21:15:24.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.675+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8003630 osd=2 3 2024-02-19T21:15:24.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.675+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 6 2024-02-19T21:15:24.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.675+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a8003630 osd=2 3 2024-02-19T21:15:24.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.675+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 2 6 2024-02-19T21:15:24.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.675+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8003630 osd=2 4 2024-02-19T21:15:24.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.675+0000 7f95c1da1700 10 client.23685.objecter _send_command 6 2024-02-19T21:15:24.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.675+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95a80030a0 con 0x7f95a800d2f0 2024-02-19T21:15:24.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95a80030a0 con 0x7f95a800d2f0 2024-02-19T21:15:24.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:24.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a8003630 osd=2 4 2024-02-19T21:15:24.677 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 2 6 2024-02-19T21:15:24.678 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:15:24.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95c1da1700 10 client.23685.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:24.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 7 2024-02-19T21:15:24.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8011970 osd=3 3 2024-02-19T21:15:24.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8011970 osd=3 3 2024-02-19T21:15:24.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8011970 osd=3 3 2024-02-19T21:15:24.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 7 2024-02-19T21:15:24.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a8011970 osd=3 3 2024-02-19T21:15:24.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 3 7 2024-02-19T21:15:24.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8011970 osd=3 4 2024-02-19T21:15:24.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95c1da1700 10 client.23685.objecter _send_command 7 2024-02-19T21:15:24.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95a8015b00 con 0x7f95a8011ff0 2024-02-19T21:15:24.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.677+0000 7f95b8ff9700 10 client.23685.objecter ms_handle_connect 0x7f95a8011ff0 2024-02-19T21:15:24.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.678+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95a8015b00 con 0x7f95a8011ff0 2024-02-19T21:15:24.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.678+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:15:24.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.678+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 7 = system:0 2024-02-19T21:15:24.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.678+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a8011970 osd=3 4 2024-02-19T21:15:24.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.678+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 3 7 2024-02-19T21:15:24.692 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.692+0000 7f95c1da1700 10 client.23685.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:24.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.692+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 8 2024-02-19T21:15:24.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.692+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8011970 osd=3 3 2024-02-19T21:15:24.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.692+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8011970 osd=3 3 2024-02-19T21:15:24.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.692+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8011970 osd=3 3 2024-02-19T21:15:24.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.692+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 8 2024-02-19T21:15:24.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.692+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a8011970 osd=3 3 2024-02-19T21:15:24.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.692+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 3 8 2024-02-19T21:15:24.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.692+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8011970 osd=3 4 2024-02-19T21:15:24.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.692+0000 7f95c1da1700 10 client.23685.objecter _send_command 8 2024-02-19T21:15:24.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.692+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95a800e270 con 0x7f95a8011ff0 2024-02-19T21:15:24.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95a800e270 con 0x7f95a8011ff0 2024-02-19T21:15:24.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:24.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a8011970 osd=3 4 2024-02-19T21:15:24.694 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 3 8 2024-02-19T21:15:24.695 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:15:24.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95c1da1700 10 client.23685.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:24.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 9 2024-02-19T21:15:24.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a800e030 osd=4 3 2024-02-19T21:15:24.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a800e030 osd=4 3 2024-02-19T21:15:24.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a800e030 osd=4 3 2024-02-19T21:15:24.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 9 2024-02-19T21:15:24.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a800e030 osd=4 3 2024-02-19T21:15:24.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 4 9 2024-02-19T21:15:24.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a800e030 osd=4 4 2024-02-19T21:15:24.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95c1da1700 10 client.23685.objecter _send_command 9 2024-02-19T21:15:24.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95a800e1d0 con 0x7f95a800ce50 2024-02-19T21:15:24.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.694+0000 7f95b8ff9700 10 client.23685.objecter ms_handle_connect 0x7f95a800ce50 2024-02-19T21:15:24.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.695+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95a800e1d0 con 0x7f95a800ce50 2024-02-19T21:15:24.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.695+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:15:24.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.695+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 9 = system:0 2024-02-19T21:15:24.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.695+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a800e030 osd=4 4 2024-02-19T21:15:24.697 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.695+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 4 9 2024-02-19T21:15:24.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.710+0000 7f95c1da1700 10 client.23685.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:24.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.710+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 10 2024-02-19T21:15:24.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.710+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a800e030 osd=4 3 2024-02-19T21:15:24.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.710+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a800e030 osd=4 3 2024-02-19T21:15:24.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.710+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a800e030 osd=4 3 2024-02-19T21:15:24.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.710+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 10 2024-02-19T21:15:24.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.710+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a800e030 osd=4 3 2024-02-19T21:15:24.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.710+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 4 10 2024-02-19T21:15:24.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.710+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a800e030 osd=4 4 2024-02-19T21:15:24.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.710+0000 7f95c1da1700 10 client.23685.objecter _send_command 10 2024-02-19T21:15:24.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.710+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95a800e1d0 con 0x7f95a800ce50 2024-02-19T21:15:24.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95a800e1d0 con 0x7f95a800ce50 2024-02-19T21:15:24.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:24.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a800e030 osd=4 4 2024-02-19T21:15:24.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 4 10 2024-02-19T21:15:24.713 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:15:24.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95c1da1700 10 client.23685.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:24.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 11 2024-02-19T21:15:24.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a800e1d0 osd=5 3 2024-02-19T21:15:24.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a800e1d0 osd=5 3 2024-02-19T21:15:24.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a800e1d0 osd=5 3 2024-02-19T21:15:24.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 11 2024-02-19T21:15:24.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a800e1d0 osd=5 3 2024-02-19T21:15:24.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 5 11 2024-02-19T21:15:24.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a800e1d0 osd=5 4 2024-02-19T21:15:24.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95c1da1700 10 client.23685.objecter _send_command 11 2024-02-19T21:15:24.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95a801c650 con 0x7f95a801a940 2024-02-19T21:15:24.714 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.712+0000 7f95b8ff9700 10 client.23685.objecter ms_handle_connect 0x7f95a801a940 2024-02-19T21:15:24.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.713+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95a801c650 con 0x7f95a801a940 2024-02-19T21:15:24.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.713+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:15:24.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.713+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 11 = system:0 2024-02-19T21:15:24.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.713+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a800e1d0 osd=5 4 2024-02-19T21:15:24.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.713+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 5 11 2024-02-19T21:15:24.722 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:15:24.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.727+0000 7f95c1da1700 10 client.23685.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:24.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.727+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 12 2024-02-19T21:15:24.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.727+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a800e1d0 osd=5 3 2024-02-19T21:15:24.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.727+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a800e1d0 osd=5 3 2024-02-19T21:15:24.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.727+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a800e1d0 osd=5 3 2024-02-19T21:15:24.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.727+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 12 2024-02-19T21:15:24.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.727+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a800e1d0 osd=5 3 2024-02-19T21:15:24.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.727+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 5 12 2024-02-19T21:15:24.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.727+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a800e1d0 osd=5 4 2024-02-19T21:15:24.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.727+0000 7f95c1da1700 10 client.23685.objecter _send_command 12 2024-02-19T21:15:24.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.727+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95a801c650 con 0x7f95a801a940 2024-02-19T21:15:24.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.729+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95a801c650 con 0x7f95a801a940 2024-02-19T21:15:24.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.729+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:24.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.729+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.729+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a800e1d0 osd=5 4 2024-02-19T21:15:24.730 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.729+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 5 12 2024-02-19T21:15:24.730 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:15:24.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.730+0000 7f95c1da1700 10 client.23685.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:24.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.730+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 13 2024-02-19T21:15:24.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.730+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a801c650 osd=6 3 2024-02-19T21:15:24.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.730+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a801c650 osd=6 3 2024-02-19T21:15:24.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.730+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a801c650 osd=6 3 2024-02-19T21:15:24.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.730+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 13 2024-02-19T21:15:24.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.730+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a801c650 osd=6 3 2024-02-19T21:15:24.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.730+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 6 13 2024-02-19T21:15:24.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.730+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a801c650 osd=6 4 2024-02-19T21:15:24.731 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.730+0000 7f95c1da1700 10 client.23685.objecter _send_command 13 2024-02-19T21:15:24.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.730+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95a8022c10 con 0x7f95a801f100 2024-02-19T21:15:24.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.730+0000 7f95b8ff9700 10 client.23685.objecter ms_handle_connect 0x7f95a801f100 2024-02-19T21:15:24.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.731+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95a8022c10 con 0x7f95a801f100 2024-02-19T21:15:24.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.731+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:15:24.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.731+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 13 = system:0 2024-02-19T21:15:24.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.731+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a801c650 osd=6 4 2024-02-19T21:15:24.732 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.731+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 6 13 2024-02-19T21:15:24.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.746+0000 7f95c1da1700 10 client.23685.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:24.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.746+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 14 2024-02-19T21:15:24.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.746+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a801c650 osd=6 3 2024-02-19T21:15:24.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.746+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a801c650 osd=6 3 2024-02-19T21:15:24.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.746+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a801c650 osd=6 3 2024-02-19T21:15:24.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.746+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 14 2024-02-19T21:15:24.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.746+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a801c650 osd=6 3 2024-02-19T21:15:24.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.746+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 6 14 2024-02-19T21:15:24.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.746+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a801c650 osd=6 4 2024-02-19T21:15:24.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.746+0000 7f95c1da1700 10 client.23685.objecter _send_command 14 2024-02-19T21:15:24.747 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.746+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95a8022c10 con 0x7f95a801f100 2024-02-19T21:15:24.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95a8022c10 con 0x7f95a801f100 2024-02-19T21:15:24.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:24.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a801c650 osd=6 4 2024-02-19T21:15:24.749 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 6 14 2024-02-19T21:15:24.749 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:15:24.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95c1da1700 10 client.23685.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:24.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 15 2024-02-19T21:15:24.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8022c10 osd=7 3 2024-02-19T21:15:24.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8022c10 osd=7 3 2024-02-19T21:15:24.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8022c10 osd=7 3 2024-02-19T21:15:24.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 15 2024-02-19T21:15:24.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a8022c10 osd=7 3 2024-02-19T21:15:24.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 7 15 2024-02-19T21:15:24.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8022c10 osd=7 4 2024-02-19T21:15:24.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95c1da1700 10 client.23685.objecter _send_command 15 2024-02-19T21:15:24.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.748+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95a8026f80 con 0x7f95a80232d0 2024-02-19T21:15:24.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.749+0000 7f95b8ff9700 10 client.23685.objecter ms_handle_connect 0x7f95a80232d0 2024-02-19T21:15:24.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.749+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f95a8026f80 con 0x7f95a80232d0 2024-02-19T21:15:24.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.749+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:15:24.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.749+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 15 = system:0 2024-02-19T21:15:24.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.749+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a8022c10 osd=7 4 2024-02-19T21:15:24.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.749+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 7 15 2024-02-19T21:15:24.751 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:23.303194+0000 mgr.y (mgr.14550) 576 : cluster [DBG] pgmap v1021: 214 pgs: 32 creating+peering, 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:24.752 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:24.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.764+0000 7f95c1da1700 10 client.23685.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:24.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.764+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign -1 16 2024-02-19T21:15:24.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.764+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8022c10 osd=7 3 2024-02-19T21:15:24.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.764+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8022c10 osd=7 3 2024-02-19T21:15:24.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.764+0000 7f95c1da1700 20 client.23685.objecter _get_session s=0x7f95a8022c10 osd=7 3 2024-02-19T21:15:24.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.764+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_remove -1 16 2024-02-19T21:15:24.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.764+0000 7f95c1da1700 20 client.23685.objecter get_session s=0x7f95a8022c10 osd=7 3 2024-02-19T21:15:24.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.764+0000 7f95c1da1700 15 client.23685.objecter _session_command_op_assign 7 16 2024-02-19T21:15:24.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.764+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8022c10 osd=7 4 2024-02-19T21:15:24.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.764+0000 7f95c1da1700 10 client.23685.objecter _send_command 16 2024-02-19T21:15:24.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.764+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f95a8026f80 con 0x7f95a80232d0 2024-02-19T21:15:24.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.766+0000 7f95b8ff9700 1 -- 172.21.15.163:0/744031373 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f95a8026f80 con 0x7f95a80232d0 2024-02-19T21:15:24.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.766+0000 7f95b8ff9700 10 client.23685.objecter ms_dispatch 0x7f95bc001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:24.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.766+0000 7f95b8ff9700 10 client.23685.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.766+0000 7f95b8ff9700 20 client.23685.objecter put_session s=0x7f95a8022c10 osd=7 4 2024-02-19T21:15:24.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.766+0000 7f95b8ff9700 15 client.23685.objecter _session_command_op_remove 7 16 2024-02-19T21:15:24.768 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:15:24.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 10 librados: watch_flush enter 2024-02-19T21:15:24.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 10 librados: watch_flush exit 2024-02-19T21:15:24.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 10 client.23685.objecter close_session for osd.0 2024-02-19T21:15:24.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 >> v1:172.21.15.163:6801/1559418115 conn(0x7f95a8004530 legacy=0x7f95a8002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:24.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a80051a0 osd=0 1 2024-02-19T21:15:24.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 10 client.23685.objecter close_session for osd.1 2024-02-19T21:15:24.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 >> v1:172.21.15.163:6805/3125047432 conn(0x7f95a8004150 legacy=0x7f95a800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:24.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8003fe0 osd=1 1 2024-02-19T21:15:24.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 10 client.23685.objecter close_session for osd.2 2024-02-19T21:15:24.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 >> v1:172.21.15.163:6809/1563158487 conn(0x7f95a800d2f0 legacy=0x7f95a800d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:24.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8003630 osd=2 1 2024-02-19T21:15:24.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 10 client.23685.objecter close_session for osd.3 2024-02-19T21:15:24.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 >> v1:172.21.15.163:6813/2112279769 conn(0x7f95a8011ff0 legacy=0x7f95a80143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:24.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95bbfff700 1 -- 172.21.15.163:0/744031373 reap_dead start 2024-02-19T21:15:24.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8011970 osd=3 1 2024-02-19T21:15:24.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 10 client.23685.objecter close_session for osd.4 2024-02-19T21:15:24.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 >> v1:172.21.15.193:6800/3129716965 conn(0x7f95a800ce50 legacy=0x7f95a8016c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:24.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95bbfff700 1 -- 172.21.15.163:0/744031373 reap_dead start 2024-02-19T21:15:24.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a800e030 osd=4 1 2024-02-19T21:15:24.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 10 client.23685.objecter close_session for osd.5 2024-02-19T21:15:24.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 >> v1:172.21.15.193:6804/805161991 conn(0x7f95a801a940 legacy=0x7f95a801ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:24.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a800e1d0 osd=5 1 2024-02-19T21:15:24.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 10 client.23685.objecter close_session for osd.6 2024-02-19T21:15:24.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 >> v1:172.21.15.193:6808/787604615 conn(0x7f95a801f100 legacy=0x7f95a80214f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:24.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a801c650 osd=6 1 2024-02-19T21:15:24.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 10 client.23685.objecter close_session for osd.7 2024-02-19T21:15:24.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 >> v1:172.21.15.193:6812/157927432 conn(0x7f95a80232d0 legacy=0x7f95a8025700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:24.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 20 client.23685.objecter put_session s=0x7f95a8022c10 osd=7 1 2024-02-19T21:15:24.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 20 client.23685.objecter shutdown clearing up homeless session... 2024-02-19T21:15:24.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 10 client.23685.objecter successfully canceled tick 2024-02-19T21:15:24.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 >> v1:172.21.15.163:6800/1552514681 conn(0x7f95a4063cf0 legacy=0x7f95a40661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:24.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 >> v1:172.21.15.163:6789/0 conn(0x7f95bc102910 legacy=0x7f95bc117bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:24.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95bbfff700 1 -- 172.21.15.163:0/744031373 reap_dead start 2024-02-19T21:15:24.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 shutdown_connections 2024-02-19T21:15:24.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 >> 172.21.15.163:0/744031373 conn(0x7f95bc06fdc0 msgr2=0x7f95bc0f4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:24.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 shutdown_connections 2024-02-19T21:15:24.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 1 -- 172.21.15.163:0/744031373 wait complete. 2024-02-19T21:15:24.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:24.767+0000 7f95c1da1700 1 librados: shutdown 2024-02-19T21:15:24.772 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.772 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.772 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.772 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.772 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.772 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.772 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.772 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:15:24.773 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:15:24.774 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:24.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:15:24.876 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:24.904 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:25.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 3 1 2024-02-19T21:15:25.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:24.626492+0000 mon.a (mon.0) 3337 : audit [DBG] from='client.? v1:172.21.15.163:0/744031373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:25.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:24.626492+0000 mon.a (mon.0) 3337 : audit [DBG] from='client.? v1:172.21.15.163:0/744031373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:25.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:24.626492+0000 mon.a (mon.0) 3337 : audit [DBG] from='client.? v1:172.21.15.163:0/744031373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:25.853 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:25.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:25.957 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:15:25.989 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:26.010 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:15:26.012 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:26.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:15:26.114 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:26.140 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:26.344 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:26.347 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:26.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:26.357 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:15:26.357 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:15:26.448 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:15:26.481 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:26.502 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:15:26.505 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:26.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:15:26.606 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:26.637 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:26.658 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:26.661 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:26.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:26.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:25.304164+0000 mgr.y (mgr.14550) 577 : cluster [DBG] pgmap v1023: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 424 B/s wr, 1 op/s 2024-02-19T21:15:26.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:25.304164+0000 mgr.y (mgr.14550) 577 : cluster [DBG] pgmap v1023: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 424 B/s wr, 1 op/s 2024-02-19T21:15:26.762 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:26.793 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:26.813 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:26.815 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:26.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:26.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:25.304164+0000 mgr.y (mgr.14550) 577 : cluster [DBG] pgmap v1023: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 424 B/s wr, 1 op/s 2024-02-19T21:15:26.916 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:26.946 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:26.967 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:26.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:27.056 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:27.057 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:27.071 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:15:27.099 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:27.548 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:15:27.550 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:27.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:15:27.559 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-19T21:15:27.651 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:27.678 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:28.071 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:15:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:15:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:15:28.792 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:15:28.820 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:27.304824+0000 mgr.y (mgr.14550) 578 : cluster [DBG] pgmap v1024: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-02-19T21:15:28.820 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:27.304824+0000 mgr.y (mgr.14550) 578 : cluster [DBG] pgmap v1024: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-02-19T21:15:28.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:27.304824+0000 mgr.y (mgr.14550) 578 : cluster [DBG] pgmap v1024: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-02-19T21:15:29.236 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:29.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:29.340 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:15:29.371 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:29.403 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:15:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:29.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:15:29.508 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:29.536 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:29.554 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:29.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:29.658 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:29.686 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:29.758 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:29.771 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:28.542800+0000 mon.a (mon.0) 3338 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:29.772 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:28.542800+0000 mon.a (mon.0) 3338 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:29.772 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:29.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:29.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:28.542800+0000 mon.a (mon.0) 3338 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:29.874 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:29.901 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:30.383 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:30.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.395+0000 7f9027b3d700 1 -- 172.21.15.163:0/1253383696 >> v1:172.21.15.163:6790/0 conn(0x7f9020102de0 legacy=0x7f90201051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:30.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.395+0000 7f9027b3d700 1 -- 172.21.15.163:0/1253383696 shutdown_connections 2024-02-19T21:15:30.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.395+0000 7f9027b3d700 1 -- 172.21.15.163:0/1253383696 >> 172.21.15.163:0/1253383696 conn(0x7f902006cc00 msgr2=0x7f90200f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:30.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.395+0000 7f9027b3d700 1 -- 172.21.15.163:0/1253383696 shutdown_connections 2024-02-19T21:15:30.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.395+0000 7f9027b3d700 1 -- 172.21.15.163:0/1253383696 wait complete. 2024-02-19T21:15:30.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.395+0000 7f9027b3d700 1 librados: starting msgr at 2024-02-19T21:15:30.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.395+0000 7f9027b3d700 1 librados: starting objecter 2024-02-19T21:15:30.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.395+0000 7f9027b3d700 1 Processor -- start 2024-02-19T21:15:30.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f9027b3d700 1 -- start start 2024-02-19T21:15:30.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f9027b3d700 1 librados: setting wanted keys 2024-02-19T21:15:30.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f9027b3d700 1 librados: calling monclient init 2024-02-19T21:15:30.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f9027b3d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f902011dc50 con 0x7f9020102de0 2024-02-19T21:15:30.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f9027b3d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f902011edf0 con 0x7f90200fdb70 2024-02-19T21:15:30.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f9027b3d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f902011ff90 con 0x7f90201068f0 2024-02-19T21:15:30.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f90250d8700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f9020102de0 0x7f90200fa690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44718/0 (socket says 172.21.15.163:44718) 2024-02-19T21:15:30.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f90250d8700 1 -- 172.21.15.163:0/249111862 learned_addr learned my addr 172.21.15.163:0/249111862 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:30.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f9016ffd700 10 client.?.objecter ms_handle_connect 0x7f9020102de0 2024-02-19T21:15:30.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f9016ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:30.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f9016ffd700 10 client.?.objecter ms_handle_connect 0x7f90201068f0 2024-02-19T21:15:30.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f9016ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:30.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86126892 0 0) 0x7f902011dc50 con 0x7f9020102de0 2024-02-19T21:15:30.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f900c002220 con 0x7f9020102de0 2024-02-19T21:15:30.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f9016ffd700 10 client.?.objecter ms_handle_connect 0x7f90200fdb70 2024-02-19T21:15:30.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f9016ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:30.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.396+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628462177 0 0) 0x7f902011ff90 con 0x7f90201068f0 2024-02-19T21:15:30.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f902011dc50 con 0x7f90201068f0 2024-02-19T21:15:30.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561493563 0 0) 0x7f902011edf0 con 0x7f90200fdb70 2024-02-19T21:15:30.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f902011ff90 con 0x7f90200fdb70 2024-02-19T21:15:30.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3733842543 0 0) 0x7f900c002220 con 0x7f9020102de0 2024-02-19T21:15:30.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f902011edf0 con 0x7f9020102de0 2024-02-19T21:15:30.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3239258137 0 0) 0x7f902011dc50 con 0x7f90201068f0 2024-02-19T21:15:30.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f900c002220 con 0x7f90201068f0 2024-02-19T21:15:30.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9010000fe0 con 0x7f9020102de0 2024-02-19T21:15:30.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f90180019f0 con 0x7f90201068f0 2024-02-19T21:15:30.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2660059871 0 0) 0x7f902011edf0 con 0x7f9020102de0 2024-02-19T21:15:30.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 >> v1:172.21.15.163:6790/0 conn(0x7f90201068f0 legacy=0x7f902011c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:30.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 >> v1:172.21.15.193:6789/0 conn(0x7f90200fdb70 legacy=0x7f90200f9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:30.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9020121170 con 0x7f9020102de0 2024-02-19T21:15:30.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9027b3d700 1 -- 172.21.15.163:0/249111862 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9020120180 con 0x7f9020102de0 2024-02-19T21:15:30.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9027b3d700 10 client.23691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:30.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9027b3d700 1 -- 172.21.15.163:0/249111862 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90201205f0 con 0x7f9020102de0 2024-02-19T21:15:30.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9027b3d700 1 librados: init done 2024-02-19T21:15:30.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f90100028d0 con 0x7f9020102de0 2024-02-19T21:15:30.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.397+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9010003810 con 0x7f9020102de0 2024-02-19T21:15:30.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.398+0000 7f9027b3d700 1 -- 172.21.15.163:0/249111862 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9004004f20 con 0x7f9020102de0 2024-02-19T21:15:30.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.398+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9010003b00 con 0x7f9020102de0 2024-02-19T21:15:30.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.399+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(680..680 src has 623..680) v4 ==== 8563+0+0 (unknown 3771508141 0 0) 0x7f90100792e0 con 0x7f9020102de0 2024-02-19T21:15:30.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.399+0000 7f9016ffd700 10 client.23691.objecter ms_dispatch 0x7f9020001550 osd_map(680..680 src has 623..680) v4 2024-02-19T21:15:30.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.399+0000 7f9016ffd700 3 client.23691.objecter handle_osd_map got epochs [680,680] > 0 2024-02-19T21:15:30.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.399+0000 7f9016ffd700 3 client.23691.objecter handle_osd_map decoding full epoch 680 2024-02-19T21:15:30.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.399+0000 7f9016ffd700 20 client.23691.objecter dump_active .. 0 homeless 2024-02-19T21:15:30.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.399+0000 7f9016ffd700 10 client.23691.objecter ms_handle_connect 0x7f900c063a20 2024-02-19T21:15:30.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.404+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f901004c560 con 0x7f9020102de0 2024-02-19T21:15:30.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:30.616+0000 7f9027b3d700 1 -- 172.21.15.163:0/249111862 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f90040051d0 con 0x7f9020102de0 2024-02-19T21:15:30.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:29.305182+0000 mgr.y (mgr.14550) 579 : cluster [DBG] pgmap v1025: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 784 B/s rd, 261 B/s wr, 1 op/s 2024-02-19T21:15:30.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:29.512726+0000 mon.a (mon.0) 3339 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:30.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:29.512779+0000 mon.a (mon.0) 3340 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2024-02-19T21:15:30.766 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:29.514207+0000 mon.a (mon.0) 3341 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-118"}]: dispatch 2024-02-19T21:15:30.766 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:29.305182+0000 mgr.y (mgr.14550) 579 : cluster [DBG] pgmap v1025: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 784 B/s rd, 261 B/s wr, 1 op/s 2024-02-19T21:15:30.766 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:29.512726+0000 mon.a (mon.0) 3339 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:30.766 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:29.512779+0000 mon.a (mon.0) 3340 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2024-02-19T21:15:30.766 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:29.514207+0000 mon.a (mon.0) 3341 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-118"}]: dispatch 2024-02-19T21:15:30.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:15:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:15:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:29.305182+0000 mgr.y (mgr.14550) 579 : cluster [DBG] pgmap v1025: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 784 B/s rd, 261 B/s wr, 1 op/s 2024-02-19T21:15:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:29.512726+0000 mon.a (mon.0) 3339 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:29.512779+0000 mon.a (mon.0) 3340 : cluster [DBG] osdmap e680: 8 total, 8 up, 8 in 2024-02-19T21:15:30.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:29.514207+0000 mon.a (mon.0) 3341 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-118"}]: dispatch 2024-02-19T21:15:30.851 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:30.853 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:30.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:30.955 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:30.985 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:31.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.526+0000 7f9016ffd700 1 -- 172.21.15.163:0/249111862 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v682) v1 ==== 91+0+0 (unknown 1926857701 0 0) 0x7f901004a9d0 con 0x7f9020102de0 2024-02-19T21:15:31.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f9027b3d700 10 librados: watch_flush enter 2024-02-19T21:15:31.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f9027b3d700 10 librados: watch_flush exit 2024-02-19T21:15:31.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f9027b3d700 20 client.23691.objecter shutdown clearing up homeless session... 2024-02-19T21:15:31.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f9027b3d700 10 client.23691.objecter successfully canceled tick 2024-02-19T21:15:31.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f9027b3d700 1 -- 172.21.15.163:0/249111862 >> v1:172.21.15.163:6800/1552514681 conn(0x7f900c063a20 legacy=0x7f900c065ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:31.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f9027b3d700 1 -- 172.21.15.163:0/249111862 >> v1:172.21.15.163:6789/0 conn(0x7f9020102de0 legacy=0x7f90200fa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:31.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f9027b3d700 1 -- 172.21.15.163:0/249111862 shutdown_connections 2024-02-19T21:15:31.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.533+0000 7f9027b3d700 1 -- 172.21.15.163:0/249111862 >> 172.21.15.163:0/249111862 conn(0x7f902006cc00 msgr2=0x7f90200f77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:31.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.533+0000 7f9027b3d700 1 -- 172.21.15.163:0/249111862 shutdown_connections 2024-02-19T21:15:31.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.533+0000 7f9027b3d700 1 -- 172.21.15.163:0/249111862 wait complete. 2024-02-19T21:15:31.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.533+0000 7f9027b3d700 1 librados: shutdown 2024-02-19T21:15:31.635 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:15:31.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:30.510850+0000 mon.a (mon.0) 3342 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:31.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:30.523828+0000 mon.a (mon.0) 3343 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-118"}]': finished 2024-02-19T21:15:31.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:30.523898+0000 mon.a (mon.0) 3344 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2024-02-19T21:15:31.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:30.617501+0000 mon.a (mon.0) 3345 : audit [INF] from='client.? v1:172.21.15.163:0/249111862' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:15:31.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:30.692847+0000 mgr.y (mgr.14550) 580 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:31.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:31.263788+0000 mon.a (mon.0) 3346 : audit [INF] from='client.? v1:172.21.15.163:0/4203170446' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:15:31.876 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:30.510850+0000 mon.a (mon.0) 3342 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:31.876 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:30.523828+0000 mon.a (mon.0) 3343 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-118"}]': finished 2024-02-19T21:15:31.876 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:30.523898+0000 mon.a (mon.0) 3344 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2024-02-19T21:15:31.876 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:30.617501+0000 mon.a (mon.0) 3345 : audit [INF] from='client.? v1:172.21.15.163:0/249111862' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:15:31.876 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:30.692847+0000 mgr.y (mgr.14550) 580 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:31.876 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:31.263788+0000 mon.a (mon.0) 3346 : audit [INF] from='client.? v1:172.21.15.163:0/4203170446' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:15:31.877 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:30.510850+0000 mon.a (mon.0) 3342 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:31.877 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:30.523828+0000 mon.a (mon.0) 3343 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-118"}]': finished 2024-02-19T21:15:31.877 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:30.523898+0000 mon.a (mon.0) 3344 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2024-02-19T21:15:31.877 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:30.617501+0000 mon.a (mon.0) 3345 : audit [INF] from='client.? v1:172.21.15.163:0/249111862' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:15:31.877 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:30.692847+0000 mgr.y (mgr.14550) 580 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:31.877 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:31.263788+0000 mon.a (mon.0) 3346 : audit [INF] from='client.? v1:172.21.15.163:0/4203170446' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:15:31.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.071+0000 7f35320dc700 1 -- 172.21.15.163:0/2055348704 >> v1:172.21.15.163:6790/0 conn(0x7f352c0fd980 legacy=0x7f352c0fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:31.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.071+0000 7f35320dc700 1 -- 172.21.15.163:0/2055348704 shutdown_connections 2024-02-19T21:15:31.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.071+0000 7f35320dc700 1 -- 172.21.15.163:0/2055348704 >> 172.21.15.163:0/2055348704 conn(0x7f352c0f35a0 msgr2=0x7f352c0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:31.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.071+0000 7f35320dc700 1 -- 172.21.15.163:0/2055348704 shutdown_connections 2024-02-19T21:15:31.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.072+0000 7f35320dc700 1 -- 172.21.15.163:0/2055348704 wait complete. 2024-02-19T21:15:31.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.072+0000 7f35320dc700 1 librados: starting msgr at 2024-02-19T21:15:31.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.072+0000 7f35320dc700 1 librados: starting objecter 2024-02-19T21:15:31.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.072+0000 7f35320dc700 1 Processor -- start 2024-02-19T21:15:31.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.072+0000 7f35320dc700 1 -- start start 2024-02-19T21:15:31.980 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.072+0000 7f35320dc700 1 librados: setting wanted keys 2024-02-19T21:15:31.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.072+0000 7f35320dc700 1 librados: calling monclient init 2024-02-19T21:15:31.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f35320dc700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f352c1172e0 con 0x7f352c0fd980 2024-02-19T21:15:31.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f35320dc700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f352c1184c0 con 0x7f352c1042c0 2024-02-19T21:15:31.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f35320dc700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f352c1196c0 con 0x7f352c100780 2024-02-19T21:15:31.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f352b7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f352c0fd980 0x7f352c0f9870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44738/0 (socket says 172.21.15.163:44738) 2024-02-19T21:15:31.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f352b7fe700 1 -- 172.21.15.163:0/4203170446 learned_addr learned my addr 172.21.15.163:0/4203170446 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:31.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 10 client.?.objecter ms_handle_connect 0x7f352c0fd980 2024-02-19T21:15:31.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:31.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 10 client.?.objecter ms_handle_connect 0x7f352c100780 2024-02-19T21:15:31.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:31.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276067569 0 0) 0x7f352c1172e0 con 0x7f352c0fd980 2024-02-19T21:15:31.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3514003030 con 0x7f352c0fd980 2024-02-19T21:15:31.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 10 client.?.objecter ms_handle_connect 0x7f352c1042c0 2024-02-19T21:15:31.981 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:31.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213773385 0 0) 0x7f352c1196c0 con 0x7f352c100780 2024-02-19T21:15:31.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f352c1172e0 con 0x7f352c100780 2024-02-19T21:15:31.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589859905 0 0) 0x7f352c1184c0 con 0x7f352c1042c0 2024-02-19T21:15:31.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f352c1196c0 con 0x7f352c1042c0 2024-02-19T21:15:31.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3023337835 0 0) 0x7f3514003030 con 0x7f352c0fd980 2024-02-19T21:15:31.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f352c1184c0 con 0x7f352c0fd980 2024-02-19T21:15:31.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f351c002d40 con 0x7f352c0fd980 2024-02-19T21:15:31.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3094876846 0 0) 0x7f352c1172e0 con 0x7f352c100780 2024-02-19T21:15:31.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3514003030 con 0x7f352c100780 2024-02-19T21:15:31.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f35200019f0 con 0x7f352c100780 2024-02-19T21:15:31.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.073+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2973481129 0 0) 0x7f352c1196c0 con 0x7f352c1042c0 2024-02-19T21:15:31.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.074+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f352c1172e0 con 0x7f352c1042c0 2024-02-19T21:15:31.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.074+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3524001b80 con 0x7f352c1042c0 2024-02-19T21:15:31.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.074+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 971011662 0 0) 0x7f352c1184c0 con 0x7f352c0fd980 2024-02-19T21:15:31.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.074+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 >> v1:172.21.15.163:6790/0 conn(0x7f352c100780 legacy=0x7f352c0f9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:31.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.074+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 >> v1:172.21.15.193:6789/0 conn(0x7f352c1042c0 legacy=0x7f352c1159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:31.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.074+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f352c11a8c0 con 0x7f352c0fd980 2024-02-19T21:15:31.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.074+0000 7f35320dc700 1 -- 172.21.15.163:0/4203170446 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f352c1198b0 con 0x7f352c0fd980 2024-02-19T21:15:31.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.074+0000 7f35320dc700 10 client.23697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:31.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.074+0000 7f35320dc700 1 -- 172.21.15.163:0/4203170446 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f352c119da0 con 0x7f352c0fd980 2024-02-19T21:15:31.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.074+0000 7f35320dc700 1 librados: init done 2024-02-19T21:15:31.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.074+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f351c002ea0 con 0x7f352c0fd980 2024-02-19T21:15:31.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.074+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f351c003800 con 0x7f352c0fd980 2024-02-19T21:15:31.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.074+0000 7f35320dc700 1 -- 172.21.15.163:0/4203170446 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3518004f20 con 0x7f352c0fd980 2024-02-19T21:15:31.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.075+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f351c001ea0 con 0x7f352c0fd980 2024-02-19T21:15:31.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.075+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(681..681 src has 623..681) v4 ==== 8555+0+0 (unknown 2397927702 0 0) 0x7f351c079100 con 0x7f352c0fd980 2024-02-19T21:15:31.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.075+0000 7f3528ff9700 10 client.23697.objecter ms_dispatch 0x7f352c001550 osd_map(681..681 src has 623..681) v4 2024-02-19T21:15:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.075+0000 7f3528ff9700 3 client.23697.objecter handle_osd_map got epochs [681,681] > 0 2024-02-19T21:15:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.075+0000 7f3528ff9700 3 client.23697.objecter handle_osd_map decoding full epoch 681 2024-02-19T21:15:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.076+0000 7f3528ff9700 20 client.23697.objecter dump_active .. 0 homeless 2024-02-19T21:15:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.076+0000 7f3528ff9700 10 client.23697.objecter ms_handle_connect 0x7f35140680c0 2024-02-19T21:15:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.080+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f351c04c3e0 con 0x7f352c0fd980 2024-02-19T21:15:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.262+0000 7f35320dc700 1 -- 172.21.15.163:0/4203170446 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f35180051d0 con 0x7f352c0fd980 2024-02-19T21:15:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.526+0000 7f3528ff9700 1 -- 172.21.15.163:0/4203170446 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v682) v1 ==== 113+0+0 (unknown 2555195379 0 0) 0x7f351c04a850 con 0x7f352c0fd980 2024-02-19T21:15:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f35320dc700 10 librados: watch_flush enter 2024-02-19T21:15:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f35320dc700 10 librados: watch_flush exit 2024-02-19T21:15:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f35320dc700 20 client.23697.objecter shutdown clearing up homeless session... 2024-02-19T21:15:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f35320dc700 10 client.23697.objecter successfully canceled tick 2024-02-19T21:15:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f35320dc700 1 -- 172.21.15.163:0/4203170446 >> v1:172.21.15.163:6800/1552514681 conn(0x7f35140680c0 legacy=0x7f351406a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f35320dc700 1 -- 172.21.15.163:0/4203170446 >> v1:172.21.15.163:6789/0 conn(0x7f352c0fd980 legacy=0x7f352c0f9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:31.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f35320dc700 1 -- 172.21.15.163:0/4203170446 shutdown_connections 2024-02-19T21:15:31.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f35320dc700 1 -- 172.21.15.163:0/4203170446 >> 172.21.15.163:0/4203170446 conn(0x7f352c0f35a0 msgr2=0x7f352c107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:31.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f35320dc700 1 -- 172.21.15.163:0/4203170446 shutdown_connections 2024-02-19T21:15:31.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f35320dc700 1 -- 172.21.15.163:0/4203170446 wait complete. 2024-02-19T21:15:31.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:31.532+0000 7f35320dc700 1 librados: shutdown 2024-02-19T21:15:31.985 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.3 to 1 (10000) 2024-02-19T21:15:32.515 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:31.305576+0000 mgr.y (mgr.14550) 581 : cluster [DBG] pgmap v1028: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 261 B/s wr, 1 op/s 2024-02-19T21:15:32.516 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:31.517182+0000 mon.a (mon.0) 3347 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:15:32.516 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:31.305576+0000 mgr.y (mgr.14550) 581 : cluster [DBG] pgmap v1028: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 261 B/s wr, 1 op/s 2024-02-19T21:15:32.516 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:31.517182+0000 mon.a (mon.0) 3347 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:15:32.516 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:31.526788+0000 mon.a (mon.0) 3348 : audit [INF] from='client.? v1:172.21.15.163:0/249111862' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:15:32.516 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:31.526852+0000 mon.a (mon.0) 3349 : audit [INF] from='client.? v1:172.21.15.163:0/4203170446' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-19T21:15:32.516 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:31.526892+0000 mon.a (mon.0) 3350 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2024-02-19T21:15:32.554 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 4 1 2024-02-19T21:15:32.777 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:31.526788+0000 mon.a (mon.0) 3348 : audit [INF] from='client.? v1:172.21.15.163:0/249111862' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:15:32.778 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:31.526852+0000 mon.a (mon.0) 3349 : audit [INF] from='client.? v1:172.21.15.163:0/4203170446' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-19T21:15:32.778 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:31.526892+0000 mon.a (mon.0) 3350 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2024-02-19T21:15:32.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:31.305576+0000 mgr.y (mgr.14550) 581 : cluster [DBG] pgmap v1028: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 261 B/s wr, 1 op/s 2024-02-19T21:15:32.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:31.517182+0000 mon.a (mon.0) 3347 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:15:32.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:31.526788+0000 mon.a (mon.0) 3348 : audit [INF] from='client.? v1:172.21.15.163:0/249111862' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:15:32.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:31.526852+0000 mon.a (mon.0) 3349 : audit [INF] from='client.? v1:172.21.15.163:0/4203170446' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-19T21:15:32.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:31.526892+0000 mon.a (mon.0) 3350 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2024-02-19T21:15:33.135 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:33.164 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:33.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:33.270 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:33.296 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:33.395 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:33.402 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:33.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:33.503 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:33.545 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:33.566 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:33.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:33.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:33.663 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:15:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:15:33] "GET /metrics HTTP/1.1" 200 192002 "" "Prometheus/2.33.4" 2024-02-19T21:15:33.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:32.210440+0000 osd.2 (osd.2) 270 : cluster [DBG] 5.a scrub starts 2024-02-19T21:15:33.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:32.211534+0000 osd.2 (osd.2) 271 : cluster [DBG] 5.a scrub ok 2024-02-19T21:15:33.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:32.527565+0000 mon.a (mon.0) 3351 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2024-02-19T21:15:33.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:32.534245+0000 mon.a (mon.0) 3352 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:33.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:32.973687+0000 osd.7 (osd.7) 313 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:15:33.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:32.976742+0000 osd.7 (osd.7) 314 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:15:33.664 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:32.210440+0000 osd.2 (osd.2) 270 : cluster [DBG] 5.a scrub starts 2024-02-19T21:15:33.664 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:32.211534+0000 osd.2 (osd.2) 271 : cluster [DBG] 5.a scrub ok 2024-02-19T21:15:33.664 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:32.527565+0000 mon.a (mon.0) 3351 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2024-02-19T21:15:33.664 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:32.534245+0000 mon.a (mon.0) 3352 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:33.664 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:32.973687+0000 osd.7 (osd.7) 313 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:15:33.664 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:32.976742+0000 osd.7 (osd.7) 314 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:15:33.670 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:33.700 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:33.738 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:33.741 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:33.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:33.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:32.210440+0000 osd.2 (osd.2) 270 : cluster [DBG] 5.a scrub starts 2024-02-19T21:15:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:32.211534+0000 osd.2 (osd.2) 271 : cluster [DBG] 5.a scrub ok 2024-02-19T21:15:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:32.527565+0000 mon.a (mon.0) 3351 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2024-02-19T21:15:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:32.534245+0000 mon.a (mon.0) 3352 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:32.973687+0000 osd.7 (osd.7) 313 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:15:33.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:32.976742+0000 osd.7 (osd.7) 314 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:15:33.842 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:33.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.841+0000 7f15c58af700 1 -- 172.21.15.163:0/3637371150 >> v1:172.21.15.163:6790/0 conn(0x7f15c01042b0 legacy=0x7f15c0106730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:33.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.841+0000 7f15c58af700 1 -- 172.21.15.163:0/3637371150 shutdown_connections 2024-02-19T21:15:33.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.841+0000 7f15c58af700 1 -- 172.21.15.163:0/3637371150 >> 172.21.15.163:0/3637371150 conn(0x7f15c00f3590 msgr2=0x7f15c00f59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:33.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.842+0000 7f15c58af700 1 -- 172.21.15.163:0/3637371150 shutdown_connections 2024-02-19T21:15:33.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.842+0000 7f15c58af700 1 -- 172.21.15.163:0/3637371150 wait complete. 2024-02-19T21:15:33.843 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.842+0000 7f15c58af700 1 librados: starting msgr at 2024-02-19T21:15:33.844 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.842+0000 7f15c58af700 1 librados: starting objecter 2024-02-19T21:15:33.844 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.842+0000 7f15c58af700 1 Processor -- start 2024-02-19T21:15:33.844 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15c58af700 1 -- start start 2024-02-19T21:15:33.844 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15c58af700 1 librados: setting wanted keys 2024-02-19T21:15:33.844 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15c58af700 1 librados: calling monclient init 2024-02-19T21:15:33.847 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15c58af700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15c0117420 con 0x7f15c0100770 2024-02-19T21:15:33.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15c58af700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15c0118600 con 0x7f15c00fd970 2024-02-19T21:15:33.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15c58af700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15c0119800 con 0x7f15c01042b0 2024-02-19T21:15:33.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15be7fc700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f15c0100770 0x7f15c00fa3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44754/0 (socket says 172.21.15.163:44754) 2024-02-19T21:15:33.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15be7fc700 1 -- 172.21.15.163:0/3637463556 learned_addr learned my addr 172.21.15.163:0/3637463556 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:33.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 10 client.?.objecter ms_handle_connect 0x7f15c0100770 2024-02-19T21:15:33.848 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:33.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 10 client.?.objecter ms_handle_connect 0x7f15c01042b0 2024-02-19T21:15:33.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:33.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996473924 0 0) 0x7f15c0117420 con 0x7f15c0100770 2024-02-19T21:15:33.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a8003030 con 0x7f15c0100770 2024-02-19T21:15:33.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 10 client.?.objecter ms_handle_connect 0x7f15c00fd970 2024-02-19T21:15:33.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:33.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216228633 0 0) 0x7f15c0119800 con 0x7f15c01042b0 2024-02-19T21:15:33.849 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c0117420 con 0x7f15c01042b0 2024-02-19T21:15:33.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118581622 0 0) 0x7f15c0118600 con 0x7f15c00fd970 2024-02-19T21:15:33.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c0119800 con 0x7f15c00fd970 2024-02-19T21:15:33.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1228936300 0 0) 0x7f15a8003030 con 0x7f15c0100770 2024-02-19T21:15:33.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15c0118600 con 0x7f15c0100770 2024-02-19T21:15:33.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f15b4001920 con 0x7f15c0100770 2024-02-19T21:15:33.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2470138571 0 0) 0x7f15c0117420 con 0x7f15c01042b0 2024-02-19T21:15:33.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.843+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15a8003030 con 0x7f15c01042b0 2024-02-19T21:15:33.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.844+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f15b80021d0 con 0x7f15c01042b0 2024-02-19T21:15:33.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.844+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3619349062 0 0) 0x7f15c0118600 con 0x7f15c0100770 2024-02-19T21:15:33.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.844+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 >> v1:172.21.15.163:6790/0 conn(0x7f15c01042b0 legacy=0x7f15c0115ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:33.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.844+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 >> v1:172.21.15.193:6789/0 conn(0x7f15c00fd970 legacy=0x7f15c00f9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:33.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.844+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c011aa00 con 0x7f15c0100770 2024-02-19T21:15:33.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.844+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c01199f0 con 0x7f15c0100770 2024-02-19T21:15:33.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.844+0000 7f15c58af700 10 client.23703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:33.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.844+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c0119ee0 con 0x7f15c0100770 2024-02-19T21:15:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.844+0000 7f15c58af700 1 librados: init done 2024-02-19T21:15:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.844+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f15b4001cb0 con 0x7f15c0100770 2024-02-19T21:15:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.844+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f15b4005b50 con 0x7f15c0100770 2024-02-19T21:15:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.844+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f15ac004f00 con 0x7f15c0100770 2024-02-19T21:15:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.845+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f15b40028b0 con 0x7f15c0100770 2024-02-19T21:15:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.845+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(684..684 src has 623..684) v4 ==== 8568+0+0 (unknown 3072077451 0 0) 0x7f15b4078f00 con 0x7f15c0100770 2024-02-19T21:15:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 osd_map(684..684 src has 623..684) v4 2024-02-19T21:15:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15a7fff700 3 client.23703.objecter handle_osd_map got epochs [684,684] > 0 2024-02-19T21:15:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15a7fff700 3 client.23703.objecter handle_osd_map decoding full epoch 684 2024-02-19T21:15:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15a7fff700 20 client.23703.objecter dump_active .. 0 homeless 2024-02-19T21:15:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v684) v1 ==== 54+0+16 (unknown 2920152435 0 2420000192) 0x7f15b407b2a0 con 0x7f15c0100770 2024-02-19T21:15:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15a7fff700 10 client.23703.objecter ms_handle_connect 0x7f15a8063990 2024-02-19T21:15:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15c58af700 10 client.23703.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 1 2024-02-19T21:15:33.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac0051a0 osd=0 3 2024-02-19T21:15:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac0051a0 osd=0 3 2024-02-19T21:15:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac0051a0 osd=0 3 2024-02-19T21:15:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 1 2024-02-19T21:15:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac0051a0 osd=0 3 2024-02-19T21:15:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 0 1 2024-02-19T21:15:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac0051a0 osd=0 4 2024-02-19T21:15:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15c58af700 10 client.23703.objecter _send_command 1 2024-02-19T21:15:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15ac004950 con 0x7f15ac004530 2024-02-19T21:15:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.846+0000 7f15a7fff700 10 client.23703.objecter ms_handle_connect 0x7f15ac004530 2024-02-19T21:15:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.847+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f15ac004950 con 0x7f15ac004530 2024-02-19T21:15:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.847+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:15:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.847+0000 7f15a7fff700 10 client.23703.objecter _finish_command 1 = system:0 2024-02-19T21:15:33.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.847+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac0051a0 osd=0 4 2024-02-19T21:15:33.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.847+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 0 1 2024-02-19T21:15:33.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.862+0000 7f15c58af700 10 client.23703.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:33.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.862+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 2 2024-02-19T21:15:33.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.862+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac0051a0 osd=0 3 2024-02-19T21:15:33.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.862+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac0051a0 osd=0 3 2024-02-19T21:15:33.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.862+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac0051a0 osd=0 3 2024-02-19T21:15:33.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.862+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 2 2024-02-19T21:15:33.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.862+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac0051a0 osd=0 3 2024-02-19T21:15:33.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.862+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 0 2 2024-02-19T21:15:33.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.862+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac0051a0 osd=0 4 2024-02-19T21:15:33.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.862+0000 7f15c58af700 10 client.23703.objecter _send_command 2 2024-02-19T21:15:33.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.862+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f15ac004d30 con 0x7f15ac004530 2024-02-19T21:15:33.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f15ac004d30 con 0x7f15ac004530 2024-02-19T21:15:33.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:33.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15a7fff700 10 client.23703.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac0051a0 osd=0 4 2024-02-19T21:15:33.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 0 2 2024-02-19T21:15:33.866 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:15:33.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15c58af700 10 client.23703.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:33.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 3 2024-02-19T21:15:33.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac003fe0 osd=1 3 2024-02-19T21:15:33.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac003fe0 osd=1 3 2024-02-19T21:15:33.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac003fe0 osd=1 3 2024-02-19T21:15:33.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 3 2024-02-19T21:15:33.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac003fe0 osd=1 3 2024-02-19T21:15:33.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 1 3 2024-02-19T21:15:33.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac003fe0 osd=1 4 2024-02-19T21:15:33.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15c58af700 10 client.23703.objecter _send_command 3 2024-02-19T21:15:33.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.863+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15ac00c000 con 0x7f15ac004150 2024-02-19T21:15:33.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.864+0000 7f15a7fff700 10 client.23703.objecter ms_handle_connect 0x7f15ac004150 2024-02-19T21:15:33.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.864+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f15ac00c000 con 0x7f15ac004150 2024-02-19T21:15:33.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.864+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:15:33.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.864+0000 7f15a7fff700 10 client.23703.objecter _finish_command 3 = system:0 2024-02-19T21:15:33.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.864+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac003fe0 osd=1 4 2024-02-19T21:15:33.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.864+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 1 3 2024-02-19T21:15:33.873 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:33.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.879+0000 7f15c58af700 10 client.23703.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:33.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.879+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 4 2024-02-19T21:15:33.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.879+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac003fe0 osd=1 3 2024-02-19T21:15:33.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.879+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac003fe0 osd=1 3 2024-02-19T21:15:33.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.879+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac003fe0 osd=1 3 2024-02-19T21:15:33.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.879+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 4 2024-02-19T21:15:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.879+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac003fe0 osd=1 3 2024-02-19T21:15:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.879+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 1 4 2024-02-19T21:15:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.879+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac003fe0 osd=1 4 2024-02-19T21:15:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.879+0000 7f15c58af700 10 client.23703.objecter _send_command 4 2024-02-19T21:15:33.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.879+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f15ac0018b0 con 0x7f15ac004150 2024-02-19T21:15:33.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.880+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f15ac0018b0 con 0x7f15ac004150 2024-02-19T21:15:33.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.880+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:33.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.880+0000 7f15a7fff700 10 client.23703.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.880+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac003fe0 osd=1 4 2024-02-19T21:15:33.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.880+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 1 4 2024-02-19T21:15:33.883 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:15:33.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15c58af700 10 client.23703.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:33.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 5 2024-02-19T21:15:33.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac003630 osd=2 3 2024-02-19T21:15:33.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac003630 osd=2 3 2024-02-19T21:15:33.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac003630 osd=2 3 2024-02-19T21:15:33.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 5 2024-02-19T21:15:33.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac003630 osd=2 3 2024-02-19T21:15:33.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 2 5 2024-02-19T21:15:33.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac003630 osd=2 4 2024-02-19T21:15:33.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15c58af700 10 client.23703.objecter _send_command 5 2024-02-19T21:15:33.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15ac00de90 con 0x7f15ac00d2f0 2024-02-19T21:15:33.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15a7fff700 10 client.23703.objecter ms_handle_connect 0x7f15ac00d2f0 2024-02-19T21:15:33.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f15ac00de90 con 0x7f15ac00d2f0 2024-02-19T21:15:33.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:15:33.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15a7fff700 10 client.23703.objecter _finish_command 5 = system:0 2024-02-19T21:15:33.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac003630 osd=2 4 2024-02-19T21:15:33.887 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.881+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 2 5 2024-02-19T21:15:33.893 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:33.895 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:33.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:33.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.896+0000 7f15c58af700 10 client.23703.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:33.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.896+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 6 2024-02-19T21:15:33.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.896+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac003630 osd=2 3 2024-02-19T21:15:33.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.896+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac003630 osd=2 3 2024-02-19T21:15:33.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.896+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac003630 osd=2 3 2024-02-19T21:15:33.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.896+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 6 2024-02-19T21:15:33.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.896+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac003630 osd=2 3 2024-02-19T21:15:33.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.896+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 2 6 2024-02-19T21:15:33.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.896+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac003630 osd=2 4 2024-02-19T21:15:33.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.896+0000 7f15c58af700 10 client.23703.objecter _send_command 6 2024-02-19T21:15:33.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.896+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f15ac0030a0 con 0x7f15ac00d2f0 2024-02-19T21:15:33.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.897+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f15ac0030a0 con 0x7f15ac00d2f0 2024-02-19T21:15:33.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.897+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:33.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.897+0000 7f15a7fff700 10 client.23703.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.897+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac003630 osd=2 4 2024-02-19T21:15:33.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.897+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 2 6 2024-02-19T21:15:33.899 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:15:33.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15c58af700 10 client.23703.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:33.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 7 2024-02-19T21:15:33.900 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac015d80 osd=3 3 2024-02-19T21:15:33.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac015d80 osd=3 3 2024-02-19T21:15:33.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac015d80 osd=3 3 2024-02-19T21:15:33.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 7 2024-02-19T21:15:33.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac015d80 osd=3 3 2024-02-19T21:15:33.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 3 7 2024-02-19T21:15:33.901 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac015d80 osd=3 4 2024-02-19T21:15:33.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15c58af700 10 client.23703.objecter _send_command 7 2024-02-19T21:15:33.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15ac019f10 con 0x7f15ac016400 2024-02-19T21:15:33.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15a7fff700 10 client.23703.objecter ms_handle_connect 0x7f15ac016400 2024-02-19T21:15:33.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f15ac019f10 con 0x7f15ac016400 2024-02-19T21:15:33.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:15:33.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15a7fff700 10 client.23703.objecter _finish_command 7 = system:0 2024-02-19T21:15:33.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac015d80 osd=3 4 2024-02-19T21:15:33.903 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.898+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 3 7 2024-02-19T21:15:33.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.913+0000 7f15c58af700 10 client.23703.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:33.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.913+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 8 2024-02-19T21:15:33.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.913+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac015d80 osd=3 3 2024-02-19T21:15:33.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.913+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac015d80 osd=3 3 2024-02-19T21:15:33.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.913+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac015d80 osd=3 3 2024-02-19T21:15:33.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.913+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 8 2024-02-19T21:15:33.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.913+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac015d80 osd=3 3 2024-02-19T21:15:33.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.913+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 3 8 2024-02-19T21:15:33.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.913+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac015d80 osd=3 4 2024-02-19T21:15:33.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.913+0000 7f15c58af700 10 client.23703.objecter _send_command 8 2024-02-19T21:15:33.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.913+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f15ac00e270 con 0x7f15ac016400 2024-02-19T21:15:33.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f15ac00e270 con 0x7f15ac016400 2024-02-19T21:15:33.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:33.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15a7fff700 10 client.23703.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac015d80 osd=3 4 2024-02-19T21:15:33.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 3 8 2024-02-19T21:15:33.916 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:15:33.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15c58af700 10 client.23703.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:33.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 9 2024-02-19T21:15:33.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac00e270 osd=4 3 2024-02-19T21:15:33.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac00e270 osd=4 3 2024-02-19T21:15:33.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac00e270 osd=4 3 2024-02-19T21:15:33.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 9 2024-02-19T21:15:33.917 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac00e270 osd=4 3 2024-02-19T21:15:33.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 4 9 2024-02-19T21:15:33.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac00e270 osd=4 4 2024-02-19T21:15:33.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15c58af700 10 client.23703.objecter _send_command 9 2024-02-19T21:15:33.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.914+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15ac00e510 con 0x7f15ac00ce50 2024-02-19T21:15:33.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.915+0000 7f15a7fff700 10 client.23703.objecter ms_handle_connect 0x7f15ac00ce50 2024-02-19T21:15:33.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.916+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f15ac00e510 con 0x7f15ac00ce50 2024-02-19T21:15:33.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.916+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:15:33.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.916+0000 7f15a7fff700 10 client.23703.objecter _finish_command 9 = system:0 2024-02-19T21:15:33.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.916+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac00e270 osd=4 4 2024-02-19T21:15:33.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.916+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 4 9 2024-02-19T21:15:33.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.930+0000 7f15c58af700 10 client.23703.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:33.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.930+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 10 2024-02-19T21:15:33.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.930+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac00e270 osd=4 3 2024-02-19T21:15:33.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.930+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac00e270 osd=4 3 2024-02-19T21:15:33.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.930+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac00e270 osd=4 3 2024-02-19T21:15:33.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.930+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 10 2024-02-19T21:15:33.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.930+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac00e270 osd=4 3 2024-02-19T21:15:33.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.930+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 4 10 2024-02-19T21:15:33.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.930+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac00e270 osd=4 4 2024-02-19T21:15:33.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.930+0000 7f15c58af700 10 client.23703.objecter _send_command 10 2024-02-19T21:15:33.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.930+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f15ac00e510 con 0x7f15ac00ce50 2024-02-19T21:15:33.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f15ac00e510 con 0x7f15ac00ce50 2024-02-19T21:15:33.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:33.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15a7fff700 10 client.23703.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.934 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:15:33.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac00e270 osd=4 4 2024-02-19T21:15:33.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 4 10 2024-02-19T21:15:33.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15c58af700 10 client.23703.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:33.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 11 2024-02-19T21:15:33.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac00e510 osd=5 3 2024-02-19T21:15:33.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac00e510 osd=5 3 2024-02-19T21:15:33.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac00e510 osd=5 3 2024-02-19T21:15:33.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 11 2024-02-19T21:15:33.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac00e510 osd=5 3 2024-02-19T21:15:33.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 5 11 2024-02-19T21:15:33.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac00e510 osd=5 4 2024-02-19T21:15:33.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15c58af700 10 client.23703.objecter _send_command 11 2024-02-19T21:15:33.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.932+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15ac01c230 con 0x7f15ac01bcb0 2024-02-19T21:15:33.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.933+0000 7f15a7fff700 10 client.23703.objecter ms_handle_connect 0x7f15ac01bcb0 2024-02-19T21:15:33.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.934+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f15ac01c230 con 0x7f15ac01bcb0 2024-02-19T21:15:33.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.934+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:15:33.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.934+0000 7f15a7fff700 10 client.23703.objecter _finish_command 11 = system:0 2024-02-19T21:15:33.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.934+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac00e510 osd=5 4 2024-02-19T21:15:33.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.934+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 5 11 2024-02-19T21:15:33.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.948+0000 7f15c58af700 10 client.23703.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:33.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.948+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 12 2024-02-19T21:15:33.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.948+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac00e510 osd=5 3 2024-02-19T21:15:33.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.948+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac00e510 osd=5 3 2024-02-19T21:15:33.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.948+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac00e510 osd=5 3 2024-02-19T21:15:33.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.948+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 12 2024-02-19T21:15:33.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.948+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac00e510 osd=5 3 2024-02-19T21:15:33.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.948+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 5 12 2024-02-19T21:15:33.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.948+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac00e510 osd=5 4 2024-02-19T21:15:33.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.948+0000 7f15c58af700 10 client.23703.objecter _send_command 12 2024-02-19T21:15:33.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.948+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f15ac01c230 con 0x7f15ac01bcb0 2024-02-19T21:15:33.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f15ac01c230 con 0x7f15ac01bcb0 2024-02-19T21:15:33.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:33.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15a7fff700 10 client.23703.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac00e510 osd=5 4 2024-02-19T21:15:33.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 5 12 2024-02-19T21:15:33.952 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:15:33.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15c58af700 10 client.23703.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:33.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 13 2024-02-19T21:15:33.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac01c230 osd=6 3 2024-02-19T21:15:33.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac01c230 osd=6 3 2024-02-19T21:15:33.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac01c230 osd=6 3 2024-02-19T21:15:33.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 13 2024-02-19T21:15:33.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac01c230 osd=6 3 2024-02-19T21:15:33.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 6 13 2024-02-19T21:15:33.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac01c230 osd=6 4 2024-02-19T21:15:33.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15c58af700 10 client.23703.objecter _send_command 13 2024-02-19T21:15:33.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.950+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15ac026fa0 con 0x7f15ac023490 2024-02-19T21:15:33.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.951+0000 7f15a7fff700 10 client.23703.objecter ms_handle_connect 0x7f15ac023490 2024-02-19T21:15:33.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.952+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f15ac026fa0 con 0x7f15ac023490 2024-02-19T21:15:33.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.952+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:15:33.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.952+0000 7f15a7fff700 10 client.23703.objecter _finish_command 13 = system:0 2024-02-19T21:15:33.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.952+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac01c230 osd=6 4 2024-02-19T21:15:33.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.952+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 6 13 2024-02-19T21:15:33.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.966+0000 7f15c58af700 10 client.23703.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:33.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.966+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 14 2024-02-19T21:15:33.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.966+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac01c230 osd=6 3 2024-02-19T21:15:33.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.966+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac01c230 osd=6 3 2024-02-19T21:15:33.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.966+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac01c230 osd=6 3 2024-02-19T21:15:33.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.966+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 14 2024-02-19T21:15:33.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.966+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac01c230 osd=6 3 2024-02-19T21:15:33.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.966+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 6 14 2024-02-19T21:15:33.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.966+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac01c230 osd=6 4 2024-02-19T21:15:33.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.966+0000 7f15c58af700 10 client.23703.objecter _send_command 14 2024-02-19T21:15:33.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.966+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f15ac028720 con 0x7f15ac023490 2024-02-19T21:15:33.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.968+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f15ac028720 con 0x7f15ac023490 2024-02-19T21:15:33.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.968+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:33.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.968+0000 7f15a7fff700 10 client.23703.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.968+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac01c230 osd=6 4 2024-02-19T21:15:33.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.968+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 6 14 2024-02-19T21:15:33.970 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:15:33.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.969+0000 7f15c58af700 10 client.23703.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:33.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.969+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 15 2024-02-19T21:15:33.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.969+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac028720 osd=7 3 2024-02-19T21:15:33.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.969+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac028720 osd=7 3 2024-02-19T21:15:33.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.969+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac028720 osd=7 3 2024-02-19T21:15:33.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.969+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 15 2024-02-19T21:15:33.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.969+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac028720 osd=7 3 2024-02-19T21:15:33.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.969+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 7 15 2024-02-19T21:15:33.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.969+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac028720 osd=7 4 2024-02-19T21:15:33.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.969+0000 7f15c58af700 10 client.23703.objecter _send_command 15 2024-02-19T21:15:33.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.969+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f15ac029350 con 0x7f15ac028da0 2024-02-19T21:15:33.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.969+0000 7f15a7fff700 10 client.23703.objecter ms_handle_connect 0x7f15ac028da0 2024-02-19T21:15:33.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.970+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f15ac029350 con 0x7f15ac028da0 2024-02-19T21:15:33.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.970+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:15:33.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.970+0000 7f15a7fff700 10 client.23703.objecter _finish_command 15 = system:0 2024-02-19T21:15:33.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.970+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac028720 osd=7 4 2024-02-19T21:15:33.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.970+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 7 15 2024-02-19T21:15:33.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.984+0000 7f15c58af700 10 client.23703.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:33.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.984+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign -1 16 2024-02-19T21:15:33.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.984+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac028720 osd=7 3 2024-02-19T21:15:33.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.984+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac028720 osd=7 3 2024-02-19T21:15:33.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.984+0000 7f15c58af700 20 client.23703.objecter _get_session s=0x7f15ac028720 osd=7 3 2024-02-19T21:15:33.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.984+0000 7f15c58af700 15 client.23703.objecter _session_command_op_remove -1 16 2024-02-19T21:15:33.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.984+0000 7f15c58af700 20 client.23703.objecter get_session s=0x7f15ac028720 osd=7 3 2024-02-19T21:15:33.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.984+0000 7f15c58af700 15 client.23703.objecter _session_command_op_assign 7 16 2024-02-19T21:15:33.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.984+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac028720 osd=7 4 2024-02-19T21:15:33.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.984+0000 7f15c58af700 10 client.23703.objecter _send_command 16 2024-02-19T21:15:33.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.984+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f15ac029350 con 0x7f15ac028da0 2024-02-19T21:15:33.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.985+0000 7f15a7fff700 1 -- 172.21.15.163:0/3637463556 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f15ac029350 con 0x7f15ac028da0 2024-02-19T21:15:33.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.985+0000 7f15a7fff700 10 client.23703.objecter ms_dispatch 0x7f15c0001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:33.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.985+0000 7f15a7fff700 10 client.23703.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.985+0000 7f15a7fff700 20 client.23703.objecter put_session s=0x7f15ac028720 osd=7 4 2024-02-19T21:15:33.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.985+0000 7f15a7fff700 15 client.23703.objecter _session_command_op_remove 7 16 2024-02-19T21:15:33.987 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:15:33.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 10 librados: watch_flush enter 2024-02-19T21:15:33.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 10 librados: watch_flush exit 2024-02-19T21:15:33.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 10 client.23703.objecter close_session for osd.0 2024-02-19T21:15:33.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 >> v1:172.21.15.163:6801/1559418115 conn(0x7f15ac004530 legacy=0x7f15ac002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:33.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac0051a0 osd=0 1 2024-02-19T21:15:33.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 10 client.23703.objecter close_session for osd.1 2024-02-19T21:15:33.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 >> v1:172.21.15.163:6805/3125047432 conn(0x7f15ac004150 legacy=0x7f15ac00a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:33.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac003fe0 osd=1 1 2024-02-19T21:15:33.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 10 client.23703.objecter close_session for osd.2 2024-02-19T21:15:33.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 >> v1:172.21.15.163:6809/1563158487 conn(0x7f15ac00d2f0 legacy=0x7f15ac00d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:33.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac003630 osd=2 1 2024-02-19T21:15:33.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 10 client.23703.objecter close_session for osd.3 2024-02-19T21:15:33.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 >> v1:172.21.15.163:6813/2112279769 conn(0x7f15ac016400 legacy=0x7f15ac0187f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:33.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15bf7fe700 1 -- 172.21.15.163:0/3637463556 reap_dead start 2024-02-19T21:15:33.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac015d80 osd=3 1 2024-02-19T21:15:33.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 10 client.23703.objecter close_session for osd.4 2024-02-19T21:15:33.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 >> v1:172.21.15.193:6800/3129716965 conn(0x7f15ac00ce50 legacy=0x7f15ac01b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac00e270 osd=4 1 2024-02-19T21:15:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15bf7fe700 1 -- 172.21.15.163:0/3637463556 reap_dead start 2024-02-19T21:15:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 10 client.23703.objecter close_session for osd.5 2024-02-19T21:15:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 >> v1:172.21.15.193:6804/805161991 conn(0x7f15ac01bcb0 legacy=0x7f15ac021860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac00e510 osd=5 1 2024-02-19T21:15:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 10 client.23703.objecter close_session for osd.6 2024-02-19T21:15:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 >> v1:172.21.15.193:6808/787604615 conn(0x7f15ac023490 legacy=0x7f15ac025880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac01c230 osd=6 1 2024-02-19T21:15:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 10 client.23703.objecter close_session for osd.7 2024-02-19T21:15:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 >> v1:172.21.15.193:6812/157927432 conn(0x7f15ac028da0 legacy=0x7f15ac026fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:33.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 20 client.23703.objecter put_session s=0x7f15ac028720 osd=7 1 2024-02-19T21:15:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 20 client.23703.objecter shutdown clearing up homeless session... 2024-02-19T21:15:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 10 client.23703.objecter successfully canceled tick 2024-02-19T21:15:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 >> v1:172.21.15.163:6800/1552514681 conn(0x7f15a8063990 legacy=0x7f15a8065e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 >> v1:172.21.15.163:6789/0 conn(0x7f15c0100770 legacy=0x7f15c00fa3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15bf7fe700 1 -- 172.21.15.163:0/3637463556 reap_dead start 2024-02-19T21:15:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 shutdown_connections 2024-02-19T21:15:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 >> 172.21.15.163:0/3637463556 conn(0x7f15c00f3590 msgr2=0x7f15c00f67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 shutdown_connections 2024-02-19T21:15:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 1 -- 172.21.15.163:0/3637463556 wait complete. 2024-02-19T21:15:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:33.986+0000 7f15c58af700 1 librados: shutdown 2024-02-19T21:15:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.991 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.992 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.992 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.992 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.992 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.992 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.992 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:15:33.997 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:34.024 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:34.044 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:34.048 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:34.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:34.149 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:34.176 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:34.213 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:34.216 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:34.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:34.317 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:34.346 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:34.366 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:34.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:34.470 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:15:34.499 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:34.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:33.305941+0000 mgr.y (mgr.14550) 582 : cluster [DBG] pgmap v1031: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:15:34.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:33.528364+0000 mon.a (mon.0) 3353 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:34.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:33.528495+0000 mon.a (mon.0) 3354 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2024-02-19T21:15:34.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:33.531550+0000 mon.a (mon.0) 3355 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:34.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:33.846943+0000 mon.a (mon.0) 3356 : audit [DBG] from='client.? v1:172.21.15.163:0/3637463556' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:34.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:34.529089+0000 mon.a (mon.0) 3357 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-120", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:34.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:34.529141+0000 mon.a (mon.0) 3358 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2024-02-19T21:15:34.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:34.529759+0000 mon.a (mon.0) 3359 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-120"}]: dispatch 2024-02-19T21:15:34.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:33.305941+0000 mgr.y (mgr.14550) 582 : cluster [DBG] pgmap v1031: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:15:34.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:33.528364+0000 mon.a (mon.0) 3353 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:34.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:33.528495+0000 mon.a (mon.0) 3354 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2024-02-19T21:15:34.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:33.531550+0000 mon.a (mon.0) 3355 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:34.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:33.846943+0000 mon.a (mon.0) 3356 : audit [DBG] from='client.? v1:172.21.15.163:0/3637463556' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:34.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:34.529089+0000 mon.a (mon.0) 3357 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-120", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:34.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:34.529141+0000 mon.a (mon.0) 3358 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2024-02-19T21:15:34.752 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:34.529759+0000 mon.a (mon.0) 3359 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-120"}]: dispatch 2024-02-19T21:15:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:33.305941+0000 mgr.y (mgr.14550) 582 : cluster [DBG] pgmap v1031: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:15:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:33.528364+0000 mon.a (mon.0) 3353 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:33.528495+0000 mon.a (mon.0) 3354 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2024-02-19T21:15:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:33.531550+0000 mon.a (mon.0) 3355 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:33.846943+0000 mon.a (mon.0) 3356 : audit [DBG] from='client.? v1:172.21.15.163:0/3637463556' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:34.529089+0000 mon.a (mon.0) 3357 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-120", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:34.529141+0000 mon.a (mon.0) 3358 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2024-02-19T21:15:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:34.529759+0000 mon.a (mon.0) 3359 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-120"}]: dispatch 2024-02-19T21:15:34.948 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:15:34.951 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:34.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:15:35.052 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:35.081 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:35.100 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:35.101 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:15:35.102 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:35.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:35.204 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:35.215 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:35.231 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:35.249 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:35.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:35.353 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:35.381 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:35.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:33.887751+0000 osd.1 (osd.1) 298 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:15:35.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:33.888715+0000 osd.1 (osd.1) 299 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:15:35.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:35.130870+0000 osd.3 (osd.3) 275 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:15:35.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:35.131762+0000 osd.3 (osd.3) 276 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:15:35.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:35.530407+0000 mon.a (mon.0) 3360 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-120"}]': finished 2024-02-19T21:15:35.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:35.530468+0000 mon.a (mon.0) 3361 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2024-02-19T21:15:35.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:35.532222+0000 mon.a (mon.0) 3362 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-120", "mode": "writeback"}]: dispatch 2024-02-19T21:15:35.861 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:33.887751+0000 osd.1 (osd.1) 298 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:15:35.861 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:33.888715+0000 osd.1 (osd.1) 299 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:15:35.861 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:35.130870+0000 osd.3 (osd.3) 275 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:15:35.861 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:35.131762+0000 osd.3 (osd.3) 276 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:15:35.861 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:35.530407+0000 mon.a (mon.0) 3360 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-120"}]': finished 2024-02-19T21:15:35.861 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:35.530468+0000 mon.a (mon.0) 3361 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2024-02-19T21:15:35.861 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:35.532222+0000 mon.a (mon.0) 3362 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-120", "mode": "writeback"}]: dispatch 2024-02-19T21:15:35.861 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:33.887751+0000 osd.1 (osd.1) 298 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:15:35.861 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:33.888715+0000 osd.1 (osd.1) 299 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:15:35.861 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:35.130870+0000 osd.3 (osd.3) 275 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:15:35.861 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:35.131762+0000 osd.3 (osd.3) 276 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:15:35.862 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:35.530407+0000 mon.a (mon.0) 3360 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-120"}]': finished 2024-02-19T21:15:35.862 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:35.530468+0000 mon.a (mon.0) 3361 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2024-02-19T21:15:35.862 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:35.532222+0000 mon.a (mon.0) 3362 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-120", "mode": "writeback"}]: dispatch 2024-02-19T21:15:36.174 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:15:36.175 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:15:36.175 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-02-19T21:15:36.175 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-02-19T21:15:36.175 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:15:36.330 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:36.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:36.433 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:36.463 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:36.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:35.003397+0000 osd.5 (osd.5) 272 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:15:36.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:35.006285+0000 osd.5 (osd.5) 273 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:15:36.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:35.306825+0000 mgr.y (mgr.14550) 583 : cluster [DBG] pgmap v1034: 214 pgs: 214 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:15:36.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:35.941823+0000 osd.7 (osd.7) 315 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:15:36.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:35.944738+0000 osd.7 (osd.7) 316 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:15:36.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:36.136216+0000 osd.3 (osd.3) 277 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:15:36.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:36.137004+0000 osd.3 (osd.3) 278 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:15:36.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:36.527734+0000 mon.a (mon.0) 3363 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:36.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:36.531890+0000 mon.a (mon.0) 3364 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-120", "mode": "writeback"}]': finished 2024-02-19T21:15:36.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:36.531948+0000 mon.a (mon.0) 3365 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2024-02-19T21:15:36.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:35.003397+0000 osd.5 (osd.5) 272 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:15:36.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:35.006285+0000 osd.5 (osd.5) 273 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:15:36.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:35.306825+0000 mgr.y (mgr.14550) 583 : cluster [DBG] pgmap v1034: 214 pgs: 214 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:15:36.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:35.941823+0000 osd.7 (osd.7) 315 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:15:36.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:35.944738+0000 osd.7 (osd.7) 316 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:15:36.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:36.136216+0000 osd.3 (osd.3) 277 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:15:36.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:36.137004+0000 osd.3 (osd.3) 278 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:15:36.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:36.527734+0000 mon.a (mon.0) 3363 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:36.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:36.531890+0000 mon.a (mon.0) 3364 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-120", "mode": "writeback"}]': finished 2024-02-19T21:15:36.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:36.531948+0000 mon.a (mon.0) 3365 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2024-02-19T21:15:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:35.003397+0000 osd.5 (osd.5) 272 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:15:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:35.006285+0000 osd.5 (osd.5) 273 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:15:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:35.306825+0000 mgr.y (mgr.14550) 583 : cluster [DBG] pgmap v1034: 214 pgs: 214 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:15:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:35.941823+0000 osd.7 (osd.7) 315 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:15:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:35.944738+0000 osd.7 (osd.7) 316 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:15:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:36.136216+0000 osd.3 (osd.3) 277 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:15:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:36.137004+0000 osd.3 (osd.3) 278 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:15:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:36.527734+0000 mon.a (mon.0) 3363 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:36.531890+0000 mon.a (mon.0) 3364 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-120", "mode": "writeback"}]': finished 2024-02-19T21:15:36.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:36.531948+0000 mon.a (mon.0) 3365 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2024-02-19T21:15:36.869 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-19T21:15:36.911 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:36.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:36.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:37.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:15:37.047 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:37.068 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:15:37.071 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:37.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:15:37.172 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:37.199 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:37.304 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:37.322 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:37.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:37.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:37.427 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:15:37.455 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:37.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:36.238733+0000 osd.6 (osd.6) 229 : cluster [DBG] 16.5 scrub starts 2024-02-19T21:15:37.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:36.239925+0000 osd.6 (osd.6) 230 : cluster [DBG] 16.5 scrub ok 2024-02-19T21:15:37.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:36.604530+0000 mon.a (mon.0) 3366 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:37.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:37.107356+0000 osd.3 (osd.3) 279 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:15:37.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:37.108216+0000 osd.3 (osd.3) 280 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:15:37.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:36.238733+0000 osd.6 (osd.6) 229 : cluster [DBG] 16.5 scrub starts 2024-02-19T21:15:37.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:36.239925+0000 osd.6 (osd.6) 230 : cluster [DBG] 16.5 scrub ok 2024-02-19T21:15:37.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:36.604530+0000 mon.a (mon.0) 3366 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:37.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:37.107356+0000 osd.3 (osd.3) 279 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:15:37.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:37.108216+0000 osd.3 (osd.3) 280 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:15:37.821 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:15:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:15:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:15:37.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:36.238733+0000 osd.6 (osd.6) 229 : cluster [DBG] 16.5 scrub starts 2024-02-19T21:15:37.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:36.239925+0000 osd.6 (osd.6) 230 : cluster [DBG] 16.5 scrub ok 2024-02-19T21:15:37.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:36.604530+0000 mon.a (mon.0) 3366 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:37.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:37.107356+0000 osd.3 (osd.3) 279 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:15:37.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:37.108216+0000 osd.3 (osd.3) 280 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:15:37.881 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:15:38.104 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:15:38.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:38.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:15:38.208 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:38.238 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:38.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:36.827354+0000 osd.1 (osd.1) 300 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:15:38.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:36.830373+0000 osd.1 (osd.1) 301 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:15:38.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:37.008323+0000 osd.5 (osd.5) 274 : cluster [DBG] 6.d scrub starts 2024-02-19T21:15:38.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:37.011255+0000 osd.5 (osd.5) 275 : cluster [DBG] 6.d scrub ok 2024-02-19T21:15:38.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:37.307235+0000 mgr.y (mgr.14550) 584 : cluster [DBG] pgmap v1037: 214 pgs: 214 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:38.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:37.550213+0000 mon.a (mon.0) 3367 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:38.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:37.550279+0000 mon.a (mon.0) 3368 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2024-02-19T21:15:38.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:37.552376+0000 mon.a (mon.0) 3369 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-120"}]: dispatch 2024-02-19T21:15:38.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:38.119940+0000 osd.3 (osd.3) 281 : cluster [DBG] 16.1a scrub starts 2024-02-19T21:15:38.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:38.121446+0000 osd.3 (osd.3) 282 : cluster [DBG] 16.1a scrub ok 2024-02-19T21:15:38.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:36.827354+0000 osd.1 (osd.1) 300 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:15:38.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:36.830373+0000 osd.1 (osd.1) 301 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:15:38.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:37.008323+0000 osd.5 (osd.5) 274 : cluster [DBG] 6.d scrub starts 2024-02-19T21:15:38.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:37.011255+0000 osd.5 (osd.5) 275 : cluster [DBG] 6.d scrub ok 2024-02-19T21:15:38.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:37.307235+0000 mgr.y (mgr.14550) 584 : cluster [DBG] pgmap v1037: 214 pgs: 214 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:38.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:37.550213+0000 mon.a (mon.0) 3367 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:38.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:37.550279+0000 mon.a (mon.0) 3368 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2024-02-19T21:15:38.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:37.552376+0000 mon.a (mon.0) 3369 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-120"}]: dispatch 2024-02-19T21:15:38.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:38.119940+0000 osd.3 (osd.3) 281 : cluster [DBG] 16.1a scrub starts 2024-02-19T21:15:38.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:38.121446+0000 osd.3 (osd.3) 282 : cluster [DBG] 16.1a scrub ok 2024-02-19T21:15:38.757 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:38.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:36.827354+0000 osd.1 (osd.1) 300 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:15:38.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:36.830373+0000 osd.1 (osd.1) 301 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:15:38.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:37.008323+0000 osd.5 (osd.5) 274 : cluster [DBG] 6.d scrub starts 2024-02-19T21:15:38.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:37.011255+0000 osd.5 (osd.5) 275 : cluster [DBG] 6.d scrub ok 2024-02-19T21:15:38.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:37.307235+0000 mgr.y (mgr.14550) 584 : cluster [DBG] pgmap v1037: 214 pgs: 214 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:38.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:37.550213+0000 mon.a (mon.0) 3367 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:38.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:37.550279+0000 mon.a (mon.0) 3368 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2024-02-19T21:15:38.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:37.552376+0000 mon.a (mon.0) 3369 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-120"}]: dispatch 2024-02-19T21:15:38.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:38.119940+0000 osd.3 (osd.3) 281 : cluster [DBG] 16.1a scrub starts 2024-02-19T21:15:38.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:38.121446+0000 osd.3 (osd.3) 282 : cluster [DBG] 16.1a scrub ok 2024-02-19T21:15:39.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.029+0000 7fda7ad5d700 1 -- 172.21.15.163:0/386899243 >> v1:172.21.15.163:6789/0 conn(0x7fda740fd980 legacy=0x7fda740fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:39.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.029+0000 7fda7ad5d700 1 -- 172.21.15.163:0/386899243 shutdown_connections 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.029+0000 7fda7ad5d700 1 -- 172.21.15.163:0/386899243 >> 172.21.15.163:0/386899243 conn(0x7fda740f35a0 msgr2=0x7fda740f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.029+0000 7fda7ad5d700 1 -- 172.21.15.163:0/386899243 shutdown_connections 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.029+0000 7fda7ad5d700 1 -- 172.21.15.163:0/386899243 wait complete. 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.029+0000 7fda7ad5d700 1 librados: starting msgr at 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.029+0000 7fda7ad5d700 1 librados: starting objecter 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.029+0000 7fda7ad5d700 1 Processor -- start 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda7ad5d700 1 -- start start 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda7ad5d700 1 librados: setting wanted keys 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda7ad5d700 1 librados: calling monclient init 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda7ad5d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fda741172c0 con 0x7fda741042c0 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda7ad5d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fda741184a0 con 0x7fda74100780 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda7ad5d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fda741196a0 con 0x7fda740fd980 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda78af9700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fda740fd980 0x7fda740f9920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:34250/0 (socket says 172.21.15.163:34250) 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda78af9700 1 -- 172.21.15.163:0/1997512087 learned_addr learned my addr 172.21.15.163:0/1997512087 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 10 client.?.objecter ms_handle_connect 0x7fda740fd980 2024-02-19T21:15:39.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:39.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 10 client.?.objecter ms_handle_connect 0x7fda741042c0 2024-02-19T21:15:39.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:39.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223563250 0 0) 0x7fda741172c0 con 0x7fda741042c0 2024-02-19T21:15:39.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda64003800 con 0x7fda741042c0 2024-02-19T21:15:39.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020286861 0 0) 0x7fda741196a0 con 0x7fda740fd980 2024-02-19T21:15:39.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda741172c0 con 0x7fda740fd980 2024-02-19T21:15:39.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 10 client.?.objecter ms_handle_connect 0x7fda74100780 2024-02-19T21:15:39.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:39.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596731771 0 0) 0x7fda741184a0 con 0x7fda74100780 2024-02-19T21:15:39.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda741196a0 con 0x7fda74100780 2024-02-19T21:15:39.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 714196714 0 0) 0x7fda741172c0 con 0x7fda740fd980 2024-02-19T21:15:39.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda741184a0 con 0x7fda740fd980 2024-02-19T21:15:39.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fda68002f60 con 0x7fda740fd980 2024-02-19T21:15:39.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2919811219 0 0) 0x7fda64003800 con 0x7fda741042c0 2024-02-19T21:15:39.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda741172c0 con 0x7fda741042c0 2024-02-19T21:15:39.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fda6c0019a0 con 0x7fda741042c0 2024-02-19T21:15:39.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 780305795 0 0) 0x7fda741196a0 con 0x7fda74100780 2024-02-19T21:15:39.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda64003800 con 0x7fda74100780 2024-02-19T21:15:39.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.030+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1245726337 0 0) 0x7fda741184a0 con 0x7fda740fd980 2024-02-19T21:15:39.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.031+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 >> v1:172.21.15.193:6789/0 conn(0x7fda74100780 legacy=0x7fda740fa030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:39.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.031+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 >> v1:172.21.15.163:6789/0 conn(0x7fda741042c0 legacy=0x7fda74115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:39.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.031+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7411a8a0 con 0x7fda740fd980 2024-02-19T21:15:39.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.031+0000 7fda7ad5d700 1 -- 172.21.15.163:0/1997512087 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda741174b0 con 0x7fda740fd980 2024-02-19T21:15:39.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.031+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fda680024f0 con 0x7fda740fd980 2024-02-19T21:15:39.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.031+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fda680038c0 con 0x7fda740fd980 2024-02-19T21:15:39.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.031+0000 7fda7ad5d700 10 client.50426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:39.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.031+0000 7fda7ad5d700 1 -- 172.21.15.163:0/1997512087 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda741179a0 con 0x7fda740fd980 2024-02-19T21:15:39.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.031+0000 7fda7ad5d700 1 librados: init done 2024-02-19T21:15:39.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.032+0000 7fda7ad5d700 1 -- 172.21.15.163:0/1997512087 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda7405f660 con 0x7fda740fd980 2024-02-19T21:15:39.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.032+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fda68002660 con 0x7fda740fd980 2024-02-19T21:15:39.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.033+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(689..689 src has 623..689) v4 ==== 8555+0+0 (unknown 289833997 0 0) 0x7fda680792d0 con 0x7fda740fd980 2024-02-19T21:15:39.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.033+0000 7fda71ffb700 10 client.50426.objecter ms_dispatch 0x7fda74001550 osd_map(689..689 src has 623..689) v4 2024-02-19T21:15:39.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.033+0000 7fda71ffb700 3 client.50426.objecter handle_osd_map got epochs [689,689] > 0 2024-02-19T21:15:39.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.033+0000 7fda71ffb700 3 client.50426.objecter handle_osd_map decoding full epoch 689 2024-02-19T21:15:39.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.033+0000 7fda71ffb700 20 client.50426.objecter dump_active .. 0 homeless 2024-02-19T21:15:39.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.033+0000 7fda71ffb700 10 client.50426.objecter ms_handle_connect 0x7fda64063c40 2024-02-19T21:15:39.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.037+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fda6804c5b0 con 0x7fda740fd980 2024-02-19T21:15:39.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.238+0000 7fda7ad5d700 1 -- 172.21.15.163:0/1997512087 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7fda74109f40 con 0x7fda740fd980 2024-02-19T21:15:39.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.385+0000 7fda71ffb700 1 -- 172.21.15.163:0/1997512087 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v690) v1 ==== 113+0+0 (unknown 774386471 0 0) 0x7fda6804aa20 con 0x7fda740fd980 2024-02-19T21:15:39.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.388+0000 7fda7ad5d700 10 librados: watch_flush enter 2024-02-19T21:15:39.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.388+0000 7fda7ad5d700 10 librados: watch_flush exit 2024-02-19T21:15:39.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.388+0000 7fda7ad5d700 20 client.50426.objecter shutdown clearing up homeless session... 2024-02-19T21:15:39.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.388+0000 7fda7ad5d700 10 client.50426.objecter successfully canceled tick 2024-02-19T21:15:39.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.389+0000 7fda7ad5d700 1 -- 172.21.15.163:0/1997512087 >> v1:172.21.15.163:6800/1552514681 conn(0x7fda64063c40 legacy=0x7fda64066100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:39.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.389+0000 7fda7ad5d700 1 -- 172.21.15.163:0/1997512087 >> v1:172.21.15.163:6790/0 conn(0x7fda740fd980 legacy=0x7fda740f9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:39.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.389+0000 7fda7ad5d700 1 -- 172.21.15.163:0/1997512087 shutdown_connections 2024-02-19T21:15:39.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.389+0000 7fda7ad5d700 1 -- 172.21.15.163:0/1997512087 >> 172.21.15.163:0/1997512087 conn(0x7fda740f35a0 msgr2=0x7fda74107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:39.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.390+0000 7fda7ad5d700 1 -- 172.21.15.163:0/1997512087 shutdown_connections 2024-02-19T21:15:39.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.390+0000 7fda7ad5d700 1 -- 172.21.15.163:0/1997512087 wait complete. 2024-02-19T21:15:39.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:39.390+0000 7fda7ad5d700 1 librados: shutdown 2024-02-19T21:15:39.392 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.4 to 1 (10000) 2024-02-19T21:15:39.712 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:37.994860+0000 osd.5 (osd.5) 276 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:15:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:37.998306+0000 osd.5 (osd.5) 277 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:15:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:38.352056+0000 osd.2 (osd.2) 272 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:15:39.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:38.352996+0000 osd.2 (osd.2) 273 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:15:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:38.548453+0000 mon.a (mon.0) 3370 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:38.553237+0000 mon.a (mon.0) 3371 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-120"}]': finished 2024-02-19T21:15:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:38.553317+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2024-02-19T21:15:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:39.108458+0000 mon.a (mon.0) 3373 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:39.145575+0000 osd.3 (osd.3) 283 : cluster [DBG] 16.14 scrub starts 2024-02-19T21:15:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:39.146757+0000 osd.3 (osd.3) 284 : cluster [DBG] 16.14 scrub ok 2024-02-19T21:15:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:39.239752+0000 mon.c (mon.2) 424 : audit [INF] from='client.? v1:172.21.15.163:0/1997512087' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:15:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:39.240054+0000 mon.a (mon.0) 3374 : audit [INF] from='client.50426 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:15:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:39.385441+0000 mon.a (mon.0) 3375 : audit [INF] from='client.50426 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-19T21:15:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:39.385507+0000 mon.a (mon.0) 3376 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2024-02-19T21:15:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:39.387282+0000 mon.b (mon.1) 741 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2024-02-19T21:15:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:39.387644+0000 mon.b (mon.1) 742 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.27", "id": [7, 2]}]: dispatch 2024-02-19T21:15:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:39.387906+0000 mon.b (mon.1) 743 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.29", "id": [7, 2]}]: dispatch 2024-02-19T21:15:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:39.390405+0000 mon.a (mon.0) 3377 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2024-02-19T21:15:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:39.390689+0000 mon.a (mon.0) 3378 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.27", "id": [7, 2]}]: dispatch 2024-02-19T21:15:39.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:39.390920+0000 mon.a (mon.0) 3379 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.29", "id": [7, 2]}]: dispatch 2024-02-19T21:15:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:37.994860+0000 osd.5 (osd.5) 276 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:15:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:37.998306+0000 osd.5 (osd.5) 277 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:15:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:38.352056+0000 osd.2 (osd.2) 272 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:15:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:38.352996+0000 osd.2 (osd.2) 273 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:15:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:38.548453+0000 mon.a (mon.0) 3370 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:38.553237+0000 mon.a (mon.0) 3371 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-120"}]': finished 2024-02-19T21:15:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:38.553317+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2024-02-19T21:15:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:39.108458+0000 mon.a (mon.0) 3373 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:39.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:39.145575+0000 osd.3 (osd.3) 283 : cluster [DBG] 16.14 scrub starts 2024-02-19T21:15:39.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:39.146757+0000 osd.3 (osd.3) 284 : cluster [DBG] 16.14 scrub ok 2024-02-19T21:15:39.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:39.239752+0000 mon.c (mon.2) 424 : audit [INF] from='client.? v1:172.21.15.163:0/1997512087' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:15:39.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:39.240054+0000 mon.a (mon.0) 3374 : audit [INF] from='client.50426 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:15:39.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:39.385441+0000 mon.a (mon.0) 3375 : audit [INF] from='client.50426 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-19T21:15:39.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:39.385507+0000 mon.a (mon.0) 3376 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2024-02-19T21:15:39.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:39.387282+0000 mon.b (mon.1) 741 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2024-02-19T21:15:39.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:39.387644+0000 mon.b (mon.1) 742 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.27", "id": [7, 2]}]: dispatch 2024-02-19T21:15:39.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:39.387906+0000 mon.b (mon.1) 743 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.29", "id": [7, 2]}]: dispatch 2024-02-19T21:15:39.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:39.390405+0000 mon.a (mon.0) 3377 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2024-02-19T21:15:39.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:39.390689+0000 mon.a (mon.0) 3378 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.27", "id": [7, 2]}]: dispatch 2024-02-19T21:15:39.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:39.390920+0000 mon.a (mon.0) 3379 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.29", "id": [7, 2]}]: dispatch 2024-02-19T21:15:40.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:37.994860+0000 osd.5 (osd.5) 276 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:15:40.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:37.998306+0000 osd.5 (osd.5) 277 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:15:40.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:38.352056+0000 osd.2 (osd.2) 272 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:15:40.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:38.352996+0000 osd.2 (osd.2) 273 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:15:40.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:38.548453+0000 mon.a (mon.0) 3370 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:40.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:38.553237+0000 mon.a (mon.0) 3371 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-120"}]': finished 2024-02-19T21:15:40.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:38.553317+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2024-02-19T21:15:40.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:39.108458+0000 mon.a (mon.0) 3373 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:40.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:39.145575+0000 osd.3 (osd.3) 283 : cluster [DBG] 16.14 scrub starts 2024-02-19T21:15:40.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:39.146757+0000 osd.3 (osd.3) 284 : cluster [DBG] 16.14 scrub ok 2024-02-19T21:15:40.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:39.239752+0000 mon.c (mon.2) 424 : audit [INF] from='client.? v1:172.21.15.163:0/1997512087' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:15:40.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:39.240054+0000 mon.a (mon.0) 3374 : audit [INF] from='client.50426 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:15:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:39.385441+0000 mon.a (mon.0) 3375 : audit [INF] from='client.50426 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-19T21:15:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:39.385507+0000 mon.a (mon.0) 3376 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2024-02-19T21:15:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:39.387282+0000 mon.b (mon.1) 741 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2024-02-19T21:15:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:39.387644+0000 mon.b (mon.1) 742 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.27", "id": [7, 2]}]: dispatch 2024-02-19T21:15:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:39.387906+0000 mon.b (mon.1) 743 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.29", "id": [7, 2]}]: dispatch 2024-02-19T21:15:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:39.390405+0000 mon.a (mon.0) 3377 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2024-02-19T21:15:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:39.390689+0000 mon.a (mon.0) 3378 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.27", "id": [7, 2]}]: dispatch 2024-02-19T21:15:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:39.390920+0000 mon.a (mon.0) 3379 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.29", "id": [7, 2]}]: dispatch 2024-02-19T21:15:40.157 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 6 1 2024-02-19T21:15:40.942 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:38.896703+0000 osd.1 (osd.1) 302 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:15:40.943 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:38.900429+0000 osd.1 (osd.1) 303 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:15:40.943 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:39.235183+0000 osd.6 (osd.6) 231 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:15:40.943 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:39.237238+0000 osd.6 (osd.6) 232 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:15:40.943 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:39.307733+0000 mgr.y (mgr.14550) 585 : cluster [DBG] pgmap v1040: 214 pgs: 214 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:15:40.943 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:40.385925+0000 mon.a (mon.0) 3380 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]': finished 2024-02-19T21:15:40.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:40.385993+0000 mon.a (mon.0) 3381 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.27", "id": [7, 2]}]': finished 2024-02-19T21:15:40.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:40.386037+0000 mon.a (mon.0) 3382 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.29", "id": [7, 2]}]': finished 2024-02-19T21:15:40.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:40.386092+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2024-02-19T21:15:40.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:40.391574+0000 mon.a (mon.0) 3384 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:40.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:38.896703+0000 osd.1 (osd.1) 302 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:15:40.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:38.900429+0000 osd.1 (osd.1) 303 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:15:40.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:39.235183+0000 osd.6 (osd.6) 231 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:15:40.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:39.237238+0000 osd.6 (osd.6) 232 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:15:40.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:39.307733+0000 mgr.y (mgr.14550) 585 : cluster [DBG] pgmap v1040: 214 pgs: 214 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:15:40.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:40.385925+0000 mon.a (mon.0) 3380 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]': finished 2024-02-19T21:15:40.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:40.385993+0000 mon.a (mon.0) 3381 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.27", "id": [7, 2]}]': finished 2024-02-19T21:15:40.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:40.386037+0000 mon.a (mon.0) 3382 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.29", "id": [7, 2]}]': finished 2024-02-19T21:15:40.945 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:40.386092+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2024-02-19T21:15:40.945 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:40.391574+0000 mon.a (mon.0) 3384 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:41.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:15:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:15:41.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:38.896703+0000 osd.1 (osd.1) 302 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:15:41.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:38.900429+0000 osd.1 (osd.1) 303 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:15:41.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:39.235183+0000 osd.6 (osd.6) 231 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:15:41.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:39.237238+0000 osd.6 (osd.6) 232 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:15:41.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:39.307733+0000 mgr.y (mgr.14550) 585 : cluster [DBG] pgmap v1040: 214 pgs: 214 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:15:41.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:40.385925+0000 mon.a (mon.0) 3380 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]': finished 2024-02-19T21:15:41.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:40.385993+0000 mon.a (mon.0) 3381 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.27", "id": [7, 2]}]': finished 2024-02-19T21:15:41.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:40.386037+0000 mon.a (mon.0) 3382 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.29", "id": [7, 2]}]': finished 2024-02-19T21:15:41.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:40.386092+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2024-02-19T21:15:41.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:40.391574+0000 mon.a (mon.0) 3384 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:39.978075+0000 osd.5 (osd.5) 278 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:15:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:39.980978+0000 osd.5 (osd.5) 279 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:15:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:40.354396+0000 osd.2 (osd.2) 274 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:15:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:40.357294+0000 osd.2 (osd.2) 275 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:15:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:40.703721+0000 mgr.y (mgr.14550) 586 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:41.080692+0000 osd.3 (osd.3) 285 : cluster [DBG] 5.13 scrub starts 2024-02-19T21:15:41.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:41.081433+0000 osd.3 (osd.3) 286 : cluster [DBG] 5.13 scrub ok 2024-02-19T21:15:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:41.388980+0000 mon.a (mon.0) 3385 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:41.389143+0000 mon.a (mon.0) 3386 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2024-02-19T21:15:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:41.391107+0000 mon.a (mon.0) 3387 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:41.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:39.978075+0000 osd.5 (osd.5) 278 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:15:41.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:39.980978+0000 osd.5 (osd.5) 279 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:15:41.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:40.354396+0000 osd.2 (osd.2) 274 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:15:41.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:40.357294+0000 osd.2 (osd.2) 275 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:15:41.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:40.703721+0000 mgr.y (mgr.14550) 586 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:41.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:41.080692+0000 osd.3 (osd.3) 285 : cluster [DBG] 5.13 scrub starts 2024-02-19T21:15:41.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:41.081433+0000 osd.3 (osd.3) 286 : cluster [DBG] 5.13 scrub ok 2024-02-19T21:15:42.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:41.388980+0000 mon.a (mon.0) 3385 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:42.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:41.389143+0000 mon.a (mon.0) 3386 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2024-02-19T21:15:42.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:41.391107+0000 mon.a (mon.0) 3387 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:42.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:39.978075+0000 osd.5 (osd.5) 278 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:15:42.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:39.980978+0000 osd.5 (osd.5) 279 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:15:42.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:40.354396+0000 osd.2 (osd.2) 274 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:15:42.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:40.357294+0000 osd.2 (osd.2) 275 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:15:42.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:40.703721+0000 mgr.y (mgr.14550) 586 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:42.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:41.080692+0000 osd.3 (osd.3) 285 : cluster [DBG] 5.13 scrub starts 2024-02-19T21:15:42.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:41.081433+0000 osd.3 (osd.3) 286 : cluster [DBG] 5.13 scrub ok 2024-02-19T21:15:42.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:41.388980+0000 mon.a (mon.0) 3385 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:42.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:41.389143+0000 mon.a (mon.0) 3386 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2024-02-19T21:15:42.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:41.391107+0000 mon.a (mon.0) 3387 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:42.287 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:42.297 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:42.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:42.408 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:15:42.440 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:42.461 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:15:42.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:42.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:15:42.565 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:15:42.594 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:42.614 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:15:42.616 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:42.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:15:42.718 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:15:42.744 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:40.993970+0000 osd.5 (osd.5) 280 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:15:42.744 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:40.997587+0000 osd.5 (osd.5) 281 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:15:42.744 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:41.308196+0000 mgr.y (mgr.14550) 587 : cluster [DBG] pgmap v1043: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:42.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:41.828036+0000 osd.4 (osd.4) 241 : cluster [DBG] 6.17 scrub starts 2024-02-19T21:15:42.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:41.829063+0000 osd.4 (osd.4) 242 : cluster [DBG] 6.17 scrub ok 2024-02-19T21:15:42.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:41.877127+0000 osd.7 (osd.7) 317 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:15:42.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:41.880073+0000 osd.7 (osd.7) 318 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:15:42.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:42.045699+0000 osd.3 (osd.3) 287 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:15:42.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:42.047794+0000 osd.3 (osd.3) 288 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:15:42.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:42.387137+0000 mon.a (mon.0) 3388 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-122", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:42.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:42.387238+0000 mon.a (mon.0) 3389 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2024-02-19T21:15:42.745 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:42.387976+0000 mon.a (mon.0) 3390 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-122"}]: dispatch 2024-02-19T21:15:42.746 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:42.766 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:15:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:42.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:15:42.870 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:42.900 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:42.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:40.993970+0000 osd.5 (osd.5) 280 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:15:42.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:40.997587+0000 osd.5 (osd.5) 281 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:15:42.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:41.308196+0000 mgr.y (mgr.14550) 587 : cluster [DBG] pgmap v1043: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:42.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:41.828036+0000 osd.4 (osd.4) 241 : cluster [DBG] 6.17 scrub starts 2024-02-19T21:15:42.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:41.829063+0000 osd.4 (osd.4) 242 : cluster [DBG] 6.17 scrub ok 2024-02-19T21:15:42.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:41.877127+0000 osd.7 (osd.7) 317 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:15:42.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:41.880073+0000 osd.7 (osd.7) 318 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:15:42.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:42.045699+0000 osd.3 (osd.3) 287 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:15:42.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:42.047794+0000 osd.3 (osd.3) 288 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:15:42.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:42.387137+0000 mon.a (mon.0) 3388 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-122", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:42.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:42.387238+0000 mon.a (mon.0) 3389 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2024-02-19T21:15:42.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:42.387976+0000 mon.a (mon.0) 3390 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-122"}]: dispatch 2024-02-19T21:15:42.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:40.993970+0000 osd.5 (osd.5) 280 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:15:42.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:40.997587+0000 osd.5 (osd.5) 281 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:15:42.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:41.308196+0000 mgr.y (mgr.14550) 587 : cluster [DBG] pgmap v1043: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:42.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:41.828036+0000 osd.4 (osd.4) 241 : cluster [DBG] 6.17 scrub starts 2024-02-19T21:15:42.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:41.829063+0000 osd.4 (osd.4) 242 : cluster [DBG] 6.17 scrub ok 2024-02-19T21:15:42.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:41.877127+0000 osd.7 (osd.7) 317 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:15:42.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:41.880073+0000 osd.7 (osd.7) 318 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:15:43.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:42.045699+0000 osd.3 (osd.3) 287 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:15:43.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:42.047794+0000 osd.3 (osd.3) 288 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:15:43.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:42.387137+0000 mon.a (mon.0) 3388 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-122", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:43.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:42.387238+0000 mon.a (mon.0) 3389 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2024-02-19T21:15:43.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:42.387976+0000 mon.a (mon.0) 3390 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-122"}]: dispatch 2024-02-19T21:15:43.024 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:43.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:43.128 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:43.157 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:43.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.649+0000 7fef7a6b3700 1 -- 172.21.15.163:0/2210752291 >> v1:172.21.15.163:6789/0 conn(0x7fef741031b0 legacy=0x7fef74105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:43.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.649+0000 7fef7a6b3700 1 -- 172.21.15.163:0/2210752291 shutdown_connections 2024-02-19T21:15:43.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.649+0000 7fef7a6b3700 1 -- 172.21.15.163:0/2210752291 >> 172.21.15.163:0/2210752291 conn(0x7fef740f2470 msgr2=0x7fef740f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:43.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.649+0000 7fef7a6b3700 1 -- 172.21.15.163:0/2210752291 shutdown_connections 2024-02-19T21:15:43.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.649+0000 7fef7a6b3700 1 -- 172.21.15.163:0/2210752291 wait complete. 2024-02-19T21:15:43.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.649+0000 7fef7a6b3700 1 librados: starting msgr at 2024-02-19T21:15:43.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.649+0000 7fef7a6b3700 1 librados: starting objecter 2024-02-19T21:15:43.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.649+0000 7fef7a6b3700 1 Processor -- start 2024-02-19T21:15:43.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.649+0000 7fef7a6b3700 1 -- start start 2024-02-19T21:15:43.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.649+0000 7fef7a6b3700 1 librados: setting wanted keys 2024-02-19T21:15:43.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.649+0000 7fef7a6b3700 1 librados: calling monclient init 2024-02-19T21:15:43.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.649+0000 7fef7a6b3700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fef74117180 con 0x7fef741031b0 2024-02-19T21:15:43.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.649+0000 7fef7a6b3700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fef74118360 con 0x7fef740ff670 2024-02-19T21:15:43.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.649+0000 7fef7a6b3700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fef74119560 con 0x7fef740fc870 2024-02-19T21:15:43.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef78c50700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fef741031b0 0x7fef74115840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:51586/0 (socket says 172.21.15.163:51586) 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef78c50700 1 -- 172.21.15.163:0/1316750101 learned_addr learned my addr 172.21.15.163:0/1316750101 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 10 client.?.objecter ms_handle_connect 0x7fef741031b0 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 10 client.?.objecter ms_handle_connect 0x7fef740fc870 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 10 client.?.objecter ms_handle_connect 0x7fef740ff670 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574941040 0 0) 0x7fef74117180 con 0x7fef741031b0 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef600038e0 con 0x7fef741031b0 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515608607 0 0) 0x7fef74119560 con 0x7fef740fc870 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef74117180 con 0x7fef740fc870 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333825172 0 0) 0x7fef74118360 con 0x7fef740ff670 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef74119560 con 0x7fef740ff670 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3649756157 0 0) 0x7fef74117180 con 0x7fef740fc870 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef74118360 con 0x7fef740fc870 2024-02-19T21:15:43.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fef64001840 con 0x7fef740fc870 2024-02-19T21:15:43.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1677909708 0 0) 0x7fef74119560 con 0x7fef740ff670 2024-02-19T21:15:43.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef74117180 con 0x7fef740ff670 2024-02-19T21:15:43.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 280300780 0 0) 0x7fef600038e0 con 0x7fef741031b0 2024-02-19T21:15:43.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef74119560 con 0x7fef741031b0 2024-02-19T21:15:43.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fef5c001800 con 0x7fef740ff670 2024-02-19T21:15:43.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fef6c001950 con 0x7fef741031b0 2024-02-19T21:15:43.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 720945871 0 0) 0x7fef74118360 con 0x7fef740fc870 2024-02-19T21:15:43.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 >> v1:172.21.15.193:6789/0 conn(0x7fef740ff670 legacy=0x7fef740f9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:43.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 >> v1:172.21.15.163:6789/0 conn(0x7fef741031b0 legacy=0x7fef74115840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:43.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7411a760 con 0x7fef740fc870 2024-02-19T21:15:43.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef74119750 con 0x7fef740fc870 2024-02-19T21:15:43.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef7a6b3700 10 client.50432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:43.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef74119c40 con 0x7fef740fc870 2024-02-19T21:15:43.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.650+0000 7fef7a6b3700 1 librados: init done 2024-02-19T21:15:43.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.651+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fef64001f50 con 0x7fef740fc870 2024-02-19T21:15:43.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.651+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fef64003310 con 0x7fef740fc870 2024-02-19T21:15:43.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.651+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fef54004f00 con 0x7fef740fc870 2024-02-19T21:15:43.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.652+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fef64001ab0 con 0x7fef740fc870 2024-02-19T21:15:43.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.652+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(694..694 src has 623..694) v4 ==== 8650+0+0 (unknown 301445139 0 0) 0x7fef640766b0 con 0x7fef740fc870 2024-02-19T21:15:43.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.652+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 osd_map(694..694 src has 623..694) v4 2024-02-19T21:15:43.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.652+0000 7fef71ffb700 3 client.50432.objecter handle_osd_map got epochs [694,694] > 0 2024-02-19T21:15:43.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.652+0000 7fef71ffb700 3 client.50432.objecter handle_osd_map decoding full epoch 694 2024-02-19T21:15:43.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.652+0000 7fef71ffb700 20 client.50432.objecter dump_active .. 0 homeless 2024-02-19T21:15:43.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.652+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v694) v1 ==== 54+0+16 (unknown 1113608185 0 2420000192) 0x7fef64078ac0 con 0x7fef740fc870 2024-02-19T21:15:43.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.652+0000 7fef71ffb700 10 client.50432.objecter ms_handle_connect 0x7fef60063db0 2024-02-19T21:15:43.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:43.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 1 2024-02-19T21:15:43.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef540051a0 osd=0 3 2024-02-19T21:15:43.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef540051a0 osd=0 3 2024-02-19T21:15:43.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef540051a0 osd=0 3 2024-02-19T21:15:43.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 1 2024-02-19T21:15:43.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef540051a0 osd=0 3 2024-02-19T21:15:43.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 0 1 2024-02-19T21:15:43.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef540051a0 osd=0 4 2024-02-19T21:15:43.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef7a6b3700 10 client.50432.objecter _send_command 1 2024-02-19T21:15:43.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef54004950 con 0x7fef54004530 2024-02-19T21:15:43.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef71ffb700 10 client.50432.objecter ms_handle_connect 0x7fef54004530 2024-02-19T21:15:43.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fef54004950 con 0x7fef54004530 2024-02-19T21:15:43.658 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:15:43.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef71ffb700 10 client.50432.objecter _finish_command 1 = system:0 2024-02-19T21:15:43.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef540051a0 osd=0 4 2024-02-19T21:15:43.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.653+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 0 1 2024-02-19T21:15:43.667 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:15:43] "GET /metrics HTTP/1.1" 200 192121 "" "Prometheus/2.33.4" 2024-02-19T21:15:43.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:41.999744+0000 osd.5 (osd.5) 282 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:15:43.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:42.003376+0000 osd.5 (osd.5) 283 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:15:43.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:42.265337+0000 osd.2 (osd.2) 276 : cluster [DBG] 6.f scrub starts 2024-02-19T21:15:43.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:42.266579+0000 osd.2 (osd.2) 277 : cluster [DBG] 6.f scrub ok 2024-02-19T21:15:43.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:43.052366+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:15:43.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:43.054046+0000 osd.3 (osd.3) 290 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:15:43.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:43.391378+0000 mon.a (mon.0) 3391 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-122"}]': finished 2024-02-19T21:15:43.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:43.391438+0000 mon.a (mon.0) 3392 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2024-02-19T21:15:43.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:43.393495+0000 mon.a (mon.0) 3393 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-122", "mode": "writeback"}]: dispatch 2024-02-19T21:15:43.669 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:43.653604+0000 mon.c (mon.2) 425 : audit [DBG] from='client.? v1:172.21.15.163:0/1316750101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:43.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.669+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:43.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.669+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 2 2024-02-19T21:15:43.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.669+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef540051a0 osd=0 3 2024-02-19T21:15:43.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.669+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef540051a0 osd=0 3 2024-02-19T21:15:43.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.669+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef540051a0 osd=0 3 2024-02-19T21:15:43.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.669+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 2 2024-02-19T21:15:43.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.669+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef540051a0 osd=0 3 2024-02-19T21:15:43.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.669+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 0 2 2024-02-19T21:15:43.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.669+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef540051a0 osd=0 4 2024-02-19T21:15:43.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.669+0000 7fef7a6b3700 10 client.50432.objecter _send_command 2 2024-02-19T21:15:43.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.669+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fef54004d30 con 0x7fef54004530 2024-02-19T21:15:43.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fef54004d30 con 0x7fef54004530 2024-02-19T21:15:43.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:43.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef71ffb700 10 client.50432.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef540051a0 osd=0 4 2024-02-19T21:15:43.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 0 2 2024-02-19T21:15:43.672 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:15:43.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:43.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 3 2024-02-19T21:15:43.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54003fe0 osd=1 3 2024-02-19T21:15:43.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54003fe0 osd=1 3 2024-02-19T21:15:43.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54003fe0 osd=1 3 2024-02-19T21:15:43.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 3 2024-02-19T21:15:43.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef54003fe0 osd=1 3 2024-02-19T21:15:43.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 1 3 2024-02-19T21:15:43.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54003fe0 osd=1 4 2024-02-19T21:15:43.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef7a6b3700 10 client.50432.objecter _send_command 3 2024-02-19T21:15:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef5400c000 con 0x7fef54004150 2024-02-19T21:15:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.671+0000 7fef71ffb700 10 client.50432.objecter ms_handle_connect 0x7fef54004150 2024-02-19T21:15:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.672+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fef5400c000 con 0x7fef54004150 2024-02-19T21:15:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.672+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:15:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.672+0000 7fef71ffb700 10 client.50432.objecter _finish_command 3 = system:0 2024-02-19T21:15:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.672+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef54003fe0 osd=1 4 2024-02-19T21:15:43.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.672+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 1 3 2024-02-19T21:15:43.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.686+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:43.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.686+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 4 2024-02-19T21:15:43.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.686+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54003fe0 osd=1 3 2024-02-19T21:15:43.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.686+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54003fe0 osd=1 3 2024-02-19T21:15:43.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.686+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54003fe0 osd=1 3 2024-02-19T21:15:43.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.686+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 4 2024-02-19T21:15:43.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.686+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef54003fe0 osd=1 3 2024-02-19T21:15:43.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.686+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 1 4 2024-02-19T21:15:43.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.686+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54003fe0 osd=1 4 2024-02-19T21:15:43.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.686+0000 7fef7a6b3700 10 client.50432.objecter _send_command 4 2024-02-19T21:15:43.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.686+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fef540018b0 con 0x7fef54004150 2024-02-19T21:15:43.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fef540018b0 con 0x7fef54004150 2024-02-19T21:15:43.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:43.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef71ffb700 10 client.50432.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef54003fe0 osd=1 4 2024-02-19T21:15:43.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 1 4 2024-02-19T21:15:43.689 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:15:43.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:43.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 5 2024-02-19T21:15:43.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54003630 osd=2 3 2024-02-19T21:15:43.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54003630 osd=2 3 2024-02-19T21:15:43.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54003630 osd=2 3 2024-02-19T21:15:43.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 5 2024-02-19T21:15:43.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef54003630 osd=2 3 2024-02-19T21:15:43.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 2 5 2024-02-19T21:15:43.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54003630 osd=2 4 2024-02-19T21:15:43.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef7a6b3700 10 client.50432.objecter _send_command 5 2024-02-19T21:15:43.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef54003e70 con 0x7fef5400d2f0 2024-02-19T21:15:43.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.688+0000 7fef71ffb700 10 client.50432.objecter ms_handle_connect 0x7fef5400d2f0 2024-02-19T21:15:43.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.689+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fef54003e70 con 0x7fef5400d2f0 2024-02-19T21:15:43.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.689+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:15:43.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.689+0000 7fef71ffb700 10 client.50432.objecter _finish_command 5 = system:0 2024-02-19T21:15:43.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.689+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef54003630 osd=2 4 2024-02-19T21:15:43.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.689+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 2 5 2024-02-19T21:15:43.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.703+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:43.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.703+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 6 2024-02-19T21:15:43.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.703+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54003630 osd=2 3 2024-02-19T21:15:43.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.703+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54003630 osd=2 3 2024-02-19T21:15:43.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.703+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54003630 osd=2 3 2024-02-19T21:15:43.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.703+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 6 2024-02-19T21:15:43.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.703+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef54003630 osd=2 3 2024-02-19T21:15:43.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.703+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 2 6 2024-02-19T21:15:43.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.703+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54003630 osd=2 4 2024-02-19T21:15:43.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.703+0000 7fef7a6b3700 10 client.50432.objecter _send_command 6 2024-02-19T21:15:43.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.703+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fef5400e730 con 0x7fef5400d2f0 2024-02-19T21:15:43.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fef5400e730 con 0x7fef5400d2f0 2024-02-19T21:15:43.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:43.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef71ffb700 10 client.50432.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef54003630 osd=2 4 2024-02-19T21:15:43.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 2 6 2024-02-19T21:15:43.706 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:15:43.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:43.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 7 2024-02-19T21:15:43.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54003410 osd=3 3 2024-02-19T21:15:43.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54003410 osd=3 3 2024-02-19T21:15:43.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54003410 osd=3 3 2024-02-19T21:15:43.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 7 2024-02-19T21:15:43.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef54003410 osd=3 3 2024-02-19T21:15:43.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 3 7 2024-02-19T21:15:43.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54003410 osd=3 4 2024-02-19T21:15:43.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef7a6b3700 10 client.50432.objecter _send_command 7 2024-02-19T21:15:43.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef54015a90 con 0x7fef54011e80 2024-02-19T21:15:43.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.705+0000 7fef71ffb700 10 client.50432.objecter ms_handle_connect 0x7fef54011e80 2024-02-19T21:15:43.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.706+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fef54015a90 con 0x7fef54011e80 2024-02-19T21:15:43.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.706+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:15:43.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.706+0000 7fef71ffb700 10 client.50432.objecter _finish_command 7 = system:0 2024-02-19T21:15:43.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.706+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef54003410 osd=3 4 2024-02-19T21:15:43.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.706+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 3 7 2024-02-19T21:15:43.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.720+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:43.720 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.720+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 8 2024-02-19T21:15:43.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.720+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54003410 osd=3 3 2024-02-19T21:15:43.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.720+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54003410 osd=3 3 2024-02-19T21:15:43.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.720+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54003410 osd=3 3 2024-02-19T21:15:43.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.720+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 8 2024-02-19T21:15:43.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.720+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef54003410 osd=3 3 2024-02-19T21:15:43.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.720+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 3 8 2024-02-19T21:15:43.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.720+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54003410 osd=3 4 2024-02-19T21:15:43.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.720+0000 7fef7a6b3700 10 client.50432.objecter _send_command 8 2024-02-19T21:15:43.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.720+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fef5400e1a0 con 0x7fef54011e80 2024-02-19T21:15:43.722 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:15:43.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fef5400e1a0 con 0x7fef54011e80 2024-02-19T21:15:43.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef71ffb700 10 client.50432.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef54003410 osd=3 4 2024-02-19T21:15:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 3 8 2024-02-19T21:15:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 9 2024-02-19T21:15:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef5400df60 osd=4 3 2024-02-19T21:15:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef5400df60 osd=4 3 2024-02-19T21:15:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef5400df60 osd=4 3 2024-02-19T21:15:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 9 2024-02-19T21:15:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef5400df60 osd=4 3 2024-02-19T21:15:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 4 9 2024-02-19T21:15:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef5400df60 osd=4 4 2024-02-19T21:15:43.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef7a6b3700 10 client.50432.objecter _send_command 9 2024-02-19T21:15:43.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.721+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef5400e100 con 0x7fef5400ce50 2024-02-19T21:15:43.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.722+0000 7fef71ffb700 10 client.50432.objecter ms_handle_connect 0x7fef5400ce50 2024-02-19T21:15:43.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.723+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fef5400e100 con 0x7fef5400ce50 2024-02-19T21:15:43.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.723+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:15:43.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.723+0000 7fef71ffb700 10 client.50432.objecter _finish_command 9 = system:0 2024-02-19T21:15:43.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.723+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef5400df60 osd=4 4 2024-02-19T21:15:43.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.723+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 4 9 2024-02-19T21:15:43.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.737+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:43.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.737+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 10 2024-02-19T21:15:43.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.737+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef5400df60 osd=4 3 2024-02-19T21:15:43.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.737+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef5400df60 osd=4 3 2024-02-19T21:15:43.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.737+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef5400df60 osd=4 3 2024-02-19T21:15:43.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.737+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 10 2024-02-19T21:15:43.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.737+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef5400df60 osd=4 3 2024-02-19T21:15:43.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.737+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 4 10 2024-02-19T21:15:43.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.737+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef5400df60 osd=4 4 2024-02-19T21:15:43.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.737+0000 7fef7a6b3700 10 client.50432.objecter _send_command 10 2024-02-19T21:15:43.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.737+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fef5400e100 con 0x7fef5400ce50 2024-02-19T21:15:43.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fef5400e100 con 0x7fef5400ce50 2024-02-19T21:15:43.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:43.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef71ffb700 10 client.50432.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef5400df60 osd=4 4 2024-02-19T21:15:43.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 4 10 2024-02-19T21:15:43.740 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:15:43.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:43.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 11 2024-02-19T21:15:43.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef5400e100 osd=5 3 2024-02-19T21:15:43.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef5400e100 osd=5 3 2024-02-19T21:15:43.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef5400e100 osd=5 3 2024-02-19T21:15:43.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 11 2024-02-19T21:15:43.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef5400e100 osd=5 3 2024-02-19T21:15:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 5 11 2024-02-19T21:15:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef5400e100 osd=5 4 2024-02-19T21:15:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef7a6b3700 10 client.50432.objecter _send_command 11 2024-02-19T21:15:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef54017fc0 con 0x7fef54017a60 2024-02-19T21:15:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef71ffb700 10 client.50432.objecter ms_handle_connect 0x7fef54017a60 2024-02-19T21:15:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fef54017fc0 con 0x7fef54017a60 2024-02-19T21:15:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:15:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef71ffb700 10 client.50432.objecter _finish_command 11 = system:0 2024-02-19T21:15:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef5400e100 osd=5 4 2024-02-19T21:15:43.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.740+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 5 11 2024-02-19T21:15:43.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.755+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:43.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.755+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 12 2024-02-19T21:15:43.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.755+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef5400e100 osd=5 3 2024-02-19T21:15:43.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.755+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef5400e100 osd=5 3 2024-02-19T21:15:43.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.755+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef5400e100 osd=5 3 2024-02-19T21:15:43.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.755+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 12 2024-02-19T21:15:43.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.755+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef5400e100 osd=5 3 2024-02-19T21:15:43.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.755+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 5 12 2024-02-19T21:15:43.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.755+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef5400e100 osd=5 4 2024-02-19T21:15:43.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.755+0000 7fef7a6b3700 10 client.50432.objecter _send_command 12 2024-02-19T21:15:43.756 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.755+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fef54017fc0 con 0x7fef54017a60 2024-02-19T21:15:43.757 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fef54017fc0 con 0x7fef54017a60 2024-02-19T21:15:43.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:43.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef71ffb700 10 client.50432.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef5400e100 osd=5 4 2024-02-19T21:15:43.758 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 5 12 2024-02-19T21:15:43.758 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:15:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 13 2024-02-19T21:15:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54017fc0 osd=6 3 2024-02-19T21:15:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54017fc0 osd=6 3 2024-02-19T21:15:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54017fc0 osd=6 3 2024-02-19T21:15:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 13 2024-02-19T21:15:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef54017fc0 osd=6 3 2024-02-19T21:15:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 6 13 2024-02-19T21:15:43.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54017fc0 osd=6 4 2024-02-19T21:15:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef7a6b3700 10 client.50432.objecter _send_command 13 2024-02-19T21:15:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef54026fe0 con 0x7fef540234a0 2024-02-19T21:15:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef71ffb700 10 client.50432.objecter ms_handle_connect 0x7fef540234a0 2024-02-19T21:15:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fef54026fe0 con 0x7fef540234a0 2024-02-19T21:15:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:15:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef71ffb700 10 client.50432.objecter _finish_command 13 = system:0 2024-02-19T21:15:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef54017fc0 osd=6 4 2024-02-19T21:15:43.760 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.758+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 6 13 2024-02-19T21:15:43.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.773+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:43.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.773+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 14 2024-02-19T21:15:43.773 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.773+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54017fc0 osd=6 3 2024-02-19T21:15:43.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.773+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54017fc0 osd=6 3 2024-02-19T21:15:43.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.773+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54017fc0 osd=6 3 2024-02-19T21:15:43.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.773+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 14 2024-02-19T21:15:43.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.773+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef54017fc0 osd=6 3 2024-02-19T21:15:43.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.773+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 6 14 2024-02-19T21:15:43.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.773+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54017fc0 osd=6 4 2024-02-19T21:15:43.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.773+0000 7fef7a6b3700 10 client.50432.objecter _send_command 14 2024-02-19T21:15:43.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.773+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fef54026fe0 con 0x7fef540234a0 2024-02-19T21:15:43.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fef54026fe0 con 0x7fef540234a0 2024-02-19T21:15:43.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:43.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef71ffb700 10 client.50432.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef54017fc0 osd=6 4 2024-02-19T21:15:43.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 6 14 2024-02-19T21:15:43.776 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:15:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 15 2024-02-19T21:15:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54026fe0 osd=7 3 2024-02-19T21:15:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54026fe0 osd=7 3 2024-02-19T21:15:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54026fe0 osd=7 3 2024-02-19T21:15:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 15 2024-02-19T21:15:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef54026fe0 osd=7 3 2024-02-19T21:15:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 7 15 2024-02-19T21:15:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54026fe0 osd=7 4 2024-02-19T21:15:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef7a6b3700 10 client.50432.objecter _send_command 15 2024-02-19T21:15:43.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef540281c0 con 0x7fef54028ca0 2024-02-19T21:15:43.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.775+0000 7fef71ffb700 10 client.50432.objecter ms_handle_connect 0x7fef54028ca0 2024-02-19T21:15:43.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.776+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fef540281c0 con 0x7fef54028ca0 2024-02-19T21:15:43.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.776+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:15:43.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.776+0000 7fef71ffb700 10 client.50432.objecter _finish_command 15 = system:0 2024-02-19T21:15:43.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.776+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef54026fe0 osd=7 4 2024-02-19T21:15:43.778 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.776+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 7 15 2024-02-19T21:15:43.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.791+0000 7fef7a6b3700 10 client.50432.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:15:43.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.791+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign -1 16 2024-02-19T21:15:43.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.791+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54026fe0 osd=7 3 2024-02-19T21:15:43.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.791+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54026fe0 osd=7 3 2024-02-19T21:15:43.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.791+0000 7fef7a6b3700 20 client.50432.objecter _get_session s=0x7fef54026fe0 osd=7 3 2024-02-19T21:15:43.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.791+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_remove -1 16 2024-02-19T21:15:43.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.791+0000 7fef7a6b3700 20 client.50432.objecter get_session s=0x7fef54026fe0 osd=7 3 2024-02-19T21:15:43.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.791+0000 7fef7a6b3700 15 client.50432.objecter _session_command_op_assign 7 16 2024-02-19T21:15:43.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.791+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54026fe0 osd=7 4 2024-02-19T21:15:43.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.791+0000 7fef7a6b3700 10 client.50432.objecter _send_command 16 2024-02-19T21:15:43.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.791+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fef540281c0 con 0x7fef54028ca0 2024-02-19T21:15:43.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef71ffb700 1 -- 172.21.15.163:0/1316750101 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fef540281c0 con 0x7fef54028ca0 2024-02-19T21:15:43.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef71ffb700 10 client.50432.objecter ms_dispatch 0x7fef74001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:15:43.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef71ffb700 10 client.50432.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef71ffb700 20 client.50432.objecter put_session s=0x7fef54026fe0 osd=7 4 2024-02-19T21:15:43.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef71ffb700 15 client.50432.objecter _session_command_op_remove 7 16 2024-02-19T21:15:43.795 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:15:43.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 10 librados: watch_flush enter 2024-02-19T21:15:43.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 10 librados: watch_flush exit 2024-02-19T21:15:43.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 10 client.50432.objecter close_session for osd.0 2024-02-19T21:15:43.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 >> v1:172.21.15.163:6801/1559418115 conn(0x7fef54004530 legacy=0x7fef54002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:43.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef540051a0 osd=0 1 2024-02-19T21:15:43.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 10 client.50432.objecter close_session for osd.1 2024-02-19T21:15:43.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 >> v1:172.21.15.163:6805/3125047432 conn(0x7fef54004150 legacy=0x7fef5400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:43.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54003fe0 osd=1 1 2024-02-19T21:15:43.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 10 client.50432.objecter close_session for osd.2 2024-02-19T21:15:43.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 >> v1:172.21.15.163:6809/1563158487 conn(0x7fef5400d2f0 legacy=0x7fef5400d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54003630 osd=2 1 2024-02-19T21:15:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 10 client.50432.objecter close_session for osd.3 2024-02-19T21:15:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 >> v1:172.21.15.163:6813/2112279769 conn(0x7fef54011e80 legacy=0x7fef54014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54003410 osd=3 1 2024-02-19T21:15:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 10 client.50432.objecter close_session for osd.4 2024-02-19T21:15:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 >> v1:172.21.15.193:6800/3129716965 conn(0x7fef5400ce50 legacy=0x7fef54015c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef5400df60 osd=4 1 2024-02-19T21:15:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 10 client.50432.objecter close_session for osd.5 2024-02-19T21:15:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 >> v1:172.21.15.193:6804/805161991 conn(0x7fef54017a60 legacy=0x7fef5401d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef78c50700 1 -- 172.21.15.163:0/1316750101 reap_dead start 2024-02-19T21:15:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef5400e100 osd=5 1 2024-02-19T21:15:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 10 client.50432.objecter close_session for osd.6 2024-02-19T21:15:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 >> v1:172.21.15.193:6808/787604615 conn(0x7fef540234a0 legacy=0x7fef54025890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef78c50700 1 -- 172.21.15.163:0/1316750101 reap_dead start 2024-02-19T21:15:43.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54017fc0 osd=6 1 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 10 client.50432.objecter close_session for osd.7 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 >> v1:172.21.15.193:6812/157927432 conn(0x7fef54028ca0 legacy=0x7fef540290e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 20 client.50432.objecter put_session s=0x7fef54026fe0 osd=7 1 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 20 client.50432.objecter shutdown clearing up homeless session... 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 10 client.50432.objecter successfully canceled tick 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 >> v1:172.21.15.163:6800/1552514681 conn(0x7fef60063db0 legacy=0x7fef60066270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef78c50700 1 -- 172.21.15.163:0/1316750101 reap_dead start 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef78c50700 1 -- 172.21.15.163:0/1316750101 reap_dead start 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 >> v1:172.21.15.163:6790/0 conn(0x7fef740fc870 legacy=0x7fef740f9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 shutdown_connections 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 >> 172.21.15.163:0/1316750101 conn(0x7fef740f2470 msgr2=0x7fef74106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 shutdown_connections 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 1 -- 172.21.15.163:0/1316750101 wait complete. 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:43.794+0000 7fef7a6b3700 1 librados: shutdown 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.798 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.799 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:15:43.924 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:41.999744+0000 osd.5 (osd.5) 282 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:15:43.924 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:42.003376+0000 osd.5 (osd.5) 283 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:15:43.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:42.265337+0000 osd.2 (osd.2) 276 : cluster [DBG] 6.f scrub starts 2024-02-19T21:15:43.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:42.266579+0000 osd.2 (osd.2) 277 : cluster [DBG] 6.f scrub ok 2024-02-19T21:15:43.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:43.052366+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:15:43.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:43.054046+0000 osd.3 (osd.3) 290 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:15:43.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:43.391378+0000 mon.a (mon.0) 3391 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-122"}]': finished 2024-02-19T21:15:43.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:43.391438+0000 mon.a (mon.0) 3392 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2024-02-19T21:15:43.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:43.393495+0000 mon.a (mon.0) 3393 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-122", "mode": "writeback"}]: dispatch 2024-02-19T21:15:43.925 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:43.653604+0000 mon.c (mon.2) 425 : audit [DBG] from='client.? v1:172.21.15.163:0/1316750101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:44.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:41.999744+0000 osd.5 (osd.5) 282 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:15:44.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:42.003376+0000 osd.5 (osd.5) 283 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:15:44.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:42.265337+0000 osd.2 (osd.2) 276 : cluster [DBG] 6.f scrub starts 2024-02-19T21:15:44.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:42.266579+0000 osd.2 (osd.2) 277 : cluster [DBG] 6.f scrub ok 2024-02-19T21:15:44.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:43.052366+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:15:44.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:43.054046+0000 osd.3 (osd.3) 290 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:15:44.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:43.391378+0000 mon.a (mon.0) 3391 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-122"}]': finished 2024-02-19T21:15:44.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:43.391438+0000 mon.a (mon.0) 3392 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2024-02-19T21:15:44.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:43.393495+0000 mon.a (mon.0) 3393 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-122", "mode": "writeback"}]: dispatch 2024-02-19T21:15:44.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:43.653604+0000 mon.c (mon.2) 425 : audit [DBG] from='client.? v1:172.21.15.163:0/1316750101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:44.307 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:44.310 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:44.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:44.411 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:15:44.440 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:44.460 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:15:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:44.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:15:44.564 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:44.592 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:44.612 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:44.614 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:44.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:44.715 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:44.747 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:42.970577+0000 osd.5 (osd.5) 284 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:15:44.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:42.974031+0000 osd.5 (osd.5) 285 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:15:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:43.160672+0000 osd.6 (osd.6) 233 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:15:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:43.162997+0000 osd.6 (osd.6) 234 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:15:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:43.308759+0000 mgr.y (mgr.14550) 588 : cluster [DBG] pgmap v1046: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:43.847732+0000 osd.4 (osd.4) 243 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:15:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:43.849911+0000 osd.4 (osd.4) 244 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:15:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:44.386378+0000 mon.a (mon.0) 3394 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:44.389967+0000 mon.a (mon.0) 3395 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-122", "mode": "writeback"}]': finished 2024-02-19T21:15:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:44.390075+0000 mon.a (mon.0) 3396 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2024-02-19T21:15:44.870 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:44.873 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:44.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:44.974 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:44.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:42.970577+0000 osd.5 (osd.5) 284 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:15:44.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:42.974031+0000 osd.5 (osd.5) 285 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:15:44.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:43.160672+0000 osd.6 (osd.6) 233 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:15:44.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:43.162997+0000 osd.6 (osd.6) 234 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:15:44.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:43.308759+0000 mgr.y (mgr.14550) 588 : cluster [DBG] pgmap v1046: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:44.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:43.847732+0000 osd.4 (osd.4) 243 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:15:44.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:43.849911+0000 osd.4 (osd.4) 244 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:15:44.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:44.386378+0000 mon.a (mon.0) 3394 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:44.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:44.389967+0000 mon.a (mon.0) 3395 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-122", "mode": "writeback"}]': finished 2024-02-19T21:15:44.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:44.390075+0000 mon.a (mon.0) 3396 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2024-02-19T21:15:44.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:42.970577+0000 osd.5 (osd.5) 284 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:15:44.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:42.974031+0000 osd.5 (osd.5) 285 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:15:44.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:43.160672+0000 osd.6 (osd.6) 233 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:15:44.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:43.162997+0000 osd.6 (osd.6) 234 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:15:44.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:43.308759+0000 mgr.y (mgr.14550) 588 : cluster [DBG] pgmap v1046: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:44.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:43.847732+0000 osd.4 (osd.4) 243 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:15:44.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:43.849911+0000 osd.4 (osd.4) 244 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:15:45.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:44.386378+0000 mon.a (mon.0) 3394 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:45.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:44.389967+0000 mon.a (mon.0) 3395 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-122", "mode": "writeback"}]': finished 2024-02-19T21:15:45.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:44.390075+0000 mon.a (mon.0) 3396 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2024-02-19T21:15:45.002 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:45.726 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:15:45.726 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:15:45.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:44.842026+0000 osd.4 (osd.4) 245 : cluster [DBG] 6.2 scrub starts 2024-02-19T21:15:45.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:44.842884+0000 osd.4 (osd.4) 246 : cluster [DBG] 6.2 scrub ok 2024-02-19T21:15:45.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:44.860908+0000 osd.3 (osd.3) 291 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:15:45.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:44.861067+0000 osd.3 (osd.3) 292 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:15:45.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:44.944644+0000 osd.1 (osd.1) 304 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:15:45.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:44.948528+0000 osd.1 (osd.1) 305 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:15:45.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:44.961525+0000 osd.5 (osd.5) 286 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:15:45.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:44.964425+0000 osd.5 (osd.5) 287 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:15:45.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:45.390794+0000 mon.a (mon.0) 3397 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2024-02-19T21:15:45.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:44.842026+0000 osd.4 (osd.4) 245 : cluster [DBG] 6.2 scrub starts 2024-02-19T21:15:45.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:44.842884+0000 osd.4 (osd.4) 246 : cluster [DBG] 6.2 scrub ok 2024-02-19T21:15:45.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:44.860908+0000 osd.3 (osd.3) 291 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:15:45.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:44.861067+0000 osd.3 (osd.3) 292 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:15:45.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:44.944644+0000 osd.1 (osd.1) 304 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:15:45.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:44.948528+0000 osd.1 (osd.1) 305 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:15:45.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:44.961525+0000 osd.5 (osd.5) 286 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:15:45.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:44.964425+0000 osd.5 (osd.5) 287 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:15:45.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:45.390794+0000 mon.a (mon.0) 3397 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2024-02-19T21:15:46.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:44.842026+0000 osd.4 (osd.4) 245 : cluster [DBG] 6.2 scrub starts 2024-02-19T21:15:46.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:44.842884+0000 osd.4 (osd.4) 246 : cluster [DBG] 6.2 scrub ok 2024-02-19T21:15:46.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:44.860908+0000 osd.3 (osd.3) 291 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:15:46.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:44.861067+0000 osd.3 (osd.3) 292 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:15:46.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:44.944644+0000 osd.1 (osd.1) 304 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:15:46.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:44.948528+0000 osd.1 (osd.1) 305 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:15:46.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:44.961525+0000 osd.5 (osd.5) 286 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:15:46.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:44.964425+0000 osd.5 (osd.5) 287 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:15:46.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:45.390794+0000 mon.a (mon.0) 3397 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2024-02-19T21:15:46.352 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:46.355 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:46.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:46.372 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-19T21:15:46.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:46.485 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:46.652 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:15:46.684 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:46.686 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:46.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:46.787 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:15:46.815 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:46.836 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:15:46.839 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:46.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:15:46.871 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:15:46.939 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:45.309730+0000 mgr.y (mgr.14550) 589 : cluster [DBG] pgmap v1049: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:46.939 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:45.956453+0000 osd.1 (osd.1) 306 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:15:46.939 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:45.957812+0000 osd.1 (osd.1) 307 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:15:46.939 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:46.398170+0000 mon.a (mon.0) 3398 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2024-02-19T21:15:46.939 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:46.434333+0000 mon.a (mon.0) 3399 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:46.941 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:15:46.970 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:46.989 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:15:46.992 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:46.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:15:46.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:45.309730+0000 mgr.y (mgr.14550) 589 : cluster [DBG] pgmap v1049: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:46.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:45.956453+0000 osd.1 (osd.1) 306 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:15:46.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:45.957812+0000 osd.1 (osd.1) 307 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:15:46.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:46.398170+0000 mon.a (mon.0) 3398 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2024-02-19T21:15:46.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:46.434333+0000 mon.a (mon.0) 3399 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:46.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:45.309730+0000 mgr.y (mgr.14550) 589 : cluster [DBG] pgmap v1049: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:46.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:45.956453+0000 osd.1 (osd.1) 306 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:15:46.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:45.957812+0000 osd.1 (osd.1) 307 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:15:46.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:46.398170+0000 mon.a (mon.0) 3398 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2024-02-19T21:15:46.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:46.434333+0000 mon.a (mon.0) 3399 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:47.093 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:47.119 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:47.241 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:47.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:47.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:47.347 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:47.375 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:47.428 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:47.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:47.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.947+0000 7f632cbdb700 1 -- 172.21.15.163:0/3869168128 >> v1:172.21.15.193:6789/0 conn(0x7f63280fbf60 legacy=0x7f63280fc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:47.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.948+0000 7f632cbdb700 1 -- 172.21.15.163:0/3869168128 shutdown_connections 2024-02-19T21:15:47.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.948+0000 7f632cbdb700 1 -- 172.21.15.163:0/3869168128 >> 172.21.15.163:0/3869168128 conn(0x7f632806cb90 msgr2=0x7f63280f3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:47.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.949+0000 7f632cbdb700 1 -- 172.21.15.163:0/3869168128 shutdown_connections 2024-02-19T21:15:47.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.949+0000 7f632cbdb700 1 -- 172.21.15.163:0/3869168128 wait complete. 2024-02-19T21:15:47.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.950+0000 7f632cbdb700 1 librados: starting msgr at 2024-02-19T21:15:47.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.950+0000 7f632cbdb700 1 librados: starting objecter 2024-02-19T21:15:47.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.950+0000 7f632cbdb700 1 Processor -- start 2024-02-19T21:15:47.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.950+0000 7f632cbdb700 1 -- start start 2024-02-19T21:15:47.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.950+0000 7f632cbdb700 1 librados: setting wanted keys 2024-02-19T21:15:47.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.950+0000 7f632cbdb700 1 librados: calling monclient init 2024-02-19T21:15:47.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f632cbdb700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6328117040 con 0x7f63281028c0 2024-02-19T21:15:47.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f632cbdb700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6328118220 con 0x7f63280fedb0 2024-02-19T21:15:47.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f632cbdb700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6328119420 con 0x7f63280fbf60 2024-02-19T21:15:47.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f6326d9d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f63281028c0 0x7f6328115700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:38148/0 (socket says 172.21.15.163:38148) 2024-02-19T21:15:47.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f6326d9d700 1 -- 172.21.15.163:0/1492407308 learned_addr learned my addr 172.21.15.163:0/1492407308 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:47.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 10 client.?.objecter ms_handle_connect 0x7f63281028c0 2024-02-19T21:15:47.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:47.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 10 client.?.objecter ms_handle_connect 0x7f63280fbf60 2024-02-19T21:15:47.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:47.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 10 client.?.objecter ms_handle_connect 0x7f63280fedb0 2024-02-19T21:15:47.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:47.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186957412 0 0) 0x7f6328117040 con 0x7f63281028c0 2024-02-19T21:15:47.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6314004a60 con 0x7f63281028c0 2024-02-19T21:15:47.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016505857 0 0) 0x7f6328118220 con 0x7f63280fedb0 2024-02-19T21:15:47.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6328117040 con 0x7f63280fedb0 2024-02-19T21:15:47.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520023851 0 0) 0x7f6328119420 con 0x7f63280fbf60 2024-02-19T21:15:47.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6328118220 con 0x7f63280fbf60 2024-02-19T21:15:47.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2627022598 0 0) 0x7f6314004a60 con 0x7f63281028c0 2024-02-19T21:15:47.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6328119420 con 0x7f63281028c0 2024-02-19T21:15:47.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6320001c40 con 0x7f63281028c0 2024-02-19T21:15:47.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 501099139 0 0) 0x7f6328117040 con 0x7f63280fedb0 2024-02-19T21:15:47.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6314004a60 con 0x7f63280fedb0 2024-02-19T21:15:47.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.951+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f63100019f0 con 0x7f63280fedb0 2024-02-19T21:15:47.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.952+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 168178520 0 0) 0x7f6328119420 con 0x7f63281028c0 2024-02-19T21:15:47.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.952+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 >> v1:172.21.15.163:6790/0 conn(0x7f63280fbf60 legacy=0x7f63280fb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:47.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.952+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 >> v1:172.21.15.193:6789/0 conn(0x7f63280fedb0 legacy=0x7f6328111f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:47.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.952+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632811a620 con 0x7f63281028c0 2024-02-19T21:15:47.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.952+0000 7f632cbdb700 1 -- 172.21.15.163:0/1492407308 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6328119610 con 0x7f63281028c0 2024-02-19T21:15:47.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.952+0000 7f632cbdb700 10 client.23721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:47.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.952+0000 7f632cbdb700 1 -- 172.21.15.163:0/1492407308 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6328119a80 con 0x7f63281028c0 2024-02-19T21:15:47.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.952+0000 7f632cbdb700 1 librados: init done 2024-02-19T21:15:47.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.953+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6320002700 con 0x7f63281028c0 2024-02-19T21:15:47.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.953+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6320005db0 con 0x7f63281028c0 2024-02-19T21:15:47.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.953+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6320018d20 con 0x7f63281028c0 2024-02-19T21:15:47.523 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.954+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(697..697 src has 623..697) v4 ==== 8650+0+0 (unknown 3670104147 0 0) 0x7f632007a160 con 0x7f63281028c0 2024-02-19T21:15:47.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.954+0000 7f632cbdb700 1 -- 172.21.15.163:0/1492407308 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6308004f20 con 0x7f63281028c0 2024-02-19T21:15:47.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.954+0000 7f631f7fe700 10 client.23721.objecter ms_dispatch 0x7f6328001550 osd_map(697..697 src has 623..697) v4 2024-02-19T21:15:47.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.954+0000 7f631f7fe700 3 client.23721.objecter handle_osd_map got epochs [697,697] > 0 2024-02-19T21:15:47.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.954+0000 7f631f7fe700 3 client.23721.objecter handle_osd_map decoding full epoch 697 2024-02-19T21:15:47.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.954+0000 7f631f7fe700 20 client.23721.objecter dump_active .. 0 homeless 2024-02-19T21:15:47.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.954+0000 7f631f7fe700 10 client.23721.objecter ms_handle_connect 0x7f6314063c40 2024-02-19T21:15:47.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:46.959+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f632004d320 con 0x7f63281028c0 2024-02-19T21:15:47.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.144+0000 7f632cbdb700 1 -- 172.21.15.163:0/1492407308 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f63080051d0 con 0x7f63281028c0 2024-02-19T21:15:47.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.395+0000 7f631f7fe700 1 -- 172.21.15.163:0/1492407308 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v698) v1 ==== 113+0+0 (unknown 1642470700 0 0) 0x7f632004b730 con 0x7f63281028c0 2024-02-19T21:15:47.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.398+0000 7f632cbdb700 10 librados: watch_flush enter 2024-02-19T21:15:47.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.398+0000 7f632cbdb700 10 librados: watch_flush exit 2024-02-19T21:15:47.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.398+0000 7f632cbdb700 20 client.23721.objecter shutdown clearing up homeless session... 2024-02-19T21:15:47.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.398+0000 7f632cbdb700 10 client.23721.objecter successfully canceled tick 2024-02-19T21:15:47.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.398+0000 7f632cbdb700 1 -- 172.21.15.163:0/1492407308 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6314063c40 legacy=0x7f6314066100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:47.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.398+0000 7f632cbdb700 1 -- 172.21.15.163:0/1492407308 >> v1:172.21.15.163:6789/0 conn(0x7f63281028c0 legacy=0x7f6328115700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:47.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.398+0000 7f632cbdb700 1 -- 172.21.15.163:0/1492407308 shutdown_connections 2024-02-19T21:15:47.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.398+0000 7f632cbdb700 1 -- 172.21.15.163:0/1492407308 >> 172.21.15.163:0/1492407308 conn(0x7f632806cb90 msgr2=0x7f6328105520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:47.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.398+0000 7f632cbdb700 1 -- 172.21.15.163:0/1492407308 shutdown_connections 2024-02-19T21:15:47.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.398+0000 7f632cbdb700 1 -- 172.21.15.163:0/1492407308 wait complete. 2024-02-19T21:15:47.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.398+0000 7f632cbdb700 1 librados: shutdown 2024-02-19T21:15:47.525 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.6 to 1 (10000) 2024-02-19T21:15:47.531 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:15:47.557 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:47.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:46.081583+0000 osd.6 (osd.6) 235 : cluster [DBG] 16.d scrub starts 2024-02-19T21:15:47.685 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:46.084702+0000 osd.6 (osd.6) 236 : cluster [DBG] 16.d scrub ok 2024-02-19T21:15:47.965 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:15:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:15:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:15:47.966 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:46.081583+0000 osd.6 (osd.6) 235 : cluster [DBG] 16.d scrub starts 2024-02-19T21:15:47.966 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:46.084702+0000 osd.6 (osd.6) 236 : cluster [DBG] 16.d scrub ok 2024-02-19T21:15:47.966 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:46.225636+0000 osd.2 (osd.2) 278 : cluster [DBG] 16.e scrub starts 2024-02-19T21:15:47.966 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:46.226413+0000 osd.2 (osd.2) 279 : cluster [DBG] 16.e scrub ok 2024-02-19T21:15:47.966 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:46.780903+0000 mon.b (mon.1) 744 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:15:47.966 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:46.941750+0000 osd.1 (osd.1) 308 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:15:47.966 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:46.944766+0000 osd.1 (osd.1) 309 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:15:47.966 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:47.145141+0000 mon.a (mon.0) 3400 : audit [INF] from='client.? v1:172.21.15.163:0/1492407308' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:15:47.966 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:47.395248+0000 mon.a (mon.0) 3401 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:47.966 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:47.395352+0000 mon.a (mon.0) 3402 : audit [INF] from='client.? v1:172.21.15.163:0/1492407308' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-19T21:15:47.967 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:47.395401+0000 mon.a (mon.0) 3403 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2024-02-19T21:15:47.967 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:47.401570+0000 mon.a (mon.0) 3404 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-122"}]: dispatch 2024-02-19T21:15:47.982 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.982+0000 7f9667ba5700 1 -- 172.21.15.163:0/4255565024 >> v1:172.21.15.163:6789/0 conn(0x7f96601042b0 legacy=0x7f9660106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:47.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.982+0000 7f9667ba5700 1 -- 172.21.15.163:0/4255565024 shutdown_connections 2024-02-19T21:15:47.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.982+0000 7f9667ba5700 1 -- 172.21.15.163:0/4255565024 >> 172.21.15.163:0/4255565024 conn(0x7f96600f2470 msgr2=0x7f96600f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:47.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.982+0000 7f9667ba5700 1 -- 172.21.15.163:0/4255565024 shutdown_connections 2024-02-19T21:15:47.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.982+0000 7f9667ba5700 1 -- 172.21.15.163:0/4255565024 wait complete. 2024-02-19T21:15:47.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9667ba5700 1 librados: starting msgr at 2024-02-19T21:15:47.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9667ba5700 1 librados: starting objecter 2024-02-19T21:15:47.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9667ba5700 1 Processor -- start 2024-02-19T21:15:47.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9667ba5700 1 -- start start 2024-02-19T21:15:47.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9667ba5700 1 librados: setting wanted keys 2024-02-19T21:15:47.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9667ba5700 1 librados: calling monclient init 2024-02-19T21:15:47.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9667ba5700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9660102aa0 con 0x7f96600f8920 2024-02-19T21:15:47.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9667ba5700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9660102c70 con 0x7f96600f94d0 2024-02-19T21:15:47.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9667ba5700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9660102e60 con 0x7f96601042b0 2024-02-19T21:15:47.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9665941700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f96600f8920 0x7f9660102300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:38166/0 (socket says 172.21.15.163:38166) 2024-02-19T21:15:47.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9666142700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f96601042b0 0x7f9660118430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:60272/0 (socket says 172.21.15.163:60272) 2024-02-19T21:15:47.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9665941700 1 -- 172.21.15.163:0/2863159766 learned_addr learned my addr 172.21.15.163:0/2863159766 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:47.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9656ffd700 10 client.?.objecter ms_handle_connect 0x7f96601042b0 2024-02-19T21:15:47.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9656ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:47.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9656ffd700 10 client.?.objecter ms_handle_connect 0x7f96600f8920 2024-02-19T21:15:47.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9656ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:47.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9656ffd700 10 client.?.objecter ms_handle_connect 0x7f96600f94d0 2024-02-19T21:15:47.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.983+0000 7f9656ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:47.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431444462 0 0) 0x7f9660102aa0 con 0x7f96600f8920 2024-02-19T21:15:47.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f964c003800 con 0x7f96600f8920 2024-02-19T21:15:47.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84360397 0 0) 0x7f9660102e60 con 0x7f96601042b0 2024-02-19T21:15:47.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9660102aa0 con 0x7f96601042b0 2024-02-19T21:15:47.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416889231 0 0) 0x7f9660102c70 con 0x7f96600f94d0 2024-02-19T21:15:47.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9660102e60 con 0x7f96600f94d0 2024-02-19T21:15:47.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2247858358 0 0) 0x7f964c003800 con 0x7f96600f8920 2024-02-19T21:15:47.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9660102c70 con 0x7f96600f8920 2024-02-19T21:15:47.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f965c001920 con 0x7f96600f8920 2024-02-19T21:15:47.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 56797667 0 0) 0x7f9660102aa0 con 0x7f96601042b0 2024-02-19T21:15:47.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f964c003800 con 0x7f96601042b0 2024-02-19T21:15:47.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9658003290 con 0x7f96601042b0 2024-02-19T21:15:47.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3887375255 0 0) 0x7f9660102e60 con 0x7f96600f94d0 2024-02-19T21:15:47.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9660102aa0 con 0x7f96600f94d0 2024-02-19T21:15:47.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f96500018c0 con 0x7f96600f94d0 2024-02-19T21:15:47.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1202920236 0 0) 0x7f964c003800 con 0x7f96601042b0 2024-02-19T21:15:47.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 >> v1:172.21.15.193:6789/0 conn(0x7f96600f94d0 legacy=0x7f9660114d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:47.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 >> v1:172.21.15.163:6789/0 conn(0x7f96600f8920 legacy=0x7f9660102300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:47.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966011cb80 con 0x7f96601042b0 2024-02-19T21:15:47.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9667ba5700 1 -- 172.21.15.163:0/2863159766 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9660119bb0 con 0x7f96601042b0 2024-02-19T21:15:47.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9667ba5700 10 client.50444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:47.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9667ba5700 1 -- 172.21.15.163:0/2863159766 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f966011a020 con 0x7f96601042b0 2024-02-19T21:15:47.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9667ba5700 1 librados: init done 2024-02-19T21:15:47.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9658001ec0 con 0x7f96601042b0 2024-02-19T21:15:47.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.984+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9658003bf0 con 0x7f96601042b0 2024-02-19T21:15:47.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.985+0000 7f9667ba5700 1 -- 172.21.15.163:0/2863159766 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9644004f20 con 0x7f96601042b0 2024-02-19T21:15:47.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.985+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9658002fd0 con 0x7f96601042b0 2024-02-19T21:15:47.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.986+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(698..698 src has 623..698) v4 ==== 8650+0+0 (unknown 3882206483 0 0) 0x7f96580794a0 con 0x7f96601042b0 2024-02-19T21:15:47.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.986+0000 7f9656ffd700 10 client.50444.objecter ms_dispatch 0x7f9660001550 osd_map(698..698 src has 623..698) v4 2024-02-19T21:15:47.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.986+0000 7f9656ffd700 3 client.50444.objecter handle_osd_map got epochs [698,698] > 0 2024-02-19T21:15:47.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.986+0000 7f9656ffd700 3 client.50444.objecter handle_osd_map decoding full epoch 698 2024-02-19T21:15:47.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.986+0000 7f9656ffd700 20 client.50444.objecter dump_active .. 0 homeless 2024-02-19T21:15:47.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.986+0000 7f9656ffd700 10 client.50444.objecter ms_handle_connect 0x7f964c063d40 2024-02-19T21:15:47.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:47.990+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f965804c6a0 con 0x7f96601042b0 2024-02-19T21:15:47.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:46.225636+0000 osd.2 (osd.2) 278 : cluster [DBG] 16.e scrub starts 2024-02-19T21:15:47.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:46.226413+0000 osd.2 (osd.2) 279 : cluster [DBG] 16.e scrub ok 2024-02-19T21:15:47.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:46.780903+0000 mon.b (mon.1) 744 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:15:47.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:46.941750+0000 osd.1 (osd.1) 308 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:15:47.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:46.944766+0000 osd.1 (osd.1) 309 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:15:47.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:47.145141+0000 mon.a (mon.0) 3400 : audit [INF] from='client.? v1:172.21.15.163:0/1492407308' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:15:47.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:47.395248+0000 mon.a (mon.0) 3401 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:47.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:47.395352+0000 mon.a (mon.0) 3402 : audit [INF] from='client.? v1:172.21.15.163:0/1492407308' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-19T21:15:47.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:47.395401+0000 mon.a (mon.0) 3403 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2024-02-19T21:15:47.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:47.401570+0000 mon.a (mon.0) 3404 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-122"}]: dispatch 2024-02-19T21:15:47.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:46.081583+0000 osd.6 (osd.6) 235 : cluster [DBG] 16.d scrub starts 2024-02-19T21:15:47.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:46.084702+0000 osd.6 (osd.6) 236 : cluster [DBG] 16.d scrub ok 2024-02-19T21:15:47.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:46.225636+0000 osd.2 (osd.2) 278 : cluster [DBG] 16.e scrub starts 2024-02-19T21:15:47.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:46.226413+0000 osd.2 (osd.2) 279 : cluster [DBG] 16.e scrub ok 2024-02-19T21:15:47.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:46.780903+0000 mon.b (mon.1) 744 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:15:47.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:46.941750+0000 osd.1 (osd.1) 308 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:15:47.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:46.944766+0000 osd.1 (osd.1) 309 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:15:47.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:47.145141+0000 mon.a (mon.0) 3400 : audit [INF] from='client.? v1:172.21.15.163:0/1492407308' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:15:47.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:47.395248+0000 mon.a (mon.0) 3401 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:47.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:47.395352+0000 mon.a (mon.0) 3402 : audit [INF] from='client.? v1:172.21.15.163:0/1492407308' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-19T21:15:47.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:47.395401+0000 mon.a (mon.0) 3403 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2024-02-19T21:15:47.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:47.401570+0000 mon.a (mon.0) 3404 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-122"}]: dispatch 2024-02-19T21:15:48.006 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:15:48.008 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:48.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:15:48.109 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:48.137 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:48.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:48.179+0000 7f9667ba5700 1 -- 172.21.15.163:0/2863159766 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f96440051d0 con 0x7f96601042b0 2024-02-19T21:15:48.236 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:48.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:48.340 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:48.368 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:48.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:48.398+0000 7f9656ffd700 1 -- 172.21.15.163:0/2863159766 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v699) v1 ==== 101+0+0 (unknown 141101666 0 0) 0x7f965804ab10 con 0x7f96601042b0 2024-02-19T21:15:48.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:48.401+0000 7f9667ba5700 10 librados: watch_flush enter 2024-02-19T21:15:48.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:48.401+0000 7f9667ba5700 10 librados: watch_flush exit 2024-02-19T21:15:48.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:48.401+0000 7f9667ba5700 20 client.50444.objecter shutdown clearing up homeless session... 2024-02-19T21:15:48.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:48.401+0000 7f9667ba5700 10 client.50444.objecter successfully canceled tick 2024-02-19T21:15:48.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:48.401+0000 7f9667ba5700 1 -- 172.21.15.163:0/2863159766 >> v1:172.21.15.163:6800/1552514681 conn(0x7f964c063d40 legacy=0x7f964c066200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:48.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:48.401+0000 7f9667ba5700 1 -- 172.21.15.163:0/2863159766 >> v1:172.21.15.163:6790/0 conn(0x7f96601042b0 legacy=0x7f9660118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:48.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:48.401+0000 7f9667ba5700 1 -- 172.21.15.163:0/2863159766 shutdown_connections 2024-02-19T21:15:48.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:48.401+0000 7f9667ba5700 1 -- 172.21.15.163:0/2863159766 >> 172.21.15.163:0/2863159766 conn(0x7f96600f2470 msgr2=0x7f9660103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:48.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:48.401+0000 7f9667ba5700 1 -- 172.21.15.163:0/2863159766 shutdown_connections 2024-02-19T21:15:48.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:48.401+0000 7f9667ba5700 1 -- 172.21.15.163:0/2863159766 wait complete. 2024-02-19T21:15:48.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:48.401+0000 7f9667ba5700 1 librados: shutdown 2024-02-19T21:15:48.402 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:15:48.491 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:15:48.492 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:15:48.507 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 0 1 2024-02-19T21:15:48.687 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:47.087284+0000 osd.6 (osd.6) 237 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:15:48.687 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:47.088237+0000 osd.6 (osd.6) 238 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:15:48.687 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:47.310344+0000 mgr.y (mgr.14550) 590 : cluster [DBG] pgmap v1052: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:48.687 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:47.944173+0000 osd.5 (osd.5) 288 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:15:48.687 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:47.947057+0000 osd.5 (osd.5) 289 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:15:48.687 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:48.128967+0000 mon.b (mon.1) 745 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:15:48.688 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:47.087284+0000 osd.6 (osd.6) 237 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:15:48.688 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:47.088237+0000 osd.6 (osd.6) 238 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:15:48.688 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:47.310344+0000 mgr.y (mgr.14550) 590 : cluster [DBG] pgmap v1052: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:48.688 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:47.944173+0000 osd.5 (osd.5) 288 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:15:48.688 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:47.947057+0000 osd.5 (osd.5) 289 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:15:48.688 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:48.128967+0000 mon.b (mon.1) 745 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:15:48.688 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:48.129763+0000 mon.b (mon.1) 746 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:15:48.688 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:48.134717+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:48.688 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:48.180334+0000 mon.c (mon.2) 426 : audit [INF] from='client.? v1:172.21.15.163:0/2863159766' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:15:48.688 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:48.180626+0000 mon.a (mon.0) 3406 : audit [INF] from='client.50444 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:15:48.688 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:48.391738+0000 mon.a (mon.0) 3407 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:15:48.688 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:48.397486+0000 mon.a (mon.0) 3408 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-122"}]': finished 2024-02-19T21:15:48.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:48.397556+0000 mon.a (mon.0) 3409 : audit [INF] from='client.50444 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:15:48.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:48.397610+0000 mon.a (mon.0) 3410 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2024-02-19T21:15:48.708 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:48.716 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:48.719 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:48.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:48.820 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:48.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:47.087284+0000 osd.6 (osd.6) 237 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:15:48.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:47.088237+0000 osd.6 (osd.6) 238 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:15:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:47.310344+0000 mgr.y (mgr.14550) 590 : cluster [DBG] pgmap v1052: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:15:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:47.944173+0000 osd.5 (osd.5) 288 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:15:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:47.947057+0000 osd.5 (osd.5) 289 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:15:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:48.128967+0000 mon.b (mon.1) 745 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:15:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:48.129763+0000 mon.b (mon.1) 746 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:15:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:48.134717+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:48.180334+0000 mon.c (mon.2) 426 : audit [INF] from='client.? v1:172.21.15.163:0/2863159766' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:15:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:48.180626+0000 mon.a (mon.0) 3406 : audit [INF] from='client.50444 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:15:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:48.391738+0000 mon.a (mon.0) 3407 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:15:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:48.397486+0000 mon.a (mon.0) 3408 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-122"}]': finished 2024-02-19T21:15:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:48.397556+0000 mon.a (mon.0) 3409 : audit [INF] from='client.50444 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:15:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:48.397610+0000 mon.a (mon.0) 3410 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2024-02-19T21:15:48.847 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:48.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:48.129763+0000 mon.b (mon.1) 746 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:15:48.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:48.134717+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:48.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:48.180334+0000 mon.c (mon.2) 426 : audit [INF] from='client.? v1:172.21.15.163:0/2863159766' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:15:48.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:48.180626+0000 mon.a (mon.0) 3406 : audit [INF] from='client.50444 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:15:48.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:48.391738+0000 mon.a (mon.0) 3407 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:15:48.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:48.397486+0000 mon.a (mon.0) 3408 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-122"}]': finished 2024-02-19T21:15:48.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:48.397556+0000 mon.a (mon.0) 3409 : audit [INF] from='client.50444 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:15:48.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:48.397610+0000 mon.a (mon.0) 3410 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2024-02-19T21:15:49.131 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-19T21:15:49.407 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:15:49.496 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:49.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:49.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:49.600 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:49.627 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:49.664 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:49.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:49.768 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:49.796 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:49.918 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:48.961409+0000 osd.1 (osd.1) 310 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:15:49.918 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:48.962236+0000 osd.1 (osd.1) 311 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:15:49.918 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:49.194302+0000 osd.3 (osd.3) 293 : cluster [DBG] 16.18 scrub starts 2024-02-19T21:15:49.918 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:49.195581+0000 osd.3 (osd.3) 294 : cluster [DBG] 16.18 scrub ok 2024-02-19T21:15:49.918 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:49.385671+0000 mon.a (mon.0) 3411 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2024-02-19T21:15:49.919 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:49.433886+0000 mon.a (mon.0) 3412 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:49.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:48.961409+0000 osd.1 (osd.1) 310 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:15:49.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:48.962236+0000 osd.1 (osd.1) 311 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:15:49.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:49.194302+0000 osd.3 (osd.3) 293 : cluster [DBG] 16.18 scrub starts 2024-02-19T21:15:49.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:49.195581+0000 osd.3 (osd.3) 294 : cluster [DBG] 16.18 scrub ok 2024-02-19T21:15:49.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:49.385671+0000 mon.a (mon.0) 3411 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2024-02-19T21:15:49.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:49.433886+0000 mon.a (mon.0) 3412 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:49.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:48.961409+0000 osd.1 (osd.1) 310 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:15:49.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:48.962236+0000 osd.1 (osd.1) 311 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:15:49.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:49.194302+0000 osd.3 (osd.3) 293 : cluster [DBG] 16.18 scrub starts 2024-02-19T21:15:49.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:49.195581+0000 osd.3 (osd.3) 294 : cluster [DBG] 16.18 scrub ok 2024-02-19T21:15:49.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:49.385671+0000 mon.a (mon.0) 3411 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2024-02-19T21:15:49.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:49.433886+0000 mon.a (mon.0) 3412 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:50.478 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:50.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:49.140534+0000 osd.6 (osd.6) 239 : cluster [DBG] 5.2 deep-scrub starts 2024-02-19T21:15:50.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:49.141773+0000 osd.6 (osd.6) 240 : cluster [DBG] 5.2 deep-scrub ok 2024-02-19T21:15:50.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:49.239868+0000 osd.2 (osd.2) 280 : cluster [DBG] 2.21 scrub starts 2024-02-19T21:15:50.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:49.242757+0000 osd.2 (osd.2) 281 : cluster [DBG] 2.21 scrub ok 2024-02-19T21:15:50.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:49.310743+0000 mgr.y (mgr.14550) 591 : cluster [DBG] pgmap v1055: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:15:50.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:49.515021+0000 osd.2 (osd.2) 282 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:15:50.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:49.515198+0000 osd.2 (osd.2) 283 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:15:50.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:49.953694+0000 osd.5 (osd.5) 290 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:15:50.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:49.956585+0000 osd.5 (osd.5) 291 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:15:50.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:49.981799+0000 osd.1 (osd.1) 312 : cluster [DBG] 16.15 scrub starts 2024-02-19T21:15:50.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:49.983117+0000 osd.1 (osd.1) 313 : cluster [DBG] 16.15 scrub ok 2024-02-19T21:15:50.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:50.221961+0000 osd.3 (osd.3) 295 : cluster [DBG] 16.4 scrub starts 2024-02-19T21:15:50.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:50.223207+0000 osd.3 (osd.3) 296 : cluster [DBG] 16.4 scrub ok 2024-02-19T21:15:50.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:50.396891+0000 mon.a (mon.0) 3413 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:50.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:50.396947+0000 mon.a (mon.0) 3414 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2024-02-19T21:15:50.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:50.398568+0000 mon.a (mon.0) 3415 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-122"}]: dispatch 2024-02-19T21:15:50.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:49.140534+0000 osd.6 (osd.6) 239 : cluster [DBG] 5.2 deep-scrub starts 2024-02-19T21:15:50.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:49.141773+0000 osd.6 (osd.6) 240 : cluster [DBG] 5.2 deep-scrub ok 2024-02-19T21:15:50.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:49.239868+0000 osd.2 (osd.2) 280 : cluster [DBG] 2.21 scrub starts 2024-02-19T21:15:50.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:49.242757+0000 osd.2 (osd.2) 281 : cluster [DBG] 2.21 scrub ok 2024-02-19T21:15:50.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:49.310743+0000 mgr.y (mgr.14550) 591 : cluster [DBG] pgmap v1055: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:15:50.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:49.515021+0000 osd.2 (osd.2) 282 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:15:50.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:49.515198+0000 osd.2 (osd.2) 283 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:15:50.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:49.953694+0000 osd.5 (osd.5) 290 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:15:50.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:49.956585+0000 osd.5 (osd.5) 291 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:15:50.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:49.981799+0000 osd.1 (osd.1) 312 : cluster [DBG] 16.15 scrub starts 2024-02-19T21:15:50.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:49.983117+0000 osd.1 (osd.1) 313 : cluster [DBG] 16.15 scrub ok 2024-02-19T21:15:50.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:50.221961+0000 osd.3 (osd.3) 295 : cluster [DBG] 16.4 scrub starts 2024-02-19T21:15:50.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:50.223207+0000 osd.3 (osd.3) 296 : cluster [DBG] 16.4 scrub ok 2024-02-19T21:15:50.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:50.396891+0000 mon.a (mon.0) 3413 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:50.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:50.396947+0000 mon.a (mon.0) 3414 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2024-02-19T21:15:50.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:50.398568+0000 mon.a (mon.0) 3415 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-122"}]: dispatch 2024-02-19T21:15:50.785 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:15:50.786 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:49.140534+0000 osd.6 (osd.6) 239 : cluster [DBG] 5.2 deep-scrub starts 2024-02-19T21:15:50.786 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:49.141773+0000 osd.6 (osd.6) 240 : cluster [DBG] 5.2 deep-scrub ok 2024-02-19T21:15:50.786 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:49.239868+0000 osd.2 (osd.2) 280 : cluster [DBG] 2.21 scrub starts 2024-02-19T21:15:50.786 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:49.242757+0000 osd.2 (osd.2) 281 : cluster [DBG] 2.21 scrub ok 2024-02-19T21:15:50.786 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:49.310743+0000 mgr.y (mgr.14550) 591 : cluster [DBG] pgmap v1055: 214 pgs: 214 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:15:50.786 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:49.515021+0000 osd.2 (osd.2) 282 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:15:50.786 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:49.515198+0000 osd.2 (osd.2) 283 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:15:50.786 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:49.953694+0000 osd.5 (osd.5) 290 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:15:50.786 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:49.956585+0000 osd.5 (osd.5) 291 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:15:50.786 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:49.981799+0000 osd.1 (osd.1) 312 : cluster [DBG] 16.15 scrub starts 2024-02-19T21:15:50.786 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:49.983117+0000 osd.1 (osd.1) 313 : cluster [DBG] 16.15 scrub ok 2024-02-19T21:15:50.786 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:50.221961+0000 osd.3 (osd.3) 295 : cluster [DBG] 16.4 scrub starts 2024-02-19T21:15:50.787 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:50.223207+0000 osd.3 (osd.3) 296 : cluster [DBG] 16.4 scrub ok 2024-02-19T21:15:50.787 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:50.396891+0000 mon.a (mon.0) 3413 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:50.787 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:50.396947+0000 mon.a (mon.0) 3414 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2024-02-19T21:15:50.787 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:50.398568+0000 mon.a (mon.0) 3415 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-122"}]: dispatch 2024-02-19T21:15:50.824 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:50.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:50.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:50.928 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:50.955 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:51.053 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:51.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:51.157 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:15:51.170 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:15:51.184 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:51.203 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:15:51.206 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:51.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:15:51.307 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:51.314 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:15:51.314 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:15:51.334 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:51.433 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:51.435 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:51.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:51.537 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:51.564 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:51.913 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:51.915 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:51.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:51.936 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:50.163476+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.7 deep-scrub starts 2024-02-19T21:15:51.936 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:50.164427+0000 osd.6 (osd.6) 242 : cluster [DBG] 2.7 deep-scrub ok 2024-02-19T21:15:51.936 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:50.714543+0000 mgr.y (mgr.14550) 592 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:51.936 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:50.912422+0000 osd.7 (osd.7) 319 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:15:51.936 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:50.915327+0000 osd.7 (osd.7) 320 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:15:51.936 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:51.015880+0000 osd.1 (osd.1) 314 : cluster [DBG] 5.c scrub starts 2024-02-19T21:15:51.937 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:51.016877+0000 osd.1 (osd.1) 315 : cluster [DBG] 5.c scrub ok 2024-02-19T21:15:51.937 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:51.394152+0000 mon.a (mon.0) 3416 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:51.937 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:51.399117+0000 mon.a (mon.0) 3417 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-122"}]': finished 2024-02-19T21:15:51.937 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:51.399178+0000 mon.a (mon.0) 3418 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2024-02-19T21:15:51.955 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-19T21:15:51.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:50.163476+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.7 deep-scrub starts 2024-02-19T21:15:51.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:50.164427+0000 osd.6 (osd.6) 242 : cluster [DBG] 2.7 deep-scrub ok 2024-02-19T21:15:51.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:50.714543+0000 mgr.y (mgr.14550) 592 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:51.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:50.912422+0000 osd.7 (osd.7) 319 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:15:51.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:50.915327+0000 osd.7 (osd.7) 320 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:15:51.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:51.015880+0000 osd.1 (osd.1) 314 : cluster [DBG] 5.c scrub starts 2024-02-19T21:15:51.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:51.016877+0000 osd.1 (osd.1) 315 : cluster [DBG] 5.c scrub ok 2024-02-19T21:15:51.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:51.394152+0000 mon.a (mon.0) 3416 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:51.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:51.399117+0000 mon.a (mon.0) 3417 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-122"}]': finished 2024-02-19T21:15:51.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:51.399178+0000 mon.a (mon.0) 3418 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2024-02-19T21:15:51.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:50.163476+0000 osd.6 (osd.6) 241 : cluster [DBG] 2.7 deep-scrub starts 2024-02-19T21:15:51.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:50.164427+0000 osd.6 (osd.6) 242 : cluster [DBG] 2.7 deep-scrub ok 2024-02-19T21:15:51.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:50.714543+0000 mgr.y (mgr.14550) 592 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:15:51.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:50.912422+0000 osd.7 (osd.7) 319 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:15:51.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:50.915327+0000 osd.7 (osd.7) 320 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:15:51.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:51.015880+0000 osd.1 (osd.1) 314 : cluster [DBG] 5.c scrub starts 2024-02-19T21:15:51.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:51.016877+0000 osd.1 (osd.1) 315 : cluster [DBG] 5.c scrub ok 2024-02-19T21:15:51.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:51.394152+0000 mon.a (mon.0) 3416 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:52.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:51.399117+0000 mon.a (mon.0) 3417 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-122"}]': finished 2024-02-19T21:15:52.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:51.399178+0000 mon.a (mon.0) 3418 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2024-02-19T21:15:52.016 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:52.044 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:52.192 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:52.195 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:52.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:52.218 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:15:52.297 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:52.327 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:52.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.415+0000 7f6319a04700 1 -- 172.21.15.163:0/399157997 >> v1:172.21.15.163:6790/0 conn(0x7f63140f9370 legacy=0x7f6314102840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.415+0000 7f6319a04700 1 -- 172.21.15.163:0/399157997 shutdown_connections 2024-02-19T21:15:52.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.415+0000 7f6319a04700 1 -- 172.21.15.163:0/399157997 >> 172.21.15.163:0/399157997 conn(0x7f63140f22f0 msgr2=0x7f63140f4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:52.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.415+0000 7f6319a04700 1 -- 172.21.15.163:0/399157997 shutdown_connections 2024-02-19T21:15:52.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.415+0000 7f6319a04700 1 -- 172.21.15.163:0/399157997 wait complete. 2024-02-19T21:15:52.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.415+0000 7f6319a04700 1 librados: starting msgr at 2024-02-19T21:15:52.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.415+0000 7f6319a04700 1 librados: starting objecter 2024-02-19T21:15:52.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.415+0000 7f6319a04700 1 Processor -- start 2024-02-19T21:15:52.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f6319a04700 1 -- start start 2024-02-19T21:15:52.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f6319a04700 1 librados: setting wanted keys 2024-02-19T21:15:52.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f6319a04700 1 librados: calling monclient init 2024-02-19T21:15:52.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f6319a04700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6314102cf0 con 0x7f63140f87c0 2024-02-19T21:15:52.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f6319a04700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f631411aa00 con 0x7f6314104150 2024-02-19T21:15:52.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f6319a04700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f631411bba0 con 0x7f63140f9370 2024-02-19T21:15:52.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f6312ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f63140f87c0 0x7f6314102380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:38184/0 (socket says 172.21.15.163:38184) 2024-02-19T21:15:52.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f6312ffd700 1 -- 172.21.15.163:0/850998758 learned_addr learned my addr 172.21.15.163:0/850998758 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:52.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 10 client.?.objecter ms_handle_connect 0x7f63140f9370 2024-02-19T21:15:52.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:52.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 10 client.?.objecter ms_handle_connect 0x7f63140f87c0 2024-02-19T21:15:52.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:52.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 10 client.?.objecter ms_handle_connect 0x7f6314104150 2024-02-19T21:15:52.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:52.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552997993 0 0) 0x7f631411bba0 con 0x7f63140f9370 2024-02-19T21:15:52.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62fc003630 con 0x7f63140f9370 2024-02-19T21:15:52.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147897702 0 0) 0x7f631411aa00 con 0x7f6314104150 2024-02-19T21:15:52.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f631411bba0 con 0x7f6314104150 2024-02-19T21:15:52.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 724971818 0 0) 0x7f62fc003630 con 0x7f63140f9370 2024-02-19T21:15:52.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f631411aa00 con 0x7f63140f9370 2024-02-19T21:15:52.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6304002f00 con 0x7f63140f9370 2024-02-19T21:15:52.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 815679955 0 0) 0x7f631411bba0 con 0x7f6314104150 2024-02-19T21:15:52.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62fc003630 con 0x7f6314104150 2024-02-19T21:15:52.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.416+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f630c001c40 con 0x7f6314104150 2024-02-19T21:15:52.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.417+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3559538528 0 0) 0x7f631411aa00 con 0x7f63140f9370 2024-02-19T21:15:52.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.417+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 >> v1:172.21.15.193:6789/0 conn(0x7f6314104150 legacy=0x7f63141182d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.417+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 >> v1:172.21.15.163:6789/0 conn(0x7f63140f87c0 legacy=0x7f6314102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.417+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631411cd80 con 0x7f63140f9370 2024-02-19T21:15:52.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.417+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6314119a50 con 0x7f63140f9370 2024-02-19T21:15:52.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.417+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6304001ec0 con 0x7f63140f9370 2024-02-19T21:15:52.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.417+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6304003960 con 0x7f63140f9370 2024-02-19T21:15:52.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.417+0000 7f6319a04700 10 client.50450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:52.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.417+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6314119ec0 con 0x7f63140f9370 2024-02-19T21:15:52.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.417+0000 7f6319a04700 1 librados: init done 2024-02-19T21:15:52.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.418+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6300004f00 con 0x7f63140f9370 2024-02-19T21:15:52.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.418+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f63040020c0 con 0x7f63140f9370 2024-02-19T21:15:52.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.419+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(703..703 src has 623..703) v4 ==== 8325+0+0 (unknown 1782079745 0 0) 0x7f63040758c0 con 0x7f63140f9370 2024-02-19T21:15:52.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.419+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 osd_map(703..703 src has 623..703) v4 2024-02-19T21:15:52.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.419+0000 7f62fbfff700 3 client.50450.objecter handle_osd_map got epochs [703,703] > 0 2024-02-19T21:15:52.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.419+0000 7f62fbfff700 3 client.50450.objecter handle_osd_map decoding full epoch 703 2024-02-19T21:15:52.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.419+0000 7f62fbfff700 20 client.50450.objecter dump_active .. 0 homeless 2024-02-19T21:15:52.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.419+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v703) v1 ==== 54+0+16 (unknown 534793240 0 2420000192) 0x7f63040789a0 con 0x7f63140f9370 2024-02-19T21:15:52.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.419+0000 7f62fbfff700 10 client.50450.objecter ms_handle_connect 0x7f62fc063b20 2024-02-19T21:15:52.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.421+0000 7f6319a04700 10 client.50450.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:52.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.421+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 1 2024-02-19T21:15:52.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.421+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f63000051a0 osd=0 3 2024-02-19T21:15:52.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.421+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f63000051a0 osd=0 3 2024-02-19T21:15:52.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.421+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f63000051a0 osd=0 3 2024-02-19T21:15:52.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.421+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 1 2024-02-19T21:15:52.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.421+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f63000051a0 osd=0 3 2024-02-19T21:15:52.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.421+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 0 1 2024-02-19T21:15:52.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.421+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f63000051a0 osd=0 4 2024-02-19T21:15:52.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.421+0000 7f6319a04700 10 client.50450.objecter _send_command 1 2024-02-19T21:15:52.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.421+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6300004970 con 0x7f6300004530 2024-02-19T21:15:52.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.421+0000 7f62fbfff700 10 client.50450.objecter ms_handle_connect 0x7f6300004530 2024-02-19T21:15:52.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.422+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6300004970 con 0x7f6300004530 2024-02-19T21:15:52.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.422+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:15:52.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.422+0000 7f62fbfff700 10 client.50450.objecter _finish_command 1 = system:0 2024-02-19T21:15:52.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.422+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f63000051a0 osd=0 4 2024-02-19T21:15:52.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.422+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 0 1 2024-02-19T21:15:52.432 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:52.439 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:52.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:52.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.441+0000 7f6319a04700 10 client.50450.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:52.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.441+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 2 2024-02-19T21:15:52.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.441+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f63000051a0 osd=0 3 2024-02-19T21:15:52.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.441+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f63000051a0 osd=0 3 2024-02-19T21:15:52.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.441+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f63000051a0 osd=0 3 2024-02-19T21:15:52.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.441+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 2 2024-02-19T21:15:52.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.441+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f63000051a0 osd=0 3 2024-02-19T21:15:52.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.441+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 0 2 2024-02-19T21:15:52.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.441+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f63000051a0 osd=0 4 2024-02-19T21:15:52.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.441+0000 7f6319a04700 10 client.50450.objecter _send_command 2 2024-02-19T21:15:52.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.441+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6300004d50 con 0x7f6300004530 2024-02-19T21:15:52.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.444+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6300004d50 con 0x7f6300004530 2024-02-19T21:15:52.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.444+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:52.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.444+0000 7f62fbfff700 10 client.50450.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.444+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f63000051a0 osd=0 4 2024-02-19T21:15:52.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.444+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 0 2 2024-02-19T21:15:52.444 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:15:52.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.445+0000 7f6319a04700 10 client.50450.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:52.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.445+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 3 2024-02-19T21:15:52.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.445+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f6300004290 osd=1 3 2024-02-19T21:15:52.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.445+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f6300004290 osd=1 3 2024-02-19T21:15:52.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.445+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f6300004290 osd=1 3 2024-02-19T21:15:52.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.445+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 3 2024-02-19T21:15:52.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.445+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f6300004290 osd=1 3 2024-02-19T21:15:52.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.445+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 1 3 2024-02-19T21:15:52.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.445+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f6300004290 osd=1 4 2024-02-19T21:15:52.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.445+0000 7f6319a04700 10 client.50450.objecter _send_command 3 2024-02-19T21:15:52.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.445+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f630000bef0 con 0x7f6300003b40 2024-02-19T21:15:52.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.445+0000 7f62fbfff700 10 client.50450.objecter ms_handle_connect 0x7f6300003b40 2024-02-19T21:15:52.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.446+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f630000bef0 con 0x7f6300003b40 2024-02-19T21:15:52.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.446+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:15:52.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.446+0000 7f62fbfff700 10 client.50450.objecter _finish_command 3 = system:0 2024-02-19T21:15:52.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.446+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f6300004290 osd=1 4 2024-02-19T21:15:52.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.446+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 1 3 2024-02-19T21:15:52.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.462+0000 7f6319a04700 10 client.50450.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:52.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.462+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 4 2024-02-19T21:15:52.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.462+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f6300004290 osd=1 3 2024-02-19T21:15:52.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.462+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f6300004290 osd=1 3 2024-02-19T21:15:52.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.462+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f6300004290 osd=1 3 2024-02-19T21:15:52.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.462+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 4 2024-02-19T21:15:52.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.462+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f6300004290 osd=1 3 2024-02-19T21:15:52.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.462+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 1 4 2024-02-19T21:15:52.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.462+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f6300004290 osd=1 4 2024-02-19T21:15:52.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.462+0000 7f6319a04700 10 client.50450.objecter _send_command 4 2024-02-19T21:15:52.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.462+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6300003160 con 0x7f6300003b40 2024-02-19T21:15:52.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6300003160 con 0x7f6300003b40 2024-02-19T21:15:52.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:52.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f62fbfff700 10 client.50450.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f6300004290 osd=1 4 2024-02-19T21:15:52.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 1 4 2024-02-19T21:15:52.464 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:15:52.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f6319a04700 10 client.50450.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:52.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 5 2024-02-19T21:15:52.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f6300001b50 osd=2 3 2024-02-19T21:15:52.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f6300001b50 osd=2 3 2024-02-19T21:15:52.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f6300001b50 osd=2 3 2024-02-19T21:15:52.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 5 2024-02-19T21:15:52.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f6300001b50 osd=2 3 2024-02-19T21:15:52.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 2 5 2024-02-19T21:15:52.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f6300001b50 osd=2 4 2024-02-19T21:15:52.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f6319a04700 10 client.50450.objecter _send_command 5 2024-02-19T21:15:52.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.464+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f630000ddb0 con 0x7f630000d210 2024-02-19T21:15:52.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.465+0000 7f62fbfff700 10 client.50450.objecter ms_handle_connect 0x7f630000d210 2024-02-19T21:15:52.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.465+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f630000ddb0 con 0x7f630000d210 2024-02-19T21:15:52.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.465+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:15:52.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.465+0000 7f62fbfff700 10 client.50450.objecter _finish_command 5 = system:0 2024-02-19T21:15:52.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.465+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f6300001b50 osd=2 4 2024-02-19T21:15:52.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.465+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 2 5 2024-02-19T21:15:52.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.480+0000 7f6319a04700 10 client.50450.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:52.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.480+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 6 2024-02-19T21:15:52.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.480+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f6300001b50 osd=2 3 2024-02-19T21:15:52.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.480+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f6300001b50 osd=2 3 2024-02-19T21:15:52.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.480+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f6300001b50 osd=2 3 2024-02-19T21:15:52.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.480+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 6 2024-02-19T21:15:52.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.480+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f6300001b50 osd=2 3 2024-02-19T21:15:52.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.480+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 2 6 2024-02-19T21:15:52.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.480+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f6300001b50 osd=2 4 2024-02-19T21:15:52.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.480+0000 7f6319a04700 10 client.50450.objecter _send_command 6 2024-02-19T21:15:52.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.480+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63000015c0 con 0x7f630000d210 2024-02-19T21:15:52.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f63000015c0 con 0x7f630000d210 2024-02-19T21:15:52.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:52.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f62fbfff700 10 client.50450.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f6300001b50 osd=2 4 2024-02-19T21:15:52.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 2 6 2024-02-19T21:15:52.481 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:15:52.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f6319a04700 10 client.50450.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:52.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 7 2024-02-19T21:15:52.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f63000019d0 osd=3 3 2024-02-19T21:15:52.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f63000019d0 osd=3 3 2024-02-19T21:15:52.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f63000019d0 osd=3 3 2024-02-19T21:15:52.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 7 2024-02-19T21:15:52.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f63000019d0 osd=3 3 2024-02-19T21:15:52.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 3 7 2024-02-19T21:15:52.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f63000019d0 osd=3 4 2024-02-19T21:15:52.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f6319a04700 10 client.50450.objecter _send_command 7 2024-02-19T21:15:52.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.481+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6300015a30 con 0x7f6300011dc0 2024-02-19T21:15:52.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.482+0000 7f62fbfff700 10 client.50450.objecter ms_handle_connect 0x7f6300011dc0 2024-02-19T21:15:52.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.482+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6300015a30 con 0x7f6300011dc0 2024-02-19T21:15:52.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.482+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:15:52.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.482+0000 7f62fbfff700 10 client.50450.objecter _finish_command 7 = system:0 2024-02-19T21:15:52.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.482+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f63000019d0 osd=3 4 2024-02-19T21:15:52.483 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.482+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 3 7 2024-02-19T21:15:52.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.497+0000 7f6319a04700 10 client.50450.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:52.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.497+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 8 2024-02-19T21:15:52.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.497+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f63000019d0 osd=3 3 2024-02-19T21:15:52.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.497+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f63000019d0 osd=3 3 2024-02-19T21:15:52.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.497+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f63000019d0 osd=3 3 2024-02-19T21:15:52.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.497+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 8 2024-02-19T21:15:52.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.497+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f63000019d0 osd=3 3 2024-02-19T21:15:52.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.497+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 3 8 2024-02-19T21:15:52.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.497+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f63000019d0 osd=3 4 2024-02-19T21:15:52.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.497+0000 7f6319a04700 10 client.50450.objecter _send_command 8 2024-02-19T21:15:52.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.497+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f630000e160 con 0x7f6300011dc0 2024-02-19T21:15:52.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f630000e160 con 0x7f6300011dc0 2024-02-19T21:15:52.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:52.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f62fbfff700 10 client.50450.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f63000019d0 osd=3 4 2024-02-19T21:15:52.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 3 8 2024-02-19T21:15:52.498 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:15:52.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f6319a04700 10 client.50450.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:52.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 9 2024-02-19T21:15:52.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f630000e160 osd=4 3 2024-02-19T21:15:52.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f630000e160 osd=4 3 2024-02-19T21:15:52.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f630000e160 osd=4 3 2024-02-19T21:15:52.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 9 2024-02-19T21:15:52.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f630000e160 osd=4 3 2024-02-19T21:15:52.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 4 9 2024-02-19T21:15:52.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f630000e160 osd=4 4 2024-02-19T21:15:52.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f6319a04700 10 client.50450.objecter _send_command 9 2024-02-19T21:15:52.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.498+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f630000cf60 con 0x7f630000e310 2024-02-19T21:15:52.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.499+0000 7f62fbfff700 10 client.50450.objecter ms_handle_connect 0x7f630000e310 2024-02-19T21:15:52.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.499+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f630000cf60 con 0x7f630000e310 2024-02-19T21:15:52.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.499+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:15:52.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.499+0000 7f62fbfff700 10 client.50450.objecter _finish_command 9 = system:0 2024-02-19T21:15:52.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.499+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f630000e160 osd=4 4 2024-02-19T21:15:52.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.499+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 4 9 2024-02-19T21:15:52.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.514+0000 7f6319a04700 10 client.50450.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:52.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.514+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 10 2024-02-19T21:15:52.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.514+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f630000e160 osd=4 3 2024-02-19T21:15:52.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.514+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f630000e160 osd=4 3 2024-02-19T21:15:52.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.514+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f630000e160 osd=4 3 2024-02-19T21:15:52.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.514+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 10 2024-02-19T21:15:52.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.514+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f630000e160 osd=4 3 2024-02-19T21:15:52.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.514+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 4 10 2024-02-19T21:15:52.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.514+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f630000e160 osd=4 4 2024-02-19T21:15:52.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.514+0000 7f6319a04700 10 client.50450.objecter _send_command 10 2024-02-19T21:15:52.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.514+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f630000cf60 con 0x7f630000e310 2024-02-19T21:15:52.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f630000cf60 con 0x7f630000e310 2024-02-19T21:15:52.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:52.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f62fbfff700 10 client.50450.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f630000e160 osd=4 4 2024-02-19T21:15:52.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 4 10 2024-02-19T21:15:52.516 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:15:52.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f6319a04700 10 client.50450.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:52.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 11 2024-02-19T21:15:52.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f630000cf60 osd=5 3 2024-02-19T21:15:52.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f630000cf60 osd=5 3 2024-02-19T21:15:52.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f630000cf60 osd=5 3 2024-02-19T21:15:52.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 11 2024-02-19T21:15:52.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f630000cf60 osd=5 3 2024-02-19T21:15:52.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 5 11 2024-02-19T21:15:52.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f630000cf60 osd=5 4 2024-02-19T21:15:52.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f6319a04700 10 client.50450.objecter _send_command 11 2024-02-19T21:15:52.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6300016d90 con 0x7f63000167f0 2024-02-19T21:15:52.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.516+0000 7f62fbfff700 10 client.50450.objecter ms_handle_connect 0x7f63000167f0 2024-02-19T21:15:52.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.517+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6300016d90 con 0x7f63000167f0 2024-02-19T21:15:52.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.517+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:15:52.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.517+0000 7f62fbfff700 10 client.50450.objecter _finish_command 11 = system:0 2024-02-19T21:15:52.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.517+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f630000cf60 osd=5 4 2024-02-19T21:15:52.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.517+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 5 11 2024-02-19T21:15:52.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.531+0000 7f6319a04700 10 client.50450.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:52.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.531+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 12 2024-02-19T21:15:52.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.531+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f630000cf60 osd=5 3 2024-02-19T21:15:52.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.531+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f630000cf60 osd=5 3 2024-02-19T21:15:52.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.531+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f630000cf60 osd=5 3 2024-02-19T21:15:52.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.531+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 12 2024-02-19T21:15:52.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.531+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f630000cf60 osd=5 3 2024-02-19T21:15:52.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.531+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 5 12 2024-02-19T21:15:52.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.531+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f630000cf60 osd=5 4 2024-02-19T21:15:52.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.531+0000 7f6319a04700 10 client.50450.objecter _send_command 12 2024-02-19T21:15:52.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.531+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6300016d90 con 0x7f63000167f0 2024-02-19T21:15:52.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6300016d90 con 0x7f63000167f0 2024-02-19T21:15:52.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:52.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f62fbfff700 10 client.50450.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f630000cf60 osd=5 4 2024-02-19T21:15:52.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 5 12 2024-02-19T21:15:52.534 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:15:52.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f6319a04700 10 client.50450.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:52.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 13 2024-02-19T21:15:52.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f6300016d90 osd=6 3 2024-02-19T21:15:52.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f6300016d90 osd=6 3 2024-02-19T21:15:52.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f6300016d90 osd=6 3 2024-02-19T21:15:52.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 13 2024-02-19T21:15:52.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f6300016d90 osd=6 3 2024-02-19T21:15:52.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 6 13 2024-02-19T21:15:52.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f6300016d90 osd=6 4 2024-02-19T21:15:52.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f6319a04700 10 client.50450.objecter _send_command 13 2024-02-19T21:15:52.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.533+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6300022b40 con 0x7f630001f030 2024-02-19T21:15:52.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.534+0000 7f62fbfff700 10 client.50450.objecter ms_handle_connect 0x7f630001f030 2024-02-19T21:15:52.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.535+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6300022b40 con 0x7f630001f030 2024-02-19T21:15:52.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.535+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:15:52.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.535+0000 7f62fbfff700 10 client.50450.objecter _finish_command 13 = system:0 2024-02-19T21:15:52.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.535+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f6300016d90 osd=6 4 2024-02-19T21:15:52.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.535+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 6 13 2024-02-19T21:15:52.541 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:52.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.549+0000 7f6319a04700 10 client.50450.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:52.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.549+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 14 2024-02-19T21:15:52.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.549+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f6300016d90 osd=6 3 2024-02-19T21:15:52.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.549+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f6300016d90 osd=6 3 2024-02-19T21:15:52.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.549+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f6300016d90 osd=6 3 2024-02-19T21:15:52.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.549+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 14 2024-02-19T21:15:52.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.549+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f6300016d90 osd=6 3 2024-02-19T21:15:52.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.549+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 6 14 2024-02-19T21:15:52.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.549+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f6300016d90 osd=6 4 2024-02-19T21:15:52.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.549+0000 7f6319a04700 10 client.50450.objecter _send_command 14 2024-02-19T21:15:52.550 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.549+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f63000242f0 con 0x7f630001f030 2024-02-19T21:15:52.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f63000242f0 con 0x7f630001f030 2024-02-19T21:15:52.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:52.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f62fbfff700 10 client.50450.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f6300016d90 osd=6 4 2024-02-19T21:15:52.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 6 14 2024-02-19T21:15:52.551 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:15:52.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f6319a04700 10 client.50450.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:15:52.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 15 2024-02-19T21:15:52.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f63000242f0 osd=7 3 2024-02-19T21:15:52.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f63000242f0 osd=7 3 2024-02-19T21:15:52.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f63000242f0 osd=7 3 2024-02-19T21:15:52.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 15 2024-02-19T21:15:52.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f63000242f0 osd=7 3 2024-02-19T21:15:52.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 7 15 2024-02-19T21:15:52.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f63000242f0 osd=7 4 2024-02-19T21:15:52.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f6319a04700 10 client.50450.objecter _send_command 15 2024-02-19T21:15:52.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.551+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6300024f20 con 0x7f6300024970 2024-02-19T21:15:52.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.552+0000 7f62fbfff700 10 client.50450.objecter ms_handle_connect 0x7f6300024970 2024-02-19T21:15:52.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.552+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6300024f20 con 0x7f6300024970 2024-02-19T21:15:52.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.552+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:15:52.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.552+0000 7f62fbfff700 10 client.50450.objecter _finish_command 15 = system:0 2024-02-19T21:15:52.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.552+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f63000242f0 osd=7 4 2024-02-19T21:15:52.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.552+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 7 15 2024-02-19T21:15:52.569 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:52.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.576+0000 7f6319a04700 10 client.50450.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:15:52.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.576+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign -1 16 2024-02-19T21:15:52.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.576+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f63000242f0 osd=7 3 2024-02-19T21:15:52.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.576+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f63000242f0 osd=7 3 2024-02-19T21:15:52.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.576+0000 7f6319a04700 20 client.50450.objecter _get_session s=0x7f63000242f0 osd=7 3 2024-02-19T21:15:52.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.576+0000 7f6319a04700 15 client.50450.objecter _session_command_op_remove -1 16 2024-02-19T21:15:52.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.576+0000 7f6319a04700 20 client.50450.objecter get_session s=0x7f63000242f0 osd=7 3 2024-02-19T21:15:52.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.576+0000 7f6319a04700 15 client.50450.objecter _session_command_op_assign 7 16 2024-02-19T21:15:52.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.576+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f63000242f0 osd=7 4 2024-02-19T21:15:52.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.576+0000 7f6319a04700 10 client.50450.objecter _send_command 16 2024-02-19T21:15:52.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.576+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f6300024f20 con 0x7f6300024970 2024-02-19T21:15:52.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.578+0000 7f62fbfff700 1 -- 172.21.15.163:0/850998758 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f6300024f20 con 0x7f6300024970 2024-02-19T21:15:52.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.578+0000 7f62fbfff700 10 client.50450.objecter ms_dispatch 0x7f6314001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:15:52.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.578+0000 7f62fbfff700 10 client.50450.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.578+0000 7f62fbfff700 20 client.50450.objecter put_session s=0x7f63000242f0 osd=7 4 2024-02-19T21:15:52.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.578+0000 7f62fbfff700 15 client.50450.objecter _session_command_op_remove 7 16 2024-02-19T21:15:52.578 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:15:52.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 10 librados: watch_flush enter 2024-02-19T21:15:52.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 10 librados: watch_flush exit 2024-02-19T21:15:52.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 10 client.50450.objecter close_session for osd.0 2024-02-19T21:15:52.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 >> v1:172.21.15.163:6801/1559418115 conn(0x7f6300004530 legacy=0x7f6300002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f63000051a0 osd=0 1 2024-02-19T21:15:52.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 10 client.50450.objecter close_session for osd.1 2024-02-19T21:15:52.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 >> v1:172.21.15.163:6805/3125047432 conn(0x7f6300003b40 legacy=0x7f630000a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f6300004290 osd=1 1 2024-02-19T21:15:52.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 10 client.50450.objecter close_session for osd.2 2024-02-19T21:15:52.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 >> v1:172.21.15.163:6809/1563158487 conn(0x7f630000d210 legacy=0x7f630000d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f6300001b50 osd=2 1 2024-02-19T21:15:52.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 10 client.50450.objecter close_session for osd.3 2024-02-19T21:15:52.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 >> v1:172.21.15.163:6813/2112279769 conn(0x7f6300011dc0 legacy=0x7f63000141b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f63000019d0 osd=3 1 2024-02-19T21:15:52.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 10 client.50450.objecter close_session for osd.4 2024-02-19T21:15:52.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 >> v1:172.21.15.193:6800/3129716965 conn(0x7f630000e310 legacy=0x7f6300015bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f630000e160 osd=4 1 2024-02-19T21:15:52.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 10 client.50450.objecter close_session for osd.5 2024-02-19T21:15:52.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 >> v1:172.21.15.193:6804/805161991 conn(0x7f63000167f0 legacy=0x7f630001d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f630000cf60 osd=5 1 2024-02-19T21:15:52.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 10 client.50450.objecter close_session for osd.6 2024-02-19T21:15:52.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 >> v1:172.21.15.193:6808/787604615 conn(0x7f630001f030 legacy=0x7f6300021420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f6300016d90 osd=6 1 2024-02-19T21:15:52.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 10 client.50450.objecter close_session for osd.7 2024-02-19T21:15:52.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 >> v1:172.21.15.193:6812/157927432 conn(0x7f6300024970 legacy=0x7f6300022b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 20 client.50450.objecter put_session s=0x7f63000242f0 osd=7 1 2024-02-19T21:15:52.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 20 client.50450.objecter shutdown clearing up homeless session... 2024-02-19T21:15:52.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 10 client.50450.objecter successfully canceled tick 2024-02-19T21:15:52.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f63137fe700 1 -- 172.21.15.163:0/850998758 reap_dead start 2024-02-19T21:15:52.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f63137fe700 1 -- 172.21.15.163:0/850998758 reap_dead start 2024-02-19T21:15:52.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f63137fe700 1 -- 172.21.15.163:0/850998758 reap_dead start 2024-02-19T21:15:52.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f63137fe700 1 -- 172.21.15.163:0/850998758 reap_dead start 2024-02-19T21:15:52.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f63137fe700 1 -- 172.21.15.163:0/850998758 reap_dead start 2024-02-19T21:15:52.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 >> v1:172.21.15.163:6800/1552514681 conn(0x7f62fc063b20 legacy=0x7f62fc065fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 >> v1:172.21.15.163:6790/0 conn(0x7f63140f9370 legacy=0x7f6314114ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 shutdown_connections 2024-02-19T21:15:52.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 >> 172.21.15.163:0/850998758 conn(0x7f63140f22f0 msgr2=0x7f63141075c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:52.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 shutdown_connections 2024-02-19T21:15:52.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 1 -- 172.21.15.163:0/850998758 wait complete. 2024-02-19T21:15:52.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.579+0000 7f6319a04700 1 librados: shutdown 2024-02-19T21:15:52.589 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.589 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.589 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.589 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.589 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.590 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.590 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.590 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:15:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.686+0000 7ff94c199700 1 -- 172.21.15.163:0/2623689682 >> v1:172.21.15.163:6790/0 conn(0x7ff9441042a0 legacy=0x7ff944106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.686+0000 7ff94c199700 1 -- 172.21.15.163:0/2623689682 shutdown_connections 2024-02-19T21:15:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.687+0000 7ff94c199700 1 -- 172.21.15.163:0/2623689682 >> 172.21.15.163:0/2623689682 conn(0x7ff9440f2470 msgr2=0x7ff9440f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.687+0000 7ff94c199700 1 -- 172.21.15.163:0/2623689682 shutdown_connections 2024-02-19T21:15:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.687+0000 7ff94c199700 1 -- 172.21.15.163:0/2623689682 wait complete. 2024-02-19T21:15:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.687+0000 7ff94c199700 1 librados: starting msgr at 2024-02-19T21:15:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.687+0000 7ff94c199700 1 librados: starting objecter 2024-02-19T21:15:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.687+0000 7ff94c199700 1 Processor -- start 2024-02-19T21:15:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.687+0000 7ff94c199700 1 -- start start 2024-02-19T21:15:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.687+0000 7ff94c199700 1 librados: setting wanted keys 2024-02-19T21:15:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.687+0000 7ff94c199700 1 librados: calling monclient init 2024-02-19T21:15:52.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.687+0000 7ff94c199700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff94410f6a0 con 0x7ff9441042a0 2024-02-19T21:15:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.687+0000 7ff94c199700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff94411a8b0 con 0x7ff9440f9490 2024-02-19T21:15:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.687+0000 7ff94c199700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff94411ba50 con 0x7ff9440f89a0 2024-02-19T21:15:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.687+0000 7ff949f35700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7ff9440f89a0 0x7ff94410ebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:60296/0 (socket says 172.21.15.163:60296) 2024-02-19T21:15:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.687+0000 7ff949f35700 1 -- 172.21.15.163:0/3092656493 learned_addr learned my addr 172.21.15.163:0/3092656493 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 10 client.?.objecter ms_handle_connect 0x7ff9440f89a0 2024-02-19T21:15:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 10 client.?.objecter ms_handle_connect 0x7ff9441042a0 2024-02-19T21:15:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 10 client.?.objecter ms_handle_connect 0x7ff9440f9490 2024-02-19T21:15:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483173487 0 0) 0x7ff94411ba50 con 0x7ff9440f89a0 2024-02-19T21:15:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff930003030 con 0x7ff9440f89a0 2024-02-19T21:15:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356059600 0 0) 0x7ff94411a8b0 con 0x7ff9440f9490 2024-02-19T21:15:52.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff94411ba50 con 0x7ff9440f9490 2024-02-19T21:15:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474750638 0 0) 0x7ff94410f6a0 con 0x7ff9441042a0 2024-02-19T21:15:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff94411a8b0 con 0x7ff9441042a0 2024-02-19T21:15:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 40733028 0 0) 0x7ff930003030 con 0x7ff9440f89a0 2024-02-19T21:15:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff94410f6a0 con 0x7ff9440f89a0 2024-02-19T21:15:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff940001980 con 0x7ff9440f89a0 2024-02-19T21:15:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1647652382 0 0) 0x7ff94411ba50 con 0x7ff9440f9490 2024-02-19T21:15:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff930003030 con 0x7ff9440f9490 2024-02-19T21:15:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff9340018a0 con 0x7ff9440f9490 2024-02-19T21:15:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 378621862 0 0) 0x7ff94411a8b0 con 0x7ff9441042a0 2024-02-19T21:15:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff94411ba50 con 0x7ff9441042a0 2024-02-19T21:15:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff93c002310 con 0x7ff9441042a0 2024-02-19T21:15:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 301982205 0 0) 0x7ff94410f6a0 con 0x7ff9440f89a0 2024-02-19T21:15:52.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 >> v1:172.21.15.193:6789/0 conn(0x7ff9440f9490 legacy=0x7ff944114a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 >> v1:172.21.15.163:6789/0 conn(0x7ff9441042a0 legacy=0x7ff944118180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94411cc30 con 0x7ff9440f89a0 2024-02-19T21:15:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff94c199700 1 -- 172.21.15.163:0/3092656493 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff944119900 con 0x7ff9440f89a0 2024-02-19T21:15:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff94c199700 10 client.50456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff94c199700 1 -- 172.21.15.163:0/3092656493 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff944119d70 con 0x7ff9440f89a0 2024-02-19T21:15:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff94c199700 1 librados: init done 2024-02-19T21:15:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff940001c40 con 0x7ff9440f89a0 2024-02-19T21:15:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.688+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff940005b70 con 0x7ff9440f89a0 2024-02-19T21:15:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.689+0000 7ff94c199700 1 -- 172.21.15.163:0/3092656493 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff928004f20 con 0x7ff9440f89a0 2024-02-19T21:15:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.689+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ff940002270 con 0x7ff9440f89a0 2024-02-19T21:15:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.690+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(703..703 src has 623..703) v4 ==== 8325+0+0 (unknown 1782079745 0 0) 0x7ff940078e20 con 0x7ff9440f89a0 2024-02-19T21:15:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.690+0000 7ff93affd700 10 client.50456.objecter ms_dispatch 0x7ff944001550 osd_map(703..703 src has 623..703) v4 2024-02-19T21:15:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.690+0000 7ff93affd700 3 client.50456.objecter handle_osd_map got epochs [703,703] > 0 2024-02-19T21:15:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.690+0000 7ff93affd700 3 client.50456.objecter handle_osd_map decoding full epoch 703 2024-02-19T21:15:52.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.690+0000 7ff93affd700 20 client.50456.objecter dump_active .. 0 homeless 2024-02-19T21:15:52.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.690+0000 7ff93affd700 10 client.50456.objecter ms_handle_connect 0x7ff930063c20 2024-02-19T21:15:52.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.694+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff94004c110 con 0x7ff9440f89a0 2024-02-19T21:15:52.877 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:52.877+0000 7ff94c199700 1 -- 172.21.15.163:0/3092656493 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7ff9280051d0 con 0x7ff9440f89a0 2024-02-19T21:15:52.937 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:52.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:51.186571+0000 osd.6 (osd.6) 243 : cluster [DBG] 2.22 deep-scrub starts 2024-02-19T21:15:52.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:51.187597+0000 osd.6 (osd.6) 244 : cluster [DBG] 2.22 deep-scrub ok 2024-02-19T21:15:52.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:51.237157+0000 osd.2 (osd.2) 284 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:15:52.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:51.240042+0000 osd.2 (osd.2) 285 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:15:52.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:51.311238+0000 mgr.y (mgr.14550) 593 : cluster [DBG] pgmap v1058: 214 pgs: 2 active+clean+snaptrim, 212 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-19T21:15:52.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:51.950519+0000 osd.7 (osd.7) 321 : cluster [DBG] 234.7 scrub starts 2024-02-19T21:15:52.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:51.952186+0000 osd.7 (osd.7) 322 : cluster [DBG] 234.7 scrub ok 2024-02-19T21:15:52.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:52.398978+0000 mon.a (mon.0) 3419 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2024-02-19T21:15:52.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:52.419238+0000 mon.c (mon.2) 427 : audit [DBG] from='client.? v1:172.21.15.163:0/850998758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:52.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:51.186571+0000 osd.6 (osd.6) 243 : cluster [DBG] 2.22 deep-scrub starts 2024-02-19T21:15:52.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:51.187597+0000 osd.6 (osd.6) 244 : cluster [DBG] 2.22 deep-scrub ok 2024-02-19T21:15:52.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:51.237157+0000 osd.2 (osd.2) 284 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:15:52.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:51.240042+0000 osd.2 (osd.2) 285 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:15:52.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:51.311238+0000 mgr.y (mgr.14550) 593 : cluster [DBG] pgmap v1058: 214 pgs: 2 active+clean+snaptrim, 212 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-19T21:15:53.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:51.950519+0000 osd.7 (osd.7) 321 : cluster [DBG] 234.7 scrub starts 2024-02-19T21:15:53.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:51.952186+0000 osd.7 (osd.7) 322 : cluster [DBG] 234.7 scrub ok 2024-02-19T21:15:53.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:52.398978+0000 mon.a (mon.0) 3419 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2024-02-19T21:15:53.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:52.419238+0000 mon.c (mon.2) 427 : audit [DBG] from='client.? v1:172.21.15.163:0/850998758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:53.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:51.186571+0000 osd.6 (osd.6) 243 : cluster [DBG] 2.22 deep-scrub starts 2024-02-19T21:15:53.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:51.187597+0000 osd.6 (osd.6) 244 : cluster [DBG] 2.22 deep-scrub ok 2024-02-19T21:15:53.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:51.237157+0000 osd.2 (osd.2) 284 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:15:53.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:51.240042+0000 osd.2 (osd.2) 285 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:15:53.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:51.311238+0000 mgr.y (mgr.14550) 593 : cluster [DBG] pgmap v1058: 214 pgs: 2 active+clean+snaptrim, 212 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-02-19T21:15:53.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:51.950519+0000 osd.7 (osd.7) 321 : cluster [DBG] 234.7 scrub starts 2024-02-19T21:15:53.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:51.952186+0000 osd.7 (osd.7) 322 : cluster [DBG] 234.7 scrub ok 2024-02-19T21:15:53.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:52.398978+0000 mon.a (mon.0) 3419 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2024-02-19T21:15:53.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:52.419238+0000 mon.c (mon.2) 427 : audit [DBG] from='client.? v1:172.21.15.163:0/850998758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:15:53.221 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:53.225 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:53.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:53.326 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:15:53.356 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:53.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:53.404+0000 7ff93affd700 1 -- 172.21.15.163:0/3092656493 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v704) v1 ==== 113+0+0 (unknown 3754568417 0 0) 0x7ff94004a520 con 0x7ff9440f89a0 2024-02-19T21:15:53.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:53.414+0000 7ff94c199700 10 librados: watch_flush enter 2024-02-19T21:15:53.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:53.414+0000 7ff94c199700 10 librados: watch_flush exit 2024-02-19T21:15:53.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:53.414+0000 7ff94c199700 20 client.50456.objecter shutdown clearing up homeless session... 2024-02-19T21:15:53.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:53.414+0000 7ff94c199700 10 client.50456.objecter successfully canceled tick 2024-02-19T21:15:53.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:53.415+0000 7ff94c199700 1 -- 172.21.15.163:0/3092656493 >> v1:172.21.15.163:6800/1552514681 conn(0x7ff930063c20 legacy=0x7ff9300660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:53.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:53.415+0000 7ff94c199700 1 -- 172.21.15.163:0/3092656493 >> v1:172.21.15.163:6790/0 conn(0x7ff9440f89a0 legacy=0x7ff94410ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:53.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:53.416+0000 7ff94c199700 1 -- 172.21.15.163:0/3092656493 shutdown_connections 2024-02-19T21:15:53.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:53.416+0000 7ff94c199700 1 -- 172.21.15.163:0/3092656493 >> 172.21.15.163:0/3092656493 conn(0x7ff9440f2470 msgr2=0x7ff944107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:53.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:53.416+0000 7ff94c199700 1 -- 172.21.15.163:0/3092656493 shutdown_connections 2024-02-19T21:15:53.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:53.416+0000 7ff94c199700 1 -- 172.21.15.163:0/3092656493 wait complete. 2024-02-19T21:15:53.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:53.416+0000 7ff94c199700 1 librados: shutdown 2024-02-19T21:15:53.422 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.0 to 1 (10000) 2024-02-19T21:15:53.554 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:15:53.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:53.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:15:53.657 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:53.686 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:53.748 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:15:53] "GET /metrics HTTP/1.1" 200 192121 "" "Prometheus/2.33.4" 2024-02-19T21:15:53.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:52.228001+0000 osd.6 (osd.6) 245 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:15:53.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:52.229064+0000 osd.6 (osd.6) 246 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:15:53.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:52.878330+0000 mon.c (mon.2) 428 : audit [INF] from='client.? v1:172.21.15.163:0/3092656493' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:15:53.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:52.878612+0000 mon.a (mon.0) 3420 : audit [INF] from='client.50456 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:15:53.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:53.011942+0000 osd.1 (osd.1) 316 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:15:53.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:53.012284+0000 osd.1 (osd.1) 317 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:15:53.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:53.079354+0000 osd.4 (osd.4) 247 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:15:53.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:53.080214+0000 osd.4 (osd.4) 248 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:15:53.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:53.404158+0000 mon.a (mon.0) 3421 : audit [INF] from='client.50456 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-19T21:15:53.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:53.404231+0000 mon.a (mon.0) 3422 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2024-02-19T21:15:53.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:53.409328+0000 mon.a (mon.0) 3423 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:53.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:52.228001+0000 osd.6 (osd.6) 245 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:15:53.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:52.229064+0000 osd.6 (osd.6) 246 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:15:53.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:52.878330+0000 mon.c (mon.2) 428 : audit [INF] from='client.? v1:172.21.15.163:0/3092656493' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:15:53.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:52.878612+0000 mon.a (mon.0) 3420 : audit [INF] from='client.50456 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:15:53.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:53.011942+0000 osd.1 (osd.1) 316 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:15:53.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:53.012284+0000 osd.1 (osd.1) 317 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:15:53.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:53.079354+0000 osd.4 (osd.4) 247 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:15:53.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:53.080214+0000 osd.4 (osd.4) 248 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:15:53.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:53.404158+0000 mon.a (mon.0) 3421 : audit [INF] from='client.50456 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-19T21:15:53.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:53.404231+0000 mon.a (mon.0) 3422 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2024-02-19T21:15:53.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:53.409328+0000 mon.a (mon.0) 3423 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:53.760 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:53.763 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:53.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:53.783 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:52.228001+0000 osd.6 (osd.6) 245 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:15:53.784 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:52.229064+0000 osd.6 (osd.6) 246 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:15:53.784 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:52.878330+0000 mon.c (mon.2) 428 : audit [INF] from='client.? v1:172.21.15.163:0/3092656493' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:15:53.784 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:52.878612+0000 mon.a (mon.0) 3420 : audit [INF] from='client.50456 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:15:53.784 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:53.011942+0000 osd.1 (osd.1) 316 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:15:53.784 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:53.012284+0000 osd.1 (osd.1) 317 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:15:53.784 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:53.079354+0000 osd.4 (osd.4) 247 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:15:53.784 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:53.080214+0000 osd.4 (osd.4) 248 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:15:53.784 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:53.404158+0000 mon.a (mon.0) 3421 : audit [INF] from='client.50456 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-19T21:15:53.784 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:53.404231+0000 mon.a (mon.0) 3422 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2024-02-19T21:15:53.784 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:53.409328+0000 mon.a (mon.0) 3423 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:15:53.864 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:15:53.891 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:54.046 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:15:54.047 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-02-19T21:15:54.047 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-02-19T21:15:54.047 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-02-19T21:15:54.047 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.047 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "MOSDECSubOpWrite(136.6s2 196/195 ECSubWrite(tid=402, reqid=client.47941.0:1, at_version=196'1, trim_to=0'0, roll_forward_to=0'0))", 2024-02-19T21:15:54.047 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:06:16.196362+0000", 2024-02-19T21:15:54.047 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 577.84811337300005, 2024-02-19T21:15:54.047 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010287370000000001, 2024-02-19T21:15:54.047 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.047 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-02-19T21:15:54.047 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.047 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.048 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.048 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196362+0000", 2024-02-19T21:15:54.048 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.048 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.048 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.048 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.048 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196362+0000", 2024-02-19T21:15:54.048 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:15:54.048 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.048 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.048 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.048 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196361+0000", 2024-02-19T21:15:54.048 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3603999999999999e-05 2024-02-19T21:15:54.048 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.049 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.049 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.049 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196395+0000", 2024-02-19T21:15:54.049 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.551e-06 2024-02-19T21:15:54.049 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.049 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.049 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.049 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196404+0000", 2024-02-19T21:15:54.049 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8769999999999993e-06 2024-02-19T21:15:54.049 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.049 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.049 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.049 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196411+0000", 2024-02-19T21:15:54.049 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00026643000000000001 2024-02-19T21:15:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196678+0000", 2024-02-19T21:15:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0366e-05 2024-02-19T21:15:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:15:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.196688+0000", 2024-02-19T21:15:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0099375339999999996 2024-02-19T21:15:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:15:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.206626+0000", 2024-02-19T21:15:54.050 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3348000000000001e-05 2024-02-19T21:15:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:16.206649+0000", 2024-02-19T21:15:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010287370000000001 2024-02-19T21:15:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.25753.0:3126 186.3 e294/290 186:c0520479:test-rados-api-smithi163-83928-38::foo:head v 294'2, mlcod=0'0)", 2024-02-19T21:15:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:54.225630+0000", 2024-02-19T21:15:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 479.81884518099997, 2024-02-19T21:15:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017481528, 2024-02-19T21:15:54.051 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.225630+0000", 2024-02-19T21:15:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.225630+0000", 2024-02-19T21:15:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:15:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.052 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.225628+0000", 2024-02-19T21:15:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015168282 2024-02-19T21:15:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.240797+0000", 2024-02-19T21:15:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3131e-05 2024-02-19T21:15:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.240810+0000", 2024-02-19T21:15:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2637e-05 2024-02-19T21:15:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.053 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.240832+0000", 2024-02-19T21:15:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4966999999999998e-05 2024-02-19T21:15:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.240877+0000", 2024-02-19T21:15:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5106e-05 2024-02-19T21:15:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.240903+0000", 2024-02-19T21:15:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0021667320000000002 2024-02-19T21:15:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.054 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:15:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.243069+0000", 2024-02-19T21:15:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2271999999999998e-05 2024-02-19T21:15:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.243112+0000", 2024-02-19T21:15:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.017481528 2024-02-19T21:15:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:1423 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94681540390272 gen 2] snapc 0=[] ondisk+write+known_if_redirected e433)", 2024-02-19T21:15:54.055 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:10:40.311888+0000", 2024-02-19T21:15:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 313.732586793, 2024-02-19T21:15:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.01168904, 2024-02-19T21:15:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:15:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:15:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:15:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:15:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1423 2024-02-19T21:15:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311888+0000", 2024-02-19T21:15:54.056 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311888+0000", 2024-02-19T21:15:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:15:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311886+0000", 2024-02-19T21:15:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0350000000000007e-06 2024-02-19T21:15:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.057 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.058 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.058 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311895+0000", 2024-02-19T21:15:54.058 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0015e-05 2024-02-19T21:15:54.058 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.058 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.058 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.058 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311905+0000", 2024-02-19T21:15:54.058 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.0219999999999999e-06 2024-02-19T21:15:54.058 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.058 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.058 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.058 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311912+0000", 2024-02-19T21:15:54.058 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00048395300000000002 2024-02-19T21:15:54.058 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.059 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.059 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.059 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.312396+0000", 2024-02-19T21:15:54.059 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7010000000000002e-06 2024-02-19T21:15:54.059 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.059 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.059 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:15:54.059 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.312404+0000", 2024-02-19T21:15:54.059 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.001316297 2024-02-19T21:15:54.059 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.059 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.059 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.059 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.313721+0000", 2024-02-19T21:15:54.059 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000109879 2024-02-19T21:15:54.060 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.060 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.060 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for missing object", 2024-02-19T21:15:54.060 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.313831+0000", 2024-02-19T21:15:54.060 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0062348129999999996 2024-02-19T21:15:54.060 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.060 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.060 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.060 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.320065+0000", 2024-02-19T21:15:54.060 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000102769 2024-02-19T21:15:54.060 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.060 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.060 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:15:54.061 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.320168+0000", 2024-02-19T21:15:54.061 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00084819799999999999 2024-02-19T21:15:54.061 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.061 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.061 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.061 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.321016+0000", 2024-02-19T21:15:54.061 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5853999999999997e-05 2024-02-19T21:15:54.061 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.061 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.061 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.061 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.321082+0000", 2024-02-19T21:15:54.061 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00249519 2024-02-19T21:15:54.061 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.061 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.062 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.062 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.323577+0000", 2024-02-19T21:15:54.062 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.01168904 2024-02-19T21:15:54.062 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.062 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.062 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.062 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.062 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.062 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(osd.3.25:34 216.16 216:6dfc172a:test-rados-api-smithi163-83928-70::aab9b7bd17003da21256cca5c7d2d7ff4ea384b6:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e452)", 2024-02-19T21:15:54.062 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:11:04.382329+0000", 2024-02-19T21:15:54.062 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 289.66214631999998, 2024-02-19T21:15:54.062 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026243446, 2024-02-19T21:15:54.062 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.062 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382329+0000", 2024-02-19T21:15:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382329+0000", 2024-02-19T21:15:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:15:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382326+0000", 2024-02-19T21:15:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0463e-05 2024-02-19T21:15:54.064 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.064 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.064 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.064 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382337+0000", 2024-02-19T21:15:54.064 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.613e-05 2024-02-19T21:15:54.064 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.064 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.064 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.064 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382353+0000", 2024-02-19T21:15:54.064 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6540000000000002e-06 2024-02-19T21:15:54.064 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.064 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.064 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.064 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382360+0000", 2024-02-19T21:15:54.065 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7012000000000002e-05 2024-02-19T21:15:54.065 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.065 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.065 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.065 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382417+0000", 2024-02-19T21:15:54.065 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000108528 2024-02-19T21:15:54.065 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.065 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.065 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.065 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382525+0000", 2024-02-19T21:15:54.065 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00020552499999999999 2024-02-19T21:15:54.065 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.065 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.065 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-02-19T21:15:54.065 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.382731+0000", 2024-02-19T21:15:54.066 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00039365899999999998 2024-02-19T21:15:54.066 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.066 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.066 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:15:54.066 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.383124+0000", 2024-02-19T21:15:54.066 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0028295439999999998 2024-02-19T21:15:54.066 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.066 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.066 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:15:54.066 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.385954+0000", 2024-02-19T21:15:54.066 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.022528835000000001 2024-02-19T21:15:54.066 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.066 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.066 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:15:54.066 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.408483+0000", 2024-02-19T21:15:54.067 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3378000000000003e-05 2024-02-19T21:15:54.067 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.067 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.067 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:15:54.067 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.408536+0000", 2024-02-19T21:15:54.067 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6229999999999997e-05 2024-02-19T21:15:54.067 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.067 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.067 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.067 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.408572+0000", 2024-02-19T21:15:54.067 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026243446 2024-02-19T21:15:54.067 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.067 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.067 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.067 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.068 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.068 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:7756 227.10 227:0abed01e:test-rados-api-smithi163-83928-91::foo:head [call] snapc 0=[] ondisk+write+known_if_redirected+returnvec e549)", 2024-02-19T21:15:54.068 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:13:05.323761+0000", 2024-02-19T21:15:54.068 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 168.72071463899999, 2024-02-19T21:15:54.068 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0066392029999999998, 2024-02-19T21:15:54.068 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.068 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:54.068 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:15:54.068 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:15:54.068 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:15:54.068 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 7756 2024-02-19T21:15:54.068 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.068 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.068 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.068 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.069 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:05.323761+0000", 2024-02-19T21:15:54.069 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.069 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.069 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.069 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.069 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:05.323761+0000", 2024-02-19T21:15:54.069 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.99999 2024-02-19T21:15:54.069 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.069 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.069 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.069 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:05.323751+0000", 2024-02-19T21:15:54.069 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1188000000000001e-05 2024-02-19T21:15:54.069 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.069 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.070 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.070 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:05.323772+0000", 2024-02-19T21:15:54.070 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8839e-05 2024-02-19T21:15:54.070 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.070 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.070 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.070 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:05.323791+0000", 2024-02-19T21:15:54.070 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1810000000000001e-06 2024-02-19T21:15:54.070 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.070 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.070 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.070 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:05.323799+0000", 2024-02-19T21:15:54.070 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0300999999999997e-05 2024-02-19T21:15:54.070 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.071 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.071 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.071 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:05.323859+0000", 2024-02-19T21:15:54.071 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9480000000000008e-06 2024-02-19T21:15:54.071 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.071 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.071 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:15:54.071 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:05.323867+0000", 2024-02-19T21:15:54.071 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001947692 2024-02-19T21:15:54.071 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.071 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.071 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.071 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:05.325815+0000", 2024-02-19T21:15:54.071 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0045818380000000004 2024-02-19T21:15:54.071 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.072 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.072 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:15:54.072 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:05.330397+0000", 2024-02-19T21:15:54.072 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1609999999999999e-06 2024-02-19T21:15:54.072 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.072 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.072 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.072 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:05.330400+0000", 2024-02-19T21:15:54.072 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0066392029999999998 2024-02-19T21:15:54.072 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.072 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.072 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.072 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.073 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.073 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.48988.0:4 231.6 231:602f83fe:::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e570)", 2024-02-19T21:15:54.073 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:13:26.354790+0000", 2024-02-19T21:15:54.073 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.68968515099999, 2024-02-19T21:15:54.073 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0095397639999999992, 2024-02-19T21:15:54.073 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.073 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:15:54.073 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:15:54.073 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.48988", 2024-02-19T21:15:54.073 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/472848668", 2024-02-19T21:15:54.073 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 4 2024-02-19T21:15:54.073 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.073 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.073 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.073 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.074 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.354790+0000", 2024-02-19T21:15:54.074 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.074 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.074 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.074 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.074 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.354790+0000", 2024-02-19T21:15:54.074 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:15:54.074 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.074 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.074 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.074 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.354788+0000", 2024-02-19T21:15:54.074 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9910000000000002e-06 2024-02-19T21:15:54.074 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.074 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.075 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.075 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.354796+0000", 2024-02-19T21:15:54.075 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7737999999999999e-05 2024-02-19T21:15:54.075 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.075 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.075 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.075 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.354814+0000", 2024-02-19T21:15:54.075 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8029999999999997e-06 2024-02-19T21:15:54.075 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.075 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.075 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.075 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.354822+0000", 2024-02-19T21:15:54.075 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6804999999999995e-05 2024-02-19T21:15:54.075 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.076 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.076 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.076 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.354888+0000", 2024-02-19T21:15:54.076 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2943999999999996e-05 2024-02-19T21:15:54.076 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.076 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.076 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.076 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.354981+0000", 2024-02-19T21:15:54.076 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0091542280000000004 2024-02-19T21:15:54.076 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.076 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.076 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.076 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.364136+0000", 2024-02-19T21:15:54.076 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000100899 2024-02-19T21:15:54.077 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.077 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.077 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.077 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.364236+0000", 2024-02-19T21:15:54.077 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.3343000000000004e-05 2024-02-19T21:15:54.077 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.077 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.077 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.077 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.364330+0000", 2024-02-19T21:15:54.077 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0095397639999999992 2024-02-19T21:15:54.077 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.077 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.077 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.077 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.078 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.078 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.48988.0:5 231.6 231:602f83fe:::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e570)", 2024-02-19T21:15:54.078 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:13:26.354840+0000", 2024-02-19T21:15:54.078 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 147.689635366, 2024-02-19T21:15:54.078 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010454751, 2024-02-19T21:15:54.078 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.078 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:54.078 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:15:54.078 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.48988", 2024-02-19T21:15:54.078 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/472848668", 2024-02-19T21:15:54.078 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 5 2024-02-19T21:15:54.078 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.078 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.078 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.354840+0000", 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.354840+0000", 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.354839+0000", 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8640000000000001e-06 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.079 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.080 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.354845+0000", 2024-02-19T21:15:54.080 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1430000000000001e-06 2024-02-19T21:15:54.080 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.080 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.080 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.080 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.354852+0000", 2024-02-19T21:15:54.080 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1669999999999998e-06 2024-02-19T21:15:54.080 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.080 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.080 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.080 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.354857+0000", 2024-02-19T21:15:54.080 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00030082699999999998 2024-02-19T21:15:54.080 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.081 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.081 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.081 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.355158+0000", 2024-02-19T21:15:54.081 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2107999999999997e-05 2024-02-19T21:15:54.081 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.081 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.081 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:15:54.081 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.355200+0000", 2024-02-19T21:15:54.081 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0091748939999999994 2024-02-19T21:15:54.081 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.081 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.081 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.081 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.364375+0000", 2024-02-19T21:15:54.081 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1203000000000003e-05 2024-02-19T21:15:54.081 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.082 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.082 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.082 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.364436+0000", 2024-02-19T21:15:54.082 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5255999999999997e-05 2024-02-19T21:15:54.082 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.082 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.082 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-02-19T21:15:54.082 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.364531+0000", 2024-02-19T21:15:54.082 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00032789199999999998 2024-02-19T21:15:54.082 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.082 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.082 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:15:54.082 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.364859+0000", 2024-02-19T21:15:54.082 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00031279799999999998 2024-02-19T21:15:54.082 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.083 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.083 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:15:54.083 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.365172+0000", 2024-02-19T21:15:54.083 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2629e-05 2024-02-19T21:15:54.083 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.083 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.083 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:15:54.083 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.365224+0000", 2024-02-19T21:15:54.083 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1742999999999997e-05 2024-02-19T21:15:54.083 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.083 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.083 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:15:54.083 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.365256+0000", 2024-02-19T21:15:54.083 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8336e-05 2024-02-19T21:15:54.084 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.084 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.084 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.084 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:13:26.365295+0000", 2024-02-19T21:15:54.084 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010454751 2024-02-19T21:15:54.084 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.084 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.084 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.084 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.084 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.084 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:425 238.11 238:88000000::::head [pgnls start_epoch 639] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e639)", 2024-02-19T21:15:54.084 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:14:43.377628+0000", 2024-02-19T21:15:54.084 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 70.666847625000003, 2024-02-19T21:15:54.084 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0016352929999999999, 2024-02-19T21:15:54.085 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.085 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:15:54.085 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:15:54.085 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:15:54.085 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:15:54.085 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 425 2024-02-19T21:15:54.085 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.085 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.085 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.085 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.085 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:43.377628+0000", 2024-02-19T21:15:54.085 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.085 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.085 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.085 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.086 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:43.377628+0000", 2024-02-19T21:15:54.086 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:15:54.086 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.086 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.086 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.086 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:43.377625+0000", 2024-02-19T21:15:54.086 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0339e-05 2024-02-19T21:15:54.086 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.086 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.086 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.086 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:43.377635+0000", 2024-02-19T21:15:54.086 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6918e-05 2024-02-19T21:15:54.086 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.086 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.087 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.087 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:43.377652+0000", 2024-02-19T21:15:54.087 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9400000000000008e-06 2024-02-19T21:15:54.087 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.087 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.087 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.087 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:43.377661+0000", 2024-02-19T21:15:54.087 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3665000000000003e-05 2024-02-19T21:15:54.087 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.087 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.087 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.087 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:43.377704+0000", 2024-02-19T21:15:54.087 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6360000000000002e-06 2024-02-19T21:15:54.087 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.088 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.088 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:15:54.088 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:43.377709+0000", 2024-02-19T21:15:54.088 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014117680000000001 2024-02-19T21:15:54.088 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.088 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.088 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.088 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:43.379121+0000", 2024-02-19T21:15:54.088 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2626000000000002e-05 2024-02-19T21:15:54.088 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.088 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.088 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.088 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:43.379153+0000", 2024-02-19T21:15:54.088 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000109341 2024-02-19T21:15:54.089 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.089 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.089 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.089 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:43.379263+0000", 2024-02-19T21:15:54.089 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0016352929999999999 2024-02-19T21:15:54.089 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.089 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.089 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.089 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.089 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.089 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "MOSDECSubOpWrite(234.5s1 659/636 ECSubWrite(tid=643, reqid=client.23517.0:796, at_version=659'9, trim_to=0'0, roll_forward_to=658'7))", 2024-02-19T21:15:54.089 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:03.386686+0000", 2024-02-19T21:15:54.089 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 50.657788947999997, 2024-02-19T21:15:54.089 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00081914599999999998, 2024-02-19T21:15:54.089 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.090 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-02-19T21:15:54.090 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.090 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.090 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.090 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.386686+0000", 2024-02-19T21:15:54.090 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.090 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.090 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.090 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.090 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.386686+0000", 2024-02-19T21:15:54.090 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:15:54.090 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.090 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.090 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.386686+0000", 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8263999999999997e-05 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.386734+0000", 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2645e-05 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.386747+0000", 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5379999999999997e-06 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.091 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.386754+0000", 2024-02-19T21:15:54.092 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3065999999999995e-05 2024-02-19T21:15:54.092 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.092 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.092 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.092 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.386827+0000", 2024-02-19T21:15:54.092 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3722e-05 2024-02-19T21:15:54.092 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.092 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.092 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:15:54.092 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.386841+0000", 2024-02-19T21:15:54.092 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00064044000000000004 2024-02-19T21:15:54.092 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.092 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.092 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:15:54.093 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387482+0000", 2024-02-19T21:15:54.093 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3765999999999999e-05 2024-02-19T21:15:54.093 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.093 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.093 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.093 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387505+0000", 2024-02-19T21:15:54.093 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00081914599999999998 2024-02-19T21:15:54.093 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.093 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.093 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.093 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.093 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.093 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "MOSDECSubOpWrite(234.5s1 659/636 ECSubWrite(tid=644, reqid=client.23517.0:796, at_version=0'0, trim_to=0'0, roll_forward_to=659'9))", 2024-02-19T21:15:54.093 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:03.387828+0000", 2024-02-19T21:15:54.093 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 50.656646680999998, 2024-02-19T21:15:54.094 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00073132399999999999, 2024-02-19T21:15:54.094 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.094 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "reached pg", 2024-02-19T21:15:54.094 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.094 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.094 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.094 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387828+0000", 2024-02-19T21:15:54.094 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.094 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.094 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.094 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.094 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387828+0000", 2024-02-19T21:15:54.094 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:15:54.094 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.094 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.095 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.095 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387827+0000", 2024-02-19T21:15:54.095 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1249e-05 2024-02-19T21:15:54.095 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.095 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.095 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.095 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387838+0000", 2024-02-19T21:15:54.095 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4059999999999994e-06 2024-02-19T21:15:54.095 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.095 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.095 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.095 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387848+0000", 2024-02-19T21:15:54.095 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6529999999999999e-06 2024-02-19T21:15:54.095 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.095 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.096 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.096 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387853+0000", 2024-02-19T21:15:54.096 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5967999999999997e-05 2024-02-19T21:15:54.096 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.096 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.096 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.096 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387909+0000", 2024-02-19T21:15:54.096 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4370000000000001e-06 2024-02-19T21:15:54.096 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.096 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.096 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:15:54.096 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387917+0000", 2024-02-19T21:15:54.096 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00062045299999999998 2024-02-19T21:15:54.097 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.097 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.097 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:15:54.097 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.388537+0000", 2024-02-19T21:15:54.097 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2557000000000001e-05 2024-02-19T21:15:54.097 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.097 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.097 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.097 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.388560+0000", 2024-02-19T21:15:54.097 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00073132399999999999 2024-02-19T21:15:54.097 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.097 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.097 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.097 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.097 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.098 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:1042 242.1c 242:38000000::::head [pgnls start_epoch 673] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e673)", 2024-02-19T21:15:54.098 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:17.439404+0000", 2024-02-19T21:15:54.098 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 36.605071199000001, 2024-02-19T21:15:54.098 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024928929999999999, 2024-02-19T21:15:54.098 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.098 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:15:54.098 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:15:54.098 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:15:54.098 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:15:54.098 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1042 2024-02-19T21:15:54.098 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.098 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.098 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.098 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.098 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.439404+0000", 2024-02-19T21:15:54.099 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.099 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.099 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.099 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.099 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.439404+0000", 2024-02-19T21:15:54.099 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:15:54.099 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.099 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.099 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.099 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.439400+0000", 2024-02-19T21:15:54.099 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3457e-05 2024-02-19T21:15:54.099 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.099 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.099 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.100 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.439414+0000", 2024-02-19T21:15:54.100 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0678000000000001e-05 2024-02-19T21:15:54.100 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.100 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.100 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.100 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.439434+0000", 2024-02-19T21:15:54.100 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8730000000000007e-06 2024-02-19T21:15:54.100 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.100 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.100 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.100 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.439442+0000", 2024-02-19T21:15:54.100 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0436999999999998e-05 2024-02-19T21:15:54.100 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.439503+0000", 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1890000000000004e-06 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.439509+0000", 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002251631 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.441761+0000", 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9153999999999999e-05 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.101 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.102 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.441790+0000", 2024-02-19T21:15:54.102 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000107169 2024-02-19T21:15:54.102 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.102 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.102 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.102 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.441897+0000", 2024-02-19T21:15:54.102 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024928929999999999 2024-02-19T21:15:54.102 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.102 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.102 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.102 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.102 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.102 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:2063 5.e 5:7759931f:::notify.3:head [watch ping cookie 94681540390272 gen 4] snapc 0=[] ondisk+write+known_if_redirected e670)", 2024-02-19T21:15:54.102 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:19.751639+0000", 2024-02-19T21:15:54.102 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 34.292836633999997, 2024-02-19T21:15:54.103 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00051566399999999999, 2024-02-19T21:15:54.103 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.103 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:15:54.103 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:15:54.103 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:15:54.103 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:15:54.103 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2063 2024-02-19T21:15:54.103 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.103 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.103 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.103 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.103 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751639+0000", 2024-02-19T21:15:54.103 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.103 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.104 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.104 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.104 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751639+0000", 2024-02-19T21:15:54.104 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:15:54.104 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.104 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.104 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.104 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751635+0000", 2024-02-19T21:15:54.104 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3261000000000001e-05 2024-02-19T21:15:54.104 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.104 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.104 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.104 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751649+0000", 2024-02-19T21:15:54.104 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5491000000000001e-05 2024-02-19T21:15:54.105 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.105 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.105 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.105 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751664+0000", 2024-02-19T21:15:54.105 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5529999999999998e-06 2024-02-19T21:15:54.105 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.105 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.105 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.105 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751671+0000", 2024-02-19T21:15:54.105 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00025979600000000002 2024-02-19T21:15:54.105 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.105 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.105 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.105 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751931+0000", 2024-02-19T21:15:54.106 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8094000000000001e-05 2024-02-19T21:15:54.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.106 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.106 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.106 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.752019+0000", 2024-02-19T21:15:54.106 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000135532 2024-02-19T21:15:54.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.106 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.106 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.106 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.752154+0000", 2024-02-19T21:15:54.106 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00051566399999999999 2024-02-19T21:15:54.106 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.106 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.106 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.106 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.107 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.107 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:1101 243.c 243:3078e400:test-rados-api-smithi163-83928-119::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-02-19T21:15:54.107 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:23.483560+0000", 2024-02-19T21:15:54.107 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 30.560915344000001, 2024-02-19T21:15:54.107 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0028494449999999999, 2024-02-19T21:15:54.107 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.107 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:54.107 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:15:54.107 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:15:54.107 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:15:54.107 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1101 2024-02-19T21:15:54.107 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.107 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.107 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.107 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.108 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.483560+0000", 2024-02-19T21:15:54.108 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.108 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.108 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.483560+0000", 2024-02-19T21:15:54.108 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:15:54.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.108 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.108 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.483557+0000", 2024-02-19T21:15:54.108 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.294e-05 2024-02-19T21:15:54.108 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.108 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.109 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.109 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.483570+0000", 2024-02-19T21:15:54.109 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2761e-05 2024-02-19T21:15:54.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.109 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.109 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.483583+0000", 2024-02-19T21:15:54.109 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6590000000000001e-06 2024-02-19T21:15:54.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.109 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.109 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.109 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.483589+0000", 2024-02-19T21:15:54.109 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1228e-05 2024-02-19T21:15:54.109 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.110 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.110 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.483621+0000", 2024-02-19T21:15:54.110 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018257000000000001 2024-02-19T21:15:54.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.110 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:15:54.110 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.483803+0000", 2024-02-19T21:15:54.110 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0012920430000000001 2024-02-19T21:15:54.110 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.110 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.110 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.110 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.485095+0000", 2024-02-19T21:15:54.110 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010070500000000001 2024-02-19T21:15:54.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.111 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.111 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.111 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.485196+0000", 2024-02-19T21:15:54.111 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015650000000000001 2024-02-19T21:15:54.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.111 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.111 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 5,7", 2024-02-19T21:15:54.111 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.485352+0000", 2024-02-19T21:15:54.111 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00051155099999999995 2024-02-19T21:15:54.111 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.111 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.111 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:15:54.111 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.485864+0000", 2024-02-19T21:15:54.112 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00038839499999999998 2024-02-19T21:15:54.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.112 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:15:54.112 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.486252+0000", 2024-02-19T21:15:54.112 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4009000000000002e-05 2024-02-19T21:15:54.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.112 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:15:54.112 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.486326+0000", 2024-02-19T21:15:54.112 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4987000000000001e-05 2024-02-19T21:15:54.112 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.112 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.112 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:15:54.113 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.486371+0000", 2024-02-19T21:15:54.113 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7938000000000002e-05 2024-02-19T21:15:54.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.113 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.113 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.113 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.486409+0000", 2024-02-19T21:15:54.113 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0028494449999999999 2024-02-19T21:15:54.113 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.113 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.113 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.113 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.113 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.113 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:1142 243.c 243:3078e400:test-rados-api-smithi163-83928-119::foo:head [omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected e679)", 2024-02-19T21:15:54.113 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:23.507904+0000", 2024-02-19T21:15:54.114 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 30.536570914999999, 2024-02-19T21:15:54.114 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00092846400000000005, 2024-02-19T21:15:54.114 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.114 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:54.114 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:15:54.114 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:15:54.114 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:15:54.114 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1142 2024-02-19T21:15:54.114 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.114 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.114 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.114 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.114 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.507904+0000", 2024-02-19T21:15:54.115 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.115 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.507904+0000", 2024-02-19T21:15:54.115 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:15:54.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.115 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.507903+0000", 2024-02-19T21:15:54.115 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6919999999999996e-06 2024-02-19T21:15:54.115 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.115 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.115 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.116 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.507910+0000", 2024-02-19T21:15:54.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0293e-05 2024-02-19T21:15:54.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.116 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.116 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.507920+0000", 2024-02-19T21:15:54.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4820000000000002e-06 2024-02-19T21:15:54.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.116 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.116 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.507925+0000", 2024-02-19T21:15:54.116 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5793000000000001e-05 2024-02-19T21:15:54.116 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.116 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.117 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.117 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.507961+0000", 2024-02-19T21:15:54.117 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8267000000000001e-05 2024-02-19T21:15:54.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.117 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.117 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.117 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.508039+0000", 2024-02-19T21:15:54.117 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000115847 2024-02-19T21:15:54.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.117 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.117 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 5,7", 2024-02-19T21:15:54.117 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.508155+0000", 2024-02-19T21:15:54.117 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00034270699999999998 2024-02-19T21:15:54.117 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.118 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:15:54.118 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.508498+0000", 2024-02-19T21:15:54.118 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000244002 2024-02-19T21:15:54.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.118 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:15:54.118 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.508742+0000", 2024-02-19T21:15:54.118 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.1681999999999999e-05 2024-02-19T21:15:54.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.118 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.118 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:15:54.118 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.508774+0000", 2024-02-19T21:15:54.118 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2830000000000002e-05 2024-02-19T21:15:54.118 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.119 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:15:54.119 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.508806+0000", 2024-02-19T21:15:54.119 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6268999999999999e-05 2024-02-19T21:15:54.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.119 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.119 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.508833+0000", 2024-02-19T21:15:54.119 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00092846400000000005 2024-02-19T21:15:54.119 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.119 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.119 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.119 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.119 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.120 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:1143 243.c 243:3078e400:test-rados-api-smithi163-83928-119::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e679)", 2024-02-19T21:15:54.120 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:23.508996+0000", 2024-02-19T21:15:54.120 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 30.535479514999999, 2024-02-19T21:15:54.120 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0012472239999999999, 2024-02-19T21:15:54.120 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.120 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:15:54.120 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:15:54.120 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:15:54.120 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:15:54.120 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1143 2024-02-19T21:15:54.120 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.120 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.120 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.120 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.508996+0000", 2024-02-19T21:15:54.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.121 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.508996+0000", 2024-02-19T21:15:54.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:15:54.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.121 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.121 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.508994+0000", 2024-02-19T21:15:54.121 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7299999999999996e-06 2024-02-19T21:15:54.121 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.121 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.508999+0000", 2024-02-19T21:15:54.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.772e-06 2024-02-19T21:15:54.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.509008+0000", 2024-02-19T21:15:54.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2479999999999998e-06 2024-02-19T21:15:54.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.122 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.122 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.122 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.509012+0000", 2024-02-19T21:15:54.122 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5270999999999999e-05 2024-02-19T21:15:54.122 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.123 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.123 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.509037+0000", 2024-02-19T21:15:54.123 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.5866999999999997e-05 2024-02-19T21:15:54.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.123 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.123 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.509113+0000", 2024-02-19T21:15:54.123 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001129592 2024-02-19T21:15:54.123 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.123 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.123 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.123 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.510243+0000", 2024-02-19T21:15:54.123 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0012472239999999999 2024-02-19T21:15:54.123 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.124 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.124 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.124 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.124 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.7.0:681 244.17 e687/683 244:e873ac47:test-rados-api-smithi163-83928-121::foo:head v 687'1, mlcod=0'0)", 2024-02-19T21:15:54.124 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:36.534056+0000", 2024-02-19T21:15:54.124 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 17.510418788999999, 2024-02-19T21:15:54.124 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00059775599999999996, 2024-02-19T21:15:54.124 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.124 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:54.124 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.124 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.124 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.124 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534056+0000", 2024-02-19T21:15:54.125 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.125 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.125 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534056+0000", 2024-02-19T21:15:54.125 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:15:54.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.125 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.125 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534056+0000", 2024-02-19T21:15:54.125 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.1859999999999998e-06 2024-02-19T21:15:54.125 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.125 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.125 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.126 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534065+0000", 2024-02-19T21:15:54.126 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6373999999999999e-05 2024-02-19T21:15:54.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.126 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.126 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534091+0000", 2024-02-19T21:15:54.126 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.075e-05 2024-02-19T21:15:54.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.126 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.126 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534102+0000", 2024-02-19T21:15:54.126 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4262999999999999e-05 2024-02-19T21:15:54.126 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.126 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.126 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534156+0000", 2024-02-19T21:15:54.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0577000000000002e-05 2024-02-19T21:15:54.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.127 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534187+0000", 2024-02-19T21:15:54.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00041668100000000002 2024-02-19T21:15:54.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.127 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:15:54.127 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534604+0000", 2024-02-19T21:15:54.127 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0485999999999997e-05 2024-02-19T21:15:54.127 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.127 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.128 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.128 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534654+0000", 2024-02-19T21:15:54.128 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00059775599999999996 2024-02-19T21:15:54.128 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.128 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.128 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.128 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.128 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.128 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.23517.0:1294 244.17 e687/683 244:e873ac47:test-rados-api-smithi163-83928-121::foo:head v 687'2, mlcod=0'0)", 2024-02-19T21:15:54.128 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:36.534710+0000", 2024-02-19T21:15:54.128 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 17.509764774000001, 2024-02-19T21:15:54.128 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0026591789999999998, 2024-02-19T21:15:54.128 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.129 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:54.129 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.129 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.129 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.129 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534710+0000", 2024-02-19T21:15:54.129 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.129 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.129 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.129 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.129 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534710+0000", 2024-02-19T21:15:54.129 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:15:54.129 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.129 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.129 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.130 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534709+0000", 2024-02-19T21:15:54.130 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1661999999999999e-05 2024-02-19T21:15:54.130 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.130 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.130 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.130 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534720+0000", 2024-02-19T21:15:54.130 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7904e-05 2024-02-19T21:15:54.130 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.130 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.130 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.130 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534738+0000", 2024-02-19T21:15:54.130 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.8269999999999998e-06 2024-02-19T21:15:54.130 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.130 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.130 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.131 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534746+0000", 2024-02-19T21:15:54.131 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1121000000000002e-05 2024-02-19T21:15:54.131 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.131 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.131 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.131 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534787+0000", 2024-02-19T21:15:54.131 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3461000000000001e-05 2024-02-19T21:15:54.131 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.131 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.131 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.131 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534811+0000", 2024-02-19T21:15:54.131 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025237530000000001 2024-02-19T21:15:54.131 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.131 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.131 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:15:54.132 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.537334+0000", 2024-02-19T21:15:54.132 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5271999999999997e-05 2024-02-19T21:15:54.132 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.132 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.132 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.132 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.537370+0000", 2024-02-19T21:15:54.132 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0026591789999999998 2024-02-19T21:15:54.132 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.132 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.132 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.132 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.132 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.132 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.23517.0:1336 244.17 e687/683 244:e873ac47:test-rados-api-smithi163-83928-121::foo:head v 687'3, mlcod=687'1)", 2024-02-19T21:15:54.132 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:36.557743+0000", 2024-02-19T21:15:54.133 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 17.486732494000002, 2024-02-19T21:15:54.135 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000474442, 2024-02-19T21:15:54.135 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.135 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:15:54.135 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.135 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.135 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.135 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.557743+0000", 2024-02-19T21:15:54.135 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.135 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.135 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.135 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.135 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.557743+0000", 2024-02-19T21:15:54.136 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:15:54.136 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.136 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.136 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.136 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.557741+0000", 2024-02-19T21:15:54.136 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2028000000000001e-05 2024-02-19T21:15:54.136 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.136 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.136 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.136 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.557753+0000", 2024-02-19T21:15:54.136 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0885e-05 2024-02-19T21:15:54.136 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.136 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.136 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.136 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.557763+0000", 2024-02-19T21:15:54.137 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9359999999999998e-06 2024-02-19T21:15:54.137 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.137 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.137 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.137 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.557768+0000", 2024-02-19T21:15:54.137 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2293999999999999e-05 2024-02-19T21:15:54.137 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.137 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.137 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.137 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.557811+0000", 2024-02-19T21:15:54.137 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3266000000000001e-05 2024-02-19T21:15:54.137 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.137 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.137 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.137 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.557834+0000", 2024-02-19T21:15:54.138 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00035704699999999999 2024-02-19T21:15:54.138 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.138 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.138 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:15:54.138 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.558191+0000", 2024-02-19T21:15:54.138 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6139000000000002e-05 2024-02-19T21:15:54.138 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.138 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.138 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.138 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.558217+0000", 2024-02-19T21:15:54.138 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000474442 2024-02-19T21:15:54.138 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.138 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.138 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.138 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.139 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.139 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:1460 244.8 244:10000000::::head [pgnls start_epoch 689] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e689)", 2024-02-19T21:15:54.139 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:38.560942+0000", 2024-02-19T21:15:54.139 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 15.48353359, 2024-02-19T21:15:54.139 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020182080000000001, 2024-02-19T21:15:54.139 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.139 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:15:54.139 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:15:54.139 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:15:54.139 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:15:54.139 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1460 2024-02-19T21:15:54.139 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.139 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.139 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.139 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.140 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.560942+0000", 2024-02-19T21:15:54.140 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.140 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.140 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.140 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.560942+0000", 2024-02-19T21:15:54.140 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:15:54.140 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.140 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.140 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.560938+0000", 2024-02-19T21:15:54.140 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4763000000000001e-05 2024-02-19T21:15:54.140 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.140 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.141 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.141 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.560953+0000", 2024-02-19T21:15:54.141 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9128999999999999e-05 2024-02-19T21:15:54.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.141 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.141 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.141 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.560972+0000", 2024-02-19T21:15:54.141 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6369000000000001e-05 2024-02-19T21:15:54.141 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.141 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.141 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.141 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.560999+0000", 2024-02-19T21:15:54.141 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0797000000000002e-05 2024-02-19T21:15:54.142 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.142 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.142 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.142 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.561049+0000", 2024-02-19T21:15:54.142 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7389999999999996e-06 2024-02-19T21:15:54.142 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.142 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.142 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:15:54.142 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.561055+0000", 2024-02-19T21:15:54.142 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0017048790000000001 2024-02-19T21:15:54.142 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.142 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.142 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.142 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.562760+0000", 2024-02-19T21:15:54.143 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6439e-05 2024-02-19T21:15:54.143 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.143 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.143 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.143 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.562806+0000", 2024-02-19T21:15:54.143 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015335699999999999 2024-02-19T21:15:54.143 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.143 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.143 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.143 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.562960+0000", 2024-02-19T21:15:54.143 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020182080000000001 2024-02-19T21:15:54.143 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.143 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.144 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.144 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.144 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.144 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:2099 5.e 5:7759931f:::notify.3:head [watch ping cookie 94681540390272 gen 4] snapc 0=[] ondisk+write+known_if_redirected e685)", 2024-02-19T21:15:54.144 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:39.755631+0000", 2024-02-19T21:15:54.144 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 14.288844011, 2024-02-19T21:15:54.144 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00065781200000000002, 2024-02-19T21:15:54.144 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:15:54.144 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:15:54.144 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:15:54.144 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:15:54.144 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:15:54.144 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2099 2024-02-19T21:15:54.144 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.145 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:15:54.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.145 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:15:54.145 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755631+0000", 2024-02-19T21:15:54.145 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:15:54.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.145 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:15:54.145 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755631+0000", 2024-02-19T21:15:54.145 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999943 2024-02-19T21:15:54.145 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.145 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.145 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:15:54.145 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755626+0000", 2024-02-19T21:15:54.146 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9372999999999999e-05 2024-02-19T21:15:54.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.146 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:15:54.146 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755645+0000", 2024-02-19T21:15:54.146 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8699000000000002e-05 2024-02-19T21:15:54.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.146 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:15:54.146 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755694+0000", 2024-02-19T21:15:54.146 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0659999999999999e-05 2024-02-19T21:15:54.146 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.146 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.146 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:15:54.147 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755714+0000", 2024-02-19T21:15:54.147 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00030899600000000002 2024-02-19T21:15:54.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.147 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.147 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:15:54.147 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.756023+0000", 2024-02-19T21:15:54.147 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012438400000000001 2024-02-19T21:15:54.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.147 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.147 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:15:54.147 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.756148+0000", 2024-02-19T21:15:54.147 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014125699999999999 2024-02-19T21:15:54.147 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:15:54.148 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:15:54.148 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:15:54.148 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.756289+0000", 2024-02-19T21:15:54.148 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00065781200000000002 2024-02-19T21:15:54.148 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.148 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.148 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.148 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:15:54.148 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:15:54.148 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:15:54.736 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-19T21:15:54.844 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:53.311613+0000 mgr.y (mgr.14550) 594 : cluster [DBG] pgmap v1061: 182 pgs: 1 active+clean+snaptrim, 181 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-19T21:15:54.844 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:53.935311+0000 osd.1 (osd.1) 318 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:15:54.845 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:53.936140+0000 osd.1 (osd.1) 319 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:15:54.845 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:54.107885+0000 osd.3 (osd.3) 297 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:15:54.845 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:54.109420+0000 osd.3 (osd.3) 298 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:15:54.845 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:54.110877+0000 mon.a (mon.0) 3424 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:54.845 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:54.403869+0000 mon.a (mon.0) 3425 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:54.845 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:54.403994+0000 mon.a (mon.0) 3426 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2024-02-19T21:15:54.845 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:54.406916+0000 mon.a (mon.0) 3427 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:54.845 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:53.311613+0000 mgr.y (mgr.14550) 594 : cluster [DBG] pgmap v1061: 182 pgs: 1 active+clean+snaptrim, 181 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-19T21:15:54.845 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:53.935311+0000 osd.1 (osd.1) 318 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:15:54.845 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:53.936140+0000 osd.1 (osd.1) 319 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:15:54.845 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:54.107885+0000 osd.3 (osd.3) 297 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:15:54.846 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:54.109420+0000 osd.3 (osd.3) 298 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:15:54.846 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:54.110877+0000 mon.a (mon.0) 3424 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:54.846 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:54.403869+0000 mon.a (mon.0) 3425 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:54.846 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:54.403994+0000 mon.a (mon.0) 3426 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2024-02-19T21:15:54.846 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:54.406916+0000 mon.a (mon.0) 3427 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:55.002 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:15:55.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:53.311613+0000 mgr.y (mgr.14550) 594 : cluster [DBG] pgmap v1061: 182 pgs: 1 active+clean+snaptrim, 181 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-19T21:15:55.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:53.935311+0000 osd.1 (osd.1) 318 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:15:55.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:53.936140+0000 osd.1 (osd.1) 319 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:15:55.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:54.107885+0000 osd.3 (osd.3) 297 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:15:55.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:54.109420+0000 osd.3 (osd.3) 298 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:15:55.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:54.110877+0000 mon.a (mon.0) 3424 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:15:55.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:54.403869+0000 mon.a (mon.0) 3425 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:15:55.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:54.403994+0000 mon.a (mon.0) 3426 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2024-02-19T21:15:55.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:54.406916+0000 mon.a (mon.0) 3427 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:15:55.140 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:15:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:55.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:15:55.245 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:15:55.272 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:55.293 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:15:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:55.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:15:55.396 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:15:55.424 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:55.461 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:15:55.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:55.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:15:55.565 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:55.592 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:55.735 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-19T21:15:55.735 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:15:55.840 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:55.842 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:54.197551+0000 osd.6 (osd.6) 247 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:15:55.842 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:54.200439+0000 osd.6 (osd.6) 248 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:15:55.842 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:55.406151+0000 mon.a (mon.0) 3428 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-124", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:55.842 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:55.406216+0000 mon.a (mon.0) 3429 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2024-02-19T21:15:55.842 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:55.406532+0000 mon.a (mon.0) 3430 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-124"}]: dispatch 2024-02-19T21:15:55.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:55.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:55.944 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:55.971 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:55.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:54.197551+0000 osd.6 (osd.6) 247 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:15:55.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:54.200439+0000 osd.6 (osd.6) 248 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:15:55.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:55.406151+0000 mon.a (mon.0) 3428 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-124", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:55.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:55.406216+0000 mon.a (mon.0) 3429 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2024-02-19T21:15:55.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:55.406532+0000 mon.a (mon.0) 3430 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-124"}]: dispatch 2024-02-19T21:15:55.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:54.197551+0000 osd.6 (osd.6) 247 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:15:55.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:54.200439+0000 osd.6 (osd.6) 248 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:15:55.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:55.406151+0000 mon.a (mon.0) 3428 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-124", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:15:55.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:55.406216+0000 mon.a (mon.0) 3429 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2024-02-19T21:15:55.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:55.406532+0000 mon.a (mon.0) 3430 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-124"}]: dispatch 2024-02-19T21:15:56.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.002+0000 7f60ec285700 1 -- 172.21.15.163:0/2858109684 >> v1:172.21.15.163:6790/0 conn(0x7f60e4100780 legacy=0x7f60e4102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:56.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.002+0000 7f60ec285700 1 -- 172.21.15.163:0/2858109684 shutdown_connections 2024-02-19T21:15:56.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.002+0000 7f60ec285700 1 -- 172.21.15.163:0/2858109684 >> 172.21.15.163:0/2858109684 conn(0x7f60e40f35a0 msgr2=0x7f60e40f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:56.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.002+0000 7f60ec285700 1 -- 172.21.15.163:0/2858109684 shutdown_connections 2024-02-19T21:15:56.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.002+0000 7f60ec285700 1 -- 172.21.15.163:0/2858109684 wait complete. 2024-02-19T21:15:56.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.003+0000 7f60ec285700 1 librados: starting msgr at 2024-02-19T21:15:56.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.003+0000 7f60ec285700 1 librados: starting objecter 2024-02-19T21:15:56.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.003+0000 7f60ec285700 1 Processor -- start 2024-02-19T21:15:56.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.003+0000 7f60ec285700 1 -- start start 2024-02-19T21:15:56.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.003+0000 7f60ec285700 1 librados: setting wanted keys 2024-02-19T21:15:56.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.003+0000 7f60ec285700 1 librados: calling monclient init 2024-02-19T21:15:56.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.003+0000 7f60ec285700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f60e41172e0 con 0x7f60e41042c0 2024-02-19T21:15:56.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.003+0000 7f60ec285700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f60e41184c0 con 0x7f60e40fd980 2024-02-19T21:15:56.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.003+0000 7f60ec285700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f60e41196c0 con 0x7f60e4100780 2024-02-19T21:15:56.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.003+0000 7f60ea822700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f60e41042c0 0x7f60e41159a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:34950/0 (socket says 172.21.15.163:34950) 2024-02-19T21:15:56.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.003+0000 7f60ea822700 1 -- 172.21.15.163:0/2440301550 learned_addr learned my addr 172.21.15.163:0/2440301550 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:15:56.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.003+0000 7f60db779700 10 client.?.objecter ms_handle_connect 0x7f60e41042c0 2024-02-19T21:15:56.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.003+0000 7f60db779700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:56.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 10 client.?.objecter ms_handle_connect 0x7f60e40fd980 2024-02-19T21:15:56.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:56.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 10 client.?.objecter ms_handle_connect 0x7f60e4100780 2024-02-19T21:15:56.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 10 client.?.objecter resend_mon_ops 2024-02-19T21:15:56.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193955966 0 0) 0x7f60e41172e0 con 0x7f60e41042c0 2024-02-19T21:15:56.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60d00038e0 con 0x7f60e41042c0 2024-02-19T21:15:56.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82546908 0 0) 0x7f60e41196c0 con 0x7f60e4100780 2024-02-19T21:15:56.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e41172e0 con 0x7f60e4100780 2024-02-19T21:15:56.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946477773 0 0) 0x7f60e41184c0 con 0x7f60e40fd980 2024-02-19T21:15:56.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e41196c0 con 0x7f60e40fd980 2024-02-19T21:15:56.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2389937730 0 0) 0x7f60d00038e0 con 0x7f60e41042c0 2024-02-19T21:15:56.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60e41184c0 con 0x7f60e41042c0 2024-02-19T21:15:56.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f60dc001c80 con 0x7f60e41042c0 2024-02-19T21:15:56.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3768081546 0 0) 0x7f60e41172e0 con 0x7f60e4100780 2024-02-19T21:15:56.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60d00038e0 con 0x7f60e4100780 2024-02-19T21:15:56.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f60e0002de0 con 0x7f60e4100780 2024-02-19T21:15:56.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 275797186 0 0) 0x7f60e41196c0 con 0x7f60e40fd980 2024-02-19T21:15:56.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60e41172e0 con 0x7f60e40fd980 2024-02-19T21:15:56.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f60d4001550 con 0x7f60e40fd980 2024-02-19T21:15:56.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2427220632 0 0) 0x7f60e41184c0 con 0x7f60e41042c0 2024-02-19T21:15:56.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 >> v1:172.21.15.163:6790/0 conn(0x7f60e4100780 legacy=0x7f60e40fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:56.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 >> v1:172.21.15.193:6789/0 conn(0x7f60e40fd980 legacy=0x7f60e40f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:56.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e411a8c0 con 0x7f60e41042c0 2024-02-19T21:15:56.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60ec285700 1 -- 172.21.15.163:0/2440301550 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e41186b0 con 0x7f60e41042c0 2024-02-19T21:15:56.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60ec285700 10 client.23739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:15:56.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60ec285700 1 -- 172.21.15.163:0/2440301550 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e4118ba0 con 0x7f60e41042c0 2024-02-19T21:15:56.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60ec285700 1 librados: init done 2024-02-19T21:15:56.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f60dc0027a0 con 0x7f60e41042c0 2024-02-19T21:15:56.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.004+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f60dc005df0 con 0x7f60e41042c0 2024-02-19T21:15:56.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.005+0000 7f60ec285700 1 -- 172.21.15.163:0/2440301550 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c8004f20 con 0x7f60e41042c0 2024-02-19T21:15:56.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.006+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f60dc002300 con 0x7f60e41042c0 2024-02-19T21:15:56.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.006+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(706..706 src has 623..706) v4 ==== 8706+0+0 (unknown 266220233 0 0) 0x7f60dc0792e0 con 0x7f60e41042c0 2024-02-19T21:15:56.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.006+0000 7f60db779700 10 client.23739.objecter ms_dispatch 0x7f60e4001550 osd_map(706..706 src has 623..706) v4 2024-02-19T21:15:56.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.006+0000 7f60db779700 3 client.23739.objecter handle_osd_map got epochs [706,706] > 0 2024-02-19T21:15:56.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.006+0000 7f60db779700 3 client.23739.objecter handle_osd_map decoding full epoch 706 2024-02-19T21:15:56.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.006+0000 7f60db779700 20 client.23739.objecter dump_active .. 0 homeless 2024-02-19T21:15:56.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.006+0000 7f60db779700 10 client.23739.objecter ms_handle_connect 0x7f60d0063be0 2024-02-19T21:15:56.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.010+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f60dc04c470 con 0x7f60e41042c0 2024-02-19T21:15:56.089 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:56.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:15:56.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.202+0000 7f60ec285700 1 -- 172.21.15.163:0/2440301550 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f60c80051d0 con 0x7f60e41042c0 2024-02-19T21:15:56.219 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:56.222 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:56.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:56.323 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:15:56.350 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:56.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.407+0000 7f60db779700 1 -- 172.21.15.163:0/2440301550 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v707) v1 ==== 87+0+0 (unknown 1310859278 0 0) 0x7f60dc04a880 con 0x7f60e41042c0 2024-02-19T21:15:56.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.410+0000 7f60ec285700 10 librados: watch_flush enter 2024-02-19T21:15:56.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.410+0000 7f60ec285700 10 librados: watch_flush exit 2024-02-19T21:15:56.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.410+0000 7f60ec285700 20 client.23739.objecter shutdown clearing up homeless session... 2024-02-19T21:15:56.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.410+0000 7f60ec285700 10 client.23739.objecter successfully canceled tick 2024-02-19T21:15:56.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.410+0000 7f60ec285700 1 -- 172.21.15.163:0/2440301550 >> v1:172.21.15.163:6800/1552514681 conn(0x7f60d0063be0 legacy=0x7f60d00660a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:56.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.410+0000 7f60ec285700 1 -- 172.21.15.163:0/2440301550 >> v1:172.21.15.163:6789/0 conn(0x7f60e41042c0 legacy=0x7f60e41159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:15:56.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.410+0000 7f60ec285700 1 -- 172.21.15.163:0/2440301550 shutdown_connections 2024-02-19T21:15:56.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.410+0000 7f60ec285700 1 -- 172.21.15.163:0/2440301550 >> 172.21.15.163:0/2440301550 conn(0x7f60e40f35a0 msgr2=0x7f60e4107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:15:56.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.411+0000 7f60ec285700 1 -- 172.21.15.163:0/2440301550 shutdown_connections 2024-02-19T21:15:56.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.411+0000 7f60ec285700 1 -- 172.21.15.163:0/2440301550 wait complete. 2024-02-19T21:15:56.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:56.411+0000 7f60ec285700 1 librados: shutdown 2024-02-19T21:15:56.415 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:15:56.548 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:15:56.551 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:56.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:15:56.653 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:56.681 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:56.779 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:56.784 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:56.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:56.789 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:15:56.789 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:15:56.789 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-02-19T21:15:56.789 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:15:56.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:55.312522+0000 mgr.y (mgr.14550) 595 : cluster [DBG] pgmap v1064: 214 pgs: 11 creating+activating, 203 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-19T21:15:56.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:56.085016+0000 osd.3 (osd.3) 299 : cluster [DBG] 6.b scrub starts 2024-02-19T21:15:56.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:56.087840+0000 osd.3 (osd.3) 300 : cluster [DBG] 6.b scrub ok 2024-02-19T21:15:56.813 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:56.203545+0000 mon.a (mon.0) 3431 : audit [INF] from='client.? v1:172.21.15.163:0/2440301550' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:15:56.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:56.404191+0000 mon.a (mon.0) 3432 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:15:56.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:56.408408+0000 mon.a (mon.0) 3433 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-124"}]': finished 2024-02-19T21:15:56.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:56.408472+0000 mon.a (mon.0) 3434 : audit [INF] from='client.? v1:172.21.15.163:0/2440301550' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:15:56.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:56.408519+0000 mon.a (mon.0) 3435 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2024-02-19T21:15:56.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:56.411675+0000 mon.a (mon.0) 3436 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-124", "mode": "writeback"}]: dispatch 2024-02-19T21:15:56.885 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:15:56.915 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:56.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:55.312522+0000 mgr.y (mgr.14550) 595 : cluster [DBG] pgmap v1064: 214 pgs: 11 creating+activating, 203 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-19T21:15:56.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:56.085016+0000 osd.3 (osd.3) 299 : cluster [DBG] 6.b scrub starts 2024-02-19T21:15:56.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:56.087840+0000 osd.3 (osd.3) 300 : cluster [DBG] 6.b scrub ok 2024-02-19T21:15:56.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:56.203545+0000 mon.a (mon.0) 3431 : audit [INF] from='client.? v1:172.21.15.163:0/2440301550' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:15:56.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:56.404191+0000 mon.a (mon.0) 3432 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:15:56.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:56.408408+0000 mon.a (mon.0) 3433 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-124"}]': finished 2024-02-19T21:15:56.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:56.408472+0000 mon.a (mon.0) 3434 : audit [INF] from='client.? v1:172.21.15.163:0/2440301550' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:15:56.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:56.408519+0000 mon.a (mon.0) 3435 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2024-02-19T21:15:56.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:56.411675+0000 mon.a (mon.0) 3436 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-124", "mode": "writeback"}]: dispatch 2024-02-19T21:15:56.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:55.312522+0000 mgr.y (mgr.14550) 595 : cluster [DBG] pgmap v1064: 214 pgs: 11 creating+activating, 203 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-19T21:15:56.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:56.085016+0000 osd.3 (osd.3) 299 : cluster [DBG] 6.b scrub starts 2024-02-19T21:15:56.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:56.087840+0000 osd.3 (osd.3) 300 : cluster [DBG] 6.b scrub ok 2024-02-19T21:15:56.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:56.203545+0000 mon.a (mon.0) 3431 : audit [INF] from='client.? v1:172.21.15.163:0/2440301550' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:15:56.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:56.404191+0000 mon.a (mon.0) 3432 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:15:56.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:56.408408+0000 mon.a (mon.0) 3433 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-124"}]': finished 2024-02-19T21:15:56.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:56.408472+0000 mon.a (mon.0) 3434 : audit [INF] from='client.? v1:172.21.15.163:0/2440301550' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:15:57.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:56.408519+0000 mon.a (mon.0) 3435 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2024-02-19T21:15:57.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:56.411675+0000 mon.a (mon.0) 3436 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-124", "mode": "writeback"}]: dispatch 2024-02-19T21:15:57.062 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:15:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:57.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:15:57.166 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:57.194 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:57.317 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:57.320 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:57.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:57.421 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:15:57.451 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:57.468 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-19T21:15:57.687 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:15:57.698 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:15:57.702 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:57.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:15:57.719 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:56.204050+0000 osd.6 (osd.6) 249 : cluster [DBG] 234.6 scrub starts 2024-02-19T21:15:57.719 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:56.206170+0000 osd.6 (osd.6) 250 : cluster [DBG] 234.6 scrub ok 2024-02-19T21:15:57.719 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:57.405714+0000 mon.a (mon.0) 3437 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:57.719 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:57.414474+0000 mon.a (mon.0) 3438 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-124", "mode": "writeback"}]': finished 2024-02-19T21:15:57.719 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:57.414529+0000 mon.a (mon.0) 3439 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-02-19T21:15:57.719 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:57.453026+0000 mon.a (mon.0) 3440 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:57.733 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:15:57.803 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:15:57.832 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:57.963 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:56.204050+0000 osd.6 (osd.6) 249 : cluster [DBG] 234.6 scrub starts 2024-02-19T21:15:57.963 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:56.206170+0000 osd.6 (osd.6) 250 : cluster [DBG] 234.6 scrub ok 2024-02-19T21:15:57.963 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:57.405714+0000 mon.a (mon.0) 3437 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:57.963 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:57.414474+0000 mon.a (mon.0) 3438 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-124", "mode": "writeback"}]': finished 2024-02-19T21:15:57.963 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:57.414529+0000 mon.a (mon.0) 3439 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-02-19T21:15:57.963 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:57.453026+0000 mon.a (mon.0) 3440 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:57.963 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:56.204050+0000 osd.6 (osd.6) 249 : cluster [DBG] 234.6 scrub starts 2024-02-19T21:15:57.963 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:56.206170+0000 osd.6 (osd.6) 250 : cluster [DBG] 234.6 scrub ok 2024-02-19T21:15:57.964 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:57.405714+0000 mon.a (mon.0) 3437 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:15:57.964 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:57.414474+0000 mon.a (mon.0) 3438 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-124", "mode": "writeback"}]': finished 2024-02-19T21:15:57.964 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:57.414529+0000 mon.a (mon.0) 3439 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2024-02-19T21:15:57.964 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:57.453026+0000 mon.a (mon.0) 3440 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:15:57.976 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:15:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:15:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:15:58.580 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:15:58.583 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:58.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:15:58.684 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:15:58.712 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:58.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:57.313011+0000 mgr.y (mgr.14550) 596 : cluster [DBG] pgmap v1067: 214 pgs: 11 creating+activating, 203 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:58.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:57.720720+0000 osd.5 (osd.5) 292 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:15:58.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:57.720892+0000 osd.5 (osd.5) 293 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:15:58.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:58.410554+0000 mon.a (mon.0) 3441 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:58.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:58.410616+0000 mon.a (mon.0) 3442 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-02-19T21:15:58.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:58.412458+0000 mon.a (mon.0) 3443 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-124"}]: dispatch 2024-02-19T21:15:58.911 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:15:58.914 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:58.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:15:58.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:57.313011+0000 mgr.y (mgr.14550) 596 : cluster [DBG] pgmap v1067: 214 pgs: 11 creating+activating, 203 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:58.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:57.720720+0000 osd.5 (osd.5) 292 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:15:58.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:57.720892+0000 osd.5 (osd.5) 293 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:15:58.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:58.410554+0000 mon.a (mon.0) 3441 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:58.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:58.410616+0000 mon.a (mon.0) 3442 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-02-19T21:15:58.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:58.412458+0000 mon.a (mon.0) 3443 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-124"}]: dispatch 2024-02-19T21:15:58.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:57.313011+0000 mgr.y (mgr.14550) 596 : cluster [DBG] pgmap v1067: 214 pgs: 11 creating+activating, 203 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:15:58.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:57.720720+0000 osd.5 (osd.5) 292 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:15:58.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:57.720892+0000 osd.5 (osd.5) 293 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:15:58.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:58.410554+0000 mon.a (mon.0) 3441 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:15:58.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:58.410616+0000 mon.a (mon.0) 3442 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2024-02-19T21:15:58.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:58.412458+0000 mon.a (mon.0) 3443 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-124"}]: dispatch 2024-02-19T21:15:59.015 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:15:59.042 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:59.140 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:15:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:59.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:15:59.177 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:15:59.244 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:59.273 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:59.293 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:15:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:59.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:15:59.396 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:15:59.411 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:15:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:15:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: "complaint_time": 30, 2024-02-19T21:15:59.411 INFO:teuthology.orchestra.run.smithi193.stdout: "num_blocked_ops": 0 2024-02-19T21:15:59.411 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:15:59.434 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:15:59.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:59.384710+0000 mon.a (mon.0) 3444 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:59.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:59.389638+0000 mon.a (mon.0) 3445 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-124"}]': finished 2024-02-19T21:15:59.723 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:15:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:59.389696+0000 mon.a (mon.0) 3446 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-02-19T21:15:59.723 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:59.384710+0000 mon.a (mon.0) 3444 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:15:59.723 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:59.389638+0000 mon.a (mon.0) 3445 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-124"}]': finished 2024-02-19T21:15:59.723 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:15:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:59.389696+0000 mon.a (mon.0) 3446 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-02-19T21:15:59.784 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:15:59.786 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:15:59.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:15:59.887 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:15:59.917 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:00.062 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:59.384710+0000 mon.a (mon.0) 3444 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:16:00.062 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:59.389638+0000 mon.a (mon.0) 3445 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-124"}]': finished 2024-02-19T21:16:00.062 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:15:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:59.389696+0000 mon.a (mon.0) 3446 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2024-02-19T21:16:00.070 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:16:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:00.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:16:00.079 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-19T21:16:00.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.521+0000 7fb945cb6700 1 -- 172.21.15.163:0/1999646583 >> v1:172.21.15.163:6790/0 conn(0x7fb9400ff610 legacy=0x7fb940101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.521+0000 7fb945cb6700 1 -- 172.21.15.163:0/1999646583 shutdown_connections 2024-02-19T21:16:00.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.521+0000 7fb945cb6700 1 -- 172.21.15.163:0/1999646583 >> 172.21.15.163:0/1999646583 conn(0x7fb9400f23d0 msgr2=0x7fb9400f47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.521+0000 7fb945cb6700 1 -- 172.21.15.163:0/1999646583 shutdown_connections 2024-02-19T21:16:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.521+0000 7fb945cb6700 1 -- 172.21.15.163:0/1999646583 wait complete. 2024-02-19T21:16:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb945cb6700 1 librados: starting msgr at 2024-02-19T21:16:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb945cb6700 1 librados: starting objecter 2024-02-19T21:16:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb945cb6700 1 Processor -- start 2024-02-19T21:16:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb945cb6700 1 -- start start 2024-02-19T21:16:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb945cb6700 1 librados: setting wanted keys 2024-02-19T21:16:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb945cb6700 1 librados: calling monclient init 2024-02-19T21:16:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb945cb6700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb9400fbda0 con 0x7fb9400ff610 2024-02-19T21:16:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb945cb6700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb9400fbf70 con 0x7fb940103150 2024-02-19T21:16:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb945cb6700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb9400fc160 con 0x7fb9400fc810 2024-02-19T21:16:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb93f7fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fb9400fc810 0x7fb9400fb600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:43048/0 (socket says 172.21.15.163:43048) 2024-02-19T21:16:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb93effd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fb9400ff610 0x7fb940114bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:34962/0 (socket says 172.21.15.163:34962) 2024-02-19T21:16:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb93f7fe700 1 -- 172.21.15.163:0/1350746871 learned_addr learned my addr 172.21.15.163:0/1350746871 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb93cff9700 10 client.?.objecter ms_handle_connect 0x7fb9400fc810 2024-02-19T21:16:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb93cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb93cff9700 10 client.?.objecter ms_handle_connect 0x7fb9400ff610 2024-02-19T21:16:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb93cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb93cff9700 10 client.?.objecter ms_handle_connect 0x7fb940103150 2024-02-19T21:16:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb93cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832995029 0 0) 0x7fb9400fc160 con 0x7fb9400fc810 2024-02-19T21:16:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9280033f0 con 0x7fb9400fc810 2024-02-19T21:16:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027608262 0 0) 0x7fb9400fbda0 con 0x7fb9400ff610 2024-02-19T21:16:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9400fc160 con 0x7fb9400ff610 2024-02-19T21:16:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394461305 0 0) 0x7fb9400fbf70 con 0x7fb940103150 2024-02-19T21:16:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.522+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9400fbda0 con 0x7fb940103150 2024-02-19T21:16:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1386552759 0 0) 0x7fb9400fc160 con 0x7fb9400ff610 2024-02-19T21:16:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9400fbf70 con 0x7fb9400ff610 2024-02-19T21:16:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb930002130 con 0x7fb9400ff610 2024-02-19T21:16:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3647145679 0 0) 0x7fb9400fbda0 con 0x7fb940103150 2024-02-19T21:16:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9400fc160 con 0x7fb940103150 2024-02-19T21:16:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1553782690 0 0) 0x7fb9280033f0 con 0x7fb9400fc810 2024-02-19T21:16:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9400fbda0 con 0x7fb9400fc810 2024-02-19T21:16:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb938001bb0 con 0x7fb940103150 2024-02-19T21:16:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb934001900 con 0x7fb9400fc810 2024-02-19T21:16:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4063692704 0 0) 0x7fb9400fbf70 con 0x7fb9400ff610 2024-02-19T21:16:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 >> v1:172.21.15.163:6790/0 conn(0x7fb9400fc810 legacy=0x7fb9400fb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 >> v1:172.21.15.193:6789/0 conn(0x7fb940103150 legacy=0x7fb9401182e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94011ca30 con 0x7fb9400ff610 2024-02-19T21:16:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb945cb6700 1 -- 172.21.15.163:0/1350746871 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb940119a60 con 0x7fb9400ff610 2024-02-19T21:16:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb945cb6700 10 client.23745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb945cb6700 1 -- 172.21.15.163:0/1350746871 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb940119f50 con 0x7fb9400ff610 2024-02-19T21:16:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb945cb6700 1 librados: init done 2024-02-19T21:16:00.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb930001200 con 0x7fb9400ff610 2024-02-19T21:16:00.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.523+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb930003620 con 0x7fb9400ff610 2024-02-19T21:16:00.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.524+0000 7fb945cb6700 1 -- 172.21.15.163:0/1350746871 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92c004f20 con 0x7fb9400ff610 2024-02-19T21:16:00.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.524+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fb930001880 con 0x7fb9400ff610 2024-02-19T21:16:00.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.525+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(710..710 src has 623..710) v4 ==== 8698+0+0 (unknown 3752819576 0 0) 0x7fb9300790c0 con 0x7fb9400ff610 2024-02-19T21:16:00.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.525+0000 7fb93cff9700 10 client.23745.objecter ms_dispatch 0x7fb940001550 osd_map(710..710 src has 623..710) v4 2024-02-19T21:16:00.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.525+0000 7fb93cff9700 3 client.23745.objecter handle_osd_map got epochs [710,710] > 0 2024-02-19T21:16:00.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.525+0000 7fb93cff9700 3 client.23745.objecter handle_osd_map decoding full epoch 710 2024-02-19T21:16:00.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.525+0000 7fb93cff9700 20 client.23745.objecter dump_active .. 0 homeless 2024-02-19T21:16:00.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.525+0000 7fb93cff9700 10 client.23745.objecter ms_handle_connect 0x7fb928063c80 2024-02-19T21:16:00.167 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.529+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb93004c310 con 0x7fb9400ff610 2024-02-19T21:16:00.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.694+0000 7fb945cb6700 1 -- 172.21.15.163:0/1350746871 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb92c000b90 con 0x7fb928063c80 2024-02-19T21:16:00.168 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:15:59.709+0000 7fb93cff9700 1 -- 172.21.15.163:0/1350746871 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+549656 (unknown 2965378022 0 3504522089) 0x7fb92c000b90 con 0x7fb928063c80 2024-02-19T21:16:00.175 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:16:00.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.165+0000 7fb945cb6700 10 librados: watch_flush enter 2024-02-19T21:16:00.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.165+0000 7fb945cb6700 10 librados: watch_flush exit 2024-02-19T21:16:00.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.165+0000 7fb945cb6700 20 client.23745.objecter shutdown clearing up homeless session... 2024-02-19T21:16:00.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.165+0000 7fb945cb6700 10 client.23745.objecter successfully canceled tick 2024-02-19T21:16:00.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.165+0000 7fb945cb6700 1 -- 172.21.15.163:0/1350746871 >> v1:172.21.15.163:6800/1552514681 conn(0x7fb928063c80 legacy=0x7fb928066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.165+0000 7fb945cb6700 1 -- 172.21.15.163:0/1350746871 >> v1:172.21.15.163:6789/0 conn(0x7fb9400ff610 legacy=0x7fb940114bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.165+0000 7fb945cb6700 1 -- 172.21.15.163:0/1350746871 shutdown_connections 2024-02-19T21:16:00.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.165+0000 7fb945cb6700 1 -- 172.21.15.163:0/1350746871 >> 172.21.15.163:0/1350746871 conn(0x7fb9400f23d0 msgr2=0x7fb940061200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:00.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.166+0000 7fb945cb6700 1 -- 172.21.15.163:0/1350746871 shutdown_connections 2024-02-19T21:16:00.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.166+0000 7fb945cb6700 1 -- 172.21.15.163:0/1350746871 wait complete. 2024-02-19T21:16:00.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.166+0000 7fb945cb6700 1 librados: shutdown 2024-02-19T21:16:00.181 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:16:00.182 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:00.209 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:00.229 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:00.231 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:00.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:00.332 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:00.361 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:00.390 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:16:00.403 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:00.414 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:00.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:00.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.425+0000 7f57a1574700 1 -- 172.21.15.163:0/3850173975 >> v1:172.21.15.163:6790/0 conn(0x7f579c1042b0 legacy=0x7f579c106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.425+0000 7f57a1574700 1 -- 172.21.15.163:0/3850173975 shutdown_connections 2024-02-19T21:16:00.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.425+0000 7f57a1574700 1 -- 172.21.15.163:0/3850173975 >> 172.21.15.163:0/3850173975 conn(0x7f579c0f2470 msgr2=0x7f579c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:00.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.425+0000 7f57a1574700 1 -- 172.21.15.163:0/3850173975 shutdown_connections 2024-02-19T21:16:00.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.425+0000 7f57a1574700 1 -- 172.21.15.163:0/3850173975 wait complete. 2024-02-19T21:16:00.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.425+0000 7f57a1574700 1 librados: starting msgr at 2024-02-19T21:16:00.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.425+0000 7f57a1574700 1 librados: starting objecter 2024-02-19T21:16:00.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.425+0000 7f57a1574700 1 Processor -- start 2024-02-19T21:16:00.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f57a1574700 1 -- start start 2024-02-19T21:16:00.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f57a1574700 1 librados: setting wanted keys 2024-02-19T21:16:00.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f57a1574700 1 librados: calling monclient init 2024-02-19T21:16:00.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f57a1574700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f579c102c80 con 0x7f579c1042b0 2024-02-19T21:16:00.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f57a1574700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f579c102e50 con 0x7f579c0f9530 2024-02-19T21:16:00.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f57a1574700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f579c11bb70 con 0x7f579c0f8980 2024-02-19T21:16:00.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f579affd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f579c0f8980 0x7f579c1024e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:43072/0 (socket says 172.21.15.163:43072) 2024-02-19T21:16:00.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f579affd700 1 -- 172.21.15.163:0/3478903843 learned_addr learned my addr 172.21.15.163:0/3478903843 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:00.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 10 client.?.objecter ms_handle_connect 0x7f579c0f8980 2024-02-19T21:16:00.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:00.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 10 client.?.objecter ms_handle_connect 0x7f579c1042b0 2024-02-19T21:16:00.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:00.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 10 client.?.objecter ms_handle_connect 0x7f579c0f9530 2024-02-19T21:16:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729768863 0 0) 0x7f579c11bb70 con 0x7f579c0f8980 2024-02-19T21:16:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5784003630 con 0x7f579c0f8980 2024-02-19T21:16:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846474841 0 0) 0x7f579c102e50 con 0x7f579c0f9530 2024-02-19T21:16:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f579c11bb70 con 0x7f579c0f9530 2024-02-19T21:16:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2083164945 0 0) 0x7f5784003630 con 0x7f579c0f8980 2024-02-19T21:16:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f579c102e50 con 0x7f579c0f8980 2024-02-19T21:16:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f578c001940 con 0x7f579c0f8980 2024-02-19T21:16:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 974496062 0 0) 0x7f579c11bb70 con 0x7f579c0f9530 2024-02-19T21:16:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5784003630 con 0x7f579c0f9530 2024-02-19T21:16:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.426+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f57900017b0 con 0x7f579c0f9530 2024-02-19T21:16:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.427+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1032051089 0 0) 0x7f579c102e50 con 0x7f579c0f8980 2024-02-19T21:16:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.427+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 >> v1:172.21.15.193:6789/0 conn(0x7f579c0f9530 legacy=0x7f579c114d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.427+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 >> v1:172.21.15.163:6789/0 conn(0x7f579c1042b0 legacy=0x7f579c118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.427+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579c11cd10 con 0x7f579c0f8980 2024-02-19T21:16:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.427+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f579c11bd60 con 0x7f579c0f8980 2024-02-19T21:16:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.427+0000 7f57a1574700 10 client.50474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.427+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f579c11c1d0 con 0x7f579c0f8980 2024-02-19T21:16:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.427+0000 7f57a1574700 1 librados: init done 2024-02-19T21:16:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.427+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f578c002c70 con 0x7f579c0f8980 2024-02-19T21:16:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.427+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f578c005aa0 con 0x7f579c0f8980 2024-02-19T21:16:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.428+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f5788004f00 con 0x7f579c0f8980 2024-02-19T21:16:00.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.428+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f578c005c20 con 0x7f579c0f8980 2024-02-19T21:16:00.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.428+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(711..711 src has 623..711) v4 ==== 8297+0+0 (unknown 875982916 0 0) 0x7f578c078e70 con 0x7f579c0f8980 2024-02-19T21:16:00.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.428+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 osd_map(711..711 src has 623..711) v4 2024-02-19T21:16:00.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.428+0000 7f5783fff700 3 client.50474.objecter handle_osd_map got epochs [711,711] > 0 2024-02-19T21:16:00.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.428+0000 7f5783fff700 3 client.50474.objecter handle_osd_map decoding full epoch 711 2024-02-19T21:16:00.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.429+0000 7f5783fff700 20 client.50474.objecter dump_active .. 0 homeless 2024-02-19T21:16:00.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.429+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v711) v1 ==== 54+0+16 (unknown 247035827 0 2420000192) 0x7f578c07b280 con 0x7f579c0f8980 2024-02-19T21:16:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.429+0000 7f5783fff700 10 client.50474.objecter ms_handle_connect 0x7f5784063c50 2024-02-19T21:16:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.429+0000 7f57a1574700 10 client.50474.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.429+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 1 2024-02-19T21:16:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.429+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f57880051a0 osd=0 3 2024-02-19T21:16:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.429+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f57880051a0 osd=0 3 2024-02-19T21:16:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.429+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f57880051a0 osd=0 3 2024-02-19T21:16:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.429+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 1 2024-02-19T21:16:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.429+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f57880051a0 osd=0 3 2024-02-19T21:16:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.429+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 0 1 2024-02-19T21:16:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.429+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f57880051a0 osd=0 4 2024-02-19T21:16:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.429+0000 7f57a1574700 10 client.50474.objecter _send_command 1 2024-02-19T21:16:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.429+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5788004950 con 0x7f5788004530 2024-02-19T21:16:00.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.429+0000 7f5783fff700 10 client.50474.objecter ms_handle_connect 0x7f5788004530 2024-02-19T21:16:00.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.430+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5788004950 con 0x7f5788004530 2024-02-19T21:16:00.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.430+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:16:00.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.430+0000 7f5783fff700 10 client.50474.objecter _finish_command 1 = system:0 2024-02-19T21:16:00.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.430+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f57880051a0 osd=0 4 2024-02-19T21:16:00.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.430+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 0 1 2024-02-19T21:16:00.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.444+0000 7f57a1574700 10 client.50474.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:00.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.445+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 2 2024-02-19T21:16:00.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.445+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f57880051a0 osd=0 3 2024-02-19T21:16:00.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.445+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f57880051a0 osd=0 3 2024-02-19T21:16:00.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.445+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f57880051a0 osd=0 3 2024-02-19T21:16:00.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.445+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 2 2024-02-19T21:16:00.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.445+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f57880051a0 osd=0 3 2024-02-19T21:16:00.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.445+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 0 2 2024-02-19T21:16:00.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.445+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f57880051a0 osd=0 4 2024-02-19T21:16:00.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.445+0000 7f57a1574700 10 client.50474.objecter _send_command 2 2024-02-19T21:16:00.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.445+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5788004d30 con 0x7f5788004530 2024-02-19T21:16:00.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5788004d30 con 0x7f5788004530 2024-02-19T21:16:00.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:00.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f5783fff700 10 client.50474.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f57880051a0 osd=0 4 2024-02-19T21:16:00.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 0 2 2024-02-19T21:16:00.448 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:16:00.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f57a1574700 10 client.50474.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:00.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 3 2024-02-19T21:16:00.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788004020 osd=1 3 2024-02-19T21:16:00.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788004020 osd=1 3 2024-02-19T21:16:00.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788004020 osd=1 3 2024-02-19T21:16:00.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 3 2024-02-19T21:16:00.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f5788004020 osd=1 3 2024-02-19T21:16:00.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 1 3 2024-02-19T21:16:00.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788004020 osd=1 4 2024-02-19T21:16:00.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f57a1574700 10 client.50474.objecter _send_command 3 2024-02-19T21:16:00.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.446+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5788004190 con 0x7f57880088d0 2024-02-19T21:16:00.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.447+0000 7f5783fff700 10 client.50474.objecter ms_handle_connect 0x7f57880088d0 2024-02-19T21:16:00.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.447+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5788004190 con 0x7f57880088d0 2024-02-19T21:16:00.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.447+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:16:00.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.447+0000 7f5783fff700 10 client.50474.objecter _finish_command 3 = system:0 2024-02-19T21:16:00.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.447+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f5788004020 osd=1 4 2024-02-19T21:16:00.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.447+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 1 3 2024-02-19T21:16:00.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.462+0000 7f57a1574700 10 client.50474.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:00.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.462+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 4 2024-02-19T21:16:00.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.462+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788004020 osd=1 3 2024-02-19T21:16:00.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.462+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788004020 osd=1 3 2024-02-19T21:16:00.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.462+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788004020 osd=1 3 2024-02-19T21:16:00.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.462+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 4 2024-02-19T21:16:00.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.462+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f5788004020 osd=1 3 2024-02-19T21:16:00.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.462+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 1 4 2024-02-19T21:16:00.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.462+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788004020 osd=1 4 2024-02-19T21:16:00.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.462+0000 7f57a1574700 10 client.50474.objecter _send_command 4 2024-02-19T21:16:00.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.462+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f57880017f0 con 0x7f57880088d0 2024-02-19T21:16:00.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f57880017f0 con 0x7f57880088d0 2024-02-19T21:16:00.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:00.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f5783fff700 10 client.50474.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f5788004020 osd=1 4 2024-02-19T21:16:00.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 1 4 2024-02-19T21:16:00.465 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:16:00.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f57a1574700 10 client.50474.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:00.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 5 2024-02-19T21:16:00.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788001ba0 osd=2 3 2024-02-19T21:16:00.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788001ba0 osd=2 3 2024-02-19T21:16:00.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788001ba0 osd=2 3 2024-02-19T21:16:00.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 5 2024-02-19T21:16:00.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f5788001ba0 osd=2 3 2024-02-19T21:16:00.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 2 5 2024-02-19T21:16:00.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788001ba0 osd=2 4 2024-02-19T21:16:00.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f57a1574700 10 client.50474.objecter _send_command 5 2024-02-19T21:16:00.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.463+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5788010df0 con 0x7f5788003b40 2024-02-19T21:16:00.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.464+0000 7f5783fff700 10 client.50474.objecter ms_handle_connect 0x7f5788003b40 2024-02-19T21:16:00.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.465+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5788010df0 con 0x7f5788003b40 2024-02-19T21:16:00.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.465+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:16:00.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.465+0000 7f5783fff700 10 client.50474.objecter _finish_command 5 = system:0 2024-02-19T21:16:00.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.465+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f5788001ba0 osd=2 4 2024-02-19T21:16:00.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.465+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 2 5 2024-02-19T21:16:00.488 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.488+0000 7f57a1574700 10 client.50474.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.488+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 6 2024-02-19T21:16:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.488+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788001ba0 osd=2 3 2024-02-19T21:16:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.488+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788001ba0 osd=2 3 2024-02-19T21:16:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.488+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788001ba0 osd=2 3 2024-02-19T21:16:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.488+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 6 2024-02-19T21:16:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.488+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f5788001ba0 osd=2 3 2024-02-19T21:16:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.488+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 2 6 2024-02-19T21:16:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.488+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788001ba0 osd=2 4 2024-02-19T21:16:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.488+0000 7f57a1574700 10 client.50474.objecter _send_command 6 2024-02-19T21:16:00.489 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.488+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f57880030a0 con 0x7f5788003b40 2024-02-19T21:16:00.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f57880030a0 con 0x7f5788003b40 2024-02-19T21:16:00.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:00.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f5783fff700 10 client.50474.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f5788001ba0 osd=2 4 2024-02-19T21:16:00.491 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 2 6 2024-02-19T21:16:00.492 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:16:00.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f57a1574700 10 client.50474.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:00.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 7 2024-02-19T21:16:00.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f578800cbb0 osd=3 3 2024-02-19T21:16:00.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f578800cbb0 osd=3 3 2024-02-19T21:16:00.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f578800cbb0 osd=3 3 2024-02-19T21:16:00.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 7 2024-02-19T21:16:00.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f578800cbb0 osd=3 3 2024-02-19T21:16:00.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 3 7 2024-02-19T21:16:00.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f578800cbb0 osd=3 4 2024-02-19T21:16:00.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f57a1574700 10 client.50474.objecter _send_command 7 2024-02-19T21:16:00.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f578800d2f0 con 0x7f5788010f90 2024-02-19T21:16:00.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.490+0000 7f5783fff700 10 client.50474.objecter ms_handle_connect 0x7f5788010f90 2024-02-19T21:16:00.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.491+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f578800d2f0 con 0x7f5788010f90 2024-02-19T21:16:00.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.491+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:16:00.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.491+0000 7f5783fff700 10 client.50474.objecter _finish_command 7 = system:0 2024-02-19T21:16:00.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.491+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f578800cbb0 osd=3 4 2024-02-19T21:16:00.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.491+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 3 7 2024-02-19T21:16:00.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.505+0000 7f57a1574700 10 client.50474.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:00.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.505+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 8 2024-02-19T21:16:00.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.505+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f578800cbb0 osd=3 3 2024-02-19T21:16:00.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.505+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f578800cbb0 osd=3 3 2024-02-19T21:16:00.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.505+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f578800cbb0 osd=3 3 2024-02-19T21:16:00.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.505+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 8 2024-02-19T21:16:00.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.505+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f578800cbb0 osd=3 3 2024-02-19T21:16:00.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.505+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 3 8 2024-02-19T21:16:00.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.505+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f578800cbb0 osd=3 4 2024-02-19T21:16:00.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.505+0000 7f57a1574700 10 client.50474.objecter _send_command 8 2024-02-19T21:16:00.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.505+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f57880122b0 con 0x7f5788010f90 2024-02-19T21:16:00.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f57880122b0 con 0x7f5788010f90 2024-02-19T21:16:00.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:00.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f5783fff700 10 client.50474.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f578800cbb0 osd=3 4 2024-02-19T21:16:00.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 3 8 2024-02-19T21:16:00.508 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:16:00.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f57a1574700 10 client.50474.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:00.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 9 2024-02-19T21:16:00.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788011ae0 osd=4 3 2024-02-19T21:16:00.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788011ae0 osd=4 3 2024-02-19T21:16:00.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788011ae0 osd=4 3 2024-02-19T21:16:00.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 9 2024-02-19T21:16:00.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f5788011ae0 osd=4 3 2024-02-19T21:16:00.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 4 9 2024-02-19T21:16:00.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788011ae0 osd=4 4 2024-02-19T21:16:00.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f57a1574700 10 client.50474.objecter _send_command 9 2024-02-19T21:16:00.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f578801a310 con 0x7f5788016790 2024-02-19T21:16:00.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.507+0000 7f5783fff700 10 client.50474.objecter ms_handle_connect 0x7f5788016790 2024-02-19T21:16:00.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.508+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f578801a310 con 0x7f5788016790 2024-02-19T21:16:00.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.508+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:16:00.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.508+0000 7f5783fff700 10 client.50474.objecter _finish_command 9 = system:0 2024-02-19T21:16:00.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.508+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f5788011ae0 osd=4 4 2024-02-19T21:16:00.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.508+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 4 9 2024-02-19T21:16:00.514 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:00.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.523+0000 7f57a1574700 10 client.50474.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:00.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.523+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 10 2024-02-19T21:16:00.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.523+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788011ae0 osd=4 3 2024-02-19T21:16:00.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.523+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788011ae0 osd=4 3 2024-02-19T21:16:00.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.523+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788011ae0 osd=4 3 2024-02-19T21:16:00.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.523+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 10 2024-02-19T21:16:00.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.523+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f5788011ae0 osd=4 3 2024-02-19T21:16:00.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.523+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 4 10 2024-02-19T21:16:00.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.523+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788011ae0 osd=4 4 2024-02-19T21:16:00.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.523+0000 7f57a1574700 10 client.50474.objecter _send_command 10 2024-02-19T21:16:00.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.523+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f578801bac0 con 0x7f5788016790 2024-02-19T21:16:00.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.524+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f578801bac0 con 0x7f5788016790 2024-02-19T21:16:00.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.524+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:00.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.524+0000 7f5783fff700 10 client.50474.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.524+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f5788011ae0 osd=4 4 2024-02-19T21:16:00.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.524+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 4 10 2024-02-19T21:16:00.526 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:16:00.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.524+0000 7f57a1574700 10 client.50474.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:00.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.524+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 11 2024-02-19T21:16:00.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.525+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f578801bac0 osd=5 3 2024-02-19T21:16:00.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.525+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f578801bac0 osd=5 3 2024-02-19T21:16:00.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.525+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f578801bac0 osd=5 3 2024-02-19T21:16:00.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.525+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 11 2024-02-19T21:16:00.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.525+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f578801bac0 osd=5 3 2024-02-19T21:16:00.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.525+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 5 11 2024-02-19T21:16:00.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.525+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f578801bac0 osd=5 4 2024-02-19T21:16:00.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.525+0000 7f57a1574700 10 client.50474.objecter _send_command 11 2024-02-19T21:16:00.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.525+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f578801c6f0 con 0x7f578801c1a0 2024-02-19T21:16:00.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.525+0000 7f5783fff700 10 client.50474.objecter ms_handle_connect 0x7f578801c1a0 2024-02-19T21:16:00.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.526+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f578801c6f0 con 0x7f578801c1a0 2024-02-19T21:16:00.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.526+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:16:00.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.526+0000 7f5783fff700 10 client.50474.objecter _finish_command 11 = system:0 2024-02-19T21:16:00.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.526+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f578801bac0 osd=5 4 2024-02-19T21:16:00.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.526+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 5 11 2024-02-19T21:16:00.541 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:00.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.541+0000 7f57a1574700 10 client.50474.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:00.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.541+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 12 2024-02-19T21:16:00.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.541+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f578801bac0 osd=5 3 2024-02-19T21:16:00.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.541+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f578801bac0 osd=5 3 2024-02-19T21:16:00.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.541+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f578801bac0 osd=5 3 2024-02-19T21:16:00.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.541+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 12 2024-02-19T21:16:00.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.541+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f578801bac0 osd=5 3 2024-02-19T21:16:00.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.541+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 5 12 2024-02-19T21:16:00.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.541+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f578801bac0 osd=5 4 2024-02-19T21:16:00.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.541+0000 7f57a1574700 10 client.50474.objecter _send_command 12 2024-02-19T21:16:00.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.541+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f578801c6f0 con 0x7f578801c1a0 2024-02-19T21:16:00.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.542+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f578801c6f0 con 0x7f578801c1a0 2024-02-19T21:16:00.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.542+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:00.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.542+0000 7f5783fff700 10 client.50474.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.542+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f578801bac0 osd=5 4 2024-02-19T21:16:00.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.542+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 5 12 2024-02-19T21:16:00.544 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:16:00.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.543+0000 7f57a1574700 10 client.50474.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:00.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.543+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 13 2024-02-19T21:16:00.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.543+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f578801c6f0 osd=6 3 2024-02-19T21:16:00.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.543+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f578801c6f0 osd=6 3 2024-02-19T21:16:00.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.543+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f578801c6f0 osd=6 3 2024-02-19T21:16:00.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.543+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 13 2024-02-19T21:16:00.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.543+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f578801c6f0 osd=6 3 2024-02-19T21:16:00.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.543+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 6 13 2024-02-19T21:16:00.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.543+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f578801c6f0 osd=6 4 2024-02-19T21:16:00.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.543+0000 7f57a1574700 10 client.50474.objecter _send_command 13 2024-02-19T21:16:00.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.543+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5788022cc0 con 0x7f578801f1b0 2024-02-19T21:16:00.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.543+0000 7f5783fff700 10 client.50474.objecter ms_handle_connect 0x7f578801f1b0 2024-02-19T21:16:00.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.544+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5788022cc0 con 0x7f578801f1b0 2024-02-19T21:16:00.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.544+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:16:00.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.544+0000 7f5783fff700 10 client.50474.objecter _finish_command 13 = system:0 2024-02-19T21:16:00.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.544+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f578801c6f0 osd=6 4 2024-02-19T21:16:00.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.544+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 6 13 2024-02-19T21:16:00.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.559+0000 7f57a1574700 10 client.50474.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:00.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.559+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 14 2024-02-19T21:16:00.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.559+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f578801c6f0 osd=6 3 2024-02-19T21:16:00.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.559+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f578801c6f0 osd=6 3 2024-02-19T21:16:00.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.559+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f578801c6f0 osd=6 3 2024-02-19T21:16:00.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.559+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 14 2024-02-19T21:16:00.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.559+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f578801c6f0 osd=6 3 2024-02-19T21:16:00.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.559+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 6 14 2024-02-19T21:16:00.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.559+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f578801c6f0 osd=6 4 2024-02-19T21:16:00.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.559+0000 7f57a1574700 10 client.50474.objecter _send_command 14 2024-02-19T21:16:00.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.559+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f5788022cc0 con 0x7f578801f1b0 2024-02-19T21:16:00.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f5788022cc0 con 0x7f578801f1b0 2024-02-19T21:16:00.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:00.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f5783fff700 10 client.50474.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f578801c6f0 osd=6 4 2024-02-19T21:16:00.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 6 14 2024-02-19T21:16:00.563 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:16:00.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f57a1574700 10 client.50474.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:00.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 15 2024-02-19T21:16:00.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788022cc0 osd=7 3 2024-02-19T21:16:00.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788022cc0 osd=7 3 2024-02-19T21:16:00.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788022cc0 osd=7 3 2024-02-19T21:16:00.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 15 2024-02-19T21:16:00.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f5788022cc0 osd=7 3 2024-02-19T21:16:00.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 7 15 2024-02-19T21:16:00.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788022cc0 osd=7 4 2024-02-19T21:16:00.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f57a1574700 10 client.50474.objecter _send_command 15 2024-02-19T21:16:00.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.561+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5788026ff0 con 0x7f5788023340 2024-02-19T21:16:00.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.562+0000 7f5783fff700 10 client.50474.objecter ms_handle_connect 0x7f5788023340 2024-02-19T21:16:00.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.562+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5788026ff0 con 0x7f5788023340 2024-02-19T21:16:00.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.562+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:16:00.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.562+0000 7f5783fff700 10 client.50474.objecter _finish_command 15 = system:0 2024-02-19T21:16:00.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.562+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f5788022cc0 osd=7 4 2024-02-19T21:16:00.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.562+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 7 15 2024-02-19T21:16:00.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.577+0000 7f57a1574700 10 client.50474.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:00.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.577+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign -1 16 2024-02-19T21:16:00.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.577+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788022cc0 osd=7 3 2024-02-19T21:16:00.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.577+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788022cc0 osd=7 3 2024-02-19T21:16:00.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.577+0000 7f57a1574700 20 client.50474.objecter _get_session s=0x7f5788022cc0 osd=7 3 2024-02-19T21:16:00.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.577+0000 7f57a1574700 15 client.50474.objecter _session_command_op_remove -1 16 2024-02-19T21:16:00.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.577+0000 7f57a1574700 20 client.50474.objecter get_session s=0x7f5788022cc0 osd=7 3 2024-02-19T21:16:00.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.577+0000 7f57a1574700 15 client.50474.objecter _session_command_op_assign 7 16 2024-02-19T21:16:00.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.577+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788022cc0 osd=7 4 2024-02-19T21:16:00.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.577+0000 7f57a1574700 10 client.50474.objecter _send_command 16 2024-02-19T21:16:00.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.577+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f57880287a0 con 0x7f5788023340 2024-02-19T21:16:00.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f5783fff700 1 -- 172.21.15.163:0/3478903843 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f57880287a0 con 0x7f5788023340 2024-02-19T21:16:00.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f5783fff700 10 client.50474.objecter ms_dispatch 0x7f579c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:00.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f5783fff700 10 client.50474.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f5783fff700 20 client.50474.objecter put_session s=0x7f5788022cc0 osd=7 4 2024-02-19T21:16:00.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f5783fff700 15 client.50474.objecter _session_command_op_remove 7 16 2024-02-19T21:16:00.580 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:16:00.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 10 librados: watch_flush enter 2024-02-19T21:16:00.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 10 librados: watch_flush exit 2024-02-19T21:16:00.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 10 client.50474.objecter close_session for osd.0 2024-02-19T21:16:00.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 >> v1:172.21.15.163:6801/1559418115 conn(0x7f5788004530 legacy=0x7f5788002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f57880051a0 osd=0 1 2024-02-19T21:16:00.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 10 client.50474.objecter close_session for osd.1 2024-02-19T21:16:00.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 >> v1:172.21.15.163:6805/3125047432 conn(0x7f57880088d0 legacy=0x7f578800acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788004020 osd=1 1 2024-02-19T21:16:00.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 10 client.50474.objecter close_session for osd.2 2024-02-19T21:16:00.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 >> v1:172.21.15.163:6809/1563158487 conn(0x7f5788003b40 legacy=0x7f578800f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788001ba0 osd=2 1 2024-02-19T21:16:00.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 10 client.50474.objecter close_session for osd.3 2024-02-19T21:16:00.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f579b7fe700 1 -- 172.21.15.163:0/3478903843 reap_dead start 2024-02-19T21:16:00.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 >> v1:172.21.15.163:6813/2112279769 conn(0x7f5788010f90 legacy=0x7f57880113d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f578800cbb0 osd=3 1 2024-02-19T21:16:00.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 10 client.50474.objecter close_session for osd.4 2024-02-19T21:16:00.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 >> v1:172.21.15.193:6800/3129716965 conn(0x7f5788016790 legacy=0x7f5788018b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788011ae0 osd=4 1 2024-02-19T21:16:00.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 10 client.50474.objecter close_session for osd.5 2024-02-19T21:16:00.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 >> v1:172.21.15.193:6804/805161991 conn(0x7f578801c1a0 legacy=0x7f578801a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f578801bac0 osd=5 1 2024-02-19T21:16:00.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 10 client.50474.objecter close_session for osd.6 2024-02-19T21:16:00.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 >> v1:172.21.15.193:6808/787604615 conn(0x7f578801f1b0 legacy=0x7f57880215a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f578801c6f0 osd=6 1 2024-02-19T21:16:00.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 10 client.50474.objecter close_session for osd.7 2024-02-19T21:16:00.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 >> v1:172.21.15.193:6812/157927432 conn(0x7f5788023340 legacy=0x7f5788025770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 20 client.50474.objecter put_session s=0x7f5788022cc0 osd=7 1 2024-02-19T21:16:00.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 20 client.50474.objecter shutdown clearing up homeless session... 2024-02-19T21:16:00.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 10 client.50474.objecter successfully canceled tick 2024-02-19T21:16:00.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f579b7fe700 1 -- 172.21.15.163:0/3478903843 reap_dead start 2024-02-19T21:16:00.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5784063c50 legacy=0x7f5784066110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 >> v1:172.21.15.163:6790/0 conn(0x7f579c0f8980 legacy=0x7f579c1024e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:00.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 shutdown_connections 2024-02-19T21:16:00.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 >> 172.21.15.163:0/3478903843 conn(0x7f579c0f2470 msgr2=0x7f579c103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:00.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 shutdown_connections 2024-02-19T21:16:00.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 1 -- 172.21.15.163:0/3478903843 wait complete. 2024-02-19T21:16:00.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:00.579+0000 7f57a1574700 1 librados: shutdown 2024-02-19T21:16:00.584 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.584 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.584 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.584 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.584 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.584 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.584 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.584 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:16:00.593 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:00.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:00.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:00.697 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:00.726 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:00.848 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:00.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:00.885 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:00.927 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:16:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:16:00.927 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:15:59.313428+0000 mgr.y (mgr.14550) 597 : cluster [DBG] pgmap v1070: 214 pgs: 11 creating+activating, 203 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:16:00.927 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:15:59.695788+0000 mgr.y (mgr.14550) 598 : audit [DBG] from='client.23745 v1:172.21.15.163:0/1350746871' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:16:00.927 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:00.397711+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-02-19T21:16:00.927 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:00.429902+0000 mon.c (mon.2) 429 : audit [DBG] from='client.? v1:172.21.15.163:0/3478903843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:00.952 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:00.977 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:15:59.313428+0000 mgr.y (mgr.14550) 597 : cluster [DBG] pgmap v1070: 214 pgs: 11 creating+activating, 203 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:16:00.977 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:15:59.695788+0000 mgr.y (mgr.14550) 598 : audit [DBG] from='client.23745 v1:172.21.15.163:0/1350746871' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:16:00.977 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:00.397711+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-02-19T21:16:00.977 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:00.429902+0000 mon.c (mon.2) 429 : audit [DBG] from='client.? v1:172.21.15.163:0/3478903843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:00.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:15:59.313428+0000 mgr.y (mgr.14550) 597 : cluster [DBG] pgmap v1070: 214 pgs: 11 creating+activating, 203 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:16:00.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:15:59.695788+0000 mgr.y (mgr.14550) 598 : audit [DBG] from='client.23745 v1:172.21.15.163:0/1350746871' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:16:00.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:00.397711+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2024-02-19T21:16:00.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:00.429902+0000 mon.c (mon.2) 429 : audit [DBG] from='client.? v1:172.21.15.163:0/3478903843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:00.979 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:01.451 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":1071,"stamp":"2024-02-19T21:15:59.388094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5874,"num_objects":208,"num_object_clones":1,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":0,"num_read":2008,"num_read_kb":1748,"num_write":751,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":512,"num_bytes_recovered":11554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":902,"ondisk_log_size":902,"up":642,"acting":642,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":591,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2449664,"kb_used_data":18624,"kb_used_omap":0,"kb_used_meta":2430976,"kb_avail":747511552,"statfs":{"total":767960285184,"available":765451829248,"internally_reserved":0,"allocated":19070976,"data_stored":7163553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489319424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.982993"},"pg_stats":[{"pgid":"246.17","version":"0'0","reported_seq":3,"reported_epoch":705,"state":"creating+activating","last_fresh":"2024-02-19T21:15:54.403639+0000","last_change":"2024-02-19T21:15:54.403639+0000","last_active":"2024-02-19T21:15:53.396321+0000","last_peered":"2024-02-19T21:15:53.396321+0000","last_clean":"2024-02-19T21:15:53.396321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:15:54.403639+0000","last_undegraded":"2024-02-19T21:15:54.403639+0000","last_fullsized":"2024-02-19T21:15:54.403639+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":["3","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"246.16","version":"0'0","reported_seq":3,"reported_epoch":705,"state":"creating+activating","last_fresh":"2024-02-19T21:15:54.412164+0000","last_change":"2024-02-19T21:15:54.412164+0000","last_active":"2024-02-19T21:15:53.396321+0000","last_peered":"2024-02-19T21:15:53.396321+0000","last_clean":"2024-02-19T21:15:53.396321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:15:54.412164+0000","last_undegraded":"2024-02-19T21:15:54.412164+0000","last_fullsized":"2024-02-19T21:15:54.412164+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":["1","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"246.15","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.437107+0000","last_change":"2024-02-19T21:15:54.437107+0000","last_active":"2024-02-19T21:15:54.437107+0000","last_peered":"2024-02-19T21:15:54.437107+0000","last_clean":"2024-02-19T21:15:54.437107+0000","last_became_active":"2024-02-19T21:15:54.436828+0000","last_became_peered":"2024-02-19T21:15:54.436828+0000","last_unstale":"2024-02-19T21:15:54.437107+0000","last_undegraded":"2024-02-19T21:15:54.437107+0000","last_fullsized":"2024-02-19T21:15:54.437107+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"246.14","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.415004+0000","last_change":"2024-02-19T21:15:54.415004+0000","last_active":"2024-02-19T21:15:54.415004+0000","last_peered":"2024-02-19T21:15:54.415004+0000","last_clean":"2024-02-19T21:15:54.415004+0000","last_became_active":"2024-02-19T21:15:54.414725+0000","last_became_peered":"2024-02-19T21:15:54.414725+0000","last_unstale":"2024-02-19T21:15:54.415004+0000","last_undegraded":"2024-02-19T21:15:54.415004+0000","last_fullsized":"2024-02-19T21:15:54.415004+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"246.13","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.456415+0000","last_change":"2024-02-19T21:15:54.456415+0000","last_active":"2024-02-19T21:15:54.456415+0000","last_peered":"2024-02-19T21:15:54.456415+0000","last_clean":"2024-02-19T21:15:54.456415+0000","last_became_active":"2024-02-19T21:15:54.456078+0000","last_became_peered":"2024-02-19T21:15:54.456078+0000","last_unstale":"2024-02-19T21:15:54.456415+0000","last_undegraded":"2024-02-19T21:15:54.456415+0000","last_fullsized":"2024-02-19T21:15:54.456415+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"246.12","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.412674+0000","last_change":"2024-02-19T21:15:54.412674+0000","last_active":"2024-02-19T21:15:54.412674+0000","last_peered":"2024-02-19T21:15:54.412674+0000","last_clean":"2024-02-19T21:15:54.412674+0000","last_became_active":"2024-02-19T21:15:54.412437+0000","last_became_peered":"2024-02-19T21:15:54.412437+0000","last_unstale":"2024-02-19T21:15:54.412674+0000","last_undegraded":"2024-02-19T21:15:54.412674+0000","last_fullsized":"2024-02-19T21:15:54.412674+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"246.11","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.436913+0000","last_change":"2024-02-19T21:15:54.436913+0000","last_active":"2024-02-19T21:15:54.436913+0000","last_peered":"2024-02-19T21:15:54.436913+0000","last_clean":"2024-02-19T21:15:54.436913+0000","last_became_active":"2024-02-19T21:15:54.436659+0000","last_became_peered":"2024-02-19T21:15:54.436659+0000","last_unstale":"2024-02-19T21:15:54.436913+0000","last_undegraded":"2024-02-19T21:15:54.436913+0000","last_fullsized":"2024-02-19T21:15:54.436913+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"246.10","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.438671+0000","last_change":"2024-02-19T21:15:54.438671+0000","last_active":"2024-02-19T21:15:54.438671+0000","last_peered":"2024-02-19T21:15:54.438671+0000","last_clean":"2024-02-19T21:15:54.438671+0000","last_became_active":"2024-02-19T21:15:54.438382+0000","last_became_peered":"2024-02-19T21:15:54.438382+0000","last_unstale":"2024-02-19T21:15:54.438671+0000","last_undegraded":"2024-02-19T21:15:54.438671+0000","last_fullsized":"2024-02-19T21:15:54.438671+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"246.f","version":"0'0","reported_seq":3,"reported_epoch":705,"state":"creating+activating","last_fresh":"2024-02-19T21:15:54.402654+0000","last_change":"2024-02-19T21:15:54.402654+0000","last_active":"2024-02-19T21:15:53.396321+0000","last_peered":"2024-02-19T21:15:53.396321+0000","last_clean":"2024-02-19T21:15:53.396321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:15:54.402654+0000","last_undegraded":"2024-02-19T21:15:54.402654+0000","last_fullsized":"2024-02-19T21:15:54.402654+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":["4","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"246.e","version":"0'0","reported_seq":3,"reported_epoch":705,"state":"creating+activating","last_fresh":"2024-02-19T21:15:54.432777+0000","last_change":"2024-02-19T21:15:54.432777+0000","last_active":"2024-02-19T21:15:53.396321+0000","last_peered":"2024-02-19T21:15:53.396321+0000","last_clean":"2024-02-19T21:15:53.396321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:15:54.432777+0000","last_undegraded":"2024-02-19T21:15:54.432777+0000","last_fullsized":"2024-02-19T21:15:54.432777+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":["5","0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"246.d","version":"0'0","reported_seq":3,"reported_epoch":705,"state":"creating+activating","last_fresh":"2024-02-19T21:15:54.410289+0000","last_change":"2024-02-19T21:15:54.410289+0000","last_active":"2024-02-19T21:15:53.396321+0000","last_peered":"2024-02-19T21:15:53.396321+0000","last_clean":"2024-02-19T21:15:53.396321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:15:54.410289+0000","last_undegraded":"2024-02-19T21:15:54.410289+0000","last_fullsized":"2024-02-19T21:15:54.410289+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":["1","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"246.c","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.457928+0000","last_change":"2024-02-19T21:15:54.457928+0000","last_active":"2024-02-19T21:15:54.457928+0000","last_peered":"2024-02-19T21:15:54.457928+0000","last_clean":"2024-02-19T21:15:54.457928+0000","last_became_active":"2024-02-19T21:15:54.457635+0000","last_became_peered":"2024-02-19T21:15:54.457635+0000","last_unstale":"2024-02-19T21:15:54.457928+0000","last_undegraded":"2024-02-19T21:15:54.457928+0000","last_fullsized":"2024-02-19T21:15:54.457928+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"246.b","version":"0'0","reported_seq":3,"reported_epoch":705,"state":"creating+activating","last_fresh":"2024-02-19T21:15:54.410673+0000","last_change":"2024-02-19T21:15:54.410673+0000","last_active":"2024-02-19T21:15:53.396321+0000","last_peered":"2024-02-19T21:15:53.396321+0000","last_clean":"2024-02-19T21:15:53.396321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:15:54.410673+0000","last_undegraded":"2024-02-19T21:15:54.410673+0000","last_fullsized":"2024-02-19T21:15:54.410673+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":["1","0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"246.a","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.458303+0000","last_change":"2024-02-19T21:15:54.458303+0000","last_active":"2024-02-19T21:15:54.458303+0000","last_peered":"2024-02-19T21:15:54.458303+0000","last_clean":"2024-02-19T21:15:54.458303+0000","last_became_active":"2024-02-19T21:15:54.458024+0000","last_became_peered":"2024-02-19T21:15:54.458024+0000","last_unstale":"2024-02-19T21:15:54.458303+0000","last_undegraded":"2024-02-19T21:15:54.458303+0000","last_fullsized":"2024-02-19T21:15:54.458303+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"246.9","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.434273+0000","last_change":"2024-02-19T21:15:54.434273+0000","last_active":"2024-02-19T21:15:54.434273+0000","last_peered":"2024-02-19T21:15:54.434273+0000","last_clean":"2024-02-19T21:15:54.434273+0000","last_became_active":"2024-02-19T21:15:54.433986+0000","last_became_peered":"2024-02-19T21:15:54.433986+0000","last_unstale":"2024-02-19T21:15:54.434273+0000","last_undegraded":"2024-02-19T21:15:54.434273+0000","last_fullsized":"2024-02-19T21:15:54.434273+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"246.8","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.412885+0000","last_change":"2024-02-19T21:15:54.412885+0000","last_active":"2024-02-19T21:15:54.412885+0000","last_peered":"2024-02-19T21:15:54.412885+0000","last_clean":"2024-02-19T21:15:54.412885+0000","last_became_active":"2024-02-19T21:15:54.412591+0000","last_became_peered":"2024-02-19T21:15:54.412591+0000","last_unstale":"2024-02-19T21:15:54.412885+0000","last_undegraded":"2024-02-19T21:15:54.412885+0000","last_fullsized":"2024-02-19T21:15:54.412885+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"246.7","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.459220+0000","last_change":"2024-02-19T21:15:54.459220+0000","last_active":"2024-02-19T21:15:54.459220+0000","last_peered":"2024-02-19T21:15:54.459220+0000","last_clean":"2024-02-19T21:15:54.459220+0000","last_became_active":"2024-02-19T21:15:54.458922+0000","last_became_peered":"2024-02-19T21:15:54.458922+0000","last_unstale":"2024-02-19T21:15:54.459220+0000","last_undegraded":"2024-02-19T21:15:54.459220+0000","last_fullsized":"2024-02-19T21:15:54.459220+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"246.6","version":"0'0","reported_seq":3,"reported_epoch":705,"state":"creating+activating","last_fresh":"2024-02-19T21:15:54.433130+0000","last_change":"2024-02-19T21:15:54.433130+0000","last_active":"2024-02-19T21:15:53.396321+0000","last_peered":"2024-02-19T21:15:53.396321+0000","last_clean":"2024-02-19T21:15:53.396321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:15:54.433130+0000","last_undegraded":"2024-02-19T21:15:54.433130+0000","last_fullsized":"2024-02-19T21:15:54.433130+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":["5","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"246.5","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.457868+0000","last_change":"2024-02-19T21:15:54.457868+0000","last_active":"2024-02-19T21:15:54.457868+0000","last_peered":"2024-02-19T21:15:54.457868+0000","last_clean":"2024-02-19T21:15:54.457868+0000","last_became_active":"2024-02-19T21:15:54.457529+0000","last_became_peered":"2024-02-19T21:15:54.457529+0000","last_unstale":"2024-02-19T21:15:54.457868+0000","last_undegraded":"2024-02-19T21:15:54.457868+0000","last_fullsized":"2024-02-19T21:15:54.457868+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"246.4","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.435316+0000","last_change":"2024-02-19T21:15:54.435316+0000","last_active":"2024-02-19T21:15:54.435316+0000","last_peered":"2024-02-19T21:15:54.435316+0000","last_clean":"2024-02-19T21:15:54.435316+0000","last_became_active":"2024-02-19T21:15:54.435013+0000","last_became_peered":"2024-02-19T21:15:54.435013+0000","last_unstale":"2024-02-19T21:15:54.435316+0000","last_undegraded":"2024-02-19T21:15:54.435316+0000","last_fullsized":"2024-02-19T21:15:54.435316+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"246.3","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.435069+0000","last_change":"2024-02-19T21:15:54.435069+0000","last_active":"2024-02-19T21:15:54.435069+0000","last_peered":"2024-02-19T21:15:54.435069+0000","last_clean":"2024-02-19T21:15:54.435069+0000","last_became_active":"2024-02-19T21:15:54.434754+0000","last_became_peered":"2024-02-19T21:15:54.434754+0000","last_unstale":"2024-02-19T21:15:54.435069+0000","last_undegraded":"2024-02-19T21:15:54.435069+0000","last_fullsized":"2024-02-19T21:15:54.435069+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"246.2","version":"0'0","reported_seq":3,"reported_epoch":705,"state":"creating+activating","last_fresh":"2024-02-19T21:15:54.411226+0000","last_change":"2024-02-19T21:15:54.411226+0000","last_active":"2024-02-19T21:15:53.396321+0000","last_peered":"2024-02-19T21:15:53.396321+0000","last_clean":"2024-02-19T21:15:53.396321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:15:54.411226+0000","last_undegraded":"2024-02-19T21:15:54.411226+0000","last_fullsized":"2024-02-19T21:15:54.411226+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"246.1","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.437517+0000","last_change":"2024-02-19T21:15:54.437517+0000","last_active":"2024-02-19T21:15:54.437517+0000","last_peered":"2024-02-19T21:15:54.437517+0000","last_clean":"2024-02-19T21:15:54.437517+0000","last_became_active":"2024-02-19T21:15:54.437259+0000","last_became_peered":"2024-02-19T21:15:54.437259+0000","last_unstale":"2024-02-19T21:15:54.437517+0000","last_undegraded":"2024-02-19T21:15:54.437517+0000","last_fullsized":"2024-02-19T21:15:54.437517+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"246.0","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.459208+0000","last_change":"2024-02-19T21:15:54.459208+0000","last_active":"2024-02-19T21:15:54.459208+0000","last_peered":"2024-02-19T21:15:54.459208+0000","last_clean":"2024-02-19T21:15:54.459208+0000","last_became_active":"2024-02-19T21:15:54.458790+0000","last_became_peered":"2024-02-19T21:15:54.458790+0000","last_unstale":"2024-02-19T21:15:54.459208+0000","last_undegraded":"2024-02-19T21:15:54.459208+0000","last_fullsized":"2024-02-19T21:15:54.459208+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"246.1b","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.458023+0000","last_change":"2024-02-19T21:15:54.458023+0000","last_active":"2024-02-19T21:15:54.458023+0000","last_peered":"2024-02-19T21:15:54.458023+0000","last_clean":"2024-02-19T21:15:54.458023+0000","last_became_active":"2024-02-19T21:15:54.457578+0000","last_became_peered":"2024-02-19T21:15:54.457578+0000","last_unstale":"2024-02-19T21:15:54.458023+0000","last_undegraded":"2024-02-19T21:15:54.458023+0000","last_fullsized":"2024-02-19T21:15:54.458023+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"234.7","version":"700'10","reported_seq":156,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.412605+0000","last_change":"2024-02-19T21:15:51.952271+0000","last_active":"2024-02-19T21:15:53.412605+0000","last_peered":"2024-02-19T21:15:53.412605+0000","last_clean":"2024-02-19T21:15:53.412605+0000","last_became_active":"2024-02-19T21:14:16.391553+0000","last_became_peered":"2024-02-19T21:14:16.391553+0000","last_unstale":"2024-02-19T21:15:53.412605+0000","last_undegraded":"2024-02-19T21:15:53.412605+0000","last_fullsized":"2024-02-19T21:15:53.412605+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"700'10","last_scrub_stamp":"2024-02-19T21:15:51.952191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:15:51.952191+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"246.1a","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.412839+0000","last_change":"2024-02-19T21:15:54.412839+0000","last_active":"2024-02-19T21:15:54.412839+0000","last_peered":"2024-02-19T21:15:54.412839+0000","last_clean":"2024-02-19T21:15:54.412839+0000","last_became_active":"2024-02-19T21:15:54.412543+0000","last_became_peered":"2024-02-19T21:15:54.412543+0000","last_unstale":"2024-02-19T21:15:54.412839+0000","last_undegraded":"2024-02-19T21:15:54.412839+0000","last_fullsized":"2024-02-19T21:15:54.412839+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"234.6","version":"671'12","reported_seq":123,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.408568+0000","last_change":"2024-02-19T21:15:49.391215+0000","last_active":"2024-02-19T21:15:53.408568+0000","last_peered":"2024-02-19T21:15:53.408568+0000","last_clean":"2024-02-19T21:15:53.408568+0000","last_became_active":"2024-02-19T21:14:16.372197+0000","last_became_peered":"2024-02-19T21:14:16.372197+0000","last_unstale":"2024-02-19T21:15:53.408568+0000","last_undegraded":"2024-02-19T21:15:53.408568+0000","last_fullsized":"2024-02-19T21:15:53.408568+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"246.19","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.459347+0000","last_change":"2024-02-19T21:15:54.459347+0000","last_active":"2024-02-19T21:15:54.459347+0000","last_peered":"2024-02-19T21:15:54.459347+0000","last_clean":"2024-02-19T21:15:54.459347+0000","last_became_active":"2024-02-19T21:15:54.459008+0000","last_became_peered":"2024-02-19T21:15:54.459008+0000","last_unstale":"2024-02-19T21:15:54.459347+0000","last_undegraded":"2024-02-19T21:15:54.459347+0000","last_fullsized":"2024-02-19T21:15:54.459347+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.5","version":"663'11","reported_seq":164,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.413937+0000","last_change":"2024-02-19T21:15:49.389636+0000","last_active":"2024-02-19T21:15:53.413937+0000","last_peered":"2024-02-19T21:15:53.413937+0000","last_clean":"2024-02-19T21:15:53.413937+0000","last_became_active":"2024-02-19T21:14:41.403889+0000","last_became_peered":"2024-02-19T21:14:41.403889+0000","last_unstale":"2024-02-19T21:15:53.413937+0000","last_undegraded":"2024-02-19T21:15:53.413937+0000","last_fullsized":"2024-02-19T21:15:53.413937+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":21,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"246.18","version":"0'0","reported_seq":3,"reported_epoch":705,"state":"creating+activating","last_fresh":"2024-02-19T21:15:54.411778+0000","last_change":"2024-02-19T21:15:54.411778+0000","last_active":"2024-02-19T21:15:53.396321+0000","last_peered":"2024-02-19T21:15:53.396321+0000","last_clean":"2024-02-19T21:15:53.396321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:15:54.411778+0000","last_undegraded":"2024-02-19T21:15:54.411778+0000","last_fullsized":"2024-02-19T21:15:54.411778+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.4","version":"679'1","reported_seq":135,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403742+0000","last_change":"2024-02-19T21:15:49.390166+0000","last_active":"2024-02-19T21:15:53.403742+0000","last_peered":"2024-02-19T21:15:53.403742+0000","last_clean":"2024-02-19T21:15:53.403742+0000","last_became_active":"2024-02-19T21:13:55.356730+0000","last_became_peered":"2024-02-19T21:13:55.356730+0000","last_unstale":"2024-02-19T21:15:53.403742+0000","last_undegraded":"2024-02-19T21:15:53.403742+0000","last_fullsized":"2024-02-19T21:15:53.403742+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"246.1f","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.458112+0000","last_change":"2024-02-19T21:15:54.458112+0000","last_active":"2024-02-19T21:15:54.458112+0000","last_peered":"2024-02-19T21:15:54.458112+0000","last_clean":"2024-02-19T21:15:54.458112+0000","last_became_active":"2024-02-19T21:15:54.457621+0000","last_became_peered":"2024-02-19T21:15:54.457621+0000","last_unstale":"2024-02-19T21:15:54.458112+0000","last_undegraded":"2024-02-19T21:15:54.458112+0000","last_fullsized":"2024-02-19T21:15:54.458112+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"234.3","version":"627'10","reported_seq":164,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404019+0000","last_change":"2024-02-19T21:15:49.404376+0000","last_active":"2024-02-19T21:15:53.404019+0000","last_peered":"2024-02-19T21:15:53.404019+0000","last_clean":"2024-02-19T21:15:53.404019+0000","last_became_active":"2024-02-19T21:14:16.398263+0000","last_became_peered":"2024-02-19T21:14:16.398263+0000","last_unstale":"2024-02-19T21:15:53.404019+0000","last_undegraded":"2024-02-19T21:15:53.404019+0000","last_fullsized":"2024-02-19T21:15:53.404019+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":21,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"246.1e","version":"0'0","reported_seq":3,"reported_epoch":705,"state":"creating+activating","last_fresh":"2024-02-19T21:15:54.429831+0000","last_change":"2024-02-19T21:15:54.429831+0000","last_active":"2024-02-19T21:15:53.396321+0000","last_peered":"2024-02-19T21:15:53.396321+0000","last_clean":"2024-02-19T21:15:53.396321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:15:54.429831+0000","last_undegraded":"2024-02-19T21:15:54.429831+0000","last_fullsized":"2024-02-19T21:15:54.429831+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":["6","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.2","version":"663'5","reported_seq":99,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410277+0000","last_change":"2024-02-19T21:15:49.396890+0000","last_active":"2024-02-19T21:15:53.410277+0000","last_peered":"2024-02-19T21:15:53.410277+0000","last_clean":"2024-02-19T21:15:53.410277+0000","last_became_active":"2024-02-19T21:14:41.390378+0000","last_became_peered":"2024-02-19T21:14:41.390378+0000","last_unstale":"2024-02-19T21:15:53.410277+0000","last_undegraded":"2024-02-19T21:15:53.410277+0000","last_fullsized":"2024-02-19T21:15:53.410277+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"246.1d","version":"0'0","reported_seq":9,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.435348+0000","last_change":"2024-02-19T21:15:54.435348+0000","last_active":"2024-02-19T21:15:54.435348+0000","last_peered":"2024-02-19T21:15:54.435348+0000","last_clean":"2024-02-19T21:15:54.435348+0000","last_became_active":"2024-02-19T21:15:54.435050+0000","last_became_peered":"2024-02-19T21:15:54.435050+0000","last_unstale":"2024-02-19T21:15:54.435348+0000","last_undegraded":"2024-02-19T21:15:54.435348+0000","last_fullsized":"2024-02-19T21:15:54.435348+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":705,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"234.1","version":"627'6","reported_seq":159,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403076+0000","last_change":"2024-02-19T21:15:49.391819+0000","last_active":"2024-02-19T21:15:53.403076+0000","last_peered":"2024-02-19T21:15:53.403076+0000","last_clean":"2024-02-19T21:15:53.403076+0000","last_became_active":"2024-02-19T21:14:41.401375+0000","last_became_peered":"2024-02-19T21:14:41.401375+0000","last_unstale":"2024-02-19T21:15:53.403076+0000","last_undegraded":"2024-02-19T21:15:53.403076+0000","last_fullsized":"2024-02-19T21:15:53.403076+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":9,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"246.1c","version":"0'0","reported_seq":3,"reported_epoch":705,"state":"creating+activating","last_fresh":"2024-02-19T21:15:54.411091+0000","last_change":"2024-02-19T21:15:54.411091+0000","last_active":"2024-02-19T21:15:53.396321+0000","last_peered":"2024-02-19T21:15:53.396321+0000","last_clean":"2024-02-19T21:15:53.396321+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:15:54.411091+0000","last_undegraded":"2024-02-19T21:15:54.411091+0000","last_fullsized":"2024-02-19T21:15:54.411091+0000","mapping_epoch":704,"log_start":"0'0","ondisk_log_start":"0'0","created":704,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:53.396321+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.396321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.0","version":"653'9","reported_seq":161,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404442+0000","last_change":"2024-02-19T21:15:49.384800+0000","last_active":"2024-02-19T21:15:53.404442+0000","last_peered":"2024-02-19T21:15:53.404442+0000","last_clean":"2024-02-19T21:15:53.404442+0000","last_became_active":"2024-02-19T21:14:16.383023+0000","last_became_peered":"2024-02-19T21:14:16.383023+0000","last_unstale":"2024-02-19T21:15:53.404442+0000","last_undegraded":"2024-02-19T21:15:53.404442+0000","last_fullsized":"2024-02-19T21:15:53.404442+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"9","length":"1"}]},{"pgid":"2.29","version":"0'0","reported_seq":101,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.407637+0000","last_change":"2024-02-19T21:15:41.405332+0000","last_active":"2024-02-19T21:15:53.407637+0000","last_peered":"2024-02-19T21:15:53.407637+0000","last_clean":"2024-02-19T21:15:53.407637+0000","last_became_active":"2024-02-19T21:15:41.404959+0000","last_became_peered":"2024-02-19T21:15:41.404959+0000","last_unstale":"2024-02-19T21:15:53.407637+0000","last_undegraded":"2024-02-19T21:15:53.407637+0000","last_fullsized":"2024-02-19T21:15:53.407637+0000","mapping_epoch":691,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":692,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.738600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.738600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":243,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403036+0000","last_change":"2024-02-19T21:14:25.068758+0000","last_active":"2024-02-19T21:15:53.403036+0000","last_peered":"2024-02-19T21:15:53.403036+0000","last_clean":"2024-02-19T21:15:53.403036+0000","last_became_active":"2024-02-19T21:11:50.303790+0000","last_became_peered":"2024-02-19T21:11:50.303790+0000","last_unstale":"2024-02-19T21:15:53.403036+0000","last_undegraded":"2024-02-19T21:15:53.403036+0000","last_fullsized":"2024-02-19T21:15:53.403036+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:25.068700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:25.068700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.25","version":"0'0","reported_seq":752,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405936+0000","last_change":"2024-02-19T21:14:16.380374+0000","last_active":"2024-02-19T21:15:53.405936+0000","last_peered":"2024-02-19T21:15:53.405936+0000","last_clean":"2024-02-19T21:15:53.405936+0000","last_became_active":"2024-02-19T21:14:16.379899+0000","last_became_peered":"2024-02-19T21:14:16.379899+0000","last_unstale":"2024-02-19T21:15:53.405936+0000","last_undegraded":"2024-02-19T21:15:53.405936+0000","last_fullsized":"2024-02-19T21:15:53.405936+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:47.724347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:47.724347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":92,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.408548+0000","last_change":"2024-02-19T21:14:16.372896+0000","last_active":"2024-02-19T21:15:53.408548+0000","last_peered":"2024-02-19T21:15:53.408548+0000","last_clean":"2024-02-19T21:15:53.408548+0000","last_became_active":"2024-02-19T21:14:16.372498+0000","last_became_peered":"2024-02-19T21:14:16.372498+0000","last_unstale":"2024-02-19T21:15:53.408548+0000","last_undegraded":"2024-02-19T21:15:53.408548+0000","last_fullsized":"2024-02-19T21:15:53.408548+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:55.034623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:13:55.034623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":23,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.416797+0000","last_change":"2024-02-19T21:15:41.399384+0000","last_active":"2024-02-19T21:15:53.416797+0000","last_peered":"2024-02-19T21:15:53.416797+0000","last_clean":"2024-02-19T21:15:53.416797+0000","last_became_active":"2024-02-19T21:15:41.398946+0000","last_became_peered":"2024-02-19T21:15:41.398946+0000","last_unstale":"2024-02-19T21:15:53.416797+0000","last_undegraded":"2024-02-19T21:15:53.416797+0000","last_fullsized":"2024-02-19T21:15:53.416797+0000","mapping_epoch":691,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":692,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:36.863734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:36.863734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.26","version":"0'0","reported_seq":251,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410789+0000","last_change":"2024-02-19T21:14:16.376340+0000","last_active":"2024-02-19T21:15:53.410789+0000","last_peered":"2024-02-19T21:15:53.410789+0000","last_clean":"2024-02-19T21:15:53.410789+0000","last_became_active":"2024-02-19T21:14:16.376017+0000","last_became_peered":"2024-02-19T21:14:16.376017+0000","last_unstale":"2024-02-19T21:15:53.410789+0000","last_undegraded":"2024-02-19T21:15:53.410789+0000","last_fullsized":"2024-02-19T21:15:53.410789+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:47.748283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:47.748283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":771,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.413828+0000","last_change":"2024-02-19T21:15:49.242815+0000","last_active":"2024-02-19T21:15:53.413828+0000","last_peered":"2024-02-19T21:15:53.413828+0000","last_clean":"2024-02-19T21:15:53.413828+0000","last_became_active":"2024-02-19T21:14:41.408037+0000","last_became_peered":"2024-02-19T21:14:41.408037+0000","last_unstale":"2024-02-19T21:15:53.413828+0000","last_undegraded":"2024-02-19T21:15:53.413828+0000","last_fullsized":"2024-02-19T21:15:53.413828+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:49.242760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:49.242760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.20","version":"397'2","reported_seq":792,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411207+0000","last_change":"2024-02-19T21:14:58.880699+0000","last_active":"2024-02-19T21:15:53.411207+0000","last_peered":"2024-02-19T21:15:53.411207+0000","last_clean":"2024-02-19T21:15:53.411207+0000","last_became_active":"2024-02-19T21:14:19.482027+0000","last_became_peered":"2024-02-19T21:14:19.482027+0000","last_unstale":"2024-02-19T21:15:53.411207+0000","last_undegraded":"2024-02-19T21:15:53.411207+0000","last_fullsized":"2024-02-19T21:15:53.411207+0000","mapping_epoch":622,"log_start":"58'1","ondisk_log_start":"58'1","created":474,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":6,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:14:58.880622+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:14:58.880622+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":256,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403466+0000","last_change":"2024-02-19T21:14:16.385009+0000","last_active":"2024-02-19T21:15:53.403466+0000","last_peered":"2024-02-19T21:15:53.403466+0000","last_clean":"2024-02-19T21:15:53.403466+0000","last_became_active":"2024-02-19T21:14:16.384406+0000","last_became_peered":"2024-02-19T21:14:16.384406+0000","last_unstale":"2024-02-19T21:15:53.403466+0000","last_undegraded":"2024-02-19T21:15:53.403466+0000","last_fullsized":"2024-02-19T21:15:53.403466+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.081993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.081993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":97,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.408095+0000","last_change":"2024-02-19T21:15:51.187729+0000","last_active":"2024-02-19T21:15:53.408095+0000","last_peered":"2024-02-19T21:15:53.408095+0000","last_clean":"2024-02-19T21:15:53.408095+0000","last_became_active":"2024-02-19T21:14:16.382869+0000","last_became_peered":"2024-02-19T21:14:16.382869+0000","last_unstale":"2024-02-19T21:15:53.408095+0000","last_undegraded":"2024-02-19T21:15:53.408095+0000","last_fullsized":"2024-02-19T21:15:53.408095+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:51.187606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:51.187606+0000","last_clean_scrub_stamp":"2024-02-19T21:15:51.187606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"623'13","reported_seq":906,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.430826+0000","last_change":"2024-02-19T21:14:48.841321+0000","last_active":"2024-02-19T21:15:54.430826+0000","last_peered":"2024-02-19T21:15:54.430826+0000","last_clean":"2024-02-19T21:15:54.430826+0000","last_became_active":"2024-02-19T21:14:16.381241+0000","last_became_peered":"2024-02-19T21:14:16.381241+0000","last_unstale":"2024-02-19T21:15:54.430826+0000","last_undegraded":"2024-02-19T21:15:54.430826+0000","last_fullsized":"2024-02-19T21:15:54.430826+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"623'13","last_scrub_stamp":"2024-02-19T21:14:48.841229+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:14:48.841229+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'28","reported_seq":770,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.402997+0000","last_change":"2024-02-19T21:14:58.119198+0000","last_active":"2024-02-19T21:15:53.402997+0000","last_peered":"2024-02-19T21:15:53.402997+0000","last_clean":"2024-02-19T21:15:53.402997+0000","last_became_active":"2024-02-19T21:14:16.394686+0000","last_became_peered":"2024-02-19T21:14:16.394686+0000","last_unstale":"2024-02-19T21:15:53.402997+0000","last_undegraded":"2024-02-19T21:15:53.402997+0000","last_fullsized":"2024-02-19T21:15:53.402997+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":621,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-19T21:14:58.119090+0000","last_deep_scrub":"473'28","last_deep_scrub_stamp":"2024-02-19T21:11:47.051823+0000","last_clean_scrub_stamp":"2024-02-19T21:14:58.119090+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":704,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.416901+0000","last_change":"2024-02-19T21:14:52.369699+0000","last_active":"2024-02-19T21:15:53.416901+0000","last_peered":"2024-02-19T21:15:53.416901+0000","last_clean":"2024-02-19T21:15:53.416901+0000","last_became_active":"2024-02-19T21:10:40.299425+0000","last_became_peered":"2024-02-19T21:10:40.299425+0000","last_unstale":"2024-02-19T21:15:53.416901+0000","last_undegraded":"2024-02-19T21:15:53.416901+0000","last_fullsized":"2024-02-19T21:15:53.416901+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:52.369624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_clean_scrub_stamp":"2024-02-19T21:14:52.369624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":705,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403380+0000","last_change":"2024-02-19T21:15:04.119671+0000","last_active":"2024-02-19T21:15:53.403380+0000","last_peered":"2024-02-19T21:15:53.403380+0000","last_clean":"2024-02-19T21:15:53.403380+0000","last_became_active":"2024-02-19T21:14:41.388356+0000","last_became_peered":"2024-02-19T21:14:41.388356+0000","last_unstale":"2024-02-19T21:15:53.403380+0000","last_undegraded":"2024-02-19T21:15:53.403380+0000","last_fullsized":"2024-02-19T21:15:53.403380+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:04.119584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:04.119584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":625,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.416709+0000","last_change":"2024-02-19T21:15:38.353068+0000","last_active":"2024-02-19T21:15:53.416709+0000","last_peered":"2024-02-19T21:15:53.416709+0000","last_clean":"2024-02-19T21:15:53.416709+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:15:53.416709+0000","last_undegraded":"2024-02-19T21:15:53.416709+0000","last_fullsized":"2024-02-19T21:15:53.416709+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:38.353000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:38.353000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'16","reported_seq":616,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411161+0000","last_change":"2024-02-19T21:15:03.898768+0000","last_active":"2024-02-19T21:15:53.411161+0000","last_peered":"2024-02-19T21:15:53.411161+0000","last_clean":"2024-02-19T21:15:53.411161+0000","last_became_active":"2024-02-19T21:14:41.391381+0000","last_became_peered":"2024-02-19T21:14:41.391381+0000","last_unstale":"2024-02-19T21:15:53.411161+0000","last_undegraded":"2024-02-19T21:15:53.411161+0000","last_fullsized":"2024-02-19T21:15:53.411161+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'16","last_scrub_stamp":"2024-02-19T21:15:03.898694+0000","last_deep_scrub":"473'16","last_deep_scrub_stamp":"2024-02-19T21:11:46.956424+0000","last_clean_scrub_stamp":"2024-02-19T21:15:03.898694+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":712,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411281+0000","last_change":"2024-02-19T21:15:05.888480+0000","last_active":"2024-02-19T21:15:53.411281+0000","last_peered":"2024-02-19T21:15:53.411281+0000","last_clean":"2024-02-19T21:15:53.411281+0000","last_became_active":"2024-02-19T21:10:41.312116+0000","last_became_peered":"2024-02-19T21:10:41.312116+0000","last_unstale":"2024-02-19T21:15:53.411281+0000","last_undegraded":"2024-02-19T21:15:53.411281+0000","last_fullsized":"2024-02-19T21:15:53.411281+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:05.888410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:05.888410+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":709,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.415474+0000","last_change":"2024-02-19T21:15:32.211604+0000","last_active":"2024-02-19T21:15:53.415474+0000","last_peered":"2024-02-19T21:15:53.415474+0000","last_clean":"2024-02-19T21:15:53.415474+0000","last_became_active":"2024-02-19T21:10:41.305574+0000","last_became_peered":"2024-02-19T21:10:41.305574+0000","last_unstale":"2024-02-19T21:15:53.415474+0000","last_undegraded":"2024-02-19T21:15:53.415474+0000","last_fullsized":"2024-02-19T21:15:53.415474+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:32.211538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:32.211538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":517,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.409844+0000","last_change":"2024-02-19T21:14:46.798482+0000","last_active":"2024-02-19T21:15:53.409844+0000","last_peered":"2024-02-19T21:15:53.409844+0000","last_clean":"2024-02-19T21:15:53.409844+0000","last_became_active":"2024-02-19T21:14:41.390553+0000","last_became_peered":"2024-02-19T21:14:41.390553+0000","last_unstale":"2024-02-19T21:15:53.409844+0000","last_undegraded":"2024-02-19T21:15:53.409844+0000","last_fullsized":"2024-02-19T21:15:53.409844+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:46.798420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:46.798420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":623,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405509+0000","last_change":"2024-02-19T21:14:18.913530+0000","last_active":"2024-02-19T21:15:53.405509+0000","last_peered":"2024-02-19T21:15:53.405509+0000","last_clean":"2024-02-19T21:15:53.405509+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:15:53.405509+0000","last_undegraded":"2024-02-19T21:15:53.405509+0000","last_fullsized":"2024-02-19T21:15:53.405509+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:18.913439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:18.913439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"459'35","reported_seq":117,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.408272+0000","last_change":"2024-02-19T21:14:47.311327+0000","last_active":"2024-02-19T21:15:53.408272+0000","last_peered":"2024-02-19T21:15:53.408272+0000","last_clean":"2024-02-19T21:15:53.408272+0000","last_became_active":"2024-02-19T21:14:16.391564+0000","last_became_peered":"2024-02-19T21:14:16.391564+0000","last_unstale":"2024-02-19T21:15:53.408272+0000","last_undegraded":"2024-02-19T21:15:53.408272+0000","last_fullsized":"2024-02-19T21:15:53.408272+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'35","last_scrub_stamp":"2024-02-19T21:14:47.311262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:47.311262+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":695,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.415176+0000","last_change":"2024-02-19T21:14:41.405996+0000","last_active":"2024-02-19T21:15:53.415176+0000","last_peered":"2024-02-19T21:15:53.415176+0000","last_clean":"2024-02-19T21:15:53.415176+0000","last_became_active":"2024-02-19T21:14:41.405016+0000","last_became_peered":"2024-02-19T21:14:41.405016+0000","last_unstale":"2024-02-19T21:15:53.415176+0000","last_undegraded":"2024-02-19T21:15:53.415176+0000","last_fullsized":"2024-02-19T21:15:53.415176+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:25.501901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:14:25.501901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":688,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.413861+0000","last_change":"2024-02-19T21:14:49.321920+0000","last_active":"2024-02-19T21:15:53.413861+0000","last_peered":"2024-02-19T21:15:53.413861+0000","last_clean":"2024-02-19T21:15:53.413861+0000","last_became_active":"2024-02-19T21:14:41.403469+0000","last_became_peered":"2024-02-19T21:14:41.403469+0000","last_unstale":"2024-02-19T21:15:53.413861+0000","last_undegraded":"2024-02-19T21:15:53.413861+0000","last_fullsized":"2024-02-19T21:15:53.413861+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:49.321813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:49.321813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":679,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411308+0000","last_change":"2024-02-19T21:14:12.566913+0000","last_active":"2024-02-19T21:15:53.411308+0000","last_peered":"2024-02-19T21:15:53.411308+0000","last_clean":"2024-02-19T21:15:53.411308+0000","last_became_active":"2024-02-19T21:08:04.265081+0000","last_became_peered":"2024-02-19T21:08:04.265081+0000","last_unstale":"2024-02-19T21:15:53.411308+0000","last_undegraded":"2024-02-19T21:15:53.411308+0000","last_fullsized":"2024-02-19T21:15:53.411308+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:12.566808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_clean_scrub_stamp":"2024-02-19T21:14:12.566808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":639,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410862+0000","last_change":"2024-02-19T21:14:59.927082+0000","last_active":"2024-02-19T21:15:53.410862+0000","last_peered":"2024-02-19T21:15:53.410862+0000","last_clean":"2024-02-19T21:15:53.410862+0000","last_became_active":"2024-02-19T21:10:41.317284+0000","last_became_peered":"2024-02-19T21:10:41.317284+0000","last_unstale":"2024-02-19T21:15:53.410862+0000","last_undegraded":"2024-02-19T21:15:53.410862+0000","last_fullsized":"2024-02-19T21:15:53.410862+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:59.927001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:59.927001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"662'39","reported_seq":334,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.406247+0000","last_change":"2024-02-19T21:15:06.578639+0000","last_active":"2024-02-19T21:15:53.406247+0000","last_peered":"2024-02-19T21:15:53.406247+0000","last_clean":"2024-02-19T21:15:53.406247+0000","last_became_active":"2024-02-19T21:10:41.313853+0000","last_became_peered":"2024-02-19T21:10:41.313853+0000","last_unstale":"2024-02-19T21:15:53.406247+0000","last_undegraded":"2024-02-19T21:15:53.406247+0000","last_fullsized":"2024-02-19T21:15:53.406247+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"662'39","last_scrub_stamp":"2024-02-19T21:15:06.578559+0000","last_deep_scrub":"662'39","last_deep_scrub_stamp":"2024-02-19T21:15:06.578559+0000","last_clean_scrub_stamp":"2024-02-19T21:15:06.578559+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"655'2","reported_seq":705,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410229+0000","last_change":"2024-02-19T21:15:00.963681+0000","last_active":"2024-02-19T21:15:53.410229+0000","last_peered":"2024-02-19T21:15:53.410229+0000","last_clean":"2024-02-19T21:15:53.410229+0000","last_became_active":"2024-02-19T21:10:41.307517+0000","last_became_peered":"2024-02-19T21:10:41.307517+0000","last_unstale":"2024-02-19T21:15:53.410229+0000","last_undegraded":"2024-02-19T21:15:53.410229+0000","last_fullsized":"2024-02-19T21:15:53.410229+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"655'2","last_scrub_stamp":"2024-02-19T21:15:00.963594+0000","last_deep_scrub":"655'2","last_deep_scrub_stamp":"2024-02-19T21:15:00.963594+0000","last_clean_scrub_stamp":"2024-02-19T21:15:00.963594+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":726,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411517+0000","last_change":"2024-02-19T21:15:51.016985+0000","last_active":"2024-02-19T21:15:53.411517+0000","last_peered":"2024-02-19T21:15:53.411517+0000","last_clean":"2024-02-19T21:15:53.411517+0000","last_became_active":"2024-02-19T21:14:41.391280+0000","last_became_peered":"2024-02-19T21:14:41.391280+0000","last_unstale":"2024-02-19T21:15:53.411517+0000","last_undegraded":"2024-02-19T21:15:53.411517+0000","last_fullsized":"2024-02-19T21:15:53.411517+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:51.016885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:51.016885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":707,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.417420+0000","last_change":"2024-02-19T21:15:42.266670+0000","last_active":"2024-02-19T21:15:53.417420+0000","last_peered":"2024-02-19T21:15:53.417420+0000","last_clean":"2024-02-19T21:15:53.417420+0000","last_became_active":"2024-02-19T21:10:41.307577+0000","last_became_peered":"2024-02-19T21:10:41.307577+0000","last_unstale":"2024-02-19T21:15:53.417420+0000","last_undegraded":"2024-02-19T21:15:53.417420+0000","last_fullsized":"2024-02-19T21:15:53.417420+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:42.266585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:15:42.266585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":287,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404951+0000","last_change":"2024-02-19T21:14:53.606588+0000","last_active":"2024-02-19T21:15:53.404951+0000","last_peered":"2024-02-19T21:15:53.404951+0000","last_clean":"2024-02-19T21:15:53.404951+0000","last_became_active":"2024-02-19T21:10:41.302670+0000","last_became_peered":"2024-02-19T21:10:41.302670+0000","last_unstale":"2024-02-19T21:15:53.404951+0000","last_undegraded":"2024-02-19T21:15:53.404951+0000","last_fullsized":"2024-02-19T21:15:53.404951+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:53.606474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:53.606474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"459'25","reported_seq":351,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404967+0000","last_change":"2024-02-19T21:14:16.417781+0000","last_active":"2024-02-19T21:15:53.404967+0000","last_peered":"2024-02-19T21:15:53.404967+0000","last_clean":"2024-02-19T21:15:53.404967+0000","last_became_active":"2024-02-19T21:14:16.381198+0000","last_became_peered":"2024-02-19T21:14:16.381198+0000","last_unstale":"2024-02-19T21:15:53.404967+0000","last_undegraded":"2024-02-19T21:15:53.404967+0000","last_fullsized":"2024-02-19T21:15:53.404967+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'25","last_scrub_stamp":"2024-02-19T21:12:51.420403+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.420403+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":692,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410696+0000","last_change":"2024-02-19T21:13:38.073731+0000","last_active":"2024-02-19T21:15:53.410696+0000","last_peered":"2024-02-19T21:15:53.410696+0000","last_clean":"2024-02-19T21:15:53.410696+0000","last_became_active":"2024-02-19T21:08:04.268406+0000","last_became_peered":"2024-02-19T21:08:04.268406+0000","last_unstale":"2024-02-19T21:15:53.410696+0000","last_undegraded":"2024-02-19T21:15:53.410696+0000","last_fullsized":"2024-02-19T21:15:53.410696+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:38.073668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:38.073668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":857,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.403988+0000","last_change":"2024-02-19T21:14:58.303172+0000","last_active":"2024-02-19T21:15:54.403988+0000","last_peered":"2024-02-19T21:15:54.403988+0000","last_clean":"2024-02-19T21:15:54.403988+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:15:54.403988+0000","last_undegraded":"2024-02-19T21:15:54.403988+0000","last_fullsized":"2024-02-19T21:15:54.403988+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:14:58.303098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:58.303098+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":282,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.406250+0000","last_change":"2024-02-19T21:12:21.380805+0000","last_active":"2024-02-19T21:15:53.406250+0000","last_peered":"2024-02-19T21:15:53.406250+0000","last_clean":"2024-02-19T21:15:53.406250+0000","last_became_active":"2024-02-19T21:10:41.298893+0000","last_became_peered":"2024-02-19T21:10:41.298893+0000","last_unstale":"2024-02-19T21:15:53.406250+0000","last_undegraded":"2024-02-19T21:15:53.406250+0000","last_fullsized":"2024-02-19T21:15:53.406250+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:21.380734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:12:21.380734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":446,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403233+0000","last_change":"2024-02-19T21:15:49.195649+0000","last_active":"2024-02-19T21:15:53.403233+0000","last_peered":"2024-02-19T21:15:53.403233+0000","last_clean":"2024-02-19T21:15:53.403233+0000","last_became_active":"2024-02-19T21:14:41.382117+0000","last_became_peered":"2024-02-19T21:14:41.382117+0000","last_unstale":"2024-02-19T21:15:53.403233+0000","last_undegraded":"2024-02-19T21:15:53.403233+0000","last_fullsized":"2024-02-19T21:15:53.403233+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:49.195586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:49.195586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"459'27","reported_seq":329,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.417211+0000","last_change":"2024-02-19T21:14:22.521813+0000","last_active":"2024-02-19T21:15:53.417211+0000","last_peered":"2024-02-19T21:15:53.417211+0000","last_clean":"2024-02-19T21:15:53.417211+0000","last_became_active":"2024-02-19T21:10:41.307576+0000","last_became_peered":"2024-02-19T21:10:41.307576+0000","last_unstale":"2024-02-19T21:15:53.417211+0000","last_undegraded":"2024-02-19T21:15:53.417211+0000","last_fullsized":"2024-02-19T21:15:53.417211+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'27","last_scrub_stamp":"2024-02-19T21:14:22.521746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:22.521746+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":734,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.409183+0000","last_change":"2024-02-19T21:14:55.003998+0000","last_active":"2024-02-19T21:15:53.409183+0000","last_peered":"2024-02-19T21:15:53.409183+0000","last_clean":"2024-02-19T21:15:53.409183+0000","last_became_active":"2024-02-19T21:12:40.369640+0000","last_became_peered":"2024-02-19T21:12:40.369640+0000","last_unstale":"2024-02-19T21:15:53.409183+0000","last_undegraded":"2024-02-19T21:15:53.409183+0000","last_fullsized":"2024-02-19T21:15:53.409183+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:55.003931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:55.003931+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"640'12","reported_seq":376,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.404209+0000","last_change":"2024-02-19T21:14:41.540201+0000","last_active":"2024-02-19T21:15:54.404209+0000","last_peered":"2024-02-19T21:15:54.404209+0000","last_clean":"2024-02-19T21:15:54.404209+0000","last_became_active":"2024-02-19T21:14:41.415828+0000","last_became_peered":"2024-02-19T21:14:41.415828+0000","last_unstale":"2024-02-19T21:15:54.404209+0000","last_undegraded":"2024-02-19T21:15:54.404209+0000","last_fullsized":"2024-02-19T21:15:54.404209+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"483'11","last_scrub_stamp":"2024-02-19T21:12:56.450044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:56.450044+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":692,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405067+0000","last_change":"2024-02-19T21:15:37.011347+0000","last_active":"2024-02-19T21:15:53.405067+0000","last_peered":"2024-02-19T21:15:53.405067+0000","last_clean":"2024-02-19T21:15:53.405067+0000","last_became_active":"2024-02-19T21:14:41.404281+0000","last_became_peered":"2024-02-19T21:14:41.404281+0000","last_unstale":"2024-02-19T21:15:53.405067+0000","last_undegraded":"2024-02-19T21:15:53.405067+0000","last_fullsized":"2024-02-19T21:15:53.405067+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:37.011259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:37.011259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":641,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404193+0000","last_change":"2024-02-19T21:14:50.038933+0000","last_active":"2024-02-19T21:15:53.404193+0000","last_peered":"2024-02-19T21:15:53.404193+0000","last_clean":"2024-02-19T21:15:53.404193+0000","last_became_active":"2024-02-19T21:14:16.397268+0000","last_became_peered":"2024-02-19T21:14:16.397268+0000","last_unstale":"2024-02-19T21:15:53.404193+0000","last_undegraded":"2024-02-19T21:15:53.404193+0000","last_fullsized":"2024-02-19T21:15:53.404193+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:50.038866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:50.038866+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"459'19","reported_seq":355,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.406193+0000","last_change":"2024-02-19T21:15:03.572770+0000","last_active":"2024-02-19T21:15:53.406193+0000","last_peered":"2024-02-19T21:15:53.406193+0000","last_clean":"2024-02-19T21:15:53.406193+0000","last_became_active":"2024-02-19T21:14:16.382074+0000","last_became_peered":"2024-02-19T21:14:16.382074+0000","last_unstale":"2024-02-19T21:15:53.406193+0000","last_undegraded":"2024-02-19T21:15:53.406193+0000","last_fullsized":"2024-02-19T21:15:53.406193+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:15:03.572687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:03.572687+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":731,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411403+0000","last_change":"2024-02-19T21:15:35.944834+0000","last_active":"2024-02-19T21:15:53.411403+0000","last_peered":"2024-02-19T21:15:53.411403+0000","last_clean":"2024-02-19T21:15:53.411403+0000","last_became_active":"2024-02-19T21:14:41.382375+0000","last_became_peered":"2024-02-19T21:14:41.382375+0000","last_unstale":"2024-02-19T21:15:53.411403+0000","last_undegraded":"2024-02-19T21:15:53.411403+0000","last_fullsized":"2024-02-19T21:15:53.411403+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:35.944741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:35.944741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":706,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405691+0000","last_change":"2024-02-19T21:14:20.914502+0000","last_active":"2024-02-19T21:15:53.405691+0000","last_peered":"2024-02-19T21:15:53.405691+0000","last_clean":"2024-02-19T21:15:53.405691+0000","last_became_active":"2024-02-19T21:14:16.393359+0000","last_became_peered":"2024-02-19T21:14:16.393359+0000","last_unstale":"2024-02-19T21:15:53.405691+0000","last_undegraded":"2024-02-19T21:15:53.405691+0000","last_fullsized":"2024-02-19T21:15:53.405691+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:20.914421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:20.914421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":439,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404141+0000","last_change":"2024-02-19T21:14:16.385816+0000","last_active":"2024-02-19T21:15:53.404141+0000","last_peered":"2024-02-19T21:15:53.404141+0000","last_clean":"2024-02-19T21:15:53.404141+0000","last_became_active":"2024-02-19T21:14:16.385474+0000","last_became_peered":"2024-02-19T21:14:16.385474+0000","last_unstale":"2024-02-19T21:15:53.404141+0000","last_undegraded":"2024-02-19T21:15:53.404141+0000","last_fullsized":"2024-02-19T21:15:53.404141+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:51.217370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:51.217370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":439,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403509+0000","last_change":"2024-02-19T21:15:38.121570+0000","last_active":"2024-02-19T21:15:53.403509+0000","last_peered":"2024-02-19T21:15:53.403509+0000","last_clean":"2024-02-19T21:15:53.403509+0000","last_became_active":"2024-02-19T21:14:16.383877+0000","last_became_peered":"2024-02-19T21:14:16.383877+0000","last_unstale":"2024-02-19T21:15:53.403509+0000","last_undegraded":"2024-02-19T21:15:53.403509+0000","last_fullsized":"2024-02-19T21:15:53.403509+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:38.121453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:38.121453+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"434'12","reported_seq":549,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.403295+0000","last_change":"2024-02-19T21:13:47.189879+0000","last_active":"2024-02-19T21:15:54.403295+0000","last_peered":"2024-02-19T21:15:54.403295+0000","last_clean":"2024-02-19T21:15:54.403295+0000","last_became_active":"2024-02-19T21:10:41.297897+0000","last_became_peered":"2024-02-19T21:10:41.297897+0000","last_unstale":"2024-02-19T21:15:54.403295+0000","last_undegraded":"2024-02-19T21:15:54.403295+0000","last_fullsized":"2024-02-19T21:15:54.403295+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'12","last_scrub_stamp":"2024-02-19T21:13:47.189812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:47.189812+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"459'28","reported_seq":374,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405017+0000","last_change":"2024-02-19T21:14:16.413467+0000","last_active":"2024-02-19T21:15:53.405017+0000","last_peered":"2024-02-19T21:15:53.405017+0000","last_clean":"2024-02-19T21:15:53.405017+0000","last_became_active":"2024-02-19T21:14:16.380694+0000","last_became_peered":"2024-02-19T21:14:16.380694+0000","last_unstale":"2024-02-19T21:15:53.405017+0000","last_undegraded":"2024-02-19T21:15:53.405017+0000","last_fullsized":"2024-02-19T21:15:53.405017+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'28","last_scrub_stamp":"2024-02-19T21:13:46.758735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:13:46.758735+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":97,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.409450+0000","last_change":"2024-02-19T21:15:50.164521+0000","last_active":"2024-02-19T21:15:53.409450+0000","last_peered":"2024-02-19T21:15:53.409450+0000","last_clean":"2024-02-19T21:15:53.409450+0000","last_became_active":"2024-02-19T21:14:16.373120+0000","last_became_peered":"2024-02-19T21:14:16.373120+0000","last_unstale":"2024-02-19T21:15:53.409450+0000","last_undegraded":"2024-02-19T21:15:53.409450+0000","last_fullsized":"2024-02-19T21:15:53.409450+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:50.164434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:50.164434+0000","last_clean_scrub_stamp":"2024-02-19T21:15:50.164434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":683,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410644+0000","last_change":"2024-02-19T21:14:16.375816+0000","last_active":"2024-02-19T21:15:53.410644+0000","last_peered":"2024-02-19T21:15:53.410644+0000","last_clean":"2024-02-19T21:15:53.410644+0000","last_became_active":"2024-02-19T21:14:16.375486+0000","last_became_peered":"2024-02-19T21:14:16.375486+0000","last_unstale":"2024-02-19T21:15:53.410644+0000","last_undegraded":"2024-02-19T21:15:53.410644+0000","last_fullsized":"2024-02-19T21:15:53.410644+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:49.749137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:13:49.749137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":520,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411469+0000","last_change":"2024-02-19T21:15:49.983230+0000","last_active":"2024-02-19T21:15:53.411469+0000","last_peered":"2024-02-19T21:15:53.411469+0000","last_clean":"2024-02-19T21:15:53.411469+0000","last_became_active":"2024-02-19T21:14:41.391010+0000","last_became_peered":"2024-02-19T21:14:41.391010+0000","last_unstale":"2024-02-19T21:15:53.411469+0000","last_undegraded":"2024-02-19T21:15:53.411469+0000","last_fullsized":"2024-02-19T21:15:53.411469+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:49.983125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:11.004006+0000","last_clean_scrub_stamp":"2024-02-19T21:15:49.983125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":287,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404794+0000","last_change":"2024-02-19T21:14:54.655007+0000","last_active":"2024-02-19T21:15:53.404794+0000","last_peered":"2024-02-19T21:15:53.404794+0000","last_clean":"2024-02-19T21:15:53.404794+0000","last_became_active":"2024-02-19T21:10:41.304695+0000","last_became_peered":"2024-02-19T21:10:41.304695+0000","last_unstale":"2024-02-19T21:15:53.404794+0000","last_undegraded":"2024-02-19T21:15:53.404794+0000","last_fullsized":"2024-02-19T21:15:53.404794+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:54.654916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:14:54.654916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"459'30","reported_seq":511,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403417+0000","last_change":"2024-02-19T21:14:52.068847+0000","last_active":"2024-02-19T21:15:53.403417+0000","last_peered":"2024-02-19T21:15:53.403417+0000","last_clean":"2024-02-19T21:15:53.403417+0000","last_became_active":"2024-02-19T21:14:41.403514+0000","last_became_peered":"2024-02-19T21:14:41.403514+0000","last_unstale":"2024-02-19T21:15:53.403417+0000","last_undegraded":"2024-02-19T21:15:53.403417+0000","last_fullsized":"2024-02-19T21:15:53.403417+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'30","last_scrub_stamp":"2024-02-19T21:14:52.068777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:52.068777+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":738,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410893+0000","last_change":"2024-02-19T21:15:48.962320+0000","last_active":"2024-02-19T21:15:53.410893+0000","last_peered":"2024-02-19T21:15:53.410893+0000","last_clean":"2024-02-19T21:15:53.410893+0000","last_became_active":"2024-02-19T21:14:16.379640+0000","last_became_peered":"2024-02-19T21:14:16.379640+0000","last_unstale":"2024-02-19T21:15:53.410893+0000","last_undegraded":"2024-02-19T21:15:53.410893+0000","last_fullsized":"2024-02-19T21:15:53.410893+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:48.962240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:48.962240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":287,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405696+0000","last_change":"2024-02-19T21:15:44.842959+0000","last_active":"2024-02-19T21:15:53.405696+0000","last_peered":"2024-02-19T21:15:53.405696+0000","last_clean":"2024-02-19T21:15:53.405696+0000","last_became_active":"2024-02-19T21:10:41.295301+0000","last_became_peered":"2024-02-19T21:10:41.295301+0000","last_unstale":"2024-02-19T21:15:53.405696+0000","last_undegraded":"2024-02-19T21:15:53.405696+0000","last_fullsized":"2024-02-19T21:15:53.405696+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:44.842888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:15:44.842888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":457,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404335+0000","last_change":"2024-02-19T21:15:39.146832+0000","last_active":"2024-02-19T21:15:53.404335+0000","last_peered":"2024-02-19T21:15:53.404335+0000","last_clean":"2024-02-19T21:15:53.404335+0000","last_became_active":"2024-02-19T21:14:41.398917+0000","last_became_peered":"2024-02-19T21:14:41.398917+0000","last_unstale":"2024-02-19T21:15:53.404335+0000","last_undegraded":"2024-02-19T21:15:53.404335+0000","last_fullsized":"2024-02-19T21:15:53.404335+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:39.146761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:39.146761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":106,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.408177+0000","last_change":"2024-02-19T21:15:49.141891+0000","last_active":"2024-02-19T21:15:53.408177+0000","last_peered":"2024-02-19T21:15:53.408177+0000","last_clean":"2024-02-19T21:15:53.408177+0000","last_became_active":"2024-02-19T21:14:41.395152+0000","last_became_peered":"2024-02-19T21:14:41.395152+0000","last_unstale":"2024-02-19T21:15:53.408177+0000","last_undegraded":"2024-02-19T21:15:53.408177+0000","last_fullsized":"2024-02-19T21:15:53.408177+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:49.141780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:49.141780+0000","last_clean_scrub_stamp":"2024-02-19T21:15:49.141780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"459'29","reported_seq":623,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404504+0000","last_change":"2024-02-19T21:15:42.003524+0000","last_active":"2024-02-19T21:15:53.404504+0000","last_peered":"2024-02-19T21:15:53.404504+0000","last_clean":"2024-02-19T21:15:53.404504+0000","last_became_active":"2024-02-19T21:14:41.404443+0000","last_became_peered":"2024-02-19T21:14:41.404443+0000","last_unstale":"2024-02-19T21:15:53.404504+0000","last_undegraded":"2024-02-19T21:15:53.404504+0000","last_fullsized":"2024-02-19T21:15:53.404504+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'29","last_scrub_stamp":"2024-02-19T21:15:42.003405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:42.003405+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":751,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410965+0000","last_change":"2024-02-19T21:15:41.880159+0000","last_active":"2024-02-19T21:15:53.410965+0000","last_peered":"2024-02-19T21:15:53.410965+0000","last_clean":"2024-02-19T21:15:53.410965+0000","last_became_active":"2024-02-19T21:14:41.415341+0000","last_became_peered":"2024-02-19T21:14:41.415341+0000","last_unstale":"2024-02-19T21:15:53.410965+0000","last_undegraded":"2024-02-19T21:15:53.410965+0000","last_fullsized":"2024-02-19T21:15:53.410965+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:41.880077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:41.880077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":683,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410020+0000","last_change":"2024-02-19T21:15:36.830480+0000","last_active":"2024-02-19T21:15:53.410020+0000","last_peered":"2024-02-19T21:15:53.410020+0000","last_clean":"2024-02-19T21:15:53.410020+0000","last_became_active":"2024-02-19T21:14:16.375008+0000","last_became_peered":"2024-02-19T21:14:16.375008+0000","last_unstale":"2024-02-19T21:15:53.410020+0000","last_undegraded":"2024-02-19T21:15:53.410020+0000","last_fullsized":"2024-02-19T21:15:53.410020+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:36.830383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:15:36.830383+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":638,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.415199+0000","last_change":"2024-02-19T21:14:41.404931+0000","last_active":"2024-02-19T21:15:53.415199+0000","last_peered":"2024-02-19T21:15:53.415199+0000","last_clean":"2024-02-19T21:15:53.415199+0000","last_became_active":"2024-02-19T21:14:41.403823+0000","last_became_peered":"2024-02-19T21:14:41.403823+0000","last_unstale":"2024-02-19T21:15:53.415199+0000","last_undegraded":"2024-02-19T21:15:53.415199+0000","last_fullsized":"2024-02-19T21:15:53.415199+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:21.483828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_clean_scrub_stamp":"2024-02-19T21:14:21.483828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":688,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405430+0000","last_change":"2024-02-19T21:14:57.083065+0000","last_active":"2024-02-19T21:15:53.405430+0000","last_peered":"2024-02-19T21:15:53.405430+0000","last_clean":"2024-02-19T21:15:53.405430+0000","last_became_active":"2024-02-19T21:14:41.404337+0000","last_became_peered":"2024-02-19T21:14:41.404337+0000","last_unstale":"2024-02-19T21:15:53.405430+0000","last_undegraded":"2024-02-19T21:15:53.405430+0000","last_fullsized":"2024-02-19T21:15:53.405430+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:57.082987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:57.082987+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"459'24","reported_seq":596,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411200+0000","last_change":"2024-02-19T21:15:45.957885+0000","last_active":"2024-02-19T21:15:53.411200+0000","last_peered":"2024-02-19T21:15:53.411200+0000","last_clean":"2024-02-19T21:15:53.411200+0000","last_became_active":"2024-02-19T21:14:41.415655+0000","last_became_peered":"2024-02-19T21:14:41.415655+0000","last_unstale":"2024-02-19T21:15:53.411200+0000","last_undegraded":"2024-02-19T21:15:53.411200+0000","last_fullsized":"2024-02-19T21:15:53.411200+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'24","last_scrub_stamp":"2024-02-19T21:15:45.957817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:45.957817+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"397'2","reported_seq":762,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411286+0000","last_change":"2024-02-19T21:15:50.915397+0000","last_active":"2024-02-19T21:15:53.411286+0000","last_peered":"2024-02-19T21:15:53.411286+0000","last_clean":"2024-02-19T21:15:53.411286+0000","last_became_active":"2024-02-19T21:14:44.370071+0000","last_became_peered":"2024-02-19T21:14:44.370071+0000","last_unstale":"2024-02-19T21:15:53.411286+0000","last_undegraded":"2024-02-19T21:15:53.411286+0000","last_fullsized":"2024-02-19T21:15:53.411286+0000","mapping_epoch":639,"log_start":"58'1","ondisk_log_start":"58'1","created":57,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:15:50.915331+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:15:50.915331+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":305,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405064+0000","last_change":"2024-02-19T21:14:41.416360+0000","last_active":"2024-02-19T21:15:53.405064+0000","last_peered":"2024-02-19T21:15:53.405064+0000","last_clean":"2024-02-19T21:15:53.405064+0000","last_became_active":"2024-02-19T21:14:41.416024+0000","last_became_peered":"2024-02-19T21:14:41.416024+0000","last_unstale":"2024-02-19T21:15:53.405064+0000","last_undegraded":"2024-02-19T21:15:53.405064+0000","last_fullsized":"2024-02-19T21:15:53.405064+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.643360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.643360+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":685,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411574+0000","last_change":"2024-02-19T21:14:20.999565+0000","last_active":"2024-02-19T21:15:53.411574+0000","last_peered":"2024-02-19T21:15:53.411574+0000","last_clean":"2024-02-19T21:15:53.411574+0000","last_became_active":"2024-02-19T21:08:04.282243+0000","last_became_peered":"2024-02-19T21:08:04.282243+0000","last_unstale":"2024-02-19T21:15:53.411574+0000","last_undegraded":"2024-02-19T21:15:53.411574+0000","last_fullsized":"2024-02-19T21:15:53.411574+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:20.999473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:20.999473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":106,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.408357+0000","last_change":"2024-02-19T21:15:47.088344+0000","last_active":"2024-02-19T21:15:53.408357+0000","last_peered":"2024-02-19T21:15:53.408357+0000","last_clean":"2024-02-19T21:15:53.408357+0000","last_became_active":"2024-02-19T21:14:41.395290+0000","last_became_peered":"2024-02-19T21:14:41.395290+0000","last_unstale":"2024-02-19T21:15:53.408357+0000","last_undegraded":"2024-02-19T21:15:53.408357+0000","last_fullsized":"2024-02-19T21:15:53.408357+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:47.088245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:47.088245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":668,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410776+0000","last_change":"2024-02-19T21:15:32.976824+0000","last_active":"2024-02-19T21:15:53.410776+0000","last_peered":"2024-02-19T21:15:53.410776+0000","last_clean":"2024-02-19T21:15:53.410776+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:15:53.410776+0000","last_undegraded":"2024-02-19T21:15:53.410776+0000","last_fullsized":"2024-02-19T21:15:53.410776+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:32.976749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:32.976749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"459'26","reported_seq":113,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.408530+0000","last_change":"2024-02-19T21:15:39.237327+0000","last_active":"2024-02-19T21:15:53.408530+0000","last_peered":"2024-02-19T21:15:53.408530+0000","last_clean":"2024-02-19T21:15:53.408530+0000","last_became_active":"2024-02-19T21:14:16.390782+0000","last_became_peered":"2024-02-19T21:14:16.390782+0000","last_unstale":"2024-02-19T21:15:53.408530+0000","last_undegraded":"2024-02-19T21:15:53.408530+0000","last_fullsized":"2024-02-19T21:15:53.408530+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'26","last_scrub_stamp":"2024-02-19T21:15:39.237243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:39.237243+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":712,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411336+0000","last_change":"2024-02-19T21:14:41.391190+0000","last_active":"2024-02-19T21:15:53.411336+0000","last_peered":"2024-02-19T21:15:53.411336+0000","last_clean":"2024-02-19T21:15:53.411336+0000","last_became_active":"2024-02-19T21:14:41.390836+0000","last_became_peered":"2024-02-19T21:14:41.390836+0000","last_unstale":"2024-02-19T21:15:53.411336+0000","last_undegraded":"2024-02-19T21:15:53.411336+0000","last_fullsized":"2024-02-19T21:15:53.411336+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:50.724362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:13:50.724362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":711,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404416+0000","last_change":"2024-02-19T21:14:14.957777+0000","last_active":"2024-02-19T21:15:53.404416+0000","last_peered":"2024-02-19T21:15:53.404416+0000","last_clean":"2024-02-19T21:15:53.404416+0000","last_became_active":"2024-02-19T21:11:48.359271+0000","last_became_peered":"2024-02-19T21:11:48.359271+0000","last_unstale":"2024-02-19T21:15:53.404416+0000","last_undegraded":"2024-02-19T21:15:53.404416+0000","last_fullsized":"2024-02-19T21:15:53.404416+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:14.957719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:14.957719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":439,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403563+0000","last_change":"2024-02-19T21:14:16.384177+0000","last_active":"2024-02-19T21:15:53.403563+0000","last_peered":"2024-02-19T21:15:53.403563+0000","last_clean":"2024-02-19T21:15:53.403563+0000","last_became_active":"2024-02-19T21:14:16.383858+0000","last_became_peered":"2024-02-19T21:14:16.383858+0000","last_unstale":"2024-02-19T21:15:53.403563+0000","last_undegraded":"2024-02-19T21:15:53.403563+0000","last_fullsized":"2024-02-19T21:15:53.403563+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:55.248259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.248259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":704,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405746+0000","last_change":"2024-02-19T21:13:53.919119+0000","last_active":"2024-02-19T21:15:53.405746+0000","last_peered":"2024-02-19T21:15:53.405746+0000","last_clean":"2024-02-19T21:15:53.405746+0000","last_became_active":"2024-02-19T21:10:41.315397+0000","last_became_peered":"2024-02-19T21:10:41.315397+0000","last_unstale":"2024-02-19T21:15:53.405746+0000","last_undegraded":"2024-02-19T21:15:53.405746+0000","last_fullsized":"2024-02-19T21:15:53.405746+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:53.919024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:53.919024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":641,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.402987+0000","last_change":"2024-02-19T21:14:41.404518+0000","last_active":"2024-02-19T21:15:53.402987+0000","last_peered":"2024-02-19T21:15:53.402987+0000","last_clean":"2024-02-19T21:15:53.402987+0000","last_became_active":"2024-02-19T21:14:41.403799+0000","last_became_peered":"2024-02-19T21:14:41.403799+0000","last_unstale":"2024-02-19T21:15:53.402987+0000","last_undegraded":"2024-02-19T21:15:53.402987+0000","last_fullsized":"2024-02-19T21:15:53.402987+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:13.932020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_clean_scrub_stamp":"2024-02-19T21:14:13.932020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":673,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.417009+0000","last_change":"2024-02-19T21:15:04.348697+0000","last_active":"2024-02-19T21:15:53.417009+0000","last_peered":"2024-02-19T21:15:53.417009+0000","last_clean":"2024-02-19T21:15:53.417009+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:15:53.417009+0000","last_undegraded":"2024-02-19T21:15:53.417009+0000","last_fullsized":"2024-02-19T21:15:53.417009+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:04.348628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:04.348628+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"459'22","reported_seq":734,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.415198+0000","last_change":"2024-02-19T21:14:41.538659+0000","last_active":"2024-02-19T21:15:53.415198+0000","last_peered":"2024-02-19T21:15:53.415198+0000","last_clean":"2024-02-19T21:15:53.415198+0000","last_became_active":"2024-02-19T21:14:41.404362+0000","last_became_peered":"2024-02-19T21:14:41.404362+0000","last_unstale":"2024-02-19T21:15:53.415198+0000","last_undegraded":"2024-02-19T21:15:53.415198+0000","last_fullsized":"2024-02-19T21:15:53.415198+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'22","last_scrub_stamp":"2024-02-19T21:14:11.471431+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:14:11.471431+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":745,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.414783+0000","last_change":"2024-02-19T21:15:51.240120+0000","last_active":"2024-02-19T21:15:53.414783+0000","last_peered":"2024-02-19T21:15:53.414783+0000","last_clean":"2024-02-19T21:15:53.414783+0000","last_became_active":"2024-02-19T21:14:41.411415+0000","last_became_peered":"2024-02-19T21:14:41.411415+0000","last_unstale":"2024-02-19T21:15:53.414783+0000","last_undegraded":"2024-02-19T21:15:53.414783+0000","last_fullsized":"2024-02-19T21:15:53.414783+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:51.240046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:51.240046+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":1574,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.412752+0000","last_change":"2024-02-19T21:15:44.948603+0000","last_active":"2024-02-19T21:15:54.412752+0000","last_peered":"2024-02-19T21:15:54.412752+0000","last_clean":"2024-02-19T21:15:54.412752+0000","last_became_active":"2024-02-19T21:14:16.388793+0000","last_became_peered":"2024-02-19T21:14:16.388793+0000","last_unstale":"2024-02-19T21:15:54.412752+0000","last_undegraded":"2024-02-19T21:15:54.412752+0000","last_fullsized":"2024-02-19T21:15:54.412752+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:15:44.948532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:44.948532+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":868,"num_read_kb":863,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":702,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410883+0000","last_change":"2024-02-19T21:14:56.859043+0000","last_active":"2024-02-19T21:15:53.410883+0000","last_peered":"2024-02-19T21:15:53.410883+0000","last_clean":"2024-02-19T21:15:53.410883+0000","last_became_active":"2024-02-19T21:14:16.390838+0000","last_became_peered":"2024-02-19T21:14:16.390838+0000","last_unstale":"2024-02-19T21:15:53.410883+0000","last_undegraded":"2024-02-19T21:15:53.410883+0000","last_fullsized":"2024-02-19T21:15:53.410883+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:56.858956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:56.858956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":637,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411437+0000","last_change":"2024-02-19T21:14:49.969389+0000","last_active":"2024-02-19T21:15:53.411437+0000","last_peered":"2024-02-19T21:15:53.411437+0000","last_clean":"2024-02-19T21:15:53.411437+0000","last_became_active":"2024-02-19T21:14:41.391420+0000","last_became_peered":"2024-02-19T21:14:41.391420+0000","last_unstale":"2024-02-19T21:15:53.411437+0000","last_undegraded":"2024-02-19T21:15:53.411437+0000","last_fullsized":"2024-02-19T21:15:53.411437+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:49.969308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_clean_scrub_stamp":"2024-02-19T21:14:49.969308+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":556,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411106+0000","last_change":"2024-02-19T21:14:55.995419+0000","last_active":"2024-02-19T21:15:53.411106+0000","last_peered":"2024-02-19T21:15:53.411106+0000","last_clean":"2024-02-19T21:15:53.411106+0000","last_became_active":"2024-02-19T21:14:41.390554+0000","last_became_peered":"2024-02-19T21:14:41.390554+0000","last_unstale":"2024-02-19T21:15:53.411106+0000","last_undegraded":"2024-02-19T21:15:53.411106+0000","last_fullsized":"2024-02-19T21:15:53.411106+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:55.995348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:14:55.995348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"459'42","reported_seq":973,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.413890+0000","last_change":"2024-02-19T21:13:47.032674+0000","last_active":"2024-02-19T21:15:54.413890+0000","last_peered":"2024-02-19T21:15:54.413890+0000","last_clean":"2024-02-19T21:15:54.413890+0000","last_became_active":"2024-02-19T21:08:04.266322+0000","last_became_peered":"2024-02-19T21:08:04.266322+0000","last_unstale":"2024-02-19T21:15:54.413890+0000","last_undegraded":"2024-02-19T21:15:54.413890+0000","last_fullsized":"2024-02-19T21:15:54.413890+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'42","last_scrub_stamp":"2024-02-19T21:13:47.032604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:13:47.032604+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":729,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403529+0000","last_change":"2024-02-19T21:15:07.044813+0000","last_active":"2024-02-19T21:15:53.403529+0000","last_peered":"2024-02-19T21:15:53.403529+0000","last_clean":"2024-02-19T21:15:53.403529+0000","last_became_active":"2024-02-19T21:14:16.395604+0000","last_became_peered":"2024-02-19T21:14:16.395604+0000","last_unstale":"2024-02-19T21:15:53.403529+0000","last_undegraded":"2024-02-19T21:15:53.403529+0000","last_fullsized":"2024-02-19T21:15:53.403529+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:07.044737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:15:07.044737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":439,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410772+0000","last_change":"2024-02-19T21:15:35.131877+0000","last_active":"2024-02-19T21:15:53.410772+0000","last_peered":"2024-02-19T21:15:53.410772+0000","last_clean":"2024-02-19T21:15:53.410772+0000","last_became_active":"2024-02-19T21:10:41.305829+0000","last_became_peered":"2024-02-19T21:10:41.305829+0000","last_unstale":"2024-02-19T21:15:53.410772+0000","last_undegraded":"2024-02-19T21:15:53.410772+0000","last_fullsized":"2024-02-19T21:15:53.410772+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:35.131770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:35.131770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":638,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.413239+0000","last_change":"2024-02-19T21:14:41.407695+0000","last_active":"2024-02-19T21:15:53.413239+0000","last_peered":"2024-02-19T21:15:53.413239+0000","last_clean":"2024-02-19T21:15:53.413239+0000","last_became_active":"2024-02-19T21:14:41.403835+0000","last_became_peered":"2024-02-19T21:14:41.403835+0000","last_unstale":"2024-02-19T21:15:53.413239+0000","last_undegraded":"2024-02-19T21:15:53.413239+0000","last_fullsized":"2024-02-19T21:15:53.413239+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:51.499300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:13:51.499300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"640'14","reported_seq":134,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.435275+0000","last_change":"2024-02-19T21:15:43.163121+0000","last_active":"2024-02-19T21:15:54.435275+0000","last_peered":"2024-02-19T21:15:54.435275+0000","last_clean":"2024-02-19T21:15:54.435275+0000","last_became_active":"2024-02-19T21:14:41.395291+0000","last_became_peered":"2024-02-19T21:14:41.395291+0000","last_unstale":"2024-02-19T21:15:54.435275+0000","last_undegraded":"2024-02-19T21:15:54.435275+0000","last_fullsized":"2024-02-19T21:15:54.435275+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"640'14","last_scrub_stamp":"2024-02-19T21:15:43.163004+0000","last_deep_scrub":"471'10","last_deep_scrub_stamp":"2024-02-19T21:11:41.775418+0000","last_clean_scrub_stamp":"2024-02-19T21:15:43.163004+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"459'20","reported_seq":741,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410807+0000","last_change":"2024-02-19T21:15:38.900504+0000","last_active":"2024-02-19T21:15:53.410807+0000","last_peered":"2024-02-19T21:15:53.410807+0000","last_clean":"2024-02-19T21:15:53.410807+0000","last_became_active":"2024-02-19T21:10:41.296275+0000","last_became_peered":"2024-02-19T21:10:41.296275+0000","last_unstale":"2024-02-19T21:15:53.410807+0000","last_undegraded":"2024-02-19T21:15:53.410807+0000","last_fullsized":"2024-02-19T21:15:53.410807+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:15:38.900433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:38.900433+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":743,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411579+0000","last_change":"2024-02-19T21:14:41.394241+0000","last_active":"2024-02-19T21:15:53.411579+0000","last_peered":"2024-02-19T21:15:53.411579+0000","last_clean":"2024-02-19T21:15:53.411579+0000","last_became_active":"2024-02-19T21:14:41.391704+0000","last_became_peered":"2024-02-19T21:14:41.391704+0000","last_unstale":"2024-02-19T21:15:53.411579+0000","last_undegraded":"2024-02-19T21:15:53.411579+0000","last_fullsized":"2024-02-19T21:15:53.411579+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:48.003444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:13:48.003444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":439,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403333+0000","last_change":"2024-02-19T21:14:41.383030+0000","last_active":"2024-02-19T21:15:53.403333+0000","last_peered":"2024-02-19T21:15:53.403333+0000","last_clean":"2024-02-19T21:15:53.403333+0000","last_became_active":"2024-02-19T21:14:41.382529+0000","last_became_peered":"2024-02-19T21:14:41.382529+0000","last_unstale":"2024-02-19T21:15:53.403333+0000","last_undegraded":"2024-02-19T21:15:53.403333+0000","last_fullsized":"2024-02-19T21:15:53.403333+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:07.221594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.221594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":439,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410819+0000","last_change":"2024-02-19T21:15:03.219533+0000","last_active":"2024-02-19T21:15:53.410819+0000","last_peered":"2024-02-19T21:15:53.410819+0000","last_clean":"2024-02-19T21:15:53.410819+0000","last_became_active":"2024-02-19T21:14:16.383904+0000","last_became_peered":"2024-02-19T21:14:16.383904+0000","last_unstale":"2024-02-19T21:15:53.410819+0000","last_undegraded":"2024-02-19T21:15:53.410819+0000","last_fullsized":"2024-02-19T21:15:53.410819+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:03.219469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:03.219469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":691,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.415533+0000","last_change":"2024-02-19T21:15:40.357385+0000","last_active":"2024-02-19T21:15:53.415533+0000","last_peered":"2024-02-19T21:15:53.415533+0000","last_clean":"2024-02-19T21:15:53.415533+0000","last_became_active":"2024-02-19T21:14:41.404657+0000","last_became_peered":"2024-02-19T21:14:41.404657+0000","last_unstale":"2024-02-19T21:15:53.415533+0000","last_undegraded":"2024-02-19T21:15:53.415533+0000","last_fullsized":"2024-02-19T21:15:53.415533+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:40.357299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:15:40.357299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"459'23","reported_seq":308,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404873+0000","last_change":"2024-02-19T21:15:43.850016+0000","last_active":"2024-02-19T21:15:53.404873+0000","last_peered":"2024-02-19T21:15:53.404873+0000","last_clean":"2024-02-19T21:15:53.404873+0000","last_became_active":"2024-02-19T21:10:41.287134+0000","last_became_peered":"2024-02-19T21:10:41.287134+0000","last_unstale":"2024-02-19T21:15:53.404873+0000","last_undegraded":"2024-02-19T21:15:53.404873+0000","last_fullsized":"2024-02-19T21:15:53.404873+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'23","last_scrub_stamp":"2024-02-19T21:15:43.849916+0000","last_deep_scrub":"420'17","last_deep_scrub_stamp":"2024-02-19T21:10:19.722303+0000","last_clean_scrub_stamp":"2024-02-19T21:15:43.849916+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":323,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.407063+0000","last_change":"2024-02-19T21:14:49.574097+0000","last_active":"2024-02-19T21:15:53.407063+0000","last_peered":"2024-02-19T21:15:53.407063+0000","last_clean":"2024-02-19T21:15:53.407063+0000","last_became_active":"2024-02-19T21:14:41.391555+0000","last_became_peered":"2024-02-19T21:14:41.391555+0000","last_unstale":"2024-02-19T21:15:53.407063+0000","last_undegraded":"2024-02-19T21:15:53.407063+0000","last_fullsized":"2024-02-19T21:15:53.407063+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:14:49.574024+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:14:49.574024+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":421,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403562+0000","last_change":"2024-02-19T21:13:00.166548+0000","last_active":"2024-02-19T21:15:53.403562+0000","last_peered":"2024-02-19T21:15:53.403562+0000","last_clean":"2024-02-19T21:15:53.403562+0000","last_became_active":"2024-02-19T21:08:04.264593+0000","last_became_peered":"2024-02-19T21:08:04.264593+0000","last_unstale":"2024-02-19T21:15:53.403562+0000","last_undegraded":"2024-02-19T21:15:53.403562+0000","last_fullsized":"2024-02-19T21:15:53.403562+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:00.166479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:13:00.166479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":620,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.416781+0000","last_change":"2024-02-19T21:13:44.567327+0000","last_active":"2024-02-19T21:15:53.416781+0000","last_peered":"2024-02-19T21:15:53.416781+0000","last_clean":"2024-02-19T21:15:53.416781+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:15:53.416781+0000","last_undegraded":"2024-02-19T21:15:53.416781+0000","last_fullsized":"2024-02-19T21:15:53.416781+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:44.567199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:13:44.567199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"459'15","reported_seq":480,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410373+0000","last_change":"2024-02-19T21:14:41.794436+0000","last_active":"2024-02-19T21:15:53.410373+0000","last_peered":"2024-02-19T21:15:53.410373+0000","last_clean":"2024-02-19T21:15:53.410373+0000","last_became_active":"2024-02-19T21:14:41.401051+0000","last_became_peered":"2024-02-19T21:14:41.401051+0000","last_unstale":"2024-02-19T21:15:53.410373+0000","last_undegraded":"2024-02-19T21:15:53.410373+0000","last_fullsized":"2024-02-19T21:15:53.410373+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:14:21.960125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:21.960125+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":690,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.415473+0000","last_change":"2024-02-19T21:14:41.405830+0000","last_active":"2024-02-19T21:15:53.415473+0000","last_peered":"2024-02-19T21:15:53.415473+0000","last_clean":"2024-02-19T21:15:53.415473+0000","last_became_active":"2024-02-19T21:14:41.404263+0000","last_became_peered":"2024-02-19T21:14:41.404263+0000","last_unstale":"2024-02-19T21:15:53.415473+0000","last_undegraded":"2024-02-19T21:15:53.415473+0000","last_fullsized":"2024-02-19T21:15:53.415473+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:12.482232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:12.482232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":421,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.409551+0000","last_change":"2024-02-19T21:14:56.048540+0000","last_active":"2024-02-19T21:15:53.409551+0000","last_peered":"2024-02-19T21:15:53.409551+0000","last_clean":"2024-02-19T21:15:53.409551+0000","last_became_active":"2024-02-19T21:08:04.264972+0000","last_became_peered":"2024-02-19T21:08:04.264972+0000","last_unstale":"2024-02-19T21:15:53.409551+0000","last_undegraded":"2024-02-19T21:15:53.409551+0000","last_fullsized":"2024-02-19T21:15:53.409551+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:56.048471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:14:56.048471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":689,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.415942+0000","last_change":"2024-02-19T21:14:48.323376+0000","last_active":"2024-02-19T21:15:53.415942+0000","last_peered":"2024-02-19T21:15:53.415942+0000","last_clean":"2024-02-19T21:15:53.415942+0000","last_became_active":"2024-02-19T21:10:41.314114+0000","last_became_peered":"2024-02-19T21:10:41.314114+0000","last_unstale":"2024-02-19T21:15:53.415942+0000","last_undegraded":"2024-02-19T21:15:53.415942+0000","last_fullsized":"2024-02-19T21:15:53.415942+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:48.323269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:48.323269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":282,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405938+0000","last_change":"2024-02-19T21:12:08.430936+0000","last_active":"2024-02-19T21:15:53.405938+0000","last_peered":"2024-02-19T21:15:53.405938+0000","last_clean":"2024-02-19T21:15:53.405938+0000","last_became_active":"2024-02-19T21:10:41.300574+0000","last_became_peered":"2024-02-19T21:10:41.300574+0000","last_unstale":"2024-02-19T21:15:53.405938+0000","last_undegraded":"2024-02-19T21:15:53.405938+0000","last_fullsized":"2024-02-19T21:15:53.405938+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:08.430874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:08.430874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"459'15","reported_seq":303,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.417199+0000","last_change":"2024-02-19T21:13:43.552100+0000","last_active":"2024-02-19T21:15:53.417199+0000","last_peered":"2024-02-19T21:15:53.417199+0000","last_clean":"2024-02-19T21:15:53.417199+0000","last_became_active":"2024-02-19T21:10:40.297867+0000","last_became_peered":"2024-02-19T21:10:40.297867+0000","last_unstale":"2024-02-19T21:15:53.417199+0000","last_undegraded":"2024-02-19T21:15:53.417199+0000","last_fullsized":"2024-02-19T21:15:53.417199+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:13:43.552026+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:13:43.552026+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":97,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.407714+0000","last_change":"2024-02-19T21:15:07.220115+0000","last_active":"2024-02-19T21:15:53.407714+0000","last_peered":"2024-02-19T21:15:53.407714+0000","last_clean":"2024-02-19T21:15:53.407714+0000","last_became_active":"2024-02-19T21:14:16.385407+0000","last_became_peered":"2024-02-19T21:14:16.385407+0000","last_unstale":"2024-02-19T21:15:53.407714+0000","last_undegraded":"2024-02-19T21:15:53.407714+0000","last_fullsized":"2024-02-19T21:15:53.407714+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:07.220022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:07.220022+0000","last_clean_scrub_stamp":"2024-02-19T21:15:07.220022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":691,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404419+0000","last_change":"2024-02-19T21:14:59.160565+0000","last_active":"2024-02-19T21:15:53.404419+0000","last_peered":"2024-02-19T21:15:53.404419+0000","last_clean":"2024-02-19T21:15:53.404419+0000","last_became_active":"2024-02-19T21:08:04.265573+0000","last_became_peered":"2024-02-19T21:08:04.265573+0000","last_unstale":"2024-02-19T21:15:53.404419+0000","last_undegraded":"2024-02-19T21:15:53.404419+0000","last_fullsized":"2024-02-19T21:15:53.404419+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:59.160470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:59.160470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":704,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410646+0000","last_change":"2024-02-19T21:14:53.918607+0000","last_active":"2024-02-19T21:15:53.410646+0000","last_peered":"2024-02-19T21:15:53.410646+0000","last_clean":"2024-02-19T21:15:53.410646+0000","last_became_active":"2024-02-19T21:14:16.379202+0000","last_became_peered":"2024-02-19T21:14:16.379202+0000","last_unstale":"2024-02-19T21:15:53.410646+0000","last_undegraded":"2024-02-19T21:15:53.410646+0000","last_fullsized":"2024-02-19T21:15:53.410646+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:53.918521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:53.918521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":659,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404111+0000","last_change":"2024-02-19T21:15:02.096155+0000","last_active":"2024-02-19T21:15:53.404111+0000","last_peered":"2024-02-19T21:15:53.404111+0000","last_clean":"2024-02-19T21:15:53.404111+0000","last_became_active":"2024-02-19T21:14:16.397730+0000","last_became_peered":"2024-02-19T21:14:16.397730+0000","last_unstale":"2024-02-19T21:15:53.404111+0000","last_undegraded":"2024-02-19T21:15:53.404111+0000","last_fullsized":"2024-02-19T21:15:53.404111+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:02.096039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:02.096039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"459'20","reported_seq":448,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404210+0000","last_change":"2024-02-19T21:14:47.087994+0000","last_active":"2024-02-19T21:15:53.404210+0000","last_peered":"2024-02-19T21:15:53.404210+0000","last_clean":"2024-02-19T21:15:53.404210+0000","last_became_active":"2024-02-19T21:08:04.267216+0000","last_became_peered":"2024-02-19T21:08:04.267216+0000","last_unstale":"2024-02-19T21:15:53.404210+0000","last_undegraded":"2024-02-19T21:15:53.404210+0000","last_fullsized":"2024-02-19T21:15:53.404210+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:14:47.087905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:47.087905+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":698,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403454+0000","last_change":"2024-02-19T21:15:42.974119+0000","last_active":"2024-02-19T21:15:53.403454+0000","last_peered":"2024-02-19T21:15:53.403454+0000","last_clean":"2024-02-19T21:15:53.403454+0000","last_became_active":"2024-02-19T21:08:04.266728+0000","last_became_peered":"2024-02-19T21:08:04.266728+0000","last_unstale":"2024-02-19T21:15:53.403454+0000","last_undegraded":"2024-02-19T21:15:53.403454+0000","last_fullsized":"2024-02-19T21:15:53.403454+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:42.974035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_clean_scrub_stamp":"2024-02-19T21:15:42.974035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"640'10","reported_seq":881,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.430573+0000","last_change":"2024-02-19T21:15:37.998427+0000","last_active":"2024-02-19T21:15:54.430573+0000","last_peered":"2024-02-19T21:15:54.430573+0000","last_clean":"2024-02-19T21:15:54.430573+0000","last_became_active":"2024-02-19T21:14:41.403775+0000","last_became_peered":"2024-02-19T21:14:41.403775+0000","last_unstale":"2024-02-19T21:15:54.430573+0000","last_undegraded":"2024-02-19T21:15:54.430573+0000","last_fullsized":"2024-02-19T21:15:54.430573+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"640'10","last_scrub_stamp":"2024-02-19T21:15:37.998310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:37.998310+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":560,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410991+0000","last_change":"2024-02-19T21:14:41.631201+0000","last_active":"2024-02-19T21:15:53.410991+0000","last_peered":"2024-02-19T21:15:53.410991+0000","last_clean":"2024-02-19T21:15:53.410991+0000","last_became_active":"2024-02-19T21:14:41.415447+0000","last_became_peered":"2024-02-19T21:14:41.415447+0000","last_unstale":"2024-02-19T21:15:53.410991+0000","last_undegraded":"2024-02-19T21:15:53.410991+0000","last_fullsized":"2024-02-19T21:15:53.410991+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:14:18.606934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:18.606934+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":101,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.407724+0000","last_change":"2024-02-19T21:14:41.395579+0000","last_active":"2024-02-19T21:15:53.407724+0000","last_peered":"2024-02-19T21:15:53.407724+0000","last_clean":"2024-02-19T21:15:53.407724+0000","last_became_active":"2024-02-19T21:14:41.395223+0000","last_became_peered":"2024-02-19T21:14:41.395223+0000","last_unstale":"2024-02-19T21:15:53.407724+0000","last_undegraded":"2024-02-19T21:15:53.407724+0000","last_fullsized":"2024-02-19T21:15:53.407724+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:13.532568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:55.202463+0000","last_clean_scrub_stamp":"2024-02-19T21:14:13.532568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"459'15","reported_seq":736,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404893+0000","last_change":"2024-02-19T21:14:15.936988+0000","last_active":"2024-02-19T21:15:53.404893+0000","last_peered":"2024-02-19T21:15:53.404893+0000","last_clean":"2024-02-19T21:15:53.404893+0000","last_became_active":"2024-02-19T21:08:04.266103+0000","last_became_peered":"2024-02-19T21:08:04.266103+0000","last_unstale":"2024-02-19T21:15:53.404893+0000","last_undegraded":"2024-02-19T21:15:53.404893+0000","last_fullsized":"2024-02-19T21:15:53.404893+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:14:15.936909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:15.936909+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":305,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405091+0000","last_change":"2024-02-19T21:14:41.415900+0000","last_active":"2024-02-19T21:15:53.405091+0000","last_peered":"2024-02-19T21:15:53.405091+0000","last_clean":"2024-02-19T21:15:53.405091+0000","last_became_active":"2024-02-19T21:14:41.415500+0000","last_became_peered":"2024-02-19T21:14:41.415500+0000","last_unstale":"2024-02-19T21:15:53.405091+0000","last_undegraded":"2024-02-19T21:15:53.405091+0000","last_fullsized":"2024-02-19T21:15:53.405091+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:07.506946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.506946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":525,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.405687+0000","last_change":"2024-02-19T21:15:43.054159+0000","last_active":"2024-02-19T21:15:54.405687+0000","last_peered":"2024-02-19T21:15:54.405687+0000","last_clean":"2024-02-19T21:15:54.405687+0000","last_became_active":"2024-02-19T21:08:04.256045+0000","last_became_peered":"2024-02-19T21:08:04.256045+0000","last_unstale":"2024-02-19T21:15:54.405687+0000","last_undegraded":"2024-02-19T21:15:54.405687+0000","last_fullsized":"2024-02-19T21:15:54.405687+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'10","last_scrub_stamp":"2024-02-19T21:15:43.054055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:43.054055+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":287,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.407147+0000","last_change":"2024-02-19T21:15:41.829174+0000","last_active":"2024-02-19T21:15:53.407147+0000","last_peered":"2024-02-19T21:15:53.407147+0000","last_clean":"2024-02-19T21:15:53.407147+0000","last_became_active":"2024-02-19T21:10:41.310279+0000","last_became_peered":"2024-02-19T21:10:41.310279+0000","last_unstale":"2024-02-19T21:15:53.407147+0000","last_undegraded":"2024-02-19T21:15:53.407147+0000","last_fullsized":"2024-02-19T21:15:53.407147+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:41.829068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_clean_scrub_stamp":"2024-02-19T21:15:41.829068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":690,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.415587+0000","last_change":"2024-02-19T21:15:03.372324+0000","last_active":"2024-02-19T21:15:53.415587+0000","last_peered":"2024-02-19T21:15:53.415587+0000","last_clean":"2024-02-19T21:15:53.415587+0000","last_became_active":"2024-02-19T21:14:16.371724+0000","last_became_peered":"2024-02-19T21:14:16.371724+0000","last_unstale":"2024-02-19T21:15:53.415587+0000","last_undegraded":"2024-02-19T21:15:53.415587+0000","last_fullsized":"2024-02-19T21:15:53.415587+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-02-19T21:15:03.372256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:03.372256+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":20,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"459'15","reported_seq":448,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410278+0000","last_change":"2024-02-19T21:14:16.723013+0000","last_active":"2024-02-19T21:15:53.410278+0000","last_peered":"2024-02-19T21:15:53.410278+0000","last_clean":"2024-02-19T21:15:53.410278+0000","last_became_active":"2024-02-19T21:14:16.377099+0000","last_became_peered":"2024-02-19T21:14:16.377099+0000","last_unstale":"2024-02-19T21:15:53.410278+0000","last_undegraded":"2024-02-19T21:15:53.410278+0000","last_fullsized":"2024-02-19T21:15:53.410278+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:13:06.958518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.958518+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":97,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.408462+0000","last_change":"2024-02-19T21:14:53.410134+0000","last_active":"2024-02-19T21:15:53.408462+0000","last_peered":"2024-02-19T21:15:53.408462+0000","last_clean":"2024-02-19T21:15:53.408462+0000","last_became_active":"2024-02-19T21:14:16.388194+0000","last_became_peered":"2024-02-19T21:14:16.388194+0000","last_unstale":"2024-02-19T21:15:53.408462+0000","last_undegraded":"2024-02-19T21:15:53.408462+0000","last_fullsized":"2024-02-19T21:15:53.408462+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:53.410025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:14:53.410025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":444,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403725+0000","last_change":"2024-02-19T21:15:41.081514+0000","last_active":"2024-02-19T21:15:53.403725+0000","last_peered":"2024-02-19T21:15:53.403725+0000","last_clean":"2024-02-19T21:15:53.403725+0000","last_became_active":"2024-02-19T21:14:41.398865+0000","last_became_peered":"2024-02-19T21:14:41.398865+0000","last_unstale":"2024-02-19T21:15:53.403725+0000","last_undegraded":"2024-02-19T21:15:53.403725+0000","last_fullsized":"2024-02-19T21:15:53.403725+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:41.081437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_clean_scrub_stamp":"2024-02-19T21:15:41.081437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":542,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403116+0000","last_change":"2024-02-19T21:15:49.956671+0000","last_active":"2024-02-19T21:15:53.403116+0000","last_peered":"2024-02-19T21:15:53.403116+0000","last_clean":"2024-02-19T21:15:53.403116+0000","last_became_active":"2024-02-19T21:14:41.388322+0000","last_became_peered":"2024-02-19T21:14:41.388322+0000","last_unstale":"2024-02-19T21:15:53.403116+0000","last_undegraded":"2024-02-19T21:15:53.403116+0000","last_fullsized":"2024-02-19T21:15:53.403116+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:49.956589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:49.956589+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":106,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.406367+0000","last_change":"2024-02-19T21:14:41.400692+0000","last_active":"2024-02-19T21:15:53.406367+0000","last_peered":"2024-02-19T21:15:53.406367+0000","last_clean":"2024-02-19T21:15:53.406367+0000","last_became_active":"2024-02-19T21:14:41.400354+0000","last_became_peered":"2024-02-19T21:14:41.400354+0000","last_unstale":"2024-02-19T21:15:53.406367+0000","last_undegraded":"2024-02-19T21:15:53.406367+0000","last_fullsized":"2024-02-19T21:15:53.406367+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:19.148620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:19.148620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"459'19","reported_seq":353,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405263+0000","last_change":"2024-02-19T21:14:56.610314+0000","last_active":"2024-02-19T21:15:53.405263+0000","last_peered":"2024-02-19T21:15:53.405263+0000","last_clean":"2024-02-19T21:15:53.405263+0000","last_became_active":"2024-02-19T21:14:16.380020+0000","last_became_peered":"2024-02-19T21:14:16.380020+0000","last_unstale":"2024-02-19T21:15:53.405263+0000","last_undegraded":"2024-02-19T21:15:53.405263+0000","last_fullsized":"2024-02-19T21:15:53.405263+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:14:56.610233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:56.610233+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":712,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.936233+0000","last_change":"2024-02-19T21:15:53.936208+0000","last_active":"2024-02-19T21:15:53.936233+0000","last_peered":"2024-02-19T21:15:53.936233+0000","last_clean":"2024-02-19T21:15:53.936233+0000","last_became_active":"2024-02-19T21:14:41.395354+0000","last_became_peered":"2024-02-19T21:14:41.395354+0000","last_unstale":"2024-02-19T21:15:53.936233+0000","last_undegraded":"2024-02-19T21:15:53.936233+0000","last_fullsized":"2024-02-19T21:15:53.936233+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.936143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.936143+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":690,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410882+0000","last_change":"2024-02-19T21:15:46.944899+0000","last_active":"2024-02-19T21:15:53.410882+0000","last_peered":"2024-02-19T21:15:53.410882+0000","last_clean":"2024-02-19T21:15:53.410882+0000","last_became_active":"2024-02-19T21:08:04.268847+0000","last_became_peered":"2024-02-19T21:08:04.268847+0000","last_unstale":"2024-02-19T21:15:53.410882+0000","last_undegraded":"2024-02-19T21:15:53.410882+0000","last_fullsized":"2024-02-19T21:15:53.410882+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:46.944774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:46.944774+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":439,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403496+0000","last_change":"2024-02-19T21:15:36.137073+0000","last_active":"2024-02-19T21:15:53.403496+0000","last_peered":"2024-02-19T21:15:53.403496+0000","last_clean":"2024-02-19T21:15:53.403496+0000","last_became_active":"2024-02-19T21:14:41.401891+0000","last_became_peered":"2024-02-19T21:14:41.401891+0000","last_unstale":"2024-02-19T21:15:53.403496+0000","last_undegraded":"2024-02-19T21:15:53.403496+0000","last_fullsized":"2024-02-19T21:15:53.403496+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:36.137008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:36.137008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":439,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.409641+0000","last_change":"2024-02-19T21:14:59.059646+0000","last_active":"2024-02-19T21:15:53.409641+0000","last_peered":"2024-02-19T21:15:53.409641+0000","last_clean":"2024-02-19T21:15:53.409641+0000","last_became_active":"2024-02-19T21:14:41.397884+0000","last_became_peered":"2024-02-19T21:14:41.397884+0000","last_unstale":"2024-02-19T21:15:53.409641+0000","last_undegraded":"2024-02-19T21:15:53.409641+0000","last_fullsized":"2024-02-19T21:15:53.409641+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:59.059572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:59.059572+0000","last_clean_scrub_stamp":"2024-02-19T21:14:59.059572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"555'10","reported_seq":332,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.402990+0000","last_change":"2024-02-19T21:15:40.997672+0000","last_active":"2024-02-19T21:15:53.402990+0000","last_peered":"2024-02-19T21:15:53.402990+0000","last_clean":"2024-02-19T21:15:53.402990+0000","last_became_active":"2024-02-19T21:14:16.395203+0000","last_became_peered":"2024-02-19T21:14:16.395203+0000","last_unstale":"2024-02-19T21:15:53.402990+0000","last_undegraded":"2024-02-19T21:15:53.402990+0000","last_fullsized":"2024-02-19T21:15:53.402990+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'10","last_scrub_stamp":"2024-02-19T21:15:40.997592+0000","last_deep_scrub":"555'10","last_deep_scrub_stamp":"2024-02-19T21:13:12.409036+0000","last_clean_scrub_stamp":"2024-02-19T21:15:40.997592+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":698,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404563+0000","last_change":"2024-02-19T21:15:44.964518+0000","last_active":"2024-02-19T21:15:53.404563+0000","last_peered":"2024-02-19T21:15:53.404563+0000","last_clean":"2024-02-19T21:15:53.404563+0000","last_became_active":"2024-02-19T21:14:16.395740+0000","last_became_peered":"2024-02-19T21:14:16.395740+0000","last_unstale":"2024-02-19T21:15:53.404563+0000","last_undegraded":"2024-02-19T21:15:53.404563+0000","last_fullsized":"2024-02-19T21:15:53.404563+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:44.964433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:44.964433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":97,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.200563+0000","last_change":"2024-02-19T21:15:54.200511+0000","last_active":"2024-02-19T21:15:54.200563+0000","last_peered":"2024-02-19T21:15:54.200563+0000","last_clean":"2024-02-19T21:15:54.200563+0000","last_became_active":"2024-02-19T21:14:16.390399+0000","last_became_peered":"2024-02-19T21:14:16.390399+0000","last_unstale":"2024-02-19T21:15:54.200563+0000","last_undegraded":"2024-02-19T21:15:54.200563+0000","last_fullsized":"2024-02-19T21:15:54.200563+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:54.200442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:54.200442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":684,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410102+0000","last_change":"2024-02-19T21:14:50.846672+0000","last_active":"2024-02-19T21:15:53.410102+0000","last_peered":"2024-02-19T21:15:53.410102+0000","last_clean":"2024-02-19T21:15:53.410102+0000","last_became_active":"2024-02-19T21:10:41.314589+0000","last_became_peered":"2024-02-19T21:10:41.314589+0000","last_unstale":"2024-02-19T21:15:53.410102+0000","last_undegraded":"2024-02-19T21:15:53.410102+0000","last_fullsized":"2024-02-19T21:15:53.410102+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:50.846595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:50.846595+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":426,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404277+0000","last_change":"2024-02-19T21:15:50.223281+0000","last_active":"2024-02-19T21:15:53.404277+0000","last_peered":"2024-02-19T21:15:53.404277+0000","last_clean":"2024-02-19T21:15:53.404277+0000","last_became_active":"2024-02-19T21:08:04.266161+0000","last_became_peered":"2024-02-19T21:08:04.266161+0000","last_unstale":"2024-02-19T21:15:53.404277+0000","last_undegraded":"2024-02-19T21:15:53.404277+0000","last_fullsized":"2024-02-19T21:15:53.404277+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:50.223212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:50.223212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"459'19","reported_seq":492,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403744+0000","last_change":"2024-02-19T21:15:42.047875+0000","last_active":"2024-02-19T21:15:53.403744+0000","last_peered":"2024-02-19T21:15:53.403744+0000","last_clean":"2024-02-19T21:15:53.403744+0000","last_became_active":"2024-02-19T21:14:16.386212+0000","last_became_peered":"2024-02-19T21:14:16.386212+0000","last_unstale":"2024-02-19T21:15:53.403744+0000","last_undegraded":"2024-02-19T21:15:53.403744+0000","last_fullsized":"2024-02-19T21:15:53.403744+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:15:42.047800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:42.047800+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":99,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.409713+0000","last_change":"2024-02-19T21:15:02.244608+0000","last_active":"2024-02-19T21:15:53.409713+0000","last_peered":"2024-02-19T21:15:53.409713+0000","last_clean":"2024-02-19T21:15:53.409713+0000","last_became_active":"2024-02-19T21:14:16.388239+0000","last_became_peered":"2024-02-19T21:14:16.388239+0000","last_unstale":"2024-02-19T21:15:53.409713+0000","last_undegraded":"2024-02-19T21:15:53.409713+0000","last_fullsized":"2024-02-19T21:15:53.409713+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:02.244550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:02.244550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":699,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411477+0000","last_change":"2024-02-19T21:14:19.588243+0000","last_active":"2024-02-19T21:15:53.411477+0000","last_peered":"2024-02-19T21:15:53.411477+0000","last_clean":"2024-02-19T21:15:53.411477+0000","last_became_active":"2024-02-19T21:14:16.375236+0000","last_became_peered":"2024-02-19T21:14:16.375236+0000","last_unstale":"2024-02-19T21:15:53.411477+0000","last_undegraded":"2024-02-19T21:15:53.411477+0000","last_fullsized":"2024-02-19T21:15:53.411477+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:19.588167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:19.588167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":457,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403748+0000","last_change":"2024-02-19T21:14:41.399175+0000","last_active":"2024-02-19T21:15:53.403748+0000","last_peered":"2024-02-19T21:15:53.403748+0000","last_clean":"2024-02-19T21:15:53.403748+0000","last_became_active":"2024-02-19T21:14:41.398484+0000","last_became_peered":"2024-02-19T21:14:41.398484+0000","last_unstale":"2024-02-19T21:15:53.403748+0000","last_undegraded":"2024-02-19T21:15:53.403748+0000","last_fullsized":"2024-02-19T21:15:53.403748+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:12.107010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:12.107010+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":106,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.406352+0000","last_change":"2024-02-19T21:15:36.240010+0000","last_active":"2024-02-19T21:15:53.406352+0000","last_peered":"2024-02-19T21:15:53.406352+0000","last_clean":"2024-02-19T21:15:53.406352+0000","last_became_active":"2024-02-19T21:14:41.400328+0000","last_became_peered":"2024-02-19T21:14:41.400328+0000","last_unstale":"2024-02-19T21:15:53.406352+0000","last_undegraded":"2024-02-19T21:15:53.406352+0000","last_fullsized":"2024-02-19T21:15:53.406352+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:36.239929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:15:36.239929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"459'20","reported_seq":625,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411095+0000","last_change":"2024-02-19T21:14:55.895841+0000","last_active":"2024-02-19T21:15:53.411095+0000","last_peered":"2024-02-19T21:15:53.411095+0000","last_clean":"2024-02-19T21:15:53.411095+0000","last_became_active":"2024-02-19T21:14:41.415491+0000","last_became_peered":"2024-02-19T21:14:41.415491+0000","last_unstale":"2024-02-19T21:15:53.411095+0000","last_undegraded":"2024-02-19T21:15:53.411095+0000","last_fullsized":"2024-02-19T21:15:53.411095+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:14:55.895741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:55.895741+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":695,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403096+0000","last_change":"2024-02-19T21:15:35.006355+0000","last_active":"2024-02-19T21:15:53.403096+0000","last_peered":"2024-02-19T21:15:53.403096+0000","last_clean":"2024-02-19T21:15:53.403096+0000","last_became_active":"2024-02-19T21:10:41.294592+0000","last_became_peered":"2024-02-19T21:10:41.294592+0000","last_unstale":"2024-02-19T21:15:53.403096+0000","last_undegraded":"2024-02-19T21:15:53.403096+0000","last_fullsized":"2024-02-19T21:15:53.403096+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:35.006289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:35.006289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"623'12","reported_seq":121,"reported_epoch":705,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.435496+0000","last_change":"2024-02-19T21:15:52.229142+0000","last_active":"2024-02-19T21:15:54.435496+0000","last_peered":"2024-02-19T21:15:54.435496+0000","last_clean":"2024-02-19T21:15:54.435496+0000","last_became_active":"2024-02-19T21:14:16.391675+0000","last_became_peered":"2024-02-19T21:14:16.391675+0000","last_unstale":"2024-02-19T21:15:54.435496+0000","last_undegraded":"2024-02-19T21:15:54.435496+0000","last_fullsized":"2024-02-19T21:15:54.435496+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"623'12","last_scrub_stamp":"2024-02-19T21:15:52.229069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:52.229069+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":668,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410966+0000","last_change":"2024-02-19T21:14:24.577379+0000","last_active":"2024-02-19T21:15:53.410966+0000","last_peered":"2024-02-19T21:15:53.410966+0000","last_clean":"2024-02-19T21:15:53.410966+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:15:53.410966+0000","last_undegraded":"2024-02-19T21:15:53.410966+0000","last_fullsized":"2024-02-19T21:15:53.410966+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:14:24.577303+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:14:24.577303+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":623,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.415647+0000","last_change":"2024-02-19T21:14:24.511577+0000","last_active":"2024-02-19T21:15:53.415647+0000","last_peered":"2024-02-19T21:15:53.415647+0000","last_clean":"2024-02-19T21:15:53.415647+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:15:53.415647+0000","last_undegraded":"2024-02-19T21:15:53.415647+0000","last_fullsized":"2024-02-19T21:15:53.415647+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:24.511507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:24.511507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"459'19","reported_seq":109,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.409636+0000","last_change":"2024-02-19T21:14:22.079972+0000","last_active":"2024-02-19T21:15:53.409636+0000","last_peered":"2024-02-19T21:15:53.409636+0000","last_clean":"2024-02-19T21:15:53.409636+0000","last_became_active":"2024-02-19T21:14:16.389206+0000","last_became_peered":"2024-02-19T21:14:16.389206+0000","last_unstale":"2024-02-19T21:15:53.409636+0000","last_undegraded":"2024-02-19T21:15:53.409636+0000","last_fullsized":"2024-02-19T21:15:53.409636+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:14:22.079893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:22.079893+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":457,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403823+0000","last_change":"2024-02-19T21:15:37.108298+0000","last_active":"2024-02-19T21:15:53.403823+0000","last_peered":"2024-02-19T21:15:53.403823+0000","last_clean":"2024-02-19T21:15:53.403823+0000","last_became_active":"2024-02-19T21:14:41.402278+0000","last_became_peered":"2024-02-19T21:14:41.402278+0000","last_unstale":"2024-02-19T21:15:53.403823+0000","last_undegraded":"2024-02-19T21:15:53.403823+0000","last_fullsized":"2024-02-19T21:15:53.403823+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:37.108220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:37.108220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":517,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411252+0000","last_change":"2024-02-19T21:14:41.416235+0000","last_active":"2024-02-19T21:15:53.411252+0000","last_peered":"2024-02-19T21:15:53.411252+0000","last_clean":"2024-02-19T21:15:53.411252+0000","last_became_active":"2024-02-19T21:14:41.415911+0000","last_became_peered":"2024-02-19T21:14:41.415911+0000","last_unstale":"2024-02-19T21:15:53.411252+0000","last_undegraded":"2024-02-19T21:15:53.411252+0000","last_fullsized":"2024-02-19T21:15:53.411252+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:11.580253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:11.580253+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":683,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411048+0000","last_change":"2024-02-19T21:15:33.888785+0000","last_active":"2024-02-19T21:15:53.411048+0000","last_peered":"2024-02-19T21:15:53.411048+0000","last_clean":"2024-02-19T21:15:53.411048+0000","last_became_active":"2024-02-19T21:10:41.312577+0000","last_became_peered":"2024-02-19T21:10:41.312577+0000","last_unstale":"2024-02-19T21:15:53.411048+0000","last_undegraded":"2024-02-19T21:15:53.411048+0000","last_fullsized":"2024-02-19T21:15:53.411048+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:33.888719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:33.888719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":637,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411353+0000","last_change":"2024-02-19T21:13:53.087054+0000","last_active":"2024-02-19T21:15:53.411353+0000","last_peered":"2024-02-19T21:15:53.411353+0000","last_clean":"2024-02-19T21:15:53.411353+0000","last_became_active":"2024-02-19T21:10:41.311050+0000","last_became_peered":"2024-02-19T21:10:41.311050+0000","last_unstale":"2024-02-19T21:15:53.411353+0000","last_undegraded":"2024-02-19T21:15:53.411353+0000","last_fullsized":"2024-02-19T21:15:53.411353+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:53.086972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:13:53.086972+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"459'34","reported_seq":770,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.416086+0000","last_change":"2024-02-19T21:13:39.513661+0000","last_active":"2024-02-19T21:15:53.416086+0000","last_peered":"2024-02-19T21:15:53.416086+0000","last_clean":"2024-02-19T21:15:53.416086+0000","last_became_active":"2024-02-19T21:08:04.248378+0000","last_became_peered":"2024-02-19T21:08:04.248378+0000","last_unstale":"2024-02-19T21:15:53.416086+0000","last_undegraded":"2024-02-19T21:15:53.416086+0000","last_fullsized":"2024-02-19T21:15:53.416086+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'34","last_scrub_stamp":"2024-02-19T21:13:39.513604+0000","last_deep_scrub":"419'22","last_deep_scrub_stamp":"2024-02-19T21:10:19.109244+0000","last_clean_scrub_stamp":"2024-02-19T21:13:39.513604+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":101,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.408209+0000","last_change":"2024-02-19T21:15:41.405941+0000","last_active":"2024-02-19T21:15:53.408209+0000","last_peered":"2024-02-19T21:15:53.408209+0000","last_clean":"2024-02-19T21:15:53.408209+0000","last_became_active":"2024-02-19T21:15:41.405527+0000","last_became_peered":"2024-02-19T21:15:41.405527+0000","last_unstale":"2024-02-19T21:15:53.408209+0000","last_undegraded":"2024-02-19T21:15:53.408209+0000","last_fullsized":"2024-02-19T21:15:53.408209+0000","mapping_epoch":691,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":692,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:10.531467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:10.531467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":703,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.411401+0000","last_change":"2024-02-19T21:14:41.415798+0000","last_active":"2024-02-19T21:15:53.411401+0000","last_peered":"2024-02-19T21:15:53.411401+0000","last_clean":"2024-02-19T21:15:53.411401+0000","last_became_active":"2024-02-19T21:14:41.415463+0000","last_became_peered":"2024-02-19T21:14:41.415463+0000","last_unstale":"2024-02-19T21:15:53.411401+0000","last_undegraded":"2024-02-19T21:15:53.411401+0000","last_fullsized":"2024-02-19T21:15:53.411401+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:23.025940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:23.025940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":305,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.406163+0000","last_change":"2024-02-19T21:14:16.382742+0000","last_active":"2024-02-19T21:15:53.406163+0000","last_peered":"2024-02-19T21:15:53.406163+0000","last_clean":"2024-02-19T21:15:53.406163+0000","last_became_active":"2024-02-19T21:14:16.381600+0000","last_became_peered":"2024-02-19T21:14:16.381600+0000","last_unstale":"2024-02-19T21:15:53.406163+0000","last_undegraded":"2024-02-19T21:15:53.406163+0000","last_fullsized":"2024-02-19T21:15:53.406163+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:43.735382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:13:43.735382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":106,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.407644+0000","last_change":"2024-02-19T21:15:04.202157+0000","last_active":"2024-02-19T21:15:53.407644+0000","last_peered":"2024-02-19T21:15:53.407644+0000","last_clean":"2024-02-19T21:15:53.407644+0000","last_became_active":"2024-02-19T21:14:41.395278+0000","last_became_peered":"2024-02-19T21:14:41.395278+0000","last_unstale":"2024-02-19T21:15:53.407644+0000","last_undegraded":"2024-02-19T21:15:53.407644+0000","last_fullsized":"2024-02-19T21:15:53.407644+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:04.202042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:04.202042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"459'18","reported_seq":489,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:54.109565+0000","last_change":"2024-02-19T21:15:54.109530+0000","last_active":"2024-02-19T21:15:54.109565+0000","last_peered":"2024-02-19T21:15:54.109565+0000","last_clean":"2024-02-19T21:15:54.109565+0000","last_became_active":"2024-02-19T21:10:41.307690+0000","last_became_peered":"2024-02-19T21:10:41.307690+0000","last_unstale":"2024-02-19T21:15:54.109565+0000","last_undegraded":"2024-02-19T21:15:54.109565+0000","last_fullsized":"2024-02-19T21:15:54.109565+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'18","last_scrub_stamp":"2024-02-19T21:15:54.109426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:54.109426+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":439,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403808+0000","last_change":"2024-02-19T21:14:16.386656+0000","last_active":"2024-02-19T21:15:53.403808+0000","last_peered":"2024-02-19T21:15:53.403808+0000","last_clean":"2024-02-19T21:15:53.403808+0000","last_became_active":"2024-02-19T21:14:16.386352+0000","last_became_peered":"2024-02-19T21:14:16.386352+0000","last_unstale":"2024-02-19T21:15:53.403808+0000","last_undegraded":"2024-02-19T21:15:53.403808+0000","last_fullsized":"2024-02-19T21:15:53.403808+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:14.063569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:14.063569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":287,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404956+0000","last_change":"2024-02-19T21:14:59.655023+0000","last_active":"2024-02-19T21:15:53.404956+0000","last_peered":"2024-02-19T21:15:53.404956+0000","last_clean":"2024-02-19T21:15:53.404956+0000","last_became_active":"2024-02-19T21:10:41.296849+0000","last_became_peered":"2024-02-19T21:10:41.296849+0000","last_unstale":"2024-02-19T21:15:53.404956+0000","last_undegraded":"2024-02-19T21:15:53.404956+0000","last_fullsized":"2024-02-19T21:15:53.404956+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:59.654949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:59.654949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":439,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.409726+0000","last_change":"2024-02-19T21:14:58.020186+0000","last_active":"2024-02-19T21:15:53.409726+0000","last_peered":"2024-02-19T21:15:53.409726+0000","last_clean":"2024-02-19T21:15:53.409726+0000","last_became_active":"2024-02-19T21:14:16.385155+0000","last_became_peered":"2024-02-19T21:14:16.385155+0000","last_unstale":"2024-02-19T21:15:53.409726+0000","last_undegraded":"2024-02-19T21:15:53.409726+0000","last_fullsized":"2024-02-19T21:15:53.409726+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:58.020115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:14:58.020115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":637,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410146+0000","last_change":"2024-02-19T21:14:25.024614+0000","last_active":"2024-02-19T21:15:53.410146+0000","last_peered":"2024-02-19T21:15:53.410146+0000","last_clean":"2024-02-19T21:15:53.410146+0000","last_became_active":"2024-02-19T21:14:16.375068+0000","last_became_peered":"2024-02-19T21:14:16.375068+0000","last_unstale":"2024-02-19T21:15:53.410146+0000","last_undegraded":"2024-02-19T21:15:53.410146+0000","last_fullsized":"2024-02-19T21:15:53.410146+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:25.024547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:25.024547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"459'19","reported_seq":339,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405776+0000","last_change":"2024-02-19T21:14:41.422811+0000","last_active":"2024-02-19T21:15:53.405776+0000","last_peered":"2024-02-19T21:15:53.405776+0000","last_clean":"2024-02-19T21:15:53.405776+0000","last_became_active":"2024-02-19T21:14:41.390502+0000","last_became_peered":"2024-02-19T21:14:41.390502+0000","last_unstale":"2024-02-19T21:15:53.405776+0000","last_undegraded":"2024-02-19T21:15:53.405776+0000","last_fullsized":"2024-02-19T21:15:53.405776+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:12:55.429801+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.429801+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":667,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410563+0000","last_change":"2024-02-19T21:14:25.616331+0000","last_active":"2024-02-19T21:15:53.410563+0000","last_peered":"2024-02-19T21:15:53.410563+0000","last_clean":"2024-02-19T21:15:53.410563+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:15:53.410563+0000","last_undegraded":"2024-02-19T21:15:53.410563+0000","last_fullsized":"2024-02-19T21:15:53.410563+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:25.616258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:25.616258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":689,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404336+0000","last_change":"2024-02-19T21:15:39.981076+0000","last_active":"2024-02-19T21:15:53.404336+0000","last_peered":"2024-02-19T21:15:53.404336+0000","last_clean":"2024-02-19T21:15:53.404336+0000","last_became_active":"2024-02-19T21:14:41.403970+0000","last_became_peered":"2024-02-19T21:14:41.403970+0000","last_unstale":"2024-02-19T21:15:53.404336+0000","last_undegraded":"2024-02-19T21:15:53.404336+0000","last_fullsized":"2024-02-19T21:15:53.404336+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:39.980983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:39.980983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":515,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.409215+0000","last_change":"2024-02-19T21:14:41.391013+0000","last_active":"2024-02-19T21:15:53.409215+0000","last_peered":"2024-02-19T21:15:53.409215+0000","last_clean":"2024-02-19T21:15:53.409215+0000","last_became_active":"2024-02-19T21:14:41.390659+0000","last_became_peered":"2024-02-19T21:14:41.390659+0000","last_unstale":"2024-02-19T21:15:53.409215+0000","last_undegraded":"2024-02-19T21:15:53.409215+0000","last_fullsized":"2024-02-19T21:15:53.409215+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:43.109579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:13:43.109579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":661,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.415306+0000","last_change":"2024-02-19T21:15:46.226483+0000","last_active":"2024-02-19T21:15:53.415306+0000","last_peered":"2024-02-19T21:15:53.415306+0000","last_clean":"2024-02-19T21:15:53.415306+0000","last_became_active":"2024-02-19T21:14:41.404407+0000","last_became_peered":"2024-02-19T21:14:41.404407+0000","last_unstale":"2024-02-19T21:15:53.415306+0000","last_undegraded":"2024-02-19T21:15:53.415306+0000","last_fullsized":"2024-02-19T21:15:53.415306+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:46.226419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:46.226419+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"459'7","reported_seq":291,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405178+0000","last_change":"2024-02-19T21:12:02.521592+0000","last_active":"2024-02-19T21:15:53.405178+0000","last_peered":"2024-02-19T21:15:53.405178+0000","last_clean":"2024-02-19T21:15:53.405178+0000","last_became_active":"2024-02-19T21:10:41.297295+0000","last_became_peered":"2024-02-19T21:10:41.297295+0000","last_unstale":"2024-02-19T21:15:53.405178+0000","last_undegraded":"2024-02-19T21:15:53.405178+0000","last_fullsized":"2024-02-19T21:15:53.405178+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'7","last_scrub_stamp":"2024-02-19T21:12:02.521523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:02.521523+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":708,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410736+0000","last_change":"2024-02-19T21:14:49.850583+0000","last_active":"2024-02-19T21:15:53.410736+0000","last_peered":"2024-02-19T21:15:53.410736+0000","last_clean":"2024-02-19T21:15:53.410736+0000","last_became_active":"2024-02-19T21:14:41.390588+0000","last_became_peered":"2024-02-19T21:14:41.390588+0000","last_unstale":"2024-02-19T21:15:53.410736+0000","last_undegraded":"2024-02-19T21:15:53.410736+0000","last_fullsized":"2024-02-19T21:15:53.410736+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:49.850506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:49.850506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":686,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410386+0000","last_change":"2024-02-19T21:14:54.900146+0000","last_active":"2024-02-19T21:15:53.410386+0000","last_peered":"2024-02-19T21:15:53.410386+0000","last_clean":"2024-02-19T21:15:53.410386+0000","last_became_active":"2024-02-19T21:10:41.307400+0000","last_became_peered":"2024-02-19T21:10:41.307400+0000","last_unstale":"2024-02-19T21:15:53.410386+0000","last_undegraded":"2024-02-19T21:15:53.410386+0000","last_fullsized":"2024-02-19T21:15:53.410386+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:54.900062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:54.900062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":692,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403126+0000","last_change":"2024-02-19T21:15:47.947147+0000","last_active":"2024-02-19T21:15:53.403126+0000","last_peered":"2024-02-19T21:15:53.403126+0000","last_clean":"2024-02-19T21:15:53.403126+0000","last_became_active":"2024-02-19T21:08:04.266100+0000","last_became_peered":"2024-02-19T21:08:04.266100+0000","last_unstale":"2024-02-19T21:15:53.403126+0000","last_undegraded":"2024-02-19T21:15:53.403126+0000","last_fullsized":"2024-02-19T21:15:53.403126+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:47.947061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:47.947061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":551,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405332+0000","last_change":"2024-02-19T21:14:41.403716+0000","last_active":"2024-02-19T21:15:53.405332+0000","last_peered":"2024-02-19T21:15:53.405332+0000","last_clean":"2024-02-19T21:15:53.405332+0000","last_became_active":"2024-02-19T21:14:41.403361+0000","last_became_peered":"2024-02-19T21:14:41.403361+0000","last_unstale":"2024-02-19T21:15:53.405332+0000","last_undegraded":"2024-02-19T21:15:53.405332+0000","last_fullsized":"2024-02-19T21:15:53.405332+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:19.904121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:19.904121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"623'19","reported_seq":350,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404725+0000","last_change":"2024-02-19T21:14:21.589221+0000","last_active":"2024-02-19T21:15:53.404725+0000","last_peered":"2024-02-19T21:15:53.404725+0000","last_clean":"2024-02-19T21:15:53.404725+0000","last_became_active":"2024-02-19T21:14:16.383577+0000","last_became_peered":"2024-02-19T21:14:16.383577+0000","last_unstale":"2024-02-19T21:15:53.404725+0000","last_undegraded":"2024-02-19T21:15:53.404725+0000","last_fullsized":"2024-02-19T21:15:53.404725+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"623'19","last_scrub_stamp":"2024-02-19T21:14:21.589138+0000","last_deep_scrub":"623'19","last_deep_scrub_stamp":"2024-02-19T21:14:21.589138+0000","last_clean_scrub_stamp":"2024-02-19T21:14:21.589138+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":439,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.410191+0000","last_change":"2024-02-19T21:15:00.101653+0000","last_active":"2024-02-19T21:15:53.410191+0000","last_peered":"2024-02-19T21:15:53.410191+0000","last_clean":"2024-02-19T21:15:53.410191+0000","last_became_active":"2024-02-19T21:14:41.400532+0000","last_became_peered":"2024-02-19T21:14:41.400532+0000","last_unstale":"2024-02-19T21:15:53.410191+0000","last_undegraded":"2024-02-19T21:15:53.410191+0000","last_fullsized":"2024-02-19T21:15:53.410191+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:00.101587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:00.101587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":667,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.409937+0000","last_change":"2024-02-19T21:13:42.086086+0000","last_active":"2024-02-19T21:15:53.409937+0000","last_peered":"2024-02-19T21:15:53.409937+0000","last_clean":"2024-02-19T21:15:53.409937+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:15:53.409937+0000","last_undegraded":"2024-02-19T21:15:53.409937+0000","last_fullsized":"2024-02-19T21:15:53.409937+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:42.086006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:13:42.086006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":287,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.405851+0000","last_change":"2024-02-19T21:15:53.080318+0000","last_active":"2024-02-19T21:15:53.405851+0000","last_peered":"2024-02-19T21:15:53.405851+0000","last_clean":"2024-02-19T21:15:53.405851+0000","last_became_active":"2024-02-19T21:10:41.309550+0000","last_became_peered":"2024-02-19T21:10:41.309550+0000","last_unstale":"2024-02-19T21:15:53.405851+0000","last_undegraded":"2024-02-19T21:15:53.405851+0000","last_fullsized":"2024-02-19T21:15:53.405851+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.080222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.080222+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":659,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403898+0000","last_change":"2024-02-19T21:14:55.013570+0000","last_active":"2024-02-19T21:15:53.403898+0000","last_peered":"2024-02-19T21:15:53.403898+0000","last_clean":"2024-02-19T21:15:53.403898+0000","last_became_active":"2024-02-19T21:14:41.403430+0000","last_became_peered":"2024-02-19T21:14:41.403430+0000","last_unstale":"2024-02-19T21:15:53.403898+0000","last_undegraded":"2024-02-19T21:15:53.403898+0000","last_fullsized":"2024-02-19T21:15:53.403898+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:55.013484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:55.013484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"459'27","reported_seq":507,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403421+0000","last_change":"2024-02-19T21:14:41.530133+0000","last_active":"2024-02-19T21:15:53.403421+0000","last_peered":"2024-02-19T21:15:53.403421+0000","last_clean":"2024-02-19T21:15:53.403421+0000","last_became_active":"2024-02-19T21:14:41.402686+0000","last_became_peered":"2024-02-19T21:14:41.402686+0000","last_unstale":"2024-02-19T21:15:53.403421+0000","last_undegraded":"2024-02-19T21:15:53.403421+0000","last_fullsized":"2024-02-19T21:15:53.403421+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'27","last_scrub_stamp":"2024-02-19T21:14:19.968192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:19.968192+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":555,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.412776+0000","last_change":"2024-02-19T21:14:41.391255+0000","last_active":"2024-02-19T21:15:53.412776+0000","last_peered":"2024-02-19T21:15:53.412776+0000","last_clean":"2024-02-19T21:15:53.412776+0000","last_became_active":"2024-02-19T21:14:41.390887+0000","last_became_peered":"2024-02-19T21:14:41.390887+0000","last_unstale":"2024-02-19T21:15:53.412776+0000","last_undegraded":"2024-02-19T21:15:53.412776+0000","last_fullsized":"2024-02-19T21:15:53.412776+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:52.779962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:52.779962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":305,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.404721+0000","last_change":"2024-02-19T21:14:55.609881+0000","last_active":"2024-02-19T21:15:53.404721+0000","last_peered":"2024-02-19T21:15:53.404721+0000","last_clean":"2024-02-19T21:15:53.404721+0000","last_became_active":"2024-02-19T21:14:16.384069+0000","last_became_peered":"2024-02-19T21:14:16.384069+0000","last_unstale":"2024-02-19T21:15:53.404721+0000","last_undegraded":"2024-02-19T21:15:53.404721+0000","last_fullsized":"2024-02-19T21:15:53.404721+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:55.609805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:55.609805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":439,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.403662+0000","last_change":"2024-02-19T21:14:16.385139+0000","last_active":"2024-02-19T21:15:53.403662+0000","last_peered":"2024-02-19T21:15:53.403662+0000","last_clean":"2024-02-19T21:15:53.403662+0000","last_became_active":"2024-02-19T21:14:16.384527+0000","last_became_peered":"2024-02-19T21:14:16.384527+0000","last_unstale":"2024-02-19T21:15:53.403662+0000","last_undegraded":"2024-02-19T21:15:53.403662+0000","last_fullsized":"2024-02-19T21:15:53.403662+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:40.304490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:40.304490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":97,"reported_epoch":704,"state":"active+clean","last_fresh":"2024-02-19T21:15:53.408611+0000","last_change":"2024-02-19T21:15:46.084809+0000","last_active":"2024-02-19T21:15:53.408611+0000","last_peered":"2024-02-19T21:15:53.408611+0000","last_clean":"2024-02-19T21:15:53.408611+0000","last_became_active":"2024-02-19T21:14:16.383734+0000","last_became_peered":"2024-02-19T21:14:16.383734+0000","last_unstale":"2024-02-19T21:15:53.408611+0000","last_undegraded":"2024-02-19T21:15:53.408611+0000","last_fullsized":"2024-02-19T21:15:53.408611+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:46.084709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:46.084709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":246,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":73,"num_objects":15,"num_object_clones":1,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":16,"num_read_kb":4,"num_write":31,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":64,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":24,"acting":24,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1102,"num_read_kb":859,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":9648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":724,"ondisk_log_size":724,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":868,"num_read_kb":863,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":126,"acting":126,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":20,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266866,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303656,"kb_used_data":2336,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441496,"statfs":{"total":95995035648,"available":95684091904,"internally_reserved":0,"allocated":2392064,"data_stored":893986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.50900000000000001,"15min":0.86499999999999999},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.215},"max":{"1min":0.68600000000000005,"5min":0.79200000000000004,"15min":89.456999999999994},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.50600000000000001,"15min":0.78900000000000003},"min":{"1min":0.32000000000000001,"5min":0.29599999999999999,"15min":0.25900000000000001},"max":{"1min":0.70999999999999996,"5min":0.81299999999999994,"15min":70.451999999999998},"last":0.26100000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.69899999999999995},"min":{"1min":0.376,"5min":0.29999999999999999,"15min":0.28199999999999997},"max":{"1min":0.66600000000000004,"5min":0.79100000000000004,"15min":34.273000000000003},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51500000000000001,"15min":0.93500000000000005},"min":{"1min":0.35899999999999999,"5min":0.29499999999999998,"15min":0.254},"max":{"1min":0.69199999999999995,"5min":0.97899999999999998,"15min":89.594999999999999},"last":0.71999999999999997}]},{"osd":2,"last update":"Mon Feb 19 21:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.57199999999999995,"15min":0.96299999999999997},"min":{"1min":0.34300000000000003,"5min":0.29099999999999998,"15min":0.24299999999999999},"max":{"1min":2.8100000000000001,"5min":4.3769999999999998,"15min":104.413},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.54500000000000004,"15min":0.95699999999999996},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.27600000000000002},"max":{"1min":0.71199999999999997,"5min":1.8480000000000001,"15min":104.268},"last":0.36599999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.56200000000000006,"15min":0.96599999999999997},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.254},"max":{"1min":0.69799999999999995,"5min":2.9769999999999999,"15min":105.3},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55200000000000005,"15min":0.61699999999999999},"min":{"1min":0.39500000000000002,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.65300000000000002,"5min":2.3050000000000002,"15min":13.243},"last":0.39000000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.45800000000000002,"15min":0.45500000000000002},"min":{"1min":0.35899999999999999,"5min":0.28999999999999998,"15min":0.24299999999999999},"max":{"1min":0.64700000000000002,"5min":0.68799999999999994,"15min":1.373},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.495,"15min":0.47999999999999998},"min":{"1min":0.39200000000000002,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":1.0740000000000001},"last":0.42899999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.46000000000000002,"15min":0.45800000000000002},"min":{"1min":0.35499999999999998,"5min":0.29099999999999998,"15min":0.252},"max":{"1min":0.70599999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.51400000000000001,"15min":0.50700000000000001},"min":{"1min":0.40600000000000003,"5min":0.32600000000000001,"15min":0.27100000000000002},"max":{"1min":0.75600000000000001,"5min":0.79300000000000004,"15min":0.85899999999999999},"last":0.625}]},{"osd":6,"last update":"Mon Feb 19 21:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.53900000000000003,"15min":0.51900000000000002},"min":{"1min":0.42399999999999999,"5min":0.30199999999999999,"15min":0.28799999999999998},"max":{"1min":0.753,"5min":0.76700000000000002,"15min":1.0049999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.56799999999999995,"15min":0.55100000000000005},"min":{"1min":0.432,"5min":0.34999999999999998,"15min":0.33700000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":1.262},"last":0.56699999999999995}]}]},{"osd":6,"up_from":45,"seq":193273528504,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311836,"kb_used_data":2324,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93433316,"statfs":{"total":95995035648,"available":95675715584,"internally_reserved":0,"allocated":2379776,"data_stored":908605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47399999999999998,"15min":0.60999999999999999},"min":{"1min":0.249,"5min":0.24099999999999999,"15min":0.19900000000000001},"max":{"1min":0.73099999999999998,"5min":0.80300000000000005,"15min":17.486999999999998},"last":0.432},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.49099999999999999,"15min":0.65500000000000003},"min":{"1min":0.30299999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.71199999999999997,"5min":1.669,"15min":17.43},"last":0.38800000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.499,"15min":0.62},"min":{"1min":0.248,"5min":0.248,"15min":0.22800000000000001},"max":{"1min":0.67400000000000004,"5min":1.6599999999999999,"15min":14.692},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.51000000000000001,"15min":0.66600000000000004},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.23400000000000001},"max":{"1min":0.69099999999999995,"5min":1.1990000000000001,"15min":14.795},"last":0.31900000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52300000000000002,"15min":0.60699999999999998},"min":{"1min":0.34100000000000003,"5min":0.253,"15min":0.253},"max":{"1min":0.72199999999999998,"5min":3.3050000000000002,"15min":6.931},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.50800000000000001,"15min":0.67400000000000004},"min":{"1min":0.36099999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":1.794,"15min":14.391},"last":0.41699999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52000000000000002,"15min":0.70199999999999996},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66800000000000004,"5min":1.4450000000000001,"15min":14.510999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.57099999999999995,"15min":0.68100000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.316},"max":{"1min":4.085,"5min":4.085,"15min":17.585000000000001},"last":0.40699999999999997}]},{"osd":4,"last update":"Mon Feb 19 21:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45600000000000002,"15min":0.42499999999999999},"min":{"1min":0.26800000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.55100000000000005,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.496,"15min":0.45200000000000001},"min":{"1min":0.29299999999999998,"5min":0.253,"15min":0.22700000000000001},"max":{"1min":0.68999999999999995,"5min":3.431,"15min":3.431},"last":0.36399999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.50800000000000001,"15min":0.47399999999999998},"min":{"1min":0.318,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60299999999999998,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.52800000000000002,"15min":0.505},"min":{"1min":0.34599999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71399999999999997,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.46100000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:15:35 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.504,"15min":0.496},"min":{"1min":0.29999999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.66500000000000004,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.51500000000000001,"15min":0.501},"min":{"1min":0.312,"5min":0.312,"15min":0.28399999999999997},"max":{"1min":0.67700000000000005,"5min":1.835,"15min":1.835},"last":0.442}]}]},{"osd":1,"up_from":13,"seq":55834575055,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299564,"kb_used_data":2340,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445588,"statfs":{"total":95995035648,"available":95688282112,"internally_reserved":0,"allocated":2396160,"data_stored":889113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.33100000000000002,"15min":0.54100000000000004},"min":{"1min":0.254,"5min":0.17999999999999999,"15min":0.16700000000000001},"max":{"1min":0.46100000000000002,"5min":2.0379999999999998,"15min":27.777000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.35299999999999998,"15min":0.57399999999999995},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.19400000000000001},"max":{"1min":0.56299999999999994,"5min":1.9870000000000001,"15min":27.631},"last":0.27000000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:15:41 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.36399999999999999,"15min":0.69299999999999995},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19600000000000001},"max":{"1min":0.50600000000000001,"5min":2.0230000000000001,"15min":27.798999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.38,"15min":0.70799999999999996},"min":{"1min":0.253,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.46200000000000002,"5min":2.089,"15min":27.606999999999999},"last":0.33700000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.36899999999999999,"15min":0.55600000000000005},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":0.52600000000000002,"5min":1.7450000000000001,"15min":15.574999999999999},"last":0.309},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.40799999999999997,"15min":0.65200000000000002},"min":{"1min":0.312,"5min":0.28999999999999998,"15min":0.25900000000000001},"max":{"1min":0.54200000000000004,"5min":2.0070000000000001,"15min":27.82},"last":0.29499999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:15:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.52700000000000002,"15min":0.64800000000000002},"min":{"1min":0.38600000000000001,"5min":0.27800000000000002,"15min":0.26800000000000002},"max":{"1min":0.80300000000000005,"5min":1.149,"15min":9.6430000000000007},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55700000000000005,"15min":0.751},"min":{"1min":0.41199999999999998,"5min":0.34799999999999998,"15min":0.308},"max":{"1min":0.77400000000000002,"5min":1.9770000000000001,"15min":27.707000000000001},"last":0.45600000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:15:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55200000000000005,"15min":0.73399999999999999},"min":{"1min":0.315,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75,"5min":1.048,"15min":27.529},"last":0.441},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53100000000000003,"15min":0.66300000000000003},"min":{"1min":0.35299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.69499999999999995,"5min":0.98099999999999998,"15min":9.6799999999999997},"last":0.42699999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53100000000000003,"15min":0.65300000000000002},"min":{"1min":0.38200000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84699999999999998,"5min":1.1659999999999999,"15min":9.6989999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59899999999999998,"15min":0.77900000000000003},"min":{"1min":0.38500000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81200000000000006,"5min":3.4359999999999999,"15min":27.733000000000001},"last":0.433}]},{"osd":7,"last update":"Mon Feb 19 21:15:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58999999999999997,"15min":0.77400000000000002},"min":{"1min":0.41699999999999998,"5min":0.371,"15min":0.36699999999999999},"max":{"1min":0.78200000000000003,"5min":3.4670000000000001,"15min":27.576000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59299999999999997,"15min":0.80500000000000005},"min":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.375},"max":{"1min":0.72599999999999998,"5min":1.1299999999999999,"15min":27.756},"last":0.40799999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738581,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307740,"kb_used_data":2324,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437412,"statfs":{"total":95995035648,"available":95679909888,"internally_reserved":0,"allocated":2379776,"data_stored":895441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:15:15 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.35199999999999998,"15min":0.624},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.188},"max":{"1min":0.46700000000000003,"5min":1.855,"15min":20.966999999999999},"last":0.252},{"interface":"front","average":{"1min":0.316,"5min":0.35899999999999999,"15min":0.63200000000000001},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.193},"max":{"1min":0.52500000000000002,"5min":1.613,"15min":20.366},"last":0.312}]},{"osd":2,"last update":"Mon Feb 19 21:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.35799999999999998,"15min":0.70999999999999996},"min":{"1min":0.23799999999999999,"5min":0.214,"15min":0.19900000000000001},"max":{"1min":0.46400000000000002,"5min":0.93600000000000005,"15min":27.916},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.373,"15min":0.63300000000000001},"min":{"1min":0.25700000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.498,"5min":0.58599999999999997,"15min":22.004000000000001},"last":0.42999999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:15:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.39400000000000002,"15min":0.73799999999999999},"min":{"1min":0.309,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.68500000000000005,"5min":0.96799999999999997,"15min":25.669},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.40999999999999998,"15min":0.68500000000000005},"min":{"1min":0.313,"5min":0.26800000000000002,"15min":0.26300000000000001},"max":{"1min":0.66800000000000004,"5min":0.77200000000000002,"15min":25.006},"last":0.35699999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:15:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.73099999999999998},"min":{"1min":0.39800000000000002,"5min":0.32500000000000001,"15min":0.28799999999999998},"max":{"1min":0.76400000000000001,"5min":1.0089999999999999,"15min":20.411999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56499999999999995,"15min":0.76000000000000001},"min":{"1min":0.39100000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.76100000000000001,"5min":2.093,"15min":19.986999999999998},"last":0.45600000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57199999999999995,"15min":0.80400000000000005},"min":{"1min":0.36199999999999999,"5min":0.31900000000000001,"15min":0.27200000000000002},"max":{"1min":0.93100000000000005,"5min":1.3560000000000001,"15min":20.021999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.59599999999999997,"15min":0.81100000000000005},"min":{"1min":0.438,"5min":0.35499999999999998,"15min":0.30399999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":20.908999999999999},"last":0.505}]},{"osd":6,"last update":"Mon Feb 19 21:15:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58299999999999996,"15min":0.75700000000000001},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.33900000000000002},"max":{"1min":0.82799999999999996,"5min":2.1459999999999999,"15min":20.445},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.59699999999999998,"15min":0.79700000000000004},"min":{"1min":0.40699999999999997,"5min":0.35099999999999998,"15min":0.33400000000000002},"max":{"1min":1.4370000000000001,"5min":2.6120000000000001,"15min":20.047999999999998},"last":0.56499999999999995}]},{"osd":7,"last update":"Mon Feb 19 21:15:34 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.58999999999999997,"15min":0.80600000000000005},"min":{"1min":0.47999999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.79300000000000004,"5min":0.84499999999999997,"15min":20.940999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59699999999999998,"15min":0.81100000000000005},"min":{"1min":0.44700000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73699999999999999,"5min":1.0660000000000001,"15min":20.471},"last":0.52200000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411530,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311836,"kb_used_data":2324,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93433316,"statfs":{"total":95995035648,"available":95675715584,"internally_reserved":0,"allocated":2379776,"data_stored":896017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.0299999999999998,"5min":0.67600000000000005,"15min":0.64700000000000002},"min":{"1min":0.222,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":28.702999999999999,"5min":28.702999999999999,"15min":28.702999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":2.0270000000000001,"5min":0.68100000000000005,"15min":0.70399999999999996},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":28.744,"5min":28.744,"15min":28.744},"last":0.251}]},{"osd":1,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.38600000000000001,"15min":0.71899999999999997},"min":{"1min":0.22600000000000001,"5min":0.20699999999999999,"15min":0.20100000000000001},"max":{"1min":3.9020000000000001,"5min":3.9020000000000001,"15min":28.882000000000001},"last":0.218},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.42299999999999999,"15min":0.69399999999999995},"min":{"1min":0.23899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":5.0739999999999998,"5min":5.0739999999999998,"15min":28.382999999999999},"last":0.309}]},{"osd":3,"last update":"Mon Feb 19 21:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.38900000000000001,"15min":0.70199999999999996},"min":{"1min":0.28499999999999998,"5min":0.27100000000000002,"15min":0.23499999999999999},"max":{"1min":0.51200000000000001,"5min":0.68899999999999995,"15min":28.742000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.44500000000000001,"15min":0.63500000000000001},"min":{"1min":0.29199999999999998,"5min":0.26200000000000001,"15min":0.25700000000000001},"max":{"1min":0.56399999999999995,"5min":4.4550000000000001,"15min":26.719999999999999},"last":0.33200000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55200000000000005,"15min":0.68600000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.999,"5min":0.999,"15min":24.356000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56799999999999995,"15min":0.70399999999999996},"min":{"1min":0.36399999999999999,"5min":0.33700000000000002,"15min":0.32400000000000001},"max":{"1min":0.83099999999999996,"5min":1.2010000000000001,"15min":20.707000000000001},"last":0.376}]},{"osd":5,"last update":"Mon Feb 19 21:14:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.55300000000000005,"15min":0.66700000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.33500000000000002},"max":{"1min":0.81399999999999995,"5min":0.99399999999999999,"15min":9.9499999999999993},"last":0.434},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57899999999999996,"15min":0.78000000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80400000000000005,"5min":1.135,"15min":26.606000000000002},"last":0.39500000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:15:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58499999999999996,"15min":0.79200000000000004},"min":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.34300000000000003},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":24.381},"last":0.442},{"interface":"front","average":{"1min":0.623,"5min":0.59499999999999997,"15min":0.73099999999999998},"min":{"1min":0.45000000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.85699999999999998,"5min":0.94999999999999996,"15min":28.706},"last":0.45200000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60399999999999998,"15min":0.78300000000000003},"min":{"1min":0.45700000000000002,"5min":0.42699999999999999,"15min":0.34200000000000003},"max":{"1min":1.05,"5min":1.05,"15min":26.658000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.61499999999999999,"15min":0.73699999999999999},"min":{"1min":0.40400000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":24.402000000000001},"last":0.41099999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182598,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311848,"kb_used_data":2336,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93433304,"statfs":{"total":95995035648,"available":95675703296,"internally_reserved":0,"allocated":2392064,"data_stored":891831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:15:01 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34599999999999997,"15min":0.71299999999999997},"min":{"1min":0.254,"5min":0.192,"15min":0.156},"max":{"1min":0.70199999999999996,"5min":1.5449999999999999,"15min":77.662000000000006},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.35399999999999998,"15min":0.48699999999999999},"min":{"1min":0.251,"5min":0.20899999999999999,"15min":0.19700000000000001},"max":{"1min":0.54900000000000004,"5min":0.60999999999999999,"15min":17.456},"last":0.443}]},{"osd":1,"last update":"Mon Feb 19 21:15:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.35799999999999998,"15min":0.52000000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.22800000000000001},"max":{"1min":0.57999999999999996,"5min":0.67600000000000005,"15min":17.327999999999999},"last":0.376},{"interface":"front","average":{"1min":0.371,"5min":0.36499999999999999,"15min":0.502},"min":{"1min":0.24099999999999999,"5min":0.20699999999999999,"15min":0.20000000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":15.004},"last":0.32700000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:15:01 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38100000000000001,"15min":0.65600000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":0.56599999999999995,"5min":0.92600000000000005,"15min":56.890999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.38100000000000001,"15min":0.64100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.20899999999999999},"max":{"1min":0.81200000000000006,"5min":1.3200000000000001,"15min":44.600999999999999},"last":0.29699999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:15:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53400000000000003,"15min":0.63400000000000001},"min":{"1min":0.41699999999999998,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":0.82599999999999996,"5min":1.3340000000000001,"15min":12.847},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53200000000000003,"15min":0.70199999999999996},"min":{"1min":0.42299999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.77500000000000002,"5min":1.3560000000000001,"15min":26.393999999999998},"last":0.36699999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54500000000000004,"15min":0.67900000000000005},"min":{"1min":0.47499999999999998,"5min":0.33200000000000002,"15min":0.313},"max":{"1min":0.82899999999999996,"5min":0.95699999999999996,"15min":17.401},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55500000000000005,"15min":0.65600000000000003},"min":{"1min":0.435,"5min":0.35999999999999999,"15min":0.313},"max":{"1min":0.871,"5min":1.397,"15min":12.882},"last":0.39200000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:15:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55200000000000005,"15min":0.65200000000000002},"min":{"1min":0.39100000000000001,"5min":0.36899999999999999,"15min":0.34300000000000003},"max":{"1min":0.83999999999999997,"5min":0.90500000000000003,"15min":17.367000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55900000000000005,"15min":0.66000000000000003},"min":{"1min":0.41299999999999998,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":0.69699999999999995,"5min":1.3660000000000001,"15min":12.904},"last":0.45500000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:15:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.65700000000000003},"min":{"1min":0.42199999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.91500000000000004,"5min":1.038,"15min":17.428999999999998},"last":0.626},{"interface":"front","average":{"1min":0.629,"5min":0.58899999999999997,"15min":0.80700000000000005},"min":{"1min":0.45000000000000001,"5min":0.374,"15min":0.34499999999999997},"max":{"1min":1.1859999999999999,"5min":1.3879999999999999,"15min":44.378},"last":0.42199999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986370,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299560,"kb_used_data":2336,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445592,"statfs":{"total":95995035648,"available":95688286208,"internally_reserved":0,"allocated":2392064,"data_stored":904132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56899999999999995,"15min":0.627},"min":{"1min":0.32500000000000001,"5min":0.28199999999999997,"15min":0.22500000000000001},"max":{"1min":0.66400000000000003,"5min":8.0530000000000008,"15min":19.878},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.85599999999999998},"min":{"1min":0.32900000000000001,"5min":0.26700000000000002,"15min":0.20499999999999999},"max":{"1min":0.67500000000000004,"5min":3.3530000000000002,"15min":70.063000000000002},"last":0.52600000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.503,"15min":0.63300000000000001},"min":{"1min":0.33800000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.71899999999999997,"5min":1.6499999999999999,"15min":26.510999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.502,"15min":0.67900000000000005},"min":{"1min":0.38500000000000001,"5min":0.32100000000000001,"15min":0.23200000000000001},"max":{"1min":0.63,"5min":0.89700000000000002,"15min":25.949000000000002},"last":0.57499999999999996}]},{"osd":2,"last update":"Mon Feb 19 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.58699999999999997,"15min":0.72099999999999997},"min":{"1min":0.376,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71399999999999997,"5min":8.1240000000000006,"15min":37.320999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53400000000000003,"15min":1.0509999999999999},"min":{"1min":0.33000000000000002,"5min":0.30599999999999999,"15min":0.253},"max":{"1min":0.66800000000000004,"5min":2.9969999999999999,"15min":69.936999999999998},"last":0.68100000000000005}]},{"osd":3,"last update":"Mon Feb 19 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54200000000000004,"15min":0.91000000000000003},"min":{"1min":0.38800000000000001,"5min":0.28999999999999998,"15min":0.27700000000000002},"max":{"1min":0.66900000000000004,"5min":3.323,"15min":69.793000000000006},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.53800000000000003,"15min":0.75700000000000001},"min":{"1min":0.33800000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70499999999999996,"5min":1.621,"15min":17.986999999999998},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Feb 19 21:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44600000000000001,"15min":0.42699999999999999},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.218},"max":{"1min":0.65800000000000003,"5min":1.532,"15min":3.2799999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.49099999999999999,"15min":0.47599999999999998},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.23899999999999999},"max":{"1min":0.76100000000000001,"5min":1.5700000000000001,"15min":4.4340000000000002},"last":0.33500000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.55500000000000005,"15min":0.48799999999999999},"min":{"1min":0.30599999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63,"5min":8.0969999999999995,"15min":8.0969999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.57699999999999996,"15min":0.53200000000000003},"min":{"1min":0.34300000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71099999999999997,"5min":3.0270000000000001,"15min":4.5609999999999999},"last":0.40500000000000003}]},{"osd":7,"last update":"Mon Feb 19 21:15:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55800000000000005,"15min":0.53300000000000003},"min":{"1min":0.374,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.77100000000000002,"5min":3.2909999999999999,"15min":6.0730000000000004},"last":0.38},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55300000000000005,"15min":0.53300000000000003},"min":{"1min":0.313,"5min":0.309,"15min":0.26400000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":3.202},"last":0.41999999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790141,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303624,"kb_used_data":2304,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441528,"statfs":{"total":95995035648,"available":95684124672,"internally_reserved":0,"allocated":2359296,"data_stored":884428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.47599999999999998,"15min":1.8720000000000001},"min":{"1min":0.34799999999999998,"5min":0.23499999999999999,"15min":0.19500000000000001},"max":{"1min":0.748,"5min":0.748,"15min":276.47899999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.49199999999999999,"15min":0.89300000000000002},"min":{"1min":0.38,"5min":0.27400000000000002,"15min":0.20899999999999999},"max":{"1min":0.70399999999999996,"5min":0.70599999999999996,"15min":52.887},"last":0.434}]},{"osd":1,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.46500000000000002,"15min":1.2569999999999999},"min":{"1min":0.40799999999999997,"5min":0.28199999999999997,"15min":0.23499999999999999},"max":{"1min":0.67700000000000005,"5min":0.77100000000000002,"15min":120.011},"last":0.502},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51200000000000001,"15min":1.0189999999999999},"min":{"1min":0.39600000000000002,"5min":0.28599999999999998,"15min":0.26200000000000001},"max":{"1min":0.68200000000000005,"5min":1.5029999999999999,"15min":118.105},"last":0.46500000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.497,"15min":0.98499999999999999},"min":{"1min":0.41499999999999998,"5min":0.32300000000000001,"15min":0.27200000000000002},"max":{"1min":0.73099999999999998,"5min":0.92300000000000004,"15min":82.242000000000004},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.54300000000000004,"15min":1.5109999999999999},"min":{"1min":0.50800000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.79000000000000004,"5min":2.9820000000000002,"15min":232.84999999999999},"last":0.47199999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.52600000000000002,"15min":1.0669999999999999},"min":{"1min":0.44700000000000001,"5min":0.32300000000000001,"15min":0.27900000000000003},"max":{"1min":0.76200000000000001,"5min":2.2210000000000001,"15min":109.70099999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.54200000000000004,"15min":0.90700000000000003},"min":{"1min":0.51600000000000001,"5min":0.32700000000000001,"15min":0.32000000000000001},"max":{"1min":0.70899999999999996,"5min":1.4590000000000001,"15min":65.519000000000005},"last":0.39500000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.44600000000000001,"15min":0.438},"min":{"1min":0.28699999999999998,"5min":0.26400000000000001,"15min":0.23300000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":1.861},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.46400000000000002,"15min":0.45500000000000002},"min":{"1min":0.313,"5min":0.255,"15min":0.248},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":1.9730000000000001},"last":0.41699999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.46400000000000002,"15min":0.46000000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.26400000000000001},"max":{"1min":0.622,"5min":0.871,"15min":1.823},"last":0.374},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.46899999999999997,"15min":0.46999999999999997},"min":{"1min":0.32700000000000001,"5min":0.30399999999999999,"15min":0.27100000000000002},"max":{"1min":0.68999999999999995,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.49199999999999999}]},{"osd":7,"last update":"Mon Feb 19 21:15:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.505,"15min":0.505},"min":{"1min":0.34699999999999998,"5min":0.30199999999999999,"15min":0.28499999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":1.8879999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.52400000000000002,"15min":0.51300000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.27800000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.96799999999999997},"last":0.45500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":246,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":246,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":246,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":246,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":246,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":246,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":246,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:16:01.461 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:16:01.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:00.724336+0000 mgr.y (mgr.14550) 599 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:01.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:01.402051+0000 mon.a (mon.0) 3448 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-02-19T21:16:01.729 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:01.403253+0000 mon.a (mon.0) 3449 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:16:01.992 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:00.724336+0000 mgr.y (mgr.14550) 599 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:01.992 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:01.402051+0000 mon.a (mon.0) 3448 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-02-19T21:16:01.992 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:01.403253+0000 mon.a (mon.0) 3449 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:16:01.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:00.724336+0000 mgr.y (mgr.14550) 599 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:01.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:01.402051+0000 mon.a (mon.0) 3448 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2024-02-19T21:16:01.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:01.403253+0000 mon.a (mon.0) 3449 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:16:02.127 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:02.130 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:02.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:02.231 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:16:02.240 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:16:02.276 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:16:02.277 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:02.281 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:16:02.284 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:02.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:16:02.385 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:02.421 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:02.727 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:02.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:02.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:02.835 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:02.862 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:02.900 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:01.313883+0000 mgr.y (mgr.14550) 600 : cluster [DBG] pgmap v1073: 182 pgs: 182 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-19T21:16:02.900 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:02.400961+0000 mon.a (mon.0) 3450 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:16:02.900 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:02.401127+0000 mon.a (mon.0) 3451 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-02-19T21:16:02.900 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:02.403398+0000 mon.a (mon.0) 3452 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:16:02.920 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-19T21:16:02.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:01.313883+0000 mgr.y (mgr.14550) 600 : cluster [DBG] pgmap v1073: 182 pgs: 182 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-19T21:16:02.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:02.400961+0000 mon.a (mon.0) 3450 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:16:02.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:02.401127+0000 mon.a (mon.0) 3451 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-02-19T21:16:02.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:02.403398+0000 mon.a (mon.0) 3452 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:16:02.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:01.313883+0000 mgr.y (mgr.14550) 600 : cluster [DBG] pgmap v1073: 182 pgs: 182 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-19T21:16:02.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:02.400961+0000 mon.a (mon.0) 3450 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:16:02.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:02.401127+0000 mon.a (mon.0) 3451 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2024-02-19T21:16:02.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:02.403398+0000 mon.a (mon.0) 3452 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:16:03.175 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:03.188 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:16:03.705 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:16:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:16:03] "GET /metrics HTTP/1.1" 200 192236 "" "Prometheus/2.33.4" 2024-02-19T21:16:03.862 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:16:04.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.899+0000 7fe4219a1700 1 -- 172.21.15.163:0/1826702356 >> v1:172.21.15.163:6790/0 conn(0x7fe41c0f8c50 legacy=0x7fe41c0f6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:04.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.899+0000 7fe4219a1700 1 -- 172.21.15.163:0/1826702356 shutdown_connections 2024-02-19T21:16:04.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.899+0000 7fe4219a1700 1 -- 172.21.15.163:0/1826702356 >> 172.21.15.163:0/1826702356 conn(0x7fe41c06cc00 msgr2=0x7fe41c0f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:04.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.899+0000 7fe4219a1700 1 -- 172.21.15.163:0/1826702356 shutdown_connections 2024-02-19T21:16:04.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.899+0000 7fe4219a1700 1 -- 172.21.15.163:0/1826702356 wait complete. 2024-02-19T21:16:04.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.899+0000 7fe4219a1700 1 librados: starting msgr at 2024-02-19T21:16:04.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.899+0000 7fe4219a1700 1 librados: starting objecter 2024-02-19T21:16:04.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe4219a1700 1 Processor -- start 2024-02-19T21:16:04.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe4219a1700 1 -- start start 2024-02-19T21:16:04.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe4219a1700 1 librados: setting wanted keys 2024-02-19T21:16:04.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe4219a1700 1 librados: calling monclient init 2024-02-19T21:16:04.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe4219a1700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe41c1195e0 con 0x7fe41c0f87b0 2024-02-19T21:16:04.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe4219a1700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe41c11a7c0 con 0x7fe41c1066c0 2024-02-19T21:16:04.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe4219a1700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe41c11b9c0 con 0x7fe41c0f8c50 2024-02-19T21:16:04.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe41affd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fe41c0f87b0 0x7fe41c102cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:34990/0 (socket says 172.21.15.163:34990) 2024-02-19T21:16:04.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe41affd700 1 -- 172.21.15.163:0/3610632511 learned_addr learned my addr 172.21.15.163:0/3610632511 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:04.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe403fff700 10 client.?.objecter ms_handle_connect 0x7fe41c0f87b0 2024-02-19T21:16:04.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe403fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:04.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe403fff700 10 client.?.objecter ms_handle_connect 0x7fe41c0f8c50 2024-02-19T21:16:04.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe403fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:04.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337585696 0 0) 0x7fe41c1195e0 con 0x7fe41c0f87b0 2024-02-19T21:16:04.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe404003030 con 0x7fe41c0f87b0 2024-02-19T21:16:04.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe403fff700 10 client.?.objecter ms_handle_connect 0x7fe41c1066c0 2024-02-19T21:16:04.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe403fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:04.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806265374 0 0) 0x7fe41c11b9c0 con 0x7fe41c0f8c50 2024-02-19T21:16:04.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe41c1195e0 con 0x7fe41c0f8c50 2024-02-19T21:16:04.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812690018 0 0) 0x7fe41c11a7c0 con 0x7fe41c1066c0 2024-02-19T21:16:04.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.900+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe41c11b9c0 con 0x7fe41c1066c0 2024-02-19T21:16:04.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1538878329 0 0) 0x7fe404003030 con 0x7fe41c0f87b0 2024-02-19T21:16:04.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe41c11a7c0 con 0x7fe41c0f87b0 2024-02-19T21:16:04.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe40c002990 con 0x7fe41c0f87b0 2024-02-19T21:16:04.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3452442897 0 0) 0x7fe41c1195e0 con 0x7fe41c0f8c50 2024-02-19T21:16:04.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe404003030 con 0x7fe41c0f8c50 2024-02-19T21:16:04.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1560575044 0 0) 0x7fe41c11b9c0 con 0x7fe41c1066c0 2024-02-19T21:16:04.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe41c1195e0 con 0x7fe41c1066c0 2024-02-19T21:16:04.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe410001e50 con 0x7fe41c0f8c50 2024-02-19T21:16:04.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe414001b00 con 0x7fe41c1066c0 2024-02-19T21:16:04.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 352992500 0 0) 0x7fe41c11a7c0 con 0x7fe41c0f87b0 2024-02-19T21:16:04.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 >> v1:172.21.15.163:6790/0 conn(0x7fe41c0f8c50 legacy=0x7fe41c1033c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:04.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 >> v1:172.21.15.193:6789/0 conn(0x7fe41c1066c0 legacy=0x7fe41c117ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:04.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c11cbc0 con 0x7fe41c0f87b0 2024-02-19T21:16:04.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe4219a1700 1 -- 172.21.15.163:0/3610632511 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41c11bbb0 con 0x7fe41c0f87b0 2024-02-19T21:16:04.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe4219a1700 10 client.23751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:04.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe4219a1700 1 -- 172.21.15.163:0/3610632511 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41c11c0a0 con 0x7fe41c0f87b0 2024-02-19T21:16:04.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe4219a1700 1 librados: init done 2024-02-19T21:16:04.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fe40c002c70 con 0x7fe41c0f87b0 2024-02-19T21:16:04.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.901+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe40c005b50 con 0x7fe41c0f87b0 2024-02-19T21:16:04.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.902+0000 7fe4219a1700 1 -- 172.21.15.163:0/3610632511 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe408004f20 con 0x7fe41c0f87b0 2024-02-19T21:16:04.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.902+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fe40c0016c0 con 0x7fe41c0f87b0 2024-02-19T21:16:04.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.903+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(714..714 src has 623..714) v4 ==== 8678+0+0 (unknown 388929752 0 0) 0x7fe40c078f40 con 0x7fe41c0f87b0 2024-02-19T21:16:04.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.903+0000 7fe403fff700 10 client.23751.objecter ms_dispatch 0x7fe41c001550 osd_map(714..714 src has 623..714) v4 2024-02-19T21:16:04.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.903+0000 7fe403fff700 3 client.23751.objecter handle_osd_map got epochs [714,714] > 0 2024-02-19T21:16:04.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.903+0000 7fe403fff700 3 client.23751.objecter handle_osd_map decoding full epoch 714 2024-02-19T21:16:04.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.903+0000 7fe403fff700 20 client.23751.objecter dump_active .. 0 homeless 2024-02-19T21:16:04.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.903+0000 7fe403fff700 10 client.23751.objecter ms_handle_connect 0x7fe404063c80 2024-02-19T21:16:04.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:03.907+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe40c04c0f0 con 0x7fe41c0f87b0 2024-02-19T21:16:04.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:04.093+0000 7fe4219a1700 1 -- 172.21.15.163:0/3610632511 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fe4080051d0 con 0x7fe41c0f87b0 2024-02-19T21:16:04.312 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:04.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:04.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:04.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:04.405+0000 7fe403fff700 1 -- 172.21.15.163:0/3610632511 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v715) v1 ==== 97+0+0 (unknown 3672201581 0 0) 0x7fe40c04a500 con 0x7fe41c0f87b0 2024-02-19T21:16:04.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:04.407+0000 7fe4219a1700 10 librados: watch_flush enter 2024-02-19T21:16:04.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:04.407+0000 7fe4219a1700 10 librados: watch_flush exit 2024-02-19T21:16:04.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:04.407+0000 7fe4219a1700 20 client.23751.objecter shutdown clearing up homeless session... 2024-02-19T21:16:04.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:04.407+0000 7fe4219a1700 10 client.23751.objecter successfully canceled tick 2024-02-19T21:16:04.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:04.407+0000 7fe4219a1700 1 -- 172.21.15.163:0/3610632511 >> v1:172.21.15.163:6800/1552514681 conn(0x7fe404063c80 legacy=0x7fe404066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:04.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:04.407+0000 7fe4219a1700 1 -- 172.21.15.163:0/3610632511 >> v1:172.21.15.163:6789/0 conn(0x7fe41c0f87b0 legacy=0x7fe41c102cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:04.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:04.407+0000 7fe4219a1700 1 -- 172.21.15.163:0/3610632511 shutdown_connections 2024-02-19T21:16:04.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:04.407+0000 7fe4219a1700 1 -- 172.21.15.163:0/3610632511 >> 172.21.15.163:0/3610632511 conn(0x7fe41c06cc00 msgr2=0x7fe41c106aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:04.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:04.409+0000 7fe4219a1700 1 -- 172.21.15.163:0/3610632511 shutdown_connections 2024-02-19T21:16:04.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:04.409+0000 7fe4219a1700 1 -- 172.21.15.163:0/3610632511 wait complete. 2024-02-19T21:16:04.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:04.409+0000 7fe4219a1700 1 librados: shutdown 2024-02-19T21:16:04.411 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:16:04.416 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:04.444 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:04.624 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:03.314311+0000 mgr.y (mgr.14550) 601 : cluster [DBG] pgmap v1076: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-19T21:16:04.624 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:03.402546+0000 mon.a (mon.0) 3453 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-126", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:16:04.624 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:03.402635+0000 mon.a (mon.0) 3454 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2024-02-19T21:16:04.624 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:03.404798+0000 mon.a (mon.0) 3455 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-126"}]: dispatch 2024-02-19T21:16:04.624 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:04.095367+0000 mon.a (mon.0) 3456 : audit [INF] from='client.? v1:172.21.15.163:0/3610632511' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:16:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:03.314311+0000 mgr.y (mgr.14550) 601 : cluster [DBG] pgmap v1076: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-19T21:16:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:03.402546+0000 mon.a (mon.0) 3453 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-126", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:16:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:03.402635+0000 mon.a (mon.0) 3454 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2024-02-19T21:16:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:03.404798+0000 mon.a (mon.0) 3455 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-126"}]: dispatch 2024-02-19T21:16:04.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:04.095367+0000 mon.a (mon.0) 3456 : audit [INF] from='client.? v1:172.21.15.163:0/3610632511' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:16:04.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:03.314311+0000 mgr.y (mgr.14550) 601 : cluster [DBG] pgmap v1076: 214 pgs: 32 unknown, 182 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-19T21:16:04.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:03.402546+0000 mon.a (mon.0) 3453 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-126", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:16:04.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:03.402635+0000 mon.a (mon.0) 3454 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2024-02-19T21:16:04.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:03.404798+0000 mon.a (mon.0) 3455 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-126"}]: dispatch 2024-02-19T21:16:04.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:04.095367+0000 mon.a (mon.0) 3456 : audit [INF] from='client.? v1:172.21.15.163:0/3610632511' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:16:04.791 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:04.794 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:04.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:04.875 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:16:04.875 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:16:04.895 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:04.926 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:05.091 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:05.126 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:05.129 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:05.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:05.231 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:05.257 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:05.355 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:05.358 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:05.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:05.459 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:16:05.486 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:04.399310+0000 mon.a (mon.0) 3457 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:16:05.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:04.404654+0000 mon.a (mon.0) 3458 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-126"}]': finished 2024-02-19T21:16:05.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:04.404727+0000 mon.a (mon.0) 3459 : audit [INF] from='client.? v1:172.21.15.163:0/3610632511' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:16:05.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:04.404777+0000 mon.a (mon.0) 3460 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2024-02-19T21:16:05.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:04.406683+0000 mon.a (mon.0) 3461 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-126", "mode": "writeback"}]: dispatch 2024-02-19T21:16:05.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:05.400708+0000 mon.a (mon.0) 3462 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:16:05.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:05.404716+0000 mon.a (mon.0) 3463 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-126", "mode": "writeback"}]': finished 2024-02-19T21:16:05.487 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:05.404778+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2024-02-19T21:16:05.488 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:05.598 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:04.399310+0000 mon.a (mon.0) 3457 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:16:05.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:04.404654+0000 mon.a (mon.0) 3458 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-126"}]': finished 2024-02-19T21:16:05.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:04.404727+0000 mon.a (mon.0) 3459 : audit [INF] from='client.? v1:172.21.15.163:0/3610632511' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:16:05.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:04.404777+0000 mon.a (mon.0) 3460 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2024-02-19T21:16:05.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:04.406683+0000 mon.a (mon.0) 3461 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-126", "mode": "writeback"}]: dispatch 2024-02-19T21:16:05.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:05.400708+0000 mon.a (mon.0) 3462 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:16:05.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:05.404716+0000 mon.a (mon.0) 3463 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-126", "mode": "writeback"}]': finished 2024-02-19T21:16:05.599 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:05.404778+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2024-02-19T21:16:05.599 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:04.399310+0000 mon.a (mon.0) 3457 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:16:05.599 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:04.404654+0000 mon.a (mon.0) 3458 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-126"}]': finished 2024-02-19T21:16:05.599 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:04.404727+0000 mon.a (mon.0) 3459 : audit [INF] from='client.? v1:172.21.15.163:0/3610632511' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:16:05.599 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:04.404777+0000 mon.a (mon.0) 3460 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2024-02-19T21:16:05.600 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:04.406683+0000 mon.a (mon.0) 3461 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-126", "mode": "writeback"}]: dispatch 2024-02-19T21:16:05.600 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:05.400708+0000 mon.a (mon.0) 3462 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:16:05.600 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:05.404716+0000 mon.a (mon.0) 3463 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-126", "mode": "writeback"}]': finished 2024-02-19T21:16:05.600 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:05.404778+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2024-02-19T21:16:05.732 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-19T21:16:05.736 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:16:05.739 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:05.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:16:05.840 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:05.869 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:05.887 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:05.890 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:05.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:05.992 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:05.999 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:16:06.019 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:06.492 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:05.315194+0000 mgr.y (mgr.14550) 602 : cluster [DBG] pgmap v1079: 214 pgs: 214 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:06.492 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:05.472767+0000 mon.a (mon.0) 3465 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:16:06.492 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:05.649879+0000 osd.4 (osd.4) 249 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:16:06.493 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:05.650325+0000 osd.4 (osd.4) 250 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:16:06.688 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:05.315194+0000 mgr.y (mgr.14550) 602 : cluster [DBG] pgmap v1079: 214 pgs: 214 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:06.688 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:05.472767+0000 mon.a (mon.0) 3465 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:16:06.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:05.649879+0000 osd.4 (osd.4) 249 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:16:06.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:05.650325+0000 osd.4 (osd.4) 250 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:16:06.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:05.315194+0000 mgr.y (mgr.14550) 602 : cluster [DBG] pgmap v1079: 214 pgs: 214 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:06.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:05.472767+0000 mon.a (mon.0) 3465 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:16:06.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:05.649879+0000 osd.4 (osd.4) 249 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:16:06.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:05.650325+0000 osd.4 (osd.4) 250 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:16:07.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.174+0000 7f3090453700 1 -- 172.21.15.163:0/923691830 >> v1:172.21.15.163:6790/0 conn(0x7f30881042c0 legacy=0x7f3088106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:07.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.174+0000 7f3090453700 1 -- 172.21.15.163:0/923691830 shutdown_connections 2024-02-19T21:16:07.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.174+0000 7f3090453700 1 -- 172.21.15.163:0/923691830 >> 172.21.15.163:0/923691830 conn(0x7f30880f35a0 msgr2=0x7f30880f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:07.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.174+0000 7f3090453700 1 -- 172.21.15.163:0/923691830 shutdown_connections 2024-02-19T21:16:07.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.174+0000 7f3090453700 1 -- 172.21.15.163:0/923691830 wait complete. 2024-02-19T21:16:07.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f3090453700 1 librados: starting msgr at 2024-02-19T21:16:07.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f3090453700 1 librados: starting objecter 2024-02-19T21:16:07.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f3090453700 1 Processor -- start 2024-02-19T21:16:07.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f3090453700 1 -- start start 2024-02-19T21:16:07.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f3090453700 1 librados: setting wanted keys 2024-02-19T21:16:07.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f3090453700 1 librados: calling monclient init 2024-02-19T21:16:07.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f3090453700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3088117320 con 0x7f30881042c0 2024-02-19T21:16:07.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f3090453700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3088118500 con 0x7f30880fd980 2024-02-19T21:16:07.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f3090453700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3088119700 con 0x7f3088100780 2024-02-19T21:16:07.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f308e9f0700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f30881042c0 0x7f30881159e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:59472/0 (socket says 172.21.15.163:59472) 2024-02-19T21:16:07.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f308d9ee700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f3088100780 0x7f30880fa000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:53328/0 (socket says 172.21.15.163:53328) 2024-02-19T21:16:07.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f308e9f0700 1 -- 172.21.15.163:0/510232126 learned_addr learned my addr 172.21.15.163:0/510232126 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:07.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f307f7fe700 10 client.?.objecter ms_handle_connect 0x7f3088100780 2024-02-19T21:16:07.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f307f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:07.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f307f7fe700 10 client.?.objecter ms_handle_connect 0x7f30881042c0 2024-02-19T21:16:07.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f307f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:07.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f307f7fe700 10 client.?.objecter ms_handle_connect 0x7f30880fd980 2024-02-19T21:16:07.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f307f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:07.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812464656 0 0) 0x7f3088119700 con 0x7f3088100780 2024-02-19T21:16:07.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.175+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3074003800 con 0x7f3088100780 2024-02-19T21:16:07.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911372837 0 0) 0x7f3088117320 con 0x7f30881042c0 2024-02-19T21:16:07.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3088119700 con 0x7f30881042c0 2024-02-19T21:16:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224156600 0 0) 0x7f3088118500 con 0x7f30880fd980 2024-02-19T21:16:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3088117320 con 0x7f30880fd980 2024-02-19T21:16:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1106213737 0 0) 0x7f3074003800 con 0x7f3088100780 2024-02-19T21:16:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3088118500 con 0x7f3088100780 2024-02-19T21:16:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3084002970 con 0x7f3088100780 2024-02-19T21:16:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2039384854 0 0) 0x7f3088119700 con 0x7f30881042c0 2024-02-19T21:16:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3074003800 con 0x7f30881042c0 2024-02-19T21:16:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3080003150 con 0x7f30881042c0 2024-02-19T21:16:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1653866585 0 0) 0x7f3088117320 con 0x7f30880fd980 2024-02-19T21:16:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3088119700 con 0x7f30880fd980 2024-02-19T21:16:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3078001620 con 0x7f30880fd980 2024-02-19T21:16:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2291379042 0 0) 0x7f3088118500 con 0x7f3088100780 2024-02-19T21:16:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 >> v1:172.21.15.193:6789/0 conn(0x7f30880fd980 legacy=0x7f30880f98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:07.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 >> v1:172.21.15.163:6789/0 conn(0x7f30881042c0 legacy=0x7f30881159e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:07.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308811a900 con 0x7f3088100780 2024-02-19T21:16:07.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f3090453700 1 -- 172.21.15.163:0/510232126 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3088117510 con 0x7f3088100780 2024-02-19T21:16:07.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f3090453700 10 client.50486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:07.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f3090453700 1 -- 172.21.15.163:0/510232126 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3088117980 con 0x7f3088100780 2024-02-19T21:16:07.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f3090453700 1 librados: init done 2024-02-19T21:16:07.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3084002d00 con 0x7f3088100780 2024-02-19T21:16:07.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.176+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3084005ae0 con 0x7f3088100780 2024-02-19T21:16:07.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.177+0000 7f3090453700 1 -- 172.21.15.163:0/510232126 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306c004f20 con 0x7f3088100780 2024-02-19T21:16:07.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.177+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f3084001660 con 0x7f3088100780 2024-02-19T21:16:07.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.178+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(717..717 src has 623..717) v4 ==== 8678+0+0 (unknown 3244685340 0 0) 0x7f3084078f20 con 0x7f3088100780 2024-02-19T21:16:07.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.178+0000 7f307f7fe700 10 client.50486.objecter ms_dispatch 0x7f3088001550 osd_map(717..717 src has 623..717) v4 2024-02-19T21:16:07.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.178+0000 7f307f7fe700 3 client.50486.objecter handle_osd_map got epochs [717,717] > 0 2024-02-19T21:16:07.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.178+0000 7f307f7fe700 3 client.50486.objecter handle_osd_map decoding full epoch 717 2024-02-19T21:16:07.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.178+0000 7f307f7fe700 20 client.50486.objecter dump_active .. 0 homeless 2024-02-19T21:16:07.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.178+0000 7f307f7fe700 10 client.50486.objecter ms_handle_connect 0x7f30740680c0 2024-02-19T21:16:07.183 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.182+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f308404c0b0 con 0x7f3088100780 2024-02-19T21:16:07.347 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.346+0000 7f3090453700 1 -- 172.21.15.163:0/510232126 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f306c000b90 con 0x7f30740680c0 2024-02-19T21:16:07.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.361+0000 7f307f7fe700 1 -- 172.21.15.163:0/510232126 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+550057 (unknown 2965378022 0 4104625055) 0x7f306c000b90 con 0x7f30740680c0 2024-02-19T21:16:07.369 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:16:07.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.365+0000 7f3090453700 10 librados: watch_flush enter 2024-02-19T21:16:07.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.365+0000 7f3090453700 10 librados: watch_flush exit 2024-02-19T21:16:07.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.365+0000 7f3090453700 20 client.50486.objecter shutdown clearing up homeless session... 2024-02-19T21:16:07.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.365+0000 7f3090453700 10 client.50486.objecter successfully canceled tick 2024-02-19T21:16:07.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.365+0000 7f3090453700 1 -- 172.21.15.163:0/510232126 >> v1:172.21.15.163:6800/1552514681 conn(0x7f30740680c0 legacy=0x7f307406a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:07.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.365+0000 7f3090453700 1 -- 172.21.15.163:0/510232126 >> v1:172.21.15.163:6790/0 conn(0x7f3088100780 legacy=0x7f30880fa000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:07.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.365+0000 7f3090453700 1 -- 172.21.15.163:0/510232126 shutdown_connections 2024-02-19T21:16:07.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.365+0000 7f3090453700 1 -- 172.21.15.163:0/510232126 >> 172.21.15.163:0/510232126 conn(0x7f30880f35a0 msgr2=0x7f3088107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:07.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.365+0000 7f3090453700 1 -- 172.21.15.163:0/510232126 shutdown_connections 2024-02-19T21:16:07.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.365+0000 7f3090453700 1 -- 172.21.15.163:0/510232126 wait complete. 2024-02-19T21:16:07.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:07.365+0000 7f3090453700 1 librados: shutdown 2024-02-19T21:16:07.375 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:16:07.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:06.425963+0000 mon.a (mon.0) 3466 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:16:07.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:06.426026+0000 mon.a (mon.0) 3467 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2024-02-19T21:16:07.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:06.427132+0000 mon.a (mon.0) 3468 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-126"}]: dispatch 2024-02-19T21:16:07.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:06.425963+0000 mon.a (mon.0) 3466 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:16:07.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:06.426026+0000 mon.a (mon.0) 3467 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2024-02-19T21:16:07.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:06.427132+0000 mon.a (mon.0) 3468 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-126"}]: dispatch 2024-02-19T21:16:07.762 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:16:07.762 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:16:07.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:06.425963+0000 mon.a (mon.0) 3466 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:16:07.774 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:06.426026+0000 mon.a (mon.0) 3467 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2024-02-19T21:16:07.774 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:06.427132+0000 mon.a (mon.0) 3468 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-126"}]: dispatch 2024-02-19T21:16:08.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:16:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:16:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:16:08.131 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":1082,"stamp":"2024-02-19T21:16:07.315380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5882,"num_objects":209,"num_object_clones":1,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":0,"num_read":2018,"num_read_kb":1758,"num_write":756,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":512,"num_bytes_recovered":11554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":907,"ondisk_log_size":907,"up":642,"acting":642,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":613,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2450956,"kb_used_data":19916,"kb_used_omap":0,"kb_used_meta":2430976,"kb_avail":747510260,"statfs":{"total":767960285184,"available":765450506240,"internally_reserved":0,"allocated":20393984,"data_stored":7908859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2489319424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001316"},"pg_stats":[{"pgid":"247.17","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.456393+0000","last_change":"2024-02-19T21:16:02.957862+0000","last_active":"2024-02-19T21:16:03.456393+0000","last_peered":"2024-02-19T21:16:03.456393+0000","last_clean":"2024-02-19T21:16:03.456393+0000","last_became_active":"2024-02-19T21:16:02.956966+0000","last_became_peered":"2024-02-19T21:16:02.956966+0000","last_unstale":"2024-02-19T21:16:03.456393+0000","last_undegraded":"2024-02-19T21:16:03.456393+0000","last_fullsized":"2024-02-19T21:16:03.456393+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"247.16","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401922+0000","last_change":"2024-02-19T21:16:02.414572+0000","last_active":"2024-02-19T21:16:03.401922+0000","last_peered":"2024-02-19T21:16:03.401922+0000","last_clean":"2024-02-19T21:16:03.401922+0000","last_became_active":"2024-02-19T21:16:02.414099+0000","last_became_peered":"2024-02-19T21:16:02.414099+0000","last_unstale":"2024-02-19T21:16:03.401922+0000","last_undegraded":"2024-02-19T21:16:03.401922+0000","last_fullsized":"2024-02-19T21:16:03.401922+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"247.15","version":"0'0","reported_seq":10,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.416212+0000","last_change":"2024-02-19T21:16:02.412788+0000","last_active":"2024-02-19T21:16:02.416212+0000","last_peered":"2024-02-19T21:16:02.416212+0000","last_clean":"2024-02-19T21:16:02.416212+0000","last_became_active":"2024-02-19T21:16:02.412471+0000","last_became_peered":"2024-02-19T21:16:02.412471+0000","last_unstale":"2024-02-19T21:16:02.416212+0000","last_undegraded":"2024-02-19T21:16:02.416212+0000","last_fullsized":"2024-02-19T21:16:02.416212+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"247.14","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402540+0000","last_change":"2024-02-19T21:16:02.411195+0000","last_active":"2024-02-19T21:16:03.402540+0000","last_peered":"2024-02-19T21:16:03.402540+0000","last_clean":"2024-02-19T21:16:03.402540+0000","last_became_active":"2024-02-19T21:16:02.410795+0000","last_became_peered":"2024-02-19T21:16:02.410795+0000","last_unstale":"2024-02-19T21:16:03.402540+0000","last_undegraded":"2024-02-19T21:16:03.402540+0000","last_fullsized":"2024-02-19T21:16:03.402540+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"247.13","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873038+0000","last_change":"2024-02-19T21:16:02.957584+0000","last_active":"2024-02-19T21:16:03.873038+0000","last_peered":"2024-02-19T21:16:03.873038+0000","last_clean":"2024-02-19T21:16:03.873038+0000","last_became_active":"2024-02-19T21:16:02.956938+0000","last_became_peered":"2024-02-19T21:16:02.956938+0000","last_unstale":"2024-02-19T21:16:03.873038+0000","last_undegraded":"2024-02-19T21:16:03.873038+0000","last_fullsized":"2024-02-19T21:16:03.873038+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"247.12","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.872829+0000","last_change":"2024-02-19T21:16:02.957389+0000","last_active":"2024-02-19T21:16:03.872829+0000","last_peered":"2024-02-19T21:16:03.872829+0000","last_clean":"2024-02-19T21:16:03.872829+0000","last_became_active":"2024-02-19T21:16:02.957018+0000","last_became_peered":"2024-02-19T21:16:02.957018+0000","last_unstale":"2024-02-19T21:16:03.872829+0000","last_undegraded":"2024-02-19T21:16:03.872829+0000","last_fullsized":"2024-02-19T21:16:03.872829+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"247.11","version":"0'0","reported_seq":10,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.960197+0000","last_change":"2024-02-19T21:16:02.959679+0000","last_active":"2024-02-19T21:16:02.960197+0000","last_peered":"2024-02-19T21:16:02.960197+0000","last_clean":"2024-02-19T21:16:02.960197+0000","last_became_active":"2024-02-19T21:16:02.959292+0000","last_became_peered":"2024-02-19T21:16:02.959292+0000","last_unstale":"2024-02-19T21:16:02.960197+0000","last_undegraded":"2024-02-19T21:16:02.960197+0000","last_fullsized":"2024-02-19T21:16:02.960197+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"247.10","version":"0'0","reported_seq":10,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.416001+0000","last_change":"2024-02-19T21:16:02.415140+0000","last_active":"2024-02-19T21:16:02.416001+0000","last_peered":"2024-02-19T21:16:02.416001+0000","last_clean":"2024-02-19T21:16:02.416001+0000","last_became_active":"2024-02-19T21:16:02.414771+0000","last_became_peered":"2024-02-19T21:16:02.414771+0000","last_unstale":"2024-02-19T21:16:02.416001+0000","last_undegraded":"2024-02-19T21:16:02.416001+0000","last_fullsized":"2024-02-19T21:16:02.416001+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"247.f","version":"0'0","reported_seq":10,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.411599+0000","last_change":"2024-02-19T21:16:02.411085+0000","last_active":"2024-02-19T21:16:02.411599+0000","last_peered":"2024-02-19T21:16:02.411599+0000","last_clean":"2024-02-19T21:16:02.411599+0000","last_became_active":"2024-02-19T21:16:02.410800+0000","last_became_peered":"2024-02-19T21:16:02.410800+0000","last_unstale":"2024-02-19T21:16:02.411599+0000","last_undegraded":"2024-02-19T21:16:02.411599+0000","last_fullsized":"2024-02-19T21:16:02.411599+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"247.e","version":"0'0","reported_seq":10,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.416136+0000","last_change":"2024-02-19T21:16:02.414873+0000","last_active":"2024-02-19T21:16:02.416136+0000","last_peered":"2024-02-19T21:16:02.416136+0000","last_clean":"2024-02-19T21:16:02.416136+0000","last_became_active":"2024-02-19T21:16:02.414563+0000","last_became_peered":"2024-02-19T21:16:02.414563+0000","last_unstale":"2024-02-19T21:16:02.416136+0000","last_undegraded":"2024-02-19T21:16:02.416136+0000","last_fullsized":"2024-02-19T21:16:02.416136+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"247.d","version":"0'0","reported_seq":10,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.415324+0000","last_change":"2024-02-19T21:16:02.414533+0000","last_active":"2024-02-19T21:16:02.415324+0000","last_peered":"2024-02-19T21:16:02.415324+0000","last_clean":"2024-02-19T21:16:02.415324+0000","last_became_active":"2024-02-19T21:16:02.414276+0000","last_became_peered":"2024-02-19T21:16:02.414276+0000","last_unstale":"2024-02-19T21:16:02.415324+0000","last_undegraded":"2024-02-19T21:16:02.415324+0000","last_fullsized":"2024-02-19T21:16:02.415324+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"247.c","version":"0'0","reported_seq":10,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.416905+0000","last_change":"2024-02-19T21:16:02.416627+0000","last_active":"2024-02-19T21:16:02.416905+0000","last_peered":"2024-02-19T21:16:02.416905+0000","last_clean":"2024-02-19T21:16:02.416905+0000","last_became_active":"2024-02-19T21:16:02.416415+0000","last_became_peered":"2024-02-19T21:16:02.416415+0000","last_unstale":"2024-02-19T21:16:02.416905+0000","last_undegraded":"2024-02-19T21:16:02.416905+0000","last_fullsized":"2024-02-19T21:16:02.416905+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"247.b","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403030+0000","last_change":"2024-02-19T21:16:02.411966+0000","last_active":"2024-02-19T21:16:03.403030+0000","last_peered":"2024-02-19T21:16:03.403030+0000","last_clean":"2024-02-19T21:16:03.403030+0000","last_became_active":"2024-02-19T21:16:02.411651+0000","last_became_peered":"2024-02-19T21:16:02.411651+0000","last_unstale":"2024-02-19T21:16:03.403030+0000","last_undegraded":"2024-02-19T21:16:03.403030+0000","last_fullsized":"2024-02-19T21:16:03.403030+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"247.a","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401502+0000","last_change":"2024-02-19T21:16:02.957840+0000","last_active":"2024-02-19T21:16:03.401502+0000","last_peered":"2024-02-19T21:16:03.401502+0000","last_clean":"2024-02-19T21:16:03.401502+0000","last_became_active":"2024-02-19T21:16:02.957558+0000","last_became_peered":"2024-02-19T21:16:02.957558+0000","last_unstale":"2024-02-19T21:16:03.401502+0000","last_undegraded":"2024-02-19T21:16:03.401502+0000","last_fullsized":"2024-02-19T21:16:03.401502+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"247.9","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.872727+0000","last_change":"2024-02-19T21:16:02.411209+0000","last_active":"2024-02-19T21:16:03.872727+0000","last_peered":"2024-02-19T21:16:03.872727+0000","last_clean":"2024-02-19T21:16:03.872727+0000","last_became_active":"2024-02-19T21:16:02.410942+0000","last_became_peered":"2024-02-19T21:16:02.410942+0000","last_unstale":"2024-02-19T21:16:03.872727+0000","last_undegraded":"2024-02-19T21:16:03.872727+0000","last_fullsized":"2024-02-19T21:16:03.872727+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"247.8","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.455770+0000","last_change":"2024-02-19T21:16:02.410015+0000","last_active":"2024-02-19T21:16:03.455770+0000","last_peered":"2024-02-19T21:16:03.455770+0000","last_clean":"2024-02-19T21:16:03.455770+0000","last_became_active":"2024-02-19T21:16:02.409721+0000","last_became_peered":"2024-02-19T21:16:02.409721+0000","last_unstale":"2024-02-19T21:16:03.455770+0000","last_undegraded":"2024-02-19T21:16:03.455770+0000","last_fullsized":"2024-02-19T21:16:03.455770+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"247.7","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403780+0000","last_change":"2024-02-19T21:16:02.414056+0000","last_active":"2024-02-19T21:16:03.403780+0000","last_peered":"2024-02-19T21:16:03.403780+0000","last_clean":"2024-02-19T21:16:03.403780+0000","last_became_active":"2024-02-19T21:16:02.413776+0000","last_became_peered":"2024-02-19T21:16:02.413776+0000","last_unstale":"2024-02-19T21:16:03.403780+0000","last_undegraded":"2024-02-19T21:16:03.403780+0000","last_fullsized":"2024-02-19T21:16:03.403780+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"247.6","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401990+0000","last_change":"2024-02-19T21:16:02.957277+0000","last_active":"2024-02-19T21:16:03.401990+0000","last_peered":"2024-02-19T21:16:03.401990+0000","last_clean":"2024-02-19T21:16:03.401990+0000","last_became_active":"2024-02-19T21:16:02.956947+0000","last_became_peered":"2024-02-19T21:16:02.956947+0000","last_unstale":"2024-02-19T21:16:03.401990+0000","last_undegraded":"2024-02-19T21:16:03.401990+0000","last_fullsized":"2024-02-19T21:16:03.401990+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"247.5","version":"0'0","reported_seq":10,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.415309+0000","last_change":"2024-02-19T21:16:02.413877+0000","last_active":"2024-02-19T21:16:02.415309+0000","last_peered":"2024-02-19T21:16:02.415309+0000","last_clean":"2024-02-19T21:16:02.415309+0000","last_became_active":"2024-02-19T21:16:02.413623+0000","last_became_peered":"2024-02-19T21:16:02.413623+0000","last_unstale":"2024-02-19T21:16:02.415309+0000","last_undegraded":"2024-02-19T21:16:02.415309+0000","last_fullsized":"2024-02-19T21:16:02.415309+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"247.4","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402630+0000","last_change":"2024-02-19T21:16:02.957370+0000","last_active":"2024-02-19T21:16:03.402630+0000","last_peered":"2024-02-19T21:16:03.402630+0000","last_clean":"2024-02-19T21:16:03.402630+0000","last_became_active":"2024-02-19T21:16:02.956959+0000","last_became_peered":"2024-02-19T21:16:02.956959+0000","last_unstale":"2024-02-19T21:16:03.402630+0000","last_undegraded":"2024-02-19T21:16:03.402630+0000","last_fullsized":"2024-02-19T21:16:03.402630+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"247.3","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873113+0000","last_change":"2024-02-19T21:16:02.412453+0000","last_active":"2024-02-19T21:16:03.873113+0000","last_peered":"2024-02-19T21:16:03.873113+0000","last_clean":"2024-02-19T21:16:03.873113+0000","last_became_active":"2024-02-19T21:16:02.412096+0000","last_became_peered":"2024-02-19T21:16:02.412096+0000","last_unstale":"2024-02-19T21:16:03.873113+0000","last_undegraded":"2024-02-19T21:16:03.873113+0000","last_fullsized":"2024-02-19T21:16:03.873113+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"247.2","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.872873+0000","last_change":"2024-02-19T21:16:02.412100+0000","last_active":"2024-02-19T21:16:03.872873+0000","last_peered":"2024-02-19T21:16:03.872873+0000","last_clean":"2024-02-19T21:16:03.872873+0000","last_became_active":"2024-02-19T21:16:02.411821+0000","last_became_peered":"2024-02-19T21:16:02.411821+0000","last_unstale":"2024-02-19T21:16:03.872873+0000","last_undegraded":"2024-02-19T21:16:03.872873+0000","last_fullsized":"2024-02-19T21:16:03.872873+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"247.1","version":"0'0","reported_seq":10,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.960186+0000","last_change":"2024-02-19T21:16:02.959550+0000","last_active":"2024-02-19T21:16:02.960186+0000","last_peered":"2024-02-19T21:16:02.960186+0000","last_clean":"2024-02-19T21:16:02.960186+0000","last_became_active":"2024-02-19T21:16:02.958858+0000","last_became_peered":"2024-02-19T21:16:02.958858+0000","last_unstale":"2024-02-19T21:16:02.960186+0000","last_undegraded":"2024-02-19T21:16:02.960186+0000","last_fullsized":"2024-02-19T21:16:02.960186+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"247.0","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401065+0000","last_change":"2024-02-19T21:16:02.412955+0000","last_active":"2024-02-19T21:16:03.401065+0000","last_peered":"2024-02-19T21:16:03.401065+0000","last_clean":"2024-02-19T21:16:03.401065+0000","last_became_active":"2024-02-19T21:16:02.412687+0000","last_became_peered":"2024-02-19T21:16:02.412687+0000","last_unstale":"2024-02-19T21:16:03.401065+0000","last_undegraded":"2024-02-19T21:16:03.401065+0000","last_fullsized":"2024-02-19T21:16:03.401065+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"247.1a","version":"0'0","reported_seq":10,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.415233+0000","last_change":"2024-02-19T21:16:02.413870+0000","last_active":"2024-02-19T21:16:02.415233+0000","last_peered":"2024-02-19T21:16:02.415233+0000","last_clean":"2024-02-19T21:16:02.415233+0000","last_became_active":"2024-02-19T21:16:02.413591+0000","last_became_peered":"2024-02-19T21:16:02.413591+0000","last_unstale":"2024-02-19T21:16:02.415233+0000","last_undegraded":"2024-02-19T21:16:02.415233+0000","last_fullsized":"2024-02-19T21:16:02.415233+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.7","version":"700'10","reported_seq":166,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405129+0000","last_change":"2024-02-19T21:15:51.952271+0000","last_active":"2024-02-19T21:16:02.405129+0000","last_peered":"2024-02-19T21:16:02.405129+0000","last_clean":"2024-02-19T21:16:02.405129+0000","last_became_active":"2024-02-19T21:14:16.391553+0000","last_became_peered":"2024-02-19T21:14:16.391553+0000","last_unstale":"2024-02-19T21:16:02.405129+0000","last_undegraded":"2024-02-19T21:16:02.405129+0000","last_fullsized":"2024-02-19T21:16:02.405129+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"700'10","last_scrub_stamp":"2024-02-19T21:15:51.952191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:15:51.952191+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"247.1b","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.872979+0000","last_change":"2024-02-19T21:16:02.413043+0000","last_active":"2024-02-19T21:16:03.872979+0000","last_peered":"2024-02-19T21:16:03.872979+0000","last_clean":"2024-02-19T21:16:03.872979+0000","last_became_active":"2024-02-19T21:16:02.412724+0000","last_became_peered":"2024-02-19T21:16:02.412724+0000","last_unstale":"2024-02-19T21:16:03.872979+0000","last_undegraded":"2024-02-19T21:16:03.872979+0000","last_fullsized":"2024-02-19T21:16:03.872979+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"234.6","version":"671'12","reported_seq":139,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.457345+0000","last_change":"2024-02-19T21:15:56.206241+0000","last_active":"2024-02-19T21:16:03.457345+0000","last_peered":"2024-02-19T21:16:03.457345+0000","last_clean":"2024-02-19T21:16:03.457345+0000","last_became_active":"2024-02-19T21:14:16.372197+0000","last_became_peered":"2024-02-19T21:14:16.372197+0000","last_unstale":"2024-02-19T21:16:03.457345+0000","last_undegraded":"2024-02-19T21:16:03.457345+0000","last_fullsized":"2024-02-19T21:16:03.457345+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"671'12","last_scrub_stamp":"2024-02-19T21:15:56.206176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:15:56.206176+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"247.18","version":"0'0","reported_seq":10,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.413224+0000","last_change":"2024-02-19T21:16:02.412737+0000","last_active":"2024-02-19T21:16:02.413224+0000","last_peered":"2024-02-19T21:16:02.413224+0000","last_clean":"2024-02-19T21:16:02.413224+0000","last_became_active":"2024-02-19T21:16:02.412421+0000","last_became_peered":"2024-02-19T21:16:02.412421+0000","last_unstale":"2024-02-19T21:16:02.413224+0000","last_undegraded":"2024-02-19T21:16:02.413224+0000","last_fullsized":"2024-02-19T21:16:02.413224+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.5","version":"663'11","reported_seq":175,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402134+0000","last_change":"2024-02-19T21:15:49.389636+0000","last_active":"2024-02-19T21:16:03.402134+0000","last_peered":"2024-02-19T21:16:03.402134+0000","last_clean":"2024-02-19T21:16:03.402134+0000","last_became_active":"2024-02-19T21:14:41.403889+0000","last_became_peered":"2024-02-19T21:14:41.403889+0000","last_unstale":"2024-02-19T21:16:03.402134+0000","last_undegraded":"2024-02-19T21:16:03.402134+0000","last_fullsized":"2024-02-19T21:16:03.402134+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":21,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"247.19","version":"0'0","reported_seq":10,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.960270+0000","last_change":"2024-02-19T21:16:02.959958+0000","last_active":"2024-02-19T21:16:02.960270+0000","last_peered":"2024-02-19T21:16:02.960270+0000","last_clean":"2024-02-19T21:16:02.960270+0000","last_became_active":"2024-02-19T21:16:02.959680+0000","last_became_peered":"2024-02-19T21:16:02.959680+0000","last_unstale":"2024-02-19T21:16:02.960270+0000","last_undegraded":"2024-02-19T21:16:02.960270+0000","last_fullsized":"2024-02-19T21:16:02.960270+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"234.4","version":"708'6","reported_seq":151,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403789+0000","last_change":"2024-02-19T21:15:49.390166+0000","last_active":"2024-02-19T21:16:03.403789+0000","last_peered":"2024-02-19T21:16:03.403789+0000","last_clean":"2024-02-19T21:16:03.403789+0000","last_became_active":"2024-02-19T21:13:55.356730+0000","last_became_peered":"2024-02-19T21:13:55.356730+0000","last_unstale":"2024-02-19T21:16:03.403789+0000","last_undegraded":"2024-02-19T21:16:03.403789+0000","last_fullsized":"2024-02-19T21:16:03.403789+0000","mapping_epoch":598,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":599,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"247.1e","version":"0'0","reported_seq":10,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.415071+0000","last_change":"2024-02-19T21:16:02.414299+0000","last_active":"2024-02-19T21:16:02.415071+0000","last_peered":"2024-02-19T21:16:02.415071+0000","last_clean":"2024-02-19T21:16:02.415071+0000","last_became_active":"2024-02-19T21:16:02.414057+0000","last_became_peered":"2024-02-19T21:16:02.414057+0000","last_unstale":"2024-02-19T21:16:02.415071+0000","last_undegraded":"2024-02-19T21:16:02.415071+0000","last_fullsized":"2024-02-19T21:16:02.415071+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.3","version":"627'10","reported_seq":174,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405780+0000","last_change":"2024-02-19T21:15:49.404376+0000","last_active":"2024-02-19T21:16:02.405780+0000","last_peered":"2024-02-19T21:16:02.405780+0000","last_clean":"2024-02-19T21:16:02.405780+0000","last_became_active":"2024-02-19T21:14:16.398263+0000","last_became_peered":"2024-02-19T21:14:16.398263+0000","last_unstale":"2024-02-19T21:16:02.405780+0000","last_undegraded":"2024-02-19T21:16:02.405780+0000","last_fullsized":"2024-02-19T21:16:02.405780+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":21,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"247.1f","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401505+0000","last_change":"2024-02-19T21:16:02.411085+0000","last_active":"2024-02-19T21:16:03.401505+0000","last_peered":"2024-02-19T21:16:03.401505+0000","last_clean":"2024-02-19T21:16:03.401505+0000","last_became_active":"2024-02-19T21:16:02.410860+0000","last_became_peered":"2024-02-19T21:16:02.410860+0000","last_unstale":"2024-02-19T21:16:03.401505+0000","last_undegraded":"2024-02-19T21:16:03.401505+0000","last_fullsized":"2024-02-19T21:16:03.401505+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"234.2","version":"663'5","reported_seq":109,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406245+0000","last_change":"2024-02-19T21:15:49.396890+0000","last_active":"2024-02-19T21:16:02.406245+0000","last_peered":"2024-02-19T21:16:02.406245+0000","last_clean":"2024-02-19T21:16:02.406245+0000","last_became_active":"2024-02-19T21:14:41.390378+0000","last_became_peered":"2024-02-19T21:14:41.390378+0000","last_unstale":"2024-02-19T21:16:02.406245+0000","last_undegraded":"2024-02-19T21:16:02.406245+0000","last_fullsized":"2024-02-19T21:16:02.406245+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"247.1c","version":"0'0","reported_seq":10,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.958615+0000","last_change":"2024-02-19T21:16:02.958284+0000","last_active":"2024-02-19T21:16:02.958615+0000","last_peered":"2024-02-19T21:16:02.958615+0000","last_clean":"2024-02-19T21:16:02.958615+0000","last_became_active":"2024-02-19T21:16:02.957932+0000","last_became_peered":"2024-02-19T21:16:02.957932+0000","last_unstale":"2024-02-19T21:16:02.958615+0000","last_undegraded":"2024-02-19T21:16:02.958615+0000","last_fullsized":"2024-02-19T21:16:02.958615+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.1","version":"627'6","reported_seq":170,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403575+0000","last_change":"2024-02-19T21:15:49.391819+0000","last_active":"2024-02-19T21:16:03.403575+0000","last_peered":"2024-02-19T21:16:03.403575+0000","last_clean":"2024-02-19T21:16:03.403575+0000","last_became_active":"2024-02-19T21:14:41.401375+0000","last_became_peered":"2024-02-19T21:14:41.401375+0000","last_unstale":"2024-02-19T21:16:03.403575+0000","last_undegraded":"2024-02-19T21:16:03.403575+0000","last_fullsized":"2024-02-19T21:16:03.403575+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":9,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"247.1d","version":"0'0","reported_seq":11,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402030+0000","last_change":"2024-02-19T21:16:02.413999+0000","last_active":"2024-02-19T21:16:03.402030+0000","last_peered":"2024-02-19T21:16:03.402030+0000","last_clean":"2024-02-19T21:16:03.402030+0000","last_became_active":"2024-02-19T21:16:02.413706+0000","last_became_peered":"2024-02-19T21:16:02.413706+0000","last_unstale":"2024-02-19T21:16:03.402030+0000","last_undegraded":"2024-02-19T21:16:03.402030+0000","last_fullsized":"2024-02-19T21:16:03.402030+0000","mapping_epoch":712,"log_start":"0'0","ondisk_log_start":"0'0","created":712,"last_epoch_clean":713,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:01.394673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:01.394673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"234.0","version":"653'9","reported_seq":172,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.872671+0000","last_change":"2024-02-19T21:15:49.384800+0000","last_active":"2024-02-19T21:16:03.872671+0000","last_peered":"2024-02-19T21:16:03.872671+0000","last_clean":"2024-02-19T21:16:03.872671+0000","last_became_active":"2024-02-19T21:14:16.383023+0000","last_became_peered":"2024-02-19T21:14:16.383023+0000","last_unstale":"2024-02-19T21:16:03.872671+0000","last_undegraded":"2024-02-19T21:16:03.872671+0000","last_fullsized":"2024-02-19T21:16:03.872671+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":111,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.456160+0000","last_change":"2024-02-19T21:15:41.405332+0000","last_active":"2024-02-19T21:16:03.456160+0000","last_peered":"2024-02-19T21:16:03.456160+0000","last_clean":"2024-02-19T21:16:03.456160+0000","last_became_active":"2024-02-19T21:15:41.404959+0000","last_became_peered":"2024-02-19T21:15:41.404959+0000","last_unstale":"2024-02-19T21:16:03.456160+0000","last_undegraded":"2024-02-19T21:16:03.456160+0000","last_fullsized":"2024-02-19T21:16:03.456160+0000","mapping_epoch":691,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":692,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.738600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.738600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":253,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402620+0000","last_change":"2024-02-19T21:14:25.068758+0000","last_active":"2024-02-19T21:16:03.402620+0000","last_peered":"2024-02-19T21:16:03.402620+0000","last_clean":"2024-02-19T21:16:03.402620+0000","last_became_active":"2024-02-19T21:11:50.303790+0000","last_became_peered":"2024-02-19T21:11:50.303790+0000","last_unstale":"2024-02-19T21:16:03.402620+0000","last_undegraded":"2024-02-19T21:16:03.402620+0000","last_fullsized":"2024-02-19T21:16:03.402620+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:25.068700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:25.068700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.25","version":"0'0","reported_seq":762,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873404+0000","last_change":"2024-02-19T21:14:16.380374+0000","last_active":"2024-02-19T21:16:03.873404+0000","last_peered":"2024-02-19T21:16:03.873404+0000","last_clean":"2024-02-19T21:16:03.873404+0000","last_became_active":"2024-02-19T21:14:16.379899+0000","last_became_peered":"2024-02-19T21:14:16.379899+0000","last_unstale":"2024-02-19T21:16:03.873404+0000","last_undegraded":"2024-02-19T21:16:03.873404+0000","last_fullsized":"2024-02-19T21:16:03.873404+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:47.724347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:47.724347+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":102,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.456532+0000","last_change":"2024-02-19T21:14:16.372896+0000","last_active":"2024-02-19T21:16:03.456532+0000","last_peered":"2024-02-19T21:16:03.456532+0000","last_clean":"2024-02-19T21:16:03.456532+0000","last_became_active":"2024-02-19T21:14:16.372498+0000","last_became_peered":"2024-02-19T21:14:16.372498+0000","last_unstale":"2024-02-19T21:16:03.456532+0000","last_undegraded":"2024-02-19T21:16:03.456532+0000","last_fullsized":"2024-02-19T21:16:03.456532+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:55.034623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:13:55.034623+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":33,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402443+0000","last_change":"2024-02-19T21:15:41.399384+0000","last_active":"2024-02-19T21:16:03.402443+0000","last_peered":"2024-02-19T21:16:03.402443+0000","last_clean":"2024-02-19T21:16:03.402443+0000","last_became_active":"2024-02-19T21:15:41.398946+0000","last_became_peered":"2024-02-19T21:15:41.398946+0000","last_unstale":"2024-02-19T21:16:03.402443+0000","last_undegraded":"2024-02-19T21:16:03.402443+0000","last_fullsized":"2024-02-19T21:16:03.402443+0000","mapping_epoch":691,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":692,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:36.863734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:36.863734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.26","version":"0'0","reported_seq":260,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.403978+0000","last_change":"2024-02-19T21:14:16.376340+0000","last_active":"2024-02-19T21:16:02.403978+0000","last_peered":"2024-02-19T21:16:02.403978+0000","last_clean":"2024-02-19T21:16:02.403978+0000","last_became_active":"2024-02-19T21:14:16.376017+0000","last_became_peered":"2024-02-19T21:14:16.376017+0000","last_unstale":"2024-02-19T21:16:02.403978+0000","last_undegraded":"2024-02-19T21:16:02.403978+0000","last_fullsized":"2024-02-19T21:16:02.403978+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:47.748283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:47.748283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":781,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402284+0000","last_change":"2024-02-19T21:15:49.242815+0000","last_active":"2024-02-19T21:16:03.402284+0000","last_peered":"2024-02-19T21:16:03.402284+0000","last_clean":"2024-02-19T21:16:03.402284+0000","last_became_active":"2024-02-19T21:14:41.408037+0000","last_became_peered":"2024-02-19T21:14:41.408037+0000","last_unstale":"2024-02-19T21:16:03.402284+0000","last_undegraded":"2024-02-19T21:16:03.402284+0000","last_fullsized":"2024-02-19T21:16:03.402284+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:49.242760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:49.242760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.20","version":"397'2","reported_seq":801,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406579+0000","last_change":"2024-02-19T21:14:58.880699+0000","last_active":"2024-02-19T21:16:02.406579+0000","last_peered":"2024-02-19T21:16:02.406579+0000","last_clean":"2024-02-19T21:16:02.406579+0000","last_became_active":"2024-02-19T21:14:19.482027+0000","last_became_peered":"2024-02-19T21:14:19.482027+0000","last_unstale":"2024-02-19T21:16:02.406579+0000","last_undegraded":"2024-02-19T21:16:02.406579+0000","last_fullsized":"2024-02-19T21:16:02.406579+0000","mapping_epoch":622,"log_start":"58'1","ondisk_log_start":"58'1","created":474,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":6,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:14:58.880622+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:14:58.880622+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":266,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403097+0000","last_change":"2024-02-19T21:14:16.385009+0000","last_active":"2024-02-19T21:16:03.403097+0000","last_peered":"2024-02-19T21:16:03.403097+0000","last_clean":"2024-02-19T21:16:03.403097+0000","last_became_active":"2024-02-19T21:14:16.384406+0000","last_became_peered":"2024-02-19T21:14:16.384406+0000","last_unstale":"2024-02-19T21:16:03.403097+0000","last_undegraded":"2024-02-19T21:16:03.403097+0000","last_fullsized":"2024-02-19T21:16:03.403097+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:51.081993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:51.081993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":107,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.457669+0000","last_change":"2024-02-19T21:15:51.187729+0000","last_active":"2024-02-19T21:16:03.457669+0000","last_peered":"2024-02-19T21:16:03.457669+0000","last_clean":"2024-02-19T21:16:03.457669+0000","last_became_active":"2024-02-19T21:14:16.382869+0000","last_became_peered":"2024-02-19T21:14:16.382869+0000","last_unstale":"2024-02-19T21:16:03.457669+0000","last_undegraded":"2024-02-19T21:16:03.457669+0000","last_fullsized":"2024-02-19T21:16:03.457669+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:51.187606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:51.187606+0000","last_clean_scrub_stamp":"2024-02-19T21:15:51.187606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"623'13","reported_seq":917,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.837803+0000","last_change":"2024-02-19T21:14:48.841321+0000","last_active":"2024-02-19T21:16:03.837803+0000","last_peered":"2024-02-19T21:16:03.837803+0000","last_clean":"2024-02-19T21:16:03.837803+0000","last_became_active":"2024-02-19T21:14:16.381241+0000","last_became_peered":"2024-02-19T21:14:16.381241+0000","last_unstale":"2024-02-19T21:16:03.837803+0000","last_undegraded":"2024-02-19T21:16:03.837803+0000","last_fullsized":"2024-02-19T21:16:03.837803+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"623'13","last_scrub_stamp":"2024-02-19T21:14:48.841229+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:14:48.841229+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'28","reported_seq":779,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405325+0000","last_change":"2024-02-19T21:14:58.119198+0000","last_active":"2024-02-19T21:16:02.405325+0000","last_peered":"2024-02-19T21:16:02.405325+0000","last_clean":"2024-02-19T21:16:02.405325+0000","last_became_active":"2024-02-19T21:14:16.394686+0000","last_became_peered":"2024-02-19T21:14:16.394686+0000","last_unstale":"2024-02-19T21:16:02.405325+0000","last_undegraded":"2024-02-19T21:16:02.405325+0000","last_fullsized":"2024-02-19T21:16:02.405325+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":621,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-19T21:14:58.119090+0000","last_deep_scrub":"473'28","last_deep_scrub_stamp":"2024-02-19T21:11:47.051823+0000","last_clean_scrub_stamp":"2024-02-19T21:14:58.119090+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":714,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402091+0000","last_change":"2024-02-19T21:14:52.369699+0000","last_active":"2024-02-19T21:16:03.402091+0000","last_peered":"2024-02-19T21:16:03.402091+0000","last_clean":"2024-02-19T21:16:03.402091+0000","last_became_active":"2024-02-19T21:10:40.299425+0000","last_became_peered":"2024-02-19T21:10:40.299425+0000","last_unstale":"2024-02-19T21:16:03.402091+0000","last_undegraded":"2024-02-19T21:16:03.402091+0000","last_fullsized":"2024-02-19T21:16:03.402091+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:52.369624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_clean_scrub_stamp":"2024-02-19T21:14:52.369624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":714,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.408919+0000","last_change":"2024-02-19T21:15:04.119671+0000","last_active":"2024-02-19T21:16:02.408919+0000","last_peered":"2024-02-19T21:16:02.408919+0000","last_clean":"2024-02-19T21:16:02.408919+0000","last_became_active":"2024-02-19T21:14:41.388356+0000","last_became_peered":"2024-02-19T21:14:41.388356+0000","last_unstale":"2024-02-19T21:16:02.408919+0000","last_undegraded":"2024-02-19T21:16:02.408919+0000","last_fullsized":"2024-02-19T21:16:02.408919+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:04.119584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:04.119584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":635,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403267+0000","last_change":"2024-02-19T21:15:38.353068+0000","last_active":"2024-02-19T21:16:03.403267+0000","last_peered":"2024-02-19T21:16:03.403267+0000","last_clean":"2024-02-19T21:16:03.403267+0000","last_became_active":"2024-02-19T21:04:39.702018+0000","last_became_peered":"2024-02-19T21:04:39.702018+0000","last_unstale":"2024-02-19T21:16:03.403267+0000","last_undegraded":"2024-02-19T21:16:03.403267+0000","last_fullsized":"2024-02-19T21:16:03.403267+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:38.353000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:38.353000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'16","reported_seq":625,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.403551+0000","last_change":"2024-02-19T21:15:03.898768+0000","last_active":"2024-02-19T21:16:02.403551+0000","last_peered":"2024-02-19T21:16:02.403551+0000","last_clean":"2024-02-19T21:16:02.403551+0000","last_became_active":"2024-02-19T21:14:41.391381+0000","last_became_peered":"2024-02-19T21:14:41.391381+0000","last_unstale":"2024-02-19T21:16:02.403551+0000","last_undegraded":"2024-02-19T21:16:02.403551+0000","last_fullsized":"2024-02-19T21:16:02.403551+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'16","last_scrub_stamp":"2024-02-19T21:15:03.898694+0000","last_deep_scrub":"473'16","last_deep_scrub_stamp":"2024-02-19T21:11:46.956424+0000","last_clean_scrub_stamp":"2024-02-19T21:15:03.898694+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":721,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404158+0000","last_change":"2024-02-19T21:15:05.888480+0000","last_active":"2024-02-19T21:16:02.404158+0000","last_peered":"2024-02-19T21:16:02.404158+0000","last_clean":"2024-02-19T21:16:02.404158+0000","last_became_active":"2024-02-19T21:10:41.312116+0000","last_became_peered":"2024-02-19T21:10:41.312116+0000","last_unstale":"2024-02-19T21:16:02.404158+0000","last_undegraded":"2024-02-19T21:16:02.404158+0000","last_fullsized":"2024-02-19T21:16:02.404158+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:05.888410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:05.888410+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":719,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401636+0000","last_change":"2024-02-19T21:15:32.211604+0000","last_active":"2024-02-19T21:16:03.401636+0000","last_peered":"2024-02-19T21:16:03.401636+0000","last_clean":"2024-02-19T21:16:03.401636+0000","last_became_active":"2024-02-19T21:10:41.305574+0000","last_became_peered":"2024-02-19T21:10:41.305574+0000","last_unstale":"2024-02-19T21:16:03.401636+0000","last_undegraded":"2024-02-19T21:16:03.401636+0000","last_fullsized":"2024-02-19T21:16:03.401636+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:32.211538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:32.211538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":526,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.403299+0000","last_change":"2024-02-19T21:14:46.798482+0000","last_active":"2024-02-19T21:16:02.403299+0000","last_peered":"2024-02-19T21:16:02.403299+0000","last_clean":"2024-02-19T21:16:02.403299+0000","last_became_active":"2024-02-19T21:14:41.390553+0000","last_became_peered":"2024-02-19T21:14:41.390553+0000","last_unstale":"2024-02-19T21:16:02.403299+0000","last_undegraded":"2024-02-19T21:16:02.403299+0000","last_fullsized":"2024-02-19T21:16:02.403299+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:46.798420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:46.798420+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":632,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406715+0000","last_change":"2024-02-19T21:14:18.913530+0000","last_active":"2024-02-19T21:16:02.406715+0000","last_peered":"2024-02-19T21:16:02.406715+0000","last_clean":"2024-02-19T21:16:02.406715+0000","last_became_active":"2024-02-19T21:04:39.764937+0000","last_became_peered":"2024-02-19T21:04:39.764937+0000","last_unstale":"2024-02-19T21:16:02.406715+0000","last_undegraded":"2024-02-19T21:16:02.406715+0000","last_fullsized":"2024-02-19T21:16:02.406715+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:18.913439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:18.913439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"459'35","reported_seq":127,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.457838+0000","last_change":"2024-02-19T21:14:47.311327+0000","last_active":"2024-02-19T21:16:03.457838+0000","last_peered":"2024-02-19T21:16:03.457838+0000","last_clean":"2024-02-19T21:16:03.457838+0000","last_became_active":"2024-02-19T21:14:16.391564+0000","last_became_peered":"2024-02-19T21:14:16.391564+0000","last_unstale":"2024-02-19T21:16:03.457838+0000","last_undegraded":"2024-02-19T21:16:03.457838+0000","last_fullsized":"2024-02-19T21:16:03.457838+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'35","last_scrub_stamp":"2024-02-19T21:14:47.311262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:47.311262+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":705,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402971+0000","last_change":"2024-02-19T21:14:41.405996+0000","last_active":"2024-02-19T21:16:03.402971+0000","last_peered":"2024-02-19T21:16:03.402971+0000","last_clean":"2024-02-19T21:16:03.402971+0000","last_became_active":"2024-02-19T21:14:41.405016+0000","last_became_peered":"2024-02-19T21:14:41.405016+0000","last_unstale":"2024-02-19T21:16:03.402971+0000","last_undegraded":"2024-02-19T21:16:03.402971+0000","last_fullsized":"2024-02-19T21:16:03.402971+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:25.501901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:14:25.501901+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":698,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401627+0000","last_change":"2024-02-19T21:14:49.321920+0000","last_active":"2024-02-19T21:16:03.401627+0000","last_peered":"2024-02-19T21:16:03.401627+0000","last_clean":"2024-02-19T21:16:03.401627+0000","last_became_active":"2024-02-19T21:14:41.403469+0000","last_became_peered":"2024-02-19T21:14:41.403469+0000","last_unstale":"2024-02-19T21:16:03.401627+0000","last_undegraded":"2024-02-19T21:16:03.401627+0000","last_fullsized":"2024-02-19T21:16:03.401627+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:49.321813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:49.321813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":688,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406664+0000","last_change":"2024-02-19T21:14:12.566913+0000","last_active":"2024-02-19T21:16:02.406664+0000","last_peered":"2024-02-19T21:16:02.406664+0000","last_clean":"2024-02-19T21:16:02.406664+0000","last_became_active":"2024-02-19T21:08:04.265081+0000","last_became_peered":"2024-02-19T21:08:04.265081+0000","last_unstale":"2024-02-19T21:16:02.406664+0000","last_undegraded":"2024-02-19T21:16:02.406664+0000","last_fullsized":"2024-02-19T21:16:02.406664+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:12.566808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:53.964703+0000","last_clean_scrub_stamp":"2024-02-19T21:14:12.566808+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":648,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404056+0000","last_change":"2024-02-19T21:14:59.927082+0000","last_active":"2024-02-19T21:16:02.404056+0000","last_peered":"2024-02-19T21:16:02.404056+0000","last_clean":"2024-02-19T21:16:02.404056+0000","last_became_active":"2024-02-19T21:10:41.317284+0000","last_became_peered":"2024-02-19T21:10:41.317284+0000","last_unstale":"2024-02-19T21:16:02.404056+0000","last_undegraded":"2024-02-19T21:16:02.404056+0000","last_fullsized":"2024-02-19T21:16:02.404056+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:59.927001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:59.927001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"662'39","reported_seq":344,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873658+0000","last_change":"2024-02-19T21:15:06.578639+0000","last_active":"2024-02-19T21:16:03.873658+0000","last_peered":"2024-02-19T21:16:03.873658+0000","last_clean":"2024-02-19T21:16:03.873658+0000","last_became_active":"2024-02-19T21:10:41.313853+0000","last_became_peered":"2024-02-19T21:10:41.313853+0000","last_unstale":"2024-02-19T21:16:03.873658+0000","last_undegraded":"2024-02-19T21:16:03.873658+0000","last_fullsized":"2024-02-19T21:16:03.873658+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"662'39","last_scrub_stamp":"2024-02-19T21:15:06.578559+0000","last_deep_scrub":"662'39","last_deep_scrub_stamp":"2024-02-19T21:15:06.578559+0000","last_clean_scrub_stamp":"2024-02-19T21:15:06.578559+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"655'2","reported_seq":714,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404684+0000","last_change":"2024-02-19T21:15:00.963681+0000","last_active":"2024-02-19T21:16:02.404684+0000","last_peered":"2024-02-19T21:16:02.404684+0000","last_clean":"2024-02-19T21:16:02.404684+0000","last_became_active":"2024-02-19T21:10:41.307517+0000","last_became_peered":"2024-02-19T21:10:41.307517+0000","last_unstale":"2024-02-19T21:16:02.404684+0000","last_undegraded":"2024-02-19T21:16:02.404684+0000","last_fullsized":"2024-02-19T21:16:02.404684+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"655'2","last_scrub_stamp":"2024-02-19T21:15:00.963594+0000","last_deep_scrub":"655'2","last_deep_scrub_stamp":"2024-02-19T21:15:00.963594+0000","last_clean_scrub_stamp":"2024-02-19T21:15:00.963594+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":735,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405590+0000","last_change":"2024-02-19T21:15:51.016985+0000","last_active":"2024-02-19T21:16:02.405590+0000","last_peered":"2024-02-19T21:16:02.405590+0000","last_clean":"2024-02-19T21:16:02.405590+0000","last_became_active":"2024-02-19T21:14:41.391280+0000","last_became_peered":"2024-02-19T21:14:41.391280+0000","last_unstale":"2024-02-19T21:16:02.405590+0000","last_undegraded":"2024-02-19T21:16:02.405590+0000","last_fullsized":"2024-02-19T21:16:02.405590+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:51.016885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:51.016885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":717,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403633+0000","last_change":"2024-02-19T21:15:42.266670+0000","last_active":"2024-02-19T21:16:03.403633+0000","last_peered":"2024-02-19T21:16:03.403633+0000","last_clean":"2024-02-19T21:16:03.403633+0000","last_became_active":"2024-02-19T21:10:41.307577+0000","last_became_peered":"2024-02-19T21:10:41.307577+0000","last_unstale":"2024-02-19T21:16:03.403633+0000","last_undegraded":"2024-02-19T21:16:03.403633+0000","last_fullsized":"2024-02-19T21:16:03.403633+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:42.266585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:15:42.266585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":297,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873021+0000","last_change":"2024-02-19T21:14:53.606588+0000","last_active":"2024-02-19T21:16:03.873021+0000","last_peered":"2024-02-19T21:16:03.873021+0000","last_clean":"2024-02-19T21:16:03.873021+0000","last_became_active":"2024-02-19T21:10:41.302670+0000","last_became_peered":"2024-02-19T21:10:41.302670+0000","last_unstale":"2024-02-19T21:16:03.873021+0000","last_undegraded":"2024-02-19T21:16:03.873021+0000","last_fullsized":"2024-02-19T21:16:03.873021+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:53.606474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:53.606474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"459'25","reported_seq":361,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873280+0000","last_change":"2024-02-19T21:14:16.417781+0000","last_active":"2024-02-19T21:16:03.873280+0000","last_peered":"2024-02-19T21:16:03.873280+0000","last_clean":"2024-02-19T21:16:03.873280+0000","last_became_active":"2024-02-19T21:14:16.381198+0000","last_became_peered":"2024-02-19T21:14:16.381198+0000","last_unstale":"2024-02-19T21:16:03.873280+0000","last_undegraded":"2024-02-19T21:16:03.873280+0000","last_fullsized":"2024-02-19T21:16:03.873280+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'25","last_scrub_stamp":"2024-02-19T21:12:51.420403+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:12:51.420403+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":701,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.403379+0000","last_change":"2024-02-19T21:13:38.073731+0000","last_active":"2024-02-19T21:16:02.403379+0000","last_peered":"2024-02-19T21:16:02.403379+0000","last_clean":"2024-02-19T21:16:02.403379+0000","last_became_active":"2024-02-19T21:08:04.268406+0000","last_became_peered":"2024-02-19T21:08:04.268406+0000","last_unstale":"2024-02-19T21:16:02.403379+0000","last_undegraded":"2024-02-19T21:16:02.403379+0000","last_fullsized":"2024-02-19T21:16:02.403379+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:38.073668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:38.073668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"65'8","reported_seq":869,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-19T21:16:04.404533+0000","last_change":"2024-02-19T21:14:58.303172+0000","last_active":"2024-02-19T21:16:04.404533+0000","last_peered":"2024-02-19T21:16:04.404533+0000","last_clean":"2024-02-19T21:16:04.404533+0000","last_became_active":"2024-02-19T21:01:20.282566+0000","last_became_peered":"2024-02-19T21:01:20.282566+0000","last_unstale":"2024-02-19T21:16:04.404533+0000","last_undegraded":"2024-02-19T21:16:04.404533+0000","last_fullsized":"2024-02-19T21:16:04.404533+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:14:58.303098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:58.303098+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":292,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873691+0000","last_change":"2024-02-19T21:12:21.380805+0000","last_active":"2024-02-19T21:16:03.873691+0000","last_peered":"2024-02-19T21:16:03.873691+0000","last_clean":"2024-02-19T21:16:03.873691+0000","last_became_active":"2024-02-19T21:10:41.298893+0000","last_became_peered":"2024-02-19T21:10:41.298893+0000","last_unstale":"2024-02-19T21:16:03.873691+0000","last_undegraded":"2024-02-19T21:16:03.873691+0000","last_fullsized":"2024-02-19T21:16:03.873691+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:21.380734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:12:21.380734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":456,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402821+0000","last_change":"2024-02-19T21:15:49.195649+0000","last_active":"2024-02-19T21:16:03.402821+0000","last_peered":"2024-02-19T21:16:03.402821+0000","last_clean":"2024-02-19T21:16:03.402821+0000","last_became_active":"2024-02-19T21:14:41.382117+0000","last_became_peered":"2024-02-19T21:14:41.382117+0000","last_unstale":"2024-02-19T21:16:03.402821+0000","last_undegraded":"2024-02-19T21:16:03.402821+0000","last_fullsized":"2024-02-19T21:16:03.402821+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:49.195586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:49.195586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"459'27","reported_seq":339,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403335+0000","last_change":"2024-02-19T21:14:22.521813+0000","last_active":"2024-02-19T21:16:03.403335+0000","last_peered":"2024-02-19T21:16:03.403335+0000","last_clean":"2024-02-19T21:16:03.403335+0000","last_became_active":"2024-02-19T21:10:41.307576+0000","last_became_peered":"2024-02-19T21:10:41.307576+0000","last_unstale":"2024-02-19T21:16:03.403335+0000","last_undegraded":"2024-02-19T21:16:03.403335+0000","last_fullsized":"2024-02-19T21:16:03.403335+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'27","last_scrub_stamp":"2024-02-19T21:14:22.521746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:22.521746+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":743,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405326+0000","last_change":"2024-02-19T21:14:55.003998+0000","last_active":"2024-02-19T21:16:02.405326+0000","last_peered":"2024-02-19T21:16:02.405326+0000","last_clean":"2024-02-19T21:16:02.405326+0000","last_became_active":"2024-02-19T21:12:40.369640+0000","last_became_peered":"2024-02-19T21:12:40.369640+0000","last_unstale":"2024-02-19T21:16:02.405326+0000","last_undegraded":"2024-02-19T21:16:02.405326+0000","last_fullsized":"2024-02-19T21:16:02.405326+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:55.003931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:55.003931+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"640'12","reported_seq":388,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-19T21:16:04.739489+0000","last_change":"2024-02-19T21:14:41.540201+0000","last_active":"2024-02-19T21:16:04.739489+0000","last_peered":"2024-02-19T21:16:04.739489+0000","last_clean":"2024-02-19T21:16:04.739489+0000","last_became_active":"2024-02-19T21:14:41.415828+0000","last_became_peered":"2024-02-19T21:14:41.415828+0000","last_unstale":"2024-02-19T21:16:04.739489+0000","last_undegraded":"2024-02-19T21:16:04.739489+0000","last_fullsized":"2024-02-19T21:16:04.739489+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"483'11","last_scrub_stamp":"2024-02-19T21:12:56.450044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:12:56.450044+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":701,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406531+0000","last_change":"2024-02-19T21:15:37.011347+0000","last_active":"2024-02-19T21:16:02.406531+0000","last_peered":"2024-02-19T21:16:02.406531+0000","last_clean":"2024-02-19T21:16:02.406531+0000","last_became_active":"2024-02-19T21:14:41.404281+0000","last_became_peered":"2024-02-19T21:14:41.404281+0000","last_unstale":"2024-02-19T21:16:02.406531+0000","last_undegraded":"2024-02-19T21:16:02.406531+0000","last_fullsized":"2024-02-19T21:16:02.406531+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:37.011259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:37.011259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":650,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405940+0000","last_change":"2024-02-19T21:14:50.038933+0000","last_active":"2024-02-19T21:16:02.405940+0000","last_peered":"2024-02-19T21:16:02.405940+0000","last_clean":"2024-02-19T21:16:02.405940+0000","last_became_active":"2024-02-19T21:14:16.397268+0000","last_became_peered":"2024-02-19T21:14:16.397268+0000","last_unstale":"2024-02-19T21:16:02.405940+0000","last_undegraded":"2024-02-19T21:16:02.405940+0000","last_fullsized":"2024-02-19T21:16:02.405940+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:50.038866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:50.038866+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"459'19","reported_seq":365,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873642+0000","last_change":"2024-02-19T21:15:03.572770+0000","last_active":"2024-02-19T21:16:03.873642+0000","last_peered":"2024-02-19T21:16:03.873642+0000","last_clean":"2024-02-19T21:16:03.873642+0000","last_became_active":"2024-02-19T21:14:16.382074+0000","last_became_peered":"2024-02-19T21:14:16.382074+0000","last_unstale":"2024-02-19T21:16:03.873642+0000","last_undegraded":"2024-02-19T21:16:03.873642+0000","last_fullsized":"2024-02-19T21:16:03.873642+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:15:03.572687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:03.572687+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":740,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406810+0000","last_change":"2024-02-19T21:15:35.944834+0000","last_active":"2024-02-19T21:16:02.406810+0000","last_peered":"2024-02-19T21:16:02.406810+0000","last_clean":"2024-02-19T21:16:02.406810+0000","last_became_active":"2024-02-19T21:14:41.382375+0000","last_became_peered":"2024-02-19T21:14:41.382375+0000","last_unstale":"2024-02-19T21:16:02.406810+0000","last_undegraded":"2024-02-19T21:16:02.406810+0000","last_fullsized":"2024-02-19T21:16:02.406810+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:35.944741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:35.944741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":715,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.407220+0000","last_change":"2024-02-19T21:14:20.914502+0000","last_active":"2024-02-19T21:16:02.407220+0000","last_peered":"2024-02-19T21:16:02.407220+0000","last_clean":"2024-02-19T21:16:02.407220+0000","last_became_active":"2024-02-19T21:14:16.393359+0000","last_became_peered":"2024-02-19T21:14:16.393359+0000","last_unstale":"2024-02-19T21:16:02.407220+0000","last_undegraded":"2024-02-19T21:16:02.407220+0000","last_fullsized":"2024-02-19T21:16:02.407220+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:20.914421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:20.914421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":449,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.404118+0000","last_change":"2024-02-19T21:14:16.385816+0000","last_active":"2024-02-19T21:16:03.404118+0000","last_peered":"2024-02-19T21:16:03.404118+0000","last_clean":"2024-02-19T21:16:03.404118+0000","last_became_active":"2024-02-19T21:14:16.385474+0000","last_became_peered":"2024-02-19T21:14:16.385474+0000","last_unstale":"2024-02-19T21:16:03.404118+0000","last_undegraded":"2024-02-19T21:16:03.404118+0000","last_fullsized":"2024-02-19T21:16:03.404118+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:51.217370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:51.217370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":449,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403614+0000","last_change":"2024-02-19T21:15:38.121570+0000","last_active":"2024-02-19T21:16:03.403614+0000","last_peered":"2024-02-19T21:16:03.403614+0000","last_clean":"2024-02-19T21:16:03.403614+0000","last_became_active":"2024-02-19T21:14:16.383877+0000","last_became_peered":"2024-02-19T21:14:16.383877+0000","last_unstale":"2024-02-19T21:16:03.403614+0000","last_undegraded":"2024-02-19T21:16:03.403614+0000","last_fullsized":"2024-02-19T21:16:03.403614+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:38.121453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:38.121453+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"434'12","reported_seq":561,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-19T21:16:04.402348+0000","last_change":"2024-02-19T21:13:47.189879+0000","last_active":"2024-02-19T21:16:04.402348+0000","last_peered":"2024-02-19T21:16:04.402348+0000","last_clean":"2024-02-19T21:16:04.402348+0000","last_became_active":"2024-02-19T21:10:41.297897+0000","last_became_peered":"2024-02-19T21:10:41.297897+0000","last_unstale":"2024-02-19T21:16:04.402348+0000","last_undegraded":"2024-02-19T21:16:04.402348+0000","last_fullsized":"2024-02-19T21:16:04.402348+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'12","last_scrub_stamp":"2024-02-19T21:13:47.189812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:13:47.189812+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"459'28","reported_seq":384,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873074+0000","last_change":"2024-02-19T21:14:16.413467+0000","last_active":"2024-02-19T21:16:03.873074+0000","last_peered":"2024-02-19T21:16:03.873074+0000","last_clean":"2024-02-19T21:16:03.873074+0000","last_became_active":"2024-02-19T21:14:16.380694+0000","last_became_peered":"2024-02-19T21:14:16.380694+0000","last_unstale":"2024-02-19T21:16:03.873074+0000","last_undegraded":"2024-02-19T21:16:03.873074+0000","last_fullsized":"2024-02-19T21:16:03.873074+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'28","last_scrub_stamp":"2024-02-19T21:13:46.758735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:13:46.758735+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":107,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.456978+0000","last_change":"2024-02-19T21:15:50.164521+0000","last_active":"2024-02-19T21:16:03.456978+0000","last_peered":"2024-02-19T21:16:03.456978+0000","last_clean":"2024-02-19T21:16:03.456978+0000","last_became_active":"2024-02-19T21:14:16.373120+0000","last_became_peered":"2024-02-19T21:14:16.373120+0000","last_unstale":"2024-02-19T21:16:03.456978+0000","last_undegraded":"2024-02-19T21:16:03.456978+0000","last_fullsized":"2024-02-19T21:16:03.456978+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:50.164434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:50.164434+0000","last_clean_scrub_stamp":"2024-02-19T21:15:50.164434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":692,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404892+0000","last_change":"2024-02-19T21:14:16.375816+0000","last_active":"2024-02-19T21:16:02.404892+0000","last_peered":"2024-02-19T21:16:02.404892+0000","last_clean":"2024-02-19T21:16:02.404892+0000","last_became_active":"2024-02-19T21:14:16.375486+0000","last_became_peered":"2024-02-19T21:14:16.375486+0000","last_unstale":"2024-02-19T21:16:02.404892+0000","last_undegraded":"2024-02-19T21:16:02.404892+0000","last_fullsized":"2024-02-19T21:16:02.404892+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:49.749137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:13:49.749137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":529,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405301+0000","last_change":"2024-02-19T21:15:49.983230+0000","last_active":"2024-02-19T21:16:02.405301+0000","last_peered":"2024-02-19T21:16:02.405301+0000","last_clean":"2024-02-19T21:16:02.405301+0000","last_became_active":"2024-02-19T21:14:41.391010+0000","last_became_peered":"2024-02-19T21:14:41.391010+0000","last_unstale":"2024-02-19T21:16:02.405301+0000","last_undegraded":"2024-02-19T21:16:02.405301+0000","last_fullsized":"2024-02-19T21:16:02.405301+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:49.983125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:11.004006+0000","last_clean_scrub_stamp":"2024-02-19T21:15:49.983125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":297,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.872911+0000","last_change":"2024-02-19T21:14:54.655007+0000","last_active":"2024-02-19T21:16:03.872911+0000","last_peered":"2024-02-19T21:16:03.872911+0000","last_clean":"2024-02-19T21:16:03.872911+0000","last_became_active":"2024-02-19T21:10:41.304695+0000","last_became_peered":"2024-02-19T21:10:41.304695+0000","last_unstale":"2024-02-19T21:16:03.872911+0000","last_undegraded":"2024-02-19T21:16:03.872911+0000","last_fullsized":"2024-02-19T21:16:03.872911+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:54.654916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:14:54.654916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"459'30","reported_seq":521,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402920+0000","last_change":"2024-02-19T21:14:52.068847+0000","last_active":"2024-02-19T21:16:03.402920+0000","last_peered":"2024-02-19T21:16:03.402920+0000","last_clean":"2024-02-19T21:16:03.402920+0000","last_became_active":"2024-02-19T21:14:41.403514+0000","last_became_peered":"2024-02-19T21:14:41.403514+0000","last_unstale":"2024-02-19T21:16:03.402920+0000","last_undegraded":"2024-02-19T21:16:03.402920+0000","last_fullsized":"2024-02-19T21:16:03.402920+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'30","last_scrub_stamp":"2024-02-19T21:14:52.068777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:52.068777+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":747,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.403707+0000","last_change":"2024-02-19T21:15:48.962320+0000","last_active":"2024-02-19T21:16:02.403707+0000","last_peered":"2024-02-19T21:16:02.403707+0000","last_clean":"2024-02-19T21:16:02.403707+0000","last_became_active":"2024-02-19T21:14:16.379640+0000","last_became_peered":"2024-02-19T21:14:16.379640+0000","last_unstale":"2024-02-19T21:16:02.403707+0000","last_undegraded":"2024-02-19T21:16:02.403707+0000","last_fullsized":"2024-02-19T21:16:02.403707+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:48.962240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:48.962240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":297,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873140+0000","last_change":"2024-02-19T21:15:44.842959+0000","last_active":"2024-02-19T21:16:03.873140+0000","last_peered":"2024-02-19T21:16:03.873140+0000","last_clean":"2024-02-19T21:16:03.873140+0000","last_became_active":"2024-02-19T21:10:41.295301+0000","last_became_peered":"2024-02-19T21:10:41.295301+0000","last_unstale":"2024-02-19T21:16:03.873140+0000","last_undegraded":"2024-02-19T21:16:03.873140+0000","last_fullsized":"2024-02-19T21:16:03.873140+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:44.842888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:15:44.842888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":467,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.404204+0000","last_change":"2024-02-19T21:15:39.146832+0000","last_active":"2024-02-19T21:16:03.404204+0000","last_peered":"2024-02-19T21:16:03.404204+0000","last_clean":"2024-02-19T21:16:03.404204+0000","last_became_active":"2024-02-19T21:14:41.398917+0000","last_became_peered":"2024-02-19T21:14:41.398917+0000","last_unstale":"2024-02-19T21:16:03.404204+0000","last_undegraded":"2024-02-19T21:16:03.404204+0000","last_fullsized":"2024-02-19T21:16:03.404204+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:39.146761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:39.146761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":116,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.457721+0000","last_change":"2024-02-19T21:15:49.141891+0000","last_active":"2024-02-19T21:16:03.457721+0000","last_peered":"2024-02-19T21:16:03.457721+0000","last_clean":"2024-02-19T21:16:03.457721+0000","last_became_active":"2024-02-19T21:14:41.395152+0000","last_became_peered":"2024-02-19T21:14:41.395152+0000","last_unstale":"2024-02-19T21:16:03.457721+0000","last_undegraded":"2024-02-19T21:16:03.457721+0000","last_fullsized":"2024-02-19T21:16:03.457721+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:49.141780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:49.141780+0000","last_clean_scrub_stamp":"2024-02-19T21:15:49.141780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"459'29","reported_seq":632,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406202+0000","last_change":"2024-02-19T21:15:42.003524+0000","last_active":"2024-02-19T21:16:02.406202+0000","last_peered":"2024-02-19T21:16:02.406202+0000","last_clean":"2024-02-19T21:16:02.406202+0000","last_became_active":"2024-02-19T21:14:41.404443+0000","last_became_peered":"2024-02-19T21:14:41.404443+0000","last_unstale":"2024-02-19T21:16:02.406202+0000","last_undegraded":"2024-02-19T21:16:02.406202+0000","last_fullsized":"2024-02-19T21:16:02.406202+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'29","last_scrub_stamp":"2024-02-19T21:15:42.003405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:42.003405+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":760,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405411+0000","last_change":"2024-02-19T21:15:41.880159+0000","last_active":"2024-02-19T21:16:02.405411+0000","last_peered":"2024-02-19T21:16:02.405411+0000","last_clean":"2024-02-19T21:16:02.405411+0000","last_became_active":"2024-02-19T21:14:41.415341+0000","last_became_peered":"2024-02-19T21:14:41.415341+0000","last_unstale":"2024-02-19T21:16:02.405411+0000","last_undegraded":"2024-02-19T21:16:02.405411+0000","last_fullsized":"2024-02-19T21:16:02.405411+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:41.880077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:41.880077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":692,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405397+0000","last_change":"2024-02-19T21:15:36.830480+0000","last_active":"2024-02-19T21:16:02.405397+0000","last_peered":"2024-02-19T21:16:02.405397+0000","last_clean":"2024-02-19T21:16:02.405397+0000","last_became_active":"2024-02-19T21:14:16.375008+0000","last_became_peered":"2024-02-19T21:14:16.375008+0000","last_unstale":"2024-02-19T21:16:02.405397+0000","last_undegraded":"2024-02-19T21:16:02.405397+0000","last_fullsized":"2024-02-19T21:16:02.405397+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:36.830383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:15:36.830383+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":648,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403946+0000","last_change":"2024-02-19T21:14:41.404931+0000","last_active":"2024-02-19T21:16:03.403946+0000","last_peered":"2024-02-19T21:16:03.403946+0000","last_clean":"2024-02-19T21:16:03.403946+0000","last_became_active":"2024-02-19T21:14:41.403823+0000","last_became_peered":"2024-02-19T21:14:41.403823+0000","last_unstale":"2024-02-19T21:16:03.403946+0000","last_undegraded":"2024-02-19T21:16:03.403946+0000","last_fullsized":"2024-02-19T21:16:03.403946+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:21.483828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_clean_scrub_stamp":"2024-02-19T21:14:21.483828+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":697,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406989+0000","last_change":"2024-02-19T21:14:57.083065+0000","last_active":"2024-02-19T21:16:02.406989+0000","last_peered":"2024-02-19T21:16:02.406989+0000","last_clean":"2024-02-19T21:16:02.406989+0000","last_became_active":"2024-02-19T21:14:41.404337+0000","last_became_peered":"2024-02-19T21:14:41.404337+0000","last_unstale":"2024-02-19T21:16:02.406989+0000","last_undegraded":"2024-02-19T21:16:02.406989+0000","last_fullsized":"2024-02-19T21:16:02.406989+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:57.082987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:57.082987+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"459'24","reported_seq":605,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404000+0000","last_change":"2024-02-19T21:15:45.957885+0000","last_active":"2024-02-19T21:16:02.404000+0000","last_peered":"2024-02-19T21:16:02.404000+0000","last_clean":"2024-02-19T21:16:02.404000+0000","last_became_active":"2024-02-19T21:14:41.415655+0000","last_became_peered":"2024-02-19T21:14:41.415655+0000","last_unstale":"2024-02-19T21:16:02.404000+0000","last_undegraded":"2024-02-19T21:16:02.404000+0000","last_fullsized":"2024-02-19T21:16:02.404000+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'24","last_scrub_stamp":"2024-02-19T21:15:45.957817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:45.957817+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"397'2","reported_seq":771,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406724+0000","last_change":"2024-02-19T21:15:50.915397+0000","last_active":"2024-02-19T21:16:02.406724+0000","last_peered":"2024-02-19T21:16:02.406724+0000","last_clean":"2024-02-19T21:16:02.406724+0000","last_became_active":"2024-02-19T21:14:44.370071+0000","last_became_peered":"2024-02-19T21:14:44.370071+0000","last_unstale":"2024-02-19T21:16:02.406724+0000","last_undegraded":"2024-02-19T21:16:02.406724+0000","last_fullsized":"2024-02-19T21:16:02.406724+0000","mapping_epoch":639,"log_start":"58'1","ondisk_log_start":"58'1","created":57,"last_epoch_clean":640,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:15:50.915331+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:15:50.915331+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":315,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873126+0000","last_change":"2024-02-19T21:14:41.416360+0000","last_active":"2024-02-19T21:16:03.873126+0000","last_peered":"2024-02-19T21:16:03.873126+0000","last_clean":"2024-02-19T21:16:03.873126+0000","last_became_active":"2024-02-19T21:14:41.416024+0000","last_became_peered":"2024-02-19T21:14:41.416024+0000","last_unstale":"2024-02-19T21:16:03.873126+0000","last_undegraded":"2024-02-19T21:16:03.873126+0000","last_fullsized":"2024-02-19T21:16:03.873126+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.643360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.643360+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":694,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405630+0000","last_change":"2024-02-19T21:14:20.999565+0000","last_active":"2024-02-19T21:16:02.405630+0000","last_peered":"2024-02-19T21:16:02.405630+0000","last_clean":"2024-02-19T21:16:02.405630+0000","last_became_active":"2024-02-19T21:08:04.282243+0000","last_became_peered":"2024-02-19T21:08:04.282243+0000","last_unstale":"2024-02-19T21:16:02.405630+0000","last_undegraded":"2024-02-19T21:16:02.405630+0000","last_fullsized":"2024-02-19T21:16:02.405630+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:20.999473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:20.999473+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":116,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.457878+0000","last_change":"2024-02-19T21:15:47.088344+0000","last_active":"2024-02-19T21:16:03.457878+0000","last_peered":"2024-02-19T21:16:03.457878+0000","last_clean":"2024-02-19T21:16:03.457878+0000","last_became_active":"2024-02-19T21:14:41.395290+0000","last_became_peered":"2024-02-19T21:14:41.395290+0000","last_unstale":"2024-02-19T21:16:03.457878+0000","last_undegraded":"2024-02-19T21:16:03.457878+0000","last_fullsized":"2024-02-19T21:16:03.457878+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:47.088245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:47.088245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":677,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404384+0000","last_change":"2024-02-19T21:15:32.976824+0000","last_active":"2024-02-19T21:16:02.404384+0000","last_peered":"2024-02-19T21:16:02.404384+0000","last_clean":"2024-02-19T21:16:02.404384+0000","last_became_active":"2024-02-19T21:01:20.295127+0000","last_became_peered":"2024-02-19T21:01:20.295127+0000","last_unstale":"2024-02-19T21:16:02.404384+0000","last_undegraded":"2024-02-19T21:16:02.404384+0000","last_fullsized":"2024-02-19T21:16:02.404384+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:32.976749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:32.976749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"459'26","reported_seq":123,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.456705+0000","last_change":"2024-02-19T21:15:39.237327+0000","last_active":"2024-02-19T21:16:03.456705+0000","last_peered":"2024-02-19T21:16:03.456705+0000","last_clean":"2024-02-19T21:16:03.456705+0000","last_became_active":"2024-02-19T21:14:16.390782+0000","last_became_peered":"2024-02-19T21:14:16.390782+0000","last_unstale":"2024-02-19T21:16:03.456705+0000","last_undegraded":"2024-02-19T21:16:03.456705+0000","last_fullsized":"2024-02-19T21:16:03.456705+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'26","last_scrub_stamp":"2024-02-19T21:15:39.237243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:39.237243+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":721,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406741+0000","last_change":"2024-02-19T21:14:41.391190+0000","last_active":"2024-02-19T21:16:02.406741+0000","last_peered":"2024-02-19T21:16:02.406741+0000","last_clean":"2024-02-19T21:16:02.406741+0000","last_became_active":"2024-02-19T21:14:41.390836+0000","last_became_peered":"2024-02-19T21:14:41.390836+0000","last_unstale":"2024-02-19T21:16:02.406741+0000","last_undegraded":"2024-02-19T21:16:02.406741+0000","last_fullsized":"2024-02-19T21:16:02.406741+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:50.724362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:13:50.724362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":720,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406157+0000","last_change":"2024-02-19T21:14:14.957777+0000","last_active":"2024-02-19T21:16:02.406157+0000","last_peered":"2024-02-19T21:16:02.406157+0000","last_clean":"2024-02-19T21:16:02.406157+0000","last_became_active":"2024-02-19T21:11:48.359271+0000","last_became_peered":"2024-02-19T21:11:48.359271+0000","last_unstale":"2024-02-19T21:16:02.406157+0000","last_undegraded":"2024-02-19T21:16:02.406157+0000","last_fullsized":"2024-02-19T21:16:02.406157+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:14.957719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:14.957719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":449,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403658+0000","last_change":"2024-02-19T21:14:16.384177+0000","last_active":"2024-02-19T21:16:03.403658+0000","last_peered":"2024-02-19T21:16:03.403658+0000","last_clean":"2024-02-19T21:16:03.403658+0000","last_became_active":"2024-02-19T21:14:16.383858+0000","last_became_peered":"2024-02-19T21:14:16.383858+0000","last_unstale":"2024-02-19T21:16:03.403658+0000","last_undegraded":"2024-02-19T21:16:03.403658+0000","last_fullsized":"2024-02-19T21:16:03.403658+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:55.248259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.248259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":713,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.407070+0000","last_change":"2024-02-19T21:13:53.919119+0000","last_active":"2024-02-19T21:16:02.407070+0000","last_peered":"2024-02-19T21:16:02.407070+0000","last_clean":"2024-02-19T21:16:02.407070+0000","last_became_active":"2024-02-19T21:10:41.315397+0000","last_became_peered":"2024-02-19T21:10:41.315397+0000","last_unstale":"2024-02-19T21:16:02.407070+0000","last_undegraded":"2024-02-19T21:16:02.407070+0000","last_fullsized":"2024-02-19T21:16:02.407070+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:53.919024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:53.919024+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":650,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404438+0000","last_change":"2024-02-19T21:14:41.404518+0000","last_active":"2024-02-19T21:16:02.404438+0000","last_peered":"2024-02-19T21:16:02.404438+0000","last_clean":"2024-02-19T21:16:02.404438+0000","last_became_active":"2024-02-19T21:14:41.403799+0000","last_became_peered":"2024-02-19T21:14:41.403799+0000","last_unstale":"2024-02-19T21:16:02.404438+0000","last_undegraded":"2024-02-19T21:16:02.404438+0000","last_fullsized":"2024-02-19T21:16:02.404438+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:13.932020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_clean_scrub_stamp":"2024-02-19T21:14:13.932020+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":683,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402236+0000","last_change":"2024-02-19T21:15:04.348697+0000","last_active":"2024-02-19T21:16:03.402236+0000","last_peered":"2024-02-19T21:16:03.402236+0000","last_clean":"2024-02-19T21:16:03.402236+0000","last_became_active":"2024-02-19T21:01:20.283841+0000","last_became_peered":"2024-02-19T21:01:20.283841+0000","last_unstale":"2024-02-19T21:16:03.402236+0000","last_undegraded":"2024-02-19T21:16:03.402236+0000","last_fullsized":"2024-02-19T21:16:03.402236+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:04.348628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:04.348628+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"459'22","reported_seq":744,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403865+0000","last_change":"2024-02-19T21:14:41.538659+0000","last_active":"2024-02-19T21:16:03.403865+0000","last_peered":"2024-02-19T21:16:03.403865+0000","last_clean":"2024-02-19T21:16:03.403865+0000","last_became_active":"2024-02-19T21:14:41.404362+0000","last_became_peered":"2024-02-19T21:14:41.404362+0000","last_unstale":"2024-02-19T21:16:03.403865+0000","last_undegraded":"2024-02-19T21:16:03.403865+0000","last_fullsized":"2024-02-19T21:16:03.403865+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'22","last_scrub_stamp":"2024-02-19T21:14:11.471431+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:14:11.471431+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":755,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402627+0000","last_change":"2024-02-19T21:15:51.240120+0000","last_active":"2024-02-19T21:16:03.402627+0000","last_peered":"2024-02-19T21:16:03.402627+0000","last_clean":"2024-02-19T21:16:03.402627+0000","last_became_active":"2024-02-19T21:14:41.411415+0000","last_became_peered":"2024-02-19T21:14:41.411415+0000","last_unstale":"2024-02-19T21:16:03.402627+0000","last_undegraded":"2024-02-19T21:16:03.402627+0000","last_fullsized":"2024-02-19T21:16:03.402627+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:51.240046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:51.240046+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":1593,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873354+0000","last_change":"2024-02-19T21:15:44.948603+0000","last_active":"2024-02-19T21:16:03.873354+0000","last_peered":"2024-02-19T21:16:03.873354+0000","last_clean":"2024-02-19T21:16:03.873354+0000","last_became_active":"2024-02-19T21:14:16.388793+0000","last_became_peered":"2024-02-19T21:14:16.388793+0000","last_unstale":"2024-02-19T21:16:03.873354+0000","last_undegraded":"2024-02-19T21:16:03.873354+0000","last_fullsized":"2024-02-19T21:16:03.873354+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:15:44.948532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:44.948532+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":878,"num_read_kb":873,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":711,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405370+0000","last_change":"2024-02-19T21:14:56.859043+0000","last_active":"2024-02-19T21:16:02.405370+0000","last_peered":"2024-02-19T21:16:02.405370+0000","last_clean":"2024-02-19T21:16:02.405370+0000","last_became_active":"2024-02-19T21:14:16.390838+0000","last_became_peered":"2024-02-19T21:14:16.390838+0000","last_unstale":"2024-02-19T21:16:02.405370+0000","last_undegraded":"2024-02-19T21:16:02.405370+0000","last_fullsized":"2024-02-19T21:16:02.405370+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:56.858956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:56.858956+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":646,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.403909+0000","last_change":"2024-02-19T21:14:49.969389+0000","last_active":"2024-02-19T21:16:02.403909+0000","last_peered":"2024-02-19T21:16:02.403909+0000","last_clean":"2024-02-19T21:16:02.403909+0000","last_became_active":"2024-02-19T21:14:41.391420+0000","last_became_peered":"2024-02-19T21:14:41.391420+0000","last_unstale":"2024-02-19T21:16:02.403909+0000","last_undegraded":"2024-02-19T21:16:02.403909+0000","last_fullsized":"2024-02-19T21:16:02.403909+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:49.969308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_clean_scrub_stamp":"2024-02-19T21:14:49.969308+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":565,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404006+0000","last_change":"2024-02-19T21:14:55.995419+0000","last_active":"2024-02-19T21:16:02.404006+0000","last_peered":"2024-02-19T21:16:02.404006+0000","last_clean":"2024-02-19T21:16:02.404006+0000","last_became_active":"2024-02-19T21:14:41.390554+0000","last_became_peered":"2024-02-19T21:14:41.390554+0000","last_unstale":"2024-02-19T21:16:02.404006+0000","last_undegraded":"2024-02-19T21:16:02.404006+0000","last_fullsized":"2024-02-19T21:16:02.404006+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:55.995348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:14:55.995348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"459'42","reported_seq":984,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.875677+0000","last_change":"2024-02-19T21:13:47.032674+0000","last_active":"2024-02-19T21:16:03.875677+0000","last_peered":"2024-02-19T21:16:03.875677+0000","last_clean":"2024-02-19T21:16:03.875677+0000","last_became_active":"2024-02-19T21:08:04.266322+0000","last_became_peered":"2024-02-19T21:08:04.266322+0000","last_unstale":"2024-02-19T21:16:03.875677+0000","last_undegraded":"2024-02-19T21:16:03.875677+0000","last_fullsized":"2024-02-19T21:16:03.875677+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'42","last_scrub_stamp":"2024-02-19T21:13:47.032604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:13:47.032604+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":738,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.409096+0000","last_change":"2024-02-19T21:15:07.044813+0000","last_active":"2024-02-19T21:16:02.409096+0000","last_peered":"2024-02-19T21:16:02.409096+0000","last_clean":"2024-02-19T21:16:02.409096+0000","last_became_active":"2024-02-19T21:14:16.395604+0000","last_became_peered":"2024-02-19T21:14:16.395604+0000","last_unstale":"2024-02-19T21:16:02.409096+0000","last_undegraded":"2024-02-19T21:16:02.409096+0000","last_fullsized":"2024-02-19T21:16:02.409096+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:07.044737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:15:07.044737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":449,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401735+0000","last_change":"2024-02-19T21:15:35.131877+0000","last_active":"2024-02-19T21:16:03.401735+0000","last_peered":"2024-02-19T21:16:03.401735+0000","last_clean":"2024-02-19T21:16:03.401735+0000","last_became_active":"2024-02-19T21:10:41.305829+0000","last_became_peered":"2024-02-19T21:10:41.305829+0000","last_unstale":"2024-02-19T21:16:03.401735+0000","last_undegraded":"2024-02-19T21:16:03.401735+0000","last_fullsized":"2024-02-19T21:16:03.401735+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:35.131770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:35.131770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":648,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401398+0000","last_change":"2024-02-19T21:14:41.407695+0000","last_active":"2024-02-19T21:16:03.401398+0000","last_peered":"2024-02-19T21:16:03.401398+0000","last_clean":"2024-02-19T21:16:03.401398+0000","last_became_active":"2024-02-19T21:14:41.403835+0000","last_became_peered":"2024-02-19T21:14:41.403835+0000","last_unstale":"2024-02-19T21:16:03.401398+0000","last_undegraded":"2024-02-19T21:16:03.401398+0000","last_fullsized":"2024-02-19T21:16:03.401398+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:51.499300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:13:51.499300+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"640'14","reported_seq":146,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-19T21:16:04.403933+0000","last_change":"2024-02-19T21:15:43.163121+0000","last_active":"2024-02-19T21:16:04.403933+0000","last_peered":"2024-02-19T21:16:04.403933+0000","last_clean":"2024-02-19T21:16:04.403933+0000","last_became_active":"2024-02-19T21:14:41.395291+0000","last_became_peered":"2024-02-19T21:14:41.395291+0000","last_unstale":"2024-02-19T21:16:04.403933+0000","last_undegraded":"2024-02-19T21:16:04.403933+0000","last_fullsized":"2024-02-19T21:16:04.403933+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"640'14","last_scrub_stamp":"2024-02-19T21:15:43.163004+0000","last_deep_scrub":"471'10","last_deep_scrub_stamp":"2024-02-19T21:11:41.775418+0000","last_clean_scrub_stamp":"2024-02-19T21:15:43.163004+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"459'20","reported_seq":750,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.403471+0000","last_change":"2024-02-19T21:15:38.900504+0000","last_active":"2024-02-19T21:16:02.403471+0000","last_peered":"2024-02-19T21:16:02.403471+0000","last_clean":"2024-02-19T21:16:02.403471+0000","last_became_active":"2024-02-19T21:10:41.296275+0000","last_became_peered":"2024-02-19T21:10:41.296275+0000","last_unstale":"2024-02-19T21:16:02.403471+0000","last_undegraded":"2024-02-19T21:16:02.403471+0000","last_fullsized":"2024-02-19T21:16:02.403471+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:15:38.900433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:38.900433+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":752,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405440+0000","last_change":"2024-02-19T21:14:41.394241+0000","last_active":"2024-02-19T21:16:02.405440+0000","last_peered":"2024-02-19T21:16:02.405440+0000","last_clean":"2024-02-19T21:16:02.405440+0000","last_became_active":"2024-02-19T21:14:41.391704+0000","last_became_peered":"2024-02-19T21:14:41.391704+0000","last_unstale":"2024-02-19T21:16:02.405440+0000","last_undegraded":"2024-02-19T21:16:02.405440+0000","last_fullsized":"2024-02-19T21:16:02.405440+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:48.003444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:13:48.003444+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":449,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402872+0000","last_change":"2024-02-19T21:14:41.383030+0000","last_active":"2024-02-19T21:16:03.402872+0000","last_peered":"2024-02-19T21:16:03.402872+0000","last_clean":"2024-02-19T21:16:03.402872+0000","last_became_active":"2024-02-19T21:14:41.382529+0000","last_became_peered":"2024-02-19T21:14:41.382529+0000","last_unstale":"2024-02-19T21:16:03.402872+0000","last_undegraded":"2024-02-19T21:16:03.402872+0000","last_fullsized":"2024-02-19T21:16:03.402872+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:07.221594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.221594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":449,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401777+0000","last_change":"2024-02-19T21:15:03.219533+0000","last_active":"2024-02-19T21:16:03.401777+0000","last_peered":"2024-02-19T21:16:03.401777+0000","last_clean":"2024-02-19T21:16:03.401777+0000","last_became_active":"2024-02-19T21:14:16.383904+0000","last_became_peered":"2024-02-19T21:14:16.383904+0000","last_unstale":"2024-02-19T21:16:03.401777+0000","last_undegraded":"2024-02-19T21:16:03.401777+0000","last_fullsized":"2024-02-19T21:16:03.401777+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:03.219469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:03.219469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":701,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401246+0000","last_change":"2024-02-19T21:15:40.357385+0000","last_active":"2024-02-19T21:16:03.401246+0000","last_peered":"2024-02-19T21:16:03.401246+0000","last_clean":"2024-02-19T21:16:03.401246+0000","last_became_active":"2024-02-19T21:14:41.404657+0000","last_became_peered":"2024-02-19T21:14:41.404657+0000","last_unstale":"2024-02-19T21:16:03.401246+0000","last_undegraded":"2024-02-19T21:16:03.401246+0000","last_fullsized":"2024-02-19T21:16:03.401246+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:40.357299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:15:40.357299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"459'23","reported_seq":318,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.872974+0000","last_change":"2024-02-19T21:15:43.850016+0000","last_active":"2024-02-19T21:16:03.872974+0000","last_peered":"2024-02-19T21:16:03.872974+0000","last_clean":"2024-02-19T21:16:03.872974+0000","last_became_active":"2024-02-19T21:10:41.287134+0000","last_became_peered":"2024-02-19T21:10:41.287134+0000","last_unstale":"2024-02-19T21:16:03.872974+0000","last_undegraded":"2024-02-19T21:16:03.872974+0000","last_fullsized":"2024-02-19T21:16:03.872974+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'23","last_scrub_stamp":"2024-02-19T21:15:43.849916+0000","last_deep_scrub":"420'17","last_deep_scrub_stamp":"2024-02-19T21:10:19.722303+0000","last_clean_scrub_stamp":"2024-02-19T21:15:43.849916+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":333,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873817+0000","last_change":"2024-02-19T21:14:49.574097+0000","last_active":"2024-02-19T21:16:03.873817+0000","last_peered":"2024-02-19T21:16:03.873817+0000","last_clean":"2024-02-19T21:16:03.873817+0000","last_became_active":"2024-02-19T21:14:41.391555+0000","last_became_peered":"2024-02-19T21:14:41.391555+0000","last_unstale":"2024-02-19T21:16:03.873817+0000","last_undegraded":"2024-02-19T21:16:03.873817+0000","last_fullsized":"2024-02-19T21:16:03.873817+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:14:49.574024+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:14:49.574024+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":436,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403274+0000","last_change":"2024-02-19T21:15:56.087923+0000","last_active":"2024-02-19T21:16:03.403274+0000","last_peered":"2024-02-19T21:16:03.403274+0000","last_clean":"2024-02-19T21:16:03.403274+0000","last_became_active":"2024-02-19T21:08:04.264593+0000","last_became_peered":"2024-02-19T21:08:04.264593+0000","last_unstale":"2024-02-19T21:16:03.403274+0000","last_undegraded":"2024-02-19T21:16:03.403274+0000","last_fullsized":"2024-02-19T21:16:03.403274+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:56.087844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:15:56.087844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":630,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402390+0000","last_change":"2024-02-19T21:13:44.567327+0000","last_active":"2024-02-19T21:16:03.402390+0000","last_peered":"2024-02-19T21:16:03.402390+0000","last_clean":"2024-02-19T21:16:03.402390+0000","last_became_active":"2024-02-19T21:04:39.696243+0000","last_became_peered":"2024-02-19T21:04:39.696243+0000","last_unstale":"2024-02-19T21:16:03.402390+0000","last_undegraded":"2024-02-19T21:16:03.402390+0000","last_fullsized":"2024-02-19T21:16:03.402390+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:44.567199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:13:44.567199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"459'15","reported_seq":490,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401692+0000","last_change":"2024-02-19T21:14:41.794436+0000","last_active":"2024-02-19T21:16:03.401692+0000","last_peered":"2024-02-19T21:16:03.401692+0000","last_clean":"2024-02-19T21:16:03.401692+0000","last_became_active":"2024-02-19T21:14:41.401051+0000","last_became_peered":"2024-02-19T21:14:41.401051+0000","last_unstale":"2024-02-19T21:16:03.401692+0000","last_undegraded":"2024-02-19T21:16:03.401692+0000","last_fullsized":"2024-02-19T21:16:03.401692+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:14:21.960125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:21.960125+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":700,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401239+0000","last_change":"2024-02-19T21:14:41.405830+0000","last_active":"2024-02-19T21:16:03.401239+0000","last_peered":"2024-02-19T21:16:03.401239+0000","last_clean":"2024-02-19T21:16:03.401239+0000","last_became_active":"2024-02-19T21:14:41.404263+0000","last_became_peered":"2024-02-19T21:14:41.404263+0000","last_unstale":"2024-02-19T21:16:03.401239+0000","last_undegraded":"2024-02-19T21:16:03.401239+0000","last_fullsized":"2024-02-19T21:16:03.401239+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:12.482232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:12.482232+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":431,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402467+0000","last_change":"2024-02-19T21:14:56.048540+0000","last_active":"2024-02-19T21:16:03.402467+0000","last_peered":"2024-02-19T21:16:03.402467+0000","last_clean":"2024-02-19T21:16:03.402467+0000","last_became_active":"2024-02-19T21:08:04.264972+0000","last_became_peered":"2024-02-19T21:08:04.264972+0000","last_unstale":"2024-02-19T21:16:03.402467+0000","last_undegraded":"2024-02-19T21:16:03.402467+0000","last_fullsized":"2024-02-19T21:16:03.402467+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:56.048471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:14:56.048471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":699,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402882+0000","last_change":"2024-02-19T21:14:48.323376+0000","last_active":"2024-02-19T21:16:03.402882+0000","last_peered":"2024-02-19T21:16:03.402882+0000","last_clean":"2024-02-19T21:16:03.402882+0000","last_became_active":"2024-02-19T21:10:41.314114+0000","last_became_peered":"2024-02-19T21:10:41.314114+0000","last_unstale":"2024-02-19T21:16:03.402882+0000","last_undegraded":"2024-02-19T21:16:03.402882+0000","last_fullsized":"2024-02-19T21:16:03.402882+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:48.323269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:48.323269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":292,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873246+0000","last_change":"2024-02-19T21:12:08.430936+0000","last_active":"2024-02-19T21:16:03.873246+0000","last_peered":"2024-02-19T21:16:03.873246+0000","last_clean":"2024-02-19T21:16:03.873246+0000","last_became_active":"2024-02-19T21:10:41.300574+0000","last_became_peered":"2024-02-19T21:10:41.300574+0000","last_unstale":"2024-02-19T21:16:03.873246+0000","last_undegraded":"2024-02-19T21:16:03.873246+0000","last_fullsized":"2024-02-19T21:16:03.873246+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:12:08.430874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:12:08.430874+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"459'15","reported_seq":313,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403164+0000","last_change":"2024-02-19T21:13:43.552100+0000","last_active":"2024-02-19T21:16:03.403164+0000","last_peered":"2024-02-19T21:16:03.403164+0000","last_clean":"2024-02-19T21:16:03.403164+0000","last_became_active":"2024-02-19T21:10:40.297867+0000","last_became_peered":"2024-02-19T21:10:40.297867+0000","last_unstale":"2024-02-19T21:16:03.403164+0000","last_undegraded":"2024-02-19T21:16:03.403164+0000","last_fullsized":"2024-02-19T21:16:03.403164+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:13:43.552026+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:13:43.552026+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":107,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.456218+0000","last_change":"2024-02-19T21:15:07.220115+0000","last_active":"2024-02-19T21:16:03.456218+0000","last_peered":"2024-02-19T21:16:03.456218+0000","last_clean":"2024-02-19T21:16:03.456218+0000","last_became_active":"2024-02-19T21:14:16.385407+0000","last_became_peered":"2024-02-19T21:14:16.385407+0000","last_unstale":"2024-02-19T21:16:03.456218+0000","last_undegraded":"2024-02-19T21:16:03.456218+0000","last_fullsized":"2024-02-19T21:16:03.456218+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:07.220022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:07.220022+0000","last_clean_scrub_stamp":"2024-02-19T21:15:07.220022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":700,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406823+0000","last_change":"2024-02-19T21:14:59.160565+0000","last_active":"2024-02-19T21:16:02.406823+0000","last_peered":"2024-02-19T21:16:02.406823+0000","last_clean":"2024-02-19T21:16:02.406823+0000","last_became_active":"2024-02-19T21:08:04.265573+0000","last_became_peered":"2024-02-19T21:08:04.265573+0000","last_unstale":"2024-02-19T21:16:02.406823+0000","last_undegraded":"2024-02-19T21:16:02.406823+0000","last_fullsized":"2024-02-19T21:16:02.406823+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:59.160470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:59.160470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":713,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405218+0000","last_change":"2024-02-19T21:14:53.918607+0000","last_active":"2024-02-19T21:16:02.405218+0000","last_peered":"2024-02-19T21:16:02.405218+0000","last_clean":"2024-02-19T21:16:02.405218+0000","last_became_active":"2024-02-19T21:14:16.379202+0000","last_became_peered":"2024-02-19T21:14:16.379202+0000","last_unstale":"2024-02-19T21:16:02.405218+0000","last_undegraded":"2024-02-19T21:16:02.405218+0000","last_fullsized":"2024-02-19T21:16:02.405218+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:53.918521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:53.918521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":668,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405865+0000","last_change":"2024-02-19T21:15:02.096155+0000","last_active":"2024-02-19T21:16:02.405865+0000","last_peered":"2024-02-19T21:16:02.405865+0000","last_clean":"2024-02-19T21:16:02.405865+0000","last_became_active":"2024-02-19T21:14:16.397730+0000","last_became_peered":"2024-02-19T21:14:16.397730+0000","last_unstale":"2024-02-19T21:16:02.405865+0000","last_undegraded":"2024-02-19T21:16:02.405865+0000","last_fullsized":"2024-02-19T21:16:02.405865+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:02.096039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:02.096039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"459'20","reported_seq":458,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.404084+0000","last_change":"2024-02-19T21:14:47.087994+0000","last_active":"2024-02-19T21:16:03.404084+0000","last_peered":"2024-02-19T21:16:03.404084+0000","last_clean":"2024-02-19T21:16:03.404084+0000","last_became_active":"2024-02-19T21:08:04.267216+0000","last_became_peered":"2024-02-19T21:08:04.267216+0000","last_unstale":"2024-02-19T21:16:03.404084+0000","last_undegraded":"2024-02-19T21:16:03.404084+0000","last_fullsized":"2024-02-19T21:16:03.404084+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:14:47.087905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:47.087905+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":707,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.409012+0000","last_change":"2024-02-19T21:15:42.974119+0000","last_active":"2024-02-19T21:16:02.409012+0000","last_peered":"2024-02-19T21:16:02.409012+0000","last_clean":"2024-02-19T21:16:02.409012+0000","last_became_active":"2024-02-19T21:08:04.266728+0000","last_became_peered":"2024-02-19T21:08:04.266728+0000","last_unstale":"2024-02-19T21:16:02.409012+0000","last_undegraded":"2024-02-19T21:16:02.409012+0000","last_fullsized":"2024-02-19T21:16:02.409012+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:42.974035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_clean_scrub_stamp":"2024-02-19T21:15:42.974035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"640'10","reported_seq":892,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401904+0000","last_change":"2024-02-19T21:15:37.998427+0000","last_active":"2024-02-19T21:16:03.401904+0000","last_peered":"2024-02-19T21:16:03.401904+0000","last_clean":"2024-02-19T21:16:03.401904+0000","last_became_active":"2024-02-19T21:14:41.403775+0000","last_became_peered":"2024-02-19T21:14:41.403775+0000","last_unstale":"2024-02-19T21:16:03.401904+0000","last_undegraded":"2024-02-19T21:16:03.401904+0000","last_fullsized":"2024-02-19T21:16:03.401904+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"640'10","last_scrub_stamp":"2024-02-19T21:15:37.998310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:37.998310+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":569,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404213+0000","last_change":"2024-02-19T21:14:41.631201+0000","last_active":"2024-02-19T21:16:02.404213+0000","last_peered":"2024-02-19T21:16:02.404213+0000","last_clean":"2024-02-19T21:16:02.404213+0000","last_became_active":"2024-02-19T21:14:41.415447+0000","last_became_peered":"2024-02-19T21:14:41.415447+0000","last_unstale":"2024-02-19T21:16:02.404213+0000","last_undegraded":"2024-02-19T21:16:02.404213+0000","last_fullsized":"2024-02-19T21:16:02.404213+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:14:18.606934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:18.606934+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":111,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.456248+0000","last_change":"2024-02-19T21:14:41.395579+0000","last_active":"2024-02-19T21:16:03.456248+0000","last_peered":"2024-02-19T21:16:03.456248+0000","last_clean":"2024-02-19T21:16:03.456248+0000","last_became_active":"2024-02-19T21:14:41.395223+0000","last_became_peered":"2024-02-19T21:14:41.395223+0000","last_unstale":"2024-02-19T21:16:03.456248+0000","last_undegraded":"2024-02-19T21:16:03.456248+0000","last_fullsized":"2024-02-19T21:16:03.456248+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:13.532568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:55.202463+0000","last_clean_scrub_stamp":"2024-02-19T21:14:13.532568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"459'15","reported_seq":745,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404610+0000","last_change":"2024-02-19T21:14:15.936988+0000","last_active":"2024-02-19T21:16:02.404610+0000","last_peered":"2024-02-19T21:16:02.404610+0000","last_clean":"2024-02-19T21:16:02.404610+0000","last_became_active":"2024-02-19T21:08:04.266103+0000","last_became_peered":"2024-02-19T21:08:04.266103+0000","last_unstale":"2024-02-19T21:16:02.404610+0000","last_undegraded":"2024-02-19T21:16:02.404610+0000","last_fullsized":"2024-02-19T21:16:02.404610+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:14:15.936909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:15.936909+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":315,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873298+0000","last_change":"2024-02-19T21:14:41.415900+0000","last_active":"2024-02-19T21:16:03.873298+0000","last_peered":"2024-02-19T21:16:03.873298+0000","last_clean":"2024-02-19T21:16:03.873298+0000","last_became_active":"2024-02-19T21:14:41.415500+0000","last_became_peered":"2024-02-19T21:14:41.415500+0000","last_unstale":"2024-02-19T21:16:03.873298+0000","last_undegraded":"2024-02-19T21:16:03.873298+0000","last_fullsized":"2024-02-19T21:16:03.873298+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:07.506946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:07.506946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"304'10","reported_seq":537,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-19T21:16:04.403852+0000","last_change":"2024-02-19T21:15:43.054159+0000","last_active":"2024-02-19T21:16:04.403852+0000","last_peered":"2024-02-19T21:16:04.403852+0000","last_clean":"2024-02-19T21:16:04.403852+0000","last_became_active":"2024-02-19T21:08:04.256045+0000","last_became_peered":"2024-02-19T21:08:04.256045+0000","last_unstale":"2024-02-19T21:16:04.403852+0000","last_undegraded":"2024-02-19T21:16:04.403852+0000","last_fullsized":"2024-02-19T21:16:04.403852+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'10","last_scrub_stamp":"2024-02-19T21:15:43.054055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:43.054055+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":297,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873808+0000","last_change":"2024-02-19T21:15:41.829174+0000","last_active":"2024-02-19T21:16:03.873808+0000","last_peered":"2024-02-19T21:16:03.873808+0000","last_clean":"2024-02-19T21:16:03.873808+0000","last_became_active":"2024-02-19T21:10:41.310279+0000","last_became_peered":"2024-02-19T21:10:41.310279+0000","last_unstale":"2024-02-19T21:16:03.873808+0000","last_undegraded":"2024-02-19T21:16:03.873808+0000","last_fullsized":"2024-02-19T21:16:03.873808+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:41.829068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_clean_scrub_stamp":"2024-02-19T21:15:41.829068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":700,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402424+0000","last_change":"2024-02-19T21:15:03.372324+0000","last_active":"2024-02-19T21:16:03.402424+0000","last_peered":"2024-02-19T21:16:03.402424+0000","last_clean":"2024-02-19T21:16:03.402424+0000","last_became_active":"2024-02-19T21:14:16.371724+0000","last_became_peered":"2024-02-19T21:14:16.371724+0000","last_unstale":"2024-02-19T21:16:03.402424+0000","last_undegraded":"2024-02-19T21:16:03.402424+0000","last_fullsized":"2024-02-19T21:16:03.402424+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-02-19T21:15:03.372256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:03.372256+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":20,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"459'15","reported_seq":457,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.402997+0000","last_change":"2024-02-19T21:14:16.723013+0000","last_active":"2024-02-19T21:16:02.402997+0000","last_peered":"2024-02-19T21:16:02.402997+0000","last_clean":"2024-02-19T21:16:02.402997+0000","last_became_active":"2024-02-19T21:14:16.377099+0000","last_became_peered":"2024-02-19T21:14:16.377099+0000","last_unstale":"2024-02-19T21:16:02.402997+0000","last_undegraded":"2024-02-19T21:16:02.402997+0000","last_fullsized":"2024-02-19T21:16:02.402997+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:13:06.958518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:13:06.958518+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":107,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.456490+0000","last_change":"2024-02-19T21:14:53.410134+0000","last_active":"2024-02-19T21:16:03.456490+0000","last_peered":"2024-02-19T21:16:03.456490+0000","last_clean":"2024-02-19T21:16:03.456490+0000","last_became_active":"2024-02-19T21:14:16.388194+0000","last_became_peered":"2024-02-19T21:14:16.388194+0000","last_unstale":"2024-02-19T21:16:03.456490+0000","last_undegraded":"2024-02-19T21:16:03.456490+0000","last_fullsized":"2024-02-19T21:16:03.456490+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:53.410025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:14:53.410025+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":454,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403409+0000","last_change":"2024-02-19T21:15:41.081514+0000","last_active":"2024-02-19T21:16:03.403409+0000","last_peered":"2024-02-19T21:16:03.403409+0000","last_clean":"2024-02-19T21:16:03.403409+0000","last_became_active":"2024-02-19T21:14:41.398865+0000","last_became_peered":"2024-02-19T21:14:41.398865+0000","last_unstale":"2024-02-19T21:16:03.403409+0000","last_undegraded":"2024-02-19T21:16:03.403409+0000","last_fullsized":"2024-02-19T21:16:03.403409+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:41.081437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_clean_scrub_stamp":"2024-02-19T21:15:41.081437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":551,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405480+0000","last_change":"2024-02-19T21:15:49.956671+0000","last_active":"2024-02-19T21:16:02.405480+0000","last_peered":"2024-02-19T21:16:02.405480+0000","last_clean":"2024-02-19T21:16:02.405480+0000","last_became_active":"2024-02-19T21:14:41.388322+0000","last_became_peered":"2024-02-19T21:14:41.388322+0000","last_unstale":"2024-02-19T21:16:02.405480+0000","last_undegraded":"2024-02-19T21:16:02.405480+0000","last_fullsized":"2024-02-19T21:16:02.405480+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:49.956589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:49.956589+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":116,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.457242+0000","last_change":"2024-02-19T21:14:41.400692+0000","last_active":"2024-02-19T21:16:03.457242+0000","last_peered":"2024-02-19T21:16:03.457242+0000","last_clean":"2024-02-19T21:16:03.457242+0000","last_became_active":"2024-02-19T21:14:41.400354+0000","last_became_peered":"2024-02-19T21:14:41.400354+0000","last_unstale":"2024-02-19T21:16:03.457242+0000","last_undegraded":"2024-02-19T21:16:03.457242+0000","last_fullsized":"2024-02-19T21:16:03.457242+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:19.148620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:19.148620+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"459'19","reported_seq":363,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873364+0000","last_change":"2024-02-19T21:14:56.610314+0000","last_active":"2024-02-19T21:16:03.873364+0000","last_peered":"2024-02-19T21:16:03.873364+0000","last_clean":"2024-02-19T21:16:03.873364+0000","last_became_active":"2024-02-19T21:14:16.380020+0000","last_became_peered":"2024-02-19T21:14:16.380020+0000","last_unstale":"2024-02-19T21:16:03.873364+0000","last_undegraded":"2024-02-19T21:16:03.873364+0000","last_fullsized":"2024-02-19T21:16:03.873364+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:14:56.610233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:56.610233+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":721,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404134+0000","last_change":"2024-02-19T21:15:53.936208+0000","last_active":"2024-02-19T21:16:02.404134+0000","last_peered":"2024-02-19T21:16:02.404134+0000","last_clean":"2024-02-19T21:16:02.404134+0000","last_became_active":"2024-02-19T21:14:41.395354+0000","last_became_peered":"2024-02-19T21:14:41.395354+0000","last_unstale":"2024-02-19T21:16:02.404134+0000","last_undegraded":"2024-02-19T21:16:02.404134+0000","last_fullsized":"2024-02-19T21:16:02.404134+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.936143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.936143+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":699,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.403552+0000","last_change":"2024-02-19T21:15:46.944899+0000","last_active":"2024-02-19T21:16:02.403552+0000","last_peered":"2024-02-19T21:16:02.403552+0000","last_clean":"2024-02-19T21:16:02.403552+0000","last_became_active":"2024-02-19T21:08:04.268847+0000","last_became_peered":"2024-02-19T21:08:04.268847+0000","last_unstale":"2024-02-19T21:16:02.403552+0000","last_undegraded":"2024-02-19T21:16:02.403552+0000","last_fullsized":"2024-02-19T21:16:02.403552+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:46.944774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:46.944774+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":449,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403843+0000","last_change":"2024-02-19T21:15:36.137073+0000","last_active":"2024-02-19T21:16:03.403843+0000","last_peered":"2024-02-19T21:16:03.403843+0000","last_clean":"2024-02-19T21:16:03.403843+0000","last_became_active":"2024-02-19T21:14:41.401891+0000","last_became_peered":"2024-02-19T21:14:41.401891+0000","last_unstale":"2024-02-19T21:16:03.403843+0000","last_undegraded":"2024-02-19T21:16:03.403843+0000","last_fullsized":"2024-02-19T21:16:03.403843+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:36.137008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:36.137008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":449,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402542+0000","last_change":"2024-02-19T21:14:59.059646+0000","last_active":"2024-02-19T21:16:03.402542+0000","last_peered":"2024-02-19T21:16:03.402542+0000","last_clean":"2024-02-19T21:16:03.402542+0000","last_became_active":"2024-02-19T21:14:41.397884+0000","last_became_peered":"2024-02-19T21:14:41.397884+0000","last_unstale":"2024-02-19T21:16:03.402542+0000","last_undegraded":"2024-02-19T21:16:03.402542+0000","last_fullsized":"2024-02-19T21:16:03.402542+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:59.059572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:59.059572+0000","last_clean_scrub_stamp":"2024-02-19T21:14:59.059572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"555'10","reported_seq":341,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405237+0000","last_change":"2024-02-19T21:15:40.997672+0000","last_active":"2024-02-19T21:16:02.405237+0000","last_peered":"2024-02-19T21:16:02.405237+0000","last_clean":"2024-02-19T21:16:02.405237+0000","last_became_active":"2024-02-19T21:14:16.395203+0000","last_became_peered":"2024-02-19T21:14:16.395203+0000","last_unstale":"2024-02-19T21:16:02.405237+0000","last_undegraded":"2024-02-19T21:16:02.405237+0000","last_fullsized":"2024-02-19T21:16:02.405237+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'10","last_scrub_stamp":"2024-02-19T21:15:40.997592+0000","last_deep_scrub":"555'10","last_deep_scrub_stamp":"2024-02-19T21:13:12.409036+0000","last_clean_scrub_stamp":"2024-02-19T21:15:40.997592+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":707,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406965+0000","last_change":"2024-02-19T21:15:44.964518+0000","last_active":"2024-02-19T21:16:02.406965+0000","last_peered":"2024-02-19T21:16:02.406965+0000","last_clean":"2024-02-19T21:16:02.406965+0000","last_became_active":"2024-02-19T21:14:16.395740+0000","last_became_peered":"2024-02-19T21:14:16.395740+0000","last_unstale":"2024-02-19T21:16:02.406965+0000","last_undegraded":"2024-02-19T21:16:02.406965+0000","last_fullsized":"2024-02-19T21:16:02.406965+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:44.964433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:44.964433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":107,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.456247+0000","last_change":"2024-02-19T21:15:54.200511+0000","last_active":"2024-02-19T21:16:03.456247+0000","last_peered":"2024-02-19T21:16:03.456247+0000","last_clean":"2024-02-19T21:16:03.456247+0000","last_became_active":"2024-02-19T21:14:16.390399+0000","last_became_peered":"2024-02-19T21:14:16.390399+0000","last_unstale":"2024-02-19T21:16:03.456247+0000","last_undegraded":"2024-02-19T21:16:03.456247+0000","last_fullsized":"2024-02-19T21:16:03.456247+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:54.200442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:54.200442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":693,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406098+0000","last_change":"2024-02-19T21:14:50.846672+0000","last_active":"2024-02-19T21:16:02.406098+0000","last_peered":"2024-02-19T21:16:02.406098+0000","last_clean":"2024-02-19T21:16:02.406098+0000","last_became_active":"2024-02-19T21:10:41.314589+0000","last_became_peered":"2024-02-19T21:10:41.314589+0000","last_unstale":"2024-02-19T21:16:02.406098+0000","last_undegraded":"2024-02-19T21:16:02.406098+0000","last_fullsized":"2024-02-19T21:16:02.406098+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:50.846595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:50.846595+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":436,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.404131+0000","last_change":"2024-02-19T21:15:50.223281+0000","last_active":"2024-02-19T21:16:03.404131+0000","last_peered":"2024-02-19T21:16:03.404131+0000","last_clean":"2024-02-19T21:16:03.404131+0000","last_became_active":"2024-02-19T21:08:04.266161+0000","last_became_peered":"2024-02-19T21:08:04.266161+0000","last_unstale":"2024-02-19T21:16:03.404131+0000","last_undegraded":"2024-02-19T21:16:03.404131+0000","last_fullsized":"2024-02-19T21:16:03.404131+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:50.223212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:50.223212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"459'19","reported_seq":502,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403924+0000","last_change":"2024-02-19T21:15:42.047875+0000","last_active":"2024-02-19T21:16:03.403924+0000","last_peered":"2024-02-19T21:16:03.403924+0000","last_clean":"2024-02-19T21:16:03.403924+0000","last_became_active":"2024-02-19T21:14:16.386212+0000","last_became_peered":"2024-02-19T21:14:16.386212+0000","last_unstale":"2024-02-19T21:16:03.403924+0000","last_undegraded":"2024-02-19T21:16:03.403924+0000","last_fullsized":"2024-02-19T21:16:03.403924+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:15:42.047800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:42.047800+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":109,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.457133+0000","last_change":"2024-02-19T21:15:02.244608+0000","last_active":"2024-02-19T21:16:03.457133+0000","last_peered":"2024-02-19T21:16:03.457133+0000","last_clean":"2024-02-19T21:16:03.457133+0000","last_became_active":"2024-02-19T21:14:16.388239+0000","last_became_peered":"2024-02-19T21:14:16.388239+0000","last_unstale":"2024-02-19T21:16:03.457133+0000","last_undegraded":"2024-02-19T21:16:03.457133+0000","last_fullsized":"2024-02-19T21:16:03.457133+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:02.244550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:02.244550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":708,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406891+0000","last_change":"2024-02-19T21:14:19.588243+0000","last_active":"2024-02-19T21:16:02.406891+0000","last_peered":"2024-02-19T21:16:02.406891+0000","last_clean":"2024-02-19T21:16:02.406891+0000","last_became_active":"2024-02-19T21:14:16.375236+0000","last_became_peered":"2024-02-19T21:14:16.375236+0000","last_unstale":"2024-02-19T21:16:02.406891+0000","last_undegraded":"2024-02-19T21:16:02.406891+0000","last_fullsized":"2024-02-19T21:16:02.406891+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:19.588167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:19.588167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":467,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403458+0000","last_change":"2024-02-19T21:14:41.399175+0000","last_active":"2024-02-19T21:16:03.403458+0000","last_peered":"2024-02-19T21:16:03.403458+0000","last_clean":"2024-02-19T21:16:03.403458+0000","last_became_active":"2024-02-19T21:14:41.398484+0000","last_became_peered":"2024-02-19T21:14:41.398484+0000","last_unstale":"2024-02-19T21:16:03.403458+0000","last_undegraded":"2024-02-19T21:16:03.403458+0000","last_fullsized":"2024-02-19T21:16:03.403458+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:12.107010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:12.107010+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":116,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.456795+0000","last_change":"2024-02-19T21:15:36.240010+0000","last_active":"2024-02-19T21:16:03.456795+0000","last_peered":"2024-02-19T21:16:03.456795+0000","last_clean":"2024-02-19T21:16:03.456795+0000","last_became_active":"2024-02-19T21:14:41.400328+0000","last_became_peered":"2024-02-19T21:14:41.400328+0000","last_unstale":"2024-02-19T21:16:03.456795+0000","last_undegraded":"2024-02-19T21:16:03.456795+0000","last_fullsized":"2024-02-19T21:16:03.456795+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:36.239929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:15:36.239929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"459'20","reported_seq":634,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404366+0000","last_change":"2024-02-19T21:14:55.895841+0000","last_active":"2024-02-19T21:16:02.404366+0000","last_peered":"2024-02-19T21:16:02.404366+0000","last_clean":"2024-02-19T21:16:02.404366+0000","last_became_active":"2024-02-19T21:14:41.415491+0000","last_became_peered":"2024-02-19T21:14:41.415491+0000","last_unstale":"2024-02-19T21:16:02.404366+0000","last_undegraded":"2024-02-19T21:16:02.404366+0000","last_fullsized":"2024-02-19T21:16:02.404366+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:14:55.895741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:55.895741+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":704,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405403+0000","last_change":"2024-02-19T21:15:35.006355+0000","last_active":"2024-02-19T21:16:02.405403+0000","last_peered":"2024-02-19T21:16:02.405403+0000","last_clean":"2024-02-19T21:16:02.405403+0000","last_became_active":"2024-02-19T21:10:41.294592+0000","last_became_peered":"2024-02-19T21:10:41.294592+0000","last_unstale":"2024-02-19T21:16:02.405403+0000","last_undegraded":"2024-02-19T21:16:02.405403+0000","last_fullsized":"2024-02-19T21:16:02.405403+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:35.006289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:35.006289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"623'12","reported_seq":133,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-02-19T21:16:04.403270+0000","last_change":"2024-02-19T21:15:52.229142+0000","last_active":"2024-02-19T21:16:04.403270+0000","last_peered":"2024-02-19T21:16:04.403270+0000","last_clean":"2024-02-19T21:16:04.403270+0000","last_became_active":"2024-02-19T21:14:16.391675+0000","last_became_peered":"2024-02-19T21:14:16.391675+0000","last_unstale":"2024-02-19T21:16:04.403270+0000","last_undegraded":"2024-02-19T21:16:04.403270+0000","last_fullsized":"2024-02-19T21:16:04.403270+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"623'12","last_scrub_stamp":"2024-02-19T21:15:52.229069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:52.229069+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":677,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404591+0000","last_change":"2024-02-19T21:14:24.577379+0000","last_active":"2024-02-19T21:16:02.404591+0000","last_peered":"2024-02-19T21:16:02.404591+0000","last_clean":"2024-02-19T21:16:02.404591+0000","last_became_active":"2024-02-19T21:01:22.297979+0000","last_became_peered":"2024-02-19T21:01:22.297979+0000","last_unstale":"2024-02-19T21:16:02.404591+0000","last_undegraded":"2024-02-19T21:16:02.404591+0000","last_fullsized":"2024-02-19T21:16:02.404591+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:14:24.577303+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:14:24.577303+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":633,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401887+0000","last_change":"2024-02-19T21:14:24.511577+0000","last_active":"2024-02-19T21:16:03.401887+0000","last_peered":"2024-02-19T21:16:03.401887+0000","last_clean":"2024-02-19T21:16:03.401887+0000","last_became_active":"2024-02-19T21:04:39.700113+0000","last_became_peered":"2024-02-19T21:04:39.700113+0000","last_unstale":"2024-02-19T21:16:03.401887+0000","last_undegraded":"2024-02-19T21:16:03.401887+0000","last_fullsized":"2024-02-19T21:16:03.401887+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:24.511507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:24.511507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"459'19","reported_seq":119,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.457069+0000","last_change":"2024-02-19T21:14:22.079972+0000","last_active":"2024-02-19T21:16:03.457069+0000","last_peered":"2024-02-19T21:16:03.457069+0000","last_clean":"2024-02-19T21:16:03.457069+0000","last_became_active":"2024-02-19T21:14:16.389206+0000","last_became_peered":"2024-02-19T21:14:16.389206+0000","last_unstale":"2024-02-19T21:16:03.457069+0000","last_undegraded":"2024-02-19T21:16:03.457069+0000","last_fullsized":"2024-02-19T21:16:03.457069+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:14:22.079893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:22.079893+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":467,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403970+0000","last_change":"2024-02-19T21:15:37.108298+0000","last_active":"2024-02-19T21:16:03.403970+0000","last_peered":"2024-02-19T21:16:03.403970+0000","last_clean":"2024-02-19T21:16:03.403970+0000","last_became_active":"2024-02-19T21:14:41.402278+0000","last_became_peered":"2024-02-19T21:14:41.402278+0000","last_unstale":"2024-02-19T21:16:03.403970+0000","last_undegraded":"2024-02-19T21:16:03.403970+0000","last_fullsized":"2024-02-19T21:16:03.403970+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:37.108220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:37.108220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":526,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404498+0000","last_change":"2024-02-19T21:14:41.416235+0000","last_active":"2024-02-19T21:16:02.404498+0000","last_peered":"2024-02-19T21:16:02.404498+0000","last_clean":"2024-02-19T21:16:02.404498+0000","last_became_active":"2024-02-19T21:14:41.415911+0000","last_became_peered":"2024-02-19T21:14:41.415911+0000","last_unstale":"2024-02-19T21:16:02.404498+0000","last_undegraded":"2024-02-19T21:16:02.404498+0000","last_fullsized":"2024-02-19T21:16:02.404498+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:11.580253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:11.580253+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":692,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.403881+0000","last_change":"2024-02-19T21:15:33.888785+0000","last_active":"2024-02-19T21:16:02.403881+0000","last_peered":"2024-02-19T21:16:02.403881+0000","last_clean":"2024-02-19T21:16:02.403881+0000","last_became_active":"2024-02-19T21:10:41.312577+0000","last_became_peered":"2024-02-19T21:10:41.312577+0000","last_unstale":"2024-02-19T21:16:02.403881+0000","last_undegraded":"2024-02-19T21:16:02.403881+0000","last_fullsized":"2024-02-19T21:16:02.403881+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:33.888719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:33.888719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":646,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.403838+0000","last_change":"2024-02-19T21:13:53.087054+0000","last_active":"2024-02-19T21:16:02.403838+0000","last_peered":"2024-02-19T21:16:02.403838+0000","last_clean":"2024-02-19T21:16:02.403838+0000","last_became_active":"2024-02-19T21:10:41.311050+0000","last_became_peered":"2024-02-19T21:10:41.311050+0000","last_unstale":"2024-02-19T21:16:02.403838+0000","last_undegraded":"2024-02-19T21:16:02.403838+0000","last_fullsized":"2024-02-19T21:16:02.403838+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:53.086972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:13:53.086972+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"459'34","reported_seq":780,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403030+0000","last_change":"2024-02-19T21:13:39.513661+0000","last_active":"2024-02-19T21:16:03.403030+0000","last_peered":"2024-02-19T21:16:03.403030+0000","last_clean":"2024-02-19T21:16:03.403030+0000","last_became_active":"2024-02-19T21:08:04.248378+0000","last_became_peered":"2024-02-19T21:08:04.248378+0000","last_unstale":"2024-02-19T21:16:03.403030+0000","last_undegraded":"2024-02-19T21:16:03.403030+0000","last_fullsized":"2024-02-19T21:16:03.403030+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'34","last_scrub_stamp":"2024-02-19T21:13:39.513604+0000","last_deep_scrub":"419'22","last_deep_scrub_stamp":"2024-02-19T21:10:19.109244+0000","last_clean_scrub_stamp":"2024-02-19T21:13:39.513604+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":111,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.457776+0000","last_change":"2024-02-19T21:15:41.405941+0000","last_active":"2024-02-19T21:16:03.457776+0000","last_peered":"2024-02-19T21:16:03.457776+0000","last_clean":"2024-02-19T21:16:03.457776+0000","last_became_active":"2024-02-19T21:15:41.405527+0000","last_became_peered":"2024-02-19T21:15:41.405527+0000","last_unstale":"2024-02-19T21:16:03.457776+0000","last_undegraded":"2024-02-19T21:16:03.457776+0000","last_fullsized":"2024-02-19T21:16:03.457776+0000","mapping_epoch":691,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":692,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:10.531467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:10.531467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":712,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405208+0000","last_change":"2024-02-19T21:14:41.415798+0000","last_active":"2024-02-19T21:16:02.405208+0000","last_peered":"2024-02-19T21:16:02.405208+0000","last_clean":"2024-02-19T21:16:02.405208+0000","last_became_active":"2024-02-19T21:14:41.415463+0000","last_became_peered":"2024-02-19T21:14:41.415463+0000","last_unstale":"2024-02-19T21:16:02.405208+0000","last_undegraded":"2024-02-19T21:16:02.405208+0000","last_fullsized":"2024-02-19T21:16:02.405208+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:23.025940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:23.025940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":315,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873637+0000","last_change":"2024-02-19T21:14:16.382742+0000","last_active":"2024-02-19T21:16:03.873637+0000","last_peered":"2024-02-19T21:16:03.873637+0000","last_clean":"2024-02-19T21:16:03.873637+0000","last_became_active":"2024-02-19T21:14:16.381600+0000","last_became_peered":"2024-02-19T21:14:16.381600+0000","last_unstale":"2024-02-19T21:16:03.873637+0000","last_undegraded":"2024-02-19T21:16:03.873637+0000","last_fullsized":"2024-02-19T21:16:03.873637+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:43.735382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:13:43.735382+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":116,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.456170+0000","last_change":"2024-02-19T21:15:04.202157+0000","last_active":"2024-02-19T21:16:03.456170+0000","last_peered":"2024-02-19T21:16:03.456170+0000","last_clean":"2024-02-19T21:16:03.456170+0000","last_became_active":"2024-02-19T21:14:41.395278+0000","last_became_peered":"2024-02-19T21:14:41.395278+0000","last_unstale":"2024-02-19T21:16:03.456170+0000","last_undegraded":"2024-02-19T21:16:03.456170+0000","last_fullsized":"2024-02-19T21:16:03.456170+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:04.202042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:04.202042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"459'18","reported_seq":499,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402203+0000","last_change":"2024-02-19T21:15:54.109530+0000","last_active":"2024-02-19T21:16:03.402203+0000","last_peered":"2024-02-19T21:16:03.402203+0000","last_clean":"2024-02-19T21:16:03.402203+0000","last_became_active":"2024-02-19T21:10:41.307690+0000","last_became_peered":"2024-02-19T21:10:41.307690+0000","last_unstale":"2024-02-19T21:16:03.402203+0000","last_undegraded":"2024-02-19T21:16:03.402203+0000","last_fullsized":"2024-02-19T21:16:03.402203+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'18","last_scrub_stamp":"2024-02-19T21:15:54.109426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:54.109426+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":449,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403501+0000","last_change":"2024-02-19T21:14:16.386656+0000","last_active":"2024-02-19T21:16:03.403501+0000","last_peered":"2024-02-19T21:16:03.403501+0000","last_clean":"2024-02-19T21:16:03.403501+0000","last_became_active":"2024-02-19T21:14:16.386352+0000","last_became_peered":"2024-02-19T21:14:16.386352+0000","last_unstale":"2024-02-19T21:16:03.403501+0000","last_undegraded":"2024-02-19T21:16:03.403501+0000","last_fullsized":"2024-02-19T21:16:03.403501+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:14.063569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:14.063569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":297,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873267+0000","last_change":"2024-02-19T21:14:59.655023+0000","last_active":"2024-02-19T21:16:03.873267+0000","last_peered":"2024-02-19T21:16:03.873267+0000","last_clean":"2024-02-19T21:16:03.873267+0000","last_became_active":"2024-02-19T21:10:41.296849+0000","last_became_peered":"2024-02-19T21:10:41.296849+0000","last_unstale":"2024-02-19T21:16:03.873267+0000","last_undegraded":"2024-02-19T21:16:03.873267+0000","last_fullsized":"2024-02-19T21:16:03.873267+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:59.654949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:59.654949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":449,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402624+0000","last_change":"2024-02-19T21:14:58.020186+0000","last_active":"2024-02-19T21:16:03.402624+0000","last_peered":"2024-02-19T21:16:03.402624+0000","last_clean":"2024-02-19T21:16:03.402624+0000","last_became_active":"2024-02-19T21:14:16.385155+0000","last_became_peered":"2024-02-19T21:14:16.385155+0000","last_unstale":"2024-02-19T21:16:03.402624+0000","last_undegraded":"2024-02-19T21:16:03.402624+0000","last_fullsized":"2024-02-19T21:16:03.402624+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:58.020115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:14:58.020115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":646,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405485+0000","last_change":"2024-02-19T21:14:25.024614+0000","last_active":"2024-02-19T21:16:02.405485+0000","last_peered":"2024-02-19T21:16:02.405485+0000","last_clean":"2024-02-19T21:16:02.405485+0000","last_became_active":"2024-02-19T21:14:16.375068+0000","last_became_peered":"2024-02-19T21:14:16.375068+0000","last_unstale":"2024-02-19T21:16:02.405485+0000","last_undegraded":"2024-02-19T21:16:02.405485+0000","last_fullsized":"2024-02-19T21:16:02.405485+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:25.024547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:25.024547+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"459'19","reported_seq":349,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873163+0000","last_change":"2024-02-19T21:14:41.422811+0000","last_active":"2024-02-19T21:16:03.873163+0000","last_peered":"2024-02-19T21:16:03.873163+0000","last_clean":"2024-02-19T21:16:03.873163+0000","last_became_active":"2024-02-19T21:14:41.390502+0000","last_became_peered":"2024-02-19T21:14:41.390502+0000","last_unstale":"2024-02-19T21:16:03.873163+0000","last_undegraded":"2024-02-19T21:16:03.873163+0000","last_fullsized":"2024-02-19T21:16:03.873163+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:12:55.429801+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:12:55.429801+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":676,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404219+0000","last_change":"2024-02-19T21:14:25.616331+0000","last_active":"2024-02-19T21:16:02.404219+0000","last_peered":"2024-02-19T21:16:02.404219+0000","last_clean":"2024-02-19T21:16:02.404219+0000","last_became_active":"2024-02-19T21:01:16.274329+0000","last_became_peered":"2024-02-19T21:01:16.274329+0000","last_unstale":"2024-02-19T21:16:02.404219+0000","last_undegraded":"2024-02-19T21:16:02.404219+0000","last_fullsized":"2024-02-19T21:16:02.404219+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:25.616258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:25.616258+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":698,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406074+0000","last_change":"2024-02-19T21:15:39.981076+0000","last_active":"2024-02-19T21:16:02.406074+0000","last_peered":"2024-02-19T21:16:02.406074+0000","last_clean":"2024-02-19T21:16:02.406074+0000","last_became_active":"2024-02-19T21:14:41.403970+0000","last_became_peered":"2024-02-19T21:14:41.403970+0000","last_unstale":"2024-02-19T21:16:02.406074+0000","last_undegraded":"2024-02-19T21:16:02.406074+0000","last_fullsized":"2024-02-19T21:16:02.406074+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:39.980983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:39.980983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":524,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.403713+0000","last_change":"2024-02-19T21:14:41.391013+0000","last_active":"2024-02-19T21:16:02.403713+0000","last_peered":"2024-02-19T21:16:02.403713+0000","last_clean":"2024-02-19T21:16:02.403713+0000","last_became_active":"2024-02-19T21:14:41.390659+0000","last_became_peered":"2024-02-19T21:14:41.390659+0000","last_unstale":"2024-02-19T21:16:02.403713+0000","last_undegraded":"2024-02-19T21:16:02.403713+0000","last_fullsized":"2024-02-19T21:16:02.403713+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:43.109579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:13:43.109579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":671,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.402487+0000","last_change":"2024-02-19T21:15:46.226483+0000","last_active":"2024-02-19T21:16:03.402487+0000","last_peered":"2024-02-19T21:16:03.402487+0000","last_clean":"2024-02-19T21:16:03.402487+0000","last_became_active":"2024-02-19T21:14:41.404407+0000","last_became_peered":"2024-02-19T21:14:41.404407+0000","last_unstale":"2024-02-19T21:16:03.402487+0000","last_undegraded":"2024-02-19T21:16:03.402487+0000","last_fullsized":"2024-02-19T21:16:03.402487+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:46.226419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:46.226419+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"459'7","reported_seq":301,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873371+0000","last_change":"2024-02-19T21:12:02.521592+0000","last_active":"2024-02-19T21:16:03.873371+0000","last_peered":"2024-02-19T21:16:03.873371+0000","last_clean":"2024-02-19T21:16:03.873371+0000","last_became_active":"2024-02-19T21:10:41.297295+0000","last_became_peered":"2024-02-19T21:10:41.297295+0000","last_unstale":"2024-02-19T21:16:03.873371+0000","last_undegraded":"2024-02-19T21:16:03.873371+0000","last_fullsized":"2024-02-19T21:16:03.873371+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'7","last_scrub_stamp":"2024-02-19T21:12:02.521523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:12:02.521523+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":717,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405288+0000","last_change":"2024-02-19T21:14:49.850583+0000","last_active":"2024-02-19T21:16:02.405288+0000","last_peered":"2024-02-19T21:16:02.405288+0000","last_clean":"2024-02-19T21:16:02.405288+0000","last_became_active":"2024-02-19T21:14:41.390588+0000","last_became_peered":"2024-02-19T21:14:41.390588+0000","last_unstale":"2024-02-19T21:16:02.405288+0000","last_undegraded":"2024-02-19T21:16:02.405288+0000","last_fullsized":"2024-02-19T21:16:02.405288+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:49.850506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:49.850506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":695,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406331+0000","last_change":"2024-02-19T21:14:54.900146+0000","last_active":"2024-02-19T21:16:02.406331+0000","last_peered":"2024-02-19T21:16:02.406331+0000","last_clean":"2024-02-19T21:16:02.406331+0000","last_became_active":"2024-02-19T21:10:41.307400+0000","last_became_peered":"2024-02-19T21:10:41.307400+0000","last_unstale":"2024-02-19T21:16:02.406331+0000","last_undegraded":"2024-02-19T21:16:02.406331+0000","last_fullsized":"2024-02-19T21:16:02.406331+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:54.900062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:54.900062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":701,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.404644+0000","last_change":"2024-02-19T21:15:47.947147+0000","last_active":"2024-02-19T21:16:02.404644+0000","last_peered":"2024-02-19T21:16:02.404644+0000","last_clean":"2024-02-19T21:16:02.404644+0000","last_became_active":"2024-02-19T21:08:04.266100+0000","last_became_peered":"2024-02-19T21:08:04.266100+0000","last_unstale":"2024-02-19T21:16:02.404644+0000","last_undegraded":"2024-02-19T21:16:02.404644+0000","last_fullsized":"2024-02-19T21:16:02.404644+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:47.947061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:47.947061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":560,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.406638+0000","last_change":"2024-02-19T21:14:41.403716+0000","last_active":"2024-02-19T21:16:02.406638+0000","last_peered":"2024-02-19T21:16:02.406638+0000","last_clean":"2024-02-19T21:16:02.406638+0000","last_became_active":"2024-02-19T21:14:41.403361+0000","last_became_peered":"2024-02-19T21:14:41.403361+0000","last_unstale":"2024-02-19T21:16:02.406638+0000","last_undegraded":"2024-02-19T21:16:02.406638+0000","last_fullsized":"2024-02-19T21:16:02.406638+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:19.904121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:19.904121+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"623'19","reported_seq":360,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.872793+0000","last_change":"2024-02-19T21:14:21.589221+0000","last_active":"2024-02-19T21:16:03.872793+0000","last_peered":"2024-02-19T21:16:03.872793+0000","last_clean":"2024-02-19T21:16:03.872793+0000","last_became_active":"2024-02-19T21:14:16.383577+0000","last_became_peered":"2024-02-19T21:14:16.383577+0000","last_unstale":"2024-02-19T21:16:03.872793+0000","last_undegraded":"2024-02-19T21:16:03.872793+0000","last_fullsized":"2024-02-19T21:16:03.872793+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"623'19","last_scrub_stamp":"2024-02-19T21:14:21.589138+0000","last_deep_scrub":"623'19","last_deep_scrub_stamp":"2024-02-19T21:14:21.589138+0000","last_clean_scrub_stamp":"2024-02-19T21:14:21.589138+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":449,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.401828+0000","last_change":"2024-02-19T21:15:00.101653+0000","last_active":"2024-02-19T21:16:03.401828+0000","last_peered":"2024-02-19T21:16:03.401828+0000","last_clean":"2024-02-19T21:16:03.401828+0000","last_became_active":"2024-02-19T21:14:41.400532+0000","last_became_peered":"2024-02-19T21:14:41.400532+0000","last_unstale":"2024-02-19T21:16:03.401828+0000","last_undegraded":"2024-02-19T21:16:03.401828+0000","last_fullsized":"2024-02-19T21:16:03.401828+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:00.101587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:00.101587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":676,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405363+0000","last_change":"2024-02-19T21:13:42.086086+0000","last_active":"2024-02-19T21:16:02.405363+0000","last_peered":"2024-02-19T21:16:02.405363+0000","last_clean":"2024-02-19T21:16:02.405363+0000","last_became_active":"2024-02-19T21:01:20.293587+0000","last_became_peered":"2024-02-19T21:01:20.293587+0000","last_unstale":"2024-02-19T21:16:02.405363+0000","last_undegraded":"2024-02-19T21:16:02.405363+0000","last_fullsized":"2024-02-19T21:16:02.405363+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:42.086006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:13:42.086006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":297,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.873226+0000","last_change":"2024-02-19T21:15:53.080318+0000","last_active":"2024-02-19T21:16:03.873226+0000","last_peered":"2024-02-19T21:16:03.873226+0000","last_clean":"2024-02-19T21:16:03.873226+0000","last_became_active":"2024-02-19T21:10:41.309550+0000","last_became_peered":"2024-02-19T21:10:41.309550+0000","last_unstale":"2024-02-19T21:16:03.873226+0000","last_undegraded":"2024-02-19T21:16:03.873226+0000","last_fullsized":"2024-02-19T21:16:03.873226+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.080222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.080222+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":668,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.410072+0000","last_change":"2024-02-19T21:14:55.013570+0000","last_active":"2024-02-19T21:16:02.410072+0000","last_peered":"2024-02-19T21:16:02.410072+0000","last_clean":"2024-02-19T21:16:02.410072+0000","last_became_active":"2024-02-19T21:14:41.403430+0000","last_became_peered":"2024-02-19T21:14:41.403430+0000","last_unstale":"2024-02-19T21:16:02.410072+0000","last_undegraded":"2024-02-19T21:16:02.410072+0000","last_fullsized":"2024-02-19T21:16:02.410072+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:55.013484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:55.013484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"459'27","reported_seq":517,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403793+0000","last_change":"2024-02-19T21:14:41.530133+0000","last_active":"2024-02-19T21:16:03.403793+0000","last_peered":"2024-02-19T21:16:03.403793+0000","last_clean":"2024-02-19T21:16:03.403793+0000","last_became_active":"2024-02-19T21:14:41.402686+0000","last_became_peered":"2024-02-19T21:14:41.402686+0000","last_unstale":"2024-02-19T21:16:03.403793+0000","last_undegraded":"2024-02-19T21:16:03.403793+0000","last_fullsized":"2024-02-19T21:16:03.403793+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'27","last_scrub_stamp":"2024-02-19T21:14:19.968192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:19.968192+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":564,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-02-19T21:16:02.405517+0000","last_change":"2024-02-19T21:14:41.391255+0000","last_active":"2024-02-19T21:16:02.405517+0000","last_peered":"2024-02-19T21:16:02.405517+0000","last_clean":"2024-02-19T21:16:02.405517+0000","last_became_active":"2024-02-19T21:14:41.390887+0000","last_became_peered":"2024-02-19T21:14:41.390887+0000","last_unstale":"2024-02-19T21:16:02.405517+0000","last_undegraded":"2024-02-19T21:16:02.405517+0000","last_fullsized":"2024-02-19T21:16:02.405517+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:52.779962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:52.779962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":315,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.872845+0000","last_change":"2024-02-19T21:14:55.609881+0000","last_active":"2024-02-19T21:16:03.872845+0000","last_peered":"2024-02-19T21:16:03.872845+0000","last_clean":"2024-02-19T21:16:03.872845+0000","last_became_active":"2024-02-19T21:14:16.384069+0000","last_became_peered":"2024-02-19T21:14:16.384069+0000","last_unstale":"2024-02-19T21:16:03.872845+0000","last_undegraded":"2024-02-19T21:16:03.872845+0000","last_fullsized":"2024-02-19T21:16:03.872845+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:55.609805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:55.609805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":449,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.403350+0000","last_change":"2024-02-19T21:14:16.385139+0000","last_active":"2024-02-19T21:16:03.403350+0000","last_peered":"2024-02-19T21:16:03.403350+0000","last_clean":"2024-02-19T21:16:03.403350+0000","last_became_active":"2024-02-19T21:14:16.384527+0000","last_became_peered":"2024-02-19T21:14:16.384527+0000","last_unstale":"2024-02-19T21:16:03.403350+0000","last_undegraded":"2024-02-19T21:16:03.403350+0000","last_fullsized":"2024-02-19T21:16:03.403350+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:40.304490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:13:40.304490+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":107,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-02-19T21:16:03.456748+0000","last_change":"2024-02-19T21:15:46.084809+0000","last_active":"2024-02-19T21:16:03.456748+0000","last_peered":"2024-02-19T21:16:03.456748+0000","last_clean":"2024-02-19T21:16:03.456748+0000","last_became_active":"2024-02-19T21:14:16.383734+0000","last_became_peered":"2024-02-19T21:14:16.383734+0000","last_unstale":"2024-02-19T21:16:03.456748+0000","last_undegraded":"2024-02-19T21:16:03.456748+0000","last_fullsized":"2024-02-19T21:16:03.456748+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:46.084709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:46.084709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":247,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":81,"num_objects":16,"num_object_clones":1,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":16,"num_read_kb":4,"num_write":36,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":64,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":24,"acting":24,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1102,"num_read_kb":859,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":9648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":724,"ondisk_log_size":724,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":878,"num_read_kb":873,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":126,"acting":126,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":20,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266868,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303804,"kb_used_data":2484,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441348,"statfs":{"total":95995035648,"available":95683940352,"internally_reserved":0,"allocated":2543616,"data_stored":978639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.50900000000000001,"15min":0.86499999999999999},"min":{"1min":0.31,"5min":0.30399999999999999,"15min":0.215},"max":{"1min":0.68600000000000005,"5min":0.79200000000000004,"15min":89.456999999999994},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.50600000000000001,"15min":0.78900000000000003},"min":{"1min":0.32000000000000001,"5min":0.29599999999999999,"15min":0.25900000000000001},"max":{"1min":0.70999999999999996,"5min":0.81299999999999994,"15min":70.451999999999998},"last":0.51600000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.69899999999999995},"min":{"1min":0.376,"5min":0.29999999999999999,"15min":0.28199999999999997},"max":{"1min":0.66600000000000004,"5min":0.79100000000000004,"15min":34.273000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51500000000000001,"15min":0.93500000000000005},"min":{"1min":0.35899999999999999,"5min":0.29499999999999998,"15min":0.254},"max":{"1min":0.69199999999999995,"5min":0.97899999999999998,"15min":89.594999999999999},"last":0.36899999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.57199999999999995,"15min":0.96299999999999997},"min":{"1min":0.34300000000000003,"5min":0.29099999999999998,"15min":0.24299999999999999},"max":{"1min":2.8100000000000001,"5min":4.3769999999999998,"15min":104.413},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.54500000000000004,"15min":0.95699999999999996},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.27600000000000002},"max":{"1min":0.71199999999999997,"5min":1.8480000000000001,"15min":104.268},"last":0.41399999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.56200000000000006,"15min":0.96599999999999997},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.254},"max":{"1min":0.69799999999999995,"5min":2.9769999999999999,"15min":105.3},"last":0.435},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55200000000000005,"15min":0.61699999999999999},"min":{"1min":0.39500000000000002,"5min":0.32000000000000001,"15min":0.315},"max":{"1min":0.65300000000000002,"5min":2.3050000000000002,"15min":13.243},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.45800000000000002,"15min":0.45500000000000002},"min":{"1min":0.35899999999999999,"5min":0.28999999999999998,"15min":0.24299999999999999},"max":{"1min":0.64700000000000002,"5min":0.68799999999999994,"15min":1.373},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.495,"15min":0.47999999999999998},"min":{"1min":0.39200000000000002,"5min":0.27400000000000002,"15min":0.25900000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":1.0740000000000001},"last":0.38100000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.46000000000000002,"15min":0.45800000000000002},"min":{"1min":0.35499999999999998,"5min":0.29099999999999998,"15min":0.252},"max":{"1min":0.70599999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.51400000000000001,"15min":0.50700000000000001},"min":{"1min":0.40600000000000003,"5min":0.32600000000000001,"15min":0.27100000000000002},"max":{"1min":0.75600000000000001,"5min":0.79300000000000004,"15min":0.85899999999999999},"last":0.50700000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.53900000000000003,"15min":0.51900000000000002},"min":{"1min":0.42399999999999999,"5min":0.30199999999999999,"15min":0.28799999999999998},"max":{"1min":0.753,"5min":0.76700000000000002,"15min":1.0049999999999999},"last":0.495},{"interface":"front","average":{"1min":0.627,"5min":0.56799999999999995,"15min":0.55100000000000005},"min":{"1min":0.432,"5min":0.34999999999999998,"15min":0.33700000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":1.262},"last":0.52600000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528506,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312000,"kb_used_data":2488,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93433152,"statfs":{"total":95995035648,"available":95675547648,"internally_reserved":0,"allocated":2547712,"data_stored":1002557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47399999999999998,"15min":0.60999999999999999},"min":{"1min":0.249,"5min":0.24099999999999999,"15min":0.19900000000000001},"max":{"1min":0.73099999999999998,"5min":0.80300000000000005,"15min":17.486999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.49099999999999999,"15min":0.65500000000000003},"min":{"1min":0.30299999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.71199999999999997,"5min":1.669,"15min":17.43},"last":0.41199999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.499,"15min":0.62},"min":{"1min":0.248,"5min":0.248,"15min":0.22800000000000001},"max":{"1min":0.67400000000000004,"5min":1.6599999999999999,"15min":14.692},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.51000000000000001,"15min":0.66600000000000004},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.23400000000000001},"max":{"1min":0.69099999999999995,"5min":1.1990000000000001,"15min":14.795},"last":0.38200000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.52300000000000002,"15min":0.60699999999999998},"min":{"1min":0.34100000000000003,"5min":0.253,"15min":0.253},"max":{"1min":0.72199999999999998,"5min":3.3050000000000002,"15min":6.931},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.50800000000000001,"15min":0.67400000000000004},"min":{"1min":0.36099999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65900000000000003,"5min":1.794,"15min":14.391},"last":0.44}]},{"osd":3,"last update":"Mon Feb 19 21:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52000000000000002,"15min":0.70199999999999996},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.66800000000000004,"5min":1.4450000000000001,"15min":14.510999999999999},"last":0.377},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.57099999999999995,"15min":0.68100000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.316},"max":{"1min":4.085,"5min":4.085,"15min":17.585000000000001},"last":0.44700000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.45600000000000002,"15min":0.42499999999999999},"min":{"1min":0.26800000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.55100000000000005,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.496,"15min":0.45200000000000001},"min":{"1min":0.29299999999999998,"5min":0.253,"15min":0.22700000000000001},"max":{"1min":0.68999999999999995,"5min":3.431,"15min":3.431},"last":0.29599999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.50800000000000001,"15min":0.47399999999999998},"min":{"1min":0.318,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60299999999999998,"5min":2.9620000000000002,"15min":2.9620000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.52800000000000002,"15min":0.505},"min":{"1min":0.34599999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71399999999999997,"5min":2.7010000000000001,"15min":2.7010000000000001},"last":0.34999999999999998}]},{"osd":7,"last update":"Mon Feb 19 21:15:35 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.504,"15min":0.496},"min":{"1min":0.29999999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.66500000000000004,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.51500000000000001,"15min":0.501},"min":{"1min":0.312,"5min":0.312,"15min":0.28399999999999997},"max":{"1min":0.67700000000000005,"5min":1.835,"15min":1.835},"last":0.39000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575057,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299716,"kb_used_data":2492,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445436,"statfs":{"total":95995035648,"available":95688126464,"internally_reserved":0,"allocated":2551808,"data_stored":977862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.33100000000000002,"15min":0.54100000000000004},"min":{"1min":0.254,"5min":0.17999999999999999,"15min":0.16700000000000001},"max":{"1min":0.46100000000000002,"5min":2.0379999999999998,"15min":27.777000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.35299999999999998,"15min":0.57399999999999995},"min":{"1min":0.218,"5min":0.20200000000000001,"15min":0.19400000000000001},"max":{"1min":0.56299999999999994,"5min":1.9870000000000001,"15min":27.631},"last":0.40400000000000003}]},{"osd":2,"last update":"Mon Feb 19 21:15:41 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.36399999999999999,"15min":0.69299999999999995},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19600000000000001},"max":{"1min":0.50600000000000001,"5min":2.0230000000000001,"15min":27.798999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.38,"15min":0.70799999999999996},"min":{"1min":0.253,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.46200000000000002,"5min":2.089,"15min":27.606999999999999},"last":0.373}]},{"osd":3,"last update":"Mon Feb 19 21:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.36899999999999999,"15min":0.55600000000000005},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":0.52600000000000002,"5min":1.7450000000000001,"15min":15.574999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.40799999999999997,"15min":0.65200000000000002},"min":{"1min":0.312,"5min":0.28999999999999998,"15min":0.25900000000000001},"max":{"1min":0.54200000000000004,"5min":2.0070000000000001,"15min":27.82},"last":0.41599999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:15:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.52700000000000002,"15min":0.64800000000000002},"min":{"1min":0.38600000000000001,"5min":0.27800000000000002,"15min":0.26800000000000002},"max":{"1min":0.80300000000000005,"5min":1.149,"15min":9.6430000000000007},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55700000000000005,"15min":0.751},"min":{"1min":0.41199999999999998,"5min":0.34799999999999998,"15min":0.308},"max":{"1min":0.77400000000000002,"5min":1.9770000000000001,"15min":27.707000000000001},"last":0.59199999999999997}]},{"osd":5,"last update":"Mon Feb 19 21:15:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55200000000000005,"15min":0.73399999999999999},"min":{"1min":0.315,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75,"5min":1.048,"15min":27.529},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53100000000000003,"15min":0.66300000000000003},"min":{"1min":0.35299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.69499999999999995,"5min":0.98099999999999998,"15min":9.6799999999999997},"last":0.60799999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53100000000000003,"15min":0.65300000000000002},"min":{"1min":0.38200000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84699999999999998,"5min":1.1659999999999999,"15min":9.6989999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.59899999999999998,"15min":0.77900000000000003},"min":{"1min":0.38500000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81200000000000006,"5min":3.4359999999999999,"15min":27.733000000000001},"last":0.61799999999999999}]},{"osd":7,"last update":"Mon Feb 19 21:15:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.58999999999999997,"15min":0.77400000000000002},"min":{"1min":0.41699999999999998,"5min":0.371,"15min":0.36699999999999999},"max":{"1min":0.78200000000000003,"5min":3.4670000000000001,"15min":27.576000000000001},"last":0.628},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59299999999999997,"15min":0.80500000000000005},"min":{"1min":0.44800000000000001,"5min":0.42099999999999999,"15min":0.375},"max":{"1min":0.72599999999999998,"5min":1.1299999999999999,"15min":27.756},"last":0.56999999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738583,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307904,"kb_used_data":2488,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437248,"statfs":{"total":95995035648,"available":95679741952,"internally_reserved":0,"allocated":2547712,"data_stored":989393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:15:15 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.35199999999999998,"15min":0.624},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.188},"max":{"1min":0.46700000000000003,"5min":1.855,"15min":20.966999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.316,"5min":0.35899999999999999,"15min":0.63200000000000001},"min":{"1min":0.23200000000000001,"5min":0.20499999999999999,"15min":0.193},"max":{"1min":0.52500000000000002,"5min":1.613,"15min":20.366},"last":0.47699999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.35799999999999998,"15min":0.70999999999999996},"min":{"1min":0.23799999999999999,"5min":0.214,"15min":0.19900000000000001},"max":{"1min":0.46400000000000002,"5min":0.93600000000000005,"15min":27.916},"last":0.496},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.373,"15min":0.63300000000000001},"min":{"1min":0.25700000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.498,"5min":0.58599999999999997,"15min":22.004000000000001},"last":0.56499999999999995}]},{"osd":3,"last update":"Mon Feb 19 21:16:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39900000000000002,"15min":0.73399999999999999},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.71999999999999997,"5min":0.96799999999999997,"15min":25.669},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.41099999999999998,"15min":0.68200000000000005},"min":{"1min":0.29999999999999999,"5min":0.27100000000000002,"15min":0.26300000000000001},"max":{"1min":0.54900000000000004,"5min":0.77200000000000002,"15min":25.006},"last":0.50700000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:15:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.73099999999999998},"min":{"1min":0.39800000000000002,"5min":0.32500000000000001,"15min":0.28799999999999998},"max":{"1min":0.76400000000000001,"5min":1.0089999999999999,"15min":20.411999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56499999999999995,"15min":0.76000000000000001},"min":{"1min":0.39100000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.76100000000000001,"5min":2.093,"15min":19.986999999999998},"last":0.59499999999999997}]},{"osd":5,"last update":"Mon Feb 19 21:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.57199999999999995,"15min":0.80400000000000005},"min":{"1min":0.36199999999999999,"5min":0.31900000000000001,"15min":0.27200000000000002},"max":{"1min":0.93100000000000005,"5min":1.3560000000000001,"15min":20.021999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.59599999999999997,"15min":0.81100000000000005},"min":{"1min":0.438,"5min":0.35499999999999998,"15min":0.30399999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":20.908999999999999},"last":0.68000000000000005}]},{"osd":6,"last update":"Mon Feb 19 21:16:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59299999999999997,"15min":0.75700000000000001},"min":{"1min":0.42299999999999999,"5min":0.34100000000000003,"15min":0.33900000000000002},"max":{"1min":0.83599999999999997,"5min":2.1459999999999999,"15min":20.445},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.61499999999999999,"15min":0.79600000000000004},"min":{"1min":0.50600000000000001,"5min":0.35099999999999998,"15min":0.33400000000000002},"max":{"1min":0.82299999999999995,"5min":2.6120000000000001,"15min":20.047999999999998},"last":0.64600000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:15:34 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.58999999999999997,"15min":0.80600000000000005},"min":{"1min":0.47999999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.79300000000000004,"5min":0.84499999999999997,"15min":20.940999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.59699999999999998,"15min":0.81100000000000005},"min":{"1min":0.44700000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73699999999999999,"5min":1.0660000000000001,"15min":20.471},"last":0.67300000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411532,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312000,"kb_used_data":2488,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93433152,"statfs":{"total":95995035648,"available":95675547648,"internally_reserved":0,"allocated":2547712,"data_stored":989969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.0299999999999998,"5min":0.67600000000000005,"15min":0.64700000000000002},"min":{"1min":0.222,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":28.702999999999999,"5min":28.702999999999999,"15min":28.702999999999999},"last":0.29599999999999999},{"interface":"front","average":{"1min":2.0270000000000001,"5min":0.68100000000000005,"15min":0.70399999999999996},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":28.744,"5min":28.744,"15min":28.744},"last":0.24299999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.38600000000000001,"15min":0.71899999999999997},"min":{"1min":0.22600000000000001,"5min":0.20699999999999999,"15min":0.20100000000000001},"max":{"1min":3.9020000000000001,"5min":3.9020000000000001,"15min":28.882000000000001},"last":0.308},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.42299999999999999,"15min":0.69399999999999995},"min":{"1min":0.23899999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":5.0739999999999998,"5min":5.0739999999999998,"15min":28.382999999999999},"last":0.26300000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:15:05 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.38900000000000001,"15min":0.70199999999999996},"min":{"1min":0.28499999999999998,"5min":0.27100000000000002,"15min":0.23499999999999999},"max":{"1min":0.51200000000000001,"5min":0.68899999999999995,"15min":28.742000000000001},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.44500000000000001,"15min":0.63500000000000001},"min":{"1min":0.29199999999999998,"5min":0.26200000000000001,"15min":0.25700000000000001},"max":{"1min":0.56399999999999995,"5min":4.4550000000000001,"15min":26.719999999999999},"last":0.32700000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55200000000000005,"15min":0.68600000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.999,"5min":0.999,"15min":24.356000000000002},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56799999999999995,"15min":0.70399999999999996},"min":{"1min":0.36399999999999999,"5min":0.33700000000000002,"15min":0.32400000000000001},"max":{"1min":0.83099999999999996,"5min":1.2010000000000001,"15min":20.707000000000001},"last":0.47599999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:15:56 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.59699999999999998,"15min":0.67300000000000004},"min":{"1min":0.375,"5min":0.35999999999999999,"15min":0.33500000000000002},"max":{"1min":3.9540000000000002,"5min":3.9540000000000002,"15min":9.9499999999999993},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.59299999999999997,"15min":0.77700000000000002},"min":{"1min":0.39500000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.079,"5min":1.135,"15min":26.606000000000002},"last":0.46600000000000003}]},{"osd":6,"last update":"Mon Feb 19 21:15:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58499999999999996,"15min":0.79200000000000004},"min":{"1min":0.36499999999999999,"5min":0.35299999999999998,"15min":0.34300000000000003},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":24.381},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.59499999999999997,"15min":0.73099999999999998},"min":{"1min":0.45000000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.85699999999999998,"5min":0.94999999999999996,"15min":28.706},"last":0.503}]},{"osd":7,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60399999999999998,"15min":0.78300000000000003},"min":{"1min":0.45700000000000002,"5min":0.42699999999999999,"15min":0.34200000000000003},"max":{"1min":1.05,"5min":1.05,"15min":26.658000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.61499999999999999,"15min":0.73699999999999999},"min":{"1min":0.40400000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":24.402000000000001},"last":0.432}]}]},{"osd":3,"up_from":25,"seq":107374182600,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312016,"kb_used_data":2504,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93433136,"statfs":{"total":95995035648,"available":95675531264,"internally_reserved":0,"allocated":2564096,"data_stored":989879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.33300000000000002,"15min":0.70999999999999996},"min":{"1min":0.20300000000000001,"5min":0.192,"15min":0.156},"max":{"1min":0.55100000000000005,"5min":1.5449999999999999,"15min":77.662000000000006},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.35099999999999998,"15min":0.48299999999999998},"min":{"1min":0.22,"5min":0.20899999999999999,"15min":0.19700000000000001},"max":{"1min":0.443,"5min":0.60199999999999998,"15min":17.456},"last":0.35699999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.35199999999999998,"15min":0.51600000000000001},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.22800000000000001},"max":{"1min":0.56599999999999995,"5min":0.67600000000000005,"15min":17.327999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.35999999999999999,"15min":0.497},"min":{"1min":0.23000000000000001,"5min":0.20699999999999999,"15min":0.20000000000000001},"max":{"1min":0.625,"5min":0.98999999999999999,"15min":15.004},"last":0.625}]},{"osd":2,"last update":"Mon Feb 19 21:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.38,"15min":0.65400000000000003},"min":{"1min":0.26000000000000001,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":0.71199999999999997,"5min":0.92600000000000005,"15min":56.890999999999998},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.376,"15min":0.63700000000000001},"min":{"1min":0.27600000000000002,"5min":0.24299999999999999,"15min":0.20899999999999999},"max":{"1min":0.50700000000000001,"5min":1.3200000000000001,"15min":44.600999999999999},"last":0.46500000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:15:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.53400000000000003,"15min":0.63400000000000001},"min":{"1min":0.41699999999999998,"5min":0.32100000000000001,"15min":0.31900000000000001},"max":{"1min":0.82599999999999996,"5min":1.3340000000000001,"15min":12.847},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53200000000000003,"15min":0.70199999999999996},"min":{"1min":0.42299999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.77500000000000002,"5min":1.3560000000000001,"15min":26.393999999999998},"last":0.39900000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54500000000000004,"15min":0.67900000000000005},"min":{"1min":0.47499999999999998,"5min":0.33200000000000002,"15min":0.313},"max":{"1min":0.82899999999999996,"5min":0.95699999999999996,"15min":17.401},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55500000000000005,"15min":0.65600000000000003},"min":{"1min":0.435,"5min":0.35999999999999999,"15min":0.313},"max":{"1min":0.871,"5min":1.397,"15min":12.882},"last":0.58699999999999997}]},{"osd":6,"last update":"Mon Feb 19 21:15:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55200000000000005,"15min":0.65200000000000002},"min":{"1min":0.39100000000000001,"5min":0.36899999999999999,"15min":0.34300000000000003},"max":{"1min":0.83999999999999997,"5min":0.90500000000000003,"15min":17.367000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55900000000000005,"15min":0.66000000000000003},"min":{"1min":0.41299999999999998,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":0.69699999999999995,"5min":1.3660000000000001,"15min":12.904},"last":0.68600000000000005}]},{"osd":7,"last update":"Mon Feb 19 21:15:35 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.55800000000000005,"15min":0.65700000000000003},"min":{"1min":0.42199999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.91500000000000004,"5min":1.038,"15min":17.428999999999998},"last":0.504},{"interface":"front","average":{"1min":0.629,"5min":0.58899999999999997,"15min":0.80700000000000005},"min":{"1min":0.45000000000000001,"5min":0.374,"15min":0.34499999999999997},"max":{"1min":1.1859999999999999,"5min":1.3879999999999999,"15min":44.378},"last":0.52800000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986372,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299724,"kb_used_data":2500,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93445428,"statfs":{"total":95995035648,"available":95688118272,"internally_reserved":0,"allocated":2560000,"data_stored":998084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.56899999999999995,"15min":0.627},"min":{"1min":0.32500000000000001,"5min":0.28199999999999997,"15min":0.22500000000000001},"max":{"1min":0.66400000000000003,"5min":8.0530000000000008,"15min":19.878},"last":0.87},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.85599999999999998},"min":{"1min":0.32900000000000001,"5min":0.26700000000000002,"15min":0.20499999999999999},"max":{"1min":0.67500000000000004,"5min":3.3530000000000002,"15min":70.063000000000002},"last":0.70399999999999996}]},{"osd":1,"last update":"Mon Feb 19 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.503,"15min":0.63300000000000001},"min":{"1min":0.33800000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.71899999999999997,"5min":1.6499999999999999,"15min":26.510999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.502,"15min":0.67900000000000005},"min":{"1min":0.38500000000000001,"5min":0.32100000000000001,"15min":0.23200000000000001},"max":{"1min":0.63,"5min":0.89700000000000002,"15min":25.949000000000002},"last":0.82699999999999996}]},{"osd":2,"last update":"Mon Feb 19 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.58699999999999997,"15min":0.72099999999999997},"min":{"1min":0.376,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71399999999999997,"5min":8.1240000000000006,"15min":37.320999999999998},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53400000000000003,"15min":1.0509999999999999},"min":{"1min":0.33000000000000002,"5min":0.30599999999999999,"15min":0.253},"max":{"1min":0.66800000000000004,"5min":2.9969999999999999,"15min":69.936999999999998},"last":0.80100000000000005}]},{"osd":3,"last update":"Mon Feb 19 21:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54200000000000004,"15min":0.91000000000000003},"min":{"1min":0.38800000000000001,"5min":0.28999999999999998,"15min":0.27700000000000002},"max":{"1min":0.66900000000000004,"5min":3.323,"15min":69.793000000000006},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.53800000000000003,"15min":0.75700000000000001},"min":{"1min":0.33800000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70499999999999996,"5min":1.621,"15min":17.986999999999998},"last":0.84899999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.44600000000000001,"15min":0.42699999999999999},"min":{"1min":0.28599999999999998,"5min":0.22900000000000001,"15min":0.218},"max":{"1min":0.65800000000000003,"5min":1.532,"15min":3.2799999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.49099999999999999,"15min":0.47599999999999998},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.23899999999999999},"max":{"1min":0.76100000000000001,"5min":1.5700000000000001,"15min":4.4340000000000002},"last":0.52600000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:15:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.55500000000000005,"15min":0.48799999999999999},"min":{"1min":0.30599999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63,"5min":8.0969999999999995,"15min":8.0969999999999995},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.57699999999999996,"15min":0.53200000000000003},"min":{"1min":0.34300000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71099999999999997,"5min":3.0270000000000001,"15min":4.5609999999999999},"last":0.57999999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:15:48 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55800000000000005,"15min":0.53300000000000003},"min":{"1min":0.374,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.77100000000000002,"5min":3.2909999999999999,"15min":6.0730000000000004},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.55300000000000005,"15min":0.53300000000000003},"min":{"1min":0.313,"5min":0.309,"15min":0.26400000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":3.202},"last":0.54000000000000004}]}]},{"osd":5,"up_from":37,"seq":158913790143,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303792,"kb_used_data":2472,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93441360,"statfs":{"total":95995035648,"available":95683952640,"internally_reserved":0,"allocated":2531328,"data_stored":982476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.47599999999999998,"15min":1.8720000000000001},"min":{"1min":0.34799999999999998,"5min":0.23499999999999999,"15min":0.19500000000000001},"max":{"1min":0.748,"5min":0.748,"15min":276.47899999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.49199999999999999,"15min":0.89300000000000002},"min":{"1min":0.38,"5min":0.27400000000000002,"15min":0.20899999999999999},"max":{"1min":0.70399999999999996,"5min":0.70599999999999996,"15min":52.887},"last":0.36599999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.46500000000000002,"15min":1.2569999999999999},"min":{"1min":0.40799999999999997,"5min":0.28199999999999997,"15min":0.23499999999999999},"max":{"1min":0.67700000000000005,"5min":0.77100000000000002,"15min":120.011},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.51200000000000001,"15min":1.0189999999999999},"min":{"1min":0.39600000000000002,"5min":0.28599999999999998,"15min":0.26200000000000001},"max":{"1min":0.68200000000000005,"5min":1.5029999999999999,"15min":118.105},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.497,"15min":0.98499999999999999},"min":{"1min":0.41499999999999998,"5min":0.32300000000000001,"15min":0.27200000000000002},"max":{"1min":0.73099999999999998,"5min":0.92300000000000004,"15min":82.242000000000004},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.54300000000000004,"15min":1.5109999999999999},"min":{"1min":0.50800000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.79000000000000004,"5min":2.9820000000000002,"15min":232.84999999999999},"last":0.63300000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.52600000000000002,"15min":1.0669999999999999},"min":{"1min":0.44700000000000001,"5min":0.32300000000000001,"15min":0.27900000000000003},"max":{"1min":0.76200000000000001,"5min":2.2210000000000001,"15min":109.70099999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.54200000000000004,"15min":0.90700000000000003},"min":{"1min":0.51600000000000001,"5min":0.32700000000000001,"15min":0.32000000000000001},"max":{"1min":0.70899999999999996,"5min":1.4590000000000001,"15min":65.519000000000005},"last":0.65400000000000003}]},{"osd":4,"last update":"Mon Feb 19 21:15:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.44600000000000001,"15min":0.438},"min":{"1min":0.28699999999999998,"5min":0.26400000000000001,"15min":0.23300000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":1.861},"last":0.505},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.46400000000000002,"15min":0.45500000000000002},"min":{"1min":0.313,"5min":0.255,"15min":0.248},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":1.9730000000000001},"last":0.58799999999999997}]},{"osd":6,"last update":"Mon Feb 19 21:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.46400000000000002,"15min":0.46000000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.26400000000000001},"max":{"1min":0.622,"5min":0.871,"15min":1.823},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.46899999999999997,"15min":0.46999999999999997},"min":{"1min":0.32700000000000001,"5min":0.30399999999999999,"15min":0.27100000000000002},"max":{"1min":0.68999999999999995,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.56000000000000005}]},{"osd":7,"last update":"Mon Feb 19 21:15:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.505,"15min":0.505},"min":{"1min":0.34699999999999998,"5min":0.30199999999999999,"15min":0.28499999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":1.8879999999999999},"last":0.624},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.52400000000000002,"15min":0.51300000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.27800000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.96799999999999997},"last":0.64300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":247,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":247,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":247,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":247,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":247,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":247,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":247,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":247,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:16:08.140 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-19T21:16:08.141 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:16:08.141 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-02-19T21:16:08.141 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-19T21:16:08.141 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-19T21:16:08.141 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-02-19T21:16:08.169 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:08.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:08.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:08.273 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:08.307 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:08.362 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:08.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:08.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:08.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.380+0000 7f00380d8700 1 -- 172.21.15.163:0/298488357 >> v1:172.21.15.163:6789/0 conn(0x7f00301042b0 legacy=0x7f0030106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:08.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.380+0000 7f00380d8700 1 -- 172.21.15.163:0/298488357 shutdown_connections 2024-02-19T21:16:08.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.380+0000 7f00380d8700 1 -- 172.21.15.163:0/298488357 >> 172.21.15.163:0/298488357 conn(0x7f00300f2470 msgr2=0x7f00300f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:08.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.380+0000 7f00380d8700 1 -- 172.21.15.163:0/298488357 shutdown_connections 2024-02-19T21:16:08.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.380+0000 7f00380d8700 1 -- 172.21.15.163:0/298488357 wait complete. 2024-02-19T21:16:08.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.381+0000 7f00380d8700 1 librados: starting msgr at 2024-02-19T21:16:08.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.381+0000 7f00380d8700 1 librados: starting objecter 2024-02-19T21:16:08.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.381+0000 7f00380d8700 1 Processor -- start 2024-02-19T21:16:08.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.381+0000 7f00380d8700 1 -- start start 2024-02-19T21:16:08.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.381+0000 7f00380d8700 1 librados: setting wanted keys 2024-02-19T21:16:08.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.381+0000 7f00380d8700 1 librados: calling monclient init 2024-02-19T21:16:08.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.381+0000 7f00380d8700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0030117470 con 0x7f00300f8980 2024-02-19T21:16:08.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f00380d8700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0030118650 con 0x7f00300f9530 2024-02-19T21:16:08.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f00380d8700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0030119850 con 0x7f00301042b0 2024-02-19T21:16:08.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0035e74700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f00300f8980 0x7f00300ff7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:59504/0 (socket says 172.21.15.163:59504) 2024-02-19T21:16:08.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0036675700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f00301042b0 0x7f0030115b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:53346/0 (socket says 172.21.15.163:53346) 2024-02-19T21:16:08.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0035e74700 1 -- 172.21.15.163:0/2924879613 learned_addr learned my addr 172.21.15.163:0/2924879613 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:08.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0035673700 1 --1- 172.21.15.163:0/2924879613 >> v1:172.21.15.193:6789/0 conn(0x7f00300f9530 0x7f00300ffee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:49208/0 (socket says 172.21.15.163:49208) 2024-02-19T21:16:08.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 10 client.?.objecter ms_handle_connect 0x7f00300f8980 2024-02-19T21:16:08.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:08.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 10 client.?.objecter ms_handle_connect 0x7f00300f9530 2024-02-19T21:16:08.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:08.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 10 client.?.objecter ms_handle_connect 0x7f00301042b0 2024-02-19T21:16:08.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:08.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538462673 0 0) 0x7f0030117470 con 0x7f00300f8980 2024-02-19T21:16:08.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001c0033f0 con 0x7f00300f8980 2024-02-19T21:16:08.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165967031 0 0) 0x7f0030118650 con 0x7f00300f9530 2024-02-19T21:16:08.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0030117470 con 0x7f00300f9530 2024-02-19T21:16:08.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370503907 0 0) 0x7f0030119850 con 0x7f00301042b0 2024-02-19T21:16:08.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0030118650 con 0x7f00301042b0 2024-02-19T21:16:08.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 927450948 0 0) 0x7f001c0033f0 con 0x7f00300f8980 2024-02-19T21:16:08.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0030119850 con 0x7f00300f8980 2024-02-19T21:16:08.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f00200019d0 con 0x7f00300f8980 2024-02-19T21:16:08.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2931558323 0 0) 0x7f0030117470 con 0x7f00300f9530 2024-02-19T21:16:08.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f001c0033f0 con 0x7f00300f9530 2024-02-19T21:16:08.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2200246026 0 0) 0x7f0030118650 con 0x7f00301042b0 2024-02-19T21:16:08.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0030117470 con 0x7f00301042b0 2024-02-19T21:16:08.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f002c001840 con 0x7f00300f9530 2024-02-19T21:16:08.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f00280023d0 con 0x7f00301042b0 2024-02-19T21:16:08.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2067573120 0 0) 0x7f0030119850 con 0x7f00300f8980 2024-02-19T21:16:08.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 >> v1:172.21.15.163:6790/0 conn(0x7f00301042b0 legacy=0x7f0030115b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:08.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 >> v1:172.21.15.193:6789/0 conn(0x7f00300f9530 legacy=0x7f00300ffee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:08.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003011aa50 con 0x7f00300f8980 2024-02-19T21:16:08.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0030118840 con 0x7f00300f8980 2024-02-19T21:16:08.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f00380d8700 10 client.23763.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:08.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0030118cb0 con 0x7f00300f8980 2024-02-19T21:16:08.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.382+0000 7f00380d8700 1 librados: init done 2024-02-19T21:16:08.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.383+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f00200023e0 con 0x7f00300f8980 2024-02-19T21:16:08.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.383+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0020005b40 con 0x7f00300f8980 2024-02-19T21:16:08.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.383+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0014004f00 con 0x7f00300f8980 2024-02-19T21:16:08.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.384+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f0020001f40 con 0x7f00300f8980 2024-02-19T21:16:08.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.384+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(718..718 src has 623..718) v4 ==== 8670+0+0 (unknown 2919477939 0 0) 0x7f0020078fc0 con 0x7f00300f8980 2024-02-19T21:16:08.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.384+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 osd_map(718..718 src has 623..718) v4 2024-02-19T21:16:08.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.384+0000 7f0026ffd700 3 client.23763.objecter handle_osd_map got epochs [718,718] > 0 2024-02-19T21:16:08.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.384+0000 7f0026ffd700 3 client.23763.objecter handle_osd_map decoding full epoch 718 2024-02-19T21:16:08.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.385+0000 7f0026ffd700 20 client.23763.objecter dump_active .. 0 homeless 2024-02-19T21:16:08.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.385+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v718) v1 ==== 54+0+16 (unknown 1396274258 0 2420000192) 0x7f00200793a0 con 0x7f00300f8980 2024-02-19T21:16:08.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.385+0000 7f0026ffd700 10 client.23763.objecter ms_handle_connect 0x7f001c063da0 2024-02-19T21:16:08.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.385+0000 7f00380d8700 10 client.23763.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:08.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.385+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 1 2024-02-19T21:16:08.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.385+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f00140051a0 osd=0 3 2024-02-19T21:16:08.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.385+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f00140051a0 osd=0 3 2024-02-19T21:16:08.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.385+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f00140051a0 osd=0 3 2024-02-19T21:16:08.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.385+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 1 2024-02-19T21:16:08.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.385+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f00140051a0 osd=0 3 2024-02-19T21:16:08.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.385+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 0 1 2024-02-19T21:16:08.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.385+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f00140051a0 osd=0 4 2024-02-19T21:16:08.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.385+0000 7f00380d8700 10 client.23763.objecter _send_command 1 2024-02-19T21:16:08.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.385+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0014004950 con 0x7f0014004530 2024-02-19T21:16:08.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.385+0000 7f0026ffd700 10 client.23763.objecter ms_handle_connect 0x7f0014004530 2024-02-19T21:16:08.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.386+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0014004950 con 0x7f0014004530 2024-02-19T21:16:08.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.386+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:16:08.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.386+0000 7f0026ffd700 10 client.23763.objecter _finish_command 1 = system:0 2024-02-19T21:16:08.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.386+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f00140051a0 osd=0 4 2024-02-19T21:16:08.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.386+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 0 1 2024-02-19T21:16:08.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.401+0000 7f00380d8700 10 client.23763.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:08.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.401+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 2 2024-02-19T21:16:08.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.401+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f00140051a0 osd=0 3 2024-02-19T21:16:08.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.401+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f00140051a0 osd=0 3 2024-02-19T21:16:08.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.401+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f00140051a0 osd=0 3 2024-02-19T21:16:08.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.401+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 2 2024-02-19T21:16:08.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.401+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f00140051a0 osd=0 3 2024-02-19T21:16:08.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.401+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 0 2 2024-02-19T21:16:08.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.401+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f00140051a0 osd=0 4 2024-02-19T21:16:08.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.401+0000 7f00380d8700 10 client.23763.objecter _send_command 2 2024-02-19T21:16:08.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.401+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0014004d30 con 0x7f0014004530 2024-02-19T21:16:08.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f0014004d30 con 0x7f0014004530 2024-02-19T21:16:08.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:08.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f0026ffd700 10 client.23763.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f00140051a0 osd=0 4 2024-02-19T21:16:08.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 0 2 2024-02-19T21:16:08.404 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:16:08.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f00380d8700 10 client.23763.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:08.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 3 2024-02-19T21:16:08.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f0014004270 osd=1 3 2024-02-19T21:16:08.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f0014004270 osd=1 3 2024-02-19T21:16:08.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f0014004270 osd=1 3 2024-02-19T21:16:08.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 3 2024-02-19T21:16:08.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f0014004270 osd=1 3 2024-02-19T21:16:08.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 1 3 2024-02-19T21:16:08.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f0014004270 osd=1 4 2024-02-19T21:16:08.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f00380d8700 10 client.23763.objecter _send_command 3 2024-02-19T21:16:08.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f001400bf60 con 0x7f0014003b40 2024-02-19T21:16:08.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.403+0000 7f0026ffd700 10 client.23763.objecter ms_handle_connect 0x7f0014003b40 2024-02-19T21:16:08.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.404+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f001400bf60 con 0x7f0014003b40 2024-02-19T21:16:08.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.404+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:16:08.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.404+0000 7f0026ffd700 10 client.23763.objecter _finish_command 3 = system:0 2024-02-19T21:16:08.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.404+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f0014004270 osd=1 4 2024-02-19T21:16:08.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.404+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 1 3 2024-02-19T21:16:08.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.418+0000 7f00380d8700 10 client.23763.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:08.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.418+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 4 2024-02-19T21:16:08.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.418+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f0014004270 osd=1 3 2024-02-19T21:16:08.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.418+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f0014004270 osd=1 3 2024-02-19T21:16:08.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.418+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f0014004270 osd=1 3 2024-02-19T21:16:08.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.418+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 4 2024-02-19T21:16:08.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.418+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f0014004270 osd=1 3 2024-02-19T21:16:08.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.418+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 1 4 2024-02-19T21:16:08.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.418+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f0014004270 osd=1 4 2024-02-19T21:16:08.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.418+0000 7f00380d8700 10 client.23763.objecter _send_command 4 2024-02-19T21:16:08.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.418+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f001400d280 con 0x7f0014003b40 2024-02-19T21:16:08.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f001400d280 con 0x7f0014003b40 2024-02-19T21:16:08.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:08.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f0026ffd700 10 client.23763.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f0014004270 osd=1 4 2024-02-19T21:16:08.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 1 4 2024-02-19T21:16:08.421 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:16:08.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f00380d8700 10 client.23763.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:08.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 5 2024-02-19T21:16:08.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f00140033a0 osd=2 3 2024-02-19T21:16:08.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f00140033a0 osd=2 3 2024-02-19T21:16:08.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f00140033a0 osd=2 3 2024-02-19T21:16:08.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 5 2024-02-19T21:16:08.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f00140033a0 osd=2 3 2024-02-19T21:16:08.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 2 5 2024-02-19T21:16:08.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f00140033a0 osd=2 4 2024-02-19T21:16:08.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f00380d8700 10 client.23763.objecter _send_command 5 2024-02-19T21:16:08.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0014010cd0 con 0x7f0014002070 2024-02-19T21:16:08.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.420+0000 7f0026ffd700 10 client.23763.objecter ms_handle_connect 0x7f0014002070 2024-02-19T21:16:08.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.421+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0014010cd0 con 0x7f0014002070 2024-02-19T21:16:08.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.421+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:16:08.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.421+0000 7f0026ffd700 10 client.23763.objecter _finish_command 5 = system:0 2024-02-19T21:16:08.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.421+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f00140033a0 osd=2 4 2024-02-19T21:16:08.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.421+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 2 5 2024-02-19T21:16:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.439+0000 7f00380d8700 10 client.23763.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.439+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 6 2024-02-19T21:16:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.439+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f00140033a0 osd=2 3 2024-02-19T21:16:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.439+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f00140033a0 osd=2 3 2024-02-19T21:16:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.439+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f00140033a0 osd=2 3 2024-02-19T21:16:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.439+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 6 2024-02-19T21:16:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.439+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f00140033a0 osd=2 3 2024-02-19T21:16:08.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.439+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 2 6 2024-02-19T21:16:08.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.439+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f00140033a0 osd=2 4 2024-02-19T21:16:08.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.439+0000 7f00380d8700 10 client.23763.objecter _send_command 6 2024-02-19T21:16:08.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.439+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0014001630 con 0x7f0014002070 2024-02-19T21:16:08.443 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:16:08.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.441+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f0014001630 con 0x7f0014002070 2024-02-19T21:16:08.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.441+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:08.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.441+0000 7f0026ffd700 10 client.23763.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.441+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f00140033a0 osd=2 4 2024-02-19T21:16:08.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.441+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 2 6 2024-02-19T21:16:08.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.442+0000 7f00380d8700 10 client.23763.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:08.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.442+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 7 2024-02-19T21:16:08.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.442+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001400c8a0 osd=3 3 2024-02-19T21:16:08.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.442+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001400c8a0 osd=3 3 2024-02-19T21:16:08.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.442+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001400c8a0 osd=3 3 2024-02-19T21:16:08.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.442+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 7 2024-02-19T21:16:08.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.442+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f001400c8a0 osd=3 3 2024-02-19T21:16:08.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.442+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 3 7 2024-02-19T21:16:08.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.442+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001400c8a0 osd=3 4 2024-02-19T21:16:08.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.442+0000 7f00380d8700 10 client.23763.objecter _send_command 7 2024-02-19T21:16:08.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.442+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f001400d020 con 0x7f0014010e40 2024-02-19T21:16:08.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.443+0000 7f0026ffd700 10 client.23763.objecter ms_handle_connect 0x7f0014010e40 2024-02-19T21:16:08.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.444+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f001400d020 con 0x7f0014010e40 2024-02-19T21:16:08.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.444+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:16:08.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.444+0000 7f0026ffd700 10 client.23763.objecter _finish_command 7 = system:0 2024-02-19T21:16:08.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.444+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f001400c8a0 osd=3 4 2024-02-19T21:16:08.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.444+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 3 7 2024-02-19T21:16:08.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.459+0000 7f00380d8700 10 client.23763.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:08.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.459+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 8 2024-02-19T21:16:08.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.459+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001400c8a0 osd=3 3 2024-02-19T21:16:08.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.459+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001400c8a0 osd=3 3 2024-02-19T21:16:08.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.459+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001400c8a0 osd=3 3 2024-02-19T21:16:08.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.459+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 8 2024-02-19T21:16:08.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.459+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f001400c8a0 osd=3 3 2024-02-19T21:16:08.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.459+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 3 8 2024-02-19T21:16:08.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.459+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001400c8a0 osd=3 4 2024-02-19T21:16:08.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.459+0000 7f00380d8700 10 client.23763.objecter _send_command 8 2024-02-19T21:16:08.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.459+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f00140121b0 con 0x7f0014010e40 2024-02-19T21:16:08.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f00140121b0 con 0x7f0014010e40 2024-02-19T21:16:08.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:08.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f0026ffd700 10 client.23763.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f001400c8a0 osd=3 4 2024-02-19T21:16:08.462 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 3 8 2024-02-19T21:16:08.462 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:16:08.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f00380d8700 10 client.23763.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:08.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 9 2024-02-19T21:16:08.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f00140121b0 osd=4 3 2024-02-19T21:16:08.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f00140121b0 osd=4 3 2024-02-19T21:16:08.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f00140121b0 osd=4 3 2024-02-19T21:16:08.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 9 2024-02-19T21:16:08.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f00140121b0 osd=4 3 2024-02-19T21:16:08.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 4 9 2024-02-19T21:16:08.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f00140121b0 osd=4 4 2024-02-19T21:16:08.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f00380d8700 10 client.23763.objecter _send_command 9 2024-02-19T21:16:08.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.461+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f001401a270 con 0x7f0014016690 2024-02-19T21:16:08.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.462+0000 7f0026ffd700 10 client.23763.objecter ms_handle_connect 0x7f0014016690 2024-02-19T21:16:08.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.462+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f001401a270 con 0x7f0014016690 2024-02-19T21:16:08.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.462+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:16:08.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.462+0000 7f0026ffd700 10 client.23763.objecter _finish_command 9 = system:0 2024-02-19T21:16:08.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.462+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f00140121b0 osd=4 4 2024-02-19T21:16:08.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.462+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 4 9 2024-02-19T21:16:08.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:16:08.467 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-19T21:16:08.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.477+0000 7f00380d8700 10 client.23763.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:08.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.477+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 10 2024-02-19T21:16:08.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.477+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f00140121b0 osd=4 3 2024-02-19T21:16:08.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.477+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f00140121b0 osd=4 3 2024-02-19T21:16:08.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.477+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f00140121b0 osd=4 3 2024-02-19T21:16:08.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.477+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 10 2024-02-19T21:16:08.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.477+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f00140121b0 osd=4 3 2024-02-19T21:16:08.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.477+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 4 10 2024-02-19T21:16:08.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.477+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f00140121b0 osd=4 4 2024-02-19T21:16:08.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.477+0000 7f00380d8700 10 client.23763.objecter _send_command 10 2024-02-19T21:16:08.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.477+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f001401ba20 con 0x7f0014016690 2024-02-19T21:16:08.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f001401ba20 con 0x7f0014016690 2024-02-19T21:16:08.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:08.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f0026ffd700 10 client.23763.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f00140121b0 osd=4 4 2024-02-19T21:16:08.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 4 10 2024-02-19T21:16:08.480 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:16:08.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f00380d8700 10 client.23763.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:08.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 11 2024-02-19T21:16:08.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001401ba20 osd=5 3 2024-02-19T21:16:08.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001401ba20 osd=5 3 2024-02-19T21:16:08.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001401ba20 osd=5 3 2024-02-19T21:16:08.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 11 2024-02-19T21:16:08.481 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f001401ba20 osd=5 3 2024-02-19T21:16:08.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 5 11 2024-02-19T21:16:08.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001401ba20 osd=5 4 2024-02-19T21:16:08.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f00380d8700 10 client.23763.objecter _send_command 11 2024-02-19T21:16:08.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f001401c620 con 0x7f001401c0d0 2024-02-19T21:16:08.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.479+0000 7f0026ffd700 10 client.23763.objecter ms_handle_connect 0x7f001401c0d0 2024-02-19T21:16:08.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.480+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f001401c620 con 0x7f001401c0d0 2024-02-19T21:16:08.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.480+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:16:08.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.480+0000 7f0026ffd700 10 client.23763.objecter _finish_command 11 = system:0 2024-02-19T21:16:08.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.480+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f001401ba20 osd=5 4 2024-02-19T21:16:08.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.480+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 5 11 2024-02-19T21:16:08.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.494+0000 7f00380d8700 10 client.23763.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:08.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.494+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 12 2024-02-19T21:16:08.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.494+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001401ba20 osd=5 3 2024-02-19T21:16:08.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.494+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001401ba20 osd=5 3 2024-02-19T21:16:08.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.494+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001401ba20 osd=5 3 2024-02-19T21:16:08.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.494+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 12 2024-02-19T21:16:08.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.494+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f001401ba20 osd=5 3 2024-02-19T21:16:08.495 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.494+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 5 12 2024-02-19T21:16:08.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.494+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001401ba20 osd=5 4 2024-02-19T21:16:08.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.494+0000 7f00380d8700 10 client.23763.objecter _send_command 12 2024-02-19T21:16:08.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.494+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f001401c620 con 0x7f001401c0d0 2024-02-19T21:16:08.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f001401c620 con 0x7f001401c0d0 2024-02-19T21:16:08.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:08.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f0026ffd700 10 client.23763.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f001401ba20 osd=5 4 2024-02-19T21:16:08.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 5 12 2024-02-19T21:16:08.498 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:16:08.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f00380d8700 10 client.23763.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:08.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 13 2024-02-19T21:16:08.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001401c620 osd=6 3 2024-02-19T21:16:08.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001401c620 osd=6 3 2024-02-19T21:16:08.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001401c620 osd=6 3 2024-02-19T21:16:08.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 13 2024-02-19T21:16:08.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f001401c620 osd=6 3 2024-02-19T21:16:08.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 6 13 2024-02-19T21:16:08.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001401c620 osd=6 4 2024-02-19T21:16:08.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f00380d8700 10 client.23763.objecter _send_command 13 2024-02-19T21:16:08.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.496+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f001401ca50 con 0x7f001401f130 2024-02-19T21:16:08.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.497+0000 7f0026ffd700 10 client.23763.objecter ms_handle_connect 0x7f001401f130 2024-02-19T21:16:08.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.497+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f001401ca50 con 0x7f001401f130 2024-02-19T21:16:08.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.497+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:16:08.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.497+0000 7f0026ffd700 10 client.23763.objecter _finish_command 13 = system:0 2024-02-19T21:16:08.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.497+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f001401c620 osd=6 4 2024-02-19T21:16:08.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.497+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 6 13 2024-02-19T21:16:08.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.512+0000 7f00380d8700 10 client.23763.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:08.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.512+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 14 2024-02-19T21:16:08.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.512+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001401c620 osd=6 3 2024-02-19T21:16:08.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.512+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001401c620 osd=6 3 2024-02-19T21:16:08.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.512+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001401c620 osd=6 3 2024-02-19T21:16:08.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.512+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 14 2024-02-19T21:16:08.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.512+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f001401c620 osd=6 3 2024-02-19T21:16:08.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.512+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 6 14 2024-02-19T21:16:08.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.512+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001401c620 osd=6 4 2024-02-19T21:16:08.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.512+0000 7f00380d8700 10 client.23763.objecter _send_command 14 2024-02-19T21:16:08.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.512+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f001401ca10 con 0x7f001401f130 2024-02-19T21:16:08.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f001401ca10 con 0x7f001401f130 2024-02-19T21:16:08.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:08.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f0026ffd700 10 client.23763.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f001401c620 osd=6 4 2024-02-19T21:16:08.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 6 14 2024-02-19T21:16:08.515 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:16:08.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f00380d8700 10 client.23763.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:08.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 15 2024-02-19T21:16:08.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001401ca10 osd=7 3 2024-02-19T21:16:08.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001401ca10 osd=7 3 2024-02-19T21:16:08.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001401ca10 osd=7 3 2024-02-19T21:16:08.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 15 2024-02-19T21:16:08.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f001401ca10 osd=7 3 2024-02-19T21:16:08.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 7 15 2024-02-19T21:16:08.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001401ca10 osd=7 4 2024-02-19T21:16:08.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f00380d8700 10 client.23763.objecter _send_command 15 2024-02-19T21:16:08.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0014023d30 con 0x7f0014024790 2024-02-19T21:16:08.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.514+0000 7f0026ffd700 10 client.23763.objecter ms_handle_connect 0x7f0014024790 2024-02-19T21:16:08.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.515+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0014023d30 con 0x7f0014024790 2024-02-19T21:16:08.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.515+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:16:08.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.515+0000 7f0026ffd700 10 client.23763.objecter _finish_command 15 = system:0 2024-02-19T21:16:08.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.515+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f001401ca10 osd=7 4 2024-02-19T21:16:08.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.515+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 7 15 2024-02-19T21:16:08.520 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:16:08.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.529+0000 7f00380d8700 10 client.23763.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:08.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.529+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign -1 16 2024-02-19T21:16:08.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.529+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001401ca10 osd=7 3 2024-02-19T21:16:08.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.529+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001401ca10 osd=7 3 2024-02-19T21:16:08.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.529+0000 7f00380d8700 20 client.23763.objecter _get_session s=0x7f001401ca10 osd=7 3 2024-02-19T21:16:08.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.529+0000 7f00380d8700 15 client.23763.objecter _session_command_op_remove -1 16 2024-02-19T21:16:08.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.529+0000 7f00380d8700 20 client.23763.objecter get_session s=0x7f001401ca10 osd=7 3 2024-02-19T21:16:08.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.529+0000 7f00380d8700 15 client.23763.objecter _session_command_op_assign 7 16 2024-02-19T21:16:08.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.529+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001401ca10 osd=7 4 2024-02-19T21:16:08.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.529+0000 7f00380d8700 10 client.23763.objecter _send_command 16 2024-02-19T21:16:08.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.529+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0014023cf0 con 0x7f0014024790 2024-02-19T21:16:08.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f0026ffd700 1 -- 172.21.15.163:0/2924879613 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f0014023cf0 con 0x7f0014024790 2024-02-19T21:16:08.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f0026ffd700 10 client.23763.objecter ms_dispatch 0x7f0030001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:08.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f0026ffd700 10 client.23763.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f0026ffd700 20 client.23763.objecter put_session s=0x7f001401ca10 osd=7 4 2024-02-19T21:16:08.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f0026ffd700 15 client.23763.objecter _session_command_op_remove 7 16 2024-02-19T21:16:08.533 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:16:08.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 10 librados: watch_flush enter 2024-02-19T21:16:08.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 10 librados: watch_flush exit 2024-02-19T21:16:08.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 10 client.23763.objecter close_session for osd.0 2024-02-19T21:16:08.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 >> v1:172.21.15.163:6801/1559418115 conn(0x7f0014004530 legacy=0x7f0014002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:08.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f00140051a0 osd=0 1 2024-02-19T21:16:08.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 10 client.23763.objecter close_session for osd.1 2024-02-19T21:16:08.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 >> v1:172.21.15.163:6805/3125047432 conn(0x7f0014003b40 legacy=0x7f001400a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:08.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f0014004270 osd=1 1 2024-02-19T21:16:08.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 10 client.23763.objecter close_session for osd.2 2024-02-19T21:16:08.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 >> v1:172.21.15.163:6809/1563158487 conn(0x7f0014002070 legacy=0x7f001400f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:08.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f00140033a0 osd=2 1 2024-02-19T21:16:08.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 10 client.23763.objecter close_session for osd.3 2024-02-19T21:16:08.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 >> v1:172.21.15.163:6813/2112279769 conn(0x7f0014010e40 legacy=0x7f0014011280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:08.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001400c8a0 osd=3 1 2024-02-19T21:16:08.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 10 client.23763.objecter close_session for osd.4 2024-02-19T21:16:08.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f0036675700 1 -- 172.21.15.163:0/2924879613 reap_dead start 2024-02-19T21:16:08.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 >> v1:172.21.15.193:6800/3129716965 conn(0x7f0014016690 legacy=0x7f0014018a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:08.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f0036675700 1 -- 172.21.15.163:0/2924879613 reap_dead start 2024-02-19T21:16:08.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f00140121b0 osd=4 1 2024-02-19T21:16:08.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 10 client.23763.objecter close_session for osd.5 2024-02-19T21:16:08.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 >> v1:172.21.15.193:6804/805161991 conn(0x7f001401c0d0 legacy=0x7f001401a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:08.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001401ba20 osd=5 1 2024-02-19T21:16:08.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 10 client.23763.objecter close_session for osd.6 2024-02-19T21:16:08.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 >> v1:172.21.15.193:6808/787604615 conn(0x7f001401f130 legacy=0x7f0014021520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:08.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001401c620 osd=6 1 2024-02-19T21:16:08.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 10 client.23763.objecter close_session for osd.7 2024-02-19T21:16:08.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 >> v1:172.21.15.193:6812/157927432 conn(0x7f0014024790 legacy=0x7f0014024bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:08.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 20 client.23763.objecter put_session s=0x7f001401ca10 osd=7 1 2024-02-19T21:16:08.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 20 client.23763.objecter shutdown clearing up homeless session... 2024-02-19T21:16:08.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 10 client.23763.objecter successfully canceled tick 2024-02-19T21:16:08.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 >> v1:172.21.15.163:6800/1552514681 conn(0x7f001c063da0 legacy=0x7f001c066260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:08.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 >> v1:172.21.15.163:6789/0 conn(0x7f00300f8980 legacy=0x7f00300ff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:08.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f0036675700 1 -- 172.21.15.163:0/2924879613 reap_dead start 2024-02-19T21:16:08.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f0036675700 1 -- 172.21.15.163:0/2924879613 reap_dead start 2024-02-19T21:16:08.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 shutdown_connections 2024-02-19T21:16:08.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 >> 172.21.15.163:0/2924879613 conn(0x7f00300f2470 msgr2=0x7f00301039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:08.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 shutdown_connections 2024-02-19T21:16:08.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 1 -- 172.21.15.163:0/2924879613 wait complete. 2024-02-19T21:16:08.537 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:08.532+0000 7f00380d8700 1 librados: shutdown 2024-02-19T21:16:08.537 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.537 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.537 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.537 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.537 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.537 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.538 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.538 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:16:08.538 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:08.559 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:16:08.562 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:08.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:16:08.663 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:08.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:07.315706+0000 mgr.y (mgr.14550) 603 : cluster [DBG] pgmap v1082: 214 pgs: 214 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:08.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:07.348204+0000 mgr.y (mgr.14550) 604 : audit [DBG] from='client.50486 v1:172.21.15.163:0/510232126' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:16:08.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:07.421535+0000 mon.a (mon.0) 3469 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:16:08.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:07.434781+0000 mon.a (mon.0) 3470 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-126"}]': finished 2024-02-19T21:16:08.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:07.434841+0000 mon.a (mon.0) 3471 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2024-02-19T21:16:08.689 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:08.385293+0000 mon.a (mon.0) 3472 : audit [DBG] from='client.? v1:172.21.15.163:0/2924879613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:08.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:07.315706+0000 mgr.y (mgr.14550) 603 : cluster [DBG] pgmap v1082: 214 pgs: 214 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:08.689 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:07.348204+0000 mgr.y (mgr.14550) 604 : audit [DBG] from='client.50486 v1:172.21.15.163:0/510232126' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:16:08.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:07.421535+0000 mon.a (mon.0) 3469 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:16:08.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:07.434781+0000 mon.a (mon.0) 3470 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-126"}]': finished 2024-02-19T21:16:08.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:07.434841+0000 mon.a (mon.0) 3471 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2024-02-19T21:16:08.690 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:08.385293+0000 mon.a (mon.0) 3472 : audit [DBG] from='client.? v1:172.21.15.163:0/2924879613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:08.692 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:08.709 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:16:08.815 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:08.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:07.315706+0000 mgr.y (mgr.14550) 603 : cluster [DBG] pgmap v1082: 214 pgs: 214 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:08.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:07.348204+0000 mgr.y (mgr.14550) 604 : audit [DBG] from='client.50486 v1:172.21.15.163:0/510232126' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:16:08.815 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:07.421535+0000 mon.a (mon.0) 3469 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:16:08.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:07.434781+0000 mon.a (mon.0) 3470 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-126"}]': finished 2024-02-19T21:16:08.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:07.434841+0000 mon.a (mon.0) 3471 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2024-02-19T21:16:08.816 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:08.385293+0000 mon.a (mon.0) 3472 : audit [DBG] from='client.? v1:172.21.15.163:0/2924879613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:08.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:08.918 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:08.944 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:09.528 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:09.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:09.592 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:09.633 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:09.659 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:09.726 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:08.437750+0000 mon.a (mon.0) 3473 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2024-02-19T21:16:09.726 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:09.109756+0000 mon.a (mon.0) 3474 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:09.726 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:09.393861+0000 mon.a (mon.0) 3475 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2024-02-19T21:16:09.726 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:09.395544+0000 mon.a (mon.0) 3476 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:16:09.727 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:16:09.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:08.437750+0000 mon.a (mon.0) 3473 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2024-02-19T21:16:09.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:09.109756+0000 mon.a (mon.0) 3474 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:09.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:09.393861+0000 mon.a (mon.0) 3475 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2024-02-19T21:16:09.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:09.395544+0000 mon.a (mon.0) 3476 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:16:09.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:08.437750+0000 mon.a (mon.0) 3473 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2024-02-19T21:16:09.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:09.109756+0000 mon.a (mon.0) 3474 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:09.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:09.393861+0000 mon.a (mon.0) 3475 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2024-02-19T21:16:09.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:09.395544+0000 mon.a (mon.0) 3476 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:16:09.858 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:09.861 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:09.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:09.962 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:09.990 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:10.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:09.316276+0000 mgr.y (mgr.14550) 605 : cluster [DBG] pgmap v1085: 182 pgs: 182 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:16:10.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:10.395373+0000 mon.a (mon.0) 3477 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:16:10.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:10.395572+0000 mon.a (mon.0) 3478 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2024-02-19T21:16:10.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:10.397704+0000 mon.a (mon.0) 3479 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:16:10.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:09.316276+0000 mgr.y (mgr.14550) 605 : cluster [DBG] pgmap v1085: 182 pgs: 182 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:16:10.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:10.395373+0000 mon.a (mon.0) 3477 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:16:10.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:10.395572+0000 mon.a (mon.0) 3478 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2024-02-19T21:16:10.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:10.397704+0000 mon.a (mon.0) 3479 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:16:10.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:09.316276+0000 mgr.y (mgr.14550) 605 : cluster [DBG] pgmap v1085: 182 pgs: 182 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:16:10.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:10.395373+0000 mon.a (mon.0) 3477 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:16:10.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:10.395572+0000 mon.a (mon.0) 3478 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2024-02-19T21:16:10.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:10.397704+0000 mon.a (mon.0) 3479 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:16:10.926 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:16:10.926 INFO:teuthology.orchestra.run.smithi193.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2024-02-19T21:16:10.926 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:16:11.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:16:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:16:11.118 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:11.121 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:11.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:11.222 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:11.254 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:11.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.356+0000 7f3e4de5e700 1 -- 172.21.15.163:0/1098848086 >> v1:172.21.15.163:6790/0 conn(0x7f3e481042b0 legacy=0x7f3e48106730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:11.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.356+0000 7f3e4de5e700 1 -- 172.21.15.163:0/1098848086 shutdown_connections 2024-02-19T21:16:11.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.356+0000 7f3e4de5e700 1 -- 172.21.15.163:0/1098848086 >> 172.21.15.163:0/1098848086 conn(0x7f3e480f3590 msgr2=0x7f3e480f59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:11.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.356+0000 7f3e4de5e700 1 -- 172.21.15.163:0/1098848086 shutdown_connections 2024-02-19T21:16:11.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.356+0000 7f3e4de5e700 1 -- 172.21.15.163:0/1098848086 wait complete. 2024-02-19T21:16:11.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.356+0000 7f3e4de5e700 1 librados: starting msgr at 2024-02-19T21:16:11.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.356+0000 7f3e4de5e700 1 librados: starting objecter 2024-02-19T21:16:11.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.356+0000 7f3e4de5e700 1 Processor -- start 2024-02-19T21:16:11.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e4de5e700 1 -- start start 2024-02-19T21:16:11.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e4de5e700 1 librados: setting wanted keys 2024-02-19T21:16:11.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e4de5e700 1 librados: calling monclient init 2024-02-19T21:16:11.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e4de5e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3e48117310 con 0x7f3e480fd970 2024-02-19T21:16:11.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e4de5e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3e481184f0 con 0x7f3e48100770 2024-02-19T21:16:11.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e4de5e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3e481196f0 con 0x7f3e481042b0 2024-02-19T21:16:11.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e477fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f3e480fd970 0x7f3e480f9a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:59524/0 (socket says 172.21.15.163:59524) 2024-02-19T21:16:11.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e477fe700 1 -- 172.21.15.163:0/2668369544 learned_addr learned my addr 172.21.15.163:0/2668369544 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:11.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 10 client.?.objecter ms_handle_connect 0x7f3e480fd970 2024-02-19T21:16:11.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:11.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 10 client.?.objecter ms_handle_connect 0x7f3e48100770 2024-02-19T21:16:11.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:11.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 10 client.?.objecter ms_handle_connect 0x7f3e481042b0 2024-02-19T21:16:11.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:11.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525143007 0 0) 0x7f3e481184f0 con 0x7f3e48100770 2024-02-19T21:16:11.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e340033f0 con 0x7f3e48100770 2024-02-19T21:16:11.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246018457 0 0) 0x7f3e481196f0 con 0x7f3e481042b0 2024-02-19T21:16:11.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e481184f0 con 0x7f3e481042b0 2024-02-19T21:16:11.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532950253 0 0) 0x7f3e48117310 con 0x7f3e480fd970 2024-02-19T21:16:11.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e481196f0 con 0x7f3e480fd970 2024-02-19T21:16:11.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 36012532 0 0) 0x7f3e340033f0 con 0x7f3e48100770 2024-02-19T21:16:11.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e48117310 con 0x7f3e48100770 2024-02-19T21:16:11.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3e300018c0 con 0x7f3e48100770 2024-02-19T21:16:11.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1482430732 0 0) 0x7f3e481184f0 con 0x7f3e481042b0 2024-02-19T21:16:11.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e340033f0 con 0x7f3e481042b0 2024-02-19T21:16:11.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3061157114 0 0) 0x7f3e481196f0 con 0x7f3e480fd970 2024-02-19T21:16:11.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e481184f0 con 0x7f3e480fd970 2024-02-19T21:16:11.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3e40002050 con 0x7f3e481042b0 2024-02-19T21:16:11.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3e38001d50 con 0x7f3e480fd970 2024-02-19T21:16:11.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3924631998 0 0) 0x7f3e48117310 con 0x7f3e48100770 2024-02-19T21:16:11.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 >> v1:172.21.15.163:6790/0 conn(0x7f3e481042b0 legacy=0x7f3e481159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:11.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.357+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 >> v1:172.21.15.163:6789/0 conn(0x7f3e480fd970 legacy=0x7f3e480f9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:11.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.358+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4811a8f0 con 0x7f3e48100770 2024-02-19T21:16:11.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.358+0000 7f3e4de5e700 1 -- 172.21.15.163:0/2668369544 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e481186e0 con 0x7f3e48100770 2024-02-19T21:16:11.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.358+0000 7f3e4de5e700 10 client.49309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:11.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.358+0000 7f3e4de5e700 1 -- 172.21.15.163:0/2668369544 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e48118bd0 con 0x7f3e48100770 2024-02-19T21:16:11.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.358+0000 7f3e4de5e700 1 librados: init done 2024-02-19T21:16:11.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.358+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3e30001f80 con 0x7f3e48100770 2024-02-19T21:16:11.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.358+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3e300032a0 con 0x7f3e48100770 2024-02-19T21:16:11.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.358+0000 7f3e4de5e700 1 -- 172.21.15.163:0/2668369544 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e28004f20 con 0x7f3e48100770 2024-02-19T21:16:11.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.359+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f3e30001ae0 con 0x7f3e48100770 2024-02-19T21:16:11.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.359+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(721..721 src has 623..721) v4 ==== 8683+0+0 (unknown 2616144867 0 0) 0x7f3e300766c0 con 0x7f3e48100770 2024-02-19T21:16:11.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.359+0000 7f3e44ff9700 10 client.49309.objecter ms_dispatch 0x7f3e48001550 osd_map(721..721 src has 623..721) v4 2024-02-19T21:16:11.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.359+0000 7f3e44ff9700 3 client.49309.objecter handle_osd_map got epochs [721,721] > 0 2024-02-19T21:16:11.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.359+0000 7f3e44ff9700 3 client.49309.objecter handle_osd_map decoding full epoch 721 2024-02-19T21:16:11.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.359+0000 7f3e44ff9700 20 client.49309.objecter dump_active .. 0 homeless 2024-02-19T21:16:11.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.360+0000 7f3e44ff9700 10 client.49309.objecter ms_handle_connect 0x7f3e34063dc0 2024-02-19T21:16:11.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.364+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3e30049860 con 0x7f3e48100770 2024-02-19T21:16:11.486 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:10.735401+0000 mgr.y (mgr.14550) 606 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:11.487 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:10.943253+0000 osd.7 (osd.7) 323 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:16:11.495 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:16:11.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:11.545+0000 7f3e4de5e700 1 -- 172.21.15.163:0/2668369544 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f3e280051d0 con 0x7f3e48100770 2024-02-19T21:16:11.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:10.735401+0000 mgr.y (mgr.14550) 606 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:11.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:10.943253+0000 osd.7 (osd.7) 323 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:16:11.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:10.943452+0000 osd.7 (osd.7) 324 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:16:11.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:11.396260+0000 mon.a (mon.0) 3480 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-128", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:16:11.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:11.396332+0000 mon.a (mon.0) 3481 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2024-02-19T21:16:11.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:11.399323+0000 mon.a (mon.0) 3482 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:16:11.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:10.943452+0000 osd.7 (osd.7) 324 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:16:11.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:11.396260+0000 mon.a (mon.0) 3480 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-128", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:16:11.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:11.396332+0000 mon.a (mon.0) 3481 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2024-02-19T21:16:11.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:11.399323+0000 mon.a (mon.0) 3482 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:16:11.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:10.735401+0000 mgr.y (mgr.14550) 606 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:11.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:10.943253+0000 osd.7 (osd.7) 323 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:16:11.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:10.943452+0000 osd.7 (osd.7) 324 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:16:11.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:11.396260+0000 mon.a (mon.0) 3480 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-128", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:16:11.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:11.396332+0000 mon.a (mon.0) 3481 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2024-02-19T21:16:11.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:11.399323+0000 mon.a (mon.0) 3482 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:16:11.761 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:12.084 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:12.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:12.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:12.197 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:12.225 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:12.244 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:12.246 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:12.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:12.347 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:12.379 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:12.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:12.398+0000 7f3e44ff9700 1 -- 172.21.15.163:0/2668369544 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v723) v1 ==== 91+0+0 (unknown 4070148641 0 0) 0x7f3e30047c70 con 0x7f3e48100770 2024-02-19T21:16:12.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:12.400+0000 7f3e4de5e700 10 librados: watch_flush enter 2024-02-19T21:16:12.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:12.400+0000 7f3e4de5e700 10 librados: watch_flush exit 2024-02-19T21:16:12.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:12.400+0000 7f3e4de5e700 20 client.49309.objecter shutdown clearing up homeless session... 2024-02-19T21:16:12.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:12.400+0000 7f3e4de5e700 10 client.49309.objecter successfully canceled tick 2024-02-19T21:16:12.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:12.400+0000 7f3e4de5e700 1 -- 172.21.15.163:0/2668369544 >> v1:172.21.15.163:6800/1552514681 conn(0x7f3e34063dc0 legacy=0x7f3e34066280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:12.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:12.400+0000 7f3e4de5e700 1 -- 172.21.15.163:0/2668369544 >> v1:172.21.15.193:6789/0 conn(0x7f3e48100770 legacy=0x7f3e480fa160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:12.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:12.401+0000 7f3e4de5e700 1 -- 172.21.15.163:0/2668369544 shutdown_connections 2024-02-19T21:16:12.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:12.401+0000 7f3e4de5e700 1 -- 172.21.15.163:0/2668369544 >> 172.21.15.163:0/2668369544 conn(0x7f3e480f3590 msgr2=0x7f3e48107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:12.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:12.401+0000 7f3e4de5e700 1 -- 172.21.15.163:0/2668369544 shutdown_connections 2024-02-19T21:16:12.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:12.401+0000 7f3e4de5e700 1 -- 172.21.15.163:0/2668369544 wait complete. 2024-02-19T21:16:12.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:12.401+0000 7f3e4de5e700 1 librados: shutdown 2024-02-19T21:16:12.403 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:12.403 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:16:12.405 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:12.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:12.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:11.316662+0000 mgr.y (mgr.14550) 607 : cluster [DBG] pgmap v1088: 214 pgs: 32 creating+peering, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-19T21:16:12.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:11.546903+0000 mon.b (mon.1) 747 : audit [INF] from='client.? v1:172.21.15.163:0/2668369544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:16:12.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:11.547321+0000 mon.a (mon.0) 3483 : audit [INF] from='client.49309 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:16:12.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:12.393762+0000 mon.a (mon.0) 3484 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:16:12.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:12.398134+0000 mon.a (mon.0) 3485 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:16:12.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:12.398206+0000 mon.a (mon.0) 3486 : audit [INF] from='client.49309 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:16:12.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:12.398264+0000 mon.a (mon.0) 3487 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2024-02-19T21:16:12.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:12.402000+0000 mon.a (mon.0) 3488 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:16:12.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:11.316662+0000 mgr.y (mgr.14550) 607 : cluster [DBG] pgmap v1088: 214 pgs: 32 creating+peering, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-19T21:16:12.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:11.546903+0000 mon.b (mon.1) 747 : audit [INF] from='client.? v1:172.21.15.163:0/2668369544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:16:12.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:11.547321+0000 mon.a (mon.0) 3483 : audit [INF] from='client.49309 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:16:12.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:12.393762+0000 mon.a (mon.0) 3484 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:16:12.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:12.398134+0000 mon.a (mon.0) 3485 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:16:12.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:12.398206+0000 mon.a (mon.0) 3486 : audit [INF] from='client.49309 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:16:12.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:12.398264+0000 mon.a (mon.0) 3487 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2024-02-19T21:16:12.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:12.402000+0000 mon.a (mon.0) 3488 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:16:12.506 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:12.535 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:12.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:11.316662+0000 mgr.y (mgr.14550) 607 : cluster [DBG] pgmap v1088: 214 pgs: 32 creating+peering, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-02-19T21:16:12.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:11.546903+0000 mon.b (mon.1) 747 : audit [INF] from='client.? v1:172.21.15.163:0/2668369544' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:16:12.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:11.547321+0000 mon.a (mon.0) 3483 : audit [INF] from='client.49309 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:16:12.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:12.393762+0000 mon.a (mon.0) 3484 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:16:12.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:12.398134+0000 mon.a (mon.0) 3485 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:16:12.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:12.398206+0000 mon.a (mon.0) 3486 : audit [INF] from='client.49309 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:16:12.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:12.398264+0000 mon.a (mon.0) 3487 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2024-02-19T21:16:12.631 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:12.402000+0000 mon.a (mon.0) 3488 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:16:12.785 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:12.787 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:12.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:12.888 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:12.916 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:12.921 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:16:12.922 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-02-19T21:16:12.922 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-02-19T21:16:12.922 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-02-19T21:16:12.922 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.922 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.49199.0:1 149.6s0 149:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e216)", 2024-02-19T21:16:12.922 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:06:36.212055+0000", 2024-02-19T21:16:12.922 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 576.70772016499996, 2024-02-19T21:16:12.922 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.008338194, 2024-02-19T21:16:12.922 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:12.922 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:12.922 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:12.922 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.49199", 2024-02-19T21:16:12.922 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2827260261", 2024-02-19T21:16:12.922 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:16:12.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.923 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:12.923 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.923 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:12.923 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212055+0000", 2024-02-19T21:16:12.923 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:12.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.923 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.923 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:12.923 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212055+0000", 2024-02-19T21:16:12.923 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:16:12.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.923 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.923 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:12.924 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212052+0000", 2024-02-19T21:16:12.924 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0199000000000001e-05 2024-02-19T21:16:12.924 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.924 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.924 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:12.924 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212073+0000", 2024-02-19T21:16:12.924 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1531000000000001e-05 2024-02-19T21:16:12.924 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.924 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.924 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:12.924 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212094+0000", 2024-02-19T21:16:12.924 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6829999999999997e-06 2024-02-19T21:16:12.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.925 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.925 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:12.925 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212103+0000", 2024-02-19T21:16:12.925 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00054734599999999999 2024-02-19T21:16:12.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.925 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.925 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.925 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212650+0000", 2024-02-19T21:16:12.925 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.8489999999999999e-06 2024-02-19T21:16:12.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.925 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.925 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:16:12.925 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:36.212654+0000", 2024-02-19T21:16:12.926 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0043875600000001 2024-02-19T21:16:12.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.926 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.926 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.217042+0000", 2024-02-19T21:16:12.926 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012985000000000001 2024-02-19T21:16:12.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.926 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:12.926 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.217171+0000", 2024-02-19T21:16:12.926 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00037497799999999999 2024-02-19T21:16:12.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.926 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.926 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:16:12.927 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.217546+0000", 2024-02-19T21:16:12.927 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0020797239999999998 2024-02-19T21:16:12.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.927 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:16:12.927 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.219626+0000", 2024-02-19T21:16:12.927 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00069722600000000001 2024-02-19T21:16:12.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.927 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:12.927 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.220323+0000", 2024-02-19T21:16:12.927 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9684000000000006e-05 2024-02-19T21:16:12.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.927 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.928 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:12.928 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:37.220393+0000", 2024-02-19T21:16:12.928 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.008338194 2024-02-19T21:16:12.928 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.928 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:12.928 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.928 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.928 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.928 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22689.0:1 172.6s0 172:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e260)", 2024-02-19T21:16:12.928 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:20.216387+0000", 2024-02-19T21:16:12.928 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 532.70338823600002, 2024-02-19T21:16:12.928 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0565913250000001, 2024-02-19T21:16:12.928 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:12.929 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:12.929 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:12.929 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22689", 2024-02-19T21:16:12.929 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/585802156", 2024-02-19T21:16:12.929 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:16:12.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.929 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:12.929 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.929 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:12.929 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216387+0000", 2024-02-19T21:16:12.929 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:12.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.929 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.930 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:12.930 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216387+0000", 2024-02-19T21:16:12.930 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:16:12.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.930 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.930 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:12.930 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216384+0000", 2024-02-19T21:16:12.930 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2157000000000001e-05 2024-02-19T21:16:12.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.930 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.930 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:12.930 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216396+0000", 2024-02-19T21:16:12.930 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.003e-05 2024-02-19T21:16:12.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:12.931 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216416+0000", 2024-02-19T21:16:12.931 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.7499999999999998e-06 2024-02-19T21:16:12.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:12.931 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.216426+0000", 2024-02-19T21:16:12.931 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00066776200000000002 2024-02-19T21:16:12.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.931 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.931 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.931 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.217093+0000", 2024-02-19T21:16:12.931 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6430000000000003e-06 2024-02-19T21:16:12.932 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.932 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.932 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:16:12.932 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:20.217098+0000", 2024-02-19T21:16:12.932 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.051702409 2024-02-19T21:16:12.932 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.932 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.932 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.932 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.268800+0000", 2024-02-19T21:16:12.932 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018998999999999999 2024-02-19T21:16:12.932 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.932 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.932 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:12.933 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.268990+0000", 2024-02-19T21:16:12.933 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00037534100000000001 2024-02-19T21:16:12.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.933 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.933 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:16:12.933 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.269366+0000", 2024-02-19T21:16:12.933 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001076088 2024-02-19T21:16:12.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.933 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.933 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:16:12.933 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.270442+0000", 2024-02-19T21:16:12.933 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024701089999999998 2024-02-19T21:16:12.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.933 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.934 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:12.934 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.272912+0000", 2024-02-19T21:16:12.934 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6147000000000002e-05 2024-02-19T21:16:12.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.934 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.934 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:12.934 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:21.272978+0000", 2024-02-19T21:16:12.934 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0565913250000001 2024-02-19T21:16:12.934 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.934 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:12.934 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.934 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.935 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.1.0:634 61.b e278/114 61:d461b0a5:test-rados-api-smithi163-83928-34::foo:14 v 278'95, mlcod=275'91)", 2024-02-19T21:16:12.935 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:38.232659+0000", 2024-02-19T21:16:12.935 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 514.68711547999999, 2024-02-19T21:16:12.935 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0151769, 2024-02-19T21:16:12.935 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:12.935 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:12.935 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:12.935 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.935 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:12.935 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.232659+0000", 2024-02-19T21:16:12.935 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:12.935 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.935 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.935 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:12.936 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.232659+0000", 2024-02-19T21:16:12.936 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:16:12.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.936 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.936 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:12.936 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.232659+0000", 2024-02-19T21:16:12.936 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3270000000000001e-06 2024-02-19T21:16:12.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.936 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.936 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:12.936 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.232667+0000", 2024-02-19T21:16:12.936 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00080230000000000004 2024-02-19T21:16:12.936 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.937 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.937 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:12.937 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.233470+0000", 2024-02-19T21:16:12.937 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7861999999999999e-05 2024-02-19T21:16:12.937 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.937 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.937 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:12.937 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.233488+0000", 2024-02-19T21:16:12.937 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3637000000000003e-05 2024-02-19T21:16:12.937 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.937 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.937 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.937 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.233531+0000", 2024-02-19T21:16:12.938 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1175999999999999e-05 2024-02-19T21:16:12.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.938 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:12.938 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.233552+0000", 2024-02-19T21:16:12.938 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.014260111000000001 2024-02-19T21:16:12.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.938 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:12.938 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.247812+0000", 2024-02-19T21:16:12.938 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3896999999999998e-05 2024-02-19T21:16:12.938 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.938 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.939 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:12.939 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:38.247836+0000", 2024-02-19T21:16:12.939 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0151769 2024-02-19T21:16:12.939 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.939 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:12.939 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.939 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.939 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.939 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.49649.0:1 201.2s0 201:5a7a12bd:::oid_SimpleStatPPNS_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e352)", 2024-02-19T21:16:12.939 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:08:52.333687+0000", 2024-02-19T21:16:12.939 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 440.58608763199999, 2024-02-19T21:16:12.939 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0054582620000001, 2024-02-19T21:16:12.939 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:12.939 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:12.940 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:12.940 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.49649", 2024-02-19T21:16:12.940 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2569144910", 2024-02-19T21:16:12.940 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:16:12.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.940 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:12.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.940 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:12.940 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333687+0000", 2024-02-19T21:16:12.940 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:12.940 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.940 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.940 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:12.941 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333687+0000", 2024-02-19T21:16:12.941 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:16:12.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.941 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:12.941 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333685+0000", 2024-02-19T21:16:12.941 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0747000000000001e-05 2024-02-19T21:16:12.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.941 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:12.941 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333695+0000", 2024-02-19T21:16:12.941 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3277e-05 2024-02-19T21:16:12.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.941 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.942 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:12.942 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333709+0000", 2024-02-19T21:16:12.942 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0219999999999993e-06 2024-02-19T21:16:12.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.942 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.942 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:12.942 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333718+0000", 2024-02-19T21:16:12.942 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8285999999999998e-05 2024-02-19T21:16:12.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.942 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.942 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.942 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333766+0000", 2024-02-19T21:16:12.942 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9040000000000002e-06 2024-02-19T21:16:12.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.942 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.943 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:16:12.943 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:52.333770+0000", 2024-02-19T21:16:12.943 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.003658465 2024-02-19T21:16:12.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.943 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.943 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.943 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:53.337428+0000", 2024-02-19T21:16:12.943 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00016740599999999999 2024-02-19T21:16:12.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.943 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.943 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:12.943 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:53.337596+0000", 2024-02-19T21:16:12.943 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00055418199999999996 2024-02-19T21:16:12.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.944 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:16:12.944 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:53.338150+0000", 2024-02-19T21:16:12.944 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000445174 2024-02-19T21:16:12.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.944 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:16:12.944 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:53.338595+0000", 2024-02-19T21:16:12.944 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00050577499999999997 2024-02-19T21:16:12.944 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.944 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.944 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:12.944 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:53.339101+0000", 2024-02-19T21:16:12.945 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.4749000000000001e-05 2024-02-19T21:16:12.945 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.945 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.945 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:12.945 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:53.339146+0000", 2024-02-19T21:16:12.945 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0054582620000001 2024-02-19T21:16:12.945 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.945 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:12.945 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.945 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.945 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.945 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:1424 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94681540386816 gen 2] snapc 0=[] ondisk+write+known_if_redirected e433)", 2024-02-19T21:16:12.945 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:10:40.311743+0000", 2024-02-19T21:16:12.946 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 332.60803231400001, 2024-02-19T21:16:12.946 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.012662921, 2024-02-19T21:16:12.946 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:12.946 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:12.946 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:12.946 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:16:12.946 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:16:12.946 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1424 2024-02-19T21:16:12.946 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.946 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:12.946 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.946 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:12.946 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311743+0000", 2024-02-19T21:16:12.946 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:12.946 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.947 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.947 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:12.947 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311743+0000", 2024-02-19T21:16:12.947 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:16:12.947 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.947 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.947 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:12.947 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311740+0000", 2024-02-19T21:16:12.947 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0318000000000001e-05 2024-02-19T21:16:12.947 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.947 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.947 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:12.947 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311760+0000", 2024-02-19T21:16:12.948 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3355e-05 2024-02-19T21:16:12.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.948 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:12.948 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311773+0000", 2024-02-19T21:16:12.948 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.3659999999999992e-06 2024-02-19T21:16:12.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.948 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:12.948 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311782+0000", 2024-02-19T21:16:12.948 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4440000000000001e-05 2024-02-19T21:16:12.948 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.948 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.948 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.948 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311836+0000", 2024-02-19T21:16:12.949 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1285e-05 2024-02-19T21:16:12.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.949 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.949 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:16:12.949 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311848+0000", 2024-02-19T21:16:12.949 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.99090941099999996 2024-02-19T21:16:12.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.949 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.949 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.949 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.302757+0000", 2024-02-19T21:16:12.949 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00026452600000000001 2024-02-19T21:16:12.949 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.949 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.949 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for degraded object", 2024-02-19T21:16:12.949 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.303021+0000", 2024-02-19T21:16:12.950 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.020182997000000001 2024-02-19T21:16:12.950 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.950 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.950 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.950 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.323204+0000", 2024-02-19T21:16:12.950 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4702999999999994e-05 2024-02-19T21:16:12.950 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.950 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.950 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:12.950 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.323269+0000", 2024-02-19T21:16:12.950 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001136399 2024-02-19T21:16:12.950 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.950 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.324406+0000", 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.012662921 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.25753.0:6559 61.2 e452/303 61:4782fe63:test-rados-api-smithi163-83928-70::foo:head v 452'97, mlcod=447'95)", 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:11:04.369908+0000", 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 308.54986715199999, 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.01199457, 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:12.951 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.952 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:12.952 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.369908+0000", 2024-02-19T21:16:12.952 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:12.952 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.952 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.952 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:12.952 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.369908+0000", 2024-02-19T21:16:12.952 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:16:12.952 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.952 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.952 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:12.952 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.369905+0000", 2024-02-19T21:16:12.952 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3260000000000001e-05 2024-02-19T21:16:12.952 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.952 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.953 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:12.953 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.369928+0000", 2024-02-19T21:16:12.953 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9531e-05 2024-02-19T21:16:12.953 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.953 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.953 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:12.953 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.369947+0000", 2024-02-19T21:16:12.953 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.2879999999999998e-06 2024-02-19T21:16:12.953 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.953 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.953 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:12.953 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.369957+0000", 2024-02-19T21:16:12.953 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.4946999999999998e-05 2024-02-19T21:16:12.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.954 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.954 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.954 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.370042+0000", 2024-02-19T21:16:12.954 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.7177000000000001e-05 2024-02-19T21:16:12.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.954 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.954 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:12.954 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.370069+0000", 2024-02-19T21:16:12.954 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.011802573 2024-02-19T21:16:12.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.954 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.954 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:12.954 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.381871+0000", 2024-02-19T21:16:12.955 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0961999999999999e-05 2024-02-19T21:16:12.955 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.955 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.955 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:12.955 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:04.381902+0000", 2024-02-19T21:16:12.955 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.01199457 2024-02-19T21:16:12.955 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.955 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:12.955 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.955 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.955 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.955 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:312 237.a 237:50000000::::head [pgnls start_epoch 629] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e629)", 2024-02-19T21:16:12.955 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:14:33.536659+0000", 2024-02-19T21:16:12.955 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 99.383116002999998, 2024-02-19T21:16:12.955 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0026515390000000001, 2024-02-19T21:16:12.956 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:12.956 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:12.956 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:12.956 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:16:12.956 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:16:12.956 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 312 2024-02-19T21:16:12.956 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.956 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:12.956 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.956 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:12.956 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.536659+0000", 2024-02-19T21:16:12.956 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:12.956 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.957 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.957 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:12.957 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.536659+0000", 2024-02-19T21:16:12.957 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:16:12.957 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.957 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.959 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:12.959 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.536656+0000", 2024-02-19T21:16:12.959 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3366e-05 2024-02-19T21:16:12.959 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.959 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.959 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:12.959 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.536670+0000", 2024-02-19T21:16:12.959 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7963999999999999e-05 2024-02-19T21:16:12.959 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.959 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.960 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:12.960 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.536687+0000", 2024-02-19T21:16:12.960 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3957000000000001e-05 2024-02-19T21:16:12.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.960 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.960 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:12.960 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.536711+0000", 2024-02-19T21:16:12.960 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9036999999999995e-05 2024-02-19T21:16:12.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.960 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.960 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.960 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.536800+0000", 2024-02-19T21:16:12.960 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4010000000000002e-06 2024-02-19T21:16:12.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.961 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.961 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:16:12.961 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.536807+0000", 2024-02-19T21:16:12.961 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002372318 2024-02-19T21:16:12.961 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.961 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.961 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.961 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.539179+0000", 2024-02-19T21:16:12.961 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4529000000000001e-05 2024-02-19T21:16:12.961 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.961 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.961 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:12.961 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.539214+0000", 2024-02-19T21:16:12.961 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6782999999999998e-05 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.539311+0000", 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0026515390000000001 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:730 240.14 240:28000000::::head [pgnls start_epoch 655] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e655)", 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:14:59.391131+0000", 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 73.528643695, 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003131858, 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:12.962 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:12.963 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:12.963 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:16:12.963 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:16:12.963 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 730 2024-02-19T21:16:12.963 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.963 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:12.963 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.963 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:12.963 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.391131+0000", 2024-02-19T21:16:12.963 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:12.963 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.963 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.963 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:12.963 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.391131+0000", 2024-02-19T21:16:12.964 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:16:12.964 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.964 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.964 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:12.964 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.391127+0000", 2024-02-19T21:16:12.964 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7195999999999999e-05 2024-02-19T21:16:12.964 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.964 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.964 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:12.964 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.391145+0000", 2024-02-19T21:16:12.964 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.249e-05 2024-02-19T21:16:12.964 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.964 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.964 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:12.965 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.391167+0000", 2024-02-19T21:16:12.965 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.058e-05 2024-02-19T21:16:12.965 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.965 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.965 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:12.965 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.391178+0000", 2024-02-19T21:16:12.965 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8851999999999997e-05 2024-02-19T21:16:12.965 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.965 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.965 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.965 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.391246+0000", 2024-02-19T21:16:12.965 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.2259999999999999e-06 2024-02-19T21:16:12.965 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.965 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.966 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:16:12.966 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.391254+0000", 2024-02-19T21:16:12.966 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0028906499999999998 2024-02-19T21:16:12.966 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.966 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.966 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.966 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.394144+0000", 2024-02-19T21:16:12.966 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6842000000000002e-05 2024-02-19T21:16:12.966 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.966 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.966 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:12.966 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.394171+0000", 2024-02-19T21:16:12.966 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.1918999999999999e-05 2024-02-19T21:16:12.966 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.967 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.967 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:12.967 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.394263+0000", 2024-02-19T21:16:12.967 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003131858 2024-02-19T21:16:12.967 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.967 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:12.967 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.967 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.967 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.967 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:797 234.6s0 234:6cb97f36:test-rados-api-smithi163-83928-115::bar:head [writefull 0~5] snapc 6=[6] ondisk+write+known_if_redirected e659)", 2024-02-19T21:16:12.967 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:03.387928+0000", 2024-02-19T21:16:12.967 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 69.531847372000001, 2024-02-19T21:16:12.967 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0051323260000000004, 2024-02-19T21:16:12.967 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:12.967 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:12.968 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:12.968 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:16:12.968 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:16:12.968 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 797 2024-02-19T21:16:12.968 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.968 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:12.968 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.968 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:12.968 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387928+0000", 2024-02-19T21:16:12.968 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:12.968 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.968 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.968 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:12.968 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387928+0000", 2024-02-19T21:16:12.969 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999952 2024-02-19T21:16:12.969 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.969 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.969 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:12.969 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387923+0000", 2024-02-19T21:16:12.969 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5981000000000002e-05 2024-02-19T21:16:12.969 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.969 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.969 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:12.969 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387939+0000", 2024-02-19T21:16:12.969 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1821e-05 2024-02-19T21:16:12.969 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.969 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.969 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387961+0000", 2024-02-19T21:16:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6879999999999994e-06 2024-02-19T21:16:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.387970+0000", 2024-02-19T21:16:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.5976000000000007e-05 2024-02-19T21:16:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.388036+0000", 2024-02-19T21:16:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1620000000000008e-06 2024-02-19T21:16:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.970 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:16:12.971 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.388045+0000", 2024-02-19T21:16:12.971 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0034814899999999998 2024-02-19T21:16:12.971 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.971 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.971 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.971 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.391526+0000", 2024-02-19T21:16:12.971 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000102123 2024-02-19T21:16:12.971 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.971 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.971 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:12.971 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.391628+0000", 2024-02-19T21:16:12.971 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00047153700000000001 2024-02-19T21:16:12.971 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.971 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.972 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:16:12.972 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.392100+0000", 2024-02-19T21:16:12.972 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00052448800000000004 2024-02-19T21:16:12.972 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.972 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.972 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:16:12.972 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.392624+0000", 2024-02-19T21:16:12.972 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00036736499999999998 2024-02-19T21:16:12.972 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.972 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.972 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:12.972 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.392992+0000", 2024-02-19T21:16:12.972 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8313999999999997e-05 2024-02-19T21:16:12.972 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.972 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.973 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:12.973 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:03.393060+0000", 2024-02-19T21:16:12.973 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0051323260000000004 2024-02-19T21:16:12.973 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.973 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:12.973 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.973 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.973 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.973 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:1040 242.14 242:28000000::::head [pgnls start_epoch 673] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e673)", 2024-02-19T21:16:12.973 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:17.438294+0000", 2024-02-19T21:16:12.973 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 55.481481047999999, 2024-02-19T21:16:12.973 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00063692100000000004, 2024-02-19T21:16:12.973 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:12.973 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:12.974 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:12.974 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:16:12.974 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:16:12.974 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1040 2024-02-19T21:16:12.974 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.974 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:12.974 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.974 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:12.974 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.438294+0000", 2024-02-19T21:16:12.974 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:12.974 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.974 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.974 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:12.974 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.438294+0000", 2024-02-19T21:16:12.975 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:16:12.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.975 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.975 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:12.975 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.438290+0000", 2024-02-19T21:16:12.975 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4205999999999999e-05 2024-02-19T21:16:12.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.975 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.975 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:12.975 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.438304+0000", 2024-02-19T21:16:12.975 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0682999999999998e-05 2024-02-19T21:16:12.975 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.975 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.975 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:12.976 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.438325+0000", 2024-02-19T21:16:12.976 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.6309999999999992e-06 2024-02-19T21:16:12.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.976 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:12.976 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.438334+0000", 2024-02-19T21:16:12.976 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00045381000000000002 2024-02-19T21:16:12.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.976 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.976 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.438787+0000", 2024-02-19T21:16:12.976 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3881999999999998e-05 2024-02-19T21:16:12.976 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.976 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.977 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:12.977 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.438821+0000", 2024-02-19T21:16:12.977 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010961800000000001 2024-02-19T21:16:12.977 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.977 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.977 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:12.977 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.438931+0000", 2024-02-19T21:16:12.977 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00063692100000000004 2024-02-19T21:16:12.977 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.977 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:12.977 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.977 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.977 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.977 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:2065 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94681540386816 gen 4] snapc 0=[] ondisk+write+known_if_redirected e670)", 2024-02-19T21:16:12.978 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:19.751640+0000", 2024-02-19T21:16:12.978 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 53.168135309999997, 2024-02-19T21:16:12.978 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00061193300000000005, 2024-02-19T21:16:12.978 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:12.978 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:12.978 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:12.978 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:16:12.978 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:16:12.978 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2065 2024-02-19T21:16:12.978 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.978 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:12.978 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.978 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:12.978 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751640+0000", 2024-02-19T21:16:12.979 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:12.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.979 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.979 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:12.979 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751640+0000", 2024-02-19T21:16:12.979 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:16:12.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.979 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.979 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:12.979 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751637+0000", 2024-02-19T21:16:12.979 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.9240000000000002e-06 2024-02-19T21:16:12.979 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.979 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.980 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:12.980 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751647+0000", 2024-02-19T21:16:12.980 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5420000000000001e-05 2024-02-19T21:16:12.980 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.980 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.980 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:12.980 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751662+0000", 2024-02-19T21:16:12.980 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.5249999999999994e-06 2024-02-19T21:16:12.980 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.980 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.980 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:12.980 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751672+0000", 2024-02-19T21:16:12.980 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00038507999999999999 2024-02-19T21:16:12.981 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.981 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.981 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.981 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.752057+0000", 2024-02-19T21:16:12.981 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4634000000000002e-05 2024-02-19T21:16:12.981 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.981 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.981 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:12.981 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.752151+0000", 2024-02-19T21:16:12.981 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010017399999999999 2024-02-19T21:16:12.981 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.981 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.981 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:12.981 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.752252+0000", 2024-02-19T21:16:12.982 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00061193300000000005 2024-02-19T21:16:12.982 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.982 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:12.982 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.982 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.982 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.982 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:2066 5.3 5:c609908c:::notify.5:head [watch ping cookie 94681540392576 gen 5] snapc 0=[] ondisk+write+known_if_redirected e670)", 2024-02-19T21:16:12.982 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:19.751690+0000", 2024-02-19T21:16:12.982 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 53.168085206000001, 2024-02-19T21:16:12.982 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00057178199999999995, 2024-02-19T21:16:12.982 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:12.982 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:12.982 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:12.983 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:16:12.983 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:16:12.983 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2066 2024-02-19T21:16:12.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.983 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:12.983 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.983 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:12.983 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751690+0000", 2024-02-19T21:16:12.983 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:12.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.983 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.983 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:12.983 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751690+0000", 2024-02-19T21:16:12.983 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:16:12.983 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.984 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.984 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:12.984 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751688+0000", 2024-02-19T21:16:12.984 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7319999999999998e-06 2024-02-19T21:16:12.984 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.984 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.984 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:12.984 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751693+0000", 2024-02-19T21:16:12.984 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6830000000000008e-06 2024-02-19T21:16:12.984 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.984 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.984 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:12.984 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751701+0000", 2024-02-19T21:16:12.984 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.8300000000000003e-06 2024-02-19T21:16:12.985 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.985 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.985 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:12.985 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.751706+0000", 2024-02-19T21:16:12.985 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00035145 2024-02-19T21:16:12.985 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.985 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.985 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.985 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.752057+0000", 2024-02-19T21:16:12.985 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6335000000000005e-05 2024-02-19T21:16:12.985 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.985 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.985 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:12.985 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.752153+0000", 2024-02-19T21:16:12.985 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00010809199999999999 2024-02-19T21:16:12.986 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.986 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.986 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:12.986 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:19.752262+0000", 2024-02-19T21:16:12.986 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00057178199999999995 2024-02-19T21:16:12.986 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.986 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:12.986 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.986 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.986 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.986 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:1102 243.0 243:00000000:test-rados-api-smithi163-83928-119:::head [pgnls start_epoch 679] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e679)", 2024-02-19T21:16:12.986 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:23.486625+0000", 2024-02-19T21:16:12.986 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 49.433149514999997, 2024-02-19T21:16:12.986 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003583096, 2024-02-19T21:16:12.986 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:12.987 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:12.987 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:12.987 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:16:12.987 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:16:12.987 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1102 2024-02-19T21:16:12.987 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.987 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:12.987 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.987 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:12.987 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.486625+0000", 2024-02-19T21:16:12.987 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:12.987 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.987 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.987 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:12.988 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.486625+0000", 2024-02-19T21:16:12.988 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:16:12.988 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.988 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.988 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:12.988 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.486622+0000", 2024-02-19T21:16:12.988 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.3127e-05 2024-02-19T21:16:12.988 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.988 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.988 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:12.988 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.486636+0000", 2024-02-19T21:16:12.988 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9723000000000001e-05 2024-02-19T21:16:12.988 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.988 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.989 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:12.989 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.486655+0000", 2024-02-19T21:16:12.989 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.9989999999999997e-06 2024-02-19T21:16:12.989 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.989 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.989 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:12.989 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.486664+0000", 2024-02-19T21:16:12.989 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7355999999999998e-05 2024-02-19T21:16:12.989 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.989 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.989 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.989 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.486722+0000", 2024-02-19T21:16:12.989 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3629999999999996e-06 2024-02-19T21:16:12.989 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.990 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.990 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:16:12.990 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.486729+0000", 2024-02-19T21:16:12.990 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0031511349999999998 2024-02-19T21:16:12.990 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.990 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.990 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.990 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.489880+0000", 2024-02-19T21:16:12.990 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6288999999999998e-05 2024-02-19T21:16:12.990 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.990 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.990 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:12.990 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.489916+0000", 2024-02-19T21:16:12.990 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00029217400000000003 2024-02-19T21:16:12.990 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.991 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.991 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:12.991 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:23.490209+0000", 2024-02-19T21:16:12.991 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003583096 2024-02-19T21:16:12.991 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.991 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:12.991 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.991 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.991 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.991 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.7.0:681 244.17 e687/683 244:e873ac47:test-rados-api-smithi163-83928-121::foo:head v 687'1, mlcod=0'0)", 2024-02-19T21:16:12.991 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:36.534110+0000", 2024-02-19T21:16:12.991 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 36.385664730000002, 2024-02-19T21:16:12.991 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00063184299999999999, 2024-02-19T21:16:12.991 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:12.992 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:12.992 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:12.992 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.992 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:12.992 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534110+0000", 2024-02-19T21:16:12.992 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:12.992 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.992 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.992 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:12.992 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534110+0000", 2024-02-19T21:16:12.992 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:16:12.992 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.992 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.993 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:12.993 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534109+0000", 2024-02-19T21:16:12.993 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.8519999999999993e-06 2024-02-19T21:16:12.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.993 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.993 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:12.993 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534118+0000", 2024-02-19T21:16:12.993 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9463999999999999e-05 2024-02-19T21:16:12.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.993 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.993 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:12.993 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534138+0000", 2024-02-19T21:16:12.993 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2129e-05 2024-02-19T21:16:12.993 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.993 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.994 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:12.994 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534150+0000", 2024-02-19T21:16:12.994 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000108325 2024-02-19T21:16:12.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.994 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.994 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.994 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534258+0000", 2024-02-19T21:16:12.994 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1886999999999999e-05 2024-02-19T21:16:12.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.994 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.994 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:12.994 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534280+0000", 2024-02-19T21:16:12.994 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00042039600000000002 2024-02-19T21:16:12.994 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.995 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.995 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:12.995 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534700+0000", 2024-02-19T21:16:12.995 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1582999999999998e-05 2024-02-19T21:16:12.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.995 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.995 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:12.995 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534742+0000", 2024-02-19T21:16:12.995 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00063184299999999999 2024-02-19T21:16:12.995 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.995 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:12.995 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:12.995 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.995 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.996 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(client.23517.0:1294 244.17 e687/683 244:e873ac47:test-rados-api-smithi163-83928-121::foo:head v 687'2, mlcod=0'0)", 2024-02-19T21:16:12.996 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:36.534893+0000", 2024-02-19T21:16:12.996 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 36.384882414000003, 2024-02-19T21:16:12.996 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014407090000000001, 2024-02-19T21:16:12.996 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:12.996 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:12.996 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:12.996 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.996 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:12.996 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534893+0000", 2024-02-19T21:16:12.996 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:12.996 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.996 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.996 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:12.997 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534893+0000", 2024-02-19T21:16:12.997 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:16:12.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.997 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.997 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:12.997 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534891+0000", 2024-02-19T21:16:12.997 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1739999999999999e-05 2024-02-19T21:16:12.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.997 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.997 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:12.997 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534902+0000", 2024-02-19T21:16:12.997 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4814000000000001e-05 2024-02-19T21:16:12.997 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.997 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.998 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:12.998 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534917+0000", 2024-02-19T21:16:12.998 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1829999999999997e-06 2024-02-19T21:16:12.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.998 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.998 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:12.998 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534924+0000", 2024-02-19T21:16:12.998 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.9036999999999998e-05 2024-02-19T21:16:12.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.998 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.998 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:12.998 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534973+0000", 2024-02-19T21:16:12.998 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0176000000000001e-05 2024-02-19T21:16:12.998 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.999 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.999 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:12.999 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534994+0000", 2024-02-19T21:16:12.999 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0012985850000000001 2024-02-19T21:16:12.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.999 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.999 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:12.999 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.536292+0000", 2024-02-19T21:16:12.999 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1069000000000003e-05 2024-02-19T21:16:12.999 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:12.999 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:12.999 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:12.999 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.536333+0000", 2024-02-19T21:16:12.999 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014407090000000001 2024-02-19T21:16:13.000 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:13.000 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:13.000 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:13.000 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.000 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "MOSDScrubReserve(16.1a REQUEST e688)", 2024-02-19T21:16:13.000 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:38.117125+0000", 2024-02-19T21:16:13.000 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 34.802649596999998, 2024-02-19T21:16:13.000 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025661490000000002, 2024-02-19T21:16:13.000 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:13.000 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:13.000 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:13.000 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.000 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:13.001 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.117125+0000", 2024-02-19T21:16:13.001 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:13.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.001 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.001 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:13.001 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.117125+0000", 2024-02-19T21:16:13.001 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:16:13.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.001 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.001 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:13.001 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.117125+0000", 2024-02-19T21:16:13.001 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.799e-06 2024-02-19T21:16:13.001 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.001 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.002 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:13.002 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.117129+0000", 2024-02-19T21:16:13.002 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5985e-05 2024-02-19T21:16:13.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.002 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.002 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:13.002 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.117145+0000", 2024-02-19T21:16:13.002 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6399999999999997e-06 2024-02-19T21:16:13.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.002 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.002 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:13.002 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.117152+0000", 2024-02-19T21:16:13.002 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.6922999999999996e-05 2024-02-19T21:16:13.002 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.003 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.003 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:13.003 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.117219+0000", 2024-02-19T21:16:13.003 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3629999999999999e-06 2024-02-19T21:16:13.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.003 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.003 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:16:13.003 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.117226+0000", 2024-02-19T21:16:13.003 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024092969999999999 2024-02-19T21:16:13.003 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.003 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.003 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:13.003 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.119635+0000", 2024-02-19T21:16:13.003 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5778e-05 2024-02-19T21:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.119651+0000", 2024-02-19T21:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0698999999999998e-05 2024-02-19T21:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.119692+0000", 2024-02-19T21:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025661490000000002 2024-02-19T21:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:13.004 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.005 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.005 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:2101 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94681540386816 gen 4] snapc 0=[] ondisk+write+known_if_redirected e685)", 2024-02-19T21:16:13.005 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:39.755649+0000", 2024-02-19T21:16:13.005 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 33.164126052, 2024-02-19T21:16:13.005 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00083203300000000003, 2024-02-19T21:16:13.005 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:13.005 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:13.005 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:13.005 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:16:13.005 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:16:13.005 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2101 2024-02-19T21:16:13.005 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.005 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:13.005 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.006 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:13.006 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755649+0000", 2024-02-19T21:16:13.006 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:13.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.006 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:13.006 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755649+0000", 2024-02-19T21:16:13.006 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999952 2024-02-19T21:16:13.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.006 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.006 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:13.006 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755644+0000", 2024-02-19T21:16:13.006 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7504e-05 2024-02-19T21:16:13.006 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.007 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.007 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:13.007 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755662+0000", 2024-02-19T21:16:13.007 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0493000000000001e-05 2024-02-19T21:16:13.007 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.007 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.007 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:13.007 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755692+0000", 2024-02-19T21:16:13.007 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.9989000000000001e-05 2024-02-19T21:16:13.007 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.007 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.007 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:13.007 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755712+0000", 2024-02-19T21:16:13.007 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00049002200000000001 2024-02-19T21:16:13.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.008 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.008 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:13.008 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.756202+0000", 2024-02-19T21:16:13.008 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00016675699999999999 2024-02-19T21:16:13.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.008 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.008 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:13.008 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.756369+0000", 2024-02-19T21:16:13.008 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000112112 2024-02-19T21:16:13.008 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.008 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.008 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:13.009 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.756481+0000", 2024-02-19T21:16:13.009 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00083203300000000003 2024-02-19T21:16:13.009 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:13.009 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:13.009 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:13.009 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.009 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.009 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:2102 5.3 5:c609908c:::notify.5:head [watch ping cookie 94681540392576 gen 5] snapc 0=[] ondisk+write+known_if_redirected e685)", 2024-02-19T21:16:13.009 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:39.755748+0000", 2024-02-19T21:16:13.009 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 33.164026823, 2024-02-19T21:16:13.009 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00067528399999999995, 2024-02-19T21:16:13.009 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:13.009 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:13.009 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:13.009 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:16:13.010 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:16:13.010 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 2102 2024-02-19T21:16:13.010 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.010 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:13.010 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.010 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:13.010 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755748+0000", 2024-02-19T21:16:13.010 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:13.010 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.010 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.010 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:13.010 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755748+0000", 2024-02-19T21:16:13.010 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:16:13.010 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.010 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.011 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:13.011 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755746+0000", 2024-02-19T21:16:13.011 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4099999999999997e-06 2024-02-19T21:16:13.011 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.011 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.011 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:13.011 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755755+0000", 2024-02-19T21:16:13.011 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0835999999999998e-05 2024-02-19T21:16:13.011 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.011 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.011 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:13.011 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755776+0000", 2024-02-19T21:16:13.011 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2886000000000001e-05 2024-02-19T21:16:13.011 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.012 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.012 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:13.012 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.755799+0000", 2024-02-19T21:16:13.012 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00041311100000000002 2024-02-19T21:16:13.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.012 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.012 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:13.012 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.756212+0000", 2024-02-19T21:16:13.012 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000119809 2024-02-19T21:16:13.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.012 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.012 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:13.012 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.756332+0000", 2024-02-19T21:16:13.012 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.1688999999999996e-05 2024-02-19T21:16:13.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.013 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.013 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:13.013 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:39.756423+0000", 2024-02-19T21:16:13.013 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00067528399999999995 2024-02-19T21:16:13.013 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:13.013 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:13.013 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:13.013 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.013 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.013 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "rep_scrubmap(5.3 e636 from shard 5)", 2024-02-19T21:16:13.013 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:43.161946+0000", 2024-02-19T21:16:13.013 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 29.757828832000001, 2024-02-19T21:16:13.013 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00062701599999999999, 2024-02-19T21:16:13.013 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:13.014 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:13.014 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:13.014 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.014 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:13.014 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:43.161946+0000", 2024-02-19T21:16:13.014 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:13.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.014 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.014 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:13.014 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:43.161946+0000", 2024-02-19T21:16:13.014 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:16:13.014 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.014 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.014 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:43.161945+0000", 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2430000000000005e-06 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:43.161953+0000", 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0128e-05 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:43.161963+0000", 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0659000000000001e-05 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:13.015 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:43.161974+0000", 2024-02-19T21:16:13.016 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00053952999999999998 2024-02-19T21:16:13.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.016 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.016 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:13.016 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:43.162513+0000", 2024-02-19T21:16:13.016 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7003999999999998e-05 2024-02-19T21:16:13.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.016 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.016 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:13.016 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:43.162530+0000", 2024-02-19T21:16:13.016 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3006000000000001e-05 2024-02-19T21:16:13.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.016 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.016 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:13.017 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:43.162573+0000", 2024-02-19T21:16:13.017 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00062701599999999999 2024-02-19T21:16:13.017 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:13.017 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:13.017 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:13.017 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.017 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.017 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:1709 246.1e 246:78000000:%01:::head [pgnls start_epoch 708] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e708)", 2024-02-19T21:16:13.017 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:57.442444+0000", 2024-02-19T21:16:13.017 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 15.477330866000001, 2024-02-19T21:16:13.017 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002908987, 2024-02-19T21:16:13.017 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:13.017 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:13.017 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:13.018 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:16:13.018 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:16:13.018 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1709 2024-02-19T21:16:13.018 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.018 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:13.018 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.018 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:13.018 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:57.442444+0000", 2024-02-19T21:16:13.018 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:13.018 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.018 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.018 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:13.018 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:57.442444+0000", 2024-02-19T21:16:13.018 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:16:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:57.442442+0000", 2024-02-19T21:16:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0851e-05 2024-02-19T21:16:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:57.442452+0000", 2024-02-19T21:16:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5046e-05 2024-02-19T21:16:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:57.442467+0000", 2024-02-19T21:16:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.404e-06 2024-02-19T21:16:13.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.020 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:13.020 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:57.442474+0000", 2024-02-19T21:16:13.020 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.1696000000000002e-05 2024-02-19T21:16:13.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.020 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:13.020 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:57.442516+0000", 2024-02-19T21:16:13.020 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8930000000000002e-06 2024-02-19T21:16:13.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.020 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.020 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:16:13.021 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:57.442521+0000", 2024-02-19T21:16:13.021 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0027323489999999998 2024-02-19T21:16:13.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.021 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.021 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:13.021 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:57.445254+0000", 2024-02-19T21:16:13.021 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4300000000000001e-05 2024-02-19T21:16:13.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.021 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.021 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:13.021 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:57.445278+0000", 2024-02-19T21:16:13.021 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.4985000000000001e-05 2024-02-19T21:16:13.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:13.021 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:13.022 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:13.022 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:57.445353+0000", 2024-02-19T21:16:13.022 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.002908987 2024-02-19T21:16:13.022 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:13.022 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:13.022 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:13.022 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:13.022 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:13.022 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:16:13.023 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:13.023 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:13.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:13.124 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:16:13.152 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:13.172 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:16:13.176 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:13.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:16:13.277 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:13.304 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:13.521 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:12.035440+0000 osd.6 (osd.6) 251 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:16:13.521 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:12.035701+0000 osd.6 (osd.6) 252 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:16:13.522 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:13.399799+0000 mon.a (mon.0) 3489 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:16:13.522 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:13.399865+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2024-02-19T21:16:13.522 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:13.401504+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:16:13.539 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-19T21:16:13.552 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:13.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:13.656 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:13.684 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:12.035440+0000 osd.6 (osd.6) 251 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:16:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:12.035701+0000 osd.6 (osd.6) 252 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:16:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:13.399799+0000 mon.a (mon.0) 3489 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:16:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:13.399865+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2024-02-19T21:16:13.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:13.401504+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:16:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:12.035440+0000 osd.6 (osd.6) 251 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:16:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:12.035701+0000 osd.6 (osd.6) 252 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:16:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:13.399799+0000 mon.a (mon.0) 3489 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:16:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:13.399865+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2024-02-19T21:16:13.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:13.401504+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-02-19T21:16:13.749 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:16:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:16:13] "GET /metrics HTTP/1.1" 200 189817 "" "Prometheus/2.33.4" 2024-02-19T21:16:13.817 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:16:14.334 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:14.337 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:14.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:14.438 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:14.467 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:14.495 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:14.497 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:14.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:14.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:13.317129+0000 mgr.y (mgr.14550) 608 : cluster [DBG] pgmap v1091: 214 pgs: 32 creating+peering, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-19T21:16:14.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:13.911147+0000 osd.7 (osd.7) 325 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:16:14.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:13.914066+0000 osd.7 (osd.7) 326 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:16:14.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:14.154536+0000 osd.4 (osd.4) 251 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:16:14.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:14.155993+0000 osd.4 (osd.4) 252 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:16:14.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:14.404088+0000 mon.a (mon.0) 3492 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-19T21:16:14.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:14.404153+0000 mon.a (mon.0) 3493 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2024-02-19T21:16:14.599 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:14.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.599+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/2050795848 >> v1:172.21.15.163:6790/0 conn(0x7f4e180fc870 legacy=0x7f4e180fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:14.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.599+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/2050795848 shutdown_connections 2024-02-19T21:16:14.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.599+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/2050795848 >> 172.21.15.163:0/2050795848 conn(0x7f4e180f2470 msgr2=0x7f4e180f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:14.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.599+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/2050795848 shutdown_connections 2024-02-19T21:16:14.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.599+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/2050795848 wait complete. 2024-02-19T21:16:14.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.599+0000 7f4e1dd9c700 1 librados: starting msgr at 2024-02-19T21:16:14.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.599+0000 7f4e1dd9c700 1 librados: starting objecter 2024-02-19T21:16:14.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e1dd9c700 1 Processor -- start 2024-02-19T21:16:14.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e1dd9c700 1 -- start start 2024-02-19T21:16:14.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e1dd9c700 1 librados: setting wanted keys 2024-02-19T21:16:14.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e1dd9c700 1 librados: calling monclient init 2024-02-19T21:16:14.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e1dd9c700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4e181172e0 con 0x7f4e181031b0 2024-02-19T21:16:14.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e1dd9c700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4e181184c0 con 0x7f4e180ff670 2024-02-19T21:16:14.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e1dd9c700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4e181196c0 con 0x7f4e180fc870 2024-02-19T21:16:14.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e17fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f4e181031b0 0x7f4e181159a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:45934/0 (socket says 172.21.15.163:45934) 2024-02-19T21:16:14.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e17fff700 1 -- 172.21.15.163:0/4140827725 learned_addr learned my addr 172.21.15.163:0/4140827725 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:14.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e14ff9700 10 client.?.objecter ms_handle_connect 0x7f4e181031b0 2024-02-19T21:16:14.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e14ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:14.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e14ff9700 10 client.?.objecter ms_handle_connect 0x7f4e180ff670 2024-02-19T21:16:14.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e14ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:14.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e14ff9700 10 client.?.objecter ms_handle_connect 0x7f4e180fc870 2024-02-19T21:16:14.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e14ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842407219 0 0) 0x7f4e181172e0 con 0x7f4e181031b0 2024-02-19T21:16:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e000033f0 con 0x7f4e181031b0 2024-02-19T21:16:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947806330 0 0) 0x7f4e181184c0 con 0x7f4e180ff670 2024-02-19T21:16:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e181172e0 con 0x7f4e180ff670 2024-02-19T21:16:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878724984 0 0) 0x7f4e181196c0 con 0x7f4e180fc870 2024-02-19T21:16:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.600+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e181184c0 con 0x7f4e180fc870 2024-02-19T21:16:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1367369941 0 0) 0x7f4e000033f0 con 0x7f4e181031b0 2024-02-19T21:16:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e181196c0 con 0x7f4e181031b0 2024-02-19T21:16:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4e10001dd0 con 0x7f4e181031b0 2024-02-19T21:16:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 246367610 0 0) 0x7f4e181184c0 con 0x7f4e180fc870 2024-02-19T21:16:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e000033f0 con 0x7f4e180fc870 2024-02-19T21:16:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4050702930 0 0) 0x7f4e181172e0 con 0x7f4e180ff670 2024-02-19T21:16:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e181184c0 con 0x7f4e180ff670 2024-02-19T21:16:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4e08001e40 con 0x7f4e180fc870 2024-02-19T21:16:14.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4e0c001840 con 0x7f4e180ff670 2024-02-19T21:16:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2068282936 0 0) 0x7f4e000033f0 con 0x7f4e180fc870 2024-02-19T21:16:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 >> v1:172.21.15.193:6789/0 conn(0x7f4e180ff670 legacy=0x7f4e180fa180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 >> v1:172.21.15.163:6789/0 conn(0x7f4e181031b0 legacy=0x7f4e181159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1811a8c0 con 0x7f4e180fc870 2024-02-19T21:16:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e181186b0 con 0x7f4e180fc870 2024-02-19T21:16:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e1dd9c700 10 client.50504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e18118ba0 con 0x7f4e180fc870 2024-02-19T21:16:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e1dd9c700 1 librados: init done 2024-02-19T21:16:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f4e080019e0 con 0x7f4e180fc870 2024-02-19T21:16:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.601+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4e08003500 con 0x7f4e180fc870 2024-02-19T21:16:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.602+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4e04004f00 con 0x7f4e180fc870 2024-02-19T21:16:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.602+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f4e08016450 con 0x7f4e180fc870 2024-02-19T21:16:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(725..725 src has 623..725) v4 ==== 8678+0+0 (unknown 1042178223 0 0) 0x7f4e08076910 con 0x7f4e180fc870 2024-02-19T21:16:14.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 osd_map(725..725 src has 623..725) v4 2024-02-19T21:16:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e14ff9700 3 client.50504.objecter handle_osd_map got epochs [725,725] > 0 2024-02-19T21:16:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e14ff9700 3 client.50504.objecter handle_osd_map decoding full epoch 725 2024-02-19T21:16:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e14ff9700 20 client.50504.objecter dump_active .. 0 homeless 2024-02-19T21:16:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v725) v1 ==== 54+0+16 (unknown 3048829041 0 2420000192) 0x7f4e08076c90 con 0x7f4e180fc870 2024-02-19T21:16:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e14ff9700 10 client.50504.objecter ms_handle_connect 0x7f4e00063c90 2024-02-19T21:16:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 1 2024-02-19T21:16:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e040051a0 osd=0 3 2024-02-19T21:16:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e040051a0 osd=0 3 2024-02-19T21:16:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e040051a0 osd=0 3 2024-02-19T21:16:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 1 2024-02-19T21:16:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e040051a0 osd=0 3 2024-02-19T21:16:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 0 1 2024-02-19T21:16:14.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e040051a0 osd=0 4 2024-02-19T21:16:14.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 1 2024-02-19T21:16:14.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e04004950 con 0x7f4e04004530 2024-02-19T21:16:14.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.603+0000 7f4e14ff9700 10 client.50504.objecter ms_handle_connect 0x7f4e04004530 2024-02-19T21:16:14.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.604+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e04004950 con 0x7f4e04004530 2024-02-19T21:16:14.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.604+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:16:14.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.604+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 1 = system:0 2024-02-19T21:16:14.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.604+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e040051a0 osd=0 4 2024-02-19T21:16:14.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.604+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 0 1 2024-02-19T21:16:14.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.618+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:14.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.619+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 2 2024-02-19T21:16:14.619 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.619+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e040051a0 osd=0 3 2024-02-19T21:16:14.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.619+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e040051a0 osd=0 3 2024-02-19T21:16:14.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.619+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e040051a0 osd=0 3 2024-02-19T21:16:14.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.619+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 2 2024-02-19T21:16:14.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.619+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e040051a0 osd=0 3 2024-02-19T21:16:14.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.619+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 0 2 2024-02-19T21:16:14.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.619+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e040051a0 osd=0 4 2024-02-19T21:16:14.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.619+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 2 2024-02-19T21:16:14.620 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.619+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4e04004d30 con 0x7f4e04004530 2024-02-19T21:16:14.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4e04004d30 con 0x7f4e04004530 2024-02-19T21:16:14.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:14.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e040051a0 osd=0 4 2024-02-19T21:16:14.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 0 2 2024-02-19T21:16:14.622 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:16:14.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:14.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 3 2024-02-19T21:16:14.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04003fe0 osd=1 3 2024-02-19T21:16:14.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04003fe0 osd=1 3 2024-02-19T21:16:14.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04003fe0 osd=1 3 2024-02-19T21:16:14.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 3 2024-02-19T21:16:14.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e04003fe0 osd=1 3 2024-02-19T21:16:14.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 1 3 2024-02-19T21:16:14.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04003fe0 osd=1 4 2024-02-19T21:16:14.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 3 2024-02-19T21:16:14.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.620+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e0400c000 con 0x7f4e04004150 2024-02-19T21:16:14.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.621+0000 7f4e14ff9700 10 client.50504.objecter ms_handle_connect 0x7f4e04004150 2024-02-19T21:16:14.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.621+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e0400c000 con 0x7f4e04004150 2024-02-19T21:16:14.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.621+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:16:14.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.621+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 3 = system:0 2024-02-19T21:16:14.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.621+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e04003fe0 osd=1 4 2024-02-19T21:16:14.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.621+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 1 3 2024-02-19T21:16:14.628 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:14.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.636+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:14.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.636+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 4 2024-02-19T21:16:14.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.636+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04003fe0 osd=1 3 2024-02-19T21:16:14.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.636+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04003fe0 osd=1 3 2024-02-19T21:16:14.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.636+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04003fe0 osd=1 3 2024-02-19T21:16:14.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.636+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 4 2024-02-19T21:16:14.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.636+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e04003fe0 osd=1 3 2024-02-19T21:16:14.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.636+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 1 4 2024-02-19T21:16:14.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.636+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04003fe0 osd=1 4 2024-02-19T21:16:14.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.636+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 4 2024-02-19T21:16:14.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.636+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4e040018b0 con 0x7f4e04004150 2024-02-19T21:16:14.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4e040018b0 con 0x7f4e04004150 2024-02-19T21:16:14.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:14.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e04003fe0 osd=1 4 2024-02-19T21:16:14.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 1 4 2024-02-19T21:16:14.638 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:16:14.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:14.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 5 2024-02-19T21:16:14.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04003630 osd=2 3 2024-02-19T21:16:14.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04003630 osd=2 3 2024-02-19T21:16:14.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04003630 osd=2 3 2024-02-19T21:16:14.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 5 2024-02-19T21:16:14.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e04003630 osd=2 3 2024-02-19T21:16:14.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 2 5 2024-02-19T21:16:14.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04003630 osd=2 4 2024-02-19T21:16:14.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 5 2024-02-19T21:16:14.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.637+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e04003e70 con 0x7f4e0400d2f0 2024-02-19T21:16:14.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.638+0000 7f4e14ff9700 10 client.50504.objecter ms_handle_connect 0x7f4e0400d2f0 2024-02-19T21:16:14.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.638+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e04003e70 con 0x7f4e0400d2f0 2024-02-19T21:16:14.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.638+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:16:14.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.638+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 5 = system:0 2024-02-19T21:16:14.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.638+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e04003630 osd=2 4 2024-02-19T21:16:14.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.638+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 2 5 2024-02-19T21:16:14.647 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:14.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:14.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:14.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.652+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:14.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.652+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 6 2024-02-19T21:16:14.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.652+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04003630 osd=2 3 2024-02-19T21:16:14.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.652+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04003630 osd=2 3 2024-02-19T21:16:14.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.652+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04003630 osd=2 3 2024-02-19T21:16:14.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.652+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 6 2024-02-19T21:16:14.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.652+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e04003630 osd=2 3 2024-02-19T21:16:14.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.652+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 2 6 2024-02-19T21:16:14.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.652+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04003630 osd=2 4 2024-02-19T21:16:14.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.652+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 6 2024-02-19T21:16:14.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.652+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4e0400e730 con 0x7f4e0400d2f0 2024-02-19T21:16:14.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4e0400e730 con 0x7f4e0400d2f0 2024-02-19T21:16:14.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:14.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e04003630 osd=2 4 2024-02-19T21:16:14.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 2 6 2024-02-19T21:16:14.655 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:16:14.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:14.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 7 2024-02-19T21:16:14.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e040033e0 osd=3 3 2024-02-19T21:16:14.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e040033e0 osd=3 3 2024-02-19T21:16:14.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e040033e0 osd=3 3 2024-02-19T21:16:14.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 7 2024-02-19T21:16:14.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e040033e0 osd=3 3 2024-02-19T21:16:14.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 3 7 2024-02-19T21:16:14.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e040033e0 osd=3 4 2024-02-19T21:16:14.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 7 2024-02-19T21:16:14.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e04015a60 con 0x7f4e04011e80 2024-02-19T21:16:14.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.654+0000 7f4e14ff9700 10 client.50504.objecter ms_handle_connect 0x7f4e04011e80 2024-02-19T21:16:14.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.655+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e04015a60 con 0x7f4e04011e80 2024-02-19T21:16:14.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.655+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:16:14.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.655+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 7 = system:0 2024-02-19T21:16:14.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.655+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e040033e0 osd=3 4 2024-02-19T21:16:14.657 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.655+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 3 7 2024-02-19T21:16:14.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.669+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:14.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.669+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 8 2024-02-19T21:16:14.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.669+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e040033e0 osd=3 3 2024-02-19T21:16:14.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.669+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e040033e0 osd=3 3 2024-02-19T21:16:14.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.669+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e040033e0 osd=3 3 2024-02-19T21:16:14.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.669+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 8 2024-02-19T21:16:14.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.669+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e040033e0 osd=3 3 2024-02-19T21:16:14.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.669+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 3 8 2024-02-19T21:16:14.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.669+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e040033e0 osd=3 4 2024-02-19T21:16:14.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.669+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 8 2024-02-19T21:16:14.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.669+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4e0400e1a0 con 0x7f4e04011e80 2024-02-19T21:16:14.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4e0400e1a0 con 0x7f4e04011e80 2024-02-19T21:16:14.671 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:14.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e040033e0 osd=3 4 2024-02-19T21:16:14.672 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 3 8 2024-02-19T21:16:14.672 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:16:14.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:14.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 9 2024-02-19T21:16:14.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e0400e1a0 osd=4 3 2024-02-19T21:16:14.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e0400e1a0 osd=4 3 2024-02-19T21:16:14.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e0400e1a0 osd=4 3 2024-02-19T21:16:14.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 9 2024-02-19T21:16:14.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e0400e1a0 osd=4 3 2024-02-19T21:16:14.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 4 9 2024-02-19T21:16:14.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e0400e1a0 osd=4 4 2024-02-19T21:16:14.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 9 2024-02-19T21:16:14.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e0400e4d0 con 0x7f4e0400ce50 2024-02-19T21:16:14.673 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.671+0000 7f4e14ff9700 10 client.50504.objecter ms_handle_connect 0x7f4e0400ce50 2024-02-19T21:16:14.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.672+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e0400e4d0 con 0x7f4e0400ce50 2024-02-19T21:16:14.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.672+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:16:14.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.672+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 9 = system:0 2024-02-19T21:16:14.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.672+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e0400e1a0 osd=4 4 2024-02-19T21:16:14.674 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.672+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 4 9 2024-02-19T21:16:14.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.686+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:14.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.686+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 10 2024-02-19T21:16:14.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.686+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e0400e1a0 osd=4 3 2024-02-19T21:16:14.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.686+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e0400e1a0 osd=4 3 2024-02-19T21:16:14.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.686+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e0400e1a0 osd=4 3 2024-02-19T21:16:14.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.686+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 10 2024-02-19T21:16:14.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.686+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e0400e1a0 osd=4 3 2024-02-19T21:16:14.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.686+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 4 10 2024-02-19T21:16:14.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.686+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e0400e1a0 osd=4 4 2024-02-19T21:16:14.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.686+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 10 2024-02-19T21:16:14.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.686+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4e0400e4d0 con 0x7f4e0400ce50 2024-02-19T21:16:14.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4e0400e4d0 con 0x7f4e0400ce50 2024-02-19T21:16:14.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:14.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e0400e1a0 osd=4 4 2024-02-19T21:16:14.689 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 4 10 2024-02-19T21:16:14.690 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:16:14.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:14.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 11 2024-02-19T21:16:14.690 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e0400e4d0 osd=5 3 2024-02-19T21:16:14.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e0400e4d0 osd=5 3 2024-02-19T21:16:14.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e0400e4d0 osd=5 3 2024-02-19T21:16:14.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 11 2024-02-19T21:16:14.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e0400e4d0 osd=5 3 2024-02-19T21:16:14.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 5 11 2024-02-19T21:16:14.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e0400e4d0 osd=5 4 2024-02-19T21:16:14.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 11 2024-02-19T21:16:14.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.688+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e04017f70 con 0x7f4e040179a0 2024-02-19T21:16:14.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.689+0000 7f4e14ff9700 10 client.50504.objecter ms_handle_connect 0x7f4e040179a0 2024-02-19T21:16:14.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.689+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e04017f70 con 0x7f4e040179a0 2024-02-19T21:16:14.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.689+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:16:14.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.689+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 11 = system:0 2024-02-19T21:16:14.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.689+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e0400e4d0 osd=5 4 2024-02-19T21:16:14.691 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.689+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 5 11 2024-02-19T21:16:14.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.704+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:14.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.704+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 12 2024-02-19T21:16:14.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.704+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e0400e4d0 osd=5 3 2024-02-19T21:16:14.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.704+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e0400e4d0 osd=5 3 2024-02-19T21:16:14.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.704+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e0400e4d0 osd=5 3 2024-02-19T21:16:14.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.704+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 12 2024-02-19T21:16:14.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.704+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e0400e4d0 osd=5 3 2024-02-19T21:16:14.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.704+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 5 12 2024-02-19T21:16:14.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.704+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e0400e4d0 osd=5 4 2024-02-19T21:16:14.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.704+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 12 2024-02-19T21:16:14.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.704+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4e04017f70 con 0x7f4e040179a0 2024-02-19T21:16:14.706 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:16:14.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.705+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4e04017f70 con 0x7f4e040179a0 2024-02-19T21:16:14.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.705+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:14.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.705+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.705+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e0400e4d0 osd=5 4 2024-02-19T21:16:14.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.705+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 5 12 2024-02-19T21:16:14.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.706+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:14.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.706+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 13 2024-02-19T21:16:14.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.706+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04017f70 osd=6 3 2024-02-19T21:16:14.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.706+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04017f70 osd=6 3 2024-02-19T21:16:14.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.706+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04017f70 osd=6 3 2024-02-19T21:16:14.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.706+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 13 2024-02-19T21:16:14.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.706+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e04017f70 osd=6 3 2024-02-19T21:16:14.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.706+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 6 13 2024-02-19T21:16:14.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.706+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04017f70 osd=6 4 2024-02-19T21:16:14.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.706+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 13 2024-02-19T21:16:14.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.706+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e04022b70 con 0x7f4e0401f060 2024-02-19T21:16:14.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.706+0000 7f4e14ff9700 10 client.50504.objecter ms_handle_connect 0x7f4e0401f060 2024-02-19T21:16:14.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.707+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e04022b70 con 0x7f4e0401f060 2024-02-19T21:16:14.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.707+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:16:14.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.707+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 13 = system:0 2024-02-19T21:16:14.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.707+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e04017f70 osd=6 4 2024-02-19T21:16:14.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.707+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 6 13 2024-02-19T21:16:14.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.720+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:14.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.720+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 14 2024-02-19T21:16:14.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.720+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04017f70 osd=6 3 2024-02-19T21:16:14.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.720+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04017f70 osd=6 3 2024-02-19T21:16:14.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.720+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04017f70 osd=6 3 2024-02-19T21:16:14.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.720+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 14 2024-02-19T21:16:14.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.720+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e04017f70 osd=6 3 2024-02-19T21:16:14.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.720+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 6 14 2024-02-19T21:16:14.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.720+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04017f70 osd=6 4 2024-02-19T21:16:14.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.720+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 14 2024-02-19T21:16:14.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.720+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4e04022b70 con 0x7f4e0401f060 2024-02-19T21:16:14.723 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:16:14.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4e04022b70 con 0x7f4e0401f060 2024-02-19T21:16:14.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:14.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e04017f70 osd=6 4 2024-02-19T21:16:14.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 6 14 2024-02-19T21:16:14.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:14.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 15 2024-02-19T21:16:14.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04022b70 osd=7 3 2024-02-19T21:16:14.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04022b70 osd=7 3 2024-02-19T21:16:14.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04022b70 osd=7 3 2024-02-19T21:16:14.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 15 2024-02-19T21:16:14.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e04022b70 osd=7 3 2024-02-19T21:16:14.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 7 15 2024-02-19T21:16:14.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04022b70 osd=7 4 2024-02-19T21:16:14.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 15 2024-02-19T21:16:14.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.722+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4e04023df0 con 0x7f4e04024870 2024-02-19T21:16:14.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.723+0000 7f4e14ff9700 10 client.50504.objecter ms_handle_connect 0x7f4e04024870 2024-02-19T21:16:14.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.723+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4e04023df0 con 0x7f4e04024870 2024-02-19T21:16:14.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.723+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:16:14.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.723+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 15 = system:0 2024-02-19T21:16:14.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.723+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e04022b70 osd=7 4 2024-02-19T21:16:14.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.723+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 7 15 2024-02-19T21:16:14.738 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.737+0000 7f4e1dd9c700 10 client.50504.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:14.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.737+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign -1 16 2024-02-19T21:16:14.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.737+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04022b70 osd=7 3 2024-02-19T21:16:14.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.737+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04022b70 osd=7 3 2024-02-19T21:16:14.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.737+0000 7f4e1dd9c700 20 client.50504.objecter _get_session s=0x7f4e04022b70 osd=7 3 2024-02-19T21:16:14.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.737+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_remove -1 16 2024-02-19T21:16:14.739 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.737+0000 7f4e1dd9c700 20 client.50504.objecter get_session s=0x7f4e04022b70 osd=7 3 2024-02-19T21:16:14.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.737+0000 7f4e1dd9c700 15 client.50504.objecter _session_command_op_assign 7 16 2024-02-19T21:16:14.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.737+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04022b70 osd=7 4 2024-02-19T21:16:14.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.737+0000 7f4e1dd9c700 10 client.50504.objecter _send_command 16 2024-02-19T21:16:14.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.737+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4e04023df0 con 0x7f4e04024870 2024-02-19T21:16:14.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e14ff9700 1 -- 172.21.15.163:0/4140827725 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4e04023df0 con 0x7f4e04024870 2024-02-19T21:16:14.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e14ff9700 10 client.50504.objecter ms_dispatch 0x7f4e18001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:14.740 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e14ff9700 10 client.50504.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e14ff9700 20 client.50504.objecter put_session s=0x7f4e04022b70 osd=7 4 2024-02-19T21:16:14.741 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e14ff9700 15 client.50504.objecter _session_command_op_remove 7 16 2024-02-19T21:16:14.741 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:16:14.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 10 librados: watch_flush enter 2024-02-19T21:16:14.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 10 librados: watch_flush exit 2024-02-19T21:16:14.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 10 client.50504.objecter close_session for osd.0 2024-02-19T21:16:14.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 >> v1:172.21.15.163:6801/1559418115 conn(0x7f4e04004530 legacy=0x7f4e04002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:14.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e040051a0 osd=0 1 2024-02-19T21:16:14.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 10 client.50504.objecter close_session for osd.1 2024-02-19T21:16:14.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 >> v1:172.21.15.163:6805/3125047432 conn(0x7f4e04004150 legacy=0x7f4e0400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:14.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04003fe0 osd=1 1 2024-02-19T21:16:14.742 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 10 client.50504.objecter close_session for osd.2 2024-02-19T21:16:14.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 >> v1:172.21.15.163:6809/1563158487 conn(0x7f4e0400d2f0 legacy=0x7f4e0400d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:14.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04003630 osd=2 1 2024-02-19T21:16:14.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 10 client.50504.objecter close_session for osd.3 2024-02-19T21:16:14.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 >> v1:172.21.15.163:6813/2112279769 conn(0x7f4e04011e80 legacy=0x7f4e04014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:14.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e040033e0 osd=3 1 2024-02-19T21:16:14.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 10 client.50504.objecter close_session for osd.4 2024-02-19T21:16:14.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 >> v1:172.21.15.193:6800/3129716965 conn(0x7f4e0400ce50 legacy=0x7f4e04015c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:14.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e0400e1a0 osd=4 1 2024-02-19T21:16:14.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 10 client.50504.objecter close_session for osd.5 2024-02-19T21:16:14.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 >> v1:172.21.15.193:6804/805161991 conn(0x7f4e040179a0 legacy=0x7f4e0401d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:14.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e0400e4d0 osd=5 1 2024-02-19T21:16:14.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e17fff700 1 -- 172.21.15.163:0/4140827725 reap_dead start 2024-02-19T21:16:14.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 10 client.50504.objecter close_session for osd.6 2024-02-19T21:16:14.743 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 >> v1:172.21.15.193:6808/787604615 conn(0x7f4e0401f060 legacy=0x7f4e04021450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:14.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e17fff700 1 -- 172.21.15.163:0/4140827725 reap_dead start 2024-02-19T21:16:14.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e17fff700 1 -- 172.21.15.163:0/4140827725 reap_dead start 2024-02-19T21:16:14.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e17fff700 1 -- 172.21.15.163:0/4140827725 reap_dead start 2024-02-19T21:16:14.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04017f70 osd=6 1 2024-02-19T21:16:14.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 10 client.50504.objecter close_session for osd.7 2024-02-19T21:16:14.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 >> v1:172.21.15.193:6812/157927432 conn(0x7f4e04024870 legacy=0x7f4e04024cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:14.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 20 client.50504.objecter put_session s=0x7f4e04022b70 osd=7 1 2024-02-19T21:16:14.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 20 client.50504.objecter shutdown clearing up homeless session... 2024-02-19T21:16:14.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 10 client.50504.objecter successfully canceled tick 2024-02-19T21:16:14.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4e00063c90 legacy=0x7f4e00066150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:14.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 >> v1:172.21.15.163:6790/0 conn(0x7f4e180fc870 legacy=0x7f4e180f9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:14.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.739+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 shutdown_connections 2024-02-19T21:16:14.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.740+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 >> 172.21.15.163:0/4140827725 conn(0x7f4e180f2470 msgr2=0x7f4e18106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:14.744 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.740+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 shutdown_connections 2024-02-19T21:16:14.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.740+0000 7f4e1dd9c700 1 -- 172.21.15.163:0/4140827725 wait complete. 2024-02-19T21:16:14.745 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:14.740+0000 7f4e1dd9c700 1 librados: shutdown 2024-02-19T21:16:14.745 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.745 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.745 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.745 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.745 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.745 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.745 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.745 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:16:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:13.317129+0000 mgr.y (mgr.14550) 608 : cluster [DBG] pgmap v1091: 214 pgs: 32 creating+peering, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-19T21:16:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:13.911147+0000 osd.7 (osd.7) 325 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:16:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:13.914066+0000 osd.7 (osd.7) 326 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:16:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:14.154536+0000 osd.4 (osd.4) 251 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:16:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:14.155993+0000 osd.4 (osd.4) 252 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:16:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:14.404088+0000 mon.a (mon.0) 3492 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-19T21:16:14.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:14.404153+0000 mon.a (mon.0) 3493 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2024-02-19T21:16:14.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:13.317129+0000 mgr.y (mgr.14550) 608 : cluster [DBG] pgmap v1091: 214 pgs: 32 creating+peering, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-02-19T21:16:14.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:13.911147+0000 osd.7 (osd.7) 325 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:16:14.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:13.914066+0000 osd.7 (osd.7) 326 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:16:14.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:14.154536+0000 osd.4 (osd.4) 251 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:16:14.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:14.155993+0000 osd.4 (osd.4) 252 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:16:14.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:14.404088+0000 mon.a (mon.0) 3492 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-128","var": "hit_set_type","val": "explicit_object"}]': finished 2024-02-19T21:16:14.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:14.404153+0000 mon.a (mon.0) 3493 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2024-02-19T21:16:14.750 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:14.778 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:14.796 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:14.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:14.913 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:14.941 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:14.994 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:14.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:15.098 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:15.126 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:15.374 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:15.376 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:15.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:15.477 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:15.507 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:15.584 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:16:15.609 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:15.610 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:13.999900+0000 osd.5 (osd.5) 294 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:16:15.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:14.002982+0000 osd.5 (osd.5) 295 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:16:15.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:14.082715+0000 osd.1 (osd.1) 320 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:16:15.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:14.086186+0000 osd.1 (osd.1) 321 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:16:15.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:14.604141+0000 mon.c (mon.2) 430 : audit [DBG] from='client.? v1:172.21.15.163:0/4140827725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:15.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:15.135097+0000 osd.3 (osd.3) 301 : cluster [DBG] 2.23 scrub starts 2024-02-19T21:16:15.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:15.135977+0000 osd.3 (osd.3) 302 : cluster [DBG] 2.23 scrub ok 2024-02-19T21:16:15.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:15.180764+0000 osd.4 (osd.4) 253 : cluster [DBG] 16.2 scrub starts 2024-02-19T21:16:15.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:15.181738+0000 osd.4 (osd.4) 254 : cluster [DBG] 16.2 scrub ok 2024-02-19T21:16:15.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:15.416810+0000 mon.a (mon.0) 3494 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:16:15.611 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:15.417127+0000 mon.a (mon.0) 3495 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-128"}]: dispatch 2024-02-19T21:16:15.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:15.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:15.620 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:16:15.620 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:16:15.714 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:15.741 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:15.760 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:15.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:13.999900+0000 osd.5 (osd.5) 294 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:16:15.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:14.002982+0000 osd.5 (osd.5) 295 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:16:15.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:14.082715+0000 osd.1 (osd.1) 320 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:16:15.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:14.086186+0000 osd.1 (osd.1) 321 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:16:15.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:14.604141+0000 mon.c (mon.2) 430 : audit [DBG] from='client.? v1:172.21.15.163:0/4140827725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:15.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:15.135097+0000 osd.3 (osd.3) 301 : cluster [DBG] 2.23 scrub starts 2024-02-19T21:16:15.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:15.135977+0000 osd.3 (osd.3) 302 : cluster [DBG] 2.23 scrub ok 2024-02-19T21:16:15.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:15.180764+0000 osd.4 (osd.4) 253 : cluster [DBG] 16.2 scrub starts 2024-02-19T21:16:15.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:15.181738+0000 osd.4 (osd.4) 254 : cluster [DBG] 16.2 scrub ok 2024-02-19T21:16:15.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:15.416810+0000 mon.a (mon.0) 3494 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:16:15.763 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:15.417127+0000 mon.a (mon.0) 3495 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-128"}]: dispatch 2024-02-19T21:16:15.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:13.999900+0000 osd.5 (osd.5) 294 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:16:15.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:14.002982+0000 osd.5 (osd.5) 295 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:16:15.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:14.082715+0000 osd.1 (osd.1) 320 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:16:15.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:14.086186+0000 osd.1 (osd.1) 321 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:16:15.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:14.604141+0000 mon.c (mon.2) 430 : audit [DBG] from='client.? v1:172.21.15.163:0/4140827725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:15.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:15.135097+0000 osd.3 (osd.3) 301 : cluster [DBG] 2.23 scrub starts 2024-02-19T21:16:15.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:15.135977+0000 osd.3 (osd.3) 302 : cluster [DBG] 2.23 scrub ok 2024-02-19T21:16:15.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:15.180764+0000 osd.4 (osd.4) 253 : cluster [DBG] 16.2 scrub starts 2024-02-19T21:16:15.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:15.181738+0000 osd.4 (osd.4) 254 : cluster [DBG] 16.2 scrub ok 2024-02-19T21:16:15.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:15.416810+0000 mon.a (mon.0) 3494 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:16:15.764 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:15.417127+0000 mon.a (mon.0) 3495 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-128"}]: dispatch 2024-02-19T21:16:15.765 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:15.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:15.858 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:15.866 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:15.894 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:16.142 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:16.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:16.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:16.248 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:16.275 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:16.380 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-19T21:16:16.508 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-02-19T21:16:16.508 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12004 ms) 2024-02-19T21:16:16.508 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-02-19T21:16:16.509 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-02-19T21:16:16.509 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: 00000010 69 73 |is| 2024-02-19T21:16:16.509 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: 00000012 2024-02-19T21:16:16.509 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12026 ms) 2024-02-19T21:16:16.509 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-02-19T21:16:16.509 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: require_osd_release = pacific 2024-02-19T21:16:16.509 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (5986 ms) 2024-02-19T21:16:16.509 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [----------] 43 tests from LibRadosTwoPoolsPP (499740 ms total) 2024-02-19T21:16:16.509 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: 2024-02-19T21:16:16.509 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-02-19T21:16:16.509 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-02-19T21:16:16.509 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1024 ms) 2024-02-19T21:16:16.509 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-02-19T21:16:16.509 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10986 ms) 2024-02-19T21:16:16.510 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-02-19T21:16:16.510 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (17979 ms) 2024-02-19T21:16:16.510 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-02-19T21:16:16.510 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-02-19T21:16:16.510 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (29992 ms total) 2024-02-19T21:16:16.510 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: 2024-02-19T21:16:16.510 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-02-19T21:16:16.510 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-02-19T21:16:16.510 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7037 ms) 2024-02-19T21:16:16.510 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-02-19T21:16:16.510 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8046 ms) 2024-02-19T21:16:16.510 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-02-19T21:16:16.510 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: waiting for scrub... 2024-02-19T21:16:16.510 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: done waiting 2024-02-19T21:16:16.510 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24933 ms) 2024-02-19T21:16:16.511 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-02-19T21:16:16.511 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10001 ms) 2024-02-19T21:16:16.511 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-02-19T21:16:16.511 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8009 ms) 2024-02-19T21:16:16.511 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-02-19T21:16:16.511 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8009 ms) 2024-02-19T21:16:16.511 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-02-19T21:16:16.511 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10005 ms) 2024-02-19T21:16:16.511 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-02-19T21:16:16.511 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (8051 ms) 2024-02-19T21:16:16.511 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-02-19T21:16:16.511 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13091 ms) 2024-02-19T21:16:16.511 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-02-19T21:16:16.511 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (7863 ms) 2024-02-19T21:16:16.511 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-02-19T21:16:16.512 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13011 ms) 2024-02-19T21:16:16.512 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-02-19T21:16:16.512 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8004 ms) 2024-02-19T21:16:16.512 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-02-19T21:16:16.512 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8039 ms) 2024-02-19T21:16:16.512 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-02-19T21:16:16.512 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: hmm, no HitSet yet 2024-02-19T21:16:16.512 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: ok, hit_set contains 248:602f83fe:::foo:head 2024-02-19T21:16:16.695 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:16:17.011 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:15.003442+0000 osd.5 (osd.5) 296 : cluster [DBG] 16.11 scrub starts 2024-02-19T21:16:17.011 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:15.006546+0000 osd.5 (osd.5) 297 : cluster [DBG] 16.11 scrub ok 2024-02-19T21:16:17.011 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:15.318003+0000 mgr.y (mgr.14550) 609 : cluster [DBG] pgmap v1094: 214 pgs: 214 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:17.011 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:15.504724+0000 mon.a (mon.0) 3496 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-128"}]': finished 2024-02-19T21:16:17.011 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:15.504801+0000 mon.a (mon.0) 3497 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2024-02-19T21:16:17.011 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:16.206535+0000 osd.4 (osd.4) 255 : cluster [DBG] 6.e scrub starts 2024-02-19T21:16:17.012 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:16.207392+0000 osd.4 (osd.4) 256 : cluster [DBG] 6.e scrub ok 2024-02-19T21:16:17.177 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:15.003442+0000 osd.5 (osd.5) 296 : cluster [DBG] 16.11 scrub starts 2024-02-19T21:16:17.177 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:15.006546+0000 osd.5 (osd.5) 297 : cluster [DBG] 16.11 scrub ok 2024-02-19T21:16:17.177 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:15.318003+0000 mgr.y (mgr.14550) 609 : cluster [DBG] pgmap v1094: 214 pgs: 214 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:17.177 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:15.504724+0000 mon.a (mon.0) 3496 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-128"}]': finished 2024-02-19T21:16:17.177 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:15.504801+0000 mon.a (mon.0) 3497 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2024-02-19T21:16:17.177 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:16.206535+0000 osd.4 (osd.4) 255 : cluster [DBG] 6.e scrub starts 2024-02-19T21:16:17.178 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:16.207392+0000 osd.4 (osd.4) 256 : cluster [DBG] 6.e scrub ok 2024-02-19T21:16:17.178 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:15.003442+0000 osd.5 (osd.5) 296 : cluster [DBG] 16.11 scrub starts 2024-02-19T21:16:17.178 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:15.006546+0000 osd.5 (osd.5) 297 : cluster [DBG] 16.11 scrub ok 2024-02-19T21:16:17.178 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:15.318003+0000 mgr.y (mgr.14550) 609 : cluster [DBG] pgmap v1094: 214 pgs: 214 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:17.178 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:15.504724+0000 mon.a (mon.0) 3496 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-128"}]': finished 2024-02-19T21:16:17.178 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:15.504801+0000 mon.a (mon.0) 3497 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2024-02-19T21:16:17.178 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:16.206535+0000 osd.4 (osd.4) 255 : cluster [DBG] 6.e scrub starts 2024-02-19T21:16:17.178 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:16.207392+0000 osd.4 (osd.4) 256 : cluster [DBG] 6.e scrub ok 2024-02-19T21:16:17.224 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:17.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:17.327 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:17.355 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:17.425 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 7, 3, 4, 6, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:16:17.426 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:16:17.426 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-19T21:16:17.426 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.000000 2024-02-19T21:16:17.426 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd primary-affinity 7 0 2024-02-19T21:16:17.503 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:17.512 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:17.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:17.525 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:16:17.613 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:17.640 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:17.678 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:17.681 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:17.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:17.782 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:17.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.785+0000 7fec1167a700 1 -- 172.21.15.163:0/4274781884 >> v1:172.21.15.193:6789/0 conn(0x7fec0c0fc890 legacy=0x7fec0c0fcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:17.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.785+0000 7fec1167a700 1 -- 172.21.15.163:0/4274781884 shutdown_connections 2024-02-19T21:16:17.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.785+0000 7fec1167a700 1 -- 172.21.15.163:0/4274781884 >> 172.21.15.163:0/4274781884 conn(0x7fec0c0f2470 msgr2=0x7fec0c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:17.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.786+0000 7fec1167a700 1 -- 172.21.15.163:0/4274781884 shutdown_connections 2024-02-19T21:16:17.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.786+0000 7fec1167a700 1 -- 172.21.15.163:0/4274781884 wait complete. 2024-02-19T21:16:17.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.786+0000 7fec1167a700 1 librados: starting msgr at 2024-02-19T21:16:17.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.786+0000 7fec1167a700 1 librados: starting objecter 2024-02-19T21:16:17.789 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.786+0000 7fec1167a700 1 Processor -- start 2024-02-19T21:16:17.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7fec1167a700 1 -- start start 2024-02-19T21:16:17.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7fec1167a700 1 librados: setting wanted keys 2024-02-19T21:16:17.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7fec1167a700 1 librados: calling monclient init 2024-02-19T21:16:17.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7fec1167a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fec0c1195d0 con 0x7fec0c0ff690 2024-02-19T21:16:17.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7fec1167a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fec0c11a7b0 con 0x7fec0c0fc890 2024-02-19T21:16:17.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7fec1167a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fec0c11b9b0 con 0x7fec0c1031d0 2024-02-19T21:16:17.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7fec0a7fc700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fec0c0ff690 0x7fec0c0f8b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:45954/0 (socket says 172.21.15.163:45954) 2024-02-19T21:16:17.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7fec0a7fc700 1 -- 172.21.15.163:0/1041660954 learned_addr learned my addr 172.21.15.163:0/1041660954 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:17.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7fec0affd700 1 --1- 172.21.15.163:0/1041660954 >> v1:172.21.15.193:6789/0 conn(0x7fec0c0fc890 0x7fec0c0f8420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:59918/0 (socket says 172.21.15.163:59918) 2024-02-19T21:16:17.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7febf3fff700 10 client.?.objecter ms_handle_connect 0x7fec0c0ff690 2024-02-19T21:16:17.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7febf3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:17.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7febf3fff700 10 client.?.objecter ms_handle_connect 0x7fec0c0fc890 2024-02-19T21:16:17.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7febf3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:17.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7febf3fff700 10 client.?.objecter ms_handle_connect 0x7fec0c1031d0 2024-02-19T21:16:17.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7febf3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:17.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7febf3fff700 1 -- 172.21.15.163:0/1041660954 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449532299 0 0) 0x7fec0c11a7b0 con 0x7fec0c0fc890 2024-02-19T21:16:17.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.787+0000 7febf3fff700 1 -- 172.21.15.163:0/1041660954 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febf4003f60 con 0x7fec0c0fc890 2024-02-19T21:16:17.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.788+0000 7febf3fff700 1 -- 172.21.15.163:0/1041660954 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2082025215 0 0) 0x7febf4003f60 con 0x7fec0c0fc890 2024-02-19T21:16:17.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.788+0000 7febf3fff700 1 -- 172.21.15.163:0/1041660954 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec0c11a7b0 con 0x7fec0c0fc890 2024-02-19T21:16:17.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.788+0000 7febf3fff700 1 -- 172.21.15.163:0/1041660954 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7febfc002d40 con 0x7fec0c0fc890 2024-02-19T21:16:17.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.788+0000 7febf3fff700 1 -- 172.21.15.163:0/1041660954 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3619500565 0 0) 0x7fec0c11a7b0 con 0x7fec0c0fc890 2024-02-19T21:16:17.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.788+0000 7febf3fff700 1 -- 172.21.15.163:0/1041660954 >> v1:172.21.15.163:6790/0 conn(0x7fec0c1031d0 legacy=0x7fec0c117c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:17.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.788+0000 7febf3fff700 1 -- 172.21.15.163:0/1041660954 >> v1:172.21.15.163:6789/0 conn(0x7fec0c0ff690 legacy=0x7fec0c0f8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:17.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.788+0000 7febf3fff700 1 -- 172.21.15.163:0/1041660954 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c11cbb0 con 0x7fec0c0fc890 2024-02-19T21:16:17.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.788+0000 7fec1167a700 1 -- 172.21.15.163:0/1041660954 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec0c11a9a0 con 0x7fec0c0fc890 2024-02-19T21:16:17.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.788+0000 7fec1167a700 10 client.49321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:17.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.788+0000 7fec1167a700 1 -- 172.21.15.163:0/1041660954 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec0c11ae50 con 0x7fec0c0fc890 2024-02-19T21:16:17.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.788+0000 7fec1167a700 1 librados: init done 2024-02-19T21:16:17.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.788+0000 7febf3fff700 1 -- 172.21.15.163:0/1041660954 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7febfc001a80 con 0x7fec0c0fc890 2024-02-19T21:16:17.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.788+0000 7febf3fff700 1 -- 172.21.15.163:0/1041660954 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7febfc003920 con 0x7fec0c0fc890 2024-02-19T21:16:17.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.789+0000 7febf3fff700 1 -- 172.21.15.163:0/1041660954 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7febfc003aa0 con 0x7fec0c0fc890 2024-02-19T21:16:17.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.790+0000 7febf3fff700 1 -- 172.21.15.163:0/1041660954 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(728..728 src has 623..728) v4 ==== 8670+0+0 (unknown 2210538086 0 0) 0x7febfc079240 con 0x7fec0c0fc890 2024-02-19T21:16:17.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.790+0000 7febf3fff700 10 client.49321.objecter ms_dispatch 0x7fec0c001550 osd_map(728..728 src has 623..728) v4 2024-02-19T21:16:17.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.790+0000 7fec1167a700 1 -- 172.21.15.163:0/1041660954 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf8004f20 con 0x7fec0c0fc890 2024-02-19T21:16:17.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.790+0000 7febf3fff700 3 client.49321.objecter handle_osd_map got epochs [728,728] > 0 2024-02-19T21:16:17.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.790+0000 7febf3fff700 3 client.49321.objecter handle_osd_map decoding full epoch 728 2024-02-19T21:16:17.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.790+0000 7febf3fff700 20 client.49321.objecter dump_active .. 0 homeless 2024-02-19T21:16:17.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.790+0000 7febf3fff700 10 client.49321.objecter ms_handle_connect 0x7febf4063c30 2024-02-19T21:16:17.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:17.795+0000 7febf3fff700 1 -- 172.21.15.163:0/1041660954 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7febfc04d360 con 0x7fec0c0fc890 2024-02-19T21:16:17.821 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:17.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:17.849 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:17.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:17.867 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:17.874 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:17.935 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:16:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:16:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:16:17.936 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:16.009706+0000 osd.2 (osd.2) 286 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:16:17.936 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:16.011185+0000 osd.2 (osd.2) 287 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:16:17.936 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:16.506239+0000 mon.a (mon.0) 3498 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2024-02-19T21:16:17.936 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:17.054539+0000 osd.3 (osd.3) 303 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:16:17.936 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:17.055465+0000 osd.3 (osd.3) 304 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:16:17.936 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:17.213772+0000 osd.4 (osd.4) 257 : cluster [DBG] 4.c scrub starts 2024-02-19T21:16:17.936 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:17.215094+0000 osd.4 (osd.4) 258 : cluster [DBG] 4.c scrub ok 2024-02-19T21:16:17.936 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:17.505903+0000 mon.a (mon.0) 3499 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2024-02-19T21:16:17.936 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:17.514012+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:16:17.951 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:17.980 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:18.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:18.014+0000 7fec1167a700 1 -- 172.21.15.163:0/1041660954 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7febf80051d0 con 0x7fec0c0fc890 2024-02-19T21:16:18.103 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:16.009706+0000 osd.2 (osd.2) 286 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:16:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:16.011185+0000 osd.2 (osd.2) 287 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:16:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:16.506239+0000 mon.a (mon.0) 3498 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2024-02-19T21:16:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:17.054539+0000 osd.3 (osd.3) 303 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:16:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:17.055465+0000 osd.3 (osd.3) 304 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:16:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:17.213772+0000 osd.4 (osd.4) 257 : cluster [DBG] 4.c scrub starts 2024-02-19T21:16:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:17.215094+0000 osd.4 (osd.4) 258 : cluster [DBG] 4.c scrub ok 2024-02-19T21:16:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:17.505903+0000 mon.a (mon.0) 3499 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2024-02-19T21:16:18.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:17.514012+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:16:18.106 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:16.009706+0000 osd.2 (osd.2) 286 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:16:18.106 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:16.011185+0000 osd.2 (osd.2) 287 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:16:18.106 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:16.506239+0000 mon.a (mon.0) 3498 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2024-02-19T21:16:18.106 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:17.054539+0000 osd.3 (osd.3) 303 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:16:18.106 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:17.055465+0000 osd.3 (osd.3) 304 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:16:18.106 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:17.213772+0000 osd.4 (osd.4) 257 : cluster [DBG] 4.c scrub starts 2024-02-19T21:16:18.106 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:17.215094+0000 osd.4 (osd.4) 258 : cluster [DBG] 4.c scrub ok 2024-02-19T21:16:18.106 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:17.505903+0000 mon.a (mon.0) 3499 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2024-02-19T21:16:18.106 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:17.514012+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:16:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:18.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:18.208 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:18.238 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:18.336 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:18.339 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:18.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:18.440 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:18.453 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:16:18.454 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:16:18.468 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:18.488 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:18.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:18.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:18.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:18.508+0000 7febf3fff700 1 -- 172.21.15.163:0/1041660954 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v729) v1 ==== 101+0+0 (unknown 3122202446 0 0) 0x7febfc002850 con 0x7fec0c0fc890 2024-02-19T21:16:18.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:18.514+0000 7fec1167a700 10 librados: watch_flush enter 2024-02-19T21:16:18.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:18.514+0000 7fec1167a700 10 librados: watch_flush exit 2024-02-19T21:16:18.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:18.514+0000 7fec1167a700 20 client.49321.objecter shutdown clearing up homeless session... 2024-02-19T21:16:18.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:18.514+0000 7fec1167a700 10 client.49321.objecter successfully canceled tick 2024-02-19T21:16:18.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:18.514+0000 7fec1167a700 1 -- 172.21.15.163:0/1041660954 >> v1:172.21.15.163:6800/1552514681 conn(0x7febf4063c30 legacy=0x7febf40660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:18.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:18.515+0000 7fec1167a700 1 -- 172.21.15.163:0/1041660954 >> v1:172.21.15.193:6789/0 conn(0x7fec0c0fc890 legacy=0x7fec0c0f8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:18.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:18.515+0000 7fec1167a700 1 -- 172.21.15.163:0/1041660954 shutdown_connections 2024-02-19T21:16:18.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:18.515+0000 7fec1167a700 1 -- 172.21.15.163:0/1041660954 >> 172.21.15.163:0/1041660954 conn(0x7fec0c0f2470 msgr2=0x7fec0c103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:18.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:18.515+0000 7fec1167a700 1 -- 172.21.15.163:0/1041660954 shutdown_connections 2024-02-19T21:16:18.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:18.515+0000 7fec1167a700 1 -- 172.21.15.163:0/1041660954 wait complete. 2024-02-19T21:16:18.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:18.515+0000 7fec1167a700 1 librados: shutdown 2024-02-19T21:16:18.517 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:16:18.591 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:18.618 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:18.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:17.318386+0000 mgr.y (mgr.14550) 610 : cluster [DBG] pgmap v1097: 182 pgs: 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:18.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:17.837491+0000 osd.7 (osd.7) 327 : cluster [DBG] 2.26 scrub starts 2024-02-19T21:16:18.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:17.840422+0000 osd.7 (osd.7) 328 : cluster [DBG] 2.26 scrub ok 2024-02-19T21:16:18.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:18.016298+0000 mon.b (mon.1) 748 : audit [INF] from='client.? v1:172.21.15.163:0/1041660954' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:16:18.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:18.016702+0000 mon.a (mon.0) 3501 : audit [INF] from='client.49321 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:16:19.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:18.078504+0000 osd.5 (osd.5) 298 : cluster [DBG] 234.3 scrub starts 2024-02-19T21:16:19.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:18.081973+0000 osd.5 (osd.5) 299 : cluster [DBG] 234.3 scrub ok 2024-02-19T21:16:19.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:18.094083+0000 osd.3 (osd.3) 305 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:16:19.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:18.094858+0000 osd.3 (osd.3) 306 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:16:19.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:18.178355+0000 osd.4 (osd.4) 259 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:16:19.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:18.179366+0000 osd.4 (osd.4) 260 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:16:19.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:18.502811+0000 mon.a (mon.0) 3502 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:16:19.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:18.506823+0000 mon.a (mon.0) 3503 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:16:19.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:18.506938+0000 mon.a (mon.0) 3504 : audit [INF] from='client.49321 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:16:19.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:18.507005+0000 mon.a (mon.0) 3505 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2024-02-19T21:16:19.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:18.508954+0000 mon.a (mon.0) 3506 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:16:19.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:18.546950+0000 osd.6 (osd.6) 253 : cluster [DBG] 2.24 scrub starts 2024-02-19T21:16:19.002 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:18.547732+0000 osd.6 (osd.6) 254 : cluster [DBG] 2.24 scrub ok 2024-02-19T21:16:19.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:17.318386+0000 mgr.y (mgr.14550) 610 : cluster [DBG] pgmap v1097: 182 pgs: 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:19.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:17.837491+0000 osd.7 (osd.7) 327 : cluster [DBG] 2.26 scrub starts 2024-02-19T21:16:19.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:17.840422+0000 osd.7 (osd.7) 328 : cluster [DBG] 2.26 scrub ok 2024-02-19T21:16:19.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:18.016298+0000 mon.b (mon.1) 748 : audit [INF] from='client.? v1:172.21.15.163:0/1041660954' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:16:19.003 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:18.016702+0000 mon.a (mon.0) 3501 : audit [INF] from='client.49321 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:16:19.003 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:18.078504+0000 osd.5 (osd.5) 298 : cluster [DBG] 234.3 scrub starts 2024-02-19T21:16:19.003 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:18.081973+0000 osd.5 (osd.5) 299 : cluster [DBG] 234.3 scrub ok 2024-02-19T21:16:19.003 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:18.094083+0000 osd.3 (osd.3) 305 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:16:19.003 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:18.094858+0000 osd.3 (osd.3) 306 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:16:19.003 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:18.178355+0000 osd.4 (osd.4) 259 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:16:19.003 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:18.179366+0000 osd.4 (osd.4) 260 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:16:19.004 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:18.502811+0000 mon.a (mon.0) 3502 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:16:19.004 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:18.506823+0000 mon.a (mon.0) 3503 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:16:19.004 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:18.506938+0000 mon.a (mon.0) 3504 : audit [INF] from='client.49321 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:16:19.004 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:18.507005+0000 mon.a (mon.0) 3505 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2024-02-19T21:16:19.004 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:18.508954+0000 mon.a (mon.0) 3506 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:16:19.004 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:18.546950+0000 osd.6 (osd.6) 253 : cluster [DBG] 2.24 scrub starts 2024-02-19T21:16:19.005 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:18.547732+0000 osd.6 (osd.6) 254 : cluster [DBG] 2.24 scrub ok 2024-02-19T21:16:19.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:17.318386+0000 mgr.y (mgr.14550) 610 : cluster [DBG] pgmap v1097: 182 pgs: 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:19.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:17.837491+0000 osd.7 (osd.7) 327 : cluster [DBG] 2.26 scrub starts 2024-02-19T21:16:19.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:17.840422+0000 osd.7 (osd.7) 328 : cluster [DBG] 2.26 scrub ok 2024-02-19T21:16:19.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:18.016298+0000 mon.b (mon.1) 748 : audit [INF] from='client.? v1:172.21.15.163:0/1041660954' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:16:19.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:18.016702+0000 mon.a (mon.0) 3501 : audit [INF] from='client.49321 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:16:19.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:18.078504+0000 osd.5 (osd.5) 298 : cluster [DBG] 234.3 scrub starts 2024-02-19T21:16:19.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:18.081973+0000 osd.5 (osd.5) 299 : cluster [DBG] 234.3 scrub ok 2024-02-19T21:16:19.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:18.094083+0000 osd.3 (osd.3) 305 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:16:19.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:18.094858+0000 osd.3 (osd.3) 306 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:16:19.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:18.178355+0000 osd.4 (osd.4) 259 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:16:19.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:18.179366+0000 osd.4 (osd.4) 260 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:16:19.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:18.502811+0000 mon.a (mon.0) 3502 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:16:19.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:18.506823+0000 mon.a (mon.0) 3503 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:16:19.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:18.506938+0000 mon.a (mon.0) 3504 : audit [INF] from='client.49321 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:16:19.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:18.507005+0000 mon.a (mon.0) 3505 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2024-02-19T21:16:19.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:18.508954+0000 mon.a (mon.0) 3506 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:16:19.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:18.546950+0000 osd.6 (osd.6) 253 : cluster [DBG] 2.24 scrub starts 2024-02-19T21:16:19.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:18.547732+0000 osd.6 (osd.6) 254 : cluster [DBG] 2.24 scrub ok 2024-02-19T21:16:19.167 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:19.170 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:19.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:19.187 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-19T21:16:19.271 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:19.303 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:19.496 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:16:19.687 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:19.690 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:19.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:19.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:19.819 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:20.168 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:20.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:20.273 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:20.303 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:20.426 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:18.854529+0000 osd.7 (osd.7) 329 : cluster [DBG] 6.8 deep-scrub starts 2024-02-19T21:16:20.426 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:18.857420+0000 osd.7 (osd.7) 330 : cluster [DBG] 6.8 deep-scrub ok 2024-02-19T21:16:20.426 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:19.318763+0000 mgr.y (mgr.14550) 611 : cluster [DBG] pgmap v1100: 214 pgs: 32 unknown, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:16:20.426 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:19.397727+0000 mon.a (mon.0) 3507 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-130", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:16:20.426 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:19.397788+0000 mon.a (mon.0) 3508 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2024-02-19T21:16:20.426 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:19.399905+0000 mon.a (mon.0) 3509 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_count","val": "3"}]: dispatch 2024-02-19T21:16:20.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:20.213857+0000 osd.4 (osd.4) 261 : cluster [DBG] 5.e scrub starts 2024-02-19T21:16:20.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:20.215743+0000 osd.4 (osd.4) 262 : cluster [DBG] 5.e scrub ok 2024-02-19T21:16:20.427 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:18.854529+0000 osd.7 (osd.7) 329 : cluster [DBG] 6.8 deep-scrub starts 2024-02-19T21:16:20.427 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:18.857420+0000 osd.7 (osd.7) 330 : cluster [DBG] 6.8 deep-scrub ok 2024-02-19T21:16:20.427 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:19.318763+0000 mgr.y (mgr.14550) 611 : cluster [DBG] pgmap v1100: 214 pgs: 32 unknown, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:16:20.427 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:19.397727+0000 mon.a (mon.0) 3507 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-130", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:16:20.427 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:19.397788+0000 mon.a (mon.0) 3508 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2024-02-19T21:16:20.427 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:19.399905+0000 mon.a (mon.0) 3509 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_count","val": "3"}]: dispatch 2024-02-19T21:16:20.427 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:20.213857+0000 osd.4 (osd.4) 261 : cluster [DBG] 5.e scrub starts 2024-02-19T21:16:20.427 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:20.215743+0000 osd.4 (osd.4) 262 : cluster [DBG] 5.e scrub ok 2024-02-19T21:16:20.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:18.854529+0000 osd.7 (osd.7) 329 : cluster [DBG] 6.8 deep-scrub starts 2024-02-19T21:16:20.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:18.857420+0000 osd.7 (osd.7) 330 : cluster [DBG] 6.8 deep-scrub ok 2024-02-19T21:16:20.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:19.318763+0000 mgr.y (mgr.14550) 611 : cluster [DBG] pgmap v1100: 214 pgs: 32 unknown, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-02-19T21:16:20.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:19.397727+0000 mon.a (mon.0) 3507 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-130", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:16:20.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:19.397788+0000 mon.a (mon.0) 3508 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2024-02-19T21:16:20.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:19.399905+0000 mon.a (mon.0) 3509 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_count","val": "3"}]: dispatch 2024-02-19T21:16:20.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:20.213857+0000 osd.4 (osd.4) 261 : cluster [DBG] 5.e scrub starts 2024-02-19T21:16:20.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:20.215743+0000 osd.4 (osd.4) 262 : cluster [DBG] 5.e scrub ok 2024-02-19T21:16:20.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.842+0000 7f04a27e2700 1 -- 172.21.15.163:0/816549506 >> v1:172.21.15.163:6789/0 conn(0x7f049c1042b0 legacy=0x7f049c106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:20.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.842+0000 7f04a27e2700 1 -- 172.21.15.163:0/816549506 shutdown_connections 2024-02-19T21:16:20.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.842+0000 7f04a27e2700 1 -- 172.21.15.163:0/816549506 >> 172.21.15.163:0/816549506 conn(0x7f049c0f2470 msgr2=0x7f049c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:20.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.842+0000 7f04a27e2700 1 -- 172.21.15.163:0/816549506 shutdown_connections 2024-02-19T21:16:20.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.842+0000 7f04a27e2700 1 -- 172.21.15.163:0/816549506 wait complete. 2024-02-19T21:16:20.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.842+0000 7f04a27e2700 1 librados: starting msgr at 2024-02-19T21:16:20.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.842+0000 7f04a27e2700 1 librados: starting objecter 2024-02-19T21:16:20.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.842+0000 7f04a27e2700 1 Processor -- start 2024-02-19T21:16:20.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.842+0000 7f04a27e2700 1 -- start start 2024-02-19T21:16:20.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.842+0000 7f04a27e2700 1 librados: setting wanted keys 2024-02-19T21:16:20.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.842+0000 7f04a27e2700 1 librados: calling monclient init 2024-02-19T21:16:20.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.843+0000 7f04a27e2700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f049c102a80 con 0x7f049c0f8920 2024-02-19T21:16:20.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.843+0000 7f04a27e2700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f049c102c50 con 0x7f049c0f94d0 2024-02-19T21:16:20.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.843+0000 7f04a27e2700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f049c102e40 con 0x7f049c1042b0 2024-02-19T21:16:20.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.843+0000 7f049bfff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f049c0f8920 0x7f049c1022e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:45964/0 (socket says 172.21.15.163:45964) 2024-02-19T21:16:20.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.843+0000 7f049bfff700 1 -- 172.21.15.163:0/2224134398 learned_addr learned my addr 172.21.15.163:0/2224134398 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:20.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.843+0000 7f04997fa700 10 client.?.objecter ms_handle_connect 0x7f049c0f8920 2024-02-19T21:16:20.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.843+0000 7f04997fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:20.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.843+0000 7f04997fa700 10 client.?.objecter ms_handle_connect 0x7f049c1042b0 2024-02-19T21:16:20.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.843+0000 7f04997fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:20.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.843+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783578381 0 0) 0x7f049c102a80 con 0x7f049c0f8920 2024-02-19T21:16:20.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.843+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04840033f0 con 0x7f049c0f8920 2024-02-19T21:16:20.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.843+0000 7f04997fa700 10 client.?.objecter ms_handle_connect 0x7f049c0f94d0 2024-02-19T21:16:20.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.843+0000 7f04997fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:20.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.843+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3747605957 0 0) 0x7f049c102e40 con 0x7f049c1042b0 2024-02-19T21:16:20.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.843+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f049c102a80 con 0x7f049c1042b0 2024-02-19T21:16:20.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782616051 0 0) 0x7f049c102c50 con 0x7f049c0f94d0 2024-02-19T21:16:20.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f049c102e40 con 0x7f049c0f94d0 2024-02-19T21:16:20.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3019064057 0 0) 0x7f04840033f0 con 0x7f049c0f8920 2024-02-19T21:16:20.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f049c102c50 con 0x7f049c0f8920 2024-02-19T21:16:20.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f048c001980 con 0x7f049c0f8920 2024-02-19T21:16:20.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 502228040 0 0) 0x7f049c102a80 con 0x7f049c1042b0 2024-02-19T21:16:20.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04840033f0 con 0x7f049c1042b0 2024-02-19T21:16:20.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0494002240 con 0x7f049c1042b0 2024-02-19T21:16:20.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3309483321 0 0) 0x7f049c102e40 con 0x7f049c0f94d0 2024-02-19T21:16:20.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f049c102a80 con 0x7f049c0f94d0 2024-02-19T21:16:20.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f04900019d0 con 0x7f049c0f94d0 2024-02-19T21:16:20.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2815499298 0 0) 0x7f049c102c50 con 0x7f049c0f8920 2024-02-19T21:16:20.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 >> v1:172.21.15.163:6790/0 conn(0x7f049c1042b0 legacy=0x7f049c118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:20.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 >> v1:172.21.15.193:6789/0 conn(0x7f049c0f94d0 legacy=0x7f049c114d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:20.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c11cb80 con 0x7f049c0f8920 2024-02-19T21:16:20.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049c119bb0 con 0x7f049c0f8920 2024-02-19T21:16:20.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04a27e2700 10 client.23784.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:20.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f049c11a020 con 0x7f049c0f8920 2024-02-19T21:16:20.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04a27e2700 1 librados: init done 2024-02-19T21:16:20.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f048c002d00 con 0x7f049c0f8920 2024-02-19T21:16:20.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.844+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f048c005b20 con 0x7f049c0f8920 2024-02-19T21:16:20.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.845+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f0488004f00 con 0x7f049c0f8920 2024-02-19T21:16:20.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.845+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f048c001ea0 con 0x7f049c0f8920 2024-02-19T21:16:20.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(731..731 src has 623..731) v4 ==== 8678+0+0 (unknown 2539968470 0 0) 0x7f048c078f70 con 0x7f049c0f8920 2024-02-19T21:16:20.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 osd_map(731..731 src has 623..731) v4 2024-02-19T21:16:20.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04997fa700 3 client.23784.objecter handle_osd_map got epochs [731,731] > 0 2024-02-19T21:16:20.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04997fa700 3 client.23784.objecter handle_osd_map decoding full epoch 731 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04997fa700 20 client.23784.objecter dump_active .. 0 homeless 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v731) v1 ==== 54+0+16 (unknown 3703141071 0 2420000192) 0x7f048c07b380 con 0x7f049c0f8920 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04997fa700 10 client.23784.objecter ms_handle_connect 0x7f0484063c60 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04a27e2700 10 client.23784.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 1 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f04880051a0 osd=0 3 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f04880051a0 osd=0 3 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f04880051a0 osd=0 3 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 1 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f04880051a0 osd=0 3 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 0 1 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f04880051a0 osd=0 4 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04a27e2700 10 client.23784.objecter _send_command 1 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.846+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0488004950 con 0x7f0488004530 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.847+0000 7f04997fa700 10 client.23784.objecter ms_handle_connect 0x7f0488004530 2024-02-19T21:16:20.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.847+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0488004950 con 0x7f0488004530 2024-02-19T21:16:20.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.847+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:16:20.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.847+0000 7f04997fa700 10 client.23784.objecter _finish_command 1 = system:0 2024-02-19T21:16:20.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.847+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f04880051a0 osd=0 4 2024-02-19T21:16:20.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.847+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 0 1 2024-02-19T21:16:20.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.862+0000 7f04a27e2700 10 client.23784.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:20.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.862+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 2 2024-02-19T21:16:20.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.862+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f04880051a0 osd=0 3 2024-02-19T21:16:20.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.862+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f04880051a0 osd=0 3 2024-02-19T21:16:20.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.862+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f04880051a0 osd=0 3 2024-02-19T21:16:20.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.862+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 2 2024-02-19T21:16:20.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.862+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f04880051a0 osd=0 3 2024-02-19T21:16:20.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.862+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 0 2 2024-02-19T21:16:20.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.862+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f04880051a0 osd=0 4 2024-02-19T21:16:20.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.862+0000 7f04a27e2700 10 client.23784.objecter _send_command 2 2024-02-19T21:16:20.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.862+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0488004d30 con 0x7f0488004530 2024-02-19T21:16:20.954 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:16:20.954 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:16:20.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f0488004d30 con 0x7f0488004530 2024-02-19T21:16:20.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:20.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04997fa700 10 client.23784.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:20.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f04880051a0 osd=0 4 2024-02-19T21:16:20.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 0 2 2024-02-19T21:16:20.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04a27e2700 10 client.23784.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:20.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 3 2024-02-19T21:16:20.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488003fe0 osd=1 3 2024-02-19T21:16:20.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488003fe0 osd=1 3 2024-02-19T21:16:20.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488003fe0 osd=1 3 2024-02-19T21:16:20.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 3 2024-02-19T21:16:20.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f0488003fe0 osd=1 3 2024-02-19T21:16:20.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 1 3 2024-02-19T21:16:20.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488003fe0 osd=1 4 2024-02-19T21:16:20.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04a27e2700 10 client.23784.objecter _send_command 3 2024-02-19T21:16:20.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f048800c000 con 0x7f0488004150 2024-02-19T21:16:20.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.864+0000 7f04997fa700 10 client.23784.objecter ms_handle_connect 0x7f0488004150 2024-02-19T21:16:20.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.865+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f048800c000 con 0x7f0488004150 2024-02-19T21:16:20.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.865+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:16:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.865+0000 7f04997fa700 10 client.23784.objecter _finish_command 3 = system:0 2024-02-19T21:16:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.865+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f0488003fe0 osd=1 4 2024-02-19T21:16:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.865+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 1 3 2024-02-19T21:16:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.880+0000 7f04a27e2700 10 client.23784.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.880+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 4 2024-02-19T21:16:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.880+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488003fe0 osd=1 3 2024-02-19T21:16:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.880+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488003fe0 osd=1 3 2024-02-19T21:16:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.880+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488003fe0 osd=1 3 2024-02-19T21:16:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.880+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 4 2024-02-19T21:16:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.880+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f0488003fe0 osd=1 3 2024-02-19T21:16:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.880+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 1 4 2024-02-19T21:16:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.880+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488003fe0 osd=1 4 2024-02-19T21:16:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.880+0000 7f04a27e2700 10 client.23784.objecter _send_command 4 2024-02-19T21:16:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.880+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f04880018b0 con 0x7f0488004150 2024-02-19T21:16:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f04880018b0 con 0x7f0488004150 2024-02-19T21:16:20.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:20.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04997fa700 10 client.23784.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:20.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f0488003fe0 osd=1 4 2024-02-19T21:16:20.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 1 4 2024-02-19T21:16:20.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04a27e2700 10 client.23784.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:20.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 5 2024-02-19T21:16:20.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488003630 osd=2 3 2024-02-19T21:16:20.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488003630 osd=2 3 2024-02-19T21:16:20.957 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:16:20.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488003630 osd=2 3 2024-02-19T21:16:20.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 5 2024-02-19T21:16:20.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f0488003630 osd=2 3 2024-02-19T21:16:20.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 2 5 2024-02-19T21:16:20.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488003630 osd=2 4 2024-02-19T21:16:20.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04a27e2700 10 client.23784.objecter _send_command 5 2024-02-19T21:16:20.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0488003e70 con 0x7f048800d2f0 2024-02-19T21:16:20.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.882+0000 7f04997fa700 10 client.23784.objecter ms_handle_connect 0x7f048800d2f0 2024-02-19T21:16:20.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.883+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0488003e70 con 0x7f048800d2f0 2024-02-19T21:16:20.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.883+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:16:20.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.883+0000 7f04997fa700 10 client.23784.objecter _finish_command 5 = system:0 2024-02-19T21:16:20.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.883+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f0488003630 osd=2 4 2024-02-19T21:16:20.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.883+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 2 5 2024-02-19T21:16:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.897+0000 7f04a27e2700 10 client.23784.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.897+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 6 2024-02-19T21:16:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.897+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488003630 osd=2 3 2024-02-19T21:16:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.897+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488003630 osd=2 3 2024-02-19T21:16:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.897+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488003630 osd=2 3 2024-02-19T21:16:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.897+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 6 2024-02-19T21:16:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.897+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f0488003630 osd=2 3 2024-02-19T21:16:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.897+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 2 6 2024-02-19T21:16:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.897+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488003630 osd=2 4 2024-02-19T21:16:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.897+0000 7f04a27e2700 10 client.23784.objecter _send_command 6 2024-02-19T21:16:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.897+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f048800e710 con 0x7f048800d2f0 2024-02-19T21:16:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f048800e710 con 0x7f048800d2f0 2024-02-19T21:16:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04997fa700 10 client.23784.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:20.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f0488003630 osd=2 4 2024-02-19T21:16:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 2 6 2024-02-19T21:16:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04a27e2700 10 client.23784.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 7 2024-02-19T21:16:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f04880033f0 osd=3 3 2024-02-19T21:16:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f04880033f0 osd=3 3 2024-02-19T21:16:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f04880033f0 osd=3 3 2024-02-19T21:16:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 7 2024-02-19T21:16:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f04880033f0 osd=3 3 2024-02-19T21:16:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 3 7 2024-02-19T21:16:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f04880033f0 osd=3 4 2024-02-19T21:16:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04a27e2700 10 client.23784.objecter _send_command 7 2024-02-19T21:16:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.899+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0488015ac0 con 0x7f0488011e80 2024-02-19T21:16:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.900+0000 7f04997fa700 10 client.23784.objecter ms_handle_connect 0x7f0488011e80 2024-02-19T21:16:20.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.900+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0488015ac0 con 0x7f0488011e80 2024-02-19T21:16:20.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.900+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:16:20.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.900+0000 7f04997fa700 10 client.23784.objecter _finish_command 7 = system:0 2024-02-19T21:16:20.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.900+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f04880033f0 osd=3 4 2024-02-19T21:16:20.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.900+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 3 7 2024-02-19T21:16:20.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.915+0000 7f04a27e2700 10 client.23784.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:20.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.915+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 8 2024-02-19T21:16:20.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.915+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f04880033f0 osd=3 3 2024-02-19T21:16:20.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.915+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f04880033f0 osd=3 3 2024-02-19T21:16:20.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.915+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f04880033f0 osd=3 3 2024-02-19T21:16:20.961 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:16:20.961 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:16:20.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.915+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 8 2024-02-19T21:16:20.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.915+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f04880033f0 osd=3 3 2024-02-19T21:16:20.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.915+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 3 8 2024-02-19T21:16:20.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.915+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f04880033f0 osd=3 4 2024-02-19T21:16:20.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.915+0000 7f04a27e2700 10 client.23784.objecter _send_command 8 2024-02-19T21:16:20.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.915+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f048800e180 con 0x7f0488011e80 2024-02-19T21:16:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f048800e180 con 0x7f0488011e80 2024-02-19T21:16:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04997fa700 10 client.23784.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f04880033f0 osd=3 4 2024-02-19T21:16:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 3 8 2024-02-19T21:16:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04a27e2700 10 client.23784.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 9 2024-02-19T21:16:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f048800df40 osd=4 3 2024-02-19T21:16:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f048800df40 osd=4 3 2024-02-19T21:16:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f048800df40 osd=4 3 2024-02-19T21:16:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 9 2024-02-19T21:16:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f048800df40 osd=4 3 2024-02-19T21:16:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 4 9 2024-02-19T21:16:20.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f048800df40 osd=4 4 2024-02-19T21:16:20.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04a27e2700 10 client.23784.objecter _send_command 9 2024-02-19T21:16:20.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.917+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f048800e0b0 con 0x7f048800ce50 2024-02-19T21:16:20.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.918+0000 7f04997fa700 10 client.23784.objecter ms_handle_connect 0x7f048800ce50 2024-02-19T21:16:20.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.918+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f048800e0b0 con 0x7f048800ce50 2024-02-19T21:16:20.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.918+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:16:20.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.918+0000 7f04997fa700 10 client.23784.objecter _finish_command 9 = system:0 2024-02-19T21:16:20.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.918+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f048800df40 osd=4 4 2024-02-19T21:16:20.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.918+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 4 9 2024-02-19T21:16:20.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.933+0000 7f04a27e2700 10 client.23784.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:20.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.933+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 10 2024-02-19T21:16:20.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.933+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f048800df40 osd=4 3 2024-02-19T21:16:20.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.933+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f048800df40 osd=4 3 2024-02-19T21:16:20.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.933+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f048800df40 osd=4 3 2024-02-19T21:16:20.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.933+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 10 2024-02-19T21:16:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.933+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f048800df40 osd=4 3 2024-02-19T21:16:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.933+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 4 10 2024-02-19T21:16:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.933+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f048800df40 osd=4 4 2024-02-19T21:16:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.933+0000 7f04a27e2700 10 client.23784.objecter _send_command 10 2024-02-19T21:16:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.933+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f048800e0b0 con 0x7f048800ce50 2024-02-19T21:16:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f048800e0b0 con 0x7f048800ce50 2024-02-19T21:16:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04997fa700 10 client.23784.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f048800df40 osd=4 4 2024-02-19T21:16:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 4 10 2024-02-19T21:16:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04a27e2700 10 client.23784.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 11 2024-02-19T21:16:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f048800e0b0 osd=5 3 2024-02-19T21:16:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f048800e0b0 osd=5 3 2024-02-19T21:16:20.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f048800e0b0 osd=5 3 2024-02-19T21:16:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 11 2024-02-19T21:16:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f048800e0b0 osd=5 3 2024-02-19T21:16:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 5 11 2024-02-19T21:16:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f048800e0b0 osd=5 4 2024-02-19T21:16:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04a27e2700 10 client.23784.objecter _send_command 11 2024-02-19T21:16:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.935+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0488016e10 con 0x7f0488016880 2024-02-19T21:16:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.936+0000 7f04997fa700 10 client.23784.objecter ms_handle_connect 0x7f0488016880 2024-02-19T21:16:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.936+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0488016e10 con 0x7f0488016880 2024-02-19T21:16:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.936+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:16:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.936+0000 7f04997fa700 10 client.23784.objecter _finish_command 11 = system:0 2024-02-19T21:16:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.936+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f048800e0b0 osd=5 4 2024-02-19T21:16:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.936+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 5 11 2024-02-19T21:16:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.951+0000 7f04a27e2700 10 client.23784.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.951+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 12 2024-02-19T21:16:20.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.951+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f048800e0b0 osd=5 3 2024-02-19T21:16:20.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.951+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f048800e0b0 osd=5 3 2024-02-19T21:16:20.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.952+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f048800e0b0 osd=5 3 2024-02-19T21:16:20.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.952+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 12 2024-02-19T21:16:20.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.952+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f048800e0b0 osd=5 3 2024-02-19T21:16:20.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.952+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 5 12 2024-02-19T21:16:20.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.952+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f048800e0b0 osd=5 4 2024-02-19T21:16:20.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.952+0000 7f04a27e2700 10 client.23784.objecter _send_command 12 2024-02-19T21:16:20.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.952+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0488016e10 con 0x7f0488016880 2024-02-19T21:16:20.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.953+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f0488016e10 con 0x7f0488016880 2024-02-19T21:16:20.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.953+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:20.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.953+0000 7f04997fa700 10 client.23784.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:20.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.953+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f048800e0b0 osd=5 4 2024-02-19T21:16:20.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.953+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 5 12 2024-02-19T21:16:20.968 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:16:20.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.954+0000 7f04a27e2700 10 client.23784.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:20.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.954+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 13 2024-02-19T21:16:20.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.954+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488016e10 osd=6 3 2024-02-19T21:16:20.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.954+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488016e10 osd=6 3 2024-02-19T21:16:20.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.954+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488016e10 osd=6 3 2024-02-19T21:16:20.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.954+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 13 2024-02-19T21:16:20.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.954+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f0488016e10 osd=6 3 2024-02-19T21:16:20.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.954+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 6 13 2024-02-19T21:16:20.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.954+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488016e10 osd=6 4 2024-02-19T21:16:20.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.954+0000 7f04a27e2700 10 client.23784.objecter _send_command 13 2024-02-19T21:16:20.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.954+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0488022bb0 con 0x7f048801f0a0 2024-02-19T21:16:20.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.954+0000 7f04997fa700 10 client.23784.objecter ms_handle_connect 0x7f048801f0a0 2024-02-19T21:16:20.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.955+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0488022bb0 con 0x7f048801f0a0 2024-02-19T21:16:20.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.955+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:16:20.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.955+0000 7f04997fa700 10 client.23784.objecter _finish_command 13 = system:0 2024-02-19T21:16:20.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.955+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f0488016e10 osd=6 4 2024-02-19T21:16:20.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.955+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 6 13 2024-02-19T21:16:20.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.969+0000 7f04a27e2700 10 client.23784.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:20.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.969+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 14 2024-02-19T21:16:20.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.969+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488016e10 osd=6 3 2024-02-19T21:16:20.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.969+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488016e10 osd=6 3 2024-02-19T21:16:20.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.969+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488016e10 osd=6 3 2024-02-19T21:16:20.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.969+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 14 2024-02-19T21:16:20.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.969+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f0488016e10 osd=6 3 2024-02-19T21:16:20.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.969+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 6 14 2024-02-19T21:16:20.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.969+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488016e10 osd=6 4 2024-02-19T21:16:20.970 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.969+0000 7f04a27e2700 10 client.23784.objecter _send_command 14 2024-02-19T21:16:20.971 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.969+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0488022bb0 con 0x7f048801f0a0 2024-02-19T21:16:20.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f0488022bb0 con 0x7f048801f0a0 2024-02-19T21:16:20.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:20.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04997fa700 10 client.23784.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:20.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f0488016e10 osd=6 4 2024-02-19T21:16:20.972 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 6 14 2024-02-19T21:16:20.973 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:16:20.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04a27e2700 10 client.23784.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:20.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 15 2024-02-19T21:16:20.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488022bb0 osd=7 3 2024-02-19T21:16:20.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488022bb0 osd=7 3 2024-02-19T21:16:20.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488022bb0 osd=7 3 2024-02-19T21:16:20.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 15 2024-02-19T21:16:20.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f0488022bb0 osd=7 3 2024-02-19T21:16:20.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 7 15 2024-02-19T21:16:20.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488022bb0 osd=7 4 2024-02-19T21:16:20.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04a27e2700 10 client.23784.objecter _send_command 15 2024-02-19T21:16:20.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.971+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0488023dd0 con 0x7f04880177c0 2024-02-19T21:16:20.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.972+0000 7f04997fa700 10 client.23784.objecter ms_handle_connect 0x7f04880177c0 2024-02-19T21:16:20.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.972+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0488023dd0 con 0x7f04880177c0 2024-02-19T21:16:20.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.972+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:16:20.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.972+0000 7f04997fa700 10 client.23784.objecter _finish_command 15 = system:0 2024-02-19T21:16:20.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.972+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f0488022bb0 osd=7 4 2024-02-19T21:16:20.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.972+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 7 15 2024-02-19T21:16:20.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.987+0000 7f04a27e2700 10 client.23784.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:20.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.987+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign -1 16 2024-02-19T21:16:20.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.987+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488022bb0 osd=7 3 2024-02-19T21:16:20.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.987+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488022bb0 osd=7 3 2024-02-19T21:16:20.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.987+0000 7f04a27e2700 20 client.23784.objecter _get_session s=0x7f0488022bb0 osd=7 3 2024-02-19T21:16:20.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.987+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_remove -1 16 2024-02-19T21:16:20.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.987+0000 7f04a27e2700 20 client.23784.objecter get_session s=0x7f0488022bb0 osd=7 3 2024-02-19T21:16:20.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.987+0000 7f04a27e2700 15 client.23784.objecter _session_command_op_assign 7 16 2024-02-19T21:16:20.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.987+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488022bb0 osd=7 4 2024-02-19T21:16:20.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.987+0000 7f04a27e2700 10 client.23784.objecter _send_command 16 2024-02-19T21:16:20.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.987+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f0488023dd0 con 0x7f04880177c0 2024-02-19T21:16:20.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04997fa700 1 -- 172.21.15.163:0/2224134398 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f0488023dd0 con 0x7f04880177c0 2024-02-19T21:16:20.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04997fa700 10 client.23784.objecter ms_dispatch 0x7f049c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:20.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04997fa700 10 client.23784.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:20.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04997fa700 20 client.23784.objecter put_session s=0x7f0488022bb0 osd=7 4 2024-02-19T21:16:20.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04997fa700 15 client.23784.objecter _session_command_op_remove 7 16 2024-02-19T21:16:20.990 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:16:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 10 librados: watch_flush enter 2024-02-19T21:16:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 10 librados: watch_flush exit 2024-02-19T21:16:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 10 client.23784.objecter close_session for osd.0 2024-02-19T21:16:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 >> v1:172.21.15.163:6801/1559418115 conn(0x7f0488004530 legacy=0x7f0488002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f04880051a0 osd=0 1 2024-02-19T21:16:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 10 client.23784.objecter close_session for osd.1 2024-02-19T21:16:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 >> v1:172.21.15.163:6805/3125047432 conn(0x7f0488004150 legacy=0x7f048800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488003fe0 osd=1 1 2024-02-19T21:16:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 10 client.23784.objecter close_session for osd.2 2024-02-19T21:16:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 >> v1:172.21.15.163:6809/1563158487 conn(0x7f048800d2f0 legacy=0x7f048800d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488003630 osd=2 1 2024-02-19T21:16:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 10 client.23784.objecter close_session for osd.3 2024-02-19T21:16:20.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 >> v1:172.21.15.163:6813/2112279769 conn(0x7f0488011e80 legacy=0x7f0488014270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a0d7f700 1 -- 172.21.15.163:0/2224134398 reap_dead start 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f04880033f0 osd=3 1 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 10 client.23784.objecter close_session for osd.4 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 >> v1:172.21.15.193:6800/3129716965 conn(0x7f048800ce50 legacy=0x7f0488015c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a0d7f700 1 -- 172.21.15.163:0/2224134398 reap_dead start 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f048800df40 osd=4 1 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 10 client.23784.objecter close_session for osd.5 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 >> v1:172.21.15.193:6804/805161991 conn(0x7f0488016880 legacy=0x7f048801d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f048800e0b0 osd=5 1 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 10 client.23784.objecter close_session for osd.6 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 >> v1:172.21.15.193:6808/787604615 conn(0x7f048801f0a0 legacy=0x7f0488021490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488016e10 osd=6 1 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 10 client.23784.objecter close_session for osd.7 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 >> v1:172.21.15.193:6812/157927432 conn(0x7f04880177c0 legacy=0x7f0488017c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 20 client.23784.objecter put_session s=0x7f0488022bb0 osd=7 1 2024-02-19T21:16:20.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 20 client.23784.objecter shutdown clearing up homeless session... 2024-02-19T21:16:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 10 client.23784.objecter successfully canceled tick 2024-02-19T21:16:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 >> v1:172.21.15.163:6800/1552514681 conn(0x7f0484063c60 legacy=0x7f0484066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 >> v1:172.21.15.163:6789/0 conn(0x7f049c0f8920 legacy=0x7f049c1022e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a0d7f700 1 -- 172.21.15.163:0/2224134398 reap_dead start 2024-02-19T21:16:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a0d7f700 1 -- 172.21.15.163:0/2224134398 reap_dead start 2024-02-19T21:16:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 shutdown_connections 2024-02-19T21:16:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 >> 172.21.15.163:0/2224134398 conn(0x7f049c0f2470 msgr2=0x7f049c103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 shutdown_connections 2024-02-19T21:16:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 1 -- 172.21.15.163:0/2224134398 wait complete. 2024-02-19T21:16:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:20.989+0000 7f04a27e2700 1 librados: shutdown 2024-02-19T21:16:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:16:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:16:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:16:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:16:20.994 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:16:20.995 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:16:20.995 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:16:20.995 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:16:21.048 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:16:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:16:21.052 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:21.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:21.157 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:21.187 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:21.300 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:16:21.300 INFO:teuthology.orchestra.run.smithi193.stdout: "size": 20, 2024-02-19T21:16:21.300 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 600, 2024-02-19T21:16:21.300 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [ 2024-02-19T21:16:21.301 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.301 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.49298.0:1 162.6s0 162:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e237)", 2024-02-19T21:16:21.301 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:06:57.212999+0000", 2024-02-19T21:16:21.301 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 564.08490849999998, 2024-02-19T21:16:21.301 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.009269615, 2024-02-19T21:16:21.301 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.301 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:21.301 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.301 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.49298", 2024-02-19T21:16:21.301 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2330262446", 2024-02-19T21:16:21.301 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:16:21.301 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.301 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.301 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.302 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.302 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.212999+0000", 2024-02-19T21:16:21.302 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.302 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.302 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.302 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.302 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.212999+0000", 2024-02-19T21:16:21.302 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:16:21.302 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.302 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.302 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.302 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.212996+0000", 2024-02-19T21:16:21.303 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.08e-05 2024-02-19T21:16:21.303 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.303 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.303 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.303 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.213007+0000", 2024-02-19T21:16:21.303 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1018e-05 2024-02-19T21:16:21.303 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.303 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.303 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.303 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.213018+0000", 2024-02-19T21:16:21.303 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7819999999999994e-06 2024-02-19T21:16:21.303 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.303 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.303 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.304 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.213026+0000", 2024-02-19T21:16:21.304 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9827999999999996e-05 2024-02-19T21:16:21.304 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.304 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.304 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.304 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.213096+0000", 2024-02-19T21:16:21.304 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.399e-06 2024-02-19T21:16:21.304 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.304 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.304 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:16:21.304 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:57.213099+0000", 2024-02-19T21:16:21.304 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.006276073 2024-02-19T21:16:21.304 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.305 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.305 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.305 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.219375+0000", 2024-02-19T21:16:21.305 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00012834399999999999 2024-02-19T21:16:21.305 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.305 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.305 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.305 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.219504+0000", 2024-02-19T21:16:21.305 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00046405399999999999 2024-02-19T21:16:21.305 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.305 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.305 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:16:21.305 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.219968+0000", 2024-02-19T21:16:21.306 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00075405700000000003 2024-02-19T21:16:21.306 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.306 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.306 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:16:21.306 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.220722+0000", 2024-02-19T21:16:21.306 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0014760260000000001 2024-02-19T21:16:21.306 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.306 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.306 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:21.306 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.222198+0000", 2024-02-19T21:16:21.306 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1095999999999998e-05 2024-02-19T21:16:21.306 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.306 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.306 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:06:58.222269+0000", 2024-02-19T21:16:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.009269615 2024-02-19T21:16:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_repop(osd.2.18:69 61.3 e294/114 61:c0520479:test-rados-api-smithi163-83928-38::foo:head v 294'69, mlcod=230'66)", 2024-02-19T21:16:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:07:54.263507+0000", 2024-02-19T21:16:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 507.034400538, 2024-02-19T21:16:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0078694200000000002, 2024-02-19T21:16:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.308 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.308 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.263507+0000", 2024-02-19T21:16:21.308 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.308 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.308 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.308 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.263507+0000", 2024-02-19T21:16:21.308 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967296 2024-02-19T21:16:21.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.308 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.308 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.308 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.263507+0000", 2024-02-19T21:16:21.308 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.003686576 2024-02-19T21:16:21.308 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.309 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.309 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.309 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.267194+0000", 2024-02-19T21:16:21.309 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4829e-05 2024-02-19T21:16:21.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.309 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.309 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.309 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.267208+0000", 2024-02-19T21:16:21.309 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.0240000000000003e-06 2024-02-19T21:16:21.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.309 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.309 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.309 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.267217+0000", 2024-02-19T21:16:21.309 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.1632000000000003e-05 2024-02-19T21:16:21.309 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.310 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.310 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.310 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.267269+0000", 2024-02-19T21:16:21.310 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0282999999999999e-05 2024-02-19T21:16:21.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.310 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.310 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.310 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.267289+0000", 2024-02-19T21:16:21.310 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0040479779999999998 2024-02-19T21:16:21.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.310 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.310 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:21.310 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.271337+0000", 2024-02-19T21:16:21.310 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9335999999999997e-05 2024-02-19T21:16:21.310 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.311 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.311 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.311 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:07:54.271377+0000", 2024-02-19T21:16:21.311 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0078694200000000002 2024-02-19T21:16:21.311 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.311 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.311 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.311 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.311 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.311 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:4348 191.8 191:10fb1d02:::199:head [delete] snapc 0=[] ondisk+write+ignore_cache+known_if_redirected e314)", 2024-02-19T21:16:21.311 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:08:14.258045+0000", 2024-02-19T21:16:21.311 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 487.03986287800001, 2024-02-19T21:16:21.311 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.039591451999999999, 2024-02-19T21:16:21.311 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.312 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:21.312 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.312 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:16:21.312 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:16:21.312 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 4348 2024-02-19T21:16:21.312 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.312 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.312 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.312 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.312 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258045+0000", 2024-02-19T21:16:21.312 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.312 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.312 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.312 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.312 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258045+0000", 2024-02-19T21:16:21.313 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:16:21.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.313 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.313 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.313 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258043+0000", 2024-02-19T21:16:21.313 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5199999999999997e-06 2024-02-19T21:16:21.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.313 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.313 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.313 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258049+0000", 2024-02-19T21:16:21.313 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6630000000000005e-06 2024-02-19T21:16:21.313 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.313 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.313 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.313 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258059+0000", 2024-02-19T21:16:21.314 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0889999999999999e-06 2024-02-19T21:16:21.314 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.314 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.314 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.314 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258064+0000", 2024-02-19T21:16:21.314 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5559999999999998e-05 2024-02-19T21:16:21.314 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.314 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.314 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.314 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258099+0000", 2024-02-19T21:16:21.314 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.4418999999999994e-05 2024-02-19T21:16:21.314 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.314 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.314 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.315 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258164+0000", 2024-02-19T21:16:21.315 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.1835000000000006e-05 2024-02-19T21:16:21.315 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.315 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.315 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,5", 2024-02-19T21:16:21.315 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.258236+0000", 2024-02-19T21:16:21.315 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.021542664 2024-02-19T21:16:21.315 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.315 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.315 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:16:21.315 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.279778+0000", 2024-02-19T21:16:21.315 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025807920000000002 2024-02-19T21:16:21.315 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.316 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.316 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:16:21.316 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.282359+0000", 2024-02-19T21:16:21.316 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.015223703 2024-02-19T21:16:21.316 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.316 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.316 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:16:21.316 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297583+0000", 2024-02-19T21:16:21.316 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0100999999999999e-05 2024-02-19T21:16:21.316 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.316 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.316 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:21.316 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297613+0000", 2024-02-19T21:16:21.317 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3626e-05 2024-02-19T21:16:21.317 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.317 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.317 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.317 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297636+0000", 2024-02-19T21:16:21.317 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.039591451999999999 2024-02-19T21:16:21.317 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.317 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.317 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.317 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.317 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.317 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:4349 191.8 191:10fdda93:::194:head [delete] snapc 0=[] ondisk+write+ignore_cache+known_if_redirected e314)", 2024-02-19T21:16:21.317 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:08:14.297868+0000", 2024-02-19T21:16:21.317 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 487.00004000500002, 2024-02-19T21:16:21.318 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026989016000000001, 2024-02-19T21:16:21.318 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.318 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:21.318 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.318 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:16:21.318 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:16:21.318 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 4349 2024-02-19T21:16:21.318 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.318 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.318 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.318 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.318 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297868+0000", 2024-02-19T21:16:21.318 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.318 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.318 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.319 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.319 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297868+0000", 2024-02-19T21:16:21.319 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999981 2024-02-19T21:16:21.319 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.319 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.319 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.319 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297866+0000", 2024-02-19T21:16:21.319 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4129999999999997e-06 2024-02-19T21:16:21.319 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.319 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.319 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.319 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297871+0000", 2024-02-19T21:16:21.319 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0733999999999999e-05 2024-02-19T21:16:21.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.320 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.320 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.320 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297882+0000", 2024-02-19T21:16:21.320 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5829999999999997e-06 2024-02-19T21:16:21.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.320 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.320 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.320 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297888+0000", 2024-02-19T21:16:21.320 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4179e-05 2024-02-19T21:16:21.320 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.320 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.320 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.320 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297912+0000", 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.7067000000000002e-05 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.297959+0000", 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9258999999999995e-05 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 3,5", 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.298028+0000", 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00027415799999999999 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.298302+0000", 2024-02-19T21:16:21.321 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00018730600000000001 2024-02-19T21:16:21.322 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.322 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.322 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:16:21.322 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.298490+0000", 2024-02-19T21:16:21.322 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026316253000000001 2024-02-19T21:16:21.322 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.322 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.322 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:16:21.322 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.324806+0000", 2024-02-19T21:16:21.322 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.9e-05 2024-02-19T21:16:21.322 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.322 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.322 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:21.322 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.324835+0000", 2024-02-19T21:16:21.323 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1923999999999999e-05 2024-02-19T21:16:21.323 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.323 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.323 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.323 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:08:14.324857+0000", 2024-02-19T21:16:21.323 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.026989016000000001 2024-02-19T21:16:21.323 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.323 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.323 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.323 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.323 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.323 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.22980.0:1 205.3s0 205:cbb9903e:::test_obj:head [omap-set-vals,omap-set-header] snapc 0=[] ondisk+write+known_if_redirected e373)", 2024-02-19T21:16:21.323 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:09:13.274270+0000", 2024-02-19T21:16:21.323 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 428.023637382, 2024-02-19T21:16:21.323 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.581927844, 2024-02-19T21:16:21.324 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.324 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:21.324 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.324 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.22980", 2024-02-19T21:16:21.324 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2118002540", 2024-02-19T21:16:21.324 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1 2024-02-19T21:16:21.324 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.324 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.324 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.324 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.324 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274270+0000", 2024-02-19T21:16:21.324 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.324 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.324 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.325 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.325 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274270+0000", 2024-02-19T21:16:21.325 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999971 2024-02-19T21:16:21.325 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.325 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.325 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.325 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274267+0000", 2024-02-19T21:16:21.325 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1556000000000001e-05 2024-02-19T21:16:21.325 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.325 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.325 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.325 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274279+0000", 2024-02-19T21:16:21.325 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1654000000000001e-05 2024-02-19T21:16:21.325 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.326 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.326 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.326 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274291+0000", 2024-02-19T21:16:21.326 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7020000000000007e-06 2024-02-19T21:16:21.326 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.326 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.326 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.326 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274299+0000", 2024-02-19T21:16:21.326 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014805800000000001 2024-02-19T21:16:21.326 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.326 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.326 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.326 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274447+0000", 2024-02-19T21:16:21.326 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.0749999999999999e-06 2024-02-19T21:16:21.327 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.327 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.327 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:16:21.327 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:13.274452+0000", 2024-02-19T21:16:21.327 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.5792793839999999 2024-02-19T21:16:21.327 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.327 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.327 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.327 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:14.853731+0000", 2024-02-19T21:16:21.327 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000125561 2024-02-19T21:16:21.327 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.327 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.328 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.328 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:14.853856+0000", 2024-02-19T21:16:21.328 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0023418330000000002 2024-02-19T21:16:21.328 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.328 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.328 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.328 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:14.856198+0000", 2024-02-19T21:16:21.328 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.581927844 2024-02-19T21:16:21.328 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.328 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.328 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.328 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.328 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.329 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:5600 208.10 208:08000000:%01:::head [pgnls start_epoch 393] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e393)", 2024-02-19T21:16:21.329 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:09:36.343158+0000", 2024-02-19T21:16:21.329 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 404.95474958599999, 2024-02-19T21:16:21.329 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.338287899, 2024-02-19T21:16:21.329 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.329 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:21.329 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.329 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:16:21.329 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:16:21.329 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 5600 2024-02-19T21:16:21.329 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.329 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.330 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.330 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.330 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343158+0000", 2024-02-19T21:16:21.330 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.330 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.330 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.330 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.330 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343158+0000", 2024-02-19T21:16:21.330 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:16:21.330 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.330 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.330 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.330 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343157+0000", 2024-02-19T21:16:21.331 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9830000000000001e-06 2024-02-19T21:16:21.331 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.331 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.331 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.331 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343163+0000", 2024-02-19T21:16:21.331 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4930000000000004e-06 2024-02-19T21:16:21.331 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.331 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.331 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.331 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343172+0000", 2024-02-19T21:16:21.331 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2700000000000004e-06 2024-02-19T21:16:21.331 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.331 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.331 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.331 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343178+0000", 2024-02-19T21:16:21.332 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3643000000000002e-05 2024-02-19T21:16:21.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.332 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.332 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.332 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343221+0000", 2024-02-19T21:16:21.332 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6779999999999997e-06 2024-02-19T21:16:21.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.332 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.332 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:16:21.332 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.343227+0000", 2024-02-19T21:16:21.332 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.33795772099999999 2024-02-19T21:16:21.332 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.332 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.332 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.333 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.681185+0000", 2024-02-19T21:16:21.333 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.0336000000000001e-05 2024-02-19T21:16:21.333 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.333 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.333 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.333 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.681235+0000", 2024-02-19T21:16:21.333 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00021113999999999999 2024-02-19T21:16:21.333 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.333 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.333 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.333 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:09:36.681446+0000", 2024-02-19T21:16:21.333 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.338287899 2024-02-19T21:16:21.333 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.333 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.333 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.34352.0:1425 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94681540394880 gen 2] snapc 0=[] ondisk+write+known_if_redirected e433)", 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:10:40.311762+0000", 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 340.98614542500002, 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.012232867, 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.34352", 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1425 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.334 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.335 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311762+0000", 2024-02-19T21:16:21.335 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.335 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.335 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.335 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.335 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311762+0000", 2024-02-19T21:16:21.335 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999967 2024-02-19T21:16:21.335 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.335 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.335 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.335 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311759+0000", 2024-02-19T21:16:21.335 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0362e-05 2024-02-19T21:16:21.335 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.335 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.336 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.336 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311769+0000", 2024-02-19T21:16:21.336 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.7405999999999999e-05 2024-02-19T21:16:21.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.336 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.336 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.336 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311787+0000", 2024-02-19T21:16:21.336 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.7710000000000006e-06 2024-02-19T21:16:21.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.336 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.336 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.336 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311795+0000", 2024-02-19T21:16:21.336 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.8292999999999999e-05 2024-02-19T21:16:21.336 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.336 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.337 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.337 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311853+0000", 2024-02-19T21:16:21.337 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0146000000000001e-05 2024-02-19T21:16:21.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.337 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:16:21.337 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:40.311863+0000", 2024-02-19T21:16:21.337 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.004993928 2024-02-19T21:16:21.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.337 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.337 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.337 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.316857+0000", 2024-02-19T21:16:21.337 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00033791000000000002 2024-02-19T21:16:21.337 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for degraded object", 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.317195+0000", 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0054299880000000002 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.322625+0000", 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.5621000000000002e-05 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.322680+0000", 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0013148350000000001 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.338 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.339 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.339 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:10:41.323995+0000", 2024-02-19T21:16:21.339 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.012232867 2024-02-19T21:16:21.339 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.339 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.339 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.339 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.339 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.339 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:6806 218.e 218:769c1872:test-rados-api-smithi163-83928-74::chunk1:head [writefull 0~10] snapc 0=[] ondisk+write+known_if_redirected e464)", 2024-02-19T21:16:21.339 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:11:32.421792+0000", 2024-02-19T21:16:21.339 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 288.876115868, 2024-02-19T21:16:21.339 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.17724985600000001, 2024-02-19T21:16:21.339 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.339 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:21.339 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.340 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:16:21.340 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:16:21.340 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 6806 2024-02-19T21:16:21.340 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.340 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.340 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.340 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.340 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421792+0000", 2024-02-19T21:16:21.340 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.340 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.340 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.340 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.340 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421792+0000", 2024-02-19T21:16:21.340 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999976 2024-02-19T21:16:21.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.341 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.341 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.341 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421790+0000", 2024-02-19T21:16:21.341 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.7069999999999998e-06 2024-02-19T21:16:21.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.341 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.341 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.341 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421798+0000", 2024-02-19T21:16:21.341 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4839e-05 2024-02-19T21:16:21.341 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.341 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.341 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.341 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421813+0000", 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1140000000000006e-06 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421821+0000", 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00013141300000000001 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421953+0000", 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.7780000000000004e-06 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for peered", 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.421959+0000", 2024-02-19T21:16:21.342 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.172664812 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.594623+0000", 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000130576 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.594754+0000", 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00015927700000000001 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 0,5", 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.594913+0000", 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0017018960000000001 2024-02-19T21:16:21.343 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.344 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:16:21.344 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.596615+0000", 2024-02-19T21:16:21.344 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.2569e-05 2024-02-19T21:16:21.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.344 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:16:21.344 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.596638+0000", 2024-02-19T21:16:21.344 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0023243410000000002 2024-02-19T21:16:21.344 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.344 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.344 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:16:21.344 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.598962+0000", 2024-02-19T21:16:21.344 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.6718000000000003e-05 2024-02-19T21:16:21.345 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.345 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.345 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:21.345 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.598999+0000", 2024-02-19T21:16:21.345 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3105999999999997e-05 2024-02-19T21:16:21.345 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.345 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.345 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.345 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:32.599042+0000", 2024-02-19T21:16:21.345 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.17724985600000001 2024-02-19T21:16:21.345 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.345 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.345 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.345 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.345 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.346 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:6815 61.10 61:0db63738:test-rados-api-smithi163-83928-74::foo:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e466)", 2024-02-19T21:16:21.346 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:11:34.300858+0000", 2024-02-19T21:16:21.346 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 286.99704971599999, 2024-02-19T21:16:21.346 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0076779409999999998, 2024-02-19T21:16:21.346 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.346 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:21.346 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.346 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:16:21.346 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:16:21.346 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 6815 2024-02-19T21:16:21.346 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.346 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.346 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.346 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.346 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300858+0000", 2024-02-19T21:16:21.347 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.347 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.347 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.347 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300858+0000", 2024-02-19T21:16:21.347 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:16:21.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.347 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.347 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.347 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300857+0000", 2024-02-19T21:16:21.347 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1149999999999999e-06 2024-02-19T21:16:21.347 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.347 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.347 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.347 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300863+0000", 2024-02-19T21:16:21.348 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1065e-05 2024-02-19T21:16:21.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.348 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.348 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.348 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300874+0000", 2024-02-19T21:16:21.348 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3249999999999998e-06 2024-02-19T21:16:21.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.348 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.348 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.348 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300879+0000", 2024-02-19T21:16:21.348 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3939999999999997e-05 2024-02-19T21:16:21.348 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.348 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.349 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.349 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300913+0000", 2024-02-19T21:16:21.349 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.9679000000000002e-05 2024-02-19T21:16:21.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.349 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.349 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.349 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.300973+0000", 2024-02-19T21:16:21.349 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0067811130000000001 2024-02-19T21:16:21.349 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.350 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.350 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 2,5", 2024-02-19T21:16:21.350 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.307754+0000", 2024-02-19T21:16:21.350 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00038248399999999999 2024-02-19T21:16:21.350 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.350 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.350 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:16:21.351 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.308136+0000", 2024-02-19T21:16:21.351 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00024405100000000001 2024-02-19T21:16:21.351 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.351 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.351 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:16:21.351 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.308381+0000", 2024-02-19T21:16:21.352 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4700999999999997e-05 2024-02-19T21:16:21.352 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.352 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.352 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:16:21.352 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.308435+0000", 2024-02-19T21:16:21.352 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4357999999999997e-05 2024-02-19T21:16:21.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.353 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.353 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:21.353 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.308490+0000", 2024-02-19T21:16:21.353 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6477999999999998e-05 2024-02-19T21:16:21.353 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.354 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.354 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.354 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:11:34.308536+0000", 2024-02-19T21:16:21.354 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0076779409999999998 2024-02-19T21:16:21.354 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.354 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.354 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.355 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.355 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.355 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.25753.0:7174 221.19 221:985d3dde:test-rados-api-smithi163-83928-80::foo:2 [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e503)", 2024-02-19T21:16:21.355 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:12:19.352715+0000", 2024-02-19T21:16:21.355 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 241.94519238999999, 2024-02-19T21:16:21.355 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010904531, 2024-02-19T21:16:21.356 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.356 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:21.356 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.356 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.25753", 2024-02-19T21:16:21.356 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:16:21.356 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 7174 2024-02-19T21:16:21.356 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.356 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.357 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.357 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.357 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352715+0000", 2024-02-19T21:16:21.357 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.357 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.357 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.357 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.357 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352715+0000", 2024-02-19T21:16:21.358 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:16:21.358 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.358 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.358 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.358 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352715+0000", 2024-02-19T21:16:21.358 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0860000000000002e-06 2024-02-19T21:16:21.358 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.358 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.358 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.358 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352718+0000", 2024-02-19T21:16:21.358 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.0129999999999999e-06 2024-02-19T21:16:21.359 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.359 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.359 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.359 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352724+0000", 2024-02-19T21:16:21.359 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.9620000000000004e-06 2024-02-19T21:16:21.359 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.359 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.359 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.360 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352728+0000", 2024-02-19T21:16:21.360 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.8980999999999999e-05 2024-02-19T21:16:21.360 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.360 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.360 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.360 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352757+0000", 2024-02-19T21:16:21.361 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3026000000000003e-05 2024-02-19T21:16:21.361 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.361 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.361 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.361 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.352820+0000", 2024-02-19T21:16:21.361 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010800346000000001 2024-02-19T21:16:21.362 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.362 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.362 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.362 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:12:19.363620+0000", 2024-02-19T21:16:21.362 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.010904531 2024-02-19T21:16:21.362 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.362 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.362 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.362 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.362 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.363 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:305 237.18 237:18000000::::head [pgnls start_epoch 629] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e629)", 2024-02-19T21:16:21.363 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:14:33.524542+0000", 2024-02-19T21:16:21.363 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 107.77336590500001, 2024-02-19T21:16:21.363 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0074946539999999999, 2024-02-19T21:16:21.363 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.363 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:21.364 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.364 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:16:21.364 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:16:21.364 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 305 2024-02-19T21:16:21.364 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.364 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.365 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.365 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.365 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.524542+0000", 2024-02-19T21:16:21.365 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.365 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.365 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.365 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.366 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.524542+0000", 2024-02-19T21:16:21.366 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999957 2024-02-19T21:16:21.366 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.366 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.366 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.366 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.524537+0000", 2024-02-19T21:16:21.366 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6256000000000002e-05 2024-02-19T21:16:21.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.367 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.367 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.367 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.524554+0000", 2024-02-19T21:16:21.367 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0934e-05 2024-02-19T21:16:21.367 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.368 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.368 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.368 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.524575+0000", 2024-02-19T21:16:21.368 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.0900000000000005e-06 2024-02-19T21:16:21.368 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.368 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.368 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.369 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.524583+0000", 2024-02-19T21:16:21.369 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.9338999999999994e-05 2024-02-19T21:16:21.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.369 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.369 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.524662+0000", 2024-02-19T21:16:21.369 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.0234999999999999e-05 2024-02-19T21:16:21.369 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.369 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.369 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:16:21.370 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.524672+0000", 2024-02-19T21:16:21.370 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0072035750000000003 2024-02-19T21:16:21.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.370 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.370 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.531876+0000", 2024-02-19T21:16:21.370 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6127000000000001e-05 2024-02-19T21:16:21.370 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.370 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.370 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.370 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.531922+0000", 2024-02-19T21:16:21.370 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00011454 2024-02-19T21:16:21.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.371 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.371 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:33.532037+0000", 2024-02-19T21:16:21.371 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0074946539999999999 2024-02-19T21:16:21.371 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.371 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.371 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.371 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.371 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.372 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:731 240.c 240:30000000::::head [pgnls start_epoch 655] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e655)", 2024-02-19T21:16:21.372 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:14:59.394608+0000", 2024-02-19T21:16:21.372 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 81.903299868999994, 2024-02-19T21:16:21.372 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025245411999999998, 2024-02-19T21:16:21.372 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.372 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:21.373 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.373 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:16:21.373 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:16:21.373 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 731 2024-02-19T21:16:21.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.373 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.373 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.373 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.373 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.394608+0000", 2024-02-19T21:16:21.373 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.373 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.374 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.374 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.374 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.394608+0000", 2024-02-19T21:16:21.374 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999952 2024-02-19T21:16:21.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.374 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.374 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.374 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.394603+0000", 2024-02-19T21:16:21.374 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8987e-05 2024-02-19T21:16:21.374 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.375 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.375 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.394622+0000", 2024-02-19T21:16:21.375 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.1327000000000001e-05 2024-02-19T21:16:21.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.375 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.375 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.394643+0000", 2024-02-19T21:16:21.375 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1616e-05 2024-02-19T21:16:21.375 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.375 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.375 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.376 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.394655+0000", 2024-02-19T21:16:21.376 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.6446999999999997e-05 2024-02-19T21:16:21.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.376 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.376 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.376 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.394711+0000", 2024-02-19T21:16:21.376 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.3429999999999998e-06 2024-02-19T21:16:21.376 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.377 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.377 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:16:21.377 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.394716+0000", 2024-02-19T21:16:21.377 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.024866266000000001 2024-02-19T21:16:21.377 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.377 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.378 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.378 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.419582+0000", 2024-02-19T21:16:21.378 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5274999999999997e-05 2024-02-19T21:16:21.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.378 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.378 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.378 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.419617+0000", 2024-02-19T21:16:21.378 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00023614299999999999 2024-02-19T21:16:21.378 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.378 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.378 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.378 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:14:59.419853+0000", 2024-02-19T21:16:21.379 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.025245411999999998 2024-02-19T21:16:21.379 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.379 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.379 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.379 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.379 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.379 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:1049 242.e 242:70000000::::head [pgnls start_epoch 673] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e673)", 2024-02-19T21:16:21.379 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:17.443969+0000", 2024-02-19T21:16:21.379 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 63.853938907, 2024-02-19T21:16:21.379 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.004054701, 2024-02-19T21:16:21.379 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.379 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:21.379 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.379 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:16:21.380 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:16:21.380 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1049 2024-02-19T21:16:21.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.380 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.380 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.380 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.380 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.443969+0000", 2024-02-19T21:16:21.380 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.380 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.380 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.380 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.443969+0000", 2024-02-19T21:16:21.380 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:16:21.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.381 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.381 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.443965+0000", 2024-02-19T21:16:21.381 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4428e-05 2024-02-19T21:16:21.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.381 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.381 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.443980+0000", 2024-02-19T21:16:21.381 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.8301e-05 2024-02-19T21:16:21.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.381 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.381 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.382 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.443998+0000", 2024-02-19T21:16:21.382 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4490000000000005e-06 2024-02-19T21:16:21.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.382 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.382 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.382 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.444007+0000", 2024-02-19T21:16:21.382 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8039000000000004e-05 2024-02-19T21:16:21.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.382 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.382 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.382 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.444075+0000", 2024-02-19T21:16:21.382 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4630999999999999e-05 2024-02-19T21:16:21.382 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.383 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.383 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:16:21.383 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.444090+0000", 2024-02-19T21:16:21.383 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0038148460000000002 2024-02-19T21:16:21.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.383 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.383 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.383 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.447905+0000", 2024-02-19T21:16:21.383 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4229000000000001e-05 2024-02-19T21:16:21.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.383 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.383 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.383 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.447929+0000", 2024-02-19T21:16:21.383 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.4438999999999995e-05 2024-02-19T21:16:21.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.384 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.384 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.384 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:17.448024+0000", 2024-02-19T21:16:21.384 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.004054701 2024-02-19T21:16:21.384 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.384 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.384 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.384 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.384 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:1294 244.17 244:e873ac47:test-rados-api-smithi163-83928-121::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e687)", 2024-02-19T21:16:21.384 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:36.533170+0000", 2024-02-19T21:16:21.384 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 44.764738184000002, 2024-02-19T21:16:21.385 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.004403433, 2024-02-19T21:16:21.385 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.385 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:21.385 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.385 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:16:21.385 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:16:21.385 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1294 2024-02-19T21:16:21.385 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.385 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.385 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.385 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.386 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.533170+0000", 2024-02-19T21:16:21.386 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.386 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.386 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.386 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.533170+0000", 2024-02-19T21:16:21.386 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:16:21.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.386 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.386 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.386 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.533166+0000", 2024-02-19T21:16:21.386 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6251000000000001e-05 2024-02-19T21:16:21.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.387 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.387 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.533182+0000", 2024-02-19T21:16:21.387 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6949e-05 2024-02-19T21:16:21.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.387 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.387 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.533199+0000", 2024-02-19T21:16:21.387 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8959999999999998e-06 2024-02-19T21:16:21.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.387 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.387 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.388 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.533209+0000", 2024-02-19T21:16:21.388 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.5043000000000003e-05 2024-02-19T21:16:21.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.388 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.388 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.388 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.533244+0000", 2024-02-19T21:16:21.388 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00017424399999999999 2024-02-19T21:16:21.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.388 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.388 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:16:21.388 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.533418+0000", 2024-02-19T21:16:21.388 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00088477 2024-02-19T21:16:21.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.389 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.389 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.389 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534303+0000", 2024-02-19T21:16:21.389 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000104916 2024-02-19T21:16:21.389 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.389 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.389 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.389 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534408+0000", 2024-02-19T21:16:21.389 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00017033 2024-02-19T21:16:21.389 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.389 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.389 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for subops from 4,6", 2024-02-19T21:16:21.389 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.534578+0000", 2024-02-19T21:16:21.389 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00062047899999999995 2024-02-19T21:16:21.390 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.390 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.390 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op_commit", 2024-02-19T21:16:21.390 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.535199+0000", 2024-02-19T21:16:21.390 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001316844 2024-02-19T21:16:21.390 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.390 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.390 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:16:21.390 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.536515+0000", 2024-02-19T21:16:21.390 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00099665999999999991 2024-02-19T21:16:21.391 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.391 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.391 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:16:21.391 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.537512+0000", 2024-02-19T21:16:21.391 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4561999999999999e-05 2024-02-19T21:16:21.391 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.391 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.391 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:21.391 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.537547+0000", 2024-02-19T21:16:21.391 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6455999999999999e-05 2024-02-19T21:16:21.391 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.391 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.391 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.391 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.537573+0000", 2024-02-19T21:16:21.392 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.004403433 2024-02-19T21:16:21.392 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.392 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.392 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.392 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.392 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.392 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:1339 244.17 244:e873ac47:test-rados-api-smithi163-83928-121::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e687)", 2024-02-19T21:16:21.392 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:36.559910+0000", 2024-02-19T21:16:21.392 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 44.737997610000001, 2024-02-19T21:16:21.392 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025691009999999999, 2024-02-19T21:16:21.392 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.392 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:21.392 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.393 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:16:21.393 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:16:21.393 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1339 2024-02-19T21:16:21.393 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.393 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.393 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.393 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.393 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.559910+0000", 2024-02-19T21:16:21.393 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.393 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.393 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.393 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.393 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.559910+0000", 2024-02-19T21:16:21.393 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999986 2024-02-19T21:16:21.393 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.394 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.394 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.394 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.559909+0000", 2024-02-19T21:16:21.394 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.6360000000000002e-06 2024-02-19T21:16:21.394 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.394 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.394 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.394 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.559914+0000", 2024-02-19T21:16:21.394 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.1249999999999993e-06 2024-02-19T21:16:21.394 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.394 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.394 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.394 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.559922+0000", 2024-02-19T21:16:21.394 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4.2950000000000004e-06 2024-02-19T21:16:21.395 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.395 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.395 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.395 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.559926+0000", 2024-02-19T21:16:21.395 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.5482000000000001e-05 2024-02-19T21:16:21.395 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.395 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.395 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.396 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.559952+0000", 2024-02-19T21:16:21.396 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.1148999999999993e-05 2024-02-19T21:16:21.396 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.396 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.396 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.396 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.560013+0000", 2024-02-19T21:16:21.396 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0022499669999999999 2024-02-19T21:16:21.396 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.396 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.396 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.396 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.562263+0000", 2024-02-19T21:16:21.396 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00014702000000000001 2024-02-19T21:16:21.397 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.397 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.397 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.397 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.562410+0000", 2024-02-19T21:16:21.397 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.9626e-05 2024-02-19T21:16:21.397 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.397 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.397 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.397 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.562479+0000", 2024-02-19T21:16:21.397 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0025691009999999999 2024-02-19T21:16:21.397 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.397 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.397 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.397 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(osd.7.53:92 234.7s0 234:e873ac47:test-rados-api-smithi163-83928-121::foo:head [copy-from ver 2] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected e687)", 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:36.560119+0000", 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 44.737788571999999, 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0017802849999999999, 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.560119+0000", 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.398 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.560119+0000", 2024-02-19T21:16:21.399 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:16:21.399 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.399 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.399 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.399 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.560118+0000", 2024-02-19T21:16:21.399 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.2039999999999996e-06 2024-02-19T21:16:21.399 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.399 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.399 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.399 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.560124+0000", 2024-02-19T21:16:21.399 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8199999999999999e-06 2024-02-19T21:16:21.399 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.399 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.399 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.399 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.560130+0000", 2024-02-19T21:16:21.400 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.4089999999999999e-06 2024-02-19T21:16:21.400 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.400 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.400 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.400 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.560134+0000", 2024-02-19T21:16:21.400 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.8917000000000001e-05 2024-02-19T21:16:21.400 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.400 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.400 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.400 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.560163+0000", 2024-02-19T21:16:21.400 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 8.2997999999999994e-05 2024-02-19T21:16:21.400 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.400 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.400 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.401 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.560246+0000", 2024-02-19T21:16:21.401 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00087688499999999999 2024-02-19T21:16:21.401 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.401 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.401 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_started", 2024-02-19T21:16:21.401 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.561123+0000", 2024-02-19T21:16:21.401 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00041383499999999998 2024-02-19T21:16:21.401 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.401 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.401 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "sub_op_committed", 2024-02-19T21:16:21.401 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.561537+0000", 2024-02-19T21:16:21.401 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00029965999999999999 2024-02-19T21:16:21.401 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.401 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.402 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "commit_sent", 2024-02-19T21:16:21.402 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.561836+0000", 2024-02-19T21:16:21.402 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.3359000000000001e-05 2024-02-19T21:16:21.402 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.402 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.402 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.402 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.561900+0000", 2024-02-19T21:16:21.402 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0017802849999999999 2024-02-19T21:16:21.402 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.402 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.402 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.402 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.402 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.402 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:1382 244.17 244:e873ac47:test-rados-api-smithi163-83928-121::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected e687)", 2024-02-19T21:16:21.403 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:36.576734+0000", 2024-02-19T21:16:21.403 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 44.721173729, 2024-02-19T21:16:21.403 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024618940000000001, 2024-02-19T21:16:21.403 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.403 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:21.403 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.403 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:16:21.403 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:16:21.403 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1382 2024-02-19T21:16:21.403 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.403 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.403 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.403 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.403 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.576734+0000", 2024-02-19T21:16:21.404 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.404 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.404 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.404 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.404 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.576734+0000", 2024-02-19T21:16:21.404 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.999999 2024-02-19T21:16:21.404 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.404 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.404 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.404 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.576733+0000", 2024-02-19T21:16:21.404 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.993e-06 2024-02-19T21:16:21.404 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.405 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.405 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.405 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.576736+0000", 2024-02-19T21:16:21.405 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.947e-06 2024-02-19T21:16:21.405 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.405 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.405 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.405 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.576742+0000", 2024-02-19T21:16:21.405 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.0989999999999999e-06 2024-02-19T21:16:21.405 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.405 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.405 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.405 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.576745+0000", 2024-02-19T21:16:21.405 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.3191999999999997e-05 2024-02-19T21:16:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.576779+0000", 2024-02-19T21:16:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000101388 2024-02-19T21:16:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for blocked object", 2024-02-19T21:16:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.576880+0000", 2024-02-19T21:16:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001308441 2024-02-19T21:16:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.578188+0000", 2024-02-19T21:16:21.406 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.8276999999999997e-05 2024-02-19T21:16:21.407 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.407 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.407 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "waiting for rw locks", 2024-02-19T21:16:21.407 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.578257+0000", 2024-02-19T21:16:21.407 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00035292599999999999 2024-02-19T21:16:21.407 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.407 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.407 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.407 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.578610+0000", 2024-02-19T21:16:21.407 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4078e-05 2024-02-19T21:16:21.407 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.407 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.407 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.407 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.578664+0000", 2024-02-19T21:16:21.407 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00053229799999999995 2024-02-19T21:16:21.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.408 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.408 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.408 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:36.579196+0000", 2024-02-19T21:16:21.408 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0024618940000000001 2024-02-19T21:16:21.408 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.408 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.408 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.408 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.408 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:1461 244.18 244:18000000::::head [pgnls start_epoch 689] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e689)", 2024-02-19T21:16:21.408 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:38.563209+0000", 2024-02-19T21:16:21.408 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 42.734698582, 2024-02-19T21:16:21.409 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001766807, 2024-02-19T21:16:21.409 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.409 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:21.409 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.409 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:16:21.409 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:16:21.409 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1461 2024-02-19T21:16:21.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.409 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.409 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.409 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.409 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.563209+0000", 2024-02-19T21:16:21.409 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.410 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.410 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.410 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.563209+0000", 2024-02-19T21:16:21.410 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999948 2024-02-19T21:16:21.410 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.410 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.410 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.410 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.563204+0000", 2024-02-19T21:16:21.410 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6804e-05 2024-02-19T21:16:21.410 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.410 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.410 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.410 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.563221+0000", 2024-02-19T21:16:21.410 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.4978e-05 2024-02-19T21:16:21.410 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.563246+0000", 2024-02-19T21:16:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.045e-05 2024-02-19T21:16:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.563256+0000", 2024-02-19T21:16:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.3435000000000004e-05 2024-02-19T21:16:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.563330+0000", 2024-02-19T21:16:21.411 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 7.6129999999999998e-06 2024-02-19T21:16:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:16:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.563338+0000", 2024-02-19T21:16:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001496809 2024-02-19T21:16:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.564834+0000", 2024-02-19T21:16:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.2138999999999998e-05 2024-02-19T21:16:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.412 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.564866+0000", 2024-02-19T21:16:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000109597 2024-02-19T21:16:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:38.564976+0000", 2024-02-19T21:16:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.001766807 2024-02-19T21:16:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "MOSDScrubReserve(6.b REQUEST e706)", 2024-02-19T21:16:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:56.082668+0000", 2024-02-19T21:16:21.413 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 25.215240166000001, 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0018688070000000001, 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:56.082668+0000", 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:56.082668+0000", 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999995 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.414 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.415 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:56.082667+0000", 2024-02-19T21:16:21.415 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.3430000000000003e-06 2024-02-19T21:16:21.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.415 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.415 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.415 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:56.082672+0000", 2024-02-19T21:16:21.415 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.3442999999999999e-05 2024-02-19T21:16:21.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.415 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.415 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.415 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:56.082696+0000", 2024-02-19T21:16:21.415 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.2150000000000001e-05 2024-02-19T21:16:21.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.415 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.416 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.416 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:56.082708+0000", 2024-02-19T21:16:21.416 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.6514999999999993e-05 2024-02-19T21:16:21.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.416 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.416 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.416 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:56.082805+0000", 2024-02-19T21:16:21.416 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.1562999999999999e-05 2024-02-19T21:16:21.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.416 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.416 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:16:21.416 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:56.082816+0000", 2024-02-19T21:16:21.416 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0016773669999999999 2024-02-19T21:16:21.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.417 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.417 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.417 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:56.084493+0000", 2024-02-19T21:16:21.417 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.6889e-05 2024-02-19T21:16:21.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.417 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.417 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.417 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:56.084510+0000", 2024-02-19T21:16:21.417 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.6149999999999999e-05 2024-02-19T21:16:21.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.417 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.417 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.417 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:56.084536+0000", 2024-02-19T21:16:21.418 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0018688070000000001 2024-02-19T21:16:21.418 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.418 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.418 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.418 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.418 INFO:teuthology.orchestra.run.smithi193.stdout: "description": "osd_op(client.23517.0:1728 246.0 246:00000000::::head [pgnls start_epoch 710] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e710)", 2024-02-19T21:16:21.418 INFO:teuthology.orchestra.run.smithi193.stdout: "initiated_at": "2024-02-19T21:15:59.393309+0000", 2024-02-19T21:16:21.418 INFO:teuthology.orchestra.run.smithi193.stdout: "age": 21.904598422999999, 2024-02-19T21:16:21.418 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0041497089999999997, 2024-02-19T21:16:21.418 INFO:teuthology.orchestra.run.smithi193.stdout: "type_data": { 2024-02-19T21:16:21.418 INFO:teuthology.orchestra.run.smithi193.stdout: "flag_point": "started", 2024-02-19T21:16:21.418 INFO:teuthology.orchestra.run.smithi193.stdout: "client_info": { 2024-02-19T21:16:21.418 INFO:teuthology.orchestra.run.smithi193.stdout: "client": "client.23517", 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: "tid": 1728 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: "events": [ 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "initiated", 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:59.393309+0000", 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "throttled", 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:59.393309+0000", 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 4294967295.9999962 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.419 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "header_read", 2024-02-19T21:16:21.420 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:59.393306+0000", 2024-02-19T21:16:21.420 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 1.4391e-05 2024-02-19T21:16:21.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.420 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.420 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "all_read", 2024-02-19T21:16:21.420 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:59.393320+0000", 2024-02-19T21:16:21.420 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 2.0667000000000001e-05 2024-02-19T21:16:21.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.420 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.420 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "dispatched", 2024-02-19T21:16:21.420 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:59.393341+0000", 2024-02-19T21:16:21.420 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 9.8819999999999996e-06 2024-02-19T21:16:21.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.421 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.421 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "queued_for_pg", 2024-02-19T21:16:21.421 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:59.393350+0000", 2024-02-19T21:16:21.421 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 5.4978000000000001e-05 2024-02-19T21:16:21.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.421 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.421 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.421 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:59.393405+0000", 2024-02-19T21:16:21.421 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 6.7510000000000002e-06 2024-02-19T21:16:21.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.421 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.421 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "op must wait for map", 2024-02-19T21:16:21.421 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:59.393412+0000", 2024-02-19T21:16:21.421 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.00388844 2024-02-19T21:16:21.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.422 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.422 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "reached_pg", 2024-02-19T21:16:21.422 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:59.397301+0000", 2024-02-19T21:16:21.422 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 3.7033999999999998e-05 2024-02-19T21:16:21.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.422 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.422 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "started", 2024-02-19T21:16:21.422 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:59.397338+0000", 2024-02-19T21:16:21.422 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.000121478 2024-02-19T21:16:21.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-02-19T21:16:21.422 INFO:teuthology.orchestra.run.smithi193.stdout: { 2024-02-19T21:16:21.422 INFO:teuthology.orchestra.run.smithi193.stdout: "event": "done", 2024-02-19T21:16:21.422 INFO:teuthology.orchestra.run.smithi193.stdout: "time": "2024-02-19T21:15:59.397459+0000", 2024-02-19T21:16:21.422 INFO:teuthology.orchestra.run.smithi193.stdout: "duration": 0.0041497089999999997 2024-02-19T21:16:21.422 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.423 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.423 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.423 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-02-19T21:16:21.423 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-02-19T21:16:21.423 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:16:21.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:19.854302+0000 osd.7 (osd.7) 331 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:16:21.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:19.857834+0000 osd.7 (osd.7) 332 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:16:21.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:20.030934+0000 osd.5 (osd.5) 300 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:16:21.498 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:20.033823+0000 osd.5 (osd.5) 301 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:16:21.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:20.397294+0000 mon.a (mon.0) 3510 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_count","val": "3"}]': finished 2024-02-19T21:16:21.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:20.397337+0000 mon.a (mon.0) 3511 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2024-02-19T21:16:21.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:20.401079+0000 mon.a (mon.0) 3512 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_period","val": "3"}]: dispatch 2024-02-19T21:16:21.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:20.745537+0000 mgr.y (mgr.14550) 612 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:21.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:20.847335+0000 mon.a (mon.0) 3513 : audit [DBG] from='client.? v1:172.21.15.163:0/2224134398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:21.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:21.029105+0000 osd.3 (osd.3) 307 : cluster [DBG] 6.c scrub starts 2024-02-19T21:16:21.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:21.029969+0000 osd.3 (osd.3) 308 : cluster [DBG] 6.c scrub ok 2024-02-19T21:16:21.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:19.854302+0000 osd.7 (osd.7) 331 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:16:21.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:19.857834+0000 osd.7 (osd.7) 332 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:16:21.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:20.030934+0000 osd.5 (osd.5) 300 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:16:21.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:20.033823+0000 osd.5 (osd.5) 301 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:16:21.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:20.397294+0000 mon.a (mon.0) 3510 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_count","val": "3"}]': finished 2024-02-19T21:16:21.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:20.397337+0000 mon.a (mon.0) 3511 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2024-02-19T21:16:21.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:20.401079+0000 mon.a (mon.0) 3512 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_period","val": "3"}]: dispatch 2024-02-19T21:16:21.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:20.745537+0000 mgr.y (mgr.14550) 612 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:21.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:20.847335+0000 mon.a (mon.0) 3513 : audit [DBG] from='client.? v1:172.21.15.163:0/2224134398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:21.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:21.029105+0000 osd.3 (osd.3) 307 : cluster [DBG] 6.c scrub starts 2024-02-19T21:16:21.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:21.029969+0000 osd.3 (osd.3) 308 : cluster [DBG] 6.c scrub ok 2024-02-19T21:16:21.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:19.854302+0000 osd.7 (osd.7) 331 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:16:21.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:19.857834+0000 osd.7 (osd.7) 332 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:16:21.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:20.030934+0000 osd.5 (osd.5) 300 : cluster [DBG] 2.3 scrub starts 2024-02-19T21:16:21.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:20.033823+0000 osd.5 (osd.5) 301 : cluster [DBG] 2.3 scrub ok 2024-02-19T21:16:21.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:20.397294+0000 mon.a (mon.0) 3510 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_count","val": "3"}]': finished 2024-02-19T21:16:21.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:20.397337+0000 mon.a (mon.0) 3511 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2024-02-19T21:16:21.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:20.401079+0000 mon.a (mon.0) 3512 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_period","val": "3"}]: dispatch 2024-02-19T21:16:21.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:20.745537+0000 mgr.y (mgr.14550) 612 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:21.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:20.847335+0000 mon.a (mon.0) 3513 : audit [DBG] from='client.? v1:172.21.15.163:0/2224134398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:21.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:21.029105+0000 osd.3 (osd.3) 307 : cluster [DBG] 6.c scrub starts 2024-02-19T21:16:21.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:21.029969+0000 osd.3 (osd.3) 308 : cluster [DBG] 6.c scrub ok 2024-02-19T21:16:22.037 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:22.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:22.141 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:22.163 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-19T21:16:22.172 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:22.192 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:22.194 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:22.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:22.296 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:16:22.323 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:22.341 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:16:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:22.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:16:22.446 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:22.473 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:22.492 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:22.493 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:22.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:22.595 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:22.621 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:22.640 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:22.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:22.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:22.743 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:22.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:21.071375+0000 osd.5 (osd.5) 302 : cluster [DBG] 5.f scrub starts 2024-02-19T21:16:22.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:21.074294+0000 osd.5 (osd.5) 303 : cluster [DBG] 5.f scrub ok 2024-02-19T21:16:22.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:21.319243+0000 mgr.y (mgr.14550) 613 : cluster [DBG] pgmap v1103: 214 pgs: 214 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:22.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:21.404474+0000 mon.a (mon.0) 3514 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_period","val": "3"}]': finished 2024-02-19T21:16:22.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:21.404536+0000 mon.a (mon.0) 3515 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2024-02-19T21:16:22.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:21.406607+0000 mon.a (mon.0) 3516 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:16:22.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:21.452579+0000 osd.0 (osd.0) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:16:22.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:21.452734+0000 osd.0 (osd.0) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:16:22.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:21.554466+0000 osd.6 (osd.6) 255 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:16:22.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:21.555387+0000 osd.6 (osd.6) 256 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:16:22.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:21.987289+0000 osd.3 (osd.3) 309 : cluster [DBG] 234.4 scrub starts 2024-02-19T21:16:22.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:21.988863+0000 osd.3 (osd.3) 310 : cluster [DBG] 234.4 scrub ok 2024-02-19T21:16:22.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:22.400600+0000 mon.a (mon.0) 3517 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:16:22.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:22.400667+0000 mon.a (mon.0) 3518 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2024-02-19T21:16:22.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:22.401052+0000 mon.a (mon.0) 3519 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-19T21:16:22.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:21.071375+0000 osd.5 (osd.5) 302 : cluster [DBG] 5.f scrub starts 2024-02-19T21:16:22.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:21.074294+0000 osd.5 (osd.5) 303 : cluster [DBG] 5.f scrub ok 2024-02-19T21:16:22.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:21.319243+0000 mgr.y (mgr.14550) 613 : cluster [DBG] pgmap v1103: 214 pgs: 214 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:22.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:21.404474+0000 mon.a (mon.0) 3514 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_period","val": "3"}]': finished 2024-02-19T21:16:22.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:21.404536+0000 mon.a (mon.0) 3515 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2024-02-19T21:16:22.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:21.406607+0000 mon.a (mon.0) 3516 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:16:22.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:21.452579+0000 osd.0 (osd.0) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:16:22.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:21.452734+0000 osd.0 (osd.0) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:16:22.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:21.554466+0000 osd.6 (osd.6) 255 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:16:22.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:21.555387+0000 osd.6 (osd.6) 256 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:16:22.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:21.987289+0000 osd.3 (osd.3) 309 : cluster [DBG] 234.4 scrub starts 2024-02-19T21:16:22.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:21.988863+0000 osd.3 (osd.3) 310 : cluster [DBG] 234.4 scrub ok 2024-02-19T21:16:22.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:22.400600+0000 mon.a (mon.0) 3517 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:16:22.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:22.400667+0000 mon.a (mon.0) 3518 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2024-02-19T21:16:22.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:22.401052+0000 mon.a (mon.0) 3519 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-19T21:16:22.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:21.071375+0000 osd.5 (osd.5) 302 : cluster [DBG] 5.f scrub starts 2024-02-19T21:16:22.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:21.074294+0000 osd.5 (osd.5) 303 : cluster [DBG] 5.f scrub ok 2024-02-19T21:16:22.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:21.319243+0000 mgr.y (mgr.14550) 613 : cluster [DBG] pgmap v1103: 214 pgs: 214 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:22.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:21.404474+0000 mon.a (mon.0) 3514 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_period","val": "3"}]': finished 2024-02-19T21:16:22.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:21.404536+0000 mon.a (mon.0) 3515 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2024-02-19T21:16:22.768 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:21.406607+0000 mon.a (mon.0) 3516 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:16:22.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:21.452579+0000 osd.0 (osd.0) 133 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:16:22.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:21.452734+0000 osd.0 (osd.0) 134 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:16:22.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:21.554466+0000 osd.6 (osd.6) 255 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:16:22.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:21.555387+0000 osd.6 (osd.6) 256 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:16:22.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:21.987289+0000 osd.3 (osd.3) 309 : cluster [DBG] 234.4 scrub starts 2024-02-19T21:16:22.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:21.988863+0000 osd.3 (osd.3) 310 : cluster [DBG] 234.4 scrub ok 2024-02-19T21:16:22.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:22.400600+0000 mon.a (mon.0) 3517 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:16:22.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:22.400667+0000 mon.a (mon.0) 3518 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2024-02-19T21:16:22.769 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:22.401052+0000 mon.a (mon.0) 3519 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-02-19T21:16:22.770 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:22.788 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:22.791 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:22.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:22.892 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:22.919 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:22.940 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:22.943 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:22.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:23.045 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:23.073 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:23.092 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:23.095 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:23.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:23.196 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:16:23.223 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:23.242 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:16:23.244 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:23.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:16:23.345 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:23.374 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:23.427 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:23.430 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:23.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:23.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.443+0000 7f0511085700 1 -- 172.21.15.163:0/807426151 >> v1:172.21.15.163:6789/0 conn(0x7f050c1042e0 legacy=0x7f050c106780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:23.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.443+0000 7f0511085700 1 -- 172.21.15.163:0/807426151 shutdown_connections 2024-02-19T21:16:23.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.443+0000 7f0511085700 1 -- 172.21.15.163:0/807426151 >> 172.21.15.163:0/807426151 conn(0x7f050c06cc00 msgr2=0x7f050c0f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:23.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.443+0000 7f0511085700 1 -- 172.21.15.163:0/807426151 shutdown_connections 2024-02-19T21:16:23.444 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.443+0000 7f0511085700 1 -- 172.21.15.163:0/807426151 wait complete. 2024-02-19T21:16:23.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.443+0000 7f0511085700 1 librados: starting msgr at 2024-02-19T21:16:23.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.443+0000 7f0511085700 1 librados: starting objecter 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f0511085700 1 Processor -- start 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f0511085700 1 -- start start 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f0511085700 1 librados: setting wanted keys 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f0511085700 1 librados: calling monclient init 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f0511085700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f050c1173b0 con 0x7f050c1042e0 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f0511085700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f050c1185b0 con 0x7f050c1007a0 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f0511085700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f050c1197b0 con 0x7f050c0fd9c0 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f050b59e700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f050c1042e0 0x7f050c115ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:45984/0 (socket says 172.21.15.163:45984) 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f050ad9d700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f050c0fd9c0 0x7f050c0f9a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:44088/0 (socket says 172.21.15.163:44088) 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f050b59e700 1 -- 172.21.15.163:0/3047273559 learned_addr learned my addr 172.21.15.163:0/3047273559 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f04f3fff700 10 client.?.objecter ms_handle_connect 0x7f050c0fd9c0 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f04f3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f04f3fff700 10 client.?.objecter ms_handle_connect 0x7f050c1007a0 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f04f3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f04f3fff700 10 client.?.objecter ms_handle_connect 0x7f050c1042e0 2024-02-19T21:16:23.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f04f3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:23.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.444+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162015411 0 0) 0x7f050c1185b0 con 0x7f050c1007a0 2024-02-19T21:16:23.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f4003800 con 0x7f050c1007a0 2024-02-19T21:16:23.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240321861 0 0) 0x7f050c1197b0 con 0x7f050c0fd9c0 2024-02-19T21:16:23.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f050c1185b0 con 0x7f050c0fd9c0 2024-02-19T21:16:23.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467688707 0 0) 0x7f050c1173b0 con 0x7f050c1042e0 2024-02-19T21:16:23.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f050c1197b0 con 0x7f050c1042e0 2024-02-19T21:16:23.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1431167219 0 0) 0x7f04f4003800 con 0x7f050c1007a0 2024-02-19T21:16:23.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f050c1173b0 con 0x7f050c1007a0 2024-02-19T21:16:23.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3471927452 0 0) 0x7f050c1197b0 con 0x7f050c1042e0 2024-02-19T21:16:23.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04f4003800 con 0x7f050c1042e0 2024-02-19T21:16:23.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3150171616 0 0) 0x7f050c1185b0 con 0x7f050c0fd9c0 2024-02-19T21:16:23.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f050c1197b0 con 0x7f050c0fd9c0 2024-02-19T21:16:23.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f05000019f0 con 0x7f050c1007a0 2024-02-19T21:16:23.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f05040017a0 con 0x7f050c1042e0 2024-02-19T21:16:23.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f04fc001840 con 0x7f050c0fd9c0 2024-02-19T21:16:23.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3307267423 0 0) 0x7f050c1173b0 con 0x7f050c1007a0 2024-02-19T21:16:23.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 >> v1:172.21.15.163:6790/0 conn(0x7f050c0fd9c0 legacy=0x7f050c0f9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:23.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 >> v1:172.21.15.163:6789/0 conn(0x7f050c1042e0 legacy=0x7f050c115ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:23.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c11a9b0 con 0x7f050c1007a0 2024-02-19T21:16:23.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f0511085700 1 -- 172.21.15.163:0/3047273559 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050c1187a0 con 0x7f050c1007a0 2024-02-19T21:16:23.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f05000023a0 con 0x7f050c1007a0 2024-02-19T21:16:23.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0500003310 con 0x7f050c1007a0 2024-02-19T21:16:23.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f0511085700 10 client.49333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:23.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f0511085700 1 -- 172.21.15.163:0/3047273559 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f050c118c10 con 0x7f050c1007a0 2024-02-19T21:16:23.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.445+0000 7f0511085700 1 librados: init done 2024-02-19T21:16:23.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.446+0000 7f0511085700 1 -- 172.21.15.163:0/3047273559 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f050c05f560 con 0x7f050c1007a0 2024-02-19T21:16:23.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.447+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f05000161a0 con 0x7f050c1007a0 2024-02-19T21:16:23.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.447+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(734..734 src has 623..734) v4 ==== 8704+0+0 (unknown 2744592465 0 0) 0x7f0500076780 con 0x7f050c1007a0 2024-02-19T21:16:23.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.447+0000 7f04f3fff700 10 client.49333.objecter ms_dispatch 0x7f050c001550 osd_map(734..734 src has 623..734) v4 2024-02-19T21:16:23.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.447+0000 7f04f3fff700 3 client.49333.objecter handle_osd_map got epochs [734,734] > 0 2024-02-19T21:16:23.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.447+0000 7f04f3fff700 3 client.49333.objecter handle_osd_map decoding full epoch 734 2024-02-19T21:16:23.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.447+0000 7f04f3fff700 20 client.49333.objecter dump_active .. 0 homeless 2024-02-19T21:16:23.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.447+0000 7f04f3fff700 10 client.49333.objecter ms_handle_connect 0x7f04f4063e70 2024-02-19T21:16:23.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.451+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f050004a710 con 0x7f050c1007a0 2024-02-19T21:16:23.455 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:23.471 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:22.024096+0000 osd.5 (osd.5) 304 : cluster [DBG] 4.15 deep-scrub starts 2024-02-19T21:16:23.471 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:22.028896+0000 osd.5 (osd.5) 305 : cluster [DBG] 4.15 deep-scrub ok 2024-02-19T21:16:23.471 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:22.170270+0000 osd.1 (osd.1) 322 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:16:23.472 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:22.171886+0000 osd.1 (osd.1) 323 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:16:23.472 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:23.171897+0000 osd.4 (osd.4) 263 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:16:23.472 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:23.173927+0000 osd.4 (osd.4) 264 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:16:23.472 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:23.401628+0000 mon.a (mon.0) 3520 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-19T21:16:23.472 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:23.401690+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2024-02-19T21:16:23.472 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:16:23] "GET /metrics HTTP/1.1" 200 189817 "" "Prometheus/2.33.4" 2024-02-19T21:16:23.472 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:22.024096+0000 osd.5 (osd.5) 304 : cluster [DBG] 4.15 deep-scrub starts 2024-02-19T21:16:23.472 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:22.028896+0000 osd.5 (osd.5) 305 : cluster [DBG] 4.15 deep-scrub ok 2024-02-19T21:16:23.472 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:22.170270+0000 osd.1 (osd.1) 322 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:16:23.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:22.171886+0000 osd.1 (osd.1) 323 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:16:23.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:23.171897+0000 osd.4 (osd.4) 263 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:16:23.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:23.173927+0000 osd.4 (osd.4) 264 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:16:23.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:23.401628+0000 mon.a (mon.0) 3520 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-19T21:16:23.473 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:23.401690+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2024-02-19T21:16:23.532 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:23.558 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:23.577 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:23.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:23.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:23.633+0000 7f0511085700 1 -- 172.21.15.163:0/3047273559 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 0.0} v 0) v1 -- 0x7f050c109f00 con 0x7f050c1007a0 2024-02-19T21:16:23.682 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:23.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:22.024096+0000 osd.5 (osd.5) 304 : cluster [DBG] 4.15 deep-scrub starts 2024-02-19T21:16:23.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:22.028896+0000 osd.5 (osd.5) 305 : cluster [DBG] 4.15 deep-scrub ok 2024-02-19T21:16:23.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:22.170270+0000 osd.1 (osd.1) 322 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:16:23.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:22.171886+0000 osd.1 (osd.1) 323 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:16:23.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:23.171897+0000 osd.4 (osd.4) 263 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:16:23.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:23.173927+0000 osd.4 (osd.4) 264 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:16:23.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:23.401628+0000 mon.a (mon.0) 3520 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-130","var": "hit_set_fpp","val": ".01"}]': finished 2024-02-19T21:16:23.706 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:23.401690+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2024-02-19T21:16:23.707 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:23.727 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:23.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:23.831 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:23.860 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:23.958 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:23.960 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:23.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:24.043 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:16:24.062 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:24.114 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:24.267 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:24.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:24.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:24.372 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:24.399 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:24.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:24.429+0000 7f04f3fff700 1 -- 172.21.15.163:0/3047273559 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]=0 set osd.7 primary-affinity to 0 (802) v735) v1 ==== 129+0+0 (unknown 1150069139 0 0) 0x7f0500048b20 con 0x7f050c1007a0 2024-02-19T21:16:24.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:24.434+0000 7f0511085700 10 librados: watch_flush enter 2024-02-19T21:16:24.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:24.434+0000 7f0511085700 10 librados: watch_flush exit 2024-02-19T21:16:24.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:24.434+0000 7f0511085700 20 client.49333.objecter shutdown clearing up homeless session... 2024-02-19T21:16:24.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:24.434+0000 7f0511085700 10 client.49333.objecter successfully canceled tick 2024-02-19T21:16:24.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:24.434+0000 7f0511085700 1 -- 172.21.15.163:0/3047273559 >> v1:172.21.15.163:6800/1552514681 conn(0x7f04f4063e70 legacy=0x7f04f4066330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:24.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:24.434+0000 7f0511085700 1 -- 172.21.15.163:0/3047273559 >> v1:172.21.15.193:6789/0 conn(0x7f050c1007a0 legacy=0x7f050c0fa180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:24.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:24.434+0000 7f0511085700 1 -- 172.21.15.163:0/3047273559 shutdown_connections 2024-02-19T21:16:24.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:24.434+0000 7f0511085700 1 -- 172.21.15.163:0/3047273559 >> 172.21.15.163:0/3047273559 conn(0x7f050c06cc00 msgr2=0x7f050c0f65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:24.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:24.434+0000 7f0511085700 1 -- 172.21.15.163:0/3047273559 shutdown_connections 2024-02-19T21:16:24.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:24.435+0000 7f0511085700 1 -- 172.21.15.163:0/3047273559 wait complete. 2024-02-19T21:16:24.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:24.435+0000 7f0511085700 1 librados: shutdown 2024-02-19T21:16:24.439 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.7 primary-affinity to 0 (802) 2024-02-19T21:16:24.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:23.015797+0000 osd.2 (osd.2) 288 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:16:24.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:23.017242+0000 osd.2 (osd.2) 289 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:16:24.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:23.204183+0000 osd.1 (osd.1) 324 : cluster [DBG] 2.a scrub starts 2024-02-19T21:16:24.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:23.207190+0000 osd.1 (osd.1) 325 : cluster [DBG] 2.a scrub ok 2024-02-19T21:16:24.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:23.319642+0000 mgr.y (mgr.14550) 614 : cluster [DBG] pgmap v1106: 214 pgs: 214 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:24.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:23.605725+0000 osd.6 (osd.6) 257 : cluster [DBG] 16.0 deep-scrub starts 2024-02-19T21:16:24.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:23.608669+0000 osd.6 (osd.6) 258 : cluster [DBG] 16.0 deep-scrub ok 2024-02-19T21:16:24.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:23.635182+0000 mon.b (mon.1) 749 : audit [INF] from='client.? v1:172.21.15.163:0/3047273559' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-02-19T21:16:24.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:23.635518+0000 mon.a (mon.0) 3522 : audit [INF] from='client.49333 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-02-19T21:16:24.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:24.108646+0000 mon.a (mon.0) 3523 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:24.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:23.015797+0000 osd.2 (osd.2) 288 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:16:24.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:23.017242+0000 osd.2 (osd.2) 289 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:16:24.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:23.204183+0000 osd.1 (osd.1) 324 : cluster [DBG] 2.a scrub starts 2024-02-19T21:16:24.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:23.207190+0000 osd.1 (osd.1) 325 : cluster [DBG] 2.a scrub ok 2024-02-19T21:16:24.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:23.319642+0000 mgr.y (mgr.14550) 614 : cluster [DBG] pgmap v1106: 214 pgs: 214 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:24.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:23.605725+0000 osd.6 (osd.6) 257 : cluster [DBG] 16.0 deep-scrub starts 2024-02-19T21:16:24.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:23.608669+0000 osd.6 (osd.6) 258 : cluster [DBG] 16.0 deep-scrub ok 2024-02-19T21:16:24.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:23.635182+0000 mon.b (mon.1) 749 : audit [INF] from='client.? v1:172.21.15.163:0/3047273559' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-02-19T21:16:24.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:23.635518+0000 mon.a (mon.0) 3522 : audit [INF] from='client.49333 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-02-19T21:16:24.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:24.108646+0000 mon.a (mon.0) 3523 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:23.015797+0000 osd.2 (osd.2) 288 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:16:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:23.017242+0000 osd.2 (osd.2) 289 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:16:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:23.204183+0000 osd.1 (osd.1) 324 : cluster [DBG] 2.a scrub starts 2024-02-19T21:16:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:23.207190+0000 osd.1 (osd.1) 325 : cluster [DBG] 2.a scrub ok 2024-02-19T21:16:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:23.319642+0000 mgr.y (mgr.14550) 614 : cluster [DBG] pgmap v1106: 214 pgs: 214 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:23.605725+0000 osd.6 (osd.6) 257 : cluster [DBG] 16.0 deep-scrub starts 2024-02-19T21:16:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:23.608669+0000 osd.6 (osd.6) 258 : cluster [DBG] 16.0 deep-scrub ok 2024-02-19T21:16:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:23.635182+0000 mon.b (mon.1) 749 : audit [INF] from='client.? v1:172.21.15.163:0/3047273559' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-02-19T21:16:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:23.635518+0000 mon.a (mon.0) 3522 : audit [INF] from='client.49333 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2024-02-19T21:16:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:24.108646+0000 mon.a (mon.0) 3523 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:24.945 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:16:25.314 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:16:25.314 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:16:25.314 INFO:teuthology.orchestra.run.smithi163.stdout: "num_ops": 0 2024-02-19T21:16:25.314 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:16:25.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:23.957818+0000 osd.7 (osd.7) 333 : cluster [DBG] 234.2 scrub starts 2024-02-19T21:16:25.663 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:23.961308+0000 osd.7 (osd.7) 334 : cluster [DBG] 234.2 scrub ok 2024-02-19T21:16:25.664 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:24.173914+0000 osd.1 (osd.1) 326 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:16:25.664 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:24.176881+0000 osd.1 (osd.1) 327 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:16:25.664 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:24.426784+0000 mon.a (mon.0) 3524 : audit [INF] from='client.49333 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2024-02-19T21:16:25.664 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:24.426859+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2024-02-19T21:16:25.664 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:25.427022+0000 mon.a (mon.0) 3526 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2024-02-19T21:16:25.664 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:23.957818+0000 osd.7 (osd.7) 333 : cluster [DBG] 234.2 scrub starts 2024-02-19T21:16:25.664 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:23.961308+0000 osd.7 (osd.7) 334 : cluster [DBG] 234.2 scrub ok 2024-02-19T21:16:25.664 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:24.173914+0000 osd.1 (osd.1) 326 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:16:25.664 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:24.176881+0000 osd.1 (osd.1) 327 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:16:25.664 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:24.426784+0000 mon.a (mon.0) 3524 : audit [INF] from='client.49333 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2024-02-19T21:16:25.664 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:24.426859+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2024-02-19T21:16:25.665 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:25.427022+0000 mon.a (mon.0) 3526 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2024-02-19T21:16:25.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:23.957818+0000 osd.7 (osd.7) 333 : cluster [DBG] 234.2 scrub starts 2024-02-19T21:16:25.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:23.961308+0000 osd.7 (osd.7) 334 : cluster [DBG] 234.2 scrub ok 2024-02-19T21:16:25.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:24.173914+0000 osd.1 (osd.1) 326 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:16:25.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:24.176881+0000 osd.1 (osd.1) 327 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:16:25.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:24.426784+0000 mon.a (mon.0) 3524 : audit [INF] from='client.49333 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2024-02-19T21:16:25.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:24.426859+0000 mon.a (mon.0) 3525 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2024-02-19T21:16:25.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:25.427022+0000 mon.a (mon.0) 3526 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2024-02-19T21:16:26.053 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:26.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:26.158 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:26.188 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:26.287 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:26.289 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:26.290 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:26.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:26.331 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:26.392 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:26.425 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:26.448 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:26.452 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:26.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:26.554 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:26.583 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:26.602 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:26.605 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:26.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:26.706 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:26.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:25.022373+0000 osd.2 (osd.2) 290 : cluster [DBG] 16.1d scrub starts 2024-02-19T21:16:26.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:25.024288+0000 osd.2 (osd.2) 291 : cluster [DBG] 16.1d scrub ok 2024-02-19T21:16:26.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:25.320995+0000 mgr.y (mgr.14550) 615 : cluster [DBG] pgmap v1109: 214 pgs: 32 peering, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 4.0 KiB/s wr, 1 op/s; 426 B/s, 0 objects/s recovering 2024-02-19T21:16:26.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:25.436301+0000 mon.a (mon.0) 3527 : cluster [WRN] Health check failed: Reduced data availability: 19 pgs inactive, 25 pgs peering (PG_AVAILABILITY) 2024-02-19T21:16:26.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:25.559360+0000 osd.6 (osd.6) 259 : cluster [DBG] 16.6 scrub starts 2024-02-19T21:16:26.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:25.560238+0000 osd.6 (osd.6) 260 : cluster [DBG] 16.6 scrub ok 2024-02-19T21:16:26.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:25.977811+0000 osd.3 (osd.3) 311 : cluster [DBG] 234.1 scrub starts 2024-02-19T21:16:26.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:25.979398+0000 osd.3 (osd.3) 312 : cluster [DBG] 234.1 scrub ok 2024-02-19T21:16:26.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:25.022373+0000 osd.2 (osd.2) 290 : cluster [DBG] 16.1d scrub starts 2024-02-19T21:16:26.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:25.024288+0000 osd.2 (osd.2) 291 : cluster [DBG] 16.1d scrub ok 2024-02-19T21:16:26.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:25.320995+0000 mgr.y (mgr.14550) 615 : cluster [DBG] pgmap v1109: 214 pgs: 32 peering, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 4.0 KiB/s wr, 1 op/s; 426 B/s, 0 objects/s recovering 2024-02-19T21:16:26.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:25.436301+0000 mon.a (mon.0) 3527 : cluster [WRN] Health check failed: Reduced data availability: 19 pgs inactive, 25 pgs peering (PG_AVAILABILITY) 2024-02-19T21:16:26.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:25.559360+0000 osd.6 (osd.6) 259 : cluster [DBG] 16.6 scrub starts 2024-02-19T21:16:26.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:25.560238+0000 osd.6 (osd.6) 260 : cluster [DBG] 16.6 scrub ok 2024-02-19T21:16:26.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:25.977811+0000 osd.3 (osd.3) 311 : cluster [DBG] 234.1 scrub starts 2024-02-19T21:16:26.734 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:25.979398+0000 osd.3 (osd.3) 312 : cluster [DBG] 234.1 scrub ok 2024-02-19T21:16:26.735 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:26.808 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:26.811 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:26.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:26.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:25.022373+0000 osd.2 (osd.2) 290 : cluster [DBG] 16.1d scrub starts 2024-02-19T21:16:26.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:25.024288+0000 osd.2 (osd.2) 291 : cluster [DBG] 16.1d scrub ok 2024-02-19T21:16:26.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:25.320995+0000 mgr.y (mgr.14550) 615 : cluster [DBG] pgmap v1109: 214 pgs: 32 peering, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 4.0 KiB/s wr, 1 op/s; 426 B/s, 0 objects/s recovering 2024-02-19T21:16:26.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:25.436301+0000 mon.a (mon.0) 3527 : cluster [WRN] Health check failed: Reduced data availability: 19 pgs inactive, 25 pgs peering (PG_AVAILABILITY) 2024-02-19T21:16:26.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:25.559360+0000 osd.6 (osd.6) 259 : cluster [DBG] 16.6 scrub starts 2024-02-19T21:16:26.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:25.560238+0000 osd.6 (osd.6) 260 : cluster [DBG] 16.6 scrub ok 2024-02-19T21:16:26.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:25.977811+0000 osd.3 (osd.3) 311 : cluster [DBG] 234.1 scrub starts 2024-02-19T21:16:26.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:25.979398+0000 osd.3 (osd.3) 312 : cluster [DBG] 234.1 scrub ok 2024-02-19T21:16:26.912 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:26.939 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:27.290 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-19T21:16:27.689 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:27.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:26.024424+0000 osd.2 (osd.2) 292 : cluster [DBG] 234.5 scrub starts 2024-02-19T21:16:27.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:26.025833+0000 osd.2 (osd.2) 293 : cluster [DBG] 234.5 scrub ok 2024-02-19T21:16:27.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:26.514902+0000 osd.6 (osd.6) 261 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:16:27.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:26.516787+0000 osd.6 (osd.6) 262 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:16:27.691 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:26.024424+0000 osd.2 (osd.2) 292 : cluster [DBG] 234.5 scrub starts 2024-02-19T21:16:27.691 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:26.025833+0000 osd.2 (osd.2) 293 : cluster [DBG] 234.5 scrub ok 2024-02-19T21:16:27.691 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:26.514902+0000 osd.6 (osd.6) 261 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:16:27.691 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:26.516787+0000 osd.6 (osd.6) 262 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:16:27.692 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:27.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:27.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:26.024424+0000 osd.2 (osd.2) 292 : cluster [DBG] 234.5 scrub starts 2024-02-19T21:16:27.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:26.025833+0000 osd.2 (osd.2) 293 : cluster [DBG] 234.5 scrub ok 2024-02-19T21:16:27.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:26.514902+0000 osd.6 (osd.6) 261 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:16:27.773 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:26.516787+0000 osd.6 (osd.6) 262 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:16:27.793 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:27.820 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:28.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:16:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:16:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:16:28.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:26.974763+0000 osd.5 (osd.5) 306 : cluster [DBG] 16.f scrub starts 2024-02-19T21:16:28.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:26.978634+0000 osd.5 (osd.5) 307 : cluster [DBG] 16.f scrub ok 2024-02-19T21:16:28.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:27.001404+0000 osd.2 (osd.2) 294 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:16:28.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:27.004571+0000 osd.2 (osd.2) 295 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:16:28.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:27.321622+0000 mgr.y (mgr.14550) 616 : cluster [DBG] pgmap v1111: 214 pgs: 32 peering, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 832 B/s rd, 3.2 KiB/s wr, 1 op/s; 346 B/s, 0 objects/s recovering 2024-02-19T21:16:28.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:26.974763+0000 osd.5 (osd.5) 306 : cluster [DBG] 16.f scrub starts 2024-02-19T21:16:28.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:26.978634+0000 osd.5 (osd.5) 307 : cluster [DBG] 16.f scrub ok 2024-02-19T21:16:28.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:27.001404+0000 osd.2 (osd.2) 294 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:16:28.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:27.004571+0000 osd.2 (osd.2) 295 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:16:28.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:27.321622+0000 mgr.y (mgr.14550) 616 : cluster [DBG] pgmap v1111: 214 pgs: 32 peering, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 832 B/s rd, 3.2 KiB/s wr, 1 op/s; 346 B/s, 0 objects/s recovering 2024-02-19T21:16:28.570 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:28.573 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:28.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:28.674 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:28.707 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:28.728 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:28.730 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:26.974763+0000 osd.5 (osd.5) 306 : cluster [DBG] 16.f scrub starts 2024-02-19T21:16:28.730 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:26.978634+0000 osd.5 (osd.5) 307 : cluster [DBG] 16.f scrub ok 2024-02-19T21:16:28.730 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:27.001404+0000 osd.2 (osd.2) 294 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:16:28.730 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:27.004571+0000 osd.2 (osd.2) 295 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:16:28.730 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:27.321622+0000 mgr.y (mgr.14550) 616 : cluster [DBG] pgmap v1111: 214 pgs: 32 peering, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 832 B/s rd, 3.2 KiB/s wr, 1 op/s; 346 B/s, 0 objects/s recovering 2024-02-19T21:16:28.731 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:28.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:28.832 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:28.862 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:28.883 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:28.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:28.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:28.986 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:29.016 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:29.218 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:29.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:29.322 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:29.354 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:29.375 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:29.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:29.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:29.480 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:29.511 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:29.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:28.051246+0000 osd.2 (osd.2) 296 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:16:29.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:28.053235+0000 osd.2 (osd.2) 297 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:16:29.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:28.051246+0000 osd.2 (osd.2) 296 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:16:29.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:28.053235+0000 osd.2 (osd.2) 297 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:16:29.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:28.051246+0000 osd.2 (osd.2) 296 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:16:29.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:28.053235+0000 osd.2 (osd.2) 297 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:16:29.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.839+0000 7f34451f5700 1 -- 172.21.15.163:0/3471028379 >> v1:172.21.15.163:6789/0 conn(0x7f34400f94d0 legacy=0x7f3440102a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:29.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.839+0000 7f34451f5700 1 -- 172.21.15.163:0/3471028379 shutdown_connections 2024-02-19T21:16:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.839+0000 7f34451f5700 1 -- 172.21.15.163:0/3471028379 >> 172.21.15.163:0/3471028379 conn(0x7f34400f2470 msgr2=0x7f34400f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.839+0000 7f34451f5700 1 -- 172.21.15.163:0/3471028379 shutdown_connections 2024-02-19T21:16:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.839+0000 7f34451f5700 1 -- 172.21.15.163:0/3471028379 wait complete. 2024-02-19T21:16:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.840+0000 7f34451f5700 1 librados: starting msgr at 2024-02-19T21:16:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.840+0000 7f34451f5700 1 librados: starting objecter 2024-02-19T21:16:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.840+0000 7f34451f5700 1 Processor -- start 2024-02-19T21:16:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.840+0000 7f34451f5700 1 -- start start 2024-02-19T21:16:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.840+0000 7f34451f5700 1 librados: setting wanted keys 2024-02-19T21:16:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.840+0000 7f34451f5700 1 librados: calling monclient init 2024-02-19T21:16:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f34451f5700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3440117380 con 0x7f34401042e0 2024-02-19T21:16:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f34451f5700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3440118520 con 0x7f34400f89e0 2024-02-19T21:16:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f34451f5700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3440119720 con 0x7f34400f94d0 2024-02-19T21:16:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f343f59e700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f34401042e0 0x7f3440115a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:45912/0 (socket says 172.21.15.163:45912) 2024-02-19T21:16:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f343f59e700 1 -- 172.21.15.163:0/2424008614 learned_addr learned my addr 172.21.15.163:0/2424008614 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f343e59c700 1 --1- 172.21.15.163:0/2424008614 >> v1:172.21.15.163:6790/0 conn(0x7f34400f94d0 0x7f34400ffe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:58088/0 (socket says 172.21.15.163:58088) 2024-02-19T21:16:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 10 client.?.objecter ms_handle_connect 0x7f34401042e0 2024-02-19T21:16:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 10 client.?.objecter ms_handle_connect 0x7f34400f94d0 2024-02-19T21:16:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862149049 0 0) 0x7f3440117380 con 0x7f34401042e0 2024-02-19T21:16:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3428003800 con 0x7f34401042e0 2024-02-19T21:16:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 10 client.?.objecter ms_handle_connect 0x7f34400f89e0 2024-02-19T21:16:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930345481 0 0) 0x7f3440119720 con 0x7f34400f94d0 2024-02-19T21:16:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3440117380 con 0x7f34400f94d0 2024-02-19T21:16:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147910330 0 0) 0x7f3440118520 con 0x7f34400f89e0 2024-02-19T21:16:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3440119720 con 0x7f34400f89e0 2024-02-19T21:16:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2806217907 0 0) 0x7f3440117380 con 0x7f34400f94d0 2024-02-19T21:16:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3440118520 con 0x7f34400f94d0 2024-02-19T21:16:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3434002f10 con 0x7f34400f94d0 2024-02-19T21:16:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3036745511 0 0) 0x7f3428003800 con 0x7f34401042e0 2024-02-19T21:16:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3440117380 con 0x7f34401042e0 2024-02-19T21:16:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 972256834 0 0) 0x7f3440119720 con 0x7f34400f89e0 2024-02-19T21:16:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3428003800 con 0x7f34400f89e0 2024-02-19T21:16:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3438001ca0 con 0x7f34401042e0 2024-02-19T21:16:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3430001840 con 0x7f34400f89e0 2024-02-19T21:16:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 815039091 0 0) 0x7f3440118520 con 0x7f34400f94d0 2024-02-19T21:16:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 >> v1:172.21.15.193:6789/0 conn(0x7f34400f89e0 legacy=0x7f34400ff720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.841+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 >> v1:172.21.15.163:6789/0 conn(0x7f34401042e0 legacy=0x7f3440115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.842+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344011a920 con 0x7f34400f94d0 2024-02-19T21:16:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.842+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3440119910 con 0x7f34400f94d0 2024-02-19T21:16:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.842+0000 7f34451f5700 10 client.50531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.842+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3440119d80 con 0x7f34400f94d0 2024-02-19T21:16:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.842+0000 7f34451f5700 1 librados: init done 2024-02-19T21:16:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.842+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3434001b40 con 0x7f34400f94d0 2024-02-19T21:16:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.842+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3434003870 con 0x7f34400f94d0 2024-02-19T21:16:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.842+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f342c004f00 con 0x7f34400f94d0 2024-02-19T21:16:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.843+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f3434002c50 con 0x7f34400f94d0 2024-02-19T21:16:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(736..736 src has 623..736) v4 ==== 8704+0+0 (unknown 505538580 0 0) 0x7f34340791d0 con 0x7f34400f94d0 2024-02-19T21:16:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 osd_map(736..736 src has 623..736) v4 2024-02-19T21:16:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f3427fff700 3 client.50531.objecter handle_osd_map got epochs [736,736] > 0 2024-02-19T21:16:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f3427fff700 3 client.50531.objecter handle_osd_map decoding full epoch 736 2024-02-19T21:16:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f3427fff700 20 client.50531.objecter dump_active .. 0 homeless 2024-02-19T21:16:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v736) v1 ==== 54+0+16 (unknown 1650709821 0 2420000192) 0x7f343407b5e0 con 0x7f34400f94d0 2024-02-19T21:16:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f3427fff700 10 client.50531.objecter ms_handle_connect 0x7f3428063d10 2024-02-19T21:16:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f34451f5700 10 client.50531.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 1 2024-02-19T21:16:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c0051a0 osd=0 3 2024-02-19T21:16:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c0051a0 osd=0 3 2024-02-19T21:16:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c0051a0 osd=0 3 2024-02-19T21:16:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 1 2024-02-19T21:16:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c0051a0 osd=0 3 2024-02-19T21:16:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 0 1 2024-02-19T21:16:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c0051a0 osd=0 4 2024-02-19T21:16:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f34451f5700 10 client.50531.objecter _send_command 1 2024-02-19T21:16:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.844+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f342c004950 con 0x7f342c004530 2024-02-19T21:16:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.845+0000 7f3427fff700 10 client.50531.objecter ms_handle_connect 0x7f342c004530 2024-02-19T21:16:29.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.845+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f342c004950 con 0x7f342c004530 2024-02-19T21:16:29.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.845+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:16:29.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.845+0000 7f3427fff700 10 client.50531.objecter _finish_command 1 = system:0 2024-02-19T21:16:29.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.845+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c0051a0 osd=0 4 2024-02-19T21:16:29.858 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.845+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 0 1 2024-02-19T21:16:29.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.860+0000 7f34451f5700 10 client.50531.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:29.860 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.860+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 2 2024-02-19T21:16:29.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.860+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c0051a0 osd=0 3 2024-02-19T21:16:29.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.860+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c0051a0 osd=0 3 2024-02-19T21:16:29.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.860+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c0051a0 osd=0 3 2024-02-19T21:16:29.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.860+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 2 2024-02-19T21:16:29.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.860+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c0051a0 osd=0 3 2024-02-19T21:16:29.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.860+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 0 2 2024-02-19T21:16:29.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.860+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c0051a0 osd=0 4 2024-02-19T21:16:29.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.860+0000 7f34451f5700 10 client.50531.objecter _send_command 2 2024-02-19T21:16:29.861 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.860+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f342c004d30 con 0x7f342c004530 2024-02-19T21:16:29.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f342c004d30 con 0x7f342c004530 2024-02-19T21:16:29.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:29.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f3427fff700 10 client.50531.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:29.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c0051a0 osd=0 4 2024-02-19T21:16:29.862 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 0 2 2024-02-19T21:16:29.863 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:16:29.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f34451f5700 10 client.50531.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:29.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 3 2024-02-19T21:16:29.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c003fe0 osd=1 3 2024-02-19T21:16:29.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c003fe0 osd=1 3 2024-02-19T21:16:29.863 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c003fe0 osd=1 3 2024-02-19T21:16:29.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 3 2024-02-19T21:16:29.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c003fe0 osd=1 3 2024-02-19T21:16:29.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 1 3 2024-02-19T21:16:29.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c003fe0 osd=1 4 2024-02-19T21:16:29.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f34451f5700 10 client.50531.objecter _send_command 3 2024-02-19T21:16:29.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.862+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f342c010410 con 0x7f342c004150 2024-02-19T21:16:29.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.863+0000 7f3427fff700 10 client.50531.objecter ms_handle_connect 0x7f342c004150 2024-02-19T21:16:29.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.863+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f342c010410 con 0x7f342c004150 2024-02-19T21:16:29.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.863+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:16:29.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.863+0000 7f3427fff700 10 client.50531.objecter _finish_command 3 = system:0 2024-02-19T21:16:29.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.863+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c003fe0 osd=1 4 2024-02-19T21:16:29.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.863+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 1 3 2024-02-19T21:16:29.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.878+0000 7f34451f5700 10 client.50531.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:29.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.878+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 4 2024-02-19T21:16:29.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.878+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c003fe0 osd=1 3 2024-02-19T21:16:29.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.878+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c003fe0 osd=1 3 2024-02-19T21:16:29.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.878+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c003fe0 osd=1 3 2024-02-19T21:16:29.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.878+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 4 2024-02-19T21:16:29.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.878+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c003fe0 osd=1 3 2024-02-19T21:16:29.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.878+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 1 4 2024-02-19T21:16:29.878 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.878+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c003fe0 osd=1 4 2024-02-19T21:16:29.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.878+0000 7f34451f5700 10 client.50531.objecter _send_command 4 2024-02-19T21:16:29.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.878+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f342c0018b0 con 0x7f342c004150 2024-02-19T21:16:29.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.879+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f342c0018b0 con 0x7f342c004150 2024-02-19T21:16:29.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.879+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:29.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.879+0000 7f3427fff700 10 client.50531.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:29.879 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.879+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c003fe0 osd=1 4 2024-02-19T21:16:29.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.879+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 1 4 2024-02-19T21:16:29.880 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:16:29.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.879+0000 7f34451f5700 10 client.50531.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:29.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.879+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 5 2024-02-19T21:16:29.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.880+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c003630 osd=2 3 2024-02-19T21:16:29.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.880+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c003630 osd=2 3 2024-02-19T21:16:29.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.880+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c003630 osd=2 3 2024-02-19T21:16:29.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.880+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 5 2024-02-19T21:16:29.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.880+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c003630 osd=2 3 2024-02-19T21:16:29.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.880+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 2 5 2024-02-19T21:16:29.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.880+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c003630 osd=2 4 2024-02-19T21:16:29.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.880+0000 7f34451f5700 10 client.50531.objecter _send_command 5 2024-02-19T21:16:29.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.880+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f342c0110b0 con 0x7f342c010580 2024-02-19T21:16:29.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.880+0000 7f3427fff700 10 client.50531.objecter ms_handle_connect 0x7f342c010580 2024-02-19T21:16:29.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.880+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f342c0110b0 con 0x7f342c010580 2024-02-19T21:16:29.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.880+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:16:29.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.880+0000 7f3427fff700 10 client.50531.objecter _finish_command 5 = system:0 2024-02-19T21:16:29.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.880+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c003630 osd=2 4 2024-02-19T21:16:29.882 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.880+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 2 5 2024-02-19T21:16:29.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.895+0000 7f34451f5700 10 client.50531.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:29.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.895+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 6 2024-02-19T21:16:29.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.895+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c003630 osd=2 3 2024-02-19T21:16:29.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.895+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c003630 osd=2 3 2024-02-19T21:16:29.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.895+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c003630 osd=2 3 2024-02-19T21:16:29.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.895+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 6 2024-02-19T21:16:29.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.895+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c003630 osd=2 3 2024-02-19T21:16:29.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.895+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 2 6 2024-02-19T21:16:29.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.895+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c003630 osd=2 4 2024-02-19T21:16:29.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.895+0000 7f34451f5700 10 client.50531.objecter _send_command 6 2024-02-19T21:16:29.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.895+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f342c0119f0 con 0x7f342c010580 2024-02-19T21:16:29.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f342c0119f0 con 0x7f342c010580 2024-02-19T21:16:29.896 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:29.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f3427fff700 10 client.50531.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:29.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c003630 osd=2 4 2024-02-19T21:16:29.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 2 6 2024-02-19T21:16:29.897 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:16:29.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f34451f5700 10 client.50531.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:29.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 7 2024-02-19T21:16:29.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c003350 osd=3 3 2024-02-19T21:16:29.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c003350 osd=3 3 2024-02-19T21:16:29.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c003350 osd=3 3 2024-02-19T21:16:29.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 7 2024-02-19T21:16:29.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c003350 osd=3 3 2024-02-19T21:16:29.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 3 7 2024-02-19T21:16:29.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c003350 osd=3 4 2024-02-19T21:16:29.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f34451f5700 10 client.50531.objecter _send_command 7 2024-02-19T21:16:29.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f342c019e70 con 0x7f342c016290 2024-02-19T21:16:29.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.897+0000 7f3427fff700 10 client.50531.objecter ms_handle_connect 0x7f342c016290 2024-02-19T21:16:29.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.898+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f342c019e70 con 0x7f342c016290 2024-02-19T21:16:29.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.898+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:16:29.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.898+0000 7f3427fff700 10 client.50531.objecter _finish_command 7 = system:0 2024-02-19T21:16:29.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.898+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c003350 osd=3 4 2024-02-19T21:16:29.899 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.898+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 3 7 2024-02-19T21:16:29.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.912+0000 7f34451f5700 10 client.50531.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:29.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.912+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 8 2024-02-19T21:16:29.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.912+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c003350 osd=3 3 2024-02-19T21:16:29.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.912+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c003350 osd=3 3 2024-02-19T21:16:29.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.912+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c003350 osd=3 3 2024-02-19T21:16:29.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.912+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 8 2024-02-19T21:16:29.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.912+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c003350 osd=3 3 2024-02-19T21:16:29.912 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.912+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 3 8 2024-02-19T21:16:29.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.912+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c003350 osd=3 4 2024-02-19T21:16:29.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.912+0000 7f34451f5700 10 client.50531.objecter _send_command 8 2024-02-19T21:16:29.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.912+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f342c011460 con 0x7f342c016290 2024-02-19T21:16:29.913 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:29.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f342c011460 con 0x7f342c016290 2024-02-19T21:16:29.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:29.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f3427fff700 10 client.50531.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:29.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c003350 osd=3 4 2024-02-19T21:16:29.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 3 8 2024-02-19T21:16:29.914 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:16:29.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f34451f5700 10 client.50531.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:29.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 9 2024-02-19T21:16:29.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c011220 osd=4 3 2024-02-19T21:16:29.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c011220 osd=4 3 2024-02-19T21:16:29.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c011220 osd=4 3 2024-02-19T21:16:29.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 9 2024-02-19T21:16:29.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c011220 osd=4 3 2024-02-19T21:16:29.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 4 9 2024-02-19T21:16:29.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c011220 osd=4 4 2024-02-19T21:16:29.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f34451f5700 10 client.50531.objecter _send_command 9 2024-02-19T21:16:29.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f342c0113d0 con 0x7f342c012870 2024-02-19T21:16:29.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.914+0000 7f3427fff700 10 client.50531.objecter ms_handle_connect 0x7f342c012870 2024-02-19T21:16:29.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.915+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f342c0113d0 con 0x7f342c012870 2024-02-19T21:16:29.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.915+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:16:29.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.915+0000 7f3427fff700 10 client.50531.objecter _finish_command 9 = system:0 2024-02-19T21:16:29.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.915+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c011220 osd=4 4 2024-02-19T21:16:29.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.915+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 4 9 2024-02-19T21:16:29.917 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:29.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:29.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.930+0000 7f34451f5700 10 client.50531.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:29.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.930+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 10 2024-02-19T21:16:29.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.930+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c011220 osd=4 3 2024-02-19T21:16:29.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.930+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c011220 osd=4 3 2024-02-19T21:16:29.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.930+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c011220 osd=4 3 2024-02-19T21:16:29.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.930+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 10 2024-02-19T21:16:29.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.930+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c011220 osd=4 3 2024-02-19T21:16:29.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.930+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 4 10 2024-02-19T21:16:29.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.930+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c011220 osd=4 4 2024-02-19T21:16:29.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.930+0000 7f34451f5700 10 client.50531.objecter _send_command 10 2024-02-19T21:16:29.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.930+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f342c0113d0 con 0x7f342c012870 2024-02-19T21:16:29.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f342c0113d0 con 0x7f342c012870 2024-02-19T21:16:29.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:29.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f3427fff700 10 client.50531.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:29.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c011220 osd=4 4 2024-02-19T21:16:29.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 4 10 2024-02-19T21:16:29.932 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:16:29.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f34451f5700 10 client.50531.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:29.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 11 2024-02-19T21:16:29.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c0113d0 osd=5 3 2024-02-19T21:16:29.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c0113d0 osd=5 3 2024-02-19T21:16:29.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c0113d0 osd=5 3 2024-02-19T21:16:29.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 11 2024-02-19T21:16:29.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c0113d0 osd=5 3 2024-02-19T21:16:29.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 5 11 2024-02-19T21:16:29.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c0113d0 osd=5 4 2024-02-19T21:16:29.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f34451f5700 10 client.50531.objecter _send_command 11 2024-02-19T21:16:29.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f342c01c3e0 con 0x7f342c01be00 2024-02-19T21:16:29.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.932+0000 7f3427fff700 10 client.50531.objecter ms_handle_connect 0x7f342c01be00 2024-02-19T21:16:29.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.933+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f342c01c3e0 con 0x7f342c01be00 2024-02-19T21:16:29.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.933+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:16:29.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.933+0000 7f3427fff700 10 client.50531.objecter _finish_command 11 = system:0 2024-02-19T21:16:29.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.933+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c0113d0 osd=5 4 2024-02-19T21:16:29.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.933+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 5 11 2024-02-19T21:16:29.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.947+0000 7f34451f5700 10 client.50531.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:29.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.947+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 12 2024-02-19T21:16:29.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.947+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c0113d0 osd=5 3 2024-02-19T21:16:29.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.947+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c0113d0 osd=5 3 2024-02-19T21:16:29.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.947+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c0113d0 osd=5 3 2024-02-19T21:16:29.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.947+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 12 2024-02-19T21:16:29.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.947+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c0113d0 osd=5 3 2024-02-19T21:16:29.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.947+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 5 12 2024-02-19T21:16:29.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.947+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c0113d0 osd=5 4 2024-02-19T21:16:29.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.947+0000 7f34451f5700 10 client.50531.objecter _send_command 12 2024-02-19T21:16:29.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.947+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f342c01c620 con 0x7f342c01be00 2024-02-19T21:16:29.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.949+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f343404a890 con 0x7f342c01be00 2024-02-19T21:16:29.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.949+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:29.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.949+0000 7f3427fff700 10 client.50531.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:29.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.949+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c0113d0 osd=5 4 2024-02-19T21:16:29.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.949+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 5 12 2024-02-19T21:16:29.950 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:16:29.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.950+0000 7f34451f5700 10 client.50531.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:29.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.950+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 13 2024-02-19T21:16:29.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.950+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c01c3e0 osd=6 3 2024-02-19T21:16:29.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.950+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c01c3e0 osd=6 3 2024-02-19T21:16:29.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.950+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c01c3e0 osd=6 3 2024-02-19T21:16:29.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.950+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 13 2024-02-19T21:16:29.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.950+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c01c3e0 osd=6 3 2024-02-19T21:16:29.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.950+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 6 13 2024-02-19T21:16:29.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.950+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c01c3e0 osd=6 4 2024-02-19T21:16:29.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.950+0000 7f34451f5700 10 client.50531.objecter _send_command 13 2024-02-19T21:16:29.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.950+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f342c027170 con 0x7f342c023470 2024-02-19T21:16:29.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.950+0000 7f3427fff700 10 client.50531.objecter ms_handle_connect 0x7f342c023470 2024-02-19T21:16:29.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.951+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f342c027170 con 0x7f342c023470 2024-02-19T21:16:29.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.951+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:16:29.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.951+0000 7f3427fff700 10 client.50531.objecter _finish_command 13 = system:0 2024-02-19T21:16:29.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.951+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c01c3e0 osd=6 4 2024-02-19T21:16:29.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.951+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 6 13 2024-02-19T21:16:29.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.965+0000 7f34451f5700 10 client.50531.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:29.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.965+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 14 2024-02-19T21:16:29.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.965+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c01c3e0 osd=6 3 2024-02-19T21:16:29.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.965+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c01c3e0 osd=6 3 2024-02-19T21:16:29.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.965+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c01c3e0 osd=6 3 2024-02-19T21:16:29.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.965+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 14 2024-02-19T21:16:29.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.965+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c01c3e0 osd=6 3 2024-02-19T21:16:29.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.965+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 6 14 2024-02-19T21:16:29.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.965+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c01c3e0 osd=6 4 2024-02-19T21:16:29.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.965+0000 7f34451f5700 10 client.50531.objecter _send_command 14 2024-02-19T21:16:29.965 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.965+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f342c027170 con 0x7f342c023470 2024-02-19T21:16:29.966 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f342c027170 con 0x7f342c023470 2024-02-19T21:16:29.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:29.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f3427fff700 10 client.50531.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:29.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c01c3e0 osd=6 4 2024-02-19T21:16:29.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 6 14 2024-02-19T21:16:29.967 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:16:29.967 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f34451f5700 10 client.50531.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:29.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 15 2024-02-19T21:16:29.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c027170 osd=7 3 2024-02-19T21:16:29.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c027170 osd=7 3 2024-02-19T21:16:29.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c027170 osd=7 3 2024-02-19T21:16:29.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 15 2024-02-19T21:16:29.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c027170 osd=7 3 2024-02-19T21:16:29.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 7 15 2024-02-19T21:16:29.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c027170 osd=7 4 2024-02-19T21:16:29.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f34451f5700 10 client.50531.objecter _send_command 15 2024-02-19T21:16:29.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f342c0283c0 con 0x7f342c028e70 2024-02-19T21:16:29.968 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.967+0000 7f3427fff700 10 client.50531.objecter ms_handle_connect 0x7f342c028e70 2024-02-19T21:16:29.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.968+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f342c0283c0 con 0x7f342c028e70 2024-02-19T21:16:29.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.968+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:16:29.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.968+0000 7f3427fff700 10 client.50531.objecter _finish_command 15 = system:0 2024-02-19T21:16:29.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.968+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c027170 osd=7 4 2024-02-19T21:16:29.969 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.968+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 7 15 2024-02-19T21:16:29.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.983+0000 7f34451f5700 10 client.50531.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:29.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.983+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign -1 16 2024-02-19T21:16:29.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.983+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c027170 osd=7 3 2024-02-19T21:16:29.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.983+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c027170 osd=7 3 2024-02-19T21:16:29.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.983+0000 7f34451f5700 20 client.50531.objecter _get_session s=0x7f342c027170 osd=7 3 2024-02-19T21:16:29.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.983+0000 7f34451f5700 15 client.50531.objecter _session_command_op_remove -1 16 2024-02-19T21:16:29.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.983+0000 7f34451f5700 20 client.50531.objecter get_session s=0x7f342c027170 osd=7 3 2024-02-19T21:16:29.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.983+0000 7f34451f5700 15 client.50531.objecter _session_command_op_assign 7 16 2024-02-19T21:16:29.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.983+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c027170 osd=7 4 2024-02-19T21:16:29.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.983+0000 7f34451f5700 10 client.50531.objecter _send_command 16 2024-02-19T21:16:29.983 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.983+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f342c0283c0 con 0x7f342c028e70 2024-02-19T21:16:29.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.984+0000 7f3427fff700 1 -- 172.21.15.163:0/2424008614 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f342c0283c0 con 0x7f342c028e70 2024-02-19T21:16:29.984 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.984+0000 7f3427fff700 10 client.50531.objecter ms_dispatch 0x7f3440001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:29.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.984+0000 7f3427fff700 10 client.50531.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:29.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.984+0000 7f3427fff700 20 client.50531.objecter put_session s=0x7f342c027170 osd=7 4 2024-02-19T21:16:29.985 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.984+0000 7f3427fff700 15 client.50531.objecter _session_command_op_remove 7 16 2024-02-19T21:16:29.985 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:16:29.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 10 librados: watch_flush enter 2024-02-19T21:16:29.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 10 librados: watch_flush exit 2024-02-19T21:16:29.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 10 client.50531.objecter close_session for osd.0 2024-02-19T21:16:29.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 >> v1:172.21.15.163:6801/1559418115 conn(0x7f342c004530 legacy=0x7f342c002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:29.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c0051a0 osd=0 1 2024-02-19T21:16:29.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 10 client.50531.objecter close_session for osd.1 2024-02-19T21:16:29.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 >> v1:172.21.15.163:6805/3125047432 conn(0x7f342c004150 legacy=0x7f342c00ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:29.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c003fe0 osd=1 1 2024-02-19T21:16:29.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 10 client.50531.objecter close_session for osd.2 2024-02-19T21:16:29.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 >> v1:172.21.15.163:6809/1563158487 conn(0x7f342c010580 legacy=0x7f342c0109a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:29.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c003630 osd=2 1 2024-02-19T21:16:29.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 10 client.50531.objecter close_session for osd.3 2024-02-19T21:16:29.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 >> v1:172.21.15.163:6813/2112279769 conn(0x7f342c016290 legacy=0x7f342c018680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:29.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c003350 osd=3 1 2024-02-19T21:16:29.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 10 client.50531.objecter close_session for osd.4 2024-02-19T21:16:29.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 >> v1:172.21.15.193:6800/3129716965 conn(0x7f342c012870 legacy=0x7f342c01a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:29.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c011220 osd=4 1 2024-02-19T21:16:29.987 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 10 client.50531.objecter close_session for osd.5 2024-02-19T21:16:29.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f343f59e700 1 -- 172.21.15.163:0/2424008614 reap_dead start 2024-02-19T21:16:29.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 >> v1:172.21.15.193:6804/805161991 conn(0x7f342c01be00 legacy=0x7f342c021840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:29.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f343f59e700 1 -- 172.21.15.163:0/2424008614 reap_dead start 2024-02-19T21:16:29.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f343f59e700 1 -- 172.21.15.163:0/2424008614 reap_dead start 2024-02-19T21:16:29.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c0113d0 osd=5 1 2024-02-19T21:16:29.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 10 client.50531.objecter close_session for osd.6 2024-02-19T21:16:29.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 >> v1:172.21.15.193:6808/787604615 conn(0x7f342c023470 legacy=0x7f342c0258c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:29.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c01c3e0 osd=6 1 2024-02-19T21:16:29.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 10 client.50531.objecter close_session for osd.7 2024-02-19T21:16:29.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 >> v1:172.21.15.193:6812/157927432 conn(0x7f342c028e70 legacy=0x7f342c0292d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:29.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 20 client.50531.objecter put_session s=0x7f342c027170 osd=7 1 2024-02-19T21:16:29.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 20 client.50531.objecter shutdown clearing up homeless session... 2024-02-19T21:16:29.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 10 client.50531.objecter successfully canceled tick 2024-02-19T21:16:29.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 >> v1:172.21.15.163:6800/1552514681 conn(0x7f3428063d10 legacy=0x7f34280661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:29.988 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 >> v1:172.21.15.163:6790/0 conn(0x7f34400f94d0 legacy=0x7f34400ffe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:29.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f343f59e700 1 -- 172.21.15.163:0/2424008614 reap_dead start 2024-02-19T21:16:29.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 shutdown_connections 2024-02-19T21:16:29.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 >> 172.21.15.163:0/2424008614 conn(0x7f34400f2470 msgr2=0x7f3440107770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:29.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 shutdown_connections 2024-02-19T21:16:29.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 1 -- 172.21.15.163:0/2424008614 wait complete. 2024-02-19T21:16:29.989 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.985+0000 7f34451f5700 1 librados: shutdown 2024-02-19T21:16:29.989 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:16:29.989 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:16:29.989 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:16:29.989 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:16:29.989 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:16:29.989 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:16:29.989 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:16:29.989 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:16:30.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.680+0000 7fadccfb6700 1 -- 172.21.15.163:0/2189956091 >> v1:172.21.15.163:6790/0 conn(0x7fadc80ff610 legacy=0x7fadc8101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:30.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.680+0000 7fadccfb6700 1 -- 172.21.15.163:0/2189956091 shutdown_connections 2024-02-19T21:16:30.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.680+0000 7fadccfb6700 1 -- 172.21.15.163:0/2189956091 >> 172.21.15.163:0/2189956091 conn(0x7fadc80f23d0 msgr2=0x7fadc80f47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:30.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.680+0000 7fadccfb6700 1 -- 172.21.15.163:0/2189956091 shutdown_connections 2024-02-19T21:16:30.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.680+0000 7fadccfb6700 1 -- 172.21.15.163:0/2189956091 wait complete. 2024-02-19T21:16:30.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.680+0000 7fadccfb6700 1 librados: starting msgr at 2024-02-19T21:16:30.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.680+0000 7fadccfb6700 1 librados: starting objecter 2024-02-19T21:16:30.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadccfb6700 1 Processor -- start 2024-02-19T21:16:30.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadccfb6700 1 -- start start 2024-02-19T21:16:30.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadccfb6700 1 librados: setting wanted keys 2024-02-19T21:16:30.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadccfb6700 1 librados: calling monclient init 2024-02-19T21:16:30.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadccfb6700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fadc80fbed0 con 0x7fadc8103150 2024-02-19T21:16:30.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadccfb6700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fadc80fc0a0 con 0x7fadc80fc810 2024-02-19T21:16:30.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadccfb6700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fadc811ba20 con 0x7fadc80ff610 2024-02-19T21:16:30.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadc5d9b700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fadc80ff610 0x7fadc8114bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:58082/0 (socket says 172.21.15.163:58082) 2024-02-19T21:16:30.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadc5d9b700 1 -- 172.21.15.163:0/1427034038 learned_addr learned my addr 172.21.15.163:0/1427034038 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:30.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadb77fe700 10 client.?.objecter ms_handle_connect 0x7fadc8103150 2024-02-19T21:16:30.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadb77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:30.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadb77fe700 10 client.?.objecter ms_handle_connect 0x7fadc80ff610 2024-02-19T21:16:30.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadb77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:30.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadb77fe700 10 client.?.objecter ms_handle_connect 0x7fadc80fc810 2024-02-19T21:16:30.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadb77fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:30.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.681+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712378294 0 0) 0x7fadc80fbed0 con 0x7fadc8103150 2024-02-19T21:16:30.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadb00038e0 con 0x7fadc8103150 2024-02-19T21:16:30.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314025687 0 0) 0x7fadc811ba20 con 0x7fadc80ff610 2024-02-19T21:16:30.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc80fbed0 con 0x7fadc80ff610 2024-02-19T21:16:30.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781225096 0 0) 0x7fadc80fc0a0 con 0x7fadc80fc810 2024-02-19T21:16:30.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc811ba20 con 0x7fadc80fc810 2024-02-19T21:16:30.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1486671863 0 0) 0x7fadb00038e0 con 0x7fadc8103150 2024-02-19T21:16:30.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadc80fc0a0 con 0x7fadc8103150 2024-02-19T21:16:30.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fadc0001d70 con 0x7fadc8103150 2024-02-19T21:16:30.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2798399428 0 0) 0x7fadc80fbed0 con 0x7fadc80ff610 2024-02-19T21:16:30.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadb00038e0 con 0x7fadc80ff610 2024-02-19T21:16:30.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fadbc002c00 con 0x7fadc80ff610 2024-02-19T21:16:30.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3689551379 0 0) 0x7fadc811ba20 con 0x7fadc80fc810 2024-02-19T21:16:30.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadc80fbed0 con 0x7fadc80fc810 2024-02-19T21:16:30.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fadb80018c0 con 0x7fadc80fc810 2024-02-19T21:16:30.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 590970735 0 0) 0x7fadc80fc0a0 con 0x7fadc8103150 2024-02-19T21:16:30.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 >> v1:172.21.15.163:6790/0 conn(0x7fadc80ff610 legacy=0x7fadc8114bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:30.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 >> v1:172.21.15.193:6789/0 conn(0x7fadc80fc810 legacy=0x7fadc80fb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:30.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc811cbc0 con 0x7fadc8103150 2024-02-19T21:16:30.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadccfb6700 1 -- 172.21.15.163:0/1427034038 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc811bc10 con 0x7fadc8103150 2024-02-19T21:16:30.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadccfb6700 10 client.23796.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:30.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadccfb6700 1 -- 172.21.15.163:0/1427034038 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc811c080 con 0x7fadc8103150 2024-02-19T21:16:30.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadccfb6700 1 librados: init done 2024-02-19T21:16:30.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fadc00026e0 con 0x7fadc8103150 2024-02-19T21:16:30.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.682+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fadc0003770 con 0x7fadc8103150 2024-02-19T21:16:30.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.683+0000 7fadccfb6700 1 -- 172.21.15.163:0/1427034038 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada8004f20 con 0x7fadc8103150 2024-02-19T21:16:30.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.683+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fadc0002240 con 0x7fadc8103150 2024-02-19T21:16:30.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.684+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(736..736 src has 623..736) v4 ==== 8704+0+0 (unknown 505538580 0 0) 0x7fadc0076a60 con 0x7fadc8103150 2024-02-19T21:16:30.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.684+0000 7fadb77fe700 10 client.23796.objecter ms_dispatch 0x7fadc8001550 osd_map(736..736 src has 623..736) v4 2024-02-19T21:16:30.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.684+0000 7fadb77fe700 3 client.23796.objecter handle_osd_map got epochs [736,736] > 0 2024-02-19T21:16:30.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.684+0000 7fadb77fe700 3 client.23796.objecter handle_osd_map decoding full epoch 736 2024-02-19T21:16:30.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.684+0000 7fadb77fe700 20 client.23796.objecter dump_active .. 0 homeless 2024-02-19T21:16:30.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.684+0000 7fadb77fe700 10 client.23796.objecter ms_handle_connect 0x7fadb0063c20 2024-02-19T21:16:30.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.688+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fadc0049bf0 con 0x7fadc8103150 2024-02-19T21:16:30.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:29.873+0000 7fadccfb6700 1 -- 172.21.15.163:0/1427034038 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7fada80051d0 con 0x7fadc8103150 2024-02-19T21:16:30.018 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:30.047 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:30.230 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:30.249 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:30.252 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:30.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:30.353 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:30.385 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:30.407 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:30.410 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:30.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:30.511 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:30.541 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:30.562 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:30.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:30.560+0000 7fadb77fe700 1 -- 172.21.15.163:0/1427034038 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v737) v1 ==== 87+0+0 (unknown 3661237169 0 0) 0x7fadc0048000 con 0x7fadc8103150 2024-02-19T21:16:30.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:30.563+0000 7fadccfb6700 10 librados: watch_flush enter 2024-02-19T21:16:30.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:30.564+0000 7fadccfb6700 10 librados: watch_flush exit 2024-02-19T21:16:30.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:30.564+0000 7fadccfb6700 20 client.23796.objecter shutdown clearing up homeless session... 2024-02-19T21:16:30.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:30.564+0000 7fadccfb6700 10 client.23796.objecter successfully canceled tick 2024-02-19T21:16:30.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:30.564+0000 7fadccfb6700 1 -- 172.21.15.163:0/1427034038 >> v1:172.21.15.163:6800/1552514681 conn(0x7fadb0063c20 legacy=0x7fadb00660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:30.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:30.564+0000 7fadccfb6700 1 -- 172.21.15.163:0/1427034038 >> v1:172.21.15.163:6789/0 conn(0x7fadc8103150 legacy=0x7fadc81182e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:30.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:30.564+0000 7fadccfb6700 1 -- 172.21.15.163:0/1427034038 shutdown_connections 2024-02-19T21:16:30.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:30.564+0000 7fadccfb6700 1 -- 172.21.15.163:0/1427034038 >> 172.21.15.163:0/1427034038 conn(0x7fadc80f23d0 msgr2=0x7fadc81065c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:30.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:30.564+0000 7fadccfb6700 1 -- 172.21.15.163:0/1427034038 shutdown_connections 2024-02-19T21:16:30.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:30.564+0000 7fadccfb6700 1 -- 172.21.15.163:0/1427034038 wait complete. 2024-02-19T21:16:30.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:30.564+0000 7fadccfb6700 1 librados: shutdown 2024-02-19T21:16:30.566 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:16:30.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:30.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:30.570 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:29.028156+0000 osd.2 (osd.2) 298 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:16:30.570 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:29.029273+0000 osd.2 (osd.2) 299 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:16:30.570 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:29.229642+0000 osd.1 (osd.1) 328 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:16:30.570 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:29.232582+0000 osd.1 (osd.1) 329 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:16:30.570 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:29.322247+0000 mgr.y (mgr.14550) 617 : cluster [DBG] pgmap v1112: 214 pgs: 32 peering, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 2.7 KiB/s wr, 0 op/s; 284 B/s, 0 objects/s recovering 2024-02-19T21:16:30.570 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:29.844378+0000 mon.c (mon.2) 431 : audit [DBG] from='client.? v1:172.21.15.163:0/2424008614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:30.570 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:29.874104+0000 mon.a (mon.0) 3528 : audit [INF] from='client.? v1:172.21.15.163:0/1427034038' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:16:30.570 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:29.893097+0000 osd.3 (osd.3) 313 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:16:30.570 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:29.894601+0000 osd.3 (osd.3) 314 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:16:30.570 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:30.207390+0000 osd.4 (osd.4) 265 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:16:30.570 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:30.209332+0000 osd.4 (osd.4) 266 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:16:30.667 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:30.694 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:29.028156+0000 osd.2 (osd.2) 298 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:16:30.694 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:29.029273+0000 osd.2 (osd.2) 299 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:16:30.694 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:29.229642+0000 osd.1 (osd.1) 328 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:16:30.694 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:29.232582+0000 osd.1 (osd.1) 329 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:16:30.694 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:29.322247+0000 mgr.y (mgr.14550) 617 : cluster [DBG] pgmap v1112: 214 pgs: 32 peering, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 2.7 KiB/s wr, 0 op/s; 284 B/s, 0 objects/s recovering 2024-02-19T21:16:30.694 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:29.844378+0000 mon.c (mon.2) 431 : audit [DBG] from='client.? v1:172.21.15.163:0/2424008614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:30.694 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:29.874104+0000 mon.a (mon.0) 3528 : audit [INF] from='client.? v1:172.21.15.163:0/1427034038' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:16:30.694 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:29.893097+0000 osd.3 (osd.3) 313 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:16:30.694 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:29.894601+0000 osd.3 (osd.3) 314 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:16:30.694 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:30.207390+0000 osd.4 (osd.4) 265 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:16:30.694 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:30.209332+0000 osd.4 (osd.4) 266 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:16:30.695 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:30.715 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:30.718 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:30.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:30.819 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:30.847 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:30.866 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:30.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:30.970 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:30.975 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:29.028156+0000 osd.2 (osd.2) 298 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:16:30.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:29.029273+0000 osd.2 (osd.2) 299 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:16:30.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:29.229642+0000 osd.1 (osd.1) 328 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:16:30.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:29.232582+0000 osd.1 (osd.1) 329 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:16:30.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:29.322247+0000 mgr.y (mgr.14550) 617 : cluster [DBG] pgmap v1112: 214 pgs: 32 peering, 182 active+clean; 3.8 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 2.7 KiB/s wr, 0 op/s; 284 B/s, 0 objects/s recovering 2024-02-19T21:16:30.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:29.844378+0000 mon.c (mon.2) 431 : audit [DBG] from='client.? v1:172.21.15.163:0/2424008614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:30.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:29.874104+0000 mon.a (mon.0) 3528 : audit [INF] from='client.? v1:172.21.15.163:0/1427034038' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:16:30.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:29.893097+0000 osd.3 (osd.3) 313 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:16:30.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:29.894601+0000 osd.3 (osd.3) 314 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:16:30.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:30.207390+0000 osd.4 (osd.4) 265 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:16:30.976 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:30.209332+0000 osd.4 (osd.4) 266 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:16:30.994 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:16:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:16:30.996 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:31.015 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:31.018 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:31.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:31.073 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 5, 7, 3, 4, 6, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:16:31.073 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:16:31.074 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [1, 2, 5, 7, 3, 4, 6, 0] 2024-02-19T21:16:31.074 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd out 2 2024-02-19T21:16:31.118 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:31.145 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:31.163 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:31.166 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:31.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:31.267 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:31.294 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:31.313 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:31.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:31.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:31.417 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:31.443 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:31.461 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:31.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:31.565 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:31.593 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:31.666 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:31.668 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:31.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:31.770 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:31.796 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:31.814 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:31.817 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:31.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:31.918 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:31.943 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:30.002831+0000 osd.5 (osd.5) 308 : cluster [DBG] 2.1c deep-scrub starts 2024-02-19T21:16:31.943 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:30.005719+0000 osd.5 (osd.5) 309 : cluster [DBG] 2.1c deep-scrub ok 2024-02-19T21:16:31.943 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:30.014775+0000 osd.2 (osd.2) 300 : cluster [DBG] 16.17 scrub starts 2024-02-19T21:16:31.943 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:30.017764+0000 osd.2 (osd.2) 301 : cluster [DBG] 16.17 scrub ok 2024-02-19T21:16:31.943 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:30.554315+0000 mon.a (mon.0) 3529 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:16:31.943 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:30.561102+0000 mon.a (mon.0) 3530 : audit [INF] from='client.? v1:172.21.15.163:0/1427034038' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:16:31.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:30.561151+0000 mon.a (mon.0) 3531 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2024-02-19T21:16:31.944 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:30.754877+0000 mgr.y (mgr.14550) 618 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:31.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:30.002831+0000 osd.5 (osd.5) 308 : cluster [DBG] 2.1c deep-scrub starts 2024-02-19T21:16:31.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:30.005719+0000 osd.5 (osd.5) 309 : cluster [DBG] 2.1c deep-scrub ok 2024-02-19T21:16:31.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:30.014775+0000 osd.2 (osd.2) 300 : cluster [DBG] 16.17 scrub starts 2024-02-19T21:16:31.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:30.017764+0000 osd.2 (osd.2) 301 : cluster [DBG] 16.17 scrub ok 2024-02-19T21:16:31.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:30.554315+0000 mon.a (mon.0) 3529 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:16:31.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:30.561102+0000 mon.a (mon.0) 3530 : audit [INF] from='client.? v1:172.21.15.163:0/1427034038' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:16:31.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:30.561151+0000 mon.a (mon.0) 3531 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2024-02-19T21:16:31.944 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:30.754877+0000 mgr.y (mgr.14550) 618 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:31.945 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:32.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:30.002831+0000 osd.5 (osd.5) 308 : cluster [DBG] 2.1c deep-scrub starts 2024-02-19T21:16:32.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:30.005719+0000 osd.5 (osd.5) 309 : cluster [DBG] 2.1c deep-scrub ok 2024-02-19T21:16:32.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:30.014775+0000 osd.2 (osd.2) 300 : cluster [DBG] 16.17 scrub starts 2024-02-19T21:16:32.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:30.017764+0000 osd.2 (osd.2) 301 : cluster [DBG] 16.17 scrub ok 2024-02-19T21:16:32.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:30.554315+0000 mon.a (mon.0) 3529 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:16:32.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:30.561102+0000 mon.a (mon.0) 3530 : audit [INF] from='client.? v1:172.21.15.163:0/1427034038' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:16:32.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:30.561151+0000 mon.a (mon.0) 3531 : cluster [DBG] osdmap e737: 8 total, 8 up, 8 in 2024-02-19T21:16:32.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:30.754877+0000 mgr.y (mgr.14550) 618 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:32.144 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:32.146 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:32.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:32.247 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:32.274 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:32.722 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:32.726 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:32.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:32.760 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:32.827 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:32.853 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:31.322898+0000 mgr.y (mgr.14550) 619 : cluster [DBG] pgmap v1114: 214 pgs: 214 active+clean; 3.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 8.4 KiB/s wr, 2 op/s; 247 B/s, 0 objects/s recovering 2024-02-19T21:16:32.853 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:31.565030+0000 mon.a (mon.0) 3532 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 19 pgs inactive, 25 pgs peering) 2024-02-19T21:16:32.853 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:31.322898+0000 mgr.y (mgr.14550) 619 : cluster [DBG] pgmap v1114: 214 pgs: 214 active+clean; 3.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 8.4 KiB/s wr, 2 op/s; 247 B/s, 0 objects/s recovering 2024-02-19T21:16:32.854 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:31.565030+0000 mon.a (mon.0) 3532 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 19 pgs inactive, 25 pgs peering) 2024-02-19T21:16:32.855 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:32.994 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:16:33.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:31.322898+0000 mgr.y (mgr.14550) 619 : cluster [DBG] pgmap v1114: 214 pgs: 214 active+clean; 3.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 8.4 KiB/s wr, 2 op/s; 247 B/s, 0 objects/s recovering 2024-02-19T21:16:33.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:31.565030+0000 mon.a (mon.0) 3532 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 19 pgs inactive, 25 pgs peering) 2024-02-19T21:16:33.207 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:33.211 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:33.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:33.312 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:33.344 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:33.365 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:33.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:33.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:33.404 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:16:33.404 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:16:33.470 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:33.497 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:33.748 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:16:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:16:33] "GET /metrics HTTP/1.1" 200 192383 "" "Prometheus/2.33.4" 2024-02-19T21:16:34.128 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:16:34.863 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:33.323301+0000 mgr.y (mgr.14550) 620 : cluster [DBG] pgmap v1115: 214 pgs: 214 active+clean; 3.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 5.2 KiB/s wr, 1 op/s 2024-02-19T21:16:34.863 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:34.402688+0000 mon.a (mon.0) 3533 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2024-02-19T21:16:34.863 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:33.323301+0000 mgr.y (mgr.14550) 620 : cluster [DBG] pgmap v1115: 214 pgs: 214 active+clean; 3.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 5.2 KiB/s wr, 1 op/s 2024-02-19T21:16:34.863 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:34.402688+0000 mon.a (mon.0) 3533 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2024-02-19T21:16:34.889 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-19T21:16:35.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:33.323301+0000 mgr.y (mgr.14550) 620 : cluster [DBG] pgmap v1115: 214 pgs: 214 active+clean; 3.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 5.2 KiB/s wr, 1 op/s 2024-02-19T21:16:35.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:34.402688+0000 mon.a (mon.0) 3533 : cluster [DBG] osdmap e738: 8 total, 8 up, 8 in 2024-02-19T21:16:35.125 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:35.147 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:35.150 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:35.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:35.251 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:35.280 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:35.379 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:35.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:35.483 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:35.516 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:35.539 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:35.541 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:35.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:35.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.620+0000 7f2fc890c700 1 -- 172.21.15.163:0/822763435 >> v1:172.21.15.163:6790/0 conn(0x7f2fc41031b0 legacy=0x7f2fc4105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:35.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.621+0000 7f2fc890c700 1 -- 172.21.15.163:0/822763435 shutdown_connections 2024-02-19T21:16:35.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.621+0000 7f2fc890c700 1 -- 172.21.15.163:0/822763435 >> 172.21.15.163:0/822763435 conn(0x7f2fc40f2470 msgr2=0x7f2fc40f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:35.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.621+0000 7f2fc890c700 1 -- 172.21.15.163:0/822763435 shutdown_connections 2024-02-19T21:16:35.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.621+0000 7f2fc890c700 1 -- 172.21.15.163:0/822763435 wait complete. 2024-02-19T21:16:35.623 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.622+0000 7f2fc890c700 1 librados: starting msgr at 2024-02-19T21:16:35.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.622+0000 7f2fc890c700 1 librados: starting objecter 2024-02-19T21:16:35.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.622+0000 7f2fc890c700 1 Processor -- start 2024-02-19T21:16:35.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.622+0000 7f2fc890c700 1 -- start start 2024-02-19T21:16:35.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.622+0000 7f2fc890c700 1 librados: setting wanted keys 2024-02-19T21:16:35.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.622+0000 7f2fc890c700 1 librados: calling monclient init 2024-02-19T21:16:35.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.623+0000 7f2fc890c700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2fc40fc1b0 con 0x7f2fc40fc870 2024-02-19T21:16:35.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.623+0000 7f2fc890c700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2fc406a570 con 0x7f2fc40ff670 2024-02-19T21:16:35.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.623+0000 7f2fc890c700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2fc406a700 con 0x7f2fc41031b0 2024-02-19T21:16:35.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.623+0000 7f2fc2d9d700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f2fc41031b0 0x7f2fc4069e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:55652/0 (socket says 172.21.15.163:55652) 2024-02-19T21:16:35.624 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.623+0000 7f2fc2d9d700 1 -- 172.21.15.163:0/3972749271 learned_addr learned my addr 172.21.15.163:0/3972749271 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:35.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.624+0000 7f2fb37fe700 10 client.?.objecter ms_handle_connect 0x7f2fc40ff670 2024-02-19T21:16:35.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.624+0000 7f2fb37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:35.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.624+0000 7f2fb37fe700 10 client.?.objecter ms_handle_connect 0x7f2fc41031b0 2024-02-19T21:16:35.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.624+0000 7f2fb37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:35.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.624+0000 7f2fb37fe700 10 client.?.objecter ms_handle_connect 0x7f2fc40fc870 2024-02-19T21:16:35.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.624+0000 7f2fb37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:35.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.624+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060193176 0 0) 0x7f2fc406a700 con 0x7f2fc41031b0 2024-02-19T21:16:35.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.624+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fac003630 con 0x7f2fc41031b0 2024-02-19T21:16:35.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.624+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147601181 0 0) 0x7f2fc406a570 con 0x7f2fc40ff670 2024-02-19T21:16:35.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.624+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc406a700 con 0x7f2fc40ff670 2024-02-19T21:16:35.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1258700385 0 0) 0x7f2fc40fc1b0 con 0x7f2fc40fc870 2024-02-19T21:16:35.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc406a570 con 0x7f2fc40fc870 2024-02-19T21:16:35.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 916056254 0 0) 0x7f2fac003630 con 0x7f2fc41031b0 2024-02-19T21:16:35.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fc40fc1b0 con 0x7f2fc41031b0 2024-02-19T21:16:35.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2fbc003020 con 0x7f2fc41031b0 2024-02-19T21:16:35.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 901929376 0 0) 0x7f2fc406a700 con 0x7f2fc40ff670 2024-02-19T21:16:35.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fac003630 con 0x7f2fc40ff670 2024-02-19T21:16:35.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2fb4001a10 con 0x7f2fc40ff670 2024-02-19T21:16:35.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2111378704 0 0) 0x7f2fac003630 con 0x7f2fc40ff670 2024-02-19T21:16:35.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 >> v1:172.21.15.163:6790/0 conn(0x7f2fc41031b0 legacy=0x7f2fc4069e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:35.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 >> v1:172.21.15.163:6789/0 conn(0x7f2fc40fc870 legacy=0x7f2fc40fb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:35.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc406a890 con 0x7f2fc40ff670 2024-02-19T21:16:35.627 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fc890c700 1 -- 172.21.15.163:0/3972749271 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc406ab40 con 0x7f2fc40ff670 2024-02-19T21:16:35.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fc890c700 10 client.49351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:35.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fc890c700 1 -- 172.21.15.163:0/3972749271 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc411b990 con 0x7f2fc40ff670 2024-02-19T21:16:35.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.625+0000 7f2fc890c700 1 librados: init done 2024-02-19T21:16:35.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.626+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f2fb4002350 con 0x7f2fc40ff670 2024-02-19T21:16:35.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.626+0000 7f2fc890c700 1 -- 172.21.15.163:0/3972749271 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc405f700 con 0x7f2fc40ff670 2024-02-19T21:16:35.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.626+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2fb4003230 con 0x7f2fc40ff670 2024-02-19T21:16:35.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.627+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f2fb4016120 con 0x7f2fc40ff670 2024-02-19T21:16:35.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.627+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(739..739 src has 623..739) v4 ==== 8670+0+0 (unknown 1539106139 0 0) 0x7f2fb4076670 con 0x7f2fc40ff670 2024-02-19T21:16:35.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.627+0000 7f2fb37fe700 10 client.49351.objecter ms_dispatch 0x7f2fc4001550 osd_map(739..739 src has 623..739) v4 2024-02-19T21:16:35.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.627+0000 7f2fb37fe700 3 client.49351.objecter handle_osd_map got epochs [739,739] > 0 2024-02-19T21:16:35.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.627+0000 7f2fb37fe700 3 client.49351.objecter handle_osd_map decoding full epoch 739 2024-02-19T21:16:35.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.627+0000 7f2fb37fe700 20 client.49351.objecter dump_active .. 0 homeless 2024-02-19T21:16:35.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.627+0000 7f2fb37fe700 10 client.49351.objecter ms_handle_connect 0x7f2fac063c30 2024-02-19T21:16:35.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.631+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f2fb404a650 con 0x7f2fc40ff670 2024-02-19T21:16:35.642 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:35.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:35.498336+0000 mon.a (mon.0) 3534 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:16:35.668 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:35.498670+0000 mon.a (mon.0) 3535 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-130"}]: dispatch 2024-02-19T21:16:35.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:35.498336+0000 mon.a (mon.0) 3534 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:16:35.668 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:35.498670+0000 mon.a (mon.0) 3535 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-130"}]: dispatch 2024-02-19T21:16:35.670 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:35.708 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:35.710 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:35.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:35.812 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:16:35.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:35.820+0000 7f2fc890c700 1 -- 172.21.15.163:0/3972749271 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd out", "ids": ["2"]} v 0) v1 -- 0x7f2fc40f80a0 con 0x7f2fc40ff670 2024-02-19T21:16:35.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:35.498336+0000 mon.a (mon.0) 3534 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:16:35.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:35.498670+0000 mon.a (mon.0) 3535 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-130"}]: dispatch 2024-02-19T21:16:35.841 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:35.861 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:16:35.864 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:35.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:16:35.965 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:35.990 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:16:35.821+0000 7fb606608700 -1 mon.a@0(leader).osd e739 definitely_dead 0 2024-02-19T21:16:35.992 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:36.013 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:36.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:36.116 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:36.145 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:36.166 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:36.168 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:36.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:36.269 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:16:36.298 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:36.316 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:16:36.319 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:36.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:16:36.421 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:36.449 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:36.523 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:36.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:36.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:36.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:36.604+0000 7f2fb37fe700 1 -- 172.21.15.163:0/3972749271 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd out", "ids": ["2"]}]=0 marked out osd.2. v740) v1 ==== 87+0+0 (unknown 2525534157 0 0) 0x7f2fb4048a60 con 0x7f2fc40ff670 2024-02-19T21:16:36.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:36.608+0000 7f2fc890c700 10 librados: watch_flush enter 2024-02-19T21:16:36.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:36.609+0000 7f2fc890c700 10 librados: watch_flush exit 2024-02-19T21:16:36.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:36.609+0000 7f2fc890c700 20 client.49351.objecter shutdown clearing up homeless session... 2024-02-19T21:16:36.610 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:36.609+0000 7f2fc890c700 10 client.49351.objecter successfully canceled tick 2024-02-19T21:16:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:36.611+0000 7f2fc890c700 1 -- 172.21.15.163:0/3972749271 >> v1:172.21.15.163:6800/1552514681 conn(0x7f2fac063c30 legacy=0x7f2fac0660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:36.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:36.611+0000 7f2fc890c700 1 -- 172.21.15.163:0/3972749271 >> v1:172.21.15.193:6789/0 conn(0x7f2fc40ff670 legacy=0x7f2fc4068740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:36.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:36.613+0000 7f2fc890c700 1 -- 172.21.15.163:0/3972749271 shutdown_connections 2024-02-19T21:16:36.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:36.613+0000 7f2fc890c700 1 -- 172.21.15.163:0/3972749271 >> 172.21.15.163:0/3972749271 conn(0x7f2fc40f2470 msgr2=0x7f2fc4106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:36.621 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:36.621+0000 7f2fc890c700 1 -- 172.21.15.163:0/3972749271 shutdown_connections 2024-02-19T21:16:36.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:36.625+0000 7f2fc890c700 1 -- 172.21.15.163:0/3972749271 wait complete. 2024-02-19T21:16:36.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:36.625+0000 7f2fc890c700 1 librados: shutdown 2024-02-19T21:16:36.627 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:36.633 INFO:teuthology.orchestra.run.smithi163.stderr:marked out osd.2. 2024-02-19T21:16:36.654 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:36.672 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:36.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:36.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:35.324111+0000 mgr.y (mgr.14550) 621 : cluster [DBG] pgmap v1117: 214 pgs: 214 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-02-19T21:16:36.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:35.590926+0000 mon.a (mon.0) 3536 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-130"}]': finished 2024-02-19T21:16:36.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:35.590987+0000 mon.a (mon.0) 3537 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2024-02-19T21:16:36.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:35.821098+0000 mon.b (mon.1) 750 : audit [INF] from='client.? v1:172.21.15.163:0/3972749271' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-19T21:16:36.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:35.821438+0000 mon.a (mon.0) 3538 : audit [INF] from='client.49351 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-19T21:16:36.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:35.821513+0000 mon.a (mon.0) 3539 : cluster [INF] Client client.admin marked osd.2 out, while it was still marked up 2024-02-19T21:16:36.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:35.324111+0000 mgr.y (mgr.14550) 621 : cluster [DBG] pgmap v1117: 214 pgs: 214 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-02-19T21:16:36.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:35.590926+0000 mon.a (mon.0) 3536 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-130"}]': finished 2024-02-19T21:16:36.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:35.590987+0000 mon.a (mon.0) 3537 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2024-02-19T21:16:36.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:35.821098+0000 mon.b (mon.1) 750 : audit [INF] from='client.? v1:172.21.15.163:0/3972749271' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-19T21:16:36.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:35.821438+0000 mon.a (mon.0) 3538 : audit [INF] from='client.49351 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-19T21:16:36.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:35.821513+0000 mon.a (mon.0) 3539 : cluster [INF] Client client.admin marked osd.2 out, while it was still marked up 2024-02-19T21:16:36.775 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:36.802 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:36.821 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:36.823 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:36.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:36.924 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:36.953 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:37.052 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:37.055 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:37.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:37.071 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:35.324111+0000 mgr.y (mgr.14550) 621 : cluster [DBG] pgmap v1117: 214 pgs: 214 active+clean; 3.9 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-02-19T21:16:37.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:35.590926+0000 mon.a (mon.0) 3536 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-130"}]': finished 2024-02-19T21:16:37.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:35.590987+0000 mon.a (mon.0) 3537 : cluster [DBG] osdmap e739: 8 total, 8 up, 8 in 2024-02-19T21:16:37.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:35.821098+0000 mon.b (mon.1) 750 : audit [INF] from='client.? v1:172.21.15.163:0/3972749271' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-19T21:16:37.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:35.821438+0000 mon.a (mon.0) 3538 : audit [INF] from='client.49351 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-02-19T21:16:37.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:35.821513+0000 mon.a (mon.0) 3539 : cluster [INF] Client client.admin marked osd.2 out, while it was still marked up 2024-02-19T21:16:37.155 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:37.183 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.151+0000 7f1b74125700 1 -- 172.21.15.163:0/3686834445 >> v1:172.21.15.163:6789/0 conn(0x7f1b6c1042c0 legacy=0x7f1b6c106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.151+0000 7f1b74125700 1 -- 172.21.15.163:0/3686834445 shutdown_connections 2024-02-19T21:16:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.151+0000 7f1b74125700 1 -- 172.21.15.163:0/3686834445 >> 172.21.15.163:0/3686834445 conn(0x7f1b6c06cc00 msgr2=0x7f1b6c0f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:37.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.151+0000 7f1b74125700 1 -- 172.21.15.163:0/3686834445 shutdown_connections 2024-02-19T21:16:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.151+0000 7f1b74125700 1 -- 172.21.15.163:0/3686834445 wait complete. 2024-02-19T21:16:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.151+0000 7f1b74125700 1 librados: starting msgr at 2024-02-19T21:16:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.151+0000 7f1b74125700 1 librados: starting objecter 2024-02-19T21:16:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.151+0000 7f1b74125700 1 Processor -- start 2024-02-19T21:16:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.151+0000 7f1b74125700 1 -- start start 2024-02-19T21:16:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.151+0000 7f1b74125700 1 librados: setting wanted keys 2024-02-19T21:16:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.151+0000 7f1b74125700 1 librados: calling monclient init 2024-02-19T21:16:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b74125700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1b6c1172d0 con 0x7f1b6c1042c0 2024-02-19T21:16:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b74125700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1b6c1184b0 con 0x7f1b6c0fd960 2024-02-19T21:16:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b74125700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1b6c1196b0 con 0x7f1b6c1007b0 2024-02-19T21:16:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b726c2700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f1b6c1042c0 0x7f1b6c115990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:53228/0 (socket says 172.21.15.163:53228) 2024-02-19T21:16:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b716c0700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f1b6c1007b0 0x7f1b6c0fa150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:55666/0 (socket says 172.21.15.163:55666) 2024-02-19T21:16:37.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b726c2700 1 -- 172.21.15.163:0/2539959624 learned_addr learned my addr 172.21.15.163:0/2539959624 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 10 client.?.objecter ms_handle_connect 0x7f1b6c1042c0 2024-02-19T21:16:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 10 client.?.objecter ms_handle_connect 0x7f1b6c1007b0 2024-02-19T21:16:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 10 client.?.objecter ms_handle_connect 0x7f1b6c0fd960 2024-02-19T21:16:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364255973 0 0) 0x7f1b6c1196b0 con 0x7f1b6c1007b0 2024-02-19T21:16:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b58003800 con 0x7f1b6c1007b0 2024-02-19T21:16:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3423242311 0 0) 0x7f1b6c1172d0 con 0x7f1b6c1042c0 2024-02-19T21:16:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b6c1196b0 con 0x7f1b6c1042c0 2024-02-19T21:16:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722395738 0 0) 0x7f1b6c1184b0 con 0x7f1b6c0fd960 2024-02-19T21:16:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b6c1172d0 con 0x7f1b6c0fd960 2024-02-19T21:16:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4124433799 0 0) 0x7f1b58003800 con 0x7f1b6c1007b0 2024-02-19T21:16:37.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b6c1184b0 con 0x7f1b6c1007b0 2024-02-19T21:16:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1b680018c0 con 0x7f1b6c1007b0 2024-02-19T21:16:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2933528724 0 0) 0x7f1b6c1196b0 con 0x7f1b6c1042c0 2024-02-19T21:16:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b58003800 con 0x7f1b6c1042c0 2024-02-19T21:16:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1716624951 0 0) 0x7f1b6c1172d0 con 0x7f1b6c0fd960 2024-02-19T21:16:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b6c1196b0 con 0x7f1b6c0fd960 2024-02-19T21:16:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1b64003150 con 0x7f1b6c1042c0 2024-02-19T21:16:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1b60001620 con 0x7f1b6c0fd960 2024-02-19T21:16:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3149503165 0 0) 0x7f1b6c1184b0 con 0x7f1b6c1007b0 2024-02-19T21:16:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 >> v1:172.21.15.193:6789/0 conn(0x7f1b6c0fd960 legacy=0x7f1b6c0f9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 >> v1:172.21.15.163:6789/0 conn(0x7f1b6c1042c0 legacy=0x7f1b6c115990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c11a8b0 con 0x7f1b6c1007b0 2024-02-19T21:16:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.152+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6c1198a0 con 0x7f1b6c1007b0 2024-02-19T21:16:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.153+0000 7f1b74125700 10 client.50543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.153+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6c119d90 con 0x7f1b6c1007b0 2024-02-19T21:16:37.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.153+0000 7f1b74125700 1 librados: init done 2024-02-19T21:16:37.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.153+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f1b68001f80 con 0x7f1b6c1007b0 2024-02-19T21:16:37.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.153+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1b68005a00 con 0x7f1b6c1007b0 2024-02-19T21:16:37.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.153+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f1b50004f00 con 0x7f1b6c1007b0 2024-02-19T21:16:37.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.154+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f1b68006c10 con 0x7f1b6c1007b0 2024-02-19T21:16:37.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.154+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(740..740 src has 623..740) v4 ==== 8094+0+0 (unknown 2841626221 0 0) 0x7f1b68078ac0 con 0x7f1b6c1007b0 2024-02-19T21:16:37.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.154+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 osd_map(740..740 src has 623..740) v4 2024-02-19T21:16:37.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.154+0000 7f1b5effd700 3 client.50543.objecter handle_osd_map got epochs [740,740] > 0 2024-02-19T21:16:37.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.154+0000 7f1b5effd700 3 client.50543.objecter handle_osd_map decoding full epoch 740 2024-02-19T21:16:37.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.154+0000 7f1b5effd700 20 client.50543.objecter dump_active .. 0 homeless 2024-02-19T21:16:37.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.154+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v740) v1 ==== 54+0+16 (unknown 1238699929 0 2420000192) 0x7f1b68078ed0 con 0x7f1b6c1007b0 2024-02-19T21:16:37.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.154+0000 7f1b5effd700 10 client.50543.objecter ms_handle_connect 0x7f1b58063f90 2024-02-19T21:16:37.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.154+0000 7f1b74125700 10 client.50543.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:37.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.154+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 1 2024-02-19T21:16:37.272 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.154+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b500051a0 osd=0 3 2024-02-19T21:16:37.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.155+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b500051a0 osd=0 3 2024-02-19T21:16:37.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.155+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b500051a0 osd=0 3 2024-02-19T21:16:37.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.155+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 1 2024-02-19T21:16:37.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.155+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b500051a0 osd=0 3 2024-02-19T21:16:37.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.155+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 0 1 2024-02-19T21:16:37.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.155+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b500051a0 osd=0 4 2024-02-19T21:16:37.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.155+0000 7f1b74125700 10 client.50543.objecter _send_command 1 2024-02-19T21:16:37.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.155+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b50004950 con 0x7f1b50004530 2024-02-19T21:16:37.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.155+0000 7f1b5effd700 10 client.50543.objecter ms_handle_connect 0x7f1b50004530 2024-02-19T21:16:37.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.155+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1b50004950 con 0x7f1b50004530 2024-02-19T21:16:37.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.155+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:16:37.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.155+0000 7f1b5effd700 10 client.50543.objecter _finish_command 1 = system:0 2024-02-19T21:16:37.274 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:16:37.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.155+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b500051a0 osd=0 4 2024-02-19T21:16:37.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.155+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 0 1 2024-02-19T21:16:37.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.172+0000 7f1b74125700 10 client.50543.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:37.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.172+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 2 2024-02-19T21:16:37.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.172+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b500051a0 osd=0 3 2024-02-19T21:16:37.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.172+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b500051a0 osd=0 3 2024-02-19T21:16:37.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.172+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b500051a0 osd=0 3 2024-02-19T21:16:37.274 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.172+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 2 2024-02-19T21:16:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.172+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b500051a0 osd=0 3 2024-02-19T21:16:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.172+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 0 2 2024-02-19T21:16:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.172+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b500051a0 osd=0 4 2024-02-19T21:16:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.172+0000 7f1b74125700 10 client.50543.objecter _send_command 2 2024-02-19T21:16:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.172+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b50004d30 con 0x7f1b50004530 2024-02-19T21:16:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1b50004d30 con 0x7f1b50004530 2024-02-19T21:16:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b5effd700 10 client.50543.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b500051a0 osd=0 4 2024-02-19T21:16:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 0 2 2024-02-19T21:16:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b74125700 10 client.50543.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 3 2024-02-19T21:16:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50003fe0 osd=1 3 2024-02-19T21:16:37.275 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50003fe0 osd=1 3 2024-02-19T21:16:37.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50003fe0 osd=1 3 2024-02-19T21:16:37.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 3 2024-02-19T21:16:37.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b50003fe0 osd=1 3 2024-02-19T21:16:37.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 1 3 2024-02-19T21:16:37.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50003fe0 osd=1 4 2024-02-19T21:16:37.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b74125700 10 client.50543.objecter _send_command 3 2024-02-19T21:16:37.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.174+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b5000c000 con 0x7f1b50004150 2024-02-19T21:16:37.276 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.175+0000 7f1b5effd700 10 client.50543.objecter ms_handle_connect 0x7f1b50004150 2024-02-19T21:16:37.276 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:16:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.175+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1b5000c000 con 0x7f1b50004150 2024-02-19T21:16:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.175+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:16:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.175+0000 7f1b5effd700 10 client.50543.objecter _finish_command 3 = system:0 2024-02-19T21:16:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.175+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b50003fe0 osd=1 4 2024-02-19T21:16:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.175+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 1 3 2024-02-19T21:16:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.194+0000 7f1b74125700 10 client.50543.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.194+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 4 2024-02-19T21:16:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.194+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50003fe0 osd=1 3 2024-02-19T21:16:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.194+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50003fe0 osd=1 3 2024-02-19T21:16:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.194+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50003fe0 osd=1 3 2024-02-19T21:16:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.194+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 4 2024-02-19T21:16:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.194+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b50003fe0 osd=1 3 2024-02-19T21:16:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.194+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 1 4 2024-02-19T21:16:37.277 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.194+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50003fe0 osd=1 4 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.194+0000 7f1b74125700 10 client.50543.objecter _send_command 4 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.194+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b500018b0 con 0x7f1b50004150 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1b500018b0 con 0x7f1b50004150 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b5effd700 10 client.50543.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b50003fe0 osd=1 4 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 1 4 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b74125700 10 client.50543.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 5 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50003630 osd=2 3 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50003630 osd=2 3 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50003630 osd=2 3 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 5 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b50003630 osd=2 3 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 2 5 2024-02-19T21:16:37.278 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50003630 osd=2 4 2024-02-19T21:16:37.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b74125700 10 client.50543.objecter _send_command 5 2024-02-19T21:16:37.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b5000de90 con 0x7f1b5000d2f0 2024-02-19T21:16:37.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.196+0000 7f1b5effd700 10 client.50543.objecter ms_handle_connect 0x7f1b5000d2f0 2024-02-19T21:16:37.279 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:16:37.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.197+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1b5000de90 con 0x7f1b5000d2f0 2024-02-19T21:16:37.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.197+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:16:37.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.197+0000 7f1b5effd700 10 client.50543.objecter _finish_command 5 = system:0 2024-02-19T21:16:37.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.197+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b50003630 osd=2 4 2024-02-19T21:16:37.279 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.197+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 2 5 2024-02-19T21:16:37.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.212+0000 7f1b74125700 10 client.50543.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:37.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.212+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 6 2024-02-19T21:16:37.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.212+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50003630 osd=2 3 2024-02-19T21:16:37.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.212+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50003630 osd=2 3 2024-02-19T21:16:37.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.212+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50003630 osd=2 3 2024-02-19T21:16:37.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.212+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 6 2024-02-19T21:16:37.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.212+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b50003630 osd=2 3 2024-02-19T21:16:37.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.212+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 2 6 2024-02-19T21:16:37.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.212+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50003630 osd=2 4 2024-02-19T21:16:37.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.212+0000 7f1b74125700 10 client.50543.objecter _send_command 6 2024-02-19T21:16:37.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.212+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b50003160 con 0x7f1b5000d2f0 2024-02-19T21:16:37.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1b50003160 con 0x7f1b5000d2f0 2024-02-19T21:16:37.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:37.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b5effd700 10 client.50543.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.280 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b50003630 osd=2 4 2024-02-19T21:16:37.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 2 6 2024-02-19T21:16:37.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b74125700 10 client.50543.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:37.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 7 2024-02-19T21:16:37.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50011970 osd=3 3 2024-02-19T21:16:37.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50011970 osd=3 3 2024-02-19T21:16:37.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50011970 osd=3 3 2024-02-19T21:16:37.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 7 2024-02-19T21:16:37.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b50011970 osd=3 3 2024-02-19T21:16:37.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 3 7 2024-02-19T21:16:37.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50011970 osd=3 4 2024-02-19T21:16:37.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b74125700 10 client.50543.objecter _send_command 7 2024-02-19T21:16:37.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b50015bd0 con 0x7f1b50011ff0 2024-02-19T21:16:37.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.214+0000 7f1b5effd700 10 client.50543.objecter ms_handle_connect 0x7f1b50011ff0 2024-02-19T21:16:37.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.215+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1b50015bd0 con 0x7f1b50011ff0 2024-02-19T21:16:37.281 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.215+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:16:37.282 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:16:37.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.215+0000 7f1b5effd700 10 client.50543.objecter _finish_command 7 = system:0 2024-02-19T21:16:37.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.215+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b50011970 osd=3 4 2024-02-19T21:16:37.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.215+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 3 7 2024-02-19T21:16:37.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.229+0000 7f1b74125700 10 client.50543.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:37.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.229+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 8 2024-02-19T21:16:37.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.229+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50011970 osd=3 3 2024-02-19T21:16:37.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.229+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50011970 osd=3 3 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.229+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50011970 osd=3 3 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.229+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 8 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.229+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b50011970 osd=3 3 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.229+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 3 8 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.229+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50011970 osd=3 4 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.229+0000 7f1b74125700 10 client.50543.objecter _send_command 8 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.229+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b5000e270 con 0x7f1b50011ff0 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1b5000e270 con 0x7f1b50011ff0 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b5effd700 10 client.50543.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b50011970 osd=3 4 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 3 8 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b74125700 10 client.50543.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 9 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b5000e270 osd=4 3 2024-02-19T21:16:37.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b5000e270 osd=4 3 2024-02-19T21:16:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b5000e270 osd=4 3 2024-02-19T21:16:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 9 2024-02-19T21:16:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b5000e270 osd=4 3 2024-02-19T21:16:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 4 9 2024-02-19T21:16:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b5000e270 osd=4 4 2024-02-19T21:16:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b74125700 10 client.50543.objecter _send_command 9 2024-02-19T21:16:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b5000cff0 con 0x7f1b5000e430 2024-02-19T21:16:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.231+0000 7f1b5effd700 10 client.50543.objecter ms_handle_connect 0x7f1b5000e430 2024-02-19T21:16:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.232+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1b5000cff0 con 0x7f1b5000e430 2024-02-19T21:16:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.232+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:16:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.232+0000 7f1b5effd700 10 client.50543.objecter _finish_command 9 = system:0 2024-02-19T21:16:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.232+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b5000e270 osd=4 4 2024-02-19T21:16:37.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.232+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 4 9 2024-02-19T21:16:37.285 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:16:37.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.247+0000 7f1b74125700 10 client.50543.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:37.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.247+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 10 2024-02-19T21:16:37.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.247+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b5000e270 osd=4 3 2024-02-19T21:16:37.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.247+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b5000e270 osd=4 3 2024-02-19T21:16:37.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.247+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b5000e270 osd=4 3 2024-02-19T21:16:37.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.247+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 10 2024-02-19T21:16:37.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.247+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b5000e270 osd=4 3 2024-02-19T21:16:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.247+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 4 10 2024-02-19T21:16:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.247+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b5000e270 osd=4 4 2024-02-19T21:16:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.247+0000 7f1b74125700 10 client.50543.objecter _send_command 10 2024-02-19T21:16:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.247+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b5000cff0 con 0x7f1b5000e430 2024-02-19T21:16:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1b5000cff0 con 0x7f1b5000e430 2024-02-19T21:16:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b5effd700 10 client.50543.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b5000e270 osd=4 4 2024-02-19T21:16:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 4 10 2024-02-19T21:16:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b74125700 10 client.50543.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 11 2024-02-19T21:16:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b5000cff0 osd=5 3 2024-02-19T21:16:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b5000cff0 osd=5 3 2024-02-19T21:16:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b5000cff0 osd=5 3 2024-02-19T21:16:37.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 11 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b5000cff0 osd=5 3 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 5 11 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b5000cff0 osd=5 4 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b74125700 10 client.50543.objecter _send_command 11 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b50018040 con 0x7f1b50017b10 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.249+0000 7f1b5effd700 10 client.50543.objecter ms_handle_connect 0x7f1b50017b10 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.250+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1b50018040 con 0x7f1b50017b10 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.250+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.250+0000 7f1b5effd700 10 client.50543.objecter _finish_command 11 = system:0 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.250+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b5000cff0 osd=5 4 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.250+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 5 11 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.265+0000 7f1b74125700 10 client.50543.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.265+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 12 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.265+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b5000cff0 osd=5 3 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.265+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b5000cff0 osd=5 3 2024-02-19T21:16:37.287 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.265+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b5000cff0 osd=5 3 2024-02-19T21:16:37.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.265+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 12 2024-02-19T21:16:37.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.265+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b5000cff0 osd=5 3 2024-02-19T21:16:37.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.265+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 5 12 2024-02-19T21:16:37.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.265+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b5000cff0 osd=5 4 2024-02-19T21:16:37.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.265+0000 7f1b74125700 10 client.50543.objecter _send_command 12 2024-02-19T21:16:37.288 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.265+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b50018040 con 0x7f1b50017b10 2024-02-19T21:16:37.288 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:16:37.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1b50018040 con 0x7f1b50017b10 2024-02-19T21:16:37.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:37.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b5effd700 10 client.50543.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b5000cff0 osd=5 4 2024-02-19T21:16:37.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 5 12 2024-02-19T21:16:37.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b74125700 10 client.50543.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:37.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 13 2024-02-19T21:16:37.289 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50018040 osd=6 3 2024-02-19T21:16:37.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50018040 osd=6 3 2024-02-19T21:16:37.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50018040 osd=6 3 2024-02-19T21:16:37.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 13 2024-02-19T21:16:37.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b50018040 osd=6 3 2024-02-19T21:16:37.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 6 13 2024-02-19T21:16:37.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50018040 osd=6 4 2024-02-19T21:16:37.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b74125700 10 client.50543.objecter _send_command 13 2024-02-19T21:16:37.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b500270f0 con 0x7f1b500235e0 2024-02-19T21:16:37.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.267+0000 7f1b5effd700 10 client.50543.objecter ms_handle_connect 0x7f1b500235e0 2024-02-19T21:16:37.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.268+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1b500270f0 con 0x7f1b500235e0 2024-02-19T21:16:37.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.268+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:16:37.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.268+0000 7f1b5effd700 10 client.50543.objecter _finish_command 13 = system:0 2024-02-19T21:16:37.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.268+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b50018040 osd=6 4 2024-02-19T21:16:37.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.268+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 6 13 2024-02-19T21:16:37.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.282+0000 7f1b74125700 10 client.50543.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:37.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.282+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 14 2024-02-19T21:16:37.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.282+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50018040 osd=6 3 2024-02-19T21:16:37.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.282+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50018040 osd=6 3 2024-02-19T21:16:37.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.282+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b50018040 osd=6 3 2024-02-19T21:16:37.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.282+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 14 2024-02-19T21:16:37.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.282+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b50018040 osd=6 3 2024-02-19T21:16:37.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.282+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 6 14 2024-02-19T21:16:37.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.282+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50018040 osd=6 4 2024-02-19T21:16:37.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.282+0000 7f1b74125700 10 client.50543.objecter _send_command 14 2024-02-19T21:16:37.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.282+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b500270f0 con 0x7f1b500235e0 2024-02-19T21:16:37.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1b500270f0 con 0x7f1b500235e0 2024-02-19T21:16:37.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:37.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b5effd700 10 client.50543.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b50018040 osd=6 4 2024-02-19T21:16:37.291 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 6 14 2024-02-19T21:16:37.292 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:16:37.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b74125700 10 client.50543.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:37.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 15 2024-02-19T21:16:37.292 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b500270f0 osd=7 3 2024-02-19T21:16:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b500270f0 osd=7 3 2024-02-19T21:16:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b500270f0 osd=7 3 2024-02-19T21:16:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 15 2024-02-19T21:16:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b500270f0 osd=7 3 2024-02-19T21:16:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 7 15 2024-02-19T21:16:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b500270f0 osd=7 4 2024-02-19T21:16:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b74125700 10 client.50543.objecter _send_command 15 2024-02-19T21:16:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b50029470 con 0x7f1b50027770 2024-02-19T21:16:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.284+0000 7f1b5effd700 10 client.50543.objecter ms_handle_connect 0x7f1b50027770 2024-02-19T21:16:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.285+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f1b50029470 con 0x7f1b50027770 2024-02-19T21:16:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.285+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:16:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.285+0000 7f1b5effd700 10 client.50543.objecter _finish_command 15 = system:0 2024-02-19T21:16:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.285+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b500270f0 osd=7 4 2024-02-19T21:16:37.293 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.285+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 7 15 2024-02-19T21:16:37.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.299+0000 7f1b74125700 10 client.50543.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:37.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.300+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign -1 16 2024-02-19T21:16:37.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.300+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b500270f0 osd=7 3 2024-02-19T21:16:37.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.300+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b500270f0 osd=7 3 2024-02-19T21:16:37.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.300+0000 7f1b74125700 20 client.50543.objecter _get_session s=0x7f1b500270f0 osd=7 3 2024-02-19T21:16:37.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.300+0000 7f1b74125700 15 client.50543.objecter _session_command_op_remove -1 16 2024-02-19T21:16:37.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.300+0000 7f1b74125700 20 client.50543.objecter get_session s=0x7f1b500270f0 osd=7 3 2024-02-19T21:16:37.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.300+0000 7f1b74125700 15 client.50543.objecter _session_command_op_assign 7 16 2024-02-19T21:16:37.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.300+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b500270f0 osd=7 4 2024-02-19T21:16:37.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.300+0000 7f1b74125700 10 client.50543.objecter _send_command 16 2024-02-19T21:16:37.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.300+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f1b50029470 con 0x7f1b50027770 2024-02-19T21:16:37.301 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.301+0000 7f1b5effd700 1 -- 172.21.15.163:0/2539959624 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f1b50029470 con 0x7f1b50027770 2024-02-19T21:16:37.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.301+0000 7f1b5effd700 10 client.50543.objecter ms_dispatch 0x7f1b6c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:37.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.301+0000 7f1b5effd700 10 client.50543.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.301+0000 7f1b5effd700 20 client.50543.objecter put_session s=0x7f1b500270f0 osd=7 4 2024-02-19T21:16:37.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.301+0000 7f1b5effd700 15 client.50543.objecter _session_command_op_remove 7 16 2024-02-19T21:16:37.302 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:16:37.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.301+0000 7f1b74125700 10 librados: watch_flush enter 2024-02-19T21:16:37.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.301+0000 7f1b74125700 10 librados: watch_flush exit 2024-02-19T21:16:37.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.301+0000 7f1b74125700 10 client.50543.objecter close_session for osd.0 2024-02-19T21:16:37.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.301+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 >> v1:172.21.15.163:6801/1559418115 conn(0x7f1b50004530 legacy=0x7f1b50002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:37.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b500051a0 osd=0 1 2024-02-19T21:16:37.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 10 client.50543.objecter close_session for osd.1 2024-02-19T21:16:37.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 >> v1:172.21.15.163:6805/3125047432 conn(0x7f1b50004150 legacy=0x7f1b5000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:37.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50003fe0 osd=1 1 2024-02-19T21:16:37.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 10 client.50543.objecter close_session for osd.2 2024-02-19T21:16:37.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 >> v1:172.21.15.163:6809/1563158487 conn(0x7f1b5000d2f0 legacy=0x7f1b5000d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:37.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50003630 osd=2 1 2024-02-19T21:16:37.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 10 client.50543.objecter close_session for osd.3 2024-02-19T21:16:37.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b726c2700 1 -- 172.21.15.163:0/2539959624 reap_dead start 2024-02-19T21:16:37.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 >> v1:172.21.15.163:6813/2112279769 conn(0x7f1b50011ff0 legacy=0x7f1b500143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:37.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50011970 osd=3 1 2024-02-19T21:16:37.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 10 client.50543.objecter close_session for osd.4 2024-02-19T21:16:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 >> v1:172.21.15.193:6800/3129716965 conn(0x7f1b5000e430 legacy=0x7f1b50015d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b5000e270 osd=4 1 2024-02-19T21:16:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 10 client.50543.objecter close_session for osd.5 2024-02-19T21:16:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 >> v1:172.21.15.193:6804/805161991 conn(0x7f1b50017b10 legacy=0x7f1b500219b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b5000cff0 osd=5 1 2024-02-19T21:16:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 10 client.50543.objecter close_session for osd.6 2024-02-19T21:16:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 >> v1:172.21.15.193:6808/787604615 conn(0x7f1b500235e0 legacy=0x7f1b500259d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b50018040 osd=6 1 2024-02-19T21:16:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 10 client.50543.objecter close_session for osd.7 2024-02-19T21:16:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 >> v1:172.21.15.193:6812/157927432 conn(0x7f1b50027770 legacy=0x7f1b50027bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 20 client.50543.objecter put_session s=0x7f1b500270f0 osd=7 1 2024-02-19T21:16:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 20 client.50543.objecter shutdown clearing up homeless session... 2024-02-19T21:16:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 10 client.50543.objecter successfully canceled tick 2024-02-19T21:16:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b726c2700 1 -- 172.21.15.163:0/2539959624 reap_dead start 2024-02-19T21:16:37.305 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 >> v1:172.21.15.163:6800/1552514681 conn(0x7f1b58063f90 legacy=0x7f1b58066450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:37.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 >> v1:172.21.15.163:6790/0 conn(0x7f1b6c1007b0 legacy=0x7f1b6c0fa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:37.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 shutdown_connections 2024-02-19T21:16:37.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 >> 172.21.15.163:0/2539959624 conn(0x7f1b6c06cc00 msgr2=0x7f1b6c107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:37.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 shutdown_connections 2024-02-19T21:16:37.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 1 -- 172.21.15.163:0/2539959624 wait complete. 2024-02-19T21:16:37.306 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:37.302+0000 7f1b74125700 1 librados: shutdown 2024-02-19T21:16:37.306 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.306 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.306 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.306 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.306 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.306 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.306 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.306 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:16:37.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:36.595140+0000 mon.a (mon.0) 3540 : audit [INF] from='client.49351 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-02-19T21:16:37.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:36.595206+0000 mon.a (mon.0) 3541 : cluster [DBG] osdmap e740: 8 total, 8 up, 7 in 2024-02-19T21:16:37.609 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:37.154615+0000 mon.c (mon.2) 432 : audit [DBG] from='client.? v1:172.21.15.163:0/2539959624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:37.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:36.595140+0000 mon.a (mon.0) 3540 : audit [INF] from='client.49351 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-02-19T21:16:37.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:36.595206+0000 mon.a (mon.0) 3541 : cluster [DBG] osdmap e740: 8 total, 8 up, 7 in 2024-02-19T21:16:37.609 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:37.154615+0000 mon.c (mon.2) 432 : audit [DBG] from='client.? v1:172.21.15.163:0/2539959624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:37.635 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:37.640 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:37.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:37.741 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:37.769 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:37.933 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:37.933 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:38.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:16:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:16:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:16:38.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:36.595140+0000 mon.a (mon.0) 3540 : audit [INF] from='client.49351 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-02-19T21:16:38.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:36.595206+0000 mon.a (mon.0) 3541 : cluster [DBG] osdmap e740: 8 total, 8 up, 7 in 2024-02-19T21:16:38.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:37.154615+0000 mon.c (mon.2) 432 : audit [DBG] from='client.? v1:172.21.15.163:0/2539959624' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:38.721 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:37.324648+0000 mgr.y (mgr.14550) 622 : cluster [DBG] pgmap v1120: 182 pgs: 182 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:16:38.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:37.605929+0000 mon.a (mon.0) 3542 : cluster [DBG] osdmap e741: 8 total, 8 up, 7 in 2024-02-19T21:16:38.722 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:37.610368+0000 mon.a (mon.0) 3543 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:16:38.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:37.324648+0000 mgr.y (mgr.14550) 622 : cluster [DBG] pgmap v1120: 182 pgs: 182 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:16:38.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:37.605929+0000 mon.a (mon.0) 3542 : cluster [DBG] osdmap e741: 8 total, 8 up, 7 in 2024-02-19T21:16:38.722 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:37.610368+0000 mon.a (mon.0) 3543 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:16:39.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:37.324648+0000 mgr.y (mgr.14550) 622 : cluster [DBG] pgmap v1120: 182 pgs: 182 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:16:39.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:37.605929+0000 mon.a (mon.0) 3542 : cluster [DBG] osdmap e741: 8 total, 8 up, 7 in 2024-02-19T21:16:39.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:37.610368+0000 mon.a (mon.0) 3543 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:16:39.977 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:38.605506+0000 mon.a (mon.0) 3544 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:16:39.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:38.605621+0000 mon.a (mon.0) 3545 : cluster [DBG] osdmap e742: 8 total, 8 up, 7 in 2024-02-19T21:16:39.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:38.653775+0000 mon.a (mon.0) 3546 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:16:39.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:39.109204+0000 mon.a (mon.0) 3547 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:39.978 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:38.605506+0000 mon.a (mon.0) 3544 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:16:39.978 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:38.605621+0000 mon.a (mon.0) 3545 : cluster [DBG] osdmap e742: 8 total, 8 up, 7 in 2024-02-19T21:16:39.978 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:38.653775+0000 mon.a (mon.0) 3546 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:16:39.978 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:39.109204+0000 mon.a (mon.0) 3547 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:40.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:38.605506+0000 mon.a (mon.0) 3544 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:16:40.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:38.605621+0000 mon.a (mon.0) 3545 : cluster [DBG] osdmap e742: 8 total, 8 up, 7 in 2024-02-19T21:16:40.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:38.653775+0000 mon.a (mon.0) 3546 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:16:40.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:39.109204+0000 mon.a (mon.0) 3547 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:40.448 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:16:40.820 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:40.824 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:40.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:40.864 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:40.925 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:40.955 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:16:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:16:40.955 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:39.325020+0000 mgr.y (mgr.14550) 623 : cluster [DBG] pgmap v1123: 214 pgs: 32 unknown, 182 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:16:40.955 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:39.621863+0000 mon.a (mon.0) 3548 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-132", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:16:40.955 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:39.621924+0000 mon.a (mon.0) 3549 : cluster [DBG] osdmap e743: 8 total, 8 up, 7 in 2024-02-19T21:16:40.955 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:39.626440+0000 mon.a (mon.0) 3550 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-132"}]: dispatch 2024-02-19T21:16:40.957 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:40.978 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:40.981 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:40.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:40.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:39.325020+0000 mgr.y (mgr.14550) 623 : cluster [DBG] pgmap v1123: 214 pgs: 32 unknown, 182 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:16:40.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:39.621863+0000 mon.a (mon.0) 3548 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-132", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:16:40.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:39.621924+0000 mon.a (mon.0) 3549 : cluster [DBG] osdmap e743: 8 total, 8 up, 7 in 2024-02-19T21:16:40.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:39.626440+0000 mon.a (mon.0) 3550 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-132"}]: dispatch 2024-02-19T21:16:40.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:39.325020+0000 mgr.y (mgr.14550) 623 : cluster [DBG] pgmap v1123: 214 pgs: 32 unknown, 182 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:16:40.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:39.621863+0000 mon.a (mon.0) 3548 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-132", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:16:40.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:39.621924+0000 mon.a (mon.0) 3549 : cluster [DBG] osdmap e743: 8 total, 8 up, 7 in 2024-02-19T21:16:40.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:39.626440+0000 mon.a (mon.0) 3550 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-132"}]: dispatch 2024-02-19T21:16:41.082 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:41.111 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:41.361 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:41.364 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:41.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:41.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:16:41.496 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:41.517 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:16:41.520 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:41.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:16:41.621 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:41.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:40.624012+0000 mon.a (mon.0) 3551 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-132"}]': finished 2024-02-19T21:16:41.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:40.624075+0000 mon.a (mon.0) 3552 : cluster [DBG] osdmap e744: 8 total, 8 up, 7 in 2024-02-19T21:16:41.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:40.627430+0000 mon.a (mon.0) 3553 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-132", "mode": "writeback"}]: dispatch 2024-02-19T21:16:41.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:40.765883+0000 mgr.y (mgr.14550) 624 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:41.635 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:41.621315+0000 mon.a (mon.0) 3554 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:16:41.635 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:40.624012+0000 mon.a (mon.0) 3551 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-132"}]': finished 2024-02-19T21:16:41.636 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:40.624075+0000 mon.a (mon.0) 3552 : cluster [DBG] osdmap e744: 8 total, 8 up, 7 in 2024-02-19T21:16:41.636 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:40.627430+0000 mon.a (mon.0) 3553 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-132", "mode": "writeback"}]: dispatch 2024-02-19T21:16:41.636 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:40.765883+0000 mgr.y (mgr.14550) 624 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:41.636 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:41.621315+0000 mon.a (mon.0) 3554 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:16:41.651 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:41.672 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:41.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:41.776 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:41.802 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:40.624012+0000 mon.a (mon.0) 3551 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-132"}]': finished 2024-02-19T21:16:41.802 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:40.624075+0000 mon.a (mon.0) 3552 : cluster [DBG] osdmap e744: 8 total, 8 up, 7 in 2024-02-19T21:16:41.802 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:40.627430+0000 mon.a (mon.0) 3553 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-132", "mode": "writeback"}]: dispatch 2024-02-19T21:16:41.802 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:40.765883+0000 mgr.y (mgr.14550) 624 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:41.802 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:41.621315+0000 mon.a (mon.0) 3554 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:16:41.804 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:41.825 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:41.827 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:41.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:41.928 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:41.959 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:41.998 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:16:41.998 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-02-19T21:16:41.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-02-19T21:16:41.998 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-02-19T21:16:41.998 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:41.998 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.25753.0:6560 61.2 e452/303 61:4782fe63:test-rados-api-smithi163-83928-70::foo:head v 452'98, mlcod=447'96)", 2024-02-19T21:16:41.998 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:11:04.382809+0000", 2024-02-19T21:16:41.998 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 337.45059797099998, 2024-02-19T21:16:41.998 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025230308999999999, 2024-02-19T21:16:41.998 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382809+0000", 2024-02-19T21:16:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382809+0000", 2024-02-19T21:16:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:16:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382807+0000", 2024-02-19T21:16:42.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5722999999999998e-05 2024-02-19T21:16:42.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382823+0000", 2024-02-19T21:16:42.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8643000000000001e-05 2024-02-19T21:16:42.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.000 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.000 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382841+0000", 2024-02-19T21:16:42.000 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2550000000000003e-06 2024-02-19T21:16:42.000 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.000 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.001 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.001 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382850+0000", 2024-02-19T21:16:42.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0214999999999999e-05 2024-02-19T21:16:42.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.001 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.001 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382911+0000", 2024-02-19T21:16:42.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9959999999999999e-05 2024-02-19T21:16:42.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.001 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.001 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:16:42.001 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.382931+0000", 2024-02-19T21:16:42.001 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025067876999999999 2024-02-19T21:16:42.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:16:42.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.407998+0000", 2024-02-19T21:16:42.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1310000000000003e-05 2024-02-19T21:16:42.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.002 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.002 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.002 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408040+0000", 2024-02-19T21:16:42.002 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.025230308999999999 2024-02-19T21:16:42.002 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.002 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.002 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.002 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.003 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.3.25:35 216.0 e452/444 216:06397337:test-rados-api-smithi163-83928-70::b1a341c59dc4c60bbe00244ac8bba87528936bc7:head v 452'2, mlcod=0'0)", 2024-02-19T21:16:42.003 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:11:04.408954+0000", 2024-02-19T21:16:42.003 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 337.42445304400002, 2024-02-19T21:16:42.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023962845999999999, 2024-02-19T21:16:42.003 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.003 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:42.003 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.003 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.003 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408954+0000", 2024-02-19T21:16:42.003 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.003 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.003 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408954+0000", 2024-02-19T21:16:42.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:16:42.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.004 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.004 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.004 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408953+0000", 2024-02-19T21:16:42.004 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0350000000000001e-06 2024-02-19T21:16:42.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408961+0000", 2024-02-19T21:16:42.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0436e-05 2024-02-19T21:16:42.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.005 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408971+0000", 2024-02-19T21:16:42.005 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8640000000000004e-06 2024-02-19T21:16:42.005 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.005 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.005 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.408976+0000", 2024-02-19T21:16:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3492999999999998e-05 2024-02-19T21:16:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.409020+0000", 2024-02-19T21:16:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4091e-05 2024-02-19T21:16:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:16:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.409044+0000", 2024-02-19T21:16:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023821754000000001 2024-02-19T21:16:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.006 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:16:42.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.432866+0000", 2024-02-19T21:16:42.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1702000000000001e-05 2024-02-19T21:16:42.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.007 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.007 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:11:04.432917+0000", 2024-02-19T21:16:42.007 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.023962845999999999 2024-02-19T21:16:42.007 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.007 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.007 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.007 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.007 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.007 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(234.2s1 658/636 ECSubWrite(tid=669, reqid=client.23517.0:794, at_version=0'0, trim_to=0'0, roll_forward_to=658'1))", 2024-02-19T21:16:42.008 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:02.411619+0000", 2024-02-19T21:16:42.008 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 99.421788129999996, 2024-02-19T21:16:42.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019196199999999999, 2024-02-19T21:16:42.008 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.008 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:16:42.008 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.008 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.008 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:02.411619+0000", 2024-02-19T21:16:42.008 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.008 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.008 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:02.411619+0000", 2024-02-19T21:16:42.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:16:42.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:02.411618+0000", 2024-02-19T21:16:42.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2075999999999999e-05 2024-02-19T21:16:42.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.009 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.009 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:02.411630+0000", 2024-02-19T21:16:42.009 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1840000000000005e-06 2024-02-19T21:16:42.009 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.009 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:02.411639+0000", 2024-02-19T21:16:42.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.533e-06 2024-02-19T21:16:42.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:02.411646+0000", 2024-02-19T21:16:42.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001344666 2024-02-19T21:16:42.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.010 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.010 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.010 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:02.412990+0000", 2024-02-19T21:16:42.010 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4490000000000005e-06 2024-02-19T21:16:42.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:16:42.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:02.413000+0000", 2024-02-19T21:16:42.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000520788 2024-02-19T21:16:42.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:16:42.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:02.413521+0000", 2024-02-19T21:16:42.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8258999999999999e-05 2024-02-19T21:16:42.011 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.011 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.011 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.011 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:02.413539+0000", 2024-02-19T21:16:42.011 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0019196199999999999 2024-02-19T21:16:42.012 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.012 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.012 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.012 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDScrubReserve(6.d REQUEST e687)", 2024-02-19T21:16:42.012 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:37.005969+0000", 2024-02-19T21:16:42.012 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 64.827437975999999, 2024-02-19T21:16:42.012 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020440359999999999, 2024-02-19T21:16:42.012 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.012 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:16:42.012 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.012 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:37.005969+0000", 2024-02-19T21:16:42.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:37.005969+0000", 2024-02-19T21:16:42.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:16:42.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.013 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.013 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.013 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:37.005969+0000", 2024-02-19T21:16:42.013 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4399999999999999e-06 2024-02-19T21:16:42.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.014 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.014 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.014 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:37.005972+0000", 2024-02-19T21:16:42.014 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2554999999999999e-05 2024-02-19T21:16:42.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.014 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.014 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.014 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:37.005984+0000", 2024-02-19T21:16:42.014 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0159999999999997e-06 2024-02-19T21:16:42.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.015 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.015 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.015 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:37.005993+0000", 2024-02-19T21:16:42.015 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3347000000000002e-05 2024-02-19T21:16:42.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.015 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.015 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.015 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:37.006057+0000", 2024-02-19T21:16:42.015 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1999999999999997e-06 2024-02-19T21:16:42.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.015 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.015 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:16:42.015 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:37.006064+0000", 2024-02-19T21:16:42.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00192107 2024-02-19T21:16:42.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.016 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.016 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.016 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:37.007985+0000", 2024-02-19T21:16:42.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0482000000000001e-05 2024-02-19T21:16:42.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.016 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.016 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:16:42.016 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:37.007995+0000", 2024-02-19T21:16:42.016 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8187999999999999e-05 2024-02-19T21:16:42.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.016 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.017 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.017 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:37.008014+0000", 2024-02-19T21:16:42.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020440359999999999 2024-02-19T21:16:42.017 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.017 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.017 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.017 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.017 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDScrubReserve(5.3 RELEASE e693)", 2024-02-19T21:16:42.017 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:43.163208+0000", 2024-02-19T21:16:42.017 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 58.670199418999999, 2024-02-19T21:16:42.017 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0011519029999999999, 2024-02-19T21:16:42.018 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.018 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:16:42.018 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:43.163208+0000", 2024-02-19T21:16:42.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.018 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.018 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.018 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:43.163208+0000", 2024-02-19T21:16:42.018 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967296 2024-02-19T21:16:42.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:43.163208+0000", 2024-02-19T21:16:42.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5799999999999999e-06 2024-02-19T21:16:42.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.019 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.019 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.019 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:43.163210+0000", 2024-02-19T21:16:42.019 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8189999999999998e-06 2024-02-19T21:16:42.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:43.163219+0000", 2024-02-19T21:16:42.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6580000000000001e-06 2024-02-19T21:16:42.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.020 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.020 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.020 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:43.163224+0000", 2024-02-19T21:16:42.020 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1145e-05 2024-02-19T21:16:42.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:43.163265+0000", 2024-02-19T21:16:42.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8550000000000001e-06 2024-02-19T21:16:42.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:16:42.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:43.163270+0000", 2024-02-19T21:16:42.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001043338 2024-02-19T21:16:42.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.021 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.021 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.021 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:43.164313+0000", 2024-02-19T21:16:42.021 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.7737e-05 2024-02-19T21:16:42.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:16:42.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:43.164331+0000", 2024-02-19T21:16:42.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8971000000000001e-05 2024-02-19T21:16:42.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.022 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.022 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.022 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:43.164360+0000", 2024-02-19T21:16:42.022 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0011519029999999999 2024-02-19T21:16:42.023 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.023 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.023 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.023 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.23517.0:1998 249.6 e734/728 249:602f83fe:::foo:head v 734'1, mlcod=0'0)", 2024-02-19T21:16:42.023 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:23.407076+0000", 2024-02-19T21:16:42.023 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 18.42633112, 2024-02-19T21:16:42.023 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00086011599999999996, 2024-02-19T21:16:42.023 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.023 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:42.023 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.023 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:23.407076+0000", 2024-02-19T21:16:42.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:23.407076+0000", 2024-02-19T21:16:42.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:16:42.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.024 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.024 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.024 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:23.407076+0000", 2024-02-19T21:16:42.024 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1046000000000001e-05 2024-02-19T21:16:42.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.025 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.025 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:23.407107+0000", 2024-02-19T21:16:42.025 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1625e-05 2024-02-19T21:16:42.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.025 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.025 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:23.407118+0000", 2024-02-19T21:16:42.025 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0397999999999999e-05 2024-02-19T21:16:42.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.025 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.025 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.026 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:23.407129+0000", 2024-02-19T21:16:42.026 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6440000000000003e-05 2024-02-19T21:16:42.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.026 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.026 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:23.407175+0000", 2024-02-19T21:16:42.026 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5908999999999999e-05 2024-02-19T21:16:42.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.026 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:16:42.026 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:23.407201+0000", 2024-02-19T21:16:42.026 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00070405499999999998 2024-02-19T21:16:42.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.026 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:16:42.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:23.407905+0000", 2024-02-19T21:16:42.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1232999999999997e-05 2024-02-19T21:16:42.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.027 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.027 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:23.407936+0000", 2024-02-19T21:16:42.027 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00086011599999999996 2024-02-19T21:16:42.027 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.027 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.027 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.027 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.027 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.027 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.23517.0:2000 249.6 e734/728 249:602f83fe:::foo:head v 734'2, mlcod=0'0)", 2024-02-19T21:16:42.028 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:24.410868+0000", 2024-02-19T21:16:42.028 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 17.422538980999999, 2024-02-19T21:16:42.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00056388100000000002, 2024-02-19T21:16:42.028 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.028 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:42.028 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:24.410868+0000", 2024-02-19T21:16:42.028 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.028 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.028 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.028 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:24.410868+0000", 2024-02-19T21:16:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999962 2024-02-19T21:16:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:24.410865+0000", 2024-02-19T21:16:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0970000000000003e-05 2024-02-19T21:16:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:24.410896+0000", 2024-02-19T21:16:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8875000000000001e-05 2024-02-19T21:16:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.029 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:24.410914+0000", 2024-02-19T21:16:42.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8809999999999996e-06 2024-02-19T21:16:42.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:24.410922+0000", 2024-02-19T21:16:42.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3353000000000003e-05 2024-02-19T21:16:42.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.030 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:24.410966+0000", 2024-02-19T21:16:42.030 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8859999999999999e-05 2024-02-19T21:16:42.030 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.030 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.030 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:16:42.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:24.410984+0000", 2024-02-19T21:16:42.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000407997 2024-02-19T21:16:42.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:16:42.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:24.411392+0000", 2024-02-19T21:16:42.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9855999999999999e-05 2024-02-19T21:16:42.031 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.031 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.031 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.031 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:24.411432+0000", 2024-02-19T21:16:42.031 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00056388100000000002 2024-02-19T21:16:42.031 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.031 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.031 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.032 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.23517.0:2002 249.6 e735/728 249:602f83fe:::foo:head v 735'3, mlcod=734'1)", 2024-02-19T21:16:42.032 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:25.413352+0000", 2024-02-19T21:16:42.032 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 16.420055506000001, 2024-02-19T21:16:42.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00058252100000000002, 2024-02-19T21:16:42.032 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.032 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:42.032 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.032 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.032 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.032 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:25.413352+0000", 2024-02-19T21:16:42.032 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:25.413352+0000", 2024-02-19T21:16:42.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:16:42.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:25.413351+0000", 2024-02-19T21:16:42.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2249000000000002e-05 2024-02-19T21:16:42.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.033 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.033 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.033 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:25.413393+0000", 2024-02-19T21:16:42.033 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0923e-05 2024-02-19T21:16:42.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.034 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:25.413404+0000", 2024-02-19T21:16:42.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.2200000000000003e-06 2024-02-19T21:16:42.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.034 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.034 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:25.413411+0000", 2024-02-19T21:16:42.034 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2592000000000001e-05 2024-02-19T21:16:42.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.034 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:25.413464+0000", 2024-02-19T21:16:42.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1787999999999999e-05 2024-02-19T21:16:42.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:16:42.035 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:25.413486+0000", 2024-02-19T21:16:42.035 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00041993 2024-02-19T21:16:42.035 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.035 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.035 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:16:42.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:25.413906+0000", 2024-02-19T21:16:42.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8802999999999998e-05 2024-02-19T21:16:42.036 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.036 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.036 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.036 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:25.413934+0000", 2024-02-19T21:16:42.036 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00058252100000000002 2024-02-19T21:16:42.036 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.036 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.036 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.037 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.037 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.4.0:662 249.6 e736/728 249:60000000:.ceph-internal::hit_set_249.6_archive_2024-02-19 21%3a16%3a23.401526Z_2024-02-19 21%3a16%3a26.415141Z:head v 736'4, has_updated_hit_set_history, mlcod=734'2)", 2024-02-19T21:16:42.037 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:26.415503+0000", 2024-02-19T21:16:42.037 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.417904432, 2024-02-19T21:16:42.037 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00066899699999999995, 2024-02-19T21:16:42.037 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.037 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:42.037 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.037 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.037 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.415503+0000", 2024-02-19T21:16:42.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.415503+0000", 2024-02-19T21:16:42.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967296 2024-02-19T21:16:42.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.038 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.038 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.415503+0000", 2024-02-19T21:16:42.038 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5209999999999999e-06 2024-02-19T21:16:42.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.038 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.415509+0000", 2024-02-19T21:16:42.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3652999999999999e-05 2024-02-19T21:16:42.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.415533+0000", 2024-02-19T21:16:42.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9340000000000006e-06 2024-02-19T21:16:42.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.039 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.039 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.415541+0000", 2024-02-19T21:16:42.039 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3260000000000003e-05 2024-02-19T21:16:42.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.039 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.415584+0000", 2024-02-19T21:16:42.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1235000000000002e-05 2024-02-19T21:16:42.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:16:42.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.415615+0000", 2024-02-19T21:16:42.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00051664199999999999 2024-02-19T21:16:42.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.040 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.040 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:16:42.040 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.416132+0000", 2024-02-19T21:16:42.040 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9988999999999999e-05 2024-02-19T21:16:42.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.041 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.041 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.416172+0000", 2024-02-19T21:16:42.041 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00066899699999999995 2024-02-19T21:16:42.041 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.041 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.041 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.041 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.041 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.23517.0:2004 249.6 e736/728 249:602f83fe:::foo:head v 736'5, mlcod=734'2)", 2024-02-19T21:16:42.041 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:26.415915+0000", 2024-02-19T21:16:42.041 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 15.417492641999999, 2024-02-19T21:16:42.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00041509400000000002, 2024-02-19T21:16:42.042 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.042 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:42.042 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.415915+0000", 2024-02-19T21:16:42.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.042 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.042 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.042 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.415915+0000", 2024-02-19T21:16:42.042 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:16:42.042 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.415914+0000", 2024-02-19T21:16:42.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0868000000000001e-05 2024-02-19T21:16:42.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.415934+0000", 2024-02-19T21:16:42.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2113000000000001e-05 2024-02-19T21:16:42.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.043 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.043 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.043 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.415947+0000", 2024-02-19T21:16:42.043 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8850000000000004e-06 2024-02-19T21:16:42.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.415952+0000", 2024-02-19T21:16:42.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0485000000000002e-05 2024-02-19T21:16:42.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.044 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.416003+0000", 2024-02-19T21:16:42.044 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4e-05 2024-02-19T21:16:42.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.044 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.044 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:16:42.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.416017+0000", 2024-02-19T21:16:42.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00028320100000000001 2024-02-19T21:16:42.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:16:42.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.416300+0000", 2024-02-19T21:16:42.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.976e-05 2024-02-19T21:16:42.045 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.045 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.045 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.045 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:26.416330+0000", 2024-02-19T21:16:42.045 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00041509400000000002 2024-02-19T21:16:42.045 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.045 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.046 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.046 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.046 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.23517.0:2006 249.6 e736/728 249:602f83fe:::foo:head v 736'6, mlcod=736'4)", 2024-02-19T21:16:42.046 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:27.418047+0000", 2024-02-19T21:16:42.046 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 14.415360604, 2024-02-19T21:16:42.046 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00055395900000000003, 2024-02-19T21:16:42.046 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.046 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:42.046 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.046 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.046 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.046 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:27.418047+0000", 2024-02-19T21:16:42.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:27.418047+0000", 2024-02-19T21:16:42.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:16:42.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.047 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.047 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.047 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:27.418044+0000", 2024-02-19T21:16:42.047 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6863999999999999e-05 2024-02-19T21:16:42.047 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:27.418071+0000", 2024-02-19T21:16:42.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9715e-05 2024-02-19T21:16:42.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.048 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:27.418091+0000", 2024-02-19T21:16:42.048 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9510000000000006e-06 2024-02-19T21:16:42.048 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.048 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.048 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:27.418099+0000", 2024-02-19T21:16:42.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4129000000000003e-05 2024-02-19T21:16:42.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:27.418153+0000", 2024-02-19T21:16:42.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.4307e-05 2024-02-19T21:16:42.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.049 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:16:42.049 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:27.418177+0000", 2024-02-19T21:16:42.049 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00039142000000000002 2024-02-19T21:16:42.049 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.049 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:16:42.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:27.418569+0000", 2024-02-19T21:16:42.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1983999999999997e-05 2024-02-19T21:16:42.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.050 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.050 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:27.418601+0000", 2024-02-19T21:16:42.050 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00055395900000000003 2024-02-19T21:16:42.050 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.050 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.050 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.050 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.050 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.050 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.23517.0:2008 249.6 e736/728 249:602f83fe:::foo:head v 736'7, mlcod=736'5)", 2024-02-19T21:16:42.051 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:28.420349+0000", 2024-02-19T21:16:42.051 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 13.413058548, 2024-02-19T21:16:42.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00064658300000000005, 2024-02-19T21:16:42.051 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.051 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:42.051 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:28.420349+0000", 2024-02-19T21:16:42.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.051 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.051 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.051 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.051 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:28.420349+0000", 2024-02-19T21:16:42.051 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999962 2024-02-19T21:16:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:28.420345+0000", 2024-02-19T21:16:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2079e-05 2024-02-19T21:16:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:28.420407+0000", 2024-02-19T21:16:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1639e-05 2024-02-19T21:16:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.052 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:28.420439+0000", 2024-02-19T21:16:42.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.26e-05 2024-02-19T21:16:42.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:28.420451+0000", 2024-02-19T21:16:42.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000102533 2024-02-19T21:16:42.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.053 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:28.420554+0000", 2024-02-19T21:16:42.053 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2005e-05 2024-02-19T21:16:42.053 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.053 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.053 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:16:42.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:28.420576+0000", 2024-02-19T21:16:42.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00038861400000000002 2024-02-19T21:16:42.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:16:42.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:28.420965+0000", 2024-02-19T21:16:42.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0836000000000001e-05 2024-02-19T21:16:42.054 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.054 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.054 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.054 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:28.420995+0000", 2024-02-19T21:16:42.054 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00064658300000000005 2024-02-19T21:16:42.054 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.054 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.054 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.055 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.4.0:666 249.6 e736/728 249:60000000:.ceph-internal::hit_set_249.6_archive_2024-02-19 21%3a16%3a26.415228Z_2024-02-19 21%3a16%3a29.422295Z:head v 736'8, has_updated_hit_set_history, mlcod=736'6)", 2024-02-19T21:16:42.055 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:29.422674+0000", 2024-02-19T21:16:42.055 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.410733629999999, 2024-02-19T21:16:42.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046344336999999999, 2024-02-19T21:16:42.055 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.055 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:42.055 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.055 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.055 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422674+0000", 2024-02-19T21:16:42.055 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.055 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.055 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.056 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.056 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422674+0000", 2024-02-19T21:16:42.056 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999957 2024-02-19T21:16:42.056 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.056 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.056 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.056 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422669+0000", 2024-02-19T21:16:42.056 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9959999999999999e-05 2024-02-19T21:16:42.056 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.056 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.056 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.056 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422689+0000", 2024-02-19T21:16:42.056 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2940999999999999e-05 2024-02-19T21:16:42.056 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.056 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422712+0000", 2024-02-19T21:16:42.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.969e-06 2024-02-19T21:16:42.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422721+0000", 2024-02-19T21:16:42.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3465000000000003e-05 2024-02-19T21:16:42.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.057 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.057 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.057 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422785+0000", 2024-02-19T21:16:42.057 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3392999999999997e-05 2024-02-19T21:16:42.057 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:16:42.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422818+0000", 2024-02-19T21:16:42.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046149449000000002 2024-02-19T21:16:42.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:16:42.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.468968+0000", 2024-02-19T21:16:42.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0578999999999998e-05 2024-02-19T21:16:42.058 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.058 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.058 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.058 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.469018+0000", 2024-02-19T21:16:42.058 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.046344336999999999 2024-02-19T21:16:42.058 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.059 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.059 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.059 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.059 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.23517.0:2010 249.6 e736/728 249:602f83fe:::foo:head v 736'9, mlcod=736'6)", 2024-02-19T21:16:42.059 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:29.422980+0000", 2024-02-19T21:16:42.059 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 12.410427057, 2024-02-19T21:16:42.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.048678160999999998, 2024-02-19T21:16:42.059 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.059 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:16:42.059 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.059 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.059 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.059 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422980+0000", 2024-02-19T21:16:42.059 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.060 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.060 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.060 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422980+0000", 2024-02-19T21:16:42.060 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:16:42.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.060 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.060 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.060 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422979+0000", 2024-02-19T21:16:42.060 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2678000000000001e-05 2024-02-19T21:16:42.060 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.060 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.060 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.060 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.423012+0000", 2024-02-19T21:16:42.061 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6739999999999997e-06 2024-02-19T21:16:42.061 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.061 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.061 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.061 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.423020+0000", 2024-02-19T21:16:42.061 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1150000000000004e-06 2024-02-19T21:16:42.061 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.061 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.061 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.061 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.423024+0000", 2024-02-19T21:16:42.061 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0899999999999997e-05 2024-02-19T21:16:42.061 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.061 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.061 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.062 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.423075+0000", 2024-02-19T21:16:42.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0253000000000001e-05 2024-02-19T21:16:42.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.062 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:16:42.062 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.423095+0000", 2024-02-19T21:16:42.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.048531584000000003 2024-02-19T21:16:42.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.062 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:16:42.062 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.471627+0000", 2024-02-19T21:16:42.062 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1977999999999998e-05 2024-02-19T21:16:42.062 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.062 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.063 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.063 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.471659+0000", 2024-02-19T21:16:42.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.048678160999999998 2024-02-19T21:16:42.063 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.063 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.063 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.063 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.063 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.063 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(234.5s2 742/740 ECSubWrite(tid=681, reqid=client.23517.0:2125, at_version=742'12, trim_to=0'0, roll_forward_to=0'0))", 2024-02-19T21:16:42.063 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:38.614740+0000", 2024-02-19T21:16:42.063 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.2186673990000001, 2024-02-19T21:16:42.063 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0011357330000000001, 2024-02-19T21:16:42.063 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.063 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:16:42.064 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.064 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.614740+0000", 2024-02-19T21:16:42.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.064 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.614740+0000", 2024-02-19T21:16:42.064 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:16:42.064 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.064 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.064 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.064 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.614740+0000", 2024-02-19T21:16:42.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4150999999999999e-05 2024-02-19T21:16:42.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.614774+0000", 2024-02-19T21:16:42.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1184e-05 2024-02-19T21:16:42.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.065 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.614785+0000", 2024-02-19T21:16:42.065 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0989999999999999e-06 2024-02-19T21:16:42.065 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.065 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.065 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.614793+0000", 2024-02-19T21:16:42.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8330000000000004e-05 2024-02-19T21:16:42.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.614871+0000", 2024-02-19T21:16:42.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1883e-05 2024-02-19T21:16:42.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.066 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:16:42.066 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.614883+0000", 2024-02-19T21:16:42.066 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00097252999999999999 2024-02-19T21:16:42.066 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.066 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:16:42.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.615856+0000", 2024-02-19T21:16:42.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0027999999999999e-05 2024-02-19T21:16:42.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.067 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.067 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.615876+0000", 2024-02-19T21:16:42.067 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0011357330000000001 2024-02-19T21:16:42.067 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.067 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.067 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.067 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.067 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.067 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(234.5s2 742/740 ECSubWrite(tid=682, reqid=client.23517.0:2125, at_version=0'0, trim_to=0'0, roll_forward_to=742'12))", 2024-02-19T21:16:42.068 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:38.619475+0000", 2024-02-19T21:16:42.068 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.2139322219999999, 2024-02-19T21:16:42.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018130920000000001, 2024-02-19T21:16:42.068 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.068 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:16:42.068 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.068 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.619475+0000", 2024-02-19T21:16:42.068 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.068 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.068 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.068 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.069 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.619475+0000", 2024-02-19T21:16:42.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:16:42.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.069 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.069 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.619474+0000", 2024-02-19T21:16:42.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3720000000000001e-05 2024-02-19T21:16:42.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.069 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.069 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.619487+0000", 2024-02-19T21:16:42.069 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2656e-05 2024-02-19T21:16:42.069 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.069 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.070 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.070 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.619500+0000", 2024-02-19T21:16:42.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6249999999999998e-06 2024-02-19T21:16:42.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.070 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.070 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.619508+0000", 2024-02-19T21:16:42.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4297e-05 2024-02-19T21:16:42.070 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.070 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.070 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.070 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.619552+0000", 2024-02-19T21:16:42.070 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0171e-05 2024-02-19T21:16:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:16:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.619562+0000", 2024-02-19T21:16:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0017164249999999999 2024-02-19T21:16:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:16:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.621279+0000", 2024-02-19T21:16:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7699999999999996e-06 2024-02-19T21:16:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.621288+0000", 2024-02-19T21:16:42.071 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018130920000000001 2024-02-19T21:16:42.072 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.072 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.072 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.072 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.072 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(234.2s1 742/735 ECSubWrite(tid=571, reqid=client.23517.0:2129, at_version=742'6, trim_to=0'0, roll_forward_to=0'0))", 2024-02-19T21:16:42.072 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:38.629193+0000", 2024-02-19T21:16:42.072 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.2042149279999999, 2024-02-19T21:16:42.072 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00042385600000000002, 2024-02-19T21:16:42.072 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.072 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:16:42.072 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.072 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.629193+0000", 2024-02-19T21:16:42.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.629193+0000", 2024-02-19T21:16:42.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:16:42.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.073 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.073 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.629192+0000", 2024-02-19T21:16:42.073 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9329000000000001e-05 2024-02-19T21:16:42.073 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.073 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.074 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.074 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.629212+0000", 2024-02-19T21:16:42.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0518e-05 2024-02-19T21:16:42.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.074 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.074 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.629222+0000", 2024-02-19T21:16:42.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4159999999999998e-06 2024-02-19T21:16:42.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.074 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.074 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.629230+0000", 2024-02-19T21:16:42.074 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6063999999999998e-05 2024-02-19T21:16:42.074 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.074 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.629276+0000", 2024-02-19T21:16:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.7849999999999992e-06 2024-02-19T21:16:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:16:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.629283+0000", 2024-02-19T21:16:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00031799900000000002 2024-02-19T21:16:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:16:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.629601+0000", 2024-02-19T21:16:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4999e-05 2024-02-19T21:16:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.076 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.076 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.629616+0000", 2024-02-19T21:16:42.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00042385600000000002 2024-02-19T21:16:42.076 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.076 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.076 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.076 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.076 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.076 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(234.5s2 742/740 ECSubWrite(tid=687, reqid=client.23517.0:2130, at_version=742'13, trim_to=0'0, roll_forward_to=742'12))", 2024-02-19T21:16:42.076 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:38.630707+0000", 2024-02-19T21:16:42.076 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.2026999749999998, 2024-02-19T21:16:42.076 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000430114, 2024-02-19T21:16:42.076 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.076 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:16:42.076 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.630707+0000", 2024-02-19T21:16:42.077 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.630707+0000", 2024-02-19T21:16:42.077 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:16:42.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.077 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.077 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.077 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.630706+0000", 2024-02-19T21:16:42.077 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2487999999999999e-05 2024-02-19T21:16:42.077 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.078 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.630729+0000", 2024-02-19T21:16:42.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1834999999999999e-05 2024-02-19T21:16:42.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.078 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.630740+0000", 2024-02-19T21:16:42.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8849999999999999e-06 2024-02-19T21:16:42.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.078 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.078 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.078 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.630748+0000", 2024-02-19T21:16:42.078 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5012000000000002e-05 2024-02-19T21:16:42.078 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.630793+0000", 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0812000000000001e-05 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.630804+0000", 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00031684999999999998 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.631121+0000", 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6592999999999999e-05 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.079 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.080 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.080 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.631138+0000", 2024-02-19T21:16:42.080 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000430114 2024-02-19T21:16:42.080 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.080 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.080 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.080 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.080 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.080 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(234.5s2 742/740 ECSubWrite(tid=688, reqid=client.23517.0:2130, at_version=0'0, trim_to=0'0, roll_forward_to=742'13))", 2024-02-19T21:16:42.080 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:38.632290+0000", 2024-02-19T21:16:42.080 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.2011169759999998, 2024-02-19T21:16:42.080 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00051898999999999999, 2024-02-19T21:16:42.080 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.080 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:16:42.080 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.632290+0000", 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.632290+0000", 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.632290+0000", 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6110000000000001e-06 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.081 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.632296+0000", 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8529999999999997e-06 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.632302+0000", 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7660000000000004e-06 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.632307+0000", 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.7031000000000001e-05 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.082 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.632394+0000", 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0350000000000007e-06 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.632403+0000", 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00038696200000000003 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.632790+0000", 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9114999999999999e-05 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.632809+0000", 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00051898999999999999 2024-02-19T21:16:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.084 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.084 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.084 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.084 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(234.1s1 742/636 ECSubWrite(tid=719, reqid=client.23517.0:2139, at_version=742'8, trim_to=0'0, roll_forward_to=742'7))", 2024-02-19T21:16:42.084 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:38.646884+0000", 2024-02-19T21:16:42.084 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.1865230530000002, 2024-02-19T21:16:42.084 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000385528, 2024-02-19T21:16:42.084 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:16:42.084 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:16:42.084 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:16:42.084 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.084 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:16:42.084 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.646884+0000", 2024-02-19T21:16:42.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:16:42.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:16:42.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.646884+0000", 2024-02-19T21:16:42.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:16:42.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:16:42.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.646883+0000", 2024-02-19T21:16:42.085 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8213e-05 2024-02-19T21:16:42.085 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.085 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.085 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:16:42.085 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.646901+0000", 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0482999999999999e-05 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.646912+0000", 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7509999999999999e-06 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.646916+0000", 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7413000000000002e-05 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.646964+0000", 2024-02-19T21:16:42.086 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2380000000000002e-06 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.646970+0000", 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00028553500000000001 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.647256+0000", 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4341999999999999e-05 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.647270+0000", 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000385528 2024-02-19T21:16:42.087 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.088 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.088 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.088 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:16:42.088 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:16:42.088 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:16:42.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.370+0000 7fe9151ac700 1 -- 172.21.15.163:0/3907866650 >> v1:172.21.15.163:6789/0 conn(0x7fe910100780 legacy=0x7fe910102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:42.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.370+0000 7fe9151ac700 1 -- 172.21.15.163:0/3907866650 shutdown_connections 2024-02-19T21:16:42.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.370+0000 7fe9151ac700 1 -- 172.21.15.163:0/3907866650 >> 172.21.15.163:0/3907866650 conn(0x7fe9100f35a0 msgr2=0x7fe9100f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:42.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.370+0000 7fe9151ac700 1 -- 172.21.15.163:0/3907866650 shutdown_connections 2024-02-19T21:16:42.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.370+0000 7fe9151ac700 1 -- 172.21.15.163:0/3907866650 wait complete. 2024-02-19T21:16:42.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.371+0000 7fe9151ac700 1 librados: starting msgr at 2024-02-19T21:16:42.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.371+0000 7fe9151ac700 1 librados: starting objecter 2024-02-19T21:16:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.371+0000 7fe9151ac700 1 Processor -- start 2024-02-19T21:16:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.371+0000 7fe9151ac700 1 -- start start 2024-02-19T21:16:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.371+0000 7fe9151ac700 1 librados: setting wanted keys 2024-02-19T21:16:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.371+0000 7fe9151ac700 1 librados: calling monclient init 2024-02-19T21:16:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.371+0000 7fe9151ac700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe91006cb60 con 0x7fe9100fd980 2024-02-19T21:16:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.371+0000 7fe9151ac700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe91006cd30 con 0x7fe9101042c0 2024-02-19T21:16:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.371+0000 7fe9151ac700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe91006cf20 con 0x7fe910100780 2024-02-19T21:16:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.371+0000 7fe90ed9d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fe9100fd980 0x7fe9100fc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:53246/0 (socket says 172.21.15.163:53246) 2024-02-19T21:16:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.371+0000 7fe90ed9d700 1 -- 172.21.15.163:0/4083781997 learned_addr learned my addr 172.21.15.163:0/4083781997 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 10 client.?.objecter ms_handle_connect 0x7fe9100fd980 2024-02-19T21:16:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 10 client.?.objecter ms_handle_connect 0x7fe910100780 2024-02-19T21:16:42.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 10 client.?.objecter ms_handle_connect 0x7fe9101042c0 2024-02-19T21:16:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797198670 0 0) 0x7fe91006cb60 con 0x7fe9100fd980 2024-02-19T21:16:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8f8003030 con 0x7fe9100fd980 2024-02-19T21:16:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314181463 0 0) 0x7fe91006cf20 con 0x7fe910100780 2024-02-19T21:16:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe91006cb60 con 0x7fe910100780 2024-02-19T21:16:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056361893 0 0) 0x7fe91006cd30 con 0x7fe9101042c0 2024-02-19T21:16:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe91006cf20 con 0x7fe9101042c0 2024-02-19T21:16:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3627092435 0 0) 0x7fe8f8003030 con 0x7fe9100fd980 2024-02-19T21:16:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe91006cd30 con 0x7fe9100fd980 2024-02-19T21:16:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2507866124 0 0) 0x7fe91006cb60 con 0x7fe910100780 2024-02-19T21:16:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8f8003030 con 0x7fe910100780 2024-02-19T21:16:42.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe904001940 con 0x7fe9100fd980 2024-02-19T21:16:42.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe900001e50 con 0x7fe910100780 2024-02-19T21:16:42.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3012251465 0 0) 0x7fe91006cf20 con 0x7fe9101042c0 2024-02-19T21:16:42.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe91006cb60 con 0x7fe9101042c0 2024-02-19T21:16:42.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe9080019a0 con 0x7fe9101042c0 2024-02-19T21:16:42.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3593634634 0 0) 0x7fe8f8003030 con 0x7fe910100780 2024-02-19T21:16:42.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 >> v1:172.21.15.193:6789/0 conn(0x7fe9101042c0 legacy=0x7fe910118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:42.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 >> v1:172.21.15.163:6789/0 conn(0x7fe9100fd980 legacy=0x7fe9100fc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:42.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91011cb90 con 0x7fe910100780 2024-02-19T21:16:42.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe9151ac700 1 -- 172.21.15.163:0/4083781997 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe910119bc0 con 0x7fe910100780 2024-02-19T21:16:42.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe9151ac700 10 client.50549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:42.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe9151ac700 1 -- 172.21.15.163:0/4083781997 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91011a030 con 0x7fe910100780 2024-02-19T21:16:42.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe9151ac700 1 librados: init done 2024-02-19T21:16:42.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fe900001430 con 0x7fe910100780 2024-02-19T21:16:42.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.372+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe900003540 con 0x7fe910100780 2024-02-19T21:16:42.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.373+0000 7fe9151ac700 1 -- 172.21.15.163:0/4083781997 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8fc004f20 con 0x7fe910100780 2024-02-19T21:16:42.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.374+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fe900016450 con 0x7fe910100780 2024-02-19T21:16:42.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.374+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(745..745 src has 623..745) v4 ==== 8475+0+0 (unknown 2210477208 0 0) 0x7fe900076870 con 0x7fe910100780 2024-02-19T21:16:42.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.374+0000 7fe8f7fff700 10 client.50549.objecter ms_dispatch 0x7fe910001550 osd_map(745..745 src has 623..745) v4 2024-02-19T21:16:42.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.374+0000 7fe8f7fff700 3 client.50549.objecter handle_osd_map got epochs [745,745] > 0 2024-02-19T21:16:42.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.374+0000 7fe8f7fff700 3 client.50549.objecter handle_osd_map decoding full epoch 745 2024-02-19T21:16:42.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.374+0000 7fe8f7fff700 20 client.50549.objecter dump_active .. 0 homeless 2024-02-19T21:16:42.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.374+0000 7fe8f7fff700 10 client.50549.objecter ms_handle_connect 0x7fe8f8063cb0 2024-02-19T21:16:42.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.378+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe900049ba0 con 0x7fe910100780 2024-02-19T21:16:42.409 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:42.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:42.513 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:42.541 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:42.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.571+0000 7fe9151ac700 1 -- 172.21.15.163:0/4083781997 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fe8fc0051d0 con 0x7fe910100780 2024-02-19T21:16:42.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.625+0000 7fe8f7fff700 1 -- 172.21.15.163:0/4083781997 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v746) v1 ==== 97+0+0 (unknown 2492100701 0 0) 0x7fe900048010 con 0x7fe910100780 2024-02-19T21:16:42.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.628+0000 7fe9151ac700 10 librados: watch_flush enter 2024-02-19T21:16:42.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.628+0000 7fe9151ac700 10 librados: watch_flush exit 2024-02-19T21:16:42.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.628+0000 7fe9151ac700 20 client.50549.objecter shutdown clearing up homeless session... 2024-02-19T21:16:42.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.628+0000 7fe9151ac700 10 client.50549.objecter successfully canceled tick 2024-02-19T21:16:42.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.628+0000 7fe9151ac700 1 -- 172.21.15.163:0/4083781997 >> v1:172.21.15.163:6800/1552514681 conn(0x7fe8f8063cb0 legacy=0x7fe8f8066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:42.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.628+0000 7fe9151ac700 1 -- 172.21.15.163:0/4083781997 >> v1:172.21.15.163:6790/0 conn(0x7fe910100780 legacy=0x7fe910114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:42.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.628+0000 7fe9151ac700 1 -- 172.21.15.163:0/4083781997 shutdown_connections 2024-02-19T21:16:42.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.628+0000 7fe9151ac700 1 -- 172.21.15.163:0/4083781997 >> 172.21.15.163:0/4083781997 conn(0x7fe9100f35a0 msgr2=0x7fe9100f5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:42.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.628+0000 7fe9151ac700 1 -- 172.21.15.163:0/4083781997 shutdown_connections 2024-02-19T21:16:42.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.628+0000 7fe9151ac700 1 -- 172.21.15.163:0/4083781997 wait complete. 2024-02-19T21:16:42.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:42.628+0000 7fe9151ac700 1 librados: shutdown 2024-02-19T21:16:42.633 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:16:42.662 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 7, 3, 4, 6, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:16:42.663 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:16:42.663 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-02-19T21:16:42.663 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-19T21:16:42.663 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-19T21:16:42.663 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-02-19T21:16:42.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:41.325460+0000 mgr.y (mgr.14550) 625 : cluster [DBG] pgmap v1126: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 5.0 KiB/s wr, 5 op/s; 977 KiB/s, 17 objects/s recovering 2024-02-19T21:16:42.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:41.625919+0000 mon.a (mon.0) 3555 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-132", "mode": "writeback"}]': finished 2024-02-19T21:16:42.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:41.625981+0000 mon.a (mon.0) 3556 : cluster [DBG] osdmap e745: 8 total, 8 up, 7 in 2024-02-19T21:16:42.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:41.628524+0000 mon.a (mon.0) 3557 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:16:42.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:42.572560+0000 mon.c (mon.2) 433 : audit [INF] from='client.? v1:172.21.15.163:0/4083781997' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:16:42.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:42.572892+0000 mon.a (mon.0) 3558 : audit [INF] from='client.50549 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:16:42.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:42.622725+0000 mon.a (mon.0) 3559 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:16:42.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:42.626493+0000 mon.a (mon.0) 3560 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:16:42.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:42.626546+0000 mon.a (mon.0) 3561 : audit [INF] from='client.50549 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:16:42.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:42.626592+0000 mon.a (mon.0) 3562 : cluster [DBG] osdmap e746: 8 total, 8 up, 7 in 2024-02-19T21:16:42.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:42.627198+0000 mon.a (mon.0) 3563 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:16:42.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:41.325460+0000 mgr.y (mgr.14550) 625 : cluster [DBG] pgmap v1126: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 5.0 KiB/s wr, 5 op/s; 977 KiB/s, 17 objects/s recovering 2024-02-19T21:16:42.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:41.625919+0000 mon.a (mon.0) 3555 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-132", "mode": "writeback"}]': finished 2024-02-19T21:16:42.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:41.625981+0000 mon.a (mon.0) 3556 : cluster [DBG] osdmap e745: 8 total, 8 up, 7 in 2024-02-19T21:16:42.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:41.628524+0000 mon.a (mon.0) 3557 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:16:42.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:42.572560+0000 mon.c (mon.2) 433 : audit [INF] from='client.? v1:172.21.15.163:0/4083781997' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:16:42.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:42.572892+0000 mon.a (mon.0) 3558 : audit [INF] from='client.50549 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:16:42.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:42.622725+0000 mon.a (mon.0) 3559 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:16:42.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:42.626493+0000 mon.a (mon.0) 3560 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:16:42.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:42.626546+0000 mon.a (mon.0) 3561 : audit [INF] from='client.50549 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:16:42.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:42.626592+0000 mon.a (mon.0) 3562 : cluster [DBG] osdmap e746: 8 total, 8 up, 7 in 2024-02-19T21:16:42.751 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:42.627198+0000 mon.a (mon.0) 3563 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:16:42.890 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:42.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:42.935 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:16:42.998 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:43.024 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:43.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:41.325460+0000 mgr.y (mgr.14550) 625 : cluster [DBG] pgmap v1126: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 5.0 KiB/s wr, 5 op/s; 977 KiB/s, 17 objects/s recovering 2024-02-19T21:16:43.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:41.625919+0000 mon.a (mon.0) 3555 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-132", "mode": "writeback"}]': finished 2024-02-19T21:16:43.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:41.625981+0000 mon.a (mon.0) 3556 : cluster [DBG] osdmap e745: 8 total, 8 up, 7 in 2024-02-19T21:16:43.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:41.628524+0000 mon.a (mon.0) 3557 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:16:43.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:42.572560+0000 mon.c (mon.2) 433 : audit [INF] from='client.? v1:172.21.15.163:0/4083781997' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:16:43.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:42.572892+0000 mon.a (mon.0) 3558 : audit [INF] from='client.50549 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:16:43.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:42.622725+0000 mon.a (mon.0) 3559 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:16:43.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:42.626493+0000 mon.a (mon.0) 3560 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:16:43.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:42.626546+0000 mon.a (mon.0) 3561 : audit [INF] from='client.50549 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:16:43.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:42.626592+0000 mon.a (mon.0) 3562 : cluster [DBG] osdmap e746: 8 total, 8 up, 7 in 2024-02-19T21:16:43.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:42.627198+0000 mon.a (mon.0) 3563 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:16:43.672 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:16:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:16:43] "GET /metrics HTTP/1.1" 200 192486 "" "Prometheus/2.33.4" 2024-02-19T21:16:43.673 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:43.676 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:43.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:43.693 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-19T21:16:43.777 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:43.804 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:43.902 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:43.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:44.005 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:16:44.035 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:44.283 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:16:44.285 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:44.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:16:44.387 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:44.418 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:44.567 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:44.569 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:44.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:44.648 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:43.325816+0000 mgr.y (mgr.14550) 626 : cluster [DBG] pgmap v1129: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 5.0 KiB/s wr, 5 op/s; 977 KiB/s, 17 objects/s recovering 2024-02-19T21:16:44.648 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:43.628013+0000 mon.a (mon.0) 3564 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:16:44.648 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:43.628081+0000 mon.a (mon.0) 3565 : cluster [DBG] osdmap e747: 8 total, 8 up, 7 in 2024-02-19T21:16:44.648 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:43.629368+0000 mon.a (mon.0) 3566 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:16:44.648 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:44.399172+0000 mon.a (mon.0) 3567 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:16:44.648 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:44.405488+0000 mon.a (mon.0) 3568 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:16:44.648 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:44.405549+0000 mon.a (mon.0) 3569 : cluster [DBG] osdmap e748: 8 total, 8 up, 7 in 2024-02-19T21:16:44.648 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:44.406911+0000 mon.a (mon.0) 3570 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:16:44.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:43.325816+0000 mgr.y (mgr.14550) 626 : cluster [DBG] pgmap v1129: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 5.0 KiB/s wr, 5 op/s; 977 KiB/s, 17 objects/s recovering 2024-02-19T21:16:44.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:43.628013+0000 mon.a (mon.0) 3564 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:16:44.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:43.628081+0000 mon.a (mon.0) 3565 : cluster [DBG] osdmap e747: 8 total, 8 up, 7 in 2024-02-19T21:16:44.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:43.629368+0000 mon.a (mon.0) 3566 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:16:44.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:44.399172+0000 mon.a (mon.0) 3567 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:16:44.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:44.405488+0000 mon.a (mon.0) 3568 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:16:44.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:44.405549+0000 mon.a (mon.0) 3569 : cluster [DBG] osdmap e748: 8 total, 8 up, 7 in 2024-02-19T21:16:44.649 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:44.406911+0000 mon.a (mon.0) 3570 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:16:44.671 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:44.701 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:44.773 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:44.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:44.784 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:16:44.784 INFO:teuthology.orchestra.run.smithi193.stdout: "success": "" 2024-02-19T21:16:44.785 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:16:44.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:43.325816+0000 mgr.y (mgr.14550) 626 : cluster [DBG] pgmap v1129: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 5.0 KiB/s wr, 5 op/s; 977 KiB/s, 17 objects/s recovering 2024-02-19T21:16:44.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:43.628013+0000 mon.a (mon.0) 3564 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:16:44.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:43.628081+0000 mon.a (mon.0) 3565 : cluster [DBG] osdmap e747: 8 total, 8 up, 7 in 2024-02-19T21:16:44.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:43.629368+0000 mon.a (mon.0) 3566 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:16:44.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:44.399172+0000 mon.a (mon.0) 3567 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:16:44.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:44.405488+0000 mon.a (mon.0) 3568 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:16:44.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:44.405549+0000 mon.a (mon.0) 3569 : cluster [DBG] osdmap e748: 8 total, 8 up, 7 in 2024-02-19T21:16:44.810 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:44.406911+0000 mon.a (mon.0) 3570 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:16:44.878 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:44.906 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.862+0000 7fa26e37b700 1 -- 172.21.15.163:0/629155672 >> v1:172.21.15.163:6789/0 conn(0x7fa268100780 legacy=0x7fa268102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.862+0000 7fa26e37b700 1 -- 172.21.15.163:0/629155672 shutdown_connections 2024-02-19T21:16:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.862+0000 7fa26e37b700 1 -- 172.21.15.163:0/629155672 >> 172.21.15.163:0/629155672 conn(0x7fa2680f35a0 msgr2=0x7fa2680f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.862+0000 7fa26e37b700 1 -- 172.21.15.163:0/629155672 shutdown_connections 2024-02-19T21:16:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.863+0000 7fa26e37b700 1 -- 172.21.15.163:0/629155672 wait complete. 2024-02-19T21:16:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.863+0000 7fa26e37b700 1 librados: starting msgr at 2024-02-19T21:16:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.863+0000 7fa26e37b700 1 librados: starting objecter 2024-02-19T21:16:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.863+0000 7fa26e37b700 1 Processor -- start 2024-02-19T21:16:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.863+0000 7fa26e37b700 1 -- start start 2024-02-19T21:16:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.863+0000 7fa26e37b700 1 librados: setting wanted keys 2024-02-19T21:16:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.863+0000 7fa26e37b700 1 librados: calling monclient init 2024-02-19T21:16:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa26e37b700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa2681173b0 con 0x7fa2681042c0 2024-02-19T21:16:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa26e37b700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa2681185b0 con 0x7fa2680fd980 2024-02-19T21:16:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa26e37b700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa2681197b0 con 0x7fa268100780 2024-02-19T21:16:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2677fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fa268100780 0x7fa2680fa110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:52540/0 (socket says 172.21.15.163:52540) 2024-02-19T21:16:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2677fe700 1 -- 172.21.15.163:0/2322208285 learned_addr learned my addr 172.21.15.163:0/2322208285 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 10 client.?.objecter ms_handle_connect 0x7fa268100780 2024-02-19T21:16:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:44.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 10 client.?.objecter ms_handle_connect 0x7fa2681042c0 2024-02-19T21:16:44.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:44.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 10 client.?.objecter ms_handle_connect 0x7fa2680fd980 2024-02-19T21:16:44.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:44.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310882622 0 0) 0x7fa2681197b0 con 0x7fa268100780 2024-02-19T21:16:44.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa250003030 con 0x7fa268100780 2024-02-19T21:16:44.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170608811 0 0) 0x7fa2681173b0 con 0x7fa2681042c0 2024-02-19T21:16:44.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2681197b0 con 0x7fa2681042c0 2024-02-19T21:16:44.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800503966 0 0) 0x7fa2681185b0 con 0x7fa2680fd980 2024-02-19T21:16:44.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2681173b0 con 0x7fa2680fd980 2024-02-19T21:16:44.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2357725735 0 0) 0x7fa250003030 con 0x7fa268100780 2024-02-19T21:16:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2681185b0 con 0x7fa268100780 2024-02-19T21:16:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa25c0020a0 con 0x7fa268100780 2024-02-19T21:16:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1101716895 0 0) 0x7fa2681197b0 con 0x7fa2681042c0 2024-02-19T21:16:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa250003030 con 0x7fa2681042c0 2024-02-19T21:16:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 517103584 0 0) 0x7fa2681173b0 con 0x7fa2680fd980 2024-02-19T21:16:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2681197b0 con 0x7fa2680fd980 2024-02-19T21:16:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa2600019a0 con 0x7fa2681042c0 2024-02-19T21:16:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.864+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa2580017c0 con 0x7fa2680fd980 2024-02-19T21:16:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.865+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1463483541 0 0) 0x7fa2681185b0 con 0x7fa268100780 2024-02-19T21:16:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.865+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 >> v1:172.21.15.193:6789/0 conn(0x7fa2680fd980 legacy=0x7fa2680f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.865+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 >> v1:172.21.15.163:6789/0 conn(0x7fa2681042c0 legacy=0x7fa268115a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.865+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26811a9b0 con 0x7fa268100780 2024-02-19T21:16:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.865+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2681175a0 con 0x7fa268100780 2024-02-19T21:16:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.865+0000 7fa26e37b700 10 client.50555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.865+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa268117a10 con 0x7fa268100780 2024-02-19T21:16:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.865+0000 7fa26e37b700 1 librados: init done 2024-02-19T21:16:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.865+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fa25c002730 con 0x7fa268100780 2024-02-19T21:16:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.865+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa25c0037f0 con 0x7fa268100780 2024-02-19T21:16:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.865+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fa254004f00 con 0x7fa268100780 2024-02-19T21:16:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.866+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fa25c0017f0 con 0x7fa268100780 2024-02-19T21:16:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.866+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(748..748 src has 623..748) v4 ==== 8501+0+0 (unknown 3996760743 0 0) 0x7fa25c079290 con 0x7fa268100780 2024-02-19T21:16:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.866+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 osd_map(748..748 src has 623..748) v4 2024-02-19T21:16:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.866+0000 7fa2657fa700 3 client.50555.objecter handle_osd_map got epochs [748,748] > 0 2024-02-19T21:16:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.866+0000 7fa2657fa700 3 client.50555.objecter handle_osd_map decoding full epoch 748 2024-02-19T21:16:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.867+0000 7fa2657fa700 20 client.50555.objecter dump_active .. 0 homeless 2024-02-19T21:16:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.867+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v748) v1 ==== 54+0+16 (unknown 515422929 0 2420000192) 0x7fa25c07b630 con 0x7fa268100780 2024-02-19T21:16:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.867+0000 7fa2657fa700 10 client.50555.objecter ms_handle_connect 0x7fa250063d00 2024-02-19T21:16:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.867+0000 7fa26e37b700 10 client.50555.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.867+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 1 2024-02-19T21:16:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.867+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa2540051a0 osd=0 3 2024-02-19T21:16:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.867+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa2540051a0 osd=0 3 2024-02-19T21:16:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.867+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa2540051a0 osd=0 3 2024-02-19T21:16:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.867+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 1 2024-02-19T21:16:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.867+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa2540051a0 osd=0 3 2024-02-19T21:16:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.867+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 0 1 2024-02-19T21:16:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.867+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa2540051a0 osd=0 4 2024-02-19T21:16:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.867+0000 7fa26e37b700 10 client.50555.objecter _send_command 1 2024-02-19T21:16:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.867+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa254004950 con 0x7fa254004530 2024-02-19T21:16:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.867+0000 7fa2657fa700 10 client.50555.objecter ms_handle_connect 0x7fa254004530 2024-02-19T21:16:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.868+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa254004950 con 0x7fa254004530 2024-02-19T21:16:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.868+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:16:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.868+0000 7fa2657fa700 10 client.50555.objecter _finish_command 1 = system:0 2024-02-19T21:16:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.868+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa2540051a0 osd=0 4 2024-02-19T21:16:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.868+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 0 1 2024-02-19T21:16:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.882+0000 7fa26e37b700 10 client.50555.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.882+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 2 2024-02-19T21:16:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.882+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa2540051a0 osd=0 3 2024-02-19T21:16:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.882+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa2540051a0 osd=0 3 2024-02-19T21:16:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.882+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa2540051a0 osd=0 3 2024-02-19T21:16:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.882+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 2 2024-02-19T21:16:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.882+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa2540051a0 osd=0 3 2024-02-19T21:16:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.882+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 0 2 2024-02-19T21:16:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.882+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa2540051a0 osd=0 4 2024-02-19T21:16:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.882+0000 7fa26e37b700 10 client.50555.objecter _send_command 2 2024-02-19T21:16:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.882+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa254004d30 con 0x7fa254004530 2024-02-19T21:16:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa254004d30 con 0x7fa254004530 2024-02-19T21:16:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:44.949 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:16:44.950 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:16:44.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa2657fa700 10 client.50555.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:44.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa2540051a0 osd=0 4 2024-02-19T21:16:44.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 0 2 2024-02-19T21:16:44.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa26e37b700 10 client.50555.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:44.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 3 2024-02-19T21:16:44.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254003fe0 osd=1 3 2024-02-19T21:16:44.950 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254003fe0 osd=1 3 2024-02-19T21:16:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254003fe0 osd=1 3 2024-02-19T21:16:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 3 2024-02-19T21:16:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa254003fe0 osd=1 3 2024-02-19T21:16:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 1 3 2024-02-19T21:16:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254003fe0 osd=1 4 2024-02-19T21:16:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa26e37b700 10 client.50555.objecter _send_command 3 2024-02-19T21:16:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa254010450 con 0x7fa254004150 2024-02-19T21:16:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.884+0000 7fa2657fa700 10 client.50555.objecter ms_handle_connect 0x7fa254004150 2024-02-19T21:16:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.885+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa254010450 con 0x7fa254004150 2024-02-19T21:16:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.885+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:16:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.885+0000 7fa2657fa700 10 client.50555.objecter _finish_command 3 = system:0 2024-02-19T21:16:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.885+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa254003fe0 osd=1 4 2024-02-19T21:16:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.885+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 1 3 2024-02-19T21:16:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.900+0000 7fa26e37b700 10 client.50555.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.900+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 4 2024-02-19T21:16:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.900+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254003fe0 osd=1 3 2024-02-19T21:16:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.900+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254003fe0 osd=1 3 2024-02-19T21:16:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.900+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254003fe0 osd=1 3 2024-02-19T21:16:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.900+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 4 2024-02-19T21:16:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.900+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa254003fe0 osd=1 3 2024-02-19T21:16:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.900+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 1 4 2024-02-19T21:16:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.900+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254003fe0 osd=1 4 2024-02-19T21:16:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.900+0000 7fa26e37b700 10 client.50555.objecter _send_command 4 2024-02-19T21:16:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.900+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa2540018b0 con 0x7fa254004150 2024-02-19T21:16:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa2540018b0 con 0x7fa254004150 2024-02-19T21:16:44.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:44.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa2657fa700 10 client.50555.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:44.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa254003fe0 osd=1 4 2024-02-19T21:16:44.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 1 4 2024-02-19T21:16:44.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa26e37b700 10 client.50555.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:44.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 5 2024-02-19T21:16:44.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254003630 osd=2 3 2024-02-19T21:16:44.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254003630 osd=2 3 2024-02-19T21:16:44.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254003630 osd=2 3 2024-02-19T21:16:44.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 5 2024-02-19T21:16:44.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa254003630 osd=2 3 2024-02-19T21:16:44.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 2 5 2024-02-19T21:16:44.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254003630 osd=2 4 2024-02-19T21:16:44.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa26e37b700 10 client.50555.objecter _send_command 5 2024-02-19T21:16:44.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2540122a0 con 0x7fa254011740 2024-02-19T21:16:44.954 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:16:44.954 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:16:44.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.902+0000 7fa2657fa700 10 client.50555.objecter ms_handle_connect 0x7fa254011740 2024-02-19T21:16:44.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.903+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa2540122a0 con 0x7fa254011740 2024-02-19T21:16:44.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.903+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:16:44.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.903+0000 7fa2657fa700 10 client.50555.objecter _finish_command 5 = system:0 2024-02-19T21:16:44.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.903+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa254003630 osd=2 4 2024-02-19T21:16:44.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.903+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 2 5 2024-02-19T21:16:44.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.917+0000 7fa26e37b700 10 client.50555.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:44.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.917+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 6 2024-02-19T21:16:44.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.917+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254003630 osd=2 3 2024-02-19T21:16:44.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.917+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254003630 osd=2 3 2024-02-19T21:16:44.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.917+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254003630 osd=2 3 2024-02-19T21:16:44.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.917+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 6 2024-02-19T21:16:44.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.917+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa254003630 osd=2 3 2024-02-19T21:16:44.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.917+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 2 6 2024-02-19T21:16:44.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.917+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254003630 osd=2 4 2024-02-19T21:16:44.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.917+0000 7fa26e37b700 10 client.50555.objecter _send_command 6 2024-02-19T21:16:44.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.917+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa2540030a0 con 0x7fa254011740 2024-02-19T21:16:44.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa2540030a0 con 0x7fa254011740 2024-02-19T21:16:44.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:44.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa2657fa700 10 client.50555.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:44.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa254003630 osd=2 4 2024-02-19T21:16:44.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 2 6 2024-02-19T21:16:44.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa26e37b700 10 client.50555.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:44.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 7 2024-02-19T21:16:44.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa2540034b0 osd=3 3 2024-02-19T21:16:44.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa2540034b0 osd=3 3 2024-02-19T21:16:44.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa2540034b0 osd=3 3 2024-02-19T21:16:44.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 7 2024-02-19T21:16:44.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa2540034b0 osd=3 3 2024-02-19T21:16:44.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 3 7 2024-02-19T21:16:44.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa2540034b0 osd=3 4 2024-02-19T21:16:44.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa26e37b700 10 client.50555.objecter _send_command 7 2024-02-19T21:16:44.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.918+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa254019ea0 con 0x7fa2540162a0 2024-02-19T21:16:44.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.919+0000 7fa2657fa700 10 client.50555.objecter ms_handle_connect 0x7fa2540162a0 2024-02-19T21:16:44.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.919+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa254019ea0 con 0x7fa2540162a0 2024-02-19T21:16:44.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.919+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:16:44.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.919+0000 7fa2657fa700 10 client.50555.objecter _finish_command 7 = system:0 2024-02-19T21:16:44.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.919+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa2540034b0 osd=3 4 2024-02-19T21:16:44.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.919+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 3 7 2024-02-19T21:16:44.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.934+0000 7fa26e37b700 10 client.50555.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:44.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.934+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 8 2024-02-19T21:16:44.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.934+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa2540034b0 osd=3 3 2024-02-19T21:16:44.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.934+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa2540034b0 osd=3 3 2024-02-19T21:16:44.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.934+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa2540034b0 osd=3 3 2024-02-19T21:16:44.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.934+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 8 2024-02-19T21:16:44.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.934+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa2540034b0 osd=3 3 2024-02-19T21:16:44.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.934+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 3 8 2024-02-19T21:16:44.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.934+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa2540034b0 osd=3 4 2024-02-19T21:16:44.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.934+0000 7fa26e37b700 10 client.50555.objecter _send_command 8 2024-02-19T21:16:44.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.934+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa254012650 con 0x7fa2540162a0 2024-02-19T21:16:44.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa254012650 con 0x7fa2540162a0 2024-02-19T21:16:44.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:44.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa2657fa700 10 client.50555.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:44.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa2540034b0 osd=3 4 2024-02-19T21:16:44.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 3 8 2024-02-19T21:16:44.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa26e37b700 10 client.50555.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:44.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 9 2024-02-19T21:16:44.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254012410 osd=4 3 2024-02-19T21:16:44.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254012410 osd=4 3 2024-02-19T21:16:44.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254012410 osd=4 3 2024-02-19T21:16:44.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 9 2024-02-19T21:16:44.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa254012410 osd=4 3 2024-02-19T21:16:44.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 4 9 2024-02-19T21:16:44.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254012410 osd=4 4 2024-02-19T21:16:44.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa26e37b700 10 client.50555.objecter _send_command 9 2024-02-19T21:16:44.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.935+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2540125c0 con 0x7fa2540112a0 2024-02-19T21:16:44.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.936+0000 7fa2657fa700 10 client.50555.objecter ms_handle_connect 0x7fa2540112a0 2024-02-19T21:16:44.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.937+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa2540125c0 con 0x7fa2540112a0 2024-02-19T21:16:44.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.937+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:16:44.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.937+0000 7fa2657fa700 10 client.50555.objecter _finish_command 9 = system:0 2024-02-19T21:16:44.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.937+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa254012410 osd=4 4 2024-02-19T21:16:44.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.937+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 4 9 2024-02-19T21:16:44.960 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:44.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.953+0000 7fa26e37b700 10 client.50555.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:44.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.953+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 10 2024-02-19T21:16:44.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.953+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254012410 osd=4 3 2024-02-19T21:16:44.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.953+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254012410 osd=4 3 2024-02-19T21:16:44.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.953+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254012410 osd=4 3 2024-02-19T21:16:44.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.953+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 10 2024-02-19T21:16:44.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.953+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa254012410 osd=4 3 2024-02-19T21:16:44.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.953+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 4 10 2024-02-19T21:16:44.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.953+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254012410 osd=4 4 2024-02-19T21:16:44.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.953+0000 7fa26e37b700 10 client.50555.objecter _send_command 10 2024-02-19T21:16:44.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.953+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa2540125c0 con 0x7fa2540112a0 2024-02-19T21:16:44.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.954+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa2540125c0 con 0x7fa2540112a0 2024-02-19T21:16:44.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.954+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:44.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.954+0000 7fa2657fa700 10 client.50555.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:44.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.954+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa254012410 osd=4 4 2024-02-19T21:16:44.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.954+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 4 10 2024-02-19T21:16:44.962 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:16:44.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.955+0000 7fa26e37b700 10 client.50555.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:44.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.955+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 11 2024-02-19T21:16:44.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.955+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa2540125c0 osd=5 3 2024-02-19T21:16:44.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.955+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa2540125c0 osd=5 3 2024-02-19T21:16:44.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.955+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa2540125c0 osd=5 3 2024-02-19T21:16:44.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.955+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 11 2024-02-19T21:16:44.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.955+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa2540125c0 osd=5 3 2024-02-19T21:16:44.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.955+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 5 11 2024-02-19T21:16:44.963 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.955+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa2540125c0 osd=5 4 2024-02-19T21:16:44.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.955+0000 7fa26e37b700 10 client.50555.objecter _send_command 11 2024-02-19T21:16:44.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.955+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa25401b290 con 0x7fa25401acb0 2024-02-19T21:16:44.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.955+0000 7fa2657fa700 10 client.50555.objecter ms_handle_connect 0x7fa25401acb0 2024-02-19T21:16:44.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.956+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa25401b290 con 0x7fa25401acb0 2024-02-19T21:16:44.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.956+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:16:44.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.956+0000 7fa2657fa700 10 client.50555.objecter _finish_command 11 = system:0 2024-02-19T21:16:44.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.956+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa2540125c0 osd=5 4 2024-02-19T21:16:44.964 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.956+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 5 11 2024-02-19T21:16:44.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.972+0000 7fa26e37b700 10 client.50555.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:44.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.972+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 12 2024-02-19T21:16:44.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.972+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa2540125c0 osd=5 3 2024-02-19T21:16:44.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.972+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa2540125c0 osd=5 3 2024-02-19T21:16:44.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.972+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa2540125c0 osd=5 3 2024-02-19T21:16:44.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.972+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 12 2024-02-19T21:16:44.973 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.972+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa2540125c0 osd=5 3 2024-02-19T21:16:44.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.972+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 5 12 2024-02-19T21:16:44.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.972+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa2540125c0 osd=5 4 2024-02-19T21:16:44.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.972+0000 7fa26e37b700 10 client.50555.objecter _send_command 12 2024-02-19T21:16:44.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.972+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa25401b290 con 0x7fa25401acb0 2024-02-19T21:16:44.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa25401b290 con 0x7fa25401acb0 2024-02-19T21:16:44.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:44.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa2657fa700 10 client.50555.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:44.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa2540125c0 osd=5 4 2024-02-19T21:16:44.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 5 12 2024-02-19T21:16:44.975 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:16:44.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa26e37b700 10 client.50555.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:44.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 13 2024-02-19T21:16:44.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa25401b290 osd=6 3 2024-02-19T21:16:44.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa25401b290 osd=6 3 2024-02-19T21:16:44.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa25401b290 osd=6 3 2024-02-19T21:16:44.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 13 2024-02-19T21:16:44.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa25401b290 osd=6 3 2024-02-19T21:16:44.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 6 13 2024-02-19T21:16:44.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa25401b290 osd=6 4 2024-02-19T21:16:44.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa26e37b700 10 client.50555.objecter _send_command 13 2024-02-19T21:16:44.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa254027080 con 0x7fa2540234a0 2024-02-19T21:16:44.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.974+0000 7fa2657fa700 10 client.50555.objecter ms_handle_connect 0x7fa2540234a0 2024-02-19T21:16:44.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.975+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa254027080 con 0x7fa2540234a0 2024-02-19T21:16:44.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.975+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:16:44.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.975+0000 7fa2657fa700 10 client.50555.objecter _finish_command 13 = system:0 2024-02-19T21:16:44.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.975+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa25401b290 osd=6 4 2024-02-19T21:16:44.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.975+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 6 13 2024-02-19T21:16:44.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.990+0000 7fa26e37b700 10 client.50555.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:44.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.990+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 14 2024-02-19T21:16:44.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.990+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa25401b290 osd=6 3 2024-02-19T21:16:44.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.991+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa25401b290 osd=6 3 2024-02-19T21:16:44.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.991+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa25401b290 osd=6 3 2024-02-19T21:16:44.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.991+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 14 2024-02-19T21:16:44.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.991+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa25401b290 osd=6 3 2024-02-19T21:16:44.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.991+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 6 14 2024-02-19T21:16:44.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.991+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa25401b290 osd=6 4 2024-02-19T21:16:44.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.991+0000 7fa26e37b700 10 client.50555.objecter _send_command 14 2024-02-19T21:16:44.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.991+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa254027080 con 0x7fa2540234a0 2024-02-19T21:16:44.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa254027080 con 0x7fa2540234a0 2024-02-19T21:16:44.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:44.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa2657fa700 10 client.50555.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:44.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa25401b290 osd=6 4 2024-02-19T21:16:44.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 6 14 2024-02-19T21:16:44.994 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:16:44.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa26e37b700 10 client.50555.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:44.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 15 2024-02-19T21:16:44.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254027080 osd=7 3 2024-02-19T21:16:44.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254027080 osd=7 3 2024-02-19T21:16:44.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254027080 osd=7 3 2024-02-19T21:16:44.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 15 2024-02-19T21:16:44.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa254027080 osd=7 3 2024-02-19T21:16:44.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 7 15 2024-02-19T21:16:44.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254027080 osd=7 4 2024-02-19T21:16:44.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa26e37b700 10 client.50555.objecter _send_command 15 2024-02-19T21:16:44.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.992+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2540282a0 con 0x7fa254028d40 2024-02-19T21:16:44.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.993+0000 7fa2657fa700 10 client.50555.objecter ms_handle_connect 0x7fa254028d40 2024-02-19T21:16:44.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.993+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fa2540282a0 con 0x7fa254028d40 2024-02-19T21:16:44.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.993+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:16:44.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.993+0000 7fa2657fa700 10 client.50555.objecter _finish_command 15 = system:0 2024-02-19T21:16:44.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.993+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa254027080 osd=7 4 2024-02-19T21:16:44.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:44.993+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 7 15 2024-02-19T21:16:45.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.008+0000 7fa26e37b700 10 client.50555.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:16:45.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.008+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign -1 16 2024-02-19T21:16:45.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.008+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254027080 osd=7 3 2024-02-19T21:16:45.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.008+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254027080 osd=7 3 2024-02-19T21:16:45.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.008+0000 7fa26e37b700 20 client.50555.objecter _get_session s=0x7fa254027080 osd=7 3 2024-02-19T21:16:45.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.008+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_remove -1 16 2024-02-19T21:16:45.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.008+0000 7fa26e37b700 20 client.50555.objecter get_session s=0x7fa254027080 osd=7 3 2024-02-19T21:16:45.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.008+0000 7fa26e37b700 15 client.50555.objecter _session_command_op_assign 7 16 2024-02-19T21:16:45.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.008+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254027080 osd=7 4 2024-02-19T21:16:45.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.008+0000 7fa26e37b700 10 client.50555.objecter _send_command 16 2024-02-19T21:16:45.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.008+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7fa2540282a0 con 0x7fa254028d40 2024-02-19T21:16:45.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.010+0000 7fa2657fa700 1 -- 172.21.15.163:0/2322208285 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7fa2540282a0 con 0x7fa254028d40 2024-02-19T21:16:45.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.010+0000 7fa2657fa700 10 client.50555.objecter ms_dispatch 0x7fa268001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:16:45.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.010+0000 7fa2657fa700 10 client.50555.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:16:45.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.010+0000 7fa2657fa700 20 client.50555.objecter put_session s=0x7fa254027080 osd=7 4 2024-02-19T21:16:45.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.010+0000 7fa2657fa700 15 client.50555.objecter _session_command_op_remove 7 16 2024-02-19T21:16:45.012 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:16:45.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.010+0000 7fa26e37b700 10 librados: watch_flush enter 2024-02-19T21:16:45.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.010+0000 7fa26e37b700 10 librados: watch_flush exit 2024-02-19T21:16:45.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.010+0000 7fa26e37b700 10 client.50555.objecter close_session for osd.0 2024-02-19T21:16:45.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.010+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 >> v1:172.21.15.163:6801/1559418115 conn(0x7fa254004530 legacy=0x7fa254002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:45.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.010+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa2540051a0 osd=0 1 2024-02-19T21:16:45.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.010+0000 7fa26e37b700 10 client.50555.objecter close_session for osd.1 2024-02-19T21:16:45.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.010+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 >> v1:172.21.15.163:6805/3125047432 conn(0x7fa254004150 legacy=0x7fa25400a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:45.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254003fe0 osd=1 1 2024-02-19T21:16:45.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 10 client.50555.objecter close_session for osd.2 2024-02-19T21:16:45.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 >> v1:172.21.15.163:6809/1563158487 conn(0x7fa254011740 legacy=0x7fa254011b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:45.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254003630 osd=2 1 2024-02-19T21:16:45.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 10 client.50555.objecter close_session for osd.3 2024-02-19T21:16:45.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 >> v1:172.21.15.163:6813/2112279769 conn(0x7fa2540162a0 legacy=0x7fa254018690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:45.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26c918700 1 -- 172.21.15.163:0/2322208285 reap_dead start 2024-02-19T21:16:45.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa2540034b0 osd=3 1 2024-02-19T21:16:45.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 10 client.50555.objecter close_session for osd.4 2024-02-19T21:16:45.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 >> v1:172.21.15.193:6800/3129716965 conn(0x7fa2540112a0 legacy=0x7fa25401a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:45.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26c918700 1 -- 172.21.15.163:0/2322208285 reap_dead start 2024-02-19T21:16:45.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254012410 osd=4 1 2024-02-19T21:16:45.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 10 client.50555.objecter close_session for osd.5 2024-02-19T21:16:45.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 >> v1:172.21.15.193:6804/805161991 conn(0x7fa25401acb0 legacy=0x7fa254021870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:45.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa2540125c0 osd=5 1 2024-02-19T21:16:45.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 10 client.50555.objecter close_session for osd.6 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 >> v1:172.21.15.193:6808/787604615 conn(0x7fa2540234a0 legacy=0x7fa254025890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa25401b290 osd=6 1 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 10 client.50555.objecter close_session for osd.7 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 >> v1:172.21.15.193:6812/157927432 conn(0x7fa254028d40 legacy=0x7fa2540291c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 20 client.50555.objecter put_session s=0x7fa254027080 osd=7 1 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 20 client.50555.objecter shutdown clearing up homeless session... 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 10 client.50555.objecter successfully canceled tick 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 >> v1:172.21.15.163:6800/1552514681 conn(0x7fa250063d00 legacy=0x7fa2500661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 >> v1:172.21.15.163:6790/0 conn(0x7fa268100780 legacy=0x7fa2680fa110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26c918700 1 -- 172.21.15.163:0/2322208285 reap_dead start 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 shutdown_connections 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 >> 172.21.15.163:0/2322208285 conn(0x7fa2680f35a0 msgr2=0x7fa268107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 shutdown_connections 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 1 -- 172.21.15.163:0/2322208285 wait complete. 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:45.011+0000 7fa26e37b700 1 librados: shutdown 2024-02-19T21:16:45.015 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:16:45.016 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:16:45.016 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:16:45.016 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:16:45.016 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:16:45.016 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:16:45.016 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:16:45.016 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:16:45.655 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:45.657 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:45.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:45.758 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:45.789 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:45.988 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:45.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:44.867855+0000 mon.c (mon.2) 434 : audit [DBG] from='client.? v1:172.21.15.163:0/2322208285' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:45.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:45.407136+0000 mon.a (mon.0) 3571 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-19T21:16:45.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:45.407181+0000 mon.a (mon.0) 3572 : cluster [DBG] osdmap e749: 8 total, 8 up, 7 in 2024-02-19T21:16:45.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:45.408081+0000 mon.a (mon.0) 3573 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-19T21:16:45.990 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:44.867855+0000 mon.c (mon.2) 434 : audit [DBG] from='client.? v1:172.21.15.163:0/2322208285' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:45.990 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:45.407136+0000 mon.a (mon.0) 3571 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-19T21:16:45.990 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:45.407181+0000 mon.a (mon.0) 3572 : cluster [DBG] osdmap e749: 8 total, 8 up, 7 in 2024-02-19T21:16:45.990 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:45.408081+0000 mon.a (mon.0) 3573 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-19T21:16:45.991 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:45.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:46.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:44.867855+0000 mon.c (mon.2) 434 : audit [DBG] from='client.? v1:172.21.15.163:0/2322208285' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:46.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:45.407136+0000 mon.a (mon.0) 3571 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-19T21:16:46.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:45.407181+0000 mon.a (mon.0) 3572 : cluster [DBG] osdmap e749: 8 total, 8 up, 7 in 2024-02-19T21:16:46.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:45.408081+0000 mon.a (mon.0) 3573 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-02-19T21:16:46.092 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:46.120 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:46.420 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:16:46.569 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:46.572 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:46.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:46.673 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:46.707 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:46.730 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:46.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:46.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:46.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:45.326658+0000 mgr.y (mgr.14550) 627 : cluster [DBG] pgmap v1132: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:46.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:46.407968+0000 mon.a (mon.0) 3574 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-19T21:16:46.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:46.408037+0000 mon.a (mon.0) 3575 : cluster [DBG] osdmap e750: 8 total, 8 up, 7 in 2024-02-19T21:16:46.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:46.408527+0000 mon.a (mon.0) 3576 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-19T21:16:46.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:45.326658+0000 mgr.y (mgr.14550) 627 : cluster [DBG] pgmap v1132: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:46.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:46.407968+0000 mon.a (mon.0) 3574 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-19T21:16:46.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:46.408037+0000 mon.a (mon.0) 3575 : cluster [DBG] osdmap e750: 8 total, 8 up, 7 in 2024-02-19T21:16:46.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:46.408527+0000 mon.a (mon.0) 3576 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-19T21:16:46.833 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:46.866 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:46.991 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:46.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:47.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:45.326658+0000 mgr.y (mgr.14550) 627 : cluster [DBG] pgmap v1132: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:47.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:46.407968+0000 mon.a (mon.0) 3574 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-02-19T21:16:47.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:46.408037+0000 mon.a (mon.0) 3575 : cluster [DBG] osdmap e750: 8 total, 8 up, 7 in 2024-02-19T21:16:47.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:46.408527+0000 mon.a (mon.0) 3576 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-02-19T21:16:47.095 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:47.122 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:47.748 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:16:47.748 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:16:47.771 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:47.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:47.876 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:47.904 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:47.977 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:47.978 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:47.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:48.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:16:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:16:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:16:48.081 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:48.113 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:48.134 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:48.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:48.238 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:48.266 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:48.305 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:48.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:48.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:48.414 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:48.440 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:48.691 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:48.692 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:47.327243+0000 mgr.y (mgr.14550) 628 : cluster [DBG] pgmap v1135: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:48.692 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:47.408000+0000 mon.a (mon.0) 3577 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-19T21:16:48.692 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:47.408066+0000 mon.a (mon.0) 3578 : cluster [DBG] osdmap e751: 8 total, 8 up, 7 in 2024-02-19T21:16:48.692 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:47.444115+0000 mon.a (mon.0) 3579 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:16:48.692 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:48.139445+0000 mon.b (mon.1) 751 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:16:48.692 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:47.327243+0000 mgr.y (mgr.14550) 628 : cluster [DBG] pgmap v1135: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:48.693 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:47.408000+0000 mon.a (mon.0) 3577 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-19T21:16:48.693 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:47.408066+0000 mon.a (mon.0) 3578 : cluster [DBG] osdmap e751: 8 total, 8 up, 7 in 2024-02-19T21:16:48.693 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:47.444115+0000 mon.a (mon.0) 3579 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:16:48.693 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:48.139445+0000 mon.b (mon.1) 751 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:16:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:48.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:48.712 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-19T21:16:48.796 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:48.821 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:47.327243+0000 mgr.y (mgr.14550) 628 : cluster [DBG] pgmap v1135: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:16:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:47.408000+0000 mon.a (mon.0) 3577 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-132","var": "hit_set_search_last_n","val": "1"}]': finished 2024-02-19T21:16:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:47.408066+0000 mon.a (mon.0) 3578 : cluster [DBG] osdmap e751: 8 total, 8 up, 7 in 2024-02-19T21:16:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:47.444115+0000 mon.a (mon.0) 3579 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:16:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:48.139445+0000 mon.b (mon.1) 751 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:16:48.825 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:48.847 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:48.852 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:48.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:48.953 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:48.981 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:49.038 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:49.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:49.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:49.142 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:49.168 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:49.366 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:49.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:49.431 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:16:49.469 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:49.505 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:49.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:48.409604+0000 mon.a (mon.0) 3580 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:16:49.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:48.409664+0000 mon.a (mon.0) 3581 : cluster [DBG] osdmap e752: 8 total, 8 up, 7 in 2024-02-19T21:16:49.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:48.411238+0000 mon.a (mon.0) 3582 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-132"}]: dispatch 2024-02-19T21:16:49.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:48.922844+0000 mon.a (mon.0) 3583 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:49.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:48.409604+0000 mon.a (mon.0) 3580 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:16:49.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:48.409664+0000 mon.a (mon.0) 3581 : cluster [DBG] osdmap e752: 8 total, 8 up, 7 in 2024-02-19T21:16:49.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:48.411238+0000 mon.a (mon.0) 3582 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-132"}]: dispatch 2024-02-19T21:16:49.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:48.922844+0000 mon.a (mon.0) 3583 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:49.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:48.409604+0000 mon.a (mon.0) 3580 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:16:49.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:48.409664+0000 mon.a (mon.0) 3581 : cluster [DBG] osdmap e752: 8 total, 8 up, 7 in 2024-02-19T21:16:49.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:48.411238+0000 mon.a (mon.0) 3582 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-132"}]: dispatch 2024-02-19T21:16:49.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:48.922844+0000 mon.a (mon.0) 3583 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:49.955 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:49.958 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:49.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:50.059 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:50.093 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:50.115 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:50.118 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:50.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:50.219 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:50.247 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:50.267 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:50.270 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:50.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:50.371 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:50.401 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:50.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.431+0000 7f53d3958700 1 -- 172.21.15.163:0/4090610713 >> v1:172.21.15.163:6790/0 conn(0x7f53cc1031d0 legacy=0x7f53cc105650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:50.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.431+0000 7f53d3958700 1 -- 172.21.15.163:0/4090610713 shutdown_connections 2024-02-19T21:16:50.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.431+0000 7f53d3958700 1 -- 172.21.15.163:0/4090610713 >> 172.21.15.163:0/4090610713 conn(0x7f53cc0f2470 msgr2=0x7f53cc0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:50.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.431+0000 7f53d3958700 1 -- 172.21.15.163:0/4090610713 shutdown_connections 2024-02-19T21:16:50.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.431+0000 7f53d3958700 1 -- 172.21.15.163:0/4090610713 wait complete. 2024-02-19T21:16:50.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.431+0000 7f53d3958700 1 librados: starting msgr at 2024-02-19T21:16:50.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.431+0000 7f53d3958700 1 librados: starting objecter 2024-02-19T21:16:50.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53d3958700 1 Processor -- start 2024-02-19T21:16:50.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53d3958700 1 -- start start 2024-02-19T21:16:50.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53d3958700 1 librados: setting wanted keys 2024-02-19T21:16:50.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53d3958700 1 librados: calling monclient init 2024-02-19T21:16:50.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53d3958700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f53cc119590 con 0x7f53cc0fc890 2024-02-19T21:16:50.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53d3958700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f53cc11a770 con 0x7f53cc1031d0 2024-02-19T21:16:50.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53d3958700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f53cc11b970 con 0x7f53cc0ff690 2024-02-19T21:16:50.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53d16f4700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f53cc0fc890 0x7f53cc060b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:34716/0 (socket says 172.21.15.163:34716) 2024-02-19T21:16:50.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53d16f4700 1 -- 172.21.15.163:0/1034190203 learned_addr learned my addr 172.21.15.163:0/1034190203 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:50.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53ca7fc700 10 client.?.objecter ms_handle_connect 0x7f53cc0fc890 2024-02-19T21:16:50.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53ca7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:50.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53ca7fc700 10 client.?.objecter ms_handle_connect 0x7f53cc0ff690 2024-02-19T21:16:50.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53ca7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:50.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53ca7fc700 10 client.?.objecter ms_handle_connect 0x7f53cc1031d0 2024-02-19T21:16:50.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53ca7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:50.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729607747 0 0) 0x7f53cc119590 con 0x7f53cc0fc890 2024-02-19T21:16:50.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b40033f0 con 0x7f53cc0fc890 2024-02-19T21:16:50.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991972599 0 0) 0x7f53cc11a770 con 0x7f53cc1031d0 2024-02-19T21:16:50.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53cc119590 con 0x7f53cc1031d0 2024-02-19T21:16:50.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384134425 0 0) 0x7f53cc11b970 con 0x7f53cc0ff690 2024-02-19T21:16:50.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.432+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53cc11a770 con 0x7f53cc0ff690 2024-02-19T21:16:50.437 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:50.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1813479239 0 0) 0x7f53b40033f0 con 0x7f53cc0fc890 2024-02-19T21:16:50.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53cc11b970 con 0x7f53cc0fc890 2024-02-19T21:16:50.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f53bc001d10 con 0x7f53cc0fc890 2024-02-19T21:16:50.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4265574702 0 0) 0x7f53cc119590 con 0x7f53cc1031d0 2024-02-19T21:16:50.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53b40033f0 con 0x7f53cc1031d0 2024-02-19T21:16:50.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f53c4001fa0 con 0x7f53cc1031d0 2024-02-19T21:16:50.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1920288177 0 0) 0x7f53cc11a770 con 0x7f53cc0ff690 2024-02-19T21:16:50.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53cc119590 con 0x7f53cc0ff690 2024-02-19T21:16:50.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f53c00018c0 con 0x7f53cc0ff690 2024-02-19T21:16:50.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1591101793 0 0) 0x7f53cc11b970 con 0x7f53cc0fc890 2024-02-19T21:16:50.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 >> v1:172.21.15.163:6790/0 conn(0x7f53cc0ff690 legacy=0x7f53cc061230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:50.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 >> v1:172.21.15.193:6789/0 conn(0x7f53cc1031d0 legacy=0x7f53cc117c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:50.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53cc11cb70 con 0x7f53cc0fc890 2024-02-19T21:16:50.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53d3958700 1 -- 172.21.15.163:0/1034190203 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53cc11bb60 con 0x7f53cc0fc890 2024-02-19T21:16:50.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53d3958700 10 client.23832.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:50.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53d3958700 1 -- 172.21.15.163:0/1034190203 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53cc11bfd0 con 0x7f53cc0fc890 2024-02-19T21:16:50.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53d3958700 1 librados: init done 2024-02-19T21:16:50.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f53bc0026c0 con 0x7f53cc0fc890 2024-02-19T21:16:50.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.433+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f53bc005e20 con 0x7f53cc0fc890 2024-02-19T21:16:50.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.434+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f53bc018cb0 con 0x7f53cc0fc890 2024-02-19T21:16:50.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.435+0000 7f53d3958700 1 -- 172.21.15.163:0/1034190203 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b8004f20 con 0x7f53cc0fc890 2024-02-19T21:16:50.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.435+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(753..753 src has 623..753) v4 ==== 8467+0+0 (unknown 2991047663 0 0) 0x7f53bc079210 con 0x7f53cc0fc890 2024-02-19T21:16:50.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.435+0000 7f53ca7fc700 10 client.23832.objecter ms_dispatch 0x7f53cc001550 osd_map(753..753 src has 623..753) v4 2024-02-19T21:16:50.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.435+0000 7f53ca7fc700 3 client.23832.objecter handle_osd_map got epochs [753,753] > 0 2024-02-19T21:16:50.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.435+0000 7f53ca7fc700 3 client.23832.objecter handle_osd_map decoding full epoch 753 2024-02-19T21:16:50.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.435+0000 7f53ca7fc700 20 client.23832.objecter dump_active .. 0 homeless 2024-02-19T21:16:50.441 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:50.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:50.441 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.435+0000 7f53ca7fc700 10 client.23832.objecter ms_handle_connect 0x7f53b4063cb0 2024-02-19T21:16:50.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.442+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f53bc04c470 con 0x7f53cc0fc890 2024-02-19T21:16:50.446 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:50.498 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:49.327830+0000 mgr.y (mgr.14550) 629 : cluster [DBG] pgmap v1138: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:16:50.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:49.417593+0000 mon.a (mon.0) 3584 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-132"}]': finished 2024-02-19T21:16:50.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:49.417646+0000 mon.a (mon.0) 3585 : cluster [DBG] osdmap e753: 8 total, 8 up, 7 in 2024-02-19T21:16:50.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:49.462231+0000 mon.b (mon.1) 752 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:16:50.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:49.462999+0000 mon.b (mon.1) 753 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:16:50.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:49.493001+0000 mon.a (mon.0) 3586 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:50.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:49.500637+0000 mon.a (mon.0) 3587 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:16:50.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:49.500943+0000 mon.a (mon.0) 3588 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-132"}]: dispatch 2024-02-19T21:16:50.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:49.940978+0000 mon.a (mon.0) 3589 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:50.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:49.327830+0000 mgr.y (mgr.14550) 629 : cluster [DBG] pgmap v1138: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:16:50.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:49.417593+0000 mon.a (mon.0) 3584 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-132"}]': finished 2024-02-19T21:16:50.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:49.417646+0000 mon.a (mon.0) 3585 : cluster [DBG] osdmap e753: 8 total, 8 up, 7 in 2024-02-19T21:16:50.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:49.462231+0000 mon.b (mon.1) 752 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:16:50.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:49.462999+0000 mon.b (mon.1) 753 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:16:50.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:49.493001+0000 mon.a (mon.0) 3586 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:50.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:49.500637+0000 mon.a (mon.0) 3587 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:16:50.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:49.500943+0000 mon.a (mon.0) 3588 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-132"}]: dispatch 2024-02-19T21:16:50.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:49.940978+0000 mon.a (mon.0) 3589 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:50.542 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:50.572 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:50.594 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:50.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:50.629 INFO:tasks.thrashosds.thrasher:in_osds: [1, 5, 7, 3, 4, 6, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:16:50.630 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:16:50.630 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [1, 5, 7, 3, 4, 6, 0] 2024-02-19T21:16:50.630 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd out 5 2024-02-19T21:16:50.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:50.641+0000 7f53d3958700 1 -- 172.21.15.163:0/1034190203 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f53b80051d0 con 0x7f53cc0fc890 2024-02-19T21:16:50.698 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:50.724 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:49.327830+0000 mgr.y (mgr.14550) 629 : cluster [DBG] pgmap v1138: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-02-19T21:16:50.724 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:49.417593+0000 mon.a (mon.0) 3584 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-132"}]': finished 2024-02-19T21:16:50.724 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:49.417646+0000 mon.a (mon.0) 3585 : cluster [DBG] osdmap e753: 8 total, 8 up, 7 in 2024-02-19T21:16:50.724 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:49.462231+0000 mon.b (mon.1) 752 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:16:50.724 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:49.462999+0000 mon.b (mon.1) 753 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:16:50.724 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:49.493001+0000 mon.a (mon.0) 3586 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:50.724 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:49.500637+0000 mon.a (mon.0) 3587 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:16:50.724 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:49.500943+0000 mon.a (mon.0) 3588 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-132"}]: dispatch 2024-02-19T21:16:50.724 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:49.940978+0000 mon.a (mon.0) 3589 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:50.726 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:50.745 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:50.747 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:50.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:50.848 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:16:50.875 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:51.072 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:16:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:16:51.323 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:16:51.325 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:51.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:16:51.426 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:51.459 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:51.479 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:51.482 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:51.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:51.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:51.503+0000 7f53ca7fc700 1 -- 172.21.15.163:0/1034190203 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v755) v1 ==== 91+0+0 (unknown 2694485920 0 0) 0x7f53bc04c210 con 0x7f53cc0fc890 2024-02-19T21:16:51.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:51.506+0000 7f53d3958700 10 librados: watch_flush enter 2024-02-19T21:16:51.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:51.507+0000 7f53d3958700 10 librados: watch_flush exit 2024-02-19T21:16:51.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:51.507+0000 7f53d3958700 20 client.23832.objecter shutdown clearing up homeless session... 2024-02-19T21:16:51.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:51.507+0000 7f53d3958700 10 client.23832.objecter successfully canceled tick 2024-02-19T21:16:51.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:51.508+0000 7f53d3958700 1 -- 172.21.15.163:0/1034190203 >> v1:172.21.15.163:6800/1552514681 conn(0x7f53b4063cb0 legacy=0x7f53b4066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:51.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:51.508+0000 7f53d3958700 1 -- 172.21.15.163:0/1034190203 >> v1:172.21.15.163:6789/0 conn(0x7f53cc0fc890 legacy=0x7f53cc060b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:51.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:51.508+0000 7f53d3958700 1 -- 172.21.15.163:0/1034190203 shutdown_connections 2024-02-19T21:16:51.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:51.508+0000 7f53d3958700 1 -- 172.21.15.163:0/1034190203 >> 172.21.15.163:0/1034190203 conn(0x7f53cc0f2470 msgr2=0x7f53cc106640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:51.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:51.508+0000 7f53d3958700 1 -- 172.21.15.163:0/1034190203 shutdown_connections 2024-02-19T21:16:51.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:51.508+0000 7f53d3958700 1 -- 172.21.15.163:0/1034190203 wait complete. 2024-02-19T21:16:51.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:51.508+0000 7f53d3958700 1 librados: shutdown 2024-02-19T21:16:51.510 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:16:51.583 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:51.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:50.351968+0000 osd.2 (osd.2) 302 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:16:51.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:50.352154+0000 osd.2 (osd.2) 303 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:16:51.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:50.494833+0000 mon.a (mon.0) 3590 : cluster [DBG] osdmap e754: 8 total, 8 up, 7 in 2024-02-19T21:16:51.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:50.747937+0000 mon.a (mon.0) 3591 : audit [INF] from='client.? v1:172.21.15.163:0/1034190203' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:16:51.604 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:50.776734+0000 mgr.y (mgr.14550) 630 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:51.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:50.351968+0000 osd.2 (osd.2) 302 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:16:51.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:50.352154+0000 osd.2 (osd.2) 303 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:16:51.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:50.494833+0000 mon.a (mon.0) 3590 : cluster [DBG] osdmap e754: 8 total, 8 up, 7 in 2024-02-19T21:16:51.604 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:50.747937+0000 mon.a (mon.0) 3591 : audit [INF] from='client.? v1:172.21.15.163:0/1034190203' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:16:51.605 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:50.776734+0000 mgr.y (mgr.14550) 630 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:51.613 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:51.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:50.351968+0000 osd.2 (osd.2) 302 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:16:51.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:50.352154+0000 osd.2 (osd.2) 303 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:16:51.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:50.494833+0000 mon.a (mon.0) 3590 : cluster [DBG] osdmap e754: 8 total, 8 up, 7 in 2024-02-19T21:16:51.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:50.747937+0000 mon.a (mon.0) 3591 : audit [INF] from='client.? v1:172.21.15.163:0/1034190203' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:16:51.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:50.776734+0000 mgr.y (mgr.14550) 630 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:16:52.161 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:52.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:52.265 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:52.297 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:52.318 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:52.321 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:52.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:52.422 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:52.452 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:52.473 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:52.475 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:52.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:52.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:51.328289+0000 mgr.y (mgr.14550) 631 : cluster [DBG] pgmap v1141: 182 pgs: 182 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:16:52.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:51.492483+0000 mon.a (mon.0) 3592 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:16:52.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:51.501920+0000 mon.a (mon.0) 3593 : audit [INF] from='client.? v1:172.21.15.163:0/1034190203' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:16:52.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:51.501978+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e755: 8 total, 8 up, 7 in 2024-02-19T21:16:52.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:51.503652+0000 mon.a (mon.0) 3595 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:16:52.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:51.653147+0000 osd.6 (osd.6) 263 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:16:52.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:51.653981+0000 osd.6 (osd.6) 264 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:16:52.577 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:52.604 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:52.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:51.328289+0000 mgr.y (mgr.14550) 631 : cluster [DBG] pgmap v1141: 182 pgs: 182 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:16:52.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:51.492483+0000 mon.a (mon.0) 3592 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:16:52.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:51.501920+0000 mon.a (mon.0) 3593 : audit [INF] from='client.? v1:172.21.15.163:0/1034190203' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:16:52.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:51.501978+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e755: 8 total, 8 up, 7 in 2024-02-19T21:16:52.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:51.503652+0000 mon.a (mon.0) 3595 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:16:52.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:51.653147+0000 osd.6 (osd.6) 263 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:16:52.993 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:51.653981+0000 osd.6 (osd.6) 264 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:16:52.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:51.328289+0000 mgr.y (mgr.14550) 631 : cluster [DBG] pgmap v1141: 182 pgs: 182 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:16:52.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:51.492483+0000 mon.a (mon.0) 3592 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:16:52.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:51.501920+0000 mon.a (mon.0) 3593 : audit [INF] from='client.? v1:172.21.15.163:0/1034190203' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:16:52.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:51.501978+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e755: 8 total, 8 up, 7 in 2024-02-19T21:16:52.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:51.503652+0000 mon.a (mon.0) 3595 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:16:52.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:51.653147+0000 osd.6 (osd.6) 263 : cluster [DBG] 2.14 scrub starts 2024-02-19T21:16:52.994 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:51.653981+0000 osd.6 (osd.6) 264 : cluster [DBG] 2.14 scrub ok 2024-02-19T21:16:53.256 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:53.258 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:53.258 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:53.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:53.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:53.361 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:53.393 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:53.414 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:53.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:53.476 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:16:53.476 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:16:53.498 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:16:53] "GET /metrics HTTP/1.1" 200 192486 "" "Prometheus/2.33.4" 2024-02-19T21:16:53.518 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:16:53.547 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:53.568 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:16:53.570 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:53.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:16:53.671 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:53.701 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:53.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:52.268132+0000 osd.5 (osd.5) 310 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:16:53.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:52.271639+0000 osd.5 (osd.5) 311 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:16:53.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:52.503293+0000 mon.a (mon.0) 3596 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:16:53.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:52.503412+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e756: 8 total, 8 up, 7 in 2024-02-19T21:16:53.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:52.516180+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:16:53.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:53.265003+0000 osd.4 (osd.4) 267 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:16:53.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:53.265926+0000 osd.4 (osd.4) 268 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:16:53.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:53.503594+0000 mon.a (mon.0) 3599 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-134", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:16:53.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:53.503656+0000 mon.a (mon.0) 3600 : cluster [DBG] osdmap e757: 8 total, 8 up, 7 in 2024-02-19T21:16:53.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:53.505605+0000 mon.a (mon.0) 3601 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-134"}]: dispatch 2024-02-19T21:16:53.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:52.268132+0000 osd.5 (osd.5) 310 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:16:53.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:52.271639+0000 osd.5 (osd.5) 311 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:16:53.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:52.503293+0000 mon.a (mon.0) 3596 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:16:53.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:52.503412+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e756: 8 total, 8 up, 7 in 2024-02-19T21:16:53.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:52.516180+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:16:53.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:53.265003+0000 osd.4 (osd.4) 267 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:16:53.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:53.265926+0000 osd.4 (osd.4) 268 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:16:53.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:53.503594+0000 mon.a (mon.0) 3599 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-134", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:16:53.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:53.503656+0000 mon.a (mon.0) 3600 : cluster [DBG] osdmap e757: 8 total, 8 up, 7 in 2024-02-19T21:16:53.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:53.505605+0000 mon.a (mon.0) 3601 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-134"}]: dispatch 2024-02-19T21:16:53.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:52.268132+0000 osd.5 (osd.5) 310 : cluster [DBG] 6.1c scrub starts 2024-02-19T21:16:54.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:52.271639+0000 osd.5 (osd.5) 311 : cluster [DBG] 6.1c scrub ok 2024-02-19T21:16:54.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:52.503293+0000 mon.a (mon.0) 3596 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:16:54.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:52.503412+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e756: 8 total, 8 up, 7 in 2024-02-19T21:16:54.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:52.516180+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:16:54.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:53.265003+0000 osd.4 (osd.4) 267 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:16:54.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:53.265926+0000 osd.4 (osd.4) 268 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:16:54.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:53.503594+0000 mon.a (mon.0) 3599 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-134", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:16:54.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:53.503656+0000 mon.a (mon.0) 3600 : cluster [DBG] osdmap e757: 8 total, 8 up, 7 in 2024-02-19T21:16:54.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:53.505605+0000 mon.a (mon.0) 3601 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-134"}]: dispatch 2024-02-19T21:16:54.182 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:16:54.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:53.203205+0000 osd.1 (osd.1) 330 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:16:54.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:53.204156+0000 osd.1 (osd.1) 331 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:16:54.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:53.328703+0000 mgr.y (mgr.14550) 632 : cluster [DBG] pgmap v1144: 214 pgs: 32 unknown, 182 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:16:54.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:54.407912+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-134"}]': finished 2024-02-19T21:16:54.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:54.407970+0000 mon.a (mon.0) 3603 : cluster [DBG] osdmap e758: 8 total, 8 up, 7 in 2024-02-19T21:16:54.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:54.409908+0000 mon.a (mon.0) 3604 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-134", "mode": "readproxy"}]: dispatch 2024-02-19T21:16:54.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:53.203205+0000 osd.1 (osd.1) 330 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:16:54.899 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:53.204156+0000 osd.1 (osd.1) 331 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:16:54.900 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:53.328703+0000 mgr.y (mgr.14550) 632 : cluster [DBG] pgmap v1144: 214 pgs: 32 unknown, 182 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:16:54.900 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:54.407912+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-134"}]': finished 2024-02-19T21:16:54.900 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:54.407970+0000 mon.a (mon.0) 3603 : cluster [DBG] osdmap e758: 8 total, 8 up, 7 in 2024-02-19T21:16:54.900 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:54.409908+0000 mon.a (mon.0) 3604 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-134", "mode": "readproxy"}]: dispatch 2024-02-19T21:16:54.900 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:53.203205+0000 osd.1 (osd.1) 330 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:16:54.900 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:53.204156+0000 osd.1 (osd.1) 331 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:16:54.900 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:53.328703+0000 mgr.y (mgr.14550) 632 : cluster [DBG] pgmap v1144: 214 pgs: 32 unknown, 182 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:16:54.900 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:54.407912+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-134"}]': finished 2024-02-19T21:16:54.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:54.407970+0000 mon.a (mon.0) 3603 : cluster [DBG] osdmap e758: 8 total, 8 up, 7 in 2024-02-19T21:16:54.901 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:54.409908+0000 mon.a (mon.0) 3604 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-134", "mode": "readproxy"}]: dispatch 2024-02-19T21:16:55.581 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-19T21:16:55.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:54.255377+0000 osd.1 (osd.1) 332 : cluster [DBG] 16.b scrub starts 2024-02-19T21:16:55.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:54.259438+0000 osd.1 (osd.1) 333 : cluster [DBG] 16.b scrub ok 2024-02-19T21:16:55.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:54.273203+0000 osd.5 (osd.5) 312 : cluster [DBG] 16.1f scrub starts 2024-02-19T21:16:55.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:54.277190+0000 osd.5 (osd.5) 313 : cluster [DBG] 16.1f scrub ok 2024-02-19T21:16:55.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:54.948865+0000 mon.a (mon.0) 3605 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:55.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:55.404639+0000 mon.a (mon.0) 3606 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:16:55.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:55.410486+0000 mon.a (mon.0) 3607 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-134", "mode": "readproxy"}]': finished 2024-02-19T21:16:55.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:55.410548+0000 mon.a (mon.0) 3608 : cluster [DBG] osdmap e759: 8 total, 8 up, 7 in 2024-02-19T21:16:55.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:54.255377+0000 osd.1 (osd.1) 332 : cluster [DBG] 16.b scrub starts 2024-02-19T21:16:55.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:54.259438+0000 osd.1 (osd.1) 333 : cluster [DBG] 16.b scrub ok 2024-02-19T21:16:55.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:54.273203+0000 osd.5 (osd.5) 312 : cluster [DBG] 16.1f scrub starts 2024-02-19T21:16:55.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:54.277190+0000 osd.5 (osd.5) 313 : cluster [DBG] 16.1f scrub ok 2024-02-19T21:16:55.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:54.948865+0000 mon.a (mon.0) 3605 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:55.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:55.404639+0000 mon.a (mon.0) 3606 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:16:55.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:55.410486+0000 mon.a (mon.0) 3607 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-134", "mode": "readproxy"}]': finished 2024-02-19T21:16:55.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:55.410548+0000 mon.a (mon.0) 3608 : cluster [DBG] osdmap e759: 8 total, 8 up, 7 in 2024-02-19T21:16:55.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:54.255377+0000 osd.1 (osd.1) 332 : cluster [DBG] 16.b scrub starts 2024-02-19T21:16:55.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:54.259438+0000 osd.1 (osd.1) 333 : cluster [DBG] 16.b scrub ok 2024-02-19T21:16:55.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:54.273203+0000 osd.5 (osd.5) 312 : cluster [DBG] 16.1f scrub starts 2024-02-19T21:16:55.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:54.277190+0000 osd.5 (osd.5) 313 : cluster [DBG] 16.1f scrub ok 2024-02-19T21:16:55.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:54.948865+0000 mon.a (mon.0) 3605 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:16:55.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:55.404639+0000 mon.a (mon.0) 3606 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:16:55.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:55.410486+0000 mon.a (mon.0) 3607 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-134", "mode": "readproxy"}]': finished 2024-02-19T21:16:56.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:55.410548+0000 mon.a (mon.0) 3608 : cluster [DBG] osdmap e759: 8 total, 8 up, 7 in 2024-02-19T21:16:56.553 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:56.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:56.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:56.557 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:16:56.658 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:16:56.689 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:56.712 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:16:56.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:56.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:16:56.779 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:55.259164+0000 osd.1 (osd.1) 334 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:16:56.780 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:55.262056+0000 osd.1 (osd.1) 335 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:16:56.780 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:55.329672+0000 mgr.y (mgr.14550) 633 : cluster [DBG] pgmap v1147: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:16:56.780 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:56.020002+0000 osd.3 (osd.3) 315 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:16:56.780 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:56.021406+0000 osd.3 (osd.3) 316 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:16:56.780 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:55.259164+0000 osd.1 (osd.1) 334 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:16:56.780 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:55.262056+0000 osd.1 (osd.1) 335 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:16:56.780 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:55.329672+0000 mgr.y (mgr.14550) 633 : cluster [DBG] pgmap v1147: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:16:56.780 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:56.020002+0000 osd.3 (osd.3) 315 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:16:56.780 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:56.021406+0000 osd.3 (osd.3) 316 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:16:56.816 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:55.259164+0000 osd.1 (osd.1) 334 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:16:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:55.262056+0000 osd.1 (osd.1) 335 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:16:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:55.329672+0000 mgr.y (mgr.14550) 633 : cluster [DBG] pgmap v1147: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-02-19T21:16:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:56.020002+0000 osd.3 (osd.3) 315 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:16:56.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:56.021406+0000 osd.3 (osd.3) 316 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:16:56.846 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:56.867 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:56.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:56.971 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:16:57.001 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:57.100 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:16:57.103 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:57.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:16:57.204 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:57.236 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:57.258 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:57.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:57.361 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:57.387 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:57.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:55.767484+0000 osd.6 (osd.6) 265 : cluster [DBG] 4.a scrub starts 2024-02-19T21:16:57.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:55.770035+0000 osd.6 (osd.6) 266 : cluster [DBG] 4.a scrub ok 2024-02-19T21:16:57.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:56.242444+0000 osd.5 (osd.5) 314 : cluster [DBG] 2.c scrub starts 2024-02-19T21:16:57.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:56.245415+0000 osd.5 (osd.5) 315 : cluster [DBG] 2.c scrub ok 2024-02-19T21:16:57.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:57.399490+0000 mon.c (mon.2) 435 : audit [INF] from='client.? v1:172.21.15.163:0/4070686890' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-19T21:16:57.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:57.399888+0000 mon.a (mon.0) 3609 : audit [INF] from='client.50567 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-19T21:16:57.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:57.400000+0000 mon.a (mon.0) 3610 : cluster [INF] Client client.admin marked osd.5 out, while it was still marked up 2024-02-19T21:16:57.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:16:57.398+0000 7fb606608700 -1 mon.a@0(leader).osd e759 definitely_dead 0 2024-02-19T21:16:57.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:55.767484+0000 osd.6 (osd.6) 265 : cluster [DBG] 4.a scrub starts 2024-02-19T21:16:57.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:55.770035+0000 osd.6 (osd.6) 266 : cluster [DBG] 4.a scrub ok 2024-02-19T21:16:57.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:56.242444+0000 osd.5 (osd.5) 314 : cluster [DBG] 2.c scrub starts 2024-02-19T21:16:57.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:56.245415+0000 osd.5 (osd.5) 315 : cluster [DBG] 2.c scrub ok 2024-02-19T21:16:57.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:57.399490+0000 mon.c (mon.2) 435 : audit [INF] from='client.? v1:172.21.15.163:0/4070686890' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-19T21:16:57.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:57.399888+0000 mon.a (mon.0) 3609 : audit [INF] from='client.50567 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-19T21:16:57.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:57.400000+0000 mon.a (mon.0) 3610 : cluster [INF] Client client.admin marked osd.5 out, while it was still marked up 2024-02-19T21:16:57.822 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:16:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:16:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:16:57.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:55.767484+0000 osd.6 (osd.6) 265 : cluster [DBG] 4.a scrub starts 2024-02-19T21:16:57.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:55.770035+0000 osd.6 (osd.6) 266 : cluster [DBG] 4.a scrub ok 2024-02-19T21:16:57.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:56.242444+0000 osd.5 (osd.5) 314 : cluster [DBG] 2.c scrub starts 2024-02-19T21:16:57.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:56.245415+0000 osd.5 (osd.5) 315 : cluster [DBG] 2.c scrub ok 2024-02-19T21:16:57.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:57.399490+0000 mon.c (mon.2) 435 : audit [INF] from='client.? v1:172.21.15.163:0/4070686890' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-19T21:16:57.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:57.399888+0000 mon.a (mon.0) 3609 : audit [INF] from='client.50567 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-19T21:16:57.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:57.400000+0000 mon.a (mon.0) 3610 : cluster [INF] Client client.admin marked osd.5 out, while it was still marked up 2024-02-19T21:16:57.842 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:16:57.845 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:57.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:16:57.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.206+0000 7f3720748700 1 -- 172.21.15.163:0/4152865550 >> v1:172.21.15.163:6789/0 conn(0x7f37180f9530 legacy=0x7f3718102a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:57.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.207+0000 7f3720748700 1 -- 172.21.15.163:0/4152865550 shutdown_connections 2024-02-19T21:16:57.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.207+0000 7f3720748700 1 -- 172.21.15.163:0/4152865550 >> 172.21.15.163:0/4152865550 conn(0x7f37180f2470 msgr2=0x7f37180f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:57.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.207+0000 7f3720748700 1 -- 172.21.15.163:0/4152865550 shutdown_connections 2024-02-19T21:16:57.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.207+0000 7f3720748700 1 -- 172.21.15.163:0/4152865550 wait complete. 2024-02-19T21:16:57.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.207+0000 7f3720748700 1 librados: starting msgr at 2024-02-19T21:16:57.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.207+0000 7f3720748700 1 librados: starting objecter 2024-02-19T21:16:57.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.207+0000 7f3720748700 1 Processor -- start 2024-02-19T21:16:57.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.207+0000 7f3720748700 1 -- start start 2024-02-19T21:16:57.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.207+0000 7f3720748700 1 librados: setting wanted keys 2024-02-19T21:16:57.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.207+0000 7f3720748700 1 librados: calling monclient init 2024-02-19T21:16:57.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.207+0000 7f3720748700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3718102b70 con 0x7f37180f8980 2024-02-19T21:16:57.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.207+0000 7f3720748700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3718102d40 con 0x7f37180f9530 2024-02-19T21:16:57.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.207+0000 7f3720748700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3718102f30 con 0x7f37181042b0 2024-02-19T21:16:57.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f371e4e4700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f37180f8980 0x7f37181023d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:47218/0 (socket says 172.21.15.163:47218) 2024-02-19T21:16:57.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f371e4e4700 1 -- 172.21.15.163:0/4070686890 learned_addr learned my addr 172.21.15.163:0/4070686890 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:57.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 10 client.?.objecter ms_handle_connect 0x7f37180f8980 2024-02-19T21:16:57.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:57.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 10 client.?.objecter ms_handle_connect 0x7f37181042b0 2024-02-19T21:16:57.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:57.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 10 client.?.objecter ms_handle_connect 0x7f37180f9530 2024-02-19T21:16:57.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:57.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243170135 0 0) 0x7f3718102b70 con 0x7f37180f8980 2024-02-19T21:16:57.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3704003800 con 0x7f37180f8980 2024-02-19T21:16:57.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721414981 0 0) 0x7f3718102f30 con 0x7f37181042b0 2024-02-19T21:16:57.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3718102b70 con 0x7f37181042b0 2024-02-19T21:16:57.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546478349 0 0) 0x7f3718102d40 con 0x7f37180f9530 2024-02-19T21:16:57.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3718102f30 con 0x7f37180f9530 2024-02-19T21:16:57.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1893671641 0 0) 0x7f3718102b70 con 0x7f37181042b0 2024-02-19T21:16:57.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3718102d40 con 0x7f37181042b0 2024-02-19T21:16:57.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f37100019a0 con 0x7f37181042b0 2024-02-19T21:16:57.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4271090701 0 0) 0x7f3704003800 con 0x7f37180f8980 2024-02-19T21:16:57.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3718102b70 con 0x7f37180f8980 2024-02-19T21:16:57.937 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3708001900 con 0x7f37180f8980 2024-02-19T21:16:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3950629059 0 0) 0x7f3718102f30 con 0x7f37180f9530 2024-02-19T21:16:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3704003800 con 0x7f37180f9530 2024-02-19T21:16:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3117982227 0 0) 0x7f3718102d40 con 0x7f37181042b0 2024-02-19T21:16:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 >> v1:172.21.15.193:6789/0 conn(0x7f37180f9530 legacy=0x7f3718114d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 >> v1:172.21.15.163:6789/0 conn(0x7f37180f8980 legacy=0x7f37181023d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f371811cbc0 con 0x7f37181042b0 2024-02-19T21:16:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f3720748700 1 -- 172.21.15.163:0/4070686890 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3718119bb0 con 0x7f37181042b0 2024-02-19T21:16:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f3720748700 10 client.50567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f3720748700 1 -- 172.21.15.163:0/4070686890 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f371811a020 con 0x7f37181042b0 2024-02-19T21:16:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f3720748700 1 librados: init done 2024-02-19T21:16:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3710002540 con 0x7f37181042b0 2024-02-19T21:16:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.208+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3710003540 con 0x7f37181042b0 2024-02-19T21:16:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.209+0000 7f3720748700 1 -- 172.21.15.163:0/4070686890 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36fc004f20 con 0x7f37181042b0 2024-02-19T21:16:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.210+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f37100020a0 con 0x7f37181042b0 2024-02-19T21:16:57.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.210+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(759..759 src has 623..759) v4 ==== 8475+0+0 (unknown 3943021735 0 0) 0x7f3710076810 con 0x7f37181042b0 2024-02-19T21:16:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.210+0000 7f370f7fe700 10 client.50567.objecter ms_dispatch 0x7f3718001550 osd_map(759..759 src has 623..759) v4 2024-02-19T21:16:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.210+0000 7f370f7fe700 3 client.50567.objecter handle_osd_map got epochs [759,759] > 0 2024-02-19T21:16:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.210+0000 7f370f7fe700 3 client.50567.objecter handle_osd_map decoding full epoch 759 2024-02-19T21:16:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.210+0000 7f370f7fe700 20 client.50567.objecter dump_active .. 0 homeless 2024-02-19T21:16:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.210+0000 7f370f7fe700 10 client.50567.objecter ms_handle_connect 0x7f3704063c90 2024-02-19T21:16:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.217+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3710049a60 con 0x7f37181042b0 2024-02-19T21:16:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.397+0000 7f3720748700 1 -- 172.21.15.163:0/4070686890 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd out", "ids": ["5"]} v 0) v1 -- 0x7f36fc0051d0 con 0x7f37181042b0 2024-02-19T21:16:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.537+0000 7f370f7fe700 1 -- 172.21.15.163:0/4070686890 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd out", "ids": ["5"]}]=0 marked out osd.5. v760) v1 ==== 87+0+0 (unknown 3721433320 0 0) 0x7f3710047e70 con 0x7f37181042b0 2024-02-19T21:16:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.542+0000 7f3720748700 10 librados: watch_flush enter 2024-02-19T21:16:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.542+0000 7f3720748700 10 librados: watch_flush exit 2024-02-19T21:16:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.542+0000 7f3720748700 20 client.50567.objecter shutdown clearing up homeless session... 2024-02-19T21:16:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.542+0000 7f3720748700 10 client.50567.objecter successfully canceled tick 2024-02-19T21:16:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.544+0000 7f3720748700 1 -- 172.21.15.163:0/4070686890 >> v1:172.21.15.163:6800/1552514681 conn(0x7f3704063c90 legacy=0x7f3704066150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.544+0000 7f3720748700 1 -- 172.21.15.163:0/4070686890 >> v1:172.21.15.163:6790/0 conn(0x7f37181042b0 legacy=0x7f3718118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:57.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.547+0000 7f3720748700 1 -- 172.21.15.163:0/4070686890 shutdown_connections 2024-02-19T21:16:57.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.547+0000 7f3720748700 1 -- 172.21.15.163:0/4070686890 >> 172.21.15.163:0/4070686890 conn(0x7f37180f2470 msgr2=0x7f3718107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:57.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.547+0000 7f3720748700 1 -- 172.21.15.163:0/4070686890 shutdown_connections 2024-02-19T21:16:57.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.547+0000 7f3720748700 1 -- 172.21.15.163:0/4070686890 wait complete. 2024-02-19T21:16:57.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:57.547+0000 7f3720748700 1 librados: shutdown 2024-02-19T21:16:57.940 INFO:teuthology.orchestra.run.smithi163.stderr:marked out osd.5. 2024-02-19T21:16:57.946 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:16:57.975 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:57.995 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:16:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:57.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:16:58.099 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:16:58.126 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:58.145 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:16:58.147 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:58.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:16:58.249 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:16:58.277 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:58.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.322+0000 7fb53de34700 1 -- 172.21.15.163:0/1126825945 >> v1:172.21.15.163:6790/0 conn(0x7fb5381042c0 legacy=0x7fb538106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:58.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.322+0000 7fb53de34700 1 -- 172.21.15.163:0/1126825945 shutdown_connections 2024-02-19T21:16:58.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.322+0000 7fb53de34700 1 -- 172.21.15.163:0/1126825945 >> 172.21.15.163:0/1126825945 conn(0x7fb53806cc00 msgr2=0x7fb5380f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:58.362 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.322+0000 7fb53de34700 1 -- 172.21.15.163:0/1126825945 shutdown_connections 2024-02-19T21:16:58.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.322+0000 7fb53de34700 1 -- 172.21.15.163:0/1126825945 wait complete. 2024-02-19T21:16:58.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.322+0000 7fb53de34700 1 librados: starting msgr at 2024-02-19T21:16:58.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.322+0000 7fb53de34700 1 librados: starting objecter 2024-02-19T21:16:58.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.322+0000 7fb53de34700 1 Processor -- start 2024-02-19T21:16:58.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb53de34700 1 -- start start 2024-02-19T21:16:58.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb53de34700 1 librados: setting wanted keys 2024-02-19T21:16:58.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb53de34700 1 librados: calling monclient init 2024-02-19T21:16:58.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb53de34700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb5381172d0 con 0x7fb5381007b0 2024-02-19T21:16:58.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb53de34700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb5381184b0 con 0x7fb5380fd960 2024-02-19T21:16:58.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb53de34700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb5381196b0 con 0x7fb5381042c0 2024-02-19T21:16:58.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb536ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fb5381007b0 0x7fb5380fa150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:47236/0 (socket says 172.21.15.163:47236) 2024-02-19T21:16:58.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb536ffd700 1 -- 172.21.15.163:0/3068753900 learned_addr learned my addr 172.21.15.163:0/3068753900 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:16:58.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 10 client.?.objecter ms_handle_connect 0x7fb5381007b0 2024-02-19T21:16:58.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:58.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 10 client.?.objecter ms_handle_connect 0x7fb5381042c0 2024-02-19T21:16:58.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:58.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 10 client.?.objecter ms_handle_connect 0x7fb5380fd960 2024-02-19T21:16:58.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:16:58.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979259378 0 0) 0x7fb5381172d0 con 0x7fb5381007b0 2024-02-19T21:16:58.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5240033f0 con 0x7fb5381007b0 2024-02-19T21:16:58.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260593478 0 0) 0x7fb5381196b0 con 0x7fb5381042c0 2024-02-19T21:16:58.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5381172d0 con 0x7fb5381042c0 2024-02-19T21:16:58.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823757830 0 0) 0x7fb5381184b0 con 0x7fb5380fd960 2024-02-19T21:16:58.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5381196b0 con 0x7fb5380fd960 2024-02-19T21:16:58.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 610214952 0 0) 0x7fb5240033f0 con 0x7fb5381007b0 2024-02-19T21:16:58.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5381184b0 con 0x7fb5381007b0 2024-02-19T21:16:58.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb520002970 con 0x7fb5381007b0 2024-02-19T21:16:58.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3451921832 0 0) 0x7fb5381172d0 con 0x7fb5381042c0 2024-02-19T21:16:58.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5240033f0 con 0x7fb5381042c0 2024-02-19T21:16:58.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb5300023d0 con 0x7fb5381042c0 2024-02-19T21:16:58.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.323+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1024429334 0 0) 0x7fb5381196b0 con 0x7fb5380fd960 2024-02-19T21:16:58.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.324+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5381172d0 con 0x7fb5380fd960 2024-02-19T21:16:58.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.324+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb528001980 con 0x7fb5380fd960 2024-02-19T21:16:58.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.324+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1090188228 0 0) 0x7fb5381184b0 con 0x7fb5381007b0 2024-02-19T21:16:58.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.324+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 >> v1:172.21.15.163:6790/0 conn(0x7fb5381042c0 legacy=0x7fb538115990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:58.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.324+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 >> v1:172.21.15.193:6789/0 conn(0x7fb5380fd960 legacy=0x7fb5380f9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:58.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.324+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53811a8b0 con 0x7fb5381007b0 2024-02-19T21:16:58.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.324+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5381198a0 con 0x7fb5381007b0 2024-02-19T21:16:58.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.324+0000 7fb53de34700 10 client.23844.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:16:58.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.324+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb538119d90 con 0x7fb5381007b0 2024-02-19T21:16:58.366 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:16:58.366 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:16:58.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.324+0000 7fb53de34700 1 librados: init done 2024-02-19T21:16:58.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.324+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb520001850 con 0x7fb5381007b0 2024-02-19T21:16:58.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.324+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb520005ae0 con 0x7fb5381007b0 2024-02-19T21:16:58.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.325+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7fb518004f00 con 0x7fb5381007b0 2024-02-19T21:16:58.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.325+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fb520018970 con 0x7fb5381007b0 2024-02-19T21:16:58.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.325+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(760..760 src has 623..760) v4 ==== 8417+0+0 (unknown 4142287843 0 0) 0x7fb520078d80 con 0x7fb5381007b0 2024-02-19T21:16:58.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.325+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 osd_map(760..760 src has 623..760) v4 2024-02-19T21:16:58.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.325+0000 7fb534ff9700 3 client.23844.objecter handle_osd_map got epochs [760,760] > 0 2024-02-19T21:16:58.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.325+0000 7fb534ff9700 3 client.23844.objecter handle_osd_map decoding full epoch 760 2024-02-19T21:16:58.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.326+0000 7fb534ff9700 20 client.23844.objecter dump_active .. 0 homeless 2024-02-19T21:16:58.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.326+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v760) v1 ==== 54+0+16 (unknown 2614430437 0 2420000192) 0x7fb52007c020 con 0x7fb5381007b0 2024-02-19T21:16:58.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.326+0000 7fb534ff9700 10 client.23844.objecter ms_handle_connect 0x7fb524063bf0 2024-02-19T21:16:58.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.326+0000 7fb53de34700 10 client.23844.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:58.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.326+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 1 2024-02-19T21:16:58.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.326+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb5180051a0 osd=0 3 2024-02-19T21:16:58.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.326+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb5180051a0 osd=0 3 2024-02-19T21:16:58.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.326+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb5180051a0 osd=0 3 2024-02-19T21:16:58.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.326+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 1 2024-02-19T21:16:58.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.326+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb5180051a0 osd=0 3 2024-02-19T21:16:58.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.326+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 0 1 2024-02-19T21:16:58.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.326+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb5180051a0 osd=0 4 2024-02-19T21:16:58.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.326+0000 7fb53de34700 10 client.23844.objecter _send_command 1 2024-02-19T21:16:58.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.326+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb518004950 con 0x7fb518004530 2024-02-19T21:16:58.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.326+0000 7fb534ff9700 10 client.23844.objecter ms_handle_connect 0x7fb518004530 2024-02-19T21:16:58.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.327+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb518004950 con 0x7fb518004530 2024-02-19T21:16:58.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.327+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:16:58.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.327+0000 7fb534ff9700 10 client.23844.objecter _finish_command 1 = system:0 2024-02-19T21:16:58.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.327+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb5180051a0 osd=0 4 2024-02-19T21:16:58.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.327+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 0 1 2024-02-19T21:16:58.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.341+0000 7fb53de34700 10 client.23844.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:58.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.341+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 2 2024-02-19T21:16:58.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.341+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb5180051a0 osd=0 3 2024-02-19T21:16:58.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.341+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb5180051a0 osd=0 3 2024-02-19T21:16:58.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.341+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb5180051a0 osd=0 3 2024-02-19T21:16:58.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.341+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 2 2024-02-19T21:16:58.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.341+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb5180051a0 osd=0 3 2024-02-19T21:16:58.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.341+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 0 2 2024-02-19T21:16:58.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.341+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb5180051a0 osd=0 4 2024-02-19T21:16:58.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.341+0000 7fb53de34700 10 client.23844.objecter _send_command 2 2024-02-19T21:16:58.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.341+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb518004d30 con 0x7fb518004530 2024-02-19T21:16:58.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb518004d30 con 0x7fb518004530 2024-02-19T21:16:58.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:58.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb534ff9700 10 client.23844.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb5180051a0 osd=0 4 2024-02-19T21:16:58.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 0 2 2024-02-19T21:16:58.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb53de34700 10 client.23844.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:58.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 3 2024-02-19T21:16:58.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518003fe0 osd=1 3 2024-02-19T21:16:58.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518003fe0 osd=1 3 2024-02-19T21:16:58.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518003fe0 osd=1 3 2024-02-19T21:16:58.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 3 2024-02-19T21:16:58.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb518003fe0 osd=1 3 2024-02-19T21:16:58.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 1 3 2024-02-19T21:16:58.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518003fe0 osd=1 4 2024-02-19T21:16:58.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb53de34700 10 client.23844.objecter _send_command 3 2024-02-19T21:16:58.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb51800c000 con 0x7fb518004150 2024-02-19T21:16:58.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.343+0000 7fb534ff9700 10 client.23844.objecter ms_handle_connect 0x7fb518004150 2024-02-19T21:16:58.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.344+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb51800c000 con 0x7fb518004150 2024-02-19T21:16:58.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.344+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:16:58.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.344+0000 7fb534ff9700 10 client.23844.objecter _finish_command 3 = system:0 2024-02-19T21:16:58.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.344+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb518003fe0 osd=1 4 2024-02-19T21:16:58.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.344+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 1 3 2024-02-19T21:16:58.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.358+0000 7fb53de34700 10 client.23844.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:58.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.358+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 4 2024-02-19T21:16:58.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.358+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518003fe0 osd=1 3 2024-02-19T21:16:58.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.358+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518003fe0 osd=1 3 2024-02-19T21:16:58.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.358+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518003fe0 osd=1 3 2024-02-19T21:16:58.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.358+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 4 2024-02-19T21:16:58.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.358+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb518003fe0 osd=1 3 2024-02-19T21:16:58.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.358+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 1 4 2024-02-19T21:16:58.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.358+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518003fe0 osd=1 4 2024-02-19T21:16:58.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.358+0000 7fb53de34700 10 client.23844.objecter _send_command 4 2024-02-19T21:16:58.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.358+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb5180018b0 con 0x7fb518004150 2024-02-19T21:16:58.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb5180018b0 con 0x7fb518004150 2024-02-19T21:16:58.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:58.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb534ff9700 10 client.23844.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb518003fe0 osd=1 4 2024-02-19T21:16:58.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 1 4 2024-02-19T21:16:58.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb53de34700 10 client.23844.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:58.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 5 2024-02-19T21:16:58.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518003630 osd=2 3 2024-02-19T21:16:58.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518003630 osd=2 3 2024-02-19T21:16:58.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518003630 osd=2 3 2024-02-19T21:16:58.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 5 2024-02-19T21:16:58.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb518003630 osd=2 3 2024-02-19T21:16:58.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 2 5 2024-02-19T21:16:58.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518003630 osd=2 4 2024-02-19T21:16:58.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb53de34700 10 client.23844.objecter _send_command 5 2024-02-19T21:16:58.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb518003e40 con 0x7fb51800d2f0 2024-02-19T21:16:58.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.360+0000 7fb534ff9700 10 client.23844.objecter ms_handle_connect 0x7fb51800d2f0 2024-02-19T21:16:58.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.361+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb518003e40 con 0x7fb51800d2f0 2024-02-19T21:16:58.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.361+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:16:58.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.361+0000 7fb534ff9700 10 client.23844.objecter _finish_command 5 = system:0 2024-02-19T21:16:58.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.361+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb518003630 osd=2 4 2024-02-19T21:16:58.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.361+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 2 5 2024-02-19T21:16:58.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.376+0000 7fb53de34700 10 client.23844.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:58.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.376+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 6 2024-02-19T21:16:58.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.376+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518003630 osd=2 3 2024-02-19T21:16:58.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.376+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518003630 osd=2 3 2024-02-19T21:16:58.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.376+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518003630 osd=2 3 2024-02-19T21:16:58.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.376+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 6 2024-02-19T21:16:58.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.376+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb518003630 osd=2 3 2024-02-19T21:16:58.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.376+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 2 6 2024-02-19T21:16:58.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.376+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518003630 osd=2 4 2024-02-19T21:16:58.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.376+0000 7fb53de34700 10 client.23844.objecter _send_command 6 2024-02-19T21:16:58.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.376+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb51800e720 con 0x7fb51800d2f0 2024-02-19T21:16:58.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.377+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb51800e720 con 0x7fb51800d2f0 2024-02-19T21:16:58.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.377+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:58.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.377+0000 7fb534ff9700 10 client.23844.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.377+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb518003630 osd=2 4 2024-02-19T21:16:58.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.377+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 2 6 2024-02-19T21:16:58.379 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:16:58.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb53de34700 10 client.23844.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:58.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 7 2024-02-19T21:16:58.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518003410 osd=3 3 2024-02-19T21:16:58.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518003410 osd=3 3 2024-02-19T21:16:58.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518003410 osd=3 3 2024-02-19T21:16:58.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 7 2024-02-19T21:16:58.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb518003410 osd=3 3 2024-02-19T21:16:58.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 3 7 2024-02-19T21:16:58.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518003410 osd=3 4 2024-02-19T21:16:58.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb53de34700 10 client.23844.objecter _send_command 7 2024-02-19T21:16:58.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb518015a60 con 0x7fb518011e50 2024-02-19T21:16:58.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb534ff9700 10 client.23844.objecter ms_handle_connect 0x7fb518011e50 2024-02-19T21:16:58.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb518015a60 con 0x7fb518011e50 2024-02-19T21:16:58.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:16:58.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb534ff9700 10 client.23844.objecter _finish_command 7 = system:0 2024-02-19T21:16:58.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb518003410 osd=3 4 2024-02-19T21:16:58.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.378+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 3 7 2024-02-19T21:16:58.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.393+0000 7fb53de34700 10 client.23844.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:58.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.393+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 8 2024-02-19T21:16:58.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.393+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518003410 osd=3 3 2024-02-19T21:16:58.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.393+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518003410 osd=3 3 2024-02-19T21:16:58.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.393+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518003410 osd=3 3 2024-02-19T21:16:58.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.393+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 8 2024-02-19T21:16:58.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.393+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb518003410 osd=3 3 2024-02-19T21:16:58.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.393+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 3 8 2024-02-19T21:16:58.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.393+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518003410 osd=3 4 2024-02-19T21:16:58.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.393+0000 7fb53de34700 10 client.23844.objecter _send_command 8 2024-02-19T21:16:58.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.393+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb51800e190 con 0x7fb518011e50 2024-02-19T21:16:58.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb51800e190 con 0x7fb518011e50 2024-02-19T21:16:58.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:58.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb534ff9700 10 client.23844.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb518003410 osd=3 4 2024-02-19T21:16:58.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 3 8 2024-02-19T21:16:58.397 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:16:58.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb53de34700 10 client.23844.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:58.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 9 2024-02-19T21:16:58.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb51800e190 osd=4 3 2024-02-19T21:16:58.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb51800e190 osd=4 3 2024-02-19T21:16:58.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb51800e190 osd=4 3 2024-02-19T21:16:58.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 9 2024-02-19T21:16:58.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb51800e190 osd=4 3 2024-02-19T21:16:58.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 4 9 2024-02-19T21:16:58.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb51800e190 osd=4 4 2024-02-19T21:16:58.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb53de34700 10 client.23844.objecter _send_command 9 2024-02-19T21:16:58.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.395+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb5180162e0 con 0x7fb51800ce50 2024-02-19T21:16:58.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.396+0000 7fb534ff9700 10 client.23844.objecter ms_handle_connect 0x7fb51800ce50 2024-02-19T21:16:58.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.396+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb5180162e0 con 0x7fb51800ce50 2024-02-19T21:16:58.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.396+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:16:58.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.396+0000 7fb534ff9700 10 client.23844.objecter _finish_command 9 = system:0 2024-02-19T21:16:58.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.396+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb51800e190 osd=4 4 2024-02-19T21:16:58.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.396+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 4 9 2024-02-19T21:16:58.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.411+0000 7fb53de34700 10 client.23844.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:58.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.411+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 10 2024-02-19T21:16:58.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.411+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb51800e190 osd=4 3 2024-02-19T21:16:58.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.411+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb51800e190 osd=4 3 2024-02-19T21:16:58.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.411+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb51800e190 osd=4 3 2024-02-19T21:16:58.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.411+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 10 2024-02-19T21:16:58.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.411+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb51800e190 osd=4 3 2024-02-19T21:16:58.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.411+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 4 10 2024-02-19T21:16:58.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.411+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb51800e190 osd=4 4 2024-02-19T21:16:58.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.411+0000 7fb53de34700 10 client.23844.objecter _send_command 10 2024-02-19T21:16:58.413 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.411+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb5180162e0 con 0x7fb51800ce50 2024-02-19T21:16:58.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb5180162e0 con 0x7fb51800ce50 2024-02-19T21:16:58.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:58.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb534ff9700 10 client.23844.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb51800e190 osd=4 4 2024-02-19T21:16:58.414 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 4 10 2024-02-19T21:16:58.415 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:16:58.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb53de34700 10 client.23844.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:58.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 11 2024-02-19T21:16:58.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb5180162e0 osd=5 3 2024-02-19T21:16:58.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb5180162e0 osd=5 3 2024-02-19T21:16:58.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb5180162e0 osd=5 3 2024-02-19T21:16:58.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 11 2024-02-19T21:16:58.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb5180162e0 osd=5 3 2024-02-19T21:16:58.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 5 11 2024-02-19T21:16:58.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb5180162e0 osd=5 4 2024-02-19T21:16:58.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb53de34700 10 client.23844.objecter _send_command 11 2024-02-19T21:16:58.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.413+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb518016ff0 con 0x7fb518016a50 2024-02-19T21:16:58.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.414+0000 7fb534ff9700 10 client.23844.objecter ms_handle_connect 0x7fb518016a50 2024-02-19T21:16:58.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.414+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb518016ff0 con 0x7fb518016a50 2024-02-19T21:16:58.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.414+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:16:58.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.414+0000 7fb534ff9700 10 client.23844.objecter _finish_command 11 = system:0 2024-02-19T21:16:58.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.414+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb5180162e0 osd=5 4 2024-02-19T21:16:58.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.414+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 5 11 2024-02-19T21:16:58.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.428+0000 7fb53de34700 10 client.23844.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:58.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.428+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 12 2024-02-19T21:16:58.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.428+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb5180162e0 osd=5 3 2024-02-19T21:16:58.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.428+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb5180162e0 osd=5 3 2024-02-19T21:16:58.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.428+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb5180162e0 osd=5 3 2024-02-19T21:16:58.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.428+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 12 2024-02-19T21:16:58.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.428+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb5180162e0 osd=5 3 2024-02-19T21:16:58.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.428+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 5 12 2024-02-19T21:16:58.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.428+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb5180162e0 osd=5 4 2024-02-19T21:16:58.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.428+0000 7fb53de34700 10 client.23844.objecter _send_command 12 2024-02-19T21:16:58.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.428+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb518016ff0 con 0x7fb518016a50 2024-02-19T21:16:58.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb518016ff0 con 0x7fb518016a50 2024-02-19T21:16:58.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:58.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb534ff9700 10 client.23844.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb5180162e0 osd=5 4 2024-02-19T21:16:58.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 5 12 2024-02-19T21:16:58.432 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:16:58.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb53de34700 10 client.23844.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:58.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 13 2024-02-19T21:16:58.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518016ff0 osd=6 3 2024-02-19T21:16:58.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518016ff0 osd=6 3 2024-02-19T21:16:58.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518016ff0 osd=6 3 2024-02-19T21:16:58.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 13 2024-02-19T21:16:58.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb518016ff0 osd=6 3 2024-02-19T21:16:58.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 6 13 2024-02-19T21:16:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518016ff0 osd=6 4 2024-02-19T21:16:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb53de34700 10 client.23844.objecter _send_command 13 2024-02-19T21:16:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.430+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb51801fc00 con 0x7fb518020670 2024-02-19T21:16:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.431+0000 7fb534ff9700 10 client.23844.objecter ms_handle_connect 0x7fb518020670 2024-02-19T21:16:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.432+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb51801fc00 con 0x7fb518020670 2024-02-19T21:16:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.432+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:16:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.432+0000 7fb534ff9700 10 client.23844.objecter _finish_command 13 = system:0 2024-02-19T21:16:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.432+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb518016ff0 osd=6 4 2024-02-19T21:16:58.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.432+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 6 13 2024-02-19T21:16:58.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.446+0000 7fb53de34700 10 client.23844.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:58.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.446+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 14 2024-02-19T21:16:58.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.446+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518016ff0 osd=6 3 2024-02-19T21:16:58.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.446+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518016ff0 osd=6 3 2024-02-19T21:16:58.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.446+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb518016ff0 osd=6 3 2024-02-19T21:16:58.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.446+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 14 2024-02-19T21:16:58.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.446+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb518016ff0 osd=6 3 2024-02-19T21:16:58.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.446+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 6 14 2024-02-19T21:16:58.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.446+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518016ff0 osd=6 4 2024-02-19T21:16:58.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.446+0000 7fb53de34700 10 client.23844.objecter _send_command 14 2024-02-19T21:16:58.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.446+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb51801fc00 con 0x7fb518020670 2024-02-19T21:16:58.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb51801fc00 con 0x7fb518020670 2024-02-19T21:16:58.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:58.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb534ff9700 10 client.23844.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb518016ff0 osd=6 4 2024-02-19T21:16:58.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 6 14 2024-02-19T21:16:58.449 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:16:58.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb53de34700 10 client.23844.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:16:58.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 15 2024-02-19T21:16:58.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb51801fc00 osd=7 3 2024-02-19T21:16:58.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb51801fc00 osd=7 3 2024-02-19T21:16:58.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb51801fc00 osd=7 3 2024-02-19T21:16:58.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 15 2024-02-19T21:16:58.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb51801fc00 osd=7 3 2024-02-19T21:16:58.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 7 15 2024-02-19T21:16:58.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb51801fc00 osd=7 4 2024-02-19T21:16:58.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb53de34700 10 client.23844.objecter _send_command 15 2024-02-19T21:16:58.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb518027360 con 0x7fb518023850 2024-02-19T21:16:58.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.448+0000 7fb534ff9700 10 client.23844.objecter ms_handle_connect 0x7fb518023850 2024-02-19T21:16:58.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.449+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb518027360 con 0x7fb518023850 2024-02-19T21:16:58.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.449+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:16:58.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.449+0000 7fb534ff9700 10 client.23844.objecter _finish_command 15 = system:0 2024-02-19T21:16:58.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.449+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb51801fc00 osd=7 4 2024-02-19T21:16:58.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.449+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 7 15 2024-02-19T21:16:58.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.463+0000 7fb53de34700 10 client.23844.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:16:58.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.463+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign -1 16 2024-02-19T21:16:58.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.463+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb51801fc00 osd=7 3 2024-02-19T21:16:58.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.463+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb51801fc00 osd=7 3 2024-02-19T21:16:58.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.463+0000 7fb53de34700 20 client.23844.objecter _get_session s=0x7fb51801fc00 osd=7 3 2024-02-19T21:16:58.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.463+0000 7fb53de34700 15 client.23844.objecter _session_command_op_remove -1 16 2024-02-19T21:16:58.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.463+0000 7fb53de34700 20 client.23844.objecter get_session s=0x7fb51801fc00 osd=7 3 2024-02-19T21:16:58.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.463+0000 7fb53de34700 15 client.23844.objecter _session_command_op_assign 7 16 2024-02-19T21:16:58.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.463+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb51801fc00 osd=7 4 2024-02-19T21:16:58.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.463+0000 7fb53de34700 10 client.23844.objecter _send_command 16 2024-02-19T21:16:58.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.463+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7fb518027360 con 0x7fb518023850 2024-02-19T21:16:58.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.465+0000 7fb534ff9700 1 -- 172.21.15.163:0/3068753900 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7fb518027360 con 0x7fb518023850 2024-02-19T21:16:58.466 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.465+0000 7fb534ff9700 10 client.23844.objecter ms_dispatch 0x7fb538001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:16:58.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.465+0000 7fb534ff9700 10 client.23844.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.465+0000 7fb534ff9700 20 client.23844.objecter put_session s=0x7fb51801fc00 osd=7 4 2024-02-19T21:16:58.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.465+0000 7fb534ff9700 15 client.23844.objecter _session_command_op_remove 7 16 2024-02-19T21:16:58.467 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:16:58.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 10 librados: watch_flush enter 2024-02-19T21:16:58.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 10 librados: watch_flush exit 2024-02-19T21:16:58.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 10 client.23844.objecter close_session for osd.0 2024-02-19T21:16:58.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 >> v1:172.21.15.163:6801/1559418115 conn(0x7fb518004530 legacy=0x7fb518002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:58.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb5180051a0 osd=0 1 2024-02-19T21:16:58.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 10 client.23844.objecter close_session for osd.1 2024-02-19T21:16:58.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 >> v1:172.21.15.163:6805/3125047432 conn(0x7fb518004150 legacy=0x7fb51800a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:58.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518003fe0 osd=1 1 2024-02-19T21:16:58.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 10 client.23844.objecter close_session for osd.2 2024-02-19T21:16:58.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 >> v1:172.21.15.163:6809/1563158487 conn(0x7fb51800d2f0 legacy=0x7fb51800d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:58.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518003630 osd=2 1 2024-02-19T21:16:58.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 10 client.23844.objecter close_session for osd.3 2024-02-19T21:16:58.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 >> v1:172.21.15.163:6813/2112279769 conn(0x7fb518011e50 legacy=0x7fb518014240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:58.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb537fff700 1 -- 172.21.15.163:0/3068753900 reap_dead start 2024-02-19T21:16:58.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518003410 osd=3 1 2024-02-19T21:16:58.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 10 client.23844.objecter close_session for osd.4 2024-02-19T21:16:58.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 >> v1:172.21.15.193:6800/3129716965 conn(0x7fb51800ce50 legacy=0x7fb518015bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:58.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb51800e190 osd=4 1 2024-02-19T21:16:58.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 10 client.23844.objecter close_session for osd.5 2024-02-19T21:16:58.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 >> v1:172.21.15.193:6804/805161991 conn(0x7fb518016a50 legacy=0x7fb51801d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:58.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb5180162e0 osd=5 1 2024-02-19T21:16:58.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 10 client.23844.objecter close_session for osd.6 2024-02-19T21:16:58.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 >> v1:172.21.15.193:6808/787604615 conn(0x7fb518020670 legacy=0x7fb518020af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:58.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb518016ff0 osd=6 1 2024-02-19T21:16:58.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 10 client.23844.objecter close_session for osd.7 2024-02-19T21:16:58.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 >> v1:172.21.15.193:6812/157927432 conn(0x7fb518023850 legacy=0x7fb518025c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:58.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 20 client.23844.objecter put_session s=0x7fb51801fc00 osd=7 1 2024-02-19T21:16:58.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 20 client.23844.objecter shutdown clearing up homeless session... 2024-02-19T21:16:58.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 10 client.23844.objecter successfully canceled tick 2024-02-19T21:16:58.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb537fff700 1 -- 172.21.15.163:0/3068753900 reap_dead start 2024-02-19T21:16:58.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 >> v1:172.21.15.163:6800/1552514681 conn(0x7fb524063bf0 legacy=0x7fb5240660b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:58.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 >> v1:172.21.15.163:6789/0 conn(0x7fb5381007b0 legacy=0x7fb5380fa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:16:58.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 shutdown_connections 2024-02-19T21:16:58.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 >> 172.21.15.163:0/3068753900 conn(0x7fb53806cc00 msgr2=0x7fb538107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:16:58.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 shutdown_connections 2024-02-19T21:16:58.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 1 -- 172.21.15.163:0/3068753900 wait complete. 2024-02-19T21:16:58.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:16:58.466+0000 7fb53de34700 1 librados: shutdown 2024-02-19T21:16:58.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.472 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:16:58.551 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:56.795036+0000 osd.6 (osd.6) 267 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:16:58.551 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:56.796030+0000 osd.6 (osd.6) 268 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:16:58.551 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:57.330042+0000 mgr.y (mgr.14550) 634 : cluster [DBG] pgmap v1149: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 212 B/s wr, 1 op/s 2024-02-19T21:16:58.551 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:57.529796+0000 mon.a (mon.0) 3611 : audit [INF] from='client.50567 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-02-19T21:16:58.551 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:57.529873+0000 mon.a (mon.0) 3612 : cluster [DBG] osdmap e760: 8 total, 8 up, 6 in 2024-02-19T21:16:58.551 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:16:58.326898+0000 mon.a (mon.0) 3613 : audit [DBG] from='client.? v1:172.21.15.163:0/3068753900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:58.551 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:56.795036+0000 osd.6 (osd.6) 267 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:16:58.551 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:56.796030+0000 osd.6 (osd.6) 268 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:16:58.552 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:57.330042+0000 mgr.y (mgr.14550) 634 : cluster [DBG] pgmap v1149: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 212 B/s wr, 1 op/s 2024-02-19T21:16:58.552 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:57.529796+0000 mon.a (mon.0) 3611 : audit [INF] from='client.50567 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-02-19T21:16:58.552 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:57.529873+0000 mon.a (mon.0) 3612 : cluster [DBG] osdmap e760: 8 total, 8 up, 6 in 2024-02-19T21:16:58.552 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:16:58.326898+0000 mon.a (mon.0) 3613 : audit [DBG] from='client.? v1:172.21.15.163:0/3068753900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:58.804 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:16:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:16:58.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:16:58.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:56.795036+0000 osd.6 (osd.6) 267 : cluster [DBG] 2.17 scrub starts 2024-02-19T21:16:58.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:56.796030+0000 osd.6 (osd.6) 268 : cluster [DBG] 2.17 scrub ok 2024-02-19T21:16:58.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:57.330042+0000 mgr.y (mgr.14550) 634 : cluster [DBG] pgmap v1149: 214 pgs: 214 active+clean; 3.8 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 212 B/s wr, 1 op/s 2024-02-19T21:16:58.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:57.529796+0000 mon.a (mon.0) 3611 : audit [INF] from='client.50567 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-02-19T21:16:58.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:57.529873+0000 mon.a (mon.0) 3612 : cluster [DBG] osdmap e760: 8 total, 8 up, 6 in 2024-02-19T21:16:58.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:16:58.326898+0000 mon.a (mon.0) 3613 : audit [DBG] from='client.? v1:172.21.15.163:0/3068753900' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:16:58.907 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:16:58.936 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:16:59.732 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:57.817373+0000 osd.6 (osd.6) 269 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:16:59.732 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:57.818309+0000 osd.6 (osd.6) 270 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:16:59.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:58.534742+0000 mon.a (mon.0) 3614 : cluster [DBG] osdmap e761: 8 total, 8 up, 6 in 2024-02-19T21:16:59.733 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:16:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:59.410417+0000 mon.a (mon.0) 3615 : cluster [DBG] osdmap e762: 8 total, 8 up, 6 in 2024-02-19T21:16:59.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:57.817373+0000 osd.6 (osd.6) 269 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:16:59.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:57.818309+0000 osd.6 (osd.6) 270 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:16:59.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:58.534742+0000 mon.a (mon.0) 3614 : cluster [DBG] osdmap e761: 8 total, 8 up, 6 in 2024-02-19T21:16:59.733 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:16:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:59.410417+0000 mon.a (mon.0) 3615 : cluster [DBG] osdmap e762: 8 total, 8 up, 6 in 2024-02-19T21:16:59.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:57.817373+0000 osd.6 (osd.6) 269 : cluster [DBG] 2.1d scrub starts 2024-02-19T21:16:59.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:57.818309+0000 osd.6 (osd.6) 270 : cluster [DBG] 2.1d scrub ok 2024-02-19T21:16:59.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:58.534742+0000 mon.a (mon.0) 3614 : cluster [DBG] osdmap e761: 8 total, 8 up, 6 in 2024-02-19T21:16:59.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:16:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:59.410417+0000 mon.a (mon.0) 3615 : cluster [DBG] osdmap e762: 8 total, 8 up, 6 in 2024-02-19T21:17:00.087 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:00.091 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:00.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:00.092 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:00.193 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:17:00.220 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:00.346 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:17:00.349 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:00.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:17:00.450 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:00.478 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:00.497 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:00.499 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:00.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:00.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:59.330536+0000 mgr.y (mgr.14550) 635 : cluster [DBG] pgmap v1152: 214 pgs: 214 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 207 B/s wr, 1 op/s 2024-02-19T21:17:00.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:16:59.408582+0000 osd.5 (osd.5) 316 : cluster [DBG] 2.20 starting backfill to osd.1 from (0'0,0'0] MAX to 397'2 2024-02-19T21:17:00.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:00.033206+0000 osd.3 (osd.3) 317 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:17:00.597 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:00.035093+0000 osd.3 (osd.3) 318 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:17:00.597 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:59.330536+0000 mgr.y (mgr.14550) 635 : cluster [DBG] pgmap v1152: 214 pgs: 214 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 207 B/s wr, 1 op/s 2024-02-19T21:17:00.598 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:16:59.408582+0000 osd.5 (osd.5) 316 : cluster [DBG] 2.20 starting backfill to osd.1 from (0'0,0'0] MAX to 397'2 2024-02-19T21:17:00.598 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:00.033206+0000 osd.3 (osd.3) 317 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:17:00.598 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:00.035093+0000 osd.3 (osd.3) 318 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:17:00.600 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:00.627 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:00.646 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:00.649 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:00.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:00.750 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:00.779 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:00.787 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:59.330536+0000 mgr.y (mgr.14550) 635 : cluster [DBG] pgmap v1152: 214 pgs: 214 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 207 B/s wr, 1 op/s 2024-02-19T21:17:00.788 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:16:59.408582+0000 osd.5 (osd.5) 316 : cluster [DBG] 2.20 starting backfill to osd.1 from (0'0,0'0] MAX to 397'2 2024-02-19T21:17:00.788 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:00.033206+0000 osd.3 (osd.3) 317 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:17:00.788 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:00.035093+0000 osd.3 (osd.3) 318 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:17:01.072 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:17:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:17:01.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.870+0000 7f003778c700 1 -- 172.21.15.163:0/3214717721 >> v1:172.21.15.163:6789/0 conn(0x7f00300fd970 legacy=0x7f00300fdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:01.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.870+0000 7f003778c700 1 -- 172.21.15.163:0/3214717721 shutdown_connections 2024-02-19T21:17:01.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.870+0000 7f003778c700 1 -- 172.21.15.163:0/3214717721 >> 172.21.15.163:0/3214717721 conn(0x7f00300f3570 msgr2=0x7f00300f5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:01.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.870+0000 7f003778c700 1 -- 172.21.15.163:0/3214717721 shutdown_connections 2024-02-19T21:17:01.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.870+0000 7f003778c700 1 -- 172.21.15.163:0/3214717721 wait complete. 2024-02-19T21:17:01.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.870+0000 7f003778c700 1 librados: starting msgr at 2024-02-19T21:17:01.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.870+0000 7f003778c700 1 librados: starting objecter 2024-02-19T21:17:01.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f003778c700 1 Processor -- start 2024-02-19T21:17:01.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f003778c700 1 -- start start 2024-02-19T21:17:01.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f003778c700 1 librados: setting wanted keys 2024-02-19T21:17:01.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f003778c700 1 librados: calling monclient init 2024-02-19T21:17:01.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f003778c700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0030119440 con 0x7f00301042b0 2024-02-19T21:17:01.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f003778c700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f003011a5e0 con 0x7f00300fd970 2024-02-19T21:17:01.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f003778c700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f003011b780 con 0x7f0030100770 2024-02-19T21:17:01.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f0034d27700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f0030100770 0x7f00300fccc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:35464/0 (socket says 172.21.15.163:35464) 2024-02-19T21:17:01.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f0035d29700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f00301042b0 0x7f0030117d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:47254/0 (socket says 172.21.15.163:47254) 2024-02-19T21:17:01.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f0034d27700 1 -- 172.21.15.163:0/3835909841 learned_addr learned my addr 172.21.15.163:0/3835909841 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:01.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f00267fc700 10 client.?.objecter ms_handle_connect 0x7f00301042b0 2024-02-19T21:17:01.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f00267fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:01.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f00267fc700 10 client.?.objecter ms_handle_connect 0x7f0030100770 2024-02-19T21:17:01.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f00267fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:01.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f00267fc700 10 client.?.objecter ms_handle_connect 0x7f00300fd970 2024-02-19T21:17:01.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f00267fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:01.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841341327 0 0) 0x7f0030119440 con 0x7f00301042b0 2024-02-19T21:17:01.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001c0038e0 con 0x7f00301042b0 2024-02-19T21:17:01.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144357426 0 0) 0x7f003011b780 con 0x7f0030100770 2024-02-19T21:17:01.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0030119440 con 0x7f0030100770 2024-02-19T21:17:01.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026146 0 0) 0x7f003011a5e0 con 0x7f00300fd970 2024-02-19T21:17:01.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.871+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f003011b780 con 0x7f00300fd970 2024-02-19T21:17:01.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2606157682 0 0) 0x7f0030119440 con 0x7f0030100770 2024-02-19T21:17:01.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f003011a5e0 con 0x7f0030100770 2024-02-19T21:17:01.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0020001940 con 0x7f0030100770 2024-02-19T21:17:01.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1652014685 0 0) 0x7f003011b780 con 0x7f00300fd970 2024-02-19T21:17:01.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0030119440 con 0x7f00300fd970 2024-02-19T21:17:01.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f002c001f20 con 0x7f00300fd970 2024-02-19T21:17:01.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2864533396 0 0) 0x7f001c0038e0 con 0x7f00301042b0 2024-02-19T21:17:01.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f003011b780 con 0x7f00301042b0 2024-02-19T21:17:01.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0028001e00 con 0x7f00301042b0 2024-02-19T21:17:01.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4039211861 0 0) 0x7f003011a5e0 con 0x7f0030100770 2024-02-19T21:17:01.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 >> v1:172.21.15.193:6789/0 conn(0x7f00300fd970 legacy=0x7f00300fc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:01.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 >> v1:172.21.15.163:6789/0 conn(0x7f00301042b0 legacy=0x7f0030117d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:01.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003011c960 con 0x7f0030100770 2024-02-19T21:17:01.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f003778c700 1 -- 172.21.15.163:0/3835909841 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003011b970 con 0x7f0030100770 2024-02-19T21:17:01.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f003778c700 10 client.50579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:01.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f003778c700 1 -- 172.21.15.163:0/3835909841 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f003011be60 con 0x7f0030100770 2024-02-19T21:17:01.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f003778c700 1 librados: init done 2024-02-19T21:17:01.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f0020001c40 con 0x7f0030100770 2024-02-19T21:17:01.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.872+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0020005bc0 con 0x7f0030100770 2024-02-19T21:17:01.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.873+0000 7f003778c700 1 -- 172.21.15.163:0/3835909841 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0014004f20 con 0x7f0030100770 2024-02-19T21:17:01.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.873+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f0020018a50 con 0x7f0030100770 2024-02-19T21:17:01.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.874+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(763..763 src has 623..763) v4 ==== 8417+0+0 (unknown 3515274436 0 0) 0x7f0020078e70 con 0x7f0030100770 2024-02-19T21:17:01.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.874+0000 7f00267fc700 10 client.50579.objecter ms_dispatch 0x7f0030001550 osd_map(763..763 src has 623..763) v4 2024-02-19T21:17:01.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.874+0000 7f00267fc700 3 client.50579.objecter handle_osd_map got epochs [763,763] > 0 2024-02-19T21:17:01.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.874+0000 7f00267fc700 3 client.50579.objecter handle_osd_map decoding full epoch 763 2024-02-19T21:17:01.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.874+0000 7f00267fc700 20 client.50579.objecter dump_active .. 0 homeless 2024-02-19T21:17:01.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.874+0000 7f00267fc700 10 client.50579.objecter ms_handle_connect 0x7f001c063cc0 2024-02-19T21:17:01.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:00.878+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f002004c120 con 0x7f0030100770 2024-02-19T21:17:01.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:01.057+0000 7f003778c700 1 -- 172.21.15.163:0/3835909841 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f00140051d0 con 0x7f0030100770 2024-02-19T21:17:01.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:01.566+0000 7f00267fc700 1 -- 172.21.15.163:0/3835909841 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v764) v1 ==== 101+0+0 (unknown 3359471907 0 0) 0x7f002004a530 con 0x7f0030100770 2024-02-19T21:17:01.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:01.568+0000 7f003778c700 10 librados: watch_flush enter 2024-02-19T21:17:01.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:01.568+0000 7f003778c700 10 librados: watch_flush exit 2024-02-19T21:17:01.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:01.568+0000 7f003778c700 20 client.50579.objecter shutdown clearing up homeless session... 2024-02-19T21:17:01.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:01.568+0000 7f003778c700 10 client.50579.objecter successfully canceled tick 2024-02-19T21:17:01.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:01.568+0000 7f003778c700 1 -- 172.21.15.163:0/3835909841 >> v1:172.21.15.163:6800/1552514681 conn(0x7f001c063cc0 legacy=0x7f001c066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:01.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:01.568+0000 7f003778c700 1 -- 172.21.15.163:0/3835909841 >> v1:172.21.15.163:6790/0 conn(0x7f0030100770 legacy=0x7f00300fccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:01.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:01.568+0000 7f003778c700 1 -- 172.21.15.163:0/3835909841 shutdown_connections 2024-02-19T21:17:01.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:01.568+0000 7f003778c700 1 -- 172.21.15.163:0/3835909841 >> 172.21.15.163:0/3835909841 conn(0x7f00300f3570 msgr2=0x7f00300f64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:01.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:01.569+0000 7f003778c700 1 -- 172.21.15.163:0/3835909841 shutdown_connections 2024-02-19T21:17:01.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:01.569+0000 7f003778c700 1 -- 172.21.15.163:0/3835909841 wait complete. 2024-02-19T21:17:01.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:01.569+0000 7f003778c700 1 librados: shutdown 2024-02-19T21:17:01.802 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:17:01.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:00.355430+0000 osd.1 (osd.1) 336 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:17:01.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:00.356280+0000 osd.1 (osd.1) 337 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:17:01.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:00.557109+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e763: 8 total, 8 up, 6 in 2024-02-19T21:17:01.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:00.787643+0000 mgr.y (mgr.14550) 636 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:01.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:01.053359+0000 osd.3 (osd.3) 319 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:17:01.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:01.055071+0000 osd.3 (osd.3) 320 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:17:01.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:01.058892+0000 mon.c (mon.2) 436 : audit [INF] from='client.? v1:172.21.15.163:0/3835909841' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:01.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:01.059175+0000 mon.a (mon.0) 3617 : audit [INF] from='client.50579 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:01.998 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:00.355430+0000 osd.1 (osd.1) 336 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:17:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:00.356280+0000 osd.1 (osd.1) 337 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:17:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:00.557109+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e763: 8 total, 8 up, 6 in 2024-02-19T21:17:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:00.787643+0000 mgr.y (mgr.14550) 636 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:01.053359+0000 osd.3 (osd.3) 319 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:17:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:01.055071+0000 osd.3 (osd.3) 320 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:17:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:01.058892+0000 mon.c (mon.2) 436 : audit [INF] from='client.? v1:172.21.15.163:0/3835909841' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:01.059175+0000 mon.a (mon.0) 3617 : audit [INF] from='client.50579 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:01.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:00.355430+0000 osd.1 (osd.1) 336 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:17:01.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:00.356280+0000 osd.1 (osd.1) 337 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:17:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:00.557109+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e763: 8 total, 8 up, 6 in 2024-02-19T21:17:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:00.787643+0000 mgr.y (mgr.14550) 636 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:01.053359+0000 osd.3 (osd.3) 319 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:17:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:01.055071+0000 osd.3 (osd.3) 320 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:17:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:01.058892+0000 mon.c (mon.2) 436 : audit [INF] from='client.? v1:172.21.15.163:0/3835909841' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:01.059175+0000 mon.a (mon.0) 3617 : audit [INF] from='client.50579 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:02.082 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:17:02.528 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:02.531 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:02.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:02.589 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:02.632 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:02.664 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:02.685 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:02.688 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:02.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:02.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:01.314030+0000 osd.1 (osd.1) 338 : cluster [DBG] 4.f scrub starts 2024-02-19T21:17:02.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:01.317502+0000 osd.1 (osd.1) 339 : cluster [DBG] 4.f scrub ok 2024-02-19T21:17:02.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:01.331059+0000 mgr.y (mgr.14550) 637 : cluster [DBG] pgmap v1155: 214 pgs: 1 active+clean+remapped, 213 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 977 KiB/s, 15 objects/s recovering 2024-02-19T21:17:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:01.555601+0000 mon.a (mon.0) 3618 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:17:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:01.563829+0000 mon.a (mon.0) 3619 : audit [INF] from='client.50579 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:17:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:01.563893+0000 mon.a (mon.0) 3620 : cluster [DBG] osdmap e764: 8 total, 8 up, 6 in 2024-02-19T21:17:02.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:01.314030+0000 osd.1 (osd.1) 338 : cluster [DBG] 4.f scrub starts 2024-02-19T21:17:02.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:01.317502+0000 osd.1 (osd.1) 339 : cluster [DBG] 4.f scrub ok 2024-02-19T21:17:02.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:01.331059+0000 mgr.y (mgr.14550) 637 : cluster [DBG] pgmap v1155: 214 pgs: 1 active+clean+remapped, 213 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 977 KiB/s, 15 objects/s recovering 2024-02-19T21:17:02.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:01.555601+0000 mon.a (mon.0) 3618 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:17:02.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:01.563829+0000 mon.a (mon.0) 3619 : audit [INF] from='client.50579 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:17:02.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:01.563893+0000 mon.a (mon.0) 3620 : cluster [DBG] osdmap e764: 8 total, 8 up, 6 in 2024-02-19T21:17:02.790 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:02.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:01.314030+0000 osd.1 (osd.1) 338 : cluster [DBG] 4.f scrub starts 2024-02-19T21:17:02.817 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:01.317502+0000 osd.1 (osd.1) 339 : cluster [DBG] 4.f scrub ok 2024-02-19T21:17:02.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:01.331059+0000 mgr.y (mgr.14550) 637 : cluster [DBG] pgmap v1155: 214 pgs: 1 active+clean+remapped, 213 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 977 KiB/s, 15 objects/s recovering 2024-02-19T21:17:02.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:01.555601+0000 mon.a (mon.0) 3618 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:17:02.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:01.563829+0000 mon.a (mon.0) 3619 : audit [INF] from='client.50579 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:17:02.818 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:01.563893+0000 mon.a (mon.0) 3620 : cluster [DBG] osdmap e764: 8 total, 8 up, 6 in 2024-02-19T21:17:02.820 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:02.840 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:02.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:02.945 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:02.974 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:02.993 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:02.996 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:02.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:03.097 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:17:03.127 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:03.426 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:17:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-02-19T21:17:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-02-19T21:17:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-02-19T21:17:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.49316.0:1 166.6s0 166:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e244)", 2024-02-19T21:17:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:07:04.203723+0000", 2024-02-19T21:17:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 599.12991101700004, 2024-02-19T21:17:03.426 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0215501570000001, 2024-02-19T21:17:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:17:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.49316", 2024-02-19T21:17:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/89414708", 2024-02-19T21:17:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:17:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:04.203723+0000", 2024-02-19T21:17:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.427 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:04.203723+0000", 2024-02-19T21:17:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:17:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:04.203720+0000", 2024-02-19T21:17:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2180000000000005e-06 2024-02-19T21:17:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:04.203729+0000", 2024-02-19T21:17:03.428 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4591e-05 2024-02-19T21:17:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:04.203744+0000", 2024-02-19T21:17:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3180000000000007e-06 2024-02-19T21:17:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:04.203752+0000", 2024-02-19T21:17:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0070971769999999997 2024-02-19T21:17:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.429 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:04.210849+0000", 2024-02-19T21:17:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4010000000000001e-06 2024-02-19T21:17:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:17:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:04.210853+0000", 2024-02-19T21:17:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0112680970000001 2024-02-19T21:17:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:05.222121+0000", 2024-02-19T21:17:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00011819699999999999 2024-02-19T21:17:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.430 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:05.222239+0000", 2024-02-19T21:17:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00043573199999999998 2024-02-19T21:17:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:17:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:05.222675+0000", 2024-02-19T21:17:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00037217400000000002 2024-02-19T21:17:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:17:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:05.223047+0000", 2024-02-19T21:17:03.431 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021503109999999998 2024-02-19T21:17:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:17:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:05.225197+0000", 2024-02-19T21:17:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5570999999999999e-05 2024-02-19T21:17:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:05.225273+0000", 2024-02-19T21:17:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0215501570000001 2024-02-19T21:17:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.432 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.48253.0:1 180.5s0 180:b3610dda:::oid_RoundTripPP2_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e276)", 2024-02-19T21:17:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:07:36.261678+0000", 2024-02-19T21:17:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 567.071955731, 2024-02-19T21:17:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.010495726, 2024-02-19T21:17:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:17:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.48253", 2024-02-19T21:17:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2746902884", 2024-02-19T21:17:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:17:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.433 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:36.261678+0000", 2024-02-19T21:17:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:36.261678+0000", 2024-02-19T21:17:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:17:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:36.261676+0000", 2024-02-19T21:17:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.6769999999999995e-06 2024-02-19T21:17:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.434 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:36.261684+0000", 2024-02-19T21:17:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4239999999999999e-06 2024-02-19T21:17:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:36.261693+0000", 2024-02-19T21:17:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5859999999999999e-06 2024-02-19T21:17:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:36.261701+0000", 2024-02-19T21:17:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0011191770000000001 2024-02-19T21:17:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.435 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:36.262820+0000", 2024-02-19T21:17:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1590000000000002e-06 2024-02-19T21:17:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:17:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:36.262823+0000", 2024-02-19T21:17:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.99171489199999996 2024-02-19T21:17:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:37.254538+0000", 2024-02-19T21:17:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000115746 2024-02-19T21:17:03.436 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:37.254654+0000", 2024-02-19T21:17:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000392469 2024-02-19T21:17:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:17:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:37.255046+0000", 2024-02-19T21:17:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032995379999999999 2024-02-19T21:17:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:17:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:37.258346+0000", 2024-02-19T21:17:03.437 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.013764788999999999 2024-02-19T21:17:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:17:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:37.272111+0000", 2024-02-19T21:17:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2924999999999998e-05 2024-02-19T21:17:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:37.272174+0000", 2024-02-19T21:17:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.010495726 2024-02-19T21:17:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.438 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.48268.0:1 181.6s0 181:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e279)", 2024-02-19T21:17:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:07:39.215621+0000", 2024-02-19T21:17:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 564.11801254299996, 2024-02-19T21:17:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0066062689999999, 2024-02-19T21:17:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:17:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.48268", 2024-02-19T21:17:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/748924268", 2024-02-19T21:17:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:17:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.439 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:39.215621+0000", 2024-02-19T21:17:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:39.215621+0000", 2024-02-19T21:17:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:17:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.440 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:39.215619+0000", 2024-02-19T21:17:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.8529999999999998e-06 2024-02-19T21:17:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:39.215628+0000", 2024-02-19T21:17:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2801e-05 2024-02-19T21:17:03.441 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:39.215640+0000", 2024-02-19T21:17:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.6680000000000007e-06 2024-02-19T21:17:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:39.215649+0000", 2024-02-19T21:17:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8321e-05 2024-02-19T21:17:03.442 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:39.215697+0000", 2024-02-19T21:17:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.405e-06 2024-02-19T21:17:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:17:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:39.215701+0000", 2024-02-19T21:17:03.443 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0039779360000001 2024-02-19T21:17:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:40.219679+0000", 2024-02-19T21:17:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000137441 2024-02-19T21:17:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:40.219816+0000", 2024-02-19T21:17:03.444 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00046030000000000002 2024-02-19T21:17:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:17:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:40.220277+0000", 2024-02-19T21:17:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0012118560000000001 2024-02-19T21:17:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:17:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:40.221488+0000", 2024-02-19T21:17:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00066912299999999998 2024-02-19T21:17:03.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:17:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:40.222157+0000", 2024-02-19T21:17:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9987999999999995e-05 2024-02-19T21:17:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:40.222227+0000", 2024-02-19T21:17:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0066062689999999 2024-02-19T21:17:03.446 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.48319.0:1 187.2s0 187:5c55be4c:::oid_RoundTripAppendPP_:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected e293)", 2024-02-19T21:17:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:07:53.230611+0000", 2024-02-19T21:17:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 550.103022345, 2024-02-19T21:17:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.006825512, 2024-02-19T21:17:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:17:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:03.447 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.48319", 2024-02-19T21:17:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/1695432171", 2024-02-19T21:17:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:17:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:53.230611+0000", 2024-02-19T21:17:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.448 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:53.230611+0000", 2024-02-19T21:17:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:17:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:53.230609+0000", 2024-02-19T21:17:03.449 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5291e-05 2024-02-19T21:17:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:53.230624+0000", 2024-02-19T21:17:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0372e-05 2024-02-19T21:17:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:53.230635+0000", 2024-02-19T21:17:03.450 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9959999999999993e-06 2024-02-19T21:17:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:53.230643+0000", 2024-02-19T21:17:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00060458799999999998 2024-02-19T21:17:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:53.231247+0000", 2024-02-19T21:17:03.451 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5540000000000003e-06 2024-02-19T21:17:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:17:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:53.231257+0000", 2024-02-19T21:17:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.004141969 2024-02-19T21:17:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.235399+0000", 2024-02-19T21:17:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00055309300000000005 2024-02-19T21:17:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.452 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.235952+0000", 2024-02-19T21:17:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00048717999999999998 2024-02-19T21:17:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:17:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.236439+0000", 2024-02-19T21:17:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000528982 2024-02-19T21:17:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:17:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.236968+0000", 2024-02-19T21:17:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00040352900000000002 2024-02-19T21:17:03.453 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:17:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.237372+0000", 2024-02-19T21:17:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5417000000000001e-05 2024-02-19T21:17:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.455 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:07:54.237437+0000", 2024-02-19T21:17:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.006825512 2024-02-19T21:17:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.22830.0:1 190.2s0 190:5455dc94:::oid_IsSafePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e307)", 2024-02-19T21:17:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:08:07.245711+0000", 2024-02-19T21:17:03.456 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 536.08792318600001, 2024-02-19T21:17:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0143075429999999, 2024-02-19T21:17:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:17:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.22830", 2024-02-19T21:17:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/4035662710", 2024-02-19T21:17:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:17:03.457 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:07.245711+0000", 2024-02-19T21:17:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.458 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:07.245711+0000", 2024-02-19T21:17:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:17:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:07.245708+0000", 2024-02-19T21:17:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.2469999999999997e-06 2024-02-19T21:17:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.459 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:07.245717+0000", 2024-02-19T21:17:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.167e-05 2024-02-19T21:17:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:07.245729+0000", 2024-02-19T21:17:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2099999999999993e-06 2024-02-19T21:17:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.460 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:07.245737+0000", 2024-02-19T21:17:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00049830399999999998 2024-02-19T21:17:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:07.246236+0000", 2024-02-19T21:17:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0189999999999998e-06 2024-02-19T21:17:03.461 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:17:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:07.246239+0000", 2024-02-19T21:17:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0120812450000001 2024-02-19T21:17:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:08.258320+0000", 2024-02-19T21:17:03.462 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010292400000000001 2024-02-19T21:17:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:08.258423+0000", 2024-02-19T21:17:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00027470899999999999 2024-02-19T21:17:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:17:03.463 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:08.258697+0000", 2024-02-19T21:17:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00063694700000000001 2024-02-19T21:17:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:17:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:08.259334+0000", 2024-02-19T21:17:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00063854500000000002 2024-02-19T21:17:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.464 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:17:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:08.259973+0000", 2024-02-19T21:17:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5118999999999999e-05 2024-02-19T21:17:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:08.260018+0000", 2024-02-19T21:17:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0143075429999999 2024-02-19T21:17:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.465 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:4600 193.18 193:18000000:%01:::head [pgnls start_epoch 334] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e334)", 2024-02-19T21:17:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:08:34.318902+0000", 2024-02-19T21:17:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 509.01473196000001, 2024-02-19T21:17:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.159811432, 2024-02-19T21:17:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.466 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:17:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:17:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 4600 2024-02-19T21:17:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.467 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:34.318902+0000", 2024-02-19T21:17:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:34.318902+0000", 2024-02-19T21:17:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999967 2024-02-19T21:17:03.468 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:34.318899+0000", 2024-02-19T21:17:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2668e-05 2024-02-19T21:17:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:34.318911+0000", 2024-02-19T21:17:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8825999999999999e-05 2024-02-19T21:17:03.469 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:34.318930+0000", 2024-02-19T21:17:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0289999999999994e-06 2024-02-19T21:17:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:34.318939+0000", 2024-02-19T21:17:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6392999999999999e-05 2024-02-19T21:17:03.470 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:34.319006+0000", 2024-02-19T21:17:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.4420000000000001e-06 2024-02-19T21:17:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:17:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:34.319013+0000", 2024-02-19T21:17:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.15952350000000001 2024-02-19T21:17:03.471 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:34.478537+0000", 2024-02-19T21:17:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9653999999999997e-05 2024-02-19T21:17:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:03.472 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:34.478576+0000", 2024-02-19T21:17:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000137038 2024-02-19T21:17:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:34.478713+0000", 2024-02-19T21:17:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.159811432 2024-02-19T21:17:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.473 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.48421.0:1 196.4s0 196:384b83b2:::oid_RoundTripWriteFullPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected e335)", 2024-02-19T21:17:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:08:35.234172+0000", 2024-02-19T21:17:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 508.09946168300002, 2024-02-19T21:17:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0211542709999999, 2024-02-19T21:17:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:17:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.48421", 2024-02-19T21:17:03.474 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/104540366", 2024-02-19T21:17:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1 2024-02-19T21:17:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:35.234172+0000", 2024-02-19T21:17:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:35.234172+0000", 2024-02-19T21:17:03.475 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999967 2024-02-19T21:17:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:35.234169+0000", 2024-02-19T21:17:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0183999999999999e-05 2024-02-19T21:17:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:35.234179+0000", 2024-02-19T21:17:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0485e-05 2024-02-19T21:17:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.476 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:35.234190+0000", 2024-02-19T21:17:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8369999999999999e-06 2024-02-19T21:17:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:35.234196+0000", 2024-02-19T21:17:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7086000000000003e-05 2024-02-19T21:17:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:35.234244+0000", 2024-02-19T21:17:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2109999999999998e-06 2024-02-19T21:17:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.477 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:17:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:35.234247+0000", 2024-02-19T21:17:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0179961399999999 2024-02-19T21:17:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:36.252243+0000", 2024-02-19T21:17:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00017406999999999999 2024-02-19T21:17:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:36.252417+0000", 2024-02-19T21:17:03.478 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00050735899999999998 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:36.252924+0000", 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0014662639999999999 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:36.254391+0000", 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00087924800000000001 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:36.255270+0000", 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6496999999999998e-05 2024-02-19T21:17:03.479 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:36.255326+0000", 2024-02-19T21:17:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0211542709999999 2024-02-19T21:17:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.25753.0:4988 200.14 200:28000000:test-rados-api-smithi163-83928-48:::head [pgnls start_epoch 358] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e358)", 2024-02-19T21:17:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:08:58.305164+0000", 2024-02-19T21:17:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 485.02846956100001, 2024-02-19T21:17:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.181372378, 2024-02-19T21:17:03.480 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.25753", 2024-02-19T21:17:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2605960922", 2024-02-19T21:17:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 4988 2024-02-19T21:17:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:58.305164+0000", 2024-02-19T21:17:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.481 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:58.305164+0000", 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:58.305161+0000", 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0298e-05 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:58.305172+0000", 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4639e-05 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:58.305186+0000", 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1809999999999998e-06 2024-02-19T21:17:03.482 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:58.305192+0000", 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.4571000000000003e-05 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:58.305247+0000", 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.8070000000000001e-05 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:58.305265+0000", 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.18112408099999999 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.483 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:58.486389+0000", 2024-02-19T21:17:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2524000000000001e-05 2024-02-19T21:17:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:58.486432+0000", 2024-02-19T21:17:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010487199999999999 2024-02-19T21:17:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:58.486537+0000", 2024-02-19T21:17:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.181372378 2024-02-19T21:17:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.484 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.14523.0:765 3.0 3:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e617)", 2024-02-19T21:17:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:14:15.555424+0000", 2024-02-19T21:17:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 167.778209734, 2024-02-19T21:17:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.83413820400000005, 2024-02-19T21:17:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.14523", 2024-02-19T21:17:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.193:0/130770101", 2024-02-19T21:17:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 765 2024-02-19T21:17:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.485 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:15.555424+0000", 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:15.555424+0000", 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999938 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:15.555418+0000", 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3938999999999999e-05 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.486 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:15.555442+0000", 2024-02-19T21:17:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2002999999999998e-05 2024-02-19T21:17:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:15.555474+0000", 2024-02-19T21:17:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5475999999999999e-05 2024-02-19T21:17:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:15.555489+0000", 2024-02-19T21:17:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015747499999999999 2024-02-19T21:17:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.487 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:15.555647+0000", 2024-02-19T21:17:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3651e-05 2024-02-19T21:17:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:17:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:15.555661+0000", 2024-02-19T21:17:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.83362766300000002 2024-02-19T21:17:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:16.389288+0000", 2024-02-19T21:17:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013237399999999999 2024-02-19T21:17:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.488 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:16.389421+0000", 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00014162300000000001 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:16.389562+0000", 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.83413820400000005 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:725 240.0 240:00000000::::head [pgnls start_epoch 655] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e655)", 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:14:59.376643+0000", 2024-02-19T21:17:03.489 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 123.956990471, 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0058050530000000001, 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 725 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:59.376643+0000", 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.490 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:59.376643+0000", 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999943 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:59.376637+0000", 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3489000000000002e-05 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:59.376661+0000", 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0221000000000001e-05 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:59.376691+0000", 2024-02-19T21:17:03.491 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2396e-05 2024-02-19T21:17:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:59.376703+0000", 2024-02-19T21:17:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00057456100000000002 2024-02-19T21:17:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:59.377278+0000", 2024-02-19T21:17:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.135e-06 2024-02-19T21:17:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op must wait for map", 2024-02-19T21:17:03.492 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:59.377283+0000", 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00491692 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:59.382200+0000", 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8720000000000002e-05 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:59.382239+0000", 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00020953200000000001 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:14:59.382448+0000", 2024-02-19T21:17:03.493 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0058050530000000001 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(osd.4.0:666 249.6 e736/728 249:60000000:.ceph-internal::hit_set_249.6_archive_2024-02-19 21%3a16%3a26.415228Z_2024-02-19 21%3a16%3a29.422295Z:head v 736'8, has_updated_hit_set_history, mlcod=736'6)", 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:29.422693+0000", 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.910940338000003, 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.048948821000000003, 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422693+0000", 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.494 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422693+0000", 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422691+0000", 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.6351e-05 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422707+0000", 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.3419999999999999e-05 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.495 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422730+0000", 2024-02-19T21:17:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7739999999999999e-06 2024-02-19T21:17:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422740+0000", 2024-02-19T21:17:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8242999999999999e-05 2024-02-19T21:17:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422788+0000", 2024-02-19T21:17:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9782e-05 2024-02-19T21:17:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.496 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422818+0000", 2024-02-19T21:17:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.048773661000000003 2024-02-19T21:17:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:17:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.471592+0000", 2024-02-19T21:17:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0441e-05 2024-02-19T21:17:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.471642+0000", 2024-02-19T21:17:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.048948821000000003 2024-02-19T21:17:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.497 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.23517.0:2010 249.6 e736/728 249:602f83fe:::foo:head v 736'9, mlcod=736'6)", 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:29.422991+0000", 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 33.910642582999998, 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.049576555000000001, 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422991+0000", 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.498 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422991+0000", 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.422990+0000", 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3649999999999998e-05 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.423024+0000", 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2048000000000001e-05 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.423036+0000", 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1709999999999999e-06 2024-02-19T21:17:03.499 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.423042+0000", 2024-02-19T21:17:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1859000000000001e-05 2024-02-19T21:17:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.423074+0000", 2024-02-19T21:17:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9882999999999999e-05 2024-02-19T21:17:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.423094+0000", 2024-02-19T21:17:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.049442709000000001 2024-02-19T21:17:03.500 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.472536+0000", 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.137e-05 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:29.472568+0000", 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.049576555000000001 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPushReply(4.12 741/740 [PushReplyOp(4:4f78e9a2:gc::gc.8:head)])", 2024-02-19T21:17:03.501 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:37.822674+0000", 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 25.510959838000002, 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039957539999999998, 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.822674+0000", 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.822674+0000", 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.502 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.822673+0000", 2024-02-19T21:17:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.0569999999999998e-06 2024-02-19T21:17:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.822679+0000", 2024-02-19T21:17:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5290000000000008e-06 2024-02-19T21:17:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.822687+0000", 2024-02-19T21:17:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8520000000000003e-06 2024-02-19T21:17:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.503 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.822692+0000", 2024-02-19T21:17:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0037310659999999999 2024-02-19T21:17:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.826423+0000", 2024-02-19T21:17:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00024641799999999998 2024-02-19T21:17:03.504 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.826670+0000", 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039957539999999998 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPushReply(4.12 741/740 [PushReplyOp(4:48f827b1:::obj_delete_at_hint.0000000064:head)])", 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:37.913022+0000", 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 25.420611844, 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0040428490000000003, 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:17:03.505 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.913022+0000", 2024-02-19T21:17:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.913022+0000", 2024-02-19T21:17:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:17:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.506 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.913021+0000", 2024-02-19T21:17:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8449999999999999e-06 2024-02-19T21:17:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.913027+0000", 2024-02-19T21:17:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.234e-06 2024-02-19T21:17:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.913036+0000", 2024-02-19T21:17:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6090000000000003e-06 2024-02-19T21:17:03.507 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.913040+0000", 2024-02-19T21:17:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0037845159999999999 2024-02-19T21:17:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.916825+0000", 2024-02-19T21:17:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00023976100000000001 2024-02-19T21:17:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:37.917065+0000", 2024-02-19T21:17:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0040428490000000003 2024-02-19T21:17:03.508 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPushReply(4.12 741/740 [PushReplyOp(4:4a2e2f65:::obj_delete_at_hint.0000000071:head)])", 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:38.003457+0000", 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 25.330176336000001, 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039292370000000004, 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.003457+0000", 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.509 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.003457+0000", 2024-02-19T21:17:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:17:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.003455+0000", 2024-02-19T21:17:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.8520000000000004e-06 2024-02-19T21:17:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.003463+0000", 2024-02-19T21:17:03.510 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2490000000000001e-05 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.003476+0000", 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.277e-06 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.003483+0000", 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003736114 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.007219+0000", 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016749700000000001 2024-02-19T21:17:03.511 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.007387+0000", 2024-02-19T21:17:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039292370000000004 2024-02-19T21:17:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPush(4.7 741/740 [PushOp(4:e47c9322:gc::gc.29:head, version: 86'17, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:e47c9322:gc::gc.29:head@86'17, size: 110, copy_subset: [0~110], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:110, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-19T21:17:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:38.157370+0000", 2024-02-19T21:17:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 25.176263937000002, 2024-02-19T21:17:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.086150854999999998, 2024-02-19T21:17:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.512 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.157370+0000", 2024-02-19T21:17:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.157370+0000", 2024-02-19T21:17:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:17:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.513 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.157369+0000", 2024-02-19T21:17:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3461999999999999e-05 2024-02-19T21:17:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.157382+0000", 2024-02-19T21:17:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5838e-05 2024-02-19T21:17:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.157398+0000", 2024-02-19T21:17:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.7860000000000008e-06 2024-02-19T21:17:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.514 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.157408+0000", 2024-02-19T21:17:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085829894000000004 2024-02-19T21:17:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.243238+0000", 2024-02-19T21:17:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3440000000000003e-06 2024-02-19T21:17:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.243245+0000", 2024-02-19T21:17:03.515 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00027562099999999998 2024-02-19T21:17:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.243521+0000", 2024-02-19T21:17:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.086150854999999998 2024-02-19T21:17:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPush(4.7 742/740 [PushOp(4:e1cdb60d:reshard::reshard.0000000008:head, version: 459'20, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:e1cdb60d:reshard::reshard.0000000008:head@459'20, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-02-19T21:17:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:38.931128+0000", 2024-02-19T21:17:03.516 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 24.402505888, 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.086212765999999996, 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.931128+0000", 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.931128+0000", 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.517 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.931126+0000", 2024-02-19T21:17:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3301e-05 2024-02-19T21:17:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.931139+0000", 2024-02-19T21:17:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3926e-05 2024-02-19T21:17:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.518 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.931153+0000", 2024-02-19T21:17:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.2090000000000005e-06 2024-02-19T21:17:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.931162+0000", 2024-02-19T21:17:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.085893785 2024-02-19T21:17:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.519 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:39.017055+0000", 2024-02-19T21:17:03.520 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4790000000000001e-06 2024-02-19T21:17:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:39.017062+0000", 2024-02-19T21:17:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00027873600000000002 2024-02-19T21:17:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.521 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:39.017341+0000", 2024-02-19T21:17:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.086212765999999996 2024-02-19T21:17:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.522 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:2410 251.0 251:00000000:test-rados-api-smithi163-83928-135:::head [pgnls start_epoch 759] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e759)", 2024-02-19T21:17:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:58.451895+0000", 2024-02-19T21:17:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.8817387669999999, 2024-02-19T21:17:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.114734395, 2024-02-19T21:17:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:17:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:17:03.523 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 2410 2024-02-19T21:17:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.451895+0000", 2024-02-19T21:17:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.524 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.451895+0000", 2024-02-19T21:17:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999962 2024-02-19T21:17:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.451891+0000", 2024-02-19T21:17:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5427999999999999e-05 2024-02-19T21:17:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.525 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.451907+0000", 2024-02-19T21:17:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0344999999999999e-05 2024-02-19T21:17:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.451927+0000", 2024-02-19T21:17:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.5280000000000003e-06 2024-02-19T21:17:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.526 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.451935+0000", 2024-02-19T21:17:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1401000000000002e-05 2024-02-19T21:17:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.452017+0000", 2024-02-19T21:17:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.9769999999999995e-06 2024-02-19T21:17:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.527 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:17:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.452027+0000", 2024-02-19T21:17:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.113902976 2024-02-19T21:17:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.565930+0000", 2024-02-19T21:17:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6086000000000001e-05 2024-02-19T21:17:03.528 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.565956+0000", 2024-02-19T21:17:03.529 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00067344400000000004 2024-02-19T21:17:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.566629+0000", 2024-02-19T21:17:03.530 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.114734395 2024-02-19T21:17:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.531 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPushReply(4.2 761/760 [PushReplyOp(4:46cb1ec1:reshard::reshard.0000000003:head)])", 2024-02-19T21:17:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:58.649675+0000", 2024-02-19T21:17:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.6839591839999999, 2024-02-19T21:17:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039182519999999997, 2024-02-19T21:17:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:17:03.532 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.649675+0000", 2024-02-19T21:17:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.649675+0000", 2024-02-19T21:17:03.533 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:17:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.649674+0000", 2024-02-19T21:17:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.9230000000000001e-06 2024-02-19T21:17:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.649679+0000", 2024-02-19T21:17:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.9519999999999994e-06 2024-02-19T21:17:03.534 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.649686+0000", 2024-02-19T21:17:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.2470000000000002e-06 2024-02-19T21:17:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.649691+0000", 2024-02-19T21:17:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0037759400000000002 2024-02-19T21:17:03.535 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.653467+0000", 2024-02-19T21:17:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012616799999999999 2024-02-19T21:17:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.653593+0000", 2024-02-19T21:17:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0039182519999999997 2024-02-19T21:17:03.536 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDPGPushReply(4.2 761/760 [PushReplyOp(4:4798f7e9:::obj_delete_at_hint.0000000007:head)])", 2024-02-19T21:17:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:58.740051+0000", 2024-02-19T21:17:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 4.5935826100000003, 2024-02-19T21:17:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003945647, 2024-02-19T21:17:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:03.537 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:17:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.740051+0000", 2024-02-19T21:17:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.740051+0000", 2024-02-19T21:17:03.538 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:17:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.740050+0000", 2024-02-19T21:17:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2789999999999998e-06 2024-02-19T21:17:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.740055+0000", 2024-02-19T21:17:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0389999999999993e-06 2024-02-19T21:17:03.539 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.740064+0000", 2024-02-19T21:17:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4800000000000003e-06 2024-02-19T21:17:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.740069+0000", 2024-02-19T21:17:03.540 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0037985559999999998 2024-02-19T21:17:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.743867+0000", 2024-02-19T21:17:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00012933899999999999 2024-02-19T21:17:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:58.743997+0000", 2024-02-19T21:17:03.541 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003945647 2024-02-19T21:17:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:03.542 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:03.542 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:17:03.722 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:17:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:17:03] "GET /metrics HTTP/1.1" 200 192620 "" "Prometheus/2.33.4" 2024-02-19T21:17:03.723 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:01.814027+0000 osd.6 (osd.6) 271 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:17:03.723 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:01.816908+0000 osd.6 (osd.6) 272 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:17:03.723 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:02.535111+0000 osd.3 (osd.3) 321 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:03.723 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:02.535297+0000 osd.3 (osd.3) 322 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:03.723 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:02.568751+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e765: 8 total, 8 up, 6 in 2024-02-19T21:17:03.723 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:02.997821+0000 osd.3 (osd.3) 323 : cluster [DBG] 2.1b scrub starts 2024-02-19T21:17:03.723 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:02.998621+0000 osd.3 (osd.3) 324 : cluster [DBG] 2.1b scrub ok 2024-02-19T21:17:03.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:01.814027+0000 osd.6 (osd.6) 271 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:17:03.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:01.816908+0000 osd.6 (osd.6) 272 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:17:03.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:02.535111+0000 osd.3 (osd.3) 321 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:03.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:02.535297+0000 osd.3 (osd.3) 322 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:03.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:02.568751+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e765: 8 total, 8 up, 6 in 2024-02-19T21:17:03.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:02.997821+0000 osd.3 (osd.3) 323 : cluster [DBG] 2.1b scrub starts 2024-02-19T21:17:03.724 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:02.998621+0000 osd.3 (osd.3) 324 : cluster [DBG] 2.1b scrub ok 2024-02-19T21:17:04.028 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-19T21:17:04.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:01.814027+0000 osd.6 (osd.6) 271 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:17:04.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:01.816908+0000 osd.6 (osd.6) 272 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:17:04.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:02.535111+0000 osd.3 (osd.3) 321 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:04.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:02.535297+0000 osd.3 (osd.3) 322 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:04.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:02.568751+0000 mon.a (mon.0) 3621 : cluster [DBG] osdmap e765: 8 total, 8 up, 6 in 2024-02-19T21:17:04.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:02.997821+0000 osd.3 (osd.3) 323 : cluster [DBG] 2.1b scrub starts 2024-02-19T21:17:04.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:02.998621+0000 osd.3 (osd.3) 324 : cluster [DBG] 2.1b scrub ok 2024-02-19T21:17:04.343 INFO:tasks.thrashosds.thrasher:in_osds: [1, 7, 3, 4, 6, 0] out_osds: [2, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:17:04.343 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:17:04.343 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [1, 7, 3, 4, 6, 0] 2024-02-19T21:17:04.344 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd out 7 2024-02-19T21:17:04.376 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:17:04.378 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:04.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:17:04.395 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:04.480 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:17:04.507 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:04.509 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:17:04.550 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:17:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:04.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:17:04.657 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:17:04.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:03.331467+0000 mgr.y (mgr.14550) 638 : cluster [DBG] pgmap v1158: 214 pgs: 1 active+clean+remapped, 213 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 977 KiB/s, 15 objects/s recovering 2024-02-19T21:17:04.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:03.995015+0000 osd.3 (osd.3) 325 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:17:04.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:03.995929+0000 osd.3 (osd.3) 326 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:17:04.684 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:04.410907+0000 mon.a (mon.0) 3622 : cluster [DBG] osdmap e766: 8 total, 8 up, 6 in 2024-02-19T21:17:04.684 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:03.331467+0000 mgr.y (mgr.14550) 638 : cluster [DBG] pgmap v1158: 214 pgs: 1 active+clean+remapped, 213 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 977 KiB/s, 15 objects/s recovering 2024-02-19T21:17:04.685 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:03.995015+0000 osd.3 (osd.3) 325 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:17:04.685 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:03.995929+0000 osd.3 (osd.3) 326 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:17:04.685 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:04.410907+0000 mon.a (mon.0) 3622 : cluster [DBG] osdmap e766: 8 total, 8 up, 6 in 2024-02-19T21:17:04.685 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:04.941 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:17:04.947 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:04.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:17:05.048 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:17:05.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:03.331467+0000 mgr.y (mgr.14550) 638 : cluster [DBG] pgmap v1158: 214 pgs: 1 active+clean+remapped, 213 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 977 KiB/s, 15 objects/s recovering 2024-02-19T21:17:05.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:03.995015+0000 osd.3 (osd.3) 325 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:17:05.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:03.995929+0000 osd.3 (osd.3) 326 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:17:05.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:04.410907+0000 mon.a (mon.0) 3622 : cluster [DBG] osdmap e766: 8 total, 8 up, 6 in 2024-02-19T21:17:05.080 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:05.101 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:17:05.104 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:05.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:17:05.205 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:17:05.234 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:05.358 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:17:05.361 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:05.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:17:05.462 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:05.493 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:05.514 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:05.517 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:05.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:05.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.571+0000 7f663e75e700 1 -- 172.21.15.163:0/2476809685 >> v1:172.21.15.163:6789/0 conn(0x7f6638100770 legacy=0x7f6638102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:05.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.571+0000 7f663e75e700 1 -- 172.21.15.163:0/2476809685 shutdown_connections 2024-02-19T21:17:05.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.571+0000 7f663e75e700 1 -- 172.21.15.163:0/2476809685 >> 172.21.15.163:0/2476809685 conn(0x7f66380f3570 msgr2=0x7f66380f5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:05.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.571+0000 7f663e75e700 1 -- 172.21.15.163:0/2476809685 shutdown_connections 2024-02-19T21:17:05.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.571+0000 7f663e75e700 1 -- 172.21.15.163:0/2476809685 wait complete. 2024-02-19T21:17:05.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.571+0000 7f663e75e700 1 librados: starting msgr at 2024-02-19T21:17:05.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.571+0000 7f663e75e700 1 librados: starting objecter 2024-02-19T21:17:05.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.571+0000 7f663e75e700 1 Processor -- start 2024-02-19T21:17:05.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.571+0000 7f663e75e700 1 -- start start 2024-02-19T21:17:05.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.571+0000 7f663e75e700 1 librados: setting wanted keys 2024-02-19T21:17:05.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.571+0000 7f663e75e700 1 librados: calling monclient init 2024-02-19T21:17:05.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f663e75e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f663806ccf0 con 0x7f6638100770 2024-02-19T21:17:05.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f663e75e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f663806ce80 con 0x7f66381042b0 2024-02-19T21:17:05.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f663e75e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f663811bb70 con 0x7f66380fd970 2024-02-19T21:17:05.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f6637fff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f66380fd970 0x7f66380fc980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:56400/0 (socket says 172.21.15.163:56400) 2024-02-19T21:17:05.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f6637fff700 1 -- 172.21.15.163:0/3829796130 learned_addr learned my addr 172.21.15.163:0/3829796130 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:05.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 10 client.?.objecter ms_handle_connect 0x7f66380fd970 2024-02-19T21:17:05.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:05.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 10 client.?.objecter ms_handle_connect 0x7f6638100770 2024-02-19T21:17:05.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:05.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 10 client.?.objecter ms_handle_connect 0x7f66381042b0 2024-02-19T21:17:05.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:05.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912653346 0 0) 0x7f663811bb70 con 0x7f66380fd970 2024-02-19T21:17:05.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6620003030 con 0x7f66380fd970 2024-02-19T21:17:05.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043145684 0 0) 0x7f663806ccf0 con 0x7f6638100770 2024-02-19T21:17:05.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f663811bb70 con 0x7f6638100770 2024-02-19T21:17:05.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826101377 0 0) 0x7f663806ce80 con 0x7f66381042b0 2024-02-19T21:17:05.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f663806ccf0 con 0x7f66381042b0 2024-02-19T21:17:05.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 362057734 0 0) 0x7f6620003030 con 0x7f66380fd970 2024-02-19T21:17:05.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f663806ce80 con 0x7f66380fd970 2024-02-19T21:17:05.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6628001980 con 0x7f66380fd970 2024-02-19T21:17:05.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2673384985 0 0) 0x7f663811bb70 con 0x7f6638100770 2024-02-19T21:17:05.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6620003030 con 0x7f6638100770 2024-02-19T21:17:05.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f662c001e50 con 0x7f6638100770 2024-02-19T21:17:05.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1089770396 0 0) 0x7f663806ccf0 con 0x7f66381042b0 2024-02-19T21:17:05.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f663811bb70 con 0x7f66381042b0 2024-02-19T21:17:05.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6630001ba0 con 0x7f66381042b0 2024-02-19T21:17:05.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1047500461 0 0) 0x7f663806ce80 con 0x7f66380fd970 2024-02-19T21:17:05.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 >> v1:172.21.15.193:6789/0 conn(0x7f66381042b0 legacy=0x7f6638118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:05.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 >> v1:172.21.15.163:6789/0 conn(0x7f6638100770 legacy=0x7f6638114d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:05.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663811cd10 con 0x7f66380fd970 2024-02-19T21:17:05.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.572+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663811bd60 con 0x7f66380fd970 2024-02-19T21:17:05.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.573+0000 7f663e75e700 10 client.50585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:05.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.573+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f663811c1d0 con 0x7f66380fd970 2024-02-19T21:17:05.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.573+0000 7f663e75e700 1 librados: init done 2024-02-19T21:17:05.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.573+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6628001d10 con 0x7f66380fd970 2024-02-19T21:17:05.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.573+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f6638060370 con 0x7f66380fd970 2024-02-19T21:17:05.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.573+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6628005b50 con 0x7f66380fd970 2024-02-19T21:17:05.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.574+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6628018a60 con 0x7f66380fd970 2024-02-19T21:17:05.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.574+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(766..766 src has 623..766) v4 ==== 8417+0+0 (unknown 3089593773 0 0) 0x7f6628079db0 con 0x7f66380fd970 2024-02-19T21:17:05.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.574+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 osd_map(766..766 src has 623..766) v4 2024-02-19T21:17:05.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.574+0000 7f66357fa700 3 client.50585.objecter handle_osd_map got epochs [766,766] > 0 2024-02-19T21:17:05.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.574+0000 7f66357fa700 3 client.50585.objecter handle_osd_map decoding full epoch 766 2024-02-19T21:17:05.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.574+0000 7f66357fa700 20 client.50585.objecter dump_active .. 0 homeless 2024-02-19T21:17:05.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.574+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v766) v1 ==== 54+0+16 (unknown 2780347667 0 2420000192) 0x7f66280028b0 con 0x7f66380fd970 2024-02-19T21:17:05.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.575+0000 7f66357fa700 10 client.50585.objecter ms_handle_connect 0x7f6620063c70 2024-02-19T21:17:05.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.575+0000 7f663e75e700 10 client.50585.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:05.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.575+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 1 2024-02-19T21:17:05.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.575+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638047540 osd=0 3 2024-02-19T21:17:05.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.575+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638047540 osd=0 3 2024-02-19T21:17:05.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.575+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638047540 osd=0 3 2024-02-19T21:17:05.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.575+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 1 2024-02-19T21:17:05.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.575+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f6638047540 osd=0 3 2024-02-19T21:17:05.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.575+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 0 1 2024-02-19T21:17:05.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.575+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638047540 osd=0 4 2024-02-19T21:17:05.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.575+0000 7f663e75e700 10 client.50585.objecter _send_command 1 2024-02-19T21:17:05.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.575+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6638109fb0 con 0x7f66380f3950 2024-02-19T21:17:05.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.575+0000 7f66357fa700 10 client.50585.objecter ms_handle_connect 0x7f66380f3950 2024-02-19T21:17:05.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.576+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6638109fb0 con 0x7f66380f3950 2024-02-19T21:17:05.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.576+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:17:05.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.576+0000 7f66357fa700 10 client.50585.objecter _finish_command 1 = system:0 2024-02-19T21:17:05.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.576+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f6638047540 osd=0 4 2024-02-19T21:17:05.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.576+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 0 1 2024-02-19T21:17:05.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.593+0000 7f663e75e700 10 client.50585.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:05.594 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.593+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 2 2024-02-19T21:17:05.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.593+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638047540 osd=0 3 2024-02-19T21:17:05.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.593+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638047540 osd=0 3 2024-02-19T21:17:05.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.593+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638047540 osd=0 3 2024-02-19T21:17:05.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.593+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 2 2024-02-19T21:17:05.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.593+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f6638047540 osd=0 3 2024-02-19T21:17:05.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.593+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 0 2 2024-02-19T21:17:05.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.593+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638047540 osd=0 4 2024-02-19T21:17:05.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.593+0000 7f663e75e700 10 client.50585.objecter _send_command 2 2024-02-19T21:17:05.595 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.593+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6638028610 con 0x7f66380f3950 2024-02-19T21:17:05.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6638028610 con 0x7f66380f3950 2024-02-19T21:17:05.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:05.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f66357fa700 10 client.50585.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f6638047540 osd=0 4 2024-02-19T21:17:05.597 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 0 2 2024-02-19T21:17:05.597 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:17:05.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f663e75e700 10 client.50585.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:05.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 3 2024-02-19T21:17:05.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f66380287e0 osd=1 3 2024-02-19T21:17:05.598 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f66380287e0 osd=1 3 2024-02-19T21:17:05.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f66380287e0 osd=1 3 2024-02-19T21:17:05.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 3 2024-02-19T21:17:05.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f66380287e0 osd=1 3 2024-02-19T21:17:05.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 1 3 2024-02-19T21:17:05.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f66380287e0 osd=1 4 2024-02-19T21:17:05.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f663e75e700 10 client.50585.objecter _send_command 3 2024-02-19T21:17:05.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6638120b40 con 0x7f66380f4680 2024-02-19T21:17:05.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.595+0000 7f66357fa700 10 client.50585.objecter ms_handle_connect 0x7f66380f4680 2024-02-19T21:17:05.599 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.596+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6638120b40 con 0x7f66380f4680 2024-02-19T21:17:05.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.596+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:17:05.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.596+0000 7f66357fa700 10 client.50585.objecter _finish_command 3 = system:0 2024-02-19T21:17:05.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.596+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f66380287e0 osd=1 4 2024-02-19T21:17:05.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.596+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 1 3 2024-02-19T21:17:05.611 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.610+0000 7f663e75e700 10 client.50585.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:05.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.610+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 4 2024-02-19T21:17:05.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.610+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f66380287e0 osd=1 3 2024-02-19T21:17:05.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.610+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f66380287e0 osd=1 3 2024-02-19T21:17:05.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.610+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f66380287e0 osd=1 3 2024-02-19T21:17:05.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.610+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 4 2024-02-19T21:17:05.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.610+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f66380287e0 osd=1 3 2024-02-19T21:17:05.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.610+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 1 4 2024-02-19T21:17:05.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.610+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f66380287e0 osd=1 4 2024-02-19T21:17:05.612 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.610+0000 7f663e75e700 10 client.50585.objecter _send_command 4 2024-02-19T21:17:05.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.610+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f663811c6c0 con 0x7f66380f4680 2024-02-19T21:17:05.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f663811c6c0 con 0x7f66380f4680 2024-02-19T21:17:05.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:05.613 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f66357fa700 10 client.50585.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f66380287e0 osd=1 4 2024-02-19T21:17:05.614 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 1 4 2024-02-19T21:17:05.614 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:17:05.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f663e75e700 10 client.50585.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:05.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 5 2024-02-19T21:17:05.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f663811cad0 osd=2 3 2024-02-19T21:17:05.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f663811cad0 osd=2 3 2024-02-19T21:17:05.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f663811cad0 osd=2 3 2024-02-19T21:17:05.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 5 2024-02-19T21:17:05.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f663811cad0 osd=2 3 2024-02-19T21:17:05.615 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 2 5 2024-02-19T21:17:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f663811cad0 osd=2 4 2024-02-19T21:17:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f663e75e700 10 client.50585.objecter _send_command 5 2024-02-19T21:17:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6638125560 con 0x7f6638059e00 2024-02-19T21:17:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.612+0000 7f66357fa700 10 client.50585.objecter ms_handle_connect 0x7f6638059e00 2024-02-19T21:17:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.613+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6638125560 con 0x7f6638059e00 2024-02-19T21:17:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.613+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:17:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.613+0000 7f66357fa700 10 client.50585.objecter _finish_command 5 = system:0 2024-02-19T21:17:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.613+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f663811cad0 osd=2 4 2024-02-19T21:17:05.616 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.613+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 2 5 2024-02-19T21:17:05.618 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:17:05.628 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.627+0000 7f663e75e700 10 client.50585.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:05.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.627+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 6 2024-02-19T21:17:05.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.627+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f663811cad0 osd=2 3 2024-02-19T21:17:05.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.627+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f663811cad0 osd=2 3 2024-02-19T21:17:05.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.627+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f663811cad0 osd=2 3 2024-02-19T21:17:05.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.627+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 6 2024-02-19T21:17:05.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.627+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f663811cad0 osd=2 3 2024-02-19T21:17:05.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.627+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 2 6 2024-02-19T21:17:05.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.627+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f663811cad0 osd=2 4 2024-02-19T21:17:05.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.627+0000 7f663e75e700 10 client.50585.objecter _send_command 6 2024-02-19T21:17:05.629 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.627+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6638059360 con 0x7f6638059e00 2024-02-19T21:17:05.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6638059360 con 0x7f6638059e00 2024-02-19T21:17:05.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:05.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f66357fa700 10 client.50585.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f663811cad0 osd=2 4 2024-02-19T21:17:05.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 2 6 2024-02-19T21:17:05.632 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:17:05.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f663e75e700 10 client.50585.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 7 2024-02-19T21:17:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638059710 osd=3 3 2024-02-19T21:17:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638059710 osd=3 3 2024-02-19T21:17:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638059710 osd=3 3 2024-02-19T21:17:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 7 2024-02-19T21:17:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f6638059710 osd=3 3 2024-02-19T21:17:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 3 7 2024-02-19T21:17:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638059710 osd=3 4 2024-02-19T21:17:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f663e75e700 10 client.50585.objecter _send_command 7 2024-02-19T21:17:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.629+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f663812a020 con 0x7f6638121990 2024-02-19T21:17:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.630+0000 7f66357fa700 10 client.50585.objecter ms_handle_connect 0x7f6638121990 2024-02-19T21:17:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.630+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f663812a020 con 0x7f6638121990 2024-02-19T21:17:05.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.630+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:17:05.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.630+0000 7f66357fa700 10 client.50585.objecter _finish_command 7 = system:0 2024-02-19T21:17:05.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.630+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f6638059710 osd=3 4 2024-02-19T21:17:05.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.630+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 3 7 2024-02-19T21:17:05.646 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:05.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.645+0000 7f663e75e700 10 client.50585.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:05.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.645+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 8 2024-02-19T21:17:05.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.645+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638059710 osd=3 3 2024-02-19T21:17:05.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.645+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638059710 osd=3 3 2024-02-19T21:17:05.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.645+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638059710 osd=3 3 2024-02-19T21:17:05.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.645+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 8 2024-02-19T21:17:05.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.645+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f6638059710 osd=3 3 2024-02-19T21:17:05.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.645+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 3 8 2024-02-19T21:17:05.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.645+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638059710 osd=3 4 2024-02-19T21:17:05.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.645+0000 7f663e75e700 10 client.50585.objecter _send_command 8 2024-02-19T21:17:05.647 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.645+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6638120ef0 con 0x7f6638121990 2024-02-19T21:17:05.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6638120ef0 con 0x7f6638121990 2024-02-19T21:17:05.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:05.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f66357fa700 10 client.50585.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f6638059710 osd=3 4 2024-02-19T21:17:05.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 3 8 2024-02-19T21:17:05.649 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:17:05.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f663e75e700 10 client.50585.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:05.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 9 2024-02-19T21:17:05.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638120ef0 osd=4 3 2024-02-19T21:17:05.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638120ef0 osd=4 3 2024-02-19T21:17:05.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638120ef0 osd=4 3 2024-02-19T21:17:05.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 9 2024-02-19T21:17:05.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f6638120ef0 osd=4 3 2024-02-19T21:17:05.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 4 9 2024-02-19T21:17:05.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638120ef0 osd=4 4 2024-02-19T21:17:05.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f663e75e700 10 client.50585.objecter _send_command 9 2024-02-19T21:17:05.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.647+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6638126570 con 0x7f66381210a0 2024-02-19T21:17:05.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.648+0000 7f66357fa700 10 client.50585.objecter ms_handle_connect 0x7f66381210a0 2024-02-19T21:17:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.649+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6638126570 con 0x7f66381210a0 2024-02-19T21:17:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.649+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:17:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.649+0000 7f66357fa700 10 client.50585.objecter _finish_command 9 = system:0 2024-02-19T21:17:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.649+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f6638120ef0 osd=4 4 2024-02-19T21:17:05.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.649+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 4 9 2024-02-19T21:17:05.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.664+0000 7f663e75e700 10 client.50585.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:05.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.664+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 10 2024-02-19T21:17:05.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.664+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638120ef0 osd=4 3 2024-02-19T21:17:05.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.664+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638120ef0 osd=4 3 2024-02-19T21:17:05.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.664+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638120ef0 osd=4 3 2024-02-19T21:17:05.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.664+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 10 2024-02-19T21:17:05.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.664+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f6638120ef0 osd=4 3 2024-02-19T21:17:05.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.664+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 4 10 2024-02-19T21:17:05.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.664+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638120ef0 osd=4 4 2024-02-19T21:17:05.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.664+0000 7f663e75e700 10 client.50585.objecter _send_command 10 2024-02-19T21:17:05.666 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.664+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6638126570 con 0x7f66381210a0 2024-02-19T21:17:05.667 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:17:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6638126570 con 0x7f66381210a0 2024-02-19T21:17:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f66357fa700 10 client.50585.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f6638120ef0 osd=4 4 2024-02-19T21:17:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 4 10 2024-02-19T21:17:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f663e75e700 10 client.50585.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:05.668 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 11 2024-02-19T21:17:05.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638126570 osd=5 3 2024-02-19T21:17:05.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638126570 osd=5 3 2024-02-19T21:17:05.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638126570 osd=5 3 2024-02-19T21:17:05.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 11 2024-02-19T21:17:05.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f6638126570 osd=5 3 2024-02-19T21:17:05.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 5 11 2024-02-19T21:17:05.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638126570 osd=5 4 2024-02-19T21:17:05.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f663e75e700 10 client.50585.objecter _send_command 11 2024-02-19T21:17:05.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.666+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6638132ae0 con 0x7f663812ee90 2024-02-19T21:17:05.669 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.667+0000 7f66357fa700 10 client.50585.objecter ms_handle_connect 0x7f663812ee90 2024-02-19T21:17:05.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.668+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6638132ae0 con 0x7f663812ee90 2024-02-19T21:17:05.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.668+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:17:05.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.668+0000 7f66357fa700 10 client.50585.objecter _finish_command 11 = system:0 2024-02-19T21:17:05.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.668+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f6638126570 osd=5 4 2024-02-19T21:17:05.670 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.668+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 5 11 2024-02-19T21:17:05.672 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:17:05.675 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:05.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:17:05.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.683+0000 7f663e75e700 10 client.50585.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:05.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.683+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 12 2024-02-19T21:17:05.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.683+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638126570 osd=5 3 2024-02-19T21:17:05.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.683+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638126570 osd=5 3 2024-02-19T21:17:05.684 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.683+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638126570 osd=5 3 2024-02-19T21:17:05.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.683+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 12 2024-02-19T21:17:05.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.683+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f6638126570 osd=5 3 2024-02-19T21:17:05.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.683+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 5 12 2024-02-19T21:17:05.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.683+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638126570 osd=5 4 2024-02-19T21:17:05.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.683+0000 7f663e75e700 10 client.50585.objecter _send_command 12 2024-02-19T21:17:05.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.683+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6638132ae0 con 0x7f663812ee90 2024-02-19T21:17:05.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.684+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6638132ae0 con 0x7f663812ee90 2024-02-19T21:17:05.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.684+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:05.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.684+0000 7f66357fa700 10 client.50585.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.684+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f6638126570 osd=5 4 2024-02-19T21:17:05.686 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.684+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 5 12 2024-02-19T21:17:05.686 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:17:05.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.685+0000 7f663e75e700 10 client.50585.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:05.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.685+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 13 2024-02-19T21:17:05.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.685+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638132ae0 osd=6 3 2024-02-19T21:17:05.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.685+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638132ae0 osd=6 3 2024-02-19T21:17:05.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.685+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638132ae0 osd=6 3 2024-02-19T21:17:05.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.685+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 13 2024-02-19T21:17:05.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.685+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f6638132ae0 osd=6 3 2024-02-19T21:17:05.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.685+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 6 13 2024-02-19T21:17:05.687 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.685+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638132ae0 osd=6 4 2024-02-19T21:17:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.685+0000 7f663e75e700 10 client.50585.objecter _send_command 13 2024-02-19T21:17:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.685+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6638133d30 con 0x7f66381347e0 2024-02-19T21:17:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.685+0000 7f66357fa700 10 client.50585.objecter ms_handle_connect 0x7f66381347e0 2024-02-19T21:17:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.686+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f6638133d30 con 0x7f66381347e0 2024-02-19T21:17:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.686+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:17:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.686+0000 7f66357fa700 10 client.50585.objecter _finish_command 13 = system:0 2024-02-19T21:17:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.686+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f6638132ae0 osd=6 4 2024-02-19T21:17:05.688 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.686+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 6 13 2024-02-19T21:17:05.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.700+0000 7f663e75e700 10 client.50585.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:05.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.700+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 14 2024-02-19T21:17:05.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.700+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638132ae0 osd=6 3 2024-02-19T21:17:05.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.700+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638132ae0 osd=6 3 2024-02-19T21:17:05.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.700+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638132ae0 osd=6 3 2024-02-19T21:17:05.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.700+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 14 2024-02-19T21:17:05.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.700+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f6638132ae0 osd=6 3 2024-02-19T21:17:05.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.700+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 6 14 2024-02-19T21:17:05.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.700+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638132ae0 osd=6 4 2024-02-19T21:17:05.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.700+0000 7f663e75e700 10 client.50585.objecter _send_command 14 2024-02-19T21:17:05.702 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.700+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f6638133d30 con 0x7f66381347e0 2024-02-19T21:17:05.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f6638133d30 con 0x7f66381347e0 2024-02-19T21:17:05.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:05.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f66357fa700 10 client.50585.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f6638132ae0 osd=6 4 2024-02-19T21:17:05.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 6 14 2024-02-19T21:17:05.704 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:17:05.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f663e75e700 10 client.50585.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:05.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 15 2024-02-19T21:17:05.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638133d30 osd=7 3 2024-02-19T21:17:05.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638133d30 osd=7 3 2024-02-19T21:17:05.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638133d30 osd=7 3 2024-02-19T21:17:05.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 15 2024-02-19T21:17:05.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f6638133d30 osd=7 3 2024-02-19T21:17:05.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 7 15 2024-02-19T21:17:05.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638133d30 osd=7 4 2024-02-19T21:17:05.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f663e75e700 10 client.50585.objecter _send_command 15 2024-02-19T21:17:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.702+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f663813b4b0 con 0x7f66381379a0 2024-02-19T21:17:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.703+0000 7f66357fa700 10 client.50585.objecter ms_handle_connect 0x7f66381379a0 2024-02-19T21:17:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.704+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f663813b4b0 con 0x7f66381379a0 2024-02-19T21:17:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.704+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:17:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.704+0000 7f66357fa700 10 client.50585.objecter _finish_command 15 = system:0 2024-02-19T21:17:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.704+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f6638133d30 osd=7 4 2024-02-19T21:17:05.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.704+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 7 15 2024-02-19T21:17:05.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.718+0000 7f663e75e700 10 client.50585.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:05.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.718+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign -1 16 2024-02-19T21:17:05.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.718+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638133d30 osd=7 3 2024-02-19T21:17:05.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.718+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638133d30 osd=7 3 2024-02-19T21:17:05.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.718+0000 7f663e75e700 20 client.50585.objecter _get_session s=0x7f6638133d30 osd=7 3 2024-02-19T21:17:05.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.718+0000 7f663e75e700 15 client.50585.objecter _session_command_op_remove -1 16 2024-02-19T21:17:05.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.718+0000 7f663e75e700 20 client.50585.objecter get_session s=0x7f6638133d30 osd=7 3 2024-02-19T21:17:05.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.718+0000 7f663e75e700 15 client.50585.objecter _session_command_op_assign 7 16 2024-02-19T21:17:05.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.718+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638133d30 osd=7 4 2024-02-19T21:17:05.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.718+0000 7f663e75e700 10 client.50585.objecter _send_command 16 2024-02-19T21:17:05.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.718+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f663813b4b0 con 0x7f66381379a0 2024-02-19T21:17:05.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f66357fa700 1 -- 172.21.15.163:0/3829796130 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f663813b4b0 con 0x7f66381379a0 2024-02-19T21:17:05.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f66357fa700 10 client.50585.objecter ms_dispatch 0x7f6638001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:05.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f66357fa700 10 client.50585.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f66357fa700 20 client.50585.objecter put_session s=0x7f6638133d30 osd=7 4 2024-02-19T21:17:05.721 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f66357fa700 15 client.50585.objecter _session_command_op_remove 7 16 2024-02-19T21:17:05.721 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:17:05.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 10 librados: watch_flush enter 2024-02-19T21:17:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 10 librados: watch_flush exit 2024-02-19T21:17:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 10 client.50585.objecter close_session for osd.0 2024-02-19T21:17:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 >> v1:172.21.15.163:6801/1559418115 conn(0x7f66380f3950 legacy=0x7f66380f3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638047540 osd=0 1 2024-02-19T21:17:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 10 client.50585.objecter close_session for osd.1 2024-02-19T21:17:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 >> v1:172.21.15.163:6805/3125047432 conn(0x7f66380f4680 legacy=0x7f663811f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f66380287e0 osd=1 1 2024-02-19T21:17:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 10 client.50585.objecter close_session for osd.2 2024-02-19T21:17:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 >> v1:172.21.15.163:6809/1563158487 conn(0x7f6638059e00 legacy=0x7f6638123e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f663811cad0 osd=2 1 2024-02-19T21:17:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 10 client.50585.objecter close_session for osd.3 2024-02-19T21:17:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 >> v1:172.21.15.163:6813/2112279769 conn(0x7f6638121990 legacy=0x7f6638128860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638059710 osd=3 1 2024-02-19T21:17:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 10 client.50585.objecter close_session for osd.4 2024-02-19T21:17:05.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 >> v1:172.21.15.193:6800/3129716965 conn(0x7f66381210a0 legacy=0x7f663812a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638120ef0 osd=4 1 2024-02-19T21:17:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 10 client.50585.objecter close_session for osd.5 2024-02-19T21:17:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 >> v1:172.21.15.193:6804/805161991 conn(0x7f663812ee90 legacy=0x7f6638131280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663ccfb700 1 -- 172.21.15.163:0/3829796130 reap_dead start 2024-02-19T21:17:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638126570 osd=5 1 2024-02-19T21:17:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 10 client.50585.objecter close_session for osd.6 2024-02-19T21:17:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 >> v1:172.21.15.193:6808/787604615 conn(0x7f66381347e0 legacy=0x7f6638134c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663ccfb700 1 -- 172.21.15.163:0/3829796130 reap_dead start 2024-02-19T21:17:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638132ae0 osd=6 1 2024-02-19T21:17:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 10 client.50585.objecter close_session for osd.7 2024-02-19T21:17:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 >> v1:172.21.15.193:6812/157927432 conn(0x7f66381379a0 legacy=0x7f6638139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 20 client.50585.objecter put_session s=0x7f6638133d30 osd=7 1 2024-02-19T21:17:05.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 20 client.50585.objecter shutdown clearing up homeless session... 2024-02-19T21:17:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 10 client.50585.objecter successfully canceled tick 2024-02-19T21:17:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663ccfb700 1 -- 172.21.15.163:0/3829796130 reap_dead start 2024-02-19T21:17:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663ccfb700 1 -- 172.21.15.163:0/3829796130 reap_dead start 2024-02-19T21:17:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6620063c70 legacy=0x7f6620066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 >> v1:172.21.15.163:6790/0 conn(0x7f66380fd970 legacy=0x7f66380fc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 shutdown_connections 2024-02-19T21:17:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 >> 172.21.15.163:0/3829796130 conn(0x7f66380f3570 msgr2=0x7f6638107700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 shutdown_connections 2024-02-19T21:17:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 1 -- 172.21.15.163:0/3829796130 wait complete. 2024-02-19T21:17:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:05.720+0000 7f663e75e700 1 librados: shutdown 2024-02-19T21:17:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.725 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.726 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.726 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.726 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:17:05.776 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:17:05.803 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:05.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:03.863625+0000 osd.6 (osd.6) 273 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:17:05.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:03.864735+0000 osd.6 (osd.6) 274 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:17:05.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:04.343425+0000 osd.1 (osd.1) 340 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:17:05.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:04.344516+0000 osd.1 (osd.1) 341 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:17:05.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:04.961975+0000 osd.3 (osd.3) 327 : cluster [DBG] 2.28 deep-scrub starts 2024-02-19T21:17:05.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:04.962978+0000 osd.3 (osd.3) 328 : cluster [DBG] 2.28 deep-scrub ok 2024-02-19T21:17:05.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:05.221800+0000 osd.4 (osd.4) 269 : cluster [DBG] 2.25 scrub starts 2024-02-19T21:17:05.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:05.222687+0000 osd.4 (osd.4) 270 : cluster [DBG] 2.25 scrub ok 2024-02-19T21:17:05.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:03.863625+0000 osd.6 (osd.6) 273 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:17:05.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:03.864735+0000 osd.6 (osd.6) 274 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:17:05.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:04.343425+0000 osd.1 (osd.1) 340 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:17:05.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:04.344516+0000 osd.1 (osd.1) 341 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:17:05.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:04.961975+0000 osd.3 (osd.3) 327 : cluster [DBG] 2.28 deep-scrub starts 2024-02-19T21:17:05.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:04.962978+0000 osd.3 (osd.3) 328 : cluster [DBG] 2.28 deep-scrub ok 2024-02-19T21:17:05.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:05.221800+0000 osd.4 (osd.4) 269 : cluster [DBG] 2.25 scrub starts 2024-02-19T21:17:05.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:05.222687+0000 osd.4 (osd.4) 270 : cluster [DBG] 2.25 scrub ok 2024-02-19T21:17:05.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:03.863625+0000 osd.6 (osd.6) 273 : cluster [DBG] 6.5 scrub starts 2024-02-19T21:17:06.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:03.864735+0000 osd.6 (osd.6) 274 : cluster [DBG] 6.5 scrub ok 2024-02-19T21:17:06.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:04.343425+0000 osd.1 (osd.1) 340 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:17:06.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:04.344516+0000 osd.1 (osd.1) 341 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:17:06.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:04.961975+0000 osd.3 (osd.3) 327 : cluster [DBG] 2.28 deep-scrub starts 2024-02-19T21:17:06.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:04.962978+0000 osd.3 (osd.3) 328 : cluster [DBG] 2.28 deep-scrub ok 2024-02-19T21:17:06.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:05.221800+0000 osd.4 (osd.4) 269 : cluster [DBG] 2.25 scrub starts 2024-02-19T21:17:06.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:05.222687+0000 osd.4 (osd.4) 270 : cluster [DBG] 2.25 scrub ok 2024-02-19T21:17:06.214 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:06.218 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:06.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:05.305798+0000 osd.1 (osd.1) 342 : cluster [DBG] 16.a scrub starts 2024-02-19T21:17:06.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:05.308982+0000 osd.1 (osd.1) 343 : cluster [DBG] 16.a scrub ok 2024-02-19T21:17:06.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:05.332335+0000 mgr.y (mgr.14550) 639 : cluster [DBG] pgmap v1160: 214 pgs: 214 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:17:06.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:05.575705+0000 mon.c (mon.2) 437 : audit [DBG] from='client.? v1:172.21.15.163:0/3829796130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:17:06.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:05.682838+0000 mon.a (mon.0) 3623 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:06.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:05.305798+0000 osd.1 (osd.1) 342 : cluster [DBG] 16.a scrub starts 2024-02-19T21:17:06.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:05.308982+0000 osd.1 (osd.1) 343 : cluster [DBG] 16.a scrub ok 2024-02-19T21:17:06.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:05.332335+0000 mgr.y (mgr.14550) 639 : cluster [DBG] pgmap v1160: 214 pgs: 214 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:17:06.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:05.575705+0000 mon.c (mon.2) 437 : audit [DBG] from='client.? v1:172.21.15.163:0/3829796130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:17:06.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:05.682838+0000 mon.a (mon.0) 3623 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:07.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:05.305798+0000 osd.1 (osd.1) 342 : cluster [DBG] 16.a scrub starts 2024-02-19T21:17:07.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:05.308982+0000 osd.1 (osd.1) 343 : cluster [DBG] 16.a scrub ok 2024-02-19T21:17:07.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:05.332335+0000 mgr.y (mgr.14550) 639 : cluster [DBG] pgmap v1160: 214 pgs: 214 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:17:07.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:05.575705+0000 mon.c (mon.2) 437 : audit [DBG] from='client.? v1:172.21.15.163:0/3829796130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:17:07.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:05.682838+0000 mon.a (mon.0) 3623 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:07.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:06.316187+0000 osd.1 (osd.1) 344 : cluster [DBG] 16.9 scrub starts 2024-02-19T21:17:07.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:06.319614+0000 osd.1 (osd.1) 345 : cluster [DBG] 16.9 scrub ok 2024-02-19T21:17:07.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:06.583309+0000 mon.a (mon.0) 3624 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:17:07.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:06.583359+0000 mon.a (mon.0) 3625 : cluster [DBG] osdmap e767: 8 total, 8 up, 6 in 2024-02-19T21:17:07.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:06.586718+0000 mon.a (mon.0) 3626 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-134"}]: dispatch 2024-02-19T21:17:07.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:07.581422+0000 mon.a (mon.0) 3627 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:17:08.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:07.590965+0000 mon.a (mon.0) 3628 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-134"}]': finished 2024-02-19T21:17:08.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:07.591031+0000 mon.a (mon.0) 3629 : cluster [DBG] osdmap e768: 8 total, 8 up, 6 in 2024-02-19T21:17:08.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:06.316187+0000 osd.1 (osd.1) 344 : cluster [DBG] 16.9 scrub starts 2024-02-19T21:17:08.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:06.319614+0000 osd.1 (osd.1) 345 : cluster [DBG] 16.9 scrub ok 2024-02-19T21:17:08.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:06.583309+0000 mon.a (mon.0) 3624 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:17:08.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:06.583359+0000 mon.a (mon.0) 3625 : cluster [DBG] osdmap e767: 8 total, 8 up, 6 in 2024-02-19T21:17:08.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:06.586718+0000 mon.a (mon.0) 3626 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-134"}]: dispatch 2024-02-19T21:17:08.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:07.581422+0000 mon.a (mon.0) 3627 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:17:08.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:07.590965+0000 mon.a (mon.0) 3628 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-134"}]': finished 2024-02-19T21:17:08.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:07.591031+0000 mon.a (mon.0) 3629 : cluster [DBG] osdmap e768: 8 total, 8 up, 6 in 2024-02-19T21:17:08.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:17:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:17:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:17:08.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:06.316187+0000 osd.1 (osd.1) 344 : cluster [DBG] 16.9 scrub starts 2024-02-19T21:17:08.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:06.319614+0000 osd.1 (osd.1) 345 : cluster [DBG] 16.9 scrub ok 2024-02-19T21:17:08.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:06.583309+0000 mon.a (mon.0) 3624 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:17:08.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:06.583359+0000 mon.a (mon.0) 3625 : cluster [DBG] osdmap e767: 8 total, 8 up, 6 in 2024-02-19T21:17:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:06.586718+0000 mon.a (mon.0) 3626 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-134"}]: dispatch 2024-02-19T21:17:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:07.581422+0000 mon.a (mon.0) 3627 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:17:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:07.590965+0000 mon.a (mon.0) 3628 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-134"}]': finished 2024-02-19T21:17:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:07.591031+0000 mon.a (mon.0) 3629 : cluster [DBG] osdmap e768: 8 total, 8 up, 6 in 2024-02-19T21:17:08.486 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:17:08.489 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:08.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:17:08.590 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:08.627 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:08.726 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:08.729 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:08.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:08.830 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:08.862 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:08.884 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:08.886 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:08.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:08.988 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:08.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:07.332932+0000 mgr.y (mgr.14550) 640 : cluster [DBG] pgmap v1162: 214 pgs: 214 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 887 B/s rd, 0 op/s 2024-02-19T21:17:08.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:07.612489+0000 mon.a (mon.0) 3630 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:08.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:07.612863+0000 mon.a (mon.0) 3631 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-134"}]: dispatch 2024-02-19T21:17:08.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:07.332932+0000 mgr.y (mgr.14550) 640 : cluster [DBG] pgmap v1162: 214 pgs: 214 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 887 B/s rd, 0 op/s 2024-02-19T21:17:08.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:07.612489+0000 mon.a (mon.0) 3630 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:08.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:07.612863+0000 mon.a (mon.0) 3631 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-134"}]: dispatch 2024-02-19T21:17:09.018 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:09.038 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:09.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:09.142 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:17:09.173 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:09.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.005+0000 7f5b77fff700 1 -- 172.21.15.163:0/457063792 >> v1:172.21.15.163:6790/0 conn(0x7f5b781005d0 legacy=0x7f5b781029c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:09.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.005+0000 7f5b77fff700 1 -- 172.21.15.163:0/457063792 shutdown_connections 2024-02-19T21:17:09.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.005+0000 7f5b77fff700 1 -- 172.21.15.163:0/457063792 >> 172.21.15.163:0/457063792 conn(0x7f5b780f33f0 msgr2=0x7f5b780f5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:09.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.005+0000 7f5b77fff700 1 -- 172.21.15.163:0/457063792 shutdown_connections 2024-02-19T21:17:09.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.005+0000 7f5b77fff700 1 -- 172.21.15.163:0/457063792 wait complete. 2024-02-19T21:17:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.005+0000 7f5b77fff700 1 librados: starting msgr at 2024-02-19T21:17:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.005+0000 7f5b77fff700 1 librados: starting objecter 2024-02-19T21:17:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.005+0000 7f5b77fff700 1 Processor -- start 2024-02-19T21:17:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b77fff700 1 -- start start 2024-02-19T21:17:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b77fff700 1 librados: setting wanted keys 2024-02-19T21:17:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b77fff700 1 librados: calling monclient init 2024-02-19T21:17:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b77fff700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b780fcf20 con 0x7f5b78104110 2024-02-19T21:17:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b77fff700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b780fd0f0 con 0x7f5b781005d0 2024-02-19T21:17:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b77fff700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b7811b9d0 con 0x7f5b780fd7d0 2024-02-19T21:17:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b76ffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f5b780fd7d0 0x7f5b780fc620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:56412/0 (socket says 172.21.15.163:56412) 2024-02-19T21:17:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b76ffd700 1 -- 172.21.15.163:0/2136691392 learned_addr learned my addr 172.21.15.163:0/2136691392 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b777fe700 1 --1- 172.21.15.163:0/2136691392 >> v1:172.21.15.163:6789/0 conn(0x7f5b78104110 0x7f5b78118290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43940/0 (socket says 172.21.15.163:43940) 2024-02-19T21:17:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 10 client.?.objecter ms_handle_connect 0x7f5b780fd7d0 2024-02-19T21:17:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:09.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 10 client.?.objecter ms_handle_connect 0x7f5b78104110 2024-02-19T21:17:09.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:09.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 10 client.?.objecter ms_handle_connect 0x7f5b781005d0 2024-02-19T21:17:09.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:09.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727634623 0 0) 0x7f5b7811b9d0 con 0x7f5b780fd7d0 2024-02-19T21:17:09.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b600033f0 con 0x7f5b780fd7d0 2024-02-19T21:17:09.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710842384 0 0) 0x7f5b780fcf20 con 0x7f5b78104110 2024-02-19T21:17:09.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b7811b9d0 con 0x7f5b78104110 2024-02-19T21:17:09.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127226640 0 0) 0x7f5b780fd0f0 con 0x7f5b781005d0 2024-02-19T21:17:09.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b780fcf20 con 0x7f5b781005d0 2024-02-19T21:17:09.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 116396762 0 0) 0x7f5b600033f0 con 0x7f5b780fd7d0 2024-02-19T21:17:09.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b780fd0f0 con 0x7f5b780fd7d0 2024-02-19T21:17:09.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4052759118 0 0) 0x7f5b7811b9d0 con 0x7f5b78104110 2024-02-19T21:17:09.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b600033f0 con 0x7f5b78104110 2024-02-19T21:17:09.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5b6c001a50 con 0x7f5b780fd7d0 2024-02-19T21:17:09.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5b70001bc0 con 0x7f5b78104110 2024-02-19T21:17:09.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 922343792 0 0) 0x7f5b780fcf20 con 0x7f5b781005d0 2024-02-19T21:17:09.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.006+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b7811b9d0 con 0x7f5b781005d0 2024-02-19T21:17:09.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.007+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5b680020b0 con 0x7f5b781005d0 2024-02-19T21:17:09.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.007+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 854781274 0 0) 0x7f5b780fd0f0 con 0x7f5b780fd7d0 2024-02-19T21:17:09.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.007+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 >> v1:172.21.15.193:6789/0 conn(0x7f5b781005d0 legacy=0x7f5b78114b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:09.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.007+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 >> v1:172.21.15.163:6789/0 conn(0x7f5b78104110 legacy=0x7f5b78118290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:09.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.007+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7811cb70 con 0x7f5b780fd7d0 2024-02-19T21:17:09.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.007+0000 7f5b77fff700 1 -- 172.21.15.163:0/2136691392 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7811bbc0 con 0x7f5b780fd7d0 2024-02-19T21:17:09.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.007+0000 7f5b77fff700 10 client.50591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:09.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.007+0000 7f5b77fff700 1 -- 172.21.15.163:0/2136691392 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7811c0b0 con 0x7f5b780fd7d0 2024-02-19T21:17:09.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.007+0000 7f5b77fff700 1 librados: init done 2024-02-19T21:17:09.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.007+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f5b6c0024b0 con 0x7f5b780fd7d0 2024-02-19T21:17:09.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.007+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5b6c005b80 con 0x7f5b780fd7d0 2024-02-19T21:17:09.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.008+0000 7f5b77fff700 1 -- 172.21.15.163:0/2136691392 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b64004f20 con 0x7f5b780fd7d0 2024-02-19T21:17:09.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.008+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f5b6c018af0 con 0x7f5b780fd7d0 2024-02-19T21:17:09.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.008+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(769..769 src has 623..769) v4 ==== 8036+0+0 (unknown 2734587785 0 0) 0x7f5b6c078dd0 con 0x7f5b780fd7d0 2024-02-19T21:17:09.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.008+0000 7f5b5ffff700 10 client.50591.objecter ms_dispatch 0x7f5b78001550 osd_map(769..769 src has 623..769) v4 2024-02-19T21:17:09.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.008+0000 7f5b5ffff700 3 client.50591.objecter handle_osd_map got epochs [769,769] > 0 2024-02-19T21:17:09.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.008+0000 7f5b5ffff700 3 client.50591.objecter handle_osd_map decoding full epoch 769 2024-02-19T21:17:09.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.008+0000 7f5b5ffff700 20 client.50591.objecter dump_active .. 0 homeless 2024-02-19T21:17:09.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.009+0000 7f5b5ffff700 10 client.50591.objecter ms_handle_connect 0x7f5b60063c60 2024-02-19T21:17:09.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.013+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5b6c04c220 con 0x7f5b780fd7d0 2024-02-19T21:17:09.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.209+0000 7f5b77fff700 1 -- 172.21.15.163:0/2136691392 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f5b640051d0 con 0x7f5b780fd7d0 2024-02-19T21:17:09.295 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:17:09.299 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:09.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:17:09.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:07.332932+0000 mgr.y (mgr.14550) 640 : cluster [DBG] pgmap v1162: 214 pgs: 214 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 887 B/s rd, 0 op/s 2024-02-19T21:17:09.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:07.612489+0000 mon.a (mon.0) 3630 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:09.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:07.612863+0000 mon.a (mon.0) 3631 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-134"}]: dispatch 2024-02-19T21:17:09.400 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:09.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.422+0000 7f5b5ffff700 1 -- 172.21.15.163:0/2136691392 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v770) v1 ==== 87+0+0 (unknown 347618272 0 0) 0x7f5b6c04a630 con 0x7f5b780fd7d0 2024-02-19T21:17:09.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.427+0000 7f5b77fff700 10 librados: watch_flush enter 2024-02-19T21:17:09.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.428+0000 7f5b77fff700 10 librados: watch_flush exit 2024-02-19T21:17:09.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.428+0000 7f5b77fff700 20 client.50591.objecter shutdown clearing up homeless session... 2024-02-19T21:17:09.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.428+0000 7f5b77fff700 10 client.50591.objecter successfully canceled tick 2024-02-19T21:17:09.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.428+0000 7f5b77fff700 1 -- 172.21.15.163:0/2136691392 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5b60063c60 legacy=0x7f5b60066120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:09.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.428+0000 7f5b77fff700 1 -- 172.21.15.163:0/2136691392 >> v1:172.21.15.163:6790/0 conn(0x7f5b780fd7d0 legacy=0x7f5b780fc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:09.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.428+0000 7f5b77fff700 1 -- 172.21.15.163:0/2136691392 shutdown_connections 2024-02-19T21:17:09.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.428+0000 7f5b77fff700 1 -- 172.21.15.163:0/2136691392 >> 172.21.15.163:0/2136691392 conn(0x7f5b780f33f0 msgr2=0x7f5b78107560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:09.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.428+0000 7f5b77fff700 1 -- 172.21.15.163:0/2136691392 shutdown_connections 2024-02-19T21:17:09.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.428+0000 7f5b77fff700 1 -- 172.21.15.163:0/2136691392 wait complete. 2024-02-19T21:17:09.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:09.428+0000 7f5b77fff700 1 librados: shutdown 2024-02-19T21:17:09.433 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:17:09.435 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:09.566 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:09.568 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:09.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:09.633 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:17:09.669 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:09.703 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:10.000 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:17:10.001 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:17:10.136 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:08.355650+0000 osd.1 (osd.1) 346 : cluster [DBG] 16.1b scrub starts 2024-02-19T21:17:10.137 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:08.359474+0000 osd.1 (osd.1) 347 : cluster [DBG] 16.1b scrub ok 2024-02-19T21:17:10.137 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:08.603290+0000 mon.a (mon.0) 3632 : cluster [DBG] osdmap e769: 8 total, 8 up, 6 in 2024-02-19T21:17:10.137 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:08.788892+0000 osd.5 (osd.5) 317 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:10.137 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:08.789352+0000 osd.5 (osd.5) 318 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:10.137 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:09.110588+0000 mon.a (mon.0) 3633 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:10.137 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:09.211071+0000 mon.c (mon.2) 438 : audit [INF] from='client.? v1:172.21.15.163:0/2136691392' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:17:10.138 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:09.211393+0000 mon.a (mon.0) 3634 : audit [INF] from='client.50591 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:17:10.138 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:09.322338+0000 osd.1 (osd.1) 348 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:17:10.138 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:09.325749+0000 osd.1 (osd.1) 349 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:17:10.138 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:09.408985+0000 mon.a (mon.0) 3635 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:17:10.138 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:09.419292+0000 mon.a (mon.0) 3636 : audit [INF] from='client.50591 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:17:10.139 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:09.419365+0000 mon.a (mon.0) 3637 : cluster [DBG] osdmap e770: 8 total, 8 up, 6 in 2024-02-19T21:17:10.139 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:09.426539+0000 mon.a (mon.0) 3638 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:17:10.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:08.355650+0000 osd.1 (osd.1) 346 : cluster [DBG] 16.1b scrub starts 2024-02-19T21:17:10.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:08.359474+0000 osd.1 (osd.1) 347 : cluster [DBG] 16.1b scrub ok 2024-02-19T21:17:10.139 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:08.603290+0000 mon.a (mon.0) 3632 : cluster [DBG] osdmap e769: 8 total, 8 up, 6 in 2024-02-19T21:17:10.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:08.788892+0000 osd.5 (osd.5) 317 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:10.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:08.789352+0000 osd.5 (osd.5) 318 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:10.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:09.110588+0000 mon.a (mon.0) 3633 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:10.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:09.211071+0000 mon.c (mon.2) 438 : audit [INF] from='client.? v1:172.21.15.163:0/2136691392' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:17:10.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:09.211393+0000 mon.a (mon.0) 3634 : audit [INF] from='client.50591 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:17:10.140 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:09.322338+0000 osd.1 (osd.1) 348 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:17:10.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:09.325749+0000 osd.1 (osd.1) 349 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:17:10.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:09.408985+0000 mon.a (mon.0) 3635 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:17:10.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:09.419292+0000 mon.a (mon.0) 3636 : audit [INF] from='client.50591 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:17:10.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:09.419365+0000 mon.a (mon.0) 3637 : cluster [DBG] osdmap e770: 8 total, 8 up, 6 in 2024-02-19T21:17:10.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:09.426539+0000 mon.a (mon.0) 3638 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:17:10.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:08.355650+0000 osd.1 (osd.1) 346 : cluster [DBG] 16.1b scrub starts 2024-02-19T21:17:10.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:08.359474+0000 osd.1 (osd.1) 347 : cluster [DBG] 16.1b scrub ok 2024-02-19T21:17:10.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:08.603290+0000 mon.a (mon.0) 3632 : cluster [DBG] osdmap e769: 8 total, 8 up, 6 in 2024-02-19T21:17:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:08.788892+0000 osd.5 (osd.5) 317 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:08.789352+0000 osd.5 (osd.5) 318 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:09.110588+0000 mon.a (mon.0) 3633 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:09.211071+0000 mon.c (mon.2) 438 : audit [INF] from='client.? v1:172.21.15.163:0/2136691392' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:17:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:09.211393+0000 mon.a (mon.0) 3634 : audit [INF] from='client.50591 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:17:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:09.322338+0000 osd.1 (osd.1) 348 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:17:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:09.325749+0000 osd.1 (osd.1) 349 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:17:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:09.408985+0000 mon.a (mon.0) 3635 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:17:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:09.419292+0000 mon.a (mon.0) 3636 : audit [INF] from='client.50591 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:17:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:09.419365+0000 mon.a (mon.0) 3637 : cluster [DBG] osdmap e770: 8 total, 8 up, 6 in 2024-02-19T21:17:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:09.426539+0000 mon.a (mon.0) 3638 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:17:11.072 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:17:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:17:11.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:09.333351+0000 mgr.y (mgr.14550) 641 : cluster [DBG] pgmap v1165: 182 pgs: 182 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:17:11.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:10.415513+0000 mon.a (mon.0) 3639 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:17:11.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:10.415599+0000 mon.a (mon.0) 3640 : cluster [DBG] osdmap e771: 8 total, 8 up, 6 in 2024-02-19T21:17:11.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:10.443576+0000 mon.a (mon.0) 3641 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:17:11.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:09.333351+0000 mgr.y (mgr.14550) 641 : cluster [DBG] pgmap v1165: 182 pgs: 182 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:17:11.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:10.415513+0000 mon.a (mon.0) 3639 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:17:11.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:10.415599+0000 mon.a (mon.0) 3640 : cluster [DBG] osdmap e771: 8 total, 8 up, 6 in 2024-02-19T21:17:11.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:10.443576+0000 mon.a (mon.0) 3641 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:17:11.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:09.333351+0000 mgr.y (mgr.14550) 641 : cluster [DBG] pgmap v1165: 182 pgs: 182 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:17:11.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:10.415513+0000 mon.a (mon.0) 3639 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:17:11.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:10.415599+0000 mon.a (mon.0) 3640 : cluster [DBG] osdmap e771: 8 total, 8 up, 6 in 2024-02-19T21:17:11.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:10.443576+0000 mon.a (mon.0) 3641 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:17:11.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:10.798460+0000 mgr.y (mgr.14550) 642 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:11.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:11.418380+0000 mon.a (mon.0) 3642 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-136", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:17:11.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:11.418450+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e772: 8 total, 8 up, 6 in 2024-02-19T21:17:11.978 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:11.422178+0000 mon.a (mon.0) 3644 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-136"}]: dispatch 2024-02-19T21:17:11.978 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:10.798460+0000 mgr.y (mgr.14550) 642 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:11.978 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:11.418380+0000 mon.a (mon.0) 3642 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-136", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:17:11.978 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:11.418450+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e772: 8 total, 8 up, 6 in 2024-02-19T21:17:11.978 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:11.422178+0000 mon.a (mon.0) 3644 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-136"}]: dispatch 2024-02-19T21:17:12.000 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-19T21:17:12.086 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:12.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:12.141 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:12.190 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:12.217 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:12.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:10.798460+0000 mgr.y (mgr.14550) 642 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:12.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:11.418380+0000 mon.a (mon.0) 3642 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-136", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:17:12.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:11.418450+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e772: 8 total, 8 up, 6 in 2024-02-19T21:17:12.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:11.422178+0000 mon.a (mon.0) 3644 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-136"}]: dispatch 2024-02-19T21:17:12.367 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:12.371 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:12.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:12.472 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:17:12.501 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:12.527 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-19T21:17:12.649 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:17:12.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:12.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:17:12.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:17:12.782 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:12.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.945+0000 7f6412bab700 1 -- 172.21.15.163:0/1027172270 >> v1:172.21.15.163:6790/0 conn(0x7f640c0ff670 legacy=0x7f640c101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:12.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.945+0000 7f6412bab700 1 -- 172.21.15.163:0/1027172270 shutdown_connections 2024-02-19T21:17:12.953 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.945+0000 7f6412bab700 1 -- 172.21.15.163:0/1027172270 >> 172.21.15.163:0/1027172270 conn(0x7f640c0f2470 msgr2=0x7f640c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:12.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.945+0000 7f6412bab700 1 -- 172.21.15.163:0/1027172270 shutdown_connections 2024-02-19T21:17:12.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.945+0000 7f6412bab700 1 -- 172.21.15.163:0/1027172270 wait complete. 2024-02-19T21:17:12.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.946+0000 7f6412bab700 1 librados: starting msgr at 2024-02-19T21:17:12.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.946+0000 7f6412bab700 1 librados: starting objecter 2024-02-19T21:17:12.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.946+0000 7f6412bab700 1 Processor -- start 2024-02-19T21:17:12.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.946+0000 7f6412bab700 1 -- start start 2024-02-19T21:17:12.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.946+0000 7f6412bab700 1 librados: setting wanted keys 2024-02-19T21:17:12.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.946+0000 7f6412bab700 1 librados: calling monclient init 2024-02-19T21:17:12.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.946+0000 7f6412bab700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f640c119350 con 0x7f640c0fc870 2024-02-19T21:17:12.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.946+0000 7f6412bab700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f640c11a4f0 con 0x7f640c0ff670 2024-02-19T21:17:12.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.946+0000 7f6412bab700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f640c11b6d0 con 0x7f640c1031b0 2024-02-19T21:17:12.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.946+0000 7f6410947700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f640c0fc870 0x7f640c0fb500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43960/0 (socket says 172.21.15.163:43960) 2024-02-19T21:17:12.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.946+0000 7f6410947700 1 -- 172.21.15.163:0/2084026801 learned_addr learned my addr 172.21.15.163:0/2084026801 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:12.954 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 10 client.?.objecter ms_handle_connect 0x7f640c0fc870 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 10 client.?.objecter ms_handle_connect 0x7f640c1031b0 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 10 client.?.objecter ms_handle_connect 0x7f640c0ff670 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795531271 0 0) 0x7f640c11b6d0 con 0x7f640c1031b0 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63fc003030 con 0x7f640c1031b0 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447722518 0 0) 0x7f640c119350 con 0x7f640c0fc870 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f640c11b6d0 con 0x7f640c0fc870 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469688946 0 0) 0x7f640c11a4f0 con 0x7f640c0ff670 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f640c119350 con 0x7f640c0ff670 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4070896469 0 0) 0x7f63fc003030 con 0x7f640c1031b0 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f640c11a4f0 con 0x7f640c1031b0 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4032697231 0 0) 0x7f640c11b6d0 con 0x7f640c0fc870 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63fc003030 con 0x7f640c0fc870 2024-02-19T21:17:12.955 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6404001bb0 con 0x7f640c1031b0 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6400001990 con 0x7f640c0fc870 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3542804798 0 0) 0x7f640c119350 con 0x7f640c0ff670 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f640c11b6d0 con 0x7f640c0ff670 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f63f8000fe0 con 0x7f640c0ff670 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4071657564 0 0) 0x7f640c11a4f0 con 0x7f640c1031b0 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 >> v1:172.21.15.193:6789/0 conn(0x7f640c0ff670 legacy=0x7f640c0fbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 >> v1:172.21.15.163:6789/0 conn(0x7f640c0fc870 legacy=0x7f640c0fb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c11c8b0 con 0x7f640c1031b0 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6412bab700 1 -- 172.21.15.163:0/2084026801 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640c11b8c0 con 0x7f640c1031b0 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6404002530 con 0x7f640c1031b0 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f64040035c0 con 0x7f640c1031b0 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6412bab700 10 client.50597.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6412bab700 1 -- 172.21.15.163:0/2084026801 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f640c11bd30 con 0x7f640c1031b0 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.947+0000 7f6412bab700 1 librados: init done 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.948+0000 7f6412bab700 1 -- 172.21.15.163:0/2084026801 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640c05f700 con 0x7f640c1031b0 2024-02-19T21:17:12.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.948+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6404002090 con 0x7f640c1031b0 2024-02-19T21:17:12.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.949+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(773..773 src has 623..773) v4 ==== 8417+0+0 (unknown 3132177892 0 0) 0x7f6404076940 con 0x7f640c1031b0 2024-02-19T21:17:12.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.949+0000 7f6409ffb700 10 client.50597.objecter ms_dispatch 0x7f640c001550 osd_map(773..773 src has 623..773) v4 2024-02-19T21:17:12.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.949+0000 7f6409ffb700 3 client.50597.objecter handle_osd_map got epochs [773,773] > 0 2024-02-19T21:17:12.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.949+0000 7f6409ffb700 3 client.50597.objecter handle_osd_map decoding full epoch 773 2024-02-19T21:17:12.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.949+0000 7f6409ffb700 20 client.50597.objecter dump_active .. 0 homeless 2024-02-19T21:17:12.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.949+0000 7f6409ffb700 10 client.50597.objecter ms_handle_connect 0x7f63fc063c90 2024-02-19T21:17:12.957 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:12.953+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6404049bd0 con 0x7f640c1031b0 2024-02-19T21:17:12.979 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:17:12.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:11.333970+0000 mgr.y (mgr.14550) 643 : cluster [DBG] pgmap v1168: 214 pgs: 32 creating+peering, 182 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:17:12.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:12.418867+0000 mon.a (mon.0) 3645 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-136"}]': finished 2024-02-19T21:17:12.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:12.418926+0000 mon.a (mon.0) 3646 : cluster [DBG] osdmap e773: 8 total, 8 up, 6 in 2024-02-19T21:17:12.983 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:12.419776+0000 mon.a (mon.0) 3647 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-136", "mode": "writeback"}]: dispatch 2024-02-19T21:17:12.983 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:11.333970+0000 mgr.y (mgr.14550) 643 : cluster [DBG] pgmap v1168: 214 pgs: 32 creating+peering, 182 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:17:12.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:12.418867+0000 mon.a (mon.0) 3645 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-136"}]': finished 2024-02-19T21:17:12.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:12.418926+0000 mon.a (mon.0) 3646 : cluster [DBG] osdmap e773: 8 total, 8 up, 6 in 2024-02-19T21:17:12.984 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:12.419776+0000 mon.a (mon.0) 3647 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-136", "mode": "writeback"}]: dispatch 2024-02-19T21:17:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:12.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:17:13.085 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:13.112 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:13.145 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:13.144+0000 7f6412bab700 1 -- 172.21.15.163:0/2084026801 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd out", "ids": ["7"]} v 0) v1 -- 0x7f640c047540 con 0x7f640c1031b0 2024-02-19T21:17:13.248 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:17:13.144+0000 7fb606608700 -1 mon.a@0(leader).osd e773 definitely_dead 0 2024-02-19T21:17:13.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:11.333970+0000 mgr.y (mgr.14550) 643 : cluster [DBG] pgmap v1168: 214 pgs: 32 creating+peering, 182 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:17:13.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:12.418867+0000 mon.a (mon.0) 3645 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi163-83928-101", "overlaypool": "test-rados-api-smithi163-83928-136"}]': finished 2024-02-19T21:17:13.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:12.418926+0000 mon.a (mon.0) 3646 : cluster [DBG] osdmap e773: 8 total, 8 up, 6 in 2024-02-19T21:17:13.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:12.419776+0000 mon.a (mon.0) 3647 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-136", "mode": "writeback"}]: dispatch 2024-02-19T21:17:13.386 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:13.427 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:13.423+0000 7f6409ffb700 1 -- 172.21.15.163:0/2084026801 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd out", "ids": ["7"]}]=0 marked out osd.7. v774) v1 ==== 87+0+0 (unknown 3080437671 0 0) 0x7f6404048040 con 0x7f640c1031b0 2024-02-19T21:17:13.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:13.431+0000 7f6412bab700 10 librados: watch_flush enter 2024-02-19T21:17:13.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:13.431+0000 7f6412bab700 10 librados: watch_flush exit 2024-02-19T21:17:13.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:13.431+0000 7f6412bab700 20 client.50597.objecter shutdown clearing up homeless session... 2024-02-19T21:17:13.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:13.431+0000 7f6412bab700 10 client.50597.objecter successfully canceled tick 2024-02-19T21:17:13.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:13.431+0000 7f6412bab700 1 -- 172.21.15.163:0/2084026801 >> v1:172.21.15.163:6800/1552514681 conn(0x7f63fc063c90 legacy=0x7f63fc066150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:13.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:13.431+0000 7f6412bab700 1 -- 172.21.15.163:0/2084026801 >> v1:172.21.15.163:6790/0 conn(0x7f640c1031b0 legacy=0x7f640c117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:13.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:13.431+0000 7f6412bab700 1 -- 172.21.15.163:0/2084026801 shutdown_connections 2024-02-19T21:17:13.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:13.431+0000 7f6412bab700 1 -- 172.21.15.163:0/2084026801 >> 172.21.15.163:0/2084026801 conn(0x7f640c0f2470 msgr2=0x7f640c106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:13.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:13.431+0000 7f6412bab700 1 -- 172.21.15.163:0/2084026801 shutdown_connections 2024-02-19T21:17:13.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:13.431+0000 7f6412bab700 1 -- 172.21.15.163:0/2084026801 wait complete. 2024-02-19T21:17:13.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:13.431+0000 7f6412bab700 1 librados: shutdown 2024-02-19T21:17:13.434 INFO:teuthology.orchestra.run.smithi163.stderr:marked out osd.7. 2024-02-19T21:17:13.639 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:13.642 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:13.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:13.743 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:13.748 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:17:13] "GET /metrics HTTP/1.1" 200 190193 "" "Prometheus/2.33.4" 2024-02-19T21:17:13.775 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:13.797 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:13.799 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:13.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:13.900 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:17:13.928 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:14.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:13.145552+0000 mon.c (mon.2) 439 : audit [INF] from='client.? v1:172.21.15.163:0/2084026801' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-19T21:17:14.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:13.145897+0000 mon.a (mon.0) 3648 : audit [INF] from='client.50597 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-19T21:17:14.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:13.145972+0000 mon.a (mon.0) 3649 : cluster [INF] Client client.admin marked osd.7 out, while it was still marked up 2024-02-19T21:17:14.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:13.417196+0000 mon.a (mon.0) 3650 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:17:14.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:13.421233+0000 mon.a (mon.0) 3651 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-136", "mode": "writeback"}]': finished 2024-02-19T21:17:14.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:13.421307+0000 mon.a (mon.0) 3652 : audit [INF] from='client.50597 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-02-19T21:17:14.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:13.421369+0000 mon.a (mon.0) 3653 : cluster [DBG] osdmap e774: 8 total, 8 up, 5 in 2024-02-19T21:17:14.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:13.145552+0000 mon.c (mon.2) 439 : audit [INF] from='client.? v1:172.21.15.163:0/2084026801' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-19T21:17:14.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:13.145897+0000 mon.a (mon.0) 3648 : audit [INF] from='client.50597 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-19T21:17:14.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:13.145972+0000 mon.a (mon.0) 3649 : cluster [INF] Client client.admin marked osd.7 out, while it was still marked up 2024-02-19T21:17:14.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:13.417196+0000 mon.a (mon.0) 3650 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:17:14.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:13.421233+0000 mon.a (mon.0) 3651 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-136", "mode": "writeback"}]': finished 2024-02-19T21:17:14.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:13.421307+0000 mon.a (mon.0) 3652 : audit [INF] from='client.50597 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-02-19T21:17:14.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:13.421369+0000 mon.a (mon.0) 3653 : cluster [DBG] osdmap e774: 8 total, 8 up, 5 in 2024-02-19T21:17:14.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:13.145552+0000 mon.c (mon.2) 439 : audit [INF] from='client.? v1:172.21.15.163:0/2084026801' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-19T21:17:14.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:13.145897+0000 mon.a (mon.0) 3648 : audit [INF] from='client.50597 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-19T21:17:14.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:13.145972+0000 mon.a (mon.0) 3649 : cluster [INF] Client client.admin marked osd.7 out, while it was still marked up 2024-02-19T21:17:14.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:13.417196+0000 mon.a (mon.0) 3650 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-02-19T21:17:14.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:13.421233+0000 mon.a (mon.0) 3651 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi163-83928-136", "mode": "writeback"}]': finished 2024-02-19T21:17:14.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:13.421307+0000 mon.a (mon.0) 3652 : audit [INF] from='client.50597 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-02-19T21:17:14.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:13.421369+0000 mon.a (mon.0) 3653 : cluster [DBG] osdmap e774: 8 total, 8 up, 5 in 2024-02-19T21:17:14.776 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:17:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:14.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:17:14.881 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:14.906 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:14.928 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:14.930 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:14.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:15.031 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:17:15.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:13.334360+0000 mgr.y (mgr.14550) 644 : cluster [DBG] pgmap v1171: 214 pgs: 32 creating+peering, 182 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:17:15.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:14.416406+0000 mon.a (mon.0) 3654 : cluster [DBG] osdmap e775: 8 total, 8 up, 5 in 2024-02-19T21:17:15.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:14.499836+0000 mon.a (mon.0) 3655 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:17:15.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:14.624116+0000 osd.1 (osd.1) 350 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:15.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:14.624280+0000 osd.1 (osd.1) 351 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:15.056 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:13.334360+0000 mgr.y (mgr.14550) 644 : cluster [DBG] pgmap v1171: 214 pgs: 32 creating+peering, 182 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:17:15.056 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:14.416406+0000 mon.a (mon.0) 3654 : cluster [DBG] osdmap e775: 8 total, 8 up, 5 in 2024-02-19T21:17:15.056 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:14.499836+0000 mon.a (mon.0) 3655 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:17:15.056 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:14.624116+0000 osd.1 (osd.1) 350 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:15.056 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:14.624280+0000 osd.1 (osd.1) 351 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:15.057 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:15.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:13.334360+0000 mgr.y (mgr.14550) 644 : cluster [DBG] pgmap v1171: 214 pgs: 32 creating+peering, 182 active+clean; 3.8 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:17:15.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:14.416406+0000 mon.a (mon.0) 3654 : cluster [DBG] osdmap e775: 8 total, 8 up, 5 in 2024-02-19T21:17:15.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:14.499836+0000 mon.a (mon.0) 3655 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_count","val": "2"}]: dispatch 2024-02-19T21:17:15.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:14.624116+0000 osd.1 (osd.1) 350 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:15.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:14.624280+0000 osd.1 (osd.1) 351 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:16.106 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:17:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:16.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:17:16.197 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:16.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.204+0000 7f8495e0a700 1 -- 172.21.15.163:0/3066775685 >> v1:172.21.15.163:6790/0 conn(0x7f84900ff670 legacy=0x7f8490101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:16.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.204+0000 7f8495e0a700 1 -- 172.21.15.163:0/3066775685 shutdown_connections 2024-02-19T21:17:16.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.204+0000 7f8495e0a700 1 -- 172.21.15.163:0/3066775685 >> 172.21.15.163:0/3066775685 conn(0x7f84900f2450 msgr2=0x7f84900f4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:16.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.204+0000 7f8495e0a700 1 -- 172.21.15.163:0/3066775685 shutdown_connections 2024-02-19T21:17:16.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.205+0000 7f8495e0a700 1 -- 172.21.15.163:0/3066775685 wait complete. 2024-02-19T21:17:16.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.205+0000 7f8495e0a700 1 librados: starting msgr at 2024-02-19T21:17:16.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.205+0000 7f8495e0a700 1 librados: starting objecter 2024-02-19T21:17:16.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.205+0000 7f8495e0a700 1 Processor -- start 2024-02-19T21:17:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.205+0000 7f8495e0a700 1 -- start start 2024-02-19T21:17:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.205+0000 7f8495e0a700 1 librados: setting wanted keys 2024-02-19T21:17:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.205+0000 7f8495e0a700 1 librados: calling monclient init 2024-02-19T21:17:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f8495e0a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8490116190 con 0x7f84900ff670 2024-02-19T21:17:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f8495e0a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8490117370 con 0x7f84900fc870 2024-02-19T21:17:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f8495e0a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8490118570 con 0x7f84901031b0 2024-02-19T21:17:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f848effd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f84900ff670 0x7f84900f8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44622/0 (socket says 172.21.15.163:44622) 2024-02-19T21:17:16.207 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f848effd700 1 -- 172.21.15.163:0/2641917926 learned_addr learned my addr 172.21.15.163:0/2641917926 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:16.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f848cff9700 10 client.?.objecter ms_handle_connect 0x7f84900ff670 2024-02-19T21:17:16.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f848cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:16.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f848cff9700 10 client.?.objecter ms_handle_connect 0x7f84901031b0 2024-02-19T21:17:16.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f848cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:16.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354003288 0 0) 0x7f8490116190 con 0x7f84900ff670 2024-02-19T21:17:16.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8478004010 con 0x7f84900ff670 2024-02-19T21:17:16.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378801424 0 0) 0x7f8490118570 con 0x7f84901031b0 2024-02-19T21:17:16.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8490116190 con 0x7f84901031b0 2024-02-19T21:17:16.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f848cff9700 10 client.?.objecter ms_handle_connect 0x7f84900fc870 2024-02-19T21:17:16.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f848cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:16.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4287374419 0 0) 0x7f8478004010 con 0x7f84900ff670 2024-02-19T21:17:16.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8490118570 con 0x7f84900ff670 2024-02-19T21:17:16.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.206+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8484001ea0 con 0x7f84900ff670 2024-02-19T21:17:16.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.207+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 676863855 0 0) 0x7f8490118570 con 0x7f84900ff670 2024-02-19T21:17:16.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.207+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 >> v1:172.21.15.163:6790/0 conn(0x7f84901031b0 legacy=0x7f8490114850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:16.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.207+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 >> v1:172.21.15.193:6789/0 conn(0x7f84900fc870 legacy=0x7f84900f8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:16.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.207+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8490119770 con 0x7f84900ff670 2024-02-19T21:17:16.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.207+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8490118760 con 0x7f84900ff670 2024-02-19T21:17:16.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.207+0000 7f8495e0a700 10 client.23874.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:16.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.207+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8490118bd0 con 0x7f84900ff670 2024-02-19T21:17:16.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.207+0000 7f8495e0a700 1 librados: init done 2024-02-19T21:17:16.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.207+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8484001ac0 con 0x7f84900ff670 2024-02-19T21:17:16.210 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.207+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8484003870 con 0x7f84900ff670 2024-02-19T21:17:16.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.208+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f847c004f30 con 0x7f84900ff670 2024-02-19T21:17:16.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f84840030c0 con 0x7f84900ff670 2024-02-19T21:17:16.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(776..776 src has 623..776) v4 ==== 8454+0+0 (unknown 4151613381 0 0) 0x7f8484079130 con 0x7f84900ff670 2024-02-19T21:17:16.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 osd_map(776..776 src has 623..776) v4 2024-02-19T21:17:16.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f848cff9700 3 client.23874.objecter handle_osd_map got epochs [776,776] > 0 2024-02-19T21:17:16.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f848cff9700 3 client.23874.objecter handle_osd_map decoding full epoch 776 2024-02-19T21:17:16.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f848cff9700 20 client.23874.objecter dump_active .. 0 homeless 2024-02-19T21:17:16.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v776) v1 ==== 54+0+16 (unknown 1886031577 0 2420000192) 0x7f84840481b0 con 0x7f84900ff670 2024-02-19T21:17:16.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f848cff9700 10 client.23874.objecter ms_handle_connect 0x7f8478063ba0 2024-02-19T21:17:16.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f8495e0a700 10 client.23874.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:16.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 1 2024-02-19T21:17:16.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c0051d0 osd=0 3 2024-02-19T21:17:16.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c0051d0 osd=0 3 2024-02-19T21:17:16.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c0051d0 osd=0 3 2024-02-19T21:17:16.211 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 1 2024-02-19T21:17:16.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c0051d0 osd=0 3 2024-02-19T21:17:16.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 0 1 2024-02-19T21:17:16.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c0051d0 osd=0 4 2024-02-19T21:17:16.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f8495e0a700 10 client.23874.objecter _send_command 1 2024-02-19T21:17:16.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.209+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f847c004980 con 0x7f847c004560 2024-02-19T21:17:16.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.210+0000 7f848cff9700 10 client.23874.objecter ms_handle_connect 0x7f847c004560 2024-02-19T21:17:16.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.210+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f847c004980 con 0x7f847c004560 2024-02-19T21:17:16.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.210+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:17:16.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.210+0000 7f848cff9700 10 client.23874.objecter _finish_command 1 = system:0 2024-02-19T21:17:16.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.210+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c0051d0 osd=0 4 2024-02-19T21:17:16.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.210+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 0 1 2024-02-19T21:17:16.212 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:16.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.231+0000 7f8495e0a700 10 client.23874.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:16.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.231+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 2 2024-02-19T21:17:16.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.231+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c0051d0 osd=0 3 2024-02-19T21:17:16.231 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.231+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c0051d0 osd=0 3 2024-02-19T21:17:16.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.231+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c0051d0 osd=0 3 2024-02-19T21:17:16.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.231+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 2 2024-02-19T21:17:16.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.231+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c0051d0 osd=0 3 2024-02-19T21:17:16.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.231+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 0 2 2024-02-19T21:17:16.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.231+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c0051d0 osd=0 4 2024-02-19T21:17:16.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.231+0000 7f8495e0a700 10 client.23874.objecter _send_command 2 2024-02-19T21:17:16.232 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.231+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f847c004d60 con 0x7f847c004560 2024-02-19T21:17:16.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.232+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f847c004d60 con 0x7f847c004560 2024-02-19T21:17:16.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.232+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:16.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.232+0000 7f848cff9700 10 client.23874.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.232+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c0051d0 osd=0 4 2024-02-19T21:17:16.233 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.232+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 0 2 2024-02-19T21:17:16.233 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:17:16.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.233+0000 7f8495e0a700 10 client.23874.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:16.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.233+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 3 2024-02-19T21:17:16.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.233+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c004010 osd=1 3 2024-02-19T21:17:16.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.233+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c004010 osd=1 3 2024-02-19T21:17:16.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.233+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c004010 osd=1 3 2024-02-19T21:17:16.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.233+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 3 2024-02-19T21:17:16.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.233+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c004010 osd=1 3 2024-02-19T21:17:16.234 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.233+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 1 3 2024-02-19T21:17:16.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.233+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c004010 osd=1 4 2024-02-19T21:17:16.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.233+0000 7f8495e0a700 10 client.23874.objecter _send_command 3 2024-02-19T21:17:16.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.233+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f847c00c030 con 0x7f847c004180 2024-02-19T21:17:16.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.233+0000 7f848cff9700 10 client.23874.objecter ms_handle_connect 0x7f847c004180 2024-02-19T21:17:16.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.234+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f847c00c030 con 0x7f847c004180 2024-02-19T21:17:16.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.234+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:17:16.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.234+0000 7f848cff9700 10 client.23874.objecter _finish_command 3 = system:0 2024-02-19T21:17:16.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.234+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c004010 osd=1 4 2024-02-19T21:17:16.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.234+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 1 3 2024-02-19T21:17:16.242 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:14.982838+0000 mon.a (mon.0) 3656 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:16.242 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:15.421733+0000 mon.a (mon.0) 3657 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:17:16.242 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:15.421801+0000 mon.a (mon.0) 3658 : cluster [DBG] osdmap e776: 8 total, 8 up, 5 in 2024-02-19T21:17:16.242 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:15.422761+0000 mon.a (mon.0) 3659 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:17:16.242 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:15.425819+0000 osd.1 (osd.1) 352 : cluster [DBG] 2.0 starting backfill to osd.3 from (0'0,0'0] MAX to 397'2 2024-02-19T21:17:16.242 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:15.426095+0000 osd.1 (osd.1) 353 : cluster [DBG] 2.20 starting backfill to osd.0 from (0'0,0'0] MAX to 397'2 2024-02-19T21:17:16.244 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:16.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.248+0000 7f8495e0a700 10 client.23874.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:16.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.248+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 4 2024-02-19T21:17:16.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.248+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c004010 osd=1 3 2024-02-19T21:17:16.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.248+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c004010 osd=1 3 2024-02-19T21:17:16.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.248+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c004010 osd=1 3 2024-02-19T21:17:16.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.248+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 4 2024-02-19T21:17:16.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.248+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c004010 osd=1 3 2024-02-19T21:17:16.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.248+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 1 4 2024-02-19T21:17:16.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.248+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c004010 osd=1 4 2024-02-19T21:17:16.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.248+0000 7f8495e0a700 10 client.23874.objecter _send_command 4 2024-02-19T21:17:16.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.248+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f847c0018e0 con 0x7f847c004180 2024-02-19T21:17:16.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:14.982838+0000 mon.a (mon.0) 3656 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:16.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:15.421733+0000 mon.a (mon.0) 3657 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:17:16.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:15.421801+0000 mon.a (mon.0) 3658 : cluster [DBG] osdmap e776: 8 total, 8 up, 5 in 2024-02-19T21:17:16.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:15.422761+0000 mon.a (mon.0) 3659 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:17:16.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:15.425819+0000 osd.1 (osd.1) 352 : cluster [DBG] 2.0 starting backfill to osd.3 from (0'0,0'0] MAX to 397'2 2024-02-19T21:17:16.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:15.426095+0000 osd.1 (osd.1) 353 : cluster [DBG] 2.20 starting backfill to osd.0 from (0'0,0'0] MAX to 397'2 2024-02-19T21:17:16.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:14.982838+0000 mon.a (mon.0) 3656 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:16.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:15.421733+0000 mon.a (mon.0) 3657 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_count","val": "2"}]': finished 2024-02-19T21:17:16.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:15.421801+0000 mon.a (mon.0) 3658 : cluster [DBG] osdmap e776: 8 total, 8 up, 5 in 2024-02-19T21:17:16.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:15.422761+0000 mon.a (mon.0) 3659 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_period","val": "600"}]: dispatch 2024-02-19T21:17:16.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:15.425819+0000 osd.1 (osd.1) 352 : cluster [DBG] 2.0 starting backfill to osd.3 from (0'0,0'0] MAX to 397'2 2024-02-19T21:17:16.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:15.426095+0000 osd.1 (osd.1) 353 : cluster [DBG] 2.20 starting backfill to osd.0 from (0'0,0'0] MAX to 397'2 2024-02-19T21:17:16.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f847c0018e0 con 0x7f847c004180 2024-02-19T21:17:16.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:16.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f848cff9700 10 client.23874.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c004010 osd=1 4 2024-02-19T21:17:16.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 1 4 2024-02-19T21:17:16.251 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:17:16.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f8495e0a700 10 client.23874.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:16.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 5 2024-02-19T21:17:16.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c003660 osd=2 3 2024-02-19T21:17:16.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c003660 osd=2 3 2024-02-19T21:17:16.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c003660 osd=2 3 2024-02-19T21:17:16.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 5 2024-02-19T21:17:16.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c003660 osd=2 3 2024-02-19T21:17:16.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 2 5 2024-02-19T21:17:16.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c003660 osd=2 4 2024-02-19T21:17:16.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f8495e0a700 10 client.23874.objecter _send_command 5 2024-02-19T21:17:16.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f847c00dec0 con 0x7f847c00d320 2024-02-19T21:17:16.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.250+0000 7f848cff9700 10 client.23874.objecter ms_handle_connect 0x7f847c00d320 2024-02-19T21:17:16.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.251+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f847c00dec0 con 0x7f847c00d320 2024-02-19T21:17:16.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.251+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:17:16.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.251+0000 7f848cff9700 10 client.23874.objecter _finish_command 5 = system:0 2024-02-19T21:17:16.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.251+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c003660 osd=2 4 2024-02-19T21:17:16.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.251+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 2 5 2024-02-19T21:17:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.265+0000 7f8495e0a700 10 client.23874.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.265+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 6 2024-02-19T21:17:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.265+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c003660 osd=2 3 2024-02-19T21:17:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.265+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c003660 osd=2 3 2024-02-19T21:17:16.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.265+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c003660 osd=2 3 2024-02-19T21:17:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.265+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 6 2024-02-19T21:17:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.265+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c003660 osd=2 3 2024-02-19T21:17:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.265+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 2 6 2024-02-19T21:17:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.265+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c003660 osd=2 4 2024-02-19T21:17:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.265+0000 7f8495e0a700 10 client.23874.objecter _send_command 6 2024-02-19T21:17:16.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.265+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f847c003190 con 0x7f847c00d320 2024-02-19T21:17:16.266 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.266+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f847c003190 con 0x7f847c00d320 2024-02-19T21:17:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.266+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.266+0000 7f848cff9700 10 client.23874.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.266+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c003660 osd=2 4 2024-02-19T21:17:16.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.266+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 2 6 2024-02-19T21:17:16.267 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:17:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.267+0000 7f8495e0a700 10 client.23874.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.267+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 7 2024-02-19T21:17:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.267+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c0119a0 osd=3 3 2024-02-19T21:17:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.267+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c0119a0 osd=3 3 2024-02-19T21:17:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.267+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c0119a0 osd=3 3 2024-02-19T21:17:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.267+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 7 2024-02-19T21:17:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.267+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c0119a0 osd=3 3 2024-02-19T21:17:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.267+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 3 7 2024-02-19T21:17:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.267+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c0119a0 osd=3 4 2024-02-19T21:17:16.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.267+0000 7f8495e0a700 10 client.23874.objecter _send_command 7 2024-02-19T21:17:16.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.267+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f847c015b90 con 0x7f847c012020 2024-02-19T21:17:16.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.267+0000 7f848cff9700 10 client.23874.objecter ms_handle_connect 0x7f847c012020 2024-02-19T21:17:16.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.267+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f847c015b90 con 0x7f847c012020 2024-02-19T21:17:16.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.268+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:17:16.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.268+0000 7f848cff9700 10 client.23874.objecter _finish_command 7 = system:0 2024-02-19T21:17:16.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.268+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c0119a0 osd=3 4 2024-02-19T21:17:16.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.268+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 3 7 2024-02-19T21:17:16.269 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:16.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:16.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.282+0000 7f8495e0a700 10 client.23874.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:16.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.282+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 8 2024-02-19T21:17:16.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.282+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c0119a0 osd=3 3 2024-02-19T21:17:16.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.282+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c0119a0 osd=3 3 2024-02-19T21:17:16.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.282+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c0119a0 osd=3 3 2024-02-19T21:17:16.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.282+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 8 2024-02-19T21:17:16.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.282+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c0119a0 osd=3 3 2024-02-19T21:17:16.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.282+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 3 8 2024-02-19T21:17:16.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.282+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c0119a0 osd=3 4 2024-02-19T21:17:16.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.282+0000 7f8495e0a700 10 client.23874.objecter _send_command 8 2024-02-19T21:17:16.282 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.282+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f847c00e270 con 0x7f847c012020 2024-02-19T21:17:16.283 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f847c00e270 con 0x7f847c012020 2024-02-19T21:17:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f848cff9700 10 client.23874.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c0119a0 osd=3 4 2024-02-19T21:17:16.284 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 3 8 2024-02-19T21:17:16.284 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:17:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f8495e0a700 10 client.23874.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 9 2024-02-19T21:17:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c00e270 osd=4 3 2024-02-19T21:17:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c00e270 osd=4 3 2024-02-19T21:17:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c00e270 osd=4 3 2024-02-19T21:17:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 9 2024-02-19T21:17:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c00e270 osd=4 3 2024-02-19T21:17:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 4 9 2024-02-19T21:17:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c00e270 osd=4 4 2024-02-19T21:17:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f8495e0a700 10 client.23874.objecter _send_command 9 2024-02-19T21:17:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.283+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f847c00d070 con 0x7f847c00e410 2024-02-19T21:17:16.285 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.284+0000 7f848cff9700 10 client.23874.objecter ms_handle_connect 0x7f847c00e410 2024-02-19T21:17:16.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.285+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f847c00d070 con 0x7f847c00e410 2024-02-19T21:17:16.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.285+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:17:16.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.285+0000 7f848cff9700 10 client.23874.objecter _finish_command 9 = system:0 2024-02-19T21:17:16.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.285+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c00e270 osd=4 4 2024-02-19T21:17:16.286 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.285+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 4 9 2024-02-19T21:17:16.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.299+0000 7f8495e0a700 10 client.23874.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:16.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.299+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 10 2024-02-19T21:17:16.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.299+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c00e270 osd=4 3 2024-02-19T21:17:16.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.299+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c00e270 osd=4 3 2024-02-19T21:17:16.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.299+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c00e270 osd=4 3 2024-02-19T21:17:16.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.299+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 10 2024-02-19T21:17:16.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.299+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c00e270 osd=4 3 2024-02-19T21:17:16.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.299+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 4 10 2024-02-19T21:17:16.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.299+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c00e270 osd=4 4 2024-02-19T21:17:16.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.299+0000 7f8495e0a700 10 client.23874.objecter _send_command 10 2024-02-19T21:17:16.300 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.299+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f847c00d070 con 0x7f847c00e410 2024-02-19T21:17:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.301+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f847c00d070 con 0x7f847c00e410 2024-02-19T21:17:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.301+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.301+0000 7f848cff9700 10 client.23874.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.301+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c00e270 osd=4 4 2024-02-19T21:17:16.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.301+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 4 10 2024-02-19T21:17:16.302 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:17:16.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.302+0000 7f8495e0a700 10 client.23874.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:16.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.302+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 11 2024-02-19T21:17:16.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.302+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c00d070 osd=5 3 2024-02-19T21:17:16.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.302+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c00d070 osd=5 3 2024-02-19T21:17:16.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.302+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c00d070 osd=5 3 2024-02-19T21:17:16.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.302+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 11 2024-02-19T21:17:16.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.302+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c00d070 osd=5 3 2024-02-19T21:17:16.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.302+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 5 11 2024-02-19T21:17:16.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.302+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c00d070 osd=5 4 2024-02-19T21:17:16.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.302+0000 7f8495e0a700 10 client.23874.objecter _send_command 11 2024-02-19T21:17:16.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.302+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f847c016ef0 con 0x7f847c016950 2024-02-19T21:17:16.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.302+0000 7f848cff9700 10 client.23874.objecter ms_handle_connect 0x7f847c016950 2024-02-19T21:17:16.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.303+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f847c016ef0 con 0x7f847c016950 2024-02-19T21:17:16.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.303+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:17:16.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.303+0000 7f848cff9700 10 client.23874.objecter _finish_command 11 = system:0 2024-02-19T21:17:16.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.303+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c00d070 osd=5 4 2024-02-19T21:17:16.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.303+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 5 11 2024-02-19T21:17:16.317 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.317+0000 7f8495e0a700 10 client.23874.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.317+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 12 2024-02-19T21:17:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.317+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c00d070 osd=5 3 2024-02-19T21:17:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.317+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c00d070 osd=5 3 2024-02-19T21:17:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.317+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c00d070 osd=5 3 2024-02-19T21:17:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.317+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 12 2024-02-19T21:17:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.317+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c00d070 osd=5 3 2024-02-19T21:17:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.317+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 5 12 2024-02-19T21:17:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.317+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c00d070 osd=5 4 2024-02-19T21:17:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.317+0000 7f8495e0a700 10 client.23874.objecter _send_command 12 2024-02-19T21:17:16.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.317+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f847c016ef0 con 0x7f847c016950 2024-02-19T21:17:16.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f847c016ef0 con 0x7f847c016950 2024-02-19T21:17:16.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:16.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f848cff9700 10 client.23874.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c00d070 osd=5 4 2024-02-19T21:17:16.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 5 12 2024-02-19T21:17:16.320 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:17:16.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f8495e0a700 10 client.23874.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:16.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 13 2024-02-19T21:17:16.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c016ef0 osd=6 3 2024-02-19T21:17:16.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c016ef0 osd=6 3 2024-02-19T21:17:16.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c016ef0 osd=6 3 2024-02-19T21:17:16.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 13 2024-02-19T21:17:16.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c016ef0 osd=6 3 2024-02-19T21:17:16.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 6 13 2024-02-19T21:17:16.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c016ef0 osd=6 4 2024-02-19T21:17:16.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f8495e0a700 10 client.23874.objecter _send_command 13 2024-02-19T21:17:16.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.319+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f847c022c80 con 0x7f847c01f170 2024-02-19T21:17:16.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.320+0000 7f848cff9700 10 client.23874.objecter ms_handle_connect 0x7f847c01f170 2024-02-19T21:17:16.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.321+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f847c022c80 con 0x7f847c01f170 2024-02-19T21:17:16.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.321+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:17:16.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.321+0000 7f848cff9700 10 client.23874.objecter _finish_command 13 = system:0 2024-02-19T21:17:16.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.321+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c016ef0 osd=6 4 2024-02-19T21:17:16.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.321+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 6 13 2024-02-19T21:17:16.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.337+0000 7f8495e0a700 10 client.23874.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:16.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.337+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 14 2024-02-19T21:17:16.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.337+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c016ef0 osd=6 3 2024-02-19T21:17:16.337 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.337+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c016ef0 osd=6 3 2024-02-19T21:17:16.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.337+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c016ef0 osd=6 3 2024-02-19T21:17:16.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.337+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 14 2024-02-19T21:17:16.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.337+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c016ef0 osd=6 3 2024-02-19T21:17:16.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.337+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 6 14 2024-02-19T21:17:16.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.337+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c016ef0 osd=6 4 2024-02-19T21:17:16.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.337+0000 7f8495e0a700 10 client.23874.objecter _send_command 14 2024-02-19T21:17:16.338 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.337+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f847c022c80 con 0x7f847c01f170 2024-02-19T21:17:16.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f847c022c80 con 0x7f847c01f170 2024-02-19T21:17:16.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:16.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f848cff9700 10 client.23874.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c016ef0 osd=6 4 2024-02-19T21:17:16.339 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 6 14 2024-02-19T21:17:16.339 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:17:16.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f8495e0a700 10 client.23874.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:16.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 15 2024-02-19T21:17:16.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c022c80 osd=7 3 2024-02-19T21:17:16.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c022c80 osd=7 3 2024-02-19T21:17:16.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c022c80 osd=7 3 2024-02-19T21:17:16.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 15 2024-02-19T21:17:16.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c022c80 osd=7 3 2024-02-19T21:17:16.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 7 15 2024-02-19T21:17:16.340 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c022c80 osd=7 4 2024-02-19T21:17:16.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f8495e0a700 10 client.23874.objecter _send_command 15 2024-02-19T21:17:16.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f847c023ed0 con 0x7f847c024980 2024-02-19T21:17:16.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.339+0000 7f848cff9700 10 client.23874.objecter ms_handle_connect 0x7f847c024980 2024-02-19T21:17:16.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.340+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f847c023ed0 con 0x7f847c024980 2024-02-19T21:17:16.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.340+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:17:16.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.340+0000 7f848cff9700 10 client.23874.objecter _finish_command 15 = system:0 2024-02-19T21:17:16.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.340+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c022c80 osd=7 4 2024-02-19T21:17:16.341 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.340+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 7 15 2024-02-19T21:17:16.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.354+0000 7f8495e0a700 10 client.23874.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:16.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.354+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign -1 16 2024-02-19T21:17:16.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.354+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c022c80 osd=7 3 2024-02-19T21:17:16.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.354+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c022c80 osd=7 3 2024-02-19T21:17:16.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.354+0000 7f8495e0a700 20 client.23874.objecter _get_session s=0x7f847c022c80 osd=7 3 2024-02-19T21:17:16.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.354+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_remove -1 16 2024-02-19T21:17:16.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.354+0000 7f8495e0a700 20 client.23874.objecter get_session s=0x7f847c022c80 osd=7 3 2024-02-19T21:17:16.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.354+0000 7f8495e0a700 15 client.23874.objecter _session_command_op_assign 7 16 2024-02-19T21:17:16.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.354+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c022c80 osd=7 4 2024-02-19T21:17:16.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.354+0000 7f8495e0a700 10 client.23874.objecter _send_command 16 2024-02-19T21:17:16.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.354+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f847c023ed0 con 0x7f847c024980 2024-02-19T21:17:16.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.356+0000 7f848cff9700 1 -- 172.21.15.163:0/2641917926 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f847c023ed0 con 0x7f847c024980 2024-02-19T21:17:16.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.356+0000 7f848cff9700 10 client.23874.objecter ms_dispatch 0x7f8490001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:16.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.356+0000 7f848cff9700 10 client.23874.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.356+0000 7f848cff9700 20 client.23874.objecter put_session s=0x7f847c022c80 osd=7 4 2024-02-19T21:17:16.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.356+0000 7f848cff9700 15 client.23874.objecter _session_command_op_remove 7 16 2024-02-19T21:17:16.357 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:17:16.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 10 librados: watch_flush enter 2024-02-19T21:17:16.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 10 librados: watch_flush exit 2024-02-19T21:17:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 10 client.23874.objecter close_session for osd.0 2024-02-19T21:17:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 >> v1:172.21.15.163:6801/1559418115 conn(0x7f847c004560 legacy=0x7f847c002780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c0051d0 osd=0 1 2024-02-19T21:17:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 10 client.23874.objecter close_session for osd.1 2024-02-19T21:17:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 >> v1:172.21.15.163:6805/3125047432 conn(0x7f847c004180 legacy=0x7f847c00a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c004010 osd=1 1 2024-02-19T21:17:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 10 client.23874.objecter close_session for osd.2 2024-02-19T21:17:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 >> v1:172.21.15.163:6809/1563158487 conn(0x7f847c00d320 legacy=0x7f847c00d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c003660 osd=2 1 2024-02-19T21:17:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 10 client.23874.objecter close_session for osd.3 2024-02-19T21:17:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 >> v1:172.21.15.163:6813/2112279769 conn(0x7f847c012020 legacy=0x7f847c014410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c0119a0 osd=3 1 2024-02-19T21:17:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f848ffff700 1 -- 172.21.15.163:0/2641917926 reap_dead start 2024-02-19T21:17:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 10 client.23874.objecter close_session for osd.4 2024-02-19T21:17:16.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 >> v1:172.21.15.193:6800/3129716965 conn(0x7f847c00e410 legacy=0x7f847c015d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f848ffff700 1 -- 172.21.15.163:0/2641917926 reap_dead start 2024-02-19T21:17:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c00e270 osd=4 1 2024-02-19T21:17:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 10 client.23874.objecter close_session for osd.5 2024-02-19T21:17:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 >> v1:172.21.15.193:6804/805161991 conn(0x7f847c016950 legacy=0x7f847c01d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c00d070 osd=5 1 2024-02-19T21:17:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 10 client.23874.objecter close_session for osd.6 2024-02-19T21:17:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 >> v1:172.21.15.193:6808/787604615 conn(0x7f847c01f170 legacy=0x7f847c021560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c016ef0 osd=6 1 2024-02-19T21:17:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 10 client.23874.objecter close_session for osd.7 2024-02-19T21:17:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 >> v1:172.21.15.193:6812/157927432 conn(0x7f847c024980 legacy=0x7f847c024de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 20 client.23874.objecter put_session s=0x7f847c022c80 osd=7 1 2024-02-19T21:17:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 20 client.23874.objecter shutdown clearing up homeless session... 2024-02-19T21:17:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 10 client.23874.objecter successfully canceled tick 2024-02-19T21:17:16.360 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8478063ba0 legacy=0x7f8478066060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 >> v1:172.21.15.163:6789/0 conn(0x7f84900ff670 legacy=0x7f84900f8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f848ffff700 1 -- 172.21.15.163:0/2641917926 reap_dead start 2024-02-19T21:17:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f848ffff700 1 -- 172.21.15.163:0/2641917926 reap_dead start 2024-02-19T21:17:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 shutdown_connections 2024-02-19T21:17:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 >> 172.21.15.163:0/2641917926 conn(0x7f84900f2450 msgr2=0x7f8490106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 shutdown_connections 2024-02-19T21:17:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 1 -- 172.21.15.163:0/2641917926 wait complete. 2024-02-19T21:17:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:16.357+0000 7f8495e0a700 1 librados: shutdown 2024-02-19T21:17:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.361 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.362 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:17:16.370 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:17:16.399 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:16.428 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:17:16.432 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:16.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:17:16.533 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:16.560 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:17.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:15.335210+0000 mgr.y (mgr.14550) 645 : cluster [DBG] pgmap v1174: 214 pgs: 4 active+recovering, 2 remapped+peering, 208 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s; 7/717 objects misplaced (0.976%); 588 B/s, 22 objects/s recovering 2024-02-19T21:17:17.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:16.209430+0000 mon.a (mon.0) 3660 : audit [DBG] from='client.? v1:172.21.15.163:0/2641917926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:17:17.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:16.418175+0000 mon.a (mon.0) 3661 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:17:17.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:16.418266+0000 mon.a (mon.0) 3662 : cluster [DBG] osdmap e777: 8 total, 8 up, 5 in 2024-02-19T21:17:17.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:16.418670+0000 mon.a (mon.0) 3663 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:17:17.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:15.335210+0000 mgr.y (mgr.14550) 645 : cluster [DBG] pgmap v1174: 214 pgs: 4 active+recovering, 2 remapped+peering, 208 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s; 7/717 objects misplaced (0.976%); 588 B/s, 22 objects/s recovering 2024-02-19T21:17:17.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:16.209430+0000 mon.a (mon.0) 3660 : audit [DBG] from='client.? v1:172.21.15.163:0/2641917926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:17:17.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:16.418175+0000 mon.a (mon.0) 3661 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:17:17.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:16.418266+0000 mon.a (mon.0) 3662 : cluster [DBG] osdmap e777: 8 total, 8 up, 5 in 2024-02-19T21:17:17.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:16.418670+0000 mon.a (mon.0) 3663 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:17:17.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:15.335210+0000 mgr.y (mgr.14550) 645 : cluster [DBG] pgmap v1174: 214 pgs: 4 active+recovering, 2 remapped+peering, 208 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s; 7/717 objects misplaced (0.976%); 588 B/s, 22 objects/s recovering 2024-02-19T21:17:17.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:16.209430+0000 mon.a (mon.0) 3660 : audit [DBG] from='client.? v1:172.21.15.163:0/2641917926' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:17:17.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:16.418175+0000 mon.a (mon.0) 3661 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_period","val": "600"}]': finished 2024-02-19T21:17:17.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:16.418266+0000 mon.a (mon.0) 3662 : cluster [DBG] osdmap e777: 8 total, 8 up, 5 in 2024-02-19T21:17:17.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:16.418670+0000 mon.a (mon.0) 3663 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_type","val": "bloom"}]: dispatch 2024-02-19T21:17:17.673 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:17:17.673 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:17:17.673 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-02-19T21:17:17.674 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-02-19T21:17:17.674 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:17:18.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:17:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:17:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:17:18.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:17.415769+0000 mon.a (mon.0) 3664 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:17:18.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:17.422472+0000 mon.a (mon.0) 3665 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:17:18.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:17.422566+0000 mon.a (mon.0) 3666 : cluster [DBG] osdmap e778: 8 total, 8 up, 5 in 2024-02-19T21:17:18.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:17.423483+0000 mon.a (mon.0) 3667 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:17:18.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:17.415769+0000 mon.a (mon.0) 3664 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:17:18.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:17.422472+0000 mon.a (mon.0) 3665 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:17:18.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:17.422566+0000 mon.a (mon.0) 3666 : cluster [DBG] osdmap e778: 8 total, 8 up, 5 in 2024-02-19T21:17:18.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:17.423483+0000 mon.a (mon.0) 3667 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:17:18.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:17.415769+0000 mon.a (mon.0) 3664 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-02-19T21:17:18.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:17.422472+0000 mon.a (mon.0) 3665 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "hit_set_type","val": "bloom"}]': finished 2024-02-19T21:17:18.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:17.422566+0000 mon.a (mon.0) 3666 : cluster [DBG] osdmap e778: 8 total, 8 up, 5 in 2024-02-19T21:17:18.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:17.423483+0000 mon.a (mon.0) 3667 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-02-19T21:17:18.310 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:18.314 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:18.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:18.330 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-19T21:17:18.415 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:17:18.445 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:18.594 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:17:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:18.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:17:18.698 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:18.726 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:19.160 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:19.175 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:19.177 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:19.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:19.178 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:17.335631+0000 mgr.y (mgr.14550) 646 : cluster [DBG] pgmap v1177: 214 pgs: 4 active+recovering, 2 remapped+peering, 208 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s; 7/717 objects misplaced (0.976%); 588 B/s, 22 objects/s recovering 2024-02-19T21:17:19.178 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:18.423689+0000 mon.a (mon.0) 3668 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-19T21:17:19.178 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:18.423740+0000 mon.a (mon.0) 3669 : cluster [DBG] osdmap e779: 8 total, 8 up, 5 in 2024-02-19T21:17:19.178 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:18.424992+0000 mon.a (mon.0) 3670 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "target_max_objects","val": "1"}]: dispatch 2024-02-19T21:17:19.178 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:17.335631+0000 mgr.y (mgr.14550) 646 : cluster [DBG] pgmap v1177: 214 pgs: 4 active+recovering, 2 remapped+peering, 208 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s; 7/717 objects misplaced (0.976%); 588 B/s, 22 objects/s recovering 2024-02-19T21:17:19.178 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:18.423689+0000 mon.a (mon.0) 3668 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-19T21:17:19.178 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:18.423740+0000 mon.a (mon.0) 3669 : cluster [DBG] osdmap e779: 8 total, 8 up, 5 in 2024-02-19T21:17:19.179 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:18.424992+0000 mon.a (mon.0) 3670 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "target_max_objects","val": "1"}]: dispatch 2024-02-19T21:17:19.279 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:19.308 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:17.335631+0000 mgr.y (mgr.14550) 646 : cluster [DBG] pgmap v1177: 214 pgs: 4 active+recovering, 2 remapped+peering, 208 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s; 7/717 objects misplaced (0.976%); 588 B/s, 22 objects/s recovering 2024-02-19T21:17:19.308 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:18.423689+0000 mon.a (mon.0) 3668 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-02-19T21:17:19.308 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:18.423740+0000 mon.a (mon.0) 3669 : cluster [DBG] osdmap e779: 8 total, 8 up, 5 in 2024-02-19T21:17:19.308 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:18.424992+0000 mon.a (mon.0) 3670 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "target_max_objects","val": "1"}]: dispatch 2024-02-19T21:17:19.310 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:19.331 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:19.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:19.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:19.434 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:17:19.465 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:19.553 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:17:19.588 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:17:19.591 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:19.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:17:19.693 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:17:19.721 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:19.741 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:17:19.743 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:19.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:17:19.844 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:17:19.880 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:20.529 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:17:20.532 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:20.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:17:20.633 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:20.659 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 6, 0] out_osds: [2, 5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:17:20.659 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:17:20.659 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-02-19T21:17:20.659 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd in 2 2024-02-19T21:17:20.665 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:20.686 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:19.336038+0000 mgr.y (mgr.14550) 647 : cluster [DBG] pgmap v1180: 214 pgs: 4 active+recovering, 2 remapped+peering, 208 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 7/717 objects misplaced (0.976%) 2024-02-19T21:17:20.686 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:19.420493+0000 mon.a (mon.0) 3671 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "target_max_objects","val": "1"}]': finished 2024-02-19T21:17:20.686 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:19.420552+0000 mon.a (mon.0) 3672 : cluster [DBG] osdmap e780: 8 total, 8 up, 5 in 2024-02-19T21:17:20.686 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:19.336038+0000 mgr.y (mgr.14550) 647 : cluster [DBG] pgmap v1180: 214 pgs: 4 active+recovering, 2 remapped+peering, 208 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 7/717 objects misplaced (0.976%) 2024-02-19T21:17:20.686 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:19.420493+0000 mon.a (mon.0) 3671 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "target_max_objects","val": "1"}]': finished 2024-02-19T21:17:20.686 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:19.420552+0000 mon.a (mon.0) 3672 : cluster [DBG] osdmap e780: 8 total, 8 up, 5 in 2024-02-19T21:17:20.687 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:20.688 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:19.336038+0000 mgr.y (mgr.14550) 647 : cluster [DBG] pgmap v1180: 214 pgs: 4 active+recovering, 2 remapped+peering, 208 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 7/717 objects misplaced (0.976%) 2024-02-19T21:17:20.688 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:19.420493+0000 mon.a (mon.0) 3671 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-136","var": "target_max_objects","val": "1"}]': finished 2024-02-19T21:17:20.688 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:19.420552+0000 mon.a (mon.0) 3672 : cluster [DBG] osdmap e780: 8 total, 8 up, 5 in 2024-02-19T21:17:20.689 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:20.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:20.753 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:20.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.758+0000 7fb4b524d700 1 -- 172.21.15.163:0/2279928231 >> v1:172.21.15.163:6789/0 conn(0x7fb4b01031c0 legacy=0x7fb4b0105640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:20.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.758+0000 7fb4b524d700 1 -- 172.21.15.163:0/2279928231 shutdown_connections 2024-02-19T21:17:20.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.758+0000 7fb4b524d700 1 -- 172.21.15.163:0/2279928231 >> 172.21.15.163:0/2279928231 conn(0x7fb4b00f24a0 msgr2=0x7fb4b00f48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:20.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.758+0000 7fb4b524d700 1 -- 172.21.15.163:0/2279928231 shutdown_connections 2024-02-19T21:17:20.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.758+0000 7fb4b524d700 1 -- 172.21.15.163:0/2279928231 wait complete. 2024-02-19T21:17:20.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.759+0000 7fb4b524d700 1 librados: starting msgr at 2024-02-19T21:17:20.759 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.759+0000 7fb4b524d700 1 librados: starting objecter 2024-02-19T21:17:20.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.759+0000 7fb4b524d700 1 Processor -- start 2024-02-19T21:17:20.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb4b524d700 1 -- start start 2024-02-19T21:17:20.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb4b524d700 1 librados: setting wanted keys 2024-02-19T21:17:20.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb4b524d700 1 librados: calling monclient init 2024-02-19T21:17:20.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb4b524d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb4b0115300 con 0x7fb4b00ff680 2024-02-19T21:17:20.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb4b524d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb4b01164e0 con 0x7fb4b00fc880 2024-02-19T21:17:20.762 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb4b524d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb4b01176e0 con 0x7fb4b01031c0 2024-02-19T21:17:20.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb4ae59c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fb4b00ff680 0x7fb4b00f8ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44640/0 (socket says 172.21.15.163:44640) 2024-02-19T21:17:20.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb4af59e700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fb4b01031c0 0x7fb4b01139c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:46608/0 (socket says 172.21.15.163:46608) 2024-02-19T21:17:20.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb4ae59c700 1 -- 172.21.15.163:0/70169480 learned_addr learned my addr 172.21.15.163:0/70169480 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:20.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb497fff700 10 client.?.objecter ms_handle_connect 0x7fb4b00ff680 2024-02-19T21:17:20.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb497fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:20.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb497fff700 10 client.?.objecter ms_handle_connect 0x7fb4b01031c0 2024-02-19T21:17:20.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb497fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:20.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb497fff700 10 client.?.objecter ms_handle_connect 0x7fb4b00fc880 2024-02-19T21:17:20.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb497fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:20.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019305163 0 0) 0x7fb4b01176e0 con 0x7fb4b01031c0 2024-02-19T21:17:20.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4980033f0 con 0x7fb4b01031c0 2024-02-19T21:17:20.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772520093 0 0) 0x7fb4b0115300 con 0x7fb4b00ff680 2024-02-19T21:17:20.763 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4b01176e0 con 0x7fb4b00ff680 2024-02-19T21:17:20.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939750844 0 0) 0x7fb4b01164e0 con 0x7fb4b00fc880 2024-02-19T21:17:20.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.760+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4b0115300 con 0x7fb4b00fc880 2024-02-19T21:17:20.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3657589531 0 0) 0x7fb4980033f0 con 0x7fb4b01031c0 2024-02-19T21:17:20.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4b01164e0 con 0x7fb4b01031c0 2024-02-19T21:17:20.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb4a8002240 con 0x7fb4b01031c0 2024-02-19T21:17:20.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1456237052 0 0) 0x7fb4b01176e0 con 0x7fb4b00ff680 2024-02-19T21:17:20.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4980033f0 con 0x7fb4b00ff680 2024-02-19T21:17:20.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1607935613 0 0) 0x7fb4b0115300 con 0x7fb4b00fc880 2024-02-19T21:17:20.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4b01176e0 con 0x7fb4b00fc880 2024-02-19T21:17:20.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb4a4001860 con 0x7fb4b00ff680 2024-02-19T21:17:20.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb4a0001990 con 0x7fb4b00fc880 2024-02-19T21:17:20.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1805282301 0 0) 0x7fb4b01164e0 con 0x7fb4b01031c0 2024-02-19T21:17:20.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 >> v1:172.21.15.193:6789/0 conn(0x7fb4b00fc880 legacy=0x7fb4b00f85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:20.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 >> v1:172.21.15.163:6789/0 conn(0x7fb4b00ff680 legacy=0x7fb4b00f8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:20.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b01188e0 con 0x7fb4b01031c0 2024-02-19T21:17:20.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb4b524d700 1 -- 172.21.15.163:0/70169480 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b01178d0 con 0x7fb4b01031c0 2024-02-19T21:17:20.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb4b524d700 10 client.50609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:20.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb4b524d700 1 -- 172.21.15.163:0/70169480 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b0117dc0 con 0x7fb4b01031c0 2024-02-19T21:17:20.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb4b524d700 1 librados: init done 2024-02-19T21:17:20.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb4a8003370 con 0x7fb4b01031c0 2024-02-19T21:17:20.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.761+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb4a8003bc0 con 0x7fb4b01031c0 2024-02-19T21:17:20.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.762+0000 7fb4b524d700 1 -- 172.21.15.163:0/70169480 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49c004f20 con 0x7fb4b01031c0 2024-02-19T21:17:20.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.763+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fb4a8003d20 con 0x7fb4b01031c0 2024-02-19T21:17:20.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.763+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(780..780 src has 623..780) v4 ==== 8447+0+0 (unknown 3494485456 0 0) 0x7fb4a8079440 con 0x7fb4b01031c0 2024-02-19T21:17:20.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.763+0000 7fb497fff700 10 client.50609.objecter ms_dispatch 0x7fb4b0001550 osd_map(780..780 src has 623..780) v4 2024-02-19T21:17:20.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.763+0000 7fb497fff700 3 client.50609.objecter handle_osd_map got epochs [780,780] > 0 2024-02-19T21:17:20.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.763+0000 7fb497fff700 3 client.50609.objecter handle_osd_map decoding full epoch 780 2024-02-19T21:17:20.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.764+0000 7fb497fff700 20 client.50609.objecter dump_active .. 0 homeless 2024-02-19T21:17:20.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.764+0000 7fb497fff700 10 client.50609.objecter ms_handle_connect 0x7fb498065f90 2024-02-19T21:17:20.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.770+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb4a804c880 con 0x7fb4b01031c0 2024-02-19T21:17:20.790 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:20.820 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:20.956 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:20.956+0000 7fb4b524d700 1 -- 172.21.15.163:0/70169480 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) v1 -- 0x7fb49c0051d0 con 0x7fb4b01031c0 2024-02-19T21:17:21.070 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:21.072 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:17:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:17:21.073 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:21.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:21.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:21.206 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:21.226 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:21.229 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:21.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:21.330 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:21.359 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:21.380 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:21.383 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:21.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:21.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:21.430+0000 7fb497fff700 1 -- 172.21.15.163:0/70169480 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v781) v1 ==== 97+0+0 (unknown 3202913883 0 0) 0x7fb4a804acf0 con 0x7fb4b01031c0 2024-02-19T21:17:21.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:21.433+0000 7fb4b524d700 10 librados: watch_flush enter 2024-02-19T21:17:21.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:21.433+0000 7fb4b524d700 10 librados: watch_flush exit 2024-02-19T21:17:21.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:21.433+0000 7fb4b524d700 20 client.50609.objecter shutdown clearing up homeless session... 2024-02-19T21:17:21.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:21.433+0000 7fb4b524d700 10 client.50609.objecter successfully canceled tick 2024-02-19T21:17:21.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:21.433+0000 7fb4b524d700 1 -- 172.21.15.163:0/70169480 >> v1:172.21.15.163:6800/1552514681 conn(0x7fb498065f90 legacy=0x7fb498068450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:21.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:21.433+0000 7fb4b524d700 1 -- 172.21.15.163:0/70169480 >> v1:172.21.15.163:6790/0 conn(0x7fb4b01031c0 legacy=0x7fb4b01139c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:21.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:21.433+0000 7fb4b524d700 1 -- 172.21.15.163:0/70169480 shutdown_connections 2024-02-19T21:17:21.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:21.433+0000 7fb4b524d700 1 -- 172.21.15.163:0/70169480 >> 172.21.15.163:0/70169480 conn(0x7fb4b00f24a0 msgr2=0x7fb4b00f5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:21.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:21.433+0000 7fb4b524d700 1 -- 172.21.15.163:0/70169480 shutdown_connections 2024-02-19T21:17:21.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:21.433+0000 7fb4b524d700 1 -- 172.21.15.163:0/70169480 wait complete. 2024-02-19T21:17:21.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:21.433+0000 7fb4b524d700 1 librados: shutdown 2024-02-19T21:17:21.434 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is set 2024-02-19T21:17:21.484 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:21.510 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:20.809284+0000 mgr.y (mgr.14550) 648 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:21.510 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:20.956730+0000 mon.c (mon.2) 440 : audit [INF] from='client.? v1:172.21.15.163:0/70169480' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:21.510 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:20.957013+0000 mon.a (mon.0) 3673 : audit [INF] from='client.50609 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:21.512 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:21.531 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:21.533 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:21.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:21.635 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:21.662 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:21.681 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:21.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:21.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:20.809284+0000 mgr.y (mgr.14550) 648 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:21.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:20.956730+0000 mon.c (mon.2) 440 : audit [INF] from='client.? v1:172.21.15.163:0/70169480' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:21.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:20.957013+0000 mon.a (mon.0) 3673 : audit [INF] from='client.50609 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:21.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:20.809284+0000 mgr.y (mgr.14550) 648 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:21.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:20.956730+0000 mon.c (mon.2) 440 : audit [INF] from='client.? v1:172.21.15.163:0/70169480' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:21.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:20.957013+0000 mon.a (mon.0) 3673 : audit [INF] from='client.50609 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:21.785 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:21.812 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:21.831 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:21.833 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:21.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:21.935 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:17:21.963 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:22.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:21.336963+0000 mgr.y (mgr.14550) 649 : cluster [DBG] pgmap v1182: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s; 204 B/s, 1 objects/s recovering 2024-02-19T21:17:22.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:21.419417+0000 mon.a (mon.0) 3674 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-19T21:17:22.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:21.419507+0000 mon.a (mon.0) 3675 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-19T21:17:22.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:21.420126+0000 mon.a (mon.0) 3676 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:17:22.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:21.429614+0000 mon.a (mon.0) 3677 : audit [INF] from='client.50609 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:17:22.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:21.429679+0000 mon.a (mon.0) 3678 : cluster [DBG] osdmap e781: 8 total, 8 up, 5 in 2024-02-19T21:17:22.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:21.477841+0000 osd.4 (osd.4) 271 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:22.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:21.478218+0000 osd.4 (osd.4) 272 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:22.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:21.336963+0000 mgr.y (mgr.14550) 649 : cluster [DBG] pgmap v1182: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s; 204 B/s, 1 objects/s recovering 2024-02-19T21:17:22.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:21.419417+0000 mon.a (mon.0) 3674 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-19T21:17:22.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:21.419507+0000 mon.a (mon.0) 3675 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-19T21:17:22.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:21.420126+0000 mon.a (mon.0) 3676 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:17:22.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:21.429614+0000 mon.a (mon.0) 3677 : audit [INF] from='client.50609 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:17:22.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:21.429679+0000 mon.a (mon.0) 3678 : cluster [DBG] osdmap e781: 8 total, 8 up, 5 in 2024-02-19T21:17:22.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:21.477841+0000 osd.4 (osd.4) 271 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:22.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:21.478218+0000 osd.4 (osd.4) 272 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:21.336963+0000 mgr.y (mgr.14550) 649 : cluster [DBG] pgmap v1182: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s; 204 B/s, 1 objects/s recovering 2024-02-19T21:17:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:21.419417+0000 mon.a (mon.0) 3674 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-02-19T21:17:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:21.419507+0000 mon.a (mon.0) 3675 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-19T21:17:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:21.420126+0000 mon.a (mon.0) 3676 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:17:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:21.429614+0000 mon.a (mon.0) 3677 : audit [INF] from='client.50609 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-19T21:17:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:21.429679+0000 mon.a (mon.0) 3678 : cluster [DBG] osdmap e781: 8 total, 8 up, 5 in 2024-02-19T21:17:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:21.477841+0000 osd.4 (osd.4) 271 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:22.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:21.478218+0000 osd.4 (osd.4) 272 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:23.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:22.449703+0000 mon.a (mon.0) 3679 : cluster [DBG] osdmap e782: 8 total, 8 up, 5 in 2024-02-19T21:17:23.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:22.449703+0000 mon.a (mon.0) 3679 : cluster [DBG] osdmap e782: 8 total, 8 up, 5 in 2024-02-19T21:17:23.749 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:17:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:17:23] "GET /metrics HTTP/1.1" 200 190193 "" "Prometheus/2.33.4" 2024-02-19T21:17:23.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:22.449703+0000 mon.a (mon.0) 3679 : cluster [DBG] osdmap e782: 8 total, 8 up, 5 in 2024-02-19T21:17:24.216 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:17:24.217 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:24.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:17:24.231 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:24.319 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:24.351 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:24.372 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:24.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:24.476 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:17:24.514 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:24.692 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:17:24.695 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:24.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:17:24.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:23.337536+0000 mgr.y (mgr.14550) 650 : cluster [DBG] pgmap v1185: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s; 204 B/s, 1 objects/s recovering 2024-02-19T21:17:24.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:23.346371+0000 osd.0 (osd.0) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:24.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:23.346647+0000 osd.0 (osd.0) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:24.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:23.455872+0000 mon.a (mon.0) 3680 : cluster [DBG] osdmap e783: 8 total, 8 up, 5 in 2024-02-19T21:17:24.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:24.110377+0000 mon.a (mon.0) 3681 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:24.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:24.421478+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e784: 8 total, 8 up, 5 in 2024-02-19T21:17:24.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:23.337536+0000 mgr.y (mgr.14550) 650 : cluster [DBG] pgmap v1185: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s; 204 B/s, 1 objects/s recovering 2024-02-19T21:17:24.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:23.346371+0000 osd.0 (osd.0) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:24.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:23.346647+0000 osd.0 (osd.0) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:24.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:23.455872+0000 mon.a (mon.0) 3680 : cluster [DBG] osdmap e783: 8 total, 8 up, 5 in 2024-02-19T21:17:24.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:24.110377+0000 mon.a (mon.0) 3681 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:24.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:24.421478+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e784: 8 total, 8 up, 5 in 2024-02-19T21:17:24.796 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:17:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:23.337536+0000 mgr.y (mgr.14550) 650 : cluster [DBG] pgmap v1185: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s; 204 B/s, 1 objects/s recovering 2024-02-19T21:17:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:23.346371+0000 osd.0 (osd.0) 135 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:23.346647+0000 osd.0 (osd.0) 136 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:23.455872+0000 mon.a (mon.0) 3680 : cluster [DBG] osdmap e783: 8 total, 8 up, 5 in 2024-02-19T21:17:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:24.110377+0000 mon.a (mon.0) 3681 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:24.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:24.421478+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e784: 8 total, 8 up, 5 in 2024-02-19T21:17:24.824 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:24.843 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:17:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:24.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:17:24.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.938+0000 7f9982ac8700 1 -- 172.21.15.163:0/3996503794 >> v1:172.21.15.163:6789/0 conn(0x7f997c1042d0 legacy=0x7f997c106770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:24.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.938+0000 7f9982ac8700 1 -- 172.21.15.163:0/3996503794 shutdown_connections 2024-02-19T21:17:24.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.938+0000 7f9982ac8700 1 -- 172.21.15.163:0/3996503794 >> 172.21.15.163:0/3996503794 conn(0x7f997c0f2470 msgr2=0x7f997c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:24.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.938+0000 7f9982ac8700 1 -- 172.21.15.163:0/3996503794 shutdown_connections 2024-02-19T21:17:24.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.938+0000 7f9982ac8700 1 -- 172.21.15.163:0/3996503794 wait complete. 2024-02-19T21:17:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.938+0000 7f9982ac8700 1 librados: starting msgr at 2024-02-19T21:17:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.938+0000 7f9982ac8700 1 librados: starting objecter 2024-02-19T21:17:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.938+0000 7f9982ac8700 1 Processor -- start 2024-02-19T21:17:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.938+0000 7f9982ac8700 1 -- start start 2024-02-19T21:17:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.938+0000 7f9982ac8700 1 librados: setting wanted keys 2024-02-19T21:17:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.938+0000 7f9982ac8700 1 librados: calling monclient init 2024-02-19T21:17:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9982ac8700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f997c1172d0 con 0x7f997c0f94f0 2024-02-19T21:17:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9982ac8700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f997c1184b0 con 0x7f997c0f8940 2024-02-19T21:17:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9982ac8700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f997c1196b0 con 0x7f997c1042d0 2024-02-19T21:17:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f997bfff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f997c0f94f0 0x7f997c0ffb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:53586/0 (socket says 172.21.15.163:53586) 2024-02-19T21:17:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f997bfff700 1 -- 172.21.15.163:0/1256198197 learned_addr learned my addr 172.21.15.163:0/1256198197 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 10 client.?.objecter ms_handle_connect 0x7f997c0f94f0 2024-02-19T21:17:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 10 client.?.objecter ms_handle_connect 0x7f997c1042d0 2024-02-19T21:17:24.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 10 client.?.objecter ms_handle_connect 0x7f997c0f8940 2024-02-19T21:17:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760220220 0 0) 0x7f997c1196b0 con 0x7f997c1042d0 2024-02-19T21:17:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f996c003030 con 0x7f997c1042d0 2024-02-19T21:17:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481528336 0 0) 0x7f997c1172d0 con 0x7f997c0f94f0 2024-02-19T21:17:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f997c1196b0 con 0x7f997c0f94f0 2024-02-19T21:17:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946377052 0 0) 0x7f997c1184b0 con 0x7f997c0f8940 2024-02-19T21:17:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f997c1172d0 con 0x7f997c0f8940 2024-02-19T21:17:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2801149282 0 0) 0x7f996c003030 con 0x7f997c1042d0 2024-02-19T21:17:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f997c1184b0 con 0x7f997c1042d0 2024-02-19T21:17:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f99740016e0 con 0x7f997c1042d0 2024-02-19T21:17:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 816795443 0 0) 0x7f997c1196b0 con 0x7f997c0f94f0 2024-02-19T21:17:24.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f996c003030 con 0x7f997c0f94f0 2024-02-19T21:17:24.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1146212141 0 0) 0x7f997c1172d0 con 0x7f997c0f8940 2024-02-19T21:17:24.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f997c1196b0 con 0x7f997c0f8940 2024-02-19T21:17:24.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9968001840 con 0x7f997c0f94f0 2024-02-19T21:17:24.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9970000fe0 con 0x7f997c0f8940 2024-02-19T21:17:24.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3548043218 0 0) 0x7f997c1184b0 con 0x7f997c1042d0 2024-02-19T21:17:24.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 >> v1:172.21.15.193:6789/0 conn(0x7f997c0f8940 legacy=0x7f997c0ff3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:24.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 >> v1:172.21.15.163:6789/0 conn(0x7f997c0f94f0 legacy=0x7f997c0ffb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:24.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997c11a8b0 con 0x7f997c1042d0 2024-02-19T21:17:24.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.939+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997c1198a0 con 0x7f997c1042d0 2024-02-19T21:17:24.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.940+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9974001da0 con 0x7f997c1042d0 2024-02-19T21:17:24.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.940+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9974003a30 con 0x7f997c1042d0 2024-02-19T21:17:24.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.940+0000 7f9982ac8700 10 client.50615.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:24.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.940+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f997c119d10 con 0x7f997c1042d0 2024-02-19T21:17:24.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.940+0000 7f9982ac8700 1 librados: init done 2024-02-19T21:17:24.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.941+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f997c060350 con 0x7f997c1042d0 2024-02-19T21:17:24.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.941+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f99740021f0 con 0x7f997c1042d0 2024-02-19T21:17:24.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.941+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(784..784 src has 623..784) v4 ==== 8447+0+0 (unknown 440741427 0 0) 0x7f9974076ca0 con 0x7f997c1042d0 2024-02-19T21:17:24.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.941+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 osd_map(784..784 src has 623..784) v4 2024-02-19T21:17:24.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.941+0000 7f9979f76700 3 client.50615.objecter handle_osd_map got epochs [784,784] > 0 2024-02-19T21:17:24.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.941+0000 7f9979f76700 3 client.50615.objecter handle_osd_map decoding full epoch 784 2024-02-19T21:17:24.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.941+0000 7f9979f76700 20 client.50615.objecter dump_active .. 0 homeless 2024-02-19T21:17:24.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.941+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v784) v1 ==== 54+0+16 (unknown 2312917249 0 2420000192) 0x7f99740790b0 con 0x7f997c1042d0 2024-02-19T21:17:24.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.941+0000 7f9979f76700 10 client.50615.objecter ms_handle_connect 0x7f996c063d90 2024-02-19T21:17:24.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9982ac8700 10 client.50615.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:24.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 1 2024-02-19T21:17:24.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c047540 osd=0 3 2024-02-19T21:17:24.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c047540 osd=0 3 2024-02-19T21:17:24.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c047540 osd=0 3 2024-02-19T21:17:24.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 1 2024-02-19T21:17:24.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c047540 osd=0 3 2024-02-19T21:17:24.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 0 1 2024-02-19T21:17:24.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c047540 osd=0 4 2024-02-19T21:17:24.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9982ac8700 10 client.50615.objecter _send_command 1 2024-02-19T21:17:24.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f997c109fa0 con 0x7f997c118640 2024-02-19T21:17:24.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9979f76700 10 client.50615.objecter ms_handle_connect 0x7f997c118640 2024-02-19T21:17:24.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f997c109fa0 con 0x7f997c118640 2024-02-19T21:17:24.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:17:24.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9979f76700 10 client.50615.objecter _finish_command 1 = system:0 2024-02-19T21:17:24.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c047540 osd=0 4 2024-02-19T21:17:24.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.942+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 0 1 2024-02-19T21:17:24.950 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:24.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.957+0000 7f9982ac8700 10 client.50615.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:24.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.957+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 2 2024-02-19T21:17:24.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.957+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c047540 osd=0 3 2024-02-19T21:17:24.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.957+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c047540 osd=0 3 2024-02-19T21:17:24.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.957+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c047540 osd=0 3 2024-02-19T21:17:24.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.957+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 2 2024-02-19T21:17:24.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.957+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c047540 osd=0 3 2024-02-19T21:17:24.958 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.957+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 0 2 2024-02-19T21:17:24.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.957+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c047540 osd=0 4 2024-02-19T21:17:24.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.957+0000 7f9982ac8700 10 client.50615.objecter _send_command 2 2024-02-19T21:17:24.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.957+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f997c119370 con 0x7f997c118640 2024-02-19T21:17:24.959 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f997c119370 con 0x7f997c118640 2024-02-19T21:17:24.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:24.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9979f76700 10 client.50615.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:24.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c047540 osd=0 4 2024-02-19T21:17:24.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 0 2 2024-02-19T21:17:24.960 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:17:24.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9982ac8700 10 client.50615.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:24.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 3 2024-02-19T21:17:24.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c1194e0 osd=1 3 2024-02-19T21:17:24.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c1194e0 osd=1 3 2024-02-19T21:17:24.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c1194e0 osd=1 3 2024-02-19T21:17:24.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 3 2024-02-19T21:17:24.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c1194e0 osd=1 3 2024-02-19T21:17:24.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 1 3 2024-02-19T21:17:24.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c1194e0 osd=1 4 2024-02-19T21:17:24.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9982ac8700 10 client.50615.objecter _send_command 3 2024-02-19T21:17:24.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.958+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f997c059cd0 con 0x7f997c0598f0 2024-02-19T21:17:24.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.959+0000 7f9979f76700 10 client.50615.objecter ms_handle_connect 0x7f997c0598f0 2024-02-19T21:17:24.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.959+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f997c059cd0 con 0x7f997c0598f0 2024-02-19T21:17:24.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.959+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:17:24.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.959+0000 7f9979f76700 10 client.50615.objecter _finish_command 3 = system:0 2024-02-19T21:17:24.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.959+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c1194e0 osd=1 4 2024-02-19T21:17:24.962 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.959+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 1 3 2024-02-19T21:17:24.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.974+0000 7f9982ac8700 10 client.50615.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:24.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.974+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 4 2024-02-19T21:17:24.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.974+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c1194e0 osd=1 3 2024-02-19T21:17:24.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.974+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c1194e0 osd=1 3 2024-02-19T21:17:24.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.974+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c1194e0 osd=1 3 2024-02-19T21:17:24.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.974+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 4 2024-02-19T21:17:24.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.974+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c1194e0 osd=1 3 2024-02-19T21:17:24.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.974+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 1 4 2024-02-19T21:17:24.975 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.974+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c1194e0 osd=1 4 2024-02-19T21:17:24.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.974+0000 7f9982ac8700 10 client.50615.objecter _send_command 4 2024-02-19T21:17:24.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.974+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f997c05a0b0 con 0x7f997c0598f0 2024-02-19T21:17:24.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f997c05a0b0 con 0x7f997c0598f0 2024-02-19T21:17:24.976 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:24.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9979f76700 10 client.50615.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:24.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c1194e0 osd=1 4 2024-02-19T21:17:24.977 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 1 4 2024-02-19T21:17:24.977 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:17:24.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9982ac8700 10 client.50615.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:24.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 5 2024-02-19T21:17:24.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c059360 osd=2 3 2024-02-19T21:17:24.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c059360 osd=2 3 2024-02-19T21:17:24.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c059360 osd=2 3 2024-02-19T21:17:24.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 5 2024-02-19T21:17:24.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c059360 osd=2 3 2024-02-19T21:17:24.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 2 5 2024-02-19T21:17:24.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c059360 osd=2 4 2024-02-19T21:17:24.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9982ac8700 10 client.50615.objecter _send_command 5 2024-02-19T21:17:24.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.975+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f997c0287b0 con 0x7f997c0594d0 2024-02-19T21:17:24.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.976+0000 7f9979f76700 10 client.50615.objecter ms_handle_connect 0x7f997c0594d0 2024-02-19T21:17:24.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.976+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f997c0287b0 con 0x7f997c0594d0 2024-02-19T21:17:24.978 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.976+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:17:24.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.976+0000 7f9979f76700 10 client.50615.objecter _finish_command 5 = system:0 2024-02-19T21:17:24.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.976+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c059360 osd=2 4 2024-02-19T21:17:24.979 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.976+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 2 5 2024-02-19T21:17:24.979 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:24.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.990+0000 7f9982ac8700 10 client.50615.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:24.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.990+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 6 2024-02-19T21:17:24.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.990+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c059360 osd=2 3 2024-02-19T21:17:24.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.990+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c059360 osd=2 3 2024-02-19T21:17:24.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.990+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c059360 osd=2 3 2024-02-19T21:17:24.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.990+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 6 2024-02-19T21:17:24.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.990+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c059360 osd=2 3 2024-02-19T21:17:24.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.990+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 2 6 2024-02-19T21:17:24.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.990+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c059360 osd=2 4 2024-02-19T21:17:24.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.990+0000 7f9982ac8700 10 client.50615.objecter _send_command 6 2024-02-19T21:17:24.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.990+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f997c027d10 con 0x7f997c0594d0 2024-02-19T21:17:24.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f997c027d10 con 0x7f997c0594d0 2024-02-19T21:17:24.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:24.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9979f76700 10 client.50615.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:24.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c059360 osd=2 4 2024-02-19T21:17:24.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 2 6 2024-02-19T21:17:24.994 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:17:24.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9982ac8700 10 client.50615.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:24.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 7 2024-02-19T21:17:24.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c028110 osd=3 3 2024-02-19T21:17:24.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c028110 osd=3 3 2024-02-19T21:17:24.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c028110 osd=3 3 2024-02-19T21:17:24.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 7 2024-02-19T21:17:24.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c028110 osd=3 3 2024-02-19T21:17:24.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 3 7 2024-02-19T21:17:24.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c028110 osd=3 4 2024-02-19T21:17:24.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9982ac8700 10 client.50615.objecter _send_command 7 2024-02-19T21:17:24.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f997c127c20 con 0x7f997c0fa630 2024-02-19T21:17:24.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.992+0000 7f9979f76700 10 client.50615.objecter ms_handle_connect 0x7f997c0fa630 2024-02-19T21:17:24.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.993+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f997c127c20 con 0x7f997c0fa630 2024-02-19T21:17:24.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.993+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:17:24.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.993+0000 7f9979f76700 10 client.50615.objecter _finish_command 7 = system:0 2024-02-19T21:17:24.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.993+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c028110 osd=3 4 2024-02-19T21:17:24.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:24.993+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 3 7 2024-02-19T21:17:24.998 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:25.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:25.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.007+0000 7f9982ac8700 10 client.50615.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:25.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.007+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 8 2024-02-19T21:17:25.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.007+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c028110 osd=3 3 2024-02-19T21:17:25.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.007+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c028110 osd=3 3 2024-02-19T21:17:25.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.007+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c028110 osd=3 3 2024-02-19T21:17:25.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.007+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 8 2024-02-19T21:17:25.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.007+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c028110 osd=3 3 2024-02-19T21:17:25.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.007+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 3 8 2024-02-19T21:17:25.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.007+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c028110 osd=3 4 2024-02-19T21:17:25.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.007+0000 7f9982ac8700 10 client.50615.objecter _send_command 8 2024-02-19T21:17:25.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.007+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f997c0f9b90 con 0x7f997c0fa630 2024-02-19T21:17:25.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f997c0f9b90 con 0x7f997c0fa630 2024-02-19T21:17:25.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:25.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9979f76700 10 client.50615.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:25.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c028110 osd=3 4 2024-02-19T21:17:25.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 3 8 2024-02-19T21:17:25.011 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:17:25.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9982ac8700 10 client.50615.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:25.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 9 2024-02-19T21:17:25.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c0f9b90 osd=4 3 2024-02-19T21:17:25.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c0f9b90 osd=4 3 2024-02-19T21:17:25.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c0f9b90 osd=4 3 2024-02-19T21:17:25.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 9 2024-02-19T21:17:25.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c0f9b90 osd=4 3 2024-02-19T21:17:25.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 4 9 2024-02-19T21:17:25.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c0f9b90 osd=4 4 2024-02-19T21:17:25.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9982ac8700 10 client.50615.objecter _send_command 9 2024-02-19T21:17:25.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.009+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f997c1240f0 con 0x7f997c0f9d40 2024-02-19T21:17:25.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.010+0000 7f9979f76700 10 client.50615.objecter ms_handle_connect 0x7f997c0f9d40 2024-02-19T21:17:25.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.010+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f997c1240f0 con 0x7f997c0f9d40 2024-02-19T21:17:25.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.010+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:17:25.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.010+0000 7f9979f76700 10 client.50615.objecter _finish_command 9 = system:0 2024-02-19T21:17:25.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.010+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c0f9b90 osd=4 4 2024-02-19T21:17:25.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.010+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 4 9 2024-02-19T21:17:25.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.024+0000 7f9982ac8700 10 client.50615.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:25.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.024+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 10 2024-02-19T21:17:25.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.024+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c0f9b90 osd=4 3 2024-02-19T21:17:25.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.024+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c0f9b90 osd=4 3 2024-02-19T21:17:25.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.024+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c0f9b90 osd=4 3 2024-02-19T21:17:25.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.024+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 10 2024-02-19T21:17:25.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.024+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c0f9b90 osd=4 3 2024-02-19T21:17:25.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.024+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 4 10 2024-02-19T21:17:25.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.024+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c0f9b90 osd=4 4 2024-02-19T21:17:25.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.024+0000 7f9982ac8700 10 client.50615.objecter _send_command 10 2024-02-19T21:17:25.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.024+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f997c1240f0 con 0x7f997c0f9d40 2024-02-19T21:17:25.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f997c1240f0 con 0x7f997c0f9d40 2024-02-19T21:17:25.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:25.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9979f76700 10 client.50615.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:25.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c0f9b90 osd=4 4 2024-02-19T21:17:25.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 4 10 2024-02-19T21:17:25.028 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:17:25.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9982ac8700 10 client.50615.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:25.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 11 2024-02-19T21:17:25.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c1240f0 osd=5 3 2024-02-19T21:17:25.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c1240f0 osd=5 3 2024-02-19T21:17:25.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c1240f0 osd=5 3 2024-02-19T21:17:25.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 11 2024-02-19T21:17:25.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c1240f0 osd=5 3 2024-02-19T21:17:25.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 5 11 2024-02-19T21:17:25.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c1240f0 osd=5 4 2024-02-19T21:17:25.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9982ac8700 10 client.50615.objecter _send_command 11 2024-02-19T21:17:25.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.026+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f997c130760 con 0x7f997c12cae0 2024-02-19T21:17:25.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.027+0000 7f9979f76700 10 client.50615.objecter ms_handle_connect 0x7f997c12cae0 2024-02-19T21:17:25.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.028+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f997c130760 con 0x7f997c12cae0 2024-02-19T21:17:25.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.028+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:17:25.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.028+0000 7f9979f76700 10 client.50615.objecter _finish_command 11 = system:0 2024-02-19T21:17:25.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.028+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c1240f0 osd=5 4 2024-02-19T21:17:25.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.028+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 5 11 2024-02-19T21:17:25.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.042+0000 7f9982ac8700 10 client.50615.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.042+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 12 2024-02-19T21:17:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.042+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c1240f0 osd=5 3 2024-02-19T21:17:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.042+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c1240f0 osd=5 3 2024-02-19T21:17:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.042+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c1240f0 osd=5 3 2024-02-19T21:17:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.042+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 12 2024-02-19T21:17:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.042+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c1240f0 osd=5 3 2024-02-19T21:17:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.042+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 5 12 2024-02-19T21:17:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.042+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c1240f0 osd=5 4 2024-02-19T21:17:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.042+0000 7f9982ac8700 10 client.50615.objecter _send_command 12 2024-02-19T21:17:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.042+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f997c130760 con 0x7f997c12cae0 2024-02-19T21:17:25.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f997c130760 con 0x7f997c12cae0 2024-02-19T21:17:25.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:25.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9979f76700 10 client.50615.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:25.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c1240f0 osd=5 4 2024-02-19T21:17:25.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 5 12 2024-02-19T21:17:25.046 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:17:25.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9982ac8700 10 client.50615.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:25.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 13 2024-02-19T21:17:25.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c130760 osd=6 3 2024-02-19T21:17:25.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c130760 osd=6 3 2024-02-19T21:17:25.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c130760 osd=6 3 2024-02-19T21:17:25.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 13 2024-02-19T21:17:25.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c130760 osd=6 3 2024-02-19T21:17:25.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 6 13 2024-02-19T21:17:25.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c130760 osd=6 4 2024-02-19T21:17:25.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9982ac8700 10 client.50615.objecter _send_command 13 2024-02-19T21:17:25.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.044+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f997c131910 con 0x7f997c132460 2024-02-19T21:17:25.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.045+0000 7f9979f76700 10 client.50615.objecter ms_handle_connect 0x7f997c132460 2024-02-19T21:17:25.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.046+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f997c131910 con 0x7f997c132460 2024-02-19T21:17:25.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.046+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:17:25.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.046+0000 7f9979f76700 10 client.50615.objecter _finish_command 13 = system:0 2024-02-19T21:17:25.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.046+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c130760 osd=6 4 2024-02-19T21:17:25.048 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.046+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 6 13 2024-02-19T21:17:25.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.061+0000 7f9982ac8700 10 client.50615.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:25.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.061+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 14 2024-02-19T21:17:25.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.061+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c130760 osd=6 3 2024-02-19T21:17:25.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.061+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c130760 osd=6 3 2024-02-19T21:17:25.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.061+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c130760 osd=6 3 2024-02-19T21:17:25.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.061+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 14 2024-02-19T21:17:25.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.061+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c130760 osd=6 3 2024-02-19T21:17:25.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.061+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 6 14 2024-02-19T21:17:25.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.061+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c130760 osd=6 4 2024-02-19T21:17:25.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.061+0000 7f9982ac8700 10 client.50615.objecter _send_command 14 2024-02-19T21:17:25.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.061+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f997c131910 con 0x7f997c132460 2024-02-19T21:17:25.064 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f997c131910 con 0x7f997c132460 2024-02-19T21:17:25.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:25.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9979f76700 10 client.50615.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:25.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c130760 osd=6 4 2024-02-19T21:17:25.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 6 14 2024-02-19T21:17:25.065 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:17:25.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9982ac8700 10 client.50615.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:25.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 15 2024-02-19T21:17:25.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c131910 osd=7 3 2024-02-19T21:17:25.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c131910 osd=7 3 2024-02-19T21:17:25.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c131910 osd=7 3 2024-02-19T21:17:25.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 15 2024-02-19T21:17:25.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c131910 osd=7 3 2024-02-19T21:17:25.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 7 15 2024-02-19T21:17:25.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c131910 osd=7 4 2024-02-19T21:17:25.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9982ac8700 10 client.50615.objecter _send_command 15 2024-02-19T21:17:25.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.063+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f997c13b1f0 con 0x7f997c131a80 2024-02-19T21:17:25.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.064+0000 7f9979f76700 10 client.50615.objecter ms_handle_connect 0x7f997c131a80 2024-02-19T21:17:25.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.064+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f997c13b1f0 con 0x7f997c131a80 2024-02-19T21:17:25.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.064+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:17:25.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.065+0000 7f9979f76700 10 client.50615.objecter _finish_command 15 = system:0 2024-02-19T21:17:25.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.065+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c131910 osd=7 4 2024-02-19T21:17:25.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.065+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 7 15 2024-02-19T21:17:25.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.079+0000 7f9982ac8700 10 client.50615.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:25.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.079+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign -1 16 2024-02-19T21:17:25.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.079+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c131910 osd=7 3 2024-02-19T21:17:25.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.079+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c131910 osd=7 3 2024-02-19T21:17:25.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.079+0000 7f9982ac8700 20 client.50615.objecter _get_session s=0x7f997c131910 osd=7 3 2024-02-19T21:17:25.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.079+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_remove -1 16 2024-02-19T21:17:25.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.079+0000 7f9982ac8700 20 client.50615.objecter get_session s=0x7f997c131910 osd=7 3 2024-02-19T21:17:25.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.079+0000 7f9982ac8700 15 client.50615.objecter _session_command_op_assign 7 16 2024-02-19T21:17:25.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.079+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c131910 osd=7 4 2024-02-19T21:17:25.080 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.079+0000 7f9982ac8700 10 client.50615.objecter _send_command 16 2024-02-19T21:17:25.081 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.079+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f997c13c9a0 con 0x7f997c131a80 2024-02-19T21:17:25.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.080+0000 7f9979f76700 1 -- 172.21.15.163:0/1256198197 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f997c13c9a0 con 0x7f997c131a80 2024-02-19T21:17:25.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.080+0000 7f9979f76700 10 client.50615.objecter ms_dispatch 0x7f997c001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:25.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.080+0000 7f9979f76700 10 client.50615.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:25.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.080+0000 7f9979f76700 20 client.50615.objecter put_session s=0x7f997c131910 osd=7 4 2024-02-19T21:17:25.082 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.080+0000 7f9979f76700 15 client.50615.objecter _session_command_op_remove 7 16 2024-02-19T21:17:25.082 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:17:25.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 10 librados: watch_flush enter 2024-02-19T21:17:25.083 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 10 librados: watch_flush exit 2024-02-19T21:17:25.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 10 client.50615.objecter close_session for osd.0 2024-02-19T21:17:25.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 >> v1:172.21.15.163:6801/1559418115 conn(0x7f997c118640 legacy=0x7f997c118a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:25.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c047540 osd=0 1 2024-02-19T21:17:25.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 10 client.50615.objecter close_session for osd.1 2024-02-19T21:17:25.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 >> v1:172.21.15.163:6805/3125047432 conn(0x7f997c0598f0 legacy=0x7f997c0273c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:25.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c1194e0 osd=1 1 2024-02-19T21:17:25.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 10 client.50615.objecter close_session for osd.2 2024-02-19T21:17:25.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 >> v1:172.21.15.163:6809/1563158487 conn(0x7f997c0594d0 legacy=0x7f997c121b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:25.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c059360 osd=2 1 2024-02-19T21:17:25.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 10 client.50615.objecter close_session for osd.3 2024-02-19T21:17:25.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 >> v1:172.21.15.163:6813/2112279769 conn(0x7f997c0fa630 legacy=0x7f997c1263f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:25.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c028110 osd=3 1 2024-02-19T21:17:25.084 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 10 client.50615.objecter close_session for osd.4 2024-02-19T21:17:25.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 >> v1:172.21.15.193:6800/3129716965 conn(0x7f997c0f9d40 legacy=0x7f997c127c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:25.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c0f9b90 osd=4 1 2024-02-19T21:17:25.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 10 client.50615.objecter close_session for osd.5 2024-02-19T21:17:25.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 >> v1:172.21.15.193:6804/805161991 conn(0x7f997c12cae0 legacy=0x7f997c12eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:25.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9981065700 1 -- 172.21.15.163:0/1256198197 reap_dead start 2024-02-19T21:17:25.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9981065700 1 -- 172.21.15.163:0/1256198197 reap_dead start 2024-02-19T21:17:25.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c1240f0 osd=5 1 2024-02-19T21:17:25.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 10 client.50615.objecter close_session for osd.6 2024-02-19T21:17:25.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 >> v1:172.21.15.193:6808/787604615 conn(0x7f997c132460 legacy=0x7f997c1328e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:25.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c130760 osd=6 1 2024-02-19T21:17:25.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 10 client.50615.objecter close_session for osd.7 2024-02-19T21:17:25.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 >> v1:172.21.15.193:6812/157927432 conn(0x7f997c131a80 legacy=0x7f997c139a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:25.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 20 client.50615.objecter put_session s=0x7f997c131910 osd=7 1 2024-02-19T21:17:25.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 20 client.50615.objecter shutdown clearing up homeless session... 2024-02-19T21:17:25.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 10 client.50615.objecter successfully canceled tick 2024-02-19T21:17:25.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9981065700 1 -- 172.21.15.163:0/1256198197 reap_dead start 2024-02-19T21:17:25.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 >> v1:172.21.15.163:6800/1552514681 conn(0x7f996c063d90 legacy=0x7f996c066250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:25.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 >> v1:172.21.15.163:6790/0 conn(0x7f997c1042d0 legacy=0x7f997c115990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:25.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 shutdown_connections 2024-02-19T21:17:25.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 >> 172.21.15.163:0/1256198197 conn(0x7f997c0f2470 msgr2=0x7f997c103990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:25.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 shutdown_connections 2024-02-19T21:17:25.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 1 -- 172.21.15.163:0/1256198197 wait complete. 2024-02-19T21:17:25.086 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:25.081+0000 7f9982ac8700 1 librados: shutdown 2024-02-19T21:17:25.086 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:17:25.086 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:17:25.086 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:17:25.086 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:17:25.086 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:17:25.086 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:17:25.086 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:17:25.087 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:17:25.102 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:17:25.129 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:25.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:24.942632+0000 mon.c (mon.2) 441 : audit [DBG] from='client.? v1:172.21.15.163:0/1256198197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:17:25.829 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:24.942632+0000 mon.c (mon.2) 441 : audit [DBG] from='client.? v1:172.21.15.163:0/1256198197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:17:25.830 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:24.942632+0000 mon.c (mon.2) 441 : audit [DBG] from='client.? v1:172.21.15.163:0/1256198197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:17:26.179 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:17:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:26.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:17:26.216 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:17:26.283 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:17:26.309 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:26.461 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:17:26.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:26.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:17:26.482 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:17:26.482 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:17:26.565 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:26.592 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:26.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:25.338380+0000 mgr.y (mgr.14550) 651 : cluster [DBG] pgmap v1188: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:17:26.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:25.338380+0000 mgr.y (mgr.14550) 651 : cluster [DBG] pgmap v1188: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:17:26.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:25.338380+0000 mgr.y (mgr.14550) 651 : cluster [DBG] pgmap v1188: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:17:27.169 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:27.173 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:27.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:27.274 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:17:27.303 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:27.401 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:17:27.404 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:27.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:17:27.505 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:27.537 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:27.558 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:27.561 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:27.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:27.662 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:27.692 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:27.712 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:27.714 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:27.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:27.816 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:27.822 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:17:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:17:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:17:27.843 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:27.941 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:27.944 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:27.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:27.948 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-19T21:17:28.045 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:17:28.073 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:28.192 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:28.200 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-19T21:17:28.204 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:17:28.210 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:28.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:17:28.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.232+0000 7f63d3969700 1 -- 172.21.15.163:0/167093693 >> v1:172.21.15.163:6789/0 conn(0x7f63cc1031c0 legacy=0x7f63cc105680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:28.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.233+0000 7f63d3969700 1 -- 172.21.15.163:0/167093693 shutdown_connections 2024-02-19T21:17:28.235 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.233+0000 7f63d3969700 1 -- 172.21.15.163:0/167093693 >> 172.21.15.163:0/167093693 conn(0x7f63cc06cc00 msgr2=0x7f63cc0f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:28.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.238+0000 7f63d3969700 1 -- 172.21.15.163:0/167093693 shutdown_connections 2024-02-19T21:17:28.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.238+0000 7f63d3969700 1 -- 172.21.15.163:0/167093693 wait complete. 2024-02-19T21:17:28.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.238+0000 7f63d3969700 1 librados: starting msgr at 2024-02-19T21:17:28.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.238+0000 7f63d3969700 1 librados: starting objecter 2024-02-19T21:17:28.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.239+0000 7f63d3969700 1 Processor -- start 2024-02-19T21:17:28.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.239+0000 7f63d3969700 1 -- start start 2024-02-19T21:17:28.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.239+0000 7f63d3969700 1 librados: setting wanted keys 2024-02-19T21:17:28.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.239+0000 7f63d3969700 1 librados: calling monclient init 2024-02-19T21:17:28.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.239+0000 7f63d3969700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f63cc116af0 con 0x7f63cc112f50 2024-02-19T21:17:28.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.239+0000 7f63d3969700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f63cc117c90 con 0x7f63cc1031c0 2024-02-19T21:17:28.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.239+0000 7f63d3969700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f63cc118e70 con 0x7f63cc0f9350 2024-02-19T21:17:28.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.239+0000 7f63d1f06700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f63cc112f50 0x7f63cc115340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:53594/0 (socket says 172.21.15.163:53594) 2024-02-19T21:17:28.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.239+0000 7f63d1f06700 1 -- 172.21.15.163:0/2533965742 learned_addr learned my addr 172.21.15.163:0/2533965742 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:28.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 10 client.?.objecter ms_handle_connect 0x7f63cc112f50 2024-02-19T21:17:28.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:28.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 10 client.?.objecter ms_handle_connect 0x7f63cc0f9350 2024-02-19T21:17:28.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:28.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698333414 0 0) 0x7f63cc116af0 con 0x7f63cc112f50 2024-02-19T21:17:28.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b8002220 con 0x7f63cc112f50 2024-02-19T21:17:28.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653119426 0 0) 0x7f63cc118e70 con 0x7f63cc0f9350 2024-02-19T21:17:28.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63cc116af0 con 0x7f63cc0f9350 2024-02-19T21:17:28.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1343502770 0 0) 0x7f63b8002220 con 0x7f63cc112f50 2024-02-19T21:17:28.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63cc118e70 con 0x7f63cc112f50 2024-02-19T21:17:28.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f63c4003210 con 0x7f63cc112f50 2024-02-19T21:17:28.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2050982549 0 0) 0x7f63cc116af0 con 0x7f63cc0f9350 2024-02-19T21:17:28.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63b8002220 con 0x7f63cc0f9350 2024-02-19T21:17:28.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f63bc001720 con 0x7f63cc0f9350 2024-02-19T21:17:28.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2547374970 0 0) 0x7f63cc118e70 con 0x7f63cc112f50 2024-02-19T21:17:28.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 >> v1:172.21.15.163:6790/0 conn(0x7f63cc0f9350 legacy=0x7f63cc111830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:28.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 >> v1:172.21.15.193:6789/0 conn(0x7f63cc1031c0 legacy=0x7f63cc0f8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:28.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc11a070 con 0x7f63cc112f50 2024-02-19T21:17:28.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63d3969700 1 -- 172.21.15.163:0/2533965742 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc119060 con 0x7f63cc112f50 2024-02-19T21:17:28.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63d3969700 10 client.23892.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:28.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63d3969700 1 -- 172.21.15.163:0/2533965742 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc1194d0 con 0x7f63cc112f50 2024-02-19T21:17:28.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.240+0000 7f63d3969700 1 librados: init done 2024-02-19T21:17:28.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.241+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f63c4001ea0 con 0x7f63cc112f50 2024-02-19T21:17:28.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.241+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f63c4003cf0 con 0x7f63cc112f50 2024-02-19T21:17:28.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.242+0000 7f63d3969700 1 -- 172.21.15.163:0/2533965742 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b0004f20 con 0x7f63cc112f50 2024-02-19T21:17:28.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.242+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f63c4003ef0 con 0x7f63cc112f50 2024-02-19T21:17:28.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.242+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(784..784 src has 623..784) v4 ==== 8447+0+0 (unknown 440741427 0 0) 0x7f63c40795c0 con 0x7f63cc112f50 2024-02-19T21:17:28.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.242+0000 7f63c27fc700 10 client.23892.objecter ms_dispatch 0x7f63cc001550 osd_map(784..784 src has 623..784) v4 2024-02-19T21:17:28.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.242+0000 7f63c27fc700 3 client.23892.objecter handle_osd_map got epochs [784,784] > 0 2024-02-19T21:17:28.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.242+0000 7f63c27fc700 3 client.23892.objecter handle_osd_map decoding full epoch 784 2024-02-19T21:17:28.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.243+0000 7f63c27fc700 20 client.23892.objecter dump_active .. 0 homeless 2024-02-19T21:17:28.246 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.243+0000 7f63c27fc700 10 client.23892.objecter ms_handle_connect 0x7f63b8063ba0 2024-02-19T21:17:28.248 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.247+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f63c404c910 con 0x7f63cc112f50 2024-02-19T21:17:28.312 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:28.343 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:28.365 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:28.367 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:28.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:28.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.436+0000 7f63d3969700 1 -- 172.21.15.163:0/2533965742 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd in", "ids": ["2"]} v 0) v1 -- 0x7f63b00051d0 con 0x7f63cc112f50 2024-02-19T21:17:28.468 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:17:28.487 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:17:28.437+0000 7fb606608700 -1 mon.a@0(leader).osd e784 definitely_dead 0 2024-02-19T21:17:28.496 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:28.520 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:17:28.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.520+0000 7f63c27fc700 1 -- 172.21.15.163:0/2533965742 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd in", "ids": ["2"]}]=0 marked in osd.2. v785) v1 ==== 85+0+0 (unknown 4025675048 0 0) 0x7f63c404ad80 con 0x7f63cc112f50 2024-02-19T21:17:28.525 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:28.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:17:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.526+0000 7f63d3969700 10 librados: watch_flush enter 2024-02-19T21:17:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.526+0000 7f63d3969700 10 librados: watch_flush exit 2024-02-19T21:17:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.526+0000 7f63d3969700 20 client.23892.objecter shutdown clearing up homeless session... 2024-02-19T21:17:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.526+0000 7f63d3969700 10 client.23892.objecter successfully canceled tick 2024-02-19T21:17:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.527+0000 7f63d3969700 1 -- 172.21.15.163:0/2533965742 >> v1:172.21.15.163:6800/1552514681 conn(0x7f63b8063ba0 legacy=0x7f63b8066060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:28.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.527+0000 7f63d3969700 1 -- 172.21.15.163:0/2533965742 >> v1:172.21.15.163:6789/0 conn(0x7f63cc112f50 legacy=0x7f63cc115340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:28.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.537+0000 7f63d3969700 1 -- 172.21.15.163:0/2533965742 shutdown_connections 2024-02-19T21:17:28.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.537+0000 7f63d3969700 1 -- 172.21.15.163:0/2533965742 >> 172.21.15.163:0/2533965742 conn(0x7f63cc06cc00 msgr2=0x7f63cc106670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:28.547 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:17:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:17:28+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 2024-02-19T21:17:28.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.546+0000 7f63d3969700 1 -- 172.21.15.163:0/2533965742 shutdown_connections 2024-02-19T21:17:28.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.551+0000 7f63d3969700 1 -- 172.21.15.163:0/2533965742 wait complete. 2024-02-19T21:17:28.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:28.551+0000 7f63d3969700 1 librados: shutdown 2024-02-19T21:17:28.556 INFO:teuthology.orchestra.run.smithi163.stderr:marked in osd.2. 2024-02-19T21:17:28.626 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:28.656 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:28.676 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:28.679 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:28.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:28.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:27.338790+0000 mgr.y (mgr.14550) 652 : cluster [DBG] pgmap v1189: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 866 B/s rd, 0 op/s 2024-02-19T21:17:28.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:28.438389+0000 mon.a (mon.0) 3683 : audit [DBG] from='client.? v1:172.21.15.163:0/2533965742' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-19T21:17:28.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:27.338790+0000 mgr.y (mgr.14550) 652 : cluster [DBG] pgmap v1189: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 866 B/s rd, 0 op/s 2024-02-19T21:17:28.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:28.438389+0000 mon.a (mon.0) 3683 : audit [DBG] from='client.? v1:172.21.15.163:0/2533965742' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-19T21:17:28.781 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:28.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:27.338790+0000 mgr.y (mgr.14550) 652 : cluster [DBG] pgmap v1189: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 866 B/s rd, 0 op/s 2024-02-19T21:17:28.806 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:28.438389+0000 mon.a (mon.0) 3683 : audit [DBG] from='client.? v1:172.21.15.163:0/2533965742' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-02-19T21:17:28.808 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:28.827 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:28.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:28.931 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:28.958 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:28.977 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:28.979 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:28.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:29.081 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:29.107 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:29.125 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:29.127 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:29.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:29.229 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:29.257 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:29.310 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:29.313 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:29.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:29.388 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:29.414 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:29.447 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:29.469 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:29.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:29.574 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:29.600 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:29.618 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:29.621 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:29.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:29.722 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:17:29.747 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:29.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:28.517168+0000 mon.a (mon.0) 3684 : audit [INF] from='client.? v1:172.21.15.163:0/2533965742' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-02-19T21:17:29.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:28.517232+0000 mon.a (mon.0) 3685 : cluster [DBG] osdmap e785: 8 total, 8 up, 6 in 2024-02-19T21:17:29.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:29.194525+0000 osd.7 (osd.7) 335 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:29.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:29.194705+0000 osd.7 (osd.7) 336 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:29.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:29.420231+0000 mon.a (mon.0) 3686 : cluster [DBG] osdmap e786: 8 total, 8 up, 6 in 2024-02-19T21:17:29.965 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:28.517168+0000 mon.a (mon.0) 3684 : audit [INF] from='client.? v1:172.21.15.163:0/2533965742' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-02-19T21:17:29.966 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:28.517232+0000 mon.a (mon.0) 3685 : cluster [DBG] osdmap e785: 8 total, 8 up, 6 in 2024-02-19T21:17:29.966 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:29.194525+0000 osd.7 (osd.7) 335 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:29.966 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:29.194705+0000 osd.7 (osd.7) 336 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:29.966 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:29.420231+0000 mon.a (mon.0) 3686 : cluster [DBG] osdmap e786: 8 total, 8 up, 6 in 2024-02-19T21:17:29.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:28.517168+0000 mon.a (mon.0) 3684 : audit [INF] from='client.? v1:172.21.15.163:0/2533965742' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-02-19T21:17:29.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:28.517232+0000 mon.a (mon.0) 3685 : cluster [DBG] osdmap e785: 8 total, 8 up, 6 in 2024-02-19T21:17:29.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:29.194525+0000 osd.7 (osd.7) 335 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:29.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:29.194705+0000 osd.7 (osd.7) 336 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:29.966 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:29.420231+0000 mon.a (mon.0) 3686 : cluster [DBG] osdmap e786: 8 total, 8 up, 6 in 2024-02-19T21:17:29.983 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-02-19T21:17:29.997 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:17:30.000 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:30.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:17:30.101 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:17:30.127 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:30.147 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:17:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:30.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:17:30.258 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:30.288 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:30.309 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:30.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:30.412 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:30.441 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:30.461 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:30.464 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:30.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:30.565 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:17:30.591 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:30.610 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:17:30.612 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:30.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:17:30.714 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:17:30.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:29.339205+0000 mgr.y (mgr.14550) 653 : cluster [DBG] pgmap v1191: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:17:30.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:29.546912+0000 mon.a (mon.0) 3687 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:30.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:30.013778+0000 mon.a (mon.0) 3688 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:30.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:30.434555+0000 mon.a (mon.0) 3689 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:17:30.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:30.434613+0000 mon.a (mon.0) 3690 : cluster [DBG] osdmap e787: 8 total, 8 up, 6 in 2024-02-19T21:17:30.732 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:30.437066+0000 mon.a (mon.0) 3691 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-136"}]: dispatch 2024-02-19T21:17:30.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:29.339205+0000 mgr.y (mgr.14550) 653 : cluster [DBG] pgmap v1191: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:17:30.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:29.546912+0000 mon.a (mon.0) 3687 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:30.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:30.013778+0000 mon.a (mon.0) 3688 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:30.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:30.434555+0000 mon.a (mon.0) 3689 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:17:30.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:30.434613+0000 mon.a (mon.0) 3690 : cluster [DBG] osdmap e787: 8 total, 8 up, 6 in 2024-02-19T21:17:30.732 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:30.437066+0000 mon.a (mon.0) 3691 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-136"}]: dispatch 2024-02-19T21:17:30.742 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:30.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:29.339205+0000 mgr.y (mgr.14550) 653 : cluster [DBG] pgmap v1191: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:17:30.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:29.546912+0000 mon.a (mon.0) 3687 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:30.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:30.013778+0000 mon.a (mon.0) 3688 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:30.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:30.434555+0000 mon.a (mon.0) 3689 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:17:30.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:30.434613+0000 mon.a (mon.0) 3690 : cluster [DBG] osdmap e787: 8 total, 8 up, 6 in 2024-02-19T21:17:30.814 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:30.437066+0000 mon.a (mon.0) 3691 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-136"}]: dispatch 2024-02-19T21:17:31.072 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:17:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:17:31.091 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:17:31.096 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:31.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:17:31.197 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:17:31.225 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:31.606 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:31.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.631+0000 7f82ec50b700 1 -- 172.21.15.163:0/2999990436 >> v1:172.21.15.163:6790/0 conn(0x7f82e40fd980 legacy=0x7f82e40fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:31.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.632+0000 7f82ec50b700 1 -- 172.21.15.163:0/2999990436 shutdown_connections 2024-02-19T21:17:31.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.632+0000 7f82ec50b700 1 -- 172.21.15.163:0/2999990436 >> 172.21.15.163:0/2999990436 conn(0x7f82e40f35a0 msgr2=0x7f82e40f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:31.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.632+0000 7f82ec50b700 1 -- 172.21.15.163:0/2999990436 shutdown_connections 2024-02-19T21:17:31.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.632+0000 7f82ec50b700 1 -- 172.21.15.163:0/2999990436 wait complete. 2024-02-19T21:17:31.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.632+0000 7f82ec50b700 1 librados: starting msgr at 2024-02-19T21:17:31.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.632+0000 7f82ec50b700 1 librados: starting objecter 2024-02-19T21:17:31.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.632+0000 7f82ec50b700 1 Processor -- start 2024-02-19T21:17:31.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.632+0000 7f82ec50b700 1 -- start start 2024-02-19T21:17:31.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.632+0000 7f82ec50b700 1 librados: setting wanted keys 2024-02-19T21:17:31.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.632+0000 7f82ec50b700 1 librados: calling monclient init 2024-02-19T21:17:31.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.632+0000 7f82ec50b700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f82e40fcf70 con 0x7f82e41042c0 2024-02-19T21:17:31.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82ec50b700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f82e40fd140 con 0x7f82e40fd980 2024-02-19T21:17:31.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82ec50b700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f82e40fd350 con 0x7f82e4100780 2024-02-19T21:17:31.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82e9aa6700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f82e4100780 0x7f82e4114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:35550/0 (socket says 172.21.15.163:35550) 2024-02-19T21:17:31.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82eaaa8700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f82e41042c0 0x7f82e4118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:53620/0 (socket says 172.21.15.163:53620) 2024-02-19T21:17:31.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82e9aa6700 1 -- 172.21.15.163:0/1686020717 learned_addr learned my addr 172.21.15.163:0/1686020717 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:31.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 10 client.?.objecter ms_handle_connect 0x7f82e4100780 2024-02-19T21:17:31.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:31.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 10 client.?.objecter ms_handle_connect 0x7f82e41042c0 2024-02-19T21:17:31.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:31.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 10 client.?.objecter ms_handle_connect 0x7f82e40fd980 2024-02-19T21:17:31.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:31.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059925132 0 0) 0x7f82e40fd350 con 0x7f82e4100780 2024-02-19T21:17:31.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d00038e0 con 0x7f82e4100780 2024-02-19T21:17:31.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470422526 0 0) 0x7f82e40fd140 con 0x7f82e40fd980 2024-02-19T21:17:31.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82e40fd350 con 0x7f82e40fd980 2024-02-19T21:17:31.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2849475347 0 0) 0x7f82d00038e0 con 0x7f82e4100780 2024-02-19T21:17:31.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82e40fd140 con 0x7f82e4100780 2024-02-19T21:17:31.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f82d4001920 con 0x7f82e4100780 2024-02-19T21:17:31.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094257218 0 0) 0x7f82e40fcf70 con 0x7f82e41042c0 2024-02-19T21:17:31.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d00038e0 con 0x7f82e41042c0 2024-02-19T21:17:31.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4270765648 0 0) 0x7f82e40fd350 con 0x7f82e40fd980 2024-02-19T21:17:31.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82e40fcf70 con 0x7f82e40fd980 2024-02-19T21:17:31.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f82e0002d10 con 0x7f82e40fd980 2024-02-19T21:17:31.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3013907781 0 0) 0x7f82e40fd140 con 0x7f82e4100780 2024-02-19T21:17:31.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 >> v1:172.21.15.193:6789/0 conn(0x7f82e40fd980 legacy=0x7f82e40fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:31.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 >> v1:172.21.15.163:6789/0 conn(0x7f82e41042c0 legacy=0x7f82e4118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:31.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e411cb90 con 0x7f82e4100780 2024-02-19T21:17:31.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82ec50b700 1 -- 172.21.15.163:0/1686020717 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e4119bc0 con 0x7f82e4100780 2024-02-19T21:17:31.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82ec50b700 10 client.50624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:31.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82ec50b700 1 -- 172.21.15.163:0/1686020717 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e411a030 con 0x7f82e4100780 2024-02-19T21:17:31.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82ec50b700 1 librados: init done 2024-02-19T21:17:31.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.633+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f82d4001c80 con 0x7f82e4100780 2024-02-19T21:17:31.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.634+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f82d4005bf0 con 0x7f82e4100780 2024-02-19T21:17:31.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.634+0000 7f82ec50b700 1 -- 172.21.15.163:0/1686020717 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c8004f20 con 0x7f82e4100780 2024-02-19T21:17:31.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.635+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f82d4002270 con 0x7f82e4100780 2024-02-19T21:17:31.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.635+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(788..788 src has 623..788) v4 ==== 8413+0+0 (unknown 2462615378 0 0) 0x7f82d4078ee0 con 0x7f82e4100780 2024-02-19T21:17:31.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.635+0000 7f82db7fe700 10 client.50624.objecter ms_dispatch 0x7f82e4001550 osd_map(788..788 src has 623..788) v4 2024-02-19T21:17:31.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.635+0000 7f82db7fe700 3 client.50624.objecter handle_osd_map got epochs [788,788] > 0 2024-02-19T21:17:31.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.635+0000 7f82db7fe700 3 client.50624.objecter handle_osd_map decoding full epoch 788 2024-02-19T21:17:31.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.635+0000 7f82db7fe700 20 client.50624.objecter dump_active .. 0 homeless 2024-02-19T21:17:31.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.636+0000 7f82db7fe700 10 client.50624.objecter ms_handle_connect 0x7f82d0063b60 2024-02-19T21:17:31.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.640+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f82d404c170 con 0x7f82e4100780 2024-02-19T21:17:31.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:30.813732+0000 mgr.y (mgr.14550) 654 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:31.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:31.436332+0000 mon.a (mon.0) 3692 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-136"}]': finished 2024-02-19T21:17:31.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:31.436389+0000 mon.a (mon.0) 3693 : cluster [DBG] osdmap e788: 8 total, 8 up, 6 in 2024-02-19T21:17:31.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:31.458372+0000 mon.a (mon.0) 3694 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:31.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:31.458652+0000 mon.a (mon.0) 3695 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-136"}]: dispatch 2024-02-19T21:17:31.823 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:31.822+0000 7f82ec50b700 1 -- 172.21.15.163:0/1686020717 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7f82c80051d0 con 0x7f82e4100780 2024-02-19T21:17:31.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:30.813732+0000 mgr.y (mgr.14550) 654 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:31.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:31.436332+0000 mon.a (mon.0) 3692 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-136"}]': finished 2024-02-19T21:17:31.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:31.436389+0000 mon.a (mon.0) 3693 : cluster [DBG] osdmap e788: 8 total, 8 up, 6 in 2024-02-19T21:17:32.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:31.458372+0000 mon.a (mon.0) 3694 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:32.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:31.458652+0000 mon.a (mon.0) 3695 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-136"}]: dispatch 2024-02-19T21:17:32.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:30.813732+0000 mgr.y (mgr.14550) 654 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:32.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:31.436332+0000 mon.a (mon.0) 3692 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-136"}]': finished 2024-02-19T21:17:32.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:31.436389+0000 mon.a (mon.0) 3693 : cluster [DBG] osdmap e788: 8 total, 8 up, 6 in 2024-02-19T21:17:32.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:31.458372+0000 mon.a (mon.0) 3694 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:32.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:31.458652+0000 mon.a (mon.0) 3695 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-136"}]: dispatch 2024-02-19T21:17:32.175 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:17:32.179 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:32.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:17:32.280 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:32.313 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:32.333 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:32.336 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:32.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:32.437 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:32.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:32.443+0000 7f82db7fe700 1 -- 172.21.15.163:0/1686020717 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v789) v1 ==== 91+0+0 (unknown 3208424421 0 0) 0x7f82d404a580 con 0x7f82e4100780 2024-02-19T21:17:32.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:32.445+0000 7f82ec50b700 10 librados: watch_flush enter 2024-02-19T21:17:32.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:32.445+0000 7f82ec50b700 10 librados: watch_flush exit 2024-02-19T21:17:32.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:32.445+0000 7f82ec50b700 20 client.50624.objecter shutdown clearing up homeless session... 2024-02-19T21:17:32.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:32.445+0000 7f82ec50b700 10 client.50624.objecter successfully canceled tick 2024-02-19T21:17:32.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:32.446+0000 7f82ec50b700 1 -- 172.21.15.163:0/1686020717 >> v1:172.21.15.163:6800/1552514681 conn(0x7f82d0063b60 legacy=0x7f82d0066020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:32.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:32.446+0000 7f82ec50b700 1 -- 172.21.15.163:0/1686020717 >> v1:172.21.15.163:6790/0 conn(0x7f82e4100780 legacy=0x7f82e4114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:32.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:32.446+0000 7f82ec50b700 1 -- 172.21.15.163:0/1686020717 shutdown_connections 2024-02-19T21:17:32.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:32.446+0000 7f82ec50b700 1 -- 172.21.15.163:0/1686020717 >> 172.21.15.163:0/1686020717 conn(0x7f82e40f35a0 msgr2=0x7f82e4107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:32.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:32.446+0000 7f82ec50b700 1 -- 172.21.15.163:0/1686020717 shutdown_connections 2024-02-19T21:17:32.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:32.446+0000 7f82ec50b700 1 -- 172.21.15.163:0/1686020717 wait complete. 2024-02-19T21:17:32.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:32.446+0000 7f82ec50b700 1 librados: shutdown 2024-02-19T21:17:32.449 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:17:32.468 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:32.487 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:32.490 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:32.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:32.591 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:32.618 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:32.636 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:32.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:32.740 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:17:32.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:31.339593+0000 mgr.y (mgr.14550) 655 : cluster [DBG] pgmap v1194: 214 pgs: 1 active+recovering, 1 active+remapped+backfill_toofull, 212 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3/711 objects misplaced (0.422%); 651 KiB/s, 14 objects/s recovering 2024-02-19T21:17:32.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:31.823886+0000 mon.c (mon.2) 442 : audit [INF] from='client.? v1:172.21.15.163:0/1686020717' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:17:32.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:31.824298+0000 mon.a (mon.0) 3696 : audit [INF] from='client.50624 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:17:32.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:32.433072+0000 mon.a (mon.0) 3697 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:17:32.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:32.437610+0000 mon.a (mon.0) 3698 : audit [INF] from='client.50624 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:17:32.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:32.437670+0000 mon.a (mon.0) 3699 : cluster [DBG] osdmap e789: 8 total, 8 up, 6 in 2024-02-19T21:17:32.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:31.339593+0000 mgr.y (mgr.14550) 655 : cluster [DBG] pgmap v1194: 214 pgs: 1 active+recovering, 1 active+remapped+backfill_toofull, 212 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3/711 objects misplaced (0.422%); 651 KiB/s, 14 objects/s recovering 2024-02-19T21:17:32.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:31.823886+0000 mon.c (mon.2) 442 : audit [INF] from='client.? v1:172.21.15.163:0/1686020717' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:17:32.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:31.824298+0000 mon.a (mon.0) 3696 : audit [INF] from='client.50624 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:17:32.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:32.433072+0000 mon.a (mon.0) 3697 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:17:32.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:32.437610+0000 mon.a (mon.0) 3698 : audit [INF] from='client.50624 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:17:32.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:32.437670+0000 mon.a (mon.0) 3699 : cluster [DBG] osdmap e789: 8 total, 8 up, 6 in 2024-02-19T21:17:32.771 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:32.799 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:17:32.802 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:32.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:17:32.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:32.930 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:33.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:31.339593+0000 mgr.y (mgr.14550) 655 : cluster [DBG] pgmap v1194: 214 pgs: 1 active+recovering, 1 active+remapped+backfill_toofull, 212 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3/711 objects misplaced (0.422%); 651 KiB/s, 14 objects/s recovering 2024-02-19T21:17:33.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:31.823886+0000 mon.c (mon.2) 442 : audit [INF] from='client.? v1:172.21.15.163:0/1686020717' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:17:33.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:31.824298+0000 mon.a (mon.0) 3696 : audit [INF] from='client.50624 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:17:33.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:32.433072+0000 mon.a (mon.0) 3697 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:17:33.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:32.437610+0000 mon.a (mon.0) 3698 : audit [INF] from='client.50624 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:17:33.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:32.437670+0000 mon.a (mon.0) 3699 : cluster [DBG] osdmap e789: 8 total, 8 up, 6 in 2024-02-19T21:17:33.332 INFO:teuthology.orchestra.run.smithi163.stdout:ERROR: (22) Invalid argument 2024-02-19T21:17:33.333 INFO:teuthology.orchestra.run.smithi163.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:17:33.614 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:33.617 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:17:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:17:33] "GET /metrics HTTP/1.1" 200 192773 "" "Prometheus/2.33.4" 2024-02-19T21:17:33.618 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:33.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:33.719 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:33.746 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:33.340182+0000 mgr.y (mgr.14550) 656 : cluster [DBG] pgmap v1197: 182 pgs: 1 active+recovering, 1 active+remapped+backfill_toofull, 180 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3/705 objects misplaced (0.426%); 977 KiB/s, 20 objects/s recovering 2024-02-19T21:17:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:33.438907+0000 mon.a (mon.0) 3700 : cluster [DBG] osdmap e790: 8 total, 8 up, 6 in 2024-02-19T21:17:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:33.440699+0000 mon.a (mon.0) 3701 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:17:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:33.450382+0000 osd.1 (osd.1) 354 : cluster [DBG] 16.13 scrub starts 2024-02-19T21:17:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:33.453341+0000 osd.1 (osd.1) 355 : cluster [DBG] 16.13 scrub ok 2024-02-19T21:17:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:33.691568+0000 osd.6 (osd.6) 275 : cluster [DBG] 6.a scrub starts 2024-02-19T21:17:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:33.692534+0000 osd.6 (osd.6) 276 : cluster [DBG] 6.a scrub ok 2024-02-19T21:17:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:33.702194+0000 mon.a (mon.0) 3702 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-19T21:17:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:33.743120+0000 osd.2 (osd.2) 304 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:17:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:33.746013+0000 osd.2 (osd.2) 305 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:17:34.906 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:33.340182+0000 mgr.y (mgr.14550) 656 : cluster [DBG] pgmap v1197: 182 pgs: 1 active+recovering, 1 active+remapped+backfill_toofull, 180 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3/705 objects misplaced (0.426%); 977 KiB/s, 20 objects/s recovering 2024-02-19T21:17:34.906 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:33.438907+0000 mon.a (mon.0) 3700 : cluster [DBG] osdmap e790: 8 total, 8 up, 6 in 2024-02-19T21:17:34.906 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:33.440699+0000 mon.a (mon.0) 3701 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:17:34.906 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:33.450382+0000 osd.1 (osd.1) 354 : cluster [DBG] 16.13 scrub starts 2024-02-19T21:17:34.906 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:33.453341+0000 osd.1 (osd.1) 355 : cluster [DBG] 16.13 scrub ok 2024-02-19T21:17:34.906 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:33.691568+0000 osd.6 (osd.6) 275 : cluster [DBG] 6.a scrub starts 2024-02-19T21:17:34.906 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:33.692534+0000 osd.6 (osd.6) 276 : cluster [DBG] 6.a scrub ok 2024-02-19T21:17:34.906 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:33.702194+0000 mon.a (mon.0) 3702 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-19T21:17:34.906 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:33.743120+0000 osd.2 (osd.2) 304 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:17:34.906 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:33.746013+0000 osd.2 (osd.2) 305 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:17:34.906 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:33.340182+0000 mgr.y (mgr.14550) 656 : cluster [DBG] pgmap v1197: 182 pgs: 1 active+recovering, 1 active+remapped+backfill_toofull, 180 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3/705 objects misplaced (0.426%); 977 KiB/s, 20 objects/s recovering 2024-02-19T21:17:34.907 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:33.438907+0000 mon.a (mon.0) 3700 : cluster [DBG] osdmap e790: 8 total, 8 up, 6 in 2024-02-19T21:17:34.907 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:33.440699+0000 mon.a (mon.0) 3701 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:17:34.907 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:33.450382+0000 osd.1 (osd.1) 354 : cluster [DBG] 16.13 scrub starts 2024-02-19T21:17:34.907 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:33.453341+0000 osd.1 (osd.1) 355 : cluster [DBG] 16.13 scrub ok 2024-02-19T21:17:34.907 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:33.691568+0000 osd.6 (osd.6) 275 : cluster [DBG] 6.a scrub starts 2024-02-19T21:17:34.907 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:33.692534+0000 osd.6 (osd.6) 276 : cluster [DBG] 6.a scrub ok 2024-02-19T21:17:34.907 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:33.702194+0000 mon.a (mon.0) 3702 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-02-19T21:17:34.907 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:33.743120+0000 osd.2 (osd.2) 304 : cluster [DBG] 6.9 scrub starts 2024-02-19T21:17:34.907 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:33.746013+0000 osd.2 (osd.2) 305 : cluster [DBG] 6.9 scrub ok 2024-02-19T21:17:34.925 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-19T21:17:34.984 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 6, 0, 2] out_osds: [5, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-19T21:17:34.984 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-19T21:17:34.984 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.meta 2024-02-19T21:17:34.984 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:17:35.197 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:35.200 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:35.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:35.301 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:17:35.332 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:35.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.405+0000 7f4b59232700 1 -- 172.21.15.163:0/1969649720 >> v1:172.21.15.163:6789/0 conn(0x7f4b541031b0 legacy=0x7f4b54105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:35.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.405+0000 7f4b59232700 1 -- 172.21.15.163:0/1969649720 shutdown_connections 2024-02-19T21:17:35.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.405+0000 7f4b59232700 1 -- 172.21.15.163:0/1969649720 >> 172.21.15.163:0/1969649720 conn(0x7f4b540f2470 msgr2=0x7f4b540f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:35.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.405+0000 7f4b59232700 1 -- 172.21.15.163:0/1969649720 shutdown_connections 2024-02-19T21:17:35.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.405+0000 7f4b59232700 1 -- 172.21.15.163:0/1969649720 wait complete. 2024-02-19T21:17:35.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.405+0000 7f4b59232700 1 librados: starting msgr at 2024-02-19T21:17:35.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.405+0000 7f4b59232700 1 librados: starting objecter 2024-02-19T21:17:35.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.406+0000 7f4b59232700 1 Processor -- start 2024-02-19T21:17:35.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.406+0000 7f4b59232700 1 -- start start 2024-02-19T21:17:35.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.406+0000 7f4b59232700 1 librados: setting wanted keys 2024-02-19T21:17:35.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.406+0000 7f4b59232700 1 librados: calling monclient init 2024-02-19T21:17:35.429 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.406+0000 7f4b59232700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b54119570 con 0x7f4b540fc870 2024-02-19T21:17:35.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.406+0000 7f4b59232700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b5411a750 con 0x7f4b541031b0 2024-02-19T21:17:35.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.406+0000 7f4b59232700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4b5411b950 con 0x7f4b540ff670 2024-02-19T21:17:35.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.406+0000 7f4b5259c700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f4b540ff670 0x7f4b54061190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38108/0 (socket says 172.21.15.163:38108) 2024-02-19T21:17:35.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.406+0000 7f4b5259c700 1 -- 172.21.15.163:0/1676639998 learned_addr learned my addr 172.21.15.163:0/1676639998 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:35.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.406+0000 7f4b52d9d700 1 --1- 172.21.15.163:0/1676639998 >> v1:172.21.15.163:6789/0 conn(0x7f4b540fc870 0x7f4b54060a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43564/0 (socket says 172.21.15.163:43564) 2024-02-19T21:17:35.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 10 client.?.objecter ms_handle_connect 0x7f4b540fc870 2024-02-19T21:17:35.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:35.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 10 client.?.objecter ms_handle_connect 0x7f4b540ff670 2024-02-19T21:17:35.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:35.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 10 client.?.objecter ms_handle_connect 0x7f4b541031b0 2024-02-19T21:17:35.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:35.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196060412 0 0) 0x7f4b54119570 con 0x7f4b540fc870 2024-02-19T21:17:35.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b3c0038e0 con 0x7f4b540fc870 2024-02-19T21:17:35.430 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072986507 0 0) 0x7f4b5411b950 con 0x7f4b540ff670 2024-02-19T21:17:35.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b54119570 con 0x7f4b540ff670 2024-02-19T21:17:35.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130565932 0 0) 0x7f4b5411a750 con 0x7f4b541031b0 2024-02-19T21:17:35.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b5411b950 con 0x7f4b541031b0 2024-02-19T21:17:35.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2645763295 0 0) 0x7f4b3c0038e0 con 0x7f4b540fc870 2024-02-19T21:17:35.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b5411a750 con 0x7f4b540fc870 2024-02-19T21:17:35.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4b440019b0 con 0x7f4b540fc870 2024-02-19T21:17:35.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 765510692 0 0) 0x7f4b54119570 con 0x7f4b540ff670 2024-02-19T21:17:35.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b3c0038e0 con 0x7f4b540ff670 2024-02-19T21:17:35.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4b48001840 con 0x7f4b540ff670 2024-02-19T21:17:35.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2230009187 0 0) 0x7f4b5411b950 con 0x7f4b541031b0 2024-02-19T21:17:35.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b54119570 con 0x7f4b541031b0 2024-02-19T21:17:35.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4b4c001b70 con 0x7f4b541031b0 2024-02-19T21:17:35.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4013901337 0 0) 0x7f4b5411a750 con 0x7f4b540fc870 2024-02-19T21:17:35.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 >> v1:172.21.15.163:6790/0 conn(0x7f4b540ff670 legacy=0x7f4b54061190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:35.431 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 >> v1:172.21.15.193:6789/0 conn(0x7f4b541031b0 legacy=0x7f4b54117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:35.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5411cb50 con 0x7f4b540fc870 2024-02-19T21:17:35.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5411bb40 con 0x7f4b540fc870 2024-02-19T21:17:35.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b59232700 10 client.23904.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:35.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5411bfb0 con 0x7f4b540fc870 2024-02-19T21:17:35.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b59232700 1 librados: init done 2024-02-19T21:17:35.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f4b44002450 con 0x7f4b540fc870 2024-02-19T21:17:35.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.407+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4b44005ae0 con 0x7f4b540fc870 2024-02-19T21:17:35.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.408+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f4b40004f00 con 0x7f4b540fc870 2024-02-19T21:17:35.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f4b44001fb0 con 0x7f4b540fc870 2024-02-19T21:17:35.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(791..791 src has 623..791) v4 ==== 8426+0+0 (unknown 2365389265 0 0) 0x7f4b44078e30 con 0x7f4b540fc870 2024-02-19T21:17:35.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 osd_map(791..791 src has 623..791) v4 2024-02-19T21:17:35.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b3bfff700 3 client.23904.objecter handle_osd_map got epochs [791,791] > 0 2024-02-19T21:17:35.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b3bfff700 3 client.23904.objecter handle_osd_map decoding full epoch 791 2024-02-19T21:17:35.432 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b3bfff700 20 client.23904.objecter dump_active .. 0 homeless 2024-02-19T21:17:35.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v791) v1 ==== 54+0+16 (unknown 3176943710 0 2420000192) 0x7f4b4407b240 con 0x7f4b540fc870 2024-02-19T21:17:35.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b3bfff700 10 client.23904.objecter ms_handle_connect 0x7f4b3c063cc0 2024-02-19T21:17:35.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b59232700 10 client.23904.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:35.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 1 2024-02-19T21:17:35.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b400051a0 osd=0 3 2024-02-19T21:17:35.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b400051a0 osd=0 3 2024-02-19T21:17:35.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b400051a0 osd=0 3 2024-02-19T21:17:35.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 1 2024-02-19T21:17:35.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b400051a0 osd=0 3 2024-02-19T21:17:35.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 0 1 2024-02-19T21:17:35.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b400051a0 osd=0 4 2024-02-19T21:17:35.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b59232700 10 client.23904.objecter _send_command 1 2024-02-19T21:17:35.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.409+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b40004950 con 0x7f4b40004530 2024-02-19T21:17:35.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.410+0000 7f4b3bfff700 10 client.23904.objecter ms_handle_connect 0x7f4b40004530 2024-02-19T21:17:35.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.410+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4b40004950 con 0x7f4b40004530 2024-02-19T21:17:35.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.410+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:17:35.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.410+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 1 = system:0 2024-02-19T21:17:35.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.410+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b400051a0 osd=0 4 2024-02-19T21:17:35.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.410+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 0 1 2024-02-19T21:17:35.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.425+0000 7f4b59232700 10 client.23904.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:35.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.425+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 2 2024-02-19T21:17:35.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.425+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b400051a0 osd=0 3 2024-02-19T21:17:35.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.425+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b400051a0 osd=0 3 2024-02-19T21:17:35.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.425+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b400051a0 osd=0 3 2024-02-19T21:17:35.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.425+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 2 2024-02-19T21:17:35.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.425+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b400051a0 osd=0 3 2024-02-19T21:17:35.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.425+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 0 2 2024-02-19T21:17:35.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.425+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b400051a0 osd=0 4 2024-02-19T21:17:35.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.425+0000 7f4b59232700 10 client.23904.objecter _send_command 2 2024-02-19T21:17:35.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.425+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4b40004d30 con 0x7f4b40004530 2024-02-19T21:17:35.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4b40004d30 con 0x7f4b40004530 2024-02-19T21:17:35.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:35.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.435 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:17:35.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b400051a0 osd=0 4 2024-02-19T21:17:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 0 2 2024-02-19T21:17:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b59232700 10 client.23904.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 3 2024-02-19T21:17:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40003fe0 osd=1 3 2024-02-19T21:17:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40003fe0 osd=1 3 2024-02-19T21:17:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40003fe0 osd=1 3 2024-02-19T21:17:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 3 2024-02-19T21:17:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b40003fe0 osd=1 3 2024-02-19T21:17:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 1 3 2024-02-19T21:17:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40003fe0 osd=1 4 2024-02-19T21:17:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b59232700 10 client.23904.objecter _send_command 3 2024-02-19T21:17:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b4000c000 con 0x7f4b40004150 2024-02-19T21:17:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.427+0000 7f4b3bfff700 10 client.23904.objecter ms_handle_connect 0x7f4b40004150 2024-02-19T21:17:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.428+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4b4000c000 con 0x7f4b40004150 2024-02-19T21:17:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.428+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:17:35.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.428+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 3 = system:0 2024-02-19T21:17:35.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.428+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b40003fe0 osd=1 4 2024-02-19T21:17:35.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.428+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 1 3 2024-02-19T21:17:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:35.437 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:17:35.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:17:35.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.450+0000 7f4b59232700 10 client.23904.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:35.451 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.450+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 4 2024-02-19T21:17:35.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.450+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40003fe0 osd=1 3 2024-02-19T21:17:35.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.450+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40003fe0 osd=1 3 2024-02-19T21:17:35.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.450+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40003fe0 osd=1 3 2024-02-19T21:17:35.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.450+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 4 2024-02-19T21:17:35.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.450+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b40003fe0 osd=1 3 2024-02-19T21:17:35.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.450+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 1 4 2024-02-19T21:17:35.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.450+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40003fe0 osd=1 4 2024-02-19T21:17:35.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.450+0000 7f4b59232700 10 client.23904.objecter _send_command 4 2024-02-19T21:17:35.452 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.450+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4b40001850 con 0x7f4b40004150 2024-02-19T21:17:35.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.452+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4b40001850 con 0x7f4b40004150 2024-02-19T21:17:35.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.452+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:35.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.452+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.452+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b40003fe0 osd=1 4 2024-02-19T21:17:35.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.452+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 1 4 2024-02-19T21:17:35.454 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:17:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.453+0000 7f4b59232700 10 client.23904.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.453+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 5 2024-02-19T21:17:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.453+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40003630 osd=2 3 2024-02-19T21:17:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.453+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40003630 osd=2 3 2024-02-19T21:17:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.453+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40003630 osd=2 3 2024-02-19T21:17:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.453+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 5 2024-02-19T21:17:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.453+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b40003630 osd=2 3 2024-02-19T21:17:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.453+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 2 5 2024-02-19T21:17:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.453+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40003630 osd=2 4 2024-02-19T21:17:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.453+0000 7f4b59232700 10 client.23904.objecter _send_command 5 2024-02-19T21:17:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.453+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b40003e70 con 0x7f4b4000d2f0 2024-02-19T21:17:35.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.453+0000 7f4b3bfff700 10 client.23904.objecter ms_handle_connect 0x7f4b4000d2f0 2024-02-19T21:17:35.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.454+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4b40003e70 con 0x7f4b4000d2f0 2024-02-19T21:17:35.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.454+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:17:35.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.454+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 5 = system:0 2024-02-19T21:17:35.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.454+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b40003630 osd=2 4 2024-02-19T21:17:35.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.454+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 2 5 2024-02-19T21:17:35.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.472+0000 7f4b59232700 10 client.23904.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:35.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.472+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 6 2024-02-19T21:17:35.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.472+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40003630 osd=2 3 2024-02-19T21:17:35.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.472+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40003630 osd=2 3 2024-02-19T21:17:35.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.472+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40003630 osd=2 3 2024-02-19T21:17:35.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.472+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 6 2024-02-19T21:17:35.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.472+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b40003630 osd=2 3 2024-02-19T21:17:35.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.472+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 2 6 2024-02-19T21:17:35.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.472+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40003630 osd=2 4 2024-02-19T21:17:35.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.472+0000 7f4b59232700 10 client.23904.objecter _send_command 6 2024-02-19T21:17:35.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.472+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4b4000e6e0 con 0x7f4b4000d2f0 2024-02-19T21:17:35.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.475+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4b4000e6e0 con 0x7f4b4000d2f0 2024-02-19T21:17:35.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.475+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:35.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.475+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.475+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b40003630 osd=2 4 2024-02-19T21:17:35.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.475+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 2 6 2024-02-19T21:17:35.478 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:17:35.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.477+0000 7f4b59232700 10 client.23904.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:35.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.477+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 7 2024-02-19T21:17:35.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.477+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40003440 osd=3 3 2024-02-19T21:17:35.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.477+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40003440 osd=3 3 2024-02-19T21:17:35.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.477+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40003440 osd=3 3 2024-02-19T21:17:35.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.477+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 7 2024-02-19T21:17:35.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.477+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b40003440 osd=3 3 2024-02-19T21:17:35.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.477+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 3 7 2024-02-19T21:17:35.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.477+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40003440 osd=3 4 2024-02-19T21:17:35.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.477+0000 7f4b59232700 10 client.23904.objecter _send_command 7 2024-02-19T21:17:35.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.477+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b40015b40 con 0x7f4b40011e80 2024-02-19T21:17:35.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.477+0000 7f4b3bfff700 10 client.23904.objecter ms_handle_connect 0x7f4b40011e80 2024-02-19T21:17:35.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.478+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4b40015b40 con 0x7f4b40011e80 2024-02-19T21:17:35.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.478+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:17:35.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.478+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 7 = system:0 2024-02-19T21:17:35.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.478+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b40003440 osd=3 4 2024-02-19T21:17:35.480 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.478+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 3 7 2024-02-19T21:17:35.491 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:35.491 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:35.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.496+0000 7f4b59232700 10 client.23904.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:35.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.496+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 8 2024-02-19T21:17:35.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.496+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40003440 osd=3 3 2024-02-19T21:17:35.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.496+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40003440 osd=3 3 2024-02-19T21:17:35.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.496+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40003440 osd=3 3 2024-02-19T21:17:35.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.496+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 8 2024-02-19T21:17:35.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.496+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b40003440 osd=3 3 2024-02-19T21:17:35.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.496+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 3 8 2024-02-19T21:17:35.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.496+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40003440 osd=3 4 2024-02-19T21:17:35.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.496+0000 7f4b59232700 10 client.23904.objecter _send_command 8 2024-02-19T21:17:35.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.496+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4b4000e120 con 0x7f4b40011e80 2024-02-19T21:17:35.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4b4000e120 con 0x7f4b40011e80 2024-02-19T21:17:35.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:35.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b40003440 osd=3 4 2024-02-19T21:17:35.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 3 8 2024-02-19T21:17:35.500 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:17:35.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b59232700 10 client.23904.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:35.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 9 2024-02-19T21:17:35.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b4000e2e0 osd=4 3 2024-02-19T21:17:35.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b4000e2e0 osd=4 3 2024-02-19T21:17:35.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b4000e2e0 osd=4 3 2024-02-19T21:17:35.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 9 2024-02-19T21:17:35.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b4000e2e0 osd=4 3 2024-02-19T21:17:35.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 4 9 2024-02-19T21:17:35.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b4000e2e0 osd=4 4 2024-02-19T21:17:35.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b59232700 10 client.23904.objecter _send_command 9 2024-02-19T21:17:35.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.498+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b4000e490 con 0x7f4b4000ce50 2024-02-19T21:17:35.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.499+0000 7f4b3bfff700 10 client.23904.objecter ms_handle_connect 0x7f4b4000ce50 2024-02-19T21:17:35.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.499+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4b4000e490 con 0x7f4b4000ce50 2024-02-19T21:17:35.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.499+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:17:35.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.499+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 9 = system:0 2024-02-19T21:17:35.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.499+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b4000e2e0 osd=4 4 2024-02-19T21:17:35.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.499+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 4 9 2024-02-19T21:17:35.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.523+0000 7f4b59232700 10 client.23904.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:35.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.523+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 10 2024-02-19T21:17:35.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.523+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b4000e2e0 osd=4 3 2024-02-19T21:17:35.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.523+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b4000e2e0 osd=4 3 2024-02-19T21:17:35.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.523+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b4000e2e0 osd=4 3 2024-02-19T21:17:35.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.523+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 10 2024-02-19T21:17:35.524 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.523+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b4000e2e0 osd=4 3 2024-02-19T21:17:35.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.523+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 4 10 2024-02-19T21:17:35.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.523+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b4000e2e0 osd=4 4 2024-02-19T21:17:35.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.523+0000 7f4b59232700 10 client.23904.objecter _send_command 10 2024-02-19T21:17:35.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.523+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4b4000e490 con 0x7f4b4000ce50 2024-02-19T21:17:35.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4b4000e490 con 0x7f4b4000ce50 2024-02-19T21:17:35.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:35.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b4000e2e0 osd=4 4 2024-02-19T21:17:35.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 4 10 2024-02-19T21:17:35.526 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:17:35.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b59232700 10 client.23904.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:35.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 11 2024-02-19T21:17:35.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b4000e490 osd=5 3 2024-02-19T21:17:35.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b4000e490 osd=5 3 2024-02-19T21:17:35.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b4000e490 osd=5 3 2024-02-19T21:17:35.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 11 2024-02-19T21:17:35.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b4000e490 osd=5 3 2024-02-19T21:17:35.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 5 11 2024-02-19T21:17:35.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b4000e490 osd=5 4 2024-02-19T21:17:35.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b59232700 10 client.23904.objecter _send_command 11 2024-02-19T21:17:35.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b40018040 con 0x7f4b40017a50 2024-02-19T21:17:35.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.525+0000 7f4b3bfff700 10 client.23904.objecter ms_handle_connect 0x7f4b40017a50 2024-02-19T21:17:35.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.526+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4b40018040 con 0x7f4b40017a50 2024-02-19T21:17:35.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.526+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:17:35.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.526+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 11 = system:0 2024-02-19T21:17:35.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.526+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b4000e490 osd=5 4 2024-02-19T21:17:35.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.526+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 5 11 2024-02-19T21:17:35.539 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:35.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.543+0000 7f4b59232700 10 client.23904.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:35.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.543+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 12 2024-02-19T21:17:35.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.543+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b4000e490 osd=5 3 2024-02-19T21:17:35.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.543+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b4000e490 osd=5 3 2024-02-19T21:17:35.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.543+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b4000e490 osd=5 3 2024-02-19T21:17:35.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.543+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 12 2024-02-19T21:17:35.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.543+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b4000e490 osd=5 3 2024-02-19T21:17:35.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.543+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 5 12 2024-02-19T21:17:35.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.543+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b4000e490 osd=5 4 2024-02-19T21:17:35.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.543+0000 7f4b59232700 10 client.23904.objecter _send_command 12 2024-02-19T21:17:35.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.543+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4b40018040 con 0x7f4b40017a50 2024-02-19T21:17:35.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4b40018040 con 0x7f4b40017a50 2024-02-19T21:17:35.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:35.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b4000e490 osd=5 4 2024-02-19T21:17:35.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 5 12 2024-02-19T21:17:35.547 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:17:35.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b59232700 10 client.23904.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:35.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 13 2024-02-19T21:17:35.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40018040 osd=6 3 2024-02-19T21:17:35.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40018040 osd=6 3 2024-02-19T21:17:35.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40018040 osd=6 3 2024-02-19T21:17:35.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 13 2024-02-19T21:17:35.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b40018040 osd=6 3 2024-02-19T21:17:35.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 6 13 2024-02-19T21:17:35.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40018040 osd=6 4 2024-02-19T21:17:35.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b59232700 10 client.23904.objecter _send_command 13 2024-02-19T21:17:35.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.545+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b40022c80 con 0x7f4b4001f140 2024-02-19T21:17:35.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.546+0000 7f4b3bfff700 10 client.23904.objecter ms_handle_connect 0x7f4b4001f140 2024-02-19T21:17:35.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.547+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4b40022c80 con 0x7f4b4001f140 2024-02-19T21:17:35.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.547+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:17:35.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.547+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 13 = system:0 2024-02-19T21:17:35.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.547+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b40018040 osd=6 4 2024-02-19T21:17:35.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.547+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 6 13 2024-02-19T21:17:35.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.563+0000 7f4b59232700 10 client.23904.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:35.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.563+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 14 2024-02-19T21:17:35.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.563+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40018040 osd=6 3 2024-02-19T21:17:35.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.563+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40018040 osd=6 3 2024-02-19T21:17:35.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.563+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40018040 osd=6 3 2024-02-19T21:17:35.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.563+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 14 2024-02-19T21:17:35.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.563+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b40018040 osd=6 3 2024-02-19T21:17:35.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.563+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 6 14 2024-02-19T21:17:35.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.563+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40018040 osd=6 4 2024-02-19T21:17:35.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.563+0000 7f4b59232700 10 client.23904.objecter _send_command 14 2024-02-19T21:17:35.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.563+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4b40024430 con 0x7f4b4001f140 2024-02-19T21:17:35.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4b40024430 con 0x7f4b4001f140 2024-02-19T21:17:35.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:35.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b40018040 osd=6 4 2024-02-19T21:17:35.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 6 14 2024-02-19T21:17:35.567 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:17:35.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b59232700 10 client.23904.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:35.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 15 2024-02-19T21:17:35.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40024430 osd=7 3 2024-02-19T21:17:35.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40024430 osd=7 3 2024-02-19T21:17:35.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40024430 osd=7 3 2024-02-19T21:17:35.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 15 2024-02-19T21:17:35.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b40024430 osd=7 3 2024-02-19T21:17:35.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 7 15 2024-02-19T21:17:35.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40024430 osd=7 4 2024-02-19T21:17:35.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b59232700 10 client.23904.objecter _send_command 15 2024-02-19T21:17:35.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.565+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b40025040 con 0x7f4b40024ab0 2024-02-19T21:17:35.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.566+0000 7f4b3bfff700 10 client.23904.objecter ms_handle_connect 0x7f4b40024ab0 2024-02-19T21:17:35.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.567+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4b40025040 con 0x7f4b40024ab0 2024-02-19T21:17:35.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.567+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:17:35.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.567+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 15 = system:0 2024-02-19T21:17:35.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.567+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b40024430 osd=7 4 2024-02-19T21:17:35.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.567+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 7 15 2024-02-19T21:17:35.572 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:35.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.581+0000 7f4b59232700 10 client.23904.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-02-19T21:17:35.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.581+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign -1 16 2024-02-19T21:17:35.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.581+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40024430 osd=7 3 2024-02-19T21:17:35.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.581+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40024430 osd=7 3 2024-02-19T21:17:35.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.581+0000 7f4b59232700 20 client.23904.objecter _get_session s=0x7f4b40024430 osd=7 3 2024-02-19T21:17:35.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.581+0000 7f4b59232700 15 client.23904.objecter _session_command_op_remove -1 16 2024-02-19T21:17:35.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.581+0000 7f4b59232700 20 client.23904.objecter get_session s=0x7f4b40024430 osd=7 3 2024-02-19T21:17:35.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.581+0000 7f4b59232700 15 client.23904.objecter _session_command_op_assign 7 16 2024-02-19T21:17:35.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.581+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40024430 osd=7 4 2024-02-19T21:17:35.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.581+0000 7f4b59232700 10 client.23904.objecter _send_command 16 2024-02-19T21:17:35.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.581+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) v1 -- 0x7f4b40025040 con 0x7f4b40024ab0 2024-02-19T21:17:35.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b3bfff700 1 -- 172.21.15.163:0/1676639998 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 ==== 39+0+3 (unknown 3822221901 0 2423995997) 0x7f4b40025040 con 0x7f4b40024ab0 2024-02-19T21:17:35.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b3bfff700 10 client.23904.objecter ms_dispatch 0x7f4b54001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'true' ) v1 2024-02-19T21:17:35.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b3bfff700 10 client.23904.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b3bfff700 20 client.23904.objecter put_session s=0x7f4b40024430 osd=7 4 2024-02-19T21:17:35.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b3bfff700 15 client.23904.objecter _session_command_op_remove 7 16 2024-02-19T21:17:35.585 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:17:35.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 10 librados: watch_flush enter 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 10 librados: watch_flush exit 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 10 client.23904.objecter close_session for osd.0 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 >> v1:172.21.15.163:6801/1559418115 conn(0x7f4b40004530 legacy=0x7f4b40002750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b400051a0 osd=0 1 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 10 client.23904.objecter close_session for osd.1 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 >> v1:172.21.15.163:6805/3125047432 conn(0x7f4b40004150 legacy=0x7f4b4000a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40003fe0 osd=1 1 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 10 client.23904.objecter close_session for osd.2 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 >> v1:172.21.15.163:6809/1563158487 conn(0x7f4b4000d2f0 legacy=0x7f4b4000d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40003630 osd=2 1 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 10 client.23904.objecter close_session for osd.3 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 >> v1:172.21.15.163:6813/2112279769 conn(0x7f4b40011e80 legacy=0x7f4b40014290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b5359e700 1 -- 172.21.15.163:0/1676639998 reap_dead start 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40003440 osd=3 1 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 10 client.23904.objecter close_session for osd.4 2024-02-19T21:17:35.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 >> v1:172.21.15.193:6800/3129716965 conn(0x7f4b4000ce50 legacy=0x7f4b40015cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b4000e2e0 osd=4 1 2024-02-19T21:17:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 10 client.23904.objecter close_session for osd.5 2024-02-19T21:17:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 >> v1:172.21.15.193:6804/805161991 conn(0x7f4b40017a50 legacy=0x7f4b4001d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b4000e490 osd=5 1 2024-02-19T21:17:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 10 client.23904.objecter close_session for osd.6 2024-02-19T21:17:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 >> v1:172.21.15.193:6808/787604615 conn(0x7f4b4001f140 legacy=0x7f4b40021530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40018040 osd=6 1 2024-02-19T21:17:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 10 client.23904.objecter close_session for osd.7 2024-02-19T21:17:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 >> v1:172.21.15.193:6812/157927432 conn(0x7f4b40024ab0 legacy=0x7f4b40022c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 20 client.23904.objecter put_session s=0x7f4b40024430 osd=7 1 2024-02-19T21:17:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 20 client.23904.objecter shutdown clearing up homeless session... 2024-02-19T21:17:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.583+0000 7f4b59232700 10 client.23904.objecter successfully canceled tick 2024-02-19T21:17:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.584+0000 7f4b5359e700 1 -- 172.21.15.163:0/1676639998 reap_dead start 2024-02-19T21:17:35.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.584+0000 7f4b5359e700 1 -- 172.21.15.163:0/1676639998 reap_dead start 2024-02-19T21:17:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.584+0000 7f4b5359e700 1 -- 172.21.15.163:0/1676639998 reap_dead start 2024-02-19T21:17:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.584+0000 7f4b5359e700 1 -- 172.21.15.163:0/1676639998 reap_dead start 2024-02-19T21:17:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.584+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4b3c063cc0 legacy=0x7f4b3c066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.584+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 >> v1:172.21.15.163:6789/0 conn(0x7f4b540fc870 legacy=0x7f4b54060a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.584+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 shutdown_connections 2024-02-19T21:17:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.584+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 >> 172.21.15.163:0/1676639998 conn(0x7f4b540f2470 msgr2=0x7f4b54106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.584+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 shutdown_connections 2024-02-19T21:17:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.584+0000 7f4b59232700 1 -- 172.21.15.163:0/1676639998 wait complete. 2024-02-19T21:17:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:35.584+0000 7f4b59232700 1 librados: shutdown 2024-02-19T21:17:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.589 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.590 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.590 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.590 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'true' 2024-02-19T21:17:35.593 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:35.596 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:35.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:35.697 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:35.728 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:35.749 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:35.751 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:35.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:35.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:34.443206+0000 mon.a (mon.0) 3703 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:17:35.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:34.443355+0000 mon.a (mon.0) 3704 : cluster [DBG] osdmap e791: 8 total, 8 up, 6 in 2024-02-19T21:17:35.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:34.514885+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:17:35.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:34.647335+0000 osd.6 (osd.6) 277 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:34.647481+0000 osd.6 (osd.6) 278 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:34.661290+0000 osd.6 (osd.6) 279 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:17:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:34.664760+0000 osd.6 (osd.6) 280 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:17:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:34.768273+0000 osd.2 (osd.2) 306 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:17:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:34.768982+0000 osd.2 (osd.2) 307 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:17:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:35.410513+0000 mon.a (mon.0) 3706 : audit [DBG] from='client.? v1:172.21.15.163:0/1676639998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:17:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:35.442371+0000 mon.a (mon.0) 3707 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-138", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:17:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:35.442432+0000 mon.a (mon.0) 3708 : cluster [DBG] osdmap e792: 8 total, 8 up, 6 in 2024-02-19T21:17:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:35.453546+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-138"}]: dispatch 2024-02-19T21:17:35.852 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:35.881 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:35.900 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:35.903 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:35.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:35.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:34.443206+0000 mon.a (mon.0) 3703 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:17:35.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:34.443355+0000 mon.a (mon.0) 3704 : cluster [DBG] osdmap e791: 8 total, 8 up, 6 in 2024-02-19T21:17:35.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:34.514885+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:17:35.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:34.647335+0000 osd.6 (osd.6) 277 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:35.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:34.647481+0000 osd.6 (osd.6) 278 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:35.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:34.661290+0000 osd.6 (osd.6) 279 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:17:35.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:34.664760+0000 osd.6 (osd.6) 280 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:17:35.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:34.768273+0000 osd.2 (osd.2) 306 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:17:35.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:34.768982+0000 osd.2 (osd.2) 307 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:17:35.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:35.410513+0000 mon.a (mon.0) 3706 : audit [DBG] from='client.? v1:172.21.15.163:0/1676639998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:17:35.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:35.442371+0000 mon.a (mon.0) 3707 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-138", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:17:35.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:35.442432+0000 mon.a (mon.0) 3708 : cluster [DBG] osdmap e792: 8 total, 8 up, 6 in 2024-02-19T21:17:35.991 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:35.453546+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-138"}]: dispatch 2024-02-19T21:17:35.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:34.443206+0000 mon.a (mon.0) 3703 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:17:35.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:34.443355+0000 mon.a (mon.0) 3704 : cluster [DBG] osdmap e791: 8 total, 8 up, 6 in 2024-02-19T21:17:35.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:34.514885+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-02-19T21:17:35.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:34.647335+0000 osd.6 (osd.6) 277 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:35.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:34.647481+0000 osd.6 (osd.6) 278 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:35.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:34.661290+0000 osd.6 (osd.6) 279 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:17:35.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:34.664760+0000 osd.6 (osd.6) 280 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:17:35.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:34.768273+0000 osd.2 (osd.2) 306 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:17:35.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:34.768982+0000 osd.2 (osd.2) 307 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:17:35.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:35.410513+0000 mon.a (mon.0) 3706 : audit [DBG] from='client.? v1:172.21.15.163:0/1676639998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:17:35.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:35.442371+0000 mon.a (mon.0) 3707 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-138", "force_nonempty": "--force-nonempty" }]': finished 2024-02-19T21:17:35.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:35.442432+0000 mon.a (mon.0) 3708 : cluster [DBG] osdmap e792: 8 total, 8 up, 6 in 2024-02-19T21:17:35.992 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:35.453546+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-138"}]: dispatch 2024-02-19T21:17:36.004 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:36.031 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:36.050 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:36.053 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:36.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:36.154 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:17:36.182 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:36.244 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:17:36.731 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:17:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:36.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:17:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:35.340986+0000 mgr.y (mgr.14550) 657 : cluster [DBG] pgmap v1200: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:17:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:35.631777+0000 osd.6 (osd.6) 281 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:17:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:35.634652+0000 osd.6 (osd.6) 282 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:17:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:35.746819+0000 osd.2 (osd.2) 308 : cluster [DBG] 5.b scrub starts 2024-02-19T21:17:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:35.747585+0000 osd.2 (osd.2) 309 : cluster [DBG] 5.b scrub ok 2024-02-19T21:17:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:36.354180+0000 osd.4 (osd.4) 273 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:17:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:36.354990+0000 osd.4 (osd.4) 274 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:17:36.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:36.443108+0000 mon.a (mon.0) 3710 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-138"}]': finished 2024-02-19T21:17:36.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:36.443172+0000 mon.a (mon.0) 3711 : cluster [DBG] osdmap e793: 8 total, 8 up, 6 in 2024-02-19T21:17:36.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:36.461578+0000 mon.a (mon.0) 3712 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:36.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:36.462005+0000 mon.a (mon.0) 3713 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-138"}]: dispatch 2024-02-19T21:17:36.834 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:36.863 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:36.882 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:36.884 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:36.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:36.985 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:36.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:35.340986+0000 mgr.y (mgr.14550) 657 : cluster [DBG] pgmap v1200: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:17:36.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:35.631777+0000 osd.6 (osd.6) 281 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:17:36.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:35.634652+0000 osd.6 (osd.6) 282 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:17:36.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:35.746819+0000 osd.2 (osd.2) 308 : cluster [DBG] 5.b scrub starts 2024-02-19T21:17:36.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:35.747585+0000 osd.2 (osd.2) 309 : cluster [DBG] 5.b scrub ok 2024-02-19T21:17:36.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:36.354180+0000 osd.4 (osd.4) 273 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:17:36.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:36.354990+0000 osd.4 (osd.4) 274 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:17:36.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:36.443108+0000 mon.a (mon.0) 3710 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-138"}]': finished 2024-02-19T21:17:36.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:36.443172+0000 mon.a (mon.0) 3711 : cluster [DBG] osdmap e793: 8 total, 8 up, 6 in 2024-02-19T21:17:36.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:36.461578+0000 mon.a (mon.0) 3712 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:36.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:36.462005+0000 mon.a (mon.0) 3713 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-138"}]: dispatch 2024-02-19T21:17:36.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:35.340986+0000 mgr.y (mgr.14550) 657 : cluster [DBG] pgmap v1200: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:17:37.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:35.631777+0000 osd.6 (osd.6) 281 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:17:37.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:35.634652+0000 osd.6 (osd.6) 282 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:17:37.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:35.746819+0000 osd.2 (osd.2) 308 : cluster [DBG] 5.b scrub starts 2024-02-19T21:17:37.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:35.747585+0000 osd.2 (osd.2) 309 : cluster [DBG] 5.b scrub ok 2024-02-19T21:17:37.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:36.354180+0000 osd.4 (osd.4) 273 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:17:37.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:36.354990+0000 osd.4 (osd.4) 274 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:17:37.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:36.443108+0000 mon.a (mon.0) 3710 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-138"}]': finished 2024-02-19T21:17:37.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:36.443172+0000 mon.a (mon.0) 3711 : cluster [DBG] osdmap e793: 8 total, 8 up, 6 in 2024-02-19T21:17:37.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:36.461578+0000 mon.a (mon.0) 3712 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:37.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:36.462005+0000 mon.a (mon.0) 3713 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-138"}]: dispatch 2024-02-19T21:17:37.013 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:37.032 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:37.035 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:37.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:37.136 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:17:37.163 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:37.182 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:17:37.184 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:37.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:17:37.285 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:37.312 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:37.331 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:37.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:37.434 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:17:37.463 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:37.482 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:17:37.484 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:37.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:17:37.586 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:17:37.613 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:37.685 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:17:37.691 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:37.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:17:37.792 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:17:37.819 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:37.837 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:17:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:37.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:17:37.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:17:37.958 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:36.700726+0000 osd.2 (osd.2) 310 : cluster [DBG] 2.e scrub starts 2024-02-19T21:17:37.958 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:36.701737+0000 osd.2 (osd.2) 311 : cluster [DBG] 2.e scrub ok 2024-02-19T21:17:37.958 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:37.082362+0000 osd.3 (osd.3) 329 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:17:37.958 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:37.083527+0000 osd.3 (osd.3) 330 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:17:37.958 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:37.446785+0000 mon.a (mon.0) 3714 : cluster [DBG] osdmap e794: 8 total, 8 up, 6 in 2024-02-19T21:17:37.959 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:36.700726+0000 osd.2 (osd.2) 310 : cluster [DBG] 2.e scrub starts 2024-02-19T21:17:37.959 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:36.701737+0000 osd.2 (osd.2) 311 : cluster [DBG] 2.e scrub ok 2024-02-19T21:17:37.959 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:37.082362+0000 osd.3 (osd.3) 329 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:17:37.959 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:37.083527+0000 osd.3 (osd.3) 330 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:17:37.959 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:37.446785+0000 mon.a (mon.0) 3714 : cluster [DBG] osdmap e794: 8 total, 8 up, 6 in 2024-02-19T21:17:37.971 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:38.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:17:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:17:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:17:38.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:36.700726+0000 osd.2 (osd.2) 310 : cluster [DBG] 2.e scrub starts 2024-02-19T21:17:38.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:36.701737+0000 osd.2 (osd.2) 311 : cluster [DBG] 2.e scrub ok 2024-02-19T21:17:38.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:37.082362+0000 osd.3 (osd.3) 329 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:17:38.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:37.083527+0000 osd.3 (osd.3) 330 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:17:38.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:37.446785+0000 mon.a (mon.0) 3714 : cluster [DBG] osdmap e794: 8 total, 8 up, 6 in 2024-02-19T21:17:38.336 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:38.696 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-19T21:17:38.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:37.341418+0000 mgr.y (mgr.14550) 658 : cluster [DBG] pgmap v1203: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:17:38.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:37.686404+0000 osd.2 (osd.2) 312 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:17:38.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:37.694885+0000 osd.2 (osd.2) 313 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:17:38.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:37.706482+0000 mon.a (mon.0) 3715 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:17:38.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:38.415879+0000 osd.1 (osd.1) 356 : cluster [DBG] 2.d scrub starts 2024-02-19T21:17:38.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:38.417087+0000 osd.1 (osd.1) 357 : cluster [DBG] 2.d scrub ok 2024-02-19T21:17:38.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:38.445465+0000 mon.a (mon.0) 3716 : cluster [DBG] osdmap e795: 8 total, 8 up, 6 in 2024-02-19T21:17:38.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:38.448303+0000 mon.a (mon.0) 3717 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:17:38.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:37.341418+0000 mgr.y (mgr.14550) 658 : cluster [DBG] pgmap v1203: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:17:38.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:37.686404+0000 osd.2 (osd.2) 312 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:17:38.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:37.694885+0000 osd.2 (osd.2) 313 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:17:38.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:37.706482+0000 mon.a (mon.0) 3715 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:17:38.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:38.415879+0000 osd.1 (osd.1) 356 : cluster [DBG] 2.d scrub starts 2024-02-19T21:17:38.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:38.417087+0000 osd.1 (osd.1) 357 : cluster [DBG] 2.d scrub ok 2024-02-19T21:17:38.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:38.445465+0000 mon.a (mon.0) 3716 : cluster [DBG] osdmap e795: 8 total, 8 up, 6 in 2024-02-19T21:17:38.994 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:38.448303+0000 mon.a (mon.0) 3717 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:17:39.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:37.341418+0000 mgr.y (mgr.14550) 658 : cluster [DBG] pgmap v1203: 214 pgs: 1 active+remapped+backfill_toofull, 213 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-19T21:17:39.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:37.686404+0000 osd.2 (osd.2) 312 : cluster [DBG] 6.12 scrub starts 2024-02-19T21:17:39.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:37.694885+0000 osd.2 (osd.2) 313 : cluster [DBG] 6.12 scrub ok 2024-02-19T21:17:39.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:37.706482+0000 mon.a (mon.0) 3715 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:17:39.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:38.415879+0000 osd.1 (osd.1) 356 : cluster [DBG] 2.d scrub starts 2024-02-19T21:17:39.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:38.417087+0000 osd.1 (osd.1) 357 : cluster [DBG] 2.d scrub ok 2024-02-19T21:17:39.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:38.445465+0000 mon.a (mon.0) 3716 : cluster [DBG] osdmap e795: 8 total, 8 up, 6 in 2024-02-19T21:17:39.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:38.448303+0000 mon.a (mon.0) 3717 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:17:39.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.227+0000 7ff57f9af700 1 -- 172.21.15.163:0/2682628894 >> v1:172.21.15.163:6789/0 conn(0x7ff5781031b0 legacy=0x7ff578105630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:39.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.227+0000 7ff57f9af700 1 -- 172.21.15.163:0/2682628894 shutdown_connections 2024-02-19T21:17:39.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.227+0000 7ff57f9af700 1 -- 172.21.15.163:0/2682628894 >> 172.21.15.163:0/2682628894 conn(0x7ff5780f2470 msgr2=0x7ff5780f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:39.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.227+0000 7ff57f9af700 1 -- 172.21.15.163:0/2682628894 shutdown_connections 2024-02-19T21:17:39.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.227+0000 7ff57f9af700 1 -- 172.21.15.163:0/2682628894 wait complete. 2024-02-19T21:17:39.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.227+0000 7ff57f9af700 1 librados: starting msgr at 2024-02-19T21:17:39.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.227+0000 7ff57f9af700 1 librados: starting objecter 2024-02-19T21:17:39.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.227+0000 7ff57f9af700 1 Processor -- start 2024-02-19T21:17:39.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.227+0000 7ff57f9af700 1 -- start start 2024-02-19T21:17:39.355 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.227+0000 7ff57f9af700 1 librados: setting wanted keys 2024-02-19T21:17:39.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.227+0000 7ff57f9af700 1 librados: calling monclient init 2024-02-19T21:17:39.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff57f9af700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff5781194b0 con 0x7ff5780fc870 2024-02-19T21:17:39.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff57f9af700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff57811a690 con 0x7ff5780ff670 2024-02-19T21:17:39.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff57f9af700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff57811b890 con 0x7ff5781031b0 2024-02-19T21:17:39.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff57d74b700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ff5780fc870 0x7ff5780f8400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43584/0 (socket says 172.21.15.163:43584) 2024-02-19T21:17:39.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff57d74b700 1 -- 172.21.15.163:0/2629641867 learned_addr learned my addr 172.21.15.163:0/2629641867 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:39.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 10 client.?.objecter ms_handle_connect 0x7ff5781031b0 2024-02-19T21:17:39.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:39.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 10 client.?.objecter ms_handle_connect 0x7ff5780fc870 2024-02-19T21:17:39.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:39.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 10 client.?.objecter ms_handle_connect 0x7ff5780ff670 2024-02-19T21:17:39.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:39.356 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792067579 0 0) 0x7ff5781194b0 con 0x7ff5780fc870 2024-02-19T21:17:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff564003800 con 0x7ff5780fc870 2024-02-19T21:17:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461525881 0 0) 0x7ff57811b890 con 0x7ff5781031b0 2024-02-19T21:17:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5781194b0 con 0x7ff5781031b0 2024-02-19T21:17:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978122127 0 0) 0x7ff57811a690 con 0x7ff5780ff670 2024-02-19T21:17:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff57811b890 con 0x7ff5780ff670 2024-02-19T21:17:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3636775383 0 0) 0x7ff564003800 con 0x7ff5780fc870 2024-02-19T21:17:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff57811a690 con 0x7ff5780fc870 2024-02-19T21:17:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff574001a70 con 0x7ff5780fc870 2024-02-19T21:17:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 293759824 0 0) 0x7ff5781194b0 con 0x7ff5781031b0 2024-02-19T21:17:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff564003800 con 0x7ff5781031b0 2024-02-19T21:17:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3826081388 0 0) 0x7ff57811b890 con 0x7ff5780ff670 2024-02-19T21:17:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5781194b0 con 0x7ff5780ff670 2024-02-19T21:17:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff570003220 con 0x7ff5781031b0 2024-02-19T21:17:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff568001840 con 0x7ff5780ff670 2024-02-19T21:17:39.357 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2126250120 0 0) 0x7ff57811a690 con 0x7ff5780fc870 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 >> v1:172.21.15.163:6790/0 conn(0x7ff5781031b0 legacy=0x7ff578117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 >> v1:172.21.15.193:6789/0 conn(0x7ff5780ff670 legacy=0x7ff5780f8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57811ca90 con 0x7ff5780fc870 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff57f9af700 1 -- 172.21.15.163:0/2629641867 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5781196a0 con 0x7ff5780fc870 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff57f9af700 10 client.23910.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff57f9af700 1 -- 172.21.15.163:0/2629641867 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff578119b10 con 0x7ff5780fc870 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.228+0000 7ff57f9af700 1 librados: init done 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.229+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff5740024b0 con 0x7ff5780fc870 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.229+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff574005b10 con 0x7ff5780fc870 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.229+0000 7ff57f9af700 1 -- 172.21.15.163:0/2629641867 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55c004f20 con 0x7ff5780fc870 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.230+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ff574002620 con 0x7ff5780fc870 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.230+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(795..795 src has 623..795) v4 ==== 8413+0+0 (unknown 3911826738 0 0) 0x7ff574078e40 con 0x7ff5780fc870 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.230+0000 7ff56e7fc700 10 client.23910.objecter ms_dispatch 0x7ff578001550 osd_map(795..795 src has 623..795) v4 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.230+0000 7ff56e7fc700 3 client.23910.objecter handle_osd_map got epochs [795,795] > 0 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.230+0000 7ff56e7fc700 3 client.23910.objecter handle_osd_map decoding full epoch 795 2024-02-19T21:17:39.358 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.230+0000 7ff56e7fc700 20 client.23910.objecter dump_active .. 0 homeless 2024-02-19T21:17:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.230+0000 7ff56e7fc700 10 client.23910.objecter ms_handle_connect 0x7ff564063d70 2024-02-19T21:17:39.359 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.235+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff57404c0f0 con 0x7ff5780fc870 2024-02-19T21:17:39.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.405+0000 7ff57f9af700 1 -- 172.21.15.163:0/2629641867 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff55c000b90 con 0x7ff564063d70 2024-02-19T21:17:39.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.419+0000 7ff56e7fc700 1 -- 172.21.15.163:0/2629641867 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+536470 (unknown 2965378022 0 3252923344) 0x7ff55c000b90 con 0x7ff564063d70 2024-02-19T21:17:39.428 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:17:39.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.427+0000 7ff57f9af700 10 librados: watch_flush enter 2024-02-19T21:17:39.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.427+0000 7ff57f9af700 10 librados: watch_flush exit 2024-02-19T21:17:39.433 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.427+0000 7ff57f9af700 20 client.23910.objecter shutdown clearing up homeless session... 2024-02-19T21:17:39.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.427+0000 7ff57f9af700 10 client.23910.objecter successfully canceled tick 2024-02-19T21:17:39.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.428+0000 7ff57f9af700 1 -- 172.21.15.163:0/2629641867 >> v1:172.21.15.163:6800/1552514681 conn(0x7ff564063d70 legacy=0x7ff564066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:39.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.428+0000 7ff57f9af700 1 -- 172.21.15.163:0/2629641867 >> v1:172.21.15.163:6789/0 conn(0x7ff5780fc870 legacy=0x7ff5780f8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:39.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.428+0000 7ff57f9af700 1 -- 172.21.15.163:0/2629641867 shutdown_connections 2024-02-19T21:17:39.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.428+0000 7ff57f9af700 1 -- 172.21.15.163:0/2629641867 >> 172.21.15.163:0/2629641867 conn(0x7ff5780f2470 msgr2=0x7ff5780f3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:39.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.433+0000 7ff57f9af700 1 -- 172.21.15.163:0/2629641867 shutdown_connections 2024-02-19T21:17:39.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.436+0000 7ff57f9af700 1 -- 172.21.15.163:0/2629641867 wait complete. 2024-02-19T21:17:39.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:39.436+0000 7ff57f9af700 1 librados: shutdown 2024-02-19T21:17:39.438 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:17:39.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:38.724916+0000 osd.6 (osd.6) 283 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:17:39.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:38.726223+0000 osd.6 (osd.6) 284 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:17:39.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:39.110205+0000 mon.a (mon.0) 3718 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:39.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:39.173125+0000 osd.3 (osd.3) 331 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:17:39.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:39.174451+0000 osd.3 (osd.3) 332 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:17:39.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:39.291451+0000 osd.4 (osd.4) 275 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:17:39.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:39.292377+0000 osd.4 (osd.4) 276 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:17:39.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:39.381968+0000 osd.1 (osd.1) 358 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:17:39.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:39.384860+0000 osd.1 (osd.1) 359 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:17:39.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:39.424699+0000 mon.a (mon.0) 3719 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:17:39.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:39.424814+0000 mon.a (mon.0) 3720 : cluster [DBG] osdmap e796: 8 total, 8 up, 6 in 2024-02-19T21:17:39.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:39.595815+0000 mon.a (mon.0) 3721 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:39.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:39.596084+0000 mon.a (mon.0) 3722 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-140"}]: dispatch 2024-02-19T21:17:39.990 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:38.724916+0000 osd.6 (osd.6) 283 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:17:39.990 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:38.726223+0000 osd.6 (osd.6) 284 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:17:39.990 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:39.110205+0000 mon.a (mon.0) 3718 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:39.990 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:39.173125+0000 osd.3 (osd.3) 331 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:17:39.990 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:39.174451+0000 osd.3 (osd.3) 332 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:17:39.990 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:39.291451+0000 osd.4 (osd.4) 275 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:17:39.990 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:39.292377+0000 osd.4 (osd.4) 276 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:17:39.990 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:39.381968+0000 osd.1 (osd.1) 358 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:17:39.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:39.384860+0000 osd.1 (osd.1) 359 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:17:39.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:39.424699+0000 mon.a (mon.0) 3719 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:17:39.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:39.424814+0000 mon.a (mon.0) 3720 : cluster [DBG] osdmap e796: 8 total, 8 up, 6 in 2024-02-19T21:17:39.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:39.595815+0000 mon.a (mon.0) 3721 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:39.991 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:39.596084+0000 mon.a (mon.0) 3722 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-140"}]: dispatch 2024-02-19T21:17:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:38.724916+0000 osd.6 (osd.6) 283 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:17:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:38.726223+0000 osd.6 (osd.6) 284 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:17:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:39.110205+0000 mon.a (mon.0) 3718 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:39.173125+0000 osd.3 (osd.3) 331 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:17:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:39.174451+0000 osd.3 (osd.3) 332 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:17:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:39.291451+0000 osd.4 (osd.4) 275 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:17:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:39.292377+0000 osd.4 (osd.4) 276 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:17:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:39.381968+0000 osd.1 (osd.1) 358 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:17:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:39.384860+0000 osd.1 (osd.1) 359 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:17:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:39.424699+0000 mon.a (mon.0) 3719 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:17:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:39.424814+0000 mon.a (mon.0) 3720 : cluster [DBG] osdmap e796: 8 total, 8 up, 6 in 2024-02-19T21:17:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:39.595815+0000 mon.a (mon.0) 3721 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:40.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:39.596084+0000 mon.a (mon.0) 3722 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-140"}]: dispatch 2024-02-19T21:17:41.023 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:17:41.026 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:41.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:17:41.127 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:17:41.132 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:17:41.132 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [], 2024-02-19T21:17:41.132 INFO:teuthology.orchestra.run.smithi163.stdout: "complaint_time": 30, 2024-02-19T21:17:41.132 INFO:teuthology.orchestra.run.smithi163.stdout: "num_blocked_ops": 0 2024-02-19T21:17:41.132 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:17:41.162 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:41.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:17:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:17:41.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:39.341807+0000 mgr.y (mgr.14550) 659 : cluster [DBG] pgmap v1206: 214 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 181 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:17:41.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:39.407133+0000 mgr.y (mgr.14550) 660 : audit [DBG] from='client.23910 v1:172.21.15.163:0/2629641867' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:17:41.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:40.039303+0000 mon.a (mon.0) 3723 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:41.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:40.125008+0000 osd.3 (osd.3) 333 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:17:41.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:40.126035+0000 osd.3 (osd.3) 334 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:17:41.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:40.436414+0000 mon.a (mon.0) 3724 : cluster [DBG] osdmap e797: 8 total, 8 up, 6 in 2024-02-19T21:17:41.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:39.341807+0000 mgr.y (mgr.14550) 659 : cluster [DBG] pgmap v1206: 214 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 181 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:17:41.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:39.407133+0000 mgr.y (mgr.14550) 660 : audit [DBG] from='client.23910 v1:172.21.15.163:0/2629641867' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:17:41.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:40.039303+0000 mon.a (mon.0) 3723 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:41.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:40.125008+0000 osd.3 (osd.3) 333 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:17:41.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:40.126035+0000 osd.3 (osd.3) 334 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:17:41.428 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:40.436414+0000 mon.a (mon.0) 3724 : cluster [DBG] osdmap e797: 8 total, 8 up, 6 in 2024-02-19T21:17:41.428 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:39.341807+0000 mgr.y (mgr.14550) 659 : cluster [DBG] pgmap v1206: 214 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 181 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:17:41.428 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:39.407133+0000 mgr.y (mgr.14550) 660 : audit [DBG] from='client.23910 v1:172.21.15.163:0/2629641867' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:17:41.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:40.039303+0000 mon.a (mon.0) 3723 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:41.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:40.125008+0000 osd.3 (osd.3) 333 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:17:41.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:40.126035+0000 osd.3 (osd.3) 334 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:17:41.429 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:40.436414+0000 mon.a (mon.0) 3724 : cluster [DBG] osdmap e797: 8 total, 8 up, 6 in 2024-02-19T21:17:41.890 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-19T21:17:42.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:40.777022+0000 osd.6 (osd.6) 285 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:17:42.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:40.778857+0000 osd.6 (osd.6) 286 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:17:42.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:40.822630+0000 mgr.y (mgr.14550) 661 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:42.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:41.438873+0000 mon.a (mon.0) 3725 : cluster [DBG] osdmap e798: 8 total, 8 up, 6 in 2024-02-19T21:17:42.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:41.439576+0000 mon.a (mon.0) 3726 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:17:42.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:41.632340+0000 osd.2 (osd.2) 314 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:17:42.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:41.633185+0000 osd.2 (osd.2) 315 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:17:42.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:40.777022+0000 osd.6 (osd.6) 285 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:17:42.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:40.778857+0000 osd.6 (osd.6) 286 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:17:42.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:40.822630+0000 mgr.y (mgr.14550) 661 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:42.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:41.438873+0000 mon.a (mon.0) 3725 : cluster [DBG] osdmap e798: 8 total, 8 up, 6 in 2024-02-19T21:17:42.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:41.439576+0000 mon.a (mon.0) 3726 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:17:42.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:41.632340+0000 osd.2 (osd.2) 314 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:17:42.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:41.633185+0000 osd.2 (osd.2) 315 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:17:42.312 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:17:42.315 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:42.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:17:42.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:40.777022+0000 osd.6 (osd.6) 285 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:17:42.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:40.778857+0000 osd.6 (osd.6) 286 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:17:42.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:40.822630+0000 mgr.y (mgr.14550) 661 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:42.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:41.438873+0000 mon.a (mon.0) 3725 : cluster [DBG] osdmap e798: 8 total, 8 up, 6 in 2024-02-19T21:17:42.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:41.439576+0000 mon.a (mon.0) 3726 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:17:42.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:41.632340+0000 osd.2 (osd.2) 314 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:17:42.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:41.633185+0000 osd.2 (osd.2) 315 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:17:42.330 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":1206,"stamp":"2024-02-19T21:17:39.341581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4006098,"num_objects":236,"num_object_clones":1,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":2104,"num_read_kb":1840,"num_write":783,"num_write_kb":4012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":12022421,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":955,"ondisk_log_size":955,"up":546,"acting":546,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":625,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":1832516,"kb_used_data":28692,"kb_used_omap":0,"kb_used_meta":1803776,"kb_avail":560638396,"statfs":{"total":575970213888,"available":574093717504,"internally_reserved":0,"allocated":29380608,"data_stored":16517086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1847066624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001256"},"pg_stats":[{"pgid":"254.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"254.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.7","version":"771'14","reported_seq":46,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.449207+0000","last_change":"2024-02-19T21:17:15.578600+0000","last_active":"2024-02-19T21:17:33.449207+0000","last_peered":"2024-02-19T21:17:33.449207+0000","last_clean":"2024-02-19T21:17:33.449207+0000","last_became_active":"2024-02-19T21:17:14.448377+0000","last_became_peered":"2024-02-19T21:17:14.448377+0000","last_unstale":"2024-02-19T21:17:33.449207+0000","last_undegraded":"2024-02-19T21:17:33.449207+0000","last_fullsized":"2024-02-19T21:17:33.449207+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"700'10","last_scrub_stamp":"2024-02-19T21:15:51.952191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:15:51.952191+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":60,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"254.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.6","version":"771'14","reported_seq":284,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.440075+0000","last_change":"2024-02-19T21:17:29.570403+0000","last_active":"2024-02-19T21:17:33.440075+0000","last_peered":"2024-02-19T21:17:33.440075+0000","last_clean":"2024-02-19T21:17:33.440075+0000","last_became_active":"2024-02-19T21:17:29.464774+0000","last_became_peered":"2024-02-19T21:17:29.464774+0000","last_unstale":"2024-02-19T21:17:33.440075+0000","last_undegraded":"2024-02-19T21:17:33.440075+0000","last_fullsized":"2024-02-19T21:17:33.440075+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"671'12","last_scrub_stamp":"2024-02-19T21:15:56.206176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:15:56.206176+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":42,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"254.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.5","version":"742'15","reported_seq":35,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444296+0000","last_change":"2024-02-19T21:17:29.574700+0000","last_active":"2024-02-19T21:17:33.444296+0000","last_peered":"2024-02-19T21:17:33.444296+0000","last_clean":"2024-02-19T21:17:33.444296+0000","last_became_active":"2024-02-19T21:17:29.523165+0000","last_became_peered":"2024-02-19T21:17:29.523165+0000","last_unstale":"2024-02-19T21:17:33.444296+0000","last_undegraded":"2024-02-19T21:17:33.444296+0000","last_fullsized":"2024-02-19T21:17:33.444296+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"663'11","last_scrub_stamp":"2024-02-19T21:16:26.025839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:16:26.025839+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":89,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"254.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.4","version":"742'8","reported_seq":268,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.445358+0000","last_change":"2024-02-19T21:17:14.517821+0000","last_active":"2024-02-19T21:17:33.445358+0000","last_peered":"2024-02-19T21:17:33.445358+0000","last_clean":"2024-02-19T21:17:33.445358+0000","last_became_active":"2024-02-19T21:17:14.448578+0000","last_became_peered":"2024-02-19T21:17:14.448578+0000","last_unstale":"2024-02-19T21:17:33.445358+0000","last_undegraded":"2024-02-19T21:17:33.445358+0000","last_fullsized":"2024-02-19T21:17:33.445358+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"708'6","last_scrub_stamp":"2024-02-19T21:16:21.988867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:16:21.988867+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":48,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"254.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.3","version":"742'14","reported_seq":94,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.439047+0000","last_change":"2024-02-19T21:17:29.568360+0000","last_active":"2024-02-19T21:17:33.439047+0000","last_peered":"2024-02-19T21:17:33.439047+0000","last_clean":"2024-02-19T21:17:33.439047+0000","last_became_active":"2024-02-19T21:17:29.465032+0000","last_became_peered":"2024-02-19T21:17:29.465032+0000","last_unstale":"2024-02-19T21:17:33.439047+0000","last_undegraded":"2024-02-19T21:17:33.439047+0000","last_fullsized":"2024-02-19T21:17:33.439047+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"716'11","last_scrub_stamp":"2024-02-19T21:16:18.081978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:16:18.081978+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4000037,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":3913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":12000108,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"254.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.2","version":"742'6","reported_seq":23,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.442779+0000","last_change":"2024-02-19T21:17:29.562756+0000","last_active":"2024-02-19T21:17:33.442779+0000","last_peered":"2024-02-19T21:17:33.442779+0000","last_clean":"2024-02-19T21:17:33.442779+0000","last_became_active":"2024-02-19T21:17:29.525837+0000","last_became_peered":"2024-02-19T21:17:29.525837+0000","last_unstale":"2024-02-19T21:17:33.442779+0000","last_undegraded":"2024-02-19T21:17:33.442779+0000","last_fullsized":"2024-02-19T21:17:33.442779+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"663'5","last_scrub_stamp":"2024-02-19T21:16:23.961316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:16:23.961316+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"254.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.1","version":"742'8","reported_seq":253,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443318+0000","last_change":"2024-02-19T21:16:25.979481+0000","last_active":"2024-02-19T21:17:33.443318+0000","last_peered":"2024-02-19T21:17:33.443318+0000","last_clean":"2024-02-19T21:17:33.443318+0000","last_became_active":"2024-02-19T21:14:41.401375+0000","last_became_peered":"2024-02-19T21:14:41.401375+0000","last_unstale":"2024-02-19T21:17:33.443318+0000","last_undegraded":"2024-02-19T21:17:33.443318+0000","last_fullsized":"2024-02-19T21:17:33.443318+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"627'6","last_scrub_stamp":"2024-02-19T21:16:25.979404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:16:25.979404+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":9,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"254.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-19T21:17:38.441171+0000","last_change":"2024-02-19T21:17:38.441171+0000","last_active":"2024-02-19T21:17:38.441171+0000","last_peered":"2024-02-19T21:17:38.441171+0000","last_clean":"2024-02-19T21:17:38.441171+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-19T21:17:38.441171+0000","last_undegraded":"2024-02-19T21:17:38.441171+0000","last_fullsized":"2024-02-19T21:17:38.441171+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:38.441171+0000","last_clean_scrub_stamp":"2024-02-19T21:17:38.441171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"234.0","version":"791'17","reported_seq":256,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-02-19T21:17:34.446208+0000","last_change":"2024-02-19T21:15:49.384800+0000","last_active":"2024-02-19T21:17:34.446208+0000","last_peered":"2024-02-19T21:17:34.446208+0000","last_clean":"2024-02-19T21:17:34.446208+0000","last_became_active":"2024-02-19T21:14:16.383023+0000","last_became_peered":"2024-02-19T21:14:16.383023+0000","last_unstale":"2024-02-19T21:17:34.446208+0000","last_undegraded":"2024-02-19T21:17:34.446208+0000","last_fullsized":"2024-02-19T21:17:34.446208+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":598,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:54.338488+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.338488+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":85,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":14,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.29","version":"0'0","reported_seq":202,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.438711+0000","last_change":"2024-02-19T21:17:14.452288+0000","last_active":"2024-02-19T21:17:33.438711+0000","last_peered":"2024-02-19T21:17:33.438711+0000","last_clean":"2024-02-19T21:17:33.438711+0000","last_became_active":"2024-02-19T21:17:14.451953+0000","last_became_peered":"2024-02-19T21:17:14.451953+0000","last_unstale":"2024-02-19T21:17:33.438711+0000","last_undegraded":"2024-02-19T21:17:33.438711+0000","last_fullsized":"2024-02-19T21:17:33.438711+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:10:52.738600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:10:52.738600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":343,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.441286+0000","last_change":"2024-02-19T21:17:14.434057+0000","last_active":"2024-02-19T21:17:33.441286+0000","last_peered":"2024-02-19T21:17:33.441286+0000","last_clean":"2024-02-19T21:17:33.441286+0000","last_became_active":"2024-02-19T21:17:14.433628+0000","last_became_peered":"2024-02-19T21:17:14.433628+0000","last_unstale":"2024-02-19T21:17:33.441286+0000","last_undegraded":"2024-02-19T21:17:33.441286+0000","last_fullsized":"2024-02-19T21:17:33.441286+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:04.962987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:04.962987+0000","last_clean_scrub_stamp":"2024-02-19T21:17:04.962987+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.25","version":"0'0","reported_seq":852,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443456+0000","last_change":"2024-02-19T21:17:05.222776+0000","last_active":"2024-02-19T21:17:33.443456+0000","last_peered":"2024-02-19T21:17:33.443456+0000","last_clean":"2024-02-19T21:17:33.443456+0000","last_became_active":"2024-02-19T21:16:58.548544+0000","last_became_peered":"2024-02-19T21:16:58.548544+0000","last_unstale":"2024-02-19T21:17:33.443456+0000","last_undegraded":"2024-02-19T21:17:33.443456+0000","last_fullsized":"2024-02-19T21:17:33.443456+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:05.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:17:05.222693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":189,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.439528+0000","last_change":"2024-02-19T21:17:14.448692+0000","last_active":"2024-02-19T21:17:33.439528+0000","last_peered":"2024-02-19T21:17:33.439528+0000","last_clean":"2024-02-19T21:17:33.439528+0000","last_became_active":"2024-02-19T21:17:14.448343+0000","last_became_peered":"2024-02-19T21:17:14.448343+0000","last_unstale":"2024-02-19T21:17:33.439528+0000","last_undegraded":"2024-02-19T21:17:33.439528+0000","last_fullsized":"2024-02-19T21:17:33.439528+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:18.547736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:16:18.547736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":551,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443867+0000","last_change":"2024-02-19T21:17:14.448847+0000","last_active":"2024-02-19T21:17:33.443867+0000","last_peered":"2024-02-19T21:17:33.443867+0000","last_clean":"2024-02-19T21:17:33.443867+0000","last_became_active":"2024-02-19T21:17:14.446959+0000","last_became_peered":"2024-02-19T21:17:14.446959+0000","last_unstale":"2024-02-19T21:17:33.443867+0000","last_undegraded":"2024-02-19T21:17:33.443867+0000","last_fullsized":"2024-02-19T21:17:33.443867+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:36.863734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:36.863734+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.26","version":"0'0","reported_seq":553,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443645+0000","last_change":"2024-02-19T21:17:14.461503+0000","last_active":"2024-02-19T21:17:33.443645+0000","last_peered":"2024-02-19T21:17:33.443645+0000","last_clean":"2024-02-19T21:17:33.443645+0000","last_became_active":"2024-02-19T21:17:14.460717+0000","last_became_peered":"2024-02-19T21:17:14.460717+0000","last_unstale":"2024-02-19T21:17:33.443645+0000","last_undegraded":"2024-02-19T21:17:33.443645+0000","last_fullsized":"2024-02-19T21:17:33.443645+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:17.840425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:16:17.840425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.442773+0000","last_change":"2024-02-19T21:17:29.522907+0000","last_active":"2024-02-19T21:17:33.442773+0000","last_peered":"2024-02-19T21:17:33.442773+0000","last_clean":"2024-02-19T21:17:33.442773+0000","last_became_active":"2024-02-19T21:17:29.522603+0000","last_became_peered":"2024-02-19T21:17:29.522603+0000","last_unstale":"2024-02-19T21:17:33.442773+0000","last_undegraded":"2024-02-19T21:17:33.442773+0000","last_fullsized":"2024-02-19T21:17:33.442773+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:49.242760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:49.242760+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.20","version":"397'2","reported_seq":614,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.447373+0000","last_change":"2024-02-19T21:17:17.429068+0000","last_active":"2024-02-19T21:17:33.447373+0000","last_peered":"2024-02-19T21:17:33.447373+0000","last_clean":"2024-02-19T21:17:33.447373+0000","last_became_active":"2024-02-19T21:17:17.428733+0000","last_became_peered":"2024-02-19T21:17:17.428733+0000","last_unstale":"2024-02-19T21:17:33.447373+0000","last_undegraded":"2024-02-19T21:17:33.447373+0000","last_fullsized":"2024-02-19T21:17:33.447373+0000","mapping_epoch":777,"log_start":"58'1","ondisk_log_start":"58'1","created":474,"last_epoch_clean":778,"parent":"0.0","parent_split_bits":6,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:14:58.880622+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:14:58.880622+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":437,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":347,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443221+0000","last_change":"2024-02-19T21:16:15.136043+0000","last_active":"2024-02-19T21:17:33.443221+0000","last_peered":"2024-02-19T21:17:33.443221+0000","last_clean":"2024-02-19T21:17:33.443221+0000","last_became_active":"2024-02-19T21:14:16.384406+0000","last_became_peered":"2024-02-19T21:14:16.384406+0000","last_unstale":"2024-02-19T21:17:33.443221+0000","last_undegraded":"2024-02-19T21:17:33.443221+0000","last_fullsized":"2024-02-19T21:17:33.443221+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:15.135981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:16:15.135981+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":180,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.438839+0000","last_change":"2024-02-19T21:15:51.187729+0000","last_active":"2024-02-19T21:17:33.438839+0000","last_peered":"2024-02-19T21:17:33.438839+0000","last_clean":"2024-02-19T21:17:33.438839+0000","last_became_active":"2024-02-19T21:14:16.382869+0000","last_became_peered":"2024-02-19T21:14:16.382869+0000","last_unstale":"2024-02-19T21:17:33.438839+0000","last_undegraded":"2024-02-19T21:17:33.438839+0000","last_fullsized":"2024-02-19T21:17:33.438839+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:51.187606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:51.187606+0000","last_clean_scrub_stamp":"2024-02-19T21:15:51.187606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"775'15","reported_seq":625,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-02-19T21:17:34.444750+0000","last_change":"2024-02-19T21:17:14.468235+0000","last_active":"2024-02-19T21:17:34.444750+0000","last_peered":"2024-02-19T21:17:34.444750+0000","last_clean":"2024-02-19T21:17:34.444750+0000","last_became_active":"2024-02-19T21:17:14.448436+0000","last_became_peered":"2024-02-19T21:17:14.448436+0000","last_unstale":"2024-02-19T21:17:34.444750+0000","last_undegraded":"2024-02-19T21:17:34.444750+0000","last_fullsized":"2024-02-19T21:17:34.444750+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"623'13","last_scrub_stamp":"2024-02-19T21:14:48.841229+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-02-19T21:04:24.046034+0000","last_clean_scrub_stamp":"2024-02-19T21:14:48.841229+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"473'28","reported_seq":744,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-02-19T21:17:34.664844+0000","last_change":"2024-02-19T21:17:34.664844+0000","last_active":"2024-02-19T21:17:34.664844+0000","last_peered":"2024-02-19T21:17:34.664844+0000","last_clean":"2024-02-19T21:17:34.664844+0000","last_became_active":"2024-02-19T21:17:14.452949+0000","last_became_peered":"2024-02-19T21:17:14.452949+0000","last_unstale":"2024-02-19T21:17:34.664844+0000","last_undegraded":"2024-02-19T21:17:34.664844+0000","last_fullsized":"2024-02-19T21:17:34.664844+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-19T21:17:34.664765+0000","last_deep_scrub":"473'28","last_deep_scrub_stamp":"2024-02-19T21:11:47.051823+0000","last_clean_scrub_stamp":"2024-02-19T21:17:34.664765+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444812+0000","last_change":"2024-02-19T21:17:29.467795+0000","last_active":"2024-02-19T21:17:33.444812+0000","last_peered":"2024-02-19T21:17:33.444812+0000","last_clean":"2024-02-19T21:17:33.444812+0000","last_became_active":"2024-02-19T21:17:29.467362+0000","last_became_peered":"2024-02-19T21:17:29.467362+0000","last_unstale":"2024-02-19T21:17:33.444812+0000","last_undegraded":"2024-02-19T21:17:33.444812+0000","last_fullsized":"2024-02-19T21:17:33.444812+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:52.369624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:16.105470+0000","last_clean_scrub_stamp":"2024-02-19T21:14:52.369624+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":680,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.692637+0000","last_change":"2024-02-19T21:17:33.692610+0000","last_active":"2024-02-19T21:17:33.692637+0000","last_peered":"2024-02-19T21:17:33.692637+0000","last_clean":"2024-02-19T21:17:33.692637+0000","last_became_active":"2024-02-19T21:17:29.465845+0000","last_became_peered":"2024-02-19T21:17:29.465845+0000","last_unstale":"2024-02-19T21:17:33.692637+0000","last_undegraded":"2024-02-19T21:17:33.692637+0000","last_fullsized":"2024-02-19T21:17:33.692637+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:33.692538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:17:33.692538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444278+0000","last_change":"2024-02-19T21:17:29.528825+0000","last_active":"2024-02-19T21:17:33.444278+0000","last_peered":"2024-02-19T21:17:33.444278+0000","last_clean":"2024-02-19T21:17:33.444278+0000","last_became_active":"2024-02-19T21:17:29.527982+0000","last_became_peered":"2024-02-19T21:17:29.527982+0000","last_unstale":"2024-02-19T21:17:33.444278+0000","last_undegraded":"2024-02-19T21:17:33.444278+0000","last_fullsized":"2024-02-19T21:17:33.444278+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:38.353000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:38.353000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"473'16","reported_seq":700,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.448295+0000","last_change":"2024-02-19T21:15:03.898768+0000","last_active":"2024-02-19T21:17:33.448295+0000","last_peered":"2024-02-19T21:17:33.448295+0000","last_clean":"2024-02-19T21:17:33.448295+0000","last_became_active":"2024-02-19T21:14:41.391381+0000","last_became_peered":"2024-02-19T21:14:41.391381+0000","last_unstale":"2024-02-19T21:17:33.448295+0000","last_undegraded":"2024-02-19T21:17:33.448295+0000","last_fullsized":"2024-02-19T21:17:33.448295+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'16","last_scrub_stamp":"2024-02-19T21:15:03.898694+0000","last_deep_scrub":"473'16","last_deep_scrub_stamp":"2024-02-19T21:11:46.956424+0000","last_clean_scrub_stamp":"2024-02-19T21:15:03.898694+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":796,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.449057+0000","last_change":"2024-02-19T21:15:05.888480+0000","last_active":"2024-02-19T21:17:33.449057+0000","last_peered":"2024-02-19T21:17:33.449057+0000","last_clean":"2024-02-19T21:17:33.449057+0000","last_became_active":"2024-02-19T21:10:41.312116+0000","last_became_peered":"2024-02-19T21:10:41.312116+0000","last_unstale":"2024-02-19T21:17:33.449057+0000","last_undegraded":"2024-02-19T21:17:33.449057+0000","last_fullsized":"2024-02-19T21:17:33.449057+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:05.888410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:05.888410+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.442576+0000","last_change":"2024-02-19T21:17:29.480237+0000","last_active":"2024-02-19T21:17:33.442576+0000","last_peered":"2024-02-19T21:17:33.442576+0000","last_clean":"2024-02-19T21:17:33.442576+0000","last_became_active":"2024-02-19T21:17:29.479730+0000","last_became_peered":"2024-02-19T21:17:29.479730+0000","last_unstale":"2024-02-19T21:17:33.442576+0000","last_undegraded":"2024-02-19T21:17:33.442576+0000","last_fullsized":"2024-02-19T21:17:33.442576+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:32.211538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:32.211538+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":20,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.746150+0000","last_change":"2024-02-19T21:17:33.746110+0000","last_active":"2024-02-19T21:17:33.746150+0000","last_peered":"2024-02-19T21:17:33.746150+0000","last_clean":"2024-02-19T21:17:33.746150+0000","last_became_active":"2024-02-19T21:17:29.523007+0000","last_became_peered":"2024-02-19T21:17:29.523007+0000","last_unstale":"2024-02-19T21:17:33.746150+0000","last_undegraded":"2024-02-19T21:17:33.746150+0000","last_fullsized":"2024-02-19T21:17:33.746150+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:33.746017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:17:33.746017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444735+0000","last_change":"2024-02-19T21:17:29.523101+0000","last_active":"2024-02-19T21:17:33.444735+0000","last_peered":"2024-02-19T21:17:33.444735+0000","last_clean":"2024-02-19T21:17:33.444735+0000","last_became_active":"2024-02-19T21:17:29.522371+0000","last_became_peered":"2024-02-19T21:17:29.522371+0000","last_unstale":"2024-02-19T21:17:33.444735+0000","last_undegraded":"2024-02-19T21:17:33.444735+0000","last_fullsized":"2024-02-19T21:17:33.444735+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:54.277197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:16:54.277197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"459'35","reported_seq":234,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.438989+0000","last_change":"2024-02-19T21:17:14.501876+0000","last_active":"2024-02-19T21:17:33.438989+0000","last_peered":"2024-02-19T21:17:33.438989+0000","last_clean":"2024-02-19T21:17:33.438989+0000","last_became_active":"2024-02-19T21:17:14.454352+0000","last_became_peered":"2024-02-19T21:17:14.454352+0000","last_unstale":"2024-02-19T21:17:33.438989+0000","last_undegraded":"2024-02-19T21:17:33.438989+0000","last_fullsized":"2024-02-19T21:17:33.438989+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'35","last_scrub_stamp":"2024-02-19T21:16:55.770042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:16:55.770042+0000","log_size":35,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444101+0000","last_change":"2024-02-19T21:17:29.528653+0000","last_active":"2024-02-19T21:17:33.444101+0000","last_peered":"2024-02-19T21:17:33.444101+0000","last_clean":"2024-02-19T21:17:33.444101+0000","last_became_active":"2024-02-19T21:17:29.527383+0000","last_became_peered":"2024-02-19T21:17:29.527383+0000","last_unstale":"2024-02-19T21:17:33.444101+0000","last_undegraded":"2024-02-19T21:17:33.444101+0000","last_fullsized":"2024-02-19T21:17:33.444101+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:56.245419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:16:56.245419+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443513+0000","last_change":"2024-02-19T21:17:29.523452+0000","last_active":"2024-02-19T21:17:33.443513+0000","last_peered":"2024-02-19T21:17:33.443513+0000","last_clean":"2024-02-19T21:17:33.443513+0000","last_became_active":"2024-02-19T21:17:29.523120+0000","last_became_peered":"2024-02-19T21:17:29.523120+0000","last_unstale":"2024-02-19T21:17:33.443513+0000","last_undegraded":"2024-02-19T21:17:33.443513+0000","last_fullsized":"2024-02-19T21:17:33.443513+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:49.321813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:49.321813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.442420+0000","last_change":"2024-02-19T21:17:29.464911+0000","last_active":"2024-02-19T21:17:33.442420+0000","last_peered":"2024-02-19T21:17:33.442420+0000","last_clean":"2024-02-19T21:17:33.442420+0000","last_became_active":"2024-02-19T21:17:29.464596+0000","last_became_peered":"2024-02-19T21:17:29.464596+0000","last_unstale":"2024-02-19T21:17:33.442420+0000","last_undegraded":"2024-02-19T21:17:33.442420+0000","last_fullsized":"2024-02-19T21:17:33.442420+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:18.857424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:18.857424+0000","last_clean_scrub_stamp":"2024-02-19T21:16:18.857424+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":411,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443996+0000","last_change":"2024-02-19T21:17:14.450313+0000","last_active":"2024-02-19T21:17:33.443996+0000","last_peered":"2024-02-19T21:17:33.443996+0000","last_clean":"2024-02-19T21:17:33.443996+0000","last_became_active":"2024-02-19T21:17:14.446750+0000","last_became_peered":"2024-02-19T21:17:14.446750+0000","last_unstale":"2024-02-19T21:17:33.443996+0000","last_undegraded":"2024-02-19T21:17:33.443996+0000","last_fullsized":"2024-02-19T21:17:33.443996+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:59.927001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:59.927001+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"662'39","reported_seq":499,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444052+0000","last_change":"2024-02-19T21:17:29.556057+0000","last_active":"2024-02-19T21:17:33.444052+0000","last_peered":"2024-02-19T21:17:33.444052+0000","last_clean":"2024-02-19T21:17:33.444052+0000","last_became_active":"2024-02-19T21:17:29.471707+0000","last_became_peered":"2024-02-19T21:17:29.471707+0000","last_unstale":"2024-02-19T21:17:33.444052+0000","last_undegraded":"2024-02-19T21:17:33.444052+0000","last_fullsized":"2024-02-19T21:17:33.444052+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"662'39","last_scrub_stamp":"2024-02-19T21:15:06.578559+0000","last_deep_scrub":"662'39","last_deep_scrub_stamp":"2024-02-19T21:15:06.578559+0000","last_clean_scrub_stamp":"2024-02-19T21:15:06.578559+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"655'2","reported_seq":743,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443357+0000","last_change":"2024-02-19T21:17:14.473224+0000","last_active":"2024-02-19T21:17:33.443357+0000","last_peered":"2024-02-19T21:17:33.443357+0000","last_clean":"2024-02-19T21:17:33.443357+0000","last_became_active":"2024-02-19T21:17:14.464799+0000","last_became_peered":"2024-02-19T21:17:14.464799+0000","last_unstale":"2024-02-19T21:17:33.443357+0000","last_undegraded":"2024-02-19T21:17:33.443357+0000","last_fullsized":"2024-02-19T21:17:33.443357+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"655'2","last_scrub_stamp":"2024-02-19T21:15:00.963594+0000","last_deep_scrub":"655'2","last_deep_scrub_stamp":"2024-02-19T21:15:00.963594+0000","last_clean_scrub_stamp":"2024-02-19T21:15:00.963594+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":810,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.448876+0000","last_change":"2024-02-19T21:15:51.016985+0000","last_active":"2024-02-19T21:17:33.448876+0000","last_peered":"2024-02-19T21:17:33.448876+0000","last_clean":"2024-02-19T21:17:33.448876+0000","last_became_active":"2024-02-19T21:14:41.391280+0000","last_became_peered":"2024-02-19T21:14:41.391280+0000","last_unstale":"2024-02-19T21:17:33.448876+0000","last_undegraded":"2024-02-19T21:17:33.448876+0000","last_fullsized":"2024-02-19T21:17:33.448876+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:51.016885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:51.016885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444809+0000","last_change":"2024-02-19T21:17:29.479795+0000","last_active":"2024-02-19T21:17:33.444809+0000","last_peered":"2024-02-19T21:17:33.444809+0000","last_clean":"2024-02-19T21:17:33.444809+0000","last_became_active":"2024-02-19T21:17:29.479445+0000","last_became_peered":"2024-02-19T21:17:29.479445+0000","last_unstale":"2024-02-19T21:17:33.444809+0000","last_undegraded":"2024-02-19T21:17:33.444809+0000","last_fullsized":"2024-02-19T21:17:33.444809+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:42.266585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:15:42.266585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":371,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443033+0000","last_change":"2024-02-19T21:14:53.606588+0000","last_active":"2024-02-19T21:17:33.443033+0000","last_peered":"2024-02-19T21:17:33.443033+0000","last_clean":"2024-02-19T21:17:33.443033+0000","last_became_active":"2024-02-19T21:10:41.302670+0000","last_became_peered":"2024-02-19T21:10:41.302670+0000","last_unstale":"2024-02-19T21:17:33.443033+0000","last_undegraded":"2024-02-19T21:17:33.443033+0000","last_fullsized":"2024-02-19T21:17:33.443033+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:53.606474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:53.606474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"459'25","reported_seq":440,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443527+0000","last_change":"2024-02-19T21:16:17.215169+0000","last_active":"2024-02-19T21:17:33.443527+0000","last_peered":"2024-02-19T21:17:33.443527+0000","last_clean":"2024-02-19T21:17:33.443527+0000","last_became_active":"2024-02-19T21:14:16.381198+0000","last_became_peered":"2024-02-19T21:14:16.381198+0000","last_unstale":"2024-02-19T21:17:33.443527+0000","last_undegraded":"2024-02-19T21:17:33.443527+0000","last_fullsized":"2024-02-19T21:17:33.443527+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'25","last_scrub_stamp":"2024-02-19T21:16:17.215100+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:16:17.215100+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":799,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.448449+0000","last_change":"2024-02-19T21:17:29.519898+0000","last_active":"2024-02-19T21:17:33.448449+0000","last_peered":"2024-02-19T21:17:33.448449+0000","last_clean":"2024-02-19T21:17:33.448449+0000","last_became_active":"2024-02-19T21:17:29.519531+0000","last_became_peered":"2024-02-19T21:17:29.519531+0000","last_unstale":"2024-02-19T21:17:33.448449+0000","last_undegraded":"2024-02-19T21:17:33.448449+0000","last_fullsized":"2024-02-19T21:17:33.448449+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:23.207199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:16:23.207199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"786'12","reported_seq":21,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-02-19T21:17:34.497920+0000","last_change":"2024-02-19T21:17:29.536694+0000","last_active":"2024-02-19T21:17:34.497920+0000","last_peered":"2024-02-19T21:17:34.497920+0000","last_clean":"2024-02-19T21:17:34.497920+0000","last_became_active":"2024-02-19T21:17:29.522613+0000","last_became_peered":"2024-02-19T21:17:29.522613+0000","last_unstale":"2024-02-19T21:17:34.497920+0000","last_undegraded":"2024-02-19T21:17:34.497920+0000","last_fullsized":"2024-02-19T21:17:34.497920+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'8","last_scrub_stamp":"2024-02-19T21:14:58.303098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:58.303098+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":407,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444222+0000","last_change":"2024-02-19T21:17:29.471449+0000","last_active":"2024-02-19T21:17:33.444222+0000","last_peered":"2024-02-19T21:17:33.444222+0000","last_clean":"2024-02-19T21:17:33.444222+0000","last_became_active":"2024-02-19T21:17:29.471013+0000","last_became_peered":"2024-02-19T21:17:29.471013+0000","last_unstale":"2024-02-19T21:17:33.444222+0000","last_undegraded":"2024-02-19T21:17:33.444222+0000","last_fullsized":"2024-02-19T21:17:33.444222+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:16.207396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:21.031577+0000","last_clean_scrub_stamp":"2024-02-19T21:16:16.207396+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":541,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.441355+0000","last_change":"2024-02-19T21:17:14.457381+0000","last_active":"2024-02-19T21:17:33.441355+0000","last_peered":"2024-02-19T21:17:33.441355+0000","last_clean":"2024-02-19T21:17:33.441355+0000","last_became_active":"2024-02-19T21:17:14.456676+0000","last_became_peered":"2024-02-19T21:17:14.456676+0000","last_unstale":"2024-02-19T21:17:33.441355+0000","last_undegraded":"2024-02-19T21:17:33.441355+0000","last_fullsized":"2024-02-19T21:17:33.441355+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:49.195586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:49.195586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"459'27","reported_seq":80,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.449491+0000","last_change":"2024-02-19T21:17:01.317578+0000","last_active":"2024-02-19T21:17:33.449491+0000","last_peered":"2024-02-19T21:17:33.449491+0000","last_clean":"2024-02-19T21:17:33.449491+0000","last_became_active":"2024-02-19T21:16:37.630325+0000","last_became_peered":"2024-02-19T21:16:37.630325+0000","last_unstale":"2024-02-19T21:17:33.449491+0000","last_undegraded":"2024-02-19T21:17:33.449491+0000","last_fullsized":"2024-02-19T21:17:33.449491+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'27","last_scrub_stamp":"2024-02-19T21:17:01.317506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:17:01.317506+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":845,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.447598+0000","last_change":"2024-02-19T21:17:29.517236+0000","last_active":"2024-02-19T21:17:33.447598+0000","last_peered":"2024-02-19T21:17:33.447598+0000","last_clean":"2024-02-19T21:17:33.447598+0000","last_became_active":"2024-02-19T21:17:29.516352+0000","last_became_peered":"2024-02-19T21:17:29.516352+0000","last_unstale":"2024-02-19T21:17:33.447598+0000","last_undegraded":"2024-02-19T21:17:33.447598+0000","last_fullsized":"2024-02-19T21:17:33.447598+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:55.003931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:14:55.003931+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"786'14","reported_seq":511,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-02-19T21:17:34.446528+0000","last_change":"2024-02-19T21:17:29.535303+0000","last_active":"2024-02-19T21:17:34.446528+0000","last_peered":"2024-02-19T21:17:34.446528+0000","last_clean":"2024-02-19T21:17:34.446528+0000","last_became_active":"2024-02-19T21:17:29.471543+0000","last_became_peered":"2024-02-19T21:17:29.471543+0000","last_unstale":"2024-02-19T21:17:34.446528+0000","last_undegraded":"2024-02-19T21:17:34.446528+0000","last_fullsized":"2024-02-19T21:17:34.446528+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"640'12","last_scrub_stamp":"2024-02-19T21:16:20.215750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:16:20.215750+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":662,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.448760+0000","last_change":"2024-02-19T21:17:29.470234+0000","last_active":"2024-02-19T21:17:33.448760+0000","last_peered":"2024-02-19T21:17:33.448760+0000","last_clean":"2024-02-19T21:17:33.448760+0000","last_became_active":"2024-02-19T21:17:29.469904+0000","last_became_peered":"2024-02-19T21:17:29.469904+0000","last_unstale":"2024-02-19T21:17:33.448760+0000","last_undegraded":"2024-02-19T21:17:33.448760+0000","last_fullsized":"2024-02-19T21:17:33.448760+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:37.011259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:37.011259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":590,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.448151+0000","last_change":"2024-02-19T21:17:08.359551+0000","last_active":"2024-02-19T21:17:33.448151+0000","last_peered":"2024-02-19T21:17:33.448151+0000","last_clean":"2024-02-19T21:17:33.448151+0000","last_became_active":"2024-02-19T21:16:58.563166+0000","last_became_peered":"2024-02-19T21:16:58.563166+0000","last_unstale":"2024-02-19T21:17:33.448151+0000","last_undegraded":"2024-02-19T21:17:33.448151+0000","last_fullsized":"2024-02-19T21:17:33.448151+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:08.359482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:17:08.359482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"459'19","reported_seq":439,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444141+0000","last_change":"2024-02-19T21:15:03.572770+0000","last_active":"2024-02-19T21:17:33.444141+0000","last_peered":"2024-02-19T21:17:33.444141+0000","last_clean":"2024-02-19T21:17:33.444141+0000","last_became_active":"2024-02-19T21:14:16.382074+0000","last_became_peered":"2024-02-19T21:14:16.382074+0000","last_unstale":"2024-02-19T21:17:33.444141+0000","last_undegraded":"2024-02-19T21:17:33.444141+0000","last_fullsized":"2024-02-19T21:17:33.444141+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:15:03.572687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:03.572687+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":726,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.447806+0000","last_change":"2024-02-19T21:17:14.439154+0000","last_active":"2024-02-19T21:17:33.447806+0000","last_peered":"2024-02-19T21:17:33.447806+0000","last_clean":"2024-02-19T21:17:33.447806+0000","last_became_active":"2024-02-19T21:17:14.438733+0000","last_became_peered":"2024-02-19T21:17:14.438733+0000","last_unstale":"2024-02-19T21:17:33.447806+0000","last_undegraded":"2024-02-19T21:17:33.447806+0000","last_fullsized":"2024-02-19T21:17:33.447806+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:35.944741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:35.944741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":657,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443970+0000","last_change":"2024-02-19T21:17:29.472833+0000","last_active":"2024-02-19T21:17:33.443970+0000","last_peered":"2024-02-19T21:17:33.443970+0000","last_clean":"2024-02-19T21:17:33.443970+0000","last_became_active":"2024-02-19T21:17:29.472162+0000","last_became_peered":"2024-02-19T21:17:29.472162+0000","last_unstale":"2024-02-19T21:17:33.443970+0000","last_undegraded":"2024-02-19T21:17:33.443970+0000","last_fullsized":"2024-02-19T21:17:33.443970+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:21.074298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:16:21.074298+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":548,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.445897+0000","last_change":"2024-02-19T21:17:14.453618+0000","last_active":"2024-02-19T21:17:33.445897+0000","last_peered":"2024-02-19T21:17:33.445897+0000","last_clean":"2024-02-19T21:17:33.445897+0000","last_became_active":"2024-02-19T21:17:14.452124+0000","last_became_peered":"2024-02-19T21:17:14.452124+0000","last_unstale":"2024-02-19T21:17:33.445897+0000","last_undegraded":"2024-02-19T21:17:33.445897+0000","last_fullsized":"2024-02-19T21:17:33.445897+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:21.029975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:16:21.029975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":534,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443127+0000","last_change":"2024-02-19T21:16:58.570774+0000","last_active":"2024-02-19T21:17:33.443127+0000","last_peered":"2024-02-19T21:17:33.443127+0000","last_clean":"2024-02-19T21:17:33.443127+0000","last_became_active":"2024-02-19T21:16:58.570575+0000","last_became_peered":"2024-02-19T21:16:58.570575+0000","last_unstale":"2024-02-19T21:17:33.443127+0000","last_undegraded":"2024-02-19T21:17:33.443127+0000","last_fullsized":"2024-02-19T21:17:33.443127+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:38.121453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:38.121453+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"434'12","reported_seq":660,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-02-19T21:17:34.447429+0000","last_change":"2024-02-19T21:16:56.021510+0000","last_active":"2024-02-19T21:17:34.447429+0000","last_peered":"2024-02-19T21:17:34.447429+0000","last_clean":"2024-02-19T21:17:34.447429+0000","last_became_active":"2024-02-19T21:10:41.297897+0000","last_became_peered":"2024-02-19T21:10:41.297897+0000","last_unstale":"2024-02-19T21:17:34.447429+0000","last_undegraded":"2024-02-19T21:17:34.447429+0000","last_fullsized":"2024-02-19T21:17:34.447429+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'12","last_scrub_stamp":"2024-02-19T21:16:56.021415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:16:56.021415+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"459'28","reported_seq":488,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443151+0000","last_change":"2024-02-19T21:16:58.601548+0000","last_active":"2024-02-19T21:17:33.443151+0000","last_peered":"2024-02-19T21:17:33.443151+0000","last_clean":"2024-02-19T21:17:33.443151+0000","last_became_active":"2024-02-19T21:16:58.563082+0000","last_became_peered":"2024-02-19T21:16:58.563082+0000","last_unstale":"2024-02-19T21:17:33.443151+0000","last_undegraded":"2024-02-19T21:17:33.443151+0000","last_fullsized":"2024-02-19T21:17:33.443151+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'28","last_scrub_stamp":"2024-02-19T21:16:30.209336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:16:30.209336+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":207,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.440101+0000","last_change":"2024-02-19T21:17:29.470333+0000","last_active":"2024-02-19T21:17:33.440101+0000","last_peered":"2024-02-19T21:17:33.440101+0000","last_clean":"2024-02-19T21:17:33.440101+0000","last_became_active":"2024-02-19T21:17:29.469878+0000","last_became_peered":"2024-02-19T21:17:29.469878+0000","last_unstale":"2024-02-19T21:17:33.440101+0000","last_undegraded":"2024-02-19T21:17:33.440101+0000","last_fullsized":"2024-02-19T21:17:33.440101+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:50.164434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:50.164434+0000","last_clean_scrub_stamp":"2024-02-19T21:15:50.164434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":685,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.439316+0000","last_change":"2024-02-19T21:17:29.465490+0000","last_active":"2024-02-19T21:17:33.439316+0000","last_peered":"2024-02-19T21:17:33.439316+0000","last_clean":"2024-02-19T21:17:33.439316+0000","last_became_active":"2024-02-19T21:17:29.464494+0000","last_became_peered":"2024-02-19T21:17:29.464494+0000","last_unstale":"2024-02-19T21:17:33.439316+0000","last_undegraded":"2024-02-19T21:17:33.439316+0000","last_fullsized":"2024-02-19T21:17:33.439316+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:19.857838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:16:19.857838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":613,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.449167+0000","last_change":"2024-02-19T21:16:58.556123+0000","last_active":"2024-02-19T21:17:33.449167+0000","last_peered":"2024-02-19T21:17:33.449167+0000","last_clean":"2024-02-19T21:17:33.449167+0000","last_became_active":"2024-02-19T21:16:58.555356+0000","last_became_peered":"2024-02-19T21:16:58.555356+0000","last_unstale":"2024-02-19T21:17:33.449167+0000","last_undegraded":"2024-02-19T21:17:33.449167+0000","last_fullsized":"2024-02-19T21:17:33.449167+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:49.983125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:11.004006+0000","last_clean_scrub_stamp":"2024-02-19T21:15:49.983125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":380,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.442899+0000","last_change":"2024-02-19T21:17:14.443325+0000","last_active":"2024-02-19T21:17:33.442899+0000","last_peered":"2024-02-19T21:17:33.442899+0000","last_clean":"2024-02-19T21:17:33.442899+0000","last_became_active":"2024-02-19T21:17:14.442937+0000","last_became_peered":"2024-02-19T21:17:14.442937+0000","last_unstale":"2024-02-19T21:17:33.442899+0000","last_undegraded":"2024-02-19T21:17:33.442899+0000","last_fullsized":"2024-02-19T21:17:33.442899+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:54.654916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:12.318389+0000","last_clean_scrub_stamp":"2024-02-19T21:14:54.654916+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"459'30","reported_seq":624,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.441458+0000","last_change":"2024-02-19T21:17:14.473233+0000","last_active":"2024-02-19T21:17:33.441458+0000","last_peered":"2024-02-19T21:17:33.441458+0000","last_clean":"2024-02-19T21:17:33.441458+0000","last_became_active":"2024-02-19T21:17:14.441435+0000","last_became_peered":"2024-02-19T21:17:14.441435+0000","last_unstale":"2024-02-19T21:17:33.441458+0000","last_undegraded":"2024-02-19T21:17:33.441458+0000","last_fullsized":"2024-02-19T21:17:33.441458+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'30","last_scrub_stamp":"2024-02-19T21:14:52.068777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:52.068777+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":822,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.449339+0000","last_change":"2024-02-19T21:15:48.962320+0000","last_active":"2024-02-19T21:17:33.449339+0000","last_peered":"2024-02-19T21:17:33.449339+0000","last_clean":"2024-02-19T21:17:33.449339+0000","last_became_active":"2024-02-19T21:14:16.379640+0000","last_became_peered":"2024-02-19T21:14:16.379640+0000","last_unstale":"2024-02-19T21:17:33.449339+0000","last_undegraded":"2024-02-19T21:17:33.449339+0000","last_fullsized":"2024-02-19T21:17:33.449339+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:48.962240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:48.962240+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":389,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443153+0000","last_change":"2024-02-19T21:17:29.471233+0000","last_active":"2024-02-19T21:17:33.443153+0000","last_peered":"2024-02-19T21:17:33.443153+0000","last_clean":"2024-02-19T21:17:33.443153+0000","last_became_active":"2024-02-19T21:17:29.470869+0000","last_became_peered":"2024-02-19T21:17:29.470869+0000","last_unstale":"2024-02-19T21:17:33.443153+0000","last_undegraded":"2024-02-19T21:17:33.443153+0000","last_fullsized":"2024-02-19T21:17:33.443153+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:44.842888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:47.538920+0000","last_clean_scrub_stamp":"2024-02-19T21:15:44.842888+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":543,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.446131+0000","last_change":"2024-02-19T21:15:39.146832+0000","last_active":"2024-02-19T21:17:33.446131+0000","last_peered":"2024-02-19T21:17:33.446131+0000","last_clean":"2024-02-19T21:17:33.446131+0000","last_became_active":"2024-02-19T21:14:41.398917+0000","last_became_peered":"2024-02-19T21:14:41.398917+0000","last_unstale":"2024-02-19T21:17:33.446131+0000","last_undegraded":"2024-02-19T21:17:33.446131+0000","last_fullsized":"2024-02-19T21:17:33.446131+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:39.146761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:39.146761+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":216,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.438891+0000","last_change":"2024-02-19T21:17:29.465821+0000","last_active":"2024-02-19T21:17:33.438891+0000","last_peered":"2024-02-19T21:17:33.438891+0000","last_clean":"2024-02-19T21:17:33.438891+0000","last_became_active":"2024-02-19T21:17:29.465269+0000","last_became_peered":"2024-02-19T21:17:29.465269+0000","last_unstale":"2024-02-19T21:17:33.438891+0000","last_undegraded":"2024-02-19T21:17:33.438891+0000","last_fullsized":"2024-02-19T21:17:33.438891+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:49.141780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:49.141780+0000","last_clean_scrub_stamp":"2024-02-19T21:15:49.141780+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"459'29","reported_seq":112,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.447993+0000","last_change":"2024-02-19T21:17:30.278157+0000","last_active":"2024-02-19T21:17:33.447993+0000","last_peered":"2024-02-19T21:17:33.447993+0000","last_clean":"2024-02-19T21:17:33.447993+0000","last_became_active":"2024-02-19T21:17:29.531157+0000","last_became_peered":"2024-02-19T21:17:29.531157+0000","last_unstale":"2024-02-19T21:17:33.447993+0000","last_undegraded":"2024-02-19T21:17:33.447993+0000","last_fullsized":"2024-02-19T21:17:33.447993+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'29","last_scrub_stamp":"2024-02-19T21:15:42.003405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:42.003405+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":745,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443570+0000","last_change":"2024-02-19T21:17:14.445275+0000","last_active":"2024-02-19T21:17:33.443570+0000","last_peered":"2024-02-19T21:17:33.443570+0000","last_clean":"2024-02-19T21:17:33.443570+0000","last_became_active":"2024-02-19T21:17:14.440945+0000","last_became_peered":"2024-02-19T21:17:14.440945+0000","last_unstale":"2024-02-19T21:17:33.443570+0000","last_undegraded":"2024-02-19T21:17:33.443570+0000","last_fullsized":"2024-02-19T21:17:33.443570+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:41.880077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:41.880077+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":794,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.447702+0000","last_change":"2024-02-19T21:17:29.517329+0000","last_active":"2024-02-19T21:17:33.447702+0000","last_peered":"2024-02-19T21:17:33.447702+0000","last_clean":"2024-02-19T21:17:33.447702+0000","last_became_active":"2024-02-19T21:17:29.516758+0000","last_became_peered":"2024-02-19T21:17:29.516758+0000","last_unstale":"2024-02-19T21:17:33.447702+0000","last_undegraded":"2024-02-19T21:17:33.447702+0000","last_fullsized":"2024-02-19T21:17:33.447702+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:36.830383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:15:36.830383+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444660+0000","last_change":"2024-02-19T21:17:29.523182+0000","last_active":"2024-02-19T21:17:33.444660+0000","last_peered":"2024-02-19T21:17:33.444660+0000","last_clean":"2024-02-19T21:17:33.444660+0000","last_became_active":"2024-02-19T21:17:29.522761+0000","last_became_peered":"2024-02-19T21:17:29.522761+0000","last_unstale":"2024-02-19T21:17:33.444660+0000","last_undegraded":"2024-02-19T21:17:33.444660+0000","last_fullsized":"2024-02-19T21:17:33.444660+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:30.017769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:02.189509+0000","last_clean_scrub_stamp":"2024-02-19T21:16:30.017769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":654,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.450441+0000","last_change":"2024-02-19T21:17:09.325859+0000","last_active":"2024-02-19T21:17:33.450441+0000","last_peered":"2024-02-19T21:17:33.450441+0000","last_clean":"2024-02-19T21:17:33.450441+0000","last_became_active":"2024-02-19T21:16:58.564416+0000","last_became_peered":"2024-02-19T21:16:58.564416+0000","last_unstale":"2024-02-19T21:17:33.450441+0000","last_undegraded":"2024-02-19T21:17:33.450441+0000","last_fullsized":"2024-02-19T21:17:33.450441+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:09.325756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:17:09.325756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"459'24","reported_seq":749,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.450262+0000","last_change":"2024-02-19T21:17:29.576869+0000","last_active":"2024-02-19T21:17:33.450262+0000","last_peered":"2024-02-19T21:17:33.450262+0000","last_clean":"2024-02-19T21:17:33.450262+0000","last_became_active":"2024-02-19T21:17:29.529550+0000","last_became_peered":"2024-02-19T21:17:29.529550+0000","last_unstale":"2024-02-19T21:17:33.450262+0000","last_undegraded":"2024-02-19T21:17:33.450262+0000","last_fullsized":"2024-02-19T21:17:33.450262+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'24","last_scrub_stamp":"2024-02-19T21:15:45.957817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:45.957817+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"397'2","reported_seq":473,"reported_epoch":790,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-19T21:17:33.447897+0000","last_change":"2024-02-19T21:17:15.430474+0000","last_active":"2024-02-19T21:17:33.447897+0000","last_peered":"2024-02-19T21:17:33.447897+0000","last_clean":"2024-02-19T21:17:12.707174+0000","last_became_active":"2024-02-19T21:17:15.429646+0000","last_became_peered":"2024-02-19T21:17:15.429646+0000","last_unstale":"2024-02-19T21:17:33.447897+0000","last_undegraded":"2024-02-19T21:17:33.447897+0000","last_fullsized":"2024-02-19T21:17:33.447897+0000","mapping_epoch":775,"log_start":"58'1","ondisk_log_start":"58'1","created":57,"last_epoch_clean":736,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:15:50.915331+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:15:50.915331+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":436,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,7],"avail_no_missing":["1","0","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":389,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443301+0000","last_change":"2024-02-19T21:14:41.416360+0000","last_active":"2024-02-19T21:17:33.443301+0000","last_peered":"2024-02-19T21:17:33.443301+0000","last_clean":"2024-02-19T21:17:33.443301+0000","last_became_active":"2024-02-19T21:14:41.416024+0000","last_became_peered":"2024-02-19T21:14:41.416024+0000","last_unstale":"2024-02-19T21:17:33.443301+0000","last_undegraded":"2024-02-19T21:17:33.443301+0000","last_fullsized":"2024-02-19T21:17:33.443301+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:54.643360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:13:54.643360+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":783,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.448935+0000","last_change":"2024-02-19T21:17:00.356354+0000","last_active":"2024-02-19T21:17:33.448935+0000","last_peered":"2024-02-19T21:17:33.448935+0000","last_clean":"2024-02-19T21:17:33.448935+0000","last_became_active":"2024-02-19T21:16:58.563088+0000","last_became_peered":"2024-02-19T21:16:58.563088+0000","last_unstale":"2024-02-19T21:17:33.448935+0000","last_undegraded":"2024-02-19T21:17:33.448935+0000","last_fullsized":"2024-02-19T21:17:33.448935+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:00.356285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:17:00.356285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":198,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.439013+0000","last_change":"2024-02-19T21:16:58.559895+0000","last_active":"2024-02-19T21:17:33.439013+0000","last_peered":"2024-02-19T21:17:33.439013+0000","last_clean":"2024-02-19T21:17:33.439013+0000","last_became_active":"2024-02-19T21:16:58.558919+0000","last_became_peered":"2024-02-19T21:16:58.558919+0000","last_unstale":"2024-02-19T21:17:33.439013+0000","last_undegraded":"2024-02-19T21:17:33.439013+0000","last_fullsized":"2024-02-19T21:17:33.439013+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:47.088245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:47.088245+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444136+0000","last_change":"2024-02-19T21:17:29.528791+0000","last_active":"2024-02-19T21:17:33.444136+0000","last_peered":"2024-02-19T21:17:33.444136+0000","last_clean":"2024-02-19T21:17:33.444136+0000","last_became_active":"2024-02-19T21:17:29.527914+0000","last_became_peered":"2024-02-19T21:17:29.527914+0000","last_unstale":"2024-02-19T21:17:33.444136+0000","last_undegraded":"2024-02-19T21:17:33.444136+0000","last_fullsized":"2024-02-19T21:17:33.444136+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:32.976749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:32.976749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.5","version":"459'26","reported_seq":221,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.439710+0000","last_change":"2024-02-19T21:17:14.482388+0000","last_active":"2024-02-19T21:17:33.439710+0000","last_peered":"2024-02-19T21:17:33.439710+0000","last_clean":"2024-02-19T21:17:33.439710+0000","last_became_active":"2024-02-19T21:17:14.453952+0000","last_became_peered":"2024-02-19T21:17:14.453952+0000","last_unstale":"2024-02-19T21:17:33.439710+0000","last_undegraded":"2024-02-19T21:17:33.439710+0000","last_fullsized":"2024-02-19T21:17:33.439710+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'26","last_scrub_stamp":"2024-02-19T21:15:39.237243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:39.237243+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":26,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.447283+0000","last_change":"2024-02-19T21:17:14.444406+0000","last_active":"2024-02-19T21:17:33.447283+0000","last_peered":"2024-02-19T21:17:33.447283+0000","last_clean":"2024-02-19T21:17:33.447283+0000","last_became_active":"2024-02-19T21:17:14.439090+0000","last_became_peered":"2024-02-19T21:17:14.439090+0000","last_unstale":"2024-02-19T21:17:33.447283+0000","last_undegraded":"2024-02-19T21:17:33.447283+0000","last_fullsized":"2024-02-19T21:17:33.447283+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:13.914069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:16:13.914069+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443592+0000","last_change":"2024-02-19T21:17:29.465863+0000","last_active":"2024-02-19T21:17:33.443592+0000","last_peered":"2024-02-19T21:17:33.443592+0000","last_clean":"2024-02-19T21:17:33.443592+0000","last_became_active":"2024-02-19T21:17:29.465528+0000","last_became_peered":"2024-02-19T21:17:29.465528+0000","last_unstale":"2024-02-19T21:17:33.443592+0000","last_undegraded":"2024-02-19T21:17:33.443592+0000","last_fullsized":"2024-02-19T21:17:33.443592+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:20.033827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:16:20.033827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":539,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443157+0000","last_change":"2024-02-19T21:16:58.561908+0000","last_active":"2024-02-19T21:17:33.443157+0000","last_peered":"2024-02-19T21:17:33.443157+0000","last_clean":"2024-02-19T21:17:33.443157+0000","last_became_active":"2024-02-19T21:16:58.561544+0000","last_became_peered":"2024-02-19T21:16:58.561544+0000","last_unstale":"2024-02-19T21:17:33.443157+0000","last_undegraded":"2024-02-19T21:17:33.443157+0000","last_fullsized":"2024-02-19T21:17:33.443157+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:17.055468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:16:17.055468+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":466,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.446845+0000","last_change":"2024-02-19T21:16:58.576208+0000","last_active":"2024-02-19T21:17:33.446845+0000","last_peered":"2024-02-19T21:17:33.446845+0000","last_clean":"2024-02-19T21:17:33.446845+0000","last_became_active":"2024-02-19T21:16:58.575873+0000","last_became_peered":"2024-02-19T21:16:58.575873+0000","last_unstale":"2024-02-19T21:17:33.446845+0000","last_undegraded":"2024-02-19T21:17:33.446845+0000","last_fullsized":"2024-02-19T21:17:33.446845+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:14.002986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:16:14.002986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":625,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.447470+0000","last_change":"2024-02-19T21:17:29.517063+0000","last_active":"2024-02-19T21:17:33.447470+0000","last_peered":"2024-02-19T21:17:33.447470+0000","last_clean":"2024-02-19T21:17:33.447470+0000","last_became_active":"2024-02-19T21:17:29.515917+0000","last_became_peered":"2024-02-19T21:17:29.515917+0000","last_unstale":"2024-02-19T21:17:33.447470+0000","last_undegraded":"2024-02-19T21:17:33.447470+0000","last_fullsized":"2024-02-19T21:17:33.447470+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:15.006550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_clean_scrub_stamp":"2024-02-19T21:16:15.006550+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444885+0000","last_change":"2024-02-19T21:17:29.510753+0000","last_active":"2024-02-19T21:17:33.444885+0000","last_peered":"2024-02-19T21:17:33.444885+0000","last_clean":"2024-02-19T21:17:33.444885+0000","last_became_active":"2024-02-19T21:17:29.510445+0000","last_became_peered":"2024-02-19T21:17:29.510445+0000","last_unstale":"2024-02-19T21:17:33.444885+0000","last_undegraded":"2024-02-19T21:17:33.444885+0000","last_fullsized":"2024-02-19T21:17:33.444885+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:04.348628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:04.348628+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"459'22","reported_seq":113,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.450242+0000","last_change":"2024-02-19T21:17:15.664608+0000","last_active":"2024-02-19T21:17:33.450242+0000","last_peered":"2024-02-19T21:17:33.450242+0000","last_clean":"2024-02-19T21:17:33.450242+0000","last_became_active":"2024-02-19T21:17:14.451567+0000","last_became_peered":"2024-02-19T21:17:14.451567+0000","last_unstale":"2024-02-19T21:17:33.450242+0000","last_undegraded":"2024-02-19T21:17:33.450242+0000","last_fullsized":"2024-02-19T21:17:33.450242+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":776,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'22","last_scrub_stamp":"2024-02-19T21:16:28.053240+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:16:28.053240+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.442514+0000","last_change":"2024-02-19T21:17:29.525765+0000","last_active":"2024-02-19T21:17:33.442514+0000","last_peered":"2024-02-19T21:17:33.442514+0000","last_clean":"2024-02-19T21:17:33.442514+0000","last_became_active":"2024-02-19T21:17:29.525497+0000","last_became_peered":"2024-02-19T21:17:29.525497+0000","last_unstale":"2024-02-19T21:17:33.442514+0000","last_undegraded":"2024-02-19T21:17:33.442514+0000","last_fullsized":"2024-02-19T21:17:33.442514+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:51.240046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:51.240046+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":1782,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-02-19T21:17:34.498529+0000","last_change":"2024-02-19T21:17:29.651638+0000","last_active":"2024-02-19T21:17:34.498529+0000","last_peered":"2024-02-19T21:17:34.498529+0000","last_clean":"2024-02-19T21:17:34.498529+0000","last_became_active":"2024-02-19T21:17:29.529134+0000","last_became_peered":"2024-02-19T21:17:29.529134+0000","last_unstale":"2024-02-19T21:17:34.498529+0000","last_undegraded":"2024-02-19T21:17:34.498529+0000","last_fullsized":"2024-02-19T21:17:34.498529+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:15:44.948532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:44.948532+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":962,"num_read_kb":957,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":695,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.439557+0000","last_change":"2024-02-19T21:17:29.464867+0000","last_active":"2024-02-19T21:17:33.439557+0000","last_peered":"2024-02-19T21:17:33.439557+0000","last_clean":"2024-02-19T21:17:33.439557+0000","last_became_active":"2024-02-19T21:17:29.464365+0000","last_became_peered":"2024-02-19T21:17:29.464365+0000","last_unstale":"2024-02-19T21:17:33.439557+0000","last_undegraded":"2024-02-19T21:17:33.439557+0000","last_fullsized":"2024-02-19T21:17:33.439557+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:03.864741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:17:03.864741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":744,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.453438+0000","last_change":"2024-02-19T21:17:33.453409+0000","last_active":"2024-02-19T21:17:33.453438+0000","last_peered":"2024-02-19T21:17:33.453438+0000","last_clean":"2024-02-19T21:17:33.453438+0000","last_became_active":"2024-02-19T21:17:29.530744+0000","last_became_peered":"2024-02-19T21:17:29.530744+0000","last_unstale":"2024-02-19T21:17:33.453438+0000","last_undegraded":"2024-02-19T21:17:33.453438+0000","last_fullsized":"2024-02-19T21:17:33.453438+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:33.453346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:11:07.947089+0000","last_clean_scrub_stamp":"2024-02-19T21:17:33.453346+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":640,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.448930+0000","last_change":"2024-02-19T21:14:55.995419+0000","last_active":"2024-02-19T21:17:33.448930+0000","last_peered":"2024-02-19T21:17:33.448930+0000","last_clean":"2024-02-19T21:17:33.448930+0000","last_became_active":"2024-02-19T21:14:41.390554+0000","last_became_peered":"2024-02-19T21:14:41.390554+0000","last_unstale":"2024-02-19T21:17:33.448930+0000","last_undegraded":"2024-02-19T21:17:33.448930+0000","last_fullsized":"2024-02-19T21:17:33.448930+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:55.995348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:14:55.995348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"786'44","reported_seq":1144,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-02-19T21:17:34.497870+0000","last_change":"2024-02-19T21:17:29.584221+0000","last_active":"2024-02-19T21:17:34.497870+0000","last_peered":"2024-02-19T21:17:34.497870+0000","last_clean":"2024-02-19T21:17:34.497870+0000","last_became_active":"2024-02-19T21:17:29.529715+0000","last_became_peered":"2024-02-19T21:17:29.529715+0000","last_unstale":"2024-02-19T21:17:34.497870+0000","last_undegraded":"2024-02-19T21:17:34.497870+0000","last_fullsized":"2024-02-19T21:17:34.497870+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'42","last_scrub_stamp":"2024-02-19T21:16:22.171895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:16:22.171895+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":673,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.448355+0000","last_change":"2024-02-19T21:16:58.564994+0000","last_active":"2024-02-19T21:17:33.448355+0000","last_peered":"2024-02-19T21:17:33.448355+0000","last_clean":"2024-02-19T21:17:33.448355+0000","last_became_active":"2024-02-19T21:16:58.564268+0000","last_became_peered":"2024-02-19T21:16:58.564268+0000","last_unstale":"2024-02-19T21:17:33.448355+0000","last_undegraded":"2024-02-19T21:17:33.448355+0000","last_fullsized":"2024-02-19T21:17:33.448355+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:07.044737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:15:07.044737+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":534,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.446608+0000","last_change":"2024-02-19T21:17:14.461336+0000","last_active":"2024-02-19T21:17:33.446608+0000","last_peered":"2024-02-19T21:17:33.446608+0000","last_clean":"2024-02-19T21:17:33.446608+0000","last_became_active":"2024-02-19T21:17:14.458478+0000","last_became_peered":"2024-02-19T21:17:14.458478+0000","last_unstale":"2024-02-19T21:17:33.446608+0000","last_undegraded":"2024-02-19T21:17:33.446608+0000","last_fullsized":"2024-02-19T21:17:33.446608+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:35.131770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:35.131770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.442483+0000","last_change":"2024-02-19T21:17:29.527355+0000","last_active":"2024-02-19T21:17:33.442483+0000","last_peered":"2024-02-19T21:17:33.442483+0000","last_clean":"2024-02-19T21:17:33.442483+0000","last_became_active":"2024-02-19T21:17:29.526643+0000","last_became_peered":"2024-02-19T21:17:29.526643+0000","last_unstale":"2024-02-19T21:17:33.442483+0000","last_undegraded":"2024-02-19T21:17:33.442483+0000","last_fullsized":"2024-02-19T21:17:33.442483+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:27.004578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:16:27.004578+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"762'15","reported_seq":250,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-02-19T21:17:34.469062+0000","last_change":"2024-02-19T21:16:58.605768+0000","last_active":"2024-02-19T21:17:34.469062+0000","last_peered":"2024-02-19T21:17:34.469062+0000","last_clean":"2024-02-19T21:17:34.469062+0000","last_became_active":"2024-02-19T21:16:58.573091+0000","last_became_peered":"2024-02-19T21:16:58.573091+0000","last_unstale":"2024-02-19T21:17:34.469062+0000","last_undegraded":"2024-02-19T21:17:34.469062+0000","last_fullsized":"2024-02-19T21:17:34.469062+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"640'14","last_scrub_stamp":"2024-02-19T21:15:43.163004+0000","last_deep_scrub":"471'10","last_deep_scrub_stamp":"2024-02-19T21:11:41.775418+0000","last_clean_scrub_stamp":"2024-02-19T21:15:43.163004+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"459'20","reported_seq":846,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.448487+0000","last_change":"2024-02-19T21:16:59.101260+0000","last_active":"2024-02-19T21:17:33.448487+0000","last_peered":"2024-02-19T21:17:33.448487+0000","last_clean":"2024-02-19T21:17:33.448487+0000","last_became_active":"2024-02-19T21:16:58.564729+0000","last_became_peered":"2024-02-19T21:16:58.564729+0000","last_unstale":"2024-02-19T21:17:33.448487+0000","last_undegraded":"2024-02-19T21:17:33.448487+0000","last_fullsized":"2024-02-19T21:17:33.448487+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:15:38.900433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:38.900433+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":850,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.448690+0000","last_change":"2024-02-19T21:17:29.530189+0000","last_active":"2024-02-19T21:17:33.448690+0000","last_peered":"2024-02-19T21:17:33.448690+0000","last_clean":"2024-02-19T21:17:33.448690+0000","last_became_active":"2024-02-19T21:17:29.529233+0000","last_became_peered":"2024-02-19T21:17:29.529233+0000","last_unstale":"2024-02-19T21:17:33.448690+0000","last_undegraded":"2024-02-19T21:17:33.448690+0000","last_fullsized":"2024-02-19T21:17:33.448690+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:53.204160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:16:53.204160+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":557,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.441432+0000","last_change":"2024-02-19T21:17:29.446888+0000","last_active":"2024-02-19T21:17:33.441432+0000","last_peered":"2024-02-19T21:17:33.441432+0000","last_clean":"2024-02-19T21:17:33.441432+0000","last_became_active":"2024-02-19T21:17:29.446569+0000","last_became_peered":"2024-02-19T21:17:29.446569+0000","last_unstale":"2024-02-19T21:17:33.441432+0000","last_undegraded":"2024-02-19T21:17:33.441432+0000","last_fullsized":"2024-02-19T21:17:33.441432+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:18.094861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:16:18.094861+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":525,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.446644+0000","last_change":"2024-02-19T21:15:03.219533+0000","last_active":"2024-02-19T21:17:33.446644+0000","last_peered":"2024-02-19T21:17:33.446644+0000","last_clean":"2024-02-19T21:17:33.446644+0000","last_became_active":"2024-02-19T21:14:16.383904+0000","last_became_peered":"2024-02-19T21:14:16.383904+0000","last_unstale":"2024-02-19T21:17:33.446644+0000","last_undegraded":"2024-02-19T21:17:33.446644+0000","last_fullsized":"2024-02-19T21:17:33.446644+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:03.219469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:03.219469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.442386+0000","last_change":"2024-02-19T21:17:29.527342+0000","last_active":"2024-02-19T21:17:33.442386+0000","last_peered":"2024-02-19T21:17:33.442386+0000","last_clean":"2024-02-19T21:17:33.442386+0000","last_became_active":"2024-02-19T21:17:29.526877+0000","last_became_peered":"2024-02-19T21:17:29.526877+0000","last_unstale":"2024-02-19T21:17:33.442386+0000","last_undegraded":"2024-02-19T21:17:33.442386+0000","last_fullsized":"2024-02-19T21:17:33.442386+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:40.357299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:15:40.357299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"459'23","reported_seq":392,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443019+0000","last_change":"2024-02-19T21:15:43.850016+0000","last_active":"2024-02-19T21:17:33.443019+0000","last_peered":"2024-02-19T21:17:33.443019+0000","last_clean":"2024-02-19T21:17:33.443019+0000","last_became_active":"2024-02-19T21:10:41.287134+0000","last_became_peered":"2024-02-19T21:10:41.287134+0000","last_unstale":"2024-02-19T21:17:33.443019+0000","last_undegraded":"2024-02-19T21:17:33.443019+0000","last_fullsized":"2024-02-19T21:17:33.443019+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'23","last_scrub_stamp":"2024-02-19T21:15:43.849916+0000","last_deep_scrub":"420'17","last_deep_scrub_stamp":"2024-02-19T21:10:19.722303+0000","last_clean_scrub_stamp":"2024-02-19T21:15:43.849916+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":422,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444277+0000","last_change":"2024-02-19T21:17:14.481724+0000","last_active":"2024-02-19T21:17:33.444277+0000","last_peered":"2024-02-19T21:17:33.444277+0000","last_clean":"2024-02-19T21:17:33.444277+0000","last_became_active":"2024-02-19T21:17:14.448448+0000","last_became_peered":"2024-02-19T21:17:14.448448+0000","last_unstale":"2024-02-19T21:17:33.444277+0000","last_undegraded":"2024-02-19T21:17:33.444277+0000","last_fullsized":"2024-02-19T21:17:33.444277+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:14:49.574024+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:14:49.574024+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":521,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443164+0000","last_change":"2024-02-19T21:17:14.447667+0000","last_active":"2024-02-19T21:17:33.443164+0000","last_peered":"2024-02-19T21:17:33.443164+0000","last_clean":"2024-02-19T21:17:33.443164+0000","last_became_active":"2024-02-19T21:17:14.446939+0000","last_became_peered":"2024-02-19T21:17:14.446939+0000","last_unstale":"2024-02-19T21:17:33.443164+0000","last_undegraded":"2024-02-19T21:17:33.443164+0000","last_fullsized":"2024-02-19T21:17:33.443164+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:56.087844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:15:56.087844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444408+0000","last_change":"2024-02-19T21:17:29.512694+0000","last_active":"2024-02-19T21:17:33.444408+0000","last_peered":"2024-02-19T21:17:33.444408+0000","last_clean":"2024-02-19T21:17:33.444408+0000","last_became_active":"2024-02-19T21:17:29.511419+0000","last_became_peered":"2024-02-19T21:17:29.511419+0000","last_unstale":"2024-02-19T21:17:33.444408+0000","last_undegraded":"2024-02-19T21:17:33.444408+0000","last_fullsized":"2024-02-19T21:17:33.444408+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:25.024296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:16:25.024296+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"459'15","reported_seq":609,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.446582+0000","last_change":"2024-02-19T21:17:29.524933+0000","last_active":"2024-02-19T21:17:33.446582+0000","last_peered":"2024-02-19T21:17:33.446582+0000","last_clean":"2024-02-19T21:17:33.446582+0000","last_became_active":"2024-02-19T21:17:29.448571+0000","last_became_peered":"2024-02-19T21:17:29.448571+0000","last_unstale":"2024-02-19T21:17:33.446582+0000","last_undegraded":"2024-02-19T21:17:33.446582+0000","last_fullsized":"2024-02-19T21:17:33.446582+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:17:01.055076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:17:01.055076+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.442334+0000","last_change":"2024-02-19T21:17:29.527439+0000","last_active":"2024-02-19T21:17:33.442334+0000","last_peered":"2024-02-19T21:17:33.442334+0000","last_clean":"2024-02-19T21:17:33.442334+0000","last_became_active":"2024-02-19T21:17:29.527055+0000","last_became_peered":"2024-02-19T21:17:29.527055+0000","last_unstale":"2024-02-19T21:17:33.442334+0000","last_undegraded":"2024-02-19T21:17:33.442334+0000","last_fullsized":"2024-02-19T21:17:33.442334+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:55.262060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:16:55.262060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":516,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.446932+0000","last_change":"2024-02-19T21:17:14.460134+0000","last_active":"2024-02-19T21:17:33.446932+0000","last_peered":"2024-02-19T21:17:33.446932+0000","last_clean":"2024-02-19T21:17:33.446932+0000","last_became_active":"2024-02-19T21:17:14.459533+0000","last_became_peered":"2024-02-19T21:17:14.459533+0000","last_unstale":"2024-02-19T21:17:33.446932+0000","last_undegraded":"2024-02-19T21:17:33.446932+0000","last_fullsized":"2024-02-19T21:17:33.446932+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:56.048471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:49.863037+0000","last_clean_scrub_stamp":"2024-02-19T21:14:56.048471+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":20,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-02-19T21:17:34.769052+0000","last_change":"2024-02-19T21:17:34.769052+0000","last_active":"2024-02-19T21:17:34.769052+0000","last_peered":"2024-02-19T21:17:34.769052+0000","last_clean":"2024-02-19T21:17:34.769052+0000","last_became_active":"2024-02-19T21:17:29.528328+0000","last_became_peered":"2024-02-19T21:17:29.528328+0000","last_unstale":"2024-02-19T21:17:34.769052+0000","last_undegraded":"2024-02-19T21:17:34.769052+0000","last_fullsized":"2024-02-19T21:17:34.769052+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:34.768986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:17:34.768986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":380,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443471+0000","last_change":"2024-02-19T21:17:14.444170+0000","last_active":"2024-02-19T21:17:33.443471+0000","last_peered":"2024-02-19T21:17:33.443471+0000","last_clean":"2024-02-19T21:17:33.443471+0000","last_became_active":"2024-02-19T21:17:14.443827+0000","last_became_peered":"2024-02-19T21:17:14.443827+0000","last_unstale":"2024-02-19T21:17:33.443471+0000","last_undegraded":"2024-02-19T21:17:33.443471+0000","last_fullsized":"2024-02-19T21:17:33.443471+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:15.181743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:16:15.181743+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"459'15","reported_seq":102,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.446711+0000","last_change":"2024-02-19T21:17:29.583319+0000","last_active":"2024-02-19T21:17:33.446711+0000","last_peered":"2024-02-19T21:17:33.446711+0000","last_clean":"2024-02-19T21:17:33.446711+0000","last_became_active":"2024-02-19T21:17:29.461171+0000","last_became_peered":"2024-02-19T21:17:29.461171+0000","last_unstale":"2024-02-19T21:17:33.446711+0000","last_undegraded":"2024-02-19T21:17:33.446711+0000","last_fullsized":"2024-02-19T21:17:33.446711+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:16:16.011190+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:16:16.011190+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":185,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.438740+0000","last_change":"2024-02-19T21:17:01.816997+0000","last_active":"2024-02-19T21:17:33.438740+0000","last_peered":"2024-02-19T21:17:33.438740+0000","last_clean":"2024-02-19T21:17:33.438740+0000","last_became_active":"2024-02-19T21:14:16.385407+0000","last_became_peered":"2024-02-19T21:14:16.385407+0000","last_unstale":"2024-02-19T21:17:33.438740+0000","last_undegraded":"2024-02-19T21:17:33.438740+0000","last_fullsized":"2024-02-19T21:17:33.438740+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:01.816912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:07.220022+0000","last_clean_scrub_stamp":"2024-02-19T21:17:01.816912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":313,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.446507+0000","last_change":"2024-02-19T21:16:58.576019+0000","last_active":"2024-02-19T21:17:33.446507+0000","last_peered":"2024-02-19T21:17:33.446507+0000","last_clean":"2024-02-19T21:17:33.446507+0000","last_became_active":"2024-02-19T21:16:58.575807+0000","last_became_peered":"2024-02-19T21:16:58.575807+0000","last_unstale":"2024-02-19T21:17:33.446507+0000","last_undegraded":"2024-02-19T21:17:33.446507+0000","last_fullsized":"2024-02-19T21:17:33.446507+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:59.160470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:59.160470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":683,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443606+0000","last_change":"2024-02-19T21:17:14.459244+0000","last_active":"2024-02-19T21:17:33.443606+0000","last_peered":"2024-02-19T21:17:33.443606+0000","last_clean":"2024-02-19T21:17:33.443606+0000","last_became_active":"2024-02-19T21:17:14.457894+0000","last_became_peered":"2024-02-19T21:17:14.457894+0000","last_unstale":"2024-02-19T21:17:33.443606+0000","last_undegraded":"2024-02-19T21:17:33.443606+0000","last_fullsized":"2024-02-19T21:17:33.443606+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:53.918521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:53.918521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":610,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443385+0000","last_change":"2024-02-19T21:17:29.471559+0000","last_active":"2024-02-19T21:17:33.443385+0000","last_peered":"2024-02-19T21:17:33.443385+0000","last_clean":"2024-02-19T21:17:33.443385+0000","last_became_active":"2024-02-19T21:17:29.470965+0000","last_became_peered":"2024-02-19T21:17:29.470965+0000","last_unstale":"2024-02-19T21:17:33.443385+0000","last_undegraded":"2024-02-19T21:17:33.443385+0000","last_fullsized":"2024-02-19T21:17:33.443385+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:02.096039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:02.096039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"459'20","reported_seq":576,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.445984+0000","last_change":"2024-02-19T21:17:29.590205+0000","last_active":"2024-02-19T21:17:33.445984+0000","last_peered":"2024-02-19T21:17:33.445984+0000","last_clean":"2024-02-19T21:17:33.445984+0000","last_became_active":"2024-02-19T21:17:29.515652+0000","last_became_peered":"2024-02-19T21:17:29.515652+0000","last_unstale":"2024-02-19T21:17:33.445984+0000","last_undegraded":"2024-02-19T21:17:33.445984+0000","last_fullsized":"2024-02-19T21:17:33.445984+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:14:47.087905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:47.087905+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":329,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443027+0000","last_change":"2024-02-19T21:17:14.458183+0000","last_active":"2024-02-19T21:17:33.443027+0000","last_peered":"2024-02-19T21:17:33.443027+0000","last_clean":"2024-02-19T21:17:33.443027+0000","last_became_active":"2024-02-19T21:17:14.457439+0000","last_became_peered":"2024-02-19T21:17:14.457439+0000","last_unstale":"2024-02-19T21:17:33.443027+0000","last_undegraded":"2024-02-19T21:17:33.443027+0000","last_fullsized":"2024-02-19T21:17:33.443027+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:42.974035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_clean_scrub_stamp":"2024-02-19T21:15:42.974035+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"762'11","reported_seq":73,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-02-19T21:17:34.499461+0000","last_change":"2024-02-19T21:16:59.035631+0000","last_active":"2024-02-19T21:17:34.499461+0000","last_peered":"2024-02-19T21:17:34.499461+0000","last_clean":"2024-02-19T21:17:34.499461+0000","last_became_active":"2024-02-19T21:16:58.571623+0000","last_became_peered":"2024-02-19T21:16:58.571623+0000","last_unstale":"2024-02-19T21:17:34.499461+0000","last_undegraded":"2024-02-19T21:17:34.499461+0000","last_fullsized":"2024-02-19T21:17:34.499461+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"640'10","last_scrub_stamp":"2024-02-19T21:15:37.998310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:37.998310+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":198,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.446468+0000","last_change":"2024-02-19T21:17:14.522246+0000","last_active":"2024-02-19T21:17:33.446468+0000","last_peered":"2024-02-19T21:17:33.446468+0000","last_clean":"2024-02-19T21:17:33.446468+0000","last_became_active":"2024-02-19T21:17:14.460658+0000","last_became_peered":"2024-02-19T21:17:14.460658+0000","last_unstale":"2024-02-19T21:17:33.446468+0000","last_undegraded":"2024-02-19T21:17:33.446468+0000","last_fullsized":"2024-02-19T21:17:33.446468+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:14:18.606934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:18.606934+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":207,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.439030+0000","last_change":"2024-02-19T21:17:29.466525+0000","last_active":"2024-02-19T21:17:33.439030+0000","last_peered":"2024-02-19T21:17:33.439030+0000","last_clean":"2024-02-19T21:17:33.439030+0000","last_became_active":"2024-02-19T21:17:29.465914+0000","last_became_peered":"2024-02-19T21:17:29.465914+0000","last_unstale":"2024-02-19T21:17:33.439030+0000","last_undegraded":"2024-02-19T21:17:33.439030+0000","last_fullsized":"2024-02-19T21:17:33.439030+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:23.608673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:23.608673+0000","last_clean_scrub_stamp":"2024-02-19T21:16:23.608673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"732'19","reported_seq":834,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.445476+0000","last_change":"2024-02-19T21:17:14.466898+0000","last_active":"2024-02-19T21:17:33.445476+0000","last_peered":"2024-02-19T21:17:33.445476+0000","last_clean":"2024-02-19T21:17:33.445476+0000","last_became_active":"2024-02-19T21:17:14.440315+0000","last_became_peered":"2024-02-19T21:17:14.440315+0000","last_unstale":"2024-02-19T21:17:33.445476+0000","last_undegraded":"2024-02-19T21:17:33.445476+0000","last_fullsized":"2024-02-19T21:17:33.445476+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'19","last_scrub_stamp":"2024-02-19T21:16:22.028901+0000","last_deep_scrub":"732'19","last_deep_scrub_stamp":"2024-02-19T21:16:22.028901+0000","last_clean_scrub_stamp":"2024-02-19T21:16:22.028901+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":394,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443596+0000","last_change":"2024-02-19T21:16:18.179449+0000","last_active":"2024-02-19T21:17:33.443596+0000","last_peered":"2024-02-19T21:17:33.443596+0000","last_clean":"2024-02-19T21:17:33.443596+0000","last_became_active":"2024-02-19T21:14:41.415500+0000","last_became_peered":"2024-02-19T21:14:41.415500+0000","last_unstale":"2024-02-19T21:17:33.443596+0000","last_undegraded":"2024-02-19T21:17:33.443596+0000","last_fullsized":"2024-02-19T21:17:33.443596+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:18.179370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:16:18.179370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"786'13","reported_seq":670,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-02-19T21:17:34.445149+0000","last_change":"2024-02-19T21:17:29.519868+0000","last_active":"2024-02-19T21:17:34.445149+0000","last_peered":"2024-02-19T21:17:34.445149+0000","last_clean":"2024-02-19T21:17:34.445149+0000","last_became_active":"2024-02-19T21:17:29.460968+0000","last_became_peered":"2024-02-19T21:17:29.460968+0000","last_unstale":"2024-02-19T21:17:34.445149+0000","last_undegraded":"2024-02-19T21:17:34.445149+0000","last_fullsized":"2024-02-19T21:17:34.445149+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"304'10","last_scrub_stamp":"2024-02-19T21:15:43.054055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:43.054055+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":407,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444322+0000","last_change":"2024-02-19T21:17:29.472952+0000","last_active":"2024-02-19T21:17:33.444322+0000","last_peered":"2024-02-19T21:17:33.444322+0000","last_clean":"2024-02-19T21:17:33.444322+0000","last_became_active":"2024-02-19T21:17:29.472526+0000","last_became_peered":"2024-02-19T21:17:29.472526+0000","last_unstale":"2024-02-19T21:17:33.444322+0000","last_undegraded":"2024-02-19T21:17:33.444322+0000","last_fullsized":"2024-02-19T21:17:33.444322+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:41.829068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:43.936153+0000","last_clean_scrub_stamp":"2024-02-19T21:15:41.829068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"100'7","reported_seq":19,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.442609+0000","last_change":"2024-02-19T21:17:29.717868+0000","last_active":"2024-02-19T21:17:33.442609+0000","last_peered":"2024-02-19T21:17:33.442609+0000","last_clean":"2024-02-19T21:17:33.442609+0000","last_became_active":"2024-02-19T21:17:29.511622+0000","last_became_peered":"2024-02-19T21:17:29.511622+0000","last_unstale":"2024-02-19T21:17:33.442609+0000","last_undegraded":"2024-02-19T21:17:33.442609+0000","last_fullsized":"2024-02-19T21:17:33.442609+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'7","last_scrub_stamp":"2024-02-19T21:15:03.372256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:03.372256+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":20,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"459'15","reported_seq":575,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.448117+0000","last_change":"2024-02-19T21:17:29.928094+0000","last_active":"2024-02-19T21:17:33.448117+0000","last_peered":"2024-02-19T21:17:33.448117+0000","last_clean":"2024-02-19T21:17:33.448117+0000","last_became_active":"2024-02-19T21:17:29.530519+0000","last_became_peered":"2024-02-19T21:17:29.530519+0000","last_unstale":"2024-02-19T21:17:33.448117+0000","last_undegraded":"2024-02-19T21:17:33.448117+0000","last_fullsized":"2024-02-19T21:17:33.448117+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:16:14.086191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:16:14.086191+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":212,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.439461+0000","last_change":"2024-02-19T21:17:29.465453+0000","last_active":"2024-02-19T21:17:33.439461+0000","last_peered":"2024-02-19T21:17:33.439461+0000","last_clean":"2024-02-19T21:17:33.439461+0000","last_became_active":"2024-02-19T21:17:29.465096+0000","last_became_peered":"2024-02-19T21:17:29.465096+0000","last_unstale":"2024-02-19T21:17:33.439461+0000","last_undegraded":"2024-02-19T21:17:33.439461+0000","last_fullsized":"2024-02-19T21:17:33.439461+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:51.653989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:01.684958+0000","last_clean_scrub_stamp":"2024-02-19T21:16:51.653989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":530,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444325+0000","last_change":"2024-02-19T21:15:41.081514+0000","last_active":"2024-02-19T21:17:33.444325+0000","last_peered":"2024-02-19T21:17:33.444325+0000","last_clean":"2024-02-19T21:17:33.444325+0000","last_became_active":"2024-02-19T21:14:41.398865+0000","last_became_peered":"2024-02-19T21:14:41.398865+0000","last_unstale":"2024-02-19T21:17:33.444325+0000","last_undegraded":"2024-02-19T21:17:33.444325+0000","last_fullsized":"2024-02-19T21:17:33.444325+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:41.081437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_clean_scrub_stamp":"2024-02-19T21:15:41.081437+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":503,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.447732+0000","last_change":"2024-02-19T21:16:58.566442+0000","last_active":"2024-02-19T21:17:33.447732+0000","last_peered":"2024-02-19T21:17:33.447732+0000","last_clean":"2024-02-19T21:17:33.447732+0000","last_became_active":"2024-02-19T21:16:58.563992+0000","last_became_peered":"2024-02-19T21:16:58.563992+0000","last_unstale":"2024-02-19T21:17:33.447732+0000","last_undegraded":"2024-02-19T21:17:33.447732+0000","last_fullsized":"2024-02-19T21:17:33.447732+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:49.956589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:49.956589+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":221,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.440068+0000","last_change":"2024-02-19T21:17:29.465960+0000","last_active":"2024-02-19T21:17:33.440068+0000","last_peered":"2024-02-19T21:17:33.440068+0000","last_clean":"2024-02-19T21:17:33.440068+0000","last_became_active":"2024-02-19T21:17:29.464433+0000","last_became_peered":"2024-02-19T21:17:29.464433+0000","last_unstale":"2024-02-19T21:17:33.440068+0000","last_undegraded":"2024-02-19T21:17:33.440068+0000","last_fullsized":"2024-02-19T21:17:33.440068+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:25.560242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:16:25.560242+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"459'19","reported_seq":437,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443750+0000","last_change":"2024-02-19T21:14:56.610314+0000","last_active":"2024-02-19T21:17:33.443750+0000","last_peered":"2024-02-19T21:17:33.443750+0000","last_clean":"2024-02-19T21:17:33.443750+0000","last_became_active":"2024-02-19T21:14:16.380020+0000","last_became_peered":"2024-02-19T21:14:16.380020+0000","last_unstale":"2024-02-19T21:17:33.443750+0000","last_undegraded":"2024-02-19T21:17:33.443750+0000","last_fullsized":"2024-02-19T21:17:33.443750+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:14:56.610233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:56.610233+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":805,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.449011+0000","last_change":"2024-02-19T21:16:58.555941+0000","last_active":"2024-02-19T21:17:33.449011+0000","last_peered":"2024-02-19T21:17:33.449011+0000","last_clean":"2024-02-19T21:17:33.449011+0000","last_became_active":"2024-02-19T21:16:58.554991+0000","last_became_peered":"2024-02-19T21:16:58.554991+0000","last_unstale":"2024-02-19T21:17:33.449011+0000","last_undegraded":"2024-02-19T21:17:33.449011+0000","last_fullsized":"2024-02-19T21:17:33.449011+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.936143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:09.756096+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.936143+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":792,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.448539+0000","last_change":"2024-02-19T21:17:14.452145+0000","last_active":"2024-02-19T21:17:33.448539+0000","last_peered":"2024-02-19T21:17:33.448539+0000","last_clean":"2024-02-19T21:17:33.448539+0000","last_became_active":"2024-02-19T21:17:14.447544+0000","last_became_peered":"2024-02-19T21:17:14.447544+0000","last_unstale":"2024-02-19T21:17:33.448539+0000","last_undegraded":"2024-02-19T21:17:33.448539+0000","last_fullsized":"2024-02-19T21:17:33.448539+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:46.944774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:46.944774+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":543,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443733+0000","last_change":"2024-02-19T21:17:29.453703+0000","last_active":"2024-02-19T21:17:33.443733+0000","last_peered":"2024-02-19T21:17:33.443733+0000","last_clean":"2024-02-19T21:17:33.443733+0000","last_became_active":"2024-02-19T21:17:29.453394+0000","last_became_peered":"2024-02-19T21:17:29.453394+0000","last_unstale":"2024-02-19T21:17:33.443733+0000","last_undegraded":"2024-02-19T21:17:33.443733+0000","last_fullsized":"2024-02-19T21:17:33.443733+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:36.137008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:36.137008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":543,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.446975+0000","last_change":"2024-02-19T21:17:29.452342+0000","last_active":"2024-02-19T21:17:33.446975+0000","last_peered":"2024-02-19T21:17:33.446975+0000","last_clean":"2024-02-19T21:17:33.446975+0000","last_became_active":"2024-02-19T21:17:29.452050+0000","last_became_peered":"2024-02-19T21:17:29.452050+0000","last_unstale":"2024-02-19T21:17:33.446975+0000","last_undegraded":"2024-02-19T21:17:33.446975+0000","last_fullsized":"2024-02-19T21:17:33.446975+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:59.059572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:59.059572+0000","last_clean_scrub_stamp":"2024-02-19T21:14:59.059572+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"555'10","reported_seq":46,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.441185+0000","last_change":"2024-02-19T21:16:58.594094+0000","last_active":"2024-02-19T21:17:33.441185+0000","last_peered":"2024-02-19T21:17:33.441185+0000","last_clean":"2024-02-19T21:17:33.441185+0000","last_became_active":"2024-02-19T21:16:58.576112+0000","last_became_peered":"2024-02-19T21:16:58.576112+0000","last_unstale":"2024-02-19T21:17:33.441185+0000","last_undegraded":"2024-02-19T21:17:33.441185+0000","last_fullsized":"2024-02-19T21:17:33.441185+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'10","last_scrub_stamp":"2024-02-19T21:15:40.997592+0000","last_deep_scrub":"555'10","last_deep_scrub_stamp":"2024-02-19T21:13:12.409036+0000","last_clean_scrub_stamp":"2024-02-19T21:15:40.997592+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":784,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.439846+0000","last_change":"2024-02-19T21:17:29.466197+0000","last_active":"2024-02-19T21:17:33.439846+0000","last_peered":"2024-02-19T21:17:33.439846+0000","last_clean":"2024-02-19T21:17:33.439846+0000","last_became_active":"2024-02-19T21:17:29.465658+0000","last_became_peered":"2024-02-19T21:17:29.465658+0000","last_unstale":"2024-02-19T21:17:33.439846+0000","last_undegraded":"2024-02-19T21:17:33.439846+0000","last_fullsized":"2024-02-19T21:17:33.439846+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:44.964433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:44.964433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":189,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.438812+0000","last_change":"2024-02-19T21:17:14.455414+0000","last_active":"2024-02-19T21:17:33.438812+0000","last_peered":"2024-02-19T21:17:33.438812+0000","last_clean":"2024-02-19T21:17:33.438812+0000","last_became_active":"2024-02-19T21:17:14.453305+0000","last_became_peered":"2024-02-19T21:17:14.453305+0000","last_unstale":"2024-02-19T21:17:33.438812+0000","last_undegraded":"2024-02-19T21:17:33.438812+0000","last_fullsized":"2024-02-19T21:17:33.438812+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:54.200442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:54.200442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.442684+0000","last_change":"2024-02-19T21:17:29.480023+0000","last_active":"2024-02-19T21:17:33.442684+0000","last_peered":"2024-02-19T21:17:33.442684+0000","last_clean":"2024-02-19T21:17:33.442684+0000","last_became_active":"2024-02-19T21:17:29.479365+0000","last_became_peered":"2024-02-19T21:17:29.479365+0000","last_unstale":"2024-02-19T21:17:33.442684+0000","last_undegraded":"2024-02-19T21:17:33.442684+0000","last_fullsized":"2024-02-19T21:17:33.442684+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:50.846595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:14:50.846595+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":530,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.446059+0000","last_change":"2024-02-19T21:17:29.464754+0000","last_active":"2024-02-19T21:17:33.446059+0000","last_peered":"2024-02-19T21:17:33.446059+0000","last_clean":"2024-02-19T21:17:33.446059+0000","last_became_active":"2024-02-19T21:17:29.464522+0000","last_became_peered":"2024-02-19T21:17:29.464522+0000","last_unstale":"2024-02-19T21:17:33.446059+0000","last_undegraded":"2024-02-19T21:17:33.446059+0000","last_fullsized":"2024-02-19T21:17:33.446059+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:50.223212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:50.223212+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"459'19","reported_seq":599,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443971+0000","last_change":"2024-02-19T21:17:14.511307+0000","last_active":"2024-02-19T21:17:33.443971+0000","last_peered":"2024-02-19T21:17:33.443971+0000","last_clean":"2024-02-19T21:17:33.443971+0000","last_became_active":"2024-02-19T21:17:14.447428+0000","last_became_peered":"2024-02-19T21:17:14.447428+0000","last_unstale":"2024-02-19T21:17:33.443971+0000","last_undegraded":"2024-02-19T21:17:33.443971+0000","last_fullsized":"2024-02-19T21:17:33.443971+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:15:42.047800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:42.047800+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":214,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.440193+0000","last_change":"2024-02-19T21:17:29.470490+0000","last_active":"2024-02-19T21:17:33.440193+0000","last_peered":"2024-02-19T21:17:33.440193+0000","last_clean":"2024-02-19T21:17:33.440193+0000","last_became_active":"2024-02-19T21:17:29.470158+0000","last_became_peered":"2024-02-19T21:17:29.470158+0000","last_unstale":"2024-02-19T21:17:33.440193+0000","last_undegraded":"2024-02-19T21:17:33.440193+0000","last_fullsized":"2024-02-19T21:17:33.440193+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:56.796039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:16:56.796039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":678,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.441603+0000","last_change":"2024-02-19T21:17:14.446541+0000","last_active":"2024-02-19T21:17:33.441603+0000","last_peered":"2024-02-19T21:17:33.441603+0000","last_clean":"2024-02-19T21:17:33.441603+0000","last_became_active":"2024-02-19T21:17:14.446205+0000","last_became_peered":"2024-02-19T21:17:14.446205+0000","last_unstale":"2024-02-19T21:17:33.441603+0000","last_undegraded":"2024-02-19T21:17:33.441603+0000","last_fullsized":"2024-02-19T21:17:33.441603+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:19.588167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:19.588167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":548,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444404+0000","last_change":"2024-02-19T21:17:03.995996+0000","last_active":"2024-02-19T21:17:33.444404+0000","last_peered":"2024-02-19T21:17:33.444404+0000","last_clean":"2024-02-19T21:17:33.444404+0000","last_became_active":"2024-02-19T21:14:41.398484+0000","last_became_peered":"2024-02-19T21:14:41.398484+0000","last_unstale":"2024-02-19T21:17:33.444404+0000","last_undegraded":"2024-02-19T21:17:33.444404+0000","last_fullsized":"2024-02-19T21:17:33.444404+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:03.995934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:17:03.995934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":198,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.439798+0000","last_change":"2024-02-19T21:16:58.562011+0000","last_active":"2024-02-19T21:17:33.439798+0000","last_peered":"2024-02-19T21:17:33.439798+0000","last_clean":"2024-02-19T21:17:33.439798+0000","last_became_active":"2024-02-19T21:16:58.561666+0000","last_became_peered":"2024-02-19T21:16:58.561666+0000","last_unstale":"2024-02-19T21:17:33.439798+0000","last_undegraded":"2024-02-19T21:17:33.439798+0000","last_fullsized":"2024-02-19T21:17:33.439798+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:36.239929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:15:36.239929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"459'20","reported_seq":423,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.446420+0000","last_change":"2024-02-19T21:17:14.473672+0000","last_active":"2024-02-19T21:17:33.446420+0000","last_peered":"2024-02-19T21:17:33.446420+0000","last_clean":"2024-02-19T21:17:33.446420+0000","last_became_active":"2024-02-19T21:17:14.447432+0000","last_became_peered":"2024-02-19T21:17:14.447432+0000","last_unstale":"2024-02-19T21:17:33.446420+0000","last_undegraded":"2024-02-19T21:17:33.446420+0000","last_fullsized":"2024-02-19T21:17:33.446420+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:14:55.895741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:14:55.895741+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":456,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.441674+0000","last_change":"2024-02-19T21:17:14.460175+0000","last_active":"2024-02-19T21:17:33.441674+0000","last_peered":"2024-02-19T21:17:33.441674+0000","last_clean":"2024-02-19T21:17:33.441674+0000","last_became_active":"2024-02-19T21:17:14.459790+0000","last_became_peered":"2024-02-19T21:17:14.459790+0000","last_unstale":"2024-02-19T21:17:33.441674+0000","last_undegraded":"2024-02-19T21:17:33.441674+0000","last_fullsized":"2024-02-19T21:17:33.441674+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:35.006289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:35.006289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"786'14","reported_seq":250,"reported_epoch":791,"state":"active+clean","last_fresh":"2024-02-19T21:17:34.469659+0000","last_change":"2024-02-19T21:17:29.532679+0000","last_active":"2024-02-19T21:17:34.469659+0000","last_peered":"2024-02-19T21:17:34.469659+0000","last_clean":"2024-02-19T21:17:34.469659+0000","last_became_active":"2024-02-19T21:17:29.470014+0000","last_became_peered":"2024-02-19T21:17:29.470014+0000","last_unstale":"2024-02-19T21:17:34.469659+0000","last_undegraded":"2024-02-19T21:17:34.469659+0000","last_fullsized":"2024-02-19T21:17:34.469659+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"623'12","last_scrub_stamp":"2024-02-19T21:15:52.229069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:52.229069+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":19,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444347+0000","last_change":"2024-02-19T21:17:29.517141+0000","last_active":"2024-02-19T21:17:33.444347+0000","last_peered":"2024-02-19T21:17:33.444347+0000","last_clean":"2024-02-19T21:17:33.444347+0000","last_became_active":"2024-02-19T21:17:29.479134+0000","last_became_peered":"2024-02-19T21:17:29.479134+0000","last_unstale":"2024-02-19T21:17:33.444347+0000","last_undegraded":"2024-02-19T21:17:33.444347+0000","last_fullsized":"2024-02-19T21:17:33.444347+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:16:52.271650+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:04:18.877385+0000","last_clean_scrub_stamp":"2024-02-19T21:16:52.271650+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.442685+0000","last_change":"2024-02-19T21:17:29.511912+0000","last_active":"2024-02-19T21:17:33.442685+0000","last_peered":"2024-02-19T21:17:33.442685+0000","last_clean":"2024-02-19T21:17:33.442685+0000","last_became_active":"2024-02-19T21:17:29.511415+0000","last_became_peered":"2024-02-19T21:17:29.511415+0000","last_unstale":"2024-02-19T21:17:33.442685+0000","last_undegraded":"2024-02-19T21:17:33.442685+0000","last_fullsized":"2024-02-19T21:17:33.442685+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:05.308987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:17:05.308987+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"459'19","reported_seq":218,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.440166+0000","last_change":"2024-02-19T21:16:58.591819+0000","last_active":"2024-02-19T21:17:33.440166+0000","last_peered":"2024-02-19T21:17:33.440166+0000","last_clean":"2024-02-19T21:17:33.440166+0000","last_became_active":"2024-02-19T21:16:58.560717+0000","last_became_peered":"2024-02-19T21:16:58.560717+0000","last_unstale":"2024-02-19T21:17:33.440166+0000","last_undegraded":"2024-02-19T21:17:33.440166+0000","last_fullsized":"2024-02-19T21:17:33.440166+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:16:26.516791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:16:26.516791+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":543,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444048+0000","last_change":"2024-02-19T21:15:37.108298+0000","last_active":"2024-02-19T21:17:33.444048+0000","last_peered":"2024-02-19T21:17:33.444048+0000","last_clean":"2024-02-19T21:17:33.444048+0000","last_became_active":"2024-02-19T21:14:41.402278+0000","last_became_peered":"2024-02-19T21:14:41.402278+0000","last_unstale":"2024-02-19T21:17:33.444048+0000","last_undegraded":"2024-02-19T21:17:33.444048+0000","last_fullsized":"2024-02-19T21:17:33.444048+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:37.108220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:37.108220+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444676+0000","last_change":"2024-02-19T21:17:29.523364+0000","last_active":"2024-02-19T21:17:33.444676+0000","last_peered":"2024-02-19T21:17:33.444676+0000","last_clean":"2024-02-19T21:17:33.444676+0000","last_became_active":"2024-02-19T21:17:29.522614+0000","last_became_peered":"2024-02-19T21:17:29.522614+0000","last_unstale":"2024-02-19T21:17:33.444676+0000","last_undegraded":"2024-02-19T21:17:33.444676+0000","last_fullsized":"2024-02-19T21:17:33.444676+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:29.029281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:16:29.029281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":785,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.448848+0000","last_change":"2024-02-19T21:17:29.516732+0000","last_active":"2024-02-19T21:17:33.448848+0000","last_peered":"2024-02-19T21:17:33.448848+0000","last_clean":"2024-02-19T21:17:33.448848+0000","last_became_active":"2024-02-19T21:17:29.516415+0000","last_became_peered":"2024-02-19T21:17:29.516415+0000","last_unstale":"2024-02-19T21:17:33.448848+0000","last_undegraded":"2024-02-19T21:17:33.448848+0000","last_fullsized":"2024-02-19T21:17:33.448848+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:33.888719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:33.888719+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":753,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.448420+0000","last_change":"2024-02-19T21:17:29.531160+0000","last_active":"2024-02-19T21:17:33.448420+0000","last_peered":"2024-02-19T21:17:33.448420+0000","last_clean":"2024-02-19T21:17:33.448420+0000","last_became_active":"2024-02-19T21:17:29.530233+0000","last_became_peered":"2024-02-19T21:17:29.530233+0000","last_unstale":"2024-02-19T21:17:33.448420+0000","last_undegraded":"2024-02-19T21:17:33.448420+0000","last_fullsized":"2024-02-19T21:17:33.448420+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:54.259446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:16:54.259446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"459'34","reported_seq":31,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444182+0000","last_change":"2024-02-19T21:17:29.545290+0000","last_active":"2024-02-19T21:17:33.444182+0000","last_peered":"2024-02-19T21:17:33.444182+0000","last_clean":"2024-02-19T21:17:33.444182+0000","last_became_active":"2024-02-19T21:17:29.511788+0000","last_became_peered":"2024-02-19T21:17:29.511788+0000","last_unstale":"2024-02-19T21:17:33.444182+0000","last_undegraded":"2024-02-19T21:17:33.444182+0000","last_fullsized":"2024-02-19T21:17:33.444182+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'34","last_scrub_stamp":"2024-02-19T21:16:23.017246+0000","last_deep_scrub":"419'22","last_deep_scrub_stamp":"2024-02-19T21:10:19.109244+0000","last_clean_scrub_stamp":"2024-02-19T21:16:23.017246+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":216,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.438922+0000","last_change":"2024-02-19T21:17:29.465695+0000","last_active":"2024-02-19T21:17:33.438922+0000","last_peered":"2024-02-19T21:17:33.438922+0000","last_clean":"2024-02-19T21:17:33.438922+0000","last_became_active":"2024-02-19T21:17:29.465426+0000","last_became_peered":"2024-02-19T21:17:29.465426+0000","last_unstale":"2024-02-19T21:17:33.438922+0000","last_undegraded":"2024-02-19T21:17:33.438922+0000","last_fullsized":"2024-02-19T21:17:33.438922+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:21.555391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:16:21.555391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":792,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.449157+0000","last_change":"2024-02-19T21:17:04.344598+0000","last_active":"2024-02-19T21:17:33.449157+0000","last_peered":"2024-02-19T21:17:33.449157+0000","last_clean":"2024-02-19T21:17:33.449157+0000","last_became_active":"2024-02-19T21:14:41.415463+0000","last_became_peered":"2024-02-19T21:14:41.415463+0000","last_unstale":"2024-02-19T21:17:33.449157+0000","last_undegraded":"2024-02-19T21:17:33.449157+0000","last_fullsized":"2024-02-19T21:17:33.449157+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:04.344520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:17:04.344520+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":403,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444118+0000","last_change":"2024-02-19T21:16:58.557861+0000","last_active":"2024-02-19T21:17:33.444118+0000","last_peered":"2024-02-19T21:17:33.444118+0000","last_clean":"2024-02-19T21:17:33.444118+0000","last_became_active":"2024-02-19T21:16:58.557229+0000","last_became_peered":"2024-02-19T21:16:58.557229+0000","last_unstale":"2024-02-19T21:17:33.444118+0000","last_undegraded":"2024-02-19T21:17:33.444118+0000","last_fullsized":"2024-02-19T21:17:33.444118+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:53.265931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:16:53.265931+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":189,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.438977+0000","last_change":"2024-02-19T21:15:04.202157+0000","last_active":"2024-02-19T21:17:33.438977+0000","last_peered":"2024-02-19T21:17:33.438977+0000","last_clean":"2024-02-19T21:17:33.438977+0000","last_became_active":"2024-02-19T21:14:41.395278+0000","last_became_peered":"2024-02-19T21:14:41.395278+0000","last_unstale":"2024-02-19T21:17:33.438977+0000","last_undegraded":"2024-02-19T21:17:33.438977+0000","last_fullsized":"2024-02-19T21:17:33.438977+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:04.202042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:04.202042+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"459'18","reported_seq":575,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.445564+0000","last_change":"2024-02-19T21:15:54.109530+0000","last_active":"2024-02-19T21:17:33.445564+0000","last_peered":"2024-02-19T21:17:33.445564+0000","last_clean":"2024-02-19T21:17:33.445564+0000","last_became_active":"2024-02-19T21:10:41.307690+0000","last_became_peered":"2024-02-19T21:10:41.307690+0000","last_unstale":"2024-02-19T21:17:33.445564+0000","last_undegraded":"2024-02-19T21:17:33.445564+0000","last_fullsized":"2024-02-19T21:17:33.445564+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'18","last_scrub_stamp":"2024-02-19T21:15:54.109426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:15:54.109426+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":539,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444478+0000","last_change":"2024-02-19T21:17:14.462215+0000","last_active":"2024-02-19T21:17:33.444478+0000","last_peered":"2024-02-19T21:17:33.444478+0000","last_clean":"2024-02-19T21:17:33.444478+0000","last_became_active":"2024-02-19T21:17:14.457742+0000","last_became_peered":"2024-02-19T21:17:14.457742+0000","last_unstale":"2024-02-19T21:17:33.444478+0000","last_undegraded":"2024-02-19T21:17:33.444478+0000","last_fullsized":"2024-02-19T21:17:33.444478+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:02.998627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:17:02.998627+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":398,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443598+0000","last_change":"2024-02-19T21:17:29.471478+0000","last_active":"2024-02-19T21:17:33.443598+0000","last_peered":"2024-02-19T21:17:33.443598+0000","last_clean":"2024-02-19T21:17:33.443598+0000","last_became_active":"2024-02-19T21:17:29.471059+0000","last_became_peered":"2024-02-19T21:17:29.471059+0000","last_unstale":"2024-02-19T21:17:33.443598+0000","last_undegraded":"2024-02-19T21:17:33.443598+0000","last_fullsized":"2024-02-19T21:17:33.443598+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:59.654949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:59.654949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":534,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.447018+0000","last_change":"2024-02-19T21:16:58.579937+0000","last_active":"2024-02-19T21:17:33.447018+0000","last_peered":"2024-02-19T21:17:33.447018+0000","last_clean":"2024-02-19T21:17:33.447018+0000","last_became_active":"2024-02-19T21:16:58.579623+0000","last_became_peered":"2024-02-19T21:16:58.579623+0000","last_unstale":"2024-02-19T21:17:33.447018+0000","last_undegraded":"2024-02-19T21:17:33.447018+0000","last_fullsized":"2024-02-19T21:17:33.447018+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:58.020115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:14:58.020115+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":744,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.447782+0000","last_change":"2024-02-19T21:17:29.521584+0000","last_active":"2024-02-19T21:17:33.447782+0000","last_peered":"2024-02-19T21:17:33.447782+0000","last_clean":"2024-02-19T21:17:33.447782+0000","last_became_active":"2024-02-19T21:17:29.521224+0000","last_became_peered":"2024-02-19T21:17:29.521224+0000","last_unstale":"2024-02-19T21:17:33.447782+0000","last_undegraded":"2024-02-19T21:17:33.447782+0000","last_fullsized":"2024-02-19T21:17:33.447782+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:17:06.319621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:17:06.319621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"459'19","reported_seq":428,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443232+0000","last_change":"2024-02-19T21:16:23.174002+0000","last_active":"2024-02-19T21:17:33.443232+0000","last_peered":"2024-02-19T21:17:33.443232+0000","last_clean":"2024-02-19T21:17:33.443232+0000","last_became_active":"2024-02-19T21:14:41.390502+0000","last_became_peered":"2024-02-19T21:14:41.390502+0000","last_unstale":"2024-02-19T21:17:33.443232+0000","last_undegraded":"2024-02-19T21:17:33.443232+0000","last_fullsized":"2024-02-19T21:17:33.443232+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:16:23.173933+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:16:23.173933+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444226+0000","last_change":"2024-02-19T21:17:29.511876+0000","last_active":"2024-02-19T21:17:33.444226+0000","last_peered":"2024-02-19T21:17:33.444226+0000","last_clean":"2024-02-19T21:17:33.444226+0000","last_became_active":"2024-02-19T21:17:29.511523+0000","last_became_peered":"2024-02-19T21:17:29.511523+0000","last_unstale":"2024-02-19T21:17:33.444226+0000","last_undegraded":"2024-02-19T21:17:33.444226+0000","last_fullsized":"2024-02-19T21:17:33.444226+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:30.005724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:30.005724+0000","last_clean_scrub_stamp":"2024-02-19T21:16:30.005724+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":47,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443207+0000","last_change":"2024-02-19T21:17:14.459191+0000","last_active":"2024-02-19T21:17:33.443207+0000","last_peered":"2024-02-19T21:17:33.443207+0000","last_clean":"2024-02-19T21:17:33.443207+0000","last_became_active":"2024-02-19T21:17:14.458449+0000","last_became_peered":"2024-02-19T21:17:14.458449+0000","last_unstale":"2024-02-19T21:17:33.443207+0000","last_undegraded":"2024-02-19T21:17:33.443207+0000","last_fullsized":"2024-02-19T21:17:33.443207+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:39.980983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:15:39.980983+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":622,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.447544+0000","last_change":"2024-02-19T21:17:29.517931+0000","last_active":"2024-02-19T21:17:33.447544+0000","last_peered":"2024-02-19T21:17:33.447544+0000","last_clean":"2024-02-19T21:17:33.447544+0000","last_became_active":"2024-02-19T21:17:29.517621+0000","last_became_peered":"2024-02-19T21:17:29.517621+0000","last_unstale":"2024-02-19T21:17:33.447544+0000","last_undegraded":"2024-02-19T21:17:33.447544+0000","last_fullsized":"2024-02-19T21:17:33.447544+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:24.176889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:16:24.176889+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":15,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444743+0000","last_change":"2024-02-19T21:17:29.523670+0000","last_active":"2024-02-19T21:17:33.444743+0000","last_peered":"2024-02-19T21:17:33.444743+0000","last_clean":"2024-02-19T21:17:33.444743+0000","last_became_active":"2024-02-19T21:17:29.523066+0000","last_became_peered":"2024-02-19T21:17:29.523066+0000","last_unstale":"2024-02-19T21:17:33.444743+0000","last_undegraded":"2024-02-19T21:17:33.444743+0000","last_fullsized":"2024-02-19T21:17:33.444743+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:46.226419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:46.226419+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"459'7","reported_seq":380,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443677+0000","last_change":"2024-02-19T21:16:14.156064+0000","last_active":"2024-02-19T21:17:33.443677+0000","last_peered":"2024-02-19T21:17:33.443677+0000","last_clean":"2024-02-19T21:17:33.443677+0000","last_became_active":"2024-02-19T21:10:41.297295+0000","last_became_peered":"2024-02-19T21:10:41.297295+0000","last_unstale":"2024-02-19T21:17:33.443677+0000","last_undegraded":"2024-02-19T21:17:33.443677+0000","last_fullsized":"2024-02-19T21:17:33.443677+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'7","last_scrub_stamp":"2024-02-19T21:16:14.155997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:16:14.155997+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":707,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.439605+0000","last_change":"2024-02-19T21:17:14.442035+0000","last_active":"2024-02-19T21:17:33.439605+0000","last_peered":"2024-02-19T21:17:33.439605+0000","last_clean":"2024-02-19T21:17:33.439605+0000","last_became_active":"2024-02-19T21:17:14.441671+0000","last_became_peered":"2024-02-19T21:17:14.441671+0000","last_unstale":"2024-02-19T21:17:33.439605+0000","last_undegraded":"2024-02-19T21:17:33.439605+0000","last_fullsized":"2024-02-19T21:17:33.439605+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:57.818313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:16:57.818313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":480,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.442835+0000","last_change":"2024-02-19T21:17:14.443798+0000","last_active":"2024-02-19T21:17:33.442835+0000","last_peered":"2024-02-19T21:17:33.442835+0000","last_clean":"2024-02-19T21:17:33.442835+0000","last_became_active":"2024-02-19T21:17:14.443414+0000","last_became_peered":"2024-02-19T21:17:14.443414+0000","last_unstale":"2024-02-19T21:17:33.442835+0000","last_undegraded":"2024-02-19T21:17:33.442835+0000","last_fullsized":"2024-02-19T21:17:33.442835+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:54.900062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:54.900062+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":323,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.447720+0000","last_change":"2024-02-19T21:17:29.517421+0000","last_active":"2024-02-19T21:17:33.447720+0000","last_peered":"2024-02-19T21:17:33.447720+0000","last_clean":"2024-02-19T21:17:33.447720+0000","last_became_active":"2024-02-19T21:17:29.517020+0000","last_became_peered":"2024-02-19T21:17:29.517020+0000","last_unstale":"2024-02-19T21:17:33.447720+0000","last_undegraded":"2024-02-19T21:17:33.447720+0000","last_fullsized":"2024-02-19T21:17:33.447720+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:47.947061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:47.947061+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":517,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.439899+0000","last_change":"2024-02-19T21:16:58.561263+0000","last_active":"2024-02-19T21:17:33.439899+0000","last_peered":"2024-02-19T21:17:33.439899+0000","last_clean":"2024-02-19T21:17:33.439899+0000","last_became_active":"2024-02-19T21:16:58.560594+0000","last_became_peered":"2024-02-19T21:16:58.560594+0000","last_unstale":"2024-02-19T21:17:33.439899+0000","last_undegraded":"2024-02-19T21:17:33.439899+0000","last_fullsized":"2024-02-19T21:17:33.439899+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:26.978641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:16:26.978641+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.18","version":"623'19","reported_seq":434,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.441690+0000","last_change":"2024-02-19T21:14:21.589221+0000","last_active":"2024-02-19T21:17:33.441690+0000","last_peered":"2024-02-19T21:17:33.441690+0000","last_clean":"2024-02-19T21:17:33.441690+0000","last_became_active":"2024-02-19T21:14:16.383577+0000","last_became_peered":"2024-02-19T21:14:16.383577+0000","last_unstale":"2024-02-19T21:17:33.441690+0000","last_undegraded":"2024-02-19T21:17:33.441690+0000","last_fullsized":"2024-02-19T21:17:33.441690+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"623'19","last_scrub_stamp":"2024-02-19T21:14:21.589138+0000","last_deep_scrub":"623'19","last_deep_scrub_stamp":"2024-02-19T21:14:21.589138+0000","last_clean_scrub_stamp":"2024-02-19T21:14:21.589138+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":552,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.446682+0000","last_change":"2024-02-19T21:17:29.453080+0000","last_active":"2024-02-19T21:17:33.446682+0000","last_peered":"2024-02-19T21:17:33.446682+0000","last_clean":"2024-02-19T21:17:33.446682+0000","last_became_active":"2024-02-19T21:17:29.452735+0000","last_became_peered":"2024-02-19T21:17:29.452735+0000","last_unstale":"2024-02-19T21:17:33.446682+0000","last_undegraded":"2024-02-19T21:17:33.446682+0000","last_fullsized":"2024-02-19T21:17:33.446682+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:00.101587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:15:00.101587+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":783,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.447634+0000","last_change":"2024-02-19T21:17:29.521140+0000","last_active":"2024-02-19T21:17:33.447634+0000","last_peered":"2024-02-19T21:17:33.447634+0000","last_clean":"2024-02-19T21:17:33.447634+0000","last_became_active":"2024-02-19T21:17:29.518437+0000","last_became_peered":"2024-02-19T21:17:29.518437+0000","last_unstale":"2024-02-19T21:17:33.447634+0000","last_undegraded":"2024-02-19T21:17:33.447634+0000","last_fullsized":"2024-02-19T21:17:33.447634+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:29.232586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:16:29.232586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":389,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443248+0000","last_change":"2024-02-19T21:17:14.447333+0000","last_active":"2024-02-19T21:17:33.443248+0000","last_peered":"2024-02-19T21:17:33.443248+0000","last_clean":"2024-02-19T21:17:33.443248+0000","last_became_active":"2024-02-19T21:17:14.446746+0000","last_became_peered":"2024-02-19T21:17:14.446746+0000","last_unstale":"2024-02-19T21:17:33.443248+0000","last_undegraded":"2024-02-19T21:17:33.443248+0000","last_fullsized":"2024-02-19T21:17:33.443248+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:53.080222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:15:53.080222+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":627,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443324+0000","last_change":"2024-02-19T21:17:29.471391+0000","last_active":"2024-02-19T21:17:33.443324+0000","last_peered":"2024-02-19T21:17:33.443324+0000","last_clean":"2024-02-19T21:17:33.443324+0000","last_became_active":"2024-02-19T21:17:29.471004+0000","last_became_peered":"2024-02-19T21:17:29.471004+0000","last_unstale":"2024-02-19T21:17:33.443324+0000","last_undegraded":"2024-02-19T21:17:33.443324+0000","last_fullsized":"2024-02-19T21:17:33.443324+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:55.013484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:14:55.013484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"459'27","reported_seq":673,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443654+0000","last_change":"2024-02-19T21:17:29.537623+0000","last_active":"2024-02-19T21:17:33.443654+0000","last_peered":"2024-02-19T21:17:33.443654+0000","last_clean":"2024-02-19T21:17:33.443654+0000","last_became_active":"2024-02-19T21:17:29.452764+0000","last_became_peered":"2024-02-19T21:17:29.452764+0000","last_unstale":"2024-02-19T21:17:33.443654+0000","last_undegraded":"2024-02-19T21:17:33.443654+0000","last_fullsized":"2024-02-19T21:17:33.443654+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'27","last_scrub_stamp":"2024-02-19T21:17:00.035102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:17:00.035102+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":554,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.444107+0000","last_change":"2024-02-19T21:17:14.449052+0000","last_active":"2024-02-19T21:17:33.444107+0000","last_peered":"2024-02-19T21:17:33.444107+0000","last_clean":"2024-02-19T21:17:33.444107+0000","last_became_active":"2024-02-19T21:17:14.447391+0000","last_became_peered":"2024-02-19T21:17:14.447391+0000","last_unstale":"2024-02-19T21:17:33.444107+0000","last_undegraded":"2024-02-19T21:17:33.444107+0000","last_fullsized":"2024-02-19T21:17:33.444107+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:13:52.779962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:13:52.779962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":389,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.441722+0000","last_change":"2024-02-19T21:14:55.609881+0000","last_active":"2024-02-19T21:17:33.441722+0000","last_peered":"2024-02-19T21:17:33.441722+0000","last_clean":"2024-02-19T21:17:33.441722+0000","last_became_active":"2024-02-19T21:14:16.384069+0000","last_became_peered":"2024-02-19T21:14:16.384069+0000","last_unstale":"2024-02-19T21:17:33.441722+0000","last_undegraded":"2024-02-19T21:17:33.441722+0000","last_fullsized":"2024-02-19T21:17:33.441722+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:14:55.609805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:14:55.609805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":539,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.443222+0000","last_change":"2024-02-19T21:17:14.447861+0000","last_active":"2024-02-19T21:17:33.443222+0000","last_peered":"2024-02-19T21:17:33.443222+0000","last_clean":"2024-02-19T21:17:33.443222+0000","last_became_active":"2024-02-19T21:17:14.447381+0000","last_became_peered":"2024-02-19T21:17:14.447381+0000","last_unstale":"2024-02-19T21:17:33.443222+0000","last_undegraded":"2024-02-19T21:17:33.443222+0000","last_fullsized":"2024-02-19T21:17:33.443222+0000","mapping_epoch":774,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":775,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:16:29.894604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:16:29.894604+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":198,"reported_epoch":790,"state":"active+clean","last_fresh":"2024-02-19T21:17:33.439752+0000","last_change":"2024-02-19T21:17:29.465065+0000","last_active":"2024-02-19T21:17:33.439752+0000","last_peered":"2024-02-19T21:17:33.439752+0000","last_clean":"2024-02-19T21:17:33.439752+0000","last_became_active":"2024-02-19T21:17:29.464415+0000","last_became_peered":"2024-02-19T21:17:29.464415+0000","last_unstale":"2024-02-19T21:17:33.439752+0000","last_undegraded":"2024-02-19T21:17:33.439752+0000","last_fullsized":"2024-02-19T21:17:33.439752+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:15:46.084709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:15:46.084709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":254,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":234,"num_pg":8,"stat_sum":{"num_bytes":4000297,"num_objects":43,"num_object_clones":1,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":21,"num_read_kb":5,"num_write":63,"num_write_kb":3965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":12000382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6537216,"data_stored":6537216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":24,"acting":24,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1099,"num_read_kb":856,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":679,"num_bytes_recovered":14038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":11876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":730,"ondisk_log_size":730,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":962,"num_read_kb":957,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":5292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":126,"acting":126,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":20,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":227633266886,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":6,"up_from":45,"seq":193273528524,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302996,"kb_used_data":5772,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93442156,"statfs":{"total":95995035648,"available":95684767744,"internally_reserved":0,"allocated":5910528,"data_stored":3785268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.48599999999999999,"15min":0.60899999999999999},"min":{"1min":0.26100000000000001,"5min":0.246,"15min":0.19900000000000001},"max":{"1min":0.59699999999999998,"5min":0.80300000000000005,"15min":17.486999999999998},"last":0.442},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.496,"15min":0.65400000000000003},"min":{"1min":0.311,"5min":0.30299999999999999,"15min":0.215},"max":{"1min":0.71899999999999997,"5min":1.669,"15min":17.43},"last":0.54600000000000004}]},{"osd":1,"last update":"Mon Feb 19 21:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.48899999999999999,"15min":0.62},"min":{"1min":0.28899999999999998,"5min":0.248,"15min":0.22800000000000001},"max":{"1min":0.64400000000000002,"5min":0.92900000000000005,"15min":14.692},"last":0.37},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.50900000000000001,"15min":0.66500000000000004},"min":{"1min":0.34399999999999997,"5min":0.30299999999999999,"15min":0.23400000000000001},"max":{"1min":0.627,"5min":1.1990000000000001,"15min":14.795},"last":0.48599999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.52400000000000002,"15min":0.60299999999999998},"min":{"1min":0.33600000000000002,"5min":0.28599999999999998,"15min":0.253},"max":{"1min":0.63,"5min":3.3050000000000002,"15min":6.931},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.52000000000000002,"15min":0.67500000000000004},"min":{"1min":0.33300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67800000000000005,"5min":1.794,"15min":14.391},"last":0.50700000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51600000000000001,"15min":0.70199999999999996},"min":{"1min":0.33900000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.70399999999999996,"5min":0.91700000000000004,"15min":14.510999999999999},"last":0.499},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.59699999999999998,"15min":0.69199999999999995},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.316},"max":{"1min":0.69199999999999995,"5min":4.085,"15min":17.585000000000001},"last":0.51800000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44700000000000001,"15min":0.433},"min":{"1min":0.30399999999999999,"5min":0.26800000000000002,"15min":0.22900000000000001},"max":{"1min":0.64500000000000002,"5min":1.972,"15min":2.9289999999999998},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.47999999999999998,"15min":0.45700000000000002},"min":{"1min":0.314,"5min":0.28799999999999998,"15min":0.22700000000000001},"max":{"1min":0.63600000000000001,"5min":2.0499999999999998,"15min":3.431},"last":0.32700000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.50700000000000001,"15min":0.48699999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.26300000000000001},"max":{"1min":0.69999999999999996,"5min":1.867,"15min":2.9620000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53600000000000003,"15min":0.51800000000000002},"min":{"1min":0.35499999999999998,"5min":0.32000000000000001,"15min":0.26700000000000002},"max":{"1min":0.70299999999999996,"5min":1.788,"15min":2.7010000000000001},"last":0.52900000000000003}]},{"osd":7,"last update":"Mon Feb 19 21:16:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.48799999999999999,"15min":0.48899999999999999},"min":{"1min":0.28000000000000003,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65800000000000003,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.28399999999999997},"max":{"1min":0.71799999999999997,"5min":1.835,"15min":1.835},"last":0.42899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575075,"num_pgs":124,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305136,"kb_used_data":3816,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93440016,"statfs":{"total":95995035648,"available":95682576384,"internally_reserved":0,"allocated":3907584,"data_stored":1726274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:17:25 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.33900000000000002,"15min":0.54200000000000004},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.16700000000000001},"max":{"1min":1.288,"5min":1.288,"15min":27.777000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.35899999999999999,"15min":0.57099999999999995},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19400000000000001},"max":{"1min":1.736,"5min":1.736,"15min":27.631},"last":0.39300000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.35799999999999998,"15min":0.68899999999999995},"min":{"1min":0.23200000000000001,"5min":0.20000000000000001,"15min":0.19600000000000001},"max":{"1min":0.495,"5min":2.0230000000000001,"15min":27.798999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.36899999999999999,"15min":0.70299999999999996},"min":{"1min":0.27600000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.434,"5min":2.089,"15min":27.606999999999999},"last":0.41699999999999998}]},{"osd":3,"last update":"Mon Feb 19 21:17:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.35499999999999998,"15min":0.54800000000000004},"min":{"1min":0.25700000000000001,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":0.75,"5min":0.75,"15min":15.574999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.40000000000000002,"15min":0.64600000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":27.82},"last":0.47899999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:17:30 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.56100000000000005,"15min":0.65000000000000002},"min":{"1min":0.39200000000000002,"5min":0.313,"15min":0.26800000000000002},"max":{"1min":1.262,"5min":1.262,"15min":9.6430000000000007},"last":0.75},{"interface":"front","average":{"1min":0.623,"5min":0.57299999999999995,"15min":0.754},"min":{"1min":0.44400000000000001,"5min":0.379,"15min":0.308},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":27.707000000000001},"last":0.66800000000000004}]},{"osd":5,"last update":"Mon Feb 19 21:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56299999999999994,"15min":0.73499999999999999},"min":{"1min":0.432,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.85999999999999999,"5min":0.98599999999999999,"15min":27.529},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53700000000000003,"15min":0.65800000000000003},"min":{"1min":0.39200000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.76200000000000001,"5min":0.98099999999999998,"15min":9.6799999999999997},"last":0.64500000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:17:15 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.55600000000000005,"15min":0.65200000000000002},"min":{"1min":0.40400000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.147,"5min":1.147,"15min":9.6989999999999998},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.59299999999999997,"15min":0.77300000000000002},"min":{"1min":0.40100000000000002,"5min":0.38500000000000001,"15min":0.35199999999999998},"max":{"1min":0.876,"5min":0.94999999999999996,"15min":27.733000000000001},"last":0.73499999999999999}]},{"osd":7,"last update":"Mon Feb 19 21:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59899999999999998,"15min":0.77400000000000002},"min":{"1min":0.40100000000000002,"5min":0.39900000000000002,"15min":0.36699999999999999},"max":{"1min":0.83599999999999997,"5min":3.4670000000000001,"15min":27.576000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58999999999999997,"15min":0.80400000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.375},"max":{"1min":0.73999999999999999,"5min":0.875,"15min":27.756},"last":0.69699999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738601,"num_pgs":110,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313304,"kb_used_data":3792,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":93431848,"statfs":{"total":95995035648,"available":95674212352,"internally_reserved":0,"allocated":3883008,"data_stored":1737299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:17:21 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.34999999999999998,"15min":0.60499999999999998},"min":{"1min":0.23599999999999999,"5min":0.19400000000000001,"15min":0.188},"max":{"1min":0.51300000000000001,"5min":1.855,"15min":20.966999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.35399999999999998,"15min":0.627},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.193},"max":{"1min":0.46500000000000002,"5min":1.613,"15min":20.366},"last":0.23300000000000001}]},{"osd":2,"last update":"Mon Feb 19 21:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35799999999999998,"15min":0.70599999999999996},"min":{"1min":0.26200000000000001,"5min":0.214,"15min":0.19900000000000001},"max":{"1min":0.52000000000000002,"5min":0.83499999999999996,"15min":27.916},"last":0.315},{"interface":"front","average":{"1min":0.38,"5min":0.379,"15min":0.629},"min":{"1min":0.28100000000000003,"5min":0.23000000000000001,"15min":0.19600000000000001},"max":{"1min":0.56499999999999995,"5min":0.58599999999999997,"15min":22.004000000000001},"last":0.30299999999999999}]},{"osd":3,"last update":"Mon Feb 19 21:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38900000000000001,"15min":0.71399999999999997},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.503,"5min":0.71999999999999997,"15min":25.669},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.41499999999999998,"15min":0.66200000000000003},"min":{"1min":0.30199999999999999,"5min":0.27100000000000002,"15min":0.26300000000000001},"max":{"1min":0.51800000000000002,"5min":0.77200000000000002,"15min":25.006},"last":0.36299999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:17:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57199999999999995,"15min":0.73699999999999999},"min":{"1min":0.38600000000000001,"5min":0.33600000000000002,"15min":0.28799999999999998},"max":{"1min":0.71799999999999997,"5min":1.0089999999999999,"15min":20.411999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58399999999999996,"15min":0.76000000000000001},"min":{"1min":0.441,"5min":0.29499999999999998,"15min":0.28699999999999998},"max":{"1min":0.80700000000000005,"5min":0.91200000000000003,"15min":19.986999999999998},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:16:55 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.58599999999999997,"15min":0.80200000000000005},"min":{"1min":0.41599999999999998,"5min":0.31900000000000001,"15min":0.27200000000000002},"max":{"1min":0.81799999999999995,"5min":0.93100000000000005,"15min":20.021999999999998},"last":0.625},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.61699999999999999,"15min":0.79800000000000004},"min":{"1min":0.503,"5min":0.35499999999999998,"15min":0.30399999999999999},"max":{"1min":0.79300000000000004,"5min":0.94299999999999995,"15min":20.908999999999999},"last":0.59399999999999997}]},{"osd":6,"last update":"Mon Feb 19 21:17:02 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58699999999999997,"15min":0.755},"min":{"1min":0.40400000000000003,"5min":0.34100000000000003,"15min":0.33900000000000002},"max":{"1min":0.83199999999999996,"5min":1.0349999999999999,"15min":20.445},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.623,"15min":0.79200000000000004},"min":{"1min":0.439,"5min":0.35099999999999998,"15min":0.33400000000000002},"max":{"1min":0.875,"5min":2.6120000000000001,"15min":20.047999999999998},"last":0.57999999999999996}]},{"osd":7,"last update":"Mon Feb 19 21:16:37 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60399999999999998,"15min":0.79200000000000004},"min":{"1min":0.48599999999999999,"5min":0.40100000000000002,"15min":0.312},"max":{"1min":0.76100000000000001,"5min":0.84499999999999997,"15min":20.940999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.81000000000000005},"min":{"1min":0.46300000000000002,"5min":0.41399999999999998,"15min":0.34899999999999998},"max":{"1min":0.89300000000000002,"5min":1.0660000000000001,"15min":20.471},"last":0.51100000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411550,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302956,"kb_used_data":5732,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93442196,"statfs":{"total":95995035648,"available":95684808704,"internally_reserved":0,"allocated":5869568,"data_stored":3743453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.29899999999999999,"5min":0.67100000000000004,"15min":0.63900000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.19400000000000001},"max":{"1min":0.376,"5min":28.702999999999999,"15min":28.702999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.67200000000000004,"15min":0.69499999999999995},"min":{"1min":0.23999999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.434,"5min":28.744,"15min":28.744},"last":0.32800000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.50800000000000001,"15min":0.754},"min":{"1min":0.218,"5min":0.218,"15min":0.20100000000000001},"max":{"1min":9.7940000000000005,"5min":9.7940000000000005,"15min":28.882000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.41399999999999998,"15min":0.68600000000000005},"min":{"1min":0.26300000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.55900000000000005,"5min":5.0739999999999998,"15min":28.382999999999999},"last":0.39900000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:17:11 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.47899999999999998,"15min":0.72099999999999997},"min":{"1min":0.26000000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.55300000000000005,"5min":4.2270000000000003,"15min":28.742000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.52300000000000002,"15min":0.65300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.25700000000000001},"max":{"1min":0.45900000000000002,"5min":6.8040000000000003,"15min":26.719999999999999},"last":0.35799999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:16:33 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.56000000000000005,"15min":0.68600000000000005},"min":{"1min":0.316,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.4079999999999999,"5min":2.4079999999999999,"15min":24.356000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.01,"5min":0.65900000000000003,"15min":0.72799999999999998},"min":{"1min":0.376,"5min":0.33700000000000002,"15min":0.32400000000000001},"max":{"1min":4.3540000000000001,"5min":4.3540000000000001,"15min":20.707000000000001},"last":0.499}]},{"osd":5,"last update":"Mon Feb 19 21:16:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.64100000000000001,"15min":0.68100000000000005},"min":{"1min":0.41399999999999998,"5min":0.35999999999999999,"15min":0.33500000000000002},"max":{"1min":4.2990000000000004,"5min":4.2990000000000004,"15min":9.9499999999999993},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.59999999999999998,"15min":0.77700000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":2.8380000000000001,"5min":2.8380000000000001,"15min":26.606000000000002},"last":0.48199999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:17:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60199999999999998,"15min":0.79200000000000004},"min":{"1min":0.372,"5min":0.36499999999999999,"15min":0.34300000000000003},"max":{"1min":0.80100000000000005,"5min":2.198,"15min":24.381},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.69999999999999996,"15min":0.75800000000000001},"min":{"1min":0.45000000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77600000000000002,"5min":4.2759999999999998,"15min":28.706},"last":0.45000000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.61799999999999999,"15min":0.78800000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.34200000000000003},"max":{"1min":2.6440000000000001,"5min":2.6440000000000001,"15min":26.658000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61699999999999999,"15min":0.73799999999999999},"min":{"1min":0.38700000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.631,"5min":2.3700000000000001,"15min":24.402000000000001},"last":0.50800000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182618,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":301044,"kb_used_data":3820,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93444108,"statfs":{"total":95995035648,"available":95686766592,"internally_reserved":0,"allocated":3911680,"data_stored":1757159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.33400000000000002,"15min":0.70699999999999996},"min":{"1min":0.245,"5min":0.192,"15min":0.156},"max":{"1min":0.47599999999999998,"5min":1.5449999999999999,"15min":77.662000000000006},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35399999999999998,"15min":0.48099999999999998},"min":{"1min":0.27000000000000002,"5min":0.20899999999999999,"15min":0.19700000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":17.456},"last":0.27700000000000002}]},{"osd":1,"last update":"Mon Feb 19 21:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34999999999999998,"15min":0.51300000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.22800000000000001},"max":{"1min":0.53900000000000003,"5min":0.67600000000000005,"15min":17.327999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35999999999999999,"15min":0.49299999999999999},"min":{"1min":0.25700000000000001,"5min":0.20699999999999999,"15min":0.20000000000000001},"max":{"1min":0.56299999999999994,"5min":0.98999999999999999,"15min":15.004},"last":0.35299999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.38300000000000001,"15min":0.65100000000000002},"min":{"1min":0.28199999999999997,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":0.67200000000000004,"5min":0.92600000000000005,"15min":56.890999999999998},"last":0.315},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.378,"15min":0.63300000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.20899999999999999},"max":{"1min":0.59099999999999997,"5min":1.3200000000000001,"15min":44.600999999999999},"last":0.25600000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:17:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.55000000000000004,"15min":0.63500000000000001},"min":{"1min":0.38200000000000001,"5min":0.33800000000000002,"15min":0.31900000000000001},"max":{"1min":0.94699999999999995,"5min":1.3340000000000001,"15min":12.847},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.54800000000000004,"15min":0.70099999999999996},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29299999999999998},"max":{"1min":0.89600000000000002,"5min":1.3560000000000001,"15min":26.393999999999998},"last":0.48599999999999999}]},{"osd":5,"last update":"Mon Feb 19 21:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.54100000000000004,"15min":0.67500000000000004},"min":{"1min":0.34899999999999998,"5min":0.33200000000000002,"15min":0.313},"max":{"1min":1.002,"5min":1.002,"15min":17.401},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55700000000000005,"15min":0.65400000000000003},"min":{"1min":0.39200000000000002,"5min":0.35999999999999999,"15min":0.313},"max":{"1min":0.82399999999999995,"5min":1.397,"15min":12.882},"last":0.47299999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:17:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.56499999999999995,"15min":0.65200000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.34799999999999998},"max":{"1min":1.036,"5min":1.036,"15min":17.367000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56699999999999995,"15min":0.65500000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.33700000000000002},"max":{"1min":0.876,"5min":1.3660000000000001,"15min":12.904},"last":0.50900000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:16:37 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55800000000000005,"15min":0.65600000000000003},"min":{"1min":0.39700000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80400000000000005,"5min":1.038,"15min":17.428999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.58199999999999996,"15min":0.80500000000000005},"min":{"1min":0.40400000000000003,"5min":0.374,"15min":0.34499999999999997},"max":{"1min":0.67100000000000004,"5min":1.3879999999999999,"15min":44.378},"last":0.46200000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986390,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307080,"kb_used_data":5760,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93438072,"statfs":{"total":95995035648,"available":95680585728,"internally_reserved":0,"allocated":5898240,"data_stored":3767633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:16:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.50600000000000001,"15min":0.625},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.22500000000000001},"max":{"1min":0.87,"5min":0.87,"15min":19.878},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.502,"15min":0.85499999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.20499999999999999},"max":{"1min":0.74199999999999999,"5min":0.82999999999999996,"15min":70.063000000000002},"last":0.53700000000000003}]},{"osd":1,"last update":"Mon Feb 19 21:16:32 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.504,"15min":0.629},"min":{"1min":0.312,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.68600000000000005,"5min":1.6499999999999999,"15min":26.510999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.504,"15min":0.67900000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.23200000000000001},"max":{"1min":0.82699999999999996,"5min":0.89700000000000002,"15min":25.949000000000002},"last":0.46999999999999997}]},{"osd":2,"last update":"Mon Feb 19 21:16:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52500000000000002,"15min":0.72099999999999997},"min":{"1min":0.29499999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.89200000000000002,"5min":2.2040000000000002,"15min":37.320999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.879,"5min":0.58399999999999996,"15min":1.0720000000000001},"min":{"1min":0.35999999999999999,"5min":0.31900000000000001,"15min":0.253},"max":{"1min":6.8700000000000001,"5min":6.8700000000000001,"15min":69.936999999999998},"last":0.45500000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:16:32 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.53000000000000003,"15min":0.91400000000000003},"min":{"1min":0.36599999999999999,"5min":0.34499999999999997,"15min":0.27700000000000002},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":69.793000000000006},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.54600000000000004,"15min":0.75800000000000001},"min":{"1min":0.39000000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":1.2430000000000001,"5min":1.621,"15min":17.986999999999998},"last":0.52500000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.432,"15min":0.42499999999999999},"min":{"1min":0.26400000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.78000000000000003,"5min":1.532,"15min":3.2799999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.33200000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.80300000000000005,"5min":1.5700000000000001,"15min":4.4340000000000002},"last":0.435}]},{"osd":6,"last update":"Mon Feb 19 21:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.503,"15min":0.51500000000000001},"min":{"1min":0.28899999999999998,"5min":0.27700000000000002,"15min":0.23499999999999999},"max":{"1min":3.3140000000000001,"5min":3.3140000000000001,"15min":8.0969999999999995},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.58399999999999996,"15min":0.56000000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.26800000000000002},"max":{"1min":3.3679999999999999,"5min":3.3679999999999999,"15min":4.5609999999999999},"last":0.501}]},{"osd":7,"last update":"Mon Feb 19 21:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52000000000000002,"15min":0.53600000000000003},"min":{"1min":0.32100000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73599999999999999,"5min":1.5049999999999999,"15min":6.0730000000000004},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.501,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33100000000000002,"5min":0.309,"15min":0.26400000000000001},"max":{"1min":0.72699999999999998,"5min":0.79300000000000004,"15min":3.202},"last":0.54700000000000004}]}]},{"osd":5,"up_from":0,"seq":158913790161,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":2888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2072576,"data_stored":2072576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":98304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2109440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2113536,"data_stored":2113536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:17:42.339 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool set default.rgw.meta pgp_num 32 2024-02-19T21:17:42.415 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.089+0000 7f867512a700 1 -- 172.21.15.163:0/3646123409 >> v1:172.21.15.163:6790/0 conn(0x7f8670106530 legacy=0x7f8670108960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:42.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.089+0000 7f867512a700 1 -- 172.21.15.163:0/3646123409 shutdown_connections 2024-02-19T21:17:42.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.089+0000 7f867512a700 1 -- 172.21.15.163:0/3646123409 >> 172.21.15.163:0/3646123409 conn(0x7f86700f35a0 msgr2=0x7f86700f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:42.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.089+0000 7f867512a700 1 -- 172.21.15.163:0/3646123409 shutdown_connections 2024-02-19T21:17:42.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.089+0000 7f867512a700 1 -- 172.21.15.163:0/3646123409 wait complete. 2024-02-19T21:17:42.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.090+0000 7f867512a700 1 librados: starting msgr at 2024-02-19T21:17:42.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.090+0000 7f867512a700 1 librados: starting objecter 2024-02-19T21:17:42.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.090+0000 7f867512a700 1 Processor -- start 2024-02-19T21:17:42.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.090+0000 7f867512a700 1 -- start start 2024-02-19T21:17:42.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.090+0000 7f867512a700 1 librados: setting wanted keys 2024-02-19T21:17:42.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.090+0000 7f867512a700 1 librados: calling monclient init 2024-02-19T21:17:42.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f867512a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8670119580 con 0x7f867006b170 2024-02-19T21:17:42.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f867512a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f867011a760 con 0x7f867006bc60 2024-02-19T21:17:42.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f867512a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f867011b960 con 0x7f8670106530 2024-02-19T21:17:42.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f866ed9d700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f867006b170 0x7f86701019c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:43598/0 (socket says 172.21.15.163:43598) 2024-02-19T21:17:42.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f866f59e700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f8670106530 0x7f8670117c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38136/0 (socket says 172.21.15.163:38136) 2024-02-19T21:17:42.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f866ed9d700 1 -- 172.21.15.163:0/3573938270 learned_addr learned my addr 172.21.15.163:0/3573938270 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:42.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 10 client.?.objecter ms_handle_connect 0x7f867006b170 2024-02-19T21:17:42.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:42.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 10 client.?.objecter ms_handle_connect 0x7f8670106530 2024-02-19T21:17:42.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:42.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 10 client.?.objecter ms_handle_connect 0x7f867006bc60 2024-02-19T21:17:42.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:42.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415340961 0 0) 0x7f867011b960 con 0x7f8670106530 2024-02-19T21:17:42.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8658003030 con 0x7f8670106530 2024-02-19T21:17:42.417 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929815426 0 0) 0x7f8670119580 con 0x7f867006b170 2024-02-19T21:17:42.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f867011b960 con 0x7f867006b170 2024-02-19T21:17:42.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486829012 0 0) 0x7f867011a760 con 0x7f867006bc60 2024-02-19T21:17:42.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8670119580 con 0x7f867006bc60 2024-02-19T21:17:42.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2296769724 0 0) 0x7f8658003030 con 0x7f8670106530 2024-02-19T21:17:42.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f867011a760 con 0x7f8670106530 2024-02-19T21:17:42.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f86680022d0 con 0x7f8670106530 2024-02-19T21:17:42.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 38841847 0 0) 0x7f867011b960 con 0x7f867006b170 2024-02-19T21:17:42.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8658003030 con 0x7f867006b170 2024-02-19T21:17:42.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3605408036 0 0) 0x7f8670119580 con 0x7f867006bc60 2024-02-19T21:17:42.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f867011b960 con 0x7f867006bc60 2024-02-19T21:17:42.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8660001920 con 0x7f867006b170 2024-02-19T21:17:42.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8664001840 con 0x7f867006bc60 2024-02-19T21:17:42.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 263777530 0 0) 0x7f867011a760 con 0x7f8670106530 2024-02-19T21:17:42.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 >> v1:172.21.15.193:6789/0 conn(0x7f867006bc60 legacy=0x7f86701020d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:42.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 >> v1:172.21.15.163:6789/0 conn(0x7f867006b170 legacy=0x7f86701019c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:42.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867011cb60 con 0x7f8670106530 2024-02-19T21:17:42.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.091+0000 7f867512a700 1 -- 172.21.15.163:0/3573938270 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f867011bb50 con 0x7f8670106530 2024-02-19T21:17:42.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.092+0000 7f867512a700 10 client.50642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:42.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.092+0000 7f867512a700 1 -- 172.21.15.163:0/3573938270 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f867011bfc0 con 0x7f8670106530 2024-02-19T21:17:42.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.092+0000 7f867512a700 1 librados: init done 2024-02-19T21:17:42.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.092+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8668002610 con 0x7f8670106530 2024-02-19T21:17:42.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.092+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8668003a40 con 0x7f8670106530 2024-02-19T21:17:42.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.093+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f8668003ba0 con 0x7f8670106530 2024-02-19T21:17:42.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.093+0000 7f867512a700 1 -- 172.21.15.163:0/3573938270 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f865c004f20 con 0x7f8670106530 2024-02-19T21:17:42.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.093+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(798..798 src has 623..798) v4 ==== 8413+0+0 (unknown 1020430013 0 0) 0x7f86680793c0 con 0x7f8670106530 2024-02-19T21:17:42.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.093+0000 7f8657fff700 10 client.50642.objecter ms_dispatch 0x7f8670001550 osd_map(798..798 src has 623..798) v4 2024-02-19T21:17:42.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.093+0000 7f8657fff700 3 client.50642.objecter handle_osd_map got epochs [798,798] > 0 2024-02-19T21:17:42.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.093+0000 7f8657fff700 3 client.50642.objecter handle_osd_map decoding full epoch 798 2024-02-19T21:17:42.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.094+0000 7f8657fff700 20 client.50642.objecter dump_active .. 0 homeless 2024-02-19T21:17:42.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.094+0000 7f8657fff700 10 client.50642.objecter ms_handle_connect 0x7f8658063dd0 2024-02-19T21:17:42.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.098+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f866804d560 con 0x7f8670106530 2024-02-19T21:17:42.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.277+0000 7f867512a700 1 -- 172.21.15.163:0/3573938270 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f865c0051d0 con 0x7f8670106530 2024-02-19T21:17:42.420 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:17:42.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.445+0000 7f8657fff700 1 -- 172.21.15.163:0/3573938270 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v799) v1 ==== 101+0+0 (unknown 1746158094 0 0) 0x7f866804b9d0 con 0x7f8670106530 2024-02-19T21:17:42.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.454+0000 7f867512a700 10 librados: watch_flush enter 2024-02-19T21:17:42.467 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:42.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.465+0000 7f867512a700 10 librados: watch_flush exit 2024-02-19T21:17:42.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.465+0000 7f867512a700 20 client.50642.objecter shutdown clearing up homeless session... 2024-02-19T21:17:42.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.465+0000 7f867512a700 10 client.50642.objecter successfully canceled tick 2024-02-19T21:17:42.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.466+0000 7f867512a700 1 -- 172.21.15.163:0/3573938270 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8658063dd0 legacy=0x7f8658066290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:42.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.466+0000 7f867512a700 1 -- 172.21.15.163:0/3573938270 >> v1:172.21.15.163:6790/0 conn(0x7f8670106530 legacy=0x7f8670117c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:42.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.466+0000 7f867512a700 1 -- 172.21.15.163:0/3573938270 shutdown_connections 2024-02-19T21:17:42.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.466+0000 7f867512a700 1 -- 172.21.15.163:0/3573938270 >> 172.21.15.163:0/3573938270 conn(0x7f86700f35a0 msgr2=0x7f8670109950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:42.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.466+0000 7f867512a700 1 -- 172.21.15.163:0/3573938270 shutdown_connections 2024-02-19T21:17:42.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.466+0000 7f867512a700 1 -- 172.21.15.163:0/3573938270 wait complete. 2024-02-19T21:17:42.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:42.466+0000 7f867512a700 1 librados: shutdown 2024-02-19T21:17:42.469 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:17:42.495 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:17:42.495 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:42.500 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:42.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:17:42.601 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:42.630 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:42.650 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:42.652 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:42.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:42.721 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:42.753 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:42.780 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:43.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:41.342243+0000 mgr.y (mgr.14550) 662 : cluster [DBG] pgmap v1209: 182 pgs: 1 active+remapped+backfill_toofull, 181 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-19T21:17:43.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:42.278908+0000 mon.c (mon.2) 443 : audit [INF] from='client.? v1:172.21.15.163:0/3573938270' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:43.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:42.279230+0000 mon.a (mon.0) 3727 : audit [INF] from='client.50642 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:43.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:42.435036+0000 mon.a (mon.0) 3728 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:17:43.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:42.442889+0000 mon.a (mon.0) 3729 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:17:43.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:42.443054+0000 mon.a (mon.0) 3730 : audit [INF] from='client.50642 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:17:43.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:42.443103+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e799: 8 total, 8 up, 6 in 2024-02-19T21:17:43.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:42.544451+0000 mon.a (mon.0) 3732 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:43.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:42.544866+0000 mon.a (mon.0) 3733 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-142"}]: dispatch 2024-02-19T21:17:43.394 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:41.342243+0000 mgr.y (mgr.14550) 662 : cluster [DBG] pgmap v1209: 182 pgs: 1 active+remapped+backfill_toofull, 181 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-19T21:17:43.395 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:42.278908+0000 mon.c (mon.2) 443 : audit [INF] from='client.? v1:172.21.15.163:0/3573938270' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:43.395 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:42.279230+0000 mon.a (mon.0) 3727 : audit [INF] from='client.50642 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:43.395 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:42.435036+0000 mon.a (mon.0) 3728 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:17:43.395 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:42.442889+0000 mon.a (mon.0) 3729 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:17:43.395 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:42.443054+0000 mon.a (mon.0) 3730 : audit [INF] from='client.50642 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:17:43.395 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:42.443103+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e799: 8 total, 8 up, 6 in 2024-02-19T21:17:43.395 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:42.544451+0000 mon.a (mon.0) 3732 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:43.395 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:42.544866+0000 mon.a (mon.0) 3733 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-142"}]: dispatch 2024-02-19T21:17:43.395 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:41.342243+0000 mgr.y (mgr.14550) 662 : cluster [DBG] pgmap v1209: 182 pgs: 1 active+remapped+backfill_toofull, 181 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-02-19T21:17:43.395 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:42.278908+0000 mon.c (mon.2) 443 : audit [INF] from='client.? v1:172.21.15.163:0/3573938270' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:43.395 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:42.279230+0000 mon.a (mon.0) 3727 : audit [INF] from='client.50642 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:17:43.395 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:42.435036+0000 mon.a (mon.0) 3728 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-19T21:17:43.395 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:42.442889+0000 mon.a (mon.0) 3729 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:17:43.396 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:42.443054+0000 mon.a (mon.0) 3730 : audit [INF] from='client.50642 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:17:43.396 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:42.443103+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e799: 8 total, 8 up, 6 in 2024-02-19T21:17:43.396 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:42.544451+0000 mon.a (mon.0) 3732 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:43.396 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:42.544866+0000 mon.a (mon.0) 3733 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-142"}]: dispatch 2024-02-19T21:17:43.748 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:17:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:17:43] "GET /metrics HTTP/1.1" 200 192864 "" "Prometheus/2.33.4" 2024-02-19T21:17:43.932 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:44.159 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:44.162 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:44.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:44.263 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:44.292 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:44.445 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:43.342651+0000 mgr.y (mgr.14550) 663 : cluster [DBG] pgmap v1212: 214 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 181 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-19T21:17:44.445 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:43.359004+0000 osd.1 (osd.1) 360 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:17:44.445 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:43.360160+0000 osd.1 (osd.1) 361 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:17:44.445 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:43.447991+0000 mon.a (mon.0) 3734 : cluster [DBG] osdmap e800: 8 total, 8 up, 6 in 2024-02-19T21:17:44.445 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:43.641942+0000 osd.2 (osd.2) 316 : cluster [DBG] 5.d scrub starts 2024-02-19T21:17:44.445 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:43.643023+0000 osd.2 (osd.2) 317 : cluster [DBG] 5.d scrub ok 2024-02-19T21:17:44.445 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:44.421209+0000 mon.a (mon.0) 3735 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:17:44.490 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:44.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:44.594 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:44.622 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:44.695 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:44.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:43.342651+0000 mgr.y (mgr.14550) 663 : cluster [DBG] pgmap v1212: 214 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 181 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-19T21:17:44.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:43.359004+0000 osd.1 (osd.1) 360 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:17:44.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:43.360160+0000 osd.1 (osd.1) 361 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:17:44.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:43.447991+0000 mon.a (mon.0) 3734 : cluster [DBG] osdmap e800: 8 total, 8 up, 6 in 2024-02-19T21:17:44.697 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:43.641942+0000 osd.2 (osd.2) 316 : cluster [DBG] 5.d scrub starts 2024-02-19T21:17:44.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:43.643023+0000 osd.2 (osd.2) 317 : cluster [DBG] 5.d scrub ok 2024-02-19T21:17:44.698 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:44.421209+0000 mon.a (mon.0) 3735 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:17:44.698 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:44.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:44.799 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2 2024-02-19T21:17:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:43.342651+0000 mgr.y (mgr.14550) 663 : cluster [DBG] pgmap v1212: 214 pgs: 32 unknown, 1 active+remapped+backfill_toofull, 181 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-19T21:17:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:43.359004+0000 osd.1 (osd.1) 360 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:17:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:43.360160+0000 osd.1 (osd.1) 361 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:17:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:43.447991+0000 mon.a (mon.0) 3734 : cluster [DBG] osdmap e800: 8 total, 8 up, 6 in 2024-02-19T21:17:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:43.641942+0000 osd.2 (osd.2) 316 : cluster [DBG] 5.d scrub starts 2024-02-19T21:17:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:43.643023+0000 osd.2 (osd.2) 317 : cluster [DBG] 5.d scrub ok 2024-02-19T21:17:44.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:44.421209+0000 mon.a (mon.0) 3735 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:17:44.827 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:45.413 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd set noscrub 2024-02-19T21:17:45.576 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:17:45.578 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:45.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2' 2024-02-19T21:17:45.680 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:17:45.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:43.776336+0000 osd.6 (osd.6) 287 : cluster [DBG] 16.5 scrub starts 2024-02-19T21:17:45.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:43.777887+0000 osd.6 (osd.6) 288 : cluster [DBG] 16.5 scrub ok 2024-02-19T21:17:45.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:44.451565+0000 mon.a (mon.0) 3736 : cluster [DBG] osdmap e801: 8 total, 8 up, 6 in 2024-02-19T21:17:45.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:44.455961+0000 mon.a (mon.0) 3737 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:17:45.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:44.617792+0000 osd.2 (osd.2) 318 : cluster [DBG] 16.1 deep-scrub starts 2024-02-19T21:17:45.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:44.622185+0000 osd.2 (osd.2) 319 : cluster [DBG] 16.1 deep-scrub ok 2024-02-19T21:17:45.708 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:45.305189+0000 osd.4 (osd.4) 277 : cluster [DBG] 234.0 deep-scrub starts 2024-02-19T21:17:45.709 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:45.309155+0000 osd.4 (osd.4) 278 : cluster [DBG] 234.0 deep-scrub ok 2024-02-19T21:17:45.710 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:45.730 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:17:45.733 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:45.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:17:45.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:43.776336+0000 osd.6 (osd.6) 287 : cluster [DBG] 16.5 scrub starts 2024-02-19T21:17:45.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:43.777887+0000 osd.6 (osd.6) 288 : cluster [DBG] 16.5 scrub ok 2024-02-19T21:17:45.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:44.451565+0000 mon.a (mon.0) 3736 : cluster [DBG] osdmap e801: 8 total, 8 up, 6 in 2024-02-19T21:17:45.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:44.455961+0000 mon.a (mon.0) 3737 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:17:45.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:44.617792+0000 osd.2 (osd.2) 318 : cluster [DBG] 16.1 deep-scrub starts 2024-02-19T21:17:45.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:44.622185+0000 osd.2 (osd.2) 319 : cluster [DBG] 16.1 deep-scrub ok 2024-02-19T21:17:45.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:45.305189+0000 osd.4 (osd.4) 277 : cluster [DBG] 234.0 deep-scrub starts 2024-02-19T21:17:45.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:45.309155+0000 osd.4 (osd.4) 278 : cluster [DBG] 234.0 deep-scrub ok 2024-02-19T21:17:45.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:43.776336+0000 osd.6 (osd.6) 287 : cluster [DBG] 16.5 scrub starts 2024-02-19T21:17:45.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:43.777887+0000 osd.6 (osd.6) 288 : cluster [DBG] 16.5 scrub ok 2024-02-19T21:17:45.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:44.451565+0000 mon.a (mon.0) 3736 : cluster [DBG] osdmap e801: 8 total, 8 up, 6 in 2024-02-19T21:17:45.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:44.455961+0000 mon.a (mon.0) 3737 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-02-19T21:17:45.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:44.617792+0000 osd.2 (osd.2) 318 : cluster [DBG] 16.1 deep-scrub starts 2024-02-19T21:17:45.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:44.622185+0000 osd.2 (osd.2) 319 : cluster [DBG] 16.1 deep-scrub ok 2024-02-19T21:17:45.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:45.305189+0000 osd.4 (osd.4) 277 : cluster [DBG] 234.0 deep-scrub starts 2024-02-19T21:17:45.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:45.309155+0000 osd.4 (osd.4) 278 : cluster [DBG] 234.0 deep-scrub ok 2024-02-19T21:17:45.834 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7 2024-02-19T21:17:45.863 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:45.883 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:17:45.885 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:45.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7' 2024-02-19T21:17:45.986 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:46.013 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:46.031 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:46.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:46.114 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-02-19T21:17:46.114 INFO:teuthology.orchestra.run.smithi163.stdout: "size": 20, 2024-02-19T21:17:46.114 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 600, 2024-02-19T21:17:46.114 INFO:teuthology.orchestra.run.smithi163.stdout: "ops": [ 2024-02-19T21:17:46.114 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.114 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_repop(client.25753.0:4348 191.8 e314/308 191:10fb1d02:::199:head v 314'8, mlcod=313'6)", 2024-02-19T21:17:46.114 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:08:14.258279+0000", 2024-02-19T21:17:46.114 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 571.85303623200002, 2024-02-19T21:17:46.114 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039096335000000003, 2024-02-19T21:17:46.114 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.114 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:17:46.114 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.114 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.114 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258279+0000", 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258279+0000", 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258277+0000", 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1771e-05 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.115 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258289+0000", 2024-02-19T21:17:46.116 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4579999999999993e-06 2024-02-19T21:17:46.116 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.116 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.116 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.116 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258297+0000", 2024-02-19T21:17:46.116 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.113e-06 2024-02-19T21:17:46.116 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.116 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.116 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.116 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258303+0000", 2024-02-19T21:17:46.116 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.4646e-05 2024-02-19T21:17:46.116 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.116 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.116 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.116 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258337+0000", 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.1143999999999999e-05 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.258358+0000", 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.038976552999999997 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.297335+0000", 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0234000000000001e-05 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:08:14.297375+0000", 2024-02-19T21:17:46.117 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.039096335000000003 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.34352.0:1422 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94681540387968 gen 4] snapc 0=[] ondisk+write+known_if_redirected e433)", 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:10:40.311647+0000", 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 425.79966765900002, 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0091623380000001, 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.34352", 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/2750222700", 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1422 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.118 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311647+0000", 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311647+0000", 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999971 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311645+0000", 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.4200000000000007e-06 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.119 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.120 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311653+0000", 2024-02-19T21:17:46.120 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1905e-05 2024-02-19T21:17:46.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.120 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.120 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.120 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311665+0000", 2024-02-19T21:17:46.120 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1360000000000002e-06 2024-02-19T21:17:46.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.120 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.120 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.120 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311671+0000", 2024-02-19T21:17:46.120 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.8489999999999998e-05 2024-02-19T21:17:46.120 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.120 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.120 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311720+0000", 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9099999999999994e-06 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:40.311728+0000", 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.98640665500000002 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:41.298135+0000", 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00030824800000000001 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for degraded object", 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:41.298443+0000", 2024-02-19T21:17:46.121 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.016746660999999999 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:41.315190+0000", 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7898000000000001e-05 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:41.315238+0000", 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.005571767 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:10:41.320810+0000", 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0091623380000001 2024-02-19T21:17:46.122 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.123 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.123 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.123 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.123 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.123 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:1683 246.4 246:21cb8c6f:test-rados-api-smithi163-83928-125::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e708)", 2024-02-19T21:17:46.123 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:57.417818+0000", 2024-02-19T21:17:46.123 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.69349719, 2024-02-19T21:17:46.123 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0040492219999999999, 2024-02-19T21:17:46.123 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.123 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:46.123 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:46.123 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:17:46.123 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:17:46.123 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1683 2024-02-19T21:17:46.123 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.124 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.124 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.124 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.124 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417818+0000", 2024-02-19T21:17:46.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.124 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.124 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.124 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417818+0000", 2024-02-19T21:17:46.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:17:46.124 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.124 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.124 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.124 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417817+0000", 2024-02-19T21:17:46.124 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4900000000000002e-06 2024-02-19T21:17:46.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417821+0000", 2024-02-19T21:17:46.125 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5449999999999998e-06 2024-02-19T21:17:46.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417829+0000", 2024-02-19T21:17:46.125 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9929999999999997e-06 2024-02-19T21:17:46.125 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.125 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.125 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.125 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417833+0000", 2024-02-19T21:17:46.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.0771000000000001e-05 2024-02-19T21:17:46.126 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.126 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.126 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.126 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417864+0000", 2024-02-19T21:17:46.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.6823999999999997e-05 2024-02-19T21:17:46.126 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.126 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.126 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.126 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417920+0000", 2024-02-19T21:17:46.126 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0037826800000000001 2024-02-19T21:17:46.126 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.126 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.126 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.126 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.421703+0000", 2024-02-19T21:17:46.127 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.9967000000000001e-05 2024-02-19T21:17:46.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.127 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.127 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.127 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.421793+0000", 2024-02-19T21:17:46.127 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3961000000000006e-05 2024-02-19T21:17:46.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.127 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.127 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.127 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.421867+0000", 2024-02-19T21:17:46.127 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0040492219999999999 2024-02-19T21:17:46.127 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.127 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.127 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.127 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.128 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.128 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:1684 246.4 246:21cb8c6f:test-rados-api-smithi163-83928-125::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e708)", 2024-02-19T21:17:46.128 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:57.417840+0000", 2024-02-19T21:17:46.128 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.693475385, 2024-02-19T21:17:46.128 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0042064559999999999, 2024-02-19T21:17:46.128 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.128 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:46.128 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:46.128 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:17:46.128 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:17:46.128 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1684 2024-02-19T21:17:46.128 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.128 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.128 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417840+0000", 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417840+0000", 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417839+0000", 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9780000000000001e-06 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.129 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417842+0000", 2024-02-19T21:17:46.130 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.968e-06 2024-02-19T21:17:46.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.130 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417846+0000", 2024-02-19T21:17:46.130 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.5050000000000002e-06 2024-02-19T21:17:46.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.130 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417848+0000", 2024-02-19T21:17:46.130 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000175482 2024-02-19T21:17:46.130 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.130 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.130 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.418024+0000", 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2610000000000001e-05 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.418057+0000", 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.003847049 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.421904+0000", 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5326000000000004e-05 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.421979+0000", 2024-02-19T21:17:46.131 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7214999999999999e-05 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.422046+0000", 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0042064559999999999 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(osd.3.25:132 234.4s0 234:21cb8c6f:test-rados-api-smithi163-83928-125::foo:head [copy-from ver 2] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected e708)", 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:57.417980+0000", 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.693334757, 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033362769999999999, 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.132 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417980+0000", 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417980+0000", 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417980+0000", 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.303e-06 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.133 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.134 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.134 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417983+0000", 2024-02-19T21:17:46.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.181e-06 2024-02-19T21:17:46.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.134 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.134 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417988+0000", 2024-02-19T21:17:46.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.999e-06 2024-02-19T21:17:46.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.134 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.134 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.417991+0000", 2024-02-19T21:17:46.134 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6661000000000001e-05 2024-02-19T21:17:46.134 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.134 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.418018+0000", 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.0245000000000003e-05 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.418088+0000", 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024585409999999999 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.420547+0000", 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00036168399999999997 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.135 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:17:46.136 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.420908+0000", 2024-02-19T21:17:46.136 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000363641 2024-02-19T21:17:46.136 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.136 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.136 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:17:46.136 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.421272+0000", 2024-02-19T21:17:46.136 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.4594e-05 2024-02-19T21:17:46.136 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.136 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.136 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.136 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.421317+0000", 2024-02-19T21:17:46.136 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0033362769999999999 2024-02-19T21:17:46.136 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.136 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.136 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(osd.3.25:133 246.4 246:21cb8c6f:test-rados-api-smithi163-83928-125::foo:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+flush+map_snap_clone+known_if_redirected e708)", 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:57.418157+0000", 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.69315780300001, 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018858830000000001, 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.418157+0000", 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.137 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.418157+0000", 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.418156+0000", 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.7570000000000004e-06 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.418161+0000", 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.024e-06 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.418168+0000", 2024-02-19T21:17:46.138 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.7850000000000002e-06 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.418172+0000", 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2518e-05 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.418194+0000", 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3650999999999998e-05 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.418248+0000", 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0017950920000000001 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.139 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.140 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.140 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.420043+0000", 2024-02-19T21:17:46.140 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0018858830000000001 2024-02-19T21:17:46.140 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.140 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.140 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.140 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.140 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.140 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:1686 246.4 246:21cb8c6f:test-rados-api-smithi163-83928-125::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e708)", 2024-02-19T21:17:46.140 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:57.423437+0000", 2024-02-19T21:17:46.140 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.687878246, 2024-02-19T21:17:46.140 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021329069999999999, 2024-02-19T21:17:46.140 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.140 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:17:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:17:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1686 2024-02-19T21:17:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423437+0000", 2024-02-19T21:17:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423437+0000", 2024-02-19T21:17:46.141 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423435+0000", 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.2650000000000004e-06 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423441+0000", 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.679e-06 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423450+0000", 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.0469999999999994e-06 2024-02-19T21:17:46.142 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423458+0000", 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.9884e-05 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423498+0000", 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3732000000000006e-05 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423562+0000", 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001902614 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.143 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.144 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.144 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.425465+0000", 2024-02-19T21:17:46.144 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3127999999999997e-05 2024-02-19T21:17:46.144 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.144 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.144 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.144 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.425518+0000", 2024-02-19T21:17:46.144 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1876999999999998e-05 2024-02-19T21:17:46.144 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.144 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.144 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.144 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.425570+0000", 2024-02-19T21:17:46.144 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0021329069999999999 2024-02-19T21:17:46.144 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:1687 246.4 246:21cb8c6f:test-rados-api-smithi163-83928-125::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected e708)", 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:57.423471+0000", 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.68784422500001, 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0022692839999999999, 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1687 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.145 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423471+0000", 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423471+0000", 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423470+0000", 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.1640000000000003e-06 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.146 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423474+0000", 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1015e-05 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423485+0000", 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6280000000000004e-06 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423490+0000", 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00021766799999999999 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423707+0000", 2024-02-19T21:17:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.8666999999999999e-05 2024-02-19T21:17:46.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.148 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.148 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", 2024-02-19T21:17:46.148 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.423766+0000", 2024-02-19T21:17:46.148 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001829551 2024-02-19T21:17:46.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.148 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.148 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.148 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.425596+0000", 2024-02-19T21:17:46.148 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.1464000000000005e-05 2024-02-19T21:17:46.148 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.148 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.148 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.148 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.425677+0000", 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.3001000000000002e-05 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.425740+0000", 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0022692839999999999 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:1690 246.4 246:21cb8c6f:test-rados-api-smithi163-83928-125::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected e708)", 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:57.426798+0000", 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.68451712300001, 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0036363509999999999, 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.149 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1690 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.426798+0000", 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.426798+0000", 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:17:46.150 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.426797+0000", 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.23e-06 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.426800+0000", 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5719999999999996e-06 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.426805+0000", 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0430000000000001e-06 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.151 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.426809+0000", 2024-02-19T21:17:46.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000174666 2024-02-19T21:17:46.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.426984+0000", 2024-02-19T21:17:46.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1588000000000001e-05 2024-02-19T21:17:46.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.152 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.152 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.427035+0000", 2024-02-19T21:17:46.152 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0032300979999999998 2024-02-19T21:17:46.152 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.152 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.430265+0000", 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5210000000000005e-05 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.430361+0000", 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3737000000000002e-05 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.430434+0000", 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0036363509999999999 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.153 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(osd.3.25:136 234.4s0 234:21cb8c6f:test-rados-api-smithi163-83928-125::foo:head [copy-from ver 6] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected e708)", 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:57.426937+0000", 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.684378129, 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0025738039999999998, 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.426937+0000", 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.154 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.426937+0000", 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.426936+0000", 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3239999999999999e-06 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.426940+0000", 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.0759999999999996e-06 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.426944+0000", 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.7669999999999999e-06 2024-02-19T21:17:46.155 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.156 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.156 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.156 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.426946+0000", 2024-02-19T21:17:46.156 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.1402999999999997e-05 2024-02-19T21:17:46.156 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.156 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.156 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.156 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.426978+0000", 2024-02-19T21:17:46.156 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3194e-05 2024-02-19T21:17:46.156 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.156 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.156 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.156 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.427021+0000", 2024-02-19T21:17:46.156 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00090560599999999999 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.427927+0000", 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00035648199999999998 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.428283+0000", 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0011615359999999999 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.429445+0000", 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.6012000000000004e-05 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.157 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.429511+0000", 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0025738039999999998 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(osd.3.25:138 234.4s0 234:21cb8c6f:test-rados-api-smithi163-83928-125::foo:head [copy-from ver 6] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected e708)", 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:57.427132+0000", 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.684183189, 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028012779999999999, 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.158 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.427132+0000", 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.427132+0000", 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.427131+0000", 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.284e-06 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.427135+0000", 2024-02-19T21:17:46.159 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7289999999999997e-06 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.427141+0000", 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5290000000000002e-06 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.427146+0000", 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.8661000000000001e-05 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.427184+0000", 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.6993999999999999e-05 2024-02-19T21:17:46.160 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for blocked object", 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.427221+0000", 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00092825299999999998 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.428150+0000", 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.1777000000000002e-05 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.428201+0000", 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00082193499999999998 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.161 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:17:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.429023+0000", 2024-02-19T21:17:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00049914200000000005 2024-02-19T21:17:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:17:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.429522+0000", 2024-02-19T21:17:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00034484899999999999 2024-02-19T21:17:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:17:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.429867+0000", 2024-02-19T21:17:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5920000000000006e-05 2024-02-19T21:17:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.429933+0000", 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0028012779999999999 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:1692 246.4 246:21cb8c6f:test-rados-api-smithi163-83928-125::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected e708)", 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:57.431647+0000", 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.67966785500001, 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024842459999999999, 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:17:46.163 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1692 2024-02-19T21:17:46.164 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.164 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.164 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.164 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.164 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431647+0000", 2024-02-19T21:17:46.164 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.164 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.164 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.164 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.164 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431647+0000", 2024-02-19T21:17:46.164 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:17:46.164 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.164 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.164 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431646+0000", 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.0050000000000004e-06 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431651+0000", 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0850000000000003e-06 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431660+0000", 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5300000000000004e-06 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.165 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431666+0000", 2024-02-19T21:17:46.166 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9658999999999998e-05 2024-02-19T21:17:46.166 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.166 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.166 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.166 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431695+0000", 2024-02-19T21:17:46.166 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.1997999999999998e-05 2024-02-19T21:17:46.166 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.166 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.166 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.166 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431757+0000", 2024-02-19T21:17:46.166 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0022099070000000001 2024-02-19T21:17:46.166 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.166 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.166 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.433967+0000", 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.0362e-05 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.434057+0000", 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.3973999999999993e-05 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.434131+0000", 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0024842459999999999 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.167 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:1693 246.4 246:21cb8c6f:test-rados-api-smithi163-83928-125::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected e708)", 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:57.431675+0000", 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.679640338, 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026776539999999998, 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 1693 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.168 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431675+0000", 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431675+0000", 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.999999 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431674+0000", 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.3890000000000001e-06 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431677+0000", 2024-02-19T21:17:46.169 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.7599999999999999e-06 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431683+0000", 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.3279999999999999e-06 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431687+0000", 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00016395900000000001 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431851+0000", 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.1425e-05 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.170 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.171 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.171 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431943+0000", 2024-02-19T21:17:46.171 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0022210329999999999 2024-02-19T21:17:46.171 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.171 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.171 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.171 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.434164+0000", 2024-02-19T21:17:46.171 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.000113427 2024-02-19T21:17:46.171 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.171 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.171 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.171 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.434277+0000", 2024-02-19T21:17:46.171 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5242999999999998e-05 2024-02-19T21:17:46.171 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.171 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.172 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.172 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.434352+0000", 2024-02-19T21:17:46.172 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0026776539999999998 2024-02-19T21:17:46.172 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.172 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.172 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.172 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.172 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.172 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(osd.3.25:140 234.4s0 234:21cb8c6f:test-rados-api-smithi163-83928-125::foo:head [copy-from ver 8] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected e708)", 2024-02-19T21:17:46.172 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:15:57.431828+0000", 2024-02-19T21:17:46.172 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 108.679486912, 2024-02-19T21:17:46.172 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001847561, 2024-02-19T21:17:46.172 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.172 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:17:46.172 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431828+0000", 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431828+0000", 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431827+0000", 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.7089999999999996e-06 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.173 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431833+0000", 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8479999999999996e-06 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431843+0000", 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.6639999999999997e-06 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431848+0000", 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.9621999999999999e-05 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431877+0000", 2024-02-19T21:17:46.174 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 8.3035999999999996e-05 2024-02-19T21:17:46.175 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.175 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.175 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.175 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.431960+0000", 2024-02-19T21:17:46.175 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00066131499999999999 2024-02-19T21:17:46.175 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.175 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.175 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:17:46.175 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.432622+0000", 2024-02-19T21:17:46.175 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00032360400000000001 2024-02-19T21:17:46.175 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.175 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.175 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:17:46.175 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.432945+0000", 2024-02-19T21:17:46.175 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00066175299999999995 2024-02-19T21:17:46.176 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.176 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.176 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:17:46.176 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.433607+0000", 2024-02-19T21:17:46.176 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8620000000000004e-05 2024-02-19T21:17:46.176 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.176 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.176 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.176 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:15:57.433676+0000", 2024-02-19T21:17:46.176 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.001847561 2024-02-19T21:17:46.176 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.176 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.176 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.177 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.177 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.177 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(234.0s2 742/619 ECSubWrite(tid=684, reqid=client.23517.0:2126, at_version=0'0, trim_to=0'0, roll_forward_to=742'10))", 2024-02-19T21:17:46.177 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:38.622049+0000", 2024-02-19T21:17:46.177 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 67.489266052999994, 2024-02-19T21:17:46.177 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045502609999999999, 2024-02-19T21:17:46.177 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.177 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:17:46.177 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.177 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.177 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.177 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622049+0000", 2024-02-19T21:17:46.177 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.177 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622049+0000", 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622047+0000", 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.2001000000000001e-05 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622059+0000", 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.5718000000000001e-05 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.178 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622075+0000", 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.1420000000000004e-06 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622082+0000", 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00010511299999999999 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622187+0000", 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.4919999999999999e-06 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:17:46.179 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622197+0000", 2024-02-19T21:17:46.180 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0043826680000000002 2024-02-19T21:17:46.180 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.180 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.180 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:17:46.180 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.626580+0000", 2024-02-19T21:17:46.180 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9680000000000001e-05 2024-02-19T21:17:46.180 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.180 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.180 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.180 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.626599+0000", 2024-02-19T21:17:46.180 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045502609999999999 2024-02-19T21:17:46.180 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.180 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.180 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.180 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "MOSDECSubOpWrite(234.0s2 742/619 ECSubWrite(tid=685, reqid=client.23517.0:2127, at_version=742'11, trim_to=0'0, roll_forward_to=742'10))", 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:16:38.622097+0000", 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 67.489218320000006, 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045125870000000002, 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "reached pg", 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622097+0000", 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622097+0000", 2024-02-19T21:17:46.181 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999995 2024-02-19T21:17:46.182 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.182 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.182 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.182 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622096+0000", 2024-02-19T21:17:46.182 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.0009000000000001e-05 2024-02-19T21:17:46.182 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.182 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.182 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.182 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622116+0000", 2024-02-19T21:17:46.182 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9569999999999996e-06 2024-02-19T21:17:46.182 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.182 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.182 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.182 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622123+0000", 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.3739999999999997e-06 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622128+0000", 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00019169200000000001 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622320+0000", 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 7.5859999999999999e-06 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_started", 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.622328+0000", 2024-02-19T21:17:46.183 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0042747490000000004 2024-02-19T21:17:46.184 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.184 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.184 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_committed", 2024-02-19T21:17:46.184 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.626602+0000", 2024-02-19T21:17:46.184 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.8569999999999997e-06 2024-02-19T21:17:46.184 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.184 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.184 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.184 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:16:38.626609+0000", 2024-02-19T21:17:46.184 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0045125870000000002 2024-02-19T21:17:46.184 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.184 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.184 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.184 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.184 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:3122 255.11 255:8db81411:test-rados-api-smithi163-83928-143::foo:head [writefull 0~10] snapc 0=[] ondisk+write+known_if_redirected e799)", 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:17:42.447221+0000", 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.6640939349999999, 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.066983210000000001, 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3122 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.447221+0000", 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.185 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.447221+0000", 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999976 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.447219+0000", 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0678e-05 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.447229+0000", 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.4612e-05 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.186 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.447244+0000", 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.3903000000000001e-05 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.447258+0000", 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0054029840000000004 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.452661+0000", 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.0795000000000001e-05 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.187 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for peered", 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.452672+0000", 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.057002223999999997 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.509674+0000", 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00013616299999999999 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.509810+0000", 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00015145299999999999 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,6", 2024-02-19T21:17:46.188 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.509961+0000", 2024-02-19T21:17:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0020294340000000001 2024-02-19T21:17:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:17:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.511991+0000", 2024-02-19T21:17:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00113521 2024-02-19T21:17:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:17:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.513126+0000", 2024-02-19T21:17:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0010064290000000001 2024-02-19T21:17:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:17:46.189 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.514133+0000", 2024-02-19T21:17:46.190 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.5651e-05 2024-02-19T21:17:46.190 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.190 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.190 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:17:46.190 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.514178+0000", 2024-02-19T21:17:46.190 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6094000000000001e-05 2024-02-19T21:17:46.190 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.190 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.190 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.190 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.514204+0000", 2024-02-19T21:17:46.190 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.066983210000000001 2024-02-19T21:17:46.190 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.190 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.190 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.190 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.191 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.191 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:3127 255.11 255:8c26628f:test-rados-api-smithi163-83928-143::foo-chunk:head [set-chunk] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected e799)", 2024-02-19T21:17:46.191 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:17:42.521883+0000", 2024-02-19T21:17:46.191 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.5894319889999999, 2024-02-19T21:17:46.191 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031335769999999998, 2024-02-19T21:17:46.191 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.191 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-19T21:17:46.191 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:46.191 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:17:46.191 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:17:46.191 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3127 2024-02-19T21:17:46.191 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.191 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.191 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.191 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.192 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.521883+0000", 2024-02-19T21:17:46.192 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.192 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.192 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.192 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.192 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.521883+0000", 2024-02-19T21:17:46.192 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:17:46.192 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.192 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.192 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.192 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.521881+0000", 2024-02-19T21:17:46.192 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.9480000000000002e-06 2024-02-19T21:17:46.192 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.192 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.192 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.521888+0000", 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.8260000000000005e-06 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.521898+0000", 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.527e-06 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.521902+0000", 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6149999999999999e-05 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.193 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.521928+0000", 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.2048e-05 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.521991+0000", 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002094378 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "waiting for subops from 0,6", 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.524085+0000", 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00083903600000000004 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.524924+0000", 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.9700000000000001e-05 2024-02-19T21:17:46.194 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "sub_op_commit_rec", 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.524944+0000", 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.1600000000000001e-05 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "op_commit", 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.524955+0000", 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 3.2500999999999999e-05 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "commit_sent", 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.524988+0000", 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.8901999999999999e-05 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.195 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.196 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.196 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.525017+0000", 2024-02-19T21:17:46.196 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0031335769999999998 2024-02-19T21:17:46.196 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.196 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.196 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.196 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.196 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.196 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:3128 255.11 255:8db81411:test-rados-api-smithi163-83928-143::foo:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected e799)", 2024-02-19T21:17:46.196 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:17:42.525244+0000", 2024-02-19T21:17:46.196 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.586070742, 2024-02-19T21:17:46.196 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002767252, 2024-02-19T21:17:46.196 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.196 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:46.196 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3128 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.525244+0000", 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.525244+0000", 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999981 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.197 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.525242+0000", 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.2521999999999999e-05 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.525265+0000", 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 1.184e-05 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.525277+0000", 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5729999999999998e-06 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.525282+0000", 2024-02-19T21:17:46.198 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6075e-05 2024-02-19T21:17:46.199 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.199 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.199 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.199 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.525308+0000", 2024-02-19T21:17:46.199 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.5147999999999994e-05 2024-02-19T21:17:46.199 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.199 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.199 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.199 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.525373+0000", 2024-02-19T21:17:46.199 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.00263818 2024-02-19T21:17:46.199 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.199 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.199 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.199 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.528012+0000", 2024-02-19T21:17:46.199 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002767252 2024-02-19T21:17:46.200 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.200 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.200 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.200 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.200 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.200 INFO:teuthology.orchestra.run.smithi163.stdout: "description": "osd_op(client.23517.0:3130 255.11 255:8c26628f:test-rados-api-smithi163-83928-143::foo-chunk:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected e799)", 2024-02-19T21:17:46.200 INFO:teuthology.orchestra.run.smithi163.stdout: "initiated_at": "2024-02-19T21:17:42.529509+0000", 2024-02-19T21:17:46.200 INFO:teuthology.orchestra.run.smithi163.stdout: "age": 3.581806474, 2024-02-19T21:17:46.200 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0025215709999999998, 2024-02-19T21:17:46.200 INFO:teuthology.orchestra.run.smithi163.stdout: "type_data": { 2024-02-19T21:17:46.200 INFO:teuthology.orchestra.run.smithi163.stdout: "flag_point": "started", 2024-02-19T21:17:46.200 INFO:teuthology.orchestra.run.smithi163.stdout: "client_info": { 2024-02-19T21:17:46.200 INFO:teuthology.orchestra.run.smithi163.stdout: "client": "client.23517", 2024-02-19T21:17:46.200 INFO:teuthology.orchestra.run.smithi163.stdout: "client_addr": "v1:172.21.15.163:0/704748104", 2024-02-19T21:17:46.201 INFO:teuthology.orchestra.run.smithi163.stdout: "tid": 3130 2024-02-19T21:17:46.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.201 INFO:teuthology.orchestra.run.smithi163.stdout: "events": [ 2024-02-19T21:17:46.201 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.201 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "initiated", 2024-02-19T21:17:46.201 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.529509+0000", 2024-02-19T21:17:46.201 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0 2024-02-19T21:17:46.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.201 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.201 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "throttled", 2024-02-19T21:17:46.201 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.529509+0000", 2024-02-19T21:17:46.201 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4294967295.9999986 2024-02-19T21:17:46.201 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.201 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.201 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "header_read", 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.529507+0000", 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 5.5489999999999997e-06 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "all_read", 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.529513+0000", 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 9.5990000000000008e-06 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "dispatched", 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.529522+0000", 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 4.809e-06 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "queued_for_pg", 2024-02-19T21:17:46.202 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.529527+0000", 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 2.6954e-05 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "reached_pg", 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.529554+0000", 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 6.4445999999999999e-05 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "started", 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.529618+0000", 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.002411669 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: { 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: "event": "done", 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: "time": "2024-02-19T21:17:42.532030+0000", 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: "duration": 0.0025215709999999998 2024-02-19T21:17:46.203 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.204 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.204 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.204 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-02-19T21:17:46.204 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-02-19T21:17:46.204 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-02-19T21:17:46.204 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:46.230 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:46.249 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:46.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:46.352 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5 2024-02-19T21:17:46.378 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:46.396 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:17:46.398 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:46.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5' 2024-02-19T21:17:46.456 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:45.343574+0000 mgr.y (mgr.14550) 664 : cluster [DBG] pgmap v1215: 214 pgs: 1 active+remapped+backfill_toofull, 32 creating+peering, 181 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-19T21:17:46.456 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:45.457799+0000 mon.a (mon.0) 3738 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:17:46.456 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:45.457932+0000 mon.a (mon.0) 3739 : cluster [DBG] osdmap e802: 8 total, 8 up, 6 in 2024-02-19T21:17:46.456 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:45.463914+0000 mon.a (mon.0) 3740 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-101","var": "dedup_tier","val": "test-rados-api-smithi163-83928-144"}]: dispatch 2024-02-19T21:17:46.456 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:45.500349+0000 mon.a (mon.0) 3741 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:46.456 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:45.500652+0000 mon.a (mon.0) 3742 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-144"}]: dispatch 2024-02-19T21:17:46.456 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:45.747471+0000 osd.6 (osd.6) 289 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:17:46.456 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:45.748401+0000 osd.6 (osd.6) 290 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:17:46.457 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:45.343574+0000 mgr.y (mgr.14550) 664 : cluster [DBG] pgmap v1215: 214 pgs: 1 active+remapped+backfill_toofull, 32 creating+peering, 181 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-19T21:17:46.457 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:45.457799+0000 mon.a (mon.0) 3738 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:17:46.457 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:45.457932+0000 mon.a (mon.0) 3739 : cluster [DBG] osdmap e802: 8 total, 8 up, 6 in 2024-02-19T21:17:46.457 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:45.463914+0000 mon.a (mon.0) 3740 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-101","var": "dedup_tier","val": "test-rados-api-smithi163-83928-144"}]: dispatch 2024-02-19T21:17:46.457 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:45.500349+0000 mon.a (mon.0) 3741 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:46.457 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:45.500652+0000 mon.a (mon.0) 3742 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-144"}]: dispatch 2024-02-19T21:17:46.457 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:45.747471+0000 osd.6 (osd.6) 289 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:17:46.457 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:45.748401+0000 osd.6 (osd.6) 290 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:17:46.500 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:46.525 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:46.544 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:46.546 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:46.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:46.647 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:17:46.676 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:46.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:45.343574+0000 mgr.y (mgr.14550) 664 : cluster [DBG] pgmap v1215: 214 pgs: 1 active+remapped+backfill_toofull, 32 creating+peering, 181 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-19T21:17:46.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:45.457799+0000 mon.a (mon.0) 3738 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi163-83928-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-02-19T21:17:46.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:45.457932+0000 mon.a (mon.0) 3739 : cluster [DBG] osdmap e802: 8 total, 8 up, 6 in 2024-02-19T21:17:46.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:45.463914+0000 mon.a (mon.0) 3740 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi163-83928-101","var": "dedup_tier","val": "test-rados-api-smithi163-83928-144"}]: dispatch 2024-02-19T21:17:46.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:45.500349+0000 mon.a (mon.0) 3741 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:46.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:45.500652+0000 mon.a (mon.0) 3742 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi163-83928-101", "tierpool": "test-rados-api-smithi163-83928-144"}]: dispatch 2024-02-19T21:17:46.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:45.747471+0000 osd.6 (osd.6) 289 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:17:46.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:45.748401+0000 osd.6 (osd.6) 290 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:17:46.875 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:17:46.878 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:46.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:17:46.979 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:47.006 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:47.024 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:47.027 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:47.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:47.128 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6 2024-02-19T21:17:47.155 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:47.173 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:17:47.175 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:47.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6' 2024-02-19T21:17:47.276 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1 2024-02-19T21:17:47.303 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:47.322 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:17:47.328 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:47.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1' 2024-02-19T21:17:47.429 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3 2024-02-19T21:17:47.456 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:47.563 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:46.460759+0000 mon.a (mon.0) 3743 : cluster [DBG] osdmap e803: 8 total, 8 up, 6 in 2024-02-19T21:17:47.563 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:46.545939+0000 osd.2 (osd.2) 320 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:17:47.563 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:46.546822+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:17:47.563 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:47.146358+0000 osd.3 (osd.3) 335 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:17:47.563 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:47.148334+0000 osd.3 (osd.3) 336 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:17:47.563 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:17:47.564 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:46.460759+0000 mon.a (mon.0) 3743 : cluster [DBG] osdmap e803: 8 total, 8 up, 6 in 2024-02-19T21:17:47.564 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:46.545939+0000 osd.2 (osd.2) 320 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:17:47.564 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:46.546822+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:17:47.564 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:47.146358+0000 osd.3 (osd.3) 335 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:17:47.564 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:47.148334+0000 osd.3 (osd.3) 336 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:17:47.566 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:47.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3' 2024-02-19T21:17:47.667 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:47.694 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:47.712 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:47.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:47.774 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:46.460759+0000 mon.a (mon.0) 3743 : cluster [DBG] osdmap e803: 8 total, 8 up, 6 in 2024-02-19T21:17:47.774 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:46.545939+0000 osd.2 (osd.2) 320 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:17:47.774 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:46.546822+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:17:47.774 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:47.146358+0000 osd.3 (osd.3) 335 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:17:47.774 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:47.148334+0000 osd.3 (osd.3) 336 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:17:47.816 DEBUG:teuthology.orchestra.run.smithi163:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0 2024-02-19T21:17:47.845 INFO:teuthology.orchestra.run.smithi163.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:47.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.872+0000 7fc3a7de7700 1 -- 172.21.15.163:0/3625400493 >> v1:172.21.15.163:6790/0 conn(0x7fc3a01042b0 legacy=0x7fc3a0106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:47.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.872+0000 7fc3a7de7700 1 -- 172.21.15.163:0/3625400493 shutdown_connections 2024-02-19T21:17:47.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.872+0000 7fc3a7de7700 1 -- 172.21.15.163:0/3625400493 >> 172.21.15.163:0/3625400493 conn(0x7fc3a00f2470 msgr2=0x7fc3a00f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:47.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.872+0000 7fc3a7de7700 1 -- 172.21.15.163:0/3625400493 shutdown_connections 2024-02-19T21:17:47.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.872+0000 7fc3a7de7700 1 -- 172.21.15.163:0/3625400493 wait complete. 2024-02-19T21:17:47.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.872+0000 7fc3a7de7700 1 librados: starting msgr at 2024-02-19T21:17:47.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.872+0000 7fc3a7de7700 1 librados: starting objecter 2024-02-19T21:17:47.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.872+0000 7fc3a7de7700 1 Processor -- start 2024-02-19T21:17:47.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.872+0000 7fc3a7de7700 1 -- start start 2024-02-19T21:17:47.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.872+0000 7fc3a7de7700 1 librados: setting wanted keys 2024-02-19T21:17:47.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.872+0000 7fc3a7de7700 1 librados: calling monclient init 2024-02-19T21:17:47.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc3a7de7700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc3a0117400 con 0x7fc3a01042b0 2024-02-19T21:17:47.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc3a7de7700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc3a01185e0 con 0x7fc3a00f94d0 2024-02-19T21:17:47.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc3a7de7700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc3a01197e0 con 0x7fc3a00f8920 2024-02-19T21:17:47.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc3a5b83700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fc3a00f8920 0x7fc3a00ff700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:43082/0 (socket says 172.21.15.163:43082) 2024-02-19T21:17:47.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc3a6384700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fc3a01042b0 0x7fc3a0115ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:51822/0 (socket says 172.21.15.163:51822) 2024-02-19T21:17:47.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc3a5b83700 1 -- 172.21.15.163:0/1094383767 learned_addr learned my addr 172.21.15.163:0/1094383767 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:47.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 10 client.?.objecter ms_handle_connect 0x7fc3a00f8920 2024-02-19T21:17:47.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:47.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 10 client.?.objecter ms_handle_connect 0x7fc3a01042b0 2024-02-19T21:17:47.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:47.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 10 client.?.objecter ms_handle_connect 0x7fc3a00f94d0 2024-02-19T21:17:47.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:47.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534804089 0 0) 0x7fc3a0117400 con 0x7fc3a01042b0 2024-02-19T21:17:47.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc38c003030 con 0x7fc3a01042b0 2024-02-19T21:17:47.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223516141 0 0) 0x7fc3a01197e0 con 0x7fc3a00f8920 2024-02-19T21:17:47.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3a0117400 con 0x7fc3a00f8920 2024-02-19T21:17:47.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950770017 0 0) 0x7fc3a01185e0 con 0x7fc3a00f94d0 2024-02-19T21:17:47.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3a01197e0 con 0x7fc3a00f94d0 2024-02-19T21:17:47.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1184877818 0 0) 0x7fc38c003030 con 0x7fc3a01042b0 2024-02-19T21:17:47.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3a01185e0 con 0x7fc3a01042b0 2024-02-19T21:17:47.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc398002320 con 0x7fc3a01042b0 2024-02-19T21:17:47.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1790325103 0 0) 0x7fc3a0117400 con 0x7fc3a00f8920 2024-02-19T21:17:47.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc38c003030 con 0x7fc3a00f8920 2024-02-19T21:17:47.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc39c001940 con 0x7fc3a00f8920 2024-02-19T21:17:47.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1290196084 0 0) 0x7fc3a01185e0 con 0x7fc3a01042b0 2024-02-19T21:17:47.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 >> v1:172.21.15.163:6790/0 conn(0x7fc3a00f8920 legacy=0x7fc3a00ff700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:47.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 >> v1:172.21.15.193:6789/0 conn(0x7fc3a00f94d0 legacy=0x7fc3a00ffe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:47.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a011a9e0 con 0x7fc3a01042b0 2024-02-19T21:17:47.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc3a7de7700 1 -- 172.21.15.163:0/1094383767 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a01175f0 con 0x7fc3a01042b0 2024-02-19T21:17:47.885 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc3a7de7700 10 client.23922.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:47.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc3a7de7700 1 -- 172.21.15.163:0/1094383767 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a0117a60 con 0x7fc3a01042b0 2024-02-19T21:17:47.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.873+0000 7fc3a7de7700 1 librados: init done 2024-02-19T21:17:47.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.874+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fc398001d20 con 0x7fc3a01042b0 2024-02-19T21:17:47.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.874+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc398003ae0 con 0x7fc3a01042b0 2024-02-19T21:17:47.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.874+0000 7fc3a7de7700 1 -- 172.21.15.163:0/1094383767 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc384004f20 con 0x7fc3a01042b0 2024-02-19T21:17:47.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.875+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fc398003ce0 con 0x7fc3a01042b0 2024-02-19T21:17:47.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.875+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(804..804 src has 633..804) v4 ==== 7588+0+0 (unknown 4096639645 0 0) 0x7fc3980791d0 con 0x7fc3a01042b0 2024-02-19T21:17:47.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.875+0000 7fc396ffd700 10 client.23922.objecter ms_dispatch 0x7fc3a0001550 osd_map(804..804 src has 633..804) v4 2024-02-19T21:17:47.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.875+0000 7fc396ffd700 3 client.23922.objecter handle_osd_map got epochs [804,804] > 0 2024-02-19T21:17:47.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.875+0000 7fc396ffd700 3 client.23922.objecter handle_osd_map decoding full epoch 804 2024-02-19T21:17:47.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.875+0000 7fc396ffd700 20 client.23922.objecter dump_active .. 0 homeless 2024-02-19T21:17:47.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.875+0000 7fc396ffd700 10 client.23922.objecter ms_handle_connect 0x7fc38c063970 2024-02-19T21:17:47.886 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:47.879+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc39804c7a0 con 0x7fc3a01042b0 2024-02-19T21:17:48.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:48.045+0000 7fc3a7de7700 1 -- 172.21.15.163:0/1094383767 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"} v 0) v1 -- 0x7fc3840051d0 con 0x7fc3a01042b0 2024-02-19T21:17:48.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:17:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:17:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:17:48.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:48.472+0000 7fc396ffd700 1 -- 172.21.15.163:0/1094383767 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]=0 set pool 6 pgp_num to 32 v805) v1 ==== 143+0+0 (unknown 4147370310 0 0) 0x7fc39804ac10 con 0x7fc3a01042b0 2024-02-19T21:17:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:48.475+0000 7fc3a7de7700 10 librados: watch_flush enter 2024-02-19T21:17:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:48.475+0000 7fc3a7de7700 10 librados: watch_flush exit 2024-02-19T21:17:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:48.475+0000 7fc3a7de7700 20 client.23922.objecter shutdown clearing up homeless session... 2024-02-19T21:17:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:48.475+0000 7fc3a7de7700 10 client.23922.objecter successfully canceled tick 2024-02-19T21:17:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:48.475+0000 7fc3a7de7700 1 -- 172.21.15.163:0/1094383767 >> v1:172.21.15.163:6800/1552514681 conn(0x7fc38c063970 legacy=0x7fc38c065e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:48.475+0000 7fc3a7de7700 1 -- 172.21.15.163:0/1094383767 >> v1:172.21.15.163:6789/0 conn(0x7fc3a01042b0 legacy=0x7fc3a0115ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:48.475+0000 7fc3a7de7700 1 -- 172.21.15.163:0/1094383767 shutdown_connections 2024-02-19T21:17:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:48.475+0000 7fc3a7de7700 1 -- 172.21.15.163:0/1094383767 >> 172.21.15.163:0/1094383767 conn(0x7fc3a00f2470 msgr2=0x7fc3a0107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:48.475+0000 7fc3a7de7700 1 -- 172.21.15.163:0/1094383767 shutdown_connections 2024-02-19T21:17:48.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:48.475+0000 7fc3a7de7700 1 -- 172.21.15.163:0/1094383767 wait complete. 2024-02-19T21:17:48.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:48.477+0000 7fc3a7de7700 1 librados: shutdown 2024-02-19T21:17:48.479 INFO:teuthology.orchestra.run.smithi163.stderr:set pool 6 pgp_num to 32 2024-02-19T21:17:48.585 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-19T21:17:48.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:47.343966+0000 mgr.y (mgr.14550) 665 : cluster [DBG] pgmap v1218: 182 pgs: 1 active+remapped+backfill_toofull, 181 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-19T21:17:48.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:47.467725+0000 mon.a (mon.0) 3744 : cluster [DBG] osdmap e804: 8 total, 8 up, 6 in 2024-02-19T21:17:48.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:47.468615+0000 mon.a (mon.0) 3745 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:48.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:48.046784+0000 mon.a (mon.0) 3746 : audit [INF] from='client.? v1:172.21.15.163:0/1094383767' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-19T21:17:48.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:48.318667+0000 osd.4 (osd.4) 279 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:17:48.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:48.319489+0000 osd.4 (osd.4) 280 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:17:48.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:47.343966+0000 mgr.y (mgr.14550) 665 : cluster [DBG] pgmap v1218: 182 pgs: 1 active+remapped+backfill_toofull, 181 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-19T21:17:48.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:47.467725+0000 mon.a (mon.0) 3744 : cluster [DBG] osdmap e804: 8 total, 8 up, 6 in 2024-02-19T21:17:48.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:47.468615+0000 mon.a (mon.0) 3745 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:48.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:48.046784+0000 mon.a (mon.0) 3746 : audit [INF] from='client.? v1:172.21.15.163:0/1094383767' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-19T21:17:48.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:48.318667+0000 osd.4 (osd.4) 279 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:17:48.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:48.319489+0000 osd.4 (osd.4) 280 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:17:48.752 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:47.343966+0000 mgr.y (mgr.14550) 665 : cluster [DBG] pgmap v1218: 182 pgs: 1 active+remapped+backfill_toofull, 181 active+clean; 3.8 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-19T21:17:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:47.467725+0000 mon.a (mon.0) 3744 : cluster [DBG] osdmap e804: 8 total, 8 up, 6 in 2024-02-19T21:17:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:47.468615+0000 mon.a (mon.0) 3745 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:48.046784+0000 mon.a (mon.0) 3746 : audit [INF] from='client.? v1:172.21.15.163:0/1094383767' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2024-02-19T21:17:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:48.318667+0000 osd.4 (osd.4) 279 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:17:48.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:48.319489+0000 osd.4 (osd.4) 280 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:17:48.851 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:17:49.439 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8063 ms) 2024-02-19T21:17:49.439 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-02-19T21:17:49.439 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708377384,0 2024-02-19T21:17:49.440 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: first is 1708377384 2024-02-19T21:17:49.440 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708377384,0 2024-02-19T21:17:49.440 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708377384,0 2024-02-19T21:17:49.440 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708377384,1708377386,1708377387,0 2024-02-19T21:17:49.440 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708377384,1708377386,1708377387,0 2024-02-19T21:17:49.440 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708377384,1708377386,1708377387,0 2024-02-19T21:17:49.440 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708377384,1708377386,1708377387,1708377389,1708377390,0 2024-02-19T21:17:49.440 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708377384,1708377386,1708377387,1708377389,1708377390,0 2024-02-19T21:17:49.440 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708377384,1708377386,1708377387,1708377389,1708377390,0 2024-02-19T21:17:49.440 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708377384,1708377386,1708377387,1708377389,1708377390,1708377392,1708377393,0 2024-02-19T21:17:49.440 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708377384,1708377386,1708377387,1708377389,1708377390,1708377392,1708377393,0 2024-02-19T21:17:49.440 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708377384,1708377386,1708377387,1708377389,1708377390,1708377392,1708377393,0 2024-02-19T21:17:49.440 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: got ls 1708377387,1708377389,1708377390,1708377392,1708377393,1708377395,1708377396,0 2024-02-19T21:17:49.440 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: first now 1708377387, trimmed 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20090 ms) 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: foo0 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13900 ms) 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18115 ms) 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (23830 ms) 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5006 ms) 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (2989 ms) 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3012 ms) 2024-02-19T21:17:49.441 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-02-19T21:17:49.442 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3016 ms) 2024-02-19T21:17:49.442 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (232121 ms total) 2024-02-19T21:17:49.442 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: 2024-02-19T21:17:49.442 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-02-19T21:17:49.442 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [==========] 72 tests from 4 test suites ran. (796568 ms total) 2024-02-19T21:17:49.442 INFO:tasks.workunit.client.0.smithi163.stdout: api_tier_pp: [ PASSED ] 72 tests. 2024-02-19T21:17:49.442 INFO:tasks.workunit.client.0.smithi163.stderr:+ exit 0 2024-02-19T21:17:49.442 INFO:tasks.workunit.client.0.smithi163.stderr:+ cleanup 2024-02-19T21:17:49.442 INFO:tasks.workunit.client.0.smithi163.stderr:+ pkill -P 83830 2024-02-19T21:17:49.489 INFO:tasks.workunit.client.0.smithi163.stderr:+ true 2024-02-19T21:17:49.490 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-19T21:17:49.490 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-19T21:17:49.525 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-02-19T21:17:49.525 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-19T21:17:49.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:48.471320+0000 mon.a (mon.0) 3747 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:17:49.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:48.471371+0000 mon.a (mon.0) 3748 : audit [INF] from='client.? v1:172.21.15.163:0/1094383767' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]': finished 2024-02-19T21:17:49.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:48.471448+0000 mon.a (mon.0) 3749 : cluster [DBG] osdmap e805: 8 total, 8 up, 6 in 2024-02-19T21:17:49.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:48.473207+0000 mon.a (mon.0) 3750 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:49.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:49.346947+0000 osd.4 (osd.4) 281 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:17:49.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:49.349074+0000 osd.4 (osd.4) 282 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:17:49.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:49.432126+0000 mon.a (mon.0) 3751 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:17:49.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:49.432204+0000 mon.a (mon.0) 3752 : cluster [DBG] osdmap e806: 8 total, 8 up, 6 in 2024-02-19T21:17:49.595 INFO:teuthology.orchestra.run.smithi163.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:17:49.599 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:49.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi163 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0' 2024-02-19T21:17:49.701 DEBUG:teuthology.orchestra.run.smithi193:> sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4 2024-02-19T21:17:49.729 INFO:teuthology.orchestra.run.smithi193.stderr:Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. 2024-02-19T21:17:49.841 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:48.471320+0000 mon.a (mon.0) 3747 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:17:49.841 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:48.471371+0000 mon.a (mon.0) 3748 : audit [INF] from='client.? v1:172.21.15.163:0/1094383767' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]': finished 2024-02-19T21:17:49.841 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:48.471448+0000 mon.a (mon.0) 3749 : cluster [DBG] osdmap e805: 8 total, 8 up, 6 in 2024-02-19T21:17:49.841 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:48.473207+0000 mon.a (mon.0) 3750 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:49.841 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:49.346947+0000 osd.4 (osd.4) 281 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:17:49.841 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:49.349074+0000 osd.4 (osd.4) 282 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:17:49.841 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:49.432126+0000 mon.a (mon.0) 3751 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:17:49.841 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:49.432204+0000 mon.a (mon.0) 3752 : cluster [DBG] osdmap e806: 8 total, 8 up, 6 in 2024-02-19T21:17:49.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:48.471320+0000 mon.a (mon.0) 3747 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:17:49.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:48.471371+0000 mon.a (mon.0) 3748 : audit [INF] from='client.? v1:172.21.15.163:0/1094383767' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]': finished 2024-02-19T21:17:49.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:48.471448+0000 mon.a (mon.0) 3749 : cluster [DBG] osdmap e805: 8 total, 8 up, 6 in 2024-02-19T21:17:49.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:48.473207+0000 mon.a (mon.0) 3750 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-101"}]: dispatch 2024-02-19T21:17:49.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:49.346947+0000 osd.4 (osd.4) 281 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:17:49.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:49.349074+0000 osd.4 (osd.4) 282 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:17:49.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:49.432126+0000 mon.a (mon.0) 3751 : audit [INF] from='client.? v1:172.21.15.163:0/704748104' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi163-83928-101"}]': finished 2024-02-19T21:17:49.842 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:49.432204+0000 mon.a (mon.0) 3752 : cluster [DBG] osdmap e806: 8 total, 8 up, 6 in 2024-02-19T21:17:49.843 DEBUG:teuthology.parallel:result is None 2024-02-19T21:17:49.843 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-02-19T21:17:49.870 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-02-19T21:17:49.871 DEBUG:teuthology.orchestra.run.smithi163:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-02-19T21:17:49.968 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-02-19T21:17:49.968 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-02-19T21:17:49.979 INFO:tasks.thrashosds:joining thrashosds 2024-02-19T21:17:50.560 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-02-19T21:17:50.560 INFO:teuthology.orchestra.run.smithi193.stdout: "ops": [], 2024-02-19T21:17:50.560 INFO:teuthology.orchestra.run.smithi193.stdout: "num_ops": 0 2024-02-19T21:17:50.560 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-02-19T21:17:50.578 INFO:teuthology.orchestra.run.smithi193.stderr:Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:17:50.581 DEBUG:teuthology.orchestra.run:got remote process result: 125 2024-02-19T21:17:50.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi193 with status 125: 'sudo docker kill -s 1 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4' 2024-02-19T21:17:50.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:49.344436+0000 mgr.y (mgr.14550) 666 : cluster [DBG] pgmap v1221: 174 pgs: 1 active+remapped+backfill_toofull, 173 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:17:50.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:49.475293+0000 mon.a (mon.0) 3753 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:17:50.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:49.500304+0000 mon.b (mon.1) 754 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:17:50.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:50.165473+0000 osd.3 (osd.3) 337 : cluster [DBG] 5.17 deep-scrub starts 2024-02-19T21:17:50.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:50.166363+0000 osd.3 (osd.3) 338 : cluster [DBG] 5.17 deep-scrub ok 2024-02-19T21:17:50.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:50.176164+0000 mon.a (mon.0) 3754 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:50.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:49.344436+0000 mgr.y (mgr.14550) 666 : cluster [DBG] pgmap v1221: 174 pgs: 1 active+remapped+backfill_toofull, 173 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:17:50.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:49.475293+0000 mon.a (mon.0) 3753 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:17:50.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:49.500304+0000 mon.b (mon.1) 754 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:17:50.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:50.165473+0000 osd.3 (osd.3) 337 : cluster [DBG] 5.17 deep-scrub starts 2024-02-19T21:17:50.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:50.166363+0000 osd.3 (osd.3) 338 : cluster [DBG] 5.17 deep-scrub ok 2024-02-19T21:17:50.920 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:50.176164+0000 mon.a (mon.0) 3754 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:50.920 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:49.344436+0000 mgr.y (mgr.14550) 666 : cluster [DBG] pgmap v1221: 174 pgs: 1 active+remapped+backfill_toofull, 173 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:17:50.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:49.475293+0000 mon.a (mon.0) 3753 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-19T21:17:50.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:49.500304+0000 mon.b (mon.1) 754 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:17:50.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:50.165473+0000 osd.3 (osd.3) 337 : cluster [DBG] 5.17 deep-scrub starts 2024-02-19T21:17:50.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:50.166363+0000 osd.3 (osd.3) 338 : cluster [DBG] 5.17 deep-scrub ok 2024-02-19T21:17:50.921 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:50.176164+0000 mon.a (mon.0) 3754 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:51.175 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:17:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:17:51.193 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-19T21:17:51.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.548+0000 7f4756ce5700 1 -- 172.21.15.163:0/2069509809 >> v1:172.21.15.163:6790/0 conn(0x7f4750104250 legacy=0x7f47501066d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:51.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.548+0000 7f4756ce5700 1 -- 172.21.15.163:0/2069509809 shutdown_connections 2024-02-19T21:17:51.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.548+0000 7f4756ce5700 1 -- 172.21.15.163:0/2069509809 >> 172.21.15.163:0/2069509809 conn(0x7f47500f3530 msgr2=0x7f47500f5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:51.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.548+0000 7f4756ce5700 1 -- 172.21.15.163:0/2069509809 shutdown_connections 2024-02-19T21:17:51.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.549+0000 7f4756ce5700 1 -- 172.21.15.163:0/2069509809 wait complete. 2024-02-19T21:17:51.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.549+0000 7f4756ce5700 1 librados: starting msgr at 2024-02-19T21:17:51.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.549+0000 7f4756ce5700 1 librados: starting objecter 2024-02-19T21:17:51.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.549+0000 7f4756ce5700 1 Processor -- start 2024-02-19T21:17:51.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.549+0000 7f4756ce5700 1 -- start start 2024-02-19T21:17:51.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.549+0000 7f4756ce5700 1 librados: setting wanted keys 2024-02-19T21:17:51.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.549+0000 7f4756ce5700 1 librados: calling monclient init 2024-02-19T21:17:51.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.549+0000 7f4756ce5700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f47500fcec0 con 0x7f4750100710 2024-02-19T21:17:51.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.549+0000 7f4756ce5700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f47500fd090 con 0x7f4750104250 2024-02-19T21:17:51.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.549+0000 7f4756ce5700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f47500fd280 con 0x7f47500fd910 2024-02-19T21:17:51.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.549+0000 7f4754a81700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f47500fd910 0x7f47500fc720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:43098/0 (socket says 172.21.15.163:43098) 2024-02-19T21:17:51.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.549+0000 7f4754a81700 1 -- 172.21.15.163:0/986067486 learned_addr learned my addr 172.21.15.163:0/986067486 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:51.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.549+0000 7f474dffb700 10 client.?.objecter ms_handle_connect 0x7f4750100710 2024-02-19T21:17:51.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:51.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 10 client.?.objecter ms_handle_connect 0x7f47500fd910 2024-02-19T21:17:51.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:51.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 10 client.?.objecter ms_handle_connect 0x7f4750104250 2024-02-19T21:17:51.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:51.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988672209 0 0) 0x7f47500fcec0 con 0x7f4750100710 2024-02-19T21:17:51.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47400038e0 con 0x7f4750100710 2024-02-19T21:17:51.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138676017 0 0) 0x7f47500fd280 con 0x7f47500fd910 2024-02-19T21:17:51.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47500fcec0 con 0x7f47500fd910 2024-02-19T21:17:51.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520633381 0 0) 0x7f47500fd090 con 0x7f4750104250 2024-02-19T21:17:51.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47500fd280 con 0x7f4750104250 2024-02-19T21:17:51.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3916964440 0 0) 0x7f47400038e0 con 0x7f4750100710 2024-02-19T21:17:51.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47500fd090 con 0x7f4750100710 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f473c001880 con 0x7f4750100710 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 37827932 0 0) 0x7f47500fcec0 con 0x7f47500fd910 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47400038e0 con 0x7f47500fd910 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 313723496 0 0) 0x7f47500fd280 con 0x7f4750104250 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47500fcec0 con 0x7f4750104250 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f47440019f0 con 0x7f47500fd910 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4748002f40 con 0x7f4750104250 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2410216181 0 0) 0x7f47500fd090 con 0x7f4750100710 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 >> v1:172.21.15.163:6790/0 conn(0x7f47500fd910 legacy=0x7f47500fc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 >> v1:172.21.15.193:6789/0 conn(0x7f4750104250 legacy=0x7f47501183d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475011cb20 con 0x7f4750100710 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4750119b50 con 0x7f4750100710 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f473c001f40 con 0x7f4750100710 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f473c003230 con 0x7f4750100710 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f4756ce5700 10 client.23928.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:51.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f475011a040 con 0x7f4750100710 2024-02-19T21:17:51.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.550+0000 7f4756ce5700 1 librados: init done 2024-02-19T21:17:51.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.551+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd ls"} v 0) v1 -- 0x7f475005f5f0 con 0x7f4750100710 2024-02-19T21:17:51.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f473c002610 con 0x7f4750100710 2024-02-19T21:17:51.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(806..806 src has 633..806) v4 ==== 7187+0+0 (unknown 2421357025 0 0) 0x7f473c076200 con 0x7f4750100710 2024-02-19T21:17:51.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 osd_map(806..806 src has 633..806) v4 2024-02-19T21:17:51.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f474dffb700 3 client.23928.objecter handle_osd_map got epochs [806,806] > 0 2024-02-19T21:17:51.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f474dffb700 3 client.23928.objecter handle_osd_map decoding full epoch 806 2024-02-19T21:17:51.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f474dffb700 20 client.23928.objecter dump_active .. 0 homeless 2024-02-19T21:17:51.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v806) v1 ==== 54+0+16 (unknown 1093676982 0 2420000192) 0x7f473c078650 con 0x7f4750100710 2024-02-19T21:17:51.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f474dffb700 10 client.23928.objecter ms_handle_connect 0x7f4740063da0 2024-02-19T21:17:51.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f4756ce5700 10 client.23928.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:51.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 1 2024-02-19T21:17:51.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750047540 osd=0 3 2024-02-19T21:17:51.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750047540 osd=0 3 2024-02-19T21:17:51.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750047540 osd=0 3 2024-02-19T21:17:51.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 1 2024-02-19T21:17:51.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f4750047540 osd=0 3 2024-02-19T21:17:51.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 0 1 2024-02-19T21:17:51.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750047540 osd=0 4 2024-02-19T21:17:51.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f4756ce5700 10 client.23928.objecter _send_command 1 2024-02-19T21:17:51.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.552+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47500f91c0 con 0x7f47500598f0 2024-02-19T21:17:51.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.553+0000 7f474dffb700 10 client.23928.objecter ms_handle_connect 0x7f47500598f0 2024-02-19T21:17:51.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.553+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f47500f91c0 con 0x7f47500598f0 2024-02-19T21:17:51.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.553+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:17:51.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.553+0000 7f474dffb700 10 client.23928.objecter _finish_command 1 = system:0 2024-02-19T21:17:51.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.553+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f4750047540 osd=0 4 2024-02-19T21:17:51.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.553+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 0 1 2024-02-19T21:17:51.388 INFO:teuthology.orchestra.run.smithi163.stdout:osd.0: {} 2024-02-19T21:17:51.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.568+0000 7f4756ce5700 10 client.23928.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:51.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.568+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 2 2024-02-19T21:17:51.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.568+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750047540 osd=0 3 2024-02-19T21:17:51.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.568+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750047540 osd=0 3 2024-02-19T21:17:51.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.568+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750047540 osd=0 3 2024-02-19T21:17:51.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.568+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 2 2024-02-19T21:17:51.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.568+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f4750047540 osd=0 3 2024-02-19T21:17:51.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.568+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 0 2 2024-02-19T21:17:51.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.568+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750047540 osd=0 4 2024-02-19T21:17:51.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.568+0000 7f4756ce5700 10 client.23928.objecter _send_command 2 2024-02-19T21:17:51.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.568+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4750109f80 con 0x7f47500598f0 2024-02-19T21:17:51.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.569+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4750109f80 con 0x7f47500598f0 2024-02-19T21:17:51.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.569+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 2: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:51.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.569+0000 7f474dffb700 10 client.23928.objecter _finish_command 2 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.569+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f4750047540 osd=0 4 2024-02-19T21:17:51.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.569+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 0 2 2024-02-19T21:17:51.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f4756ce5700 10 client.23928.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:51.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 3 2024-02-19T21:17:51.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750059cd0 osd=1 3 2024-02-19T21:17:51.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750059cd0 osd=1 3 2024-02-19T21:17:51.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750059cd0 osd=1 3 2024-02-19T21:17:51.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 3 2024-02-19T21:17:51.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f4750059cd0 osd=1 3 2024-02-19T21:17:51.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 1 3 2024-02-19T21:17:51.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750059cd0 osd=1 4 2024-02-19T21:17:51.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f4756ce5700 10 client.23928.objecter _send_command 3 2024-02-19T21:17:51.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6805/3125047432 -- command(tid 3: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4750120950 con 0x7f4750059e40 2024-02-19T21:17:51.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f474dffb700 10 client.23928.objecter ms_handle_connect 0x7f4750059e40 2024-02-19T21:17:51.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 3: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4750120950 con 0x7f4750059e40 2024-02-19T21:17:51.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 3: 0 ) v1 2024-02-19T21:17:51.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f474dffb700 10 client.23928.objecter _finish_command 3 = system:0 2024-02-19T21:17:51.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f4750059cd0 osd=1 4 2024-02-19T21:17:51.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.570+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 1 3 2024-02-19T21:17:51.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.585+0000 7f4756ce5700 10 client.23928.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:51.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.585+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 4 2024-02-19T21:17:51.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.585+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750059cd0 osd=1 3 2024-02-19T21:17:51.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.585+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750059cd0 osd=1 3 2024-02-19T21:17:51.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.585+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750059cd0 osd=1 3 2024-02-19T21:17:51.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.585+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 4 2024-02-19T21:17:51.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.585+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f4750059cd0 osd=1 3 2024-02-19T21:17:51.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.585+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 1 4 2024-02-19T21:17:51.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.585+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750059cd0 osd=1 4 2024-02-19T21:17:51.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.585+0000 7f4756ce5700 10 client.23928.objecter _send_command 4 2024-02-19T21:17:51.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.585+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6805/3125047432 -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4750059360 con 0x7f4750059e40 2024-02-19T21:17:51.391 INFO:teuthology.orchestra.run.smithi163.stdout:osd.1: {} 2024-02-19T21:17:51.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4750059360 con 0x7f4750059e40 2024-02-19T21:17:51.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 4: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:51.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f474dffb700 10 client.23928.objecter _finish_command 4 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f4750059cd0 osd=1 4 2024-02-19T21:17:51.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 1 4 2024-02-19T21:17:51.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f4756ce5700 10 client.23928.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:51.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 5 2024-02-19T21:17:51.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750059710 osd=2 3 2024-02-19T21:17:51.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750059710 osd=2 3 2024-02-19T21:17:51.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750059710 osd=2 3 2024-02-19T21:17:51.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 5 2024-02-19T21:17:51.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f4750059710 osd=2 3 2024-02-19T21:17:51.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 2 5 2024-02-19T21:17:51.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750059710 osd=2 4 2024-02-19T21:17:51.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f4756ce5700 10 client.23928.objecter _send_command 5 2024-02-19T21:17:51.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6809/1563158487 -- command(tid 5: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4750125370 con 0x7f4750105340 2024-02-19T21:17:51.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.587+0000 7f474dffb700 10 client.23928.objecter ms_handle_connect 0x7f4750105340 2024-02-19T21:17:51.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.588+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 5: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4750125370 con 0x7f4750105340 2024-02-19T21:17:51.393 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.588+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 5: 0 ) v1 2024-02-19T21:17:51.393 INFO:teuthology.orchestra.run.smithi163.stdout:osd.2: {} 2024-02-19T21:17:51.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.588+0000 7f474dffb700 10 client.23928.objecter _finish_command 5 = system:0 2024-02-19T21:17:51.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.588+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f4750059710 osd=2 4 2024-02-19T21:17:51.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.588+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 2 5 2024-02-19T21:17:51.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.602+0000 7f4756ce5700 10 client.23928.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:51.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.602+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 6 2024-02-19T21:17:51.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.602+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750059710 osd=2 3 2024-02-19T21:17:51.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.602+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750059710 osd=2 3 2024-02-19T21:17:51.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.602+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750059710 osd=2 3 2024-02-19T21:17:51.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.602+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 6 2024-02-19T21:17:51.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.602+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f4750059710 osd=2 3 2024-02-19T21:17:51.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.602+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 2 6 2024-02-19T21:17:51.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.602+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750059710 osd=2 4 2024-02-19T21:17:51.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.602+0000 7f4756ce5700 10 client.23928.objecter _send_command 6 2024-02-19T21:17:51.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.602+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6809/1563158487 -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47501048a0 con 0x7f4750105340 2024-02-19T21:17:51.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f47501048a0 con 0x7f4750105340 2024-02-19T21:17:51.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 6: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:51.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f474dffb700 10 client.23928.objecter _finish_command 6 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f4750059710 osd=2 4 2024-02-19T21:17:51.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 2 6 2024-02-19T21:17:51.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f4756ce5700 10 client.23928.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:51.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 7 2024-02-19T21:17:51.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750104c50 osd=3 3 2024-02-19T21:17:51.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750104c50 osd=3 3 2024-02-19T21:17:51.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750104c50 osd=3 3 2024-02-19T21:17:51.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 7 2024-02-19T21:17:51.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f4750104c50 osd=3 3 2024-02-19T21:17:51.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 3 7 2024-02-19T21:17:51.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750104c50 osd=3 4 2024-02-19T21:17:51.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f4756ce5700 10 client.23928.objecter _send_command 7 2024-02-19T21:17:51.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.603+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6813/2112279769 -- command(tid 7: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4750129d90 con 0x7f47501217a0 2024-02-19T21:17:51.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.604+0000 7f474dffb700 10 client.23928.objecter ms_handle_connect 0x7f47501217a0 2024-02-19T21:17:51.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.604+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 7: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4750129d90 con 0x7f47501217a0 2024-02-19T21:17:51.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.604+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 7: 0 ) v1 2024-02-19T21:17:51.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.604+0000 7f474dffb700 10 client.23928.objecter _finish_command 7 = system:0 2024-02-19T21:17:51.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.604+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f4750104c50 osd=3 4 2024-02-19T21:17:51.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.604+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 3 7 2024-02-19T21:17:51.396 INFO:teuthology.orchestra.run.smithi163.stdout:osd.3: {} 2024-02-19T21:17:51.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.618+0000 7f4756ce5700 10 client.23928.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:51.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.618+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 8 2024-02-19T21:17:51.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.618+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750104c50 osd=3 3 2024-02-19T21:17:51.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.618+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750104c50 osd=3 3 2024-02-19T21:17:51.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.618+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750104c50 osd=3 3 2024-02-19T21:17:51.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.618+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 8 2024-02-19T21:17:51.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.618+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f4750104c50 osd=3 3 2024-02-19T21:17:51.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.618+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 3 8 2024-02-19T21:17:51.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.618+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750104c50 osd=3 4 2024-02-19T21:17:51.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.618+0000 7f4756ce5700 10 client.23928.objecter _send_command 8 2024-02-19T21:17:51.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.618+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.163:6813/2112279769 -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4750120d00 con 0x7f47501217a0 2024-02-19T21:17:51.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4750120d00 con 0x7f47501217a0 2024-02-19T21:17:51.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 8: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:51.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f474dffb700 10 client.23928.objecter _finish_command 8 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f4750104c50 osd=3 4 2024-02-19T21:17:51.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 3 8 2024-02-19T21:17:51.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f4756ce5700 10 client.23928.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:51.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 9 2024-02-19T21:17:51.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750120d00 osd=4 3 2024-02-19T21:17:51.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750120d00 osd=4 3 2024-02-19T21:17:51.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750120d00 osd=4 3 2024-02-19T21:17:51.398 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 9 2024-02-19T21:17:51.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f4750120d00 osd=4 3 2024-02-19T21:17:51.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 4 9 2024-02-19T21:17:51.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750120d00 osd=4 4 2024-02-19T21:17:51.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f4756ce5700 10 client.23928.objecter _send_command 9 2024-02-19T21:17:51.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.620+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.193:6800/3129716965 -- command(tid 9: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4750120fc0 con 0x7f47501261c0 2024-02-19T21:17:51.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.621+0000 7f474dffb700 10 client.23928.objecter ms_handle_connect 0x7f47501261c0 2024-02-19T21:17:51.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.621+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 9: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4750120fc0 con 0x7f47501261c0 2024-02-19T21:17:51.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.621+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 9: 0 ) v1 2024-02-19T21:17:51.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.621+0000 7f474dffb700 10 client.23928.objecter _finish_command 9 = system:0 2024-02-19T21:17:51.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.621+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f4750120d00 osd=4 4 2024-02-19T21:17:51.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.621+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 4 9 2024-02-19T21:17:51.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.636+0000 7f4756ce5700 10 client.23928.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:51.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.636+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 10 2024-02-19T21:17:51.399 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.636+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750120d00 osd=4 3 2024-02-19T21:17:51.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.636+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750120d00 osd=4 3 2024-02-19T21:17:51.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.636+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750120d00 osd=4 3 2024-02-19T21:17:51.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.636+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 10 2024-02-19T21:17:51.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.636+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f4750120d00 osd=4 3 2024-02-19T21:17:51.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.636+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 4 10 2024-02-19T21:17:51.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.636+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750120d00 osd=4 4 2024-02-19T21:17:51.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.636+0000 7f4756ce5700 10 client.23928.objecter _send_command 10 2024-02-19T21:17:51.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.636+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.193:6800/3129716965 -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4750120fc0 con 0x7f47501261c0 2024-02-19T21:17:51.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4750120fc0 con 0x7f47501261c0 2024-02-19T21:17:51.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 10: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:51.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f474dffb700 10 client.23928.objecter _finish_command 10 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.400 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f4750120d00 osd=4 4 2024-02-19T21:17:51.401 INFO:teuthology.orchestra.run.smithi163.stdout:osd.4: {} 2024-02-19T21:17:51.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 4 10 2024-02-19T21:17:51.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f4756ce5700 10 client.23928.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:51.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 11 2024-02-19T21:17:51.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750120fc0 osd=5 3 2024-02-19T21:17:51.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750120fc0 osd=5 3 2024-02-19T21:17:51.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750120fc0 osd=5 3 2024-02-19T21:17:51.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 11 2024-02-19T21:17:51.401 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f4750120fc0 osd=5 3 2024-02-19T21:17:51.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 5 11 2024-02-19T21:17:51.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750120fc0 osd=5 4 2024-02-19T21:17:51.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f4756ce5700 10 client.23928.objecter _send_command 11 2024-02-19T21:17:51.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.638+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.193:6804/805161991 -- command(tid 11: {"prefix": "get_command_descriptions"}) v1 -- 0x7f475012b0c0 con 0x7f475012ab20 2024-02-19T21:17:51.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.639+0000 7f474dffb700 10 client.23928.objecter ms_handle_connect 0x7f475012ab20 2024-02-19T21:17:51.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.639+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 11: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f475012b0c0 con 0x7f475012ab20 2024-02-19T21:17:51.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.639+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 11: 0 ) v1 2024-02-19T21:17:51.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.639+0000 7f474dffb700 10 client.23928.objecter _finish_command 11 = system:0 2024-02-19T21:17:51.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.639+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f4750120fc0 osd=5 4 2024-02-19T21:17:51.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.639+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 5 11 2024-02-19T21:17:51.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.654+0000 7f4756ce5700 10 client.23928.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:51.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.654+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 12 2024-02-19T21:17:51.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.654+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750120fc0 osd=5 3 2024-02-19T21:17:51.402 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.654+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750120fc0 osd=5 3 2024-02-19T21:17:51.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.654+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750120fc0 osd=5 3 2024-02-19T21:17:51.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.654+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 12 2024-02-19T21:17:51.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.654+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f4750120fc0 osd=5 3 2024-02-19T21:17:51.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.654+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 5 12 2024-02-19T21:17:51.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.654+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750120fc0 osd=5 4 2024-02-19T21:17:51.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.654+0000 7f4756ce5700 10 client.23928.objecter _send_command 12 2024-02-19T21:17:51.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.654+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.193:6804/805161991 -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f475012b0c0 con 0x7f475012ab20 2024-02-19T21:17:51.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f473c04ae90 con 0x7f475012ab20 2024-02-19T21:17:51.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 12: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:51.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f474dffb700 10 client.23928.objecter _finish_command 12 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f4750120fc0 osd=5 4 2024-02-19T21:17:51.403 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 5 12 2024-02-19T21:17:51.404 INFO:teuthology.orchestra.run.smithi163.stdout:osd.5: {} 2024-02-19T21:17:51.404 INFO:teuthology.orchestra.run.smithi163.stdout:osd.6: {} 2024-02-19T21:17:51.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f4756ce5700 10 client.23928.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:51.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 13 2024-02-19T21:17:51.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f475012b2e0 osd=6 3 2024-02-19T21:17:51.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f475012b2e0 osd=6 3 2024-02-19T21:17:51.404 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f475012b2e0 osd=6 3 2024-02-19T21:17:51.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 13 2024-02-19T21:17:51.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f475012b2e0 osd=6 3 2024-02-19T21:17:51.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 6 13 2024-02-19T21:17:51.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f475012b2e0 osd=6 4 2024-02-19T21:17:51.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f4756ce5700 10 client.23928.objecter _send_command 13 2024-02-19T21:17:51.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.656+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.193:6808/787604615 -- command(tid 13: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4750136fb0 con 0x7f4750133360 2024-02-19T21:17:51.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.657+0000 7f474dffb700 10 client.23928.objecter ms_handle_connect 0x7f4750133360 2024-02-19T21:17:51.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.657+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 13: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f4750136fb0 con 0x7f4750133360 2024-02-19T21:17:51.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.657+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 13: 0 ) v1 2024-02-19T21:17:51.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.657+0000 7f474dffb700 10 client.23928.objecter _finish_command 13 = system:0 2024-02-19T21:17:51.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.657+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f475012b2e0 osd=6 4 2024-02-19T21:17:51.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.657+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 6 13 2024-02-19T21:17:51.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.671+0000 7f4756ce5700 10 client.23928.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:51.405 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.671+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 14 2024-02-19T21:17:51.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.671+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f475012b2e0 osd=6 3 2024-02-19T21:17:51.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.671+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f475012b2e0 osd=6 3 2024-02-19T21:17:51.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.671+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f475012b2e0 osd=6 3 2024-02-19T21:17:51.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.671+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 14 2024-02-19T21:17:51.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.671+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f475012b2e0 osd=6 3 2024-02-19T21:17:51.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.671+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 6 14 2024-02-19T21:17:51.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.671+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f475012b2e0 osd=6 4 2024-02-19T21:17:51.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.671+0000 7f4756ce5700 10 client.23928.objecter _send_command 14 2024-02-19T21:17:51.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.671+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.193:6808/787604615 -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f4750136fb0 con 0x7f4750133360 2024-02-19T21:17:51.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f4750136fb0 con 0x7f4750133360 2024-02-19T21:17:51.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 14: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:51.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f474dffb700 10 client.23928.objecter _finish_command 14 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f475012b2e0 osd=6 4 2024-02-19T21:17:51.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 6 14 2024-02-19T21:17:51.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f4756ce5700 10 client.23928.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:17:51.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 15 2024-02-19T21:17:51.407 INFO:teuthology.orchestra.run.smithi163.stdout:osd.7: {} 2024-02-19T21:17:51.407 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750136fb0 osd=7 3 2024-02-19T21:17:51.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750136fb0 osd=7 3 2024-02-19T21:17:51.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750136fb0 osd=7 3 2024-02-19T21:17:51.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 15 2024-02-19T21:17:51.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f4750136fb0 osd=7 3 2024-02-19T21:17:51.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 7 15 2024-02-19T21:17:51.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750136fb0 osd=7 4 2024-02-19T21:17:51.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f4756ce5700 10 client.23928.objecter _send_command 15 2024-02-19T21:17:51.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.673+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.193:6812/157927432 -- command(tid 15: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47501381d0 con 0x7f4750138c70 2024-02-19T21:17:51.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.674+0000 7f474dffb700 10 client.23928.objecter ms_handle_connect 0x7f4750138c70 2024-02-19T21:17:51.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.675+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 15: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f47501381d0 con 0x7f4750138c70 2024-02-19T21:17:51.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.675+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 15: 0 ) v1 2024-02-19T21:17:51.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.675+0000 7f474dffb700 10 client.23928.objecter _finish_command 15 = system:0 2024-02-19T21:17:51.408 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.675+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f4750136fb0 osd=7 4 2024-02-19T21:17:51.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.675+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 7 15 2024-02-19T21:17:51.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.689+0000 7f4756ce5700 10 client.23928.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-02-19T21:17:51.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.689+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign -1 16 2024-02-19T21:17:51.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.689+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750136fb0 osd=7 3 2024-02-19T21:17:51.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.689+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750136fb0 osd=7 3 2024-02-19T21:17:51.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.689+0000 7f4756ce5700 20 client.23928.objecter _get_session s=0x7f4750136fb0 osd=7 3 2024-02-19T21:17:51.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.689+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_remove -1 16 2024-02-19T21:17:51.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.689+0000 7f4756ce5700 20 client.23928.objecter get_session s=0x7f4750136fb0 osd=7 3 2024-02-19T21:17:51.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.689+0000 7f4756ce5700 15 client.23928.objecter _session_command_op_assign 7 16 2024-02-19T21:17:51.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.689+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750136fb0 osd=7 4 2024-02-19T21:17:51.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.689+0000 7f4756ce5700 10 client.23928.objecter _send_command 16 2024-02-19T21:17:51.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.689+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 --> v1:172.21.15.193:6812/157927432 -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) v1 -- 0x7f47501381d0 con 0x7f4750138c70 2024-02-19T21:17:51.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f474dffb700 1 -- 172.21.15.163:0/986067486 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 ==== 40+0+3 (unknown 2619146361 0 2423995997) 0x7f47501381d0 con 0x7f4750138c70 2024-02-19T21:17:51.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f474dffb700 10 client.23928.objecter ms_dispatch 0x7f4750001550 command_reply(tid 16: 0 osd_enable_op_tracker = 'false' ) v1 2024-02-19T21:17:51.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f474dffb700 10 client.23928.objecter _finish_command 16 = system:0 osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f474dffb700 20 client.23928.objecter put_session s=0x7f4750136fb0 osd=7 4 2024-02-19T21:17:51.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f474dffb700 15 client.23928.objecter _session_command_op_remove 7 16 2024-02-19T21:17:51.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 10 librados: watch_flush enter 2024-02-19T21:17:51.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 10 librados: watch_flush exit 2024-02-19T21:17:51.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 10 client.23928.objecter close_session for osd.0 2024-02-19T21:17:51.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 >> v1:172.21.15.163:6801/1559418115 conn(0x7f47500598f0 legacy=0x7f475011c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:51.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750047540 osd=0 1 2024-02-19T21:17:51.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 10 client.23928.objecter close_session for osd.1 2024-02-19T21:17:51.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 >> v1:172.21.15.163:6805/3125047432 conn(0x7f4750059e40 legacy=0x7f475011f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:51.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750059cd0 osd=1 1 2024-02-19T21:17:51.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 10 client.23928.objecter close_session for osd.2 2024-02-19T21:17:51.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 >> v1:172.21.15.163:6809/1563158487 conn(0x7f4750105340 legacy=0x7f4750123c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:51.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750059710 osd=2 1 2024-02-19T21:17:51.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 10 client.23928.objecter close_session for osd.3 2024-02-19T21:17:51.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 >> v1:172.21.15.163:6813/2112279769 conn(0x7f47501217a0 legacy=0x7f4750128670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:51.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750104c50 osd=3 1 2024-02-19T21:17:51.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 10 client.23928.objecter close_session for osd.4 2024-02-19T21:17:51.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 >> v1:172.21.15.193:6800/3129716965 conn(0x7f47501261c0 legacy=0x7f4750129f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:51.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750120d00 osd=4 1 2024-02-19T21:17:51.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 10 client.23928.objecter close_session for osd.5 2024-02-19T21:17:51.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 >> v1:172.21.15.193:6804/805161991 conn(0x7f475012ab20 legacy=0x7f4750131730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:51.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4755282700 1 -- 172.21.15.163:0/986067486 reap_dead start 2024-02-19T21:17:51.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750120fc0 osd=5 1 2024-02-19T21:17:51.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 10 client.23928.objecter close_session for osd.6 2024-02-19T21:17:51.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 >> v1:172.21.15.193:6808/787604615 conn(0x7f4750133360 legacy=0x7f4750135750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:51.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4755282700 1 -- 172.21.15.163:0/986067486 reap_dead start 2024-02-19T21:17:51.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f475012b2e0 osd=6 1 2024-02-19T21:17:51.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 10 client.23928.objecter close_session for osd.7 2024-02-19T21:17:51.411 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 >> v1:172.21.15.193:6812/157927432 conn(0x7f4750138c70 legacy=0x7f47501390f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:51.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 20 client.23928.objecter put_session s=0x7f4750136fb0 osd=7 1 2024-02-19T21:17:51.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 20 client.23928.objecter shutdown clearing up homeless session... 2024-02-19T21:17:51.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 10 client.23928.objecter successfully canceled tick 2024-02-19T21:17:51.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4755282700 1 -- 172.21.15.163:0/986067486 reap_dead start 2024-02-19T21:17:51.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4755282700 1 -- 172.21.15.163:0/986067486 reap_dead start 2024-02-19T21:17:51.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4740063da0 legacy=0x7f4740066260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:51.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 >> v1:172.21.15.163:6789/0 conn(0x7f4750100710 legacy=0x7f4750114ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:51.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 shutdown_connections 2024-02-19T21:17:51.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 >> 172.21.15.163:0/986067486 conn(0x7f47500f3530 msgr2=0x7f47501076c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:51.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 shutdown_connections 2024-02-19T21:17:51.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 1 -- 172.21.15.163:0/986067486 wait complete. 2024-02-19T21:17:51.412 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:50.691+0000 7f4756ce5700 1 librados: shutdown 2024-02-19T21:17:51.412 INFO:teuthology.orchestra.run.smithi163.stderr:osd.0: osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.413 INFO:teuthology.orchestra.run.smithi163.stderr:osd.1: osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.413 INFO:teuthology.orchestra.run.smithi163.stderr:osd.2: osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.413 INFO:teuthology.orchestra.run.smithi163.stderr:osd.3: osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.413 INFO:teuthology.orchestra.run.smithi163.stderr:osd.4: osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.413 INFO:teuthology.orchestra.run.smithi163.stderr:osd.5: osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.413 INFO:teuthology.orchestra.run.smithi163.stderr:osd.6: osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.413 INFO:teuthology.orchestra.run.smithi163.stderr:osd.7: osd_enable_op_tracker = 'false' 2024-02-19T21:17:51.456 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:17:51.794 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:50.553390+0000 mon.a (mon.0) 3755 : audit [DBG] from='client.? v1:172.21.15.163:0/986067486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:17:51.794 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:50.646510+0000 osd.6 (osd.6) 291 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:17:51.794 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:50.647573+0000 osd.6 (osd.6) 292 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:17:51.794 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:50.833555+0000 mgr.y (mgr.14550) 667 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:51.794 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:51.152046+0000 osd.3 (osd.3) 339 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:17:51.794 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:51.152855+0000 osd.3 (osd.3) 340 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:17:51.795 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:50.553390+0000 mon.a (mon.0) 3755 : audit [DBG] from='client.? v1:172.21.15.163:0/986067486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:17:51.795 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:50.646510+0000 osd.6 (osd.6) 291 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:17:51.795 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:50.647573+0000 osd.6 (osd.6) 292 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:17:51.795 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:50.833555+0000 mgr.y (mgr.14550) 667 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:51.795 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:51.152046+0000 osd.3 (osd.3) 339 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:17:51.795 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:51.152855+0000 osd.3 (osd.3) 340 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:17:51.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:50.553390+0000 mon.a (mon.0) 3755 : audit [DBG] from='client.? v1:172.21.15.163:0/986067486' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-19T21:17:51.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:50.646510+0000 osd.6 (osd.6) 291 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:17:51.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:50.647573+0000 osd.6 (osd.6) 292 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:17:51.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:50.833555+0000 mgr.y (mgr.14550) 667 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:17:51.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:51.152046+0000 osd.3 (osd.3) 339 : cluster [DBG] 2.1e scrub starts 2024-02-19T21:17:51.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:51.152855+0000 osd.3 (osd.3) 340 : cluster [DBG] 2.1e scrub ok 2024-02-19T21:17:52.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.000+0000 7f91152f9700 1 -- 172.21.15.163:0/2311968723 >> v1:172.21.15.163:6790/0 conn(0x7f91101031c0 legacy=0x7f9110105680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:52.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.001+0000 7f91152f9700 1 -- 172.21.15.163:0/2311968723 shutdown_connections 2024-02-19T21:17:52.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.001+0000 7f91152f9700 1 -- 172.21.15.163:0/2311968723 >> 172.21.15.163:0/2311968723 conn(0x7f911006cc00 msgr2=0x7f91100f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:52.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.001+0000 7f91152f9700 1 -- 172.21.15.163:0/2311968723 shutdown_connections 2024-02-19T21:17:52.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.001+0000 7f91152f9700 1 -- 172.21.15.163:0/2311968723 wait complete. 2024-02-19T21:17:52.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.001+0000 7f91152f9700 1 librados: starting msgr at 2024-02-19T21:17:52.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.001+0000 7f91152f9700 1 librados: starting objecter 2024-02-19T21:17:52.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.001+0000 7f91152f9700 1 Processor -- start 2024-02-19T21:17:52.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.001+0000 7f91152f9700 1 -- start start 2024-02-19T21:17:52.005 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.001+0000 7f91152f9700 1 librados: setting wanted keys 2024-02-19T21:17:52.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.001+0000 7f91152f9700 1 librados: calling monclient init 2024-02-19T21:17:52.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f91152f9700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9110117320 con 0x7f91100ff6b0 2024-02-19T21:17:52.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f91152f9700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9110118500 con 0x7f91101031c0 2024-02-19T21:17:52.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f91152f9700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9110119700 con 0x7f91100fc880 2024-02-19T21:17:52.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f910effd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f91100fc880 0x7f91100f9be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:43108/0 (socket says 172.21.15.163:43108) 2024-02-19T21:17:52.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f910effd700 1 -- 172.21.15.163:0/3249517926 learned_addr learned my addr 172.21.15.163:0/3249517926 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:52.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 10 client.?.objecter ms_handle_connect 0x7f91100fc880 2024-02-19T21:17:52.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:52.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 10 client.?.objecter ms_handle_connect 0x7f91100ff6b0 2024-02-19T21:17:52.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:52.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 10 client.?.objecter ms_handle_connect 0x7f91101031c0 2024-02-19T21:17:52.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:52.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024959409 0 0) 0x7f9110119700 con 0x7f91100fc880 2024-02-19T21:17:52.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90f80033f0 con 0x7f91100fc880 2024-02-19T21:17:52.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713719796 0 0) 0x7f9110117320 con 0x7f91100ff6b0 2024-02-19T21:17:52.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9110119700 con 0x7f91100ff6b0 2024-02-19T21:17:52.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262355974 0 0) 0x7f9110118500 con 0x7f91101031c0 2024-02-19T21:17:52.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9110117320 con 0x7f91101031c0 2024-02-19T21:17:52.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3599224748 0 0) 0x7f9110117320 con 0x7f91101031c0 2024-02-19T21:17:52.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9110118500 con 0x7f91101031c0 2024-02-19T21:17:52.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9108002250 con 0x7f91101031c0 2024-02-19T21:17:52.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3168384317 0 0) 0x7f90f80033f0 con 0x7f91100fc880 2024-02-19T21:17:52.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9110117320 con 0x7f91100fc880 2024-02-19T21:17:52.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2880063658 0 0) 0x7f9110119700 con 0x7f91100ff6b0 2024-02-19T21:17:52.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90f80033f0 con 0x7f91100ff6b0 2024-02-19T21:17:52.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9104001940 con 0x7f91100fc880 2024-02-19T21:17:52.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9100001840 con 0x7f91100ff6b0 2024-02-19T21:17:52.007 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3906037667 0 0) 0x7f9110118500 con 0x7f91101031c0 2024-02-19T21:17:52.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 >> v1:172.21.15.163:6790/0 conn(0x7f91100fc880 legacy=0x7f91100f9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:52.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 >> v1:172.21.15.163:6789/0 conn(0x7f91100ff6b0 legacy=0x7f91100fa2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:52.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911011a900 con 0x7f91101031c0 2024-02-19T21:17:52.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f91152f9700 1 -- 172.21.15.163:0/3249517926 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9110117510 con 0x7f91101031c0 2024-02-19T21:17:52.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f91152f9700 10 client.49471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:52.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f91152f9700 1 -- 172.21.15.163:0/3249517926 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9110117980 con 0x7f91101031c0 2024-02-19T21:17:52.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.002+0000 7f91152f9700 1 librados: init done 2024-02-19T21:17:52.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.003+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f91080031d0 con 0x7f91101031c0 2024-02-19T21:17:52.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.003+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9108003a00 con 0x7f91101031c0 2024-02-19T21:17:52.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.003+0000 7f91152f9700 1 -- 172.21.15.163:0/3249517926 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90fc004f20 con 0x7f91101031c0 2024-02-19T21:17:52.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.004+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9108001b70 con 0x7f91101031c0 2024-02-19T21:17:52.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.004+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(806..806 src has 633..806) v4 ==== 7187+0+0 (unknown 2421357025 0 0) 0x7f91080767b0 con 0x7f91101031c0 2024-02-19T21:17:52.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.004+0000 7f90f7fff700 10 client.49471.objecter ms_dispatch 0x7f9110001550 osd_map(806..806 src has 633..806) v4 2024-02-19T21:17:52.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.004+0000 7f90f7fff700 3 client.49471.objecter handle_osd_map got epochs [806,806] > 0 2024-02-19T21:17:52.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.004+0000 7f90f7fff700 3 client.49471.objecter handle_osd_map decoding full epoch 806 2024-02-19T21:17:52.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.004+0000 7f90f7fff700 20 client.49471.objecter dump_active .. 0 homeless 2024-02-19T21:17:52.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.004+0000 7f90f7fff700 10 client.49471.objecter ms_handle_connect 0x7f90f8063db0 2024-02-19T21:17:52.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.009+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9108049f70 con 0x7f91101031c0 2024-02-19T21:17:52.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.183+0000 7f91152f9700 1 -- 172.21.15.163:0/3249517926 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) v1 -- 0x7f90fc0051d0 con 0x7f91101031c0 2024-02-19T21:17:52.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.644+0000 7f90f7fff700 1 -- 172.21.15.163:0/3249517926 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v807) v1 ==== 87+0+0 (unknown 2773576570 0 0) 0x7f91080483e0 con 0x7f91101031c0 2024-02-19T21:17:52.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.646+0000 7f91152f9700 10 librados: watch_flush enter 2024-02-19T21:17:52.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.646+0000 7f91152f9700 10 librados: watch_flush exit 2024-02-19T21:17:52.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.646+0000 7f91152f9700 20 client.49471.objecter shutdown clearing up homeless session... 2024-02-19T21:17:52.648 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.646+0000 7f91152f9700 10 client.49471.objecter successfully canceled tick 2024-02-19T21:17:52.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.646+0000 7f91152f9700 1 -- 172.21.15.163:0/3249517926 >> v1:172.21.15.163:6800/1552514681 conn(0x7f90f8063db0 legacy=0x7f90f8066270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:52.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.646+0000 7f91152f9700 1 -- 172.21.15.163:0/3249517926 >> v1:172.21.15.193:6789/0 conn(0x7f91101031c0 legacy=0x7f91101159e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:52.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.647+0000 7f91152f9700 1 -- 172.21.15.163:0/3249517926 shutdown_connections 2024-02-19T21:17:52.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.647+0000 7f91152f9700 1 -- 172.21.15.163:0/3249517926 >> 172.21.15.163:0/3249517926 conn(0x7f911006cc00 msgr2=0x7f9110106670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:52.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.647+0000 7f91152f9700 1 -- 172.21.15.163:0/3249517926 shutdown_connections 2024-02-19T21:17:52.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.647+0000 7f91152f9700 1 -- 172.21.15.163:0/3249517926 wait complete. 2024-02-19T21:17:52.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:52.647+0000 7f91152f9700 1 librados: shutdown 2024-02-19T21:17:52.649 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is set 2024-02-19T21:17:52.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:51.326517+0000 osd.1 (osd.1) 362 : cluster [DBG] 4.b deep-scrub starts 2024-02-19T21:17:52.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:51.331010+0000 osd.1 (osd.1) 363 : cluster [DBG] 4.b deep-scrub ok 2024-02-19T21:17:52.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:51.345273+0000 mgr.y (mgr.14550) 668 : cluster [DBG] pgmap v1223: 174 pgs: 174 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:17:52.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:51.446406+0000 osd.2 (osd.2) 322 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:17:52.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:51.447267+0000 osd.2 (osd.2) 323 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:17:52.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:51.522365+0000 mon.a (mon.0) 3756 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-19T21:17:52.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:51.631903+0000 mon.b (mon.1) 755 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:17:52.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:51.632767+0000 mon.b (mon.1) 756 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:17:52.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:51.637108+0000 mon.a (mon.0) 3757 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:52.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:52.185299+0000 mon.b (mon.1) 757 : audit [INF] from='client.? v1:172.21.15.163:0/3249517926' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:17:52.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:52.185795+0000 mon.a (mon.0) 3758 : audit [INF] from='client.49471 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:17:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:51.326517+0000 osd.1 (osd.1) 362 : cluster [DBG] 4.b deep-scrub starts 2024-02-19T21:17:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:51.331010+0000 osd.1 (osd.1) 363 : cluster [DBG] 4.b deep-scrub ok 2024-02-19T21:17:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:51.345273+0000 mgr.y (mgr.14550) 668 : cluster [DBG] pgmap v1223: 174 pgs: 174 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:17:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:51.446406+0000 osd.2 (osd.2) 322 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:17:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:51.447267+0000 osd.2 (osd.2) 323 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:17:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:51.522365+0000 mon.a (mon.0) 3756 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-19T21:17:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:51.631903+0000 mon.b (mon.1) 755 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:17:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:51.632767+0000 mon.b (mon.1) 756 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:17:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:51.637108+0000 mon.a (mon.0) 3757 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:52.185299+0000 mon.b (mon.1) 757 : audit [INF] from='client.? v1:172.21.15.163:0/3249517926' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:17:52.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:52.185795+0000 mon.a (mon.0) 3758 : audit [INF] from='client.49471 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:17:52.787 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:51.326517+0000 osd.1 (osd.1) 362 : cluster [DBG] 4.b deep-scrub starts 2024-02-19T21:17:52.787 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:51.331010+0000 osd.1 (osd.1) 363 : cluster [DBG] 4.b deep-scrub ok 2024-02-19T21:17:52.787 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:51.345273+0000 mgr.y (mgr.14550) 668 : cluster [DBG] pgmap v1223: 174 pgs: 174 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T21:17:52.787 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:51.446406+0000 osd.2 (osd.2) 322 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:17:52.787 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:51.447267+0000 osd.2 (osd.2) 323 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:17:52.787 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:51.522365+0000 mon.a (mon.0) 3756 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-19T21:17:52.788 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:51.631903+0000 mon.b (mon.1) 755 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:17:52.788 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:51.632767+0000 mon.b (mon.1) 756 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:17:52.788 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:51.637108+0000 mon.a (mon.0) 3757 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:52.788 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:52.185299+0000 mon.b (mon.1) 757 : audit [INF] from='client.? v1:172.21.15.163:0/3249517926' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:17:52.788 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:52.185795+0000 mon.a (mon.0) 3758 : audit [INF] from='client.49471 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-19T21:17:53.285 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:17:53.286 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:17:53.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:52.635465+0000 mon.a (mon.0) 3759 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:17:53.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:52.644330+0000 mon.a (mon.0) 3760 : audit [INF] from='client.49471 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:17:53.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:52.644396+0000 mon.a (mon.0) 3761 : cluster [DBG] osdmap e807: 8 total, 8 up, 6 in 2024-02-19T21:17:53.749 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:17:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:17:53] "GET /metrics HTTP/1.1" 200 192864 "" "Prometheus/2.33.4" 2024-02-19T21:17:53.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:52.635465+0000 mon.a (mon.0) 3759 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:17:53.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:52.644330+0000 mon.a (mon.0) 3760 : audit [INF] from='client.49471 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:17:53.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:52.644396+0000 mon.a (mon.0) 3761 : cluster [DBG] osdmap e807: 8 total, 8 up, 6 in 2024-02-19T21:17:53.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:52.635465+0000 mon.a (mon.0) 3759 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-19T21:17:53.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:52.644330+0000 mon.a (mon.0) 3760 : audit [INF] from='client.49471 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-19T21:17:53.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:52.644396+0000 mon.a (mon.0) 3761 : cluster [DBG] osdmap e807: 8 total, 8 up, 6 in 2024-02-19T21:17:53.991 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-19T21:17:54.256 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:17:54.597 INFO:tasks.thrashosds.thrasher:inning osd 2024-02-19T21:17:54.597 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-02-19T21:17:54.597 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd in 5 2024-02-19T21:17:54.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:53.309143+0000 osd.2 (osd.2) 324 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:54.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:53.309356+0000 osd.2 (osd.2) 325 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:54.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:53.345670+0000 mgr.y (mgr.14550) 669 : cluster [DBG] pgmap v1225: 174 pgs: 174 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 870 B/s rd, 0 op/s 2024-02-19T21:17:54.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:53.649102+0000 mon.a (mon.0) 3762 : cluster [DBG] osdmap e808: 8 total, 8 up, 6 in 2024-02-19T21:17:54.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:54.110488+0000 mon.a (mon.0) 3763 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:54.878 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:54.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:53.309143+0000 osd.2 (osd.2) 324 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:54.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:53.309356+0000 osd.2 (osd.2) 325 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:54.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:53.345670+0000 mgr.y (mgr.14550) 669 : cluster [DBG] pgmap v1225: 174 pgs: 174 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 870 B/s rd, 0 op/s 2024-02-19T21:17:54.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:53.649102+0000 mon.a (mon.0) 3762 : cluster [DBG] osdmap e808: 8 total, 8 up, 6 in 2024-02-19T21:17:54.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:54.110488+0000 mon.a (mon.0) 3763 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:54.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:53.309143+0000 osd.2 (osd.2) 324 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:17:54.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:53.309356+0000 osd.2 (osd.2) 325 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:17:54.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:53.345670+0000 mgr.y (mgr.14550) 669 : cluster [DBG] pgmap v1225: 174 pgs: 174 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 870 B/s rd, 0 op/s 2024-02-19T21:17:54.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:53.649102+0000 mon.a (mon.0) 3762 : cluster [DBG] osdmap e808: 8 total, 8 up, 6 in 2024-02-19T21:17:55.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:54.110488+0000 mon.a (mon.0) 3763 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:17:55.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset noscrub 2024-02-19T21:17:56.034 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:17:56.034 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:17:56.650 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:55.346768+0000 mgr.y (mgr.14550) 670 : cluster [DBG] pgmap v1227: 174 pgs: 174 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:17:56.674 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:56.695 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:55.346768+0000 mgr.y (mgr.14550) 670 : cluster [DBG] pgmap v1227: 174 pgs: 174 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:17:56.713 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-19T21:17:56.913 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:55.346768+0000 mgr.y (mgr.14550) 670 : cluster [DBG] pgmap v1227: 174 pgs: 174 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:17:56.977 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:17:57.774 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:57.599237+0000 mon.a (mon.0) 3764 : audit [DBG] from='client.? v1:172.21.15.163:0/2192947096' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-02-19T21:17:57.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.409+0000 7fbc673ee700 1 -- 172.21.15.163:0/4036443358 >> v1:172.21.15.163:6789/0 conn(0x7fbc60100780 legacy=0x7fbc60102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:57.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.409+0000 7fbc673ee700 1 -- 172.21.15.163:0/4036443358 shutdown_connections 2024-02-19T21:17:57.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.409+0000 7fbc673ee700 1 -- 172.21.15.163:0/4036443358 >> 172.21.15.163:0/4036443358 conn(0x7fbc600f35a0 msgr2=0x7fbc600f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:57.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.410+0000 7fbc673ee700 1 -- 172.21.15.163:0/4036443358 shutdown_connections 2024-02-19T21:17:57.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.410+0000 7fbc673ee700 1 -- 172.21.15.163:0/4036443358 wait complete. 2024-02-19T21:17:57.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.410+0000 7fbc673ee700 1 librados: starting msgr at 2024-02-19T21:17:57.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.410+0000 7fbc673ee700 1 librados: starting objecter 2024-02-19T21:17:57.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.410+0000 7fbc673ee700 1 Processor -- start 2024-02-19T21:17:57.825 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.410+0000 7fbc673ee700 1 -- start start 2024-02-19T21:17:57.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.410+0000 7fbc673ee700 1 librados: setting wanted keys 2024-02-19T21:17:57.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.410+0000 7fbc673ee700 1 librados: calling monclient init 2024-02-19T21:17:57.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.410+0000 7fbc673ee700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbc60117330 con 0x7fbc60100780 2024-02-19T21:17:57.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.410+0000 7fbc673ee700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbc60118510 con 0x7fbc601042c0 2024-02-19T21:17:57.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.410+0000 7fbc673ee700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbc60119710 con 0x7fbc600fd980 2024-02-19T21:17:57.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.410+0000 7fbc6518a700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fbc600fd980 0x7fbc600f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:56822/0 (socket says 172.21.15.163:56822) 2024-02-19T21:17:57.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.410+0000 7fbc6518a700 1 -- 172.21.15.163:0/2192947096 learned_addr learned my addr 172.21.15.163:0/2192947096 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:57.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 10 client.?.objecter ms_handle_connect 0x7fbc60100780 2024-02-19T21:17:57.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:57.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 10 client.?.objecter ms_handle_connect 0x7fbc600fd980 2024-02-19T21:17:57.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:57.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 10 client.?.objecter ms_handle_connect 0x7fbc601042c0 2024-02-19T21:17:57.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:57.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480021428 0 0) 0x7fbc60117330 con 0x7fbc60100780 2024-02-19T21:17:57.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc4c0033f0 con 0x7fbc60100780 2024-02-19T21:17:57.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852899823 0 0) 0x7fbc60119710 con 0x7fbc600fd980 2024-02-19T21:17:57.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc60117330 con 0x7fbc600fd980 2024-02-19T21:17:57.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013125019 0 0) 0x7fbc60118510 con 0x7fbc601042c0 2024-02-19T21:17:57.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc60119710 con 0x7fbc601042c0 2024-02-19T21:17:57.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3172725680 0 0) 0x7fbc4c0033f0 con 0x7fbc60100780 2024-02-19T21:17:57.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc60118510 con 0x7fbc60100780 2024-02-19T21:17:57.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbc5c002050 con 0x7fbc60100780 2024-02-19T21:17:57.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 253854872 0 0) 0x7fbc60119710 con 0x7fbc601042c0 2024-02-19T21:17:57.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc4c0033f0 con 0x7fbc601042c0 2024-02-19T21:17:57.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1710334727 0 0) 0x7fbc60117330 con 0x7fbc600fd980 2024-02-19T21:17:57.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc60119710 con 0x7fbc600fd980 2024-02-19T21:17:57.827 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbc58001bb0 con 0x7fbc601042c0 2024-02-19T21:17:57.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbc50001900 con 0x7fbc600fd980 2024-02-19T21:17:57.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3351954737 0 0) 0x7fbc60118510 con 0x7fbc60100780 2024-02-19T21:17:57.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 >> v1:172.21.15.163:6790/0 conn(0x7fbc600fd980 legacy=0x7fbc600f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:57.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 >> v1:172.21.15.193:6789/0 conn(0x7fbc601042c0 legacy=0x7fbc601159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:57.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6011a910 con 0x7fbc60100780 2024-02-19T21:17:57.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc673ee700 1 -- 172.21.15.163:0/2192947096 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc60119900 con 0x7fbc60100780 2024-02-19T21:17:57.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc673ee700 10 client.23940.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:57.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc673ee700 1 -- 172.21.15.163:0/2192947096 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc60119d70 con 0x7fbc60100780 2024-02-19T21:17:57.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc673ee700 1 librados: init done 2024-02-19T21:17:57.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fbc5c001200 con 0x7fbc60100780 2024-02-19T21:17:57.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.411+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fbc5c0035f0 con 0x7fbc60100780 2024-02-19T21:17:57.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.412+0000 7fbc673ee700 1 -- 172.21.15.163:0/2192947096 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc44004f20 con 0x7fbc60100780 2024-02-19T21:17:57.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.413+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fbc5c001880 con 0x7fbc60100780 2024-02-19T21:17:57.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.413+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(808..808 src has 633..808) v4 ==== 7187+0+0 (unknown 497925489 0 0) 0x7fbc5c0763b0 con 0x7fbc60100780 2024-02-19T21:17:57.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.413+0000 7fbc567fc700 10 client.23940.objecter ms_dispatch 0x7fbc60001550 osd_map(808..808 src has 633..808) v4 2024-02-19T21:17:57.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.413+0000 7fbc567fc700 3 client.23940.objecter handle_osd_map got epochs [808,808] > 0 2024-02-19T21:17:57.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.413+0000 7fbc567fc700 3 client.23940.objecter handle_osd_map decoding full epoch 808 2024-02-19T21:17:57.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.413+0000 7fbc567fc700 20 client.23940.objecter dump_active .. 0 homeless 2024-02-19T21:17:57.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.413+0000 7fbc567fc700 10 client.23940.objecter ms_handle_connect 0x7fbc4c063e20 2024-02-19T21:17:57.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.418+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fbc5c049b70 con 0x7fbc60100780 2024-02-19T21:17:57.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.597+0000 7fbc673ee700 1 -- 172.21.15.163:0/2192947096 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd in", "ids": ["5"]} v 0) v1 -- 0x7fbc440051d0 con 0x7fbc60100780 2024-02-19T21:17:57.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.661+0000 7fbc567fc700 1 -- 172.21.15.163:0/2192947096 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd in", "ids": ["5"]}]=0 marked in osd.5. v809) v1 ==== 85+0+0 (unknown 870512969 0 0) 0x7fbc5c047fe0 con 0x7fbc60100780 2024-02-19T21:17:57.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.667+0000 7fbc673ee700 10 librados: watch_flush enter 2024-02-19T21:17:57.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.668+0000 7fbc673ee700 10 librados: watch_flush exit 2024-02-19T21:17:57.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.668+0000 7fbc673ee700 20 client.23940.objecter shutdown clearing up homeless session... 2024-02-19T21:17:57.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.668+0000 7fbc673ee700 10 client.23940.objecter successfully canceled tick 2024-02-19T21:17:57.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.671+0000 7fbc673ee700 1 -- 172.21.15.163:0/2192947096 >> v1:172.21.15.163:6800/1552514681 conn(0x7fbc4c063e20 legacy=0x7fbc4c0662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:57.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.671+0000 7fbc673ee700 1 -- 172.21.15.163:0/2192947096 >> v1:172.21.15.163:6789/0 conn(0x7fbc60100780 legacy=0x7fbc600fa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:57.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.671+0000 7fbc673ee700 1 -- 172.21.15.163:0/2192947096 shutdown_connections 2024-02-19T21:17:57.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.671+0000 7fbc673ee700 1 -- 172.21.15.163:0/2192947096 >> 172.21.15.163:0/2192947096 conn(0x7fbc600f35a0 msgr2=0x7fbc60107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:57.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.676+0000 7fbc673ee700 1 -- 172.21.15.163:0/2192947096 shutdown_connections 2024-02-19T21:17:57.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.679+0000 7fbc673ee700 1 -- 172.21.15.163:0/2192947096 wait complete. 2024-02-19T21:17:57.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:57.680+0000 7fbc673ee700 1 librados: shutdown 2024-02-19T21:17:57.830 INFO:teuthology.orchestra.run.smithi163.stderr:marked in osd.5. 2024-02-19T21:17:57.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:57.599237+0000 mon.a (mon.0) 3764 : audit [DBG] from='client.? v1:172.21.15.163:0/2192947096' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-02-19T21:17:57.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:17:57.598+0000 7fb606608700 -1 mon.a@0(leader).osd e808 definitely_dead 0 2024-02-19T21:17:57.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:57.599237+0000 mon.a (mon.0) 3764 : audit [DBG] from='client.? v1:172.21.15.163:0/2192947096' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-02-19T21:17:58.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:17:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:17:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:17:58.514 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-02-19T21:17:58.514 INFO:tasks.thrashosds.thrasher:inning osd 2024-02-19T21:17:58.514 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-02-19T21:17:58.515 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd in 7 2024-02-19T21:17:58.648 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:17:58.649 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:17:58.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:57.347100+0000 mgr.y (mgr.14550) 671 : cluster [DBG] pgmap v1228: 174 pgs: 174 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:17:58.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:57.656792+0000 mon.a (mon.0) 3765 : audit [INF] from='client.? v1:172.21.15.163:0/2192947096' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-02-19T21:17:58.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:57.656852+0000 mon.a (mon.0) 3766 : cluster [DBG] osdmap e809: 8 total, 8 up, 7 in 2024-02-19T21:17:58.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:57.347100+0000 mgr.y (mgr.14550) 671 : cluster [DBG] pgmap v1228: 174 pgs: 174 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:17:58.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:57.656792+0000 mon.a (mon.0) 3765 : audit [INF] from='client.? v1:172.21.15.163:0/2192947096' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-02-19T21:17:58.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:57.656852+0000 mon.a (mon.0) 3766 : cluster [DBG] osdmap e809: 8 total, 8 up, 7 in 2024-02-19T21:17:58.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:57.347100+0000 mgr.y (mgr.14550) 671 : cluster [DBG] pgmap v1228: 174 pgs: 174 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:17:58.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:57.656792+0000 mon.a (mon.0) 3765 : audit [INF] from='client.? v1:172.21.15.163:0/2192947096' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-02-19T21:17:58.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:57.656852+0000 mon.a (mon.0) 3766 : cluster [DBG] osdmap e809: 8 total, 8 up, 7 in 2024-02-19T21:17:59.399 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-19T21:17:59.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.514+0000 7fc0507bd700 1 -- 172.21.15.163:0/3502634885 >> v1:172.21.15.163:6789/0 conn(0x7fc0480f9490 legacy=0x7fc048102a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:59.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.514+0000 7fc0507bd700 1 -- 172.21.15.163:0/3502634885 shutdown_connections 2024-02-19T21:17:59.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.514+0000 7fc0507bd700 1 -- 172.21.15.163:0/3502634885 >> 172.21.15.163:0/3502634885 conn(0x7fc0480f2470 msgr2=0x7fc0480f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:17:59.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.514+0000 7fc0507bd700 1 -- 172.21.15.163:0/3502634885 shutdown_connections 2024-02-19T21:17:59.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.514+0000 7fc0507bd700 1 -- 172.21.15.163:0/3502634885 wait complete. 2024-02-19T21:17:59.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.515+0000 7fc0507bd700 1 librados: starting msgr at 2024-02-19T21:17:59.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.515+0000 7fc0507bd700 1 librados: starting objecter 2024-02-19T21:17:59.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.515+0000 7fc0507bd700 1 Processor -- start 2024-02-19T21:17:59.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.515+0000 7fc0507bd700 1 -- start start 2024-02-19T21:17:59.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.515+0000 7fc0507bd700 1 librados: setting wanted keys 2024-02-19T21:17:59.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.515+0000 7fc0507bd700 1 librados: calling monclient init 2024-02-19T21:17:59.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.515+0000 7fc0507bd700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc0481172c0 con 0x7fc0480f89a0 2024-02-19T21:17:59.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.515+0000 7fc0507bd700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc0481184a0 con 0x7fc0480f9490 2024-02-19T21:17:59.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.515+0000 7fc0507bd700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc0481196a0 con 0x7fc0481042a0 2024-02-19T21:17:59.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.515+0000 7fc04dd58700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fc0480f9490 0x7fc0480ffb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:58556/0 (socket says 172.21.15.163:58556) 2024-02-19T21:17:59.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.515+0000 7fc04dd58700 1 -- 172.21.15.163:0/3794916233 learned_addr learned my addr 172.21.15.163:0/3794916233 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:17:59.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.515+0000 7fc04ed5a700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fc0481042a0 0x7fc048115980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:56830/0 (socket says 172.21.15.163:56830) 2024-02-19T21:17:59.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.515+0000 7fc04e559700 1 --1- 172.21.15.163:0/3794916233 >> v1:172.21.15.163:6789/0 conn(0x7fc0480f89a0 0x7fc0480ff480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:52850/0 (socket says 172.21.15.163:52850) 2024-02-19T21:17:59.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 10 client.?.objecter ms_handle_connect 0x7fc0481042a0 2024-02-19T21:17:59.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:59.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 10 client.?.objecter ms_handle_connect 0x7fc0480f9490 2024-02-19T21:17:59.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:59.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 10 client.?.objecter ms_handle_connect 0x7fc0480f89a0 2024-02-19T21:17:59.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:17:59.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847309821 0 0) 0x7fc0481184a0 con 0x7fc0480f9490 2024-02-19T21:17:59.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc034003800 con 0x7fc0480f9490 2024-02-19T21:17:59.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476828045 0 0) 0x7fc0481196a0 con 0x7fc0481042a0 2024-02-19T21:17:59.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0481184a0 con 0x7fc0481042a0 2024-02-19T21:17:59.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893789934 0 0) 0x7fc0481172c0 con 0x7fc0480f89a0 2024-02-19T21:17:59.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0481196a0 con 0x7fc0480f89a0 2024-02-19T21:17:59.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 470367615 0 0) 0x7fc034003800 con 0x7fc0480f9490 2024-02-19T21:17:59.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0481172c0 con 0x7fc0480f9490 2024-02-19T21:17:59.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc038002ea0 con 0x7fc0480f9490 2024-02-19T21:17:59.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3997697755 0 0) 0x7fc0481196a0 con 0x7fc0480f89a0 2024-02-19T21:17:59.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc034003800 con 0x7fc0480f89a0 2024-02-19T21:17:59.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1187767198 0 0) 0x7fc0481184a0 con 0x7fc0481042a0 2024-02-19T21:17:59.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0481196a0 con 0x7fc0481042a0 2024-02-19T21:17:59.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc0440018d0 con 0x7fc0480f89a0 2024-02-19T21:17:59.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc040001bc0 con 0x7fc0481042a0 2024-02-19T21:17:59.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2922271856 0 0) 0x7fc0481172c0 con 0x7fc0480f9490 2024-02-19T21:17:59.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 >> v1:172.21.15.163:6790/0 conn(0x7fc0481042a0 legacy=0x7fc048115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:59.520 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 >> v1:172.21.15.163:6789/0 conn(0x7fc0480f89a0 legacy=0x7fc0480ff480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:17:59.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04811a8a0 con 0x7fc0480f9490 2024-02-19T21:17:59.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc0507bd700 1 -- 172.21.15.163:0/3794916233 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc048118690 con 0x7fc0480f9490 2024-02-19T21:17:59.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc0507bd700 10 client.49483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:17:59.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc0507bd700 1 -- 172.21.15.163:0/3794916233 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc048118b00 con 0x7fc0480f9490 2024-02-19T21:17:59.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc0507bd700 1 librados: init done 2024-02-19T21:17:59.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fc038001fb0 con 0x7fc0480f9490 2024-02-19T21:17:59.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.516+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc038003920 con 0x7fc0480f9490 2024-02-19T21:17:59.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.517+0000 7fc0507bd700 1 -- 172.21.15.163:0/3794916233 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc02c004f20 con 0x7fc0480f9490 2024-02-19T21:17:59.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.518+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fc038003a80 con 0x7fc0480f9490 2024-02-19T21:17:59.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.518+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(810..810 src has 633..810) v4 ==== 7220+0+0 (unknown 1014976769 0 0) 0x7fc038078d20 con 0x7fc0480f9490 2024-02-19T21:17:59.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.518+0000 7fc03f7fe700 10 client.49483.objecter ms_dispatch 0x7fc048001550 osd_map(810..810 src has 633..810) v4 2024-02-19T21:17:59.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.518+0000 7fc03f7fe700 3 client.49483.objecter handle_osd_map got epochs [810,810] > 0 2024-02-19T21:17:59.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.518+0000 7fc03f7fe700 3 client.49483.objecter handle_osd_map decoding full epoch 810 2024-02-19T21:17:59.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.518+0000 7fc03f7fe700 20 client.49483.objecter dump_active .. 0 homeless 2024-02-19T21:17:59.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.519+0000 7fc03f7fe700 10 client.49483.objecter ms_handle_connect 0x7fc034063cc0 2024-02-19T21:17:59.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.527+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc03804c560 con 0x7fc0480f9490 2024-02-19T21:17:59.540 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:17:59.679 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:58.661021+0000 mon.a (mon.0) 3767 : cluster [DBG] osdmap e810: 8 total, 8 up, 7 in 2024-02-19T21:17:59.680 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:17:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:59.659760+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e811: 8 total, 8 up, 7 in 2024-02-19T21:17:59.686 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:17:59.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:17:59.717+0000 7fc0507bd700 1 -- 172.21.15.163:0/3794916233 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) v1 -- 0x7fc02c0051d0 con 0x7fc0480f9490 2024-02-19T21:17:59.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:58.661021+0000 mon.a (mon.0) 3767 : cluster [DBG] osdmap e810: 8 total, 8 up, 7 in 2024-02-19T21:18:00.042 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:17:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:59.659760+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e811: 8 total, 8 up, 7 in 2024-02-19T21:18:00.042 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:58.661021+0000 mon.a (mon.0) 3767 : cluster [DBG] osdmap e810: 8 total, 8 up, 7 in 2024-02-19T21:18:00.043 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:17:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:59.659760+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e811: 8 total, 8 up, 7 in 2024-02-19T21:18:00.693 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:00.692+0000 7fc03f7fe700 1 -- 172.21.15.163:0/3794916233 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v812) v1 ==== 91+0+0 (unknown 2608503075 0 0) 0x7fc03804a9d0 con 0x7fc0480f9490 2024-02-19T21:18:00.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:00.694+0000 7fc0507bd700 10 librados: watch_flush enter 2024-02-19T21:18:00.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:00.694+0000 7fc0507bd700 10 librados: watch_flush exit 2024-02-19T21:18:00.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:00.694+0000 7fc0507bd700 20 client.49483.objecter shutdown clearing up homeless session... 2024-02-19T21:18:00.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:00.694+0000 7fc0507bd700 10 client.49483.objecter successfully canceled tick 2024-02-19T21:18:00.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:00.694+0000 7fc0507bd700 1 -- 172.21.15.163:0/3794916233 >> v1:172.21.15.163:6800/1552514681 conn(0x7fc034063cc0 legacy=0x7fc034066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:00.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:00.694+0000 7fc0507bd700 1 -- 172.21.15.163:0/3794916233 >> v1:172.21.15.193:6789/0 conn(0x7fc0480f9490 legacy=0x7fc0480ffb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:00.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:00.694+0000 7fc0507bd700 1 -- 172.21.15.163:0/3794916233 shutdown_connections 2024-02-19T21:18:00.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:00.694+0000 7fc0507bd700 1 -- 172.21.15.163:0/3794916233 >> 172.21.15.163:0/3794916233 conn(0x7fc0480f2470 msgr2=0x7fc048107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:00.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:00.694+0000 7fc0507bd700 1 -- 172.21.15.163:0/3794916233 shutdown_connections 2024-02-19T21:18:00.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:00.694+0000 7fc0507bd700 1 -- 172.21.15.163:0/3794916233 wait complete. 2024-02-19T21:18:00.696 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:00.695+0000 7fc0507bd700 1 librados: shutdown 2024-02-19T21:18:00.699 INFO:teuthology.orchestra.run.smithi163.stderr:noscrub is unset 2024-02-19T21:18:00.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:59.347654+0000 mgr.y (mgr.14550) 672 : cluster [DBG] pgmap v1231: 174 pgs: 174 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:18:00.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:17:59.658227+0000 osd.0 (osd.0) 137 : cluster [DBG] 2.20 starting backfill to osd.5 from (0'0,0'0] MAX to 397'2 2024-02-19T21:18:00.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:59.717640+0000 mon.b (mon.1) 758 : audit [INF] from='client.? v1:172.21.15.163:0/3794916233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:18:00.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:17:59.717972+0000 mon.a (mon.0) 3769 : audit [INF] from='client.49483 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:18:00.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:00.052811+0000 mon.a (mon.0) 3770 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:18:00.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:59.347654+0000 mgr.y (mgr.14550) 672 : cluster [DBG] pgmap v1231: 174 pgs: 174 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:18:00.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:17:59.658227+0000 osd.0 (osd.0) 137 : cluster [DBG] 2.20 starting backfill to osd.5 from (0'0,0'0] MAX to 397'2 2024-02-19T21:18:00.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:59.717640+0000 mon.b (mon.1) 758 : audit [INF] from='client.? v1:172.21.15.163:0/3794916233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:18:01.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:17:59.717972+0000 mon.a (mon.0) 3769 : audit [INF] from='client.49483 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:18:01.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:00.052811+0000 mon.a (mon.0) 3770 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:18:01.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:59.347654+0000 mgr.y (mgr.14550) 672 : cluster [DBG] pgmap v1231: 174 pgs: 174 active+clean; 5.7 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:18:01.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:17:59.658227+0000 osd.0 (osd.0) 137 : cluster [DBG] 2.20 starting backfill to osd.5 from (0'0,0'0] MAX to 397'2 2024-02-19T21:18:01.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:59.717640+0000 mon.b (mon.1) 758 : audit [INF] from='client.? v1:172.21.15.163:0/3794916233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:18:01.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:17:59.717972+0000 mon.a (mon.0) 3769 : audit [INF] from='client.49483 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-19T21:18:01.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:00.052811+0000 mon.a (mon.0) 3770 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:18:01.123 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:18:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:18:01.328 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-19T21:18:01.332 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:18:01.332 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:18:01.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.936+0000 7f8de1746700 1 -- 172.21.15.163:0/2304567028 >> v1:172.21.15.163:6790/0 conn(0x7f8ddc1042a0 legacy=0x7f8ddc106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:01.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.936+0000 7f8de1746700 1 -- 172.21.15.163:0/2304567028 shutdown_connections 2024-02-19T21:18:01.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.936+0000 7f8de1746700 1 -- 172.21.15.163:0/2304567028 >> 172.21.15.163:0/2304567028 conn(0x7f8ddc0f24b0 msgr2=0x7f8ddc0f48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:01.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.936+0000 7f8de1746700 1 -- 172.21.15.163:0/2304567028 shutdown_connections 2024-02-19T21:18:01.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.936+0000 7f8de1746700 1 -- 172.21.15.163:0/2304567028 wait complete. 2024-02-19T21:18:01.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.937+0000 7f8de1746700 1 librados: starting msgr at 2024-02-19T21:18:01.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.937+0000 7f8de1746700 1 librados: starting objecter 2024-02-19T21:18:01.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.937+0000 7f8de1746700 1 Processor -- start 2024-02-19T21:18:01.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.937+0000 7f8de1746700 1 -- start start 2024-02-19T21:18:01.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.937+0000 7f8de1746700 1 librados: setting wanted keys 2024-02-19T21:18:01.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.937+0000 7f8de1746700 1 librados: calling monclient init 2024-02-19T21:18:01.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8de1746700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8ddc102bb0 con 0x7f8ddc0f9430 2024-02-19T21:18:01.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8de1746700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8ddc102d80 con 0x7f8ddc1042a0 2024-02-19T21:18:01.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8de1746700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8ddc11bb60 con 0x7f8ddc0f8940 2024-02-19T21:18:01.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8ddaffd700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f8ddc0f8940 0x7f8ddc102410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:56842/0 (socket says 172.21.15.163:56842) 2024-02-19T21:18:01.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8ddaffd700 1 -- 172.21.15.163:0/3055338601 learned_addr learned my addr 172.21.15.163:0/3055338601 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:01.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 10 client.?.objecter ms_handle_connect 0x7f8ddc0f8940 2024-02-19T21:18:01.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:01.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120467677 0 0) 0x7f8ddc11bb60 con 0x7f8ddc0f8940 2024-02-19T21:18:01.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dc40033f0 con 0x7f8ddc0f8940 2024-02-19T21:18:01.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 10 client.?.objecter ms_handle_connect 0x7f8ddc0f9430 2024-02-19T21:18:01.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:01.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 10 client.?.objecter ms_handle_connect 0x7f8ddc1042a0 2024-02-19T21:18:01.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:01.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538159787 0 0) 0x7f8ddc102bb0 con 0x7f8ddc0f9430 2024-02-19T21:18:01.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ddc11bb60 con 0x7f8ddc0f9430 2024-02-19T21:18:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091358728 0 0) 0x7f8ddc102d80 con 0x7f8ddc1042a0 2024-02-19T21:18:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ddc102bb0 con 0x7f8ddc1042a0 2024-02-19T21:18:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3424137593 0 0) 0x7f8dc40033f0 con 0x7f8ddc0f8940 2024-02-19T21:18:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ddc102d80 con 0x7f8ddc0f8940 2024-02-19T21:18:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8dcc0029d0 con 0x7f8ddc0f8940 2024-02-19T21:18:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1385848888 0 0) 0x7f8ddc102bb0 con 0x7f8ddc1042a0 2024-02-19T21:18:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8dc40033f0 con 0x7f8ddc1042a0 2024-02-19T21:18:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8dd4003280 con 0x7f8ddc1042a0 2024-02-19T21:18:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1622109766 0 0) 0x7f8ddc11bb60 con 0x7f8ddc0f9430 2024-02-19T21:18:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ddc102bb0 con 0x7f8ddc0f9430 2024-02-19T21:18:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8dd0002740 con 0x7f8ddc0f9430 2024-02-19T21:18:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4055570739 0 0) 0x7f8ddc102d80 con 0x7f8ddc0f8940 2024-02-19T21:18:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 >> v1:172.21.15.193:6789/0 conn(0x7f8ddc1042a0 legacy=0x7f8ddc118420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 >> v1:172.21.15.163:6789/0 conn(0x7f8ddc0f9430 legacy=0x7f8ddc114cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:01.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.938+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc11cd00 con 0x7f8ddc0f8940 2024-02-19T21:18:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.939+0000 7f8de1746700 1 -- 172.21.15.163:0/3055338601 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ddc11bd50 con 0x7f8ddc0f8940 2024-02-19T21:18:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.939+0000 7f8de1746700 10 client.50678.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.939+0000 7f8de1746700 1 -- 172.21.15.163:0/3055338601 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ddc11c1c0 con 0x7f8ddc0f8940 2024-02-19T21:18:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.939+0000 7f8de1746700 1 librados: init done 2024-02-19T21:18:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.939+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8dcc002da0 con 0x7f8ddc0f8940 2024-02-19T21:18:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.939+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8dcc005c30 con 0x7f8ddc0f8940 2024-02-19T21:18:01.944 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.940+0000 7f8de1746700 1 -- 172.21.15.163:0/3055338601 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc8004f20 con 0x7f8ddc0f8940 2024-02-19T21:18:01.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.940+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f8dcc018b60 con 0x7f8ddc0f8940 2024-02-19T21:18:01.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.941+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(813..813 src has 633..813) v4 ==== 7187+0+0 (unknown 3687481556 0 0) 0x7f8dcc078a90 con 0x7f8ddc0f8940 2024-02-19T21:18:01.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.941+0000 7f8dc3fff700 10 client.50678.objecter ms_dispatch 0x7f8ddc001550 osd_map(813..813 src has 633..813) v4 2024-02-19T21:18:01.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.941+0000 7f8dc3fff700 3 client.50678.objecter handle_osd_map got epochs [813,813] > 0 2024-02-19T21:18:01.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.941+0000 7f8dc3fff700 3 client.50678.objecter handle_osd_map decoding full epoch 813 2024-02-19T21:18:01.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.941+0000 7f8dc3fff700 20 client.50678.objecter dump_active .. 0 homeless 2024-02-19T21:18:01.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.941+0000 7f8dc3fff700 10 client.50678.objecter ms_handle_connect 0x7f8dc4063c10 2024-02-19T21:18:01.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:01.945+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8dcc04c3a0 con 0x7f8ddc0f8940 2024-02-19T21:18:01.965 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:00.677477+0000 mon.a (mon.0) 3771 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-02-19T21:18:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:00.689435+0000 mon.a (mon.0) 3772 : audit [INF] from='client.49483 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:18:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:00.689530+0000 mon.a (mon.0) 3773 : cluster [DBG] osdmap e812: 8 total, 8 up, 7 in 2024-02-19T21:18:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:00.843477+0000 mgr.y (mgr.14550) 673 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:01.207129+0000 osd.3 (osd.3) 341 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:18:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:01.208181+0000 osd.3 (osd.3) 342 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:18:01.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:00.677477+0000 mon.a (mon.0) 3771 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-02-19T21:18:01.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:00.689435+0000 mon.a (mon.0) 3772 : audit [INF] from='client.49483 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:18:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:00.689530+0000 mon.a (mon.0) 3773 : cluster [DBG] osdmap e812: 8 total, 8 up, 7 in 2024-02-19T21:18:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:00.843477+0000 mgr.y (mgr.14550) 673 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:01.207129+0000 osd.3 (osd.3) 341 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:18:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:01.208181+0000 osd.3 (osd.3) 342 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:18:02.039 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:00.677477+0000 mon.a (mon.0) 3771 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-02-19T21:18:02.040 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:00.689435+0000 mon.a (mon.0) 3772 : audit [INF] from='client.49483 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-19T21:18:02.040 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:00.689530+0000 mon.a (mon.0) 3773 : cluster [DBG] osdmap e812: 8 total, 8 up, 7 in 2024-02-19T21:18:02.040 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:00.843477+0000 mgr.y (mgr.14550) 673 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:02.040 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:01.207129+0000 osd.3 (osd.3) 341 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:18:02.040 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:01.208181+0000 osd.3 (osd.3) 342 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:18:02.057 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-19T21:18:02.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:02.126+0000 7f8de1746700 1 -- 172.21.15.163:0/3055338601 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd in", "ids": ["7"]} v 0) v1 -- 0x7f8dc80051d0 con 0x7f8ddc0f8940 2024-02-19T21:18:02.361 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:18:02.453 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:18:02.127+0000 7fb606608700 -1 mon.a@0(leader).osd e813 definitely_dead 0 2024-02-19T21:18:02.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:02.703+0000 7f8dc3fff700 1 -- 172.21.15.163:0/3055338601 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd in", "ids": ["7"]}]=0 marked in osd.7. v814) v1 ==== 85+0+0 (unknown 3939476133 0 0) 0x7f8dcc04a7b0 con 0x7f8ddc0f8940 2024-02-19T21:18:02.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:02.706+0000 7f8de1746700 10 librados: watch_flush enter 2024-02-19T21:18:02.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:02.709+0000 7f8de1746700 10 librados: watch_flush exit 2024-02-19T21:18:02.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:02.709+0000 7f8de1746700 20 client.50678.objecter shutdown clearing up homeless session... 2024-02-19T21:18:02.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:02.709+0000 7f8de1746700 10 client.50678.objecter successfully canceled tick 2024-02-19T21:18:02.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:02.709+0000 7f8de1746700 1 -- 172.21.15.163:0/3055338601 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8dc4063c10 legacy=0x7f8dc40660d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:02.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:02.710+0000 7f8de1746700 1 -- 172.21.15.163:0/3055338601 >> v1:172.21.15.163:6790/0 conn(0x7f8ddc0f8940 legacy=0x7f8ddc102410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:02.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:02.710+0000 7f8de1746700 1 -- 172.21.15.163:0/3055338601 shutdown_connections 2024-02-19T21:18:02.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:02.710+0000 7f8de1746700 1 -- 172.21.15.163:0/3055338601 >> 172.21.15.163:0/3055338601 conn(0x7f8ddc0f24b0 msgr2=0x7f8ddc107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:02.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:02.710+0000 7f8de1746700 1 -- 172.21.15.163:0/3055338601 shutdown_connections 2024-02-19T21:18:02.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:02.710+0000 7f8de1746700 1 -- 172.21.15.163:0/3055338601 wait complete. 2024-02-19T21:18:02.717 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:02.710+0000 7f8de1746700 1 librados: shutdown 2024-02-19T21:18:02.718 INFO:teuthology.orchestra.run.smithi163.stderr:marked in osd.7. 2024-02-19T21:18:02.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:01.348057+0000 mgr.y (mgr.14550) 674 : cluster [DBG] pgmap v1234: 174 pgs: 1 active+recovering, 1 active+clean+remapped, 172 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 422 B/s, 7 objects/s recovering 2024-02-19T21:18:02.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:01.695324+0000 mon.a (mon.0) 3774 : cluster [DBG] osdmap e813: 8 total, 8 up, 7 in 2024-02-19T21:18:02.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:02.127468+0000 mon.c (mon.2) 444 : audit [DBG] from='client.? v1:172.21.15.163:0/3055338601' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-19T21:18:02.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:02.127812+0000 mon.a (mon.0) 3775 : audit [DBG] from='client.50678 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-19T21:18:02.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:02.249640+0000 osd.4 (osd.4) 283 : cluster [DBG] 2.f scrub starts 2024-02-19T21:18:02.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:02.251270+0000 osd.4 (osd.4) 284 : cluster [DBG] 2.f scrub ok 2024-02-19T21:18:02.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:01.348057+0000 mgr.y (mgr.14550) 674 : cluster [DBG] pgmap v1234: 174 pgs: 1 active+recovering, 1 active+clean+remapped, 172 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 422 B/s, 7 objects/s recovering 2024-02-19T21:18:02.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:01.695324+0000 mon.a (mon.0) 3774 : cluster [DBG] osdmap e813: 8 total, 8 up, 7 in 2024-02-19T21:18:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:02.127468+0000 mon.c (mon.2) 444 : audit [DBG] from='client.? v1:172.21.15.163:0/3055338601' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-19T21:18:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:02.127812+0000 mon.a (mon.0) 3775 : audit [DBG] from='client.50678 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-19T21:18:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:02.249640+0000 osd.4 (osd.4) 283 : cluster [DBG] 2.f scrub starts 2024-02-19T21:18:02.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:02.251270+0000 osd.4 (osd.4) 284 : cluster [DBG] 2.f scrub ok 2024-02-19T21:18:03.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:01.348057+0000 mgr.y (mgr.14550) 674 : cluster [DBG] pgmap v1234: 174 pgs: 1 active+recovering, 1 active+clean+remapped, 172 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 422 B/s, 7 objects/s recovering 2024-02-19T21:18:03.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:01.695324+0000 mon.a (mon.0) 3774 : cluster [DBG] osdmap e813: 8 total, 8 up, 7 in 2024-02-19T21:18:03.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:02.127468+0000 mon.c (mon.2) 444 : audit [DBG] from='client.? v1:172.21.15.163:0/3055338601' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-19T21:18:03.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:02.127812+0000 mon.a (mon.0) 3775 : audit [DBG] from='client.50678 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-19T21:18:03.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:02.249640+0000 osd.4 (osd.4) 283 : cluster [DBG] 2.f scrub starts 2024-02-19T21:18:03.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:02.251270+0000 osd.4 (osd.4) 284 : cluster [DBG] 2.f scrub ok 2024-02-19T21:18:03.429 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-02-19T21:18:03.429 INFO:tasks.thrashosds.thrasher:fixing pg num pool .rgw.root 2024-02-19T21:18:03.430 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd pool set .rgw.root pgp_num 42 2024-02-19T21:18:03.705 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:18:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:18:03] "GET /metrics HTTP/1.1" 200 187969 "" "Prometheus/2.33.4" 2024-02-19T21:18:03.756 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:02.699422+0000 mon.a (mon.0) 3776 : audit [INF] from='client.50678 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-02-19T21:18:03.756 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:02.699503+0000 mon.a (mon.0) 3777 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2024-02-19T21:18:03.756 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:03.260602+0000 osd.5 (osd.5) 319 : cluster [DBG] 16.c scrub starts 2024-02-19T21:18:03.756 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:03.263519+0000 osd.5 (osd.5) 320 : cluster [DBG] 16.c scrub ok 2024-02-19T21:18:03.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:02.699422+0000 mon.a (mon.0) 3776 : audit [INF] from='client.50678 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-02-19T21:18:03.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:02.699503+0000 mon.a (mon.0) 3777 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2024-02-19T21:18:03.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:03.260602+0000 osd.5 (osd.5) 319 : cluster [DBG] 16.c scrub starts 2024-02-19T21:18:03.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:03.263519+0000 osd.5 (osd.5) 320 : cluster [DBG] 16.c scrub ok 2024-02-19T21:18:03.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:02.699422+0000 mon.a (mon.0) 3776 : audit [INF] from='client.50678 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-02-19T21:18:03.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:02.699503+0000 mon.a (mon.0) 3777 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2024-02-19T21:18:04.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:03.260602+0000 osd.5 (osd.5) 319 : cluster [DBG] 16.c scrub starts 2024-02-19T21:18:04.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:03.263519+0000 osd.5 (osd.5) 320 : cluster [DBG] 16.c scrub ok 2024-02-19T21:18:04.092 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:18:04.092 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:18:04.370 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:04.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.382+0000 7fa637964700 1 -- 172.21.15.163:0/3689219129 >> v1:172.21.15.163:6789/0 conn(0x7fa6300f8920 legacy=0x7fa6300f8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:04.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.382+0000 7fa637964700 1 -- 172.21.15.163:0/3689219129 shutdown_connections 2024-02-19T21:18:04.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.382+0000 7fa637964700 1 -- 172.21.15.163:0/3689219129 >> 172.21.15.163:0/3689219129 conn(0x7fa6300f2470 msgr2=0x7fa6300f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:04.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.382+0000 7fa637964700 1 -- 172.21.15.163:0/3689219129 shutdown_connections 2024-02-19T21:18:04.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.382+0000 7fa637964700 1 -- 172.21.15.163:0/3689219129 wait complete. 2024-02-19T21:18:04.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.383+0000 7fa637964700 1 librados: starting msgr at 2024-02-19T21:18:04.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.383+0000 7fa637964700 1 librados: starting objecter 2024-02-19T21:18:04.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.383+0000 7fa637964700 1 Processor -- start 2024-02-19T21:18:04.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.383+0000 7fa637964700 1 -- start start 2024-02-19T21:18:04.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.383+0000 7fa637964700 1 librados: setting wanted keys 2024-02-19T21:18:04.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.383+0000 7fa637964700 1 librados: calling monclient init 2024-02-19T21:18:04.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.383+0000 7fa637964700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa63010f8f0 con 0x7fa6300f8920 2024-02-19T21:18:04.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.383+0000 7fa637964700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa63010fac0 con 0x7fa6300f94d0 2024-02-19T21:18:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.383+0000 7fa637964700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa63011fc40 con 0x7fa6301042b0 2024-02-19T21:18:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.383+0000 7fa635700700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fa6300f8920 0x7fa630102200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:56032/0 (socket says 172.21.15.163:56032) 2024-02-19T21:18:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.383+0000 7fa635700700 1 -- 172.21.15.163:0/1274968692 learned_addr learned my addr 172.21.15.163:0/1274968692 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.383+0000 7fa635f01700 1 --1- 172.21.15.163:0/1274968692 >> v1:172.21.15.163:6790/0 conn(0x7fa6301042b0 0x7fa63011c500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38780/0 (socket says 172.21.15.163:38780) 2024-02-19T21:18:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 10 client.?.objecter ms_handle_connect 0x7fa6300f8920 2024-02-19T21:18:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 10 client.?.objecter ms_handle_connect 0x7fa6301042b0 2024-02-19T21:18:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 10 client.?.objecter ms_handle_connect 0x7fa6300f94d0 2024-02-19T21:18:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813328949 0 0) 0x7fa63010f8f0 con 0x7fa6300f8920 2024-02-19T21:18:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa61c0033f0 con 0x7fa6300f8920 2024-02-19T21:18:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102784645 0 0) 0x7fa63010fac0 con 0x7fa6300f94d0 2024-02-19T21:18:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa63010f8f0 con 0x7fa6300f94d0 2024-02-19T21:18:04.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412268706 0 0) 0x7fa63011fc40 con 0x7fa6301042b0 2024-02-19T21:18:04.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa63010fac0 con 0x7fa6301042b0 2024-02-19T21:18:04.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2773041674 0 0) 0x7fa61c0033f0 con 0x7fa6300f8920 2024-02-19T21:18:04.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa63011fc40 con 0x7fa6300f8920 2024-02-19T21:18:04.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa62c001440 con 0x7fa6300f8920 2024-02-19T21:18:04.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1436629562 0 0) 0x7fa63010f8f0 con 0x7fa6300f94d0 2024-02-19T21:18:04.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa61c0033f0 con 0x7fa6300f94d0 2024-02-19T21:18:04.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa6200019d0 con 0x7fa6300f94d0 2024-02-19T21:18:04.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2184175538 0 0) 0x7fa63011fc40 con 0x7fa6300f8920 2024-02-19T21:18:04.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 >> v1:172.21.15.163:6790/0 conn(0x7fa6301042b0 legacy=0x7fa63011c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:04.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 >> v1:172.21.15.193:6789/0 conn(0x7fa6300f94d0 legacy=0x7fa630102910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:04.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa630120e20 con 0x7fa6300f8920 2024-02-19T21:18:04.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa637964700 1 -- 172.21.15.163:0/1274968692 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa63011fe30 con 0x7fa6300f8920 2024-02-19T21:18:04.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa637964700 10 client.23958.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:04.388 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa637964700 1 -- 172.21.15.163:0/1274968692 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6301202a0 con 0x7fa6300f8920 2024-02-19T21:18:04.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.384+0000 7fa637964700 1 librados: init done 2024-02-19T21:18:04.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.385+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fa62c0019e0 con 0x7fa6300f8920 2024-02-19T21:18:04.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.385+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fa62c0036c0 con 0x7fa6300f8920 2024-02-19T21:18:04.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.386+0000 7fa637964700 1 -- 172.21.15.163:0/1274968692 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa614004f20 con 0x7fa6300f8920 2024-02-19T21:18:04.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.386+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fa62c003840 con 0x7fa6300f8920 2024-02-19T21:18:04.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.386+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(815..815 src has 633..815) v4 ==== 7253+0+0 (unknown 557547220 0 0) 0x7fa62c078c60 con 0x7fa6300f8920 2024-02-19T21:18:04.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.386+0000 7fa6267fc700 10 client.23958.objecter ms_dispatch 0x7fa630001550 osd_map(815..815 src has 633..815) v4 2024-02-19T21:18:04.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.386+0000 7fa6267fc700 3 client.23958.objecter handle_osd_map got epochs [815,815] > 0 2024-02-19T21:18:04.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.386+0000 7fa6267fc700 3 client.23958.objecter handle_osd_map decoding full epoch 815 2024-02-19T21:18:04.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.387+0000 7fa6267fc700 20 client.23958.objecter dump_active .. 0 homeless 2024-02-19T21:18:04.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.387+0000 7fa6267fc700 10 client.23958.objecter ms_handle_connect 0x7fa61c063c80 2024-02-19T21:18:04.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.391+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fa62c04c4a0 con 0x7fa6300f8920 2024-02-19T21:18:04.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:04.602+0000 7fa637964700 1 -- 172.21.15.163:0/1274968692 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7fa6140051d0 con 0x7fa6300f8920 2024-02-19T21:18:04.722 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-19T21:18:04.882 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:03.348491+0000 mgr.y (mgr.14550) 675 : cluster [DBG] pgmap v1237: 174 pgs: 1 active+recovering, 1 active+clean+remapped, 172 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 422 B/s, 7 objects/s recovering 2024-02-19T21:18:04.883 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:03.497945+0000 osd.2 (osd.2) 326 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:18:04.883 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:03.500833+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:18:04.883 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:03.706207+0000 mon.a (mon.0) 3778 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2024-02-19T21:18:04.883 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:04.432330+0000 osd.1 (osd.1) 364 : cluster [DBG] 2.0 starting backfill to osd.7 from (0'0,0'0] MAX to 397'2 2024-02-19T21:18:04.883 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:04.433137+0000 mon.a (mon.0) 3779 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2024-02-19T21:18:04.883 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:04.433785+0000 osd.5 (osd.5) 321 : cluster [DBG] 2.20 starting backfill to osd.7 from (0'0,0'0] MAX to 397'2 2024-02-19T21:18:04.883 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:04.603447+0000 mon.a (mon.0) 3780 : audit [INF] from='client.? v1:172.21.15.163:0/1274968692' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:18:04.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:03.348491+0000 mgr.y (mgr.14550) 675 : cluster [DBG] pgmap v1237: 174 pgs: 1 active+recovering, 1 active+clean+remapped, 172 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 422 B/s, 7 objects/s recovering 2024-02-19T21:18:04.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:03.497945+0000 osd.2 (osd.2) 326 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:18:04.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:03.500833+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:18:04.883 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:03.706207+0000 mon.a (mon.0) 3778 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2024-02-19T21:18:04.884 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:04.432330+0000 osd.1 (osd.1) 364 : cluster [DBG] 2.0 starting backfill to osd.7 from (0'0,0'0] MAX to 397'2 2024-02-19T21:18:04.884 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:04.433137+0000 mon.a (mon.0) 3779 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2024-02-19T21:18:04.884 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:04.433785+0000 osd.5 (osd.5) 321 : cluster [DBG] 2.20 starting backfill to osd.7 from (0'0,0'0] MAX to 397'2 2024-02-19T21:18:04.884 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:04.603447+0000 mon.a (mon.0) 3780 : audit [INF] from='client.? v1:172.21.15.163:0/1274968692' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:18:04.995 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:18:05.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:03.348491+0000 mgr.y (mgr.14550) 675 : cluster [DBG] pgmap v1237: 174 pgs: 1 active+recovering, 1 active+clean+remapped, 172 active+clean; 5.7 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 422 B/s, 7 objects/s recovering 2024-02-19T21:18:05.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:03.497945+0000 osd.2 (osd.2) 326 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:18:05.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:03.500833+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:18:05.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:03.706207+0000 mon.a (mon.0) 3778 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2024-02-19T21:18:05.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:04.432330+0000 osd.1 (osd.1) 364 : cluster [DBG] 2.0 starting backfill to osd.7 from (0'0,0'0] MAX to 397'2 2024-02-19T21:18:05.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:04.433137+0000 mon.a (mon.0) 3779 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2024-02-19T21:18:05.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:04.433785+0000 osd.5 (osd.5) 321 : cluster [DBG] 2.20 starting backfill to osd.7 from (0'0,0'0] MAX to 397'2 2024-02-19T21:18:05.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:04.603447+0000 mon.a (mon.0) 3780 : audit [INF] from='client.? v1:172.21.15.163:0/1274968692' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-19T21:18:05.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:05.443+0000 7fa6267fc700 1 -- 172.21.15.163:0/1274968692 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v817) v1 ==== 101+0+0 (unknown 2959041364 0 0) 0x7fa62c04c060 con 0x7fa6300f8920 2024-02-19T21:18:05.446 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:05.445+0000 7fa637964700 10 librados: watch_flush enter 2024-02-19T21:18:05.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:05.446+0000 7fa637964700 10 librados: watch_flush exit 2024-02-19T21:18:05.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:05.446+0000 7fa637964700 20 client.23958.objecter shutdown clearing up homeless session... 2024-02-19T21:18:05.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:05.446+0000 7fa637964700 10 client.23958.objecter successfully canceled tick 2024-02-19T21:18:05.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:05.446+0000 7fa637964700 1 -- 172.21.15.163:0/1274968692 >> v1:172.21.15.163:6800/1552514681 conn(0x7fa61c063c80 legacy=0x7fa61c066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:05.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:05.446+0000 7fa637964700 1 -- 172.21.15.163:0/1274968692 >> v1:172.21.15.163:6789/0 conn(0x7fa6300f8920 legacy=0x7fa630102200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:05.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:05.446+0000 7fa637964700 1 -- 172.21.15.163:0/1274968692 shutdown_connections 2024-02-19T21:18:05.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:05.446+0000 7fa637964700 1 -- 172.21.15.163:0/1274968692 >> 172.21.15.163:0/1274968692 conn(0x7fa6300f2470 msgr2=0x7fa630107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:05.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:05.446+0000 7fa637964700 1 -- 172.21.15.163:0/1274968692 shutdown_connections 2024-02-19T21:18:05.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:05.446+0000 7fa637964700 1 -- 172.21.15.163:0/1274968692 wait complete. 2024-02-19T21:18:05.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:05.446+0000 7fa637964700 1 librados: shutdown 2024-02-19T21:18:05.448 INFO:teuthology.orchestra.run.smithi163.stderr:nodeep-scrub is unset 2024-02-19T21:18:05.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:05.237288+0000 osd.3 (osd.3) 343 : cluster [DBG] 4.14 deep-scrub starts 2024-02-19T21:18:05.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:05.240806+0000 osd.3 (osd.3) 344 : cluster [DBG] 4.14 deep-scrub ok 2024-02-19T21:18:05.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:05.440475+0000 mon.a (mon.0) 3781 : audit [INF] from='client.? v1:172.21.15.163:0/1274968692' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:18:05.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:05.440573+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2024-02-19T21:18:05.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:05.237288+0000 osd.3 (osd.3) 343 : cluster [DBG] 4.14 deep-scrub starts 2024-02-19T21:18:05.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:05.240806+0000 osd.3 (osd.3) 344 : cluster [DBG] 4.14 deep-scrub ok 2024-02-19T21:18:06.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:05.440475+0000 mon.a (mon.0) 3781 : audit [INF] from='client.? v1:172.21.15.163:0/1274968692' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:18:06.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:05.440573+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2024-02-19T21:18:06.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:05.237288+0000 osd.3 (osd.3) 343 : cluster [DBG] 4.14 deep-scrub starts 2024-02-19T21:18:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:05.240806+0000 osd.3 (osd.3) 344 : cluster [DBG] 4.14 deep-scrub ok 2024-02-19T21:18:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:05.440475+0000 mon.a (mon.0) 3781 : audit [INF] from='client.? v1:172.21.15.163:0/1274968692' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-19T21:18:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:05.440573+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2024-02-19T21:18:06.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.859+0000 7f3618629700 1 -- 172.21.15.163:0/1167249829 >> v1:172.21.15.163:6789/0 conn(0x7f36100fc870 legacy=0x7f36100fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:06.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.860+0000 7f3618629700 1 -- 172.21.15.163:0/1167249829 shutdown_connections 2024-02-19T21:18:06.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.860+0000 7f3618629700 1 -- 172.21.15.163:0/1167249829 >> 172.21.15.163:0/1167249829 conn(0x7f36100f2470 msgr2=0x7f36100f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:06.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.860+0000 7f3618629700 1 -- 172.21.15.163:0/1167249829 shutdown_connections 2024-02-19T21:18:06.864 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.860+0000 7f3618629700 1 -- 172.21.15.163:0/1167249829 wait complete. 2024-02-19T21:18:06.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.860+0000 7f3618629700 1 librados: starting msgr at 2024-02-19T21:18:06.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.860+0000 7f3618629700 1 librados: starting objecter 2024-02-19T21:18:06.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.860+0000 7f3618629700 1 Processor -- start 2024-02-19T21:18:06.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.860+0000 7f3618629700 1 -- start start 2024-02-19T21:18:06.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.860+0000 7f3618629700 1 librados: setting wanted keys 2024-02-19T21:18:06.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.860+0000 7f3618629700 1 librados: calling monclient init 2024-02-19T21:18:06.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.860+0000 7f3618629700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f36100fbe70 con 0x7f36101031b0 2024-02-19T21:18:06.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.860+0000 7f3618629700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f36100fc040 con 0x7f36100ff670 2024-02-19T21:18:06.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.860+0000 7f3618629700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f36100fc230 con 0x7f36100fc870 2024-02-19T21:18:06.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36163c5700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f36100fc870 0x7f36100fb6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38802/0 (socket says 172.21.15.163:38802) 2024-02-19T21:18:06.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36163c5700 1 -- 172.21.15.163:0/3939355146 learned_addr learned my addr 172.21.15.163:0/3939355146 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:06.865 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f3616bc6700 1 --1- 172.21.15.163:0/3939355146 >> v1:172.21.15.163:6789/0 conn(0x7f36101031b0 0x7f3610118340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:56052/0 (socket says 172.21.15.163:56052) 2024-02-19T21:18:06.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 10 client.?.objecter ms_handle_connect 0x7f36100fc870 2024-02-19T21:18:06.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:06.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 10 client.?.objecter ms_handle_connect 0x7f36101031b0 2024-02-19T21:18:06.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:06.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 10 client.?.objecter ms_handle_connect 0x7f36100ff670 2024-02-19T21:18:06.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:06.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941355285 0 0) 0x7f36100fc040 con 0x7f36100ff670 2024-02-19T21:18:06.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35fc0033f0 con 0x7f36100ff670 2024-02-19T21:18:06.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715207267 0 0) 0x7f36100fc230 con 0x7f36100fc870 2024-02-19T21:18:06.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36100fc040 con 0x7f36100fc870 2024-02-19T21:18:06.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832441923 0 0) 0x7f36100fbe70 con 0x7f36101031b0 2024-02-19T21:18:06.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36100fc230 con 0x7f36101031b0 2024-02-19T21:18:06.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3769454214 0 0) 0x7f35fc0033f0 con 0x7f36100ff670 2024-02-19T21:18:06.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36100fbe70 con 0x7f36100ff670 2024-02-19T21:18:06.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f360c0019d0 con 0x7f36100ff670 2024-02-19T21:18:06.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1021019211 0 0) 0x7f36100fc230 con 0x7f36101031b0 2024-02-19T21:18:06.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35fc0033f0 con 0x7f36101031b0 2024-02-19T21:18:06.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2417959629 0 0) 0x7f36100fc040 con 0x7f36100fc870 2024-02-19T21:18:06.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36100fc230 con 0x7f36100fc870 2024-02-19T21:18:06.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3608001b80 con 0x7f36101031b0 2024-02-19T21:18:06.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3604002df0 con 0x7f36100fc870 2024-02-19T21:18:06.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3425291109 0 0) 0x7f36100fbe70 con 0x7f36100ff670 2024-02-19T21:18:06.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 >> v1:172.21.15.163:6790/0 conn(0x7f36100fc870 legacy=0x7f36100fb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:06.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 >> v1:172.21.15.163:6789/0 conn(0x7f36101031b0 legacy=0x7f3610118340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:06.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361011ca90 con 0x7f36100ff670 2024-02-19T21:18:06.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f3618629700 1 -- 172.21.15.163:0/3939355146 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3610119ac0 con 0x7f36100ff670 2024-02-19T21:18:06.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f3618629700 10 client.49501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:06.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f3618629700 1 -- 172.21.15.163:0/3939355146 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3610119fb0 con 0x7f36100ff670 2024-02-19T21:18:06.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f3618629700 1 librados: init done 2024-02-19T21:18:06.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f360c0023a0 con 0x7f36100ff670 2024-02-19T21:18:06.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.861+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f360c0032d0 con 0x7f36100ff670 2024-02-19T21:18:06.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.862+0000 7f3618629700 1 -- 172.21.15.163:0/3939355146 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f4004f20 con 0x7f36100ff670 2024-02-19T21:18:06.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.863+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f360c001f00 con 0x7f36100ff670 2024-02-19T21:18:06.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.863+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(818..818 src has 633..818) v4 ==== 7187+0+0 (unknown 2281346150 0 0) 0x7f360c076190 con 0x7f36100ff670 2024-02-19T21:18:06.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.863+0000 7f36037fe700 10 client.49501.objecter ms_dispatch 0x7f3610001550 osd_map(818..818 src has 633..818) v4 2024-02-19T21:18:06.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.863+0000 7f36037fe700 3 client.49501.objecter handle_osd_map got epochs [818,818] > 0 2024-02-19T21:18:06.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.863+0000 7f36037fe700 3 client.49501.objecter handle_osd_map decoding full epoch 818 2024-02-19T21:18:06.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.864+0000 7f36037fe700 20 client.49501.objecter dump_active .. 0 homeless 2024-02-19T21:18:06.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.864+0000 7f36037fe700 10 client.49501.objecter ms_handle_connect 0x7f35fc063d70 2024-02-19T21:18:06.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:06.868+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f360c0498f0 con 0x7f36100ff670 2024-02-19T21:18:06.908 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:18:06.908 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:18:06.937 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:05.349335+0000 mgr.y (mgr.14550) 676 : cluster [DBG] pgmap v1240: 174 pgs: 2 active+clean+remapped, 172 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 739 B/s, 18 objects/s recovering 2024-02-19T21:18:06.937 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:06.293667+0000 osd.7 (osd.7) 337 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:18:06.937 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:06.296575+0000 osd.7 (osd.7) 338 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:18:06.937 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:06.440849+0000 mon.a (mon.0) 3783 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2024-02-19T21:18:06.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:05.349335+0000 mgr.y (mgr.14550) 676 : cluster [DBG] pgmap v1240: 174 pgs: 2 active+clean+remapped, 172 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 739 B/s, 18 objects/s recovering 2024-02-19T21:18:06.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:06.293667+0000 osd.7 (osd.7) 337 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:18:06.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:06.296575+0000 osd.7 (osd.7) 338 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:18:06.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:06.440849+0000 mon.a (mon.0) 3783 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2024-02-19T21:18:06.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:05.349335+0000 mgr.y (mgr.14550) 676 : cluster [DBG] pgmap v1240: 174 pgs: 2 active+clean+remapped, 172 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 739 B/s, 18 objects/s recovering 2024-02-19T21:18:06.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:06.293667+0000 osd.7 (osd.7) 337 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:18:06.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:06.296575+0000 osd.7 (osd.7) 338 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:18:07.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:06.440849+0000 mon.a (mon.0) 3783 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2024-02-19T21:18:07.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:07.030+0000 7f3618629700 1 -- 172.21.15.163:0/3939355146 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"} v 0) v1 -- 0x7f35f40051d0 con 0x7f36100ff670 2024-02-19T21:18:07.611 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-19T21:18:07.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:07.722+0000 7f36037fe700 1 -- 172.21.15.163:0/3939355146 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]=0 set pool 2 pgp_num to 42 v819) v1 ==== 136+0+0 (unknown 2002098527 0 0) 0x7f360c047d60 con 0x7f36100ff670 2024-02-19T21:18:07.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:07.724+0000 7f3618629700 10 librados: watch_flush enter 2024-02-19T21:18:07.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:07.724+0000 7f3618629700 10 librados: watch_flush exit 2024-02-19T21:18:07.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:07.724+0000 7f3618629700 20 client.49501.objecter shutdown clearing up homeless session... 2024-02-19T21:18:07.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:07.724+0000 7f3618629700 10 client.49501.objecter successfully canceled tick 2024-02-19T21:18:07.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:07.724+0000 7f3618629700 1 -- 172.21.15.163:0/3939355146 >> v1:172.21.15.163:6800/1552514681 conn(0x7f35fc063d70 legacy=0x7f35fc066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:07.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:07.724+0000 7f3618629700 1 -- 172.21.15.163:0/3939355146 >> v1:172.21.15.193:6789/0 conn(0x7f36100ff670 legacy=0x7f3610114c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:07.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:07.724+0000 7f3618629700 1 -- 172.21.15.163:0/3939355146 shutdown_connections 2024-02-19T21:18:07.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:07.724+0000 7f3618629700 1 -- 172.21.15.163:0/3939355146 >> 172.21.15.163:0/3939355146 conn(0x7f36100f2470 msgr2=0x7f3610106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:07.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:07.724+0000 7f3618629700 1 -- 172.21.15.163:0/3939355146 shutdown_connections 2024-02-19T21:18:07.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:07.724+0000 7f3618629700 1 -- 172.21.15.163:0/3939355146 wait complete. 2024-02-19T21:18:07.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:07.724+0000 7f3618629700 1 librados: shutdown 2024-02-19T21:18:07.726 INFO:teuthology.orchestra.run.smithi163.stderr:set pool 2 pgp_num to 42 2024-02-19T21:18:07.884 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:18:07.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:07.031135+0000 mon.b (mon.1) 759 : audit [INF] from='client.? v1:172.21.15.163:0/3939355146' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]: dispatch 2024-02-19T21:18:07.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:07.031518+0000 mon.a (mon.0) 3784 : audit [INF] from='client.49501 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]: dispatch 2024-02-19T21:18:07.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:07.282956+0000 osd.1 (osd.1) 365 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:18:07.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:07.286414+0000 osd.1 (osd.1) 366 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:18:07.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:07.292789+0000 osd.3 (osd.3) 345 : cluster [DBG] 16.7 scrub starts 2024-02-19T21:18:07.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:07.294976+0000 osd.3 (osd.3) 346 : cluster [DBG] 16.7 scrub ok 2024-02-19T21:18:07.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:07.031135+0000 mon.b (mon.1) 759 : audit [INF] from='client.? v1:172.21.15.163:0/3939355146' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]: dispatch 2024-02-19T21:18:08.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:07.031518+0000 mon.a (mon.0) 3784 : audit [INF] from='client.49501 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]: dispatch 2024-02-19T21:18:08.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:07.282956+0000 osd.1 (osd.1) 365 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:18:08.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:07.286414+0000 osd.1 (osd.1) 366 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:18:08.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:07.292789+0000 osd.3 (osd.3) 345 : cluster [DBG] 16.7 scrub starts 2024-02-19T21:18:08.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:07.294976+0000 osd.3 (osd.3) 346 : cluster [DBG] 16.7 scrub ok 2024-02-19T21:18:08.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:18:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:18:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:18:08.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:07.031135+0000 mon.b (mon.1) 759 : audit [INF] from='client.? v1:172.21.15.163:0/3939355146' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]: dispatch 2024-02-19T21:18:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:07.031518+0000 mon.a (mon.0) 3784 : audit [INF] from='client.49501 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]: dispatch 2024-02-19T21:18:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:07.282956+0000 osd.1 (osd.1) 365 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:18:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:07.286414+0000 osd.1 (osd.1) 366 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:18:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:07.292789+0000 osd.3 (osd.3) 345 : cluster [DBG] 16.7 scrub starts 2024-02-19T21:18:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:07.294976+0000 osd.3 (osd.3) 346 : cluster [DBG] 16.7 scrub ok 2024-02-19T21:18:08.388 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-02-19T21:18:08.660 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:08.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:07.349867+0000 mgr.y (mgr.14550) 677 : cluster [DBG] pgmap v1243: 174 pgs: 2 active+clean+remapped, 172 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 739 B/s, 18 objects/s recovering 2024-02-19T21:18:08.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:07.643672+0000 osd.6 (osd.6) 293 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:18:09.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:07.644482+0000 osd.6 (osd.6) 294 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:18:09.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:07.717239+0000 mon.a (mon.0) 3785 : audit [INF] from='client.49501 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]': finished 2024-02-19T21:18:09.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:07.717301+0000 mon.a (mon.0) 3786 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2024-02-19T21:18:09.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:08.118007+0000 osd.4 (osd.4) 285 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:18:09.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:08.119668+0000 osd.4 (osd.4) 286 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:18:09.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:08.312137+0000 osd.3 (osd.3) 347 : cluster [DBG] 16.16 deep-scrub starts 2024-02-19T21:18:09.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:08.313430+0000 osd.3 (osd.3) 348 : cluster [DBG] 16.16 deep-scrub ok 2024-02-19T21:18:09.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:07.349867+0000 mgr.y (mgr.14550) 677 : cluster [DBG] pgmap v1243: 174 pgs: 2 active+clean+remapped, 172 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 739 B/s, 18 objects/s recovering 2024-02-19T21:18:09.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:07.643672+0000 osd.6 (osd.6) 293 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:18:09.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:07.644482+0000 osd.6 (osd.6) 294 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:18:09.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:07.717239+0000 mon.a (mon.0) 3785 : audit [INF] from='client.49501 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]': finished 2024-02-19T21:18:09.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:07.717301+0000 mon.a (mon.0) 3786 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2024-02-19T21:18:09.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:08.118007+0000 osd.4 (osd.4) 285 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:18:09.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:08.119668+0000 osd.4 (osd.4) 286 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:18:09.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:08.312137+0000 osd.3 (osd.3) 347 : cluster [DBG] 16.16 deep-scrub starts 2024-02-19T21:18:09.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:08.313430+0000 osd.3 (osd.3) 348 : cluster [DBG] 16.16 deep-scrub ok 2024-02-19T21:18:09.023 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:07.349867+0000 mgr.y (mgr.14550) 677 : cluster [DBG] pgmap v1243: 174 pgs: 2 active+clean+remapped, 172 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 739 B/s, 18 objects/s recovering 2024-02-19T21:18:09.024 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:07.643672+0000 osd.6 (osd.6) 293 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:18:09.024 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:07.644482+0000 osd.6 (osd.6) 294 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:18:09.024 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:07.717239+0000 mon.a (mon.0) 3785 : audit [INF] from='client.49501 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num", "val": "42"}]': finished 2024-02-19T21:18:09.024 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:07.717301+0000 mon.a (mon.0) 3786 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2024-02-19T21:18:09.024 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:08.118007+0000 osd.4 (osd.4) 285 : cluster [DBG] 5.9 scrub starts 2024-02-19T21:18:09.024 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:08.119668+0000 osd.4 (osd.4) 286 : cluster [DBG] 5.9 scrub ok 2024-02-19T21:18:09.024 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:08.312137+0000 osd.3 (osd.3) 347 : cluster [DBG] 16.16 deep-scrub starts 2024-02-19T21:18:09.024 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:08.313430+0000 osd.3 (osd.3) 348 : cluster [DBG] 16.16 deep-scrub ok 2024-02-19T21:18:09.538 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:18:09.538 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:18:10.231 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:09.113576+0000 mon.a (mon.0) 3787 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:18:10.231 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:09.116580+0000 osd.4 (osd.4) 287 : cluster [DBG] 2.b scrub starts 2024-02-19T21:18:10.231 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:09.118089+0000 osd.4 (osd.4) 288 : cluster [DBG] 2.b scrub ok 2024-02-19T21:18:10.232 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:09.278381+0000 osd.1 (osd.1) 367 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:18:10.232 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:09.281861+0000 osd.1 (osd.1) 368 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:18:10.232 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:09.349608+0000 osd.3 (osd.3) 349 : cluster [DBG] 16.1a scrub starts 2024-02-19T21:18:10.232 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:09.350210+0000 mgr.y (mgr.14550) 678 : cluster [DBG] pgmap v1245: 174 pgs: 2 active+clean+remapped, 172 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 601 B/s, 15 objects/s recovering 2024-02-19T21:18:10.232 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:09.350635+0000 osd.3 (osd.3) 350 : cluster [DBG] 16.1a scrub ok 2024-02-19T21:18:10.232 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:09.430951+0000 osd.2 (osd.2) 328 : cluster [DBG] 2.21 scrub starts 2024-02-19T21:18:10.232 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:09.434200+0000 osd.2 (osd.2) 329 : cluster [DBG] 2.21 scrub ok 2024-02-19T21:18:10.232 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:09.435627+0000 mon.a (mon.0) 3788 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-02-19T21:18:10.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:09.113576+0000 mon.a (mon.0) 3787 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:18:10.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:09.116580+0000 osd.4 (osd.4) 287 : cluster [DBG] 2.b scrub starts 2024-02-19T21:18:10.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:09.118089+0000 osd.4 (osd.4) 288 : cluster [DBG] 2.b scrub ok 2024-02-19T21:18:10.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:09.278381+0000 osd.1 (osd.1) 367 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:18:10.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:09.281861+0000 osd.1 (osd.1) 368 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:18:10.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:09.349608+0000 osd.3 (osd.3) 349 : cluster [DBG] 16.1a scrub starts 2024-02-19T21:18:10.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:09.350210+0000 mgr.y (mgr.14550) 678 : cluster [DBG] pgmap v1245: 174 pgs: 2 active+clean+remapped, 172 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 601 B/s, 15 objects/s recovering 2024-02-19T21:18:10.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:09.350635+0000 osd.3 (osd.3) 350 : cluster [DBG] 16.1a scrub ok 2024-02-19T21:18:10.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:09.430951+0000 osd.2 (osd.2) 328 : cluster [DBG] 2.21 scrub starts 2024-02-19T21:18:10.246 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:09.434200+0000 osd.2 (osd.2) 329 : cluster [DBG] 2.21 scrub ok 2024-02-19T21:18:10.247 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:09.435627+0000 mon.a (mon.0) 3788 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-02-19T21:18:10.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:09.113576+0000 mon.a (mon.0) 3787 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:18:10.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:09.116580+0000 osd.4 (osd.4) 287 : cluster [DBG] 2.b scrub starts 2024-02-19T21:18:10.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:09.118089+0000 osd.4 (osd.4) 288 : cluster [DBG] 2.b scrub ok 2024-02-19T21:18:10.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:09.278381+0000 osd.1 (osd.1) 367 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:18:10.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:09.281861+0000 osd.1 (osd.1) 368 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:18:10.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:09.349608+0000 osd.3 (osd.3) 349 : cluster [DBG] 16.1a scrub starts 2024-02-19T21:18:10.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:09.350210+0000 mgr.y (mgr.14550) 678 : cluster [DBG] pgmap v1245: 174 pgs: 2 active+clean+remapped, 172 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 601 B/s, 15 objects/s recovering 2024-02-19T21:18:10.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:09.350635+0000 osd.3 (osd.3) 350 : cluster [DBG] 16.1a scrub ok 2024-02-19T21:18:10.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:09.430951+0000 osd.2 (osd.2) 328 : cluster [DBG] 2.21 scrub starts 2024-02-19T21:18:10.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:09.434200+0000 osd.2 (osd.2) 329 : cluster [DBG] 2.21 scrub ok 2024-02-19T21:18:10.247 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:09.435627+0000 mon.a (mon.0) 3788 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-02-19T21:18:10.251 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-19T21:18:10.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.491+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/2501070106 >> v1:172.21.15.163:6790/0 conn(0x7fb2b0100770 legacy=0x7fb2b0102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:10.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.491+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/2501070106 shutdown_connections 2024-02-19T21:18:10.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.491+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/2501070106 >> 172.21.15.163:0/2501070106 conn(0x7fb2b00f3570 msgr2=0x7fb2b00f5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:10.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.491+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/2501070106 shutdown_connections 2024-02-19T21:18:10.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.491+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/2501070106 wait complete. 2024-02-19T21:18:10.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.491+0000 7fb2b4fd6700 1 librados: starting msgr at 2024-02-19T21:18:10.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.491+0000 7fb2b4fd6700 1 librados: starting objecter 2024-02-19T21:18:10.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb2b4fd6700 1 Processor -- start 2024-02-19T21:18:10.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb2b4fd6700 1 -- start start 2024-02-19T21:18:10.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb2b4fd6700 1 librados: setting wanted keys 2024-02-19T21:18:10.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb2b4fd6700 1 librados: calling monclient init 2024-02-19T21:18:10.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb2b4fd6700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb2b01172d0 con 0x7fb2b0100770 2024-02-19T21:18:10.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb2b4fd6700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb2b01184b0 con 0x7fb2b00fd970 2024-02-19T21:18:10.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb2b4fd6700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb2b01196b0 con 0x7fb2b01042b0 2024-02-19T21:18:10.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb2add9b700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fb2b0100770 0x7fb2b00fa060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:56070/0 (socket says 172.21.15.163:56070) 2024-02-19T21:18:10.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb2add9b700 1 -- 172.21.15.163:0/3646473181 learned_addr learned my addr 172.21.15.163:0/3646473181 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:10.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb2aed9d700 1 --1- 172.21.15.163:0/3646473181 >> v1:172.21.15.163:6790/0 conn(0x7fb2b01042b0 0x7fb2b0115990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38828/0 (socket says 172.21.15.163:38828) 2024-02-19T21:18:10.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb29f7fe700 10 client.?.objecter ms_handle_connect 0x7fb2b0100770 2024-02-19T21:18:10.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb29f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:10.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb29f7fe700 10 client.?.objecter ms_handle_connect 0x7fb2b01042b0 2024-02-19T21:18:10.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb29f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:10.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb29f7fe700 10 client.?.objecter ms_handle_connect 0x7fb2b00fd970 2024-02-19T21:18:10.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb29f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:10.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622974504 0 0) 0x7fb2b01172d0 con 0x7fb2b0100770 2024-02-19T21:18:10.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb298003800 con 0x7fb2b0100770 2024-02-19T21:18:10.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309893993 0 0) 0x7fb2b01196b0 con 0x7fb2b01042b0 2024-02-19T21:18:10.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.492+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b01172d0 con 0x7fb2b01042b0 2024-02-19T21:18:10.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006189924 0 0) 0x7fb2b01184b0 con 0x7fb2b00fd970 2024-02-19T21:18:10.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b01196b0 con 0x7fb2b00fd970 2024-02-19T21:18:10.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3995513171 0 0) 0x7fb2b01172d0 con 0x7fb2b01042b0 2024-02-19T21:18:10.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2b01184b0 con 0x7fb2b01042b0 2024-02-19T21:18:10.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb2a8001b80 con 0x7fb2b01042b0 2024-02-19T21:18:10.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2248171724 0 0) 0x7fb298003800 con 0x7fb2b0100770 2024-02-19T21:18:10.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2b01172d0 con 0x7fb2b0100770 2024-02-19T21:18:10.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb2a0002f60 con 0x7fb2b0100770 2024-02-19T21:18:10.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1245996793 0 0) 0x7fb2b01196b0 con 0x7fb2b00fd970 2024-02-19T21:18:10.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb298003800 con 0x7fb2b00fd970 2024-02-19T21:18:10.506 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb2a40018c0 con 0x7fb2b00fd970 2024-02-19T21:18:10.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2481324682 0 0) 0x7fb2b01184b0 con 0x7fb2b01042b0 2024-02-19T21:18:10.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.193:6789/0 conn(0x7fb2b00fd970 legacy=0x7fb2b00f9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:10.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.163:6789/0 conn(0x7fb2b0100770 legacy=0x7fb2b00fa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:10.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b011a8b0 con 0x7fb2b01042b0 2024-02-19T21:18:10.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b01198a0 con 0x7fb2b01042b0 2024-02-19T21:18:10.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb2b4fd6700 10 client.50696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:10.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b0119d90 con 0x7fb2b01042b0 2024-02-19T21:18:10.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb2b4fd6700 1 librados: init done 2024-02-19T21:18:10.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb2a8002640 con 0x7fb2b01042b0 2024-02-19T21:18:10.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.493+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb2a8005d20 con 0x7fb2b01042b0 2024-02-19T21:18:10.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.494+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb290004f40 con 0x7fb2b01042b0 2024-02-19T21:18:10.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.494+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fb2a8018c90 con 0x7fb2b01042b0 2024-02-19T21:18:10.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.495+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(820..820 src has 633..820) v4 ==== 7187+0+0 (unknown 62866080 0 0) 0x7fb2a8078bd0 con 0x7fb2b01042b0 2024-02-19T21:18:10.507 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.495+0000 7fb29f7fe700 10 client.50696.objecter ms_dispatch 0x7fb2b0001550 osd_map(820..820 src has 633..820) v4 2024-02-19T21:18:10.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.495+0000 7fb29f7fe700 3 client.50696.objecter handle_osd_map got epochs [820,820] > 0 2024-02-19T21:18:10.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.495+0000 7fb29f7fe700 3 client.50696.objecter handle_osd_map decoding full epoch 820 2024-02-19T21:18:10.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.495+0000 7fb29f7fe700 20 client.50696.objecter dump_active .. 0 homeless 2024-02-19T21:18:10.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.495+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1111 (unknown 409899127 0 545308006) 0x7fb2a8079000 con 0x7fb2b01042b0 2024-02-19T21:18:10.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.495+0000 7fb29f7fe700 10 client.50696.objecter ms_handle_connect 0x7fb298063c30 2024-02-19T21:18:10.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.495+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fb290004cb0 con 0x7fb2900027b0 2024-02-19T21:18:10.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.495+0000 7fb29f7fe700 10 client.50696.objecter ms_handle_connect 0x7fb2900027b0 2024-02-19T21:18:10.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.495+0000 7fb29f7fe700 10 client.50696.objecter resend_mon_ops 2024-02-19T21:18:10.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.495+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47355481 0 0) 0x7fb290004cb0 con 0x7fb2900027b0 2024-02-19T21:18:10.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.495+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b01184b0 con 0x7fb2900027b0 2024-02-19T21:18:10.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.495+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1697008476 0 0) 0x7fb2b01184b0 con 0x7fb2900027b0 2024-02-19T21:18:10.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.495+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb290004cb0 con 0x7fb2900027b0 2024-02-19T21:18:10.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.495+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb2a00024e0 con 0x7fb2900027b0 2024-02-19T21:18:10.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.496+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2069764837 0 0) 0x7fb290004cb0 con 0x7fb2900027b0 2024-02-19T21:18:10.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.496+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6789/0 -- command(tid 2: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb290002260 con 0x7fb2900027b0 2024-02-19T21:18:10.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.496+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.0 v1:172.21.15.163:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+8167 (unknown 0 0 1565185916) 0x7fb2a0002300 con 0x7fb2900027b0 2024-02-19T21:18:10.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.496+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.163:6789/0 conn(0x7fb2900027b0 legacy=0x7fb290005920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:10.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.496+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.163:6789/0 conn(0x7fb2900027b0 legacy=0x7fb290005920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-19T21:18:10.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.503+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fb29000ada0 con 0x7fb2900070d0 2024-02-19T21:18:10.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.503+0000 7fb29f7fe700 10 client.50696.objecter ms_handle_connect 0x7fb2900070d0 2024-02-19T21:18:10.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.503+0000 7fb29f7fe700 10 client.50696.objecter resend_mon_ops 2024-02-19T21:18:10.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.503+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734326153 0 0) 0x7fb29000ada0 con 0x7fb2900070d0 2024-02-19T21:18:10.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.503+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb290004cb0 con 0x7fb2900070d0 2024-02-19T21:18:10.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.503+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2125258570 0 0) 0x7fb290004cb0 con 0x7fb2900070d0 2024-02-19T21:18:10.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.503+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb29000ada0 con 0x7fb2900070d0 2024-02-19T21:18:10.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.503+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb2a00033a0 con 0x7fb2900070d0 2024-02-19T21:18:10.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.503+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3986438973 0 0) 0x7fb29000ada0 con 0x7fb2900070d0 2024-02-19T21:18:10.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.503+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6789/0 -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7fb29000c000 con 0x7fb2900070d0 2024-02-19T21:18:10.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.505+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.0 v1:172.21.15.163:6789/0 5 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '600' (not observed, change may require restart) ) v1 ==== 85+0+3 (unknown 1746277125 0 2423995997) 0x7fb2a0006970 con 0x7fb2900070d0 2024-02-19T21:18:10.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.505+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.163:6789/0 conn(0x7fb2900070d0 legacy=0x7fb290009510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:10.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.505+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.163:6789/0 conn(0x7fb2900070d0 legacy=0x7fb290009510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-19T21:18:10.511 INFO:teuthology.orchestra.run.smithi163.stdout:mon.a: {} 2024-02-19T21:18:10.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.505+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fb290001f20 con 0x7fb290001ab0 2024-02-19T21:18:10.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.505+0000 7fb29f7fe700 10 client.50696.objecter ms_handle_connect 0x7fb290001ab0 2024-02-19T21:18:10.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.505+0000 7fb29f7fe700 10 client.50696.objecter resend_mon_ops 2024-02-19T21:18:10.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.505+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1657906476 0 0) 0x7fb290001f20 con 0x7fb290001ab0 2024-02-19T21:18:10.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.505+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb29000ada0 con 0x7fb290001ab0 2024-02-19T21:18:10.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.506+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3033583710 0 0) 0x7fb29000ada0 con 0x7fb290001ab0 2024-02-19T21:18:10.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.506+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb290001f20 con 0x7fb290001ab0 2024-02-19T21:18:10.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.506+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb2a0003910 con 0x7fb290001ab0 2024-02-19T21:18:10.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.506+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3210594803 0 0) 0x7fb290001f20 con 0x7fb290001ab0 2024-02-19T21:18:10.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.506+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.193:6789/0 -- command(tid 4: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb290012960 con 0x7fb290001ab0 2024-02-19T21:18:10.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.506+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.1 v1:172.21.15.193:6789/0 5 ==== command_reply(tid 4: 0 ) v1 ==== 8+0+8167 (unknown 0 0 1565185916) 0x7fb2a0006ae0 con 0x7fb290001ab0 2024-02-19T21:18:10.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.506+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.193:6789/0 conn(0x7fb290001ab0 legacy=0x7fb29000b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:10.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.506+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.193:6789/0 conn(0x7fb290001ab0 legacy=0x7fb29000b700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-19T21:18:10.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.506+0000 7fb2aed9d700 1 -- 172.21.15.163:0/3646473181 reap_dead start 2024-02-19T21:18:10.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.513+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fb2900130e0 con 0x7fb290012c80 2024-02-19T21:18:10.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.513+0000 7fb29f7fe700 10 client.50696.objecter ms_handle_connect 0x7fb290012c80 2024-02-19T21:18:10.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.513+0000 7fb29f7fe700 10 client.50696.objecter resend_mon_ops 2024-02-19T21:18:10.515 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.513+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366504682 0 0) 0x7fb2900130e0 con 0x7fb290012c80 2024-02-19T21:18:10.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.513+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb290001f20 con 0x7fb290012c80 2024-02-19T21:18:10.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.514+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1200072273 0 0) 0x7fb290001f20 con 0x7fb290012c80 2024-02-19T21:18:10.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.514+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2900130e0 con 0x7fb290012c80 2024-02-19T21:18:10.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.514+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb2a0003e60 con 0x7fb290012c80 2024-02-19T21:18:10.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.514+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 867197880 0 0) 0x7fb2900130e0 con 0x7fb290012c80 2024-02-19T21:18:10.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.514+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.193:6789/0 -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7fb290014340 con 0x7fb290012c80 2024-02-19T21:18:10.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.515+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.1 v1:172.21.15.193:6789/0 5 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '600' (not observed, change may require restart) ) v1 ==== 85+0+3 (unknown 1746277125 0 2423995997) 0x7fb2a0003d80 con 0x7fb290012c80 2024-02-19T21:18:10.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.515+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.193:6789/0 conn(0x7fb290012c80 legacy=0x7fb290011200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:10.519 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.515+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.193:6789/0 conn(0x7fb290012c80 legacy=0x7fb290011200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-19T21:18:10.519 INFO:teuthology.orchestra.run.smithi163.stdout:mon.b: {} 2024-02-19T21:18:10.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.516+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fb290013f80 con 0x7fb290013360 2024-02-19T21:18:10.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.516+0000 7fb29f7fe700 10 client.50696.objecter ms_handle_connect 0x7fb290013360 2024-02-19T21:18:10.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.516+0000 7fb29f7fe700 10 client.50696.objecter resend_mon_ops 2024-02-19T21:18:10.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.516+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959101727 0 0) 0x7fb290013f80 con 0x7fb290013360 2024-02-19T21:18:10.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.516+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2900130e0 con 0x7fb290013360 2024-02-19T21:18:10.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.516+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2571165029 0 0) 0x7fb2900130e0 con 0x7fb290013360 2024-02-19T21:18:10.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.516+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb290013f80 con 0x7fb290013360 2024-02-19T21:18:10.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.516+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb2a0007390 con 0x7fb290013360 2024-02-19T21:18:10.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.516+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3461034039 0 0) 0x7fb290013f80 con 0x7fb290013360 2024-02-19T21:18:10.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.516+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6790/0 -- command(tid 6: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb29001aca0 con 0x7fb290013360 2024-02-19T21:18:10.521 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.516+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 5 ==== command_reply(tid 6: 0 ) v1 ==== 8+0+8167 (unknown 0 0 1565185916) 0x7fb2a00072e0 con 0x7fb290013360 2024-02-19T21:18:10.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.516+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.163:6790/0 conn(0x7fb290013360 legacy=0x7fb2900137b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:10.522 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.516+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.163:6790/0 conn(0x7fb290013360 legacy=0x7fb2900137b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-19T21:18:10.522 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:18:10.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.523+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 3) v1 -- 0x7fb29001b1f0 con 0x7fb29001ae10 2024-02-19T21:18:10.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.523+0000 7fb29f7fe700 10 client.50696.objecter ms_handle_connect 0x7fb29001ae10 2024-02-19T21:18:10.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.523+0000 7fb29f7fe700 10 client.50696.objecter resend_mon_ops 2024-02-19T21:18:10.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.523+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075849848 0 0) 0x7fb29001b1f0 con 0x7fb29001ae10 2024-02-19T21:18:10.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.523+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2900130e0 con 0x7fb29001ae10 2024-02-19T21:18:10.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.523+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3098219723 0 0) 0x7fb2900130e0 con 0x7fb29001ae10 2024-02-19T21:18:10.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.523+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb29001b1f0 con 0x7fb29001ae10 2024-02-19T21:18:10.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.523+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb2a0007920 con 0x7fb29001ae10 2024-02-19T21:18:10.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.523+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3832678284 0 0) 0x7fb29001b1f0 con 0x7fb29001ae10 2024-02-19T21:18:10.525 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.523+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 --> v1:172.21.15.163:6790/0 -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) v1 -- 0x7fb29001c430 con 0x7fb29001ae10 2024-02-19T21:18:10.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.525+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 <== mon.2 v1:172.21.15.163:6790/0 5 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '600' (not observed, change may require restart) ) v1 ==== 85+0+3 (unknown 1746277125 0 2423995997) 0x7fb2a0007b90 con 0x7fb29001ae10 2024-02-19T21:18:10.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.525+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.163:6790/0 conn(0x7fb29001ae10 legacy=0x7fb290019540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:10.526 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.525+0000 7fb29f7fe700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.163:6790/0 conn(0x7fb29001ae10 legacy=0x7fb290019540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-02-19T21:18:10.527 INFO:teuthology.orchestra.run.smithi163.stdout:mon.c: {} 2024-02-19T21:18:10.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.525+0000 7fb2b4fd6700 10 librados: watch_flush enter 2024-02-19T21:18:10.527 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.525+0000 7fb2b4fd6700 10 librados: watch_flush exit 2024-02-19T21:18:10.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.525+0000 7fb2b4fd6700 20 client.50696.objecter shutdown clearing up homeless session... 2024-02-19T21:18:10.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.525+0000 7fb2b4fd6700 10 client.50696.objecter successfully canceled tick 2024-02-19T21:18:10.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.525+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.163:6800/1552514681 conn(0x7fb298063c30 legacy=0x7fb2980660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:10.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.525+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/3646473181 >> v1:172.21.15.163:6790/0 conn(0x7fb2b01042b0 legacy=0x7fb2b0115990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:10.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.525+0000 7fb2aed9d700 1 -- 172.21.15.163:0/3646473181 reap_dead start 2024-02-19T21:18:10.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.525+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/3646473181 shutdown_connections 2024-02-19T21:18:10.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.525+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/3646473181 >> 172.21.15.163:0/3646473181 conn(0x7fb2b00f3570 msgr2=0x7fb2b0107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:10.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.525+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/3646473181 shutdown_connections 2024-02-19T21:18:10.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.525+0000 7fb2b4fd6700 1 -- 172.21.15.163:0/3646473181 wait complete. 2024-02-19T21:18:10.528 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:10.525+0000 7fb2b4fd6700 1 librados: shutdown 2024-02-19T21:18:10.528 INFO:teuthology.orchestra.run.smithi163.stderr:mon.a: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-19T21:18:10.528 INFO:teuthology.orchestra.run.smithi163.stderr:mon.b: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-19T21:18:10.528 INFO:teuthology.orchestra.run.smithi163.stderr:mon.c: mon_osd_down_out_interval = '600' (not observed, change may require restart) 2024-02-19T21:18:11.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:18:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:18:11.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:10.235091+0000 osd.1 (osd.1) 369 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:18:11.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:10.237986+0000 osd.1 (osd.1) 370 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:18:11.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:10.334069+0000 osd.3 (osd.3) 351 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:18:11.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:10.335638+0000 osd.3 (osd.3) 352 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:18:11.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:10.423535+0000 osd.2 (osd.2) 330 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:18:11.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:10.426426+0000 osd.2 (osd.2) 331 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:18:11.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:10.496061+0000 mon.c (mon.2) 445 : audit [DBG] from='client.? v1:172.21.15.163:0/3646473181' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:18:11.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:10.854380+0000 mgr.y (mgr.14550) 679 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:11.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:10.235091+0000 osd.1 (osd.1) 369 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:18:11.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:10.237986+0000 osd.1 (osd.1) 370 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:18:11.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:10.334069+0000 osd.3 (osd.3) 351 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:18:11.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:10.335638+0000 osd.3 (osd.3) 352 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:18:11.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:10.423535+0000 osd.2 (osd.2) 330 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:18:11.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:10.426426+0000 osd.2 (osd.2) 331 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:18:11.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:10.496061+0000 mon.c (mon.2) 445 : audit [DBG] from='client.? v1:172.21.15.163:0/3646473181' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:18:11.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:10.854380+0000 mgr.y (mgr.14550) 679 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:11.325 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 0 1 2024-02-19T21:18:11.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:10.235091+0000 osd.1 (osd.1) 369 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:18:11.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:10.237986+0000 osd.1 (osd.1) 370 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:18:11.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:10.334069+0000 osd.3 (osd.3) 351 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:18:11.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:10.335638+0000 osd.3 (osd.3) 352 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:18:11.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:10.423535+0000 osd.2 (osd.2) 330 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:18:11.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:10.426426+0000 osd.2 (osd.2) 331 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:18:11.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:10.496061+0000 mon.c (mon.2) 445 : audit [DBG] from='client.? v1:172.21.15.163:0/3646473181' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T21:18:11.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:10.854380+0000 mgr.y (mgr.14550) 679 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:11.594 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:12.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:11.302926+0000 osd.3 (osd.3) 353 : cluster [DBG] 5.13 scrub starts 2024-02-19T21:18:12.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:11.303774+0000 osd.3 (osd.3) 354 : cluster [DBG] 5.13 scrub ok 2024-02-19T21:18:12.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:11.350975+0000 mgr.y (mgr.14550) 680 : cluster [DBG] pgmap v1247: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-02-19T21:18:12.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:11.384907+0000 osd.5 (osd.5) 322 : cluster [DBG] 16.3 scrub starts 2024-02-19T21:18:12.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:11.389724+0000 osd.5 (osd.5) 323 : cluster [DBG] 16.3 scrub ok 2024-02-19T21:18:12.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:11.462018+0000 osd.2 (osd.2) 332 : cluster [DBG] 5.a scrub starts 2024-02-19T21:18:12.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:11.462898+0000 osd.2 (osd.2) 333 : cluster [DBG] 5.a scrub ok 2024-02-19T21:18:12.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:11.302926+0000 osd.3 (osd.3) 353 : cluster [DBG] 5.13 scrub starts 2024-02-19T21:18:12.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:11.303774+0000 osd.3 (osd.3) 354 : cluster [DBG] 5.13 scrub ok 2024-02-19T21:18:12.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:11.350975+0000 mgr.y (mgr.14550) 680 : cluster [DBG] pgmap v1247: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-02-19T21:18:12.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:11.384907+0000 osd.5 (osd.5) 322 : cluster [DBG] 16.3 scrub starts 2024-02-19T21:18:12.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:11.389724+0000 osd.5 (osd.5) 323 : cluster [DBG] 16.3 scrub ok 2024-02-19T21:18:12.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:11.462018+0000 osd.2 (osd.2) 332 : cluster [DBG] 5.a scrub starts 2024-02-19T21:18:12.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:11.462898+0000 osd.2 (osd.2) 333 : cluster [DBG] 5.a scrub ok 2024-02-19T21:18:12.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:11.302926+0000 osd.3 (osd.3) 353 : cluster [DBG] 5.13 scrub starts 2024-02-19T21:18:12.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:11.303774+0000 osd.3 (osd.3) 354 : cluster [DBG] 5.13 scrub ok 2024-02-19T21:18:12.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:11.350975+0000 mgr.y (mgr.14550) 680 : cluster [DBG] pgmap v1247: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-02-19T21:18:12.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:11.384907+0000 osd.5 (osd.5) 322 : cluster [DBG] 16.3 scrub starts 2024-02-19T21:18:12.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:11.389724+0000 osd.5 (osd.5) 323 : cluster [DBG] 16.3 scrub ok 2024-02-19T21:18:12.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:11.462018+0000 osd.2 (osd.2) 332 : cluster [DBG] 5.a scrub starts 2024-02-19T21:18:12.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:11.462898+0000 osd.2 (osd.2) 333 : cluster [DBG] 5.a scrub ok 2024-02-19T21:18:12.612 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:18:12.612 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:18:13.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:12.070646+0000 osd.4 (osd.4) 289 : cluster [DBG] 16.19 scrub starts 2024-02-19T21:18:13.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:12.072417+0000 osd.4 (osd.4) 290 : cluster [DBG] 16.19 scrub ok 2024-02-19T21:18:13.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:12.308116+0000 osd.1 (osd.1) 371 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:18:13.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:12.311304+0000 osd.1 (osd.1) 372 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:18:13.241 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:12.342633+0000 osd.5 (osd.5) 324 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:18:13.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:12.345790+0000 osd.5 (osd.5) 325 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:18:13.242 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:12.070646+0000 osd.4 (osd.4) 289 : cluster [DBG] 16.19 scrub starts 2024-02-19T21:18:13.242 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:12.072417+0000 osd.4 (osd.4) 290 : cluster [DBG] 16.19 scrub ok 2024-02-19T21:18:13.242 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:12.308116+0000 osd.1 (osd.1) 371 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:18:13.242 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:12.311304+0000 osd.1 (osd.1) 372 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:18:13.242 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:12.342633+0000 osd.5 (osd.5) 324 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:18:13.242 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:12.345790+0000 osd.5 (osd.5) 325 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:18:13.296 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:12.070646+0000 osd.4 (osd.4) 289 : cluster [DBG] 16.19 scrub starts 2024-02-19T21:18:13.296 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:12.072417+0000 osd.4 (osd.4) 290 : cluster [DBG] 16.19 scrub ok 2024-02-19T21:18:13.296 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:12.308116+0000 osd.1 (osd.1) 371 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:18:13.296 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:12.311304+0000 osd.1 (osd.1) 372 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:18:13.296 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:12.342633+0000 osd.5 (osd.5) 324 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:18:13.296 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:12.345790+0000 osd.5 (osd.5) 325 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:18:13.314 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-19T21:18:13.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.469+0000 7fb6eaf43700 1 -- 172.21.15.163:0/3868261535 >> v1:172.21.15.163:6790/0 conn(0x7fb6e40ff6b0 legacy=0x7fb6e4101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:13.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.469+0000 7fb6eaf43700 1 -- 172.21.15.163:0/3868261535 shutdown_connections 2024-02-19T21:18:13.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.469+0000 7fb6eaf43700 1 -- 172.21.15.163:0/3868261535 >> 172.21.15.163:0/3868261535 conn(0x7fb6e406cc00 msgr2=0x7fb6e40f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:13.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.469+0000 7fb6eaf43700 1 -- 172.21.15.163:0/3868261535 shutdown_connections 2024-02-19T21:18:13.471 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.469+0000 7fb6eaf43700 1 -- 172.21.15.163:0/3868261535 wait complete. 2024-02-19T21:18:13.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6eaf43700 1 librados: starting msgr at 2024-02-19T21:18:13.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6eaf43700 1 librados: starting objecter 2024-02-19T21:18:13.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6eaf43700 1 Processor -- start 2024-02-19T21:18:13.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6eaf43700 1 -- start start 2024-02-19T21:18:13.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6eaf43700 1 librados: setting wanted keys 2024-02-19T21:18:13.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6eaf43700 1 librados: calling monclient init 2024-02-19T21:18:13.473 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6eaf43700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6e4116200 con 0x7fb6e41031c0 2024-02-19T21:18:13.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6eaf43700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6e4117400 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6eaf43700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6e4118600 con 0x7fb6e40fc880 2024-02-19T21:18:13.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6e8cdf700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fb6e40fc880 0x7fb6e40f8ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:38854/0 (socket says 172.21.15.163:38854) 2024-02-19T21:18:13.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6e8cdf700 1 -- 172.21.15.163:0/2623178851 learned_addr learned my addr 172.21.15.163:0/2623178851 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:13.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6e1ffb700 10 client.?.objecter ms_handle_connect 0x7fb6e40fc880 2024-02-19T21:18:13.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6e1ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:13.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6e1ffb700 10 client.?.objecter ms_handle_connect 0x7fb6e40ff6b0 2024-02-19T21:18:13.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6e1ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:13.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6e1ffb700 10 client.?.objecter ms_handle_connect 0x7fb6e41031c0 2024-02-19T21:18:13.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6e1ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:13.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120903422 0 0) 0x7fb6e4117400 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6d4003800 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.474 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387122275 0 0) 0x7fb6e4116200 con 0x7fb6e41031c0 2024-02-19T21:18:13.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6e4117400 con 0x7fb6e41031c0 2024-02-19T21:18:13.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235937343 0 0) 0x7fb6e4118600 con 0x7fb6e40fc880 2024-02-19T21:18:13.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.470+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6e4116200 con 0x7fb6e40fc880 2024-02-19T21:18:13.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3942134573 0 0) 0x7fb6d4003800 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6e4118600 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb6d0002f60 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3935658438 0 0) 0x7fb6e4117400 con 0x7fb6e41031c0 2024-02-19T21:18:13.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6d4003800 con 0x7fb6e41031c0 2024-02-19T21:18:13.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3136803483 0 0) 0x7fb6e4116200 con 0x7fb6e40fc880 2024-02-19T21:18:13.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6e4117400 con 0x7fb6e40fc880 2024-02-19T21:18:13.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb6dc001bc0 con 0x7fb6e41031c0 2024-02-19T21:18:13.475 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb6d80019f0 con 0x7fb6e40fc880 2024-02-19T21:18:13.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3482261247 0 0) 0x7fb6e4118600 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 >> v1:172.21.15.163:6790/0 conn(0x7fb6e40fc880 legacy=0x7fb6e40f8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:13.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 >> v1:172.21.15.163:6789/0 conn(0x7fb6e41031c0 legacy=0x7fb6e41148c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:13.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e4119800 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6eaf43700 1 -- 172.21.15.163:0/2623178851 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e41175f0 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb6d0000ed0 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb6d0003810 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6eaf43700 10 client.49519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:13.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6eaf43700 1 -- 172.21.15.163:0/2623178851 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e4117a60 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.471+0000 7fb6eaf43700 1 librados: init done 2024-02-19T21:18:13.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.472+0000 7fb6eaf43700 1 -- 172.21.15.163:0/2623178851 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e405f4e0 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.472+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fb6d00166a0 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.473+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(820..820 src has 633..820) v4 ==== 7187+0+0 (unknown 62866080 0 0) 0x7fb6d0076400 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.473+0000 7fb6e1ffb700 10 client.49519.objecter ms_dispatch 0x7fb6e4001550 osd_map(820..820 src has 633..820) v4 2024-02-19T21:18:13.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.473+0000 7fb6e1ffb700 3 client.49519.objecter handle_osd_map got epochs [820,820] > 0 2024-02-19T21:18:13.477 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.473+0000 7fb6e1ffb700 3 client.49519.objecter handle_osd_map decoding full epoch 820 2024-02-19T21:18:13.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.473+0000 7fb6e1ffb700 20 client.49519.objecter dump_active .. 0 homeless 2024-02-19T21:18:13.478 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.473+0000 7fb6e1ffb700 10 client.49519.objecter ms_handle_connect 0x7fb6d4063d30 2024-02-19T21:18:13.482 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.481+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb6d0049d10 con 0x7fb6e40ff6b0 2024-02-19T21:18:13.498 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:18:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:18:13] "GET /metrics HTTP/1.1" 200 187997 "" "Prometheus/2.33.4" 2024-02-19T21:18:13.585 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:18:13.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:13.655+0000 7fb6eaf43700 1 -- 172.21.15.163:0/2623178851 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) v1 -- 0x7fb6e4047540 con 0x7fb6e40ff6b0 2024-02-19T21:18:14.138 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:14.137+0000 7fb6e1ffb700 1 -- 172.21.15.163:0/2623178851 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v821) v1 ==== 113+0+0 (unknown 516988825 0 0) 0x7fb6d0048180 con 0x7fb6e40ff6b0 2024-02-19T21:18:14.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:14.139+0000 7fb6eaf43700 10 librados: watch_flush enter 2024-02-19T21:18:14.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:14.139+0000 7fb6eaf43700 10 librados: watch_flush exit 2024-02-19T21:18:14.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:14.139+0000 7fb6eaf43700 20 client.49519.objecter shutdown clearing up homeless session... 2024-02-19T21:18:14.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:14.139+0000 7fb6eaf43700 10 client.49519.objecter successfully canceled tick 2024-02-19T21:18:14.141 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:14.139+0000 7fb6eaf43700 1 -- 172.21.15.163:0/2623178851 >> v1:172.21.15.163:6800/1552514681 conn(0x7fb6d4063d30 legacy=0x7fb6d40661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:14.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:14.139+0000 7fb6eaf43700 1 -- 172.21.15.163:0/2623178851 >> v1:172.21.15.193:6789/0 conn(0x7fb6e40ff6b0 legacy=0x7fb6e40f93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:14.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:14.139+0000 7fb6eaf43700 1 -- 172.21.15.163:0/2623178851 shutdown_connections 2024-02-19T21:18:14.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:14.139+0000 7fb6eaf43700 1 -- 172.21.15.163:0/2623178851 >> 172.21.15.163:0/2623178851 conn(0x7fb6e406cc00 msgr2=0x7fb6e4106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:14.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:14.140+0000 7fb6eaf43700 1 -- 172.21.15.163:0/2623178851 shutdown_connections 2024-02-19T21:18:14.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:14.140+0000 7fb6eaf43700 1 -- 172.21.15.163:0/2623178851 wait complete. 2024-02-19T21:18:14.142 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:14.140+0000 7fb6eaf43700 1 librados: shutdown 2024-02-19T21:18:14.142 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.0 to 1 (10000) 2024-02-19T21:18:14.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:13.248986+0000 osd.3 (osd.3) 355 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:18:14.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:13.250892+0000 osd.3 (osd.3) 356 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:18:14.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:13.351333+0000 mgr.y (mgr.14550) 681 : cluster [DBG] pgmap v1248: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-02-19T21:18:14.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:13.369072+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:18:14.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:13.371971+0000 osd.5 (osd.5) 327 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:18:14.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:13.656753+0000 mon.b (mon.1) 760 : audit [INF] from='client.? v1:172.21.15.163:0/2623178851' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:18:14.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:13.657094+0000 mon.a (mon.0) 3789 : audit [INF] from='client.49519 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:18:14.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:13.248986+0000 osd.3 (osd.3) 355 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:18:14.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:13.250892+0000 osd.3 (osd.3) 356 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:18:14.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:13.351333+0000 mgr.y (mgr.14550) 681 : cluster [DBG] pgmap v1248: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-02-19T21:18:14.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:13.369072+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:18:14.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:13.371971+0000 osd.5 (osd.5) 327 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:18:14.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:13.656753+0000 mon.b (mon.1) 760 : audit [INF] from='client.? v1:172.21.15.163:0/2623178851' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:18:14.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:13.657094+0000 mon.a (mon.0) 3789 : audit [INF] from='client.49519 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:18:14.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:13.248986+0000 osd.3 (osd.3) 355 : cluster [DBG] 4.11 scrub starts 2024-02-19T21:18:14.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:13.250892+0000 osd.3 (osd.3) 356 : cluster [DBG] 4.11 scrub ok 2024-02-19T21:18:14.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:13.351333+0000 mgr.y (mgr.14550) 681 : cluster [DBG] pgmap v1248: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 741 B/s rd, 0 op/s 2024-02-19T21:18:14.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:13.369072+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:18:14.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:13.371971+0000 osd.5 (osd.5) 327 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:18:14.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:13.656753+0000 mon.b (mon.1) 760 : audit [INF] from='client.? v1:172.21.15.163:0/2623178851' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:18:14.564 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:13.657094+0000 mon.a (mon.0) 3789 : audit [INF] from='client.49519 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:18:15.033 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd primary-affinity 0 1 2024-02-19T21:18:15.306 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:15.367 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:18:15.369 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:18:15.391 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:13.753557+0000 osd.3 (osd.3) 357 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:15.391 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:13.753707+0000 osd.3 (osd.3) 358 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:15.391 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:14.137153+0000 mon.a (mon.0) 3790 : audit [INF] from='client.49519 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-19T21:18:15.391 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:14.137230+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-02-19T21:18:15.392 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:14.310757+0000 osd.1 (osd.1) 373 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:18:15.392 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:14.311758+0000 osd.1 (osd.1) 374 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:18:15.392 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:14.399118+0000 osd.5 (osd.5) 328 : cluster [DBG] 6.d scrub starts 2024-02-19T21:18:15.392 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:14.402000+0000 osd.5 (osd.5) 329 : cluster [DBG] 6.d scrub ok 2024-02-19T21:18:15.392 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:14.436334+0000 mon.a (mon.0) 3792 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-02-19T21:18:15.392 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:15.094223+0000 mon.a (mon.0) 3793 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:18:15.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:13.753557+0000 osd.3 (osd.3) 357 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:15.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:13.753707+0000 osd.3 (osd.3) 358 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:15.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:14.137153+0000 mon.a (mon.0) 3790 : audit [INF] from='client.49519 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-19T21:18:15.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:14.137230+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-02-19T21:18:15.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:14.310757+0000 osd.1 (osd.1) 373 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:18:15.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:14.311758+0000 osd.1 (osd.1) 374 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:18:15.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:14.399118+0000 osd.5 (osd.5) 328 : cluster [DBG] 6.d scrub starts 2024-02-19T21:18:15.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:14.402000+0000 osd.5 (osd.5) 329 : cluster [DBG] 6.d scrub ok 2024-02-19T21:18:15.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:14.436334+0000 mon.a (mon.0) 3792 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-02-19T21:18:15.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:15.094223+0000 mon.a (mon.0) 3793 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:18:15.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:13.753557+0000 osd.3 (osd.3) 357 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:15.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:13.753707+0000 osd.3 (osd.3) 358 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:15.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:14.137153+0000 mon.a (mon.0) 3790 : audit [INF] from='client.49519 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-19T21:18:15.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:14.137230+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-02-19T21:18:15.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:14.310757+0000 osd.1 (osd.1) 373 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:18:15.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:14.311758+0000 osd.1 (osd.1) 374 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:18:15.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:14.399118+0000 osd.5 (osd.5) 328 : cluster [DBG] 6.d scrub starts 2024-02-19T21:18:15.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:14.402000+0000 osd.5 (osd.5) 329 : cluster [DBG] 6.d scrub ok 2024-02-19T21:18:15.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:14.436334+0000 mon.a (mon.0) 3792 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-02-19T21:18:15.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:15.094223+0000 mon.a (mon.0) 3793 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:18:16.033 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-19T21:18:16.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:15.131902+0000 osd.4 (osd.4) 291 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:18:16.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:15.134200+0000 osd.4 (osd.4) 292 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:18:16.141 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:15.352195+0000 mgr.y (mgr.14550) 682 : cluster [DBG] pgmap v1251: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:18:16.304 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:18:16.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:15.131902+0000 osd.4 (osd.4) 291 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:18:16.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:15.134200+0000 osd.4 (osd.4) 292 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:18:16.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:15.352195+0000 mgr.y (mgr.14550) 682 : cluster [DBG] pgmap v1251: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:18:16.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:15.131902+0000 osd.4 (osd.4) 291 : cluster [DBG] 5.18 scrub starts 2024-02-19T21:18:16.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:15.134200+0000 osd.4 (osd.4) 292 : cluster [DBG] 5.18 scrub ok 2024-02-19T21:18:16.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:15.352195+0000 mgr.y (mgr.14550) 682 : cluster [DBG] pgmap v1251: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:18:17.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.103+0000 7fb00c74c700 1 -- 172.21.15.163:0/4281920231 >> v1:172.21.15.163:6790/0 conn(0x7fb0041042c0 legacy=0x7fb004106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:17.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.103+0000 7fb00c74c700 1 -- 172.21.15.163:0/4281920231 shutdown_connections 2024-02-19T21:18:17.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.103+0000 7fb00c74c700 1 -- 172.21.15.163:0/4281920231 >> 172.21.15.163:0/4281920231 conn(0x7fb0040f35a0 msgr2=0x7fb0040f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:17.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.103+0000 7fb00c74c700 1 -- 172.21.15.163:0/4281920231 shutdown_connections 2024-02-19T21:18:17.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.103+0000 7fb00c74c700 1 -- 172.21.15.163:0/4281920231 wait complete. 2024-02-19T21:18:17.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.104+0000 7fb00c74c700 1 librados: starting msgr at 2024-02-19T21:18:17.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.104+0000 7fb00c74c700 1 librados: starting objecter 2024-02-19T21:18:17.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.104+0000 7fb00c74c700 1 Processor -- start 2024-02-19T21:18:17.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.104+0000 7fb00c74c700 1 -- start start 2024-02-19T21:18:17.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.104+0000 7fb00c74c700 1 librados: setting wanted keys 2024-02-19T21:18:17.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.104+0000 7fb00c74c700 1 librados: calling monclient init 2024-02-19T21:18:17.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.104+0000 7fb00c74c700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb0040fcee0 con 0x7fb0041042c0 2024-02-19T21:18:17.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.104+0000 7fb00c74c700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb0040fd0b0 con 0x7fb004100780 2024-02-19T21:18:17.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.104+0000 7fb00c74c700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb0040fd2a0 con 0x7fb0040fd980 2024-02-19T21:18:17.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.104+0000 7fb00ace9700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fb0041042c0 0x7fb004118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44048/0 (socket says 172.21.15.163:44048) 2024-02-19T21:18:17.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.104+0000 7fb00ace9700 1 -- 172.21.15.163:0/198736836 learned_addr learned my addr 172.21.15.163:0/198736836 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:17.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.104+0000 7faffb7fe700 10 client.?.objecter ms_handle_connect 0x7fb0041042c0 2024-02-19T21:18:17.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.104+0000 7faffb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:17.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.104+0000 7faffb7fe700 10 client.?.objecter ms_handle_connect 0x7fb0040fd980 2024-02-19T21:18:17.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.104+0000 7faffb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:17.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 10 client.?.objecter ms_handle_connect 0x7fb004100780 2024-02-19T21:18:17.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:17.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017017930 0 0) 0x7fb0040fcee0 con 0x7fb0041042c0 2024-02-19T21:18:17.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff00033f0 con 0x7fb0041042c0 2024-02-19T21:18:17.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898535375 0 0) 0x7fb0040fd2a0 con 0x7fb0040fd980 2024-02-19T21:18:17.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0040fcee0 con 0x7fb0040fd980 2024-02-19T21:18:17.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416872171 0 0) 0x7fb0040fd0b0 con 0x7fb004100780 2024-02-19T21:18:17.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0040fd2a0 con 0x7fb004100780 2024-02-19T21:18:17.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1523435188 0 0) 0x7faff00033f0 con 0x7fb0041042c0 2024-02-19T21:18:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0040fd0b0 con 0x7fb0041042c0 2024-02-19T21:18:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1808872174 0 0) 0x7fb0040fcee0 con 0x7fb0040fd980 2024-02-19T21:18:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faff00033f0 con 0x7fb0040fd980 2024-02-19T21:18:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7faffc001420 con 0x7fb0041042c0 2024-02-19T21:18:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7faff4002870 con 0x7fb0040fd980 2024-02-19T21:18:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1215819199 0 0) 0x7fb0040fd2a0 con 0x7fb004100780 2024-02-19T21:18:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0040fcee0 con 0x7fb004100780 2024-02-19T21:18:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb000001830 con 0x7fb004100780 2024-02-19T21:18:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 708060216 0 0) 0x7fb0040fd0b0 con 0x7fb0041042c0 2024-02-19T21:18:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 >> v1:172.21.15.163:6790/0 conn(0x7fb0040fd980 legacy=0x7fb0040fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 >> v1:172.21.15.193:6789/0 conn(0x7fb004100780 legacy=0x7fb004114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00411cb90 con 0x7fb0041042c0 2024-02-19T21:18:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7fb00c74c700 1 -- 172.21.15.163:0/198736836 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb004119bc0 con 0x7fb0041042c0 2024-02-19T21:18:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7fb00c74c700 10 client.23988.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:17.113 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7fb00c74c700 1 -- 172.21.15.163:0/198736836 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00411a030 con 0x7fb0041042c0 2024-02-19T21:18:17.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7fb00c74c700 1 librados: init done 2024-02-19T21:18:17.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7faffc0015b0 con 0x7fb0041042c0 2024-02-19T21:18:17.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.105+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7faffc003b60 con 0x7fb0041042c0 2024-02-19T21:18:17.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.106+0000 7fb00c74c700 1 -- 172.21.15.163:0/198736836 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe8004f20 con 0x7fb0041042c0 2024-02-19T21:18:17.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.107+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7faffc001b80 con 0x7fb0041042c0 2024-02-19T21:18:17.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.107+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(822..822 src has 633..822) v4 ==== 7187+0+0 (unknown 2200427853 0 0) 0x7faffc079060 con 0x7fb0041042c0 2024-02-19T21:18:17.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.107+0000 7faffb7fe700 10 client.23988.objecter ms_dispatch 0x7fb004001550 osd_map(822..822 src has 633..822) v4 2024-02-19T21:18:17.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.107+0000 7faffb7fe700 3 client.23988.objecter handle_osd_map got epochs [822,822] > 0 2024-02-19T21:18:17.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.107+0000 7faffb7fe700 3 client.23988.objecter handle_osd_map decoding full epoch 822 2024-02-19T21:18:17.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.107+0000 7faffb7fe700 20 client.23988.objecter dump_active .. 0 homeless 2024-02-19T21:18:17.114 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.107+0000 7faffb7fe700 10 client.23988.objecter ms_handle_connect 0x7faff0063d00 2024-02-19T21:18:17.115 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.111+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7faffc04c820 con 0x7fb0041042c0 2024-02-19T21:18:17.143 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:16.096585+0000 osd.4 (osd.4) 293 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:18:17.143 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:16.097472+0000 osd.4 (osd.4) 294 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:18:17.143 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:16.712686+0000 osd.6 (osd.6) 295 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:18:17.143 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:16.713476+0000 osd.6 (osd.6) 296 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:18:17.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:16.096585+0000 osd.4 (osd.4) 293 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:18:17.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:16.097472+0000 osd.4 (osd.4) 294 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:18:17.143 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:16.712686+0000 osd.6 (osd.6) 295 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:18:17.144 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:16.713476+0000 osd.6 (osd.6) 296 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:18:17.290 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:17.288+0000 7fb00c74c700 1 -- 172.21.15.163:0/198736836 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) v1 -- 0x7fafe80059e0 con 0x7fb0041042c0 2024-02-19T21:18:17.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:16.096585+0000 osd.4 (osd.4) 293 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:18:17.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:16.097472+0000 osd.4 (osd.4) 294 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:18:17.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:16.712686+0000 osd.6 (osd.6) 295 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:18:17.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:16.713476+0000 osd.6 (osd.6) 296 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:18:17.965 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:18:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:18:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:18:18.153 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:18.151+0000 7faffb7fe700 1 -- 172.21.15.163:0/198736836 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v823) v1 ==== 133+0+0 (unknown 764697318 0 0) 0x7faffc04ac90 con 0x7fb0041042c0 2024-02-19T21:18:18.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:18.155+0000 7fb00c74c700 10 librados: watch_flush enter 2024-02-19T21:18:18.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:18.155+0000 7fb00c74c700 10 librados: watch_flush exit 2024-02-19T21:18:18.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:18.155+0000 7fb00c74c700 20 client.23988.objecter shutdown clearing up homeless session... 2024-02-19T21:18:18.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:18.155+0000 7fb00c74c700 10 client.23988.objecter successfully canceled tick 2024-02-19T21:18:18.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:18.155+0000 7fb00c74c700 1 -- 172.21.15.163:0/198736836 >> v1:172.21.15.163:6800/1552514681 conn(0x7faff0063d00 legacy=0x7faff00661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:18.157 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:18.155+0000 7fb00c74c700 1 -- 172.21.15.163:0/198736836 >> v1:172.21.15.163:6789/0 conn(0x7fb0041042c0 legacy=0x7fb004118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:18.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:18.158+0000 7fb00c74c700 1 -- 172.21.15.163:0/198736836 shutdown_connections 2024-02-19T21:18:18.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:18.158+0000 7fb00c74c700 1 -- 172.21.15.163:0/198736836 >> 172.21.15.163:0/198736836 conn(0x7fb0040f35a0 msgr2=0x7fb004107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:18.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:18.160+0000 7fb00c74c700 1 -- 172.21.15.163:0/198736836 shutdown_connections 2024-02-19T21:18:18.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:18.160+0000 7fb00c74c700 1 -- 172.21.15.163:0/198736836 wait complete. 2024-02-19T21:18:18.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:18.161+0000 7fb00c74c700 1 librados: shutdown 2024-02-19T21:18:18.163 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-02-19T21:18:18.212 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:18:18.212 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:18:18.235 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:17.138648+0000 osd.4 (osd.4) 295 : cluster [DBG] 4.d scrub starts 2024-02-19T21:18:18.235 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:17.140347+0000 osd.4 (osd.4) 296 : cluster [DBG] 4.d scrub ok 2024-02-19T21:18:18.235 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:17.289836+0000 mon.a (mon.0) 3794 : audit [INF] from='client.? v1:172.21.15.163:0/198736836' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:18:18.235 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:17.352534+0000 mgr.y (mgr.14550) 683 : cluster [DBG] pgmap v1252: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:18:18.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:17.138648+0000 osd.4 (osd.4) 295 : cluster [DBG] 4.d scrub starts 2024-02-19T21:18:18.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:17.140347+0000 osd.4 (osd.4) 296 : cluster [DBG] 4.d scrub ok 2024-02-19T21:18:18.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:17.289836+0000 mon.a (mon.0) 3794 : audit [INF] from='client.? v1:172.21.15.163:0/198736836' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:18:18.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:17.352534+0000 mgr.y (mgr.14550) 683 : cluster [DBG] pgmap v1252: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:18:18.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:17.138648+0000 osd.4 (osd.4) 295 : cluster [DBG] 4.d scrub starts 2024-02-19T21:18:18.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:17.140347+0000 osd.4 (osd.4) 296 : cluster [DBG] 4.d scrub ok 2024-02-19T21:18:18.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:17.289836+0000 mon.a (mon.0) 3794 : audit [INF] from='client.? v1:172.21.15.163:0/198736836' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-19T21:18:18.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:17.352534+0000 mgr.y (mgr.14550) 683 : cluster [DBG] pgmap v1252: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:18:18.900 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-19T21:18:18.919 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 1 1 2024-02-19T21:18:19.169 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.b/config 2024-02-19T21:18:19.203 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:19.417 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:18.149441+0000 mon.a (mon.0) 3795 : audit [INF] from='client.? v1:172.21.15.163:0/198736836' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-19T21:18:19.417 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:18.149524+0000 mon.a (mon.0) 3796 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-02-19T21:18:19.417 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:18.186520+0000 osd.4 (osd.4) 297 : cluster [DBG] 5.1 scrub starts 2024-02-19T21:18:19.417 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:18.187348+0000 osd.4 (osd.4) 298 : cluster [DBG] 5.1 scrub ok 2024-02-19T21:18:19.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:18.149441+0000 mon.a (mon.0) 3795 : audit [INF] from='client.? v1:172.21.15.163:0/198736836' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-19T21:18:19.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:18.149524+0000 mon.a (mon.0) 3796 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-02-19T21:18:19.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:18.186520+0000 osd.4 (osd.4) 297 : cluster [DBG] 5.1 scrub starts 2024-02-19T21:18:19.441 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:18.187348+0000 osd.4 (osd.4) 298 : cluster [DBG] 5.1 scrub ok 2024-02-19T21:18:19.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:18.149441+0000 mon.a (mon.0) 3795 : audit [INF] from='client.? v1:172.21.15.163:0/198736836' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-19T21:18:19.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:18.149524+0000 mon.a (mon.0) 3796 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-02-19T21:18:19.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:18.186520+0000 osd.4 (osd.4) 297 : cluster [DBG] 5.1 scrub starts 2024-02-19T21:18:19.442 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:18.187348+0000 osd.4 (osd.4) 298 : cluster [DBG] 5.1 scrub ok 2024-02-19T21:18:20.322 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:19.157610+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-02-19T21:18:20.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:19.161341+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:18:20.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:19.162236+0000 osd.4 (osd.4) 300 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:18:20.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:19.352918+0000 mgr.y (mgr.14550) 684 : cluster [DBG] pgmap v1255: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 982 B/s rd, 0 op/s 2024-02-19T21:18:20.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:19.723632+0000 osd.2 (osd.2) 334 : cluster [DBG] 6.f scrub starts 2024-02-19T21:18:20.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:19.724707+0000 osd.2 (osd.2) 335 : cluster [DBG] 6.f scrub ok 2024-02-19T21:18:20.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:20.102022+0000 mon.a (mon.0) 3798 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:18:20.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:19.157610+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-02-19T21:18:20.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:19.161341+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:18:20.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:19.162236+0000 osd.4 (osd.4) 300 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:18:20.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:19.352918+0000 mgr.y (mgr.14550) 684 : cluster [DBG] pgmap v1255: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 982 B/s rd, 0 op/s 2024-02-19T21:18:20.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:19.723632+0000 osd.2 (osd.2) 334 : cluster [DBG] 6.f scrub starts 2024-02-19T21:18:20.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:19.724707+0000 osd.2 (osd.2) 335 : cluster [DBG] 6.f scrub ok 2024-02-19T21:18:20.499 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:20.102022+0000 mon.a (mon.0) 3798 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:18:20.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:19.157610+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-02-19T21:18:20.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:19.161341+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:18:20.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:19.162236+0000 osd.4 (osd.4) 300 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:18:20.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:19.352918+0000 mgr.y (mgr.14550) 684 : cluster [DBG] pgmap v1255: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 982 B/s rd, 0 op/s 2024-02-19T21:18:20.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:19.723632+0000 osd.2 (osd.2) 334 : cluster [DBG] 6.f scrub starts 2024-02-19T21:18:20.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:19.724707+0000 osd.2 (osd.2) 335 : cluster [DBG] 6.f scrub ok 2024-02-19T21:18:20.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:20.102022+0000 mon.a (mon.0) 3798 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:18:20.951 INFO:teuthology.orchestra.run.smithi193.stdout:ERROR: (22) Invalid argument 2024-02-19T21:18:20.951 INFO:teuthology.orchestra.run.smithi193.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-19T21:18:20.975 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:18:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:18:21.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.119+0000 7f5a6c4d6700 1 -- 172.21.15.163:0/3026998067 >> v1:172.21.15.163:6789/0 conn(0x7f5a64100780 legacy=0x7f5a64102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:21.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.119+0000 7f5a6c4d6700 1 -- 172.21.15.163:0/3026998067 shutdown_connections 2024-02-19T21:18:21.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.119+0000 7f5a6c4d6700 1 -- 172.21.15.163:0/3026998067 >> 172.21.15.163:0/3026998067 conn(0x7f5a640f35a0 msgr2=0x7f5a640f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:21.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.119+0000 7f5a6c4d6700 1 -- 172.21.15.163:0/3026998067 shutdown_connections 2024-02-19T21:18:21.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.119+0000 7f5a6c4d6700 1 -- 172.21.15.163:0/3026998067 wait complete. 2024-02-19T21:18:21.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.120+0000 7f5a6c4d6700 1 librados: starting msgr at 2024-02-19T21:18:21.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.120+0000 7f5a6c4d6700 1 librados: starting objecter 2024-02-19T21:18:21.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.120+0000 7f5a6c4d6700 1 Processor -- start 2024-02-19T21:18:21.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.120+0000 7f5a6c4d6700 1 -- start start 2024-02-19T21:18:21.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.120+0000 7f5a6c4d6700 1 librados: setting wanted keys 2024-02-19T21:18:21.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.120+0000 7f5a6c4d6700 1 librados: calling monclient init 2024-02-19T21:18:21.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.120+0000 7f5a6c4d6700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5a641171c0 con 0x7f5a640fd980 2024-02-19T21:18:21.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.120+0000 7f5a6c4d6700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5a64118360 con 0x7f5a64100780 2024-02-19T21:18:21.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.120+0000 7f5a6c4d6700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5a64119560 con 0x7f5a641042c0 2024-02-19T21:18:21.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.120+0000 7f5a6a272700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f5a640fd980 0x7f5a640f9750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44064/0 (socket says 172.21.15.163:44064) 2024-02-19T21:18:21.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.120+0000 7f5a6a272700 1 -- 172.21.15.163:0/2931740271 learned_addr learned my addr 172.21.15.163:0/2931740271 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:21.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.120+0000 7f5a5b7fe700 10 client.?.objecter ms_handle_connect 0x7f5a641042c0 2024-02-19T21:18:21.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.120+0000 7f5a5b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:21.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.120+0000 7f5a5b7fe700 10 client.?.objecter ms_handle_connect 0x7f5a640fd980 2024-02-19T21:18:21.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.120+0000 7f5a5b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:21.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698669878 0 0) 0x7f5a641171c0 con 0x7f5a640fd980 2024-02-19T21:18:21.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a50003800 con 0x7f5a640fd980 2024-02-19T21:18:21.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 10 client.?.objecter ms_handle_connect 0x7f5a64100780 2024-02-19T21:18:21.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:21.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711222787 0 0) 0x7f5a64119560 con 0x7f5a641042c0 2024-02-19T21:18:21.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a641171c0 con 0x7f5a641042c0 2024-02-19T21:18:21.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48725150 0 0) 0x7f5a64118360 con 0x7f5a64100780 2024-02-19T21:18:21.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a64119560 con 0x7f5a64100780 2024-02-19T21:18:21.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2142904775 0 0) 0x7f5a50003800 con 0x7f5a640fd980 2024-02-19T21:18:21.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a64118360 con 0x7f5a640fd980 2024-02-19T21:18:21.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5a540019d0 con 0x7f5a640fd980 2024-02-19T21:18:21.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 494703267 0 0) 0x7f5a641171c0 con 0x7f5a641042c0 2024-02-19T21:18:21.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a50003800 con 0x7f5a641042c0 2024-02-19T21:18:21.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5a5c001d70 con 0x7f5a641042c0 2024-02-19T21:18:21.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2215845061 0 0) 0x7f5a64119560 con 0x7f5a64100780 2024-02-19T21:18:21.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a641171c0 con 0x7f5a64100780 2024-02-19T21:18:21.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4288960754 0 0) 0x7f5a64118360 con 0x7f5a640fd980 2024-02-19T21:18:21.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 >> v1:172.21.15.163:6790/0 conn(0x7f5a641042c0 legacy=0x7f5a64115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:21.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 >> v1:172.21.15.193:6789/0 conn(0x7f5a64100780 legacy=0x7f5a640f9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:21.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6411a760 con 0x7f5a640fd980 2024-02-19T21:18:21.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a6c4d6700 1 -- 172.21.15.163:0/2931740271 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a641173b0 con 0x7f5a640fd980 2024-02-19T21:18:21.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a6c4d6700 10 client.23994.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:21.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a6c4d6700 1 -- 172.21.15.163:0/2931740271 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a641178a0 con 0x7f5a640fd980 2024-02-19T21:18:21.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a6c4d6700 1 librados: init done 2024-02-19T21:18:21.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f5a540023c0 con 0x7f5a640fd980 2024-02-19T21:18:21.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.121+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5a54005a90 con 0x7f5a640fd980 2024-02-19T21:18:21.127 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.122+0000 7f5a6c4d6700 1 -- 172.21.15.163:0/2931740271 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a48004f20 con 0x7f5a640fd980 2024-02-19T21:18:21.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.123+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f5a54018920 con 0x7f5a640fd980 2024-02-19T21:18:21.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.123+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(824..824 src has 633..824) v4 ==== 7187+0+0 (unknown 249076189 0 0) 0x7f5a54078900 con 0x7f5a640fd980 2024-02-19T21:18:21.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.123+0000 7f5a5b7fe700 10 client.23994.objecter ms_dispatch 0x7f5a64001550 osd_map(824..824 src has 633..824) v4 2024-02-19T21:18:21.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.123+0000 7f5a5b7fe700 3 client.23994.objecter handle_osd_map got epochs [824,824] > 0 2024-02-19T21:18:21.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.123+0000 7f5a5b7fe700 3 client.23994.objecter handle_osd_map decoding full epoch 824 2024-02-19T21:18:21.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.123+0000 7f5a5b7fe700 20 client.23994.objecter dump_active .. 0 homeless 2024-02-19T21:18:21.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.123+0000 7f5a5b7fe700 10 client.23994.objecter ms_handle_connect 0x7f5a50063e90 2024-02-19T21:18:21.128 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.127+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5a5404c0c0 con 0x7f5a640fd980 2024-02-19T21:18:21.239 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:20.188111+0000 osd.4 (osd.4) 301 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:18:21.239 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:20.189979+0000 osd.4 (osd.4) 302 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:18:21.239 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:20.190422+0000 osd.3 (osd.3) 359 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:18:21.239 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:20.191144+0000 osd.3 (osd.3) 360 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:18:21.239 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:20.397036+0000 osd.5 (osd.5) 330 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:18:21.239 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:20.399931+0000 osd.5 (osd.5) 331 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:18:21.239 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:20.864438+0000 mgr.y (mgr.14550) 685 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:21.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:21.303+0000 7f5a6c4d6700 1 -- 172.21.15.163:0/2931740271 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) v1 -- 0x7f5a480051d0 con 0x7f5a640fd980 2024-02-19T21:18:21.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:20.188111+0000 osd.4 (osd.4) 301 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:18:21.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:20.189979+0000 osd.4 (osd.4) 302 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:18:21.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:20.190422+0000 osd.3 (osd.3) 359 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:18:21.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:20.191144+0000 osd.3 (osd.3) 360 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:18:21.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:20.397036+0000 osd.5 (osd.5) 330 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:18:21.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:20.399931+0000 osd.5 (osd.5) 331 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:18:21.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:20.864438+0000 mgr.y (mgr.14550) 685 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:21.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:20.188111+0000 osd.4 (osd.4) 301 : cluster [DBG] 4.13 scrub starts 2024-02-19T21:18:21.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:20.189979+0000 osd.4 (osd.4) 302 : cluster [DBG] 4.13 scrub ok 2024-02-19T21:18:21.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:20.190422+0000 osd.3 (osd.3) 359 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:18:21.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:20.191144+0000 osd.3 (osd.3) 360 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:18:21.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:20.397036+0000 osd.5 (osd.5) 330 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:18:21.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:20.399931+0000 osd.5 (osd.5) 331 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:18:21.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:20.864438+0000 mgr.y (mgr.14550) 685 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:22.172 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:22.170+0000 7f5a5b7fe700 1 -- 172.21.15.163:0/2931740271 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v825) v1 ==== 113+0+0 (unknown 945897597 0 0) 0x7f5a5404a4d0 con 0x7f5a640fd980 2024-02-19T21:18:22.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:22.172+0000 7f5a6c4d6700 10 librados: watch_flush enter 2024-02-19T21:18:22.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:22.172+0000 7f5a6c4d6700 10 librados: watch_flush exit 2024-02-19T21:18:22.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:22.172+0000 7f5a6c4d6700 20 client.23994.objecter shutdown clearing up homeless session... 2024-02-19T21:18:22.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:22.172+0000 7f5a6c4d6700 10 client.23994.objecter successfully canceled tick 2024-02-19T21:18:22.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:22.172+0000 7f5a6c4d6700 1 -- 172.21.15.163:0/2931740271 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5a50063e90 legacy=0x7f5a50066350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:22.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:22.172+0000 7f5a6c4d6700 1 -- 172.21.15.163:0/2931740271 >> v1:172.21.15.163:6789/0 conn(0x7f5a640fd980 legacy=0x7f5a640f9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:22.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:22.172+0000 7f5a6c4d6700 1 -- 172.21.15.163:0/2931740271 shutdown_connections 2024-02-19T21:18:22.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:22.173+0000 7f5a6c4d6700 1 -- 172.21.15.163:0/2931740271 >> 172.21.15.163:0/2931740271 conn(0x7f5a640f35a0 msgr2=0x7f5a64107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:22.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:22.173+0000 7f5a6c4d6700 1 -- 172.21.15.163:0/2931740271 shutdown_connections 2024-02-19T21:18:22.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:22.173+0000 7f5a6c4d6700 1 -- 172.21.15.163:0/2931740271 wait complete. 2024-02-19T21:18:22.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:22.173+0000 7f5a6c4d6700 1 librados: shutdown 2024-02-19T21:18:22.175 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.1 to 1 (10000) 2024-02-19T21:18:22.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:21.218848+0000 osd.3 (osd.3) 361 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:18:22.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:21.219683+0000 osd.3 (osd.3) 362 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:18:22.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:21.225903+0000 osd.4 (osd.4) 303 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:18:22.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:21.226665+0000 osd.4 (osd.4) 304 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:18:22.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:21.304659+0000 mon.a (mon.0) 3799 : audit [INF] from='client.? v1:172.21.15.163:0/2931740271' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:18:22.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:21.353720+0000 mgr.y (mgr.14550) 686 : cluster [DBG] pgmap v1256: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-19T21:18:22.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:21.384257+0000 osd.0 (osd.0) 138 : cluster [DBG] 16.14 deep-scrub starts 2024-02-19T21:18:22.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:21.385242+0000 osd.0 (osd.0) 139 : cluster [DBG] 16.14 deep-scrub ok 2024-02-19T21:18:22.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:21.390698+0000 osd.1 (osd.1) 375 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:18:22.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:21.391499+0000 osd.1 (osd.1) 376 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:18:22.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:21.619602+0000 osd.5 (osd.5) 332 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:22.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:21.619829+0000 osd.5 (osd.5) 333 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:22.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:21.218848+0000 osd.3 (osd.3) 361 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:18:22.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:21.219683+0000 osd.3 (osd.3) 362 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:18:22.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:21.225903+0000 osd.4 (osd.4) 303 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:18:22.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:21.226665+0000 osd.4 (osd.4) 304 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:18:22.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:21.304659+0000 mon.a (mon.0) 3799 : audit [INF] from='client.? v1:172.21.15.163:0/2931740271' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:18:22.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:21.353720+0000 mgr.y (mgr.14550) 686 : cluster [DBG] pgmap v1256: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-19T21:18:22.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:21.384257+0000 osd.0 (osd.0) 138 : cluster [DBG] 16.14 deep-scrub starts 2024-02-19T21:18:22.501 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:21.385242+0000 osd.0 (osd.0) 139 : cluster [DBG] 16.14 deep-scrub ok 2024-02-19T21:18:22.501 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:21.390698+0000 osd.1 (osd.1) 375 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:18:22.501 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:21.391499+0000 osd.1 (osd.1) 376 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:18:22.501 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:21.619602+0000 osd.5 (osd.5) 332 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:22.501 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:21.619829+0000 osd.5 (osd.5) 333 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:22.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:21.218848+0000 osd.3 (osd.3) 361 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:18:22.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:21.219683+0000 osd.3 (osd.3) 362 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:18:22.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:21.225903+0000 osd.4 (osd.4) 303 : cluster [DBG] 5.1a scrub starts 2024-02-19T21:18:22.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:21.226665+0000 osd.4 (osd.4) 304 : cluster [DBG] 5.1a scrub ok 2024-02-19T21:18:22.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:21.304659+0000 mon.a (mon.0) 3799 : audit [INF] from='client.? v1:172.21.15.163:0/2931740271' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:18:22.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:21.353720+0000 mgr.y (mgr.14550) 686 : cluster [DBG] pgmap v1256: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-19T21:18:22.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:21.384257+0000 osd.0 (osd.0) 138 : cluster [DBG] 16.14 deep-scrub starts 2024-02-19T21:18:22.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:21.385242+0000 osd.0 (osd.0) 139 : cluster [DBG] 16.14 deep-scrub ok 2024-02-19T21:18:22.574 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:21.390698+0000 osd.1 (osd.1) 375 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:18:22.574 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:21.391499+0000 osd.1 (osd.1) 376 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:18:22.574 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:21.619602+0000 osd.5 (osd.5) 332 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:22.574 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:21.619829+0000 osd.5 (osd.5) 333 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:22.974 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd primary-affinity 1 1 2024-02-19T21:18:23.246 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:23.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:22.171035+0000 mon.a (mon.0) 3800 : audit [INF] from='client.? v1:172.21.15.163:0/2931740271' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-19T21:18:23.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:22.171105+0000 mon.a (mon.0) 3801 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-02-19T21:18:23.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:22.362537+0000 osd.1 (osd.1) 377 : cluster [DBG] 2.15 deep-scrub starts 2024-02-19T21:18:23.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:22.364369+0000 osd.1 (osd.1) 378 : cluster [DBG] 2.15 deep-scrub ok 2024-02-19T21:18:23.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:22.692757+0000 osd.6 (osd.6) 297 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:18:23.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:22.695722+0000 osd.6 (osd.6) 298 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:18:23.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:22.698634+0000 osd.2 (osd.2) 336 : cluster [DBG] 16.e scrub starts 2024-02-19T21:18:23.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:22.699500+0000 osd.2 (osd.2) 337 : cluster [DBG] 16.e scrub ok 2024-02-19T21:18:23.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:22.171035+0000 mon.a (mon.0) 3800 : audit [INF] from='client.? v1:172.21.15.163:0/2931740271' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-19T21:18:23.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:22.171105+0000 mon.a (mon.0) 3801 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-02-19T21:18:23.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:22.362537+0000 osd.1 (osd.1) 377 : cluster [DBG] 2.15 deep-scrub starts 2024-02-19T21:18:23.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:22.364369+0000 osd.1 (osd.1) 378 : cluster [DBG] 2.15 deep-scrub ok 2024-02-19T21:18:23.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:22.692757+0000 osd.6 (osd.6) 297 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:18:23.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:22.695722+0000 osd.6 (osd.6) 298 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:18:23.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:22.698634+0000 osd.2 (osd.2) 336 : cluster [DBG] 16.e scrub starts 2024-02-19T21:18:23.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:22.699500+0000 osd.2 (osd.2) 337 : cluster [DBG] 16.e scrub ok 2024-02-19T21:18:23.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:22.171035+0000 mon.a (mon.0) 3800 : audit [INF] from='client.? v1:172.21.15.163:0/2931740271' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-19T21:18:23.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:22.171105+0000 mon.a (mon.0) 3801 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-02-19T21:18:23.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:22.362537+0000 osd.1 (osd.1) 377 : cluster [DBG] 2.15 deep-scrub starts 2024-02-19T21:18:23.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:22.364369+0000 osd.1 (osd.1) 378 : cluster [DBG] 2.15 deep-scrub ok 2024-02-19T21:18:23.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:22.692757+0000 osd.6 (osd.6) 297 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:18:23.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:22.695722+0000 osd.6 (osd.6) 298 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:18:23.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:22.698634+0000 osd.2 (osd.2) 336 : cluster [DBG] 16.e scrub starts 2024-02-19T21:18:23.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:22.699500+0000 osd.2 (osd.2) 337 : cluster [DBG] 16.e scrub ok 2024-02-19T21:18:23.741 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:18:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:18:23] "GET /metrics HTTP/1.1" 200 187997 "" "Prometheus/2.33.4" 2024-02-19T21:18:24.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:23.228776+0000 osd.3 (osd.3) 363 : cluster [DBG] 16.4 scrub starts 2024-02-19T21:18:24.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:23.230608+0000 osd.3 (osd.3) 364 : cluster [DBG] 16.4 scrub ok 2024-02-19T21:18:24.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:23.268403+0000 osd.4 (osd.4) 305 : cluster [DBG] 4.e scrub starts 2024-02-19T21:18:24.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:23.269791+0000 osd.4 (osd.4) 306 : cluster [DBG] 4.e scrub ok 2024-02-19T21:18:24.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:23.354080+0000 mgr.y (mgr.14550) 687 : cluster [DBG] pgmap v1258: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:18:24.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:23.432761+0000 osd.5 (osd.5) 334 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:18:24.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:23.435746+0000 osd.5 (osd.5) 335 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:18:24.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:23.228776+0000 osd.3 (osd.3) 363 : cluster [DBG] 16.4 scrub starts 2024-02-19T21:18:24.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:23.230608+0000 osd.3 (osd.3) 364 : cluster [DBG] 16.4 scrub ok 2024-02-19T21:18:24.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:23.268403+0000 osd.4 (osd.4) 305 : cluster [DBG] 4.e scrub starts 2024-02-19T21:18:24.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:23.269791+0000 osd.4 (osd.4) 306 : cluster [DBG] 4.e scrub ok 2024-02-19T21:18:24.501 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:23.354080+0000 mgr.y (mgr.14550) 687 : cluster [DBG] pgmap v1258: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:18:24.501 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:23.432761+0000 osd.5 (osd.5) 334 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:18:24.501 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:23.435746+0000 osd.5 (osd.5) 335 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:18:24.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:23.228776+0000 osd.3 (osd.3) 363 : cluster [DBG] 16.4 scrub starts 2024-02-19T21:18:24.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:23.230608+0000 osd.3 (osd.3) 364 : cluster [DBG] 16.4 scrub ok 2024-02-19T21:18:24.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:23.268403+0000 osd.4 (osd.4) 305 : cluster [DBG] 4.e scrub starts 2024-02-19T21:18:24.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:23.269791+0000 osd.4 (osd.4) 306 : cluster [DBG] 4.e scrub ok 2024-02-19T21:18:24.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:23.354080+0000 mgr.y (mgr.14550) 687 : cluster [DBG] pgmap v1258: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:18:24.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:23.432761+0000 osd.5 (osd.5) 334 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:18:24.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:23.435746+0000 osd.5 (osd.5) 335 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:18:25.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.036+0000 7fcbdac43700 1 -- 172.21.15.163:0/1028100739 >> v1:172.21.15.163:6790/0 conn(0x7fcbd4100770 legacy=0x7fcbd4102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:25.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.036+0000 7fcbdac43700 1 -- 172.21.15.163:0/1028100739 shutdown_connections 2024-02-19T21:18:25.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.036+0000 7fcbdac43700 1 -- 172.21.15.163:0/1028100739 >> 172.21.15.163:0/1028100739 conn(0x7fcbd40f3590 msgr2=0x7fcbd40f59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:25.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.036+0000 7fcbdac43700 1 -- 172.21.15.163:0/1028100739 shutdown_connections 2024-02-19T21:18:25.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.036+0000 7fcbdac43700 1 -- 172.21.15.163:0/1028100739 wait complete. 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.036+0000 7fcbdac43700 1 librados: starting msgr at 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.036+0000 7fcbdac43700 1 librados: starting objecter 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.036+0000 7fcbdac43700 1 Processor -- start 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.036+0000 7fcbdac43700 1 -- start start 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.036+0000 7fcbdac43700 1 librados: setting wanted keys 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.036+0000 7fcbdac43700 1 librados: calling monclient init 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbdac43700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcbd4117420 con 0x7fcbd4100770 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbdac43700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcbd4118600 con 0x7fcbd41042b0 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbdac43700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcbd4119800 con 0x7fcbd40fd970 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd89df700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fcbd40fd970 0x7fcbd40f9c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:37506/0 (socket says 172.21.15.163:37506) 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd89df700 1 -- 172.21.15.163:0/3976730646 learned_addr learned my addr 172.21.15.163:0/3976730646 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd3fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fcbd4100770 0x7fcbd40fa360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:56436/0 (socket says 172.21.15.163:56436) 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 10 client.?.objecter ms_handle_connect 0x7fcbd4100770 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 10 client.?.objecter ms_handle_connect 0x7fcbd40fd970 2024-02-19T21:18:25.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:25.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261811043 0 0) 0x7fcbd4117420 con 0x7fcbd4100770 2024-02-19T21:18:25.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc4003030 con 0x7fcbd4100770 2024-02-19T21:18:25.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 10 client.?.objecter ms_handle_connect 0x7fcbd41042b0 2024-02-19T21:18:25.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:25.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907011563 0 0) 0x7fcbd4119800 con 0x7fcbd40fd970 2024-02-19T21:18:25.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd4117420 con 0x7fcbd40fd970 2024-02-19T21:18:25.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742280484 0 0) 0x7fcbd4118600 con 0x7fcbd41042b0 2024-02-19T21:18:25.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd4119800 con 0x7fcbd41042b0 2024-02-19T21:18:25.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2617351360 0 0) 0x7fcbc4003030 con 0x7fcbd4100770 2024-02-19T21:18:25.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbd4118600 con 0x7fcbd4100770 2024-02-19T21:18:25.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcbc0001f90 con 0x7fcbd4100770 2024-02-19T21:18:25.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 685532069 0 0) 0x7fcbd4117420 con 0x7fcbd40fd970 2024-02-19T21:18:25.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbc4003030 con 0x7fcbd40fd970 2024-02-19T21:18:25.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcbc8001940 con 0x7fcbd40fd970 2024-02-19T21:18:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1928080730 0 0) 0x7fcbd4119800 con 0x7fcbd41042b0 2024-02-19T21:18:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbd4117420 con 0x7fcbd41042b0 2024-02-19T21:18:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1800283254 0 0) 0x7fcbd4118600 con 0x7fcbd4100770 2024-02-19T21:18:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 >> v1:172.21.15.163:6790/0 conn(0x7fcbd40fd970 legacy=0x7fcbd40f9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 >> v1:172.21.15.193:6789/0 conn(0x7fcbd41042b0 legacy=0x7fcbd4115ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd411aa00 con 0x7fcbd4100770 2024-02-19T21:18:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.037+0000 7fcbdac43700 1 -- 172.21.15.163:0/3976730646 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbd41199f0 con 0x7fcbd4100770 2024-02-19T21:18:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.038+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fcbc00022d0 con 0x7fcbd4100770 2024-02-19T21:18:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.038+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fcbc0003950 con 0x7fcbd4100770 2024-02-19T21:18:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.038+0000 7fcbdac43700 10 client.24000.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.038+0000 7fcbdac43700 1 -- 172.21.15.163:0/3976730646 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbd4119ee0 con 0x7fcbd4100770 2024-02-19T21:18:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.038+0000 7fcbdac43700 1 librados: init done 2024-02-19T21:18:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.039+0000 7fcbdac43700 1 -- 172.21.15.163:0/3976730646 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbd4060350 con 0x7fcbd4100770 2024-02-19T21:18:25.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.039+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fcbc0003b90 con 0x7fcbd4100770 2024-02-19T21:18:25.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.039+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(826..826 src has 633..826) v4 ==== 7187+0+0 (unknown 2386560560 0 0) 0x7fcbc0078d30 con 0x7fcbd4100770 2024-02-19T21:18:25.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.039+0000 7fcbd1ffb700 10 client.24000.objecter ms_dispatch 0x7fcbd4001550 osd_map(826..826 src has 633..826) v4 2024-02-19T21:18:25.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.039+0000 7fcbd1ffb700 3 client.24000.objecter handle_osd_map got epochs [826,826] > 0 2024-02-19T21:18:25.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.039+0000 7fcbd1ffb700 3 client.24000.objecter handle_osd_map decoding full epoch 826 2024-02-19T21:18:25.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.039+0000 7fcbd1ffb700 20 client.24000.objecter dump_active .. 0 homeless 2024-02-19T21:18:25.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.039+0000 7fcbd1ffb700 10 client.24000.objecter ms_handle_connect 0x7fcbc4063e10 2024-02-19T21:18:25.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.043+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fcbc004c550 con 0x7fcbd4100770 2024-02-19T21:18:25.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:25.221+0000 7fcbdac43700 1 -- 172.21.15.163:0/3976730646 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) v1 -- 0x7fcbd40f91a0 con 0x7fcbd4100770 2024-02-19T21:18:25.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:23.902157+0000 osd.1 (osd.1) 379 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:25.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:23.902511+0000 osd.1 (osd.1) 380 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:25.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:24.178429+0000 mon.a (mon.0) 3802 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-02-19T21:18:25.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:24.250886+0000 osd.3 (osd.3) 365 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:18:25.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:24.251804+0000 osd.3 (osd.3) 366 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:18:25.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:24.259882+0000 osd.4 (osd.4) 307 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:18:25.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:24.261728+0000 osd.4 (osd.4) 308 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:18:25.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:23.902157+0000 osd.1 (osd.1) 379 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:25.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:23.902511+0000 osd.1 (osd.1) 380 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:25.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:24.178429+0000 mon.a (mon.0) 3802 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-02-19T21:18:25.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:24.250886+0000 osd.3 (osd.3) 365 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:18:25.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:24.251804+0000 osd.3 (osd.3) 366 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:18:25.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:24.259882+0000 osd.4 (osd.4) 307 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:18:25.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:24.261728+0000 osd.4 (osd.4) 308 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:18:25.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:23.902157+0000 osd.1 (osd.1) 379 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:25.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:23.902511+0000 osd.1 (osd.1) 380 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:25.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:24.178429+0000 mon.a (mon.0) 3802 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-02-19T21:18:25.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:24.250886+0000 osd.3 (osd.3) 365 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:18:25.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:24.251804+0000 osd.3 (osd.3) 366 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:18:25.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:24.259882+0000 osd.4 (osd.4) 307 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:18:25.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:24.261728+0000 osd.4 (osd.4) 308 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:18:26.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:26.182+0000 7fcbd1ffb700 1 -- 172.21.15.163:0/3976730646 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v827) v1 ==== 133+0+0 (unknown 2339008553 0 0) 0x7fcbc004a9c0 con 0x7fcbd4100770 2024-02-19T21:18:26.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:26.185+0000 7fcbdac43700 10 librados: watch_flush enter 2024-02-19T21:18:26.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:26.185+0000 7fcbdac43700 10 librados: watch_flush exit 2024-02-19T21:18:26.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:26.185+0000 7fcbdac43700 20 client.24000.objecter shutdown clearing up homeless session... 2024-02-19T21:18:26.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:26.185+0000 7fcbdac43700 10 client.24000.objecter successfully canceled tick 2024-02-19T21:18:26.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:26.185+0000 7fcbdac43700 1 -- 172.21.15.163:0/3976730646 >> v1:172.21.15.163:6800/1552514681 conn(0x7fcbc4063e10 legacy=0x7fcbc40662d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:26.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:26.185+0000 7fcbdac43700 1 -- 172.21.15.163:0/3976730646 >> v1:172.21.15.163:6789/0 conn(0x7fcbd4100770 legacy=0x7fcbd40fa360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:26.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:26.187+0000 7fcbdac43700 1 -- 172.21.15.163:0/3976730646 shutdown_connections 2024-02-19T21:18:26.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:26.187+0000 7fcbdac43700 1 -- 172.21.15.163:0/3976730646 >> 172.21.15.163:0/3976730646 conn(0x7fcbd40f3590 msgr2=0x7fcbd4107700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:26.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:26.187+0000 7fcbdac43700 1 -- 172.21.15.163:0/3976730646 shutdown_connections 2024-02-19T21:18:26.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:26.187+0000 7fcbdac43700 1 -- 172.21.15.163:0/3976730646 wait complete. 2024-02-19T21:18:26.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:26.187+0000 7fcbdac43700 1 librados: shutdown 2024-02-19T21:18:26.189 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-02-19T21:18:26.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:25.222599+0000 mon.a (mon.0) 3803 : audit [INF] from='client.? v1:172.21.15.163:0/3976730646' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:18:26.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:25.259483+0000 osd.3 (osd.3) 367 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:18:26.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:25.261043+0000 osd.3 (osd.3) 368 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:18:26.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:25.318620+0000 osd.0 (osd.0) 140 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:18:26.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:25.321776+0000 osd.0 (osd.0) 141 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:18:26.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:25.354817+0000 mgr.y (mgr.14550) 688 : cluster [DBG] pgmap v1260: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-19T21:18:26.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:25.222599+0000 mon.a (mon.0) 3803 : audit [INF] from='client.? v1:172.21.15.163:0/3976730646' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:18:26.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:25.259483+0000 osd.3 (osd.3) 367 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:18:26.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:25.261043+0000 osd.3 (osd.3) 368 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:18:26.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:25.318620+0000 osd.0 (osd.0) 140 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:18:26.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:25.321776+0000 osd.0 (osd.0) 141 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:18:26.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:25.354817+0000 mgr.y (mgr.14550) 688 : cluster [DBG] pgmap v1260: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-19T21:18:26.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:25.222599+0000 mon.a (mon.0) 3803 : audit [INF] from='client.? v1:172.21.15.163:0/3976730646' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-19T21:18:26.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:25.259483+0000 osd.3 (osd.3) 367 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:18:26.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:25.261043+0000 osd.3 (osd.3) 368 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:18:26.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:25.318620+0000 osd.0 (osd.0) 140 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:18:26.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:25.321776+0000 osd.0 (osd.0) 141 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:18:26.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:25.354817+0000 mgr.y (mgr.14550) 688 : cluster [DBG] pgmap v1260: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-19T21:18:27.411 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:26.183721+0000 mon.a (mon.0) 3804 : audit [INF] from='client.? v1:172.21.15.163:0/3976730646' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-19T21:18:27.411 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:26.183783+0000 mon.a (mon.0) 3805 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-02-19T21:18:27.411 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:26.220743+0000 osd.3 (osd.3) 369 : cluster [DBG] 6.b scrub starts 2024-02-19T21:18:27.411 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:26.221894+0000 osd.3 (osd.3) 370 : cluster [DBG] 6.b scrub ok 2024-02-19T21:18:27.411 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:26.259188+0000 osd.4 (osd.4) 309 : cluster [DBG] 6.2 scrub starts 2024-02-19T21:18:27.411 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:26.260002+0000 osd.4 (osd.4) 310 : cluster [DBG] 6.2 scrub ok 2024-02-19T21:18:27.411 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:26.442435+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:18:27.412 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:26.445524+0000 osd.1 (osd.1) 382 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:18:27.412 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:26.183721+0000 mon.a (mon.0) 3804 : audit [INF] from='client.? v1:172.21.15.163:0/3976730646' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-19T21:18:27.412 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:26.183783+0000 mon.a (mon.0) 3805 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-02-19T21:18:27.412 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:26.220743+0000 osd.3 (osd.3) 369 : cluster [DBG] 6.b scrub starts 2024-02-19T21:18:27.412 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:26.221894+0000 osd.3 (osd.3) 370 : cluster [DBG] 6.b scrub ok 2024-02-19T21:18:27.412 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:26.259188+0000 osd.4 (osd.4) 309 : cluster [DBG] 6.2 scrub starts 2024-02-19T21:18:27.412 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:26.260002+0000 osd.4 (osd.4) 310 : cluster [DBG] 6.2 scrub ok 2024-02-19T21:18:27.412 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:26.442435+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:18:27.412 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:26.445524+0000 osd.1 (osd.1) 382 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:18:27.434 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 2 1 2024-02-19T21:18:27.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:26.183721+0000 mon.a (mon.0) 3804 : audit [INF] from='client.? v1:172.21.15.163:0/3976730646' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-19T21:18:27.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:26.183783+0000 mon.a (mon.0) 3805 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-02-19T21:18:27.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:26.220743+0000 osd.3 (osd.3) 369 : cluster [DBG] 6.b scrub starts 2024-02-19T21:18:27.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:26.221894+0000 osd.3 (osd.3) 370 : cluster [DBG] 6.b scrub ok 2024-02-19T21:18:27.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:26.259188+0000 osd.4 (osd.4) 309 : cluster [DBG] 6.2 scrub starts 2024-02-19T21:18:27.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:26.260002+0000 osd.4 (osd.4) 310 : cluster [DBG] 6.2 scrub ok 2024-02-19T21:18:27.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:26.442435+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:18:27.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:26.445524+0000 osd.1 (osd.1) 382 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:18:27.707 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:28.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:18:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:18:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:18:28.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:27.355319+0000 mgr.y (mgr.14550) 689 : cluster [DBG] pgmap v1262: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:18:28.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:27.718997+0000 osd.6 (osd.6) 299 : cluster [DBG] 16.d scrub starts 2024-02-19T21:18:28.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:27.721642+0000 osd.6 (osd.6) 300 : cluster [DBG] 16.d scrub ok 2024-02-19T21:18:28.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:27.761108+0000 osd.2 (osd.2) 338 : cluster [DBG] 16.1d scrub starts 2024-02-19T21:18:28.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:27.763130+0000 osd.2 (osd.2) 339 : cluster [DBG] 16.1d scrub ok 2024-02-19T21:18:28.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:27.355319+0000 mgr.y (mgr.14550) 689 : cluster [DBG] pgmap v1262: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:18:28.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:27.718997+0000 osd.6 (osd.6) 299 : cluster [DBG] 16.d scrub starts 2024-02-19T21:18:28.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:27.721642+0000 osd.6 (osd.6) 300 : cluster [DBG] 16.d scrub ok 2024-02-19T21:18:28.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:27.761108+0000 osd.2 (osd.2) 338 : cluster [DBG] 16.1d scrub starts 2024-02-19T21:18:28.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:27.763130+0000 osd.2 (osd.2) 339 : cluster [DBG] 16.1d scrub ok 2024-02-19T21:18:28.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:27.355319+0000 mgr.y (mgr.14550) 689 : cluster [DBG] pgmap v1262: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:18:28.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:27.718997+0000 osd.6 (osd.6) 299 : cluster [DBG] 16.d scrub starts 2024-02-19T21:18:28.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:27.721642+0000 osd.6 (osd.6) 300 : cluster [DBG] 16.d scrub ok 2024-02-19T21:18:28.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:27.761108+0000 osd.2 (osd.2) 338 : cluster [DBG] 16.1d scrub starts 2024-02-19T21:18:28.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:27.763130+0000 osd.2 (osd.2) 339 : cluster [DBG] 16.1d scrub ok 2024-02-19T21:18:29.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:28.240717+0000 osd.3 (osd.3) 371 : cluster [DBG] 2.23 scrub starts 2024-02-19T21:18:29.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:28.241992+0000 osd.3 (osd.3) 372 : cluster [DBG] 2.23 scrub ok 2024-02-19T21:18:29.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:28.424776+0000 osd.5 (osd.5) 336 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:18:29.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:28.428238+0000 osd.5 (osd.5) 337 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:18:29.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:28.499798+0000 osd.1 (osd.1) 383 : cluster [DBG] 5.c scrub starts 2024-02-19T21:18:29.248 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:28.500938+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.c scrub ok 2024-02-19T21:18:29.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:28.240717+0000 osd.3 (osd.3) 371 : cluster [DBG] 2.23 scrub starts 2024-02-19T21:18:29.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:28.241992+0000 osd.3 (osd.3) 372 : cluster [DBG] 2.23 scrub ok 2024-02-19T21:18:29.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:28.424776+0000 osd.5 (osd.5) 336 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:18:29.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:28.428238+0000 osd.5 (osd.5) 337 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:18:29.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:28.499798+0000 osd.1 (osd.1) 383 : cluster [DBG] 5.c scrub starts 2024-02-19T21:18:29.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:28.500938+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.c scrub ok 2024-02-19T21:18:29.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.461+0000 7f1f35a26700 1 -- 172.21.15.163:0/398206959 >> v1:172.21.15.163:6790/0 conn(0x7f1f300fc870 legacy=0x7f1f300fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:29.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.461+0000 7f1f35a26700 1 -- 172.21.15.163:0/398206959 shutdown_connections 2024-02-19T21:18:29.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.461+0000 7f1f35a26700 1 -- 172.21.15.163:0/398206959 >> 172.21.15.163:0/398206959 conn(0x7f1f300f2470 msgr2=0x7f1f300f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:29.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.461+0000 7f1f35a26700 1 -- 172.21.15.163:0/398206959 shutdown_connections 2024-02-19T21:18:29.463 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.461+0000 7f1f35a26700 1 -- 172.21.15.163:0/398206959 wait complete. 2024-02-19T21:18:29.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.462+0000 7f1f35a26700 1 librados: starting msgr at 2024-02-19T21:18:29.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.462+0000 7f1f35a26700 1 librados: starting objecter 2024-02-19T21:18:29.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.462+0000 7f1f35a26700 1 Processor -- start 2024-02-19T21:18:29.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.462+0000 7f1f35a26700 1 -- start start 2024-02-19T21:18:29.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.462+0000 7f1f35a26700 1 librados: setting wanted keys 2024-02-19T21:18:29.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.462+0000 7f1f35a26700 1 librados: calling monclient init 2024-02-19T21:18:29.464 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f35a26700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1f301161b0 con 0x7f1f300fc870 2024-02-19T21:18:29.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f35a26700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1f30117390 con 0x7f1f301031b0 2024-02-19T21:18:29.465 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f35a26700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1f30118590 con 0x7f1f300ff670 2024-02-19T21:18:29.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f2e7fc700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f1f300ff670 0x7f1f300f8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:37534/0 (socket says 172.21.15.163:37534) 2024-02-19T21:18:29.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f2e7fc700 1 -- 172.21.15.163:0/1719908383 learned_addr learned my addr 172.21.15.163:0/1719908383 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:29.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 10 client.?.objecter ms_handle_connect 0x7f1f300ff670 2024-02-19T21:18:29.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:29.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 10 client.?.objecter ms_handle_connect 0x7f1f300fc870 2024-02-19T21:18:29.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:29.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 10 client.?.objecter ms_handle_connect 0x7f1f301031b0 2024-02-19T21:18:29.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:29.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862658788 0 0) 0x7f1f30118590 con 0x7f1f300ff670 2024-02-19T21:18:29.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f18003800 con 0x7f1f300ff670 2024-02-19T21:18:29.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035081008 0 0) 0x7f1f301161b0 con 0x7f1f300fc870 2024-02-19T21:18:29.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f30118590 con 0x7f1f300fc870 2024-02-19T21:18:29.467 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933211714 0 0) 0x7f1f30117390 con 0x7f1f301031b0 2024-02-19T21:18:29.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f301161b0 con 0x7f1f301031b0 2024-02-19T21:18:29.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1487711438 0 0) 0x7f1f30118590 con 0x7f1f300fc870 2024-02-19T21:18:29.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f30117390 con 0x7f1f300fc870 2024-02-19T21:18:29.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2972172778 0 0) 0x7f1f18003800 con 0x7f1f300ff670 2024-02-19T21:18:29.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f30118590 con 0x7f1f300ff670 2024-02-19T21:18:29.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4136688740 0 0) 0x7f1f301161b0 con 0x7f1f301031b0 2024-02-19T21:18:29.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f18003800 con 0x7f1f301031b0 2024-02-19T21:18:29.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1f20002f60 con 0x7f1f300fc870 2024-02-19T21:18:29.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.463+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1f240018d0 con 0x7f1f300ff670 2024-02-19T21:18:29.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.464+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1f28001bc0 con 0x7f1f301031b0 2024-02-19T21:18:29.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.464+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3749059482 0 0) 0x7f1f18003800 con 0x7f1f301031b0 2024-02-19T21:18:29.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.464+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 >> v1:172.21.15.163:6790/0 conn(0x7f1f300ff670 legacy=0x7f1f300f8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:29.468 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.464+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 >> v1:172.21.15.163:6789/0 conn(0x7f1f300fc870 legacy=0x7f1f300f8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:29.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.464+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f30119790 con 0x7f1f301031b0 2024-02-19T21:18:29.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.464+0000 7f1f35a26700 1 -- 172.21.15.163:0/1719908383 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f301163a0 con 0x7f1f301031b0 2024-02-19T21:18:29.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.464+0000 7f1f35a26700 10 client.49543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:29.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.464+0000 7f1f35a26700 1 -- 172.21.15.163:0/1719908383 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f30116810 con 0x7f1f301031b0 2024-02-19T21:18:29.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.464+0000 7f1f35a26700 1 librados: init done 2024-02-19T21:18:29.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.464+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f1f28002200 con 0x7f1f301031b0 2024-02-19T21:18:29.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.464+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f1f28003420 con 0x7f1f301031b0 2024-02-19T21:18:29.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.465+0000 7f1f35a26700 1 -- 172.21.15.163:0/1719908383 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1c004f20 con 0x7f1f301031b0 2024-02-19T21:18:29.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.466+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f1f28001d20 con 0x7f1f301031b0 2024-02-19T21:18:29.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.466+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(828..828 src has 633..828) v4 ==== 7187+0+0 (unknown 168153334 0 0) 0x7f1f28076350 con 0x7f1f301031b0 2024-02-19T21:18:29.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.466+0000 7f1f17fff700 10 client.49543.objecter ms_dispatch 0x7f1f30001550 osd_map(828..828 src has 633..828) v4 2024-02-19T21:18:29.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.466+0000 7f1f17fff700 3 client.49543.objecter handle_osd_map got epochs [828,828] > 0 2024-02-19T21:18:29.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.466+0000 7f1f17fff700 3 client.49543.objecter handle_osd_map decoding full epoch 828 2024-02-19T21:18:29.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.466+0000 7f1f17fff700 20 client.49543.objecter dump_active .. 0 homeless 2024-02-19T21:18:29.470 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.466+0000 7f1f17fff700 10 client.49543.objecter ms_handle_connect 0x7f1f18066010 2024-02-19T21:18:29.472 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.471+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f1f28049ab0 con 0x7f1f301031b0 2024-02-19T21:18:29.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:28.240717+0000 osd.3 (osd.3) 371 : cluster [DBG] 2.23 scrub starts 2024-02-19T21:18:29.572 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:28.241992+0000 osd.3 (osd.3) 372 : cluster [DBG] 2.23 scrub ok 2024-02-19T21:18:29.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:28.424776+0000 osd.5 (osd.5) 336 : cluster [DBG] 5.15 scrub starts 2024-02-19T21:18:29.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:28.428238+0000 osd.5 (osd.5) 337 : cluster [DBG] 5.15 scrub ok 2024-02-19T21:18:29.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:28.499798+0000 osd.1 (osd.1) 383 : cluster [DBG] 5.c scrub starts 2024-02-19T21:18:29.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:28.500938+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.c scrub ok 2024-02-19T21:18:29.651 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:29.649+0000 7f1f35a26700 1 -- 172.21.15.163:0/1719908383 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) v1 -- 0x7f1f1c0051d0 con 0x7f1f301031b0 2024-02-19T21:18:30.449 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:30.447+0000 7f1f17fff700 1 -- 172.21.15.163:0/1719908383 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v829) v1 ==== 113+0+0 (unknown 1360966034 0 0) 0x7f1f28047ec0 con 0x7f1f301031b0 2024-02-19T21:18:30.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:30.451+0000 7f1f35a26700 10 librados: watch_flush enter 2024-02-19T21:18:30.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:30.451+0000 7f1f35a26700 10 librados: watch_flush exit 2024-02-19T21:18:30.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:30.451+0000 7f1f35a26700 20 client.49543.objecter shutdown clearing up homeless session... 2024-02-19T21:18:30.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:30.451+0000 7f1f35a26700 10 client.49543.objecter successfully canceled tick 2024-02-19T21:18:30.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:30.451+0000 7f1f35a26700 1 -- 172.21.15.163:0/1719908383 >> v1:172.21.15.163:6800/1552514681 conn(0x7f1f18066010 legacy=0x7f1f180684d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:30.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:30.452+0000 7f1f35a26700 1 -- 172.21.15.163:0/1719908383 >> v1:172.21.15.193:6789/0 conn(0x7f1f301031b0 legacy=0x7f1f30114870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:30.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:30.452+0000 7f1f35a26700 1 -- 172.21.15.163:0/1719908383 shutdown_connections 2024-02-19T21:18:30.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:30.452+0000 7f1f35a26700 1 -- 172.21.15.163:0/1719908383 >> 172.21.15.163:0/1719908383 conn(0x7f1f300f2470 msgr2=0x7f1f30106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:30.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:30.452+0000 7f1f35a26700 1 -- 172.21.15.163:0/1719908383 shutdown_connections 2024-02-19T21:18:30.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:30.452+0000 7f1f35a26700 1 -- 172.21.15.163:0/1719908383 wait complete. 2024-02-19T21:18:30.454 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:30.452+0000 7f1f35a26700 1 librados: shutdown 2024-02-19T21:18:30.455 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.2 to 1 (10000) 2024-02-19T21:18:30.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:29.315470+0000 osd.0 (osd.0) 142 : cluster [DBG] 16.15 scrub starts 2024-02-19T21:18:31.199 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:29.317035+0000 osd.0 (osd.0) 143 : cluster [DBG] 16.15 scrub ok 2024-02-19T21:18:31.199 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:29.355706+0000 mgr.y (mgr.14550) 690 : cluster [DBG] pgmap v1263: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 712 B/s rd, 0 op/s 2024-02-19T21:18:31.199 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:29.439805+0000 mon.a (mon.0) 3806 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-02-19T21:18:31.199 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:29.451404+0000 osd.5 (osd.5) 338 : cluster [DBG] 2.18 deep-scrub starts 2024-02-19T21:18:31.200 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:29.454653+0000 osd.5 (osd.5) 339 : cluster [DBG] 2.18 deep-scrub ok 2024-02-19T21:18:31.200 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:29.524285+0000 osd.1 (osd.1) 385 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:18:31.200 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:29.527728+0000 osd.1 (osd.1) 386 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:18:31.200 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:29.651218+0000 mon.b (mon.1) 761 : audit [INF] from='client.? v1:172.21.15.163:0/1719908383' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:18:31.200 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:29.651877+0000 mon.a (mon.0) 3807 : audit [INF] from='client.49543 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:18:31.200 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:29.710139+0000 osd.2 (osd.2) 340 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:18:31.200 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:29.713203+0000 osd.2 (osd.2) 341 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:18:31.200 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:29.315470+0000 osd.0 (osd.0) 142 : cluster [DBG] 16.15 scrub starts 2024-02-19T21:18:31.200 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:29.317035+0000 osd.0 (osd.0) 143 : cluster [DBG] 16.15 scrub ok 2024-02-19T21:18:31.200 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:29.355706+0000 mgr.y (mgr.14550) 690 : cluster [DBG] pgmap v1263: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 712 B/s rd, 0 op/s 2024-02-19T21:18:31.201 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:29.439805+0000 mon.a (mon.0) 3806 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-02-19T21:18:31.201 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:29.451404+0000 osd.5 (osd.5) 338 : cluster [DBG] 2.18 deep-scrub starts 2024-02-19T21:18:31.201 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:29.454653+0000 osd.5 (osd.5) 339 : cluster [DBG] 2.18 deep-scrub ok 2024-02-19T21:18:31.201 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:29.524285+0000 osd.1 (osd.1) 385 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:18:31.201 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:29.527728+0000 osd.1 (osd.1) 386 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:18:31.201 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:29.651218+0000 mon.b (mon.1) 761 : audit [INF] from='client.? v1:172.21.15.163:0/1719908383' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:18:31.201 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:29.651877+0000 mon.a (mon.0) 3807 : audit [INF] from='client.49543 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:18:31.201 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:29.710139+0000 osd.2 (osd.2) 340 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:18:31.201 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:29.713203+0000 osd.2 (osd.2) 341 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:18:31.202 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:29.315470+0000 osd.0 (osd.0) 142 : cluster [DBG] 16.15 scrub starts 2024-02-19T21:18:31.202 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:29.317035+0000 osd.0 (osd.0) 143 : cluster [DBG] 16.15 scrub ok 2024-02-19T21:18:31.202 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:29.355706+0000 mgr.y (mgr.14550) 690 : cluster [DBG] pgmap v1263: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 712 B/s rd, 0 op/s 2024-02-19T21:18:31.202 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:29.439805+0000 mon.a (mon.0) 3806 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-02-19T21:18:31.202 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:29.451404+0000 osd.5 (osd.5) 338 : cluster [DBG] 2.18 deep-scrub starts 2024-02-19T21:18:31.202 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:29.454653+0000 osd.5 (osd.5) 339 : cluster [DBG] 2.18 deep-scrub ok 2024-02-19T21:18:31.202 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:29.524285+0000 osd.1 (osd.1) 385 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:18:31.202 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:29.527728+0000 osd.1 (osd.1) 386 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:18:31.202 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:29.651218+0000 mon.b (mon.1) 761 : audit [INF] from='client.? v1:172.21.15.163:0/1719908383' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:18:31.202 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:29.651877+0000 mon.a (mon.0) 3807 : audit [INF] from='client.49543 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:18:31.202 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:29.710139+0000 osd.2 (osd.2) 340 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:18:31.202 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:29.713203+0000 osd.2 (osd.2) 341 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:18:31.232 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd primary-affinity 2 1 2024-02-19T21:18:31.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:18:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:18:31.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:30.422864+0000 osd.5 (osd.5) 340 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:18:31.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:30.425854+0000 osd.5 (osd.5) 341 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:18:31.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:30.447500+0000 mon.a (mon.0) 3808 : audit [INF] from='client.49543 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-19T21:18:31.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:30.447551+0000 mon.a (mon.0) 3809 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-02-19T21:18:31.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:30.673714+0000 osd.2 (osd.2) 342 : cluster [DBG] 4.1c deep-scrub starts 2024-02-19T21:18:31.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:30.676419+0000 osd.2 (osd.2) 343 : cluster [DBG] 4.1c deep-scrub ok 2024-02-19T21:18:31.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:30.873351+0000 mgr.y (mgr.14550) 691 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:31.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:30.422864+0000 osd.5 (osd.5) 340 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:18:31.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:30.425854+0000 osd.5 (osd.5) 341 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:18:31.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:30.447500+0000 mon.a (mon.0) 3808 : audit [INF] from='client.49543 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-19T21:18:31.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:30.447551+0000 mon.a (mon.0) 3809 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-02-19T21:18:31.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:30.673714+0000 osd.2 (osd.2) 342 : cluster [DBG] 4.1c deep-scrub starts 2024-02-19T21:18:31.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:30.676419+0000 osd.2 (osd.2) 343 : cluster [DBG] 4.1c deep-scrub ok 2024-02-19T21:18:31.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:30.873351+0000 mgr.y (mgr.14550) 691 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:31.506 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:31.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:30.422864+0000 osd.5 (osd.5) 340 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:18:31.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:30.425854+0000 osd.5 (osd.5) 341 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:18:31.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:30.447500+0000 mon.a (mon.0) 3808 : audit [INF] from='client.49543 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-19T21:18:31.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:30.447551+0000 mon.a (mon.0) 3809 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-02-19T21:18:31.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:30.673714+0000 osd.2 (osd.2) 342 : cluster [DBG] 4.1c deep-scrub starts 2024-02-19T21:18:31.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:30.676419+0000 osd.2 (osd.2) 343 : cluster [DBG] 4.1c deep-scrub ok 2024-02-19T21:18:31.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:30.873351+0000 mgr.y (mgr.14550) 691 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:32.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:31.356083+0000 mgr.y (mgr.14550) 692 : cluster [DBG] pgmap v1266: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:18:32.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:31.533005+0000 osd.1 (osd.1) 387 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:18:32.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:31.537106+0000 osd.1 (osd.1) 388 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:18:32.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:31.715149+0000 osd.2 (osd.2) 344 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:18:32.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:31.718159+0000 osd.2 (osd.2) 345 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:18:32.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:31.356083+0000 mgr.y (mgr.14550) 692 : cluster [DBG] pgmap v1266: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:18:32.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:31.533005+0000 osd.1 (osd.1) 387 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:18:32.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:31.537106+0000 osd.1 (osd.1) 388 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:18:32.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:31.715149+0000 osd.2 (osd.2) 344 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:18:32.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:31.718159+0000 osd.2 (osd.2) 345 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:18:32.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:31.356083+0000 mgr.y (mgr.14550) 692 : cluster [DBG] pgmap v1266: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:18:32.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:31.533005+0000 osd.1 (osd.1) 387 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:18:32.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:31.537106+0000 osd.1 (osd.1) 388 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:18:32.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:31.715149+0000 osd.2 (osd.2) 344 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:18:32.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:31.718159+0000 osd.2 (osd.2) 345 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:18:33.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.245+0000 7faeb4d64700 1 -- 172.21.15.163:0/3888864081 >> v1:172.21.15.163:6789/0 conn(0x7faeb0100780 legacy=0x7faeb0102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:33.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.245+0000 7faeb4d64700 1 -- 172.21.15.163:0/3888864081 shutdown_connections 2024-02-19T21:18:33.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.245+0000 7faeb4d64700 1 -- 172.21.15.163:0/3888864081 >> 172.21.15.163:0/3888864081 conn(0x7faeb00f35a0 msgr2=0x7faeb00f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:33.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.245+0000 7faeb4d64700 1 -- 172.21.15.163:0/3888864081 shutdown_connections 2024-02-19T21:18:33.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.245+0000 7faeb4d64700 1 -- 172.21.15.163:0/3888864081 wait complete. 2024-02-19T21:18:33.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.245+0000 7faeb4d64700 1 librados: starting msgr at 2024-02-19T21:18:33.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.245+0000 7faeb4d64700 1 librados: starting objecter 2024-02-19T21:18:33.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7faeb4d64700 1 Processor -- start 2024-02-19T21:18:33.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7faeb4d64700 1 -- start start 2024-02-19T21:18:33.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7faeb4d64700 1 librados: setting wanted keys 2024-02-19T21:18:33.249 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7faeb4d64700 1 librados: calling monclient init 2024-02-19T21:18:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7faeb4d64700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faeb0117160 con 0x7faeb0100780 2024-02-19T21:18:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7faeb4d64700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faeb0118340 con 0x7faeb00fd980 2024-02-19T21:18:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7faeb4d64700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7faeb0119540 con 0x7faeb01042c0 2024-02-19T21:18:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7faeaed9d700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7faeb01042c0 0x7faeb0115940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:37540/0 (socket says 172.21.15.163:37540) 2024-02-19T21:18:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7faeaed9d700 1 -- 172.21.15.163:0/3928677302 learned_addr learned my addr 172.21.15.163:0/3928677302 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7faeadd9b700 1 --1- 172.21.15.163:0/3928677302 >> v1:172.21.15.163:6789/0 conn(0x7faeb0100780 0x7faeb00f9e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:56464/0 (socket says 172.21.15.163:56464) 2024-02-19T21:18:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 10 client.?.objecter ms_handle_connect 0x7faeb0100780 2024-02-19T21:18:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 10 client.?.objecter ms_handle_connect 0x7faeb01042c0 2024-02-19T21:18:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 10 client.?.objecter ms_handle_connect 0x7faeb00fd980 2024-02-19T21:18:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919148757 0 0) 0x7faeb0117160 con 0x7faeb0100780 2024-02-19T21:18:33.250 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae980038e0 con 0x7faeb0100780 2024-02-19T21:18:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011431446 0 0) 0x7faeb0119540 con 0x7faeb01042c0 2024-02-19T21:18:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeb0117160 con 0x7faeb01042c0 2024-02-19T21:18:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144202757 0 0) 0x7faeb0118340 con 0x7faeb00fd980 2024-02-19T21:18:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeb0119540 con 0x7faeb00fd980 2024-02-19T21:18:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 26876022 0 0) 0x7fae980038e0 con 0x7faeb0100780 2024-02-19T21:18:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faeb0118340 con 0x7faeb0100780 2024-02-19T21:18:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7faea4001ff0 con 0x7faeb0100780 2024-02-19T21:18:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3484589843 0 0) 0x7faeb0117160 con 0x7faeb01042c0 2024-02-19T21:18:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae980038e0 con 0x7faeb01042c0 2024-02-19T21:18:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.246+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7faea8001c50 con 0x7faeb01042c0 2024-02-19T21:18:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.247+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3367897767 0 0) 0x7faeb0119540 con 0x7faeb00fd980 2024-02-19T21:18:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.247+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faeb0117160 con 0x7faeb00fd980 2024-02-19T21:18:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.247+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7faea00019f0 con 0x7faeb00fd980 2024-02-19T21:18:33.251 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.247+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2316506530 0 0) 0x7faeb0118340 con 0x7faeb0100780 2024-02-19T21:18:33.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.247+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 >> v1:172.21.15.163:6790/0 conn(0x7faeb01042c0 legacy=0x7faeb0115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:33.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.247+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 >> v1:172.21.15.193:6789/0 conn(0x7faeb00fd980 legacy=0x7faeb00f9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:33.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.247+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb011a740 con 0x7faeb0100780 2024-02-19T21:18:33.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.247+0000 7faeb4d64700 1 -- 172.21.15.163:0/3928677302 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb0119730 con 0x7faeb0100780 2024-02-19T21:18:33.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.247+0000 7faeb4d64700 10 client.24012.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:33.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.247+0000 7faeb4d64700 1 -- 172.21.15.163:0/3928677302 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb0119c20 con 0x7faeb0100780 2024-02-19T21:18:33.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.247+0000 7faeb4d64700 1 librados: init done 2024-02-19T21:18:33.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.247+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7faea40019f0 con 0x7faeb0100780 2024-02-19T21:18:33.252 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.247+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7faea4003720 con 0x7faeb0100780 2024-02-19T21:18:33.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.248+0000 7faeb4d64700 1 -- 172.21.15.163:0/3928677302 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae90004f20 con 0x7faeb0100780 2024-02-19T21:18:33.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.248+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7faea4002ce0 con 0x7faeb0100780 2024-02-19T21:18:33.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.249+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(829..829 src has 633..829) v4 ==== 7187+0+0 (unknown 3368002680 0 0) 0x7faea4078c00 con 0x7faeb0100780 2024-02-19T21:18:33.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.249+0000 7fae9f7fe700 10 client.24012.objecter ms_dispatch 0x7faeb0001550 osd_map(829..829 src has 633..829) v4 2024-02-19T21:18:33.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.249+0000 7fae9f7fe700 3 client.24012.objecter handle_osd_map got epochs [829,829] > 0 2024-02-19T21:18:33.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.249+0000 7fae9f7fe700 3 client.24012.objecter handle_osd_map decoding full epoch 829 2024-02-19T21:18:33.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.249+0000 7fae9f7fe700 20 client.24012.objecter dump_active .. 0 homeless 2024-02-19T21:18:33.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.249+0000 7fae9f7fe700 10 client.24012.objecter ms_handle_connect 0x7fae98063c90 2024-02-19T21:18:33.254 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.253+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7faea404c420 con 0x7faeb0100780 2024-02-19T21:18:33.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.434+0000 7faeb4d64700 1 -- 172.21.15.163:0/3928677302 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} v 0) v1 -- 0x7fae900059e0 con 0x7faeb0100780 2024-02-19T21:18:33.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.454+0000 7fae9f7fe700 1 -- 172.21.15.163:0/3928677302 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]=0 set osd.2 primary-affinity to 1 (8655362) v830) v1 ==== 133+0+0 (unknown 1176086124 0 0) 0x7faea4003920 con 0x7faeb0100780 2024-02-19T21:18:33.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.458+0000 7faeb4d64700 10 librados: watch_flush enter 2024-02-19T21:18:33.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.458+0000 7faeb4d64700 10 librados: watch_flush exit 2024-02-19T21:18:33.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.458+0000 7faeb4d64700 20 client.24012.objecter shutdown clearing up homeless session... 2024-02-19T21:18:33.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.458+0000 7faeb4d64700 10 client.24012.objecter successfully canceled tick 2024-02-19T21:18:33.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.458+0000 7faeb4d64700 1 -- 172.21.15.163:0/3928677302 >> v1:172.21.15.163:6800/1552514681 conn(0x7fae98063c90 legacy=0x7fae98066150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:33.460 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.458+0000 7faeb4d64700 1 -- 172.21.15.163:0/3928677302 >> v1:172.21.15.163:6789/0 conn(0x7faeb0100780 legacy=0x7faeb00f9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:33.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.458+0000 7faeb4d64700 1 -- 172.21.15.163:0/3928677302 shutdown_connections 2024-02-19T21:18:33.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.458+0000 7faeb4d64700 1 -- 172.21.15.163:0/3928677302 >> 172.21.15.163:0/3928677302 conn(0x7faeb00f35a0 msgr2=0x7faeb0107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:33.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.458+0000 7faeb4d64700 1 -- 172.21.15.163:0/3928677302 shutdown_connections 2024-02-19T21:18:33.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.458+0000 7faeb4d64700 1 -- 172.21.15.163:0/3928677302 wait complete. 2024-02-19T21:18:33.461 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:33.458+0000 7faeb4d64700 1 librados: shutdown 2024-02-19T21:18:33.461 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-02-19T21:18:33.749 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:18:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:18:33] "GET /metrics HTTP/1.1" 200 187976 "" "Prometheus/2.33.4" 2024-02-19T21:18:33.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:31.793664+0000 osd.6 (osd.6) 301 : cluster [DBG] 2.22 scrub starts 2024-02-19T21:18:33.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:31.794669+0000 osd.6 (osd.6) 302 : cluster [DBG] 2.22 scrub ok 2024-02-19T21:18:33.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:32.374503+0000 osd.0 (osd.0) 144 : cluster [DBG] 16.18 scrub starts 2024-02-19T21:18:33.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:32.377901+0000 osd.0 (osd.0) 145 : cluster [DBG] 16.18 scrub ok 2024-02-19T21:18:33.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:32.743460+0000 osd.2 (osd.2) 346 : cluster [DBG] 16.17 deep-scrub starts 2024-02-19T21:18:33.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:32.745711+0000 osd.2 (osd.2) 347 : cluster [DBG] 16.17 deep-scrub ok 2024-02-19T21:18:33.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:33.435583+0000 mon.a (mon.0) 3810 : audit [INF] from='client.? v1:172.21.15.163:0/3928677302' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:18:33.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:31.793664+0000 osd.6 (osd.6) 301 : cluster [DBG] 2.22 scrub starts 2024-02-19T21:18:33.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:31.794669+0000 osd.6 (osd.6) 302 : cluster [DBG] 2.22 scrub ok 2024-02-19T21:18:33.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:32.374503+0000 osd.0 (osd.0) 144 : cluster [DBG] 16.18 scrub starts 2024-02-19T21:18:33.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:32.377901+0000 osd.0 (osd.0) 145 : cluster [DBG] 16.18 scrub ok 2024-02-19T21:18:33.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:32.743460+0000 osd.2 (osd.2) 346 : cluster [DBG] 16.17 deep-scrub starts 2024-02-19T21:18:33.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:32.745711+0000 osd.2 (osd.2) 347 : cluster [DBG] 16.17 deep-scrub ok 2024-02-19T21:18:33.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:33.435583+0000 mon.a (mon.0) 3810 : audit [INF] from='client.? v1:172.21.15.163:0/3928677302' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:18:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:31.793664+0000 osd.6 (osd.6) 301 : cluster [DBG] 2.22 scrub starts 2024-02-19T21:18:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:31.794669+0000 osd.6 (osd.6) 302 : cluster [DBG] 2.22 scrub ok 2024-02-19T21:18:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:32.374503+0000 osd.0 (osd.0) 144 : cluster [DBG] 16.18 scrub starts 2024-02-19T21:18:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:32.377901+0000 osd.0 (osd.0) 145 : cluster [DBG] 16.18 scrub ok 2024-02-19T21:18:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:32.743460+0000 osd.2 (osd.2) 346 : cluster [DBG] 16.17 deep-scrub starts 2024-02-19T21:18:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:32.745711+0000 osd.2 (osd.2) 347 : cluster [DBG] 16.17 deep-scrub ok 2024-02-19T21:18:33.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:33 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:33.435583+0000 mon.a (mon.0) 3810 : audit [INF] from='client.? v1:172.21.15.163:0/3928677302' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-19T21:18:34.247 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 3 1 2024-02-19T21:18:34.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:32.811051+0000 osd.6 (osd.6) 303 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:18:34.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:32.812082+0000 osd.6 (osd.6) 304 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:18:34.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:33.356496+0000 mgr.y (mgr.14550) 693 : cluster [DBG] pgmap v1267: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 713 B/s rd, 0 op/s 2024-02-19T21:18:34.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:33.399474+0000 osd.0 (osd.0) 146 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:18:34.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:33.400793+0000 osd.0 (osd.0) 147 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:18:34.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:33.409871+0000 osd.5 (osd.5) 342 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:18:34.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:33.412942+0000 osd.5 (osd.5) 343 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:18:34.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:33.455367+0000 mon.a (mon.0) 3811 : audit [INF] from='client.? v1:172.21.15.163:0/3928677302' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-19T21:18:34.499 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:33.455423+0000 mon.a (mon.0) 3812 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-02-19T21:18:34.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:34.358111+0000 osd.4 (osd.4) 311 : cluster [DBG] 6.17 scrub starts 2024-02-19T21:18:34.500 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:34.359240+0000 osd.4 (osd.4) 312 : cluster [DBG] 6.17 scrub ok 2024-02-19T21:18:34.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:32.811051+0000 osd.6 (osd.6) 303 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:18:34.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:32.812082+0000 osd.6 (osd.6) 304 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:18:34.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:33.356496+0000 mgr.y (mgr.14550) 693 : cluster [DBG] pgmap v1267: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 713 B/s rd, 0 op/s 2024-02-19T21:18:34.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:33.399474+0000 osd.0 (osd.0) 146 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:18:34.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:33.400793+0000 osd.0 (osd.0) 147 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:18:34.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:33.409871+0000 osd.5 (osd.5) 342 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:18:34.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:33.412942+0000 osd.5 (osd.5) 343 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:18:34.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:33.455367+0000 mon.a (mon.0) 3811 : audit [INF] from='client.? v1:172.21.15.163:0/3928677302' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-19T21:18:34.500 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:33.455423+0000 mon.a (mon.0) 3812 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-02-19T21:18:34.501 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:34.358111+0000 osd.4 (osd.4) 311 : cluster [DBG] 6.17 scrub starts 2024-02-19T21:18:34.501 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:34.359240+0000 osd.4 (osd.4) 312 : cluster [DBG] 6.17 scrub ok 2024-02-19T21:18:34.523 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:32.811051+0000 osd.6 (osd.6) 303 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:18:34.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:32.812082+0000 osd.6 (osd.6) 304 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:18:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:33.356496+0000 mgr.y (mgr.14550) 693 : cluster [DBG] pgmap v1267: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 713 B/s rd, 0 op/s 2024-02-19T21:18:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:33.399474+0000 osd.0 (osd.0) 146 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:18:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:33.400793+0000 osd.0 (osd.0) 147 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:18:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:33.409871+0000 osd.5 (osd.5) 342 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:18:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:33.412942+0000 osd.5 (osd.5) 343 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:18:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:33.455367+0000 mon.a (mon.0) 3811 : audit [INF] from='client.? v1:172.21.15.163:0/3928677302' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-19T21:18:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:33.455423+0000 mon.a (mon.0) 3812 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-02-19T21:18:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:34.358111+0000 osd.4 (osd.4) 311 : cluster [DBG] 6.17 scrub starts 2024-02-19T21:18:34.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:34.359240+0000 osd.4 (osd.4) 312 : cluster [DBG] 6.17 scrub ok 2024-02-19T21:18:35.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:34.441349+0000 osd.5 (osd.5) 344 : cluster [DBG] 2.3 deep-scrub starts 2024-02-19T21:18:35.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:34.444312+0000 osd.5 (osd.5) 345 : cluster [DBG] 2.3 deep-scrub ok 2024-02-19T21:18:35.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:34.711938+0000 osd.2 (osd.2) 348 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:18:35.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:34.714832+0000 osd.2 (osd.2) 349 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:18:35.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:34.749400+0000 osd.6 (osd.6) 305 : cluster [DBG] 2.24 scrub starts 2024-02-19T21:18:35.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:34.750373+0000 osd.6 (osd.6) 306 : cluster [DBG] 2.24 scrub ok 2024-02-19T21:18:35.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:35.350233+0000 osd.4 (osd.4) 313 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:18:35.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:35.351186+0000 osd.4 (osd.4) 314 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:18:35.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:34.441349+0000 osd.5 (osd.5) 344 : cluster [DBG] 2.3 deep-scrub starts 2024-02-19T21:18:35.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:34.444312+0000 osd.5 (osd.5) 345 : cluster [DBG] 2.3 deep-scrub ok 2024-02-19T21:18:35.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:34.711938+0000 osd.2 (osd.2) 348 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:18:35.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:34.714832+0000 osd.2 (osd.2) 349 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:18:35.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:34.749400+0000 osd.6 (osd.6) 305 : cluster [DBG] 2.24 scrub starts 2024-02-19T21:18:35.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:34.750373+0000 osd.6 (osd.6) 306 : cluster [DBG] 2.24 scrub ok 2024-02-19T21:18:35.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:35.350233+0000 osd.4 (osd.4) 313 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:18:35.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:35 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:35.351186+0000 osd.4 (osd.4) 314 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:18:35.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:34.441349+0000 osd.5 (osd.5) 344 : cluster [DBG] 2.3 deep-scrub starts 2024-02-19T21:18:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:34.444312+0000 osd.5 (osd.5) 345 : cluster [DBG] 2.3 deep-scrub ok 2024-02-19T21:18:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:34.711938+0000 osd.2 (osd.2) 348 : cluster [DBG] 2.10 scrub starts 2024-02-19T21:18:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:34.714832+0000 osd.2 (osd.2) 349 : cluster [DBG] 2.10 scrub ok 2024-02-19T21:18:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:34.749400+0000 osd.6 (osd.6) 305 : cluster [DBG] 2.24 scrub starts 2024-02-19T21:18:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:34.750373+0000 osd.6 (osd.6) 306 : cluster [DBG] 2.24 scrub ok 2024-02-19T21:18:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:35.350233+0000 osd.4 (osd.4) 313 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:18:35.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:35 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:35.351186+0000 osd.4 (osd.4) 314 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:18:36.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.314+0000 7f94a6af4700 1 -- 172.21.15.163:0/2189369315 >> v1:172.21.15.163:6790/0 conn(0x7f94a00fc870 legacy=0x7f94a00fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:36.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.314+0000 7f94a6af4700 1 -- 172.21.15.163:0/2189369315 shutdown_connections 2024-02-19T21:18:36.318 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.314+0000 7f94a6af4700 1 -- 172.21.15.163:0/2189369315 >> 172.21.15.163:0/2189369315 conn(0x7f94a00f2470 msgr2=0x7f94a00f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.314+0000 7f94a6af4700 1 -- 172.21.15.163:0/2189369315 shutdown_connections 2024-02-19T21:18:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.314+0000 7f94a6af4700 1 -- 172.21.15.163:0/2189369315 wait complete. 2024-02-19T21:18:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.314+0000 7f94a6af4700 1 librados: starting msgr at 2024-02-19T21:18:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.314+0000 7f94a6af4700 1 librados: starting objecter 2024-02-19T21:18:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f94a6af4700 1 Processor -- start 2024-02-19T21:18:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f94a6af4700 1 -- start start 2024-02-19T21:18:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f94a6af4700 1 librados: setting wanted keys 2024-02-19T21:18:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f94a6af4700 1 librados: calling monclient init 2024-02-19T21:18:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f94a6af4700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f94a0116280 con 0x7f94a00ff670 2024-02-19T21:18:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f94a6af4700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f94a0117460 con 0x7f94a01031b0 2024-02-19T21:18:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f94a6af4700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f94a0118660 con 0x7f94a00fc870 2024-02-19T21:18:36.319 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949ffff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f94a00ff670 0x7f94a00f9260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:55530/0 (socket says 172.21.15.163:55530) 2024-02-19T21:18:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949ffff700 1 -- 172.21.15.163:0/805858146 learned_addr learned my addr 172.21.15.163:0/805858146 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 10 client.?.objecter ms_handle_connect 0x7f94a00ff670 2024-02-19T21:18:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 10 client.?.objecter ms_handle_connect 0x7f94a00fc870 2024-02-19T21:18:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 10 client.?.objecter ms_handle_connect 0x7f94a01031b0 2024-02-19T21:18:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359704095 0 0) 0x7f94a0118660 con 0x7f94a00fc870 2024-02-19T21:18:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9490003030 con 0x7f94a00fc870 2024-02-19T21:18:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672041079 0 0) 0x7f94a0116280 con 0x7f94a00ff670 2024-02-19T21:18:36.320 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a0118660 con 0x7f94a00ff670 2024-02-19T21:18:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986762324 0 0) 0x7f94a0117460 con 0x7f94a01031b0 2024-02-19T21:18:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a0116280 con 0x7f94a01031b0 2024-02-19T21:18:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1880496671 0 0) 0x7f9490003030 con 0x7f94a00fc870 2024-02-19T21:18:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94a0117460 con 0x7f94a00fc870 2024-02-19T21:18:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9494001e80 con 0x7f94a00fc870 2024-02-19T21:18:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.315+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1811114069 0 0) 0x7f94a0118660 con 0x7f94a00ff670 2024-02-19T21:18:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.316+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9490003030 con 0x7f94a00ff670 2024-02-19T21:18:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.316+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f948c001860 con 0x7f94a00ff670 2024-02-19T21:18:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.316+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2383075087 0 0) 0x7f94a0116280 con 0x7f94a01031b0 2024-02-19T21:18:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.316+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94a0118660 con 0x7f94a01031b0 2024-02-19T21:18:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.316+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9498001d70 con 0x7f94a01031b0 2024-02-19T21:18:36.321 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.316+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 72520167 0 0) 0x7f94a0117460 con 0x7f94a00fc870 2024-02-19T21:18:36.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.316+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 >> v1:172.21.15.193:6789/0 conn(0x7f94a01031b0 legacy=0x7f94a0114940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:36.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.316+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 >> v1:172.21.15.163:6789/0 conn(0x7f94a00ff670 legacy=0x7f94a00f9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:36.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.316+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a0119860 con 0x7f94a00fc870 2024-02-19T21:18:36.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.316+0000 7f94a6af4700 1 -- 172.21.15.163:0/805858146 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a0118850 con 0x7f94a00fc870 2024-02-19T21:18:36.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.316+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9494002e00 con 0x7f94a00fc870 2024-02-19T21:18:36.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.316+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f94940036c0 con 0x7f94a00fc870 2024-02-19T21:18:36.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.316+0000 7f94a6af4700 10 client.50744.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:36.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.316+0000 7f94a6af4700 1 -- 172.21.15.163:0/805858146 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a0118cc0 con 0x7f94a00fc870 2024-02-19T21:18:36.322 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.316+0000 7f94a6af4700 1 librados: init done 2024-02-19T21:18:36.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.317+0000 7f94a6af4700 1 -- 172.21.15.163:0/805858146 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a00fc210 con 0x7f94a00fc870 2024-02-19T21:18:36.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.317+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9494001770 con 0x7f94a00fc870 2024-02-19T21:18:36.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.318+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(830..830 src has 633..830) v4 ==== 7187+0+0 (unknown 2325565211 0 0) 0x7f9494076300 con 0x7f94a00fc870 2024-02-19T21:18:36.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.318+0000 7f949dffb700 10 client.50744.objecter ms_dispatch 0x7f94a0001550 osd_map(830..830 src has 633..830) v4 2024-02-19T21:18:36.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.318+0000 7f949dffb700 3 client.50744.objecter handle_osd_map got epochs [830,830] > 0 2024-02-19T21:18:36.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.318+0000 7f949dffb700 3 client.50744.objecter handle_osd_map decoding full epoch 830 2024-02-19T21:18:36.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.318+0000 7f949dffb700 20 client.50744.objecter dump_active .. 0 homeless 2024-02-19T21:18:36.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.318+0000 7f949dffb700 10 client.50744.objecter ms_handle_connect 0x7f9490063c30 2024-02-19T21:18:36.324 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.322+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9494049ac0 con 0x7f94a00fc870 2024-02-19T21:18:36.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:36.497+0000 7f94a6af4700 1 -- 172.21.15.163:0/805858146 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f94a005f700 con 0x7f94a00fc870 2024-02-19T21:18:36.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:35.357427+0000 mgr.y (mgr.14550) 694 : cluster [DBG] pgmap v1269: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:18:36.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:35.700621+0000 osd.2 (osd.2) 350 : cluster [DBG] 16.a scrub starts 2024-02-19T21:18:36.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:35.702042+0000 osd.2 (osd.2) 351 : cluster [DBG] 16.a scrub ok 2024-02-19T21:18:36.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:36.303829+0000 osd.3 (osd.3) 373 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:18:36.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:36.305189+0000 osd.3 (osd.3) 374 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:18:36.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:35.357427+0000 mgr.y (mgr.14550) 694 : cluster [DBG] pgmap v1269: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:18:36.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:35.700621+0000 osd.2 (osd.2) 350 : cluster [DBG] 16.a scrub starts 2024-02-19T21:18:36.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:35.702042+0000 osd.2 (osd.2) 351 : cluster [DBG] 16.a scrub ok 2024-02-19T21:18:36.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:36.303829+0000 osd.3 (osd.3) 373 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:18:36.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:36.305189+0000 osd.3 (osd.3) 374 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:18:36.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:35.357427+0000 mgr.y (mgr.14550) 694 : cluster [DBG] pgmap v1269: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:18:36.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:35.700621+0000 osd.2 (osd.2) 350 : cluster [DBG] 16.a scrub starts 2024-02-19T21:18:36.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:35.702042+0000 osd.2 (osd.2) 351 : cluster [DBG] 16.a scrub ok 2024-02-19T21:18:36.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:36.303829+0000 osd.3 (osd.3) 373 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:18:36.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:36.305189+0000 osd.3 (osd.3) 374 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:18:37.476 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:37.474+0000 7f949dffb700 1 -- 172.21.15.163:0/805858146 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v831) v1 ==== 113+0+0 (unknown 1455095297 0 0) 0x7f9494047f30 con 0x7f94a00fc870 2024-02-19T21:18:37.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:37.477+0000 7f94a6af4700 10 librados: watch_flush enter 2024-02-19T21:18:37.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:37.477+0000 7f94a6af4700 10 librados: watch_flush exit 2024-02-19T21:18:37.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:37.477+0000 7f94a6af4700 20 client.50744.objecter shutdown clearing up homeless session... 2024-02-19T21:18:37.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:37.477+0000 7f94a6af4700 10 client.50744.objecter successfully canceled tick 2024-02-19T21:18:37.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:37.477+0000 7f94a6af4700 1 -- 172.21.15.163:0/805858146 >> v1:172.21.15.163:6800/1552514681 conn(0x7f9490063c30 legacy=0x7f94900660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:37.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:37.477+0000 7f94a6af4700 1 -- 172.21.15.163:0/805858146 >> v1:172.21.15.163:6790/0 conn(0x7f94a00fc870 legacy=0x7f94a00f8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:37.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:37.477+0000 7f94a6af4700 1 -- 172.21.15.163:0/805858146 shutdown_connections 2024-02-19T21:18:37.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:37.477+0000 7f94a6af4700 1 -- 172.21.15.163:0/805858146 >> 172.21.15.163:0/805858146 conn(0x7f94a00f2470 msgr2=0x7f94a0106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:37.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:37.477+0000 7f94a6af4700 1 -- 172.21.15.163:0/805858146 shutdown_connections 2024-02-19T21:18:37.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:37.477+0000 7f94a6af4700 1 -- 172.21.15.163:0/805858146 wait complete. 2024-02-19T21:18:37.479 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:37.477+0000 7f94a6af4700 1 librados: shutdown 2024-02-19T21:18:37.480 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.3 to 1 (10000) 2024-02-19T21:18:37.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:36.498792+0000 mon.c (mon.2) 446 : audit [INF] from='client.? v1:172.21.15.163:0/805858146' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:18:37.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:36.499087+0000 mon.a (mon.0) 3813 : audit [INF] from='client.50744 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:18:37.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:36.723527+0000 osd.2 (osd.2) 352 : cluster [DBG] 2.c scrub starts 2024-02-19T21:18:37.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:36.724482+0000 osd.2 (osd.2) 353 : cluster [DBG] 2.c scrub ok 2024-02-19T21:18:37.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:36.498792+0000 mon.c (mon.2) 446 : audit [INF] from='client.? v1:172.21.15.163:0/805858146' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:18:37.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:36.499087+0000 mon.a (mon.0) 3813 : audit [INF] from='client.50744 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:18:37.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:36.723527+0000 osd.2 (osd.2) 352 : cluster [DBG] 2.c scrub starts 2024-02-19T21:18:37.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:36.724482+0000 osd.2 (osd.2) 353 : cluster [DBG] 2.c scrub ok 2024-02-19T21:18:37.774 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:36.498792+0000 mon.c (mon.2) 446 : audit [INF] from='client.? v1:172.21.15.163:0/805858146' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:18:37.775 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:36.499087+0000 mon.a (mon.0) 3813 : audit [INF] from='client.50744 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:18:37.775 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:36.723527+0000 osd.2 (osd.2) 352 : cluster [DBG] 2.c scrub starts 2024-02-19T21:18:37.775 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:36.724482+0000 osd.2 (osd.2) 353 : cluster [DBG] 2.c scrub ok 2024-02-19T21:18:38.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:18:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:18:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:18:38.305 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd primary-affinity 3 1 2024-02-19T21:18:38.580 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:38.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:37.342619+0000 osd.0 (osd.0) 148 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:18:38.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:37.343706+0000 osd.0 (osd.0) 149 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:18:38.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:37.357878+0000 mgr.y (mgr.14550) 695 : cluster [DBG] pgmap v1270: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:18:38.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:37.475130+0000 mon.a (mon.0) 3814 : audit [INF] from='client.50744 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-19T21:18:38.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:37.475212+0000 mon.a (mon.0) 3815 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-02-19T21:18:38.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:37.673806+0000 osd.6 (osd.6) 307 : cluster [DBG] 16.6 deep-scrub starts 2024-02-19T21:18:38.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:37.674815+0000 osd.6 (osd.6) 308 : cluster [DBG] 16.6 deep-scrub ok 2024-02-19T21:18:38.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:37.342619+0000 osd.0 (osd.0) 148 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:18:38.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:37.343706+0000 osd.0 (osd.0) 149 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:18:38.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:37.357878+0000 mgr.y (mgr.14550) 695 : cluster [DBG] pgmap v1270: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:18:38.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:37.475130+0000 mon.a (mon.0) 3814 : audit [INF] from='client.50744 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-19T21:18:38.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:37.475212+0000 mon.a (mon.0) 3815 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-02-19T21:18:38.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:37.673806+0000 osd.6 (osd.6) 307 : cluster [DBG] 16.6 deep-scrub starts 2024-02-19T21:18:38.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:37.674815+0000 osd.6 (osd.6) 308 : cluster [DBG] 16.6 deep-scrub ok 2024-02-19T21:18:38.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:37.342619+0000 osd.0 (osd.0) 148 : cluster [DBG] 2.5 scrub starts 2024-02-19T21:18:38.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:37.343706+0000 osd.0 (osd.0) 149 : cluster [DBG] 2.5 scrub ok 2024-02-19T21:18:38.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:37.357878+0000 mgr.y (mgr.14550) 695 : cluster [DBG] pgmap v1270: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:18:38.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:37.475130+0000 mon.a (mon.0) 3814 : audit [INF] from='client.50744 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-19T21:18:38.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:37.475212+0000 mon.a (mon.0) 3815 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-02-19T21:18:38.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:37.673806+0000 osd.6 (osd.6) 307 : cluster [DBG] 16.6 deep-scrub starts 2024-02-19T21:18:38.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:37.674815+0000 osd.6 (osd.6) 308 : cluster [DBG] 16.6 deep-scrub ok 2024-02-19T21:18:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:38.301785+0000 osd.0 (osd.0) 150 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:18:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:38.304036+0000 osd.0 (osd.0) 151 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:18:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:38.714202+0000 osd.4 (osd.4) 315 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:38.714648+0000 osd.4 (osd.4) 316 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:39.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:39.443898+0000 mon.a (mon.0) 3816 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-02-19T21:18:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:38.301785+0000 osd.0 (osd.0) 150 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:18:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:38.304036+0000 osd.0 (osd.0) 151 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:18:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:38.714202+0000 osd.4 (osd.4) 315 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:38.714648+0000 osd.4 (osd.4) 316 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:39.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:39.443898+0000 mon.a (mon.0) 3816 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-02-19T21:18:39.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:38.301785+0000 osd.0 (osd.0) 150 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:18:39.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:38.304036+0000 osd.0 (osd.0) 151 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:18:39.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:38.714202+0000 osd.4 (osd.4) 315 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:39.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:38.714648+0000 osd.4 (osd.4) 316 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:39.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:39.443898+0000 mon.a (mon.0) 3816 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-02-19T21:18:40.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.378+0000 7f5a8a2ce700 1 -- 172.21.15.163:0/3794692215 >> v1:172.21.15.163:6790/0 conn(0x7f5a840fd980 legacy=0x7f5a840fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:40.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.378+0000 7f5a8a2ce700 1 -- 172.21.15.163:0/3794692215 shutdown_connections 2024-02-19T21:18:40.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.378+0000 7f5a8a2ce700 1 -- 172.21.15.163:0/3794692215 >> 172.21.15.163:0/3794692215 conn(0x7f5a840f35a0 msgr2=0x7f5a840f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:40.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.378+0000 7f5a8a2ce700 1 -- 172.21.15.163:0/3794692215 shutdown_connections 2024-02-19T21:18:40.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.378+0000 7f5a8a2ce700 1 -- 172.21.15.163:0/3794692215 wait complete. 2024-02-19T21:18:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.378+0000 7f5a8a2ce700 1 librados: starting msgr at 2024-02-19T21:18:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.378+0000 7f5a8a2ce700 1 librados: starting objecter 2024-02-19T21:18:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.378+0000 7f5a8a2ce700 1 Processor -- start 2024-02-19T21:18:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.378+0000 7f5a8a2ce700 1 -- start start 2024-02-19T21:18:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.378+0000 7f5a8a2ce700 1 librados: setting wanted keys 2024-02-19T21:18:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.378+0000 7f5a8a2ce700 1 librados: calling monclient init 2024-02-19T21:18:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a8a2ce700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5a840fcf70 con 0x7f5a840fd980 2024-02-19T21:18:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a8a2ce700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5a840fd100 con 0x7f5a841042c0 2024-02-19T21:18:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a8a2ce700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5a840fd2d0 con 0x7f5a84100780 2024-02-19T21:18:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a83fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f5a840fd980 0x7f5a840fc7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:55548/0 (socket says 172.21.15.163:55548) 2024-02-19T21:18:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a83fff700 1 -- 172.21.15.163:0/180273402 learned_addr learned my addr 172.21.15.163:0/180273402 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a837fe700 1 --1- 172.21.15.163:0/180273402 >> v1:172.21.15.163:6790/0 conn(0x7f5a84100780 0x7f5a84114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:47376/0 (socket says 172.21.15.163:47376) 2024-02-19T21:18:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 10 client.?.objecter ms_handle_connect 0x7f5a840fd980 2024-02-19T21:18:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:40.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 10 client.?.objecter ms_handle_connect 0x7f5a84100780 2024-02-19T21:18:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 10 client.?.objecter ms_handle_connect 0x7f5a841042c0 2024-02-19T21:18:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209131701 0 0) 0x7f5a840fd2d0 con 0x7f5a84100780 2024-02-19T21:18:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6c003030 con 0x7f5a84100780 2024-02-19T21:18:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307973556 0 0) 0x7f5a840fcf70 con 0x7f5a840fd980 2024-02-19T21:18:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a840fd2d0 con 0x7f5a840fd980 2024-02-19T21:18:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261102456 0 0) 0x7f5a840fd100 con 0x7f5a841042c0 2024-02-19T21:18:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a840fcf70 con 0x7f5a841042c0 2024-02-19T21:18:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2046211825 0 0) 0x7f5a6c003030 con 0x7f5a84100780 2024-02-19T21:18:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a840fd100 con 0x7f5a84100780 2024-02-19T21:18:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5a78001840 con 0x7f5a84100780 2024-02-19T21:18:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2332598664 0 0) 0x7f5a840fd2d0 con 0x7f5a840fd980 2024-02-19T21:18:40.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a6c003030 con 0x7f5a840fd980 2024-02-19T21:18:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5a740015d0 con 0x7f5a840fd980 2024-02-19T21:18:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3530728453 0 0) 0x7f5a840fcf70 con 0x7f5a841042c0 2024-02-19T21:18:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a840fd2d0 con 0x7f5a841042c0 2024-02-19T21:18:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5a7c001950 con 0x7f5a841042c0 2024-02-19T21:18:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1353594261 0 0) 0x7f5a840fd100 con 0x7f5a84100780 2024-02-19T21:18:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 >> v1:172.21.15.193:6789/0 conn(0x7f5a841042c0 legacy=0x7f5a84118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 >> v1:172.21.15.163:6789/0 conn(0x7f5a840fd980 legacy=0x7f5a840fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8411cb90 con 0x7f5a84100780 2024-02-19T21:18:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.379+0000 7f5a8a2ce700 1 -- 172.21.15.163:0/180273402 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a84119bc0 con 0x7f5a84100780 2024-02-19T21:18:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.380+0000 7f5a8a2ce700 10 client.50750.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.380+0000 7f5a8a2ce700 1 -- 172.21.15.163:0/180273402 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8411a0b0 con 0x7f5a84100780 2024-02-19T21:18:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.380+0000 7f5a8a2ce700 1 librados: init done 2024-02-19T21:18:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.380+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f5a78002290 con 0x7f5a84100780 2024-02-19T21:18:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.380+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5a78005990 con 0x7f5a84100780 2024-02-19T21:18:40.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.381+0000 7f5a8a2ce700 1 -- 172.21.15.163:0/180273402 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a70004f20 con 0x7f5a84100780 2024-02-19T21:18:40.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.381+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f5a78001df0 con 0x7f5a84100780 2024-02-19T21:18:40.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.381+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(832..832 src has 633..832) v4 ==== 7187+0+0 (unknown 2133456079 0 0) 0x7f5a780787d0 con 0x7f5a84100780 2024-02-19T21:18:40.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.381+0000 7f5a81775700 10 client.50750.objecter ms_dispatch 0x7f5a84001550 osd_map(832..832 src has 633..832) v4 2024-02-19T21:18:40.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.381+0000 7f5a81775700 3 client.50750.objecter handle_osd_map got epochs [832,832] > 0 2024-02-19T21:18:40.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.381+0000 7f5a81775700 3 client.50750.objecter handle_osd_map decoding full epoch 832 2024-02-19T21:18:40.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.382+0000 7f5a81775700 20 client.50750.objecter dump_active .. 0 homeless 2024-02-19T21:18:40.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.382+0000 7f5a81775700 10 client.50750.objecter ms_handle_connect 0x7f5a6c063dd0 2024-02-19T21:18:40.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.386+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5a7804bf90 con 0x7f5a84100780 2024-02-19T21:18:40.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:40.574+0000 7f5a8a2ce700 1 -- 172.21.15.163:0/180273402 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} v 0) v1 -- 0x7f5a700059e0 con 0x7f5a84100780 2024-02-19T21:18:40.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:39.358295+0000 mgr.y (mgr.14550) 696 : cluster [DBG] pgmap v1272: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:18:40.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:39.507754+0000 osd.1 (osd.1) 389 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:18:40.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:39.510639+0000 osd.1 (osd.1) 390 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:18:40.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:39.669662+0000 osd.2 (osd.2) 354 : cluster [DBG] 5.b deep-scrub starts 2024-02-19T21:18:40.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:39.670786+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.b deep-scrub ok 2024-02-19T21:18:40.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:39.708061+0000 osd.6 (osd.6) 309 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:18:40.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:39.709219+0000 osd.6 (osd.6) 310 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:18:40.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.399754+0000 mon.b (mon.1) 762 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-02-19T21:18:40.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.399976+0000 mon.b (mon.1) 763 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 1]}]: dispatch 2024-02-19T21:18:40.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.400168+0000 mon.a (mon.0) 3817 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-02-19T21:18:40.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.400178+0000 mon.b (mon.1) 764 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-19T21:18:40.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.400361+0000 mon.b (mon.1) 765 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2024-02-19T21:18:40.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.400543+0000 mon.b (mon.1) 766 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-02-19T21:18:40.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.400725+0000 mon.b (mon.1) 767 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-19T21:18:40.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.400783+0000 mon.a (mon.0) 3818 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 1]}]: dispatch 2024-02-19T21:18:40.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.400909+0000 mon.b (mon.1) 768 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-19T21:18:40.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.401101+0000 mon.b (mon.1) 769 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-19T21:18:40.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.401238+0000 mon.a (mon.0) 3819 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-19T21:18:40.824 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.401564+0000 mon.a (mon.0) 3820 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2024-02-19T21:18:40.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.401750+0000 mon.a (mon.0) 3821 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-02-19T21:18:40.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.401925+0000 mon.a (mon.0) 3822 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-19T21:18:40.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.402096+0000 mon.a (mon.0) 3823 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-19T21:18:40.825 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.402274+0000 mon.a (mon.0) 3824 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-19T21:18:40.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:39.358295+0000 mgr.y (mgr.14550) 696 : cluster [DBG] pgmap v1272: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:18:40.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:39.507754+0000 osd.1 (osd.1) 389 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:18:40.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:39.510639+0000 osd.1 (osd.1) 390 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:18:41.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:39.669662+0000 osd.2 (osd.2) 354 : cluster [DBG] 5.b deep-scrub starts 2024-02-19T21:18:41.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:39.670786+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.b deep-scrub ok 2024-02-19T21:18:41.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:39.708061+0000 osd.6 (osd.6) 309 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:18:41.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:39.709219+0000 osd.6 (osd.6) 310 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:18:41.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.399754+0000 mon.b (mon.1) 762 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.399976+0000 mon.b (mon.1) 763 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 1]}]: dispatch 2024-02-19T21:18:41.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.400168+0000 mon.a (mon.0) 3817 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.400178+0000 mon.b (mon.1) 764 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.400361+0000 mon.b (mon.1) 765 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.400543+0000 mon.b (mon.1) 766 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.400725+0000 mon.b (mon.1) 767 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-19T21:18:41.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.400783+0000 mon.a (mon.0) 3818 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 1]}]: dispatch 2024-02-19T21:18:41.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.400909+0000 mon.b (mon.1) 768 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-19T21:18:41.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:39.358295+0000 mgr.y (mgr.14550) 696 : cluster [DBG] pgmap v1272: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:18:41.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:39.507754+0000 osd.1 (osd.1) 389 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:18:41.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:39.510639+0000 osd.1 (osd.1) 390 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:18:41.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:39.669662+0000 osd.2 (osd.2) 354 : cluster [DBG] 5.b deep-scrub starts 2024-02-19T21:18:41.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:39.670786+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.b deep-scrub ok 2024-02-19T21:18:41.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:39.708061+0000 osd.6 (osd.6) 309 : cluster [DBG] 6.3 scrub starts 2024-02-19T21:18:41.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:39.709219+0000 osd.6 (osd.6) 310 : cluster [DBG] 6.3 scrub ok 2024-02-19T21:18:41.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.399754+0000 mon.b (mon.1) 762 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.399976+0000 mon.b (mon.1) 763 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 1]}]: dispatch 2024-02-19T21:18:41.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.400168+0000 mon.a (mon.0) 3817 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.400178+0000 mon.b (mon.1) 764 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.002 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.400361+0000 mon.b (mon.1) 765 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.002 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.400543+0000 mon.b (mon.1) 766 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.002 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.400725+0000 mon.b (mon.1) 767 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-19T21:18:41.002 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.400783+0000 mon.a (mon.0) 3818 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 1]}]: dispatch 2024-02-19T21:18:41.002 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.400909+0000 mon.b (mon.1) 768 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-19T21:18:41.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.401101+0000 mon.b (mon.1) 769 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-19T21:18:41.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.401238+0000 mon.a (mon.0) 3819 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.401564+0000 mon.a (mon.0) 3820 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.401750+0000 mon.a (mon.0) 3821 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.401925+0000 mon.a (mon.0) 3822 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-19T21:18:41.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.402096+0000 mon.a (mon.0) 3823 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-19T21:18:41.003 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.402274+0000 mon.a (mon.0) 3824 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-19T21:18:41.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.401101+0000 mon.b (mon.1) 769 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-19T21:18:41.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.401238+0000 mon.a (mon.0) 3819 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.401564+0000 mon.a (mon.0) 3820 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.401750+0000 mon.a (mon.0) 3821 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]: dispatch 2024-02-19T21:18:41.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.401925+0000 mon.a (mon.0) 3822 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-19T21:18:41.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.402096+0000 mon.a (mon.0) 3823 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-19T21:18:41.003 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.402274+0000 mon.a (mon.0) 3824 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-19T21:18:41.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:18:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:18:41.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:41.536+0000 7f5a81775700 1 -- 172.21.15.163:0/180273402 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]=0 set osd.3 primary-affinity to 1 (8655362) v834) v1 ==== 133+0+0 (unknown 1096195991 0 0) 0x7f5a7804a400 con 0x7f5a84100780 2024-02-19T21:18:41.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:41.540+0000 7f5a8a2ce700 10 librados: watch_flush enter 2024-02-19T21:18:41.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:41.543+0000 7f5a8a2ce700 10 librados: watch_flush exit 2024-02-19T21:18:41.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:41.543+0000 7f5a8a2ce700 20 client.50750.objecter shutdown clearing up homeless session... 2024-02-19T21:18:41.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:41.543+0000 7f5a8a2ce700 10 client.50750.objecter successfully canceled tick 2024-02-19T21:18:41.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:41.543+0000 7f5a8a2ce700 1 -- 172.21.15.163:0/180273402 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5a6c063dd0 legacy=0x7f5a6c066290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:41.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:41.543+0000 7f5a8a2ce700 1 -- 172.21.15.163:0/180273402 >> v1:172.21.15.163:6790/0 conn(0x7f5a84100780 legacy=0x7f5a84114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:41.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:41.545+0000 7f5a8a2ce700 1 -- 172.21.15.163:0/180273402 shutdown_connections 2024-02-19T21:18:41.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:41.545+0000 7f5a8a2ce700 1 -- 172.21.15.163:0/180273402 >> 172.21.15.163:0/180273402 conn(0x7f5a840f35a0 msgr2=0x7f5a84107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:41.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:41.547+0000 7f5a8a2ce700 1 -- 172.21.15.163:0/180273402 shutdown_connections 2024-02-19T21:18:41.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:41.547+0000 7f5a8a2ce700 1 -- 172.21.15.163:0/180273402 wait complete. 2024-02-19T21:18:41.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:41.547+0000 7f5a8a2ce700 1 librados: shutdown 2024-02-19T21:18:41.550 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-02-19T21:18:41.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.534270+0000 mon.a (mon.0) 3825 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-02-19T21:18:41.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.534356+0000 mon.a (mon.0) 3826 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 1]}]': finished 2024-02-19T21:18:41.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.534419+0000 mon.a (mon.0) 3827 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-02-19T21:18:41.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.534481+0000 mon.a (mon.0) 3828 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]': finished 2024-02-19T21:18:41.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.534541+0000 mon.a (mon.0) 3829 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-02-19T21:18:41.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.534601+0000 mon.a (mon.0) 3830 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-19T21:18:41.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.534663+0000 mon.a (mon.0) 3831 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-02-19T21:18:41.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.534722+0000 mon.a (mon.0) 3832 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-02-19T21:18:41.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:40.534776+0000 mon.a (mon.0) 3833 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-02-19T21:18:41.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.575567+0000 mon.c (mon.2) 447 : audit [INF] from='client.? v1:172.21.15.163:0/180273402' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:18:41.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.575841+0000 mon.a (mon.0) 3834 : audit [INF] from='client.50750 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:18:41.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:40.884408+0000 mgr.y (mgr.14550) 697 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:41.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:41.537055+0000 mon.a (mon.0) 3835 : audit [INF] from='client.50750 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-19T21:18:41.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:41.537134+0000 mon.a (mon.0) 3836 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-02-19T21:18:41.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.534270+0000 mon.a (mon.0) 3825 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-02-19T21:18:41.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.534356+0000 mon.a (mon.0) 3826 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 1]}]': finished 2024-02-19T21:18:41.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.534419+0000 mon.a (mon.0) 3827 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-02-19T21:18:41.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.534481+0000 mon.a (mon.0) 3828 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]': finished 2024-02-19T21:18:41.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.534541+0000 mon.a (mon.0) 3829 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-02-19T21:18:41.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.534601+0000 mon.a (mon.0) 3830 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-19T21:18:41.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.534663+0000 mon.a (mon.0) 3831 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-02-19T21:18:41.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.534722+0000 mon.a (mon.0) 3832 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-02-19T21:18:41.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:40.534776+0000 mon.a (mon.0) 3833 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-02-19T21:18:41.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.575567+0000 mon.c (mon.2) 447 : audit [INF] from='client.? v1:172.21.15.163:0/180273402' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:18:41.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.575841+0000 mon.a (mon.0) 3834 : audit [INF] from='client.50750 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:18:41.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:40.884408+0000 mgr.y (mgr.14550) 697 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:41.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:41.537055+0000 mon.a (mon.0) 3835 : audit [INF] from='client.50750 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-19T21:18:41.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:41.537134+0000 mon.a (mon.0) 3836 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-02-19T21:18:41.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.534270+0000 mon.a (mon.0) 3825 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-02-19T21:18:41.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.534356+0000 mon.a (mon.0) 3826 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 1]}]': finished 2024-02-19T21:18:41.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.534419+0000 mon.a (mon.0) 3827 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-02-19T21:18:41.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.534481+0000 mon.a (mon.0) 3828 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 2]}]': finished 2024-02-19T21:18:41.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.534541+0000 mon.a (mon.0) 3829 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [7, 2]}]': finished 2024-02-19T21:18:41.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.534601+0000 mon.a (mon.0) 3830 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-19T21:18:41.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.534663+0000 mon.a (mon.0) 3831 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-02-19T21:18:41.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.534722+0000 mon.a (mon.0) 3832 : audit [INF] from='mgr.14550 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-02-19T21:18:41.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:40.534776+0000 mon.a (mon.0) 3833 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-02-19T21:18:41.835 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.575567+0000 mon.c (mon.2) 447 : audit [INF] from='client.? v1:172.21.15.163:0/180273402' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:18:41.836 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.575841+0000 mon.a (mon.0) 3834 : audit [INF] from='client.50750 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-19T21:18:41.836 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:40.884408+0000 mgr.y (mgr.14550) 697 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:41.836 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:41.537055+0000 mon.a (mon.0) 3835 : audit [INF] from='client.50750 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-19T21:18:41.836 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:41.537134+0000 mon.a (mon.0) 3836 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-02-19T21:18:42.221 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 4 1 2024-02-19T21:18:42.504 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:42.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:41.358826+0000 mgr.y (mgr.14550) 698 : cluster [DBG] pgmap v1275: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:18:42.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:41.547683+0000 osd.1 (osd.1) 391 : cluster [DBG] 2.a scrub starts 2024-02-19T21:18:42.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:41.550789+0000 osd.1 (osd.1) 392 : cluster [DBG] 2.a scrub ok 2024-02-19T21:18:42.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:42.340383+0000 osd.3 (osd.3) 375 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:18:42.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:42.341274+0000 osd.3 (osd.3) 376 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:18:42.804 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:41.358826+0000 mgr.y (mgr.14550) 698 : cluster [DBG] pgmap v1275: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:18:42.804 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:41.547683+0000 osd.1 (osd.1) 391 : cluster [DBG] 2.a scrub starts 2024-02-19T21:18:42.804 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:41.550789+0000 osd.1 (osd.1) 392 : cluster [DBG] 2.a scrub ok 2024-02-19T21:18:42.804 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:42.340383+0000 osd.3 (osd.3) 375 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:18:42.804 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:42.341274+0000 osd.3 (osd.3) 376 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:18:42.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:41.358826+0000 mgr.y (mgr.14550) 698 : cluster [DBG] pgmap v1275: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:18:42.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:41.547683+0000 osd.1 (osd.1) 391 : cluster [DBG] 2.a scrub starts 2024-02-19T21:18:42.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:41.550789+0000 osd.1 (osd.1) 392 : cluster [DBG] 2.a scrub ok 2024-02-19T21:18:42.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:42.340383+0000 osd.3 (osd.3) 375 : cluster [DBG] 3.2 scrub starts 2024-02-19T21:18:42.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:42.341274+0000 osd.3 (osd.3) 376 : cluster [DBG] 3.2 scrub ok 2024-02-19T21:18:43.746 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:18:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:18:43] "GET /metrics HTTP/1.1" 200 187979 "" "Prometheus/2.33.4" 2024-02-19T21:18:43.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:42.495857+0000 osd.1 (osd.1) 393 : cluster [DBG] 4.4 deep-scrub starts 2024-02-19T21:18:43.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:42.503138+0000 osd.1 (osd.1) 394 : cluster [DBG] 4.4 deep-scrub ok 2024-02-19T21:18:43.747 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:42.752869+0000 osd.2 (osd.2) 356 : cluster [DBG] 2.1d deep-scrub starts 2024-02-19T21:18:43.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:42.753929+0000 osd.2 (osd.2) 357 : cluster [DBG] 2.1d deep-scrub ok 2024-02-19T21:18:43.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:43.348550+0000 osd.3 (osd.3) 377 : cluster [DBG] 6.c deep-scrub starts 2024-02-19T21:18:43.748 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:43.349525+0000 osd.3 (osd.3) 378 : cluster [DBG] 6.c deep-scrub ok 2024-02-19T21:18:43.748 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:42.495857+0000 osd.1 (osd.1) 393 : cluster [DBG] 4.4 deep-scrub starts 2024-02-19T21:18:43.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:42.503138+0000 osd.1 (osd.1) 394 : cluster [DBG] 4.4 deep-scrub ok 2024-02-19T21:18:43.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:42.752869+0000 osd.2 (osd.2) 356 : cluster [DBG] 2.1d deep-scrub starts 2024-02-19T21:18:43.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:42.753929+0000 osd.2 (osd.2) 357 : cluster [DBG] 2.1d deep-scrub ok 2024-02-19T21:18:43.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:43.348550+0000 osd.3 (osd.3) 377 : cluster [DBG] 6.c deep-scrub starts 2024-02-19T21:18:43.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:43.349525+0000 osd.3 (osd.3) 378 : cluster [DBG] 6.c deep-scrub ok 2024-02-19T21:18:43.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:42.495857+0000 osd.1 (osd.1) 393 : cluster [DBG] 4.4 deep-scrub starts 2024-02-19T21:18:43.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:42.503138+0000 osd.1 (osd.1) 394 : cluster [DBG] 4.4 deep-scrub ok 2024-02-19T21:18:43.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:42.752869+0000 osd.2 (osd.2) 356 : cluster [DBG] 2.1d deep-scrub starts 2024-02-19T21:18:43.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:42.753929+0000 osd.2 (osd.2) 357 : cluster [DBG] 2.1d deep-scrub ok 2024-02-19T21:18:43.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:43.348550+0000 osd.3 (osd.3) 377 : cluster [DBG] 6.c deep-scrub starts 2024-02-19T21:18:43.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:43.349525+0000 osd.3 (osd.3) 378 : cluster [DBG] 6.c deep-scrub ok 2024-02-19T21:18:44.262 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.259+0000 7f7426fd1700 1 -- 172.21.15.163:0/2954773004 >> v1:172.21.15.163:6790/0 conn(0x7f7420100780 legacy=0x7f7420102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:44.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.259+0000 7f7426fd1700 1 -- 172.21.15.163:0/2954773004 shutdown_connections 2024-02-19T21:18:44.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.259+0000 7f7426fd1700 1 -- 172.21.15.163:0/2954773004 >> 172.21.15.163:0/2954773004 conn(0x7f74200f35a0 msgr2=0x7f74200f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:44.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.259+0000 7f7426fd1700 1 -- 172.21.15.163:0/2954773004 shutdown_connections 2024-02-19T21:18:44.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.259+0000 7f7426fd1700 1 -- 172.21.15.163:0/2954773004 wait complete. 2024-02-19T21:18:44.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f7426fd1700 1 librados: starting msgr at 2024-02-19T21:18:44.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f7426fd1700 1 librados: starting objecter 2024-02-19T21:18:44.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f7426fd1700 1 Processor -- start 2024-02-19T21:18:44.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f7426fd1700 1 -- start start 2024-02-19T21:18:44.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f7426fd1700 1 librados: setting wanted keys 2024-02-19T21:18:44.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f7426fd1700 1 librados: calling monclient init 2024-02-19T21:18:44.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f7426fd1700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f74201172c0 con 0x7f74201042c0 2024-02-19T21:18:44.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f7426fd1700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f74201184a0 con 0x7f7420100780 2024-02-19T21:18:44.263 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f7426fd1700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f74201196a0 con 0x7f74200fd980 2024-02-19T21:18:44.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f7424d6d700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f74200fd980 0x7f74200f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:47386/0 (socket says 172.21.15.163:47386) 2024-02-19T21:18:44.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f7424d6d700 1 -- 172.21.15.163:0/1311942969 learned_addr learned my addr 172.21.15.163:0/1311942969 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:44.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f741dffb700 10 client.?.objecter ms_handle_connect 0x7f74200fd980 2024-02-19T21:18:44.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f741dffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:44.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f741dffb700 10 client.?.objecter ms_handle_connect 0x7f74201042c0 2024-02-19T21:18:44.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f741dffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:44.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f741dffb700 10 client.?.objecter ms_handle_connect 0x7f7420100780 2024-02-19T21:18:44.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f741dffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:44.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546013773 0 0) 0x7f74201172c0 con 0x7f74201042c0 2024-02-19T21:18:44.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7410003030 con 0x7f74201042c0 2024-02-19T21:18:44.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093662061 0 0) 0x7f74201196a0 con 0x7f74200fd980 2024-02-19T21:18:44.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.260+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74201172c0 con 0x7f74200fd980 2024-02-19T21:18:44.264 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461027403 0 0) 0x7f74201184a0 con 0x7f7420100780 2024-02-19T21:18:44.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74201196a0 con 0x7f7420100780 2024-02-19T21:18:44.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 561330023 0 0) 0x7f74201172c0 con 0x7f74200fd980 2024-02-19T21:18:44.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74201184a0 con 0x7f74200fd980 2024-02-19T21:18:44.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f74140019f0 con 0x7f74200fd980 2024-02-19T21:18:44.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3719644143 0 0) 0x7f7410003030 con 0x7f74201042c0 2024-02-19T21:18:44.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74201172c0 con 0x7f74201042c0 2024-02-19T21:18:44.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7418001b00 con 0x7f74201042c0 2024-02-19T21:18:44.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2741351554 0 0) 0x7f74201196a0 con 0x7f7420100780 2024-02-19T21:18:44.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7410003030 con 0x7f7420100780 2024-02-19T21:18:44.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f740c0020f0 con 0x7f7420100780 2024-02-19T21:18:44.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4208364352 0 0) 0x7f74201184a0 con 0x7f74200fd980 2024-02-19T21:18:44.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 >> v1:172.21.15.193:6789/0 conn(0x7f7420100780 legacy=0x7f74200fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:44.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 >> v1:172.21.15.163:6789/0 conn(0x7f74201042c0 legacy=0x7f7420115980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:44.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742011a8a0 con 0x7f74200fd980 2024-02-19T21:18:44.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f7426fd1700 1 -- 172.21.15.163:0/1311942969 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74201174b0 con 0x7f74200fd980 2024-02-19T21:18:44.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f7414002520 con 0x7f74200fd980 2024-02-19T21:18:44.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7414005c20 con 0x7f74200fd980 2024-02-19T21:18:44.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f7426fd1700 10 client.50756.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:44.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f7426fd1700 1 -- 172.21.15.163:0/1311942969 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74201179a0 con 0x7f74200fd980 2024-02-19T21:18:44.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.261+0000 7f7426fd1700 1 librados: init done 2024-02-19T21:18:44.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.262+0000 7f7426fd1700 1 -- 172.21.15.163:0/1311942969 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742005f6a0 con 0x7f74200fd980 2024-02-19T21:18:44.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.263+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f7414002080 con 0x7f74200fd980 2024-02-19T21:18:44.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.264+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(835..835 src has 633..835) v4 ==== 7419+0+0 (unknown 1035221250 0 0) 0x7f7414078bf0 con 0x7f74200fd980 2024-02-19T21:18:44.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.264+0000 7f741dffb700 10 client.50756.objecter ms_dispatch 0x7f7420001550 osd_map(835..835 src has 633..835) v4 2024-02-19T21:18:44.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.264+0000 7f741dffb700 3 client.50756.objecter handle_osd_map got epochs [835,835] > 0 2024-02-19T21:18:44.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.264+0000 7f741dffb700 3 client.50756.objecter handle_osd_map decoding full epoch 835 2024-02-19T21:18:44.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.264+0000 7f741dffb700 20 client.50756.objecter dump_active .. 0 homeless 2024-02-19T21:18:44.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.264+0000 7f741dffb700 10 client.50756.objecter ms_handle_connect 0x7f7410063c90 2024-02-19T21:18:44.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.267+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f741404c260 con 0x7f74200fd980 2024-02-19T21:18:44.443 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:44.442+0000 7f7426fd1700 1 -- 172.21.15.163:0/1311942969 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) v1 -- 0x7f7420047540 con 0x7f74200fd980 2024-02-19T21:18:44.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:43.016949+0000 osd.0 (osd.0) 152 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:44.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:43.017099+0000 osd.0 (osd.0) 153 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:44.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:43.359229+0000 mgr.y (mgr.14550) 699 : cluster [DBG] pgmap v1277: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-02-19T21:18:44.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:43.559396+0000 mon.a (mon.0) 3837 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-02-19T21:18:44.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:44.444887+0000 mon.a (mon.0) 3838 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-02-19T21:18:44.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:44.445048+0000 mon.c (mon.2) 448 : audit [INF] from='client.? v1:172.21.15.163:0/1311942969' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:18:44.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:44.446566+0000 mon.a (mon.0) 3839 : audit [INF] from='client.50756 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:18:44.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:43.016949+0000 osd.0 (osd.0) 152 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:44.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:43.017099+0000 osd.0 (osd.0) 153 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:44.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:43.359229+0000 mgr.y (mgr.14550) 699 : cluster [DBG] pgmap v1277: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-02-19T21:18:44.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:43.559396+0000 mon.a (mon.0) 3837 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-02-19T21:18:44.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:44.444887+0000 mon.a (mon.0) 3838 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-02-19T21:18:44.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:44.445048+0000 mon.c (mon.2) 448 : audit [INF] from='client.? v1:172.21.15.163:0/1311942969' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:18:44.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:44.446566+0000 mon.a (mon.0) 3839 : audit [INF] from='client.50756 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:18:45.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:43.016949+0000 osd.0 (osd.0) 152 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:45.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:43.017099+0000 osd.0 (osd.0) 153 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:45.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:43.359229+0000 mgr.y (mgr.14550) 699 : cluster [DBG] pgmap v1277: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-02-19T21:18:45.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:43.559396+0000 mon.a (mon.0) 3837 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-02-19T21:18:45.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:44.444887+0000 mon.a (mon.0) 3838 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-02-19T21:18:45.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:44.445048+0000 mon.c (mon.2) 448 : audit [INF] from='client.? v1:172.21.15.163:0/1311942969' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:18:45.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:44.446566+0000 mon.a (mon.0) 3839 : audit [INF] from='client.50756 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:18:45.453 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:45.452+0000 7f741dffb700 1 -- 172.21.15.163:0/1311942969 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v837) v1 ==== 113+0+0 (unknown 2392958725 0 0) 0x7f741404a670 con 0x7f74200fd980 2024-02-19T21:18:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:45.454+0000 7f7426fd1700 10 librados: watch_flush enter 2024-02-19T21:18:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:45.454+0000 7f7426fd1700 10 librados: watch_flush exit 2024-02-19T21:18:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:45.454+0000 7f7426fd1700 20 client.50756.objecter shutdown clearing up homeless session... 2024-02-19T21:18:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:45.454+0000 7f7426fd1700 10 client.50756.objecter successfully canceled tick 2024-02-19T21:18:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:45.455+0000 7f7426fd1700 1 -- 172.21.15.163:0/1311942969 >> v1:172.21.15.163:6800/1552514681 conn(0x7f7410063c90 legacy=0x7f7410066150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:45.455+0000 7f7426fd1700 1 -- 172.21.15.163:0/1311942969 >> v1:172.21.15.163:6790/0 conn(0x7f74200fd980 legacy=0x7f74200f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:45.455+0000 7f7426fd1700 1 -- 172.21.15.163:0/1311942969 shutdown_connections 2024-02-19T21:18:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:45.455+0000 7f7426fd1700 1 -- 172.21.15.163:0/1311942969 >> 172.21.15.163:0/1311942969 conn(0x7f74200f35a0 msgr2=0x7f7420107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:45.455+0000 7f7426fd1700 1 -- 172.21.15.163:0/1311942969 shutdown_connections 2024-02-19T21:18:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:45.455+0000 7f7426fd1700 1 -- 172.21.15.163:0/1311942969 wait complete. 2024-02-19T21:18:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:45.455+0000 7f7426fd1700 1 librados: shutdown 2024-02-19T21:18:45.457 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.4 to 1 (10000) 2024-02-19T21:18:45.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:44.275444+0000 osd.0 (osd.0) 154 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:18:45.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:44.276373+0000 osd.0 (osd.0) 155 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:18:45.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:44.487769+0000 osd.5 (osd.5) 346 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:18:45.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:44.490711+0000 osd.5 (osd.5) 347 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:18:45.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:45.451632+0000 mon.a (mon.0) 3840 : audit [INF] from='client.50756 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-19T21:18:45.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:45.451715+0000 mon.a (mon.0) 3841 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-02-19T21:18:45.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:44.275444+0000 osd.0 (osd.0) 154 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:18:45.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:44.276373+0000 osd.0 (osd.0) 155 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:18:45.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:44.487769+0000 osd.5 (osd.5) 346 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:18:45.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:44.490711+0000 osd.5 (osd.5) 347 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:18:45.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:45.451632+0000 mon.a (mon.0) 3840 : audit [INF] from='client.50756 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-19T21:18:45.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:45.451715+0000 mon.a (mon.0) 3841 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-02-19T21:18:45.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:44.275444+0000 osd.0 (osd.0) 154 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:18:45.822 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:44.276373+0000 osd.0 (osd.0) 155 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:18:45.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:44.487769+0000 osd.5 (osd.5) 346 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:18:45.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:44.490711+0000 osd.5 (osd.5) 347 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:18:45.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:45.451632+0000 mon.a (mon.0) 3840 : audit [INF] from='client.50756 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-19T21:18:45.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:45.451715+0000 mon.a (mon.0) 3841 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-02-19T21:18:46.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd primary-affinity 4 1 2024-02-19T21:18:46.458 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:46.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:45.360035+0000 mgr.y (mgr.14550) 700 : cluster [DBG] pgmap v1280: 174 pgs: 174 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 45 B/s, 2 objects/s recovering 2024-02-19T21:18:46.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:45.430569+0000 osd.1 (osd.1) 395 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:18:46.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:45.432022+0000 osd.1 (osd.1) 396 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:18:46.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:45.536279+0000 osd.5 (osd.5) 348 : cluster [DBG] 16.11 scrub starts 2024-02-19T21:18:46.823 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:45.539755+0000 osd.5 (osd.5) 349 : cluster [DBG] 16.11 scrub ok 2024-02-19T21:18:46.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:45.360035+0000 mgr.y (mgr.14550) 700 : cluster [DBG] pgmap v1280: 174 pgs: 174 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 45 B/s, 2 objects/s recovering 2024-02-19T21:18:46.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:45.430569+0000 osd.1 (osd.1) 395 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:18:46.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:45.432022+0000 osd.1 (osd.1) 396 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:18:46.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:45.536279+0000 osd.5 (osd.5) 348 : cluster [DBG] 16.11 scrub starts 2024-02-19T21:18:46.947 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:45.539755+0000 osd.5 (osd.5) 349 : cluster [DBG] 16.11 scrub ok 2024-02-19T21:18:46.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:45.360035+0000 mgr.y (mgr.14550) 700 : cluster [DBG] pgmap v1280: 174 pgs: 174 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 45 B/s, 2 objects/s recovering 2024-02-19T21:18:46.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:45.430569+0000 osd.1 (osd.1) 395 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:18:46.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:45.432022+0000 osd.1 (osd.1) 396 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:18:46.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:45.536279+0000 osd.5 (osd.5) 348 : cluster [DBG] 16.11 scrub starts 2024-02-19T21:18:46.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:45.539755+0000 osd.5 (osd.5) 349 : cluster [DBG] 16.11 scrub ok 2024-02-19T21:18:47.726 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:46.433817+0000 osd.4 (osd.4) 317 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:18:47.726 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:46.435438+0000 osd.4 (osd.4) 318 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:18:47.727 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:46.746119+0000 osd.6 (osd.6) 311 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:18:47.727 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:46.748068+0000 osd.6 (osd.6) 312 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:18:47.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:46.433817+0000 osd.4 (osd.4) 317 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:18:47.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:46.435438+0000 osd.4 (osd.4) 318 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:18:47.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:46.746119+0000 osd.6 (osd.6) 311 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:18:47.727 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:46.748068+0000 osd.6 (osd.6) 312 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:18:48.073 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:18:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:18:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:18:48.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:46.433817+0000 osd.4 (osd.4) 317 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:18:48.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:46.435438+0000 osd.4 (osd.4) 318 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:18:48.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:46.746119+0000 osd.6 (osd.6) 311 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:18:48.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:46.748068+0000 osd.6 (osd.6) 312 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:18:48.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.235+0000 7fb768889700 1 -- 172.21.15.163:0/784018031 >> v1:172.21.15.163:6789/0 conn(0x7fb764100780 legacy=0x7fb764102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:48.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.235+0000 7fb768889700 1 -- 172.21.15.163:0/784018031 shutdown_connections 2024-02-19T21:18:48.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.235+0000 7fb768889700 1 -- 172.21.15.163:0/784018031 >> 172.21.15.163:0/784018031 conn(0x7fb7640f35a0 msgr2=0x7fb7640f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:48.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.235+0000 7fb768889700 1 -- 172.21.15.163:0/784018031 shutdown_connections 2024-02-19T21:18:48.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.235+0000 7fb768889700 1 -- 172.21.15.163:0/784018031 wait complete. 2024-02-19T21:18:48.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.235+0000 7fb768889700 1 librados: starting msgr at 2024-02-19T21:18:48.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.235+0000 7fb768889700 1 librados: starting objecter 2024-02-19T21:18:48.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.235+0000 7fb768889700 1 Processor -- start 2024-02-19T21:18:48.239 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.235+0000 7fb768889700 1 -- start start 2024-02-19T21:18:48.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb768889700 1 librados: setting wanted keys 2024-02-19T21:18:48.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb768889700 1 librados: calling monclient init 2024-02-19T21:18:48.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb768889700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb764117320 con 0x7fb7640fd980 2024-02-19T21:18:48.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb768889700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb764118500 con 0x7fb7641042c0 2024-02-19T21:18:48.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb768889700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb764119700 con 0x7fb764100780 2024-02-19T21:18:48.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb76259c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fb7640fd980 0x7fb7640f9a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:38390/0 (socket says 172.21.15.163:38390) 2024-02-19T21:18:48.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb761d9b700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fb764100780 0x7fb7640fa170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:56010/0 (socket says 172.21.15.163:56010) 2024-02-19T21:18:48.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb76259c700 1 -- 172.21.15.163:0/3503046829 learned_addr learned my addr 172.21.15.163:0/3503046829 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:48.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 10 client.?.objecter ms_handle_connect 0x7fb764100780 2024-02-19T21:18:48.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:48.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 10 client.?.objecter ms_handle_connect 0x7fb7640fd980 2024-02-19T21:18:48.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:48.240 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 10 client.?.objecter ms_handle_connect 0x7fb7641042c0 2024-02-19T21:18:48.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:48.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100963531 0 0) 0x7fb764117320 con 0x7fb7640fd980 2024-02-19T21:18:48.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb74c002300 con 0x7fb7640fd980 2024-02-19T21:18:48.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012277890 0 0) 0x7fb764119700 con 0x7fb764100780 2024-02-19T21:18:48.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb764117320 con 0x7fb764100780 2024-02-19T21:18:48.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088260198 0 0) 0x7fb764118500 con 0x7fb7641042c0 2024-02-19T21:18:48.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb764119700 con 0x7fb7641042c0 2024-02-19T21:18:48.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4218795794 0 0) 0x7fb764117320 con 0x7fb764100780 2024-02-19T21:18:48.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb764118500 con 0x7fb764100780 2024-02-19T21:18:48.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1069817883 0 0) 0x7fb74c002300 con 0x7fb7640fd980 2024-02-19T21:18:48.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb764117320 con 0x7fb7640fd980 2024-02-19T21:18:48.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb7580015d0 con 0x7fb764100780 2024-02-19T21:18:48.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.236+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb75c0018c0 con 0x7fb7640fd980 2024-02-19T21:18:48.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.237+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3744278325 0 0) 0x7fb764119700 con 0x7fb7641042c0 2024-02-19T21:18:48.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.237+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb74c002300 con 0x7fb7641042c0 2024-02-19T21:18:48.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.237+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb754001950 con 0x7fb7641042c0 2024-02-19T21:18:48.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.237+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3056931324 0 0) 0x7fb764117320 con 0x7fb7640fd980 2024-02-19T21:18:48.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.237+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 >> v1:172.21.15.163:6790/0 conn(0x7fb764100780 legacy=0x7fb7640fa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:48.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.237+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 >> v1:172.21.15.193:6789/0 conn(0x7fb7641042c0 legacy=0x7fb7641159e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:48.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.237+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76411a900 con 0x7fb7640fd980 2024-02-19T21:18:48.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.237+0000 7fb768889700 1 -- 172.21.15.163:0/3503046829 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb764117510 con 0x7fb7640fd980 2024-02-19T21:18:48.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.237+0000 7fb768889700 10 client.24036.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:48.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.237+0000 7fb768889700 1 -- 172.21.15.163:0/3503046829 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb764117a00 con 0x7fb7640fd980 2024-02-19T21:18:48.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.237+0000 7fb768889700 1 librados: init done 2024-02-19T21:18:48.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.237+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fb75c001f00 con 0x7fb7640fd980 2024-02-19T21:18:48.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.237+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fb75c0059b0 con 0x7fb7640fd980 2024-02-19T21:18:48.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.238+0000 7fb768889700 1 -- 172.21.15.163:0/3503046829 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb744004f20 con 0x7fb7640fd980 2024-02-19T21:18:48.242 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.238+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fb75c001a20 con 0x7fb7640fd980 2024-02-19T21:18:48.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.239+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(837..837 src has 633..837) v4 ==== 7419+0+0 (unknown 2522972820 0 0) 0x7fb75c078860 con 0x7fb7640fd980 2024-02-19T21:18:48.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.239+0000 7fb7537fe700 10 client.24036.objecter ms_dispatch 0x7fb764001550 osd_map(837..837 src has 633..837) v4 2024-02-19T21:18:48.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.239+0000 7fb7537fe700 3 client.24036.objecter handle_osd_map got epochs [837,837] > 0 2024-02-19T21:18:48.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.239+0000 7fb7537fe700 3 client.24036.objecter handle_osd_map decoding full epoch 837 2024-02-19T21:18:48.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.239+0000 7fb7537fe700 20 client.24036.objecter dump_active .. 0 homeless 2024-02-19T21:18:48.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.239+0000 7fb7537fe700 10 client.24036.objecter ms_handle_connect 0x7fb74c063c50 2024-02-19T21:18:48.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.243+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fb75c04bf30 con 0x7fb7640fd980 2024-02-19T21:18:48.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.418+0000 7fb768889700 1 -- 172.21.15.163:0/3503046829 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) v1 -- 0x7fb7440059e0 con 0x7fb7640fd980 2024-02-19T21:18:48.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.583+0000 7fb7537fe700 1 -- 172.21.15.163:0/3503046829 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v838) v1 ==== 133+0+0 (unknown 2440925959 0 0) 0x7fb75c04a340 con 0x7fb7640fd980 2024-02-19T21:18:48.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.586+0000 7fb768889700 10 librados: watch_flush enter 2024-02-19T21:18:48.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.586+0000 7fb768889700 10 librados: watch_flush exit 2024-02-19T21:18:48.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.586+0000 7fb768889700 20 client.24036.objecter shutdown clearing up homeless session... 2024-02-19T21:18:48.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.586+0000 7fb768889700 10 client.24036.objecter successfully canceled tick 2024-02-19T21:18:48.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.586+0000 7fb768889700 1 -- 172.21.15.163:0/3503046829 >> v1:172.21.15.163:6800/1552514681 conn(0x7fb74c063c50 legacy=0x7fb74c066110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:48.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.586+0000 7fb768889700 1 -- 172.21.15.163:0/3503046829 >> v1:172.21.15.163:6789/0 conn(0x7fb7640fd980 legacy=0x7fb7640f9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:48.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.586+0000 7fb768889700 1 -- 172.21.15.163:0/3503046829 shutdown_connections 2024-02-19T21:18:48.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.586+0000 7fb768889700 1 -- 172.21.15.163:0/3503046829 >> 172.21.15.163:0/3503046829 conn(0x7fb7640f35a0 msgr2=0x7fb764107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:48.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.586+0000 7fb768889700 1 -- 172.21.15.163:0/3503046829 shutdown_connections 2024-02-19T21:18:48.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.586+0000 7fb768889700 1 -- 172.21.15.163:0/3503046829 wait complete. 2024-02-19T21:18:48.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:48.586+0000 7fb768889700 1 librados: shutdown 2024-02-19T21:18:48.589 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-02-19T21:18:48.928 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:47.356417+0000 osd.0 (osd.0) 156 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:18:48.928 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:47.359308+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:18:48.928 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:47.360403+0000 mgr.y (mgr.14550) 701 : cluster [DBG] pgmap v1282: 174 pgs: 174 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 878 B/s rd, 0 op/s; 37 B/s, 2 objects/s recovering 2024-02-19T21:18:48.928 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:47.415528+0000 osd.4 (osd.4) 319 : cluster [DBG] 2.26 deep-scrub starts 2024-02-19T21:18:48.928 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:47.416501+0000 osd.4 (osd.4) 320 : cluster [DBG] 2.26 deep-scrub ok 2024-02-19T21:18:48.928 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:47.715515+0000 osd.2 (osd.2) 358 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:18:48.928 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:47.716566+0000 osd.2 (osd.2) 359 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:18:48.928 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:47.744393+0000 osd.6 (osd.6) 313 : cluster [DBG] 4.a deep-scrub starts 2024-02-19T21:18:48.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:47.751134+0000 osd.6 (osd.6) 314 : cluster [DBG] 4.a deep-scrub ok 2024-02-19T21:18:48.929 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:48.418750+0000 mon.a (mon.0) 3842 : audit [INF] from='client.? v1:172.21.15.163:0/3503046829' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:18:48.929 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:47.356417+0000 osd.0 (osd.0) 156 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:18:48.929 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:47.359308+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:18:48.929 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:47.360403+0000 mgr.y (mgr.14550) 701 : cluster [DBG] pgmap v1282: 174 pgs: 174 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 878 B/s rd, 0 op/s; 37 B/s, 2 objects/s recovering 2024-02-19T21:18:48.929 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:47.415528+0000 osd.4 (osd.4) 319 : cluster [DBG] 2.26 deep-scrub starts 2024-02-19T21:18:48.929 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:47.416501+0000 osd.4 (osd.4) 320 : cluster [DBG] 2.26 deep-scrub ok 2024-02-19T21:18:48.929 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:47.715515+0000 osd.2 (osd.2) 358 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:18:48.929 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:47.716566+0000 osd.2 (osd.2) 359 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:18:48.929 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:47.744393+0000 osd.6 (osd.6) 313 : cluster [DBG] 4.a deep-scrub starts 2024-02-19T21:18:48.929 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:47.751134+0000 osd.6 (osd.6) 314 : cluster [DBG] 4.a deep-scrub ok 2024-02-19T21:18:48.929 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:48.418750+0000 mon.a (mon.0) 3842 : audit [INF] from='client.? v1:172.21.15.163:0/3503046829' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:18:49.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:47.356417+0000 osd.0 (osd.0) 156 : cluster [DBG] 6.0 scrub starts 2024-02-19T21:18:49.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:47.359308+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.0 scrub ok 2024-02-19T21:18:49.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:47.360403+0000 mgr.y (mgr.14550) 701 : cluster [DBG] pgmap v1282: 174 pgs: 174 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 878 B/s rd, 0 op/s; 37 B/s, 2 objects/s recovering 2024-02-19T21:18:49.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:47.415528+0000 osd.4 (osd.4) 319 : cluster [DBG] 2.26 deep-scrub starts 2024-02-19T21:18:49.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:47.416501+0000 osd.4 (osd.4) 320 : cluster [DBG] 2.26 deep-scrub ok 2024-02-19T21:18:49.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:47.715515+0000 osd.2 (osd.2) 358 : cluster [DBG] 5.6 scrub starts 2024-02-19T21:18:49.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:47.716566+0000 osd.2 (osd.2) 359 : cluster [DBG] 5.6 scrub ok 2024-02-19T21:18:49.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:47.744393+0000 osd.6 (osd.6) 313 : cluster [DBG] 4.a deep-scrub starts 2024-02-19T21:18:49.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:47.751134+0000 osd.6 (osd.6) 314 : cluster [DBG] 4.a deep-scrub ok 2024-02-19T21:18:49.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:48.418750+0000 mon.a (mon.0) 3842 : audit [INF] from='client.? v1:172.21.15.163:0/3503046829' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-19T21:18:49.347 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 5 1 2024-02-19T21:18:49.634 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:49.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:48.583765+0000 mon.a (mon.0) 3843 : audit [INF] from='client.? v1:172.21.15.163:0/3503046829' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-19T21:18:49.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:48.583856+0000 mon.a (mon.0) 3844 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-02-19T21:18:49.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:48.599402+0000 osd.5 (osd.5) 350 : cluster [DBG] 4.15 deep-scrub starts 2024-02-19T21:18:49.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:48.603063+0000 osd.5 (osd.5) 351 : cluster [DBG] 4.15 deep-scrub ok 2024-02-19T21:18:49.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:48.709133+0000 osd.6 (osd.6) 315 : cluster [DBG] 2.29 scrub starts 2024-02-19T21:18:49.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:48.712805+0000 osd.6 (osd.6) 316 : cluster [DBG] 2.29 scrub ok 2024-02-19T21:18:49.882 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:48.583765+0000 mon.a (mon.0) 3843 : audit [INF] from='client.? v1:172.21.15.163:0/3503046829' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-19T21:18:49.882 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:48.583856+0000 mon.a (mon.0) 3844 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-02-19T21:18:49.882 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:48.599402+0000 osd.5 (osd.5) 350 : cluster [DBG] 4.15 deep-scrub starts 2024-02-19T21:18:49.883 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:48.603063+0000 osd.5 (osd.5) 351 : cluster [DBG] 4.15 deep-scrub ok 2024-02-19T21:18:49.883 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:48.709133+0000 osd.6 (osd.6) 315 : cluster [DBG] 2.29 scrub starts 2024-02-19T21:18:49.883 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:48.712805+0000 osd.6 (osd.6) 316 : cluster [DBG] 2.29 scrub ok 2024-02-19T21:18:50.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:48.583765+0000 mon.a (mon.0) 3843 : audit [INF] from='client.? v1:172.21.15.163:0/3503046829' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-19T21:18:50.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:48.583856+0000 mon.a (mon.0) 3844 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-02-19T21:18:50.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:48.599402+0000 osd.5 (osd.5) 350 : cluster [DBG] 4.15 deep-scrub starts 2024-02-19T21:18:50.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:48.603063+0000 osd.5 (osd.5) 351 : cluster [DBG] 4.15 deep-scrub ok 2024-02-19T21:18:50.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:48.709133+0000 osd.6 (osd.6) 315 : cluster [DBG] 2.29 scrub starts 2024-02-19T21:18:50.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:48.712805+0000 osd.6 (osd.6) 316 : cluster [DBG] 2.29 scrub ok 2024-02-19T21:18:50.892 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:49.360785+0000 mgr.y (mgr.14550) 702 : cluster [DBG] pgmap v1284: 174 pgs: 174 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 882 B/s rd, 0 op/s; 37 B/s, 2 objects/s recovering 2024-02-19T21:18:50.892 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:49.663235+0000 osd.6 (osd.6) 317 : cluster [DBG] 2.17 deep-scrub starts 2024-02-19T21:18:50.892 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:49.664159+0000 osd.6 (osd.6) 318 : cluster [DBG] 2.17 deep-scrub ok 2024-02-19T21:18:50.892 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:50.287677+0000 osd.3 (osd.3) 379 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:18:50.893 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:50.289441+0000 osd.3 (osd.3) 380 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:18:50.893 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:49.360785+0000 mgr.y (mgr.14550) 702 : cluster [DBG] pgmap v1284: 174 pgs: 174 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 882 B/s rd, 0 op/s; 37 B/s, 2 objects/s recovering 2024-02-19T21:18:50.893 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:49.663235+0000 osd.6 (osd.6) 317 : cluster [DBG] 2.17 deep-scrub starts 2024-02-19T21:18:50.893 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:49.664159+0000 osd.6 (osd.6) 318 : cluster [DBG] 2.17 deep-scrub ok 2024-02-19T21:18:50.893 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:50.287677+0000 osd.3 (osd.3) 379 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:18:50.893 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:50.289441+0000 osd.3 (osd.3) 380 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:18:50.896 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:49.360785+0000 mgr.y (mgr.14550) 702 : cluster [DBG] pgmap v1284: 174 pgs: 174 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 882 B/s rd, 0 op/s; 37 B/s, 2 objects/s recovering 2024-02-19T21:18:50.896 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:49.663235+0000 osd.6 (osd.6) 317 : cluster [DBG] 2.17 deep-scrub starts 2024-02-19T21:18:50.896 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:49.664159+0000 osd.6 (osd.6) 318 : cluster [DBG] 2.17 deep-scrub ok 2024-02-19T21:18:50.896 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:50.287677+0000 osd.3 (osd.3) 379 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:18:50.896 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:50.289441+0000 osd.3 (osd.3) 380 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:18:51.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:18:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:18:51.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.415+0000 7f6f7efd3700 1 -- 172.21.15.163:0/2923051163 >> v1:172.21.15.163:6789/0 conn(0x7f6f781042c0 legacy=0x7f6f78106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:51.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.415+0000 7f6f7efd3700 1 -- 172.21.15.163:0/2923051163 shutdown_connections 2024-02-19T21:18:51.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.415+0000 7f6f7efd3700 1 -- 172.21.15.163:0/2923051163 >> 172.21.15.163:0/2923051163 conn(0x7f6f780f35a0 msgr2=0x7f6f780f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:51.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.415+0000 7f6f7efd3700 1 -- 172.21.15.163:0/2923051163 shutdown_connections 2024-02-19T21:18:51.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.415+0000 7f6f7efd3700 1 -- 172.21.15.163:0/2923051163 wait complete. 2024-02-19T21:18:51.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f7efd3700 1 librados: starting msgr at 2024-02-19T21:18:51.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f7efd3700 1 librados: starting objecter 2024-02-19T21:18:51.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f7efd3700 1 Processor -- start 2024-02-19T21:18:51.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f7efd3700 1 -- start start 2024-02-19T21:18:51.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f7efd3700 1 librados: setting wanted keys 2024-02-19T21:18:51.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f7efd3700 1 librados: calling monclient init 2024-02-19T21:18:51.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f7efd3700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6f780fcfa0 con 0x7f6f78100780 2024-02-19T21:18:51.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f7efd3700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6f780fd130 con 0x7f6f780fd980 2024-02-19T21:18:51.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f7efd3700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6f780fd320 con 0x7f6f781042c0 2024-02-19T21:18:51.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f77fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f6f78100780 0x7f6f78114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:38410/0 (socket says 172.21.15.163:38410) 2024-02-19T21:18:51.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f77fff700 1 -- 172.21.15.163:0/3841333493 learned_addr learned my addr 172.21.15.163:0/3841333493 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:51.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f7d570700 1 --1- 172.21.15.163:0/3841333493 >> v1:172.21.15.163:6790/0 conn(0x7f6f781042c0 0x7f6f78118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:56030/0 (socket says 172.21.15.163:56030) 2024-02-19T21:18:51.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f75ffb700 10 client.?.objecter ms_handle_connect 0x7f6f78100780 2024-02-19T21:18:51.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f75ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:51.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f75ffb700 10 client.?.objecter ms_handle_connect 0x7f6f781042c0 2024-02-19T21:18:51.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f75ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:51.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562138329 0 0) 0x7f6f780fcfa0 con 0x7f6f78100780 2024-02-19T21:18:51.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f680038e0 con 0x7f6f78100780 2024-02-19T21:18:51.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957674683 0 0) 0x7f6f780fd320 con 0x7f6f781042c0 2024-02-19T21:18:51.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f780fcfa0 con 0x7f6f781042c0 2024-02-19T21:18:51.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f75ffb700 10 client.?.objecter ms_handle_connect 0x7f6f780fd980 2024-02-19T21:18:51.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.416+0000 7f6f75ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:51.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413973214 0 0) 0x7f6f780fd130 con 0x7f6f780fd980 2024-02-19T21:18:51.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f780fd320 con 0x7f6f780fd980 2024-02-19T21:18:51.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3646354236 0 0) 0x7f6f680038e0 con 0x7f6f78100780 2024-02-19T21:18:51.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f780fd130 con 0x7f6f78100780 2024-02-19T21:18:51.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6f640019f0 con 0x7f6f78100780 2024-02-19T21:18:51.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 913526570 0 0) 0x7f6f780fcfa0 con 0x7f6f781042c0 2024-02-19T21:18:51.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f680038e0 con 0x7f6f781042c0 2024-02-19T21:18:51.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6f70001360 con 0x7f6f781042c0 2024-02-19T21:18:51.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2618456418 0 0) 0x7f6f780fd130 con 0x7f6f78100780 2024-02-19T21:18:51.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 >> v1:172.21.15.163:6790/0 conn(0x7f6f781042c0 legacy=0x7f6f78118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:51.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 >> v1:172.21.15.193:6789/0 conn(0x7f6f780fd980 legacy=0x7f6f780fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:51.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7811cb90 con 0x7f6f78100780 2024-02-19T21:18:51.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f7efd3700 1 -- 172.21.15.163:0/3841333493 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f78119bc0 con 0x7f6f78100780 2024-02-19T21:18:51.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6f64001c60 con 0x7f6f78100780 2024-02-19T21:18:51.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6f64005ab0 con 0x7f6f78100780 2024-02-19T21:18:51.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f7efd3700 10 client.24042.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:51.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f7efd3700 1 -- 172.21.15.163:0/3841333493 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7811a030 con 0x7f6f78100780 2024-02-19T21:18:51.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.417+0000 7f6f7efd3700 1 librados: init done 2024-02-19T21:18:51.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.418+0000 7f6f7efd3700 1 -- 172.21.15.163:0/3841333493 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f7805f660 con 0x7f6f78100780 2024-02-19T21:18:51.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.418+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6f640021b0 con 0x7f6f78100780 2024-02-19T21:18:51.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.419+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(838..838 src has 784..838) v4 ==== 7419+0+0 (unknown 3172826037 0 0) 0x7f6f640789a0 con 0x7f6f78100780 2024-02-19T21:18:51.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.419+0000 7f6f75ffb700 10 client.24042.objecter ms_dispatch 0x7f6f78001550 osd_map(838..838 src has 784..838) v4 2024-02-19T21:18:51.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.419+0000 7f6f75ffb700 3 client.24042.objecter handle_osd_map got epochs [838,838] > 0 2024-02-19T21:18:51.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.419+0000 7f6f75ffb700 3 client.24042.objecter handle_osd_map decoding full epoch 838 2024-02-19T21:18:51.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.419+0000 7f6f75ffb700 20 client.24042.objecter dump_active .. 0 homeless 2024-02-19T21:18:51.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.419+0000 7f6f75ffb700 10 client.24042.objecter ms_handle_connect 0x7f6f680639c0 2024-02-19T21:18:51.426 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.423+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6f6404c010 con 0x7f6f78100780 2024-02-19T21:18:51.600 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.599+0000 7f6f7efd3700 1 -- 172.21.15.163:0/3841333493 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f6f78047540 con 0x7f6f78100780 2024-02-19T21:18:51.622 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.621+0000 7f6f75ffb700 1 -- 172.21.15.163:0/3841333493 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v839) v1 ==== 113+0+0 (unknown 2298865814 0 0) 0x7f6f6404a420 con 0x7f6f78100780 2024-02-19T21:18:51.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.624+0000 7f6f7efd3700 10 librados: watch_flush enter 2024-02-19T21:18:51.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.624+0000 7f6f7efd3700 10 librados: watch_flush exit 2024-02-19T21:18:51.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.624+0000 7f6f7efd3700 20 client.24042.objecter shutdown clearing up homeless session... 2024-02-19T21:18:51.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.624+0000 7f6f7efd3700 10 client.24042.objecter successfully canceled tick 2024-02-19T21:18:51.625 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.624+0000 7f6f7efd3700 1 -- 172.21.15.163:0/3841333493 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6f680639c0 legacy=0x7f6f68065e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:51.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.624+0000 7f6f7efd3700 1 -- 172.21.15.163:0/3841333493 >> v1:172.21.15.163:6789/0 conn(0x7f6f78100780 legacy=0x7f6f78114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:51.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.624+0000 7f6f7efd3700 1 -- 172.21.15.163:0/3841333493 shutdown_connections 2024-02-19T21:18:51.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.624+0000 7f6f7efd3700 1 -- 172.21.15.163:0/3841333493 >> 172.21.15.163:0/3841333493 conn(0x7f6f780f35a0 msgr2=0x7f6f78107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:51.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.624+0000 7f6f7efd3700 1 -- 172.21.15.163:0/3841333493 shutdown_connections 2024-02-19T21:18:51.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.624+0000 7f6f7efd3700 1 -- 172.21.15.163:0/3841333493 wait complete. 2024-02-19T21:18:51.626 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:51.624+0000 7f6f7efd3700 1 librados: shutdown 2024-02-19T21:18:51.626 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.5 to 1 (10000) 2024-02-19T21:18:51.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:50.586643+0000 osd.5 (osd.5) 352 : cluster [DBG] 5.f scrub starts 2024-02-19T21:18:51.989 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:50.589564+0000 osd.5 (osd.5) 353 : cluster [DBG] 5.f scrub ok 2024-02-19T21:18:51.989 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:50.895038+0000 mgr.y (mgr.14550) 703 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:51.989 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:51.302174+0000 osd.3 (osd.3) 381 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:18:51.989 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:51.303852+0000 osd.3 (osd.3) 382 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:18:51.989 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:51.600470+0000 mon.a (mon.0) 3845 : audit [INF] from='client.? v1:172.21.15.163:0/3841333493' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:18:51.989 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:50.586643+0000 osd.5 (osd.5) 352 : cluster [DBG] 5.f scrub starts 2024-02-19T21:18:51.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:50.589564+0000 osd.5 (osd.5) 353 : cluster [DBG] 5.f scrub ok 2024-02-19T21:18:51.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:50.895038+0000 mgr.y (mgr.14550) 703 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:51.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:51.302174+0000 osd.3 (osd.3) 381 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:18:51.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:51.303852+0000 osd.3 (osd.3) 382 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:18:51.990 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:51.600470+0000 mon.a (mon.0) 3845 : audit [INF] from='client.? v1:172.21.15.163:0/3841333493' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:18:52.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:50.586643+0000 osd.5 (osd.5) 352 : cluster [DBG] 5.f scrub starts 2024-02-19T21:18:52.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:50.589564+0000 osd.5 (osd.5) 353 : cluster [DBG] 5.f scrub ok 2024-02-19T21:18:52.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:50.895038+0000 mgr.y (mgr.14550) 703 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:18:52.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:51.302174+0000 osd.3 (osd.3) 381 : cluster [DBG] 6.1b scrub starts 2024-02-19T21:18:52.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:51.303852+0000 osd.3 (osd.3) 382 : cluster [DBG] 6.1b scrub ok 2024-02-19T21:18:52.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:51.600470+0000 mon.a (mon.0) 3845 : audit [INF] from='client.? v1:172.21.15.163:0/3841333493' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:18:52.382 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd primary-affinity 5 1 2024-02-19T21:18:52.660 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:52.884 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:51.346631+0000 osd.0 (osd.0) 158 : cluster [DBG] 16.0 scrub starts 2024-02-19T21:18:52.884 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:51.349673+0000 osd.0 (osd.0) 159 : cluster [DBG] 16.0 scrub ok 2024-02-19T21:18:52.885 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:51.361528+0000 mgr.y (mgr.14550) 704 : cluster [DBG] pgmap v1285: 174 pgs: 174 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 31 B/s, 1 objects/s recovering 2024-02-19T21:18:52.885 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:51.561042+0000 osd.5 (osd.5) 354 : cluster [DBG] 2.2 scrub starts 2024-02-19T21:18:52.885 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:51.564014+0000 osd.5 (osd.5) 355 : cluster [DBG] 2.2 scrub ok 2024-02-19T21:18:52.885 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:51.620821+0000 mon.a (mon.0) 3846 : audit [INF] from='client.? v1:172.21.15.163:0/3841333493' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-19T21:18:52.885 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:51.620879+0000 mon.a (mon.0) 3847 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-02-19T21:18:52.885 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:51.641916+0000 mon.b (mon.1) 770 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:18:52.885 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:51.943299+0000 mon.b (mon.1) 771 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:18:52.885 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:51.943968+0000 mon.b (mon.1) 772 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:18:52.885 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:51.948173+0000 mon.a (mon.0) 3848 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:18:52.885 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:51.346631+0000 osd.0 (osd.0) 158 : cluster [DBG] 16.0 scrub starts 2024-02-19T21:18:52.885 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:51.349673+0000 osd.0 (osd.0) 159 : cluster [DBG] 16.0 scrub ok 2024-02-19T21:18:52.886 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:51.361528+0000 mgr.y (mgr.14550) 704 : cluster [DBG] pgmap v1285: 174 pgs: 174 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 31 B/s, 1 objects/s recovering 2024-02-19T21:18:52.886 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:51.561042+0000 osd.5 (osd.5) 354 : cluster [DBG] 2.2 scrub starts 2024-02-19T21:18:52.886 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:51.564014+0000 osd.5 (osd.5) 355 : cluster [DBG] 2.2 scrub ok 2024-02-19T21:18:52.886 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:51.620821+0000 mon.a (mon.0) 3846 : audit [INF] from='client.? v1:172.21.15.163:0/3841333493' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-19T21:18:52.886 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:51.620879+0000 mon.a (mon.0) 3847 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-02-19T21:18:52.886 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:51.641916+0000 mon.b (mon.1) 770 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:18:52.886 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:51.943299+0000 mon.b (mon.1) 771 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:18:52.886 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:51.943968+0000 mon.b (mon.1) 772 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:18:52.886 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:51.948173+0000 mon.a (mon.0) 3848 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:18:53.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:51.346631+0000 osd.0 (osd.0) 158 : cluster [DBG] 16.0 scrub starts 2024-02-19T21:18:53.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:51.349673+0000 osd.0 (osd.0) 159 : cluster [DBG] 16.0 scrub ok 2024-02-19T21:18:53.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:51.361528+0000 mgr.y (mgr.14550) 704 : cluster [DBG] pgmap v1285: 174 pgs: 174 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 31 B/s, 1 objects/s recovering 2024-02-19T21:18:53.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:51.561042+0000 osd.5 (osd.5) 354 : cluster [DBG] 2.2 scrub starts 2024-02-19T21:18:53.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:51.564014+0000 osd.5 (osd.5) 355 : cluster [DBG] 2.2 scrub ok 2024-02-19T21:18:53.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:51.620821+0000 mon.a (mon.0) 3846 : audit [INF] from='client.? v1:172.21.15.163:0/3841333493' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-19T21:18:53.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:51.620879+0000 mon.a (mon.0) 3847 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-02-19T21:18:53.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:51.641916+0000 mon.b (mon.1) 770 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:18:53.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:51.943299+0000 mon.b (mon.1) 771 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:18:53.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:51.943968+0000 mon.b (mon.1) 772 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:18:53.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:51.948173+0000 mon.a (mon.0) 3848 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:18:53.749 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:18:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:18:53] "GET /metrics HTTP/1.1" 200 187979 "" "Prometheus/2.33.4" 2024-02-19T21:18:53.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:52.536374+0000 osd.5 (osd.5) 356 : cluster [DBG] 16.1f scrub starts 2024-02-19T21:18:53.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:52.540273+0000 osd.5 (osd.5) 357 : cluster [DBG] 16.1f scrub ok 2024-02-19T21:18:53.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:52.536374+0000 osd.5 (osd.5) 356 : cluster [DBG] 16.1f scrub starts 2024-02-19T21:18:53.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:52.540273+0000 osd.5 (osd.5) 357 : cluster [DBG] 16.1f scrub ok 2024-02-19T21:18:54.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:52.536374+0000 osd.5 (osd.5) 356 : cluster [DBG] 16.1f scrub starts 2024-02-19T21:18:54.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:52.540273+0000 osd.5 (osd.5) 357 : cluster [DBG] 16.1f scrub ok 2024-02-19T21:18:54.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.369+0000 7f4b017a1700 1 -- 172.21.15.163:0/2197339106 >> v1:172.21.15.163:6790/0 conn(0x7f4afc0ff690 legacy=0x7f4afc101a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:54.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.369+0000 7f4b017a1700 1 -- 172.21.15.163:0/2197339106 shutdown_connections 2024-02-19T21:18:54.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.369+0000 7f4b017a1700 1 -- 172.21.15.163:0/2197339106 >> 172.21.15.163:0/2197339106 conn(0x7f4afc0f2470 msgr2=0x7f4afc0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:54.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.369+0000 7f4b017a1700 1 -- 172.21.15.163:0/2197339106 shutdown_connections 2024-02-19T21:18:54.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.369+0000 7f4b017a1700 1 -- 172.21.15.163:0/2197339106 wait complete. 2024-02-19T21:18:54.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.369+0000 7f4b017a1700 1 librados: starting msgr at 2024-02-19T21:18:54.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.369+0000 7f4b017a1700 1 librados: starting objecter 2024-02-19T21:18:54.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.369+0000 7f4b017a1700 1 Processor -- start 2024-02-19T21:18:54.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4b017a1700 1 -- start start 2024-02-19T21:18:54.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4b017a1700 1 librados: setting wanted keys 2024-02-19T21:18:54.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4b017a1700 1 librados: calling monclient init 2024-02-19T21:18:54.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4b017a1700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4afc0fbf60 con 0x7f4afc1031d0 2024-02-19T21:18:54.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4b017a1700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4afc0fc130 con 0x7f4afc0fc890 2024-02-19T21:18:54.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4b017a1700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4afc11baa0 con 0x7f4afc0ff690 2024-02-19T21:18:54.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4afa7fc700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f4afc0ff690 0x7f4afc114c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:42292/0 (socket says 172.21.15.163:42292) 2024-02-19T21:18:54.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4afb7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f4afc1031d0 0x7f4afc118360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:42598/0 (socket says 172.21.15.163:42598) 2024-02-19T21:18:54.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4afa7fc700 1 -- 172.21.15.163:0/1346996961 learned_addr learned my addr 172.21.15.163:0/1346996961 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:54.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 10 client.?.objecter ms_handle_connect 0x7f4afc1031d0 2024-02-19T21:18:54.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:54.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 10 client.?.objecter ms_handle_connect 0x7f4afc0ff690 2024-02-19T21:18:54.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:54.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 10 client.?.objecter ms_handle_connect 0x7f4afc0fc890 2024-02-19T21:18:54.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:54.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923493365 0 0) 0x7f4afc0fbf60 con 0x7f4afc1031d0 2024-02-19T21:18:54.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae4003800 con 0x7f4afc1031d0 2024-02-19T21:18:54.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480958373 0 0) 0x7f4afc11baa0 con 0x7f4afc0ff690 2024-02-19T21:18:54.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4afc0fbf60 con 0x7f4afc0ff690 2024-02-19T21:18:54.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220816214 0 0) 0x7f4afc0fc130 con 0x7f4afc0fc890 2024-02-19T21:18:54.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4afc11baa0 con 0x7f4afc0fc890 2024-02-19T21:18:54.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 481718676 0 0) 0x7f4ae4003800 con 0x7f4afc1031d0 2024-02-19T21:18:54.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4afc0fc130 con 0x7f4afc1031d0 2024-02-19T21:18:54.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4af4001e00 con 0x7f4afc1031d0 2024-02-19T21:18:54.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3092893082 0 0) 0x7f4afc0fbf60 con 0x7f4afc0ff690 2024-02-19T21:18:54.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ae4003800 con 0x7f4afc0ff690 2024-02-19T21:18:54.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4af0002f60 con 0x7f4afc0ff690 2024-02-19T21:18:54.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2598094769 0 0) 0x7f4afc11baa0 con 0x7f4afc0fc890 2024-02-19T21:18:54.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4afc0fbf60 con 0x7f4afc0fc890 2024-02-19T21:18:54.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.370+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4aec0018c0 con 0x7f4afc0fc890 2024-02-19T21:18:54.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.371+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4085704586 0 0) 0x7f4afc0fc130 con 0x7f4afc1031d0 2024-02-19T21:18:54.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.371+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 >> v1:172.21.15.163:6790/0 conn(0x7f4afc0ff690 legacy=0x7f4afc114c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:54.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.371+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 >> v1:172.21.15.193:6789/0 conn(0x7f4afc0fc890 legacy=0x7f4afc0fb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:54.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.371+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc11cc40 con 0x7f4afc1031d0 2024-02-19T21:18:54.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.371+0000 7f4b017a1700 1 -- 172.21.15.163:0/1346996961 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4afc11bc90 con 0x7f4afc1031d0 2024-02-19T21:18:54.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.371+0000 7f4b017a1700 10 client.24048.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:54.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.371+0000 7f4b017a1700 1 -- 172.21.15.163:0/1346996961 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4afc11c100 con 0x7f4afc1031d0 2024-02-19T21:18:54.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.371+0000 7f4b017a1700 1 librados: init done 2024-02-19T21:18:54.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.371+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f4af4002770 con 0x7f4afc1031d0 2024-02-19T21:18:54.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.371+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4af4005d00 con 0x7f4afc1031d0 2024-02-19T21:18:54.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.372+0000 7f4b017a1700 1 -- 172.21.15.163:0/1346996961 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ae8004f20 con 0x7f4afc1031d0 2024-02-19T21:18:54.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.372+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f4af40028e0 con 0x7f4afc1031d0 2024-02-19T21:18:54.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.372+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(839..839 src has 784..839) v4 ==== 7419+0+0 (unknown 1927340707 0 0) 0x7f4af4078cc0 con 0x7f4afc1031d0 2024-02-19T21:18:54.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.372+0000 7f4ae3fff700 10 client.24048.objecter ms_dispatch 0x7f4afc001550 osd_map(839..839 src has 784..839) v4 2024-02-19T21:18:54.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.372+0000 7f4ae3fff700 3 client.24048.objecter handle_osd_map got epochs [839,839] > 0 2024-02-19T21:18:54.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.372+0000 7f4ae3fff700 3 client.24048.objecter handle_osd_map decoding full epoch 839 2024-02-19T21:18:54.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.373+0000 7f4ae3fff700 20 client.24048.objecter dump_active .. 0 homeless 2024-02-19T21:18:54.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.373+0000 7f4ae3fff700 10 client.24048.objecter ms_handle_connect 0x7f4ae4063c90 2024-02-19T21:18:54.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.377+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f4af404c330 con 0x7f4afc1031d0 2024-02-19T21:18:54.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.557+0000 7f4b017a1700 1 -- 172.21.15.163:0/1346996961 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) v1 -- 0x7f4ae80059e0 con 0x7f4afc1031d0 2024-02-19T21:18:54.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.633+0000 7f4ae3fff700 1 -- 172.21.15.163:0/1346996961 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v840) v1 ==== 133+0+0 (unknown 3730654700 0 0) 0x7f4af404a740 con 0x7f4afc1031d0 2024-02-19T21:18:54.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.636+0000 7f4b017a1700 10 librados: watch_flush enter 2024-02-19T21:18:54.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.636+0000 7f4b017a1700 10 librados: watch_flush exit 2024-02-19T21:18:54.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.636+0000 7f4b017a1700 20 client.24048.objecter shutdown clearing up homeless session... 2024-02-19T21:18:54.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.636+0000 7f4b017a1700 10 client.24048.objecter successfully canceled tick 2024-02-19T21:18:54.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.636+0000 7f4b017a1700 1 -- 172.21.15.163:0/1346996961 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4ae4063c90 legacy=0x7f4ae4066150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:54.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.636+0000 7f4b017a1700 1 -- 172.21.15.163:0/1346996961 >> v1:172.21.15.163:6789/0 conn(0x7f4afc1031d0 legacy=0x7f4afc118360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:54.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.636+0000 7f4b017a1700 1 -- 172.21.15.163:0/1346996961 shutdown_connections 2024-02-19T21:18:54.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.636+0000 7f4b017a1700 1 -- 172.21.15.163:0/1346996961 >> 172.21.15.163:0/1346996961 conn(0x7f4afc0f2470 msgr2=0x7f4afc106640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:54.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.637+0000 7f4b017a1700 1 -- 172.21.15.163:0/1346996961 shutdown_connections 2024-02-19T21:18:54.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.637+0000 7f4b017a1700 1 -- 172.21.15.163:0/1346996961 wait complete. 2024-02-19T21:18:54.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:54.637+0000 7f4b017a1700 1 librados: shutdown 2024-02-19T21:18:54.640 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-02-19T21:18:54.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:53.317485+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:18:54.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:53.320474+0000 osd.0 (osd.0) 161 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:18:54.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:53.361876+0000 mgr.y (mgr.14550) 705 : cluster [DBG] pgmap v1287: 174 pgs: 174 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2024-02-19T21:18:54.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:53.370769+0000 osd.1 (osd.1) 397 : cluster [DBG] 4.f scrub starts 2024-02-19T21:18:54.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:53.374246+0000 osd.1 (osd.1) 398 : cluster [DBG] 4.f scrub ok 2024-02-19T21:18:54.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:53.553532+0000 osd.4 (osd.4) 321 : cluster [DBG] 5.e scrub starts 2024-02-19T21:18:55.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:53.555062+0000 osd.4 (osd.4) 322 : cluster [DBG] 5.e scrub ok 2024-02-19T21:18:55.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:53.674035+0000 osd.6 (osd.6) 319 : cluster [DBG] 6.5 deep-scrub starts 2024-02-19T21:18:55.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:53.675134+0000 osd.6 (osd.6) 320 : cluster [DBG] 6.5 deep-scrub ok 2024-02-19T21:18:55.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:54.558626+0000 mon.a (mon.0) 3849 : audit [INF] from='client.? v1:172.21.15.163:0/1346996961' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:18:55.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:53.317485+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:18:55.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:53.320474+0000 osd.0 (osd.0) 161 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:18:55.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:53.361876+0000 mgr.y (mgr.14550) 705 : cluster [DBG] pgmap v1287: 174 pgs: 174 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2024-02-19T21:18:55.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:53.370769+0000 osd.1 (osd.1) 397 : cluster [DBG] 4.f scrub starts 2024-02-19T21:18:55.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:53.374246+0000 osd.1 (osd.1) 398 : cluster [DBG] 4.f scrub ok 2024-02-19T21:18:55.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:53.553532+0000 osd.4 (osd.4) 321 : cluster [DBG] 5.e scrub starts 2024-02-19T21:18:55.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:53.555062+0000 osd.4 (osd.4) 322 : cluster [DBG] 5.e scrub ok 2024-02-19T21:18:55.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:53.674035+0000 osd.6 (osd.6) 319 : cluster [DBG] 6.5 deep-scrub starts 2024-02-19T21:18:55.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:53.675134+0000 osd.6 (osd.6) 320 : cluster [DBG] 6.5 deep-scrub ok 2024-02-19T21:18:55.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:54.558626+0000 mon.a (mon.0) 3849 : audit [INF] from='client.? v1:172.21.15.163:0/1346996961' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:18:55.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:53.317485+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.18 scrub starts 2024-02-19T21:18:55.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:53.320474+0000 osd.0 (osd.0) 161 : cluster [DBG] 6.18 scrub ok 2024-02-19T21:18:55.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:53.361876+0000 mgr.y (mgr.14550) 705 : cluster [DBG] pgmap v1287: 174 pgs: 174 active+clean; 5.7 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2024-02-19T21:18:55.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:53.370769+0000 osd.1 (osd.1) 397 : cluster [DBG] 4.f scrub starts 2024-02-19T21:18:55.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:53.374246+0000 osd.1 (osd.1) 398 : cluster [DBG] 4.f scrub ok 2024-02-19T21:18:55.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:53.553532+0000 osd.4 (osd.4) 321 : cluster [DBG] 5.e scrub starts 2024-02-19T21:18:55.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:53.555062+0000 osd.4 (osd.4) 322 : cluster [DBG] 5.e scrub ok 2024-02-19T21:18:55.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:53.674035+0000 osd.6 (osd.6) 319 : cluster [DBG] 6.5 deep-scrub starts 2024-02-19T21:18:55.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:53.675134+0000 osd.6 (osd.6) 320 : cluster [DBG] 6.5 deep-scrub ok 2024-02-19T21:18:55.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:54.558626+0000 mon.a (mon.0) 3849 : audit [INF] from='client.? v1:172.21.15.163:0/1346996961' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-19T21:18:55.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 6 1 2024-02-19T21:18:55.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:54.413893+0000 osd.1 (osd.1) 399 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:18:55.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:54.416843+0000 osd.1 (osd.1) 400 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:18:55.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:54.532385+0000 osd.4 (osd.4) 323 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:18:55.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:54.534289+0000 osd.4 (osd.4) 324 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:18:55.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:54.634726+0000 mon.a (mon.0) 3850 : audit [INF] from='client.? v1:172.21.15.163:0/1346996961' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-19T21:18:55.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:54.634767+0000 mon.a (mon.0) 3851 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2024-02-19T21:18:55.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:55.029943+0000 osd.7 (osd.7) 339 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:55.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:55.030174+0000 osd.7 (osd.7) 340 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:55.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:54.413893+0000 osd.1 (osd.1) 399 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:18:55.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:54.416843+0000 osd.1 (osd.1) 400 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:18:55.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:54.532385+0000 osd.4 (osd.4) 323 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:18:55.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:54.534289+0000 osd.4 (osd.4) 324 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:18:55.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:54.634726+0000 mon.a (mon.0) 3850 : audit [INF] from='client.? v1:172.21.15.163:0/1346996961' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-19T21:18:55.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:54.634767+0000 mon.a (mon.0) 3851 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2024-02-19T21:18:55.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:55.029943+0000 osd.7 (osd.7) 339 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:55.751 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:55.030174+0000 osd.7 (osd.7) 340 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:55.755 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:56.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:54.413893+0000 osd.1 (osd.1) 399 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:18:56.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:54.416843+0000 osd.1 (osd.1) 400 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:18:56.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:54.532385+0000 osd.4 (osd.4) 323 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:18:56.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:54.534289+0000 osd.4 (osd.4) 324 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:18:56.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:54.634726+0000 mon.a (mon.0) 3850 : audit [INF] from='client.? v1:172.21.15.163:0/1346996961' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-19T21:18:56.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:54.634767+0000 mon.a (mon.0) 3851 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2024-02-19T21:18:56.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:55.029943+0000 osd.7 (osd.7) 339 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:56.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:55.030174+0000 osd.7 (osd.7) 340 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:56.867 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:55.362723+0000 mgr.y (mgr.14550) 706 : cluster [DBG] pgmap v1289: 174 pgs: 174 active+clean; 5.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-19T21:18:56.868 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:55.649020+0000 mon.a (mon.0) 3852 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2024-02-19T21:18:56.868 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:55.362723+0000 mgr.y (mgr.14550) 706 : cluster [DBG] pgmap v1289: 174 pgs: 174 active+clean; 5.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-19T21:18:56.868 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:55.649020+0000 mon.a (mon.0) 3852 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2024-02-19T21:18:57.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:55.362723+0000 mgr.y (mgr.14550) 706 : cluster [DBG] pgmap v1289: 174 pgs: 174 active+clean; 5.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-19T21:18:57.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:55.649020+0000 mon.a (mon.0) 3852 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2024-02-19T21:18:57.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.367+0000 7f5c397eb700 1 -- 172.21.15.163:0/1679885828 >> v1:172.21.15.163:6790/0 conn(0x7f5c340fd960 legacy=0x7f5c340fdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:57.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.367+0000 7f5c397eb700 1 -- 172.21.15.163:0/1679885828 shutdown_connections 2024-02-19T21:18:57.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.367+0000 7f5c397eb700 1 -- 172.21.15.163:0/1679885828 >> 172.21.15.163:0/1679885828 conn(0x7f5c3406cc00 msgr2=0x7f5c340f4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:57.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.367+0000 7f5c397eb700 1 -- 172.21.15.163:0/1679885828 shutdown_connections 2024-02-19T21:18:57.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.367+0000 7f5c397eb700 1 -- 172.21.15.163:0/1679885828 wait complete. 2024-02-19T21:18:57.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c397eb700 1 librados: starting msgr at 2024-02-19T21:18:57.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c397eb700 1 librados: starting objecter 2024-02-19T21:18:57.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c397eb700 1 Processor -- start 2024-02-19T21:18:57.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c397eb700 1 -- start start 2024-02-19T21:18:57.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c397eb700 1 librados: setting wanted keys 2024-02-19T21:18:57.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c397eb700 1 librados: calling monclient init 2024-02-19T21:18:57.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c397eb700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c341172b0 con 0x7f5c340fd960 2024-02-19T21:18:57.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c397eb700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c34118490 con 0x7f5c341007b0 2024-02-19T21:18:57.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c397eb700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c34119690 con 0x7f5c341042c0 2024-02-19T21:18:57.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c337fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f5c341042c0 0x7f5c34115970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:42306/0 (socket says 172.21.15.163:42306) 2024-02-19T21:18:57.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c337fe700 1 -- 172.21.15.163:0/1472916080 learned_addr learned my addr 172.21.15.163:0/1472916080 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:18:57.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c32ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f5c340fd960 0x7f5c340f9c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:42618/0 (socket says 172.21.15.163:42618) 2024-02-19T21:18:57.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c1bfff700 10 client.?.objecter ms_handle_connect 0x7f5c340fd960 2024-02-19T21:18:57.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c1bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:57.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c1bfff700 10 client.?.objecter ms_handle_connect 0x7f5c341042c0 2024-02-19T21:18:57.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c1bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:57.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c1bfff700 10 client.?.objecter ms_handle_connect 0x7f5c341007b0 2024-02-19T21:18:57.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c1bfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:18:57.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.368+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677384420 0 0) 0x7f5c34119690 con 0x7f5c341042c0 2024-02-19T21:18:57.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c1c003030 con 0x7f5c341042c0 2024-02-19T21:18:57.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962259043 0 0) 0x7f5c341172b0 con 0x7f5c340fd960 2024-02-19T21:18:57.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c34119690 con 0x7f5c340fd960 2024-02-19T21:18:57.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065903210 0 0) 0x7f5c34118490 con 0x7f5c341007b0 2024-02-19T21:18:57.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c341172b0 con 0x7f5c341007b0 2024-02-19T21:18:57.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2351240485 0 0) 0x7f5c1c003030 con 0x7f5c341042c0 2024-02-19T21:18:57.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c34118490 con 0x7f5c341042c0 2024-02-19T21:18:57.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5c2c001c80 con 0x7f5c341042c0 2024-02-19T21:18:57.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1888530365 0 0) 0x7f5c34119690 con 0x7f5c340fd960 2024-02-19T21:18:57.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c1c003030 con 0x7f5c340fd960 2024-02-19T21:18:57.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5c28002020 con 0x7f5c340fd960 2024-02-19T21:18:57.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3038319604 0 0) 0x7f5c341172b0 con 0x7f5c341007b0 2024-02-19T21:18:57.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c34119690 con 0x7f5c341007b0 2024-02-19T21:18:57.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5c24001970 con 0x7f5c341007b0 2024-02-19T21:18:57.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3115672376 0 0) 0x7f5c34118490 con 0x7f5c341042c0 2024-02-19T21:18:57.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 >> v1:172.21.15.193:6789/0 conn(0x7f5c341007b0 legacy=0x7f5c340fa310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:57.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 >> v1:172.21.15.163:6789/0 conn(0x7f5c340fd960 legacy=0x7f5c340f9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:57.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3411a890 con 0x7f5c341042c0 2024-02-19T21:18:57.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c397eb700 1 -- 172.21.15.163:0/1472916080 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c34119880 con 0x7f5c341042c0 2024-02-19T21:18:57.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c397eb700 10 client.50780.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:18:57.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c397eb700 1 -- 172.21.15.163:0/1472916080 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c34119d70 con 0x7f5c341042c0 2024-02-19T21:18:57.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.369+0000 7f5c397eb700 1 librados: init done 2024-02-19T21:18:57.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.370+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f5c2c0027a0 con 0x7f5c341042c0 2024-02-19T21:18:57.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.370+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5c2c005df0 con 0x7f5c341042c0 2024-02-19T21:18:57.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.370+0000 7f5c397eb700 1 -- 172.21.15.163:0/1472916080 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c20004f20 con 0x7f5c341042c0 2024-02-19T21:18:57.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.370+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f5c2c018da0 con 0x7f5c341042c0 2024-02-19T21:18:57.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.371+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(841..841 src has 784..841) v4 ==== 7419+0+0 (unknown 3132139427 0 0) 0x7f5c2c078d80 con 0x7f5c341042c0 2024-02-19T21:18:57.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.371+0000 7f5c1bfff700 10 client.50780.objecter ms_dispatch 0x7f5c34001550 osd_map(841..841 src has 784..841) v4 2024-02-19T21:18:57.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.371+0000 7f5c1bfff700 3 client.50780.objecter handle_osd_map got epochs [841,841] > 0 2024-02-19T21:18:57.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.371+0000 7f5c1bfff700 3 client.50780.objecter handle_osd_map decoding full epoch 841 2024-02-19T21:18:57.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.371+0000 7f5c1bfff700 20 client.50780.objecter dump_active .. 0 homeless 2024-02-19T21:18:57.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.371+0000 7f5c1bfff700 10 client.50780.objecter ms_handle_connect 0x7f5c1c063c20 2024-02-19T21:18:57.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.375+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f5c2c04d380 con 0x7f5c341042c0 2024-02-19T21:18:57.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.557+0000 7f5c397eb700 1 -- 172.21.15.163:0/1472916080 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) v1 -- 0x7f5c200051d0 con 0x7f5c341042c0 2024-02-19T21:18:57.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.650+0000 7f5c1bfff700 1 -- 172.21.15.163:0/1472916080 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v842) v1 ==== 113+0+0 (unknown 1393785421 0 0) 0x7f5c2c04b790 con 0x7f5c341042c0 2024-02-19T21:18:57.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.652+0000 7f5c397eb700 10 librados: watch_flush enter 2024-02-19T21:18:57.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.652+0000 7f5c397eb700 10 librados: watch_flush exit 2024-02-19T21:18:57.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.653+0000 7f5c397eb700 20 client.50780.objecter shutdown clearing up homeless session... 2024-02-19T21:18:57.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.653+0000 7f5c397eb700 10 client.50780.objecter successfully canceled tick 2024-02-19T21:18:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.653+0000 7f5c397eb700 1 -- 172.21.15.163:0/1472916080 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5c1c063c20 legacy=0x7f5c1c0660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.653+0000 7f5c397eb700 1 -- 172.21.15.163:0/1472916080 >> v1:172.21.15.163:6790/0 conn(0x7f5c341042c0 legacy=0x7f5c34115970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:18:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.653+0000 7f5c397eb700 1 -- 172.21.15.163:0/1472916080 shutdown_connections 2024-02-19T21:18:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.653+0000 7f5c397eb700 1 -- 172.21.15.163:0/1472916080 >> 172.21.15.163:0/1472916080 conn(0x7f5c3406cc00 msgr2=0x7f5c34107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:18:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.653+0000 7f5c397eb700 1 -- 172.21.15.163:0/1472916080 shutdown_connections 2024-02-19T21:18:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.653+0000 7f5c397eb700 1 -- 172.21.15.163:0/1472916080 wait complete. 2024-02-19T21:18:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:18:57.653+0000 7f5c397eb700 1 librados: shutdown 2024-02-19T21:18:57.656 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.6 to 1 (10000) 2024-02-19T21:18:57.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:56.473207+0000 osd.1 (osd.1) 401 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:18:57.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:56.474367+0000 osd.1 (osd.1) 402 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:18:57.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:57.558741+0000 mon.c (mon.2) 449 : audit [INF] from='client.? v1:172.21.15.163:0/1472916080' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:18:57.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:57.559030+0000 mon.a (mon.0) 3853 : audit [INF] from='client.50780 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:18:58.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:56.473207+0000 osd.1 (osd.1) 401 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:18:58.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:56.474367+0000 osd.1 (osd.1) 402 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:18:58.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:57.558741+0000 mon.c (mon.2) 449 : audit [INF] from='client.? v1:172.21.15.163:0/1472916080' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:18:58.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:57 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:57.559030+0000 mon.a (mon.0) 3853 : audit [INF] from='client.50780 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:18:58.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:18:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:18:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:18:58.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:56.473207+0000 osd.1 (osd.1) 401 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:18:58.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:56.474367+0000 osd.1 (osd.1) 402 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:18:58.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:57.558741+0000 mon.c (mon.2) 449 : audit [INF] from='client.? v1:172.21.15.163:0/1472916080' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:18:58.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:57.559030+0000 mon.a (mon.0) 3853 : audit [INF] from='client.50780 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:18:58.439 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd primary-affinity 6 1 2024-02-19T21:18:58.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:57.363109+0000 mgr.y (mgr.14550) 707 : cluster [DBG] pgmap v1291: 174 pgs: 174 active+clean; 5.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:18:58.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:18:57.651181+0000 mon.a (mon.0) 3854 : audit [INF] from='client.50780 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-19T21:18:58.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:57.651243+0000 mon.a (mon.0) 3855 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2024-02-19T21:18:58.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:57.661657+0000 osd.6 (osd.6) 321 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:58.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:57.661956+0000 osd.6 (osd.6) 322 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:58.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:58.208379+0000 osd.3 (osd.3) 383 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:18:58.670 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:58.210816+0000 osd.3 (osd.3) 384 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:18:58.670 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:57.363109+0000 mgr.y (mgr.14550) 707 : cluster [DBG] pgmap v1291: 174 pgs: 174 active+clean; 5.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:18:58.670 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:18:57.651181+0000 mon.a (mon.0) 3854 : audit [INF] from='client.50780 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-19T21:18:58.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:57.651243+0000 mon.a (mon.0) 3855 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2024-02-19T21:18:58.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:57.661657+0000 osd.6 (osd.6) 321 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:58.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:57.661956+0000 osd.6 (osd.6) 322 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:58.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:58.208379+0000 osd.3 (osd.3) 383 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:18:58.671 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:58.210816+0000 osd.3 (osd.3) 384 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:18:58.715 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:18:59.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:57.363109+0000 mgr.y (mgr.14550) 707 : cluster [DBG] pgmap v1291: 174 pgs: 174 active+clean; 5.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:18:59.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:18:57.651181+0000 mon.a (mon.0) 3854 : audit [INF] from='client.50780 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-19T21:18:59.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:57.651243+0000 mon.a (mon.0) 3855 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2024-02-19T21:18:59.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:57.661657+0000 osd.6 (osd.6) 321 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:18:59.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:57.661956+0000 osd.6 (osd.6) 322 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:18:59.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:58.208379+0000 osd.3 (osd.3) 383 : cluster [DBG] 4.19 scrub starts 2024-02-19T21:18:59.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:58.210816+0000 osd.3 (osd.3) 384 : cluster [DBG] 4.19 scrub ok 2024-02-19T21:18:59.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:58.661232+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2024-02-19T21:18:59.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:58.951867+0000 osd.2 (osd.2) 360 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:18:59.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:58.952921+0000 osd.2 (osd.2) 361 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:18:59.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:59.212217+0000 osd.3 (osd.3) 385 : cluster [DBG] 2.28 scrub starts 2024-02-19T21:19:00.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:18:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:59.213415+0000 osd.3 (osd.3) 386 : cluster [DBG] 2.28 scrub ok 2024-02-19T21:19:00.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:58.661232+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2024-02-19T21:19:00.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:58.951867+0000 osd.2 (osd.2) 360 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:19:00.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:58.952921+0000 osd.2 (osd.2) 361 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:19:00.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:59.212217+0000 osd.3 (osd.3) 385 : cluster [DBG] 2.28 scrub starts 2024-02-19T21:19:00.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:18:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:59.213415+0000 osd.3 (osd.3) 386 : cluster [DBG] 2.28 scrub ok 2024-02-19T21:19:00.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:58.661232+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2024-02-19T21:19:00.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:58.951867+0000 osd.2 (osd.2) 360 : cluster [DBG] 16.1c scrub starts 2024-02-19T21:19:00.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:58.952921+0000 osd.2 (osd.2) 361 : cluster [DBG] 16.1c scrub ok 2024-02-19T21:19:00.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:59.212217+0000 osd.3 (osd.3) 385 : cluster [DBG] 2.28 scrub starts 2024-02-19T21:19:00.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:18:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:59.213415+0000 osd.3 (osd.3) 386 : cluster [DBG] 2.28 scrub ok 2024-02-19T21:19:00.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.525+0000 7ff2c98d7700 1 -- 172.21.15.163:0/2322729264 >> v1:172.21.15.163:6790/0 conn(0x7ff2c40fd980 legacy=0x7ff2c40fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:00.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.525+0000 7ff2c98d7700 1 -- 172.21.15.163:0/2322729264 shutdown_connections 2024-02-19T21:19:00.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.525+0000 7ff2c98d7700 1 -- 172.21.15.163:0/2322729264 >> 172.21.15.163:0/2322729264 conn(0x7ff2c40f35a0 msgr2=0x7ff2c40f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:00.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.525+0000 7ff2c98d7700 1 -- 172.21.15.163:0/2322729264 shutdown_connections 2024-02-19T21:19:00.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.525+0000 7ff2c98d7700 1 -- 172.21.15.163:0/2322729264 wait complete. 2024-02-19T21:19:00.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.525+0000 7ff2c98d7700 1 librados: starting msgr at 2024-02-19T21:19:00.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.525+0000 7ff2c98d7700 1 librados: starting objecter 2024-02-19T21:19:00.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2c98d7700 1 Processor -- start 2024-02-19T21:19:00.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2c98d7700 1 -- start start 2024-02-19T21:19:00.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2c98d7700 1 librados: setting wanted keys 2024-02-19T21:19:00.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2c98d7700 1 librados: calling monclient init 2024-02-19T21:19:00.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2c98d7700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff2c406cbb0 con 0x7ff2c40fd980 2024-02-19T21:19:00.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2c98d7700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff2c406cd40 con 0x7ff2c41042c0 2024-02-19T21:19:00.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2c98d7700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff2c406cf10 con 0x7ff2c4100780 2024-02-19T21:19:00.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2c27fc700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7ff2c4100780 0x7ff2c4114d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:42328/0 (socket says 172.21.15.163:42328) 2024-02-19T21:19:00.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2c27fc700 1 -- 172.21.15.163:0/683067062 learned_addr learned my addr 172.21.15.163:0/683067062 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:00.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2c2ffd700 1 --1- 172.21.15.163:0/683067062 >> v1:172.21.15.163:6789/0 conn(0x7ff2c40fd980 0x7ff2c40fc990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:42646/0 (socket says 172.21.15.163:42646) 2024-02-19T21:19:00.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2abfff700 10 client.?.objecter ms_handle_connect 0x7ff2c4100780 2024-02-19T21:19:00.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2abfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:00.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2abfff700 10 client.?.objecter ms_handle_connect 0x7ff2c40fd980 2024-02-19T21:19:00.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2abfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:00.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789186434 0 0) 0x7ff2c406cf10 con 0x7ff2c4100780 2024-02-19T21:19:00.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2ac0038e0 con 0x7ff2c4100780 2024-02-19T21:19:00.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2abfff700 10 client.?.objecter ms_handle_connect 0x7ff2c41042c0 2024-02-19T21:19:00.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2abfff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:00.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830382855 0 0) 0x7ff2c406cbb0 con 0x7ff2c40fd980 2024-02-19T21:19:00.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2c406cf10 con 0x7ff2c40fd980 2024-02-19T21:19:00.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660770960 0 0) 0x7ff2c406cd40 con 0x7ff2c41042c0 2024-02-19T21:19:00.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.526+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2c406cbb0 con 0x7ff2c41042c0 2024-02-19T21:19:00.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 715967000 0 0) 0x7ff2c406cf10 con 0x7ff2c40fd980 2024-02-19T21:19:00.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2c406cd40 con 0x7ff2c40fd980 2024-02-19T21:19:00.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3602504928 0 0) 0x7ff2ac0038e0 con 0x7ff2c4100780 2024-02-19T21:19:00.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2c406cf10 con 0x7ff2c4100780 2024-02-19T21:19:00.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff2b8002c90 con 0x7ff2c40fd980 2024-02-19T21:19:00.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff2b4001860 con 0x7ff2c4100780 2024-02-19T21:19:00.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3192770203 0 0) 0x7ff2c406cd40 con 0x7ff2c40fd980 2024-02-19T21:19:00.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 >> v1:172.21.15.163:6790/0 conn(0x7ff2c4100780 legacy=0x7ff2c4114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:00.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 >> v1:172.21.15.193:6789/0 conn(0x7ff2c41042c0 legacy=0x7ff2c4118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:00.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c411cb90 con 0x7ff2c40fd980 2024-02-19T21:19:00.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2c98d7700 1 -- 172.21.15.163:0/683067062 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c4119bc0 con 0x7ff2c40fd980 2024-02-19T21:19:00.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2c98d7700 10 client.24060.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:00.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2c98d7700 1 -- 172.21.15.163:0/683067062 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c411a030 con 0x7ff2c40fd980 2024-02-19T21:19:00.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2c98d7700 1 librados: init done 2024-02-19T21:19:00.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff2b80022f0 con 0x7ff2c40fd980 2024-02-19T21:19:00.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.527+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff2b80036e0 con 0x7ff2c40fd980 2024-02-19T21:19:00.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.528+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ff2b8003840 con 0x7ff2c40fd980 2024-02-19T21:19:00.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.528+0000 7ff2c98d7700 1 -- 172.21.15.163:0/683067062 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b0004f20 con 0x7ff2c40fd980 2024-02-19T21:19:00.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.529+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(844..844 src has 784..844) v4 ==== 7419+0+0 (unknown 1150271998 0 0) 0x7ff2b8078cf0 con 0x7ff2c40fd980 2024-02-19T21:19:00.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.529+0000 7ff2abfff700 10 client.24060.objecter ms_dispatch 0x7ff2c4001550 osd_map(844..844 src has 784..844) v4 2024-02-19T21:19:00.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.529+0000 7ff2abfff700 3 client.24060.objecter handle_osd_map got epochs [844,844] > 0 2024-02-19T21:19:00.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.529+0000 7ff2abfff700 3 client.24060.objecter handle_osd_map decoding full epoch 844 2024-02-19T21:19:00.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.529+0000 7ff2abfff700 20 client.24060.objecter dump_active .. 0 homeless 2024-02-19T21:19:00.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.529+0000 7ff2abfff700 10 client.24060.objecter ms_handle_connect 0x7ff2ac0639a0 2024-02-19T21:19:00.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.533+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7ff2b804c510 con 0x7ff2c40fd980 2024-02-19T21:19:00.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:00.713+0000 7ff2c98d7700 1 -- 172.21.15.163:0/683067062 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} v 0) v1 -- 0x7ff2b00059e0 con 0x7ff2c40fd980 2024-02-19T21:19:00.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:59.363504+0000 mgr.y (mgr.14550) 708 : cluster [DBG] pgmap v1294: 174 pgs: 174 active+clean; 5.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:19:00.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:59.422829+0000 osd.2 (osd.2) 362 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:19:00.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:59.423011+0000 osd.2 (osd.2) 363 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:19:00.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:59.441567+0000 osd.4 (osd.4) 325 : cluster [DBG] 16.2 scrub starts 2024-02-19T21:19:00.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:59.442793+0000 osd.4 (osd.4) 326 : cluster [DBG] 16.2 scrub ok 2024-02-19T21:19:01.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:18:59.672663+0000 mon.a (mon.0) 3857 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2024-02-19T21:19:01.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:59.363504+0000 mgr.y (mgr.14550) 708 : cluster [DBG] pgmap v1294: 174 pgs: 174 active+clean; 5.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:19:01.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:59.422829+0000 osd.2 (osd.2) 362 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:19:01.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:59.423011+0000 osd.2 (osd.2) 363 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:19:01.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:59.441567+0000 osd.4 (osd.4) 325 : cluster [DBG] 16.2 scrub starts 2024-02-19T21:19:01.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:59.442793+0000 osd.4 (osd.4) 326 : cluster [DBG] 16.2 scrub ok 2024-02-19T21:19:01.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:18:59.672663+0000 mon.a (mon.0) 3857 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2024-02-19T21:19:01.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:19:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:19:01.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:59.363504+0000 mgr.y (mgr.14550) 708 : cluster [DBG] pgmap v1294: 174 pgs: 174 active+clean; 5.2 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:19:01.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:59.422829+0000 osd.2 (osd.2) 362 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:19:01.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:59.423011+0000 osd.2 (osd.2) 363 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:19:01.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:59.441567+0000 osd.4 (osd.4) 325 : cluster [DBG] 16.2 scrub starts 2024-02-19T21:19:01.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:59.442793+0000 osd.4 (osd.4) 326 : cluster [DBG] 16.2 scrub ok 2024-02-19T21:19:01.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:18:59.672663+0000 mon.a (mon.0) 3857 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2024-02-19T21:19:01.676 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:01.674+0000 7ff2abfff700 1 -- 172.21.15.163:0/683067062 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]=0 set osd.6 primary-affinity to 1 (8655362) v845) v1 ==== 133+0+0 (unknown 321819561 0 0) 0x7ff2b804a980 con 0x7ff2c40fd980 2024-02-19T21:19:01.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:01.677+0000 7ff2c98d7700 10 librados: watch_flush enter 2024-02-19T21:19:01.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:01.677+0000 7ff2c98d7700 10 librados: watch_flush exit 2024-02-19T21:19:01.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:01.677+0000 7ff2c98d7700 20 client.24060.objecter shutdown clearing up homeless session... 2024-02-19T21:19:01.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:01.677+0000 7ff2c98d7700 10 client.24060.objecter successfully canceled tick 2024-02-19T21:19:01.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:01.677+0000 7ff2c98d7700 1 -- 172.21.15.163:0/683067062 >> v1:172.21.15.163:6800/1552514681 conn(0x7ff2ac0639a0 legacy=0x7ff2ac065e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:01.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:01.677+0000 7ff2c98d7700 1 -- 172.21.15.163:0/683067062 >> v1:172.21.15.163:6789/0 conn(0x7ff2c40fd980 legacy=0x7ff2c40fc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:01.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:01.677+0000 7ff2c98d7700 1 -- 172.21.15.163:0/683067062 shutdown_connections 2024-02-19T21:19:01.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:01.677+0000 7ff2c98d7700 1 -- 172.21.15.163:0/683067062 >> 172.21.15.163:0/683067062 conn(0x7ff2c40f35a0 msgr2=0x7ff2c40f5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:01.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:01.678+0000 7ff2c98d7700 1 -- 172.21.15.163:0/683067062 shutdown_connections 2024-02-19T21:19:01.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:01.678+0000 7ff2c98d7700 1 -- 172.21.15.163:0/683067062 wait complete. 2024-02-19T21:19:01.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:01.678+0000 7ff2c98d7700 1 librados: shutdown 2024-02-19T21:19:01.681 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-02-19T21:19:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:00.331863+0000 osd.0 (osd.0) 162 : cluster [DBG] 16.f deep-scrub starts 2024-02-19T21:19:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:00.334602+0000 osd.0 (osd.0) 163 : cluster [DBG] 16.f deep-scrub ok 2024-02-19T21:19:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:00.568843+0000 osd.6 (osd.6) 323 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:19:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:00.571758+0000 osd.6 (osd.6) 324 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:19:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:00.714643+0000 mon.a (mon.0) 3858 : audit [INF] from='client.? v1:172.21.15.163:0/683067062' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:19:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:00.898887+0000 mgr.y (mgr.14550) 709 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:01.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:01.394553+0000 osd.1 (osd.1) 403 : cluster [DBG] 16.b scrub starts 2024-02-19T21:19:02.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:01.398536+0000 osd.1 (osd.1) 404 : cluster [DBG] 16.b scrub ok 2024-02-19T21:19:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:00.331863+0000 osd.0 (osd.0) 162 : cluster [DBG] 16.f deep-scrub starts 2024-02-19T21:19:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:00.334602+0000 osd.0 (osd.0) 163 : cluster [DBG] 16.f deep-scrub ok 2024-02-19T21:19:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:00.568843+0000 osd.6 (osd.6) 323 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:19:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:00.571758+0000 osd.6 (osd.6) 324 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:19:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:00.714643+0000 mon.a (mon.0) 3858 : audit [INF] from='client.? v1:172.21.15.163:0/683067062' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:19:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:00.898887+0000 mgr.y (mgr.14550) 709 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:01.394553+0000 osd.1 (osd.1) 403 : cluster [DBG] 16.b scrub starts 2024-02-19T21:19:02.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:01.398536+0000 osd.1 (osd.1) 404 : cluster [DBG] 16.b scrub ok 2024-02-19T21:19:02.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:00.331863+0000 osd.0 (osd.0) 162 : cluster [DBG] 16.f deep-scrub starts 2024-02-19T21:19:02.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:00.334602+0000 osd.0 (osd.0) 163 : cluster [DBG] 16.f deep-scrub ok 2024-02-19T21:19:02.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:00.568843+0000 osd.6 (osd.6) 323 : cluster [DBG] 2.11 scrub starts 2024-02-19T21:19:02.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:00.571758+0000 osd.6 (osd.6) 324 : cluster [DBG] 2.11 scrub ok 2024-02-19T21:19:02.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:00.714643+0000 mon.a (mon.0) 3858 : audit [INF] from='client.? v1:172.21.15.163:0/683067062' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-19T21:19:02.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:00.898887+0000 mgr.y (mgr.14550) 709 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:02.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:01.394553+0000 osd.1 (osd.1) 403 : cluster [DBG] 16.b scrub starts 2024-02-19T21:19:02.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:01.398536+0000 osd.1 (osd.1) 404 : cluster [DBG] 16.b scrub ok 2024-02-19T21:19:02.433 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd reweight 7 1 2024-02-19T21:19:02.690 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:01.352035+0000 osd.0 (osd.0) 164 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:19:02.690 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:01.353424+0000 osd.0 (osd.0) 165 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:19:02.690 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:01.364523+0000 mgr.y (mgr.14550) 710 : cluster [DBG] pgmap v1296: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-02-19T21:19:02.690 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:01.486473+0000 osd.4 (osd.4) 327 : cluster [DBG] 4.c scrub starts 2024-02-19T21:19:02.690 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:01.488081+0000 osd.4 (osd.4) 328 : cluster [DBG] 4.c scrub ok 2024-02-19T21:19:02.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:01.675472+0000 mon.a (mon.0) 3859 : audit [INF] from='client.? v1:172.21.15.163:0/683067062' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-19T21:19:02.691 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:01.675551+0000 mon.a (mon.0) 3860 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2024-02-19T21:19:02.716 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:02.938 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:01.352035+0000 osd.0 (osd.0) 164 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:19:02.939 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:01.353424+0000 osd.0 (osd.0) 165 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:19:02.939 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:01.364523+0000 mgr.y (mgr.14550) 710 : cluster [DBG] pgmap v1296: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-02-19T21:19:02.939 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:01.486473+0000 osd.4 (osd.4) 327 : cluster [DBG] 4.c scrub starts 2024-02-19T21:19:02.939 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:01.488081+0000 osd.4 (osd.4) 328 : cluster [DBG] 4.c scrub ok 2024-02-19T21:19:02.939 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:01.675472+0000 mon.a (mon.0) 3859 : audit [INF] from='client.? v1:172.21.15.163:0/683067062' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-19T21:19:02.939 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:01.675551+0000 mon.a (mon.0) 3860 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2024-02-19T21:19:03.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:01.352035+0000 osd.0 (osd.0) 164 : cluster [DBG] 5.1e scrub starts 2024-02-19T21:19:03.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:01.353424+0000 osd.0 (osd.0) 165 : cluster [DBG] 5.1e scrub ok 2024-02-19T21:19:03.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:01.364523+0000 mgr.y (mgr.14550) 710 : cluster [DBG] pgmap v1296: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-02-19T21:19:03.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:01.486473+0000 osd.4 (osd.4) 327 : cluster [DBG] 4.c scrub starts 2024-02-19T21:19:03.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:01.488081+0000 osd.4 (osd.4) 328 : cluster [DBG] 4.c scrub ok 2024-02-19T21:19:03.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:01.675472+0000 mon.a (mon.0) 3859 : audit [INF] from='client.? v1:172.21.15.163:0/683067062' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-19T21:19:03.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:01.675551+0000 mon.a (mon.0) 3860 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2024-02-19T21:19:03.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:02.683716+0000 mon.a (mon.0) 3861 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2024-02-19T21:19:03.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:03.246010+0000 osd.3 (osd.3) 387 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:19:03.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:03.247430+0000 osd.3 (osd.3) 388 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:19:03.749 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:19:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:19:03] "GET /metrics HTTP/1.1" 200 187971 "" "Prometheus/2.33.4" 2024-02-19T21:19:03.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:02.683716+0000 mon.a (mon.0) 3861 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2024-02-19T21:19:03.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:03.246010+0000 osd.3 (osd.3) 387 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:19:03.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:03.247430+0000 osd.3 (osd.3) 388 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:19:04.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:02.683716+0000 mon.a (mon.0) 3861 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2024-02-19T21:19:04.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:03.246010+0000 osd.3 (osd.3) 387 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:19:04.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:03.247430+0000 osd.3 (osd.3) 388 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:19:04.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.555+0000 7fd7680fa700 1 -- 172.21.15.163:0/4240343970 >> v1:172.21.15.163:6790/0 conn(0x7fd760100780 legacy=0x7fd760102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:04.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.556+0000 7fd7680fa700 1 -- 172.21.15.163:0/4240343970 shutdown_connections 2024-02-19T21:19:04.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.556+0000 7fd7680fa700 1 -- 172.21.15.163:0/4240343970 >> 172.21.15.163:0/4240343970 conn(0x7fd7600f35a0 msgr2=0x7fd7600f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:04.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.556+0000 7fd7680fa700 1 -- 172.21.15.163:0/4240343970 shutdown_connections 2024-02-19T21:19:04.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.556+0000 7fd7680fa700 1 -- 172.21.15.163:0/4240343970 wait complete. 2024-02-19T21:19:04.560 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.556+0000 7fd7680fa700 1 librados: starting msgr at 2024-02-19T21:19:04.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.556+0000 7fd7680fa700 1 librados: starting objecter 2024-02-19T21:19:04.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.556+0000 7fd7680fa700 1 Processor -- start 2024-02-19T21:19:04.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.556+0000 7fd7680fa700 1 -- start start 2024-02-19T21:19:04.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.556+0000 7fd7680fa700 1 librados: setting wanted keys 2024-02-19T21:19:04.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.556+0000 7fd7680fa700 1 librados: calling monclient init 2024-02-19T21:19:04.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.556+0000 7fd7680fa700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd760117160 con 0x7fd7601042c0 2024-02-19T21:19:04.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd7680fa700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd760118340 con 0x7fd760100780 2024-02-19T21:19:04.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd7680fa700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd760119520 con 0x7fd7600fd980 2024-02-19T21:19:04.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd765e96700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fd7600fd980 0x7fd7600f9710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:46174/0 (socket says 172.21.15.163:46174) 2024-02-19T21:19:04.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd765e96700 1 -- 172.21.15.163:0/3392876327 learned_addr learned my addr 172.21.15.163:0/3392876327 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:04.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 10 client.?.objecter ms_handle_connect 0x7fd7600fd980 2024-02-19T21:19:04.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:04.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 10 client.?.objecter ms_handle_connect 0x7fd7601042c0 2024-02-19T21:19:04.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:04.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 10 client.?.objecter ms_handle_connect 0x7fd760100780 2024-02-19T21:19:04.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:04.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463625957 0 0) 0x7fd760117160 con 0x7fd7601042c0 2024-02-19T21:19:04.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd74c003030 con 0x7fd7601042c0 2024-02-19T21:19:04.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1767919517 0 0) 0x7fd760119520 con 0x7fd7600fd980 2024-02-19T21:19:04.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd760117160 con 0x7fd7600fd980 2024-02-19T21:19:04.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215209585 0 0) 0x7fd760118340 con 0x7fd760100780 2024-02-19T21:19:04.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd760119520 con 0x7fd760100780 2024-02-19T21:19:04.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1369326648 0 0) 0x7fd74c003030 con 0x7fd7601042c0 2024-02-19T21:19:04.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd760118340 con 0x7fd7601042c0 2024-02-19T21:19:04.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd758001bc0 con 0x7fd7601042c0 2024-02-19T21:19:04.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4140707207 0 0) 0x7fd760117160 con 0x7fd7600fd980 2024-02-19T21:19:04.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd74c003030 con 0x7fd7600fd980 2024-02-19T21:19:04.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2778897022 0 0) 0x7fd760119520 con 0x7fd760100780 2024-02-19T21:19:04.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd760117160 con 0x7fd760100780 2024-02-19T21:19:04.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd75c0019d0 con 0x7fd7600fd980 2024-02-19T21:19:04.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd750000fe0 con 0x7fd760100780 2024-02-19T21:19:04.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2943459865 0 0) 0x7fd760118340 con 0x7fd7601042c0 2024-02-19T21:19:04.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 >> v1:172.21.15.163:6790/0 conn(0x7fd7600fd980 legacy=0x7fd7600f9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:04.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 >> v1:172.21.15.193:6789/0 conn(0x7fd760100780 legacy=0x7fd7600f9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:04.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76011a720 con 0x7fd7601042c0 2024-02-19T21:19:04.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd7680fa700 1 -- 172.21.15.163:0/3392876327 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd760117350 con 0x7fd7601042c0 2024-02-19T21:19:04.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd7680fa700 10 client.24066.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:04.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd7680fa700 1 -- 172.21.15.163:0/3392876327 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd760117840 con 0x7fd7601042c0 2024-02-19T21:19:04.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.557+0000 7fd7680fa700 1 librados: init done 2024-02-19T21:19:04.563 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.558+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fd758002280 con 0x7fd7601042c0 2024-02-19T21:19:04.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.558+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd758003620 con 0x7fd7601042c0 2024-02-19T21:19:04.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.558+0000 7fd7680fa700 1 -- 172.21.15.163:0/3392876327 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd744004f20 con 0x7fd7601042c0 2024-02-19T21:19:04.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.559+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fd758001de0 con 0x7fd7601042c0 2024-02-19T21:19:04.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.559+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(846..846 src has 796..846) v4 ==== 7419+0+0 (unknown 2503803538 0 0) 0x7fd758076490 con 0x7fd7601042c0 2024-02-19T21:19:04.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.559+0000 7fd756ffd700 10 client.24066.objecter ms_dispatch 0x7fd760001550 osd_map(846..846 src has 796..846) v4 2024-02-19T21:19:04.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.559+0000 7fd756ffd700 3 client.24066.objecter handle_osd_map got epochs [846,846] > 0 2024-02-19T21:19:04.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.559+0000 7fd756ffd700 3 client.24066.objecter handle_osd_map decoding full epoch 846 2024-02-19T21:19:04.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.559+0000 7fd756ffd700 20 client.24066.objecter dump_active .. 0 homeless 2024-02-19T21:19:04.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.559+0000 7fd756ffd700 10 client.24066.objecter ms_handle_connect 0x7fd74c063da0 2024-02-19T21:19:04.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.563+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd758049b60 con 0x7fd7601042c0 2024-02-19T21:19:04.693 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:03.364902+0000 mgr.y (mgr.14550) 711 : cluster [DBG] pgmap v1299: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:19:04.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:03.501913+0000 osd.4 (osd.4) 329 : cluster [DBG] 6.e deep-scrub starts 2024-02-19T21:19:04.694 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:03.502931+0000 osd.4 (osd.4) 330 : cluster [DBG] 6.e deep-scrub ok 2024-02-19T21:19:04.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:04.746+0000 7fd7680fa700 1 -- 172.21.15.163:0/3392876327 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) v1 -- 0x7fd7440051d0 con 0x7fd7601042c0 2024-02-19T21:19:04.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:03.364902+0000 mgr.y (mgr.14550) 711 : cluster [DBG] pgmap v1299: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:19:04.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:03.501913+0000 osd.4 (osd.4) 329 : cluster [DBG] 6.e deep-scrub starts 2024-02-19T21:19:04.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:03.502931+0000 osd.4 (osd.4) 330 : cluster [DBG] 6.e deep-scrub ok 2024-02-19T21:19:05.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:03.364902+0000 mgr.y (mgr.14550) 711 : cluster [DBG] pgmap v1299: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:19:05.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:03.501913+0000 osd.4 (osd.4) 329 : cluster [DBG] 6.e deep-scrub starts 2024-02-19T21:19:05.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:03.502931+0000 osd.4 (osd.4) 330 : cluster [DBG] 6.e deep-scrub ok 2024-02-19T21:19:05.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:05.705+0000 7fd756ffd700 1 -- 172.21.15.163:0/3392876327 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v847) v1 ==== 113+0+0 (unknown 3335243421 0 0) 0x7fd758047f70 con 0x7fd7601042c0 2024-02-19T21:19:05.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:05.708+0000 7fd7680fa700 10 librados: watch_flush enter 2024-02-19T21:19:05.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:05.709+0000 7fd7680fa700 10 librados: watch_flush exit 2024-02-19T21:19:05.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:05.709+0000 7fd7680fa700 20 client.24066.objecter shutdown clearing up homeless session... 2024-02-19T21:19:05.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:05.709+0000 7fd7680fa700 10 client.24066.objecter successfully canceled tick 2024-02-19T21:19:05.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:05.709+0000 7fd7680fa700 1 -- 172.21.15.163:0/3392876327 >> v1:172.21.15.163:6800/1552514681 conn(0x7fd74c063da0 legacy=0x7fd74c066260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:05.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:05.709+0000 7fd7680fa700 1 -- 172.21.15.163:0/3392876327 >> v1:172.21.15.163:6789/0 conn(0x7fd7601042c0 legacy=0x7fd760115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:05.711 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:05.709+0000 7fd7680fa700 1 -- 172.21.15.163:0/3392876327 shutdown_connections 2024-02-19T21:19:05.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:05.709+0000 7fd7680fa700 1 -- 172.21.15.163:0/3392876327 >> 172.21.15.163:0/3392876327 conn(0x7fd7600f35a0 msgr2=0x7fd760107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:05.712 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:05.711+0000 7fd7680fa700 1 -- 172.21.15.163:0/3392876327 shutdown_connections 2024-02-19T21:19:05.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:05.711+0000 7fd7680fa700 1 -- 172.21.15.163:0/3392876327 wait complete. 2024-02-19T21:19:05.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:05.711+0000 7fd7680fa700 1 librados: shutdown 2024-02-19T21:19:05.713 INFO:teuthology.orchestra.run.smithi163.stderr:reweighted osd.7 to 1 (10000) 2024-02-19T21:19:05.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:04.550732+0000 osd.4 (osd.4) 331 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:19:05.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:04.552595+0000 osd.4 (osd.4) 332 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:19:05.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:04.748348+0000 mon.a (mon.0) 3862 : audit [INF] from='client.? v1:172.21.15.163:0/3392876327' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:19:06.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:05.234704+0000 osd.3 (osd.3) 389 : cluster [DBG] 2.1b deep-scrub starts 2024-02-19T21:19:06.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:05.235847+0000 osd.3 (osd.3) 390 : cluster [DBG] 2.1b deep-scrub ok 2024-02-19T21:19:06.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:04.550732+0000 osd.4 (osd.4) 331 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:19:06.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:04.552595+0000 osd.4 (osd.4) 332 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:19:06.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:04.748348+0000 mon.a (mon.0) 3862 : audit [INF] from='client.? v1:172.21.15.163:0/3392876327' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:19:06.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:05.234704+0000 osd.3 (osd.3) 389 : cluster [DBG] 2.1b deep-scrub starts 2024-02-19T21:19:06.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:05.235847+0000 osd.3 (osd.3) 390 : cluster [DBG] 2.1b deep-scrub ok 2024-02-19T21:19:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:04.550732+0000 osd.4 (osd.4) 331 : cluster [DBG] 4.1 scrub starts 2024-02-19T21:19:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:04.552595+0000 osd.4 (osd.4) 332 : cluster [DBG] 4.1 scrub ok 2024-02-19T21:19:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:04.748348+0000 mon.a (mon.0) 3862 : audit [INF] from='client.? v1:172.21.15.163:0/3392876327' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:19:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:05.234704+0000 osd.3 (osd.3) 389 : cluster [DBG] 2.1b deep-scrub starts 2024-02-19T21:19:06.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:05.235847+0000 osd.3 (osd.3) 390 : cluster [DBG] 2.1b deep-scrub ok 2024-02-19T21:19:06.447 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd primary-affinity 7 1 2024-02-19T21:19:06.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:05.309633+0000 osd.0 (osd.0) 166 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:19:06.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:05.311489+0000 osd.0 (osd.0) 167 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:19:06.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:05.365746+0000 mgr.y (mgr.14550) 712 : cluster [DBG] pgmap v1300: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:19:06.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:05.706515+0000 mon.a (mon.0) 3863 : audit [INF] from='client.? v1:172.21.15.163:0/3392876327' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-19T21:19:06.706 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:05.706591+0000 mon.a (mon.0) 3864 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2024-02-19T21:19:06.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:05.941288+0000 osd.2 (osd.2) 364 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:19:06.707 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:05.942406+0000 osd.2 (osd.2) 365 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:19:06.707 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:05.309633+0000 osd.0 (osd.0) 166 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:19:06.707 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:05.311489+0000 osd.0 (osd.0) 167 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:19:06.707 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:05.365746+0000 mgr.y (mgr.14550) 712 : cluster [DBG] pgmap v1300: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:19:06.707 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:05.706515+0000 mon.a (mon.0) 3863 : audit [INF] from='client.? v1:172.21.15.163:0/3392876327' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-19T21:19:06.707 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:05.706591+0000 mon.a (mon.0) 3864 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2024-02-19T21:19:06.707 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:05.941288+0000 osd.2 (osd.2) 364 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:19:06.707 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:05.942406+0000 osd.2 (osd.2) 365 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:19:06.721 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:07.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:05.309633+0000 osd.0 (osd.0) 166 : cluster [DBG] 4.16 scrub starts 2024-02-19T21:19:07.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:05.311489+0000 osd.0 (osd.0) 167 : cluster [DBG] 4.16 scrub ok 2024-02-19T21:19:07.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:05.365746+0000 mgr.y (mgr.14550) 712 : cluster [DBG] pgmap v1300: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-19T21:19:07.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:05.706515+0000 mon.a (mon.0) 3863 : audit [INF] from='client.? v1:172.21.15.163:0/3392876327' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-19T21:19:07.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:05.706591+0000 mon.a (mon.0) 3864 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2024-02-19T21:19:07.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:05.941288+0000 osd.2 (osd.2) 364 : cluster [DBG] 5.4 scrub starts 2024-02-19T21:19:07.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:05.942406+0000 osd.2 (osd.2) 365 : cluster [DBG] 5.4 scrub ok 2024-02-19T21:19:07.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:07.541852+0000 osd.5 (osd.5) 358 : cluster [DBG] 6.1c deep-scrub starts 2024-02-19T21:19:07.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:07.545447+0000 osd.5 (osd.5) 359 : cluster [DBG] 6.1c deep-scrub ok 2024-02-19T21:19:08.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:07.541852+0000 osd.5 (osd.5) 358 : cluster [DBG] 6.1c deep-scrub starts 2024-02-19T21:19:08.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:07.545447+0000 osd.5 (osd.5) 359 : cluster [DBG] 6.1c deep-scrub ok 2024-02-19T21:19:08.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:19:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:19:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:19:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:07.541852+0000 osd.5 (osd.5) 358 : cluster [DBG] 6.1c deep-scrub starts 2024-02-19T21:19:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:07.545447+0000 osd.5 (osd.5) 359 : cluster [DBG] 6.1c deep-scrub ok 2024-02-19T21:19:08.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.550+0000 7f41f38bf700 1 -- 172.21.15.163:0/3550895005 >> v1:172.21.15.163:6789/0 conn(0x7f41ec100780 legacy=0x7f41ec102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:08.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.550+0000 7f41f38bf700 1 -- 172.21.15.163:0/3550895005 shutdown_connections 2024-02-19T21:19:08.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.550+0000 7f41f38bf700 1 -- 172.21.15.163:0/3550895005 >> 172.21.15.163:0/3550895005 conn(0x7f41ec0f35a0 msgr2=0x7f41ec0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:08.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.550+0000 7f41f38bf700 1 -- 172.21.15.163:0/3550895005 shutdown_connections 2024-02-19T21:19:08.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.550+0000 7f41f38bf700 1 -- 172.21.15.163:0/3550895005 wait complete. 2024-02-19T21:19:08.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.550+0000 7f41f38bf700 1 librados: starting msgr at 2024-02-19T21:19:08.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.550+0000 7f41f38bf700 1 librados: starting objecter 2024-02-19T21:19:08.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41f38bf700 1 Processor -- start 2024-02-19T21:19:08.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41f38bf700 1 -- start start 2024-02-19T21:19:08.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41f38bf700 1 librados: setting wanted keys 2024-02-19T21:19:08.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41f38bf700 1 librados: calling monclient init 2024-02-19T21:19:08.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41f38bf700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f41ec117240 con 0x7f41ec1042c0 2024-02-19T21:19:08.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41f38bf700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f41ec118420 con 0x7f41ec100780 2024-02-19T21:19:08.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41f38bf700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f41ec119620 con 0x7f41ec0fd980 2024-02-19T21:19:08.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41f1e5c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f41ec1042c0 0x7f41ec115940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:59314/0 (socket says 172.21.15.163:59314) 2024-02-19T21:19:08.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41f1e5c700 1 -- 172.21.15.163:0/2767927213 learned_addr learned my addr 172.21.15.163:0/2767927213 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:08.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41f165b700 1 --1- 172.21.15.163:0/2767927213 >> v1:172.21.15.163:6790/0 conn(0x7f41ec0fd980 0x7f41ec0f98e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:46188/0 (socket says 172.21.15.163:46188) 2024-02-19T21:19:08.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41e27fc700 10 client.?.objecter ms_handle_connect 0x7f41ec1042c0 2024-02-19T21:19:08.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41e27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:08.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41e27fc700 10 client.?.objecter ms_handle_connect 0x7f41ec0fd980 2024-02-19T21:19:08.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41e27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:08.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41e27fc700 10 client.?.objecter ms_handle_connect 0x7f41ec100780 2024-02-19T21:19:08.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41e27fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:08.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706567562 0 0) 0x7f41ec117240 con 0x7f41ec1042c0 2024-02-19T21:19:08.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41d8003030 con 0x7f41ec1042c0 2024-02-19T21:19:08.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664686370 0 0) 0x7f41ec119620 con 0x7f41ec0fd980 2024-02-19T21:19:08.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ec117240 con 0x7f41ec0fd980 2024-02-19T21:19:08.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014583689 0 0) 0x7f41ec118420 con 0x7f41ec100780 2024-02-19T21:19:08.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ec119620 con 0x7f41ec100780 2024-02-19T21:19:08.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.551+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2872664805 0 0) 0x7f41d8003030 con 0x7f41ec1042c0 2024-02-19T21:19:08.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41ec118420 con 0x7f41ec1042c0 2024-02-19T21:19:08.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f41e4001d70 con 0x7f41ec1042c0 2024-02-19T21:19:08.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 492458367 0 0) 0x7f41ec119620 con 0x7f41ec100780 2024-02-19T21:19:08.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41d8003030 con 0x7f41ec100780 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3298494704 0 0) 0x7f41ec117240 con 0x7f41ec0fd980 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41ec119620 con 0x7f41ec0fd980 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f41e8002070 con 0x7f41ec100780 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f41dc001840 con 0x7f41ec0fd980 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3821323743 0 0) 0x7f41ec118420 con 0x7f41ec1042c0 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 >> v1:172.21.15.163:6790/0 conn(0x7f41ec0fd980 legacy=0x7f41ec0f98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 >> v1:172.21.15.193:6789/0 conn(0x7f41ec100780 legacy=0x7f41ec0f9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ec11a820 con 0x7f41ec1042c0 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41f38bf700 1 -- 172.21.15.163:0/2767927213 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ec119810 con 0x7f41ec1042c0 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41f38bf700 10 client.24072.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41f38bf700 1 -- 172.21.15.163:0/2767927213 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ec119d00 con 0x7f41ec1042c0 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41f38bf700 1 librados: init done 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f41e4002890 con 0x7f41ec1042c0 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.552+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f41e4005ef0 con 0x7f41ec1042c0 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.553+0000 7f41f38bf700 1 -- 172.21.15.163:0/2767927213 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d0004f20 con 0x7f41ec1042c0 2024-02-19T21:19:08.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.553+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f41e40023f0 con 0x7f41ec1042c0 2024-02-19T21:19:08.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.554+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(847..847 src has 796..847) v4 ==== 7419+0+0 (unknown 1522572164 0 0) 0x7f41e4078dc0 con 0x7f41ec1042c0 2024-02-19T21:19:08.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.554+0000 7f41e27fc700 10 client.24072.objecter ms_dispatch 0x7f41ec001550 osd_map(847..847 src has 796..847) v4 2024-02-19T21:19:08.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.554+0000 7f41e27fc700 3 client.24072.objecter handle_osd_map got epochs [847,847] > 0 2024-02-19T21:19:08.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.554+0000 7f41e27fc700 3 client.24072.objecter handle_osd_map decoding full epoch 847 2024-02-19T21:19:08.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.554+0000 7f41e27fc700 20 client.24072.objecter dump_active .. 0 homeless 2024-02-19T21:19:08.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.554+0000 7f41e27fc700 10 client.24072.objecter ms_handle_connect 0x7f41d8063c70 2024-02-19T21:19:08.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.558+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f41e404c490 con 0x7f41ec1042c0 2024-02-19T21:19:08.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:07.366158+0000 mgr.y (mgr.14550) 713 : cluster [DBG] pgmap v1302: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:08.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:07.511731+0000 osd.6 (osd.6) 325 : cluster [DBG] 2.14 deep-scrub starts 2024-02-19T21:19:08.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:07.512849+0000 osd.6 (osd.6) 326 : cluster [DBG] 2.14 deep-scrub ok 2024-02-19T21:19:08.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:07.533233+0000 osd.4 (osd.4) 333 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:19:08.712 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:07.534177+0000 osd.4 (osd.4) 334 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:19:08.736 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:08.734+0000 7f41f38bf700 1 -- 172.21.15.163:0/2767927213 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) v1 -- 0x7f41d00059e0 con 0x7f41ec1042c0 2024-02-19T21:19:08.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:07.366158+0000 mgr.y (mgr.14550) 713 : cluster [DBG] pgmap v1302: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:08.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:07.511731+0000 osd.6 (osd.6) 325 : cluster [DBG] 2.14 deep-scrub starts 2024-02-19T21:19:08.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:07.512849+0000 osd.6 (osd.6) 326 : cluster [DBG] 2.14 deep-scrub ok 2024-02-19T21:19:08.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:07.533233+0000 osd.4 (osd.4) 333 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:19:08.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:07.534177+0000 osd.4 (osd.4) 334 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:19:09.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:07.366158+0000 mgr.y (mgr.14550) 713 : cluster [DBG] pgmap v1302: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:09.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:07.511731+0000 osd.6 (osd.6) 325 : cluster [DBG] 2.14 deep-scrub starts 2024-02-19T21:19:09.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:07.512849+0000 osd.6 (osd.6) 326 : cluster [DBG] 2.14 deep-scrub ok 2024-02-19T21:19:09.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:07.533233+0000 osd.4 (osd.4) 333 : cluster [DBG] 6.1e scrub starts 2024-02-19T21:19:09.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:07.534177+0000 osd.4 (osd.4) 334 : cluster [DBG] 6.1e scrub ok 2024-02-19T21:19:09.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:09.713+0000 7f41e27fc700 1 -- 172.21.15.163:0/2767927213 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v848) v1 ==== 133+0+0 (unknown 2521654915 0 0) 0x7f41e404a8a0 con 0x7f41ec1042c0 2024-02-19T21:19:09.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:09.716+0000 7f41f38bf700 10 librados: watch_flush enter 2024-02-19T21:19:09.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:09.716+0000 7f41f38bf700 10 librados: watch_flush exit 2024-02-19T21:19:09.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:09.716+0000 7f41f38bf700 20 client.24072.objecter shutdown clearing up homeless session... 2024-02-19T21:19:09.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:09.716+0000 7f41f38bf700 10 client.24072.objecter successfully canceled tick 2024-02-19T21:19:09.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:09.716+0000 7f41f38bf700 1 -- 172.21.15.163:0/2767927213 >> v1:172.21.15.163:6800/1552514681 conn(0x7f41d8063c70 legacy=0x7f41d8066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:09.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:09.716+0000 7f41f38bf700 1 -- 172.21.15.163:0/2767927213 >> v1:172.21.15.163:6789/0 conn(0x7f41ec1042c0 legacy=0x7f41ec115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:09.718 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:09.716+0000 7f41f38bf700 1 -- 172.21.15.163:0/2767927213 shutdown_connections 2024-02-19T21:19:09.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:09.716+0000 7f41f38bf700 1 -- 172.21.15.163:0/2767927213 >> 172.21.15.163:0/2767927213 conn(0x7f41ec0f35a0 msgr2=0x7f41ec107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:09.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:09.716+0000 7f41f38bf700 1 -- 172.21.15.163:0/2767927213 shutdown_connections 2024-02-19T21:19:09.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:09.716+0000 7f41f38bf700 1 -- 172.21.15.163:0/2767927213 wait complete. 2024-02-19T21:19:09.719 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:09.716+0000 7f41f38bf700 1 librados: shutdown 2024-02-19T21:19:09.723 INFO:teuthology.orchestra.run.smithi163.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-02-19T21:19:09.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:08.736270+0000 mon.a (mon.0) 3865 : audit [INF] from='client.? v1:172.21.15.163:0/2767927213' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:19:09.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:09.544901+0000 osd.5 (osd.5) 360 : cluster [DBG] 2.20 scrub starts 2024-02-19T21:19:09.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:09.548375+0000 osd.5 (osd.5) 361 : cluster [DBG] 2.20 scrub ok 2024-02-19T21:19:09.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:08.736270+0000 mon.a (mon.0) 3865 : audit [INF] from='client.? v1:172.21.15.163:0/2767927213' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:19:09.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:09.544901+0000 osd.5 (osd.5) 360 : cluster [DBG] 2.20 scrub starts 2024-02-19T21:19:10.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:09.548375+0000 osd.5 (osd.5) 361 : cluster [DBG] 2.20 scrub ok 2024-02-19T21:19:10.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:08.736270+0000 mon.a (mon.0) 3865 : audit [INF] from='client.? v1:172.21.15.163:0/2767927213' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-19T21:19:10.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:09.544901+0000 osd.5 (osd.5) 360 : cluster [DBG] 2.20 scrub starts 2024-02-19T21:19:10.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:09.548375+0000 osd.5 (osd.5) 361 : cluster [DBG] 2.20 scrub ok 2024-02-19T21:19:10.496 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-02-19T21:19:10.496 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-02-19T21:19:10.496 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-02-19T21:19:10.496 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-02-19T21:19:10.496 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-19T21:19:10.496 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-19T21:19:10.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:09.366553+0000 mgr.y (mgr.14550) 714 : cluster [DBG] pgmap v1303: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 665 B/s rd, 0 op/s 2024-02-19T21:19:10.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:09.484722+0000 osd.6 (osd.6) 327 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:19:10.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:09.486904+0000 osd.6 (osd.6) 328 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:19:10.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:09.713877+0000 mon.a (mon.0) 3866 : audit [INF] from='client.? v1:172.21.15.163:0/2767927213' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-19T21:19:10.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:09.713936+0000 mon.a (mon.0) 3867 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2024-02-19T21:19:10.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:10.212362+0000 osd.3 (osd.3) 391 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:19:10.749 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:10.213667+0000 osd.3 (osd.3) 392 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:19:10.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:10.569656+0000 osd.5 (osd.5) 362 : cluster [DBG] 16.1b scrub starts 2024-02-19T21:19:10.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:10.573460+0000 osd.5 (osd.5) 363 : cluster [DBG] 16.1b scrub ok 2024-02-19T21:19:10.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:09.366553+0000 mgr.y (mgr.14550) 714 : cluster [DBG] pgmap v1303: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 665 B/s rd, 0 op/s 2024-02-19T21:19:10.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:09.484722+0000 osd.6 (osd.6) 327 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:19:10.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:09.486904+0000 osd.6 (osd.6) 328 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:19:10.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:09.713877+0000 mon.a (mon.0) 3866 : audit [INF] from='client.? v1:172.21.15.163:0/2767927213' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-19T21:19:10.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:09.713936+0000 mon.a (mon.0) 3867 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2024-02-19T21:19:10.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:10.212362+0000 osd.3 (osd.3) 391 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:19:10.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:10.213667+0000 osd.3 (osd.3) 392 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:19:10.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:10.569656+0000 osd.5 (osd.5) 362 : cluster [DBG] 16.1b scrub starts 2024-02-19T21:19:10.750 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:10.573460+0000 osd.5 (osd.5) 363 : cluster [DBG] 16.1b scrub ok 2024-02-19T21:19:10.780 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:11.072 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:19:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:19:11.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:09.366553+0000 mgr.y (mgr.14550) 714 : cluster [DBG] pgmap v1303: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 665 B/s rd, 0 op/s 2024-02-19T21:19:11.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:09.484722+0000 osd.6 (osd.6) 327 : cluster [DBG] 4.5 scrub starts 2024-02-19T21:19:11.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:09.486904+0000 osd.6 (osd.6) 328 : cluster [DBG] 4.5 scrub ok 2024-02-19T21:19:11.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:09.713877+0000 mon.a (mon.0) 3866 : audit [INF] from='client.? v1:172.21.15.163:0/2767927213' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-19T21:19:11.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:09.713936+0000 mon.a (mon.0) 3867 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2024-02-19T21:19:11.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:10.212362+0000 osd.3 (osd.3) 391 : cluster [DBG] 6.13 scrub starts 2024-02-19T21:19:11.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:10.213667+0000 osd.3 (osd.3) 392 : cluster [DBG] 6.13 scrub ok 2024-02-19T21:19:11.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:10.569656+0000 osd.5 (osd.5) 362 : cluster [DBG] 16.1b scrub starts 2024-02-19T21:19:11.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:10.573460+0000 osd.5 (osd.5) 363 : cluster [DBG] 16.1b scrub ok 2024-02-19T21:19:11.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:10.263672+0000 osd.0 (osd.0) 168 : cluster [DBG] 6.9 deep-scrub starts 2024-02-19T21:19:11.969 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:10.266662+0000 osd.0 (osd.0) 169 : cluster [DBG] 6.9 deep-scrub ok 2024-02-19T21:19:11.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:10.499211+0000 osd.4 (osd.4) 335 : cluster [DBG] 2.25 scrub starts 2024-02-19T21:19:11.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:10.500211+0000 osd.4 (osd.4) 336 : cluster [DBG] 2.25 scrub ok 2024-02-19T21:19:11.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:10.727778+0000 mon.a (mon.0) 3868 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2024-02-19T21:19:11.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:10.902452+0000 mgr.y (mgr.14550) 715 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:11.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:10.982840+0000 osd.2 (osd.2) 366 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:19:11.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:10.983945+0000 osd.2 (osd.2) 367 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:19:11.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:11.191020+0000 osd.3 (osd.3) 393 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:19:11.970 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:11.193030+0000 osd.3 (osd.3) 394 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:19:11.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:10.263672+0000 osd.0 (osd.0) 168 : cluster [DBG] 6.9 deep-scrub starts 2024-02-19T21:19:11.970 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:10.266662+0000 osd.0 (osd.0) 169 : cluster [DBG] 6.9 deep-scrub ok 2024-02-19T21:19:11.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:10.499211+0000 osd.4 (osd.4) 335 : cluster [DBG] 2.25 scrub starts 2024-02-19T21:19:11.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:10.500211+0000 osd.4 (osd.4) 336 : cluster [DBG] 2.25 scrub ok 2024-02-19T21:19:11.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:10.727778+0000 mon.a (mon.0) 3868 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2024-02-19T21:19:11.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:10.902452+0000 mgr.y (mgr.14550) 715 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:11.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:10.982840+0000 osd.2 (osd.2) 366 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:19:11.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:10.983945+0000 osd.2 (osd.2) 367 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:19:11.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:11.191020+0000 osd.3 (osd.3) 393 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:19:11.972 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:11.193030+0000 osd.3 (osd.3) 394 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:19:12.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:10.263672+0000 osd.0 (osd.0) 168 : cluster [DBG] 6.9 deep-scrub starts 2024-02-19T21:19:12.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:10.266662+0000 osd.0 (osd.0) 169 : cluster [DBG] 6.9 deep-scrub ok 2024-02-19T21:19:12.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:10.499211+0000 osd.4 (osd.4) 335 : cluster [DBG] 2.25 scrub starts 2024-02-19T21:19:12.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:10.500211+0000 osd.4 (osd.4) 336 : cluster [DBG] 2.25 scrub ok 2024-02-19T21:19:12.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:10.727778+0000 mon.a (mon.0) 3868 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2024-02-19T21:19:12.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:10.902452+0000 mgr.y (mgr.14550) 715 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:12.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:10.982840+0000 osd.2 (osd.2) 366 : cluster [DBG] 6.14 scrub starts 2024-02-19T21:19:12.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:10.983945+0000 osd.2 (osd.2) 367 : cluster [DBG] 6.14 scrub ok 2024-02-19T21:19:12.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:11.191020+0000 osd.3 (osd.3) 393 : cluster [DBG] 4.0 scrub starts 2024-02-19T21:19:12.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:11.193030+0000 osd.3 (osd.3) 394 : cluster [DBG] 4.0 scrub ok 2024-02-19T21:19:12.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.421+0000 7f6a5388e700 1 -- 172.21.15.163:0/1831277511 >> v1:172.21.15.163:6790/0 conn(0x7f6a4c0ff670 legacy=0x7f6a4c101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:12.434 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.421+0000 7f6a5388e700 1 -- 172.21.15.163:0/1831277511 shutdown_connections 2024-02-19T21:19:12.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.421+0000 7f6a5388e700 1 -- 172.21.15.163:0/1831277511 >> 172.21.15.163:0/1831277511 conn(0x7f6a4c0f2470 msgr2=0x7f6a4c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:12.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.421+0000 7f6a5388e700 1 -- 172.21.15.163:0/1831277511 shutdown_connections 2024-02-19T21:19:12.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.421+0000 7f6a5388e700 1 -- 172.21.15.163:0/1831277511 wait complete. 2024-02-19T21:19:12.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.421+0000 7f6a5388e700 1 librados: starting msgr at 2024-02-19T21:19:12.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.421+0000 7f6a5388e700 1 librados: starting objecter 2024-02-19T21:19:12.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.421+0000 7f6a5388e700 1 Processor -- start 2024-02-19T21:19:12.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.421+0000 7f6a5388e700 1 -- start start 2024-02-19T21:19:12.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.421+0000 7f6a5388e700 1 librados: setting wanted keys 2024-02-19T21:19:12.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.421+0000 7f6a5388e700 1 librados: calling monclient init 2024-02-19T21:19:12.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a5388e700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6a4c1171a0 con 0x7f6a4c1031b0 2024-02-19T21:19:12.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a5388e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6a4c118380 con 0x7f6a4c0fc870 2024-02-19T21:19:12.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a5388e700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6a4c119580 con 0x7f6a4c0ff670 2024-02-19T21:19:12.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a50e29700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f6a4c0ff670 0x7f6a4c0612f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:46202/0 (socket says 172.21.15.163:46202) 2024-02-19T21:19:12.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a50e29700 1 -- 172.21.15.163:0/2612334112 learned_addr learned my addr 172.21.15.163:0/2612334112 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:12.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 10 client.?.objecter ms_handle_connect 0x7f6a4c1031b0 2024-02-19T21:19:12.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:12.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 10 client.?.objecter ms_handle_connect 0x7f6a4c0ff670 2024-02-19T21:19:12.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:12.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 10 client.?.objecter ms_handle_connect 0x7f6a4c0fc870 2024-02-19T21:19:12.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:12.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212384564 0 0) 0x7f6a4c1171a0 con 0x7f6a4c1031b0 2024-02-19T21:19:12.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a38003030 con 0x7f6a4c1031b0 2024-02-19T21:19:12.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297918220 0 0) 0x7f6a4c119580 con 0x7f6a4c0ff670 2024-02-19T21:19:12.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a4c1171a0 con 0x7f6a4c0ff670 2024-02-19T21:19:12.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846373316 0 0) 0x7f6a4c118380 con 0x7f6a4c0fc870 2024-02-19T21:19:12.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a4c119580 con 0x7f6a4c0fc870 2024-02-19T21:19:12.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 690723733 0 0) 0x7f6a38003030 con 0x7f6a4c1031b0 2024-02-19T21:19:12.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a4c118380 con 0x7f6a4c1031b0 2024-02-19T21:19:12.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6a44001bc0 con 0x7f6a4c1031b0 2024-02-19T21:19:12.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2762371652 0 0) 0x7f6a4c1171a0 con 0x7f6a4c0ff670 2024-02-19T21:19:12.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a38003030 con 0x7f6a4c0ff670 2024-02-19T21:19:12.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6a3c002070 con 0x7f6a4c0ff670 2024-02-19T21:19:12.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2691126465 0 0) 0x7f6a4c119580 con 0x7f6a4c0fc870 2024-02-19T21:19:12.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a4c1171a0 con 0x7f6a4c0fc870 2024-02-19T21:19:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6a480015a0 con 0x7f6a4c0fc870 2024-02-19T21:19:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1087714382 0 0) 0x7f6a4c118380 con 0x7f6a4c1031b0 2024-02-19T21:19:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 >> v1:172.21.15.163:6790/0 conn(0x7f6a4c0ff670 legacy=0x7f6a4c0612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 >> v1:172.21.15.193:6789/0 conn(0x7f6a4c0fc870 legacy=0x7f6a4c060be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4c11a780 con 0x7f6a4c1031b0 2024-02-19T21:19:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a5388e700 1 -- 172.21.15.163:0/2612334112 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a4c119770 con 0x7f6a4c1031b0 2024-02-19T21:19:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a5388e700 10 client.24078.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a5388e700 1 -- 172.21.15.163:0/2612334112 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a4c119c60 con 0x7f6a4c1031b0 2024-02-19T21:19:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.422+0000 7f6a5388e700 1 librados: init done 2024-02-19T21:19:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.423+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6a44002280 con 0x7f6a4c1031b0 2024-02-19T21:19:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.423+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6a44003620 con 0x7f6a4c1031b0 2024-02-19T21:19:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.423+0000 7f6a5388e700 1 -- 172.21.15.163:0/2612334112 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a30004f20 con 0x7f6a4c1031b0 2024-02-19T21:19:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.424+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6a44001de0 con 0x7f6a4c1031b0 2024-02-19T21:19:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.424+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(849..849 src has 796..849) v4 ==== 7419+0+0 (unknown 1436230568 0 0) 0x7f6a440764f0 con 0x7f6a4c1031b0 2024-02-19T21:19:12.439 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.424+0000 7f6a427fc700 10 client.24078.objecter ms_dispatch 0x7f6a4c001550 osd_map(849..849 src has 796..849) v4 2024-02-19T21:19:12.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.424+0000 7f6a427fc700 3 client.24078.objecter handle_osd_map got epochs [849,849] > 0 2024-02-19T21:19:12.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.424+0000 7f6a427fc700 3 client.24078.objecter handle_osd_map decoding full epoch 849 2024-02-19T21:19:12.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.424+0000 7f6a427fc700 20 client.24078.objecter dump_active .. 0 homeless 2024-02-19T21:19:12.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.424+0000 7f6a427fc700 10 client.24078.objecter ms_handle_connect 0x7f6a38063d10 2024-02-19T21:19:12.440 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.429+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6a44049b60 con 0x7f6a4c1031b0 2024-02-19T21:19:12.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.601+0000 7f6a5388e700 1 -- 172.21.15.163:0/2612334112 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a300051d0 con 0x7f6a4c1031b0 2024-02-19T21:19:12.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.602+0000 7f6a427fc700 1 -- 172.21.15.163:0/2612334112 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v849) v1 ==== 74+0+22115 (unknown 1979137330 0 176545836) 0x7f6a44047f70 con 0x7f6a4c1031b0 2024-02-19T21:19:12.605 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:19:12.605 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":849,"fsid":"2591809a-cf69-11ee-95bb-87774f69a715","created":"2024-02-19T20:55:37.794885+0000","modified":"2024-02-19T21:19:10.718277+0000","last_up_change":"2024-02-19T21:01:09.214134+0000","last_in_change":"2024-02-19T21:18:02.692983+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":109,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":256,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-19T20:59:09.425169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"597","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-19T21:01:14.658945+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":42,"pg_placement_num":42,"pg_placement_num_target":42,"pg_num_target":42,"pg_num_pending":42,"last_pg_merge_meta":{"source_pgid":"2.20","ready_epoch":309,"last_epoch_started":308,"last_epoch_clean":308,"source_version":"58'1","target_version":"58'1"},"last_change":"819","last_force_op_resend":"0","last_force_op_resend_prenautilus":"309","last_force_op_resend_preluminous":"474","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-19T21:01:14.958800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-19T21:01:16.543816+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-19T21:01:18.521482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-19T21:01:20.586564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"805","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-02-19T21:04:35.333192+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4a14b406-7c21-40f3-8b12-8cf9df811088","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":833,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6801","nonce":1559418115}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6802","nonce":1559418115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6804","nonce":1559418115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6803","nonce":1559418115}]},"public_addr":"172.21.15.163:6801/1559418115","cluster_addr":"172.21.15.163:6802/1559418115","heartbeat_back_addr":"172.21.15.163:6804/1559418115","heartbeat_front_addr":"172.21.15.163:6803/1559418115","state":["exists","up"]},{"osd":1,"uuid":"e74a65fa-cb9c-42c3-9293-602cf7e9dc67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":833,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6805","nonce":3125047432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6806","nonce":3125047432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6808","nonce":3125047432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6807","nonce":3125047432}]},"public_addr":"172.21.15.163:6805/3125047432","cluster_addr":"172.21.15.163:6806/3125047432","heartbeat_back_addr":"172.21.15.163:6808/3125047432","heartbeat_front_addr":"172.21.15.163:6807/3125047432","state":["exists","up"]},{"osd":2,"uuid":"6980a625-cdc9-487e-8f8c-7997ba716a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":833,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6809","nonce":1563158487}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6810","nonce":1563158487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6812","nonce":1563158487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6811","nonce":1563158487}]},"public_addr":"172.21.15.163:6809/1563158487","cluster_addr":"172.21.15.163:6810/1563158487","heartbeat_back_addr":"172.21.15.163:6812/1563158487","heartbeat_front_addr":"172.21.15.163:6811/1563158487","state":["exists","up"]},{"osd":3,"uuid":"359e2ee0-9d1d-4074-8119-2d69bc2ca3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":814,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6813","nonce":2112279769}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6814","nonce":2112279769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6816","nonce":2112279769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6815","nonce":2112279769}]},"public_addr":"172.21.15.163:6813/2112279769","cluster_addr":"172.21.15.163:6814/2112279769","heartbeat_back_addr":"172.21.15.163:6816/2112279769","heartbeat_front_addr":"172.21.15.163:6815/2112279769","state":["exists","up"]},{"osd":4,"uuid":"2bf5ef05-292b-4693-85a9-97991f2091e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":814,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6800","nonce":3129716965}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6801","nonce":3129716965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6803","nonce":3129716965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6802","nonce":3129716965}]},"public_addr":"172.21.15.193:6800/3129716965","cluster_addr":"172.21.15.193:6801/3129716965","heartbeat_back_addr":"172.21.15.193:6803/3129716965","heartbeat_front_addr":"172.21.15.193:6802/3129716965","state":["exists","up"]},{"osd":5,"uuid":"59065f9b-9240-414b-9c91-e91cda4aed6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":833,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6804","nonce":805161991}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6805","nonce":805161991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6807","nonce":805161991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6806","nonce":805161991}]},"public_addr":"172.21.15.193:6804/805161991","cluster_addr":"172.21.15.193:6805/805161991","heartbeat_back_addr":"172.21.15.193:6807/805161991","heartbeat_front_addr":"172.21.15.193:6806/805161991","state":["exists","up"]},{"osd":6,"uuid":"302c95df-3842-4818-8afd-d79d8184684f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":845,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6808","nonce":787604615}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6809","nonce":787604615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6811","nonce":787604615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6810","nonce":787604615}]},"public_addr":"172.21.15.193:6808/787604615","cluster_addr":"172.21.15.193:6809/787604615","heartbeat_back_addr":"172.21.15.193:6811/787604615","heartbeat_front_addr":"172.21.15.193:6810/787604615","state":["exists","up"]},{"osd":7,"uuid":"9ebeff27-c2f8-4a66-923f-a0d7573ddd3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":848,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6812","nonce":157927432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6813","nonce":157927432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6815","nonce":157927432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6814","nonce":157927432}]},"public_addr":"172.21.15.193:6812/157927432","cluster_addr":"172.21.15.193:6813/157927432","heartbeat_back_addr":"172.21.15.193:6815/157927432","heartbeat_front_addr":"172.21.15.193:6814/157927432","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:43.017101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:23.902515+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:59.423013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:13.753710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:38.714651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:21.619833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:57.661959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:55.030177+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":7,"to":2}]},{"pgid":"2.7","mappings":[{"from":7,"to":1}]},{"pgid":"2.9","mappings":[{"from":7,"to":2}]},{"pgid":"2.1a","mappings":[{"from":7,"to":2}]},{"pgid":"2.1d","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:0/1427774891":"2024-02-20T21:01:38.896227+0000","172.21.15.163:0/3282102947":"2024-02-20T21:01:38.896227+0000","172.21.15.163:6800/2735092549":"2024-02-20T21:01:38.896227+0000","172.21.15.163:0/1235588846":"2024-02-20T20:56:29.603030+0000","172.21.15.163:6800/2731225015":"2024-02-20T20:56:29.603030+0000","172.21.15.163:0/506491651":"2024-02-20T20:56:02.185409+0000","172.21.15.163:0/168117826":"2024-02-20T21:01:38.896227+0000","172.21.15.163:0/1458009465":"2024-02-20T20:56:29.603030+0000","172.21.15.163:0/541631472":"2024-02-20T20:56:02.185409+0000","172.21.15.163:6800/2248999541":"2024-02-20T20:56:02.185409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":25,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":57,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":76,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":85,"snaps":[{"begin":2,"length":1}]},{"pool":92,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":94,"snaps":[{"begin":4,"length":4}]},{"pool":107,"snaps":[{"begin":2,"length":1}]},{"pool":112,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":126,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":132,"snaps":[{"begin":12,"length":1}]},{"pool":154,"snaps":[{"begin":14,"length":1}]},{"pool":164,"snaps":[{"begin":17,"length":1}]},{"pool":178,"snaps":[{"begin":20,"length":1}]},{"pool":237,"snaps":[{"begin":2,"length":1}]},{"pool":238,"snaps":[{"begin":4,"length":1}]},{"pool":241,"snaps":[{"begin":6,"length":1}]},{"pool":245,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T21:19:12.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.604+0000 7f6a5388e700 10 librados: watch_flush enter 2024-02-19T21:19:12.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.604+0000 7f6a5388e700 10 librados: watch_flush exit 2024-02-19T21:19:12.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.604+0000 7f6a5388e700 20 client.24078.objecter shutdown clearing up homeless session... 2024-02-19T21:19:12.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.604+0000 7f6a5388e700 10 client.24078.objecter successfully canceled tick 2024-02-19T21:19:12.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.604+0000 7f6a5388e700 1 -- 172.21.15.163:0/2612334112 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6a38063d10 legacy=0x7f6a380661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:12.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.604+0000 7f6a5388e700 1 -- 172.21.15.163:0/2612334112 >> v1:172.21.15.163:6789/0 conn(0x7f6a4c1031b0 legacy=0x7f6a4c115860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:12.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.604+0000 7f6a5388e700 1 -- 172.21.15.163:0/2612334112 shutdown_connections 2024-02-19T21:19:12.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.604+0000 7f6a5388e700 1 -- 172.21.15.163:0/2612334112 >> 172.21.15.163:0/2612334112 conn(0x7f6a4c0f2470 msgr2=0x7f6a4c106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:12.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.604+0000 7f6a5388e700 1 -- 172.21.15.163:0/2612334112 shutdown_connections 2024-02-19T21:19:12.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.604+0000 7f6a5388e700 1 -- 172.21.15.163:0/2612334112 wait complete. 2024-02-19T21:19:12.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:12.605+0000 7f6a5388e700 1 librados: shutdown 2024-02-19T21:19:12.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:11.367140+0000 mgr.y (mgr.14550) 716 : cluster [DBG] pgmap v1306: 174 pgs: 19 peering, 155 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:12.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:11.433836+0000 osd.6 (osd.6) 329 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:19:12.987 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:11.435010+0000 osd.6 (osd.6) 330 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:19:12.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:11.521136+0000 osd.4 (osd.4) 337 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:19:12.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:11.522939+0000 osd.4 (osd.4) 338 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:19:12.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:11.724561+0000 mon.a (mon.0) 3869 : cluster [WRN] Health check failed: Reduced data availability: 17 pgs inactive, 19 pgs peering (PG_AVAILABILITY) 2024-02-19T21:19:12.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:12.573931+0000 osd.5 (osd.5) 364 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:19:12.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:12.576889+0000 osd.5 (osd.5) 365 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:19:12.988 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:12.603405+0000 mon.a (mon.0) 3870 : audit [DBG] from='client.? v1:172.21.15.163:0/2612334112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T21:19:12.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:11.367140+0000 mgr.y (mgr.14550) 716 : cluster [DBG] pgmap v1306: 174 pgs: 19 peering, 155 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:12.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:11.433836+0000 osd.6 (osd.6) 329 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:19:12.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:11.435010+0000 osd.6 (osd.6) 330 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:19:12.988 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:11.521136+0000 osd.4 (osd.4) 337 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:19:12.989 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:11.522939+0000 osd.4 (osd.4) 338 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:19:12.989 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:11.724561+0000 mon.a (mon.0) 3869 : cluster [WRN] Health check failed: Reduced data availability: 17 pgs inactive, 19 pgs peering (PG_AVAILABILITY) 2024-02-19T21:19:12.989 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:12.573931+0000 osd.5 (osd.5) 364 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:19:12.989 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:12.576889+0000 osd.5 (osd.5) 365 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:19:12.989 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:12.603405+0000 mon.a (mon.0) 3870 : audit [DBG] from='client.? v1:172.21.15.163:0/2612334112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T21:19:13.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:11.367140+0000 mgr.y (mgr.14550) 716 : cluster [DBG] pgmap v1306: 174 pgs: 19 peering, 155 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:13.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:11.433836+0000 osd.6 (osd.6) 329 : cluster [DBG] 5.1f scrub starts 2024-02-19T21:19:13.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:11.435010+0000 osd.6 (osd.6) 330 : cluster [DBG] 5.1f scrub ok 2024-02-19T21:19:13.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:11.521136+0000 osd.4 (osd.4) 337 : cluster [DBG] 4.18 scrub starts 2024-02-19T21:19:13.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:11.522939+0000 osd.4 (osd.4) 338 : cluster [DBG] 4.18 scrub ok 2024-02-19T21:19:13.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:11.724561+0000 mon.a (mon.0) 3869 : cluster [WRN] Health check failed: Reduced data availability: 17 pgs inactive, 19 pgs peering (PG_AVAILABILITY) 2024-02-19T21:19:13.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:12.573931+0000 osd.5 (osd.5) 364 : cluster [DBG] 5.7 scrub starts 2024-02-19T21:19:13.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:12.576889+0000 osd.5 (osd.5) 365 : cluster [DBG] 5.7 scrub ok 2024-02-19T21:19:13.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:12.603405+0000 mon.a (mon.0) 3870 : audit [DBG] from='client.? v1:172.21.15.163:0/2612334112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T21:19:13.339 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-19T21:19:13.339 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-19T21:19:13.613 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:13.741 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:19:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:19:13] "GET /metrics HTTP/1.1" 200 187978 "" "Prometheus/2.33.4" 2024-02-19T21:19:13.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:12.313710+0000 osd.0 (osd.0) 170 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:19:13.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:12.315241+0000 osd.0 (osd.0) 171 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:19:13.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:13.151318+0000 osd.3 (osd.3) 395 : cluster [DBG] 2.1e deep-scrub starts 2024-02-19T21:19:13.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:13.152230+0000 osd.3 (osd.3) 396 : cluster [DBG] 2.1e deep-scrub ok 2024-02-19T21:19:14.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:12.313710+0000 osd.0 (osd.0) 170 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:19:14.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:12.315241+0000 osd.0 (osd.0) 171 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:19:14.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:13.151318+0000 osd.3 (osd.3) 395 : cluster [DBG] 2.1e deep-scrub starts 2024-02-19T21:19:14.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:13.152230+0000 osd.3 (osd.3) 396 : cluster [DBG] 2.1e deep-scrub ok 2024-02-19T21:19:14.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:12.313710+0000 osd.0 (osd.0) 170 : cluster [DBG] 6.16 scrub starts 2024-02-19T21:19:14.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:12.315241+0000 osd.0 (osd.0) 171 : cluster [DBG] 6.16 scrub ok 2024-02-19T21:19:14.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:13.151318+0000 osd.3 (osd.3) 395 : cluster [DBG] 2.1e deep-scrub starts 2024-02-19T21:19:14.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:13.152230+0000 osd.3 (osd.3) 396 : cluster [DBG] 2.1e deep-scrub ok 2024-02-19T21:19:14.876 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:13.367599+0000 mgr.y (mgr.14550) 717 : cluster [DBG] pgmap v1307: 174 pgs: 19 peering, 155 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 668 B/s rd, 0 op/s 2024-02-19T21:19:14.876 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:13.595721+0000 osd.7 (osd.7) 341 : cluster [DBG] 6.12 deep-scrub starts 2024-02-19T21:19:14.876 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:13.598695+0000 osd.7 (osd.7) 342 : cluster [DBG] 6.12 deep-scrub ok 2024-02-19T21:19:14.876 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:13.915421+0000 osd.2 (osd.2) 368 : cluster [DBG] 2.e deep-scrub starts 2024-02-19T21:19:14.876 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:13.916655+0000 osd.2 (osd.2) 369 : cluster [DBG] 2.e deep-scrub ok 2024-02-19T21:19:14.877 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:13.367599+0000 mgr.y (mgr.14550) 717 : cluster [DBG] pgmap v1307: 174 pgs: 19 peering, 155 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 668 B/s rd, 0 op/s 2024-02-19T21:19:14.877 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:13.595721+0000 osd.7 (osd.7) 341 : cluster [DBG] 6.12 deep-scrub starts 2024-02-19T21:19:14.877 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:13.598695+0000 osd.7 (osd.7) 342 : cluster [DBG] 6.12 deep-scrub ok 2024-02-19T21:19:14.877 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:13.915421+0000 osd.2 (osd.2) 368 : cluster [DBG] 2.e deep-scrub starts 2024-02-19T21:19:14.877 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:14 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:13.916655+0000 osd.2 (osd.2) 369 : cluster [DBG] 2.e deep-scrub ok 2024-02-19T21:19:15.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:13.367599+0000 mgr.y (mgr.14550) 717 : cluster [DBG] pgmap v1307: 174 pgs: 19 peering, 155 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 668 B/s rd, 0 op/s 2024-02-19T21:19:15.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:13.595721+0000 osd.7 (osd.7) 341 : cluster [DBG] 6.12 deep-scrub starts 2024-02-19T21:19:15.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:13.598695+0000 osd.7 (osd.7) 342 : cluster [DBG] 6.12 deep-scrub ok 2024-02-19T21:19:15.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:13.915421+0000 osd.2 (osd.2) 368 : cluster [DBG] 2.e deep-scrub starts 2024-02-19T21:19:15.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:13.916655+0000 osd.2 (osd.2) 369 : cluster [DBG] 2.e deep-scrub ok 2024-02-19T21:19:15.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.370+0000 7f9eefb91700 1 -- 172.21.15.163:0/1359960816 >> v1:172.21.15.163:6790/0 conn(0x7f9ee8100780 legacy=0x7f9ee8102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:15.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.370+0000 7f9eefb91700 1 -- 172.21.15.163:0/1359960816 shutdown_connections 2024-02-19T21:19:15.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.370+0000 7f9eefb91700 1 -- 172.21.15.163:0/1359960816 >> 172.21.15.163:0/1359960816 conn(0x7f9ee80f35a0 msgr2=0x7f9ee80f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:15.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.370+0000 7f9eefb91700 1 -- 172.21.15.163:0/1359960816 shutdown_connections 2024-02-19T21:19:15.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.370+0000 7f9eefb91700 1 -- 172.21.15.163:0/1359960816 wait complete. 2024-02-19T21:19:15.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.371+0000 7f9eefb91700 1 librados: starting msgr at 2024-02-19T21:19:15.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.371+0000 7f9eefb91700 1 librados: starting objecter 2024-02-19T21:19:15.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.371+0000 7f9eefb91700 1 Processor -- start 2024-02-19T21:19:15.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.371+0000 7f9eefb91700 1 -- start start 2024-02-19T21:19:15.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.371+0000 7f9eefb91700 1 librados: setting wanted keys 2024-02-19T21:19:15.375 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.371+0000 7f9eefb91700 1 librados: calling monclient init 2024-02-19T21:19:15.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.371+0000 7f9eefb91700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9ee8117330 con 0x7f9ee8100780 2024-02-19T21:19:15.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.371+0000 7f9eefb91700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9ee8118510 con 0x7f9ee80fd980 2024-02-19T21:19:15.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.371+0000 7f9eefb91700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9ee8119710 con 0x7f9ee81042c0 2024-02-19T21:19:15.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9eee12e700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f9ee81042c0 0x7f9ee81159f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:41994/0 (socket says 172.21.15.163:41994) 2024-02-19T21:19:15.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9eee12e700 1 -- 172.21.15.163:0/851159755 learned_addr learned my addr 172.21.15.163:0/851159755 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:15.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9eed92d700 1 --1- 172.21.15.163:0/851159755 >> v1:172.21.15.193:6789/0 conn(0x7f9ee80fd980 0x7f9ee80f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.163:34246/0 (socket says 172.21.15.163:34246) 2024-02-19T21:19:15.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 10 client.?.objecter ms_handle_connect 0x7f9ee81042c0 2024-02-19T21:19:15.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:15.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 10 client.?.objecter ms_handle_connect 0x7f9ee80fd980 2024-02-19T21:19:15.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:15.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856476409 0 0) 0x7f9ee8119710 con 0x7f9ee81042c0 2024-02-19T21:19:15.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ed0003030 con 0x7f9ee81042c0 2024-02-19T21:19:15.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 10 client.?.objecter ms_handle_connect 0x7f9ee8100780 2024-02-19T21:19:15.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:15.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702571317 0 0) 0x7f9ee8118510 con 0x7f9ee80fd980 2024-02-19T21:19:15.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee8119710 con 0x7f9ee80fd980 2024-02-19T21:19:15.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602518834 0 0) 0x7f9ee8117330 con 0x7f9ee8100780 2024-02-19T21:19:15.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee8118510 con 0x7f9ee8100780 2024-02-19T21:19:15.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2975691659 0 0) 0x7f9ed0003030 con 0x7f9ee81042c0 2024-02-19T21:19:15.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ee8117330 con 0x7f9ee81042c0 2024-02-19T21:19:15.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9ed8001900 con 0x7f9ee81042c0 2024-02-19T21:19:15.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3455193097 0 0) 0x7f9ee8119710 con 0x7f9ee80fd980 2024-02-19T21:19:15.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ed0003030 con 0x7f9ee80fd980 2024-02-19T21:19:15.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9ee0001e00 con 0x7f9ee80fd980 2024-02-19T21:19:15.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3190220029 0 0) 0x7f9ee8118510 con 0x7f9ee8100780 2024-02-19T21:19:15.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ee8119710 con 0x7f9ee8100780 2024-02-19T21:19:15.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9edc0020f0 con 0x7f9ee8100780 2024-02-19T21:19:15.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3698326339 0 0) 0x7f9ee8117330 con 0x7f9ee81042c0 2024-02-19T21:19:15.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 >> v1:172.21.15.193:6789/0 conn(0x7f9ee80fd980 legacy=0x7f9ee80f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:15.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 >> v1:172.21.15.163:6789/0 conn(0x7f9ee8100780 legacy=0x7f9ee80fa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:15.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee811a910 con 0x7f9ee81042c0 2024-02-19T21:19:15.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9eefb91700 1 -- 172.21.15.163:0/851159755 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee8118700 con 0x7f9ee81042c0 2024-02-19T21:19:15.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9eefb91700 10 client.50810.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:15.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9eefb91700 1 -- 172.21.15.163:0/851159755 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee8118b70 con 0x7f9ee81042c0 2024-02-19T21:19:15.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.372+0000 7f9eefb91700 1 librados: init done 2024-02-19T21:19:15.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.373+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9ed8001c90 con 0x7f9ee81042c0 2024-02-19T21:19:15.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.373+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9ed8005b50 con 0x7f9ee81042c0 2024-02-19T21:19:15.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.374+0000 7f9eefb91700 1 -- 172.21.15.163:0/851159755 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed4004f20 con 0x7f9ee81042c0 2024-02-19T21:19:15.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.374+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9ed80028b0 con 0x7f9ee81042c0 2024-02-19T21:19:15.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.374+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(849..849 src has 796..849) v4 ==== 7419+0+0 (unknown 1436230568 0 0) 0x7f9ed80789f0 con 0x7f9ee81042c0 2024-02-19T21:19:15.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.374+0000 7f9ee6ffd700 10 client.50810.objecter ms_dispatch 0x7f9ee8001550 osd_map(849..849 src has 796..849) v4 2024-02-19T21:19:15.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.374+0000 7f9ee6ffd700 3 client.50810.objecter handle_osd_map got epochs [849,849] > 0 2024-02-19T21:19:15.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.374+0000 7f9ee6ffd700 3 client.50810.objecter handle_osd_map decoding full epoch 849 2024-02-19T21:19:15.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.374+0000 7f9ee6ffd700 20 client.50810.objecter dump_active .. 0 homeless 2024-02-19T21:19:15.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.375+0000 7f9ee6ffd700 10 client.50810.objecter ms_handle_connect 0x7f9ed0063c70 2024-02-19T21:19:15.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.378+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f9ed804c0c0 con 0x7f9ee81042c0 2024-02-19T21:19:15.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.542+0000 7f9eefb91700 1 -- 172.21.15.163:0/851159755 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ed40051d0 con 0x7f9ee81042c0 2024-02-19T21:19:15.545 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.543+0000 7f9ee6ffd700 1 -- 172.21.15.163:0/851159755 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v849) v1 ==== 74+0+22115 (unknown 1979137330 0 176545836) 0x7f9ed804a4d0 con 0x7f9ee81042c0 2024-02-19T21:19:15.546 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:19:15.546 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":849,"fsid":"2591809a-cf69-11ee-95bb-87774f69a715","created":"2024-02-19T20:55:37.794885+0000","modified":"2024-02-19T21:19:10.718277+0000","last_up_change":"2024-02-19T21:01:09.214134+0000","last_in_change":"2024-02-19T21:18:02.692983+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":109,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":256,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-02-19T20:59:09.425169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"597","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-19T21:01:14.658945+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":42,"pg_placement_num":42,"pg_placement_num_target":42,"pg_num_target":42,"pg_num_pending":42,"last_pg_merge_meta":{"source_pgid":"2.20","ready_epoch":309,"last_epoch_started":308,"last_epoch_clean":308,"source_version":"58'1","target_version":"58'1"},"last_change":"819","last_force_op_resend":"0","last_force_op_resend_prenautilus":"309","last_force_op_resend_preluminous":"474","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"iscsi","create_time":"2024-02-19T21:01:14.958800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-19T21:01:16.543816+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-19T21:01:18.521482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-19T21:01:20.586564+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"805","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-02-19T21:04:35.333192+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4a14b406-7c21-40f3-8b12-8cf9df811088","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":833,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6801","nonce":1559418115}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6802","nonce":1559418115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6804","nonce":1559418115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6803","nonce":1559418115}]},"public_addr":"172.21.15.163:6801/1559418115","cluster_addr":"172.21.15.163:6802/1559418115","heartbeat_back_addr":"172.21.15.163:6804/1559418115","heartbeat_front_addr":"172.21.15.163:6803/1559418115","state":["exists","up"]},{"osd":1,"uuid":"e74a65fa-cb9c-42c3-9293-602cf7e9dc67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":833,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6805","nonce":3125047432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6806","nonce":3125047432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6808","nonce":3125047432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6807","nonce":3125047432}]},"public_addr":"172.21.15.163:6805/3125047432","cluster_addr":"172.21.15.163:6806/3125047432","heartbeat_back_addr":"172.21.15.163:6808/3125047432","heartbeat_front_addr":"172.21.15.163:6807/3125047432","state":["exists","up"]},{"osd":2,"uuid":"6980a625-cdc9-487e-8f8c-7997ba716a97","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":833,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6809","nonce":1563158487}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6810","nonce":1563158487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6812","nonce":1563158487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6811","nonce":1563158487}]},"public_addr":"172.21.15.163:6809/1563158487","cluster_addr":"172.21.15.163:6810/1563158487","heartbeat_back_addr":"172.21.15.163:6812/1563158487","heartbeat_front_addr":"172.21.15.163:6811/1563158487","state":["exists","up"]},{"osd":3,"uuid":"359e2ee0-9d1d-4074-8119-2d69bc2ca3a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":814,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6813","nonce":2112279769}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6814","nonce":2112279769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6816","nonce":2112279769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.163:6815","nonce":2112279769}]},"public_addr":"172.21.15.163:6813/2112279769","cluster_addr":"172.21.15.163:6814/2112279769","heartbeat_back_addr":"172.21.15.163:6816/2112279769","heartbeat_front_addr":"172.21.15.163:6815/2112279769","state":["exists","up"]},{"osd":4,"uuid":"2bf5ef05-292b-4693-85a9-97991f2091e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":814,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6800","nonce":3129716965}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6801","nonce":3129716965}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6803","nonce":3129716965}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6802","nonce":3129716965}]},"public_addr":"172.21.15.193:6800/3129716965","cluster_addr":"172.21.15.193:6801/3129716965","heartbeat_back_addr":"172.21.15.193:6803/3129716965","heartbeat_front_addr":"172.21.15.193:6802/3129716965","state":["exists","up"]},{"osd":5,"uuid":"59065f9b-9240-414b-9c91-e91cda4aed6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":833,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6804","nonce":805161991}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6805","nonce":805161991}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6807","nonce":805161991}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6806","nonce":805161991}]},"public_addr":"172.21.15.193:6804/805161991","cluster_addr":"172.21.15.193:6805/805161991","heartbeat_back_addr":"172.21.15.193:6807/805161991","heartbeat_front_addr":"172.21.15.193:6806/805161991","state":["exists","up"]},{"osd":6,"uuid":"302c95df-3842-4818-8afd-d79d8184684f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":845,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6808","nonce":787604615}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6809","nonce":787604615}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6811","nonce":787604615}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6810","nonce":787604615}]},"public_addr":"172.21.15.193:6808/787604615","cluster_addr":"172.21.15.193:6809/787604615","heartbeat_back_addr":"172.21.15.193:6811/787604615","heartbeat_front_addr":"172.21.15.193:6810/787604615","state":["exists","up"]},{"osd":7,"uuid":"9ebeff27-c2f8-4a66-923f-a0d7573ddd3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":848,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6812","nonce":157927432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6813","nonce":157927432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6815","nonce":157927432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.193:6814","nonce":157927432}]},"public_addr":"172.21.15.193:6812/157927432","cluster_addr":"172.21.15.193:6813/157927432","heartbeat_back_addr":"172.21.15.193:6815/157927432","heartbeat_front_addr":"172.21.15.193:6814/157927432","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:43.017101+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:23.902515+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:59.423013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:13.753710+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:38.714651+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:21.619833+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:57.661959+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T21:18:55.030177+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":7,"to":2}]},{"pgid":"2.7","mappings":[{"from":7,"to":1}]},{"pgid":"2.9","mappings":[{"from":7,"to":2}]},{"pgid":"2.1a","mappings":[{"from":7,"to":2}]},{"pgid":"2.1d","mappings":[{"from":7,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:0/1427774891":"2024-02-20T21:01:38.896227+0000","172.21.15.163:0/3282102947":"2024-02-20T21:01:38.896227+0000","172.21.15.163:6800/2735092549":"2024-02-20T21:01:38.896227+0000","172.21.15.163:0/1235588846":"2024-02-20T20:56:29.603030+0000","172.21.15.163:6800/2731225015":"2024-02-20T20:56:29.603030+0000","172.21.15.163:0/506491651":"2024-02-20T20:56:02.185409+0000","172.21.15.163:0/168117826":"2024-02-20T21:01:38.896227+0000","172.21.15.163:0/1458009465":"2024-02-20T20:56:29.603030+0000","172.21.15.163:0/541631472":"2024-02-20T20:56:02.185409+0000","172.21.15.163:6800/2248999541":"2024-02-20T20:56:02.185409+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":25,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":57,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":76,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":85,"snaps":[{"begin":2,"length":1}]},{"pool":92,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":94,"snaps":[{"begin":4,"length":4}]},{"pool":107,"snaps":[{"begin":2,"length":1}]},{"pool":112,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":126,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":132,"snaps":[{"begin":12,"length":1}]},{"pool":154,"snaps":[{"begin":14,"length":1}]},{"pool":164,"snaps":[{"begin":17,"length":1}]},{"pool":178,"snaps":[{"begin":20,"length":1}]},{"pool":237,"snaps":[{"begin":2,"length":1}]},{"pool":238,"snaps":[{"begin":4,"length":1}]},{"pool":241,"snaps":[{"begin":6,"length":1}]},{"pool":245,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T21:19:15.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.545+0000 7f9eefb91700 10 librados: watch_flush enter 2024-02-19T21:19:15.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.545+0000 7f9eefb91700 10 librados: watch_flush exit 2024-02-19T21:19:15.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.545+0000 7f9eefb91700 20 client.50810.objecter shutdown clearing up homeless session... 2024-02-19T21:19:15.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.545+0000 7f9eefb91700 10 client.50810.objecter successfully canceled tick 2024-02-19T21:19:15.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.545+0000 7f9eefb91700 1 -- 172.21.15.163:0/851159755 >> v1:172.21.15.163:6800/1552514681 conn(0x7f9ed0063c70 legacy=0x7f9ed0066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:15.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.545+0000 7f9eefb91700 1 -- 172.21.15.163:0/851159755 >> v1:172.21.15.163:6790/0 conn(0x7f9ee81042c0 legacy=0x7f9ee81159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:15.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.545+0000 7f9eefb91700 1 -- 172.21.15.163:0/851159755 shutdown_connections 2024-02-19T21:19:15.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.545+0000 7f9eefb91700 1 -- 172.21.15.163:0/851159755 >> 172.21.15.163:0/851159755 conn(0x7f9ee80f35a0 msgr2=0x7f9ee8107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:15.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.545+0000 7f9eefb91700 1 -- 172.21.15.163:0/851159755 shutdown_connections 2024-02-19T21:19:15.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.545+0000 7f9eefb91700 1 -- 172.21.15.163:0/851159755 wait complete. 2024-02-19T21:19:15.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:15.545+0000 7f9eefb91700 1 librados: shutdown 2024-02-19T21:19:15.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:14.433880+0000 osd.6 (osd.6) 331 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:19:15.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:14.434791+0000 osd.6 (osd.6) 332 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:19:15.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:14.944762+0000 osd.2 (osd.2) 370 : cluster [DBG] 16.1 scrub starts 2024-02-19T21:19:15.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:14.948328+0000 osd.2 (osd.2) 371 : cluster [DBG] 16.1 scrub ok 2024-02-19T21:19:15.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:15.544105+0000 mon.c (mon.2) 450 : audit [DBG] from='client.? v1:172.21.15.163:0/851159755' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T21:19:15.911 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:15.623021+0000 osd.5 (osd.5) 366 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:19:15.912 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:15.626637+0000 osd.5 (osd.5) 367 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:19:15.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:14.433880+0000 osd.6 (osd.6) 331 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:19:15.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:14.434791+0000 osd.6 (osd.6) 332 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:19:15.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:14.944762+0000 osd.2 (osd.2) 370 : cluster [DBG] 16.1 scrub starts 2024-02-19T21:19:15.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:14.948328+0000 osd.2 (osd.2) 371 : cluster [DBG] 16.1 scrub ok 2024-02-19T21:19:15.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:15.544105+0000 mon.c (mon.2) 450 : audit [DBG] from='client.? v1:172.21.15.163:0/851159755' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T21:19:15.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:15.623021+0000 osd.5 (osd.5) 366 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:19:15.912 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:15 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:15.626637+0000 osd.5 (osd.5) 367 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:19:16.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:14.433880+0000 osd.6 (osd.6) 331 : cluster [DBG] 16.12 scrub starts 2024-02-19T21:19:16.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:14.434791+0000 osd.6 (osd.6) 332 : cluster [DBG] 16.12 scrub ok 2024-02-19T21:19:16.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:14.944762+0000 osd.2 (osd.2) 370 : cluster [DBG] 16.1 scrub starts 2024-02-19T21:19:16.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:14.948328+0000 osd.2 (osd.2) 371 : cluster [DBG] 16.1 scrub ok 2024-02-19T21:19:16.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:15.544105+0000 mon.c (mon.2) 450 : audit [DBG] from='client.? v1:172.21.15.163:0/851159755' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T21:19:16.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:15.623021+0000 osd.5 (osd.5) 366 : cluster [DBG] 4.8 scrub starts 2024-02-19T21:19:16.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:15.626637+0000 osd.5 (osd.5) 367 : cluster [DBG] 4.8 scrub ok 2024-02-19T21:19:16.307 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-19T21:19:16.583 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:17.066 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:15.368507+0000 mgr.y (mgr.14550) 718 : cluster [DBG] pgmap v1308: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:19:17.066 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:15.372825+0000 osd.0 (osd.0) 172 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:19:17.066 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:15.374614+0000 osd.0 (osd.0) 173 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:19:17.066 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:15.741301+0000 mon.a (mon.0) 3871 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 17 pgs inactive, 19 pgs peering) 2024-02-19T21:19:17.066 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:15.984515+0000 osd.2 (osd.2) 372 : cluster [DBG] 5.d scrub starts 2024-02-19T21:19:17.066 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:15.986156+0000 osd.2 (osd.2) 373 : cluster [DBG] 5.d scrub ok 2024-02-19T21:19:17.066 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:16.215905+0000 osd.3 (osd.3) 397 : cluster [DBG] 5.17 deep-scrub starts 2024-02-19T21:19:17.066 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:16.216784+0000 osd.3 (osd.3) 398 : cluster [DBG] 5.17 deep-scrub ok 2024-02-19T21:19:17.067 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:15.368507+0000 mgr.y (mgr.14550) 718 : cluster [DBG] pgmap v1308: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:19:17.067 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:15.372825+0000 osd.0 (osd.0) 172 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:19:17.067 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:15.374614+0000 osd.0 (osd.0) 173 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:19:17.067 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:15.741301+0000 mon.a (mon.0) 3871 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 17 pgs inactive, 19 pgs peering) 2024-02-19T21:19:17.067 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:15.984515+0000 osd.2 (osd.2) 372 : cluster [DBG] 5.d scrub starts 2024-02-19T21:19:17.067 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:15.986156+0000 osd.2 (osd.2) 373 : cluster [DBG] 5.d scrub ok 2024-02-19T21:19:17.067 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:16.215905+0000 osd.3 (osd.3) 397 : cluster [DBG] 5.17 deep-scrub starts 2024-02-19T21:19:17.067 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:16.216784+0000 osd.3 (osd.3) 398 : cluster [DBG] 5.17 deep-scrub ok 2024-02-19T21:19:17.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:15.368507+0000 mgr.y (mgr.14550) 718 : cluster [DBG] pgmap v1308: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:19:17.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:15.372825+0000 osd.0 (osd.0) 172 : cluster [DBG] 5.3 scrub starts 2024-02-19T21:19:17.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:15.374614+0000 osd.0 (osd.0) 173 : cluster [DBG] 5.3 scrub ok 2024-02-19T21:19:17.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:15.741301+0000 mon.a (mon.0) 3871 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 17 pgs inactive, 19 pgs peering) 2024-02-19T21:19:17.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:15.984515+0000 osd.2 (osd.2) 372 : cluster [DBG] 5.d scrub starts 2024-02-19T21:19:17.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:15.986156+0000 osd.2 (osd.2) 373 : cluster [DBG] 5.d scrub ok 2024-02-19T21:19:17.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:16.215905+0000 osd.3 (osd.3) 397 : cluster [DBG] 5.17 deep-scrub starts 2024-02-19T21:19:17.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:16.216784+0000 osd.3 (osd.3) 398 : cluster [DBG] 5.17 deep-scrub ok 2024-02-19T21:19:17.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:16.560769+0000 osd.7 (osd.7) 343 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:19:17.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:16.563694+0000 osd.7 (osd.7) 344 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:19:17.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:16.995843+0000 osd.2 (osd.2) 374 : cluster [DBG] 5.a scrub starts 2024-02-19T21:19:17.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:16.996693+0000 osd.2 (osd.2) 375 : cluster [DBG] 5.a scrub ok 2024-02-19T21:19:17.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:17.625853+0000 osd.5 (osd.5) 368 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:19:17.881 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:17.628862+0000 osd.5 (osd.5) 369 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:19:17.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:16.560769+0000 osd.7 (osd.7) 343 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:19:17.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:16.563694+0000 osd.7 (osd.7) 344 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:19:17.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:16.995843+0000 osd.2 (osd.2) 374 : cluster [DBG] 5.a scrub starts 2024-02-19T21:19:17.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:16.996693+0000 osd.2 (osd.2) 375 : cluster [DBG] 5.a scrub ok 2024-02-19T21:19:17.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:17.625853+0000 osd.5 (osd.5) 368 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:19:17.882 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:17 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:17.628862+0000 osd.5 (osd.5) 369 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:19:18.072 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:19:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:19:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:19:18.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:16.560769+0000 osd.7 (osd.7) 343 : cluster [DBG] 5.10 scrub starts 2024-02-19T21:19:18.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:16.563694+0000 osd.7 (osd.7) 344 : cluster [DBG] 5.10 scrub ok 2024-02-19T21:19:18.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:16.995843+0000 osd.2 (osd.2) 374 : cluster [DBG] 5.a scrub starts 2024-02-19T21:19:18.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:16.996693+0000 osd.2 (osd.2) 375 : cluster [DBG] 5.a scrub ok 2024-02-19T21:19:18.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:17.625853+0000 osd.5 (osd.5) 368 : cluster [DBG] 5.16 scrub starts 2024-02-19T21:19:18.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:17 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:17.628862+0000 osd.5 (osd.5) 369 : cluster [DBG] 5.16 scrub ok 2024-02-19T21:19:18.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.373+0000 7f7072c87700 1 -- 172.21.15.163:0/2736394648 >> v1:172.21.15.163:6789/0 conn(0x7f706c0f8920 legacy=0x7f706c0f8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:18.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.373+0000 7f7072c87700 1 -- 172.21.15.163:0/2736394648 shutdown_connections 2024-02-19T21:19:18.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.373+0000 7f7072c87700 1 -- 172.21.15.163:0/2736394648 >> 172.21.15.163:0/2736394648 conn(0x7f706c0f2470 msgr2=0x7f706c0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:18.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.373+0000 7f7072c87700 1 -- 172.21.15.163:0/2736394648 shutdown_connections 2024-02-19T21:19:18.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.373+0000 7f7072c87700 1 -- 172.21.15.163:0/2736394648 wait complete. 2024-02-19T21:19:18.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.374+0000 7f7072c87700 1 librados: starting msgr at 2024-02-19T21:19:18.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.374+0000 7f7072c87700 1 librados: starting objecter 2024-02-19T21:19:18.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.374+0000 7f7072c87700 1 Processor -- start 2024-02-19T21:19:18.376 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.374+0000 7f7072c87700 1 -- start start 2024-02-19T21:19:18.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.374+0000 7f7072c87700 1 librados: setting wanted keys 2024-02-19T21:19:18.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.374+0000 7f7072c87700 1 librados: calling monclient init 2024-02-19T21:19:18.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.374+0000 7f7072c87700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f706c102b40 con 0x7f706c0f94d0 2024-02-19T21:19:18.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.374+0000 7f7072c87700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f706c102d10 con 0x7f706c1042b0 2024-02-19T21:19:18.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.374+0000 7f7072c87700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f706c102f00 con 0x7f706c0f8920 2024-02-19T21:19:18.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.374+0000 7f706bfff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f706c0f94d0 0x7f706c114d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:53712/0 (socket says 172.21.15.163:53712) 2024-02-19T21:19:18.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.374+0000 7f7070a23700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f706c0f8920 0x7f706c1023a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:42014/0 (socket says 172.21.15.163:42014) 2024-02-19T21:19:18.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.374+0000 7f706bfff700 1 -- 172.21.15.163:0/163917489 learned_addr learned my addr 172.21.15.163:0/163917489 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:18.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 10 client.?.objecter ms_handle_connect 0x7f706c0f8920 2024-02-19T21:19:18.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:18.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 10 client.?.objecter ms_handle_connect 0x7f706c0f94d0 2024-02-19T21:19:18.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:18.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 10 client.?.objecter ms_handle_connect 0x7f706c1042b0 2024-02-19T21:19:18.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:18.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616512647 0 0) 0x7f706c102f00 con 0x7f706c0f8920 2024-02-19T21:19:18.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70540038e0 con 0x7f706c0f8920 2024-02-19T21:19:18.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393461409 0 0) 0x7f706c102b40 con 0x7f706c0f94d0 2024-02-19T21:19:18.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f706c102f00 con 0x7f706c0f94d0 2024-02-19T21:19:18.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122076504 0 0) 0x7f706c102d10 con 0x7f706c1042b0 2024-02-19T21:19:18.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f706c102b40 con 0x7f706c1042b0 2024-02-19T21:19:18.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 438143750 0 0) 0x7f70540038e0 con 0x7f706c0f8920 2024-02-19T21:19:18.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f706c102d10 con 0x7f706c0f8920 2024-02-19T21:19:18.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f70640030c0 con 0x7f706c0f8920 2024-02-19T21:19:18.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3281041250 0 0) 0x7f706c102f00 con 0x7f706c0f94d0 2024-02-19T21:19:18.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70540038e0 con 0x7f706c0f94d0 2024-02-19T21:19:18.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f70600019f0 con 0x7f706c0f94d0 2024-02-19T21:19:18.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2008893707 0 0) 0x7f706c102b40 con 0x7f706c1042b0 2024-02-19T21:19:18.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f706c102f00 con 0x7f706c1042b0 2024-02-19T21:19:18.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f705c001780 con 0x7f706c1042b0 2024-02-19T21:19:18.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1999736590 0 0) 0x7f706c102d10 con 0x7f706c0f8920 2024-02-19T21:19:18.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 >> v1:172.21.15.193:6789/0 conn(0x7f706c1042b0 legacy=0x7f706c118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:18.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 >> v1:172.21.15.163:6789/0 conn(0x7f706c0f94d0 legacy=0x7f706c114d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:18.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c11cbc0 con 0x7f706c0f8920 2024-02-19T21:19:18.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7072c87700 1 -- 172.21.15.163:0/163917489 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f706c119bb0 con 0x7f706c0f8920 2024-02-19T21:19:18.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f7064003220 con 0x7f706c0f8920 2024-02-19T21:19:18.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.375+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7064003c20 con 0x7f706c0f8920 2024-02-19T21:19:18.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.376+0000 7f7072c87700 10 client.50816.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:18.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.376+0000 7f7072c87700 1 -- 172.21.15.163:0/163917489 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f706c11a020 con 0x7f706c0f8920 2024-02-19T21:19:18.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.376+0000 7f7072c87700 1 librados: init done 2024-02-19T21:19:18.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.376+0000 7f7072c87700 10 client.50816.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:19:18.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.376+0000 7f7072c87700 15 client.50816.objecter _session_command_op_assign -1 1 2024-02-19T21:19:18.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.376+0000 7f7072c87700 20 client.50816.objecter _get_session osd=-1 returning homeless 2024-02-19T21:19:18.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.376+0000 7f7072c87700 10 client.50816.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:18.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.376+0000 7f7072c87700 1 -- 172.21.15.163:0/163917489 --> v1:172.21.15.163:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f706c05f700 con 0x7f706c0f8920 2024-02-19T21:19:18.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f7064003e60 con 0x7f706c0f8920 2024-02-19T21:19:18.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(849..849 src has 796..849) v4 ==== 7419+0+0 (unknown 1436230568 0 0) 0x7f7064079010 con 0x7f706c0f8920 2024-02-19T21:19:18.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 10 client.50816.objecter ms_dispatch 0x7f706c001550 osd_map(849..849 src has 796..849) v4 2024-02-19T21:19:18.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 3 client.50816.objecter handle_osd_map got epochs [849,849] > 0 2024-02-19T21:19:18.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 3 client.50816.objecter handle_osd_map decoding full epoch 849 2024-02-19T21:19:18.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 10 client.50816.objecter checking command 1 2024-02-19T21:19:18.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 20 client.50816.objecter _get_session s=0x7f705406f950 osd=0 3 2024-02-19T21:19:18.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 20 client.50816.objecter put_session s=0x7f705406f950 osd=0 3 2024-02-19T21:19:18.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 15 client.50816.objecter _session_command_op_remove -1 1 2024-02-19T21:19:18.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 20 client.50816.objecter _get_session s=0x7f705406f950 osd=0 3 2024-02-19T21:19:18.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 20 client.50816.objecter get_session s=0x7f705406f950 osd=0 3 2024-02-19T21:19:18.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 15 client.50816.objecter _session_command_op_assign 0 1 2024-02-19T21:19:18.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 20 client.50816.objecter put_session s=0x7f705406f950 osd=0 4 2024-02-19T21:19:18.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 10 client.50816.objecter _send_command 1 2024-02-19T21:19:18.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 --> v1:172.21.15.163:6801/1559418115 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7054073d10 con 0x7f7054070000 2024-02-19T21:19:18.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 20 client.50816.objecter dump_active .. 0 homeless 2024-02-19T21:19:18.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_get_version_reply(handle=1 version=849) v2 ==== 24+0+0 (unknown 4047222891 0 0) 0x7f70640793f0 con 0x7f706c0f8920 2024-02-19T21:19:18.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 10 client.50816.objecter ms_handle_connect 0x7f7054063c30 2024-02-19T21:19:18.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.377+0000 7f7069ffb700 10 client.50816.objecter ms_handle_connect 0x7f7054070000 2024-02-19T21:19:18.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.378+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== osd.0 v1:172.21.15.163:6801/1559418115 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f7054073d10 con 0x7f7054070000 2024-02-19T21:19:18.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.378+0000 7f7069ffb700 10 client.50816.objecter ms_dispatch 0x7f706c001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:19:18.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.378+0000 7f7069ffb700 10 client.50816.objecter _finish_command 1 = system:0 2024-02-19T21:19:18.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.378+0000 7f7069ffb700 20 client.50816.objecter put_session s=0x7f705406f950 osd=0 4 2024-02-19T21:19:18.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.378+0000 7f7069ffb700 15 client.50816.objecter _session_command_op_remove 0 1 2024-02-19T21:19:18.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7072c87700 10 client.50816.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:19:18.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7072c87700 15 client.50816.objecter _session_command_op_assign -1 2 2024-02-19T21:19:18.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7072c87700 20 client.50816.objecter _get_session s=0x7f705406f950 osd=0 3 2024-02-19T21:19:18.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7072c87700 20 client.50816.objecter put_session s=0x7f705406f950 osd=0 3 2024-02-19T21:19:18.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7072c87700 20 client.50816.objecter _get_session s=0x7f705406f950 osd=0 3 2024-02-19T21:19:18.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7072c87700 15 client.50816.objecter _session_command_op_remove -1 2 2024-02-19T21:19:18.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7072c87700 20 client.50816.objecter get_session s=0x7f705406f950 osd=0 3 2024-02-19T21:19:18.394 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7072c87700 15 client.50816.objecter _session_command_op_assign 0 2 2024-02-19T21:19:18.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7072c87700 20 client.50816.objecter put_session s=0x7f705406f950 osd=0 4 2024-02-19T21:19:18.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7072c87700 10 client.50816.objecter _send_command 2 2024-02-19T21:19:18.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7072c87700 1 -- 172.21.15.163:0/163917489 --> v1:172.21.15.163:6801/1559418115 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f706c047540 con 0x7f7054070000 2024-02-19T21:19:18.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7069ffb700 1 -- 172.21.15.163:0/163917489 <== osd.0 v1:172.21.15.163:6801/1559418115 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 735264872) 0x7f706c047540 con 0x7f7054070000 2024-02-19T21:19:18.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7069ffb700 10 client.50816.objecter ms_dispatch 0x7f706c001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:19:18.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7069ffb700 10 client.50816.objecter _finish_command 2 = system:0 2024-02-19T21:19:18.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7069ffb700 20 client.50816.objecter put_session s=0x7f705406f950 osd=0 4 2024-02-19T21:19:18.395 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7069ffb700 15 client.50816.objecter _session_command_op_remove 0 2 2024-02-19T21:19:18.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.392+0000 7f7072c87700 10 librados: watch_flush enter 2024-02-19T21:19:18.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.393+0000 7f7072c87700 10 librados: watch_flush exit 2024-02-19T21:19:18.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.393+0000 7f7072c87700 10 client.50816.objecter close_session for osd.0 2024-02-19T21:19:18.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.393+0000 7f7072c87700 1 -- 172.21.15.163:0/163917489 >> v1:172.21.15.163:6801/1559418115 conn(0x7f7054070000 legacy=0x7f70540724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:18.396 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.393+0000 7f7072c87700 20 client.50816.objecter put_session s=0x7f705406f950 osd=0 1 2024-02-19T21:19:18.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.393+0000 7f7072c87700 20 client.50816.objecter shutdown clearing up homeless session... 2024-02-19T21:19:18.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.393+0000 7f7072c87700 10 client.50816.objecter successfully canceled tick 2024-02-19T21:19:18.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.393+0000 7f7072c87700 1 -- 172.21.15.163:0/163917489 >> v1:172.21.15.163:6800/1552514681 conn(0x7f7054063c30 legacy=0x7f70540660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:18.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.393+0000 7f7072c87700 1 -- 172.21.15.163:0/163917489 >> v1:172.21.15.163:6790/0 conn(0x7f706c0f8920 legacy=0x7f706c1023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:18.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.393+0000 7f7071224700 1 -- 172.21.15.163:0/163917489 reap_dead start 2024-02-19T21:19:18.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.393+0000 7f7072c87700 1 -- 172.21.15.163:0/163917489 shutdown_connections 2024-02-19T21:19:18.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.393+0000 7f7072c87700 1 -- 172.21.15.163:0/163917489 >> 172.21.15.163:0/163917489 conn(0x7f706c0f2470 msgr2=0x7f706c107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:18.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.393+0000 7f7072c87700 1 -- 172.21.15.163:0/163917489 shutdown_connections 2024-02-19T21:19:18.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.393+0000 7f7072c87700 1 -- 172.21.15.163:0/163917489 wait complete. 2024-02-19T21:19:18.397 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:18.393+0000 7f7072c87700 1 librados: shutdown 2024-02-19T21:19:18.961 INFO:teuthology.orchestra.run.smithi163.stdout:34359738622 2024-02-19T21:19:18.962 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-19T21:19:19.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:17.368917+0000 mgr.y (mgr.14550) 719 : cluster [DBG] pgmap v1309: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:19:19.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:17.425287+0000 osd.6 (osd.6) 333 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:19:19.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:17.426171+0000 osd.6 (osd.6) 334 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:19:19.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:18.644760+0000 osd.5 (osd.5) 370 : cluster [DBG] 6.a deep-scrub starts 2024-02-19T21:19:19.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:18 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:18.647840+0000 osd.5 (osd.5) 371 : cluster [DBG] 6.a deep-scrub ok 2024-02-19T21:19:19.232 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:19.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:17.368917+0000 mgr.y (mgr.14550) 719 : cluster [DBG] pgmap v1309: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:19:19.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:17.425287+0000 osd.6 (osd.6) 333 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:19:19.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:17.426171+0000 osd.6 (osd.6) 334 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:19:19.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:18.644760+0000 osd.5 (osd.5) 370 : cluster [DBG] 6.a deep-scrub starts 2024-02-19T21:19:19.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:18.647840+0000 osd.5 (osd.5) 371 : cluster [DBG] 6.a deep-scrub ok 2024-02-19T21:19:19.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:17.368917+0000 mgr.y (mgr.14550) 719 : cluster [DBG] pgmap v1309: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:19:19.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:17.425287+0000 osd.6 (osd.6) 333 : cluster [DBG] 2.7 scrub starts 2024-02-19T21:19:19.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:17.426171+0000 osd.6 (osd.6) 334 : cluster [DBG] 2.7 scrub ok 2024-02-19T21:19:19.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:18.644760+0000 osd.5 (osd.5) 370 : cluster [DBG] 6.a deep-scrub starts 2024-02-19T21:19:19.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:18.647840+0000 osd.5 (osd.5) 371 : cluster [DBG] 6.a deep-scrub ok 2024-02-19T21:19:20.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:18.343787+0000 osd.0 (osd.0) 174 : cluster [DBG] 16.5 scrub starts 2024-02-19T21:19:20.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:18.345207+0000 osd.0 (osd.0) 175 : cluster [DBG] 16.5 scrub ok 2024-02-19T21:19:20.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:18.560439+0000 osd.7 (osd.7) 345 : cluster [DBG] 5.9 deep-scrub starts 2024-02-19T21:19:20.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:18.563920+0000 osd.7 (osd.7) 346 : cluster [DBG] 5.9 deep-scrub ok 2024-02-19T21:19:20.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:18.747407+0000 osd.1 (osd.1) 405 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:19:20.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:18.748333+0000 osd.1 (osd.1) 406 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:19:20.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:18.988368+0000 osd.2 (osd.2) 376 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:19:20.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:19 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:18.991314+0000 osd.2 (osd.2) 377 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:19:20.159 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:18.343787+0000 osd.0 (osd.0) 174 : cluster [DBG] 16.5 scrub starts 2024-02-19T21:19:20.160 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:18.345207+0000 osd.0 (osd.0) 175 : cluster [DBG] 16.5 scrub ok 2024-02-19T21:19:20.160 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:18.560439+0000 osd.7 (osd.7) 345 : cluster [DBG] 5.9 deep-scrub starts 2024-02-19T21:19:20.160 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:18.563920+0000 osd.7 (osd.7) 346 : cluster [DBG] 5.9 deep-scrub ok 2024-02-19T21:19:20.160 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:18.747407+0000 osd.1 (osd.1) 405 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:19:20.160 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:18.748333+0000 osd.1 (osd.1) 406 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:19:20.160 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:18.988368+0000 osd.2 (osd.2) 376 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:19:20.160 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:18.991314+0000 osd.2 (osd.2) 377 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:19:20.160 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:18.343787+0000 osd.0 (osd.0) 174 : cluster [DBG] 16.5 scrub starts 2024-02-19T21:19:20.160 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:18.345207+0000 osd.0 (osd.0) 175 : cluster [DBG] 16.5 scrub ok 2024-02-19T21:19:20.161 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:18.560439+0000 osd.7 (osd.7) 345 : cluster [DBG] 5.9 deep-scrub starts 2024-02-19T21:19:20.161 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:18.563920+0000 osd.7 (osd.7) 346 : cluster [DBG] 5.9 deep-scrub ok 2024-02-19T21:19:20.161 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:18.747407+0000 osd.1 (osd.1) 405 : cluster [DBG] 6.4 scrub starts 2024-02-19T21:19:20.161 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:18.748333+0000 osd.1 (osd.1) 406 : cluster [DBG] 6.4 scrub ok 2024-02-19T21:19:20.161 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:18.988368+0000 osd.2 (osd.2) 376 : cluster [DBG] 5.8 scrub starts 2024-02-19T21:19:20.161 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:18.991314+0000 osd.2 (osd.2) 377 : cluster [DBG] 5.8 scrub ok 2024-02-19T21:19:20.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:19.369619+0000 mgr.y (mgr.14550) 720 : cluster [DBG] pgmap v1310: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:19:20.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:19.547373+0000 osd.7 (osd.7) 347 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:19:20.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:19.550358+0000 osd.7 (osd.7) 348 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:19:20.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:19.736867+0000 osd.3 (osd.3) 399 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:19:20.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:19.737028+0000 osd.3 (osd.3) 400 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:19:20.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:20.245572+0000 osd.3 (osd.3) 401 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:19:20.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:20.246868+0000 osd.3 (osd.3) 402 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:19:20.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:20.460744+0000 mon.a (mon.0) 3872 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2024-02-19T21:19:20.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:19.369619+0000 mgr.y (mgr.14550) 720 : cluster [DBG] pgmap v1310: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:19:20.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:19.547373+0000 osd.7 (osd.7) 347 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:19:20.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:19.550358+0000 osd.7 (osd.7) 348 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:19:20.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:19.736867+0000 osd.3 (osd.3) 399 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:19:20.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:19.737028+0000 osd.3 (osd.3) 400 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:19:20.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:20.245572+0000 osd.3 (osd.3) 401 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:19:20.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:20.246868+0000 osd.3 (osd.3) 402 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:19:20.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:20.460744+0000 mon.a (mon.0) 3872 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2024-02-19T21:19:21.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.031+0000 7f21d2647700 1 -- 172.21.15.163:0/426824039 >> v1:172.21.15.163:6790/0 conn(0x7f21cc0ff670 legacy=0x7f21cc101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:21.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.031+0000 7f21d2647700 1 -- 172.21.15.163:0/426824039 shutdown_connections 2024-02-19T21:19:21.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.031+0000 7f21d2647700 1 -- 172.21.15.163:0/426824039 >> 172.21.15.163:0/426824039 conn(0x7f21cc0f2470 msgr2=0x7f21cc0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:21.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.031+0000 7f21d2647700 1 -- 172.21.15.163:0/426824039 shutdown_connections 2024-02-19T21:19:21.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.031+0000 7f21d2647700 1 -- 172.21.15.163:0/426824039 wait complete. 2024-02-19T21:19:21.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.031+0000 7f21d2647700 1 librados: starting msgr at 2024-02-19T21:19:21.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.031+0000 7f21d2647700 1 librados: starting objecter 2024-02-19T21:19:21.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21d2647700 1 Processor -- start 2024-02-19T21:19:21.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21d2647700 1 -- start start 2024-02-19T21:19:21.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21d2647700 1 librados: setting wanted keys 2024-02-19T21:19:21.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21d2647700 1 librados: calling monclient init 2024-02-19T21:19:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21d2647700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f21cc1172b0 con 0x7f21cc1031b0 2024-02-19T21:19:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21d2647700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f21cc118490 con 0x7f21cc0fc870 2024-02-19T21:19:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21d2647700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f21cc119690 con 0x7f21cc0ff670 2024-02-19T21:19:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21cb7fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f21cc0ff670 0x7f21cc0613b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:42022/0 (socket says 172.21.15.163:42022) 2024-02-19T21:19:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21d0be4700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f21cc1031b0 0x7f21cc115970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:53722/0 (socket says 172.21.15.163:53722) 2024-02-19T21:19:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21cb7fe700 1 -- 172.21.15.163:0/3765773875 learned_addr learned my addr 172.21.15.163:0/3765773875 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21c97fa700 10 client.?.objecter ms_handle_connect 0x7f21cc0ff670 2024-02-19T21:19:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21c97fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21c97fa700 10 client.?.objecter ms_handle_connect 0x7f21cc1031b0 2024-02-19T21:19:21.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21c97fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21c97fa700 10 client.?.objecter ms_handle_connect 0x7f21cc0fc870 2024-02-19T21:19:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21c97fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080947386 0 0) 0x7f21cc1172b0 con 0x7f21cc1031b0 2024-02-19T21:19:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b8003030 con 0x7f21cc1031b0 2024-02-19T21:19:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871960473 0 0) 0x7f21cc119690 con 0x7f21cc0ff670 2024-02-19T21:19:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21cc1172b0 con 0x7f21cc0ff670 2024-02-19T21:19:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343477388 0 0) 0x7f21cc118490 con 0x7f21cc0fc870 2024-02-19T21:19:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.032+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21cc119690 con 0x7f21cc0fc870 2024-02-19T21:19:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3543256272 0 0) 0x7f21b8003030 con 0x7f21cc1031b0 2024-02-19T21:19:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21cc118490 con 0x7f21cc1031b0 2024-02-19T21:19:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f21c4001b40 con 0x7f21cc1031b0 2024-02-19T21:19:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1588317757 0 0) 0x7f21cc1172b0 con 0x7f21cc0ff670 2024-02-19T21:19:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21b8003030 con 0x7f21cc0ff670 2024-02-19T21:19:21.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f21b4002020 con 0x7f21cc0ff670 2024-02-19T21:19:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3879355255 0 0) 0x7f21cc119690 con 0x7f21cc0fc870 2024-02-19T21:19:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21cc1172b0 con 0x7f21cc0fc870 2024-02-19T21:19:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f21bc0019f0 con 0x7f21cc0fc870 2024-02-19T21:19:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3232312729 0 0) 0x7f21cc118490 con 0x7f21cc1031b0 2024-02-19T21:19:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 >> v1:172.21.15.163:6790/0 conn(0x7f21cc0ff670 legacy=0x7f21cc0613b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 >> v1:172.21.15.193:6789/0 conn(0x7f21cc0fc870 legacy=0x7f21cc060ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc11a890 con 0x7f21cc1031b0 2024-02-19T21:19:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21d2647700 1 -- 172.21.15.163:0/3765773875 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21cc1174a0 con 0x7f21cc1031b0 2024-02-19T21:19:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21d2647700 10 client.24096.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21d2647700 1 -- 172.21.15.163:0/3765773875 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21cc117990 con 0x7f21cc1031b0 2024-02-19T21:19:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21d2647700 1 librados: init done 2024-02-19T21:19:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f21c40025d0 con 0x7f21cc1031b0 2024-02-19T21:19:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.033+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f21c4005ca0 con 0x7f21cc1031b0 2024-02-19T21:19:21.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.034+0000 7f21d2647700 10 client.24096.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:19:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.034+0000 7f21d2647700 15 client.24096.objecter _session_command_op_assign -1 1 2024-02-19T21:19:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.034+0000 7f21d2647700 20 client.24096.objecter _get_session osd=-1 returning homeless 2024-02-19T21:19:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.034+0000 7f21d2647700 10 client.24096.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:21.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.034+0000 7f21d2647700 1 -- 172.21.15.163:0/3765773875 --> v1:172.21.15.163:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f21ac0028b0 con 0x7f21cc1031b0 2024-02-19T21:19:21.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.034+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f21c4002130 con 0x7f21cc1031b0 2024-02-19T21:19:21.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(850..850 src has 796..850) v4 ==== 7419+0+0 (unknown 7470691 0 0) 0x7f21c4078b00 con 0x7f21cc1031b0 2024-02-19T21:19:21.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 10 client.24096.objecter ms_dispatch 0x7f21cc001550 osd_map(850..850 src has 796..850) v4 2024-02-19T21:19:21.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 3 client.24096.objecter handle_osd_map got epochs [850,850] > 0 2024-02-19T21:19:21.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 3 client.24096.objecter handle_osd_map decoding full epoch 850 2024-02-19T21:19:21.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 10 client.24096.objecter checking command 1 2024-02-19T21:19:21.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 20 client.24096.objecter _get_session s=0x7f21b4002020 osd=1 3 2024-02-19T21:19:21.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 20 client.24096.objecter put_session s=0x7f21b4002020 osd=1 3 2024-02-19T21:19:21.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 15 client.24096.objecter _session_command_op_remove -1 1 2024-02-19T21:19:21.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 20 client.24096.objecter _get_session s=0x7f21b4002020 osd=1 3 2024-02-19T21:19:21.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 20 client.24096.objecter get_session s=0x7f21b4002020 osd=1 3 2024-02-19T21:19:21.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 15 client.24096.objecter _session_command_op_assign 1 1 2024-02-19T21:19:21.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 20 client.24096.objecter put_session s=0x7f21b4002020 osd=1 4 2024-02-19T21:19:21.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 10 client.24096.objecter _send_command 1 2024-02-19T21:19:21.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 --> v1:172.21.15.163:6805/3125047432 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21b8073cc0 con 0x7f21b806ffb0 2024-02-19T21:19:21.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 20 client.24096.objecter dump_active .. 0 homeless 2024-02-19T21:19:21.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_get_version_reply(handle=1 version=850) v2 ==== 24+0+0 (unknown 3804485784 0 0) 0x7f21c4078ee0 con 0x7f21cc1031b0 2024-02-19T21:19:21.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 10 client.24096.objecter ms_handle_connect 0x7f21b8063ce0 2024-02-19T21:19:21.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.035+0000 7f21c97fa700 10 client.24096.objecter ms_handle_connect 0x7f21b806ffb0 2024-02-19T21:19:21.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.036+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== osd.1 v1:172.21.15.163:6805/3125047432 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f21b8073cc0 con 0x7f21b806ffb0 2024-02-19T21:19:21.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.036+0000 7f21c97fa700 10 client.24096.objecter ms_dispatch 0x7f21cc001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:19:21.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.036+0000 7f21c97fa700 10 client.24096.objecter _finish_command 1 = system:0 2024-02-19T21:19:21.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.036+0000 7f21c97fa700 20 client.24096.objecter put_session s=0x7f21b4002020 osd=1 4 2024-02-19T21:19:21.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.036+0000 7f21c97fa700 15 client.24096.objecter _session_command_op_remove 1 1 2024-02-19T21:19:21.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 10 client.24096.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:19:21.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 15 client.24096.objecter _session_command_op_assign -1 2 2024-02-19T21:19:21.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 20 client.24096.objecter _get_session s=0x7f21b4002020 osd=1 3 2024-02-19T21:19:21.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 20 client.24096.objecter put_session s=0x7f21b4002020 osd=1 3 2024-02-19T21:19:21.052 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 20 client.24096.objecter _get_session s=0x7f21b4002020 osd=1 3 2024-02-19T21:19:21.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 15 client.24096.objecter _session_command_op_remove -1 2 2024-02-19T21:19:21.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 20 client.24096.objecter get_session s=0x7f21b4002020 osd=1 3 2024-02-19T21:19:21.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 15 client.24096.objecter _session_command_op_assign 1 2 2024-02-19T21:19:21.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 20 client.24096.objecter put_session s=0x7f21b4002020 osd=1 4 2024-02-19T21:19:21.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 10 client.24096.objecter _send_command 2 2024-02-19T21:19:21.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 1 -- 172.21.15.163:0/3765773875 --> v1:172.21.15.163:6805/3125047432 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f21ac003c60 con 0x7f21b806ffb0 2024-02-19T21:19:21.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21c97fa700 1 -- 172.21.15.163:0/3765773875 <== osd.1 v1:172.21.15.163:6805/3125047432 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 716926331) 0x7f21ac003c60 con 0x7f21b806ffb0 2024-02-19T21:19:21.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21c97fa700 10 client.24096.objecter ms_dispatch 0x7f21cc001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:19:21.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21c97fa700 10 client.24096.objecter _finish_command 2 = system:0 2024-02-19T21:19:21.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21c97fa700 20 client.24096.objecter put_session s=0x7f21b4002020 osd=1 4 2024-02-19T21:19:21.053 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21c97fa700 15 client.24096.objecter _session_command_op_remove 1 2 2024-02-19T21:19:21.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 10 librados: watch_flush enter 2024-02-19T21:19:21.054 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 10 librados: watch_flush exit 2024-02-19T21:19:21.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 10 client.24096.objecter close_session for osd.1 2024-02-19T21:19:21.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 1 -- 172.21.15.163:0/3765773875 >> v1:172.21.15.163:6805/3125047432 conn(0x7f21b806ffb0 legacy=0x7f21b8072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:21.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 20 client.24096.objecter put_session s=0x7f21b4002020 osd=1 1 2024-02-19T21:19:21.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 20 client.24096.objecter shutdown clearing up homeless session... 2024-02-19T21:19:21.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.050+0000 7f21d2647700 10 client.24096.objecter successfully canceled tick 2024-02-19T21:19:21.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.051+0000 7f21d2647700 1 -- 172.21.15.163:0/3765773875 >> v1:172.21.15.163:6800/1552514681 conn(0x7f21b8063ce0 legacy=0x7f21b80661a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:21.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.051+0000 7f21d2647700 1 -- 172.21.15.163:0/3765773875 >> v1:172.21.15.163:6789/0 conn(0x7f21cc1031b0 legacy=0x7f21cc115970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:21.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.051+0000 7f21d0be4700 1 -- 172.21.15.163:0/3765773875 reap_dead start 2024-02-19T21:19:21.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.051+0000 7f21d2647700 1 -- 172.21.15.163:0/3765773875 shutdown_connections 2024-02-19T21:19:21.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.051+0000 7f21d2647700 1 -- 172.21.15.163:0/3765773875 >> 172.21.15.163:0/3765773875 conn(0x7f21cc0f2470 msgr2=0x7f21cc106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:21.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.051+0000 7f21d2647700 1 -- 172.21.15.163:0/3765773875 shutdown_connections 2024-02-19T21:19:21.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.051+0000 7f21d2647700 1 -- 172.21.15.163:0/3765773875 wait complete. 2024-02-19T21:19:21.055 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:21.051+0000 7f21d2647700 1 librados: shutdown 2024-02-19T21:19:21.072 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:19:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:19:21.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:19.369619+0000 mgr.y (mgr.14550) 720 : cluster [DBG] pgmap v1310: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-19T21:19:21.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:19.547373+0000 osd.7 (osd.7) 347 : cluster [DBG] 2.0 scrub starts 2024-02-19T21:19:21.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:19.550358+0000 osd.7 (osd.7) 348 : cluster [DBG] 2.0 scrub ok 2024-02-19T21:19:21.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:19.736867+0000 osd.3 (osd.3) 399 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:19:21.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:19.737028+0000 osd.3 (osd.3) 400 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:19:21.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:20.245572+0000 osd.3 (osd.3) 401 : cluster [DBG] 6.1f scrub starts 2024-02-19T21:19:21.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:20.246868+0000 osd.3 (osd.3) 402 : cluster [DBG] 6.1f scrub ok 2024-02-19T21:19:21.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:20 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:20.460744+0000 mon.a (mon.0) 3872 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2024-02-19T21:19:21.834 INFO:teuthology.orchestra.run.smithi163.stdout:55834575097 2024-02-19T21:19:21.834 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-19T21:19:22.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:20.567289+0000 osd.4 (osd.4) 339 : cluster [DBG] 5.18 deep-scrub starts 2024-02-19T21:19:22.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:20.568915+0000 osd.4 (osd.4) 340 : cluster [DBG] 5.18 deep-scrub ok 2024-02-19T21:19:22.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:20.703312+0000 osd.1 (osd.1) 407 : cluster [DBG] 16.9 scrub starts 2024-02-19T21:19:22.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:20.706560+0000 osd.1 (osd.1) 408 : cluster [DBG] 16.9 scrub ok 2024-02-19T21:19:22.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:20.910464+0000 mgr.y (mgr.14550) 721 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:22.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:20.987060+0000 osd.2 (osd.2) 378 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:19:22.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:21 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:20.989036+0000 osd.2 (osd.2) 379 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:19:22.103 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:22.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:20.567289+0000 osd.4 (osd.4) 339 : cluster [DBG] 5.18 deep-scrub starts 2024-02-19T21:19:22.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:20.568915+0000 osd.4 (osd.4) 340 : cluster [DBG] 5.18 deep-scrub ok 2024-02-19T21:19:22.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:20.703312+0000 osd.1 (osd.1) 407 : cluster [DBG] 16.9 scrub starts 2024-02-19T21:19:22.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:20.706560+0000 osd.1 (osd.1) 408 : cluster [DBG] 16.9 scrub ok 2024-02-19T21:19:22.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:20.910464+0000 mgr.y (mgr.14550) 721 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:22.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:20.987060+0000 osd.2 (osd.2) 378 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:19:22.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:20.989036+0000 osd.2 (osd.2) 379 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:19:22.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:20.567289+0000 osd.4 (osd.4) 339 : cluster [DBG] 5.18 deep-scrub starts 2024-02-19T21:19:22.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:20.568915+0000 osd.4 (osd.4) 340 : cluster [DBG] 5.18 deep-scrub ok 2024-02-19T21:19:22.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:20.703312+0000 osd.1 (osd.1) 407 : cluster [DBG] 16.9 scrub starts 2024-02-19T21:19:22.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:20.706560+0000 osd.1 (osd.1) 408 : cluster [DBG] 16.9 scrub ok 2024-02-19T21:19:22.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:20.910464+0000 mgr.y (mgr.14550) 721 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:22.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:20.987060+0000 osd.2 (osd.2) 378 : cluster [DBG] 4.2 scrub starts 2024-02-19T21:19:22.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:21 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:20.989036+0000 osd.2 (osd.2) 379 : cluster [DBG] 4.2 scrub ok 2024-02-19T21:19:23.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:21.370389+0000 mgr.y (mgr.14550) 722 : cluster [DBG] pgmap v1312: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:19:23.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:21.516314+0000 osd.4 (osd.4) 341 : cluster [DBG] 2.f scrub starts 2024-02-19T21:19:23.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:21.517483+0000 osd.4 (osd.4) 342 : cluster [DBG] 2.f scrub ok 2024-02-19T21:19:23.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:21.591627+0000 osd.7 (osd.7) 349 : cluster [DBG] 2.27 scrub starts 2024-02-19T21:19:23.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:21.592558+0000 osd.7 (osd.7) 350 : cluster [DBG] 2.27 scrub ok 2024-02-19T21:19:23.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:21.682711+0000 osd.1 (osd.1) 409 : cluster [DBG] 2.d scrub starts 2024-02-19T21:19:23.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:21.683498+0000 osd.1 (osd.1) 410 : cluster [DBG] 2.d scrub ok 2024-02-19T21:19:23.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:22.278871+0000 osd.3 (osd.3) 403 : cluster [DBG] 16.7 scrub starts 2024-02-19T21:19:23.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:22.280880+0000 osd.3 (osd.3) 404 : cluster [DBG] 16.7 scrub ok 2024-02-19T21:19:23.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:22.666055+0000 osd.5 (osd.5) 372 : cluster [DBG] 16.c deep-scrub starts 2024-02-19T21:19:23.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:22 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:22.669088+0000 osd.5 (osd.5) 373 : cluster [DBG] 16.c deep-scrub ok 2024-02-19T21:19:23.157 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:21.370389+0000 mgr.y (mgr.14550) 722 : cluster [DBG] pgmap v1312: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:19:23.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:21.516314+0000 osd.4 (osd.4) 341 : cluster [DBG] 2.f scrub starts 2024-02-19T21:19:23.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:21.517483+0000 osd.4 (osd.4) 342 : cluster [DBG] 2.f scrub ok 2024-02-19T21:19:23.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:21.591627+0000 osd.7 (osd.7) 349 : cluster [DBG] 2.27 scrub starts 2024-02-19T21:19:23.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:21.592558+0000 osd.7 (osd.7) 350 : cluster [DBG] 2.27 scrub ok 2024-02-19T21:19:23.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:21.682711+0000 osd.1 (osd.1) 409 : cluster [DBG] 2.d scrub starts 2024-02-19T21:19:23.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:21.683498+0000 osd.1 (osd.1) 410 : cluster [DBG] 2.d scrub ok 2024-02-19T21:19:23.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:22.278871+0000 osd.3 (osd.3) 403 : cluster [DBG] 16.7 scrub starts 2024-02-19T21:19:23.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:22.280880+0000 osd.3 (osd.3) 404 : cluster [DBG] 16.7 scrub ok 2024-02-19T21:19:23.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:22.666055+0000 osd.5 (osd.5) 372 : cluster [DBG] 16.c deep-scrub starts 2024-02-19T21:19:23.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:22.669088+0000 osd.5 (osd.5) 373 : cluster [DBG] 16.c deep-scrub ok 2024-02-19T21:19:23.158 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:21.370389+0000 mgr.y (mgr.14550) 722 : cluster [DBG] pgmap v1312: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:19:23.159 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:21.516314+0000 osd.4 (osd.4) 341 : cluster [DBG] 2.f scrub starts 2024-02-19T21:19:23.159 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:21.517483+0000 osd.4 (osd.4) 342 : cluster [DBG] 2.f scrub ok 2024-02-19T21:19:23.159 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:21.591627+0000 osd.7 (osd.7) 349 : cluster [DBG] 2.27 scrub starts 2024-02-19T21:19:23.159 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:21.592558+0000 osd.7 (osd.7) 350 : cluster [DBG] 2.27 scrub ok 2024-02-19T21:19:23.159 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:21.682711+0000 osd.1 (osd.1) 409 : cluster [DBG] 2.d scrub starts 2024-02-19T21:19:23.159 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:21.683498+0000 osd.1 (osd.1) 410 : cluster [DBG] 2.d scrub ok 2024-02-19T21:19:23.159 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:22.278871+0000 osd.3 (osd.3) 403 : cluster [DBG] 16.7 scrub starts 2024-02-19T21:19:23.159 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:22.280880+0000 osd.3 (osd.3) 404 : cluster [DBG] 16.7 scrub ok 2024-02-19T21:19:23.159 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:22.666055+0000 osd.5 (osd.5) 372 : cluster [DBG] 16.c deep-scrub starts 2024-02-19T21:19:23.159 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:22 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:22.669088+0000 osd.5 (osd.5) 373 : cluster [DBG] 16.c deep-scrub ok 2024-02-19T21:19:23.749 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:19:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:19:23] "GET /metrics HTTP/1.1" 200 187978 "" "Prometheus/2.33.4" 2024-02-19T21:19:24.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:22.935229+0000 osd.2 (osd.2) 380 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:19:24.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:22.938129+0000 osd.2 (osd.2) 381 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:19:24.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:23.325911+0000 osd.3 (osd.3) 405 : cluster [DBG] 5.13 scrub starts 2024-02-19T21:19:24.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:23 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:23.326779+0000 osd.3 (osd.3) 406 : cluster [DBG] 5.13 scrub ok 2024-02-19T21:19:24.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.166+0000 7f9e08d69700 1 -- 172.21.15.163:0/956660640 >> v1:172.21.15.163:6790/0 conn(0x7f9e040fd970 legacy=0x7f9e040fdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:24.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.166+0000 7f9e08d69700 1 -- 172.21.15.163:0/956660640 shutdown_connections 2024-02-19T21:19:24.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.166+0000 7f9e08d69700 1 -- 172.21.15.163:0/956660640 >> 172.21.15.163:0/956660640 conn(0x7f9e040f3590 msgr2=0x7f9e040f59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:24.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.166+0000 7f9e08d69700 1 -- 172.21.15.163:0/956660640 shutdown_connections 2024-02-19T21:19:24.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.166+0000 7f9e08d69700 1 -- 172.21.15.163:0/956660640 wait complete. 2024-02-19T21:19:24.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.167+0000 7f9e08d69700 1 librados: starting msgr at 2024-02-19T21:19:24.169 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.167+0000 7f9e08d69700 1 librados: starting objecter 2024-02-19T21:19:24.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.167+0000 7f9e08d69700 1 Processor -- start 2024-02-19T21:19:24.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.167+0000 7f9e08d69700 1 -- start start 2024-02-19T21:19:24.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.167+0000 7f9e08d69700 1 librados: setting wanted keys 2024-02-19T21:19:24.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.167+0000 7f9e08d69700 1 librados: calling monclient init 2024-02-19T21:19:24.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.167+0000 7f9e08d69700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9e04119440 con 0x7f9e040fd970 2024-02-19T21:19:24.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.167+0000 7f9e08d69700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9e0411a5e0 con 0x7f9e04100770 2024-02-19T21:19:24.170 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.167+0000 7f9e08d69700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9e0411b780 con 0x7f9e041042b0 2024-02-19T21:19:24.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.167+0000 7f9e0259c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f9e040fd970 0x7f9e041022e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:53744/0 (socket says 172.21.15.163:53744) 2024-02-19T21:19:24.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9e0259c700 1 -- 172.21.15.163:0/1256008983 learned_addr learned my addr 172.21.15.163:0/1256008983 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:24.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9e02d9d700 1 --1- 172.21.15.163:0/1256008983 >> v1:172.21.15.163:6790/0 conn(0x7f9e041042b0 0x7f9e04117d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:42046/0 (socket says 172.21.15.163:42046) 2024-02-19T21:19:24.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 10 client.?.objecter ms_handle_connect 0x7f9e040fd970 2024-02-19T21:19:24.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:24.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 10 client.?.objecter ms_handle_connect 0x7f9e041042b0 2024-02-19T21:19:24.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:24.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 10 client.?.objecter ms_handle_connect 0x7f9e04100770 2024-02-19T21:19:24.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:24.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059429282 0 0) 0x7f9e04119440 con 0x7f9e040fd970 2024-02-19T21:19:24.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dec003030 con 0x7f9e040fd970 2024-02-19T21:19:24.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924723785 0 0) 0x7f9e0411a5e0 con 0x7f9e04100770 2024-02-19T21:19:24.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e04119440 con 0x7f9e04100770 2024-02-19T21:19:24.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24223039 0 0) 0x7f9e0411b780 con 0x7f9e041042b0 2024-02-19T21:19:24.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e0411a5e0 con 0x7f9e041042b0 2024-02-19T21:19:24.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2462592706 0 0) 0x7f9dec003030 con 0x7f9e040fd970 2024-02-19T21:19:24.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e0411b780 con 0x7f9e040fd970 2024-02-19T21:19:24.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9df8002dc0 con 0x7f9e040fd970 2024-02-19T21:19:24.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1548770790 0 0) 0x7f9e04119440 con 0x7f9e04100770 2024-02-19T21:19:24.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9dec003030 con 0x7f9e04100770 2024-02-19T21:19:24.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9df4001970 con 0x7f9e04100770 2024-02-19T21:19:24.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 724135579 0 0) 0x7f9e0411b780 con 0x7f9e040fd970 2024-02-19T21:19:24.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 >> v1:172.21.15.163:6790/0 conn(0x7f9e041042b0 legacy=0x7f9e04117d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:24.174 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 >> v1:172.21.15.193:6789/0 conn(0x7f9e04100770 legacy=0x7f9e041029f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:24.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0411c960 con 0x7f9e040fd970 2024-02-19T21:19:24.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9e08d69700 1 -- 172.21.15.163:0/1256008983 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0411b970 con 0x7f9e040fd970 2024-02-19T21:19:24.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9e08d69700 10 client.24102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:24.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9e08d69700 1 -- 172.21.15.163:0/1256008983 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0411bde0 con 0x7f9e040fd970 2024-02-19T21:19:24.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9e08d69700 1 librados: init done 2024-02-19T21:19:24.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f9df8001aa0 con 0x7f9e040fd970 2024-02-19T21:19:24.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.168+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f9df8003870 con 0x7f9e040fd970 2024-02-19T21:19:24.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.169+0000 7f9e08d69700 10 client.24102.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:19:24.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.169+0000 7f9e08d69700 15 client.24102.objecter _session_command_op_assign -1 1 2024-02-19T21:19:24.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.169+0000 7f9e08d69700 20 client.24102.objecter _get_session osd=-1 returning homeless 2024-02-19T21:19:24.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.169+0000 7f9e08d69700 10 client.24102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:24.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.169+0000 7f9e08d69700 1 -- 172.21.15.163:0/1256008983 --> v1:172.21.15.163:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9de40028b0 con 0x7f9e040fd970 2024-02-19T21:19:24.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f9df8003ab0 con 0x7f9e040fd970 2024-02-19T21:19:24.175 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(850..850 src has 796..850) v4 ==== 7419+0+0 (unknown 7470691 0 0) 0x7f9df8078d30 con 0x7f9e040fd970 2024-02-19T21:19:24.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 10 client.24102.objecter ms_dispatch 0x7f9e04001550 osd_map(850..850 src has 796..850) v4 2024-02-19T21:19:24.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 3 client.24102.objecter handle_osd_map got epochs [850,850] > 0 2024-02-19T21:19:24.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 3 client.24102.objecter handle_osd_map decoding full epoch 850 2024-02-19T21:19:24.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 10 client.24102.objecter checking command 1 2024-02-19T21:19:24.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 20 client.24102.objecter _get_session s=0x7f9dec06f620 osd=2 3 2024-02-19T21:19:24.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 20 client.24102.objecter put_session s=0x7f9dec06f620 osd=2 3 2024-02-19T21:19:24.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 15 client.24102.objecter _session_command_op_remove -1 1 2024-02-19T21:19:24.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 20 client.24102.objecter _get_session s=0x7f9dec06f620 osd=2 3 2024-02-19T21:19:24.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 20 client.24102.objecter get_session s=0x7f9dec06f620 osd=2 3 2024-02-19T21:19:24.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 15 client.24102.objecter _session_command_op_assign 2 1 2024-02-19T21:19:24.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 20 client.24102.objecter put_session s=0x7f9dec06f620 osd=2 4 2024-02-19T21:19:24.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 10 client.24102.objecter _send_command 1 2024-02-19T21:19:24.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 --> v1:172.21.15.163:6809/1563158487 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9dec0739e0 con 0x7f9dec06fcd0 2024-02-19T21:19:24.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 20 client.24102.objecter dump_active .. 0 homeless 2024-02-19T21:19:24.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_get_version_reply(handle=1 version=850) v2 ==== 24+0+0 (unknown 3804485784 0 0) 0x7f9df80790f0 con 0x7f9e040fd970 2024-02-19T21:19:24.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 10 client.24102.objecter ms_handle_connect 0x7f9dec063990 2024-02-19T21:19:24.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.170+0000 7f9df37fe700 10 client.24102.objecter ms_handle_connect 0x7f9dec06fcd0 2024-02-19T21:19:24.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.171+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 <== osd.2 v1:172.21.15.163:6809/1563158487 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f9dec0739e0 con 0x7f9dec06fcd0 2024-02-19T21:19:24.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.171+0000 7f9df37fe700 10 client.24102.objecter ms_dispatch 0x7f9e04001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:19:24.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.171+0000 7f9df37fe700 10 client.24102.objecter _finish_command 1 = system:0 2024-02-19T21:19:24.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.171+0000 7f9df37fe700 20 client.24102.objecter put_session s=0x7f9dec06f620 osd=2 4 2024-02-19T21:19:24.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.171+0000 7f9df37fe700 15 client.24102.objecter _session_command_op_remove 2 1 2024-02-19T21:19:24.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9e08d69700 10 client.24102.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:19:24.187 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9e08d69700 15 client.24102.objecter _session_command_op_assign -1 2 2024-02-19T21:19:24.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9e08d69700 20 client.24102.objecter _get_session s=0x7f9dec06f620 osd=2 3 2024-02-19T21:19:24.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9e08d69700 20 client.24102.objecter put_session s=0x7f9dec06f620 osd=2 3 2024-02-19T21:19:24.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9e08d69700 20 client.24102.objecter _get_session s=0x7f9dec06f620 osd=2 3 2024-02-19T21:19:24.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9e08d69700 15 client.24102.objecter _session_command_op_remove -1 2 2024-02-19T21:19:24.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9e08d69700 20 client.24102.objecter get_session s=0x7f9dec06f620 osd=2 3 2024-02-19T21:19:24.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9e08d69700 15 client.24102.objecter _session_command_op_assign 2 2 2024-02-19T21:19:24.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9e08d69700 20 client.24102.objecter put_session s=0x7f9dec06f620 osd=2 4 2024-02-19T21:19:24.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9e08d69700 10 client.24102.objecter _send_command 2 2024-02-19T21:19:24.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9e08d69700 1 -- 172.21.15.163:0/1256008983 --> v1:172.21.15.163:6809/1563158487 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9de4003c60 con 0x7f9dec06fcd0 2024-02-19T21:19:24.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9df37fe700 1 -- 172.21.15.163:0/1256008983 <== osd.2 v1:172.21.15.163:6809/1563158487 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 2506265566) 0x7f9de4003c60 con 0x7f9dec06fcd0 2024-02-19T21:19:24.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9df37fe700 10 client.24102.objecter ms_dispatch 0x7f9e04001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:19:24.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9df37fe700 10 client.24102.objecter _finish_command 2 = system:0 2024-02-19T21:19:24.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9df37fe700 20 client.24102.objecter put_session s=0x7f9dec06f620 osd=2 4 2024-02-19T21:19:24.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.185+0000 7f9df37fe700 15 client.24102.objecter _session_command_op_remove 2 2 2024-02-19T21:19:24.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.186+0000 7f9e08d69700 10 librados: watch_flush enter 2024-02-19T21:19:24.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.186+0000 7f9e08d69700 10 librados: watch_flush exit 2024-02-19T21:19:24.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.186+0000 7f9e08d69700 10 client.24102.objecter close_session for osd.2 2024-02-19T21:19:24.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.186+0000 7f9e08d69700 1 -- 172.21.15.163:0/1256008983 >> v1:172.21.15.163:6809/1563158487 conn(0x7f9dec06fcd0 legacy=0x7f9dec072170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:24.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.186+0000 7f9e08d69700 20 client.24102.objecter put_session s=0x7f9dec06f620 osd=2 1 2024-02-19T21:19:24.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.186+0000 7f9e08d69700 20 client.24102.objecter shutdown clearing up homeless session... 2024-02-19T21:19:24.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.186+0000 7f9e08d69700 10 client.24102.objecter successfully canceled tick 2024-02-19T21:19:24.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.186+0000 7f9e08d69700 1 -- 172.21.15.163:0/1256008983 >> v1:172.21.15.163:6800/1552514681 conn(0x7f9dec063990 legacy=0x7f9dec065e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:24.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.186+0000 7f9e08d69700 1 -- 172.21.15.163:0/1256008983 >> v1:172.21.15.163:6789/0 conn(0x7f9e040fd970 legacy=0x7f9e041022e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:24.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.186+0000 7f9e02d9d700 1 -- 172.21.15.163:0/1256008983 reap_dead start 2024-02-19T21:19:24.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.186+0000 7f9e08d69700 1 -- 172.21.15.163:0/1256008983 shutdown_connections 2024-02-19T21:19:24.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.186+0000 7f9e08d69700 1 -- 172.21.15.163:0/1256008983 >> 172.21.15.163:0/1256008983 conn(0x7f9e040f3590 msgr2=0x7f9e04107720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:24.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.186+0000 7f9e08d69700 1 -- 172.21.15.163:0/1256008983 shutdown_connections 2024-02-19T21:19:24.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.186+0000 7f9e08d69700 1 -- 172.21.15.163:0/1256008983 wait complete. 2024-02-19T21:19:24.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:24.186+0000 7f9e08d69700 1 librados: shutdown 2024-02-19T21:19:24.207 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:22.935229+0000 osd.2 (osd.2) 380 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:19:24.207 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:22.938129+0000 osd.2 (osd.2) 381 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:19:24.208 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:23.325911+0000 osd.3 (osd.3) 405 : cluster [DBG] 5.13 scrub starts 2024-02-19T21:19:24.208 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:23.326779+0000 osd.3 (osd.3) 406 : cluster [DBG] 5.13 scrub ok 2024-02-19T21:19:24.208 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:22.935229+0000 osd.2 (osd.2) 380 : cluster [DBG] 2.1 scrub starts 2024-02-19T21:19:24.208 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:22.938129+0000 osd.2 (osd.2) 381 : cluster [DBG] 2.1 scrub ok 2024-02-19T21:19:24.208 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:23.325911+0000 osd.3 (osd.3) 405 : cluster [DBG] 5.13 scrub starts 2024-02-19T21:19:24.208 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:23.326779+0000 osd.3 (osd.3) 406 : cluster [DBG] 5.13 scrub ok 2024-02-19T21:19:24.922 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:23.370761+0000 mgr.y (mgr.14550) 723 : cluster [DBG] pgmap v1313: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:19:24.923 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:24 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:23.370761+0000 mgr.y (mgr.14550) 723 : cluster [DBG] pgmap v1313: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:19:24.941 INFO:teuthology.orchestra.run.smithi163.stdout:77309411572 2024-02-19T21:19:24.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-19T21:19:25.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:24 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:23.370761+0000 mgr.y (mgr.14550) 723 : cluster [DBG] pgmap v1313: 174 pgs: 174 active+clean; 5.2 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-19T21:19:25.212 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:26.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:24.344702+0000 osd.0 (osd.0) 176 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:19:26.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:24.347702+0000 osd.0 (osd.0) 177 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:19:26.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:24.529588+0000 osd.4 (osd.4) 343 : cluster [DBG] 16.19 scrub starts 2024-02-19T21:19:26.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:24.531755+0000 osd.4 (osd.4) 344 : cluster [DBG] 16.19 scrub ok 2024-02-19T21:19:26.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:24.656337+0000 osd.7 (osd.7) 351 : cluster [DBG] 2.20 scrub starts 2024-02-19T21:19:26.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:24.659856+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.20 scrub ok 2024-02-19T21:19:26.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:25.657085+0000 osd.1 (osd.1) 411 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:19:26.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:25 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:25.659975+0000 osd.1 (osd.1) 412 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:19:26.098 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:24.344702+0000 osd.0 (osd.0) 176 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:19:26.098 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:24.347702+0000 osd.0 (osd.0) 177 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:19:26.098 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:24.529588+0000 osd.4 (osd.4) 343 : cluster [DBG] 16.19 scrub starts 2024-02-19T21:19:26.098 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:24.531755+0000 osd.4 (osd.4) 344 : cluster [DBG] 16.19 scrub ok 2024-02-19T21:19:26.099 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:24.656337+0000 osd.7 (osd.7) 351 : cluster [DBG] 2.20 scrub starts 2024-02-19T21:19:26.099 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:24.659856+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.20 scrub ok 2024-02-19T21:19:26.099 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:25.657085+0000 osd.1 (osd.1) 411 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:19:26.099 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:25.659975+0000 osd.1 (osd.1) 412 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:19:26.099 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:24.344702+0000 osd.0 (osd.0) 176 : cluster [DBG] 3.1 scrub starts 2024-02-19T21:19:26.099 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:24.347702+0000 osd.0 (osd.0) 177 : cluster [DBG] 3.1 scrub ok 2024-02-19T21:19:26.099 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:24.529588+0000 osd.4 (osd.4) 343 : cluster [DBG] 16.19 scrub starts 2024-02-19T21:19:26.099 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:24.531755+0000 osd.4 (osd.4) 344 : cluster [DBG] 16.19 scrub ok 2024-02-19T21:19:26.099 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:24.656337+0000 osd.7 (osd.7) 351 : cluster [DBG] 2.20 scrub starts 2024-02-19T21:19:26.099 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:24.659856+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.20 scrub ok 2024-02-19T21:19:26.099 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:25.657085+0000 osd.1 (osd.1) 411 : cluster [DBG] 2.9 scrub starts 2024-02-19T21:19:26.100 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:25 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:25.659975+0000 osd.1 (osd.1) 412 : cluster [DBG] 2.9 scrub ok 2024-02-19T21:19:27.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.004+0000 7f5db1de9700 1 -- 172.21.15.163:0/3108507483 >> v1:172.21.15.163:6790/0 conn(0x7f5dac0ff670 legacy=0x7f5dac101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:27.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.004+0000 7f5db1de9700 1 -- 172.21.15.163:0/3108507483 shutdown_connections 2024-02-19T21:19:27.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.004+0000 7f5db1de9700 1 -- 172.21.15.163:0/3108507483 >> 172.21.15.163:0/3108507483 conn(0x7f5dac0f2470 msgr2=0x7f5dac0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:27.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.004+0000 7f5db1de9700 1 -- 172.21.15.163:0/3108507483 shutdown_connections 2024-02-19T21:19:27.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.004+0000 7f5db1de9700 1 -- 172.21.15.163:0/3108507483 wait complete. 2024-02-19T21:19:27.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.005+0000 7f5db1de9700 1 librados: starting msgr at 2024-02-19T21:19:27.006 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.005+0000 7f5db1de9700 1 librados: starting objecter 2024-02-19T21:19:27.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.005+0000 7f5db1de9700 1 Processor -- start 2024-02-19T21:19:27.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.005+0000 7f5db1de9700 1 -- start start 2024-02-19T21:19:27.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.005+0000 7f5db1de9700 1 librados: setting wanted keys 2024-02-19T21:19:27.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.005+0000 7f5db1de9700 1 librados: calling monclient init 2024-02-19T21:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5db1de9700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5dac0fbf00 con 0x7f5dac0ff670 2024-02-19T21:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5db1de9700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5dac0fc0d0 con 0x7f5dac1031b0 2024-02-19T21:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5db1de9700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5dac11ba80 con 0x7f5dac0fc870 2024-02-19T21:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5daaffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f5dac0ff670 0x7f5dac114c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44930/0 (socket says 172.21.15.163:44930) 2024-02-19T21:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5daaffd700 1 -- 172.21.15.163:0/3563560710 learned_addr learned my addr 172.21.15.163:0/3563560710 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5dab7fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f5dac0fc870 0x7f5dac0fb760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:54160/0 (socket says 172.21.15.163:54160) 2024-02-19T21:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 10 client.?.objecter ms_handle_connect 0x7f5dac0fc870 2024-02-19T21:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 10 client.?.objecter ms_handle_connect 0x7f5dac0ff670 2024-02-19T21:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 10 client.?.objecter ms_handle_connect 0x7f5dac1031b0 2024-02-19T21:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055503698 0 0) 0x7f5dac11ba80 con 0x7f5dac0fc870 2024-02-19T21:19:27.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d940038e0 con 0x7f5dac0fc870 2024-02-19T21:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272828512 0 0) 0x7f5dac0fbf00 con 0x7f5dac0ff670 2024-02-19T21:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dac11ba80 con 0x7f5dac0ff670 2024-02-19T21:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901410711 0 0) 0x7f5dac0fc0d0 con 0x7f5dac1031b0 2024-02-19T21:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dac0fbf00 con 0x7f5dac1031b0 2024-02-19T21:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3875481931 0 0) 0x7f5d940038e0 con 0x7f5dac0fc870 2024-02-19T21:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dac0fc0d0 con 0x7f5dac0fc870 2024-02-19T21:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5d9c0018c0 con 0x7f5dac0fc870 2024-02-19T21:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1552817039 0 0) 0x7f5dac11ba80 con 0x7f5dac0ff670 2024-02-19T21:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d940038e0 con 0x7f5dac0ff670 2024-02-19T21:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5da0002ce0 con 0x7f5dac0ff670 2024-02-19T21:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1245155975 0 0) 0x7f5dac0fc0d0 con 0x7f5dac0fc870 2024-02-19T21:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 >> v1:172.21.15.193:6789/0 conn(0x7f5dac1031b0 legacy=0x7f5dac118340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 >> v1:172.21.15.163:6789/0 conn(0x7f5dac0ff670 legacy=0x7f5dac114c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:27.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dac11cc20 con 0x7f5dac0fc870 2024-02-19T21:19:27.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.006+0000 7f5db1de9700 1 -- 172.21.15.163:0/3563560710 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dac11bc10 con 0x7f5dac0fc870 2024-02-19T21:19:27.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.007+0000 7f5db1de9700 10 client.50834.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:27.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.007+0000 7f5db1de9700 1 -- 172.21.15.163:0/3563560710 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dac11c080 con 0x7f5dac0fc870 2024-02-19T21:19:27.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.007+0000 7f5db1de9700 1 librados: init done 2024-02-19T21:19:27.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.007+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f5d9c002350 con 0x7f5dac0fc870 2024-02-19T21:19:27.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.007+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f5d9c005a70 con 0x7f5dac0fc870 2024-02-19T21:19:27.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.007+0000 7f5db1de9700 10 client.50834.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:19:27.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.007+0000 7f5db1de9700 15 client.50834.objecter _session_command_op_assign -1 1 2024-02-19T21:19:27.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.007+0000 7f5db1de9700 20 client.50834.objecter _get_session osd=-1 returning homeless 2024-02-19T21:19:27.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.007+0000 7f5db1de9700 10 client.50834.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:27.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.007+0000 7f5db1de9700 1 -- 172.21.15.163:0/3563560710 --> v1:172.21.15.163:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5d980028b0 con 0x7f5dac0fc870 2024-02-19T21:19:27.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f5d9c002af0 con 0x7f5dac0fc870 2024-02-19T21:19:27.012 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(850..850 src has 796..850) v4 ==== 7419+0+0 (unknown 7470691 0 0) 0x7f5d9c078900 con 0x7f5dac0fc870 2024-02-19T21:19:27.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 10 client.50834.objecter ms_dispatch 0x7f5dac001550 osd_map(850..850 src has 796..850) v4 2024-02-19T21:19:27.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 3 client.50834.objecter handle_osd_map got epochs [850,850] > 0 2024-02-19T21:19:27.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 3 client.50834.objecter handle_osd_map decoding full epoch 850 2024-02-19T21:19:27.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 10 client.50834.objecter checking command 1 2024-02-19T21:19:27.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 20 client.50834.objecter _get_session s=0x7f5d9406f830 osd=3 3 2024-02-19T21:19:27.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 20 client.50834.objecter put_session s=0x7f5d9406f830 osd=3 3 2024-02-19T21:19:27.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 15 client.50834.objecter _session_command_op_remove -1 1 2024-02-19T21:19:27.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 20 client.50834.objecter _get_session s=0x7f5d9406f830 osd=3 3 2024-02-19T21:19:27.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 20 client.50834.objecter get_session s=0x7f5d9406f830 osd=3 3 2024-02-19T21:19:27.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 15 client.50834.objecter _session_command_op_assign 3 1 2024-02-19T21:19:27.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 20 client.50834.objecter put_session s=0x7f5d9406f830 osd=3 4 2024-02-19T21:19:27.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 10 client.50834.objecter _send_command 1 2024-02-19T21:19:27.013 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 --> v1:172.21.15.163:6813/2112279769 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d94073bf0 con 0x7f5d9406fee0 2024-02-19T21:19:27.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 20 client.50834.objecter dump_active .. 0 homeless 2024-02-19T21:19:27.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.008+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_get_version_reply(handle=1 version=850) v2 ==== 24+0+0 (unknown 3804485784 0 0) 0x7f5d9c078d00 con 0x7f5dac0fc870 2024-02-19T21:19:27.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.009+0000 7f5da8ff9700 10 client.50834.objecter ms_handle_connect 0x7f5d94063bb0 2024-02-19T21:19:27.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.009+0000 7f5da8ff9700 10 client.50834.objecter ms_handle_connect 0x7f5d9406fee0 2024-02-19T21:19:27.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.009+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 <== osd.3 v1:172.21.15.163:6813/2112279769 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f5d94073bf0 con 0x7f5d9406fee0 2024-02-19T21:19:27.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.009+0000 7f5da8ff9700 10 client.50834.objecter ms_dispatch 0x7f5dac001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:19:27.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.009+0000 7f5da8ff9700 10 client.50834.objecter _finish_command 1 = system:0 2024-02-19T21:19:27.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.009+0000 7f5da8ff9700 20 client.50834.objecter put_session s=0x7f5d9406f830 osd=3 4 2024-02-19T21:19:27.014 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.009+0000 7f5da8ff9700 15 client.50834.objecter _session_command_op_remove 3 1 2024-02-19T21:19:27.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5db1de9700 10 client.50834.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:19:27.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5db1de9700 15 client.50834.objecter _session_command_op_assign -1 2 2024-02-19T21:19:27.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5db1de9700 20 client.50834.objecter _get_session s=0x7f5d9406f830 osd=3 3 2024-02-19T21:19:27.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5db1de9700 20 client.50834.objecter put_session s=0x7f5d9406f830 osd=3 3 2024-02-19T21:19:27.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5db1de9700 20 client.50834.objecter _get_session s=0x7f5d9406f830 osd=3 3 2024-02-19T21:19:27.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5db1de9700 15 client.50834.objecter _session_command_op_remove -1 2 2024-02-19T21:19:27.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5db1de9700 20 client.50834.objecter get_session s=0x7f5d9406f830 osd=3 3 2024-02-19T21:19:27.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5db1de9700 15 client.50834.objecter _session_command_op_assign 3 2 2024-02-19T21:19:27.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5db1de9700 20 client.50834.objecter put_session s=0x7f5d9406f830 osd=3 4 2024-02-19T21:19:27.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5db1de9700 10 client.50834.objecter _send_command 2 2024-02-19T21:19:27.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5db1de9700 1 -- 172.21.15.163:0/3563560710 --> v1:172.21.15.163:6813/2112279769 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5d98003c60 con 0x7f5d9406fee0 2024-02-19T21:19:27.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5da8ff9700 1 -- 172.21.15.163:0/3563560710 <== osd.3 v1:172.21.15.163:6813/2112279769 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 328681653) 0x7f5d98003c60 con 0x7f5d9406fee0 2024-02-19T21:19:27.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5da8ff9700 10 client.50834.objecter ms_dispatch 0x7f5dac001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:19:27.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5da8ff9700 10 client.50834.objecter _finish_command 2 = system:0 2024-02-19T21:19:27.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5da8ff9700 20 client.50834.objecter put_session s=0x7f5d9406f830 osd=3 4 2024-02-19T21:19:27.029 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.023+0000 7f5da8ff9700 15 client.50834.objecter _session_command_op_remove 3 2 2024-02-19T21:19:27.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.024+0000 7f5db1de9700 10 librados: watch_flush enter 2024-02-19T21:19:27.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.024+0000 7f5db1de9700 10 librados: watch_flush exit 2024-02-19T21:19:27.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.024+0000 7f5db1de9700 10 client.50834.objecter close_session for osd.3 2024-02-19T21:19:27.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.024+0000 7f5db1de9700 1 -- 172.21.15.163:0/3563560710 >> v1:172.21.15.163:6813/2112279769 conn(0x7f5d9406fee0 legacy=0x7f5d94072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:27.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.024+0000 7f5db1de9700 20 client.50834.objecter put_session s=0x7f5d9406f830 osd=3 1 2024-02-19T21:19:27.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.024+0000 7f5db1de9700 20 client.50834.objecter shutdown clearing up homeless session... 2024-02-19T21:19:27.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.024+0000 7f5db1de9700 10 client.50834.objecter successfully canceled tick 2024-02-19T21:19:27.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.024+0000 7f5db1de9700 1 -- 172.21.15.163:0/3563560710 >> v1:172.21.15.163:6800/1552514681 conn(0x7f5d94063bb0 legacy=0x7f5d94066070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:27.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.024+0000 7f5db1de9700 1 -- 172.21.15.163:0/3563560710 >> v1:172.21.15.163:6790/0 conn(0x7f5dac0fc870 legacy=0x7f5dac0fb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:27.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.024+0000 7f5dabfff700 1 -- 172.21.15.163:0/3563560710 reap_dead start 2024-02-19T21:19:27.030 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.024+0000 7f5db1de9700 1 -- 172.21.15.163:0/3563560710 shutdown_connections 2024-02-19T21:19:27.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.024+0000 7f5db1de9700 1 -- 172.21.15.163:0/3563560710 >> 172.21.15.163:0/3563560710 conn(0x7f5dac0f2470 msgr2=0x7f5dac106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:27.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.024+0000 7f5db1de9700 1 -- 172.21.15.163:0/3563560710 shutdown_connections 2024-02-19T21:19:27.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.024+0000 7f5db1de9700 1 -- 172.21.15.163:0/3563560710 wait complete. 2024-02-19T21:19:27.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:27.024+0000 7f5db1de9700 1 librados: shutdown 2024-02-19T21:19:27.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:25.371663+0000 mgr.y (mgr.14550) 724 : cluster [DBG] pgmap v1314: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:19:27.158 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:25.626620+0000 osd.7 (osd.7) 353 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:19:27.159 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:25.629579+0000 osd.7 (osd.7) 354 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:19:27.159 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:25.996706+0000 osd.2 (osd.2) 382 : cluster [DBG] 2.21 scrub starts 2024-02-19T21:19:27.159 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:25.999837+0000 osd.2 (osd.2) 383 : cluster [DBG] 2.21 scrub ok 2024-02-19T21:19:27.159 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:26.627270+0000 osd.5 (osd.5) 374 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:19:27.159 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:26.630288+0000 osd.5 (osd.5) 375 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:19:27.159 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:25.371663+0000 mgr.y (mgr.14550) 724 : cluster [DBG] pgmap v1314: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:19:27.159 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:25.626620+0000 osd.7 (osd.7) 353 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:19:27.160 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:25.629579+0000 osd.7 (osd.7) 354 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:19:27.160 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:25.996706+0000 osd.2 (osd.2) 382 : cluster [DBG] 2.21 scrub starts 2024-02-19T21:19:27.160 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:25.999837+0000 osd.2 (osd.2) 383 : cluster [DBG] 2.21 scrub ok 2024-02-19T21:19:27.160 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:26.627270+0000 osd.5 (osd.5) 374 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:19:27.160 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:26.630288+0000 osd.5 (osd.5) 375 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:19:27.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:25.371663+0000 mgr.y (mgr.14550) 724 : cluster [DBG] pgmap v1314: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:19:27.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:25.626620+0000 osd.7 (osd.7) 353 : cluster [DBG] 1.0 scrub starts 2024-02-19T21:19:27.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:25.629579+0000 osd.7 (osd.7) 354 : cluster [DBG] 1.0 scrub ok 2024-02-19T21:19:27.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:25.996706+0000 osd.2 (osd.2) 382 : cluster [DBG] 2.21 scrub starts 2024-02-19T21:19:27.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:25.999837+0000 osd.2 (osd.2) 383 : cluster [DBG] 2.21 scrub ok 2024-02-19T21:19:27.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:26.627270+0000 osd.5 (osd.5) 374 : cluster [DBG] 6.19 scrub starts 2024-02-19T21:19:27.161 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:26 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:26.630288+0000 osd.5 (osd.5) 375 : cluster [DBG] 6.19 scrub ok 2024-02-19T21:19:27.580 INFO:teuthology.orchestra.run.smithi163.stdout:107374182641 2024-02-19T21:19:27.581 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-19T21:19:27.849 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:27.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:26.353924+0000 osd.0 (osd.0) 178 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:19:27.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:26.355816+0000 osd.0 (osd.0) 179 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:19:27.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:26.673041+0000 osd.1 (osd.1) 413 : cluster [DBG] 16.13 deep-scrub starts 2024-02-19T21:19:27.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:26.677157+0000 osd.1 (osd.1) 414 : cluster [DBG] 16.13 deep-scrub ok 2024-02-19T21:19:27.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:27.578676+0000 osd.5 (osd.5) 376 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:19:28.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:27.581562+0000 osd.5 (osd.5) 377 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:19:28.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:26.353924+0000 osd.0 (osd.0) 178 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:19:28.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:26.355816+0000 osd.0 (osd.0) 179 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:19:28.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:26.673041+0000 osd.1 (osd.1) 413 : cluster [DBG] 16.13 deep-scrub starts 2024-02-19T21:19:28.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:26.677157+0000 osd.1 (osd.1) 414 : cluster [DBG] 16.13 deep-scrub ok 2024-02-19T21:19:28.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:27.578676+0000 osd.5 (osd.5) 376 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:19:28.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:27 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:27.581562+0000 osd.5 (osd.5) 377 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:19:28.073 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:19:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:19:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:19:28.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:26.353924+0000 osd.0 (osd.0) 178 : cluster [DBG] 4.1e scrub starts 2024-02-19T21:19:28.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:26.355816+0000 osd.0 (osd.0) 179 : cluster [DBG] 4.1e scrub ok 2024-02-19T21:19:28.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:26.673041+0000 osd.1 (osd.1) 413 : cluster [DBG] 16.13 deep-scrub starts 2024-02-19T21:19:28.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:26.677157+0000 osd.1 (osd.1) 414 : cluster [DBG] 16.13 deep-scrub ok 2024-02-19T21:19:28.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:27.578676+0000 osd.5 (osd.5) 376 : cluster [DBG] 2.12 scrub starts 2024-02-19T21:19:28.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:27 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:27.581562+0000 osd.5 (osd.5) 377 : cluster [DBG] 2.12 scrub ok 2024-02-19T21:19:28.962 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:27.372221+0000 mgr.y (mgr.14550) 725 : cluster [DBG] pgmap v1315: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:19:28.962 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:28.211212+0000 osd.3 (osd.3) 407 : cluster [DBG] 16.1a scrub starts 2024-02-19T21:19:28.962 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:28.212487+0000 osd.3 (osd.3) 408 : cluster [DBG] 16.1a scrub ok 2024-02-19T21:19:28.963 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:27.372221+0000 mgr.y (mgr.14550) 725 : cluster [DBG] pgmap v1315: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:19:28.963 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:28.211212+0000 osd.3 (osd.3) 407 : cluster [DBG] 16.1a scrub starts 2024-02-19T21:19:28.963 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:28 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:28.212487+0000 osd.3 (osd.3) 408 : cluster [DBG] 16.1a scrub ok 2024-02-19T21:19:29.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:27.372221+0000 mgr.y (mgr.14550) 725 : cluster [DBG] pgmap v1315: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:19:29.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:28.211212+0000 osd.3 (osd.3) 407 : cluster [DBG] 16.1a scrub starts 2024-02-19T21:19:29.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:28 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:28.212487+0000 osd.3 (osd.3) 408 : cluster [DBG] 16.1a scrub ok 2024-02-19T21:19:29.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.846+0000 7f2358b30700 1 -- 172.21.15.163:0/940038985 >> v1:172.21.15.163:6790/0 conn(0x7f2354100780 legacy=0x7f2354102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:29.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.846+0000 7f2358b30700 1 -- 172.21.15.163:0/940038985 shutdown_connections 2024-02-19T21:19:29.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.846+0000 7f2358b30700 1 -- 172.21.15.163:0/940038985 >> 172.21.15.163:0/940038985 conn(0x7f23540f35a0 msgr2=0x7f23540f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:29.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.846+0000 7f2358b30700 1 -- 172.21.15.163:0/940038985 shutdown_connections 2024-02-19T21:19:29.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.846+0000 7f2358b30700 1 -- 172.21.15.163:0/940038985 wait complete. 2024-02-19T21:19:29.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.846+0000 7f2358b30700 1 librados: starting msgr at 2024-02-19T21:19:29.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.846+0000 7f2358b30700 1 librados: starting objecter 2024-02-19T21:19:29.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f2358b30700 1 Processor -- start 2024-02-19T21:19:29.850 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f2358b30700 1 -- start start 2024-02-19T21:19:29.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f2358b30700 1 librados: setting wanted keys 2024-02-19T21:19:29.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f2358b30700 1 librados: calling monclient init 2024-02-19T21:19:29.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f2358b30700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f23540fcee0 con 0x7f23540fd980 2024-02-19T21:19:29.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f2358b30700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f23540fd0d0 con 0x7f2354100780 2024-02-19T21:19:29.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f2358b30700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f23540fd2c0 con 0x7f23541042c0 2024-02-19T21:19:29.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f235259c700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f23540fd980 0x7f23540fc7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44938/0 (socket says 172.21.15.163:44938) 2024-02-19T21:19:29.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f2352d9d700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f23541042c0 0x7f2354118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:54176/0 (socket says 172.21.15.163:54176) 2024-02-19T21:19:29.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f235259c700 1 -- 172.21.15.163:0/3394664392 learned_addr learned my addr 172.21.15.163:0/3394664392 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:29.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f23437fe700 10 client.?.objecter ms_handle_connect 0x7f23541042c0 2024-02-19T21:19:29.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f23437fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:29.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f23437fe700 10 client.?.objecter ms_handle_connect 0x7f23540fd980 2024-02-19T21:19:29.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f23437fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:29.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f23437fe700 10 client.?.objecter ms_handle_connect 0x7f2354100780 2024-02-19T21:19:29.851 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f23437fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:29.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336969347 0 0) 0x7f23540fd2c0 con 0x7f23541042c0 2024-02-19T21:19:29.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f233c003030 con 0x7f23541042c0 2024-02-19T21:19:29.852 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292130053 0 0) 0x7f23540fcee0 con 0x7f23540fd980 2024-02-19T21:19:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.847+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23540fd2c0 con 0x7f23540fd980 2024-02-19T21:19:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477437904 0 0) 0x7f23540fd0d0 con 0x7f2354100780 2024-02-19T21:19:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23540fcee0 con 0x7f2354100780 2024-02-19T21:19:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1405803330 0 0) 0x7f23540fd2c0 con 0x7f23540fd980 2024-02-19T21:19:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23540fd0d0 con 0x7f23540fd980 2024-02-19T21:19:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f234c0019f0 con 0x7f23540fd980 2024-02-19T21:19:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1012133859 0 0) 0x7f233c003030 con 0x7f23541042c0 2024-02-19T21:19:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23540fd2c0 con 0x7f23541042c0 2024-02-19T21:19:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2344001b40 con 0x7f23541042c0 2024-02-19T21:19:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3027929831 0 0) 0x7f23540fcee0 con 0x7f2354100780 2024-02-19T21:19:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f233c003030 con 0x7f2354100780 2024-02-19T21:19:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f2348001430 con 0x7f2354100780 2024-02-19T21:19:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1343694738 0 0) 0x7f23540fd0d0 con 0x7f23540fd980 2024-02-19T21:19:29.853 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 >> v1:172.21.15.163:6790/0 conn(0x7f23541042c0 legacy=0x7f2354118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 >> v1:172.21.15.193:6789/0 conn(0x7f2354100780 legacy=0x7f2354114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235411cb90 con 0x7f23540fd980 2024-02-19T21:19:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f2358b30700 1 -- 172.21.15.163:0/3394664392 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2354119bc0 con 0x7f23540fd980 2024-02-19T21:19:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f2358b30700 10 client.24114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f2358b30700 1 -- 172.21.15.163:0/3394664392 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f235411a0b0 con 0x7f23540fd980 2024-02-19T21:19:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f2358b30700 1 librados: init done 2024-02-19T21:19:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f234c002520 con 0x7f23540fd980 2024-02-19T21:19:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.848+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f234c005bb0 con 0x7f23540fd980 2024-02-19T21:19:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.849+0000 7f2358b30700 10 client.24114.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:19:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.849+0000 7f2358b30700 15 client.24114.objecter _session_command_op_assign -1 1 2024-02-19T21:19:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.849+0000 7f2358b30700 20 client.24114.objecter _get_session osd=-1 returning homeless 2024-02-19T21:19:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.849+0000 7f2358b30700 10 client.24114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:29.854 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.849+0000 7f2358b30700 1 -- 172.21.15.163:0/3394664392 --> v1:172.21.15.163:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f23340028b0 con 0x7f23540fd980 2024-02-19T21:19:29.855 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.849+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f234c018b00 con 0x7f23540fd980 2024-02-19T21:19:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(850..850 src has 796..850) v4 ==== 7419+0+0 (unknown 7470691 0 0) 0x7f234c078b50 con 0x7f23540fd980 2024-02-19T21:19:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 10 client.24114.objecter ms_dispatch 0x7f2354001550 osd_map(850..850 src has 796..850) v4 2024-02-19T21:19:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 3 client.24114.objecter handle_osd_map got epochs [850,850] > 0 2024-02-19T21:19:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 3 client.24114.objecter handle_osd_map decoding full epoch 850 2024-02-19T21:19:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 10 client.24114.objecter checking command 1 2024-02-19T21:19:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 20 client.24114.objecter _get_session s=0x7f233c06f950 osd=4 3 2024-02-19T21:19:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 20 client.24114.objecter put_session s=0x7f233c06f950 osd=4 3 2024-02-19T21:19:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 15 client.24114.objecter _session_command_op_remove -1 1 2024-02-19T21:19:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 20 client.24114.objecter _get_session s=0x7f233c06f950 osd=4 3 2024-02-19T21:19:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 20 client.24114.objecter get_session s=0x7f233c06f950 osd=4 3 2024-02-19T21:19:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 15 client.24114.objecter _session_command_op_assign 4 1 2024-02-19T21:19:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 20 client.24114.objecter put_session s=0x7f233c06f950 osd=4 4 2024-02-19T21:19:29.856 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 10 client.24114.objecter _send_command 1 2024-02-19T21:19:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 --> v1:172.21.15.193:6800/3129716965 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f233c073c70 con 0x7f233c070000 2024-02-19T21:19:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 20 client.24114.objecter dump_active .. 0 homeless 2024-02-19T21:19:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_get_version_reply(handle=1 version=850) v2 ==== 24+0+0 (unknown 3804485784 0 0) 0x7f234c078f60 con 0x7f23540fd980 2024-02-19T21:19:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 10 client.24114.objecter ms_handle_connect 0x7f233c063c20 2024-02-19T21:19:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.850+0000 7f23437fe700 10 client.24114.objecter ms_handle_connect 0x7f233c070000 2024-02-19T21:19:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.851+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== osd.4 v1:172.21.15.193:6800/3129716965 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f233c073c70 con 0x7f233c070000 2024-02-19T21:19:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.851+0000 7f23437fe700 10 client.24114.objecter ms_dispatch 0x7f2354001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:19:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.851+0000 7f23437fe700 10 client.24114.objecter _finish_command 1 = system:0 2024-02-19T21:19:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.851+0000 7f23437fe700 20 client.24114.objecter put_session s=0x7f233c06f950 osd=4 4 2024-02-19T21:19:29.857 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.851+0000 7f23437fe700 15 client.24114.objecter _session_command_op_remove 4 1 2024-02-19T21:19:29.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.865+0000 7f2358b30700 10 client.24114.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:19:29.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.865+0000 7f2358b30700 15 client.24114.objecter _session_command_op_assign -1 2 2024-02-19T21:19:29.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.865+0000 7f2358b30700 20 client.24114.objecter _get_session s=0x7f233c06f950 osd=4 3 2024-02-19T21:19:29.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.865+0000 7f2358b30700 20 client.24114.objecter put_session s=0x7f233c06f950 osd=4 3 2024-02-19T21:19:29.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.865+0000 7f2358b30700 20 client.24114.objecter _get_session s=0x7f233c06f950 osd=4 3 2024-02-19T21:19:29.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.865+0000 7f2358b30700 15 client.24114.objecter _session_command_op_remove -1 2 2024-02-19T21:19:29.866 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.865+0000 7f2358b30700 20 client.24114.objecter get_session s=0x7f233c06f950 osd=4 3 2024-02-19T21:19:29.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.865+0000 7f2358b30700 15 client.24114.objecter _session_command_op_assign 4 2 2024-02-19T21:19:29.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.865+0000 7f2358b30700 20 client.24114.objecter put_session s=0x7f233c06f950 osd=4 4 2024-02-19T21:19:29.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.865+0000 7f2358b30700 10 client.24114.objecter _send_command 2 2024-02-19T21:19:29.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.865+0000 7f2358b30700 1 -- 172.21.15.163:0/3394664392 --> v1:172.21.15.193:6800/3129716965 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2334001140 con 0x7f233c070000 2024-02-19T21:19:29.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.866+0000 7f23437fe700 1 -- 172.21.15.163:0/3394664392 <== osd.4 v1:172.21.15.193:6800/3129716965 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 1340886204) 0x7f2334001140 con 0x7f233c070000 2024-02-19T21:19:29.867 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.866+0000 7f23437fe700 10 client.24114.objecter ms_dispatch 0x7f2354001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:19:29.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.866+0000 7f23437fe700 10 client.24114.objecter _finish_command 2 = system:0 2024-02-19T21:19:29.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.866+0000 7f23437fe700 20 client.24114.objecter put_session s=0x7f233c06f950 osd=4 4 2024-02-19T21:19:29.868 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.866+0000 7f23437fe700 15 client.24114.objecter _session_command_op_remove 4 2 2024-02-19T21:19:29.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.866+0000 7f2358b30700 10 librados: watch_flush enter 2024-02-19T21:19:29.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.866+0000 7f2358b30700 10 librados: watch_flush exit 2024-02-19T21:19:29.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.866+0000 7f2358b30700 10 client.24114.objecter close_session for osd.4 2024-02-19T21:19:29.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.866+0000 7f2358b30700 1 -- 172.21.15.163:0/3394664392 >> v1:172.21.15.193:6800/3129716965 conn(0x7f233c070000 legacy=0x7f233c0724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:29.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.866+0000 7f2358b30700 20 client.24114.objecter put_session s=0x7f233c06f950 osd=4 1 2024-02-19T21:19:29.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.866+0000 7f2358b30700 20 client.24114.objecter shutdown clearing up homeless session... 2024-02-19T21:19:29.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.866+0000 7f2358b30700 10 client.24114.objecter successfully canceled tick 2024-02-19T21:19:29.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.866+0000 7f2358b30700 1 -- 172.21.15.163:0/3394664392 >> v1:172.21.15.163:6800/1552514681 conn(0x7f233c063c20 legacy=0x7f233c0660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:29.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.867+0000 7f2358b30700 1 -- 172.21.15.163:0/3394664392 >> v1:172.21.15.163:6789/0 conn(0x7f23540fd980 legacy=0x7f23540fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:29.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.867+0000 7f2352d9d700 1 -- 172.21.15.163:0/3394664392 reap_dead start 2024-02-19T21:19:29.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.867+0000 7f2358b30700 1 -- 172.21.15.163:0/3394664392 shutdown_connections 2024-02-19T21:19:29.869 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.867+0000 7f2358b30700 1 -- 172.21.15.163:0/3394664392 >> 172.21.15.163:0/3394664392 conn(0x7f23540f35a0 msgr2=0x7f2354107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:29.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.867+0000 7f2358b30700 1 -- 172.21.15.163:0/3394664392 shutdown_connections 2024-02-19T21:19:29.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.867+0000 7f2358b30700 1 -- 172.21.15.163:0/3394664392 wait complete. 2024-02-19T21:19:29.870 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:29.867+0000 7f2358b30700 1 librados: shutdown 2024-02-19T21:19:29.888 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:29.207339+0000 osd.3 (osd.3) 409 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:19:29.888 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:29.208396+0000 osd.3 (osd.3) 410 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:19:29.888 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:29.207339+0000 osd.3 (osd.3) 409 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:19:29.888 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:29 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:29.208396+0000 osd.3 (osd.3) 410 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:19:30.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:29.207339+0000 osd.3 (osd.3) 409 : cluster [DBG] 2.19 scrub starts 2024-02-19T21:19:30.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:29 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:29.208396+0000 osd.3 (osd.3) 410 : cluster [DBG] 2.19 scrub ok 2024-02-19T21:19:30.570 INFO:teuthology.orchestra.run.smithi163.stdout:133143986413 2024-02-19T21:19:30.571 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-19T21:19:30.802 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:29.285610+0000 osd.0 (osd.0) 180 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:19:30.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:29.288493+0000 osd.0 (osd.0) 181 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:19:30.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:29.372730+0000 mgr.y (mgr.14550) 726 : cluster [DBG] pgmap v1316: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:19:30.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:30.190762+0000 osd.3 (osd.3) 411 : cluster [DBG] 4.14 deep-scrub starts 2024-02-19T21:19:30.803 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:30.192702+0000 osd.3 (osd.3) 412 : cluster [DBG] 4.14 deep-scrub ok 2024-02-19T21:19:30.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:29.285610+0000 osd.0 (osd.0) 180 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:19:30.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:29.288493+0000 osd.0 (osd.0) 181 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:19:30.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:29.372730+0000 mgr.y (mgr.14550) 726 : cluster [DBG] pgmap v1316: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:19:30.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:30.190762+0000 osd.3 (osd.3) 411 : cluster [DBG] 4.14 deep-scrub starts 2024-02-19T21:19:30.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:30.192702+0000 osd.3 (osd.3) 412 : cluster [DBG] 4.14 deep-scrub ok 2024-02-19T21:19:30.845 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:31.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:19:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:19:31.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:29.285610+0000 osd.0 (osd.0) 180 : cluster [DBG] 16.8 scrub starts 2024-02-19T21:19:31.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:29.288493+0000 osd.0 (osd.0) 181 : cluster [DBG] 16.8 scrub ok 2024-02-19T21:19:31.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:29.372730+0000 mgr.y (mgr.14550) 726 : cluster [DBG] pgmap v1316: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:19:31.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:30.190762+0000 osd.3 (osd.3) 411 : cluster [DBG] 4.14 deep-scrub starts 2024-02-19T21:19:31.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:30.192702+0000 osd.3 (osd.3) 412 : cluster [DBG] 4.14 deep-scrub ok 2024-02-19T21:19:31.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:30.437596+0000 osd.4 (osd.4) 345 : cluster [DBG] 4.d scrub starts 2024-02-19T21:19:31.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:30.439405+0000 osd.4 (osd.4) 346 : cluster [DBG] 4.d scrub ok 2024-02-19T21:19:31.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:30.707819+0000 osd.7 (osd.7) 355 : cluster [DBG] 2.b scrub starts 2024-02-19T21:19:31.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:30.711289+0000 osd.7 (osd.7) 356 : cluster [DBG] 2.b scrub ok 2024-02-19T21:19:32.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:30.750908+0000 osd.1 (osd.1) 415 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:19:32.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:30.754376+0000 osd.1 (osd.1) 416 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:19:32.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:30.918565+0000 mgr.y (mgr.14550) 727 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:32.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:30.437596+0000 osd.4 (osd.4) 345 : cluster [DBG] 4.d scrub starts 2024-02-19T21:19:32.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:30.439405+0000 osd.4 (osd.4) 346 : cluster [DBG] 4.d scrub ok 2024-02-19T21:19:32.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:30.707819+0000 osd.7 (osd.7) 355 : cluster [DBG] 2.b scrub starts 2024-02-19T21:19:32.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:30.711289+0000 osd.7 (osd.7) 356 : cluster [DBG] 2.b scrub ok 2024-02-19T21:19:32.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:30.750908+0000 osd.1 (osd.1) 415 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:19:32.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:30.754376+0000 osd.1 (osd.1) 416 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:19:32.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:30.918565+0000 mgr.y (mgr.14550) 727 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:32.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:30.437596+0000 osd.4 (osd.4) 345 : cluster [DBG] 4.d scrub starts 2024-02-19T21:19:32.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:30.439405+0000 osd.4 (osd.4) 346 : cluster [DBG] 4.d scrub ok 2024-02-19T21:19:32.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:30.707819+0000 osd.7 (osd.7) 355 : cluster [DBG] 2.b scrub starts 2024-02-19T21:19:32.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:30.711289+0000 osd.7 (osd.7) 356 : cluster [DBG] 2.b scrub ok 2024-02-19T21:19:32.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:30.750908+0000 osd.1 (osd.1) 415 : cluster [DBG] 3.0 scrub starts 2024-02-19T21:19:32.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:30.754376+0000 osd.1 (osd.1) 416 : cluster [DBG] 3.0 scrub ok 2024-02-19T21:19:32.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:31 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:30.918565+0000 mgr.y (mgr.14550) 727 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.555+0000 7f0cee71d700 1 -- 172.21.15.163:0/2727899300 >> v1:172.21.15.163:6790/0 conn(0x7f0ce80fd980 legacy=0x7f0ce80fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.555+0000 7f0cee71d700 1 -- 172.21.15.163:0/2727899300 shutdown_connections 2024-02-19T21:19:32.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.555+0000 7f0cee71d700 1 -- 172.21.15.163:0/2727899300 >> 172.21.15.163:0/2727899300 conn(0x7f0ce80f35a0 msgr2=0x7f0ce80f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.555+0000 7f0cee71d700 1 -- 172.21.15.163:0/2727899300 shutdown_connections 2024-02-19T21:19:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.555+0000 7f0cee71d700 1 -- 172.21.15.163:0/2727899300 wait complete. 2024-02-19T21:19:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.555+0000 7f0cee71d700 1 librados: starting msgr at 2024-02-19T21:19:32.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.555+0000 7f0cee71d700 1 librados: starting objecter 2024-02-19T21:19:32.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.555+0000 7f0cee71d700 1 Processor -- start 2024-02-19T21:19:32.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.555+0000 7f0cee71d700 1 -- start start 2024-02-19T21:19:32.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.555+0000 7f0cee71d700 1 librados: setting wanted keys 2024-02-19T21:19:32.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.555+0000 7f0cee71d700 1 librados: calling monclient init 2024-02-19T21:19:32.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0cee71d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0ce8117280 con 0x7f0ce80fd980 2024-02-19T21:19:32.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0cee71d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0ce8118460 con 0x7f0ce81042c0 2024-02-19T21:19:32.569 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0cee71d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0ce8119660 con 0x7f0ce8100780 2024-02-19T21:19:32.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce7fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f0ce80fd980 0x7f0ce80f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44966/0 (socket says 172.21.15.163:44966) 2024-02-19T21:19:32.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce7fff700 1 -- 172.21.15.163:0/435656463 learned_addr learned my addr 172.21.15.163:0/435656463 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:32.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 10 client.?.objecter ms_handle_connect 0x7f0ce80fd980 2024-02-19T21:19:32.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:32.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 10 client.?.objecter ms_handle_connect 0x7f0ce8100780 2024-02-19T21:19:32.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:32.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578978400 0 0) 0x7f0ce8117280 con 0x7f0ce80fd980 2024-02-19T21:19:32.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd00033f0 con 0x7f0ce80fd980 2024-02-19T21:19:32.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 10 client.?.objecter ms_handle_connect 0x7f0ce81042c0 2024-02-19T21:19:32.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:32.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529735231 0 0) 0x7f0ce8119660 con 0x7f0ce8100780 2024-02-19T21:19:32.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce8117280 con 0x7f0ce8100780 2024-02-19T21:19:32.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1726659993 0 0) 0x7f0cd00033f0 con 0x7f0ce80fd980 2024-02-19T21:19:32.570 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ce8119660 con 0x7f0ce80fd980 2024-02-19T21:19:32.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0cd80010c0 con 0x7f0ce80fd980 2024-02-19T21:19:32.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1678264520 0 0) 0x7f0ce8117280 con 0x7f0ce8100780 2024-02-19T21:19:32.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cd00033f0 con 0x7f0ce8100780 2024-02-19T21:19:32.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604781766 0 0) 0x7f0ce8118460 con 0x7f0ce81042c0 2024-02-19T21:19:32.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce8117280 con 0x7f0ce81042c0 2024-02-19T21:19:32.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.556+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0cdc001840 con 0x7f0ce8100780 2024-02-19T21:19:32.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.557+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1085489041 0 0) 0x7f0ce8119660 con 0x7f0ce80fd980 2024-02-19T21:19:32.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.557+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 >> v1:172.21.15.163:6790/0 conn(0x7f0ce8100780 legacy=0x7f0ce80fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:32.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.557+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 >> v1:172.21.15.193:6789/0 conn(0x7f0ce81042c0 legacy=0x7f0ce8115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:32.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.557+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce811a860 con 0x7f0ce80fd980 2024-02-19T21:19:32.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.557+0000 7f0cee71d700 1 -- 172.21.15.163:0/435656463 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce8119850 con 0x7f0ce80fd980 2024-02-19T21:19:32.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.557+0000 7f0cee71d700 10 client.24120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:32.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.557+0000 7f0cee71d700 1 -- 172.21.15.163:0/435656463 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce8119d40 con 0x7f0ce80fd980 2024-02-19T21:19:32.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.557+0000 7f0cee71d700 1 librados: init done 2024-02-19T21:19:32.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.557+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f0cd8002830 con 0x7f0ce80fd980 2024-02-19T21:19:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.557+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f0cd80038d0 con 0x7f0ce80fd980 2024-02-19T21:19:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.557+0000 7f0cee71d700 10 client.24120.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:19:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.557+0000 7f0cee71d700 15 client.24120.objecter _session_command_op_assign -1 1 2024-02-19T21:19:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.557+0000 7f0cee71d700 20 client.24120.objecter _get_session osd=-1 returning homeless 2024-02-19T21:19:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.557+0000 7f0cee71d700 10 client.24120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.557+0000 7f0cee71d700 1 -- 172.21.15.163:0/435656463 --> v1:172.21.15.163:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0cd40028b0 con 0x7f0ce80fd980 2024-02-19T21:19:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.558+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f0cd8002390 con 0x7f0ce80fd980 2024-02-19T21:19:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.558+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(850..850 src has 796..850) v4 ==== 7419+0+0 (unknown 7470691 0 0) 0x7f0cd8078d80 con 0x7f0ce80fd980 2024-02-19T21:19:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.558+0000 7f0ce57fa700 10 client.24120.objecter ms_dispatch 0x7f0ce8001550 osd_map(850..850 src has 796..850) v4 2024-02-19T21:19:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.558+0000 7f0ce57fa700 3 client.24120.objecter handle_osd_map got epochs [850,850] > 0 2024-02-19T21:19:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.558+0000 7f0ce57fa700 3 client.24120.objecter handle_osd_map decoding full epoch 850 2024-02-19T21:19:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.558+0000 7f0ce57fa700 10 client.24120.objecter checking command 1 2024-02-19T21:19:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.558+0000 7f0ce57fa700 20 client.24120.objecter _get_session s=0x7f0cd80010c0 osd=5 3 2024-02-19T21:19:32.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.558+0000 7f0ce57fa700 20 client.24120.objecter put_session s=0x7f0cd80010c0 osd=5 3 2024-02-19T21:19:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.558+0000 7f0ce57fa700 15 client.24120.objecter _session_command_op_remove -1 1 2024-02-19T21:19:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.559+0000 7f0ce57fa700 20 client.24120.objecter _get_session s=0x7f0cd80010c0 osd=5 3 2024-02-19T21:19:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.559+0000 7f0ce57fa700 20 client.24120.objecter get_session s=0x7f0cd80010c0 osd=5 3 2024-02-19T21:19:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.559+0000 7f0ce57fa700 15 client.24120.objecter _session_command_op_assign 5 1 2024-02-19T21:19:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.559+0000 7f0ce57fa700 20 client.24120.objecter put_session s=0x7f0cd80010c0 osd=5 4 2024-02-19T21:19:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.559+0000 7f0ce57fa700 10 client.24120.objecter _send_command 1 2024-02-19T21:19:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.559+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 --> v1:172.21.15.193:6804/805161991 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0cd0073b70 con 0x7f0cd006fe60 2024-02-19T21:19:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.559+0000 7f0ce57fa700 20 client.24120.objecter dump_active .. 0 homeless 2024-02-19T21:19:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.559+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_get_version_reply(handle=1 version=850) v2 ==== 24+0+0 (unknown 3804485784 0 0) 0x7f0cd8079190 con 0x7f0ce80fd980 2024-02-19T21:19:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.559+0000 7f0ce57fa700 10 client.24120.objecter ms_handle_connect 0x7f0cd0063c20 2024-02-19T21:19:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.559+0000 7f0ce57fa700 10 client.24120.objecter ms_handle_connect 0x7f0cd006fe60 2024-02-19T21:19:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.560+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 <== osd.5 v1:172.21.15.193:6804/805161991 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0cd0073b70 con 0x7f0cd006fe60 2024-02-19T21:19:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.560+0000 7f0ce57fa700 10 client.24120.objecter ms_dispatch 0x7f0ce8001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:19:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.560+0000 7f0ce57fa700 10 client.24120.objecter _finish_command 1 = system:0 2024-02-19T21:19:32.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.560+0000 7f0ce57fa700 20 client.24120.objecter put_session s=0x7f0cd80010c0 osd=5 4 2024-02-19T21:19:32.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.560+0000 7f0ce57fa700 15 client.24120.objecter _session_command_op_remove 5 1 2024-02-19T21:19:32.587 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.574+0000 7f0cee71d700 10 client.24120.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:19:32.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.574+0000 7f0cee71d700 15 client.24120.objecter _session_command_op_assign -1 2 2024-02-19T21:19:32.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.574+0000 7f0cee71d700 20 client.24120.objecter _get_session s=0x7f0cd80010c0 osd=5 3 2024-02-19T21:19:32.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.574+0000 7f0cee71d700 20 client.24120.objecter put_session s=0x7f0cd80010c0 osd=5 3 2024-02-19T21:19:32.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.574+0000 7f0cee71d700 20 client.24120.objecter _get_session s=0x7f0cd80010c0 osd=5 3 2024-02-19T21:19:32.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.574+0000 7f0cee71d700 15 client.24120.objecter _session_command_op_remove -1 2 2024-02-19T21:19:32.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.574+0000 7f0cee71d700 20 client.24120.objecter get_session s=0x7f0cd80010c0 osd=5 3 2024-02-19T21:19:32.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.574+0000 7f0cee71d700 15 client.24120.objecter _session_command_op_assign 5 2 2024-02-19T21:19:32.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.574+0000 7f0cee71d700 20 client.24120.objecter put_session s=0x7f0cd80010c0 osd=5 4 2024-02-19T21:19:32.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.574+0000 7f0cee71d700 10 client.24120.objecter _send_command 2 2024-02-19T21:19:32.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.574+0000 7f0cee71d700 1 -- 172.21.15.163:0/435656463 --> v1:172.21.15.193:6804/805161991 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0cd4001140 con 0x7f0cd006fe60 2024-02-19T21:19:32.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.575+0000 7f0ce57fa700 1 -- 172.21.15.163:0/435656463 <== osd.5 v1:172.21.15.193:6804/805161991 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 2596373458) 0x7f0cd4001140 con 0x7f0cd006fe60 2024-02-19T21:19:32.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.575+0000 7f0ce57fa700 10 client.24120.objecter ms_dispatch 0x7f0ce8001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:19:32.588 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.575+0000 7f0ce57fa700 10 client.24120.objecter _finish_command 2 = system:0 2024-02-19T21:19:32.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.575+0000 7f0ce57fa700 20 client.24120.objecter put_session s=0x7f0cd80010c0 osd=5 4 2024-02-19T21:19:32.589 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.575+0000 7f0ce57fa700 15 client.24120.objecter _session_command_op_remove 5 2 2024-02-19T21:19:32.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.575+0000 7f0cee71d700 10 librados: watch_flush enter 2024-02-19T21:19:32.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.575+0000 7f0cee71d700 10 librados: watch_flush exit 2024-02-19T21:19:32.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.575+0000 7f0cee71d700 10 client.24120.objecter close_session for osd.5 2024-02-19T21:19:32.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.575+0000 7f0cee71d700 1 -- 172.21.15.163:0/435656463 >> v1:172.21.15.193:6804/805161991 conn(0x7f0cd006fe60 legacy=0x7f0cd0072300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:32.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.575+0000 7f0cee71d700 20 client.24120.objecter put_session s=0x7f0cd80010c0 osd=5 1 2024-02-19T21:19:32.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.575+0000 7f0cee71d700 20 client.24120.objecter shutdown clearing up homeless session... 2024-02-19T21:19:32.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.575+0000 7f0cee71d700 10 client.24120.objecter successfully canceled tick 2024-02-19T21:19:32.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.575+0000 7f0cee71d700 1 -- 172.21.15.163:0/435656463 >> v1:172.21.15.163:6800/1552514681 conn(0x7f0cd0063c20 legacy=0x7f0cd00660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:32.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.575+0000 7f0cee71d700 1 -- 172.21.15.163:0/435656463 >> v1:172.21.15.163:6789/0 conn(0x7f0ce80fd980 legacy=0x7f0ce80f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:32.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.575+0000 7f0ceccba700 1 -- 172.21.15.163:0/435656463 reap_dead start 2024-02-19T21:19:32.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.576+0000 7f0cee71d700 1 -- 172.21.15.163:0/435656463 shutdown_connections 2024-02-19T21:19:32.590 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.576+0000 7f0cee71d700 1 -- 172.21.15.163:0/435656463 >> 172.21.15.163:0/435656463 conn(0x7f0ce80f35a0 msgr2=0x7f0ce8107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:32.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.576+0000 7f0cee71d700 1 -- 172.21.15.163:0/435656463 shutdown_connections 2024-02-19T21:19:32.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.576+0000 7f0cee71d700 1 -- 172.21.15.163:0/435656463 wait complete. 2024-02-19T21:19:32.591 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:32.576+0000 7f0cee71d700 1 librados: shutdown 2024-02-19T21:19:32.908 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:31.272071+0000 osd.0 (osd.0) 182 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:19:32.908 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:31.272985+0000 osd.0 (osd.0) 183 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:19:32.908 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:31.373461+0000 mgr.y (mgr.14550) 728 : cluster [DBG] pgmap v1317: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-19T21:19:32.908 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:31.690872+0000 osd.7 (osd.7) 357 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:19:32.908 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:31.694041+0000 osd.7 (osd.7) 358 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:19:32.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:31.272071+0000 osd.0 (osd.0) 182 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:19:32.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:31.272985+0000 osd.0 (osd.0) 183 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:19:32.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:31.373461+0000 mgr.y (mgr.14550) 728 : cluster [DBG] pgmap v1317: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-19T21:19:32.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:31.690872+0000 osd.7 (osd.7) 357 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:19:32.909 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:32 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:31.694041+0000 osd.7 (osd.7) 358 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:19:33.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:31.272071+0000 osd.0 (osd.0) 182 : cluster [DBG] 5.5 scrub starts 2024-02-19T21:19:33.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:31.272985+0000 osd.0 (osd.0) 183 : cluster [DBG] 5.5 scrub ok 2024-02-19T21:19:33.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:31.373461+0000 mgr.y (mgr.14550) 728 : cluster [DBG] pgmap v1317: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-19T21:19:33.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:31.690872+0000 osd.7 (osd.7) 357 : cluster [DBG] 2.8 scrub starts 2024-02-19T21:19:33.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:32 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:31.694041+0000 osd.7 (osd.7) 358 : cluster [DBG] 2.8 scrub ok 2024-02-19T21:19:33.250 INFO:teuthology.orchestra.run.smithi163.stdout:158913790185 2024-02-19T21:19:33.251 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-19T21:19:33.499 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:19:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:19:33] "GET /metrics HTTP/1.1" 200 187976 "" "Prometheus/2.33.4" 2024-02-19T21:19:33.520 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:34.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:33.373950+0000 mgr.y (mgr.14550) 729 : cluster [DBG] pgmap v1318: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-19T21:19:34.815 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:34.249387+0000 osd.3 (osd.3) 413 : cluster [DBG] 4.11 deep-scrub starts 2024-02-19T21:19:34.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:33.373950+0000 mgr.y (mgr.14550) 729 : cluster [DBG] pgmap v1318: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-19T21:19:34.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:34.249387+0000 osd.3 (osd.3) 413 : cluster [DBG] 4.11 deep-scrub starts 2024-02-19T21:19:35.072 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:33.373950+0000 mgr.y (mgr.14550) 729 : cluster [DBG] pgmap v1318: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-19T21:19:35.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:34.249387+0000 osd.3 (osd.3) 413 : cluster [DBG] 4.11 deep-scrub starts 2024-02-19T21:19:35.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:34.253605+0000 osd.3 (osd.3) 414 : cluster [DBG] 4.11 deep-scrub ok 2024-02-19T21:19:35.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:34.525818+0000 osd.5 (osd.5) 378 : cluster [DBG] 6.d scrub starts 2024-02-19T21:19:35.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:34 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:34.529205+0000 osd.5 (osd.5) 379 : cluster [DBG] 6.d scrub ok 2024-02-19T21:19:35.164 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:34.253605+0000 osd.3 (osd.3) 414 : cluster [DBG] 4.11 deep-scrub ok 2024-02-19T21:19:35.164 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:34.525818+0000 osd.5 (osd.5) 378 : cluster [DBG] 6.d scrub starts 2024-02-19T21:19:35.164 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:34.529205+0000 osd.5 (osd.5) 379 : cluster [DBG] 6.d scrub ok 2024-02-19T21:19:35.165 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:34.253605+0000 osd.3 (osd.3) 414 : cluster [DBG] 4.11 deep-scrub ok 2024-02-19T21:19:35.165 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:34.525818+0000 osd.5 (osd.5) 378 : cluster [DBG] 6.d scrub starts 2024-02-19T21:19:35.165 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:34 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:34.529205+0000 osd.5 (osd.5) 379 : cluster [DBG] 6.d scrub ok 2024-02-19T21:19:35.418 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.415+0000 7ffa6054a700 1 -- 172.21.15.163:0/3824512870 >> v1:172.21.15.163:6790/0 conn(0x7ffa580fd980 legacy=0x7ffa580fdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:35.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.415+0000 7ffa6054a700 1 -- 172.21.15.163:0/3824512870 shutdown_connections 2024-02-19T21:19:35.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.415+0000 7ffa6054a700 1 -- 172.21.15.163:0/3824512870 >> 172.21.15.163:0/3824512870 conn(0x7ffa580f35a0 msgr2=0x7ffa580f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:35.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.415+0000 7ffa6054a700 1 -- 172.21.15.163:0/3824512870 shutdown_connections 2024-02-19T21:19:35.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.415+0000 7ffa6054a700 1 -- 172.21.15.163:0/3824512870 wait complete. 2024-02-19T21:19:35.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa6054a700 1 librados: starting msgr at 2024-02-19T21:19:35.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa6054a700 1 librados: starting objecter 2024-02-19T21:19:35.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa6054a700 1 Processor -- start 2024-02-19T21:19:35.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa6054a700 1 -- start start 2024-02-19T21:19:35.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa6054a700 1 librados: setting wanted keys 2024-02-19T21:19:35.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa6054a700 1 librados: calling monclient init 2024-02-19T21:19:35.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa6054a700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa580fcf70 con 0x7ffa581042c0 2024-02-19T21:19:35.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa6054a700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa580fd140 con 0x7ffa580fd980 2024-02-19T21:19:35.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa6054a700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa580fd350 con 0x7ffa58100780 2024-02-19T21:19:35.419 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa5eae7700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ffa581042c0 0x7ffa58118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46850/0 (socket says 172.21.15.163:46850) 2024-02-19T21:19:35.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa5eae7700 1 -- 172.21.15.163:0/141986744 learned_addr learned my addr 172.21.15.163:0/141986744 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:35.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa4f7fe700 10 client.?.objecter ms_handle_connect 0x7ffa581042c0 2024-02-19T21:19:35.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa4f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:35.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa4f7fe700 10 client.?.objecter ms_handle_connect 0x7ffa58100780 2024-02-19T21:19:35.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa4f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:35.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854823531 0 0) 0x7ffa580fcf70 con 0x7ffa581042c0 2024-02-19T21:19:35.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa440038e0 con 0x7ffa581042c0 2024-02-19T21:19:35.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa4f7fe700 10 client.?.objecter ms_handle_connect 0x7ffa580fd980 2024-02-19T21:19:35.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa4f7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:35.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398765379 0 0) 0x7ffa580fd350 con 0x7ffa58100780 2024-02-19T21:19:35.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa580fcf70 con 0x7ffa58100780 2024-02-19T21:19:35.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942673904 0 0) 0x7ffa580fd140 con 0x7ffa580fd980 2024-02-19T21:19:35.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.416+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa580fd350 con 0x7ffa580fd980 2024-02-19T21:19:35.420 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1440302332 0 0) 0x7ffa440038e0 con 0x7ffa581042c0 2024-02-19T21:19:35.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa580fd140 con 0x7ffa581042c0 2024-02-19T21:19:35.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa50001c80 con 0x7ffa581042c0 2024-02-19T21:19:35.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3255104658 0 0) 0x7ffa580fcf70 con 0x7ffa58100780 2024-02-19T21:19:35.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa440038e0 con 0x7ffa58100780 2024-02-19T21:19:35.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa540016a0 con 0x7ffa58100780 2024-02-19T21:19:35.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1648621102 0 0) 0x7ffa580fd350 con 0x7ffa580fd980 2024-02-19T21:19:35.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa580fcf70 con 0x7ffa580fd980 2024-02-19T21:19:35.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa480015d0 con 0x7ffa580fd980 2024-02-19T21:19:35.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 852141235 0 0) 0x7ffa580fd140 con 0x7ffa581042c0 2024-02-19T21:19:35.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 >> v1:172.21.15.163:6790/0 conn(0x7ffa58100780 legacy=0x7ffa58114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:35.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 >> v1:172.21.15.193:6789/0 conn(0x7ffa580fd980 legacy=0x7ffa580fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:35.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5811cb90 con 0x7ffa581042c0 2024-02-19T21:19:35.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa6054a700 1 -- 172.21.15.163:0/141986744 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa58119bc0 con 0x7ffa581042c0 2024-02-19T21:19:35.421 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa6054a700 10 client.24126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:35.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa6054a700 1 -- 172.21.15.163:0/141986744 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa5811a030 con 0x7ffa581042c0 2024-02-19T21:19:35.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa6054a700 1 librados: init done 2024-02-19T21:19:35.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ffa50002010 con 0x7ffa581042c0 2024-02-19T21:19:35.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ffa50005ed0 con 0x7ffa581042c0 2024-02-19T21:19:35.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa6054a700 10 client.24126.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:19:35.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa6054a700 15 client.24126.objecter _session_command_op_assign -1 1 2024-02-19T21:19:35.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa6054a700 20 client.24126.objecter _get_session osd=-1 returning homeless 2024-02-19T21:19:35.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa6054a700 10 client.24126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:35.422 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.417+0000 7ffa6054a700 1 -- 172.21.15.163:0/141986744 --> v1:172.21.15.163:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ffa3c0028b0 con 0x7ffa581042c0 2024-02-19T21:19:35.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.418+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ffa50002c30 con 0x7ffa581042c0 2024-02-19T21:19:35.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(850..850 src has 796..850) v4 ==== 7419+0+0 (unknown 7470691 0 0) 0x7ffa50078c20 con 0x7ffa581042c0 2024-02-19T21:19:35.423 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 10 client.24126.objecter ms_dispatch 0x7ffa58001550 osd_map(850..850 src has 796..850) v4 2024-02-19T21:19:35.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 3 client.24126.objecter handle_osd_map got epochs [850,850] > 0 2024-02-19T21:19:35.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 3 client.24126.objecter handle_osd_map decoding full epoch 850 2024-02-19T21:19:35.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 10 client.24126.objecter checking command 1 2024-02-19T21:19:35.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 20 client.24126.objecter _get_session s=0x7ffa4406f940 osd=6 3 2024-02-19T21:19:35.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 20 client.24126.objecter put_session s=0x7ffa4406f940 osd=6 3 2024-02-19T21:19:35.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 15 client.24126.objecter _session_command_op_remove -1 1 2024-02-19T21:19:35.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 20 client.24126.objecter _get_session s=0x7ffa4406f940 osd=6 3 2024-02-19T21:19:35.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 20 client.24126.objecter get_session s=0x7ffa4406f940 osd=6 3 2024-02-19T21:19:35.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 15 client.24126.objecter _session_command_op_assign 6 1 2024-02-19T21:19:35.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 20 client.24126.objecter put_session s=0x7ffa4406f940 osd=6 4 2024-02-19T21:19:35.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 10 client.24126.objecter _send_command 1 2024-02-19T21:19:35.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 --> v1:172.21.15.193:6808/787604615 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffa44073d00 con 0x7ffa4406fff0 2024-02-19T21:19:35.424 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 20 client.24126.objecter dump_active .. 0 homeless 2024-02-19T21:19:35.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_get_version_reply(handle=1 version=850) v2 ==== 24+0+0 (unknown 3804485784 0 0) 0x7ffa50078fd0 con 0x7ffa581042c0 2024-02-19T21:19:35.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 10 client.24126.objecter ms_handle_connect 0x7ffa44063c20 2024-02-19T21:19:35.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.419+0000 7ffa4f7fe700 10 client.24126.objecter ms_handle_connect 0x7ffa4406fff0 2024-02-19T21:19:35.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.420+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== osd.6 v1:172.21.15.193:6808/787604615 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ffa44073d00 con 0x7ffa4406fff0 2024-02-19T21:19:35.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.420+0000 7ffa4f7fe700 10 client.24126.objecter ms_dispatch 0x7ffa58001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:19:35.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.420+0000 7ffa4f7fe700 10 client.24126.objecter _finish_command 1 = system:0 2024-02-19T21:19:35.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.420+0000 7ffa4f7fe700 20 client.24126.objecter put_session s=0x7ffa4406f940 osd=6 4 2024-02-19T21:19:35.425 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.420+0000 7ffa4f7fe700 15 client.24126.objecter _session_command_op_remove 6 1 2024-02-19T21:19:35.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa6054a700 10 client.24126.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:19:35.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa6054a700 15 client.24126.objecter _session_command_op_assign -1 2 2024-02-19T21:19:35.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa6054a700 20 client.24126.objecter _get_session s=0x7ffa4406f940 osd=6 3 2024-02-19T21:19:35.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa6054a700 20 client.24126.objecter put_session s=0x7ffa4406f940 osd=6 3 2024-02-19T21:19:35.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa6054a700 20 client.24126.objecter _get_session s=0x7ffa4406f940 osd=6 3 2024-02-19T21:19:35.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa6054a700 15 client.24126.objecter _session_command_op_remove -1 2 2024-02-19T21:19:35.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa6054a700 20 client.24126.objecter get_session s=0x7ffa4406f940 osd=6 3 2024-02-19T21:19:35.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa6054a700 15 client.24126.objecter _session_command_op_assign 6 2 2024-02-19T21:19:35.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa6054a700 20 client.24126.objecter put_session s=0x7ffa4406f940 osd=6 4 2024-02-19T21:19:35.435 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa6054a700 10 client.24126.objecter _send_command 2 2024-02-19T21:19:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa6054a700 1 -- 172.21.15.163:0/141986744 --> v1:172.21.15.193:6808/787604615 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ffa3c003c60 con 0x7ffa4406fff0 2024-02-19T21:19:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa4f7fe700 1 -- 172.21.15.163:0/141986744 <== osd.6 v1:172.21.15.193:6808/787604615 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 3423269531) 0x7ffa3c003c60 con 0x7ffa4406fff0 2024-02-19T21:19:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa4f7fe700 10 client.24126.objecter ms_dispatch 0x7ffa58001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:19:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa4f7fe700 10 client.24126.objecter _finish_command 2 = system:0 2024-02-19T21:19:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa4f7fe700 20 client.24126.objecter put_session s=0x7ffa4406f940 osd=6 4 2024-02-19T21:19:35.436 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.434+0000 7ffa4f7fe700 15 client.24126.objecter _session_command_op_remove 6 2 2024-02-19T21:19:35.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.435+0000 7ffa6054a700 10 librados: watch_flush enter 2024-02-19T21:19:35.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.435+0000 7ffa6054a700 10 librados: watch_flush exit 2024-02-19T21:19:35.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.435+0000 7ffa6054a700 10 client.24126.objecter close_session for osd.6 2024-02-19T21:19:35.437 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.435+0000 7ffa6054a700 1 -- 172.21.15.163:0/141986744 >> v1:172.21.15.193:6808/787604615 conn(0x7ffa4406fff0 legacy=0x7ffa44072490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:35.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.435+0000 7ffa6054a700 20 client.24126.objecter put_session s=0x7ffa4406f940 osd=6 1 2024-02-19T21:19:35.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.435+0000 7ffa6054a700 20 client.24126.objecter shutdown clearing up homeless session... 2024-02-19T21:19:35.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.435+0000 7ffa6054a700 10 client.24126.objecter successfully canceled tick 2024-02-19T21:19:35.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.435+0000 7ffa6054a700 1 -- 172.21.15.163:0/141986744 >> v1:172.21.15.163:6800/1552514681 conn(0x7ffa44063c20 legacy=0x7ffa440660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:35.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.435+0000 7ffa6054a700 1 -- 172.21.15.163:0/141986744 >> v1:172.21.15.163:6789/0 conn(0x7ffa581042c0 legacy=0x7ffa58118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:35.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.435+0000 7ffa5eae7700 1 -- 172.21.15.163:0/141986744 reap_dead start 2024-02-19T21:19:35.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.435+0000 7ffa6054a700 1 -- 172.21.15.163:0/141986744 shutdown_connections 2024-02-19T21:19:35.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.435+0000 7ffa6054a700 1 -- 172.21.15.163:0/141986744 >> 172.21.15.163:0/141986744 conn(0x7ffa580f35a0 msgr2=0x7ffa58107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:35.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.435+0000 7ffa6054a700 1 -- 172.21.15.163:0/141986744 shutdown_connections 2024-02-19T21:19:35.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.435+0000 7ffa6054a700 1 -- 172.21.15.163:0/141986744 wait complete. 2024-02-19T21:19:35.438 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:35.435+0000 7ffa6054a700 1 librados: shutdown 2024-02-19T21:19:36.129 INFO:teuthology.orchestra.run.smithi163.stdout:193273528549 2024-02-19T21:19:36.129 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-19T21:19:36.404 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:36.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:35.374731+0000 mgr.y (mgr.14550) 730 : cluster [DBG] pgmap v1319: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:19:36.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:35.385281+0000 osd.4 (osd.4) 347 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:19:36.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:35.387145+0000 osd.4 (osd.4) 348 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:19:37.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:35.374731+0000 mgr.y (mgr.14550) 730 : cluster [DBG] pgmap v1319: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:19:37.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:35.385281+0000 osd.4 (osd.4) 347 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:19:37.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:36 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:35.387145+0000 osd.4 (osd.4) 348 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:19:37.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:35.374731+0000 mgr.y (mgr.14550) 730 : cluster [DBG] pgmap v1319: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:19:37.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:35.385281+0000 osd.4 (osd.4) 347 : cluster [DBG] 4.9 scrub starts 2024-02-19T21:19:37.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:35.387145+0000 osd.4 (osd.4) 348 : cluster [DBG] 4.9 scrub ok 2024-02-19T21:19:37.828 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:36.615749+0000 osd.7 (osd.7) 359 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:19:37.828 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:36.618946+0000 osd.7 (osd.7) 360 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:19:37.828 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:36.726310+0000 osd.1 (osd.1) 417 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:19:37.828 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:36.729353+0000 osd.1 (osd.1) 418 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:19:37.828 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:36.851129+0000 osd.1 (osd.1) 419 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:19:37.828 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:36.851314+0000 osd.1 (osd.1) 420 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:19:37.828 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:37.266572+0000 osd.3 (osd.3) 415 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:19:37.829 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:37.268099+0000 osd.3 (osd.3) 416 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:19:38.073 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:19:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:19:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:19:38.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:36.615749+0000 osd.7 (osd.7) 359 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:19:38.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:36.618946+0000 osd.7 (osd.7) 360 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:19:38.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:36.726310+0000 osd.1 (osd.1) 417 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:19:38.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:36.729353+0000 osd.1 (osd.1) 418 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:19:38.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:36.851129+0000 osd.1 (osd.1) 419 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:19:38.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:36.851314+0000 osd.1 (osd.1) 420 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:19:38.075 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:37.266572+0000 osd.3 (osd.3) 415 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:19:38.075 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:37 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:37.268099+0000 osd.3 (osd.3) 416 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:19:38.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.100+0000 7ff539e40700 1 -- 172.21.15.163:0/1142301273 >> v1:172.21.15.163:6790/0 conn(0x7ff5340fc870 legacy=0x7ff5340fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:38.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.100+0000 7ff539e40700 1 -- 172.21.15.163:0/1142301273 shutdown_connections 2024-02-19T21:19:38.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.100+0000 7ff539e40700 1 -- 172.21.15.163:0/1142301273 >> 172.21.15.163:0/1142301273 conn(0x7ff5340f2470 msgr2=0x7ff5340f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:38.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.100+0000 7ff539e40700 1 -- 172.21.15.163:0/1142301273 shutdown_connections 2024-02-19T21:19:38.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.100+0000 7ff539e40700 1 -- 172.21.15.163:0/1142301273 wait complete. 2024-02-19T21:19:38.103 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.101+0000 7ff539e40700 1 librados: starting msgr at 2024-02-19T21:19:38.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.101+0000 7ff539e40700 1 librados: starting objecter 2024-02-19T21:19:38.104 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.101+0000 7ff539e40700 1 Processor -- start 2024-02-19T21:19:38.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.101+0000 7ff539e40700 1 -- start start 2024-02-19T21:19:38.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.101+0000 7ff539e40700 1 librados: setting wanted keys 2024-02-19T21:19:38.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.101+0000 7ff539e40700 1 librados: calling monclient init 2024-02-19T21:19:38.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff539e40700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff5341172a0 con 0x7ff5340ff670 2024-02-19T21:19:38.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff539e40700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff534118480 con 0x7ff5341031b0 2024-02-19T21:19:38.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff539e40700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff534119680 con 0x7ff5340fc870 2024-02-19T21:19:38.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff532ffd700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ff5340ff670 0x7ff534061290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46872/0 (socket says 172.21.15.163:46872) 2024-02-19T21:19:38.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff5337fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7ff5340fc870 0x7ff534060b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:50572/0 (socket says 172.21.15.163:50572) 2024-02-19T21:19:38.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff532ffd700 1 -- 172.21.15.163:0/2399390069 learned_addr learned my addr 172.21.15.163:0/2399390069 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:38.107 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 10 client.?.objecter ms_handle_connect 0x7ff5340ff670 2024-02-19T21:19:38.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:38.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 10 client.?.objecter ms_handle_connect 0x7ff5340fc870 2024-02-19T21:19:38.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:38.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 10 client.?.objecter ms_handle_connect 0x7ff5341031b0 2024-02-19T21:19:38.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:38.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422423188 0 0) 0x7ff5341172a0 con 0x7ff5340ff670 2024-02-19T21:19:38.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff520003030 con 0x7ff5340ff670 2024-02-19T21:19:38.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543368620 0 0) 0x7ff534119680 con 0x7ff5340fc870 2024-02-19T21:19:38.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5341172a0 con 0x7ff5340fc870 2024-02-19T21:19:38.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600115950 0 0) 0x7ff534118480 con 0x7ff5341031b0 2024-02-19T21:19:38.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff534119680 con 0x7ff5341031b0 2024-02-19T21:19:38.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2570052390 0 0) 0x7ff520003030 con 0x7ff5340ff670 2024-02-19T21:19:38.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff534118480 con 0x7ff5340ff670 2024-02-19T21:19:38.108 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff5240018e0 con 0x7ff5340ff670 2024-02-19T21:19:38.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 460190126 0 0) 0x7ff5341172a0 con 0x7ff5340fc870 2024-02-19T21:19:38.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff520003030 con 0x7ff5340fc870 2024-02-19T21:19:38.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff51c0015d0 con 0x7ff5340fc870 2024-02-19T21:19:38.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2627235569 0 0) 0x7ff534119680 con 0x7ff5341031b0 2024-02-19T21:19:38.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5341172a0 con 0x7ff5341031b0 2024-02-19T21:19:38.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff52c001920 con 0x7ff5341031b0 2024-02-19T21:19:38.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 756102374 0 0) 0x7ff534118480 con 0x7ff5340ff670 2024-02-19T21:19:38.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 >> v1:172.21.15.163:6790/0 conn(0x7ff5340fc870 legacy=0x7ff534060b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:38.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 >> v1:172.21.15.193:6789/0 conn(0x7ff5341031b0 legacy=0x7ff5341159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:38.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53411a880 con 0x7ff5340ff670 2024-02-19T21:19:38.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.102+0000 7ff539e40700 1 -- 172.21.15.163:0/2399390069 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff534119870 con 0x7ff5340ff670 2024-02-19T21:19:38.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.103+0000 7ff539e40700 10 client.24132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:38.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.103+0000 7ff539e40700 1 -- 172.21.15.163:0/2399390069 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff534119d60 con 0x7ff5340ff670 2024-02-19T21:19:38.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.103+0000 7ff539e40700 1 librados: init done 2024-02-19T21:19:38.109 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.103+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7ff524001c90 con 0x7ff5340ff670 2024-02-19T21:19:38.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.103+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7ff524005b70 con 0x7ff5340ff670 2024-02-19T21:19:38.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.103+0000 7ff539e40700 10 client.24132.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-02-19T21:19:38.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.103+0000 7ff539e40700 15 client.24132.objecter _session_command_op_assign -1 1 2024-02-19T21:19:38.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.103+0000 7ff539e40700 20 client.24132.objecter _get_session osd=-1 returning homeless 2024-02-19T21:19:38.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.103+0000 7ff539e40700 10 client.24132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:38.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.103+0000 7ff539e40700 1 -- 172.21.15.163:0/2399390069 --> v1:172.21.15.163:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff5140028b0 con 0x7ff5340ff670 2024-02-19T21:19:38.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7ff524002270 con 0x7ff5340ff670 2024-02-19T21:19:38.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(851..851 src has 796..851) v4 ==== 7419+0+0 (unknown 3482085237 0 0) 0x7ff524078bb0 con 0x7ff5340ff670 2024-02-19T21:19:38.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 10 client.24132.objecter ms_dispatch 0x7ff534001550 osd_map(851..851 src has 796..851) v4 2024-02-19T21:19:38.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 3 client.24132.objecter handle_osd_map got epochs [851,851] > 0 2024-02-19T21:19:38.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 3 client.24132.objecter handle_osd_map decoding full epoch 851 2024-02-19T21:19:38.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 10 client.24132.objecter checking command 1 2024-02-19T21:19:38.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 20 client.24132.objecter _get_session s=0x7ff52006fa60 osd=7 3 2024-02-19T21:19:38.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 20 client.24132.objecter put_session s=0x7ff52006fa60 osd=7 3 2024-02-19T21:19:38.110 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 15 client.24132.objecter _session_command_op_remove -1 1 2024-02-19T21:19:38.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 20 client.24132.objecter _get_session s=0x7ff52006fa60 osd=7 3 2024-02-19T21:19:38.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 20 client.24132.objecter get_session s=0x7ff52006fa60 osd=7 3 2024-02-19T21:19:38.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 15 client.24132.objecter _session_command_op_assign 7 1 2024-02-19T21:19:38.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 20 client.24132.objecter put_session s=0x7ff52006fa60 osd=7 4 2024-02-19T21:19:38.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 10 client.24132.objecter _send_command 1 2024-02-19T21:19:38.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 --> v1:172.21.15.193:6812/157927432 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff520073e20 con 0x7ff520070110 2024-02-19T21:19:38.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 20 client.24132.objecter dump_active .. 0 homeless 2024-02-19T21:19:38.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.104+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_get_version_reply(handle=1 version=851) v2 ==== 24+0+0 (unknown 282056038 0 0) 0x7ff524078fc0 con 0x7ff5340ff670 2024-02-19T21:19:38.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.105+0000 7ff530ff9700 10 client.24132.objecter ms_handle_connect 0x7ff520063cc0 2024-02-19T21:19:38.111 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.105+0000 7ff530ff9700 10 client.24132.objecter ms_handle_connect 0x7ff520070110 2024-02-19T21:19:38.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.106+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== osd.7 v1:172.21.15.193:6812/157927432 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7ff520073e20 con 0x7ff520070110 2024-02-19T21:19:38.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.106+0000 7ff530ff9700 10 client.24132.objecter ms_dispatch 0x7ff534001550 command_reply(tid 1: 0 ) v1 2024-02-19T21:19:38.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.106+0000 7ff530ff9700 10 client.24132.objecter _finish_command 1 = system:0 2024-02-19T21:19:38.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.106+0000 7ff530ff9700 20 client.24132.objecter put_session s=0x7ff52006fa60 osd=7 4 2024-02-19T21:19:38.112 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.106+0000 7ff530ff9700 15 client.24132.objecter _session_command_op_remove 7 1 2024-02-19T21:19:38.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 10 client.24132.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-02-19T21:19:38.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 15 client.24132.objecter _session_command_op_assign -1 2 2024-02-19T21:19:38.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 20 client.24132.objecter _get_session s=0x7ff52006fa60 osd=7 3 2024-02-19T21:19:38.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 20 client.24132.objecter put_session s=0x7ff52006fa60 osd=7 3 2024-02-19T21:19:38.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 20 client.24132.objecter _get_session s=0x7ff52006fa60 osd=7 3 2024-02-19T21:19:38.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 15 client.24132.objecter _session_command_op_remove -1 2 2024-02-19T21:19:38.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 20 client.24132.objecter get_session s=0x7ff52006fa60 osd=7 3 2024-02-19T21:19:38.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 15 client.24132.objecter _session_command_op_assign 7 2 2024-02-19T21:19:38.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 20 client.24132.objecter put_session s=0x7ff52006fa60 osd=7 4 2024-02-19T21:19:38.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 10 client.24132.objecter _send_command 2 2024-02-19T21:19:38.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 1 -- 172.21.15.163:0/2399390069 --> v1:172.21.15.193:6812/157927432 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff514001140 con 0x7ff520070110 2024-02-19T21:19:38.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff530ff9700 1 -- 172.21.15.163:0/2399390069 <== osd.7 v1:172.21.15.193:6812/157927432 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 1981069406) 0x7ff514001140 con 0x7ff520070110 2024-02-19T21:19:38.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff530ff9700 10 client.24132.objecter ms_dispatch 0x7ff534001550 command_reply(tid 2: 0 ) v1 2024-02-19T21:19:38.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff530ff9700 10 client.24132.objecter _finish_command 2 = system:0 2024-02-19T21:19:38.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff530ff9700 20 client.24132.objecter put_session s=0x7ff52006fa60 osd=7 4 2024-02-19T21:19:38.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff530ff9700 15 client.24132.objecter _session_command_op_remove 7 2 2024-02-19T21:19:38.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 10 librados: watch_flush enter 2024-02-19T21:19:38.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 10 librados: watch_flush exit 2024-02-19T21:19:38.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 10 client.24132.objecter close_session for osd.7 2024-02-19T21:19:38.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 1 -- 172.21.15.163:0/2399390069 >> v1:172.21.15.193:6812/157927432 conn(0x7ff520070110 legacy=0x7ff5200725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:38.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 20 client.24132.objecter put_session s=0x7ff52006fa60 osd=7 1 2024-02-19T21:19:38.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 20 client.24132.objecter shutdown clearing up homeless session... 2024-02-19T21:19:38.124 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 10 client.24132.objecter successfully canceled tick 2024-02-19T21:19:38.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 1 -- 172.21.15.163:0/2399390069 >> v1:172.21.15.163:6800/1552514681 conn(0x7ff520063cc0 legacy=0x7ff520066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:38.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff539e40700 1 -- 172.21.15.163:0/2399390069 >> v1:172.21.15.163:6789/0 conn(0x7ff5340ff670 legacy=0x7ff534061290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:38.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.120+0000 7ff533fff700 1 -- 172.21.15.163:0/2399390069 reap_dead start 2024-02-19T21:19:38.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.121+0000 7ff539e40700 1 -- 172.21.15.163:0/2399390069 shutdown_connections 2024-02-19T21:19:38.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.121+0000 7ff539e40700 1 -- 172.21.15.163:0/2399390069 >> 172.21.15.163:0/2399390069 conn(0x7ff5340f2470 msgr2=0x7ff534106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:38.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.121+0000 7ff539e40700 1 -- 172.21.15.163:0/2399390069 shutdown_connections 2024-02-19T21:19:38.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.121+0000 7ff539e40700 1 -- 172.21.15.163:0/2399390069 wait complete. 2024-02-19T21:19:38.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:38.121+0000 7ff539e40700 1 librados: shutdown 2024-02-19T21:19:38.150 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:36.615749+0000 osd.7 (osd.7) 359 : cluster [DBG] 16.1e scrub starts 2024-02-19T21:19:38.150 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:36.618946+0000 osd.7 (osd.7) 360 : cluster [DBG] 16.1e scrub ok 2024-02-19T21:19:38.150 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:36.726310+0000 osd.1 (osd.1) 417 : cluster [DBG] 6.1 scrub starts 2024-02-19T21:19:38.150 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:36.729353+0000 osd.1 (osd.1) 418 : cluster [DBG] 6.1 scrub ok 2024-02-19T21:19:38.150 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:36.851129+0000 osd.1 (osd.1) 419 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:19:38.150 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:36.851314+0000 osd.1 (osd.1) 420 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:19:38.150 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:37.266572+0000 osd.3 (osd.3) 415 : cluster [DBG] 4.10 scrub starts 2024-02-19T21:19:38.150 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:37 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:37.268099+0000 osd.3 (osd.3) 416 : cluster [DBG] 4.10 scrub ok 2024-02-19T21:19:38.720 INFO:teuthology.orchestra.run.smithi163.stdout:227633266911 2024-02-19T21:19:38.720 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-19T21:19:38.992 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:38.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:37.375130+0000 mgr.y (mgr.14550) 731 : cluster [DBG] pgmap v1320: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:38.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:37.826822+0000 mon.a (mon.0) 3873 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2024-02-19T21:19:38.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:38.306702+0000 osd.3 (osd.3) 417 : cluster [DBG] 16.16 scrub starts 2024-02-19T21:19:38.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:38.307707+0000 osd.3 (osd.3) 418 : cluster [DBG] 16.16 scrub ok 2024-02-19T21:19:38.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:38.484033+0000 osd.5 (osd.5) 380 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:19:38.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:38.484298+0000 osd.5 (osd.5) 381 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:19:39.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:37.375130+0000 mgr.y (mgr.14550) 731 : cluster [DBG] pgmap v1320: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:39.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:37.826822+0000 mon.a (mon.0) 3873 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2024-02-19T21:19:39.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:38.306702+0000 osd.3 (osd.3) 417 : cluster [DBG] 16.16 scrub starts 2024-02-19T21:19:39.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:38.307707+0000 osd.3 (osd.3) 418 : cluster [DBG] 16.16 scrub ok 2024-02-19T21:19:39.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:38.484033+0000 osd.5 (osd.5) 380 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:19:39.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:38 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:38.484298+0000 osd.5 (osd.5) 381 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:19:39.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:37.375130+0000 mgr.y (mgr.14550) 731 : cluster [DBG] pgmap v1320: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:39.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:37.826822+0000 mon.a (mon.0) 3873 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2024-02-19T21:19:39.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:38.306702+0000 osd.3 (osd.3) 417 : cluster [DBG] 16.16 scrub starts 2024-02-19T21:19:39.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:38.307707+0000 osd.3 (osd.3) 418 : cluster [DBG] 16.16 scrub ok 2024-02-19T21:19:39.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:38.484033+0000 osd.5 (osd.5) 380 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:19:39.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:38 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:38.484298+0000 osd.5 (osd.5) 381 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:19:40.226 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:38.322170+0000 osd.0 (osd.0) 184 : cluster [DBG] 4.b scrub starts 2024-02-19T21:19:40.226 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:38.323632+0000 osd.0 (osd.0) 185 : cluster [DBG] 4.b scrub ok 2024-02-19T21:19:40.226 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:38.828451+0000 mon.a (mon.0) 3874 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2024-02-19T21:19:40.227 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:38.322170+0000 osd.0 (osd.0) 184 : cluster [DBG] 4.b scrub starts 2024-02-19T21:19:40.227 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:38.323632+0000 osd.0 (osd.0) 185 : cluster [DBG] 4.b scrub ok 2024-02-19T21:19:40.227 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:39 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:38.828451+0000 mon.a (mon.0) 3874 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2024-02-19T21:19:40.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:38.322170+0000 osd.0 (osd.0) 184 : cluster [DBG] 4.b scrub starts 2024-02-19T21:19:40.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:38.323632+0000 osd.0 (osd.0) 185 : cluster [DBG] 4.b scrub ok 2024-02-19T21:19:40.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:39 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:38.828451+0000 mon.a (mon.0) 3874 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2024-02-19T21:19:40.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.746+0000 7f7f08e26700 1 -- 172.21.15.163:0/1146005825 >> v1:172.21.15.163:6789/0 conn(0x7f7f040ff670 legacy=0x7f7f04101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:40.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.747+0000 7f7f08e26700 1 -- 172.21.15.163:0/1146005825 shutdown_connections 2024-02-19T21:19:40.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.747+0000 7f7f08e26700 1 -- 172.21.15.163:0/1146005825 >> 172.21.15.163:0/1146005825 conn(0x7f7f040f2470 msgr2=0x7f7f040f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:40.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.747+0000 7f7f08e26700 1 -- 172.21.15.163:0/1146005825 shutdown_connections 2024-02-19T21:19:40.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.747+0000 7f7f08e26700 1 -- 172.21.15.163:0/1146005825 wait complete. 2024-02-19T21:19:40.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.747+0000 7f7f08e26700 1 librados: starting msgr at 2024-02-19T21:19:40.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.747+0000 7f7f08e26700 1 librados: starting objecter 2024-02-19T21:19:40.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.747+0000 7f7f08e26700 1 Processor -- start 2024-02-19T21:19:40.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.747+0000 7f7f08e26700 1 -- start start 2024-02-19T21:19:40.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.747+0000 7f7f08e26700 1 librados: setting wanted keys 2024-02-19T21:19:40.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.747+0000 7f7f08e26700 1 librados: calling monclient init 2024-02-19T21:19:40.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7f08e26700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7f04119350 con 0x7f7f040ff670 2024-02-19T21:19:40.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7f08e26700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7f0411a530 con 0x7f7f041031b0 2024-02-19T21:19:40.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7f08e26700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7f0411b710 con 0x7f7f040fc870 2024-02-19T21:19:40.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7f0259c700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f7f040fc870 0x7f7f040f8400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:50590/0 (socket says 172.21.15.163:50590) 2024-02-19T21:19:40.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7f0259c700 1 -- 172.21.15.163:0/649463557 learned_addr learned my addr 172.21.15.163:0/649463557 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:40.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7f01d9b700 1 --1- 172.21.15.163:0/649463557 >> v1:172.21.15.163:6789/0 conn(0x7f7f040ff670 0x7f7f040f8b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46888/0 (socket says 172.21.15.163:46888) 2024-02-19T21:19:40.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 10 client.?.objecter ms_handle_connect 0x7f7f040fc870 2024-02-19T21:19:40.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:40.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 10 client.?.objecter ms_handle_connect 0x7f7f040ff670 2024-02-19T21:19:40.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:40.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 10 client.?.objecter ms_handle_connect 0x7f7f041031b0 2024-02-19T21:19:40.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:40.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377190644 0 0) 0x7f7f0411b710 con 0x7f7f040fc870 2024-02-19T21:19:40.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eec0038e0 con 0x7f7f040fc870 2024-02-19T21:19:40.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739346507 0 0) 0x7f7f04119350 con 0x7f7f040ff670 2024-02-19T21:19:40.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f0411b710 con 0x7f7f040ff670 2024-02-19T21:19:40.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748454572 0 0) 0x7f7f0411a530 con 0x7f7f041031b0 2024-02-19T21:19:40.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f04119350 con 0x7f7f041031b0 2024-02-19T21:19:40.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3919471211 0 0) 0x7f7eec0038e0 con 0x7f7f040fc870 2024-02-19T21:19:40.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f0411a530 con 0x7f7f040fc870 2024-02-19T21:19:40.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7ef4001990 con 0x7f7f040fc870 2024-02-19T21:19:40.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4221948301 0 0) 0x7f7f0411b710 con 0x7f7f040ff670 2024-02-19T21:19:40.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7eec0038e0 con 0x7f7f040ff670 2024-02-19T21:19:40.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7ef8001f50 con 0x7f7f040ff670 2024-02-19T21:19:40.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4090989365 0 0) 0x7f7f04119350 con 0x7f7f041031b0 2024-02-19T21:19:40.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f0411b710 con 0x7f7f041031b0 2024-02-19T21:19:40.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7efc001bd0 con 0x7f7f041031b0 2024-02-19T21:19:40.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 610059737 0 0) 0x7f7f0411a530 con 0x7f7f040fc870 2024-02-19T21:19:40.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 >> v1:172.21.15.193:6789/0 conn(0x7f7f041031b0 legacy=0x7f7f04117c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:40.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 >> v1:172.21.15.163:6789/0 conn(0x7f7f040ff670 legacy=0x7f7f040f8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:40.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0411c910 con 0x7f7f040fc870 2024-02-19T21:19:40.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7f08e26700 1 -- 172.21.15.163:0/649463557 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f04119540 con 0x7f7f040fc870 2024-02-19T21:19:40.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7f08e26700 10 client.50864.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:40.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7f08e26700 1 -- 172.21.15.163:0/649463557 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f04119a30 con 0x7f7f040fc870 2024-02-19T21:19:40.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7f08e26700 1 librados: init done 2024-02-19T21:19:40.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.748+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f7ef4001d20 con 0x7f7f040fc870 2024-02-19T21:19:40.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.749+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f7ef4005bc0 con 0x7f7f040fc870 2024-02-19T21:19:40.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.749+0000 7f7f08e26700 1 -- 172.21.15.163:0/649463557 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee4004f20 con 0x7f7f040fc870 2024-02-19T21:19:40.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.750+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f7ef4002ba0 con 0x7f7f040fc870 2024-02-19T21:19:40.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.750+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(852..852 src has 796..852) v4 ==== 7419+0+0 (unknown 2879669749 0 0) 0x7f7ef4078a80 con 0x7f7f040fc870 2024-02-19T21:19:40.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.750+0000 7f7ef37fe700 10 client.50864.objecter ms_dispatch 0x7f7f04001550 osd_map(852..852 src has 796..852) v4 2024-02-19T21:19:40.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.750+0000 7f7ef37fe700 3 client.50864.objecter handle_osd_map got epochs [852,852] > 0 2024-02-19T21:19:40.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.750+0000 7f7ef37fe700 3 client.50864.objecter handle_osd_map decoding full epoch 852 2024-02-19T21:19:40.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.750+0000 7f7ef37fe700 20 client.50864.objecter dump_active .. 0 homeless 2024-02-19T21:19:40.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.750+0000 7f7ef37fe700 10 client.50864.objecter ms_handle_connect 0x7f7eec063c70 2024-02-19T21:19:40.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.754+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f7ef404c130 con 0x7f7f040fc870 2024-02-19T21:19:40.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.923+0000 7f7f08e26700 1 -- 172.21.15.163:0/649463557 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7ee40051d0 con 0x7f7f040fc870 2024-02-19T21:19:40.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.923+0000 7f7ef37fe700 1 -- 172.21.15.163:0/649463557 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 3462156553) 0x7f7ef404a540 con 0x7f7f040fc870 2024-02-19T21:19:40.926 INFO:teuthology.orchestra.run.smithi163.stdout:34359738626 2024-02-19T21:19:40.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.925+0000 7f7f08e26700 10 librados: watch_flush enter 2024-02-19T21:19:40.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.925+0000 7f7f08e26700 10 librados: watch_flush exit 2024-02-19T21:19:40.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.925+0000 7f7f08e26700 20 client.50864.objecter shutdown clearing up homeless session... 2024-02-19T21:19:40.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.925+0000 7f7f08e26700 10 client.50864.objecter successfully canceled tick 2024-02-19T21:19:40.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.925+0000 7f7f08e26700 1 -- 172.21.15.163:0/649463557 >> v1:172.21.15.163:6800/1552514681 conn(0x7f7eec063c70 legacy=0x7f7eec066130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:40.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.925+0000 7f7f08e26700 1 -- 172.21.15.163:0/649463557 >> v1:172.21.15.163:6790/0 conn(0x7f7f040fc870 legacy=0x7f7f040f8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:40.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.925+0000 7f7f08e26700 1 -- 172.21.15.163:0/649463557 shutdown_connections 2024-02-19T21:19:40.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.925+0000 7f7f08e26700 1 -- 172.21.15.163:0/649463557 >> 172.21.15.163:0/649463557 conn(0x7f7f040f2470 msgr2=0x7f7f04106620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:40.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.925+0000 7f7f08e26700 1 -- 172.21.15.163:0/649463557 shutdown_connections 2024-02-19T21:19:40.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.925+0000 7f7f08e26700 1 -- 172.21.15.163:0/649463557 wait complete. 2024-02-19T21:19:40.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:40.926+0000 7f7f08e26700 1 librados: shutdown 2024-02-19T21:19:41.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:39.375521+0000 mgr.y (mgr.14550) 732 : cluster [DBG] pgmap v1323: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:19:41.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:39.377560+0000 osd.4 (osd.4) 349 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:19:41.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:39.378777+0000 osd.4 (osd.4) 350 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:19:41.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:40.347168+0000 osd.4 (osd.4) 351 : cluster [DBG] 6.2 deep-scrub starts 2024-02-19T21:19:41.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:40.348270+0000 osd.4 (osd.4) 352 : cluster [DBG] 6.2 deep-scrub ok 2024-02-19T21:19:41.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:40.507697+0000 osd.6 (osd.6) 335 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:19:41.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:40.508615+0000 osd.6 (osd.6) 336 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:19:41.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:40.657622+0000 osd.5 (osd.5) 382 : cluster [DBG] 16.3 deep-scrub starts 2024-02-19T21:19:41.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:40.661297+0000 osd.5 (osd.5) 383 : cluster [DBG] 16.3 deep-scrub ok 2024-02-19T21:19:41.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:39.375521+0000 mgr.y (mgr.14550) 732 : cluster [DBG] pgmap v1323: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:19:41.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:39.377560+0000 osd.4 (osd.4) 349 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:19:41.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:39.378777+0000 osd.4 (osd.4) 350 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:19:41.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:40.347168+0000 osd.4 (osd.4) 351 : cluster [DBG] 6.2 deep-scrub starts 2024-02-19T21:19:41.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:40.348270+0000 osd.4 (osd.4) 352 : cluster [DBG] 6.2 deep-scrub ok 2024-02-19T21:19:41.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:40.507697+0000 osd.6 (osd.6) 335 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:19:41.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:40.508615+0000 osd.6 (osd.6) 336 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:19:41.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:40.657622+0000 osd.5 (osd.5) 382 : cluster [DBG] 16.3 deep-scrub starts 2024-02-19T21:19:41.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:40.661297+0000 osd.5 (osd.5) 383 : cluster [DBG] 16.3 deep-scrub ok 2024-02-19T21:19:41.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:19:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:19:41.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:39.375521+0000 mgr.y (mgr.14550) 732 : cluster [DBG] pgmap v1323: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:19:41.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:39.377560+0000 osd.4 (osd.4) 349 : cluster [DBG] 5.1c scrub starts 2024-02-19T21:19:41.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:39.378777+0000 osd.4 (osd.4) 350 : cluster [DBG] 5.1c scrub ok 2024-02-19T21:19:41.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:40.347168+0000 osd.4 (osd.4) 351 : cluster [DBG] 6.2 deep-scrub starts 2024-02-19T21:19:41.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:40.348270+0000 osd.4 (osd.4) 352 : cluster [DBG] 6.2 deep-scrub ok 2024-02-19T21:19:41.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:40.507697+0000 osd.6 (osd.6) 335 : cluster [DBG] 2.1a scrub starts 2024-02-19T21:19:41.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:40.508615+0000 osd.6 (osd.6) 336 : cluster [DBG] 2.1a scrub ok 2024-02-19T21:19:41.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:40.657622+0000 osd.5 (osd.5) 382 : cluster [DBG] 16.3 deep-scrub starts 2024-02-19T21:19:41.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:40 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:40.661297+0000 osd.5 (osd.5) 383 : cluster [DBG] 16.3 deep-scrub ok 2024-02-19T21:19:41.652 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738622 got 34359738626 for osd.0 2024-02-19T21:19:41.653 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-19T21:19:41.922 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:40.458783+0000 osd.4 (osd.4) 353 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:19:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:40.459105+0000 osd.4 (osd.4) 354 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:19:41.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:40.759465+0000 osd.1 (osd.1) 421 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:19:42.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:40.760275+0000 osd.1 (osd.1) 422 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:19:42.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:40.851588+0000 mon.a (mon.0) 3875 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2024-02-19T21:19:42.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:40.921859+0000 mgr.y (mgr.14550) 733 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:42.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:40.925108+0000 mon.c (mon.2) 451 : audit [DBG] from='client.? v1:172.21.15.163:0/649463557' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-19T21:19:42.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:41.535926+0000 osd.6 (osd.6) 337 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:19:42.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:41.536876+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:19:42.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:40.458783+0000 osd.4 (osd.4) 353 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:19:42.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:40.459105+0000 osd.4 (osd.4) 354 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:19:42.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:40.759465+0000 osd.1 (osd.1) 421 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:19:42.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:40.760275+0000 osd.1 (osd.1) 422 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:19:42.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:40.851588+0000 mon.a (mon.0) 3875 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2024-02-19T21:19:42.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:40.921859+0000 mgr.y (mgr.14550) 733 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:42.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:40.925108+0000 mon.c (mon.2) 451 : audit [DBG] from='client.? v1:172.21.15.163:0/649463557' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-19T21:19:42.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:41.535926+0000 osd.6 (osd.6) 337 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:19:42.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:41.536876+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:19:42.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:40.458783+0000 osd.4 (osd.4) 353 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:19:42.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:40.459105+0000 osd.4 (osd.4) 354 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:19:42.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:40.759465+0000 osd.1 (osd.1) 421 : cluster [DBG] 2.15 scrub starts 2024-02-19T21:19:42.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:40.760275+0000 osd.1 (osd.1) 422 : cluster [DBG] 2.15 scrub ok 2024-02-19T21:19:42.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:40.851588+0000 mon.a (mon.0) 3875 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2024-02-19T21:19:42.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:40.921859+0000 mgr.y (mgr.14550) 733 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:42.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:40.925108+0000 mon.c (mon.2) 451 : audit [DBG] from='client.? v1:172.21.15.163:0/649463557' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-19T21:19:42.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:41.535926+0000 osd.6 (osd.6) 337 : cluster [DBG] 5.2 scrub starts 2024-02-19T21:19:42.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:41 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:41.536876+0000 osd.6 (osd.6) 338 : cluster [DBG] 5.2 scrub ok 2024-02-19T21:19:43.105 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:41.375900+0000 mgr.y (mgr.14550) 734 : cluster [DBG] pgmap v1325: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:43.106 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:42 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:41.375900+0000 mgr.y (mgr.14550) 734 : cluster [DBG] pgmap v1325: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:43.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:42 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:41.375900+0000 mgr.y (mgr.14550) 734 : cluster [DBG] pgmap v1325: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:43.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.571+0000 7f3cf7cd8700 1 -- 172.21.15.163:0/2550430924 >> v1:172.21.15.163:6790/0 conn(0x7f3cf01042c0 legacy=0x7f3cf0106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:43.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.571+0000 7f3cf7cd8700 1 -- 172.21.15.163:0/2550430924 shutdown_connections 2024-02-19T21:19:43.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.571+0000 7f3cf7cd8700 1 -- 172.21.15.163:0/2550430924 >> 172.21.15.163:0/2550430924 conn(0x7f3cf00f35a0 msgr2=0x7f3cf00f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:43.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.571+0000 7f3cf7cd8700 1 -- 172.21.15.163:0/2550430924 shutdown_connections 2024-02-19T21:19:43.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.571+0000 7f3cf7cd8700 1 -- 172.21.15.163:0/2550430924 wait complete. 2024-02-19T21:19:43.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3cf7cd8700 1 librados: starting msgr at 2024-02-19T21:19:43.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3cf7cd8700 1 librados: starting objecter 2024-02-19T21:19:43.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3cf7cd8700 1 Processor -- start 2024-02-19T21:19:43.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3cf7cd8700 1 -- start start 2024-02-19T21:19:43.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3cf7cd8700 1 librados: setting wanted keys 2024-02-19T21:19:43.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3cf7cd8700 1 librados: calling monclient init 2024-02-19T21:19:43.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3cf7cd8700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3cf0117350 con 0x7f3cf0100780 2024-02-19T21:19:43.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3cf7cd8700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3cf0118530 con 0x7f3cf01042c0 2024-02-19T21:19:43.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3cf7cd8700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3cf0119730 con 0x7f3cf00fd980 2024-02-19T21:19:43.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3cf5a74700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f3cf00fd980 0x7f3cf00f9960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:50616/0 (socket says 172.21.15.163:50616) 2024-02-19T21:19:43.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3cf5a74700 1 -- 172.21.15.163:0/3765693670 learned_addr learned my addr 172.21.15.163:0/3765693670 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:43.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3cf5273700 1 --1- 172.21.15.163:0/3765693670 >> v1:172.21.15.163:6789/0 conn(0x7f3cf0100780 0x7f3cf00fa0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:46904/0 (socket says 172.21.15.163:46904) 2024-02-19T21:19:43.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3ce6ffd700 10 client.?.objecter ms_handle_connect 0x7f3cf00fd980 2024-02-19T21:19:43.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3ce6ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:43.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3ce6ffd700 10 client.?.objecter ms_handle_connect 0x7f3cf0100780 2024-02-19T21:19:43.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3ce6ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:43.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530292891 0 0) 0x7f3cf0119730 con 0x7f3cf00fd980 2024-02-19T21:19:43.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cdc003030 con 0x7f3cf00fd980 2024-02-19T21:19:43.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669512215 0 0) 0x7f3cf0117350 con 0x7f3cf0100780 2024-02-19T21:19:43.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.572+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cf0119730 con 0x7f3cf0100780 2024-02-19T21:19:43.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 10 client.?.objecter ms_handle_connect 0x7f3cf01042c0 2024-02-19T21:19:43.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:43.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2090060004 0 0) 0x7f3cdc003030 con 0x7f3cf00fd980 2024-02-19T21:19:43.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cf0117350 con 0x7f3cf00fd980 2024-02-19T21:19:43.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3ce0001940 con 0x7f3cf00fd980 2024-02-19T21:19:43.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171016136 0 0) 0x7f3cf0118530 con 0x7f3cf01042c0 2024-02-19T21:19:43.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cdc003030 con 0x7f3cf01042c0 2024-02-19T21:19:43.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2827999293 0 0) 0x7f3cf0119730 con 0x7f3cf0100780 2024-02-19T21:19:43.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cf0118530 con 0x7f3cf0100780 2024-02-19T21:19:43.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3cec001a30 con 0x7f3cf0100780 2024-02-19T21:19:43.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 718595437 0 0) 0x7f3cf0117350 con 0x7f3cf00fd980 2024-02-19T21:19:43.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 >> v1:172.21.15.193:6789/0 conn(0x7f3cf01042c0 legacy=0x7f3cf0115a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:43.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 >> v1:172.21.15.163:6789/0 conn(0x7f3cf0100780 legacy=0x7f3cf00fa0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:43.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf011a930 con 0x7f3cf00fd980 2024-02-19T21:19:43.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3cf7cd8700 1 -- 172.21.15.163:0/3765693670 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf0117540 con 0x7f3cf00fd980 2024-02-19T21:19:43.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3cf7cd8700 10 client.50870.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:43.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3cf7cd8700 1 -- 172.21.15.163:0/3765693670 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf01179b0 con 0x7f3cf00fd980 2024-02-19T21:19:43.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3cf7cd8700 1 librados: init done 2024-02-19T21:19:43.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f3ce0001c90 con 0x7f3cf00fd980 2024-02-19T21:19:43.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.573+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f3ce0005b50 con 0x7f3cf00fd980 2024-02-19T21:19:43.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.574+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f3ce00189e0 con 0x7f3cf00fd980 2024-02-19T21:19:43.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.574+0000 7f3cf7cd8700 1 -- 172.21.15.163:0/3765693670 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf005f6a0 con 0x7f3cf00fd980 2024-02-19T21:19:43.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.575+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(853..853 src has 796..853) v4 ==== 7419+0+0 (unknown 1683610851 0 0) 0x7f3ce00799b0 con 0x7f3cf00fd980 2024-02-19T21:19:43.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.575+0000 7f3ce6ffd700 10 client.50870.objecter ms_dispatch 0x7f3cf0001550 osd_map(853..853 src has 796..853) v4 2024-02-19T21:19:43.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.575+0000 7f3ce6ffd700 3 client.50870.objecter handle_osd_map got epochs [853,853] > 0 2024-02-19T21:19:43.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.575+0000 7f3ce6ffd700 3 client.50870.objecter handle_osd_map decoding full epoch 853 2024-02-19T21:19:43.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.575+0000 7f3ce6ffd700 20 client.50870.objecter dump_active .. 0 homeless 2024-02-19T21:19:43.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.575+0000 7f3ce6ffd700 10 client.50870.objecter ms_handle_connect 0x7f3cdc063cb0 2024-02-19T21:19:43.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.579+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f3ce004d080 con 0x7f3cf00fd980 2024-02-19T21:19:43.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.746+0000 7f3cf7cd8700 1 -- 172.21.15.163:0/3765693670 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3cf0109fc0 con 0x7f3cf00fd980 2024-02-19T21:19:43.748 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.746+0000 7f3ce6ffd700 1 -- 172.21.15.163:0/3765693670 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 2856532818) 0x7f3ce004b490 con 0x7f3cf00fd980 2024-02-19T21:19:43.749 INFO:teuthology.orchestra.run.smithi163.stdout:55834575101 2024-02-19T21:19:43.749 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:19:43] "GET /metrics HTTP/1.1" 200 187977 "" "Prometheus/2.33.4" 2024-02-19T21:19:43.750 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.748+0000 7f3cf7cd8700 10 librados: watch_flush enter 2024-02-19T21:19:43.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.748+0000 7f3cf7cd8700 10 librados: watch_flush exit 2024-02-19T21:19:43.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.748+0000 7f3cf7cd8700 20 client.50870.objecter shutdown clearing up homeless session... 2024-02-19T21:19:43.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.748+0000 7f3cf7cd8700 10 client.50870.objecter successfully canceled tick 2024-02-19T21:19:43.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.748+0000 7f3cf7cd8700 1 -- 172.21.15.163:0/3765693670 >> v1:172.21.15.163:6800/1552514681 conn(0x7f3cdc063cb0 legacy=0x7f3cdc066170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:43.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.748+0000 7f3cf7cd8700 1 -- 172.21.15.163:0/3765693670 >> v1:172.21.15.163:6790/0 conn(0x7f3cf00fd980 legacy=0x7f3cf00f9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:43.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.748+0000 7f3cf7cd8700 1 -- 172.21.15.163:0/3765693670 shutdown_connections 2024-02-19T21:19:43.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.748+0000 7f3cf7cd8700 1 -- 172.21.15.163:0/3765693670 >> 172.21.15.163:0/3765693670 conn(0x7f3cf00f35a0 msgr2=0x7f3cf0107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:43.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.748+0000 7f3cf7cd8700 1 -- 172.21.15.163:0/3765693670 shutdown_connections 2024-02-19T21:19:43.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.748+0000 7f3cf7cd8700 1 -- 172.21.15.163:0/3765693670 wait complete. 2024-02-19T21:19:43.751 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:43.748+0000 7f3cf7cd8700 1 librados: shutdown 2024-02-19T21:19:44.090 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:42.380611+0000 osd.4 (osd.4) 355 : cluster [DBG] 6.17 deep-scrub starts 2024-02-19T21:19:44.090 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:42.381649+0000 osd.4 (osd.4) 356 : cluster [DBG] 6.17 deep-scrub ok 2024-02-19T21:19:44.090 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:42.441594+0000 osd.3 (osd.3) 419 : cluster [DBG] 16.4 scrub starts 2024-02-19T21:19:44.090 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:42.443156+0000 osd.3 (osd.3) 420 : cluster [DBG] 16.4 scrub ok 2024-02-19T21:19:44.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:42.484371+0000 osd.7 (osd.7) 361 : cluster [DBG] 5.1a deep-scrub starts 2024-02-19T21:19:44.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:42.487263+0000 osd.7 (osd.7) 362 : cluster [DBG] 5.1a deep-scrub ok 2024-02-19T21:19:44.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:43.681815+0000 osd.5 (osd.5) 384 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:19:44.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:43.684701+0000 osd.5 (osd.5) 385 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:19:44.091 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:43.747584+0000 mon.c (mon.2) 452 : audit [DBG] from='client.? v1:172.21.15.163:0/3765693670' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-19T21:19:44.091 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:42.380611+0000 osd.4 (osd.4) 355 : cluster [DBG] 6.17 deep-scrub starts 2024-02-19T21:19:44.091 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:42.381649+0000 osd.4 (osd.4) 356 : cluster [DBG] 6.17 deep-scrub ok 2024-02-19T21:19:44.091 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:42.441594+0000 osd.3 (osd.3) 419 : cluster [DBG] 16.4 scrub starts 2024-02-19T21:19:44.091 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:42.443156+0000 osd.3 (osd.3) 420 : cluster [DBG] 16.4 scrub ok 2024-02-19T21:19:44.091 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:42.484371+0000 osd.7 (osd.7) 361 : cluster [DBG] 5.1a deep-scrub starts 2024-02-19T21:19:44.091 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:42.487263+0000 osd.7 (osd.7) 362 : cluster [DBG] 5.1a deep-scrub ok 2024-02-19T21:19:44.091 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:43.681815+0000 osd.5 (osd.5) 384 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:19:44.092 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:43.684701+0000 osd.5 (osd.5) 385 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:19:44.092 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:43 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:43.747584+0000 mon.c (mon.2) 452 : audit [DBG] from='client.? v1:172.21.15.163:0/3765693670' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-19T21:19:44.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:42.380611+0000 osd.4 (osd.4) 355 : cluster [DBG] 6.17 deep-scrub starts 2024-02-19T21:19:44.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:42.381649+0000 osd.4 (osd.4) 356 : cluster [DBG] 6.17 deep-scrub ok 2024-02-19T21:19:44.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:42.441594+0000 osd.3 (osd.3) 419 : cluster [DBG] 16.4 scrub starts 2024-02-19T21:19:44.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:42.443156+0000 osd.3 (osd.3) 420 : cluster [DBG] 16.4 scrub ok 2024-02-19T21:19:44.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:42.484371+0000 osd.7 (osd.7) 361 : cluster [DBG] 5.1a deep-scrub starts 2024-02-19T21:19:44.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:42.487263+0000 osd.7 (osd.7) 362 : cluster [DBG] 5.1a deep-scrub ok 2024-02-19T21:19:44.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:43.681815+0000 osd.5 (osd.5) 384 : cluster [DBG] 2.16 scrub starts 2024-02-19T21:19:44.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:43.684701+0000 osd.5 (osd.5) 385 : cluster [DBG] 2.16 scrub ok 2024-02-19T21:19:44.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:43 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:43.747584+0000 mon.c (mon.2) 452 : audit [DBG] from='client.? v1:172.21.15.163:0/3765693670' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-19T21:19:44.450 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575097 got 55834575101 for osd.1 2024-02-19T21:19:44.451 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-19T21:19:44.730 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:45.200 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:43.376459+0000 mgr.y (mgr.14550) 735 : cluster [DBG] pgmap v1326: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:45.200 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:43.473551+0000 osd.3 (osd.3) 421 : cluster [DBG] 6.b scrub starts 2024-02-19T21:19:45.201 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:43.474797+0000 osd.3 (osd.3) 422 : cluster [DBG] 6.b scrub ok 2024-02-19T21:19:45.201 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:43.700614+0000 osd.1 (osd.1) 423 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:19:45.201 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:43.701485+0000 osd.1 (osd.1) 424 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:19:45.201 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:43.376459+0000 mgr.y (mgr.14550) 735 : cluster [DBG] pgmap v1326: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:45.202 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:43.473551+0000 osd.3 (osd.3) 421 : cluster [DBG] 6.b scrub starts 2024-02-19T21:19:45.202 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:43.474797+0000 osd.3 (osd.3) 422 : cluster [DBG] 6.b scrub ok 2024-02-19T21:19:45.202 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:43.700614+0000 osd.1 (osd.1) 423 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:19:45.202 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:43.701485+0000 osd.1 (osd.1) 424 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:19:45.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:43.376459+0000 mgr.y (mgr.14550) 735 : cluster [DBG] pgmap v1326: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:45.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:43.473551+0000 osd.3 (osd.3) 421 : cluster [DBG] 6.b scrub starts 2024-02-19T21:19:45.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:43.474797+0000 osd.3 (osd.3) 422 : cluster [DBG] 6.b scrub ok 2024-02-19T21:19:45.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:43.700614+0000 osd.1 (osd.1) 423 : cluster [DBG] 2.6 scrub starts 2024-02-19T21:19:45.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:44 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:43.701485+0000 osd.1 (osd.1) 424 : cluster [DBG] 2.6 scrub ok 2024-02-19T21:19:45.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:44.461824+0000 osd.7 (osd.7) 363 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:19:45.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:44.465219+0000 osd.7 (osd.7) 364 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:19:46.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:44.690590+0000 osd.1 (osd.1) 425 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:19:46.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:44.691514+0000 osd.1 (osd.1) 426 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:19:46.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:45.176458+0000 osd.0 (osd.0) 186 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:19:46.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:45.177497+0000 osd.0 (osd.0) 187 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:19:46.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:45.497620+0000 osd.6 (osd.6) 339 : cluster [DBG] 16.d scrub starts 2024-02-19T21:19:46.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:45.499991+0000 osd.6 (osd.6) 340 : cluster [DBG] 16.d scrub ok 2024-02-19T21:19:46.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:45.676431+0000 osd.5 (osd.5) 386 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:19:46.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:45.679376+0000 osd.5 (osd.5) 387 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:19:46.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:44.461824+0000 osd.7 (osd.7) 363 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:19:46.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:44.465219+0000 osd.7 (osd.7) 364 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:19:46.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:44.690590+0000 osd.1 (osd.1) 425 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:19:46.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:44.691514+0000 osd.1 (osd.1) 426 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:19:46.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:45.176458+0000 osd.0 (osd.0) 186 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:19:46.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:45.177497+0000 osd.0 (osd.0) 187 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:19:46.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:45.497620+0000 osd.6 (osd.6) 339 : cluster [DBG] 16.d scrub starts 2024-02-19T21:19:46.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:45.499991+0000 osd.6 (osd.6) 340 : cluster [DBG] 16.d scrub ok 2024-02-19T21:19:46.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:45.676431+0000 osd.5 (osd.5) 386 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:19:46.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:45.679376+0000 osd.5 (osd.5) 387 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:19:46.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:44.461824+0000 osd.7 (osd.7) 363 : cluster [DBG] 6.8 scrub starts 2024-02-19T21:19:46.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:44.465219+0000 osd.7 (osd.7) 364 : cluster [DBG] 6.8 scrub ok 2024-02-19T21:19:46.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:44.690590+0000 osd.1 (osd.1) 425 : cluster [DBG] 6.1d scrub starts 2024-02-19T21:19:46.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:44.691514+0000 osd.1 (osd.1) 426 : cluster [DBG] 6.1d scrub ok 2024-02-19T21:19:46.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:45.176458+0000 osd.0 (osd.0) 186 : cluster [DBG] 2.1f scrub starts 2024-02-19T21:19:46.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:45.177497+0000 osd.0 (osd.0) 187 : cluster [DBG] 2.1f scrub ok 2024-02-19T21:19:46.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:45.497620+0000 osd.6 (osd.6) 339 : cluster [DBG] 16.d scrub starts 2024-02-19T21:19:46.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:45.499991+0000 osd.6 (osd.6) 340 : cluster [DBG] 16.d scrub ok 2024-02-19T21:19:46.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:45.676431+0000 osd.5 (osd.5) 386 : cluster [DBG] 5.1b scrub starts 2024-02-19T21:19:46.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:45 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:45.679376+0000 osd.5 (osd.5) 387 : cluster [DBG] 5.1b scrub ok 2024-02-19T21:19:46.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.576+0000 7f61c5bb2700 1 -- 172.21.15.163:0/2570514237 >> v1:172.21.15.163:6790/0 conn(0x7f61c0100780 legacy=0x7f61c0102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:46.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.577+0000 7f61c5bb2700 1 -- 172.21.15.163:0/2570514237 shutdown_connections 2024-02-19T21:19:46.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.577+0000 7f61c5bb2700 1 -- 172.21.15.163:0/2570514237 >> 172.21.15.163:0/2570514237 conn(0x7f61c00f35a0 msgr2=0x7f61c00f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:46.602 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.577+0000 7f61c5bb2700 1 -- 172.21.15.163:0/2570514237 shutdown_connections 2024-02-19T21:19:46.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.577+0000 7f61c5bb2700 1 -- 172.21.15.163:0/2570514237 wait complete. 2024-02-19T21:19:46.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.577+0000 7f61c5bb2700 1 librados: starting msgr at 2024-02-19T21:19:46.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.577+0000 7f61c5bb2700 1 librados: starting objecter 2024-02-19T21:19:46.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.577+0000 7f61c5bb2700 1 Processor -- start 2024-02-19T21:19:46.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.577+0000 7f61c5bb2700 1 -- start start 2024-02-19T21:19:46.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.577+0000 7f61c5bb2700 1 librados: setting wanted keys 2024-02-19T21:19:46.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.577+0000 7f61c5bb2700 1 librados: calling monclient init 2024-02-19T21:19:46.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61c5bb2700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f61c0117370 con 0x7f61c01042c0 2024-02-19T21:19:46.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61c5bb2700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f61c0118550 con 0x7f61c0100780 2024-02-19T21:19:46.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61c5bb2700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f61c0119750 con 0x7f61c00fd980 2024-02-19T21:19:46.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bf7fe700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f61c00fd980 0x7f61c00f9b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:48478/0 (socket says 172.21.15.163:48478) 2024-02-19T21:19:46.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bf7fe700 1 -- 172.21.15.163:0/1549781119 learned_addr learned my addr 172.21.15.163:0/1549781119 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:46.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 10 client.?.objecter ms_handle_connect 0x7f61c01042c0 2024-02-19T21:19:46.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:46.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 10 client.?.objecter ms_handle_connect 0x7f61c00fd980 2024-02-19T21:19:46.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:46.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 10 client.?.objecter ms_handle_connect 0x7f61c0100780 2024-02-19T21:19:46.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:46.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265978007 0 0) 0x7f61c0117370 con 0x7f61c01042c0 2024-02-19T21:19:46.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a8003030 con 0x7f61c01042c0 2024-02-19T21:19:46.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737752436 0 0) 0x7f61c0119750 con 0x7f61c00fd980 2024-02-19T21:19:46.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c0117370 con 0x7f61c00fd980 2024-02-19T21:19:46.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131670918 0 0) 0x7f61c0118550 con 0x7f61c0100780 2024-02-19T21:19:46.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c0119750 con 0x7f61c0100780 2024-02-19T21:19:46.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3209539217 0 0) 0x7f61a8003030 con 0x7f61c01042c0 2024-02-19T21:19:46.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61c0118550 con 0x7f61c01042c0 2024-02-19T21:19:46.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 129291790 0 0) 0x7f61c0117370 con 0x7f61c00fd980 2024-02-19T21:19:46.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61a8003030 con 0x7f61c00fd980 2024-02-19T21:19:46.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f61b8001b80 con 0x7f61c01042c0 2024-02-19T21:19:46.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f61b00019f0 con 0x7f61c00fd980 2024-02-19T21:19:46.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4033559763 0 0) 0x7f61c0119750 con 0x7f61c0100780 2024-02-19T21:19:46.606 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61c0117370 con 0x7f61c0100780 2024-02-19T21:19:46.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f61b4001430 con 0x7f61c0100780 2024-02-19T21:19:46.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2672498767 0 0) 0x7f61c0118550 con 0x7f61c01042c0 2024-02-19T21:19:46.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 >> v1:172.21.15.163:6790/0 conn(0x7f61c00fd980 legacy=0x7f61c00f9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:46.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 >> v1:172.21.15.193:6789/0 conn(0x7f61c0100780 legacy=0x7f61c00fa230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:46.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c011a950 con 0x7f61c01042c0 2024-02-19T21:19:46.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61c5bb2700 1 -- 172.21.15.163:0/1549781119 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c0119940 con 0x7f61c01042c0 2024-02-19T21:19:46.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61c5bb2700 10 client.24150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:46.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61c5bb2700 1 -- 172.21.15.163:0/1549781119 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c0119db0 con 0x7f61c01042c0 2024-02-19T21:19:46.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.578+0000 7f61c5bb2700 1 librados: init done 2024-02-19T21:19:46.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.579+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f61b8002210 con 0x7f61c01042c0 2024-02-19T21:19:46.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.579+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f61b80035b0 con 0x7f61c01042c0 2024-02-19T21:19:46.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.579+0000 7f61c5bb2700 1 -- 172.21.15.163:0/1549781119 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61ac004f20 con 0x7f61c01042c0 2024-02-19T21:19:46.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.580+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f61b8001d70 con 0x7f61c01042c0 2024-02-19T21:19:46.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.580+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(853..853 src has 796..853) v4 ==== 7419+0+0 (unknown 1683610851 0 0) 0x7f61b8076470 con 0x7f61c01042c0 2024-02-19T21:19:46.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.580+0000 7f61bcff9700 10 client.24150.objecter ms_dispatch 0x7f61c0001550 osd_map(853..853 src has 796..853) v4 2024-02-19T21:19:46.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.580+0000 7f61bcff9700 3 client.24150.objecter handle_osd_map got epochs [853,853] > 0 2024-02-19T21:19:46.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.580+0000 7f61bcff9700 3 client.24150.objecter handle_osd_map decoding full epoch 853 2024-02-19T21:19:46.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.580+0000 7f61bcff9700 20 client.24150.objecter dump_active .. 0 homeless 2024-02-19T21:19:46.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.581+0000 7f61bcff9700 10 client.24150.objecter ms_handle_connect 0x7f61a8063c30 2024-02-19T21:19:46.609 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.584+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f61b8049ae0 con 0x7f61c01042c0 2024-02-19T21:19:46.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.749+0000 7f61c5bb2700 1 -- 172.21.15.163:0/1549781119 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f61ac0051d0 con 0x7f61c01042c0 2024-02-19T21:19:46.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.750+0000 7f61bcff9700 1 -- 172.21.15.163:0/1549781119 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 1240788364) 0x7f61b8047ef0 con 0x7f61c01042c0 2024-02-19T21:19:46.752 INFO:teuthology.orchestra.run.smithi163.stdout:77309411577 2024-02-19T21:19:46.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.751+0000 7f61c5bb2700 10 librados: watch_flush enter 2024-02-19T21:19:46.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.751+0000 7f61c5bb2700 10 librados: watch_flush exit 2024-02-19T21:19:46.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.751+0000 7f61c5bb2700 20 client.24150.objecter shutdown clearing up homeless session... 2024-02-19T21:19:46.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.751+0000 7f61c5bb2700 10 client.24150.objecter successfully canceled tick 2024-02-19T21:19:46.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.751+0000 7f61c5bb2700 1 -- 172.21.15.163:0/1549781119 >> v1:172.21.15.163:6800/1552514681 conn(0x7f61a8063c30 legacy=0x7f61a80660f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:46.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.751+0000 7f61c5bb2700 1 -- 172.21.15.163:0/1549781119 >> v1:172.21.15.163:6789/0 conn(0x7f61c01042c0 legacy=0x7f61c0115a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:46.754 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.751+0000 7f61c5bb2700 1 -- 172.21.15.163:0/1549781119 shutdown_connections 2024-02-19T21:19:46.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.751+0000 7f61c5bb2700 1 -- 172.21.15.163:0/1549781119 >> 172.21.15.163:0/1549781119 conn(0x7f61c00f35a0 msgr2=0x7f61c0107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:46.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.752+0000 7f61c5bb2700 1 -- 172.21.15.163:0/1549781119 shutdown_connections 2024-02-19T21:19:46.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.752+0000 7f61c5bb2700 1 -- 172.21.15.163:0/1549781119 wait complete. 2024-02-19T21:19:46.755 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:46.752+0000 7f61c5bb2700 1 librados: shutdown 2024-02-19T21:19:47.105 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:45.377169+0000 mgr.y (mgr.14550) 736 : cluster [DBG] pgmap v1327: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:19:47.105 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:45.454653+0000 osd.4 (osd.4) 357 : cluster [DBG] 4.13 deep-scrub starts 2024-02-19T21:19:47.106 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:45.458783+0000 osd.4 (osd.4) 358 : cluster [DBG] 4.13 deep-scrub ok 2024-02-19T21:19:47.106 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:46.170688+0000 osd.0 (osd.0) 188 : cluster [DBG] 16.18 deep-scrub starts 2024-02-19T21:19:47.106 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:46.174083+0000 osd.0 (osd.0) 189 : cluster [DBG] 16.18 deep-scrub ok 2024-02-19T21:19:47.106 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:46.751355+0000 mon.a (mon.0) 3876 : audit [DBG] from='client.? v1:172.21.15.163:0/1549781119' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-19T21:19:47.106 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:45.377169+0000 mgr.y (mgr.14550) 736 : cluster [DBG] pgmap v1327: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:19:47.106 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:45.454653+0000 osd.4 (osd.4) 357 : cluster [DBG] 4.13 deep-scrub starts 2024-02-19T21:19:47.106 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:45.458783+0000 osd.4 (osd.4) 358 : cluster [DBG] 4.13 deep-scrub ok 2024-02-19T21:19:47.106 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:46.170688+0000 osd.0 (osd.0) 188 : cluster [DBG] 16.18 deep-scrub starts 2024-02-19T21:19:47.106 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:46.174083+0000 osd.0 (osd.0) 189 : cluster [DBG] 16.18 deep-scrub ok 2024-02-19T21:19:47.106 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:46 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:46.751355+0000 mon.a (mon.0) 3876 : audit [DBG] from='client.? v1:172.21.15.163:0/1549781119' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-19T21:19:47.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:45.377169+0000 mgr.y (mgr.14550) 736 : cluster [DBG] pgmap v1327: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:19:47.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:45.454653+0000 osd.4 (osd.4) 357 : cluster [DBG] 4.13 deep-scrub starts 2024-02-19T21:19:47.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:45.458783+0000 osd.4 (osd.4) 358 : cluster [DBG] 4.13 deep-scrub ok 2024-02-19T21:19:47.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:46.170688+0000 osd.0 (osd.0) 188 : cluster [DBG] 16.18 deep-scrub starts 2024-02-19T21:19:47.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:46.174083+0000 osd.0 (osd.0) 189 : cluster [DBG] 16.18 deep-scrub ok 2024-02-19T21:19:47.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:46.751355+0000 mon.a (mon.0) 3876 : audit [DBG] from='client.? v1:172.21.15.163:0/1549781119' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-19T21:19:47.515 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411572 got 77309411577 for osd.2 2024-02-19T21:19:47.515 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-19T21:19:47.787 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:48.073 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:19:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:19:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:19:48.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:46.285911+0000 osd.2 (osd.2) 384 : cluster [DBG] 6.f scrub starts 2024-02-19T21:19:48.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:46.286991+0000 osd.2 (osd.2) 385 : cluster [DBG] 6.f scrub ok 2024-02-19T21:19:48.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:46.482743+0000 osd.4 (osd.4) 359 : cluster [DBG] 5.1 deep-scrub starts 2024-02-19T21:19:48.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:46.483744+0000 osd.4 (osd.4) 360 : cluster [DBG] 5.1 deep-scrub ok 2024-02-19T21:19:48.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:46.547959+0000 osd.3 (osd.3) 423 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:19:48.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:46.548848+0000 osd.3 (osd.3) 424 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:19:48.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:47.456780+0000 osd.7 (osd.7) 365 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:19:48.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:47.459709+0000 osd.7 (osd.7) 366 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:19:48.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:47.527518+0000 osd.6 (osd.6) 341 : cluster [DBG] 16.6 scrub starts 2024-02-19T21:19:48.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:47.528339+0000 osd.6 (osd.6) 342 : cluster [DBG] 16.6 scrub ok 2024-02-19T21:19:48.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:46.285911+0000 osd.2 (osd.2) 384 : cluster [DBG] 6.f scrub starts 2024-02-19T21:19:48.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:46.286991+0000 osd.2 (osd.2) 385 : cluster [DBG] 6.f scrub ok 2024-02-19T21:19:48.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:46.482743+0000 osd.4 (osd.4) 359 : cluster [DBG] 5.1 deep-scrub starts 2024-02-19T21:19:48.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:46.483744+0000 osd.4 (osd.4) 360 : cluster [DBG] 5.1 deep-scrub ok 2024-02-19T21:19:48.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:46.547959+0000 osd.3 (osd.3) 423 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:19:48.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:46.548848+0000 osd.3 (osd.3) 424 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:19:48.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:47.456780+0000 osd.7 (osd.7) 365 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:19:48.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:47.459709+0000 osd.7 (osd.7) 366 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:19:48.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:47.527518+0000 osd.6 (osd.6) 341 : cluster [DBG] 16.6 scrub starts 2024-02-19T21:19:48.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:47.528339+0000 osd.6 (osd.6) 342 : cluster [DBG] 16.6 scrub ok 2024-02-19T21:19:48.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:46.285911+0000 osd.2 (osd.2) 384 : cluster [DBG] 6.f scrub starts 2024-02-19T21:19:48.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:46.286991+0000 osd.2 (osd.2) 385 : cluster [DBG] 6.f scrub ok 2024-02-19T21:19:48.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:46.482743+0000 osd.4 (osd.4) 359 : cluster [DBG] 5.1 deep-scrub starts 2024-02-19T21:19:48.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:46.483744+0000 osd.4 (osd.4) 360 : cluster [DBG] 5.1 deep-scrub ok 2024-02-19T21:19:48.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:46.547959+0000 osd.3 (osd.3) 423 : cluster [DBG] 6.6 scrub starts 2024-02-19T21:19:48.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:46.548848+0000 osd.3 (osd.3) 424 : cluster [DBG] 6.6 scrub ok 2024-02-19T21:19:48.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:47.456780+0000 osd.7 (osd.7) 365 : cluster [DBG] 6.15 scrub starts 2024-02-19T21:19:48.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:47.459709+0000 osd.7 (osd.7) 366 : cluster [DBG] 6.15 scrub ok 2024-02-19T21:19:48.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:47.527518+0000 osd.6 (osd.6) 341 : cluster [DBG] 16.6 scrub starts 2024-02-19T21:19:48.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:47 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:47.528339+0000 osd.6 (osd.6) 342 : cluster [DBG] 16.6 scrub ok 2024-02-19T21:19:48.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:47.302273+0000 osd.2 (osd.2) 386 : cluster [DBG] 16.e scrub starts 2024-02-19T21:19:48.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:47.303086+0000 osd.2 (osd.2) 387 : cluster [DBG] 16.e scrub ok 2024-02-19T21:19:48.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:47.377492+0000 mgr.y (mgr.14550) 737 : cluster [DBG] pgmap v1328: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:19:48.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:47.579026+0000 osd.3 (osd.3) 425 : cluster [DBG] 2.23 scrub starts 2024-02-19T21:19:49.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:47.579963+0000 osd.3 (osd.3) 426 : cluster [DBG] 2.23 scrub ok 2024-02-19T21:19:49.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:47.302273+0000 osd.2 (osd.2) 386 : cluster [DBG] 16.e scrub starts 2024-02-19T21:19:49.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:47.303086+0000 osd.2 (osd.2) 387 : cluster [DBG] 16.e scrub ok 2024-02-19T21:19:49.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:47.377492+0000 mgr.y (mgr.14550) 737 : cluster [DBG] pgmap v1328: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:19:49.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:47.579026+0000 osd.3 (osd.3) 425 : cluster [DBG] 2.23 scrub starts 2024-02-19T21:19:49.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:48 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:47.579963+0000 osd.3 (osd.3) 426 : cluster [DBG] 2.23 scrub ok 2024-02-19T21:19:49.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:47.302273+0000 osd.2 (osd.2) 386 : cluster [DBG] 16.e scrub starts 2024-02-19T21:19:49.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:47.303086+0000 osd.2 (osd.2) 387 : cluster [DBG] 16.e scrub ok 2024-02-19T21:19:49.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:47.377492+0000 mgr.y (mgr.14550) 737 : cluster [DBG] pgmap v1328: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:19:49.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:47.579026+0000 osd.3 (osd.3) 425 : cluster [DBG] 2.23 scrub starts 2024-02-19T21:19:49.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:47.579963+0000 osd.3 (osd.3) 426 : cluster [DBG] 2.23 scrub ok 2024-02-19T21:19:49.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.636+0000 7fd12d0af700 1 -- 172.21.15.163:0/778917069 >> v1:172.21.15.163:6789/0 conn(0x7fd1281042c0 legacy=0x7fd128106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:49.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.636+0000 7fd12d0af700 1 -- 172.21.15.163:0/778917069 shutdown_connections 2024-02-19T21:19:49.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.636+0000 7fd12d0af700 1 -- 172.21.15.163:0/778917069 >> 172.21.15.163:0/778917069 conn(0x7fd1280f35a0 msgr2=0x7fd1280f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:49.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.636+0000 7fd12d0af700 1 -- 172.21.15.163:0/778917069 shutdown_connections 2024-02-19T21:19:49.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.636+0000 7fd12d0af700 1 -- 172.21.15.163:0/778917069 wait complete. 2024-02-19T21:19:49.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.637+0000 7fd12d0af700 1 librados: starting msgr at 2024-02-19T21:19:49.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.637+0000 7fd12d0af700 1 librados: starting objecter 2024-02-19T21:19:49.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.637+0000 7fd12d0af700 1 Processor -- start 2024-02-19T21:19:49.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.637+0000 7fd12d0af700 1 -- start start 2024-02-19T21:19:49.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.637+0000 7fd12d0af700 1 librados: setting wanted keys 2024-02-19T21:19:49.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.637+0000 7fd12d0af700 1 librados: calling monclient init 2024-02-19T21:19:49.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.637+0000 7fd12d0af700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd128117330 con 0x7fd1281042c0 2024-02-19T21:19:49.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.637+0000 7fd12d0af700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd128118510 con 0x7fd1280fd980 2024-02-19T21:19:49.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.637+0000 7fd12d0af700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd128119710 con 0x7fd128100780 2024-02-19T21:19:49.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd12759e700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fd1281042c0 0x7fd1281159f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33774/0 (socket says 172.21.15.163:33774) 2024-02-19T21:19:49.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd12759e700 1 -- 172.21.15.163:0/1279296110 learned_addr learned my addr 172.21.15.163:0/1279296110 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:49.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 10 client.?.objecter ms_handle_connect 0x7fd128100780 2024-02-19T21:19:49.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:49.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 10 client.?.objecter ms_handle_connect 0x7fd1281042c0 2024-02-19T21:19:49.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:49.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 10 client.?.objecter ms_handle_connect 0x7fd1280fd980 2024-02-19T21:19:49.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:49.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182818500 0 0) 0x7fd128119710 con 0x7fd128100780 2024-02-19T21:19:49.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd110003030 con 0x7fd128100780 2024-02-19T21:19:49.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915634559 0 0) 0x7fd128117330 con 0x7fd1281042c0 2024-02-19T21:19:49.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd128119710 con 0x7fd1281042c0 2024-02-19T21:19:49.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567598169 0 0) 0x7fd128118510 con 0x7fd1280fd980 2024-02-19T21:19:49.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd128117330 con 0x7fd1280fd980 2024-02-19T21:19:49.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2847933654 0 0) 0x7fd128119710 con 0x7fd1281042c0 2024-02-19T21:19:49.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd128118510 con 0x7fd1281042c0 2024-02-19T21:19:49.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd1200022d0 con 0x7fd1281042c0 2024-02-19T21:19:49.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1524390946 0 0) 0x7fd110003030 con 0x7fd128100780 2024-02-19T21:19:49.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd128119710 con 0x7fd128100780 2024-02-19T21:19:49.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd11c001920 con 0x7fd128100780 2024-02-19T21:19:49.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2749276892 0 0) 0x7fd128117330 con 0x7fd1280fd980 2024-02-19T21:19:49.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd110003030 con 0x7fd1280fd980 2024-02-19T21:19:49.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd1180017c0 con 0x7fd1280fd980 2024-02-19T21:19:49.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2065915122 0 0) 0x7fd128118510 con 0x7fd1281042c0 2024-02-19T21:19:49.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 >> v1:172.21.15.163:6790/0 conn(0x7fd128100780 legacy=0x7fd1280fa030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:49.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 >> v1:172.21.15.193:6789/0 conn(0x7fd1280fd980 legacy=0x7fd1280f9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:49.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12811a910 con 0x7fd1281042c0 2024-02-19T21:19:49.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd12d0af700 1 -- 172.21.15.163:0/1279296110 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd128117520 con 0x7fd1281042c0 2024-02-19T21:19:49.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd12d0af700 10 client.24156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:49.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd12d0af700 1 -- 172.21.15.163:0/1279296110 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd128117a10 con 0x7fd1281042c0 2024-02-19T21:19:49.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.638+0000 7fd12d0af700 1 librados: init done 2024-02-19T21:19:49.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.639+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fd120002610 con 0x7fd1281042c0 2024-02-19T21:19:49.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.639+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fd120003a70 con 0x7fd1281042c0 2024-02-19T21:19:49.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.639+0000 7fd12d0af700 1 -- 172.21.15.163:0/1279296110 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd114004f20 con 0x7fd1281042c0 2024-02-19T21:19:49.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.640+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fd120001bc0 con 0x7fd1281042c0 2024-02-19T21:19:49.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.640+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(853..853 src has 796..853) v4 ==== 7419+0+0 (unknown 1683610851 0 0) 0x7fd1200790e0 con 0x7fd1281042c0 2024-02-19T21:19:49.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.640+0000 7fd10ffff700 10 client.24156.objecter ms_dispatch 0x7fd128001550 osd_map(853..853 src has 796..853) v4 2024-02-19T21:19:49.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.640+0000 7fd10ffff700 3 client.24156.objecter handle_osd_map got epochs [853,853] > 0 2024-02-19T21:19:49.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.640+0000 7fd10ffff700 3 client.24156.objecter handle_osd_map decoding full epoch 853 2024-02-19T21:19:49.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.640+0000 7fd10ffff700 20 client.24156.objecter dump_active .. 0 homeless 2024-02-19T21:19:49.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.641+0000 7fd10ffff700 10 client.24156.objecter ms_handle_connect 0x7fd110063ca0 2024-02-19T21:19:49.646 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.644+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fd12004c870 con 0x7fd1281042c0 2024-02-19T21:19:49.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.808+0000 7fd12d0af700 1 -- 172.21.15.163:0/1279296110 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd1140051d0 con 0x7fd1281042c0 2024-02-19T21:19:49.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.808+0000 7fd10ffff700 1 -- 172.21.15.163:0/1279296110 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (unknown 383520633 0 2728376441) 0x7fd12004ace0 con 0x7fd1281042c0 2024-02-19T21:19:49.811 INFO:teuthology.orchestra.run.smithi163.stdout:107374182645 2024-02-19T21:19:49.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.810+0000 7fd12d0af700 10 librados: watch_flush enter 2024-02-19T21:19:49.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.810+0000 7fd12d0af700 10 librados: watch_flush exit 2024-02-19T21:19:49.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.810+0000 7fd12d0af700 20 client.24156.objecter shutdown clearing up homeless session... 2024-02-19T21:19:49.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.810+0000 7fd12d0af700 10 client.24156.objecter successfully canceled tick 2024-02-19T21:19:49.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.810+0000 7fd12d0af700 1 -- 172.21.15.163:0/1279296110 >> v1:172.21.15.163:6800/1552514681 conn(0x7fd110063ca0 legacy=0x7fd110066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:49.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.810+0000 7fd12d0af700 1 -- 172.21.15.163:0/1279296110 >> v1:172.21.15.163:6789/0 conn(0x7fd1281042c0 legacy=0x7fd1281159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:49.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.810+0000 7fd12d0af700 1 -- 172.21.15.163:0/1279296110 shutdown_connections 2024-02-19T21:19:49.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.810+0000 7fd12d0af700 1 -- 172.21.15.163:0/1279296110 >> 172.21.15.163:0/1279296110 conn(0x7fd1280f35a0 msgr2=0x7fd128107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:49.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.810+0000 7fd12d0af700 1 -- 172.21.15.163:0/1279296110 shutdown_connections 2024-02-19T21:19:49.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.810+0000 7fd12d0af700 1 -- 172.21.15.163:0/1279296110 wait complete. 2024-02-19T21:19:49.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:49.810+0000 7fd12d0af700 1 librados: shutdown 2024-02-19T21:19:50.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:48.616045+0000 osd.3 (osd.3) 427 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:19:50.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:48.616958+0000 osd.3 (osd.3) 428 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:19:50.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:48.703785+0000 osd.1 (osd.1) 427 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:19:50.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:48.706757+0000 osd.1 (osd.1) 428 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:19:50.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:49.542769+0000 osd.6 (osd.6) 343 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:19:50.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:49.545676+0000 osd.6 (osd.6) 344 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:19:50.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:49.654987+0000 osd.5 (osd.5) 388 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:19:50.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:49.657886+0000 osd.5 (osd.5) 389 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:19:50.145 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:49.809897+0000 mon.a (mon.0) 3877 : audit [DBG] from='client.? v1:172.21.15.163:0/1279296110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-19T21:19:50.146 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:48.616045+0000 osd.3 (osd.3) 427 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:19:50.146 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:48.616958+0000 osd.3 (osd.3) 428 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:19:50.146 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:48.703785+0000 osd.1 (osd.1) 427 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:19:50.146 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:48.706757+0000 osd.1 (osd.1) 428 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:19:50.146 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:49.542769+0000 osd.6 (osd.6) 343 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:19:50.146 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:49.545676+0000 osd.6 (osd.6) 344 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:19:50.146 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:49.654987+0000 osd.5 (osd.5) 388 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:19:50.146 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:49.657886+0000 osd.5 (osd.5) 389 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:19:50.146 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:49 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:49.809897+0000 mon.a (mon.0) 3877 : audit [DBG] from='client.? v1:172.21.15.163:0/1279296110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-19T21:19:50.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:48.616045+0000 osd.3 (osd.3) 427 : cluster [DBG] 6.11 scrub starts 2024-02-19T21:19:50.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:48.616958+0000 osd.3 (osd.3) 428 : cluster [DBG] 6.11 scrub ok 2024-02-19T21:19:50.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:48.703785+0000 osd.1 (osd.1) 427 : cluster [DBG] 5.12 scrub starts 2024-02-19T21:19:50.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:48.706757+0000 osd.1 (osd.1) 428 : cluster [DBG] 5.12 scrub ok 2024-02-19T21:19:50.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:49.542769+0000 osd.6 (osd.6) 343 : cluster [DBG] 5.11 scrub starts 2024-02-19T21:19:50.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:49.545676+0000 osd.6 (osd.6) 344 : cluster [DBG] 5.11 scrub ok 2024-02-19T21:19:50.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:49.654987+0000 osd.5 (osd.5) 388 : cluster [DBG] 2.18 scrub starts 2024-02-19T21:19:50.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:49.657886+0000 osd.5 (osd.5) 389 : cluster [DBG] 2.18 scrub ok 2024-02-19T21:19:50.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:49.809897+0000 mon.a (mon.0) 3877 : audit [DBG] from='client.? v1:172.21.15.163:0/1279296110' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-19T21:19:50.462 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182641 got 107374182645 for osd.3 2024-02-19T21:19:50.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-19T21:19:50.734 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:51.207 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:49.378044+0000 mgr.y (mgr.14550) 738 : cluster [DBG] pgmap v1329: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:19:51.208 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:49.499904+0000 osd.4 (osd.4) 361 : cluster [DBG] 4.e scrub starts 2024-02-19T21:19:51.208 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:49.501368+0000 osd.4 (osd.4) 362 : cluster [DBG] 4.e scrub ok 2024-02-19T21:19:51.208 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:50.572746+0000 osd.6 (osd.6) 345 : cluster [DBG] 4.a scrub starts 2024-02-19T21:19:51.208 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:50.574915+0000 osd.6 (osd.6) 346 : cluster [DBG] 4.a scrub ok 2024-02-19T21:19:51.208 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:49.378044+0000 mgr.y (mgr.14550) 738 : cluster [DBG] pgmap v1329: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:19:51.208 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:49.499904+0000 osd.4 (osd.4) 361 : cluster [DBG] 4.e scrub starts 2024-02-19T21:19:51.208 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:49.501368+0000 osd.4 (osd.4) 362 : cluster [DBG] 4.e scrub ok 2024-02-19T21:19:51.208 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:50.572746+0000 osd.6 (osd.6) 345 : cluster [DBG] 4.a scrub starts 2024-02-19T21:19:51.208 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:50.574915+0000 osd.6 (osd.6) 346 : cluster [DBG] 4.a scrub ok 2024-02-19T21:19:51.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:19:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:19:51.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:49.378044+0000 mgr.y (mgr.14550) 738 : cluster [DBG] pgmap v1329: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:19:51.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:49.499904+0000 osd.4 (osd.4) 361 : cluster [DBG] 4.e scrub starts 2024-02-19T21:19:51.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:49.501368+0000 osd.4 (osd.4) 362 : cluster [DBG] 4.e scrub ok 2024-02-19T21:19:51.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:50.572746+0000 osd.6 (osd.6) 345 : cluster [DBG] 4.a scrub starts 2024-02-19T21:19:51.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:50.574915+0000 osd.6 (osd.6) 346 : cluster [DBG] 4.a scrub ok 2024-02-19T21:19:52.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:50.615025+0000 osd.3 (osd.3) 429 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:19:52.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:50.616617+0000 osd.3 (osd.3) 430 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:19:52.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:50.931493+0000 mgr.y (mgr.14550) 739 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:52.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:50.615025+0000 osd.3 (osd.3) 429 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:19:52.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:50.616617+0000 osd.3 (osd.3) 430 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:19:52.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:51 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:50.931493+0000 mgr.y (mgr.14550) 739 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:52.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:50.615025+0000 osd.3 (osd.3) 429 : cluster [DBG] 5.14 scrub starts 2024-02-19T21:19:52.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:50.616617+0000 osd.3 (osd.3) 430 : cluster [DBG] 5.14 scrub ok 2024-02-19T21:19:52.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:50.931493+0000 mgr.y (mgr.14550) 739 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:19:52.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.533+0000 7f4fb8d5d700 1 -- 172.21.15.163:0/393485743 >> v1:172.21.15.163:6790/0 conn(0x7f4fb41042c0 legacy=0x7f4fb4106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:52.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.533+0000 7f4fb8d5d700 1 -- 172.21.15.163:0/393485743 shutdown_connections 2024-02-19T21:19:52.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.533+0000 7f4fb8d5d700 1 -- 172.21.15.163:0/393485743 >> 172.21.15.163:0/393485743 conn(0x7f4fb40f35a0 msgr2=0x7f4fb40f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:52.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.533+0000 7f4fb8d5d700 1 -- 172.21.15.163:0/393485743 shutdown_connections 2024-02-19T21:19:52.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.533+0000 7f4fb8d5d700 1 -- 172.21.15.163:0/393485743 wait complete. 2024-02-19T21:19:52.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fb8d5d700 1 librados: starting msgr at 2024-02-19T21:19:52.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fb8d5d700 1 librados: starting objecter 2024-02-19T21:19:52.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fb8d5d700 1 Processor -- start 2024-02-19T21:19:52.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fb8d5d700 1 -- start start 2024-02-19T21:19:52.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fb8d5d700 1 librados: setting wanted keys 2024-02-19T21:19:52.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fb8d5d700 1 librados: calling monclient init 2024-02-19T21:19:52.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fb8d5d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4fb4117460 con 0x7f4fb41042c0 2024-02-19T21:19:52.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fb8d5d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4fb4118660 con 0x7f4fb4100780 2024-02-19T21:19:52.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fb8d5d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4fb4119860 con 0x7f4fb40fd980 2024-02-19T21:19:52.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fb259c700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f4fb40fd980 0x7f4fb40f9bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:48516/0 (socket says 172.21.15.163:48516) 2024-02-19T21:19:52.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fb259c700 1 -- 172.21.15.163:0/3588912672 learned_addr learned my addr 172.21.15.163:0/3588912672 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:52.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fb2d9d700 1 --1- 172.21.15.163:0/3588912672 >> v1:172.21.15.163:6789/0 conn(0x7f4fb41042c0 0x7f4fb4115b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:33786/0 (socket says 172.21.15.163:33786) 2024-02-19T21:19:52.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fa37fe700 10 client.?.objecter ms_handle_connect 0x7f4fb40fd980 2024-02-19T21:19:52.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fa37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:52.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fa37fe700 10 client.?.objecter ms_handle_connect 0x7f4fb41042c0 2024-02-19T21:19:52.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fa37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:52.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fa37fe700 10 client.?.objecter ms_handle_connect 0x7f4fb4100780 2024-02-19T21:19:52.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.534+0000 7f4fa37fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:52.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684741884 0 0) 0x7f4fb4117460 con 0x7f4fb41042c0 2024-02-19T21:19:52.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f9c0033f0 con 0x7f4fb41042c0 2024-02-19T21:19:52.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938594935 0 0) 0x7f4fb4119860 con 0x7f4fb40fd980 2024-02-19T21:19:52.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fb4117460 con 0x7f4fb40fd980 2024-02-19T21:19:52.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601618636 0 0) 0x7f4fb4118660 con 0x7f4fb4100780 2024-02-19T21:19:52.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fb4119860 con 0x7f4fb4100780 2024-02-19T21:19:52.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3639263506 0 0) 0x7f4f9c0033f0 con 0x7f4fb41042c0 2024-02-19T21:19:52.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fb4118660 con 0x7f4fb41042c0 2024-02-19T21:19:52.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3301577300 0 0) 0x7f4fb4119860 con 0x7f4fb4100780 2024-02-19T21:19:52.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f9c0033f0 con 0x7f4fb4100780 2024-02-19T21:19:52.540 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4fac0023e0 con 0x7f4fb41042c0 2024-02-19T21:19:52.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2358275413 0 0) 0x7f4fb4117460 con 0x7f4fb40fd980 2024-02-19T21:19:52.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fb4119860 con 0x7f4fb40fd980 2024-02-19T21:19:52.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4fa4001800 con 0x7f4fb4100780 2024-02-19T21:19:52.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4fa80028d0 con 0x7f4fb40fd980 2024-02-19T21:19:52.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2879325514 0 0) 0x7f4f9c0033f0 con 0x7f4fb4100780 2024-02-19T21:19:52.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 >> v1:172.21.15.163:6790/0 conn(0x7f4fb40fd980 legacy=0x7f4fb40f9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:52.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 >> v1:172.21.15.163:6789/0 conn(0x7f4fb41042c0 legacy=0x7f4fb4115b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:52.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb411aa60 con 0x7f4fb4100780 2024-02-19T21:19:52.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fb8d5d700 1 -- 172.21.15.163:0/3588912672 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb4117650 con 0x7f4fb4100780 2024-02-19T21:19:52.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fb8d5d700 10 client.49699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:52.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fb8d5d700 1 -- 172.21.15.163:0/3588912672 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb4117b40 con 0x7f4fb4100780 2024-02-19T21:19:52.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fb8d5d700 1 librados: init done 2024-02-19T21:19:52.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f4fa4001ec0 con 0x7f4fb4100780 2024-02-19T21:19:52.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.535+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4fa40030f0 con 0x7f4fb4100780 2024-02-19T21:19:52.541 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.536+0000 7f4fb8d5d700 1 -- 172.21.15.163:0/3588912672 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb405f660 con 0x7f4fb4100780 2024-02-19T21:19:52.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.536+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f4fa4001a20 con 0x7f4fb4100780 2024-02-19T21:19:52.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.537+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(853..853 src has 796..853) v4 ==== 7419+0+0 (unknown 1683610851 0 0) 0x7f4fa40761f0 con 0x7f4fb4100780 2024-02-19T21:19:52.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.537+0000 7f4fa37fe700 10 client.49699.objecter ms_dispatch 0x7f4fb4001550 osd_map(853..853 src has 796..853) v4 2024-02-19T21:19:52.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.537+0000 7f4fa37fe700 3 client.49699.objecter handle_osd_map got epochs [853,853] > 0 2024-02-19T21:19:52.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.537+0000 7f4fa37fe700 3 client.49699.objecter handle_osd_map decoding full epoch 853 2024-02-19T21:19:52.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.537+0000 7f4fa37fe700 20 client.49699.objecter dump_active .. 0 homeless 2024-02-19T21:19:52.542 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.537+0000 7f4fa37fe700 10 client.49699.objecter ms_handle_connect 0x7f4f9c063d50 2024-02-19T21:19:52.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.541+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f4fa4049860 con 0x7f4fb4100780 2024-02-19T21:19:52.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.711+0000 7f4fb8d5d700 1 -- 172.21.15.163:0/3588912672 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4fb4109f20 con 0x7f4fb4100780 2024-02-19T21:19:52.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.711+0000 7f4fa37fe700 1 -- 172.21.15.163:0/3588912672 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (unknown 1823589948 0 1023939148) 0x7f4fa4002030 con 0x7f4fb4100780 2024-02-19T21:19:52.713 INFO:teuthology.orchestra.run.smithi163.stdout:133143986418 2024-02-19T21:19:52.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.713+0000 7f4fb8d5d700 10 librados: watch_flush enter 2024-02-19T21:19:52.715 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.713+0000 7f4fb8d5d700 10 librados: watch_flush exit 2024-02-19T21:19:52.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.713+0000 7f4fb8d5d700 20 client.49699.objecter shutdown clearing up homeless session... 2024-02-19T21:19:52.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.713+0000 7f4fb8d5d700 10 client.49699.objecter successfully canceled tick 2024-02-19T21:19:52.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.713+0000 7f4fb8d5d700 1 -- 172.21.15.163:0/3588912672 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4f9c063d50 legacy=0x7f4f9c066210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:52.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.713+0000 7f4fb8d5d700 1 -- 172.21.15.163:0/3588912672 >> v1:172.21.15.193:6789/0 conn(0x7f4fb4100780 legacy=0x7f4fb40fa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:52.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.713+0000 7f4fb8d5d700 1 -- 172.21.15.163:0/3588912672 shutdown_connections 2024-02-19T21:19:52.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.713+0000 7f4fb8d5d700 1 -- 172.21.15.163:0/3588912672 >> 172.21.15.163:0/3588912672 conn(0x7f4fb40f35a0 msgr2=0x7f4fb4107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:52.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.713+0000 7f4fb8d5d700 1 -- 172.21.15.163:0/3588912672 shutdown_connections 2024-02-19T21:19:52.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.713+0000 7f4fb8d5d700 1 -- 172.21.15.163:0/3588912672 wait complete. 2024-02-19T21:19:52.716 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:52.713+0000 7f4fb8d5d700 1 librados: shutdown 2024-02-19T21:19:52.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:51.378907+0000 mgr.y (mgr.14550) 740 : cluster [DBG] pgmap v1330: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-02-19T21:19:52.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:51.952643+0000 mon.b (mon.1) 773 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:19:52.999 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:52.186989+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:19:53.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:52.189884+0000 osd.0 (osd.0) 191 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:19:53.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:52.373478+0000 mon.b (mon.1) 774 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:19:53.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:52.374347+0000 mon.b (mon.1) 775 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:19:53.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:52.378621+0000 mon.a (mon.0) 3878 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:19:53.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:52.568120+0000 osd.6 (osd.6) 347 : cluster [DBG] 2.22 scrub starts 2024-02-19T21:19:53.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:52.568912+0000 osd.6 (osd.6) 348 : cluster [DBG] 2.22 scrub ok 2024-02-19T21:19:53.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:52.595476+0000 osd.5 (osd.5) 390 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:19:53.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:52.598552+0000 osd.5 (osd.5) 391 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:19:53.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:52.712645+0000 mon.b (mon.1) 776 : audit [DBG] from='client.? v1:172.21.15.163:0/3588912672' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-19T21:19:53.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:51.378907+0000 mgr.y (mgr.14550) 740 : cluster [DBG] pgmap v1330: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-02-19T21:19:53.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:51.952643+0000 mon.b (mon.1) 773 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:19:53.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:52.186989+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:19:53.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:52.189884+0000 osd.0 (osd.0) 191 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:19:53.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:52.373478+0000 mon.b (mon.1) 774 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:19:53.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:52.374347+0000 mon.b (mon.1) 775 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:19:53.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:52.378621+0000 mon.a (mon.0) 3878 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:19:53.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:52.568120+0000 osd.6 (osd.6) 347 : cluster [DBG] 2.22 scrub starts 2024-02-19T21:19:53.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:52.568912+0000 osd.6 (osd.6) 348 : cluster [DBG] 2.22 scrub ok 2024-02-19T21:19:53.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:52.595476+0000 osd.5 (osd.5) 390 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:19:53.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:52.598552+0000 osd.5 (osd.5) 391 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:19:53.001 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:52 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:52.712645+0000 mon.b (mon.1) 776 : audit [DBG] from='client.? v1:172.21.15.163:0/3588912672' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-19T21:19:53.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:51.378907+0000 mgr.y (mgr.14550) 740 : cluster [DBG] pgmap v1330: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-02-19T21:19:53.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:51.952643+0000 mon.b (mon.1) 773 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:19:53.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:52.186989+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.10 scrub starts 2024-02-19T21:19:53.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:52.189884+0000 osd.0 (osd.0) 191 : cluster [DBG] 6.10 scrub ok 2024-02-19T21:19:53.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:52.373478+0000 mon.b (mon.1) 774 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:19:53.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:52.374347+0000 mon.b (mon.1) 775 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:19:53.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:52.378621+0000 mon.a (mon.0) 3878 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:19:53.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:52.568120+0000 osd.6 (osd.6) 347 : cluster [DBG] 2.22 scrub starts 2024-02-19T21:19:53.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:52.568912+0000 osd.6 (osd.6) 348 : cluster [DBG] 2.22 scrub ok 2024-02-19T21:19:53.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:52.595476+0000 osd.5 (osd.5) 390 : cluster [DBG] 6.7 scrub starts 2024-02-19T21:19:53.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:52.598552+0000 osd.5 (osd.5) 391 : cluster [DBG] 6.7 scrub ok 2024-02-19T21:19:53.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:52.712645+0000 mon.b (mon.1) 776 : audit [DBG] from='client.? v1:172.21.15.163:0/3588912672' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-19T21:19:53.418 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986413 got 133143986418 for osd.4 2024-02-19T21:19:53.419 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-19T21:19:53.692 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:53.749 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:19:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:19:53] "GET /metrics HTTP/1.1" 200 187977 "" "Prometheus/2.33.4" 2024-02-19T21:19:54.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:52.315266+0000 osd.2 (osd.2) 388 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:19:54.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:52.316998+0000 osd.2 (osd.2) 389 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:19:54.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:52.683666+0000 osd.1 (osd.1) 429 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:19:54.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:52.687373+0000 osd.1 (osd.1) 430 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:19:54.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:53.637021+0000 osd.1 (osd.1) 431 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:19:54.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:53.638060+0000 osd.1 (osd.1) 432 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:19:54.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:52.315266+0000 osd.2 (osd.2) 388 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:19:54.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:52.316998+0000 osd.2 (osd.2) 389 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:19:54.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:52.683666+0000 osd.1 (osd.1) 429 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:19:54.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:52.687373+0000 osd.1 (osd.1) 430 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:19:54.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:53.637021+0000 osd.1 (osd.1) 431 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:19:54.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:53 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:53.638060+0000 osd.1 (osd.1) 432 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:19:54.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:52.315266+0000 osd.2 (osd.2) 388 : cluster [DBG] 4.1c scrub starts 2024-02-19T21:19:54.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:52.316998+0000 osd.2 (osd.2) 389 : cluster [DBG] 4.1c scrub ok 2024-02-19T21:19:54.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:52.683666+0000 osd.1 (osd.1) 429 : cluster [DBG] 4.7 scrub starts 2024-02-19T21:19:54.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:52.687373+0000 osd.1 (osd.1) 430 : cluster [DBG] 4.7 scrub ok 2024-02-19T21:19:54.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:53.637021+0000 osd.1 (osd.1) 431 : cluster [DBG] 2.4 scrub starts 2024-02-19T21:19:54.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:53.638060+0000 osd.1 (osd.1) 432 : cluster [DBG] 2.4 scrub ok 2024-02-19T21:19:55.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:53.379456+0000 mgr.y (mgr.14550) 741 : cluster [DBG] pgmap v1331: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:55.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:53.495391+0000 osd.3 (osd.3) 431 : cluster [DBG] 6.c scrub starts 2024-02-19T21:19:55.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:53.496630+0000 osd.3 (osd.3) 432 : cluster [DBG] 6.c scrub ok 2024-02-19T21:19:55.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:54.542204+0000 osd.6 (osd.6) 349 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:19:55.232 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:54.544015+0000 osd.6 (osd.6) 350 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:19:55.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:53.379456+0000 mgr.y (mgr.14550) 741 : cluster [DBG] pgmap v1331: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:55.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:53.495391+0000 osd.3 (osd.3) 431 : cluster [DBG] 6.c scrub starts 2024-02-19T21:19:55.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:53.496630+0000 osd.3 (osd.3) 432 : cluster [DBG] 6.c scrub ok 2024-02-19T21:19:55.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:54.542204+0000 osd.6 (osd.6) 349 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:19:55.233 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:54 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:54.544015+0000 osd.6 (osd.6) 350 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:19:55.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:53.379456+0000 mgr.y (mgr.14550) 741 : cluster [DBG] pgmap v1331: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:55.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:53.495391+0000 osd.3 (osd.3) 431 : cluster [DBG] 6.c scrub starts 2024-02-19T21:19:55.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:53.496630+0000 osd.3 (osd.3) 432 : cluster [DBG] 6.c scrub ok 2024-02-19T21:19:55.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:54.542204+0000 osd.6 (osd.6) 349 : cluster [DBG] 4.1f scrub starts 2024-02-19T21:19:55.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:54.544015+0000 osd.6 (osd.6) 350 : cluster [DBG] 4.1f scrub ok 2024-02-19T21:19:55.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.719+0000 7f8d94068700 1 -- 172.21.15.163:0/3711121102 >> v1:172.21.15.163:6790/0 conn(0x7f8d8c1042c0 legacy=0x7f8d8c106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:55.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.719+0000 7f8d94068700 1 -- 172.21.15.163:0/3711121102 shutdown_connections 2024-02-19T21:19:55.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.719+0000 7f8d94068700 1 -- 172.21.15.163:0/3711121102 >> 172.21.15.163:0/3711121102 conn(0x7f8d8c0f35a0 msgr2=0x7f8d8c0f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:55.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.719+0000 7f8d94068700 1 -- 172.21.15.163:0/3711121102 shutdown_connections 2024-02-19T21:19:55.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.719+0000 7f8d94068700 1 -- 172.21.15.163:0/3711121102 wait complete. 2024-02-19T21:19:55.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d94068700 1 librados: starting msgr at 2024-02-19T21:19:55.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d94068700 1 librados: starting objecter 2024-02-19T21:19:55.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d94068700 1 Processor -- start 2024-02-19T21:19:55.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d94068700 1 -- start start 2024-02-19T21:19:55.722 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d94068700 1 librados: setting wanted keys 2024-02-19T21:19:55.723 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d94068700 1 librados: calling monclient init 2024-02-19T21:19:55.725 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d94068700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8d8c1174d0 con 0x7f8d8c0fd980 2024-02-19T21:19:55.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d94068700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8d8c1186d0 con 0x7f8d8c100780 2024-02-19T21:19:55.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d94068700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8d8c1198d0 con 0x7f8d8c1042c0 2024-02-19T21:19:55.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d91e04700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8d8c0fd980 0x7f8d8c0f9ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:51536/0 (socket says 172.21.15.163:51536) 2024-02-19T21:19:55.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d91e04700 1 -- 172.21.15.163:0/3634099635 learned_addr learned my addr 172.21.15.163:0/3634099635 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:55.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d92605700 1 --1- 172.21.15.163:0/3634099635 >> v1:172.21.15.163:6790/0 conn(0x7f8d8c1042c0 0x7f8d8c115bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45300/0 (socket says 172.21.15.163:45300) 2024-02-19T21:19:55.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d82ffd700 10 client.?.objecter ms_handle_connect 0x7f8d8c1042c0 2024-02-19T21:19:55.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d82ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:55.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d82ffd700 10 client.?.objecter ms_handle_connect 0x7f8d8c0fd980 2024-02-19T21:19:55.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.720+0000 7f8d82ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:55.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 10 client.?.objecter ms_handle_connect 0x7f8d8c100780 2024-02-19T21:19:55.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:55.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840014367 0 0) 0x7f8d8c1198d0 con 0x7f8d8c1042c0 2024-02-19T21:19:55.726 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d78003630 con 0x7f8d8c1042c0 2024-02-19T21:19:55.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86046091 0 0) 0x7f8d8c1174d0 con 0x7f8d8c0fd980 2024-02-19T21:19:55.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d8c1198d0 con 0x7f8d8c0fd980 2024-02-19T21:19:55.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632929337 0 0) 0x7f8d8c1186d0 con 0x7f8d8c100780 2024-02-19T21:19:55.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d8c1174d0 con 0x7f8d8c100780 2024-02-19T21:19:55.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3586927768 0 0) 0x7f8d78003630 con 0x7f8d8c1042c0 2024-02-19T21:19:55.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d8c1186d0 con 0x7f8d8c1042c0 2024-02-19T21:19:55.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8d84003030 con 0x7f8d8c1042c0 2024-02-19T21:19:55.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2581134047 0 0) 0x7f8d8c1198d0 con 0x7f8d8c0fd980 2024-02-19T21:19:55.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d78003630 con 0x7f8d8c0fd980 2024-02-19T21:19:55.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8d7c001880 con 0x7f8d8c0fd980 2024-02-19T21:19:55.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4004039786 0 0) 0x7f8d8c1174d0 con 0x7f8d8c100780 2024-02-19T21:19:55.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d8c1198d0 con 0x7f8d8c100780 2024-02-19T21:19:55.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8d88001840 con 0x7f8d8c100780 2024-02-19T21:19:55.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1770619974 0 0) 0x7f8d8c1186d0 con 0x7f8d8c1042c0 2024-02-19T21:19:55.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 >> v1:172.21.15.193:6789/0 conn(0x7f8d8c100780 legacy=0x7f8d8c0fa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:55.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 >> v1:172.21.15.163:6789/0 conn(0x7f8d8c0fd980 legacy=0x7f8d8c0f9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:55.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c11aad0 con 0x7f8d8c1042c0 2024-02-19T21:19:55.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d94068700 1 -- 172.21.15.163:0/3634099635 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d8c1176c0 con 0x7f8d8c1042c0 2024-02-19T21:19:55.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d94068700 10 client.50894.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:55.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d94068700 1 -- 172.21.15.163:0/3634099635 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d8c117b30 con 0x7f8d8c1042c0 2024-02-19T21:19:55.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d94068700 1 librados: init done 2024-02-19T21:19:55.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f8d84001ee0 con 0x7f8d8c1042c0 2024-02-19T21:19:55.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.721+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f8d84003b00 con 0x7f8d8c1042c0 2024-02-19T21:19:55.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.722+0000 7f8d94068700 1 -- 172.21.15.163:0/3634099635 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d70004f20 con 0x7f8d8c1042c0 2024-02-19T21:19:55.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.723+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f8d84003c60 con 0x7f8d8c1042c0 2024-02-19T21:19:55.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.723+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(853..853 src has 796..853) v4 ==== 7419+0+0 (unknown 1683610851 0 0) 0x7f8d84078eb0 con 0x7f8d8c1042c0 2024-02-19T21:19:55.728 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.723+0000 7f8d82ffd700 10 client.50894.objecter ms_dispatch 0x7f8d8c001550 osd_map(853..853 src has 796..853) v4 2024-02-19T21:19:55.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.723+0000 7f8d82ffd700 3 client.50894.objecter handle_osd_map got epochs [853,853] > 0 2024-02-19T21:19:55.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.723+0000 7f8d82ffd700 3 client.50894.objecter handle_osd_map decoding full epoch 853 2024-02-19T21:19:55.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.723+0000 7f8d82ffd700 20 client.50894.objecter dump_active .. 0 homeless 2024-02-19T21:19:55.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.723+0000 7f8d82ffd700 10 client.50894.objecter ms_handle_connect 0x7f8d78063d70 2024-02-19T21:19:55.729 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.728+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f8d8404c640 con 0x7f8d8c1042c0 2024-02-19T21:19:55.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.893+0000 7f8d94068700 1 -- 172.21.15.163:0/3634099635 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8d700051d0 con 0x7f8d8c1042c0 2024-02-19T21:19:55.895 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.893+0000 7f8d82ffd700 1 -- 172.21.15.163:0/3634099635 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (unknown 2131975755 0 2316229251) 0x7f8d8404aab0 con 0x7f8d8c1042c0 2024-02-19T21:19:55.896 INFO:teuthology.orchestra.run.smithi163.stdout:158913790189 2024-02-19T21:19:55.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.895+0000 7f8d94068700 10 librados: watch_flush enter 2024-02-19T21:19:55.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.895+0000 7f8d94068700 10 librados: watch_flush exit 2024-02-19T21:19:55.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.895+0000 7f8d94068700 20 client.50894.objecter shutdown clearing up homeless session... 2024-02-19T21:19:55.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.895+0000 7f8d94068700 10 client.50894.objecter successfully canceled tick 2024-02-19T21:19:55.897 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.895+0000 7f8d94068700 1 -- 172.21.15.163:0/3634099635 >> v1:172.21.15.163:6800/1552514681 conn(0x7f8d78063d70 legacy=0x7f8d78066230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:55.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.895+0000 7f8d94068700 1 -- 172.21.15.163:0/3634099635 >> v1:172.21.15.163:6790/0 conn(0x7f8d8c1042c0 legacy=0x7f8d8c115bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:55.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.895+0000 7f8d94068700 1 -- 172.21.15.163:0/3634099635 shutdown_connections 2024-02-19T21:19:55.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.895+0000 7f8d94068700 1 -- 172.21.15.163:0/3634099635 >> 172.21.15.163:0/3634099635 conn(0x7f8d8c0f35a0 msgr2=0x7f8d8c107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:55.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.895+0000 7f8d94068700 1 -- 172.21.15.163:0/3634099635 shutdown_connections 2024-02-19T21:19:55.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.895+0000 7f8d94068700 1 -- 172.21.15.163:0/3634099635 wait complete. 2024-02-19T21:19:55.898 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:55.895+0000 7f8d94068700 1 librados: shutdown 2024-02-19T21:19:56.190 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:54.311325+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.5 deep-scrub starts 2024-02-19T21:19:56.190 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:54.312398+0000 osd.2 (osd.2) 391 : cluster [DBG] 2.5 deep-scrub ok 2024-02-19T21:19:56.190 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:55.259603+0000 osd.0 (osd.0) 192 : cluster [DBG] 16.14 scrub starts 2024-02-19T21:19:56.190 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:55.260696+0000 osd.0 (osd.0) 193 : cluster [DBG] 16.14 scrub ok 2024-02-19T21:19:56.190 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:55.894667+0000 mon.c (mon.2) 453 : audit [DBG] from='client.? v1:172.21.15.163:0/3634099635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-19T21:19:56.190 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:54.311325+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.5 deep-scrub starts 2024-02-19T21:19:56.191 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:54.312398+0000 osd.2 (osd.2) 391 : cluster [DBG] 2.5 deep-scrub ok 2024-02-19T21:19:56.191 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:55.259603+0000 osd.0 (osd.0) 192 : cluster [DBG] 16.14 scrub starts 2024-02-19T21:19:56.191 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:55.260696+0000 osd.0 (osd.0) 193 : cluster [DBG] 16.14 scrub ok 2024-02-19T21:19:56.191 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:55 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:55.894667+0000 mon.c (mon.2) 453 : audit [DBG] from='client.? v1:172.21.15.163:0/3634099635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-19T21:19:56.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:54.311325+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.5 deep-scrub starts 2024-02-19T21:19:56.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:54.312398+0000 osd.2 (osd.2) 391 : cluster [DBG] 2.5 deep-scrub ok 2024-02-19T21:19:56.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:55.259603+0000 osd.0 (osd.0) 192 : cluster [DBG] 16.14 scrub starts 2024-02-19T21:19:56.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:55.260696+0000 osd.0 (osd.0) 193 : cluster [DBG] 16.14 scrub ok 2024-02-19T21:19:56.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:55.894667+0000 mon.c (mon.2) 453 : audit [DBG] from='client.? v1:172.21.15.163:0/3634099635' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-19T21:19:56.590 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790185 got 158913790189 for osd.5 2024-02-19T21:19:56.590 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-19T21:19:56.858 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:19:57.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:55.380199+0000 mgr.y (mgr.14550) 742 : cluster [DBG] pgmap v1332: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:19:57.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:55.700215+0000 osd.1 (osd.1) 433 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:19:57.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:55.703198+0000 osd.1 (osd.1) 434 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:19:57.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:56.237144+0000 osd.0 (osd.0) 194 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:19:57.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:56.238469+0000 osd.0 (osd.0) 195 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:19:57.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:55.380199+0000 mgr.y (mgr.14550) 742 : cluster [DBG] pgmap v1332: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:19:57.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:55.700215+0000 osd.1 (osd.1) 433 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:19:57.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:55.703198+0000 osd.1 (osd.1) 434 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:19:57.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:56.237144+0000 osd.0 (osd.0) 194 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:19:57.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:56 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:56.238469+0000 osd.0 (osd.0) 195 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:19:57.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:55.380199+0000 mgr.y (mgr.14550) 742 : cluster [DBG] pgmap v1332: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:19:57.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:55.700215+0000 osd.1 (osd.1) 433 : cluster [DBG] 5.19 scrub starts 2024-02-19T21:19:57.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:55.703198+0000 osd.1 (osd.1) 434 : cluster [DBG] 5.19 scrub ok 2024-02-19T21:19:57.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:56.237144+0000 osd.0 (osd.0) 194 : cluster [DBG] 4.6 scrub starts 2024-02-19T21:19:57.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:56.238469+0000 osd.0 (osd.0) 195 : cluster [DBG] 4.6 scrub ok 2024-02-19T21:19:58.073 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:19:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:19:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:19:58.639 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.635+0000 7fdb85f79700 1 -- 172.21.15.163:0/2311964301 >> v1:172.21.15.163:6789/0 conn(0x7fdb800fc870 legacy=0x7fdb800fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:58.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.635+0000 7fdb85f79700 1 -- 172.21.15.163:0/2311964301 shutdown_connections 2024-02-19T21:19:58.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.635+0000 7fdb85f79700 1 -- 172.21.15.163:0/2311964301 >> 172.21.15.163:0/2311964301 conn(0x7fdb800f2470 msgr2=0x7fdb800f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:58.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.635+0000 7fdb85f79700 1 -- 172.21.15.163:0/2311964301 shutdown_connections 2024-02-19T21:19:58.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.635+0000 7fdb85f79700 1 -- 172.21.15.163:0/2311964301 wait complete. 2024-02-19T21:19:58.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.635+0000 7fdb85f79700 1 librados: starting msgr at 2024-02-19T21:19:58.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.635+0000 7fdb85f79700 1 librados: starting objecter 2024-02-19T21:19:58.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb85f79700 1 Processor -- start 2024-02-19T21:19:58.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb85f79700 1 -- start start 2024-02-19T21:19:58.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb85f79700 1 librados: setting wanted keys 2024-02-19T21:19:58.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb85f79700 1 librados: calling monclient init 2024-02-19T21:19:58.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb85f79700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdb80117160 con 0x7fdb800fc870 2024-02-19T21:19:58.640 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb85f79700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdb80118340 con 0x7fdb800ff670 2024-02-19T21:19:58.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb85f79700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdb80119520 con 0x7fdb801031b0 2024-02-19T21:19:58.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb7f7fe700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fdb800fc870 0x7fdb80060ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:51560/0 (socket says 172.21.15.163:51560) 2024-02-19T21:19:58.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb7f7fe700 1 -- 172.21.15.163:0/2253374221 learned_addr learned my addr 172.21.15.163:0/2253374221 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:19:58.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb7ffff700 1 --1- 172.21.15.163:0/2253374221 >> v1:172.21.15.163:6790/0 conn(0x7fdb801031b0 0x7fdb80115840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45314/0 (socket says 172.21.15.163:45314) 2024-02-19T21:19:58.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb7cff9700 10 client.?.objecter ms_handle_connect 0x7fdb800fc870 2024-02-19T21:19:58.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb7cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:58.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb7cff9700 10 client.?.objecter ms_handle_connect 0x7fdb801031b0 2024-02-19T21:19:58.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb7cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:58.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb7cff9700 10 client.?.objecter ms_handle_connect 0x7fdb800ff670 2024-02-19T21:19:58.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb7cff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:19:58.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651065158 0 0) 0x7fdb80117160 con 0x7fdb800fc870 2024-02-19T21:19:58.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb68003030 con 0x7fdb800fc870 2024-02-19T21:19:58.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407123339 0 0) 0x7fdb80119520 con 0x7fdb801031b0 2024-02-19T21:19:58.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb80117160 con 0x7fdb801031b0 2024-02-19T21:19:58.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.636+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656807966 0 0) 0x7fdb80118340 con 0x7fdb800ff670 2024-02-19T21:19:58.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb80119520 con 0x7fdb800ff670 2024-02-19T21:19:58.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3353100547 0 0) 0x7fdb68003030 con 0x7fdb800fc870 2024-02-19T21:19:58.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb80118340 con 0x7fdb800fc870 2024-02-19T21:19:58.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2004032170 0 0) 0x7fdb80117160 con 0x7fdb801031b0 2024-02-19T21:19:58.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb68003030 con 0x7fdb801031b0 2024-02-19T21:19:58.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdb740010c0 con 0x7fdb800fc870 2024-02-19T21:19:58.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdb78001c80 con 0x7fdb801031b0 2024-02-19T21:19:58.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 981313325 0 0) 0x7fdb80119520 con 0x7fdb800ff670 2024-02-19T21:19:58.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb80117160 con 0x7fdb800ff670 2024-02-19T21:19:58.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdb70001840 con 0x7fdb800ff670 2024-02-19T21:19:58.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4010390047 0 0) 0x7fdb80118340 con 0x7fdb800fc870 2024-02-19T21:19:58.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 >> v1:172.21.15.163:6790/0 conn(0x7fdb801031b0 legacy=0x7fdb80115840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:58.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 >> v1:172.21.15.193:6789/0 conn(0x7fdb800ff670 legacy=0x7fdb800612b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:58.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8011a720 con 0x7fdb800fc870 2024-02-19T21:19:58.642 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb85f79700 1 -- 172.21.15.163:0/2253374221 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb80119710 con 0x7fdb800fc870 2024-02-19T21:19:58.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb85f79700 10 client.24174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:19:58.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb85f79700 1 -- 172.21.15.163:0/2253374221 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb80119c00 con 0x7fdb800fc870 2024-02-19T21:19:58.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb85f79700 1 librados: init done 2024-02-19T21:19:58.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fdb74002390 con 0x7fdb800fc870 2024-02-19T21:19:58.643 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.637+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fdb740037d0 con 0x7fdb800fc870 2024-02-19T21:19:58.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.638+0000 7fdb85f79700 1 -- 172.21.15.163:0/2253374221 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6c004f20 con 0x7fdb800fc870 2024-02-19T21:19:58.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.638+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fdb74001880 con 0x7fdb800fc870 2024-02-19T21:19:58.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.639+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(853..853 src has 796..853) v4 ==== 7419+0+0 (unknown 1683610851 0 0) 0x7fdb74078d90 con 0x7fdb800fc870 2024-02-19T21:19:58.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.639+0000 7fdb7cff9700 10 client.24174.objecter ms_dispatch 0x7fdb80001550 osd_map(853..853 src has 796..853) v4 2024-02-19T21:19:58.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.639+0000 7fdb7cff9700 3 client.24174.objecter handle_osd_map got epochs [853,853] > 0 2024-02-19T21:19:58.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.639+0000 7fdb7cff9700 3 client.24174.objecter handle_osd_map decoding full epoch 853 2024-02-19T21:19:58.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.639+0000 7fdb7cff9700 20 client.24174.objecter dump_active .. 0 homeless 2024-02-19T21:19:58.644 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.639+0000 7fdb7cff9700 10 client.24174.objecter ms_handle_connect 0x7fdb68063d00 2024-02-19T21:19:58.645 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.643+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fdb7404c460 con 0x7fdb800fc870 2024-02-19T21:19:58.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.811+0000 7fdb85f79700 1 -- 172.21.15.163:0/2253374221 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fdb6c0051d0 con 0x7fdb800fc870 2024-02-19T21:19:58.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.812+0000 7fdb7cff9700 1 -- 172.21.15.163:0/2253374221 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (unknown 1274345170 0 2132743374) 0x7fdb7404a8d0 con 0x7fdb800fc870 2024-02-19T21:19:58.815 INFO:teuthology.orchestra.run.smithi163.stdout:193273528553 2024-02-19T21:19:58.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.814+0000 7fdb85f79700 10 librados: watch_flush enter 2024-02-19T21:19:58.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.814+0000 7fdb85f79700 10 librados: watch_flush exit 2024-02-19T21:19:58.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.814+0000 7fdb85f79700 20 client.24174.objecter shutdown clearing up homeless session... 2024-02-19T21:19:58.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.814+0000 7fdb85f79700 10 client.24174.objecter successfully canceled tick 2024-02-19T21:19:58.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.814+0000 7fdb85f79700 1 -- 172.21.15.163:0/2253374221 >> v1:172.21.15.163:6800/1552514681 conn(0x7fdb68063d00 legacy=0x7fdb680661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:58.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.814+0000 7fdb85f79700 1 -- 172.21.15.163:0/2253374221 >> v1:172.21.15.163:6789/0 conn(0x7fdb800fc870 legacy=0x7fdb80060ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:19:58.816 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.814+0000 7fdb85f79700 1 -- 172.21.15.163:0/2253374221 shutdown_connections 2024-02-19T21:19:58.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.814+0000 7fdb85f79700 1 -- 172.21.15.163:0/2253374221 >> 172.21.15.163:0/2253374221 conn(0x7fdb800f2470 msgr2=0x7fdb800f5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:19:58.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.814+0000 7fdb85f79700 1 -- 172.21.15.163:0/2253374221 shutdown_connections 2024-02-19T21:19:58.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.814+0000 7fdb85f79700 1 -- 172.21.15.163:0/2253374221 wait complete. 2024-02-19T21:19:58.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:19:58.814+0000 7fdb85f79700 1 librados: shutdown 2024-02-19T21:19:59.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:57.223486+0000 osd.2 (osd.2) 392 : cluster [DBG] 16.1d deep-scrub starts 2024-02-19T21:19:59.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:57.225201+0000 osd.2 (osd.2) 393 : cluster [DBG] 16.1d deep-scrub ok 2024-02-19T21:19:59.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:57.380622+0000 mgr.y (mgr.14550) 743 : cluster [DBG] pgmap v1333: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:59.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:58.282068+0000 osd.0 (osd.0) 196 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:19:59.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:58.284114+0000 osd.0 (osd.0) 197 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:19:59.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:58.350051+0000 osd.7 (osd.7) 367 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:19:59.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:58.352942+0000 osd.7 (osd.7) 368 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:19:59.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:19:58.813462+0000 mon.a (mon.0) 3879 : audit [DBG] from='client.? v1:172.21.15.163:0/2253374221' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-19T21:19:59.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:57.223486+0000 osd.2 (osd.2) 392 : cluster [DBG] 16.1d deep-scrub starts 2024-02-19T21:19:59.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:57.225201+0000 osd.2 (osd.2) 393 : cluster [DBG] 16.1d deep-scrub ok 2024-02-19T21:19:59.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:57.380622+0000 mgr.y (mgr.14550) 743 : cluster [DBG] pgmap v1333: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:59.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:58.282068+0000 osd.0 (osd.0) 196 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:19:59.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:58.284114+0000 osd.0 (osd.0) 197 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:19:59.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:58.350051+0000 osd.7 (osd.7) 367 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:19:59.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:58.352942+0000 osd.7 (osd.7) 368 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:19:59.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:58 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:19:58.813462+0000 mon.a (mon.0) 3879 : audit [DBG] from='client.? v1:172.21.15.163:0/2253374221' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-19T21:19:59.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:57.223486+0000 osd.2 (osd.2) 392 : cluster [DBG] 16.1d deep-scrub starts 2024-02-19T21:19:59.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:57.225201+0000 osd.2 (osd.2) 393 : cluster [DBG] 16.1d deep-scrub ok 2024-02-19T21:19:59.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:57.380622+0000 mgr.y (mgr.14550) 743 : cluster [DBG] pgmap v1333: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:19:59.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:58.282068+0000 osd.0 (osd.0) 196 : cluster [DBG] 4.3 scrub starts 2024-02-19T21:19:59.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:58.284114+0000 osd.0 (osd.0) 197 : cluster [DBG] 4.3 scrub ok 2024-02-19T21:19:59.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:58.350051+0000 osd.7 (osd.7) 367 : cluster [DBG] 2.1c scrub starts 2024-02-19T21:19:59.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:58.352942+0000 osd.7 (osd.7) 368 : cluster [DBG] 2.1c scrub ok 2024-02-19T21:19:59.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:19:58.813462+0000 mon.a (mon.0) 3879 : audit [DBG] from='client.? v1:172.21.15.163:0/2253374221' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-19T21:19:59.820 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528549 got 193273528553 for osd.6 2024-02-19T21:19:59.820 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-19T21:20:00.098 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:20:00.229 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:59.218103+0000 osd.7 (osd.7) 369 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:20:00.230 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:19:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:59.218371+0000 osd.7 (osd.7) 370 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:20:00.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:59.218103+0000 osd.7 (osd.7) 369 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:20:00.230 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:19:59 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:59.218371+0000 osd.7 (osd.7) 370 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:20:00.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:59.218103+0000 osd.7 (osd.7) 369 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:20:00.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:19:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:59.218371+0000 osd.7 (osd.7) 370 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:20:01.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:59.229543+0000 osd.2 (osd.2) 394 : cluster [DBG] 2.10 deep-scrub starts 2024-02-19T21:20:01.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:59.232433+0000 osd.2 (osd.2) 395 : cluster [DBG] 2.10 deep-scrub ok 2024-02-19T21:20:01.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:59.381133+0000 mgr.y (mgr.14550) 744 : cluster [DBG] pgmap v1334: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:20:01.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:59.556753+0000 osd.4 (osd.4) 363 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:20:01.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:59.557891+0000 osd.4 (osd.4) 364 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:20:01.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:19:59.919495+0000 mon.a (mon.0) 3880 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2024-02-19T21:20:01.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:00.000108+0000 mon.a (mon.0) 3881 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-02-19T21:20:01.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:00.000128+0000 mon.a (mon.0) 3882 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-19T21:20:01.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:00.000152+0000 mon.a (mon.0) 3883 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-19T21:20:01.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:00.000159+0000 mon.a (mon.0) 3884 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-19T21:20:01.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:59.229543+0000 osd.2 (osd.2) 394 : cluster [DBG] 2.10 deep-scrub starts 2024-02-19T21:20:01.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:59.232433+0000 osd.2 (osd.2) 395 : cluster [DBG] 2.10 deep-scrub ok 2024-02-19T21:20:01.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:59.381133+0000 mgr.y (mgr.14550) 744 : cluster [DBG] pgmap v1334: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:20:01.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:59.556753+0000 osd.4 (osd.4) 363 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:20:01.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:59.557891+0000 osd.4 (osd.4) 364 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:20:01.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:19:59.919495+0000 mon.a (mon.0) 3880 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2024-02-19T21:20:01.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:00.000108+0000 mon.a (mon.0) 3881 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-02-19T21:20:01.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:00.000128+0000 mon.a (mon.0) 3882 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-19T21:20:01.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:00.000152+0000 mon.a (mon.0) 3883 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-19T21:20:01.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:00.000159+0000 mon.a (mon.0) 3884 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-19T21:20:01.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:20:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:20:01.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:59.229543+0000 osd.2 (osd.2) 394 : cluster [DBG] 2.10 deep-scrub starts 2024-02-19T21:20:01.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:59.232433+0000 osd.2 (osd.2) 395 : cluster [DBG] 2.10 deep-scrub ok 2024-02-19T21:20:01.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:59.381133+0000 mgr.y (mgr.14550) 744 : cluster [DBG] pgmap v1334: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-19T21:20:01.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:59.556753+0000 osd.4 (osd.4) 363 : cluster [DBG] 6.1a scrub starts 2024-02-19T21:20:01.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:59.557891+0000 osd.4 (osd.4) 364 : cluster [DBG] 6.1a scrub ok 2024-02-19T21:20:01.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:19:59.919495+0000 mon.a (mon.0) 3880 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2024-02-19T21:20:01.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:00.000108+0000 mon.a (mon.0) 3881 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-02-19T21:20:01.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:00.000128+0000 mon.a (mon.0) 3882 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-02-19T21:20:01.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:00.000152+0000 mon.a (mon.0) 3883 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-02-19T21:20:01.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:00.000159+0000 mon.a (mon.0) 3884 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-19T21:20:01.905 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.901+0000 7f6be0d91700 1 -- 172.21.15.163:0/1890933358 >> v1:172.21.15.163:6790/0 conn(0x7f6bdc1031d0 legacy=0x7f6bdc105650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.901+0000 7f6be0d91700 1 -- 172.21.15.163:0/1890933358 shutdown_connections 2024-02-19T21:20:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.901+0000 7f6be0d91700 1 -- 172.21.15.163:0/1890933358 >> 172.21.15.163:0/1890933358 conn(0x7f6bdc0f2470 msgr2=0x7f6bdc0f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:20:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.901+0000 7f6be0d91700 1 -- 172.21.15.163:0/1890933358 shutdown_connections 2024-02-19T21:20:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.901+0000 7f6be0d91700 1 -- 172.21.15.163:0/1890933358 wait complete. 2024-02-19T21:20:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.901+0000 7f6be0d91700 1 librados: starting msgr at 2024-02-19T21:20:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.901+0000 7f6be0d91700 1 librados: starting objecter 2024-02-19T21:20:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.901+0000 7f6be0d91700 1 Processor -- start 2024-02-19T21:20:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6be0d91700 1 -- start start 2024-02-19T21:20:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6be0d91700 1 librados: setting wanted keys 2024-02-19T21:20:01.906 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6be0d91700 1 librados: calling monclient init 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6be0d91700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6bdc1162c0 con 0x7f6bdc1031d0 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6be0d91700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6bdc1174c0 con 0x7f6bdc0ff690 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6be0d91700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6bdc1186c0 con 0x7f6bdc0fc890 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bda59c700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f6bdc0fc890 0x7f6bdc0f8870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:45342/0 (socket says 172.21.15.163:45342) 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bda59c700 1 -- 172.21.15.163:0/4051971531 learned_addr learned my addr 172.21.15.163:0/4051971531 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bcb7fe700 10 client.?.objecter ms_handle_connect 0x7f6bdc0fc890 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bcb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bcb7fe700 10 client.?.objecter ms_handle_connect 0x7f6bdc1031d0 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bcb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bcb7fe700 10 client.?.objecter ms_handle_connect 0x7f6bdc0ff690 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bcb7fe700 10 client.?.objecter resend_mon_ops 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984895954 0 0) 0x7f6bdc1186c0 con 0x7f6bdc0fc890 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc40038e0 con 0x7f6bdc0fc890 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630924773 0 0) 0x7f6bdc1162c0 con 0x7f6bdc1031d0 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bdc1186c0 con 0x7f6bdc1031d0 2024-02-19T21:20:01.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012926300 0 0) 0x7f6bdc1174c0 con 0x7f6bdc0ff690 2024-02-19T21:20:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bdc1162c0 con 0x7f6bdc0ff690 2024-02-19T21:20:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1264637626 0 0) 0x7f6bc40038e0 con 0x7f6bdc0fc890 2024-02-19T21:20:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bdc1174c0 con 0x7f6bdc0fc890 2024-02-19T21:20:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.902+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6bcc001a70 con 0x7f6bdc0fc890 2024-02-19T21:20:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1529446796 0 0) 0x7f6bdc1186c0 con 0x7f6bdc1031d0 2024-02-19T21:20:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bc40038e0 con 0x7f6bdc1031d0 2024-02-19T21:20:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 656207563 0 0) 0x7f6bdc1162c0 con 0x7f6bdc0ff690 2024-02-19T21:20:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bdc1186c0 con 0x7f6bdc0ff690 2024-02-19T21:20:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6bd4001950 con 0x7f6bdc1031d0 2024-02-19T21:20:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6bd0001840 con 0x7f6bdc0ff690 2024-02-19T21:20:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.2 v1:172.21.15.163:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1511570017 0 0) 0x7f6bdc1174c0 con 0x7f6bdc0fc890 2024-02-19T21:20:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 >> v1:172.21.15.193:6789/0 conn(0x7f6bdc0ff690 legacy=0x7f6bdc0f9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:01.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 >> v1:172.21.15.163:6789/0 conn(0x7f6bdc1031d0 legacy=0x7f6bdc114980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 --> v1:172.21.15.163:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc1198c0 con 0x7f6bdc0fc890 2024-02-19T21:20:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6be0d91700 1 -- 172.21.15.163:0/4051971531 --> v1:172.21.15.163:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bdc1164b0 con 0x7f6bdc0fc890 2024-02-19T21:20:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6be0d91700 10 client.50906.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:20:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6be0d91700 1 -- 172.21.15.163:0/4051971531 --> v1:172.21.15.163:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bdc116920 con 0x7f6bdc0fc890 2024-02-19T21:20:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6be0d91700 1 librados: init done 2024-02-19T21:20:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.2 v1:172.21.15.163:6790/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f6bcc0023e0 con 0x7f6bdc0fc890 2024-02-19T21:20:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.903+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.2 v1:172.21.15.163:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f6bcc005a60 con 0x7f6bdc0fc890 2024-02-19T21:20:01.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.904+0000 7f6be0d91700 1 -- 172.21.15.163:0/4051971531 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bbc004f20 con 0x7f6bdc0fc890 2024-02-19T21:20:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.904+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.2 v1:172.21.15.163:6790/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f6bcc001f40 con 0x7f6bdc0fc890 2024-02-19T21:20:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.905+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.2 v1:172.21.15.163:6790/0 8 ==== osd_map(854..854 src has 796..854) v4 ==== 7419+0+0 (unknown 833814824 0 0) 0x7f6bcc078a20 con 0x7f6bdc0fc890 2024-02-19T21:20:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.905+0000 7f6bcb7fe700 10 client.50906.objecter ms_dispatch 0x7f6bdc001550 osd_map(854..854 src has 796..854) v4 2024-02-19T21:20:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.905+0000 7f6bcb7fe700 3 client.50906.objecter handle_osd_map got epochs [854,854] > 0 2024-02-19T21:20:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.905+0000 7f6bcb7fe700 3 client.50906.objecter handle_osd_map decoding full epoch 854 2024-02-19T21:20:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.905+0000 7f6bcb7fe700 20 client.50906.objecter dump_active .. 0 homeless 2024-02-19T21:20:01.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.905+0000 7f6bcb7fe700 10 client.50906.objecter ms_handle_connect 0x7f6bc4063cc0 2024-02-19T21:20:01.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:01.909+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.2 v1:172.21.15.163:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f6bcc04c0f0 con 0x7f6bdc0fc890 2024-02-19T21:20:02.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:02.074+0000 7f6be0d91700 1 -- 172.21.15.163:0/4051971531 --> v1:172.21.15.163:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f6bbc0051d0 con 0x7f6bdc0fc890 2024-02-19T21:20:02.076 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:02.074+0000 7f6bcb7fe700 1 -- 172.21.15.163:0/4051971531 <== mon.2 v1:172.21.15.163:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (unknown 1482059429 0 3970207793) 0x7f6bcc04a500 con 0x7f6bdc0fc890 2024-02-19T21:20:02.076 INFO:teuthology.orchestra.run.smithi163.stdout:227633266916 2024-02-19T21:20:02.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:02.075+0000 7f6be0d91700 10 librados: watch_flush enter 2024-02-19T21:20:02.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:02.076+0000 7f6be0d91700 10 librados: watch_flush exit 2024-02-19T21:20:02.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:02.076+0000 7f6be0d91700 20 client.50906.objecter shutdown clearing up homeless session... 2024-02-19T21:20:02.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:02.076+0000 7f6be0d91700 10 client.50906.objecter successfully canceled tick 2024-02-19T21:20:02.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:02.076+0000 7f6be0d91700 1 -- 172.21.15.163:0/4051971531 >> v1:172.21.15.163:6800/1552514681 conn(0x7f6bc4063cc0 legacy=0x7f6bc4066180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:02.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:02.076+0000 7f6be0d91700 1 -- 172.21.15.163:0/4051971531 >> v1:172.21.15.163:6790/0 conn(0x7f6bdc0fc890 legacy=0x7f6bdc0f8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:02.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:02.076+0000 7f6be0d91700 1 -- 172.21.15.163:0/4051971531 shutdown_connections 2024-02-19T21:20:02.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:02.076+0000 7f6be0d91700 1 -- 172.21.15.163:0/4051971531 >> 172.21.15.163:0/4051971531 conn(0x7f6bdc0f2470 msgr2=0x7f6bdc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:20:02.078 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:02.076+0000 7f6be0d91700 1 -- 172.21.15.163:0/4051971531 shutdown_connections 2024-02-19T21:20:02.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:02.076+0000 7f6be0d91700 1 -- 172.21.15.163:0/4051971531 wait complete. 2024-02-19T21:20:02.079 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:02.076+0000 7f6be0d91700 1 librados: shutdown 2024-02-19T21:20:02.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:00.675608+0000 osd.1 (osd.1) 435 : cluster [DBG] 5.c scrub starts 2024-02-19T21:20:02.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:00.676578+0000 osd.1 (osd.1) 436 : cluster [DBG] 5.c scrub ok 2024-02-19T21:20:02.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:20:00.942424+0000 mgr.y (mgr.14550) 745 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:20:02.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:01.304928+0000 osd.0 (osd.0) 198 : cluster [DBG] 16.15 deep-scrub starts 2024-02-19T21:20:02.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:01.306082+0000 osd.0 (osd.0) 199 : cluster [DBG] 16.15 deep-scrub ok 2024-02-19T21:20:02.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:01.336404+0000 osd.7 (osd.7) 371 : cluster [DBG] 2.26 scrub starts 2024-02-19T21:20:02.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:01.339295+0000 osd.7 (osd.7) 372 : cluster [DBG] 2.26 scrub ok 2024-02-19T21:20:02.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:01.521105+0000 osd.6 (osd.6) 351 : cluster [DBG] 2.29 scrub starts 2024-02-19T21:20:02.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:01.523995+0000 osd.6 (osd.6) 352 : cluster [DBG] 2.29 scrub ok 2024-02-19T21:20:02.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:00.675608+0000 osd.1 (osd.1) 435 : cluster [DBG] 5.c scrub starts 2024-02-19T21:20:02.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:00.676578+0000 osd.1 (osd.1) 436 : cluster [DBG] 5.c scrub ok 2024-02-19T21:20:02.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:20:00.942424+0000 mgr.y (mgr.14550) 745 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:20:02.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:01.304928+0000 osd.0 (osd.0) 198 : cluster [DBG] 16.15 deep-scrub starts 2024-02-19T21:20:02.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:01.306082+0000 osd.0 (osd.0) 199 : cluster [DBG] 16.15 deep-scrub ok 2024-02-19T21:20:02.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:01.336404+0000 osd.7 (osd.7) 371 : cluster [DBG] 2.26 scrub starts 2024-02-19T21:20:02.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:01.339295+0000 osd.7 (osd.7) 372 : cluster [DBG] 2.26 scrub ok 2024-02-19T21:20:02.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:01.521105+0000 osd.6 (osd.6) 351 : cluster [DBG] 2.29 scrub starts 2024-02-19T21:20:02.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:01 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:01.523995+0000 osd.6 (osd.6) 352 : cluster [DBG] 2.29 scrub ok 2024-02-19T21:20:02.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:00.675608+0000 osd.1 (osd.1) 435 : cluster [DBG] 5.c scrub starts 2024-02-19T21:20:02.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:00.676578+0000 osd.1 (osd.1) 436 : cluster [DBG] 5.c scrub ok 2024-02-19T21:20:02.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:20:00.942424+0000 mgr.y (mgr.14550) 745 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:20:02.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:01.304928+0000 osd.0 (osd.0) 198 : cluster [DBG] 16.15 deep-scrub starts 2024-02-19T21:20:02.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:01.306082+0000 osd.0 (osd.0) 199 : cluster [DBG] 16.15 deep-scrub ok 2024-02-19T21:20:02.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:01.336404+0000 osd.7 (osd.7) 371 : cluster [DBG] 2.26 scrub starts 2024-02-19T21:20:02.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:01.339295+0000 osd.7 (osd.7) 372 : cluster [DBG] 2.26 scrub ok 2024-02-19T21:20:02.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:01.521105+0000 osd.6 (osd.6) 351 : cluster [DBG] 2.29 scrub starts 2024-02-19T21:20:02.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:01.523995+0000 osd.6 (osd.6) 352 : cluster [DBG] 2.29 scrub ok 2024-02-19T21:20:02.738 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266911 got 227633266916 for osd.7 2024-02-19T21:20:02.738 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-19T21:20:02.738 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:20:02.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:01.381938+0000 mgr.y (mgr.14550) 746 : cluster [DBG] pgmap v1336: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:20:02.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:20:02.075541+0000 mon.c (mon.2) 454 : audit [DBG] from='client.? v1:172.21.15.163:0/4051971531' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-19T21:20:03.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:01.381938+0000 mgr.y (mgr.14550) 746 : cluster [DBG] pgmap v1336: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:20:03.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:02 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:20:02.075541+0000 mon.c (mon.2) 454 : audit [DBG] from='client.? v1:172.21.15.163:0/4051971531' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-19T21:20:03.015 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:20:03.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:01.381938+0000 mgr.y (mgr.14550) 746 : cluster [DBG] pgmap v1336: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:20:03.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:20:02.075541+0000 mon.c (mon.2) 454 : audit [DBG] from='client.? v1:172.21.15.163:0/4051971531' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-19T21:20:03.749 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:20:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:20:03] "GET /metrics HTTP/1.1" 200 187981 "" "Prometheus/2.33.4" 2024-02-19T21:20:04.199 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:02.239773+0000 osd.2 (osd.2) 396 : cluster [DBG] 16.a scrub starts 2024-02-19T21:20:04.200 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:02.241050+0000 osd.2 (osd.2) 397 : cluster [DBG] 16.a scrub ok 2024-02-19T21:20:04.200 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:02.401218+0000 osd.3 (osd.3) 433 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:20:04.200 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:02.402697+0000 osd.3 (osd.3) 434 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:20:04.200 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:03.540044+0000 osd.6 (osd.6) 353 : cluster [DBG] 2.24 scrub starts 2024-02-19T21:20:04.200 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:03.540984+0000 osd.6 (osd.6) 354 : cluster [DBG] 2.24 scrub ok 2024-02-19T21:20:04.200 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:02.239773+0000 osd.2 (osd.2) 396 : cluster [DBG] 16.a scrub starts 2024-02-19T21:20:04.200 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:02.241050+0000 osd.2 (osd.2) 397 : cluster [DBG] 16.a scrub ok 2024-02-19T21:20:04.200 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:02.401218+0000 osd.3 (osd.3) 433 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:20:04.200 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:02.402697+0000 osd.3 (osd.3) 434 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:20:04.200 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:03.540044+0000 osd.6 (osd.6) 353 : cluster [DBG] 2.24 scrub starts 2024-02-19T21:20:04.200 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:03.540984+0000 osd.6 (osd.6) 354 : cluster [DBG] 2.24 scrub ok 2024-02-19T21:20:04.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:02.239773+0000 osd.2 (osd.2) 396 : cluster [DBG] 16.a scrub starts 2024-02-19T21:20:04.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:02.241050+0000 osd.2 (osd.2) 397 : cluster [DBG] 16.a scrub ok 2024-02-19T21:20:04.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:02.401218+0000 osd.3 (osd.3) 433 : cluster [DBG] 4.1d scrub starts 2024-02-19T21:20:04.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:02.402697+0000 osd.3 (osd.3) 434 : cluster [DBG] 4.1d scrub ok 2024-02-19T21:20:04.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:03.540044+0000 osd.6 (osd.6) 353 : cluster [DBG] 2.24 scrub starts 2024-02-19T21:20:04.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:03.540984+0000 osd.6 (osd.6) 354 : cluster [DBG] 2.24 scrub ok 2024-02-19T21:20:04.764 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.759+0000 7f4d1a930700 1 -- 172.21.15.163:0/3888208124 >> v1:172.21.15.163:6790/0 conn(0x7f4d140fc870 legacy=0x7f4d140fcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:04.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.760+0000 7f4d1a930700 1 -- 172.21.15.163:0/3888208124 shutdown_connections 2024-02-19T21:20:04.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.760+0000 7f4d1a930700 1 -- 172.21.15.163:0/3888208124 >> 172.21.15.163:0/3888208124 conn(0x7f4d140f2470 msgr2=0x7f4d140f4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:20:04.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.760+0000 7f4d1a930700 1 -- 172.21.15.163:0/3888208124 shutdown_connections 2024-02-19T21:20:04.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.760+0000 7f4d1a930700 1 -- 172.21.15.163:0/3888208124 wait complete. 2024-02-19T21:20:04.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.760+0000 7f4d1a930700 1 librados: starting msgr at 2024-02-19T21:20:04.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.760+0000 7f4d1a930700 1 librados: starting objecter 2024-02-19T21:20:04.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.760+0000 7f4d1a930700 1 Processor -- start 2024-02-19T21:20:04.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.760+0000 7f4d1a930700 1 -- start start 2024-02-19T21:20:04.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.760+0000 7f4d1a930700 1 librados: setting wanted keys 2024-02-19T21:20:04.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.760+0000 7f4d1a930700 1 librados: calling monclient init 2024-02-19T21:20:04.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d1a930700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4d14117230 con 0x7f4d141031b0 2024-02-19T21:20:04.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d1a930700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4d14118410 con 0x7f4d140ff670 2024-02-19T21:20:04.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d1a930700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4d14119610 con 0x7f4d140fc870 2024-02-19T21:20:04.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d13fff700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7f4d140fc870 0x7f4d140f98a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:58962/0 (socket says 172.21.15.163:58962) 2024-02-19T21:20:04.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d13fff700 1 -- 172.21.15.163:0/2639476106 learned_addr learned my addr 172.21.15.163:0/2639476106 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:20:04.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 10 client.?.objecter ms_handle_connect 0x7f4d141031b0 2024-02-19T21:20:04.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:20:04.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 10 client.?.objecter ms_handle_connect 0x7f4d140fc870 2024-02-19T21:20:04.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:20:04.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 10 client.?.objecter ms_handle_connect 0x7f4d140ff670 2024-02-19T21:20:04.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:20:04.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965382112 0 0) 0x7f4d14117230 con 0x7f4d141031b0 2024-02-19T21:20:04.766 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cfc003800 con 0x7f4d141031b0 2024-02-19T21:20:04.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098035810 0 0) 0x7f4d14119610 con 0x7f4d140fc870 2024-02-19T21:20:04.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d14117230 con 0x7f4d140fc870 2024-02-19T21:20:04.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617756572 0 0) 0x7f4d14118410 con 0x7f4d140ff670 2024-02-19T21:20:04.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d14119610 con 0x7f4d140ff670 2024-02-19T21:20:04.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2221562241 0 0) 0x7f4cfc003800 con 0x7f4d141031b0 2024-02-19T21:20:04.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d14118410 con 0x7f4d141031b0 2024-02-19T21:20:04.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3752369985 0 0) 0x7f4d14117230 con 0x7f4d140fc870 2024-02-19T21:20:04.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cfc003800 con 0x7f4d140fc870 2024-02-19T21:20:04.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4d0c001c40 con 0x7f4d141031b0 2024-02-19T21:20:04.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4d04002dd0 con 0x7f4d140fc870 2024-02-19T21:20:04.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1358590102 0 0) 0x7f4d14119610 con 0x7f4d140ff670 2024-02-19T21:20:04.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d14117230 con 0x7f4d140ff670 2024-02-19T21:20:04.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4d08001620 con 0x7f4d140ff670 2024-02-19T21:20:04.767 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2982789321 0 0) 0x7f4d14118410 con 0x7f4d141031b0 2024-02-19T21:20:04.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 >> v1:172.21.15.163:6790/0 conn(0x7f4d140fc870 legacy=0x7f4d140f98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:04.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 >> v1:172.21.15.193:6789/0 conn(0x7f4d140ff670 legacy=0x7f4d140f9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:04.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1411a810 con 0x7f4d141031b0 2024-02-19T21:20:04.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d1a930700 1 -- 172.21.15.163:0/2639476106 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d14119800 con 0x7f4d141031b0 2024-02-19T21:20:04.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d1a930700 10 client.24186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:20:04.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.761+0000 7f4d1a930700 1 -- 172.21.15.163:0/2639476106 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d14119cf0 con 0x7f4d141031b0 2024-02-19T21:20:04.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.762+0000 7f4d1a930700 1 librados: init done 2024-02-19T21:20:04.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.762+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7f4d0c0022d0 con 0x7f4d141031b0 2024-02-19T21:20:04.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.762+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7f4d0c003640 con 0x7f4d141031b0 2024-02-19T21:20:04.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.762+0000 7f4d1a930700 1 -- 172.21.15.163:0/2639476106 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d00004f20 con 0x7f4d141031b0 2024-02-19T21:20:04.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.763+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7f4d0c0037a0 con 0x7f4d141031b0 2024-02-19T21:20:04.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.763+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(854..854 src has 796..854) v4 ==== 7419+0+0 (unknown 833814824 0 0) 0x7f4d0c076450 con 0x7f4d141031b0 2024-02-19T21:20:04.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.763+0000 7f4d117fa700 10 client.24186.objecter ms_dispatch 0x7f4d14001550 osd_map(854..854 src has 796..854) v4 2024-02-19T21:20:04.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.763+0000 7f4d117fa700 3 client.24186.objecter handle_osd_map got epochs [854,854] > 0 2024-02-19T21:20:04.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.763+0000 7f4d117fa700 3 client.24186.objecter handle_osd_map decoding full epoch 854 2024-02-19T21:20:04.769 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.763+0000 7f4d117fa700 20 client.24186.objecter dump_active .. 0 homeless 2024-02-19T21:20:04.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.764+0000 7f4d117fa700 10 client.24186.objecter ms_handle_connect 0x7f4cfc063df0 2024-02-19T21:20:04.770 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.767+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7f4d0c049c70 con 0x7f4d141031b0 2024-02-19T21:20:04.931 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.929+0000 7f4d1a930700 1 -- 172.21.15.163:0/2639476106 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4d00000b90 con 0x7f4cfc063df0 2024-02-19T21:20:04.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.945+0000 7f4d117fa700 1 -- 172.21.15.163:0/2639476106 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+454498 (unknown 2965378022 0 215417214) 0x7f4d00000b90 con 0x7f4cfc063df0 2024-02-19T21:20:04.955 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:20:04.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.948+0000 7f4d1a930700 10 librados: watch_flush enter 2024-02-19T21:20:04.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.948+0000 7f4d1a930700 10 librados: watch_flush exit 2024-02-19T21:20:04.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.948+0000 7f4d1a930700 20 client.24186.objecter shutdown clearing up homeless session... 2024-02-19T21:20:04.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.948+0000 7f4d1a930700 10 client.24186.objecter successfully canceled tick 2024-02-19T21:20:04.960 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.948+0000 7f4d1a930700 1 -- 172.21.15.163:0/2639476106 >> v1:172.21.15.163:6800/1552514681 conn(0x7f4cfc063df0 legacy=0x7f4cfc0662b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:04.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.948+0000 7f4d1a930700 1 -- 172.21.15.163:0/2639476106 >> v1:172.21.15.163:6789/0 conn(0x7f4d141031b0 legacy=0x7f4d141158f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:04.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.948+0000 7f4d1a930700 1 -- 172.21.15.163:0/2639476106 shutdown_connections 2024-02-19T21:20:04.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.948+0000 7f4d1a930700 1 -- 172.21.15.163:0/2639476106 >> 172.21.15.163:0/2639476106 conn(0x7f4d140f2470 msgr2=0x7f4d14106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:20:04.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.948+0000 7f4d1a930700 1 -- 172.21.15.163:0/2639476106 shutdown_connections 2024-02-19T21:20:04.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.948+0000 7f4d1a930700 1 -- 172.21.15.163:0/2639476106 wait complete. 2024-02-19T21:20:04.961 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:04.948+0000 7f4d1a930700 1 librados: shutdown 2024-02-19T21:20:04.961 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:20:05.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:03.382442+0000 mgr.y (mgr.14550) 747 : cluster [DBG] pgmap v1337: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:20:05.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:03.593578+0000 osd.4 (osd.4) 365 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:20:05.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:03.595406+0000 osd.4 (osd.4) 366 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:20:05.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:03.690669+0000 osd.1 (osd.1) 437 : cluster [DBG] 4.f scrub starts 2024-02-19T21:20:05.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:03.694139+0000 osd.1 (osd.1) 438 : cluster [DBG] 4.f scrub ok 2024-02-19T21:20:05.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:04.330468+0000 osd.0 (osd.0) 200 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:20:05.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:04.331469+0000 osd.0 (osd.0) 201 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:20:05.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:03.382442+0000 mgr.y (mgr.14550) 747 : cluster [DBG] pgmap v1337: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:20:05.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:03.593578+0000 osd.4 (osd.4) 365 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:20:05.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:03.595406+0000 osd.4 (osd.4) 366 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:20:05.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:03.690669+0000 osd.1 (osd.1) 437 : cluster [DBG] 4.f scrub starts 2024-02-19T21:20:05.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:03.694139+0000 osd.1 (osd.1) 438 : cluster [DBG] 4.f scrub ok 2024-02-19T21:20:05.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:04.330468+0000 osd.0 (osd.0) 200 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:20:05.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:04 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:04.331469+0000 osd.0 (osd.0) 201 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:20:05.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:03.382442+0000 mgr.y (mgr.14550) 747 : cluster [DBG] pgmap v1337: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-19T21:20:05.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:03.593578+0000 osd.4 (osd.4) 365 : cluster [DBG] 4.1b scrub starts 2024-02-19T21:20:05.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:03.595406+0000 osd.4 (osd.4) 366 : cluster [DBG] 4.1b scrub ok 2024-02-19T21:20:05.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:03.690669+0000 osd.1 (osd.1) 437 : cluster [DBG] 4.f scrub starts 2024-02-19T21:20:05.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:03.694139+0000 osd.1 (osd.1) 438 : cluster [DBG] 4.f scrub ok 2024-02-19T21:20:05.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:04.330468+0000 osd.0 (osd.0) 200 : cluster [DBG] 2.13 scrub starts 2024-02-19T21:20:05.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:04.331469+0000 osd.0 (osd.0) 201 : cluster [DBG] 2.13 scrub ok 2024-02-19T21:20:05.721 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":1337,"stamp":"2024-02-19T21:20:03.382159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":193,"num_object_clones":0,"num_object_copies":579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":2229,"num_read_kb":1981,"num_write":720,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":775,"num_bytes_recovered":19845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":908,"ondisk_log_size":908,"up":522,"acting":522,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":522,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2437144,"kb_used_data":14296,"kb_used_omap":0,"kb_used_meta":2422784,"kb_avail":747524072,"statfs":{"total":767960285184,"available":765464649728,"internally_reserved":0,"allocated":14639104,"data_stored":4543387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2480930816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003028"},"pg_stats":[{"pgid":"2.29","version":"0'0","reported_seq":287,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968943+0000","last_change":"2024-02-19T21:18:48.712881+0000","last_active":"2024-02-19T21:19:40.968943+0000","last_peered":"2024-02-19T21:19:40.968943+0000","last_clean":"2024-02-19T21:19:40.968943+0000","last_became_active":"2024-02-19T21:18:03.725668+0000","last_became_peered":"2024-02-19T21:18:03.725668+0000","last_unstale":"2024-02-19T21:19:40.968943+0000","last_undegraded":"2024-02-19T21:19:40.968943+0000","last_fullsized":"2024-02-19T21:19:40.968943+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:48.712812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:18:48.712812+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":429,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852374+0000","last_change":"2024-02-19T21:18:59.213488+0000","last_active":"2024-02-19T21:19:40.852374+0000","last_peered":"2024-02-19T21:19:40.852374+0000","last_clean":"2024-02-19T21:19:40.852374+0000","last_became_active":"2024-02-19T21:18:03.741123+0000","last_became_peered":"2024-02-19T21:18:03.741123+0000","last_unstale":"2024-02-19T21:19:40.852374+0000","last_undegraded":"2024-02-19T21:19:40.852374+0000","last_fullsized":"2024-02-19T21:19:40.852374+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:59.213418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:04.962987+0000","last_clean_scrub_stamp":"2024-02-19T21:18:59.213418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.25","version":"0'0","reported_seq":928,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.379648+0000","last_change":"2024-02-19T21:19:10.500294+0000","last_active":"2024-02-19T21:19:39.379648+0000","last_peered":"2024-02-19T21:19:39.379648+0000","last_clean":"2024-02-19T21:19:39.379648+0000","last_became_active":"2024-02-19T21:17:58.673707+0000","last_became_peered":"2024-02-19T21:17:58.673707+0000","last_unstale":"2024-02-19T21:19:39.379648+0000","last_undegraded":"2024-02-19T21:19:39.379648+0000","last_fullsized":"2024-02-19T21:19:39.379648+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:10.500215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:10.500215+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":265,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968022+0000","last_change":"2024-02-19T21:18:34.750459+0000","last_active":"2024-02-19T21:19:40.968022+0000","last_peered":"2024-02-19T21:19:40.968022+0000","last_clean":"2024-02-19T21:19:40.968022+0000","last_became_active":"2024-02-19T21:18:03.725448+0000","last_became_peered":"2024-02-19T21:18:03.725448+0000","last_unstale":"2024-02-19T21:19:40.968022+0000","last_undegraded":"2024-02-19T21:19:40.968022+0000","last_fullsized":"2024-02-19T21:19:40.968022+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:34.750379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:18:34.750379+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":608,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969424+0000","last_change":"2024-02-19T21:19:21.592636+0000","last_active":"2024-02-19T21:19:40.969424+0000","last_peered":"2024-02-19T21:19:40.969424+0000","last_clean":"2024-02-19T21:19:40.969424+0000","last_became_active":"2024-02-19T21:19:10.740468+0000","last_became_peered":"2024-02-19T21:19:10.740468+0000","last_unstale":"2024-02-19T21:19:40.969424+0000","last_undegraded":"2024-02-19T21:19:40.969424+0000","last_fullsized":"2024-02-19T21:19:40.969424+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:21.592563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:21.592563+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.26","version":"0'0","reported_seq":603,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969724+0000","last_change":"2024-02-19T21:19:10.742290+0000","last_active":"2024-02-19T21:19:40.969724+0000","last_peered":"2024-02-19T21:19:40.969724+0000","last_clean":"2024-02-19T21:19:40.969724+0000","last_became_active":"2024-02-19T21:19:10.741937+0000","last_became_peered":"2024-02-19T21:19:10.741937+0000","last_unstale":"2024-02-19T21:19:40.969724+0000","last_undegraded":"2024-02-19T21:19:40.969724+0000","last_fullsized":"2024-02-19T21:19:40.969724+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:47.416506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:47.416506+0000","last_clean_scrub_stamp":"2024-02-19T21:18:47.416506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":96,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.829586+0000","last_change":"2024-02-19T21:19:25.999950+0000","last_active":"2024-02-19T21:19:38.829586+0000","last_peered":"2024-02-19T21:19:38.829586+0000","last_clean":"2024-02-19T21:19:38.829586+0000","last_became_active":"2024-02-19T21:18:03.741741+0000","last_became_peered":"2024-02-19T21:18:03.741741+0000","last_unstale":"2024-02-19T21:19:38.829586+0000","last_undegraded":"2024-02-19T21:19:38.829586+0000","last_fullsized":"2024-02-19T21:19:38.829586+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:25.999844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:25.999844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.20","version":"397'2","reported_seq":649,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967232+0000","last_change":"2024-02-19T21:19:24.659934+0000","last_active":"2024-02-19T21:19:40.967232+0000","last_peered":"2024-02-19T21:19:40.967232+0000","last_clean":"2024-02-19T21:19:40.967232+0000","last_became_active":"2024-02-19T21:19:10.741757+0000","last_became_peered":"2024-02-19T21:19:10.741757+0000","last_unstale":"2024-02-19T21:19:40.967232+0000","last_undegraded":"2024-02-19T21:19:40.967232+0000","last_fullsized":"2024-02-19T21:19:40.967232+0000","mapping_epoch":848,"log_start":"58'1","ondisk_log_start":"58'1","created":474,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":6,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:19:24.659860+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:19:24.659860+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":420,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:47.580055+0000","last_change":"2024-02-19T21:19:47.580032+0000","last_active":"2024-02-19T21:19:47.580055+0000","last_peered":"2024-02-19T21:19:47.580055+0000","last_clean":"2024-02-19T21:19:47.580055+0000","last_became_active":"2024-02-19T21:14:16.384406+0000","last_became_peered":"2024-02-19T21:14:16.384406+0000","last_unstale":"2024-02-19T21:19:47.580055+0000","last_undegraded":"2024-02-19T21:19:47.580055+0000","last_fullsized":"2024-02-19T21:19:47.580055+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:47.579967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:47.579967+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":252,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:52.569047+0000","last_change":"2024-02-19T21:19:52.569011+0000","last_active":"2024-02-19T21:19:52.569047+0000","last_peered":"2024-02-19T21:19:52.569047+0000","last_clean":"2024-02-19T21:19:52.569047+0000","last_became_active":"2024-02-19T21:14:16.382869+0000","last_became_peered":"2024-02-19T21:14:16.382869+0000","last_unstale":"2024-02-19T21:19:52.569047+0000","last_undegraded":"2024-02-19T21:19:52.569047+0000","last_fullsized":"2024-02-19T21:19:52.569047+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:52.568919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:51.187606+0000","last_clean_scrub_stamp":"2024-02-19T21:19:52.568919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"849'18","reported_seq":687,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:59.802252+0000","last_change":"2024-02-19T21:19:18.564012+0000","last_active":"2024-02-19T21:19:59.802252+0000","last_peered":"2024-02-19T21:19:59.802252+0000","last_clean":"2024-02-19T21:19:59.802252+0000","last_became_active":"2024-02-19T21:19:10.741991+0000","last_became_peered":"2024-02-19T21:19:10.741991+0000","last_unstale":"2024-02-19T21:19:59.802252+0000","last_undegraded":"2024-02-19T21:19:59.802252+0000","last_fullsized":"2024-02-19T21:19:59.802252+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"849'18","last_scrub_stamp":"2024-02-19T21:19:18.563931+0000","last_deep_scrub":"849'18","last_deep_scrub_stamp":"2024-02-19T21:19:18.563931+0000","last_clean_scrub_stamp":"2024-02-19T21:19:18.563931+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'28","reported_seq":99,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.802714+0000","last_change":"2024-02-19T21:19:15.626733+0000","last_active":"2024-02-19T21:19:39.802714+0000","last_peered":"2024-02-19T21:19:39.802714+0000","last_clean":"2024-02-19T21:19:39.802714+0000","last_became_active":"2024-02-19T21:18:03.734723+0000","last_became_peered":"2024-02-19T21:18:03.734723+0000","last_unstale":"2024-02-19T21:19:39.802714+0000","last_undegraded":"2024-02-19T21:19:39.802714+0000","last_fullsized":"2024-02-19T21:19:39.802714+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-19T21:19:15.626643+0000","last_deep_scrub":"473'28","last_deep_scrub_stamp":"2024-02-19T21:11:47.051823+0000","last_clean_scrub_stamp":"2024-02-19T21:19:15.626643+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":96,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.828372+0000","last_change":"2024-02-19T21:19:13.916725+0000","last_active":"2024-02-19T21:19:38.828372+0000","last_peered":"2024-02-19T21:19:38.828372+0000","last_clean":"2024-02-19T21:19:38.828372+0000","last_became_active":"2024-02-19T21:18:03.742827+0000","last_became_peered":"2024-02-19T21:18:03.742827+0000","last_unstale":"2024-02-19T21:19:38.828372+0000","last_undegraded":"2024-02-19T21:19:38.828372+0000","last_fullsized":"2024-02-19T21:19:38.828372+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:13.916659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:13.916659+0000","last_clean_scrub_stamp":"2024-02-19T21:19:13.916659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":58,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.803733+0000","last_change":"2024-02-19T21:19:18.647920+0000","last_active":"2024-02-19T21:19:39.803733+0000","last_peered":"2024-02-19T21:19:39.803733+0000","last_clean":"2024-02-19T21:19:39.803733+0000","last_became_active":"2024-02-19T21:17:58.690808+0000","last_became_peered":"2024-02-19T21:17:58.690808+0000","last_unstale":"2024-02-19T21:19:39.803733+0000","last_undegraded":"2024-02-19T21:19:39.803733+0000","last_fullsized":"2024-02-19T21:19:39.803733+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:18.647844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:18.647844+0000","last_clean_scrub_stamp":"2024-02-19T21:19:18.647844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":96,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.827861+0000","last_change":"2024-02-19T21:18:58.953026+0000","last_active":"2024-02-19T21:19:38.827861+0000","last_peered":"2024-02-19T21:19:38.827861+0000","last_clean":"2024-02-19T21:19:38.827861+0000","last_became_active":"2024-02-19T21:17:58.688372+0000","last_became_peered":"2024-02-19T21:17:58.688372+0000","last_unstale":"2024-02-19T21:19:38.827861+0000","last_undegraded":"2024-02-19T21:19:38.827861+0000","last_fullsized":"2024-02-19T21:19:38.827861+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:58.952929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:18:58.952929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"806'19","reported_seq":782,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967626+0000","last_change":"2024-02-19T21:19:38.323704+0000","last_active":"2024-02-19T21:19:40.967626+0000","last_peered":"2024-02-19T21:19:40.967626+0000","last_clean":"2024-02-19T21:19:40.967626+0000","last_became_active":"2024-02-19T21:18:41.545335+0000","last_became_peered":"2024-02-19T21:18:41.545335+0000","last_unstale":"2024-02-19T21:19:40.967626+0000","last_undegraded":"2024-02-19T21:19:40.967626+0000","last_fullsized":"2024-02-19T21:19:40.967626+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"806'19","last_scrub_stamp":"2024-02-19T21:19:38.323637+0000","last_deep_scrub":"806'19","last_deep_scrub_stamp":"2024-02-19T21:17:51.331019+0000","last_clean_scrub_stamp":"2024-02-19T21:19:38.323637+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":867,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.071132+0000","last_change":"2024-02-19T21:19:21.683567+0000","last_active":"2024-02-19T21:19:39.071132+0000","last_peered":"2024-02-19T21:19:39.071132+0000","last_clean":"2024-02-19T21:19:39.071132+0000","last_became_active":"2024-02-19T21:10:41.312116+0000","last_became_peered":"2024-02-19T21:10:41.312116+0000","last_unstale":"2024-02-19T21:19:39.071132+0000","last_undegraded":"2024-02-19T21:19:39.071132+0000","last_fullsized":"2024-02-19T21:19:39.071132+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:21.683502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:21.683502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":87,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.828766+0000","last_change":"2024-02-19T21:19:16.996763+0000","last_active":"2024-02-19T21:19:38.828766+0000","last_peered":"2024-02-19T21:19:38.828766+0000","last_clean":"2024-02-19T21:19:38.828766+0000","last_became_active":"2024-02-19T21:17:29.479730+0000","last_became_peered":"2024-02-19T21:17:29.479730+0000","last_unstale":"2024-02-19T21:19:38.828766+0000","last_undegraded":"2024-02-19T21:19:38.828766+0000","last_fullsized":"2024-02-19T21:19:38.828766+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:16.996698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:16.996698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":487,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967097+0000","last_change":"2024-02-19T21:19:10.266742+0000","last_active":"2024-02-19T21:19:40.967097+0000","last_peered":"2024-02-19T21:19:40.967097+0000","last_clean":"2024-02-19T21:19:40.967097+0000","last_became_active":"2024-02-19T21:18:19.172204+0000","last_became_peered":"2024-02-19T21:18:19.172204+0000","last_unstale":"2024-02-19T21:19:40.967097+0000","last_undegraded":"2024-02-19T21:19:40.967097+0000","last_fullsized":"2024-02-19T21:19:40.967097+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:10.266666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:10.266666+0000","last_clean_scrub_stamp":"2024-02-19T21:19:10.266666+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":67,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.802949+0000","last_change":"2024-02-19T21:18:52.540347+0000","last_active":"2024-02-19T21:19:39.802949+0000","last_peered":"2024-02-19T21:19:39.802949+0000","last_clean":"2024-02-19T21:19:39.802949+0000","last_became_active":"2024-02-19T21:18:03.735016+0000","last_became_peered":"2024-02-19T21:18:03.735016+0000","last_unstale":"2024-02-19T21:19:39.802949+0000","last_undegraded":"2024-02-19T21:19:39.802949+0000","last_fullsized":"2024-02-19T21:19:39.802949+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:52.540277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:18:52.540277+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"837'44","reported_seq":373,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:50.575040+0000","last_change":"2024-02-19T21:19:50.575003+0000","last_active":"2024-02-19T21:19:50.575040+0000","last_peered":"2024-02-19T21:19:50.575040+0000","last_clean":"2024-02-19T21:19:50.575040+0000","last_became_active":"2024-02-19T21:18:03.731077+0000","last_became_peered":"2024-02-19T21:18:03.731077+0000","last_unstale":"2024-02-19T21:19:50.575040+0000","last_undegraded":"2024-02-19T21:19:50.575040+0000","last_fullsized":"2024-02-19T21:19:50.575040+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"837'44","last_scrub_stamp":"2024-02-19T21:19:50.574920+0000","last_deep_scrub":"837'44","last_deep_scrub_stamp":"2024-02-19T21:18:47.751140+0000","last_clean_scrub_stamp":"2024-02-19T21:19:50.574920+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":91,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.827616+0000","last_change":"2024-02-19T21:18:36.724586+0000","last_active":"2024-02-19T21:19:38.827616+0000","last_peered":"2024-02-19T21:19:38.827616+0000","last_clean":"2024-02-19T21:19:38.827616+0000","last_became_active":"2024-02-19T21:17:58.688829+0000","last_became_peered":"2024-02-19T21:17:58.688829+0000","last_unstale":"2024-02-19T21:19:38.827616+0000","last_undegraded":"2024-02-19T21:19:38.827616+0000","last_fullsized":"2024-02-19T21:19:38.827616+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:36.724491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:18:36.724491+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":96,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.827297+0000","last_change":"2024-02-19T21:18:39.670892+0000","last_active":"2024-02-19T21:19:38.827297+0000","last_peered":"2024-02-19T21:19:38.827297+0000","last_clean":"2024-02-19T21:19:38.827297+0000","last_became_active":"2024-02-19T21:17:58.680830+0000","last_became_peered":"2024-02-19T21:17:58.680830+0000","last_unstale":"2024-02-19T21:19:38.827297+0000","last_undegraded":"2024-02-19T21:19:38.827297+0000","last_fullsized":"2024-02-19T21:19:38.827297+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:39.670792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:39.670792+0000","last_clean_scrub_stamp":"2024-02-19T21:18:39.670792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":61,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:44.465390+0000","last_change":"2024-02-19T21:19:44.465348+0000","last_active":"2024-02-19T21:19:44.465390+0000","last_peered":"2024-02-19T21:19:44.465390+0000","last_clean":"2024-02-19T21:19:44.465390+0000","last_became_active":"2024-02-19T21:19:10.738170+0000","last_became_peered":"2024-02-19T21:19:10.738170+0000","last_unstale":"2024-02-19T21:19:44.465390+0000","last_undegraded":"2024-02-19T21:19:44.465390+0000","last_fullsized":"2024-02-19T21:19:44.465390+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:44.465227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:18.857424+0000","last_clean_scrub_stamp":"2024-02-19T21:19:44.465227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":37,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969373+0000","last_change":"2024-02-19T21:19:36.619077+0000","last_active":"2024-02-19T21:19:40.969373+0000","last_peered":"2024-02-19T21:19:40.969373+0000","last_clean":"2024-02-19T21:19:40.969373+0000","last_became_active":"2024-02-19T21:19:10.740906+0000","last_became_peered":"2024-02-19T21:19:10.740906+0000","last_unstale":"2024-02-19T21:19:40.969373+0000","last_undegraded":"2024-02-19T21:19:40.969373+0000","last_fullsized":"2024-02-19T21:19:40.969373+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:36.618955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:36.618955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"662'39","reported_seq":571,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.380268+0000","last_change":"2024-02-19T21:19:30.439506+0000","last_active":"2024-02-19T21:19:39.380268+0000","last_peered":"2024-02-19T21:19:39.380268+0000","last_clean":"2024-02-19T21:19:39.380268+0000","last_became_active":"2024-02-19T21:17:29.471707+0000","last_became_peered":"2024-02-19T21:17:29.471707+0000","last_unstale":"2024-02-19T21:19:39.380268+0000","last_undegraded":"2024-02-19T21:19:39.380268+0000","last_fullsized":"2024-02-19T21:19:39.380268+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"662'39","last_scrub_stamp":"2024-02-19T21:19:30.439411+0000","last_deep_scrub":"662'39","last_deep_scrub_stamp":"2024-02-19T21:15:06.578559+0000","last_clean_scrub_stamp":"2024-02-19T21:19:30.439411+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"655'2","reported_seq":802,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968947+0000","last_change":"2024-02-19T21:19:30.711382+0000","last_active":"2024-02-19T21:19:40.968947+0000","last_peered":"2024-02-19T21:19:40.968947+0000","last_clean":"2024-02-19T21:19:40.968947+0000","last_became_active":"2024-02-19T21:19:10.741201+0000","last_became_peered":"2024-02-19T21:19:10.741201+0000","last_unstale":"2024-02-19T21:19:40.968947+0000","last_undegraded":"2024-02-19T21:19:40.968947+0000","last_fullsized":"2024-02-19T21:19:40.968947+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"655'2","last_scrub_stamp":"2024-02-19T21:19:30.711294+0000","last_deep_scrub":"655'2","last_deep_scrub_stamp":"2024-02-19T21:15:00.963594+0000","last_clean_scrub_stamp":"2024-02-19T21:19:30.711294+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":876,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.071075+0000","last_change":"2024-02-19T21:18:28.501041+0000","last_active":"2024-02-19T21:19:39.071075+0000","last_peered":"2024-02-19T21:19:39.071075+0000","last_clean":"2024-02-19T21:19:39.071075+0000","last_became_active":"2024-02-19T21:14:41.391280+0000","last_became_peered":"2024-02-19T21:14:41.391280+0000","last_unstale":"2024-02-19T21:19:39.071075+0000","last_undegraded":"2024-02-19T21:19:39.071075+0000","last_fullsized":"2024-02-19T21:19:39.071075+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:28.500946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:18:28.500946+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":87,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:46.287061+0000","last_change":"2024-02-19T21:19:46.287061+0000","last_active":"2024-02-19T21:19:46.287061+0000","last_peered":"2024-02-19T21:19:46.287061+0000","last_clean":"2024-02-19T21:19:46.287061+0000","last_became_active":"2024-02-19T21:17:29.479445+0000","last_became_peered":"2024-02-19T21:17:29.479445+0000","last_unstale":"2024-02-19T21:19:46.287061+0000","last_undegraded":"2024-02-19T21:19:46.287061+0000","last_fullsized":"2024-02-19T21:19:46.287061+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:46.286995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:19:46.286995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":443,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.378914+0000","last_change":"2024-02-19T21:19:24.531845+0000","last_active":"2024-02-19T21:19:39.378914+0000","last_peered":"2024-02-19T21:19:39.378914+0000","last_clean":"2024-02-19T21:19:39.378914+0000","last_became_active":"2024-02-19T21:10:41.302670+0000","last_became_peered":"2024-02-19T21:10:41.302670+0000","last_unstale":"2024-02-19T21:19:39.378914+0000","last_undegraded":"2024-02-19T21:19:39.378914+0000","last_fullsized":"2024-02-19T21:19:39.378914+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:24.531759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:24.531759+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"459'25","reported_seq":507,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.379438+0000","last_change":"2024-02-19T21:19:01.488178+0000","last_active":"2024-02-19T21:19:39.379438+0000","last_peered":"2024-02-19T21:19:39.379438+0000","last_clean":"2024-02-19T21:19:39.379438+0000","last_became_active":"2024-02-19T21:14:16.381198+0000","last_became_peered":"2024-02-19T21:14:16.381198+0000","last_unstale":"2024-02-19T21:19:39.379438+0000","last_undegraded":"2024-02-19T21:19:39.379438+0000","last_fullsized":"2024-02-19T21:19:39.379438+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'25","last_scrub_stamp":"2024-02-19T21:19:01.488097+0000","last_deep_scrub":"82'15","last_deep_scrub_stamp":"2024-02-19T21:03:04.086282+0000","last_clean_scrub_stamp":"2024-02-19T21:19:01.488097+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":883,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.070424+0000","last_change":"2024-02-19T21:18:41.550936+0000","last_active":"2024-02-19T21:19:39.070424+0000","last_peered":"2024-02-19T21:19:39.070424+0000","last_clean":"2024-02-19T21:19:39.070424+0000","last_became_active":"2024-02-19T21:18:03.726011+0000","last_became_peered":"2024-02-19T21:18:03.726011+0000","last_unstale":"2024-02-19T21:19:39.070424+0000","last_undegraded":"2024-02-19T21:19:39.070424+0000","last_fullsized":"2024-02-19T21:19:39.070424+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:41.550800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:18:41.550800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"816'14","reported_seq":148,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:54.800742+0000","last_change":"2024-02-19T21:19:15.986228+0000","last_active":"2024-02-19T21:19:54.800742+0000","last_peered":"2024-02-19T21:19:54.800742+0000","last_clean":"2024-02-19T21:19:54.800742+0000","last_became_active":"2024-02-19T21:18:03.744226+0000","last_became_peered":"2024-02-19T21:18:03.744226+0000","last_unstale":"2024-02-19T21:19:54.800742+0000","last_undegraded":"2024-02-19T21:19:54.800742+0000","last_fullsized":"2024-02-19T21:19:54.800742+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'14","last_scrub_stamp":"2024-02-19T21:19:15.986160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:15.986160+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":474,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.380386+0000","last_change":"2024-02-19T21:19:03.503021+0000","last_active":"2024-02-19T21:19:39.380386+0000","last_peered":"2024-02-19T21:19:39.380386+0000","last_clean":"2024-02-19T21:19:39.380386+0000","last_became_active":"2024-02-19T21:17:29.471013+0000","last_became_peered":"2024-02-19T21:17:29.471013+0000","last_unstale":"2024-02-19T21:19:39.380386+0000","last_undegraded":"2024-02-19T21:19:39.380386+0000","last_fullsized":"2024-02-19T21:19:39.380386+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:03.502937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:03.502937+0000","last_clean_scrub_stamp":"2024-02-19T21:19:03.502937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":617,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:46.174198+0000","last_change":"2024-02-19T21:19:46.174160+0000","last_active":"2024-02-19T21:19:46.174198+0000","last_peered":"2024-02-19T21:19:46.174198+0000","last_clean":"2024-02-19T21:19:46.174198+0000","last_became_active":"2024-02-19T21:18:19.172795+0000","last_became_peered":"2024-02-19T21:18:19.172795+0000","last_unstale":"2024-02-19T21:19:46.174198+0000","last_undegraded":"2024-02-19T21:19:46.174198+0000","last_fullsized":"2024-02-19T21:19:46.174198+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:46.174087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:46.174087+0000","last_clean_scrub_stamp":"2024-02-19T21:19:46.174087+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"459'27","reported_seq":146,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.071054+0000","last_change":"2024-02-19T21:18:53.374321+0000","last_active":"2024-02-19T21:19:39.071054+0000","last_peered":"2024-02-19T21:19:39.071054+0000","last_clean":"2024-02-19T21:19:39.071054+0000","last_became_active":"2024-02-19T21:16:37.630325+0000","last_became_peered":"2024-02-19T21:16:37.630325+0000","last_unstale":"2024-02-19T21:19:39.071054+0000","last_undegraded":"2024-02-19T21:19:39.071054+0000","last_fullsized":"2024-02-19T21:19:39.071054+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'27","last_scrub_stamp":"2024-02-19T21:18:53.374257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:18:53.374257+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":934,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.070321+0000","last_change":"2024-02-19T21:19:25.660038+0000","last_active":"2024-02-19T21:19:39.070321+0000","last_peered":"2024-02-19T21:19:39.070321+0000","last_clean":"2024-02-19T21:19:39.070321+0000","last_became_active":"2024-02-19T21:18:41.545465+0000","last_became_peered":"2024-02-19T21:18:41.545465+0000","last_unstale":"2024-02-19T21:19:39.070321+0000","last_undegraded":"2024-02-19T21:19:39.070321+0000","last_fullsized":"2024-02-19T21:19:39.070321+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:25.659979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:25.659979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"811'15","reported_seq":620,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:54.800958+0000","last_change":"2024-02-19T21:18:53.555180+0000","last_active":"2024-02-19T21:19:54.800958+0000","last_peered":"2024-02-19T21:19:54.800958+0000","last_clean":"2024-02-19T21:19:54.800958+0000","last_became_active":"2024-02-19T21:17:58.675780+0000","last_became_peered":"2024-02-19T21:17:58.675780+0000","last_unstale":"2024-02-19T21:19:54.800958+0000","last_undegraded":"2024-02-19T21:19:54.800958+0000","last_fullsized":"2024-02-19T21:19:54.800958+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"811'15","last_scrub_stamp":"2024-02-19T21:18:53.555067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:18:53.555067+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":63,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.804029+0000","last_change":"2024-02-19T21:19:34.529292+0000","last_active":"2024-02-19T21:19:39.804029+0000","last_peered":"2024-02-19T21:19:39.804029+0000","last_clean":"2024-02-19T21:19:39.804029+0000","last_became_active":"2024-02-19T21:17:58.687157+0000","last_became_peered":"2024-02-19T21:17:58.687157+0000","last_unstale":"2024-02-19T21:19:39.804029+0000","last_undegraded":"2024-02-19T21:19:39.804029+0000","last_fullsized":"2024-02-19T21:19:39.804029+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:34.529209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:34.529209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":58,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.803607+0000","last_change":"2024-02-19T21:19:10.573539+0000","last_active":"2024-02-19T21:19:39.803607+0000","last_peered":"2024-02-19T21:19:39.803607+0000","last_clean":"2024-02-19T21:19:39.803607+0000","last_became_active":"2024-02-19T21:17:58.689118+0000","last_became_peered":"2024-02-19T21:17:58.689118+0000","last_unstale":"2024-02-19T21:19:39.803607+0000","last_undegraded":"2024-02-19T21:19:39.803607+0000","last_fullsized":"2024-02-19T21:19:39.803607+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:10.573465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:10.573465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"459'19","reported_seq":511,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:49.501449+0000","last_change":"2024-02-19T21:19:49.501449+0000","last_active":"2024-02-19T21:19:49.501449+0000","last_peered":"2024-02-19T21:19:49.501449+0000","last_clean":"2024-02-19T21:19:49.501449+0000","last_became_active":"2024-02-19T21:14:16.382074+0000","last_became_peered":"2024-02-19T21:14:16.382074+0000","last_unstale":"2024-02-19T21:19:49.501449+0000","last_undegraded":"2024-02-19T21:19:49.501449+0000","last_fullsized":"2024-02-19T21:19:49.501449+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:19:49.501373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:49.501373+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":774,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967708+0000","last_change":"2024-02-19T21:19:31.694138+0000","last_active":"2024-02-19T21:19:40.967708+0000","last_peered":"2024-02-19T21:19:40.967708+0000","last_clean":"2024-02-19T21:19:40.967708+0000","last_became_active":"2024-02-19T21:19:10.734034+0000","last_became_peered":"2024-02-19T21:19:10.734034+0000","last_unstale":"2024-02-19T21:19:40.967708+0000","last_undegraded":"2024-02-19T21:19:40.967708+0000","last_fullsized":"2024-02-19T21:19:40.967708+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:31.694046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:31.694046+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":58,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.804158+0000","last_change":"2024-02-19T21:18:50.589644+0000","last_active":"2024-02-19T21:19:39.804158+0000","last_peered":"2024-02-19T21:19:39.804158+0000","last_clean":"2024-02-19T21:19:39.804158+0000","last_became_active":"2024-02-19T21:17:58.688047+0000","last_became_peered":"2024-02-19T21:17:58.688047+0000","last_unstale":"2024-02-19T21:19:39.804158+0000","last_undegraded":"2024-02-19T21:19:39.804158+0000","last_fullsized":"2024-02-19T21:19:39.804158+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:50.589569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:18:50.589569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":630,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:53.496726+0000","last_change":"2024-02-19T21:19:53.496701+0000","last_active":"2024-02-19T21:19:53.496726+0000","last_peered":"2024-02-19T21:19:53.496726+0000","last_clean":"2024-02-19T21:19:53.496726+0000","last_became_active":"2024-02-19T21:17:58.673286+0000","last_became_peered":"2024-02-19T21:17:58.673286+0000","last_unstale":"2024-02-19T21:19:53.496726+0000","last_undegraded":"2024-02-19T21:19:53.496726+0000","last_fullsized":"2024-02-19T21:19:53.496726+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:53.496636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:43.349529+0000","last_clean_scrub_stamp":"2024-02-19T21:19:53.496636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851104+0000","last_change":"2024-02-19T21:19:28.212564+0000","last_active":"2024-02-19T21:19:40.851104+0000","last_peered":"2024-02-19T21:19:40.851104+0000","last_clean":"2024-02-19T21:19:40.851104+0000","last_became_active":"2024-02-19T21:17:58.675523+0000","last_became_peered":"2024-02-19T21:17:58.675523+0000","last_unstale":"2024-02-19T21:19:40.851104+0000","last_undegraded":"2024-02-19T21:19:40.851104+0000","last_fullsized":"2024-02-19T21:19:40.851104+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:28.212491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:28.212491+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"434'12","reported_seq":757,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:59.801851+0000","last_change":"2024-02-19T21:19:03.247504+0000","last_active":"2024-02-19T21:19:59.801851+0000","last_peered":"2024-02-19T21:19:59.801851+0000","last_clean":"2024-02-19T21:19:59.801851+0000","last_became_active":"2024-02-19T21:10:41.297897+0000","last_became_peered":"2024-02-19T21:10:41.297897+0000","last_unstale":"2024-02-19T21:19:59.801851+0000","last_undegraded":"2024-02-19T21:19:59.801851+0000","last_fullsized":"2024-02-19T21:19:59.801851+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'12","last_scrub_stamp":"2024-02-19T21:19:03.247435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:03.247435+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"459'28","reported_seq":580,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.379110+0000","last_change":"2024-02-19T21:19:04.552679+0000","last_active":"2024-02-19T21:19:39.379110+0000","last_peered":"2024-02-19T21:19:39.379110+0000","last_clean":"2024-02-19T21:19:39.379110+0000","last_became_active":"2024-02-19T21:17:58.672547+0000","last_became_peered":"2024-02-19T21:17:58.672547+0000","last_unstale":"2024-02-19T21:19:39.379110+0000","last_undegraded":"2024-02-19T21:19:39.379110+0000","last_fullsized":"2024-02-19T21:19:39.379110+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'28","last_scrub_stamp":"2024-02-19T21:19:04.552600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:04.552600+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":297,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968652+0000","last_change":"2024-02-19T21:19:17.426250+0000","last_active":"2024-02-19T21:19:40.968652+0000","last_peered":"2024-02-19T21:19:40.968652+0000","last_clean":"2024-02-19T21:19:40.968652+0000","last_became_active":"2024-02-19T21:18:41.545753+0000","last_became_peered":"2024-02-19T21:18:41.545753+0000","last_unstale":"2024-02-19T21:19:40.968652+0000","last_undegraded":"2024-02-19T21:19:40.968652+0000","last_fullsized":"2024-02-19T21:19:40.968652+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:17.426176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:50.164434+0000","last_clean_scrub_stamp":"2024-02-19T21:19:17.426176+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":726,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967797+0000","last_change":"2024-02-19T21:19:10.742006+0000","last_active":"2024-02-19T21:19:40.967797+0000","last_peered":"2024-02-19T21:19:40.967797+0000","last_clean":"2024-02-19T21:19:40.967797+0000","last_became_active":"2024-02-19T21:19:10.741672+0000","last_became_peered":"2024-02-19T21:19:10.741672+0000","last_unstale":"2024-02-19T21:19:40.967797+0000","last_undegraded":"2024-02-19T21:19:40.967797+0000","last_fullsized":"2024-02-19T21:19:40.967797+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:39.709227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:18:39.709227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":678,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968638+0000","last_change":"2024-02-19T21:18:29.317105+0000","last_active":"2024-02-19T21:19:40.968638+0000","last_peered":"2024-02-19T21:19:40.968638+0000","last_clean":"2024-02-19T21:19:40.968638+0000","last_became_active":"2024-02-19T21:18:19.177273+0000","last_became_peered":"2024-02-19T21:18:19.177273+0000","last_unstale":"2024-02-19T21:19:40.968638+0000","last_undegraded":"2024-02-19T21:19:40.968638+0000","last_fullsized":"2024-02-19T21:19:40.968638+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:29.317039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:13:11.004006+0000","last_clean_scrub_stamp":"2024-02-19T21:18:29.317039+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":461,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:46.483834+0000","last_change":"2024-02-19T21:19:46.483834+0000","last_active":"2024-02-19T21:19:46.483834+0000","last_peered":"2024-02-19T21:19:46.483834+0000","last_clean":"2024-02-19T21:19:46.483834+0000","last_became_active":"2024-02-19T21:18:03.726906+0000","last_became_peered":"2024-02-19T21:18:03.726906+0000","last_unstale":"2024-02-19T21:19:46.483834+0000","last_undegraded":"2024-02-19T21:19:46.483834+0000","last_fullsized":"2024-02-19T21:19:46.483834+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:46.483749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:46.483749+0000","last_clean_scrub_stamp":"2024-02-19T21:19:46.483749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"459'30","reported_seq":726,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852611+0000","last_change":"2024-02-19T21:19:11.193104+0000","last_active":"2024-02-19T21:19:40.852611+0000","last_peered":"2024-02-19T21:19:40.852611+0000","last_clean":"2024-02-19T21:19:40.852611+0000","last_became_active":"2024-02-19T21:18:03.741555+0000","last_became_peered":"2024-02-19T21:18:03.741555+0000","last_unstale":"2024-02-19T21:19:40.852611+0000","last_undegraded":"2024-02-19T21:19:40.852611+0000","last_fullsized":"2024-02-19T21:19:40.852611+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'30","last_scrub_stamp":"2024-02-19T21:19:11.193034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:11.193034+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":893,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:43.701552+0000","last_change":"2024-02-19T21:19:43.701552+0000","last_active":"2024-02-19T21:19:43.701552+0000","last_peered":"2024-02-19T21:19:43.701552+0000","last_clean":"2024-02-19T21:19:43.701552+0000","last_became_active":"2024-02-19T21:14:16.379640+0000","last_became_peered":"2024-02-19T21:14:16.379640+0000","last_unstale":"2024-02-19T21:19:43.701552+0000","last_undegraded":"2024-02-19T21:19:43.701552+0000","last_fullsized":"2024-02-19T21:19:43.701552+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:43.701488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:43.701488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":461,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.348406+0000","last_change":"2024-02-19T21:19:40.348360+0000","last_active":"2024-02-19T21:19:40.348406+0000","last_peered":"2024-02-19T21:19:40.348406+0000","last_clean":"2024-02-19T21:19:40.348406+0000","last_became_active":"2024-02-19T21:17:29.470869+0000","last_became_peered":"2024-02-19T21:17:29.470869+0000","last_unstale":"2024-02-19T21:19:40.348406+0000","last_undegraded":"2024-02-19T21:19:40.348406+0000","last_fullsized":"2024-02-19T21:19:40.348406+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:40.348274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:40.348274+0000","last_clean_scrub_stamp":"2024-02-19T21:19:40.348274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":427,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:55.260831+0000","last_change":"2024-02-19T21:19:55.260797+0000","last_active":"2024-02-19T21:19:55.260831+0000","last_peered":"2024-02-19T21:19:55.260831+0000","last_clean":"2024-02-19T21:19:55.260831+0000","last_became_active":"2024-02-19T21:18:19.171295+0000","last_became_peered":"2024-02-19T21:18:19.171295+0000","last_unstale":"2024-02-19T21:19:55.260831+0000","last_undegraded":"2024-02-19T21:19:55.260831+0000","last_fullsized":"2024-02-19T21:19:55.260831+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:55.260704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:21.385247+0000","last_clean_scrub_stamp":"2024-02-19T21:19:55.260704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":297,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:41.536988+0000","last_change":"2024-02-19T21:19:41.536960+0000","last_active":"2024-02-19T21:19:41.536988+0000","last_peered":"2024-02-19T21:19:41.536988+0000","last_clean":"2024-02-19T21:19:41.536988+0000","last_became_active":"2024-02-19T21:17:58.681256+0000","last_became_peered":"2024-02-19T21:17:58.681256+0000","last_unstale":"2024-02-19T21:19:41.536988+0000","last_undegraded":"2024-02-19T21:19:41.536988+0000","last_fullsized":"2024-02-19T21:19:41.536988+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:41.536880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:49.141780+0000","last_clean_scrub_stamp":"2024-02-19T21:19:41.536880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"459'29","reported_seq":576,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:58.284209+0000","last_change":"2024-02-19T21:19:58.284186+0000","last_active":"2024-02-19T21:19:58.284209+0000","last_peered":"2024-02-19T21:19:58.284209+0000","last_clean":"2024-02-19T21:19:58.284209+0000","last_became_active":"2024-02-19T21:18:19.172356+0000","last_became_peered":"2024-02-19T21:18:19.172356+0000","last_unstale":"2024-02-19T21:19:58.284209+0000","last_undegraded":"2024-02-19T21:19:58.284209+0000","last_fullsized":"2024-02-19T21:19:58.284209+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'29","last_scrub_stamp":"2024-02-19T21:19:58.284118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:58.284118+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":34,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:54.312479+0000","last_change":"2024-02-19T21:19:54.312479+0000","last_active":"2024-02-19T21:19:54.312479+0000","last_peered":"2024-02-19T21:19:54.312479+0000","last_clean":"2024-02-19T21:19:54.312479+0000","last_became_active":"2024-02-19T21:18:41.546175+0000","last_became_peered":"2024-02-19T21:18:41.546175+0000","last_unstale":"2024-02-19T21:19:54.312479+0000","last_undegraded":"2024-02-19T21:19:54.312479+0000","last_fullsized":"2024-02-19T21:19:54.312479+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:54.312403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:54.312403+0000","last_clean_scrub_stamp":"2024-02-19T21:19:54.312403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":865,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.070417+0000","last_change":"2024-02-19T21:19:36.729428+0000","last_active":"2024-02-19T21:19:39.070417+0000","last_peered":"2024-02-19T21:19:39.070417+0000","last_clean":"2024-02-19T21:19:39.070417+0000","last_became_active":"2024-02-19T21:17:29.516758+0000","last_became_peered":"2024-02-19T21:17:29.516758+0000","last_unstale":"2024-02-19T21:19:39.070417+0000","last_undegraded":"2024-02-19T21:19:39.070417+0000","last_fullsized":"2024-02-19T21:19:39.070417+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:36.729358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:19:36.729358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":82,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.829659+0000","last_change":"2024-02-19T21:18:32.745789+0000","last_active":"2024-02-19T21:19:38.829659+0000","last_peered":"2024-02-19T21:19:38.829659+0000","last_clean":"2024-02-19T21:19:38.829659+0000","last_became_active":"2024-02-19T21:17:29.522761+0000","last_became_peered":"2024-02-19T21:17:29.522761+0000","last_unstale":"2024-02-19T21:19:38.829659+0000","last_undegraded":"2024-02-19T21:19:38.829659+0000","last_fullsized":"2024-02-19T21:19:38.829659+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:32.745715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:32.745715+0000","last_clean_scrub_stamp":"2024-02-19T21:18:32.745715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":58,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.804234+0000","last_change":"2024-02-19T21:19:12.576968+0000","last_active":"2024-02-19T21:19:39.804234+0000","last_peered":"2024-02-19T21:19:39.804234+0000","last_clean":"2024-02-19T21:19:39.804234+0000","last_became_active":"2024-02-19T21:17:58.685999+0000","last_became_peered":"2024-02-19T21:17:58.685999+0000","last_unstale":"2024-02-19T21:19:39.804234+0000","last_undegraded":"2024-02-19T21:19:39.804234+0000","last_fullsized":"2024-02-19T21:19:39.804234+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:12.576894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:12.576894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"459'24","reported_seq":555,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:56.238618+0000","last_change":"2024-02-19T21:19:56.238578+0000","last_active":"2024-02-19T21:19:56.238618+0000","last_peered":"2024-02-19T21:19:56.238618+0000","last_clean":"2024-02-19T21:19:56.238618+0000","last_became_active":"2024-02-19T21:18:19.170850+0000","last_became_peered":"2024-02-19T21:18:19.170850+0000","last_unstale":"2024-02-19T21:19:56.238618+0000","last_undegraded":"2024-02-19T21:19:56.238618+0000","last_fullsized":"2024-02-19T21:19:56.238618+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'24","last_scrub_stamp":"2024-02-19T21:19:56.238476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:56.238476+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"397'2","reported_seq":544,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967532+0000","last_change":"2024-02-19T21:19:19.550446+0000","last_active":"2024-02-19T21:19:40.967532+0000","last_peered":"2024-02-19T21:19:40.967532+0000","last_clean":"2024-02-19T21:19:40.967532+0000","last_became_active":"2024-02-19T21:19:10.733472+0000","last_became_peered":"2024-02-19T21:19:10.733472+0000","last_unstale":"2024-02-19T21:19:40.967532+0000","last_undegraded":"2024-02-19T21:19:40.967532+0000","last_fullsized":"2024-02-19T21:19:40.967532+0000","mapping_epoch":848,"log_start":"58'1","ondisk_log_start":"58'1","created":57,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:19:19.550363+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:19:19.550363+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":277,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.966930+0000","last_change":"2024-02-19T21:19:24.347774+0000","last_active":"2024-02-19T21:19:40.966930+0000","last_peered":"2024-02-19T21:19:40.966930+0000","last_clean":"2024-02-19T21:19:40.966930+0000","last_became_active":"2024-02-19T21:18:19.172569+0000","last_became_peered":"2024-02-19T21:18:19.172569+0000","last_unstale":"2024-02-19T21:19:40.966930+0000","last_undegraded":"2024-02-19T21:19:40.966930+0000","last_fullsized":"2024-02-19T21:19:40.966930+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:24.347707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:19:24.347707+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":858,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.071115+0000","last_change":"2024-02-19T21:19:18.748401+0000","last_active":"2024-02-19T21:19:39.071115+0000","last_peered":"2024-02-19T21:19:39.071115+0000","last_clean":"2024-02-19T21:19:39.071115+0000","last_became_active":"2024-02-19T21:17:58.691420+0000","last_became_peered":"2024-02-19T21:17:58.691420+0000","last_unstale":"2024-02-19T21:19:39.071115+0000","last_undegraded":"2024-02-19T21:19:39.071115+0000","last_fullsized":"2024-02-19T21:19:39.071115+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:18.748338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:18.748338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":279,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969761+0000","last_change":"2024-02-19T21:19:14.434878+0000","last_active":"2024-02-19T21:19:40.969761+0000","last_peered":"2024-02-19T21:19:40.969761+0000","last_clean":"2024-02-19T21:19:40.969761+0000","last_became_active":"2024-02-19T21:17:58.682748+0000","last_became_peered":"2024-02-19T21:17:58.682748+0000","last_unstale":"2024-02-19T21:19:40.969761+0000","last_undegraded":"2024-02-19T21:19:40.969761+0000","last_fullsized":"2024-02-19T21:19:40.969761+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:14.434797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:14.434797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":70,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968955+0000","last_change":"2024-02-19T21:19:10.741533+0000","last_active":"2024-02-19T21:19:40.968955+0000","last_peered":"2024-02-19T21:19:40.968955+0000","last_clean":"2024-02-19T21:19:40.968955+0000","last_became_active":"2024-02-19T21:19:10.740867+0000","last_became_peered":"2024-02-19T21:19:10.740867+0000","last_unstale":"2024-02-19T21:19:40.968955+0000","last_undegraded":"2024-02-19T21:19:40.968955+0000","last_fullsized":"2024-02-19T21:19:40.968955+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:05.942411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:05.942411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"459'26","reported_seq":318,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968331+0000","last_change":"2024-02-19T21:19:09.487024+0000","last_active":"2024-02-19T21:19:40.968331+0000","last_peered":"2024-02-19T21:19:40.968331+0000","last_clean":"2024-02-19T21:19:40.968331+0000","last_became_active":"2024-02-19T21:18:03.727966+0000","last_became_peered":"2024-02-19T21:18:03.727966+0000","last_unstale":"2024-02-19T21:19:40.968331+0000","last_undegraded":"2024-02-19T21:19:40.968331+0000","last_fullsized":"2024-02-19T21:19:40.968331+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'26","last_scrub_stamp":"2024-02-19T21:19:09.486913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:09.486913+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":662,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967796+0000","last_change":"2024-02-19T21:19:25.629678+0000","last_active":"2024-02-19T21:19:40.967796+0000","last_peered":"2024-02-19T21:19:40.967796+0000","last_clean":"2024-02-19T21:19:40.967796+0000","last_became_active":"2024-02-19T21:19:10.742230+0000","last_became_peered":"2024-02-19T21:19:10.742230+0000","last_unstale":"2024-02-19T21:19:40.967796+0000","last_undegraded":"2024-02-19T21:19:40.967796+0000","last_fullsized":"2024-02-19T21:19:40.967796+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:25.629584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:19:25.629584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":67,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.802916+0000","last_change":"2024-02-19T21:18:34.444400+0000","last_active":"2024-02-19T21:19:39.802916+0000","last_peered":"2024-02-19T21:19:39.802916+0000","last_clean":"2024-02-19T21:19:39.802916+0000","last_became_active":"2024-02-19T21:18:03.735603+0000","last_became_peered":"2024-02-19T21:18:03.735603+0000","last_unstale":"2024-02-19T21:19:39.802916+0000","last_undegraded":"2024-02-19T21:19:39.802916+0000","last_fullsized":"2024-02-19T21:19:39.802916+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:34.444317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:34.444317+0000","last_clean_scrub_stamp":"2024-02-19T21:18:34.444317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851181+0000","last_change":"2024-02-19T21:18:42.341345+0000","last_active":"2024-02-19T21:19:40.851181+0000","last_peered":"2024-02-19T21:19:40.851181+0000","last_clean":"2024-02-19T21:19:40.851181+0000","last_became_active":"2024-02-19T21:17:58.675488+0000","last_became_peered":"2024-02-19T21:17:58.675488+0000","last_unstale":"2024-02-19T21:19:40.851181+0000","last_undegraded":"2024-02-19T21:19:40.851181+0000","last_fullsized":"2024-02-19T21:19:40.851181+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:42.341279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:18:42.341279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":63,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:52.598627+0000","last_change":"2024-02-19T21:19:52.598627+0000","last_active":"2024-02-19T21:19:52.598627+0000","last_peered":"2024-02-19T21:19:52.598627+0000","last_clean":"2024-02-19T21:19:52.598627+0000","last_became_active":"2024-02-19T21:17:58.680832+0000","last_became_peered":"2024-02-19T21:17:58.680832+0000","last_unstale":"2024-02-19T21:19:52.598627+0000","last_undegraded":"2024-02-19T21:19:52.598627+0000","last_fullsized":"2024-02-19T21:19:52.598627+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:52.598556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:52.598556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":58,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.803310+0000","last_change":"2024-02-19T21:18:45.539887+0000","last_active":"2024-02-19T21:19:39.803310+0000","last_peered":"2024-02-19T21:19:39.803310+0000","last_clean":"2024-02-19T21:19:39.803310+0000","last_became_active":"2024-02-19T21:17:58.684312+0000","last_became_peered":"2024-02-19T21:17:58.684312+0000","last_unstale":"2024-02-19T21:19:39.803310+0000","last_undegraded":"2024-02-19T21:19:39.803310+0000","last_fullsized":"2024-02-19T21:19:39.803310+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:45.539763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_clean_scrub_stamp":"2024-02-19T21:18:45.539763+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":105,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.828444+0000","last_change":"2024-02-19T21:18:47.716644+0000","last_active":"2024-02-19T21:19:38.828444+0000","last_peered":"2024-02-19T21:19:38.828444+0000","last_clean":"2024-02-19T21:19:38.828444+0000","last_became_active":"2024-02-19T21:18:03.742383+0000","last_became_peered":"2024-02-19T21:18:03.742383+0000","last_unstale":"2024-02-19T21:19:38.828444+0000","last_undegraded":"2024-02-19T21:19:38.828444+0000","last_fullsized":"2024-02-19T21:19:38.828444+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:47.716570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:18:47.716570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"459'22","reported_seq":205,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:52.687446+0000","last_change":"2024-02-19T21:19:52.687446+0000","last_active":"2024-02-19T21:19:52.687446+0000","last_peered":"2024-02-19T21:19:52.687446+0000","last_clean":"2024-02-19T21:19:52.687446+0000","last_became_active":"2024-02-19T21:17:58.686399+0000","last_became_peered":"2024-02-19T21:17:58.686399+0000","last_unstale":"2024-02-19T21:19:52.687446+0000","last_undegraded":"2024-02-19T21:19:52.687446+0000","last_fullsized":"2024-02-19T21:19:52.687446+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'22","last_scrub_stamp":"2024-02-19T21:19:52.687378+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:19:52.687378+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":87,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.829523+0000","last_change":"2024-02-19T21:19:22.938201+0000","last_active":"2024-02-19T21:19:38.829523+0000","last_peered":"2024-02-19T21:19:38.829523+0000","last_clean":"2024-02-19T21:19:38.829523+0000","last_became_active":"2024-02-19T21:17:29.525497+0000","last_became_peered":"2024-02-19T21:17:29.525497+0000","last_unstale":"2024-02-19T21:19:38.829523+0000","last_undegraded":"2024-02-19T21:19:38.829523+0000","last_fullsized":"2024-02-19T21:19:38.829523+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:22.938133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:22.938133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":1998,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:58.768070+0000","last_change":"2024-02-19T21:19:30.754451+0000","last_active":"2024-02-19T21:19:58.768070+0000","last_peered":"2024-02-19T21:19:58.768070+0000","last_clean":"2024-02-19T21:19:58.768070+0000","last_became_active":"2024-02-19T21:17:29.529134+0000","last_became_peered":"2024-02-19T21:17:29.529134+0000","last_unstale":"2024-02-19T21:19:58.768070+0000","last_undegraded":"2024-02-19T21:19:58.768070+0000","last_fullsized":"2024-02-19T21:19:58.768070+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:19:30.754381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:30.754381+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1107,"num_read_kb":1102,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":736,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969077+0000","last_change":"2024-02-19T21:19:10.742903+0000","last_active":"2024-02-19T21:19:40.969077+0000","last_peered":"2024-02-19T21:19:40.969077+0000","last_clean":"2024-02-19T21:19:40.969077+0000","last_became_active":"2024-02-19T21:19:10.742195+0000","last_became_peered":"2024-02-19T21:19:10.742195+0000","last_unstale":"2024-02-19T21:19:40.969077+0000","last_undegraded":"2024-02-19T21:19:40.969077+0000","last_fullsized":"2024-02-19T21:19:40.969077+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:53.675138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:53.675138+0000","last_clean_scrub_stamp":"2024-02-19T21:18:53.675138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":819,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.070768+0000","last_change":"2024-02-19T21:19:26.677237+0000","last_active":"2024-02-19T21:19:39.070768+0000","last_peered":"2024-02-19T21:19:39.070768+0000","last_clean":"2024-02-19T21:19:39.070768+0000","last_became_active":"2024-02-19T21:18:03.734552+0000","last_became_peered":"2024-02-19T21:18:03.734552+0000","last_unstale":"2024-02-19T21:19:39.070768+0000","last_undegraded":"2024-02-19T21:19:39.070768+0000","last_fullsized":"2024-02-19T21:19:39.070768+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:26.677161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:26.677161+0000","last_clean_scrub_stamp":"2024-02-19T21:19:26.677161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":524,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968697+0000","last_change":"2024-02-19T21:19:31.273086+0000","last_active":"2024-02-19T21:19:40.968697+0000","last_peered":"2024-02-19T21:19:40.968697+0000","last_clean":"2024-02-19T21:19:40.968697+0000","last_became_active":"2024-02-19T21:18:19.171202+0000","last_became_peered":"2024-02-19T21:18:19.171202+0000","last_unstale":"2024-02-19T21:19:40.968697+0000","last_undegraded":"2024-02-19T21:19:40.968697+0000","last_fullsized":"2024-02-19T21:19:40.968697+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:31.272992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:19:31.272992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"834'54","reported_seq":1249,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:54.800712+0000","last_change":"2024-02-19T21:18:42.503258+0000","last_active":"2024-02-19T21:19:54.800712+0000","last_peered":"2024-02-19T21:19:54.800712+0000","last_clean":"2024-02-19T21:19:54.800712+0000","last_became_active":"2024-02-19T21:17:29.529715+0000","last_became_peered":"2024-02-19T21:17:29.529715+0000","last_unstale":"2024-02-19T21:19:54.800712+0000","last_undegraded":"2024-02-19T21:19:54.800712+0000","last_fullsized":"2024-02-19T21:19:54.800712+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"834'54","last_scrub_stamp":"2024-02-19T21:18:42.503149+0000","last_deep_scrub":"834'54","last_deep_scrub_stamp":"2024-02-19T21:18:42.503149+0000","last_clean_scrub_stamp":"2024-02-19T21:18:42.503149+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":58,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.803601+0000","last_change":"2024-02-19T21:18:51.564128+0000","last_active":"2024-02-19T21:19:39.803601+0000","last_peered":"2024-02-19T21:19:39.803601+0000","last_clean":"2024-02-19T21:19:39.803601+0000","last_became_active":"2024-02-19T21:17:58.689167+0000","last_became_peered":"2024-02-19T21:17:58.689167+0000","last_unstale":"2024-02-19T21:19:39.803601+0000","last_undegraded":"2024-02-19T21:19:39.803601+0000","last_fullsized":"2024-02-19T21:19:39.803601+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:51.564022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:18:51.564022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:46.548959+0000","last_change":"2024-02-19T21:19:46.548923+0000","last_active":"2024-02-19T21:19:46.548959+0000","last_peered":"2024-02-19T21:19:46.548959+0000","last_clean":"2024-02-19T21:19:46.548959+0000","last_became_active":"2024-02-19T21:18:03.731371+0000","last_became_peered":"2024-02-19T21:18:03.731371+0000","last_unstale":"2024-02-19T21:19:46.548959+0000","last_undegraded":"2024-02-19T21:19:46.548959+0000","last_fullsized":"2024-02-19T21:19:46.548959+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:46.548853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:46.548853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":91,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.827083+0000","last_change":"2024-02-19T21:18:29.713301+0000","last_active":"2024-02-19T21:19:38.827083+0000","last_peered":"2024-02-19T21:19:38.827083+0000","last_clean":"2024-02-19T21:19:38.827083+0000","last_became_active":"2024-02-19T21:17:58.688197+0000","last_became_peered":"2024-02-19T21:17:58.688197+0000","last_unstale":"2024-02-19T21:19:38.827083+0000","last_undegraded":"2024-02-19T21:19:38.827083+0000","last_fullsized":"2024-02-19T21:19:38.827083+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:29.713207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:18:29.713207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"824'17","reported_seq":103,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:59.801837+0000","last_change":"2024-02-19T21:19:15.374737+0000","last_active":"2024-02-19T21:19:59.801837+0000","last_peered":"2024-02-19T21:19:59.801837+0000","last_clean":"2024-02-19T21:19:59.801837+0000","last_became_active":"2024-02-19T21:18:19.165036+0000","last_became_peered":"2024-02-19T21:18:19.165036+0000","last_unstale":"2024-02-19T21:19:59.801837+0000","last_undegraded":"2024-02-19T21:19:59.801837+0000","last_fullsized":"2024-02-19T21:19:59.801837+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"824'17","last_scrub_stamp":"2024-02-19T21:19:15.374622+0000","last_deep_scrub":"471'10","last_deep_scrub_stamp":"2024-02-19T21:11:41.775418+0000","last_clean_scrub_stamp":"2024-02-19T21:19:15.374622+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"459'20","reported_seq":46,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.828585+0000","last_change":"2024-02-19T21:19:20.989153+0000","last_active":"2024-02-19T21:19:38.828585+0000","last_peered":"2024-02-19T21:19:38.828585+0000","last_clean":"2024-02-19T21:19:38.828585+0000","last_became_active":"2024-02-19T21:18:41.543526+0000","last_became_peered":"2024-02-19T21:18:41.543526+0000","last_unstale":"2024-02-19T21:19:38.828585+0000","last_undegraded":"2024-02-19T21:19:38.828585+0000","last_fullsized":"2024-02-19T21:19:38.828585+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:19:20.989044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:20.989044+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":930,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:53.638179+0000","last_change":"2024-02-19T21:19:53.638179+0000","last_active":"2024-02-19T21:19:53.638179+0000","last_peered":"2024-02-19T21:19:53.638179+0000","last_clean":"2024-02-19T21:19:53.638179+0000","last_became_active":"2024-02-19T21:18:03.734529+0000","last_became_peered":"2024-02-19T21:18:03.734529+0000","last_unstale":"2024-02-19T21:19:53.638179+0000","last_undegraded":"2024-02-19T21:19:53.638179+0000","last_fullsized":"2024-02-19T21:19:53.638179+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:53.638068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:19:53.638068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":590,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967159+0000","last_change":"2024-02-19T21:18:47.359382+0000","last_active":"2024-02-19T21:19:40.967159+0000","last_peered":"2024-02-19T21:19:40.967159+0000","last_clean":"2024-02-19T21:19:40.967159+0000","last_became_active":"2024-02-19T21:18:19.169773+0000","last_became_peered":"2024-02-19T21:18:19.169773+0000","last_unstale":"2024-02-19T21:19:40.967159+0000","last_undegraded":"2024-02-19T21:19:40.967159+0000","last_fullsized":"2024-02-19T21:19:40.967159+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:47.359312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:18:47.359312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":598,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851723+0000","last_change":"2024-02-19T21:19:38.307780+0000","last_active":"2024-02-19T21:19:40.851723+0000","last_peered":"2024-02-19T21:19:40.851723+0000","last_clean":"2024-02-19T21:19:40.851723+0000","last_became_active":"2024-02-19T21:14:16.383904+0000","last_became_peered":"2024-02-19T21:14:16.383904+0000","last_unstale":"2024-02-19T21:19:40.851723+0000","last_undegraded":"2024-02-19T21:19:40.851723+0000","last_fullsized":"2024-02-19T21:19:40.851723+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:38.307710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:08.313435+0000","last_clean_scrub_stamp":"2024-02-19T21:19:38.307710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":87,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.826948+0000","last_change":"2024-02-19T21:19:18.991408+0000","last_active":"2024-02-19T21:19:38.826948+0000","last_peered":"2024-02-19T21:19:38.826948+0000","last_clean":"2024-02-19T21:19:38.826948+0000","last_became_active":"2024-02-19T21:17:29.526877+0000","last_became_peered":"2024-02-19T21:17:29.526877+0000","last_unstale":"2024-02-19T21:19:38.826948+0000","last_undegraded":"2024-02-19T21:19:38.826948+0000","last_fullsized":"2024-02-19T21:19:38.826948+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:18.991321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:19:18.991321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"459'23","reported_seq":464,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.378848+0000","last_change":"2024-02-19T21:19:35.387240+0000","last_active":"2024-02-19T21:19:39.378848+0000","last_peered":"2024-02-19T21:19:39.378848+0000","last_clean":"2024-02-19T21:19:39.378848+0000","last_became_active":"2024-02-19T21:10:41.287134+0000","last_became_peered":"2024-02-19T21:10:41.287134+0000","last_unstale":"2024-02-19T21:19:39.378848+0000","last_undegraded":"2024-02-19T21:19:39.378848+0000","last_fullsized":"2024-02-19T21:19:39.378848+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'23","last_scrub_stamp":"2024-02-19T21:19:35.387152+0000","last_deep_scrub":"420'17","last_deep_scrub_stamp":"2024-02-19T21:10:19.722303+0000","last_clean_scrub_stamp":"2024-02-19T21:19:35.387152+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":524,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.379753+0000","last_change":"2024-02-19T21:19:21.517612+0000","last_active":"2024-02-19T21:19:39.379753+0000","last_peered":"2024-02-19T21:19:39.379753+0000","last_clean":"2024-02-19T21:19:39.379753+0000","last_became_active":"2024-02-19T21:18:03.726841+0000","last_became_peered":"2024-02-19T21:18:03.726841+0000","last_unstale":"2024-02-19T21:19:39.379753+0000","last_undegraded":"2024-02-19T21:19:39.379753+0000","last_fullsized":"2024-02-19T21:19:39.379753+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:19:21.517494+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:19:21.517494+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":603,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:43.474941+0000","last_change":"2024-02-19T21:19:43.474904+0000","last_active":"2024-02-19T21:19:43.474941+0000","last_peered":"2024-02-19T21:19:43.474941+0000","last_clean":"2024-02-19T21:19:43.474941+0000","last_became_active":"2024-02-19T21:18:03.730876+0000","last_became_peered":"2024-02-19T21:18:03.730876+0000","last_unstale":"2024-02-19T21:19:43.474941+0000","last_undegraded":"2024-02-19T21:19:43.474941+0000","last_fullsized":"2024-02-19T21:19:43.474941+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:43.474803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:19:43.474803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":96,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:57.225281+0000","last_change":"2024-02-19T21:19:57.225281+0000","last_active":"2024-02-19T21:19:57.225281+0000","last_peered":"2024-02-19T21:19:57.225281+0000","last_clean":"2024-02-19T21:19:57.225281+0000","last_became_active":"2024-02-19T21:18:03.742208+0000","last_became_peered":"2024-02-19T21:18:03.742208+0000","last_unstale":"2024-02-19T21:19:57.225281+0000","last_undegraded":"2024-02-19T21:19:57.225281+0000","last_fullsized":"2024-02-19T21:19:57.225281+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:57.225206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:57.225206+0000","last_clean_scrub_stamp":"2024-02-19T21:19:57.225206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"459'15","reported_seq":440,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968353+0000","last_change":"2024-02-19T21:19:05.311570+0000","last_active":"2024-02-19T21:19:40.968353+0000","last_peered":"2024-02-19T21:19:40.968353+0000","last_clean":"2024-02-19T21:19:40.968353+0000","last_became_active":"2024-02-19T21:18:19.174039+0000","last_became_peered":"2024-02-19T21:18:19.174039+0000","last_unstale":"2024-02-19T21:19:40.968353+0000","last_undegraded":"2024-02-19T21:19:40.968353+0000","last_fullsized":"2024-02-19T21:19:40.968353+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:19:05.311494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:05.311494+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":87,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:59.232504+0000","last_change":"2024-02-19T21:19:59.232504+0000","last_active":"2024-02-19T21:19:59.232504+0000","last_peered":"2024-02-19T21:19:59.232504+0000","last_clean":"2024-02-19T21:19:59.232504+0000","last_became_active":"2024-02-19T21:17:29.527055+0000","last_became_peered":"2024-02-19T21:17:29.527055+0000","last_unstale":"2024-02-19T21:19:59.232504+0000","last_undegraded":"2024-02-19T21:19:59.232504+0000","last_fullsized":"2024-02-19T21:19:59.232504+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:59.232438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:59.232438+0000","last_clean_scrub_stamp":"2024-02-19T21:19:59.232438+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":598,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851458+0000","last_change":"2024-02-19T21:19:16.216864+0000","last_active":"2024-02-19T21:19:40.851458+0000","last_peered":"2024-02-19T21:19:40.851458+0000","last_clean":"2024-02-19T21:19:40.851458+0000","last_became_active":"2024-02-19T21:18:03.730784+0000","last_became_peered":"2024-02-19T21:18:03.730784+0000","last_unstale":"2024-02-19T21:19:40.851458+0000","last_undegraded":"2024-02-19T21:19:40.851458+0000","last_fullsized":"2024-02-19T21:19:40.851458+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:16.216788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:16.216788+0000","last_clean_scrub_stamp":"2024-02-19T21:19:16.216788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":96,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.827998+0000","last_change":"2024-02-19T21:19:10.984020+0000","last_active":"2024-02-19T21:19:38.827998+0000","last_peered":"2024-02-19T21:19:38.827998+0000","last_clean":"2024-02-19T21:19:38.827998+0000","last_became_active":"2024-02-19T21:18:03.743303+0000","last_became_peered":"2024-02-19T21:18:03.743303+0000","last_unstale":"2024-02-19T21:19:38.827998+0000","last_undegraded":"2024-02-19T21:19:38.827998+0000","last_fullsized":"2024-02-19T21:19:38.827998+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:10.983950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:10.983950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":456,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.379506+0000","last_change":"2024-02-19T21:18:59.442878+0000","last_active":"2024-02-19T21:19:39.379506+0000","last_peered":"2024-02-19T21:19:39.379506+0000","last_clean":"2024-02-19T21:19:39.379506+0000","last_became_active":"2024-02-19T21:18:03.725494+0000","last_became_peered":"2024-02-19T21:18:03.725494+0000","last_unstale":"2024-02-19T21:19:39.379506+0000","last_undegraded":"2024-02-19T21:19:39.379506+0000","last_fullsized":"2024-02-19T21:19:39.379506+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:59.442799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:18:59.442799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"459'15","reported_seq":187,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851179+0000","last_change":"2024-02-19T21:18:50.289558+0000","last_active":"2024-02-19T21:19:40.851179+0000","last_peered":"2024-02-19T21:19:40.851179+0000","last_clean":"2024-02-19T21:19:40.851179+0000","last_became_active":"2024-02-19T21:18:03.730847+0000","last_became_peered":"2024-02-19T21:18:03.730847+0000","last_unstale":"2024-02-19T21:19:40.851179+0000","last_undegraded":"2024-02-19T21:19:40.851179+0000","last_fullsized":"2024-02-19T21:19:40.851179+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:18:50.289450+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:18:50.289450+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":252,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969053+0000","last_change":"2024-02-19T21:19:00.571835+0000","last_active":"2024-02-19T21:19:40.969053+0000","last_peered":"2024-02-19T21:19:40.969053+0000","last_clean":"2024-02-19T21:19:40.969053+0000","last_became_active":"2024-02-19T21:14:16.385407+0000","last_became_peered":"2024-02-19T21:14:16.385407+0000","last_unstale":"2024-02-19T21:19:40.969053+0000","last_undegraded":"2024-02-19T21:19:40.969053+0000","last_fullsized":"2024-02-19T21:19:40.969053+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:00.571763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:07.220022+0000","last_clean_scrub_stamp":"2024-02-19T21:19:00.571763+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":58,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.804079+0000","last_change":"2024-02-19T21:19:17.628960+0000","last_active":"2024-02-19T21:19:39.804079+0000","last_peered":"2024-02-19T21:19:39.804079+0000","last_clean":"2024-02-19T21:19:39.804079+0000","last_became_active":"2024-02-19T21:17:58.686411+0000","last_became_peered":"2024-02-19T21:17:58.686411+0000","last_unstale":"2024-02-19T21:19:39.804079+0000","last_undegraded":"2024-02-19T21:19:39.804079+0000","last_fullsized":"2024-02-19T21:19:39.804079+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:17.628866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:17.628866+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":741,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:47.459819+0000","last_change":"2024-02-19T21:19:47.459791+0000","last_active":"2024-02-19T21:19:47.459819+0000","last_peered":"2024-02-19T21:19:47.459819+0000","last_clean":"2024-02-19T21:19:47.459819+0000","last_became_active":"2024-02-19T21:19:10.742604+0000","last_became_peered":"2024-02-19T21:19:10.742604+0000","last_unstale":"2024-02-19T21:19:47.459819+0000","last_undegraded":"2024-02-19T21:19:47.459819+0000","last_fullsized":"2024-02-19T21:19:47.459819+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:47.459713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:47.459713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":63,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.661411+0000","last_change":"2024-02-19T21:19:40.661379+0000","last_active":"2024-02-19T21:19:40.661411+0000","last_peered":"2024-02-19T21:19:40.661411+0000","last_clean":"2024-02-19T21:19:40.661411+0000","last_became_active":"2024-02-19T21:17:58.689211+0000","last_became_peered":"2024-02-19T21:17:58.689211+0000","last_unstale":"2024-02-19T21:19:40.661411+0000","last_undegraded":"2024-02-19T21:19:40.661411+0000","last_fullsized":"2024-02-19T21:19:40.661411+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:40.661301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:40.661301+0000","last_clean_scrub_stamp":"2024-02-19T21:19:40.661301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"816'25","reported_seq":675,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851502+0000","last_change":"2024-02-19T21:19:30.192778+0000","last_active":"2024-02-19T21:19:40.851502+0000","last_peered":"2024-02-19T21:19:40.851502+0000","last_clean":"2024-02-19T21:19:40.851502+0000","last_became_active":"2024-02-19T21:18:03.732603+0000","last_became_peered":"2024-02-19T21:18:03.732603+0000","last_unstale":"2024-02-19T21:19:40.851502+0000","last_undegraded":"2024-02-19T21:19:40.851502+0000","last_fullsized":"2024-02-19T21:19:40.851502+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'25","last_scrub_stamp":"2024-02-19T21:19:30.192706+0000","last_deep_scrub":"816'25","last_deep_scrub_stamp":"2024-02-19T21:19:30.192706+0000","last_clean_scrub_stamp":"2024-02-19T21:19:30.192706+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":72,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.802746+0000","last_change":"2024-02-19T21:19:27.581663+0000","last_active":"2024-02-19T21:19:39.802746+0000","last_peered":"2024-02-19T21:19:39.802746+0000","last_clean":"2024-02-19T21:19:39.802746+0000","last_became_active":"2024-02-19T21:18:03.732785+0000","last_became_peered":"2024-02-19T21:18:03.732785+0000","last_unstale":"2024-02-19T21:19:39.802746+0000","last_undegraded":"2024-02-19T21:19:39.802746+0000","last_fullsized":"2024-02-19T21:19:39.802746+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:27.581567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_clean_scrub_stamp":"2024-02-19T21:19:27.581567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"811'12","reported_seq":87,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:54.800960+0000","last_change":"2024-02-19T21:18:28.428328+0000","last_active":"2024-02-19T21:19:54.800960+0000","last_peered":"2024-02-19T21:19:54.800960+0000","last_clean":"2024-02-19T21:19:54.800960+0000","last_became_active":"2024-02-19T21:17:58.687483+0000","last_became_peered":"2024-02-19T21:17:58.687483+0000","last_unstale":"2024-02-19T21:19:54.800960+0000","last_undegraded":"2024-02-19T21:19:54.800960+0000","last_fullsized":"2024-02-19T21:19:54.800960+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"811'12","last_scrub_stamp":"2024-02-19T21:18:28.428243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:18:28.428243+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":528,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968206+0000","last_change":"2024-02-19T21:19:12.315318+0000","last_active":"2024-02-19T21:19:40.968206+0000","last_peered":"2024-02-19T21:19:40.968206+0000","last_clean":"2024-02-19T21:19:40.968206+0000","last_became_active":"2024-02-19T21:18:19.174474+0000","last_became_peered":"2024-02-19T21:18:19.174474+0000","last_unstale":"2024-02-19T21:19:40.968206+0000","last_undegraded":"2024-02-19T21:19:40.968206+0000","last_fullsized":"2024-02-19T21:19:40.968206+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:19:12.315246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:12.315246+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":240,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.966849+0000","last_change":"2024-02-19T21:18:51.349780+0000","last_active":"2024-02-19T21:19:40.966849+0000","last_peered":"2024-02-19T21:19:40.966849+0000","last_clean":"2024-02-19T21:19:40.966849+0000","last_became_active":"2024-02-19T21:18:19.169061+0000","last_became_peered":"2024-02-19T21:18:19.169061+0000","last_unstale":"2024-02-19T21:19:40.966849+0000","last_undegraded":"2024-02-19T21:19:40.966849+0000","last_fullsized":"2024-02-19T21:19:40.966849+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:51.349682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:23.608673+0000","last_clean_scrub_stamp":"2024-02-19T21:18:51.349682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"732'19","reported_seq":87,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.802834+0000","last_change":"2024-02-19T21:18:48.603165+0000","last_active":"2024-02-19T21:19:39.802834+0000","last_peered":"2024-02-19T21:19:39.802834+0000","last_clean":"2024-02-19T21:19:39.802834+0000","last_became_active":"2024-02-19T21:18:03.732567+0000","last_became_peered":"2024-02-19T21:18:03.732567+0000","last_unstale":"2024-02-19T21:19:39.802834+0000","last_undegraded":"2024-02-19T21:19:39.802834+0000","last_fullsized":"2024-02-19T21:19:39.802834+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'19","last_scrub_stamp":"2024-02-19T21:18:48.603069+0000","last_deep_scrub":"732'19","last_deep_scrub_stamp":"2024-02-19T21:18:48.603069+0000","last_clean_scrub_stamp":"2024-02-19T21:18:48.603069+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":277,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967703+0000","last_change":"2024-02-19T21:18:44.276441+0000","last_active":"2024-02-19T21:19:40.967703+0000","last_peered":"2024-02-19T21:19:40.967703+0000","last_clean":"2024-02-19T21:19:40.967703+0000","last_became_active":"2024-02-19T21:18:19.171116+0000","last_became_peered":"2024-02-19T21:18:19.171116+0000","last_unstale":"2024-02-19T21:19:40.967703+0000","last_undegraded":"2024-02-19T21:19:40.967703+0000","last_fullsized":"2024-02-19T21:19:40.967703+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:44.276378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:18:44.276378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"816'14","reported_seq":785,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:59.801871+0000","last_change":"2024-02-19T21:19:50.616694+0000","last_active":"2024-02-19T21:19:59.801871+0000","last_peered":"2024-02-19T21:19:59.801871+0000","last_clean":"2024-02-19T21:19:59.801871+0000","last_became_active":"2024-02-19T21:18:03.731454+0000","last_became_peered":"2024-02-19T21:18:03.731454+0000","last_unstale":"2024-02-19T21:19:59.801871+0000","last_undegraded":"2024-02-19T21:19:59.801871+0000","last_fullsized":"2024-02-19T21:19:59.801871+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'14","last_scrub_stamp":"2024-02-19T21:19:50.616622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:50.616622+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":488,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:42.381733+0000","last_change":"2024-02-19T21:19:42.381733+0000","last_active":"2024-02-19T21:19:42.381733+0000","last_peered":"2024-02-19T21:19:42.381733+0000","last_clean":"2024-02-19T21:19:42.381733+0000","last_became_active":"2024-02-19T21:17:58.680636+0000","last_became_peered":"2024-02-19T21:17:58.680636+0000","last_unstale":"2024-02-19T21:19:42.381733+0000","last_undegraded":"2024-02-19T21:19:42.381733+0000","last_fullsized":"2024-02-19T21:19:42.381733+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:42.381654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:42.381654+0000","last_clean_scrub_stamp":"2024-02-19T21:19:42.381654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"801'8","reported_seq":92,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.829575+0000","last_change":"2024-02-19T21:19:14.948411+0000","last_active":"2024-02-19T21:19:38.829575+0000","last_peered":"2024-02-19T21:19:38.829575+0000","last_clean":"2024-02-19T21:19:38.829575+0000","last_became_active":"2024-02-19T21:17:29.511622+0000","last_became_peered":"2024-02-19T21:17:29.511622+0000","last_unstale":"2024-02-19T21:19:38.829575+0000","last_undegraded":"2024-02-19T21:19:38.829575+0000","last_fullsized":"2024-02-19T21:19:38.829575+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"801'8","last_scrub_stamp":"2024-02-19T21:19:14.948334+0000","last_deep_scrub":"801'8","last_deep_scrub_stamp":"2024-02-19T21:17:44.622190+0000","last_clean_scrub_stamp":"2024-02-19T21:19:14.948334+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":20,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"459'15","reported_seq":39,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.803253+0000","last_change":"2024-02-19T21:18:42.235565+0000","last_active":"2024-02-19T21:19:39.803253+0000","last_peered":"2024-02-19T21:19:39.803253+0000","last_clean":"2024-02-19T21:19:39.803253+0000","last_became_active":"2024-02-19T21:18:41.543916+0000","last_became_peered":"2024-02-19T21:18:41.543916+0000","last_unstale":"2024-02-19T21:19:39.803253+0000","last_undegraded":"2024-02-19T21:19:39.803253+0000","last_fullsized":"2024-02-19T21:19:39.803253+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:18:29.527734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:18:29.527734+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":288,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967930+0000","last_change":"2024-02-19T21:19:07.512940+0000","last_active":"2024-02-19T21:19:40.967930+0000","last_peered":"2024-02-19T21:19:40.967930+0000","last_clean":"2024-02-19T21:19:40.967930+0000","last_became_active":"2024-02-19T21:17:58.682002+0000","last_became_peered":"2024-02-19T21:17:58.682002+0000","last_unstale":"2024-02-19T21:19:40.967930+0000","last_undegraded":"2024-02-19T21:19:40.967930+0000","last_fullsized":"2024-02-19T21:19:40.967930+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:07.512855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:07.512855+0000","last_clean_scrub_stamp":"2024-02-19T21:19:07.512855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":603,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852111+0000","last_change":"2024-02-19T21:19:23.326855+0000","last_active":"2024-02-19T21:19:40.852111+0000","last_peered":"2024-02-19T21:19:40.852111+0000","last_clean":"2024-02-19T21:19:40.852111+0000","last_became_active":"2024-02-19T21:14:41.398865+0000","last_became_peered":"2024-02-19T21:14:41.398865+0000","last_unstale":"2024-02-19T21:19:40.852111+0000","last_undegraded":"2024-02-19T21:19:40.852111+0000","last_fullsized":"2024-02-19T21:19:40.852111+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:23.326783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_clean_scrub_stamp":"2024-02-19T21:19:23.326783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":516,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:52.189992+0000","last_change":"2024-02-19T21:19:52.189967+0000","last_active":"2024-02-19T21:19:52.189992+0000","last_peered":"2024-02-19T21:19:52.189992+0000","last_clean":"2024-02-19T21:19:52.189992+0000","last_became_active":"2024-02-19T21:18:19.169406+0000","last_became_peered":"2024-02-19T21:18:19.169406+0000","last_unstale":"2024-02-19T21:19:52.189992+0000","last_undegraded":"2024-02-19T21:19:52.189992+0000","last_fullsized":"2024-02-19T21:19:52.189992+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:52.189887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:52.189887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":302,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:47.528440+0000","last_change":"2024-02-19T21:19:47.528413+0000","last_active":"2024-02-19T21:19:47.528440+0000","last_peered":"2024-02-19T21:19:47.528440+0000","last_clean":"2024-02-19T21:19:47.528440+0000","last_became_active":"2024-02-19T21:17:58.675970+0000","last_became_peered":"2024-02-19T21:17:58.675970+0000","last_unstale":"2024-02-19T21:19:47.528440+0000","last_undegraded":"2024-02-19T21:19:47.528440+0000","last_fullsized":"2024-02-19T21:19:47.528440+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:47.528343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:37.674820+0000","last_clean_scrub_stamp":"2024-02-19T21:19:47.528343+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"853'24","reported_seq":514,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:45.458867+0000","last_change":"2024-02-19T21:19:45.458867+0000","last_active":"2024-02-19T21:19:45.458867+0000","last_peered":"2024-02-19T21:19:45.458867+0000","last_clean":"2024-02-19T21:19:45.458867+0000","last_became_active":"2024-02-19T21:14:16.380020+0000","last_became_peered":"2024-02-19T21:14:16.380020+0000","last_unstale":"2024-02-19T21:19:45.458867+0000","last_undegraded":"2024-02-19T21:19:45.458867+0000","last_fullsized":"2024-02-19T21:19:45.458867+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"853'24","last_scrub_stamp":"2024-02-19T21:19:45.458789+0000","last_deep_scrub":"853'24","last_deep_scrub_stamp":"2024-02-19T21:19:45.458789+0000","last_clean_scrub_stamp":"2024-02-19T21:19:45.458789+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":885,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.760362+0000","last_change":"2024-02-19T21:19:40.760340+0000","last_active":"2024-02-19T21:19:40.760362+0000","last_peered":"2024-02-19T21:19:40.760362+0000","last_clean":"2024-02-19T21:19:40.760362+0000","last_became_active":"2024-02-19T21:17:58.694655+0000","last_became_peered":"2024-02-19T21:17:58.694655+0000","last_unstale":"2024-02-19T21:19:40.760362+0000","last_undegraded":"2024-02-19T21:19:40.760362+0000","last_fullsized":"2024-02-19T21:19:40.760362+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:40.760279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:22.364374+0000","last_clean_scrub_stamp":"2024-02-19T21:19:40.760279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":872,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:48.706829+0000","last_change":"2024-02-19T21:19:48.706829+0000","last_active":"2024-02-19T21:19:48.706829+0000","last_peered":"2024-02-19T21:19:48.706829+0000","last_clean":"2024-02-19T21:19:48.706829+0000","last_became_active":"2024-02-19T21:17:58.687747+0000","last_became_peered":"2024-02-19T21:17:58.687747+0000","last_unstale":"2024-02-19T21:19:48.706829+0000","last_undegraded":"2024-02-19T21:19:48.706829+0000","last_fullsized":"2024-02-19T21:19:48.706829+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:48.706764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:48.706764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":625,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:48.617052+0000","last_change":"2024-02-19T21:19:48.617028+0000","last_active":"2024-02-19T21:19:48.617052+0000","last_peered":"2024-02-19T21:19:48.617052+0000","last_clean":"2024-02-19T21:19:48.617052+0000","last_became_active":"2024-02-19T21:17:58.674337+0000","last_became_peered":"2024-02-19T21:17:58.674337+0000","last_unstale":"2024-02-19T21:19:48.617052+0000","last_undegraded":"2024-02-19T21:19:48.617052+0000","last_fullsized":"2024-02-19T21:19:48.617052+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:48.616963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:48.616963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851545+0000","last_change":"2024-02-19T21:19:22.280953+0000","last_active":"2024-02-19T21:19:40.851545+0000","last_peered":"2024-02-19T21:19:40.851545+0000","last_clean":"2024-02-19T21:19:40.851545+0000","last_became_active":"2024-02-19T21:17:29.452050+0000","last_became_peered":"2024-02-19T21:17:29.452050+0000","last_unstale":"2024-02-19T21:19:40.851545+0000","last_undegraded":"2024-02-19T21:19:40.851545+0000","last_fullsized":"2024-02-19T21:19:40.851545+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:22.280883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:59.059572+0000","last_clean_scrub_stamp":"2024-02-19T21:19:22.280883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"555'10","reported_seq":119,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852294+0000","last_change":"2024-02-19T21:19:37.268171+0000","last_active":"2024-02-19T21:19:40.852294+0000","last_peered":"2024-02-19T21:19:40.852294+0000","last_clean":"2024-02-19T21:19:40.852294+0000","last_became_active":"2024-02-19T21:16:58.576112+0000","last_became_peered":"2024-02-19T21:16:58.576112+0000","last_unstale":"2024-02-19T21:19:40.852294+0000","last_undegraded":"2024-02-19T21:19:40.852294+0000","last_fullsized":"2024-02-19T21:19:40.852294+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'10","last_scrub_stamp":"2024-02-19T21:19:37.268103+0000","last_deep_scrub":"555'10","last_deep_scrub_stamp":"2024-02-19T21:13:12.409036+0000","last_clean_scrub_stamp":"2024-02-19T21:19:37.268103+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":63,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:43.684798+0000","last_change":"2024-02-19T21:19:43.684798+0000","last_active":"2024-02-19T21:19:43.684798+0000","last_peered":"2024-02-19T21:19:43.684798+0000","last_clean":"2024-02-19T21:19:43.684798+0000","last_became_active":"2024-02-19T21:17:58.688322+0000","last_became_peered":"2024-02-19T21:17:58.688322+0000","last_unstale":"2024-02-19T21:19:43.684798+0000","last_undegraded":"2024-02-19T21:19:43.684798+0000","last_fullsized":"2024-02-19T21:19:43.684798+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:43.684706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:43.684706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":270,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:49.545775+0000","last_change":"2024-02-19T21:19:49.545749+0000","last_active":"2024-02-19T21:19:49.545775+0000","last_peered":"2024-02-19T21:19:49.545775+0000","last_clean":"2024-02-19T21:19:49.545775+0000","last_became_active":"2024-02-19T21:18:03.725686+0000","last_became_peered":"2024-02-19T21:18:03.725686+0000","last_unstale":"2024-02-19T21:19:49.545775+0000","last_undegraded":"2024-02-19T21:19:49.545775+0000","last_fullsized":"2024-02-19T21:19:49.545775+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:49.545680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:49.545680+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":66,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967539+0000","last_change":"2024-02-19T21:19:13.598831+0000","last_active":"2024-02-19T21:19:40.967539+0000","last_peered":"2024-02-19T21:19:40.967539+0000","last_clean":"2024-02-19T21:19:40.967539+0000","last_became_active":"2024-02-19T21:19:10.740674+0000","last_became_peered":"2024-02-19T21:19:10.740674+0000","last_unstale":"2024-02-19T21:19:40.967539+0000","last_undegraded":"2024-02-19T21:19:40.967539+0000","last_fullsized":"2024-02-19T21:19:40.967539+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:13.598706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:13.598706+0000","last_clean_scrub_stamp":"2024-02-19T21:19:13.598706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":612,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:42.443267+0000","last_change":"2024-02-19T21:19:42.443225+0000","last_active":"2024-02-19T21:19:42.443267+0000","last_peered":"2024-02-19T21:19:42.443267+0000","last_clean":"2024-02-19T21:19:42.443267+0000","last_became_active":"2024-02-19T21:18:03.737373+0000","last_became_peered":"2024-02-19T21:18:03.737373+0000","last_unstale":"2024-02-19T21:19:42.443267+0000","last_undegraded":"2024-02-19T21:19:42.443267+0000","last_fullsized":"2024-02-19T21:19:42.443267+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:42.443160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:42.443160+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"850'24","reported_seq":719,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852240+0000","last_change":"2024-02-19T21:19:34.253671+0000","last_active":"2024-02-19T21:19:40.852240+0000","last_peered":"2024-02-19T21:19:40.852240+0000","last_clean":"2024-02-19T21:19:40.852240+0000","last_became_active":"2024-02-19T21:18:03.722651+0000","last_became_peered":"2024-02-19T21:18:03.722651+0000","last_unstale":"2024-02-19T21:19:40.852240+0000","last_undegraded":"2024-02-19T21:19:40.852240+0000","last_fullsized":"2024-02-19T21:19:40.852240+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"850'24","last_scrub_stamp":"2024-02-19T21:19:34.253611+0000","last_deep_scrub":"850'24","last_deep_scrub_stamp":"2024-02-19T21:19:34.253611+0000","last_clean_scrub_stamp":"2024-02-19T21:19:34.253611+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":290,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969127+0000","last_change":"2024-02-19T21:18:49.664237+0000","last_active":"2024-02-19T21:19:40.969127+0000","last_peered":"2024-02-19T21:19:40.969127+0000","last_clean":"2024-02-19T21:19:40.969127+0000","last_became_active":"2024-02-19T21:17:58.683408+0000","last_became_peered":"2024-02-19T21:17:58.683408+0000","last_unstale":"2024-02-19T21:19:40.969127+0000","last_undegraded":"2024-02-19T21:19:40.969127+0000","last_fullsized":"2024-02-19T21:19:40.969127+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:49.664164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:49.664164+0000","last_clean_scrub_stamp":"2024-02-19T21:18:49.664164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":729,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967956+0000","last_change":"2024-02-19T21:19:16.563780+0000","last_active":"2024-02-19T21:19:40.967956+0000","last_peered":"2024-02-19T21:19:40.967956+0000","last_clean":"2024-02-19T21:19:40.967956+0000","last_became_active":"2024-02-19T21:19:10.742227+0000","last_became_peered":"2024-02-19T21:19:10.742227+0000","last_unstale":"2024-02-19T21:19:40.967956+0000","last_undegraded":"2024-02-19T21:19:40.967956+0000","last_fullsized":"2024-02-19T21:19:40.967956+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:16.563699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:16.563699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852194+0000","last_change":"2024-02-19T21:19:10.213745+0000","last_active":"2024-02-19T21:19:40.852194+0000","last_peered":"2024-02-19T21:19:40.852194+0000","last_clean":"2024-02-19T21:19:40.852194+0000","last_became_active":"2024-02-19T21:14:41.398484+0000","last_became_peered":"2024-02-19T21:14:41.398484+0000","last_unstale":"2024-02-19T21:19:40.852194+0000","last_undegraded":"2024-02-19T21:19:40.852194+0000","last_fullsized":"2024-02-19T21:19:40.852194+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:10.213671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:10.213671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":269,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968441+0000","last_change":"2024-02-19T21:19:18.345285+0000","last_active":"2024-02-19T21:19:40.968441+0000","last_peered":"2024-02-19T21:19:40.968441+0000","last_clean":"2024-02-19T21:19:40.968441+0000","last_became_active":"2024-02-19T21:18:19.166023+0000","last_became_peered":"2024-02-19T21:18:19.166023+0000","last_unstale":"2024-02-19T21:19:40.968441+0000","last_undegraded":"2024-02-19T21:19:40.968441+0000","last_fullsized":"2024-02-19T21:19:40.968441+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:18.345211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:19:18.345211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"459'20","reported_seq":580,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968162+0000","last_change":"2024-02-19T21:19:26.355932+0000","last_active":"2024-02-19T21:19:40.968162+0000","last_peered":"2024-02-19T21:19:40.968162+0000","last_clean":"2024-02-19T21:19:40.968162+0000","last_became_active":"2024-02-19T21:18:19.174836+0000","last_became_peered":"2024-02-19T21:18:19.174836+0000","last_unstale":"2024-02-19T21:19:40.968162+0000","last_undegraded":"2024-02-19T21:19:40.968162+0000","last_fullsized":"2024-02-19T21:19:40.968162+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:19:26.355824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:26.355824+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":72,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:49.657969+0000","last_change":"2024-02-19T21:19:49.657969+0000","last_active":"2024-02-19T21:19:49.657969+0000","last_peered":"2024-02-19T21:19:49.657969+0000","last_clean":"2024-02-19T21:19:49.657969+0000","last_became_active":"2024-02-19T21:18:03.734371+0000","last_became_peered":"2024-02-19T21:18:03.734371+0000","last_unstale":"2024-02-19T21:19:49.657969+0000","last_undegraded":"2024-02-19T21:19:49.657969+0000","last_fullsized":"2024-02-19T21:19:49.657969+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:49.657890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:29.454663+0000","last_clean_scrub_stamp":"2024-02-19T21:19:49.657890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"816'15","reported_seq":364,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:59.802249+0000","last_change":"2024-02-19T21:19:11.435110+0000","last_active":"2024-02-19T21:19:59.802249+0000","last_peered":"2024-02-19T21:19:59.802249+0000","last_clean":"2024-02-19T21:19:59.802249+0000","last_became_active":"2024-02-19T21:18:03.726129+0000","last_became_peered":"2024-02-19T21:18:03.726129+0000","last_unstale":"2024-02-19T21:19:59.802249+0000","last_undegraded":"2024-02-19T21:19:59.802249+0000","last_fullsized":"2024-02-19T21:19:59.802249+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'15","last_scrub_stamp":"2024-02-19T21:19:11.435014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:11.435014+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":36,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969211+0000","last_change":"2024-02-19T21:19:10.740904+0000","last_active":"2024-02-19T21:19:40.969211+0000","last_peered":"2024-02-19T21:19:40.969211+0000","last_clean":"2024-02-19T21:19:40.969211+0000","last_became_active":"2024-02-19T21:19:10.740395+0000","last_became_peered":"2024-02-19T21:19:10.740395+0000","last_unstale":"2024-02-19T21:19:40.969211+0000","last_undegraded":"2024-02-19T21:19:40.969211+0000","last_fullsized":"2024-02-19T21:19:40.969211+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:19:07.545452+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:19:07.545452+0000","last_clean_scrub_stamp":"2024-02-19T21:19:07.545452+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":91,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.828834+0000","last_change":"2024-02-19T21:18:35.702138+0000","last_active":"2024-02-19T21:19:38.828834+0000","last_peered":"2024-02-19T21:19:38.828834+0000","last_clean":"2024-02-19T21:19:38.828834+0000","last_became_active":"2024-02-19T21:18:03.742082+0000","last_became_peered":"2024-02-19T21:18:03.742082+0000","last_unstale":"2024-02-19T21:19:38.828834+0000","last_undegraded":"2024-02-19T21:19:38.828834+0000","last_fullsized":"2024-02-19T21:19:38.828834+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:35.702049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:18:35.702049+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"459'19","reported_seq":311,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:54.544131+0000","last_change":"2024-02-19T21:19:54.544099+0000","last_active":"2024-02-19T21:19:54.544131+0000","last_peered":"2024-02-19T21:19:54.544131+0000","last_clean":"2024-02-19T21:19:54.544131+0000","last_became_active":"2024-02-19T21:17:58.682999+0000","last_became_peered":"2024-02-19T21:17:58.682999+0000","last_unstale":"2024-02-19T21:19:54.544131+0000","last_undegraded":"2024-02-19T21:19:54.544131+0000","last_fullsized":"2024-02-19T21:19:54.544131+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:19:54.544019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:54.544019+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852324+0000","last_change":"2024-02-19T21:19:29.208508+0000","last_active":"2024-02-19T21:19:40.852324+0000","last_peered":"2024-02-19T21:19:40.852324+0000","last_clean":"2024-02-19T21:19:40.852324+0000","last_became_active":"2024-02-19T21:14:41.402278+0000","last_became_peered":"2024-02-19T21:14:41.402278+0000","last_unstale":"2024-02-19T21:19:40.852324+0000","last_undegraded":"2024-02-19T21:19:40.852324+0000","last_fullsized":"2024-02-19T21:19:40.852324+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:29.208404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:29.208404+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":492,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968080+0000","last_change":"2024-02-19T21:19:01.353521+0000","last_active":"2024-02-19T21:19:40.968080+0000","last_peered":"2024-02-19T21:19:40.968080+0000","last_clean":"2024-02-19T21:19:40.968080+0000","last_became_active":"2024-02-19T21:18:19.175181+0000","last_became_peered":"2024-02-19T21:18:19.175181+0000","last_unstale":"2024-02-19T21:19:40.968080+0000","last_undegraded":"2024-02-19T21:19:40.968080+0000","last_fullsized":"2024-02-19T21:19:40.968080+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:01.353430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:01.353430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":865,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:44.691588+0000","last_change":"2024-02-19T21:19:44.691588+0000","last_active":"2024-02-19T21:19:44.691588+0000","last_peered":"2024-02-19T21:19:44.691588+0000","last_clean":"2024-02-19T21:19:44.691588+0000","last_became_active":"2024-02-19T21:17:58.695090+0000","last_became_peered":"2024-02-19T21:17:58.695090+0000","last_unstale":"2024-02-19T21:19:44.691588+0000","last_undegraded":"2024-02-19T21:19:44.691588+0000","last_fullsized":"2024-02-19T21:19:44.691588+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:44.691518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:44.691518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":819,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.070706+0000","last_change":"2024-02-19T21:19:01.398622+0000","last_active":"2024-02-19T21:19:39.070706+0000","last_peered":"2024-02-19T21:19:39.070706+0000","last_clean":"2024-02-19T21:19:39.070706+0000","last_became_active":"2024-02-19T21:17:29.530233+0000","last_became_peered":"2024-02-19T21:17:29.530233+0000","last_unstale":"2024-02-19T21:19:39.070706+0000","last_undegraded":"2024-02-19T21:19:39.070706+0000","last_fullsized":"2024-02-19T21:19:39.070706+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:01.398540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:01.398540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"459'34","reported_seq":103,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:52.317077+0000","last_change":"2024-02-19T21:19:52.317077+0000","last_active":"2024-02-19T21:19:52.317077+0000","last_peered":"2024-02-19T21:19:52.317077+0000","last_clean":"2024-02-19T21:19:52.317077+0000","last_became_active":"2024-02-19T21:17:29.511788+0000","last_became_peered":"2024-02-19T21:17:29.511788+0000","last_unstale":"2024-02-19T21:19:52.317077+0000","last_undegraded":"2024-02-19T21:19:52.317077+0000","last_fullsized":"2024-02-19T21:19:52.317077+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'34","last_scrub_stamp":"2024-02-19T21:19:52.317003+0000","last_deep_scrub":"459'34","last_deep_scrub_stamp":"2024-02-19T21:18:30.676424+0000","last_clean_scrub_stamp":"2024-02-19T21:19:52.317003+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":306,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969665+0000","last_change":"2024-02-19T21:19:40.508731+0000","last_active":"2024-02-19T21:19:40.969665+0000","last_peered":"2024-02-19T21:19:40.969665+0000","last_clean":"2024-02-19T21:19:40.969665+0000","last_became_active":"2024-02-19T21:18:41.542346+0000","last_became_peered":"2024-02-19T21:18:41.542346+0000","last_unstale":"2024-02-19T21:19:40.969665+0000","last_undegraded":"2024-02-19T21:19:40.969665+0000","last_fullsized":"2024-02-19T21:19:40.969665+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:40.508619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:40.508619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":858,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.071170+0000","last_change":"2024-02-19T21:18:56.474437+0000","last_active":"2024-02-19T21:19:39.071170+0000","last_peered":"2024-02-19T21:19:39.071170+0000","last_clean":"2024-02-19T21:19:39.071170+0000","last_became_active":"2024-02-19T21:14:41.415463+0000","last_became_peered":"2024-02-19T21:14:41.415463+0000","last_unstale":"2024-02-19T21:19:39.071170+0000","last_undegraded":"2024-02-19T21:19:39.071170+0000","last_fullsized":"2024-02-19T21:19:39.071170+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:56.474372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:18:56.474372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":479,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.380229+0000","last_change":"2024-02-19T21:19:07.534257+0000","last_active":"2024-02-19T21:19:39.380229+0000","last_peered":"2024-02-19T21:19:39.380229+0000","last_clean":"2024-02-19T21:19:39.380229+0000","last_became_active":"2024-02-19T21:17:58.675678+0000","last_became_peered":"2024-02-19T21:17:58.675678+0000","last_unstale":"2024-02-19T21:19:39.380229+0000","last_undegraded":"2024-02-19T21:19:39.380229+0000","last_fullsized":"2024-02-19T21:19:39.380229+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:07.534181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:19:07.534181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":73,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.966885+0000","last_change":"2024-02-19T21:19:29.288567+0000","last_active":"2024-02-19T21:19:40.966885+0000","last_peered":"2024-02-19T21:19:40.966885+0000","last_clean":"2024-02-19T21:19:40.966885+0000","last_became_active":"2024-02-19T21:18:19.169386+0000","last_became_peered":"2024-02-19T21:18:19.169386+0000","last_unstale":"2024-02-19T21:19:40.966885+0000","last_undegraded":"2024-02-19T21:19:40.966885+0000","last_fullsized":"2024-02-19T21:19:40.966885+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:29.288497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:29.288497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"459'18","reported_seq":643,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851311+0000","last_change":"2024-02-19T21:18:36.305279+0000","last_active":"2024-02-19T21:19:40.851311+0000","last_peered":"2024-02-19T21:19:40.851311+0000","last_clean":"2024-02-19T21:19:40.851311+0000","last_became_active":"2024-02-19T21:10:41.307690+0000","last_became_peered":"2024-02-19T21:10:41.307690+0000","last_unstale":"2024-02-19T21:19:40.851311+0000","last_undegraded":"2024-02-19T21:19:40.851311+0000","last_fullsized":"2024-02-19T21:19:40.851311+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'18","last_scrub_stamp":"2024-02-19T21:18:36.305194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:18:36.305194+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852299+0000","last_change":"2024-02-19T21:19:05.235928+0000","last_active":"2024-02-19T21:19:40.852299+0000","last_peered":"2024-02-19T21:19:40.852299+0000","last_clean":"2024-02-19T21:19:40.852299+0000","last_became_active":"2024-02-19T21:18:03.731257+0000","last_became_peered":"2024-02-19T21:18:03.731257+0000","last_unstale":"2024-02-19T21:19:40.852299+0000","last_undegraded":"2024-02-19T21:19:40.852299+0000","last_fullsized":"2024-02-19T21:19:40.852299+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:05.235851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:05.235851+0000","last_clean_scrub_stamp":"2024-02-19T21:19:05.235851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":470,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.379334+0000","last_change":"2024-02-19T21:19:39.378918+0000","last_active":"2024-02-19T21:19:39.379334+0000","last_peered":"2024-02-19T21:19:39.379334+0000","last_clean":"2024-02-19T21:19:39.379334+0000","last_became_active":"2024-02-19T21:17:29.471059+0000","last_became_peered":"2024-02-19T21:17:29.471059+0000","last_unstale":"2024-02-19T21:19:39.379334+0000","last_undegraded":"2024-02-19T21:19:39.379334+0000","last_fullsized":"2024-02-19T21:19:39.379334+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:39.378799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:39.378799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851631+0000","last_change":"2024-02-19T21:19:20.246947+0000","last_active":"2024-02-19T21:19:40.851631+0000","last_peered":"2024-02-19T21:19:40.851631+0000","last_clean":"2024-02-19T21:19:40.851631+0000","last_became_active":"2024-02-19T21:17:58.674931+0000","last_became_peered":"2024-02-19T21:17:58.674931+0000","last_unstale":"2024-02-19T21:19:40.851631+0000","last_undegraded":"2024-02-19T21:19:40.851631+0000","last_fullsized":"2024-02-19T21:19:40.851631+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:20.246873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:19:20.246873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":810,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.070481+0000","last_change":"2024-02-19T21:19:20.706643+0000","last_active":"2024-02-19T21:19:39.070481+0000","last_peered":"2024-02-19T21:19:39.070481+0000","last_clean":"2024-02-19T21:19:39.070481+0000","last_became_active":"2024-02-19T21:17:29.521224+0000","last_became_peered":"2024-02-19T21:17:29.521224+0000","last_unstale":"2024-02-19T21:19:39.070481+0000","last_undegraded":"2024-02-19T21:19:39.070481+0000","last_fullsized":"2024-02-19T21:19:39.070481+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:20.706564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:20.706564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"459'19","reported_seq":495,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.379345+0000","last_change":"2024-02-19T21:18:54.534380+0000","last_active":"2024-02-19T21:19:39.379345+0000","last_peered":"2024-02-19T21:19:39.379345+0000","last_clean":"2024-02-19T21:19:39.379345+0000","last_became_active":"2024-02-19T21:14:41.390502+0000","last_became_peered":"2024-02-19T21:14:41.390502+0000","last_unstale":"2024-02-19T21:19:39.379345+0000","last_undegraded":"2024-02-19T21:19:39.379345+0000","last_fullsized":"2024-02-19T21:19:39.379345+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:18:54.534294+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:18:54.534294+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":37,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:58.353057+0000","last_change":"2024-02-19T21:19:58.353026+0000","last_active":"2024-02-19T21:19:58.353057+0000","last_peered":"2024-02-19T21:19:58.353057+0000","last_clean":"2024-02-19T21:19:58.353057+0000","last_became_active":"2024-02-19T21:19:10.740286+0000","last_became_peered":"2024-02-19T21:19:10.740286+0000","last_unstale":"2024-02-19T21:19:58.353057+0000","last_undegraded":"2024-02-19T21:19:58.353057+0000","last_fullsized":"2024-02-19T21:19:58.353057+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:58.352947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:30.005724+0000","last_clean_scrub_stamp":"2024-02-19T21:19:58.352947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":72,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:45.679492+0000","last_change":"2024-02-19T21:19:45.679492+0000","last_active":"2024-02-19T21:19:45.679492+0000","last_peered":"2024-02-19T21:19:45.679492+0000","last_clean":"2024-02-19T21:19:45.679492+0000","last_became_active":"2024-02-19T21:18:03.735544+0000","last_became_peered":"2024-02-19T21:18:03.735544+0000","last_unstale":"2024-02-19T21:19:45.679492+0000","last_undegraded":"2024-02-19T21:19:45.679492+0000","last_fullsized":"2024-02-19T21:19:45.679492+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:45.679388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:45.679388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":692,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.966954+0000","last_change":"2024-02-19T21:18:53.320550+0000","last_active":"2024-02-19T21:19:40.966954+0000","last_peered":"2024-02-19T21:19:40.966954+0000","last_clean":"2024-02-19T21:19:40.966954+0000","last_became_active":"2024-02-19T21:18:19.174191+0000","last_became_peered":"2024-02-19T21:18:19.174191+0000","last_unstale":"2024-02-19T21:19:40.966954+0000","last_undegraded":"2024-02-19T21:19:40.966954+0000","last_fullsized":"2024-02-19T21:19:40.966954+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:53.320479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:18:53.320479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":87,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:47.303158+0000","last_change":"2024-02-19T21:19:47.303158+0000","last_active":"2024-02-19T21:19:47.303158+0000","last_peered":"2024-02-19T21:19:47.303158+0000","last_clean":"2024-02-19T21:19:47.303158+0000","last_became_active":"2024-02-19T21:17:29.523066+0000","last_became_peered":"2024-02-19T21:17:29.523066+0000","last_unstale":"2024-02-19T21:19:47.303158+0000","last_undegraded":"2024-02-19T21:19:47.303158+0000","last_fullsized":"2024-02-19T21:19:47.303158+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:47.303090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:47.303090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"459'7","reported_seq":447,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.379467+0000","last_change":"2024-02-19T21:18:46.435528+0000","last_active":"2024-02-19T21:19:39.379467+0000","last_peered":"2024-02-19T21:19:39.379467+0000","last_clean":"2024-02-19T21:19:39.379467+0000","last_became_active":"2024-02-19T21:10:41.297295+0000","last_became_peered":"2024-02-19T21:10:41.297295+0000","last_unstale":"2024-02-19T21:19:39.379467+0000","last_undegraded":"2024-02-19T21:19:39.379467+0000","last_fullsized":"2024-02-19T21:19:39.379467+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'7","last_scrub_stamp":"2024-02-19T21:18:46.435443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:18:46.435443+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":34,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:38.828965+0000","last_change":"2024-02-19T21:18:42.754006+0000","last_active":"2024-02-19T21:19:38.828965+0000","last_peered":"2024-02-19T21:19:38.828965+0000","last_clean":"2024-02-19T21:19:38.828965+0000","last_became_active":"2024-02-19T21:18:41.543717+0000","last_became_peered":"2024-02-19T21:18:41.543717+0000","last_unstale":"2024-02-19T21:19:38.828965+0000","last_undegraded":"2024-02-19T21:19:38.828965+0000","last_fullsized":"2024-02-19T21:19:38.828965+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:42.753934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:42.753934+0000","last_clean_scrub_stamp":"2024-02-19T21:18:42.753934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":526,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:42.487367+0000","last_change":"2024-02-19T21:19:42.487341+0000","last_active":"2024-02-19T21:19:42.487367+0000","last_peered":"2024-02-19T21:19:42.487367+0000","last_clean":"2024-02-19T21:19:42.487367+0000","last_became_active":"2024-02-19T21:19:10.740847+0000","last_became_peered":"2024-02-19T21:19:10.740847+0000","last_unstale":"2024-02-19T21:19:42.487367+0000","last_undegraded":"2024-02-19T21:19:42.487367+0000","last_fullsized":"2024-02-19T21:19:42.487367+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:42.487268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:42.487268+0000","last_clean_scrub_stamp":"2024-02-19T21:19:42.487268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":63,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.803241+0000","last_change":"2024-02-19T21:19:26.630405+0000","last_active":"2024-02-19T21:19:39.803241+0000","last_peered":"2024-02-19T21:19:39.803241+0000","last_clean":"2024-02-19T21:19:39.803241+0000","last_became_active":"2024-02-19T21:17:58.685724+0000","last_became_peered":"2024-02-19T21:17:58.685724+0000","last_unstale":"2024-02-19T21:19:39.803241+0000","last_undegraded":"2024-02-19T21:19:39.803241+0000","last_fullsized":"2024-02-19T21:19:39.803241+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:26.630297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:26.630297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":525,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968704+0000","last_change":"2024-02-19T21:19:00.334681+0000","last_active":"2024-02-19T21:19:40.968704+0000","last_peered":"2024-02-19T21:19:40.968704+0000","last_clean":"2024-02-19T21:19:40.968704+0000","last_became_active":"2024-02-19T21:18:19.164839+0000","last_became_peered":"2024-02-19T21:18:19.164839+0000","last_unstale":"2024-02-19T21:19:40.968704+0000","last_undegraded":"2024-02-19T21:19:40.968704+0000","last_fullsized":"2024-02-19T21:19:40.968704+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:00.334607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:00.334607+0000","last_clean_scrub_stamp":"2024-02-19T21:19:00.334607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"623'19","reported_seq":506,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.378717+0000","last_change":"2024-02-19T21:19:11.523034+0000","last_active":"2024-02-19T21:19:39.378717+0000","last_peered":"2024-02-19T21:19:39.378717+0000","last_clean":"2024-02-19T21:19:39.378717+0000","last_became_active":"2024-02-19T21:14:16.383577+0000","last_became_peered":"2024-02-19T21:14:16.383577+0000","last_unstale":"2024-02-19T21:19:39.378717+0000","last_undegraded":"2024-02-19T21:19:39.378717+0000","last_fullsized":"2024-02-19T21:19:39.378717+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"623'19","last_scrub_stamp":"2024-02-19T21:19:11.522944+0000","last_deep_scrub":"623'19","last_deep_scrub_stamp":"2024-02-19T21:14:21.589138+0000","last_clean_scrub_stamp":"2024-02-19T21:19:11.522944+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":634,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851797+0000","last_change":"2024-02-19T21:19:13.152305+0000","last_active":"2024-02-19T21:19:40.851797+0000","last_peered":"2024-02-19T21:19:40.851797+0000","last_clean":"2024-02-19T21:19:40.851797+0000","last_became_active":"2024-02-19T21:17:58.673528+0000","last_became_peered":"2024-02-19T21:17:58.673528+0000","last_unstale":"2024-02-19T21:19:40.851797+0000","last_undegraded":"2024-02-19T21:19:40.851797+0000","last_fullsized":"2024-02-19T21:19:40.851797+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:13.152235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:13.152235+0000","last_clean_scrub_stamp":"2024-02-19T21:19:13.152235+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":872,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:55.703282+0000","last_change":"2024-02-19T21:19:55.703282+0000","last_active":"2024-02-19T21:19:55.703282+0000","last_peered":"2024-02-19T21:19:55.703282+0000","last_clean":"2024-02-19T21:19:55.703282+0000","last_became_active":"2024-02-19T21:18:03.726183+0000","last_became_peered":"2024-02-19T21:18:03.726183+0000","last_unstale":"2024-02-19T21:19:55.703282+0000","last_undegraded":"2024-02-19T21:19:55.703282+0000","last_fullsized":"2024-02-19T21:19:55.703282+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:55.703203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:19:55.703203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":470,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:59.557979+0000","last_change":"2024-02-19T21:19:59.557979+0000","last_active":"2024-02-19T21:19:59.557979+0000","last_peered":"2024-02-19T21:19:59.557979+0000","last_clean":"2024-02-19T21:19:59.557979+0000","last_became_active":"2024-02-19T21:17:58.679633+0000","last_became_peered":"2024-02-19T21:17:58.679633+0000","last_unstale":"2024-02-19T21:19:59.557979+0000","last_undegraded":"2024-02-19T21:19:59.557979+0000","last_fullsized":"2024-02-19T21:19:59.557979+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:59.557898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:59.557898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":63,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.803810+0000","last_change":"2024-02-19T21:19:22.669178+0000","last_active":"2024-02-19T21:19:39.803810+0000","last_peered":"2024-02-19T21:19:39.803810+0000","last_clean":"2024-02-19T21:19:39.803810+0000","last_became_active":"2024-02-19T21:17:58.678930+0000","last_became_peered":"2024-02-19T21:17:58.678930+0000","last_unstale":"2024-02-19T21:19:39.803810+0000","last_undegraded":"2024-02-19T21:19:39.803810+0000","last_fullsized":"2024-02-19T21:19:39.803810+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:22.669094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:22.669094+0000","last_clean_scrub_stamp":"2024-02-19T21:19:22.669094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"459'27","reported_seq":741,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852000+0000","last_change":"2024-02-19T21:18:58.210897+0000","last_active":"2024-02-19T21:19:40.852000+0000","last_peered":"2024-02-19T21:19:40.852000+0000","last_clean":"2024-02-19T21:19:40.852000+0000","last_became_active":"2024-02-19T21:17:29.452764+0000","last_became_peered":"2024-02-19T21:17:29.452764+0000","last_unstale":"2024-02-19T21:19:40.852000+0000","last_undegraded":"2024-02-19T21:19:40.852000+0000","last_fullsized":"2024-02-19T21:19:40.852000+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'27","last_scrub_stamp":"2024-02-19T21:18:58.210821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:18:58.210821+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":630,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:45.177663+0000","last_change":"2024-02-19T21:19:45.177620+0000","last_active":"2024-02-19T21:19:45.177663+0000","last_peered":"2024-02-19T21:19:45.177663+0000","last_clean":"2024-02-19T21:19:45.177663+0000","last_became_active":"2024-02-19T21:18:19.172145+0000","last_became_peered":"2024-02-19T21:18:19.172145+0000","last_unstale":"2024-02-19T21:19:45.177663+0000","last_undegraded":"2024-02-19T21:19:45.177663+0000","last_fullsized":"2024-02-19T21:19:45.177663+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:45.177506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:45.177506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":461,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-02-19T21:19:39.378960+0000","last_change":"2024-02-19T21:19:20.569004+0000","last_active":"2024-02-19T21:19:39.378960+0000","last_peered":"2024-02-19T21:19:39.378960+0000","last_clean":"2024-02-19T21:19:39.378960+0000","last_became_active":"2024-02-19T21:14:16.384069+0000","last_became_peered":"2024-02-19T21:14:16.384069+0000","last_unstale":"2024-02-19T21:19:39.378960+0000","last_undegraded":"2024-02-19T21:19:39.378960+0000","last_fullsized":"2024-02-19T21:19:39.378960+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:20.568921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:20.568921+0000","last_clean_scrub_stamp":"2024-02-19T21:19:20.568921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852033+0000","last_change":"2024-02-19T21:18:51.303957+0000","last_active":"2024-02-19T21:19:40.852033+0000","last_peered":"2024-02-19T21:19:40.852033+0000","last_clean":"2024-02-19T21:19:40.852033+0000","last_became_active":"2024-02-19T21:18:03.731332+0000","last_became_peered":"2024-02-19T21:18:03.731332+0000","last_unstale":"2024-02-19T21:19:40.852033+0000","last_undegraded":"2024-02-19T21:19:40.852033+0000","last_fullsized":"2024-02-19T21:19:40.852033+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:51.303860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:18:51.303860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":270,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:45.500128+0000","last_change":"2024-02-19T21:19:45.500073+0000","last_active":"2024-02-19T21:19:45.500128+0000","last_peered":"2024-02-19T21:19:45.500128+0000","last_clean":"2024-02-19T21:19:45.500128+0000","last_became_active":"2024-02-19T21:17:29.464415+0000","last_became_peered":"2024-02-19T21:17:29.464415+0000","last_unstale":"2024-02-19T21:19:45.500128+0000","last_undegraded":"2024-02-19T21:19:45.500128+0000","last_fullsized":"2024-02-19T21:19:45.500128+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:45.499995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:45.499995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1100,"num_read_kb":857,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":14412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":767,"ondisk_log_size":767,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1107,"num_read_kb":1102,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":2342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":126,"acting":126,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":20,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266916,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307208,"kb_used_data":1792,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437944,"statfs":{"total":95995035648,"available":95680454656,"internally_reserved":0,"allocated":1835008,"data_stored":568707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.50700000000000001,"15min":0.86899999999999999},"min":{"1min":0.29499999999999998,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":0.70999999999999996,"5min":1.069,"15min":89.456999999999994},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54900000000000004,"15min":0.80800000000000005},"min":{"1min":0.39100000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.75600000000000001,"5min":4.9560000000000004,"15min":70.451999999999998},"last":0.501}]},{"osd":1,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.628,"15min":0.73799999999999999},"min":{"1min":0.33900000000000002,"5min":0.33700000000000002,"15min":0.28199999999999997},"max":{"1min":0.71599999999999997,"5min":7.0590000000000002,"15min":34.273000000000003},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51200000000000001,"15min":0.93600000000000005},"min":{"1min":0.34999999999999998,"5min":0.255,"15min":0.254},"max":{"1min":0.73699999999999999,"5min":1.1020000000000001,"15min":89.594999999999999},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56499999999999995,"15min":0.97999999999999998},"min":{"1min":0.31,"5min":0.29799999999999999,"15min":0.27600000000000002},"max":{"1min":0.64800000000000002,"5min":4.9000000000000004,"15min":104.413},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59799999999999998,"15min":0.97999999999999998},"min":{"1min":0.40200000000000002,"5min":0.32100000000000001,"15min":0.27600000000000002},"max":{"1min":0.74399999999999999,"5min":6.9039999999999999,"15min":104.268},"last":0.51300000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.621,"15min":0.998},"min":{"1min":0.38300000000000001,"5min":0.29999999999999999,"15min":0.29099999999999998},"max":{"1min":0.747,"5min":7.984,"15min":105.3},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.63,"15min":0.65700000000000003},"min":{"1min":0.36199999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.67100000000000004,"5min":10.853,"15min":13.243},"last":0.45500000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.496,"15min":0.45500000000000002},"min":{"1min":0.35199999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.78400000000000003,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.375},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58699999999999997,"15min":0.50700000000000001},"min":{"1min":0.376,"5min":0.28699999999999998,"15min":0.26600000000000001},"max":{"1min":0.88900000000000001,"5min":4.6500000000000004,"15min":4.6500000000000004},"last":0.442}]},{"osd":5,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.46600000000000003},"min":{"1min":0.40000000000000002,"5min":0.27900000000000003,"15min":0.252},"max":{"1min":0.81000000000000005,"5min":4.1289999999999996,"15min":4.1289999999999996},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55200000000000005,"15min":0.51100000000000001},"min":{"1min":0.442,"5min":0.30099999999999999,"15min":0.27100000000000002},"max":{"1min":0.85599999999999998,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.41899999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58099999999999996,"15min":0.52900000000000003},"min":{"1min":0.44500000000000001,"5min":0.313,"15min":0.28799999999999998},"max":{"1min":0.84899999999999998,"5min":3.597,"15min":3.597},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58999999999999997,"15min":0.55200000000000005},"min":{"1min":0.501,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.878,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.48599999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528554,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303092,"kb_used_data":1772,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93442060,"statfs":{"total":95995035648,"available":95684669440,"internally_reserved":0,"allocated":1814528,"data_stored":568049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.60999999999999999},"min":{"1min":0.34399999999999997,"5min":0.249,"15min":0.19900000000000001},"max":{"1min":0.71599999999999997,"5min":0.73099999999999998,"15min":17.486999999999998},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.49299999999999999,"15min":0.65700000000000003},"min":{"1min":0.44,"5min":0.26800000000000002,"15min":0.215},"max":{"1min":0.69899999999999995,"5min":0.71899999999999997,"15min":17.43},"last":0.49199999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.498,"15min":0.625},"min":{"1min":0.39100000000000001,"5min":0.248,"15min":0.22800000000000001},"max":{"1min":0.747,"5min":0.747,"15min":14.692},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.49299999999999999,"15min":0.66600000000000004},"min":{"1min":0.38400000000000001,"5min":0.24099999999999999,"15min":0.23400000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":14.795},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Feb 19 21:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.49099999999999999,"15min":0.60299999999999998},"min":{"1min":0.34999999999999998,"5min":0.32600000000000001,"15min":0.253},"max":{"1min":0.65800000000000003,"5min":0.72199999999999998,"15min":6.931},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.498,"15min":0.67600000000000005},"min":{"1min":0.376,"5min":0.33300000000000002,"15min":0.26000000000000001},"max":{"1min":0.68500000000000005,"5min":0.83099999999999996,"15min":14.391},"last":0.55900000000000005}]},{"osd":3,"last update":"Mon Feb 19 21:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.50700000000000001,"15min":0.70299999999999996},"min":{"1min":0.41999999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":14.510999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59399999999999997,"15min":0.69199999999999995},"min":{"1min":0.41399999999999998,"5min":0.33100000000000002,"15min":0.316},"max":{"1min":0.73699999999999999,"5min":4.085,"15min":17.585000000000001},"last":0.61399999999999999}]},{"osd":4,"last update":"Mon Feb 19 21:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.41999999999999998,"15min":0.42599999999999999},"min":{"1min":0.318,"5min":0.24399999999999999,"15min":0.22900000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":2.9289999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.45400000000000001,"15min":0.45800000000000002},"min":{"1min":0.34399999999999997,"5min":0.28799999999999998,"15min":0.22700000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":3.431},"last":0.60499999999999998}]},{"osd":5,"last update":"Mon Feb 19 21:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.48299999999999998,"15min":0.48199999999999998},"min":{"1min":0.41299999999999998,"5min":0.27500000000000002,"15min":0.26300000000000001},"max":{"1min":0.64800000000000002,"5min":1.778,"15min":2.9620000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.26700000000000002},"max":{"1min":0.77900000000000003,"5min":1.788,"15min":2.7010000000000001},"last":0.628}]},{"osd":7,"last update":"Mon Feb 19 21:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.48899999999999999,"15min":0.49199999999999999},"min":{"1min":0.40699999999999997,"5min":0.28000000000000003,"15min":0.26000000000000001},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":1.7170000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.502,"15min":0.495},"min":{"1min":0.432,"5min":0.30299999999999999,"15min":0.28399999999999997},"max":{"1min":0.71099999999999997,"5min":0.72699999999999998,"15min":1.835},"last":0.59299999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575105,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307228,"kb_used_data":1812,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437924,"statfs":{"total":95995035648,"available":95680434176,"internally_reserved":0,"allocated":1855488,"data_stored":568260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:19:30 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.33400000000000002,"15min":0.53400000000000003},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.16700000000000001},"max":{"1min":0.41999999999999998,"5min":1.288,"15min":27.777000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.56999999999999995},"min":{"1min":0.25600000000000001,"5min":0.19900000000000001,"15min":0.19400000000000001},"max":{"1min":0.627,"5min":1.736,"15min":27.631},"last":0.35599999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.371,"15min":0.68300000000000005},"min":{"1min":0.26700000000000002,"5min":0.20000000000000001,"15min":0.19600000000000001},"max":{"1min":0.53700000000000003,"5min":0.97499999999999998,"15min":27.798999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.373,"15min":0.69999999999999996},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.56100000000000005,"5min":1.7729999999999999,"15min":27.606999999999999},"last":0.32600000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.54600000000000004},"min":{"1min":0.29599999999999999,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":0.55400000000000005,"5min":0.75,"15min":15.574999999999999},"last":0.314},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.40600000000000003,"15min":0.64700000000000002},"min":{"1min":0.28799999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.51300000000000001,"5min":0.70499999999999996,"15min":27.82},"last":0.33700000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:19:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54800000000000004,"15min":0.64200000000000002},"min":{"1min":0.38400000000000001,"5min":0.27400000000000002,"15min":0.26800000000000002},"max":{"1min":0.67100000000000004,"5min":1.262,"15min":9.6430000000000007},"last":0.5},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57899999999999996,"15min":0.751},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.308},"max":{"1min":0.73699999999999999,"5min":0.84899999999999998,"15min":27.707000000000001},"last":0.55700000000000005}]},{"osd":5,"last update":"Mon Feb 19 21:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.56899999999999995,"15min":0.73699999999999999},"min":{"1min":0.33800000000000002,"5min":0.315,"15min":0.29999999999999999},"max":{"1min":0.77200000000000002,"5min":1.829,"15min":27.529},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55600000000000005,"15min":0.65800000000000003},"min":{"1min":0.374,"5min":0.35299999999999998,"15min":0.316},"max":{"1min":0.73399999999999999,"5min":1.228,"15min":9.6799999999999997},"last":0.48699999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55300000000000005,"15min":0.64600000000000002},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71699999999999997,"5min":1.147,"15min":9.6989999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59599999999999997,"15min":0.77500000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.35199999999999998},"max":{"1min":0.78400000000000003,"5min":0.876,"15min":27.733000000000001},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Feb 19 21:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.58899999999999997,"15min":0.77100000000000002},"min":{"1min":0.40200000000000002,"5min":0.36799999999999999,"15min":0.36699999999999999},"max":{"1min":0.84399999999999997,"5min":1.8029999999999999,"15min":27.576000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61199999999999999,"15min":0.80300000000000005},"min":{"1min":0.437,"5min":0.38400000000000001,"15min":0.375},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":27.756},"last":0.49099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738631,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299008,"kb_used_data":1784,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93446144,"statfs":{"total":95995035648,"available":95688851456,"internally_reserved":0,"allocated":1826816,"data_stored":567173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.35899999999999999,"15min":0.61099999999999999},"min":{"1min":0.23200000000000001,"5min":0.19400000000000001,"15min":0.188},"max":{"1min":2.6720000000000002,"5min":2.6720000000000002,"15min":20.966999999999999},"last":0.252},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.34399999999999997,"15min":0.63},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.193},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":20.366},"last":0.28199999999999997}]},{"osd":2,"last update":"Mon Feb 19 21:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.35299999999999998,"15min":0.69399999999999995},"min":{"1min":0.22900000000000001,"5min":0.22800000000000001,"15min":0.19900000000000001},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":27.916},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.372,"15min":0.60999999999999999},"min":{"1min":0.25700000000000001,"5min":0.23799999999999999,"15min":0.19600000000000001},"max":{"1min":1.968,"5min":1.968,"15min":22.004000000000001},"last":0.33300000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.40200000000000002,"15min":0.71999999999999997},"min":{"1min":0.251,"5min":0.219,"15min":0.219},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":25.669},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.40699999999999997,"15min":0.66500000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.26300000000000001},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":25.006},"last":0.35599999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.56599999999999995,"15min":0.73799999999999999},"min":{"1min":0.40699999999999997,"5min":0.36099999999999999,"15min":0.28799999999999998},"max":{"1min":2.157,"5min":2.157,"15min":20.411999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.59799999999999998,"15min":0.77000000000000002},"min":{"1min":0.40799999999999997,"5min":0.309,"15min":0.28699999999999998},"max":{"1min":2.2999999999999998,"5min":2.2999999999999998,"15min":19.986999999999998},"last":0.55700000000000005}]},{"osd":5,"last update":"Mon Feb 19 21:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.58799999999999997,"15min":0.80000000000000004},"min":{"1min":0.42399999999999999,"5min":0.36199999999999999,"15min":0.27200000000000002},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":20.021999999999998},"last":0.496},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.61199999999999999,"15min":0.79300000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.30399999999999999},"max":{"1min":2.032,"5min":2.032,"15min":20.908999999999999},"last":0.47599999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58899999999999997,"15min":0.75700000000000001},"min":{"1min":0.41099999999999998,"5min":0.34100000000000003,"15min":0.33900000000000002},"max":{"1min":2.1299999999999999,"5min":2.1299999999999999,"15min":20.445},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.622,"5min":0.61199999999999999,"15min":0.79600000000000004},"min":{"1min":0.42699999999999999,"5min":0.35199999999999998,"15min":0.33400000000000002},"max":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":20.047999999999998},"last":0.46800000000000003}]},{"osd":7,"last update":"Mon Feb 19 21:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.61299999999999999,"15min":0.77900000000000003},"min":{"1min":0.45100000000000001,"5min":0.39900000000000002,"15min":0.312},"max":{"1min":2.004,"5min":2.004,"15min":20.940999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.59599999999999997,"15min":0.80400000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.34899999999999998},"max":{"1min":2.181,"5min":2.181,"15min":20.471},"last":0.51500000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411580,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303092,"kb_used_data":1772,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93442060,"statfs":{"total":95995035648,"available":95684669440,"internally_reserved":0,"allocated":1814528,"data_stored":567419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:19:54 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.66800000000000004,"15min":0.62},"min":{"1min":0.223,"5min":0.183,"15min":0.183},"max":{"1min":0.53100000000000003,"5min":28.702999999999999,"15min":28.702999999999999},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.221,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.45000000000000001,"5min":28.744,"15min":28.744},"last":0.22}]},{"osd":1,"last update":"Mon Feb 19 21:19:54 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.50700000000000001,"15min":0.73399999999999999},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20599999999999999},"max":{"1min":0.47999999999999998,"5min":9.7940000000000005,"15min":28.882000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.41599999999999998,"15min":0.66500000000000004},"min":{"1min":0.26000000000000001,"5min":0.21299999999999999,"15min":0.17999999999999999},"max":{"1min":0.505,"5min":5.0739999999999998,"15min":28.382999999999999},"last":0.26800000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.47399999999999998,"15min":0.71999999999999997},"min":{"1min":0.34399999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.56599999999999995,"5min":4.2270000000000003,"15min":28.742000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.47999999999999998,"15min":0.65200000000000002},"min":{"1min":0.28599999999999998,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":0.60599999999999998,"5min":6.8040000000000003,"15min":26.719999999999999},"last":0.314}]},{"osd":4,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.67000000000000004},"min":{"1min":0.38700000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.84399999999999997,"5min":2.4079999999999999,"15min":24.356000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.67600000000000005,"15min":0.73699999999999999},"min":{"1min":0.47499999999999998,"5min":0.36399999999999999,"15min":0.32800000000000001},"max":{"1min":0.84499999999999997,"5min":4.3540000000000001,"15min":20.707000000000001},"last":0.54200000000000004}]},{"osd":5,"last update":"Mon Feb 19 21:18:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.66000000000000003,"15min":0.68899999999999995},"min":{"1min":0.41499999999999998,"5min":0.35999999999999999,"15min":0.33500000000000002},"max":{"1min":0.84699999999999998,"5min":4.2990000000000004,"15min":9.9499999999999993},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60799999999999998,"15min":0.77800000000000002},"min":{"1min":0.38900000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.82699999999999996,"5min":2.8380000000000001,"15min":26.606000000000002},"last":0.52800000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.60599999999999998,"15min":0.78900000000000003},"min":{"1min":0.41699999999999998,"5min":0.36499999999999999,"15min":0.34300000000000003},"max":{"1min":0.90800000000000003,"5min":2.198,"15min":24.381},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.69699999999999995,"15min":0.75800000000000001},"min":{"1min":0.41399999999999998,"5min":0.36099999999999999,"15min":0.35999999999999999},"max":{"1min":0.86599999999999999,"5min":4.2759999999999998,"15min":28.706},"last":0.52100000000000002}]},{"osd":7,"last update":"Mon Feb 19 21:19:54 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.627,"15min":0.77200000000000002},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.34200000000000003},"max":{"1min":0.87,"5min":2.6440000000000001,"15min":26.658000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61699999999999999,"15min":0.71399999999999997},"min":{"1min":0.42299999999999999,"5min":0.38700000000000001,"15min":0.35399999999999998},"max":{"1min":0.80300000000000005,"5min":2.3700000000000001,"15min":24.402000000000001},"last":0.49099999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182648,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303120,"kb_used_data":1800,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93442032,"statfs":{"total":95995035648,"available":95684640768,"internally_reserved":0,"allocated":1843200,"data_stored":567659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33300000000000002,"15min":0.70899999999999996},"min":{"1min":0.23300000000000001,"5min":0.20300000000000001,"15min":0.156},"max":{"1min":0.51000000000000001,"5min":0.70199999999999996,"15min":77.662000000000006},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.34599999999999997,"15min":0.47999999999999998},"min":{"1min":0.254,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.499,"5min":0.60399999999999998,"15min":17.456},"last":0.41899999999999998}]},{"osd":1,"last update":"Mon Feb 19 21:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.34699999999999998,"15min":0.51500000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.56399999999999995,"5min":0.57999999999999996,"15min":17.327999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.35599999999999998,"15min":0.496},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20000000000000001},"max":{"1min":0.502,"5min":0.98999999999999999,"15min":15.004},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Feb 19 21:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.38500000000000001,"15min":0.65300000000000002},"min":{"1min":0.252,"5min":0.23499999999999999,"15min":0.219},"max":{"1min":0.51900000000000002,"5min":0.71199999999999997,"15min":56.890999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.374,"5min":0.373,"15min":0.63600000000000001},"min":{"1min":0.29399999999999998,"5min":0.24299999999999999,"15min":0.20899999999999999},"max":{"1min":0.52100000000000002,"5min":0.81200000000000006,"15min":44.600999999999999},"last":0.35099999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:19:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54400000000000004,"15min":0.63300000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.31900000000000001},"max":{"1min":0.93700000000000006,"5min":0.94699999999999995,"15min":12.847},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55000000000000004,"15min":0.70099999999999996},"min":{"1min":0.35499999999999998,"5min":0.29899999999999999,"15min":0.29299999999999998},"max":{"1min":0.82999999999999996,"5min":0.89600000000000002,"15min":26.393999999999998},"last":0.55500000000000005}]},{"osd":5,"last update":"Mon Feb 19 21:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.56599999999999995,"15min":0.67800000000000005},"min":{"1min":0.443,"5min":0.34899999999999998,"15min":0.313},"max":{"1min":0.80300000000000005,"5min":1.002,"15min":17.401},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55600000000000005,"15min":0.65300000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.313},"max":{"1min":0.78300000000000003,"5min":0.871,"15min":12.882},"last":0.53100000000000003}]},{"osd":6,"last update":"Mon Feb 19 21:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.56499999999999995,"15min":0.65500000000000003},"min":{"1min":0.41799999999999998,"5min":0.373,"15min":0.34799999999999998},"max":{"1min":0.84299999999999997,"5min":1.036,"15min":17.367000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56599999999999995,"15min":0.65700000000000003},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.33700000000000002},"max":{"1min":0.72399999999999998,"5min":0.876,"15min":12.904},"last":0.498}]},{"osd":7,"last update":"Mon Feb 19 21:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.66100000000000003},"min":{"1min":0.46400000000000002,"5min":0.39700000000000002,"15min":0.35499999999999998},"max":{"1min":0.67900000000000005,"5min":0.91500000000000004,"15min":17.428999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.59799999999999998,"15min":0.81000000000000005},"min":{"1min":0.42099999999999999,"5min":0.40400000000000003,"15min":0.34499999999999997},"max":{"1min":0.67700000000000005,"5min":1.1859999999999999,"15min":44.378},"last":0.52200000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986420,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307204,"kb_used_data":1788,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437948,"statfs":{"total":95995035648,"available":95680458752,"internally_reserved":0,"allocated":1830912,"data_stored":567672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.61099999999999999,"15min":0.66200000000000003},"min":{"1min":0.32100000000000001,"5min":0.27200000000000002,"15min":0.23599999999999999},"max":{"1min":0.75900000000000001,"5min":7.4109999999999996,"15min":19.878},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58399999999999996,"15min":0.88300000000000001},"min":{"1min":0.35799999999999998,"5min":0.249,"15min":0.20499999999999999},"max":{"1min":0.74199999999999999,"5min":3.4079999999999999,"15min":70.063000000000002},"last":0.59399999999999997}]},{"osd":1,"last update":"Mon Feb 19 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.64900000000000002,"15min":0.67200000000000004},"min":{"1min":0.38900000000000001,"5min":0.30599999999999999,"15min":0.19700000000000001},"max":{"1min":0.70599999999999996,"5min":10.086,"15min":26.510999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54100000000000004,"15min":0.69399999999999995},"min":{"1min":0.40999999999999998,"5min":0.29799999999999999,"15min":0.23200000000000001},"max":{"1min":0.72799999999999998,"5min":2.2719999999999998,"15min":25.949000000000002},"last":0.623}]},{"osd":2,"last update":"Mon Feb 19 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.54200000000000004,"15min":0.72699999999999998},"min":{"1min":0.46000000000000002,"5min":0.29499999999999998,"15min":0.26700000000000002},"max":{"1min":0.79000000000000004,"5min":1.7729999999999999,"15min":37.320999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.97599999999999998,"15min":1.2050000000000001},"min":{"1min":0.40400000000000003,"5min":0.309,"15min":0.253},"max":{"1min":0.71799999999999997,"5min":23.504999999999999,"15min":69.936999999999998},"last":0.52500000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.69499999999999995,"15min":0.97199999999999998},"min":{"1min":0.438,"5min":0.33400000000000002,"15min":0.27700000000000002},"max":{"1min":0.70999999999999996,"5min":11.500999999999999,"15min":69.793000000000006},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.71399999999999997,"15min":0.81000000000000005},"min":{"1min":0.46800000000000003,"5min":0.33800000000000002,"15min":0.28299999999999997},"max":{"1min":0.75,"5min":6.1970000000000001,"15min":17.986999999999998},"last":0.55300000000000005}]},{"osd":5,"last update":"Mon Feb 19 21:19:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48199999999999998,"15min":0.443},"min":{"1min":0.32500000000000001,"5min":0.26400000000000001,"15min":0.22900000000000001},"max":{"1min":0.64400000000000002,"5min":4.7439999999999998,"15min":4.7439999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54000000000000004,"15min":0.499},"min":{"1min":0.32900000000000001,"5min":0.30399999999999999,"15min":0.252},"max":{"1min":0.76600000000000001,"5min":4.7130000000000001,"15min":4.7130000000000001},"last":0.42899999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.50900000000000001,"15min":0.51200000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.23499999999999999},"max":{"1min":0.65200000000000002,"5min":3.3140000000000001,"15min":8.0969999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58199999999999996,"15min":0.56299999999999994},"min":{"1min":0.34599999999999997,"5min":0.307,"15min":0.26800000000000002},"max":{"1min":0.69899999999999995,"5min":3.3679999999999999,"15min":4.5609999999999999},"last":0.46899999999999997}]},{"osd":7,"last update":"Mon Feb 19 21:19:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57999999999999996,"15min":0.56100000000000005},"min":{"1min":0.35199999999999998,"5min":0.32100000000000001,"15min":0.27600000000000002},"max":{"1min":0.749,"5min":4.4779999999999998,"15min":6.0730000000000004},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55800000000000005,"15min":0.54600000000000004},"min":{"1min":0.42699999999999999,"5min":0.313,"15min":0.26400000000000001},"max":{"1min":0.79200000000000004,"5min":0.79300000000000004,"15min":3.202},"last":0.60299999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790191,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307192,"kb_used_data":1776,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437960,"statfs":{"total":95995035648,"available":95680471040,"internally_reserved":0,"allocated":1818624,"data_stored":568448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49399999999999999,"15min":1.877},"min":{"1min":0.33200000000000002,"5min":0.26400000000000001,"15min":0.19500000000000001},"max":{"1min":0.67100000000000004,"5min":0.748,"15min":276.47899999999998},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51900000000000002,"15min":0.89200000000000002},"min":{"1min":0.32400000000000001,"5min":0.28999999999999998,"15min":0.20899999999999999},"max":{"1min":0.68500000000000005,"5min":0.99299999999999999,"15min":52.887},"last":0.434}]},{"osd":1,"last update":"Mon Feb 19 21:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.50700000000000001,"15min":1.2609999999999999},"min":{"1min":0.374,"5min":0.27100000000000002,"15min":0.23499999999999999},"max":{"1min":0.629,"5min":0.75,"15min":120.011},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52700000000000002,"15min":1.016},"min":{"1min":0.314,"5min":0.314,"15min":0.26200000000000001},"max":{"1min":0.63900000000000001,"5min":0.89400000000000002,"15min":118.105},"last":0.58199999999999996}]},{"osd":2,"last update":"Mon Feb 19 21:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51700000000000002,"15min":0.98799999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.27200000000000002},"max":{"1min":0.628,"5min":0.73099999999999998,"15min":82.242000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52800000000000002,"15min":1.512},"min":{"1min":0.40699999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66700000000000004,"5min":0.79000000000000004,"15min":232.84999999999999},"last":0.55800000000000005}]},{"osd":3,"last update":"Mon Feb 19 21:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53600000000000003,"15min":1.069},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.27900000000000003},"max":{"1min":0.70599999999999996,"5min":0.80600000000000005,"15min":109.70099999999999},"last":0.499},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55200000000000005,"15min":0.90900000000000003},"min":{"1min":0.40300000000000002,"5min":0.35499999999999998,"15min":0.32000000000000001},"max":{"1min":0.67700000000000005,"5min":0.70899999999999996,"15min":65.519000000000005},"last":0.51400000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.45900000000000002,"15min":0.442},"min":{"1min":0.35799999999999998,"5min":0.253,"15min":0.23300000000000001},"max":{"1min":0.61599999999999999,"5min":0.77200000000000002,"15min":1.861},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.48399999999999999,"15min":0.46200000000000002},"min":{"1min":0.373,"5min":0.27300000000000002,"15min":0.248},"max":{"1min":0.64800000000000002,"5min":0.72999999999999998,"15min":1.9730000000000001},"last":0.64900000000000002}]},{"osd":6,"last update":"Mon Feb 19 21:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47099999999999997,"15min":0.46800000000000003},"min":{"1min":0.28299999999999997,"5min":0.246,"15min":0.246},"max":{"1min":0.65000000000000002,"5min":0.95099999999999996,"15min":1.823},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.497,"15min":0.47999999999999998},"min":{"1min":0.29399999999999998,"5min":0.28799999999999998,"15min":0.27100000000000002},"max":{"1min":0.73399999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.59499999999999997}]},{"osd":7,"last update":"Mon Feb 19 21:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.51000000000000001},"min":{"1min":0.40500000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.73899999999999999,"5min":1.052,"15min":1.8879999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54100000000000004,"15min":0.51700000000000002},"min":{"1min":0.42499999999999999,"5min":0.34300000000000003,"15min":0.27800000000000002},"max":{"1min":0.73599999999999999,"5min":0.80500000000000005,"15min":0.96799999999999997},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:20:05.729 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T21:20:05.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:04.173803+0000 osd.2 (osd.2) 398 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:20:06.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:04.173984+0000 osd.2 (osd.2) 399 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:20:06.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:04.637238+0000 osd.4 (osd.4) 367 : cluster [DBG] 4.c deep-scrub starts 2024-02-19T21:20:06.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:04.638732+0000 osd.4 (osd.4) 368 : cluster [DBG] 4.c deep-scrub ok 2024-02-19T21:20:06.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:20:04.930707+0000 mgr.y (mgr.14550) 748 : audit [DBG] from='client.24186 v1:172.21.15.163:0/2639476106' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:20:06.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:04.940294+0000 mon.a (mon.0) 3885 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-02-19T21:20:06.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:04.173803+0000 osd.2 (osd.2) 398 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:20:06.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:04.173984+0000 osd.2 (osd.2) 399 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:20:06.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:04.637238+0000 osd.4 (osd.4) 367 : cluster [DBG] 4.c deep-scrub starts 2024-02-19T21:20:06.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:04.638732+0000 osd.4 (osd.4) 368 : cluster [DBG] 4.c deep-scrub ok 2024-02-19T21:20:06.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:20:04.930707+0000 mgr.y (mgr.14550) 748 : audit [DBG] from='client.24186 v1:172.21.15.163:0/2639476106' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:20:06.000 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:05 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:04.940294+0000 mon.a (mon.0) 3885 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-02-19T21:20:06.004 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:20:06.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:04.173803+0000 osd.2 (osd.2) 398 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:20:06.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:04.173984+0000 osd.2 (osd.2) 399 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:20:06.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:04.637238+0000 osd.4 (osd.4) 367 : cluster [DBG] 4.c deep-scrub starts 2024-02-19T21:20:06.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:04.638732+0000 osd.4 (osd.4) 368 : cluster [DBG] 4.c deep-scrub ok 2024-02-19T21:20:06.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:20:04.930707+0000 mgr.y (mgr.14550) 748 : audit [DBG] from='client.24186 v1:172.21.15.163:0/2639476106' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:20:06.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:04.940294+0000 mon.a (mon.0) 3885 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-02-19T21:20:07.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:05.241157+0000 osd.2 (osd.2) 400 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:20:07.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:05.244027+0000 osd.2 (osd.2) 401 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:20:07.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:05.383093+0000 mgr.y (mgr.14550) 749 : cluster [DBG] pgmap v1339: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:20:07.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:05.590702+0000 osd.4 (osd.4) 369 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:20:07.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:05.592425+0000 osd.4 (osd.4) 370 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:20:07.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:06.107890+0000 osd.0 (osd.0) 202 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:20:07.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:06.108039+0000 osd.0 (osd.0) 203 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:20:07.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:05.241157+0000 osd.2 (osd.2) 400 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:20:07.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:05.244027+0000 osd.2 (osd.2) 401 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:20:07.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:05.383093+0000 mgr.y (mgr.14550) 749 : cluster [DBG] pgmap v1339: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:20:07.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:05.590702+0000 osd.4 (osd.4) 369 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:20:07.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:05.592425+0000 osd.4 (osd.4) 370 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:20:07.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:06.107890+0000 osd.0 (osd.0) 202 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:20:07.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:06 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:06.108039+0000 osd.0 (osd.0) 203 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:20:07.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:05.241157+0000 osd.2 (osd.2) 400 : cluster [DBG] 16.10 scrub starts 2024-02-19T21:20:07.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:05.244027+0000 osd.2 (osd.2) 401 : cluster [DBG] 16.10 scrub ok 2024-02-19T21:20:07.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:05.383093+0000 mgr.y (mgr.14550) 749 : cluster [DBG] pgmap v1339: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-19T21:20:07.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:05.590702+0000 osd.4 (osd.4) 369 : cluster [DBG] 4.1a scrub starts 2024-02-19T21:20:07.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:05.592425+0000 osd.4 (osd.4) 370 : cluster [DBG] 4.1a scrub ok 2024-02-19T21:20:07.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:06.107890+0000 osd.0 (osd.0) 202 : cluster [DBG] purged_snaps scrub starts 2024-02-19T21:20:07.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:06.108039+0000 osd.0 (osd.0) 203 : cluster [DBG] purged_snaps scrub ok 2024-02-19T21:20:07.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.812+0000 7fe81a8cb700 1 -- 172.21.15.163:0/3311699142 >> v1:172.21.15.163:6789/0 conn(0x7fe8141042c0 legacy=0x7fe814106740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:07.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.812+0000 7fe81a8cb700 1 -- 172.21.15.163:0/3311699142 shutdown_connections 2024-02-19T21:20:07.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.812+0000 7fe81a8cb700 1 -- 172.21.15.163:0/3311699142 >> 172.21.15.163:0/3311699142 conn(0x7fe8140f35a0 msgr2=0x7fe8140f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:20:07.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.812+0000 7fe81a8cb700 1 -- 172.21.15.163:0/3311699142 shutdown_connections 2024-02-19T21:20:07.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.812+0000 7fe81a8cb700 1 -- 172.21.15.163:0/3311699142 wait complete. 2024-02-19T21:20:07.817 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.813+0000 7fe81a8cb700 1 librados: starting msgr at 2024-02-19T21:20:07.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.813+0000 7fe81a8cb700 1 librados: starting objecter 2024-02-19T21:20:07.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.813+0000 7fe81a8cb700 1 Processor -- start 2024-02-19T21:20:07.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.813+0000 7fe81a8cb700 1 -- start start 2024-02-19T21:20:07.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.813+0000 7fe81a8cb700 1 librados: setting wanted keys 2024-02-19T21:20:07.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.813+0000 7fe81a8cb700 1 librados: calling monclient init 2024-02-19T21:20:07.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe81a8cb700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe814119520 con 0x7fe814100780 2024-02-19T21:20:07.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe81a8cb700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe81411a700 con 0x7fe8140fd980 2024-02-19T21:20:07.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe81a8cb700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe81411b900 con 0x7fe8141042c0 2024-02-19T21:20:07.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe818e68700 1 --1- >> v1:172.21.15.163:6790/0 conn(0x7fe8141042c0 0x7fe814117d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6790/0 says I am v1:172.21.15.163:58986/0 (socket says 172.21.15.163:58986) 2024-02-19T21:20:07.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe818e68700 1 -- 172.21.15.163:0/1916127829 learned_addr learned my addr 172.21.15.163:0/1916127829 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:20:07.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 10 client.?.objecter ms_handle_connect 0x7fe8141042c0 2024-02-19T21:20:07.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:20:07.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 10 client.?.objecter ms_handle_connect 0x7fe814100780 2024-02-19T21:20:07.818 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:20:07.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 10 client.?.objecter ms_handle_connect 0x7fe8140fd980 2024-02-19T21:20:07.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 10 client.?.objecter resend_mon_ops 2024-02-19T21:20:07.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229420910 0 0) 0x7fe814119520 con 0x7fe814100780 2024-02-19T21:20:07.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7fc003030 con 0x7fe814100780 2024-02-19T21:20:07.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011138206 0 0) 0x7fe81411b900 con 0x7fe8141042c0 2024-02-19T21:20:07.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe814119520 con 0x7fe8141042c0 2024-02-19T21:20:07.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668354124 0 0) 0x7fe81411a700 con 0x7fe8140fd980 2024-02-19T21:20:07.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe81411b900 con 0x7fe8140fd980 2024-02-19T21:20:07.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 160400367 0 0) 0x7fe7fc003030 con 0x7fe814100780 2024-02-19T21:20:07.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe81411a700 con 0x7fe814100780 2024-02-19T21:20:07.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe808001880 con 0x7fe814100780 2024-02-19T21:20:07.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 902208681 0 0) 0x7fe814119520 con 0x7fe8141042c0 2024-02-19T21:20:07.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7fc003030 con 0x7fe8141042c0 2024-02-19T21:20:07.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe80c002250 con 0x7fe8141042c0 2024-02-19T21:20:07.819 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2565956090 0 0) 0x7fe81411b900 con 0x7fe8140fd980 2024-02-19T21:20:07.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe814119520 con 0x7fe8140fd980 2024-02-19T21:20:07.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe8040015d0 con 0x7fe8140fd980 2024-02-19T21:20:07.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3400278044 0 0) 0x7fe81411a700 con 0x7fe814100780 2024-02-19T21:20:07.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 >> v1:172.21.15.163:6790/0 conn(0x7fe8141042c0 legacy=0x7fe814117d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:07.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 >> v1:172.21.15.193:6789/0 conn(0x7fe8140fd980 legacy=0x7fe814102450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:07.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81411cb00 con 0x7fe814100780 2024-02-19T21:20:07.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe81a8cb700 1 -- 172.21.15.163:0/1916127829 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe814119710 con 0x7fe814100780 2024-02-19T21:20:07.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe81a8cb700 10 client.24192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:20:07.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe81a8cb700 1 -- 172.21.15.163:0/1916127829 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe814119c00 con 0x7fe814100780 2024-02-19T21:20:07.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.814+0000 7fe81a8cb700 1 librados: init done 2024-02-19T21:20:07.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.815+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fe808002010 con 0x7fe814100780 2024-02-19T21:20:07.820 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.815+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fe808003310 con 0x7fe814100780 2024-02-19T21:20:07.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.815+0000 7fe81a8cb700 1 -- 172.21.15.163:0/1916127829 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe800004f20 con 0x7fe814100780 2024-02-19T21:20:07.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.816+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fe808001b70 con 0x7fe814100780 2024-02-19T21:20:07.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.816+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(856..856 src has 796..856) v4 ==== 7419+0+0 (unknown 4192420904 0 0) 0x7fe808076290 con 0x7fe814100780 2024-02-19T21:20:07.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.816+0000 7fe8117fa700 10 client.24192.objecter ms_dispatch 0x7fe814001550 osd_map(856..856 src has 796..856) v4 2024-02-19T21:20:07.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.816+0000 7fe8117fa700 3 client.24192.objecter handle_osd_map got epochs [856,856] > 0 2024-02-19T21:20:07.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.816+0000 7fe8117fa700 3 client.24192.objecter handle_osd_map decoding full epoch 856 2024-02-19T21:20:07.821 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.816+0000 7fe8117fa700 20 client.24192.objecter dump_active .. 0 homeless 2024-02-19T21:20:07.822 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.816+0000 7fe8117fa700 10 client.24192.objecter ms_handle_connect 0x7fe7fc063d80 2024-02-19T21:20:07.822 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.820+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fe808049960 con 0x7fe814100780 2024-02-19T21:20:07.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:06.437358+0000 osd.3 (osd.3) 435 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:20:07.954 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:06.438838+0000 osd.3 (osd.3) 436 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:06.761937+0000 osd.1 (osd.1) 439 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:06.763559+0000 osd.1 (osd.1) 440 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:06.949856+0000 mon.a (mon.0) 3886 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-02-19T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:07.282130+0000 osd.0 (osd.0) 204 : cluster [DBG] 16.0 deep-scrub starts 2024-02-19T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:07.285173+0000 osd.0 (osd.0) 205 : cluster [DBG] 16.0 deep-scrub ok 2024-02-19T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:07.661984+0000 osd.5 (osd.5) 392 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:07.665655+0000 osd.5 (osd.5) 393 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:20:07.955 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:06.437358+0000 osd.3 (osd.3) 435 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:20:07.955 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:06.438838+0000 osd.3 (osd.3) 436 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:20:07.955 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:06.761937+0000 osd.1 (osd.1) 439 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:20:07.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:06.763559+0000 osd.1 (osd.1) 440 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:20:07.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:06.949856+0000 mon.a (mon.0) 3886 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-02-19T21:20:07.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:07.282130+0000 osd.0 (osd.0) 204 : cluster [DBG] 16.0 deep-scrub starts 2024-02-19T21:20:07.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:07.285173+0000 osd.0 (osd.0) 205 : cluster [DBG] 16.0 deep-scrub ok 2024-02-19T21:20:07.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:07.661984+0000 osd.5 (osd.5) 392 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:20:07.956 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:07 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:07.665655+0000 osd.5 (osd.5) 393 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:20:07.986 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.984+0000 7fe81a8cb700 1 -- 172.21.15.163:0/1916127829 --> v1:172.21.15.163:6800/1552514681 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe800000b90 con 0x7fe7fc063d80 2024-02-19T21:20:07.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.996+0000 7fe8117fa700 1 -- 172.21.15.163:0/1916127829 <== mgr.14550 v1:172.21.15.163:6800/1552514681 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+454408 (unknown 2965378022 0 2636700156) 0x7fe800000b90 con 0x7fe7fc063d80 2024-02-19T21:20:08.004 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-02-19T21:20:08.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.999+0000 7fe81a8cb700 10 librados: watch_flush enter 2024-02-19T21:20:08.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.999+0000 7fe81a8cb700 10 librados: watch_flush exit 2024-02-19T21:20:08.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.999+0000 7fe81a8cb700 20 client.24192.objecter shutdown clearing up homeless session... 2024-02-19T21:20:08.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.999+0000 7fe81a8cb700 10 client.24192.objecter successfully canceled tick 2024-02-19T21:20:08.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.999+0000 7fe81a8cb700 1 -- 172.21.15.163:0/1916127829 >> v1:172.21.15.163:6800/1552514681 conn(0x7fe7fc063d80 legacy=0x7fe7fc066240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:08.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.999+0000 7fe81a8cb700 1 -- 172.21.15.163:0/1916127829 >> v1:172.21.15.163:6789/0 conn(0x7fe814100780 legacy=0x7fe814102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:08.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.999+0000 7fe81a8cb700 1 -- 172.21.15.163:0/1916127829 shutdown_connections 2024-02-19T21:20:08.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.999+0000 7fe81a8cb700 1 -- 172.21.15.163:0/1916127829 >> 172.21.15.163:0/1916127829 conn(0x7fe8140f35a0 msgr2=0x7fe814107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:20:08.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.999+0000 7fe81a8cb700 1 -- 172.21.15.163:0/1916127829 shutdown_connections 2024-02-19T21:20:08.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:07.999+0000 7fe81a8cb700 1 -- 172.21.15.163:0/1916127829 wait complete. 2024-02-19T21:20:08.009 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:08.000+0000 7fe81a8cb700 1 librados: shutdown 2024-02-19T21:20:08.009 INFO:teuthology.orchestra.run.smithi163.stderr:dumped all 2024-02-19T21:20:08.073 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:20:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x[40786]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:20:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-19T21:20:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:06.437358+0000 osd.3 (osd.3) 435 : cluster [DBG] 4.17 scrub starts 2024-02-19T21:20:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:06.438838+0000 osd.3 (osd.3) 436 : cluster [DBG] 4.17 scrub ok 2024-02-19T21:20:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:06.761937+0000 osd.1 (osd.1) 439 : cluster [DBG] 4.4 scrub starts 2024-02-19T21:20:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:06.763559+0000 osd.1 (osd.1) 440 : cluster [DBG] 4.4 scrub ok 2024-02-19T21:20:08.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:06.949856+0000 mon.a (mon.0) 3886 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-02-19T21:20:08.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:07.282130+0000 osd.0 (osd.0) 204 : cluster [DBG] 16.0 deep-scrub starts 2024-02-19T21:20:08.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:07.285173+0000 osd.0 (osd.0) 205 : cluster [DBG] 16.0 deep-scrub ok 2024-02-19T21:20:08.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:07.661984+0000 osd.5 (osd.5) 392 : cluster [DBG] 4.12 scrub starts 2024-02-19T21:20:08.074 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:07.665655+0000 osd.5 (osd.5) 393 : cluster [DBG] 4.12 scrub ok 2024-02-19T21:20:08.725 INFO:teuthology.orchestra.run.smithi163.stdout:{"pg_ready":true,"pg_map":{"version":1341,"stamp":"2024-02-19T21:20:07.383226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5801,"num_objects":193,"num_object_clones":0,"num_object_copies":579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":2234,"num_read_kb":1986,"num_write":720,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":775,"num_bytes_recovered":19845,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":908,"ondisk_log_size":908,"up":522,"acting":522,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":522,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2437240,"kb_used_data":14392,"kb_used_omap":0,"kb_used_meta":2422784,"kb_avail":747523976,"statfs":{"total":767960285184,"available":765464551424,"internally_reserved":0,"allocated":14737408,"data_stored":4604675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2480930816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.462651"},"pg_stats":[{"pgid":"2.29","version":"0'0","reported_seq":292,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:01.524122+0000","last_change":"2024-02-19T21:20:01.524122+0000","last_active":"2024-02-19T21:20:01.524122+0000","last_peered":"2024-02-19T21:20:01.524122+0000","last_clean":"2024-02-19T21:20:01.524122+0000","last_became_active":"2024-02-19T21:18:03.725668+0000","last_became_peered":"2024-02-19T21:18:03.725668+0000","last_unstale":"2024-02-19T21:20:01.524122+0000","last_undegraded":"2024-02-19T21:20:01.524122+0000","last_fullsized":"2024-02-19T21:20:01.524122+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:20:01.524003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:20:01.524003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.28","version":"0'0","reported_seq":429,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852374+0000","last_change":"2024-02-19T21:18:59.213488+0000","last_active":"2024-02-19T21:19:40.852374+0000","last_peered":"2024-02-19T21:19:40.852374+0000","last_clean":"2024-02-19T21:19:40.852374+0000","last_became_active":"2024-02-19T21:18:03.741123+0000","last_became_peered":"2024-02-19T21:18:03.741123+0000","last_unstale":"2024-02-19T21:19:40.852374+0000","last_undegraded":"2024-02-19T21:19:40.852374+0000","last_fullsized":"2024-02-19T21:19:40.852374+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:59.213418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:17:04.962987+0000","last_clean_scrub_stamp":"2024-02-19T21:18:59.213418+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.25","version":"0'0","reported_seq":930,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.482644+0000","last_change":"2024-02-19T21:19:10.500294+0000","last_active":"2024-02-19T21:20:00.482644+0000","last_peered":"2024-02-19T21:20:00.482644+0000","last_clean":"2024-02-19T21:20:00.482644+0000","last_became_active":"2024-02-19T21:17:58.673707+0000","last_became_peered":"2024-02-19T21:17:58.673707+0000","last_unstale":"2024-02-19T21:20:00.482644+0000","last_undegraded":"2024-02-19T21:20:00.482644+0000","last_fullsized":"2024-02-19T21:20:00.482644+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:10.500215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:10.500215+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.24","version":"0'0","reported_seq":270,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:03.541067+0000","last_change":"2024-02-19T21:20:03.541067+0000","last_active":"2024-02-19T21:20:03.541067+0000","last_peered":"2024-02-19T21:20:03.541067+0000","last_clean":"2024-02-19T21:20:03.541067+0000","last_became_active":"2024-02-19T21:18:03.725448+0000","last_became_peered":"2024-02-19T21:18:03.725448+0000","last_unstale":"2024-02-19T21:20:03.541067+0000","last_undegraded":"2024-02-19T21:20:03.541067+0000","last_fullsized":"2024-02-19T21:20:03.541067+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:20:03.540988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:20:03.540988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.27","version":"0'0","reported_seq":608,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969424+0000","last_change":"2024-02-19T21:19:21.592636+0000","last_active":"2024-02-19T21:19:40.969424+0000","last_peered":"2024-02-19T21:19:40.969424+0000","last_clean":"2024-02-19T21:19:40.969424+0000","last_became_active":"2024-02-19T21:19:10.740468+0000","last_became_peered":"2024-02-19T21:19:10.740468+0000","last_unstale":"2024-02-19T21:19:40.969424+0000","last_undegraded":"2024-02-19T21:19:40.969424+0000","last_fullsized":"2024-02-19T21:19:40.969424+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:21.592563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:21.592563+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.26","version":"0'0","reported_seq":608,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:01.339371+0000","last_change":"2024-02-19T21:20:01.339371+0000","last_active":"2024-02-19T21:20:01.339371+0000","last_peered":"2024-02-19T21:20:01.339371+0000","last_clean":"2024-02-19T21:20:01.339371+0000","last_became_active":"2024-02-19T21:19:10.741937+0000","last_became_peered":"2024-02-19T21:19:10.741937+0000","last_unstale":"2024-02-19T21:20:01.339371+0000","last_undegraded":"2024-02-19T21:20:01.339371+0000","last_fullsized":"2024-02-19T21:20:01.339371+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:20:01.339299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:47.416506+0000","last_clean_scrub_stamp":"2024-02-19T21:20:01.339299+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.21","version":"0'0","reported_seq":98,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.228770+0000","last_change":"2024-02-19T21:19:25.999950+0000","last_active":"2024-02-19T21:20:00.228770+0000","last_peered":"2024-02-19T21:20:00.228770+0000","last_clean":"2024-02-19T21:20:00.228770+0000","last_became_active":"2024-02-19T21:18:03.741741+0000","last_became_peered":"2024-02-19T21:18:03.741741+0000","last_unstale":"2024-02-19T21:20:00.228770+0000","last_undegraded":"2024-02-19T21:20:00.228770+0000","last_fullsized":"2024-02-19T21:20:00.228770+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:25.999844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:25.999844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.20","version":"397'2","reported_seq":649,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967232+0000","last_change":"2024-02-19T21:19:24.659934+0000","last_active":"2024-02-19T21:19:40.967232+0000","last_peered":"2024-02-19T21:19:40.967232+0000","last_clean":"2024-02-19T21:19:40.967232+0000","last_became_active":"2024-02-19T21:19:10.741757+0000","last_became_peered":"2024-02-19T21:19:10.741757+0000","last_unstale":"2024-02-19T21:19:40.967232+0000","last_undegraded":"2024-02-19T21:19:40.967232+0000","last_fullsized":"2024-02-19T21:19:40.967232+0000","mapping_epoch":848,"log_start":"58'1","ondisk_log_start":"58'1","created":474,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":6,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:19:24.659860+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:19:24.659860+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.23","version":"0'0","reported_seq":420,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:47.580055+0000","last_change":"2024-02-19T21:19:47.580032+0000","last_active":"2024-02-19T21:19:47.580055+0000","last_peered":"2024-02-19T21:19:47.580055+0000","last_clean":"2024-02-19T21:19:47.580055+0000","last_became_active":"2024-02-19T21:14:16.384406+0000","last_became_peered":"2024-02-19T21:14:16.384406+0000","last_unstale":"2024-02-19T21:19:47.580055+0000","last_undegraded":"2024-02-19T21:19:47.580055+0000","last_fullsized":"2024-02-19T21:19:47.580055+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:47.579967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:47.579967+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.22","version":"0'0","reported_seq":252,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:52.569047+0000","last_change":"2024-02-19T21:19:52.569011+0000","last_active":"2024-02-19T21:19:52.569047+0000","last_peered":"2024-02-19T21:19:52.569047+0000","last_clean":"2024-02-19T21:19:52.569047+0000","last_became_active":"2024-02-19T21:14:16.382869+0000","last_became_peered":"2024-02-19T21:14:16.382869+0000","last_unstale":"2024-02-19T21:19:52.569047+0000","last_undegraded":"2024-02-19T21:19:52.569047+0000","last_fullsized":"2024-02-19T21:19:52.569047+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":474,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:52.568919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:51.187606+0000","last_clean_scrub_stamp":"2024-02-19T21:19:52.568919+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"849'18","reported_seq":688,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:19:59.921900+0000","last_change":"2024-02-19T21:19:18.564012+0000","last_active":"2024-02-19T21:19:59.921900+0000","last_peered":"2024-02-19T21:19:59.921900+0000","last_clean":"2024-02-19T21:19:59.921900+0000","last_became_active":"2024-02-19T21:19:10.741991+0000","last_became_peered":"2024-02-19T21:19:10.741991+0000","last_unstale":"2024-02-19T21:19:59.921900+0000","last_undegraded":"2024-02-19T21:19:59.921900+0000","last_fullsized":"2024-02-19T21:19:59.921900+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"849'18","last_scrub_stamp":"2024-02-19T21:19:18.563931+0000","last_deep_scrub":"849'18","last_deep_scrub_stamp":"2024-02-19T21:19:18.563931+0000","last_clean_scrub_stamp":"2024-02-19T21:19:18.563931+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"473'28","reported_seq":100,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969867+0000","last_change":"2024-02-19T21:19:15.626733+0000","last_active":"2024-02-19T21:19:40.969867+0000","last_peered":"2024-02-19T21:19:40.969867+0000","last_clean":"2024-02-19T21:19:40.969867+0000","last_became_active":"2024-02-19T21:18:03.734723+0000","last_became_peered":"2024-02-19T21:18:03.734723+0000","last_unstale":"2024-02-19T21:19:40.969867+0000","last_undegraded":"2024-02-19T21:19:40.969867+0000","last_fullsized":"2024-02-19T21:19:40.969867+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"473'28","last_scrub_stamp":"2024-02-19T21:19:15.626643+0000","last_deep_scrub":"473'28","last_deep_scrub_stamp":"2024-02-19T21:11:47.051823+0000","last_clean_scrub_stamp":"2024-02-19T21:19:15.626643+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":98,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.229577+0000","last_change":"2024-02-19T21:19:13.916725+0000","last_active":"2024-02-19T21:20:00.229577+0000","last_peered":"2024-02-19T21:20:00.229577+0000","last_clean":"2024-02-19T21:20:00.229577+0000","last_became_active":"2024-02-19T21:18:03.742827+0000","last_became_peered":"2024-02-19T21:18:03.742827+0000","last_unstale":"2024-02-19T21:20:00.229577+0000","last_undegraded":"2024-02-19T21:20:00.229577+0000","last_fullsized":"2024-02-19T21:20:00.229577+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:13.916659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:13.916659+0000","last_clean_scrub_stamp":"2024-02-19T21:19:13.916659+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":59,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.970395+0000","last_change":"2024-02-19T21:19:18.647920+0000","last_active":"2024-02-19T21:19:40.970395+0000","last_peered":"2024-02-19T21:19:40.970395+0000","last_clean":"2024-02-19T21:19:40.970395+0000","last_became_active":"2024-02-19T21:17:58.690808+0000","last_became_peered":"2024-02-19T21:17:58.690808+0000","last_unstale":"2024-02-19T21:19:40.970395+0000","last_undegraded":"2024-02-19T21:19:40.970395+0000","last_fullsized":"2024-02-19T21:19:40.970395+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:18.647844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:18.647844+0000","last_clean_scrub_stamp":"2024-02-19T21:19:18.647844+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":98,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.229587+0000","last_change":"2024-02-19T21:18:58.953026+0000","last_active":"2024-02-19T21:20:00.229587+0000","last_peered":"2024-02-19T21:20:00.229587+0000","last_clean":"2024-02-19T21:20:00.229587+0000","last_became_active":"2024-02-19T21:17:58.688372+0000","last_became_peered":"2024-02-19T21:17:58.688372+0000","last_unstale":"2024-02-19T21:20:00.229587+0000","last_undegraded":"2024-02-19T21:20:00.229587+0000","last_fullsized":"2024-02-19T21:20:00.229587+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:58.952929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:18:58.952929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"806'19","reported_seq":782,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967626+0000","last_change":"2024-02-19T21:19:38.323704+0000","last_active":"2024-02-19T21:19:40.967626+0000","last_peered":"2024-02-19T21:19:40.967626+0000","last_clean":"2024-02-19T21:19:40.967626+0000","last_became_active":"2024-02-19T21:18:41.545335+0000","last_became_peered":"2024-02-19T21:18:41.545335+0000","last_unstale":"2024-02-19T21:19:40.967626+0000","last_undegraded":"2024-02-19T21:19:40.967626+0000","last_fullsized":"2024-02-19T21:19:40.967626+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"806'19","last_scrub_stamp":"2024-02-19T21:19:38.323637+0000","last_deep_scrub":"806'19","last_deep_scrub_stamp":"2024-02-19T21:17:51.331019+0000","last_clean_scrub_stamp":"2024-02-19T21:19:38.323637+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":868,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852424+0000","last_change":"2024-02-19T21:19:21.683567+0000","last_active":"2024-02-19T21:19:40.852424+0000","last_peered":"2024-02-19T21:19:40.852424+0000","last_clean":"2024-02-19T21:19:40.852424+0000","last_became_active":"2024-02-19T21:10:41.312116+0000","last_became_peered":"2024-02-19T21:10:41.312116+0000","last_unstale":"2024-02-19T21:19:40.852424+0000","last_undegraded":"2024-02-19T21:19:40.852424+0000","last_fullsized":"2024-02-19T21:19:40.852424+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:21.683502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:21.683502+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":89,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.228870+0000","last_change":"2024-02-19T21:19:16.996763+0000","last_active":"2024-02-19T21:20:00.228870+0000","last_peered":"2024-02-19T21:20:00.228870+0000","last_clean":"2024-02-19T21:20:00.228870+0000","last_became_active":"2024-02-19T21:17:29.479730+0000","last_became_peered":"2024-02-19T21:17:29.479730+0000","last_unstale":"2024-02-19T21:20:00.228870+0000","last_undegraded":"2024-02-19T21:20:00.228870+0000","last_fullsized":"2024-02-19T21:20:00.228870+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:16.996698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:16.996698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":487,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967097+0000","last_change":"2024-02-19T21:19:10.266742+0000","last_active":"2024-02-19T21:19:40.967097+0000","last_peered":"2024-02-19T21:19:40.967097+0000","last_clean":"2024-02-19T21:19:40.967097+0000","last_became_active":"2024-02-19T21:18:19.172204+0000","last_became_peered":"2024-02-19T21:18:19.172204+0000","last_unstale":"2024-02-19T21:19:40.967097+0000","last_undegraded":"2024-02-19T21:19:40.967097+0000","last_fullsized":"2024-02-19T21:19:40.967097+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:10.266666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:10.266666+0000","last_clean_scrub_stamp":"2024-02-19T21:19:10.266666+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":68,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969291+0000","last_change":"2024-02-19T21:18:52.540347+0000","last_active":"2024-02-19T21:19:40.969291+0000","last_peered":"2024-02-19T21:19:40.969291+0000","last_clean":"2024-02-19T21:19:40.969291+0000","last_became_active":"2024-02-19T21:18:03.735016+0000","last_became_peered":"2024-02-19T21:18:03.735016+0000","last_unstale":"2024-02-19T21:19:40.969291+0000","last_undegraded":"2024-02-19T21:19:40.969291+0000","last_fullsized":"2024-02-19T21:19:40.969291+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:52.540277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:18:52.540277+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"837'44","reported_seq":373,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:50.575040+0000","last_change":"2024-02-19T21:19:50.575003+0000","last_active":"2024-02-19T21:19:50.575040+0000","last_peered":"2024-02-19T21:19:50.575040+0000","last_clean":"2024-02-19T21:19:50.575040+0000","last_became_active":"2024-02-19T21:18:03.731077+0000","last_became_peered":"2024-02-19T21:18:03.731077+0000","last_unstale":"2024-02-19T21:19:50.575040+0000","last_undegraded":"2024-02-19T21:19:50.575040+0000","last_fullsized":"2024-02-19T21:19:50.575040+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"837'44","last_scrub_stamp":"2024-02-19T21:19:50.574920+0000","last_deep_scrub":"837'44","last_deep_scrub_stamp":"2024-02-19T21:18:47.751140+0000","last_clean_scrub_stamp":"2024-02-19T21:19:50.574920+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":93,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.229343+0000","last_change":"2024-02-19T21:18:36.724586+0000","last_active":"2024-02-19T21:20:00.229343+0000","last_peered":"2024-02-19T21:20:00.229343+0000","last_clean":"2024-02-19T21:20:00.229343+0000","last_became_active":"2024-02-19T21:17:58.688829+0000","last_became_peered":"2024-02-19T21:17:58.688829+0000","last_unstale":"2024-02-19T21:20:00.229343+0000","last_undegraded":"2024-02-19T21:20:00.229343+0000","last_fullsized":"2024-02-19T21:20:00.229343+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:36.724491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:49.051732+0000","last_clean_scrub_stamp":"2024-02-19T21:18:36.724491+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":98,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.228935+0000","last_change":"2024-02-19T21:18:39.670892+0000","last_active":"2024-02-19T21:20:00.228935+0000","last_peered":"2024-02-19T21:20:00.228935+0000","last_clean":"2024-02-19T21:20:00.228935+0000","last_became_active":"2024-02-19T21:17:58.680830+0000","last_became_peered":"2024-02-19T21:17:58.680830+0000","last_unstale":"2024-02-19T21:20:00.228935+0000","last_undegraded":"2024-02-19T21:20:00.228935+0000","last_fullsized":"2024-02-19T21:20:00.228935+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:39.670792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:39.670792+0000","last_clean_scrub_stamp":"2024-02-19T21:18:39.670792+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":61,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:44.465390+0000","last_change":"2024-02-19T21:19:44.465348+0000","last_active":"2024-02-19T21:19:44.465390+0000","last_peered":"2024-02-19T21:19:44.465390+0000","last_clean":"2024-02-19T21:19:44.465390+0000","last_became_active":"2024-02-19T21:19:10.738170+0000","last_became_peered":"2024-02-19T21:19:10.738170+0000","last_unstale":"2024-02-19T21:19:44.465390+0000","last_undegraded":"2024-02-19T21:19:44.465390+0000","last_fullsized":"2024-02-19T21:19:44.465390+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:44.465227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:18.857424+0000","last_clean_scrub_stamp":"2024-02-19T21:19:44.465227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":37,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969373+0000","last_change":"2024-02-19T21:19:36.619077+0000","last_active":"2024-02-19T21:19:40.969373+0000","last_peered":"2024-02-19T21:19:40.969373+0000","last_clean":"2024-02-19T21:19:40.969373+0000","last_became_active":"2024-02-19T21:19:10.740906+0000","last_became_peered":"2024-02-19T21:19:10.740906+0000","last_unstale":"2024-02-19T21:19:40.969373+0000","last_undegraded":"2024-02-19T21:19:40.969373+0000","last_fullsized":"2024-02-19T21:19:40.969373+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:36.618955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:36.618955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"662'39","reported_seq":573,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.482815+0000","last_change":"2024-02-19T21:19:30.439506+0000","last_active":"2024-02-19T21:20:00.482815+0000","last_peered":"2024-02-19T21:20:00.482815+0000","last_clean":"2024-02-19T21:20:00.482815+0000","last_became_active":"2024-02-19T21:17:29.471707+0000","last_became_peered":"2024-02-19T21:17:29.471707+0000","last_unstale":"2024-02-19T21:20:00.482815+0000","last_undegraded":"2024-02-19T21:20:00.482815+0000","last_fullsized":"2024-02-19T21:20:00.482815+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"662'39","last_scrub_stamp":"2024-02-19T21:19:30.439411+0000","last_deep_scrub":"662'39","last_deep_scrub_stamp":"2024-02-19T21:15:06.578559+0000","last_clean_scrub_stamp":"2024-02-19T21:19:30.439411+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"655'2","reported_seq":802,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968947+0000","last_change":"2024-02-19T21:19:30.711382+0000","last_active":"2024-02-19T21:19:40.968947+0000","last_peered":"2024-02-19T21:19:40.968947+0000","last_clean":"2024-02-19T21:19:40.968947+0000","last_became_active":"2024-02-19T21:19:10.741201+0000","last_became_peered":"2024-02-19T21:19:10.741201+0000","last_unstale":"2024-02-19T21:19:40.968947+0000","last_undegraded":"2024-02-19T21:19:40.968947+0000","last_fullsized":"2024-02-19T21:19:40.968947+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"655'2","last_scrub_stamp":"2024-02-19T21:19:30.711294+0000","last_deep_scrub":"655'2","last_deep_scrub_stamp":"2024-02-19T21:15:00.963594+0000","last_clean_scrub_stamp":"2024-02-19T21:19:30.711294+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":882,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.676642+0000","last_change":"2024-02-19T21:20:00.676642+0000","last_active":"2024-02-19T21:20:00.676642+0000","last_peered":"2024-02-19T21:20:00.676642+0000","last_clean":"2024-02-19T21:20:00.676642+0000","last_became_active":"2024-02-19T21:14:41.391280+0000","last_became_peered":"2024-02-19T21:14:41.391280+0000","last_unstale":"2024-02-19T21:20:00.676642+0000","last_undegraded":"2024-02-19T21:20:00.676642+0000","last_fullsized":"2024-02-19T21:20:00.676642+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:20:00.676581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:20:00.676581+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":89,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.229870+0000","last_change":"2024-02-19T21:19:46.287061+0000","last_active":"2024-02-19T21:20:00.229870+0000","last_peered":"2024-02-19T21:20:00.229870+0000","last_clean":"2024-02-19T21:20:00.229870+0000","last_became_active":"2024-02-19T21:17:29.479445+0000","last_became_peered":"2024-02-19T21:17:29.479445+0000","last_unstale":"2024-02-19T21:20:00.229870+0000","last_undegraded":"2024-02-19T21:20:00.229870+0000","last_fullsized":"2024-02-19T21:20:00.229870+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:46.286995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:19.845609+0000","last_clean_scrub_stamp":"2024-02-19T21:19:46.286995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":445,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.482134+0000","last_change":"2024-02-19T21:19:24.531845+0000","last_active":"2024-02-19T21:20:00.482134+0000","last_peered":"2024-02-19T21:20:00.482134+0000","last_clean":"2024-02-19T21:20:00.482134+0000","last_became_active":"2024-02-19T21:10:41.302670+0000","last_became_peered":"2024-02-19T21:10:41.302670+0000","last_unstale":"2024-02-19T21:20:00.482134+0000","last_undegraded":"2024-02-19T21:20:00.482134+0000","last_fullsized":"2024-02-19T21:20:00.482134+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:24.531759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:24.531759+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.c","version":"459'25","reported_seq":514,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:04.638853+0000","last_change":"2024-02-19T21:20:04.638823+0000","last_active":"2024-02-19T21:20:04.638853+0000","last_peered":"2024-02-19T21:20:04.638853+0000","last_clean":"2024-02-19T21:20:04.638853+0000","last_became_active":"2024-02-19T21:14:16.381198+0000","last_became_peered":"2024-02-19T21:14:16.381198+0000","last_unstale":"2024-02-19T21:20:04.638853+0000","last_undegraded":"2024-02-19T21:20:04.638853+0000","last_fullsized":"2024-02-19T21:20:04.638853+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'25","last_scrub_stamp":"2024-02-19T21:20:04.638738+0000","last_deep_scrub":"459'25","last_deep_scrub_stamp":"2024-02-19T21:20:04.638738+0000","last_clean_scrub_stamp":"2024-02-19T21:20:04.638738+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":884,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851075+0000","last_change":"2024-02-19T21:18:41.550936+0000","last_active":"2024-02-19T21:19:40.851075+0000","last_peered":"2024-02-19T21:19:40.851075+0000","last_clean":"2024-02-19T21:19:40.851075+0000","last_became_active":"2024-02-19T21:18:03.726011+0000","last_became_peered":"2024-02-19T21:18:03.726011+0000","last_unstale":"2024-02-19T21:19:40.851075+0000","last_undegraded":"2024-02-19T21:19:40.851075+0000","last_fullsized":"2024-02-19T21:19:40.851075+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:41.550800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:18:41.550800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"816'14","reported_seq":151,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:04.802728+0000","last_change":"2024-02-19T21:19:15.986228+0000","last_active":"2024-02-19T21:20:04.802728+0000","last_peered":"2024-02-19T21:20:04.802728+0000","last_clean":"2024-02-19T21:20:04.802728+0000","last_became_active":"2024-02-19T21:18:03.744226+0000","last_became_peered":"2024-02-19T21:18:03.744226+0000","last_unstale":"2024-02-19T21:20:04.802728+0000","last_undegraded":"2024-02-19T21:20:04.802728+0000","last_fullsized":"2024-02-19T21:20:04.802728+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'14","last_scrub_stamp":"2024-02-19T21:19:15.986160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:15.986160+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":476,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.483890+0000","last_change":"2024-02-19T21:19:03.503021+0000","last_active":"2024-02-19T21:20:00.483890+0000","last_peered":"2024-02-19T21:20:00.483890+0000","last_clean":"2024-02-19T21:20:00.483890+0000","last_became_active":"2024-02-19T21:17:29.471013+0000","last_became_peered":"2024-02-19T21:17:29.471013+0000","last_unstale":"2024-02-19T21:20:00.483890+0000","last_undegraded":"2024-02-19T21:20:00.483890+0000","last_fullsized":"2024-02-19T21:20:00.483890+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:03.502937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:03.502937+0000","last_clean_scrub_stamp":"2024-02-19T21:19:03.502937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":617,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:46.174198+0000","last_change":"2024-02-19T21:19:46.174160+0000","last_active":"2024-02-19T21:19:46.174198+0000","last_peered":"2024-02-19T21:19:46.174198+0000","last_clean":"2024-02-19T21:19:46.174198+0000","last_became_active":"2024-02-19T21:18:19.172795+0000","last_became_peered":"2024-02-19T21:18:19.172795+0000","last_unstale":"2024-02-19T21:19:46.174198+0000","last_undegraded":"2024-02-19T21:19:46.174198+0000","last_fullsized":"2024-02-19T21:19:46.174198+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:46.174087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:46.174087+0000","last_clean_scrub_stamp":"2024-02-19T21:19:46.174087+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"459'27","reported_seq":152,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:03.694222+0000","last_change":"2024-02-19T21:20:03.694222+0000","last_active":"2024-02-19T21:20:03.694222+0000","last_peered":"2024-02-19T21:20:03.694222+0000","last_clean":"2024-02-19T21:20:03.694222+0000","last_became_active":"2024-02-19T21:16:37.630325+0000","last_became_peered":"2024-02-19T21:16:37.630325+0000","last_unstale":"2024-02-19T21:20:03.694222+0000","last_undegraded":"2024-02-19T21:20:03.694222+0000","last_fullsized":"2024-02-19T21:20:03.694222+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'27","last_scrub_stamp":"2024-02-19T21:20:03.694144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:20:03.694144+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":935,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852159+0000","last_change":"2024-02-19T21:19:25.660038+0000","last_active":"2024-02-19T21:19:40.852159+0000","last_peered":"2024-02-19T21:19:40.852159+0000","last_clean":"2024-02-19T21:19:40.852159+0000","last_became_active":"2024-02-19T21:18:41.545465+0000","last_became_peered":"2024-02-19T21:18:41.545465+0000","last_unstale":"2024-02-19T21:19:40.852159+0000","last_undegraded":"2024-02-19T21:19:40.852159+0000","last_fullsized":"2024-02-19T21:19:40.852159+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:25.659979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:25.659979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"811'15","reported_seq":623,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:04.802884+0000","last_change":"2024-02-19T21:18:53.555180+0000","last_active":"2024-02-19T21:20:04.802884+0000","last_peered":"2024-02-19T21:20:04.802884+0000","last_clean":"2024-02-19T21:20:04.802884+0000","last_became_active":"2024-02-19T21:17:58.675780+0000","last_became_peered":"2024-02-19T21:17:58.675780+0000","last_unstale":"2024-02-19T21:20:04.802884+0000","last_undegraded":"2024-02-19T21:20:04.802884+0000","last_fullsized":"2024-02-19T21:20:04.802884+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"811'15","last_scrub_stamp":"2024-02-19T21:18:53.555067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:18:53.555067+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":64,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968932+0000","last_change":"2024-02-19T21:19:34.529292+0000","last_active":"2024-02-19T21:19:40.968932+0000","last_peered":"2024-02-19T21:19:40.968932+0000","last_clean":"2024-02-19T21:19:40.968932+0000","last_became_active":"2024-02-19T21:17:58.687157+0000","last_became_peered":"2024-02-19T21:17:58.687157+0000","last_unstale":"2024-02-19T21:19:40.968932+0000","last_undegraded":"2024-02-19T21:19:40.968932+0000","last_fullsized":"2024-02-19T21:19:40.968932+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:34.529209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:34.529209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":59,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969715+0000","last_change":"2024-02-19T21:19:10.573539+0000","last_active":"2024-02-19T21:19:40.969715+0000","last_peered":"2024-02-19T21:19:40.969715+0000","last_clean":"2024-02-19T21:19:40.969715+0000","last_became_active":"2024-02-19T21:17:58.689118+0000","last_became_peered":"2024-02-19T21:17:58.689118+0000","last_unstale":"2024-02-19T21:19:40.969715+0000","last_undegraded":"2024-02-19T21:19:40.969715+0000","last_fullsized":"2024-02-19T21:19:40.969715+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:10.573465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:10.573465+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"459'19","reported_seq":513,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.483582+0000","last_change":"2024-02-19T21:19:49.501449+0000","last_active":"2024-02-19T21:20:00.483582+0000","last_peered":"2024-02-19T21:20:00.483582+0000","last_clean":"2024-02-19T21:20:00.483582+0000","last_became_active":"2024-02-19T21:14:16.382074+0000","last_became_peered":"2024-02-19T21:14:16.382074+0000","last_unstale":"2024-02-19T21:20:00.483582+0000","last_undegraded":"2024-02-19T21:20:00.483582+0000","last_fullsized":"2024-02-19T21:20:00.483582+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:19:49.501373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:49.501373+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":774,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967708+0000","last_change":"2024-02-19T21:19:31.694138+0000","last_active":"2024-02-19T21:19:40.967708+0000","last_peered":"2024-02-19T21:19:40.967708+0000","last_clean":"2024-02-19T21:19:40.967708+0000","last_became_active":"2024-02-19T21:19:10.734034+0000","last_became_peered":"2024-02-19T21:19:10.734034+0000","last_unstale":"2024-02-19T21:19:40.967708+0000","last_undegraded":"2024-02-19T21:19:40.967708+0000","last_fullsized":"2024-02-19T21:19:40.967708+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:31.694046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:31.694046+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":59,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968980+0000","last_change":"2024-02-19T21:18:50.589644+0000","last_active":"2024-02-19T21:19:40.968980+0000","last_peered":"2024-02-19T21:19:40.968980+0000","last_clean":"2024-02-19T21:19:40.968980+0000","last_became_active":"2024-02-19T21:17:58.688047+0000","last_became_peered":"2024-02-19T21:17:58.688047+0000","last_unstale":"2024-02-19T21:19:40.968980+0000","last_undegraded":"2024-02-19T21:19:40.968980+0000","last_fullsized":"2024-02-19T21:19:40.968980+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:50.589569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:18:50.589569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":630,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:53.496726+0000","last_change":"2024-02-19T21:19:53.496701+0000","last_active":"2024-02-19T21:19:53.496726+0000","last_peered":"2024-02-19T21:19:53.496726+0000","last_clean":"2024-02-19T21:19:53.496726+0000","last_became_active":"2024-02-19T21:17:58.673286+0000","last_became_peered":"2024-02-19T21:17:58.673286+0000","last_unstale":"2024-02-19T21:19:53.496726+0000","last_undegraded":"2024-02-19T21:19:53.496726+0000","last_fullsized":"2024-02-19T21:19:53.496726+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:53.496636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:43.349529+0000","last_clean_scrub_stamp":"2024-02-19T21:19:53.496636+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851104+0000","last_change":"2024-02-19T21:19:28.212564+0000","last_active":"2024-02-19T21:19:40.851104+0000","last_peered":"2024-02-19T21:19:40.851104+0000","last_clean":"2024-02-19T21:19:40.851104+0000","last_became_active":"2024-02-19T21:17:58.675523+0000","last_became_peered":"2024-02-19T21:17:58.675523+0000","last_unstale":"2024-02-19T21:19:40.851104+0000","last_undegraded":"2024-02-19T21:19:40.851104+0000","last_fullsized":"2024-02-19T21:19:40.851104+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:28.212491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:28.212491+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"434'12","reported_seq":758,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:19:59.921333+0000","last_change":"2024-02-19T21:19:03.247504+0000","last_active":"2024-02-19T21:19:59.921333+0000","last_peered":"2024-02-19T21:19:59.921333+0000","last_clean":"2024-02-19T21:19:59.921333+0000","last_became_active":"2024-02-19T21:10:41.297897+0000","last_became_peered":"2024-02-19T21:10:41.297897+0000","last_unstale":"2024-02-19T21:19:59.921333+0000","last_undegraded":"2024-02-19T21:19:59.921333+0000","last_fullsized":"2024-02-19T21:19:59.921333+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'12","last_scrub_stamp":"2024-02-19T21:19:03.247435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:03.247435+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"459'28","reported_seq":582,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.482227+0000","last_change":"2024-02-19T21:19:04.552679+0000","last_active":"2024-02-19T21:20:00.482227+0000","last_peered":"2024-02-19T21:20:00.482227+0000","last_clean":"2024-02-19T21:20:00.482227+0000","last_became_active":"2024-02-19T21:17:58.672547+0000","last_became_peered":"2024-02-19T21:17:58.672547+0000","last_unstale":"2024-02-19T21:20:00.482227+0000","last_undegraded":"2024-02-19T21:20:00.482227+0000","last_fullsized":"2024-02-19T21:20:00.482227+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'28","last_scrub_stamp":"2024-02-19T21:19:04.552600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:04.552600+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":297,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968652+0000","last_change":"2024-02-19T21:19:17.426250+0000","last_active":"2024-02-19T21:19:40.968652+0000","last_peered":"2024-02-19T21:19:40.968652+0000","last_clean":"2024-02-19T21:19:40.968652+0000","last_became_active":"2024-02-19T21:18:41.545753+0000","last_became_peered":"2024-02-19T21:18:41.545753+0000","last_unstale":"2024-02-19T21:19:40.968652+0000","last_undegraded":"2024-02-19T21:19:40.968652+0000","last_fullsized":"2024-02-19T21:19:40.968652+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:17.426176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:50.164434+0000","last_clean_scrub_stamp":"2024-02-19T21:19:17.426176+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":726,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967797+0000","last_change":"2024-02-19T21:19:10.742006+0000","last_active":"2024-02-19T21:19:40.967797+0000","last_peered":"2024-02-19T21:19:40.967797+0000","last_clean":"2024-02-19T21:19:40.967797+0000","last_became_active":"2024-02-19T21:19:10.741672+0000","last_became_peered":"2024-02-19T21:19:10.741672+0000","last_unstale":"2024-02-19T21:19:40.967797+0000","last_undegraded":"2024-02-19T21:19:40.967797+0000","last_fullsized":"2024-02-19T21:19:40.967797+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:39.709227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:05:03.248506+0000","last_clean_scrub_stamp":"2024-02-19T21:18:39.709227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":683,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:01.306153+0000","last_change":"2024-02-19T21:20:01.306153+0000","last_active":"2024-02-19T21:20:01.306153+0000","last_peered":"2024-02-19T21:20:01.306153+0000","last_clean":"2024-02-19T21:20:01.306153+0000","last_became_active":"2024-02-19T21:18:19.177273+0000","last_became_peered":"2024-02-19T21:18:19.177273+0000","last_unstale":"2024-02-19T21:20:01.306153+0000","last_undegraded":"2024-02-19T21:20:01.306153+0000","last_fullsized":"2024-02-19T21:20:01.306153+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:20:01.306085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:20:01.306085+0000","last_clean_scrub_stamp":"2024-02-19T21:20:01.306085+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":463,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.481929+0000","last_change":"2024-02-19T21:19:46.483834+0000","last_active":"2024-02-19T21:20:00.481929+0000","last_peered":"2024-02-19T21:20:00.481929+0000","last_clean":"2024-02-19T21:20:00.481929+0000","last_became_active":"2024-02-19T21:18:03.726906+0000","last_became_peered":"2024-02-19T21:18:03.726906+0000","last_unstale":"2024-02-19T21:20:00.481929+0000","last_undegraded":"2024-02-19T21:20:00.481929+0000","last_fullsized":"2024-02-19T21:20:00.481929+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:46.483749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:46.483749+0000","last_clean_scrub_stamp":"2024-02-19T21:19:46.483749+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"459'30","reported_seq":726,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852611+0000","last_change":"2024-02-19T21:19:11.193104+0000","last_active":"2024-02-19T21:19:40.852611+0000","last_peered":"2024-02-19T21:19:40.852611+0000","last_clean":"2024-02-19T21:19:40.852611+0000","last_became_active":"2024-02-19T21:18:03.741555+0000","last_became_peered":"2024-02-19T21:18:03.741555+0000","last_unstale":"2024-02-19T21:19:40.852611+0000","last_undegraded":"2024-02-19T21:19:40.852611+0000","last_fullsized":"2024-02-19T21:19:40.852611+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'30","last_scrub_stamp":"2024-02-19T21:19:11.193034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:11.193034+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":894,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:43.701589+0000","last_change":"2024-02-19T21:19:43.701552+0000","last_active":"2024-02-19T21:19:43.701589+0000","last_peered":"2024-02-19T21:19:43.701589+0000","last_clean":"2024-02-19T21:19:43.701589+0000","last_became_active":"2024-02-19T21:14:16.379640+0000","last_became_peered":"2024-02-19T21:14:16.379640+0000","last_unstale":"2024-02-19T21:19:43.701589+0000","last_undegraded":"2024-02-19T21:19:43.701589+0000","last_fullsized":"2024-02-19T21:19:43.701589+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:43.701488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:43.701488+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":463,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.484187+0000","last_change":"2024-02-19T21:19:40.348360+0000","last_active":"2024-02-19T21:20:00.484187+0000","last_peered":"2024-02-19T21:20:00.484187+0000","last_clean":"2024-02-19T21:20:00.484187+0000","last_became_active":"2024-02-19T21:17:29.470869+0000","last_became_peered":"2024-02-19T21:17:29.470869+0000","last_unstale":"2024-02-19T21:20:00.484187+0000","last_undegraded":"2024-02-19T21:20:00.484187+0000","last_fullsized":"2024-02-19T21:20:00.484187+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:40.348274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:40.348274+0000","last_clean_scrub_stamp":"2024-02-19T21:19:40.348274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":427,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:55.260831+0000","last_change":"2024-02-19T21:19:55.260797+0000","last_active":"2024-02-19T21:19:55.260831+0000","last_peered":"2024-02-19T21:19:55.260831+0000","last_clean":"2024-02-19T21:19:55.260831+0000","last_became_active":"2024-02-19T21:18:19.171295+0000","last_became_peered":"2024-02-19T21:18:19.171295+0000","last_unstale":"2024-02-19T21:19:55.260831+0000","last_undegraded":"2024-02-19T21:19:55.260831+0000","last_fullsized":"2024-02-19T21:19:55.260831+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:55.260704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:21.385247+0000","last_clean_scrub_stamp":"2024-02-19T21:19:55.260704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":297,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:41.536988+0000","last_change":"2024-02-19T21:19:41.536960+0000","last_active":"2024-02-19T21:19:41.536988+0000","last_peered":"2024-02-19T21:19:41.536988+0000","last_clean":"2024-02-19T21:19:41.536988+0000","last_became_active":"2024-02-19T21:17:58.681256+0000","last_became_peered":"2024-02-19T21:17:58.681256+0000","last_unstale":"2024-02-19T21:19:41.536988+0000","last_undegraded":"2024-02-19T21:19:41.536988+0000","last_fullsized":"2024-02-19T21:19:41.536988+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:41.536880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:49.141780+0000","last_clean_scrub_stamp":"2024-02-19T21:19:41.536880+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"459'29","reported_seq":576,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:58.284209+0000","last_change":"2024-02-19T21:19:58.284186+0000","last_active":"2024-02-19T21:19:58.284209+0000","last_peered":"2024-02-19T21:19:58.284209+0000","last_clean":"2024-02-19T21:19:58.284209+0000","last_became_active":"2024-02-19T21:18:19.172356+0000","last_became_peered":"2024-02-19T21:18:19.172356+0000","last_unstale":"2024-02-19T21:19:58.284209+0000","last_undegraded":"2024-02-19T21:19:58.284209+0000","last_fullsized":"2024-02-19T21:19:58.284209+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'29","last_scrub_stamp":"2024-02-19T21:19:58.284118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:58.284118+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":36,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.229261+0000","last_change":"2024-02-19T21:19:54.312479+0000","last_active":"2024-02-19T21:20:00.229261+0000","last_peered":"2024-02-19T21:20:00.229261+0000","last_clean":"2024-02-19T21:20:00.229261+0000","last_became_active":"2024-02-19T21:18:41.546175+0000","last_became_peered":"2024-02-19T21:18:41.546175+0000","last_unstale":"2024-02-19T21:20:00.229261+0000","last_undegraded":"2024-02-19T21:20:00.229261+0000","last_fullsized":"2024-02-19T21:20:00.229261+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:54.312403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:54.312403+0000","last_clean_scrub_stamp":"2024-02-19T21:19:54.312403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":866,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852305+0000","last_change":"2024-02-19T21:19:36.729428+0000","last_active":"2024-02-19T21:19:40.852305+0000","last_peered":"2024-02-19T21:19:40.852305+0000","last_clean":"2024-02-19T21:19:40.852305+0000","last_became_active":"2024-02-19T21:17:29.516758+0000","last_became_peered":"2024-02-19T21:17:29.516758+0000","last_unstale":"2024-02-19T21:19:40.852305+0000","last_undegraded":"2024-02-19T21:19:40.852305+0000","last_fullsized":"2024-02-19T21:19:40.852305+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:36.729358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:07.473070+0000","last_clean_scrub_stamp":"2024-02-19T21:19:36.729358+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":84,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.229826+0000","last_change":"2024-02-19T21:18:32.745789+0000","last_active":"2024-02-19T21:20:00.229826+0000","last_peered":"2024-02-19T21:20:00.229826+0000","last_clean":"2024-02-19T21:20:00.229826+0000","last_became_active":"2024-02-19T21:17:29.522761+0000","last_became_peered":"2024-02-19T21:17:29.522761+0000","last_unstale":"2024-02-19T21:20:00.229826+0000","last_undegraded":"2024-02-19T21:20:00.229826+0000","last_fullsized":"2024-02-19T21:20:00.229826+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:32.745715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:32.745715+0000","last_clean_scrub_stamp":"2024-02-19T21:18:32.745715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":59,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969212+0000","last_change":"2024-02-19T21:19:12.576968+0000","last_active":"2024-02-19T21:19:40.969212+0000","last_peered":"2024-02-19T21:19:40.969212+0000","last_clean":"2024-02-19T21:19:40.969212+0000","last_became_active":"2024-02-19T21:17:58.685999+0000","last_became_peered":"2024-02-19T21:17:58.685999+0000","last_unstale":"2024-02-19T21:19:40.969212+0000","last_undegraded":"2024-02-19T21:19:40.969212+0000","last_fullsized":"2024-02-19T21:19:40.969212+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:12.576894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:12.576894+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"459'24","reported_seq":555,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:56.238618+0000","last_change":"2024-02-19T21:19:56.238578+0000","last_active":"2024-02-19T21:19:56.238618+0000","last_peered":"2024-02-19T21:19:56.238618+0000","last_clean":"2024-02-19T21:19:56.238618+0000","last_became_active":"2024-02-19T21:18:19.170850+0000","last_became_peered":"2024-02-19T21:18:19.170850+0000","last_unstale":"2024-02-19T21:19:56.238618+0000","last_undegraded":"2024-02-19T21:19:56.238618+0000","last_fullsized":"2024-02-19T21:19:56.238618+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'24","last_scrub_stamp":"2024-02-19T21:19:56.238476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:56.238476+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"397'2","reported_seq":544,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967532+0000","last_change":"2024-02-19T21:19:19.550446+0000","last_active":"2024-02-19T21:19:40.967532+0000","last_peered":"2024-02-19T21:19:40.967532+0000","last_clean":"2024-02-19T21:19:40.967532+0000","last_became_active":"2024-02-19T21:19:10.733472+0000","last_became_peered":"2024-02-19T21:19:10.733472+0000","last_unstale":"2024-02-19T21:19:40.967532+0000","last_undegraded":"2024-02-19T21:19:40.967532+0000","last_fullsized":"2024-02-19T21:19:40.967532+0000","mapping_epoch":848,"log_start":"58'1","ondisk_log_start":"58'1","created":57,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'2","last_scrub_stamp":"2024-02-19T21:19:19.550363+0000","last_deep_scrub":"397'2","last_deep_scrub_stamp":"2024-02-19T21:09:40.934474+0000","last_clean_scrub_stamp":"2024-02-19T21:19:19.550363+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":277,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.966930+0000","last_change":"2024-02-19T21:19:24.347774+0000","last_active":"2024-02-19T21:19:40.966930+0000","last_peered":"2024-02-19T21:19:40.966930+0000","last_clean":"2024-02-19T21:19:40.966930+0000","last_became_active":"2024-02-19T21:18:19.172569+0000","last_became_peered":"2024-02-19T21:18:19.172569+0000","last_unstale":"2024-02-19T21:19:40.966930+0000","last_undegraded":"2024-02-19T21:19:40.966930+0000","last_fullsized":"2024-02-19T21:19:40.966930+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:24.347707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:57.505029+0000","last_clean_scrub_stamp":"2024-02-19T21:19:24.347707+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":859,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852997+0000","last_change":"2024-02-19T21:19:18.748401+0000","last_active":"2024-02-19T21:19:40.852997+0000","last_peered":"2024-02-19T21:19:40.852997+0000","last_clean":"2024-02-19T21:19:40.852997+0000","last_became_active":"2024-02-19T21:17:58.691420+0000","last_became_peered":"2024-02-19T21:17:58.691420+0000","last_unstale":"2024-02-19T21:19:40.852997+0000","last_undegraded":"2024-02-19T21:19:40.852997+0000","last_fullsized":"2024-02-19T21:19:40.852997+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:18.748338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:18.748338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":279,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969761+0000","last_change":"2024-02-19T21:19:14.434878+0000","last_active":"2024-02-19T21:19:40.969761+0000","last_peered":"2024-02-19T21:19:40.969761+0000","last_clean":"2024-02-19T21:19:40.969761+0000","last_became_active":"2024-02-19T21:17:58.682748+0000","last_became_peered":"2024-02-19T21:17:58.682748+0000","last_unstale":"2024-02-19T21:19:40.969761+0000","last_undegraded":"2024-02-19T21:19:40.969761+0000","last_fullsized":"2024-02-19T21:19:40.969761+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:14.434797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:14.434797+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":70,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968955+0000","last_change":"2024-02-19T21:19:10.741533+0000","last_active":"2024-02-19T21:19:40.968955+0000","last_peered":"2024-02-19T21:19:40.968955+0000","last_clean":"2024-02-19T21:19:40.968955+0000","last_became_active":"2024-02-19T21:19:10.740867+0000","last_became_peered":"2024-02-19T21:19:10.740867+0000","last_unstale":"2024-02-19T21:19:40.968955+0000","last_undegraded":"2024-02-19T21:19:40.968955+0000","last_fullsized":"2024-02-19T21:19:40.968955+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:05.942411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:05.942411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"459'26","reported_seq":318,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968331+0000","last_change":"2024-02-19T21:19:09.487024+0000","last_active":"2024-02-19T21:19:40.968331+0000","last_peered":"2024-02-19T21:19:40.968331+0000","last_clean":"2024-02-19T21:19:40.968331+0000","last_became_active":"2024-02-19T21:18:03.727966+0000","last_became_peered":"2024-02-19T21:18:03.727966+0000","last_unstale":"2024-02-19T21:19:40.968331+0000","last_undegraded":"2024-02-19T21:19:40.968331+0000","last_fullsized":"2024-02-19T21:19:40.968331+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'26","last_scrub_stamp":"2024-02-19T21:19:09.486913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:09.486913+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":662,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967796+0000","last_change":"2024-02-19T21:19:25.629678+0000","last_active":"2024-02-19T21:19:40.967796+0000","last_peered":"2024-02-19T21:19:40.967796+0000","last_clean":"2024-02-19T21:19:40.967796+0000","last_became_active":"2024-02-19T21:19:10.742230+0000","last_became_peered":"2024-02-19T21:19:10.742230+0000","last_unstale":"2024-02-19T21:19:40.967796+0000","last_undegraded":"2024-02-19T21:19:40.967796+0000","last_fullsized":"2024-02-19T21:19:40.967796+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:25.629584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T20:59:10.174707+0000","last_clean_scrub_stamp":"2024-02-19T21:19:25.629584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":68,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969637+0000","last_change":"2024-02-19T21:18:34.444400+0000","last_active":"2024-02-19T21:19:40.969637+0000","last_peered":"2024-02-19T21:19:40.969637+0000","last_clean":"2024-02-19T21:19:40.969637+0000","last_became_active":"2024-02-19T21:18:03.735603+0000","last_became_peered":"2024-02-19T21:18:03.735603+0000","last_unstale":"2024-02-19T21:19:40.969637+0000","last_undegraded":"2024-02-19T21:19:40.969637+0000","last_fullsized":"2024-02-19T21:19:40.969637+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:34.444317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:34.444317+0000","last_clean_scrub_stamp":"2024-02-19T21:18:34.444317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851181+0000","last_change":"2024-02-19T21:18:42.341345+0000","last_active":"2024-02-19T21:19:40.851181+0000","last_peered":"2024-02-19T21:19:40.851181+0000","last_clean":"2024-02-19T21:19:40.851181+0000","last_became_active":"2024-02-19T21:17:58.675488+0000","last_became_peered":"2024-02-19T21:17:58.675488+0000","last_unstale":"2024-02-19T21:19:40.851181+0000","last_undegraded":"2024-02-19T21:19:40.851181+0000","last_fullsized":"2024-02-19T21:19:40.851181+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:42.341279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:18:42.341279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":64,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:52.598648+0000","last_change":"2024-02-19T21:19:52.598627+0000","last_active":"2024-02-19T21:19:52.598648+0000","last_peered":"2024-02-19T21:19:52.598648+0000","last_clean":"2024-02-19T21:19:52.598648+0000","last_became_active":"2024-02-19T21:17:58.680832+0000","last_became_peered":"2024-02-19T21:17:58.680832+0000","last_unstale":"2024-02-19T21:19:52.598648+0000","last_undegraded":"2024-02-19T21:19:52.598648+0000","last_fullsized":"2024-02-19T21:19:52.598648+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:52.598556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:52.598556+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":59,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968141+0000","last_change":"2024-02-19T21:18:45.539887+0000","last_active":"2024-02-19T21:19:40.968141+0000","last_peered":"2024-02-19T21:19:40.968141+0000","last_clean":"2024-02-19T21:19:40.968141+0000","last_became_active":"2024-02-19T21:17:58.684312+0000","last_became_peered":"2024-02-19T21:17:58.684312+0000","last_unstale":"2024-02-19T21:19:40.968141+0000","last_undegraded":"2024-02-19T21:19:40.968141+0000","last_fullsized":"2024-02-19T21:19:40.968141+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:45.539763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:10:20.204880+0000","last_clean_scrub_stamp":"2024-02-19T21:18:45.539763+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":107,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.229659+0000","last_change":"2024-02-19T21:18:47.716644+0000","last_active":"2024-02-19T21:20:00.229659+0000","last_peered":"2024-02-19T21:20:00.229659+0000","last_clean":"2024-02-19T21:20:00.229659+0000","last_became_active":"2024-02-19T21:18:03.742383+0000","last_became_peered":"2024-02-19T21:18:03.742383+0000","last_unstale":"2024-02-19T21:20:00.229659+0000","last_undegraded":"2024-02-19T21:20:00.229659+0000","last_fullsized":"2024-02-19T21:20:00.229659+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:47.716570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:18:47.716570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"459'22","reported_seq":206,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:52.687469+0000","last_change":"2024-02-19T21:19:52.687446+0000","last_active":"2024-02-19T21:19:52.687469+0000","last_peered":"2024-02-19T21:19:52.687469+0000","last_clean":"2024-02-19T21:19:52.687469+0000","last_became_active":"2024-02-19T21:17:58.686399+0000","last_became_peered":"2024-02-19T21:17:58.686399+0000","last_unstale":"2024-02-19T21:19:52.687469+0000","last_undegraded":"2024-02-19T21:19:52.687469+0000","last_fullsized":"2024-02-19T21:19:52.687469+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'22","last_scrub_stamp":"2024-02-19T21:19:52.687378+0000","last_deep_scrub":"86'18","last_deep_scrub_stamp":"2024-02-19T21:03:27.966528+0000","last_clean_scrub_stamp":"2024-02-19T21:19:52.687378+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":89,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.228682+0000","last_change":"2024-02-19T21:19:22.938201+0000","last_active":"2024-02-19T21:20:00.228682+0000","last_peered":"2024-02-19T21:20:00.228682+0000","last_clean":"2024-02-19T21:20:00.228682+0000","last_became_active":"2024-02-19T21:17:29.525497+0000","last_became_peered":"2024-02-19T21:17:29.525497+0000","last_unstale":"2024-02-19T21:20:00.228682+0000","last_undegraded":"2024-02-19T21:20:00.228682+0000","last_fullsized":"2024-02-19T21:20:00.228682+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:22.938133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:22.938133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"65'5","reported_seq":2004,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:03.776495+0000","last_change":"2024-02-19T21:19:30.754451+0000","last_active":"2024-02-19T21:20:03.776495+0000","last_peered":"2024-02-19T21:20:03.776495+0000","last_clean":"2024-02-19T21:20:03.776495+0000","last_became_active":"2024-02-19T21:17:29.529134+0000","last_became_peered":"2024-02-19T21:17:29.529134+0000","last_unstale":"2024-02-19T21:20:03.776495+0000","last_undegraded":"2024-02-19T21:20:03.776495+0000","last_fullsized":"2024-02-19T21:20:03.776495+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'5","last_scrub_stamp":"2024-02-19T21:19:30.754381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:30.754381+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1112,"num_read_kb":1107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":736,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969077+0000","last_change":"2024-02-19T21:19:10.742903+0000","last_active":"2024-02-19T21:19:40.969077+0000","last_peered":"2024-02-19T21:19:40.969077+0000","last_clean":"2024-02-19T21:19:40.969077+0000","last_became_active":"2024-02-19T21:19:10.742195+0000","last_became_peered":"2024-02-19T21:19:10.742195+0000","last_unstale":"2024-02-19T21:19:40.969077+0000","last_undegraded":"2024-02-19T21:19:40.969077+0000","last_fullsized":"2024-02-19T21:19:40.969077+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:53.675138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:53.675138+0000","last_clean_scrub_stamp":"2024-02-19T21:18:53.675138+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":820,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851644+0000","last_change":"2024-02-19T21:19:26.677237+0000","last_active":"2024-02-19T21:19:40.851644+0000","last_peered":"2024-02-19T21:19:40.851644+0000","last_clean":"2024-02-19T21:19:40.851644+0000","last_became_active":"2024-02-19T21:18:03.734552+0000","last_became_peered":"2024-02-19T21:18:03.734552+0000","last_unstale":"2024-02-19T21:19:40.851644+0000","last_undegraded":"2024-02-19T21:19:40.851644+0000","last_fullsized":"2024-02-19T21:19:40.851644+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:26.677161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:26.677161+0000","last_clean_scrub_stamp":"2024-02-19T21:19:26.677161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":524,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968697+0000","last_change":"2024-02-19T21:19:31.273086+0000","last_active":"2024-02-19T21:19:40.968697+0000","last_peered":"2024-02-19T21:19:40.968697+0000","last_clean":"2024-02-19T21:19:40.968697+0000","last_became_active":"2024-02-19T21:18:19.171202+0000","last_became_peered":"2024-02-19T21:18:19.171202+0000","last_unstale":"2024-02-19T21:19:40.968697+0000","last_undegraded":"2024-02-19T21:19:40.968697+0000","last_fullsized":"2024-02-19T21:19:40.968697+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:31.272992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:02:56.510474+0000","last_clean_scrub_stamp":"2024-02-19T21:19:31.272992+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"834'54","reported_seq":1251,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.482106+0000","last_change":"2024-02-19T21:18:42.503258+0000","last_active":"2024-02-19T21:20:00.482106+0000","last_peered":"2024-02-19T21:20:00.482106+0000","last_clean":"2024-02-19T21:20:00.482106+0000","last_became_active":"2024-02-19T21:17:29.529715+0000","last_became_peered":"2024-02-19T21:17:29.529715+0000","last_unstale":"2024-02-19T21:20:00.482106+0000","last_undegraded":"2024-02-19T21:20:00.482106+0000","last_fullsized":"2024-02-19T21:20:00.482106+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"834'54","last_scrub_stamp":"2024-02-19T21:18:42.503149+0000","last_deep_scrub":"834'54","last_deep_scrub_stamp":"2024-02-19T21:18:42.503149+0000","last_clean_scrub_stamp":"2024-02-19T21:18:42.503149+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":59,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.970410+0000","last_change":"2024-02-19T21:18:51.564128+0000","last_active":"2024-02-19T21:19:40.970410+0000","last_peered":"2024-02-19T21:19:40.970410+0000","last_clean":"2024-02-19T21:19:40.970410+0000","last_became_active":"2024-02-19T21:17:58.689167+0000","last_became_peered":"2024-02-19T21:17:58.689167+0000","last_unstale":"2024-02-19T21:19:40.970410+0000","last_undegraded":"2024-02-19T21:19:40.970410+0000","last_fullsized":"2024-02-19T21:19:40.970410+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:51.564022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:19.845394+0000","last_clean_scrub_stamp":"2024-02-19T21:18:51.564022+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:46.548959+0000","last_change":"2024-02-19T21:19:46.548923+0000","last_active":"2024-02-19T21:19:46.548959+0000","last_peered":"2024-02-19T21:19:46.548959+0000","last_clean":"2024-02-19T21:19:46.548959+0000","last_became_active":"2024-02-19T21:18:03.731371+0000","last_became_peered":"2024-02-19T21:18:03.731371+0000","last_unstale":"2024-02-19T21:19:46.548959+0000","last_undegraded":"2024-02-19T21:19:46.548959+0000","last_fullsized":"2024-02-19T21:19:46.548959+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:46.548853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:46.548853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":93,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.228732+0000","last_change":"2024-02-19T21:18:29.713301+0000","last_active":"2024-02-19T21:20:00.228732+0000","last_peered":"2024-02-19T21:20:00.228732+0000","last_clean":"2024-02-19T21:20:00.228732+0000","last_became_active":"2024-02-19T21:17:58.688197+0000","last_became_peered":"2024-02-19T21:17:58.688197+0000","last_unstale":"2024-02-19T21:20:00.228732+0000","last_undegraded":"2024-02-19T21:20:00.228732+0000","last_fullsized":"2024-02-19T21:20:00.228732+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:29.713207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:18:29.713207+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"824'17","reported_seq":104,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:19:59.919872+0000","last_change":"2024-02-19T21:19:15.374737+0000","last_active":"2024-02-19T21:19:59.919872+0000","last_peered":"2024-02-19T21:19:59.919872+0000","last_clean":"2024-02-19T21:19:59.919872+0000","last_became_active":"2024-02-19T21:18:19.165036+0000","last_became_peered":"2024-02-19T21:18:19.165036+0000","last_unstale":"2024-02-19T21:19:59.919872+0000","last_undegraded":"2024-02-19T21:19:59.919872+0000","last_fullsized":"2024-02-19T21:19:59.919872+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"824'17","last_scrub_stamp":"2024-02-19T21:19:15.374622+0000","last_deep_scrub":"471'10","last_deep_scrub_stamp":"2024-02-19T21:11:41.775418+0000","last_clean_scrub_stamp":"2024-02-19T21:19:15.374622+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"459'20","reported_seq":48,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.228661+0000","last_change":"2024-02-19T21:19:20.989153+0000","last_active":"2024-02-19T21:20:00.228661+0000","last_peered":"2024-02-19T21:20:00.228661+0000","last_clean":"2024-02-19T21:20:00.228661+0000","last_became_active":"2024-02-19T21:18:41.543526+0000","last_became_peered":"2024-02-19T21:18:41.543526+0000","last_unstale":"2024-02-19T21:20:00.228661+0000","last_undegraded":"2024-02-19T21:20:00.228661+0000","last_fullsized":"2024-02-19T21:20:00.228661+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:19:20.989044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:20.989044+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":931,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:53.638224+0000","last_change":"2024-02-19T21:19:53.638179+0000","last_active":"2024-02-19T21:19:53.638224+0000","last_peered":"2024-02-19T21:19:53.638224+0000","last_clean":"2024-02-19T21:19:53.638224+0000","last_became_active":"2024-02-19T21:18:03.734529+0000","last_became_peered":"2024-02-19T21:18:03.734529+0000","last_unstale":"2024-02-19T21:19:53.638224+0000","last_undegraded":"2024-02-19T21:19:53.638224+0000","last_fullsized":"2024-02-19T21:19:53.638224+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:53.638068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:01.530108+0000","last_clean_scrub_stamp":"2024-02-19T21:19:53.638068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":590,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967159+0000","last_change":"2024-02-19T21:18:47.359382+0000","last_active":"2024-02-19T21:19:40.967159+0000","last_peered":"2024-02-19T21:19:40.967159+0000","last_clean":"2024-02-19T21:19:40.967159+0000","last_became_active":"2024-02-19T21:18:19.169773+0000","last_became_peered":"2024-02-19T21:18:19.169773+0000","last_unstale":"2024-02-19T21:19:40.967159+0000","last_undegraded":"2024-02-19T21:19:40.967159+0000","last_fullsized":"2024-02-19T21:19:40.967159+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:47.359312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:18:47.359312+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":598,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851723+0000","last_change":"2024-02-19T21:19:38.307780+0000","last_active":"2024-02-19T21:19:40.851723+0000","last_peered":"2024-02-19T21:19:40.851723+0000","last_clean":"2024-02-19T21:19:40.851723+0000","last_became_active":"2024-02-19T21:14:16.383904+0000","last_became_peered":"2024-02-19T21:14:16.383904+0000","last_unstale":"2024-02-19T21:19:40.851723+0000","last_undegraded":"2024-02-19T21:19:40.851723+0000","last_fullsized":"2024-02-19T21:19:40.851723+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:38.307710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:08.313435+0000","last_clean_scrub_stamp":"2024-02-19T21:19:38.307710+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":89,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.228648+0000","last_change":"2024-02-19T21:19:18.991408+0000","last_active":"2024-02-19T21:20:00.228648+0000","last_peered":"2024-02-19T21:20:00.228648+0000","last_clean":"2024-02-19T21:20:00.228648+0000","last_became_active":"2024-02-19T21:17:29.526877+0000","last_became_peered":"2024-02-19T21:17:29.526877+0000","last_unstale":"2024-02-19T21:20:00.228648+0000","last_undegraded":"2024-02-19T21:20:00.228648+0000","last_fullsized":"2024-02-19T21:20:00.228648+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:18.991321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:16.866116+0000","last_clean_scrub_stamp":"2024-02-19T21:19:18.991321+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"459'23","reported_seq":466,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.482030+0000","last_change":"2024-02-19T21:19:35.387240+0000","last_active":"2024-02-19T21:20:00.482030+0000","last_peered":"2024-02-19T21:20:00.482030+0000","last_clean":"2024-02-19T21:20:00.482030+0000","last_became_active":"2024-02-19T21:10:41.287134+0000","last_became_peered":"2024-02-19T21:10:41.287134+0000","last_unstale":"2024-02-19T21:20:00.482030+0000","last_undegraded":"2024-02-19T21:20:00.482030+0000","last_fullsized":"2024-02-19T21:20:00.482030+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'23","last_scrub_stamp":"2024-02-19T21:19:35.387152+0000","last_deep_scrub":"420'17","last_deep_scrub_stamp":"2024-02-19T21:10:19.722303+0000","last_clean_scrub_stamp":"2024-02-19T21:19:35.387152+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"84'4","reported_seq":526,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.482903+0000","last_change":"2024-02-19T21:19:21.517612+0000","last_active":"2024-02-19T21:20:00.482903+0000","last_peered":"2024-02-19T21:20:00.482903+0000","last_clean":"2024-02-19T21:20:00.482903+0000","last_became_active":"2024-02-19T21:18:03.726841+0000","last_became_peered":"2024-02-19T21:18:03.726841+0000","last_unstale":"2024-02-19T21:20:00.482903+0000","last_undegraded":"2024-02-19T21:20:00.482903+0000","last_fullsized":"2024-02-19T21:20:00.482903+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'4","last_scrub_stamp":"2024-02-19T21:19:21.517494+0000","last_deep_scrub":"84'4","last_deep_scrub_stamp":"2024-02-19T21:03:21.164286+0000","last_clean_scrub_stamp":"2024-02-19T21:19:21.517494+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":603,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:43.474941+0000","last_change":"2024-02-19T21:19:43.474904+0000","last_active":"2024-02-19T21:19:43.474941+0000","last_peered":"2024-02-19T21:19:43.474941+0000","last_clean":"2024-02-19T21:19:43.474941+0000","last_became_active":"2024-02-19T21:18:03.730876+0000","last_became_peered":"2024-02-19T21:18:03.730876+0000","last_unstale":"2024-02-19T21:19:43.474941+0000","last_undegraded":"2024-02-19T21:19:43.474941+0000","last_fullsized":"2024-02-19T21:19:43.474941+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:43.474803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:17.625711+0000","last_clean_scrub_stamp":"2024-02-19T21:19:43.474803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":98,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.229438+0000","last_change":"2024-02-19T21:19:57.225281+0000","last_active":"2024-02-19T21:20:00.229438+0000","last_peered":"2024-02-19T21:20:00.229438+0000","last_clean":"2024-02-19T21:20:00.229438+0000","last_became_active":"2024-02-19T21:18:03.742208+0000","last_became_peered":"2024-02-19T21:18:03.742208+0000","last_unstale":"2024-02-19T21:20:00.229438+0000","last_undegraded":"2024-02-19T21:20:00.229438+0000","last_fullsized":"2024-02-19T21:20:00.229438+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:57.225206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:57.225206+0000","last_clean_scrub_stamp":"2024-02-19T21:19:57.225206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"459'15","reported_seq":440,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968353+0000","last_change":"2024-02-19T21:19:05.311570+0000","last_active":"2024-02-19T21:19:40.968353+0000","last_peered":"2024-02-19T21:19:40.968353+0000","last_clean":"2024-02-19T21:19:40.968353+0000","last_became_active":"2024-02-19T21:18:19.174039+0000","last_became_peered":"2024-02-19T21:18:19.174039+0000","last_unstale":"2024-02-19T21:19:40.968353+0000","last_undegraded":"2024-02-19T21:19:40.968353+0000","last_fullsized":"2024-02-19T21:19:40.968353+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:19:05.311494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:05.311494+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":89,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.228593+0000","last_change":"2024-02-19T21:19:59.232504+0000","last_active":"2024-02-19T21:20:00.228593+0000","last_peered":"2024-02-19T21:20:00.228593+0000","last_clean":"2024-02-19T21:20:00.228593+0000","last_became_active":"2024-02-19T21:17:29.527055+0000","last_became_peered":"2024-02-19T21:17:29.527055+0000","last_unstale":"2024-02-19T21:20:00.228593+0000","last_undegraded":"2024-02-19T21:20:00.228593+0000","last_fullsized":"2024-02-19T21:20:00.228593+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:59.232438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:59.232438+0000","last_clean_scrub_stamp":"2024-02-19T21:19:59.232438+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":598,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851458+0000","last_change":"2024-02-19T21:19:16.216864+0000","last_active":"2024-02-19T21:19:40.851458+0000","last_peered":"2024-02-19T21:19:40.851458+0000","last_clean":"2024-02-19T21:19:40.851458+0000","last_became_active":"2024-02-19T21:18:03.730784+0000","last_became_peered":"2024-02-19T21:18:03.730784+0000","last_unstale":"2024-02-19T21:19:40.851458+0000","last_undegraded":"2024-02-19T21:19:40.851458+0000","last_fullsized":"2024-02-19T21:19:40.851458+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:16.216788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:16.216788+0000","last_clean_scrub_stamp":"2024-02-19T21:19:16.216788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":98,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.229674+0000","last_change":"2024-02-19T21:19:10.984020+0000","last_active":"2024-02-19T21:20:00.229674+0000","last_peered":"2024-02-19T21:20:00.229674+0000","last_clean":"2024-02-19T21:20:00.229674+0000","last_became_active":"2024-02-19T21:18:03.743303+0000","last_became_peered":"2024-02-19T21:18:03.743303+0000","last_unstale":"2024-02-19T21:20:00.229674+0000","last_undegraded":"2024-02-19T21:20:00.229674+0000","last_fullsized":"2024-02-19T21:20:00.229674+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:10.983950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:10.983950+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":458,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.484297+0000","last_change":"2024-02-19T21:18:59.442878+0000","last_active":"2024-02-19T21:20:00.484297+0000","last_peered":"2024-02-19T21:20:00.484297+0000","last_clean":"2024-02-19T21:20:00.484297+0000","last_became_active":"2024-02-19T21:18:03.725494+0000","last_became_peered":"2024-02-19T21:18:03.725494+0000","last_unstale":"2024-02-19T21:20:00.484297+0000","last_undegraded":"2024-02-19T21:20:00.484297+0000","last_fullsized":"2024-02-19T21:20:00.484297+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:59.442799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:18:59.442799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"459'15","reported_seq":187,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851179+0000","last_change":"2024-02-19T21:18:50.289558+0000","last_active":"2024-02-19T21:19:40.851179+0000","last_peered":"2024-02-19T21:19:40.851179+0000","last_clean":"2024-02-19T21:19:40.851179+0000","last_became_active":"2024-02-19T21:18:03.730847+0000","last_became_peered":"2024-02-19T21:18:03.730847+0000","last_unstale":"2024-02-19T21:19:40.851179+0000","last_undegraded":"2024-02-19T21:19:40.851179+0000","last_fullsized":"2024-02-19T21:19:40.851179+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:18:50.289450+0000","last_deep_scrub":"78'9","last_deep_scrub_stamp":"2024-02-19T21:02:33.968182+0000","last_clean_scrub_stamp":"2024-02-19T21:18:50.289450+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":252,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969053+0000","last_change":"2024-02-19T21:19:00.571835+0000","last_active":"2024-02-19T21:19:40.969053+0000","last_peered":"2024-02-19T21:19:40.969053+0000","last_clean":"2024-02-19T21:19:40.969053+0000","last_became_active":"2024-02-19T21:14:16.385407+0000","last_became_peered":"2024-02-19T21:14:16.385407+0000","last_unstale":"2024-02-19T21:19:40.969053+0000","last_undegraded":"2024-02-19T21:19:40.969053+0000","last_fullsized":"2024-02-19T21:19:40.969053+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:00.571763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:15:07.220022+0000","last_clean_scrub_stamp":"2024-02-19T21:19:00.571763+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":59,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.970177+0000","last_change":"2024-02-19T21:19:17.628960+0000","last_active":"2024-02-19T21:19:40.970177+0000","last_peered":"2024-02-19T21:19:40.970177+0000","last_clean":"2024-02-19T21:19:40.970177+0000","last_became_active":"2024-02-19T21:17:58.686411+0000","last_became_peered":"2024-02-19T21:17:58.686411+0000","last_unstale":"2024-02-19T21:19:40.970177+0000","last_undegraded":"2024-02-19T21:19:40.970177+0000","last_fullsized":"2024-02-19T21:19:40.970177+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:17.628866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:17.628866+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":741,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:47.459819+0000","last_change":"2024-02-19T21:19:47.459791+0000","last_active":"2024-02-19T21:19:47.459819+0000","last_peered":"2024-02-19T21:19:47.459819+0000","last_clean":"2024-02-19T21:19:47.459819+0000","last_became_active":"2024-02-19T21:19:10.742604+0000","last_became_peered":"2024-02-19T21:19:10.742604+0000","last_unstale":"2024-02-19T21:19:47.459819+0000","last_undegraded":"2024-02-19T21:19:47.459819+0000","last_fullsized":"2024-02-19T21:19:47.459819+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:47.459713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:47.459713+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":64,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969559+0000","last_change":"2024-02-19T21:19:40.661379+0000","last_active":"2024-02-19T21:19:40.969559+0000","last_peered":"2024-02-19T21:19:40.969559+0000","last_clean":"2024-02-19T21:19:40.969559+0000","last_became_active":"2024-02-19T21:17:58.689211+0000","last_became_peered":"2024-02-19T21:17:58.689211+0000","last_unstale":"2024-02-19T21:19:40.969559+0000","last_undegraded":"2024-02-19T21:19:40.969559+0000","last_fullsized":"2024-02-19T21:19:40.969559+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:40.661301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:40.661301+0000","last_clean_scrub_stamp":"2024-02-19T21:19:40.661301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"816'25","reported_seq":675,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851502+0000","last_change":"2024-02-19T21:19:30.192778+0000","last_active":"2024-02-19T21:19:40.851502+0000","last_peered":"2024-02-19T21:19:40.851502+0000","last_clean":"2024-02-19T21:19:40.851502+0000","last_became_active":"2024-02-19T21:18:03.732603+0000","last_became_peered":"2024-02-19T21:18:03.732603+0000","last_unstale":"2024-02-19T21:19:40.851502+0000","last_undegraded":"2024-02-19T21:19:40.851502+0000","last_fullsized":"2024-02-19T21:19:40.851502+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'25","last_scrub_stamp":"2024-02-19T21:19:30.192706+0000","last_deep_scrub":"816'25","last_deep_scrub_stamp":"2024-02-19T21:19:30.192706+0000","last_clean_scrub_stamp":"2024-02-19T21:19:30.192706+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":73,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.970448+0000","last_change":"2024-02-19T21:19:27.581663+0000","last_active":"2024-02-19T21:19:40.970448+0000","last_peered":"2024-02-19T21:19:40.970448+0000","last_clean":"2024-02-19T21:19:40.970448+0000","last_became_active":"2024-02-19T21:18:03.732785+0000","last_became_peered":"2024-02-19T21:18:03.732785+0000","last_unstale":"2024-02-19T21:19:40.970448+0000","last_undegraded":"2024-02-19T21:19:40.970448+0000","last_fullsized":"2024-02-19T21:19:40.970448+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:27.581567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:09:40.986635+0000","last_clean_scrub_stamp":"2024-02-19T21:19:27.581567+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"811'12","reported_seq":89,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.483383+0000","last_change":"2024-02-19T21:18:28.428328+0000","last_active":"2024-02-19T21:20:00.483383+0000","last_peered":"2024-02-19T21:20:00.483383+0000","last_clean":"2024-02-19T21:20:00.483383+0000","last_became_active":"2024-02-19T21:17:58.687483+0000","last_became_peered":"2024-02-19T21:17:58.687483+0000","last_unstale":"2024-02-19T21:20:00.483383+0000","last_undegraded":"2024-02-19T21:20:00.483383+0000","last_fullsized":"2024-02-19T21:20:00.483383+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"811'12","last_scrub_stamp":"2024-02-19T21:18:28.428243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:18:28.428243+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"65'1","reported_seq":528,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968206+0000","last_change":"2024-02-19T21:19:12.315318+0000","last_active":"2024-02-19T21:19:40.968206+0000","last_peered":"2024-02-19T21:19:40.968206+0000","last_clean":"2024-02-19T21:19:40.968206+0000","last_became_active":"2024-02-19T21:18:19.174474+0000","last_became_peered":"2024-02-19T21:18:19.174474+0000","last_unstale":"2024-02-19T21:19:40.968206+0000","last_undegraded":"2024-02-19T21:19:40.968206+0000","last_fullsized":"2024-02-19T21:19:40.968206+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-02-19T21:19:12.315246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:12.315246+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":240,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.966849+0000","last_change":"2024-02-19T21:18:51.349780+0000","last_active":"2024-02-19T21:19:40.966849+0000","last_peered":"2024-02-19T21:19:40.966849+0000","last_clean":"2024-02-19T21:19:40.966849+0000","last_became_active":"2024-02-19T21:18:19.169061+0000","last_became_peered":"2024-02-19T21:18:19.169061+0000","last_unstale":"2024-02-19T21:19:40.966849+0000","last_undegraded":"2024-02-19T21:19:40.966849+0000","last_fullsized":"2024-02-19T21:19:40.966849+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:51.349682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:23.608673+0000","last_clean_scrub_stamp":"2024-02-19T21:18:51.349682+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"732'19","reported_seq":88,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969030+0000","last_change":"2024-02-19T21:18:48.603165+0000","last_active":"2024-02-19T21:19:40.969030+0000","last_peered":"2024-02-19T21:19:40.969030+0000","last_clean":"2024-02-19T21:19:40.969030+0000","last_became_active":"2024-02-19T21:18:03.732567+0000","last_became_peered":"2024-02-19T21:18:03.732567+0000","last_unstale":"2024-02-19T21:19:40.969030+0000","last_undegraded":"2024-02-19T21:19:40.969030+0000","last_fullsized":"2024-02-19T21:19:40.969030+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'19","last_scrub_stamp":"2024-02-19T21:18:48.603069+0000","last_deep_scrub":"732'19","last_deep_scrub_stamp":"2024-02-19T21:18:48.603069+0000","last_clean_scrub_stamp":"2024-02-19T21:18:48.603069+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":282,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:04.331544+0000","last_change":"2024-02-19T21:20:04.331544+0000","last_active":"2024-02-19T21:20:04.331544+0000","last_peered":"2024-02-19T21:20:04.331544+0000","last_clean":"2024-02-19T21:20:04.331544+0000","last_became_active":"2024-02-19T21:18:19.171116+0000","last_became_peered":"2024-02-19T21:18:19.171116+0000","last_unstale":"2024-02-19T21:20:04.331544+0000","last_undegraded":"2024-02-19T21:20:04.331544+0000","last_fullsized":"2024-02-19T21:20:04.331544+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:20:04.331474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:20:04.331474+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"816'14","reported_seq":786,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:19:59.921944+0000","last_change":"2024-02-19T21:19:50.616694+0000","last_active":"2024-02-19T21:19:59.921944+0000","last_peered":"2024-02-19T21:19:59.921944+0000","last_clean":"2024-02-19T21:19:59.921944+0000","last_became_active":"2024-02-19T21:18:03.731454+0000","last_became_peered":"2024-02-19T21:18:03.731454+0000","last_unstale":"2024-02-19T21:19:59.921944+0000","last_undegraded":"2024-02-19T21:19:59.921944+0000","last_fullsized":"2024-02-19T21:19:59.921944+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'14","last_scrub_stamp":"2024-02-19T21:19:50.616622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:50.616622+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":490,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.482954+0000","last_change":"2024-02-19T21:19:42.381733+0000","last_active":"2024-02-19T21:20:00.482954+0000","last_peered":"2024-02-19T21:20:00.482954+0000","last_clean":"2024-02-19T21:20:00.482954+0000","last_became_active":"2024-02-19T21:17:58.680636+0000","last_became_peered":"2024-02-19T21:17:58.680636+0000","last_unstale":"2024-02-19T21:20:00.482954+0000","last_undegraded":"2024-02-19T21:20:00.482954+0000","last_fullsized":"2024-02-19T21:20:00.482954+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:42.381654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:42.381654+0000","last_clean_scrub_stamp":"2024-02-19T21:19:42.381654+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1","version":"801'8","reported_seq":94,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.228742+0000","last_change":"2024-02-19T21:19:14.948411+0000","last_active":"2024-02-19T21:20:00.228742+0000","last_peered":"2024-02-19T21:20:00.228742+0000","last_clean":"2024-02-19T21:20:00.228742+0000","last_became_active":"2024-02-19T21:17:29.511622+0000","last_became_peered":"2024-02-19T21:17:29.511622+0000","last_unstale":"2024-02-19T21:20:00.228742+0000","last_undegraded":"2024-02-19T21:20:00.228742+0000","last_fullsized":"2024-02-19T21:20:00.228742+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"801'8","last_scrub_stamp":"2024-02-19T21:19:14.948334+0000","last_deep_scrub":"801'8","last_deep_scrub_stamp":"2024-02-19T21:17:44.622190+0000","last_clean_scrub_stamp":"2024-02-19T21:19:14.948334+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":20,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"459'15","reported_seq":40,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.970160+0000","last_change":"2024-02-19T21:18:42.235565+0000","last_active":"2024-02-19T21:19:40.970160+0000","last_peered":"2024-02-19T21:19:40.970160+0000","last_clean":"2024-02-19T21:19:40.970160+0000","last_became_active":"2024-02-19T21:18:41.543916+0000","last_became_peered":"2024-02-19T21:18:41.543916+0000","last_unstale":"2024-02-19T21:19:40.970160+0000","last_undegraded":"2024-02-19T21:19:40.970160+0000","last_fullsized":"2024-02-19T21:19:40.970160+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'15","last_scrub_stamp":"2024-02-19T21:18:29.527734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:18:29.527734+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":288,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967930+0000","last_change":"2024-02-19T21:19:07.512940+0000","last_active":"2024-02-19T21:19:40.967930+0000","last_peered":"2024-02-19T21:19:40.967930+0000","last_clean":"2024-02-19T21:19:40.967930+0000","last_became_active":"2024-02-19T21:17:58.682002+0000","last_became_peered":"2024-02-19T21:17:58.682002+0000","last_unstale":"2024-02-19T21:19:40.967930+0000","last_undegraded":"2024-02-19T21:19:40.967930+0000","last_fullsized":"2024-02-19T21:19:40.967930+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:07.512855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:07.512855+0000","last_clean_scrub_stamp":"2024-02-19T21:19:07.512855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":603,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852111+0000","last_change":"2024-02-19T21:19:23.326855+0000","last_active":"2024-02-19T21:19:40.852111+0000","last_peered":"2024-02-19T21:19:40.852111+0000","last_clean":"2024-02-19T21:19:40.852111+0000","last_became_active":"2024-02-19T21:14:41.398865+0000","last_became_peered":"2024-02-19T21:14:41.398865+0000","last_unstale":"2024-02-19T21:19:40.852111+0000","last_undegraded":"2024-02-19T21:19:40.852111+0000","last_fullsized":"2024-02-19T21:19:40.852111+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:23.326783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:08:12.856015+0000","last_clean_scrub_stamp":"2024-02-19T21:19:23.326783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":516,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:52.189992+0000","last_change":"2024-02-19T21:19:52.189967+0000","last_active":"2024-02-19T21:19:52.189992+0000","last_peered":"2024-02-19T21:19:52.189992+0000","last_clean":"2024-02-19T21:19:52.189992+0000","last_became_active":"2024-02-19T21:18:19.169406+0000","last_became_peered":"2024-02-19T21:18:19.169406+0000","last_unstale":"2024-02-19T21:19:52.189992+0000","last_undegraded":"2024-02-19T21:19:52.189992+0000","last_fullsized":"2024-02-19T21:19:52.189992+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:52.189887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:52.189887+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":302,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:47.528440+0000","last_change":"2024-02-19T21:19:47.528413+0000","last_active":"2024-02-19T21:19:47.528440+0000","last_peered":"2024-02-19T21:19:47.528440+0000","last_clean":"2024-02-19T21:19:47.528440+0000","last_became_active":"2024-02-19T21:17:58.675970+0000","last_became_peered":"2024-02-19T21:17:58.675970+0000","last_unstale":"2024-02-19T21:19:47.528440+0000","last_undegraded":"2024-02-19T21:19:47.528440+0000","last_fullsized":"2024-02-19T21:19:47.528440+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:47.528343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:37.674820+0000","last_clean_scrub_stamp":"2024-02-19T21:19:47.528343+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"853'24","reported_seq":516,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.481739+0000","last_change":"2024-02-19T21:19:45.458867+0000","last_active":"2024-02-19T21:20:00.481739+0000","last_peered":"2024-02-19T21:20:00.481739+0000","last_clean":"2024-02-19T21:20:00.481739+0000","last_became_active":"2024-02-19T21:14:16.380020+0000","last_became_peered":"2024-02-19T21:14:16.380020+0000","last_unstale":"2024-02-19T21:20:00.481739+0000","last_undegraded":"2024-02-19T21:20:00.481739+0000","last_fullsized":"2024-02-19T21:20:00.481739+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"853'24","last_scrub_stamp":"2024-02-19T21:19:45.458789+0000","last_deep_scrub":"853'24","last_deep_scrub_stamp":"2024-02-19T21:19:45.458789+0000","last_clean_scrub_stamp":"2024-02-19T21:19:45.458789+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":886,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852347+0000","last_change":"2024-02-19T21:19:40.760340+0000","last_active":"2024-02-19T21:19:40.852347+0000","last_peered":"2024-02-19T21:19:40.852347+0000","last_clean":"2024-02-19T21:19:40.852347+0000","last_became_active":"2024-02-19T21:17:58.694655+0000","last_became_peered":"2024-02-19T21:17:58.694655+0000","last_unstale":"2024-02-19T21:19:40.852347+0000","last_undegraded":"2024-02-19T21:19:40.852347+0000","last_fullsized":"2024-02-19T21:19:40.852347+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:40.760279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:22.364374+0000","last_clean_scrub_stamp":"2024-02-19T21:19:40.760279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":873,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:48.706854+0000","last_change":"2024-02-19T21:19:48.706829+0000","last_active":"2024-02-19T21:19:48.706854+0000","last_peered":"2024-02-19T21:19:48.706854+0000","last_clean":"2024-02-19T21:19:48.706854+0000","last_became_active":"2024-02-19T21:17:58.687747+0000","last_became_peered":"2024-02-19T21:17:58.687747+0000","last_unstale":"2024-02-19T21:19:48.706854+0000","last_undegraded":"2024-02-19T21:19:48.706854+0000","last_fullsized":"2024-02-19T21:19:48.706854+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:48.706764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:48.706764+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":625,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:48.617052+0000","last_change":"2024-02-19T21:19:48.617028+0000","last_active":"2024-02-19T21:19:48.617052+0000","last_peered":"2024-02-19T21:19:48.617052+0000","last_clean":"2024-02-19T21:19:48.617052+0000","last_became_active":"2024-02-19T21:17:58.674337+0000","last_became_peered":"2024-02-19T21:17:58.674337+0000","last_unstale":"2024-02-19T21:19:48.617052+0000","last_undegraded":"2024-02-19T21:19:48.617052+0000","last_fullsized":"2024-02-19T21:19:48.617052+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:48.616963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:48.616963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851545+0000","last_change":"2024-02-19T21:19:22.280953+0000","last_active":"2024-02-19T21:19:40.851545+0000","last_peered":"2024-02-19T21:19:40.851545+0000","last_clean":"2024-02-19T21:19:40.851545+0000","last_became_active":"2024-02-19T21:17:29.452050+0000","last_became_peered":"2024-02-19T21:17:29.452050+0000","last_unstale":"2024-02-19T21:19:40.851545+0000","last_undegraded":"2024-02-19T21:19:40.851545+0000","last_fullsized":"2024-02-19T21:19:40.851545+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:22.280883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:14:59.059572+0000","last_clean_scrub_stamp":"2024-02-19T21:19:22.280883+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"555'10","reported_seq":119,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852294+0000","last_change":"2024-02-19T21:19:37.268171+0000","last_active":"2024-02-19T21:19:40.852294+0000","last_peered":"2024-02-19T21:19:40.852294+0000","last_clean":"2024-02-19T21:19:40.852294+0000","last_became_active":"2024-02-19T21:16:58.576112+0000","last_became_peered":"2024-02-19T21:16:58.576112+0000","last_unstale":"2024-02-19T21:19:40.852294+0000","last_undegraded":"2024-02-19T21:19:40.852294+0000","last_fullsized":"2024-02-19T21:19:40.852294+0000","mapping_epoch":760,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":761,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'10","last_scrub_stamp":"2024-02-19T21:19:37.268103+0000","last_deep_scrub":"555'10","last_deep_scrub_stamp":"2024-02-19T21:13:12.409036+0000","last_clean_scrub_stamp":"2024-02-19T21:19:37.268103+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":64,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:43.684827+0000","last_change":"2024-02-19T21:19:43.684798+0000","last_active":"2024-02-19T21:19:43.684827+0000","last_peered":"2024-02-19T21:19:43.684827+0000","last_clean":"2024-02-19T21:19:43.684827+0000","last_became_active":"2024-02-19T21:17:58.688322+0000","last_became_peered":"2024-02-19T21:17:58.688322+0000","last_unstale":"2024-02-19T21:19:43.684827+0000","last_undegraded":"2024-02-19T21:19:43.684827+0000","last_fullsized":"2024-02-19T21:19:43.684827+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:43.684706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:43.684706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":270,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:49.545775+0000","last_change":"2024-02-19T21:19:49.545749+0000","last_active":"2024-02-19T21:19:49.545775+0000","last_peered":"2024-02-19T21:19:49.545775+0000","last_clean":"2024-02-19T21:19:49.545775+0000","last_became_active":"2024-02-19T21:18:03.725686+0000","last_became_peered":"2024-02-19T21:18:03.725686+0000","last_unstale":"2024-02-19T21:19:49.545775+0000","last_undegraded":"2024-02-19T21:19:49.545775+0000","last_fullsized":"2024-02-19T21:19:49.545775+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:49.545680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:49.545680+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":66,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967539+0000","last_change":"2024-02-19T21:19:13.598831+0000","last_active":"2024-02-19T21:19:40.967539+0000","last_peered":"2024-02-19T21:19:40.967539+0000","last_clean":"2024-02-19T21:19:40.967539+0000","last_became_active":"2024-02-19T21:19:10.740674+0000","last_became_peered":"2024-02-19T21:19:10.740674+0000","last_unstale":"2024-02-19T21:19:40.967539+0000","last_undegraded":"2024-02-19T21:19:40.967539+0000","last_fullsized":"2024-02-19T21:19:40.967539+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:13.598706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:13.598706+0000","last_clean_scrub_stamp":"2024-02-19T21:19:13.598706+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":612,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:42.443267+0000","last_change":"2024-02-19T21:19:42.443225+0000","last_active":"2024-02-19T21:19:42.443267+0000","last_peered":"2024-02-19T21:19:42.443267+0000","last_clean":"2024-02-19T21:19:42.443267+0000","last_became_active":"2024-02-19T21:18:03.737373+0000","last_became_peered":"2024-02-19T21:18:03.737373+0000","last_unstale":"2024-02-19T21:19:42.443267+0000","last_undegraded":"2024-02-19T21:19:42.443267+0000","last_fullsized":"2024-02-19T21:19:42.443267+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:42.443160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:42.443160+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"850'24","reported_seq":719,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852240+0000","last_change":"2024-02-19T21:19:34.253671+0000","last_active":"2024-02-19T21:19:40.852240+0000","last_peered":"2024-02-19T21:19:40.852240+0000","last_clean":"2024-02-19T21:19:40.852240+0000","last_became_active":"2024-02-19T21:18:03.722651+0000","last_became_peered":"2024-02-19T21:18:03.722651+0000","last_unstale":"2024-02-19T21:19:40.852240+0000","last_undegraded":"2024-02-19T21:19:40.852240+0000","last_fullsized":"2024-02-19T21:19:40.852240+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"850'24","last_scrub_stamp":"2024-02-19T21:19:34.253611+0000","last_deep_scrub":"850'24","last_deep_scrub_stamp":"2024-02-19T21:19:34.253611+0000","last_clean_scrub_stamp":"2024-02-19T21:19:34.253611+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":290,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969127+0000","last_change":"2024-02-19T21:18:49.664237+0000","last_active":"2024-02-19T21:19:40.969127+0000","last_peered":"2024-02-19T21:19:40.969127+0000","last_clean":"2024-02-19T21:19:40.969127+0000","last_became_active":"2024-02-19T21:17:58.683408+0000","last_became_peered":"2024-02-19T21:17:58.683408+0000","last_unstale":"2024-02-19T21:19:40.969127+0000","last_undegraded":"2024-02-19T21:19:40.969127+0000","last_fullsized":"2024-02-19T21:19:40.969127+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:49.664164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:49.664164+0000","last_clean_scrub_stamp":"2024-02-19T21:18:49.664164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":729,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.967956+0000","last_change":"2024-02-19T21:19:16.563780+0000","last_active":"2024-02-19T21:19:40.967956+0000","last_peered":"2024-02-19T21:19:40.967956+0000","last_clean":"2024-02-19T21:19:40.967956+0000","last_became_active":"2024-02-19T21:19:10.742227+0000","last_became_peered":"2024-02-19T21:19:10.742227+0000","last_unstale":"2024-02-19T21:19:40.967956+0000","last_undegraded":"2024-02-19T21:19:40.967956+0000","last_fullsized":"2024-02-19T21:19:40.967956+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:16.563699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:16.563699+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852194+0000","last_change":"2024-02-19T21:19:10.213745+0000","last_active":"2024-02-19T21:19:40.852194+0000","last_peered":"2024-02-19T21:19:40.852194+0000","last_clean":"2024-02-19T21:19:40.852194+0000","last_became_active":"2024-02-19T21:14:41.398484+0000","last_became_peered":"2024-02-19T21:14:41.398484+0000","last_unstale":"2024-02-19T21:19:40.852194+0000","last_undegraded":"2024-02-19T21:19:40.852194+0000","last_fullsized":"2024-02-19T21:19:40.852194+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:10.213671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:10.213671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":269,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968441+0000","last_change":"2024-02-19T21:19:18.345285+0000","last_active":"2024-02-19T21:19:40.968441+0000","last_peered":"2024-02-19T21:19:40.968441+0000","last_clean":"2024-02-19T21:19:40.968441+0000","last_became_active":"2024-02-19T21:18:19.166023+0000","last_became_peered":"2024-02-19T21:18:19.166023+0000","last_unstale":"2024-02-19T21:19:40.968441+0000","last_undegraded":"2024-02-19T21:19:40.968441+0000","last_fullsized":"2024-02-19T21:19:40.968441+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:18.345211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.664814+0000","last_clean_scrub_stamp":"2024-02-19T21:19:18.345211+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1e","version":"459'20","reported_seq":580,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968162+0000","last_change":"2024-02-19T21:19:26.355932+0000","last_active":"2024-02-19T21:19:40.968162+0000","last_peered":"2024-02-19T21:19:40.968162+0000","last_clean":"2024-02-19T21:19:40.968162+0000","last_became_active":"2024-02-19T21:18:19.174836+0000","last_became_peered":"2024-02-19T21:18:19.174836+0000","last_unstale":"2024-02-19T21:19:40.968162+0000","last_undegraded":"2024-02-19T21:19:40.968162+0000","last_fullsized":"2024-02-19T21:19:40.968162+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'20","last_scrub_stamp":"2024-02-19T21:19:26.355824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:26.355824+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":73,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:49.658000+0000","last_change":"2024-02-19T21:19:49.657969+0000","last_active":"2024-02-19T21:19:49.658000+0000","last_peered":"2024-02-19T21:19:49.658000+0000","last_clean":"2024-02-19T21:19:49.658000+0000","last_became_active":"2024-02-19T21:18:03.734371+0000","last_became_peered":"2024-02-19T21:18:03.734371+0000","last_unstale":"2024-02-19T21:19:49.658000+0000","last_undegraded":"2024-02-19T21:19:49.658000+0000","last_fullsized":"2024-02-19T21:19:49.658000+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:49.657890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:29.454663+0000","last_clean_scrub_stamp":"2024-02-19T21:19:49.657890+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"816'15","reported_seq":365,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:19:59.924522+0000","last_change":"2024-02-19T21:19:11.435110+0000","last_active":"2024-02-19T21:19:59.924522+0000","last_peered":"2024-02-19T21:19:59.924522+0000","last_clean":"2024-02-19T21:19:59.924522+0000","last_became_active":"2024-02-19T21:18:03.726129+0000","last_became_peered":"2024-02-19T21:18:03.726129+0000","last_unstale":"2024-02-19T21:19:59.924522+0000","last_undegraded":"2024-02-19T21:19:59.924522+0000","last_fullsized":"2024-02-19T21:19:59.924522+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"816'15","last_scrub_stamp":"2024-02-19T21:19:11.435014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:11.435014+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"95'2","reported_seq":36,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969211+0000","last_change":"2024-02-19T21:19:10.740904+0000","last_active":"2024-02-19T21:19:40.969211+0000","last_peered":"2024-02-19T21:19:40.969211+0000","last_clean":"2024-02-19T21:19:40.969211+0000","last_became_active":"2024-02-19T21:19:10.740395+0000","last_became_peered":"2024-02-19T21:19:10.740395+0000","last_unstale":"2024-02-19T21:19:40.969211+0000","last_undegraded":"2024-02-19T21:19:40.969211+0000","last_fullsized":"2024-02-19T21:19:40.969211+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-19T21:19:07.545452+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-19T21:19:07.545452+0000","last_clean_scrub_stamp":"2024-02-19T21:19:07.545452+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":98,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:02.241142+0000","last_change":"2024-02-19T21:20:02.241119+0000","last_active":"2024-02-19T21:20:02.241142+0000","last_peered":"2024-02-19T21:20:02.241142+0000","last_clean":"2024-02-19T21:20:02.241142+0000","last_became_active":"2024-02-19T21:18:03.742082+0000","last_became_peered":"2024-02-19T21:18:03.742082+0000","last_unstale":"2024-02-19T21:20:02.241142+0000","last_undegraded":"2024-02-19T21:20:02.241142+0000","last_fullsized":"2024-02-19T21:20:02.241142+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:20:02.241054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:20:02.241054+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1f","version":"459'19","reported_seq":311,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:54.544131+0000","last_change":"2024-02-19T21:19:54.544099+0000","last_active":"2024-02-19T21:19:54.544131+0000","last_peered":"2024-02-19T21:19:54.544131+0000","last_clean":"2024-02-19T21:19:54.544131+0000","last_became_active":"2024-02-19T21:17:58.682999+0000","last_became_peered":"2024-02-19T21:17:58.682999+0000","last_unstale":"2024-02-19T21:19:54.544131+0000","last_undegraded":"2024-02-19T21:19:54.544131+0000","last_fullsized":"2024-02-19T21:19:54.544131+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:19:54.544019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:19:54.544019+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852324+0000","last_change":"2024-02-19T21:19:29.208508+0000","last_active":"2024-02-19T21:19:40.852324+0000","last_peered":"2024-02-19T21:19:40.852324+0000","last_clean":"2024-02-19T21:19:40.852324+0000","last_became_active":"2024-02-19T21:14:41.402278+0000","last_became_peered":"2024-02-19T21:14:41.402278+0000","last_unstale":"2024-02-19T21:19:40.852324+0000","last_undegraded":"2024-02-19T21:19:40.852324+0000","last_fullsized":"2024-02-19T21:19:40.852324+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:29.208404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:29.208404+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":492,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968080+0000","last_change":"2024-02-19T21:19:01.353521+0000","last_active":"2024-02-19T21:19:40.968080+0000","last_peered":"2024-02-19T21:19:40.968080+0000","last_clean":"2024-02-19T21:19:40.968080+0000","last_became_active":"2024-02-19T21:18:19.175181+0000","last_became_peered":"2024-02-19T21:18:19.175181+0000","last_unstale":"2024-02-19T21:19:40.968080+0000","last_undegraded":"2024-02-19T21:19:40.968080+0000","last_fullsized":"2024-02-19T21:19:40.968080+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:01.353430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:01.353430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":866,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:44.691611+0000","last_change":"2024-02-19T21:19:44.691588+0000","last_active":"2024-02-19T21:19:44.691611+0000","last_peered":"2024-02-19T21:19:44.691611+0000","last_clean":"2024-02-19T21:19:44.691611+0000","last_became_active":"2024-02-19T21:17:58.695090+0000","last_became_peered":"2024-02-19T21:17:58.695090+0000","last_unstale":"2024-02-19T21:19:44.691611+0000","last_undegraded":"2024-02-19T21:19:44.691611+0000","last_fullsized":"2024-02-19T21:19:44.691611+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:44.691518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:44.691518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":820,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851572+0000","last_change":"2024-02-19T21:19:01.398622+0000","last_active":"2024-02-19T21:19:40.851572+0000","last_peered":"2024-02-19T21:19:40.851572+0000","last_clean":"2024-02-19T21:19:40.851572+0000","last_became_active":"2024-02-19T21:17:29.530233+0000","last_became_peered":"2024-02-19T21:17:29.530233+0000","last_unstale":"2024-02-19T21:19:40.851572+0000","last_undegraded":"2024-02-19T21:19:40.851572+0000","last_fullsized":"2024-02-19T21:19:40.851572+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:01.398540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:01.398540+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"459'34","reported_seq":105,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.229467+0000","last_change":"2024-02-19T21:19:52.317077+0000","last_active":"2024-02-19T21:20:00.229467+0000","last_peered":"2024-02-19T21:20:00.229467+0000","last_clean":"2024-02-19T21:20:00.229467+0000","last_became_active":"2024-02-19T21:17:29.511788+0000","last_became_peered":"2024-02-19T21:17:29.511788+0000","last_unstale":"2024-02-19T21:20:00.229467+0000","last_undegraded":"2024-02-19T21:20:00.229467+0000","last_fullsized":"2024-02-19T21:20:00.229467+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'34","last_scrub_stamp":"2024-02-19T21:19:52.317003+0000","last_deep_scrub":"459'34","last_deep_scrub_stamp":"2024-02-19T21:18:30.676424+0000","last_clean_scrub_stamp":"2024-02-19T21:19:52.317003+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":306,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.969665+0000","last_change":"2024-02-19T21:19:40.508731+0000","last_active":"2024-02-19T21:19:40.969665+0000","last_peered":"2024-02-19T21:19:40.969665+0000","last_clean":"2024-02-19T21:19:40.969665+0000","last_became_active":"2024-02-19T21:18:41.542346+0000","last_became_peered":"2024-02-19T21:18:41.542346+0000","last_unstale":"2024-02-19T21:19:40.969665+0000","last_undegraded":"2024-02-19T21:19:40.969665+0000","last_fullsized":"2024-02-19T21:19:40.969665+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:40.508619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:40.508619+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":859,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852482+0000","last_change":"2024-02-19T21:18:56.474437+0000","last_active":"2024-02-19T21:19:40.852482+0000","last_peered":"2024-02-19T21:19:40.852482+0000","last_clean":"2024-02-19T21:19:40.852482+0000","last_became_active":"2024-02-19T21:14:41.415463+0000","last_became_peered":"2024-02-19T21:14:41.415463+0000","last_unstale":"2024-02-19T21:19:40.852482+0000","last_undegraded":"2024-02-19T21:19:40.852482+0000","last_fullsized":"2024-02-19T21:19:40.852482+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:56.474372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:18:56.474372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":481,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.483834+0000","last_change":"2024-02-19T21:19:07.534257+0000","last_active":"2024-02-19T21:20:00.483834+0000","last_peered":"2024-02-19T21:20:00.483834+0000","last_clean":"2024-02-19T21:20:00.483834+0000","last_became_active":"2024-02-19T21:17:58.675678+0000","last_became_peered":"2024-02-19T21:17:58.675678+0000","last_unstale":"2024-02-19T21:20:00.483834+0000","last_undegraded":"2024-02-19T21:20:00.483834+0000","last_fullsized":"2024-02-19T21:20:00.483834+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:07.534181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:03:12.081227+0000","last_clean_scrub_stamp":"2024-02-19T21:19:07.534181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":73,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.966885+0000","last_change":"2024-02-19T21:19:29.288567+0000","last_active":"2024-02-19T21:19:40.966885+0000","last_peered":"2024-02-19T21:19:40.966885+0000","last_clean":"2024-02-19T21:19:40.966885+0000","last_became_active":"2024-02-19T21:18:19.169386+0000","last_became_peered":"2024-02-19T21:18:19.169386+0000","last_unstale":"2024-02-19T21:19:40.966885+0000","last_undegraded":"2024-02-19T21:19:40.966885+0000","last_fullsized":"2024-02-19T21:19:40.966885+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:29.288497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:29.288497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1d","version":"459'18","reported_seq":648,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:02.402812+0000","last_change":"2024-02-19T21:20:02.402812+0000","last_active":"2024-02-19T21:20:02.402812+0000","last_peered":"2024-02-19T21:20:02.402812+0000","last_clean":"2024-02-19T21:20:02.402812+0000","last_became_active":"2024-02-19T21:10:41.307690+0000","last_became_peered":"2024-02-19T21:10:41.307690+0000","last_unstale":"2024-02-19T21:20:02.402812+0000","last_undegraded":"2024-02-19T21:20:02.402812+0000","last_fullsized":"2024-02-19T21:20:02.402812+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'18","last_scrub_stamp":"2024-02-19T21:20:02.402706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:20:02.402706+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852299+0000","last_change":"2024-02-19T21:19:05.235928+0000","last_active":"2024-02-19T21:19:40.852299+0000","last_peered":"2024-02-19T21:19:40.852299+0000","last_clean":"2024-02-19T21:19:40.852299+0000","last_became_active":"2024-02-19T21:18:03.731257+0000","last_became_peered":"2024-02-19T21:18:03.731257+0000","last_unstale":"2024-02-19T21:19:40.852299+0000","last_undegraded":"2024-02-19T21:19:40.852299+0000","last_fullsized":"2024-02-19T21:19:40.852299+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:05.235851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:05.235851+0000","last_clean_scrub_stamp":"2024-02-19T21:19:05.235851+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":472,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.482642+0000","last_change":"2024-02-19T21:19:39.378918+0000","last_active":"2024-02-19T21:20:00.482642+0000","last_peered":"2024-02-19T21:20:00.482642+0000","last_clean":"2024-02-19T21:20:00.482642+0000","last_became_active":"2024-02-19T21:17:29.471059+0000","last_became_peered":"2024-02-19T21:17:29.471059+0000","last_unstale":"2024-02-19T21:20:00.482642+0000","last_undegraded":"2024-02-19T21:20:00.482642+0000","last_fullsized":"2024-02-19T21:20:00.482642+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:39.378799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:39.378799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851631+0000","last_change":"2024-02-19T21:19:20.246947+0000","last_active":"2024-02-19T21:19:40.851631+0000","last_peered":"2024-02-19T21:19:40.851631+0000","last_clean":"2024-02-19T21:19:40.851631+0000","last_became_active":"2024-02-19T21:17:58.674931+0000","last_became_peered":"2024-02-19T21:17:58.674931+0000","last_unstale":"2024-02-19T21:19:40.851631+0000","last_undegraded":"2024-02-19T21:19:40.851631+0000","last_fullsized":"2024-02-19T21:19:40.851631+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:20.246873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:54.723156+0000","last_clean_scrub_stamp":"2024-02-19T21:19:20.246873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":811,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852429+0000","last_change":"2024-02-19T21:19:20.706643+0000","last_active":"2024-02-19T21:19:40.852429+0000","last_peered":"2024-02-19T21:19:40.852429+0000","last_clean":"2024-02-19T21:19:40.852429+0000","last_became_active":"2024-02-19T21:17:29.521224+0000","last_became_peered":"2024-02-19T21:17:29.521224+0000","last_unstale":"2024-02-19T21:19:40.852429+0000","last_undegraded":"2024-02-19T21:19:40.852429+0000","last_fullsized":"2024-02-19T21:19:40.852429+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:20.706564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:20.706564+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"459'19","reported_seq":497,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.484232+0000","last_change":"2024-02-19T21:18:54.534380+0000","last_active":"2024-02-19T21:20:00.484232+0000","last_peered":"2024-02-19T21:20:00.484232+0000","last_clean":"2024-02-19T21:20:00.484232+0000","last_became_active":"2024-02-19T21:14:41.390502+0000","last_became_peered":"2024-02-19T21:14:41.390502+0000","last_unstale":"2024-02-19T21:20:00.484232+0000","last_undegraded":"2024-02-19T21:20:00.484232+0000","last_fullsized":"2024-02-19T21:20:00.484232+0000","mapping_epoch":636,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'19","last_scrub_stamp":"2024-02-19T21:18:54.534294+0000","last_deep_scrub":"194'13","last_deep_scrub_stamp":"2024-02-19T21:06:14.772417+0000","last_clean_scrub_stamp":"2024-02-19T21:18:54.534294+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":37,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:58.353057+0000","last_change":"2024-02-19T21:19:58.353026+0000","last_active":"2024-02-19T21:19:58.353057+0000","last_peered":"2024-02-19T21:19:58.353057+0000","last_clean":"2024-02-19T21:19:58.353057+0000","last_became_active":"2024-02-19T21:19:10.740286+0000","last_became_peered":"2024-02-19T21:19:10.740286+0000","last_unstale":"2024-02-19T21:19:58.353057+0000","last_undegraded":"2024-02-19T21:19:58.353057+0000","last_fullsized":"2024-02-19T21:19:58.353057+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:58.352947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:16:30.005724+0000","last_clean_scrub_stamp":"2024-02-19T21:19:58.352947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":73,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:45.679539+0000","last_change":"2024-02-19T21:19:45.679492+0000","last_active":"2024-02-19T21:19:45.679539+0000","last_peered":"2024-02-19T21:19:45.679539+0000","last_clean":"2024-02-19T21:19:45.679539+0000","last_became_active":"2024-02-19T21:18:03.735544+0000","last_became_peered":"2024-02-19T21:18:03.735544+0000","last_unstale":"2024-02-19T21:19:45.679539+0000","last_undegraded":"2024-02-19T21:19:45.679539+0000","last_fullsized":"2024-02-19T21:19:45.679539+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:45.679388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:19.270373+0000","last_clean_scrub_stamp":"2024-02-19T21:19:45.679388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":692,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.966954+0000","last_change":"2024-02-19T21:18:53.320550+0000","last_active":"2024-02-19T21:19:40.966954+0000","last_peered":"2024-02-19T21:19:40.966954+0000","last_clean":"2024-02-19T21:19:40.966954+0000","last_became_active":"2024-02-19T21:18:19.174191+0000","last_became_peered":"2024-02-19T21:18:19.174191+0000","last_unstale":"2024-02-19T21:19:40.966954+0000","last_undegraded":"2024-02-19T21:19:40.966954+0000","last_fullsized":"2024-02-19T21:19:40.966954+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:53.320479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:25.785366+0000","last_clean_scrub_stamp":"2024-02-19T21:18:53.320479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":89,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.229498+0000","last_change":"2024-02-19T21:19:47.303158+0000","last_active":"2024-02-19T21:20:00.229498+0000","last_peered":"2024-02-19T21:20:00.229498+0000","last_clean":"2024-02-19T21:20:00.229498+0000","last_became_active":"2024-02-19T21:17:29.523066+0000","last_became_peered":"2024-02-19T21:17:29.523066+0000","last_unstale":"2024-02-19T21:20:00.229498+0000","last_undegraded":"2024-02-19T21:20:00.229498+0000","last_fullsized":"2024-02-19T21:20:00.229498+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:47.303090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:47.303090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"459'7","reported_seq":454,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:03.595519+0000","last_change":"2024-02-19T21:20:03.595488+0000","last_active":"2024-02-19T21:20:03.595519+0000","last_peered":"2024-02-19T21:20:03.595519+0000","last_clean":"2024-02-19T21:20:03.595519+0000","last_became_active":"2024-02-19T21:10:41.297295+0000","last_became_peered":"2024-02-19T21:10:41.297295+0000","last_unstale":"2024-02-19T21:20:03.595519+0000","last_undegraded":"2024-02-19T21:20:03.595519+0000","last_fullsized":"2024-02-19T21:20:03.595519+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'7","last_scrub_stamp":"2024-02-19T21:20:03.595411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:20:03.595411+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":36,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.229167+0000","last_change":"2024-02-19T21:18:42.754006+0000","last_active":"2024-02-19T21:20:00.229167+0000","last_peered":"2024-02-19T21:20:00.229167+0000","last_clean":"2024-02-19T21:20:00.229167+0000","last_became_active":"2024-02-19T21:18:41.543717+0000","last_became_peered":"2024-02-19T21:18:41.543717+0000","last_unstale":"2024-02-19T21:20:00.229167+0000","last_undegraded":"2024-02-19T21:20:00.229167+0000","last_fullsized":"2024-02-19T21:20:00.229167+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:42.753934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:18:42.753934+0000","last_clean_scrub_stamp":"2024-02-19T21:18:42.753934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":526,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:42.487367+0000","last_change":"2024-02-19T21:19:42.487341+0000","last_active":"2024-02-19T21:19:42.487367+0000","last_peered":"2024-02-19T21:19:42.487367+0000","last_clean":"2024-02-19T21:19:42.487367+0000","last_became_active":"2024-02-19T21:19:10.740847+0000","last_became_peered":"2024-02-19T21:19:10.740847+0000","last_unstale":"2024-02-19T21:19:42.487367+0000","last_undegraded":"2024-02-19T21:19:42.487367+0000","last_fullsized":"2024-02-19T21:19:42.487367+0000","mapping_epoch":848,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":849,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:42.487268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:42.487268+0000","last_clean_scrub_stamp":"2024-02-19T21:19:42.487268+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":64,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968092+0000","last_change":"2024-02-19T21:19:26.630405+0000","last_active":"2024-02-19T21:19:40.968092+0000","last_peered":"2024-02-19T21:19:40.968092+0000","last_clean":"2024-02-19T21:19:40.968092+0000","last_became_active":"2024-02-19T21:17:58.685724+0000","last_became_peered":"2024-02-19T21:17:58.685724+0000","last_unstale":"2024-02-19T21:19:40.968092+0000","last_undegraded":"2024-02-19T21:19:40.968092+0000","last_fullsized":"2024-02-19T21:19:40.968092+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:26.630297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:26.630297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":525,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968704+0000","last_change":"2024-02-19T21:19:00.334681+0000","last_active":"2024-02-19T21:19:40.968704+0000","last_peered":"2024-02-19T21:19:40.968704+0000","last_clean":"2024-02-19T21:19:40.968704+0000","last_became_active":"2024-02-19T21:18:19.164839+0000","last_became_peered":"2024-02-19T21:18:19.164839+0000","last_unstale":"2024-02-19T21:19:40.968704+0000","last_undegraded":"2024-02-19T21:19:40.968704+0000","last_fullsized":"2024-02-19T21:19:40.968704+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:00.334607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:00.334607+0000","last_clean_scrub_stamp":"2024-02-19T21:19:00.334607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"623'19","reported_seq":508,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.481586+0000","last_change":"2024-02-19T21:19:11.523034+0000","last_active":"2024-02-19T21:20:00.481586+0000","last_peered":"2024-02-19T21:20:00.481586+0000","last_clean":"2024-02-19T21:20:00.481586+0000","last_became_active":"2024-02-19T21:14:16.383577+0000","last_became_peered":"2024-02-19T21:14:16.383577+0000","last_unstale":"2024-02-19T21:20:00.481586+0000","last_undegraded":"2024-02-19T21:20:00.481586+0000","last_fullsized":"2024-02-19T21:20:00.481586+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"623'19","last_scrub_stamp":"2024-02-19T21:19:11.522944+0000","last_deep_scrub":"623'19","last_deep_scrub_stamp":"2024-02-19T21:14:21.589138+0000","last_clean_scrub_stamp":"2024-02-19T21:19:11.522944+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":634,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.851797+0000","last_change":"2024-02-19T21:19:13.152305+0000","last_active":"2024-02-19T21:19:40.851797+0000","last_peered":"2024-02-19T21:19:40.851797+0000","last_clean":"2024-02-19T21:19:40.851797+0000","last_became_active":"2024-02-19T21:17:58.673528+0000","last_became_peered":"2024-02-19T21:17:58.673528+0000","last_unstale":"2024-02-19T21:19:40.851797+0000","last_undegraded":"2024-02-19T21:19:40.851797+0000","last_fullsized":"2024-02-19T21:19:40.851797+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:13.152235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:13.152235+0000","last_clean_scrub_stamp":"2024-02-19T21:19:13.152235+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":873,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:55.703307+0000","last_change":"2024-02-19T21:19:55.703282+0000","last_active":"2024-02-19T21:19:55.703307+0000","last_peered":"2024-02-19T21:19:55.703307+0000","last_clean":"2024-02-19T21:19:55.703307+0000","last_became_active":"2024-02-19T21:18:03.726183+0000","last_became_peered":"2024-02-19T21:18:03.726183+0000","last_unstale":"2024-02-19T21:19:55.703307+0000","last_undegraded":"2024-02-19T21:19:55.703307+0000","last_fullsized":"2024-02-19T21:19:55.703307+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:55.703203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:06:24.702197+0000","last_clean_scrub_stamp":"2024-02-19T21:19:55.703203+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":472,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.484243+0000","last_change":"2024-02-19T21:19:59.557979+0000","last_active":"2024-02-19T21:20:00.484243+0000","last_peered":"2024-02-19T21:20:00.484243+0000","last_clean":"2024-02-19T21:20:00.484243+0000","last_became_active":"2024-02-19T21:17:58.679633+0000","last_became_peered":"2024-02-19T21:17:58.679633+0000","last_unstale":"2024-02-19T21:20:00.484243+0000","last_undegraded":"2024-02-19T21:20:00.484243+0000","last_fullsized":"2024-02-19T21:20:00.484243+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:59.557898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:19:59.557898+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":64,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.968533+0000","last_change":"2024-02-19T21:19:22.669178+0000","last_active":"2024-02-19T21:19:40.968533+0000","last_peered":"2024-02-19T21:19:40.968533+0000","last_clean":"2024-02-19T21:19:40.968533+0000","last_became_active":"2024-02-19T21:17:58.678930+0000","last_became_peered":"2024-02-19T21:17:58.678930+0000","last_unstale":"2024-02-19T21:19:40.968533+0000","last_undegraded":"2024-02-19T21:19:40.968533+0000","last_fullsized":"2024-02-19T21:19:40.968533+0000","mapping_epoch":809,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:22.669094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:22.669094+0000","last_clean_scrub_stamp":"2024-02-19T21:19:22.669094+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"459'27","reported_seq":741,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852000+0000","last_change":"2024-02-19T21:18:58.210897+0000","last_active":"2024-02-19T21:19:40.852000+0000","last_peered":"2024-02-19T21:19:40.852000+0000","last_clean":"2024-02-19T21:19:40.852000+0000","last_became_active":"2024-02-19T21:17:29.452764+0000","last_became_peered":"2024-02-19T21:17:29.452764+0000","last_unstale":"2024-02-19T21:19:40.852000+0000","last_undegraded":"2024-02-19T21:19:40.852000+0000","last_fullsized":"2024-02-19T21:19:40.852000+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'27","last_scrub_stamp":"2024-02-19T21:18:58.210821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:17.258309+0000","last_clean_scrub_stamp":"2024-02-19T21:18:58.210821+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":630,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:45.177663+0000","last_change":"2024-02-19T21:19:45.177620+0000","last_active":"2024-02-19T21:19:45.177663+0000","last_peered":"2024-02-19T21:19:45.177663+0000","last_clean":"2024-02-19T21:19:45.177663+0000","last_became_active":"2024-02-19T21:18:19.172145+0000","last_became_peered":"2024-02-19T21:18:19.172145+0000","last_unstale":"2024-02-19T21:19:45.177663+0000","last_undegraded":"2024-02-19T21:19:45.177663+0000","last_fullsized":"2024-02-19T21:19:45.177663+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:45.177506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:15.241725+0000","last_clean_scrub_stamp":"2024-02-19T21:19:45.177506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":463,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-02-19T21:20:00.481297+0000","last_change":"2024-02-19T21:19:20.569004+0000","last_active":"2024-02-19T21:20:00.481297+0000","last_peered":"2024-02-19T21:20:00.481297+0000","last_clean":"2024-02-19T21:20:00.481297+0000","last_became_active":"2024-02-19T21:14:16.384069+0000","last_became_peered":"2024-02-19T21:14:16.384069+0000","last_unstale":"2024-02-19T21:20:00.481297+0000","last_undegraded":"2024-02-19T21:20:00.481297+0000","last_fullsized":"2024-02-19T21:20:00.481297+0000","mapping_epoch":619,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":620,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:20.568921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:19:20.568921+0000","last_clean_scrub_stamp":"2024-02-19T21:19:20.568921+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":616,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:40.852033+0000","last_change":"2024-02-19T21:18:51.303957+0000","last_active":"2024-02-19T21:19:40.852033+0000","last_peered":"2024-02-19T21:19:40.852033+0000","last_clean":"2024-02-19T21:19:40.852033+0000","last_became_active":"2024-02-19T21:18:03.731332+0000","last_became_peered":"2024-02-19T21:18:03.731332+0000","last_unstale":"2024-02-19T21:19:40.852033+0000","last_undegraded":"2024-02-19T21:19:40.852033+0000","last_fullsized":"2024-02-19T21:19:40.852033+0000","mapping_epoch":814,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":815,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:18:51.303860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:01:21.273400+0000","last_clean_scrub_stamp":"2024-02-19T21:18:51.303860+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":270,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-02-19T21:19:45.500128+0000","last_change":"2024-02-19T21:19:45.500073+0000","last_active":"2024-02-19T21:19:45.500128+0000","last_peered":"2024-02-19T21:19:45.500128+0000","last_clean":"2024-02-19T21:19:45.500128+0000","last_became_active":"2024-02-19T21:17:29.464415+0000","last_became_peered":"2024-02-19T21:17:29.464415+0000","last_unstale":"2024-02-19T21:19:45.500128+0000","last_undegraded":"2024-02-19T21:19:45.500128+0000","last_fullsized":"2024-02-19T21:19:45.500128+0000","mapping_epoch":785,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":786,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T21:19:45.499995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T21:04:38.494818+0000","last_clean_scrub_stamp":"2024-02-19T21:19:45.499995+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1100,"num_read_kb":857,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":716,"num_bytes_recovered":14412,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":767,"ondisk_log_size":767,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1112,"num_read_kb":1107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":42,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":2342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":126,"acting":126,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":20,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":53,"seq":227633266917,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307220,"kb_used_data":1804,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437932,"statfs":{"total":95995035648,"available":95680442368,"internally_reserved":0,"allocated":1847296,"data_stored":576368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.50700000000000001,"15min":0.86899999999999999},"min":{"1min":0.29499999999999998,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":0.70999999999999996,"5min":1.069,"15min":89.456999999999994},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54900000000000004,"15min":0.80800000000000005},"min":{"1min":0.39100000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.75600000000000001,"5min":4.9560000000000004,"15min":70.451999999999998},"last":0.438}]},{"osd":1,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.628,"15min":0.73799999999999999},"min":{"1min":0.33900000000000002,"5min":0.33700000000000002,"15min":0.28199999999999997},"max":{"1min":0.71599999999999997,"5min":7.0590000000000002,"15min":34.273000000000003},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51200000000000001,"15min":0.93600000000000005},"min":{"1min":0.34999999999999998,"5min":0.255,"15min":0.254},"max":{"1min":0.73699999999999999,"5min":1.1020000000000001,"15min":89.594999999999999},"last":0.61899999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56499999999999995,"15min":0.97999999999999998},"min":{"1min":0.31,"5min":0.29799999999999999,"15min":0.27600000000000002},"max":{"1min":0.64800000000000002,"5min":4.9000000000000004,"15min":104.413},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.59799999999999998,"15min":0.97999999999999998},"min":{"1min":0.40200000000000002,"5min":0.32100000000000001,"15min":0.27600000000000002},"max":{"1min":0.74399999999999999,"5min":6.9039999999999999,"15min":104.268},"last":0.66000000000000003}]},{"osd":3,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.621,"15min":0.998},"min":{"1min":0.38300000000000001,"5min":0.29999999999999999,"15min":0.29099999999999998},"max":{"1min":0.747,"5min":7.984,"15min":105.3},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.63,"15min":0.65700000000000003},"min":{"1min":0.36199999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.67100000000000004,"5min":10.853,"15min":13.243},"last":0.67800000000000005}]},{"osd":4,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.496,"15min":0.45500000000000002},"min":{"1min":0.35199999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.78400000000000003,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58699999999999997,"15min":0.50700000000000001},"min":{"1min":0.376,"5min":0.28699999999999998,"15min":0.26600000000000001},"max":{"1min":0.88900000000000001,"5min":4.6500000000000004,"15min":4.6500000000000004},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.50800000000000001,"15min":0.46600000000000003},"min":{"1min":0.40000000000000002,"5min":0.27900000000000003,"15min":0.252},"max":{"1min":0.81000000000000005,"5min":4.1289999999999996,"15min":4.1289999999999996},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55200000000000005,"15min":0.51100000000000001},"min":{"1min":0.442,"5min":0.30099999999999999,"15min":0.27100000000000002},"max":{"1min":0.85599999999999998,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.72899999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58099999999999996,"15min":0.52900000000000003},"min":{"1min":0.44500000000000001,"5min":0.313,"15min":0.28799999999999998},"max":{"1min":0.84899999999999998,"5min":3.597,"15min":3.597},"last":0.749},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58999999999999997,"15min":0.55200000000000005},"min":{"1min":0.501,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.878,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.755}]}]},{"osd":6,"up_from":45,"seq":193273528555,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303104,"kb_used_data":1784,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93442048,"statfs":{"total":95995035648,"available":95684657152,"internally_reserved":0,"allocated":1826816,"data_stored":575710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.60999999999999999},"min":{"1min":0.34399999999999997,"5min":0.249,"15min":0.19900000000000001},"max":{"1min":0.71599999999999997,"5min":0.73099999999999998,"15min":17.486999999999998},"last":0.436},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.49299999999999999,"15min":0.65700000000000003},"min":{"1min":0.44,"5min":0.26800000000000002,"15min":0.215},"max":{"1min":0.69899999999999995,"5min":0.71899999999999997,"15min":17.43},"last":0.59399999999999997}]},{"osd":1,"last update":"Mon Feb 19 21:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.498,"15min":0.625},"min":{"1min":0.39100000000000001,"5min":0.248,"15min":0.22800000000000001},"max":{"1min":0.747,"5min":0.747,"15min":14.692},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.49299999999999999,"15min":0.66600000000000004},"min":{"1min":0.38400000000000001,"5min":0.24099999999999999,"15min":0.23400000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":14.795},"last":0.61399999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.49099999999999999,"15min":0.60299999999999998},"min":{"1min":0.34999999999999998,"5min":0.32600000000000001,"15min":0.253},"max":{"1min":0.65800000000000003,"5min":0.72199999999999998,"15min":6.931},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.498,"15min":0.67600000000000005},"min":{"1min":0.376,"5min":0.33300000000000002,"15min":0.26000000000000001},"max":{"1min":0.68500000000000005,"5min":0.83099999999999996,"15min":14.391},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.50700000000000001,"15min":0.70299999999999996},"min":{"1min":0.41999999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":14.510999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59399999999999997,"15min":0.69199999999999995},"min":{"1min":0.41399999999999998,"5min":0.33100000000000002,"15min":0.316},"max":{"1min":0.73699999999999999,"5min":4.085,"15min":17.585000000000001},"last":0.52900000000000003}]},{"osd":4,"last update":"Mon Feb 19 21:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.41999999999999998,"15min":0.42599999999999999},"min":{"1min":0.318,"5min":0.24399999999999999,"15min":0.22900000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":2.9289999999999998},"last":0.499},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.45400000000000001,"15min":0.45800000000000002},"min":{"1min":0.34399999999999997,"5min":0.28799999999999998,"15min":0.22700000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":3.431},"last":0.51400000000000001}]},{"osd":5,"last update":"Mon Feb 19 21:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.48299999999999998,"15min":0.48199999999999998},"min":{"1min":0.41299999999999998,"5min":0.27500000000000002,"15min":0.26300000000000001},"max":{"1min":0.64800000000000002,"5min":1.778,"15min":2.9620000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.52000000000000002,"15min":0.51600000000000001},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.26700000000000002},"max":{"1min":0.77900000000000003,"5min":1.788,"15min":2.7010000000000001},"last":0.57099999999999995}]},{"osd":7,"last update":"Mon Feb 19 21:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.48899999999999999,"15min":0.49199999999999999},"min":{"1min":0.40699999999999997,"5min":0.28000000000000003,"15min":0.26000000000000001},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":1.7170000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.502,"15min":0.495},"min":{"1min":0.432,"5min":0.30299999999999999,"15min":0.28399999999999997},"max":{"1min":0.71099999999999997,"5min":0.72699999999999998,"15min":1.835},"last":0.58499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575106,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307240,"kb_used_data":1824,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437912,"statfs":{"total":95995035648,"available":95680421888,"internally_reserved":0,"allocated":1867776,"data_stored":575921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:19:30 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.33400000000000002,"15min":0.53400000000000003},"min":{"1min":0.215,"5min":0.20300000000000001,"15min":0.16700000000000001},"max":{"1min":0.41999999999999998,"5min":1.288,"15min":27.777000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.56999999999999995},"min":{"1min":0.25600000000000001,"5min":0.19900000000000001,"15min":0.19400000000000001},"max":{"1min":0.627,"5min":1.736,"15min":27.631},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Feb 19 21:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.371,"15min":0.68300000000000005},"min":{"1min":0.26700000000000002,"5min":0.20000000000000001,"15min":0.19600000000000001},"max":{"1min":0.53700000000000003,"5min":0.97499999999999998,"15min":27.798999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.373,"15min":0.69999999999999996},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.56100000000000005,"5min":1.7729999999999999,"15min":27.606999999999999},"last":0.442}]},{"osd":3,"last update":"Mon Feb 19 21:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.54600000000000004},"min":{"1min":0.29599999999999999,"5min":0.22700000000000001,"15min":0.217},"max":{"1min":0.55400000000000005,"5min":0.75,"15min":15.574999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.40600000000000003,"15min":0.64700000000000002},"min":{"1min":0.28799999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.51300000000000001,"5min":0.70499999999999996,"15min":27.82},"last":0.52100000000000002}]},{"osd":4,"last update":"Mon Feb 19 21:19:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54800000000000004,"15min":0.64200000000000002},"min":{"1min":0.38400000000000001,"5min":0.27400000000000002,"15min":0.26800000000000002},"max":{"1min":0.67100000000000004,"5min":1.262,"15min":9.6430000000000007},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57899999999999996,"15min":0.751},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.308},"max":{"1min":0.73699999999999999,"5min":0.84899999999999998,"15min":27.707000000000001},"last":0.872}]},{"osd":5,"last update":"Mon Feb 19 21:20:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.56999999999999995,"15min":0.72999999999999998},"min":{"1min":0.33700000000000002,"5min":0.315,"15min":0.29999999999999999},"max":{"1min":0.79400000000000004,"5min":1.829,"15min":27.529},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56399999999999995,"15min":0.65400000000000003},"min":{"1min":0.42999999999999999,"5min":0.35299999999999998,"15min":0.316},"max":{"1min":0.88800000000000001,"5min":1.228,"15min":9.6799999999999997},"last":0.79300000000000004}]},{"osd":6,"last update":"Mon Feb 19 21:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.55300000000000005,"15min":0.64600000000000002},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71699999999999997,"5min":1.147,"15min":9.6989999999999998},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59599999999999997,"15min":0.77500000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.35199999999999998},"max":{"1min":0.78400000000000003,"5min":0.876,"15min":27.733000000000001},"last":0.90700000000000003}]},{"osd":7,"last update":"Mon Feb 19 21:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.58899999999999997,"15min":0.77100000000000002},"min":{"1min":0.40200000000000002,"5min":0.36799999999999999,"15min":0.36699999999999999},"max":{"1min":0.84399999999999997,"5min":1.8029999999999999,"15min":27.576000000000001},"last":0.751},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61199999999999999,"15min":0.80300000000000005},"min":{"1min":0.437,"5min":0.38400000000000001,"15min":0.375},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":27.756},"last":0.85299999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738632,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299020,"kb_used_data":1796,"kb_used_omap":0,"kb_used_meta":297216,"kb_avail":93446132,"statfs":{"total":95995035648,"available":95688839168,"internally_reserved":0,"allocated":1839104,"data_stored":574834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304349184},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 21:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.35899999999999999,"15min":0.61099999999999999},"min":{"1min":0.23200000000000001,"5min":0.19400000000000001,"15min":0.188},"max":{"1min":2.6720000000000002,"5min":2.6720000000000002,"15min":20.966999999999999},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.34399999999999997,"15min":0.63},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.193},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":20.366},"last":0.29599999999999999}]},{"osd":2,"last update":"Mon Feb 19 21:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.35299999999999998,"15min":0.69399999999999995},"min":{"1min":0.22900000000000001,"5min":0.22800000000000001,"15min":0.19900000000000001},"max":{"1min":2.2429999999999999,"5min":2.2429999999999999,"15min":27.916},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.372,"15min":0.60999999999999999},"min":{"1min":0.25700000000000001,"5min":0.23799999999999999,"15min":0.19600000000000001},"max":{"1min":1.968,"5min":1.968,"15min":22.004000000000001},"last":0.26100000000000001}]},{"osd":3,"last update":"Mon Feb 19 21:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.40200000000000002,"15min":0.71999999999999997},"min":{"1min":0.251,"5min":0.219,"15min":0.219},"max":{"1min":2.3220000000000001,"5min":2.3220000000000001,"15min":25.669},"last":0.315},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.40699999999999997,"15min":0.66500000000000004},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.26300000000000001},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":25.006},"last":0.28799999999999998}]},{"osd":4,"last update":"Mon Feb 19 21:19:23 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.56599999999999995,"15min":0.73799999999999999},"min":{"1min":0.40699999999999997,"5min":0.36099999999999999,"15min":0.28799999999999998},"max":{"1min":2.157,"5min":2.157,"15min":20.411999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.59799999999999998,"15min":0.77000000000000002},"min":{"1min":0.40799999999999997,"5min":0.309,"15min":0.28699999999999998},"max":{"1min":2.2999999999999998,"5min":2.2999999999999998,"15min":19.986999999999998},"last":0.52600000000000002}]},{"osd":5,"last update":"Mon Feb 19 21:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.58799999999999997,"15min":0.80000000000000004},"min":{"1min":0.42399999999999999,"5min":0.36199999999999999,"15min":0.27200000000000002},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":20.021999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.61199999999999999,"15min":0.79300000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.30399999999999999},"max":{"1min":2.032,"5min":2.032,"15min":20.908999999999999},"last":0.56499999999999995}]},{"osd":6,"last update":"Mon Feb 19 21:19:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.58899999999999997,"15min":0.75700000000000001},"min":{"1min":0.41099999999999998,"5min":0.34100000000000003,"15min":0.33900000000000002},"max":{"1min":2.1299999999999999,"5min":2.1299999999999999,"15min":20.445},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.61199999999999999,"15min":0.79600000000000004},"min":{"1min":0.42699999999999999,"5min":0.35199999999999998,"15min":0.33400000000000002},"max":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":20.047999999999998},"last":0.57199999999999995}]},{"osd":7,"last update":"Mon Feb 19 21:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.61299999999999999,"15min":0.77900000000000003},"min":{"1min":0.45100000000000001,"5min":0.39900000000000002,"15min":0.312},"max":{"1min":2.004,"5min":2.004,"15min":20.940999999999999},"last":0.622},{"interface":"front","average":{"1min":0.629,"5min":0.59599999999999997,"15min":0.80400000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.34899999999999998},"max":{"1min":2.181,"5min":2.181,"15min":20.471},"last":0.55900000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411581,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303104,"kb_used_data":1784,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93442048,"statfs":{"total":95995035648,"available":95684657152,"internally_reserved":0,"allocated":1826816,"data_stored":575080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:19:54 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.66800000000000004,"15min":0.62},"min":{"1min":0.223,"5min":0.183,"15min":0.183},"max":{"1min":0.53100000000000003,"5min":28.702999999999999,"15min":28.702999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.221,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.45000000000000001,"5min":28.744,"15min":28.744},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Feb 19 21:19:54 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.50700000000000001,"15min":0.73399999999999999},"min":{"1min":0.22600000000000001,"5min":0.20999999999999999,"15min":0.20599999999999999},"max":{"1min":0.47999999999999998,"5min":9.7940000000000005,"15min":28.882000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.41599999999999998,"15min":0.66500000000000004},"min":{"1min":0.26000000000000001,"5min":0.21299999999999999,"15min":0.17999999999999999},"max":{"1min":0.505,"5min":5.0739999999999998,"15min":28.382999999999999},"last":0.308}]},{"osd":3,"last update":"Mon Feb 19 21:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.47399999999999998,"15min":0.71999999999999997},"min":{"1min":0.34399999999999997,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.56599999999999995,"5min":4.2270000000000003,"15min":28.742000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.47999999999999998,"15min":0.65200000000000002},"min":{"1min":0.28599999999999998,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":0.60599999999999998,"5min":6.8040000000000003,"15min":26.719999999999999},"last":0.377}]},{"osd":4,"last update":"Mon Feb 19 21:19:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.56999999999999995,"15min":0.67000000000000004},"min":{"1min":0.38700000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.84399999999999997,"5min":2.4079999999999999,"15min":24.356000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.67600000000000005,"15min":0.73699999999999999},"min":{"1min":0.47499999999999998,"5min":0.36399999999999999,"15min":0.32800000000000001},"max":{"1min":0.84499999999999997,"5min":4.3540000000000001,"15min":20.707000000000001},"last":0.56899999999999995}]},{"osd":5,"last update":"Mon Feb 19 21:20:00 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.66200000000000003,"15min":0.68500000000000005},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.81499999999999995,"5min":4.2990000000000004,"15min":9.9499999999999993},"last":0.497},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60999999999999999,"15min":0.75600000000000001},"min":{"1min":0.39900000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.74199999999999999,"5min":2.8380000000000001,"15min":26.606000000000002},"last":0.44700000000000001}]},{"osd":6,"last update":"Mon Feb 19 21:19:36 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.60599999999999998,"15min":0.78900000000000003},"min":{"1min":0.41699999999999998,"5min":0.36499999999999999,"15min":0.34300000000000003},"max":{"1min":0.90800000000000003,"5min":2.198,"15min":24.381},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.69699999999999995,"15min":0.75800000000000001},"min":{"1min":0.41399999999999998,"5min":0.36099999999999999,"15min":0.35999999999999999},"max":{"1min":0.86599999999999999,"5min":4.2759999999999998,"15min":28.706},"last":0.54900000000000004}]},{"osd":7,"last update":"Mon Feb 19 21:19:54 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.627,"15min":0.77200000000000002},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.34200000000000003},"max":{"1min":0.87,"5min":2.6440000000000001,"15min":26.658000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61699999999999999,"15min":0.71399999999999997},"min":{"1min":0.42299999999999999,"5min":0.38700000000000001,"15min":0.35399999999999998},"max":{"1min":0.80300000000000005,"5min":2.3700000000000001,"15min":24.402000000000001},"last":0.48599999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182649,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303132,"kb_used_data":1812,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":93442020,"statfs":{"total":95995035648,"available":95684628480,"internally_reserved":0,"allocated":1855488,"data_stored":575320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33300000000000002,"15min":0.70899999999999996},"min":{"1min":0.23300000000000001,"5min":0.20300000000000001,"15min":0.156},"max":{"1min":0.51000000000000001,"5min":0.70199999999999996,"15min":77.662000000000006},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.34599999999999997,"15min":0.47999999999999998},"min":{"1min":0.254,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.499,"5min":0.60399999999999998,"15min":17.456},"last":0.25600000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.34699999999999998,"15min":0.51500000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.56399999999999995,"5min":0.57999999999999996,"15min":17.327999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.35599999999999998,"15min":0.496},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.20000000000000001},"max":{"1min":0.502,"5min":0.98999999999999999,"15min":15.004},"last":0.312}]},{"osd":2,"last update":"Mon Feb 19 21:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.38500000000000001,"15min":0.65300000000000002},"min":{"1min":0.252,"5min":0.23499999999999999,"15min":0.219},"max":{"1min":0.51900000000000002,"5min":0.71199999999999997,"15min":56.890999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.374,"5min":0.373,"15min":0.63600000000000001},"min":{"1min":0.29399999999999998,"5min":0.24299999999999999,"15min":0.20899999999999999},"max":{"1min":0.52100000000000002,"5min":0.81200000000000006,"15min":44.600999999999999},"last":0.32500000000000001}]},{"osd":4,"last update":"Mon Feb 19 21:19:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54400000000000004,"15min":0.63300000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.31900000000000001},"max":{"1min":0.93700000000000006,"5min":0.94699999999999995,"15min":12.847},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.55000000000000004,"15min":0.70099999999999996},"min":{"1min":0.35499999999999998,"5min":0.29899999999999999,"15min":0.29299999999999998},"max":{"1min":0.82999999999999996,"5min":0.89600000000000002,"15min":26.393999999999998},"last":0.53200000000000003}]},{"osd":5,"last update":"Mon Feb 19 21:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.56599999999999995,"15min":0.67800000000000005},"min":{"1min":0.443,"5min":0.34899999999999998,"15min":0.313},"max":{"1min":0.80300000000000005,"5min":1.002,"15min":17.401},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.55600000000000005,"15min":0.65300000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.313},"max":{"1min":0.78300000000000003,"5min":0.871,"15min":12.882},"last":0.48899999999999999}]},{"osd":6,"last update":"Mon Feb 19 21:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.56499999999999995,"15min":0.65500000000000003},"min":{"1min":0.41799999999999998,"5min":0.373,"15min":0.34799999999999998},"max":{"1min":0.84299999999999997,"5min":1.036,"15min":17.367000000000001},"last":0.624},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56599999999999995,"15min":0.65700000000000003},"min":{"1min":0.40100000000000002,"5min":0.35699999999999998,"15min":0.33700000000000002},"max":{"1min":0.72399999999999998,"5min":0.876,"15min":12.904},"last":0.50900000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58199999999999996,"15min":0.66100000000000003},"min":{"1min":0.46400000000000002,"5min":0.39700000000000002,"15min":0.35499999999999998},"max":{"1min":0.67900000000000005,"5min":0.91500000000000004,"15min":17.428999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.59799999999999998,"15min":0.81000000000000005},"min":{"1min":0.42099999999999999,"5min":0.40400000000000003,"15min":0.34499999999999997},"max":{"1min":0.67700000000000005,"5min":1.1859999999999999,"15min":44.378},"last":0.55500000000000005}]}]},{"osd":4,"up_from":31,"seq":133143986421,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307216,"kb_used_data":1800,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437936,"statfs":{"total":95995035648,"available":95680446464,"internally_reserved":0,"allocated":1843200,"data_stored":575333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.61099999999999999,"15min":0.66200000000000003},"min":{"1min":0.32100000000000001,"5min":0.27200000000000002,"15min":0.23599999999999999},"max":{"1min":0.75900000000000001,"5min":7.4109999999999996,"15min":19.878},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.58399999999999996,"15min":0.88300000000000001},"min":{"1min":0.35799999999999998,"5min":0.249,"15min":0.20499999999999999},"max":{"1min":0.74199999999999999,"5min":3.4079999999999999,"15min":70.063000000000002},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Feb 19 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.64900000000000002,"15min":0.67200000000000004},"min":{"1min":0.38900000000000001,"5min":0.30599999999999999,"15min":0.19700000000000001},"max":{"1min":0.70599999999999996,"5min":10.086,"15min":26.510999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54100000000000004,"15min":0.69399999999999995},"min":{"1min":0.40999999999999998,"5min":0.29799999999999999,"15min":0.23200000000000001},"max":{"1min":0.72799999999999998,"5min":2.2719999999999998,"15min":25.949000000000002},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Feb 19 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.54200000000000004,"15min":0.72699999999999998},"min":{"1min":0.46000000000000002,"5min":0.29499999999999998,"15min":0.26700000000000002},"max":{"1min":0.79000000000000004,"5min":1.7729999999999999,"15min":37.320999999999998},"last":0.5},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.97599999999999998,"15min":1.2050000000000001},"min":{"1min":0.40400000000000003,"5min":0.309,"15min":0.253},"max":{"1min":0.71799999999999997,"5min":23.504999999999999,"15min":69.936999999999998},"last":0.58899999999999997}]},{"osd":3,"last update":"Mon Feb 19 21:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.69499999999999995,"15min":0.97199999999999998},"min":{"1min":0.438,"5min":0.33400000000000002,"15min":0.27700000000000002},"max":{"1min":0.70999999999999996,"5min":11.500999999999999,"15min":69.793000000000006},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.71399999999999997,"15min":0.81000000000000005},"min":{"1min":0.46800000000000003,"5min":0.33800000000000002,"15min":0.28299999999999997},"max":{"1min":0.75,"5min":6.1970000000000001,"15min":17.986999999999998},"last":0.57399999999999995}]},{"osd":5,"last update":"Mon Feb 19 21:19:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48199999999999998,"15min":0.443},"min":{"1min":0.32500000000000001,"5min":0.26400000000000001,"15min":0.22900000000000001},"max":{"1min":0.64400000000000002,"5min":4.7439999999999998,"15min":4.7439999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54000000000000004,"15min":0.499},"min":{"1min":0.32900000000000001,"5min":0.30399999999999999,"15min":0.252},"max":{"1min":0.76600000000000001,"5min":4.7130000000000001,"15min":4.7130000000000001},"last":0.60199999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.50900000000000001,"15min":0.51200000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.23499999999999999},"max":{"1min":0.65200000000000002,"5min":3.3140000000000001,"15min":8.0969999999999995},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58199999999999996,"15min":0.56299999999999994},"min":{"1min":0.34599999999999997,"5min":0.307,"15min":0.26800000000000002},"max":{"1min":0.69899999999999995,"5min":3.3679999999999999,"15min":4.5609999999999999},"last":0.63300000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:19:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57999999999999996,"15min":0.56100000000000005},"min":{"1min":0.35199999999999998,"5min":0.32100000000000001,"15min":0.27600000000000002},"max":{"1min":0.749,"5min":4.4779999999999998,"15min":6.0730000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55800000000000005,"15min":0.54600000000000004},"min":{"1min":0.42699999999999999,"5min":0.313,"15min":0.26400000000000001},"max":{"1min":0.79200000000000004,"5min":0.79300000000000004,"15min":3.202},"last":0.60999999999999999}]}]},{"osd":5,"up_from":37,"seq":158913790192,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307204,"kb_used_data":1788,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":93437948,"statfs":{"total":95995035648,"available":95680458752,"internally_reserved":0,"allocated":1830912,"data_stored":576109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 21:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49399999999999999,"15min":1.877},"min":{"1min":0.33200000000000002,"5min":0.26400000000000001,"15min":0.19500000000000001},"max":{"1min":0.67100000000000004,"5min":0.748,"15min":276.47899999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51900000000000002,"15min":0.89200000000000002},"min":{"1min":0.32400000000000001,"5min":0.28999999999999998,"15min":0.20899999999999999},"max":{"1min":0.68500000000000005,"5min":0.99299999999999999,"15min":52.887},"last":0.55100000000000005}]},{"osd":1,"last update":"Mon Feb 19 21:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.50700000000000001,"15min":1.2609999999999999},"min":{"1min":0.374,"5min":0.27100000000000002,"15min":0.23499999999999999},"max":{"1min":0.629,"5min":0.75,"15min":120.011},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52700000000000002,"15min":1.016},"min":{"1min":0.314,"5min":0.314,"15min":0.26200000000000001},"max":{"1min":0.63900000000000001,"5min":0.89400000000000002,"15min":118.105},"last":0.57499999999999996}]},{"osd":2,"last update":"Mon Feb 19 21:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51700000000000002,"15min":0.98799999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.27200000000000002},"max":{"1min":0.628,"5min":0.73099999999999998,"15min":82.242000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52800000000000002,"15min":1.512},"min":{"1min":0.40699999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66700000000000004,"5min":0.79000000000000004,"15min":232.84999999999999},"last":0.52600000000000002}]},{"osd":3,"last update":"Mon Feb 19 21:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53600000000000003,"15min":1.069},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.27900000000000003},"max":{"1min":0.70599999999999996,"5min":0.80600000000000005,"15min":109.70099999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.55200000000000005,"15min":0.90900000000000003},"min":{"1min":0.40300000000000002,"5min":0.35499999999999998,"15min":0.32000000000000001},"max":{"1min":0.67700000000000005,"5min":0.70899999999999996,"15min":65.519000000000005},"last":0.434}]},{"osd":4,"last update":"Mon Feb 19 21:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.45900000000000002,"15min":0.442},"min":{"1min":0.35799999999999998,"5min":0.253,"15min":0.23300000000000001},"max":{"1min":0.61599999999999999,"5min":0.77200000000000002,"15min":1.861},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.48399999999999999,"15min":0.46200000000000002},"min":{"1min":0.373,"5min":0.27300000000000002,"15min":0.248},"max":{"1min":0.64800000000000002,"5min":0.72999999999999998,"15min":1.9730000000000001},"last":0.34699999999999998}]},{"osd":6,"last update":"Mon Feb 19 21:19:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47099999999999997,"15min":0.46800000000000003},"min":{"1min":0.28299999999999997,"5min":0.246,"15min":0.246},"max":{"1min":0.65000000000000002,"5min":0.95099999999999996,"15min":1.823},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.497,"15min":0.47999999999999998},"min":{"1min":0.29399999999999998,"5min":0.28799999999999998,"15min":0.27100000000000002},"max":{"1min":0.73399999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.38600000000000001}]},{"osd":7,"last update":"Mon Feb 19 21:19:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53600000000000003,"15min":0.51000000000000001},"min":{"1min":0.40500000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.73899999999999999,"5min":1.052,"15min":1.8879999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54100000000000004,"15min":0.51700000000000002},"min":{"1min":0.42499999999999999,"5min":0.34300000000000003,"15min":0.27800000000000002},"max":{"1min":0.73599999999999999,"5min":0.80500000000000005,"15min":0.96799999999999997},"last":0.42499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T21:20:08.732 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-19T21:20:08.733 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell --fsid 2591809a-cf69-11ee-95bb-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-02-19T21:20:09.003 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/mon.a/config 2024-02-19T21:20:09.249 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:07.383447+0000 mgr.y (mgr.14550) 750 : cluster [DBG] pgmap v1341: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:20:09.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:07.476362+0000 osd.3 (osd.3) 437 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:20:09.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:07.478211+0000 osd.3 (osd.3) 438 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:20:09.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:20:07.985944+0000 mgr.y (mgr.14550) 751 : audit [DBG] from='client.24192 v1:172.21.15.163:0/1916127829' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:20:09.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:07.383447+0000 mgr.y (mgr.14550) 750 : cluster [DBG] pgmap v1341: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:20:09.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:07.476362+0000 osd.3 (osd.3) 437 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:20:09.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:07.478211+0000 osd.3 (osd.3) 438 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:20:09.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:08 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:20:07.985944+0000 mgr.y (mgr.14550) 751 : audit [DBG] from='client.24192 v1:172.21.15.163:0/1916127829' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:20:09.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:07.383447+0000 mgr.y (mgr.14550) 750 : cluster [DBG] pgmap v1341: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-19T21:20:09.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:07.476362+0000 osd.3 (osd.3) 437 : cluster [DBG] 5.0 scrub starts 2024-02-19T21:20:09.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:07.478211+0000 osd.3 (osd.3) 438 : cluster [DBG] 5.0 scrub ok 2024-02-19T21:20:09.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:20:07.985944+0000 mgr.y (mgr.14550) 751 : audit [DBG] from='client.24192 v1:172.21.15.163:0/1916127829' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T21:20:10.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:08.203641+0000 osd.2 (osd.2) 402 : cluster [DBG] 16.17 scrub starts 2024-02-19T21:20:10.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:08.206318+0000 osd.2 (osd.2) 403 : cluster [DBG] 16.17 scrub ok 2024-02-19T21:20:10.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:08.467968+0000 osd.3 (osd.3) 439 : cluster [DBG] 2.28 deep-scrub starts 2024-02-19T21:20:10.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:08.468771+0000 osd.3 (osd.3) 440 : cluster [DBG] 2.28 deep-scrub ok 2024-02-19T21:20:10.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:08.795326+0000 osd.1 (osd.1) 441 : cluster [DBG] 16.b scrub starts 2024-02-19T21:20:10.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:08.799128+0000 osd.1 (osd.1) 442 : cluster [DBG] 16.b scrub ok 2024-02-19T21:20:10.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:09.458355+0000 osd.7 (osd.7) 373 : cluster [DBG] 6.3 deep-scrub starts 2024-02-19T21:20:10.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:09.461306+0000 osd.7 (osd.7) 374 : cluster [DBG] 6.3 deep-scrub ok 2024-02-19T21:20:10.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:08.203641+0000 osd.2 (osd.2) 402 : cluster [DBG] 16.17 scrub starts 2024-02-19T21:20:10.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:08.206318+0000 osd.2 (osd.2) 403 : cluster [DBG] 16.17 scrub ok 2024-02-19T21:20:10.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:08.467968+0000 osd.3 (osd.3) 439 : cluster [DBG] 2.28 deep-scrub starts 2024-02-19T21:20:10.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:08.468771+0000 osd.3 (osd.3) 440 : cluster [DBG] 2.28 deep-scrub ok 2024-02-19T21:20:10.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:08.795326+0000 osd.1 (osd.1) 441 : cluster [DBG] 16.b scrub starts 2024-02-19T21:20:10.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:08.799128+0000 osd.1 (osd.1) 442 : cluster [DBG] 16.b scrub ok 2024-02-19T21:20:10.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:09.458355+0000 osd.7 (osd.7) 373 : cluster [DBG] 6.3 deep-scrub starts 2024-02-19T21:20:10.251 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:09 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:09.461306+0000 osd.7 (osd.7) 374 : cluster [DBG] 6.3 deep-scrub ok 2024-02-19T21:20:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:08.203641+0000 osd.2 (osd.2) 402 : cluster [DBG] 16.17 scrub starts 2024-02-19T21:20:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:08.206318+0000 osd.2 (osd.2) 403 : cluster [DBG] 16.17 scrub ok 2024-02-19T21:20:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:08.467968+0000 osd.3 (osd.3) 439 : cluster [DBG] 2.28 deep-scrub starts 2024-02-19T21:20:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:08.468771+0000 osd.3 (osd.3) 440 : cluster [DBG] 2.28 deep-scrub ok 2024-02-19T21:20:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:08.795326+0000 osd.1 (osd.1) 441 : cluster [DBG] 16.b scrub starts 2024-02-19T21:20:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:08.799128+0000 osd.1 (osd.1) 442 : cluster [DBG] 16.b scrub ok 2024-02-19T21:20:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:09.458355+0000 osd.7 (osd.7) 373 : cluster [DBG] 6.3 deep-scrub starts 2024-02-19T21:20:10.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:09.461306+0000 osd.7 (osd.7) 374 : cluster [DBG] 6.3 deep-scrub ok 2024-02-19T21:20:10.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.827+0000 7fc64e01d700 1 -- 172.21.15.163:0/4096208263 >> v1:172.21.15.163:6790/0 conn(0x7fc648100780 legacy=0x7fc648102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:10.828 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.827+0000 7fc64e01d700 1 -- 172.21.15.163:0/4096208263 shutdown_connections 2024-02-19T21:20:10.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.827+0000 7fc64e01d700 1 -- 172.21.15.163:0/4096208263 >> 172.21.15.163:0/4096208263 conn(0x7fc6480f35a0 msgr2=0x7fc6480f59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:20:10.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.827+0000 7fc64e01d700 1 -- 172.21.15.163:0/4096208263 shutdown_connections 2024-02-19T21:20:10.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.827+0000 7fc64e01d700 1 -- 172.21.15.163:0/4096208263 wait complete. 2024-02-19T21:20:10.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.827+0000 7fc64e01d700 1 librados: starting msgr at 2024-02-19T21:20:10.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.827+0000 7fc64e01d700 1 librados: starting objecter 2024-02-19T21:20:10.829 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.827+0000 7fc64e01d700 1 Processor -- start 2024-02-19T21:20:10.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc64e01d700 1 -- start start 2024-02-19T21:20:10.831 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc64e01d700 1 librados: setting wanted keys 2024-02-19T21:20:10.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc64e01d700 1 librados: calling monclient init 2024-02-19T21:20:10.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc64e01d700 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6480fcf70 con 0x7fc6481042c0 2024-02-19T21:20:10.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc64e01d700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6480fd140 con 0x7fc6480fd980 2024-02-19T21:20:10.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc64e01d700 1 -- --> v1:172.21.15.163:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6480fd350 con 0x7fc648100780 2024-02-19T21:20:10.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc647fff700 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fc6481042c0 0x7fc648118440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:52478/0 (socket says 172.21.15.163:52478) 2024-02-19T21:20:10.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc647fff700 1 -- 172.21.15.163:0/161611972 learned_addr learned my addr 172.21.15.163:0/161611972 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-02-19T21:20:10.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc644ff9700 10 client.?.objecter ms_handle_connect 0x7fc6481042c0 2024-02-19T21:20:10.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc644ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:20:10.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc644ff9700 10 client.?.objecter ms_handle_connect 0x7fc648100780 2024-02-19T21:20:10.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc644ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:20:10.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.0 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842017833 0 0) 0x7fc6480fcf70 con 0x7fc6481042c0 2024-02-19T21:20:10.832 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc630003110 con 0x7fc6481042c0 2024-02-19T21:20:10.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc644ff9700 10 client.?.objecter ms_handle_connect 0x7fc6480fd980 2024-02-19T21:20:10.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc644ff9700 10 client.?.objecter resend_mon_ops 2024-02-19T21:20:10.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.2 v1:172.21.15.163:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758402815 0 0) 0x7fc6480fd350 con 0x7fc648100780 2024-02-19T21:20:10.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 --> v1:172.21.15.163:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6480fcf70 con 0x7fc648100780 2024-02-19T21:20:10.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304911820 0 0) 0x7fc6480fd140 con 0x7fc6480fd980 2024-02-19T21:20:10.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.828+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6480fd350 con 0x7fc6480fd980 2024-02-19T21:20:10.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.0 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3314261226 0 0) 0x7fc630003110 con 0x7fc6481042c0 2024-02-19T21:20:10.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6480fd140 con 0x7fc6481042c0 2024-02-19T21:20:10.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.0 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc640001c60 con 0x7fc6481042c0 2024-02-19T21:20:10.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.2 v1:172.21.15.163:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1086570923 0 0) 0x7fc6480fcf70 con 0x7fc648100780 2024-02-19T21:20:10.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 --> v1:172.21.15.163:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc630003110 con 0x7fc648100780 2024-02-19T21:20:10.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.2 v1:172.21.15.163:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc638002cf0 con 0x7fc648100780 2024-02-19T21:20:10.833 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1619329680 0 0) 0x7fc6480fd350 con 0x7fc6480fd980 2024-02-19T21:20:10.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6480fcf70 con 0x7fc6480fd980 2024-02-19T21:20:10.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc63c001880 con 0x7fc6480fd980 2024-02-19T21:20:10.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.0 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1386135003 0 0) 0x7fc6480fd140 con 0x7fc6481042c0 2024-02-19T21:20:10.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 >> v1:172.21.15.163:6790/0 conn(0x7fc648100780 legacy=0x7fc648114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:10.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 >> v1:172.21.15.193:6789/0 conn(0x7fc6480fd980 legacy=0x7fc6480fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:10.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64811cb90 con 0x7fc6481042c0 2024-02-19T21:20:10.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc64e01d700 1 -- 172.21.15.163:0/161611972 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc648119bc0 con 0x7fc6481042c0 2024-02-19T21:20:10.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc64e01d700 10 client.24198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-02-19T21:20:10.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc64e01d700 1 -- 172.21.15.163:0/161611972 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc64811a030 con 0x7fc6481042c0 2024-02-19T21:20:10.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc64e01d700 1 librados: init done 2024-02-19T21:20:10.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.0 v1:172.21.15.163:6789/0 5 ==== config(25 keys) v1 ==== 1080+0+0 (unknown 192281390 0 0) 0x7fc640002370 con 0x7fc6481042c0 2024-02-19T21:20:10.834 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.829+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.0 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1973004795 0 0) 0x7fc640005dd0 con 0x7fc6481042c0 2024-02-19T21:20:10.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.830+0000 7fc64e01d700 1 -- 172.21.15.163:0/161611972 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc634004f20 con 0x7fc6481042c0 2024-02-19T21:20:10.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.830+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.0 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 19) v1 ==== 77088+0+0 (unknown 3366196778 0 0) 0x7fc640001ed0 con 0x7fc6481042c0 2024-02-19T21:20:10.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.831+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.0 v1:172.21.15.163:6789/0 8 ==== osd_map(856..856 src has 806..856) v4 ==== 7419+0+0 (unknown 1041781516 0 0) 0x7fc640078d10 con 0x7fc6481042c0 2024-02-19T21:20:10.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.831+0000 7fc644ff9700 10 client.24198.objecter ms_dispatch 0x7fc648001550 osd_map(856..856 src has 806..856) v4 2024-02-19T21:20:10.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.831+0000 7fc644ff9700 3 client.24198.objecter handle_osd_map got epochs [856,856] > 0 2024-02-19T21:20:10.835 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.831+0000 7fc644ff9700 3 client.24198.objecter handle_osd_map decoding full epoch 856 2024-02-19T21:20:10.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.831+0000 7fc644ff9700 20 client.24198.objecter dump_active .. 0 homeless 2024-02-19T21:20:10.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.831+0000 7fc644ff9700 10 client.24198.objecter ms_handle_connect 0x7fc630063de0 2024-02-19T21:20:10.836 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:10.835+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.0 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+163658 (unknown 1092875540 0 292100661) 0x7fc64004c380 con 0x7fc6481042c0 2024-02-19T21:20:11.004 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:11.003+0000 7fc64e01d700 1 -- 172.21.15.163:0/161611972 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} v 0) v1 -- 0x7fc6340051d0 con 0x7fc6481042c0 2024-02-19T21:20:11.008 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:11.007+0000 7fc644ff9700 1 -- 172.21.15.163:0/161611972 <== mon.0 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]=0 v30) v1 ==== 118+0+0 (unknown 1095616568 0 0) 0x7fc64004a790 con 0x7fc6481042c0 2024-02-19T21:20:11.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:11.009+0000 7fc64e01d700 10 librados: watch_flush enter 2024-02-19T21:20:11.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:11.009+0000 7fc64e01d700 10 librados: watch_flush exit 2024-02-19T21:20:11.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:11.009+0000 7fc64e01d700 20 client.24198.objecter shutdown clearing up homeless session... 2024-02-19T21:20:11.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:11.009+0000 7fc64e01d700 10 client.24198.objecter successfully canceled tick 2024-02-19T21:20:11.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:11.009+0000 7fc64e01d700 1 -- 172.21.15.163:0/161611972 >> v1:172.21.15.163:6800/1552514681 conn(0x7fc630063de0 legacy=0x7fc6300662a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:11.010 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:11.009+0000 7fc64e01d700 1 -- 172.21.15.163:0/161611972 >> v1:172.21.15.163:6789/0 conn(0x7fc6481042c0 legacy=0x7fc648118440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T21:20:11.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:11.009+0000 7fc64e01d700 1 -- 172.21.15.163:0/161611972 shutdown_connections 2024-02-19T21:20:11.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:11.009+0000 7fc64e01d700 1 -- 172.21.15.163:0/161611972 >> 172.21.15.163:0/161611972 conn(0x7fc6480f35a0 msgr2=0x7fc648107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T21:20:11.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:11.010+0000 7fc64e01d700 1 -- 172.21.15.163:0/161611972 shutdown_connections 2024-02-19T21:20:11.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:11.010+0000 7fc64e01d700 1 -- 172.21.15.163:0/161611972 wait complete. 2024-02-19T21:20:11.011 INFO:teuthology.orchestra.run.smithi163.stderr:2024-02-19T21:20:11.010+0000 7fc64e01d700 1 librados: shutdown 2024-02-19T21:20:11.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:09.231551+0000 osd.2 (osd.2) 404 : cluster [DBG] 2.c scrub starts 2024-02-19T21:20:11.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:09.232559+0000 osd.2 (osd.2) 405 : cluster [DBG] 2.c scrub ok 2024-02-19T21:20:11.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:09.383813+0000 mgr.y (mgr.14550) 752 : cluster [DBG] pgmap v1342: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:20:11.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:10.376356+0000 osd.0 (osd.0) 206 : cluster [DBG] 6.18 deep-scrub starts 2024-02-19T21:20:11.250 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:10.379324+0000 osd.0 (osd.0) 207 : cluster [DBG] 6.18 deep-scrub ok 2024-02-19T21:20:11.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:09.231551+0000 osd.2 (osd.2) 404 : cluster [DBG] 2.c scrub starts 2024-02-19T21:20:11.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:09.232559+0000 osd.2 (osd.2) 405 : cluster [DBG] 2.c scrub ok 2024-02-19T21:20:11.250 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:09.383813+0000 mgr.y (mgr.14550) 752 : cluster [DBG] pgmap v1342: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:20:11.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:10.376356+0000 osd.0 (osd.0) 206 : cluster [DBG] 6.18 deep-scrub starts 2024-02-19T21:20:11.251 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:10.379324+0000 osd.0 (osd.0) 207 : cluster [DBG] 6.18 deep-scrub ok 2024-02-19T21:20:11.251 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:20:11] ENGINE Bus STOPPING 2024-02-19T21:20:11.251 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:20:11] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-19T21:20:11.251 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:20:11] ENGINE Bus STOPPED 2024-02-19T21:20:11.251 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:20:11] ENGINE Bus STARTING 2024-02-19T21:20:11.251 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:20:11] ENGINE Serving on http://:::9283 2024-02-19T21:20:11.251 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: [19/Feb/2024:21:20:11] ENGINE Bus STARTED 2024-02-19T21:20:11.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:20:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug there is no tcmu-runner data available 2024-02-19T21:20:11.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:09.231551+0000 osd.2 (osd.2) 404 : cluster [DBG] 2.c scrub starts 2024-02-19T21:20:11.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:09.232559+0000 osd.2 (osd.2) 405 : cluster [DBG] 2.c scrub ok 2024-02-19T21:20:11.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:09.383813+0000 mgr.y (mgr.14550) 752 : cluster [DBG] pgmap v1342: 174 pgs: 174 active+clean; 5.7 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-19T21:20:11.323 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:10.376356+0000 osd.0 (osd.0) 206 : cluster [DBG] 6.18 deep-scrub starts 2024-02-19T21:20:11.324 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:10.379324+0000 osd.0 (osd.0) 207 : cluster [DBG] 6.18 deep-scrub ok 2024-02-19T21:20:11.744 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-19T21:20:11.754 INFO:tasks.cephadm:Teardown begin 2024-02-19T21:20:11.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-19T21:20:11.784 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-19T21:20:11.822 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-19T21:20:11.822 DEBUG:teuthology.orchestra.run.smithi163:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T21:20:11.841 DEBUG:teuthology.orchestra.run.smithi193:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T21:20:11.882 INFO:tasks.cephadm:Stopping all daemons... 2024-02-19T21:20:11.882 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-19T21:20:11.882 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-2591809a-cf69-11ee-95bb-87774f69a715@mon.a 2024-02-19T21:20:12.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:10.553433+0000 osd.4 (osd.4) 371 : cluster [DBG] 6.e scrub starts 2024-02-19T21:20:12.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:10.554454+0000 osd.4 (osd.4) 372 : cluster [DBG] 6.e scrub ok 2024-02-19T21:20:12.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:10.847213+0000 osd.1 (osd.1) 443 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:20:12.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:10.848428+0000 osd.1 (osd.1) 444 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:20:12.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:20:10.947351+0000 mgr.y (mgr.14550) 753 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:20:12.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:20:11.003907+0000 mon.a (mon.0) 3887 : audit [INF] from='client.? v1:172.21.15.163:0/161611972' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-19T21:20:12.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:20:11.007346+0000 mon.a (mon.0) 3888 : audit [INF] from='client.? v1:172.21.15.163:0/161611972' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-19T21:20:12.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:20:11.039258+0000 mon.b (mon.1) 777 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:20:12.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:20:11.040777+0000 mon.b (mon.1) 778 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:20:12.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:20:11.041602+0000 mon.b (mon.1) 779 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:20:12.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: audit 2024-02-19T21:20:11.048506+0000 mon.a (mon.0) 3889 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:20:12.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:11.598426+0000 osd.5 (osd.5) 394 : cluster [DBG] 5.15 deep-scrub starts 2024-02-19T21:20:12.000 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: cluster 2024-02-19T21:20:11.602479+0000 osd.5 (osd.5) 395 : cluster [DBG] 5.15 deep-scrub ok 2024-02-19T21:20:12.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:11 smithi163 systemd[1]: Stopping Ceph mon.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:20:12.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:11 smithi163 bash[261506]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon.a" found: no such container 2024-02-19T21:20:12.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:10.553433+0000 osd.4 (osd.4) 371 : cluster [DBG] 6.e scrub starts 2024-02-19T21:20:12.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:10.554454+0000 osd.4 (osd.4) 372 : cluster [DBG] 6.e scrub ok 2024-02-19T21:20:12.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:10.847213+0000 osd.1 (osd.1) 443 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:20:12.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:10.848428+0000 osd.1 (osd.1) 444 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:20:12.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:20:10.947351+0000 mgr.y (mgr.14550) 753 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:20:12.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:20:11.003907+0000 mon.a (mon.0) 3887 : audit [INF] from='client.? v1:172.21.15.163:0/161611972' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-19T21:20:12.001 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:20:11.007346+0000 mon.a (mon.0) 3888 : audit [INF] from='client.? v1:172.21.15.163:0/161611972' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-19T21:20:12.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:20:11.039258+0000 mon.b (mon.1) 777 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:20:12.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:20:11.040777+0000 mon.b (mon.1) 778 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:20:12.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:20:11.041602+0000 mon.b (mon.1) 779 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:20:12.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: audit 2024-02-19T21:20:11.048506+0000 mon.a (mon.0) 3889 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:20:12.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:11.598426+0000 osd.5 (osd.5) 394 : cluster [DBG] 5.15 deep-scrub starts 2024-02-19T21:20:12.002 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:11 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: cluster 2024-02-19T21:20:11.602479+0000 osd.5 (osd.5) 395 : cluster [DBG] 5.15 deep-scrub ok 2024-02-19T21:20:12.076 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:10.553433+0000 osd.4 (osd.4) 371 : cluster [DBG] 6.e scrub starts 2024-02-19T21:20:12.077 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:10.554454+0000 osd.4 (osd.4) 372 : cluster [DBG] 6.e scrub ok 2024-02-19T21:20:12.077 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:10.847213+0000 osd.1 (osd.1) 443 : cluster [DBG] 5.1d scrub starts 2024-02-19T21:20:12.077 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:10.848428+0000 osd.1 (osd.1) 444 : cluster [DBG] 5.1d scrub ok 2024-02-19T21:20:12.078 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:20:10.947351+0000 mgr.y (mgr.14550) 753 : audit [DBG] from='client.34454 v1:172.21.15.193:0/1065798186' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-19T21:20:12.078 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:20:11.003907+0000 mon.a (mon.0) 3887 : audit [INF] from='client.? v1:172.21.15.163:0/161611972' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-19T21:20:12.078 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:20:11.007346+0000 mon.a (mon.0) 3888 : audit [INF] from='client.? v1:172.21.15.163:0/161611972' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-19T21:20:12.078 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:20:11.039258+0000 mon.b (mon.1) 777 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T21:20:12.078 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:20:11.040777+0000 mon.b (mon.1) 778 : audit [DBG] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T21:20:12.079 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:20:11.041602+0000 mon.b (mon.1) 779 : audit [INF] from='mgr.14550 v1:172.21.15.163:0/3673044579' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T21:20:12.079 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: audit 2024-02-19T21:20:11.048506+0000 mon.a (mon.0) 3889 : audit [INF] from='mgr.14550 ' entity='mgr.y' 2024-02-19T21:20:12.079 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:11.598426+0000 osd.5 (osd.5) 394 : cluster [DBG] 5.15 deep-scrub starts 2024-02-19T21:20:12.079 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: cluster 2024-02-19T21:20:11.602479+0000 osd.5 (osd.5) 395 : cluster [DBG] 5.15 deep-scrub ok 2024-02-19T21:20:12.489 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:20:12.182+0000 7fb60ce15700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T21:20:12.490 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:12 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a[38224]: debug 2024-02-19T21:20:12.182+0000 7fb60ce15700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-19T21:20:12.749 INFO:journalctl@ceph.mon.a.smithi163.stdout:Feb 19 21:20:12 smithi163 bash[261546]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-a 2024-02-19T21:20:12.995 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@mon.a.service' 2024-02-19T21:20:13.034 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T21:20:13.035 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-19T21:20:13.035 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-19T21:20:13.035 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-2591809a-cf69-11ee-95bb-87774f69a715@mon.c 2024-02-19T21:20:13.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:13 smithi163 systemd[1]: Stopping Ceph mon.c for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:20:13.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:13 smithi163 bash[261664]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon.c" found: no such container 2024-02-19T21:20:13.750 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:20:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: ::ffff:172.21.15.193 - - [19/Feb/2024:21:20:13] "GET /metrics HTTP/1.1" 200 187985 "" "Prometheus/2.33.4" 2024-02-19T21:20:13.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T21:20:13.353+0000 7f441f200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T21:20:13.750 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c[45145]: debug 2024-02-19T21:20:13.353+0000 7f441f200700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-02-19T21:20:14.249 INFO:journalctl@ceph.mon.c.smithi163.stdout:Feb 19 21:20:14 smithi163 bash[261702]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-c 2024-02-19T21:20:14.369 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@mon.c.service' 2024-02-19T21:20:14.405 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T21:20:14.405 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-19T21:20:14.406 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-19T21:20:14.406 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-2591809a-cf69-11ee-95bb-87774f69a715@mon.b 2024-02-19T21:20:14.727 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:14 smithi193 systemd[1]: Stopping Ceph mon.b for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:20:14.727 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:14 smithi193 bash[118539]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon.b" found: no such container 2024-02-19T21:20:15.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T21:20:14.724+0000 7f8afd421700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T21:20:15.073 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b[39315]: debug 2024-02-19T21:20:14.724+0000 7f8afd421700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-02-19T21:20:15.573 INFO:journalctl@ceph.mon.b.smithi193.stdout:Feb 19 21:20:15 smithi193 bash[118577]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-mon-b 2024-02-19T21:20:15.749 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@mon.b.service' 2024-02-19T21:20:15.796 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T21:20:15.797 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-19T21:20:15.797 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-19T21:20:15.797 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-2591809a-cf69-11ee-95bb-87774f69a715@mgr.y 2024-02-19T21:20:16.111 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:20:15 smithi163 systemd[1]: Stopping Ceph mgr.y for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:20:16.111 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:20:15 smithi163 bash[261819]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr.y" found: no such container 2024-02-19T21:20:16.424 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:20:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:20:16.109+0000 7f8d6a967700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-19T21:20:16.424 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:20:16 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y[38498]: debug 2024-02-19T21:20:16.109+0000 7f8d6a967700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-19T21:20:16.749 INFO:journalctl@ceph.mgr.y.smithi163.stdout:Feb 19 21:20:16 smithi163 bash[261858]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-y 2024-02-19T21:20:16.946 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@mgr.y.service' 2024-02-19T21:20:16.985 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T21:20:16.986 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-19T21:20:16.986 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-19T21:20:16.986 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-2591809a-cf69-11ee-95bb-87774f69a715@mgr.x 2024-02-19T21:20:17.323 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:20:17 smithi193 systemd[1]: Stopping Ceph mgr.x for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:20:17.323 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:20:17 smithi193 bash[118697]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr.x" found: no such container 2024-02-19T21:20:17.823 INFO:journalctl@ceph.mgr.x.smithi193.stdout:Feb 19 21:20:17 smithi193 bash[118737]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-mgr-x 2024-02-19T21:20:18.047 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@mgr.x.service' 2024-02-19T21:20:18.082 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T21:20:18.082 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-19T21:20:18.083 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-19T21:20:18.083 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.0 2024-02-19T21:20:18.366 INFO:journalctl@ceph.osd.0.smithi163.stdout:Feb 19 21:20:18 smithi163 systemd[1]: Stopping Ceph osd.0 for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:20:18.366 INFO:journalctl@ceph.osd.0.smithi163.stdout:Feb 19 21:20:18 smithi163 bash[261978]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.0" found: no such container 2024-02-19T21:20:18.749 INFO:journalctl@ceph.osd.0.smithi163.stdout:Feb 19 21:20:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-0[48619]: debug 2024-02-19T21:20:18.364+0000 7f63420c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-19T21:20:19.367 INFO:journalctl@ceph.osd.0.smithi163.stdout:Feb 19 21:20:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-0[48619]: debug 2024-02-19T21:20:18.364+0000 7f63420c1700 -1 osd.0 856 *** Got signal Terminated *** 2024-02-19T21:20:19.367 INFO:journalctl@ceph.osd.0.smithi163.stdout:Feb 19 21:20:18 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-0[48619]: debug 2024-02-19T21:20:18.364+0000 7f63420c1700 -1 osd.0 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T21:20:23.999 INFO:journalctl@ceph.osd.0.smithi163.stdout:Feb 19 21:20:23 smithi163 bash[262019]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-0 2024-02-19T21:20:25.740 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.0.service' 2024-02-19T21:20:25.781 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T21:20:25.781 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-19T21:20:25.782 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-19T21:20:25.782 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.1 2024-02-19T21:20:25.999 INFO:journalctl@ceph.osd.1.smithi163.stdout:Feb 19 21:20:25 smithi163 systemd[1]: Stopping Ceph osd.1 for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:20:26.000 INFO:journalctl@ceph.osd.1.smithi163.stdout:Feb 19 21:20:25 smithi163 bash[262254]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.1" found: no such container 2024-02-19T21:20:26.500 INFO:journalctl@ceph.osd.1.smithi163.stdout:Feb 19 21:20:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-1[53263]: debug 2024-02-19T21:20:26.091+0000 7f78fb8f1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-19T21:20:26.500 INFO:journalctl@ceph.osd.1.smithi163.stdout:Feb 19 21:20:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-1[53263]: debug 2024-02-19T21:20:26.091+0000 7f78fb8f1700 -1 osd.1 856 *** Got signal Terminated *** 2024-02-19T21:20:26.500 INFO:journalctl@ceph.osd.1.smithi163.stdout:Feb 19 21:20:26 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-1[53263]: debug 2024-02-19T21:20:26.091+0000 7f78fb8f1700 -1 osd.1 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T21:20:30.499 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:20:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:20:30.146Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused" 2024-02-19T21:20:30.500 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:20:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:20:30.147Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:20:31.750 INFO:journalctl@ceph.osd.1.smithi163.stdout:Feb 19 21:20:31 smithi163 bash[262291]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-1 2024-02-19T21:20:33.426 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.1.service' 2024-02-19T21:20:33.466 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T21:20:33.467 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-19T21:20:33.467 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-19T21:20:33.467 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.2 2024-02-19T21:20:33.750 INFO:journalctl@ceph.osd.2.smithi163.stdout:Feb 19 21:20:33 smithi163 systemd[1]: Stopping Ceph osd.2 for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:20:33.750 INFO:journalctl@ceph.osd.2.smithi163.stdout:Feb 19 21:20:33 smithi163 bash[262532]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.2" found: no such container 2024-02-19T21:20:34.249 INFO:journalctl@ceph.osd.2.smithi163.stdout:Feb 19 21:20:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-2[57708]: debug 2024-02-19T21:20:33.807+0000 7f5b50696700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-19T21:20:34.249 INFO:journalctl@ceph.osd.2.smithi163.stdout:Feb 19 21:20:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-2[57708]: debug 2024-02-19T21:20:33.807+0000 7f5b50696700 -1 osd.2 856 *** Got signal Terminated *** 2024-02-19T21:20:34.249 INFO:journalctl@ceph.osd.2.smithi163.stdout:Feb 19 21:20:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-2[57708]: debug 2024-02-19T21:20:33.807+0000 7f5b50696700 -1 osd.2 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T21:20:39.500 INFO:journalctl@ceph.osd.2.smithi163.stdout:Feb 19 21:20:39 smithi163 bash[262569]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-2 2024-02-19T21:20:40.250 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:20:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=error ts=2024-02-19T21:20:40.146Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:20:40.250 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:20:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:20:40.146Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused" 2024-02-19T21:20:40.250 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:20:40 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:20:40.147Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:20:40.920 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.2.service' 2024-02-19T21:20:40.950 INFO:journalctl@ceph.osd.2.smithi163.stdout:Feb 19 21:20:40 smithi163 systemd[1]: ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.2.service: Succeeded. 2024-02-19T21:20:40.950 INFO:journalctl@ceph.osd.2.smithi163.stdout:Feb 19 21:20:40 smithi163 systemd[1]: Stopped Ceph osd.2 for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T21:20:40.963 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T21:20:40.963 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-19T21:20:40.963 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-19T21:20:40.963 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.3 2024-02-19T21:20:41.249 INFO:journalctl@ceph.osd.3.smithi163.stdout:Feb 19 21:20:41 smithi163 systemd[1]: Stopping Ceph osd.3 for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:20:41.249 INFO:journalctl@ceph.osd.3.smithi163.stdout:Feb 19 21:20:41 smithi163 bash[262810]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.3" found: no such container 2024-02-19T21:20:41.750 INFO:journalctl@ceph.osd.3.smithi163.stdout:Feb 19 21:20:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-3[62371]: debug 2024-02-19T21:20:41.284+0000 7f397023e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-19T21:20:41.750 INFO:journalctl@ceph.osd.3.smithi163.stdout:Feb 19 21:20:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-3[62371]: debug 2024-02-19T21:20:41.284+0000 7f397023e700 -1 osd.3 856 *** Got signal Terminated *** 2024-02-19T21:20:41.750 INFO:journalctl@ceph.osd.3.smithi163.stdout:Feb 19 21:20:41 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-3[62371]: debug 2024-02-19T21:20:41.284+0000 7f397023e700 -1 osd.3 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T21:20:44.999 INFO:journalctl@ceph.osd.3.smithi163.stdout:Feb 19 21:20:44 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-3[62371]: debug 2024-02-19T21:20:44.583+0000 7f396d1bb700 -1 osd.3 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.107487+0000 front 2024-02-19T21:20:23.107612+0000 (oldest deadline 2024-02-19T21:20:44.207654+0000) 2024-02-19T21:20:45.999 INFO:journalctl@ceph.osd.3.smithi163.stdout:Feb 19 21:20:45 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-3[62371]: debug 2024-02-19T21:20:45.566+0000 7f396d1bb700 -1 osd.3 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.107487+0000 front 2024-02-19T21:20:23.107612+0000 (oldest deadline 2024-02-19T21:20:44.207654+0000) 2024-02-19T21:20:46.823 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:46.510+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:21.207479+0000 front 2024-02-19T21:20:21.207612+0000 (oldest deadline 2024-02-19T21:20:45.907494+0000) 2024-02-19T21:20:46.824 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:46 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:46.655+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:20:46.999 INFO:journalctl@ceph.osd.3.smithi163.stdout:Feb 19 21:20:46 smithi163 bash[262851]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-3 2024-02-19T21:20:47.823 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:47.477+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:21.207479+0000 front 2024-02-19T21:20:21.207612+0000 (oldest deadline 2024-02-19T21:20:45.907494+0000) 2024-02-19T21:20:47.823 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:47 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:47.613+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:20:48.498 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.3.service' 2024-02-19T21:20:48.540 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T21:20:48.540 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-19T21:20:48.540 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-19T21:20:48.540 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.4 2024-02-19T21:20:48.823 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:48.508+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:21.207479+0000 front 2024-02-19T21:20:21.207612+0000 (oldest deadline 2024-02-19T21:20:45.907494+0000) 2024-02-19T21:20:48.823 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:48.621+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:20:48.824 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 21:20:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-4[43484]: debug 2024-02-19T21:20:48.418+0000 7f914fdc4700 -1 osd.4 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.471466+0000 front 2024-02-19T21:20:22.471561+0000 (oldest deadline 2024-02-19T21:20:48.371503+0000) 2024-02-19T21:20:48.824 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 21:20:48 smithi193 systemd[1]: Stopping Ceph osd.4 for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:20:48.824 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 21:20:48 smithi193 bash[118872]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.4" found: no such container 2024-02-19T21:20:49.323 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 21:20:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-4[43484]: debug 2024-02-19T21:20:48.840+0000 7f9152e47700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-19T21:20:49.323 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 21:20:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-4[43484]: debug 2024-02-19T21:20:48.840+0000 7f9152e47700 -1 osd.4 856 *** Got signal Terminated *** 2024-02-19T21:20:49.323 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 21:20:48 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-4[43484]: debug 2024-02-19T21:20:48.840+0000 7f9152e47700 -1 osd.4 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T21:20:49.645 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:49.493+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:21.207479+0000 front 2024-02-19T21:20:21.207612+0000 (oldest deadline 2024-02-19T21:20:45.907494+0000) 2024-02-19T21:20:49.645 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 21:20:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-4[43484]: debug 2024-02-19T21:20:49.384+0000 7f914fdc4700 -1 osd.4 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.471466+0000 front 2024-02-19T21:20:22.471561+0000 (oldest deadline 2024-02-19T21:20:48.371503+0000) 2024-02-19T21:20:49.645 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:49.452+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:20:50.073 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:49.642+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:20:50.500 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:20:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=error ts=2024-02-19T21:20:50.147Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:20:50.500 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:20:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:20:50.148Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused" 2024-02-19T21:20:50.500 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:20:50 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:20:50.148Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:20:50.823 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:50.621+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:20:50.824 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:50.456+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:20:50.824 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 21:20:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-4[43484]: debug 2024-02-19T21:20:50.395+0000 7f914fdc4700 -1 osd.4 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.471466+0000 front 2024-02-19T21:20:22.471561+0000 (oldest deadline 2024-02-19T21:20:48.371503+0000) 2024-02-19T21:20:50.825 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:50 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:50.487+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:21.207479+0000 front 2024-02-19T21:20:21.207612+0000 (oldest deadline 2024-02-19T21:20:45.907494+0000) 2024-02-19T21:20:51.823 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:51.486+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:21.207479+0000 front 2024-02-19T21:20:21.207612+0000 (oldest deadline 2024-02-19T21:20:45.907494+0000) 2024-02-19T21:20:51.823 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:51.611+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:20:51.823 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:51.486+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:20:51.824 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 21:20:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-4[43484]: debug 2024-02-19T21:20:51.388+0000 7f914fdc4700 -1 osd.4 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.471466+0000 front 2024-02-19T21:20:22.471561+0000 (oldest deadline 2024-02-19T21:20:48.371503+0000) 2024-02-19T21:20:51.824 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 21:20:51 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-4[43484]: debug 2024-02-19T21:20:51.388+0000 7f914fdc4700 -1 osd.4 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:28.372133+0000 front 2024-02-19T21:20:28.372239+0000 (oldest deadline 2024-02-19T21:20:51.272174+0000) 2024-02-19T21:20:52.823 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:52.535+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:21.207479+0000 front 2024-02-19T21:20:21.207612+0000 (oldest deadline 2024-02-19T21:20:45.907494+0000) 2024-02-19T21:20:52.823 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 21:20:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-4[43484]: debug 2024-02-19T21:20:52.350+0000 7f914fdc4700 -1 osd.4 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.471466+0000 front 2024-02-19T21:20:22.471561+0000 (oldest deadline 2024-02-19T21:20:48.371503+0000) 2024-02-19T21:20:52.823 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 21:20:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-4[43484]: debug 2024-02-19T21:20:52.350+0000 7f914fdc4700 -1 osd.4 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:28.372133+0000 front 2024-02-19T21:20:28.372239+0000 (oldest deadline 2024-02-19T21:20:51.272174+0000) 2024-02-19T21:20:52.824 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:52.566+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:20:52.824 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:52.566+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:20:52.824 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:52.445+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:20:52.824 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:52 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:52.445+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:20:53.824 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:53.574+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:20:53.824 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:53.574+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:20:53.824 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 21:20:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-4[43484]: debug 2024-02-19T21:20:53.361+0000 7f914fdc4700 -1 osd.4 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.471466+0000 front 2024-02-19T21:20:22.471561+0000 (oldest deadline 2024-02-19T21:20:48.371503+0000) 2024-02-19T21:20:53.824 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 21:20:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-4[43484]: debug 2024-02-19T21:20:53.361+0000 7f914fdc4700 -1 osd.4 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:28.372133+0000 front 2024-02-19T21:20:28.372239+0000 (oldest deadline 2024-02-19T21:20:51.272174+0000) 2024-02-19T21:20:53.824 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:53.496+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:21.207479+0000 front 2024-02-19T21:20:21.207612+0000 (oldest deadline 2024-02-19T21:20:45.907494+0000) 2024-02-19T21:20:53.825 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:53.496+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:27.008604+0000 front 2024-02-19T21:20:27.008649+0000 (oldest deadline 2024-02-19T21:20:53.409141+0000) 2024-02-19T21:20:53.825 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:53.494+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:20:53.825 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:53 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:53.494+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:20:54.465 INFO:journalctl@ceph.osd.4.smithi193.stdout:Feb 19 21:20:54 smithi193 bash[118913]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-4 2024-02-19T21:20:54.823 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:54.462+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:21.207479+0000 front 2024-02-19T21:20:21.207612+0000 (oldest deadline 2024-02-19T21:20:45.907494+0000) 2024-02-19T21:20:54.823 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:54.462+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:27.008604+0000 front 2024-02-19T21:20:27.008649+0000 (oldest deadline 2024-02-19T21:20:53.409141+0000) 2024-02-19T21:20:54.823 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:54.477+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:20:54.823 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:54.477+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:20:54.824 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:54.613+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:20:54.824 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:54 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:54.613+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:20:55.669 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:55.467+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:21.207479+0000 front 2024-02-19T21:20:21.207612+0000 (oldest deadline 2024-02-19T21:20:45.907494+0000) 2024-02-19T21:20:55.669 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:55.467+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:27.008604+0000 front 2024-02-19T21:20:27.008649+0000 (oldest deadline 2024-02-19T21:20:53.409141+0000) 2024-02-19T21:20:55.669 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:55.466+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:20:55.669 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:55.466+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:20:55.669 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:55.582+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:20:55.669 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:55 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:55.582+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:20:56.062 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.4.service' 2024-02-19T21:20:56.102 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T21:20:56.102 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-19T21:20:56.102 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-19T21:20:56.103 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.5 2024-02-19T21:20:56.323 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:56 smithi193 systemd[1]: Stopping Ceph osd.5 for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:20:56.323 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:56 smithi193 bash[119150]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.5" found: no such container 2024-02-19T21:20:56.823 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:56.604+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:20:56.823 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:56.604+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:20:56.823 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:56.458+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:20:56.824 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:56.458+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:20:56.824 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:56.406+0000 7faad8632700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-19T21:20:56.824 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:56.406+0000 7faad8632700 -1 osd.5 856 *** Got signal Terminated *** 2024-02-19T21:20:56.824 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:56.406+0000 7faad8632700 -1 osd.5 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T21:20:56.824 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:56.499+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:21.207479+0000 front 2024-02-19T21:20:21.207612+0000 (oldest deadline 2024-02-19T21:20:45.907494+0000) 2024-02-19T21:20:56.824 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:56 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:56.499+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:27.008604+0000 front 2024-02-19T21:20:27.008649+0000 (oldest deadline 2024-02-19T21:20:53.409141+0000) 2024-02-19T21:20:57.823 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:57.462+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:21.207479+0000 front 2024-02-19T21:20:21.207612+0000 (oldest deadline 2024-02-19T21:20:45.907494+0000) 2024-02-19T21:20:57.823 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:57.462+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:27.008604+0000 front 2024-02-19T21:20:27.008649+0000 (oldest deadline 2024-02-19T21:20:53.409141+0000) 2024-02-19T21:20:57.824 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:57.615+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:20:57.824 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:57.615+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:20:57.824 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:57.457+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:20:57.824 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:57 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:57.457+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:20:58.823 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:58.485+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:21.207479+0000 front 2024-02-19T21:20:21.207612+0000 (oldest deadline 2024-02-19T21:20:45.907494+0000) 2024-02-19T21:20:58.823 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:58.485+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:27.008604+0000 front 2024-02-19T21:20:27.008649+0000 (oldest deadline 2024-02-19T21:20:53.409141+0000) 2024-02-19T21:20:58.823 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:58.635+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:20:58.824 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:58.635+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:20:58.824 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:58.466+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:20:58.824 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:58 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:58.466+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:20:59.823 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:59.454+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:21.207479+0000 front 2024-02-19T21:20:21.207612+0000 (oldest deadline 2024-02-19T21:20:45.907494+0000) 2024-02-19T21:20:59.823 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:20:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:20:59.454+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:27.008604+0000 front 2024-02-19T21:20:27.008649+0000 (oldest deadline 2024-02-19T21:20:53.409141+0000) 2024-02-19T21:20:59.824 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:59.600+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:20:59.824 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:20:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:20:59.600+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:20:59.824 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:59.500+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:20:59.824 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:20:59 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:20:59.500+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:00.500 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=error ts=2024-02-19T21:21:00.147Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:21:00.500 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:00.148Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused" 2024-02-19T21:21:00.500 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:00 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:00.148Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:21:00.823 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:21:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:21:00.485+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:21.207479+0000 front 2024-02-19T21:20:21.207612+0000 (oldest deadline 2024-02-19T21:20:45.907494+0000) 2024-02-19T21:21:00.823 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:21:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5[47880]: debug 2024-02-19T21:21:00.485+0000 7faad15a7700 -1 osd.5 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:27.008604+0000 front 2024-02-19T21:20:27.008649+0000 (oldest deadline 2024-02-19T21:20:53.409141+0000) 2024-02-19T21:21:00.823 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:00.634+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:21:00.824 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:00.634+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:21:00.824 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:00.533+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:00.824 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:00.533+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:00.824 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:00 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:00.533+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:01.823 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:01.591+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:21:01.823 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:01.591+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:21:01.823 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:01.567+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:01.823 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:01.567+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:01.823 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:01 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:01.567+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:02.323 INFO:journalctl@ceph.osd.5.smithi193.stdout:Feb 19 21:21:01 smithi193 bash[119191]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-5 2024-02-19T21:21:03.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:02.617+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:03.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:02.617+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:03.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:02.617+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:03.074 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:02.595+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:21:03.074 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:02 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:02.595+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:21:03.806 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.5.service' 2024-02-19T21:21:03.846 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:03.584+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:03.846 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:03.584+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:03.846 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:03.584+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:03.846 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:03.598+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:21:03.846 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:03.598+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:21:03.846 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:03 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:03.598+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:37.397667+0000 front 2024-02-19T21:20:37.397705+0000 (oldest deadline 2024-02-19T21:21:02.697627+0000) 2024-02-19T21:21:03.848 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T21:21:03.848 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-19T21:21:03.848 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-19T21:21:03.848 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.6 2024-02-19T21:21:03.999 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:03.585Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused" 2024-02-19T21:21:04.000 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:03 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:03.585Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:21:04.158 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:03 smithi193 systemd[1]: Stopping Ceph osd.6 for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:21:04.158 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:03 smithi193 bash[119424]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.6" found: no such container 2024-02-19T21:21:04.158 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:04.154+0000 7f824cb88700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-19T21:21:04.573 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:04.154+0000 7f824cb88700 -1 osd.6 856 *** Got signal Terminated *** 2024-02-19T21:21:04.573 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:04.154+0000 7f824cb88700 -1 osd.6 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T21:21:05.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:04.571+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:05.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:04.571+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:05.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:04.571+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:05.074 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:04.636+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:21:05.074 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:04.636+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:21:05.074 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:04 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:04.636+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:37.397667+0000 front 2024-02-19T21:20:37.397705+0000 (oldest deadline 2024-02-19T21:21:02.697627+0000) 2024-02-19T21:21:06.073 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:05.628+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:21:06.073 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:05.628+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:21:06.073 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:05.628+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:37.397667+0000 front 2024-02-19T21:20:37.397705+0000 (oldest deadline 2024-02-19T21:21:02.697627+0000) 2024-02-19T21:21:06.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:05.606+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:06.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:05.606+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:06.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:05 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:05.606+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:07.073 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:06.620+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:21:07.073 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:06.620+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:21:07.073 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:06.620+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:37.397667+0000 front 2024-02-19T21:20:37.397705+0000 (oldest deadline 2024-02-19T21:21:02.697627+0000) 2024-02-19T21:21:07.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:06.646+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:07.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:06.646+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:07.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:06 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:06.646+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:08.073 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:07.665+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:21:08.073 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:07.665+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:21:08.073 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:07.665+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:37.397667+0000 front 2024-02-19T21:20:37.397705+0000 (oldest deadline 2024-02-19T21:21:02.697627+0000) 2024-02-19T21:21:08.074 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:07.665+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-02-19T21:20:42.698136+0000 front 2024-02-19T21:20:42.698098+0000 (oldest deadline 2024-02-19T21:21:06.798147+0000) 2024-02-19T21:21:08.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:07.642+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:08.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:07.642+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:08.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:07.642+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:08.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:07 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:07.642+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-02-19T21:20:43.486920+0000 front 2024-02-19T21:20:43.486957+0000 (oldest deadline 2024-02-19T21:21:07.586906+0000) 2024-02-19T21:21:09.073 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:08.700+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:22.294887+0000 front 2024-02-19T21:20:22.294903+0000 (oldest deadline 2024-02-19T21:20:46.295483+0000) 2024-02-19T21:21:09.073 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:08.700+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:26.296098+0000 front 2024-02-19T21:20:26.296029+0000 (oldest deadline 2024-02-19T21:20:52.195958+0000) 2024-02-19T21:21:09.073 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:08.700+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:37.397667+0000 front 2024-02-19T21:20:37.397705+0000 (oldest deadline 2024-02-19T21:21:02.697627+0000) 2024-02-19T21:21:09.073 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6[52285]: debug 2024-02-19T21:21:08.700+0000 7f8248b03700 -1 osd.6 856 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-02-19T21:20:42.698136+0000 front 2024-02-19T21:20:42.698098+0000 (oldest deadline 2024-02-19T21:21:06.798147+0000) 2024-02-19T21:21:09.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:08.638+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:09.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:08.638+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:09.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:08.638+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:09.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:08 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:08.638+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-02-19T21:20:43.486920+0000 front 2024-02-19T21:20:43.486957+0000 (oldest deadline 2024-02-19T21:21:07.586906+0000) 2024-02-19T21:21:09.823 INFO:journalctl@ceph.osd.6.smithi193.stdout:Feb 19 21:21:09 smithi193 bash[119463]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-6 2024-02-19T21:21:09.823 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:09.683+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:09.823 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:09.683+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:09.824 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:09.683+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:09.824 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:09 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:09.683+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-02-19T21:20:43.486920+0000 front 2024-02-19T21:20:43.486957+0000 (oldest deadline 2024-02-19T21:21:07.586906+0000) 2024-02-19T21:21:10.500 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=error ts=2024-02-19T21:21:10.148Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:21:10.500 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:10.149Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused" 2024-02-19T21:21:10.500 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:10 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:10.149Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:21:11.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:10.681+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:11.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:10.681+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:11.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:10.681+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:11.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:10 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:10.681+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-02-19T21:20:43.486920+0000 front 2024-02-19T21:20:43.486957+0000 (oldest deadline 2024-02-19T21:21:07.586906+0000) 2024-02-19T21:21:11.579 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.6.service' 2024-02-19T21:21:11.620 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T21:21:11.620 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-19T21:21:11.620 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-19T21:21:11.620 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.7 2024-02-19T21:21:11.945 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:11.661+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:11.946 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:11.661+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:11.946 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:11.661+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:11.946 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:11.661+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-02-19T21:20:43.486920+0000 front 2024-02-19T21:20:43.486957+0000 (oldest deadline 2024-02-19T21:21:07.586906+0000) 2024-02-19T21:21:11.946 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:11 smithi193 systemd[1]: Stopping Ceph osd.7 for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:21:11.946 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:11 smithi193 bash[119699]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd.7" found: no such container 2024-02-19T21:21:11.946 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:11.943+0000 7fe7cb2e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-19T21:21:11.946 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:11.943+0000 7fe7cb2e0700 -1 osd.7 856 *** Got signal Terminated *** 2024-02-19T21:21:11.946 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:11 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:11.943+0000 7fe7cb2e0700 -1 osd.7 856 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T21:21:13.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:12.670+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:13.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:12.670+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:13.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:12.670+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:13.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:12 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:12.670+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-02-19T21:20:43.486920+0000 front 2024-02-19T21:20:43.486957+0000 (oldest deadline 2024-02-19T21:21:07.586906+0000) 2024-02-19T21:21:13.999 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=error ts=2024-02-19T21:21:13.584Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:21:13.999 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:13.585Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused" 2024-02-19T21:21:14.000 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:13 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:13.586Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:21:14.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:13.664+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:14.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:13.664+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:14.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:13.664+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:14.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:13 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:13.664+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-02-19T21:20:43.486920+0000 front 2024-02-19T21:20:43.486957+0000 (oldest deadline 2024-02-19T21:21:07.586906+0000) 2024-02-19T21:21:15.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:14.647+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:15.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:14.647+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:15.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:14.647+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:15.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:14.647+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-02-19T21:20:43.486920+0000 front 2024-02-19T21:20:43.486957+0000 (oldest deadline 2024-02-19T21:21:07.586906+0000) 2024-02-19T21:21:15.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:14 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:14.648+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.193:6802 osd.4 since back 2024-02-19T21:20:53.388208+0000 front 2024-02-19T21:20:53.388268+0000 (oldest deadline 2024-02-19T21:21:13.888341+0000) 2024-02-19T21:21:16.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:15.678+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:16.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:15.678+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:16.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:15.678+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:16.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:15.678+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-02-19T21:20:43.486920+0000 front 2024-02-19T21:20:43.486957+0000 (oldest deadline 2024-02-19T21:21:07.586906+0000) 2024-02-19T21:21:16.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:15 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:15.678+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.193:6802 osd.4 since back 2024-02-19T21:20:53.388208+0000 front 2024-02-19T21:20:53.388268+0000 (oldest deadline 2024-02-19T21:21:13.888341+0000) 2024-02-19T21:21:17.073 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:16.652+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6803 osd.0 since back 2024-02-19T21:20:23.283254+0000 front 2024-02-19T21:20:23.283362+0000 (oldest deadline 2024-02-19T21:20:48.583176+0000) 2024-02-19T21:21:17.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:16.652+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6807 osd.1 since back 2024-02-19T21:20:29.084175+0000 front 2024-02-19T21:20:29.084046+0000 (oldest deadline 2024-02-19T21:20:51.983993+0000) 2024-02-19T21:21:17.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:16.652+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6811 osd.2 since back 2024-02-19T21:20:38.285920+0000 front 2024-02-19T21:20:38.286007+0000 (oldest deadline 2024-02-19T21:20:59.985870+0000) 2024-02-19T21:21:17.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:16.652+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.163:6815 osd.3 since back 2024-02-19T21:20:43.486920+0000 front 2024-02-19T21:20:43.486957+0000 (oldest deadline 2024-02-19T21:21:07.586906+0000) 2024-02-19T21:21:17.074 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:16 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7[56915]: debug 2024-02-19T21:21:16.652+0000 7fe7c8a5e700 -1 osd.7 856 heartbeat_check: no reply from 172.21.15.193:6802 osd.4 since back 2024-02-19T21:20:53.388208+0000 front 2024-02-19T21:20:53.388268+0000 (oldest deadline 2024-02-19T21:21:13.888341+0000) 2024-02-19T21:21:17.573 INFO:journalctl@ceph.osd.7.smithi193.stdout:Feb 19 21:21:17 smithi193 bash[119737]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-osd-7 2024-02-19T21:21:19.173 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@osd.7.service' 2024-02-19T21:21:19.217 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T21:21:19.217 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-19T21:21:19.217 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-02-19T21:21:19.217 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-2591809a-cf69-11ee-95bb-87774f69a715@rgw.foo.a 2024-02-19T21:21:19.499 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Feb 19 21:21:19 smithi163 systemd[1]: Stopping Ceph rgw.foo.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:21:19.500 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Feb 19 21:21:19 smithi163 bash[263084]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-rgw.foo.a" found: no such container 2024-02-19T21:21:20.000 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Feb 19 21:21:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-rgw-foo-a[65209]: debug 2024-02-19T21:21:19.500+0000 7f828e72d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-02-19T21:21:20.000 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Feb 19 21:21:19 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-rgw-foo-a[65209]: debug 2024-02-19T21:21:19.500+0000 7f82a57f7880 -1 shutting down 2024-02-19T21:21:20.500 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=error ts=2024-02-19T21:21:20.149Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused" 2024-02-19T21:21:20.500 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:20.149Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused" 2024-02-19T21:21:20.500 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:20 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:20.149Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:21:24.000 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=error ts=2024-02-19T21:21:23.585Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:21:24.000 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:23.586Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused" 2024-02-19T21:21:24.000 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:23 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:23.586Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:21:29.830 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Feb 19 21:21:29 smithi163 bash[263125]: time="2024-02-19T21:21:29Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-2591809a-cf69-11ee-95bb-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-02-19T21:21:30.151 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Feb 19 21:21:30 smithi163 bash[263125]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-rgw-foo-a 2024-02-19T21:21:30.406 INFO:journalctl@ceph.rgw.foo.a.smithi163.stdout:Feb 19 21:21:30 smithi163 systemd[1]: ceph-2591809a-cf69-11ee-95bb-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-02-19T21:21:30.406 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=error ts=2024-02-19T21:21:30.149Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:21:30.406 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:30.150Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused" 2024-02-19T21:21:30.406 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:30 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:30.150Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:21:30.411 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@rgw.foo.a.service' 2024-02-19T21:21:30.447 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T21:21:30.448 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-02-19T21:21:30.448 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-19T21:21:30.448 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-2591809a-cf69-11ee-95bb-87774f69a715@prometheus.a 2024-02-19T21:21:30.745 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:21:30 smithi193 systemd[1]: Stopping Ceph prometheus.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:21:30.745 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:21:30 smithi193 bash[119975]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus.a" found: no such container 2024-02-19T21:21:31.055 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:21:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:21:30.743Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-19T21:21:31.056 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:21:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:21:30.743Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-19T21:21:31.056 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:21:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:21:30.743Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-19T21:21:31.056 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:21:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:21:30.743Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-19T21:21:31.056 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:21:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:21:30.743Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-19T21:21:31.056 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:21:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:21:30.743Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-19T21:21:31.056 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:21:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:21:30.743Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-19T21:21:31.056 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:21:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:21:30.743Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-19T21:21:31.056 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:21:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:21:30.743Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-19T21:21:31.056 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:21:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:21:30.743Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-19T21:21:31.056 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:21:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:21:30.743Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-19T21:21:31.056 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:21:30 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a[62647]: ts=2024-02-19T21:21:30.743Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-19T21:21:31.323 INFO:journalctl@ceph.prometheus.a.smithi193.stdout:Feb 19 21:21:31 smithi193 bash[120015]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-prometheus-a 2024-02-19T21:21:31.575 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2591809a-cf69-11ee-95bb-87774f69a715@prometheus.a.service' 2024-02-19T21:21:31.609 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T21:21:31.610 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-19T21:21:31.610 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2591809a-cf69-11ee-95bb-87774f69a715 --force --keep-logs 2024-02-19T21:21:31.902 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:31.625Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi163.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.163:8443: connect: connection refused" 2024-02-19T21:21:31.902 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:31 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=warn ts=2024-02-19T21:21:31.626Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi193.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.193:8443: connect: connection refused" 2024-02-19T21:21:33.000 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:32 smithi163 systemd[1]: Stopping Ceph alertmanager.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:21:33.000 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:32 smithi163 bash[263550]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager.a" found: no such container 2024-02-19T21:21:33.343 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:33 smithi163 ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a[72084]: level=info ts=2024-02-19T21:21:33.004Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-19T21:21:33.750 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:33 smithi163 bash[263591]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-alertmanager-a 2024-02-19T21:21:34.250 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:21:34 smithi163 systemd[1]: Stopping Ceph node-exporter.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:21:34.250 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:21:34 smithi163 bash[263695]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter.a" found: no such container 2024-02-19T21:21:34.251 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:33 smithi163 systemd[1]: ceph-2591809a-cf69-11ee-95bb-87774f69a715@alertmanager.a.service: Succeeded. 2024-02-19T21:21:34.251 INFO:journalctl@ceph.alertmanager.a.smithi163.stdout:Feb 19 21:21:33 smithi163 systemd[1]: Stopped Ceph alertmanager.a for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T21:21:34.899 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:21:34 smithi163 bash[263735]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-a 2024-02-19T21:21:35.200 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:21:34 smithi163 systemd[1]: ceph-2591809a-cf69-11ee-95bb-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-19T21:21:35.200 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:21:35 smithi163 systemd[1]: ceph-2591809a-cf69-11ee-95bb-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-19T21:21:35.200 INFO:journalctl@ceph.node-exporter.a.smithi163.stdout:Feb 19 21:21:35 smithi163 systemd[1]: Stopped Ceph node-exporter.a for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T21:21:35.438 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2591809a-cf69-11ee-95bb-87774f69a715 --force --keep-logs 2024-02-19T21:21:36.649 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:21:36 smithi193 systemd[1]: Stopping Ceph iscsi.iscsi.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:21:36.649 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:21:36 smithi193 bash[120379]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi.iscsi.a" found: no such container 2024-02-19T21:21:37.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:21:36 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a[60486]: debug Shutdown received 2024-02-19T21:21:46.950 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:21:46 smithi193 bash[120417]: time="2024-02-19T21:21:46Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-02-19T21:21:47.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:21:47 smithi193 bash[120417]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a 2024-02-19T21:21:47.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:21:47 smithi193 systemd[1]: ceph-2591809a-cf69-11ee-95bb-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-02-19T21:21:48.165 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:21:48 smithi193 systemd[1]: Stopping Ceph node-exporter.b for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:21:48.165 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:21:47 smithi193 bash[120491]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-iscsi-iscsi-a-tcmu 2024-02-19T21:21:48.165 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:21:47 smithi193 systemd[1]: ceph-2591809a-cf69-11ee-95bb-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-02-19T21:21:48.165 INFO:journalctl@ceph.iscsi.iscsi.a.smithi193.stdout:Feb 19 21:21:47 smithi193 systemd[1]: Stopped Ceph iscsi.iscsi.a for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T21:21:48.573 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:21:48 smithi193 bash[120568]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter.b" found: no such container 2024-02-19T21:21:48.943 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:21:48 smithi193 bash[120609]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-node-exporter-b 2024-02-19T21:21:49.226 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:21:48 smithi193 systemd[1]: ceph-2591809a-cf69-11ee-95bb-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-19T21:21:49.227 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:21:49 smithi193 systemd[1]: ceph-2591809a-cf69-11ee-95bb-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-19T21:21:49.227 INFO:journalctl@ceph.node-exporter.b.smithi193.stdout:Feb 19 21:21:49 smithi193 systemd[1]: Stopped Ceph node-exporter.b for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T21:21:49.573 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:21:49 smithi193 systemd[1]: Stopping Ceph grafana.a for 2591809a-cf69-11ee-95bb-87774f69a715... 2024-02-19T21:21:49.573 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:21:49 smithi193 bash[120745]: Error: no container with name or ID "ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana.a" found: no such container 2024-02-19T21:21:49.903 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:21:49 smithi193 ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a[63013]: t=2024-02-19T21:21:49+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-19T21:21:50.220 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:21:49 smithi193 bash[120785]: ceph-2591809a-cf69-11ee-95bb-87774f69a715-grafana-a 2024-02-19T21:21:50.477 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:21:50 smithi193 systemd[1]: ceph-2591809a-cf69-11ee-95bb-87774f69a715@grafana.a.service: Succeeded. 2024-02-19T21:21:50.477 INFO:journalctl@ceph.grafana.a.smithi193.stdout:Feb 19 21:21:50 smithi193 systemd[1]: Stopped Ceph grafana.a for 2591809a-cf69-11ee-95bb-87774f69a715. 2024-02-19T21:21:50.723 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-19T21:21:50.761 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-19T21:21:50.791 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-19T21:21:50.792 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566727/remote/ubuntu@smithi163.front.sepia.ceph.com/crash 2024-02-19T21:21:50.793 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/crash -- . 2024-02-19T21:21:50.840 INFO:teuthology.orchestra.run.smithi163.stderr:tar: /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/crash: Cannot open: No such file or directory 2024-02-19T21:21:50.840 INFO:teuthology.orchestra.run.smithi163.stderr:tar: Error is not recoverable: exiting now 2024-02-19T21:21:50.842 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566727/remote/ubuntu@smithi193.front.sepia.ceph.com/crash 2024-02-19T21:21:50.843 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/crash -- . 2024-02-19T21:21:50.875 INFO:teuthology.orchestra.run.smithi193.stderr:tar: /var/lib/ceph/2591809a-cf69-11ee-95bb-87774f69a715/crash: Cannot open: No such file or directory 2024-02-19T21:21:50.875 INFO:teuthology.orchestra.run.smithi193.stderr:tar: Error is not recoverable: exiting now 2024-02-19T21:21:50.876 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-19T21:21:50.877 INFO:tasks.cephadm:Removing cluster... 2024-02-19T21:21:50.877 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2591809a-cf69-11ee-95bb-87774f69a715 --force 2024-02-19T21:21:51.724 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2591809a-cf69-11ee-95bb-87774f69a715 --force 2024-02-19T21:21:52.533 INFO:tasks.cephadm:Removing cephadm ... 2024-02-19T21:21:52.533 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-19T21:21:52.555 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-19T21:21:52.574 INFO:tasks.cephadm:Teardown complete 2024-02-19T21:21:52.574 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1508, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 238, in ceph_log if first_in_ceph_log('\[ERR\]|\[WRN\]|\[SEC\]', File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 229, in first_in_ceph_log r = ctx.ceph[cluster_name].bootstrap_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 447, in run r = RemoteProcess(client, args, check_status=check_status, hostname=name, File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 65, in __init__ self.command = quote(args) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 257, in quote return ' '.join(_quote(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 255, in _quote yield pipes.quote(a) File "/usr/lib/python3.8/shlex.py", line 325, in quote if _find_unsafe(s) is None: TypeError: expected string or bytes-like object 2024-02-19T21:21:52.577 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-19T21:21:52.587 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1508, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 238, in ceph_log if first_in_ceph_log('\[ERR\]|\[WRN\]|\[SEC\]', File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 229, in first_in_ceph_log r = ctx.ceph[cluster_name].bootstrap_remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 447, in run r = RemoteProcess(client, args, check_status=check_status, hostname=name, File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 65, in __init__ self.command = quote(args) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 257, in quote return ' '.join(_quote(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 255, in _quote yield pipes.quote(a) File "/usr/lib/python3.8/shlex.py", line 325, in quote if _find_unsafe(s) is None: TypeError: expected string or bytes-like object 2024-02-19T21:21:52.587 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-19T21:21:52.587 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T21:21:52.597 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T21:21:52.690 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-19T21:21:52.699 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-19T21:21:52.700 DEBUG:teuthology.orchestra.run.smithi163:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T21:21:52.702 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T21:21:52.719 INFO:teuthology.orchestra.run.smithi193.stderr:bash: ntpq: command not found 2024-02-19T21:21:52.720 INFO:teuthology.orchestra.run.smithi163.stderr:bash: ntpq: command not found 2024-02-19T21:21:52.724 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T21:21:52.724 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-02-19T21:21:52.724 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 60 +573us[ +563us] +/- 93ms 2024-02-19T21:21:52.724 INFO:teuthology.orchestra.run.smithi193.stdout:^* hv02.front.sepia.ceph.com 3 7 377 59 -586us[ -597us] +/- 73ms 2024-02-19T21:21:52.724 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +263us[ +263us] +/- 78ms 2024-02-19T21:21:52.724 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:21:52.725 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T21:21:52.725 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-02-19T21:21:52.725 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 +553us[ +553us] +/- 93ms 2024-02-19T21:21:52.725 INFO:teuthology.orchestra.run.smithi163.stdout:^* hv02.front.sepia.ceph.com 3 6 377 57 -639us[ -632us] +/- 73ms 2024-02-19T21:21:52.725 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 +246us[ +246us] +/- 78ms 2024-02-19T21:21:52.725 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T21:21:52.726 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-19T21:21:52.759 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-19T21:21:52.759 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-19T21:21:52.768 DEBUG:teuthology.orchestra.run.smithi163:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T21:21:52.864 DEBUG:teuthology.orchestra.run.smithi193:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T21:21:52.947 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T21:21:52.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:21:52.980 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T21:21:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:21:53.012 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-19T21:21:53.023 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-19T21:21:53.138 INFO:teuthology.task.internal:Duration was 2220.062752 seconds 2024-02-19T21:21:53.138 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-19T21:21:53.154 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-19T21:21:53.155 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T21:21:53.157 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T21:21:53.202 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T21:21:53.203 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T21:21:53.583 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-19T21:21:53.583 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2024-02-19T21:21:53.583 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T21:21:53.607 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2024-02-19T21:21:53.607 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T21:21:53.631 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-19T21:21:53.632 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T21:21:53.650 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T21:21:53.765 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-19T21:21:53.765 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T21:21:53.767 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T21:21:54.061 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-19T21:21:54.072 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-19T21:21:54.072 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T21:21:54.099 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T21:21:54.127 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-19T21:21:54.225 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T21:21:54.228 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T21:21:54.251 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2024-02-19T21:21:54.253 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2024-02-19T21:21:54.280 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T21:21:54.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:21:54.323 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T21:21:54.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:21:54.340 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-19T21:21:54.350 INFO:teuthology.task.internal:Transferring archived files... 2024-02-19T21:21:54.351 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566727/remote/smithi163 2024-02-19T21:21:54.352 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T21:21:54.426 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566727/remote/smithi193 2024-02-19T21:21:54.427 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T21:21:54.475 INFO:teuthology.task.internal:Removing archive directory... 2024-02-19T21:21:54.475 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T21:21:54.478 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T21:21:54.524 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-19T21:21:54.535 INFO:teuthology.task.internal:Not uploading archives. 2024-02-19T21:21:54.535 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-19T21:21:54.562 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-19T21:21:54.563 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T21:21:54.565 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T21:21:54.580 INFO:teuthology.orchestra.run.smithi163.stdout: 525520 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 21:21 /home/ubuntu/cephtest 2024-02-19T21:21:54.583 INFO:teuthology.orchestra.run.smithi193.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 21:21 /home/ubuntu/cephtest 2024-02-19T21:21:54.584 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-19T21:21:54.611 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-19T21:21:54.662 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: TypeError: expected string or bytes-like object 2024-02-19T21:21:54.663 INFO:teuthology.nuke:Checking targets against current locks 2024-02-19T21:21:54.690 DEBUG:teuthology.nuke:shortname: smithi163 2024-02-19T21:21:54.691 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T21:21:54.713 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566727', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 20:35:52.352211', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKXeXo44orPSizxUoHEwGL0jM2Z+DSjc1UXCU9yhbhT8G1mP2VXCzJWMGf23UNEZln9z9RXM6QIFHFdgLfHOolQ='} 2024-02-19T21:21:54.716 DEBUG:teuthology.nuke:shortname: smithi193 2024-02-19T21:21:54.717 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T21:21:54.740 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566727', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 20:35:52.350719', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA6jJ4zYMBuiRZrks/R4fN/SUBXMxsHBdM6sjXLOjLfzYKXFLDmg21GyuY++/Envxy0Q870W5QaMOmuxbSLPpmg='} 2024-02-19T21:21:54.769 INFO:teuthology.orchestra.console.smithi163:Power off 2024-02-19T21:21:54.770 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T21:21:54.794 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off 2024-02-19T21:21:54.794 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T21:21:54.809 INFO:teuthology.orchestra.console.smithi193:Power off 2024-02-19T21:21:54.809 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T21:21:54.817 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-02-19T21:21:54.831 DEBUG:teuthology.orchestra.console.smithi193:power off output: Chassis Power Control: Down/Off 2024-02-19T21:21:54.831 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T21:21:54.853 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-02-19T21:21:58.820 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T21:21:58.854 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T21:21:58.931 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-02-19T21:21:58.964 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-02-19T21:22:02.936 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T21:22:02.965 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T21:22:03.047 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-02-19T21:22:03.075 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is off 2024-02-19T21:22:03.176 INFO:teuthology.orchestra.console.smithi193:Power off completed 2024-02-19T21:22:07.051 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T21:22:07.162 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off 2024-02-19T21:22:07.262 INFO:teuthology.orchestra.console.smithi163:Power off completed 2024-02-19T21:22:07.366 INFO:teuthology.run:Summary data: description: rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} duration: 2220.0627522468567 failure_reason: expected string or bytes-like object flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-02-19T21:22:07.366 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T21:22:07.462 INFO:teuthology.run:FAIL